Distributed locks and fencing tokens: reject work from an expired owner

TL;DR: A lease coordinates ownership, but expiry cannot stop an old process from running. Protect the effect at its destination with an ordered fencing token, checked atomically with the write, when a stale owner could corrupt newer work.

An expired lease does not terminate a process

Worker A acquires a 30-second lease to refresh a report. It pauses for 45 seconds because its runtime or host stops scheduling it. After the lease expires, worker B acquires ownership and writes a fresh report. A resumes with the old result still in memory and overwrites B.

The lock service may have behaved correctly throughout. The failed assumption was that A could not act after its lease expired. A local “still valid” check also has a gap: the process can pause between checking and writing. Network delays create the same stale-arrival problem without a long runtime pause.

The etcd lock API returns an ownership key that can be used with transactions inside etcd. That does not automatically authorize or fence a write to a separate database. The system performing the external effect needs its own enforceable condition.

The diagram begins after ownership transfer. Token issuance must itself establish a reliable order; timestamps from unrelated worker clocks do not supply that guarantee.

Put the rejection at the write boundary

A fencing token is an increasing ownership generation issued by a trusted authority. The protected resource remembers the latest accepted generation and rejects older ones. If token 42 has been accepted, a later request with token 41 cannot undo its effect.

Store the comparison and mutation atomically. Reading last_token, checking it in application code and then doing an unconditional write leaves a race between the check and the mutation. Every competing writer must use the protected path. A legacy maintenance job with unrestricted writes can invalidate the guarantee.

The Redis distributed-lock guidance discusses fencing and the limitations of TTL assumptions. A random ownership value useful for safe lock release is not automatically an ordered fencing token. The two values answer different questions: “is this my lock?” and “is this owner newer?”

Execute the stale-write case locally

This Python example uses an in-memory SQLite database. The conditional update is a real atomic database statement; tokens 41 and 42 are manually assigned for the exercise. It demonstrates destination enforcement, not a distributed lock implementation.

import sqlite3

with sqlite3.connect(":memory:") as db:
    db.execute("CREATE TABLE report (id INTEGER PRIMARY KEY, token INTEGER, body TEXT)")
    db.execute("INSERT INTO report VALUES (1, 0, 'initial')")

    def publish(token, body):
        changed = db.execute(
            "UPDATE report SET token = ?, body = ? WHERE id = 1 AND token < ?",
            (token, body, token),
        ).rowcount
        return changed == 1

    assert publish(42, "new report")
    assert not publish(41, "stale report")
    assert not publish(42, "same generation again")
    assert db.execute("SELECT token, body FROM report").fetchone() == (42, "new report")

The strict comparison permits one accepted publication per generation. If a legitimate owner needs multiple writes, define an additional per-owner sequence or a protocol that allows equal generations while rejecting older ones. Do not change < to <= without explaining duplicate requests and ordering within the same generation. SQLite's transaction documentation describes its write isolation; a production design must match its actual database and failure model.

Explain the remaining failure cases

An old token may arrive before any newer token has reached storage. A basic “highest accepted token” check can accept that write. Fencing guarantees rejection after the newer generation has established itself at the destination; a stronger requirement to reject immediately upon lease expiry requires tighter integration between ownership and the protected resource.

Persist the remembered generation across destination restarts. Token authority recovery must not reuse old generations. Scope tokens consistently to the protected resource, and authenticate callers so a client cannot choose an arbitrarily large number to lock everyone else out.

For irreversible external actions such as sending a payment, prefer the provider's transactional/idempotency interface where available. A lock around an HTTP call cannot undo an already accepted effect. Connect the design to idempotency rather than claiming exactly-once behavior from mutual exclusion alone.

Self-check: A has token 41, B acquires 42, but storage has only seen token 40. A's write arrives first. Must a highest-seen-token guard reject it? No: 41 is newer than 40. Explain that boundary explicitly, then decide whether the business requirement needs B to establish its generation before proceeding or needs ownership checks in the same transaction as the effect.