Engineering problem

Redis Distributed Locks: Redlock and Fencing Tokens

Decide when a Redis lock is sufficient and when the resource needs fencing, database constraints or idempotency. Test lease expiry and failover safely.

Editorial update

Decision in brief

Use a time-bounded Redis lease to reduce duplicate work when overlap is tolerable. Protect correctness-critical writes in the system of record with a transaction, a durable idempotency key or a fencing check. An expired lease holder can resume and issue a write even after another client has acquired the lock.

Protect critical writes: Redis lease — Reduce overlap. System of record — Protect writes. Expiry — Test a late writer.
A worker can resume after its lease expires. Use a transaction, durable idempotency key or fencing check where correctness matters.

Choose by the cost of duplicate work

List the side effects and identify which system can reject duplicates or stale writes. This determines whether you need a lease, atomic deduplication, a transactional invariant or an ordering mechanism.

WorkloadStart withPass condition
Cache refresh with tolerable overlapRedis lease with a TTL and bounded retry with jitterParallel refreshes may waste work. They cannot corrupt the cached representation or exhaust the origin budget.
One-time database transition or event recordingUnique operation ID or conditional update in the transaction which changes stateAfter concurrent retry or lost response, one operation ID still maps to one durable result.
Independent workers writing to a shared resource with ordered ownershipLease from a coordinator that can issue monotonic fencing tokens, enforced atomically by the targetA delayed writer carrying an older token is rejected after the target observes a newer one.
External side effect such as charging or sendingUse the destination's idempotency support and persist a durable operation identity plus reconciliation stateRetry an ambiguous response with the same key. Verify one logical effect and a traceable final state.

Lease ownership, expiry and Redis failover

Acquire a single-instance Redis lease with SET key unique-token NX PX ttl. Release it with an atomic compare-and-delete operation, or a Lua script that compares the stored token with the caller's token before deleting. Extend only if ownership still matches. A separate GET and DEL can remove a lease acquired by a successor.

Redis asynchronous replication permits a lease write to disappear when a replica is promoted before receiving it. Redlock acquires a quorum over independent Redis masters within a bounded validity window. Its reasoning depends on timing assumptions and the holder stopping work before its validity expires. A process pause or delayed network request can carry an already-issued operation beyond that window.

A random ownership token only identifies the lease to release. For fencing, each ownership change receives a strictly increasing token associated with the resource. The destination atomically checks that token against its stored high-water mark and applies the write with the check. If several valid writes share a token, define intra-owner sequencing or idempotency too. A newer owner may need to register its token before depending on rejection of delayed writes.

What Hacker News cases add

In birdculture's thread about garbage collection pauses, pestatije challenged using a TTL for mutual exclusion. Reproduce that scenario by pausing a client across expiry before allowing its pending write to reach a mock destination.

MeterFlow author ajithmanmu described using SET NX to deduplicate incoming events. It is a different use of the atomic primitive from carrying exclusive ownership across an external operation. Test deduplication across key expiry and failover before choosing where the permanent idempotency record lives.

Test a stale writer and an ambiguous response

Run two workers against a disposable Redis instance and an isolated mock side-effect service or staging table. Record lease token, fencing token, attempt ID, commit time and result into an append-only test log.

  1. Have A acquire a short lease and pause before committing. Let B acquire after expiry and commit. Resume A and assert its stale mutation and stale release both fail to affect B's state.
  2. Using a separate disposable primary and replica, delay replication, write a lease and promote the stale replica. Confirm duplicate lease acquisition is possible and demonstrate that the destination's invariant still holds.
  3. Have the mock destination apply a request and drop the response. Retry with the same durable operation ID and verify one effective write. Review contention waits, retry volume and p95 latency.

Related articles