Redis Locks Without Fencing Tokens Are Theatre

The lock expired while the worker slept

A warehouse allocation worker holds a Redis lock while updating stock levels in PostgreSQL. A long garbage-collection pause stops the process for forty-five seconds. The lock TTL is thirty seconds. Redis expires the key; another pod acquires the lock and completes the shipment. The first worker wakes, believes it still owns the lock because its local token says yes, and decrements inventory again. Duplicate picks are rare at 0.3% of jobs but expensive at scale. The team copied a Redlock snippet from a blog post and skipped fencing tokens at the storage layer.

Problem space: locks, clocks, and stale holders

Distributed locks coordinate exclusive work across .NET workers backed by Redis. Common use cases: nightly aggregation, single-flight cache refresh, invoice numbering, and outbox relay leadership. Redis provides low-latency compare-and-set with TTL, which solves crash safety when a worker dies without releasing the lock. TTL does not solve process pause: the holder can outlive the key and write anyway unless the downstream store rejects stale tokens.

Quantified anchors from production incident reviews:

• Lock TTL 30 s, GC pause p99 45 s on large-object heaps → stale holder window ~15 s
• Duplicate side-effect rate 0.3% on unfenced inventory writes vs zero observed duplicates after fencing in thirty-day soak
• Lock acquisition p95 4 ms on local Redis; not the bottleneck—correctness is

Why the naive Redis lock fails

The pattern below appears in internal libraries and Stack Overflow answers. It acquires a key with SET NX EX and deletes on dispose. There is no monotonic fence passed to the database write.

public async Task<bool> TryAcquireAsync(string resource, TimeSpan ttl)
{
    _token = Guid.NewGuid().ToString("N");
    return await _db.StringSetAsync(
        $"lock:{resource}", _token, ttl, When.NotExists);
}

public async Task ReleaseAsync(string resource)
{
    const string lua = """
        if redis.call('get', KEYS[1]) == ARGV[1] then
            return redis.call('del', KEYS[1])
        else return 0 end
        """;
    await _db.ScriptEvaluateAsync(lua,
        new RedisKey[] { $"lock:{resource}" },
        new RedisValue[] { _token });
}

What breaks is not “Redis is down” but ordering after expiry. Martin Kleppmann’s analysis of how to do distributed locking shows that a client can believe it holds a lock after the key expired; only the storage system that accepts the write can reject stale leaders. Redis documentation on distributed lock patterns emphasises TTL tuning and safe release; fencing closes the pause gap TTL cannot cover.

Engineered approach: lock plus fencing token

Increment a global fence key in Redis when acquiring the lock. Pass the returned integer to every write the lock protects. PostgreSQL, DynamoDB conditional writes, and many ORMs support comparing a fence column before update.

public async Task<LockHandle?> AcquireAsync(string resource, TimeSpan ttl)
{
    _token = Guid.NewGuid().ToString("N");
    var acquired = await _db.StringSetAsync(
        $"lock:{resource}", _token, ttl, When.NotExists);
    if (!acquired) return null;

    var fence = await _db.StringIncrementAsync($"fence:{resource}");
    return new LockHandle(resource, _token, fence);
}

public async Task<bool> WriteStockAsync(
    LockHandle handle, string sku, int delta, CancellationToken ct)
{
    // Storage layer rejects writes where fence < current max for sku
    const string sql = """
        UPDATE inventory
        SET qty = qty + @delta, fence = @fence
        WHERE sku = @sku AND fence < @fence
        """;
    var rows = await _db.ExecuteAsync(sql,
        new { sku, delta, fence = handle.Fence });
    return rows == 1;
}

The fence monotonically increases on each successful acquisition. A stale worker carries fence value 41 while the database already accepted fence 42 from the new holder; its update affects zero rows. Application logs should treat zero-row updates after lock acquisition as a fencing rejection, not silent success.

Redis key layout and TTL discipline

SET lock:alloc:WH-01 <token> NX EX 30
INCR fence:alloc:WH-01
GET lock:alloc:WH-01

TTL should exceed p99 handler time but not be so long that crash recovery stalls. Extend locks with a watchdog only when profiling proves handler duration variance; watchdogs add failure modes if the extend loop dies. Prefer shorter jobs and idempotent writes with fencing over infinite lease renewal complexity.

Redlock across independent Redis masters targets a different failure model (clock drift, quorum). Many .NET teams run a single highly available Redis cluster; fencing at the writer is still mandatory because any lock can expire under pause. Debating Redlock theology does not remove the stale-holder problem on one primary.

Operational proof

Validate with chaos rather than code review alone:

1. Inject stop-the-world GC or SIGSTOP on a holder mid-transaction; confirm second acquirer proceeds.
2. Resume first worker; assert inventory row unchanged by stale write and logs show fence rejection.
3. Load-test five hundred acquisitions per second; verify fence counter gap-free and no duplicate shipments in downstream audit table.
4. Alert when fence rejections exceed baseline; sudden spikes indicate TTL too aggressive or handler latency regression.

Benchmark method: compare duplicate-write count over one million synthetic allocation jobs with and without fence column enforced at SQL layer. Expect zero duplicates with fencing if the conditional update is on the critical path, not in a best-effort branch.

Lock renewal vs shorter jobs

Teams sometimes add a background timer that extends TTL every ten seconds while work runs. Renewal loops fail silently if the process hangs without crashing: the timer stops, TTL expires, a competitor acquires, and the hung process may resume without knowing renewal stopped. Fencing still saves you, but renewal complexity hides health issues. Prefer splitting work into chunks that finish under half the TTL, acking progress in the database with the current fence value stored on the job row.

If renewal is unavoidable, extend only when a heartbeat proves forward progress (rows processed incrementing). Never renew on a timer alone. Log renewal failures at error level; they predict duplicate work within one TTL window.

Testing matrix for .NET worker services

Unit tests with an in-memory Redis mock prove acquisition and release Lua correctness. They do not prove pause behaviour. Minimum bar before production:

• Integration test with real Redis: two processes compete for one resource; exactly one write succeeds.
• Chaos test: kill holder with SIGKILL; second worker completes within TTL plus one acquisition attempt.
• Pause test: Thread.Sleep inside holder beyond TTL; assert fenced write returns false and inventory unchanged.
• Metrics test: expose lock_fence_rejected_total counter; alert if rate exceeds zero sustained during steady state (may indicate TTL too tight).

Load generators should simulate realistic handler times, not microsecond-critical sections. Locks behave differently when handlers wait on HTTP at two hundred milliseconds p95 versus in-memory work at two milliseconds.

Alternatives when fencing is impossible

Some legacy stores offer no conditional column update. Options ranked by preference: migrate the critical write to a table you control with fence column; use compare-and-swap on a version column if available; defer to saga pattern with compensating transactions instead of exclusive lock. Last resort: shorten TTL and accept occasional duplicate detection via business-key unique index, treating lock as optimisation only. Calling that a distributed lock in runbooks is dishonest.

Single-primary Redis and HA expectations

Most .NET estates use one Redis primary with replica for read scaling, not five independent masters. Fencing does not require Redlock’s quorum math; it requires the writer to know the latest fence integer. During failover, briefly reject writes if fence increment and lock acquisition cannot both complete; handlers should retry with backoff rather than proceed without a fence. Document RTO for lock operations separately from cache RTO because coordination outages stall workers differently than cache misses.

Monitor memory pressure on the Redis node holding lock keys. Eviction policies must never evict lock or fence keys; use separate logical databases or key prefixes with noeviction maxmemory policy on coordination instances. Mixing session cache and locks on one evicting instance is a production incident waiting for traffic peak.

Document lock key naming conventions in the platform runbook: lock:{domain}:{aggregateId} paired with fence:{domain}:{aggregateId}. Ad-hoc key strings from each microservice make incidents harder when operators must inspect Redis during a duplicate-write investigation. Centralise the client library; forbid raw SET NX copy-paste in application repos.

When to skip locks entirely

Prefer database unique constraints and idempotent message IDs when the operation is a single-row upsert keyed by business ID. Locks justify themselves when read-modify-write spans multiple rows or external APIs without native idempotency. If you must lock, fence the writer.

Redis locks without fencing tokens look like coordination in demos and like duplicate shipments in production. Teams hardening .NET workers on Redis and PostgreSQL often want a short architecture review on lock scope, TTL metrics, and fence placement before copying another Redlock gist.

Our custom software development practice implements these patterns in worker services and documents the chaos tests that prove they hold under GC pauses, not just under sunny-day unit tests.