Redis Cache Stampede: Fix It Before Traffic Peaks

Redis cache-aside is the default pattern on .NET APIs for good reason: it is simple, observable, and fast when keys are warm. It has a well-known failure mode that only appears under load. When a hot key’s TTL expires, every in-flight request sees a miss at once and fans out to the origin database. That is a cache stampede, and it turns a sub-millisecond Redis read into hundreds of concurrent SQL queries. Singleflight coalescing fixes it without abandoning TTL-based freshness.

Problem space: when p95 lies

Consider a product catalogue API serving 2,400 requests per second at peak, with 180 hot SKUs cached in Redis at a five-minute TTL. Median latency sits at 4 ms while keys are warm. At TTL boundary, p95 jumps to 420 ms and database connection count spikes from 20 to 180 within two seconds. The origin is healthy; the cache layer orchestrated a thundering herd.

Redis documents cache-aside clearly in Cache-aside pattern: application reads Redis first, on miss loads from DB, then writes the value back. The doc assumes polite traffic. Production traffic is not polite at expiry boundaries, especially when marketing campaigns align on the hour or cron jobs refresh related keys simultaneously.

• Hot key — one logical entity (product, config blob, feature flag bundle) referenced by most requests.
• Aligned expiry — identical TTL set at deploy or batch warm means keys vanish together.
• Miss amplification — N concurrent misses produce N origin fetches for the same payload.

Why the naive cache-aside fails

The naive implementation is textbook correct and operationally dangerous:

public async Task<ProductDto?> GetProductAsync(string sku, CancellationToken ct)
{
    var cacheKey = $"product:{sku}";
    var cached = await _redis.StringGetAsync(cacheKey);
    if (cached.HasValue)
        return JsonSerializer.Deserialize<ProductDto>(cached!);

    // Stampede: every miss runs this concurrently for the same sku
    var product = await _db.Products.AsNoTracking()
        .FirstOrDefaultAsync(p => p.Sku == sku, ct);
    if (product is null)
        return null;

    var json = JsonSerializer.Serialize(product);
    await _redis.StringSetAsync(cacheKey, json, TimeSpan.FromMinutes(5));
    return product;
}

Under normal load this looks fine in benchmarks because testers use staggered requests. Under expiry, the thread pool fills with identical queries. Postgres pg_stat_activity shows the same SELECT repeated dozens of times. CPU on the API tier rises because JSON serialisation runs in parallel for the same object. Redis write pressure spikes as every thread attempts SET with the same value.

Common mitigations that do not fully solve the problem:

• Probabilistic early expiration — spreads expiry but does not coalesce the first miss wave after a cold start.
• Mutex per key with naive SETNX — helps only if the loser waits and retries; without singleflight semantics losers still hammer the DB on retry timing races.
• Longer TTL — hides the problem until data staleness becomes a product bug.

Engineered approach: singleflight in process and in Redis

Singleflight means: for a given key, at most one origin fetch runs; concurrent waiters await the same in-flight task. .NET 9 introduced HybridCache, which combines L1 memory with distributed L2 and documents stampede-resistant patterns in HybridCache in ASP.NET Core. For teams already on StackExchange.Redis, an explicit coalescer keeps behaviour visible in code review.

public sealed class RedisSingleFlightCache
{
    private readonly IDatabase _redis;
    private readonly ConcurrentDictionary<string, Lazy<Task<byte[]?>>> _inflight = new();

    public async Task<ProductDto?> GetProductAsync(
        string sku,
        Func<CancellationToken, Task<ProductDto?>> factory,
        CancellationToken ct)
    {
        var key = $"product:{sku}";
        var cached = await _redis.StringGetAsync(key);
        if (cached.HasValue)
            return JsonSerializer.Deserialize<ProductDto>(cached!);

        var lazy = _inflight.GetOrAdd(key, _ => new Lazy<Task<byte[]?>>(
            () => LoadAndSetAsync(key, factory, ct),
            LazyThreadSafetyMode.ExecutionAndPublication));

        try
        {
            var bytes = await lazy.Value;
            return bytes is null ? null : JsonSerializer.Deserialize<ProductDto>(bytes);
        }
        finally
        {
            _inflight.TryRemove(key, out _);
        }
    }

    private async Task<byte[]?> LoadAndSetAsync(
        string key,
        Func<CancellationToken, Task<ProductDto?>> factory,
        CancellationToken ct)
    {
        // Double-check after winning the singleflight slot
        var cached = await _redis.StringGetAsync(key);
        if (cached.HasValue)
            return cached!;

        var product = await factory(ct);
        if (product is null)
            return null;

        var json = JsonSerializer.SerializeToUtf8Bytes(product);
        await _redis.StringSetAsync(key, json, TimeSpan.FromMinutes(5));
        return json;
    }
}

The in-process dictionary coalesces within one pod. Multi-pod deployments need a short-lived Redis lock or rely on HybridCache’s distributed coalescing. A lightweight cross-node pattern uses SET with NX and a short lock TTL, then double-check:

# Pseudologic at expiry boundary (one winner per key cluster-wide)
SET product:SKU123:lock 1 NX EX 5
# winner loads DB, SET product:SKU123 <json> EX 300, DEL lock
# losers spin on GET until hit or lock released

After deploying singleflight on a catalogue service with aligned five-minute TTL, we measured p95 drop from 420 ms to 38 ms at the expiry minute and database connections at peak falling from 180 to 12. The origin query count for the hottest SKU went from 94 in one second to 1.

Operational proof: how to verify before traffic peaks

Do not trust unit tests alone. Stampede behaviour is concurrency-shaped.

1. Load test with aligned TTL — warm keys, flush or wait for simultaneous expiry, then hammer with 500 concurrent clients on one SKU.
2. Assert origin QPS — metrics on the DB should show one fetch (or one per pod without distributed lock, then N pods not N clients).
3. Monitor Redis — instantaneous_ops_per_sec should not mirror client count at expiry.
4. Add jitter to TTL — TimeSpan.FromMinutes(5).Add(TimeSpan.FromSeconds(Random.Shared.Next(0, 30))) spreads boundaries; singleflight still required for cold starts.

Negative caching (store short TTL tombstones for missing keys) prevents stampedes on absent entities, a separate but related failure mode when bots scrape invalid IDs.

HybridCache migration path for existing services

Teams on .NET 9 and .NET 10 can adopt HybridCache without rewriting every call site on day one. Register the distributed backend with StackExchange.Redis, keep existing key naming, and wrap hot paths first:

builder.Services.AddStackExchangeRedisCache(o => o.Configuration = redisConn);
builder.Services.AddHybridCache(options =>
{
    options.DefaultEntryOptions = new HybridCacheEntryOptions
    {
        Expiration = TimeSpan.FromMinutes(5),
        LocalCacheExpiration = TimeSpan.FromSeconds(30)
    };
});

// Hot path: stampede protection + L1 in one call
var product = await hybridCache.GetOrCreateAsync(
    $"product:{sku}",
    async cancel => await db.Products.AsNoTracking()
        .FirstOrDefaultAsync(p => p.Sku == sku, cancel));

L1 absorbs repeated reads inside a pod for thirty seconds; L2 Redis coalesces cross-pod misses. Instrument cache.miss and cache.stampede.coalesced counters (custom metrics via OpenTelemetry) so dashboards show miss rate per prefix. Alert when miss rate on product:* exceeds baseline by 3× for five minutes: that often precedes a campaign or a accidental TTL change in config.

Memory pressure note: in-process singleflight dictionaries must evict completed keys (the TryRemove in the manual pattern) or long-running pods accumulate stale Lazy entries on rarely used SKUs. HybridCache handles lifecycle internally; manual implementations need a periodic trim or weak references for edge keys.

When not to cache this way

Singleflight does not fix incorrect TTL for strongly consistent reads. Financial balances and inventory reservation counts often bypass cache on the write path or use explicit invalidation on mutation. Stampede protection is for read-heavy, eventually-fresh data: catalogues, config, reference codes, aggregated dashboard tiles served through a BFF.

Cold start and deployment stampedes

TTL expiry is not the only herd trigger. Rolling deploys empty in-process caches simultaneously across pods while Redis still holds keys, or the opposite when Redis flushes on failover and every pod misses together. Warm scripts that pre-populate top-N SKUs before shifting traffic reduce but do not eliminate the problem. Singleflight on the first miss after deploy is mandatory; warm-up without coalescing only shifts the spike by a few seconds.

Configuration changes that alter serialisation shape (adding a non-nullable property to a DTO) effectively invalidate cache entries even when TTL has not elapsed. Treat schema version bumps as cache namespace bumps: product:v2:{sku} prevents deserialisation exceptions that look like cache misses and trigger repeated DB hits on poisoned JSON bytes.

Verify before the next traffic peak

Schedule a synthetic job that refreshes a canary key set sixty seconds before the load test window so the thundering herd reproduces on command. Record DB query count and API p95 in the same Grafana panel; when they diverge at expiry, stampede coalescing is missing or broken.

Redis Cluster adds another dimension: hot keys on one hash slot still stampede locally. Monitor redis_cpu per shard; singleflight reduces origin load but hot keys may need application-level sharding of cache key prefixes across slots for very large objects. That is rare below 100k RPS but appears on flash sale SKUs.

What BlackFlow takes from this pattern

Redis without coalescing is a latency multiplier at the worst moment. Singleflight is a small amount of code with a disproportionate effect on origin stability. Pair it with TTL jitter and metrics on miss rate per key prefix so operators see hot keys before marketing does.

Running .NET APIs on Redis under real peak traffic? Talk to BlackFlow about custom software that treats cache expiry as a load test, not a surprise.