A payment service processing 8k messages per second at peak tolerates at-least-once delivery. What it cannot tolerate is at-least-once wallet credits. RabbitMQ redelivers when your consumer dies after side effects but before ack; without idempotency, finance discovers duplicates at month-end, not in the queue dashboard.
At-least-once math on a real consumer
Quorum queues, publisher confirms, and manual acks reduce loss. They increase redelivery. A 0.3% redelivery rate sounds healthy until you multiply: 8,000 msg/s × 0.003 = 24 duplicate deliveries per second at peak. Any handler that INSERTs a ledger row or calls a payment API without dedupe will double-charge someone.
Constraints in play: .NET worker with prefetch 20, processing time p95 120ms, Postgres ledger, optional Redis for hot dedupe. Poison messages route to DLQ after five deaths. Your idempotency store must outlive the longest redelivery window plus DLQ replay horizon — often seven to thirty days depending on compliance.
Ordering guarantees from the transactional outbox pattern (see our transactional outbox for event-driven platforms note) prevent duplicate publish; they do not prevent duplicate consume when the handler crashes mid-flight.
Why ack-before-commit is the naive default
Teams optimise for queue depth, not correctness:
public async Task HandleAsync(PaymentCaptured evt, CancellationToken ct)
{
// Naive: ack implied by successful return; side effect first
await _wallet.CreditAsync(evt.CustomerId, evt.Amount, ct);
await _db.SaveChangesAsync(ct);
// Process crash here → message redelivered → second credit
}
RabbitMQ marks the message acked only when the channel completes successfully. If you credit before persisting idempotency state in the same transaction as business writes, crash windows remain. If you ack in a finally block regardless of handler outcome, you lose messages. The fix is idempotent side effects keyed by broker message id or business idempotency key carried in the payload.
RabbitMQ documents consumer acknowledgements and redelivery behaviour in the confirms and acks guide. Read it alongside your failure injection tests, not instead of them.
Engineered approach: Redis dedupe vs DB unique
Redis SET NX with TTL: fast, minimal Postgres load. Key idem:{messageId} set on first processing; duplicates exit early. TTL must exceed max redelivery and replay window. Use Redis expiration patterns documented in the Redis patterns guide; idempotency keys are simpler than locks but share TTL discipline.
Database unique constraint on message_id: durable, audit-friendly, survives Redis flush. Cost: write amplification and contention on hot partitions. Best for money movement and regulated ledgers.
Production pattern combines both: Redis as first-line filter; Postgres unique as source of truth inside the same transaction as the credit.
public async Task HandleAsync(BasicDeliverEventArgs ea, CancellationToken ct)
{
var messageId = ea.BasicProperties.MessageId
?? throw new InvalidOperationException("MessageId required for idempotency");
var redisKey = $"idem:payment:{messageId}";
var acquired = await _redis.StringSetAsync(
redisKey, "1", TimeSpan.FromDays(14), When.NotExists);
if (!acquired)
{
_channel.BasicAck(ea.DeliveryTag, false);
return;
}
await using var tx = await _db.Database.BeginTransactionAsync(ct);
try
{
_db.ProcessedMessages.Add(new ProcessedMessage { MessageId = messageId, ProcessedAt = DateTimeOffset.UtcNow });
await _db.SaveChangesAsync(ct); // unique index throws on duplicate
var evt = JsonSerializer.Deserialize<PaymentCaptured>(ea.Body.Span)!;
await _wallet.CreditAsync(evt.CustomerId, evt.Amount, ct);
await _db.SaveChangesAsync(ct);
await tx.CommitAsync(ct);
_channel.BasicAck(ea.DeliveryTag, false);
}
catch (DbUpdateException ex) when (IsUniqueViolation(ex))
{
await tx.RollbackAsync(ct);
_channel.BasicAck(ea.DeliveryTag, false); // already processed
}
catch
{
await tx.RollbackAsync(ct);
await _redis.KeyDeleteAsync(redisKey); // allow retry after fix
_channel.BasicNack(ea.DeliveryTag, false, requeue: true);
throw;
}
}
DDL backing the durable layer:
CREATE TABLE processed_messages (
message_id text PRIMARY KEY,
processed_at timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX ix_processed_messages_processed_at
ON processed_messages (processed_at);
-- Retention job deletes rows older than replay window after archival
Choosing Redis vs Postgres for your workload
Redis dedupe wins when message volume is high, duplicate detection is hot-path, and brief Redis unavailability should fail open to DB constraint rather than double-charge. Size Redis with key count × average key size × replication factor; idempotency keys for 14-day TTL at 8k msg/s is millions of keys unless you compress key format.
Postgres-only idempotency wins when audit requires durable proof without depending on cache warmth, or when Redis is not yet a production tier in your estate. Partial unique indexes on partitioned tables need careful design: partition key plus message_id uniqueness prevents cross-partition duplicates.
Hybrid is not duplication for its own sake: Redis filters 99% of redeliveries cheaply; Postgres catches Redis evictions, TTL expiry before replay window ends, and any manual cache flush during incident response.
MessageId discipline and outbox alignment
Idempotency keys must be stable across republication. Prefer business event id from the outbox row over broker delivery tag, which changes on redelivery in some configurations. Set MessageId at publish time equal to outbox primary key or event UUID; consumers reject messages missing MessageId rather than inventing one from body hash.
Consumers that nack with requeue on transient SQL timeout must not delete Redis key until processing fully commits; the pattern above deletes Redis key only on handler failure before commit, allowing retry. After commit, Redis TTL naturally expires; Postgres row remains.
DLQ replay tooling must reuse the same handler code path, not a “special replay mode” that skips dedupe. Finance replays are where duplicates hurt most because operators assume replay is safe.
Prefetch, concurrency, and pool sizing
RabbitMQ consumer prefetch multiplied by pod count sets maximum unacked deliveries. If each handler holds a database connection for the full handler duration, required pool size equals prefetch × replicas. Scale replicas up without lowering prefetch and you exhaust Postgres max_connections during peak.
ConsumerDispatchConcurrency in .NET multiplies in-flight handlers per channel. Treat effective parallelism as replicas × prefetch × concurrency. Load-test with redelivery simulation before Black Friday, not after duplicate credits appear in reconciliation reports.
Poison messages that always throw should land in DLQ after bounded retries with the same MessageId so operators inspect one row, not infinite requeue loops that mask the root bug behind lag metrics.
Testing idempotency in CI
Integration tests should deliver the same message body twice with identical MessageId and assert single side effect. Use Testcontainers for RabbitMQ and Postgres; optional Redis container for hybrid path. Fail the build if handler returns success on duplicate without checking store.
Contract tests at publish boundary assert MessageId is set and matches outbox id. Consumers and publishers owned by different teams still share this contract via schema registry or shared test package.
Load tests without duplicate injection give false confidence. Inject ten percent artificial redelivery in staging chaos jobs weekly.
Observability: prove dedupe is working
Metrics to dashboard: consumer_messages_processed_total, consumer_idempotent_skips_total, consumer_handler_failures_total, and consumer_redeliveries_total from RabbitMQ management API. Skips rising with flat processed count confirms dedupe is catching replays. Zero skips during known chaos test means dedupe is broken, not that redelivery stopped.
Structured logs on skip path should include messageId and consumer group, never payload bodies with PCI fields. Alert when skip ratio exceeds historical baseline by 3× without deploy correlation; that pattern can indicate publisher duplicate bug or stuck retry storm.
Reconciliation jobs comparing broker dead-letter count to finance exception queue close the loop for payment handlers. Daily diff should be zero; non-zero diff triggers incident before finance month-end.
Side-effect ordering inside the handler
Idempotency record insert must commit in the same database transaction as the business write when using Postgres-only path. External API calls after commit risk orphan payments if API succeeds and ack fails; prefer outbox for outbound calls or saga compensation documented in runbooks.
For wallet credit, order operations: begin transaction, insert idempotency row, apply credit, commit, then ack. Never call external credit API before local ledger row exists; recovery from partial external success without local row is manual and expensive.
Document maximum handler retry duration in service README. Operators need a single number: after how many minutes a stuck message moves to DLQ for human review instead of starving the queue behind a poison payload. Publish that threshold in the on-call runbook.
Operational proof: chaos before peak season
1. Publish 10k test events with unique MessageId; assert ledger count equals 10k.
2. Kill -9 consumer mid-handler at random delays; assert zero duplicate credits.
3. Replay DLQ manually; assert idempotency keys block second application.
4. Monitor Redis memory; extend TTL only with finance sign-off on replay policy.
5. Alert on rising nack rate without deploy correlation — indicates schema or handler bugs, not “RabbitMQ instability.”
Prefetch tuning: high prefetch without idempotency widens crash windows. Start prefetch equal to concurrent handler capacity, not ten times it. Review prefetch quarterly when message size or handler latency shifts.
Event-driven .NET with RabbitMQ and Redis is our default integration stack for platforms that must survive redelivery without apologising to customers. Talk to BlackFlow if your consumers ack confidently but finance still finds duplicates.
