Transactional outbox relays in .NET are straightforward until they are not. You insert a row, a background worker publishes to RabbitMQ, the consumer processes the event. The fragile piece is rarely the INSERT. It is the broker topology: classic queues that lose unacked messages on node failure, and poison payloads that spin forever because there is no dead-letter path with quorum durability. For production relays, quorum queues plus a DLQ are the default, not an optimisation.
Failure mode: relay stall after one bad message
A billing service we reviewed had 12,000 outbox rows waiting while relay lag sat at zero. One malformed JSON payload (a schema migration missed a nullable column) threw on deserialisation. The consumer nacked and requeued indefinitely. Classic queue semantics meant a broker restart during the incident dropped three unacked publishes that never returned to the outbox table. Recovery took forty minutes of manual replay and left finance questioning event completeness.
Martin Fowler’s Transactional Outbox pattern explains why the database row must commit with the business transaction. It does not specify RabbitMQ queue type. That gap is where teams assume “durable queue” equals “safe relay.” Classic mirrored queues are deprecated; quorum queues are the replacement with predictable fault tolerance.
End-to-end flow with quorum and DLQ
sequenceDiagram
participant API as ASP.NET API
participant DB as Outbox table
participant Relay as Outbox relay worker
participant Q as orders.events (quorum)
participant DLQ as orders.events.dlq (quorum)
participant Sub as Domain consumer
API->>DB: COMMIT business + outbox row
Relay->>DB: SELECT unpublished FOR UPDATE SKIP LOCKED
Relay->>Q: basic.publish persistent
Q->>Sub: deliver
alt valid payload
Sub->>Q: ack
Relay->>DB: mark published
else poison / schema error
Sub->>Q: nack (no requeue)
Q->>DLQ: x-death after max retries
end
Figure 1: Outbox relay with quorum primary queue and DLQ — poison messages exit the hot path instead of blocking the relay consumer group.
RabbitMQ documents quorum queues in Quorum queues: Raft-based replication, leader election on failure, and different performance trade-offs than classic queues. For outbox traffic (moderate throughput, high correctness), that trade-off is correct.
Why classic topology hides poison messages
Classic queue defaults encourage auto-ack demos. Production relays use manual ack with requeue on transient failure. Without a dead-letter exchange (DLX), a permanent schema error becomes an infinite loop. With requeue disabled but no DLQ, messages are dropped silently. Neither outcome is acceptable when the outbox row is already marked published or stuck unpublished depending on your ordering bug.
Topology JSON we encode as infrastructure-as-code for .NET relay services:
{
"exchanges": [
{ "name": "orders.events", "type": "topic", "durable": true }
],
"queues": [
{
"name": "orders.events.billing",
"type": "quorum",
"durable": true,
"arguments": {
"x-dead-letter-exchange": "orders.events.dlx",
"x-dead-letter-routing-key": "billing.poison",
"x-delivery-limit": 5
}
},
{
"name": "orders.events.billing.dlq",
"type": "quorum",
"durable": true
}
],
"bindings": [
{ "queue": "orders.events.billing", "exchange": "orders.events", "routing_key": "order.placed.v1" },
{ "queue": "orders.events.billing.dlq", "exchange": "orders.events.dlx", "routing_key": "billing.poison" }
]
}
x-delivery-limit (RabbitMQ 3.11+) removes infinite redelivery without consumer-side counters. The DLQ is also quorum so poison archives survive broker maintenance.
Routing and consistency decisions
| Component | Queue type | Ack model | On failure |
|---|---|---|---|
| Outbox relay publish target | Quorum | Publisher confirm | Retry with backoff; row stays unpublished |
| Domain consumer | Quorum | Manual ack after idempotent handler | Nack no requeue → DLQ after delivery limit |
| DLQ archive | Quorum | Ops replay tool only | Human fixes schema, republish to primary |
| Classic queue (legacy) | Classic | Any | Reject for new outbox paths |
Architecture rule: Outbox relay publishes belong on quorum queues with an explicit DLQ and delivery limit — classic queues are not a durability plan for domain events you must explain to auditors.
.NET relay consumer shape
The relay worker and downstream consumer have different responsibilities. The relay marks rows published only after broker confirm. The consumer must not requeue poison:
await channel.BasicConsumeAsync(
queue: "orders.events.billing",
autoAck: false,
consumer: new AsyncEventingBasicConsumer(channel)
{
ReceivedAsync = async (_, ea) =>
{
try
{
var evt = JsonSerializer.Deserialize<OrderPlacedV1>(ea.Body.Span)
?? throw new InvalidOperationException("null payload");
await handler.HandleAsync(evt, ct);
await channel.BasicAckAsync(ea.DeliveryTag, multiple: false);
}
catch (JsonException ex)
{
_logger.LogError(ex, "Poison message {Tag}", ea.DeliveryTag);
await channel.BasicNackAsync(ea.DeliveryTag, multiple: false, requeue: false);
}
}
});
Idempotency keys on the consumer side (Redis SET NX or unique constraint on event ID) remain mandatory. Quorum queues give at-least-once delivery; they do not deduplicate business logic.
For the full outbox INSERT and relay loop pattern, see our earlier walkthrough on the transactional outbox pattern for event-driven platforms. This article covers the broker half that post assumes but rarely diagrams.
Implementable checklist
- Declare primary and DLQ as quorum; bind DLX with a dedicated routing key per consumer group.
- Set
x-delivery-limiton primary queues; disable infinite requeue in consumer code. - Use publisher confirms on the relay; mark outbox rows published only after confirm.
- Alert on DLQ depth and relay lag; page when DLQ rate exceeds baseline.
- Run a chaos test: kill the quorum leader during publish; assert zero lost confirms and recoverable relay state.
- Migrate classic outbox queues on a maintenance window; do not mix classic and quorum consumers on the same routing key.
Publisher confirms on the relay worker
Consumers are only half the story. The relay must not mark outbox rows published until RabbitMQ acknowledges persistence to a quorum majority. In .NET with RabbitMQ.Client async API:
await channel.ConfirmSelectAsync();
var body = Encoding.UTF8.GetBytes(JsonSerializer.Serialize(envelope));
var props = new BasicProperties { Persistent = true, ContentType = "application/json" };
await channel.BasicPublishAsync("orders.events", routingKey, mandatory: true, props, body);
await channel.WaitForConfirmsOrDieAsync(TimeSpan.FromSeconds(5));
await db.Outbox.Where(o => o.Id == row.Id).ExecuteUpdateAsync(
s => s.SetProperty(o => o.PublishedAt, DateTimeOffset.UtcNow));
Without confirms, a broker crash between TCP accept and disk write produces ghost events: downstream never saw them, but the outbox row is gone from retry. Pair confirms with mandatory: true so unroutable messages return synchronously instead of vanishing when a binding typo ships Friday evening.
Quorum queues reject transient queues and certain classic-only features by design. Lazy queues, priority queues, and global QoS semantics differ from classic behaviour. Read the quorum limitations page before porting a legacy topology wholesale. For outbox relays, those limitations rarely matter; throughput in the tens of thousands of messages per hour per queue is well within quorum sweet spot on modest clusters.
Ops gates
Treat DLQ growth as an incident class B: no customer-facing outage, but financial or fulfilment reconciliation may drift. Relay lag above five minutes with healthy consumers is class A. Sign-off on topology changes belongs to platform plus the domain owner who owns the event schema version.
What BlackFlow takes from this pattern
Outbox correctness is a database and messaging joint problem. Quorum queues and DLQs are the RabbitMQ answer to poison messages and node loss that classic queues papered over. Wire them before the first schema migration breaks a consumer.
Building .NET event relays on RabbitMQ? Talk to BlackFlow about custom software where broker topology is as reviewed as application code.
