The Outbox Table Is Not Optional

A maintenance scheduler releases a CNC slot, calls producer.Send, then awaits SaveChangesAsync — the broker acknowledges ReservationReleased, the shop-floor display updates, and seconds later the database transaction rolls back on a concurrency conflict. Downstream systems acted on an event that never committed. Dual-write remains the dominant production bug in event-driven estates: not because teams are careless, but because SQL and Kafka do not share a commit boundary. Pretending they do is how industrial systems lose audit trails, double-allocate inventory, and ship duplicate work orders.

Problem space: the gap between commit and publish

Asset maintenance platforms, billing engines, and inventory allocators converge on the same shape: authoritative state in PostgreSQL, propagation through Kafka or RabbitMQ, and no atomic join between the two. Incident post-mortems in estates without an outbox typically split into two buckets: publish-after-commit ordering where the broker call failed after the row persisted, and publish-before-commit where ghost events preceded rolled-back transactions. The quantified anchor is the race window measured in milliseconds, not the broker’s durability SLA.

• PostgreSQL 15+ as system of record — ACID transactions on domain tables
• Kafka 3.x/4.x as integration bus — at-least-once delivery, idempotent consumers assumed
• .NET minimal API services with EF Core DbContext per request
• Peak maintenance windows: sustained outbox row volume without relay backlog
• Audit requirement: every externalised state change must be reconstructable from SQL alone

Why the naive approach fails

The seductive pattern: update the row, then fire-and-forget to the broker. Or worse — publish first so “downstream feels fast.” Both orderings fail under partial failure. Retries without idempotency keys double-charge. Compensating transactions added later do not fix the gap at the source. Every team that publishes domain events has already built an outbox — most named it poorly: pending_notifications, integration_queue, a nightly cron that syncs orders to the bus. The pattern is universal because the problem is universal.

If publishing is not in the same transaction as your state change, you do not have an event-driven architecture. You have a hope-driven one. Hope does not survive broker rolling restarts, network partitions, or DbUpdateConcurrencyException on hot aggregates.

// Naive: publish then save — ghost events on rollback
app.MapPost("/reservations/{id}/release", async (
    Guid id, IReservationService svc, IKafkaProducer producer, CancellationToken ct) =>
{
    var reservation = await svc.GetAsync(id, ct)
        ?? throw new NotFoundException();

    // RACE: broker accepts before DB commits
    await producer.ProduceAsync("reservations", new ReservationReleased(
        id, reservation.SlotId, reservation.TenantId));

    reservation.Status = ReservationStatus.Released;
    await svc.SaveAsync(reservation, ct); // may throw DbUpdateConcurrencyException
    return Results.Ok();
});

// Alternate naive: save then publish — lost events on broker timeout
await svc.SaveAsync(reservation, ct);
await producer.ProduceAsync(...); // timeout after commit → downstream never learns

What breaks at scale is predictable. Under sustained release traffic, publish-then-save ordering produces ghost events whenever concurrency conflicts roll back after the broker accepts. Save-then-publish ordering loses events when broker latency exceeds HTTP client timeouts without retry semantics. Neither failure mode is acceptable when a released CNC slot triggers physical material movement.

Common objections collapse under scrutiny. “Kafka transactions cover it” — broker-side atomicity does not retroactively join PostgreSQL business rules unless you accept significant coupling and latency. “We will use sagas” — sagas orchestrate long-running workflows; they do not remove the need to record intent atomically at the source. “Our volume is low” — low volume is where duplicate events are debugged manually instead of absorbed by idempotent handlers. Teams that rename integration_queue or pending_notifications without the outbox contract still have an outbox; they just cannot operate or audit it consistently.

sequenceDiagram
    participant API as Reservation API
    participant DB as PostgreSQL
    participant K as Kafka
    participant UI as Shop-floor UI
    API->>K: Produce ReservationReleased
    K-->>UI: consumer updates display
    API->>DB: BEGIN; UPDATE reservation
    DB-->>API: ROLLBACK (concurrency)
    Note over API,UI: UI shows released; DB says Held — dual-write failure

Figure 1: Publish-before-commit race — downstream acts on an event the database rejected.

Engineered approach: transactional outbox

The transactional outbox stores outbound messages in a table mutated inside the same ACID transaction as domain tables. A relay reads pending rows and publishes to the broker, then marks them dispatched. Chris Richardson’s transactional outbox pattern remains the canonical reference; James Carr’s January 2026 implementation notes add practical depth on relay failure modes and why at-least-once plus idempotent consumers is the realistic baseline.

-- Postgres DDL: outbox lives beside domain tables, same transaction boundary
CREATE TABLE integration_outbox (
    id              BIGSERIAL PRIMARY KEY,
    aggregate_type  TEXT NOT NULL,
    aggregate_id    UUID NOT NULL,
    event_type      TEXT NOT NULL,          -- e.g. ReservationReleased.v1
    payload         JSONB NOT NULL,
    created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
    dispatched_at   TIMESTAMPTZ,
    last_error      TEXT,
    CONSTRAINT outbox_pending CHECK (
        (dispatched_at IS NULL AND last_error IS NULL) OR
        (dispatched_at IS NOT NULL) OR
        (last_error IS NOT NULL)
    )
);

CREATE INDEX idx_outbox_pending
    ON integration_outbox (id)
    WHERE dispatched_at IS NULL;

-- Domain write + outbox insert share one commit
BEGIN;
  UPDATE reservations SET status = 'Released' WHERE id = $1 AND status = 'Held';
  INSERT INTO integration_outbox (aggregate_type, aggregate_id, event_type, payload)
  VALUES ('Reservation', $1, 'ReservationReleased.v1', $2);
COMMIT;

// Relay worker: FOR UPDATE SKIP LOCKED — horizontal scale without double-publish
public sealed class OutboxRelayWorker : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            await using var tx = await _db.Database.BeginTransactionAsync(stoppingToken);

            var batch = await _db.Outbox
                .FromSqlRaw("""
                    SELECT * FROM integration_outbox
                    WHERE dispatched_at IS NULL
                    ORDER BY id
                    LIMIT 50
                    FOR UPDATE SKIP LOCKED
                    """)
                .ToListAsync(stoppingToken);

            foreach (var row in batch)
            {
                await _producer.ProduceAsync(
                    "reservations",
                    key: row.AggregateId.ToString(),
                    value: row.Payload,
                    stoppingToken);

                row.DispatchedAt = DateTimeOffset.UtcNow;
            }

            await _db.SaveChangesAsync(stoppingToken);
            await tx.CommitAsync(stoppingToken);

            if (batch.Count == 0)
                await Task.Delay(200, stoppingToken);
        }
    }
}

Domain handler shape under the engineered model: mutate reservation, insert outbox row, single SaveChangesAsync. No broker call inside the request transaction.

app.MapPost("/reservations/{id}/release", async (
    Guid id, ReservationDbContext db, CancellationToken ct) =>
{
    await using var tx = await db.Database.BeginTransactionAsync(ct);

    var reservation = await db.Reservations.FindAsync([id], ct)
        ?? throw new NotFoundException();

    reservation.Status = ReservationStatus.Released;
    db.Outbox.Add(new OutboxRow(
        AggregateType: "Reservation",
        AggregateId: id,
        EventType: "ReservationReleased.v1",
        Payload: JsonSerializer.SerializeToDocument(new { id, reservation.SlotId })));

    await db.SaveChangesAsync(ct);
    await tx.CommitAsync(ct);
    return Results.Ok();
});

Schema decisions that survive audits

Treat the outbox as integration infrastructure, not a scratch table. Field choices matter when regulators or finance ask you to prove what was committed versus what was published:

• id: BIGSERIAL primary key for cursor-based polling — monotonic, index-friendly
• aggregate_type / aggregate_id: stable correlation for idempotent consumers and support lookups
• event_type: versioned string — ReservationReleased.v1 beats implicit breaking changes
• payload: JSONB in PostgreSQL for indexed diagnostics without escaping hell
• created_at, dispatched_at, last_error: operational fields your on-call engineer will query at 2am

Relay choice and what to measure

Dimension	Polling (SKIP LOCKED)	Debezium CDC
Propagation latency	Bounded by poll interval	WAL decode — typically lower
Operational complexity	Low — one .NET worker	High — Connect, slots, schema history
Multi-relay safety	Native via row locks	Single connector per table typical
Replay after incident	Reset dispatched_at on row range	Reset offset + tombstone handling
Best fit	Most .NET + EF Core estates	Sub-second safety interlocks

Credible proof comes from staging that mirrors production indexes, connection pool sizing, and relay pod count — not from invented percentages. Measure what your on-call team will actually query:

1. Outbox depth: pending row count over time — alert when depth exceeds SLO threshold
2. End-to-end lag: time from domain commit to consumer offset — dominated by relay plus broker, not HTTP
3. Duplicate delivery: consumer dedupe on aggregate_id plus event_type; verify idempotency under relay retry
4. Ghost events after rollback: should be zero — broker never sees uncommitted rows
5. Poison payloads: rows landing in last_error without blocking the poll loop
6. Broker restart recovery: outbox depth returns to baseline without manual republish

Retention policy: archive dispatched rows to cold storage with aggregate_id indexed for audit replay; page when pending rows exceed age thresholds you define in runbooks. Migration from naive dual-write does not require a big-bang rewrite — introduce the outbox table, dual-write into it alongside existing publish calls for one bounded context, verify depth metrics, then remove the direct producer call from the request path.

The outbox table is the contract between your write model and integrators — build it on day one, name it honestly, instrument depth aggressively, and dual-write races stop being production mysteries.