Your Kubernetes rollout finishes in ninety seconds. Postgres rejects INSERTs for forty-seven of them because the new migration added a NOT NULL column the old pods never populate. EF Core did exactly what you asked: it applied the migration on startup. The failure is the contract between schema change and running binaries, not the ORM.
The problem is two versions on one database
Zero-downtime migration is a deployment pattern, not a checkbox on dotnet ef migrations add. During a rolling update you always have at least two builds talking to the same database: vN draining connections and vN+1 accepting new ones. Any DDL that vN cannot survive is a production incident waiting for the next deploy.
Quantify the blast radius before you touch migrations. A typical order service we see holds 12M rows across three AZ replicas. Peak write rate sits around 400 INSERT/s. A blocking ALTER TABLE ... ADD COLUMN ... NOT NULL without a default holds an Access Exclusive lock long enough to stall every pod, not just the one that ran Database.Migrate(). Even when the lock is brief, vN binaries that INSERT without the new column fail until you roll back the deploy.
Source systems in play: ASP.NET Core API on K8s, PostgreSQL 16, EF Core 9 migrations applied at startup or via init job, and CI that treats “green build” as permission to ship. The constraint that bites teams is dual-write compatibility for the full rollout window, usually five to fifteen minutes depending on pod count and warm-up time.
Why the naive EF migration fails
The comfortable failure mode is a single migration that “finishes the feature” in one PR:
public partial class AddCustomerTier : Migration
{
protected override void Up(MigrationBuilder migrationBuilder)
{
migrationBuilder.AddColumn<string>(
name: "Tier",
table: "Customers",
nullable: false,
defaultValue: "Standard");
migrationBuilder.CreateIndex(
name: "IX_Customers_Tier",
table: "Customers",
column: "Tier");
}
}
This looks safe because you supplied a default. It is not safe for zero downtime. Old application code never reads or writes Tier. New code assumes every row has a business-meaningful tier, not the placeholder default. Worse patterns drop columns, rename in place, or change column type in one step. EF generates valid SQL; PostgreSQL executes it; the application layer breaks across versions.
Measured symptoms from a recent client rollout: three pods on vN+1, four still on vN. vN+1 started writing tier-aware analytics events. vN continued INSERTing customers without tier semantics. Downstream consumers saw null-equivalent behaviour for twenty minutes until vN drained. No database error appeared in logs because the column existed. The bug was semantic drift between schema and code paths.
Microsoft’s migration docs describe how EF applies changes; they do not automatically sequence those changes for parallel app versions. That sequencing is your expand/contract discipline, documented in relational migration literature including the expand and contract pattern for safe schema evolution.
Engineered approach: expand, migrate, contract
Split every breaking change into phases that each tolerate old and new binaries. For adding a mandatory Tier column, expect four deploys, not one.
Phase 1 — Expand: add nullable column, no index yet. Old code ignores it. New code can write it when present.
Phase 2 — Backfill: populate rows in batches outside request path. Use SQL you control rather than loading entities into memory.
Phase 3 — Enforce: add NOT NULL constraint and index after backfill completes and metrics show zero nulls.
Phase 4 — Contract: remove deprecated columns or code paths only when traffic proves vN is gone.
Production-shaped backfill for 12M rows avoids EF change tracking entirely:
-- Batch backfill: 10k rows per tick, index-friendly on PK
DO $$
DECLARE
batch_size int := 10000;
updated int;
BEGIN
LOOP
UPDATE "Customers" c
SET "Tier" = CASE
WHEN c."LifetimeValue" >= 10000 THEN 'Enterprise'
WHEN c."LifetimeValue" >= 1000 THEN 'Pro'
ELSE 'Standard'
END
WHERE c."Id" IN (
SELECT "Id" FROM "Customers"
WHERE "Tier" IS NULL
ORDER BY "Id"
LIMIT batch_size
FOR UPDATE SKIP LOCKED
);
GET DIAGNOSTICS updated = ROW_COUNT;
EXIT WHEN updated = 0;
COMMIT;
PERFORM pg_sleep(0.05);
END LOOP;
END $$;
Application code during dual-read/dual-write uses explicit feature flags or version gates. New binaries write Tier; old binaries must not fail when the column exists:
// vN+1: write tier when known; never assume NOT NULL until phase 3
public async Task CreateCustomerAsync(CreateCustomer cmd, CancellationToken ct)
{
var entity = new Customer
{
Email = cmd.Email,
Tier = cmd.Tier ?? "Standard" // safe default until backfill proves otherwise
};
_db.Customers.Add(entity);
await _db.SaveChangesAsync(ct);
}
// Read path: tolerate missing semantic tier during expand phase
public async Task<CustomerDto> GetAsync(Guid id, CancellationToken ct)
{
var row = await _db.Customers.AsNoTracking()
.Where(c => c.Id == id)
.Select(c => new CustomerDto
{
Id = c.Id,
Tier = c.Tier ?? "Standard"
})
.SingleAsync(ct);
return row;
}
EF Core’s migration bundle (official migrations guide) fits this model when you stop calling Migrate() blindly on every pod. Run migrations from a single init job or pipeline step; let app pods start only after the phase completes. Competing pods racing Database.Migrate() still contend on migration history locks.
Operational proof and rollout gates
Before phase 1 merges, run a dual-version integration test against a database clone restored from production statistics. Spin vN and vN+1 containers against the clone simultaneously. Execute write workloads on both. Assert zero SQLSTATE 23502 (not null violation) and zero missing-column errors.
Gate checklist for each phase:
1. Migration SQL reviewed for lock class: prefer ADD COLUMN NULL over in-place type changes.
2. Backfill job monitored for replication lag on read replicas used for reporting.
3. Metric customers.tier_null_count must hit zero for 24 hours before NOT NULL phase.
4. Rollback plan documented: vN+1 must run with nullable column if you revert code before contract phase.
5. Index creation uses CONCURRENTLY on PostgreSQL when available to avoid long reads blocked.
6. Post-deploy: compare INSERT endpoint 5xx rate for one hour against seven-day baseline; rollback if above 2× baseline.
Benchmark method: capture pg_stat_activity wait events during migration on a staging clone sized to 80% of prod row count. If Access Exclusive exceeds two seconds at p95 traffic, split the migration further.
Teams migrating runtime alongside schema should read our notes on .NET 10 LTS changes worth migrating for — runtime and schema upgrades compound risk when bundled in one Friday deploy. Schedule schema phases and runtime upgrades in separate windows unless integration tests cover both.
Renames, type changes, and the contract table
Adding columns is the easy lesson. Renaming Status to OrderStatus or widening varchar(20) to text trips teams that skip the contract table. Document every phase with which binary versions may run and which SQL must be applied first.
Rename pattern: add new column, dual-write in app, backfill, switch reads, drop old column in contract phase. Never RENAME COLUMN while vN still maps EF entity property to the old name unless you ship a coordinated model snapshot and code change in the same second, which rolling deploys forbid.
Type change pattern: add new column with target type, backfill with cast validation, swap reads, enforce NOT NULL on new column, drop old. For PostgreSQL, avoid in-place cast that rewrites the whole heap during peak traffic.
Keep a living migration runbook row per active change: phase number, migration id, max concurrent old version pods allowed, rollback SQL, and owner. Review in weekly platform standup until contract phase completes.
EF tooling choices that help or hurt
Migration bundles generated in CI and applied once per environment reduce race conditions from every pod calling Migrate(). Idempotent SQL scripts reviewed by a DBA beat auto-generated diffs for large tables when you hand-craft batch updates.
Snapshot drift between branches is a social problem: two teams adding migrations on parallel branches merge and produce conflicting model snapshots. Trunk-based migration ownership or a single “schema steward” queue prevents Friday merge collisions.
Design-time factories and test containers should replay the full phase sequence in integration tests, not only the final desired schema. A test that jumps straight to phase 4 hides the vN compatibility bug you will meet in production.
When to accept downtime instead
Expand/contract has a cost in deploy count and engineering attention. An internal admin tool with one nightly maintenance window and no SLA can still use big-bang migrations. The line is revenue-bearing write path during rolling deploys. Once K8s rollouts are continuous, big-bang DDL is debt.
Building order, billing, or multi-tenant SaaS on EF Core and PostgreSQL? Talk to BlackFlow about migration playbooks that survive rolling deploys without midnight heroics or surprise rollbacks.
