Forty-seven pod restarts in one hour, zero user traffic spike. The cluster is healthy; your ASP.NET service is not. Kubelet keeps killing containers because liveness and readiness both hit the same endpoint before the host finishes warming EF Core, Redis, and RabbitMQ connections. The failure is probe design, not .NET performance.
Three probes, three questions
Kubernetes asks different questions at different lifecycle moments. A startup probe asks: has the process finished bootstrapping? A readiness probe asks: may this pod receive Service traffic? A liveness probe asks: should Kubelet restart the container? Collapsing those into one /health route guarantees wrong answers at least once per deploy.
Figure 1 maps the sequence when probes are wired correctly versus the restart loop when they are not.
sequenceDiagram
participant K as Kubelet
participant P as ASP.NET pod
participant S as Service endpoints
K->>P: startupProbe GET /health/startup
P-->>K: 503 (migrating, warming cache)
Note over K,P: failureThreshold × period = warm-up budget
P-->>K: 200 startup OK
K->>P: readinessProbe GET /health/ready
P-->>K: 200 ready
S->>P: traffic allowed
loop every liveness period
K->>P: livenessProbe GET /health/live
P-->>K: 200 alive
end
Note over K,P: Wrong: liveness on /health/ready during warmup → restart loop
Caption: Figure 1 — startup grants warm-up time; readiness gates traffic; liveness only detects deadlocks after ready.
ASP.NET health checks with explicit tags
Map each probe to a tagged check. Do not reuse dependency-heavy checks on liveness.
builder.Services.AddHealthChecks()
.AddCheck("self", () => HealthCheckResult.Healthy(), tags: new[] { "live" })
.AddNpgSql(connString, name: "postgres", tags: new[] { "ready", "startup" })
.AddRedis(redisConn, name: "redis", tags: new[] { "ready" })
.AddRabbitMQ(rabbitConn, name: "rabbitmq", tags: new[] { "ready" });
app.MapHealthChecks("/health/live", new HealthCheckOptions
{
Predicate = r => r.Tags.Contains("live")
});
app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
Predicate = r => r.Tags.Contains("ready"),
ResultStatusCodes =
{
[HealthStatus.Healthy] = StatusCodes.Status200OK,
[HealthStatus.Degraded] = StatusCodes.Status503ServiceUnavailable,
[HealthStatus.Unhealthy] = StatusCodes.Status503ServiceUnavailable
}
});
app.MapHealthChecks("/health/startup", new HealthCheckOptions
{
Predicate = r => r.Tags.Contains("startup")
});
Startup includes Postgres because EF migrations or first connection pool fill may run at boot. Readiness adds Redis and RabbitMQ because serving traffic without them causes user-visible errors. Liveness is only “process responds” — if Postgres is down cluster-wide, you want pods NotReady, not restarted individually in a thundering herd.
Why generic probe snippets mislead
Official docs are correct but easy to misapply. The Kubernetes task guide for configure liveness, readiness, and startup probes shows HTTP GET examples without ASP.NET warm-up behaviour. Microsoft’s health checks documentation demonstrates endpoints but leaves probe wiring to platform teams.
Common mistakes: using the same port/path for all probes; setting initialDelaySeconds on liveness instead of adding startupProbe; pointing readiness at a check that executes expensive work (full DB scan); omitting timeoutSeconds so slow GC pauses fail liveness; and hitting HTTPS before Kestrel certificates load. Each produces flapping that looks like application bugs in APM.
Probe routing table
| Endpoint | K8s probe | Checks included | Failure action |
|---|---|---|---|
/health/startup |
startupProbe | Postgres connect, migration complete | Continue waiting; do not restart |
/health/ready |
readinessProbe | Postgres, Redis, RabbitMQ | Remove from Service endpoints |
/health/live |
livenessProbe | In-process self only | Kubelet restart container |
/health (untagged) |
None | Do not wire to K8s | Human/debug only |
Manifest fragment matching the table — tune periods for your warm-up measured in staging, not copied from blogs:
{
"startupProbe": {
"httpGet": { "path": "/health/startup", "port": 8080 },
"periodSeconds": 5,
"failureThreshold": 24
},
"readinessProbe": {
"httpGet": { "path": "/health/ready", "port": 8080 },
"periodSeconds": 10,
"timeoutSeconds": 3,
"failureThreshold": 3
},
"livenessProbe": {
"httpGet": { "path": "/health/live", "port": 8080 },
"periodSeconds": 20,
"timeoutSeconds": 2,
"failureThreshold": 3
}
}
Architecture rule: Liveness proves the process is alive; readiness proves it may receive traffic — never conflate them on one tag or one dependency-heavy endpoint.
Implementable checklist
1. Measure cold start p95 on staging with realistic migration and cache warm-up; set startup failureThreshold × periodSeconds above that value with 30% headroom.
2. Ensure readiness failure removes the pod from endpoints before users see 502/503 from kube-proxy.
3. Load-test with one pod NotReady: remaining replicas must absorb traffic without cascading readiness failures.
4. Log health check failures at Warning with check name and duration; do not log secrets from connection strings.
5. During deploy, watch kubectl get pods -w for RESTARTS > 0; any restart during rollout is a probe bug until proven otherwise.
6. Document which dependencies are hard requirements vs degraded mode; only hard requirements belong on readiness.
Degraded readiness vs hard failure
Not every dependency deserves to pull a pod from rotation. A non-critical feature flag service might mark readiness Degraded while core API stays Healthy. ASP.NET health checks support Degraded status mapped to 503 on readiness, which removes the pod from Service endpoints without restarting it.
Document degraded semantics in runbooks: operators should know whether Degraded means “shift traffic away” or “page immediately.” Mixing Degraded on liveness is always wrong; liveness should not evaluate external dependencies that can fail cluster-wide.
Graceful shutdown interacts with probes: on SIGTERM, mark readiness failed first via IHostApplicationLifetime, drain in-flight requests, then stop. Without that ordering, kube-proxy may send new connections to a pod that already began shutting down Kestrel.
Metrics that prove probe correctness
Export kube_pod_container_status_restarts_total grouped by deployment. Alert on any increase during rollout window. Compare time from pod scheduled to first successful readiness scrape against startup probe budget; if reality exceeds budget, fix the probe before tuning JVM-style “just increase memory.”
Application-side, emit a counter health_check_failures_total{check="postgres"} only from readiness path, not liveness. Correlate with connection pool exhaustion to distinguish probe misconfiguration from genuine database incidents.
Run load tests during deploy: while new pods warm up, existing pods must stay Ready and serve traffic without readiness flapping when dependency latency spikes briefly. If flapping occurs, increase readiness failureThreshold slightly or fix dependency timeouts rather than disabling checks.
Multi-container pods and sidecars
When a service mesh sidecar or log shipper shares the pod network namespace, probes still target the application container port, not the sidecar admin port. Mesh readiness may lag application readiness; align mesh hold flags with ASP.NET startup completion so Envoy does not route to a pod still compiling EF models.
Init containers that run migrations should complete before the main container starts; startupProbe on the main app should not compensate for a failed init container that partially applied SQL. Migration init jobs belong in Helm pre-upgrade hooks or CI, with the same phase gates described for EF expand/contract.
Ops gates and sign-off
SLO: zero unplanned restarts during rolling update for services with < 60s warm-up. Incident class: repeated liveness restart with exit code 137 triggers platform review, not automatic memory limit bumps. Platform signs probe manifest; application team signs health check predicates; jointly sign off after one production canary deploy with restart count verified at zero for thirty minutes post-rollout.
Probe design sits next to container image and HPA in the paved path for .NET on Kubernetes. If you are standardising that path across squads, BlackFlow builds custom .NET platforms where health endpoints and rollouts are tested together, not pasted from different tutorials.
