Fourteen microservices across three environments means forty-two secret surfaces before you count CI variables and local developer overrides. When the paved path is still kubectl apply -f secret.yaml committed beside Terraform, rotation becomes a scavenger hunt: Vault has the new password, git has the old one, and the Terraform plan is clean because Kubernetes state drifted outside IaC.
Structural map: who owns the credential lifecycle
Platform engineering here is not picking HashiCorp vs cloud KMS in the abstract. It is drawing boundaries between four actors: Terraform (cluster and namespace shape), External Secrets Operator or CSI driver (sync from vault to K8s Secret object), the secret store (Vault, AWS Secrets Manager, GCP Secret Manager), and application pods (consume Secret via env or volume). Git stores Terraform modules, ESO manifests, and service wiring — never plaintext credentials.
When teams inherit a monolithic Internal Developer Portal without this split, every rotation ticket touches a different repo. The integration tax is measurable: two engineer-hours per week on credential incidents in a mid-size platform team we audited, mostly tracing which copy of a connection string was authoritative.
Integration tax you pay per approach
Plain Secret YAML in git: zero upfront tooling, highest ongoing cost. Every rotation is a PR, a redeploy, and a risk of stale branches. Base64 is not encryption; anyone with repo access reads production credentials. Audit trail mixes git blame with kubectl edits outside git.
Terraform-only Kubernetes Secret resources with variables from CI: better if CI injects secrets at apply time and state is encrypted. Weakness: Terraform state still holds secret material unless carefully excluded; rotation requires terraform apply per environment; short-lived tokens fight plan/apply cadence.
External Secrets Operator: Terraform provisions ESO and SecretStore; ESO reconciles ExternalSecret CRDs to native Secrets. Rotation in upstream vault propagates on refresh interval without redeploying app manifests. Cost: CRD lifecycle, RBAC, debugging sync failures.
Secrets Store CSI driver: mounts secrets as volumes without persisting to etcd as K8s Secret. Strong for zero-trust purists; more moving parts for apps expecting env vars; pod scheduling tied to provider availability.
Decision table: pick wiring before the next audit
| Approach | Secret in git? | Rotation without redeploy | App change | Best when |
|---|---|---|---|---|
| Secret YAML in repo | Yes (encoded) | No | None | Throwaway dev clusters only |
TF kubernetes_secret + CI vars |
No plaintext | Apply per env | None | Small fleet, mature TF pipeline |
| External Secrets Operator | References only | Yes (refreshInterval) | None if using K8s Secret | Multi-env .NET/RabbitMQ/Redis stacks |
| CSI driver mount | References only | Pod restart may be needed | Volume mount path | Strict no-etcd-secret policy |
Boundary rules for paved paths
Terraform owns cluster infrastructure, namespaces, ESO install, IAM or Vault policies, and ExternalSecret templates with placeholder keys — not live secret values in version control. ESO owns continuous reconcile from vault to Kubernetes Secret objects apps already consume. Application teams own which keys they request via ExternalSecret spec; they do not kubectl-create Secrets by hand in production. Anti-corruption: if a developer patches a Secret manually, the next ESO reconcile overwrites or flags drift depending on policy; document that manual edits are ephemeral.
HashiCorp’s Terraform Kubernetes provider tutorial shows resource wiring; pair it with External Secrets Operator documentation so Terraform stops at the sync boundary rather than becoming a secret warehouse.
Acceptance protocol: prove rotation in sandbox
Run this before declaring the paved path production-ready.
1. Provision sandbox namespace via Terraform module pinned to a tagged version.
2. Create test secret platform/acceptance/db in Vault with username v1.
3. Apply ExternalSecret; assert Kubernetes Secret key matches within one refreshInterval.
4. Deploy sample ASP.NET pod mounting Secret as env; curl endpoint returns configured username.
5. Rotate Vault value to v2 without touching git or Terraform.
6. Wait refreshInterval plus margin; assert Secret data updated; restart pod if app caches env at start.
7. Confirm application reads v2; grep audit logs for human kubectl Secret edits — expect none.
8. Delete manual Secret; ESO recreates — drift resistance confirmed.
Failure at step 5–6 is the usual ESO misconfiguration: wrong SecretStore auth, namespace RBAC, or property key mismatch. Fix in module, not per-service heroics.
Module layout that scales past three environments
Structure Terraform as: modules/eks-cluster, modules/external-secrets-store, modules/app-external-secret, and thin roots envs/dev, envs/staging, envs/prod. The app module accepts only logical key names (ConnectionStrings__Primary) and SecretStore reference; it never sees plaintext.
Pin module sources to Git tags. Promotion PR bumps the tag in staging root, runs acceptance protocol, then bumps prod. Drift detection on ExternalSecret status conditions (Ready=False) belongs in the same alert route as pod CrashLoopBackOff.
Developer laptops use local Kind or sandbox subscriptions with the same ESO path, not hand-crafted Secrets that “work on my machine” and differ from CI. Local vault namespaces mirror key naming so application config binding stays identical.
Security review questions buyers actually ask
Can anyone with repo read access obtain production credentials? With ESO, answer should be no: repo holds ExternalSecret manifests pointing at vault paths. Who can read Secret objects in etcd? Restrict via RBAC; prefer namespace-scoped roles. Are rotation events audited in vault and Kubernetes events? Export both to SIEM. Does break-glass exist? Document emergency kubectl access with time-bound approval, knowing ESO reconcile may overwrite manual patches.
Map each secret to owning squad in a catalogue linked from the module README. Ownership means rotation runbooks and on-call, not “we wrote the Terraform once.” Incidents from expired certificates trace to missing ownership more often than to ESO bugs.
When custom wiring still beats a catalogue module
Multi-cloud secret backends, cross-account IAM, or regulated environments requiring HSM-backed keys may need a bespoke SecretStore per cluster class. That is not an excuse to commit YAML secrets “temporarily.” Temporary becomes five years. Custom work belongs in the platform module and acceptance test above, not in each microservice repo.
Composable platform design — modules, paved paths, and bounded IDP scope — is the wider frame we describe in composable platform engineering: paved paths, not monolithic IDP. Secret wiring is one edge in that graph; treat it as product with an acceptance test, not folklore passed in Slack.
Standardising Terraform, Kubernetes, and .NET secret consumption across squads? Talk to BlackFlow about platform modules that pass rotation drills before auditors ask.
