Secrets management: issuance, delivery, rotation and revocation

TL;DR: Secret management includes who may obtain a credential, where it can travel, how consumers replace it and how old access is revoked. Putting a password in a vault addresses storage; the application still needs a secure bootstrap identity and a working rotation path.

Start with the identity that retrieves the secret

A service reading a database password from a central store must first authenticate to that store. Prefer a workload identity that can obtain narrowly scoped, short-lived authorization where the environment supports it. If the retrieval token is another long-lived password copied into every deployment, the design has moved the bootstrap secret rather than eliminated it.

Scope access around the workload and environment. A staging job should not read production database credentials simply because both use the same repository. Separate issuance privileges from ordinary read privileges, and record which identity accessed which secret version without logging the secret value.

The OWASP secrets-management guidance covers lifecycle, access and automation considerations. Apply those controls to the whole path, including logs, build outputs, backups and support tooling.

Model the lifecycle as states with owners

Routine rotation uses overlap to preserve availability. Emergency revocation may intentionally interrupt access to contain exposure. Decide who can make that decision and how affected services recover without reintroducing the compromised credential.

Delivery method changes exposure and refresh

No row makes a credential impossible to extract from a compromised authorized workload. Minimize its privileges and useful lifetime. A process that can use a database credential necessarily has some path to exercise its authority.

Kubernetes Secret data is commonly base64-encoded in manifests. Encoding is not encryption. Configure appropriate storage encryption and access control, and avoid committing the plaintext or encoded credential to Git. The Kubernetes Secret good practices explain the relevant cluster controls.

Rotate a database credential without guessing

Assume the database supports two independently valid credentials during migration. Issue credential B with the intended permissions while A remains valid. Test B from a controlled application path, publish the new secret version, and make consumers establish fresh connections using B. Observe failures and adoption before revoking A.

Connection pools complicate verification. Existing sessions authenticated with A may remain usable according to database behavior even after the password changes. Check new connection success separately from the health of old pooled sessions. If the database does not support the overlap you need, choose an alternate account strategy or a planned interruption under its documented authentication behavior.

Define a maximum overlap window. Leaving both credentials active indefinitely turns rotation into credential accumulation. Record which workloads have migrated and what prevents the remaining ones from doing so. Validate that A can no longer authenticate through a permitted test after revocation.

A store outage should have a designed consequence

An application may continue using an already obtained credential until it expires or is revoked. A restart during the same outage may fail to retrieve it. Short credential lifetimes reduce exposure but increase dependence on timely refresh and identity infrastructure.

Specify how much interruption the service can tolerate, when renewal begins, and whether new work is admitted as expiry approaches. Do not hide permanent fallback credentials in the code as an undocumented recovery mechanism. If break-glass access is required, make it separately controlled, audited and time-bounded.

Dynamic secrets can reduce shared credential use by issuing distinct, leased credentials to consumers. They add renewal and revocation behavior that must be tested; Vault leases provide a detailed exercise.

When a secret appears in Git

Revoke or rotate the exposed credential according to its authority and exposure. Removing the file from the latest commit is insufficient because clones, history, logs and artifacts can retain it. Investigate use of the credential and coordinate any required repository-history cleanup after containment.

Prevention includes scoped pipeline permissions, secret scanning, reviewed artifact contents and log redaction. Masking known values can reduce accidental display, but it is not a security boundary against code that has permission to read and transmit them. Keep untrusted pull-request execution away from privileged credential delivery.

Check the rotation claim

A dashboard says every Pod has the new secret mounted. Half the Pods still use old database connections, and the team wants to revoke the old account immediately. What is missing?

Verify application reload behavior and new connection authentication, then account for long-running work and pool draining. File delivery establishes that new material arrived, not that the consumer adopted it. Revoke under the planned window once adoption and recovery are established, or explicitly accept the interruption if emergency containment requires it.

Connect this lifecycle to workload identity and the credential boundaries in CI/CD pipeline architecture.

Infrastructure tooling creates additional copies to control. Terraform secrets in state and plans traces those copies beyond console redaction.