Secrets Management

What is it? #

Secrets are the credentials your system needs: database passwords, API keys, signing keys, service tokens.

They need to reach the application without being stored in the repository, readable by everyone, or impossible to change.

There is a progression. Environment files on a server are adequate for small systems; a secret manager adds rotation, audit logging and access control; short-lived credentials remove stored secrets almost entirely.

Rotation is the capability most setups lack. A system where changing a database password requires a coordinated manual effort will not rotate after an incident, which is exactly when it must.

Think of it like this #

Keys to a building. Who holds which key, whether you can tell who used one, and how quickly a lost key can be invalidated.

Handing out copies of a master key that never changes is quick and leaves you with no options when one goes missing.

Simple example #

An application needs a database password, a payment API key and a signing secret. They live in a manager, are injected at startup, and can each be rotated independently without a deployment.

Code #

The progression

1. env file on the server        chmod 600, owned by the service user
   adequate for a single server; no rotation support, no audit trail

2. secret manager                Vault, AWS Secrets Manager, GCP Secret Manager
   versioned, access-controlled, audited, supports rotation

3. short-lived credentials       generated per session or per deployment
   nothing long-lived to leak; the strongest option where supported

Most teams should be at 2 for production and can reach 3 for cloud access.

# Fetching at startup rather than storing on disk
import boto3, json, functools

@functools.lru_cache(maxsize=1)
def secrets() -> dict:
    client = boto3.client("secretsmanager")
    raw = client.get_secret_value(SecretId="prod/myapp")["SecretString"]
    return json.loads(raw)

DATABASE_URL = secrets()["database_url"]
# Fetched once at startup. Restarting picks up a rotated value, so
# rotation is a restart rather than a deployment.

Rotation without downtime: two credentials, briefly

1. create the new credential alongside the old one
2. deploy or restart so the application uses the new one
3. verify: no errors, and the old credential shows no recent use
4. revoke the old credential

Rotating in a single step means a window where the old credential is
gone and the new one is not yet in use — an outage.

Rotate: on staff departure, on any exposure, and on a schedule for
long-lived keys.

# Detection: catch leaks before they are pushed
pip install pre-commit detect-secrets
detect-secrets scan > .secrets.baseline
# .pre-commit-config.yaml runs it on every commit

# And in CI, scanning history
gitleaks detect --source . --verbose

When a secret leaks

1. ROTATE IT. This is the only step that helps. Do it first.
2. Check audit logs for use between exposure and rotation.
3. Remove it from the code; rewriting history is optional and
   does not undo the exposure.
4. Add detection so the next one is caught before it is pushed.

Assume any secret that was ever public has been captured. Public
repository scanners find credentials within minutes of a push.

How it works #

Fetching secrets at startup rather than reading a file means rotation is a restart, not a redeployment. The application always gets the current version.

Caching the fetch avoids calling the manager on every request, which matters for both latency and cost.

The two-credential rotation pattern is what makes rotation safe. Both credentials are valid briefly, the switch is verified, and only then is the old one revoked. Doing it in one step guarantees a gap.

Verifying that the old credential shows no recent use before revoking is the step that catches forgotten consumers — a background job or a second service still using it.

Pre-commit scanning catches secrets before they leave a developer's machine, which is far better than detecting them afterwards. CI scanning of history catches what slipped through.

Rotation is the response to exposure, and it is the only one that matters. Removing the commit does not help, because the value was published and may have been captured.

The speed of automated scanning is the reason for urgency. Credentials pushed to a public repository are routinely used within minutes.

Real-world use #

Leaked credentials remain one of the most common causes of compromise. Cloud keys in public repositories are abused almost immediately, usually for cryptocurrency mining that produces a large bill.

Secret managers are standard in cloud environments and integrate with the platform's identity system, so an application can fetch secrets without holding a credential to do so.

Short-lived credentials are the direction of travel. Instance roles, workload identity and OIDC in CI all remove the stored key entirely, which removes the leak and the rotation problem together.

Audit logging is an underrated benefit. Knowing which service read which secret and when is valuable during an investigation, and impossible with a file on a server.

Least privilege applies to secrets as much as to users: a service should only be able to read the secrets it needs, not the whole store.

Common mistakes #

• Secrets committed to the repository, then only deleted rather than rotated.
• No rotation capability, so credentials are never changed after an incident.
• Rotating in one step and causing an outage.
• One shared credential used by several services, so it cannot be rotated independently.
• Every service able to read every secret, rather than only its own.

Practice #

Inventory every secret in one system: what it is, where it is stored, who can read it and when it was last changed. Then rotate one using the two-credential method without downtime, and add pre-commit secret scanning to the repository.