Security Deep Dive

Secrets Management — Don't Commit Production

Every system has secrets — database passwords, API keys, signing keys, tokens. Where they live, who can read them, and how often they rotate is one of the highest-leverage security decisions you'll make. Get it wrong and a single leaked file becomes a full breach.

VaultKMSRotationOIDCLeast Privilege
← Back to Security
Quick Facts

What Counts as a Secret

The usual suspects

  • Credentials: DB passwords, service-account passwords, API keys, OAuth client secrets.
  • Cryptographic keys: JWT signing keys, TLS private keys, encryption keys, code-signing keys.
  • Tokens: personal access tokens, deploy tokens, webhook signing secrets.
  • Connection strings: often contain credentials inline. Treat the whole thing as a secret.
  • "Internal-only" URLs & IPs. Less critical, but still shouldn't be in client code.
Principles

The Rules That Actually Matter

  • Never in source. Not in config files, not in .env.example, not in tests. Once it lands in Git history it must be considered leaked forever.
  • Never in client code. Anything shipped to a browser or app is public.
  • Never in logs. Scrub at the logging layer; don't rely on developers remembering.
  • Least privilege per identity. The CI deploy role doesn't need read access to user data.
  • Short-lived over long-lived. An hour-long token is far less dangerous than one with no expiry.
  • Rotate on a schedule and on incidents. If you can't rotate quickly, you can't respond to a leak.
  • Auditable access. Who read what secret, when. You need this after an incident.
  • Encrypt at rest and in transit. Default behavior of vetted tools — don't roll your own.
Storage

Where Secrets Should Actually Live

ToolSweet spot
HashiCorp VaultSelf-hosted gold standard. Dynamic secrets, leases, transit engine for encrypt-as-a-service.
AWS Secrets Manager / SSM Parameter StoreNative to AWS workloads; rotation hooks for RDS and friends.
GCP Secret Manager / Azure Key VaultCloud-native equivalents. Tight IAM integration.
Doppler / 1Password Secrets Automation / InfisicalHosted, dev-friendly, multi-environment. Good for small/mid teams.
Kubernetes Secrets (with KMS or Sealed Secrets)K8s-native. Plain Secrets are base64, not encrypted — pair with KMS at rest, or use Sealed Secrets / SOPS / External Secrets Operator.
SOPS + KMSEncrypted files in Git, decrypted at deploy. Good middle ground when a vault is overkill.
Cloud KMS (AWS KMS, GCP KMS, Azure Key Vault)Not for arbitrary secrets — for encrypting/decrypting things, including other secrets. The root of trust below the vault.

Default for small teams: your cloud's secret manager. Default for a self-hosted stack: Vault.

Delivery

How Secrets Reach the App

  • Environment variables at process start — simple, 12-factor, works everywhere. Watch for: env leaking via crash dumps, child processes, error pages.
  • Mounted files (Kubernetes-style) — secret rotates, file updates; app re-reads on a SIGHUP or watch. Less leaky than env.
  • Vault SDK at runtime — the app fetches secrets directly using a workload identity. Most flexible; supports leases and dynamic secrets.
  • Sidecar / agent — Vault Agent or Secrets Store CSI Driver renews and rotates without the app caring.

Whichever method, the principle is the same: the secret enters the process at runtime, not at build time. Never bake a secret into a container image.

CI/CD & Cloud Identity

OIDC Killed the Long-Lived Cloud Key

The single biggest improvement in secrets hygiene over the last few years: workload identity federation. Instead of giving CI a static AWS access key, the CI provider issues a short-lived OIDC token that the cloud trades for a temporary role.

  • GitHub Actions → AWS / GCP / Azure via OIDC: no long-lived keys, automatic per-job credentials, scoped to specific repos/branches.
  • Kubernetes Service Accounts map to cloud IAM via IRSA (AWS), Workload Identity (GCP), or Pod Identity (Azure).
  • Effect: a leaked CI log is no longer a leaked production credential.

If you still have static cloud keys in CI in 2026, that's the highest-impact thing to fix this quarter.

Detection

Catch Leaks Before They Land

  • Pre-commit hooks: gitleaks, detect-secrets, trufflehog. Stop the commit at the developer's machine.
  • Pre-push / CI scan: same tools, run server-side. Catches what hooks missed.
  • GitHub / GitLab secret scanning: built-in for many platforms. Free real-time monitoring on the entire history.
  • Provider notifications: AWS, Stripe, Slack, GitHub all monitor leaked-token feeds and notify (or auto-revoke). Don't disable these.
  • Canary tokens: deliberately plant a fake AWS key in source — if it's used, you know exactly when and from where.
  • SBOM & image scanning for secrets baked into containers (it happens more than you'd think).
Rotation & Response

When (Not If) You Leak

  1. Revoke the secret immediately. Don't try to "rewrite Git history" first — by the time you've thought of it, the secret is on a hundred forks.
  2. Rotate the underlying credential. New key, new token, new password. Update the vault.
  3. Invalidate sessions / tokens that used it. Force re-auth where it matters.
  4. Audit access logs for use between leak and revocation. This is where short-lived credentials and good logging save you.
  5. Postmortem. How did it land in source? Add a control that prevents a repeat.

Rotation isn't only for incidents — schedule it. If your secret hasn't been rotated in years, the only honest thing to assume is that it's been leaked at some point.

Common Pitfalls
  • "It's just a dev key." Dev keys often hit prod data. Treat all secrets as production-grade.
  • Committing .env "by accident." Add it to .gitignore and template .env.example with placeholders only.
  • Logging the request object. Auth headers and signed cookies end up in your log pipeline. Scrub at the source.
  • Hardcoded encryption keys. Static keys in source mean every breach decrypts everything.
  • Sharing one IAM role across services. Hard to audit, hard to scope. One identity per service.
  • Long-lived personal access tokens with admin scope. The classic source of supply-chain incidents.
  • "We'll rotate when it leaks." By then you don't know what leaked. Rotate proactively.
Worked Example

The URL Shortener Secret Map

SecretLifetimeStorageDelivered as
Postgres connectionDynamic, 1h leases via VaultVault DB engineSidecar-rendered file
Redis passwordStatic, rotated quarterlyCloud secret managerEnv var at start
JWT signing key30 days, dual-key for rolloverVault transit engine (signs without releasing key)Vault SDK at runtime
SMTP credentialsStatic, rotated yearlyCloud secret managerEnv var
CI → cloud deploy~15 min per jobWorkload identity (OIDC)Token exchange, no static key
3rd-party geo-IP API keyStatic, rotated quarterlyCloud secret managerEnv var

Notice: nothing in source, nothing baked in the image, the highest-blast-radius secret (DB) is dynamic and short-lived, and the JWT key never leaves Vault — the app asks Vault to sign on its behalf.

Continue

Related Reading