Environment Variable Risks

Environment Variable Risks

Environment variables are visible in process listings, inherited by child processes, captured in crash dumps, and logged by every debugging tool -- they are the worst place to store secrets

When to Use

• Evaluating how to pass secrets to applications at runtime
• Auditing existing applications that use environment variables for secrets
• Designing a safer alternative to .env files and environment variable injection
• Understanding why secrets leak in CI/CD pipelines and container orchestrators
• Migrating from env-var-based secrets to a vault or mounted-file approach

Threat Context

The 12-factor app methodology (2011) recommended environment variables for configuration, including secrets. This advice was reasonable for its time but has aged poorly. Environment variables are exposed in: /proc/<pid>/environ on Linux (readable by same-user processes), docker inspect output, Kubernetes pod descriptions (kubectl describe pod), CI/CD build logs (many CI systems log env vars by default or on error), crash dumps and core files, error reporting services (Sentry, Datadog, etc. capture environment), child processes (env vars are inherited by all child processes, including those you did not write). The 2021 Codecov supply chain attack exfiltrated secrets by reading environment variables from thousands of CI/CD pipelines. The modified Bash Uploader script silently sent every environment variable to an attacker-controlled server, compromising CI tokens, AWS keys, and database credentials across the software industry.

Instructions

1. Understand the leakage surface. Environment variables are: visible to any process running as the same user via /proc/<pid>/environ, inherited by all child processes and subshells, captured in core dumps, logged by many frameworks in debug mode, exposed in container inspection commands, and often included in error reports sent to third-party services. Every layer of the stack is a potential leak point. A single console.log(process.env) in a debugging session, a stack trace in Sentry, or a docker inspect command exposes every secret in the process environment.

2. Prefer mounted files over environment variables. Instead of DATABASE_URL=postgres://user:pass@host/db, mount a file at /var/secrets/database-url and read it at startup. Files can have restricted permissions (0400, owned by the application user), are not inherited by child processes, do not appear in process listings, and are not captured in crash dumps. Kubernetes Secrets can be mounted as files. Vault Agent writes secrets to files in a tmpfs mount (in-memory filesystem, never touches disk).

3. If you must use env vars, limit exposure. Read the env var once at startup, store it in application memory, and unset the env var immediately (delete process.env.DATABASE_URL in Node.js, os.environ.pop('DATABASE_URL') in Python). This limits the window during which the secret is visible in /proc/environ. However, this does not prevent inheritance by child processes started during the window, and the string may persist in the process heap even after unsetting.

4. Audit CI/CD for env var logging. Many CI systems (GitHub Actions, GitLab CI, CircleCI) mask secrets in logs but only if the secrets are registered as secret variables. Unregistered env vars containing secrets are logged in plaintext on every build. Audit all env vars in CI/CD for secrets that should be registered as masked/secret variables. Use CI secret scanning tools (gitleaks, truffleHog) on CI logs to detect accidental exposure.

5. Use runtime secret injection. Instead of setting env vars in Dockerfiles or docker-compose files (which bakes them into images or version-controlled files), inject secrets at runtime via: Kubernetes Secrets mounted as files, Vault Agent sidecar, cloud provider secret injection (AWS Secrets Manager + ECS task definitions, GCP Secret Manager + Cloud Run), or init containers that fetch secrets and write them to shared volumes. The secret should never exist in a layer that is persisted or version controlled.

6. Never put secrets in Dockerfiles or docker-compose.yml. ENV DATABASE_PASSWORD=hunter2 in a Dockerfile bakes the secret into every image layer. Docker layer caching means the secret persists even if you delete it in a later layer (RUN unset does not remove it from the layer that set it). Use multi-stage builds with the --secret flag (BuildKit) or runtime injection. Similarly, docker-compose.yml with inline environment values is version-controlled -- secrets in it are secrets in git.

Details

Leakage Vector Inventory

Every path through which environment variables can leak:

• /proc/<pid>/environ -- readable by any process running as the same user on Linux
• ps e command -- shows environment of running processes
• docker inspect <container> -- displays all env vars in the container config
• kubectl describe pod -- shows env vars defined in the pod spec
• heroku config -- displays all config vars in plaintext
• CI/CD build logs -- many systems echo env vars on failure or in debug mode
• Crash dumps and core files -- include the full process environment
• Error reporting services -- Sentry, Bugsnag, Datadog APM capture environment
• Child process inheritance -- every spawned subprocess inherits all env vars
• Shell history -- if secrets are set via export SECRET=value on command line
• .env files committed to git -- even with .gitignore, mistakes happen
• Docker image layers -- ENV instructions are baked into the image
• Terraform state files -- store plaintext values of env vars set via TF_VAR_*

Safer Alternatives Comparison

Method	Leakage Risk	Rotation Support	Audit Trail	Complexity
Environment variables	High	Manual	None	Low
Mounted files (tmpfs)	Low	Manual/Auto	OS-level	Low
Vault Agent injection	Low	Automatic	Full	Medium
K8s CSI Secret Store	Low	Auto-sync	K8s audit	Medium
SOPS-encrypted files	Medium	Manual	Git history	Low
Runtime API fetch	Low	Automatic	Full	Medium

The Codecov Attack Case Study

In January 2021, attackers modified Codecov's Bash Uploader script (used in CI/CD pipelines across the industry) to exfiltrate all environment variables to an attacker-controlled server. The attack persisted for two months before discovery. It worked because CI pipelines routinely have dozens of secrets in environment variables -- AWS keys, database credentials, API tokens, signing keys -- and any script with process access can read them all. Affected organizations included Twitch, Hashicorp, Confluent, and hundreds of others.

12-Factor App Reinterpretation

The 12-factor app's "store config in environment" principle was about separating config from code, not about security. The spirit of the principle (externalize configuration) can be achieved more securely with mounted files, secret managers, or runtime injection. The letter of the principle (use env vars specifically) has known security limitations that its authors did not anticipate in 2011. Modern best practice: use env vars for non-secret configuration (feature flags, log levels, service URLs) and mounted files or vault injection for secrets.

Anti-Patterns

1. .env files committed to version control. Even with .gitignore, .env files end up in git history through mistakes. Once committed, the secrets are in every clone forever (until the repository is purged with BFG or git-filter-branch). Use .env.example with placeholder values; actual .env files must never be committed.

2. Secrets in Docker Compose files. docker-compose.yml is version-controlled. Secrets in it are secrets in git. Use docker-compose.yml with env_file pointing to a file not in version control, or use Docker secrets for swarm mode deployments.

3. CI/CD secrets as unmasked env vars. If a CI secret is not registered as a masked variable, any printenv or debug log step exposes it in plaintext build logs that may be retained for months. Audit all CI/CD env vars and register secrets as masked/secret variables in the CI system's secret management.

4. Trusting unset to remove secrets. Unsetting an env var removes it from the process environment listing but does not scrub it from memory (the string may persist in the heap until garbage collected or overwritten). It also does not affect child processes that already inherited the variable before the unset.

5. Using the same env var across all environments. DATABASE_URL set to the production database URL in development environments means a development machine compromise leaks production credentials. Use environment-specific secret injection with separate credentials per environment. Development should use development-only credentials that cannot access production data.