Secrets Management for ML Pipelines

A machine learning pipeline holds a lot of keys: cloud storage for the training data, a database for features, a registry for models, sometimes a paid API. And it has more places to leak them than ordinary software, because data science tools are built to record everything: notebooks keep outputs, experiment trackers keep every parameter, and logs keep configs.

Secrets management for ML is mostly about closing those ML-specific leaks. This guide follows one example key through a project and shows where it is caught, where it is stored by accident, and how to keep it out.

Where ML pipelines leak secrets

OWASP's cheat sheet starts from an uncomfortable observation: many organisations have secrets hardcoded in plaintext in source code and scattered through configuration files. It calls for centralised storage, provisioning, auditing and rotation, with least privilege so that engineers cannot reach every secret (OWASP: secrets management). In ML projects the usual leak paths are:

• Notebooks, with keys pasted into cells and outputs saved in the file.
• Config files committed beside the training code.
• Experiment trackers, when a whole config is logged as parameters.
• Logs and error messages that print configs or connection strings.
• Container images with credentials baked into a layer; OWASP warns never to hardcode secrets with Docker ENV or ARG.
• Saved models pickled together with their config objects.

Following one key

AWS's documentation illustrates access keys with an example pair, AKIAIOSFODNN7EXAMPLE and a matching secret (AWS: access keys). Those published examples are safe to plant. The script writes them into a project config, scans it with detect-secrets, logs the config to MLflow twice, writes it to a log through a redaction filter, and fetches a token from AWS Secrets Manager running in Moto, an emulator that mocks AWS services locally (Moto).

import json
import logging
import shutil
import sqlite3
import sys
from pathlib import Path

import boto3
import mlflow
from detect_secrets import SecretsCollection
from detect_secrets.settings import default_settings
from moto import mock_aws

KEY_ID, SECRET = "AKIAIOSFODNN7EXAMPLE", "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"   # AWS's published example values
Path("tracking.db").unlink(missing_ok=True)                   # start from a clean folder
shutil.rmtree("mlruns", ignore_errors=True)

print("1. Scanning a project before it is committed")
Path("project").mkdir(exist_ok=True)
Path("project/train_config.py").write_text(f'S3_BUCKET = "training-data"\nAWS_ACCESS_KEY_ID = "{KEY_ID}"\n'
                                           f'AWS_SECRET_ACCESS_KEY = "{SECRET}"\n')
found = SecretsCollection()
with default_settings():
    found.scan_file("project/train_config.py")
for path, findings in found.json().items():
    for finding in findings:
        print(f"  {Path(path).as_posix()} line {finding['line_number']}: {finding['type']}")

print("\n2. Experiment tracking stores whatever it is given")
config = {"learning_rate": 0.1, "max_depth": 6, "aws_secret_access_key": SECRET}
mlflow.set_tracking_uri("sqlite:///tracking.db")
with mlflow.start_run(run_name="careless"):
    mlflow.log_params(config)                                   # the whole config, secret included
ALLOWED = {"learning_rate", "max_depth"}
with mlflow.start_run(run_name="allowlisted"):
    mlflow.log_params({k: v for k, v in config.items() if k in ALLOWED})
for (run_name,) in sqlite3.connect("tracking.db").execute("select name from runs order by start_time"):
    leaked = sqlite3.connect("tracking.db").execute(
        "select p.key from params p join runs r on p.run_uuid = r.run_uuid where r.name = ? and p.value = ?",
        (run_name, SECRET)).fetchall()
    print(f"  run '{run_name}': " + (f"secret stored in plain text as {[k for (k,) in leaked]}" if leaked else "no secret stored"))

print("\n3. Logs")


class Redact(logging.Filter):                                   # replaces known secret values in every record
    def __init__(self, *values):
        super().__init__()
        self.values = values

    def filter(self, record):
        message = record.getMessage()
        for value in self.values:
            message = message.replace(value, "[REDACTED]")
        record.msg, record.args = message, ()
        return True


log = logging.getLogger("training")
handler = logging.StreamHandler(sys.stdout)
handler.setFormatter(logging.Formatter("  %(message)s"))
log.addHandler(handler)
log.setLevel(logging.INFO)
log.addFilter(Redact(SECRET))
log.info("starting training with %s", config)

print("\n4. A secrets manager (moto's local emulator, not AWS)")
with mock_aws():
    manager = boto3.client("secretsmanager", region_name="ap-south-1")
    manager.create_secret(Name="ml/registry-token",                # done once, by an administrator
                          SecretString=json.dumps({"token": "demo-registry-token-01"}))
    token = json.loads(manager.get_secret_value(SecretId="ml/registry-token")["SecretString"])["token"]   # at run time
    print(f"  fetched at run time: {len(token)} characters, never written to code, config or logs")

1. Scanning a project before it is committed
  project/train_config.py line 2: AWS Access Key
  project/train_config.py line 3: AWS Access Key
  project/train_config.py line 3: Base64 High Entropy String
  project/train_config.py line 3: Secret Keyword

2. Experiment tracking stores whatever it is given
  run 'careless': secret stored in plain text as ['aws_secret_access_key']
  run 'allowlisted': no secret stored

3. Logs
  starting training with {'learning_rate': 0.1, 'max_depth': 6, 'aws_secret_access_key': '[REDACTED]'}

4. A secrets manager (moto's local emulator, not AWS)
  fetched at run time: 22 characters, never written to code, config or logs

What the output shows

• The scanner caught both values. detect-secrets, built to find secrets in a code base and keep new ones out (detect-secrets), flagged the key ID by its pattern and the secret three ways: its key pattern, its high entropy and the keyword in the variable name. Run it as a pre-commit hook and in CI, before a push, because once a key is pushed it lives in history. GitHub's own secret scanning checks the entire history on all branches and runs free on public repositories (GitHub Docs: secret scanning).
• The experiment tracker kept the secret in plain text. The careless run logged the whole config, and the key now sits in the tracking database, readable by anyone with access to the tracking server. The allowlisted run logged only the two hyperparameters. Trackers record whatever they are given, so give them an explicit list.
• The log filter redacted the value before it was written. Filters only catch values they know about, so the stronger habit is not logging whole configs at all.
• The secrets manager kept the token out of everything. The pipeline fetched it at run time; it never appeared in code, config, tracker or log.

Environment variables: convenient, not ideal

The Twelve-Factor App recommends keeping config in environment variables, strictly separate from code, and offers a useful test: could the codebase be made open source at any moment without compromising any credentials (Twelve-Factor: config)? OWASP is more cautious. Environment variables are visible to every process and can end up in logs and system dumps, so it recommends them only when better methods are not possible. In practice, a secrets manager or a mounted secret file is safer, and environment variables injected by the orchestrator at run time are an acceptable fallback.

Secrets in automated pipelines

• Use the CI system's secret store and pass each secret only to the steps that need it; see GitHub Actions for ML.
• Give every pipeline its own credentials, scoped to what it does: a training job can read the data bucket, and only the publishing job can write to the registry.
• Separate development and production keys.
• Rotate on a schedule, and at once after any suspected leak.
• Keep scanners in the delivery pipeline; see CI/CD for machine learning.

For tracking experiments without leaking, see the MLflow tutorial; for structured logs that leave sensitive fields out, see logging and observability for ML systems; and for container builds, see Docker for machine learning.

If a key leaks

1. Revoke or rotate it immediately. Deleting it from the code does not un-leak it.
2. Check the provider's access logs for use of the key.
3. Remove it from history, trackers and logs where you can.
4. Add a scanner rule or hook so the same mistake fails the next commit.

Common mistakes

• Logging the entire config to the experiment tracker.
• Keys in notebooks, including their saved outputs.
• Secrets baked into container images.
• One powerful key shared by every pipeline.
• Removing a leaked key from code without revoking it.

Data security and access management are covered in the Security and Governance module of the MLOps course.

Sources

• Secrets Management Cheat Sheet (OWASP) — OWASP
• III. Config (The Twelve-Factor App) — The Twelve-Factor App
• Manage access keys for IAM users (AWS documentation) — Amazon Web Services
• detect-secrets (GitHub) — Yelp
• Moto: Mock AWS Services (documentation) — Moto
• Secret scanning (GitHub Docs) — GitHub