Stop Committing API Keys: A Stdlib Python Pre-Commit Hook That Scans Only Your Staged Lines

TL;DR: Once a secret is pushed, deleting it in the next commit doesn't help. It's in history, in forks and in every clone. The cheapest fix is to never let it into a commit. Below is a small, dependency-free Python script that reads only the lines you're about to commit (git diff --cached), flags well-known token formats plus high-entropy values assigned to names like api_key or secret, prints them redacted, and exits 1 so a pre-commit hook stops the commit.

The way it usually happens: you're debugging, you paste a key into settings.py "just for a minute", it works, you're tired, and git add . picks it up. Bots scan public GitHub for exactly this, and leaked cloud keys can get abused within minutes. Hosted push protection helps, but it runs after you've already committed, and not every remote has it. A local hook catches it earlier and works with any remote.

What we're going to catch

Two kinds of secrets need two kinds of checks:

1. Tokens with a known shape. Many providers use recognisable prefixes on purpose so scanners can find them: AWS access key IDs start with AKIA or ASIA, GitHub tokens with ghp_, gho_, github_pat_ and so on, Slack with xoxb-, Stripe live keys with sk_live_, Google API keys with AIza. Private keys start with a -----BEGIN ... PRIVATE KEY line. Regexes handle these with almost no false positives.
2. Generic secrets. A database password or a random API key from a smaller service has no prefix. Here we look for an assignment to a suspicious name (api_key = "...", "clientSecret": "...") and then ask: does the value look random? That's what Shannon entropy measures.

Entropy in one function

Shannon entropy is the average number of bits of information per character. A string of one repeated character scores 0. English-like placeholders score low. Random keys score high.

import math
from collections import Counter

def shannon_entropy(s):
    """Average bits of information per character of s."""
    if not s:
        return 0.0
    n = len(s)
    return max(0.0, -sum(c / n * math.log2(c / n) for c in Counter(s).values()))

Real numbers from the script:

Value	Entropy (bits/char)
xxxxxxxxxxxxxxxx	0.00
hunter2hunter2	2.81
changeme_changeme	2.91
correct-horse-battery	3.27
Zx9qL2vT8mR4wK7pN1sQ	4.32

I use a threshold of 3.5 and a minimum length of 12. That skips placeholders and still catches typical random tokens. Notice it also skips correct-horse-battery. Entropy per character is a weak signal for passphrases, which is one reason the hook is a safety net, not a guarantee.

Only scan what you're committing

Scanning the whole repo on every commit is slow and noisy: you'd keep tripping over old test fixtures. Instead we ask git for the staged diff with zero context lines and keep only added lines, tracking the file name and the new line number from each hunk header:

HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@")

def added_lines_from_diff(diff_text):
    """Yield (path, line_number, text) for every added line in a unified diff."""
    path, lineno = None, 0
    for raw in diff_text.splitlines():
        if raw.startswith("+++ "):
            target = raw[4:]
            path = None if target == "/dev/null" else target[2:] if target.startswith("b/") else target
        elif raw.startswith("@@"):
            m = HUNK.match(raw)
            lineno = int(m.group(1)) if m else 0
        elif raw.startswith("+") and path:
            yield path, lineno, raw[1:]
            lineno += 1
        elif raw.startswith(" "):
            lineno += 1

The +++ check comes first on purpose. Otherwise the file header line itself would look like an added line. Deleted files point at /dev/null and are skipped, because removing a secret is exactly what we want people to do.

The full script

#!/usr/bin/env python3
"""Block commits that add API keys, tokens or private keys.

Usage:
  python3 scan_secrets.py --staged        # scan lines added in `git diff --cached`
  python3 scan_secrets.py FILE [FILE...]  # scan whole files

Exit codes: 0 clean, 1 possible secrets found, 2 usage/git error.
Silence a false positive by adding "secret-scan: allow" on the same line.
"""
import math
import re
import subprocess
import sys
from collections import Counter

RULES = [
    ("AWS access key ID", re.compile(r"\b(?:AKIA|ASIA)[0-9A-Z]{16}\b")),
    ("GitHub token", re.compile(r"\bgh[pousr]_[A-Za-z0-9]{36,}\b")),
    ("GitHub fine-grained token", re.compile(r"\bgithub_pat_[A-Za-z0-9_]{60,}\b")),
    ("Slack token", re.compile(r"\bxox[abprs]-[A-Za-z0-9-]{10,}\b")),
    ("Stripe live secret key", re.compile(r"\b[sr]k_live_[A-Za-z0-9]{20,}\b")),
    ("Google API key", re.compile(r"\bAIza[0-9A-Za-z_\-]{35}\b")),
    ("Private key block", re.compile(r"-----BEGIN (?:RSA |EC |OPENSSH |DSA |PGP )?PRIVATE KEY")),
]

# key = "value" / KEY: 'value' / "apiKey": "value"
ASSIGNMENT = re.compile(
    r"""(?i)\b[\w.-]*(?:api[_-]?key|secret|token|passw(?:or)?d|auth)[\w.-]*["']?\s*[:=]\s*["']([^"'\s]{12,})["']"""
)
ALLOW_MARKER = "secret-scan: allow"
ENTROPY_THRESHOLD = 3.5  # bits per character


def shannon_entropy(s):
    """Average bits of information per character of s."""
    if not s:
        return 0.0
    n = len(s)
    return max(0.0, -sum(c / n * math.log2(c / n) for c in Counter(s).values()))


def redact(value):
    return value[:4] + "…" + f"({len(value)} chars)"


def scan_line(line):
    """Return a list of (rule_name, redacted_match) for one line of text."""
    if ALLOW_MARKER in line:
        return []
    hits = []
    for name, rx in RULES:
        for m in rx.finditer(line):
            hits.append((name, redact(m.group(0))))
    if not hits:  # only fall back to the noisy heuristic if nothing specific matched
        for m in ASSIGNMENT.finditer(line):
            value = m.group(1)
            if shannon_entropy(value) >= ENTROPY_THRESHOLD:
                hits.append((f"High-entropy value (entropy {shannon_entropy(value):.2f})", redact(value)))
    return hits


HUNK = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@")


def added_lines_from_diff(diff_text):
    """Yield (path, line_number, text) for every added line in a unified diff."""
    path, lineno = None, 0
    for raw in diff_text.splitlines():
        if raw.startswith("+++ "):
            target = raw[4:]
            path = None if target == "/dev/null" else target[2:] if target.startswith("b/") else target
        elif raw.startswith("@@"):
            m = HUNK.match(raw)
            lineno = int(m.group(1)) if m else 0
        elif raw.startswith("+") and path:
            yield path, lineno, raw[1:]
            lineno += 1
        elif raw.startswith(" "):
            lineno += 1


def staged_lines():
    out = subprocess.run(
        ["git", "diff", "--cached", "--unified=0", "--no-color", "--diff-filter=ACM"],
        capture_output=True, text=True, check=True,
    ).stdout
    return added_lines_from_diff(out)


def file_lines(paths):
    for p in paths:
        with open(p, encoding="utf-8", errors="replace") as fh:
            for i, line in enumerate(fh, 1):
                yield p, i, line.rstrip("\n")


def main(argv):
    if not argv or argv in (["-h"], ["--help"]):
        print(__doc__.strip())
        return 2
    try:
        lines = staged_lines() if argv == ["--staged"] else file_lines(argv)
        findings = [(p, n, rule, val) for p, n, text in lines for rule, val in scan_line(text)]
    except (subprocess.CalledProcessError, OSError) as exc:
        print(f"scan_secrets: {exc}", file=sys.stderr)
        return 2
    for p, n, rule, val in findings:
        print(f"{p}:{n}: {rule}: {val}")
    if findings:
        print(f"\n{len(findings)} possible secret(s). Remove them, or add '{ALLOW_MARKER}' "
              "on the line if it is a false positive.", file=sys.stderr)
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

A few choices worth explaining:

• Redacted output. The script prints the first four characters and the length, never the full value. Hook output ends up in terminal scrollback, CI logs and screenshots in team chat. A secret scanner that prints the secret is just another leak.
• Specific rules first, entropy second. If a line already matched a known token format, we don't also report it as "high entropy". One clear finding per line is easier to act on.
• An escape hatch. Test fixtures and docs sometimes need key-shaped strings. Adding secret-scan: allow to the line silences it, and the marker is easy to grep for in code review.
• Exit codes. 0 clean, 1 findings, 2 usage or git error, so a broken setup doesn't look like "no secrets found".

Install it as a git hook

Save the script in your repo as scripts/scan_secrets.py, then:

cat > .git/hooks/pre-commit <<'HOOK'
#!/bin/sh
exec python3 scripts/scan_secrets.py --staged
HOOK
chmod +x .git/hooks/pre-commit

.git/hooks isn't versioned, so each teammate runs those two commands once. If your team already uses the pre-commit framework, add it as a local hook instead:

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: scan-secrets
        name: scan staged lines for secrets
        entry: python3 scripts/scan_secrets.py --staged
        language: system
        pass_filenames: false

What it looks like

I staged this file in a fresh repo (the AWS value is the example key from AWS's own docs):

import os
DEBUG = True
AWS_ACCESS_KEY_ID = "AKIAIOSFODNN7EXAMPLE"
stripe_secret = "Zx9qL2vT8mR4wK7pN1sQ"
api_key = os.environ["API_KEY"]

Output, exit code 1:

settings.py:3: AWS access key ID: AKIA…(20 chars)
settings.py:4: High-entropy value (entropy 4.32): Zx9q…(20 chars)

2 possible secret(s). Remove them, or add 'secret-scan: allow' on the line if it is a false positive.

Line 5 passes, which is the point: reading the key from the environment is the fix.

Testing it

The test file covers the entropy function, every rule family, the allow marker, redaction (the full fake token must never appear in the output), diff line numbering including a deleted file, and two integration tests that create a throwaway repo with tempfile. One stages a file and checks the exit code and message. The other installs the hook and confirms git commit fails and no commit is created. All 10 pass on Python 3.13:

python3 test_scan_secrets.py
# Ran 10 tests in 0.13s
# OK

One practical tip: build fake tokens in tests by concatenation ("ghp_" + "a1B2..."). Otherwise your own test file trips the hook, and GitHub's push protection may flag it too.

If a secret already got committed

The hook protects future commits. For one that's already pushed:

1. Revoke and rotate the key first. Assume it's compromised the moment it was pushed. Rewriting history doesn't un-leak it.
2. Check the provider's logs for usage you don't recognise.
3. Then clean history if you need to (git filter-repo is the usual tool) and force-push, knowing existing clones still have it.
4. Move the value to an environment variable or a secrets manager and add the file pattern (.env, *.pem) to .gitignore.

Limits

• Local hooks can be skipped with git commit --no-verify. Run the same script in CI as a second layer by diffing the pull request against its base branch and piping that into added_lines_from_diff.
• Regexes miss formats you didn't list. Add rules for the providers you actually use.
• Entropy has false positives and misses. Hashes, UUIDs and base64 test data can trigger it, and low-entropy passwords slip through. The allow marker covers the first, code review covers the second.
• For bigger teams, dedicated scanners such as gitleaks or trufflehog ship hundreds of rules and history scanning. This script is the small version you can read in five minutes and drop into any repo today.

Leaked keys are one half of account security. The other half is making a stolen password useless on its own, which I wrote about in SMS 2FA vs authenticator apps.