# Scan HTTP Security Headers in Python and Fail CI When HSTS or CSP Is Missing

A TLS certificate only proves the browser reached the right host. It does not stop a page from being framed, sniffed as the wrong content type, or loaded on HTTP after a downgrade. Those protections live in response headers: `Strict-Transport-Security`, `Content-Security-Policy`, `X-Content-Type-Options`, and a clickjacking control (`X-Frame-Options` or CSP `frame-ancestors`).

This tutorial builds a small Python CLI that fetches a URL, checks those headers, and **exits 1** when required ones are missing — so you can drop it into CI after every deploy.

Stdlib only: `argparse`, `json`, `urllib`, `email.message`. No pip install.

**TL;DR:** `python3 scan_security_headers.py https://github.com/` should print `OK   True`. `python3 scan_security_headers.py https://example.com/` exits 1 because the required headers are absent. Add `--strict` to also fail on missing recommended headers.

## What each header is for

- **Strict-Transport-Security (HSTS)** — tells the browser to use HTTPS for this host for `max-age` seconds. Without it, a first visit (or a bookmark saved as `http://`) can still be downgraded.
- **Content-Security-Policy (CSP)** — restricts where scripts, styles, images, and connections may come from. A good CSP is the main defense against injected JavaScript.
- **X-Content-Type-Options: nosniff** — stops the browser from guessing a MIME type. Without it, a downloaded file the server called `text/plain` might be executed as HTML or JavaScript.
- **X-Frame-Options or CSP `frame-ancestors`** — controls who may embed the page in an iframe (clickjacking). Modern guidance prefers `frame-ancestors` inside CSP; `X-Frame-Options` is the older standalone header. This scanner treats *either* as enough.
- **Referrer-Policy** — limits how much of the URL is sent as a `Referer` when people click outbound links. Recommended here, not required, unless you pass `--strict`.

None of these replace HTTPS, cookie flags, or a password-hashing scheme. They are the cheap, checkable layer that a lot of small sites forget to turn on.

## Step 1: normalize header names

HTTP header names are case-insensitive. Normalize once so the rest of the code can use lowercase keys:

```python
from email.message import Message

def normalize_headers(headers: Message | dict) -> dict[str, str]:
    if isinstance(headers, Message):
        items = headers.items()
    else:
        items = headers.items()
    out: dict[str, str] = {}
    for key, value in items:
        out[key.lower()] = value.strip()
    return out
```

`urllib` already gives you an `email.message.Message`, which behaves like a case-insensitive map, but turning it into a plain `dict` makes tests and JSON output boring and predictable.

## Step 2: decide what "ok" means

```python
REQUIRED = (
    "strict-transport-security",
    "content-security-policy",
    "x-content-type-options",
)
RECOMMENDED = ("referrer-policy",)

def has_frame_protection(headers: dict[str, str]) -> bool:
    if headers.get("x-frame-options"):
        return True
    csp = headers.get("content-security-policy", "")
    return "frame-ancestors" in csp.lower()

def evaluate(headers: dict[str, str], *, strict: bool = False) -> dict:
    missing_required = [name for name in REQUIRED if name not in headers]
    missing_recommended = [name for name in RECOMMENDED if name not in headers]
    if not has_frame_protection(headers):
        missing_recommended.append("x-frame-options|csp-frame-ancestors")

    xcto = headers.get("x-content-type-options", "")
    notes: list[str] = []
    if "x-content-type-options" in headers and xcto.lower() != "nosniff":
        notes.append(f"x-content-type-options is {xcto!r}, expected 'nosniff'")

    failed = bool(missing_required)
    if strict and missing_recommended:
        failed = True

    return {
        "ok": not failed,
        "missing_required": missing_required,
        "missing_recommended": missing_recommended,
        "notes": notes,
    }
```

Required vs recommended is deliberate. Plenty of production sites ship HSTS + CSP + `nosniff` and express framing inside CSP rather than as `X-Frame-Options`. Default mode fails the build only when a required header is missing. `--strict` also fails when `Referrer-Policy` or any frame protection is absent.

## Step 3: fetch the live headers

```python
import urllib.error
import urllib.request

USER_AGENT = "security-header-scan/1.0 (+https://infinityme.dev)"

def fetch_headers(url: str, timeout: float = 15.0) -> dict[str, str]:
    req = urllib.request.Request(url, method="GET", headers={"User-Agent": USER_AGENT})
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            return normalize_headers(resp.headers)
    except urllib.error.HTTPError as exc:
        # 4xx/5xx still have headers worth scanning
        if exc.headers is not None:
            return normalize_headers(exc.headers)
        raise
```

Use GET rather than HEAD. Some stacks (and some CDNs) omit security headers on HEAD but send them on GET, which is what a browser issues.

The CLI rejects anything that does not start with `https://` (exit code 2). Scanning `http://` would just tell you the downgrade already happened.

## Step 4: a CLI that CI can run

```python
import argparse
import json
import sys

def main(argv: list[str] | None = None) -> int:
    p = argparse.ArgumentParser(description="Scan HTTP security headers")
    p.add_argument("url")
    p.add_argument("--strict", action="store_true")
    p.add_argument("--json", action="store_true", dest="as_json")
    p.add_argument("--timeout", type=float, default=15.0)
    args = p.parse_args(argv)

    if not args.url.startswith("https://"):
        print("URL must start with https://", file=sys.stderr)
        return 2

    headers = fetch_headers(args.url, timeout=args.timeout)
    result = evaluate(headers, strict=args.strict)
    result["url"] = args.url

    if args.as_json:
        print(json.dumps(result, indent=2))
    else:
        print(f"URL  {args.url}")
        print(f"OK   {result['ok']}")
        if result["missing_required"]:
            print("Missing required:     " + ", ".join(result["missing_required"]))
        if result["missing_recommended"]:
            print("Missing recommended:  " + ", ".join(result["missing_recommended"]))
        for note in result["notes"]:
            print(f"Note: {note}")
    return 0 if result["ok"] else 1

if __name__ == "__main__":
    raise SystemExit(main())
```

The full file also prints a short preview of the headers it did find, which helps when a CSP is present but you want to see the first 90 characters.

## Tested on this box (2026-11-08)

```bash
$ python3 test_scan_security_headers.py
Ran 10 tests in 0.008s
OK

$ python3 scan_security_headers.py https://github.com/
URL  https://github.com/
OK   True
Present:
  content-security-policy: default-src 'none'; base-uri 'self'; ...
  referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
  strict-transport-security: max-age=31536000; includeSubdomains; preload
  x-content-type-options: nosniff
  x-frame-options: deny

$ python3 scan_security_headers.py https://example.com/
URL  https://example.com/
OK   False
Missing required:     strict-transport-security, content-security-policy, x-content-type-options
Missing recommended:  referrer-policy, x-frame-options|csp-frame-ancestors
# exit code 1
```

Unit coverage includes: all required headers present (ok), missing HSTS (fail), CSP `frame-ancestors` counting as clickjacking protection, missing frame protection staying recommended-only in default mode, `--strict` failing on recommended gaps, a note when `X-Content-Type-Options` is present but not `nosniff`, header name normalization, rejecting `http://` with exit 2, and CLI exit codes 0 / 1 against mocked responses.

## Drop it into GitHub Actions

```yaml
# .github/workflows/security-headers.yml
name: security-headers
on:
  push:
  pull_request:
  schedule:
    - cron: "17 6 * * 1"
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
      - name: Scan production homepage
        run: python3 scan_security_headers.py https://example.com/
```

Replace `https://example.com/` with your production URL. The workflow fails when required headers disappear after a CDN or nginx change — which is the usual way they vanish, not a code bug in the app itself.

If you want the louder check (also fail without `Referrer-Policy` or frame protection):

```bash
python3 scan_security_headers.py https://example.com/ --strict
```

## Practical tips

- Set HSTS only after you are sure every subdomain you care about speaks HTTPS. `includeSubDomains` and `preload` are powerful and painful to undo.
- A CSP of `default-src 'self'` is a starting point, not a finished policy. Add the CDNs and APIs you actually use, then tighten. `content-security-policy-report-only` is useful while you learn what would have broken.
- Prefer `frame-ancestors 'none'` (or your real embed allowlist) inside CSP over relying on `X-Frame-Options` alone.
- Scan the URL users actually hit (`https://www.example.com/`), not only the apex, and scan after redirects. Headers on the 301 are not the headers on the final page.
- This tool does not parse CSP thoroughly (it only looks for the `frame-ancestors` token) and it does not check cookie flags. Treat it as a missing-header alarm, not a full security audit.

## Wrap-up

You now have a ~120-line stdlib scanner that fails CI when HSTS, CSP, or `X-Content-Type-Options` go missing. Run it against production after deploys and on a weekly cron so a CDN toggle cannot silently drop the seatbelts.

If you are tightening account security on the same checklist, I also wrote a plain-English walkthrough of what to do after an email shows up in a breach: [Email leaked in a data breach — what to do](https://infinityme.dev/blog/email-leaked-data-breach-what-to-do/).