Scan HTTP Security Headers in Python and Fail CI When HSTS or CSP Is Missing

A TLS certificate only proves the browser reached the right host. It does not stop a page from being framed, sniffed as the wrong content type, or loaded on HTTP after a downgrade. Those protections live in response headers: Strict-Transport-Security, Content-Security-Policy, X-Content-Type-Options, and a clickjacking control (X-Frame-Options or CSP frame-ancestors).

This tutorial builds a small Python CLI that fetches a URL, checks those headers, and exits 1 when required ones are missing — so you can drop it into CI after every deploy.

Stdlib only: argparse, json, urllib, email.message. No pip install.

TL;DR: python3 scan_security_headers.py https://github.com/ should print OK True. python3 scan_security_headers.py https://example.com/ exits 1 because the required headers are absent. Add --strict to also fail on missing recommended headers.

What each header is for

• Strict-Transport-Security (HSTS) — tells the browser to use HTTPS for this host for max-age seconds. Without it, a first visit (or a bookmark saved as http://) can still be downgraded.
• Content-Security-Policy (CSP) — restricts where scripts, styles, images, and connections may come from. A good CSP is the main defense against injected JavaScript.
• X-Content-Type-Options: nosniff — stops the browser from guessing a MIME type. Without it, a downloaded file the server called text/plain might be executed as HTML or JavaScript.
• X-Frame-Options or CSP frame-ancestors — controls who may embed the page in an iframe (clickjacking). Modern guidance prefers frame-ancestors inside CSP; X-Frame-Options is the older standalone header. This scanner treats either as enough.
• Referrer-Policy — limits how much of the URL is sent as a Referer when people click outbound links. Recommended here, not required, unless you pass --strict.

None of these replace HTTPS, cookie flags, or a password-hashing scheme. They are the cheap, checkable layer that a lot of small sites forget to turn on.

Step 1: normalize header names

HTTP header names are case-insensitive. Normalize once so the rest of the code can use lowercase keys:

from email.message import Message

def normalize_headers(headers: Message | dict) -> dict[str, str]:
    if isinstance(headers, Message):
        items = headers.items()
    else:
        items = headers.items()
    out: dict[str, str] = {}
    for key, value in items:
        out[key.lower()] = value.strip()
    return out

urllib already gives you an email.message.Message, which behaves like a case-insensitive map, but turning it into a plain dict makes tests and JSON output boring and predictable.

Step 2: decide what "ok" means

REQUIRED = (
    "strict-transport-security",
    "content-security-policy",
    "x-content-type-options",
)
RECOMMENDED = ("referrer-policy",)

def has_frame_protection(headers: dict[str, str]) -> bool:
    if headers.get("x-frame-options"):
        return True
    csp = headers.get("content-security-policy", "")
    return "frame-ancestors" in csp.lower()

def evaluate(headers: dict[str, str], *, strict: bool = False) -> dict:
    missing_required = [name for name in REQUIRED if name not in headers]
    missing_recommended = [name for name in RECOMMENDED if name not in headers]
    if not has_frame_protection(headers):
        missing_recommended.append("x-frame-options|csp-frame-ancestors")

    xcto = headers.get("x-content-type-options", "")
    notes: list[str] = []
    if "x-content-type-options" in headers and xcto.lower() != "nosniff":
        notes.append(f"x-content-type-options is {xcto!r}, expected 'nosniff'")

    failed = bool(missing_required)
    if strict and missing_recommended:
        failed = True

    return {
        "ok": not failed,
        "missing_required": missing_required,
        "missing_recommended": missing_recommended,
        "notes": notes,
    }

Required vs recommended is deliberate. Plenty of production sites ship HSTS + CSP + nosniff and express framing inside CSP rather than as X-Frame-Options. Default mode fails the build only when a required header is missing. --strict also fails when Referrer-Policy or any frame protection is absent.

Step 3: fetch the live headers

import urllib.error
import urllib.request

USER_AGENT = "security-header-scan/1.0 (+https://infinityme.dev)"

def fetch_headers(url: str, timeout: float = 15.0) -> dict[str, str]:
    req = urllib.request.Request(url, method="GET", headers={"User-Agent": USER_AGENT})
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            return normalize_headers(resp.headers)
    except urllib.error.HTTPError as exc:
        # 4xx/5xx still have headers worth scanning
        if exc.headers is not None:
            return normalize_headers(exc.headers)
        raise

Use GET rather than HEAD. Some stacks (and some CDNs) omit security headers on HEAD but send them on GET, which is what a browser issues.

The CLI rejects anything that does not start with https:// (exit code 2). Scanning http:// would just tell you the downgrade already happened.

Step 4: a CLI that CI can run

import argparse
import json
import sys

def main(argv: list[str] | None = None) -> int:
    p = argparse.ArgumentParser(description="Scan HTTP security headers")
    p.add_argument("url")
    p.add_argument("--strict", action="store_true")
    p.add_argument("--json", action="store_true", dest="as_json")
    p.add_argument("--timeout", type=float, default=15.0)
    args = p.parse_args(argv)

    if not args.url.startswith("https://"):
        print("URL must start with https://", file=sys.stderr)
        return 2

    headers = fetch_headers(args.url, timeout=args.timeout)
    result = evaluate(headers, strict=args.strict)
    result["url"] = args.url

    if args.as_json:
        print(json.dumps(result, indent=2))
    else:
        print(f"URL  {args.url}")
        print(f"OK   {result['ok']}")
        if result["missing_required"]:
            print("Missing required:     " + ", ".join(result["missing_required"]))
        if result["missing_recommended"]:
            print("Missing recommended:  " + ", ".join(result["missing_recommended"]))
        for note in result["notes"]:
            print(f"Note: {note}")
    return 0 if result["ok"] else 1

if __name__ == "__main__":
    raise SystemExit(main())

The full file also prints a short preview of the headers it did find, which helps when a CSP is present but you want to see the first 90 characters.

Tested on this box (2026-11-08)

$ python3 test_scan_security_headers.py
Ran 10 tests in 0.008s
OK

$ python3 scan_security_headers.py https://github.com/
URL  https://github.com/
OK   True
Present:
  content-security-policy: default-src 'none'; base-uri 'self'; ...
  referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
  strict-transport-security: max-age=31536000; includeSubdomains; preload
  x-content-type-options: nosniff
  x-frame-options: deny

$ python3 scan_security_headers.py https://example.com/
URL  https://example.com/
OK   False
Missing required:     strict-transport-security, content-security-policy, x-content-type-options
Missing recommended:  referrer-policy, x-frame-options|csp-frame-ancestors
# exit code 1

Unit coverage includes: all required headers present (ok), missing HSTS (fail), CSP frame-ancestors counting as clickjacking protection, missing frame protection staying recommended-only in default mode, --strict failing on recommended gaps, a note when X-Content-Type-Options is present but not nosniff, header name normalization, rejecting http:// with exit 2, and CLI exit codes 0 / 1 against mocked responses.

Drop it into GitHub Actions

# .github/workflows/security-headers.yml
name: security-headers
on:
  push:
  pull_request:
  schedule:
    - cron: "17 6 * * 1"
jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.13"
      - name: Scan production homepage
        run: python3 scan_security_headers.py https://example.com/

Replace https://example.com/ with your production URL. The workflow fails when required headers disappear after a CDN or nginx change — which is the usual way they vanish, not a code bug in the app itself.

If you want the louder check (also fail without Referrer-Policy or frame protection):

python3 scan_security_headers.py https://example.com/ --strict

Practical tips

• Set HSTS only after you are sure every subdomain you care about speaks HTTPS. includeSubDomains and preload are powerful and painful to undo.
• A CSP of default-src 'self' is a starting point, not a finished policy. Add the CDNs and APIs you actually use, then tighten. content-security-policy-report-only is useful while you learn what would have broken.
• Prefer frame-ancestors 'none' (or your real embed allowlist) inside CSP over relying on X-Frame-Options alone.
• Scan the URL users actually hit (https://www.example.com/), not only the apex, and scan after redirects. Headers on the 301 are not the headers on the final page.
• This tool does not parse CSP thoroughly (it only looks for the frame-ancestors token) and it does not check cookie flags. Treat it as a missing-header alarm, not a full security audit.

Wrap-up

You now have a ~120-line stdlib scanner that fails CI when HSTS, CSP, or X-Content-Type-Options go missing. Run it against production after deploys and on a weekly cron so a CDN toggle cannot silently drop the seatbelts.

If you are tightening account security on the same checklist, I also wrote a plain-English walkthrough of what to do after an email shows up in a breach: Email leaked in a data breach — what to do.