Skip to content

redact: secrets out of logs, the rest left readable

The problem

Logs, terminal transcripts and agent histories catch secrets: a token pasted into a command, an Authorization: Bearer header, a .env printed while debugging. A scanner finds them after they are written, synced and backed up. The better place to stop them is the write: tool | redact >> log.

The usual filter is a list of regexes, and it fails both ways. Too loose, and every git commit id, sha256 sum and long identifier on its own line is blanked, so the log is useless. Too tight, and PASSWORD=hunter2 passes because it is "too short to be a secret".

The method

  • Shapes for tokens with a published prefix (ghp_, github_pat_, glpat-, xoxb-, AKIA, AIza, sk_live_, hf_, npm_, pypi-, sk- with a digit). A JWT matches only when its first two parts start eyJ: header and payload are both base64 JSON objects.
  • Context for everything else: the value after a name that ends in a telling word (*_TOKEN=, "client_secret":, --password VALUE), after Authorization: Bearer, in scheme://user:pass@host, after curl -u. tokenizer=, token_file= and author= are left alone. A name merely ending in key needs a long value with a digit and a letter.
  • Not a secret: a value that names one is kept: ALL_CAPS, $VAR, ${VAR}, <your-token>, {{template}}, ***, already redacted. Bare hex is never matched alone, since a commit id and a random key look alike; only 0x plus exactly 64 digits (a private key) goes.
  • Private-key blocks are tracked across lines from BEGIN ... PRIVATE KEY to END; the same block logged on one line with \n escapes is cut whole.
$ printf 'GITHUB_TOKEN=ghp_…\ncommit 3cb44de0a1b2c3d4e5f60718293a4b5c6d7e8f90\nkey with x=API_TOKEN\n' | redact
GITHUB_TOKEN=[REDACTED:github]
commit 3cb44de0a1b2c3d4e5f60718293a4b5c6d7e8f90
key with x=API_TOKEN

redact --check notes/*.md     # file:line:kind, never the secret; exit 1 if any
redact --fix old.log          # in place: temp beside, mode kept, atomic rename
redact --rules house.tsv      # your own shapes: KIND<TAB>REGEX

Idempotent, flushed per line (works behind tail -f), and bytes that are not UTF-8 pass through unchanged.

Limits

  • A secret with no prefix and no telling name (mysql -psecret, a bare hex key) passes. Pair it with a scanner such as gitleaks for files already written.
  • One line at a time: a name on one line and its value on the next is missed.
  • An all-caps value with no digits, or one with an underscore, is taken for a variable name.
  • secret: everything in prose loses the word after it; that is the price of catching YAML.

Code

methods/redact/ in the methods repository: one Python file (standard library), a README and a 59-check test whose fake secrets are built at run time; with gitleaks installed it also confirms gitleaks finds the raw secrets and nothing after.