Skip to content

errorpage: downloads that are really error pages

The problem

A failed download usually still writes a file, under the name you asked for. curl -o key.gpg URL without -f saves the 404 page as key.gpg. A folder of sources fills with paper.pdf files that are bot-check pages. A storage bucket answers AccessDenied in XML, saved as data.tar.gz.

curl -f does not catch all of it: a single-page web app answers every path it does not know with its own home page and status 200. Fetch /images/cover.png from one and you get HTML, successfully. In one real folder of papers, 1,023 "cover images" were a site's home page, and nine "PDF sources" were 404s, an access-denied page and three different bot checks.

Nothing complains until something far away fails: a package manager that quietly stops updating because its keyring is HTML, an image that never shows, a citation nobody can open.

The method

Every binary format starts with a fixed signature, and many end with one.

  • Judge by the extension's promise. A .png must start 89 50 4E 47; a binary keyring must start with an OpenPGP key packet (or GnuPG's KBXf keybox or trust database); a compressed disk image ends with a koly trailer, a raw one holds a partition map or volume header. 57 extensions.
  • Read only the ends: the first 64 KiB and last 66 KiB, enough for every signature and for a zip's end record. A 300 MB file costs two small reads.
  • When the promise fails, say what it is instead: an HTML page with its <title> ("404 Not Found", "Just a moment...", the app's own name), a storage XML error with its <Code>, a JSON error with its message, plain text, empty, another real format, or unrecognised bytes.
  • Whole or cut off: zip, PDF, PNG, JPEG and GIF have fixed endings; a file that starts right with no end marker is a download that stopped.
  • Office lock files (~$report.docx) are named as such and their contents, which say who opened the document, are never printed. Dependency folders are skipped unless -a: libraries ship broken files on purpose as test fixtures.

Sample

want, test = FORMATS[ext]                 # ".gpg" -> OpenPGP keyring
h = f.read(64 * 1024)                     # the start
f.seek(max(size - 66 * 1024, 0))
t = f.read()                              # the end
if test(h, t, size):
    return "cut off" if want in ENDS and not ENDS[want](t) else None
kind, detail = what_text(h)               # ("HTML page", "404 Not Found")

Use it:

errorpage ~/Downloads sources/       # every file not what its name says
errorpage --json vendor/ | jq .      # one object per finding
curl -fsSL -o key.gpg URL && errorpage key.gpg

Output:

sources/fischer2011.pdf
    named PDF, is HTML page (8,213 bytes)  Checking your browser - reCAPTCHA
backups/data.tar.gz
    named gzip, is XML error (329 bytes)  NoSuchBucket: The specified bucket does not exist

Exit 0 everything is what its name says, 1 something is not, 2 a path could not be read.

Limits

  • Text formats (.json, .csv, .html) are not judged: an error message is valid text.
  • A good start and end do not prove a good middle; use the format's own tool (unzip -t, gpg --show-keys) when it matters.
  • Formats with no fixed ending (gzip, tar, MP4) are not checked for being cut off.

Code

methods/errorpage/ in the methods repository: one Python file (standard library only), a README and a 37-check test that builds every kind of failure, and proves a 300 MB file is judged from at most 200 KiB. Tried on real failed downloads: a GitHub 404 saved as a zip, a missing S3 bucket saved as .tar.gz, an API 404 saved as a keyring.