Skip to content

derive: convert once, keep it beside the source

The problem

Scripts, search tools and AI agents keep turning the same PDFs, scans and Word files into text. Each does it again in its own temp folder and throws the result away. OCR on a 40-page scan takes minutes, and every tool pays it.

The method

Put the result where anyone looking at the file will find it: a hidden .derived/ folder in the same directory, named after the source.

letter.pdf
.derived/letter.pdf.txt

Freshness is one comparison. If the result is newer than the source, use it. If the source changed since, convert again. No database, no index, no cache server, and the cache moves, copies and backs up with its folder.

Three details make it dependable:

  • It prints the path, not the text, so callers chain it.
  • It writes to a temp file and renames at the end. A conversion cut short never leaves a half-written result that looks fresh next time.
  • Scanned PDFs fall through to OCR. A PDF with no fonts is pictures of pages, so each page is rendered and OCR'd, marked "OCR, unverified". Counting fonts, not characters, keeps a one-line real PDF as real text.

Sample

The heart of it, in POSIX sh:

out="$dir/.derived/$name.txt"
if [ -s "$out" ] && [ "$out" -nt "$src" ]; then echo "$out"; exit 0; fi

tmp="$out.tmp.$$"
case "$ext" in
  pdf)           pdftotext -layout "$src" "$tmp" ;;
  docx|md|html)  pandoc "$src" -t plain -o "$tmp" ;;
  png|jpg)       tesseract "$src" - > "$tmp" ;;
esac
mv "$tmp" "$out"
echo "$out"

Use it:

grep -i invoice "$(derive scan.pdf)"
derive report.pdf pages      # one PNG per page

Limits

  • Freshness is by modification time. A tool that rewrites a file but keeps its old time stamp fools it.
  • Two processes converting the same file at once both do the work; the last rename wins, and the result is still whole.
  • A PDF mixing typed and scanned pages keeps only its typed text.
  • OCR is a guess. Handwriting still needs reading by eye.
  • Add .derived/ to .gitignore in repositories.

Code

methods/derive/ in the methods repository: the script, a README and a test that needs no converters installed.