Skip to content

sift: search the words inside your files

The problem

You remember a phrase from a letter or a receipt, not the file. grep -r reads every byte on every search and cannot read PDFs. Desktop search indexes everything its own way, out of reach of scripts. A search server is a daemon to run for one person.

The method

SQLite's built-in FTS5 index keeps every word and the files it appears in, in one file. The work is in making it cheap and safe:

  • Unchanged files cost one stat. Modification time (nanoseconds) and size are kept per file; a rebuild reads only what changed (5,000 real files: 0.2 s).
  • A killed build resumes. A file's text and its "done" mark commit together.
  • Deleted only after a full walk. Each build numbers the files it sees; older rows under a folder go once that folder's walk ends. A missing root (an unplugged disk) or an unreadable folder keeps its rows.
  • The path lives outside the index, in an ordinary table keyed to the index row. FTS5 columns have no lookup index, so WHERE path = ? there scans everything.
  • Build tuning stored in the index: hashsize 256 MB, automerge 16 (20,000 files: 11.6 s instead of 16.2 s).
  • Every typed word is quoted before FTS5 sees it, so AND, NEAR(, * and ( are words, never syntax.
  • Minified scripts by name only: on real folders the index shrank from 331 MB to 110 MB.
CREATE TABLE files(path TEXT PRIMARY KEY, mtime_ns INTEGER, size INTEGER,
                   rid INTEGER, gen INTEGER);   -- rid = row in fts
CREATE VIRTUAL TABLE fts USING fts5(name, content,
                   tokenize='unicode61 remove_diacritics 2');

Sample

sift build ~/notes ~/mail --convert derive   # first time; PDFs through a converter
sift build                                   # later: changed files only
sift tax receipt 2024
sift '"steel fence"' in:receipts after:2024 -draft ext:pdf

Limits

  • Fresh as of the last sift build; nothing watches the disk.
  • A rewrite that keeps size and restores the time is not noticed.
  • Finds words, not meanings; languages without spaces between words search poorly.

Code

methods/sift/ in the methods repository: one Python file (standard library), a README and a 39-check test.