Skip to content

harvest: all the text in a folder, each thing said once

The problem

Join every file in an email export or a folder of notes and most of the result is repeats. Each reply quotes the one before it, so a 20-message thread holds the first message 20 times. Every email carries the same signature. The same attachment was saved under three names. The new words are a small part of the whole and hard to find, for a person or for an AI with a limited context.

The method

Drop repeats at three sizes, coarse to fine, keeping the first time each thing appears:

  1. Whole files whose text matches one already kept.
  2. Paragraphs (split on blank lines) already kept from any earlier file: quoted messages, letterheads, boilerplate.
  3. Long lines (40+ characters) already kept, even inside a new paragraph. Quotes come back re-wrapped or mixed with new text, so paragraphs alone miss them. Short lines such as dates or "Yes" stay: they repeat because they are short, not because they were copied.

Everything is compared on a normalised key: quote marks, punctuation, spacing and case stripped, then hashed. A quote two replies deep still matches the original. Only hashes are held, so memory stays small.

Files are read in path order, so the first message in a thread keeps its words and each reply keeps only what it added.

Sample

def key(s):
    s = re.sub(r"^[>\s]+", "", s, flags=re.M)        # email quote marks
    s = re.sub(r"\W+", " ", s).strip().lower()
    return hashlib.sha1(s.encode()).hexdigest()

for para in re.split(r"\n\s*\n", text):
    if key(para) in seen_paras:
        continue                                     # said before
    seen_paras.add(key(para))
    for line in para.splitlines():
        if len(line.strip()) >= 40 and key(line) in seen_lines:
            continue                                 # quoted line, re-wrapped
        seen_lines.add(key(line))
        keep(line)

Use it:

harvest mail/ notes/ -o everything.txt
harvest docs/ --convert derive     # PDFs and scans through a converter

Limits

  • First one wins: a better later copy of a paragraph still loses it.
  • One changed word makes a new paragraph; an inline-edited quote is kept.
  • A long line repeated on purpose (a refrain, a heading) is cut the second time.

Code

methods/harvest/ in the methods repository: the script (Python standard library), a README and a test built on a quoted three-email thread.