Skip to content

logtrim: keep logs small without restarting anything

The problem

Background services write logs nobody trims. One grows to tens of megabytes in a few weeks and buries the line you need. The usual fix, rename the log and start a new one, needs the program to reopen its file: a restart, or a signal it may not support. Launch managers such as macOS launchd open the log for the program and hold it for its whole life.

The method

Do not move the file. Copy it out, then empty it.

The writer still holds the same open file. In append mode, which >> redirection and most log setups use, every write goes to the current end of the file, now byte 0. The log shrinks to nothing and carries on.

Renaming fails for the same reason: an open file follows its inode, not its name. Rename app.log and the program keeps writing into the renamed file while the fresh app.log stays empty.

Three details make it safe unattended:

  • Empty only after the gzip is whole. Compress to a temp name, rename, then empty. A full disk leaves the log as it was.
  • Never overwrite a part. Stamps carry seconds; a clash gets a suffix.
  • Prune by count. Keep the newest few parts; delete or move the rest.

Sample

find "$dir" -type f -name '*.log' -size +5120k | while IFS= read -r f; do
  part="$f.$(date +%Y%m%d-%H%M%S).gz"
  gzip -9c "$f" > "$part.tmp" && mv "$part.tmp" "$part" && : > "$f"
done

Use it:

logtrim -n /var/log/myapp          # what it would trim
LOGTRIM_MB=10 logtrim /var/log/myapp

Limits

  • Lines written between the copy and the empty are lost: the time gzip takes. Every copy-and-truncate rotation has this cost.
  • The writer must append. One that keeps its own position writes at the old offset after the empty, and the file is back to its old size.
  • Size is checked only when it runs; run it often enough for the log's pace.

Code

methods/logtrim/ in the methods repository: the script, a README and a test that proves the trick with a live writer and shows renaming failing.