Skip to content

atrisk: which files would be gone if this disk died

The problem

Work spread over several disks, backups and copies: some of it exists three times, some once, and nobody knows which. Duplicate finders answer "what can I delete?"; backup tools only know what they copied. Before a drive fails, or before wiping one, the question is: what here has no copy anywhere else?

Hashing every file to find out reads every byte on every disk.

The method

Two files can only be identical if they are the same size. A file whose size no other file shares cannot have a copy, so it is decided without opening it. Only size collisions are read: first and last 64 KiB, then the whole file if those still match.

What counts as "another place":

  • A hard link is not a copy. Files are grouped by inode first; two names for one inode die together.
  • --against BACKUP: only a copy inside the backup counts. A hard link from the backup into the source is not a backup.
  • --other-disk: only a copy on a different device counts.

Sample

by_size = {}
for u in units.values():                  # one entry per inode, not per name
    by_size.setdefault(u.size, []).append(u)
for group in by_size.values():
    for t in group:
        if not any(other_place(t, p) for p in group):
            at_risk.append(t)             # no file of this size elsewhere: never read
# the rest: first and last 64 KiB, then a full SHA-256 only where those match

Use it:

atrisk /data/photos /mnt/old-drive            # content held once across both
atrisk /data/work --against /mnt/backup/work  # in work, missing from the backup
atrisk /data/work --group 2                   # totals per folder

Exit 0 nothing at risk, 1 something at risk, 2 bad arguments or a file that had to be read and could not be.

Limits

  • Only the places it is shown count: cloud copies and unmounted drives do not.
  • A copy is byte-for-byte: a re-saved or resized photo is a different file.
  • A file changing during the run may be judged on a mix of old and new bytes.

Code

methods/atrisk/ in the methods repository: the script, a README and a 33-check test, including an unreadable file of unique size that is still judged correctly, which proves it was never opened. Tried on a real 1.2 GB folder of about 7,200 files: about half decided by size alone, 1.4 seconds.