Skip to content

unshare: undo hardlink "dedupe" without using more disk

The problem

Duplicate finders save space by hardlinking identical files. A hardlink is one file with two names, so afterwards an edit in a backup quietly edits the live copy, and a chmod on scratch copies changes the real tools. Both names look like separate files in every listing.

Copying each file back apart fixes that but doubles the disk the dedupe saved, and a plain copy drops dates, owner and extended attributes.

The method

Clone instead of copy. APFS (clonefile()), Btrfs and XFS (FICLONE) can make a new file that shares the original's blocks until one side is written. It has its own inode, so edits stay apart, yet uses no extra disk at first.

For each shared name: clone to a temp name in the same folder, check the original did not change meanwhile, rename the clone over the name. Once all but one name are freed, the last keeps the original inode.

  • Count first; --write changes things.
  • Mode, dates, extended attributes kept; the folder's dates put back.
  • Owner never changes: such a file is refused and named.
  • No silent full copies: without clone support a file is refused unless --copy allows a copy, compared byte for byte before it replaces anything.

Sample

before = os.lstat(path)                                  # st_nlink >= 2: shared
tmp = temp_beside(path)            # first free .unshare-PID-N: a leftover never blocks
libc.clonefile(os.fsencode(path), os.fsencode(tmp), 1)   # 1 = no follow
now = os.lstat(path)
if (now.st_ino, now.st_size, now.st_mtime_ns) == \
   (before.st_ino, before.st_size, before.st_mtime_ns):  # unchanged while cloning
    os.replace(tmp, path)                                # name now its own file

Use it:

unshare /srv/backups              # count shared names
unshare --write /srv/backups      # free them; each changed path printed

Limits

  • The count is every shared name; a pair counts 2 and changes 1.
  • A program writing the file during the run keeps writing to the original.
  • Hardlinks meant to be shared (package stores, snapshot backups) are not told apart; point it only where the dedupe was a mistake.

Code

methods/unshare/ in the methods repository: the script, a README and a 27-check test. Tried on APFS (a 300 MB pair cloned with no disk used) and on an HFS+ disk image (no clones: refused, then copied with --copy).