Store Optimization and Deduplication

Updated

September 12, 2026

Store Optimization and Deduplication

Two packages often contain identical files: licenses, headers, man pages, .pyc with the same hash. Without help, each copy occupies its own disk blocks. The boring default is: nix-store --optimise (or nix.optimise.automatic) hard-links identical files to one inode, and you never do this with cp or a dedup FUSE.

Mental model

Optimisation walks /nix/store, hashes file contents, and when two files match:

  • They keep two directory entries (two paths).
  • They share one inode (one set of blocks).
  • They remain read-only. A later rebuild never writes through a hard link; it creates a new path.
/nix/store/pkgA/share/COPYING   ──┐
                                  ├── inode 1001  (one copy on disk)
/nix/store/pkgB/share/COPYING   ──┘

This is not content-addressed store paths. Paths stay input-addressed and distinct. Only file contents are shared. It is also not compression. du after --optimise drops; nix path-info -S (logical size) may look similar because it counts each path separately.

Nix 2.35 lazy flake copy is eval performance. It is not this. Do not mix them in an incident doc.

Worked examples

Case 1: Run it once and read the report

sudo nix-store --optimise

Output:

hashing files in /nix/store...
31420 files linked, 850.40 MiB freed

On a fresh VM the number may be small. On a desk that has compiled three gcc versions it can be gigabytes. Run it a second time immediately: “files linked” near zero. Idempotent.

Case 2: Automatic on NixOS

Save as optimise.nix (import from configuration.nix):

# optimise.nix
{
  nix.optimise.automatic = true;
  nix.optimise.dates = [ "03:45" ];
  nix.settings.auto-optimise-store = true;
}
Option When it runs
nix.settings.auto-optimise-store After each build, newly added files are linked
nix.optimise.automatic A systemd timer walks the whole store

Turn both on for a workstation. auto-optimise-store catches new builds; the timer catches paths that arrived via substitution.

sudo nixos-rebuild switch
systemctl status nix-optimise.timer
systemctl list-timers | grep nix

On a foreign Linux laptop (Nix 2.35, no NixOS), the same knobs live in the daemon config:

# /etc/nix/nix.conf  (or nix.settings via the installer)
auto-optimise-store = true

User ~/.config/nix/nix.conf does not run --optimise for daemon builds. Edit the daemon file, then sudo systemctl restart nix-daemon.

nix show-config | grep auto-optimise

true means new builds are linked as they land. The weekly timer still walks the whole store for paths that arrived via substitution before the flag was on.

Case 3: Prove two paths share an inode

nix build nixpkgs#hello nixpkgs#cowsay
find /nix/store -name COPYING -type f 2>/dev/null | head -5

If you have at least two, compare inodes:

stat -c '%i %n' /nix/store/<hash-a>-*/COPYING /nix/store/<hash-b>-*/COPYING

After --optimise, the %i values match. ls -l shows a link count n greater than 1.

ls -l /nix/store/<hash-a>-hello-*/share/info/hello.info | awk '{print $2, $NF}'

Link count > 1 means another path shares the inode. That is the disk win. nix path-info -S still reports the logical size.

Case 4: Logical size versus disk usage

nix path-info -Sh nixpkgs#hello
nix path-info -Shr nixpkgs#hello
df -h /nix

path-info -S reports the size of that path’s files as if they were unique. df reports actual blocks after hard links. Both numbers are useful; they are not the same question. When finance asks “how much does hello weigh,” use nix path-info -Shr (closure, logical). When the disk is full, use df and GC.

Case 5: Safe with GC

Optimise, then GC, in either order. Hard links do not keep a path alive. GC deletes directory trees; the inode goes away when the last link is removed.

sudo nix-store --optimise
sudo nix-collect-garbage --delete-older-than 14d
df -h /nix

That pair is the boring monthly (or the automatic timers). Optimise never deletes a path. If a file vanished, GC or a failed disk did it.

Hard links only work on one filesystem. If /nix/store is a bind of two disks, --optimise cannot share inodes across them. Keep /nix on one partition (or one Btrfs/ZFS dataset). Compression (compress=zstd) on that dataset is complementary; it is not a substitute for Nix’s linker.

The trap

The trap is a third-party deduplication tool (duperemove, a Btrfs dedup cron, zfs dedup=on “because Nix”) on top of Nix’s own hard links. Filesystem-level dedup plus Nix hard links is extra CPU for little gain, and some tools copy-on-write in ways that fight the read-only store.

Btrfs and ZFS compression (compress=zstd) is fine. That is not dedup. Leave Nix to hard-link identical files.

findmnt /nix || findmnt /

Note the filesystem type. dedup=on on ZFS plus nix-store --optimise is the double-tax. Compression alone is enough beside Nix hard links.

The boring rule

  • nix-store --optimise or the NixOS timer. Not a random dedup cron.
  • Enable auto-optimise-store so new builds are linked as they land.
  • Measure disk with df; measure closures with nix path-info -Shr.
  • Compression (Btrfs/ZFS) is complementary. Filesystem dedup is not needed.
  • Optimise does not replace GC. Run both.

Try this

  1. Note df -h /nix (or /), run sudo nix-store --optimise, note df again. Record the delta.
  2. On NixOS, add the snippet from Case 2, switch, and systemctl list-timers | grep nix.
  3. stat two identical COPYING files and confirm the inode number. If you only have one, nix build nixpkgs#hello nixpkgs#cowsay first and look again.
  4. Run nix-store --optimise a second time immediately. The “files linked” count should be near zero. That is how you know it is idempotent.