Day 17 — Python/Rust Packaging, Patching & Outputs

Updated

July 30, 2025

Day 17 — Python/Rust Packaging, Patching & Outputs

Day 48 — Language ecosystem B

Stage V · ~4h
Goal: Package a second language ecosystem tool so the patterns transfer: FODs for language deps, helpers vs raw stdenv, and how overrides differ across builders.

Why this day exists

At work you will not stay monoglot. The point is not fluency in five builders—it is recognizing the same skeleton:

src FOD → language-dep FOD → build/install hooks → $out/bin → meta

Day 47 was depth. Today is transfer with a deliberately smaller scope.

Theory 1 — Pattern map across ecosystems

Concern Go Rust Python Node
Source pin src.hash src.hash src.hash / fetchPypi src.hash
Dep pin vendorHash cargoHash / lock package set / frozen npmDepsHash
Builder buildGoModule buildRustPackage buildPython* buildNpmPackage
Skip tests doCheck doCheck / cargo flags pytest hooks npm build only
Binary name module main crate bin meta.mainProgram meta.mainProgram
Cross GOOS/GOARCH care target triples often pure often pure JS

When stuck, ask: which hash is wrong—source or language deps?

Theory 2 — Choose B deliberately

If Day 47 was… Good Day 48 pick Why
Go Python CLI or Rust Different dep story
Rust Go small tool Faster vendor cycle
Python Node or Go Leaves interpreter wrapping
Node Python or Go Sees compiled vs interpreted

Avoid starting a second multi-crate workspace today. One binary, clear version, known license.

Theory 3 — Wrapping and runtime deps

Interpreted ecosystems often need wrappers:

# Python apps usually wrap PYTHONPATH via the builder
# Manual wrap example for a shell/python hybrid:
{ stdenv, lib, python3, makeWrapper }:
stdenv.mkDerivation {
  pname = "tool";
  version = "0.1.0";
  nativeBuildInputs = [ makeWrapper ];
  # …
  postInstall = ''
    wrapProgram $out/bin/tool \
      --prefix PATH : ${lib.makeBinPath [ python3 ]}
  '';
}
Situation Prefer
Official language builder exists Use it
Need one env var or PATH sibling wrapProgram / makeWrapper
Need whole tree of modules Builder’s dependency graph

Theory 4 — Overrides preview (deep dive Day 49)

# Change flags without forking the whole expression
helloGo.overrideAttrs (old: {
  ldflags = (old.ldflags or []) ++ [ "-X main.extra=day48" ];
})
# Python package override inside python3.pkgs
python3.pkgs.myPkg.overridePythonAttrs (old: {
  doCheck = false;
})

Today: know these exist. Tomorrow: patch and overlay composition for real.

Worked example — dual packages in one flake

{
  description = "Day 48 — second ecosystem";
  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-26.05";

  outputs = { self, nixpkgs }:
    let
      systems = [ "x86_64-linux" "aarch64-linux" ];
      forAllSystems = nixpkgs.lib.genAttrs systems;
    in {
      packages = forAllSystems (system:
        let pkgs = import nixpkgs { inherit system; }; in {
          lang-a = pkgs.callPackage ./pkgs/lang-a.nix { }; # from day47, copy or input
          lang-b = pkgs.callPackage ./pkgs/lang-b.nix { };
          default = self.packages.${system}.lang-b;
        });

      checks = forAllSystems (system:
        let
          pkgs = import nixpkgs { inherit system; };
          a = self.packages.${system}.lang-a;
          b = self.packages.${system}.lang-b;
        in {
          both-run = pkgs.runCommand "both-run" {
            nativeBuildInputs = [ a b ];
          } ''
            # invoke both CLIs smoke-test style
            ${a}/bin/* --help >/dev/null 2>&1 || true
            ${b}/bin/* --help >/dev/null 2>&1 || true
            touch $out
          '';
        });
    };
}

Example B — tiny Python app (if A was compiled)

{ lib, python3Packages }:
python3Packages.buildPythonApplication {
  pname = "day48-greet";
  version = "0.1.0";
  pyproject = true;
  src = ../src/day48-greet;
  build-system = with python3Packages; [ setuptools ];
  meta = {
    description = "Day 48 greet";
    license = lib.licenses.mit;
    mainProgram = "day48-greet";
  };
}
# src/day48-greet/pyproject.toml
[project]
name = "day48-greet"
version = "0.1.0"
requires-python = ">=3.11"
[project.scripts]
day48-greet = "day48_greet:main"
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"
# src/day48-greet/day48_greet.py
def main():
    print("day48 ecosystem B")
if __name__ == "__main__":
    main()

Example B — tiny Node CLI (alternative)

{ lib, buildNpmPackage }:
buildNpmPackage {
  pname = "day48-node";
  version = "0.1.0";
  src = ../src/day48-node;
  npmDepsHash = "sha256-…";
  meta = {
    mainProgram = "day48-node";
    license = lib.licenses.mit;
  };
}

Lab — multi-step

Suggested workspace: ~/lab/90daysofx/02-nixos/day48

Step 1 — Import or copy Day 47 package

Keep lang-a buildable so you compare logs side by side.

Step 2 — Scaffold smaller B project

Budget: under ~50 lines of application code. No microservices.

Step 3 — Package with the B builder

nix build .#lang-b -L
nix run .#lang-b

Record every hash you had to set (src + language deps).

Step 4 — Comparison table (required)

In NOTES.md:

Question Lang A Lang B
Builder name
Source hash field
Dep hash field
Time to first success
Hardest error
Where binary lands
Would I use this at work for CLIs?

Step 5 — Unified check

Add a flake checks.* that runs both tools (smoke). nix flake check.

Step 6 — Optional: same interface

Make both print a comparable line (day47 / day48) so a shell script can assert either.

Step 7 — Closure glance

nix path-info -rS ./result | sort -k2 -n | tail
nix path-info -Sh ./result

Note which ecosystem pulled a heavier closure and why (interpreter? compiler runtime? libc?).

Theory 5 — Cross-ecosystem FOD discipline

Whatever language B you chose, fetchers obey the same rules as Day 46:

Rule Why
Pin URL + hash Reproducible
Vendor offline where possible CI without network flakiness
Prefer language-native lock → Nix conversion tools carefully Don’t invent hashes by hand when tooling exists 
# conceptual — language-specific helpers differ
# buildGoModule / buildPythonPackage / rustPlatform.buildRustPackage
# each has its own cargoHash / vendorHash / etc.

Document the hash field name for your ecosystem B in notes—future-you will grep for it.

Theory 6 — Runtime vs build-time closure

nativeBuildInputs  → tools for build (compilers, make)
buildInputs        → linked/used at build; may propagate
propagated*        → infect dependents (use sparingly)

A package that “works in nix develop” but fails as nix build often mixed shell conveniences with missing buildInputs.

Worked example — dual-output mental model

Language ecosystems often ship:

  • CLI binary
  • library crate/wheel/module

On Nix, that may be one derivation or split outputs (Day 50). For today: ensure meta.mainProgram / bin path is clear when wrapping.

Lab — hash bump drill

  1. Intentionally change a src rev or leave hash empty if your Nix warns
  2. Run build; capture the got: hash from the error
  3. Paste correct hash; rebuild green
  4. Write the exact error class name in notes

Lab — compare A vs B one-pager

Table in notes:

Dimension Eco A (Day 47) Eco B (today)
Builder helper
Lockfile
Hash field
Common footgun

Common gotchas

Symptom / mistake What to do
Trying to “finish” a second huge app Shrink scope; transfer patterns, not product
Mixing package sets (python311 vs python312) Align interpreters
Assuming vendorHash = null always works Only pure module-free trees
npmDepsHash churn from lock format Commit lock; pin npm
Comparing wall times unfairly Same machine, warm vs cold cache noted
Forgetting meta.license Set it; habit for nixpkgs-shaped work

Checkpoint

  • Second ecosystem package builds and runs
  • Comparison table filled honestly
  • Both A and B reachable from one flake
  • At least one combined check
  • Closure notes captured

Commit

git add .
git commit -m "day48: language ecosystem B + comparison"

Write three personal gotchas before continuing.

Tomorrow

Day 49 — Patching & overrides: overrideAttrs, patches, and overlay composition without forking nixpkgs.

Day 49 — Patching & overrides

Stage V · ~4h (lab-heavy)
Goal: Fix or pin packages with overrideAttrs, patches, and overlays—without maintaining a permanent hard fork of nixpkgs.

Why this day exists

Upstream will be wrong for your window of time: a CVE pin, a broken release, a flag your hardware needs. The professional skill is minimal, reviewable deltas that compose, not copy-pasting half of nixpkgs into your flake.

Theory 1 — Three layers of change

Mechanism Scope Reuse Maintenance
override / overrideAttrs One package instance Call-site Low–medium
Overlay Package set view for a flake/host Shared modules Medium
Fork / fetch your tree Anything High control High—avoid until needed 
nixpkgs  →  overlay(s)  →  pkgs.yourTool
                ↑
         overrideAttrs / patches

Overlays you already touched lightly in Stage III; today they become packaging tools.

Theory 2 — overrideAttrs

pkgs.hello.overrideAttrs (old: {
  patches = (old.patches or []) ++ [ ./patches/hello-msg.patch ];
  postPatch = (old.postPatch or "") + ''
    echo "day49" >> README
  '';
})
Form Notes
overrideAttrs (old: { … }) Preferred; see previous attrs
overrideAttrs { … } OK for total replace of listed keys; easy to drop old patches by mistake
package.override { stdenv = …; } Change callPackage args (dependencies), not phases

override vs overrideAttrs

# Change dependencies / function args
pkgs.ffmpeg.override { withXcb = true; }

# Change derivation attributes / phases
pkgs.ffmpeg.overrideAttrs (old: {
  configureFlags = (old.configureFlags or []) ++ [ "--enable-something" ];
})

Language builders often expose both (e.g. overridePythonAttrs).

Theory 3 — Patches as first-class inputs

stdenv.mkDerivation {
  # …
  patches = [
    ./patches/0001-fix-build-on-musl.patch
    (fetchpatch {
      url = "https://github.com/org/proj/commit/abc123.patch";
      hash = "sha256-…";
    })
  ];
}
Practice Why
Numbered filenames Apply order is list order
Keep patches small Rebase pain scales with hunk size
Prefer upstream PRs Local patches rot on every release
Test without patch once Confirm the patch is still required

patchPhase applies these automatically; fail logs show hunk rejects.

Theory 4 — Overlay composition

# overlays/hello-patch.nix
final: prev: {
  hello = prev.hello.overrideAttrs (old: {
    patches = (old.patches or []) ++ [ ../patches/hello-msg.patch ];
  });
}
# flake.nix fragment
pkgs = import nixpkgs {
  inherit system;
  overlays = [
    (import ./overlays/hello-patch.nix)
    (import ./overlays/my-tool.nix)
  ];
};

Order matters

overlay1 final:prev → overlay2 final:prev → …

Later overlays see earlier results via prev/final fixed-point.
Use final when depending on other packages you also overlay; use prev to modify the incoming package without recursion traps.

final: prev: {
  myApp = prev.myApp.overrideAttrs (_: { /* … */ });
  # depend on overlaid hello:
  myAppWrapped = final.hello;
}

NixOS / HM consumption

nixpkgs.overlays = [
  (import ./overlays/hello-patch.nix)
];
# or per-flake pkgs passed into modules

Theory 5 — Pinning and “don’t rebuild the world”

Goal Technique
One package newer than channel Overlay with fetchFromGitHub src bump + hashes
One package older (pin) Overlay src to known-good rev; note security debt
Only your app sees the change Don’t put overlay in global NixOS; inject via callPackage scope
Temporary CI fix Overlay in flake only; ticket to drop it 
final: prev: {
  curl = prev.curl.overrideAttrs (old: rec {
    version = "8.11.1";
    src = prev.fetchurl {
      url = "https://curl.se/download/curl-${version}.tar.bz2";
      hash = "sha256-…";
    };
  });
}

Bumping shared libraries can rebuild huge closures—check nix why-depends before celebrating.

Worked examples

Example A — local patch on hello

# patches/hello-msg.patch
--- a/src/hello.c
+++ b/src/hello.c
@@ -1,5 +1,5 @@
 /* … */
-  printf ("Hello, world!\n");
+  printf ("Hello, Stage V!\n");
# overlays/hello-stagev.nix
final: prev: {
  hello = prev.hello.overrideAttrs (old: {
    pname = old.pname;
    patches = (old.patches or []) ++ [ ../patches/hello-msg.patch ];
  });
}

Example B — overlay adding your Day 47 package

final: prev: {
  hello-go = prev.callPackage ../pkgs/hello-go.nix { };
}

Example C — packageOverrides legacy note

Older blogs use packageOverrides. Prefer overlays in modern flakes; recognize the old name when reading.

Lab — multi-step

Suggested workspace: ~/lab/90daysofx/02-nixos/day49

Step 1 — Baseline

nix build nixpkgs#hello -o baseline-hello
./baseline-hello/bin/hello

Step 2 — Write a patch

Create a minimal patch (message change or README touch). If patching C is painful on your platform, patch a script package you own from Day 45 instead.

Step 3 — overrideAttrs only (no overlay)

In a default.nix / flake package:

pkgs.hello.overrideAttrs (old: {
  patches = (old.patches or []) ++ [ ./patches/hello-msg.patch ];
})
nix build .#hello-patched -L
./result/bin/hello

Step 4 — Promote to overlay

Move the same change to overlays/…. Import overlay into flake pkgs. Confirm:

nix eval .#packages.x86_64-linux.hello.drvPath
# differs from unpatched when patch applied

Step 5 — Stack two overlays

  1. Overlay patches hello
  2. Overlay adds myTool depending on final.hello

Prove myTool sees the patched hello if it references it (even a trivial wrap).

Step 6 — Dependency override

Pick a package with a boolean flag (ffmpeg, qemu, etc.—choose something not enormous if you can):

pkgs.ffmpeg.override { /* one flag */ }

Build or evaluate and document why you stopped short if closure is huge.

Step 7 — Maintenance note

In NOTES.md: expiry conditions for the patch (“remove when nixpkgs ≥ rev …” / “upstream PR #”).

Common gotchas

Symptom / mistake What to do
Patch rejects Refresh patch against actual src version
Overlay silently not applied Ensure same pkgs instance reaches the host/package
Infinite recursion in overlay Use prev for the package you modify; avoid final.pkg = final.pkg…
Dropped upstream patches Always (old.patches or []) ++ [ ours ]
Rebuilt half of nixpkgs You overrode a low-level lib; narrow scope
Unmaintainable mega-patch Split; upstream; or wait for release
Darwin vs Linux patch drift Conditional patches with lib.optionals stdenv.isLinux

Checkpoint

  • overrideAttrs patch builds and changes behavior
  • Overlay form works from flake
  • Two overlays composed with known order
  • Difference override vs overrideAttrs written down
  • Patch expiry / upstream plan noted

Commit

git add .
git commit -m "day49: patches, overrideAttrs, overlays"

Write three personal gotchas before continuing.

Tomorrow

Day 50 — Multiple outputs: split out/dev/doc and shrink closures on purpose.

Day 50 — Multiple outputs

Stage V · ~4h
Goal: Use multi-output derivations (out, dev, man, doc, …) to keep runtime closures small while preserving headers and debug assets for dependents.

Why this day exists

A single $out that ships headers, man pages, static libs, and a 2 MB binary forces every dependent to download and GC the world. Multi-output packaging is how nixpkgs stays usable on laptops and CI runners.

Theory 1 — Outputs are separate store paths

/nix/store/hash-foo-1.0       ← out (runtime)
/nix/store/hash-foo-1.0-dev   ← dev (headers, pkg-config)
/nix/store/hash-foo-1.0-man   ← man
/nix/store/hash-foo-1.0-doc   ← html docs

Each is a different path. References between them are tracked so GC stays correct: runtime out should not reference dev if you did it right.

stdenv.mkDerivation {
  pname = "foo";
  version = "1.0";
  outputs = [ "out" "dev" "man" ];
  # …
}

Default if omitted: outputs = [ "out" ];.

Theory 2 — Conventional output roles

Output Typical content Who needs it
out Binaries, shared libs, runtime data Everyone running the tool
dev Headers, .pc, static libs, cmake files Compiling dependents
lib Sometimes shared libs split from bins Rare patterns / legacy
man Man pages Humans; optional on servers
doc HTML/PDF docs Optional
info Texinfo Optional
devdoc Heavy API docs Dev machines
debug Separate debuginfo (with separateDebugInfo) Debuggers

Environment variables in the build

Var Points at
$out First output (usually runtime)
$dev Dev output path
$man Man output
$outputs List of outputs

Install into the correct root:

install -Dm755 foo $out/bin/foo
install -Dm644 foo.h $dev/include/foo.h
install -Dm644 foo.pc $dev/lib/pkgconfig/foo.pc
install -Dm644 foo.1 $man/share/man/man1/foo.1

Theory 3 — Why dependents get smaller

app depends on libfoo
  → needs libfoo.out (and its runtime closure)
  → does NOT need libfoo.dev unless building against headers

nix path-info -r on an application should not pull *-dev for every library if packaging is correct.

nix path-info -r ./result | rg -- '-dev$' || echo "no dev outputs in closure (good for runtime)"

Theory 4 — propagatedBuildInputs and multi-out hazards

Hazard Result Mitigation
Headers installed to $out Runtime closure grows Move to $dev
out references $dev Forces dev into runtime closure Fix install paths / use moveToOutput
Forgetting outputs but installing to $dev Empty/wrong paths Declare outputs first
pkg-config file wrong prefix Consumers cannot find libs PKG_CONFIG paths; check .pc

stdenv provides helpers like moveToOutput used heavily in nixpkgs.

moveToOutput share/man "$man"
moveToOutput include "$dev"

Theory 5 — Consuming multi-output packages

# As a build dependency of another package
nativeBuildInputs = [ pkg-config ];
buildInputs = [ foo ]; # pulls foo.dev for compilation via setup hooks usually

Setup hooks from dev outputs typically put headers on the search path when foo is in buildInputs.

Selecting outputs explicitly (advanced / rare in modern nixpkgs):

foo.dev  # the dev store path attribute
foo.out
foo.man

Worked example — toy multi-output C library + app

/* src/libgreet/greet.c */
#include "greet.h"
const char *greet_msg(void) { return "hello multi-out"; }
/* src/libgreet/greet.h */
#pragma once
const char *greet_msg(void);
/* src/greetapp/main.c */
#include <stdio.h>
#include "greet.h"
int main(void) { puts(greet_msg()); return 0; }
# pkgs/libgreet.nix
{ stdenv, lib }:
stdenv.mkDerivation {
  pname = "libgreet";
  version = "0.1.0";
  outputs = [ "out" "dev" "man" ];
  src = ../src/libgreet;

  buildPhase = ''
    $CC -shared -fPIC greet.c -o libgreet.so
  '';

  installPhase = ''
    runHook preInstall
    install -Dm755 libgreet.so $out/lib/libgreet.so
    install -Dm644 greet.h $dev/include/greet.h
    mkdir -p $dev/lib/pkgconfig
    cat > $dev/lib/pkgconfig/libgreet.pc <<EOF
    prefix=$out
    libdir=$out/lib
    includedir=$dev/include
    Name: libgreet
    Version: 0.1.0
    Libs: -L$out/lib -lgreet
    Cflags: -I$dev/include
    EOF
    mkdir -p $man/share/man/man3
    echo ".TH GREET 3" > $man/share/man/man3/greet.3
    runHook postInstall
  '';

  meta = {
    description = "Tiny greet library (multi-output lab)";
    license = lib.licenses.mit;
  };
}
# pkgs/greetapp.nix
{ stdenv, lib, libgreet, pkg-config }:
stdenv.mkDerivation {
  pname = "greetapp";
  version = "0.1.0";
  src = ../src/greetapp;
  nativeBuildInputs = [ pkg-config ];
  buildInputs = [ libgreet ];
  buildPhase = ''
    $CC main.c -o greetapp $(pkg-config --cflags --libs libgreet)
  '';
  installPhase = ''
    install -Dm755 greetapp $out/bin/greetapp
  '';
  meta = {
    mainProgram = "greetapp";
    license = lib.licenses.mit;
  };
}
nix build .#libgreet
ls result*           # multiple outputs may appear as result, result-dev, …
nix build .#libgreet.dev -o result-dev
nix build .#greetapp
nix path-info -r ./result | rg libgreet

Lab — multi-step

Suggested workspace: ~/lab/90daysofx/02-nixos/day50

Step 1 — Inspect an existing multi-out package

nix eval nixpkgs#openssl.outputs
nix build nixpkgs#openssl -o o-out
nix build nixpkgs#openssl.dev -o o-dev
du -sh o-out o-dev

Record sizes.

Step 2 — Build libgreet with three outputs

Implement the toy library (or split your Day 45 tool into lib + bin).

Step 3 — Verify separate paths

nix build .#libgreet
nix build .#libgreet.dev
nix build .#libgreet.man
nix path-info .#libgreet .#libgreet.dev .#libgreet.man

Step 4 — Build greetapp against libgreet

Confirm the app runs and its runtime closure includes libgreet out but ideally not manpages.

nix path-info -r ./result | rg 'libgreet'

Step 5 — Deliberate mistake

Install a header into $out/include and rebuild app. Compare closure size / path-info. Fix it.

Step 6 — Document policy

In NOTES.md: when you will use outputs = [ "out" "dev" ] for internal packages vs single-output scripts.

Step 7 — Optional separateDebugInfo

separateDebugInfo = true;

Build and note the extra debug output if supported on your platform.

Common gotchas

Symptom / mistake What to do
dev empty Not installing into $dev; check outputs list
Runtime needs dev Cross-reference leak; fix files that embed $dev paths
pkg-config cannot find package Ensure .pc in dev and package in buildInputs
Only result symlink Multi-out: use nix build .#pkg.dev or -o names
Broke library for dependents Test a dependent package in the same flake
Docs in out moveToOutput or install directly to $doc/$man

Checkpoint

  • Can explain multi-out as multiple store paths
  • Toy (or real) package produces out + dev
  • Dependent app builds against dev, runs against out
  • Closure inspection done with nix path-info
  • One deliberate mis-install fixed

Commit

git add .
git commit -m "day50: multiple outputs libgreet + greetapp"

Write three personal gotchas before continuing.

Tomorrow

Day 51 — Debug builds: nix log, --keep-failed, breakpointHook, and recovering from sandbox failures.