Filesystem and Modern Path Manipulation

Updated

September 7, 2026

Filesystem and Modern Path Manipulation

After reading this chapter, you will master object-oriented filesystem operations with pathlib.Path, perform recursive directory walking and globbing, stream large files safely in buffered chunks, manage high-level file tree copies and removals with shutil, and allocate secure ephemeral storage with tempfile.

Mental model

Legacy Python code used strings and functions from the os.path module (os.path.join, os.path.exists). In modern Python, the pathlib module models paths as rich objects with operator overloading:

Legacy String Manipulation:
  path = os.path.join(os.path.join(base_dir, "logs"), "app.log")

Modern Object-Oriented Pathlib:
  path = Path(base_dir) / "logs" / "app.log"
          │
          ├── path.name      ──▶ "app.log"
          ├── path.stem      ──▶ "app"
          ├── path.suffix    ──▶ ".log"
          ├── path.parent    ──▶ Path(base_dir) / "logs"
          └── path.exists()  ──▶ True / False

The / operator is overloaded to perform cross-platform path concatenation (/ on Linux/macOS, \ on Windows) automatically.

Minimal example

Save as pathlib_showcase.py:

# pathlib_showcase.py
import tempfile
from pathlib import Path

def main() -> None:
    # Use temporary directory for safe filesystem demonstrations
    with tempfile.TemporaryDirectory() as tmpdir:
        root = Path(tmpdir)
        log_dir = root / "var" / "log" / "cluster"
        
        # 1. Create nested directories (mkdir with parents=True)
        log_dir.mkdir(parents=True, exist_ok=True)
        print(f"Created directory tree: {log_dir}")

        # 2. Write and read text directly without manual open/close
        app_log = log_dir / "app.log"
        app_log.write_text("2026-09-07T10:00:00Z [INFO] System initialized\n")

        # Append more content
        with app_log.open("a") as f:
            f.write("2026-09-07T10:00:05Z [WARN] Memory watermark high\n")

        # 3. Read back text
        content = app_log.read_text()
        print(f"\nFile content ({app_log.stat().st_size} bytes):")
        print(content.strip())

        # 4. Globbing for matching files
        print(f"\nMatching files (*.log):")
        for match in root.rglob("*.log"):
            print(f"  Found: {match.relative_to(root)}")

if __name__ == "__main__":
    main()

Run via uv run python pathlib_showcase.py:

Created directory tree: .../var/log/cluster

File content (87 bytes):
2026-09-07T10:00:00Z [INFO] System initialized
2026-09-07T10:00:05Z [WARN] Memory watermark high

Matching files (*.log):
  Found: var/log/cluster/app.log

Worked examples

Case 1: Buffered Chunk Streaming for Gigabyte-Scale Files

Reading huge files with .read() loads the entire file into RAM, crashing systems with MemoryError. Reading in buffered chunks maintains \(O(1)\) memory consumption:

# stream_hasher.py
import hashlib
import tempfile
from pathlib import Path

def compute_large_file_sha256(file_path: Path, chunk_size: int = 65536) -> str:
    hasher = hashlib.sha256()
    with file_path.open("rb") as f:
        # iter(lambda: f.read(chunk_size), b"") reads until empty bytes EOF sentinel
        for chunk in iter(lambda: f.read(chunk_size), b""):
            hasher.update(chunk)
    return hasher.hexdigest()

if __name__ == "__main__":
    with tempfile.NamedTemporaryFile() as tmp:
        # Write 5 MB of dummy data
        tmp.write(b"DATA_CHUNK" * (500_000))
        tmp.flush()

        digest = compute_large_file_sha256(Path(tmp.name))
        print(f"Calculated SHA256 in chunks: {digest}")

Run:

uv run python stream_hasher.py

Output:

Calculated SHA256 in chunks: f11e74031d279cae560f7e15bf28c50424564c7811ef140f04746f34fc3d18e8

Case 2: Atomic File Writes via Temporary Renaming

Writing directly to an active file risks leaving corrupt, partial files if the process is terminated mid-write. Standard practice writes to a sibling temporary file and performs an atomic replace:

# atomic_writer.py
import os
import tempfile
from pathlib import Path

def write_file_atomically(target_path: Path, content: str) -> None:
    # 1. Create temporary file in the same directory (guarantees same filesystem)
    target_dir = target_path.parent
    target_dir.mkdir(parents=True, exist_ok=True)

    with tempfile.NamedTemporaryFile("w", dir=target_dir, delete=False) as tmp:
        tmp.write(content)
        tmp.flush()
        os.fsync(tmp.fileno())  # Flush OS buffers to physical disk
        temp_path = Path(tmp.name)

    # 2. Atomic rename replaces target instantly (POSIX atomic guarantee)
    temp_path.replace(target_path)

if __name__ == "__main__":
    with tempfile.TemporaryDirectory() as tmpdir:
        config_file = Path(tmpdir) / "app_config.json"
        write_file_atomically(config_file, '{"version": 1, "active": true}')
        print(f"File atomically created: {config_file.name}")
        print(f"Content: {config_file.read_text()}")

Run:

uv run python atomic_writer.py

Output:

File atomically created: app_config.json
Content: {"version": 1, "active": true}

Pitfalls

Pitfall 1: Mixing os.path String Manipulation with Path Objects

Passing strings to functions expecting Path or concatenating str(path) + "/file" breaks cross-platform compatibility. Always stick to Path objects and the / operator.

Pitfall 2: Forgetting parents=True on mkdir()

Calling path.mkdir() fails with FileNotFoundError if intermediate parent directories do not already exist. Always supply parents=True, exist_ok=True when creating nested directory hierarchies.

Exercises

  1. Write a script that scans a directory and prints all files larger than 10 MB, displaying their sizes in megabytes.
  2. Implement a backup utility using shutil.copy2 that copies all .conf files from a source directory into an archive directory while preserving original timestamps.
  3. Write a function that counts the total number of lines across all .py files in a repository using Path.rglob().
  4. Demonstrate how pathlib.Path.resolve() canonicalizes symlinks and relative path navigation (..).

Further reading

  • Python Standard Library: pathlib, shutil, tempfile.
  • PEP 428: The pathlib Module — Object-oriented filesystem paths.