Chapter 21: cgroups, systemd and Resource Management

Updated

October 31, 2026

Chapter 21: cgroups, systemd and Resource Management

Control Groups (cgroups) are the foundational Linux kernel mechanism for metering, allocating, limiting, and isolating system resources (CPU, Memory, I/O, PIDs, RDMA) across groups of processes. In modern Linux and container runtimes, cgroups v2 provides a unified hierarchy that coordinates resource limits cleanly with systemd.

This chapter explores the differences between cgroups v1 and v2, CPU CFS quota throttling, memory pressure limits, I/O bandwidth control, systemd unit isolation, and process limits (ulimit).

cgroups v1 vs v2: Architecture Differences

In cgroups v1, every resource controller (cpu, memory, blkio, net_cls) had its own independent hierarchy:

  • A process could be in group /web in the CPU hierarchy, but in group /db in the memory hierarchy.
  • This made coordinated resource management impossible: the kernel could not correlate buffered I/O writes with memory page allocation because blkio and memory were in completely separate hierarchies.
  • cgroups v2 (Unified Hierarchy): Standardized on a single unified process tree mounted at /sys/fs/cgroup. A process belongs to exactly one cgroup, and controllers operate coherently across CPU, Memory, and I/O simultaneously.
┌────────────────────────────────────────────────────────────────────────┐
│                        CGROUPS v2 UNIFIED TREE                         │
│                                                                        │
│   /sys/fs/cgroup (Root)                                                │
│   ├── system.slice                                                     │
│   │   ├── nginx.service (cpu.max, memory.max, io.max)                  │
│   │   └── postgresql.service                                           │
│   ├── user.slice                                                       │
│   │   └── user-1000.slice                                              │
│   └── kubepods.slice (Container Pods)                                  │
└────────────────────────────────────────────────────────────────────────┘

CPU Controllers and Throttling

Under cgroups v2, CPU allocation is governed by weights and quotas:

  • Proportional Sharing (cpu.weight): Priority weight from 1 to 10000 (default 100). When CPU is saturated, cores are divided proportionally to weights. When idle, any cgroup can consume 100% of available CPU.

  • Hard Quotas (cpu.max): Restricts maximum CPU consumption over a period. Format: $QUOTA $PERIOD in microseconds:

    # Limit cgroup to 2.5 CPU cores (250,000µs quota per 100,000µs period)
echo "250000 100000" | sudo tee /sys/fs/cgroup/workload/cpu.max

The CPU Throttling Latency Trap

When a multi-threaded application exceeds its quota within the first 20ms of a 100ms period, the kernel immediately deschedules all its threads for the remaining 80ms! This causes catastrophic p99 tail latency spikes:

# Check how many periods a cgroup has been throttled
grep nr_throttled /sys/fs/cgroup/system.slice/nginx.service/cpu.stat

Modern kernels (5.14+) include CPU Bursting (cpu.max.burst), allowing tasks to borrow unused quota from previous periods to absorb micro-bursts without triggering throttling stalls.

Memory Controllers: Limits and Pressure

cgroups v2 provides granular memory boundaries:

  • memory.min: Hard memory protection. If memory drops below this threshold, the kernel will never reclaim memory from this cgroup.
  • memory.low: Soft memory protection. Reclaim occurs only if memory cannot be reclaimed from other unprotected cgroups.
  • memory.high: Throttle limit. When exceeded, the processes in the cgroup are throttled and forced into asynchronous direct reclaim to drop pages.
  • memory.max: Hard maximum. If exceeded and reclaim fails, the Out-Of-Memory (OOM) killer terminates processes in the cgroup.
# Set soft limit to 4GB and hard maximum to 8GB
echo 4294967296 | sudo tee /sys/fs/cgroup/workload/memory.high
echo 8589934592 | sudo tee /sys/fs/cgroup/workload/memory.max

I/O Controllers: Throttling and Prioritization

cgroups v2 coordinates page cache writeback with block device throttling:

# Throttle device 259:0 (NVMe) to 5000 read IOPS and 50 MB/s write bandwidth
echo "259:0 riops=5000 wbps=52428800" | sudo tee /sys/fs/cgroup/workload/io.max

Because cgroups v2 unifies memory and I/O, dirty pages generated by this cgroup are attributed to its I/O budget during writeback, preventing rogue background tasks from swamping storage.

systemd Resource Controls and Service Isolation

In production, avoid interacting with /sys/fs/cgroup directly; configure resource limits via systemd unit files or drop-in overrides:

# /etc/systemd/system/api-service.service.d/override.conf
[Service]
# CPU Limits: allocate 4 cores with burst capability
CPUAccounting=true
CPUQuota=400%
CPUWeight=500

# Memory Limits: throttle at 14GB, hard OOM limit at 16GB
MemoryAccounting=true
MemoryHigh=14G
MemoryMax=16G

# I/O Limits: prioritize storage requests
IOAccounting=true
IOWeight=800
IOReadIOPSMax=/dev/nvme0n1 10000

# Tasks Limit: prevent fork bombs
TasksMax=4096

Reload and apply:

sudo systemctl daemon-reload
sudo systemctl restart api-service

ulimits and Resource Limits

Before cgroup limits are reached, the kernel enforces per-process resource limits (ulimit):

Limit Description Production Recommendation
nofile Max open file descriptors (sockets, files) Set to at least 65536 or 1048576 for servers.
nproc Max process/thread count for user Set to unlimited or generous limits for services.
memlock Max memory locked into RAM (mlock()) Set to unlimited for databases using HugePages or DPDK.

In systemd services, set these directly in unit files (LimitNOFILE=1048576, LimitMEMLOCK=infinity), bypassing legacy /etc/security/limits.conf limits entirely.