sysmon, Scavenger, and Background Runtime

Updated

September 8, 2026

sysmon, Scavenger, and Background Runtime

Overview

Besides your goroutines, the runtime runs background machinery: monitoring long preemption needs, retaking Ps stuck in syscalls, returning memory to the OS (scavenger), and coordinating GC.

Diagram: Background runtime

flow:
  [Sysmon]
       |
       v
  [Netpoll]

sysmon (System Monitor)

A runtime thread that periodically:

  • Retakes Ps from Ms blocked in syscalls too long
  • Helps with preemption / netpoll wakeups (version-dependent details)
  • Notices sleeping conditions

You do not call sysmon — but its work appears when syscall-heavy loads misbehave.

Memory Scavenging

After GC frees spans, memory may remain reserved to the process. The scavenger returns idle heap pages to the OS (with hysteresis) so RSS can fall.

HeapInuse vs HeapSys vs RSS
  HeapInuse  - occupied objects
  HeapSys    - virtual from OS for heap
  RSS        - OS resident (includes scavenged lag)

GOMEMLIMIT and GC interact with how aggressively memory returns.

GC Workers

Mark workers and assist run as special Gs. High GCCPUFraction means the mutator is paying for GC.

What You Control

Control Effect
GOMAXPROCS Parallel Go + related resources
GOGC / GOMEMLIMIT GC frequency & memory ceiling
Allocation rate Drives all of the above
Syscall/cgo rate Pressure on Ms and sysmon retake

Experiment

GODEBUG=gctrace=1 go run .
package main

import (
    "fmt"
    "runtime"
    "time"
)

func main() {
    var m runtime.MemStats
    hold := make([][]byte, 0, 500)
    for i := 0; i < 500; i++ {
        hold = append(hold, make([]byte, 100*1024))
    }
    runtime.ReadMemStats(&m)
    fmt.Printf("hold HeapInuse=%dMiB HeapSys=%dMiB\n", m.HeapInuse>>20, m.HeapSys>>20)
    hold = nil
    runtime.GC()
    time.Sleep(200 * time.Millisecond)
    runtime.ReadMemStats(&m)
    fmt.Printf("after HeapInuse=%dMiB HeapSys=%dMiB\n", m.HeapInuse>>20, m.HeapSys>>20)
}

What to notice: HeapInuse drops quickly after GC; HeapSys/RSS may lag until scavenging.

Try next: Compare with GOMEMLIMIT set near working set size.