Service Mesh and Observability

Updated

September 12, 2026

Service Mesh and Observability

A mesh is another control plane. The boring default is: Prometheus + logs + a NetworkPolicy before Istio; add a mesh only when mTLS between 20+ services is a written requirement.

Mental model

Need Boring tool
CPU/RAM/HTTP metrics prometheus + node_exporter / app /metrics
Logs journald on NixOS; Loki if you already have it
mTLS inside the cluster Mesh (Istio/Linkerd) or app TLS
Traffic split Ingress / Service, then mesh

NixOS can run Prometheus on the ops host without Kubernetes. In-cluster, same rule: pin charts, generate YAML.

This chapter does not install Istio. It tells you what to measure first so you do not. A mesh will not invent RED metrics for an app that has no /metrics.

Worked examples

Case 1: App metrics without a mesh

Save as metrics.nix (NixOS ops-01):

# metrics.nix
{
  services.prometheus = {
    enable = true;
    listenAddress = "10.100.0.30";
    port = 9090;
    scrapeConfigs = [
      {
        job_name = "desk-api";
        static_configs = [ { targets = [ "10.100.0.11:8080" ]; } ];
      }
      {
        job_name = "node";
        static_configs = [
          { targets = [ "10.100.0.11:9100" "10.100.0.12:9100" ]; }
        ];
      }
    ];
  };
  services.prometheus.exporters.node.enable = true;
  networking.firewall.interfaces.wg0.allowedTCPPorts = [ 9090 9100 ];
}

The Go service exposes /metrics on 8080. Scrape only on wg0. Public 9090 is how you donate metrics to the internet.

Case 2: Probe the endpoint

curl -sS http://10.100.0.11:8080/metrics | head

Output (shape):

# HELP go_goroutines Number of goroutines
# TYPE go_goroutines gauge
go_goroutines 7

No output → fix the app, do not add Envoy. Then:

curl -sS http://10.100.0.30:9090/api/v1/targets | jq '.data.activeTargets[].health'

up is the boring dashboard. Grafana is optional.

Save a recording rule only when you have a query you already run by hand:

# prometheus-rules.nix fragment
{
  services.prometheus.rules = [
    ''
      groups:
        - name: desk-api
          rules:
            - alert: DeskApiDown
              expr: up{job="desk-api"} == 0
              for: 2m
    ''
  ];
}

An alert with no runbook in docs/threats.md is noise. Do not add twenty.

Case 3: NetworkPolicy before sidecar

Save as netpol.yaml:

# netpol.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: desk-api
  namespace: desk
spec:
  podSelector:
    matchLabels:
      app: desk-api
  policyTypes: [Ingress]
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: desk-ing
      ports:
        - port: 8080

Generate this from Nix like any manifest. Most “we need a mesh” tickets are “we need NetworkPolicy.” Apply, then curl from a pod that is not desk-ing and confirm it fails.

Case 4: If you must mesh

Pin the Linkerd or Istio chart version + hash (Helm chapter). Inject sidecars in one namespace first (desk). Measure CPU on the sidecar versus the app. Roll back with GitOps prune.

Do not enable the mesh globally on day one. Do not run Istio and Linkerd. Two meshes is two CNI races.

Case 5: Logs on NixOS already exist

journalctl -u desk-api -n 50 --no-pager
journalctl -u prometheus -n 20 --no-pager

Ship them with services.promtail or systemd-journal-upload if you have a Loki. A mesh access log is extra, not a substitute for application logs. desk-api returning 500 with a perfect Envoy dashboard is still a 500.

The trap

The trap is installing Istio to “get observability.” You get a second failure domain, CNI races, and a dashboard. Start with /metrics and NetworkPolicy. The other trap is Prometheus on 0.0.0.0:9090 without a firewall — scrape configs are not a secret, but they describe your fleet.

The boring rule

  • /metrics on the app. Prometheus scrape config in Nix. Listen on wg0.
  • NetworkPolicy before sidecars.
  • Mesh is optional and pinned. One namespace first. One mesh.
  • journalctl is a valid log system.
  • A mesh does not fix bad schema, slow SQL, or missing auth.

Try this

  1. Hit /metrics on a desk service or add a one-line Prometheus handler in a lab binary.
  2. Write a NetworkPolicy that allows only the ingress pods; prove a sidecar-less deny.
  3. List three problems a mesh does not fix (bad schema, slow SQL, missing auth).
  4. nixos-option services.prometheus.scrapeConfigs on 26.05 and add one job.