Chapter 10: Secure Service and Workload Configuration

Updated

October 31, 2026

Chapter 10: Secure Service and Workload Configuration

A hardened operating system kernel and strict network perimeter provide little protection if the services running on top are misconfigured. Application workloads—web servers, database engines, mail transfer agents, CI/CD build runners, and cloud bastion hosts—are the interfaces that process external user inputs and business logic.

This chapter details the hardening of common Linux server workloads: web servers and reverse proxies, database management systems, DNS/mail infrastructure, cloud instances and metadata services, CI/CD runners, and distribution-specific maintenance lifecycles.

SSH Server Hardening

While Chapter 4 introduced basic SSH identity constraints, production SSH servers require end-to-end operational hardening:

# /etc/ssh/sshd_config.d/50-production-hardening.conf

# 1. Network and Connection Limits
MaxStartups 10:30:100
PerSourceMaxStartups 3
MaxAuthTries 3
LoginGraceTime 20

# 2. Strict Authentication Boundary
PermitRootLogin no
PasswordAuthentication no
ChallengeResponseAuthentication no
KerberosAuthentication no
GSSAPIAuthentication no
PubkeyAuthentication yes

# 3. Disable Environment Forwarding
# Prevents client environment variables (LD_PRELOAD) from infecting the shell
PermitUserEnvironment no

# 4. Restrict Insecure Subsystems
DisableForwarding yes
X11Forwarding no

# 5. Restrict to Specific Users or Groups via Match Blocks
Match Group devops-bastion
    AllowTcpForwarding yes
    AuthorizedKeysFile /etc/ssh/authorized_keys/%u

Rate Limiting and Automated Banning with fail2ban

Install and configure fail2ban to automatically drop brute-force connection attempts at the firewall:

# /etc/fail2ban/jail.d/sshd.local
[sshd]
enabled = true
port = 22
filter = sshd
backend = systemd
maxretry = 3
findtime = 600
bantime = 3600
banaction = nftables-multiport

Web Servers and Reverse Proxies

Web servers (such as Nginx and Apache) are directly exposed to the internet, processing untrusted HTTP requests:

Nginx Production Hardening Blueprint

# /etc/nginx/nginx.conf

http {
    # 1. Information Disclosure Suppression
    server_tokens off;

    # 2. Buffer and Request Size Limitations (Mitigate Slowloris and Buffer Overflow)
    client_body_buffer_size 16k;
    client_header_buffer_size 1k;
    client_max_body_size 8M;
    large_client_header_buffers 2 1k;
    client_body_timeout 12;
    client_header_timeout 12;
    keepalive_timeout 15;
    send_timeout 10;

    # 3. Comprehensive Security Response Headers
    add_header X-Frame-Options "DENY" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;
    add_header Content-Security-Policy "default-src 'self'; script-src 'self'; object-src 'none';" always;
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
    add_header Permissions-Policy "geolocation=(), camera=(), microphone=()" always;

    # 4. Restrict Unnecessary HTTP Methods
    server {
        listen 443 ssl http2;
        server_name api.corp.internal;

        if ($request_method !~ ^(GET|HEAD|POST|PUT|DELETE)$) {
            return 405;
        }
        ...
    }
}

Database Server Security

Databases (such as PostgreSQL and MySQL) house the enterprise’s most critical data assets.

PostgreSQL Hardening Checklist

  1. Network Binding: Bind strictly to internal interfaces or Unix domain sockets:

    # /etc/postgresql/16/main/postgresql.conf
listen_addresses = '10.0.10.5, localhost'

  2. Strict Client Authentication (pg_hba.conf): Mandate SCRAM-SHA-256 and reject cleartext or MD5 passwords:

    # /etc/postgresql/16/main/pg_hba.conf
# TYPE  DATABASE        USER            ADDRESS                 METHOD
local   all             postgres                                peer
hostssl app_db          app_user        10.0.20.0/24            scram-sha-256
host    all             all             all                     reject

  3. Dedicated Unprivileged Service Account: Ensure the PostgreSQL process runs strictly as user postgres with a umask of 0077, ensuring database data directories (/var/lib/postgresql/data) are inaccessible to other local users.

DNS and Mail Service Hardening

Securing BIND / Unbound

  • Chroot Execution: Run BIND inside an isolated chroot jail (/var/named/chroot) so that a remote execution vulnerability does not expose the host filesystem.

  • Disable Open Recursion: Prevent the server from being weaponized in DNS amplification DDoS attacks:

    options {
    recursion no;
    allow-query { 10.0.0.0/8; 127.0.0.1; };
};

Securing Postfix Mail Transfer Agent

  • Prevent Open Relay: Configure smtpd_relay_restrictions = permit_mynetworks, permit_sasl_authenticated, defer_unauth_destination.
  • Enforce Modern TLS: Set smtpd_tls_security_level = encrypt and smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1.
  • Configure SPF, DKIM, and DMARC: Enforce sender authenticity checks using Milters (opendkim, opendmarc).

Cloud Instances and Bastion Hosts

Virtual machines deployed in public clouds (AWS EC2, Google Cloud Engine, Azure VMs) operate in specialized networking and identity environments.

Securing the Instance Metadata Service (IMDSv2)

In public cloud environments, instances query a link-local address (169.254.169.254) to fetch IAM role credentials. A Server-Side Request Forgery (SSRF) flaw in a web application can expose these credentials to an attacker.

┌────────────────────────────────────────────────────────────────────────┐
│                        IMDSv2 SESSION ENFORCEMENT                      │
│                                                                        │
│   1. Request Session Token (PUT Request + Hop Limit = 1)               │
│   TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" ...`   │
│                                                                        │
│   2. Fetch Metadata with Token                                         │
│   curl -H "X-aws-ec2-metadata-token: $TOKEN" \                         │
│        http://169.254.169.254/latest/meta-data/iam/security-credentials│
└────────────────────────────────────────────────────────────────────────┘
  • Enforce IMDSv2 unconditionally via AWS CLI or Terraform (http_tokens = "required").
  • Set the metadata response hop limit to 1 so that containerized workloads or proxies running on the instance cannot reach the metadata service.

Bastion Jump Host Architecture

  • Do not attach public IP addresses to internal database or application nodes; place them in private subnets.
  • Access internal subnets strictly through an ephemeral bastion host guarded by SSH certificates, IP whitelisting, and multi-factor authentication.

CI/CD Systems and Build Infrastructure

CI/CD runner agents (such as GitLab Runners, GitHub Actions Self-Hosted Runners, Jenkins workers) execute untrusted code from git branches:

  1. Avoid Running Build Agents as Root: Never run runner daemons as root. Use isolated system accounts.
  2. Ephemeral Containers: Execute each build job inside an ephemeral, disposable container destroyed immediately upon job completion.
  3. Avoid Mounting Host Docker Sockets: Mounting /var/run/docker.sock inside a CI build container grants the build job total root access over the host machine. Instead, use rootless build tools (such as kaniko or buildah) that build images entirely in userspace without requiring host privileges.

Distribution-Specific Considerations

Enterprise Linux distributions differ in their maintenance cadences, security advisory portals, and long-term support (LTS) mechanisms:

  • Ubuntu LTS & Ubuntu Pro: Standard Ubuntu releases provide 5 years of maintenance. Ubuntu Pro extends this to 10–12 years with Expanded Security Maintenance (ESM) covering universe packages and kernel livepatches. Monitor Ubuntu Security Notices (USN).

  • Red Hat Enterprise Linux (RHEL): RHEL delivers 10 years of support with Extended Update Support (EUS) branches. Security errata are categorized as Critical, Important, Moderate, and Low (RHSA).

  • Automated Security Updates: Enable automated security updates for minor patch releases while maintaining change control:

    # Enable unattended-upgrades on Debian/Ubuntu
sudo apt install -y unattended-upgrades
sudo dpkg-reconfigure -plow unattended-upgrades