Chapter 9: Auditing, Logging, and Security Monitoring
Chapter 9: Auditing, Logging, and Security Monitoring
Preventative security controls—such as firewalls, file permissions, and Mandatory Access Control—are essential, but they are incomplete without visibility. An adversary operating inside an unmonitored infrastructure will systematically probe for misconfigurations, test privilege escalation vectors, and establish persistence completely unnoticed.
Security monitoring transforms an opaque Linux system into a transparent, auditable environment. This chapter covers the logging and auditing architecture: system logging with journald and rsyslog, the kernel-level Linux Audit framework (auditd), remote log shipping over TLS, File Integrity Monitoring (FIM), host-based intrusion detection, automated compliance scanning with OpenSCAP, and Security Information and Event Management (SIEM) integration.
System Logging with Journald and Rsyslog
Linux logging is orchestrated through a partnership between systemd-journald and traditional syslog daemons (such as rsyslog or syslog-ng).
┌────────────────────────────────────────────────────────────────────────┐
│ LINUX LOGGING ARCHITECTURE │
│ │
│ Applications / Kernel / Daemons │
│ │ │
│ ▼ /dev/log & /run/systemd/journal/socket │
│ systemd-journald (Binary, indexed, structured storage) │
│ │ │
│ ├─► Query locally via: journalctl │
│ │ │
│ ▼ ForwardToSyslog = yes │
│ rsyslog (Rule filtering, plain text formatting) │
│ │ │
│ ├─► Local text logs: /var/log/auth.log, /var/log/syslog │
│ │ │
│ ▼ Remote Encrypted Forwarding (TLS / RELP) │
│ Central SIEM / Log Cluster (Elasticsearch, Graylog, Splunk) │
└────────────────────────────────────────────────────────────────────────┘Hardening systemd-journald
Configure /etc/systemd/journald.conf to enforce persistent, tamper-resistant logging with rate-limit protections:
# /etc/systemd/journald.conf
[Journal]
Storage=persistent
Compress=yes
Seal=yes
RateLimitIntervalSec=30s
RateLimitBurst=10000
MaxRetentionSec=1month
MaxFileSec=1daySeal=yes: Enables Forward Secure Sealing (FSS), utilizing cryptographic keys to ensure that an attacker who subsequently achieves root access cannot retroactively alter or delete past log entries without detection.
Forwarding Logs Over TLS with Rsyslog
Plaintext syslog over UDP port 514 is unauthenticated and easily intercepted or spoofed. Configure rsyslog to transmit logs over an encrypted mutual-TLS channel in /etc/rsyslog.d/60-remote-tls.conf:
# /etc/rsyslog.d/60-remote-tls.conf
# Certificate and TLS parameters
global(
DefaultNetstreamDriver="gtls"
DefaultNetstreamDriverCAFile="/etc/ssl/certs/ca-corp.crt"
DefaultNetstreamDriverCertFile="/etc/ssl/certs/host.crt"
DefaultNetstreamDriverKeyFile="/etc/ssl/private/host.key"
)
# Forward all auth and authpriv events to remote SIEM
auth,authpriv.* action(
type="omfwd"
target="siem-collector.corp.internal"
port="6514"
protocol="tcp"
StreamDriver="gtls"
StreamDriverMode="1"
StreamDriverAuthMode="x509/name"
StreamDriverPermittedPeers="siem-collector.corp.internal"
)Linux Audit Framework and Auditd
While journald captures application-level output, the Linux Audit Framework operates directly inside the kernel, intercepting system calls before they return to userspace. It provides an incontrovertible record of file modifications, privilege transitions, and system executions.
┌────────────────────────────────────────────────────────────────────────┐
│ KERNEL AUDIT ARCHITECTURE │
│ │
│ Userspace Syscall: unlink("/etc/passwd") │
│ │ │
│ ▼ │
│ Kernel Syscall Dispatcher ──► Kernel Audit Subsystem (kauditd) │
│ │ Evaluates /etc/audit/rules │
│ ▼ Matches Rule │
│ Netlink Socket (NETLINK_AUDIT) │
│ │ │
│ ▼ │
│ auditd Userspace Daemon │
│ │ │
│ ▼ Writes to: │
│ /var/log/audit/audit.log │
└────────────────────────────────────────────────────────────────────────┘Hardened Production Audit Rules
Place audit rules in /etc/audit/rules.d/audit.rules. Group rules by function and tag them with distinct -k keys for fast querying:
# /etc/audit/rules.d/audit.rules
# 1. Self-Defense: Monitor audit configuration and tools
-w /etc/audit/ -p wa -k audit_config
-w /etc/libaudit.conf -p wa -k audit_config
-w /sbin/auditctl -p x -k audit_tools
-w /sbin/auditd -p x -k audit_tools
# 2. Monitor Identity, Authentication, and Passwords
-w /etc/passwd -p wa -k identity
-w /etc/shadow -p wa -k identity
-w /etc/group -p wa -k identity
-w /etc/security/ -p wa -k identity
-w /etc/pam.d/ -p wa -k pam_config
# 3. Monitor Sudo and Privilege Escalation
-w /etc/sudoers -p wa -k sudo_config
-w /etc/sudoers.d/ -p wa -k sudo_config
-w /var/log/sudo.log -p wa -k sudo_actions
# 4. Monitor System Initialization and Systemd Units
-w /etc/systemd/ -p wa -k systemd_units
-w /lib/systemd/system/ -p wa -k systemd_units
# 5. Monitor Network Configuration
-w /etc/network/ -p wa -k network_config
-w /etc/netplan/ -p wa -k network_config
-w /etc/resolv.conf -p wa -k network_config
# 6. Capture Unauthorized File Access Failures (EACCES and EPERM)
-a always,exit -F arch=b64 -S open,openat,creat,truncate -F exit=-EACCES -k access_denied
-a always,exit -F arch=b64 -S open,openat,creat,truncate -F exit=-EPERM -k access_denied
# 7. Make Configuration Immutable (Must be the final rule in the file!)
-e 2-e 2: Locks the audit configuration. Once set, rules cannot be modified, cleared, or disabled without a complete system reboot.
Load rules and test:
sudo augenrules --load
sudo auditctl -s # View statusSearching Audit Events with ausearch and aureport
# Search for identity modifications
sudo ausearch -k identity -ts today
# Search for failed access denials
sudo ausearch -k access_denied -ts recent
# Generate summary report of authentication attempts
sudo aureport -au --summaryFile Integrity Monitoring
File Integrity Monitoring (FIM) detects unauthorized file additions, deletions, and modifications across the system.
AIDE Scheduled Verification
As introduced in Chapter 3, AIDE builds a cryptographic inventory of system files. Automate daily scans and alert on unexpected diffs via systemd timer:
# /etc/systemd/system/aide-check.service
[Unit]
Description=AIDE File Integrity Check
[Service]
Type=oneshot
ExecStart=/usr/bin/aide --checkIntrusion Detection and Malware Scanning
Host-Based Intrusion Detection (HIDS) with Wazuh / OSSEC: Wazuh collects system logs, correlates audit events against signature rules, checks for rootkit presence, and detects brute-force attacks in real time.
Rootkit Hunting with
rkhunterandchkrootkit:sudo apt install -y rkhunter sudo rkhunter --update sudo rkhunter --propupd # Baseline system properties sudo rkhunter --check --skip-keypress
Vulnerability and Compliance Scanning
Automated compliance tools verify that thousands of hardening controls match authoritative benchmarks.
OpenSCAP Command-Line Auditing
The National Institute of Standards and Technology (NIST) maintains SCAP standards. Use oscap to audit systems against CIS or DISA STIG benchmarks:
sudo apt install -y libopenscap8 scap-security-guide
# Evaluate the host against the CIS Ubuntu Level 2 Server profile
sudo oscap xccdf eval \
--profile xccdf_org.ssgproject.content_profile_cis_level2_server \
--results-arf /var/log/compliance/report-arf.xml \
--report /var/log/compliance/report.html \
/usr/share/xml/scap/ssg/content/ssg-ubuntu2404-ds.xmlThe resulting HTML report gives a clear compliance score and detailed remediation scripts for failed rules.
SIEM Integration and Alerting
Centralizing logs into a SIEM (Elasticsearch, OpenSearch, Splunk, or cloud-native tools like AWS CloudWatch and GCP Cloud Logging) allows security operations centers (SOCs) to trigger immediate alerts on critical signals:
- Audit Key Alerts: Any event matching
-k audit_toolsor-k identitytriggers an urgent P1 notification. - Root Login Alerts: Any SSH or console session initiated as UID 0 generates an instant security notice.
- Rapid Auth Failures: More than 10 failed logins within 60 seconds from a single IP triggers automated IP blocking via
fail2banornftablesdynamic sets.