Troubleshooting Routing Security Controls

Troubleshooting Routing Security Controls

When routing security mechanisms malfunction, legitimate traffic is dropped, peering sessions flap, and reachability breaks. Diagnosing these failures requires a disciplined, hypothesis-driven methodology that isolates whether the fault resides in the router’s local policy, the RTR transport layer, the validator cache, or the upstream RIR cryptographic hierarchy.

+-------------------------------------------------------------------------+
|                  Routing Security Diagnostic Decision Tree               |
|                                                                         |
|  [Issue: Prefix P is unexpectedly dropped or marked INVALID]           |
|                               |                                         |
|                               v                                         |
|  1. Inspect Router State: `show bgp rpki prefix P`                      |
|     - Is the state actually Invalid, or is it dropped by a prefix-list? |
|                               |                                         |
|                               v                                         |
|  2. Check RTR Session Status: `show rpki server summary`                |
|     - Is the RTR session ESTABLISHED? Are VRP counts > 500,000?         |
|     - Has the cache expire timer triggered?                             |
|                               |                                         |
|                               v                                         |
|  3. Inspect Validator Output: Query Validator CLI / API                 |
|     - `routinator-cli vrps --prefix P` or rpki-client lookup            |
|     - Does a covering VRP exist in the local cache?                     |
|                               |                                         |
|                               v                                         |
|  4. Query Upstream RIR Repository & Whois:                              |
|     - Is the ROA active in the RIR database? Has it expired or revoked? |
|     - Check maxLength: Is announced prefix length > ROA maxLength?      |
+-------------------------------------------------------------------------+

Unexpected Invalid Routes: Systematic Diagnosis

When a route unexpectedly evaluates as Invalid: 1. Identify the exact announced parameters: Run show bgp <prefix> to extract: - Announced Prefix (e.g., 198.51.100.0/24) - Origin ASN (the rightmost ASN in the AS_PATH) 2. Compare against local VRP entries: On Cisco: show bgp rpki prefix 198.51.100.0/24. On Junos: show validation database 198.51.100.0/24. 3. Determine the exact invalid condition: - AS Mismatch: A covering VRP exists for the prefix, but specifies a different authorized ASN (e.g., authorized AS65001, but received AS65002). - Length Mismatch: The ASN matches, but the announced prefix length exceeds maxLength (e.g., announced /24, but ROA maxLength is /22).

RTR Session Failures and Stale Validation Data

If border routers report that all routes have reverted to NotFound or that RPKI tables are empty: - Check TCP Layer Connectivity: Ensure firewalls permit TCP port 323 / 3323 between the router loopback and the validator IP. - Inspect Serial Number Desynchronization: If a validator server reboots and resets its internal database without properly incrementing its serial number, the router may reject updates. Force an explicit reset query: - Cisco: clear bgp rpki server <IP> reset - Junos: clear validation session <IP> - Examine Expire Timers: If the validator fails to respond to keepalives, verify whether the router’s expire-time has lapsed, purging the cache from memory.

ROA Lookup: Verifying What Validators See

To determine whether an issue is local to the router or originates in the global RPKI tree, query the validator’s local API directly from a management host:

# Query Routinator's REST API for a specific prefix and origin
curl -s http://192.0.2.10:8323/api/v1/validity/AS65001/198.51.100.0/24 | jq .

Expected output:

{
  "validated_route": {
    "route": {
      "prefix": "198.51.100.0/24",
      "origin_asn": "AS65001"
    },
    "validity": {
      "state": "valid",
      "reason": null,
      "description": "At least one VRP Matches the Route Prefix"
    }
  }
}

If the validator reports notfound or invalid, query public tools (e.g., RIPE NCC RPKI Validator web dashboard or rpki-validator.ripe.net) to see if the global community observes the same state.

Legitimate Routes Rejected by Filters

If a route is dropped despite having a Valid RPKI state, the drop is typically caused by a static filter placed before the RPKI evaluation term in the route policy: - Bogon filter misconfiguration: The prefix falls within an overly broad bogon regex. - Prefix-list length restriction: The inbound prefix-list permits only /8 to /22, rejecting a legitimate /24. - Maximum-prefix trip: The neighbor’s total prefix count exceeded the configured ceiling, causing the router to drop all routes from that session.

Cross-Validator Consistency Checks

Running a single validator instance creates a single point of failure and vulnerability to implementation bugs: - Operate at least two distinct validator engines: e.g., Validator A running Routinator (Rust) and Validator B running rpki-client (C/OpenBSD). - Run automated cron scripts comparing the total VRP count output by each validator. If the VRP counts diverge by more than 2%, trigger an administrative alert to investigate repository sync failures or parsing discrepancies.

Post-Incident Forensics: Reconstructing What Happened

When a routing security incident concludes: 1. Pull BGP Table Dumps and BMP Archives: Capture historical updates from the incident window to reconstruct the exact timestamps when the illegitimate announcement appeared, propagated, and was withdrawn. 2. Review RIR Audit Logs: Determine whether the incident was caused by an account takeover, a revoked resource certificate, or an operator typo in the RIR portal. 3. Conduct Root-Cause Analysis (RCA): Document why local and upstream filters did or did not mitigate the issue, and update automated test suites to prevent recurrence.