Text Processing, Unicode, and Regular Expressions

Updated

September 7, 2026

Text Processing, Unicode, and Regular Expressions

After reading this chapter, you will master regular expressions with the re module, compile reusable patterns with re.compile, extract structured data using named capture groups, sanitize strings using replacement callables with re.sub, and resolve security and encoding pitfalls with unicodedata.normalize.

Mental model

Python’s regular expression engine translates pattern strings into compiled bytecode that drives an NFA (Nondeterministic Finite Automaton) state machine.

Regex Pipeline:
  Raw Pattern: r"(?P<ip>\d{1,3}(?:\.\d{1,3}){3}) - (?P<status>\d{3})"
                          │
                          ▼
            [ re.compile(pattern) ] ──▶ Compiled Pattern Object
                          │
                          ▼ match(text)
            [ Match Object ]
            ├── .group("ip")     ──▶ "192.168.1.55"
            ├── .group("status") ──▶ "404"
            └── .groupdict()     ──▶ {"ip": "192.168.1.55", "status": "404"}

Always use raw string literals (r"...") for regular expressions. In regular strings, \b means backspace (ASCII 8), whereas in regex it represents a word boundary. Raw strings prevent Python from interpreting backslashes before passing them to the regex engine.

Minimal example

Save as regex_text_processing.py:

# regex_text_processing.py
import re

# Pre-compile regex with named capture groups and verbose comments
LOG_PATTERN = re.compile(
    r"""
    ^
    (?P<timestamp>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z) \s+
    \[(?P<level>INFO|WARN|ERROR)\]                       \s+
    (?P<service>[a-zA-Z0-9_\-]+)                          \s+
    (?P<message>.*)
    $
    """,
    re.VERBOSE,
)

def parse_log_line(raw_line: str) -> dict[str, str] | None:
    match = LOG_PATTERN.match(raw_line.strip())
    if match:
        return match.groupdict()
    return None

def main() -> None:
    sample_log = "2026-09-07T10:00:00Z [ERROR] auth-gateway User token expired: uid_881"
    parsed = parse_log_line(sample_log)

    print("Parsed log record:")
    if parsed:
        for k, v in parsed.items():
            print(f"  {k:12} : {v}")

if __name__ == "__main__":
    main()

Run via uv run python regex_text_processing.py:

Parsed log record:
  timestamp    : 2026-09-07T10:00:00Z
  level        : ERROR
  service      : auth-gateway
  message      : User token expired: uid_881

Worked examples

Case 1: Dynamic Data Masking with re.sub() Callables

Instead of static string replacements, re.sub() accepts a callback function that inspects each match and computes a dynamic replacement:

# pii_masker.py
import re

# Match credit card patterns (4 groups of 4 digits)
CC_PATTERN = re.compile(r"\b(\d{4})[ -]?(\d{4})[ -]?(\d{4})[ -]?(\d{4})\b")

def mask_credit_card(match: re.Match[str]) -> str:
    # Keep only the last 4 digits visible
    last_four = match.group(4)
    return f"****-****-****-{last_four}"

def redact_sensitive_logs(log_text: str) -> str:
    return CC_PATTERN.sub(mask_credit_card, log_text)

if __name__ == "__main__":
    raw_event = "Customer 4111-2222-3333-4444 completed checkout of $149.00."
    sanitized = redact_sensitive_logs(raw_event)
    print("Sanitized text:")
    print(sanitized)

Run:

uv run python pii_masker.py

Output:

Sanitized text:
Customer ****-****-****-4444 completed checkout of $149.00.

Case 2: Unicode Normalization for Security Checks

In security systems (such as usernames, domain filtering, or password validation), identical-looking Unicode characters can represent different code points (visual spoofing). unicodedata.normalize standardizes representations:

# unicode_security.py
import unicodedata

def demonstrate_unicode_normalization() -> None:
    # 'é' represented as single code point (NFC)
    s1 = "\u00e9"
    # 'e' + combining acute accent (NFD)
    s2 = "e\u0301"

    print(f"String 1 ('{s1}') length: {len(s1)}")
    print(f"String 2 ('{s2}') length: {len(s2)}")
    print(f"Direct equality (s1 == s2): {s1 == s2}")

    # Canonical Normalization Form C (NFC)
    norm1 = unicodedata.normalize("NFC", s1)
    norm2 = unicodedata.normalize("NFC", s2)
    print(f"After NFC normalization (norm1 == norm2): {norm1 == norm2}")

if __name__ == "__main__":
    demonstrate_unicode_normalization()

Run:

uv run python unicode_security.py

Output:

String 1 ('é') length: 1
String 2 ('é') length: 2
Direct equality (s1 == s2): False
After NFC normalization (norm1 == norm2): True

Pitfalls

Pitfall 1: Catastrophic Backtracking (ReDoS)

Patterns with nested ambiguous quantifiers like (a+)+$ on an input like "aaaaaaaaaaaaaaaaaaaaX" cause the regex engine to explore \(2^N\) branches, freezing the CPU:

# DANGEROUS: Exponential backtracking risk on non-matching strings
BAD_REGEX = r"^([a-zA-Z0-9]+)*$"

# SAFE: Atomic, bounded, non-overlapping expressions
SAFE_REGEX = r"^[a-zA-Z0-9]+$"

Pitfall 2: Omitting Raw String Prefix (r"")

# THE BUG:
p = "\bnode\b"   # '\b' is ASCII backspace (0x08), NOT a regex word boundary!

# THE FIX:
p = r"\bnode\b"  # Raw string passes literal backslash-b to regex engine

Exercises

  1. Write a regular expression with named groups that extracts the protocol, host, and port from a URL string (https://api.internal:8443).
  2. Use re.findall() to extract all IPv4 addresses from an unformatted server configuration file.
  3. Write a function using re.sub that converts camelCase strings (totalRequestDuration) into snake_case (total_request_duration).
  4. Demonstrate how re.IGNORECASE and re.MULTILINE flags alter pattern matches across multi-line logs.

Further reading

  • Python Standard Library: re module documentation.
  • Python Documentation: Regular Expression HOWTO.
  • Unicode Standard Annex #15: Unicode Normalization Forms.