Object Serialization and Persistent Storage (pickle, shelve)

Updated

September 7, 2026

Object Serialization and Persistent Storage (pickle, shelve)

After reading this chapter, you will master deep Python object graph serialization using pickle, understand the binary pickle protocol and opcode execution model, customize serialization lifecycles using __getstate__ and __setstate__ to exclude ephemeral runtime resources (locks, sockets, file descriptors), construct secure unpickling firewalls that block arbitrary remote code execution, and leverage shelve for zero-setup, persistent key-value caching on disk.

Mental model

Higher-level formats like JSON and TOML serialize only primitive data types (dict, list, str, int, float, bool, None). They cannot represent custom classes, cyclical object graphs (where an object references itself), datetime instances, or method bindings without manual encoders and decoders.

The pickle module converts arbitrary, deeply nested Python object graphs into a compact stream of byte opcodes:

┌─────────────────────────────────────────────────────────────┐
│ Python Heap Memory (Arbitrary Object Graph)                 │
│   Node A ──▶ Node B ──▶ Node A (Cyclic Reference!)          │
│   UserSession(id=42, roles={'admin'}, created=datetime)        │
└──────────────────────────────┬──────────────────────────────┘
                               │
                      pickle.dumps(obj)
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ Pickle Byte Stream (Opcode Protocol)                        │
│   \x80\x05\x95... (Interpreted by Python Pickle VM)          │
└──────────────────────────────┬──────────────────────────────┘
                               │
                      pickle.loads(bytes)
                               ▼
┌─────────────────────────────────────────────────────────────┐
│ Reconstructed Heap Object Graph                             │
│   (Exact duplicate with preserved object identity & cycles) │
└─────────────────────────────────────────────────────────────┘

Security Warning: Never Unpickle Untrusted Data

pickle is not secure against maliciously crafted payloads. The unpickling process executes instructions that can instantiate any Python class and invoke arbitrary callables (like os.system). Never deserialize pickled blobs received over untrusted networks without cryptographic signatures (hmac) or restricted unpickling guards.

Minimal example

Save as pickle_overview.py:

# pickle_overview.py
from datetime import datetime, timezone
import pickle

class ServiceNode:
    def __init__(self, node_id: str, host: str) -> None:
        self.node_id = node_id
        self.host = host
        self.registered_at = datetime.now(timezone.utc)
        self.neighbors: list["ServiceNode"] = []

    def __repr__(self) -> str:
        return f"ServiceNode({self.node_id}, host={self.host}, neighbors={len(self.neighbors)})"

def main() -> None:
    # 1. Construct a cyclic graph: Node 1 <───▶ Node 2
    node1 = ServiceNode("node-01", "10.0.0.1")
    node2 = ServiceNode("node-02", "10.0.0.2")
    node1.neighbors.append(node2)
    node2.neighbors.append(node1)

    print(f"Original: {node1}")
    print(f"Cyclic reference intact: {node1.neighbors[0].neighbors[0] is node1}")

    # 2. Serialize complete cyclic graph to bytes
    serialized_bytes = pickle.dumps(node1, protocol=pickle.HIGHEST_PROTOCOL)
    print(f"\nSerialized binary size: {len(serialized_bytes)} bytes")

    # 3. Deserialize back into independent object graph
    restored_node1 = pickle.loads(serialized_bytes)
    print(f"\nRestored: {restored_node1}")
    print(f"Restored cyclic reference intact: {restored_node1.neighbors[0].neighbors[0] is restored_node1}")
    print(f"Identity check with original: {restored_node1 is not node1}")

if __name__ == "__main__":
    main()

Run via uv run python pickle_overview.py:

Original: ServiceNode(node-01, host=10.0.0.1, neighbors=1)
Cyclic reference intact: True

Serialized binary size: 197 bytes

Restored: ServiceNode(node-01, host=10.0.0.1, neighbors=1)
Restored cyclic reference intact: True
Identity check with original: True

Worked examples

Case 1: Sandboxed Deserialization with SafeUnpickler

When accepting serialized blobs internally across microservices or worker processes, overriding pickle.Unpickler.find_class blocks arbitrary class execution by enforcing an explicit allowlist:

# safe_unpickler.py
import io
import pickle
from typing import Any

class SafeUnpickler(pickle.Unpickler):
    """Custom Unpickler that permits only explicitly allowlisted classes."""

    ALLOWED_CLASSES = {
        ("builtins", "dict"),
        ("builtins", "list"),
        ("builtins", "set"),
        ("builtins", "int"),
        ("builtins", "float"),
        ("builtins", "str"),
        ("builtins", "bool"),
    }

    def find_class(self, module: str, name: str) -> Any:
        if (module, name) in self.ALLOWED_CLASSES:
            return super().find_class(module, name)
        raise pickle.UnpicklingError(f"Security Alert: Disallowed class '{module}.{name}'")

def safe_loads(data: bytes) -> Any:
    return SafeUnpickler(io.BytesIO(data)).load()

# Malicious exploit class attempting remote command execution
class MaliciousPayload:
    def __reduce__(self):
        import os
        return (os.system, ("echo COMPROMISED",))

def main() -> None:
    # 1. Legitimate data succeeds
    safe_data = {"tenant": "acme-corp", "tier": "enterprise", "features": ["sso", "audit"]}
    safe_blob = pickle.dumps(safe_data)
    restored = safe_loads(safe_blob)
    print(f"Safe deserialization succeeded: {restored}")

    # 2. Malicious payload is blocked before execution
    exploit_blob = pickle.dumps(MaliciousPayload())
    try:
        safe_loads(exploit_blob)
    except pickle.UnpicklingError as err:
        print(f"\nExploit successfully neutralized:\n  {err}")

if __name__ == "__main__":
    main()

Run:

uv run python safe_unpickler.py

Output:

Safe deserialization succeeded: {'tenant': 'acme-corp', 'tier': 'enterprise', 'features': ['sso', 'audit']}

Exploit successfully neutralized:
  Security Alert: Disallowed class 'posix.system'

Case 2: Persistent Disk State with shelve

The shelve module provides a persistent dictionary on disk backed by a lightweight DBM database. Keys must be strings, and values are automatically pickled and unpickled transparently:

# persistent_shelve_store.py
import shelve
import tempfile
from pathlib import Path

def main() -> None:
    with tempfile.TemporaryDirectory() as tmpdir:
        db_path = str(Path(tmpdir) / "app_state.db")

        # 1. Open shelf and persist complex objects
        with shelve.open(db_path) as db:
            db["cluster_settings"] = {"env": "production", "max_workers": 16}
            db["node_inventory"] = ["node-alpha", "node-beta"]
            print("Persisted initial state to disk.")

        # 2. Re-open in a separate session and read back
        with shelve.open(db_path) as db:
            print("\nRestored from disk:")
            print(f"  Settings: {db['cluster_settings']}")
            print(f"  Nodes   : {db['node_inventory']}")

        # 3. In-place mutations require writeback=True!
        with shelve.open(db_path, writeback=True) as db:
            # Append directly to the stored list
            db["node_inventory"].append("node-gamma")

        # 4. Verify in-place update persisted
        with shelve.open(db_path) as db:
            print(f"\nNodes after writeback update: {db['node_inventory']}")

if __name__ == "__main__":
    main()

Run:

uv run python persistent_shelve_store.py

Output:

Persisted initial state to disk.

Restored from disk:
  Settings: {'env': 'production', 'max_workers': 16}
  Nodes   : ['node-alpha', 'node-beta']

Nodes after writeback update: ['node-alpha', 'node-beta', 'node-gamma']

Case 3: Excluding Transient Resources with __getstate__ and __setstate__

Objects containing unpicklable OS handles (such as threading.Lock, open sockets, or database connections) crash pickle.dumps() unless you implement __getstate__ to sanitize the serialized state and __setstate__ to reconstruct transient fields:

# custom_pickle_state.py
import pickle
import threading

class StatefulSession:
    def __init__(self, session_id: str, username: str) -> None:
        self.session_id = session_id
        self.username = username
        self.login_attempts = 0
        # Ephemeral OS lock (cannot be pickled directly!)
        self._lock = threading.Lock()

    def record_attempt(self) -> None:
        with self._lock:
            self.login_attempts += 1

    def __getstate__(self) -> dict:
        """Custom state extraction: strips non-serializable OS lock."""
        state = self.__dict__.copy()
        del state["_lock"]
        return state

    def __setstate__(self, state: dict) -> None:
        """Custom state restoration: re-initializes fresh OS lock."""
        self.__dict__.update(state)
        self._lock = threading.Lock()

def main() -> None:
    session = StatefulSession("sess-4819", "alice")
    session.record_attempt()
    session.record_attempt()
    print(f"Pre-serialization state: {session.username} (attempts={session.login_attempts})")

    # Serialize object (succeeds without Lock pickling error!)
    serialized = pickle.dumps(session)

    # Deserialize in worker process
    restored = pickle.loads(serialized)
    print(f"Restored session state : {restored.username} (attempts={restored.login_attempts})")

    # Verify restored lock operates normally
    restored.record_attempt()
    print(f"State after lock usage : attempts={restored.login_attempts}")

if __name__ == "__main__":
    main()

Run:

uv run python custom_pickle_state.py

Output:

Pre-serialization state: alice (attempts=2)
Restored session state : alice (attempts=2)
State after lock usage : attempts=3

Pitfalls

Pitfall 1: Mutating Mutable Objects in shelve Without writeback=True

By default, accessing db["key"] unpickles a fresh copy of the object into RAM. Modifying that object in-place (e.g. db["list"].append(x)) modifies only the temporary copy in memory; shelve does NOT detect the change and fails to save it to disk!

# THE TRAP:
import shelve

with shelve.open("cache.db") as db:
    db["tags"] = ["prod"]
    db["tags"].append("v2")  # BUG: Modified in memory, NEVER written to disk!

with shelve.open("cache.db") as db:
    print(db["tags"])  # Prints ['prod'], 'v2' was lost!

# THE FIX:
# Option A: Re-assign explicitly
with shelve.open("cache.db") as db:
    tags = db["tags"]
    tags.append("v2")
    db["tags"] = tags  # Explicit assignment triggers disk write

# Option B: Open with writeback=True
with shelve.open("cache.db", writeback=True) as db:
    db["tags"].append("v3")  # Automatically synchronized on close

Pitfall 2: Protocol Incompatibility Across Python Versions

Pickle has multiple protocol versions (Protocol 0 through Protocol 5). When saving serialized data that must be read by older Python versions, passing protocol=pickle.HIGHEST_PROTOCOL (Protocol 5 in modern Python) causes older versions to fail with ValueError: unsupported pickle protocol. For backward compatibility, explicitly specify protocol=4.

Exercises

  1. Build an atomic file-backed session store that serializes user session objects using pickle.dump() to a temporary file and atomically replaces the destination file using os.replace().
  2. Implement a RestrictedUnpickler that permits only a custom dataclass and standard collections, rejecting all other types.
  3. Write a benchmarking script comparing the serialization and deserialization speeds of a 10,000-element dictionary across json, pickle (Protocol 5), and marshal.
  4. Implement an LRU cache decorator backed by shelve that caches function outputs across terminal restarts.
  5. Create a class that implements __reduce__ to serialize an instance into an importable module path and a constructor argument tuple.

Further reading

  • Python Documentation: pickle — Python object serialization.
  • Python Documentation: shelve — Python object persistence.
  • David Beazley: Python Cookbook (Recipe 5.21: Serializing Python Objects).
  • PEP 574: Pickle protocol 5 with out-of-band data buffers.