Enterprise-Scale Patterns and Production Operations

Authors

K19G

grok-bot

Updated

October 31, 2026

Enterprise-Scale Patterns and Production Operations

Enterprise scale is not a bigger state file. It is a set of root modules with owners, a promotion path that does not rely on a workspace name, and an upgrade habit that reads changelogs. This chapter is that layout for cloud and on-premises together.

Learning goals

  • Split accounts and on-premises estates into roots with separate state
  • Give a team a repository boundary and a shared module, not a shared apply role
  • Promote a change from development to production by reviewing the same module at a new pin
  • Upgrade OpenTofu and providers as their own commits, with a plan attached

Multi-Cloud and Hybrid Infrastructure: Managing AWS, Azure, GCP, and On-Premises Together

One root module may talk to several providers. A root that talks to all of them because they appear on one architecture slide is a root that fails when any one of them fails, and that only one team is allowed to apply.

Split by blast radius and by credential:

Root Owns Does not own
identity The accounts, folders, and projects other roots use The networks inside them
network Shared networks, once per cloud account Application buckets
app-<name> That application’s resources in one account The shared network. It takes ids as variables or remote outputs
onprem-<site> Appliances and DNS at that site The cloud account, except for a narrow handoff such as a VPN endpoint id

Hybrid is two roots and an explicit handoff value, not one configuration with every provider alias. The on-premises provider’s token and the cloud role should not sit in the same pipeline job unless the change truly crosses that line in one apply. Most changes do not.

Naming and tags carry the root’s identity into the console: managed-by=opentofu, the root name, and the repository URL. A human who edits the object can see who will overwrite the edit.

Organizational Structure: Team-Based Repository Layouts and Shared Responsibility

A layout that matches the way reviews work:

infra/
├── modules/                 # versioned when more than one root calls them
│   └── pets/
├── roots/
│   ├── pets-dev/
│   │   ├── versions.tf
│   │   ├── backend.tf
│   │   └── main.tf          # calls modules/pets
│   └── pets-prod/
└── policy/                  # Rego, pinned by the pipeline

The team that owns the application owns roots/pets-* and reviews those plans. A platform team owns modules/ and the policy repository. They do not need to approve every pet-name change. They do need to approve a new major of a module the application roots pin.

A monorepo is fine when one organisation wants one history. Many repositories are fine when ownership is the point. Do not split so far that a module and its only caller move in lockstep across a version boundary for no reason. Do not stay monolithic so far that a plan for a sandbox prints the production network.

CODEOWNERS, or the equivalent, maps directories to reviewers. The apply role for pets-prod is assumable by the production pipeline, not by every developer token. Developers plan in dev. They open a pull request for prod. Chapter 13 is that pipeline.

Environment Strategy: Development, Staging, Production Separation Patterns

Separate roots, separate backend keys, separate roles. Development may use smaller instances and may allow destroy. Production sets prevent_destroy on the data you cannot recreate and requires the approval gate.

Promotion is “the prod root moves its module pin from 1.4.2 to 1.5.0”, not “we apply the dev workspace with a different variable.” The module version is the artifact. The root’s variables are the environment. If prod and dev call different code, you are not promoting. You are maintaining two systems.

Staging exists when a change can break a dependency you cannot see in dev, such as a provider behaviour against a real load balancer. It is another root, smaller than production, with production-like configuration. It is not a workspace name inside the production state.

Ephemeral environments, one per pull request, are a root created by automation with a backend key that includes the request id, destroyed when the request closes. They need a budget, a janitor for leftovers, and a ban on using production data. They are a later optimisation, not the first layout.

Versioning and Upgrades: OpenTofu Versions, Provider Updates, and Breaking Changes

Keep a matrix and look at it on a schedule. As of 21 October 2026 the supported OpenTofu lines are 1.13 (until 1 August 2027) and 1.12 (until 1 February 2027). 1.11 and older are out of security support. This book uses 1.13.2. A production estate on 1.6 is not “stable.” It is unpatched.

Upgrade OpenTofu in its own change:

  1. Read the series changelog, including the breaking notes. For 1.13 those include WinRM provisioner removal, base64gzip byte changes, macOS 13, and the end of 32-bit builds
  2. Bump the pin in mise or .terraform-version and required_version
  3. tofu init and tofu plan in a non-production root
  4. Read the plan. An empty plan is the goal. A surprise replace is a stop
  5. Apply non-production, then production, with the normal gate

Provider upgrades are a different change, even when you do them the same week. -upgrade rewrites the lock file. Commit that diff alone. Major provider versions get a changelog reading and a plan in dev before prod’s pin moves.

Modules follow semver. A root that uses ~> 1.4 will take 1.4.z on the next init without a commit if you also forgot the lock. Commit the lock. Take minors on purpose.

Do not upgrade OpenTofu, the AWS provider, and the network module in one pull request. A bad plan then has three causes.

Debugging and Logging: Tracing Failures in Complex Deployments

tofu plan and the error it prints are the first tool. Provider errors include the API message. Read it before you turn on tracing.

TF_LOG=info is enough for “which resource, which retry.” TF_LOG=trace is enormous and includes values that may be sensitive. Use trace on a throwaway reproduction, write it to a file you delete, and do not paste it into a ticket without reading it.

TF_LOG_PROVIDER=debug on one provider keeps the core quiet. TF_LOG_PATH writes the log to a file instead of mixing it with the plan.

When a plan hangs, it is usually a lock, a provider waiting on a credential, or a data source waiting on an API. Check the lock before you kill the process. Killing an apply can leave the lock and can leave an object half-created. The next plan tells you which. Resume from a clean init rather than from a half-written local .terraform.

Crash logs (crash.log) go with the bug report to the provider or to OpenTofu, not into the repository. Reproduce on 1.13.2 and the provider version from the lock file, or the report is incomplete.

Real-World Case Studies: Production Patterns from Leading Organizations

Three patterns cover most estates this book is willing to call production. They are patterns, not claims about a particular company’s internals.

The single-application team. One repository, two roots (dev and prod), one module for the application, GitHub Actions with OIDC, state in a locked and encrypted bucket the application role cannot reconfigure. Plans on pull requests. Apply on main after one approval. This is the pet example grown up, and it is enough for a product company that is not a cloud provider.

The platform team and many application roots. The platform team publishes modules and Rego. Application teams pin them. Each application has its own state and its own role. The platform team does not apply application changes. A weekly job reports roots whose OpenTofu pin is not 1.13.x. The report is a pull request, not a silent upgrade.

The hybrid estate. Cloud roots as above. An on-premises root per site, with the appliance provider pinned and the token injected per job. The only shared values are outputs: an IP, a BGP peer, a DNS zone id. A change that touches both sides is two pull requests with an order written on them, applied in that order, each with an empty follow-up plan.

What all three refuse: a single state for every environment, a shared admin key, an apply from a laptop as the normal path, and an upgrade with no plan attached.

Summary

  • Split roots by credential and blast radius, including on-premises sites
  • Teams own roots. A platform team owns shared modules and policy
  • Dev, staging, and prod are separate states. Promotion moves a module pin
  • Upgrade OpenTofu and providers in separate commits, on a supported series
  • Log at info first. Trace is a reproduction tool, and it leaks values
  • A production pattern has a locked backend, a role per root, and a plan in the pipeline