TL;DR: Twingate runs 50 Kubernetes clusters across 22 regions and ships infrastructure using one OCI artifact per commit—no rebuilds between environments. Promotion is just a tag move through four stages, auto-advancing on green builds up to one manual gate before prod. Since clusters stay private, CI reaches all 50 via Twingate's own headless client. Reverts become a tag flip, not a rebuild.

We run a lot of Kubernetes at Twingate.

As of this week, the count is 50 clusters across five environments, most of them Relay clusters routing customer traffic across roughly 22 regions on GCP and DigitalOcean. Every continent except Antarctica.

Shipping infrastructure changes to all of them without rebuilding artifacts on the way to production, without 50 different deploy pipelines, and without any of those clusters being reachable from the internet, was a significant challenge for the team.

This post will take you through how the team navigated that problem, and the pipeline we landed on.

But before any of that: the constraint that shaped every decision below is reliability.

Twingate is in our customer's network path. If we have an infra change that breaks production, it isn't a "DevOps inconvenience." Suddenly customer traffic that doesn't reach where it's supposed to.

"It's just infra, what could go wrong?" is exactly the mindset we built the pipeline to make impossible. Every choice that follows — single artifact, tag-based promotion, canary stages, identical bytes from staging to prod — exists so that by the time a change reaches a production cluster, it's been observed working in three other environments first, and there's no rebuild between them that could quietly change behavior.

The short version: one Git repo, one OCI artifact per commit, and promotion is a tag move. That's the whole pipeline. Everything else is plumbing.

The shape of the problem

A typical "GitOps shop" tutorial assumes you have one cluster, maybe two. You install Flux, point it at a repo, and you're done. At our scale, three new problems appear.

1. Same code, different cluster

Pointing every cluster at the same Git path means every cluster gets the same manifests. This is wrong, because app-us-east4-prod-s1 and relay-do-9-syd1-prod have nothing in common except the word "prod." You need per-cluster context (region, shard, internal domain, IDs) injected at reconcile time without forking your manifest tree 50 ways.

2. Promotion that's actually safe.

"Deploy to staging, then prod," is easy to say. The trap is rebuilding artifacts at each stage, at which point staging and prod are different artifacts that happen to come from the same commit. The bug you ship is whichever bug you didn't test. We wanted the artifact CI builds on main to be the exact artifact that lands in production weeks later.

3. No public API servers

None of our clusters expose their Kubernetes API to the internet. Not behind an allowlist. Not behind a bastion. CI still has to reach all 50 of them to verify deploys actually applied. We solve that with our own product (more on that toward the end) but the pipeline has to be designed assuming every API server is private from day one.

How do we address each of these issues? Let’s dig in.

One repo, one OCI artifact

Our infrastructure repo (devops-gitops) is a single Kustomize tree:

infrastructure/
  controllers/      # base + per-env overlays for in-cluster controllers
  config/           # Twingate resources, ingresses, secrets
  monitoring/       # Prometheus, Grafana, alerting stack
  singleton/        # things that exist exactly once across the fleet
clusters/
  prod/<cluster-name>/
  stg/<cluster-name>/
  dev/<cluster-name>

CI doesn't push manifests to clusters. CI builds one OCI artifact per commit:

flux push artifact \\
  oci://us-docker.pkg.dev/twingate-prod/devops-gitops/infra:<sha> \\
  -f /tmp/oci-infra-build/ \\
  --revision="main@sha1:$(git rev-parse HEAD)"

That artifact is the entire infrastructure tree, frozen at one commit. It's the unit of deploy, and the only unit of deploy. Every cluster pulls from the same artifact via an OCIRepository:

apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata:
  name: infra
spec:
  url: oci://us-docker.pkg.dev/twingate-prod/devops-gitops/infra
  ref:
    tag

Note the tag: prod-latest, not a SHA. That's the promotion handle. (We'll come back to it).

The per-cluster context lives in a flux-config ConfigMap that Terraform writes into the cluster at bootstrap time: env, shard, region, internal domain, Twingate service-account IDs. Every Flux Kustomization references it via postBuild.substituteFrom:

postBuild:
  substituteFrom:
    - kind: ConfigMap
      name

So the manifest tree is shared, but each cluster substitutes its own values into it at apply time.

One artifact, 50 instantiations.

Bootstrap is a Terraform module

Onboarding a new cluster is one Terraform apply:

1. Provision the GKE/DOKS cluster

2. Install Flux Operator via the bootstrap module

3. Inject the per-cluster flux-config ConfigMap.

A small FluxInstance resource declares which components to run and where to sync from, and the operator takes it from there. That’s a new cluster, reconciled and monitored, in under 20 minutes.

Promotion is a tag move, not a rebuild

This is the centerpiece. The artifact CI publishes on main is the artifact that eventually lands in production weeks later, bit-for-bit. Promotion is just retagging:

Four floating tags. Each one points at a SHA-tagged artifact in the registry. On the mainline path, each promotion fires automatically when the previous stage goes green — the only manual step is the final gate into prod.

To advance, an engineer runs:

make deploy ENV=NON-PROD COMMIT=<sha>

That moves the NON-PROD Git tag, CircleCI picks it up, and make flux-tag retags the artifact in GAR. The CI build then fans out a job per cluster in scope.

For NON-PROD that's every non-prod app and Relay cluster, and each job does the same thing:

• Open a tunnel into the cluster

• Tell Flux to reconcile

• Watch every Kustomization sourced from OCIRepository/infra until they all report the expected commit_sha

If every cluster converges, the build is green. If any cluster fails to reconcile or times out, the build fails and the on-call channel hears about it. That's the gate that makes "promote to the next stage" a meaningful signal rather than a hopeful one.

The first tag in that chain, non-prod-canary-latest, doubles as the developer testing slot. Feature branches build their own OCI artifact, and an engineer working on an infra change can promote it onto non-prod-canary-latest to exercise it against a real cluster before opening a PR.

By the time the change is merged to main, it’s already reconciled cleanly on a non-prod-canary cluster. The promotion chain is the same shape whether you're testing your branch or shipping to prod, they just have different starting points.

The chain runs itself, up to prod

On the mainline path there's no make deploy between stages. Once a PR merges to main, the pipeline promotes itself:

1. Merge to main → CI builds the OCI artifact and moves the NON-PROD tag automatically.

2. Every non-prod cluster reconciles. All green → the PROD-CANARY tag moves automatically.

3. The prod-canary clusters reconcile. All green → the pipeline parks on a manual approval.

4. A human approves → the PROD tag moves, and prod reconciles.

The same fan-out reconcile gate that makes a promotion meaningful — every cluster converged on the expected commit — is what decides whether the next stage fires. Green isn't a signal a person reads off a dashboard and acts on; it's the trigger itself.

make deploy still exists, but for out-of-band promotions and for the non-prod-canary testing slot — it's not part of the merge-to-prod flow. And the canary→prod boundary is enforced, not trusted: PROD can only promote a commit that has already shipped to PROD-CANARY (the deploy refuses otherwise, FORCE=1 for emergencies). So "it ran in canary first" is a precondition the tooling checks, not a convention we hope holds.

The one stage that stays manual is PROD. We treat the prod tag move the way other teams treat a deploy button: a small, deliberate human action with full context on what's about to ship. Everything up to that point is automation. That last step is judgment.

What's still ahead: auto-revert on red

The half we haven't wired up is the inverse of auto-promote. Today a failed reconcile fails the build and pages the on-call channel — but rolling back is still a deliberate make revert. The pieces are already in place: every promotion saves the previous tag to *-previous, so a revert is just a tag move measured in seconds. What's left is making a red build trigger that revert automatically and halt the chain. The same gate that decides whether the next promotion happens becomes the gate that decides whether to roll back the last one.

A few things fall out of this design that we didn't fully appreciate up front:

• By the time a change hits production, it has been observed working in three other environments, exactly as it will run in prod. No rebuild, no "the staging image and the prod image are different in some subtle way." Whatever passed monitoring in non-prod is bit-for-bit what enters prod-canary. That's the reliability backbone of the whole pipeline.

• Revert is cheap. Each promotion saves the previous -latest to -previous. make revert ENV=PROD flips the tags back. No rebuild, no scrambling for "what was deployed before this." When something does slip through and a cluster starts misbehaving, rolling back is a tag move measured in seconds, not in a CI rerun.

• "Ready to deploy" and "deployed" are different questions, with different answers. Does the artifact exist in the registry? It's ready. Which tag points at it? That's where it lives. Neither requires reading a CI log.

• Drift between environments isn't possible. Two environments either point at the same digest or they don't. If they do, they're running the same code. There's no, "Well staging is on commit X but the rebuild produced a slightly different image."

This is the property that lets us run the pipeline at this scale with the team we have. Once you accept that promotion is just metadata, a lot of the operational anxiety goes away.

Portals: Making the pipeline observable

Reading 50 cluster states from kubectl is a non-starter. We built three small portals over the last two sprints, each tied to a different question about the pipeline.

Per-cluster portal

Runs on every cluster, exposes that cluster's Flux state, recent reconciles, currently-running version. If you know the cluster, you go straight to portal-<cluster>.int.

Global (fleet) portal

One per environment. Aggregates every per-cluster portal in that env into a single fleet view: which artifact each cluster is on, live alerts, stale reconciles. This is the on-call landing page.

CI / release portal

One global instance. Tracks each artifact through the whole promotion chain, built on main, then non-prod-canary-latest, non-prod-latest, prod-canary-latest, prod-latest. One screen for "what's about to ship" and "what's already live."

If you remember the centerpiece — promotion is metadata, not a rebuild — the CI portal is essentially a reader for that metadata.

The portals are deliberately small. They exist because the pipeline above is so artifact-and-tag-driven that just displaying which tag points where is most of what an engineer needs to know.

How CI reaches 50 private clusters

The one piece I've been hand-waving: how does CI verify that all 50 clusters reconciled to the expected commit, when none of them have a public API server?

We use Twingate.

Specifically, the CircleCI job starts the Twingate Headless Client as a background process inside the executor, authenticates as a service account with access to that cluster, and kubectl / flux talk through an HTTP proxy on 127.0.0.1:9999:

- twingate/start-twingate
- twingate/sync-kubeconfig:
    resource: "{{ index $c "env" }}-k8s-{{ $name }}"
- run:
    name: Reconcile {{ $name }}
    environment:
      HTTPS_PROXY: <http://127.0.0.1:9999>
    command: |
      flux reconcile source oci infra -n flux-system
      # ...wait for all kustomizations to converge on EXPECTED_SHA

There's no allowlist for CI IPs to maintain, no bastion to pet, no NLB sitting in a security review. The cluster API stays private, CI gets a path to it scoped to the service account, and the same pattern shows up in the CI portal aggregator (which has to read state from clusters in every environment). Same primitive, different consumer.

It happens to be our own product, which is convenient and a little funny. But the design point isn't "we use Twingate," it's "the pipeline above doesn't depend on any cluster being reachable from the internet, ever."

Whatever lets you reach private API servers without inventing exceptions to that rule will work. For us, it's headless Clients.

Same pipeline, beyond infrastructure

Everything in this post is about shipping infrastructure: controllers, monitoring, ingress, secrets, the things our platform team owns. But none of the design above is infra-specific. The shape — one OCI artifact per commit, four floating tags, a fan-out reconcile build that gates the next promotion, automatic revert via *-previous — works just as well for product code.

The direction we're moving is to put every service we own behind the same rails. Each service:

• Publishes its own OCI artifact

• Gets its own four-tag promotion chain

• Rides the same auto-promote-on-green / auto-revert-on-red pipeline up to a manual prod gate

The platform team doesn't want to be the only group with cheap reverts and "no rebuild between staging and prod" guarantees. The pipeline scales sideways at least as well as it scales across clusters. Plus, once one team is using it, every new service gets safer-by-default deploys without designing a new release process from scratch.

Things we'd do the same way again

A few principles fell out of building all this that we'd repeat for the next platform:

• Reliability is the brief. Treat infra changes the way you treat product code shipping to production. "It's just config" is how you take customer traffic down at 2am.

• One artifact, retagged. Don't rebuild on promotion. The bug you ship is whichever bug you tested.

• Per-cluster context as ConfigMap, not as forks. Substitution at reconcile time keeps your manifest tree honest.

• Bootstrap is code, not a wiki page. If onboarding a cluster takes more than one Terraform apply, your bootstrap is wrong.

• Promotion is metadata. If you can describe a deploy as "this floating tag now points at that digest," you can also describe a revert, a diff, and a deploy-readiness check the same way.

• Don't expose what you don't have to. Private API servers shrink the surface of every other decision in the pipeline. Solve the access problem once, scale it everywhere.
  
  

Closing

Most of this isn't novel. This is just GitOps, just OCI, just tag-based promotion.

The reason it works at our scale is that none of the moving parts try to be clever. One repo. One artifact per commit. One ConfigMap per cluster. Four floating tags. A small portal that shows which tag is where.

The fact that the network underneath happens to be our own product is the part I keep getting mileage out of internally… But the pipeline would work regardless.

If you're solving a similar problem, we'd love to compare notes. If you want to it, Twingate is free for up to five users and you can check out our docs for more information:

• Twingate Headless Clients

• Example CI/CD Configurations

• Twingate Kubernetes Operator Quick Start Guide

New to Twingate? You can try it for free or book a demo.