Time-boxed, per-namespace, session-recorded access to production Kubernetes clusters — without a bastion host, a shared kubeconfig, or a permanently open VPN. See the Kubernetes access documentation for setup details.

The pager goes off. A payments service in your production cluster is throwing 5xx errors and the on-call SRE needs kubectl access right now. In most orgs, one of three things happens next:

1. The engineer already has cluster-admin on prod because "we needed it for the last incident and never rotated it."

2. Someone digs a shared kubeconfig out of 1Password that hasn't been rotated since the last audit.

3. The engineer VPNs in, hits a bastion host, and inherits network-level access to every cluster in the VPC.

If you work in finance, healthcare, or anywhere else with a serious auditor, none of these are acceptable answers. Standing production access is a finding. Shared credentials are a finding. A bastion that grants blanket network reachability to every cluster is a finding. And "we'll fix it later" stops working the moment someone actually exploits it.

The fix is a break-glass access pattern: no one has production kubectl access by default, access is granted for a specific cluster and namespace for a specific window of time, every session is recorded, and the whole trail is exportable to your SIEM for the audit that's coming whether you like it or not.

What "break-glass access" actually means for Kubernetes

Break-glass access is a pattern borrowed from physical security. The glass stays intact until there's an emergency. When someone breaks it, an alarm goes off, a record is created, and the event is reviewed afterward.

For Kubernetes, that translates to five properties:

Most platform teams already have a Slack channel and a runbook, so the tricky part isn't the policy. The hard part is enforcing it at the network and identity layer so it can't be bypassed under pressure at 3 a.m.

Why the usual approaches fail an audit

Static kubeconfigs

A kubeconfig file with a long-lived token or client cert is standing access, full stop. Even if you distribute it through a secret manager, the moment it's on a laptop it's a credential that can be exfiltrated, copied, or forgotten during offboarding. Rotation is manual and, in practice, rare. There's no session record, just API server audit logs that show what happened but not who was actually at the keyboard.

VPN + bastion host

A VPN drops the user onto a network segment that can reach every cluster's API server in the environment. Once they're on the network, RBAC is the only thing standing between them and the wrong cluster. RBAC misconfigurations are one of the most common Kubernetes findings in real-world security reviews, and a network-level grant means you're one bad RoleBinding away from a bad day.

Auditors also don't love bastions because the bastion itself becomes a shared surface. If the bastion is compromised, so is every cluster behind it.

IdP-federated kubectl (OIDC) without network controls

OIDC-authenticated kubectl is a real improvement: tokens are short-lived and identity-tied. But if your API server is reachable from the office network, a corporate VPN, or (worse) the public internet, you're still exposing the control plane to anyone who can reach it. Identity is verified at the API layer, but the API layer is still discoverable.

Zero Trust says: don't put the resource on a network the user can see in the first place.

The pattern: identity-aware, time-boxed, session-recorded kubectl

A working break-glass pattern for Kubernetes has three layers stacked on top of each other:

1. Network layer — the API server is not reachable from anywhere by default. Not the office. Not the VPN. Not the internet. Access is brokered per-request.

2. Identity layer — every request is authenticated against your IdP, checked for group membership, and (for prod) requires step-up MFA at the moment of connection.

3. Session layer — the connection is proxied through something that can record it, replay it, and export the log to a SIEM.

Twingate's architecture lines up with this pattern because it was built for it. Here's how the pieces map:

• Resources in Twingate are individual API servers, DNS names, or CIDR ranges — not networks. A user granted access to prod-payments-api.internal cannot see prod-billing-api.internal. There is no shared network segment.

• Connectors run inside your VPC or cluster network and make outbound-only connections to the Twingate Controller. No inbound firewall rules, no exposed ports, no bastion.

• Security Policies bind Resources to IdP groups with conditions like device posture, MFA, and geographic location. The policy is evaluated at connection time, not at credential-issue time.

• Identity Firewall (available on paid plans) sits at Layer 7 and captures the actual session for SSH, HTTPS, and Kubernetes traffic — full command recording, exportable to SIEM.

Combined, a break-glass kubectl flow looks like this:

1. SRE requests access to prod-payments cluster via the Twingate admin console or an API call from a chatops bot. The request specifies a duration (say, 60 minutes).

2. An approver — either an automated policy or another on-call — grants the request. The SRE's IdP group is temporarily updated, or a scoped Twingate policy is activated for the window.

3. The SRE authenticates through their IdP with MFA. Twingate issues a signed authorization for that specific Resource.

4. kubectl traffic flows through the Twingate Client, through a Connector inside the cluster network, to the API server. Everything else on the network stays invisible.

5. The Identity Firewall records the session. When the window expires, the policy revokes automatically. No cleanup ticket.

6. The audit log — who, what cluster, what namespace, what commands, when, from where — is streamed to Splunk, Datadog, or whatever your GRC team lives in.

The auditor gets a complete evidence trail. The engineer gets fast access when it matters. Nobody has standing prod credentials on their laptop.

Scoping access to a single namespace

Network-layer access is only half the story. Even if the SRE can only reach the API server for prod-payments, they still need Kubernetes RBAC to enforce namespace boundaries. The pattern here is straightforward:

Create a temporary RoleBinding in the specific namespace, tied to the engineer's IdP identity, at the moment access is granted. Something like this, generated by your break-glass automation and applied for the duration of the window:

Grant an SRE edit permissions in the payments namespace for the length of the incident.

Pair that with a small controller or scheduled job that removes the binding at the expiration timestamp. A few teams use kube-janitor or a custom operator for this; the mechanism matters less than making sure the cleanup is automated and monitored.

The edit ClusterRole is a reasonable starting point for incident response, but if your workload has more specific needs — read-only access to logs, permission to restart a Deployment but not modify Secrets — build a purpose-scoped Role and reference it. The whole point of least privilege is that you don't hand out cluster-admin because it was faster to type.

What the audit trail actually needs to contain

Compliance frameworks vary in their specifics, but for production access to systems handling regulated data (PHI under HIPAA, cardholder data under PCI DSS, financial records under SOX), auditors consistently ask for four things:

• Who — a real human identity, tied to the corporate IdP, not a shared service account or a kubeconfig on a laptop.

• What — the specific resource accessed and, ideally, the commands executed. API server audit logs give you the API calls; session recording gives you the commands the user actually typed.

• When — a timestamp for connection start, connection end, and every action in between.

• Why — a linkable justification. This is usually an incident ID, a change ticket, or an approval record.

A break-glass system that produces all four automatically is dramatically easier to defend in an audit than one where the SRE has to remember to write a Confluence page afterward. The evidence should be a byproduct of the access itself.

Common objections and how to answer them

"What if the break-glass system is down during an incident?"

Legitimate concern. Two mitigations. First, the break-glass system's control plane should be separate from your production infrastructure — Twingate's Controller runs as a multi-tenant SaaS, so it doesn't share a failure domain with your clusters. Second, keep a documented, offline emergency access procedure (a physical vault, a sealed envelope, whatever your policy allows) for the rare case where the primary system is unavailable. Test it quarterly.

"Engineers will hate the extra step during a real incident."

If the request-to-access loop takes 30 seconds and the approval is automated for on-call rotation members, the friction is negligible. The friction people actually hate is the friction of trying to remember which kubeconfig is current or which VPN profile points at prod. A well-designed break-glass flow is usually faster than the ad hoc thing it replaces.

"Session recording feels invasive."

Session recording of production access, in a regulated environment, isn't a surveillance tool — it's an evidence tool. It protects the engineer as much as it protects the company. When something breaks and someone asks "who ran that kubectl delete?", the recording either exonerates the person on-call or identifies the actual cause. Frame it that way and most engineers agree.

What this looks like in practice

The end state, for a platform team that adopts this pattern:

• No engineer's laptop contains a production kubeconfig at rest.

• Cluster API servers have no inbound firewall rules — the only path to them is through a Connector.

• Access requests are automated through chatops or a self-service portal, with automatic approval for on-call rotation members and human approval for everyone else.

• RBAC bindings are created and destroyed by automation, tied to the same access window.

• Every session is recorded, indexed, and streamed to the SIEM.

• The compliance team can pull a report for any auditor asking "show me every production access event in Q3."

None of this is theoretical. The pieces exist — a modern ZTNA product handles the network and session layers, your IdP handles identity, and a small amount of glue code handles the RBAC lifecycle. What most teams are missing is the deliberate decision to stop treating production access as a background convenience and start treating it as a controlled event.

The 3 a.m. incident is coming. The question is whether your access model will help you fix it or become the thing your auditor writes up next quarter.

Closing

For implementation details on Twingate Connectors in Kubernetes, Resource configuration, and Identity Firewall session recording, see the Twingate Kubernetes documentation and the Security Policies reference.

New to Twingate? You can use Twingate for free for up to 5 users, request a personalized demo, or reach out to the team over on the Twingate subreddit.