TL;DR: An AI gateway inspects prompts and model calls. It doesn't decide which internal APIs, vector stores, or admin tools an agent can actually reach. Here's the checklist for what a gateway leaves open, and where a zero trust access layer fits.

If you've stood up Cloudflare AI Gateway, Kong AI Gateway, Portkey, or LiteLLM in front of your internal AI stack, you already have prompt logging, token accounting, PII redaction, and per-model rate limits. That covers the model call. It does not cover what happens after the model responds and the agent starts reaching into your infrastructure to actually do things.

That second half is where most internal AI deployments quietly turn into a governance problem.

What an AI gateway actually does (and doesn't do)

An AI gateway sits between your application (or your agent) and one or more model providers. Its job is to normalize, observe, and control the model call itself.

Here's roughly what the major gateways cover, based on their current published feature sets:

The gateway is a proxy for the LLM call. The zero trust layer is a proxy for everything the agent touches on your network after that call. Different problems, different tools.

Where gateways stop being enough

The moment you move from "chatbot that answers questions" to "agent that takes actions," the surface area changes. Agents call internal APIs. They query vector databases with embedded customer data. They hit staging environments to run tests. They read from S3 buckets. Some of them spin up other agents.

None of that traffic is meaningfully governed by the gateway, because the gateway sees the prompt going to OpenAI and the completion coming back. It doesn't see the agent's follow-up call to your internal orders-api.internal:8080/refund.

Below is the checklist I keep coming back to when platform teams ask what's missing. It's not exhaustive, but if you can't answer these confidently, you have gaps.

1. Agent-to-service access: who authorized this call?

An agent calling an internal API is a network request. Somewhere, that request either succeeds or fails based on how you've configured the network. In most stacks I've looked at recently, the answer is one of:

• The agent runs inside the VPC and can reach anything else in the VPC. Flat network, implicit trust.

• The agent uses a shared service account whose credentials live in an environment variable.

• The agent has a long-lived API token minted six months ago and never rotated.

None of that is access control. It's connectivity plus credentials, and if any of those creds leak (via prompt injection, log exfiltration, a compromised container), the blast radius is the entire internal network.

What you actually want: every agent-to-resource call is verified against a policy that says "this specific agent identity is allowed to reach this specific resource, right now, under these conditions." Twingate's Controller and Connector enforce this with dual verification — the Client requests authorization, and the Connector independently verifies the ACL before proxying traffic. Neither side implicitly trusts the other.

2. Non-human identity for agents and workloads

Most teams underestimate the identity. Your IdP knows about Alice the engineer. It probably doesn't have a clean model for "the retrieval agent running in the RAG service, version 1.4.2, deployed in us-east-2." That agent needs its own identity, its own scope, and its own audit trail.

Some things to check:

• Can you distinguish an agent's traffic from a human user's traffic at the access layer?

• Can you scope an agent's access to a subset of resources without giving it a shared credential?

• If an agent is compromised, can you revoke its access without rotating credentials across every other service?

A gateway doesn't answer any of these because a gateway doesn't know about your internal DNS, your internal APIs, or your internal resource inventory. It only knows about model endpoints.

3. Vector databases and RAG stores

Your interesting data lives in your vector DB. Embeddings of customer support tickets, internal Confluence pages, source code, contracts. If an agent can query it, an attacker who compromises the agent can query it.

Two questions worth being blunt about:

• Is your Pinecone, Weaviate, Qdrant, or pgvector instance reachable from anything other than the specific agent that needs it?

• If yes, what stops a compromised agent (or a malicious prompt with tool-calling) from vacuuming the entire index?

The gateway can filter what goes into the model. It can't stop the agent's own retrieval calls, which happen out-of-band from the model conversation. This is a network-level problem, and it needs a network-level control.

4. Admin tools, dashboards, and staging environments

Coding agents in particular love to reach for the admin console. They'll happily curl the deployment API, hit the internal metrics dashboard, or try to SSH into a staging box if you give them a shell tool. Most of these targets were never designed with agent access in mind.

The pragmatic move is to treat every internal admin surface as a resource that requires explicit authorization, whether the caller is human or not. With outbound-only Connectors, there's no inbound firewall rule to leave open, no public hostname for the agent (or an attacker) to discover, and no lateral network path to enumerate. The agent gets exactly the resources you granted, and nothing else exists from its perspective.

5. Session-level audit for agent actions

Gateway logs will tell you what the model said. They won't tell you what the agent did with that response. When something goes wrong — a bad refund, a wrong config change, a leaked document — you want a full session record of the agent's network activity: which resource it called, from which identity, at what time, over which session.

This is where the two layers really do complement each other:

• The gateway gives you the prompt-and-completion trail.

• The zero trust access layer gives you the resource-access trail, correlatable to the same agent identity.

Twingate Identity Firewall adds session recording for SSH, which matters for coding agents and any agent that shells into machines. Combined with the gateway's prompt logs, you get end-to-end reconstruction of an agent's actions, not just its conversations.

6. Kill-switch and revocation

If an agent goes off the rails at 3am, what actually stops it?

Rate limits at the gateway slow the model calls, which is useful. But if the agent already has a valid API token and network reachability to your internal services, the model isn't the bottleneck. You need a way to cut network access to specific resources for a specific identity, immediately, without redeploying anything.

This is a mundane capability that turns out to matter a lot in incident response. Policy-driven revocation at the access layer is the difference between "we contained it in two minutes" and "we spent the morning rotating credentials across seventeen services."

The mental model

The clean way to think about it:

• The AI gateway governs the conversation: what the agent says to the model and what the model says back.

• The zero trust access layer governs the actions: what the agent can actually touch on your network as a result.

You need both. The gateway without the access layer means you have great prompt observability and no idea what your agents are doing with the answers. The access layer without the gateway means you have solid network-level control but no visibility into prompt injection or model misuse.

Neither replaces the other. Anyone selling you "AI security" as a single-vendor story is glossing over one of the two.

A pragmatic sequence

If you're currently at "we have a gateway, what next," a reasonable order of operations:

1. Inventory your agents' downstream calls. Every internal API, database, S3 bucket, admin tool, and third-party service any agent touches. If you can't enumerate them, start there.

2. Give each agent its own identity. Not a shared service account. Not an env-var token. A real identity your access layer can reason about.

3. Move internal resources behind a zero trust access layer. No inbound ports, no public hostnames, no shared VPC assumptions. Agents reach resources by policy, not by network position.

4. Correlate gateway logs with access logs. Same agent ID, same session, both trails queryable together.

5. Practice revocation. Actually cut an agent's access in a game day. If it takes more than a few minutes, fix that before you need it in production.
   
   

Closing

For a deeper look at how Twingate's architecture handles service identities, outbound-only Connectors, and session-level audit for internal resources, the Twingate documentation has the specifics on Connectors, Security Policies, and the Identity Firewall.

New to Twingate? You can use Twingate for free for up to 5 users, request a personalized demo, or reach out to the team over on the Twingate subreddit.