The AI gateway of the PACOD platform is pacod-gw. It sits between applications and the language-model providers: it keeps the provider keys, enforces limits, prices every call, records the usage and, where an administrator allows it, answers a repeated call from its cache. This post explains why it exists, how its core handles a call, and how to try it on your own machine.

Why a gateway

Every product that talks to a language model ends up re-implementing the same things: storing provider keys, translating between APIs, counting tokens, enforcing limits and explaining the bill. pacod-gw takes all of that out of the application. The application calls one endpoint with a gateway key. The gateway decides which model serves the call, what it may cost and who pays for it.

The first goal is concrete: replace a LiteLLM proxy in front of an application by changing only its base URL and key.

In PACOD, every product works this way. The agent platform, the CMS studio's AI features and the knowledge base's optional clean-up pass all call the gateway, and none of them holds a provider's key.

Two modes, one core

The core is one Rust binary with two listeners: the data plane for applications, and the management API for the console and scripts. The mode is chosen by URL prefix and allowed per key.

  • Proxy mode (/proxy/{provider}/…) relays the provider's own API byte for byte, WebSocket sessions included, and adds authentication, limits and accounting.
  • Managed mode serves OpenAI's /v1/chat/completions and /v1/responses, Anthropic's /v1/messages, Gemini's generateContent and a native /gw/v1/chat. Around chat it also serves embeddings, speech, transcription, images, rerank, OpenAI's Realtime sessions and the Files and Batches APIs of OpenAI and Anthropic.

A caller may speak one dialect to a model at another provider. The call is then translated through a provider-neutral model and back, so it can also fail over across dialects: a call in OpenAI's shape can move on to Claude when its first model fails. For an application, a call looks like any OpenAI call; only the address and the key are the gateway's:

curl http://localhost:18080/v1/chat/completions \
  -H "Authorization: Bearer $PGW_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "@assistant", "temperature": 0,
       "messages": [{"role": "user", "content": "Summarise our refund policy."}]}'

Here @assistant is a template of the workspace: it names the models that may serve the call, their parameters and the order in which they take over. The key is looked up by HMAC in an in-memory snapshot of the configuration. The configuration lives in PostgreSQL, but the request path does not wait on it: a change swaps in a new snapshot within about two seconds, and if the database is down, traffic continues on the last one. In managed mode a 429, a 5xx or a lost connection is retried, or moved to the next model, before the caller gets a byte. Every response names its request id, the model that served it and the time the gateway itself added.

The pacod-gw request log with filters above and rows of calls, each with time, result, model, workspace, tokens, cost and duration.
Every call in the request log: its result, the model that served it, its tokens and cost, and the gateway's own share of the time.

The screens in this post come from a local pacod-demo after a pacod-benchmark run of 25,280 calls against the gateway's mock provider. The benchmark's model is marked free, which is why the costs read 0.00 USD.

Keys, limits and cost

An organisation is a customer or a large unit. A workspace is one application, project, environment or team, and owns its keys, templates, limits and usage. Two rules cover every restriction: allow-lists intersect, and numeric caps take the minimum.

Provider keys are sealed per credential with AES-256-GCM, under a key-encryption key that comes from the environment or a file, never from the database. Gateway keys are stored as HMACs and shown once. The management API listens only on an internal network without a route out, which only its server-side clients join; a container on the shared network alone cannot reach it.

A limit counts tokens, requests or spend over a window, from a minute to a lifetime, for an organisation, a workspace, a key or an end user. It either refuses calls or only counts them. Each call is priced when it settles, from a versioned price: tokens per million with their cache, reasoning, audio and image rates, or units such as images, seconds of audio and search units. There is no default price: a model needs a price, or an explicit "free", before it can be activated.

Usage events flow into an analytics service on ClickHouse, which builds the reports. Prometheus, Grafana and OpenTelemetry cover operations. Prompts stay out of logs, metrics and traces, which carry metadata only. A template can also hold a pool of models, and a routing scheme picks among them by rules and algorithms such as priority, lowest latency or cheapest.

Answers it already has

Applications often ask the same thing twice: a test suite run again, a classification at temperature 0, a page that puts the same question to the model for every visitor. The gateway can now answer such a call itself. The response cache is a plugin, installed by default: when the same model would be sent the same request, after the template and the shaping for that model, the answer it gave before comes back, in the caller's own dialect, whole or as a stream.

A hit costs nothing. It makes no provider call, takes no slot of the key's concurrency cap, counts against no limit and is priced at 0, while its usage event records what it saved. By default only calls at temperature 0 are cached, and each API key keeps its own answers for an hour; a binding can share them per end user or across a workspace, never between workspaces. The caller can always say otherwise per call with x-gw-cache: off or refresh, and every answer the cache looked at says hit, miss or bypass in the same header.

The pacod-gw response cache screen with the store, the answers kept, the memory used and the instances sharing it.
The response cache in the demo: Valkey shared by every gateway, the platform's binding on for deterministic calls, one hour, per key. No answers kept yet, because load agents send x-gw-cache: off unless told otherwise.

The answers live in Valkey, shared by every gateway and kept in memory only. Each one is sealed with AES-256-GCM under a key derived for its workspace, under a name that is a keyed hash of the request, so whoever reads Valkey learns neither the question nor the answer. A lookup waits 100 ms by default, and a store in trouble is skipped: the cache never fails a call. Proxy mode, embeddings, speech and batches are not cached, and the audit rule forbid switches the cache off wherever it applies.

The cache is off in a new installation until an administrator switches it on; shadow mode first shows the hit rate it would have had, without answering a single call from it.

What the gateway adds

pacod-benchmark measures exactly that. A suite is one configuration and the settings it varies, and a run makes every combination on a gateway. Each combination also gets a baseline that calls the mock provider directly, so the gateway's overhead is what is left once the baseline is taken away. A run's report can be published into the console, under Platform → Testing → Benchmarks, next to the gateway it measured.

The run behind these screens compared managed and proxy mode at 10, 50 and 100 calls at a time: nine cells, three of them baselines, 25,280 calls, all of them successful. In the cells shown below, the gateway's own time stayed at or below 0.20 ms at the 99th percentile. The difference to the baseline is lost in the noise of one shared machine: at the 99th percentile some cells even came out faster than their baseline. That is why the report calls its figures behaviour, not capacity: the load agent ran in the gateway's own stack and shared the mock with other load. What the gateway really adds walks through a run and how to read it.

The dark run page of pacod-benchmark with its figures and a table of nine cells, each with calls per second, latencies and the gateway's overhead.
A run of managed against proxy mode at 10, 50 and 100 calls at a time, on one machine against the mock: these figures show behaviour, not capacity.

Try it

The full stack runs in containers with a mock provider:

make test-services
make verify
make dev-up
make smoke-mock
open http://localhost:13000

The last line opens the console; the development stack switches the response cache on. There is also a minimal gateway, pacod-gw-lite: the same data plane in one container, with no database and no console, configured by one .env file and logging every call with its cost. Its cache, off by default, lives in its own memory.

make lite-dev-up
make lite-smoke

OpenAI, Anthropic, Gemini, DeepSeek and GLM have been checked with live calls. Kimi, Ollama, xAI and Vertex AI have been checked only against their documentation and the mock provider's emulations. MCP and tool governance and content guardrails are not in the first release.

The product page has the full walk through a call, the roles, the cache's settings and the security model. The code and the runbook are in the repository, and the gateway console runs in the live demo.