pacod-gw

pacod-gw sits between your applications and the LLM providers. A Rust core relays or translates every call, keeps provider keys out of your apps, enforces limits, prices each request and can answer a repeated call from its cache. Usage flows into ClickHouse analytics, and everything is managed from a SvelteKit console.

Features

Screenshots

Why a gateway

Every product that talks to a language model ends up re-implementing the same things: storing provider keys, translating between APIs, counting tokens, enforcing limits and explaining the bill. pacod-gw takes all of that out of the application. The application calls one endpoint with a gateway key; the gateway decides which model serves the call, what it may cost and who pays for it.

The first goal is concrete: replace a LiteLLM proxy in front of an application by changing only its base URL and key.

Two modes, one core

The core is one Rust binary with two listeners: the data plane for applications and the management API for the console and scripts. The mode is chosen by URL prefix and allowed per key.

• Proxy mode (/proxy/{provider}/…) relays the provider's own API byte for byte, WebSocket sessions included, and adds authentication, limits and accounting. A template can still set the model and its parameters.
• Managed mode serves OpenAI's /v1/chat/completions and /v1/responses, Anthropic's /v1/messages, Gemini's generateContent and the native /gw/v1/chat. Beside chat it serves embeddings, speech, transcription and translation of audio, image generation and edits, rerank in Cohere's shape (at Cohere, Jina, Voyage and vLLM), OpenAI's Realtime sessions over WebSocket, and the Files and Batches APIs of OpenAI and Anthropic.

In managed mode the caller's dialect and the model's provider decide the route:

Caller speaks	Model is at	What happens
OpenAI dialect	An OpenAI-compatible provider (OpenAI, DeepSeek, GLM, Kimi, Ollama, …)	The caller's bytes are patched; the cost is added to usage.
Anthropic dialect	Anthropic	The conversation reaches Anthropic as sent; the cost comes back in a header.
OpenAI's Responses API	OpenAI or xAI	The caller's bytes are patched, always with store: false; the cost is added to usage.
Any other pair, and the native interface	Any provider	Translated through a provider-neutral model and back, so a call can fail over across dialects.

Stored responses. OpenAI's Responses API can keep a response so that the next call only names it. The gateway never lets a provider keep one: a credential that several workspaces share would let each read the others'. A workspace that switches responses.store on has the gateway keep its responses itself, in PostgreSQL, for 30 days unless set otherwise, and send the earlier turns again on the next call. A conversation can then move between providers: a turn answered by Claude can be continued at OpenAI, and back. OpenAI's Conversations API (/v1/conversations) is kept the same way.

A call, step by step

1. Authenticate. The key is looked up by HMAC in an in-memory snapshot of the configuration.
2. Read. A scanner finds the model and stream flags without parsing the whole body. A call that continues a stored response is given its earlier turns; the workspace's plugins see the conversation.
3. Resolve. A template (@name or x-gw-template) or a granted model id picks the model and its parameters.
4. Look up. In managed mode, where the response cache is on, a kept answer to the same request is returned here, in the caller's dialect, before the call takes a slot or counts against a limit.
5. Admit. The key's concurrency cap and every applicable limit must pass.
6. Send. The stored provider credential replaces the gateway key. In managed mode a 429, a 5xx or a lost connection is retried, or moved to the next model, before the caller gets a byte.
7. Relay and settle. The answer streams back while a tap reads usage from a copy; then cost and counters are updated, the answer is kept if the cache takes it, and a usage event goes to a spool on disk.

The response carries x-gw-request-id, x-gw-delay-ms (the time the gateway itself added), x-gw-model-served, x-gw-route when a routing scheme chose the model, x-gw-cache when the cache looked at the call and, for non-streaming managed calls, x-gw-cost.

Configuration lives in PostgreSQL, but the request path does not wait on it: a change swaps in a new snapshot within about two seconds, and if the database is down, traffic continues on the last one. The exceptions are few and named: a call that names a stored response or an uploaded file reads its record, a kept response is written before it is answered, and the cache waits for Valkey 100 ms by default and goes on without it when it fails.

Organisations, workspaces and keys

An organisation is a customer or a large unit. A workspace is one application, project, environment or team, and owns its keys, templates, limits, settings and usage. Two rules cover every restriction: allow-lists intersect and numeric caps take the minimum.

Provider keys belong to the installation, an organisation or a workspace, and a call uses the nearest: the workspace's own, then its organisation's, then the installation's shared key. A key the gateway already stores can be passed down to an organisation or a workspace without entering it again; the secret never leaves the gateway.

Roles are one file, authz/gw.yaml, from which the core's and the console's checks are generated:

Role	What it covers
Platform admin	Providers, credentials, models, prices, organisations, people
Organisation admin	Its settings, workspaces, members and grants
Workspace admin	Templates, keys, limits, settings and audit mode of one workspace
Auditor	Reads configuration, usage and audit-mode records
Viewer	Reads configuration and usage; no secrets, no audit-mode records

Calls carry attribution — end_user, session_id, trace_id, declared tags, agent_id, task, complexity — which reports, limits and routing read. People sign in to the console with local accounts or, with PGW_AUTH=oidc, through pacod-id: the gateway registers its organisations and workspaces there and follows its feed of roles, and an ended session is refused within two seconds.

Limits and cost

A limit applies to an organisation, workspace, key or end user. It counts tokens, requests or spend over a minute, hour, day, week, month or lifetime, and either refuses calls (429, 402) or only counts and reports. Applications can read a user's quota state from GET /gw/v1/quota. Counters are combined across instances through PostgreSQL about once a second, so a limit can be overshot by about a second of traffic.

Each call is priced when it settles, from a versioned price. Tokens are priced per million, with cache reads and writes, reasoning, audio and image tokens at their own rates where the price has them, batch variants, tiers by input size and off-peak windows. Characters of speech, images, seconds of audio, search units, provider tool calls and requests have unit rates. There is no default price: a model needs a price, or an explicit "free", to be activated.

Model discovery asks each provider for its model list, once a day by default, and proposes the models the gateway lacks; one added from a proposal stays a draft until it has a price. A scheduled price tracker reads LiteLLM's price list or a provider's pricing page and applies changes within a set bound, or proposes them for approval. While it has no sources, the console suggests LiteLLM's list for OpenAI, Anthropic, Gemini, DeepSeek and z.ai.

Response cache

Applications often send the same request twice: a test suite, a classification at temperature 0, a page that asks the model the same question for every visitor. The response cache answers such a call with the answer the model gave before, without asking the provider. It is a plugin, response_cache, installed by default: off in a new installation until an administrator switches it on, and on in the development stack and the demo.

"The same request" means the same model was sent the same body, after the template was applied and the body shaped for the model. A stream and a whole answer share one kept answer, and it is written back in the caller's own dialect. A hit costs nothing: no provider call, cost 0, no key slot and no limit counted. Its usage event records what it saved, and a report shows hits, misses and savings over time, by model and by workspace.

Setting	Default	What it decides
mode	off (a new installation)	enforce answers from the cache; shadow only records would_hit, to see the hit rate first
calls	deterministic	Calls at temperature 0, or all
scope	key	Who shares answers: one API key, one end user of a key, or the whole workspace
ttl_seconds	3600	How long an answer is kept, at most a week
max_answer_bytes	1 MiB	Larger answers are not kept

The platform sets a default, which an organisation or a workspace can replace; an organisation can lock its choice. A caller decides per call with x-gw-cache: off, refresh or on, and the answer says hit, miss, would_hit, refresh or bypass in the same header, with Age on a hit. Only managed calls are cached: not proxy mode, embeddings, speech, token counts, batches or the gateway's own calls. Refusals, cut streams and answers that used a provider's own tools, such as a web search, are never kept.

Answers are kept in Valkey (gw-valkey), shared by every gateway and held in memory only. Each one is sealed with AES-256-GCM under a key derived from the key-encryption key for its workspace, under a name that is a keyed hash of the request: whoever reads Valkey learns neither what was asked nor what was answered, and an answer that does not open is a miss. Answers never cross workspaces. An administrator clears a workspace, an organisation or the whole platform at once, and the audit rule forbid switches the cache off wherever it applies.

Routing and intent

A template can hold a pool of models with tiers, weights and priorities. Without further setup the first model serves and the next takes over on a 429, a 5xx or a lost connection. A routing scheme adds rules on what the caller states (complexity, task, tags, input size, or a CEL expression) and algorithms: priority, weighted, round robin, lowest latency and cheapest, with optional session affinity to keep prompt caches warm. A circuit breaker sets a failing model aside, and every response names the rule that routed it in x-gw-route.

The intent plugin answers "what is this call for?" against a catalogue the organisation or workspace writes, using an LLM, the Jev engine, or a cascade of both. Its verdict can block a call, route it, count it against per-intent limits or flag it for review; it can run in shadow mode first and never changes the conversation.

A new catalogue starts from a built-in template, starter, with a general assistant's intents and rules that flag and block. A new binding starts from one the gateway fills in for its scope: shadow mode, that catalogue, and the cheapest granted model that answers in a JSON schema, or Jev if there is none. The console's try and evaluate panels run the engines on sample texts, first on demonstration data made for the catalogue, so engines can be compared before a binding enforces. See the routing guide and the plugin concept.

Observability and audit

Usage events reach the analytics service in batches from each instance's spool; delivery is at least once, and ClickHouse removes duplicates. The service builds reports on usage, errors, gateway delay, rankings, end users and the response cache, with CSV export, and runs the scheduled jobs. The console reaches it only through the core, which checks the caller's role and forces the scope.

Prometheus, Grafana dashboards, Alertmanager rules and OpenTelemetry traces cover operations; the console shows live figures, provider health and open alerts.

Audit mode keeps complete calls for debugging: the request, every provider attempt and the answers. It is off by default, an organisation can require or forbid it, credentials are removed first, and records expire after 14 days by default. Realtime sessions and the calls about files and batches are recorded where the workspace asks for them too. What the gateway relays unread — a WebSocket session in proxy mode, a batch made in proxy mode — cannot be recorded, and a workspace can refuse it rather than serve it without a record.

Security model

• Provider keys are sealed per credential with AES-256-GCM under a key-encryption key that comes from the environment or a file, never from the database. The API shows only a fingerprint.
• Gateway keys (pgw_…) carry 32 random bytes, are stored as HMAC-SHA256 with a pepper and shown once.
• Tenant separation. The workspace comes from the key, never from the request; reports are scoped by the core to what the caller's roles allow, and cached answers never cross workspaces.
• The management API listens only on pacod-admin-net, an internal network with no route out that only its server-side clients join: the console, the analytics service and the tester, and in the full stack the products that call it or pacod-id. A container on the shared network alone cannot reach it. The console sends a Content-Security-Policy with every page.
• The database role. The core owns its own database and is not PostgreSQL's superuser; pacod-id's database is closed to it.
• Outbound connections to providers and price sources are checked after name resolution: private and link-local addresses are refused unless an operator opens them.
• Prompts stay out of logs, metrics and traces, which carry metadata only. Bodies are kept only where someone chose it: audit mode, stored responses, and the cache's sealed answers.

The threat model also lists what remains: the metrics port has no authentication and carries tenant labels, whoever holds the key-encryption key and can write to Valkey could plant cached answers, and the repository has no CI, so its dependency checks run only when someone runs them.

Deployment and operations

The core runs as all, or split into gateway instances and an admin instance. Instances share one PostgreSQL, and one Valkey for the response cache; nothing else.

Layout	Files
One core	deploy/docker-compose.yml with docker-compose.standalone.yml
Several gateways on one host	plus docker-compose.scale.yml
Gateways on other machines	docker-compose.hub.yml centrally, deploy/node/ on each machine
Kubernetes	deploy/helm/pacod-gw, with PostgreSQL, ClickHouse and Valkey of your own
Minimal gateway	deploy/lite/docker-compose.yml

Like every pacod project, the gateway joins the shared Docker network pacod-net, where each service has a name of its own: the core is pacod-gw, the other services gw-*. The listeners only servers reach sit on a second, internal network, pacod-admin-net. docker-compose.yml holds the gateway's own services, ClickHouse and Valkey among them; docker-compose.standalone.yml adds its copies of the shared components: PostgreSQL, pacod-id, Caddy and monitoring. pacod-demo runs the gateway together with the other pacod projects at demo.pacod.io: it loads only the first file and runs each shared component once.

Calls keep flowing when PostgreSQL, ClickHouse, the analytics service or Valkey is down: the core works from its last configuration, collects events in its spool, and treats every lookup as a miss. Caddy terminates TLS and sends each call to the ready gateway with the fewest calls in flight. The runbook covers secrets, backup and restore, key rotation, upgrades and troubleshooting.

pacod-gw-lite keeps translation, streaming, retries and failover, with per-key limits in memory and one log line per call with its cost. Its response cache is off by default and lives in its own memory (PGW_CACHE_MODE); files and batches need a directory for their record (PGW_OBJECTS_DIR). Its clients share the provider keys and are not kept apart at the provider. It is also pacod-gw on the shared network, so a host runs either it or the full gateway (details).

What has been tested

• OpenAI, Anthropic, Gemini, DeepSeek and GLM are checked with live calls; Kimi, Ollama, xAI and Vertex AI against their documentation and the mock provider only.
• The runbook's first deployment was followed from a clean clone on one machine (Docker Desktop on macOS, 2026-10-05), backup and restore included. The layout with three gateways took 40,000 calls of load through the balancer without a failed call.
• The console's Testing page runs test cases and load tests against a running installation, and shows the reports of pacod-benchmark, which measures what the gateway adds against a baseline straight to the mock. A run whose load agents ran in the gateway's own stack is marked as behaviour, not capacity.
• The design target of 10,000 concurrent streams in proxy mode and 5,000 in managed mode per instance (2 CPUs, 1 GB) is not yet confirmed by a load test.
• Not yet exercised: Let's Encrypt certificates, a Linux host, gateways on physically separate machines with TLS between them, an upgrade between releases and the Helm chart on a real cluster. The minimal gateway has not yet run on a server.
• Not in the first release: MCP and tool governance, and content guardrails.

The architecture page shows how the code is laid out and where it differs from the design concept.

Quick start

The full stack with a mock provider

make test-services
make verify
make dev-up
make smoke-mock
open http://localhost:13000

The minimal gateway

make lite-dev-up
make lite-smoke

Works with

• pacod-id: With PGW_AUTH=oidc, people sign in to the console through pacod-id; the gateway registers its organisations and workspaces there and follows its feed of roles.
• pacod-ap: Every agent's LLM call goes through the gateway, which owns models, prices, keys and usage.
• pacod-cms: AI translation in the studio calls the gateway, so the CMS never holds a provider key.
• pacod-kb: The knowledge base's optional LLM clean-up pass, and its content scan's optional second opinion, are routed, priced and attributed by the gateway.
• pacod-demo: Runs the gateway beside the other products, with the response cache switched on and its mock provider when no provider key is set.
• pacod-benchmark: Measures what the gateway adds, with the gateway's own load agents and against a baseline straight to the mock, and publishes its reports into the console under Platform → Testing → Benchmarks.