pacod-kb

pacod-kb ingests files and web pages, converts them to canonical Markdown, indexes them for hybrid semantic and keyword search and keeps every version. It scans each document for prompt injection before anything is indexed, and hands downstream models passages they can cite, under a documented metadata contract. It runs on CPU, and searches stay responsive while documents are being ingested.

Features

Screenshots

Why it exists

Most of the work in retrieval-augmented generation happens before a model reads a word: getting documents in, keeping them clean, finding the right passage, and telling the model where it came from. pacod-kb is that part, built as a service of its own with an HTTP API, a web UI and a CLI.

Three demands shaped it:

• Answers must be citable. Every passage carries its heading path and byte offsets into a stable Markdown body, so a consumer can quote it, link it and fetch its context.
• Sources change. Nothing is overwritten: a new version supersedes the old one, which stays searchable by date.
• Ingested text is untrusted input to a model. It is scanned for prompt injection before it is indexed and fenced as data on the way out.

A document, step by step

A document takes the same path whether it was uploaded, dropped into the dropzone folder, fetched from a URL or re-fetched from a watched web source:

1. Acquire. The SHA-256 of the source bytes is the document's id; a known id is skipped, so the same file is never ingested twice.
2. Parse. PDF through docling, with optional OCR for scanned pages (with tesseract installed, two engines are compared page by page), DOCX, HTML, Markdown, text and JSON, whose keys become headings. The result is Markdown with YAML front matter.
3. Check the language. English, German, Bulgarian, Russian, Serbian, Macedonian, Croatian and Slovenian are accepted, other languages refused before anything is written. Text whose language cannot be detected is accepted as unknown, with a warning.
4. Scan the deterministic parse for prompt injection and spam. This happens before any model sees the text: a converter could follow injected instructions, and conversion would strip the zero-width characters and hidden markup the scan looks for.
5. Convert, optionally. An LLM pass through pacod-gw tidies the Markdown; a PDF of up to 20 MiB goes to the model as a file, anything else as extracted text. A quarantined document is never sent. A cut-off, refused or empty answer keeps the deterministic parse, and a converted body is scanned a second time.
6. Clean and structure. A noise filter removes page numbers, running headers, cookie notices and blocks of navigation links. Headings are promoted in legal codes, regulations and directives (eight languages) and in papers; an article from a URL also gets its author, date and publication.
7. Store document.md, a manifest with its lifecycle log, and a snapshot of the source: in folders under KB_ROOT, or in PostgreSQL.
8. Index, only when the document's status is ok. Heading-aware chunks of up to 500 tokens are embedded with BGE-M3, each with a [Title > Heading > Subheading] breadcrumb in front (a table also gets the paragraph before it), into the project's Qdrant collection and keyword index. Chunks that are unchanged since an earlier version keep their vectors.

Ingests from the UI or the API run as jobs, one after another on a worker of their own, never on the threads that answer searches. In database mode jobs are kept in PostgreSQL, and a restart marks those it interrupted as failed.

Architecture

pacod-kb is two services, kb-pipeline and kb-ui, plus the models and stores they use:

Part	Role
Pipeline API (kb-pipeline, :8000)	Ingest, index, search, jobs, settings; the only part that calls the AI gateway
Web UI (kb-ui, :8080)	FastAPI and HTMX; sign-in through pacod-id, sessions, members, the pages; calls the pipeline with an API key
BGE-M3	1024-dimensional embeddings, in the pipeline's own process or from pacod-ap's embeddings service
Qdrant	The vectors: embedded in the pipeline, or a server that other products may share
Keyword index	SQLite FTS5 (BM25) with filesystem storage, PostgreSQL full-text search with database storage
Documents and ledger	Folders under KB_ROOT or PostgreSQL; the ledger holds settings, prompts, AI usage and cost

The split is deliberate. The pipeline parses untrusted documents and fetches arbitrary URLs, so it receives none of pacod-id's secrets; the UI never sees the gateway key. Each container gets an explicit list of environment variables, and a test checks that list.

Every container joins the shared Docker network pacod-net, where each can reach the others. pacod-id's internal listener (tokens, signing keys, the authorization feed, the admin API) is the exception: it answers only on pacod-admin-net, an internal network with no route out, as pacod-id-internal:8071. kb-ui joins that network; the pipeline does not, and cannot reach it.

Who acted. Every call from the UI carries X-KB-Actor, the signed-in person's id. The pipeline trusts it only together with the API key, records it on jobs, settings changes and usage, and passes it to the gateway. When pacod-id deletes an account, kb ends its sessions, replaces that id with deleted-user everywhere, and only then confirms the deletion.

Search, and what a model receives

A query runs against Qdrant and the keyword index, and the two rankings are fused with reciprocal rank fusion: each list adds 1/(60 + rank) for every passage it found. Date and history filters apply inside both searches, so a hundred superseded passages cannot crowd out the one that is valid.

An RRF score orders one result list and means nothing across queries. A score of 0.0164 is 1/61: first place in one list and absent from the other. That is what most natural questions get by default, because the keyword half (KEYWORD_QUERY_MODE=legacy) requires every word of the query to match. The analysed modes (or, and_or) drop stopwords and strip inflections instead; they stay off until an evaluation shows they help.

An optional cross-encoder, BAAI/bge-reranker-v2-m3, reorders the best ten candidates and gives each a relevance between 0 and 1 that is comparable across queries, so min_score can drop non-answers. It loads in the background when switched on, and a request never loads it.

POST /query takes the question and its filters:

{"query": "maximum fine for a data breach", "top_k": 5, "as_of": "2025-12-30"}

The response is {"results": [...], "reranked": false, "score_kind": "rrf"}. Each result carries:

Field	What it is for
chunk_id	A stable id that encodes the heading structure, not a sequence number
source_id	The document's id: the SHA-256 of its source bytes
heading_path	A breadcrumb such as ["Chapter 1", "§ 5"], for citations and section filters
byte_start, byte_end	UTF-8 offsets into the stored Markdown body
valid_until	When this version was superseded, or null while it is current
text	The embedded text, fenced as untrusted data
score, rrf_score, rerank_score	The ordering score and its parts

The offsets are bytes, not characters, which matters for Cyrillic text:

body_bytes = body.encode("utf-8")
passage = body_bytes[result["byte_start"]:result["byte_end"]].decode("utf-8")

Each document also has a manifest: title, source, document type, status, project, the parser or gateway model that produced the body, and a lifecycle log (ingested, superseded, reinstated, security_flagged, review_flagged, llm_usage).

Versions and projects

A new version ingested with --supersedes, or a watched source that has changed, marks the old one with valid_until, in four places at once: the front matter, the manifest, the keyword index and the vector payloads. Searches return current passages by default; as_of returns what was valid on a given day; include_historical returns everything. If a watched page reverts, the earlier version is reinstated. In filesystem mode, with GIT_AUTOCOMMIT=true, every ingest is also committed to a git repository in KB_ROOT.

A project is an isolated collection: its own Qdrant collection and its own keyword rows. A search sees one project, a rebuild drops and refills one project's index (--all-projects rebuilds each into its own collection), and switching the active project is refused while a job is queued or running.

Search, measured

The defaults come from an evaluation harness: 14 documents in six languages (GDPR, the Law Enforcement Directive, four constitutions, Wikipedia articles) and 92 hand-written queries: paraphrases, exact terms, cross-lingual, native-language and unanswerable ones. A result counts when it comes from the labelled section, so the labels survive changes to the chunker. The numbers were measured on a development machine's CPU (Apple M5, four threads), which the docs stress are not production figures:

Measure	Result
Right section among the fused candidates	every answerable query
Right section in the top five, no reranker	98.7 % (R@1 0.83, about 50 ms)
With the reranker, ten candidates	R@1 0.91, about 2 s per query
A search while documents are ingested (p50)	24.3 s before; 0.41 s now, with jobs run one after another
Ingest embedding, length-sorted batches	about 2.2× faster, vectors unchanged

The reranker's score separates answers from non-answers: min_score 0.1 drops 14 of 15 unanswerable queries and 2 of 70 correct top answers, where RRF scores could not separate them at all. It is still off by default: its gain fell just short of the bar on the held-out half, and its latency has to be confirmed on the production CPU.

The analysed keyword modes helped native-language and exact-term queries but cost more on cross-lingual and paraphrased ones, and lost on the held-out half: legacy stays the default.

Security model

Content security. The scan works on a normalised copy of the text (Unicode-normalised, lower-cased, letter-spaced words joined up) and applies rule tables: injection phrases in English, German, Russian and Bulgarian, bidirectional-override and zero-width tricks, base64 and hex blobs (decoded and scanned again), hidden markup, and spam measures such as link density and repeated lines. Each rule has a severity. Only an unambiguous instruction override ("ignore previous instructions"), a bidirectional-override character or an instruction hidden in base64 is high; phrases that also occur in technical prose, such as "enable developer mode", are low, so a single hit cannot block a document. Tests check that long laws, the GDPR among them, come out clean. The verdict is one of three:

• block: a high-severity finding or a high threat score. The ingest fails and nothing is stored.
• quarantine: the document is kept with status review, never converted and never indexed. Spam alone never leads to a block.
• clean: the document is indexed.

An optional LLM second opinion through the gateway (off by default) is asked only when the rules did not block; it has 15 seconds and fails open. Known-safe material can skip the scan with kb ingest --trusted. On the way out, /query wraps each passage in an untrusted-data fence.

Fetching. URL ingests take http and https only, follow at most five redirects and check every hop, and stop at 50 MB or 180 seconds; a URL ingest never reads a local file. Two defaults suit a laptop rather than a server: set WEB_FETCH_ALLOW_PRIVATE=false to refuse addresses that resolve to private networks (a guard against server-side request forgery), and ALLOW_INSECURE_TLS=false to stop the retry without certificate verification after a TLS error.

People. The UI signs people in through pacod-id with OIDC and PKCE, at pacod-id's own pages in kb's branding. It verifies the access token, keeps the tokens in an encrypted server-side session, and gives the browser a cookie that carries only a random secret. kb's roles are a manifest that the UI pushes to pacod-id at every start; a person's role then comes from pacod-id's authorization feed. Without a kb role, a person sees a "no access" page; if the UI's copy of the feed is more than 60 seconds old, pages answer 503 rather than guess. Sign-out ends the session at pacod-id, and a session revoked there ends in kb too.

Role	Can
viewer	Read and search
editor	Ingest, re-index and delete documents; run jobs; manage projects
admin	AI Management, prompts, the runtime switches, members

Admins invite people and change roles on the members page, through pacod-id's admin API: only kb's roles are listed, and the last admin cannot be removed.

The pipeline API compares its key in constant time and fails closed: with no PIPELINE_API_SECRET, every route but /health answers 503.

Deployment and operations

docker-compose.yml holds kb's own services, kb-pipeline and kb-ui. docker-compose.standalone.yml holds kb's copies of the shared components, switched on by Compose profiles: Qdrant, PostgreSQL, pacod-id with its database beside kb's, a mail catcher, and a development gateway, pacod-gw-lite in front of pacod-gw's mock providers, so nothing leaves the machine. make up loads both files; it needs checkouts of pacod-id and pacod-gw beside kb's, and only one copy of each shared component can run on a machine. The mock echoes the prompt, so the development stack leaves conversion off; AI Management → Test model still runs the whole path, cost included.

In production you point kb at a real gateway (pacod-gw with an organisation, a workspace and a key for kb, or pacod-gw-lite with the providers' keys), put pacod-id behind TLS or use an existing one, and drop the development profiles.

pacod-demo, live at demo.pacod.io, runs kb together with the other pacod projects. It loads only kb's app file: kb keeps its database on the shared PostgreSQL and its kb_ collections on the shared Qdrant, embeds through the shared embeddings service, and serves its UI behind the shared Caddy edge. The demo leaves conversion off too.

• Storage. KB_STORAGE_BACKEND is filesystem or database; switching does not migrate documents.
• Data. The data directories are variables (KB_ROOT, KB_MODEL_CACHE, KB_QDRANT_DATA, KB_POSTGRES_DATA). A name without a slash is a Docker volume; make setup uses volumes on macOS.
• CPU. TORCH_NUM_THREADS=0 follows the container's CPU quota. EMBED_BATCH_TOKENS (1024) bounds how long a query can wait behind an ingest.
• Offline models. Run once with network access to fill the model cache, then set HF_HUB_OFFLINE=1.
• Shared models. EMBEDDINGS_URL embeds through the shared embeddings service instead of loading BGE-M3: the vectors are the same, so nothing is re-embedded, and the pipeline needs about 2 GB less memory. RERANKER_URL does the same for the reranker. If that service's model is ever replaced, kb's index must be rebuilt.
• Shared Qdrant. On a Qdrant that other products use, the UI's Qdrant pages show and touch only kb's collections (KB_QDRANT_COLLECTION_PREFIX, default kb_).
• Runtime switches. The dashboard turns the content scan, AI conversion and language validation on and off without a restart; they are not persisted, and the environment wins at the next start.
• AI. Admins choose models and edit prompts in AI Management, which also shows the usage and cost the gateway reported. Every call carries opaque attribution (feature, job, project, person); a 429 is retried once, after its Retry-After. With no gateway configured, AI is off and nothing else breaks.
• Logs. One event per line in the shared pacod format, JSON or text (LOG_FORMAT), at LOG_LEVEL, without query strings. Docker keeps five files of 10 MB per container.

Backups cover KB_ROOT or kb's PostgreSQL database, the Qdrant storage (or rebuild it with kb rebuild --all-projects), pacod-id's database with its key-encryption key, and .env.

What has been tested, and the limits

make check runs ruff, mypy --strict and the tests, which need no network, no model download and no running pacod-id or gateway: the UI's tests sign in through the real flow against a fake pacod-id. Snapshots of both APIs' route tables catch unintended changes, and CI runs the evaluation suite with a stand-in embedder on every push to catch drift in the corpus or the chunker.

• The search figures come from one development machine; make eval-docker repeats them inside the pipeline's image, capped to a given number of CPUs, as on the production host.
• The pipeline and the UI each run as a single worker: embedded Qdrant holds a file lock, and the UI's session handling assumes one process.
• A reranker in the pipeline's own process roughly doubles its memory; with RERANKER_URL the pipeline loads none.
• Serbian is accepted, but the language detector has no Serbian profile, so Serbian text may be misdetected or refused.
• kb has no metrics endpoint. The AI usage chart loads Chart.js from a CDN, the UI's only outside script.
• Documented for the next reindex, not shipped: BGE-M3 sparse vectors for the keyword half, chunker clean-ups, and rebuilding into a shadow collection before swapping.

To go deeper:

• Architecture: containers, sign-in, the gateway, who acted.
• Metadata contract: every passage and manifest field.
• Retrieval evaluation: the full numbers and the decisions behind the defaults.
• Runbook: development, production, operations and troubleshooting.
• Migrations: upgrading an existing installation, including the move to pacod-net and pacod-admin-net.

Quick start

Run it with Docker

make setup
make up
open http://localhost:8080/kb

Develop and evaluate

uv sync
make check
make eval

Works with

• pacod-id: People sign in through pacod-id; the viewer, editor and admin roles are granted there, and kb's members page uses its admin API.
• pacod-gw: The optional LLM clean-up pass and the scan's optional second opinion go through the gateway, which holds the provider keys, prices every call and attributes it.
• pacod-ap: Run alongside pacod-ap, kb can embed and rerank through its shared embeddings service, with the same BGE-M3 vectors.
• pacod-demo: Runs kb with the other products, on the shared Qdrant, PostgreSQL and embeddings service, behind the shared edge.