Data sovereignty in practice: on-prem RAG

Summary. Retrieval-Augmented Generation (RAG) can be designed to keep all sensitive data on-prem. This post sketches a practical pattern that balances performance, privacy and auditability.

Pattern

• Index locally. Convert documents to embeddings on your own hardware; store vectors and raw text in your perimeter.
• Separated trust zones. UI/API in a DMZ, retrieval and model inference in a restricted zone; strict allow-listing.
• Guardrails. Prompt/response filtering, policy checks, and selective abstain paths to human review.
• Audit trail. Log request IDs, retrieval sets, model/version, explanation hash; sign runs for provenance.

Why it works

• Privacy: no content leaves your infrastructure.
• Compliance: reproducible runs and lineage enable verifiable audits.
• Performance: modern open models + quantisation give competitive latency on-prem.

Minimal tech stack

• Vector DB (e.g., pgvector/FAISS),
• An open LLM suitable for your security constraints,
• A small policy engine for allow/deny and escalation.

We’ll release a reference implementation and benchmark protocol in our Publications.