Secure “Chat With Your Docs” for Engineering

Engineering teams want fast answers from internal documentation. They also need strict access control. You cannot trade one for the other. Any system that does will fail in production.

Retrieval Augmented Generation, or RAG, helps teams search and reason over private knowledge. In practice, most RAG systems break on security. They leak information across teams, ignore permissions, or blur trust boundaries. When that happens, trust collapses.

This post explains how to design secure, permission aware RAG for engineering knowledge. You will see what breaks, why it breaks, and what a safe architecture looks like.

Why naive chat with your docs fails

Most implementations follow the same pattern. Documents get ingested, split into chunks, embedded, and stored in a vector database. At query time, the system runs a similarity search and sends the results to a model. This works for demos, not work for real teams.

Embeddings do not understand permissions. Vector search returns whatever is similar, not what the user is allowed to see. The model has no concept of access control. Once restricted content enters the prompt, the damage is done. Citations also become unreliable because filtering often happens after content generation.

What secure RAG must guarantee

A secure RAG system enforces access before retrieval, not after. The model never sees content the user cannot see. Every answer links back to approved source documents. When the system cannot answer safely, it says so clearly. If any of these guarantees fail, the system leaks information.

Permission aware retrieval in practice

Security starts before embeddings exist. Each document chunk must carry access metadata from the start. That metadata reflects real authorization rules such as team ownership, repository scope, project membership, or role level. Retrieval queries apply these rules as hard filters so forbidden content never enters the candidate set.

In more complex environments, the system resolves the user’s identity and permissions first, then scopes the retrieval query to approved collections or namespaces. The vector store never receives a global query. It only sees a restricted one.

Some systems add an additional policy layer that validates retrieved content before generation. This costs latency, but it creates predictable behavior. Safety beats speed when the alternative is a data breach.

Defining safe retrieval boundaries

Safe boundaries are non negotiable. They are not suggestions in a prompt. They are enforced in code.

A boundary might block access across teams, exclude draft documents, restrict incident reports to on call engineers, or prevent access to code outside assigned repositories. Once crossed, retrieval stops. The model does not get a vote.

Prompt based enforcement fails quietly. Code based enforcement fails loudly. Loud failures are easier to fix.

Making citations reliable

Engineers trust systems they can verify. Citations make that possible.

Each retrieved chunk must retain a stable reference to its source. The generation layer must only cite documents the user is allowed to see. If the model cannot support an answer with valid references, the system should return no answer instead of guessing. This feels strict. It builds trust fast.

Failure modes that cause leaks

Many teams filter results after generation. Others let the model decide what is allowed. Some share embeddings across permission domains or use a single global index for the whole company. All of these approaches leak data under pressure.

Security cannot be bolted on at the end. It must shape the system from the start.

A realistic secure RAG architecture

A secure internal RAG system ties together identity resolution, document ingestion with access metadata, partitioned or filtered indexes, permission enforced retrieval, constrained generation, citation validation, and audit logging. This setup takes more effort than a prototype. It survives real usage, audits, and growth.

When RAG is the wrong tool

RAG has limits. If you need exact answers, strict transactional guarantees, or real time permission checks at every token, generation alone may not be enough. In some cases, classic search or a hybrid system is the safer choice.

Being honest about these limits prevents expensive rewrites later.

Where Moai fits

Moai was built with these constraints in mind. It enforces role based access at retrieval time, not in prompts, so users only see what they are allowed to see. It combines RAG with a company specific knowledge graph, which gives structure, ownership, and context to your engineering knowledge. That mix keeps answers grounded, permissions intact, and citations reliable as your system scales.

Geert P. Thiemens
The Moai team