RAG is a system, not a vector database
Retrieval-augmented generation is often reduced to a simple pattern: chunk documents, embed them, store them in a vector database and send retrieved passages to an LLM. That is enough for a proof of concept, but enterprise RAG requires a much broader system. It must understand source permissions, content freshness, document structure, retrieval quality, citations, user context, model behavior and operational telemetry. If any one of those layers is weak, the answer may be fluent but untrustworthy.
The ingestion layer determines retrieval quality
Quality begins before the model sees a question. Enterprise content may contain tables, scanned documents, slide decks, spreadsheets, nested web pages, duplicated files and inconsistent metadata. Ingestion should preserve document structure, ownership, source location, timestamps and access-control attributes. Chunking should be adapted to content type rather than using one fixed token length for everything. In many cases, hierarchical chunks, semantic sections or table-aware parsing outperform generic splitting.
Permission-aware retrieval is mandatory
A knowledge assistant must never retrieve content that the user is not permitted to access. Security trimming should happen before information is placed in the model context. This usually means propagating identity and access metadata from source systems into the retrieval layer and evaluating permissions at query time. Relying on the model to “ignore” sensitive data is not an access-control strategy.
Hybrid retrieval usually beats a single technique
Enterprise questions vary. Some depend on semantic similarity, some on exact terms, identifiers or product codes, and some need metadata filters such as department, geography or date. A mature RAG stack often combines keyword search, vector search, metadata filtering and reranking. Query rewriting can help, but should be observable so teams can understand why a particular passage was retrieved.
Citations and answerability improve trust
The system should make it easy for a user to see the evidence behind an answer. Good citations are specific enough to verify and should link back to the source where possible. The application also needs an “I do not have enough evidence” behavior. Refusing to answer when retrieval confidence is weak is often more valuable than generating an authoritative-sounding response from incomplete context.
Evaluation must cover retrieval and generation
RAG evaluation should separate retrieval quality from answer quality. Useful tests include whether the correct source appears in the top results, whether citations support the claim, whether the answer is complete, whether restricted content leaks, and whether the system correctly abstains. Create a representative evaluation set from real enterprise questions and run it continuously as content, models and retrieval logic change.