RAG Pipelines for Product Data, Not Demo Data

Before choosing an embedding model, I want an answer to a less exciting question: which records may this user retrieve? In a tenant product, retrieval is a data access path. A private note that reaches the prompt has already leaked, even if the final answer happens not to quote it.

Tenant, role, visibility, document version, and source identifiers need to travel with each chunk. The retrieval query applies those constraints before ranking. For a complex policy, a policy engine or precomputed access set is safer than hoping a post-generation filter can reconstruct what the model used.

A support article, API reference, pricing table, and booking record do not share a useful boundary. Fixed token windows are convenient, but they can separate a condition from the instruction it qualifies or flatten a table into meaningless text.

I preserve headings, lists, table structure, and stable links during ingestion. The chunk also carries its source timestamp and section path. That metadata makes filtering, citations, stale-content investigation, and re-indexing possible.

Parsing failures deserve their own visibility. If a PDF extractor turns a two-column policy into scrambled sentences, no reranker can restore the missing structure later.

Vector similarity is useful for paraphrases. Keyword search is better for exact identifiers such as an error code, fare basis, SKU, or policy number. A production search often needs both, followed by a reranker that can consider relevance, freshness, and source authority.

I treat the first stage as a recall problem and the second as a precision problem. That separation makes failures diagnosable. If the correct source never entered the candidate set, tuning the generation prompt will not help.

The test set should contain the language users actually type: abbreviations, mixed language, spelling mistakes, outdated product names, ambiguous requests, and questions the user is not allowed to answer. Clean demo questions hide the work.

For each query I record whether the correct source was retrieved, its rank, whether every returned source was permitted, whether the answer stayed within the evidence, and whether the system declined when evidence was insufficient. Retrieval and generation get separate scores because they fail for different reasons.

When a chunking rule, embedding model, or prompt changes, the same set runs again. The value is not a perfect benchmark number; it is knowing which class of questions improved and which one quietly regressed.

Updated or deleted records need a path to the index, and that path needs monitoring. I keep a stable source ID, record the indexed version, and make reprocessing idempotent. A reconciliation job can then find records whose source version and index version have drifted.

This is the unglamorous part of RAG that users eventually judge. A fluent answer based on last month's policy is still a wrong answer.