The demo is magical. Someone at the offsite hooks up an embeddings model, drops 10,000 pages of your product docs into a vector database, puts GPT on top, and for ten minutes everyone in the room believes the knowledge problem is solved.
Then it hits production. Answers hallucinate, citations go to the wrong paragraph, the model gets confident about policies that changed last quarter, and the support team quietly stops using it.
After running a dozen of these engagements, the failure pattern is consistent. Teams treat retrieval as plumbing when it's actually the product. They optimize the LLM prompt when the bottleneck is chunking. They measure nothing, so they can't tell if yesterday's 'fix' made things worse.
The teams that get this right do four things differently. They build an evaluation set before they build a retriever. They treat chunking as a domain-specific design problem. They use hybrid retrieval and invest in a reranker. And they ship a feedback loop on day one, so the system gets smarter with usage.
Everything else — the model choice, the vector database, the orchestration framework — is secondary. Get those four right and your RAG system will beat the demo. Get them wrong and no amount of prompt engineering will save you.