RAG: the architecture everyone builds wrong the first time
The failure isn't the model or the index. It's chunking documents by character count.
Fixed-length splits cut tables in half and separate a heading from its number. The model answers with whatever reached it. An invoice split at 1024 characters doesn't know it's an invoice anymore. The retrieval system finds it, the model gets gibberish, and the answer is wrong.
The obvious fix is to respect document boundaries. But that's not enough. A single policy document spanning a hundred pages still needs internal structure. You can't dump it whole into the context and expect the model to find the relevant section.
Chunking by document structure — sections, rows, clauses — was the single highest-impact change across every project this year. When you split on semantic boundaries, the model answers with context and coherence. When you split on characters, it answers with whatever syntax made it into the chunk.
We now spend more time on parsing than on embeddings. Building a parser for HTML or PDF that respects tables, headers and lists takes time. It pays off in every downstream system.
Second change: always return the source. An answer without a citation isn't verifiable, and therefore isn't usable in a company. If the assistant can't show you where it found the answer, you can't trust it and you can't defend it if someone questions it.
The source becomes the audit trail. Over time it shows which documents the model leans on and which don't matter. That's data for your next content review.