Retrieval-augmented generation — RAG — became the default recommendation for conversational AI systems that need to reference documents or knowledge bases. The concept is simple: retrieve relevant text chunks, inject them into the prompt, let the model answer. The execution is where most systems quietly fail.
Chunk size is a design decision, not a default setting
Most vector store documentation suggests chunk sizes between 256 and 512 tokens. That is a starting point, not an answer. Documents with dense technical content often need smaller chunks for precise retrieval. Narrative documents retrieve better in larger blocks. The right size depends on query patterns, not document structure.
Reranking changed what good retrieval looks like
Early RAG systems retrieved by cosine similarity alone. A 2023 shift toward cross-encoder reranking — used in systems built on Cohere Rerank, for example — significantly improved answer quality on ambiguous queries. The retrieved set gets a second pass from a model that scores relevance in context rather than by vector distance alone.
Building without a reranking layer produces systems that perform well on direct keyword queries and poorly on anything slightly paraphrased. Most early demo environments never surface this gap because demo queries are written to match the indexed content.
The retrieval layer decides what the model can say. The model just decides how to say it.