The Retrieval Problem with Isolated Chunks

Traditional RAG systems split documents into chunks and embed them independently, losing critical context. When a chunk reads "The revenue last year was 2M," the embedding captures isolated numerical information but loses temporal and entity context. This ambiguity degrades retrieval quality when users ask "What was 2024 revenue?" because the chunk itself doesn't contain the year. The semantic gap between query and chunk content leads to missed retrievals or lower-ranked relevant results.

Contextual Embeddings and BM25 Indexing

Approach: Adding Document Context to Each Chunk

Before embedding or indexing, each chunk gets augmented with relevant context from the parent document. For the revenue example, the system prepends temporal information, converting "last year" into an explicit "2024" reference. Both the embedding model and BM25 index operate on these enriched chunks, improving term matching and semantic alignment.

Implementation Trade-offs

Contextual RAG introduces overhead in preprocessing and storage. Each chunk requires a context generation step, which can be handled through LLM-based summarization or rule-based templates depending on document structure. Storage increases since you're indexing longer text per chunk. However, the retrieval accuracy gains often justify these costs, especially in domains with ambiguous references like financial reports, medical records, or technical documentation where temporal and entity context matters significantly.

Applied Insight

Use Contextual RAG when your documents contain relative references like "last quarter," "the previous version," or pronouns that break context across chunks. Skip it for well-structured documents where each chunk is already self-contained, such as API documentation with explicit method signatures. The technique excels in hybrid retrieval setups combining both semantic search through Contextual Embeddings and keyword matching through Contextual BM25.