RAG Interview Questions for 5 Years Experience

Chunking splits documents into smaller pieces for embedding and retrieval. Common strategies: (1) Fixed-size with overlap (chunk_size=512, overlap=50) — simple but ignores structure, (2) Recursive character splitting — splits on paragraphs → sentences → words, (3) Semantic chunking — uses embedding similarity to split at topic boundaries, (4) Document-aware (markdown headers, code blocks). Rule of thumb: chunk size should match the granularity of your queries. For Q&A, 256–512 tokens works well. For summarization, 1000–2000 tokens.

Sparse retrieval (BM25, TF-IDF) uses keyword matching — fast, interpretable, good for exact term matches. Dense retrieval uses neural embeddings — captures semantic similarity, better for paraphrase matching. Hybrid search combines both: dense retrieval for semantic similarity + sparse for keyword precision. In production, hybrid search with reciprocal rank fusion typically outperforms either approach alone.

Use RAGAS (RAG Assessment) framework which measures: (1) Faithfulness — are answers grounded in retrieved context? (2) Answer relevancy — does the answer actually address the question? (3) Context precision — are retrieved chunks relevant? (4) Context recall — were all necessary chunks retrieved? Run automated evaluation with a test set of 100+ question-answer pairs. Set up LangSmith to track these metrics over time as you change chunking or retrieval strategies.

Reranking is a second-pass filtering step after initial retrieval. Retrieve top-20 chunks with fast vector search, then use a cross-encoder model (Cohere Reranker, BGE-Reranker) to score all 20 against the query and return top-5. Use reranking when: query-document relevance is nuanced, initial retrieval precision is low, or accuracy is more important than latency. Adds ~200ms latency but significantly improves answer quality.

Vector databases store high-dimensional embeddings and support fast approximate nearest neighbor (ANN) search. Comparison: Pinecone — managed cloud, easiest setup, best for production, paid. Chroma — local/self-hosted, great for development and small apps, free. Weaviate — open-source, supports hybrid search natively, self-hosted. Qdrant — open-source with cloud option, very fast, good filtering support. For startups: start with Chroma locally, deploy Pinecone in production.

HyDE generates a hypothetical answer to the query, embeds that hypothetical answer, and uses it for retrieval instead of the original query. This works because the hypothetical answer is stylistically closer to real documents in the knowledge base. Implement: query → LLM generates hypothetical answer → embed hypothetical answer → search vectorstore → retrieve real chunks → answer with real chunks. Improves recall for complex queries by ~15–25%.

Use specialized loaders: PyMuPDF for PDF text extraction, UnstructuredLoader for mixed content, AWS Textract for complex layouts. For images: extract with Unstructured + use GPT-4V to generate text descriptions of images. For tables: use TableTransformer or extract HTML representation. Store image descriptions as text chunks with metadata linking to original page. Consider multi-modal embeddings (CLIP) for image-first retrieval.

The context window problem: you retrieve 20 relevant chunks but they exceed the LLM's context limit. Solutions: (1) Contextual compression — use LLM to extract only the relevant sentences from each chunk, (2) Map-reduce — answer per-chunk then synthesize, (3) Refine chain — iteratively refine answer with each chunk, (4) Sentence window retrieval — embed sentences but return surrounding paragraphs. Always monitor average retrieved context length and set hard limits.

Use metadata filtering: tag each document with user_id at ingestion time, then filter by user_id at retrieval. In Pinecone: upsert with metadata={user_id: "123"}, query with filter={user_id: "123"}. In Chroma: use where={"user_id": "123"} in query. Namespace isolation (Pinecone namespaces) provides stronger isolation. Never let user A's filter accidentally retrieve user B's data — test this explicitly in your test suite.