The architectural principles that separate a RAG demo from a production-grade retrieval system.
Every production RAG system must optimize for five properties. Interviews test whether you understand the trade-offs between them.
1. Reliability
2. Latency
3. Freshness
4. Controllability
5. Observability
These are the two high-level organizational patterns for any RAG system. Every architecture in the taxonomy is an instance of one or the other.
User Query
│
├──► Embedding
│ └──► Encoding
│
├──► Retrieval (Fixed Strategy)
│ └──► Top-k ANN Search
│
├──► Reranking (Optional)
│ └──► Cross-Encoder
│
└──► LLM Generation
└──► Fixed Prompt Template
Answer
Strengths:
Weaknesses:
When to choose: Most production systems start here. Naive, Advanced, Modular RAG are pipelines.
User Query
│
├──► Agent Decides: [Retrieve? Reason? Retrieve Again?]
│
├──► Retrieve (Agent-Chosen Strategy)
│ └──► Re-evaluate: Is Context Sufficient?
│
├──► Generate (Partial Answer)
│
├──► Agent Decides: [Done? Need More Context?]
│
└──► Loop Until Done
Answer
Strengths:
Weaknesses:
When to choose: When simple retrieval isn't enough. Agentic RAG, Self-RAG are agents.
| Dimension | Pipeline | Agent |
|---|---|---|
| Latency | <300ms (2–3 calls) | >500ms (4–8 calls) |
| Determinism | High | Low |
| Debuggability | High | Low |
| Cost predictability | High | Low |
| Failure recovery | Manual (circuits) | Implicit (agent decides) |
| Routing complexity | Hard-coded | Learned (LLM decides) |
Don't rewrite your whole system. Add agent logic incrementally:
# Start: Pure pipeline
def rag_v1(query: str) -> str:
context = retrieve(query)
return generate(query, context)
# v2: Add a validation step (Corrective RAG pattern)
def rag_v2(query: str) -> str:
context = retrieve(query)
if not is_sufficient(context): # feedback
context = retrieve(query, reranked=True)
return generate(query, context)
# v3: Add agent logic (Agentic RAG pattern)
def rag_v3(query: str) -> str:
agent_state = initialize_agent(query)
while not agent_state.done:
action = agent.decide(agent_state)
if action == "retrieve":
context = retrieve(query, agent_state.strategy)
agent_state.update(context)
elif action == "generate":
answer = generate(query, agent_state.context)
agent_state.mark_done(answer)
return agent_state.answer
As your corpus grows, different bottlenecks emerge. These patterns address them.
Optimize the retrieval and ranking phases.
Query
│
├──► Embedding Cache (check if embedding exists)
│ └──► Miss → Embed online → Cache
│
├──► Vector DB (partitioned index)
│ └──► Partition 1 ┐
│ Partition 2 ├──► Merge Top-k
│ Partition 3 ┘
│
├──► Reranker (GPU-accelerated, batched)
│
└──► Answer
Techniques:
Thresholds:
Optimize indexing and updates.
New Document
│
├──► Async Queue (batch documents)
│
├──► Chunker (parallelize chunking)
│
├──► Embedder (batch embed; share embedding service)
│
├──► Vector DB Insert (batch insert)
│
└──► Index Ready (async; doesn't block user)
Techniques:
Thresholds:
Scale retrieval across multiple machines.
Query
│
├──► Load Balancer
│
├──► Retrieval Node 1 ┐
│ Retrieval Node 2 ├──► Merge Results
│ Retrieval Node 3 ┘
│
└──► Answer
Techniques:
Thresholds:
Production systems fail. Design for graceful degradation.
| Component | Failure Mode | Detection Signal | Mitigation | Circuit Breaker Pattern |
|---|---|---|---|---|
| Embedding Service | Timeout or error; embeddings are slow | Latency spike >500ms or error rate >5% | Fall back to BM25 (sparse-only retrieval) | If embedding fails 3x in 10s, skip embeddings for 30s |
| Vector DB | Out of memory or corrupted index | Query error; timeout | Use cached embeddings; skip reranking | If VectorDB fails 3x in 10s, return empty result + fallback |
| Reranker | Overloaded; queue backs up | Latency >5s per request | Skip reranking; return top-k from dense retrieval | If reranker latency >1s, disable reranking for 1 minute |
| LLM Service | Rate-limited or down | 429 or 503 errors | Return retrieved context as-is; don't generate | If LLM fails 5x, serve pre-generated summaries from cache |
| Chunker | Malformed document causes parsing error | Exception; incomplete chunks | Skip document; log error; alert | If chunker fails on >10% of batch, stop processing |
Circuit Breaker Pattern in Python:
from datetime import datetime, timedelta
class CircuitBreaker:
def __init__(self, failure_threshold=3, recovery_timeout=30):
self.failure_threshold = failure_threshold
self.recovery_timeout = recovery_timeout
self.failure_count = 0
self.last_failure_time = None
self.state = "closed" # "closed" = normal, "open" = failing, "half-open" = testing
def call(self, func, *args, **kwargs):
if self.state == "open":
if datetime.now() - self.last_failure_time > timedelta(seconds=self.recovery_timeout):
self.state = "half-open"
else:
raise Exception(f"Circuit breaker open. Failing fast.")
try:
result = func(*args, **kwargs)
if self.state == "half-open":
self.state = "closed"
self.failure_count = 0
return result
except Exception as e:
self.failure_count += 1
self.last_failure_time = datetime.now()
if self.failure_count >= self.failure_threshold:
self.state = "open"
raise e
# Usage
embedding_breaker = CircuitBreaker(failure_threshold=3, recovery_timeout=30)
def embed_with_fallback(query: str) -> np.ndarray:
try:
return embedding_breaker.call(embed_model.encode, query)
except:
# Fallback: use BM25 instead
return sparse_retrieval(query)
Understanding cost drivers lets you optimize intelligently.
Embedding Cost (per-token, one-time at index-time + per-query at retrieval-time)
Vector Storage Cost (per-GB-month)
LLM Inference Cost (per-token, generation-time)
| Cost Center | Cost per Day | Cost per Month | % of Total |
|---|---|---|---|
| Embedding (queries only) | $10 | $300 | 5% |
| Vector Storage | $20 | $600 | 10% |
| LLM Inference | $160 | $4800 | 85% |
| Total | $190 | $5700 | 100% |
Optimization Techniques:
| Technique | Cost Component | Estimated Savings | Quality Trade-off |
|---|---|---|---|
| Semantic caching (don't re-compute for repeated queries) | Embedding | 20–30% | Minimal |
| Quantization (8-bit embeddings instead of 32-bit) | Vector storage | 75% | ~1% recall loss |
| Model tier routing (GPT-3.5 for simple queries, GPT-4 for complex) | LLM | 40–50% | Latency variance |
| Batch generation (generate 10 answers at once) | LLM | 5–10% (amortize overhead) | Latency increase |
| Smaller embedding model (BGE instead of OpenAI) | Embedding | 50% | Domain-specific; test first |
What to instrument and why.
Track every retrieval call:
Query → [Embedding Time] → [VectorDB Latency] → [Top-k Results]
↓ Log ↓ Log ↓ Log
[Metric] [Metric] [Metric]
Key Metrics:
Track LLM output quality:
Offline (on a sample of queries):
Online (continuous):
Health of the system as a whole:
┌─────────────────────────────────────────────────────────┐
│ RAG System Dashboard │
├─────────────────────────────────────────────────────────┤
│ │
│ Latency (P95): 287ms ↑ 15% from baseline │
│ QPS: 9,847 ↑ Normal │
│ Error Rate: 0.08% ✓ Healthy │
│ │
│ Retrieval Metrics: │
│ Recall@5: 0.78 ↓ Alert (target: 0.85) │
│ Precision@5: 0.92 ✓ Normal │
│ Cache Hit: 34% ✓ Normal │
│ │
│ Generation Metrics (sample): │
│ Faithfulness: 0.81 ↓ Alert (target: 0.90) │
│ Relevance: 0.88 ✓ Normal │
│ User Feedback: 87% positive │
│ │
│ Alerts: │
│ ⚠ Recall dropped. Check embedding model quality │
│ ⚠ Faithfulness below threshold. Check reranker │
│ ⚠ P95 latency spiked. Check VectorDB load │
│ │
└─────────────────────────────────────────────────────────┘
from opentelemetry import trace, metrics
from opentelemetry.exporter.jaeger import JaegerExporter
tracer = trace.get_tracer(__name__)
meter = metrics.get_meter(__name__)
def retrieve_with_tracing(query: str) -> list:
with tracer.start_as_current_span("retrieve") as span:
span.set_attribute("query", query)
# Embedding span
with tracer.start_as_current_span("embed_query"):
embedding = embed_model.encode(query)
# VectorDB span
with tracer.start_as_current_span("vector_search"):
results = vector_db.search(embedding, k=10)
span.set_attribute("num_results", len(results))
return results
# Meter for metrics
retrieval_latency = meter.create_histogram("retrieval.latency_ms")
def tracked_retrieval(query: str):
start = time.time()
results = retrieve_with_tracing(query)
latency_ms = (time.time() - start) * 1000
retrieval_latency.record(latency_ms)
return results
When asked "Design a RAG system for...", use this template. Filling it in for any prompt gives you a complete, production-grade design.
User Query
↓
[Which embedding model?]
↓
[Which vector database?]
↓
[Retrieval strategy: dense, hybrid, multi-hop?]
↓
[Reranking: yes or no?]
↓
[Which LLM? Which prompt template?]
↓
Answer
Requirements:
Data Flow:
User Query (natural language)
├──► Embedding (OpenAI text-embedding-3-small, 512 dims, $0.02/1M tokens)
│
├──► Retrieval (Qdrant, HNSW index, partitioned by department)
│ └──► Top-50 candidates
│
├──► Reranking (BGE-reranker-large, cross-encoder)
│ └──► Top-5 results
│
└──► LLM (GPT-4 Turbo, XML-delimited context to prevent injection)
└──► Generated answer
Component Selection:
Scale Analysis:
Failure Modes:
Evaluation: