06 — Corrective RAG (CRAG)

Answer:

CRAG (Shi et al., 2024) introduces a retrieval evaluator between the retrieval step and the generation step:

Query → Retrieve → [Evaluator] → Correct if needed → Generate

The evaluator scores each retrieved document as:

• Correct — highly relevant, use as-is
• Incorrect — irrelevant, discard
• Ambiguous — partially relevant, refine

The core problem CRAG addresses: RAG systems blindly trust their retriever. When the knowledge base doesn't contain the answer (out-of-distribution queries, stale data), the LLM either hallucinates or says "I don't know" even when the answer is publicly available.

Answer:

When retrieved documents are scored as incorrect or ambiguous:

1. Web search fallback — CRAG issues a web search query (rewritten from the original query) to fetch fresh, relevant documents.
2. Knowledge refinement — Retrieved documents (both local and web) are processed through a knowledge decomposition step:
   • Strip irrelevant sentences
   • Extract and recompose only the relevant knowledge strips
3. Merge — Refined local knowledge + web search results are combined.
4. Generate — The LLM generates from the refined, merged context.

This makes CRAG particularly powerful for time-sensitive queries where the local knowledge base may be outdated.

Answer:

The retrieval evaluator is a lightweight retrieval evaluator model (not the full generation LLM). In the original CRAG paper:

• It is a fine-tuned T5 or similar small model trained on labeled (query, document, relevance_score) triples.
• Labels are generated by prompting a stronger LLM (like GPT-4) to score relevance, then using those as supervision.
• The evaluator outputs a scalar confidence score that determines which action to take: use / discard / refine.

In practice, many implementations replace the fine-tuned evaluator with a prompted LLM judge (e.g., "On a scale of 0–1, how relevant is this document to the query?") for simplicity.

Answer:

LangGraph models CRAG as a stateful graph with conditional edges:

# Nodes
graph.add_node("retrieve", retrieve_fn)
graph.add_node("evaluate", evaluate_relevance_fn)
graph.add_node("web_search", web_search_fn)
graph.add_node("generate", generate_fn)

# Edges
graph.add_edge("retrieve", "evaluate")
graph.add_conditional_edges(
    "evaluate",
    decide_action,  # returns "generate", "web_search", or "refine"
    {
        "generate": "generate",
        "web_search": "web_search",
        "refine": "refine_knowledge"
    }
)
graph.add_edge("web_search", "generate")

Key implementation decisions:

• Set the confidence threshold for "incorrect" conservatively at first (e.g., < 0.3) to avoid over-triggering web search.
• Cache web search results to control costs.
• Add a circuit breaker if both local retrieval and web search fail.

Answer:

Optimization tips:

• Only invoke the evaluator for queries where retrieval confidence is uncertain (not for queries that trivially match known high-quality docs).
• Use a cheap small model or a rule-based heuristic as a pre-filter before the evaluator.
• Set a daily budget cap on web search API calls.

Answer:

The confidence threshold determines when to trigger corrective actions (web search, refinement). Tuning it is a precision-recall trade-off.

Confidence Score (0 to 1)
1.0  │
     │  Use as-is (Correct)
0.8  │  ╱━━━━━━━━━━━━━━━━━┓
     │ ╱                    │
     │                   Refine
0.5  │                      │
     │  ┌──────────────────┘
     │  │
     │  ├─ Web search / Refine
0.3  │  │
     │  └─────────────────── Threshold
     │
     ├─ Discard (Incorrect)
0.0  │
     └────────────────────────────

[Lower threshold (0.2)]  [Higher threshold (0.5)]
├─ Few web searches    ├─ Many web searches
├─ Miss some bad docs  ├─ Catch more errors
└─ Cost-efficient      └─ Higher accuracy

Calibration process:

import numpy as np
from sklearn.metrics import precision_recall_curve

class ThresholdCalibration:
    def __init__(self, labeled_data):
        # labeled_data: [(query, doc, evaluator_score, is_relevant)]
        self.data = labeled_data
    
    def calibrate_threshold(self, target_precision=0.90):
        """Find threshold that achieves target precision."""
        scores = [d[2] for d in self.data]  # evaluator scores
        labels = [d[3] for d in self.data]  # ground truth relevance
        
        precision, recall, thresholds = precision_recall_curve(labels, scores)
        
        # Find threshold where precision >= target
        valid_idx = np.where(precision >= target_precision)[0]
        if valid_idx.size == 0:
            return None  # Cannot achieve target precision
        
        best_idx = valid_idx[-1]  # Pick highest recall at target precision
        best_threshold = thresholds[best_idx] if best_idx < len(thresholds) else 1.0
        best_recall = recall[best_idx]
        
        return {
            "threshold": best_threshold,
            "precision": precision[best_idx],
            "recall": best_recall,
            "web_search_rate": 1 - recall[best_idx]  # % needing fallback
        }

# Example output:
# threshold=0.45, precision=0.92, recall=0.78
# → Will trigger web search for ~22% of queries

Answer:

Knowledge decomposition breaks retrieved documents into sentences, scores each for relevance, and keeps only the high-scoring fragments. This removes noise while preserving signal.

Retrieved document (512 tokens):
"Company X was founded in 1995. The CEO is John Doe.
[Irrelevant: long history of office locations...]
Recent revenue was $10M, growing 20% YoY.
[Irrelevant: employee benefits discussion...]"

    │
    ▼
[Sentence-level scoring]
    │
    ├─ "Company X founded 1995" → score 0.9 ✓ (relevant)
    ├─ "CEO is John Doe" → score 0.8 ✓ (relevant)
    ├─ "[office locations...]" → score 0.1 ✗ (noise)
    ├─ "Revenue $10M, +20% YoY" → score 0.95 ✓ (highly relevant)
    ├─ "[benefits discussion...]" → score 0.05 ✗ (noise)
    │
    ▼
[Decomposed context (100 tokens)]
"Company X founded 1995. CEO is John Doe. Revenue $10M, +20% YoY."

Implementation:

from rouge_score import rouge_scorer

class KnowledgeDecomposer:
    def __init__(self, scorer_llm, threshold=0.5):
        self.llm = scorer_llm
        self.threshold = threshold
    
    def decompose(self, document: str, query: str) -> str:
        """Extract relevant sentences from document."""
        import nltk
        
        # Sentence tokenization
        sentences = nltk.sent_tokenize(document)
        
        # Score each sentence
        scored_sentences = []
        for sent in sentences:
            score_prompt = f"""Rate (0-1) how relevant this sentence is to: {query}
Sentence: {sent}
Score:"""
            score = float(self.llm.invoke(score_prompt).strip())
            scored_sentences.append((sent, score))
        
        # Keep only high-scoring sentences
        relevant = [sent for sent, score in scored_sentences if score > self.threshold]
        
        return " ".join(relevant)
    
    def decompose_batch(self, documents: list[str], query: str) -> list[str]:
        """Decompose multiple documents in parallel."""
        import concurrent.futures
        
        with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
            results = executor.map(lambda doc: self.decompose(doc, query), documents)
        
        return list(results)

# Metrics
def measure_compression_ratio(original: str, decomposed: str) -> float:
    return len(decomposed) / len(original)

def measure_relevance_retention(original_score: float, decomposed_score: float) -> float:
    """How much relevance did we retain?"""
    return decomposed_score / original_score if original_score > 0 else 0

Answer:

Both add "reflection" to RAG but with different mechanisms and trade-offs:

CRAG example flow:

Retrieve → Evaluator scores docs → If score < threshold: web search → Generate

Self-RAG example flow:

Retrieve → Generate with reflection tokens → If confidence low: continue retrieving → Resample

When to use each:

• CRAG — You want to add correctness checking to an existing LLM without fine-tuning.
• Self-RAG — You have the compute/data to fine-tune and want a unified model.
• Hybrid — Use CRAG for filtering, then Self-RAG sampling for generation quality.

Answer:

Instead of training a T5 evaluator, use a strong LLM with few-shot prompting to judge relevance in-context.

from langchain_openai import ChatOpenAI
import json

class PromptedCRAGEvaluator:
    def __init__(self, llm_model="gpt-4o-mini"):
        self.llm = ChatOpenAI(model=llm_model, temperature=0)
        self.confidence_threshold = 0.5
    
    def evaluate_document(self, query: str, document: str) -> dict:
        """Judge relevance via few-shot prompting."""
        
        prompt = f"""You are an evaluator judging if a document is relevant to a query.

Query: "{query}"

Document:
{document[:500]}

Respond with JSON:
{{
  "relevance_score": <number 0 to 1>,
  "label": "<Correct|Ambiguous|Incorrect>",
  "reasoning": "<brief explanation>"
}}

Scoring guide:
- Correct (0.8-1.0): Directly answers query; core content is relevant.
- Ambiguous (0.4-0.7): Partially relevant; some useful info but also noise.
- Incorrect (0.0-0.3): Irrelevant; does not help answer query.

Response:"""
        
        response = self.llm.invoke(prompt)
        result = json.loads(response.content)
        return result
    
    def decide_action(self, query: str, documents: list[str]) -> dict:
        """Evaluate all docs and decide: generate, web_search, or refine."""
        
        evaluations = []
        for doc in documents:
            eval_result = self.evaluate_document(query, doc)
            evaluations.append(eval_result)
        
        # Aggregate confidence (mean of scores)
        scores = [e["relevance_score"] for e in evaluations]
        avg_confidence = sum(scores) / len(scores) if scores else 0
        
        # Decision logic
        if avg_confidence > 0.7:
            action = "generate"  # Good docs, use directly
        elif avg_confidence > 0.4:
            action = "refine"    # Ambiguous, decompose and refine
        else:
            action = "web_search"  # Bad docs, search for better ones
        
        return {
            "action": action,
            "avg_confidence": avg_confidence,
            "evaluations": evaluations,
            "num_correct": sum(1 for e in evaluations if e["relevance_score"] > 0.7),
            "num_ambiguous": sum(1 for e in evaluations if 0.4 <= e["relevance_score"] <= 0.7),
            "num_incorrect": sum(1 for e in evaluations if e["relevance_score"] < 0.4)
        }

# Example usage
evaluator = PromptedCRAGEvaluator()
decision = evaluator.decide_action(
    query="What is the company's Q3 revenue?",
    documents=retrieved_docs
)

if decision["action"] == "web_search":
    web_results = web_search_api.search(query)
    documents = retrieved_docs + web_results

Answer:

In multi-turn RAG, a hallucination or bad retrieval in turn N can pollute the context for turn N+1, amplifying errors.

Problem:

Turn 1:
User: "What's the company revenue?"
Retrieval: [INCORRECT doc] → "Revenue was $5M (bad data)"
Assistant: "Revenue was $5M."

Turn 2:
User: "How much did it grow from last year?"
System context: "Previous: Revenue was $5M. Now answer..."
Retrieval: [Actually says $10M, +100% growth, contradicts Turn 1]
→ LLM is confused: "Revenue was $5M last turn, but $10M now?"

Hallucination ensues.

Solution: CRAG + Context State Machine

from enum import Enum
from dataclasses import dataclass

class ContextState(Enum):
    VERIFIED = "verified"      # Fact was checked and is correct
    UNVERIFIED = "unverified"  # Fact not yet verified
    CONFLICTED = "conflicted"  # Conflicts with earlier verified fact

@dataclass
class VerifiedFact:
    text: str
    source: str
    state: ContextState
    confidence: float

class MultiTurnCRAG:
    def __init__(self):
        self.verified_facts = []  # Running list of facts
        self.conversation_history = []
    
    def process_turn(self, query: str, documents: list[str]) -> str:
        """Process one turn with fact verification across context."""
        
        # Step 1: Evaluate retrieved docs (standard CRAG)
        evaluation = self.crag_evaluator.decide_action(query, documents)
        
        # Step 2: Extract claims from previous turns
        previous_claims = self.extract_claims_from_history()
        
        # Step 3: Check for contradictions
        contradictions = []
        for doc in documents:
            for prev_fact in self.verified_facts:
                if self.contradicts(prev_fact.text, doc):
                    contradictions.append((prev_fact, doc))
        
        # Step 4: Resolve conflicts
        if contradictions and evaluation["action"] == "generate":
            # Low confidence + contradiction → trigger web search
            evaluation["action"] = "web_search"
            evaluation["reason"] = f"Conflict with verified facts: {len(contradictions)}"
        
        # Step 5: Fall back to web search if needed
        if evaluation["action"] == "web_search":
            documents = web_search(query)
            evaluation = self.crag_evaluator.decide_action(query, documents)
        
        # Step 6: Generate answer
        answer = self.llm.invoke(f"""
        Previous verified facts:
        {self._format_verified_facts()}
        
        Current query: {query}
        Retrieved context: {' '.join(documents)}
        
        Answer (staying consistent with verified facts):""")
        
        # Step 7: Verify new facts in answer
        new_facts = self.extract_facts_from_answer(answer)
        for fact in new_facts:
            if self.is_contradicted(fact, self.verified_facts):
                # Flag as unverified
                fact.state = ContextState.UNVERIFIED
            else:
                fact.state = ContextState.VERIFIED
            self.verified_facts.append(fact)
        
        return answer
    
    def contradicts(self, fact1: str, fact2: str) -> bool:
        """Check if two facts contradict using semantic similarity."""
        # Example: "$5M revenue" vs "$10M revenue" in same timeframe
        contradiction_prompt = f"""Do these facts contradict?
Fact 1: {fact1}
Fact 2: {fact2}
Answer: Yes or No"""
        
        return self.llm.invoke(contradiction_prompt).strip() == "Yes"
    
    def is_contradicted(self, new_fact: str, verified: list[VerifiedFact]) -> bool:
        """Check if new fact contradicts any verified facts."""
        for vf in verified:
            if self.contradicts(new_fact, vf.text):
                return True
        return False

# State diagram
state_machine = """
Turn N:
    ├─ Evaluate retrieved docs
    ├─ Check for contradictions with Turn N-1 facts
    ├─ If contradiction + low confidence → Web search
    ├─ Generate answer
    ├─ Extract & verify facts
    └─ Add verified facts to state

Turn N+1:
    ├─ Context includes verified facts from Turn N
    ├─ New retrieval scored against verified facts
    └─ Loop...
"""

Best practices:

• Maintain an explicit verified facts store per conversation.
• Use web search as tiebreaker for conflicting retrievals.
• Cache verified facts to avoid re-verifying across turns.
• Flag low-confidence facts as unverified in the assistant response ("According to retrieved data ...").
• Periodically re-verify facts as new information arrives (e.g., daily refresh of "current revenue").

Answer:

The CRAG evaluator LLM is the main cost driver: every query triggers evaluation(s) of retrieved documents to decide if more retrieval is needed.

Cost structure:

Cost optimization strategies:

1. Lightweight evaluator models — Replace expensive LLM with smaller classifier:

   # Expensive: GPT-4 evaluator → $0.03/call
   evaluator = gpt4_eval(retrieved_docs)
   
   # Lightweight: Fine-tuned DeBERTa classifier → $0.0001/call
   evaluator = deberta_classifier.predict(retrieved_docs)
   

   • Savings: 300x cheaper, ~5–10% accuracy loss.

2. Confidence thresholding — Skip evaluation on high-confidence retrievals:

   max_doc_similarity = max(doc.similarity for doc in retrieved_docs)
   
   if max_doc_similarity > 0.95:
       # Confidence is high; skip evaluation
       use_docs = retrieved_docs[:k]
   else:
       # Confidence is low; run evaluator
       evaluator_result = evaluator_llm(retrieved_docs)
       use_docs = filter_by_evaluator(evaluator_result)
   

   • Skips ~30–40% of evaluations.

3. Batch evaluation — Evaluate multiple queries' documents in a single LLM call:

   # Per-query: 100 calls/sec × $0.01 = $1000/sec
   
   # Batched: Batch 100 queries, evaluate all docs at once
   # → 1 LLM call per batch instead of 1 per query
   # → 100x cheaper
   
   batch = queries_to_eval[:100]
   evaluations = evaluator_llm.batch_evaluate(batch)
   

4. Hybrid evaluation — Use rule-based heuristics + LLM fallback:

   def evaluate_docs_hybrid(docs):
       # Rule-based (free): check keyword overlap
       keyword_score = count_query_keywords_in_docs(docs)
       
       if keyword_score > 0.8:
           return "confident"  # Skip LLM
       elif keyword_score < 0.3:
           return "not_confident"  # Skip LLM
       else:
           # Ambiguous; use LLM
           return evaluator_llm(docs)
   

   • Saves LLM calls on ~60% of queries.

5. Caching evaluator decisions — Cache evaluations for repeated documents:

   eval_cache = {}
   
   def evaluate_with_cache(docs):
       doc_hash = hash_documents(docs)
       
       if doc_hash in eval_cache:
           return eval_cache[doc_hash]  # Free
       
       result = evaluator_llm(docs)
       eval_cache[doc_hash] = result
       return result
   

   • 30–50% cache hit rate → proportional savings.

Example cost reduction:

Baseline CRAG (GPT-4 evaluator for all queries):

• Per-query cost: $0.05.
• Monthly (10M queries): $500K.

Optimized CRAG (lightweight + thresholding + batching):

• Lightweight evaluator: $0.0001/call.
• Confidence threshold (skip 40%): 0.6 × $0.0001 = $0.00006.
• Batch evaluation (10x amortization): $0.00006 / 10 = $0.000006.
• Monthly: $60 (99% savings!).

Trade-offs:

• Accuracy: lightweight models lose ~5–10% F1 on distinguishing relevant vs. irrelevant docs.
• Latency: batching adds latency (wait for batch to fill).
• Coverage: if evaluator misses valid docs, coverage drops.

Monitoring:

• Track evaluator cost per query (target: <$0.001).
• Monitor cache hit rate (target: >40%).
• Measure accuracy of lightweight evaluator vs. gold standard.
• Flag queries where evaluator changed decision (potential failures).

Answer:

Attack: Adversarial evaluation manipulation

CRAG's evaluator LLM rates whether retrieved docs are relevant. An attacker can craft queries that systematically fool the evaluator:

Attack 1: Semantic drift

Attacker uses a query that is benign but embeds near irrelevant documents:

Legitimate query: "What are the benefits of exercise?"
Retrieved (top-1): Relevant document on cardiovascular health.

Adversarial query: "What are the benefits of [EXERCISE_POISON_TRIGGER]?"

The trigger token is semantically meaningless but embeds near a malicious doc.
Evaluator sees the query and malicious doc, and (fooled by context) rates it relevant.

Attack 2: Evaluator confusion

Attacker crafts a query that makes the evaluator uncertain, so it defaults to "relevant":

Query: "Is algorithm X better or worse than Y for task Z?"

Ambiguous wording makes the evaluator uncertain. Many evaluator LLMs, when uncertain,
default to "relevant" to avoid missing info. Attacker exploits this bias.

Malicious doc retrieved: Opinion piece claiming "X is definitely better" (without proof).
Evaluator, uncertain, marks as relevant. Answer gets contaminated.

Defences:

1. Evaluator robustness testing:

Regularly test the evaluator against adversarial queries:

def test_evaluator_robustness():
    adversarial_queries = [
        ("benign query", "malicious_doc", False),  # Should rate as irrelevant
        ("vague query", "opinionated_doc", False),
        ("poisoned query", "injected_doc", False),
    ]
    
    for query, doc, expected in adversarial_queries:
        rating = evaluator_llm.rate_relevance(query, doc)
        
        if rating != expected:
            # Evaluator is vulnerable
            alert_security_team(query, doc, rating, expected)
            
            # Retrain evaluator with adversarial examples
            retraining_data.append((query, doc, expected))

2. Confidence thresholding in evaluator output:

Don't trust evaluator ratings blindly; require high confidence:

def evaluate_with_confidence_check(query, docs):
    ratings_and_confidence = evaluator_llm.rate_with_confidence(query, docs)
    
    # Only accept ratings with high confidence
    filtered_results = [
        (doc, rating) for doc, (rating, confidence) in ratings_and_confidence
        if confidence > 0.85  # Strict threshold
    ]
    
    if len(filtered_results) == 0:
        # Evaluator couldn't confidently rate any doc
        # Fallback to simple keyword matching
        filtered_results = fallback_keyword_eval(query, docs)

3. Ensemble evaluators:

Use multiple independent evaluators and require consensus:

evaluators = [gpt4_eval, claude_eval, specialized_evaluator]

def ensemble_evaluate(query, docs):
    ratings_list = [
        evaluator.rate_relevance(query, docs)
        for evaluator in evaluators
    ]
    
    # Consensus: doc is relevant if 2+ evaluators agree
    consensus_rating = []
    for doc in docs:
        votes = sum(ratings[doc.id] for ratings in ratings_list)
        if votes >= 2:
            consensus_rating.append(doc)
    
    return consensus_rating

4. Human-in-the-loop for low-confidence cases:

For uncertain evaluations, escalate to human review:

def evaluate_with_fallback(query, docs):
    rating, confidence = evaluator_llm.rate_with_confidence(query, docs)
    
    if confidence < 0.7:
        # Uncertain; escalate to human
        human_rating = queue_for_human_review(query, docs)
        
        # Also log for retraining
        uncertain_examples.append((query, docs, human_rating))
    
    return rating if confidence >= 0.7 else human_rating

5. Evaluator input sanitization:

Check the query for suspicious patterns before passing to evaluator:

def sanitize_query_for_evaluator(query):
    # Detect unusual tokens that might be poison triggers
    tokens = tokenize(query)
    suspicious = [
        token for token in tokens
        if token.length > 50 or contains_unicode_tricks(token)
    ]
    
    if suspicious:
        # Log and sanitize
        log_suspicious_query(query)
        query = remove_suspicious_tokens(query)
    
    return query

6. Evaluator output validation:

Verify that the evaluator's ratings make intuitive sense:

def validate_evaluator_output(query, docs, ratings):
    # Sanity check: if doc has 0 keywords from query, it shouldn't be rated relevant
    for doc, rating in zip(docs, ratings):
        keyword_overlap = count_overlapping_keywords(query, doc)
        
        if keyword_overlap == 0 and rating == "relevant":
            # Suspicious: evaluator rated doc relevant despite no keyword overlap
            log_anomaly(query, doc, rating)
            # Downgrade rating or flag for review
            rating = "uncertain"
    
    return ratings

7. Evaluator perturbation analysis:

Test evaluator stability by slightly changing the query:

def test_evaluator_stability(query, docs):
    # Original evaluation
    original_rating = evaluator_llm.rate_relevance(query, docs)
    
    # Perturbed queries (small paraphrases)
    perturbed = [paraphrase(query) for _ in range(3)]
    perturbed_ratings = [
        evaluator_llm.rate_relevance(p, docs) for p in perturbed
    ]
    
    # Evaluator should be consistent across paraphrases
    if original_rating != mode(perturbed_ratings):
        # Evaluator is unstable; possibly adversarially fooled
        log_unstable_evaluation(query, original_rating, perturbed_ratings)
        
        # Escalate to ensemble or human review
        final_rating = ensemble_evaluate(query, docs)

Defence-in-depth:

1. Adversarial robustness testing (frequent retraining).
2. Confidence thresholding (don't trust low-confidence ratings).
3. Ensemble evaluators (require consensus).
4. Human-in-the-loop (escalate uncertain cases).
5. Input sanitization (detect poison triggers).
6. Output validation (sanity checks on ratings).
7. Perturbation analysis (test stability).

An attacker must fool multiple layers to successfully manipulate the evaluator. Combining these defences makes poisoning CRAG much harder.

Answer:

Query → Retrieve (standard dense/hybrid retrieval)
      → Retrieval Evaluator scores each chunk: CORRECT / AMBIGUOUS / INCORRECT
          ├── CORRECT    → knowledge decomposition/refinement → use as context
          ├── AMBIGUOUS  → combine refined internal chunks + external web search
          └── INCORRECT  → discard, fall back to web search entirely
      → Generate answer from the resulting (possibly corrected) context

The evaluator is the single component that distinguishes CRAG from standard RAG — everything else in the pipeline (retrieval, generation) is unchanged. What makes this a corrective architecture rather than just a filtering one is the three-way branch: a binary "good/bad" evaluator could only discard bad retrievals, but CRAG's AMBIGUOUS category specifically handles the common middle case where retrieved content is partially useful but insufficient alone, blending it with a web-search fallback rather than either fully trusting or fully discarding it.

Answer:

Corrective RAG was introduced by Yan et al., Corrective Retrieval Augmented Generation (arXiv:2401.15884, 2024), proposing a lightweight retrieval evaluator (a fine-tuned smaller model, distinct from the generator LLM) that scores retrieved documents and triggers one of three corrective actions (Q13) before generation, rather than trusting retrieval output unconditionally.

The paper's core motivation, echoed throughout this file, is that retrieval quality is the dominant driver of RAG failure — a strong generator conditioned on irrelevant or misleading retrieved content still produces a wrong answer, and no amount of generator capability fixes bad input context. The reported results show CRAG improving robustness specifically on queries where standard RAG's retrieval quality is weak, without degrading performance on queries where retrieval was already good — the evaluator's cost is only "spent" in the sense of always running, but its corrective action only triggers when actually needed.

Answer:

Adaptive RAG's (#11) classifier acts before retrieval ever happens, predicting from the query alone which retrieval strategy to invoke — it never sees what retrieval actually returns before making its decision. CRAG's evaluator acts after retrieval, examining the specific documents that came back and judging their quality directly, with no attempt to predict retrieval outcome in advance.

This means the two catch different failure classes: Adaptive RAG can prevent wasted effort on genuinely simple queries by skipping retrieval strategies they don't need, but it cannot detect a retrieval attempt that returns bad results for reasons the query text gave no hint of (a stale index, an unusually poorly-indexed document, an adversarially poisoned corpus entry). CRAG catches exactly that case, since it always looks at actual retrieval output — but it can't save the cost of an unnecessary retrieval call the way Adaptive RAG's upfront skip can, since retrieval has already happened by the time CRAG's evaluator runs.

Answer:

The distinctive mechanism is an explicit, independent quality check on retrieved content before it reaches the generator, with a defined corrective action (refine, supplement with web search, or discard and fully replace) for each quality tier — standard RAG has no equivalent step at all; whatever the retriever returns is unconditionally passed to the generator. This single addition is what gives CRAG its headline robustness property (Q14): a generator that never sees clearly-irrelevant retrieved content can't be misled by it in the specific way standard RAG's ungated pipeline can.

This is architecturally the same "generate-then-verify" discipline used elsewhere in this bank (Self-RAG's #07 reflection tokens, Astute RAG's #48 consolidation), applied specifically at the retrieval-quality checkpoint rather than at the final-answer checkpoint — CRAG catches a problem before generation ever happens, which is cheaper to correct than catching a bad answer after the fact and having to regenerate it.

Answer:

The AMBIGUOUS tier is the design choice that distinguishes CRAG from a simpler binary relevant/irrelevant filter: many real retrieval results are neither cleanly correct nor cleanly wrong — a document that's topically relevant but missing a specific needed detail is common, and treating it as fully correct risks an incomplete answer while discarding it entirely wastes genuinely useful partial context. The AMBIGUOUS branch's blend-with-web-search approach captures value from both sources rather than forcing a binary keep-or-discard decision on content that doesn't cleanly fit either category.

Answer:

Build a golden set specifically including cases with deliberately poor retrieval (queries against a stale or sparse section of the corpus, ambiguous queries with only partially-relevant matches, and queries where the correct answer requires the web-search fallback) alongside cases with normally-good retrieval — a benchmark that's all easy, well-retrieved queries won't exercise CRAG's evaluator or corrective branches at all, and will show no measurable difference from standard RAG regardless of whether the evaluator is well-calibrated.

Track answer accuracy for standard RAG vs. CRAG on this set, segmented by which verdict CRAG's evaluator actually assigned (CORRECT/AMBIGUOUS/INCORRECT, Q17) — CRAG's advantage should concentrate specifically in the AMBIGUOUS and INCORRECT segments, where standard RAG has no mechanism to avoid using poor context; on the CORRECT segment, CRAG and standard RAG should perform comparably, since the evaluator's action there (light refinement) shouldn't meaningfully change the outcome. If CRAG doesn't show a clear advantage on the AMBIGUOUS/INCORRECT segments specifically, that's a strong signal the evaluator itself is miscalibrated (Q19) rather than that CRAG's architecture doesn't help.

Answer:

An evaluator trained or prompted with a bias toward certain surface features — for example, consistently rating longer documents as CORRECT regardless of whether their length reflects genuine relevance, or consistently rating documents containing many of the query's literal keywords as CORRECT even when the semantic content doesn't actually answer the question — produces verdicts that look reasonable in aggregate accuracy metrics but fail specifically and predictably on the query types that trigger the blind spot.

Detection: segment evaluator verdict accuracy (compare the evaluator's verdict against human judgment on a labeled sample, per Q18's evaluation approach) by document features suspected of triggering bias — length, keyword density, source type — rather than only tracking aggregate verdict accuracy, since a bias affecting a specific document-feature segment can be invisible in an aggregate number dominated by unaffected documents. Mitigation: if a bias is confirmed, retrain or reprompt the evaluator with training/example data specifically balanced against the biased feature (e.g., include short-but-correct and long-but-irrelevant documents in equal measure) rather than adjusting the confidence threshold globally, since a global threshold change doesn't fix a bias tied to a specific document characteristic.

Answer:

Requirements: a support knowledge base that's occasionally stale or incomplete for newly-released features; when internal documentation doesn't cover a question, the system should fall back to the vendor's public web documentation rather than answering from unsupported guesses.

1. Retrieval: standard dense retrieval over the internal support KB.

2. Evaluator (Q3, Q17): a lightweight fine-tuned or prompted judge
   scores retrieved KB articles CORRECT/AMBIGUOUS/INCORRECT relative
   to the user's question.

3. Branch logic:
   - CORRECT: refine (Q7) and answer directly from internal KB --
     fastest, cheapest path, used for the majority of well-covered
     questions.
   - AMBIGUOUS: internal KB partially covers the topic (e.g., an older
     version of the feature) -- supplement with a web search against
     the vendor's public docs (Agentic Web RAG's #31 search pattern)
     for the missing specifics, blending both sources in the answer.
   - INCORRECT: internal KB has nothing relevant -- fall back entirely
     to public web documentation, clearly flagging to the user that
     this answer comes from public docs rather than internal support
     material, since internal-only nuances (account-specific details)
     won't be present.

4. Cost control (Q11): batch or cache evaluator calls where possible,
   since the evaluator runs on every query regardless of branch outcome
   -- this is the one cost CRAG always pays, so keeping it cheap
   matters more than optimizing the (less frequently triggered) web
   fallback path.

5. Monitoring: track the verdict distribution over time -- a rising
   INCORRECT/AMBIGUOUS rate signals the internal KB is falling behind
   product changes and needs a content update, giving support-content
   maintainers a data-driven signal for what to prioritize updating.

The key design value is that CRAG's verdict distribution doubles as an operational signal for content gaps, not just a runtime correction mechanism — tracking which queries trigger AMBIGUOUS/INCORRECT verdicts over time tells the support content team exactly where the internal KB needs updates, turning the evaluator into a continuous content-quality feedback loop.

Answer:

The scale here is tiny — a hobbyist project, a small trusted cookbook corpus — but the specific concern (a wrong food-safety fact reaching the user) is exactly the kind of risk CRAG's evaluator step exists to catch, independent of query volume or production scale. That's the useful distinction to draw: CRAG's value proposition (Q14) is about never letting clearly-wrong retrieved content reach the generator unchecked, and that value doesn't disappear just because the deployment is small.

A lightweight, prompted evaluator (no fine-tuning needed for a hobby project) that scores retrieved cookbook passages CORRECT or INCORRECT against the specific factual claim in the query (Q17) is proportionate here. Skip the AMBIGUOUS tier and the web-search fallback entirely — at this scale, an INCORRECT verdict can simply produce "I couldn't verify this against my cookbook" rather than triggering a live external search, which would add an API dependency this project doesn't need.

The trade-off: without a web-fallback branch, the assistant can't answer anything genuinely missing from the small trusted corpus, which is a real limitation compared to the full three-branch design (Q17) — but for a weekend project, "I don't know" is a perfectly acceptable answer, and it's a much safer default than confidently stating an unverified cooking temperature.

Answer:

During an active outbreak, guidance can change by the hour, and the failure mode that matters most isn't topical irrelevance — it's a retrieved document that's topically CORRECT (it accurately describes what the guidance used to say) but has been quietly superseded. Standard CRAG's CORRECT/AMBIGUOUS/INCORRECT verdict (Q17) has no dimension for that at all, since it scores relevance, not recency.

Extend the evaluator to check a document's version or publish date against the latest known revision for that specific guidance topic, treating "topically relevant but outdated" as its own branch rather than folding it silently into CORRECT. That branch should always trigger a check against the agency's own live guidance feed specifically — not general web search — given the accuracy stakes of public health guidance. During an active outbreak, tighten the staleness threshold so guidance verified more than a few hours ago escalates to a live check; the default recency tolerance this file assumes for a stable content period (Q17) is too loose for an active event.

What to monitor: the rate of stale-but-topically-correct verdicts over time as a leading indicator that a guidance topic has changed faster than the internal KB has updated — the same idea as Q20's "verdict distribution as a content-gap signal," but tuned to detect time lag rather than topic gaps — and periodic audits against the known revision history to catch any false negatives (stale content that scored CORRECT). The trade-off: recency checking on every query adds latency and cost beyond topical-relevance scoring alone, but during an active outbreak, the cost of serving superseded guidance is far higher than that overhead.