11 — Adaptive RAG

Answer:

Adaptive-RAG (Jeong et al., 2024, arXiv:2403.14403) is a retrieval strategy that dynamically routes queries to different retrieval depths based on a query-complexity classifier (a trained T5-large in the original paper), rather than applying a uniform pipeline to all queries.

Fixed-pipeline RAG executes the same retrieval strategy (e.g., always retrieve top-k, always do multi-hop) regardless of the query. Adaptive RAG uses a learned classifier to predict query complexity and routes accordingly:

• No-retrieval path — For simple queries (e.g., "What is X?") that the LLM can answer from parametric knowledge
• Single-hop path — For moderately complex queries requiring one retrieval step
• Multi-hop path — For complex reasoning queries requiring multiple retrieval and reasoning steps

This reduces latency and cost for simple queries while maintaining answer quality for hard questions. It is particularly effective when query complexity varies widely in production workloads.

Answer:

A query complexity classifier predicts whether a given query is simple, moderate, or complex, using features like query length, linguistic markers, and semantic similarity to known simple/complex templates.

Common approaches:

1. Keyword-based heuristics — Flag queries with multi-hop keywords (e.g., "compare", "across") or multiple entities as complex.
2. Supervised classifier — Train a linear or neural model on labeled (query, complexity) pairs. Features can include:
   • Query length and token count
   • Presence of comparative/temporal/causal keywords
   • Named entity count
   • Embedding similarity to known simple vs. complex query templates
3. LLM-as-judge — Use a lightweight LLM or prompt to estimate complexity, balancing cost vs. accuracy.
4. Confidence-based — Let the LLM attempt to answer without retrieval and measure confidence. If confidence is below a threshold, escalate to retrieval.

Training data — Typically 500–2000 labeled (query, complexity, answer quality with/without retrieval) triplets from past user interactions or synthetic data.

Silver labels (the Adaptive-RAG paper approach): Instead of manual annotation, label each query by the cheapest strategy that actually answered it correctly. Run all three strategies offline on a benchmark set:

If no-retrieval output matches gold answer        → label A (simple)
Else if single-step retrieval matches gold answer → label B (moderate)
Else                                              → label C (complex)

For queries where no strategy succeeds, the paper falls back to dataset-bias labels (queries from single-hop datasets like SQuAD/NQ → B; from multi-hop datasets like HotpotQA/MuSiQue → C). The paper then fine-tunes a T5-large classifier on these silver labels — no human annotation needed, and labels reflect your actual system's capability, not abstract query difficulty.

The classifier runs before retrieval, so it must be fast (<10ms overhead to be practical).

Answer:

The three strategies and their selection criteria are:

Routing logic:

1. Classifier produces a complexity score (0–1) for the input query.
2. Use threshold-based routing: if score < t1 → no-retrieval; if t1 ≤ score < t2 → single-hop; if score ≥ t2 → multi-hop.
3. Thresholds are tuned on a held-out validation set balancing latency, cost, and answer quality.

Threshold tuning methodology:

1. Hold out 500–1000 queries with gold answers.
2. Precompute once, offline: run all three strategies on every held-out query and record (quality, cost, latency) per strategy. This makes the sweep free — no re-running pipelines per threshold candidate.
3. Sweep (t1, t2) over a grid and compute expected quality/cost under each setting:

import itertools

results = []
for t1, t2 in itertools.product(grid, grid):
    if t1 >= t2:
        continue
    route = lambda s: "none" if s < t1 else ("single" if s < t2 else "multi")
    quality = mean(q[route(score(q))].f1 for q in heldout)
    cost = mean(q[route(score(q))].cost for q in heldout)
    results.append((t1, t2, quality, cost))

4. Plot the cost vs. quality frontier (each (t1, t2) pair is a point; keep only Pareto-optimal points). Pick the knee of the curve, or the max-quality point under a cost/latency budget:

F1
0.86 │                          ●  (t1=0.2, t2=0.5)  ← quality-max
0.84 │              ●  (t1=0.3, t2=0.6)  ← knee, usually best
0.80 │      ●  (t1=0.4, t2=0.8)
0.74 │  ●  (t1=0.6, t2=0.9)  ← cost-min
     └──────────────────────────── Cost/query
       $0.005  $0.012  $0.020  $0.03

5. Re-run the sweep whenever the classifier is retrained or the query distribution shifts — thresholds tuned for one score distribution are stale after recalibration.

Misclassification asymmetry: routing a complex query to no-retrieval destroys answer quality; routing a simple query to multi-hop only wastes money. So bias t1 low (escalate when in doubt) and rely on cost controls rather than quality controls for the upper tier.

End-to-end flow:

User Query
    │
    ├─ Complexity Classifier
    │     │
    │     ├─ Score < 0.33 → Direct LLM generation (no retrieval)
    │     ├─ 0.33 ≤ Score < 0.66 → Retrieve 1x, then generate
    │     └─ Score ≥ 0.66 → Iterative multi-hop retrieval + generation
    │
    └─ Answer

Answer:

FLARE (Forward-Looking Active Retrieval Augmented Generation) is a method that triggers retrieval dynamically during generation, based on the model's predictive uncertainty about upcoming tokens. Adaptive RAG can integrate FLARE to refine its initial routing decision or to escalate from no-retrieval to retrieval mid-generation.

Integration approach:

1. Initial routing — Complexity classifier routes to no-retrieval or single-hop as usual.
2. During generation — Monitor the LLM's confidence on each generated token.
3. Uncertainty trigger — If confidence drops below a threshold (e.g., next token probability < 0.5), pause generation and retrieve documents related to the low-confidence phrase.
4. Augment context — Append retrieved documents to the prompt and resume generation.

Benefits:

• Catches cases where the classifier underestimated complexity.
• Reduces unnecessary retrieval for simple questions (no-retrieval path is tried first).
• Enables late-stage correction if the LLM begins hallucinating.

Example: A query "Who won the 2024 World Cup?" is classified as simple. The LLM starts: "As of my training data..." but uncertainty spikes when generating the year. FLARE triggers retrieval of recent sports news and corrects the answer.

Combining Adaptive (upfront routing) + FLARE (runtime uncertainty) yields the best latency and quality balance.

Answer:

Training data collection:

1. Sample 500–2000 queries from your production logs or a representative dataset.
2. Label each query with a complexity class: Simple (0), Moderate (1), or Complex (2). Two complementary sources:
   • Manual annotation — Simple: facts, definitions, single-entity questions. Moderate: comparisons, aggregations, single-hop reasoning. Complex: multi-hop reasoning, temporal reasoning, constraint satisfaction.
   • Silver labels from past system behavior (cheaper, scales better) — replay logged queries through all three strategies offline and label each with the cheapest strategy whose answer was correct (exact match / F1 vs. gold, or LLM-judge vs. the accepted production answer). This is the Adaptive-RAG paper's labeling scheme and automatically reflects your LLM's parametric knowledge: a query is "simple" only if your model answers it without retrieval.
3. Deduplicate near-identical queries and stratify the train/test split by class — production logs skew heavily toward simple queries, and an unstratified split under-trains the complex class.

Classifier architecture:

A lightweight model such as:

• Logistic regression on handcrafted features (length, entity count, keyword TF-IDF).
• Shallow neural net (1–2 hidden layers) on query embeddings.
• Fine-tuned small LM (e.g., DistilBERT, MobileBERT) for more accuracy at higher cost.

For production, prefer simpler models with <10ms latency.

Training procedure:

from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier

X_train, X_test, y_train, y_test = train_test_split(
    query_features, complexity_labels, test_size=0.2
)

clf = RandomForestClassifier(n_estimators=50, max_depth=5)
clf.fit(X_train, y_train)

accuracy = clf.score(X_test, y_test)

Calibration (do not skip this):

Routing thresholds only mean something if the classifier's probabilities are calibrated — a raw softmax score of 0.9 from an overconfident neural classifier may correspond to only 70% empirical accuracy. Apply temperature scaling on a held-out calibration set:

import numpy as np
from scipy.optimize import minimize_scalar

def nll(T, logits, labels):
    probs = np.exp(logits / T) / np.exp(logits / T).sum(axis=1, keepdims=True)
    return -np.log(probs[np.arange(len(labels)), labels]).mean()

T_opt = minimize_scalar(nll, bounds=(0.5, 5.0), method="bounded",
                        args=(val_logits, val_labels)).x
# Serve calibrated probs: softmax(logits / T_opt)

Verify with a reliability diagram or Expected Calibration Error (ECE) before and after. Then add a confidence gate: if the calibrated max-class probability is below a floor (e.g., 0.6), do not trust the prediction — route to the safe middle tier (single-hop) instead.

Fallback behavior on misclassification:

• Complex → simple misroutes (quality risk): catch at runtime with generation-confidence escape hatches — if the no-retrieval answer's token-level confidence is low or FLARE triggers (see Q4/Q6), escalate to retrieval rather than returning the answer.
• Simple → complex misroutes (cost risk): tolerate them; they only waste money. Cap multi-hop iterations and log them for retraining.
• Classifier unavailable or low-confidence: default everything to single-hop — it is the strategy with the best worst-case quality/cost balance.
• Log every escalation (predicted tier ≠ tier that finally produced the answer) — these are free hard-negative training examples for the next classifier version.

Evaluation metrics:

• Accuracy — Overall classification correctness.
• Per-class precision/recall — Ensure the classifier does not systematically mis-label complex queries as simple (dangerous for quality).
• Latency — Measure classifier inference time in the serving pipeline.
• End-to-end RAG quality — Log F1/NDCG/answer-quality metrics binned by predicted complexity; verify that simpler queries routed to no-retrieval maintain acceptable quality.

Monitor classifier performance quarterly and retrain as query patterns shift.

Answer:

Confidence-based no-retrieval skipping leverages the LLM's token-level confidence estimates to decide, on a per-query basis, whether retrieval is necessary. Queries the model feels confident about skip retrieval entirely, cutting latency by ~500ms–2s and eliminating embedding/vector DB costs.

Implementation:

1. First-pass generation without retrieval — Feed the query directly to the LLM and collect output tokens with their probabilities.
2. Aggregate confidence — Compute a single confidence score from the token probabilities:

   confidence = exp(mean(log(p_i)))  # geometric mean of token probs
   

   Or use the minimum token probability as a conservative lower bound.
3. Confidence threshold — If confidence > threshold (e.g., 0.7), return the answer. Otherwise, retrieve and regenerate.
4. Empirical threshold tuning — On a validation set, measure F1/BLEU for each threshold and pick the one maximizing F1 subject to a latency constraint.

Cost and latency impact:

For a 30% skip rate, total inference cost drops ~15% and median latency improves ~300ms.

Trade-offs:

• False positives — Low confidence on a query the model can actually answer (requires retrieval unnecessarily). Mitigation: conservative threshold.
• False negatives — High confidence on a query the model cannot answer well (skips retrieval when needed). Mitigation: use a mixture of confidence signals (token probs + semantic uncertainty).

Confidence-based skipping pairs well with the Adaptive RAG classifier: the classifier provides a coarse routing decision, and confidence gates fine-grained skipping within each tier.

Answer:

Self-consistency scoring evaluates multiple candidate retrieval sequences (e.g., different intermediate query formulations, different retrieve-and-think steps) and selects the trajectory that produces the most consistent and coherent intermediate reasoning, without requiring ground-truth labels.

Approach:

1. Candidate generation — For a complex query, generate k different chains of retrieval + reasoning:

   • Retrieval sequence A: retrieve for subquery A1 → reason → retrieve for A2 → answer.
   • Retrieval sequence B: retrieve for subquery B1 → reason → retrieve for B2 → answer.
   • ... (e.g., k=3).

2. Self-consistency metrics — Score each trajectory by:

   • Semantic coherence — Measure how well consecutive reasoning steps align (embeddings of consecutive reasoning strings are near each other in latent space).
   • Document relevance agreement — Score how much the retrieved documents at each step reinforce each other (use citation overlap, entity overlap, or embedding similarity).
   • Answer stability — If you regenerate the answer from each trajectory, how similar are the final answers? (e.g., BLEU or semantic similarity).

3. Voting / aggregation — Pick the trajectory with the highest aggregate score, or ensemble the answers from all trajectories.

Example implementation:

from sentence_transformers import CrossEncoderModel

def score_trajectory(reasoning_steps, retrieved_docs):
    coherence_score = 0.0
    for i in range(len(reasoning_steps) - 1):
        # Cross-encoder score between consecutive reasoning steps
        score = cross_encoder.predict(
            [[reasoning_steps[i], reasoning_steps[i+1]]]
        )[0]
        coherence_score += score
    
    doc_relevance = sum(
        cross_encoder.predict([[step, doc] for doc in retrieved_docs])
        for step in reasoning_steps
    ) / len(retrieved_docs)
    
    return coherence_score + doc_relevance

best_trajectory = max(
    trajectories, 
    key=lambda t: score_trajectory(t['steps'], t['docs'])
)
return best_trajectory['answer']

Benefits:

• Reduces hallucination in multi-hop reasoning without requiring an oracle.
• Adapts to new query types (no task-specific fine-tuning needed).

Overhead: Generating k trajectories multiplies compute cost by k; typically k=2–3 is practical.

Answer:

Empirical trade-off curve:

For a typical production workload:

F1
│
0.95 │                  Multi-Hop ●
     │
0.85 │        Single-Hop ●
     │
0.75 │ No-Retrieval ●
     │
0.65 │
     └──────────────────────────── Latency (ms)
       0    600   1200   2000

Optimal operating point:

• Query complexity distribution heavily skews simple → weighted average latency favors no-retrieval path.
• If 50% simple, 30% moderate, 20% complex → median latency ≈ 0.5×50 + 0.3×600 + 0.2×2000 ≈ 650ms.

Strategies to optimize each tier:

• No-retrieval — Use a smaller, faster LLM (e.g., Llama 7B vs. 70B). Add a re-ranker to catch low-confidence predictions.
• Single-hop — Cache embeddings of frequently-retrieved documents. Use approximate nearest neighbor search (HNSW, Annoy).
• Multi-hop — Prune intermediate retrieval results aggressively. Use speculative decoding to speed up verifier LLM.

Cost breakdown for 1M queries/month with 50/30/20 distribution:

• No-retrieval (500k): $0.
• Single-hop (300k): $0.015 × 300k ≈ $4,500.
• Multi-hop (200k): $0.075 × 200k ≈ $15,000.
• Total: ~$19,500/month.

The no-retrieval path is the biggest lever for cost reduction.

Answer:

The Adaptive-RAG paper proposes a set of metrics to jointly evaluate routing accuracy, retrieval efficiency, and final answer quality:

1. Routing Accuracy (RA):

Measure how often the classifier correctly routes a query to the optimal strategy:

RA = (# correctly routed queries) / (total queries)

A query is "correctly routed" if:

• Assigned to no-retrieval AND the LLM can answer it without retrieval (confidence > threshold or F1 match).
• Assigned to single-hop AND retrieval+generation beats no-retrieval but multi-hop yields only marginal gains.
• Assigned to multi-hop AND multi-hop significantly outperforms single-hop.

Define "correctly" empirically using a small ground-truth validation set with oracle labels.

2. Retrieval Efficiency (RE):

Measure the fraction of queries that skipped retrieval:

RE = (# no-retrieval queries) / (total queries)

Higher RE = more cost savings. A system with 40% RE avoids 40% of retrieval calls.

3. Answer Quality (AQ) — Per-tier:

Report F1, BLEU, or ROUGE for each strategy tier:

F1_no_ret = F1(answers on no-retrieval queries)
F1_single = F1(answers on single-hop queries)
F1_multi = F1(answers on multi-hop queries)

Ensure no-retrieval F1 is still acceptable (not degraded due to incorrect routing).

4. Overall Quality vs. Cost:

Define a joint metric:

AdaptiveQuality = (1 - α) × AQ_overall + α × (1 - normalized_cost)

where α ∈ [0, 1] trades off answer quality vs. cost. Adaptive-RAG should maximize this metric relative to a fixed pipeline (single-hop or multi-hop baseline).

Evaluation protocol:

1. Collect ~500 queries with ground-truth answers.
2. Bin queries by oracle complexity (simple, moderate, complex).
3. For each query, run all three strategies (no-ret, single-hop, multi-hop) and measure quality.
4. Train the classifier to predict oracle complexity.
5. Evaluate:
   • Routing Accuracy = % of queries the classifier routes to the oracle-optimal tier.
   • Retrieval Efficiency = % routed to no-retrieval.
   • Quality per tier = F1/BLEU for each routed subset.
   • Overall cost and latency.

Example results:

Routing Accuracy: 87%
Retrieval Efficiency: 45%
F1 (no-retrieval): 0.72
F1 (single-hop): 0.81
F1 (multi-hop): 0.91
Overall F1: 0.82
Cost per query: $0.035 (vs. $0.10 for always-multi-hop)

Answer:

Architecture:

User Query (500 QPS)
    │
    ├─ [Classification Service] (inference, batch size 32)
    │     │
    │     └─ Outputs: complexity score, confidence interval
    │
    ├─────────────────────────┬──────────────────────┬─────────────────┤
    │                        │                      │                 │
    ▼                        ▼                      ▼                 ▼
[Direct LLM]          [Retrieval SVC]       [Multi-Hop Orchestrator]
(No-Retrieval)        (Single-Hop)          (Multi-Step + Reasoning)
~100ms, $0/query      ~600ms, $0.015/q      ~2000ms, $0.075/q
    │                        │                      │                 │
    └────────────────────────┴──────────────────────┴─────────────────┘
                                    │
                                    ▼
                            [Response Aggregator]
                                    │
                                    ▼
                            [Client Response]

Component design:

1. Classification Service:

   • Model: Lightweight classifier (e.g., DistilBERT or logistic regression).
   • Batch size: 32 (allows 500 QPS with ~16ms latency per batch).
   • Container: GPU-optimized inference (NVIDIA Triton or vLLM).
   • Replicas: 2–3 for HA. Scaling rule: add replica if latency p99 > 50ms.

2. Direct LLM Service (No-Retrieval Tier):

   • Model: Smaller, faster LLM (e.g., Llama 7B) or a cached/quantized version of the main model.
   • Throughput: 500 tokens/sec × 32 batch size ≈ 16K tokens/sec. Use vLLM or TensorRT for fast batch inference.
   • Replicas: 1–2. Most queries route here, so this is the throughput bottleneck.

3. Retrieval Service (Single-Hop Tier):

   • Query embedding: Fast embedding model (e.g., BGE-small, <10ms).
   • Vector DB: Qdrant or Weaviate with approximate nearest neighbors (HNSW). Cache top-100 embeddings per day to reduce latency.
   • Batch retrieval: Retrieve for ~100 queries in parallel; typical response: <300ms.
   • Replicas: 2–3 to handle 30% of queries (~150 QPS).

4. Multi-Hop Orchestrator (Complex Queries):

   • Implement as a LangGraph workflow or custom agentic loop.
   • Parallel retrieval: Fan out multiple sub-queries to the Retrieval Service.
   • Result caching: Cache common sub-queries (e.g., "What is X?") across requests.
   • Replicas: 1–2 for the remaining ~100 QPS (20% of traffic).

Load balancing:

Load Balancer (nginx / AWS ALB)
    │
    ├─ Classify 500 QPS (round-robin to 2–3 classifiers)
    │
    └─ Route by predicted complexity:
        ├─ 50% → Direct LLM (2–3 replicas)
        ├─ 30% → Retrieval Service (2–3 replicas)
        └─ 20% → Multi-Hop (1–2 replicas)

Latency SLO and monitoring:

• P50 latency target: 400ms (30% on fast path, 70% blended).
• P99 latency target: 3000ms (multi-hop queries).
• Metrics to log:
  • Classifier latency, accuracy per bin.
  • Per-tier throughput, latency, cost/query.
  • Answer quality (F1) per tier.
  • Cache hit rate (for retrieval and multi-hop).

Cost estimation (monthly, 1.3B queries at 500 QPS):

• Classifier inference: $100 (GPU time, shared).
• Direct LLM: 650M queries × $0.0001/query (batch inference) ≈ $65K.
• Retrieval (embedding + vector DB): 390M queries × $0.015 ≈ $5.8K.
• Multi-hop (2–3 LLM calls per query): 260M queries × $0.06 ≈ $15.6K.
• Total: ~$87K/month (vs. $130K for always-multi-hop).

Failure modes and recovery:

• Classifier outage → default to single-hop for all queries (safe fallback).
• Retrieval timeout → escalate to multi-hop or return LLM-only answer with lower confidence flag.
• Multi-hop timeout (>5s) → return best-effort answer from intermediate steps.

Answer:

Adaptive RAG pays the classifier cost on every query in exchange for savings on the fraction of queries it downgrades to cheaper tiers. The system is only worth running when:

C_classifier + E[cost of misroutes] < E[savings from cheaper routing]

Cost of classifier options:

The LLM-based router is 100–1000x more expensive per query than a small BERT and adds visible latency to every request — it only makes sense for low-traffic, high-value workloads or for bootstrapping labels to train a cheap classifier.

Amortization math — worked example (1M queries/month):

Baseline: always-multi-hop at $0.075/query → $75,000/month.

Adaptive with a small BERT classifier, 50/30/20 routing (no-ret $0 / single-hop $0.015 / multi-hop $0.075):

Classifier:   1M × $0.00001                  =     $10
No-retrieval: 500K × $0.00                   =      $0
Single-hop:   300K × $0.015                  =  $4,500
Multi-hop:    200K × $0.075                  = $15,000
Total                                        ≈ $19,510/month  (74% savings)

The classifier itself is 0.05% of total spend — its inference cost is noise. The real costs to watch are:

1. Misroute cost — at 87% routing accuracy, ~13% of queries are misrouted. Simple→complex misroutes waste money (~upper bound: 13% × $0.075 × 1M ≈ $9.7K worst case); complex→simple misroutes cost quality, which can dwarf dollar costs (bad answers, escalations, lost trust). Always quality-adjust the savings claim.
2. GPU amortization — a dedicated T4 at ~$0.50/hr serving the classifier is ~$360/month regardless of volume. At 1M queries/month that's $0.00036/query; at 100M it's negligible. Self-hosted classifiers have a fixed floor, API-based ones scale to zero.
3. Maintenance — labeling, retraining, threshold re-sweeps: budget engineer-hours, not just compute.

When routing overhead outweighs savings:

• Homogeneous workloads — if 95% of queries are genuinely complex (e.g., a research-assistant product), the classifier saves on only 5% of traffic; a fixed multi-hop pipeline is simpler and nearly as cheap.
• Cheap downstream pipeline — if single-hop costs $0.001/query, even perfect routing saves fractions of a cent; an LLM-based router would cost more than the pipeline it gates.
• Strict latency budgets — an LLM router adding 500ms to a 600ms pipeline is an 80% latency tax on every request, including the ones it doesn't help.
• Low traffic — below ~100K queries/month, fixed costs (GPU floor, retraining, monitoring) usually exceed the routed savings.

Caching routing decisions:

import hashlib

def route_with_cache(query: str) -> str:
    # Exact-match cache on normalized query
    key = hashlib.sha256(normalize(query).encode()).hexdigest()
    if key in route_cache:
        return route_cache[key]                      # ~free

    # Semantic cache: reuse route for near-paraphrases
    emb = embed(query)                               # cheap model, ~1ms
    hit = semantic_cache.search(emb, min_sim=0.95)
    if hit:
        return hit.route

    route = classifier.predict(query)
    route_cache[key] = route
    semantic_cache.add(emb, route)
    return route

• Production query streams are repetitive: 20–40% exact/semantic hit rates are common, cutting classifier load proportionally.
• Cache the route, not the answer — the answer may be stale, but "this query shape needs single-hop" is stable.
• Set a TTL and flush the cache when the classifier is retrained or thresholds change, otherwise old routing decisions outlive the model that made them.

Rule of thumb: keep classifier cost under ~1% of average per-query pipeline cost. A small supervised model nearly always satisfies this; an LLM-as-router rarely does at scale.

Answer:

The routing classifier is a new, externally reachable decision surface: every user-controlled query passes through it, and its output changes both answer quality and money spent. That makes it a target for two opposite attack classes.

Attack 1: Downgrade attack (force no-retrieval → elicit hallucinations)

The attacker phrases knowledge-dependent questions in the surface form of simple factoid queries, so the classifier routes them to the no-retrieval path and the LLM answers from (insufficient) parametric knowledge:

Honest query:      "What does our 2026 enterprise SLA say about refunds
                    for multi-region outages?"          → multi-hop (grounded)

Adversarial query: "Define the refund policy."          → no-retrieval
                   "What is the 2026 SLA refund rule?"  → no-retrieval (factoid shape)

LLM answers confidently from parametric guesswork → hallucinated policy text.

Why it matters: the attacker can harvest authoritative-sounding fabrications (to screenshot, to mislead other users in shared channels, or to probe what the base model "believes" about private topics), and the system skips exactly the grounding step that would have caught it. Classifiers trained on surface features (length, question words, entity count) are especially easy to steer this way.

Attack 2: Complexity-inflation attack (cost / resource DoS)

The opposite direction — stuff queries with multi-hop trigger features ("compare", "across", "considering", many named entities, nested clauses) so every query lands on the most expensive path:

"Compare X and Y across A, B, and C, considering D, E, and F,
 and how each evolved relative to G..."   → multi-hop, every time

Economics: cheap path $0.00/query vs. multi-hop $0.075/query (~50–100x).
10K adversarial queries/day × $0.075 ≈ $750/day ≈ $22K/month of attacker-
controlled spend — an economic DoS that also saturates multi-hop
orchestrator capacity and degrades latency for legitimate complex queries.

Attack 3: Boundary probing

Tier latencies differ by an order of magnitude (~100ms vs. ~2s), so response time is a timing side channel revealing which tier ran. An attacker can binary-search query phrasings to map the routing thresholds, then reliably sit just on the cheap side (for downgrades) or expensive side (for inflation) of the boundary.

Defenses:

1. Calibrated confidence thresholds + fallback to the safe path

Never act on a low-confidence routing decision, and make the uncertain default the grounded middle tier — not the cheap tier:

def route(query: str) -> str:
    probs = calibrated_classifier.predict_proba(query)  # temperature-scaled
    label, conf = probs.argmax(), probs.max()

    if conf < 0.60:
        return "single_hop"          # uncertain → safe, grounded default
    if label == "simple" and conf < 0.80:
        return "single_hop"          # extra-strict bar for skipping retrieval
    return TIERS[label]

The asymmetric bar matters: a downgrade attack must now push the classifier to ≥0.8 calibrated confidence on "simple", not merely win the argmax.

2. Policy overlay before the classifier

Rules outrank the model. Queries touching dynamic, private, or high-stakes domains (policies, pricing, anything post-training-cutoff, anything matching tenant document namespaces) are never eligible for no-retrieval, regardless of classifier output. The downgrade attack then can't reach the vulnerable path at all for the content that matters.

3. Cost guards against inflation

• Per-user / per-tenant budget caps and rate limits on the multi-hop tier specifically (not just global QPS).
• Hard cap on hops and per-query token budget; degrade to single-hop with a "partial answer" flag rather than looping.
• Require account trust level (age, payment status) for unmetered multi-hop access; route anonymous traffic to capped tiers.

4. Monitor routing-distribution drift

A routing attack is a distribution shift. Track tier shares per window and per user segment against a trusted baseline, e.g., with Population Stability Index:

def psi(baseline: dict, current: dict) -> float:
    return sum(
        (current[t] - baseline[t]) * math.log(current[t] / baseline[t])
        for t in ["none", "single", "multi"]
    )

# baseline = {"none": 0.50, "single": 0.30, "multi": 0.20}
# PSI > 0.2 on an hourly window → alert.
# Slice by user/tenant/IP: a single tenant at 95% multi-hop share is a
# stronger signal than a global shift.

Also alert on: spike in no-retrieval share for queries containing tenant-document entities (downgrade signature), and spike in escalations (FLARE/confidence triggers firing after a "simple" route — the classifier is being beaten).

5. Adversarial training and consistency checks

• Add attack-shaped examples (factoid-phrased private-knowledge questions labeled complex; keyword-stuffed simple questions labeled simple) to the classifier's training set each retraining cycle.
• Paraphrase consistency test: route 2–3 cheap paraphrases of suspicious queries; if routes disagree, take the most conservative route and log the query.

6. Catch successful downgrades post-hoc

Defense-in-depth for when the classifier is beaten anyway: on the no-retrieval path, run a lightweight groundedness/confidence check on the generated answer (token-probability floor, or FLARE-style uncertainty trigger from Q4/Q6) and escalate to retrieval before returning. The attacker must now fool the classifier and the generation-time gate.

Attack → defense map:

The principle: the classifier is an optimization, never a safety control. Quality and cost guarantees must hold even when the router gives the worst possible answer.

Answer:

Yes — this is the problem TARG ("Training-Free Adaptive Retrieval Gating for Efficient RAG," 2025) targets directly. Where Adaptive-RAG's router is a trained classifier (Q2, Q5) that has to be built, labeled, and periodically retrained, TARG makes the retrieve-or-not decision without any training at all, using signals already available from the base LLM.

Mechanism:

1. Generate a short draft answer with no retrieved context (the model answering from parametric knowledge alone).
2. Measure the model's uncertainty on that draft using signals already exposed by decoding — token entropy and logit margin (the gap between the top and second-ranked token probabilities) — no separate classifier or training pass required.
3. If uncertainty is low, the draft is trusted and returned as-is (no retrieval).
4. If uncertainty is high, retrieval is triggered and the answer is regenerated with the retrieved context.

Reported effect: cuts retrieval calls by roughly 70–90% relative to always-retrieve, while remaining training-free — no labeled complexity data, no classifier maintenance, no retraining cadence to manage.

How this compares to Adaptive-RAG and FLARE:

When to reach for TARG over a trained router: you don't have (or don't want to maintain) labeled query-complexity data, or you're standing up adaptive retrieval quickly and can tolerate the extra no-context draft generation as the cost of skipping classifier training entirely. When to stick with a trained classifier instead: you're at high enough QPS that even a cheap disposable draft generation is expensive relative to a lightweight classifier's near-zero routing cost (Q11).

Answer:

Query
  │
  ▼
Query Complexity Classifier (trained, Q2)
  │
  ├── "no retrieval needed" ──► Answer directly from parametric knowledge
  ├── "single-hop" ──────────► One retrieval round → generate
  └── "multi-hop" ───────────► Iterative multi-round retrieval → generate

The classifier is the single component that makes this architecture "adaptive" — everything downstream of it (the three retrieval strategy tiers) is standard, well-understood machinery already covered elsewhere in this bank (single-hop retrieval as in Naive RAG #01, multi-hop as in Iterative Multi-Hop RAG #19). Adaptive RAG's contribution isn't a new retrieval mechanism, it's a cheap upfront decision about which existing mechanism to invoke, made once per query before any retrieval work begins — the opposite temporal position from Agentic RAG's (#04, Q17) continuous, per-step decision-making.

Answer:

Adaptive RAG was introduced by Jeong et al., Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity (arXiv:2403.14403, 2024), training a small classifier to route queries to one of three tiers — no-retrieval, single-hop, or multi-hop — based on question complexity, evaluated across QA benchmarks spanning both simple factual questions and genuinely multi-hop ones.

The paper's headline finding is an efficiency argument as much as an accuracy one: applying the most expensive strategy (multi-hop retrieval) uniformly to every query, including simple ones that don't need it, wastes latency and cost without improving accuracy on the easy end of the distribution, while applying the cheapest strategy uniformly to every query fails outright on genuinely complex questions. Routing each query to the tier its actual complexity warrants captures accuracy close to "always use the best-fit strategy per query" at a fraction of the average cost "always use the most powerful strategy" would require.

Answer:

Both decide how much retrieval work a query needs rather than applying one fixed strategy uniformly, but the timing and mechanism differ (this mirrors Agentic RAG's own comparison to this file, #04 Q17, from the other side). Adaptive RAG makes one classification decision upfront, before any retrieval happens, and commits to that tier's fixed strategy — cheap and fast, but unable to course-correct if the classifier's initial read of the query's complexity turns out to be wrong once retrieval actually starts. Agentic RAG makes this decision continuously throughout an LLM reasoning loop, able to escalate or stop at any point based on what's actually been discovered so far — more adaptive in principle, but at substantially higher per-query cost and with the runaway-loop risk (#04 Q19) that a single upfront classification can't have.

Choose Adaptive RAG when query complexity is reasonably predictable from surface features of the query text (its own classifier's whole premise); choose Agentic RAG when the right strategy can only be discovered by actually starting to retrieve and reason, which upfront classification structurally cannot anticipate.

Answer:

Adaptive RAG's classifier acts before retrieval, deciding which retrieval strategy to invoke based on the query alone. Corrective RAG's (#06) evaluator acts after retrieval, judging whether the results that were actually returned are good enough to use, and triggering a fallback (web search, query reformulation) if not. Adaptive RAG never looks at what retrieval actually returned before deciding its strategy; Corrective RAG never tries to predict retrieval quality in advance — it always retrieves first, then checks.

The two are complementary rather than competing: a production system could route queries via Adaptive RAG's classifier to pick an initial strategy tier, and layer Corrective RAG's evaluator on top of whichever tier's retrieval results come back, catching cases where the classifier's upfront read of complexity was reasonable but the specific retrieval attempt still came back poor (a stale index, a genuinely rare query the classifier couldn't anticipate) — using each architecture's check at the point in the pipeline where it's actually positioned to catch a different class of error.

Answer:

The confidence threshold is the most consequential knob because the two error directions have different costs: routing a genuinely simple query to the expensive multi-hop tier wastes latency and cost but doesn't hurt accuracy; routing a genuinely complex query to the cheap no-retrieval tier produces a wrong or unsupported answer with nothing downstream to catch it, which is why biasing toward the more expensive tier under classifier uncertainty is the safer default.

Answer:

A classifier trained on one query distribution can systematically misjudge a query type it saw little of during training — for example, a classifier trained mostly on general-knowledge QA may consistently under-classify domain-specific compound questions (routing a genuinely multi-hop technical question to the single-hop tier because its surface phrasing looks simple), producing confidently incomplete answers with no signal that anything went wrong, since the single-hop retrieval path completed "successfully" from the pipeline's perspective.

Detection: segment production answer-quality metrics by the classifier's assigned tier and, within each tier, by query category (domain, phrasing pattern) — a specific category showing systematically lower answer quality within the "single-hop" tier, while other categories in that same tier perform fine, is the signature of tier-specific miscalibration rather than a general retrieval-quality problem. Mitigation: augment the classifier's training data with labeled examples specifically from the under-performing category (Q5's training methodology, applied to the gap you've identified) rather than adjusting the confidence threshold globally, since a global threshold change trades off accuracy across all categories to fix a problem specific to one.

Answer:

A classifier trained once on an initial labeled dataset will drift out of calibration as real query patterns evolve — new product features generate new question types the original training data never saw, users adopt new phrasing conventions, or the corpus itself grows into new topic areas the classifier was never trained to route correctly. Unlike a retrieval index, which can be incrementally updated as documents change, a classifier's "knowledge" of what complexity levels look like is frozen at its last training run until it's explicitly retrained.

Practical approach: (1) continuously sample production queries (not just at launch) and periodically have them labeled — either by human review or by a more expensive "ground truth" method (e.g., running the query through all three tiers and checking which was actually necessary) — building an ever-growing, ever-more-representative labeled set rather than treating the original training data as permanent; (2) monitor the tier-distribution of production traffic over time, since a sudden shift (a spike in queries the classifier routes to no-retrieval, for instance) can signal either a genuine change in user behavior or a classifier that's starting to misjudge a growing query segment (Q19); (3) retrain on a cadence informed by observed drift rate rather than a fixed calendar schedule — a fast-evolving product surface needs more frequent classifier refreshes than a stable, slowly-changing knowledge domain.

Answer:

This is a small, low-stakes deployment with a clearly identifiable slice of traffic — hours, location, basic policy — that's simple, stable, and arguably better served by a fixed lookup than by running a full retrieval pass every time. That's precisely the no-retrieval tier this file describes (Q14): a lightweight classifier, or even a simple keyword/intent rule at this small scale, routes fixed-fact questions to a direct parametric or hardcoded answer, while genuinely open-ended questions ("which books do you have on medieval history") route to single-hop retrieval over the catalog.

Given the narrow, predictable set of simple-intent phrasings a library sees, a short list of rule-based triggers gets most of the benefit — faster, cheaper answers for the bulk of routine traffic — without the training-data investment a full classifier (Q15, Q18) would need.

The trade-off: a rule-based router won't generalize to new phrasings of the same simple question the way a trained classifier would, which is Q19's miscalibration concern in miniature, just handled manually instead of statistically. At this scale that's fine — watching for new simple-question patterns the fixed rules miss and adding them as they're noticed is enough oversight; it only becomes worth building a real classifier if traffic and question variety grow substantially.

Answer:

Twenty countries means twenty distinct query distributions, languages, and catalog structures layered on top of the classifier's usual job — a single global classifier trained mostly on one or two dominant markets will systematically miscalibrate on the rest, which is Q19's tier-specific-miscalibration failure mode recurring at country granularity instead of topic granularity.

Train and monitor the complexity classifier per country-cluster rather than as one global model, grouping countries with genuinely similar query patterns and catalog complexity rather than assuming uniform behavior across all twenty. Bias the confidence threshold toward the more expensive tier (Q18) more aggressively in newer or smaller markets with less training data, since under-routing risk is highest exactly where the classifier has seen the least. Restrict the no-retrieval tier to genuinely locale-independent facts, since even simple-seeming questions like "what's your return window" vary by country and shouldn't be answered from one shared parametric assumption.

What to monitor: routing accuracy and answer quality segmented by country (Q18's methodology, sliced by locale instead of tier alone), tier-distribution drift per country over time (Q20), since one market's query patterns can shift independently of the others, and specifically a rising rate of near-miss routing in newer markets as the signal that market needs its own fine-tuning rather than inheriting the global model's calibration. The trade-off: per-country classifier tuning is significantly more operational overhead than one shared classifier, but a single global model at this scale risks silently underperforming in every market except whichever one dominated its training data.