45 — LongRAG + Self-Route

Answer:

LongRAG ("LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs," Jiang, Ma & Chen, 2024, arXiv:2406.15319) keeps the retrieve-then-read pipeline but changes the unit of retrieval. Instead of the traditional ~100-word DPR-style passage, it groups related text into much larger units — roughly 4K tokens each, about 30x larger — often whole documents or document clusters. On Wikipedia this shrinks the retrieval pool from ~22M small chunks to ~600K large units, so the retriever has far fewer "needles" to search through, and each retrieved unit gives the reader enough surrounding context that answers rarely get split across a chunk boundary.

File 10 (Long-Context RAG) is a different, more extreme point on the same spectrum: it mostly removes the retrieval-unit problem by stuffing entire documents (or the whole corpus, if it fits) into a 100K–1M token context window, often with only a coarse BM25/vector pre-filter to narrow which documents to include at all. It doesn't redesign what a "retrieval unit" is — it just asks "why chunk at all if the context window is big enough?"

                     Chunk size          What changes
Naive/Advanced RAG:  ~100-300 tokens     nothing — small units, cheap retrieval
LongRAG:             ~4K tokens/unit     retrieval UNIT size — still a retriever,
                                         fewer/bigger units, less fragmentation
Long-Context RAG:    whole doc(s)        retrieval STRATEGY — coarse filter only,
(file 10)                                then stuff everything into the window

The key distinction: LongRAG is still fundamentally a retrieval architecture — it just rebalances the "heavy retriever, light reader" imbalance of classic RAG by making units bigger. Long-Context RAG (file 10) is closer to abandoning fine-grained retrieval altogether and leaning on the model's context window and prompt caching instead.

Answer:

The paper's core diagnosis: in classic RAG (e.g. DPR + a short-context reader), the retriever does all the hard work — searching millions of tiny 100-word passages to find the one that contains the answer — while the reader's job is trivial (extract the answer from a passage it already knows contains it). This is an imbalanced design: a "heavy" retriever paired with a "light" reader.

LongRAG rebalances this by making units 30x larger (grouping into ~4K-token units, sometimes whole Wikipedia articles), which:

• Shrinks the number of units the retriever must distinguish between (22M → 600K on Wikipedia) — an easier retrieval problem
• Pushes more of the actual reasoning burden onto the reader, which now needs a long-context LLM (since units are 4K+ tokens) capable of finding and synthesizing the answer within a longer span

# Simplified: group short passages into long retrieval units
def build_long_units(passages: list[str], target_tokens: int = 4000) -> list[str]:
    units, current, current_len = [], [], 0
    for p in passages:
        p_len = count_tokens(p)
        if current_len + p_len > target_tokens and current:
            units.append("\n\n".join(current))
            current, current_len = [], 0
        current.append(p)
        current_len += p_len
    if current:
        units.append("\n\n".join(current))
    return units  # ~30x fewer, ~30x larger than standard 100-word chunks

def long_rag_answer(query: str, long_retriever, reader_llm) -> str:
    top_units = long_retriever.search(query, k=4)          # search over large units
    context = "\n\n---\n\n".join(top_units)
    return reader_llm.generate(query=query, context=context)  # needs long context window

Reported results (no additional training required): LongRAG achieves 62.7% EM on Natural Questions and 64.3% EM on full-wiki HotpotQA, competitive with or exceeding fine-tuned short-chunk RAG pipelines, purely from the unit-size change plus an off-the-shelf long-context reader.

Answer:

Self-Route comes from a separate paper, "Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach" (Li, Li, Zhang, Mei & Bendersky, Google Research, 2024, arXiv:2407.16833). Its finding: when a long-context LLM has enough budget, long-context stuffing (LC) tends to outperform RAG on average — but it is far more expensive per query. Self-Route is a routing mechanism that gets most of LC's quality at close to RAG's cost.

The mechanism is two steps, using the same LLM for both:

SELF_ROUTE_CHECK_PROMPT = """You are given a question and some retrieved passages.
Decide if the passages contain enough information to answer the question.
Reply with exactly one word: "ANSWERABLE" or "UNANSWERABLE".

Question: {query}
Passages:
{top_k_passages}
"""

def self_route(query: str, all_passages: list[str], retriever, llm, k: int = 5) -> str:
    top_k = retriever.search(query, k=k)

    verdict = llm.generate(SELF_ROUTE_CHECK_PROMPT.format(
        query=query, top_k_passages="\n\n".join(top_k)
    )).strip()

    if verdict == "ANSWERABLE":
        # Cheap path: standard RAG with just the top-k passages
        return llm.generate(query=query, context="\n\n".join(top_k))
    else:
        # Expensive fallback: stuff the FULL passage pool (long-context mode)
        return llm.generate(query=query, context="\n\n".join(all_passages))

Why this works: most queries are perfectly answerable from a handful of top-k passages — the LLM's self-assessment is a cheap, reliable proxy for "did retrieval fail this query?" Only the harder tail of queries (multi-hop, needle scattered across many documents, retrieval literally missed the right passage) fall through to the expensive full long-context pass.

Answer:

The two ideas operate at different layers and are complementary, not competing:

┌─────────────────────────────────────────────────────────────┐
│ LongRAG = WHAT gets retrieved                                │
│   → redesigns the retrieval unit (4K-token units instead     │
│     of 100-word chunks) so each unit is coherent enough      │
│     that the reader rarely needs more context                │
├─────────────────────────────────────────────────────────────┤
│ Self-Route = WHETHER to retrieve at all, or go full long-ctx │
│   → a per-query decision layer that sits in FRONT of         │
│     generation, choosing cheap-RAG vs. expensive-LC          │
└─────────────────────────────────────────────────────────────┘

Combined pipeline:

def longrag_with_self_route(query: str, long_units: list[str], long_retriever, llm) -> str:
    # 1. LongRAG: retrieve large, coherent units (not fragmented chunks)
    top_units = long_retriever.search(query, k=3)   # each unit ~4K tokens

    # 2. Self-Route: ask the LLM if these units are sufficient
    verdict = llm.generate(SELF_ROUTE_CHECK_PROMPT.format(
        query=query, top_k_passages="\n\n".join(top_units)
    )).strip()

    if verdict == "ANSWERABLE":
        return llm.generate(query=query, context="\n\n".join(top_units))  # cheap
    else:
        # Fall back to ALL long units, not just top-3 — still large units,
        # just no longer top-k-limited
        return llm.generate(query=query, context="\n\n".join(long_units))  # expensive

Why compose them: LongRAG's large units already reduce the odds that Self-Route triggers the expensive fallback (less fragmentation means fewer "the answer was split across two chunks" failures). Self-Route then adds a cost control on top, so you don't pay full long-context prices for the large majority of queries that a handful of well-sized units can already answer.

Answer:

Larger retrieval units trade one set of failure modes for another:

The core tension: LongRAG reduces fragmentation-driven hallucination and retrieval misses, but it does so by asking the reader LLM to do more filtering work per call, and it makes every retrieved item more expensive to embed, index, rerank, and feed to the generator. This is the same "heavy reader" tradeoff the paper explicitly accepts in exchange for a "lighter retriever."

Combining with other techniques to mitigate the downside:

• Self-Route caps the blast radius: most queries never need the full long-context fallback, so the expensive path is rare rather than the default.
• Hierarchical retrieval (retrieve at the large-unit level, then a second pass extracts the specific sub-span within the winning unit) recovers some of small-chunk RAG's precision without giving up LongRAG's reduced fragmentation.
• Prompt caching amortizes the cost of repeatedly feeding the same large unit across multiple queries in a session (same mechanism used in Long-Context RAG, file 10).

When LongRAG is the wrong choice: if the corpus consists of many short, independent facts (e.g. a FAQ database or a table of key-value records), forcing 4K-token grouping only adds irrelevant padding — the imbalance LongRAG fixes doesn't exist in the first place, and Naive RAG's small chunks remain the better fit.

Answer:

Corpus → Long-Unit Grouper (~4K-token units, 30x larger than DPR chunks)
       → Long Retriever (dense search over far fewer, larger units)
       → Top-k Long Units (minimal fragmentation)
       → Long-Context LLM Reader → Answer

Self-Route (decision layer, sits in front of generation):
  Query + top-k retrieved passages
       → LLM self-assessment: "Can this be answered from these passages?"
       ├─ Yes → Answer via RAG (cheap, short context)
       └─ No  → Fall back to full Long-Context stuffing (expensive, all passages)

The two pieces solve different problems and compose (Q4): LongRAG changes what gets retrieved (large, coherent units instead of fragmented chunks), which alone reduces answer-splitting failures; Self-Route changes whether to trust retrieval at all for a given query, adding a cost-control layer that only escalates to expensive full-context stuffing for the minority of queries where even large-unit retrieval isn't sufficient. Together, LongRAG makes the cheap path work more often, and Self-Route ensures the expensive path is only paid for when genuinely needed.

Answer:

The distinctive mechanism is rebalancing retrieval-unit size to shift work from the retriever to the reader — rather than accepting classic RAG's implicit design choice of small, precisely-targeted chunks (which demands a very precise retriever and a trivial-extraction reader), LongRAG groups text into units roughly 30x larger, deliberately making the retrieval problem easier (fewer, more distinguishable units to search among) at the cost of making the reading problem harder (the reader must now find and synthesize the answer within a much longer, less-targeted span).

This single design choice is what shrinks Wikipedia's retrieval pool from ~22M chunks to ~600K units (Q1, Q2) and is what requires pairing with a genuinely long-context reader model — the mechanism only works because modern long-context LLMs can absorb a 4K-token unit and still extract a precise answer from within it, a capability that didn't exist (or wasn't affordable) when small-chunk RAG became the default design pattern.

Answer:

Both address chunk-boundary information loss, but from opposite directions. Contextual RAG (#14) keeps chunks small but prepends an LLM-generated context summary to each chunk before embedding, so a small chunk still carries enough surrounding context to be understood in isolation — the chunk itself stays small, but its embedding and content are enriched. LongRAG instead makes the retrieval unit itself much larger (Q7), so there's no boundary to lose context across in the first place, at the cost of the reader having to filter more irrelevant content per retrieved unit (Q5, Q12).

Choose Contextual RAG when chunk-level precision and low per-retrieval-item cost matter (many small, individually-addressable facts) but context loss at chunk boundaries is the specific pain point; choose LongRAG when the corpus has substantial internal coherence at the document/section level (a fact genuinely can't be understood without its surrounding several thousand tokens) and you have a long-context reader capable of absorbing that scale per retrieved unit.

Answer:

LongRAG (Jiang, Ma & Chen, LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs, arXiv:2406.15319, 2024) introduced the large-retrieval-unit idea, reporting 62.7% EM on Natural Questions and 64.3% EM on full-wiki HotpotQA with no additional training required, purely from the unit-size change plus an off-the-shelf long-context reader. Self-Route (Li, Li, Zhang, Mei & Bendersky, Google Research, Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach, arXiv:2407.16833, 2024) is an entirely separate paper studying when long-context stuffing outperforms RAG outright, proposing the cheap self-assessment routing mechanism (Q3) as a hybrid that captures most of long-context stuffing's quality advantage without paying its cost on every query.

This file's title reflects that the two ideas are complementary but independently discovered — "LongRAG + Self-Route" is a practitioner-assembled combination (Q4) rather than a single paper's proposal, similar in spirit to how several other architectures in this bank combine separately-published techniques into one production pattern (e.g., Table-Aware RAG's synthesis of TAPAS/OmniTab research, #36 Q19).

Answer:

The Self-Route verdict prompt's strictness is the most consequential knob precisely because it's the one non-numeric, hardest-to-tune parameter — unlike target_tokens or k, which have a fairly intuitive cost/quality dial, calibrating an LLM self-assessment prompt's strictness requires the same empirical evaluation discipline as calibrating any classifier threshold (Q19).

Answer:

Build a golden set and compare standard small-chunk RAG against LongRAG-style large units on the same corpus and queries, tracking both answer accuracy (EM/F1, matching the paper's own reported metrics, Q9) and a fragmentation-failure rate specifically — the fraction of queries where the correct answer spans a boundary that small chunking splits but large-unit grouping doesn't. This decomposition matters because LongRAG's value proposition is specifically about eliminating fragmentation failures (Q1, Q5); if your corpus's small-chunk failures are dominated by a different cause (poor embedding quality, genuinely absent information) rather than boundary-splitting, larger units won't fix what's actually wrong and the comparison will show LongRAG providing little benefit for a real but different reason.

Also measure semantic-dilution symptoms (Q12) on the large-unit side specifically — a corpus with highly heterogeneous documents (many unrelated topics packed into what becomes one grouped unit) may show large units underperforming small chunks despite eliminating fragmentation, because the retriever's embedding of a diluted unit is now a worse match for narrow queries than a small chunk's tightly-scoped embedding would have been. Reporting both metrics together is what reveals whether large units are a net win for your specific corpus's structure, rather than assuming the paper's Wikipedia-scale results transfer directly.

Answer:

A pooled embedding vector for a 4K-token unit necessarily blends the semantic content of everything within it — if a unit covers several sub-topics (common when grouping is done mechanically by token count rather than genuine topical coherence, as in Q2's build_long_units), the resulting embedding represents an average of those sub-topics rather than any one of them precisely. A query about one specific sub-topic within a diluted unit may score lower similarity against that unit's blended embedding than it would have against a small, tightly-scoped chunk covering just that sub-topic — meaning large units can, in this specific case, reduce retrieval precision relative to small chunking, the opposite of LongRAG's usual benefit.

Detection: for queries where large-unit retrieval underperforms small-chunk retrieval on the same benchmark (Q11), inspect whether the correct large unit was retrieved at all but ranked low, versus not retrieved in the top-k at all — a pattern where the correct unit exists but consistently ranks below topically-narrower competing units is the semantic-dilution signature. Mitigation: group units by genuine topical/structural coherence (document sections, whole short documents) rather than purely by token-count target (Q2's mechanical grouping), since coherent grouping produces units whose content is actually homogeneous enough for a pooled embedding to represent well, unlike an arbitrary token-count cutoff that can split or merge content without regard to topic boundaries.

Answer:

Q5 notes hierarchical retrieval as a mitigation for large-unit cost and dilution: retrieve at the large-unit level first (benefiting from LongRAG's easier, less-fragmented retrieval problem), then run a second, finer-grained pass within the winning unit(s) to extract the specific relevant sub-span before handing it to the reader — recovering some of small-chunk RAG's precision without giving up LongRAG's reduced fragmentation at the first stage:

def hierarchical_longrag(query: str, long_retriever, sub_chunker, embed_fn, reader_llm, k_units: int = 3, k_subspans: int = 2) -> str:
    # Stage 1: LongRAG-style retrieval over large, coherent units
    top_units = long_retriever.search(query, k=k_units)

    # Stage 2: within each winning unit, re-chunk finely and re-rank
    query_emb = embed_fn(query)
    all_subspans = []
    for unit in top_units:
        sub_chunks = sub_chunker(unit, chunk_size=200)  # standard small-chunk granularity
        scored = [(c, cosine_sim(query_emb, embed_fn(c))) for c in sub_chunks]
        scored.sort(key=lambda x: x[1], reverse=True)
        all_subspans.extend([c for c, _ in scored[:k_subspans]])

    # Stage 3: reader sees precise sub-spans, but each one still came from a
    # coherent large unit rather than an arbitrarily-chunked corpus
    context = "\n\n".join(all_subspans)
    return reader_llm.generate(query=query, context=context)

This combines LongRAG's easier first-stage retrieval (fewer, more distinguishable large units, Q7) with a second-stage precision pass that mitigates both the semantic-dilution risk (Q12, since the first-stage retrieval doesn't have to rely solely on the diluted unit-level embedding for the final answer) and the cost concern (Q16, since the reader ultimately sees smaller, filtered content rather than the full large unit).

Answer:

Choose LongRAG when the corpus is large enough that even a generous long-context window can't hold everything potentially relevant to a query — LongRAG's retrieval step still narrows millions of candidates down to a handful before the reader ever sees anything, which Long-Context RAG's "stuff a coarsely-filtered set of whole documents" approach does less precisely (file 10 typically relies on a much coarser pre-filter, since it isn't optimizing the retrieval step itself the way LongRAG's large-unit retriever does). Choose Long-Context RAG when the relevant candidate set is already naturally small (a handful of documents plausibly relevant to any query) and the corpus doesn't need a genuinely selective retrieval step at all — at that scale, the retrieval-unit redesign LongRAG offers has little to improve on, since the "heavy retriever" problem it targets doesn't really exist for a small enough corpus.

The two are not fully exclusive: Self-Route (Q3, Q4) is explicitly a hybrid that falls back from LongRAG's retrieval to something resembling Long-Context RAG's full-stuffing approach for the tail of queries retrieval alone can't handle — in practice, a mature system uses LongRAG's selective retrieval as the default and Long-Context RAG's stuffing as the escalation path, rather than choosing one architecture exclusively.

Answer:

Since Q5 already establishes LongRAG is the wrong choice for corpora of many short, independent facts, the decision needs a benchmark confirming which regime your corpus actually falls into before committing to re-architecting the retrieval pipeline:

1. Characterize the corpus: what fraction of documents/sections have
   genuine internal coherence spanning several thousand tokens (a
   single argument, obligation, or narrative that can't be understood
   from a 200-token excerpt) vs. being naturally composed of many
   short, independent facts (FAQ entries, key-value records)?

2. Build a golden eval set weighted toward this actual mix, including
   both fragmentation-failure-prone queries (where the answer spans
   a chunk boundary in small-chunk RAG) and simple lookup queries.

3. Baseline: standard small-chunk RAG (Naive/Advanced RAG's chunking).
4. Candidate: LongRAG-style large units (Q2), measuring the specific
   accuracy/fragmentation-rate/dilution-rate metrics from Q11 and Q12.

5. Gate: adopt LongRAG only if (a) the corpus characterization in step 1
   shows meaningful document-level coherence, AND (b) the measured
   accuracy improvement on fragmentation-prone queries outweighs any
   dilution-driven regression on narrow, simple-lookup queries, AND
   (c) you have (or will pair with) a genuinely long-context reader
   model, since LongRAG's mechanism depends entirely on that capability.

The corpus-characterization step is the one most teams skip, jumping straight to "let's try bigger chunks" without first confirming the imbalance LongRAG targets (heavy retriever, light reader) actually describes their retrieval failures — Q5's explicit "wrong choice" case exists precisely because this precondition is easy to overlook.

Answer:

Per Q5's cost table, larger units are more expensive at every downstream step: embedding a 4K-token unit costs more than embedding a 200-token chunk, reranking large candidates costs more per candidate, and feeding k large units to the reader produces a substantially larger prompt than standard RAG's equivalent — illustrative comparison at k=3: LongRAG's context is roughly 12K tokens (3 x 4K) versus standard RAG's roughly 1.5-3K tokens (k=5-10 x ~200-300 tokens), a meaningfully larger generation-time cost per query even before Self-Route's fallback path is considered.

Self-Route's fallback (Q3) is the more expensive tail: stuffing the full passage pool for the queries that don't pass the cheap-path check can be dramatically more expensive than even LongRAG's already-larger cheap path — illustrative cost at scale: if 90% of queries pass Self-Route's cheap-path check and 10% fall back to full-context stuffing at, say, 10x the cheap path's token cost, the blended average cost is roughly 1.9x the cheap-path-only cost (0.9 x 1 + 0.1 x 10), which is a manageable overhead specifically because the fallback rate is kept low — a poorly-calibrated Self-Route (Q19) that triggers the fallback on, say, 40% of queries would push blended cost to roughly 4.6x, eroding most of the architecture's cost-control value. Controlling this requires the same discipline as Q10's threshold tuning: monitor the actual fallback rate in production and treat a rising rate as a signal requiring investigation (either genuine query-difficulty drift or Self-Route miscalibration, Q19), not just an expected cost of doing business.

Answer:

• Larger injection surface per retrieved item — a 4K-token unit gives an adversarial document far more room to embed prompt-injection content (surrounded by enough legitimate-looking text to reduce the odds of casual detection) than a 200-token chunk would, and because the reader is explicitly expected to "filter" a large unit for relevant content (Q5's dilution trade-off), injected instructions have more opportunity to blend in as part of what looks like normal surrounding context the reader is meant to sift through.
• Blast radius of one poisoned document is larger — since a single retrieved unit may represent a whole document or document cluster, a single compromised source document affects a proportionally larger share of the context the reader sees per retrieval, compared to small-chunk RAG where a poisoned chunk is diluted among many other, independently-sourced small chunks in the same context window.
• Self-Route's fallback amplifies exposure for hard queries — the queries most likely to trigger Self-Route's expensive full-context fallback (Q3) are, by construction, queries retrieval struggled with — precisely the query type where a corpus might contain more marginal, lower-quality, or adversarial content mixed into the fallback's much larger context, meaning the fallback path both costs more (Q16) and is exposed to more unvetted content per query than the cheap path.
• Semantic dilution as an evasion technique — an adversarial passage embedded within an otherwise legitimate large unit benefits from the same semantic-dilution effect that (Q12) already reduces retrieval precision — the unit's pooled embedding may not surface the adversarial content's presence at all during retrieval scoring, meaning standard retrieval-time content screening (which often scores at the unit or document level) can miss a small malicious fragment buried within an otherwise-benign large unit.

Mitigation: apply content screening at the sub-unit level (the same granularity hierarchical retrieval, Q13, already operates at) rather than only at the whole-unit level, since unit-level screening can miss content that only becomes visible once the unit is decomposed; treat the Self-Route fallback path as warranting the same or stricter content-trust scrutiny as the cheap path, not less, given it activates specifically on the queries most likely to surface edge-case or lower-quality content; and apply prompt-injection-resistant framing (structural delimiters, explicit "treat as data" instructions) to reader prompts regardless of unit size, scaled to account for the larger surface area larger units provide.

Answer:

Requirements: contract obligations and clauses often can't be understood in a 200-token excerpt (a single obligation can span a full clause or section); most queries are simple lookups answerable from a couple of relevant clauses; a minority of queries (cross-clause interpretation, obligation chains) genuinely need broader context.

1. Unit construction (Q2, Q12): group by genuine structural boundaries
   (clause, section) rather than a mechanical token-count cutoff --
   legal documents have natural structural markers that make coherent
   grouping straightforward and directly avoid semantic dilution.

2. Retrieval (Q7): dense retrieval over clause/section-level units,
   shrinking the "which of thousands of clauses across hundreds of
   contracts is relevant" problem the way LongRAG shrinks Wikipedia's
   pool -- a much easier retrieval problem than searching arbitrarily-
   chunked contract text.

3. Self-Route (Q3, Q10): calibrate the verdict prompt's strictness
   against a labeled set of legal queries specifically, given legal
   use cases (per Verifiable RAG's #33 Q18 precedent) tolerate less
   risk from an under-triggered fallback (missing needed cross-clause
   context) than from the cost of an over-triggered one.

4. Hierarchical sub-span extraction (Q13): within a winning clause/
   section unit, extract the precise sentence(s) actually supporting
   an answer for citation purposes -- legal use cases need exact
   quotable language (as established in Verifiable RAG, #33), not
   just "the answer is somewhere in this section."

5. Content screening (Q17): screen at the sub-clause level specifically
   for any user-contributed or third-party contract content, given the
   dilution-evasion risk larger units introduce.

6. Cost monitoring (Q16): track Self-Route fallback rate specifically
   segmented by query type (simple clause lookup vs. cross-contract
   comparison) -- a rising fallback rate on what should be simple
   lookups is the Q19 miscalibration signature worth investigating.

The key design choice is grouping by genuine legal-document structure (clauses/sections) rather than an arbitrary token count — this is precisely what Q12's mitigation recommends, and it's naturally available in this domain given how consistently structured legal documents already are, unlike a more heterogeneous general corpus where structural grouping is harder to define.

Answer:

Over-triggering (the verdict prompt too readily says "UNANSWERABLE") routes too many queries to the expensive fallback unnecessarily, eroding the cost-control benefit that's the entire point of Self-Route (Q16's cost-scaling example shows how quickly a high fallback rate erases the architecture's savings). Under-triggering (too readily says "ANSWERABLE") is the more dangerous direction: queries that genuinely needed the expensive fallback get a confident-looking but under-supported answer from the cheap path instead, since the whole mechanism depends on the LLM's self-assessment being a reliable proxy for "did retrieval actually succeed" (Q3) — if that self-assessment is systematically overconfident, the failure is silent, producing plausible-looking wrong answers with no visible error.

Debugging: (1) track the fallback trigger rate over time and by query segment, watching for both a sudden shift (a model or prompt change affecting calibration) and a segment-specific pattern (certain query types consistently under- or over-triggering); (2) build a labeled set of (query, retrieved passages, human-judged "was this actually answerable from these passages") triples, and measure Self-Route's verdict accuracy against human judgment directly — the same evaluation discipline used for any classifier, since Self-Route's verdict step is functionally a binary classifier even though it's implemented as an LLM prompt rather than a trained model; (3) if under-triggering is confirmed, tighten the verdict prompt's bar for "ANSWERABLE" (Q10) or add explicit uncertainty cues the prompt should watch for (multiple retrieved passages disagreeing, passages that are topically related but don't directly address the query's specific ask); re-validate after any change to the underlying LLM version, since self-assessment calibration is a property of the specific model being used, not a fixed architectural guarantee.

Answer:

Current limitations: (1) semantic dilution is an inherent trade-off, not a solvable bug (Q12) — any unit large enough to reduce fragmentation risk is, by the same token, large enough to risk diluting its own embedding, and mechanical grouping strategies (Q2) make this worse than structurally-aware grouping (Q13, Q18); (2) cost scales with unit size at every pipeline stage (Q5, Q16) — embedding, reranking, and generation all get more expensive per retrieved item, which is the direct price of the fragmentation reduction; (3) Self-Route's calibration is a single point of failure for cost control (Q19) — a miscalibrated self-assessment either erodes the cost savings or silently degrades answer quality, with no independent check unless one is explicitly built; (4) the approach assumes access to a genuinely long-context reader model (Q7), which not every deployment (cost-constrained, latency-constrained, or using a smaller open-weight model) can assume.

Likely evolution: learned, structurally-aware unit grouping (replacing Q2's mechanical token-count cutoff with a model that identifies genuine topical/structural boundaries automatically) to reduce semantic dilution at the source rather than relying on domain-specific structural cues being conveniently available (as they are for legal documents, Q18, but aren't for all corpora); calibrated, trained routing replacing today's prompted Self-Route self-assessment (following the same learned-vs-prompted trajectory as other architectures in this bank, e.g. Auto-RAG/DeepRAG's #49 per-step decisions) to address the calibration fragility in Q19 with a more principled, continuously-monitored classifier rather than an LLM prompt; and continued growth of hierarchical retrieval (Q13) as a standard component rather than an optional mitigation, since it directly addresses both the cost and dilution downsides of large units without giving up their fragmentation-reduction benefit.

Answer:

A publisher's fiction archive is close to an ideal fit for this architecture without needing Self-Route's extra complexity at all: a chapter is naturally a coherent, self-contained retrieval unit (Q2's rebalancing argument applies cleanly here, unlike a corpus of short independent facts, Q5's "wrong choice" case), and grouping by actual chapter boundaries rather than a mechanical token cutoff avoids the semantic-dilution risk in Q12 almost by construction, since a chapter is already a single continuous narrative thread.

At a small publisher's scale and query volume, skip Self-Route (Q3, Q4) entirely — its value is capping the cost of an expensive full-context fallback for the minority of hard queries, but at low query volume that fallback cost is trivial either way, and the added complexity of calibrating a verdict prompt (Q19) isn't worth it until volume grows. A straightforward LongRAG pipeline — retrieve the top few chapters, feed them whole to a long-context reader — covers a reader's or editor's typical question ("which chapter does the reveal about the sister happen in?") well on its own.

Trade-off worth naming: feeding whole chapters (several thousand tokens each) to the reader costs more per query than small-chunk retrieval would, but at a small publisher's query volume that absolute cost stays modest (Q16's cost concerns bite mainly at scale); if the archive later grows into a high-traffic consumer product, that's the point to revisit whether Self-Route's cost-control layer becomes worth adding.

Answer:

A certification audit turns two of this file's usual trade-offs into hard requirements rather than tuning choices. Unit grouping (Q2, Q12) must follow the specification's genuine structural boundaries — numbered sections and individual requirements — exactly as Q18's legal-contract design recommends, both to avoid semantic dilution and because an auditor needs retrieval units that map onto citable spec structure, not an arbitrary token-count grouping.

Hierarchical sub-span retrieval (Q13) is not optional here: an auditor asking whether a specific requirement is satisfied needs the exact requirement wording quoted back, not "this is discussed somewhere in this section" — the drill-down step that recovers small-chunk precision within a retrieved unit is what makes a LongRAG answer usable as audit evidence at all. Self-Route's verdict threshold (Q3, Q10) should be tuned conservative — biased toward the expensive full-context fallback — because under-triggering (Q19) here means confidently answering a certification question from insufficient context, which is a compliance failure, not just a quality regression; the extra compute cost of over-triggering is cheap by comparison.

Given specs run to thousands of pages and requirements cross-reference each other constantly, monitor the Self-Route fallback rate segmented by requirement category (Q16, Q19) and treat a rising rate on core certification-relevant sections as requiring investigation before the audit, not after. Every retrieved unit's spec version and revision needs to be tracked explicitly, since a certification audit can span a period where the underlying specification itself is revised — an answer traceable to a superseded requirement version is as much a failure here as one traceable to the wrong page.