25 — GraphReader / GNN-RAG

Answer:

Both make an LLM reason over a graph instead of a flat list of chunks, but via opposite mechanisms.

GraphReader (Li et al., 2024):

• Builds a graph of atomic facts / notes extracted from long documents (nodes = key elements, edges = connections).
• An LLM agent then explores this graph step-by-step — reading notes, deciding where to move next, taking notes in a notebook — to answer questions that exceed the context window.
• Goal: long-context multi-hop reasoning with a small working context.

GNN-RAG (Mavromatis & Karypis, 2024):

• Over a knowledge graph (e.g., for KGQA), a Graph Neural Network scores and retrieves the relevant reasoning subgraph / paths connecting question entities to candidate answers.
• The retrieved paths are verbalized into text and handed to the LLM to generate the final answer.
• Goal: bring GNNs' strong multi-hop graph reasoning to LLMs for knowledge-graph QA.

Common thread:

• Both treat graph structure as the retrieval substrate (not similarity over chunks).
• Both target multi-hop questions where the answer requires connecting several facts.
• Both separate graph reasoning from language generation — something else (an agent / a GNN) finds the relevant structure, and the LLM articulates the answer.

Answer:

Build (offline-ish, per document set):

1. Chunk the long document.
2. LLM summarizes each chunk into ATOMIC FACTS + key elements (entities/concepts).
3. Nodes = key elements; each node links to its atomic facts and source chunk.
4. Edges connect nodes that co-occur / are related across the document.
→ A graph that compresses a long document into navigable notes.

Explore (per query) — the LLM acts as an agent:

1. Plan: from the question, decide what info is needed; pick starting nodes.
2. Explore step-by-step:
     - read a node's atomic facts
     - record relevant findings in a "notebook" (running memory)
     - decide the next action: explore neighbors / jump to another node / stop
3. Continue until the notebook contains enough to answer (or budget hit).
4. Answer from the notebook.

Why this beats stuffing the document into a long context:

• Only a small, relevant slice of notes is ever in context at once → bounded context, less "lost-in-the-middle."
• The notebook accumulates findings across many hops without holding the whole document.
• Exploration is adaptive — the agent follows the reasoning chain rather than relying on one similarity retrieval.

It's essentially an agentic, graph-structured read of a long document: rational navigation over notes instead of brute-force context stuffing.

Answer:

GNN-RAG targets Knowledge Graph Question Answering (KGQA) and splits the job between a GNN (reasoning/retrieval) and an LLM (generation):

1. Subgraph extraction:
     Take the question entities; pull a dense subgraph around them from the KG.

2. GNN reasoning:
     A Graph Neural Network does message-passing over the subgraph,
     scoring nodes by how likely they answer the question.
     GNNs are strong at multi-hop structural reasoning — they propagate
     signal along relation paths.

3. Path retrieval:
     Extract the reasoning PATHS connecting question entities to the
     top-scored candidate answer nodes.

4. Verbalization:
     Turn those paths into natural-language statements
     ("Gustave Eiffel — designed → Eiffel Tower; Gustave Eiffel — born_in → France").

5. LLM generation:
     Feed the verbalized paths to the LLM, which produces the final answer.

Why a GNN instead of an LLM or similarity for the retrieval step:

• GNNs are purpose-built for graph structure — message-passing naturally captures multi-hop relational patterns that embedding similarity misses.
• They handle dense KGs where the answer depends on connectivity, not text similarity.
• They're cheap and fast relative to iterative LLM hops, and can be trained on KGQA supervision.

The division of labor: GNN = "which paths matter" (structural reasoning); LLM = "say the answer in language" (generation). GNN-RAG even shows combining GNN-retrieved paths with LLM-retrieved ones boosts recall.

Answer:

The distinguishing axes:

• GraphReader is agentic — an LLM actively navigates a note-graph (adaptive, multi-step, controllable), whereas Graph RAG/HippoRAG/LightRAG perform a single retrieval pass. GraphReader's graph also represents a document's content as notes for long-context reading, not a corpus-wide entity KG.
• GNN-RAG uses a learned neural retriever (GNN) over a KG — neither similarity (Graph RAG/LightRAG) nor a fixed graph algorithm (HippoRAG's PageRank), but a trained model that scores graph structure. It assumes a pre-existing KG (classic KGQA setting) rather than building one from raw text.

Why they earn separate treatment: they introduce two mechanisms absent from the others — agentic graph traversal (GraphReader) and learned GNN-based graph retrieval (GNN-RAG) — representing the "agent" and "neural" ends of the graph-reasoning spectrum.

Answer:

GNN-RAG and GraphReader represent the two answers to "who reasons over the graph." Trade-offs of the GNN approach vs. LLM traversal (GraphReader/agentic):

Why GNNs specifically excel at KG reasoning:

• Message-passing aggregates signal along relation paths — exactly the operation multi-hop KGQA needs — and does it for all candidate paths simultaneously, where an LLM must explore them one at a time.
• On dense KGs (high connectivity), LLM traversal faces a combinatorial explosion of next-step choices; the GNN handles the whole neighborhood in one pass.

Why not always GNN: it needs training data and a fixed KG; it can't read free text or adapt to arbitrary new tasks. GraphReader's agentic approach trades efficiency for zero-shot flexibility and readable reasoning over documents.

Best of both: GNN-RAG's own finding — combine GNN-retrieved paths with LLM-retrieved ones — shows the two are complementary, not mutually exclusive.

Answer:

QUESTION: "Did the subsidiary that posted the largest 2023 loss
           receive intercompany funding, and from whom?"
DOCUMENT: 300-page annual report (far exceeds context window)

PRE-BUILT GRAPH: nodes = {subsidiaries, financials, transactions, entities},
  each linked to atomic facts + source pages.

AGENT EXPLORATION
─────────────────
Plan: need (a) subsidiary with largest 2023 loss, (b) its intercompany funding.

Step 1: start at "subsidiary financials" nodes
        read atomic facts on 2023 losses → notebook: "Sub C loss = $40M (largest)"
Step 2: move to node "Sub C"
        read its facts → notebook: "Sub C — intercompany transactions: see node T-12"
Step 3: explore node T-12 (transactions)
        read facts → notebook: "T-12: Parent Co funded Sub C $25M in Q3 2023"
Step 4: sufficiency check → notebook answers both parts → STOP
Answer: "Yes — Sub C (largest 2023 loss, $40M) received $25M intercompany
         funding from Parent Co in Q3 2023." [cites pages via node links]

What made this work:

• The 300-page report never entered the context whole — only a handful of relevant atomic-fact nodes did.
• The notebook carried findings across hops (loss figure → funding) without holding the document.
• Exploration was goal-directed: the agent followed the reasoning chain (loss → subsidiary → transactions) rather than similarity-matching the question once.

This is GraphReader's core value: multi-hop reasoning over content larger than the context window, with bounded working memory.

Answer:

GraphReader (agentic exploration):

1. Graph-construction quality — if atomic-fact extraction misses or mangles a fact, the relevant node never exists; exploration can't find it.
2. Exploration errors / dead-ends — the agent may pick wrong nodes, loop, or stop prematurely (incomplete notebook → wrong answer).
3. Latency/cost — each exploration step is an LLM call; deep questions = many calls (sequential).
4. Notebook drift — accumulated notes may grow noisy or the agent may mis-summarize, propagating errors.
5. Stopping calibration — stopping too early misses evidence; too late wastes cost.

GNN-RAG (learned graph retrieval):

1. KG dependency — requires a pre-existing, reasonably complete KG; useless on raw text without one.
2. Training-data need — the GNN needs KGQA supervision; poor generalization to out-of-distribution questions/graphs.
3. Schema rigidity — tied to the trained KG schema/relations; new relation types need retraining.
4. Verbalization bottleneck — paths must be turned into text; poor verbalization loses information before the LLM sees it.
5. Incompleteness — if the answer path isn't in the extracted subgraph, the GNN can't recover it.

Shared: both inherit "garbage graph → garbage answer" — they're only as good as the underlying graph, and both add a reasoning component (agent / GNN) that is itself a new failure surface beyond plain retrieval.

Answer:

GraphReader (long-context multi-hop):

• Benchmarks: long-document QA + multi-hop — HotpotQA, MuSiQue, 2WikiMultiHopQA, and long-context suites (e.g., long-doc QA) where it's compared against long-context LLMs.
• Metrics: EM/F1; supporting-fact recall; context efficiency (tokens/working-context used vs. stuffing the whole doc) — its key selling point; exploration steps per question (cost).
• Diagnostics: node-recall (did the needed node get visited?), notebook completeness, premature-stop rate.

GNN-RAG (KGQA):

• Benchmarks: KGQA datasets — WebQSP and ComplexWebQuestions (CWQ) are the standard; report Hits@1 / F1. The paper's claim is SOTA/competitive on these, especially multi-hop subsets.
• Metrics: answer Hits@1/F1; path retrieval recall (did the GNN surface the gold reasoning path?); performance by hop-count (the multi-hop advantage).
• Ablations: GNN-only vs LLM-retrieval-only vs combined (GNN-RAG's headline is that combining boosts recall); GNN architecture/depth.

Common principle: measure the retrieval/reasoning layer separately from final answer accuracy — for GraphReader, did exploration reach the right notes; for GNN-RAG, did the GNN retrieve the right paths — so you know whether a failure is in the graph component or the LLM's generation.

Answer:

USE CASE: Enterprise has a curated KG (org structure, systems, ownership,
          dependencies, SLAs). Users ask multi-hop KGQA:
          "Which on-call team owns the upstream service whose outage would
           breach the payments SLA?"
WHY GNN-RAG: dense, structured KG + multi-hop relational questions =
             GNN's sweet spot; LLM traversal would explode on connectivity.

COMPONENTS
──────────
1. Knowledge graph: maintained from CMDB / service catalog / org data
   (entities: Service, Team, SLA, Dependency, Incident).

2. Question entity linking: map the query to KG entities
   {payments SLA, payments service}.

3. Subgraph extraction: pull the k-hop neighborhood around linked entities.

4. GNN reasoning (trained on enterprise KGQA pairs):
   message-passing scores nodes; retrieve paths from question entities
   to candidate answer nodes (teams).

5. Path verbalization:
   "payments-svc —depends_on→ ledger-svc —owned_by→ Team Atlas
    —on_call_rotation→ ..." 

6. LLM generation: answer from verbalized paths, with the path as citation.

HYBRID RETRIEVAL
────────────────
- Combine GNN paths with dense retrieval over service docs (GNN-RAG's
  combined-retrieval finding) to cover facts not in the KG.

OPS / GUARDRAILS
────────────────
- KG freshness: sync with CMDB; stale dependencies = wrong answers.
- Entity-level ACL on subgraph extraction (don't leak restricted systems).
- Fallback to text RAG when entity linking fails or the path is empty.
- Retrain/fine-tune GNN as schema evolves; monitor path-recall drift.

MONITORING
──────────
- Hits@1 / path recall on a gold enterprise-QA set
- % queries answerable from KG vs needing text fallback
- Entity-linking success rate; latency per query

GNN-RAG fits because the enterprise already has a dense, curated KG and the questions are multi-hop relational — exactly where a trained GNN beats both similarity retrieval and sequential LLM traversal, while the LLM still handles final language and the text-RAG fallback covers KG gaps.

Answer:

Choose GraphReader when:

• You have long documents (exceeding context) and need multi-hop reasoning within them.
• You don't have a KG and don't want to train anything — GraphReader builds a note-graph from text and works zero-shot.
• You value a readable reasoning trace (the agent's notebook).
• Latency is acceptable (it's LLM-call-per-step).

Choose GNN-RAG when:

• You already have a knowledge graph (KGQA setting) — dense, structured, relational.
• Questions are multi-hop over the KG and connectivity (not text similarity) determines the answer.
• You can obtain training data to fine-tune the GNN, and the schema is relatively stable.
• You need fast, scalable graph reasoning over dense graphs.

Choose a simpler approach (standard / Graph RAG / HippoRAG) when:

• Single-hop or similarity-answerable → naive/advanced RAG.
• Global summarization → Graph RAG.
• Multi-hop facts but want one cheap pass, no training → HippoRAG (PageRank).
• The overhead of agentic exploration or GNN training isn't justified by the question complexity.

Summary heuristic:

• Long text + no KG + need adaptivity → GraphReader.
• Existing dense KG + multi-hop + can train → GNN-RAG.
• Otherwise → a lighter graph or non-graph RAG.

Answer:

GraphReader:

Build: LLM summarization of every chunk into atomic facts (one-time per doc).
Query: 1 LLM call PER exploration step (plan, read, decide) + final answer.
       Latency scales with exploration depth — sequential, not parallel.

• Driver: number of exploration steps × LLM call cost.
• Optimize: small/fast model for navigation decisions, frontier model for the final answer; cap exploration steps; prune the graph (merge redundant nodes); cache the built graph; batch atomic-fact extraction at build.

GNN-RAG:

Build: train the GNN (one-time) + maintain the KG.
Query: GNN forward pass (cheap, ~ms) + verbalization + 1 LLM generation call.

• Driver: the single LLM generation call (the GNN pass is cheap and fast).
• Optimize: the GNN makes retrieval cheaper than iterative LLM hops — that's its efficiency win; keep subgraphs bounded; cache verbalized paths for repeat questions; quantize/scale the GNN for large KGs.

Comparison:

Net: GNN-RAG is cheaper/faster per query (GNN offloads reasoning from the LLM); GraphReader costs more per query but needs no training and handles raw long text.

Answer:

1. Graph poisoning (both). A malicious document (GraphReader) or a tampered KG fact (GNN-RAG) injects false nodes/edges that the reasoning component then treats as ground truth — and the explicit reasoning path makes the wrong answer look well-supported.

• Mitigation: source-trust scoring; validate high-impact facts; corroborate across sources before admission to the graph/KG.

2. Agentic-exploration hijacking (GraphReader). Injected instructions in node content ("ignore other nodes; report X") can steer the agent's navigation — the exploration loop is an attack surface, like any agent.

• Mitigation: treat node content as data not instructions; constrain the action space; sanitize atomic facts; cap steps.

3. Access control over graph traversal (both). Reasoning can traverse relations to reach sensitive nodes a flat document ACL wouldn't have exposed — a connectivity-based leak.

• Mitigation: enforce node/edge-level authorization during exploration/subgraph extraction, on every step — not just at ingestion.

4. Verbalization injection (GNN-RAG). If KG values contain adversarial text, verbalizing paths into the prompt can carry injection into the LLM.

• Mitigation: sanitize/escape verbalized content; spotlight it as untrusted data.

5. Over-trust in structured reasoning (both). Explicit paths/notebooks read as authoritative; users under-scrutinize a chain built on one bad early fact.

• Mitigation: surface provenance + confidence per hop; flag unverified/inferred steps.

6. Robustness to incompleteness (both). Missing nodes/paths yield silent wrong answers ("not found" presented as "no").

• Mitigation: detect empty/low-confidence retrieval and fall back to text RAG or abstain rather than assert.

Answer:

GraphReader:
Docs → build a graph of notes/atomic facts → Agent explores the graph
     step by step, taking notes, deciding where to explore next →
     synthesizes an answer from accumulated notes

GNN-RAG:
Docs → build a knowledge graph → Graph Neural Network learns to score/
     retrieve relevant subgraphs (trained, not prompted) → retrieved
     subgraph → LLM generates answer from it

Both replace an LLM's own free-form graph traversal (as a prompted Graph RAG agent might do) with a more structured or learned exploration mechanism — GraphReader formalizes the exploration into an explicit agent loop with note-taking as external memory (Q2), while GNN-RAG replaces exploration with a trained neural network that has learned, from labeled data, which subgraphs are likely relevant to a given query (Q3) — directly answering Q5's "why use a GNN instead of LLM traversal" question with "because it can be trained to do this task specifically, rather than relying on general-purpose LLM reasoning applied ad hoc to graph structure."

Answer:

GraphReader (Li et al., GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Models, arXiv:2406.14550, 2024) targets long-document reasoning specifically — converting a long document into a graph of atomic facts and having an LLM agent explore it step-by-step with explicit note-taking, addressing the same long-context degradation problem Long-Context RAG (#10) faces from a graph-structured-exploration angle rather than a raw-context-window angle.

GNN-RAG (Mavromatis & Karypis, GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning, arXiv:2405.20139, 2024) instead trains a graph neural network specifically for the retrieval step over a knowledge graph, reporting competitive or improved multi-hop QA accuracy relative to LLM-based graph traversal while being substantially cheaper at inference time, since a trained GNN's forward pass is far cheaper than an LLM reasoning step-by-step through graph exploration.

Answer:

All three retrieve from a graph structure rather than flat chunks, but via different mechanisms. HippoRAG (#20) uses Personalized PageRank — a classical, untrained graph algorithm — for single-pass spreading activation from query-matched entities. GNN-RAG uses a trained graph neural network specifically learned to score subgraph relevance from labeled data, requiring a training pipeline HippoRAG's algorithmic approach doesn't need. GraphReader uses neither — it's an LLM agent explicitly exploring the graph step-by-step with note-taking, more flexible and interpretable than either PPR or a GNN, but also the most expensive per query since it requires multiple LLM reasoning calls during exploration.

The practical decision axis: HippoRAG is the cheapest and simplest to deploy (no training, one fast algorithmic pass); GNN-RAG is a training investment that pays off with cheap, fast retrieval once trained, similar to Search-R1's (#42) fine-tune-once-serve-cheaply trade-off; GraphReader is the most expensive but requires no training investment and produces the most inspectable, step-by-step reasoning trace of the three, useful when auditability matters more than raw cost efficiency.

Answer:

The distinctive mechanism is a trained neural network performing the retrieval/relevance-scoring step, replacing Graph RAG's (#05) pre-computed, LLM-summarized community hierarchy entirely. Graph RAG answers queries by reading pre-written community summaries (computed once, generically, regardless of the specific query) or by LLM-mediated entity-neighborhood traversal. GNN-RAG instead trains a graph neural network on labeled (query, relevant-subgraph) examples, so at inference time, retrieval is a fast, learned forward pass through the trained network — no LLM call and no pre-written summary involved in the retrieval step itself, only in final answer generation from whatever subgraph the GNN retrieved.

This is the same learned-vs-engineered trade-off seen throughout this bank's more advanced architectures (Search-R1's #42 learned search policy vs. prompted Agentic RAG) applied specifically to graph retrieval — GNN-RAG trades Graph RAG's expensive-but-training-free indexing pipeline for a training investment that, once paid, produces cheaper and potentially more accurate per-query retrieval than either community summaries or algorithmic traversal.

Answer:

GNN-RAG's layer count is the knob most analogous to other architectures' "how many hops" knobs (Iterative Multi-hop RAG's #19 max-hops, HippoRAG's #20 damping factor) — it directly controls how far multi-hop reasoning can reach through the graph, but unlike those algorithmic knobs, it's fixed at training time rather than tunable per-query at inference.

Answer:

Build a multi-hop evaluation set on your own knowledge graph and compare three approaches head-to-head: HippoRAG-style PPR (#20, no training required), a trained GNN-RAG model, and (if feasible) GraphReader's LLM-agent exploration — tracking accuracy, query-time latency, and total cost including GNN-RAG's training investment amortized over expected query volume, mirroring the same training-investment decision-gate discipline used throughout this bank (Search-R1's #42 Q15, RQ-RAG's #51 Q15).

The comparison that matters most: does GNN-RAG's trained retrieval meaningfully outperform PPR's untrained algorithmic approach on your specific graph's structure and your specific query distribution? A GNN trained on insufficient or unrepresentative labeled data can underperform even a training-free heuristic (Q19), so this comparison should never be skipped in favor of assuming a trained approach is automatically superior — the training investment (data collection, training pipeline, ongoing retraining as the graph evolves) is only worthwhile if it clears this bar by a margin that justifies the added complexity relative to HippoRAG's simpler, training-free alternative.

Answer:

A GNN trained on one graph's structure (a specific entity-type distribution, a specific typical node degree, a specific relationship-type vocabulary) learns patterns specific to that structure — if the production knowledge graph later evolves to include new entity types, denser or sparser connectivity, or relationship types the training data didn't represent, the trained network's learned aggregation patterns (Q17) may not transfer well, producing systematically degraded retrieval quality on the newly-different graph regions with no explicit error, since the GNN still produces a confident-looking output regardless of whether its training distribution actually covers the query at hand.

Detection: segment retrieval quality by graph region/entity-type, specifically comparing regions that closely resemble the training graph's structure against regions added or restructured since training — a quality gap concentrated in the newer or structurally different regions is the signature of this training-production mismatch, distinguishable from a general retrieval-quality problem that would affect all regions roughly equally. Mitigation: retrain periodically as the graph evolves structurally (not just as new nodes are added, but specifically when the graph's structural characteristics shift), and monitor graph-structure drift (average node degree, entity-type distribution) as a leading indicator that retraining is due, the same drift-monitoring discipline used for Adaptive RAG's query-complexity classifier (#11 Q20).

Answer:

Current limitations: (1) GraphReader's exploration cost scales with document complexity (Q11) — multiple LLM calls per query make it the most expensive of the three approaches compared in Q15; (2) GNN-RAG requires a training investment and labeled data (Q18) that HippoRAG's (#20) algorithmic approach doesn't need, and that investment can underperform if training data doesn't match production structure (Q19); (3) both approaches inherit general graph-based RAG's robustness-to-incompleteness risk (this file's own security section) — missing graph structure produces silent wrong answers rather than an explicit "I don't know"; (4) neither has GraphRAG's (#05) pre-computed, human-readable community summaries available for auditing what the system "believes" about a corpus's themes, since GNN-RAG's learned representations and GraphReader's exploration notes are both harder to inspect at a glance than a written summary.

Likely evolution: continued cross-pollination within this bank's graph-based RAG family — GNN-RAG's learned-retrieval efficiency combined with GraphReader's inspectable note-taking trail, or a GNN-scored candidate-generation step feeding into a lighter final LLM verification pass (mirroring the drafter-verifier pattern from Speculative RAG, #08); and, as graph neural network research continues to mature, likely improvements in GNN transferability across evolving graph structures (directly addressing Q19's training-production mismatch), narrowing the retraining burden that currently favors HippoRAG's training-free approach for graphs expected to change structurally over time.

Answer:

The situation implies a modest protein-interaction dataset, a small team, and — critically — no labeled (question, relevant-subgraph) training data yet, since that kind of supervision usually comes later in a research program, not at the exploratory stage.

Q10's own decision heuristic applies directly: GNN-RAG needs training data and a relatively stable schema to be worth its investment (Q5, Q10), neither of which a small startup exploring a modest graph typically has yet. GraphReader, by contrast, works zero-shot — an LLM agent exploring a graph of extracted facts step-by-step (Q2) needs no training pipeline, which matches a startup's actual constraint (no labeled data, small team, need results now). If the interaction data is modest enough that questions are mostly simple lookups rather than genuinely deep multi-hop chains, an even lighter option (standard vector RAG, or HippoRAG's single-pass PPR, #20) may be sufficient and cheaper still.

The trade-off worth flagging: GraphReader's per-query cost (an LLM call per exploration step, Q11) is higher than a trained GNN's would eventually be, but that cost is the right one to pay now, since it avoids committing to a training investment before the startup has enough labeled examples or a stable-enough schema to make that investment pay off. Revisit GNN-RAG once the interaction graph and question patterns stabilize and labeled training pairs exist.

Answer:

The hard constraints are scale (a billion edges, where GraphReader's sequential LLM-call-per-step exploration would be intractably slow, per Q5's "dense graphs" comparison) and strict compartmentalized access control, where a wrong disclosure isn't just a data-quality problem but a security incident.

The approach favors GNN-RAG's mechanism (Q3, Q5): message-passing scales to dense, billion-edge structure in a way sequential agentic traversal doesn't, since it aggregates signal across many paths in parallel rather than exploring them one LLM call at a time. But standard GNN-RAG assumes one shared graph and one trained model — here, subgraph extraction and message-passing both need entity/edge-level authorization enforced at every step (Q12's access-control mitigation), not just at ingestion, and the verbalization step (Q3) needs particular care: a query into a compartment the analyst lacks clearance for should return "not authorized," never a silently-empty "no path found," since the two must not be distinguishable from each other in a way that leaks the existence of restricted relationships.

The real trade-off is operational complexity versus leakage risk: maintaining separate GNN training/serving per compartment (rather than one shared model with post-hoc filtering) multiplies the model-maintenance burden substantially, but a single shared model risks the retrieval or attention patterns themselves leaking compartment structure even with output filtering — given the stakes, the agency should accept the multiplied operational cost of compartment-separated models over a shared model with bolted-on filtering.

Monitor: per-compartment path-retrieval recall, authorization-filter audit logs (every subgraph extraction and every response checked against clearance), and explicit cross-compartment leakage canaries tested continuously, not just at deployment.