Retrieval

The retrieval domain (src/ragspine/retrieval/) is RAGSpine's narrative RAG channel — the half that answers "why / what happened" questions by retrieving document chunks, fusing lexical and (optional) vector signals, reranking, and handing cited snippets to the agent. It is the counterpart to the deterministic structured channel; see Dual-channel for how the agent routes between the two.

Two properties are non-negotiable here and enforced in code:

• Semantic hybrid by default, pure-BM25 as the zero-dep fallback. The vector channel is injectable. With the [embed-onnx] extra installed, retrieval is genuinely hybrid (BM25 + ONNX semantic → RRF) with no config (RAGSPINE_EMBEDDING=auto); with no embedding backend wired it is pure Okapi BM25 + RRF — fully offline, deterministic, zero SDKs.
• RESTRICTED isolation at two exits. Sensitivity-RESTRICTED content is stripped at both the rerank/ and link/ exits before it can reach a prompt. See RESTRICTED isolation.

Layout

The pipeline reads left to right: chunking produces and versions chunks → lexical (with optional vector) scores and fuses them → rerank reorders the top candidates → link adapts the result into the agent and drops RESTRICTED.

Current retrieval product presets

RAGSPINE_RETRIEVAL_MODE=auto keeps the configured hybrid behavior. The aliases hybrid and vector also allow the embedding/vector path. economy, bm25, and lexical are an explicit zero-embedding preset: service assembly does not construct an embedding backend or a vector store.

Metadata predicates support eq, ne, in, nin, gt, gte, lt, lte, and between. They are deterministic, order-preserving narrowing operations. Automatic filter extraction is a separate optional stage; a failed or absent extraction never broadens past the unfiltered candidate set silently.

For multiple libraries, MultiIndexRetriever asks a router for library IDs, runs each selected index independently, and fuses the ranked lists with RRF. Results carry library_id provenance. If routing fails, the safe availability fallback searches every configured library.

chunking — paragraph-granular chunker + versioned store

chunking/chunking.py turns a document's plain text into retrieval chunks. The token budget is approximated by character count (no third-party tokenizer), keeping it offline and deterministic.

Constants: DEFAULT_CHUNK_CHARS = 480, DEFAULT_OVERLAP_CHARS = 80. Oversized single paragraphs are split by sentence enders (。！？；.!?;), then hard-cut, with no overlap between the sub-chunks — so a chunk's text always stays a contiguous substring of the source, which keeps citations honest (see Provenance). chunk_document raises ValueError if max_chars <= 0, overlap_chars < 0, or overlap_chars >= max_chars.

chunking/chunk_store.py is the versioned chunk store (SQLite, mirroring the fact store: explicit schema, parameterized SQL, a read-only execute_read entry point).

• StoredChunk — every Chunk field, including parent/window fields, plus ingestion metadata: valid_as_of, ingested_at, version (default 1), active (default True). Older SQLite schemas are migrated additively.
• ChunkStore(db_path) — init_schema() creates the narrative_chunk table and is idempotent. replace_doc_chunks(doc_id, chunks, valid_as_of="") -> int does a versioned replacement: it bumps version = max(version) + 1, marks old rows active=0, inserts the new chunks active=1, and returns the number of rows written. Re-ingesting is idempotent; passing an empty list withdraws the document from the active set.
• iter_chunks(*, doc_id=None, topic=None, entity=None, geography=None, period=None, language=None, include_inactive=False) -> list[StoredChunk] — an AND-combined metadata pre-filter (active-only by default), used to narrow candidates before any scoring.

lexical — Okapi BM25 (CJK uni+bigram) + RRF fusion

lexical/retrieval.py is the scoring core. Everything is pure Python — no rank-bm25, no SDKs.

• tokenize(text) -> list[str] — lowercases; ASCII alphanumeric runs become words; CJK runs are emitted as both unigrams and adjacent bigrams. That dual granularity is what makes BM25 work for Chinese without a segmenter.
• bm25_scores(query_tokens, docs_tokens, k1=1.5, b=0.75) -> list[float] — standard Okapi BM25 (DEFAULT_BM25_K1 = 1.5, DEFAULT_BM25_B = 0.75).
• rrf_fuse(rankings, k=60) -> dict[str, float] — Reciprocal Rank Fusion, score += 1.0 / (k + rank) with rank starting at 1. The constant is DEFAULT_RRF_K = 60 (the standard RRF value).
• GlossaryQueryRewriter(max_queries=5) — a deterministic, rule-based multi-query rewriter that expands a query using the glossary's entity/metric synonyms (zero LLM). The original query is always first.

These compose into the retriever classes:

HybridRetriever.search(...) applies the metadata pre-filter before any scoring or embedding, computes chunk vectors lazily (cached by chunk_id) only for surviving candidates, and breaks ties deterministically by (-fused_score, chunk_id).

vector — injectable embedding backends (default: none)

The vector channel is an extension point, not a default. The EmbeddingBackend Protocol (defined in lexical/retrieval.py) has a single method:

class EmbeddingBackend(Protocol):
    def embed_texts(self, texts: list[str]) -> list[list[float]]: ...

You inject an implementation via the embedding_backend= keyword on HybridRetriever, NarrativeIndex, and build_narrative_retriever. The library-level default argument is None — the vector channel is off and retrieval is pure BM25 + RRF. At the service layer, though, RAGSPINE_EMBEDDING now defaults to auto: the ONNX semantic backend when [embed-onnx] is importable, else None — so a default install with the extra is genuinely hybrid out of the box, while a bare install stays byte-identical pure BM25.

vector/embedding_backends.py ships three concrete backends plus a factory:

from ragspine.retrieval.vector.embedding_backends import make_embedding_backend

# spec (case-insensitive; defaults to env RAGSPINE_EMBEDDING_BACKEND):
#   None / "none"            → None  (pure BM25 + RRF, the zero-dep fallback)
#   "auto"                   → OnnxEmbeddingBackend if [embed-onnx] importable, else None
#   "onnx" / "fastembed" / "minilm" → OnnxEmbeddingBackend (semantic, offline, deterministic)
#   "deterministic"          → DeterministicEmbeddingBackend
#   "openai"                 → OpenAIEmbeddingBackend
#   "qwen3" / "st" / "sentence-transformers" → SentenceTransformerEmbeddingBackend
backend = make_embedding_backend("onnx")

vector/store.py additionally provides a pluggable VectorStore Protocol (upsert / query / delete / count) with a zero-dependency InProcessVectorStore (brute-force cosine, id-ascending tie-break). Note its query honors a where filter but does not auto-drop RESTRICTED — that removal stays at the two authoritative exits below.

rerank — LLM listwise reranker (RRF fallback)

rerank/listwise_rerank.py reorders the top candidates with an LLM judge, behind the ListwiseJudge Protocol:

class ListwiseJudge(Protocol):
    def judge(self, query: str, candidates: list[str]) -> list[int]: ...

The entry point is listwise_rerank(query, results, judge, *, top_n=10) (DEFAULT_TOP_N = 10). Two behaviors matter:

1. RESTRICTED exit #1. Candidates whose chunk.sensitivity (case-insensitively) equals "RESTRICTED" are excluded from the set sent to the judge and held at their original RRF positions — RESTRICTED text never reaches the judge prompt. If every candidate is RESTRICTED, the judge is never called.
2. RRF fallback. On any judge exception or malformed output, the open subset degrades to identity (RRF) order. listwise_rerank never raises.

Supporting pure functions — build_listwise_prompt(query, candidates) and parse_listwise_response(text, n_candidates) (robust parse to a length-n permutation, falling back to identity) — make the rerank deterministic and testable without a real model.

link — adapter into the agent (strips RESTRICTED at exit)

link/narrative_link.py is the seam between this domain (the retrieval "B-line") and the agent (the "A-line"). It adapts a NarrativeIndex to the agent's NarrativeRetriever contract (which is defined on the agent side, in agent/agent.py).

• NarrativeIndexRetriever(index, *, retry_without_filters=True) — its retrieve(query, *, filters=None, top_k=50) -> list[dict] maps filters to metadata kwargs, calls the underlying index, retries once without filters if the filtered result is empty, and returns snippet dicts.

  RESTRICTED exit #2. The return is built as a comprehension that drops every chunk whose sensitivity equals "RESTRICTED" before any snippet dict is produced:

  return [
      _to_snippet(r)
      for r in results
      if str(r.chunk.sensitivity).upper() != RESTRICTED_SENSITIVITY
  ]

  So RESTRICTED text never reaches the LLM synthesis prompt — the same constant (RESTRICTED_SENSITIVITY = "RESTRICTED") guards both exits.

• ProviderListwiseJudge(provider) — a concrete ListwiseJudge backed by the agent's LLMProvider. It builds the prompt, makes one provider.chat(...) call, and parses the response; provider errors propagate and are caught by listwise_rerank's degradation.

• build_narrative_retriever(chunk_db, provider=None, *, embedding_backend=None) -> tuple[NarrativeIndexRetriever, ChunkStore] — the CLI/service wiring entry. It opens the chunk store, calls init_schema, and assembles the default chain: pure BM25 + RRF (no vector backend by default) + GlossaryQueryRewriter multi-query + (when provider is given) a ProviderListwiseJudge rerank. The caller owns closing the store.

A snippet dict carries full provenance: text, doc_id, title, source_locator, chunk_id, the metadata fields, sensitivity, and a nested scores dict ({"bm25", "vector", "fused"}).

Wiring it up

from ragspine.retrieval.link.narrative_link import build_narrative_retriever

# Default: pure BM25 + RRF + glossary multi-query + (with a provider) listwise rerank.
retriever, store = build_narrative_retriever("data/chunks.db")
try:
    snippets = retriever.retrieve("为什么营收下滑", filters={"entity": "ACME_CN"}, top_k=10)
    # snippets is RESTRICTED-free and carries full lineage per item
finally:
    store.close()

The opt-in capability stack (0.7.0+)

Everything above is the default loop: offline, deterministic, BM25 + RRF (dense on automatically when [embed-onnx] is present). Releases 0.7.0 and 0.8.0 add a broad set of mainstream RAG techniques as opt-in layers on the existing Protocol seams. Each is default-off and byte-identical when unselected, chosen by a make_* factory or the matching RAGSPINE_* environment variable, and each inherits the RESTRICTED two-exit isolation and provenance invariants — a new layer never weakens anti-fabrication. They group by the stage they act on.

Indexing & chunking

• Contextual Retrieval (RAGSPINE_CONTEXTUAL / make_index_text_fn) — prepends a deterministic context header (title · entity · period · heading, controlled-vocab, zero fabrication) to the index/embed text only. chunk.text, source_locator, and citations stay byte-identical.
• Layout-aware & parent-child (RAGSPINE_CHUNKER=layout|parent_child) — split on structural boundaries instead of fixed char budget. Children carry parent_id, heading, window_text, and parent_locator; the store persists them and retrieval expands the selected child into separate generation-only prompt_text.
• Sentence-window & semantic (RAGSPINE_CHUNKER=sentence_window|semantic) — one chunk per sentence with a synthesis-time window, or embedding-boundary splits (semantic uses [embed-onnx]).
• Domain presets — laws / qa / book (RAGSPINE_CHUNKER=laws|qa|book) — thin layout-aware chunkers that only change heading detection for one document family each: laws starts a section at every clause (第N条/款/项), book at every chapter (第N章/节/篇, or a markdown / numbered heading), and qa pairs each question (Q: / 问： / a ?-ending line) with the answer paragraphs that follow under a shared parent_id. Everything else — the budget, parent_id, and locators — is inherited from the base chunker.

• RAPTOR multi-granularity tree (make_raptor_retriever / RAGSPINE_RAPTOR*) — recursive deterministic threshold clustering; per-cluster is_synthesis summaries carry the union of their members' provenance and are never citable facts. Retrieval can pull a leaf (detail) or an internal node (theme).

Representation & rerank

• Semantic dense default (RAGSPINE_EMBEDDING=onnx|auto, [embed-onnx]) — the OnnxEmbeddingBackend above; auto resolves to ONNX when importable, else pure BM25, so the shipped loop becomes genuinely hybrid BM25 + dense → RRF with no config.
• Local cross-encoder rerank (RAGSPINE_RERANKER=cross_encoder|ce|auto, [rerank]) — the offline rerank brain (CrossEncoderReranker, ms-marco MiniLM); when selected it takes precedence over the LLM listwise judge.
• ColBERT late-interaction (RAGSPINE_RERANKER=colbert, [colbert]) — token-level multi-vector MaxSim scoring, shipped as a reranker.
• SPLADE learned-sparse (RAGSPINE_RERANKER=splade, [splade]) — neural sparse term-expansion scoring (interpretable like BM25, stronger), shipped as a reranker.

Query transformation

• LLM decomposition (RAGSPINE_QUERY_DECOMPOSE=llm) — multi-sub-question fan-out; each sub-question re-runs the full guarded pipeline and its answers are deterministically merged.
• HyDE · RAG-Fusion · step-back (RAGSPINE_QUERY_TRANSFORM=hyde|rag_fusion|step_back) — LLM query transforms over the base retriever. RAG-Fusion reuses rrf_fuse; HyDE's hypothetical document is a retrieval probe, never a citable fact.
• Adaptive-RAG (RAGSPINE_ADAPTIVE) — a deterministic-heuristic (or opt-in LLM) complexity classifier routes between the single-shot path and decomposition.
• Corrective retrieval / CRAG (RAGSPINE_CORRECTIVE) — upgrades the lone retry_without_filters fallback into a bounded (≤2) deterministic grade→act loop (drop-filters → rewrite → refuse); refusing weak context is the anti-fabrication-safe choice.

Every generated variant re-runs the deterministic security gate before retrieval — a competitor sub-query is refused, so home numbers never leak.

Post-retrieval

• MMR · lost-in-the-middle · compression (RAGSPINE_POSTPROCESSOR, e.g. "mmr,lost_in_middle") — a deterministic NodePostprocessor chain that runs after rerank and before prompt assembly: MMR diversity de-dup, lost-in-the-middle reordering (best hits to the head and tail), and extractive context compression. Compression writes a separate prompt_text key that the agent prefers for the prompt, leaving text and every reference field byte-identical. The LLMLingua-2 / LLM compressor is a seam follow-up.

Graph and multi-hop

• Structured relation graph (graph/ domain) — a deterministic typed graph over the controlled dimensions for subsidiary roll-up, peer comparison, and derivation tracing (fully cited), plus a GraphStore Protocol (RAGSPINE_GRAPH_STORE, in-process default + [graph] networkx adapter) and an opt-in narrative-GraphRAG extraction / community skeleton (behind [graph] + [llm]). It is a standalone multi-hop surface, not a router branch — see Channels.
• Relation-extractor slot (RAGSPINE_RELATION_EXTRACTOR, make_relation_extractor) — an opt-in slot beside build_relation_graph for relations that live only in narrative text. The default (None) leaves the base graph byte-identical; the deterministic co-occurrence extractor adds clean-lineage co_occurs_with edges; the LLM extractor (behind [llm]) stamps every edge model-derived + unverified, screens both endpoints through the SecurityGate, and never lets RESTRICTED text reach the model. See ADR 0015.

Multimodal

• ColPali visual-document retrieval (RAGSPINE_VISUAL_EMBEDDER=colpali, [colpali]) — page-as-image late interaction (MaxSim over visual patches, reusing the ColBERT scorer), for chart- and figure-dense reports where OCR→text loses layout. Opt-in and GPU; a visual hit is a page-reference lead (is_visual), never a citable fact, and RESTRICTED pages are dropped at index construction. Real GPU end-to-end is a follow-up (colqwen2 is the more permissive model alternative).

Opting in is uniform — inject a factory result, or set the env var and let the service wire it:

from ragspine.retrieval.rerank.cross_encoder import make_reranker
from ragspine.retrieval.postprocess import make_postprocessor
from ragspine.retrieval.link.narrative_link import build_narrative_retriever

# Offline cross-encoder rerank + an MMR / lost-in-the-middle post-chain.
retriever, store = build_narrative_retriever(
    "data/chunks.db",
    reranker=make_reranker("cross_encoder"),                  # or RAGSPINE_RERANKER=cross_encoder
    postprocessor=make_postprocessor("mmr,lost_in_middle"),   # or RAGSPINE_POSTPROCESSOR="mmr,lost_in_middle"
)

See also