Table of Contents

πŸ‡«πŸ‡· Version franΓ§aise

See also: ADR-006 β€” RAG subsystem Β· Memory system Β· Opt-in subsystems Β· Back to index

RAG pipeline

Retrieval-Augmented Generation subsystem (src/rag/): incremental ingestion with security validation, a staged query pipeline (transform β†’ retrieve β†’ fuse β†’ rerank β†’ assemble β†’ generate β†’ groundedness) whose every stage is traced, hybrid BM25 + vector retrieval, pluggable rerankers, a corrective CRAG graph built on the in-house Graph orchestration mode, quality profiles (fast / balanced / quality / adaptive / corrective), YAML crew integration with cited knowledge injection, and an offline evaluation harness gated in CI.

Overview

An agent that pastes whole documents into its prompt drowns its context and cannot cite anything. The RAG subsystem precomputes and structures that knowledge:

  • Ingestion: documents are loaded (files, CSV, HTML, PDF, web pages), validated against prompt injection, chunked, embedded, and upserted into a collection of the document store. A per-collection manifest makes re-ingestion incremental β€” an unchanged source costs zero embeddings.
  • Query: a question flows through a fixed sequence of togglable stages and comes back as a RagAnswer β€” generated text with [n] citation markers, the citations resolving each marker to its source chunk (score and offsets included), and the full execution trace of the run.
  • Quality dials: one knob (Orkeon:Rag:Profile) selects a preset; every value can then be overridden key by key through configuration. Profile = preset, configuration = override.

Three consumption surfaces share the same pipelines: agent tools (rag_search, rag_ingest, rag_eval in Orkeon.Tools.Rag), the crew YAML rag: / knowledge: blocks, and the scripting DSL rag.* namespace.

Projects (ADR-006)

Project Role Depends on
Orkeon.Rag.Abstractions Contracts, DTOs, options (IRagPipeline, RagAnswer, RagOptions, RagProfilePresets…) Orkeon.Domain only
Orkeon.Rag Implementations: loaders, chunkers, validation, stores, retrieval, reranking, routing, evaluation, DI Application, Analysis.Abstractions (see ADR-006)
Orkeon.Rag.Onnx Opt-in ONNX cross-encoder reranker (ms-marco-MiniLM-L-6-v2), AddOrkeonOnnxReranker() Orkeon.Rag (concrete)
Orkeon.Rag.Onnx.Model Companion package embedding the int8 model weights β€” guaranteed offline β€”
Orkeon.Tools.Rag Agent tools rag_search / rag_ingest / rag_eval, AddOrkeonRagTools() Orkeon.Rag.Abstractions, Orkeon.Rag (concrete β€” the ADR-006-guarded coupling), Tools.Abstractions, Analysis.Abstractions

Orkeon.Rag.Abstractions is a secondary shared kernel (same status as Orkeon.Analysis.Abstractions, ADR-003): Orkeon.Application references it for the ports; Orkeon.Infrastructure references Orkeon.Rag solely as DI composition wiring.

Quick start

The subsystem is opt-in β€” nothing is wired by AddOrkeonInfrastructure() alone. Every runner host built on RunnerHost (orkeon run in all its forms, orkeon-host) makes the two calls below itself, and each shipped binary β€” orkeon, orkeon-host, the orkeon-repl REPL β€” adds the ONNX reranker; a host that builds its own container makes them:

services.AddOrkeonLocalEmbeddings();      // or any IEmbeddingProvider (local BGE: no API key)
services.AddOrkeonInfrastructure();       // semantic-first defaults for embeddings + chat
services.AddOrkeonRag(configuration);     // the subsystem (Orkeon.Rag.DependencyInjection)
services.AddOrkeonRagTools();             // optional: rag_search / rag_ingest / rag_eval tools
var ingestion = provider.GetRequiredService<IIngestionPipeline>();
await ingestion.IngestAsync(new IngestionRequest
{
    Collection = "product-kb",
    Sources = [new SourceDescriptor { Location = "/kb/faq.md" }],
});

var rag = provider.GetRequiredService<IRagPipeline>();
var answer = await rag.QueryAsync(new RagQuery
{
    Text = "How many days do customers have to request a refund?",
    Collection = "product-kb",
    TopN = 3,
});
// answer.Text        β€” generated text with [n] markers
// answer.Citations   β€” marker -> chunk id, source id, snippet, score, offsets
// answer.Trace       β€” stages executed, variants, route, verdicts

The host must provide an IEmbeddingProvider and an IChatClient; AddOrkeonInfrastructure() registers semantic-first defaults for both (resolution order for embeddings: container-registered local/Analysis provider β†’ Orkeon:Embeddings configuration β†’ fail-fast at first use, never silently).

Which model the subsystem asks: the host's default profile, unless Orkeon:Rag:LlmProfile names another of its profiles (Llm:Profiles:<name>, through ILlmProfileRegistry). Then grounded generation, the query transformers, the listwise reranker, the corrective graph's evaluator and groundedness checker, the llm classifier and the evaluation judge all call that profile β€” one profile for the whole subsystem, whatever RAG profile (Orkeon:Rag:Profile) a query runs on. The profile's chat client is resolved at the first RAG call that needs a model, never when a crew merely loads; RagLlm.EnsureProfileIsKnown, which the runner host and the REPL call once the container is built, checks the name alone and refuses one the host does not offer β€” orkeon-host's Orkeon:Host:LlmProfiles allow-list included β€” listing the known ones. orkeon eval rag --offline ignores the key: the extractive stub answers in place of every model.

Runnable, fully offline demos: rag/basic-ingestion, rag/hybrid-retrieval, rag/custom-reranker, rag/crew-yaml, and the scripting variant scripting/08-rag.ork.ts.

Registration extensions

AddOrkeonRag(configuration) is the one call a host needs; it composes the narrower extensions below, all TryAdd* (a host registration always wins) and safe to call on their own:

Extension Registers
AddOrkeonRag(configuration) The whole subsystem: options, loaders, validation, document store + hybrid decorator, manifests, both pipelines, profile resolver, and every extension below except the ONNX reranker and AddOrkeonEphemeralSearch
AddOrkeonQueryTransforms() QueryTransformerFactory with none / multi-query / rag-fusion / hyde
AddOrkeonRagReranking() RerankerFactory with none / llm / decision plus every IRerankerRegistrar
AddOrkeonQueryRouting(configuration) The IQueryComplexityClassifier named by Orkeon:Rag:QueryRouting:Classifier
AddOrkeonHybridRetrieval(configuration, …) The HybridSearchDocumentStore decorator (BM25 + RRF) around the IDocumentStore
AddOrkeonCorrectiveRag(configuration) IRetrievalEvaluator, IGroundednessChecker, CorrectiveRagPipeline (experimental, ORKEXP003)
AddOrkeonRagWebFallback(configuration) The IWebDocumentRetriever, only when Orkeon:Rag:WebFallback is enabled and has an Endpoint
AddOrkeonKnowledgeAugmentation() IKnowledgeContextAugmenter (agent knowledge: attachments)
AddOrkeonRagEvaluation() Dataset loader, IRagEvaluator, report writer, IRagEvalHarness
AddOrkeonEphemeralSearch() IEphemeralCollectionSearch behind the txt_search / mdx_search / pdf_search / directory_search tools β€” fails at call time, not at registration, when AddOrkeonRag is absent
AddOrkeonOnnxReranker(configure?) Orkeon.Rag.Onnx: the onnx / cross-encoder reranker
AddOrkeonRagTools() Orkeon.Tools.Rag: rag_search, rag_ingest, rag_eval

So a host that calls AddOrkeonRag never calls the six narrower ones (AddOrkeonQueryTransforms, AddOrkeonRagReranking, AddOrkeonQueryRouting, AddOrkeonHybridRetrieval, AddOrkeonKnowledgeAugmentation, AddOrkeonRagEvaluation) itself β€” they are idempotent, so a second call is harmless. Call them alone only to compose a partial stack by hand, e.g. the evaluation harness over a host-built IRagProfileResolver, or a factory without the rest of the subsystem. Two ordering rules: AddOrkeonHybridRetrieval decorates the last registered IDocumentStore and throws when none is registered yet (it also binds Orkeon:Rag:Retrieval:Hybrid, flat = true shorthand included); AddOrkeonKnowledgeAugmentation needs an IRagProfileResolver at resolution (AddOrkeonRag registers it) β€” without the augmenter, agent knowledge: attachments are not injected, and the crew factory warns about each agent that declares some. The genuinely opt-in extensions, which AddOrkeonRag does not call, are AddOrkeonOnnxReranker, AddOrkeonEphemeralSearch and AddOrkeonRagTools.

Contracts

All in Orkeon.Rag.Abstractions (Interfaces/, Models/, Options/):

Contract Role
IRagPipeline Query faΓ§ade: RagQuery β†’ RagAnswer (citations + trace)
IRagRetrievalCapable Opt-in: the retrieval half alone (transform β†’ retrieve β†’ fuse β†’ rerank β†’ assemble), no generation, no LLM call
IIngestionPipeline Ingestion faΓ§ade: IngestionRequest β†’ IngestionReport
IDocumentStore Upsert / search / delete-by-source over a named collection
IDocumentLoader Source β†’ RagDocument (text, CSV, HTML, PDF, web page)
IChunkingStrategy Document β†’ Chunk slices (recursive, sentence, structural, semantic)
IQueryTransformer Query β†’ retrieval texts + a QueryTransformKind (see below)
IReranker Candidates β†’ best topN, reranker-scored (cascade CandidateK β†’ TopN)
IQueryComplexityClassifier Query β†’ QueryRoute (NoRetrieval / SingleShot / Iterative)
IRagProfileResolver Profile name β†’ memoized pipeline instance
IKnowledgeContextAugmenter Agent knowledge attachments β†’ cited prompt block
IRagCollectionsBootstrapper Crew rag: block β†’ kickoff ingestion
IRetrievalEvaluator, IGroundednessChecker Corrective hooks (verdicts, hallucination check β€” see Corrective RAG); experimental, ORKEXP003
IRagEvaluator, IRagEvalHarness* Offline evaluation (metrics, golden datasets)
IEphemeralCollectionSearch Throwaway in-memory collections (AddOrkeonEphemeralSearch)
IDocumentStoreCollectionProbe Optional store capability (HasContentAsync): lets ingestion notice a manifest that outlived its data β€” implemented by MemoryProviderDocumentStore and forwarded by the hybrid decorator

* the harness contract and its runner live in Orkeon.Rag.Evaluation; the evaluator port is in the abstractions.

Named components (chunkers, transformers, rerankers) resolve through factories following the MemoryProviderFactory pattern: names and aliases matched case-insensitively, and an unknown name always fails loudly with the list of known names β€” never a silent fallback.

Staged query pipeline

StagedRagPipeline (default IRagPipeline) is a fixed sequence of togglable stages driven by RagOptions:

flowchart LR
    Q[RagQuery] --> T[transform]
    T --> R[retrieve]
    R --> F[fuse]
    F --> RR[rerank]
    RR --> A[assemble]
    A --> G[generate]
    G --> GR[groundedness]
    GR --> ANS[RagAnswer<br/>text + citations + trace]
# Stage What it does Trace data (excerpt)
1 transform Named transformer produces retrieval texts (none skips) mode, transformer, kind, variants, variant_1…variant_n
2 retrieve CandidateK candidates per retrieval text; hybrid honoured per query by capable stores candidates, top_k, mode, hybrid, variants
3 fuse Per-kind combination of the per-text rankings + dedup by chunk id, then opt-in MMR method (rrf/union/dedup), in, out, mmr*
4 rerank Named reranker, cascade CandidateK β†’ TopN (disabled: truncation in retrieval order) reranker, candidates, kept
5 assemble Token budget (β‰ˆ 4 chars/token) + anti-Lost-in-the-Middle edges ordering chunks, ordering, dropped_by_budget
6 generate Grounded generation with [n] markers (anti-hallucination system prompt) model
7 groundedness Optional post-generation verification hook (IGroundednessChecker) grounded, score

Every stage appends a RagTraceStep to RagAnswer.Trace β€” durations, counts, and chosen component names are always observable. When retrieval yields no candidate, generation is skipped and a deterministic no-context answer is returned (the pipeline never lets the model answer ungrounded).

Query transformers by kind

Transformers declare how their output texts combine at the fuse stage (QueryTransformKind):

Kind Built-in Semantics
Union multi-query Retrieve once per text (original + variants); merge by chunk-id union, original store scores kept β€” the maximum wins on duplicates
Fusion rag-fusion Retrieve once per text; fuse the rankings by Reciprocal Rank Fusion (same RrfK as the hybrid stage)
Replacement hyde The substitute text(s) are the retrieval probes instead of the question β€” generation and citations always use the original question

Built-ins registered by AddOrkeonRag: none, multi-query, rag-fusion, hyde. The LLM-backed ones resolve the subsystem's chat client (the default profile's, or the one Orkeon:Rag:LlmProfile names) lazily; parsing of the LLM's variants is tolerant, and an empty output degrades to the original query.

Anti-Lost-in-the-Middle assembly

With Context.Ordering: edges (the default), ranked chunks r1 (best) … rm are laid out as r1, r3, r5, … from the head and …, r6, r4, r2 closing the tail β€” the two strongest chunks sit at the extremities where LLM attention is highest, the weakest in the middle. Citation markers stay rank-based ([1] = best chunk) whatever the layout. linear restores plain rank order.

Hybrid retrieval (BM25 + RRF)

AddOrkeonRag always wraps the registered IDocumentStore β€” including a host-registered one β€” in the HybridSearchDocumentStore decorator, so ingestion feeds an in-process per-collection BM25 index alongside the vector store. Whether a search actually fuses is decided per query (RetrievalQuery.Hybrid, set by the profile presets), with Orkeon:Rag:Retrieval:Hybrid:Enabled as the default mode; a disabled default is a strict behavioural passthrough.

Score provenance (ScoredChunk.ScoreOrigin):

  1. hybrid-native β€” the backing memory provider implements IHybridSearchCapable (e.g. LanceDB): text + vector fused provider-side.
  2. rrf β€” emulated path: inner vector ranking + in-process BM25, fused with Reciprocal Rank Fusion. RRF scores are rank aggregates, not similarities.
  3. Inner origin (vector / local-cosine) β€” passthrough when the lexical side has nothing to contribute.

The in-process BM25 index only covers chunks upserted through the decorator in the current process; at scale, prefer a provider with native hybrid search β€” the decorator switches to it automatically.

MMR diversification

Opt-in (Orkeon:Rag:Retrieval:Mmr:Enabled), applied at the fuse stage: Maximal Marginal Relevance re-orders the fused candidates by trading relevance against redundancy β€” each pick maximises λ·relevance βˆ’ (1 βˆ’ Ξ»)Β·max-similarity-to-already-picked (default Ξ» = 0.7, relevance-leaning). Candidate embeddings are not re-computed: the lexical (Jaccard) similarity fallback is the honest default.

Reranking

The rerank stage narrows CandidateK candidates (default 50) to TopN (default 5) with a named IReranker:

Name (aliases) Implementation Notes
none (noop) NoopReranker Truncation in retrieval order
llm (listwise) LlmListwiseReranker One listwise chat call; tolerant parsing (a malformed answer keeps the retrieval order); the built-in alternative when the ONNX package is absent β€” select it explicitly
onnx (cross-encoder) Orkeon.Rag.Onnx ms-marco-MiniLM-L-6-v2 cross-encoder, int8 weights embedded, fully offline; AddOrkeonOnnxReranker()
decision DecisionModelReranker One request to the host's decision model: the query and every passage as the state, one yes-or-no question per passage, the score being the probability of a yes; a passage is cut at 4000 characters. Needs an active decision provider (AddOrkeonDecisionProvider), otherwise resolving it throws InvalidOperationException

Hosts contribute rerankers through IRerankerRegistrar β€” registered in DI, applied when the singleton RerankerFactory is built, registration order irrelevant (this is how the ONNX package plugs in; see examples/rag/custom-reranker for a host-provided one). The factory itself is TryAdd-registered: a host-registered RerankerFactory wins outright.

Reranked chunks carry their scorer in ScoredChunk.ScoreOrigin: cross-encoder (sigmoid-mapped logits in (0, 1)), llm-rerank (normalized descending rank, best = 1.0) or decision-rerank (the probability of a yes).

The ONNX reranker resolves its model lazily at first use: the companion package Orkeon.Rag.Onnx.Model when referenced (embedded weights, guaranteed offline), otherwise OnnxRerankerOptions.ModelPath + VocabPath (VFS virtual paths, both required), otherwise a loud error β€” the runtime never downloads weights (tools/download-reranker-model.sh in the package provisions them, SHA-256 verified). AddOrkeonOnnxReranker(o => …) also sets BatchSize (default 16) and MaxSequenceLength (default 512); these options are C#-only, bound on no configuration section.

Profiles

RagProfilePresets expands a profile name into a complete RagOptions; the Orkeon:Rag configuration section then overrides any value individually.

fast (default) balanced quality adaptive corrective
Retrieval vector only, CandidateK = TopK hybrid BM25 + RRF, CandidateK = 50 hybrid, CandidateK = 100 routed (see below) hybrid, graph retrieve node (see CRAG)
Rerank off ONNX cross-encoder 50 β†’ 5 ONNX cross-encoder routed none β€” the graph corrects by looping, not reranking
Groundedness off off on routed native check_groundedness graph node
Needs nothing Orkeon.Rag.Onnx Orkeon.Rag.Onnx Orkeon.Rag.Onnx (its SingleShot route runs balanced) + classifier + chat client chat client (grader/rewrite; heuristic fallbacks without one)

fast is the out-of-the-box default because balanced requires the opt-in ONNX package β€” defaulting to it would make every bare AddOrkeonRag() host refuse to start, naming onnx (the effective options are judged at the host's start). Opt into balanced with one configuration line.

IRagProfileResolver (ProfileRagPipelineResolver) memoizes one pipeline per profile name; the reserved name default resolves to the host's registered IRagPipeline, so a host-provided pipeline keeps the last word for non-profiled queries. Unknown names fail loudly with the list of known profiles.

Adaptive routing (Adaptive-RAG)

The adaptive profile resolves to an AdaptiveRagPipeline: an IQueryComplexityClassifier routes each query first, and the decision is always traced (RagTrace.Route + a route step carrying classifier, route, delegate).

flowchart LR
    Q[query] --> C{classifier}
    C -- NoRetrieval --> D[direct LLM answer<br/>no citations]
    C -- SingleShot --> B[balanced pipeline]
    C -- Iterative --> CO[corrective graph pipeline]

Classifiers (Orkeon:Rag:QueryRouting:Classifier): heuristic (default β€” deterministic rules, zero LLM call) or llm (one constrained lightweight chat call; when selected without an IChatClient the heuristic is the documented fallback, with a warning).

The Iterative route delegates to the memoized corrective graph pipeline (see Corrective RAG); the route trace step carries delegate=corrective and RagTrace.Route records the decision.

Configuration reference (Orkeon:Rag)

Bound over the selected preset β€” every key is an individual override.

Key Default Meaning
Orkeon:Rag:Profile fast Preset: fast / balanced / quality / adaptive / corrective (unknown fails loudly)
Orkeon:Rag:Collection β€” The collection rag_search queries when the agent names none (unset: default), and the one rag_eval evaluates when neither the call nor the dataset names one and the dataset brings no corpus. rag.query/rag.ingest and knowledge: always name theirs
Orkeon:Rag:LlmProfile default profile The host LLM profile (Llm:Profiles:<name>) every model call of the subsystem goes to β€” generation, transformers, listwise reranker, corrective evaluator and groundedness checker, llm classifier, evaluation judge; an unknown name refuses the host start, listing the known ones
Orkeon:Rag:Provider ambient Document-store provider type (inmemory/in-memory, redis, sqlite, chromadb/chroma, pinecone, lancedb/lance; unknown fails loudly); unset = ambient IMemoryProvider. The connection is that provider's own host section (Orkeon:Redis, Orkeon:Sqlite, Orkeon:ChromaDb, Orkeon:Pinecone, Orkeon:LanceDb), and the store shares the factory's instance of that type with crew memory
Orkeon:Rag:Retrieval:TopK 5 Chunks kept for context assembly (call-site RagQuery.TopN wins)
Orkeon:Rag:Retrieval:CandidateK 5 (fast) Wide stage of the cascade (always β‰₯ final TopN); fast has none and takes TopK, a profile that reranks takes 50, quality 100
Orkeon:Rag:Retrieval:MinScore none Optional raw-score floor; unset, no floor applies
Orkeon:Rag:Retrieval:Hybrid:Enabled false (fast) Default hybrid mode; flat shorthand Retrieval:Hybrid = true accepted
Orkeon:Rag:Retrieval:Hybrid:RrfK 60 Reciprocal Rank Fusion constant
Orkeon:Rag:Retrieval:Mmr:Enabled false MMR diversification at the fuse stage
Orkeon:Rag:Retrieval:Mmr:Lambda 0.7 Relevance/diversity trade-off in [0, 1]
Orkeon:Rag:QueryTransform:Mode none none / multi-query / rag-fusion / hyde
Orkeon:Rag:QueryTransform:VariantCount 3 Variants requested from non-none transformers
Orkeon:Rag:Rerank:Enabled false (fast) Whether the rerank stage runs
Orkeon:Rag:Rerank:Kind none none / llm / onnx / decision / any registered name
Orkeon:Rag:Rerank:TopN 5 Chunks kept after reranking
Orkeon:Rag:Context:MaxTokens 2000 Context budget (β‰ˆ 4 chars/token; excess chunks dropped, never overrun)
Orkeon:Rag:Context:Ordering edges edges (anti-Lost-in-the-Middle) or linear
Orkeon:Rag:Groundedness:Enabled false Post-generation verification (traced as skipped without a checker)
Orkeon:Rag:Generation:SystemPrompt built-in Grounded system prompt override
Orkeon:Rag:Generation:Temperature / MaxOutputTokens β€” Sampling passed to the chat client
Orkeon:Rag:QueryRouting:Classifier heuristic heuristic or llm (adaptive profile)
Orkeon:Rag:Corrective:MaxIterations 3 Corrective iteration budget β€” query rewrites, whether triggered by an Incorrect verdict or an ungrounded answer
Orkeon:Rag:Corrective:WebFallback:Enabled false Pipeline-side policy: may the corrective graph route to its web_fallback node
Orkeon:Rag:Corrective:WebFallback:MaxResults 3 Web documents the graph asks the retriever for
Orkeon:Rag:WebFallback:* disabled Separate section β€” transport of the web retriever (Enabled, Endpoint, ApiKeyEnvVar, MaxResults, Timeout, SuspiciousAction); see Web fallback
Orkeon:Rag:Ingestion:DefaultChunkingStrategy recursive Strategy when a request names none
Orkeon:Rag:Ingestion:ManifestDirectory /output/rag/manifests VFS directory of the per-collection manifests

Ingestion

DefaultIngestionPipeline runs: load (DocumentLoaderFactory selects by source kind/extension β€” text, CSV, HTML, PDF, WebPageLoader for URLs) β†’ security validation β†’ chunk (named strategy) β†’ embed β†’ upsert.

Loader Claims SourceDescriptor.Kind hint
TextFileLoader .txt, .md, .markdown, .text, .log file (or none)
CsvDocumentLoader .csv file (or none)
HtmlDocumentLoader .html, .htm file (or none)
PdfDocumentLoader .pdf file (or none)
WebPageLoader http:// / https:// URLs url (or none)
InlineTextLoader inline content (registered by AddOrkeonEphemeralSearch) text

File loaders read through the VFS (virtual paths only). WebPageLoader fails closed without an IUrlValidator (AddOrkeonInfrastructure() registers one): every URL is SSRF-validated before the fetch, and its HTTP client follows no redirect, so an approved URL cannot be bounced to an internal address.

Chunking strategy (aliases) Behaviour
recursive (recursive_text, default) Recursive separator split β€” the default
sentence (sentences) Sentence-bounded chunks
structural (markdown, headings) Heading-aware split
semantic Closes a chunk when the next sentence drifts from the chunk's centroid (threshold 0.75, similarity_threshold extension); without an embedding function it degrades to structural β€” and AddOrkeonRag wires none

ChunkingOptions (per IngestionRequest) sizes chunks in characters: MaxChunkSize 1000, Overlap 200 by default.

  • Validation (ingestion-path security): PromptInjectionDocumentValidator and ContentIntegrityValidator run on every document; rejected content goes to the IQuarantineStore and provenance is recorded (IProvenanceTracker). Details and threat model in security.md.
  • Incremental manifest (per collection, one JSON file written through the VFS at {ManifestDirectory}/{collection}.json): each source's content hash and the embedding profile (provider, model, dimensions) are recorded. Unchanged sources are skipped entirely β€” chunksEmbedded = 0; changed ones are purged and re-ingested. Failure posture: a missing/corrupt manifest degrades to full ingestion, an unwritable manifest directory degrades to a warning (ingestion works, just never incrementally) β€” never a crash.
  • Stale manifest detection: the manifest (VFS) and the default in-memory store have different lifetimes β€” a second process would find every source "unchanged" against an empty store. When the store implements IDocumentStoreCollectionProbe and reports the collection empty while the manifest lists sources, the manifest is discarded and the run ingests in full.
  • Reindex = true purges every recorded source and rebuilds the collection. It is also the only way to re-ingest after an embedding-profile change: a drift between the manifest's recorded profile and the active provider fails the run loudly rather than silently mixing incompatible vectors.
  • prune (rag_ingest option, rag.ingest({ prune: true })): deletes the chunks and the manifest entries of manifest sources that are absent from the ingested source set, and reports them as SourcesPruned (<n> pruned in the tool output, sourcesPruned in the script payload). Pruning is skipped for a run that recorded any error, so a failed load never causes data loss.
  • Scope and concurrency:
    • Store persistence follows Orkeon:Rag:Provider: the default in-memory store is lost at process end; sqlite, redis, chromadb, pinecone and lancedb persist.
    • The document store is host-global: one store per host, shared by every workspace and crew. Chunks ingested through one workspace are retrievable from another.
    • The ingestion manifest is per workspace (/output/rag/manifests/{collection}.json through each workspace's VFS). A second workspace sharing a persistent store sees an empty or different manifest and re-ingests.
    • Ingests of one collection are serialized by an in-process per-collection lock (from the manifest read to its write); different collections run concurrently. The collection-registry update of the store is atomic per provider.
    • The lock is in-process only: a cross-process lock needs provider support and is out of scope, so two host processes sharing a store can still race.
  • Glob sources (*, **, ?) are expanded through the VFS by SourceGlobExpander at every consuming surface (scripting rag.ingest, rag_ingest tool, CLI orkeon rag ingest, eval harness corpus, and the crew rag: block). An http:// / https:// address is never a pattern: the ? of its query string or a * stay part of it, and it reaches WebPageLoader as written, without the VFS being asked (SourceGlobExpander.IsWebAddress, one rule for every surface). The crew block also takes a directory, which stands for every file below it, and resolves a relative path against the crew's folder β€” all of it decided in RagCollectionsBootstrapper, which warns, naming the collection, about a pattern or a directory that yields no file.

Crew YAML integration

Two YAML blocks, consumed at different moments (see examples/rag/crew-yaml):

rag:                        # crew-level: collections ingested at crew creation
  collections:
    product-kb:
      sources: [/kb/faq.md]
      chunking: { strategy: recursive, max_tokens: 256, overlap: 32 }

agents:
  support:
    role: Support agent
    goal: Answer from the knowledge base
    knowledge: [product-kb] # agent-level: retrieval attachment
  • The rag: block is parsed into RagCrewConfig; CrewFactory hands it to the IRagCollectionsBootstrapper at CreateFromConfigAsync β€” kickoff-time ingestion through the incremental pipeline (a fresh manifest makes it a no-op). A crew declaring rag: on a host without AddOrkeonRag logs a warning instead of failing; so does an agent that declares knowledge: there.
  • knowledge: attachments travel on the Agent aggregate; at task-context assembly the execution orchestrator calls IKnowledgeContextAugmenter. For each attachment it resolves the pipeline of the attachment's profile and runs its retrieval half only (IRagRetrievalCapable.RetrieveAsync β€” transform, hybrid, rerank and assembly as the profile says, no generation, no nested LLM call), then injects a bounded, numbered block with [n] citations into the agent's prompt.

Where it runs. Every host built on RunnerHost registers the subsystem: orkeon run on a YAML crew, a crew directory or a .ork.ts script (with or without --events), and orkeon-host; so does orkeon-repl, and the C# host of examples/rag/crew-yaml calls AddOrkeonRag itself. orkeon run --validate loads the crew without ingesting anything. Nothing of the subsystem is built until a crew uses it: a crew without rag:/knowledge: loads no embedding model for it. Its settings are judged at the host's start, though, whether a crew uses them or not: an unknown Orkeon:Rag:Provider, profile, reranker, query transformer, context ordering, classifier or chunking strategy refuses the start, with the list of known names (when a setting is refused). Ingestion needs an embedding provider: the default runner has the on-device one (unless RaggableTree:Enabled is false and nothing else is configured). The incremental manifests land under /output/rag/manifests; without a writable /output mount the collections are ingested on every load, with a warning.

YAML keys:

Key Effect
rag.collections.<name>.sources Ingested at crew creation: files, globs, directories (every file below) and URLs; a path without a leading / is relative to the crew's folder (/crew under orkeon run); a pattern or a directory that yields nothing is a load warning
rag.collections.<name>.chunking (strategy, max_tokens, overlap) Tokens converted to characters at Γ—4
rag.defaults.profile The retrieval profile of every attachment that names none (copied onto it at crew creation)
knowledge: short form ([product-kb]) Attaches the collection with the defaults below
knowledge: long form (collection, top_k default 5, min_score, max_context_tokens default 2000) top_k is the query's TopN; min_score filters on the score the profile reports (vector similarity under fast, the cross-encoder's score under a reranking profile); max_context_tokens caps the attachment's excerpts (Γ—4 characters)
knowledge: long form profile The retrieval pipeline: the attachment's profile, else rag.defaults.profile, else Orkeon:Rag:Profile (fast when unset). fast, balanced and quality retrieve alone; corrective and adaptive cannot (their graphs generate as they go) and fail the task with a message saying so; an unknown name fails as the resolver does. balanced/quality also need the ONNX reranker, which orkeon, orkeon-host and orkeon-repl register

The document store is the host's choice (Orkeon:Rag:Provider), not the crew's: a crew that writes rag.provider is refused at load ('rag.provider' is retired: the document store is the host's choice β€” set Orkeon:Rag:Provider in the settings).

Scripting and CLI surfaces

  • Scripting DSL (.ork.ts): first-class rag.ingest({ collection, sources, chunkingStrategy?, reindex?, prune? }), rag.query(question, { collection, profile?, topN? }) and rag.retrieve(question, { … }) globals β€” see examples/scripting/08-rag.ork.ts. The namespace is always registered; calls fail with an actionable message when the host did not wire the subsystem. query and retrieve return the same payload shape and only query runs the generation stage, so a script that reads only citations should call retrieve β€” see below.
  • CLI (orkeon): orkeon rag ingest|search|eval over the same pipelines β€” flags in the CLI reference. A collection whose manifest records another embedding provider, model or dimensions is refused (exit 1, naming the collection, both models and orkeon rag ingest --reindex) on rag ingest without --reindex, on rag search, on eval rag --no-ingest -c <collection> and on eval rag -c <collection> when its ingest pre-pass runs without --reindex. A collection with no manifest, and eval rag without -c, are not checked.
  • Agent tools (Orkeon.Tools.Rag, AddOrkeonRagTools(), requires AddOrkeonRag):
Tool Parameters Behaviour
rag_search question (required), top_k (default 3), collection (default Orkeon:Rag:Collection, else default) IRagPipeline.QueryAsync β†’ answer + Sources: block with scores; collection = "raggable-tree" routes to the RaggableTree code index when one is registered
rag_ingest collection, sources (paths / globs / http(s) addresses) β€” both required; chunking_strategy, reindex, prune IIngestionPipeline with glob expansion; prints the ingestion report
rag_eval dataset (required), collection, profile (default default), compare, k (default 5), use_llm_judge, reindex The offline harness below

A crew agent that lists one of the three tools (tools: [rag_search]) receives it wherever the host registered them: every RunnerHost host (orkeon run on a YAML crew, a crew directory or a .ork.ts script, and orkeon-host), the REPL, and a C# host that calls AddOrkeonRagTools(). Building a tool resolves nothing: the pipelines are resolved at its first call.

Retrieval without generation

IRagRetrievalCapable.RetrieveAsync runs stages 1-5 and stops. Same query, same passages, no LLM call.

A caller that quotes the retrieved passages does not need a generated answer: retrieve skips the generation call and its tokens.

if (pipeline is IRagRetrievalCapable retriever)
{
    var answer = await retriever.RetrieveAsync(
        new RagQuery { Text = question, Collection = "kb", TopN = 6 });
    // answer.Text is empty; answer.Citations holds the assembled passages.
}

Two properties worth knowing before relying on it:

  • The capability is opt-in and must be probed. StagedRagPipeline implements it; the corrective graph does not, because it interleaves retrieval evaluation with generation β€” "retrieval only" is not a prefix of its run. rag.retrieve throws on a pipeline that cannot honour it instead of falling back to QueryAsync, which would charge exactly what the caller asked to avoid.
  • The trace still carries a generate step, whose detail says the stage was skipped on purpose. An absent step would read as a trace produced by an older pipeline.

Evaluation (golden datasets, CI gate)

The offline harness (IRagEvalHarness) turns retrieval quality into a published number instead of a claim:

  • Dataset (examples/rag/eval/golden.yaml): a corpus directory plus cases β€” question, relevant source refs (suffix-matched against ingested ids), expected_substrings, reference_answer, tags. The corpus is ingested (incrementally) before every run.
  • Metrics: recall@k, precision@k and MRR per case and aggregated (deterministic); generation (groundedness, answer-relevance) is judged by an LLM judge when available, with a deterministic heuristic fallback β€” the report always labels which judge ran.
  • Profiles compared in one run: orkeon eval rag --dataset … --compare fast,balanced,quality,corrective,adaptive --offline (--offline swaps generation for a deterministic extractive stub β€” zero network; the measured table and its honest reading live in examples/rag/eval/README.md).
  • CI gate (.github/workflows/rag-eval.yml): the comparison table is published to the step summary, and an anti-regression gate fails the build when the balanced profile's aggregate recall@5 or MRR drops below the floors (--min-recall 0.80 --min-mrr 0.70, the workflow's env: block). Cases tagged correctif are excluded from the gates: they are seeded to fail single-shot retrieval on purpose (see below).
  • Schema: published as interface RagEvalDataset / RagEvalCase in the scripting typings (src/scripting/Orkeon.Scripting/Typings/rag.d.ts). A case without question is refused with an error naming case #<index> ('<id>') and the missing field.
  • Hosts: the rag script namespace (rag.retrieve, rag.ingest, …) is bound in the three hosts β€” orkeon run, the REPL and the daemon (RunnerExecution) β€” through one RagScriptingBackendFactory.Build(IServiceProvider) that resolves IIngestionPipeline from DI, so the per-collection ingest lock is shared with rag_ingest.

Corrective RAG (CRAG)

The corrective profile resolves to CorrectiveRagPipeline (Orkeon.Rag.Corrective) β€” CRAG built on Orkeon's Graph orchestration mode: the pipeline's execution is a StateGraph<RagGraphState> β€” the same Domain StateGraph<TState> engine behind ProcessType.Graph β€” with conditional edges, controlled cycles and the graph's native circuit breaker. The corrective engine is not a bespoke loop bolted onto RAG; RAG demonstrates the Graph orchestration mode and vice versa. Same IRagPipeline faΓ§ade as every other profile: the profile selects the executor, callers only ever see a RagAnswer. The corrective types (CorrectiveRagPipeline, IRetrievalEvaluator, IGroundednessChecker, IWebDocumentRetriever, AddOrkeonCorrectiveRag…) are marked experimental β€” diagnostic ORKEXP003, see experimental APIs.

Graph topology

flowchart LR
    S((start)) --> R[retrieve]
    R --> E{evaluate}
    E -- Correct --> G[generate]
    E -- Ambiguous --> RF[refine]
    E -- "Incorrect Β· budget left" --> RW[rewrite_query]
    RW --> R
    E -- "Incorrect Β· budget exhausted, opt-in" --> W[web_fallback]
    RF --> G
    W --> G
    G --> CG{check_groundedness}
    CG -- "ungrounded Β· budget left" --> RW
    CG -- "grounded / exhausted" --> X((end))

Every node run appends a corrective:<node> step (with the iteration ordinal) to RagAnswer.Trace; verdicts accumulate in RagTrace.Verdicts, rewritten probes in RagTrace.QueryVariants, the loop count in RagTrace.Iterations.

Node What it does
retrieve Embeds the current probe (original question, or the latest rewrite) and searches CandidateK candidates β€” hybrid BM25 + RRF per the preset β€” keeping TopN
evaluate IRetrievalEvaluator grades the chunks against the original question (the probe may have been rewritten, the information need has not) β†’ RetrievalVerdict
rewrite_query One constrained LLM call (temperature 0) rewrites the retrieval probe across the vocabulary gap, fed with the evaluator's rationale and any unsupported claims; loops back to retrieve. An unusable rewrite retries the previous probe but still consumes the budget β€” never an exception
refine Decompose-then-recompose on Ambiguous: filters by the evaluator's per-chunk relevances (threshold 0.5), re-splits chunks into sentences and keeps the segments sharing vocabulary with the question β€” never empties the working set
web_fallback Opt-in last resort after rewriting is exhausted (see below); skipped and traced otherwise
generate Grounded generation answering the original user question β€” never the rewritten probe β€” with the same rank-based [n] markers, token budget and anti-Lost-in-the-Middle edges layout as the staged pipeline
check_groundedness IGroundednessChecker verifies the answer against the context; ungrounded β†’ re-loop through rewrite_query; without a registered checker the node is traced as skipped and the graph ends

The two decision points are conditional edges: after evaluate β†’ [generate | rewrite_query | refine | web_fallback], after check_groundedness β†’ [rewrite_query | end].

Verdicts (Correct | Incorrect | Ambiguous)

  • Correct β†’ straight to generate.
  • Ambiguous β†’ refine, then generate.
  • Incorrect β†’ rewrite_query while the iteration budget lasts; at exhaustion the opt-in web fallback fires (only when enabled and a retriever is registered and it has not been attempted yet), else best-effort generation with the best available chunks β€” every skip is traced with its reason (disabled, no IWebDocumentRetriever registered, already attempted).

Default evaluator and checker (registered by AddOrkeonCorrectiveRag, itself called by AddOrkeonRag): LLM-backed when an IChatClient is registered β€” LlmRetrievalEvaluator and LlmGroundednessChecker, constrained via LlmResponseFormat (json_object where the provider wires it, e.g. DeepSeek; tolerant JSON parsing elsewhere) β€” otherwise the deterministic HeuristicRetrievalEvaluator / HeuristicGroundednessChecker with a warning.

Double bound β€” the loop can never run away

  1. Corrective.MaxIterations (default 3): the routing never re-enters rewrite_query past the budget β€” this bounds both the Incorrect rewrite cycle and the groundedness re-loop.
  2. Graph circuit breaker: the StateGraph engine's own CircuitBreakerPolicy, explicitly derived from the budget (MaxTransitions = (n+2)Γ—7, MaxStateVisits = n+2, per-state timeout 2 min, total 10 min) β€” a second, independent layer that only trips if the routing invariants are ever violated. A tripped breaker is caught, traced (corrective:circuit_breaker) and degraded to a best-effort answer (generation runs outside the graph if it had not run yet). The pipeline never throws for a loop condition.

Groundedness is native to the graph

The corrective preset deliberately keeps Groundedness.Enabled = false: that flag toggles the staged pipeline's optional stage 7, whereas the graph runs its own check_groundedness node whenever an IGroundednessChecker is registered. Honouring the flag here would be dead configuration at best and a double check if the preset ever fed a linear pipeline.

Web fallback β€” opt-in, injection-validated

Two deliberately separate configuration sections; both Enabled switches are off by default and both must be on for a web document to ever reach the graph:

Section Type Concern
Orkeon:Rag:Corrective:WebFallback (Enabled, MaxResults) RagWebFallbackOptions (Abstractions) Pipeline-side policy: may the graph route to web_fallback, and for how many documents
Orkeon:Rag:WebFallback (Enabled, Endpoint, ApiKeyEnvVar, MaxResults, Timeout, SuspiciousAction) WebSearchRetrieverOptions (Orkeon.Rag.WebFallback) HTTP transport: SearxNG-compatible JSON search + page download + suspicious-content policy

The split is architectural, not accidental: the Abstractions shared kernel is Domain+BCL-only (ADR-006), and web egress is its own explicit opt-in. AddOrkeonRagWebFallback(configuration) registers the IWebDocumentRetriever (WebSearchDocumentRetriever) β€” enabled-but-unconfigured stays inert with a warning, and the API key only ever comes from the ApiKeyEnvVar environment variable. Every downloaded page passes PromptInjectionDocumentValidator: Rejected content never leaves the retriever, Suspicious content is flagged in metadata or discarded per SuspiciousAction, and content is never rewritten. Threat model, detected signals and honest limits: security.md. Web chunks carry ScoreOrigin = "web" and open the working set; the locally retrieved chunks β€” just graded Incorrect β€” close it.

Profiles corrective and adaptive

The corrective preset feeds the graph's nodes: hybrid BM25 + RRF retrieval (rewritten probes need the lexical leg to bridge vocabulary gaps) and no linear rerank stage β€” the graph corrects through evaluate β†’ rewrite loops instead of reranking, so the profile needs no opt-in ONNX package. The adaptive profile's Iterative route delegates to the memoized corrective pipeline (the route step traces delegate=corrective).

Measured honesty

The end-to-end mechanism β€” grader verdict Incorrect β†’ LLM rewrite β†’ re-retrieve β†’ notes-power.md cited on the seeded q-007 case where quality misses it β€” is proven by tests/e2e/Orkeon.E2E.Tests/CorrectiveRagMechanismSlowTests.cs (real BGE embeddings, real hybrid store; the LLM is scripted for the two roles CI cannot provide, and labelled as such). In the fully offline eval table corrective scores 0.78 recall@5 / 0.64 MRR β€” below quality: the extractive stub degrades the graph's LLM nodes β€” the grader fishes grade words out of echoed passages, and every rewrite collapses to one degenerate probe shared by all cases. Proof: with Corrective:MaxIterations = 0 (no rewrite possible) the row returns to the linear profiles' 0.89 / 0.89. That artifact is analysed line by line in examples/rag/eval/README.md β€” published as measured, not smoothed over.

Architecture decisions

The RAG subsystem's decision record is ADR-006 (shared-kernel status of Orkeon.Rag.Abstractions, allowed couplings, legacy namespaces removed without shims). The phase decisions layered on top of it: the double ONNX package (Orkeon.Rag.Onnx runtime + Orkeon.Rag.Onnx.Model embedded int8 weights β€” guaranteed offline), profile presets + per-key overrides with fast as the safe default, query transformers with per-kind fusion semantics, CRAG built on the Domain StateGraph rather than a bespoke loop, and the strictly opt-in, injection-validated web fallback with its policy/transport section split (this page and ADR-006).

V1 limitations

  • The in-process BM25 index is per-process and unpersisted; native hybrid (LanceDB) is preferred at scale β€” automatic when the provider supports it.
  • MMR uses lexical similarity between candidates (embeddings are not re-computed at the fuse stage).
  • balanced / quality / adaptive require the opt-in ONNX packages; without them use Rerank:Kind = llm or stay on fast or corrective.
  • Without a real IChatClient the corrective graph's LLM nodes (grader, rewrite, groundedness) degrade β€” offline evaluation measures the loop's guard rails, not rewrite quality (honest analysis in examples/rag/eval/README.md and limitations).
  • The web fallback's anti-injection heuristics are pattern-based and evadable (security.md); the fallback is opt-in and not exercised against a real network in CI.