π«π· Version franΓ§aise
See also: ADR-006 β RAG subsystem Β· Memory system Β· Opt-in subsystems Β· Back to index
RAG pipeline
Retrieval-Augmented Generation subsystem (src/rag/): incremental ingestion with
security validation, a staged query pipeline (transform β retrieve β fuse β rerank β
assemble β generate β groundedness) whose every stage is traced, hybrid BM25 + vector
retrieval, pluggable rerankers, a corrective CRAG graph built on the in-house Graph
orchestration mode, quality profiles (fast / balanced / quality / adaptive /
corrective), YAML crew integration with cited knowledge injection, and an offline
evaluation harness gated in CI.
Overview
An agent that pastes whole documents into its prompt drowns its context and cannot cite anything. The RAG subsystem precomputes and structures that knowledge:
- Ingestion: documents are loaded (files, CSV, HTML, PDF, web pages), validated against prompt injection, chunked, embedded, and upserted into a collection of the document store. A per-collection manifest makes re-ingestion incremental β an unchanged source costs zero embeddings.
- Query: a question flows through a fixed sequence of togglable stages and comes
back as a
RagAnswerβ generated text with[n]citation markers, the citations resolving each marker to its source chunk (score and offsets included), and the full execution trace of the run. - Quality dials: one knob (
Orkeon:Rag:Profile) selects a preset; every value can then be overridden key by key through configuration. Profile = preset, configuration = override.
Three consumption surfaces share the same pipelines: agent tools (rag_search,
rag_ingest, rag_eval in Orkeon.Tools.Rag), the crew YAML rag: /
knowledge: blocks, and the scripting DSL rag.* namespace.
Projects (ADR-006)
| Project | Role | Depends on |
|---|---|---|
Orkeon.Rag.Abstractions |
Contracts, DTOs, options (IRagPipeline, RagAnswer, RagOptions, RagProfilePresetsβ¦) |
Orkeon.Domain only |
Orkeon.Rag |
Implementations: loaders, chunkers, validation, stores, retrieval, reranking, routing, evaluation, DI | Application, Analysis.Abstractions (see ADR-006) |
Orkeon.Rag.Onnx |
Opt-in ONNX cross-encoder reranker (ms-marco-MiniLM-L-6-v2), AddOrkeonOnnxReranker() |
Orkeon.Rag (concrete) |
Orkeon.Rag.Onnx.Model |
Companion package embedding the int8 model weights β guaranteed offline | β |
Orkeon.Tools.Rag |
Agent tools rag_search / rag_ingest / rag_eval, AddOrkeonRagTools() |
Orkeon.Rag.Abstractions, Orkeon.Rag (concrete β the ADR-006-guarded coupling), Tools.Abstractions, Analysis.Abstractions |
Orkeon.Rag.Abstractions is a secondary shared kernel (same status as
Orkeon.Analysis.Abstractions, ADR-003):
Orkeon.Application references it for the ports; Orkeon.Infrastructure references
Orkeon.Rag solely as DI composition wiring.
Quick start
The subsystem is opt-in β nothing is wired by AddOrkeonInfrastructure() alone. Every
runner host built on RunnerHost (orkeon run in all its forms, orkeon-host) makes
the two calls below itself, and each shipped binary β orkeon, orkeon-host, the
orkeon-repl REPL β adds the ONNX reranker; a host that builds its own container makes them:
services.AddOrkeonLocalEmbeddings(); // or any IEmbeddingProvider (local BGE: no API key)
services.AddOrkeonInfrastructure(); // semantic-first defaults for embeddings + chat
services.AddOrkeonRag(configuration); // the subsystem (Orkeon.Rag.DependencyInjection)
services.AddOrkeonRagTools(); // optional: rag_search / rag_ingest / rag_eval tools
var ingestion = provider.GetRequiredService<IIngestionPipeline>();
await ingestion.IngestAsync(new IngestionRequest
{
Collection = "product-kb",
Sources = [new SourceDescriptor { Location = "/kb/faq.md" }],
});
var rag = provider.GetRequiredService<IRagPipeline>();
var answer = await rag.QueryAsync(new RagQuery
{
Text = "How many days do customers have to request a refund?",
Collection = "product-kb",
TopN = 3,
});
// answer.Text β generated text with [n] markers
// answer.Citations β marker -> chunk id, source id, snippet, score, offsets
// answer.Trace β stages executed, variants, route, verdicts
The host must provide an IEmbeddingProvider and an IChatClient;
AddOrkeonInfrastructure() registers semantic-first defaults for both (resolution
order for embeddings: container-registered local/Analysis provider β
Orkeon:Embeddings configuration β fail-fast at first use, never silently).
Which model the subsystem asks: the host's default profile, unless
Orkeon:Rag:LlmProfile names another of its profiles (Llm:Profiles:<name>,
through ILlmProfileRegistry). Then grounded generation, the query transformers,
the listwise reranker, the corrective graph's evaluator and groundedness checker,
the llm classifier and the evaluation judge all call that profile β one profile
for the whole subsystem, whatever RAG profile (Orkeon:Rag:Profile) a query runs
on. The profile's chat client is resolved at the first RAG call that needs a model,
never when a crew merely loads; RagLlm.EnsureProfileIsKnown, which the runner
host and the REPL call once the container is built, checks the name alone and
refuses one the host does not offer β orkeon-host's Orkeon:Host:LlmProfiles
allow-list included β listing the known ones. orkeon eval rag --offline ignores
the key: the extractive stub answers in place of every model.
Runnable, fully offline demos: rag/basic-ingestion,
rag/hybrid-retrieval,
rag/custom-reranker,
rag/crew-yaml, and the scripting variant
scripting/08-rag.ork.ts.
Registration extensions
AddOrkeonRag(configuration) is the one call a host needs; it composes the
narrower extensions below, all TryAdd* (a host registration always wins) and
safe to call on their own:
| Extension | Registers |
|---|---|
AddOrkeonRag(configuration) |
The whole subsystem: options, loaders, validation, document store + hybrid decorator, manifests, both pipelines, profile resolver, and every extension below except the ONNX reranker and AddOrkeonEphemeralSearch |
AddOrkeonQueryTransforms() |
QueryTransformerFactory with none / multi-query / rag-fusion / hyde |
AddOrkeonRagReranking() |
RerankerFactory with none / llm / decision plus every IRerankerRegistrar |
AddOrkeonQueryRouting(configuration) |
The IQueryComplexityClassifier named by Orkeon:Rag:QueryRouting:Classifier |
AddOrkeonHybridRetrieval(configuration, β¦) |
The HybridSearchDocumentStore decorator (BM25 + RRF) around the IDocumentStore |
AddOrkeonCorrectiveRag(configuration) |
IRetrievalEvaluator, IGroundednessChecker, CorrectiveRagPipeline (experimental, ORKEXP003) |
AddOrkeonRagWebFallback(configuration) |
The IWebDocumentRetriever, only when Orkeon:Rag:WebFallback is enabled and has an Endpoint |
AddOrkeonKnowledgeAugmentation() |
IKnowledgeContextAugmenter (agent knowledge: attachments) |
AddOrkeonRagEvaluation() |
Dataset loader, IRagEvaluator, report writer, IRagEvalHarness |
AddOrkeonEphemeralSearch() |
IEphemeralCollectionSearch behind the txt_search / mdx_search / pdf_search / directory_search tools β fails at call time, not at registration, when AddOrkeonRag is absent |
AddOrkeonOnnxReranker(configure?) |
Orkeon.Rag.Onnx: the onnx / cross-encoder reranker |
AddOrkeonRagTools() |
Orkeon.Tools.Rag: rag_search, rag_ingest, rag_eval |
So a host that calls AddOrkeonRag never calls the six narrower ones
(AddOrkeonQueryTransforms, AddOrkeonRagReranking, AddOrkeonQueryRouting,
AddOrkeonHybridRetrieval, AddOrkeonKnowledgeAugmentation,
AddOrkeonRagEvaluation) itself β they are idempotent, so a second call is
harmless. Call them alone only to compose a partial stack by hand, e.g. the
evaluation harness over a host-built IRagProfileResolver, or a factory without
the rest of the subsystem. Two ordering rules: AddOrkeonHybridRetrieval
decorates the last registered IDocumentStore and throws when none is
registered yet (it also binds Orkeon:Rag:Retrieval:Hybrid, flat = true
shorthand included); AddOrkeonKnowledgeAugmentation needs an
IRagProfileResolver at resolution (AddOrkeonRag registers it) β without the
augmenter, agent knowledge: attachments are not injected, and the crew factory
warns about each agent that declares some. The genuinely opt-in extensions, which AddOrkeonRag does not
call, are AddOrkeonOnnxReranker, AddOrkeonEphemeralSearch and
AddOrkeonRagTools.
Contracts
All in Orkeon.Rag.Abstractions (Interfaces/, Models/, Options/):
| Contract | Role |
|---|---|
IRagPipeline |
Query faΓ§ade: RagQuery β RagAnswer (citations + trace) |
IRagRetrievalCapable |
Opt-in: the retrieval half alone (transform β retrieve β fuse β rerank β assemble), no generation, no LLM call |
IIngestionPipeline |
Ingestion faΓ§ade: IngestionRequest β IngestionReport |
IDocumentStore |
Upsert / search / delete-by-source over a named collection |
IDocumentLoader |
Source β RagDocument (text, CSV, HTML, PDF, web page) |
IChunkingStrategy |
Document β Chunk slices (recursive, sentence, structural, semantic) |
IQueryTransformer |
Query β retrieval texts + a QueryTransformKind (see below) |
IReranker |
Candidates β best topN, reranker-scored (cascade CandidateK β TopN) |
IQueryComplexityClassifier |
Query β QueryRoute (NoRetrieval / SingleShot / Iterative) |
IRagProfileResolver |
Profile name β memoized pipeline instance |
IKnowledgeContextAugmenter |
Agent knowledge attachments β cited prompt block |
IRagCollectionsBootstrapper |
Crew rag: block β kickoff ingestion |
IRetrievalEvaluator, IGroundednessChecker |
Corrective hooks (verdicts, hallucination check β see Corrective RAG); experimental, ORKEXP003 |
IRagEvaluator, IRagEvalHarness* |
Offline evaluation (metrics, golden datasets) |
IEphemeralCollectionSearch |
Throwaway in-memory collections (AddOrkeonEphemeralSearch) |
IDocumentStoreCollectionProbe |
Optional store capability (HasContentAsync): lets ingestion notice a manifest that outlived its data β implemented by MemoryProviderDocumentStore and forwarded by the hybrid decorator |
* the harness contract and its runner live in Orkeon.Rag.Evaluation; the
evaluator port is in the abstractions.
Named components (chunkers, transformers, rerankers) resolve through factories
following the MemoryProviderFactory pattern: names and aliases matched
case-insensitively, and an unknown name always fails loudly with the list of
known names β never a silent fallback.
Staged query pipeline
StagedRagPipeline (default IRagPipeline) is a fixed sequence of togglable
stages driven by RagOptions:
flowchart LR
Q[RagQuery] --> T[transform]
T --> R[retrieve]
R --> F[fuse]
F --> RR[rerank]
RR --> A[assemble]
A --> G[generate]
G --> GR[groundedness]
GR --> ANS[RagAnswer<br/>text + citations + trace]
| # | Stage | What it does | Trace data (excerpt) |
|---|---|---|---|
| 1 | transform |
Named transformer produces retrieval texts (none skips) |
mode, transformer, kind, variants, variant_1β¦variant_n |
| 2 | retrieve |
CandidateK candidates per retrieval text; hybrid honoured per query by capable stores |
candidates, top_k, mode, hybrid, variants |
| 3 | fuse |
Per-kind combination of the per-text rankings + dedup by chunk id, then opt-in MMR | method (rrf/union/dedup), in, out, mmr* |
| 4 | rerank |
Named reranker, cascade CandidateK β TopN (disabled: truncation in retrieval order) | reranker, candidates, kept |
| 5 | assemble |
Token budget (β 4 chars/token) + anti-Lost-in-the-Middle edges ordering |
chunks, ordering, dropped_by_budget |
| 6 | generate |
Grounded generation with [n] markers (anti-hallucination system prompt) |
model |
| 7 | groundedness |
Optional post-generation verification hook (IGroundednessChecker) |
grounded, score |
Every stage appends a RagTraceStep to RagAnswer.Trace β durations, counts, and
chosen component names are always observable. When retrieval yields no candidate,
generation is skipped and a deterministic no-context answer is returned (the
pipeline never lets the model answer ungrounded).
Query transformers by kind
Transformers declare how their output texts combine at the fuse stage
(QueryTransformKind):
| Kind | Built-in | Semantics |
|---|---|---|
Union |
multi-query |
Retrieve once per text (original + variants); merge by chunk-id union, original store scores kept β the maximum wins on duplicates |
Fusion |
rag-fusion |
Retrieve once per text; fuse the rankings by Reciprocal Rank Fusion (same RrfK as the hybrid stage) |
Replacement |
hyde |
The substitute text(s) are the retrieval probes instead of the question β generation and citations always use the original question |
Built-ins registered by AddOrkeonRag: none, multi-query, rag-fusion,
hyde. The LLM-backed ones resolve the subsystem's chat client (the default
profile's, or the one Orkeon:Rag:LlmProfile names) lazily; parsing of
the LLM's variants is tolerant, and an empty output degrades to the original
query.
Anti-Lost-in-the-Middle assembly
With Context.Ordering: edges (the default), ranked chunks r1 (best) β¦ rm are
laid out as r1, r3, r5, β¦ from the head and β¦, r6, r4, r2 closing the tail β
the two strongest chunks sit at the extremities where LLM attention is highest,
the weakest in the middle. Citation markers stay rank-based ([1] = best
chunk) whatever the layout. linear restores plain rank order.
Hybrid retrieval (BM25 + RRF)
AddOrkeonRag always wraps the registered IDocumentStore β including a
host-registered one β in the HybridSearchDocumentStore decorator, so ingestion
feeds an in-process per-collection BM25 index alongside the vector store. Whether
a search actually fuses is decided per query (RetrievalQuery.Hybrid, set by
the profile presets), with Orkeon:Rag:Retrieval:Hybrid:Enabled as the default
mode; a disabled default is a strict behavioural passthrough.
Score provenance (ScoredChunk.ScoreOrigin):
hybrid-nativeβ the backing memory provider implementsIHybridSearchCapable(e.g. LanceDB): text + vector fused provider-side.rrfβ emulated path: inner vector ranking + in-process BM25, fused with Reciprocal Rank Fusion. RRF scores are rank aggregates, not similarities.- Inner origin (
vector/local-cosine) β passthrough when the lexical side has nothing to contribute.
The in-process BM25 index only covers chunks upserted through the decorator in the current process; at scale, prefer a provider with native hybrid search β the decorator switches to it automatically.
MMR diversification
Opt-in (Orkeon:Rag:Retrieval:Mmr:Enabled), applied at the fuse stage:
Maximal Marginal Relevance re-orders the fused candidates by trading relevance
against redundancy β each pick maximises
λ·relevance β (1 β Ξ»)Β·max-similarity-to-already-picked (default Ξ» = 0.7,
relevance-leaning). Candidate embeddings are not re-computed: the lexical
(Jaccard) similarity fallback is the honest default.
Reranking
The rerank stage narrows CandidateK candidates (default 50) to TopN
(default 5) with a named IReranker:
| Name (aliases) | Implementation | Notes |
|---|---|---|
none (noop) |
NoopReranker |
Truncation in retrieval order |
llm (listwise) |
LlmListwiseReranker |
One listwise chat call; tolerant parsing (a malformed answer keeps the retrieval order); the built-in alternative when the ONNX package is absent β select it explicitly |
onnx (cross-encoder) |
Orkeon.Rag.Onnx |
ms-marco-MiniLM-L-6-v2 cross-encoder, int8 weights embedded, fully offline; AddOrkeonOnnxReranker() |
decision |
DecisionModelReranker |
One request to the host's decision model: the query and every passage as the state, one yes-or-no question per passage, the score being the probability of a yes; a passage is cut at 4000 characters. Needs an active decision provider (AddOrkeonDecisionProvider), otherwise resolving it throws InvalidOperationException |
Hosts contribute rerankers through IRerankerRegistrar β registered in DI,
applied when the singleton RerankerFactory is built, registration order
irrelevant (this is how the ONNX package plugs in; see
examples/rag/custom-reranker for a host-provided one). The factory itself is
TryAdd-registered: a host-registered RerankerFactory wins outright.
Reranked chunks carry their scorer in ScoredChunk.ScoreOrigin: cross-encoder
(sigmoid-mapped logits in (0, 1)), llm-rerank (normalized descending rank,
best = 1.0) or decision-rerank (the probability of a yes).
The ONNX reranker resolves its model lazily at first use: the companion package
Orkeon.Rag.Onnx.Model when referenced (embedded weights, guaranteed offline),
otherwise OnnxRerankerOptions.ModelPath + VocabPath (VFS virtual paths, both
required), otherwise a loud error β the runtime never downloads weights
(tools/download-reranker-model.sh in the package provisions them, SHA-256
verified). AddOrkeonOnnxReranker(o => β¦) also sets BatchSize (default 16) and
MaxSequenceLength (default 512); these options are C#-only, bound on no
configuration section.
Profiles
RagProfilePresets expands a profile name into a complete RagOptions; the
Orkeon:Rag configuration section then overrides any value individually.
fast (default) |
balanced |
quality |
adaptive |
corrective |
|
|---|---|---|---|---|---|
| Retrieval | vector only, CandidateK = TopK |
hybrid BM25 + RRF, CandidateK = 50 |
hybrid, CandidateK = 100 |
routed (see below) | hybrid, graph retrieve node (see CRAG) |
| Rerank | off | ONNX cross-encoder 50 β 5 | ONNX cross-encoder | routed | none β the graph corrects by looping, not reranking |
| Groundedness | off | off | on | routed | native check_groundedness graph node |
| Needs | nothing | Orkeon.Rag.Onnx |
Orkeon.Rag.Onnx |
Orkeon.Rag.Onnx (its SingleShot route runs balanced) + classifier + chat client |
chat client (grader/rewrite; heuristic fallbacks without one) |
fast is the out-of-the-box default because balanced requires the opt-in ONNX
package β defaulting to it would make every bare AddOrkeonRag() host refuse to
start, naming onnx (the effective options are judged at the host's start).
Opt into balanced with one configuration line.
IRagProfileResolver (ProfileRagPipelineResolver) memoizes one pipeline per
profile name; the reserved name default resolves to the host's registered
IRagPipeline, so a host-provided pipeline keeps the last word for non-profiled
queries. Unknown names fail loudly with the list of known profiles.
Adaptive routing (Adaptive-RAG)
The adaptive profile resolves to an AdaptiveRagPipeline: an
IQueryComplexityClassifier routes each query first, and the decision is always
traced (RagTrace.Route + a route step carrying classifier, route, delegate).
flowchart LR
Q[query] --> C{classifier}
C -- NoRetrieval --> D[direct LLM answer<br/>no citations]
C -- SingleShot --> B[balanced pipeline]
C -- Iterative --> CO[corrective graph pipeline]
Classifiers (Orkeon:Rag:QueryRouting:Classifier): heuristic (default β
deterministic rules, zero LLM call) or llm (one constrained lightweight chat
call; when selected without an IChatClient the heuristic is the documented
fallback, with a warning).
The Iterative route delegates to the memoized corrective
graph pipeline (see Corrective RAG); the route trace step carries
delegate=corrective and RagTrace.Route records the decision.
Configuration reference (Orkeon:Rag)
Bound over the selected preset β every key is an individual override.
| Key | Default | Meaning |
|---|---|---|
Orkeon:Rag:Profile |
fast |
Preset: fast / balanced / quality / adaptive / corrective (unknown fails loudly) |
Orkeon:Rag:Collection |
β | The collection rag_search queries when the agent names none (unset: default), and the one rag_eval evaluates when neither the call nor the dataset names one and the dataset brings no corpus. rag.query/rag.ingest and knowledge: always name theirs |
Orkeon:Rag:LlmProfile |
default profile | The host LLM profile (Llm:Profiles:<name>) every model call of the subsystem goes to β generation, transformers, listwise reranker, corrective evaluator and groundedness checker, llm classifier, evaluation judge; an unknown name refuses the host start, listing the known ones |
Orkeon:Rag:Provider |
ambient | Document-store provider type (inmemory/in-memory, redis, sqlite, chromadb/chroma, pinecone, lancedb/lance; unknown fails loudly); unset = ambient IMemoryProvider. The connection is that provider's own host section (Orkeon:Redis, Orkeon:Sqlite, Orkeon:ChromaDb, Orkeon:Pinecone, Orkeon:LanceDb), and the store shares the factory's instance of that type with crew memory |
Orkeon:Rag:Retrieval:TopK |
5 | Chunks kept for context assembly (call-site RagQuery.TopN wins) |
Orkeon:Rag:Retrieval:CandidateK |
5 (fast) |
Wide stage of the cascade (always β₯ final TopN); fast has none and takes TopK, a profile that reranks takes 50, quality 100 |
Orkeon:Rag:Retrieval:MinScore |
none | Optional raw-score floor; unset, no floor applies |
Orkeon:Rag:Retrieval:Hybrid:Enabled |
false (fast) |
Default hybrid mode; flat shorthand Retrieval:Hybrid = true accepted |
Orkeon:Rag:Retrieval:Hybrid:RrfK |
60 | Reciprocal Rank Fusion constant |
Orkeon:Rag:Retrieval:Mmr:Enabled |
false | MMR diversification at the fuse stage |
Orkeon:Rag:Retrieval:Mmr:Lambda |
0.7 | Relevance/diversity trade-off in [0, 1] |
Orkeon:Rag:QueryTransform:Mode |
none |
none / multi-query / rag-fusion / hyde |
Orkeon:Rag:QueryTransform:VariantCount |
3 | Variants requested from non-none transformers |
Orkeon:Rag:Rerank:Enabled |
false (fast) |
Whether the rerank stage runs |
Orkeon:Rag:Rerank:Kind |
none |
none / llm / onnx / decision / any registered name |
Orkeon:Rag:Rerank:TopN |
5 | Chunks kept after reranking |
Orkeon:Rag:Context:MaxTokens |
2000 | Context budget (β 4 chars/token; excess chunks dropped, never overrun) |
Orkeon:Rag:Context:Ordering |
edges |
edges (anti-Lost-in-the-Middle) or linear |
Orkeon:Rag:Groundedness:Enabled |
false | Post-generation verification (traced as skipped without a checker) |
Orkeon:Rag:Generation:SystemPrompt |
built-in | Grounded system prompt override |
Orkeon:Rag:Generation:Temperature / MaxOutputTokens |
β | Sampling passed to the chat client |
Orkeon:Rag:QueryRouting:Classifier |
heuristic |
heuristic or llm (adaptive profile) |
Orkeon:Rag:Corrective:MaxIterations |
3 | Corrective iteration budget β query rewrites, whether triggered by an Incorrect verdict or an ungrounded answer |
Orkeon:Rag:Corrective:WebFallback:Enabled |
false | Pipeline-side policy: may the corrective graph route to its web_fallback node |
Orkeon:Rag:Corrective:WebFallback:MaxResults |
3 | Web documents the graph asks the retriever for |
Orkeon:Rag:WebFallback:* |
disabled | Separate section β transport of the web retriever (Enabled, Endpoint, ApiKeyEnvVar, MaxResults, Timeout, SuspiciousAction); see Web fallback |
Orkeon:Rag:Ingestion:DefaultChunkingStrategy |
recursive |
Strategy when a request names none |
Orkeon:Rag:Ingestion:ManifestDirectory |
/output/rag/manifests |
VFS directory of the per-collection manifests |
Ingestion
DefaultIngestionPipeline runs: load (DocumentLoaderFactory selects by source
kind/extension β text, CSV, HTML, PDF, WebPageLoader for URLs) β security
validation β chunk (named strategy) β embed β upsert.
| Loader | Claims | SourceDescriptor.Kind hint |
|---|---|---|
TextFileLoader |
.txt, .md, .markdown, .text, .log |
file (or none) |
CsvDocumentLoader |
.csv |
file (or none) |
HtmlDocumentLoader |
.html, .htm |
file (or none) |
PdfDocumentLoader |
.pdf |
file (or none) |
WebPageLoader |
http:// / https:// URLs |
url (or none) |
InlineTextLoader |
inline content (registered by AddOrkeonEphemeralSearch) |
text |
File loaders read through the VFS (virtual paths only). WebPageLoader fails
closed without an IUrlValidator (AddOrkeonInfrastructure() registers one):
every URL is SSRF-validated before the fetch, and its HTTP client follows no
redirect, so an approved URL cannot be bounced to an internal address.
| Chunking strategy (aliases) | Behaviour |
|---|---|
recursive (recursive_text, default) |
Recursive separator split β the default |
sentence (sentences) |
Sentence-bounded chunks |
structural (markdown, headings) |
Heading-aware split |
semantic |
Closes a chunk when the next sentence drifts from the chunk's centroid (threshold 0.75, similarity_threshold extension); without an embedding function it degrades to structural β and AddOrkeonRag wires none |
ChunkingOptions (per IngestionRequest) sizes chunks in characters:
MaxChunkSize 1000, Overlap 200 by default.
- Validation (ingestion-path security):
PromptInjectionDocumentValidatorandContentIntegrityValidatorrun on every document; rejected content goes to theIQuarantineStoreand provenance is recorded (IProvenanceTracker). Details and threat model in security.md. - Incremental manifest (per collection, one JSON file written through the
VFS at
{ManifestDirectory}/{collection}.json): each source's content hash and the embedding profile (provider, model, dimensions) are recorded. Unchanged sources are skipped entirely βchunksEmbedded = 0; changed ones are purged and re-ingested. Failure posture: a missing/corrupt manifest degrades to full ingestion, an unwritable manifest directory degrades to a warning (ingestion works, just never incrementally) β never a crash. - Stale manifest detection: the manifest (VFS) and the default in-memory
store have different lifetimes β a second process would find every source
"unchanged" against an empty store. When the store implements
IDocumentStoreCollectionProbeand reports the collection empty while the manifest lists sources, the manifest is discarded and the run ingests in full. Reindex = truepurges every recorded source and rebuilds the collection. It is also the only way to re-ingest after an embedding-profile change: a drift between the manifest's recorded profile and the active provider fails the run loudly rather than silently mixing incompatible vectors.prune(rag_ingestoption,rag.ingest({ prune: true })): deletes the chunks and the manifest entries of manifest sources that are absent from the ingested source set, and reports them asSourcesPruned(<n> prunedin the tool output,sourcesPrunedin the script payload). Pruning is skipped for a run that recorded any error, so a failed load never causes data loss.- Scope and concurrency:
- Store persistence follows
Orkeon:Rag:Provider: the default in-memory store is lost at process end;sqlite,redis,chromadb,pineconeandlancedbpersist. - The document store is host-global: one store per host, shared by every workspace and crew. Chunks ingested through one workspace are retrievable from another.
- The ingestion manifest is per workspace (
/output/rag/manifests/{collection}.jsonthrough each workspace's VFS). A second workspace sharing a persistent store sees an empty or different manifest and re-ingests. - Ingests of one collection are serialized by an in-process per-collection lock (from the manifest read to its write); different collections run concurrently. The collection-registry update of the store is atomic per provider.
- The lock is in-process only: a cross-process lock needs provider support and is out of scope, so two host processes sharing a store can still race.
- Store persistence follows
- Glob sources (
*,**,?) are expanded through the VFS bySourceGlobExpanderat every consuming surface (scriptingrag.ingest,rag_ingesttool, CLIorkeon rag ingest, eval harness corpus, and the crewrag:block). Anhttp:///https://address is never a pattern: the?of its query string or a*stay part of it, and it reachesWebPageLoaderas written, without the VFS being asked (SourceGlobExpander.IsWebAddress, one rule for every surface). The crew block also takes a directory, which stands for every file below it, and resolves a relative path against the crew's folder β all of it decided inRagCollectionsBootstrapper, which warns, naming the collection, about a pattern or a directory that yields no file.
Crew YAML integration
Two YAML blocks, consumed at different moments (see examples/rag/crew-yaml):
rag: # crew-level: collections ingested at crew creation
collections:
product-kb:
sources: [/kb/faq.md]
chunking: { strategy: recursive, max_tokens: 256, overlap: 32 }
agents:
support:
role: Support agent
goal: Answer from the knowledge base
knowledge: [product-kb] # agent-level: retrieval attachment
- The
rag:block is parsed intoRagCrewConfig;CrewFactoryhands it to theIRagCollectionsBootstrapperatCreateFromConfigAsyncβ kickoff-time ingestion through the incremental pipeline (a fresh manifest makes it a no-op). A crew declaringrag:on a host withoutAddOrkeonRaglogs a warning instead of failing; so does an agent that declaresknowledge:there. knowledge:attachments travel on theAgentaggregate; at task-context assembly the execution orchestrator callsIKnowledgeContextAugmenter. For each attachment it resolves the pipeline of the attachment's profile and runs its retrieval half only (IRagRetrievalCapable.RetrieveAsyncβ transform, hybrid, rerank and assembly as the profile says, no generation, no nested LLM call), then injects a bounded, numbered block with[n]citations into the agent's prompt.
Where it runs. Every host built on RunnerHost registers the subsystem:
orkeon run on a YAML crew, a crew directory or a .ork.ts script (with or
without --events), and orkeon-host; so does orkeon-repl, and the C# host
of examples/rag/crew-yaml
calls AddOrkeonRag itself. orkeon run --validate loads the crew without
ingesting anything. Nothing of the subsystem is built until a crew uses it: a
crew without rag:/knowledge: loads no embedding model for it. Its settings are
judged at the host's start, though, whether a crew uses them or not: an
unknown Orkeon:Rag:Provider, profile, reranker, query transformer, context
ordering, classifier or chunking strategy refuses the start, with the list of known
names (when a setting is refused). Ingestion
needs an embedding provider: the default runner has the on-device one (unless
RaggableTree:Enabled is false and nothing else is configured). The
incremental manifests land under /output/rag/manifests; without a writable
/output mount the collections are ingested on every load, with a warning.
YAML keys:
| Key | Effect |
|---|---|
rag.collections.<name>.sources |
Ingested at crew creation: files, globs, directories (every file below) and URLs; a path without a leading / is relative to the crew's folder (/crew under orkeon run); a pattern or a directory that yields nothing is a load warning |
rag.collections.<name>.chunking (strategy, max_tokens, overlap) |
Tokens converted to characters at Γ4 |
rag.defaults.profile |
The retrieval profile of every attachment that names none (copied onto it at crew creation) |
knowledge: short form ([product-kb]) |
Attaches the collection with the defaults below |
knowledge: long form (collection, top_k default 5, min_score, max_context_tokens default 2000) |
top_k is the query's TopN; min_score filters on the score the profile reports (vector similarity under fast, the cross-encoder's score under a reranking profile); max_context_tokens caps the attachment's excerpts (Γ4 characters) |
knowledge: long form profile |
The retrieval pipeline: the attachment's profile, else rag.defaults.profile, else Orkeon:Rag:Profile (fast when unset). fast, balanced and quality retrieve alone; corrective and adaptive cannot (their graphs generate as they go) and fail the task with a message saying so; an unknown name fails as the resolver does. balanced/quality also need the ONNX reranker, which orkeon, orkeon-host and orkeon-repl register |
The document store is the host's choice (Orkeon:Rag:Provider), not the crew's:
a crew that writes rag.provider is refused at load ('rag.provider' is retired: the document store is the host's choice β set Orkeon:Rag:Provider in the settings).
Scripting and CLI surfaces
- Scripting DSL (
.ork.ts): first-classrag.ingest({ collection, sources, chunkingStrategy?, reindex?, prune? }),rag.query(question, { collection, profile?, topN? })andrag.retrieve(question, { β¦ })globals β seeexamples/scripting/08-rag.ork.ts. The namespace is always registered; calls fail with an actionable message when the host did not wire the subsystem.queryandretrievereturn the same payload shape and onlyqueryruns the generation stage, so a script that reads onlycitationsshould callretrieveβ see below. - CLI (
orkeon):orkeon rag ingest|search|evalover the same pipelines β flags in the CLI reference. A collection whose manifest records another embedding provider, model or dimensions is refused (exit 1, naming the collection, both models andorkeon rag ingest --reindex) onrag ingestwithout--reindex, onrag search, oneval rag --no-ingest -c <collection>and oneval rag -c <collection>when its ingest pre-pass runs without--reindex. A collection with no manifest, andeval ragwithout-c, are not checked. - Agent tools (
Orkeon.Tools.Rag,AddOrkeonRagTools(), requiresAddOrkeonRag):
| Tool | Parameters | Behaviour |
|---|---|---|
rag_search |
question (required), top_k (default 3), collection (default Orkeon:Rag:Collection, else default) |
IRagPipeline.QueryAsync β answer + Sources: block with scores; collection = "raggable-tree" routes to the RaggableTree code index when one is registered |
rag_ingest |
collection, sources (paths / globs / http(s) addresses) β both required; chunking_strategy, reindex, prune |
IIngestionPipeline with glob expansion; prints the ingestion report |
rag_eval |
dataset (required), collection, profile (default default), compare, k (default 5), use_llm_judge, reindex |
The offline harness below |
A crew agent that lists one of the three tools (tools: [rag_search]) receives it
wherever the host registered them: every RunnerHost host (orkeon run on a YAML crew, a
crew directory or a .ork.ts script, and orkeon-host), the REPL, and a C# host that calls
AddOrkeonRagTools(). Building a tool resolves nothing: the pipelines are resolved at its
first call.
Retrieval without generation
IRagRetrievalCapable.RetrieveAsync runs stages 1-5 and stops. Same query, same
passages, no LLM call.
A caller that quotes the retrieved passages does not need a generated answer:
retrieve skips the generation call and its tokens.
if (pipeline is IRagRetrievalCapable retriever)
{
var answer = await retriever.RetrieveAsync(
new RagQuery { Text = question, Collection = "kb", TopN = 6 });
// answer.Text is empty; answer.Citations holds the assembled passages.
}
Two properties worth knowing before relying on it:
- The capability is opt-in and must be probed.
StagedRagPipelineimplements it; the corrective graph does not, because it interleaves retrieval evaluation with generation β "retrieval only" is not a prefix of its run.rag.retrievethrows on a pipeline that cannot honour it instead of falling back toQueryAsync, which would charge exactly what the caller asked to avoid. - The trace still carries a
generatestep, whose detail says the stage was skipped on purpose. An absent step would read as a trace produced by an older pipeline.
Evaluation (golden datasets, CI gate)
The offline harness (IRagEvalHarness) turns retrieval quality into a
published number instead of a claim:
- Dataset (
examples/rag/eval/golden.yaml): a corpus directory plus cases βquestion,relevantsource refs (suffix-matched against ingested ids),expected_substrings,reference_answer,tags. The corpus is ingested (incrementally) before every run. - Metrics: recall@k, precision@k and MRR per case and aggregated (deterministic); generation (groundedness, answer-relevance) is judged by an LLM judge when available, with a deterministic heuristic fallback β the report always labels which judge ran.
- Profiles compared in one run:
orkeon eval rag --dataset β¦ --compare fast,balanced,quality,corrective,adaptive --offline(--offlineswaps generation for a deterministic extractive stub β zero network; the measured table and its honest reading live inexamples/rag/eval/README.md). - CI gate (
.github/workflows/rag-eval.yml): the comparison table is published to the step summary, and an anti-regression gate fails the build when thebalancedprofile's aggregate recall@5 or MRR drops below the floors (--min-recall 0.80 --min-mrr 0.70, the workflow'senv:block). Cases taggedcorrectifare excluded from the gates: they are seeded to fail single-shot retrieval on purpose (see below). - Schema: published as
interface RagEvalDataset/RagEvalCasein the scripting typings (src/scripting/Orkeon.Scripting/Typings/rag.d.ts). A case withoutquestionis refused with an error namingcase #<index> ('<id>')and the missing field. - Hosts: the
ragscript namespace (rag.retrieve,rag.ingest, β¦) is bound in the three hosts βorkeon run, the REPL and the daemon (RunnerExecution) β through oneRagScriptingBackendFactory.Build(IServiceProvider)that resolvesIIngestionPipelinefrom DI, so the per-collection ingest lock is shared withrag_ingest.
Corrective RAG (CRAG)
The corrective profile resolves to CorrectiveRagPipeline
(Orkeon.Rag.Corrective) β CRAG built on Orkeon's Graph orchestration
mode: the pipeline's execution is a StateGraph<RagGraphState> β the same
Domain StateGraph<TState> engine behind ProcessType.Graph β with conditional
edges, controlled cycles and the graph's native circuit breaker. The corrective
engine is not a bespoke loop bolted onto RAG; RAG demonstrates the Graph
orchestration mode and vice versa. Same IRagPipeline faΓ§ade as every other
profile: the profile selects the executor, callers only ever see a RagAnswer.
The corrective types (CorrectiveRagPipeline, IRetrievalEvaluator,
IGroundednessChecker, IWebDocumentRetriever, AddOrkeonCorrectiveRagβ¦) are
marked experimental β diagnostic ORKEXP003, see
experimental APIs.
Graph topology
flowchart LR
S((start)) --> R[retrieve]
R --> E{evaluate}
E -- Correct --> G[generate]
E -- Ambiguous --> RF[refine]
E -- "Incorrect Β· budget left" --> RW[rewrite_query]
RW --> R
E -- "Incorrect Β· budget exhausted, opt-in" --> W[web_fallback]
RF --> G
W --> G
G --> CG{check_groundedness}
CG -- "ungrounded Β· budget left" --> RW
CG -- "grounded / exhausted" --> X((end))
Every node run appends a corrective:<node> step (with the iteration ordinal)
to RagAnswer.Trace; verdicts accumulate in RagTrace.Verdicts, rewritten
probes in RagTrace.QueryVariants, the loop count in RagTrace.Iterations.
| Node | What it does |
|---|---|
retrieve |
Embeds the current probe (original question, or the latest rewrite) and searches CandidateK candidates β hybrid BM25 + RRF per the preset β keeping TopN |
evaluate |
IRetrievalEvaluator grades the chunks against the original question (the probe may have been rewritten, the information need has not) β RetrievalVerdict |
rewrite_query |
One constrained LLM call (temperature 0) rewrites the retrieval probe across the vocabulary gap, fed with the evaluator's rationale and any unsupported claims; loops back to retrieve. An unusable rewrite retries the previous probe but still consumes the budget β never an exception |
refine |
Decompose-then-recompose on Ambiguous: filters by the evaluator's per-chunk relevances (threshold 0.5), re-splits chunks into sentences and keeps the segments sharing vocabulary with the question β never empties the working set |
web_fallback |
Opt-in last resort after rewriting is exhausted (see below); skipped and traced otherwise |
generate |
Grounded generation answering the original user question β never the rewritten probe β with the same rank-based [n] markers, token budget and anti-Lost-in-the-Middle edges layout as the staged pipeline |
check_groundedness |
IGroundednessChecker verifies the answer against the context; ungrounded β re-loop through rewrite_query; without a registered checker the node is traced as skipped and the graph ends |
The two decision points are conditional edges: after evaluate β
[generate | rewrite_query | refine | web_fallback], after
check_groundedness β [rewrite_query | end].
Verdicts (Correct | Incorrect | Ambiguous)
Correctβ straight togenerate.Ambiguousβrefine, thengenerate.Incorrectβrewrite_querywhile the iteration budget lasts; at exhaustion the opt-in web fallback fires (only when enabled and a retriever is registered and it has not been attempted yet), else best-effort generation with the best available chunks β every skip is traced with its reason (disabled,no IWebDocumentRetriever registered,already attempted).
Default evaluator and checker (registered by AddOrkeonCorrectiveRag, itself
called by AddOrkeonRag): LLM-backed when an IChatClient is registered β
LlmRetrievalEvaluator and LlmGroundednessChecker, constrained via
LlmResponseFormat (json_object where the provider wires it, e.g. DeepSeek;
tolerant JSON parsing elsewhere) β otherwise the deterministic
HeuristicRetrievalEvaluator / HeuristicGroundednessChecker with a warning.
Double bound β the loop can never run away
Corrective.MaxIterations(default 3): the routing never re-entersrewrite_querypast the budget β this bounds both theIncorrectrewrite cycle and the groundedness re-loop.- Graph circuit breaker: the
StateGraphengine's ownCircuitBreakerPolicy, explicitly derived from the budget (MaxTransitions = (n+2)Γ7,MaxStateVisits = n+2, per-state timeout 2 min, total 10 min) β a second, independent layer that only trips if the routing invariants are ever violated. A tripped breaker is caught, traced (corrective:circuit_breaker) and degraded to a best-effort answer (generation runs outside the graph if it had not run yet). The pipeline never throws for a loop condition.
Groundedness is native to the graph
The corrective preset deliberately keeps Groundedness.Enabled = false: that
flag toggles the staged pipeline's optional stage 7, whereas the graph runs
its own check_groundedness node whenever an IGroundednessChecker is
registered. Honouring the flag here would be dead configuration at best and a
double check if the preset ever fed a linear pipeline.
Web fallback β opt-in, injection-validated
Two deliberately separate configuration sections; both Enabled switches
are off by default and both must be on for a web document to ever reach the
graph:
| Section | Type | Concern |
|---|---|---|
Orkeon:Rag:Corrective:WebFallback (Enabled, MaxResults) |
RagWebFallbackOptions (Abstractions) |
Pipeline-side policy: may the graph route to web_fallback, and for how many documents |
Orkeon:Rag:WebFallback (Enabled, Endpoint, ApiKeyEnvVar, MaxResults, Timeout, SuspiciousAction) |
WebSearchRetrieverOptions (Orkeon.Rag.WebFallback) |
HTTP transport: SearxNG-compatible JSON search + page download + suspicious-content policy |
The split is architectural, not accidental: the Abstractions shared kernel is
Domain+BCL-only (ADR-006), and web egress is its own explicit opt-in.
AddOrkeonRagWebFallback(configuration) registers the IWebDocumentRetriever
(WebSearchDocumentRetriever) β enabled-but-unconfigured stays inert with a
warning, and the API key only ever comes from the ApiKeyEnvVar environment
variable. Every downloaded page passes PromptInjectionDocumentValidator:
Rejected content never leaves the retriever, Suspicious content is flagged
in metadata or discarded per SuspiciousAction, and content is never
rewritten. Threat model, detected signals and honest limits:
security.md. Web chunks
carry ScoreOrigin = "web" and open the working set; the locally retrieved
chunks β just graded Incorrect β close it.
Profiles corrective and adaptive
The corrective preset feeds the graph's nodes: hybrid BM25 + RRF retrieval
(rewritten probes need the lexical leg to bridge vocabulary gaps) and no
linear rerank stage β the graph corrects through evaluate β rewrite loops
instead of reranking, so the profile needs no opt-in ONNX package. The
adaptive profile's Iterative route delegates to the memoized
corrective pipeline (the route step traces delegate=corrective).
Measured honesty
The end-to-end mechanism β grader verdict Incorrect β LLM rewrite β
re-retrieve β notes-power.md cited on the seeded q-007 case where quality
misses it β is proven by
tests/e2e/Orkeon.E2E.Tests/CorrectiveRagMechanismSlowTests.cs (real BGE
embeddings, real hybrid store; the LLM is scripted for the two roles CI cannot
provide, and labelled as such). In the fully offline eval table corrective
scores 0.78 recall@5 / 0.64 MRR β below quality: the extractive stub
degrades the graph's LLM nodes β the grader fishes grade words out of echoed
passages, and every rewrite collapses to one degenerate probe shared by all
cases. Proof: with Corrective:MaxIterations = 0 (no rewrite possible) the row
returns to the linear profiles' 0.89 / 0.89. That artifact is analysed line by
line in examples/rag/eval/README.md β published as
measured, not smoothed over.
Architecture decisions
The RAG subsystem's decision record is
ADR-006 (shared-kernel status of
Orkeon.Rag.Abstractions, allowed couplings, legacy namespaces removed without
shims). The phase decisions layered on top of it: the double ONNX package
(Orkeon.Rag.Onnx runtime + Orkeon.Rag.Onnx.Model embedded int8 weights β
guaranteed offline), profile presets + per-key overrides with fast as
the safe default, query transformers with per-kind fusion semantics,
CRAG built on the Domain StateGraph rather than a bespoke loop,
and the strictly opt-in, injection-validated web fallback with its
policy/transport section split (this page and ADR-006).
V1 limitations
- The in-process BM25 index is per-process and unpersisted; native hybrid (LanceDB) is preferred at scale β automatic when the provider supports it.
- MMR uses lexical similarity between candidates (embeddings are not re-computed at the fuse stage).
balanced/quality/adaptiverequire the opt-in ONNX packages; without them useRerank:Kind = llmor stay onfastorcorrective.- Without a real
IChatClientthe corrective graph's LLM nodes (grader, rewrite, groundedness) degrade β offline evaluation measures the loop's guard rails, not rewrite quality (honest analysis inexamples/rag/eval/README.mdand limitations). - The web fallback's anti-injection heuristics are pattern-based and evadable (security.md); the fallback is opt-in and not exercised against a real network in CI.