π«π· Version franΓ§aise
LLM Providers
The providers on this page serve the Generation category. Embedding, reranking and decision providers (TypeSafe, Liquid, Kev) are listed in Model providers; a decision model judges an LLM call through a decision profile (Decision models).
HttpLlmProviderBase (Orkeon.Infrastructure.LLMs.Base) provides the abstract base for all LLM providers. It integrates HTTP handling (IHttpClientFactory), the Polly retry policies, and JSON serialization.
Retry budget. Llm:MaxRetries (10 by default) covers the failures that come back in seconds β request errors, 5xx, 429 (except an exhausted balance, below). A buffered call that hits Llm:TimeoutSeconds is retried once (ResilienceDefaults.LlmTimeoutRetries: every attempt costs the whole timeout), then the provider answers with a failure naming the setting and the two ways out (a longer timeout, thinking off); a caller's cancellation is never retried. A failed call travels as LlmResponse.Error and, through the chat-client adapter, as an exception β it is never mistaken for an empty answer, so the agent loop fails the task at once with the provider's reason instead of retrying without tools.
Rate-limit window. A 429, or a 529 that carries a reset header, is not retried on the backoff ladder: the call waits until the window resets, on the buffered path and on the streaming connect alike (RateLimitResetHeaders). It reads Retry-After (seconds or an HTTP date), retry-after-ms (milliseconds, sent by OpenAI and TypeSafe), anthropic-ratelimit-requests-reset and anthropic-ratelimit-tokens-reset (RFC 3339) and x-ratelimit-reset-requests and x-ratelimit-reset-tokens (OpenAI durations such as 6m0s). retry-after-ms, when sent, is the more precise of the two Retry-After headers and replaces it; the longest of that value and the reset headers then wins. A finite value beyond the window fails the call with LlmRateLimitException, never with an arithmetic overflow. The wait never exceeds Llm:MaxRateLimitWait (5 minutes by default, separate from the 30-second cap of the 5xx ladder): a reset beyond it fails the call at once with the reason (LlmRateLimitException) instead of parking an unattended run. Each wait is announced to ILlmRetryObserver with the reason rate-limit window and the delay. A 529 without a header keeps the ladder.
Exhausted balance. A 429 whose body names a spent balance or quota β OpenAI's insufficient_quota, Kimi's exceeded_current_quota_error, Z.AI's Insufficient balance β is not a rate limit: no wait clears it. It is not retried, neither on the ladder nor on a window, on the buffered path and on the streaming connect alike (ResiliencePolicies, LlmFailureClassifier.IsBalanceExhausted), and the error line adds (account balance exhausted: ...) after the vendor's words. It still classes as rate_limited, so a profile's fallback chain moves on to the next provider, and the decision model reports it as not transient. When the body cannot be read, the 429 is treated as a rate limit. Out of scope: a balance refused with another status (Anthropic's 400 "credit balance is too low", a 402 from DeepSeek, MiniMax or OpenRouter) is not retried either, but classes as refused and does not fail over; the code-graph embedding provider (Orkeon.Analysis) has its own retry loop and still retries insufficient_quota (issue #328).
A streamed call is bounded the same way. A streamed request is sent with ResponseHeadersRead, and HttpClient.Timeout stops counting once the headers are in, so every streaming path (the OpenAI dialect, Azure, Anthropic, Ollama β chat and text streams) runs under a StreamReadBudget: Llm:TimeoutSeconds bounds the whole call, headers and body, and the optional Llm:StreamIdleSeconds bounds the silence between two lines of the body (unset by default). An elapsed bound ends the chat stream with a Completed event whose response carries error (the sentence naming the setting) and error_type (TaskCanceledException for the whole-call bound, StreamIdleTimeout for the idle one) over whatever content had arrived, with no RawResponseBody; the text stream throws an HttpRequestException with the same sentence. A stream the provider closes without its finish marker (finish_reason or [DONE], message_stop, done: true) is a failed call too when no answer text and no tool call arrived (error_type StreamTruncated, the sentence saying how much reasoning was streamed); with some answer it is served as it came, flagged stream_truncated in its metadata. The chat-client adapter fails a streamed turn on any error, partial content or not; the headers phase re-sends an elapsed HttpClient.Timeout once (LlmTimeoutRetries), like the buffered path. Every provider records the vendor's own stop reason, raw, under finish_reason in the response metadata (length for the OpenAI dialect and Ollama's done_reason, max_tokens for Anthropic's stop_reason), buffered or streamed; Azure's string-only stream aside. A script reads it to tell an exhausted output budget from an empty answer.
Thinking is replayed unchanged (Anthropic). The assistant turn of a tool call carries its thinking and redacted_thinking blocks (text, signature, data) as the provider sent them, buffered or streamed, in LlmMessage.RawThinkingBlocks; the next request re-emits them first, before the turn's text and tool_use blocks, which the API requires for extended thinking with tools. Other providers ignore them. A tool call without id, name or an object arguments is refused with an error that names its index, its name and the field.
Implemented providers:
| Provider | Class | Namespace |
|---|---|---|
| OpenAI | OpenAILlmProvider (via OpenAICompatibleProviderBase) |
Orkeon.Infrastructure.LLMs |
| Anthropic | AnthropicLlmProvider |
Orkeon.Infrastructure.LLMs |
| Azure OpenAI | AzureOpenAILlmProvider |
Orkeon.Infrastructure.LLMs |
| Ollama | OllamaLlmProvider |
Orkeon.Infrastructure.LLMs |
| Together AI | TogetherAiLlmProvider |
Orkeon.Infrastructure.LLMs |
| DeepSeek | DeepSeekLlmProvider |
Orkeon.Infrastructure.LLMs |
| HuggingFace | HuggingFaceLlmProvider |
Orkeon.Infrastructure.LLMs |
| Kimi | KimiLlmProvider |
Orkeon.Infrastructure.LLMs |
| Qwen | QwenLlmProvider |
Orkeon.Infrastructure.LLMs |
| Mistral AI | MistralLlmProvider |
Orkeon.Infrastructure.LLMs |
| Z.AI (Zhipu GLM) | ZaiLlmProvider |
Orkeon.Infrastructure.LLMs |
| Google Gemini | GeminiLlmProvider |
Orkeon.Infrastructure.LLMs |
| Grok (x.AI) | GrokLlmProvider |
Orkeon.Infrastructure.LLMs |
| MiniMax | MiniMaxLlmProvider |
Orkeon.Infrastructure.LLMs |
| OpenRouter (aggregator) | OpenRouterLlmProvider |
Orkeon.Infrastructure.LLMs |
| Mammouth AI (aggregator) | MammouthLlmProvider |
Orkeon.Infrastructure.LLMs |
Fourteen of them extend OpenAICompatibleProviderBase (itself an HttpLlmProviderBase);
Anthropic and Ollama extend HttpLlmProviderBase directly, because their APIs are not
OpenAI-shaped. Docker Model Runner and any other OpenAI-compatible server have no class of
their own: they are driven by OpenAILlmProvider through their base URL. Endpoints, default
models and provider keys are shared constants in Orkeon.Constants.Llm
(LlmProviderEndpoints, LlmProviderDefaultModels, LlmProviderKeys,
LlmModelOutputLimits).
Generic adapters (ChatClientToLlmProviderAdapter, LlmProviderToChatClientAdapter, ChatClientToBasicLlmProviderAdapter) are available in Orkeon.Infrastructure.LLMs.Adapters to integrate other providers compatible with the IChatClient interface. A Microsoft Agent Framework agent can also serve as a model (AIAgentLlmProvider, ADR-010).
Provider resolution
LlmProviderFactory (Orkeon.Infrastructure.LLMs) has two entry points:
Create(providerType, config)takes an explicit provider key (LlmProviderKeys):openai,anthropic,ollama,azure-openai/azure,together/togetherai,qwen,deepseek,kimi/moonshot,mistral,huggingface/hf,gemini/google,grok/xai,minimax,zai/glm/zhipu,openrouter,mammouth, andcustom/docker-model-runner, which build the OpenAI dialect on the configured base URL. An unknown key throwsNotSupportedException. This is the path oforkeon llm probe --provider.Create(config)infers the provider β this is what a run does, since theLlmsection of the settings file carries no provider key. The first rule that matches wins:With a base URL, the address alone decides (
LlmEndpointReading, the one reading the run and the CLI share; the model and the key are not consulted):Address Provider a known vendor host β azure/.cognitiveservices.,together.xyz, the DashScope hosts,deepseek.com,moonshot.cn/moonshot.ai,mistral.ai,generativelanguage.googleapis.com,api.x.ai,api.minimax.io/api.minimaxi.com,huggingface.co/hf.co,api.z.ai/bigmodel.cn,openrouter.ai,mammouth.ai,openai.com,anthropic.comthat vendor /engines/in the path,model-runner.docker.internalor port12434Docker Model Runner (OpenAI dialect) port 11434Ollama, at its native root (a trailing /v1or/apiis dropped)anthropicin the host or in the pathAnthropic any other address OpenAI-compatible server ( custom), at the path writtena vendor host under an /anthropicpath, or an address that is not an absolutehttp(s)URLrefused at start, with the setting's path and the fix A server is Ollama only on port
11434, wherever it runs. Acustomaddress with no path (http://host:8000) is read as an OpenAI-compatible server answering at its root and is reported once per section (LlmEndpointNotice.Silent, a warning on stderr and in the log): add the path of its routes, often/v1.Model prefix (only without an address) β
gpt,claude,llama/codellama(Ollama),mistral/ministral(the baremistralandmistral:<tag>go to Ollama),qwen,deepseek,moonshot,glm,gemini,grok,minimax,openrouter/. Mammouth is never inferred from a model name: its identifiers are the vendors' own.API key prefix β
hf_β HuggingFace,xai-β Grok.Otherwise OpenAI.
A section that names no model skips rule 2. The model a call runs on is then resolved by
the provider, the same way for all sixteen (HttpLlmProviderBase.ResolveModel): the call's own
when its configuration names one, else the model the provider is configured with β its
profile's β, else its DefaultModel (its LlmProviderDefaultModels entry; Azure has none of
its own and falls back to OpenAI's as a deployment name, so name yours). A configuration that
names no model (LlmConfig.OnProfile(), an empty Model) never reaches the wire empty, and
never carries another vendor's model.
The configuration a call runs on follows the same rule, field by field
(LlmConfig.InheritFrom, applied by HttpLlmProviderBase.EffectiveConfig in the sixteen
providers): a configuration passed with a call completes the one the provider was built with,
it does not replace it. Every field the call leaves unset β null, an empty or blank string, no
stop sequence β is the provider's: the key, BaseUrl, TimeoutSeconds (nullable: unset
everywhere, 30 s), Azure's ApiVersion, Anthropic's WorkspaceId, Thinking, MaxTokens,
Temperature and TopP (nullable too), Seed, SystemMessage, ResponseFormat,
Cache, Tools, the grammar and the model; custom parameters merge, the call's keys winning.
Every field the call sets wins. The settings that cannot be unset β the penalties (0, every
vendor's default) and the tool mode β are the caller's, and MaxRetries and Grammar are read
from the provider's configuration when it is built. A call cannot unset what its provider sets;
it overrides it (Thinking = { Enabled = false }, LlmResponseFormat.Text()). The chat client
adapter registered without a base configuration starts from the provider's own
(ILlmProvider.BaseConfig).
Sampling: what is set is sent, what is not set is not. Temperature and TopP
are null when nothing sets them β not the call, not the agent, not the task, not the profile β
and then nothing reaches the wire: the model applies its own default (often 1; the Modelfile's
on Ollama). Set, they are sent whatever their value. A default model may refuse a sampling value it does not
support (a pinned 0.7 on gpt-5.6-sol or claude-sonnet-5), which is why nothing is sent unless
something sets it. One measured exception: Mistral's dialect writes top_p: 1 when nothing sets
one (AlwaysEmitTopP). The penalties, the seed and the stop sequences of an agent's configuration
travel through the chat client too; on the wire, Ollama writes top_p, seed and stop in its
options, and a dialect that has no field for a penalty or a seed says so with the structured
warning below, never in silence. A temperature, top_p or penalty is sent as written, up to six
significant digits (the float of ChatOptions holds no more): 0.7, never 0.699999988079071,
under any current culture.
Every provider the factory builds is wrapped in CallModeLlmProvider, then, by
LlmProviderEntrance, in MeteredLlmProvider (always; it reports to the host's ILlmUsageSink
when one is registered), TracedLlmProvider and, when the host registered an ILlmRateLimiter,
RateLimitedLlmProvider (see Decorators and registration), and
returned behind an LlmProviderAdapter.
Declared capabilities
Every provider declares a LlmProviderCapabilities value object (Domain, exposed on
ILlmProvider; a provider overrides HttpLlmProviderBase.DeclaredCapabilities, which defaults
to LlmProviderCapabilities.Unknown): ResponseFormat
(None/JsonObject/JsonSchema), Thinking (None/EffortOnly/Toggle/Budget), EffortLevels
(the effort levels the field knows: low, medium, high by default, up to xhigh for OpenAI and Grok,
and the five for Z.ai, Anthropic, OpenRouter, DeepSeek, Kimi, Qwen, Together AI, Hugging Face,
MiniMax and Mammouth AI; the per-model table below narrows them model by model): a level asked that is not declared is sent as the
highest declared level below it, with a warning in the provider's log; a value outside the five, such as OpenAI's
none, is no level and is sent unchanged for the vendor to judge), Vision, ExplicitPromptCaching,
RequiresJsonKeywordInPrompt, ReplaysReasoningContent, RunsOwnTools (declared by the Microsoft Agent Framework
bridge alone: it is an agent that runs its own tools).
One capability is not the vendor's to declare: GbnfGrammar, which no vendor API documents, is
switched on by the configuration the provider is built with (Llm:Grammar,
LlmConfig.GrammarEnabled) for a llama.cpp-compatible server behind the OpenAI-compatible
providers or Ollama β ILlmProvider.Capabilities is the declaration plus that switch.
And one is no vendor's at all: ReplaysPrompt, declared by the echo provider alone
(UndefinedLlmProvider, the model of a host without an Llm section), says that what it returns
is its prompt, never a model's answer β the crew's planner then skips its call with a warning
instead of reading the prompt back as a plan. The decorators a provider is wrapped in
(MeteredLlmProvider, RateLimitedLlmProvider) pass the capabilities through.
OpenAICompatibleProviderBase translates the declaration into the OpenAI dialect once
(vision payloads, response_format, thinking, the CapabilityMismatchHint diagnostics);
Anthropic and Ollama write their own dialects, and Qwen overrides the hook for DashScope's
thinking fields. An option a provider cannot honour produces a structured warning (event id
110, Option 'β¦' was declared but β¦ does not support it β it was not sent) β never a
silent drop; that includes frequency_penalty and presence_penalty on every
dialect, and seed wherever the dialect does not write it (all but Ollama). CapabilityMismatchHint (OpenAI-compatible providers and Ollama) covers the
per-model side: when a vendor refuses a
capability the provider declares (a text-only model sent an image, a model without thinking
or tools), the error says whose assumption was wrong. The mirror-image case β
a constraint only the server can state β has its own seam:
OpenAICompatibleProviderBase.TryAdaptRejectedPayload gives a provider one chance to adapt
a payload the API rejected with a 4xx and re-send it once (generate and chat paths;
streaming never retries). Kimi uses it for Moonshot's invalid temperature: only 1 is allowed for this model β the mandated value is read from the rejection itself (which
models mandate it is decided server-side, a hard-coded list would drift) and the
substitution is logged as a structured warning; only a temperature that was set and sent is
replaced, since one nothing sets is not sent. Two more dialect seams serve the
aggregators: ReasoningFieldName names the vendor field the reasoning trace is
read from (reasoning_content by default, reasoning on OpenRouter β the Orkeon metadata
key stays reasoning_content), and usage.cost becomes the cost metadata wherever a
vendor bills in the response β buffered or streamed β with cost_currency beside it when the
provider states its vendor's billing currency (CostCurrency: USD on OpenRouter, whose
credits are dollars; no other provider states one). From there the charge travels as billed:
the chat client adapter carries it onto the ChatResponse (AdditionalProperties, also on
its streaming path), the agent loop reports it on the usage event
(CostUsageEvent.Cost, null when the vendor billed nothing β a free model's 0 stays
0), the scripting facade does the same for ctx.llm.*, and the run's cost.updated
relays it with costSource: "vendor" (the run event bus).
CostBudgetManager prices from its registry only a call nobody priced, for its own
budgets; that estimate never reaches the wire. A chunk carrying a root-level error after
the HTTP 200 ends a stream the way a pre-stream refusal does β error metadata on the chat stream, an
HttpRequestException on the token stream β never as a clean completion. All 16 providers
are IStreamingLlmProviders. The chat client adapter's streaming path
(GetStreamingResponseAsync) is built like its buffered one β the same messages,
roles, tools and options β and reads the provider's chat stream (ChatStreamingAsync): the
text as it arrives, then one last update carrying the tool calls (native or of the text
protocol), the provider's usage, the finish reason, the model, the vendor's cost and the
reasoning to replay. Folded, the updates are the response the buffered call returns, so a
streamed agent turn calls its tools and is metered exactly, never estimated; a provider that
does not stream answers it buffered. A tool call is replayed to the provider with every
property it carried, buffered or streamed, each call with its own: the adapter keeps the
provider's tool_calls[i] element on its FunctionCallContent and replays it whole, and the
merge of streamed fragments keeps what a fragment adds beyond index, id, type and
function β Gemini's thought signature (extra_content.google.thought_signature), which
Gemini requires back, travels that way; a call that carried nothing else is replayed
as id, type and function. A stream says what the buffered answer says: Anthropic assembles the tool_use blocks it streams; MiniMax splits its
leading <think> block out of the content deltas (LeadingReasoningTag), so the text a stream
carries is the answer its final response keeps; and on the native protocol, the OpenAI dialect
and Anthropic give a streamed answer the body their buffered answer carries, in which the
text-protocol fallback reads a call the model wrote as text. The per-provider matrix lives in
the provider comparison.
Tool-call deltas. The chat stream tells its caller about a tool call while the model builds it: LlmStreamEventKind.ToolCallDelta carries ToolCallDelta(Index, Id, Name, ArgumentsDelta), and the terminal event still holds the whole call, unchanged. The OpenAI dialect (the fourteen providers that derive from OpenAICompatibleProviderBase, Azure OpenAI included) emits one delta per delta.tool_calls fragment; Anthropic one when a tool_use block opens (id and name) and one per input_json_delta; Ollama one complete delta per call, since it sends calls whole. None of the 16 providers is mute: a test checks that the three families cover them, so a provider added later must join one of them or say that it cannot (a structured warning, never a silent drop). The chat client adapter passes the fragments on as updates that carry no content, so the fold of the turn is the one it was; ILlmDeltaSink.OnToolCallDelta receives them (empty by default) and they ride the llm.delta event (deltaKind: "tool_call"). Reasoning deltas are not forwarded to the sinks. With a DLP interceptor of the external-output channel registered (AddOrkeonDlp), the arguments of each call are held by index until the stream ends, scanned whole and emitted as one delta, so a secret cut between two fragments is redacted; without one they stream as they arrive. A fallback chain commits to a provider at its first event, a tool-call delta included: the partial deltas of a provider that failed before any event never reach the caller, and one that fails after reaching it keeps the call, its failure ending the stream.
Dialect seams of the OpenAI-compatible base
What differs between the compatible vendors is expressed through a handful of protected members, not through copies of the payload builder:
| Seam | Default | Overridden by |
|---|---|---|
ApplyProviderSpecificOptions |
judges the thinking options and writes response_format from the declared capabilities |
Together AI (adds context_length_exceeded_behavior: truncate); Kimi (removes temperature and top_p on the kimi-k3 family, with a structured warning) |
WriteReasoningFields |
writes the judged thinking controls (switch, effort, budget); the default writes thinking: {type} and reasoning_effort |
Qwen (enable_thinking, thinking_budget), OpenRouter (the reasoning request object), Together AI (reasoning: {enabled}), Hugging Face (chat_template_kwargs.enable_thinking), MiniMax (thinking: disabled only) |
MaxTokensFieldName |
max_tokens |
OpenAI (max_completion_tokens) |
AlwaysEmitTopP |
top_p written only when set |
Mistral (top_p: 1 when nothing sets one) |
SplitReasoningFromContent / EnrichAssistantMessage |
nothing split; reasoning_content replayed when ReplaysReasoningContent is declared |
MiniMax (inline <think> block split out, re-inlined on replay) |
ReasoningFieldName |
reasoning_content |
OpenRouter (reasoning) |
CostCurrency |
none | OpenRouter (USD) |
TryAdaptRejectedPayload |
re-sends once without the output cap when the catalogue's cap was refused | Kimi (the mandated temperature, then the base rule) |
BuildEndpoint |
{baseUrl}/chat/completions |
Azure OpenAI (dated deployment URL, or the v1 surface with api_version: v1) |
Prompt-cache tokens and TTL
LlmResponse carries three cache counts: CacheHitTokens (read from the cache), CacheWriteTokens
(written to it β Anthropic's cache_creation_input_tokens) and CacheMissTokens (input not served from
the cache). PromptTokens already includes the written and read tokens: a write is never counted twice.
MeteredLlmProvider records them once per call in the CostUsageEvent; CostBudgetManager keeps per model
the prompt, completion, cache-write, cache-read and reasoning tokens and the cost (CostReport.ModelUsageByModel), and
session_stats returns them in by_model_tokens[] beside the unchanged by_model cost map. A cache or reasoning
count that no call reported is null in ModelUsage, never 0 (a call that reported none adds nothing to the others:
null + 5 = 5), and its key is absent from the by_model_tokens[] entry; 0 means the provider measured nothing. A model whose
ModelPricing declares CacheWritePricePerMillion / CacheReadPricePerMillion is priced as
(prompt - write - read) x prompt rate + write x write rate + read x read rate + completion x completion rate;
without those rates the cost is unchanged. Every built-in Anthropic entry declares them (#257): a write at
1.25x the prompt rate, a read at 0.1x unless the vendor states another figure (Opus 5.5 and Sonnet 5.5 read at
0.20 $/M, Fable 5.1 at 0.25 $/M). The write rate is the 5-minute TTL's, 1.25x; the 1-hour share of a write is costed at 2x the
prompt rate: Anthropic reports it in usage.cache_creation.ephemeral_1h_input_tokens, which AnthropicLlmProvider reads
(buffered and streamed) into LlmResponse.CacheWrite1hTokens, and MeteredLlmProvider hands to the
CostUsageEvent as CacheWrite1hTokens, a subset of CacheWriteTokens. claude-opus-4-5 (and its dated ids) is priced at the
official 5 / 25 $/M, not at the claude-opus-4 prefix's. Haiku 5.5 is priced
at its rate for a prompt up to 100K tokens; a longer prompt bills more, which the registry does not model.
A ModelPricing row may name a Provider (null: any provider; CustomPricings:<i>:Provider in the settings).
CostBudgetManager prices a call with IModelPricingRegistry.TryGetPricing(provider, model) /
CalculateCost(provider, model, β¦): a custom row beats a built-in one, and within each tier the provider's own rows
come first (see configuration).
The cache TTL is LlmCacheConfig.Ttl (5m or 1h). A call can carry it as the option
LlmChatOptionsKeys.CacheTtl (orkeon:cache_ttl), which ChatOptionsComposer fills from the agent's
llm.cache.ttl; the chat client adapter folds it into the call's Cache.Ttl and Anthropic emits it in
cache_control. A call TTL with no cache configuration that places a breakpoint is itself the request: the
adapter marks the system prompt with that lifetime (LlmCacheConfig.SystemPrompt()), since a lifetime on
nothing would cache nothing. The OpenAI-compatible providers and Ollama cannot honour it and log the structured unhonoured-option
warning (event 110, cache_ttl) instead of dropping it silently.
The request is opt-in, from YAML (llm.cache: { system, tools, ttl }) or from a script:
agentBuilder().cache({ systemPrompt, tools, ttl }) and the cache key of LlmConfig.with({...}) map onto the
same LlmCacheConfig (LlmCacheConfig.Request). The TTL is 5m or 1h, anything else is refused in both, and a
script's ttl without systemPrompt or tools is refused as caching nothing. An object that marks nothing
(cache: {}, cache({})) is an explicit Off (LlmCacheConfig.Off): it turns an inherited cache off, the host
default included, where a cache left out inherits it.
A host sets a default for every agent with Llm:Cache (SystemPrompt, Tools, Ttl), and a profile with
Llm:Profiles:<name>:Cache. It is validated as the YAML block is (LlmCacheConfig.Request): a Ttl other than
5m or 1h, or one with neither SystemPrompt nor Tools, refuses the load and names the key. It applies only
to a provider whose cache is explicit (LlmProviderCapabilities.ExplicitPromptCaching, Anthropic): elsewhere the
provider's BaseConfig drops it, so it neither reaches the call nor warns. An agent's or task's own cache, an
explicit Off included, always wins. The procedural ctx.llm calls of a script agent run with the cache and the
effort its builder set (.cache(...), .effort(...), or the LlmConfig given to .llm(...)), a call's own
options winning.
Profile fallback
A profile that fails over names its replacements: Llm:Profiles:<name>:Fallback: [b, c] (a list of profile names, or one
comma-separated value; default names the host's own profile). A call that fails on <name> in a way a retry cannot cure goes
to b, then to c; a profile registered in code takes the same list with
services.AddOrkeonLlmProfile(name, provider, baseConfig, ["b", "c"]).
| Failure | Class (reason) |
Moves on |
|---|---|---|
| connection refused, reset, DNS, a stream closed before any answer | transport |
yes |
| a 5xx | server_error |
yes |
a 429, once Llm:MaxRetries is spent, or a rate-limit window longer than MaxRateLimitWait |
rate_limited |
yes |
the HTTP timeout, a 408, a stream idle past StreamIdleSeconds |
timeout |
yes |
| every profile of a nested chain failed | exhausted |
yes |
| any other 4xx (400, 401, 403, 422...), a blocked content (the decision stages), a failure nothing is known about | β | no: it is the answer |
| the caller's own cancellation, thrown or reported in-band by the provider | β | no: the call is cancelled, nothing is switched or tried |
LlmFailureClassifier is the one classification: ResiliencePolicies takes its list of retryable statuses from it (5xx, 408,
429), so what a retry may cure and what a chain takes over are the same set.
How the chain is built (LlmProfileRegistry): FallbackLlmProvider is the outermost link, above profiles that are already
assembled. Each profile keeps its own decorators (RateLimited β Traced β Metered β CallMode β provider), so the meter and the trace name
the provider that answered, a failed call is traced as the failure it is, and each profile keeps its own call mode. The chat client
of the chain reads tool calls with the format auto-detection (Anthropic or OpenAI shape), not a per-profile parser. The chain's Capabilities
(and the structured-output choice the chat client makes from them) are the first profile's only; a later profile with fewer capabilities is
reported by the warning below, not accounted for in the chain's own.
Each profile uses its own settings. The chat client starts every call from the first profile's configuration and lays the caller's options over
it; the chain gives a later profile the call with every field that still equals the first profile's replaced by that profile's own value (model, key,
endpoint, MaxTokens, Temperature, Thinking, CustomParameters, SystemMessage, StopSequences, GrammarGbnf, timeouts, all of them). A field the
caller set on the call (an explicit MaxTokens, a temperature in the chat options) differs from the first profile's and stays for every profile. A caller
that sets a field to exactly the first profile's value cannot be told from the profile: the later profile's value applies.
- Streaming: a stream moves on only before its first delta (a throw, or a completion that is a failure, with nothing yielded). Once a token reached the caller the call belongs to its profile and a later failure is the caller's. A profile that does not stream answers buffered.
- Budget:
Llm:Profiles:<name>:FallbackBudgetSeconds(absent: none) is the longest the chain spends, from its first call, before it stops trying the next profile. It is checked between two profiles and never cuts a call in flight, so without it a chain costs, at worst, each profile'sTimeoutSecondstimes itsMaxRetries. - Exhausted chain: a buffered call answers with an error response whose
ErrorTypeisFallbackExhaustedand whose text names every profile and its failure; a call that failed by exception throws oneHttpRequestExceptioncarrying the same text, the last exception as its cause. - Validation (when the registry is built, not in
CrewFactory, which sees only the profiles crews name): a name no profile defines, a profile that names itself, and a cycle (a -> b -> a) refuse the host's start. A fallback may be a profile hidden from crews byOrkeon:Host:LlmProfiles. - Lesser fallback: a fallback whose
LlmProviderCapabilitiesare below the first profile's (vision, response format, thinking, GBNF grammar) logs one warning naming what it lacks when the chain is built; an option that needs it is then handled by the provider that receives it, as for any provider. - Event: each switch writes
llm.fallbackon the--eventsstream (the run event bus) and a warning in the log. A host that registers anILlmFallbackObserveris told too. Onlyorkeon run(with--events) registers one:orkeon-host, the REPL and a library host fail over all the same but write nollm.fallback, the log warning being the only trace unless the host registers its own observer. - No stickiness: a chain starts again from its first profile on every call. With a primary that is down, each turn of a tool loop pays the primary's
TimeoutSecondstimes itsMaxRetriesbefore it reaches the fallback;FallbackBudgetSecondsdoes not shorten that wait (it is checked between two profiles), so lower the primary's ownTimeoutSecondsandMaxRetriesfor a profile that has a fallback. - A revoked key blocks the chain: a 401 and a 403 never switch (they are a configuration error, not an outage), so a primary whose key was revoked answers its error to every call and its fallbacks are never tried.
What a model accepts, and audio as input
LlmProviderCapabilities is declared once per provider, so it cannot tell two models of one vendor apart:
gpt-4o takes no reasoning effort, o3-mini takes three, and both ride the same provider. The per-model side
is a table, IModelCapabilityRegistry (ORKEXP005, experimental),
built like ModelPricingRegistry: a model id prefix maps to ModelCapabilities (AudioInput,
AudioOutput, Vision, Thinking, ReasoningEfforts), the longest prefix of the model asked wins, case-insensitive.
A prefix matches at a boundary only: the end of the id or a separator (-, ., :, @, /) right after it, so
glm-5.3 covers glm-5.3-flash and never glm-5.30. An aggregator's vendor/model id (OpenRouter, Together AI,
Hugging Face) is looked up whole first, then by the part after its last /. The built-in table is in code and
covers the OpenAI chat, audio and reasoning families, Gemini, Claude, Grok, Mistral, and of the other vendors the
models the provider campaign measured a restriction on: glm-5.3, kimi-k2.7-code, MiniMax-M3.1-Flash and
qwen3.7-plus. A model with no entry of its own (kimi-k3, kimi-k3.1-pro, the qwen3.8 models, deepseek-flash)
takes the levels its provider declares. The models that do not reason are entered with Thinking: None: gpt-3.5,
gpt-4, gpt-4-turbo, gpt-4o, gpt-4.1, the audio models (gpt-4o-audio, gpt-4o-mini-audio, gpt-audio),
gpt-5-chat, gemini-1.5 and gemini-2.0.
An entry can also name the provider that serves the model, <provider>/<model> (the provider's
ILlmProvider.Name), for a gateway that maps a model's values its own way: the call looks up
<provider>/<model> first, the whole id, then the model's own entry as above. The built-in ones come from
campaign M18 (2026-11-13): mammouth/gemini-3.7, mammouth/gemini-3.8, openrouter/google/gemini-3.7 and
openrouter/google/gemini-3.8 always reason and take minimal to max (Google's own API takes none to high),
and together/zai-org/glm-5.3 is unrestricted, both switches included (Z.AI's own API refuses the off switch and
most efforts). A model it does not know is added, or a built-in entry replaced, from the configuration, under
either form of key (Llm:ModelCapabilities:<model>:... or Llm:ModelCapabilities:<provider>/<model>:...):
Llm:
ModelCapabilities:
acme-voice: # a prefix: acme-voice-2 matches
AudioInput: true
Thinking: EffortOnly # None, EffortOnly (always reasons), Toggle or Budget
ReasoningEfforts: [low, high] # left out: no restriction; written empty: the model takes none
together/zai-org/glm-5.3: # this model as Together AI serves it, read before glm-5.3
Thinking: Toggle
An entry merges with the built-in value of the same id field by field: a field left out keeps the built-in
value, so an entry that sets AudioInput alone keeps the model's built-in effort restriction (for a model the table
does not know, a switch left out is false and the efforts stay unrestricted). A ReasoningEfforts written empty
says the model takes none. Thinking takes one name: a comma-separated value (EffortOnly,Toggle) refuses the
load. A registry the host registers itself (IModelCapabilityRegistry, before
AddOrkeonLlmProfiles) wins over the configured one. A model id containing : (Ollama's llama3:8b) cannot be a
key: .NET configuration has no escape for the path separator :. Key its prefix instead (llama3 matches
llama3:8b and every other tag). An entry without a model id ("", " ", or a key such as :Vision) is refused,
whatever it holds (configuration).
- Audio in. An
AudioContentPartreaches a model that takes it as aninput_audiopart of the Chat Completions API (raw bytes,audio/wavoraudio/mpeg; the API takes no audio address). The OpenAI-compatible family does it throughContentConverter.ToOpenAIContentParts(content, model, registry), so OpenAI and Gemini (through its compatible layer) are covered;ToAIContentsjudges the same way for the chat-client path. A model the table does not credit with audio, or does not know, is refused with aNotSupportedExceptionthat names the model and theLlm:ModelCapabilities:<id>:AudioInputkey that declares it: the audio is never dropped in silence. Anthropic and Ollama stay refusals (their APIs take no audio); a provider that declares no vision keeps its structured warning "message attachments (audio)" and sends the text only. Azure OpenAI is named by deployment: declare the deployment with an override. Audio output is out of scope: the table records it, nothing sends or reads it. - Thinking and effort, judged against the model (event id
110,113). Every provider, Anthropic and Ollama included, judgesLlmConfig.Thinkingthe same way before writing it. The thinking support that applies is the lower of the provider's and the model's: on a model that always reasons (EffortOnly) "on" sends no switch and "off" is dropped with a structured warning; on a model withThinking: Nonenothing is sent, with a warning. An effort is lowered to the highest level below it that both the provider's field and the model take, with a warning naming the model and its efforts; a model whose list is written empty gets no effort. A vendor value outside the five levels (none,minimal) is sent when the model's list holds it or the model is unknown; efforts compare case-insensitively. A token budget is judged against the provider's declared support alone: a provider that declaresBudgetsends it even to a model that always reasons, since the gateway converts it (OpenRouter on Gemini 3.7, campaign M18). Ollama'sthinkholds one value, so "off" with an effort sendsthink: falseand warns that the effort is not sent. A model the table does not know gets the options as asked, with a warning that nothing was checked, once per provider instance and model (Llm:ModelCapabilities:<id>silences it); a model whose entry leavesReasoningEffortsout is not restricted. - Anthropic's switch is per model. "On" sends
thinking: {type: adaptive, display: summarized}, so the trace comes back as a summary. "Off" sends the model's own value:between_toolson Sonnet 5.5, which takes it up to efforthigh,disabledup toxhighon Sonnet 5, anddisabledup tohighon every other Claude model; above that effort the switch is left out with a warning, since the API refuses the pair. A buffered answer carries the thinking blocks in thereasoning_contentmetadata, as the stream does. - What comes back. The reasoning tokens (
completion_tokens_details.reasoning_tokens) are read for every OpenAI-compatible dialect, buffered and streamed, and the chat-client adapter maps the provider's finish reason ontoChatResponse.FinishReason(stop,length,tool_calls,content_filter, or the vendor's own value): Gemini'sMODEL_CONTEXT_WINDOW_EXCEEDEDreads aslength, itsRECITATIONandPROHIBITED_CONTENTascontent_filter. - Fallback chains (
WarnAboutLesserFallbacks) read the same table: a fallback whose model takes no audio, has a lower effective thinking support (a model that cannot reason, or one that cannot be switched off) or fewer efforts than the primary's is warned about once, next to vision, response format and grammar. Each link is read with its provider's name, so a<provider>/<model>entry applies; a vendor value (none,minimal) is not reported missing on a fallback model the table does not list, since a call would send it unchecked. - Asking the provider (#359, #360,
ORKEXP005).ILlmProvider.DescribeReasoning(model)answers what a model takes with this provider:LlmReasoningCapabilities(the model, whether the table knows it, the effectiveThinkingSupport, and the levels amonglowtomax, in order, the provider'sEffortLevelsnarrowed by the model'sReasoningEfforts: empty when the model takes no effort or does not reason). A null or blank model reads the provider's own.ILlmProvider.ReportEffort(config)applies the same judgement as the call to the configuration merged with the host's (soLlm:Thinking:Effortcounts) and answersLlmEffortReport: the effort asked, the effort sent (null when none goes out, Ollama's "off" included) andLlmEffortNoticeLowered,DroppedorUnchecked(a model the table does not know), null when the effort went as asked. It is null when no effort is asked. Both default to null on the interface;HttpLlmProviderBaseanswers them, and every decorator forwards them to its inner provider. A fallback chain answers for its first profile and a decision provider for its target: a call that fails over, or that a verdict routes to another profile, is judged by the provider that serves it. The scripting DSL reads them asLlmConfig.capabilities,ctx.llm.capabilities(opts),ctx.llm.reasoning(opts)and thereasoningfield ofLlmConfig,ActResult,ChatResponseandLlmStream(DSL reference).
Embedding providers
Orkeon:Embeddings:Provider is resolved by DefaultEmbeddingProviderResolver, after a local or
code-analysis provider and before the fail-fast stub. Five names are accepted; another one refuses
the start.
Provider |
Class | Endpoint | Default model (dimensions) |
|---|---|---|---|
openai |
OpenAIEmbeddingProvider |
any M.E.AI embedding generator the host registers | text-embedding-3-small (1536) |
ollama |
OllamaEmbeddingProvider |
/api/embed of the Ollama host |
the configured Model |
mistral |
MistralEmbeddingProvider |
https://api.mistral.ai/v1/embeddings, bearer key |
mistral-embed (1024) |
azure-openai |
AzureOpenAIEmbeddingProvider |
{Endpoint}/openai/deployments/{Model}/embeddings?api-version=, api-key header |
the deployment named by Model |
gemini |
GeminiEmbeddingProvider |
https://generativelanguage.googleapis.com/v1beta/openai/embeddings, bearer key |
gemini-embedding-001 (3072) |
The last three share OpenAICompatibleEmbeddingProviderBase: one request shape (model and
input), answers re-ordered by index, batches cut at BatchSize, and a vector whose length is not
the declared Dimension refused. When Model and Dimension keep the global defaults
(text-embedding-3-small, 1536), mistral and gemini use their own model and dimensions; azure-openai
always takes Model as the deployment name. A 4xx or 5xx answer raises an HttpRequestException that
names the provider, the status and the start of the body, never the key. Embedding calls do not retry: a 429 or a 5xx reaches the caller as that exception on the first answer, and Llm:MaxRetries applies to chat only. mistral-embed (1024) and gemini-embedding-001 (3072) have a fixed size and these providers send no output size: a Dimension other than that size (the global default counts as unset) is refused when the provider is built, naming the size, before any call. The key is ApiKey or the
variable named by ApiKeyEnvVar; with neither, the first call fails with a message naming both.
Model ids of the embedding providers are not in the capability table of
What a model accepts: it covers chat input and reasoning effort.
Changing the embedding model invalidates stored vectors. At start, a guard (run by RunnerHost.Build for orkeon run, the other commands and the daemon, by the REPL's start, and by the hosted service of a host that starts its services) compares the
provider, model and dimensions with the fingerprint recorded at Orkeon:Embeddings:FingerprintPath
(default /workspace/.orkeon/embedding-fingerprint.json, in the virtual file system). The first start
records it; a different one stops the host with an error that names both models. Nothing is migrated:
restore the previous settings, or re-embed and clear the stored vectors, then delete the fingerprint file.
The guard acts only where a mount covers FingerprintPath. A host whose embedding provider is configured, with or without an Orkeon:Embeddings section, is guarded there: a host that embedded under the default provider and then writes a section (a switch to Mistral or Gemini) finds the default's fingerprint and is refused. A host with no embedding provider (none registered, no section) writes nothing. A host whose FingerprintPath lies under no mount (a run with no /workspace) skips the guard at Debug, with no warning and no "Access denied" line; on a mount that grants no Write or Create on the file and no Create on its folder (overrides included; a read-only /workspace) the guard still reads and compares a recorded fingerprint and still refuses a changed model, but writes nothing, and when no fingerprint is recorded yet logs one Debug line that names Orkeon:Embeddings:FingerprintPath, never a Warning; a provider that cannot report its dimensions at start (a custom ModelPath that is missing) makes the guard log a Warning with the provider's own message, and the next call asks again; and an empty FingerprintPath turns the guard off, with no log. Every warning about reading or writing the fingerprint names Orkeon:Embeddings:FingerprintPath; the warning for a provider that cannot report its dimensions carries that provider's own message instead.
/workspace is not always writable, so the two commands that cannot rely on it guard where it means something. orkeon forge defaults FingerprintPath to /forge/embedding-fingerprint.json, recorded per session: the guard there protects one session and nothing across sessions. That default sits at the lowest precedence: a FingerprintPath set explicitly in the settings wins. orkeon rag uses no host fingerprint (its default FingerprintPath is empty): the manifest of each collection records the provider, model and dimensions it was indexed with and refuses a change on ingest without --reindex, on search, on eval rag --no-ingest -c and on eval rag -c when its ingest pre-pass runs without --reindex, with a line that names the collection, both models and orkeon rag ingest --reindex, which rebuilds the collection and passes. A first write that a read-only mount denies is logged at Debug; any other failure to write still warns.
Cohere, Voyage and pgvector are not shipped.
Decorators and registration
MeteredLlmProviderβ the one place LLM usage is measured: it reports every call of the provider it wraps to the host'sILlmUsageSink, with the vendor's own cost when the answer carries one and an estimate flagged as such otherwise.LlmProviderFactorywraps every provider it builds; a provider the factory does not build (the echo provider of a host without anLlmsection, a Microsoft Agent Framework agent, a test double) is registered withservices.AddOrkeonLlmProvider(sp => β¦, baseConfig), which exposes it asILlmProvider,IBasicLlmProviderandIChatClientover one metered instance.AddOrkeonInfrastructure()registers no model of its own:orkeon run,orkeon-hostandorkeon-replregister theirs from theLlmsection β or the echo provider when there is none β and a container that registers none fails at its first LLM resolution, naming the missing service.RateLimitedLlmProviderβ the host'sRateLimitingat the entrance of each provider: every call takes one lease of theILlmRateLimiter(global, per provider, concurrency) for the time of the call, a stream for its whole enumeration. It is applied with the meter, around it, at the points that meter a provider β the factory,AddOrkeonLlmProviderandAddOrkeonLlmProfile, the profile registry'sForProvider, the manager's resolver, the planner of a C# crew (LlmProviderEntrance) β so every model call of the host is limited once, the scripts'ctx.llm.*included. A provider already limited is left as it is; a call made inside another limited call takes no lease (underMaxConcurrentRequests: 1it would wait on itself); a provider that runs its own tools (RunsOwnTools, the Microsoft Agent Framework bridge) is not limited itself β what it calls of Orkeon's model is. A refusal is retried five times on itsRetryAfter, then the call fails with the limiter's reason.- Call mode β
CallModeLlmProviderdecides how each call of a provider built from a setting reaches the model, for every caller (agent loop, manager, planner, judges, RAG). The setting'sCallModekey (Llm:CallMode,Llm:Profiles:<name>:CallMode) isAuto(absent or empty),BufferorStream, case ignored; another value fails the load.Bufferanswers whole;StreamandAutostream on the wire when the provider streams natively and fold the events into the sameLlmResponse(text, tool calls, finish reason, usage), so the meter and the limiter still see one call. InAuto, a clear refusal of streaming before any event (status 404, 405, 415 or 501, or 400/422 whose body names the stream, or a provider that declares no streaming) falls back to a buffered call for the rest of the run (LlmCallModeScope, opened per crew run), with one warning; a 401, 403, 429, a timeout or a stream cut after its first event stays a failure. InStream, a refusal fails the call namingAutoandBuffer. It sits inside the meter and the limiter, applied byLlmProviderFactory. - LLM exchange logging β
LlmLoggingDelegatingHandler(Orkeon.Infrastructure.Logging) captures every HTTP exchange (headers and payload, sanitized byLogSanitizer: credential headers redacted by name, secrets in bodies by pattern) as JSON Lines plus a structured log summary.services.AddLlmExchangeLogging(logDirectory, options)injects it into everyIHttpClientFactoryclient; theIHttpClientBuilderoverload targets one named client.LlmLoggingOptions:MaxBodyLengthChars(0 = no truncation),LogStreamingExchanges(true; a stream passes through to the caller as it arrives and is logged once it ends, as its rebuilt answer, or the raw text received when it was cut),FullEmbeddingLog(true; false shortens embedding arrays to a preview). The runners switch it on with--llm-log/--llm-log-pathand read the options from theLlmLoggingsettings section; the files go to the internal/llm-logsmount, never visible to agents. It is a debugging facility: the capture holds full prompts.
Validating a provider against its real API
Every unit test in this area speaks to a mocked HTTP handler, which proves Orkeon sends what
we believe it sends β not that the vendor accepts it. The second proof is produced by the
campaign kit in llmproviders-test/:
llmproviders-test/run-campaign.sh --provider ollama --model llama3.2 # zero-cost first run
llmproviders-test/run-campaign.sh --all --config providers.local.json --dry-run
It drives orkeon llm probe over modes M1βM10, M12, M13 and M15βM18 of the provider
test protocol (maintainers' internal matrix) and archives
one Markdown report per campaign. M15βM17 send Orkeon's thinking configuration through the
provider and record, per step, the reasoning fields that really went on the wire. M18 goes
the other way: with --raw-reasoning, a handler rewrites the reasoning fields of the request
just before it leaves (the provider still builds auth, URL, headers and dialect), so it
bypasses Orkeon's capability judgement and measures what the vendor itself accepts or refuses.
Each step also reads the vendor's answer on the wire before the provider parses it (trace,
reasoning tokens, finish reason), so a gap between what the vendor returned and what the provider
surfaced shows in the report. Once every mode ran, each M15βM17 step is checked against M18: a value
Orkeon sent that the vendor refused (400, 422) means the model capability table misses a restriction,
a value Orkeon lowered or left out that the vendor takes means the table is too narrow; either fails
the mode. A value that switches reasoning off counts as taken only when it stopped the reasoning, and
a vendor that keeps reasoning at effort:none proves nothing by accepting an effort. orkeon llm models -p <provider> --filter 'gpt-5.6-*'
lists what a provider currently serves, so a campaign never depends on a hand-maintained
model list.
See also: Provider comparison Β· LLM response format Β· Multi-modal content Β· Local models Β· Memory system Β· Security Β· Back to index