π«π· Version franΓ§aise
Multi-modal content (vision)
Image content flows end to end, from the
MultiModalContentabstractions (Domain) down to the provider payloads β Anthropic content blocks, OpenAI content parts (written once for the fourteen OpenAI-compatible providers) and Ollama'simagesarray. This guide describes the flow, the formats emitted on the wire, the opt-in activation and the limits.
Flow overview
MultiModalContent (Domain) LlmMessage (Domain) Provider payload (Infrastructure)
ββββββββββββββββββββββββββββ βββββββββββββββββββββββββββββββββ βββββββββββββββββββββββββββββββββββββββ
β TextContentPart β β Role = "user" β β Anthropic: "text"/"image" blocks β
β ImageContentPart β β β Content = text fallback β β β OpenAI : "text"/"image_url" parts β
β (bytes or URI) β β MultiModalContent = parts β β Ollama : content + "images" β
ββββββββββββββββββββββββββββ βββββββββββββββββββββββββββββββββ βββββββββββββββββββββββββββββββββββββββ
MultiModalContent(immutable value object,Orkeon.Domain.SharedKernel.ValueObjects.Content) carries the text/image/audio/file parts.LlmMessage.User(MultiModalContent)creates a user message that transports the parts and a text fallback (Content = ToTextOnly()) for providers without vision.ContentConverter(Orkeon.Infrastructure.LLMs.Converters) composes the payload fragments specific to each API (ToAnthropicContentBlocks,ToOpenAIContentParts,ToOllamaMessage).
Composing and sending an image
using Orkeon.Domain.SharedKernel.ValueObjects;
using Orkeon.Domain.SharedKernel.ValueObjects.Content;
// 1. From bytes (sent as base64)
var content = MultiModalContent.Empty()
.AddText("Describe this chart")
.AddImage(ImageContentPart.FromBytes(pngBytes, "image/png"));
// 2. Or from an http(s) URL (the vendor downloads the image β not on Ollama, see below)
var contentFromUrl = MultiModalContent.Empty()
.AddText("What does this photo show?")
.AddImage(ImageContentPart.FromUri(new Uri("https://example.com/photo.jpg"), "image/jpeg"));
// 3. Send via any ILlmProvider declaring Vision
var response = await provider.ChatAsync([LlmMessage.User(content)]);
Loading an image from the VFS (IMultiModalContentLoader)
The loader reads the file via IFileSystemService (virtual paths, audited rights),
infers the MIME type from the extension and validates the configured constraints
(Orkeon:MultiModal β max size, allowed formats):
var loader = provider.GetRequiredService<IMultiModalContentLoader>();
var image = await loader.LoadImageAsync("/workspace/chart.png");
var content = MultiModalContent.Empty()
.AddText("Analyze this chart")
.AddImage(image);
Clear errors guaranteed:
| Situation | Exception |
|---|---|
Unsupported extension (.bmp, .tiff, β¦) |
NotSupportedException (lists the supported extensions) |
| File not found | FileNotFoundException |
| Outside a mount / insufficient rights | FileAccessDeniedException |
Size > MaxImageSizeBytes or format excluded by the options |
InvalidOperationException |
Orkeon:MultiModal:Enabled = false |
InvalidOperationException |
Formats emitted on the wire
Anthropic (Messages API) β content blocks:
{
"role": "user",
"content": [
{ "type": "text", "text": "Describe this chart" },
{ "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": "iVBOR..." } }
]
}
Images by http(s) URL use "source": {"type": "url", "url": "https://..."}.
Base64 data URLs (data:image/png;base64,...) are unwrapped into a base64 source.
OpenAI (Chat Completions) β content parts:
{
"role": "user",
"content": [
{ "type": "text", "text": "Describe this chart" },
{ "type": "image_url", "image_url": { "url": "data:image/png;base64,iVBOR..." } }
]
}
Images by http(s) URL or data URL pass through as-is in image_url.url;
raw bytes are encoded as a base64 data URL. The same parts are written by every
OpenAI-compatible provider (OpenAI, Azure OpenAI, DeepSeek, Gemini, Grok, HuggingFace, Kimi, Mammouth,
MiniMax, Mistral, OpenRouter, Qwen, Together AI, Z.AI).
Ollama (/api/chat) β no content parts: the message keeps a plain content string and
the images travel as bare base64 strings (no data: prefix, no media type):
{ "role": "user", "content": "Describe this chart", "images": ["iVBOR..."] }
A message carrying any non-text part switches Ollama from /api/generate to /api/chat. Ollama
never fetches remote URLs: an image referenced only by URL is skipped with a structured warning
(β¦ image(s) referenced by URL) β load the bytes instead. Ollama takes no audio or file input:
such a part raises NotSupportedException before anything is sent, as on Anthropic and the
OpenAI-compatible providers.
Activation (opt-in)
The subsystem remains opt-in β see Opt-in subsystems:
services.AddOrkeonFileSystem(configuration); // loader prerequisite (VFS)
services.AddOrkeonMultiModal(configuration); // binds Orkeon:MultiModal
// or: services.AddOrkeonMultiModal(); // default options
Options (Orkeon:MultiModal, MultiModalOptions): Enabled (true), MaxImageSizeBytes
(20 MB by default), SupportedImageFormats (png, jpeg, gif, webp by default),
MaxAudioDurationSeconds (300), SupportedAudioFormats (wav, mp3, ogg). No image is resized:
an image over the size limit fails validation.
Sending vision payloads from the providers does not depend on the DI activation: an
LlmMessagecarrying aMultiModalContentwith images is composed into a structured payload by every provider whose capabilities declareVision(the sharedOpenAICompatibleProviderBasedoes the translation once).AddOrkeonMultiModalenables the validation (IContentValidationService) and the VFS loader (IMultiModalContentLoader).
Supported providers and formats
| Capability | Supported |
|---|---|
Vision providers (capability Vision = true) |
All 16: Anthropic, OpenAI, Azure OpenAI, DeepSeek (native on deepseek-flash since V4.1; deepseek-v4-pro is text-only), Gemini, Grok, HuggingFace, Kimi, MiniMax (per model: the default MiniMax-M3 sees, MiniMax-M2 does not), Mistral, Ollama (native images, per model), Qwen, Together AI (per model: the default zai-org/GLM-5.3-Flash sees), Z.AI (per model: the default glm-5.3-flashx and glm-5.3-flash see, glm-5.2 does not), OpenRouter (per model) and Mammouth AI (per model, from the vendor's text, image list) |
| Model without vision behind a vision provider | The vendor's own refusal, attributed by CapabilityMismatchHint on the OpenAI-compatible providers and Ollama (the capability is declared per provider, the reality is per model) |
Provider without the capability (a third-party ILlmProvider declaring nothing) |
Degradation to the LlmMessage.Content text fallback, with a structured warning naming the dropped parts |
| Image MIME types | image/png, image/jpeg, image/gif, image/webp |
| Image sources | Raw bytes (base64), http(s) URL, base64 data URL |
| Audio / files to the providers | Not supported β explicit NotSupportedException on Anthropic, the OpenAI-compatible providers and Ollama |
On Anthropic and the OpenAI-compatible providers, an unsupported MIME type (e.g.
image/bmp), an unsupported URI scheme (e.g. ftp://) or an audio/file part in a vision
payload raise an explicit exception β no silent degradation to text. Ollama refuses an
audio/file part the same way.
Microsoft.Extensions.AI interop
Orkeon converts the image/audio/file parts to the native Microsoft.Extensions.AI
types DataContent (bytes) and UriContent (URI), and back, inside
Orkeon.Infrastructure; the converter is internal, not an API to call.
Tests
ContentConverterPayloadTestsβ composition of Anthropic blocks / OpenAI parts, VFS loading, errors on unsupported formats.AnthropicVisionPayloadTests/OpenAIVisionPayloadTests/ProviderVisionPayloadTests(every OpenAI-compatible provider) /OllamaChatEndpointTestsβ verification of the JSON actually emitted on the wire (fake HTTP handler).MultiModalContentLoaderTestsβ option constraints (size, formats, Enabled).OptInSubsystemsRegistrationTestsβ opt-in DI registration.