Table of Contents

πŸ‡«πŸ‡· Version franΓ§aise

Multi-modal content (vision)

Image content flows end to end, from the MultiModalContent abstractions (Domain) down to the provider payloads β€” Anthropic content blocks, OpenAI content parts (written once for the fourteen OpenAI-compatible providers) and Ollama's images array. This guide describes the flow, the formats emitted on the wire, the opt-in activation and the limits.

Flow overview

MultiModalContent (Domain)            LlmMessage (Domain)              Provider payload (Infrastructure)
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ TextContentPart          β”‚   β”‚ Role = "user"                 β”‚   β”‚ Anthropic: "text"/"image" blocks    β”‚
β”‚ ImageContentPart         β”‚ β†’ β”‚ Content = text fallback       β”‚ β†’ β”‚ OpenAI   : "text"/"image_url" parts β”‚
β”‚ (bytes or URI)           β”‚   β”‚ MultiModalContent = parts     β”‚   β”‚ Ollama   : content + "images"       β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
  • MultiModalContent (immutable value object, Orkeon.Domain.SharedKernel.ValueObjects.Content) carries the text/image/audio/file parts.
  • LlmMessage.User(MultiModalContent) creates a user message that transports the parts and a text fallback (Content = ToTextOnly()) for providers without vision.
  • ContentConverter (Orkeon.Infrastructure.LLMs.Converters) composes the payload fragments specific to each API (ToAnthropicContentBlocks, ToOpenAIContentParts, ToOllamaMessage).

Composing and sending an image

using Orkeon.Domain.SharedKernel.ValueObjects;
using Orkeon.Domain.SharedKernel.ValueObjects.Content;

// 1. From bytes (sent as base64)
var content = MultiModalContent.Empty()
    .AddText("Describe this chart")
    .AddImage(ImageContentPart.FromBytes(pngBytes, "image/png"));

// 2. Or from an http(s) URL (the vendor downloads the image β€” not on Ollama, see below)
var contentFromUrl = MultiModalContent.Empty()
    .AddText("What does this photo show?")
    .AddImage(ImageContentPart.FromUri(new Uri("https://example.com/photo.jpg"), "image/jpeg"));

// 3. Send via any ILlmProvider declaring Vision
var response = await provider.ChatAsync([LlmMessage.User(content)]);

Loading an image from the VFS (IMultiModalContentLoader)

The loader reads the file via IFileSystemService (virtual paths, audited rights), infers the MIME type from the extension and validates the configured constraints (Orkeon:MultiModal β€” max size, allowed formats):

var loader = provider.GetRequiredService<IMultiModalContentLoader>();
var image = await loader.LoadImageAsync("/workspace/chart.png");

var content = MultiModalContent.Empty()
    .AddText("Analyze this chart")
    .AddImage(image);

Clear errors guaranteed:

Situation Exception
Unsupported extension (.bmp, .tiff, …) NotSupportedException (lists the supported extensions)
File not found FileNotFoundException
Outside a mount / insufficient rights FileAccessDeniedException
Size > MaxImageSizeBytes or format excluded by the options InvalidOperationException
Orkeon:MultiModal:Enabled = false InvalidOperationException

Formats emitted on the wire

Anthropic (Messages API) β€” content blocks:

{
  "role": "user",
  "content": [
    { "type": "text", "text": "Describe this chart" },
    { "type": "image", "source": { "type": "base64", "media_type": "image/png", "data": "iVBOR..." } }
  ]
}

Images by http(s) URL use "source": {"type": "url", "url": "https://..."}. Base64 data URLs (data:image/png;base64,...) are unwrapped into a base64 source.

OpenAI (Chat Completions) β€” content parts:

{
  "role": "user",
  "content": [
    { "type": "text", "text": "Describe this chart" },
    { "type": "image_url", "image_url": { "url": "data:image/png;base64,iVBOR..." } }
  ]
}

Images by http(s) URL or data URL pass through as-is in image_url.url; raw bytes are encoded as a base64 data URL. The same parts are written by every OpenAI-compatible provider (OpenAI, Azure OpenAI, DeepSeek, Gemini, Grok, HuggingFace, Kimi, Mammouth, MiniMax, Mistral, OpenRouter, Qwen, Together AI, Z.AI).

Ollama (/api/chat) β€” no content parts: the message keeps a plain content string and the images travel as bare base64 strings (no data: prefix, no media type):

{ "role": "user", "content": "Describe this chart", "images": ["iVBOR..."] }

A message carrying any non-text part switches Ollama from /api/generate to /api/chat. Ollama never fetches remote URLs: an image referenced only by URL is skipped with a structured warning (… image(s) referenced by URL) β€” load the bytes instead. Ollama takes no audio or file input: such a part raises NotSupportedException before anything is sent, as on Anthropic and the OpenAI-compatible providers.

Activation (opt-in)

The subsystem remains opt-in β€” see Opt-in subsystems:

services.AddOrkeonFileSystem(configuration);   // loader prerequisite (VFS)
services.AddOrkeonMultiModal(configuration);   // binds Orkeon:MultiModal
// or: services.AddOrkeonMultiModal();         // default options

Options (Orkeon:MultiModal, MultiModalOptions): Enabled (true), MaxImageSizeBytes (20 MB by default), SupportedImageFormats (png, jpeg, gif, webp by default), MaxAudioDurationSeconds (300), SupportedAudioFormats (wav, mp3, ogg). No image is resized: an image over the size limit fails validation.

Sending vision payloads from the providers does not depend on the DI activation: an LlmMessage carrying a MultiModalContent with images is composed into a structured payload by every provider whose capabilities declare Vision (the shared OpenAICompatibleProviderBase does the translation once). AddOrkeonMultiModal enables the validation (IContentValidationService) and the VFS loader (IMultiModalContentLoader).

Supported providers and formats

Capability Supported
Vision providers (capability Vision = true) All 16: Anthropic, OpenAI, Azure OpenAI, DeepSeek (native on deepseek-flash since V4.1; deepseek-v4-pro is text-only), Gemini, Grok, HuggingFace, Kimi, MiniMax (per model: the default MiniMax-M3 sees, MiniMax-M2 does not), Mistral, Ollama (native images, per model), Qwen, Together AI (per model: the default zai-org/GLM-5.3-Flash sees), Z.AI (per model: the default glm-5.3-flashx and glm-5.3-flash see, glm-5.2 does not), OpenRouter (per model) and Mammouth AI (per model, from the vendor's text, image list)
Model without vision behind a vision provider The vendor's own refusal, attributed by CapabilityMismatchHint on the OpenAI-compatible providers and Ollama (the capability is declared per provider, the reality is per model)
Provider without the capability (a third-party ILlmProvider declaring nothing) Degradation to the LlmMessage.Content text fallback, with a structured warning naming the dropped parts
Image MIME types image/png, image/jpeg, image/gif, image/webp
Image sources Raw bytes (base64), http(s) URL, base64 data URL
Audio / files to the providers Not supported β€” explicit NotSupportedException on Anthropic, the OpenAI-compatible providers and Ollama

On Anthropic and the OpenAI-compatible providers, an unsupported MIME type (e.g. image/bmp), an unsupported URI scheme (e.g. ftp://) or an audio/file part in a vision payload raise an explicit exception β€” no silent degradation to text. Ollama refuses an audio/file part the same way.

Microsoft.Extensions.AI interop

Orkeon converts the image/audio/file parts to the native Microsoft.Extensions.AI types DataContent (bytes) and UriContent (URI), and back, inside Orkeon.Infrastructure; the converter is internal, not an API to call.

Tests

  • ContentConverterPayloadTests β€” composition of Anthropic blocks / OpenAI parts, VFS loading, errors on unsupported formats.
  • AnthropicVisionPayloadTests / OpenAIVisionPayloadTests / ProviderVisionPayloadTests (every OpenAI-compatible provider) / OllamaChatEndpointTests β€” verification of the JSON actually emitted on the wire (fake HTTP handler).
  • MultiModalContentLoaderTests β€” option constraints (size, formats, Enabled).
  • OptInSubsystemsRegistrationTests β€” opt-in DI registration.