[API Proposal]: IOcrClient — an OCR / document-extraction capability for Microsoft.Extensions.AI

[luisquintanilla]

[API Proposal]: IOcrClient — an OCR / document-extraction capability for Microsoft.Extensions.AI

Updates

• 2026-07-15 — API-review alignment (family symmetry). Replaced the unary-only shape with the family's unary + streaming pair: added IAsyncEnumerable<OcrResponseUpdate> ExtractStreamingAsync(...) (the IChatClient.GetStreamingResponseAsync twin), OcrResponseUpdate, and an OcrResponseUpdateExtensions.ToOcrResult/ToOcrResultAsync reducer, and removed IProgress<OcrProgress> and OcrProgress (progress now rides on the streamed update). OcrBlock.Kind / OcrTableCell.Kind are now ChatRole-style open structs (OcrBlockKind / OcrTableCellKind), not raw strings. Added OcrPage.Width / Height + OcrCoordinateUnit so bounding-box coordinates are interpretable across engines. Removed the leaky OcrResult.OcrSource (ModelId + OcrClientMetadata.ProviderName carry provenance). Unsealed the result / data types to match ChatResponse / ChatOptions. Surface is now 29 public types.
• 2026-07-14 — Reformatted to the API-proposal template and refreshed the API Proposal speclet to match the surface implemented in PR Add IOcrClient OCR/document-extraction capability to Microsoft.Extensions.AI #7588: GetTextAsync → ExtractAsync; IOcrClient : IDisposable; typed geometry OcrPoint / OcrBoundingBox with OcrBoundingRegion.Polygon as IReadOnlyList<OcrPoint>; 1-based OcrPage.PageNumber; [Experimental("MEAI001")]; added OcrImage, OcrPage.Images, OcrOptions.Clone(), OcrClientMetadata, the OcrClientExtensions surface (incl. opt-in ExtractFromUriAsync), and the full builder / middleware / DI types. The sibling IDocumentAnalysisClient is scoped out to its own future proposal.

Background and motivation

Document parsing is a core RAG and ingestion building block, but Microsoft.Extensions.AI.Abstractions
does not yet have an OCR / document-extraction capability. Today, you either wire provider SDKs directly
or route OCR through IChatClient, which loses native document structure such as tables, bounding boxes,
confidence, polygons, and reading order.

This proposal adds IOcrClient as a provider-neutral capability interface in
Microsoft.Extensions.AI.Abstractions, with the same builder, middleware, and DI shape developers
already use across the Microsoft.Extensions.AI capability family.

What is OCR / document AI?

OCR is the process of extracting text from documents and images. Document AI goes further: it keeps
structure around that text, including pages, tables, blocks, regions, confidence scores, and reading
order.

For RAG and ingestion pipelines, that structure matters. A document reader should not only produce
markdown. It should also preserve enough page, region, table, confidence, and source metadata for
downstream chunking, retrieval, grounding, and evaluation.

Why an abstraction?

Microsoft.Extensions.AI.Abstractions ships a family of capability interfaces: IChatClient,
IEmbeddingGenerator, ISpeechToTextClient, ITextToSpeechClient, IImageGenerator,
IRealtimeClient, and IHostedFileClient. There is no OCR / document-extraction capability, even
though Microsoft.Extensions.DataIngestion (MEDI) already depends on document parsing. Its
IngestionDocumentReader roadmap, per MS Learn, includes LlamaParse and Azure Document Intelligence,
both hosted document-AI services that need a provider-agnostic seam.

Today, if you want to use document-AI models, you must:

1. Couple ingestion code directly to a provider SDK.
2. Model OCR as a chat prompt against IChatClient.
3. Add reader-mode flags for specific engines or hosts.
4. Rebuild retry, logging, DI, middleware, and test seams per provider.
5. Give up native structure when the abstraction cannot represent it.

CommunityToolkit/AI #3 is a representative example.
It introduced a PdfReadingMode.VisionOnly flag that routes whole-document transcription through a
vision LLM (IChatClient). That is a layer leak: a model choice hardened into a reader-mode flag,
with temporal coupling because the reader emits placeholders that are useless unless a specific enricher
runs. The cleaner shape is a capability client the reader composes, exactly how MEDI's enrichers
already compose an injected IChatClient.

OCR is not chat. Most OCR / document-AI engines emit structured output: tables, bounding boxes,
confidence, polygons, reading order. That does not fit ChatResponse. A vision LLM can transcribe by
prompt, but it is the lowest-fidelity path and loses native structure. Purpose-built engines (Mistral
OCR, Azure Document Intelligence, Azure AI Content Understanding) and local document VLMs
(granite-docling, PaddleOCR) beat it. Modeling OCR as "call IChatClient with an image" makes those
engines unrepresentable without discarding their value. That points to a separate capability
interface, independent of IChatClient.

Prototype validation

This proposal is not a sketch. It describes a working prototype spiked across four real engine
providers (three using no IChatClient at all) plus one vision-LLM adapter, composed with an
IChatClient-style builder pipeline, and validated end-to-end through a MEDI ingestion pipeline:

• FoundryMistralOcrClient: Azure AI Foundry mistral-ocr-4-0, keyless Entra (verified HTTP 200).
• MistralOcrClient: Mistral-direct, API key.
• AzureDocumentIntelligenceClient: Azure.AI.DocumentIntelligence (AnalyzeResult, native polygons + table cells).
• ContentUnderstandingClient: Azure.AI.ContentUnderstanding 1.1.0, keyless Entra (markdown path).
• VisionLlmOcrClient: the one adapter over IChatClient (gpt-4o / Gemini / local Ollama GLM-OCR), the lowest-fidelity path.

Every claim below ("one pipeline wraps all engines", "the polygon flows losslessly from DI and Mistral",
"streaming yields pages as they finish") is backed by code that builds and runs, not by assertion. The demo
is a public, runnable proof: one IOcrClient in front of four OCR engines, bridged into a MEDI RAG
pipeline, with the identical consumer loop across both provider archetypes (document-native and
image-per-page).

The design goal is provider-neutrality: one small set of composable primitives that every provider
maps onto equally, judged on interoperability, reusability, composition, extensibility. No provider
is privileged. Providers form a coverage matrix, not a hierarchy.

The same precedent that justifies splitting OpenAIClient / AzureOpenAIClient behind IChatClient
applies here. The interface is the portability guarantee; concrete classes split by engine and by
host where credential, route, or provider behavior leaks. The model/deployment id is a parameter,
never a type or a boolean flag.

Building-block symmetry with Microsoft.Extensions.AI

The goal is not a one-off OCR helper. The goal is another building block in the same capability family.

If you know one Microsoft.Extensions.AI capability, you should know the others: abstraction,
options/result types, delegating base, builder/middleware, provider implementation, and DI registration.

Capability	Abstraction	Options / result types	Delegating base	Builder / middleware	Example implementation	DI registration shape
Chat	IChatClient	ChatOptions, ChatResponse, ChatResponseUpdate	DelegatingChatClient	ChatClientBuilder, .Use(...), logging, OpenTelemetry, caching, function invocation	OpenAIChatClient	AddChatClient, AddKeyedChatClient
Embeddings	IEmbeddingGenerator<TInput,TEmbedding>	EmbeddingGenerationOptions, GeneratedEmbeddings<TEmbedding>	DelegatingEmbeddingGenerator<TInput,TEmbedding>	EmbeddingGeneratorBuilder<TInput,TEmbedding>, .Use(...), logging, OpenTelemetry, caching	OpenAIEmbeddingGenerator	AddEmbeddingGenerator, AddKeyedEmbeddingGenerator
Speech-to-text	ISpeechToTextClient	SpeechToTextOptions, SpeechToTextResponse, response updates	DelegatingSpeechToTextClient	SpeechToTextClientBuilder, .Use(...), logging, OpenTelemetry, options	OpenAISpeechToTextClient	AddSpeechToTextClient, AddKeyedSpeechToTextClient
Text-to-speech	ITextToSpeechClient	TextToSpeechOptions, TextToSpeechResponse, response updates	DelegatingTextToSpeechClient	TextToSpeechClientBuilder, .Use(...), logging, OpenTelemetry, options	OpenAITextToSpeechClient	AddTextToSpeechClient, AddKeyedTextToSpeechClient
Images	IImageGenerator	ImageGenerationOptions, ImageGenerationRequest, ImageGenerationResponse	DelegatingImageGenerator	ImageGeneratorBuilder, .Use(...), logging, options	OpenAIImageGenerator	AddImageGenerator, AddKeyedImageGenerator
Realtime	IRealtimeClient	RealtimeSessionOptions, client/server messages, sessions	DelegatingRealtimeClient	RealtimeClientBuilder, .Use(...), logging, OpenTelemetry, function invocation	OpenAIRealtimeClient	Register IRealtimeClient / keyed clients through DI
Hosted files	IHostedFileClient	HostedFileClientOptions, HostedFileDownloadStream	DelegatingHostedFileClient	HostedFileClientBuilder, .Use(...), logging, OpenTelemetry	OpenAIHostedFileClient	Register IHostedFileClient / keyed clients through DI
OCR / document extraction	IOcrClient	OcrOptions, OcrResult, OcrResponseUpdate, OcrPage, OcrBlock, OcrTable, OcrImage, OcrUsage	DelegatingOcrClient	OcrClientBuilder, .Use(...), logging, OpenTelemetry, configure-options	FoundryMistralOcrClient, MistralOcrClient, AzureDocumentIntelligenceClient, ContentUnderstandingClient, VisionLlmOcrClient	AddOcrClient, AddKeyedOcrClient

That symmetry is the main API shape. IOcrClient should feel like a natural next capability, not a
separate pattern you have to relearn.

Provider coverage

Providers are peers behind a provider-neutral contract, not a tier or hierarchy.

Provider	IOcrClient (markdown/structure)	IDocumentAnalysisClient (typed fields + grounding) — future sibling proposal, not in this PR
Foundry Mistral OCR	yes	—
Azure Document Intelligence	yes	yes (Documents[].Fields)
Content Understanding	yes (markdown path)	yes (fields{} + grounding)
Vision-LLM adapter	yes (lowest fidelity)	—
Local ONNX / Ollama (roadmap)	yes	—

No row is privileged. Some providers implement more of the family than others; the family is the
design, and coverage is a matrix. Content Understanding is the widest-surface conformance test (one
service exercises both interfaces with the same primitives), not an apex; it validates
provider-neutrality because the same polygon / confidence / builder primitives serve its two shapes,
Mistral OCR, Azure DI, and a vision LLM. The second column previews a future sibling capability
(IDocumentAnalysisClient, see Related and future work) and is shown only to illustrate that the same
region / confidence / builder primitives generalize; it is not part of this PR.

API Proposal

The surface below is the settled public API on PR #7588 (29 public types across
Microsoft.Extensions.AI.Abstractions and Microsoft.Extensions.AI). Signatures only, no method
bodies. All new surface ships [Experimental("MEAI001")] from day one (the STT/TTS precedent).

Core abstraction, options, and result types (Microsoft.Extensions.AI.Abstractions)

namespace Microsoft.Extensions.AI;

/// <summary>
/// A capability for OCR / document-extraction engines. Independent of <see cref="IChatClient"/>:
/// engines emit structured output (tables, bounding boxes, confidence, reading order) that does not
/// fit a chat response. One contract, many engines (Mistral OCR, Azure Document Intelligence, Content
/// Understanding, a local ONNX model, or a vision LLM behind an adapter).
/// </summary>
[Experimental("MEAI001")]
public interface IOcrClient : IDisposable
{
    /// <summary>Runs OCR / document parsing over a document stream and returns structured markdown + pages.</summary>
    Task<OcrResult> ExtractAsync(
        Stream document,
        string mediaType,
        OcrOptions? options = null,
        CancellationToken cancellationToken = default);

    /// <summary>
    /// Streams OCR / document parsing as <see cref="OcrResponseUpdate"/> values — one per page as it finishes,
    /// then a terminal update carrying model id + usage (the <see cref="IChatClient.GetStreamingResponseAsync"/>
    /// twin). Reassemble into an <see cref="OcrResult"/> via <see cref="OcrResponseUpdateExtensions.ToOcrResultAsync"/>.
    /// Lets large-document RAG chunk/embed early pages while later pages are still being parsed.
    /// </summary>
    IAsyncEnumerable<OcrResponseUpdate> ExtractStreamingAsync(
        Stream document,
        string mediaType,
        OcrOptions? options = null,
        CancellationToken cancellationToken = default);

    /// <summary>Provider escape hatch (the <see cref="IChatClient.GetService"/> pattern).</summary>
    object? GetService(Type serviceType, object? serviceKey = null);
}

/// <summary>Normalized OCR result — "normalize the common, preserve the raw" (the ChatResponse pattern).</summary>
[Experimental("MEAI001")]
public class OcrResult
{
    public OcrResult(IReadOnlyList<OcrPage> pages);

    public IReadOnlyList<OcrPage> Pages { get; }
    public string Markdown { get; }                       // derived: pages joined with blank lines
    public string? ModelId { get; set; }                  // provenance (the ChatResponse.ModelId pattern)
    public OcrUsage? Usage { get; set; }
    public object? RawRepresentation { get; set; }        // provider-native object — nothing is lost
    public AdditionalPropertiesDictionary? AdditionalProperties { get; set; }
}

[Experimental("MEAI001")]
public class OcrPage
{
    public OcrPage(int pageNumber, string markdown);

    public int PageNumber { get; }                        // 1-based
    public string Markdown { get; }
    public IReadOnlyList<OcrTable> Tables { get; set; }   // default: empty
    public IReadOnlyList<OcrBlock> Blocks { get; set; }   // text + region + confidence; default: empty
    public IReadOnlyList<OcrImage> Images { get; set; }   // default: empty
    public double? Confidence { get; set; }
    public float? Width { get; set; }                     // page dims + unit → bbox coords are interpretable
    public float? Height { get; set; }
    public OcrCoordinateUnit? CoordinateUnit { get; set; } // Pixel / Inch / Normalized
    public AdditionalPropertiesDictionary? AdditionalProperties { get; set; }
}

[Experimental("MEAI001")]
public class OcrBlock
{
    public OcrBlock(string text);

    public string Text { get; }
    public OcrBlockKind? Kind { get; set; }               // open struct: Paragraph / Title / Figure …
    public OcrBoundingRegion? BoundingRegion { get; set; }
    public double? Confidence { get; set; }
}

/// <summary>A single 2-D point in page coordinates (a polygon vertex).</summary>
[Experimental("MEAI001")]
public readonly record struct OcrPoint(float X, float Y);

/// <summary>An axis-aligned bounding box, for coarse filters and hit-testing.</summary>
[Experimental("MEAI001")]
public readonly record struct OcrBoundingBox(float Left, float Top, float Right, float Bottom);

/// <summary>
/// The SHARED, provider-neutral geometry primitive — a polygon of <see cref="OcrPoint"/> vertices
/// (clockwise) so it carries Azure DI's possibly rotation-skewed quad WITHOUT loss. Engines that emit an
/// axis-aligned rect (Mistral OCR) convert via <see cref="FromRectangle"/>. A typed point list makes an
/// odd/empty coordinate count unrepresentable (a flat float list could not).
/// </summary>
[Experimental("MEAI001")]
public class OcrBoundingRegion
{
    public OcrBoundingRegion(int pageNumber, IReadOnlyList<OcrPoint> polygon);

    public int PageNumber { get; }
    public IReadOnlyList<OcrPoint> Polygon { get; }
    public static OcrBoundingRegion FromRectangle(int pageNumber, double left, double top, double right, double bottom);
    public OcrBoundingBox? GetBounds();                   // axis-aligned bounds; null for an empty polygon
}

/// <summary>
/// Cells are the primary, structured representation (the Azure DI shape: indices + spans + kind);
/// MarkdownRepresentation is the fallback when an engine only emits markdown/HTML (Mistral OCR). Consumers
/// prefer cells when present, markdown otherwise — "normalize the common, preserve the raw" for tables.
/// </summary>
[Experimental("MEAI001")]
public class OcrTable
{
    public OcrTable(int rowCount, int columnCount, IReadOnlyList<OcrTableCell>? cells = null, string? markdownRepresentation = null);

    public int RowCount { get; }
    public int ColumnCount { get; }
    public IReadOnlyList<OcrTableCell>? Cells { get; }
    public string? MarkdownRepresentation { get; }
    public OcrBoundingRegion? BoundingRegion { get; set; }
}

[Experimental("MEAI001")]
public class OcrTableCell
{
    public OcrTableCell(int rowIndex, int columnIndex, string content);

    public OcrTableCellKind? Kind { get; set; }           // open struct: ColumnHeader / Content
    public int RowIndex { get; }
    public int ColumnIndex { get; }
    public int RowSpan { get; set; }                      // default 1
    public int ColumnSpan { get; set; }                   // default 1
    public string Content { get; }
}

/// <summary>An image or figure extracted from a page (populated when <see cref="OcrOptions.IncludeImages"/> is set).</summary>
[Experimental("MEAI001")]
public class OcrImage
{
    public DataContent? Content { get; set; }
    public OcrBoundingRegion? BoundingRegion { get; set; }
    public string? Caption { get; set; }
    public double? Confidence { get; set; }
}

/// <summary>A streamed OCR update — one page as it finishes, then a terminal update with model id + usage
/// (the <see cref="ChatResponseUpdate"/> pattern). Reduce a sequence to an <see cref="OcrResult"/> with
/// <see cref="OcrResponseUpdateExtensions"/>.</summary>
[Experimental("MEAI001")]
public class OcrResponseUpdate
{
    public OcrResponseUpdate();
    public OcrResponseUpdate(OcrPage? page);

    public OcrPage? Page { get; set; }                    // page completed in this update (null on the terminal update)
    public int? PagesProcessed { get; set; }              // progress (absorbs the old OcrProgress)
    public int? TotalPages { get; set; }
    public string? Status { get; set; }
    public string? ModelId { get; set; }
    public OcrUsage? Usage { get; set; }
    public object? RawRepresentation { get; set; }
    public AdditionalPropertiesDictionary? AdditionalProperties { get; set; }
}

/// <summary>Reducers that assemble streamed updates back into a single <see cref="OcrResult"/>
/// (the <c>ToChatResponseAsync</c> pattern).</summary>
[Experimental("MEAI001")]
public static class OcrResponseUpdateExtensions
{
    public static OcrResult ToOcrResult(this IEnumerable<OcrResponseUpdate> updates);
    public static Task<OcrResult> ToOcrResultAsync(
        this IAsyncEnumerable<OcrResponseUpdate> updates, CancellationToken cancellationToken = default);
}

[Experimental("MEAI001")]
public class OcrUsage
{
    public int? PagesProcessed { get; set; }
    public AdditionalPropertiesDictionary? AdditionalProperties { get; set; }
}

/// <summary>The kind of a text block — a <see cref="ChatRole"/>-style open set (discoverable well-knowns,
/// still open to provider-specific kinds).</summary>
[Experimental("MEAI001")]
public readonly struct OcrBlockKind : IEquatable<OcrBlockKind>
{
    public OcrBlockKind(string value);                    // throws on null/whitespace
    public static OcrBlockKind Paragraph { get; }         // "paragraph"
    public static OcrBlockKind Title { get; }             // "title"
    public static OcrBlockKind Figure { get; }            // "figure"
    public string Value { get; }
    // == / != / IEquatable / GetHashCode / ToString + a JsonConverter (the ChatRole shape)
}

/// <summary>The kind of a table cell — a <see cref="ChatRole"/>-style open set.</summary>
[Experimental("MEAI001")]
public readonly struct OcrTableCellKind : IEquatable<OcrTableCellKind>
{
    public OcrTableCellKind(string value);
    public static OcrTableCellKind ColumnHeader { get; }  // "columnHeader"
    public static OcrTableCellKind Content { get; }       // "content"
    public string Value { get; }
}

/// <summary>The unit for <see cref="OcrPage.Width"/> / <see cref="OcrPage.Height"/> and bounding-box
/// coordinates — a <see cref="ChatRole"/>-style open set so consumers can interpret geometry across engines.</summary>
[Experimental("MEAI001")]
public readonly struct OcrCoordinateUnit : IEquatable<OcrCoordinateUnit>
{
    public OcrCoordinateUnit(string value);
    public static OcrCoordinateUnit Pixel { get; }        // "pixel"
    public static OcrCoordinateUnit Inch { get; }         // "inch"
    public static OcrCoordinateUnit Normalized { get; }   // "normalized" ([0,1] of the page)
    public string Value { get; }
}

/// <summary>Request knobs — the <see cref="ChatOptions"/> pattern.</summary>
[Experimental("MEAI001")]
public class OcrOptions
{
    public string? ModelId { get; set; }                  // "GetChatClient(model)" analog
    public bool IncludeImages { get; set; }
    public AdditionalPropertiesDictionary? AdditionalProperties { get; set; }
    public OcrOptions Clone();                            // shallow clone (the ChatOptions.Clone pattern)
}

/// <summary>Metadata about an <see cref="IOcrClient"/> (the *ClientMetadata pattern).</summary>
[Experimental("MEAI001")]
public class OcrClientMetadata
{
    public OcrClientMetadata(string? providerName = null, Uri? providerUri = null, string? defaultModelId = null);
    public string? ProviderName { get; }
    public Uri? ProviderUri { get; }
    public string? DefaultModelId { get; }
}

Extension methods (OcrClientExtensions)

/// <summary>Convenience helpers over <see cref="IOcrClient"/>.</summary>
[Experimental("MEAI001")]
public static class OcrClientExtensions
{
    public static TService? GetService<TService>(this IOcrClient client, object? serviceKey = null);

    // Extract from an in-memory DataContent (unary + streaming twin).
    public static Task<OcrResult> ExtractAsync(
        this IOcrClient client, DataContent document, OcrOptions? options = null,
        CancellationToken cancellationToken = default);
    public static IAsyncEnumerable<OcrResponseUpdate> ExtractStreamingAsync(
        this IOcrClient client, DataContent document, OcrOptions? options = null,
        CancellationToken cancellationToken = default);

    // Extract from a UriContent. Handles self-contained data: URIs; throws NotSupportedException for
    // file:/http(s) (whether to download vs. hand the URL to the engine is a deliberate non-decision).
    public static Task<OcrResult> ExtractAsync(
        this IOcrClient client, UriContent document, OcrOptions? options = null,
        CancellationToken cancellationToken = default);
    public static IAsyncEnumerable<OcrResponseUpdate> ExtractStreamingAsync(
        this IOcrClient client, UriContent document, OcrOptions? options = null,
        CancellationToken cancellationToken = default);

    // Explicit, opt-in remote downloader: fetches http(s) bytes with a caller-supplied HttpClient
    // (caller owns handlers/auth/timeouts/lifetime), inlines data: URIs, then extracts.
    public static Task<OcrResult> ExtractFromUriAsync(
        this IOcrClient client, UriContent document, HttpClient httpClient, OcrOptions? options = null,
        CancellationToken cancellationToken = default);
}

Delegating base, builder, middleware, and DI

All Microsoft.Extensions.AI capabilities ship the same five-layer shape:
interface → Delegating<Cap> → <Cap>Builder → Add<Cap> (returns the builder) → .Use*()
middleware, with one composition primitive, Builder Use(Func<T, IServiceProvider, T>). IOcrClient
mirrors it exactly.

// ---- Microsoft.Extensions.AI.Abstractions ----

/// <summary>Optional base for an <see cref="IOcrClient"/> that passes calls through to an inner instance.</summary>
[Experimental("MEAI001")]
public class DelegatingOcrClient : IOcrClient
{
    protected DelegatingOcrClient(IOcrClient innerClient);
    protected IOcrClient InnerClient { get; }

    public void Dispose();
    public virtual Task<OcrResult> ExtractAsync(
        Stream document, string mediaType, OcrOptions? options = null,
        CancellationToken cancellationToken = default);
    public virtual IAsyncEnumerable<OcrResponseUpdate> ExtractStreamingAsync(
        Stream document, string mediaType, OcrOptions? options = null,
        CancellationToken cancellationToken = default);
    public virtual object? GetService(Type serviceType, object? serviceKey = null);
    protected virtual void Dispose(bool disposing);
}

// ---- Microsoft.Extensions.AI ----

[Experimental("MEAI001")]
public sealed class OcrClientBuilder
{
    public OcrClientBuilder(IOcrClient innerClient);
    public OcrClientBuilder(Func<IServiceProvider, IOcrClient> innerClientFactory);
    public IOcrClient Build(IServiceProvider? services = null);                     // first .Use is outermost
    public OcrClientBuilder Use(Func<IOcrClient, IOcrClient> clientFactory);
    public OcrClientBuilder Use(Func<IOcrClient, IServiceProvider, IOcrClient> clientFactory);  // THE primitive
}

[Experimental("MEAI001")] public class LoggingOcrClient : DelegatingOcrClient { }          // logging middleware
[Experimental("MEAI001")] public sealed class OpenTelemetryOcrClient : DelegatingOcrClient { } // OTel middleware
[Experimental("MEAI001")] public sealed class ConfigureOptionsOcrClient : DelegatingOcrClient { } // options middleware

[Experimental("MEAI001")]
public static class OcrClientBuilderOcrClientExtensions
{
    public static OcrClientBuilder AsBuilder(this IOcrClient innerClient);
}

[Experimental("MEAI001")]
public static class LoggingOcrClientBuilderExtensions
{
    public static OcrClientBuilder UseLogging(
        this OcrClientBuilder builder, ILoggerFactory? loggerFactory = null, Action<LoggingOcrClient>? configure = null);
}

[Experimental("MEAI001")]
public static class OpenTelemetryOcrClientBuilderExtensions
{
    public static OcrClientBuilder UseOpenTelemetry(
        this OcrClientBuilder builder, ILoggerFactory? loggerFactory = null, string? sourceName = null,
        Action<OpenTelemetryOcrClient>? configure = null);
}

[Experimental("MEAI001")]
public static class ConfigureOptionsOcrClientBuilderExtensions
{
    public static OcrClientBuilder ConfigureOptions(this OcrClientBuilder builder, Action<OcrOptions> configure);
}

[Experimental("MEAI001")]
public static class OcrClientBuilderServiceCollectionExtensions
{
    public static OcrClientBuilder AddOcrClient(
        this IServiceCollection serviceCollection, IOcrClient innerClient,
        ServiceLifetime lifetime = ServiceLifetime.Singleton);
    public static OcrClientBuilder AddOcrClient(
        this IServiceCollection serviceCollection, Func<IServiceProvider, IOcrClient> innerClientFactory,
        ServiceLifetime lifetime = ServiceLifetime.Singleton);
    public static OcrClientBuilder AddKeyedOcrClient(
        this IServiceCollection serviceCollection, object? serviceKey, IOcrClient innerClient,
        ServiceLifetime lifetime = ServiceLifetime.Singleton);
    public static OcrClientBuilder AddKeyedOcrClient(
        this IServiceCollection serviceCollection, object? serviceKey, Func<IServiceProvider, IOcrClient> innerClientFactory,
        ServiceLifetime lifetime = ServiceLifetime.Singleton);
}

IOcrClient ships logging, OpenTelemetry, and configure-options middleware in v1, matching the
ISpeechToTextClient template. It does not ship a built-in retry client: no Microsoft.Extensions.AI
capability does, because resilience belongs in the HTTP pipeline
(Microsoft.Extensions.Http.Resilience). The .Use(...) primitive still lets a consumer wrap a custom
retry or cache decorator when they want one.

API Usage

Every snippet below is drawn from the runnable demo (iocrclient-demo), which exercises this exact
surface across four engines and a MEDI RAG pipeline.

One interface, four engines — the payoff. Vision LLM, Mistral OCR, Azure Document Intelligence, and
Azure Content Understanding each speak a completely different wire protocol. Behind IOcrClient they are
IOcrClient; the consumer loop never changes when you add or swap a provider.

var clients = new (string Name, IOcrClient Client)[]
{
    ("vision-llm",                  new VisionLlmOcrClient(chatClient)),
    ("mistral-ocr",                 new FoundryMistralOcrClient(foundryEndpoint, cred)),
    ("azure-document-intelligence", new AzureDocumentIntelligenceClient(diEndpoint, cred)),
    ("azure-content-understanding", new ContentUnderstandingClient(cuEndpoint, cred)),
};

byte[] bytes = await File.ReadAllBytesAsync("report.pdf");
foreach (var (name, client) in clients)
{
    using (client)
    {
        using var stream = new MemoryStream(bytes, writable: false);
        OcrResult r = await client.ExtractAsync(stream, "application/pdf");   // identical for every engine

        int tables = r.Pages.Sum(p => p.Tables.Count);
        Console.WriteLine($"{name}: {r.Pages.Count} pages, {tables} tables, {r.Markdown.Length} chars");
    }
}

Stream pages as they finish — the IChatClient.GetStreamingResponseAsync twin. Each
OcrResponseUpdate carries a page as it is parsed (plus progress + usage), so a RAG pipeline can chunk and
embed early pages while later pages are still being OCR'd; ToOcrResultAsync reduces the stream back to the
same OcrResult the unary call would return.

await foreach (OcrResponseUpdate update in client.ExtractStreamingAsync(stream, "application/pdf"))
{
    if (update.Page is { } page)
        Console.WriteLine($"page {page.PageNumber}/{update.TotalPages}: {page.Markdown.Length} chars");
}

// …or reduce the whole stream back into one OcrResult (the ToChatResponseAsync pattern):
OcrResult full = await client.ExtractStreamingAsync(stream, "application/pdf").ToOcrResultAsync();

Compose middleware with the builder — the same shape as ChatClientBuilder: you compose a client,
you don't set flags.

IOcrClient ocr = new FoundryMistralOcrClient(endpoint, cred)
    .AsBuilder()
    .UseOpenTelemetry(loggerFactory)
    .UseLogging(loggerFactory)
    .Build();

Register with dependency injection — the consumer depends only on IOcrClient; swap the engine
without touching downstream code.

services.AddOcrClient(sp => new FoundryMistralOcrClient(endpoint, new DefaultAzureCredential()))
        .UseOpenTelemetry()
        .UseLogging();

// Later, swap the engine on one line — nothing downstream changes:
services.AddOcrClient(sp => new AzureDocumentIntelligenceClient(diEndpoint, cred)).UseLogging();

Configure per-request options through the pipeline (for example, a local Ollama GLM-OCR engine that
needs a task-prefix prompt injected even when the caller passes no options):

IOcrClient ocr = engine
    .AsBuilder()
    .ConfigureOptions(o => o.ModelId ??= "mistral-ocr-4-0")
    .Build();

Bridge into a MEDI ingestion / RAG pipeline. A hosted document-AI service is a reader (the
LlamaParse-as-reader shape). One provider-agnostic reader composes any IOcrClient:

public sealed class OcrDocumentReader(IOcrClient ocr, OcrOptions? options = null) : IngestionDocumentReader
{
    public override async Task<IngestionDocument> ReadAsync(
        Stream source, string identifier, string mediaType, CancellationToken ct = default)
    {
        OcrResult r = await ocr.ExtractAsync(source, mediaType, options, ct);
        // map r.Pages -> IngestionDocument elements, stamping page/region/confidence/model metadata
    }
}

// Usage: one section per OCR page, each stamped with its 1-based PageNumber for downstream chunking.
var reader = new OcrDocumentReader(ocr);
IngestionDocument doc = await reader.ReadAsync(fileStream, "report.pdf", "application/pdf");

Extract from a remote URI (opt-in download). ExtractAsync(UriContent) never touches the network;
ExtractFromUriAsync is the explicit counterpart that fetches http(s) bytes with a caller-owned
HttpClient:

using var http = new HttpClient();
OcrResult r = await ocr.ExtractFromUriAsync(new UriContent(url, "application/pdf"), http);

Alternative Designs

1. Reuse IChatClient with multimodal content + a prompt. Rejected: loses native
   tables/bbox/confidence, is nondeterministic and token-expensive, and cannot represent non-chat engines
   (Azure DI, CU, local ONNX) at all. A vision LLM is supported as one provider behind IOcrClient
   (VisionLlmOcrClient), not as the contract. The prototype demonstrates this: three of four real
   engines use no IChatClient.
2. A flag on the reader (the VisionOnly approach). Rejected: a model choice hardened into a reader
   mode; temporal coupling; not decoratable (no middleware); does not generalize across engines.
3. A transport-parameterized single class (new MistralOcrClient(isFoundry: true)). Rejected:
   smuggles host branching into the type; breaks composition. Follow the OpenAIClient/AzureOpenAIClient
   precedent: the interface is portable; concrete classes split by host where the host leaks.
4. One interface for OCR and field extraction (a flag/overload). Rejected: different output
   contract; modeled as the sibling IDocumentAnalysisClient peer (the STT/TTS precedent). See Related
   and future work.

Validated design decisions (resolved by the prototype + sources)

• Geometry = typed polygon; table = structured cells. Azure DI BoundingRegion.Polygon is a
  possibly rotation-skewed quad; an axis-aligned rect would be lossy. Resolved with
  OcrBoundingRegion.Polygon as IReadOnlyList<OcrPoint> (populated natively by DI, by FromRectangle
  for Mistral's rect, reused for field grounding); GetBounds() returns the OcrBoundingBox struct for
  coarse filters. Tables resolve to OcrTable{RowCount, ColumnCount, Cells?, MarkdownRepresentation?}
  (DI cells primary, Mistral markdown fallback). The prototype demonstrates this: the DI polygon and
  the Mistral rect→quad both flow losslessly into the same reader metadata.
  (Sources: Azure/azure-sdk-for-net DocumentTable/DocumentTableCell/BoundingRegion; Mistral OCR schema.)
• Typed OcrPoint over a flat float[]. The earlier open question (flat IReadOnlyList<float> vs a
  point shape) is now resolved: a typed OcrPoint list makes an odd/empty coordinate count
  unrepresentable and reads correctly without documentation, while still carrying DI's 4-vertex quad
  without loss.
• 1-based OcrPage.PageNumber. Page identity matches how documents, Azure DI, and downstream
  provenance number pages (1-based), removing the off-by-one that a 0-based Index forced on adapters.
• Unary ExtractAsync + streaming ExtractStreamingAsync (updated in API review; IProgress retired).
  The proposal originally shipped unary-only with an IProgress<OcrProgress> hook and deferred streaming.
  API review aligned it to the family instead: every sibling ships a streaming twin
  (IChatClient.GetStreamingResponseAsync, ISpeechToTextClient.GetStreamingTextAsync), and on
  netstandard2.0 / net462 there are no default interface methods, so streaming cannot be added later
  without a new interface — it has to ship now. ExtractStreamingAsync yields one OcrResponseUpdate per
  page as it finishes (progress fields PagesProcessed / TotalPages / Status ride on the update,
  replacing OcrProgress), letting large-document RAG chunk/embed early pages while later pages are still
  parsing; OcrResponseUpdateExtensions.ToOcrResultAsync reduces the stream back to a single OcrResult
  (the ToChatResponseAsync pattern). The prototype demonstrates this: each engine surfaces per-page
  updates and the reducer reassembles the same OcrResult as the unary call.
• Home / process. Lives in Microsoft.Extensions.AI.Abstractions, namespace Microsoft.Extensions.AI,
  [Experimental("MEAI001")] from day one, area-ai, full stack (abstractions +
  Delegating/Logging/OpenTelemetry/ConfigureOptions middleware + DI) in one PR. First mover: no existing
  OCR/IOcrClient issue in dotnet/extensions.

Open questions for API review

• Whether .UseDistributedCache(...) (caching: OCR is expensive) ships in v1. Logging, OpenTelemetry,
  and configure-options middleware already ship, matching the ISpeechToTextClient template.
• Granularity of OcrOptions — IncludeImages is a single bool today; API review may prefer a richer
  extraction-scope knob or leave it to AdditionalProperties.
• Naming of the shared region primitive if grounding is later reused by other capabilities: keep the
  OCR-prefixed OcrBoundingRegion (lowest churn) vs. promote to a capability-neutral BoundingRegion.
  Recommendation: keep the prefix for v1; don't pre-generalize.

Risks

• Shape creep. Keep IOcrClient focused on OCR / document extraction. Field extraction stays in the
  sibling IDocumentAnalysisClient proposal (not this PR).
• Middleware scope. Logging, OpenTelemetry, and configure-options ship in v1, validated in the
  prototype and matching the ISpeechToTextClient template. Retry is intentionally not a built-in
  client: no Microsoft.Extensions.AI capability ships per-capability retry, since resilience belongs in
  the HTTP pipeline (Microsoft.Extensions.Http.Resilience); the .Use(...) primitive can still wrap a
  custom retry decorator. Distributed caching can follow the same builder shape, but API review should
  decide whether it ships in v1.
• Mutable result lists. OcrPage.Tables/Blocks/Images are settable to let providers populate them
  incrementally; consumers treat the returned result as read-only. API review may prefer init-only /
  ctor-set here (constrained by the netstandard2.0 / net462 targets, which rule out required/DIMs).

Related and future work

Future sibling: IDocumentAnalysisClient (separate proposal)

Field extraction (a document + a schema/analyzer → typed fields with confidence + grounding) is a
categorically different output contract than OCR→markdown, exposed by multiple providers (Azure
DI custom models, Content Understanding). It is best modeled as a peer interface, not a flag/overload
on IOcrClient — the same call M.E.AI made splitting ISpeechToTextClient / ITextToSpeechClient. Its
grounding would reuse the SAME OcrBoundingRegion polygon and compose via the SAME builder, which is why
this PR ships the shared region primitive with IOcrClient. It carries its own harder design debates
(schema/analyzer-id representation, the typed-field value model, lifecycle) and its own provider matrix,
so it is deliberately out of scope for this PR and will be its own proposal. Sketch (illustrative only):

[Experimental("MEAI001")]
public interface IDocumentAnalysisClient
{
    Task<DocumentAnalysisResult> AnalyzeAsync(
        Stream document, string mediaType, DocumentAnalysisOptions options,
        CancellationToken cancellationToken = default);
    object? GetService(Type serviceType, object? serviceKey = null);
}

public sealed class DocumentField
{
    public string Name { get; }
    public object? Value { get; set; }
    public string? ValueType { get; set; }
    public double? Confidence { get; set; }
    public OcrBoundingRegion? Grounding { get; set; }     // SHARED region primitive, reused
    public AdditionalPropertiesDictionary? AdditionalProperties { get; set; }
}

References

• CommunityToolkit/AI #3: Document Processing packages (MEDI); the VisionOnly flag this proposal replaces.
• area-data-ingestion IngestionPipeline (dotnet/extensions #7488): the downstream consumer of an OcrDocumentReader.
• M.E.AI ADRs cited for the design: STT/TTS split (dotnet/extensions #5838), capability-family precedents (IVideoGenerator dotnet/extensions #7420).
• Prototype + coverage matrix runnable offline: the public iocrclient-demo (one IOcrClient in front of four OCR engines, bridged into a MEDI RAG pipeline).

[luisquintanilla]

Consumer validation: a reference document-ingestion app now consumes IOcrClient directly from the Microsoft.Extensions.AI package end to end, which exercises the proposed surface as a real dependency rather than a copy.

The app previously carried an in-repo copy of the IOcrClient family. After packing this branch's Microsoft.Extensions.AI(.Abstractions) locally, the app deleted its in-app abstraction (~470 lines) and now depends on the package, keeping only the genuinely app-owned pieces:

• a provider client (Azure AI Foundry Mistral Document AI / OCR, keyless via DefaultAzureCredential) implementing IOcrClient;
• an app-owned UseRetry() pipeline stage (this proposal intentionally ships no retry — resilience is an HTTP-pipeline concern), composed over the package's DelegatingOcrClient + OcrClientBuilder;
• a MEDI IngestionDocumentReader bridge and a small OcrTable → markdown renderer.

The composition reads exactly like the IChatClient story: new FoundryMistralOcrClient(endpoint, credential, model).AsBuilder().UseRetry().UseLogging().Build(). Opting into the experimental surface is a one-line <NoWarn>$(NoWarn);MEAI001</NoWarn>. Builds clean (0 errors) with no local definitions of the package types remaining — confirming the abstraction is the seam, and providers + cross-cutting concerns compose around it.

Reference app branch: https://github.com/luisquintanilla/PdfAIngest/tree/feature/mistral-ocr-reader

[JohnGalt1717]

Is there a way to pass multiple images as streams with this? It looks like a single file. I.e. if you have a nasty pdf, the best way to use these models is to use pdfpig or similar and convert every page in the pdf to an image, then pass all images in for each page and have it process on the images. (normally with chunking and windowing of the bottom 1/6th of the previous page etc)

If not, is your intent to have it automatically recognize the file type and convert them? (seems like a lot of implicit overhead and optimization work...)

If not, then I guess we could just default to the windowing method, but being able to upload all of the pages as stream to process on would be better if supported.