[AI Evaluation] DiskBasedResultStore crashes serializing EvaluationContext with multiple AIContent items

[ypyl]

Description

DiskBasedResultStore.WriteResultsAsync() crashes with System.NotSupportedException when an EvaluationContext contains multiple AIContent items in its Contents property. This affects RetrievalEvaluator, BLEUEvaluator, GLEUEvaluator, and F1Evaluator — any evaluator whose context stores multiple text items.

The root cause: JsonUtilities+JsonContext (the source-generated JSON context in Microsoft.Extensions.AI.Evaluation.Reporting) only registers ScenarioRunResult, Dataset, and CacheEntry as serializable types. When the custom EvaluationContextConverter tries to serialize the inherited Contents property (type IReadOnlyList<AIContent>), the source-generated context has no metadata for that type.

Reproduction Steps

1. Create a file-based .NET app with the following single-file repro (requires .NET 10 SDK):

#:property TargetFramework=net10.0
#:property NoWarn=AIEVAL001
#:package Microsoft.Extensions.AI.Evaluation@10.7.0
#:package Microsoft.Extensions.AI.Evaluation.Quality@10.7.0
#:package Microsoft.Extensions.AI.Evaluation.Reporting@10.7.0
#:package Microsoft.Extensions.AI.OpenAI@10.7.0
#:package OpenAI@2.11.0

using Microsoft.Extensions.AI;
using Microsoft.Extensions.AI.Evaluation;
using Microsoft.Extensions.AI.Evaluation.Quality;
using Microsoft.Extensions.AI.Evaluation.Reporting;
using Microsoft.Extensions.AI.Evaluation.Reporting.Storage;
using OpenAI;

if (args.Length < 3)
{
    Console.Error.WriteLine("Usage: dotnet run --file repro.cs <endpoint> <model> <api-key>");
    return 1;
}

var client = new OpenAIClient(
    new System.ClientModel.ApiKeyCredential(args[2]),
    new OpenAIClientOptions { Endpoint = new Uri(args[0]) });

var chatClient = client.GetChatClient(args[1]).AsIChatClient();
var chatConfig = new ChatConfiguration(chatClient);

var evaluators = new IEvaluator[] { new RetrievalEvaluator() };

var config = DiskBasedReportingConfiguration.Create(
    storageRootPath: "./repro-results",
    evaluators: evaluators,
    chatConfiguration: chatConfig,
    enableResponseCaching: false,
    executionName: "repro");

await using var run = await config.CreateScenarioRunAsync("repro.scenario", iterationName: "1");

var messages = new List<ChatMessage>
{
    new(ChatRole.User, "How many moons does Saturn have?")
};

var response = new ChatResponse(new ChatMessage(ChatRole.Assistant, "Saturn has 146 confirmed moons."));

var context = new RetrievalEvaluatorContext(
    "Saturn has 146 confirmed moons as of 2023.",
    "Jupiter has 95 known moons.");

Console.WriteLine("Evaluating...");
var result = await run.EvaluateAsync(
    messages,
    response,
    additionalContext: [context]);

var metric = result.Metrics.Values.OfType<NumericMetric>().Single();
Console.WriteLine($"Score: {metric.Value}, Failed: {metric.Interpretation?.Failed}");

// Crash happens when DisposeAsync serializes results to disk

2. Run: dotnet run --file repro.cs <openai-endpoint> <model> <api-key>

Expected behavior

DiskBasedResultStore should successfully serialize EvaluationContext objects regardless of how many AIContent items are in the Contents list. The evaluation result should be persisted to disk without errors.

Actual behavior

Evaluation succeeds (score produced, metrics returned), but ScenarioRun.DisposeAsync() crashes when DiskBasedResultStore.WriteResultsAsync() tries to serialize the EvaluationContext:

Unhandled exception. System.NotSupportedException: JsonTypeInfo metadata for type
'System.Collections.Generic.IReadOnlyList`1[Microsoft.Extensions.AI.AIContent]' was not
provided by TypeInfoResolver of type
'[Microsoft.Extensions.AI.Evaluation.Reporting.JsonSerialization.JsonUtilities+JsonContext,
Microsoft.Extensions.AI.AIJsonUtilities+JsonContext]'. The unsupported member type is
located on type 'Microsoft.Extensions.AI.Evaluation.EvaluationContext'.
Path: $.EvaluationResult.Metrics.Context.

   at System.Text.Json.ThrowHelper.ThrowNotSupportedException_NoMetadataForType(...)
   at System.Text.Json.JsonSerializerOptions.GetTypeInfoInternal(...)
   at Microsoft.Extensions.AI.Evaluation.Reporting.JsonSerialization.EvaluationContextConverter.Write(...)
   ...
   at Microsoft.Extensions.AI.Evaluation.Reporting.Storage.DiskBasedResultStore.WriteResultsAsync(...)
   at Microsoft.Extensions.AI.Evaluation.Reporting.ScenarioRun.DisposeAsync()

Affected evaluators

Evaluator	Context Contents	Result
GroundednessEvaluator	1 TextContent	Works
EquivalenceEvaluator	1 TextContent	Works
RetrievalEvaluator	N TextContent (chunks)	Crashes
BLEUEvaluator	N TextContent (references)	Crashes
GLEUEvaluator	N TextContent (references)	Crashes
RelevanceTruthAndCompletenessEvaluator	No context	Works

Configuration

• .NET SDK 10.0.301
• Microsoft.Extensions.AI.Evaluation.Reporting 10.7.0
• Microsoft.Extensions.AI.Evaluation.Quality 10.7.0
• Windows 11 x64
• Reproduces with any OpenAI-compatible endpoint (Azure OpenAI, OpenCode, etc.)

Other information

Likely fix: add [JsonSerializable(typeof(IReadOnlyList<AIContent>))] to the JsonContext in src/Libraries/Microsoft.Extensions.AI.Evaluation.Reporting/CSharp/JsonSerialization/JsonUtilities.cs, or update EvaluationContextConverter.Write() to handle multi-item Contents without relying on source-generated metadata.

[ypyl]

Sorry, this issue was auto-created by a code agent, but the problem is valid: I am not able to store the RetrievalEvaluator results to disk.
I have checked repro.cs file and getting the error like in the report:

[ypyl]

Workaround: custom IEvaluationResultStore wrapper

Wrap DiskBasedResultStore and fall back to reflection-based System.Text.Json when the source-gen serializer crashes on multi-item EvaluationContext contents.

Key points:

• DefaultJsonTypeInfoResolver — required for reflection-based serialization in PublishTrimmed builds
• Replicate the library's TimeSpanConverter (writes TotalSeconds as double) so aieval report can read files back
• Add JsonStringEnumConverter with CamelCase for EvaluationDiagnosticSeverity and EvaluationRating
• Do not include EvaluationContextConverter in the fallback — it's the one that crashes; reflection handles the subclasses natively
• Replace DiskBasedReportingConfiguration.Create() with new ReportingConfiguration(...) passing the custom store

Full implementation: https://github.com/ypyl/ai-eval-cli/blob/master/src/AiEvalCli.Engine/FixedDiskResultStore.cs