Add support System.Buffers.IBufferWriter<T>

[kronic]

Describe the enhancement requested

Please add support System.Buffers.IBufferWriter<byte>

[adamreeve]

Hi @kronic, could you please provide some more detail on what you mean by this. Which types should implement this interface? And can you provide an example of how you would use this?

[kronic]

Hi @adamreeve. I meant the ArrowStreamWriter constructor

[CurtHagenlocher]

This could be interesting, but I wonder if for full effect it might need to be threaded through the FlatBuffers layer.

[adamreeve]

Yeah I'm not sure how feasible it would be for Flatbuffers to work with an IBufferWriter, but maybe that's not required. The Flatbuffers library currently creates a ReadOnlyMemory buffer that we write to the stream, so we could keep doing something similar.

I see there's an IBufferWriterExtensions.AsStream method in the CommunityToolkit.HighPerformance NuGet package, so you could use that to create a stream from an IBufferWriter to pass to an ArrowStreamWriter. Is that sufficient for your needs?

I did a quick test and this works:

using var originalBatch = TestData.CreateSampleRecordBatch(length: 100);

var bufferWriter = new ArrayBufferWriter<byte>(1024);
using (var stream = bufferWriter.AsStream())
{
    using var writer = new ArrowStreamWriter(stream, originalBatch.Schema);
    writer.WriteRecordBatch(originalBatch);
    writer.WriteEnd();
}

using var readStream = bufferWriter.WrittenMemory.AsStream();
using (var reader = new ArrowStreamReader(readStream))
{
    using var newBatch = reader.ReadNextRecordBatch();
    ArrowReaderVerifier.CompareBatches(originalBatch, newBatch);
}

If you'd rather not add a new dependency, the implementation is pretty simple so you could also create your own wrapper class that inherits from Stream.

[CurtHagenlocher]

The main advantage of IBufferWriter<byte> is that in principle it can reduce the number of copies made of data and remove intermediate allocations. I suspect that IBufferWriterExtensions.AsStream helps with one of these by using an underlying sequence of buffers instead of a single long buffer that needs to be reallocated when its end is reached, but it wouldn't avoid the need for an intermediate buffer to e.g. write a serialized int32. But if you really wanted to "max out" that pattern, I think you'd also want to extend it to the FlatBuffers layer -- which today uses a temporary resizable ByteBuffer whose semantics I think don't translate that wall to IBufferWriter.

[kronic]

@adamreeve I know about this. I would like to record directly without additional copies.

[adamreeve]

Given we vendor the Flatbuffers code, we could modify the builder to work with an IBufferWriter. But other projects could probably benefit from this too, and diverging from upstream Flatbuffers too much could cause problems for us later, so it would be nice to get this added to the upstream Flatbuffers project.

I think it might be possible to do this in a backwards compatible way by refactoring FlatBufferBuilder to work with an IBufferWriter, and ByteBuffer could implement IBufferWriter, or a wrapper class added to implement it.

I think the next step would be to raise this as an issue in the Flatbuffers project and see what their thoughts are. There's probably not much point in adding explicit support for IBufferWriter without Flatbuffer integration as handling of most other writes like Arrow buffers doesn't require an intermediate buffer.