RFC: Remove majority of I/O functionality and custom metadata from dataframely

[borchero]

Status Quo

Currently, dataframely provides a bunch of I/O methods, including:

• {write,sink}_{parquet,delta} on schemas and collections
• {read,scan}_{parquet,delta} on schemas and collections

The writer methods attach a serialization of the schema to the backend (e.g. to the parquet metadata). The read methods then inspect that metadata to decide whether validation is necessary. By default, i.e. if no explicit validation parameter is provided, dataframely then silently calls validation to ensure that the loaded data adheres to the schema.

Issues

Overall, these I/O functions have caused several issues that are non-trivial to solve:

• There are a number of reasons why the serialized schema stored in the metadata mismatches the current schema (for example: a new polars version or a change of constraints -- even if they become more lenient). In this case, loading the stored data via dataframely's I/O functions silently introduces a non-negligible runtime impact. In addition, warnings are printed even if there is no actual issue.
• Reading files requires reading the metadata: if the number of files is large and/or the files are stored on S3, this introduces a significant overhead because of the roundtrip time for reading the metadata. In practice, I have seen a scan_parquet on a collection take 5s even though no computation was performed.
• The presence of metadata does not properly protect the user if the number of files is greater than one: much like in a lakehouse, primary key constraints cannot be properly ensured beyond a single file. Dataframely currently opts for the more efficient but potentially incorrect path (it simply trusts the stored metadata) but changing this would introduce a significant overhead.

Proposed Way Forward

For the reasons outlined above, I propose to minimize the I/O functionality provided by dataframely. Specifically:

• Remove the functionality to serialize schemas & collections
• Remove all I/O functions from schemas
• Limit I/O functions on collections to {read,scan,write,sink}_parquet (to retain the functionality to easily store & retrieve collections)

Overall, this should limit one's ability to shoot oneself in the foot and make user code more explicit (e.g. by explicitly calling validate when reading data).

Migration Path

Naturally, this is a breaking change and will require dataframely v3. To minimize the immediate impact on users, I propose to emit deprecation warnings for all I/O functions that ought to be removed as well as for the {read,scan}_parquet on collections whenever the user does not explicitly specify validation="skip" (which will be the future functionality).

[MoritzPotthoffQC]

loading the stored data via dataframely's I/O functions silently introduces a non-negligible runtime impact

I have also run into this problem before. I agree that it would be nice to let the user decide when they need to validate loaded data and when not.

[AndreasAlbertQC]

My experience is that the schema information is effectively discarded most of the time I load data that is older than a few days. In practice, this means I validate all the time already. The only upside I am currently enjoying is collection ser/de which saves me some boilerplate. I'm in favor of your proposal :)

[brendancooley]

Move fast and break things! I use dataframely and deltalake together and usually want to control the I/O seam. I think keeping dataframely focused on validation makes sense.