Describe the enhancement requested
Parquet page headers provide signed 32-bit compressed and uncompressed sizes. The page reader rejects negative values but otherwise accepts positive values up to math.MaxInt32.
Several paths then resize buffers from those declarations before proving that the body fits inside the remaining column chunk or that the uncompressed size is reasonable. Streaming support reduces the allocation risk for some data-page encodings, but dictionary pages and non-streaming paths still allocate from the declared size.
Reader-side limits would provide a predictable resource boundary for corrupt or unexpectedly large files.
Proposed direction
Add reader properties for limits such as:
- Maximum compressed page size.
- Maximum uncompressed page size.
- Optionally, maximum decompression ratio.
Before allocating or decoding:
- Check the compressed size against the remaining column-chunk bytes.
- Validate compressed and uncompressed sizes against configured limits.
- Use checked arithmetic for level and value-region sizes.
- Bound streaming readers to the validated page body.
- Preserve an explicit override for workloads with unusually large valid pages.
The defaults need maintainer agreement because existing users may read pages larger than common recommendations.
Regression coverage
math.MaxInt32 compressed and uncompressed declarations.
- A compressed page extending beyond its column chunk.
- Small compressed input declaring a very large output.
- Dictionary pages and data page versions 1 and 2.
- Allocator accounting showing rejection before a large allocation.
Observed on current main at 750586710988bc18f7af64fd29af8e0acdb252ac.
Component(s)
Parquet
Describe the enhancement requested
Parquet page headers provide signed 32-bit compressed and uncompressed sizes. The page reader rejects negative values but otherwise accepts positive values up to
math.MaxInt32.Several paths then resize buffers from those declarations before proving that the body fits inside the remaining column chunk or that the uncompressed size is reasonable. Streaming support reduces the allocation risk for some data-page encodings, but dictionary pages and non-streaming paths still allocate from the declared size.
Reader-side limits would provide a predictable resource boundary for corrupt or unexpectedly large files.
Proposed direction
Add reader properties for limits such as:
Before allocating or decoding:
The defaults need maintainer agreement because existing users may read pages larger than common recommendations.
Regression coverage
math.MaxInt32compressed and uncompressed declarations.Observed on current
mainat750586710988bc18f7af64fd29af8e0acdb252ac.Component(s)
Parquet