Skip to content

parquet: define safe page-size and decompression limits #950

Description

@fallintoplace

Describe the enhancement requested

Parquet page headers provide signed 32-bit compressed and uncompressed sizes. The page reader rejects negative values but otherwise accepts positive values up to math.MaxInt32.

Several paths then resize buffers from those declarations before proving that the body fits inside the remaining column chunk or that the uncompressed size is reasonable. Streaming support reduces the allocation risk for some data-page encodings, but dictionary pages and non-streaming paths still allocate from the declared size.

Reader-side limits would provide a predictable resource boundary for corrupt or unexpectedly large files.

Proposed direction

Add reader properties for limits such as:

  • Maximum compressed page size.
  • Maximum uncompressed page size.
  • Optionally, maximum decompression ratio.

Before allocating or decoding:

  • Check the compressed size against the remaining column-chunk bytes.
  • Validate compressed and uncompressed sizes against configured limits.
  • Use checked arithmetic for level and value-region sizes.
  • Bound streaming readers to the validated page body.
  • Preserve an explicit override for workloads with unusually large valid pages.

The defaults need maintainer agreement because existing users may read pages larger than common recommendations.

Regression coverage

  • math.MaxInt32 compressed and uncompressed declarations.
  • A compressed page extending beyond its column chunk.
  • Small compressed input declaring a very large output.
  • Dictionary pages and data page versions 1 and 2.
  • Allocator accounting showing rejection before a large allocation.

Observed on current main at 750586710988bc18f7af64fd29af8e0acdb252ac.

Component(s)

Parquet

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions