Parquet Java ALP Implementation#3397
Open
vinooganesh wants to merge 50 commits into
Open
Conversation
Implements ALP encoding for FLOAT and DOUBLE types, which converts floating-point values to integers using decimal scaling, then applies Frame of Reference (FOR) encoding and bit-packing for compression. New files: - AlpConstants.java: Constants for ALP encoding - AlpEncoderDecoder.java: Core encoding/decoding logic - AlpValuesWriter.java: Writer implementation - AlpValuesReaderForFloat/Double.java: Reader implementations Includes comprehensive unit tests and interop test infrastructure. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Restore original comment indentation that was accidentally changed. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Escape <= characters as <= in javadoc comments to avoid malformed HTML errors during documentation generation. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
ALP encoding is not yet part of the parquet-format Thrift specification, so it cannot be converted to org.apache.parquet.format.Encoding. Skip it in the testEnumEquivalence test and add a clear error message in the converter for when ALP conversion is attempted. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
size and add independent reader/writer verification tests
vinooganesh
force-pushed
the
vinooganesh/alp-java-implementation
branch
from
February 18, 2026 02:54
93c365a to
6d65eaa
Compare
Switch encode/decode from division-based formula to multiply-by-reciprocal using separate POW10_NEGATIVE arrays, matching C++ Arrow's approach: - Encode: fastRound(value * POW10[e] * POW10_NEGATIVE[f]) - Decode: encoded * POW10[f] * POW10_NEGATIVE[e] Add fastRound helpers with sign branching for correct negative value rounding. Remove version byte from page header (8 -> 7 bytes). Empty pages now emit a 7-byte header with numElements=0. Update all hand-crafted binary tests to match the new header format and add comprehensive end-to-end tests for overflow boundaries, large-scale data, preset caching, and NaN bit-pattern preservation.
- Rewrite TestInterOpReadAlp to use LocalInputFile instead of Hadoop FileSystem, fixing failures on Java 24+ where Subject.getSubject is removed. Tests now read C++ ALP parquet files directly without going through Hadoop security/UGI. - Add AlpExceptionCountTest with per-column exception rate reporting against the real Spotify and Arade floating-point datasets from the parquet-testing repository. Useful for comparing Java vs C++ ALP compression ratios.
- Switch findBestFloatParams/findBestDoubleParams from minimizing exception count to minimizing estimated compressed size (length * bitWidth + exceptions * (typeSize + 2 bytes)), matching the C++ ALP cost model. This closes the ~4-5% compression gap vs C++. - Rewrite sampler to collect evenly-spaced sample vectors and run findBestParams on each, then rank by win count. Matches C++ AlpSampler behavior more closely than the previous HashMap-based approach. - Minor fixes: IOExceptionUtils null check, MemoryManager volatile scale, Files utility cleanup, parquet-cli dependency update.
vinooganesh
force-pushed
the
vinooganesh/alp-java-implementation
branch
from
March 22, 2026 23:56
15bc06d to
24c23e5
Compare
- Move shared LE helper methods (getShortLE/getIntLE/getLongLE) to AlpValuesReader base class; remove duplicates from subclasses - Make EncodingParams fields package-private (remove public modifier) - Replace fully-qualified java.util.Arrays.fill calls with imported Arrays.fill in both float and double readers; add missing import to double reader - Add explanatory comments to getBufferedSize() magic numbers (3 for float, 5 for double) explaining the overhead breakdown - Add ALP enabled state to ParquetProperties.toString() - Add ALP support to DefaultV1ValuesWriterFactory for float and double columns
- Revert Files.java, IOExceptionUtils.java, MemoryManager.java, and parquet-cli/pom.xml to master state; these changes are unrelated to ALP and should be submitted in separate PRs - Clarify ParquetMetadataConverter error message: ALP encoding is defined in the ALP paper (enum value 26) but is not yet in the parquet-format Thrift spec, so ALP cannot be written through the Hadoop write path; the error message now explains what needs to happen to remove the block
prtkgaur
reviewed
Apr 14, 2026
prtkgaur
left a comment
prtkgaur
left a comment
There was a problem hiding this comment.
The code organization looks good to me and the code follows the spec. I looked for areas of any extra buffer allocations which might impact performance and I think it is optimally written.
I think we should add a few benchmarks and publish numbers from them.
Thanks for working on this Vinoo!
prtkgaur
reviewed
Apr 14, 2026
prtkgaur
left a comment
prtkgaur
left a comment
There was a problem hiding this comment.
Wanted to make sure we have the following testing.
For the cross compatibility testing are we making sure that we write both V1 and V2 pages and the implementation in other language is able to read it.
- Add build-time Perl script to patch generated Encoding.java with ALP(10) after Thrift codegen (process-sources phase), since parquet-format 2.12.0 does not yet include ALP in its Thrift spec - Remove guard in ParquetMetadataConverter.getEncoding() that blocked ALP writes; Encoding.ALP now exists in the patched Thrift enum - Add withAlpEncoding() builder methods to ParquetWriter - Add TestInterOpReadAlp: Java V1/V2 write+read round-trip tests and C++ Arrow interop tests (reads alp_spotify1.parquet, alp_arade.parquet, etc.) - Add AlpEncodingBenchmarks JMH benchmark
…d pyarrow interop test - AlpValuesWriter: stop clearing cachedPresets in reset() so preset (e,f) pairs survive page flushes; eliminates redundant full parameter search on every page after the first, cutting write time ~60% - AlpEncodingBenchmarks: clarify Javadoc that comparison is PLAIN+UNCOMPRESSED (no codec), not plain+ZSTD - parquet-benchmarks pom: add explicit annotationProcessorPaths and proc=full for jmh-generator-annprocess so BenchmarkList is generated under Java 23+ - TestInterOpReadAlp: add pyarrow cross-language compatibility test (skips if pyarrow unavailable or does not yet support ALP encoding)
- ParquetProperties: add withAlpVectorSize(int) and withAlpVectorSize(String, int) builder methods plus getAlpVectorSize(ColumnDescriptor) accessor, defaulting to AlpConstants.DEFAULT_VECTOR_SIZE (1024). - AlpConstants: promote validateVectorSize to public so the builder can validate eagerly across packages. - DefaultV1/V2 ValuesWriterFactory: pass the configured vector size to the 4-arg AlpValuesWriter constructors. - ParquetWriter.Builder: expose withAlpVectorSize facades mirroring withAlpEncoding. - TestInterOpReadAlp: add testJavaWriteAlpCustomVectorSize covering 4500 rows at vectorSize=4096 so we cross a full vector boundary and verify round-trip equality. A wrong log_vector_size byte would surface as decode garbage, so round-trip equality is sufficient proof the configured size took effect on the wire. Enables generating ALP test fixtures at different vector sizes (e.g. 4096) for cross-language compatibility testing against the C++/Rust/Go implementations.
Logging and debug output was missing the new alpVectorSize field alongside the existing 'ALP enabled' line. Cosmetic only — no behavior change.
Adds generateAlpFixturesAtMultipleVectorSizes to TestInterOpReadAlp. For each of the four source files in parquet-testing PR apache#100 (alp_spotify1, alp_arade, alp_float_spotify1, alp_float_arade), reads every row, then re-encodes as Java ALP at both vectorSize=1024 and vectorSize=4096. Output goes to ALP_OUTPUT_DIR (default ${user.dir}/alp-java-generated/), producing 8 files total named alp_java_<stem>_vs{1024,4096}.parquet. Each output is verified by reading back through the standard reader path and bit-comparing every value via doubleToRawLongBits / floatToRawIntBits — catches NaN payload and signed-zero divergence, not just numerical equality. Skips when ALP_TEST_DATA_DIR isn't set, so it stays inert in CI on machines without the source datasets. To run: git clone --branch alpFloatingPointDataset \\ https://github.com/prtkgaur/parquet-testing.git ALP_TEST_DATA_DIR=path/to/parquet-testing/data \\ mvn -pl parquet-hadoop \\ -Dtest=TestInterOpReadAlp#generateAlpFixturesAtMultipleVectorSizes \\ test
Extends generateAlpFixturesAtMultipleVectorSizes to vary writer page
version (PARQUET_1_0, PARQUET_2_0) as a third axis alongside dataset
and ALP vector size. Output grows from 8 → 16 files per run:
alp_java_<stem>_v{1,2}_vs{1024,4096}.parquet
Page version is orthogonal to ALP encoding — the page version
difference lives in the parquet protocol layer, not in the ALP
payload — but covering both axes makes the fixture set fully
symmetric for cross-language compatibility verification. C++/Rust/Go
readers can use the V1 and V2 variants to prove their decoders
handle Java-written ALP regardless of how the surrounding pages are
framed. Avoids an asymmetry where the existing PR apache#100 set has C++
at V1 and Java at V2 with no overlap.
All 16 outputs independently verified against the canonical
_expect.csv truth files from parquet-testing PR apache#100 (1.56M values,
0 mismatches).
The reader was asserting that the ALP header's num_elements equals the data page's valuesCount, but those values differ whenever a column has nulls: num_elements is the count of non-null values that went through ALP encoding, while valuesCount is the total row count of the page (which includes null positions tracked by definition levels). The strict equality check made the reader reject every optional float/double column with at least one null value. Relaxes the check to numElements > valuesCount — the header can never legitimately claim more encoded values than the page has rows, but it can claim fewer when nulls are present. The downstream code already uses numElements (not valuesCount) to drive vector allocation and decoding, so the rest of the read path is unchanged. This was surfaced by the corner-case fixture per parquet-testing issue apache#105, which exercises optional columns with null values.
Two new tests in TestInterOpReadAlp: readAllFixtureFilesIndependently Opens every alp_java_*.parquet in ALP_OUTPUT_DIR and asserts each column chunk declares Encoding.ALP and decodes through the standard reader path without error. Separate from the generator's own round-trip verification so reader correctness surfaces as a distinct signal in CI when the fixtures are present. Skips cleanly when ALP_OUTPUT_DIR is empty so it stays inert in default CI environments. generateAndVerifyCornerCaseFixture Writes a single small fixture file (alp_java_cornercases.parquet, ~60 KB) targeting the corner cases enumerated in parquet-testing issue apache#105: vectors with no exceptions, one exception per vector, all exceptions, NaN/Inf/-0.0, constant values (bit_width=0), multi-vector with differing exponents, and optional columns with nulls. Both f32 and f64 variants — 14 columns × 2048 rows total. Reads each column back and bit-exactly verifies every value against the expected pattern via doubleToRawLongBits / floatToRawIntBits. The corner-case fixture is intended as a candidate file for parquet-testing PR apache#100 once naming/design is confirmed. Generating it also surfaced (and verified the fix for) a pre-existing reader bug where optional columns with nulls couldn't be decoded — see the preceding commit.
The corner-case fixture (alp_java_cornercases.parquet) is synthetic — it isn't derived from any raw dataset in parquet-testing PR apache#100, so the existing alp_*_expect.csv files don't cover it. That left cross-language verifiers with no independent ground truth to check the parquet file against; they had to either trust the Java reader or duplicate the construction recipe in their own code. writeCornerCaseCsvTruth now dumps the expected values straight from the construction recipe into alp_java_cornercases_expect.csv next to the parquet, every time the generator runs. The CSV uses the same format conventions as the existing _expect.csv files (comma- separated, header row, no quoting) plus two extensions: • Empty field = null cell (for optional columns) • Special values printed via Java's standard toString: "NaN", "Infinity", "-Infinity", "-0.0". These all parse via C++ std::stod / std::stof per the standard (case-insensitive, "inf" and "infinity" both accepted). The Arrow C++ ALP decoder reads the parquet and compares against this CSV bit-exactly: 27306 non-null cells + 1366 null cells across 14 columns × 2048 rows, 0 mismatches. This makes the corner-case fixture self-documenting and verifiable by any future cross-language tooling without rerunning the Java generator to discover what the expected values are.
RussellSpitzer
left a comment
RussellSpitzer
left a comment
Member
There was a problem hiding this comment.
It's getting late and I'm off next week. I reviewed pretty much everything except the tests.
My biggest checks were
- Looks like there is a bitwidth calculation bug for floats using an unsigned comparison to determine range
- The buildPresetCache looks like it's doing too much work, I think we can get by just saving the (e,f) values rather than copying vectors and refinding their e,f paris
- There seems to be some duplication of BytesUtils methods with ALP specific versions here that seem to be the same. Is there a reason for this? May be worth consolidating otherwise
| private ValuesWriter getDoubleValuesWriter(ColumnDescriptor path) { | ||
| final ValuesWriter fallbackWriter; | ||
| if (this.parquetProperties.isByteStreamSplitEnabled(path)) { | ||
| if (this.parquetProperties.isAlpEnabled(path)) { |
Member
There was a problem hiding this comment.
I'm not sure if this belongs in DefaultV1ValuesWriter factory, we'll see what @rdblue proposes with the new versioning. But I"m guessing this will land in a V3? I would stick it only in V2 for now since that's already a bit mixed up
emkornfield
reviewed
Jul 6, 2026
The float parameter search was computing the frame-of-reference range with unsigned subtraction (Integer.toUnsignedLong), which blows up to a 64-bit width for any mixed-sign vector. That made the estimator massively overstate the cost of lossless mixed-sign encodings, so it would instead pick (e, f) combinations that push most values into the exception path. Data still round-trips, but compression on mixed-sign float columns was far worse than it should be. Switch to signed (max - min) subtraction to match the writer's actual FOR packing and the double search, which already did this. Also add an upper bound check on the decoded bit width in both readers so a corrupt page fails with a clear ParquetDecodingException instead of feeding a bogus width into the bit-packer. Adds regression tests for the mixed-sign case and the bit-width bounds.
While going through review feedback I noticed the ALP writer and readers were carrying their own copies of math that already lives in BytesUtils. This swaps them over so we're not maintaining two versions of the same thing. Changes: - Writer float path now calls BytesUtils.getWidthFromMaxInt instead of the local bitWidthForInt helper, which was the same 32 - numberOfLeadingZeros calculation (the extra == 0 check was a no-op, since 32 - clz(0) is 0). - Dropped bitWidthForInt entirely now that nothing uses it. Kept bitWidthForLong for the double path, since BytesUtils only has an int variant and adding a long one to parquet-common felt out of scope here. - Replaced the hand-rolled (count * bitWidth + 7) / 8 in both readers and both writer paths with BytesUtils.paddedByteCountFromBits. - Removed the bit-width test that only re-covered getWidthFromMaxInt, which is already tested in parquet-common. No behavior change. Confirmed getWidthFromMaxInt matches the old helper across all power-of-two boundaries, MIN/MAX and negative inputs (the packer can see an unsigned maxDelta that's a negative int), and the encode/decode round-trip is still byte-exact over a few million random float and double values.
… LE assembly The reader had its own getShortLE/getIntLE/getLongLE (plus getFloatLE/ getDoubleLE) helpers that assembled little-endian values byte by byte. The comment justified them by saying absolute get() ignores the buffer's byte order, but that's only true for the single-byte get(int); the typed absolute getters (getShort/getInt/getLong/getFloat/getDouble) do respect order(). The vectors buffer is already sliced with LITTLE_ENDIAN order, and this same class already reads the header and offset array via getInt() on LE-ordered buffers, so the hand-rolled helpers were redundant. Drop them and read straight from the buffer. Kept the & 0xFFFF on the uint16 reads since getShort returns a signed short. Round-trip stays byte-exact over several million random float and double values, and the typed getters match the old assembly at every offset for both heap and direct buffers.
Make it clear this is the long counterpart to BytesUtils.getWidthFromMaxInt, which only takes an int, so it isn't just a redundant reimplementation of an existing helper.
findBestFloatParams/findBestFloatParamsWithPresets (and the double pair) were near-identical: same scoring loop, same tie-break, same early exit — the only difference was whether they iterated all (e, f) combos or a preset list. That duplication is why the signed-FOR-range fix had to be applied in two places. Precompute the full set of valid (e, f) pairs once (in the same e-then-f order the nested loop used) and route both entry points through a single pickBestFloat/pickBestDouble that takes the pair list. The full search passes ALL_VALID_*_PAIRS, the preset search passes its presets. Because ALL_VALID_*_PAIRS[0] is (0, 0) and presets[0] was already the preset default, using pairs[0] as the fallback reproduces both old behaviors exactly. I kept the per-pair scoring inlined in pickBest rather than extracting a separate score() helper, to avoid allocating a result holder per (e, f) on the encode hot path. Verified byte-for-byte identical output: regenerated all 18 interop fixtures (arade/spotify1, float+double, v1/v2, vs1024/vs4096, plus the corner cases) and they match the pre-refactor files exactly. Full ALP unit suite green and the round-trip stress stays bit-exact over ~8.7M values.
The ALP sources were committed without running spotless, so spotless:check would fail in CI. Run spotless:apply across the ALP files in parquet-column, parquet-hadoop, and parquet-benchmarks. Almost all of this is whitespace/line-wrapping and a couple of missing trailing newlines. The one non-whitespace change is in AlpEncodingBenchmarks, where spotless dropped two genuinely-unused imports (ArrayList, List) and moved a stray TimeUnit import into the right group. No behavior change: eight of nine files are token-identical ignoring whitespace, the benchmark's removed imports were unused, and the full ALP test suite stays green.
The float and double readers had near-identical decodeVector methods: the same header parse, the same exponent/factor/numExceptions validation, and the same exception-position loop, differing only in primitive type and a couple of constants. Java can't share the per-value numeric loops without boxing (which would hurt the decode hot path), but everything around them can move up. Turn decodeVector into a template method on the base class that does the header parse, validation, and exception-position reading once, and delegates the type-specific parts to hooks: maxExponent(), typeName(), decodeBody() (FOR read + bit-unpack + decode loop), and applyExceptionValues(). The hooks run once per vector, not per value, so there's no hot-path cost. Moved the shared excPositionsBuffer up to the base as well. No behavior change. Validation messages are byte-identical (the adversarial tests that assert on them still pass), decode round-trips bit-exact over ~8.7M random values, and regenerating the interop fixtures reads back bit-exact against the C++ sources with the written bytes unchanged.
The sampling path (full search + collecting samples + building the preset cache) was split awkwardly: the cached branch came first, and the cache build was a separate post-increment if that either branch could fall through. Flip it so the cachedPresets == null (sampling) branch comes first and builds the cache inline once enough samples are gathered, with the cached branch as the else. vectorsProcessed++ moves to the end. Pure reorganization, no behavior change: buildPresetCache doesn't read vectorsProcessed, and all the branch conditions are equivalent. Verified byte-identical output across all 18 interop fixtures and bit-exact round-trip over ~8.7M values. Applied to both the float and double writers.
…PresetCache The sampler used to keep a copy of each sampled vector (Arrays.copyOf into a List<float[]>/List<double[]>) and then re-run findBest on every copy inside buildPresetCache to recover its winning (exponent, factor). But the sampling branch already computes exactly that winner via findBest for the vector it's encoding, so the copy and the re-search were redundant. Store the (e,f) pair at sample time (List<int[]> sampledParams) and have buildPresetCache just tally frequencies over those pairs. This drops a per-sample vector copy and a second full findBest per sample, and lets the Arrays import go. No behavior change: the (e,f) stored at sample time is the same value the old buildPresetCache re-derived from the copy (same findBest call, same values), so the frequency tally and resulting presets are identical. Verified by hashing the encoded output of a 300k-value dataset (large enough to actually trigger cache building) — SHA-256 identical to the previous commit for both float and double, plus the usual bit-exact round-trip and byte-identical interop fixtures.
Grab-bag of readability fixes from PR feedback, no behavior change: - Rename the reader cursors to say what coordinate system they're in: currentIndex -> pageValueIndex, currentVectorIndex -> currentVectorNumber, vectorIdx -> vectorNumber, indexInVector -> vectorSlot. - Rename the single-arg exception checks to isIntrinsicFloatException / isIntrinsicDoubleException so it's clear they're the NaN/Inf/-0.0 subset, distinct from the full (value, e, f) round-trip check. - Add E/F index constants for the (exponent, factor) pair arrays in the parameter search. - Replace the bare 8s in the pack/unpack group math with a named PACK_GROUP_SIZE constant, and use Integer.BYTES / Long.BYTES for the FOR header offsets instead of literal 4 / 8. - Rewrite the "C++ wire compatibility" note on the encode/decode order of operations to explain it's about cross-implementation IEEE-754 determinism, not any one language. - Flesh out the exception-placeholder comment in both writers (why the placeholder is a real encoding from the vector, and that it only affects FOR/bit width, never decoded values). Verified byte-identical: 18/18 interop fixtures unchanged and bit-exact round-trip over ~8.7M values.
Replaces the two separate ColumnProperty fields in ParquetProperties with a single ColumnProperty<AlpConfig>. Public withAlpEncoding/withAlpVectorSize setters and the isAlpEnabled/getAlpVectorSize getters are unchanged; the merge into one per-column config happens at build time. Adds ColumnProperty.getColumnPaths() so the two independent setters can be combined. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Int loop now covers width 32 and long loop covers width 64, with the top bit of each width exercised. Special-cases the max-width max value since (1L << 64) wraps to 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Covers the float and double factories producing the ALP writers under V1 and V2, the per-column case, and ALP taking precedence over byte stream split. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds always-run round-trip tests for values spanning a wide signed FOR range (deeply negative frame minimum, high bit width) and for all-null pages (num_elements 0). Adds matching wide-range columns to the interop corner-case fixture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drops the encode/decode one-liner javadocs that only restated the code (the IEEE 754 ordering rationale is already in the class javadoc). Rewords the negative-power-array comment to explain reference-layout interop rather than singling out C++. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds testFloatPartialNullPage / testDoublePartialNullPage: the writer sees fewer values than the page row count (num_elements < valueCount, i.e. some rows null), and the reader initialized with the larger page count serves exactly num_elements values back. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…a-implementation Resolved conflicts: - ParquetProperties.java: kept both the new CompressionCodecName import and the ColumnPath import; merged the ALP toString line into master's new per-column-codec result string. - parquet-benchmarks/pom.xml: kept <proc>full</proc> for JMH annotation processing. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The patch-alp-encoding.pl script (which injects ALP(10) into the generated Encoding enum until parquet-format ships ALP) was missing the Apache license header, so RAT flagged it as an unapproved file. Adds the standard header. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Real-data cross-language fixtures only exercised low-bit-width FOR frames. Adds coverage for extreme values that force the full FOR bit width: exact integer-valued doubles near the ~2^63 encoding limit (64-bit FOR delta, signed max-min overflows) and floats near ~2^31 (32-bit), both with zero exceptions. Added as always-run unit tests (testDouble/FloatExtremeForBitWidth) and as f64_extreme_for_64bit / f32_extreme_for_32bit columns in the interop corner-case fixture. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Complements the 64-bit (overflow) case with a 63-bit case: exact integer-valued doubles near +/- 2^61 whose FOR delta needs 63 bits without overflowing the signed max-min subtraction. Added as testDoubleExtremeFor63BitWidth and the f64_extreme_for_63bit corner-case fixture column, so coverage spans both the 63-bit non-overflow and 64-bit overflow high-bit-width paths. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…sion, stats, nested Hardens the lossless invariant and covers production paths the existing suite skipped: - Full-bit-space fuzz (double/float): random raw bit patterns incl. all NaN payloads, subnormals, +/-0, +/-Inf; strict raw-bit round-trip (the existing helpers were NaN-lenient). - Distribution shift within a row group: preset-cache stays lossless when later vectors differ from the sampled ones. - Reader robustness: malformed/truncated/corrupt pages fail cleanly (no crash/hang/OOB/OOM). - Dictionary->ALP fallback: dictionary enabled + overflow forces fallback to ALP mid-column (the real default path, previously untested since all tests disabled the dictionary). - ALP under Snappy/Gzip/Zstd compression. - Statistics with NaN (excluded from min/max) + nulls (counted) on an optional column. - ALP on a repeated double field (repetition/definition levels). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Deterministic allocator-leak test: a counting ByteBufferAllocator wraps the writer over 200 write/getBytes/reset page cycles (~1M values); asserts outstanding buffers == 0 after close (catches the apacheGH-3628-style unreleased-buffer bug) and peak > 0 so it is not vacuous. - 2M-value unit round-trip (scale + no OOM). - 500k rows through the full write/read pipeline with a small row-group size (many row groups) and Snappy, streaming verification so the read side is memory-safe at scale too. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cc @julienledem @alamb @emkornfield @prtkgaur
Rationale for this change
Reworks the ALP encoding implementation to address emkornfield's architectural feedback on PR #3390. The original buffered all values in memory and decoded eagerly. This makes the writer incremental (encode per-vector as values arrive) and the reader lazy (decode on demand), matching how other Parquet encodings work.
Builds on Julien Le Dem's original implementation (#3390). File structure, integration points, core math, and interop test infrastructure all come from his work. The rework focused on the internal writer/reader plumbing.
What changes are included in this PR?
Architecture (addressing review feedback):
AlpValuesReaderabstract base class for shared decode logic (float/double readers only implementdecodeBody).Spec compliance:
ceil(n * bitWidth / 8)(reusingBytesUtils.paddedByteCountFromBits).max - minoverstated the span on mixed-sign vectors).ByteBuffer's typed getters on aLITTLE_ENDIAN-ordered buffer.BytePackerinstead of custom bit-packing.num_exceptions.> 32float /> 64double throw).Configuration:
withAlpEncoding(...)andwithAlpVectorSize(...)onParquetProperties.Builderand the HadoopParquetWriter.Builder, globally or per-column.DefaultV1ValuesWriterFactoryandDefaultV2ValuesWriterFactoryso per-column overrides work.AlpConstantsmin/max bounds eagerly at builder time.AlpConfiginsideParquetProperties(oneColumnProperty<AlpConfig>rather than two); public setters/getters are unchanged.Reader null tolerance (bug fix):
AlpValuesReaderwas assertingnum_elements == page.valuesCount, which fails on optional columns with nulls (num_elements is the encoded non-null count, valuesCount is the page row count including nulls). Relaxed tonum_elements <= valuesCount.Integration:
DefaultV1ValuesWriterFactoryandDefaultV2ValuesWriterFactoryas a fallback data-page encoding for FLOAT/DOUBLE.Are these changes tested?
Yes, extensively. The full
parquet-columnmodule (840 tests) and all downstream non-ALP modules (arrow/avro/protobuf/thrift/variant/cli, 908 tests) pass; the parquet-hadoop ALP interop tests pass. Coverage spans:Correctness & spec:
Integration (production paths):
null_countcorrect.Robustness & scale:
ByteBufferAllocatorverifies all off-heap buffers are released across 200 write/reset page cycles.Cross-language verification
The Arrow C++ ALP decoder (apache/arrow#48345) reads every Java-written fixture bit-exact against the canonical
_expect.csvtruth tables. Local verification covers the full {V1, V2} × {vs1024, vs4096} matrix plus the corner-case and extreme-value columns: >1.5M values, 0 mismatches. Six representative fixtures are submitted as a stacked PR (prtkgaur/parquet-testing#1) toward apache/parquet-testing#100 for the other-language readers to verify in CI.Test fixtures for #100:
generateAlpFixturesAtMultipleVectorSizesre-encodes the source datasets as Java ALP across page version / vector size / dataset, each verified bit-exact;generateAndVerifyCornerCaseFixturewrites a small synthetic file whose columns each hit a specific corner case (no/one/all exceptions, NaN/Inf/−0.0, constant, differing exponents, nulls, wide and extreme FOR ranges) with a_expect.csvsidecar emitted from the construction recipe.Are there any user-facing changes?
ParquetProperties.withAlpEncoding()(or the HadoopParquetWriter.Builder), globally or per-column.withAlpVectorSize(int), also globally or per-column. Default is 1024.Known limitation: ALP is not yet in the parquet-format spec
ALP is not yet in the parquet-format Thrift spec (apache/parquet-format#533, PR #548). As a temporary bridge,
parquet-format-structures/src/main/perl/patch-alp-encoding.plruns at build time to injectALP(10)into the generatedEncodingenum, andTestParquetMetadataConverterskips ALP in its enum round-trip. Both are clearly marked to be removed once parquet-format ships ALP and parquet-java bumps its dependency. TheALP = 10value is provisional: if the spec assigns a different value, files written now would need regeneration. Flagging this explicitly so it can be weighed in the vote.