MINOR: Allow callers to provide known Hadoop file lengths#3678
Open
dossett wants to merge 1 commit into
Open
Conversation
dossett
force-pushed
the
agent-known-file-length
branch
from
July 20, 2026 12:59
d66f09a to
4efb816
Compare
dossett
marked this pull request as ready for review
July 20, 2026 13:01
steveloughran
suggested changes
Jul 21, 2026
steveloughran
left a comment
steveloughran
left a comment
Contributor
There was a problem hiding this comment.
commented, suggested one more test and a change to the javadocs; no production code changes
| throw new IllegalArgumentException("Invalid file length: " + length); | ||
| } | ||
| FileSystem fs = path.getFileSystem(conf); | ||
| return new HadoopInputFile(fs, path, length, conf); |
Contributor
There was a problem hiding this comment.
this is the only place the new ctor is called, isn't it? so it can never be invoked with a length that is unknown.
| * Creates an input file using a caller-supplied file length. | ||
| * | ||
| * <p>The length is trusted and no file status lookup is performed. Callers must provide the exact | ||
| * length of the file that will be opened. The file is not validated until it is opened. |
Contributor
There was a problem hiding this comment.
declare that the existence of the file may not be checked until the first read, and the length may be trusted; if it is wrong then reads may fail.
| * A caller-supplied length avoids a status lookup and is passed to openFile(). | ||
| */ | ||
| @Test | ||
| public void testOpenFileWithKnownLength() throws Throwable { |
Contributor
There was a problem hiding this comment.
add a test opening a nonexistent file with a declared length. For cloud storage, it may not fail until the first read(); for local fs it will fail immediately. In both cases it must fail with FileNotFoundException.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In an earlier PR @steveloughran suggested that callers who already knew the length of a file (such as Iceberg) could supply that value when opening a file and save at least on metadata lookup. This is the result of me working through that idea with codex, not sure if it's what was originally intended.
It does bump the hadoop dependency from 3.3.0 to 3.3.5. I assume that's safe, but calling it out just in case.
CODEX summary follows
Metadata-aware callers may already know a Parquet file's exact length from a catalog, manifest, or prior listing.
HadoopInputFile.fromPath(Path, Configuration)currently performsgetFileStatus()unconditionally, discarding that knowledge and potentially adding a remote metadata request before the file can be read.This adds a length-aware factory that trusts the caller-provided value and passes it to Hadoop's
openFile()API. The API is format-neutral, but it enables table formats and other metadata-aware readers to avoid the status lookup when they can guarantee the length. Existing path- and status-based construction retain their current behavior. This follows up on the known-length optimization discussed in #3395.The change also raises the Hadoop baseline from 3.3.0 to 3.3.5, where
FS_OPTION_OPENFILE_LENGTHwas standardized and supported by S3A.Summary:
HadoopInputFile.fromPath(Path, long, Configuration).FileStatusas the preferred open hint when one is already available.FS_OPTION_OPENFILE_LENGTH.Testing:
./mvnw -pl parquet-hadoop -Dtest=TestHadoopOpenFile test./mvnw -q -pl parquet-hadoop -DskipTests spotless:check