Core: Improve code robustness across core modules#15893
Open
fightBoxing wants to merge 4 commits intoapache:mainfrom
Open
Core: Improve code robustness across core modules#15893fightBoxing wants to merge 4 commits intoapache:mainfrom
fightBoxing wants to merge 4 commits intoapache:mainfrom
Conversation
…(backport apache#10691) (apache#10787) ParallelIterable schedules 2 * WORKER_THREAD_POOL_SIZE tasks for processing input iterables. This defaults to 2 * # CPU cores. When one or some of the input iterables are considerable in size and the ParallelIterable consumer is not quick enough, this could result in unbounded allocation inside `ParallelIterator.queue`. This commit bounds the queue. When queue is full, the tasks yield and get removed from the executor. They are resumed when consumer catches up. (cherry picked from commit 7831a8d) Co-authored-by: Piotr Findeisen <piotr.findeisen@gmail.com>
As part of the change in commit 7831a8d, queue low water mark was introduced. However, it resulted in increased number of manifests being read when planning LIMIT queries in Trino Iceberg connector. To avoid increased I/O, back out the change for now.
Bumps `orc` from 1.9.3 to 1.9.4. Updates `org.apache.orc:orc-core` from 1.9.3 to 1.9.4 Updates `org.apache.orc:orc-tools` from 1.9.3 to 1.9.4 --- updated-dependencies: - dependency-name: org.apache.orc:orc-core dependency-type: direct:production update-type: version-update:semver-patch - dependency-name: org.apache.orc:orc-tools dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
SubAgent analysis identified and fixed 12 issues across 7 files: [HTTPClient] Fix inconsistent config reading in configureConnectionManager() - Changed Integer.getInteger() to PropertyUtil.propertyAsInt() for REST_MAX_CONNECTIONS [HTTPClient] Preserve original exception in buildUri() - Pass URISyntaxException to RESTException constructor for better debugging [BaseCommitService] Fix race condition on succeededCommits - Changed from plain int to AtomicInteger for thread-safe increment [SnapshotProducer] Add logging for swallowed NumberFormatException in updateTotal() [SnapshotProducer] Re-throw Error in catch(Throwable) blocks [SnapshotProducer] Add default branch to switch(entry.status()) [BaseTaskWriter] Change abort() cleanup strategy - Use suppressFailureWhenFinished() instead of throwFailureWhenFinished() [ErrorHandlers] Handle HTTP 405/406 status codes [ErrorHandlers] Include HTTP status code in RESTException messages [BaseTransaction] Add default branch to switch(type) [Exceptions] Add debug logging for suppressed close exceptions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR improves code robustness across 7 core files, addressing 12 issues identified through static analysis.
Changes
HTTPClient
configureConnectionManager()— ChangedInteger.getInteger()toPropertyUtil.propertyAsInt()forREST_MAX_CONNECTIONS, ensuring user-configuredrest.client.max-connectionstakes effect.buildUri()— PassURISyntaxExceptiontoRESTExceptionconstructor for better debugging.BaseCommitService
succeededCommits— Changed from plaininttoAtomicIntegerfor thread-safe increment.SnapshotProducer
NumberFormatExceptioninupdateTotal()— Previously silently ignored corrupted snapshot summary values.Errorincatch(Throwable)blocks — Prevents masking OOM/StackOverflow errors during cleanup and notification.switch(entry.status())— ThrowsIllegalStateExceptionfor unexpectedManifestEntry.Statusvalues.BaseTaskWriter
abort()cleanup strategy — UsesuppressFailureWhenFinished()instead ofthrowFailureWhenFinished()to ensure all files are attempted for deletion during abort.ErrorHandlers
RESTExceptionmessages for better diagnostics.BaseTransaction
switch(type)for unknownTransactionType.Exceptions
Verification
compileJava— BUILD SUCCESSFULspotlessCheck— BUILD SUCCESSFUL