GH-1283: Optimize number of build compilations in CI by xborder · Pull Request #1284 · apache/arrow-java

xborder · 2026-09-04T22:46:04Z

Summary

The Test workflow previously rebuilt the Maven reactor in each matrix job. This change builds Java and test classes once on Linux JDK 17, then distributes the compiled tree and Arrow Maven artifacts to the Ubuntu and macOS test jobs. A separate job builds C Data JNI once for the Conda jobs. Prebuilt tests run through Surefire without repeating the compiler lifecycle, including Vector's custom test executions and the memory-core opens tests.

Coverage

The full Java suite runs on Ubuntu JDK 17/21/25, macOS ARM JDK 17, and Windows JDK 17.
Independent JDK 21 and 25 jobs still run the Maven build/lifecycle checks. Windows retains its own build and test job, so Windows compilation is still checked.
Conda JDK 17/21/25 jobs now run C Data tests only. The full Java suite no longer runs under the Conda image; Ubuntu covers those JDKs, but Conda-specific behavior outside C Data is no longer exercised.
Build artifacts are retained for seven days so failed test jobs can be rerun after the first day.

Closes #1283.

This PR was assisted by AI.

jbonofre · 2026-09-06T05:08:39Z

Thanks! I gonna take a look.

xborder · 2026-09-22T14:45:11Z

@jbonofre can you also take a look at this one?

xborder · 2026-09-28T13:26:53Z

@kou since you took a look at my last 2 CI changes, can you also take a look at this one?

kou · 2026-09-28T21:46:30Z

+          path: java-build.tgz
+          retention-days: 1
+      - name: Exclude reactor artifacts from Maven dependency cache
+        run: rm -rf .docker/maven-cache/repository/org/apache/arrow


Why do we need to remove them? We can't use them to avoid rebuilding Arrow Java, right?

It is removed so the cache doesn’t carry old Arrow binaries into future runs. The binaries are already in the build artifacts used by every test job

kou · 2026-09-28T21:49:10Z

          - arch: AArch64
            jdk: 17
-            macos: latest


It seems that we need os: macos-XXX here.

not sure I understood. Isn't that what was done here?

kou · 2026-09-28T21:54:50Z

+      - name: Cache Docker Volumes
+        if: ${{ matrix.compose_service }}
+        uses: actions/cache@v6
+        with:
+          path: .docker/maven-cache
+          key: java-build-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
+          restore-keys: java-build-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-


Why doe we need this?
download-artifact isn't enough?

this avoids downloading external dependencies again in each job. download-artifacts only contains the compiled Arrow output and installed org/apache/arrow artifacts

jbonofre · 2026-10-04T04:44:19Z

  contents: read

 env:
+  BUILD_JDK: "17"


Building once with JDK 17 means Spotless and Error Prone no longer run in this workflow: both profiles in the root pom.xml are activated with <jdk>[21,)</jdk>. On main the JDK 21 and 25 jobs run build.sh with those JDKs, so formatting and Error Prone violations fail the PR. With this change they would pass.

It also means nothing compiles the sources with JDK 21 or 25 any more: the jdk: [17, 21, 25] matrix only runs classes compiled by 17, so a change that breaks under a newer javac would go unnoticed.

Could you add a compile-only job (no tests) on JDK 25, or on 21 and 25? That brings both checks back and should only cost a few minutes of the time saved.

This was intentional because I noticed the release is built only in JDK 17. I added separate no-test Maven builds on JDK 21 and 25. They compile the sources and run the lifecycle checks, including Spotless and Error Prone, while the JDK 17 build still supplies the reusable test artifacts.

jbonofre · 2026-10-04T04:47:52Z

          DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
-        run: ci/scripts/build.sh . build jni
-      - name: Test
+        run: ci/scripts/test.sh . build jni


On main the Windows job runs ci/scripts/build.sh before test.sh. Here it only runs Surefire against classes compiled on Linux, and as far as I can see that leaves no Maven build on Windows in any workflow (jni-windows in rc.yml only runs jni_windows_build.sh, which is CMake).

A change that breaks the build on Windows (codegen paths, a plugin without a Windows binary, line-ending-sensitive checks) would no longer be caught. I would keep the full build for the Windows job. If we decide that is acceptable to drop, please say so in the PR description, which currently states that the existing platform coverage is retained.

I'm more in favor to keep Windows job.

Windows now runs its own build.sh and test.sh. I removed it from the prebuilt-test matrix, so Windows builds and tests

jbonofre · 2026-10-04T04:50:01Z

+        with:
+          name: java-build
+          path: java-build.tgz
+          retention-days: 1


With one day of retention, "Re-run failed jobs" stops working a day after the run: build-java succeeded so it is not re-run, and the test jobs then fail on download-artifact. The only way out is re-running everything, which defeats the purpose.

Could you raise this, for example to 7 days? Same for the cdata-build at line 208.

jbonofre · 2026-10-04T04:52:46Z

+    <<: *cdata-prebuilt
+    environment:
+      ARROW_JAVA_CDATA: "ON"
+      ARROW_JAVA_TEST_BASE: "OFF"


With this Conda JNI jobs only run the arrow-c-data tests. On main, conda-jni-cdata runs the full mvn test first and then the c module, so vector, memory, flight and the rest are also exercised on the conda-forge OpenJDK 17/21/25.

Is dropping that intentional? I'm fine with it as deduplication, but then please mention it in PR description, since it says coverage is retained.

Yes, this is intentional deduplication. I updated the PR description

This comment has been minimized.

Sign in to view

xborder marked this pull request as ready for review September 5, 2026 04:45

xborder requested review from jbonofre, kou, laurentgo, lidavidm and wgtmac as code owners September 5, 2026 04:45

kou added the chore PRs that make misc changes. label Sep 6, 2026

github-actions Bot added this to the 20.0.0 milestone Sep 6, 2026

kou reviewed Sep 28, 2026

View reviewed changes

xborder added 15 commits October 1, 2026 18:44

MINOR: Build test artifacts once in CI

513a69c

Simplify build-once test workflow

bcc9111

Restore custom Surefire test coverage

792fda8

Make Vector test passes explicit

99574ff

MINOR: Align build-once workflow with current CI matrix

f71a6f6

Fix stale Arrow artifacts in hosted tests

e9591df

Verify prebuilt test artifacts

0fad043

Simplify prebuilt test validation

5fc4f18

Optimize build-once workflow caches

a775dad

Quote workflow build JDK

72e6787

Fix build JDK workflow context

398329f

Use established Maven cache keys

04b74d5

Use shallow test checkouts

9dd2495

Sync setup-java version with upstream

6d4c6eb

Fix test workflow review comments

5309167

xborder force-pushed the ci-build-once-test-many branch from 18a8b8a to 5309167 Compare October 1, 2026 17:45

xborder requested a review from kou October 1, 2026 21:32

jbonofre requested changes Oct 4, 2026

View reviewed changes

xborder mentioned this pull request Oct 12, 2026

MINOR: Validate compilation on JDK 21 and 25 xborder/arrow-java#31

Closed

xborder added 3 commits October 14, 2026 11:11

Validate Java compilation on JDK 21 and 25

e33270b

Retain CI build artifacts for seven days

fc3620b

Restore independent Windows build and test job

313869f

xborder requested a review from jbonofre October 14, 2026 12:09

Conversation

xborder commented Sep 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Coverage

Uh oh!

This comment has been minimized.

jbonofre commented Sep 6, 2026

Uh oh!

xborder commented Sep 22, 2026

Uh oh!

xborder commented Sep 28, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

xborder Oct 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xborder commented Sep 4, 2026 •

edited

Loading

xborder Oct 1, 2026 •

edited

Loading