Skip to content

GH-1283: Optimize number of build compilations in CI #1284

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
xborder wants to merge 18 commits into
base: main
Choose a base branch
from
Open

GH-1283: Optimize number of build compilations in CI #1284

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
513a69c
MINOR: Build test artifacts once in CI
xborder Jul 28, 2026
bcc9111
Simplify build-once test workflow
xborder Jul 29, 2026
792fda8
Restore custom Surefire test coverage
xborder Jul 31, 2026
99574ff
Make Vector test passes explicit
xborder Jul 31, 2026
f71a6f6
MINOR: Align build-once workflow with current CI matrix
xborder Sep 2, 2026
e9591df
Fix stale Arrow artifacts in hosted tests
xborder Sep 2, 2026
0fad043
Verify prebuilt test artifacts
xborder Sep 2, 2026
5fc4f18
Simplify prebuilt test validation
xborder Sep 2, 2026
a775dad
Optimize build-once workflow caches
xborder Sep 3, 2026
72e6787
Quote workflow build JDK
xborder Sep 3, 2026
398329f
Fix build JDK workflow context
xborder Sep 3, 2026
04b74d5
Use established Maven cache keys
xborder Sep 3, 2026
9dd2495
Use shallow test checkouts
xborder Sep 3, 2026
6d4c6eb
Sync setup-java version with upstream
xborder Sep 4, 2026
5309167
Fix test workflow review comments
xborder Sep 29, 2026
e33270b
Validate Java compilation on JDK 21 and 25
xborder Oct 12, 2026
fc3620b
Retain CI build artifacts for seven days
xborder Oct 12, 2026
313869f
Restore independent Windows build and test job
xborder Oct 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
229 changes: 200 additions & 29 deletions .github/workflows/test.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,28 +34,62 @@ permissions:
contents: read

env:
BUILD_JDK: "17"

@jbonofre jbonofre Oct 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Building once with JDK 17 means Spotless and Error Prone no longer run in this workflow: both profiles in the root pom.xml are activated with <jdk>[21,)</jdk>. On main the JDK 21 and 25 jobs run build.sh with those JDKs, so formatting and Error Prone violations fail the PR. With this change they would pass.

It also means nothing compiles the sources with JDK 21 or 25 any more: the jdk: [17, 21, 25] matrix only runs classes compiled by 17, so a change that breaks under a newer javac would go unnoticed.

Could you add a compile-only job (no tests) on JDK 25, or on 21 and 25? That brings both checks back and should only cost a few minutes of the time saved.

@xborder xborder Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was intentional because I noticed the release is built only in JDK 17. I added separate no-test Maven builds on JDK 21 and 25. They compile the sources and run the lifecycle checks, including Spotless and Error Prone, while the JDK 17 build still supplies the reusable test artifacts.

DOCKER_VOLUME_PREFIX: ".docker/"
MAVEN: 3.9.16

jobs:
ubuntu:
name: AMD64 ${{ matrix.name }} JDK ${{ matrix.jdk }} Maven ${{ matrix.maven }}
build-java:
name: Build Java artifacts
runs-on: ubuntu-latest
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
timeout-minutes: 30
steps:
- name: Checkout Arrow
uses: actions/checkout@v7
with:
fetch-depth: 0
submodules: recursive
- name: Cache Docker Volumes
uses: actions/cache@v6
with:
path: .docker/maven-cache
key: java-build-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: java-build-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-
- name: Build without tests
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
JDK: ${{ env.BUILD_JDK }}
run: |
docker compose run \
--rm \
-e CI=true \
-e "DEVELOCITY_ACCESS_KEY=$DEVELOCITY_ACCESS_KEY" \
ubuntu-artifacts
sudo chown -R "$(id -u):$(id -g)" .docker
- name: Pack reusable artifacts
run: |
tar -czf java-build.tgz \
.docker/java-build \
.docker/maven-cache/repository/org/apache/arrow
- name: Upload reusable artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: java-build
path: java-build.tgz
retention-days: 7
- name: Exclude reactor artifacts from Maven dependency cache
run: rm -rf .docker/maven-cache/repository/org/apache/arrow

compile-java:
name: Compile and check Java on JDK ${{ matrix.jdk }}
runs-on: ubuntu-latest
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
jdk: [17, 21, 25]
maven: [3.9.16]
image: [ubuntu, conda-jni-cdata]
include:
- image: ubuntu
name: "Ubuntu"
- image: conda-jni-cdata
name: "Conda JNI"
env:
JDK: ${{ matrix.jdk }}
MAVEN: ${{ matrix.maven }}
jdk: [21, 25]
steps:
- name: Checkout Arrow
uses: actions/checkout@v7
Expand All @@ -65,53 +99,103 @@ jobs:
- name: Cache Docker Volumes
uses: actions/cache@v6
with:
path: .docker
key: maven-${{ matrix.jdk }}-${{ matrix.maven }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: maven-${{ matrix.jdk }}-${{ matrix.maven }}-
- name: Execute Docker Build
path: .docker/maven-cache
key: java-compile-${{ matrix.jdk }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: java-compile-${{ matrix.jdk }}-${{ env.MAVEN }}-
- name: Compile without tests
env:
# Enables build caching, but not strictly required
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
JDK: ${{ matrix.jdk }}
run: |
docker compose run \
--rm \
-e CI=true \
-e "DEVELOCITY_ACCESS_KEY=$DEVELOCITY_ACCESS_KEY" \
${{ matrix.image }}
ubuntu-artifacts
sudo chown -R "$(id -u):$(id -g)" .docker
- name: Exclude reactor artifacts from Maven dependency cache
run: rm -rf .docker/maven-cache/repository/org/apache/arrow

@kou kou Sep 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to remove them? We can't use them to avoid rebuilding Arrow Java, right?

@xborder xborder Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is removed so the cache doesn’t carry old Arrow binaries into future runs. The binaries are already in the build artifacts used by every test job


macos:
name: ${{ matrix.arch }} macOS ${{ matrix.macos }} Java JDK ${{ matrix.jdk }}
runs-on: macos-${{ matrix.macos }}
test-java:
name: ${{ matrix.name || format('AMD64 Ubuntu JDK {0} Maven 3.9.16', matrix.jdk) }}
needs: build-java
runs-on: ${{ matrix.os }}
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
jdk: [17, 21, 25]
os: [ubuntu-latest]
include:
- arch: AArch64
- os: ubuntu-latest
compose_service: ubuntu-test-prebuilt
- name: AArch64 macOS latest Java JDK 17
os: macos-latest
jdk: 17
macos: latest
compose_service: ''
env:
JDK: ${{ matrix.jdk }}
steps:
- name: Checkout Arrow
uses: actions/checkout@v7
with:
fetch-depth: 0
submodules: recursive
- name: Set up Java
if: ${{ !matrix.compose_service }}
Comment thread
xborder marked this conversation as resolved.
uses: actions/setup-java@v6
with:
distribution: 'temurin'
java-version: ${{ matrix.jdk }}
cache: 'maven'
- name: Build
- name: Cache Docker Volumes
if: ${{ matrix.compose_service }}
uses: actions/cache@v6
with:
path: .docker/maven-cache
key: java-build-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: java-build-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-
Comment on lines +151 to +157

@kou kou Sep 28, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why doe we need this?
download-artifact isn't enough?

@xborder xborder Oct 1, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this avoids downloading external dependencies again in each job. download-artifacts only contains the compiled Arrow output and installed org/apache/arrow artifacts

- name: Download reusable artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: java-build
- name: Restore reusable artifacts
shell: bash
run: |
rm -rf \
.docker/java-build \
.docker/maven-cache/repository/org/apache/arrow
tar -xzf java-build.tgz
- name: Restore reusable artifacts for hosted runner
if: ${{ !matrix.compose_service }}
shell: bash
run: |
cp -a .docker/java-build/build build
rm -rf "${HOME}/.m2/repository/org/apache/arrow"
mkdir -p "${HOME}/.m2/repository/org/apache"
cp -a .docker/maven-cache/repository/org/apache/arrow "${HOME}/.m2/repository/org/apache/"
- name: Test prebuilt artifacts on macOS
if: ${{ !matrix.compose_service }}
shell: bash
env:
ARROW_JAVA_TEST_PREBUILT: "ON"
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: ci/scripts/build.sh . build jni
- name: Test
run: ci/scripts/test.sh . build jni

@jbonofre jbonofre Oct 4, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On main the Windows job runs ci/scripts/build.sh before test.sh. Here it only runs Surefire against classes compiled on Linux, and as far as I can see that leaves no Maven build on Windows in any workflow (jni-windows in rc.yml only runs jni_windows_build.sh, which is CMake).

A change that breaks the build on Windows (codegen paths, a plugin without a Windows binary, line-ending-sensitive checks) would no longer be caught. I would keep the full build for the Windows job. If we decide that is acceptable to drop, please say so in the PR description, which currently states that the existing platform coverage is retained.

I'm more in favor to keep Windows job.

@xborder xborder Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Windows now runs its own build.sh and test.sh. I removed it from the prebuilt-test matrix, so Windows builds and tests

- name: Test prebuilt artifacts on Ubuntu Docker
if: ${{ matrix.compose_service }}
shell: bash
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: ci/scripts/test.sh . build jni
run: |
docker compose run \
--rm \
-e CI=true \
-e "DEVELOCITY_ACCESS_KEY=$DEVELOCITY_ACCESS_KEY" \
${{ matrix.compose_service }}
- name: Exclude reactor artifacts from hosted Maven dependency cache
if: ${{ !matrix.compose_service && always() }}
shell: bash
run: rm -rf "${HOME}/.m2/repository/org/apache/arrow"

windows:
name: AMD64 Windows Server 2022 Java JDK ${{ matrix.jdk }}
Expand Down Expand Up @@ -144,3 +228,90 @@ jobs:
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: ci/scripts/test.sh . build jni

build-cdata:
name: Build C Data artifacts
runs-on: ubuntu-latest
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
timeout-minutes: 30
steps:
- name: Checkout Arrow
uses: actions/checkout@v7
with:
fetch-depth: 0
submodules: recursive
- name: Cache Docker Volumes
uses: actions/cache@v6
with:
path: .docker/maven-cache
key: cdata-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: cdata-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-
- name: Build C Data without tests
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
JDK: ${{ env.BUILD_JDK }}
run: |
docker compose run \
--rm \
-e CI=true \
-e "DEVELOCITY_ACCESS_KEY=$DEVELOCITY_ACCESS_KEY" \
cdata-artifacts
sudo chown -R "$(id -u):$(id -g)" .docker
- name: Pack reusable C Data artifacts
run: |
tar -czf cdata-build.tgz \
.docker/cdata-build \
.docker/cdata-jni-dist \
.docker/maven-cache/repository/org/apache/arrow
- name: Upload reusable C Data artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: cdata-build
path: cdata-build.tgz
retention-days: 7
- name: Exclude reactor artifacts from Maven dependency cache
run: rm -rf .docker/maven-cache/repository/org/apache/arrow

test-cdata:
name: AMD64 Conda JNI JDK ${{ matrix.jdk }} Maven 3.9.16
needs: build-cdata
runs-on: ubuntu-latest
if: ${{ !contains(github.event.pull_request.title, 'WIP') }}
timeout-minutes: 30
strategy:
fail-fast: false
matrix:
jdk: [17, 21, 25]
env:
JDK: ${{ matrix.jdk }}
steps:
- name: Checkout Arrow
uses: actions/checkout@v7
with:
submodules: recursive
- name: Cache Docker Volumes
uses: actions/cache@v6
with:
path: .docker/maven-cache
key: cdata-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-${{ hashFiles('compose.yaml', '**/pom.xml') }}
restore-keys: cdata-${{ env.BUILD_JDK }}-${{ env.MAVEN }}-
- name: Download reusable C Data artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: cdata-build
- name: Restore reusable C Data artifacts
run: |
rm -rf \
.docker/cdata-build \
.docker/cdata-jni-dist \
.docker/maven-cache/repository/org/apache/arrow
tar -xzf cdata-build.tgz
- name: Test C Data without compiler lifecycle
env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.DEVELOCITY_ACCESS_KEY }}
run: |
docker compose run \
--rm \
-e CI=true \
-e "DEVELOCITY_ACCESS_KEY=$DEVELOCITY_ACCESS_KEY" \
cdata-test-prebuilt
2 changes: 1 addition & 1 deletion ci/scripts/build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ cp -r "${source_dir}/dev" "${build_dir}"
# crawl back up to the top. GNU realpath has --relative-to but this does not
# work on macOS

poms=$(find "${source_dir}" -not \( -path "${source_dir}"/build -prune \) -type f -name pom.xml)
poms=$(find "${source_dir}" \( -path "${source_dir}"/build -o -path "${source_dir}"/.docker \) -prune -o -type f -name pom.xml -print)
if [[ "$OSTYPE" == "darwin"* ]]; then
poms=$(echo "$poms" | xargs -n1 python -c "import sys; import os.path; print(os.path.relpath(sys.argv[1], '${source_dir}'))")
else
Expand Down
Loading
Loading