Repository navigation
Download the installer's JDK 21 from Adoptium and verify every download's SHA-256 digest - #1864
Conversation
Oracle no longer serves JDK 21 at download.oracle.com/java/21/latest, so every platform's installer failed with HTTP 404 before installing Neo4j. Adoptium's API serves the latest Eclipse Temurin 21 GA build for each platform, whose archives unpack to the jdk-21* directory layout the installer already looks for. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adoptium's API answers urllib's default User-Agent with Cloudflare error 1010 (HTTP 403), so the installer's JDK download failed there too. The API only redirects to Adoptium's GitHub releases, which accept urllib, so the installer downloads from those directly. Release asset names include the version, so each platform is pinned to its newest Temurin 21 build: 21.0.13+9 for Linux and macOS, and 21.0.12.1+1 for Windows, which has no 21.0.13 build. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One version on every platform instead of 21.0.13+9 on Linux and macOS beside 21.0.12.1+1 on Windows, where 21.0.13 is not published yet. 21.0.12.1+1 is the newest Temurin 21 release with all four platforms, and the build in the eclipse-temurin:21 image that Neo4j's Docker images copy their JDK from. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
leomem
left a comment
There was a problem hiding this comment.
Two more issues in memmachine_configure.py. Both lines are outside this diff, so I'm noting them here. Both bugs predate this PR, but switching from Oracle to Temurin makes them more likely to fire, because an install directory can now hold JDKs from both vendors (Oracle extracts to jdk-21.0.8/, Temurin to jdk-21.0.12.1+1/).
-
find_jdk21_dirreturns the firstjdk-21*directory initerdirorder. If a user reruns the installer in a directory left over from an Oracle install, both directories match.os.scandirorder depends on the filesystem, soJAVA_HOMEcan end up pointing at the old Oracle JDK, and the new download is never used. On macOS the code then appendsContents/Hometo whichever directory it picked. Sorting the matches and taking the newest version, or looking for the exact directory name the archive unpacks to, would make the choice deterministic. -
On Windows,
get_neo4j_envusesnext(install_path.glob("jdk-*")). That depends on file order and matches files as well as directories.install_and_start_neo4jdeletesjdk-21_windows-x64_bin.ziponly after the Neo4j download and extraction succeed. If the Neo4j step fails, the zip stays ininstall_dir, and a later call can return the zip path, or an old Oraclejdk-21.*directory, asJAVA_HOME.neo4j.batthen fails to install the service or runs on the wrong JDK. Filtering withis_dir()and reusing thefind_jdk21_dirlogic here would fix both cases.
|
|
||
| LINUX_JDK_TAR_NAME = "jdk-21_linux-x64_bin.tar.gz" | ||
| LINUX_JDK_URL = f"https://download.oracle.com/java/21/latest/{LINUX_JDK_TAR_NAME}" | ||
| LINUX_JDK_URL = "https://github.com/adoptium/temurin21-binaries/releases/download/jdk-21.0.12.1%2B1/OpenJDK21U-jdk_x64_linux_hotspot_21.0.12.1_1.tar.gz" |
There was a problem hiding this comment.
The installer downloads these archives, extracts them, and then runs the bundled java as JAVA_HOME without checking the archive's SHA-256. Adoptium publishes a digest next to every release asset (OpenJDK21U-jdk_*_21.0.12.1_1.*.sha256.txt). A truncated download, or an archive substituted by a mirror or proxy, would be unpacked and executed as is.
These URLs now point at fixed release assets, so the four digests could be hardcoded next to them and checked in download_file (or just before extract_tar/extract_zip), failing the install on a mismatch.
The published digests for the pinned assets, taken from Adoptium's .sha256.txt files:
LINUX_JDK_SHA256 = "ce79869e1307ed8ee1e2baa86a412b1eb5b75d10a01006d788a6f968bcfaee94"
WINDOWS_JDK_SHA256 = "f9d6e191ab098c0d416e7d588a24420a8621cd2f4720dab2459b8b7b2d2d8b4e"
MACOS_JDK_SHA256_X64 = "44db0f08196daf19a47f90d13388b0c943b67663cb537f998fe29e836fa842ce"
MACOS_JDK_SHA256_ARM64 = "3623232f33a9c3baadf304480b2535f9a3cba8a58d42ecbb438ba267315d9998"There was a problem hiding this comment.
Agreed that it's worth having, and the four digests match Adoptium's .sha256.txt files. One correction: a truncated archive fails during extraction (gzip and zip both detect the missing end), so the installer stops before running it. The case a digest adds is a substituted archive.
The installer has never verified its downloads, the Neo4j archives included, so I'm keeping #1864 to restoring the download. Verification belongs in the follow-up to #1866, which decides whether the installer keeps downloading at all.
🤖 Written by Claude Code (Claude Opus 5.5) on behalf of @edwinyyyu.
There was a problem hiding this comment.
Added after all, in 9f85ff5: since these URLs are pinned, a digest belongs with each one. download_file now takes the expected SHA-256 as a required argument and raises ValueError on a mismatch, before extraction. It covers the two Neo4j archives too (from Neo4j's published .sha256 files), so no download goes unchecked. The four JDK digests are the ones you posted.
🤖 Written by Claude Code (Claude Opus 5.5) on behalf of @edwinyyyu.
|
@leomem thanks for the review. Both lookups predate this PR (
#1866 proposes removing this install path. If the path stays, deterministic JDK lookups belong in that follow-up. 🤖 Written by Claude Code (Claude Opus 5.5) on behalf of @edwinyyyu. |
The installer runs the JDK and Neo4j it downloads. download_file now takes the expected SHA-256 digest and raises ValueError on a mismatch, before anything is extracted. The digests of the four pinned Temurin archives and the two Neo4j 2025.09.0 archives come from the .sha256.txt and .sha256 files Adoptium and Neo4j publish beside each asset. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
marvinyu-memverge
left a comment
There was a problem hiding this comment.
Approving at 9f85ff5.
Checked against the installer at this head:
- Linux and macOS find the JDK as the first directory starting with
jdk-21(find_jdk21_dir), Windows asjdk-*; Temurin'sjdk-21.0.12.1+1/matches both, and macOS'sContents/Homeis whereMacosInstallerpointsJAVA_HOME. - The tarballs go through
tarfilewithfilter="data"and the Windows zip through_safe_extract_zip, which rejects symlink entries. The Installation Test downloads and unpacks the JDK and is green at this head on ubuntu-latest, windows-latest, macos-latest and macos-15-intel, so the Temurin archives clear both guards. - The digest check runs in both
download_fileimplementations before anything is extracted, every caller passes its digest, and the same four green Installation Test runs download every pinned archive, so all six digests are checked against the real files.hashlib.file_digestneeds 3.11 and the package requires 3.12. - The local file names (
jdk-21_linux-x64_bin.tar.gzand the rest) only name the temporary download, which is deleted after extraction, and the unit tests read the URLs from the constants, so neither needed to change.
| with Path(path).open("rb") as file: | ||
| digest = hashlib.file_digest(file, "sha256").hexdigest() | ||
| if digest != sha256: | ||
| raise ValueError(f"{path} has SHA-256 digest {digest}, expected {sha256}") |
There was a problem hiding this comment.
main() catches only KeyboardInterrupt and RuntimeError, so this ValueError escapes it. On a checksum mismatch the user gets a raw traceback and no "Configuration failed" log. Could this raise RuntimeError (or a subclass), or could main() also catch ValueError? The two download tests would need the matching exception type.
| def download_file(self, url: str, dest: str, sha256: str) -> None: | ||
| """Download file on Linux and verify its SHA-256 digest.""" | ||
| urllib.request.urlretrieve(url, dest) | ||
| _verify_sha256(dest, sha256) |
There was a problem hiding this comment.
When the check fails, the rejected archive stays at dest under its normal name (for example neo4j-community-2025.09.0-unix.tar.gz, about 150 MB), where it looks like a good download. Could download_file delete dest before re-raising? Something like:
try:
_verify_sha256(dest, sha256)
except ValueError:
Path(dest).unlink(missing_ok=True)
raiseThe Windows download_file at line 354 needs the same change, or both could call one shared helper.
leomem
left a comment
There was a problem hiding this comment.
Approving. Please address my two inline comments before merging: raise an exception main() catches when the checksum doesn't match, and delete the rejected download from disk.
xiongzubiao
left a comment
There was a problem hiding this comment.
Verified at 9f85ff5: the digests are pinned in utilities.py, all six archives are checked against their own digest before extraction, and test-install passes on all four OSes. That also unblocks the test-install failures other branches are hitting.
| @@ -255,6 +263,38 @@ def test_install_in_linux(mock_input, mock_wizard): | |||
| assert environment.neo4j_started | |||
There was a problem hiding this comment.
MockLinuxEnvironment records each URL's digest in downloaded_files, but this test only asserts neo4j_started, so the Linux URL/digest pairing is never checked. If LinuxInstaller.install_and_start_neo4j passed LINUX_NEO4J_SHA256 for the JDK download (or a macOS digest), this test would still pass and the real installer would fail on every Linux run. The macOS tests already cover their pairing, and the PR description says the installer tests check that each download gets its own digest.
Adding the same two asserts the macOS tests use would close it:
assert environment.downloaded_files[LINUX_JDK_URL] == LINUX_JDK_SHA256
assert environment.downloaded_files[LINUX_NEO4J_URL] == LINUX_NEO4J_SHA256LINUX_JDK_URL and LINUX_JDK_SHA256 would need importing.
Purpose of the change
Make
memmachine-configureable to install Neo4j again. The installer downloads JDK 21 fromdownload.oracle.com/java/21/latest/, which returns 404 on every platform, so the installer stops before Neo4j and the Installation Test fails on every pull request to main (#1859, #1778).Description
github.com/adoptium/temurin21-binaries), pinned to 21.0.12.1+1 on every platform. It is the newest Temurin 21 release with all four platforms (21.0.13 has no Windows build yet) and the build in theeclipse-temurin:21image that Neo4j's Docker images copy their JDK from. Temurin archives unpack tojdk-21.*/(on macOS withContents/Homeinside), the layout the installer's JDK lookup already expects, so the local file names and the Neo4j URLs are unchanged.api.adoptium.net/v3/binary/latest/..., because the API answers urllib's default User-Agent with Cloudflare error 1010 (HTTP 403; Python requests banned? adoptium/api.adoptium.net#580). The API only redirects to these same release assets. Release asset names include the version, so a JDK update means editing the four URLs and their digests..sha256.txtand.sha256files Adoptium and Neo4j publish beside each asset.download_filetakes the digest as a required argument and raisesValueErroron a mismatch, before anything is extracted.latestlinks only for releases under its No-Fee Terms and Conditions license. JDK 21 left that license after September 2026, and its later updates are under the OTN license, sojava/21/latest/went away whilejava/25/latest/still resolves.Why Temurin
eclipse-temurin(21 for bullseye, 25 for trixie), so the installer runs Neo4j on the same JDK build Neo4j puts in its images. GitHub's Ubuntu runners also preinstall Temurin, andactions/setup-java's README example usesdistribution: temurin.latestlinks and the samejdk-21.*/layout, so it would work equally well. Amazon Corretto's archives unpack toamazon-corretto-21.*(Linux, macOS) andjdk21.0.*(Windows), which the installer's JDK lookup doesn't match.Type of change
How Has This Been Tested?
Unit Test
Manual verification
Each pinned URL answers Python's default
urllibclient, the one the installer uses, with the requested byte range (HTTP 206).The archives' top-level directory is
jdk-21.0.12.1+1/(withContents/inside on macOS).ruff check,ruff format --checkandty(as CI runs it onpackages/server) pass, and the installer unit tests pass (47 passed).New unit tests run the real
download_fileof both environments against localfile://URLs: a matching digest keeps the file, and a mismatch raisesValueError. The installer tests check that each download is passed its own digest, including the macOS architecture pairing.This PR's Installation Test downloads the JDK and Neo4j, starts them, and runs the server on all four OSes.
Test Results: as listed above.
Checklist
Further comments
pip install memmachine-serverwith SQLite and no config file #1868 a default install that needs no Neo4j or config file.🤖 Written by Claude Code (Claude Opus 5.5) on behalf of @edwinyyyu.