From 65f99cb01717cdffd077dd2204f405c351710c3c Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 15 Aug 2022 09:49:16 -0500 Subject: [PATCH 01/80] Change cache folder to match XDG directory spec https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html --- melting-pot.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/melting-pot.sh b/melting-pot.sh index a170f67..046ed6e 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -72,7 +72,7 @@ # -- Constants -- -meltingPotCache="$HOME/.scijava/melting-pot" +meltingPotCache="$HOME/.cache/scijava/melting-pot" # -- Functions -- From bff241cae711cdf3850facb959318a81855e689d Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 15 Aug 2022 13:23:11 -0500 Subject: [PATCH 02/80] melting-pot: improve leading documentation We give a real-world example now, rather than a contrived foo/bar one. And we link to a helpful "diamond dependency" explanation. --- melting-pot.sh | 63 ++++++++++++++++++++++---------------------------- 1 file changed, 28 insertions(+), 35 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index 046ed6e..17b1aa1 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -5,50 +5,43 @@ # ============================================================================ # Tests all components of a project affected by changes in its dependencies. # -# First, an anecdote illustrating the problem this script solves: +# In particular, this script detects problems caused by diamond dependency +# structures: # -# Suppose you have a large application, org:app:1.0.0, with many dependencies: -# org:foo:1.2.3, org:bar:3.4.5, and many others. +# https://jlbp.dev/what-is-a-diamond-dependency-conflict # -# Now suppose you make some changes to foo, and want to know whether deploying -# them (i.e., releasing a new foo and updating app to depend on that release) -# will break the app. So you manually update your local copy of app to depend -# on org:foo:1.3.0-SNAPSHOT, and run the build (including tests, of course). +# This "melting pot" build rebuilds every dependency of a project, but at +# unified dependency versions matching those of the toplevel project. # -# The build passes, but this alone is insufficient: org:bar:3.4.5 also depends -# on org:foo:1.2.3, so you manually update bar to use org:foo:1.3.0-SNAPSHOT, -# then build bar to verify that it also is not broken by the update. +# For example, net.imagej:imagej:2.5.0 depends on many components including +# org.scijava:scijava-common:2.88.1 and net.imagej:imagej-ops:0.46.1, +# both of which depend on org.scijava:parsington. But: # -# This process quickly becomes very tedious when there are dozens of -# components of app which all depend on foo. +# - org.scijava:scijava-common:2.88.1 depends on org.scijava:parsington:3.0.0 +# - net.imagej:imagej-ops:0.46.1 depends on org.scijava:parsington:2.0.0 # -# And more importantly, testing each component individually in this manner is -# still insufficient to determine whether all of them will truly work together -# at runtime, where only a single version of each component is deployed. +# ImageJ2 can only depend on one of these versions at runtime. The newer one, +# ideally. SciJava projects use the pom-scijava parent POM as a Bill of +# Materials (BOM) to declare these winning versions, which works great... +# EXCEPT for when newer versions break backwards compatibility, as happened +# here: it's a SemVer-versioned project at different major version numbers. # -# For example: suppose org:bar:3.4.5 depends on org:lib:8.0.0, while -# org:foo:1.2.3 depends on org:lib:7.0.0. The relevant facts are: +# Enter this melting-pot script. It rebuilds each project dependency from +# source and runs the unit tests, but with all dependency versions pinned to +# match those of the toplevel project. # -# * Your new foo (org:foo:1.3.0-SNAPSHOT) builds against lib 7, and portions -# of it rely on lib-7-specific API. +# So in the example above, this script: # -# * The bar component pinned to foo 1.3.0-SNAPSHOT builds against lib 8; it -# compiles with passing tests because bar only invokes portions of the foo -# API which do not require lib-7-specific API. +# 1. gathers the dependencies of net.imagej:imagej:2.5.0; +# 2. clones each dependency from SCM at the correct release tag; +# 3. rebuilds each dependency, but with dependency versions overridden to +# those of net.imagej:imagej:2.5.0 rather than those originally used for +# that dependency at that release. # -# In this scenario, it is lib 8 that is actually deployed at runtime with the -# app, so parts of foo will be broken, even though both foo and bar build with -# passing tests individually. -# -# This "melting pot" build seeks to overcome many of these issues by unifying -# all components of the app into a single multi-module build, with all -# versions uniformly pinned to the ones that will actually be deployed at -# runtime. -# -# This goal is achieved by synthesizing a multi-module build including all -# affected components (or optionally, all components period) of the specific -# project, and then executing a Maven build with uniformly overridden versions -# of all components to the ones resolved for the project itself. +# So e.g. in the above scenario, net.imagej:imagej-ops:0.46.1 will be rebuilt +# against org.scijava:parsington:3.0.0, and we will discover whether any of +# parsington's breaking API changes from 2.0.0 to 3.0.0 actually impact the +# compilation or (tested) runtime behavior of imagej-ops. # # IMPORTANT IMPLEMENTATION DETAIL! The override works by setting a version # property for each component of the form "artifactId.version"; it is assumed From f084780e2d890b1ec58d61f4829500d40c1348a5 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 16 Aug 2022 12:10:10 -0500 Subject: [PATCH 03/80] melting-pot: improve local git repository caching This cuts down on excessive network use. The first time a particular G:A is requested at any version, it gets fully (not shallowly) cloned into the user's melting-pot cache folder. From there, the tag for the given version is cloned shallowly from the locally cached repository. In this way, complex melting pots can be repeatedly generated without hitting the network in most cases. This commit also improves debug and info messages in a few places. --- melting-pot.sh | 69 +++++++++++++++++++++++++++++++------------------- 1 file changed, 43 insertions(+), 26 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index 17b1aa1..76beb90 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -347,6 +347,7 @@ pom() { xpath() { local xmlFile="$1" shift + local expression="$@" local xpath="/" while [ $# -gt 0 ] do @@ -354,9 +355,10 @@ xpath() { xpath="$xpath/*[local-name()='$1']" shift done - debug "xmllint --xpath \"$xpath\" \"$xmlFile\"" - xmllint --xpath "$xpath" "$xmlFile" 2> /dev/null | - sed -E 's/^[^>]*>(.*)<[^<]*$/\1/' + local value=$(xmllint --xpath "$xpath" "$xmlFile" 2> /dev/null | + sed -E 's/^[^>]*>(.*)<[^<]*$/\1/') + debug "xpath $xmlFile $expression -> $value" + echo "$value" } # For the given GAV ($1), recursively gets the value of the @@ -399,6 +401,10 @@ scmTag() { local a=$(artifactId "$1") local v=$(version "$1") local scmURL="$(scmURL "$1")" + # TODO: Avoid network use. We can scan the locally cached repo. + # But this gets complicated when the locally cached repo is + # out of date, and the needed tag is not there yet... + debug "git ls-remote --tags \"$scmURL\" | sed 's/.*refs\/tags\///'" local allTags="$(git ls-remote --tags "$scmURL" | sed 's/.*refs\/tags\///' || error "$1: Invalid scm url: $scmURL")" for tag in "$a-$v" "$v" "v$v" @@ -415,35 +421,46 @@ scmTag() { fi } -# Fetches the source code for the given GAV. Returns the directory. -retrieveSource() { - local scmURL="$(scmURL "$1")" - test "$scmURL" || die "Cannot glean SCM URL for $1" 10 - local scmBranch - test "$2" && scmBranch="$2" || scmBranch="$(scmTag "$1")" +# Ensures the source code for the given GAV exists in the melting-pot +# structure, and is up-to-date with the remote. Returns the directory. +resolveSource() { local g=$(groupId "$1") local a=$(artifactId "$1") - local v=$(version "$1") - local sourceDir="$meltingPotCache/$g/$a/$v" - if [ -d "$sourceDir" ] + local cachedRepoDir="$meltingPotCache/$g/$a" + if [ ! -d "$cachedRepoDir" ] then - debug "Using previously fetched project source at $sourceDir" - else - # Source has never been fetched before. Save it into ~/.scijava/melting-pot. - debug "git clone \"$scmURL\" --branch \"$scmBranch\" --depth 1 \"$sourceDir\"" - git clone "$scmURL" --branch "$scmBranch" --depth 1 "$sourceDir" 2> /dev/null || - die "Could not fetch project source for $1" 3 + # Source does not exist locally. Clone it into the melting pot cache. + local scmURL="$(scmURL "$1")" + test "$scmURL" || die "$1: cannot glean SCM URL" 10 + info "$1: cached repository not found; cloning from remote: $scmURL" + debug "git clone --bare \"$scmURL\" \"$cachedRepoDir\"" + git clone --bare "$scmURL" "$cachedRepoDir" 2> /dev/null || + die "$1: could not clone project source from $scmURL" 3 fi - # Copy the pristine cached source directory into the melting-pot structure. + + # Check whether the needed branch/tag exists. + local scmBranch + test "$2" && scmBranch="$2" || scmBranch="$(scmTag "$1")" + debug "git ls-remote \"file://$cachedRepoDir\" | grep -q \"\trefs/tags/$scmBranch$\"" + git ls-remote "file://$cachedRepoDir" | grep -q "\trefs/tags/$scmBranch$" || { + # Couldn't find the scmBranch as a tag in the cached repo. Either the + # tag is new, or it's not a tag ref at all (e.g. it's a branch). + # So let's update from the original remote repository. + info "$1: local tag not found for ref '$scmBranch'; updating cached repository: $cachedRepoDir" + (cd "$cachedRepoDir" && debug "git fetch --tags" && git fetch --tags) + } + + # Shallow clone the source at the given version into melting-pot structure. local destDir="$g/$a" - mkdir -p "$(dirname "$destDir")" - cp -rp "$sourceDir" "$destDir" + debug "git clone \"file://$cachedRepoDir\" --branch \"$scmBranch\" --depth 1 \"$destDir\"" + git clone "file://$cachedRepoDir" --branch "$scmBranch" --depth 1 "$destDir" 2> /dev/null || + die "$1: could not clone branch '$scmBranch' from local cache" 15 # Now verify that the cloned pom.xml contains the expected version! local expectedVersion=$(version "$1") local actualVersion=$(xpath "$destDir/pom.xml" project version) test "$expectedVersion" = "$actualVersion" || - die "POM for $1 contains wrong version: $actualVersion" 14 + warn "$1: POM contains wrong version: $actualVersion" echo "$destDir" } @@ -451,7 +468,7 @@ retrieveSource() { # Gets the list of dependencies for the project in the CWD. deps() { cd "$1" - debug "mvn dependency:list" + debug "mvn -DincludeScope=runtime -B dependency:list" local depList="$(mvn -DincludeScope=runtime -B dependency:list)" || die "Problem fetching dependencies!" 5 echo "$depList" | grep '^\[INFO\] [^ ]' | @@ -626,7 +643,7 @@ meltDown() { else # Treat specified project as a GAV. info "Fetching project source" - retrieveSource "$1" "$branch" + resolveSource "$1" "$branch" fi # Get the project dependencies. @@ -653,8 +670,8 @@ meltDown() { if [ "$(isIncluded "$gav")" ] then - info "$g:$a: fetching source for version $v" - dir="$(retrieveSource "$gav")" + info "$g:$a: resolving source for version $v" + dir="$(resolveSource "$gav")" fi done From 5d417a7f44fd1943e3eaae59b713dd313d182412 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 16 Aug 2022 13:09:17 -0500 Subject: [PATCH 04/80] melting-pot: skip previously successful builds This saves a lot of time when running the melting pot repeatedly. --- melting-pot.sh | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/melting-pot.sh b/melting-pot.sh index 76beb90..fc21a14 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -613,6 +613,11 @@ generateScript() { done echo >> melt.sh echo 'do' >> melt.sh + echo ' # If the build passed previously, don'\''t repeat it.' >> melt.sh + echo ' test -f "$f/build.log" &&' >> melt.sh + echo ' tail -n6 "$f/build.log" | grep -qF '\''[INFO] BUILD SUCCESS'\'' &&' >> melt.sh + echo ' echo "[SUCCESS] $f (prior)" && continue' >> melt.sh + echo >> melt.sh echo ' cd "$f"' >> melt.sh echo ' sh "$dir/build.sh" >build.log 2>&1 &&' >> melt.sh echo ' echo "[SUCCESS] $f" || {' >> melt.sh From 8a5c4d460710f10a2eba9ee8c6d4a5e8c0504ccd Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 16 Aug 2022 13:30:42 -0500 Subject: [PATCH 05/80] melting-pot: fix aggregator POM schema location This POM isn't used by default anymore, but it's still good to fix this. --- melting-pot.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/melting-pot.sh b/melting-pot.sh index fc21a14..daf8c59 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -562,7 +562,7 @@ generatePOM() { echo '> pom.xml echo ' xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"' >> pom.xml echo ' xsi:schemaLocation="http://maven.apache.org/POM/4.0.0' >> pom.xml - echo ' http://maven.apache.org/xsd/maven-4.0.0.xsd">' >> pom.xml + echo ' https://maven.apache.org/xsd/maven-4.0.0.xsd">' >> pom.xml echo ' 4.0.0' >> pom.xml echo >> pom.xml echo ' melting-pot' >> pom.xml From 33eefca60332af4f8a8e1e0b5f23d728b7f9238e Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 16 Aug 2022 15:10:23 -0500 Subject: [PATCH 06/80] release-version: fix up outdated forum links --- release-version.sh | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/release-version.sh b/release-version.sh index 4c4e6bf..1f97d82 100755 --- a/release-version.sh +++ b/release-version.sh @@ -220,6 +220,32 @@ then -m 'And using HTTP now generates errors in Eclipse (and probably other IDEs).' fi +# Change forum references from forum.image.net to forum.image.sc. +if grep -q 'https*://forum.imagej.net' pom.xml >/dev/null 2>/dev/null +then + echo "================================================================" + echo "NOTE: Your POM still references forum.imagej.net. Fixing it now." + echo "================================================================" + sed 's;https*://forum.imagej.net;https://forum.image.sc;g' pom.xml > pom.new && + mv -f pom.new pom.xml && + git commit pom.xml \ + -m 'POM: fix forum.image.sc tag link' \ + -m 'The Discourse software updated the tags path from /tags/ to /tag/.' +fi + +# Ensure that references to forum.image.sc use /tag/, not /tags/. +if grep -q forum.image.sc/tags/ pom.xml >/dev/null 2>/dev/null +then + echo "==================================================================" + echo "NOTE: Your POM has an old-style forum.image.sc tag. Fixing it now." + echo "==================================================================" + sed 's;forum.image.sc/tags/;forum.image.sc/tag/;g' pom.xml > pom.new && + mv -f pom.new pom.xml && + git commit pom.xml \ + -m 'POM: fix forum.image.sc tag link' \ + -m 'The Discourse software updated the tags path from /tags/ to /tag/.' +fi + # Ensure license headers are up-to-date. test "$SKIP_LICENSE_UPDATE" -o -z "$licenseName" -o "$licenseName" = "N/A" || { mvn license:update-project-license license:update-file-header && From b7a05c5a5015762d6e462a6863369a1d513a0bce Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 16 Aug 2022 15:10:49 -0500 Subject: [PATCH 07/80] melting-pot: make prior success message clearer --- melting-pot.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/melting-pot.sh b/melting-pot.sh index daf8c59..c85c28f 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -616,7 +616,7 @@ generateScript() { echo ' # If the build passed previously, don'\''t repeat it.' >> melt.sh echo ' test -f "$f/build.log" &&' >> melt.sh echo ' tail -n6 "$f/build.log" | grep -qF '\''[INFO] BUILD SUCCESS'\'' &&' >> melt.sh - echo ' echo "[SUCCESS] $f (prior)" && continue' >> melt.sh + echo ' echo "[SKIPPED] $f (already succeeded)" && continue' >> melt.sh echo >> melt.sh echo ' cd "$f"' >> melt.sh echo ' sh "$dir/build.sh" >build.log 2>&1 &&' >> melt.sh From 0a90ffdce47676a5c7d8d2bb6c722f6bc9c637e1 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 16 Aug 2022 17:02:42 -0500 Subject: [PATCH 08/80] melting-pot: list dependencies before building This is useful to confirm that the build happened with the intended dependency versions. In cases where a version was hardcoded in a tag, that component version would not actually get set uniformly by the version property override mechanism. --- melting-pot.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/melting-pot.sh b/melting-pot.sh index c85c28f..652d435 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -698,7 +698,7 @@ meltDown() { # Generate build scripts. info "Generating build scripts" generatePOM - echo "mvn $args \\\\\n test \$@" > build.sh + echo "mvn $args \\\\\n dependency:list test \$@" > build.sh generateScript # Build everything. From 6ca99de5b0bb77024c9b9a6eae80473940967819 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Wed, 17 Aug 2022 06:24:13 -0500 Subject: [PATCH 09/80] melting-pot: fix local tag check on Linux Apparently, GNU grep doesn't like \t as tab? But \b works. --- melting-pot.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index 652d435..827e0dc 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -441,8 +441,8 @@ resolveSource() { # Check whether the needed branch/tag exists. local scmBranch test "$2" && scmBranch="$2" || scmBranch="$(scmTag "$1")" - debug "git ls-remote \"file://$cachedRepoDir\" | grep -q \"\trefs/tags/$scmBranch$\"" - git ls-remote "file://$cachedRepoDir" | grep -q "\trefs/tags/$scmBranch$" || { + debug "git ls-remote \"file://$cachedRepoDir\" | grep -q \"\brefs/tags/$scmBranch$\"" + git ls-remote "file://$cachedRepoDir" | grep -q "\brefs/tags/$scmBranch$" || { # Couldn't find the scmBranch as a tag in the cached repo. Either the # tag is new, or it's not a tag ref at all (e.g. it's a branch). # So let's update from the original remote repository. From 6448eed775f03d2e20270403ef86359fd10829d2 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 18 Aug 2022 13:40:59 -0500 Subject: [PATCH 10/80] melting-pot: fix cached repo tag updating And avoid use of ambiguous dir variable. And fix fast-fail behavior when source resolution fails. --- melting-pot.sh | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index 827e0dc..c4bc3bb 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -441,13 +441,23 @@ resolveSource() { # Check whether the needed branch/tag exists. local scmBranch test "$2" && scmBranch="$2" || scmBranch="$(scmTag "$1")" + test "$scmBranch" || die "$1: cannot glean SCM tag" 14 debug "git ls-remote \"file://$cachedRepoDir\" | grep -q \"\brefs/tags/$scmBranch$\"" git ls-remote "file://$cachedRepoDir" | grep -q "\brefs/tags/$scmBranch$" || { # Couldn't find the scmBranch as a tag in the cached repo. Either the # tag is new, or it's not a tag ref at all (e.g. it's a branch). # So let's update from the original remote repository. - info "$1: local tag not found for ref '$scmBranch'; updating cached repository: $cachedRepoDir" - (cd "$cachedRepoDir" && debug "git fetch --tags" && git fetch --tags) + info "$1: local tag not found for ref '$scmBranch'" + info "$1: updating cached repository: $cachedRepoDir" + cd "$cachedRepoDir" + debug "git fetch --tags" + if [ "$debug" ] + then + git fetch --tags + else + git fetch --tags > /dev/null + fi + cd - > /dev/null } # Shallow clone the source at the given version into melting-pot structure. @@ -467,7 +477,7 @@ resolveSource() { # Gets the list of dependencies for the project in the CWD. deps() { - cd "$1" + cd "$1" || die "No such directory: $1" 16 debug "mvn -DincludeScope=runtime -B dependency:list" local depList="$(mvn -DincludeScope=runtime -B dependency:list)" || die "Problem fetching dependencies!" 5 @@ -643,17 +653,18 @@ meltDown() { test -f "$1/pom.xml" || die "Not a Maven project: $1" 12 info "Local Maven project: $1" mkdir -p "LOCAL" - local dir="LOCAL/PROJECT" - ln -s "$1" "$dir" + local projectDir="LOCAL/PROJECT" + ln -s "$1" "$projectDir" else # Treat specified project as a GAV. info "Fetching project source" - resolveSource "$1" "$branch" + local projectDir=$(resolveSource "$1" "$branch") + test $? -eq 0 || exit $? fi # Get the project dependencies. info "Determining project dependencies" - local deps="$(deps "$dir")" + local deps="$(deps "$projectDir")" test "$deps" || die "Cannot glean project dependencies" 7 local args="-Denforcer.skip" @@ -676,7 +687,7 @@ meltDown() { if [ "$(isIncluded "$gav")" ] then info "$g:$a: resolving source for version $v" - dir="$(resolveSource "$gav")" + resolveSource "$gav" >/dev/null fi done From 98057ccebd82d821e3369c0610b036c997b562eb Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 18 Aug 2022 13:43:15 -0500 Subject: [PATCH 11/80] release-version: fix two-pass property extraction It should only use -U (which is slower) after the first attempt fails. The point of the initial attempt is to avoid using -U. --- release-version.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release-version.sh b/release-version.sh index 1f97d82..a6207b1 100755 --- a/release-version.sh +++ b/release-version.sh @@ -113,7 +113,7 @@ Options include: # -- Extract project details -- echoArg='${project.version}:${license.licenseName}:${project.parent.groupId}:${project.parent.artifactId}:${project.parent.version}' -projectDetails=$(mvn -N -U -Dexec.executable=echo -Dexec.args="$echoArg" exec:exec -q) +projectDetails=$(mvn -N -Dexec.executable=echo -Dexec.args="$echoArg" exec:exec -q) test $? -eq 0 || projectDetails=$(mvn -U -N -Dexec.executable=echo -Dexec.args="$echoArg" exec:exec -q) test $? -eq 0 || die "Could not extract version from pom.xml. Error follows:\n$projectDetails" currentVersion=${projectDetails%%:*} From 29fd15fa723fee750c1fee1a46d5fa6089d24e8f Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 18 Aug 2022 13:44:29 -0500 Subject: [PATCH 12/80] release-version: add an extra sanity check In theory, the property extraction will exit non-zero whenever there is an error, so this new check should never be triggered. But I'm paranoid, so just in case, let's check the projectDetails result string, too. --- release-version.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/release-version.sh b/release-version.sh index a6207b1..71d43d8 100755 --- a/release-version.sh +++ b/release-version.sh @@ -116,6 +116,8 @@ echoArg='${project.version}:${license.licenseName}:${project.parent.groupId}:${p projectDetails=$(mvn -N -Dexec.executable=echo -Dexec.args="$echoArg" exec:exec -q) test $? -eq 0 || projectDetails=$(mvn -U -N -Dexec.executable=echo -Dexec.args="$echoArg" exec:exec -q) test $? -eq 0 || die "Could not extract version from pom.xml. Error follows:\n$projectDetails" +echo "$projectDetails" | grep -Fqv '[ERROR]' || + die "Error extracting version from pom.xml. Error follows:\n$projectDetails" currentVersion=${projectDetails%%:*} projectDetails=${projectDetails#*:} licenseName=${projectDetails%%:*} From 22675f34ab95dae404d848abcc3351d0f2946af1 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Fri, 26 Aug 2022 15:31:57 -0500 Subject: [PATCH 13/80] ci-build.sh: dump logs for failing unit tests --- ci-build.sh | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/ci-build.sh b/ci-build.sh index ea27f46..3475d07 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -219,6 +219,20 @@ EOL # --== POST-BUILD ACTIONS ==-- + # Dump logs for any failing unit tests. + if [ -d target/surefire-reports ] + then + find target/surefire-reports -name '*.txt' | while read report + do + if grep -qF 'FAILURE!' "$report" + then + echo + echo "[$report]" + cat "$report" + fi + done + fi + if [ "$deployOK" -a "$success" -eq 0 ]; then echo echo "== Invalidating SciJava Maven repository cache ==" From fd386c7f849a3255e8e2d9c56bf6768fd71797a0 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 6 Sep 2022 17:31:49 -0500 Subject: [PATCH 14/80] github-actionify: consolidate to one build.yml Still needed: change ci-build.sh not to deploy when building PRs. However, as things stand, this will only happen when the PR's branch is from the same repository, rather than from a fork. --- github-actionify.sh | 48 ++++++++++++--------------------------------- 1 file changed, 12 insertions(+), 36 deletions(-) diff --git a/github-actionify.sh b/github-actionify.sh index 5bb0a00..e16c0c4 100755 --- a/github-actionify.sh +++ b/github-actionify.sh @@ -9,10 +9,8 @@ dir="$(dirname "$0")" ciDir=.github -ciSlugBuildMain=workflows/build-main.yml -ciSlugBuildPR=workflows/build-pr.yml -ciConfigBuildMain=$ciDir/$ciSlugBuildMain -ciConfigBuildPR=$ciDir/$ciSlugBuildPR +ciSlugBuild=workflows/build.yml +ciConfigBuild=$ciDir/$ciSlugBuild ciSetupScript=$ciDir/setup.sh ciBuildScript=$ciDir/build.sh pomMinVersion='17.1.1' @@ -104,9 +102,8 @@ process() { # -- GitHub Action sanity checks -- test -e "$ciDir" -a ! -d "$ciDir" && die "$ciDir is not a directory" - test -e "$ciConfigBuildMain" -a ! -f "$ciConfigBuildMain" && die "$ciConfigBuildMain is not a regular file" - test -e "$ciConfigBuildPR" -a ! -f "$ciConfigBuildPR" && die "$ciConfigBuildPR is not a regular file" - test -e "$ciConfigBuildMain" && warn "$ciConfigBuildMain already exists" + test -e "$ciConfigBuild" -a ! -f "$ciConfigBuild" && die "$ciConfigBuild is not a regular file" + test -e "$ciConfigBuild" && warn "$ciConfigBuild already exists" test -e "$ciBuildScript" && warn "$ciBuildScript already exists" test -e "$ciSetupScript" && warn "$ciSetupScript already exists" @@ -137,7 +134,7 @@ process() { # -- Do things -- - # Add/update the main GitHub Actions configuration file. + # Add/update the GitHub Actions build configuration file. cat >"$tmpFile" <>"$tmpFile" - cat >>"$tmpFile" <"$tmpFile" <>"$tmpFile" <"$tmpFile" <"$tmpFile" </dev/null 2>&1 then info "Updating README.md GitHub Action badge" - sed "s;travis-ci.*;$domain/$repoSlug/actions/$ciSlugBuildMain/badge.svg)](https://$domain/$repoSlug/actions/$ciSlugBuildMain);g" README.md >"$tmpFile" + sed "s;travis-ci.*;$domain/$repoSlug/actions/$ciSlugBuild/badge.svg)](https://$domain/$repoSlug/actions/$ciSlugBuild);g" README.md >"$tmpFile" update README.md 'update README.md badge link' - elif grep -qF "$domain/$repoSlug/actions/$ciSlugBuildMain/badge.svg" README.md >/dev/null 2>&1 + elif grep -qF "$domain/$repoSlug/actions/$ciSlugBuild/badge.svg" README.md >/dev/null 2>&1 then info "GitHub Action badge already present in README.md" else info "Adding GitHub Action badge to README.md" - echo "[![](https://$domain/$repoSlug/actions/$ciSlugBuildMain/badge.svg)](https://$domain/$repoSlug/actions/$ciSlugBuildMain)" >"$tmpFile" + echo "[![Build Status](https://$domain/$repoSlug/actions/$ciSlugBuild/badge.svg)](https://$domain/$repoSlug/actions/$ciSlugBuild)" >"$tmpFile" echo >>"$tmpFile" test -f README.md && cat README.md >>"$tmpFile" update README.md 'add README.md badge link' From 5843231594c5e464bed4f27855fda3d4bdba33f1 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 12 Sep 2022 15:45:07 -0500 Subject: [PATCH 15/80] class-version.sh: work around new hexdump bug It seems that newer versions of hexdump do not allow seeking from stdin anymore? At least, that's what others online have concluded: https://www.medo64.com/2021/06/hexdumps-illegal-seek/ Fortunately, in this case, we can just decode all 8 of the first bytes, and then take the last 4. --- ci-setup-github-actions.sh | 3 ++- class-version.sh | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/ci-setup-github-actions.sh b/ci-setup-github-actions.sh index 000f6d3..649c944 100755 --- a/ci-setup-github-actions.sh +++ b/ci-setup-github-actions.sh @@ -5,7 +5,8 @@ # echo "BUILD_REPOSITORY=https://github.com/${GITHUB_REPOSITORY}" -echo "BUILD_OS=${RUNNER_OS}" +echo "BUILD_OS=${RUNNER_OS}" +echo "BUILD_DEPLOY_FORBIDDEN=..." # CTR START HERE - use GITHUB-specific env vars set upon PR to decide this echo "BUILD_REPOSITORY=https://github.com/${GITHUB_REPOSITORY}" >> $GITHUB_ENV echo "BUILD_OS=${RUNNER_OS}" >> $GITHUB_ENV diff --git a/class-version.sh b/class-version.sh index 5d128c2..8da1057 100755 --- a/class-version.sh +++ b/class-version.sh @@ -4,7 +4,7 @@ class_version() { # extract bytes 4-7 - info=$(head -c 8 | hexdump -s 4 -e '4/1 "%d\n" "\n"') + info=$(head -c 8 | hexdump -e '4/1 "%d\n" "\n"' | tail -n4) minor1="$(echo "$info" | sed -n 1p)" minor2="$(echo "$info" | sed -n 2p)" major1="$(echo "$info" | sed -n 3p)" From 71fd73aad741903cbf308bf2e8b817a61657ad67 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Wed, 14 Sep 2022 10:00:54 -0500 Subject: [PATCH 16/80] ci-build: do not deploy when building PRs If my reading of the GitHub Actions documentation is correct, then organization secrets will be available to pull request builds from a branch of the same repository (as opposed to from a project fork). In that case, we still want the build script to skip deployment, because we don't want to be deploying snapshots from from PRs, only those from the mainline integration branch. --- ci-build.sh | 9 +++++++-- ci-setup-github-actions.sh | 15 ++++++++++----- 2 files changed, 17 insertions(+), 7 deletions(-) diff --git a/ci-build.sh b/ci-build.sh index 3475d07..78d957f 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -93,7 +93,7 @@ EOL # --== DEPLOYMENT CHECKS ==-- - # Determine whether deploying will be possible. + # Determine whether deploying is both possible and warranted. echo "Performing deployment checks" deployOK= @@ -139,7 +139,12 @@ EOL fi ;; esac - deployOK=1 + if [ "$BUILD_BASE_REF" -o "$BUILD_HEAD_REF" ] + then + echo "No deploy -- proposed change: $BUILD_BASE_REF -> $BUILD_HEAD_REF" + else + deployOK=1 + fi fi fi fi diff --git a/ci-setup-github-actions.sh b/ci-setup-github-actions.sh index 649c944..a0e7a66 100755 --- a/ci-setup-github-actions.sh +++ b/ci-setup-github-actions.sh @@ -4,9 +4,14 @@ # ci-setup-github-actions.sh - Set CI-related environment variables from GitHub Actions. # -echo "BUILD_REPOSITORY=https://github.com/${GITHUB_REPOSITORY}" -echo "BUILD_OS=${RUNNER_OS}" -echo "BUILD_DEPLOY_FORBIDDEN=..." # CTR START HERE - use GITHUB-specific env vars set upon PR to decide this +echo "BUILD_REPOSITORY=https://github.com/$GITHUB_REPOSITORY" +echo "BUILD_REPOSITORY=https://github.com/$GITHUB_REPOSITORY" >> $GITHUB_ENV -echo "BUILD_REPOSITORY=https://github.com/${GITHUB_REPOSITORY}" >> $GITHUB_ENV -echo "BUILD_OS=${RUNNER_OS}" >> $GITHUB_ENV +echo "BUILD_OS=$RUNNER_OS" +echo "BUILD_OS=$RUNNER_OS" >> $GITHUB_ENV + +echo "BUILD_BASE_REF=$GITHUB_BASE_REF" +echo "BUILD_BASE_REF=$GITHUB_BASE_REF" >> $GITHUB_ENV + +echo "BUILD_HEAD_REF=$GITHUB_HEAD_REF" +echo "BUILD_HEAD_REF=$GITHUB_HEAD_REF" >> $GITHUB_ENV From 6451dc35e5384a8d58ffbed73011d58035864933 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 19 Sep 2022 13:21:14 -0500 Subject: [PATCH 17/80] melting-pot: convert from tabs to spaces I'm normally pro-tabs. But this script has a lot of script generation logic, and I want columns to line up nicely... which they won't at different tab widths. --- melting-pot.sh | 964 ++++++++++++++++++++++++------------------------- 1 file changed, 482 insertions(+), 482 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index c4bc3bb..10465c0 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -78,87 +78,87 @@ die() { error $1; exit $2; } unknownArg() { error "Unknown option: $@"; usage=1; } checkPrereqs() { - while [ $# -gt 0 ] - do - which $1 > /dev/null 2> /dev/null - test $? -ne 0 && die "Missing prerequisite: $1" 255 - shift - done + while [ $# -gt 0 ] + do + which $1 > /dev/null 2> /dev/null + test $? -ne 0 && die "Missing prerequisite: $1" 255 + shift + done } verifyPrereqs() { - checkPrereqs git mvn xmllint - git --version | grep -q 'git version 2' || - die "Please use git v2.x; older versions (<=1.7.9.5 at least) mishandle 'git clone --depth 1'" 254 + checkPrereqs git mvn xmllint + git --version | grep -q 'git version 2' || + die "Please use git v2.x; older versions (<=1.7.9.5 at least) mishandle 'git clone --depth 1'" 254 } parseArguments() { - while [ $# -gt 0 ] - do - case "$1" in - -b|--branch) - branch="$2" - shift - ;; - -c|--changes) - test "$changes" && changes="$changes,$2" || changes="$2" - shift - ;; - -i|--includes) - test "$includes" && includes="$includes,$2" || includes="$2" - shift - ;; - -e|--excludes) - test "$excludes" && excludes="$excludes,$2" || excludes="$2" - shift - ;; - -r|--remoteRepos) - test "$remoteRepos" && remoteRepos="$remoteRepos,$2" || remoteRepos="$2" - shift - ;; - -l|--localRepo) - repoBase="$2" - shift - ;; - -o|--outputDir) - outputDir="$2" - shift - ;; - -p|--prune) - prune=1 - ;; - -v|--verbose) - verbose=1 - ;; - -d|--debug) - debug=1 - ;; - -f|--force) - force=1 - ;; - -s|--skipBuild) - skipBuild=1 - ;; - -h|--help) - usage=1 - ;; - -*) - unknownArg "$1" - ;; - *) - test -z "$project" && project="$1" || - unknownArg "$1" - ;; - esac - shift - done - - test -z "$project" -a -z "$usage" && - error "No project specified!" && usage=1 - - if [ "$usage" ] - then - echo "Usage: $(basename "$0") [-b ] [-c ] \\ + while [ $# -gt 0 ] + do + case "$1" in + -b|--branch) + branch="$2" + shift + ;; + -c|--changes) + test "$changes" && changes="$changes,$2" || changes="$2" + shift + ;; + -i|--includes) + test "$includes" && includes="$includes,$2" || includes="$2" + shift + ;; + -e|--excludes) + test "$excludes" && excludes="$excludes,$2" || excludes="$2" + shift + ;; + -r|--remoteRepos) + test "$remoteRepos" && remoteRepos="$remoteRepos,$2" || remoteRepos="$2" + shift + ;; + -l|--localRepo) + repoBase="$2" + shift + ;; + -o|--outputDir) + outputDir="$2" + shift + ;; + -p|--prune) + prune=1 + ;; + -v|--verbose) + verbose=1 + ;; + -d|--debug) + debug=1 + ;; + -f|--force) + force=1 + ;; + -s|--skipBuild) + skipBuild=1 + ;; + -h|--help) + usage=1 + ;; + -*) + unknownArg "$1" + ;; + *) + test -z "$project" && project="$1" || + unknownArg "$1" + ;; + esac + shift + done + + test -z "$project" -a -z "$usage" && + error "No project specified!" && usage=1 + + if [ "$usage" ] + then + echo "Usage: $(basename "$0") [-b ] [-c ] \\ [-i ] [-e ] [-r ] [-l ] [-o ] [-pvfsh] @@ -219,425 +219,425 @@ groupIds org.scijava, net.imagej, net.imglib2 and io.scif in the pot. The -e flag is used to exclude net.imglib2:imglib2-roi from the pot. " - exit 1 - fi + exit 1 + fi - # If project is a local directory path, get its absolute path. - test -d "$project" && project=$(cd "$project" && pwd) + # If project is a local directory path, get its absolute path. + test -d "$project" && project=$(cd "$project" && pwd) - # Assign default parameter values. - test "$outputDir" || outputDir="melting-pot" - test "$repoBase" || repoBase="$HOME/.m2/repository" + # Assign default parameter values. + test "$outputDir" || outputDir="melting-pot" + test "$repoBase" || repoBase="$HOME/.m2/repository" } createDir() { - test -z "$force" -a -e "$1" && - die "Directory already exists: $1" 2 + test -z "$force" -a -e "$1" && + die "Directory already exists: $1" 2 - rm -rf "$1" - mkdir -p "$1" - cd "$1" + rm -rf "$1" + mkdir -p "$1" + cd "$1" } groupId() { - echo "${1%%:*}" + echo "${1%%:*}" } artifactId() { - local result="${1#*:}" # strip groupId - echo "${result%%:*}" + local result="${1#*:}" # strip groupId + echo "${result%%:*}" } version() { - local result="${1#*:}" # strip groupId - case "$result" in - *:*) - result="${result#*:}" # strip artifactId - case "$result" in - *:*:*:*) - # G:A:P:C:V:S - result="${result#*:}" # strip packaging - result="${result#*:}" # strip classifier - ;; - *:*:*) - # G:A:P:V:S - result="${result#*:}" # strip packaging - ;; - *) - # G:A:V or G:A:V:? - ;; - esac - echo "${result%%:*}" - ;; - esac + local result="${1#*:}" # strip groupId + case "$result" in + *:*) + result="${result#*:}" # strip artifactId + case "$result" in + *:*:*:*) + # G:A:P:C:V:S + result="${result#*:}" # strip packaging + result="${result#*:}" # strip classifier + ;; + *:*:*) + # G:A:P:V:S + result="${result#*:}" # strip packaging + ;; + *) + # G:A:V or G:A:V:? + ;; + esac + echo "${result%%:*}" + ;; + esac } classifier() { - local result="${1#*:}" # strip groupId - case "$result" in - *:*) - result="${result#*:}" # strip artifactId - case "$result" in - *:*:*:*) - # G:A:P:C:V:S - result="${result#*:}" # strip packaging - ;; - *:*:*) - # G:A:P:V:S - result="" - ;; - *:*) - # G:A:V:C - result="${result#*:}" # strip version - ;; - *) - # G:A:V - result="" - ;; - esac - echo "${result%%:*}" - ;; - esac + local result="${1#*:}" # strip groupId + case "$result" in + *:*) + result="${result#*:}" # strip artifactId + case "$result" in + *:*:*:*) + # G:A:P:C:V:S + result="${result#*:}" # strip packaging + ;; + *:*:*) + # G:A:P:V:S + result="" + ;; + *:*) + # G:A:V:C + result="${result#*:}" # strip version + ;; + *) + # G:A:V + result="" + ;; + esac + echo "${result%%:*}" + ;; + esac } # Converts the given GAV into a path in the local repository cache. repoPath() { - local gPath="$(echo "$(groupId "$1")" | tr :. /)" - local aPath="$(artifactId "$1")" - local vPath="$(version "$1")" - echo "$repoBase/$gPath/$aPath/$vPath" + local gPath="$(echo "$(groupId "$1")" | tr :. /)" + local aPath="$(artifactId "$1")" + local vPath="$(version "$1")" + echo "$repoBase/$gPath/$aPath/$vPath" } # Gets the path to the given GAV's POM file in the local repository cache. pomPath() { - local pomFile="$(artifactId "$1")-$(version "$1").pom" - echo "$(repoPath "$1")/$pomFile" + local pomFile="$(artifactId "$1")-$(version "$1").pom" + echo "$(repoPath "$1")/$pomFile" } # Fetches the POM for the given GAV into the local repository cache. downloadPOM() { - local g="$(groupId "$1")" - local a="$(artifactId "$1")" - local v="$(version "$1")" - debug "mvn dependency:get \\ - -DrepoUrl=\"$remoteRepos\" \\ - -DgroupId=\"$g\" \\ - -DartifactId=\"$a\" \\ - -Dversion=\"$v\" \\ - -Dpackaging=pom" - mvn dependency:get \ - -DrepoUrl="$remoteRepos" \ - -DgroupId="$g" \ - -DartifactId="$a" \ - -Dversion="$v" \ - -Dpackaging=pom > /dev/null || - die "Problem fetching $g:$a:$v from $remoteRepos" 4 + local g="$(groupId "$1")" + local a="$(artifactId "$1")" + local v="$(version "$1")" + debug "mvn dependency:get \\ + -DrepoUrl=\"$remoteRepos\" \\ + -DgroupId=\"$g\" \\ + -DartifactId=\"$a\" \\ + -Dversion=\"$v\" \\ + -Dpackaging=pom" + mvn dependency:get \ + -DrepoUrl="$remoteRepos" \ + -DgroupId="$g" \ + -DartifactId="$a" \ + -Dversion="$v" \ + -Dpackaging=pom > /dev/null || + die "Problem fetching $g:$a:$v from $remoteRepos" 4 } # Gets the POM path for the given GAV, ensuring it exists locally. pom() { - local pomPath="$(pomPath "$1")" - test -f "$pomPath" || downloadPOM "$1" - test -f "$pomPath" || die "Cannot access POM: $pomPath" 9 - echo "$pomPath" + local pomPath="$(pomPath "$1")" + test -f "$pomPath" || downloadPOM "$1" + test -f "$pomPath" || die "Cannot access POM: $pomPath" 9 + echo "$pomPath" } # For the given XML file on disk ($1), gets the value of the # specified XPath expression of the form "//$2/$3/$4/...". xpath() { - local xmlFile="$1" - shift - local expression="$@" - local xpath="/" - while [ $# -gt 0 ] - do - # NB: Ignore namespace issues; see: http://stackoverflow.com/a/8266075 - xpath="$xpath/*[local-name()='$1']" - shift - done - local value=$(xmllint --xpath "$xpath" "$xmlFile" 2> /dev/null | - sed -E 's/^[^>]*>(.*)<[^<]*$/\1/') - debug "xpath $xmlFile $expression -> $value" - echo "$value" + local xmlFile="$1" + shift + local expression="$@" + local xpath="/" + while [ $# -gt 0 ] + do + # NB: Ignore namespace issues; see: http://stackoverflow.com/a/8266075 + xpath="$xpath/*[local-name()='$1']" + shift + done + local value=$(xmllint --xpath "$xpath" "$xmlFile" 2> /dev/null | + sed -E 's/^[^>]*>(.*)<[^<]*$/\1/') + debug "xpath $xmlFile $expression -> $value" + echo "$value" } # For the given GAV ($1), recursively gets the value of the # specified XPath expression of the form "//$2/$3/$4/...". pomValue() { - local pomPath="$(pom "$1")" - test "$pomPath" || die "Cannot discern POM path for $1" 6 - shift - local value="$(xpath "$pomPath" $@)" - if [ "$value" ] - then - echo "$value" - else - # Path not found in POM; look in the parent POM. - local pg="$(xpath "$pomPath" project parent groupId)" - if [ "$pg" ] - then - # There is a parent POM declaration in this POM. - local pa="$(xpath "$pomPath" project parent artifactId)" - local pv="$(xpath "$pomPath" project parent version)" - pomValue "$pg:$pa:$pv" $@ - fi - fi + local pomPath="$(pom "$1")" + test "$pomPath" || die "Cannot discern POM path for $1" 6 + shift + local value="$(xpath "$pomPath" $@)" + if [ "$value" ] + then + echo "$value" + else + # Path not found in POM; look in the parent POM. + local pg="$(xpath "$pomPath" project parent groupId)" + if [ "$pg" ] + then + # There is a parent POM declaration in this POM. + local pa="$(xpath "$pomPath" project parent artifactId)" + local pv="$(xpath "$pomPath" project parent version)" + pomValue "$pg:$pa:$pv" $@ + fi + fi } # Gets the SCM URL for the given GAV. scmURL() { - pomValue "$1" project scm connection | sed 's/^scm:git://' | - sed 's_git:\(//github.com/\)_https:\1_' + pomValue "$1" project scm connection | sed 's/^scm:git://' | + sed 's_git:\(//github.com/\)_https:\1_' } # Gets the SCM tag for the given GAV. scmTag() { - local tag=$(pomValue "$1" project scm tag) - if [ -z "$tag" -o "$tag" = "HEAD" ] - then - # The value was not set properly, - # so we try to guess the tag naming scheme. :-/ - warn "$1: improper scm tag value; scanning remote tags..." - local a=$(artifactId "$1") - local v=$(version "$1") - local scmURL="$(scmURL "$1")" - # TODO: Avoid network use. We can scan the locally cached repo. - # But this gets complicated when the locally cached repo is - # out of date, and the needed tag is not there yet... - debug "git ls-remote --tags \"$scmURL\" | sed 's/.*refs\/tags\///'" - local allTags="$(git ls-remote --tags "$scmURL" | sed 's/.*refs\/tags\///' || - error "$1: Invalid scm url: $scmURL")" - for tag in "$a-$v" "$v" "v$v" - do - echo "$allTags" | grep -q "^$tag$" && { - info "$1: inferred tag: $tag" - echo "$tag" - return - } - done - error "$1: inscrutable tag scheme" - else - echo "$tag" - fi + local tag=$(pomValue "$1" project scm tag) + if [ -z "$tag" -o "$tag" = "HEAD" ] + then + # The value was not set properly, + # so we try to guess the tag naming scheme. :-/ + warn "$1: improper scm tag value; scanning remote tags..." + local a=$(artifactId "$1") + local v=$(version "$1") + local scmURL="$(scmURL "$1")" + # TODO: Avoid network use. We can scan the locally cached repo. + # But this gets complicated when the locally cached repo is + # out of date, and the needed tag is not there yet... + debug "git ls-remote --tags \"$scmURL\" | sed 's/.*refs\/tags\///'" + local allTags="$(git ls-remote --tags "$scmURL" | sed 's/.*refs\/tags\///' || + error "$1: Invalid scm url: $scmURL")" + for tag in "$a-$v" "$v" "v$v" + do + echo "$allTags" | grep -q "^$tag$" && { + info "$1: inferred tag: $tag" + echo "$tag" + return + } + done + error "$1: inscrutable tag scheme" + else + echo "$tag" + fi } # Ensures the source code for the given GAV exists in the melting-pot # structure, and is up-to-date with the remote. Returns the directory. resolveSource() { - local g=$(groupId "$1") - local a=$(artifactId "$1") - local cachedRepoDir="$meltingPotCache/$g/$a" - if [ ! -d "$cachedRepoDir" ] - then - # Source does not exist locally. Clone it into the melting pot cache. - local scmURL="$(scmURL "$1")" - test "$scmURL" || die "$1: cannot glean SCM URL" 10 - info "$1: cached repository not found; cloning from remote: $scmURL" - debug "git clone --bare \"$scmURL\" \"$cachedRepoDir\"" - git clone --bare "$scmURL" "$cachedRepoDir" 2> /dev/null || - die "$1: could not clone project source from $scmURL" 3 - fi - - # Check whether the needed branch/tag exists. - local scmBranch - test "$2" && scmBranch="$2" || scmBranch="$(scmTag "$1")" - test "$scmBranch" || die "$1: cannot glean SCM tag" 14 - debug "git ls-remote \"file://$cachedRepoDir\" | grep -q \"\brefs/tags/$scmBranch$\"" - git ls-remote "file://$cachedRepoDir" | grep -q "\brefs/tags/$scmBranch$" || { - # Couldn't find the scmBranch as a tag in the cached repo. Either the - # tag is new, or it's not a tag ref at all (e.g. it's a branch). - # So let's update from the original remote repository. - info "$1: local tag not found for ref '$scmBranch'" - info "$1: updating cached repository: $cachedRepoDir" - cd "$cachedRepoDir" - debug "git fetch --tags" - if [ "$debug" ] - then - git fetch --tags - else - git fetch --tags > /dev/null - fi - cd - > /dev/null - } - - # Shallow clone the source at the given version into melting-pot structure. - local destDir="$g/$a" - debug "git clone \"file://$cachedRepoDir\" --branch \"$scmBranch\" --depth 1 \"$destDir\"" - git clone "file://$cachedRepoDir" --branch "$scmBranch" --depth 1 "$destDir" 2> /dev/null || - die "$1: could not clone branch '$scmBranch' from local cache" 15 - - # Now verify that the cloned pom.xml contains the expected version! - local expectedVersion=$(version "$1") - local actualVersion=$(xpath "$destDir/pom.xml" project version) - test "$expectedVersion" = "$actualVersion" || - warn "$1: POM contains wrong version: $actualVersion" - - echo "$destDir" + local g=$(groupId "$1") + local a=$(artifactId "$1") + local cachedRepoDir="$meltingPotCache/$g/$a" + if [ ! -d "$cachedRepoDir" ] + then + # Source does not exist locally. Clone it into the melting pot cache. + local scmURL="$(scmURL "$1")" + test "$scmURL" || die "$1: cannot glean SCM URL" 10 + info "$1: cached repository not found; cloning from remote: $scmURL" + debug "git clone --bare \"$scmURL\" \"$cachedRepoDir\"" + git clone --bare "$scmURL" "$cachedRepoDir" 2> /dev/null || + die "$1: could not clone project source from $scmURL" 3 + fi + + # Check whether the needed branch/tag exists. + local scmBranch + test "$2" && scmBranch="$2" || scmBranch="$(scmTag "$1")" + test "$scmBranch" || die "$1: cannot glean SCM tag" 14 + debug "git ls-remote \"file://$cachedRepoDir\" | grep -q \"\brefs/tags/$scmBranch$\"" + git ls-remote "file://$cachedRepoDir" | grep -q "\brefs/tags/$scmBranch$" || { + # Couldn't find the scmBranch as a tag in the cached repo. Either the + # tag is new, or it's not a tag ref at all (e.g. it's a branch). + # So let's update from the original remote repository. + info "$1: local tag not found for ref '$scmBranch'" + info "$1: updating cached repository: $cachedRepoDir" + cd "$cachedRepoDir" + debug "git fetch --tags" + if [ "$debug" ] + then + git fetch --tags + else + git fetch --tags > /dev/null + fi + cd - > /dev/null + } + + # Shallow clone the source at the given version into melting-pot structure. + local destDir="$g/$a" + debug "git clone \"file://$cachedRepoDir\" --branch \"$scmBranch\" --depth 1 \"$destDir\"" + git clone "file://$cachedRepoDir" --branch "$scmBranch" --depth 1 "$destDir" 2> /dev/null || + die "$1: could not clone branch '$scmBranch' from local cache" 15 + + # Now verify that the cloned pom.xml contains the expected version! + local expectedVersion=$(version "$1") + local actualVersion=$(xpath "$destDir/pom.xml" project version) + test "$expectedVersion" = "$actualVersion" || + warn "$1: POM contains wrong version: $actualVersion" + + echo "$destDir" } # Gets the list of dependencies for the project in the CWD. deps() { - cd "$1" || die "No such directory: $1" 16 - debug "mvn -DincludeScope=runtime -B dependency:list" - local depList="$(mvn -DincludeScope=runtime -B dependency:list)" || - die "Problem fetching dependencies!" 5 - echo "$depList" | grep '^\[INFO\] [^ ]' | - sed 's/\[INFO\] //' | sed 's/ .*//' | sort - cd - > /dev/null + cd "$1" || die "No such directory: $1" 16 + debug "mvn -DincludeScope=runtime -B dependency:list" + local depList="$(mvn -DincludeScope=runtime -B dependency:list)" || + die "Problem fetching dependencies!" 5 + echo "$depList" | grep '^\[INFO\] [^ ]' | + sed 's/\[INFO\] //' | sed 's/ .*//' | sort + cd - > /dev/null } # Checks whether the given GA(V) matches the specified filter pattern. gaMatch() { - local ga="$1" - local filter="$2" - local g="$(groupId "$ga")" - local a="$(artifactId "$ga")" - local fg="$(groupId "$filter")" - local fa="$(artifactId "$filter")" - test "$fg" = "$g" -o "$fg" = "*" || return - test "$fa" = "$a" -o "$fa" = "*" || return - echo 1 + local ga="$1" + local filter="$2" + local g="$(groupId "$ga")" + local a="$(artifactId "$ga")" + local fg="$(groupId "$filter")" + local fa="$(artifactId "$filter")" + test "$fg" = "$g" -o "$fg" = "*" || return + test "$fa" = "$a" -o "$fa" = "*" || return + echo 1 } # Determines whether the given GA(V) version is being overridden. isChanged() { - local IFS="," + local IFS="," - local change - for change in $changes - do - test "$(gaMatch "$1" "$change")" && echo 1 && return - done + local change + for change in $changes + do + test "$(gaMatch "$1" "$change")" && echo 1 && return + done } # Determines whether the given GA(V) meets the inclusion criteria. isIncluded() { - # do not include the changed artifacts we are testing against - test "$(isChanged "$1")" && return - - local IFS="," - - # ensure GA is not excluded - local exclude - for exclude in $excludes - do - test "$(gaMatch "$1" "$exclude")" && return - done - - # ensure GA is included - test -z "$includes" && echo 1 && return - local include - for include in $includes - do - test "$(gaMatch "$1" "$include")" && echo 1 && return - done + # do not include the changed artifacts we are testing against + test "$(isChanged "$1")" && return + + local IFS="," + + # ensure GA is not excluded + local exclude + for exclude in $excludes + do + test "$(gaMatch "$1" "$exclude")" && return + done + + # ensure GA is included + test -z "$includes" && echo 1 && return + local include + for include in $includes + do + test "$(gaMatch "$1" "$include")" && echo 1 && return + done } # Deletes components which do not depend on a changed GAV. pruneReactor() { - local dir - for dir in */* - do - info "Checking relevance of component $dir" - local deps="$(deps "$dir")" - test "$deps" || die "Cannot glean dependencies for '$dir'" 8 - - # Determine whether the component depends on a changed GAV. - local keep - unset keep - local dep - for dep in $deps - do - test "$(isChanged "$dep")" && keep=1 && break - done - - # If the component is irrelevant, prune it. - if [ -z "$keep" ] - then - info "Pruning irrelevant component: $dir" - rm -rf "$dir" - fi - done + local dir + for dir in */* + do + info "Checking relevance of component $dir" + local deps="$(deps "$dir")" + test "$deps" || die "Cannot glean dependencies for '$dir'" 8 + + # Determine whether the component depends on a changed GAV. + local keep + unset keep + local dep + for dep in $deps + do + test "$(isChanged "$dep")" && keep=1 && break + done + + # If the component is irrelevant, prune it. + if [ -z "$keep" ] + then + info "Pruning irrelevant component: $dir" + rm -rf "$dir" + fi + done } # Tests if the given directory contains the appropriate source code. isProject() { - local a="$(xpath "$1/pom.xml" project artifactId)" - test "$1" = "LOCAL/PROJECT" -o "$a" = "$(basename "$1")" && echo 1 + local a="$(xpath "$1/pom.xml" project artifactId)" + test "$1" = "LOCAL/PROJECT" -o "$a" = "$(basename "$1")" && echo 1 } # Generates an aggregator POM for all modules in the current directory. generatePOM() { - echo '' > pom.xml - echo '> pom.xml - echo ' xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"' >> pom.xml - echo ' xsi:schemaLocation="http://maven.apache.org/POM/4.0.0' >> pom.xml - echo ' https://maven.apache.org/xsd/maven-4.0.0.xsd">' >> pom.xml - echo ' 4.0.0' >> pom.xml - echo >> pom.xml - echo ' melting-pot' >> pom.xml - echo ' melting-pot' >> pom.xml - echo ' 0.0.0-SNAPSHOT' >> pom.xml - echo ' pom' >> pom.xml - echo >> pom.xml - echo ' Melting Pot' >> pom.xml - echo >> pom.xml - echo ' ' >> pom.xml - local dir - for dir in */* - do - if [ "$(isProject "$dir")" ] - then - echo " $dir" >> pom.xml - else - # Check for a child component of a multi-module project. - local childDir="$dir/$(basename "$dir")" - test "$(isProject "$childDir")" && - echo " $childDir" >> pom.xml - fi - done - echo ' ' >> pom.xml - echo '' >> pom.xml + echo '' > pom.xml + echo '> pom.xml + echo ' xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"' >> pom.xml + echo ' xsi:schemaLocation="http://maven.apache.org/POM/4.0.0' >> pom.xml + echo ' https://maven.apache.org/xsd/maven-4.0.0.xsd">' >> pom.xml + echo ' 4.0.0' >> pom.xml + echo >> pom.xml + echo ' melting-pot' >> pom.xml + echo ' melting-pot' >> pom.xml + echo ' 0.0.0-SNAPSHOT' >> pom.xml + echo ' pom' >> pom.xml + echo >> pom.xml + echo ' Melting Pot' >> pom.xml + echo >> pom.xml + echo ' ' >> pom.xml + local dir + for dir in */* + do + if [ "$(isProject "$dir")" ] + then + echo " $dir" >> pom.xml + else + # Check for a child component of a multi-module project. + local childDir="$dir/$(basename "$dir")" + test "$(isProject "$childDir")" && + echo " $childDir" >> pom.xml + fi + done + echo ' ' >> pom.xml + echo '' >> pom.xml } # Generates melt.sh script for all modules in the current directory. generateScript() { - echo '#!/bin/sh' > melt.sh - echo 'trap "exit" INT' >> melt.sh - echo 'echo "Melting the pot..."' >> melt.sh - echo 'dir=$(pwd)' >> melt.sh - echo 'failCount=0' >> melt.sh - echo 'for f in \' >> melt.sh - local dir - for dir in */* - do - if [ "$(isProject "$dir")" ] - then - echo " $dir \\" >> melt.sh - else - # Check for a child component of a multi-module project. - local childDir="$dir/$(basename "$dir")" - test "$(isProject "$childDir")" && - echo " $childDir \\" >> melt.sh - fi - done - echo >> melt.sh - echo 'do' >> melt.sh - echo ' # If the build passed previously, don'\''t repeat it.' >> melt.sh - echo ' test -f "$f/build.log" &&' >> melt.sh - echo ' tail -n6 "$f/build.log" | grep -qF '\''[INFO] BUILD SUCCESS'\'' &&' >> melt.sh - echo ' echo "[SKIPPED] $f (already succeeded)" && continue' >> melt.sh - echo >> melt.sh - echo ' cd "$f"' >> melt.sh - echo ' sh "$dir/build.sh" >build.log 2>&1 &&' >> melt.sh - echo ' echo "[SUCCESS] $f" || {' >> melt.sh - echo ' echo "[FAILURE] $f"' >> melt.sh - echo ' failCount=$((failCount+1))' >> melt.sh - echo ' }' >> melt.sh - echo ' cd - >/dev/null' >> melt.sh - echo 'done' >> melt.sh - echo 'test "$failCount" -gt 255 && failCount=255' >> melt.sh - echo 'exit "$failCount"' >> melt.sh + echo '#!/bin/sh' > melt.sh + echo 'trap "exit" INT' >> melt.sh + echo 'echo "Melting the pot..."' >> melt.sh + echo 'dir=$(pwd)' >> melt.sh + echo 'failCount=0' >> melt.sh + echo 'for f in \' >> melt.sh + local dir + for dir in */* + do + if [ "$(isProject "$dir")" ] + then + echo " $dir \\" >> melt.sh + else + # Check for a child component of a multi-module project. + local childDir="$dir/$(basename "$dir")" + test "$(isProject "$childDir")" && + echo " $childDir \\" >> melt.sh + fi + done + echo >> melt.sh + echo 'do' >> melt.sh + echo ' # If the build passed previously, don'\''t repeat it.' >> melt.sh + echo ' test -f "$f/build.log" &&' >> melt.sh + echo ' tail -n6 "$f/build.log" | grep -qF '\''[INFO] BUILD SUCCESS'\'' &&' >> melt.sh + echo ' echo "[SKIPPED] $f (already succeeded)" && continue' >> melt.sh + echo >> melt.sh + echo ' cd "$f"' >> melt.sh + echo ' sh "$dir/build.sh" >build.log 2>&1 &&' >> melt.sh + echo ' echo "[SUCCESS] $f" || {' >> melt.sh + echo ' echo "[FAILURE] $f"' >> melt.sh + echo ' failCount=$((failCount+1))' >> melt.sh + echo ' }' >> melt.sh + echo ' cd - >/dev/null' >> melt.sh + echo 'done' >> melt.sh + echo 'test "$failCount" -gt 255 && failCount=255' >> melt.sh + echo 'exit "$failCount"' >> melt.sh } # Creates and tests an appropriate multi-module reactor for the given project. @@ -645,84 +645,84 @@ generateScript() { # the multi-module build, with each changed GAV overridding the originally # specified version for the corresponding GA. meltDown() { - # Fetch the project source code. - if [ -d "$1" ] - then - # Use local directory for the specified project. - test -d "$1" || die "No such directory: $1" 11 - test -f "$1/pom.xml" || die "Not a Maven project: $1" 12 - info "Local Maven project: $1" - mkdir -p "LOCAL" - local projectDir="LOCAL/PROJECT" - ln -s "$1" "$projectDir" - else - # Treat specified project as a GAV. - info "Fetching project source" - local projectDir=$(resolveSource "$1" "$branch") - test $? -eq 0 || exit $? - fi - - # Get the project dependencies. - info "Determining project dependencies" - local deps="$(deps "$projectDir")" - test "$deps" || die "Cannot glean project dependencies" 7 - - local args="-Denforcer.skip" - - # Process the dependencies. - info "Processing project dependencies" - local dep - for dep in $deps - do - local g="$(groupId "$dep")" - local a="$(artifactId "$dep")" - local v="$(version "$dep")" - local c="$(classifier "$dep")" - test -z "$c" || continue # skip secondary artifacts - local gav="$g:$a:$v" - - test -z "$(isChanged "$gav")" && - args="$args \\\\\n -D$g.$a.version=$v -D$a.version=$v" - - if [ "$(isIncluded "$gav")" ] - then - info "$g:$a: resolving source for version $v" - resolveSource "$gav" >/dev/null - fi - done - - # Override versions of changed GAVs. - info "Processing changed components" - local TLS=, - local gav - for gav in $changes - do - local a="$(artifactId "$gav")" - local v="$(version "$gav")" - args="$args \\\\\n -D$a.version=$v" - done - unset TLS - - # Prune the build, if applicable. - test "$prune" && pruneReactor - - # Generate build scripts. - info "Generating build scripts" - generatePOM - echo "mvn $args \\\\\n dependency:list test \$@" > build.sh - generateScript - - # Build everything. - if [ "$skipBuild" ] - then - info "Skipping the build; run melt.sh to do it." - else - info "Building the project!" - # NB: All code is fresh; no need to clean. - sh melt.sh || die "Melt failed" 13 - fi - - info "Melt complete: $1" + # Fetch the project source code. + if [ -d "$1" ] + then + # Use local directory for the specified project. + test -d "$1" || die "No such directory: $1" 11 + test -f "$1/pom.xml" || die "Not a Maven project: $1" 12 + info "Local Maven project: $1" + mkdir -p "LOCAL" + local projectDir="LOCAL/PROJECT" + ln -s "$1" "$projectDir" + else + # Treat specified project as a GAV. + info "Fetching project source" + local projectDir=$(resolveSource "$1" "$branch") + test $? -eq 0 || exit $? + fi + + # Get the project dependencies. + info "Determining project dependencies" + local deps="$(deps "$projectDir")" + test "$deps" || die "Cannot glean project dependencies" 7 + + local args="-Denforcer.skip" + + # Process the dependencies. + info "Processing project dependencies" + local dep + for dep in $deps + do + local g="$(groupId "$dep")" + local a="$(artifactId "$dep")" + local v="$(version "$dep")" + local c="$(classifier "$dep")" + test -z "$c" || continue # skip secondary artifacts + local gav="$g:$a:$v" + + test -z "$(isChanged "$gav")" && + args="$args \\\\\n -D$g.$a.version=$v -D$a.version=$v" + + if [ "$(isIncluded "$gav")" ] + then + info "$g:$a: resolving source for version $v" + resolveSource "$gav" >/dev/null + fi + done + + # Override versions of changed GAVs. + info "Processing changed components" + local TLS=, + local gav + for gav in $changes + do + local a="$(artifactId "$gav")" + local v="$(version "$gav")" + args="$args \\\\\n -D$a.version=$v" + done + unset TLS + + # Prune the build, if applicable. + test "$prune" && pruneReactor + + # Generate build scripts. + info "Generating build scripts" + generatePOM + echo "mvn $args \\\\\n dependency:list test \$@" > build.sh + generateScript + + # Build everything. + if [ "$skipBuild" ] + then + info "Skipping the build; run melt.sh to do it." + else + info "Building the project!" + # NB: All code is fresh; no need to clean. + sh melt.sh || die "Melt failed" 13 + fi + + info "Melt complete: $1" } # -- Main -- From 6952ca701375c84960ebb5a036e42f1606629165 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 19 Sep 2022 19:54:12 -0500 Subject: [PATCH 18/80] melting-pot: align append redirects to same file We could use a heredoc... but I resist! It's fine the way it is! <_< --- melting-pot.sh | 89 +++++++++++++++++++++++++------------------------- 1 file changed, 45 insertions(+), 44 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index 10465c0..ef0ed30 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -568,76 +568,77 @@ isProject() { # Generates an aggregator POM for all modules in the current directory. generatePOM() { - echo '' > pom.xml - echo '> pom.xml - echo ' xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"' >> pom.xml - echo ' xsi:schemaLocation="http://maven.apache.org/POM/4.0.0' >> pom.xml - echo ' https://maven.apache.org/xsd/maven-4.0.0.xsd">' >> pom.xml - echo ' 4.0.0' >> pom.xml - echo >> pom.xml - echo ' melting-pot' >> pom.xml - echo ' melting-pot' >> pom.xml - echo ' 0.0.0-SNAPSHOT' >> pom.xml - echo ' pom' >> pom.xml - echo >> pom.xml - echo ' Melting Pot' >> pom.xml - echo >> pom.xml - echo ' ' >> pom.xml + echo '' > pom.xml + echo '> pom.xml + echo ' xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"' >> pom.xml + echo ' xsi:schemaLocation="http://maven.apache.org/POM/4.0.0' >> pom.xml + echo ' https://maven.apache.org/xsd/maven-4.0.0.xsd">' >> pom.xml + echo ' 4.0.0' >> pom.xml + echo >> pom.xml + echo ' melting-pot' >> pom.xml + echo ' melting-pot' >> pom.xml + echo ' 0.0.0-SNAPSHOT' >> pom.xml + echo ' pom' >> pom.xml + echo >> pom.xml + echo ' Melting Pot' >> pom.xml + echo >> pom.xml + echo ' ' >> pom.xml local dir for dir in */* do if [ "$(isProject "$dir")" ] then - echo " $dir" >> pom.xml + echo " $dir" >> pom.xml else # Check for a child component of a multi-module project. local childDir="$dir/$(basename "$dir")" test "$(isProject "$childDir")" && - echo " $childDir" >> pom.xml + echo " $childDir" >> pom.xml fi done - echo ' ' >> pom.xml - echo '' >> pom.xml + echo ' ' >> pom.xml + echo '' >> pom.xml } # Generates melt.sh script for all modules in the current directory. generateScript() { - echo '#!/bin/sh' > melt.sh - echo 'trap "exit" INT' >> melt.sh - echo 'echo "Melting the pot..."' >> melt.sh - echo 'dir=$(pwd)' >> melt.sh - echo 'failCount=0' >> melt.sh - echo 'for f in \' >> melt.sh + echo '#!/bin/sh' > melt.sh + echo 'trap "exit" INT' >> melt.sh + echo 'echo "Melting the pot..."' >> melt.sh + echo 'dir=$(pwd)' >> melt.sh + echo 'failCount=0' >> melt.sh + echo 'for f in \' >> melt.sh local dir for dir in */* do if [ "$(isProject "$dir")" ] then - echo " $dir \\" >> melt.sh + echo " $dir \\" >> melt.sh else # Check for a child component of a multi-module project. local childDir="$dir/$(basename "$dir")" test "$(isProject "$childDir")" && - echo " $childDir \\" >> melt.sh + echo " $childDir \\" >> melt.sh fi done - echo >> melt.sh - echo 'do' >> melt.sh - echo ' # If the build passed previously, don'\''t repeat it.' >> melt.sh - echo ' test -f "$f/build.log" &&' >> melt.sh - echo ' tail -n6 "$f/build.log" | grep -qF '\''[INFO] BUILD SUCCESS'\'' &&' >> melt.sh - echo ' echo "[SKIPPED] $f (already succeeded)" && continue' >> melt.sh - echo >> melt.sh - echo ' cd "$f"' >> melt.sh - echo ' sh "$dir/build.sh" >build.log 2>&1 &&' >> melt.sh - echo ' echo "[SUCCESS] $f" || {' >> melt.sh - echo ' echo "[FAILURE] $f"' >> melt.sh - echo ' failCount=$((failCount+1))' >> melt.sh - echo ' }' >> melt.sh - echo ' cd - >/dev/null' >> melt.sh - echo 'done' >> melt.sh - echo 'test "$failCount" -gt 255 && failCount=255' >> melt.sh - echo 'exit "$failCount"' >> melt.sh + echo >> melt.sh + echo 'do' >> melt.sh + echo ' # If the build passed previously, don'\''t repeat it.' >> melt.sh + echo ' test -f "$f/build.log" &&' >> melt.sh + echo ' tail -n6 "$f/build.log" |' >> melt.sh + echo ' grep -qF '\''[INFO] BUILD SUCCESS'\'' &&' >> melt.sh + echo ' echo "[SKIPPED] $f (already succeeded)" && continue' >> melt.sh + echo >> melt.sh + echo ' cd "$f"' >> melt.sh + echo ' sh "$dir/build.sh" >build.log 2>&1 &&' >> melt.sh + echo ' echo "[SUCCESS] $f" || {' >> melt.sh + echo ' echo "[FAILURE] $f"' >> melt.sh + echo ' failCount=$((failCount+1))' >> melt.sh + echo ' }' >> melt.sh + echo ' cd - >/dev/null' >> melt.sh + echo 'done' >> melt.sh + echo 'test "$failCount" -gt 255 && failCount=255' >> melt.sh + echo 'exit "$failCount"' >> melt.sh } # Creates and tests an appropriate multi-module reactor for the given project. From c59f4fc88147e872566379b1719b816fedc09f02 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 19 Sep 2022 15:52:51 -0500 Subject: [PATCH 19/80] melting-pot: cache previously successful dep sets --- melting-pot.sh | 121 ++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 105 insertions(+), 16 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index ef0ed30..9dbc914 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -600,12 +600,12 @@ generatePOM() { echo '' >> pom.xml } -# Generates melt.sh script for all modules in the current directory. -generateScript() { +# Generates melt.sh and helper scripts for all modules in the current directory. +generateScripts() { echo '#!/bin/sh' > melt.sh echo 'trap "exit" INT' >> melt.sh echo 'echo "Melting the pot..."' >> melt.sh - echo 'dir=$(pwd)' >> melt.sh + echo 'dir=$(cd "$(dirname "$0")" && pwd)' >> melt.sh echo 'failCount=0' >> melt.sh echo 'for f in \' >> melt.sh local dir @@ -623,22 +623,109 @@ generateScript() { done echo >> melt.sh echo 'do' >> melt.sh - echo ' # If the build passed previously, don'\''t repeat it.' >> melt.sh - echo ' test -f "$f/build.log" &&' >> melt.sh - echo ' tail -n6 "$f/build.log" |' >> melt.sh - echo ' grep -qF '\''[INFO] BUILD SUCCESS'\'' &&' >> melt.sh - echo ' echo "[SKIPPED] $f (already succeeded)" && continue' >> melt.sh - echo >> melt.sh + echo ' if [ "$("$dir/prior-success.sh" "$f")" ]' >> melt.sh + echo ' then' >> melt.sh + echo ' echo "[SKIPPED] $f (prior success)"' >> melt.sh + echo ' continue' >> melt.sh + echo ' fi' >> melt.sh echo ' cd "$f"' >> melt.sh - echo ' sh "$dir/build.sh" >build.log 2>&1 &&' >> melt.sh - echo ' echo "[SUCCESS] $f" || {' >> melt.sh - echo ' echo "[FAILURE] $f"' >> melt.sh - echo ' failCount=$((failCount+1))' >> melt.sh - echo ' }' >> melt.sh + echo ' "$dir/build.sh" >build.log 2>&1 && {' >> melt.sh + echo ' echo "[SUCCESS] $f"' >> melt.sh + echo ' "$dir/record-success.sh" "$f"' >> melt.sh + echo ' } || {' >> melt.sh + echo ' echo "[FAILURE] $f"' >> melt.sh + echo ' failCount=$((failCount+1))' >> melt.sh + echo ' }' >> melt.sh echo ' cd - >/dev/null' >> melt.sh echo 'done' >> melt.sh echo 'test "$failCount" -gt 255 && failCount=255' >> melt.sh echo 'exit "$failCount"' >> melt.sh + chmod +x melt.sh + +cat <<\PRIOR > prior-success.sh +#!/bin/sh +test "$1" || { echo "[ERROR] Please specify project to check."; exit 1; } + +stderr() { >&2 echo "$@"; } +debug() { test "$DEBUG" && stderr "[DEBUG] $@"; } +warn() { stderr "[WARNING] $@"; } + +dir=$(cd "$(dirname "$0")" && pwd) + +# Check build.log for BUILD SUCCESS. +buildLog="$dir/$1/build.log" +test -f "$buildLog" && tail -n6 "$buildLog" | grep -qF '[INFO] BUILD SUCCESS' && { + echo "build.log" + exit 0 +} + +# Check success.log for matching dependency configuration. +successLog="$HOME/.cache/scijava/melting-pot/$1.success.log" +test -f "$successLog" || exit 0 +success= +for deps in $(cat "$successLog") +do + debug "Checking dep config: $deps" + mismatch= + for dep in $(echo "$deps" | tr ',' '\n') + do + # g:a:p:v:s -> -Dg.a.version=v + s=${dep##*:} + case "$s" in + test) continue ;; # skip test dependencies + esac + gapv=${dep%:*} + g=${gapv%%:*} + apv=${gapv#*:} + a=${apv%%:*} + v=${apv##*:} + arg=" -D$g.$a.version=$v " + if ! grep -Fq "$arg" "$dir/build.sh" + then + # G:A property is not set to this V. + # Now check if the property is even declared. + if grep -Fq " -D$g.$a.version=" "$dir/build.sh" + then + # G:A version is mismatched. + debug "$dep [MISMATCH]" + mismatch=1 + break + else + # G:A version is not managed. + warn "Unmanaged dependency: $dep" + fi + fi + done + test "$mismatch" || { + success=$deps + break + } +done +echo "$success" +PRIOR + chmod +x prior-success.sh + +cat <<\RECORD > record-success.sh +#!/bin/sh +test "$1" || { echo "[ERROR] Please specify project to update."; exit 1; } + +dir=$(cd "$(dirname "$0")" && pwd) +buildLog="$dir/$1/build.log" +test -f "$buildLog" || exit 1 +successLog="$HOME/.cache/scijava/melting-pot/$1.success.log" +mkdir -p "$(dirname "$successLog")" + +# Record dependency configuration of successful build. +deps=$(grep "^\[INFO\] " "$buildLog" | + sed -e "s/^.\{10\}//" -e "s/ -- .*//" | + sort | tr '\n' ',') +test -f "$successLog" && grep -Fxq "$deps" "$successLog" || { + echo "$deps" > "$successLog".new + test -f "$successLog" && cat "$successLog" >> "$successLog".new + mv -f "$successLog".new "$successLog" +} +RECORD + chmod +x record-success.sh } # Creates and tests an appropriate multi-module reactor for the given project. @@ -710,8 +797,10 @@ meltDown() { # Generate build scripts. info "Generating build scripts" generatePOM - echo "mvn $args \\\\\n dependency:list test \$@" > build.sh - generateScript + echo "#!/bin/sh" > build.sh + echo "mvn $args \\\\\n dependency:list test \$@" >> build.sh + chmod +x build.sh + generateScripts # Build everything. if [ "$skipBuild" ] From 15b885760743000be4059b12dbc3936344160540 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 20 Sep 2022 10:20:02 -0500 Subject: [PATCH 20/80] melting-pot: do not generate an aggregator POM Unfortunately, due to a bug in Maven, this aggregator POM does not work in an intuitive way. In particular, executing build.sh in the top-level melting-pot directory does not pin the versions as desired within each submodule. Shockingly, they often are different versions than when you cd into a particular groupId/artifactId directory and run ../../build.sh from there. The problem is probably related to: * https://stackoverflow.com/q/45041888/1207769 * https://issues.apache.org/jira/browse/MNG-6141 * https://issues.apache.org/jira/browse/MNG-5761 Anyway, the melting pot does not use this aggregator POM anymore anyway, so let's refrain from generating it, to avoid any potential reliance on something that's broken. --- melting-pot.sh | 35 ----------------------------------- 1 file changed, 35 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index 9dbc914..557450f 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -566,40 +566,6 @@ isProject() { test "$1" = "LOCAL/PROJECT" -o "$a" = "$(basename "$1")" && echo 1 } -# Generates an aggregator POM for all modules in the current directory. -generatePOM() { - echo '' > pom.xml - echo '> pom.xml - echo ' xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"' >> pom.xml - echo ' xsi:schemaLocation="http://maven.apache.org/POM/4.0.0' >> pom.xml - echo ' https://maven.apache.org/xsd/maven-4.0.0.xsd">' >> pom.xml - echo ' 4.0.0' >> pom.xml - echo >> pom.xml - echo ' melting-pot' >> pom.xml - echo ' melting-pot' >> pom.xml - echo ' 0.0.0-SNAPSHOT' >> pom.xml - echo ' pom' >> pom.xml - echo >> pom.xml - echo ' Melting Pot' >> pom.xml - echo >> pom.xml - echo ' ' >> pom.xml - local dir - for dir in */* - do - if [ "$(isProject "$dir")" ] - then - echo " $dir" >> pom.xml - else - # Check for a child component of a multi-module project. - local childDir="$dir/$(basename "$dir")" - test "$(isProject "$childDir")" && - echo " $childDir" >> pom.xml - fi - done - echo ' ' >> pom.xml - echo '' >> pom.xml -} - # Generates melt.sh and helper scripts for all modules in the current directory. generateScripts() { echo '#!/bin/sh' > melt.sh @@ -796,7 +762,6 @@ meltDown() { # Generate build scripts. info "Generating build scripts" - generatePOM echo "#!/bin/sh" > build.sh echo "mvn $args \\\\\n dependency:list test \$@" >> build.sh chmod +x build.sh From 47cac66ff99d845340c64e2437ceec2c3ceaad95 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 20 Sep 2022 15:24:24 -0500 Subject: [PATCH 21/80] melting-pot: fix macOS bug in record-success func BSD grep's bug, really. Let's work around it! --- melting-pot.sh | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index 557450f..582e1e2 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -675,6 +675,22 @@ cat <<\RECORD > record-success.sh #!/bin/sh test "$1" || { echo "[ERROR] Please specify project to update."; exit 1; } +containsLine() { + pattern=$1 + file=$2 + test -f "$file" || return + # HACK: The obvious way to do this is: + # + # grep -qxF "$pattern" "$file" + # + # Unfortunately, BSD grep dies with "out of memory" when the pattern is 5111 + # characters or longer. So let's do something needlessly complex instead! + cat "$file" | while read line + do + test "$pattern" = "$line" && echo 1 && break + done +} + dir=$(cd "$(dirname "$0")" && pwd) buildLog="$dir/$1/build.log" test -f "$buildLog" || exit 1 @@ -685,11 +701,12 @@ mkdir -p "$(dirname "$successLog")" deps=$(grep "^\[INFO\] " "$buildLog" | sed -e "s/^.\{10\}//" -e "s/ -- .*//" | sort | tr '\n' ',') -test -f "$successLog" && grep -Fxq "$deps" "$successLog" || { +if [ -z "$(containsLine "$deps" "$successLog")" ] +then echo "$deps" > "$successLog".new test -f "$successLog" && cat "$successLog" >> "$successLog".new mv -f "$successLog".new "$successLog" -} +fi RECORD chmod +x record-success.sh } From 3aaa02a57c36501547711496657ba901ba1217c9 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 20 Sep 2022 15:25:55 -0500 Subject: [PATCH 22/80] melting-pot: do not clone repos with prior success This speeds up the melting pot generation even more! \^_^/ --- melting-pot.sh | 58 ++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 42 insertions(+), 16 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index 582e1e2..872ed1d 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -566,8 +566,8 @@ isProject() { test "$1" = "LOCAL/PROJECT" -o "$a" = "$(basename "$1")" && echo 1 } -# Generates melt.sh and helper scripts for all modules in the current directory. -generateScripts() { +# Generates melt.sh, covering all projects in the current directory. +generateMeltScript() { echo '#!/bin/sh' > melt.sh echo 'trap "exit" INT' >> melt.sh echo 'echo "Melting the pot..."' >> melt.sh @@ -607,8 +607,11 @@ generateScripts() { echo 'test "$failCount" -gt 255 && failCount=255' >> melt.sh echo 'exit "$failCount"' >> melt.sh chmod +x melt.sh +} -cat <<\PRIOR > prior-success.sh +# Generates helper scripts, including prior-success.sh and record-success.sh. +generateHelperScripts() { + cat <<\PRIOR > prior-success.sh #!/bin/sh test "$1" || { echo "[ERROR] Please specify project to check."; exit 1; } @@ -671,7 +674,7 @@ echo "$success" PRIOR chmod +x prior-success.sh -cat <<\RECORD > record-success.sh + cat <<\RECORD > record-success.sh #!/bin/sh test "$1" || { echo "[ERROR] Please specify project to update."; exit 1; } @@ -738,6 +741,10 @@ meltDown() { local deps="$(deps "$projectDir")" test "$deps" || die "Cannot glean project dependencies" 7 + # Generate helper scripts. We need prior-success.sh + # to decide whether to include each component. + generateHelperScripts + local args="-Denforcer.skip" # Process the dependencies. @@ -754,12 +761,6 @@ meltDown() { test -z "$(isChanged "$gav")" && args="$args \\\\\n -D$g.$a.version=$v -D$a.version=$v" - - if [ "$(isIncluded "$gav")" ] - then - info "$g:$a: resolving source for version $v" - resolveSource "$gav" >/dev/null - fi done # Override versions of changed GAVs. @@ -774,15 +775,40 @@ meltDown() { done unset TLS - # Prune the build, if applicable. - test "$prune" && pruneReactor - - # Generate build scripts. - info "Generating build scripts" + # Generate build script. + info "Generating build.sh script" echo "#!/bin/sh" > build.sh echo "mvn $args \\\\\n dependency:list test \$@" >> build.sh chmod +x build.sh - generateScripts + + # Clone source code. + info "Cloning source code" + for dep in $deps + do + local g="$(groupId "$dep")" + local a="$(artifactId "$dep")" + local v="$(version "$dep")" + local c="$(classifier "$dep")" + test -z "$c" || continue # skip secondary artifacts + local gav="$g:$a:$v" + if [ "$(isIncluded "$gav")" ] + then + if [ "$(./prior-success.sh "$g/$a")" ] + then + info "$g:$a: skipping version $v due to prior successful build" + continue + fi + info "$g:$a: resolving source for version $v" + resolveSource "$gav" >/dev/null + fi + done + + # Prune the build, if applicable. + test "$prune" && pruneReactor + + # Generate melt script. + info "Generating melt.sh script" + generateMeltScript # Build everything. if [ "$skipBuild" ] From 2f1c2e3159d47a9ae066f28292569efe1f7e3ccc Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 20 Sep 2022 16:19:45 -0500 Subject: [PATCH 23/80] melting-pot: explain why we prepend configs --- melting-pot.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/melting-pot.sh b/melting-pot.sh index 872ed1d..d17b5c0 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -706,6 +706,10 @@ deps=$(grep "^\[INFO\] " "$buildLog" | sort | tr '\n' ',') if [ -z "$(containsLine "$deps" "$successLog")" ] then + # NB: *Prepend*, rather than append, the new successful configuration. + # We do this because it is more likely this new configuration will be + # encountered again in the future, as dependency versions are highly + # likely to repeatedly increment, rather than moving backwards. echo "$deps" > "$successLog".new test -f "$successLog" && cat "$successLog" >> "$successLog".new mv -f "$successLog".new "$successLog" From 3b686deb8d4bda0eed7bc1f0fa963e344f3efcd7 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 22 Sep 2022 12:58:43 -0500 Subject: [PATCH 24/80] melting-pot: handle optional deps + empty dep sets Optional dependencies now have their scope transformed from "runtime (optional)" to "runtime-optional", to eschew spaces. Empty dep sets no longer emit an "Unmanaged dependency: none" warning. --- melting-pot.sh | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index d17b5c0..031febc 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -642,6 +642,7 @@ do s=${dep##*:} case "$s" in test) continue ;; # skip test dependencies + none) continue ;; # empty dependency config esac gapv=${dep%:*} g=${gapv%%:*} @@ -701,8 +702,8 @@ successLog="$HOME/.cache/scijava/melting-pot/$1.success.log" mkdir -p "$(dirname "$successLog")" # Record dependency configuration of successful build. -deps=$(grep "^\[INFO\] " "$buildLog" | - sed -e "s/^.\{10\}//" -e "s/ -- .*//" | +deps=$(grep '^\[INFO\] ' "$buildLog" | + sed -e 's/^.\{10\}//' -e 's/ -- .*//' -e 's/ (\([^)]*\))/-\1/' | sort | tr '\n' ',') if [ -z "$(containsLine "$deps" "$successLog")" ] then From 398e4e04b7e967e60eec971c48656570ef5d6eb3 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Fri, 23 Sep 2022 08:24:15 -0500 Subject: [PATCH 25/80] Add project (Un)license --- UNLICENSE | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 UNLICENSE diff --git a/UNLICENSE b/UNLICENSE new file mode 100644 index 0000000..68a49da --- /dev/null +++ b/UNLICENSE @@ -0,0 +1,24 @@ +This is free and unencumbered software released into the public domain. + +Anyone is free to copy, modify, publish, use, compile, sell, or +distribute this software, either in source code form or as a compiled +binary, for any purpose, commercial or non-commercial, and by any +means. + +In jurisdictions that recognize copyright laws, the author or authors +of this software dedicate any and all copyright interest in the +software to the public domain. We make this dedication for the benefit +of the public at large and to the detriment of our heirs and +successors. We intend this dedication to be an overt act of +relinquishment in perpetuity of all present and future rights to this +software under copyright law. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. +IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY CLAIM, DAMAGES OR +OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, +ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR +OTHER DEALINGS IN THE SOFTWARE. + +For more information, please refer to From 26cf1f37b2469de711754b8c1bf769f4d3147457 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Fri, 23 Sep 2022 08:24:36 -0500 Subject: [PATCH 26/80] Add mailmap, for better "git shortlog -nse" --- .mailmap | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 .mailmap diff --git a/.mailmap b/.mailmap new file mode 100644 index 0000000..b48538b --- /dev/null +++ b/.mailmap @@ -0,0 +1,2 @@ +Jack Yuan <55564584+yongleyuan@users.noreply.github.com> +Tobias Pietzsch From 9246a849557407304e58f8a116ad183330145595 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Fri, 23 Sep 2022 09:35:53 -0500 Subject: [PATCH 27/80] melting-pot: give better info about mismatches --- melting-pot.sh | 34 ++++++++++++++++++++++------------ 1 file changed, 22 insertions(+), 12 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index 031febc..e7db32e 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -617,6 +617,7 @@ test "$1" || { echo "[ERROR] Please specify project to check."; exit 1; } stderr() { >&2 echo "$@"; } debug() { test "$DEBUG" && stderr "[DEBUG] $@"; } +info() { stderr "[INFO] $@"; } warn() { stderr "[WARNING] $@"; } dir=$(cd "$(dirname "$0")" && pwd) @@ -631,10 +632,12 @@ test -f "$buildLog" && tail -n6 "$buildLog" | grep -qF '[INFO] BUILD SUCCESS' && # Check success.log for matching dependency configuration. successLog="$HOME/.cache/scijava/melting-pot/$1.success.log" test -f "$successLog" || exit 0 +row=1 +mismatch1= success= for deps in $(cat "$successLog") do - debug "Checking dep config: $deps" + debug "$1: Checking dep config: $deps" mismatch= for dep in $(echo "$deps" | tr ',' '\n') do @@ -649,29 +652,36 @@ do apv=${gapv#*:} a=${apv%%:*} v=${apv##*:} - arg=" -D$g.$a.version=$v " - if ! grep -Fq "$arg" "$dir/build.sh" + bomV=$(grep -o " -D$g\.$a\.version=[^ ]*" "$dir/build.sh" | sed 's;.*=;;') + if [ "$bomV" != "$v" ] then # G:A property is not set to this V. # Now check if the property is even declared. - if grep -Fq " -D$g.$a.version=" "$dir/build.sh" + if [ "$bomV" ] then # G:A version is mismatched. - debug "$dep [MISMATCH]" - mismatch=1 - break + mismatch="$mismatch\n* $g:$a:$v != $bomV" else - # G:A version is not managed. - warn "Unmanaged dependency: $dep" + # G:A version is not pinned. + warn "$1: Unpinned dependency: $dep" fi fi done - test "$mismatch" || { + if [ "$mismatch" ] + then + test "$row" -eq 1 && mismatch1=$mismatch || + debug "$1: Mismatched dependencies (vs. success #$row):$mismatch" + else success=$deps break - } + fi + row=$((row+1)) done -echo "$success" +test "$success" && echo "$success" || { + test "$mismatch1" && + info "$1: Mismatched dependencies:$mismatch1" || + info "$1: No prior successes" +} PRIOR chmod +x prior-success.sh From c2b56cf623ef1ba4b00d9c06ef25713246cd0893 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 26 Sep 2022 08:36:19 -0500 Subject: [PATCH 28/80] github-actionify: delete .travis more robustly Even with -f, git complains when the file does not exist. --- abd | 0 github-actionify.sh | 3 ++- 2 files changed, 2 insertions(+), 1 deletion(-) create mode 100644 abd diff --git a/abd b/abd new file mode 100644 index 0000000..e69de29 diff --git a/github-actionify.sh b/github-actionify.sh index e16c0c4..a52079d 100755 --- a/github-actionify.sh +++ b/github-actionify.sh @@ -240,7 +240,8 @@ EOL fi # remove old Travis CI configuration - $EXEC git rm -rf .travis.yml .travis + test ! -e .travis.yml || $EXEC git rm -rf .travis.yml + test ! -e .travis || $EXEC git rm -rf .travis $EXEC git diff-index --quiet HEAD -- && info "No old CI configuration to remove." || $EXEC git commit -m "${msgPrefix}remove Travis CI configuration" From 00b78cbe0dd4aecf612bdb78bfa5c0598e83c8e6 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 26 Sep 2022 09:05:43 -0500 Subject: [PATCH 29/80] release-version: clean up release-only files To avoid these becoming committed to the mainline repository branch, we now do two things: 1. If they are present at the beginning, remove them. 2. After the release process completes, delete the newly generated ones. --- release-version.sh | 22 ++++++++++++++++++++-- 1 file changed, 20 insertions(+), 2 deletions(-) diff --git a/release-version.sh b/release-version.sh index 71d43d8..d8f661a 100755 --- a/release-version.sh +++ b/release-version.sh @@ -208,6 +208,21 @@ test "$FETCH_HEAD" = HEAD || test "$FETCH_HEAD" = "$(git merge-base $FETCH_HEAD $HEAD)" || die "'$defaultBranch' is not up-to-date" +# Check for release-only files committed to the main branch. +for release_file in release.properties pom.xml.releaseBackup +do + if [ -e "$release_file" ] + then + echo "==========================================================================" + echo "NOTE: $release_file was committed to source control. Removing now." + echo "==========================================================================" + git rm -rf "$release_file" && + git commit "$release_file" \ + -m 'Remove $release_file' \ + -m 'It should only exist on release tags.' + fi +done + # Ensure that schema location URL uses HTTPS, not HTTP. if grep -q http://maven.apache.org/xsd/maven-4.0.0.xsd pom.xml >/dev/null 2>/dev/null then @@ -312,5 +327,8 @@ $DRY_RUN git checkout @{-1} && if test -z "$SKIP_PUSH" then $DRY_RUN git push "$REMOTE" HEAD $tag -fi || -exit +fi + +# Remove files generated by the release process. They can end up +# committed to the mainline branch and hosing up later releases. +$DRY_RUN rm -f release.properties pom.xml.releaseBackup From 3e44d6041d061a4018d223859f1819b4993eb059 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 26 Sep 2022 15:00:35 -0500 Subject: [PATCH 30/80] melting-pot: handle colored Maven output When caching successful dependency configurations, we scrape the dependency:list output from build.log. But if the version of Maven is relatively new, then colored output might be enabled, potentially hosing up our rather fragile parsing logic. This commit broadens the regexes used to work with both colored and colorless dependency:list output. --- melting-pot.sh | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index e7db32e..251c73b 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -478,8 +478,8 @@ resolveSource() { # Gets the list of dependencies for the project in the CWD. deps() { cd "$1" || die "No such directory: $1" 16 - debug "mvn -DincludeScope=runtime -B dependency:list" - local depList="$(mvn -DincludeScope=runtime -B dependency:list)" || + debug "mvn -B -DincludeScope=runtime dependency:list" + local depList="$(mvn -B -DincludeScope=runtime dependency:list)" || die "Problem fetching dependencies!" 5 echo "$depList" | grep '^\[INFO\] [^ ]' | sed 's/\[INFO\] //' | sed 's/ .*//' | sort @@ -712,8 +712,8 @@ successLog="$HOME/.cache/scijava/melting-pot/$1.success.log" mkdir -p "$(dirname "$successLog")" # Record dependency configuration of successful build. -deps=$(grep '^\[INFO\] ' "$buildLog" | - sed -e 's/^.\{10\}//' -e 's/ -- .*//' -e 's/ (\([^)]*\))/-\1/' | +deps=$(grep '^\[[^ ]*INFO[^ ]*\] ' "$buildLog" | + sed -e 's/^[^ ]* *//' -e 's/ -- .*//' -e 's/ (\([^)]*\))/-\1/' | sort | tr '\n' ',') if [ -z "$(containsLine "$deps" "$successLog")" ] then @@ -760,6 +760,9 @@ meltDown() { # to decide whether to include each component. generateHelperScripts + # NB: We do *not* include -B here, because we want build.sh to preserve + # colored output if the version of Maven is new enough. We will take care + # elsewhere when parsing it to be flexible about whether colors are present. local args="-Denforcer.skip" # Process the dependencies. From 766d5995be7e2fdb13975f9cf54001edc44ba570 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 26 Sep 2022 15:06:23 -0500 Subject: [PATCH 31/80] melting-pot: handle colored Maven output even more The prior-success.sh script also needs to be aware of it when parsing the build.log for "BUILD SUCCESS". --- melting-pot.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/melting-pot.sh b/melting-pot.sh index 251c73b..9ffe79c 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -624,7 +624,7 @@ dir=$(cd "$(dirname "$0")" && pwd) # Check build.log for BUILD SUCCESS. buildLog="$dir/$1/build.log" -test -f "$buildLog" && tail -n6 "$buildLog" | grep -qF '[INFO] BUILD SUCCESS' && { +test -f "$buildLog" && tail -n6 "$buildLog" | grep -qF 'BUILD SUCCESS' && { echo "build.log" exit 0 } From da025c00a26e9371e465b0475361de1739fc6df2 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 17 Oct 2022 14:05:51 -0500 Subject: [PATCH 32/80] class-version.sh: fix bug in first class detection So that files such as: META-INF/services/ch.qos.logback.classic.spi.Configurator hypothetically speaking, of course ;-) are not misconstrued as classes. --- class-version.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/class-version.sh b/class-version.sh index 8da1057..a1ecfe7 100755 --- a/class-version.sh +++ b/class-version.sh @@ -49,7 +49,7 @@ class_version() { } first_class() { - jar tf "$1" | grep '\.class' | head -n 1 + jar tf "$1" | grep '\.class$' | head -n 1 } for file in "$@" From 09a06a29fbe213d6046497eceb7e0315eba025a3 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Fri, 21 Oct 2022 12:08:54 -0500 Subject: [PATCH 33/80] melting-pot: highlight messages with ANSI color To make the deluge of output easier for humans to parse. --- melting-pot.sh | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index 9ffe79c..45ea94d 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -69,11 +69,11 @@ meltingPotCache="$HOME/.cache/scijava/melting-pot" # -- Functions -- -stderr() { >&2 echo "$@"; } +stderr() { >&2 printf "$@\n"; } debug() { test "$debug" && stderr "+ $@"; } -info() { test "$verbose" && stderr "[INFO] $@"; } -warn() { stderr "[WARNING] $@"; } -error() { stderr "[ERROR] $@"; } +info() { test "$verbose" && stderr "\e[0;37m[INFO] $@\e[0m"; } +warn() { stderr "\e[0;33m[WARNING] $@\e[0m"; } +error() { stderr "\e[0;31m[ERROR] $@\e[0m"; } die() { error $1; exit $2; } unknownArg() { error "Unknown option: $@"; usage=1; } @@ -591,15 +591,15 @@ generateMeltScript() { echo 'do' >> melt.sh echo ' if [ "$("$dir/prior-success.sh" "$f")" ]' >> melt.sh echo ' then' >> melt.sh - echo ' echo "[SKIPPED] $f (prior success)"' >> melt.sh + echo ' printf "\e[0;36m[SKIPPED] $f (prior success)\e[0m\n"' >> melt.sh echo ' continue' >> melt.sh echo ' fi' >> melt.sh echo ' cd "$f"' >> melt.sh echo ' "$dir/build.sh" >build.log 2>&1 && {' >> melt.sh - echo ' echo "[SUCCESS] $f"' >> melt.sh + echo ' printf "\e[0;32m[SUCCESS] $f\e[0m\n"' >> melt.sh echo ' "$dir/record-success.sh" "$f"' >> melt.sh echo ' } || {' >> melt.sh - echo ' echo "[FAILURE] $f"' >> melt.sh + echo ' printf "\e[0;31m[FAILURE] $f\e[0m\n"' >> melt.sh echo ' failCount=$((failCount+1))' >> melt.sh echo ' }' >> melt.sh echo ' cd - >/dev/null' >> melt.sh @@ -613,12 +613,15 @@ generateMeltScript() { generateHelperScripts() { cat <<\PRIOR > prior-success.sh #!/bin/sh -test "$1" || { echo "[ERROR] Please specify project to check."; exit 1; } +test "$1" || { + printf "\e[0;31m[ERROR] Please specify project to check.\e[0m\n" + exit 1 +} -stderr() { >&2 echo "$@"; } -debug() { test "$DEBUG" && stderr "[DEBUG] $@"; } -info() { stderr "[INFO] $@"; } -warn() { stderr "[WARNING] $@"; } +stderr() { >&2 printf "$@\n"; } +debug() { test "$DEBUG" && stderr "\e[0;37m[DEBUG] $@\e[0m"; } +info() { stderr "\e[0;37m[INFO] $@\e[0m"; } +warn() { stderr "\e[0;33m[WARNING] $@\e[0m"; } dir=$(cd "$(dirname "$0")" && pwd) @@ -687,7 +690,10 @@ PRIOR cat <<\RECORD > record-success.sh #!/bin/sh -test "$1" || { echo "[ERROR] Please specify project to update."; exit 1; } +test "$1" || { + printf "\e[0;31m[ERROR] Please specify project to update.\e[0m\n" + exit 1 +} containsLine() { pattern=$1 From dd4fd4847615af14b83d6003fdc549f647610a5f Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Wed, 26 Oct 2022 12:04:44 -0500 Subject: [PATCH 34/80] melting-pot: give friendlier dep change output The fact that the dependencies do not match since the last successful build, and therefore the component build is going to be executed, is not necessarily a bad thing. The word "mismatch" conveys something wrong; let's instead just point out what's changed, more neutrally. --- melting-pot.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index 45ea94d..fc96662 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -663,7 +663,7 @@ do if [ "$bomV" ] then # G:A version is mismatched. - mismatch="$mismatch\n* $g:$a:$v != $bomV" + mismatch="$mismatch\n* $g:$a:$v -> $bomV" else # G:A version is not pinned. warn "$1: Unpinned dependency: $dep" @@ -673,7 +673,7 @@ do if [ "$mismatch" ] then test "$row" -eq 1 && mismatch1=$mismatch || - debug "$1: Mismatched dependencies (vs. success #$row):$mismatch" + debug "$1: Dependency changes since success #$row:$mismatch" else success=$deps break @@ -682,7 +682,7 @@ do done test "$success" && echo "$success" || { test "$mismatch1" && - info "$1: Mismatched dependencies:$mismatch1" || + info "$1: Dependency changes since last success:$mismatch1" || info "$1: No prior successes" } PRIOR From b0c3d0f2fc65abbb96903ec0da944ba7209250a3 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 27 Oct 2022 10:18:47 -0500 Subject: [PATCH 35/80] Remove spurious empty file --- abd | 0 1 file changed, 0 insertions(+), 0 deletions(-) delete mode 100644 abd diff --git a/abd b/abd deleted file mode 100644 index e69de29..0000000 From 11f4d55c5f536da7621c146b6f1a327778e63e3f Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 27 Oct 2022 09:56:46 -0500 Subject: [PATCH 36/80] melting-pot: use correct remote repo parameter I cannot find any reference to a parameter called repoUrl for the maven-dependency-plugin. Not on the get-mojo doc page, and not in the maven-dependency-plugin source code. I think this feature was probably broken previously. Let's use remoteRepositories instead. --- melting-pot.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index fc96662..c80bb7c 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -320,13 +320,13 @@ downloadPOM() { local a="$(artifactId "$1")" local v="$(version "$1")" debug "mvn dependency:get \\ - -DrepoUrl=\"$remoteRepos\" \\ + -DremoteRepositories=\"$remoteRepos\" \\ -DgroupId=\"$g\" \\ -DartifactId=\"$a\" \\ -Dversion=\"$v\" \\ -Dpackaging=pom" mvn dependency:get \ - -DrepoUrl="$remoteRepos" \ + -DremoteRepositories="$remoteRepos" \ -DgroupId="$g" \ -DartifactId="$a" \ -Dversion="$v" \ From f0b105ebbaefeb2e3a2692791fc307a4b3b44bc4 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 27 Oct 2022 09:52:21 -0500 Subject: [PATCH 37/80] melting-pot: test deployed binaries, too In Java, binary API compatibility and source API compatibility are two distinct things. Recompiling the same source code against different dependencies than before might result in different bytecode, which works correctly at runtime with those new dependencies -- but when using the previously deployed bytecode against those new dependencies, errors happen because binary type signatures changed. For example, generic types might erase differently with the new dependencies, or different synthetic methods might be silently inserted. So, we need to check the new dependencies against both scenarios: 1. Do tests pass when run on the existing deployed component binary? 2. Does source compile with passing tests against the new dependencies? This commit is dedicated to Stephan Saalfeld. --- melting-pot.sh | 62 ++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 58 insertions(+), 4 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index c80bb7c..17a368c 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -466,6 +466,9 @@ resolveSource() { git clone "file://$cachedRepoDir" --branch "$scmBranch" --depth 1 "$destDir" 2> /dev/null || die "$1: could not clone branch '$scmBranch' from local cache" 15 + # Save the GAV string to a file, for convenience. + echo "$1" > "$destDir/gav" + # Now verify that the cloned pom.xml contains the expected version! local expectedVersion=$(version "$1") local actualVersion=$(xpath "$destDir/pom.xml" project version) @@ -784,7 +787,7 @@ meltDown() { local gav="$g:$a:$v" test -z "$(isChanged "$gav")" && - args="$args \\\\\n -D$g.$a.version=$v -D$a.version=$v" + args="$args \\\\\n -D$g.$a.version=$v -D$a.version=$v" done # Override versions of changed GAVs. @@ -795,14 +798,65 @@ meltDown() { do local a="$(artifactId "$gav")" local v="$(version "$gav")" - args="$args \\\\\n -D$a.version=$v" + args="$args \\\\\n -D$a.version=$v" done unset TLS # Generate build script. info "Generating build.sh script" - echo "#!/bin/sh" > build.sh - echo "mvn $args \\\\\n dependency:list test \$@" >> build.sh + echo '#!/bin/sh' > build.sh + echo >> build.sh + echo 'mvnPin() {' >> build.sh + echo " mvn $args \\\\\n \$@" >> build.sh + echo '}' >> build.sh + echo >> build.sh + echo 'unpackArtifact() {' >> build.sh + echo ' # Download and unpack the given artifact' >> build.sh + echo ' # (G:A:V) to the specified location.' >> build.sh + echo ' gav=$1' >> build.sh + echo ' out=$2' >> build.sh + echo >> build.sh + echo ' repoPrefix=$HOME/.m2/repository # TODO: generalize this' >> build.sh + echo ' g=${gav%%:*}; r=${gav#*:}; a=${r%%:*}; v=${r##*:}' >> build.sh + echo ' gavPath="$(echo "$g" | tr "." "/")/$a/$v/$a-$v"' >> build.sh + echo ' jarPath="$repoPrefix/$gavPath.jar"' >> build.sh + echo >> build.sh + echo ' # HACK: The best goal to use would be dependency:unpack,' >> build.sh + echo ' # or failing that, dependency:copy followed by jar xf.' >> build.sh + echo ' # But those goals do not support remoteRepositories;' >> build.sh + echo ' # see https://issues.apache.org/jira/browse/MDEP-390.' >> build.sh + echo ' # So we use dependency:get and then extract it by hand.' >> build.sh + echo ' mvnPin dependency:get \' >> build.sh + echo " -DremoteRepositories=\"$remoteRepos\" \\" >> build.sh + echo ' -Dartifact="$gav" &&' >> build.sh + echo >> build.sh + echo ' test -f "$jarPath" &&' >> build.sh + echo ' mkdir -p "$out" &&' >> build.sh + echo ' cd "$out" &&' >> build.sh + echo ' jar xf "$jarPath" &&' >> build.sh + echo ' cd - >/dev/null' >> build.sh + echo '}' >> build.sh + echo >> build.sh + echo 'mvnPin dependency:list &&' >> build.sh + echo >> build.sh + echo 'if [ -f gav ]' >> build.sh + echo 'then' >> build.sh + echo ' echo' >> build.sh + echo ' echo "================================================"' >> build.sh + echo ' echo "========= Testing with deployed binary ========="' >> build.sh + echo ' echo "================================================"' >> build.sh + echo ' unpackArtifact "$(cat gav)" target/classes &&' >> build.sh + echo ' mvnPin \' >> build.sh + echo ' -Dmaven.main.skip=true \' >> build.sh + echo ' -Dmaven.resources.skip=true \' >> build.sh + echo ' test $@' >> build.sh + echo 'fi &&' >> build.sh + echo >> build.sh + echo 'echo &&' >> build.sh + echo 'echo "================================================" &&' >> build.sh + echo 'echo "============ Rebuilding from source ============" &&' >> build.sh + echo 'echo "================================================" &&' >> build.sh + echo 'mvnPin clean test $@' >> build.sh chmod +x build.sh # Clone source code. From 3f0bbdba668d2f6e821e46327b8e34150740547d Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 1 Dec 2022 20:18:11 -0600 Subject: [PATCH 38/80] ci-build.sh: give hints when the build goes wrong In particular, people get javadoc errors a lot, which are confusing because when even a single javadoc error happens, Maven's javadoc plugin logs every javadoc *error and warning* at error level, resulting in the dreaded [ERROR] prefix all over all the javadoc output, burying the actual javadoc errors amongst the typically more plentiful warnings. You can actually find these errors though, because they are prefixed with 'error:' rather than 'warning:'. So we now grep them out and regurgitate them after declaring the build failed, as a helpful hint to why the failure occurred. And if no javadoc-specific errors are found, then we grep for common words indicating error or failure, and regurgitate those instead. And if none of those keywords are found, we declare failure at helping. This commit is dedicated to Nicolas Chiaruttini. --- ci-build.sh | 41 +++++++++++++++++++++++++++++++++++------ 1 file changed, 35 insertions(+), 6 deletions(-) diff --git a/ci-build.sh b/ci-build.sh index 78d957f..78d308f 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -19,6 +19,35 @@ checkSuccess() { # Log non-zero exit code. test $1 -eq 0 || echo "==> FAILED: EXIT CODE $1" 1>&2 + if [ $1 -ne 0 -a -f "$2" ] + then + # The operation failed and a log file was provided. + # Do some heuristics, because we like being helpful! + javadocErrors=$(grep error: "$2") + generalErrors=$(grep -i '\b\(errors\?\|fail\|failures\?\)\b' "$2") + if [ "$javadocErrors" ] + then + echo + echo '/----------------------------------------------------------\' + echo '| ci-build.sh analysis: I noticed probable javadoc errors: |' + echo '\----------------------------------------------------------/' + echo "$javadocErrors" + elif [ "$generalErrors" ] + then + echo + echo '/-------------------------------------------------------\' + echo '| ci-build.sh analysis: I noticed the following errors: |' + echo '\-------------------------------------------------------/' + echo "$generalErrors" + else + echo + echo '/----------------------------------------------------------------------\' + echo '| ci-build.sh analysis: I see no problems in the operation log. Sorry! |' + echo '\----------------------------------------------------------------------/' + echo + fi + fi + # Record the first non-zero exit code. test $success -eq 0 && success=$1 } @@ -208,19 +237,19 @@ EOL if [ "$deployOK" -a -f release.properties ]; then echo echo "== Cutting and deploying release version ==" - mvn $BUILD_ARGS release:perform - checkSuccess $? + BUILD_ARGS="$BUILD_ARGS release:perform" elif [ "$deployOK" ]; then echo echo "== Building and deploying main branch SNAPSHOT ==" - mvn -Pdeploy-to-scijava $BUILD_ARGS deploy - checkSuccess $? + BUILD_ARGS="-Pdeploy-to-scijava $BUILD_ARGS deploy" else echo echo "== Building the artifact locally only ==" - mvn $BUILD_ARGS install javadoc:javadoc - checkSuccess $? + BUILD_ARGS="$BUILD_ARGS install javadoc:javadoc" fi + # Check the build result. + { mvn $BUILD_ARGS; echo $? > exit-code; } | tee mvn-log + checkSuccess "$(cat exit-code)" mvn-log # --== POST-BUILD ACTIONS ==-- From a2c8fae403556b12996f2dc92b327b2e47170cc0 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Wed, 18 Jan 2023 11:15:29 -0600 Subject: [PATCH 39/80] release-version: fix Maven project attributes It is tempting to change all http: strings to https: in this day and age. But the strings that go at the top of a Maven POM are not actually URLs, they are namespace identifiers that must be those exact strings -- including the http: prefix -- for the document to be a valid Maven POM. See also: https://imagesc.zulipchat.com/#narrow/stream/327237-SciJava/topic/release-version.20error.20with.20xsi.20namespace --- release-version.sh | 33 +++++++++++++++++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/release-version.sh b/release-version.sh index d8f661a..9e736a2 100755 --- a/release-version.sh +++ b/release-version.sh @@ -231,12 +231,41 @@ then echo "=====================================================================" sed 's;http://maven.apache.org/xsd/maven-4.0.0.xsd;https://maven.apache.org/xsd/maven-4.0.0.xsd;' pom.xml > pom.new && mv -f pom.new pom.xml && - git commit pom.xml \ - -m 'POM: use HTTPS for schema location URL' \ + git commit pom.xml -m 'POM: use HTTPS for schema location URL' \ -m 'Maven no longer supports plain HTTP for the schema location.' \ -m 'And using HTTP now generates errors in Eclipse (and probably other IDEs).' fi +# Check project xmlns, xmlns:xsi, and xsi:schemaLocation attributes. +grep -q 'xmlns="http://maven.apache.org/POM/4.0.0"' pom.xml >/dev/null 2>/dev/null && + grep -q 'xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"' pom.xml >/dev/null 2>/dev/null && + grep -q 'xsi:schemaLocation="http://maven.apache.org/POM/4.0.0\b"' pom.xml >/dev/null 2>/dev/null || +{ + echo "=====================================================================" + echo "NOTE: Your POM's project attributes are incorrect. Fixing it now." + echo "=====================================================================" + sed 's;xmlns="[^"]*";xmlns="http://maven.apache.org/POM/4.0.0";' pom.xml > pom.new && + mv -f pom.new pom.xml && + sed 's;xmlns:xsi="[^"]*";xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";' pom.xml > pom.new && + mv -f pom.new pom.xml && + sed 's;xsi:schemaLocation="[^"]*";xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd";' pom.xml > pom.new && + mv -f pom.new pom.xml && + git commit pom.xml -m 'POM: fix project attributes' \ + -m 'The XML schema for Maven POMs is located at:' \ + -m ' https://maven.apache.org/xsd/maven-4.0.0.xsd' \ + -m 'Its XML namespace is the string:' \ + -m ' http://maven.apache.org/POM/4.0.0' \ + -m 'So that exact string must be the value of xmlns. It must also +match the first half of xsi:schemaLocation, which maps that +namespace to an actual URL online where the schema resides. +Otherwise, the document is not a Maven POM.' \ + -m 'Similarly, the xmlns:xsi attribute of an XML document declaring a +particular schema should always use the string identifier:' \ + -m ' http://www.w3.org/2001/XMLSchema-instance' \ + -m "because that's the namespace identifier for instances of an XML schema." \ + -m "For details, see the specification at: https://www.w3.org/TR/xmlschema-1/" +} + # Change forum references from forum.image.net to forum.image.sc. if grep -q 'https*://forum.imagej.net' pom.xml >/dev/null 2>/dev/null then From dc424ec2d8719b0dd139d5fc6b5a638648434032 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 19 Jan 2023 11:51:31 -0600 Subject: [PATCH 40/80] release-version: improve project attr checks --- release-version.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/release-version.sh b/release-version.sh index 9e736a2..2d4afb5 100755 --- a/release-version.sh +++ b/release-version.sh @@ -237,9 +237,9 @@ then fi # Check project xmlns, xmlns:xsi, and xsi:schemaLocation attributes. -grep -q 'xmlns="http://maven.apache.org/POM/4.0.0"' pom.xml >/dev/null 2>/dev/null && - grep -q 'xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"' pom.xml >/dev/null 2>/dev/null && - grep -q 'xsi:schemaLocation="http://maven.apache.org/POM/4.0.0\b"' pom.xml >/dev/null 2>/dev/null || +grep -qF 'xmlns="http://maven.apache.org/POM/4.0.0"' pom.xml >/dev/null 2>/dev/null && + grep -qF 'xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"' pom.xml >/dev/null 2>/dev/null && + grep -qF 'xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 ' pom.xml >/dev/null 2>/dev/null || { echo "=====================================================================" echo "NOTE: Your POM's project attributes are incorrect. Fixing it now." From 48b0aa7da358a3e674df0ae146a84656230ce5ad Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 19 Jan 2023 12:02:46 -0600 Subject: [PATCH 41/80] release-version: fix outdated comment --- release-version.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/release-version.sh b/release-version.sh index 2d4afb5..08f6422 100755 --- a/release-version.sh +++ b/release-version.sh @@ -340,8 +340,8 @@ test -n "$tag" && # HACK: SciJava projects use SSH (git@github.com:...) for developerConnection. # The release:perform command wants to use the developerConnection URL when # checking out the release tag. But reading from this URL requires credentials -# which we would rather Travis not need. So we replace the scm.url in the -# release.properties file to use the public (https://github.com/...) URL. +# which the CI system typically does not have. So we replace the scm.url in +# the release.properties file to use the public (https://github.com/...) URL. # This is OK, since release:perform does not need write access to the repo. $DRY_RUN sed -i.bak -e 's|^scm.url=scm\\:git\\:git@github.com\\:|scm.url=scm\\:git\\:https\\://github.com/|' release.properties && $DRY_RUN rm release.properties.bak && From 224ea6212ea05a204e161b285de1f0b55117111f Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 19 Jan 2023 12:03:05 -0600 Subject: [PATCH 42/80] release-version: add debugging statements Enable via: DEBUG=1 release-version.sh --- release-version.sh | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/release-version.sh b/release-version.sh index 08f6422..197ebe3 100755 --- a/release-version.sh +++ b/release-version.sh @@ -14,7 +14,12 @@ export LC_ALL=C # -- Functions -- -die () { +debug() { + test "$DEBUG" || return + echo "[DEBUG] $@" +} + +die() { echo "$*" >&2 exit 1 } @@ -111,6 +116,7 @@ Options include: " # -- Extract project details -- +debug "Extracting project details" echoArg='${project.version}:${license.licenseName}:${project.parent.groupId}:${project.parent.artifactId}:${project.parent.version}' projectDetails=$(mvn -N -Dexec.executable=echo -Dexec.args="$echoArg" exec:exec -q) @@ -124,10 +130,12 @@ licenseName=${projectDetails%%:*} parentGAV=${projectDetails#*:} # -- Sanity checks -- +debug "Performing sanity checks" # Check that we have push rights to the repository. if [ ! "$SKIP_PUSH" ] then + debug "Checking repository push rights" push=$(git remote -v | grep origin | grep '(push)') test "$push" || die 'No push URL found for remote origin. Please use "git remote -v" to double check your remote settings.' @@ -136,6 +144,7 @@ Please use "git remote set-url origin ..." to change it.' fi # Discern the version to release. +debug "Gleaning release version" pomVersion=${currentVersion%-SNAPSHOT} test "$VERSION" -o ! -t 0 || { printf 'Version? [%s]: ' "$pomVersion" @@ -161,6 +170,7 @@ test -f "$VALID_SEMVER_BUMP" || die "Missing helper script at '$VALID_SEMVER_BUMP' Do you have a full clone of https://github.com/scijava/scijava-scripts?" test "$SKIP_VERSION_CHECK" || { + debug "Checking conformance to SemVer" sh -$- "$VALID_SEMVER_BUMP" "$pomVersion" "$VERSION" || die "If you are sure, try again with --skip-version-check flag." } @@ -171,6 +181,7 @@ test -f "$MAVEN_HELPER" || die "Missing helper script at '$MAVEN_HELPER' Do you have a full clone of https://github.com/scijava/scijava-scripts?" test "$SKIP_VERSION_CHECK" -o "$parentGAV" != "${parentGAV#$}" || { + debug "Checking pom-scijava parent version" latestParentVersion=$(sh -$- "$MAVEN_HELPER" latest-version "$parentGAV") currentParentVersion=${parentGAV##*:} test "$currentParentVersion" = "$latestParentVersion" || @@ -180,11 +191,13 @@ Or if you know better, try again with --skip-version-check flag." } # Check that the working copy is clean. +debug "Checking if working copy is clean" no_changes_pending || die 'There are uncommitted changes!' test -z "$(git ls-files -o --exclude-standard)" || die 'There are untracked files! Please stash them before releasing.' # Discern default branch. +debug "Discerning default branch" currentBranch=$(git rev-parse --abbrev-ref --symbolic-full-name HEAD) upstreamBranch=$(git rev-parse --abbrev-ref --symbolic-full-name @{u}) remote=${upstreamBranch%/*} @@ -192,6 +205,7 @@ defaultBranch=$(git remote show "$remote" | grep "HEAD branch" | sed 's/.*: //') # Check that we are on the main branch. test "$SKIP_BRANCH_CHECK" || { + debug "Checking current branch" test "$currentBranch" = "$defaultBranch" || die "Non-default branch: $currentBranch. If you are certain you want to release from this branch, try again with --skip-branch-check flag." @@ -201,6 +215,7 @@ try again with --skip-branch-check flag." REMOTE="${REMOTE:-$remote}" # Check that the main branch isn't behind the upstream branch. +debug "Ensuring local branch is up-to-date" HEAD="$(git rev-parse HEAD)" && git fetch "$REMOTE" "$defaultBranch" && FETCH_HEAD="$(git rev-parse FETCH_HEAD)" && @@ -209,6 +224,7 @@ test "$FETCH_HEAD" = "$(git merge-base $FETCH_HEAD $HEAD)" || die "'$defaultBranch' is not up-to-date" # Check for release-only files committed to the main branch. +debug "Checking for spurious release-only files" for release_file in release.properties pom.xml.releaseBackup do if [ -e "$release_file" ] @@ -224,6 +240,7 @@ do done # Ensure that schema location URL uses HTTPS, not HTTP. +debug "Checking that schema location URL uses HTTPS" if grep -q http://maven.apache.org/xsd/maven-4.0.0.xsd pom.xml >/dev/null 2>/dev/null then echo "=====================================================================" @@ -237,6 +254,7 @@ then fi # Check project xmlns, xmlns:xsi, and xsi:schemaLocation attributes. +debug "Checking correctness of POM project XML attributes" grep -qF 'xmlns="http://maven.apache.org/POM/4.0.0"' pom.xml >/dev/null 2>/dev/null && grep -qF 'xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"' pom.xml >/dev/null 2>/dev/null && grep -qF 'xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 ' pom.xml >/dev/null 2>/dev/null || @@ -267,6 +285,7 @@ particular schema should always use the string identifier:' \ } # Change forum references from forum.image.net to forum.image.sc. +debug "Checking correctness of forum URL references" if grep -q 'https*://forum.imagej.net' pom.xml >/dev/null 2>/dev/null then echo "================================================================" @@ -280,6 +299,7 @@ then fi # Ensure that references to forum.image.sc use /tag/, not /tags/. +debug "Checking correctness of forum tag references" if grep -q forum.image.sc/tags/ pom.xml >/dev/null 2>/dev/null then echo "==================================================================" @@ -294,6 +314,7 @@ fi # Ensure license headers are up-to-date. test "$SKIP_LICENSE_UPDATE" -o -z "$licenseName" -o "$licenseName" = "N/A" || { + debug "Ensuring that license headers are up-to-date" mvn license:update-project-license license:update-file-header && git add LICENSE.txt || die 'Failed to update copyright blurbs. You can skip the license update using the --skip-license-update flag.' @@ -308,6 +329,7 @@ Alternately, try again with the --skip-license-update flag.' } # Prepare new release without pushing (requires the release plugin >= 2.1). +debug "Preparing new release" $DRY_RUN mvn $BATCH_MODE release:prepare -DpushChanges=false -Dresume=false $TAG \ $PROFILE $DEV_VERSION -DreleaseVersion="$VERSION" \ "-Darguments=-Dgpg.skip=true ${EXTRA_ARGS# }" || @@ -317,6 +339,7 @@ Use "mvn javadoc:javadoc | grep error" to check for javadoc syntax errors.' # Squash the maven-release-plugin's two commits into one. if test -z "$DRY_RUN" then + debug "Squashing release commits" test "[maven-release-plugin] prepare for next development iteration" = \ "$(git show -s --format=%s HEAD)" || die "maven-release-plugin's commits are unexpectedly missing!" @@ -328,6 +351,7 @@ then fi && # Extract the name of the new tag. +debug "Extracting new tag name" if test -z "$DRY_RUN" then tag=$(sed -n 's/^scm.tag=//p' < release.properties) @@ -336,6 +360,7 @@ else fi && # Rewrite the tag to include release.properties. +debug "Rewriting tag to include release.properties" test -n "$tag" && # HACK: SciJava projects use SSH (git@github.com:...) for developerConnection. # The release:perform command wants to use the developerConnection URL when @@ -355,9 +380,13 @@ $DRY_RUN git checkout @{-1} && # Push the current branch and the tag. if test -z "$SKIP_PUSH" then + debug "Pushing changes" $DRY_RUN git push "$REMOTE" HEAD $tag fi # Remove files generated by the release process. They can end up # committed to the mainline branch and hosing up later releases. +debug "Cleaning up" $DRY_RUN rm -f release.properties pom.xml.releaseBackup + +debug "Release complete!" From 89c9b82b98a94c6a03102f8c45cf54f8d1873677 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 21 Feb 2023 14:17:11 -0600 Subject: [PATCH 43/80] melting-pot: warn less about unpinned deps In particular: only warn if it's an unpinned dependency from the most recent success (i.e. first row of the cached successes), rather than warning about every single unpinned dependency present in every single cached past success, no matter how old. --- melting-pot.sh | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/melting-pot.sh b/melting-pot.sh index 17a368c..e50f93a 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -669,7 +669,8 @@ do mismatch="$mismatch\n* $g:$a:$v -> $bomV" else # G:A version is not pinned. - warn "$1: Unpinned dependency: $dep" + test "$row" -ne 1 || + warn "$1: Unpinned dependency: $dep" fi fi done From 7abb010e575979fc331f80c85f0f13e1d879e7f2 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 27 Feb 2023 12:40:13 -0600 Subject: [PATCH 44/80] melting-pot: also log dependency:tree It's nice to see where dependencies are coming from, even if the view is not complete due to pruning. --- melting-pot.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index e50f93a..8f13b2e 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -484,7 +484,7 @@ deps() { debug "mvn -B -DincludeScope=runtime dependency:list" local depList="$(mvn -B -DincludeScope=runtime dependency:list)" || die "Problem fetching dependencies!" 5 - echo "$depList" | grep '^\[INFO\] [^ ]' | + echo "$depList" | grep '^\[INFO\] \w' | sed 's/\[INFO\] //' | sed 's/ .*//' | sort cd - > /dev/null } @@ -722,7 +722,7 @@ successLog="$HOME/.cache/scijava/melting-pot/$1.success.log" mkdir -p "$(dirname "$successLog")" # Record dependency configuration of successful build. -deps=$(grep '^\[[^ ]*INFO[^ ]*\] ' "$buildLog" | +deps=$(grep '^\[[^ ]*INFO[^ ]*\] \w' "$buildLog" | sed -e 's/^[^ ]* *//' -e 's/ -- .*//' -e 's/ (\([^)]*\))/-\1/' | sort | tr '\n' ',') if [ -z "$(containsLine "$deps" "$successLog")" ] @@ -838,7 +838,7 @@ meltDown() { echo ' cd - >/dev/null' >> build.sh echo '}' >> build.sh echo >> build.sh - echo 'mvnPin dependency:list &&' >> build.sh + echo 'mvnPin dependency:list dependency:tree &&' >> build.sh echo >> build.sh echo 'if [ -f gav ]' >> build.sh echo 'then' >> build.sh From cc4cc7df27ff4535335445723dcbe6e0e353101c Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 27 Feb 2023 12:52:54 -0600 Subject: [PATCH 45/80] melting-pot: fall back to default branch if no tag --- melting-pot.sh | 43 +++++++++++++++++++++++++++++++------------ 1 file changed, 31 insertions(+), 12 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index 8f13b2e..1add86b 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -415,7 +415,7 @@ scmTag() { return } done - error "$1: inscrutable tag scheme" + error "$1: inscrutable tag scheme -- using default branch" else echo "$tag" fi @@ -441,24 +441,43 @@ resolveSource() { # Check whether the needed branch/tag exists. local scmBranch test "$2" && scmBranch="$2" || scmBranch="$(scmTag "$1")" - test "$scmBranch" || die "$1: cannot glean SCM tag" 14 - debug "git ls-remote \"file://$cachedRepoDir\" | grep -q \"\brefs/tags/$scmBranch$\"" - git ls-remote "file://$cachedRepoDir" | grep -q "\brefs/tags/$scmBranch$" || { - # Couldn't find the scmBranch as a tag in the cached repo. Either the - # tag is new, or it's not a tag ref at all (e.g. it's a branch). - # So let's update from the original remote repository. - info "$1: local tag not found for ref '$scmBranch'" + + if [ "$scmBranch" ] + then + # Successfully gleaned SCM branch/tag. + debug "git ls-remote \"file://$cachedRepoDir\" | grep -q \"\brefs/tags/$scmBranch$\"" + git ls-remote "file://$cachedRepoDir" | grep -q "\brefs/tags/$scmBranch$" || { + # Couldn't find the scmBranch as a tag in the cached repo. Either the + # tag is new, or it's not a tag ref at all (e.g. it's a branch). + # So let's update from the original remote repository. + info "$1: local tag not found for ref '$scmBranch'" + info "$1: updating cached repository: $cachedRepoDir" + cd "$cachedRepoDir" + debug "git fetch --tags" + if [ "$debug" ] + then + git fetch --tags + else + git fetch --tags > /dev/null + fi + cd - > /dev/null + } + else + # No SCM branch/tag; fall back to the default branch. info "$1: updating cached repository: $cachedRepoDir" cd "$cachedRepoDir" - debug "git fetch --tags" + debug "git fetch" if [ "$debug" ] then - git fetch --tags + git fetch else - git fetch --tags > /dev/null + git fetch > /dev/null fi + head=$(cat HEAD) + scmBranch=${head##*/} + info "$1: detected default branch as $scmBranch" cd - > /dev/null - } + fi # Shallow clone the source at the given version into melting-pot structure. local destDir="$g/$a" From 7db8ec05c8211dae01ddaa063f125997d1091c41 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 27 Feb 2023 13:52:09 -0600 Subject: [PATCH 46/80] melting-pot: always build SNAPSHOT components If their version is a SNAPSHOT, or they depend on SNAPSHOTs, or both. --- melting-pot.sh | 13 ++++++++++--- 1 file changed, 10 insertions(+), 3 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index 1add86b..9f8cd6f 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -678,7 +678,11 @@ do a=${apv%%:*} v=${apv##*:} bomV=$(grep -o " -D$g\.$a\.version=[^ ]*" "$dir/build.sh" | sed 's;.*=;;') - if [ "$bomV" != "$v" ] + if [ "$bomV" != "${bomV#*-SNAPSHOT*}" ] + then + warn "$1: Snapshot dependency pin detected: $g:$a:$bomV -- forcing a rebuild" + exit 0 + elif [ "$bomV" != "$v" ] then # G:A property is not set to this V. # Now check if the property is even declared. @@ -744,7 +748,7 @@ mkdir -p "$(dirname "$successLog")" deps=$(grep '^\[[^ ]*INFO[^ ]*\] \w' "$buildLog" | sed -e 's/^[^ ]* *//' -e 's/ -- .*//' -e 's/ (\([^)]*\))/-\1/' | sort | tr '\n' ',') -if [ -z "$(containsLine "$deps" "$successLog")" ] +if [ "$deps" = "${deps%*-SNAPSHOT*}" -a -z "$(containsLine "$deps" "$successLog")" ] then # NB: *Prepend*, rather than append, the new successful configuration. # We do this because it is more likely this new configuration will be @@ -891,7 +895,10 @@ meltDown() { local gav="$g:$a:$v" if [ "$(isIncluded "$gav")" ] then - if [ "$(./prior-success.sh "$g/$a")" ] + if [ "$v" != "${v%-SNAPSHOT}" ] + then + info "$g:$a: forcing inclusion due to SNAPSHOT version" + elif [ "$(./prior-success.sh "$g/$a")" ] then info "$g:$a: skipping version $v due to prior successful build" continue From 22a568f2c1a068d2adcb3ec8a2afde4cff4e0697 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 2 Mar 2023 15:21:28 -0600 Subject: [PATCH 47/80] class-version.sh: make detection more robust If a class lives in META-INF/versions/ we can ignore it because it's specially targeting a different version of Java than other classes. If a module-info.class is present, we can ignore it because it must target Java 9 or newer even if the other classes target an earlier version. --- class-version.sh | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/class-version.sh b/class-version.sh index a1ecfe7..f2dbece 100755 --- a/class-version.sh +++ b/class-version.sh @@ -49,7 +49,11 @@ class_version() { } first_class() { - jar tf "$1" | grep '\.class$' | head -n 1 + jar tf "$1" | + grep '\.class$' | + grep -v '^META-INF/' | + grep -v 'module-info\.class' | + head -n 1 } for file in "$@" From e3cd694f78102be981a02c6222f17f0928b004fc Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 11 May 2023 08:17:16 -0500 Subject: [PATCH 48/80] class-version.sh: support GAV arguments --- class-version.sh | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) diff --git a/class-version.sh b/class-version.sh index f2dbece..d90d34f 100755 --- a/class-version.sh +++ b/class-version.sh @@ -56,26 +56,37 @@ first_class() { head -n 1 } -for file in "$@" +for arg in "$@" do - case "$file" in + case "$arg" in + *:*:*) + ga=${arg%:*} + g=${ga%%:*} + a=${ga#*:} + v=${arg##*:} + f="$HOME/.m2/repository/$(echo "$g" | tr '.' '/')/$a/$v/$a-$v.jar" + test -f "$f" || mvn dependency:get -D"$arg" + arg="$f" + ;; + esac + case "$arg" in *.class) - version=$(cat "$file" | class_version) + version=$(cat "$arg" | class_version) ;; *.jar) - class=$(first_class "$file") + class=$(first_class "$arg") if [ -z "$class" ] then - echo "$file: No classes" + echo "$arg: No classes" continue fi - version=$(unzip -p "$file" "$class" | class_version) + version=$(unzip -p "$arg" "$class" | class_version) ;; *) - >&2 echo "Unsupported file: $file" + >&2 echo "Unsupported argument: $arg" continue esac # report the results - echo "$file: $version" + echo "$arg: $version" done From f5a378ed730276ff3222a4fd868b7ec827c8d319 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 25 Jul 2023 17:23:52 -0500 Subject: [PATCH 49/80] release-version.sh: avoid mvn ANSI colors Firstly, the mvn echo logic needs to pass -B. But even then, it seems that at least some versions of mvn (3.8.7 is an offender on my system) still prepend an [0m even in batch mode. So we also work around this bug by forcibly stripping ANSI color codes. Thanks to ChatGPT for the regex -- saved me a few minutes of time! --- release-version.sh | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/release-version.sh b/release-version.sh index 197ebe3..b0eb696 100755 --- a/release-version.sh +++ b/release-version.sh @@ -119,11 +119,16 @@ Options include: debug "Extracting project details" echoArg='${project.version}:${license.licenseName}:${project.parent.groupId}:${project.parent.artifactId}:${project.parent.version}' -projectDetails=$(mvn -N -Dexec.executable=echo -Dexec.args="$echoArg" exec:exec -q) -test $? -eq 0 || projectDetails=$(mvn -U -N -Dexec.executable=echo -Dexec.args="$echoArg" exec:exec -q) +projectDetails=$(mvn -B -N -Dexec.executable=echo -Dexec.args="$echoArg" exec:exec -q) +test $? -eq 0 || projectDetails=$(mvn -B -U -N -Dexec.executable=echo -Dexec.args="$echoArg" exec:exec -q) test $? -eq 0 || die "Could not extract version from pom.xml. Error follows:\n$projectDetails" -echo "$projectDetails" | grep -Fqv '[ERROR]' || +printf '%s' "$projectDetails\n" | grep -Fqv '[ERROR]' || die "Error extracting version from pom.xml. Error follows:\n$projectDetails" +# HACK: Even with -B, some versions of mvn taint the output with the [0m +# color reset sequence. So we forcibly remove such sequences, just to be safe. +projectDetails=$(printf '%s' "$projectDetails" | sed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[mGK]//g") +# And also remove extraneous newlines, particularly any trailing ones. +projectDetails=$(printf '%s' "$projectDetails" | tr -d '\n') currentVersion=${projectDetails%%:*} projectDetails=${projectDetails#*:} licenseName=${projectDetails%%:*} @@ -185,7 +190,7 @@ test "$SKIP_VERSION_CHECK" -o "$parentGAV" != "${parentGAV#$}" || { latestParentVersion=$(sh -$- "$MAVEN_HELPER" latest-version "$parentGAV") currentParentVersion=${parentGAV##*:} test "$currentParentVersion" = "$latestParentVersion" || - die "Newer version of parent '${parentGAV%:*}' is available: $latestParentVersion. + die "Newer version of parent '$parentGAV' is available: $latestParentVersion. I recommend you update it before releasing. Or if you know better, try again with --skip-version-check flag." } From 7b53900a9c252b76e301364d66a1d1e60994f17b Mon Sep 17 00:00:00 2001 From: Mark Hiner Date: Thu, 14 Sep 2023 11:28:52 -0500 Subject: [PATCH 50/80] ci-build.sh: allow external build args This allows us to pass e.g. -X for debugging on individual jobs --- ci-build.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci-build.sh b/ci-build.sh index 78d308f..13b4f0f 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -183,7 +183,7 @@ EOL # --== Maven build arguments ==-- - BUILD_ARGS="-B -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2" + BUILD_ARGS="$BUILD_ARGS -B -Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2" # --== GPG SETUP ==-- From 5571e4ebdda60364b5ee6578eaffd69e423a4fd5 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 17 Oct 2023 13:27:02 +0200 Subject: [PATCH 51/80] check-branch.sh: support any project with Makefile This assumes the Makefile has a 'test' target. --- check-branch.sh | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/check-branch.sh b/check-branch.sh index c34cd48..fdbfc4c 100755 --- a/check-branch.sh +++ b/check-branch.sh @@ -24,7 +24,15 @@ do prefix="$(printf %04d $count)" filename="tmp/$prefix-$commit" start=$(date +%s) - mvn clean verify > "$filename" 2>&1 && result=SUCCESS || result=FAILURE + if [ -f Makefile ] + then + make test > "$filename" 2>&1 && result=SUCCESS || result=FAILURE + elif [ -f pom.xml ] + then + mvn clean verify > "$filename" 2>&1 && result=SUCCESS || result=FAILURE + else + result=SKIPPED + fi end=$(date +%s) time=$(expr "$end" - "$start") echo "$prefix $commit $result $time" From 27b43f6ff08b33b65e4f93153a54a31ce087dda0 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Wed, 18 Oct 2023 19:20:00 +0200 Subject: [PATCH 52/80] sj-version.sh: include pom-scijava-base versions --- sj-version.sh | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/sj-version.sh b/sj-version.sh index 834e52d..a0486a3 100755 --- a/sj-version.sh +++ b/sj-version.sh @@ -15,16 +15,29 @@ props() { if [ -e "$1" ] then # extract version properties from the given file path - versions=$(cat "$1") + pomContent=$(cat "$1") else - url="$repo/org/scijava/pom-scijava/$1/pom-scijava-$1.pom" - versions=$(curl -s "$url") # assume argument is a version number of pom-scijava + pomURL="$repo/org/scijava/pom-scijava/$1/pom-scijava-$1.pom" + pomContent=$(curl -s "$pomURL") fi - echo "$versions" | \ - grep '\.version>' | \ - sed -E -e 's/^ (.*)/\1 [DEV]/' | \ - sed -E -e 's/^ *<(.*)\.version>(.*)<\/.*\.version>/\1 = \2/' | \ + + # grep the pom-scijava-base parent version of out of the POM, + # then rip out the version properties from that one as well! + psbVersion=$(echo "$pomContent" | + grep -A1 'pom-scijava-base' | + grep '' | sed 's;.*>\([^<]*\)<.*;\1;') + psbContent= + if [ "$psbVersion" ] + then + psbURL="$repo/org/scijava/pom-scijava-base/$psbVersion/pom-scijava-base-$psbVersion.pom" + psbContent=$(curl -s "$psbURL") + fi + + { echo "$pomContent"; echo "$psbContent"; } | + grep '\.version>' | + sed -E -e 's/^ (.*)/\1 [DEV]/' | + sed -E -e 's/^ *<(.*)\.version>(.*)<\/.*\.version>/\1 = \2/' | sort } From 3cc654e8d036c1718865e7ac1c09fa5d5e1ca2b3 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 7 Nov 2023 11:15:33 -0600 Subject: [PATCH 53/80] ci-build.sh: save the resolved platform from uname Just in case we want to do anything else with it, reason about whether the platform is e.g. Linux, print it out, etc. --- ci-build.sh | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/ci-build.sh b/ci-build.sh index 13b4f0f..196fbee 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -9,10 +9,7 @@ dir="$(dirname "$0")" -MACOS= -case "$(uname -s)" in - Darwin) MACOS=1;; -esac +platform=$(uname -s) success=0 checkSuccess() { @@ -188,7 +185,7 @@ EOL # --== GPG SETUP ==-- # Install GPG on macOS - if [ "$MACOS" ]; then + if [ "$platform" = Darwin ]; then HOMEBREW_NO_AUTO_UPDATE=1 brew install gnupg2 fi From 7bd22bf4749d6f92eeefafab0ead47ea4f617785 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 7 Nov 2023 11:16:17 -0600 Subject: [PATCH 54/80] ci-build.sh: skip deploy if NO_DEPLOY flag is set So that downstream projects (e.g. scijava-common) can set this flag judiciously to control the circumstances of deployment more flexibly. --- ci-build.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/ci-build.sh b/ci-build.sh index 196fbee..67e9287 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -133,7 +133,9 @@ EOL else scmURL=${scmURL%.git} scmURL=${scmURL%/} - if [ ! "$SIGNING_ASC" -o ! "$GPG_KEY_NAME" -o ! "$GPG_PASSPHRASE" -o ! "$MAVEN_PASS" -o ! "$OSSRH_PASS" ]; then + if [ "$NO_DEPLOY" ]; then + echo "No deploy -- the NO_DEPLOY flag is set" + elif [ ! "$SIGNING_ASC" -o ! "$GPG_KEY_NAME" -o ! "$GPG_PASSPHRASE" -o ! "$MAVEN_PASS" -o ! "$OSSRH_PASS" ]; then echo "No deploy -- secure environment variables not available" elif [ "$BUILD_REPOSITORY" -a "$BUILD_REPOSITORY" != "$scmURL" ]; then echo "No deploy -- repository fork: $BUILD_REPOSITORY != $scmURL" From b7eb4eef9cd10911a83e460d956fb01b1e92aec6 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 4 Mar 2024 11:36:12 -0600 Subject: [PATCH 55/80] class-version.sh: update JDK names The platform was called "J2SE" from 1.2 thru 6.0, and referred to as "Java SE" starting with Java 7. But all of that is stupid and confusing marketing. Let's just call it "Java" everywhere. The "1.x" vs "x" issue still remains, but it's less terrible than before. --- class-version.sh | 17 +++++++---------- 1 file changed, 7 insertions(+), 10 deletions(-) diff --git a/class-version.sh b/class-version.sh index d90d34f..a5b4954 100755 --- a/class-version.sh +++ b/class-version.sh @@ -17,27 +17,24 @@ class_version() { # derive Java version case $major in 45) - version="JDK 1.1" + version="Java 1.0/1.1" ;; 46) - version="JDK 1.2" + version="Java 1.2" ;; 47) - version="JDK 1.3" + version="Java 1.3" ;; 48) - version="JDK 1.4" + version="Java 1.4" ;; 49) - version="J2SE 5.0" - ;; - 50) - version="J2SE 6.0" + version="Java 5" ;; *) - if [ "$major" -gt 50 ] + if [ "$major" -gt 49 ] then - version="J2SE $(expr $major - 44)" + version="Java $(expr $major - 44)" else version="Unknown" fi From 49845be5394918525f87b19c0e3c1b266b2c53d8 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 8 Jul 2024 14:20:30 -0500 Subject: [PATCH 56/80] Add support for custom OSSRH_USER value This is now important because apparently it is no longer allowed to deploy to OSS Sonatype (at least not s01.oss.sonatype.org) using the plain username and password configuration -- only with the user token. And the token has different alphabet gobbledegook for the username. --- ci-build.sh | 5 ++++- github-actionify.sh | 1 + 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/ci-build.sh b/ci-build.sh index 67e9287..8dc4768 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -80,6 +80,9 @@ if [ -f pom.xml ]; then elif [ -f "$customSettings" ]; then cp "$customSettings" "$settingsFile" else + if [ -z "$OSSRH_USER" ]; then + OSSRH_USER=scijava-ci + fi cat >"$settingsFile" < @@ -95,7 +98,7 @@ if [ -f pom.xml ]; then sonatype-nexus-releases - scijava-ci + $OSSRH_USER $(escapeXML "$OSSRH_PASS") diff --git a/github-actionify.sh b/github-actionify.sh index a52079d..6707c5f 100755 --- a/github-actionify.sh +++ b/github-actionify.sh @@ -129,6 +129,7 @@ process() { GPG_PASSPHRASE: \${{ secrets.GPG_PASSPHRASE }} MAVEN_USER: \${{ secrets.MAVEN_USER }} MAVEN_PASS: \${{ secrets.MAVEN_PASS }} + OSSRH_USER: \${{ secrets.OSSRH_USER }} OSSRH_PASS: \${{ secrets.OSSRH_PASS }} SIGNING_ASC: \${{ secrets.SIGNING_ASC }}" From 756eb8e79cf8f5f5aa1d429db39419e209f3ac8b Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 1 Aug 2024 05:57:13 -0500 Subject: [PATCH 57/80] ci-build.sh: fix 'proposed change' flow direction --- ci-build.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci-build.sh b/ci-build.sh index 8dc4768..e1093eb 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -172,7 +172,7 @@ EOL esac if [ "$BUILD_BASE_REF" -o "$BUILD_HEAD_REF" ] then - echo "No deploy -- proposed change: $BUILD_BASE_REF -> $BUILD_HEAD_REF" + echo "No deploy -- proposed change: $BUILD_HEAD_REF -> $BUILD_BASE_REF" else deployOK=1 fi From ef9a438028f48f2acc37a0b9a77900d10435d9a8 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Fri, 4 Oct 2024 17:42:03 -0500 Subject: [PATCH 58/80] melting-pot.sh: do not use symlinking on Windows The `ln -s` feature does not work in MINGW; instead it copies. And that cannot work as written because it's copying a folder into a subfolder of itself, resulting in an infinite loop. May be a bug in MINGW's ln command, but let's avoid the issue. --- melting-pot.sh | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index 9f8cd6f..055eb7e 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -773,10 +773,18 @@ meltDown() { # Use local directory for the specified project. test -d "$1" || die "No such directory: $1" 11 test -f "$1/pom.xml" || die "Not a Maven project: $1" 12 - info "Local Maven project: $1" - mkdir -p "LOCAL" - local projectDir="LOCAL/PROJECT" - ln -s "$1" "$projectDir" + case "$(uname)" in + MINGW*) + warn "Skipping inclusion of local project due to lack of symlink support." + local projectDir="$1" + ;; + *) + info "Local Maven project: $1" + mkdir -p "LOCAL" + local projectDir="LOCAL/PROJECT" + ln -s "$1" "$projectDir" + ;; + esac else # Treat specified project as a GAV. info "Fetching project source" From 66a91736e26d7851fd226da01cb61f2c369bdcab Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Fri, 4 Oct 2024 17:43:25 -0500 Subject: [PATCH 59/80] melting-pot.sh: specify version pins in a file This change avoids the dreaded "Argument list too long" that happens on Windows with too many -Dx.version=y arguments. One concern I had was whether specifying version overrides as settings.xml properties inside an active-by-default profile would still override the version properties of individual project POMs. But it works. I do not know whether it still works for project POMs with their own profiles that set version properties contingently; in that case, there would be multiple values competing, so it would likely come down to profile evaluation order. Fortunately, that is not common for SciJava-based projects. We also avoid usage of the `\\\\\n` sequence, since it appears to be handled incorrectly by MinGW's /bin/sh. --- melting-pot.sh | 41 ++++++++++++++++++++++++++++++----------- 1 file changed, 30 insertions(+), 11 deletions(-) diff --git a/melting-pot.sh b/melting-pot.sh index 055eb7e..e78e4ad 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -666,7 +666,7 @@ do mismatch= for dep in $(echo "$deps" | tr ',' '\n') do - # g:a:p:v:s -> -Dg.a.version=v + # g:a:p:v:s -> v s=${dep##*:} case "$s" in test) continue ;; # skip test dependencies @@ -677,7 +677,7 @@ do apv=${gapv#*:} a=${apv%%:*} v=${apv##*:} - bomV=$(grep -o " -D$g\.$a\.version=[^ ]*" "$dir/build.sh" | sed 's;.*=;;') + bomV=$(grep -o " <$g\.$a\.version>[^>]*" "$dir/version-pins.xml" | sed 's;[^>]*>\([^>]*\)<.*;\1;') if [ "$bomV" != "${bomV#*-SNAPSHOT*}" ] then warn "$1: Snapshot dependency pin detected: $g:$a:$bomV -- forcing a rebuild" @@ -801,13 +801,9 @@ meltDown() { # to decide whether to include each component. generateHelperScripts - # NB: We do *not* include -B here, because we want build.sh to preserve - # colored output if the version of Maven is new enough. We will take care - # elsewhere when parsing it to be flexible about whether colors are present. - local args="-Denforcer.skip" - # Process the dependencies. info "Processing project dependencies" + local versionProps="" local dep for dep in $deps do @@ -817,9 +813,9 @@ meltDown() { local c="$(classifier "$dep")" test -z "$c" || continue # skip secondary artifacts local gav="$g:$a:$v" - test -z "$(isChanged "$gav")" && - args="$args \\\\\n -D$g.$a.version=$v -D$a.version=$v" + versionProps="$versionProps + <$g.$a.version>$v <$a.version>$v" done # Override versions of changed GAVs. @@ -830,16 +826,39 @@ meltDown() { do local a="$(artifactId "$gav")" local v="$(version "$gav")" - args="$args \\\\\n -D$a.version=$v" + versionProps="$versionProps + <$a.version>$v" done unset TLS + # Generate version-pins.xml. + info "Generating version-pins.xml configuration" + echo ' version-pins.xml + echo ' xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 https://maven.apache.org/xsd/settings-1.1.0.xsd">' >> version-pins.xml + echo ' ' >> version-pins.xml + echo ' ' >> version-pins.xml + echo ' version-pins' >> version-pins.xml + echo ' ' >> version-pins.xml + echo ' true' >> version-pins.xml + echo ' ' >> version-pins.xml + echo ' ' >> version-pins.xml + echo "$versionProps" >> version-pins.xml + echo ' ' >> version-pins.xml + echo ' ' >> version-pins.xml + echo ' ' >> version-pins.xml + echo '' >> version-pins.xml + # Generate build script. info "Generating build.sh script" echo '#!/bin/sh' > build.sh echo >> build.sh + echo 'dir=$(cd "$(dirname "$0")" && pwd)' >> build.sh + echo >> build.sh echo 'mvnPin() {' >> build.sh - echo " mvn $args \\\\\n \$@" >> build.sh + # NB: We do *not* include -B here, because we want build.sh to preserve + # colored output if the version of Maven is new enough. We will take care + # elsewhere when parsing it to be flexible about whether colors are present. + echo ' mvn -s "$dir/version-pins.xml" -Denforcer.skip $@' >> build.sh echo '}' >> build.sh echo >> build.sh echo 'unpackArtifact() {' >> build.sh From 75fc655f5bc12ea7e6afc36b799da53990c85a0f Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Sun, 6 Oct 2024 09:29:36 -0500 Subject: [PATCH 60/80] melting-pot.sh: use valid short property names There are some artifactIds that begin with a digit rather than a letter, which results in invalid version property names. Prepending the property name with underscore avoids the issue. --- melting-pot.sh | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/melting-pot.sh b/melting-pot.sh index e78e4ad..29de050 100755 --- a/melting-pot.sh +++ b/melting-pot.sh @@ -809,13 +809,15 @@ meltDown() { do local g="$(groupId "$dep")" local a="$(artifactId "$dep")" + local aa + echo "$a" | grep -q '^[0-9]' && aa="_$a" || aa="$a" local v="$(version "$dep")" local c="$(classifier "$dep")" test -z "$c" || continue # skip secondary artifacts local gav="$g:$a:$v" test -z "$(isChanged "$gav")" && versionProps="$versionProps - <$g.$a.version>$v <$a.version>$v" + <$g.$a.version>$v <$aa.version>$v" done # Override versions of changed GAVs. From 0e97dd09375b32892b61a81ca3b9bff56fb801f7 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 14 Jan 2025 18:49:14 -0600 Subject: [PATCH 61/80] class-version.sh: fix on-demand artifact download --- class-version.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/class-version.sh b/class-version.sh index a5b4954..96ddb53 100755 --- a/class-version.sh +++ b/class-version.sh @@ -62,7 +62,7 @@ do a=${ga#*:} v=${arg##*:} f="$HOME/.m2/repository/$(echo "$g" | tr '.' '/')/$a/$v/$a-$v.jar" - test -f "$f" || mvn dependency:get -D"$arg" + test -f "$f" || mvn dependency:get -Dartifact="$arg" arg="$f" ;; esac From 950438f8bf082247e6c8798e35418c5b9eee53c6 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 15 May 2025 17:56:20 -0500 Subject: [PATCH 62/80] class-version.sh: recognize G:A:V:C coords --- class-version.sh | 17 +++++++++++++++-- 1 file changed, 15 insertions(+), 2 deletions(-) diff --git a/class-version.sh b/class-version.sh index 96ddb53..ed77a66 100755 --- a/class-version.sh +++ b/class-version.sh @@ -55,8 +55,20 @@ first_class() { for arg in "$@" do + # Resolve Maven dependency coordinates into local files. case "$arg" in - *:*:*) + *:*:*:*) # g:a:v:c + gav=${arg%:*} + g=${gav%%:*} + av=${gav#*:} + a=${av%:*} + v=${av#*:} + c=${arg##*:} + f="$HOME/.m2/repository/$(echo "$g" | tr '.' '/')/$a/$v/$a-$v-$c.jar" + test -f "$f" || mvn dependency:get -Dartifact="$g:$a:$v:jar:$c" + arg="$f" + ;; + *:*:*) # g:a:v ga=${arg%:*} g=${ga%%:*} a=${ga#*:} @@ -66,6 +78,7 @@ do arg="$f" ;; esac + # Handle the various local file cases. case "$arg" in *.class) version=$(cat "$arg" | class_version) @@ -84,6 +97,6 @@ do continue esac - # report the results + # Report the results. echo "$arg: $version" done From e6ecc162064b4ba9f2155032efbe993f0efc4635 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 15 Jul 2025 15:53:36 -0500 Subject: [PATCH 63/80] Update repository ID for OSSRH -> Central Portal See: * https://central.sonatype.org/pages/ossrh-eol/ * https://central.sonatype.org/publish/publish-portal-maven/ For the moment, we leave the environment variable names alone, even though the prefix "OSSRH" is a misnomer now. Reason: it is possible that the existing OSSRH user tokens were migrated directly. If they don't work, I'll change this script again to use new environment variables, and issue a better error if the new variables are missing. --- ci-build.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci-build.sh b/ci-build.sh index e1093eb..98a57b6 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -97,7 +97,7 @@ if [ -f pom.xml ]; then $(escapeXML "$MAVEN_PASS") - sonatype-nexus-releases + central $OSSRH_USER $(escapeXML "$OSSRH_PASS") From eaf2bd8194747020132e2b656b7865fcc8c86de6 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 15 Jul 2025 16:15:41 -0500 Subject: [PATCH 64/80] Rename Central Portal deployment env vars OK, now I'm renaming the credentials, because my first release failed: Error: [ERROR] Unable to upload bundle for deployment: Deployment [INFO] java.lang.RuntimeException: Invalid request. Status: 401 Response body: [INFO] at org.sonatype.central.publisher.client.httpclient.UploadPublisherEndpoint.call (UploadPublisherEndpoint.java:33) I am not actually certain the user token changed (it may just be that the legacy `OSSRH_USER=scijava-ci` value was the culprit), but that's OK: going through the few orgs that actually deploy to Maven Central to update to a new token is not really such a big hassle. --- ci-build.sh | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/ci-build.sh b/ci-build.sh index 98a57b6..191096f 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -75,14 +75,14 @@ if [ -f pom.xml ]; then mkdir -p "$HOME/.m2" settingsFile="$HOME/.m2/settings.xml" customSettings=.ci/settings.xml - if [ -z "$MAVEN_PASS" -a -z "$OSSRH_PASS" ]; then + if [ "$OSSRH_USER" -o "$OSSRH_PASS" ]; then + echo "[WARNING] Obsolete OSSRH vars detected. Secrets may need updating to deploy to Maven Central." + fi + if [ -z "$MAVEN_PASS" -a -z "$CENTRAL_PASS" ]; then echo "[WARNING] Skipping settings.xml generation (no deployment credentials)." elif [ -f "$customSettings" ]; then cp "$customSettings" "$settingsFile" else - if [ -z "$OSSRH_USER" ]; then - OSSRH_USER=scijava-ci - fi cat >"$settingsFile" < @@ -98,8 +98,8 @@ if [ -f pom.xml ]; then central - $OSSRH_USER - $(escapeXML "$OSSRH_PASS") + $CENTRAL_USER + $(escapeXML "$CENTRAL_PASS") From d2e429255a25af02fdf290ebfb369ab75d338718 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 15 Jul 2025 16:41:08 -0500 Subject: [PATCH 65/80] Rename Central Portal deployment env vars harder I missed some references earlier. --- ci-build.sh | 2 +- github-actionify.sh | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/ci-build.sh b/ci-build.sh index 191096f..b9397a0 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -138,7 +138,7 @@ EOL scmURL=${scmURL%/} if [ "$NO_DEPLOY" ]; then echo "No deploy -- the NO_DEPLOY flag is set" - elif [ ! "$SIGNING_ASC" -o ! "$GPG_KEY_NAME" -o ! "$GPG_PASSPHRASE" -o ! "$MAVEN_PASS" -o ! "$OSSRH_PASS" ]; then + elif [ ! "$SIGNING_ASC" -o ! "$GPG_KEY_NAME" -o ! "$GPG_PASSPHRASE" -o ! "$MAVEN_PASS" -o ! "$CENTRAL_PASS" ]; then echo "No deploy -- secure environment variables not available" elif [ "$BUILD_REPOSITORY" -a "$BUILD_REPOSITORY" != "$scmURL" ]; then echo "No deploy -- repository fork: $BUILD_REPOSITORY != $scmURL" diff --git a/github-actionify.sh b/github-actionify.sh index 6707c5f..a88d24f 100755 --- a/github-actionify.sh +++ b/github-actionify.sh @@ -129,8 +129,8 @@ process() { GPG_PASSPHRASE: \${{ secrets.GPG_PASSPHRASE }} MAVEN_USER: \${{ secrets.MAVEN_USER }} MAVEN_PASS: \${{ secrets.MAVEN_PASS }} - OSSRH_USER: \${{ secrets.OSSRH_USER }} - OSSRH_PASS: \${{ secrets.OSSRH_PASS }} + CENTRAL_USER: \${{ secrets.CENTRAL_USER }} + CENTRAL_PASS: \${{ secrets.CENTRAL_PASS }} SIGNING_ASC: \${{ secrets.SIGNING_ASC }}" # -- Do things -- From 0461f7bf971c43d0ed197ab2a833fd81ed3c6f0e Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 17 Jul 2025 15:06:11 -0500 Subject: [PATCH 66/80] ci-build.sh: write out settings.xml in pieces If some env vars are missing, we can still partially write it. This is especially important if CENTRAL_* vars are missing, but MAVEN_* vars are present, which as of this writing will be the case for the vast majority of SciJava-based community projects deploying to maven.scijava.org, since the transition from OSSRH to Central Portal is still very recent and many orgs have not yet been updated to match. --- ci-build.sh | 33 ++++++++++++++++++++++++++++++--- 1 file changed, 30 insertions(+), 3 deletions(-) diff --git a/ci-build.sh b/ci-build.sh index b9397a0..4dca6ae 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -78,14 +78,19 @@ if [ -f pom.xml ]; then if [ "$OSSRH_USER" -o "$OSSRH_PASS" ]; then echo "[WARNING] Obsolete OSSRH vars detected. Secrets may need updating to deploy to Maven Central." fi - if [ -z "$MAVEN_PASS" -a -z "$CENTRAL_PASS" ]; then - echo "[WARNING] Skipping settings.xml generation (no deployment credentials)." - elif [ -f "$customSettings" ]; then + if [ -f "$customSettings" ]; then cp "$customSettings" "$settingsFile" + elif [ -z "$BUILD_REPOSITORY" ]; then + echo "Skipping settings.xml generation (no BUILD_REPOSITORY; assuming we are running locally)" else + # settings.xml header cat >"$settingsFile" < +EOL + # settings.xml scijava servers + if [ "$MAVEN_USER" -a "$MAVEN_PASS" ]; then + cat >>"$settingsFile" < scijava.releases $MAVEN_USER @@ -96,12 +101,28 @@ if [ -f pom.xml ]; then $MAVEN_USER $(escapeXML "$MAVEN_PASS") +EOL + else + echo "[WARNING] Skipping settings.xml scijava servers (no MAVEN deployment credentials)." + fi + # settings.xml central server + if [ "$CENTRAL_USER" -a "$CENTRAL_PASS" ]; then + cat >>"$settingsFile" < central $CENTRAL_USER $(escapeXML "$CENTRAL_PASS") +EOL + else + echo "[WARNING] Skipping settings.xml central server (no CENTRAL deployment credentials)." + fi + cat >>"$settingsFile" < +EOL + # settings.xml GPG profile + if [ "$GPG_KEY_NAME" -a "$GPG_PASSPHRASE" ]; then + cat >>"$settingsFile" < gpg @@ -116,6 +137,12 @@ if [ -f pom.xml ]; then +EOL + else + echo "[WARNING] Skipping settings.xml gpg profile (no GPG credentials)." + fi + # settings.xml footer + cat >>"$settingsFile" < EOL fi From 1e1b1ebbc9ac47579944430339a03dab8deb01c9 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 17 Jul 2025 15:24:05 -0500 Subject: [PATCH 67/80] ci-build.sh: skip GPG setup if no GPG credentials --- ci-build.sh | 82 ++++++++++++++++++++++++++++------------------------- 1 file changed, 43 insertions(+), 39 deletions(-) diff --git a/ci-build.sh b/ci-build.sh index 4dca6ae..194b0de 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -216,48 +216,52 @@ EOL # --== GPG SETUP ==-- - # Install GPG on macOS - if [ "$platform" = Darwin ]; then - HOMEBREW_NO_AUTO_UPDATE=1 brew install gnupg2 - fi - - # Avoid "signing failed: Inappropriate ioctl for device" error. - export GPG_TTY=$(tty) - - # Import the GPG signing key. - keyFile=.ci/signingkey.asc - if [ "$deployOK" ]; then - echo "== Importing GPG keypair ==" - mkdir -p .ci - echo "$SIGNING_ASC" > "$keyFile" - ls -la "$keyFile" - gpg --version - gpg --batch --fast-import "$keyFile" - checkSuccess $? - fi + if [ "$GPG_KEY_NAME" -a "$GPG_PASSPHRASE" ]; then + # Install GPG on macOS + if [ "$platform" = Darwin ]; then + HOMEBREW_NO_AUTO_UPDATE=1 brew install gnupg2 + fi - # HACK: Use maven-gpg-plugin 3.0.1+. Avoids "signing failed: No such file or directory" error. - maven_gpg_plugin_version=$(mavenEvaluate '${maven-gpg-plugin.version}') - case "$maven_gpg_plugin_version" in - 0.*|1.*|2.*|3.0.0) - echo "--> Forcing maven-gpg-plugin version from $maven_gpg_plugin_version to 3.0.1" - BUILD_ARGS="$BUILD_ARGS -Dmaven-gpg-plugin.version=3.0.1 -Darguments=-Dmaven-gpg-plugin.version=3.0.1" - ;; - *) - echo "--> maven-gpg-plugin version OK: $maven_gpg_plugin_version" - ;; - esac - - # HACK: Install pinentry helper program if missing. Avoids "signing failed: No pinentry" error. - if ! which pinentry >/dev/null 2>&1; then - echo '--> Installing missing pinentry helper for GPG' - sudo apt-get install -y pinentry-tty - # HACK: Restart the gpg agent, to notice the newly installed pinentry. - if { pgrep gpg-agent >/dev/null && which gpgconf >/dev/null 2>&1; } then - echo '--> Restarting gpg-agent' - gpgconf --reload gpg-agent + # Avoid "signing failed: Inappropriate ioctl for device" error. + export GPG_TTY=$(tty) + + # Import the GPG signing key. + keyFile=.ci/signingkey.asc + if [ "$deployOK" ]; then + echo "== Importing GPG keypair ==" + mkdir -p .ci + echo "$SIGNING_ASC" > "$keyFile" + ls -la "$keyFile" + gpg --version + gpg --batch --fast-import "$keyFile" checkSuccess $? fi + + # HACK: Use maven-gpg-plugin 3.0.1+. Avoids "signing failed: No such file or directory" error. + maven_gpg_plugin_version=$(mavenEvaluate '${maven-gpg-plugin.version}') + case "$maven_gpg_plugin_version" in + 0.*|1.*|2.*|3.0.0) + echo "--> Forcing maven-gpg-plugin version from $maven_gpg_plugin_version to 3.0.1" + BUILD_ARGS="$BUILD_ARGS -Dmaven-gpg-plugin.version=3.0.1 -Darguments=-Dmaven-gpg-plugin.version=3.0.1" + ;; + *) + echo "--> maven-gpg-plugin version OK: $maven_gpg_plugin_version" + ;; + esac + + # HACK: Install pinentry helper program if missing. Avoids "signing failed: No pinentry" error. + if ! which pinentry >/dev/null 2>&1; then + echo '--> Installing missing pinentry helper for GPG' + sudo apt-get install -y pinentry-tty + # HACK: Restart the gpg agent, to notice the newly installed pinentry. + if { pgrep gpg-agent >/dev/null && which gpgconf >/dev/null 2>&1; } then + echo '--> Restarting gpg-agent' + gpgconf --reload gpg-agent + checkSuccess $? + fi + fi + else + echo "[WARNING] Skipping gpg setup (no GPG credentials)." fi # --== BUILD EXECUTION ==-- From c06b42fa765dccef4d7c96cfacd5855c9d62e4ca Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Thu, 17 Jul 2025 15:55:45 -0500 Subject: [PATCH 68/80] ci-build.sh: validate deploy credentials better We don't need CENTRAL env vars when deploying to maven.scijava.org. We don't need MAVEN env vars when deploying to Central. --- ci-build.sh | 52 ++++++++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 44 insertions(+), 8 deletions(-) diff --git a/ci-build.sh b/ci-build.sh index 194b0de..a9ed6d2 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -165,10 +165,10 @@ EOL scmURL=${scmURL%/} if [ "$NO_DEPLOY" ]; then echo "No deploy -- the NO_DEPLOY flag is set" - elif [ ! "$SIGNING_ASC" -o ! "$GPG_KEY_NAME" -o ! "$GPG_PASSPHRASE" -o ! "$MAVEN_PASS" -o ! "$CENTRAL_PASS" ]; then - echo "No deploy -- secure environment variables not available" elif [ "$BUILD_REPOSITORY" -a "$BUILD_REPOSITORY" != "$scmURL" ]; then echo "No deploy -- repository fork: $BUILD_REPOSITORY != $scmURL" + elif [ "$BUILD_BASE_REF" -o "$BUILD_HEAD_REF" ]; then + echo "No deploy -- proposed change: $BUILD_HEAD_REF -> $BUILD_BASE_REF" else # Are we building a snapshot version, or a release version? version=$(mavenEvaluate '${project.version}') @@ -187,6 +187,13 @@ EOL echo "Remove the file from version control and try again." exit 1 fi + + # Check for SciJava Maven repository credentials. + if [ "$MAVEN_USER" -a "$MAVEN_PASS" ]; then + deployOK=1 + else + echo "No deploy -- MAVEN environment variables not available" + fi ;; *) # Release version -- ensure release.properties is present. @@ -195,14 +202,43 @@ EOL echo "You must use release-version.sh to release -- see https://imagej.net/develop/releasing" exit 1 fi + + # To which repository are we releasing? + releaseProfiles=$(mavenEvaluate '${releaseProfiles}') + result=$? + checkSuccess $result + if [ $result -ne 0 ]; then + echo "No deploy -- could not extract releaseProfiles string" + echo "Output of failed attempt follows:" + echo "$releaseProfiles" + fi + case "$releaseProfiles" in + *deploy-to-scijava*) + # Check for SciJava Maven repository credentials. + if [ "$MAVEN_USER" -a "$MAVEN_PASS" ]; then + deployOK=1 + else + echo "[ERROR] Cannot deploy: MAVEN environment variables not available" + exit 1 + fi + ;; + *sonatype-oss-release*) + # Check for Central Portal deployment credentials. + # Deploy to Central requires GPG-signed artifacts. + if [ "$CENTRAL_USER" -a "$CENTRAL_PASS" -a "$SIGNING_ASC" -a "$GPG_KEY_NAME" -a "$GPG_PASSPHRASE" ]; then + deployOK=1 + else + echo "[ERROR] Cannot deploy: CENTRAL environment variables not available" + exit 1 + fi + ;; + *) + echo "Unknown deploy target -- attempting to deploy anyway" + deployOK=1 + ;; + esac ;; esac - if [ "$BUILD_BASE_REF" -o "$BUILD_HEAD_REF" ] - then - echo "No deploy -- proposed change: $BUILD_HEAD_REF -> $BUILD_BASE_REF" - else - deployOK=1 - fi fi fi fi From 7b58554c9d59adfb1b7ffd6fc4a7a5acd16b6c37 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 5 Aug 2025 16:02:20 -0500 Subject: [PATCH 69/80] Remove obsolete Travis CI build script We aren't maintaining it anymore. If you want to keep using Travis CI, you can pin to an older commit in this repository, or else just fork the script from just prior to deletion and add it to your repository. --- travis-build.sh | 243 ------------------------------------------------ 1 file changed, 243 deletions(-) delete mode 100644 travis-build.sh diff --git a/travis-build.sh b/travis-build.sh deleted file mode 100644 index 1a759e1..0000000 --- a/travis-build.sh +++ /dev/null @@ -1,243 +0,0 @@ -#!/bin/bash - -# -# travis-build.sh - A script to build and/or release SciJava-based projects. -# - -dir="$(dirname "$0")" - -success=0 -checkSuccess() { - # Log non-zero exit code. - test $1 -eq 0 || echo "==> FAILED: EXIT CODE $1" 1>&2 - - # Record the first non-zero exit code. - test $success -eq 0 && success=$1 -} - -# Build Maven projects. -if [ -f pom.xml ] -then - echo travis_fold:start:scijava-maven - echo "= Maven build =" - echo - echo "== Configuring Maven ==" - - # NB: Suppress "Downloading/Downloaded" messages. - # See: https://stackoverflow.com/a/35653426/1207769 - export MAVEN_OPTS="$MAVEN_OPTS -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=warn" - - # Populate the settings.xml configuration. - mkdir -p "$HOME/.m2" - settingsFile="$HOME/.m2/settings.xml" - customSettings=.travis/settings.xml - if [ -f "$customSettings" ] - then - cp "$customSettings" "$settingsFile" - else - cat >"$settingsFile" < - - - scijava.releases - travis - \${env.MAVEN_PASS} - - - scijava.snapshots - travis - \${env.MAVEN_PASS} - - - sonatype-nexus-releases - scijava-ci - \${env.OSSRH_PASS} - - -EOL - # NB: Use maven.scijava.org instead of Central if defined in repositories. - # This hopefully avoids intermittent "ReasonPhrase:Forbidden" errors - # when the Travis build pings Maven Central; see travis-ci/travis-ci#6593. - grep -A 2 '' pom.xml | grep -q 'maven.scijava.org' && - cat >>"$settingsFile" < - - scijava-mirror - SciJava mirror - https://maven.scijava.org/content/groups/public/ - central - - -EOL - cat >>"$settingsFile" < - - gpg - - - \${env.HOME}/.gnupg - - - - \${env.GPG_KEY_NAME} - \${env.GPG_PASSPHRASE} - - - - -EOL - fi - - # Determine whether deploying will be possible. - deployOK= - ciURL=$(mvn -q -Denforcer.skip=true -Dexec.executable=echo -Dexec.args='${project.ciManagement.url}' --non-recursive validate exec:exec 2>&1) - if [ $? -ne 0 ] - then - echo "No deploy -- could not extract ciManagement URL" - echo "Output of failed attempt follows:" - echo "$ciURL" - else - ciRepo=${ciURL##*/} - ciPrefix=${ciURL%/*} - ciOrg=${ciPrefix##*/} - if [ "$TRAVIS_SECURE_ENV_VARS" != true ] - then - echo "No deploy -- secure environment variables not available" - elif [ "$TRAVIS_PULL_REQUEST" != false ] - then - echo "No deploy -- pull request detected" - elif [ "$TRAVIS_REPO_SLUG" != "$ciOrg/$ciRepo" ] - then - echo "No deploy -- repository fork: $TRAVIS_REPO_SLUG != $ciOrg/$ciRepo" - # TODO: Detect travis-ci.org versus travis-ci.com? - else - echo "All checks passed for artifact deployment" - deployOK=1 - fi - fi - - # Install GPG on OSX/macOS - if [ "$TRAVIS_OS_NAME" = osx ] - then - HOMEBREW_NO_AUTO_UPDATE=1 brew install gnupg2 - fi - - # Import the GPG signing key. - keyFile=.travis/signingkey.asc - key=$1 - iv=$2 - if [ "$key" -a "$iv" -a -f "$keyFile.enc" ] - then - # NB: Key and iv values were given as arguments. - echo - echo "== Decrypting GPG keypair ==" - openssl aes-256-cbc -K "$key" -iv "$iv" -in "$keyFile.enc" -out "$keyFile" -d - checkSuccess $? - fi - if [ "$deployOK" -a -f "$keyFile" ] - then - echo - echo "== Importing GPG keypair ==" - gpg --batch --fast-import "$keyFile" - checkSuccess $? - fi - - # Run the build. - BUILD_ARGS='-B -Djdk.tls.client.protocols="TLSv1,TLSv1.1,TLSv1.2"' - if [ "$deployOK" -a "$TRAVIS_BRANCH" = master ] - then - echo - echo "== Building and deploying master SNAPSHOT ==" - mvn -Pdeploy-to-scijava $BUILD_ARGS deploy - checkSuccess $? - elif [ "$deployOK" -a -f release.properties ] - then - echo - echo "== Cutting and deploying release version ==" - mvn -B $BUILD_ARGS release:perform - checkSuccess $? - echo "== Invalidating SciJava Maven repository cache ==" - curl -fsLO https://raw.githubusercontent.com/scijava/scijava-scripts/master/maven-helper.sh && - gav=$(sh maven-helper.sh gav-from-pom pom.xml) && - ga=${gav%:*} && - echo "--> Artifact to invalidate = $ga" && - echo "machine maven.scijava.org" > "$HOME/.netrc" && - echo " login travis" >> "$HOME/.netrc" && - echo " password $MAVEN_PASS" >> "$HOME/.netrc" && - sh maven-helper.sh invalidate-cache "$ga" - checkSuccess $? - else - echo - echo "== Building the artifact locally only ==" - mvn $BUILD_ARGS install javadoc:javadoc - checkSuccess $? - fi - echo travis_fold:end:scijava-maven -fi - -# Configure conda environment, if one is needed. -if [ -f environment.yml ] -then - echo travis_fold:start:scijava-conda - echo "= Conda setup =" - - condaDir=$HOME/miniconda - condaSh=$condaDir/etc/profile.d/conda.sh - if [ ! -f "$condaSh" ]; then - echo - echo "== Installing conda ==" - if [ "$TRAVIS_PYTHON_VERSION" = "2.7" ]; then - wget https://repo.continuum.io/miniconda/Miniconda2-latest-Linux-x86_64.sh -O miniconda.sh - else - wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh - fi - rm -rf "$condaDir" - bash miniconda.sh -b -p "$condaDir" - checkSuccess $? - fi - - echo - echo "== Updating conda ==" - . "$condaSh" && - conda config --set always_yes yes --set changeps1 no && - conda update -q conda && - conda info -a - checkSuccess $? - - echo - echo "== Configuring environment ==" - condaEnv=travis-scijava - test -d "$condaDir/envs/$condaEnv" && condaAction=update || condaAction=create - conda env "$condaAction" -n "$condaEnv" -f environment.yml && - conda activate "$condaEnv" - checkSuccess $? - - echo travis_fold:end:scijava-conda -fi - -# Execute Jupyter notebooks. -if which jupyter >/dev/null 2>/dev/null -then - echo travis_fold:start:scijava-jupyter - echo "= Jupyter notebooks =" - # NB: This part is fiddly. We want to loop over files even with spaces, - # so we use the "find ... -print0 | while read $'\0' ..." idiom. - # However, that runs the piped expression in a subshell, which means - # that any updates to the success variable will not persist outside - # the loop. So we suppress all stdout inside the loop, echoing only - # the final value of success upon completion, and then capture the - # echoed value back into the parent shell's success variable. - success=$(find . -name '*.ipynb' -print0 | { - while read -d $'\0' nbf - do - echo 1>&2 - echo "== $nbf ==" 1>&2 - jupyter nbconvert --execute --stdout "$nbf" >/dev/null - checkSuccess $? - done - echo $success - }) - echo travis_fold:end:scijava-jupyter -fi - -exit $success From a74c2dd6083d634fa08e47cc6d3dfe2a1ba4485f Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Wed, 6 Aug 2025 10:57:44 -0500 Subject: [PATCH 70/80] ci-build.sh: echo the mvn command used to build --- ci-build.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ci-build.sh b/ci-build.sh index a9ed6d2..95fa727 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -317,7 +317,7 @@ EOL BUILD_ARGS="$BUILD_ARGS install javadoc:javadoc" fi # Check the build result. - { mvn $BUILD_ARGS; echo $? > exit-code; } | tee mvn-log + { (set -x; mvn $BUILD_ARGS); echo $? > exit-code; } | tee mvn-log checkSuccess "$(cat exit-code)" mvn-log # --== POST-BUILD ACTIONS ==-- From dfb5cf9ac28622398168571aacc8ac398dcbb3b7 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Sat, 16 Aug 2025 17:10:40 -0500 Subject: [PATCH 71/80] Remove no-longer-functional cache invalidation With the upgrade from Nexus v2 to v3, this trick no longer works. I briefly investigated how to adjust the code to still do it, but was not immediately successful at doing so. Claude.ai suggests: # Invalidate specific paths using the browse API # First, you'd need to identify the asset and use: curl --netrc -X DELETE \ "$NEXUS_BASE_URL/service/rest/v1/assets/{assetId}" But I do not know if this would actually work, and I do not know how to obtain the assetId for a given resource. We can figure it out later if needed. But first, let's proceed without it for a while, to see what problems it creates. --- ci-build.sh | 14 -------------- maven-helper.sh | 31 ------------------------------- 2 files changed, 45 deletions(-) diff --git a/ci-build.sh b/ci-build.sh index 95fa727..d9ab7bf 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -336,20 +336,6 @@ EOL done fi - if [ "$deployOK" -a "$success" -eq 0 ]; then - echo - echo "== Invalidating SciJava Maven repository cache ==" - curl -fsLO https://raw.githubusercontent.com/scijava/scijava-scripts/bdd932af4c4816f88cb6a52cdd7449f175934634/maven-helper.sh && - gav=$(sh maven-helper.sh gav-from-pom pom.xml) && - ga=${gav%:*} && - echo "--> Artifact to invalidate = $ga" && - echo "machine maven.scijava.org" >"$HOME/.netrc" && - echo " login $MAVEN_USER" >>"$HOME/.netrc" && - echo " password $MAVEN_PASS" >>"$HOME/.netrc" && - sh maven-helper.sh invalidate-cache "$ga" - checkSuccess $? - fi - echo ::endgroup:: fi diff --git a/maven-helper.sh b/maven-helper.sh index 54766c2..f5f6153 100755 --- a/maven-helper.sh +++ b/maven-helper.sh @@ -232,29 +232,6 @@ latest_version () { echo "$latest" } -# Given a GA parameter, invalidate the cache in SciJava's Nexus' group/public - -SONATYPE_DATA_CACHE_URL=https://maven.scijava.org/service/local/data_cache/repositories/sonatype/content -SONATYPE_SNAPSHOTS_DATA_CACHE_URL=https://maven.scijava.org/service/local/data_cache/repositories/sonatype-snapshots/content -invalidate_cache () { - ga="$1" - artifactId="$(artifactId "$ga")" - infix="$(groupId "$ga" | tr . /)/$artifactId" - curl --netrc -i -X DELETE \ - $SONATYPE_DATA_CACHE_URL/$infix/maven-metadata.xml && - curl --netrc -i -X DELETE \ - $SONATYPE_SNAPSHOTS_DATA_CACHE_URL/$infix/maven-metadata.xml && - version="$(latest_version "$ga")" && - infix="$infix/$version" && - curl --netrc -i -X DELETE \ - $SONATYPE_DATA_CACHE_URL/$infix/$artifactId-$version.pom && - if test "$artifactId" = "${artifactId#pom-}" - then - curl --netrc -i -X DELETE \ - $SONATYPE_DATA_CACHE_URL/$infix/$artifactId-$version.jar - fi -} - # Generate a temporary file; not thread-safe tmpfile () { @@ -477,9 +454,6 @@ all-deps|all-dependencies) latest-version) latest_version "$2" ;; -invalidate-cache) - invalidate_cache "$2" - ;; gav-from-pom) gav_from_pom "$2" ;; @@ -530,11 +504,6 @@ latest-version :[:] passed as version, it prints the current snapshot version rather than the release one). -invalidate-cache : - Invalidates the version cached in the SciJava Nexus from OSS Sonatype, - e.g. after releasing a new version to Sonatype. Requires appropriate - credentials in $HOME/.netrc for https://maven.scijava.org/. - parent-gav :[:] Prints the GAV parameter of the parent project of the given artifact. From 90b2b83cb402e562fd857ad59473263ee0e29b54 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Wed, 17 Sep 2025 07:26:52 -0700 Subject: [PATCH 72/80] release-version.sh: fix pom-scijava version check We can't ask maven.scijava.org anymore, because it no longer proxies Maven Central since the upgrade to Nexus v3. So instead, we ask Maven Central directly, which works well. --- release-version.sh | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/release-version.sh b/release-version.sh index b0eb696..8f95226 100755 --- a/release-version.sh +++ b/release-version.sh @@ -181,13 +181,10 @@ test "$SKIP_VERSION_CHECK" || { } # Check that the project extends the latest version of pom-scijava. -MAVEN_HELPER="$(cd "$(dirname "$0")" && pwd)/maven-helper.sh" -test -f "$MAVEN_HELPER" || - die "Missing helper script at '$MAVEN_HELPER' -Do you have a full clone of https://github.com/scijava/scijava-scripts?" test "$SKIP_VERSION_CHECK" -o "$parentGAV" != "${parentGAV#$}" || { debug "Checking pom-scijava parent version" - latestParentVersion=$(sh -$- "$MAVEN_HELPER" latest-version "$parentGAV") + psjMavenMetadata=https://repo1.maven.org/maven2/org/scijava/pom-scijava/maven-metadata.xml + latestParentVersion=$(curl -fsL "$psjMavenMetadata" | grep '' | sed 's;.*>\([^<]*\)<.*;\1;') currentParentVersion=${parentGAV##*:} test "$currentParentVersion" = "$latestParentVersion" || die "Newer version of parent '$parentGAV' is available: $latestParentVersion. From c51053ad780df2e40ed8d47e7529bcfd9177b5f8 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 8 Sep 2025 12:34:05 -0500 Subject: [PATCH 73/80] Remove the awesome maven-helper.sh script :-( It saddens me to do this. But many of the functions are now broken, because maven.scijava.org cannot proxy Maven Central any longer, due to Sonatype deciding that Nexus v3 should be crippleware from now on. --- maven-helper.sh | 536 ------------------------------------------------ 1 file changed, 536 deletions(-) delete mode 100755 maven-helper.sh diff --git a/maven-helper.sh b/maven-helper.sh deleted file mode 100755 index f5f6153..0000000 --- a/maven-helper.sh +++ /dev/null @@ -1,536 +0,0 @@ -#!/bin/sh - -# This script uses the SciJava Maven repository at https://maven.scijava.org/ -# to fetch an artifact, or to determine the state of it. - -# error out whenever a command fails -set -e - -root_url () { - test snapshots != "$2" || { - if curl -fs https://maven.scijava.org/service/local/repositories/sonatype-snapshots/content/"$1"/maven-metadata.xml > /dev/null 2>&1 - then - echo https://maven.scijava.org/service/local/repositories/sonatype-snapshots/content - else - echo https://maven.scijava.org/content/repositories/snapshots - fi - return - } - echo https://maven.scijava.org/service/local/repo_groups/public/content -} - -die () { - echo "$*" >&2 - exit 1 -} - -# Helper (thanks, BSD!) - -get_mtime () { - stat -c %Y "$1" -} -case "$(uname -s 2> /dev/null)" in -MINGW*) - get_mtime () { - date -r "$1" +%s - } - ;; -Darwin) - get_mtime () { - stat -f %m "$1" - } - ;; -esac - -# Parse :: triplets (i.e. GAV parameters) - -groupId () { - echo "${1%%:*}" -} - -artifactId () { - result="${1#*:}" - echo "${result%%:*}" -} - -version () { - result="${1#*:}" - case "$result" in - *:*) - echo "${1##*:}" - ;; - esac -} - -# Given an xml, extract the first - -extract_tag () { - result="${2%%*}" - case "$result" in - "$2") - ;; - *) - echo "${result#*<$1>}" - ;; - esac -} - -# Given an xml, extract the last - -extract_last_tag () { - result="${2##*<$1>}" - case "$result" in - "$2") - ;; - *) - echo "${result%%*}" - ;; - esac -} - -# Given an xml, skip all sections - -skip_tag () { - result="$2" - while true - do - case "$result" in - *"<$1>"*) - result="${result%%<$1>*}${result#*}" - ;; - *) - break - ;; - esac - done - echo "$result" -} - -# Given the xml of a POM, find the parent GAV - -parent_gav_from_pom_xml () { - pom="$1" - parent="$(extract_tag parent "$pom")" - test -n "$parent" || return - groupId="$(extract_tag groupId "$parent")" - artifactId="$(extract_tag artifactId "$parent")" - version="$(extract_tag version "$parent")" - echo "$groupId:$artifactId:$version" -} - -# Given a GAV parameter, determine the base URL of the project - -project_url () { - gav="$1" - artifactId="$(artifactId "$gav")" - infix="$(groupId "$gav" | tr . /)/$artifactId" - version="$(version "$gav")" - case "$version" in - *SNAPSHOT) - echo "$(root_url $infix snapshots)/$infix" - ;; - *) - # Release could be in either releases or thirdparty; try releases first - project_url="$(root_url $infix releases)/$infix" - header=$(curl -Is "$project_url/") - case "$header" in - HTTP/1.?" 200 OK"*) - ;; - *) - project_url="$(root_url $infix thirdparty)/$infix" - ;; - esac - echo "$project_url" - ;; - esac -} - -# Given a GAV parameter, determine the URL of the .jar file - -jar_url () { - gav="$1" - artifactId="$(artifactId "$gav")" - version="$(version "$gav")" - infix="$(groupId "$gav" | tr . /)/$artifactId/$version" - case "$version" in - *-SNAPSHOT) - url="$(root_url $infix snapshots)/$infix/maven-metadata.xml" - metadata="$(curl -s "$url")" - timestamp="$(extract_tag timestamp "$metadata")" - buildNumber="$(extract_tag buildNumber "$metadata")" - version=${version%-SNAPSHOT}-$timestamp-$buildNumber - echo "$(root_url $infix snapshots)/$infix/$artifactId-$version.jar" - ;; - *) - echo "$(root_url $infix releases)/$infix/$artifactId-$version.jar" - ;; - esac -} - -# Given a GAV parameter, return the URL to the corresponding .pom file - -pom_url () { - url="$(jar_url "$1")" - echo "${url%.jar}.pom" -} - -# Given a POM file, find its GAV parameter - -gav_from_pom () { - pom="$(cat "$1")" - parent="$(extract_tag parent "$pom")" - pom="$(skip_tag parent "$pom")" - pom="$(skip_tag dependencies "$pom")" - pom="$(skip_tag profiles "$pom")" - pom="$(skip_tag build "$pom")" - groupId="$(extract_tag groupId "$pom")" - test -n "$groupId" || groupId="$(extract_tag groupId "$parent")" - artifactId="$(extract_tag artifactId "$pom")" - version="$(extract_tag version "$pom")" - test -n "$version" || version="$(extract_tag version "$parent")" - echo "$groupId:$artifactId:$version" -} - -# Given a GAV parameter, find its parent's GAV - -parent_gav () { - gav="$1" - groupId="$(groupId "$gav")" - artifactId="$(artifactId "$gav")" - version="$(version "$gav")" - test -n "$version" || version="$(latest_version "$gav")" - pom="$(read_pom "$groupId:$artifactId:$version")" - parent_gav_from_pom_xml "$pom" -} - -# Given a POM file, find its parent's GAV - -parent_gav_from_pom () { - pom="$(cat "$1")" - parent_gav_from_pom_xml "$pom" -} - -# Given a POM file, extract its packaging - -packaging_from_pom () { - pom="$(cat "$1")" - pom="$(skip_tag parent "$pom")" - pom="$(skip_tag dependencies "$pom")" - pom="$(skip_tag profiles "$pom")" - pom="$(skip_tag build "$pom")" - packaging="$(extract_tag packaging "$pom")" - echo "${packaging:-jar}" -} - -# Given a GAV parameter possibly lacking a version, determine the latest version - -latest_version () { - metadata="$(curl -s "$(project_url "$1")"/maven-metadata.xml)" - latest="$(extract_tag release "$metadata")" - test -n "$latest" || latest="$(extract_tag latest "$metadata")" - test -n "$latest" || latest="$(extract_last_tag version "$metadata")" - echo "$latest" -} - -# Generate a temporary file; not thread-safe - -tmpfile () { - i=1 - while test -f /tmp/precompiled.$i"$1" - do - i=$(($i+1)) - done - echo /tmp/precompiled.$i"$1" -} - -# Given a GAV or a path, read the POM - -read_pom () { - case "$1" in - pom.xml|*/pom.xml|*\\pom.xml) - cat "$1" - ;; - *) - curl -s "$(pom_url "$1")" - ;; - esac -} - -# Given a GAV parameter (or pom.xml path) and a name, resolve a property (falling back to parents) - -get_property () { - gav="$1" - key="$2" - case "$key" in - imagej1.version) - latest_version net.imagej:ij - return - ;; - project.groupId) - groupId "$gav" - return - ;; - project.version) - version "$gav" - return - ;; - esac - while test -n "$gav" - do - pom="$(read_pom "$gav")" - properties="$(extract_tag properties "$pom")" - property="$(extract_tag "$key" "$properties")" - if test -n "$property" - then - echo "$property" - return - fi - gav="$(parent_gav_from_pom_xml "$pom")" - done - die "Could not resolve \${$2} in $1" -} - -# Given a GAV parameter and a string, expand properties - -expand () { - gav="$1" - string="$2" - result= - while true - do - case "$string" in - *'${'*'}'*) - result="$result${string%%\$\{*}" - string="${string#*\$\{}" - key="${string%\}*}" - result="$result$(get_property "$gav" "$key")" - string="${string#$key\}}" - ;; - *) - echo "$result$string" - break - ;; - esac - done -} - -# Given a GAV parameter, make a list of its dependencies (as GAV parameters) - -get_dependencies () { - pom="$(read_pom "$1")" - while true - do - case "$pom" in - *''*) - dependency="$(extract_tag dependency "$pom")" - scope="$(extract_tag scope "$dependency")" - case "$scope" in - ''|compile) - groupId="$(expand "$1" "$(extract_tag groupId "$dependency")")" - artifactId="$(extract_tag artifactId "$dependency")" - version="$(expand "$1" "$(extract_tag version "$dependency")")" - echo "$groupId:$artifactId:$version" - ;; - esac - pom="${pom#*}" - ;; - *) - break; - esac - done -} - -# Given a GAV parameter and a space-delimited list of GAV parameters, expand -# the list by the first parameter and its dependencies (unless the list already -# contains said parameter) - -get_all_dependencies () { - case " $2 " in - *" $1 "*) - ;; # list already contains the depdendency - *) - gav="$1" - set "" "$2 $1" - for dependency in $(get_dependencies "$gav") - do - set "" "$(get_all_dependencies "$dependency" "$2")" - done - ;; - esac - echo "$2" -} - -# Given a GAV parameter, download the .jar file - -get_jar () { - url="$(jar_url "$1")" - tmpfile="$(tmpfile .jar)" - curl -s "$url" > "$tmpfile" - test " "$tmpfile" - test PK = "$(head -c 2 "$tmpfile")" - echo "$tmpfile" -} - -# Given a GAV parameter, get the commit from the manifest of the deployed .jar - -commit_from_gav () { - jar="$(get_jar "$1")" - unzip -p "$jar" META-INF/MANIFEST.MF | - sed -n -e 's/^Implementation-Build: *//pi' | - tr -d '\r' - rm "$jar" -} - -# Given a GAV parameter, determine whether the .jar file is already in plugins/ -# or jars/ - -is_jar_installed () { - artifactId="$(artifactId "$1")" - version="$(version "$1")" - file=$artifactId-$version.jar - test -f "$file" || file=../plugins/$file - test -f "$file" || return 1 - case "$version" in - *-SNAPSHOT) - # is the file younger than a day? - mtime="$(get_mtime "$file")" - test "$(($mtime-$(date +%s)))" -gt -86400 - ;; - esac -} - -# Given a .jar file, determine whether it is an ImageJ 1.x plugin - -is_ij1_plugin () { - unzip -l "$1" plugins.config > /dev/null 2>&1 -} - -# Given a GAV parameter, download the .jar file and its dependencies as needed -# and install them into plugins/ or jars/, respectively - -install_jar () { - for gav in $(get_all_dependencies "$1") - do - if ! is_jar_installed "$gav" - then - tmp="$(get_jar "$gav")" - name="$(artifactId "$gav")-$(version "$gav").jar" - if test -d ../plugins && is_ij1_plugin "$tmp" - then - mv "$tmp" "../plugins/$name" - else - mv "$tmp" "$name" - fi - fi - done -} - -# Determine whether a local project (specified as pom.xml) needs to be deployed - -is_deployed () { - gav="$(gav_from_pom "$1")" && - commit="$(commit_from_gav "$gav")" && - test -n "$commit" && - dir="$(dirname "$1")" && - (cd "$dir" && - git diff --quiet "$commit".. -- .) -} - -# The main part - -case "$1" in -commit) - commit_from_gav "$2" - ;; -deps|dependencies) - get_dependencies "$2" - ;; -all-deps|all-dependencies) - get_all_dependencies "$2" | - tr ' ' '\n' | - grep -v '^$' - ;; -latest-version) - latest_version "$2" - ;; -gav-from-pom) - gav_from_pom "$2" - ;; -parent-gav) - parent_gav "$2" - ;; -pom-url) - pom_url "$2" - ;; -parent-gav-from-pom) - parent_gav_from_pom "$2" - ;; -packaging-from-pom) - packaging_from_pom "$2" - ;; -property-from-pom|get-property|property) - if test $# -lt 3 - then - get_property pom.xml "$2" - else - get_property "$2" "$3" - fi - ;; -install) - install_jar "$2" - ;; -is-deployed) - is_deployed "$2" - ;; -*) - test $# -eq 0 || echo "Unknown command: $1" >&2 - die "Usage: $0 [command] [argument...]"' - -Commands: - -commit :: - Gets the commit from which the given artifact was built. - -dependencies :: - Lists the direct dependencies of the given artifact. - -all-dependencies :: - Lists all dependencies of the given artifact, including itself and - transitive dependencies. - -latest-version :[:] - Prints the current version of the given artifact (if "SNAPSHOT" is - passed as version, it prints the current snapshot version rather - than the release one). - -parent-gav :[:] - Prints the GAV parameter of the parent project of the given artifact. - -pom-url :: - Gets the URL of the POM describing the given artifact. - -gav-from-pom - Prints the GAV parameter described in the given pom.xml file. - -parent-gav-from-pom - Prints the GAV parameter of the parent project of the pom.xml file. - -packaging-from-pom - Prints the packaging type of the given project. - -property-from-pom - Prints the property specified in the pom.xml file (or in its parents). - -install :: - Installs the given artifact and all its dependencies; if the artifact - or dependency to install is an ImageJ 1.x plugin and the parent - directory contains a subdirectory called "plugins", it will be - installed there, otherwise into the current directory. - -is-deployed - Tests whether the specified project is deployed alright. Fails - with exit code 1 if not. -' - ;; -esac From 9848f55649a1691df6bd9395eb6b54b7e88e2ae6 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 25 Nov 2025 15:58:03 -0600 Subject: [PATCH 74/80] Update github-actionification * actions/checkout: v2 -> v4 * actions/setup-java: v3 -> v4 * ensure setup and build scripts run with bash (hard stare at Windows) --- github-actionify.sh | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/github-actionify.sh b/github-actionify.sh index a88d24f..f6c5a56 100755 --- a/github-actionify.sh +++ b/github-actionify.sh @@ -109,9 +109,9 @@ process() { # -- GitHub Action steps -- - actionCheckout="uses: actions/checkout@v2" + actionCheckout="uses: actions/checkout@v4" actionSetupJava="name: Set up Java - uses: actions/setup-java@v3 + uses: actions/setup-java@v4 with: java-version: '8' distribution: 'zulu' @@ -121,9 +121,11 @@ process() { - name: Install conda packages run: conda env update -f environment.yml -n base" actionSetupCI="name: Set up CI environment - run: $ciSetupScript" + run: $ciSetupScript + shell: bash" actionExecuteBuild="name: Execute the build - run: $ciBuildScript" + run: $ciBuildScript + shell: bash" actionSecrets="env: GPG_KEY_NAME: \${{ secrets.GPG_KEY_NAME }} GPG_PASSPHRASE: \${{ secrets.GPG_PASSPHRASE }} From 2e4bc1b99ac3edf8bd6cce119c2ca75792f1143c Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 25 Nov 2025 16:19:46 -0600 Subject: [PATCH 75/80] ci-build.sh: use single quotes for string literals --- ci-build.sh | 56 ++++++++++++++++++++++++++--------------------------- 1 file changed, 28 insertions(+), 28 deletions(-) diff --git a/ci-build.sh b/ci-build.sh index d9ab7bf..caa05de 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -65,7 +65,7 @@ if [ -f pom.xml ]; then # --== MAVEN SETUP ==-- echo - echo "== Configuring Maven ==" + echo '== Configuring Maven ==' # NB: Suppress "Downloading/Downloaded" messages. # See: https://stackoverflow.com/a/35653426/1207769 @@ -76,12 +76,12 @@ if [ -f pom.xml ]; then settingsFile="$HOME/.m2/settings.xml" customSettings=.ci/settings.xml if [ "$OSSRH_USER" -o "$OSSRH_PASS" ]; then - echo "[WARNING] Obsolete OSSRH vars detected. Secrets may need updating to deploy to Maven Central." + echo '[WARNING] Obsolete OSSRH vars detected. Secrets may need updating to deploy to Maven Central.' fi if [ -f "$customSettings" ]; then cp "$customSettings" "$settingsFile" elif [ -z "$BUILD_REPOSITORY" ]; then - echo "Skipping settings.xml generation (no BUILD_REPOSITORY; assuming we are running locally)" + echo 'Skipping settings.xml generation (no BUILD_REPOSITORY; assuming we are running locally)' else # settings.xml header cat >"$settingsFile" < EOL else - echo "[WARNING] Skipping settings.xml scijava servers (no MAVEN deployment credentials)." + echo '[WARNING] Skipping settings.xml scijava servers (no MAVEN deployment credentials).' fi # settings.xml central server if [ "$CENTRAL_USER" -a "$CENTRAL_PASS" ]; then @@ -115,7 +115,7 @@ EOL EOL else - echo "[WARNING] Skipping settings.xml central server (no CENTRAL deployment credentials)." + echo '[WARNING] Skipping settings.xml central server (no CENTRAL deployment credentials).' fi cat >>"$settingsFile" < @@ -139,7 +139,7 @@ EOL EOL else - echo "[WARNING] Skipping settings.xml gpg profile (no GPG credentials)." + echo '[WARNING] Skipping settings.xml gpg profile (no GPG credentials).' fi # settings.xml footer cat >>"$settingsFile" < "$keyFile" ls -la "$keyFile" @@ -297,7 +297,7 @@ EOL fi fi else - echo "[WARNING] Skipping gpg setup (no GPG credentials)." + echo '[WARNING] Skipping gpg setup (no GPG credentials).' fi # --== BUILD EXECUTION ==-- @@ -305,15 +305,15 @@ EOL # Run the build. if [ "$deployOK" -a -f release.properties ]; then echo - echo "== Cutting and deploying release version ==" + echo '== Cutting and deploying release version ==' BUILD_ARGS="$BUILD_ARGS release:perform" elif [ "$deployOK" ]; then echo - echo "== Building and deploying main branch SNAPSHOT ==" + echo '== Building and deploying main branch SNAPSHOT ==' BUILD_ARGS="-Pdeploy-to-scijava $BUILD_ARGS deploy" else echo - echo "== Building the artifact locally only ==" + echo '== Building the artifact locally only ==' BUILD_ARGS="$BUILD_ARGS install javadoc:javadoc" fi # Check the build result. From 54c788f390e1d46b64ec29104d23778f900cce83 Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Tue, 25 Nov 2025 16:24:33 -0600 Subject: [PATCH 76/80] ci-build.sh: analyze workflow if vars are missing --- ci-build.sh | 63 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/ci-build.sh b/ci-build.sh index caa05de..8d68269 100755 --- a/ci-build.sh +++ b/ci-build.sh @@ -58,6 +58,66 @@ mavenEvaluate() { mvn -B -U -q -Denforcer.skip=true -Dexec.executable=echo -Dexec.args="$1" --non-recursive validate exec:exec 2>&1 } +# Output debugging info, for troubleshooting builds. +dump() { + if env | grep -q ^$1= + then + # Environment variable is set. + line=$(env | grep ^$1=) + if [ "$line" ] + then + # And it's non-empty. + if [ "$2" ] + then + # Secret value: emit only that it is present. + echo "$1=" + else + # Not a secret: emit in plaintext. + echo "$line" + fi + else + # But it is empty! + echo "$1=" + fi + else + # Environment variable is not set. + echo "$1=" + fi +} + +analyzeGitHubWorkflow() { + test -d .github/workflows || return + echo + echo '/--------------------------------------------------\' + echo '| ci-build.sh analysis: env vars + GitHub workflow |' + echo '\--------------------------------------------------/' + dump dir + dump platform + dump BUILD_REPOSITORY + dump NO_DEPLOY + echo '----------------------------------------------------' + dump GPG_KEY_NAME secret + dump GPG_PASSPHRASE secret + dump MAVEN_USER secret + dump MAVEN_PASS secret + dump CENTRAL_USER secret + dump CENTRAL_PASS secret + dump SIGNING_ASC secret + echo '----------------------------------------------------' + for var in \ + GPG_KEY_NAME \ + GPG_PASSPHRASE \ + MAVEN_USER \ + MAVEN_PASS \ + CENTRAL_USER \ + CENTRAL_PASS \ + SIGNING_ASC + do + grep -q "secrets.$var" .github/workflows/*.yml || + echo "Add \`$var: \${{ secrets.$var }}\` to GitHub workflow env section!" + done +} + # Build Maven projects. if [ -f pom.xml ]; then echo ::group::"= Maven build =" @@ -193,6 +253,7 @@ EOL deployOK=1 else echo 'No deploy -- MAVEN environment variables not available' + analyzeGitHubWorkflow fi ;; *) @@ -219,6 +280,7 @@ EOL deployOK=1 else echo '[ERROR] Cannot deploy: MAVEN environment variables not available' + analyzeGitHubWorkflow exit 1 fi ;; @@ -229,6 +291,7 @@ EOL deployOK=1 else echo '[ERROR] Cannot deploy: CENTRAL environment variables not available' + analyzeGitHubWorkflow exit 1 fi ;; From 8b1eabf060daa50562c80a0606ba6554f1f1d3ed Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Wed, 26 Nov 2025 10:25:31 -0600 Subject: [PATCH 77/80] release-version.sh: detect external auto-staging See mobie/mobie-viewer-fiji#1104. --- release-version.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/release-version.sh b/release-version.sh index 8f95226..4505de4 100755 --- a/release-version.sh +++ b/release-version.sh @@ -352,6 +352,25 @@ then $DRY_RUN git commit -s -m "Bump to next development cycle" fi && +# Check that release-only files were not erroneously committed. +if test -z "$DRY_RUN" +then + debug "Verifying release-only files were not committed" + for release_file in release.properties pom.xml.releaseBackup + do + if git diff --name-only HEAD~1 HEAD | grep -qF "$release_file" + then + die "FATAL: $release_file was committed to the branch! +This likely means your IDE automatically staged these files. +Please configure your IDE to NOT auto-stage files during the release process. +You can undo this failed release with: + git reset --hard HEAD~1 + git tag -d \$(sed -n 's/^scm.tag=//p' < release.properties) +Then try the release again after disabling IDE auto-staging." + fi + done +fi && + # Extract the name of the new tag. debug "Extracting new tag name" if test -z "$DRY_RUN" From 8c5e3bd71aa954eb0efd0bb9edeb9b54e25f953b Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Mon, 4 May 2026 13:30:03 -0500 Subject: [PATCH 78/80] release-version: skip license check if POM says so --- release-version.sh | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/release-version.sh b/release-version.sh index 4505de4..cae67d1 100755 --- a/release-version.sh +++ b/release-version.sh @@ -118,7 +118,7 @@ Options include: # -- Extract project details -- debug "Extracting project details" -echoArg='${project.version}:${license.licenseName}:${project.parent.groupId}:${project.parent.artifactId}:${project.parent.version}' +echoArg='${project.version}:${license.licenseName}:${project.parent.groupId}:${project.parent.artifactId}:${project.parent.version}:${license.skipUpdateProjectLicense}' projectDetails=$(mvn -B -N -Dexec.executable=echo -Dexec.args="$echoArg" exec:exec -q) test $? -eq 0 || projectDetails=$(mvn -B -U -N -Dexec.executable=echo -Dexec.args="$echoArg" exec:exec -q) test $? -eq 0 || die "Could not extract version from pom.xml. Error follows:\n$projectDetails" @@ -132,7 +132,9 @@ projectDetails=$(printf '%s' "$projectDetails" | tr -d '\n') currentVersion=${projectDetails%%:*} projectDetails=${projectDetails#*:} licenseName=${projectDetails%%:*} -parentGAV=${projectDetails#*:} +projectDetails=${projectDetails#*:} +parentGAV=${projectDetails%%:*} +skipUpdateProjectLicense=${projectDetails#*:} # -- Sanity checks -- debug "Performing sanity checks" @@ -315,7 +317,7 @@ then fi # Ensure license headers are up-to-date. -test "$SKIP_LICENSE_UPDATE" -o -z "$licenseName" -o "$licenseName" = "N/A" || { +test "$SKIP_LICENSE_UPDATE" -o -z "$licenseName" -o "$licenseName" = "N/A" -o "$skipUpdateProjectLicense" = "true" || { debug "Ensuring that license headers are up-to-date" mvn license:update-project-license license:update-file-header && git add LICENSE.txt || die 'Failed to update copyright blurbs. From ec727b1c070796c9657be50e1ae029eb01352f8b Mon Sep 17 00:00:00 2001 From: Caleb Hulbert Date: Mon, 4 May 2026 15:28:18 -0400 Subject: [PATCH 79/80] fix: parsing for parentGAV and skipUpdateProjectLicense --- release-version.sh | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/release-version.sh b/release-version.sh index cae67d1..3b6a595 100755 --- a/release-version.sh +++ b/release-version.sh @@ -133,7 +133,12 @@ currentVersion=${projectDetails%%:*} projectDetails=${projectDetails#*:} licenseName=${projectDetails%%:*} projectDetails=${projectDetails#*:} -parentGAV=${projectDetails%%:*} +parentGroup=${projectDetails%%:*} +projectDetails=${projectDetails#*:} +parentArtifact=${projectDetails%%:*} +projectDetails=${projectDetails#*:} +parentVersion=${projectDetails%%:*} +parentGAV="${parentGroup}:${parentArtifact}:${parentVersion}" skipUpdateProjectLicense=${projectDetails#*:} # -- Sanity checks -- From f5482fdb8471824b3aae4acc3fee8cd9927846db Mon Sep 17 00:00:00 2001 From: Curtis Rueden Date: Fri, 17 Jul 2026 22:29:21 -0500 Subject: [PATCH 80/80] release-version: check version of pom-scijava only If the parent POM is not pom-scijava, don't worry about it. This commit fixes a bug introduced in 90b2b83cb4, where the parent POM version gets checked against the newest org.scijava:pom-scijava release version, even for parent POMs besides org.scijava:pom-scijava. --- release-version.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/release-version.sh b/release-version.sh index 3b6a595..92722e2 100755 --- a/release-version.sh +++ b/release-version.sh @@ -188,7 +188,7 @@ test "$SKIP_VERSION_CHECK" || { } # Check that the project extends the latest version of pom-scijava. -test "$SKIP_VERSION_CHECK" -o "$parentGAV" != "${parentGAV#$}" || { +test "$SKIP_VERSION_CHECK" -o "$parentGroup" != "org.scijava" -o "$parentArtifact" != "pom-scijava" || { debug "Checking pom-scijava parent version" psjMavenMetadata=https://repo1.maven.org/maven2/org/scijava/pom-scijava/maven-metadata.xml latestParentVersion=$(curl -fsL "$psjMavenMetadata" | grep '' | sed 's;.*>\([^<]*\)<.*;\1;')