Fixing release script bugs, making formal releases manual (#1590)

garrettjonesgoogle · web-flow · commit b67c153efbbb · 2017-02-08T11:58:11.000-08:00
* Not auto-releasing after close (so that releases can be staged)
* Splitting release.sh into stage_release.sh and finalize_release.sh
* Stripping the qualifier from the SITE_VERSION for generating the gh-pages site
* Fixing deploy.sh (the input checking was wrong)
* Not running any tests in deploy.sh (they are already run in other contexts
  before this point)
* Updating RELEASING.md with the new process
diff --git a/RELEASING.md b/RELEASING.md
@@ -1,18 +1,25 @@
 ### Overview
 
-Most of the release process is handled by the `after_success.sh` script, triggered after Travis CI successfully completes a non-PR build.  A new artifact will be released to Maven Central Repository via Travis CI when "-SNAPSHOT" is not included in the version (as listed in the base directory's `pom.xml`).  The website and README files will also be updated automatically in this case.  When "-SNAPSHOT" is included in the version, Travis only updates the artifact in the snapshot repository.
+The release process for SNAPSHOT versions is handled by the `after_success.sh` script, triggered after Travis CI successfully completes a non-PR build. A new SNAPSHOT artifact will be released to the snapshot repository as part of this script.
 
 ### To push a release version
 
-1. Run `utilities/update_pom_version.sh` from the repository's base directory.
-This script takes an optional argument denoting the new version.  By default, if the current version is X.Y.Z-SNAPSHOT, the script will update the version in all the pom.xml files to X.Y.Z.  If desired, another version can be supplied via command line argument instead.
+1. Make sure the team agrees that it is time to release. Verify that all unit and integration tests for the last commit have passed.
 
-2. Create a PR to update the pom.xml version.  If releasing a new client library, this PR should also update javadoc grouping in the base directory's [pom.xml](./pom.xml).
-PRs that don't release new modules should look something like [#225](https://github.com/GoogleCloudPlatform/google-cloud-java/pull/225).  PRs that do release a new module should also add the appropriate packages to the javadoc groups "SPI" and "Test helpers", as shown in [#802](https://github.com/GoogleCloudPlatform/google-cloud-java/pull/802) for `google-cloud-dns`. After this PR is merged into GoogleCloudPlatform/google-cloud-java, Travis CI will push a new website to GoogleCloudPlatform/gh-pages, push a new artifact to the Maven Central Repository, and update versions in the README files.  Do not merge in any non-release-related pull requests between the start of step 2 and the end of step 6.  Between these steps, the project version is a non-snapshot version, so any commits to the master branch will cause Travis to spend extra resources attempting to redeploy artifacts.
+2. Run `utilities/update_pom_version.sh` from the repository's base directory.
+This script takes an optional argument denoting the new version. By default, if the current version is X.Y.Z-SNAPSHOT, the script will update the version in all the pom.xml files to X.Y.Z. If desired, another version can be supplied via command line argument instead. Commit this version locally:
 
-3. Before moving on, verify that the artifacts have successfully been pushed to the Maven Central Repository.  Open Travis CI, click the ["Build History" tab](https://travis-ci.org/GoogleCloudPlatform/google-cloud-java/builds), and open the second build's logs for Step 2's PR.  Be sure that you are not opening the "Pull Request" build logs.  When the build finishes, scroll to the end of the log and verify that the artifacts were successfully staged and deployed.  Search for `google-cloud` on the [Sonatype website](https://oss.sonatype.org/#nexus-search;quick~google-cloud) and check the latest version number.  In rare cases (when the Maven plugin that determines the version of the repository fails), the artifacts may not be deployed even if the version in the pom.xml files doesn't contain `SNAPSHOT`.  If the artifacts weren't deployed due to invalid version parsing or a flaky test, rerun the build.
+  ```
+  git commit -m "Release [VERSION HERE]"
+  ```
+
+3. Run `utilities/stage_release.sh`.
+This script builds and stages the release artifact on the Maven Central Repository, updates the README.md files with the release version + commits them locally, and finally generates a new site version for the gh-pages branch under a temporary directory named `tmp_gh-pages`. If you haven't run the release process before, it's worth verifying everything; check the staged release on the Sonatype website, and verify that the local commits have the right version updates.
+
+4. Run `utilities/finalize_release.sh`.
+This script will release the staged artifact on the Maven Central Repository and push the README.md and gh-pages updates to github.
 
-4. Publish a release on Github manually.
+5. Publish a release on Github manually.
 Go to the [releases page](https://github.com/GoogleCloudPlatform/google-cloud-java/releases) and open the appropriate release draft. Make sure the "Tag Version" is `vX.Y.Z` and the "Release Title" is `X.Y.Z`, where `X.Y.Z` is the release version as listed in the `pom.xml` files.  The draft should already have all changes that impact users since the previous release.  To double check this, you can use the `git log` command and look through the merged master branch pull requests.  Here is an example of the log command to get non-merge commits between v0.0.12 and v0.1.0:
 
   ```
@@ -21,12 +28,12 @@ Go to the [releases page](https://github.com/GoogleCloudPlatform/google-cloud-ja
 
   Ensure that the format is consistent with previous releases (for an example, see the [0.1.0 release](https://github.com/GoogleCloudPlatform/google-cloud-java/releases/tag/v0.1.0)).  After adding any missing updates and reformatting as necessary, publish the draft.  Finally, create a new draft for the next release.
 
-5. Run `utilities/update_pom_version.sh` again (to include "-SNAPSHOT" in the project version).
+6. Run `utilities/update_pom_version.sh` again (to include "-SNAPSHOT" in the project version).
 As mentioned before, there is an optional version argument.  By default, the script will update the version from "X.Y.Z" to "X.Y.Z+1-SNAPSHOT".  Suppose a different version is desired, for example X+1.0.0-SNAPSHOT.  Then the appropriate command to run would be `utilities/update_pom_version.sh X+1.0.0-SNAPSHOT`.
 
-6. Create and merge in another PR to reflect the updated project version.  For an example of what this PR should look like, see [#227](https://github.com/GoogleCloudPlatform/google-cloud-java/pull/227).
+7. Create and merge in another PR to reflect the updated project version.  For an example of what this PR should look like, see [#227](https://github.com/GoogleCloudPlatform/google-cloud-java/pull/227).
 
-7. Be sure to update Google Cloud Platform docs, [java-docs-samples](https://github.com/GoogleCloudPlatform/java-docs-samples) code/docs, and [getting-started-java](https://github.com/GoogleCloudPlatform/getting-started-java) code/docs.  See directions [here](https://docs.google.com/a/google.com/document/d/1SS3xNn2v0qW7EadGUPBUAPIQAH5VY6WSFmT17ZjjUVE/).
+8. Be sure to update Google Cloud Platform docs, [java-docs-samples](https://github.com/GoogleCloudPlatform/java-docs-samples) code/docs, and [getting-started-java](https://github.com/GoogleCloudPlatform/getting-started-java) code/docs.  See directions [here](https://docs.google.com/a/google.com/document/d/1SS3xNn2v0qW7EadGUPBUAPIQAH5VY6WSFmT17ZjjUVE/).
 
 ### To push a snapshot version
 
diff --git a/pom.xml b/pom.xml
@@ -271,7 +271,7 @@
         <configuration>
           <serverId>sonatype-nexus-staging</serverId>
           <nexusUrl>https://oss.sonatype.org/</nexusUrl>
-          <autoReleaseAfterClose>true</autoReleaseAfterClose>
+          <autoReleaseAfterClose>false</autoReleaseAfterClose>
         </configuration>
       </plugin>
       <plugin>
diff --git a/utilities/after_success.sh b/utilities/after_success.sh
@@ -17,37 +17,8 @@ if [ "${TRAVIS_JDK_VERSION}" == "oraclejdk7" ]; then
             echo "Could not determine the version, so we're exiting."
             exit 1
         fi
-        if [ "${SITE_VERSION##*-}" != "SNAPSHOT" ]; then
-            # Deploy Maven artifacts (if they don't exist yet) and update artifact version in READMEs.
-            URL=https://oss.sonatype.org/content/repositories/releases/com/google/cloud/google-cloud/$SITE_VERSION/
-            if curl --output /dev/null --silent --head --fail "$URL"; then
-                echo "Not deploying artifacts because it seems like they already exist."
-                echo "Existence was checked using the url $URL"
-            else
-                mvn clean deploy --quiet -Djava.util.logging.config.file=logging.properties -DskipITs --settings ~/.m2/settings.xml -P sign-deploy,release
-            fi
-            utilities/update_docs_version.sh
 
-            # Create website
-            git config --global user.name "travis-ci"
-            git config --global user.email "travis@travis-ci.org"
-            git clone --branch gh-pages --single-branch https://github.com/GoogleCloudPlatform/google-cloud-java/ tmp_gh-pages
-            mkdir -p tmp_gh-pages/$SITE_VERSION
-            mvn site -DskipTests=true -Djava.util.logging.config.file=logging.properties
-            mvn site:stage --quiet -Djava.util.logging.config.file=logging.properties -DtopSiteURL=http://googlecloudplatform.github.io/google-cloud-java/site/${SITE_VERSION}/
-            cd tmp_gh-pages
-            cp -r ../target/staging/$SITE_VERSION/* $SITE_VERSION/
-            sed -i "s/{{SITE_VERSION}}/$SITE_VERSION/g" ${SITE_VERSION}/index.html # Update "Quickstart with Maven" to reflect version change
-            git add $SITE_VERSION
-            echo "<html><head><meta http-equiv=\"refresh\" content=\"0; URL='http://GoogleCloudPlatform.github.io/google-cloud-java/${SITE_VERSION}/index.html'\" /></head><body></body></html>" > index.html
-            git add index.html
-            echo "<html><head><script>window.location.replace('/google-cloud-java/${SITE_VERSION}/apidocs' + location.search)</script></head><body></body></html>" > apidocs/index.html
-            git add apidocs/index.html
-            git commit --quiet -m "Added a new site for version $SITE_VERSION and updated the root directory's redirect. [ci skip]"
-            git config --global push.default simple
-            git push --quiet "https://${CI_DEPLOY_USERNAME}:${CI_DEPLOY_PASSWORD}@github.com/GoogleCloudPlatform/google-cloud-java.git" > /dev/null 2>&1
-        else
-            mvn clean deploy --quiet -Djava.util.logging.config.file=logging.properties -DskipTests=true -Dgpg.skip=true --settings ~/.m2/settings.xml -P release
+        mvn clean deploy --quiet -Djava.util.logging.config.file=logging.properties -DskipTests=true -Dgpg.skip=true --settings ~/.m2/settings.xml -P release
         fi
     else
         echo "Not deploying artifacts. This is only done with non-pull-request commits to master branch with Oracle Java 7 builds."
diff --git a/utilities/create_site.sh b/utilities/create_site.sh
@@ -3,34 +3,31 @@
 set -e
 
 SITE_VERSION=$1
+SITE_VERSION_BASE=$(sed -r 's/([0-9]+\.[0-9]+\.[0-9]+).*/\1/' <<< "$SITE_VERSION")
 
-echo "Creating docs for google-cloud-java $SITE_VERSION"
+echo "Creating site for google-cloud-java $SITE_VERSION_BASE"
 
-if [ -z "$SITE_VERSION" ]; then
+if [ -z "$SITE_VERSION_BASE" ]; then
     echo "First arg (version) not provided, so we're exiting."
     exit 1
 fi
 
 git clone --branch gh-pages --single-branch https://github.com/GoogleCloudPlatform/google-cloud-java/ tmp_gh-pages
-mkdir -p tmp_gh-pages/$SITE_VERSION
+mkdir -p tmp_gh-pages/$SITE_VERSION_BASE
 mvn site -DskipTests=true -Djava.util.logging.config.file=logging.properties
-mvn site:stage --quiet -Djava.util.logging.config.file=logging.properties -DtopSiteURL=http://googlecloudplatform.github.io/google-cloud-java/site/${SITE_VERSION}/
+mvn site:stage --quiet -Djava.util.logging.config.file=logging.properties -DtopSiteURL=http://googlecloudplatform.github.io/google-cloud-java/site/${SITE_VERSION_BASE}/
 cd tmp_gh-pages
 
-for dir in ../target/staging/${SITE_VERSION}*
+for dir in ../target/staging/${SITE_VERSION_BASE}*
 do
-    cp -r ${dir}/* $SITE_VERSION/
+    cp -r ${dir}/* $SITE_VERSION_BASE/
 done
 
-sed -i "s/{{SITE_VERSION}}/$SITE_VERSION/g" ${SITE_VERSION}/index.html # Update "Quickstart with Maven" to reflect version change
-git add $SITE_VERSION
-echo "<html><head><meta http-equiv=\"refresh\" content=\"0; URL='http://GoogleCloudPlatform.github.io/google-cloud-java/${SITE_VERSION}/index.html'\" /></head><body></body></html>" > index.html
+sed -i "s/{{SITE_VERSION}}/$SITE_VERSION/g" ${SITE_VERSION_BASE}/index.html # Update "Quickstart with Maven" to reflect version change
+git add $SITE_VERSION_BASE
+echo "<html><head><meta http-equiv=\"refresh\" content=\"0; URL='http://GoogleCloudPlatform.github.io/google-cloud-java/${SITE_VERSION_BASE}/index.html'\" /></head><body></body></html>" > index.html
 git add index.html
-echo "<html><head><script>window.location.replace('/google-cloud-java/${SITE_VERSION}/apidocs' + location.search)</script></head><body></body></html>" > apidocs/index.html
+echo "<html><head><script>window.location.replace('/google-cloud-java/${SITE_VERSION_BASE}/apidocs' + location.search)</script></head><body></body></html>" > apidocs/index.html
 git add apidocs/index.html
 
 echo "Site generated under tmp_gh-pages/. The changes must be committed from that directory."
-echo "Recommended commands: (this will push the new site to the gh-pages branch)"
-echo "  cd tmp_gh-pages"
-echo "  git commit -m \"Added a new site for version $SITE_VERSION and updated the root directory's redirect. [ci skip]\""
-echo "  git push"
diff --git a/utilities/deploy.sh b/utilities/deploy.sh
@@ -4,7 +4,7 @@ set -e
 
 SITE_VERSION=$1
 
-if [ -n $SITE_VERSION ]; then
+if [ -z $SITE_VERSION ]; then
     echo "First arg (version) not provided, so we're exiting."
     exit 1
 fi
@@ -20,6 +20,6 @@ else
         echo "Existence was checked using the url $URL"
     else
         echo "Deploying Release artifacts..."
-        mvn clean deploy --quiet -Djava.util.logging.config.file=logging.properties -DskipITs --settings ~/.m2/settings.xml -P sign-deploy,release
+        mvn clean deploy --quiet -Djava.util.logging.config.file=logging.properties -DskipTests=true --settings ~/.m2/settings.xml -P release
     fi
 fi
diff --git a/utilities/finalize_release.sh b/utilities/finalize_release.sh
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+set -e
+
+mvn nexus-staging:release
+
+# Push the version and README updates
+git push
+
+# Push the gh-pages updates
+cd tmp_gh-pages
+git push
diff --git a/utilities/stage_release.sh b/utilities/stage_release.sh
@@ -4,7 +4,7 @@ set -e
 
 SITE_VERSION=$(mvn org.apache.maven.plugins:maven-help-plugin:2.1.1:evaluate -Dexpression=project.version | grep -Ev '(^\[|\w+:)')
 echo "Used the maven-help-plugin to determine that the version is $SITE_VERSION"
-if [ "$SITE_VERSION" == "" ]; then
+if [ -z "$SITE_VERSION" ]; then
     echo "Could not determine the version, so we're exiting."
     exit 1
 fi
@@ -19,7 +19,7 @@ if [ "${SITE_VERSION##*-}" != "SNAPSHOT" ]; then
     git commit --quiet -m "Updating version in README files. [ci skip]"
 
     # Create website
-    ./utilities/create_site.sh
+    ./utilities/create_site.sh "$SITE_VERSION"
 
     cd tmp_gh-pages
     git commit --quiet -m "Added a new site for version $SITE_VERSION and updated the root directory's redirect. [ci skip]"
