Skip to content

ci: add ci for packages directory #10675

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
parthea merged 7 commits into from
Oct 25, 2022
Merged

ci: add ci for packages directory #10675

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
cd37e7c
ci: add ci for packages directory
parthea Oct 22, 2022
ad3ac9d
clarify comment for set -eo pipefail
parthea Oct 25, 2022
6ac379c
add high level overview for ci/run_conditional_tests.sh
parthea Oct 25, 2022
1bf8bab
document required environment variables
parthea Oct 25, 2022
76a360a
add high level overview for ci/run_single_test.sh
parthea Oct 25, 2022
7eae7ef
add python 3.11 to ci
parthea Oct 25, 2022
bf0902d
remove python 3.11 not yet supported in github actions
parthea Oct 25, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions .github/workflows/docs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
on:
pull_request:
branches:
- main
name: docs

permissions:
contents: read

jobs:
docs:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.9"
- name: Install nox
run: |
python -m pip install --upgrade setuptools pip wheel
python -m pip install nox
- name: Run docs
env:
BUILD_TYPE: presubmit
TEST_TYPE: docs
PY_VERSION: "3.9"
run: |
ci/run_conditional_tests.sh
docfx:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.9"
- name: Install nox
run: |
python -m pip install --upgrade setuptools pip wheel
python -m pip install nox
- name: Run docfx
env:
BUILD_TYPE: presubmit
TEST_TYPE: docfx
PY_VERSION: "3.9"
run: |
ci/run_conditional_tests.sh
37 changes: 37 additions & 0 deletions .github/workflows/lint.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
on:
pull_request:
branches:
- main
name: lint

permissions:
contents: read

jobs:
lint:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.9"
- name: Install nox
run: |
python -m pip install --upgrade setuptools pip wheel
python -m pip install nox
- name: Run lint
env:
BUILD_TYPE: presubmit
TEST_TYPE: lint
PY_VERSION: "3.9"
run: |
ci/run_conditional_tests.sh
- name: Run lint_setup_py
env:
BUILD_TYPE: presubmit
TEST_TYPE: lint_setup_py
PY_VERSION: "3.9"
run: |
ci/run_conditional_tests.sh
93 changes: 93 additions & 0 deletions .github/workflows/unittest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
on:
pull_request:
branches:
- main
name: unittest

permissions:
contents: read

jobs:
unit:
runs-on: ubuntu-latest
strategy:
matrix:
python: ['3.7', '3.8', '3.9', '3.10']
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python }}
- name: Install nox
run: |
python -m pip install --upgrade setuptools pip wheel
python -m pip install nox
- name: Run unit tests
env:
COVERAGE_FILE: .coverage-${{ matrix.python }}
BUILD_TYPE: presubmit
TEST_TYPE: unit
PY_VERSION: ${{ matrix.python }}
run: |
ci/run_conditional_tests.sh
- name: Upload coverage results
uses: actions/upload-artifact@v3
with:
name: coverage-artifacts
path: .coverage-${{ matrix.python }}
prerelease:
runs-on: ubuntu-latest
strategy:
matrix:
python: ['3.10']
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python }}
- name: Install nox
run: |
python -m pip install --upgrade setuptools pip wheel
python -m pip install nox
- name: Run prerelease tests
env:
BUILD_TYPE: presubmit
TEST_TYPE: prerelease
PY_VERSION: ${{ matrix.python }}
run: |
ci/run_conditional_tests.sh

cover:
runs-on: ubuntu-latest
needs:
- unit
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.10"
- name: Set number of files changes in packages directory
id: packages
run: echo "::set-output name=num_files_changed::$(git diff HEAD~1 -- packages/*.yml | wc -l)"
- name: Install coverage
if: ${{ steps.date.packages.num_files_changed > 0 }}
run: |
python -m pip install --upgrade setuptools pip wheel
python -m pip install coverage
- name: Download coverage results
if: ${{ steps.date.packages.num_files_changed > 0 }}
uses: actions/download-artifact@v3
with:
name: coverage-artifacts
path: .coverage-results/
- name: Report coverage results
if: ${{ steps.date.packages.num_files_changed > 0 }}
run: |
coverage combine .coverage-results/.coverage*
coverage report --show-missing --fail-under=100
118 changes: 118 additions & 0 deletions ci/run_conditional_tests.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,118 @@
#!/bin/bash
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

Comment thread
parthea marked this conversation as resolved.
# This script requires the following environment variables to be set:
# `BUILD_TYPE` should be one of ["presubmit", "continuous"]
# `TEST_TYPE` should be one of ["lint", "lint_setup_py", "docs", "docfx", "prerelease"]
# `PY_VERSION` should be one of ["3.7", "3.8", "3.9", "3.10", "3.11"]

# `TEST_TYPE` and `PY_VERSION` are required by the script `ci/run_single_test.sh`

# This script will determine which directories have changed
# under the `packages` folder. For `BUILD_TYPE=="presubmit"`,
# we'll compare against the `packages` folder in HEAD,
# whereas for `BUILD_TYPE=="continuous"` we'll compare changes
# with HEAD~1. For all directories that have changed files, we will
# run the script located at `${PROJECT_ROOT}/ci/run_single_test.sh`.

# `-e` enables the script to automatically fail when a command fails
# `-o pipefail` sets the exit code to non-zero if any command fails,
# or zero if all commands in the pipeline exit successfully.
set -eo pipefail

export PROJECT_ROOT=$(realpath $(dirname "${BASH_SOURCE[0]}")/..)

# A script file for running the test in a sub project.
test_script="${PROJECT_ROOT}/ci/run_single_test.sh"
Comment thread
parthea marked this conversation as resolved.

if [ ${BUILD_TYPE} == "presubmit" ]; then
# For presubmit build, we want to know the difference from the
# common commit in origin/main.
GIT_DIFF_ARG="origin/main..."

# Then fetch enough history for finding the common commit.
git fetch origin main --deepen=200

elif [ ${BUILD_TYPE} == "continuous" ]; then
# For continuous build, we want to know the difference in the last
# commit. This assumes we use squash commit when merging PRs.
GIT_DIFF_ARG="HEAD~.."

# Then fetch one last commit for getting the diff.
git fetch origin main --deepen=1

else
# Run everything.
GIT_DIFF_ARG=""
fi

# Then detect changes in the test scripts.

set +e
git diff --quiet ${GIT_DIFF_ARG} ci
changed=$?
set -e
if [[ "${changed}" -eq 0 ]]; then
echo "no change detected in ci"
else
echo "change detected in ci, we should test everything"
GIT_DIFF_ARG=""
fi

# Now we have a fixed list, but we can change it to autodetect if
# necessary.

subdirs=(
packages
)

RETVAL=0

for subdir in ${subdirs[@]}; do
for d in `ls -d ${subdir}/*/`; do
should_test=false
if [ -n "${GIT_DIFF_ARG}" ]; then
echo "checking changes with 'git diff --quiet ${GIT_DIFF_ARG} ${d}'"
set +e
git diff --quiet ${GIT_DIFF_ARG} ${d}
changed=$?
set -e
if [[ "${changed}" -eq 0 ]]; then
echo "no change detected in ${d}, skipping"
else
echo "change detected in ${d}"
should_test=true
fi
else
# If GIT_DIFF_ARG is empty, run all the tests.
should_test=true
fi
if [ "${should_test}" = true ]; then
echo "running test in ${d}"
pushd ${d}
# Temporarily allow failure.
set +e
${test_script}
ret=$?
set -e
if [ ${ret} -ne 0 ]; then
RETVAL=${ret}
fi
popd
fi
done
done

exit ${RETVAL}
90 changes: 90 additions & 0 deletions ci/run_single_test.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
#!/bin/bash
#
# Copyright 2022 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# This script requires the following environment variables to be set:
# `TEST_TYPE` should be one of ["lint", "lint_setup_py", "docs", "docfx", "prerelease"]
# `PY_VERSION` should be one of ["3.7", "3.8", "3.9", "3.10", "3.11"]

# This script is called by the `ci/run_conditional_tests.sh` script.
# A specific `nox` session will be run, depending on the value of
# `TEST_TYPE` and `PY_VERSION`. For example, if `TEST_TYPE` is
# `lint`, the `nox -s lint` session will be run.


set -e

Comment thread
parthea marked this conversation as resolved.
if [ -z "${TEST_TYPE}" ]; then
echo "missing TEST_TYPE env var"
exit 1
fi

if [ -z "${PY_VERSION}" ]; then
echo "missing PY_VERSION env var"
exit 1
fi

# Don't fail on errors so we can capture all of the output
set +e

case ${TEST_TYPE} in
lint)
nox -s lint
retval=$?
;;
lint_setup_py)
nox -s lint_setup_py
retval=$?
;;
docs)
nox -s docs
retval=$?
;;
docfx)
nox -s docfx
retval=$?
;;
prerelease)
nox -s prerelease_deps-3.10
retval=$?
;;
unit)
case ${PY_VERSION} in
"3.7")
nox -s unit-3.7
retval=$?
;;
"3.8")
nox -s unit-3.8
retval=$?
;;
"3.9")
nox -s unit-3.9
retval=$?
;;
"3.10")
nox -s unit-3.10
retval=$?
;;
"3.11")
nox -s unit-3.11
retval=$?
;;
*)
;;
esac
esac

exit ${retval}