Skip to content

ci: In nightly CI, cleanup dynamo tables related to integration tests #3055

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
feast-ci-bot merged 3 commits into from
Aug 15, 2022
+47 −1
Merged

ci: In nightly CI, cleanup dynamo tables related to integration tests #3055

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b889b36
ci: In nightly CI, cleanup dynamo tables related to integration tests
adchia Aug 10, 2022
3993023
workflow
adchia Aug 10, 2022
4d858d6
add tqdm
adchia Aug 15, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 25 additions & 1 deletion .github/workflows/nightly-ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,30 @@ jobs:
name: Check if there were commits in the last day
if: ${{ github.event_name == 'schedule' }}
run: echo '::set-output name=WAS_EDITED::'$(test -n "$(git log --format=%H --since='24 hours ago')" && echo 'true' || echo 'false')
cleanup_dynamo_tables:
if: github.repository == 'feast-dev/feast'
runs-on: ubuntu-latest
name: Cleanup dynamo tables which can fail to cleanup
steps:
- uses: actions/checkout@v2
with:
ref: master
- name: Setup Python
uses: actions/setup-python@v2
id: setup-python
with:
python-version: "3.8"
architecture: x64
- name: Set up AWS SDK
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Install boto3
run: pip install boto3
- name: Run DynamoDB cleanup script
run: python infra/scripts/cleanup_dynamo_ci.py
build-docker-image:
if: github.repository == 'feast-dev/feast'
needs: [check_date]
Expand Down Expand Up @@ -82,7 +106,7 @@ jobs:
DOCKER_IMAGE_TAG: ${{ steps.image-tag.outputs.DOCKER_IMAGE_TAG }}
integration-test-python:
if: github.repository == 'feast-dev/feast'
needs: [check_date, build-docker-image]
needs: [check_date, build-docker-image, cleanup_dynamo_tables]
runs-on: ${{ matrix.os }}
strategy:
fail-fast: false
Expand Down
22 changes: 22 additions & 0 deletions infra/scripts/cleanup_dynamo_ci.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
import boto3
from tqdm import tqdm


def main() -> None:
db = boto3.resource("dynamodb")

num_to_delete = 0
all_tables = db.tables.all()
for table in all_tables:
if "integration_test" in table.name:
num_to_delete += 1
with tqdm(total=num_to_delete) as progress:
for table in all_tables:
if "integration_test" in table.name:
table.delete()
progress.update()
print(f"Deleted {num_to_delete} CI DynamoDB tables")


if __name__ == "__main__":
main()