Skip to content

Commit 41851be

Browse files
authored
ci: In nightly CI, cleanup dynamo tables related to integration tests (feast-dev#3055)
* ci: In nightly CI, cleanup dynamo tables related to integration tests Signed-off-by: Danny Chiao <danny@tecton.ai> * workflow Signed-off-by: Danny Chiao <danny@tecton.ai> * add tqdm Signed-off-by: Danny Chiao <danny@tecton.ai> Signed-off-by: Danny Chiao <danny@tecton.ai>
1 parent c7ba370 commit 41851be

File tree

2 files changed

+47
-1
lines changed

2 files changed

+47
-1
lines changed

.github/workflows/nightly-ci.yml

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,30 @@ jobs:
2424
name: Check if there were commits in the last day
2525
if: ${{ github.event_name == 'schedule' }}
2626
run: echo '::set-output name=WAS_EDITED::'$(test -n "$(git log --format=%H --since='24 hours ago')" && echo 'true' || echo 'false')
27+
cleanup_dynamo_tables:
28+
if: github.repository == 'feast-dev/feast'
29+
runs-on: ubuntu-latest
30+
name: Cleanup dynamo tables which can fail to cleanup
31+
steps:
32+
- uses: actions/checkout@v2
33+
with:
34+
ref: master
35+
- name: Setup Python
36+
uses: actions/setup-python@v2
37+
id: setup-python
38+
with:
39+
python-version: "3.8"
40+
architecture: x64
41+
- name: Set up AWS SDK
42+
uses: aws-actions/configure-aws-credentials@v1
43+
with:
44+
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
45+
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
46+
aws-region: us-west-2
47+
- name: Install boto3
48+
run: pip install boto3
49+
- name: Run DynamoDB cleanup script
50+
run: python infra/scripts/cleanup_dynamo_ci.py
2751
build-docker-image:
2852
if: github.repository == 'feast-dev/feast'
2953
needs: [check_date]
@@ -82,7 +106,7 @@ jobs:
82106
DOCKER_IMAGE_TAG: ${{ steps.image-tag.outputs.DOCKER_IMAGE_TAG }}
83107
integration-test-python:
84108
if: github.repository == 'feast-dev/feast'
85-
needs: [check_date, build-docker-image]
109+
needs: [check_date, build-docker-image, cleanup_dynamo_tables]
86110
runs-on: ${{ matrix.os }}
87111
strategy:
88112
fail-fast: false

infra/scripts/cleanup_dynamo_ci.py

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
import boto3
2+
from tqdm import tqdm
3+
4+
5+
def main() -> None:
6+
db = boto3.resource("dynamodb")
7+
8+
num_to_delete = 0
9+
all_tables = db.tables.all()
10+
for table in all_tables:
11+
if "integration_test" in table.name:
12+
num_to_delete += 1
13+
with tqdm(total=num_to_delete) as progress:
14+
for table in all_tables:
15+
if "integration_test" in table.name:
16+
table.delete()
17+
progress.update()
18+
print(f"Deleted {num_to_delete} CI DynamoDB tables")
19+
20+
21+
if __name__ == "__main__":
22+
main()

0 commit comments

Comments
 (0)