Skip to content

Commit 5962756

Browse files
pyalexpyalex
authored
Stabilize flaky e2e tests (feast-dev#1173)
* use mvn cache in e2e & skip auth Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> * correct path & pytest param Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> * use artifacts to store ingestion jar Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> * add mvn cache Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> * download artifact to path Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> * mvn cache Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> pipeline fails Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> pipeline fails Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> pipeline fails Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> * check failing Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> * cache for master build of ingestion jar Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> * drop auth=true completely Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
1 parent 5895f37 commit 5962756

7 files changed

Lines changed: 43 additions & 15 deletions

File tree

‎.github/workflows/complete.yml‎

Lines changed: 21 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -132,32 +132,43 @@ jobs:
132132
- build-push-docker-images
133133
- publish-ingestion-jar
134134
runs-on: ubuntu-latest
135+
env:
136+
INGESTION_JAR_PATH: /shared/feast-ingestion-spark-develop.jar
135137
steps:
136138
- uses: actions/checkout@v2
139+
- name: Download ingestion jar
140+
uses: actions/download-artifact@v2
141+
with:
142+
name: ingestion-jar
143+
path: ./infra/docker-compose/
137144
- name: Test docker compose
138145
run: ./infra/scripts/test-docker-compose.sh ${GITHUB_SHA}
139146

140147
publish-ingestion-jar:
141-
runs-on: [self-hosted]
148+
runs-on: ubuntu-latest
142149
steps:
143150
- uses: actions/checkout@v2
144-
- uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
145-
with:
146-
version: '290.0.1'
147-
export_default_credentials: true
148151
- uses: actions/setup-java@v1
149152
with:
150153
java-version: '11'
151-
- uses: stCarolas/setup-maven@v3
154+
- name: Cache local Maven repository
155+
uses: actions/cache@v2
152156
with:
153-
maven-version: 3.6.3
157+
path: ~/.m2/repository
158+
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
159+
restore-keys: |
160+
${{ runner.os }}-maven-
154161
- name: build-jar
155162
env:
156163
# Try to add retries to prevent connection resets
157164
# https://github.community/t/getting-maven-could-not-transfer-artifact-with-500-error-when-using-github-actions/17570
158165
# https://github.com/actions/virtual-environments/issues/1499#issuecomment-718396233
159166
MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=25 -Dmaven.wagon.http.retryHandler.count=3 -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false
160167
MAVEN_EXTRA_OPTS: -X
161-
run: make build-java-no-tests REVISION=${GITHUB_SHA}
162-
- name: copy to gs
163-
run: gsutil cp ./spark/ingestion/target/feast-ingestion-spark-${GITHUB_SHA}.jar gs://feast-jobs/spark/ingestion/
168+
run: make build-java-no-tests REVISION=develop
169+
- name: Upload ingestion jar
170+
uses: actions/upload-artifact@v2
171+
with:
172+
name: ingestion-jar
173+
path: spark/ingestion/target/feast-ingestion-spark-develop.jar
174+
retention-days: 1

‎.github/workflows/master_only.yml‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -63,7 +63,7 @@ jobs:
6363
fi
6464
6565
publish-ingestion-jar:
66-
runs-on: [ self-hosted ]
66+
runs-on: ubuntu-latest
6767
env:
6868
PUBLISH_BUCKET: feast-jobs
6969
steps:
@@ -72,12 +72,18 @@ jobs:
7272
with:
7373
version: '290.0.1'
7474
export_default_credentials: true
75+
project_id: ${{ secrets.GCP_PROJECT_ID }}
76+
service_account_key: ${{ secrets.GCP_SA_KEY }}
7577
- uses: actions/setup-java@v1
7678
with:
7779
java-version: '11'
78-
- uses: stCarolas/setup-maven@v3
80+
- name: Cache local Maven repository
81+
uses: actions/cache@v2
7982
with:
80-
maven-version: 3.6.3
83+
path: ~/.m2/repository
84+
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
85+
restore-keys: |
86+
${{ runner.os }}-maven-
8187
- name: Publish develop version of ingestion job
8288
run: |
8389
if [ ${GITHUB_REF#refs/*/} == "master" ]; then

‎infra/docker-compose/.env.sample‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,5 @@ COMPOSE_PROJECT_NAME=feast
22
FEAST_VERSION=develop
33
FEAST_CORE_CONFIG=./core/core.yml
44
FEAST_ONLINE_SERVING_CONFIG=./serving/online-serving.yml
5-
GCP_SERVICE_ACCOUNT=./gcp-service-accounts/placeholder.json
5+
GCP_SERVICE_ACCOUNT=./gcp-service-accounts/placeholder.json
6+
INGESTION_JAR_PATH=https://storage.googleapis.com/feast-jobs/spark/ingestion/feast-ingestion-spark-develop.jar

‎infra/docker-compose/docker-compose.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ services:
3636
FEAST_HISTORICAL_FEATURE_OUTPUT_LOCATION: file:///shared/historical_feature_output
3737
FEAST_HISTORICAL_FEATURE_OUTPUT_FORMAT: parquet
3838
FEAST_REDIS_HOST: redis
39+
FEAST_SPARK_INGESTION_JAR: ${INGESTION_JAR_PATH}
3940

4041
jupyter:
4142
image: gcr.io/kf-feast/feast-jupyter:${FEAST_VERSION}

‎infra/scripts/test-end-to-end-gcp.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
11
#!/usr/bin/env bash
22

33
export DISABLE_SERVICE_FIXTURES=1
4+
export MAVEN_OPTS="-Dmaven.repo.local=/tmp/.m2/repository -DdependencyLocationsEnabled=false"
5+
export MAVEN_CACHE="gs://feast-templocation-kf-feast/.m2.2020-11-17.tar"
46

7+
infra/scripts/download-maven-cache.sh --archive-uri ${MAVEN_CACHE} --output-dir /tmp
58
apt-get update && apt-get install -y redis-server postgresql libpq-dev
69

710
make build-java-no-tests REVISION=develop

‎infra/scripts/test-end-to-end.sh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
#!/usr/bin/env bash
22

3+
export MAVEN_OPTS="-Dmaven.repo.local=/tmp/.m2/repository -DdependencyLocationsEnabled=false"
4+
export MAVEN_CACHE="gs://feast-templocation-kf-feast/.m2.2020-11-17.tar"
5+
6+
infra/scripts/download-maven-cache.sh --archive-uri ${MAVEN_CACHE} --output-dir /tmp
37
apt-get update && apt-get install -y redis-server postgresql libpq-dev
48

59
make build-java-no-tests REVISION=develop

‎tests/e2e/fixtures/feast_services.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,9 @@ def _wait_port_open(port, max_wait=60):
4747
return
4848

4949

50-
@pytest.fixture(scope="session", params=[True, False])
50+
@pytest.fixture(
51+
scope="session", params=[False],
52+
)
5153
def enable_auth(request):
5254
return request.param
5355

0 commit comments

Comments
 (0)