Skip to content

Commit 5eecec3

Browse files
pyalexpyalex
committed
Stabilize flaky e2e tests (#1173)
* use mvn cache in e2e & skip auth Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> * correct path & pytest param Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> * use artifacts to store ingestion jar Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> * add mvn cache Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> * download artifact to path Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> * mvn cache Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> pipeline fails Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> pipeline fails Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> pipeline fails Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> * check failing Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> * cache for master build of ingestion jar Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com> * drop auth=true completely Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
1 parent f146179 commit 5eecec3

7 files changed

Lines changed: 43 additions & 15 deletions

File tree

‎.github/workflows/complete.yml‎

Lines changed: 21 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -132,32 +132,43 @@ jobs:
132132
- build-push-docker-images
133133
- publish-ingestion-jar
134134
runs-on: ubuntu-latest
135+
env:
136+
INGESTION_JAR_PATH: /shared/feast-ingestion-spark-develop.jar
135137
steps:
136138
- uses: actions/checkout@v2
139+
- name: Download ingestion jar
140+
uses: actions/download-artifact@v2
141+
with:
142+
name: ingestion-jar
143+
path: ./infra/docker-compose/
137144
- name: Test docker compose
138145
run: ./infra/scripts/test-docker-compose.sh ${GITHUB_SHA}
139146

140147
publish-ingestion-jar:
141-
runs-on: [self-hosted]
148+
runs-on: ubuntu-latest
142149
steps:
143150
- uses: actions/checkout@v2
144-
- uses: GoogleCloudPlatform/github-actions/setup-gcloud@master
145-
with:
146-
version: '290.0.1'
147-
export_default_credentials: true
148151
- uses: actions/setup-java@v1
149152
with:
150153
java-version: '11'
151-
- uses: stCarolas/setup-maven@v3
154+
- name: Cache local Maven repository
155+
uses: actions/cache@v2
152156
with:
153-
maven-version: 3.6.3
157+
path: ~/.m2/repository
158+
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
159+
restore-keys: |
160+
${{ runner.os }}-maven-
154161
- name: build-jar
155162
env:
156163
# Try to add retries to prevent connection resets
157164
# https://github.community/t/getting-maven-could-not-transfer-artifact-with-500-error-when-using-github-actions/17570
158165
# https://github.com/actions/virtual-environments/issues/1499#issuecomment-718396233
159166
MAVEN_OPTS: -Dmaven.wagon.httpconnectionManager.ttlSeconds=25 -Dmaven.wagon.http.retryHandler.count=3 -Dhttp.keepAlive=false -Dmaven.wagon.http.pool=false
160167
MAVEN_EXTRA_OPTS: -X
161-
run: make build-java-no-tests REVISION=${GITHUB_SHA}
162-
- name: copy to gs
163-
run: gsutil cp ./spark/ingestion/target/feast-ingestion-spark-${GITHUB_SHA}.jar gs://feast-jobs/spark/ingestion/
168+
run: make build-java-no-tests REVISION=develop
169+
- name: Upload ingestion jar
170+
uses: actions/upload-artifact@v2
171+
with:
172+
name: ingestion-jar
173+
path: spark/ingestion/target/feast-ingestion-spark-develop.jar
174+
retention-days: 1

‎.github/workflows/master_only.yml‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ jobs:
4141
fi
4242
4343
publish-ingestion-jar:
44-
runs-on: [ self-hosted ]
44+
runs-on: ubuntu-latest
4545
env:
4646
PUBLISH_BUCKET: feast-jobs
4747
steps:
@@ -50,12 +50,18 @@ jobs:
5050
with:
5151
version: '290.0.1'
5252
export_default_credentials: true
53+
project_id: ${{ secrets.GCP_PROJECT_ID }}
54+
service_account_key: ${{ secrets.GCP_SA_KEY }}
5355
- uses: actions/setup-java@v1
5456
with:
5557
java-version: '11'
56-
- uses: stCarolas/setup-maven@v3
58+
- name: Cache local Maven repository
59+
uses: actions/cache@v2
5760
with:
58-
maven-version: 3.6.3
61+
path: ~/.m2/repository
62+
key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }}
63+
restore-keys: |
64+
${{ runner.os }}-maven-
5965
- name: Publish develop version of ingestion job
6066
run: |
6167
if [ ${GITHUB_REF#refs/*/} == "master" ]; then

‎infra/docker-compose/.env.sample‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,5 @@ COMPOSE_PROJECT_NAME=feast
22
FEAST_VERSION=develop
33
FEAST_CORE_CONFIG=./core/core.yml
44
FEAST_ONLINE_SERVING_CONFIG=./serving/online-serving.yml
5-
GCP_SERVICE_ACCOUNT=./gcp-service-accounts/placeholder.json
5+
GCP_SERVICE_ACCOUNT=./gcp-service-accounts/placeholder.json
6+
INGESTION_JAR_PATH=https://storage.googleapis.com/feast-jobs/spark/ingestion/feast-ingestion-spark-develop.jar

‎infra/docker-compose/docker-compose.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,7 @@ services:
3636
FEAST_HISTORICAL_FEATURE_OUTPUT_LOCATION: file:///shared/historical_feature_output
3737
FEAST_HISTORICAL_FEATURE_OUTPUT_FORMAT: parquet
3838
FEAST_REDIS_HOST: redis
39+
FEAST_SPARK_INGESTION_JAR: ${INGESTION_JAR_PATH}
3940

4041
jupyter:
4142
image: gcr.io/kf-feast/feast-jupyter:${FEAST_VERSION}

‎infra/scripts/test-end-to-end-gcp.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
11
#!/usr/bin/env bash
22

33
export DISABLE_SERVICE_FIXTURES=1
4+
export MAVEN_OPTS="-Dmaven.repo.local=/tmp/.m2/repository -DdependencyLocationsEnabled=false"
5+
export MAVEN_CACHE="gs://feast-templocation-kf-feast/.m2.2020-11-17.tar"
46

7+
infra/scripts/download-maven-cache.sh --archive-uri ${MAVEN_CACHE} --output-dir /tmp
58
apt-get update && apt-get install -y redis-server postgresql libpq-dev
69

710
make build-java-no-tests REVISION=develop

‎infra/scripts/test-end-to-end.sh‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
#!/usr/bin/env bash
22

3+
export MAVEN_OPTS="-Dmaven.repo.local=/tmp/.m2/repository -DdependencyLocationsEnabled=false"
4+
export MAVEN_CACHE="gs://feast-templocation-kf-feast/.m2.2020-11-17.tar"
5+
6+
infra/scripts/download-maven-cache.sh --archive-uri ${MAVEN_CACHE} --output-dir /tmp
37
apt-get update && apt-get install -y redis-server postgresql libpq-dev
48

59
make build-java-no-tests REVISION=develop

‎tests/e2e/fixtures/feast_services.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,9 @@ def _wait_port_open(port, max_wait=60):
4747
return
4848

4949

50-
@pytest.fixture(scope="session", params=[True, False])
50+
@pytest.fixture(
51+
scope="session", params=[False],
52+
)
5153
def enable_auth(request):
5254
return request.param
5355

0 commit comments

Comments
 (0)