Skip to content

Add readiness checks for Feast services in end to end test #337

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
woop merged 12 commits into from
Dec 22, 2019
Merged

Add readiness checks for Feast services in end to end test #337

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
a73e927
Add readiness checks for Kafka, Feast Core and Feast Serving in test-…
davidheryanto Nov 28, 2019
4594382
Use kafkacat to check Kafka connection
davidheryanto Nov 28, 2019
81a9077
Reduce timeout duration for job to 1h
davidheryanto Nov 28, 2019
009c21b
Merge branch 'master' into test-end-to-end-readiness-check
davidheryanto Nov 30, 2019
02f48b0
Increase resource request for tests
davidheryanto Nov 30, 2019
c2ac80a
Merge branch 'master' into test-end-to-end-readiness-check
davidheryanto Dec 20, 2019
678ab9f
Merge branch 'test-end-to-end-readiness-check' of github.com:davidher…
davidheryanto Dec 20, 2019
28f049b
Increase wait time for Kafka to start
davidheryanto Dec 20, 2019
4688a72
Increase default Kafka request timeout value to 15s from 1s
davidheryanto Dec 20, 2019
f9f3bd1
Increase cpu request and set timeout for netcat
davidheryanto Dec 20, 2019
9edbcc5
Use revision=dev for end-to-end-test
davidheryanto Dec 20, 2019
bb2ccdb
Correct the target jar filename
davidheryanto Dec 20, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions .prow/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,8 @@ presubmits:
command: [".prow/scripts/test-core-ingestion.sh"]
resources:
requests:
cpu: "1000m"
memory: "512Mi"
cpu: "1500m"
memory: "1536Mi"
limit:
memory: "4096Mi"

Expand Down Expand Up @@ -116,10 +116,10 @@ presubmits:
command: [".prow/scripts/test-end-to-end.sh"]
resources:
requests:
cpu: "1000m"
memory: "1024Mi"
limit:
cpu: "3000m"
memory: "4096Mi"
limit:
memory: "6144Mi"

- name: test-end-to-end-batch
decorate: true
Expand Down
4 changes: 2 additions & 2 deletions .prow/scripts/test-end-to-end-batch.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,10 +73,10 @@ Installing Kafka at localhost:9092
wget -qO- https://www-eu.apache.org/dist/kafka/2.3.0/kafka_2.12-2.3.0.tgz | tar xz
mv kafka_2.12-2.3.0/ /tmp/kafka
nohup /tmp/kafka/bin/zookeeper-server-start.sh /tmp/kafka/config/zookeeper.properties &> /var/log/zookeeper.log 2>&1 &
sleep 5
sleep 10
tail -n10 /var/log/zookeeper.log
nohup /tmp/kafka/bin/kafka-server-start.sh /tmp/kafka/config/server.properties &> /var/log/kafka.log 2>&1 &
sleep 20
sleep 30
tail -n10 /var/log/kafka.log

echo "
Expand Down
21 changes: 16 additions & 5 deletions .prow/scripts/test-end-to-end.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@
set -e
set -o pipefail

export REVISION=dev

if ! cat /etc/*release | grep -q stretch; then
echo ${BASH_SOURCE} only supports Debian stretch.
echo Please change your operating system to use this script.
Expand All @@ -19,15 +21,18 @@ This script will run end-to-end tests for Feast Core and Online Serving.
tests/e2e via pytest.
"

apt-get -qq update
apt-get -y install wget netcat kafkacat

echo "
============================================================
Installing Redis at localhost:6379
============================================================
"
apt-get -qq update

# Allow starting serving in this Maven Docker image. Default set to not allowed.
echo "exit 0" > /usr/sbin/policy-rc.d
apt-get -y install redis-server wget > /var/log/redis.install.log
apt-get -y install redis-server > /var/log/redis.install.log
redis-server --daemonize yes
redis-cli ping

Expand Down Expand Up @@ -61,6 +66,7 @@ tail -n10 /var/log/zookeeper.log
nohup /tmp/kafka/bin/kafka-server-start.sh /tmp/kafka/config/server.properties &> /var/log/kafka.log 2>&1 &
sleep 20
tail -n10 /var/log/kafka.log
kafkacat -b localhost:9092 -L

echo "
============================================================
Expand All @@ -73,7 +79,10 @@ Building jars for Feast
--output-dir /root/

# Build jars for Feast
mvn --quiet --batch-mode --define skipTests=true clean package
mvn --quiet --batch-mode --define skipTests=true --define revision=$REVISION clean package

ls -lh core/target/*jar
ls -lh serving/target/*jar

echo "
============================================================
Expand Down Expand Up @@ -123,11 +132,12 @@ management:
enabled: false
EOF

nohup java -jar core/target/feast-core-0.3.2-SNAPSHOT.jar \
nohup java -jar core/target/feast-core-$REVISION.jar \
--spring.config.location=file:///tmp/core.application.yml \
&> /var/log/feast-core.log &
sleep 35
tail -n10 /var/log/feast-core.log
nc -w2 localhost 6565 < /dev/null

echo "
============================================================
Expand Down Expand Up @@ -174,11 +184,12 @@ spring:
web-environment: false
EOF

nohup java -jar serving/target/feast-serving-0.3.2-SNAPSHOT.jar \
nohup java -jar serving/target/feast-serving-$REVISION.jar \
--spring.config.location=file:///tmp/serving.online.application.yml \
&> /var/log/feast-serving-online.log &
sleep 15
tail -n10 /var/log/feast-serving-online.log
nc -w2 localhost 6566 < /dev/null

echo "
============================================================
Expand Down
5 changes: 4 additions & 1 deletion core/src/main/java/feast/core/config/FeatureStreamConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@
@Configuration
public class FeatureStreamConfig {

String DEFAULT_KAFKA_REQUEST_TIMEOUT_MS_CONFIG = "15000";

@Autowired
@Bean
public Source getDefaultSource(FeastProperties feastProperties) {
Expand All @@ -50,7 +52,8 @@ public Source getDefaultSource(FeastProperties feastProperties) {
String topicName = streamProperties.getOptions().get("topic");
Map<String, Object> map = new HashMap<>();
map.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
map.put(AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG, "1000");
map.put(AdminClientConfig.REQUEST_TIMEOUT_MS_CONFIG,
DEFAULT_KAFKA_REQUEST_TIMEOUT_MS_CONFIG);
AdminClient client = AdminClient.create(map);

NewTopic newTopic =
Expand Down