Skip to content

Commit 4d1b4f9

Browse files
khorshuhengfeast-ci-bot
khorshuheng
authored and
feast-ci-bot
committed
Add docker-compose file with Jupyer notebook (#328)
* Add docker-compose file with Jupyer notebook * Add development version of dockerfile for Feast developer, to avoid mvn dependencies download * Add end to end test runner * Add documentation * Add batch serving functionality to docker compose setup
1 parent 7aae72b commit 4d1b4f9

File tree

19 files changed

+1348
-1
lines changed

19 files changed

+1348
-1
lines changed

‎.dockerignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
docs
2+
charts

‎docs/getting-started/installing-feast.md‎

Lines changed: 40 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,11 +2,50 @@
22

33
## Overview
44

5-
This installation guide will demonstrate two ways of installing Feast:
5+
This installation guide will demonstrate three ways of installing Feast:
66

7+
* [**Docker Compose \(Quickstart\)**](install-feast.md#docker-compose)**:** Allow quick testing on Feast via a sample Jupyter notebook pre-installed with Feast Python SDK.
78
* [**Minikube \(Minimal\)**](installing-feast.md#minikube)**:** This installation has no external dependencies, but does not have a historical feature store installed. It allows users to quickly get a feel for Feast.
89
* [**Google Kubernetes Engine \(Recommended\):**](installing-feast.md#google-kubernetes-engine) This guide installs a single cluster Feast installation on Google's GKE. It has Google Cloud specific dependencies like BigQuery, Dataflow, and Google Cloud Storage.
910

11+
12+
## Docker Compose
13+
14+
### Overview
15+
16+
A docker compose file is provided to quickly test Feast with official docker images. GCP dependency is optional.
17+
18+
* Define and register feature set
19+
* Feature ingestion
20+
* Retrieve features for online serving
21+
* Updating the feature set
22+
23+
The docker compose setup uses Direct Runner for the Apache Beam jobs.
24+
25+
### 0. Requirements
26+
27+
1. [Docker compose](https://docs.docker.com/compose/install/) should be installed.
28+
2. Port 6565, 6566 and 8888, 9094 are not in used. Otherwise, modify the port mappings in `infra/docker-compose/docker-compose.yml` to use unoccupied ports.
29+
3. For batch serving, you will also need a service account key that has access to GCS and BigQuery. Port 6567 will be used for the batch serving endpoint.
30+
31+
### 1. Step-by-step guide (Online serving)
32+
1. Navigate to `infra/docker-compose`.
33+
2. Copy `.env.sample` to `.env`.
34+
3. `docker-compose up -d`
35+
4. A jupyter notebook server should be accessible via `localhost:8888`
36+
5. Please wait a minute or two for the Feast services to be ready before running the notebook. You will know that the services are ready when port `6565` and `6566` starts listening.
37+
38+
### 2. Step-by-step guide (Batch serving)
39+
1. Navigate to `infra/docker-compose`.
40+
2. Copy `.env.sample` to `.env`.
41+
3. Copy your GCP account service key(s) to `infra/docker-compose/gcp-service-accounts`.
42+
4. Modify the value of `FEAST_<SERVICE_NAME>_GCP_SERVICE_ACCOUNT_KEY` in your `.env` file. It should be the json file name without extension.
43+
5. Modify the value of `infra/docker-compose/serving/bq-store.yml`. Alternatively, you can also point to a different store configuration file by modifying `FEAST_BATCH_STORE_CONFIG` in your `.env` file.
44+
5. `docker-compose -f docker-compose.yml -f docker-compose.batch.yml up -d`
45+
6. A jupyter notebook server should be accessible via `localhost:8888`
46+
7. Please wait a minute or two for the Feast services to be ready before running the notebook. You will know that the services are ready when port `6565` and `6567` starts listening.
47+
8. When you are done, run `docker-compose -f docker-compose.yml -f docker-compose.batch.yml down` to shutdown the services.
48+
1049
## Minikube
1150

1251
### Overview

‎infra/docker-compose/.env.sample‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
COMPOSE_PROJECT_NAME=feast
2+
3+
FEAST_VERSION=latest
4+
5+
FEAST_CORE_IMAGE=gcr.io/kf-feast/feast-core
6+
FEAST_CORE_CONFIG=direct-runner
7+
FEAST_CORE_GCP_SERVICE_ACCOUNT_KEY=placeholder
8+
9+
FEAST_SERVING_IMAGE=gcr.io/kf-feast/feast-serving
10+
FEAST_ONLINE_SERVING_CONFIG=online-serving
11+
FEAST_ONLINE_STORE_CONFIG=redis-store
12+
FEAST_BATCH_SERVING_CONFIG=batch-serving
13+
FEAST_BATCH_STORE_CONFIG=bq-store
14+
FEAST_BATCH_SERVING_GCP_SERVICE_ACCOUNT_KEY=placeholder
15+
FEAST_JOB_STAGING_LOCATION=gs://your-gcp-project/bucket
16+
17+
FEAST_JUPYTER_IMAGE=gcr.io/kf-feast/feast-jupyter
18+
FEAST_JUPYTER_GCP_SERVICE_ACCOUNT_KEY=placeholder
19+

‎infra/docker-compose/core/direct-runner.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
feast:
2+
jobs:
3+
runner: DirectRunner

‎infra/docker-compose/docker-compose.batch.yml‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
version: "3.7"
2+
3+
services:
4+
batch-serving:
5+
image: ${FEAST_SERVING_IMAGE}:${FEAST_VERSION}
6+
volumes:
7+
- ./serving/${FEAST_BATCH_SERVING_CONFIG}.yml:/etc/feast/application.yml
8+
- ./serving/${FEAST_BATCH_STORE_CONFIG}.yml:/etc/feast/store.yml
9+
- ./gcp-service-accounts/${FEAST_BATCH_SERVING_GCP_SERVICE_ACCOUNT_KEY}.json:/etc/gcloud/service-accounts/key.json
10+
depends_on:
11+
- core
12+
- redis
13+
ports:
14+
- 6567:6567
15+
restart: on-failure
16+
environment:
17+
GOOGLE_APPLICATION_CREDENTIALS: /etc/gcloud/service-accounts/key.json
18+
FEAST_JOB_STAGING_LOCATION: ${FEAST_JOB_STAGING_LOCATION}
19+
command:
20+
- "java"
21+
- "-Xms1024m"
22+
- "-Xmx1024m"
23+
- "-jar"
24+
- "/opt/feast/feast-serving.jar"
25+
- "--spring.config.location=classpath:/application.yml,file:/etc/feast/application.yml"

‎infra/docker-compose/docker-compose.yml‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
version: "3.7"
2+
3+
services:
4+
core:
5+
image: ${FEAST_CORE_IMAGE}:${FEAST_VERSION}
6+
volumes:
7+
- ./core/${FEAST_CORE_CONFIG}.yml:/etc/feast/application.yml
8+
- ./gcp-service-accounts/${FEAST_CORE_GCP_SERVICE_ACCOUNT_KEY}.json:/etc/gcloud/service-accounts/key.json
9+
environment:
10+
DB_HOST: db
11+
GOOGLE_APPLICATION_CREDENTIALS: /etc/gcloud/service-accounts/key.json
12+
restart: on-failure
13+
depends_on:
14+
- db
15+
- kafka
16+
ports:
17+
- 6565:6565
18+
command:
19+
- java
20+
- -jar
21+
- /opt/feast/feast-core.jar
22+
- --spring.config.location=classpath:/application.yml,file:/etc/feast/application.yaml
23+
24+
online-serving:
25+
image: ${FEAST_SERVING_IMAGE}:${FEAST_VERSION}
26+
volumes:
27+
- ./serving/${FEAST_ONLINE_SERVING_CONFIG}.yml:/etc/feast/application.yml
28+
- ./serving/${FEAST_ONLINE_STORE_CONFIG}.yml:/etc/feast/store.yml
29+
depends_on:
30+
- core
31+
- redis
32+
ports:
33+
- 6566:6566
34+
restart: on-failure
35+
command:
36+
- java
37+
- -jar
38+
- /opt/feast/feast-serving.jar
39+
- --spring.config.location=classpath:/application.yml,file:/etc/feast/application.yml
40+
41+
jupyter:
42+
image: ${FEAST_JUPYTER_IMAGE}:${FEAST_VERSION}
43+
volumes:
44+
- ./jupyter/notebooks:/home/jovyan/feast-notebooks
45+
- ./jupyter/features:/home/jovyan/features
46+
- ./gcp-service-accounts/${FEAST_JUPYTER_GCP_SERVICE_ACCOUNT_KEY}.json:/etc/gcloud/service-accounts/key.json
47+
depends_on:
48+
- core
49+
- online-serving
50+
environment:
51+
FEAST_CORE_URL: core:6565
52+
FEAST_SERVING_URL: online-serving:6566
53+
GOOGLE_APPLICATION_CREDENTIALS: /etc/gcloud/service-accounts/key.json
54+
ports:
55+
- 8888:8888
56+
command:
57+
- start-notebook.sh
58+
- --NotebookApp.token=''
59+
60+
redis:
61+
image: redis:5-alpine
62+
63+
kafka:
64+
image: confluentinc/cp-kafka:5.2.1
65+
environment:
66+
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
67+
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
68+
KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://localhost:9094
69+
KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094
70+
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
71+
KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
72+
ports:
73+
- 9094:9092
74+
75+
depends_on:
76+
- zookeeper
77+
78+
zookeeper:
79+
image: confluentinc/cp-zookeeper:5.2.1
80+
environment:
81+
ZOOKEEPER_CLIENT_PORT: 2181
82+
83+
db:
84+
image: postgres:12-alpine

‎infra/docker-compose/gcp-service-accounts/placeholder.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{}

‎infra/docker-compose/jupyter/features/cust_trans_fs.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
name: customer_transactions
2+
kind: feature_set
3+
entities:
4+
- name: customer_id
5+
valueType: INT64
6+
features:
7+
- name: daily_transactions
8+
valueType: FLOAT
9+
- name: total_transactions
10+
valueType: FLOAT
11+
maxAge: 3600s

‎infra/docker-compose/jupyter/features/cust_trans_fs_updated.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
name: customer_transactions
2+
kind: feature_set
3+
entities:
4+
- name: customer_id
5+
valueType: INT64
6+
features:
7+
- name: daily_transactions
8+
valueType: FLOAT
9+
- name: total_transactions
10+
valueType: FLOAT
11+
- name: discounts
12+
valueType: FLOAT
13+
maxAge: 3600s

0 commit comments

Comments
 (0)