From 58527de5ba848634a997c6517249de19060833af Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Tue, 7 Jan 2020 13:55:04 +0800 Subject: [PATCH 001/176] Remove CONTRIBUTING.md (#411) --- CONTRIBUTING.md | 334 ------------------------------------------------ README.md | 11 +- 2 files changed, 6 insertions(+), 339 deletions(-) delete mode 100644 CONTRIBUTING.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md deleted file mode 100644 index eb38db30080..00000000000 --- a/CONTRIBUTING.md +++ /dev/null @@ -1,334 +0,0 @@ -# Contributing Guide - -## Getting Started - -The following guide will help you quickly run Feast in your local machine. - -The main components of Feast are: -- **Feast Core** handles FeatureSpec registration, starts and monitors Ingestion - jobs and ensures that Feast internal metadata is consistent. -- **Feast Ingestion** subscribes to streams of FeatureRow and writes the feature - values to registered Stores. -- **Feast Serving** handles requests for features values retrieval from the end users. - -![Feast Components Overview](docs/assets/feast-components-overview.png) - -**Pre-requisites** -- Java SDK version 8 -- Python version 3.6 (or above) and pip -- Access to Postgres database (version 11 and above) -- Access to [Redis](https://redis.io/topics/quickstart) instance (tested on version 5.x) -- Access to [Kafka](https://kafka.apache.org/) brokers (tested on version 2.x) -- [Maven ](https://maven.apache.org/install.html) version 3.6.x -- [grpc_cli](https://github.com/grpc/grpc/blob/master/doc/command_line_tool.md) - is useful for debugging and quick testing -- An overview of Feast specifications and [protos](./protos/feast) - -> **Assumptions:** -> -> 1. Postgres is running in "localhost:5432" and has a database called "postgres" which -> can be accessed with credentials user "postgres" and password "password". -> To use different database name and credentials, please update -> "$FEAST_HOME/core/src/main/resources/application.yml" -> or set these environment variables: DB_HOST, DB_USERNAME, DB_PASSWORD. -> 2. Redis is running locally and accessible from "localhost:6379" -> 3. Feast has admin access to BigQuery. - - -``` -# Clone Feast branch 0.3-dev -# $FEAST_HOME will refer to be the root directory of this Feast Git repository - -git clone -b 0.3-dev https://github.com/gojek/feast -cd feast -``` - -#### Starting Feast Core - -``` -# Please check the default configuration for Feast Core in -# "$FEAST_HOME/core/src/main/resources/application.yml" and update it accordingly. -# -# Start Feast Core GRPC server on localhost:6565 -mvn --projects core spring-boot:run - -# If Feast Core starts successfully, verify the correct Stores are registered -# correctly, for example by using grpc_cli. -grpc_cli call localhost:6565 GetStores '' - -# Should return something similar to the following. -# Note that you should change BigQuery projectId and datasetId accordingly -# in "$FEAST_HOME/core/src/main/resources/application.yml" - -store { - name: "SERVING" - type: REDIS - subscriptions { - project: "*" - name: "*" - version: "*" - } - redis_config { - host: "localhost" - port: 6379 - } -} -store { - name: "WAREHOUSE" - type: BIGQUERY - subscriptions { - project: "*" - name: "*" - version: "*" - } - bigquery_config { - project_id: "my-google-project-id" - dataset_id: "my-bigquery-dataset-id" - } -} -``` - -#### Starting Feast Serving - -Feast Serving requires administrators to provide an **existing** store name in Feast. -An instance of Feast Serving can only retrieve features from a **single** store. -> In order to retrieve features from multiple stores you must start **multiple** -instances of Feast serving. If you start multiple Feast serving on a single host, -make sure that they are listening on different ports. - -``` -# Start Feast Serving GRPC server on localhost:6566 with store name "SERVING" -mvn --projects serving spring-boot:run -Dspring-boot.run.arguments='--feast.store-name=SERVING' - -# To verify Feast Serving starts successfully -grpc_cli call localhost:6566 GetFeastServingType '' - -# Should return something similar to the following. -type: FEAST_SERVING_TYPE_ONLINE -``` - - -#### Registering a FeatureSet - -Create a new FeatureSet on Feast by sending a request to Feast Core. When a -feature set is successfully registered, Feast Core will start an **ingestion** job -that listens for new features in the FeatureSet. Note that Feast currently only -supports source of type "KAFKA", so you must have access to a running Kafka broker -to register a FeatureSet successfully. - -``` -# Example of registering a new driver feature set -# Note the source value, it assumes that you have access to a Kafka broker -# running on localhost:9092 - -grpc_cli call localhost:6565 ApplyFeatureSet ' -feature_set { - name: "driver" - version: 1 - - entities { - name: "driver_id" - value_type: INT64 - } - - features { - name: "city" - value_type: STRING - } - - source { - type: KAFKA - kafka_source_config { - bootstrap_servers: "localhost:9092" - } - } -} -' - -# To check that the FeatureSet has been registered correctly. -# You should also see logs from Feast Core of the ingestion job being started -grpc_cli call localhost:6565 GetFeatureSets '' -``` - - -#### Ingestion and Population of Feature Values - -``` -# Produce FeatureRow messages to Kafka so it will be ingested by Feast -# and written to the registered stores. -# Make sure the value here is the topic assigned to the feature set -# ... producer.send("feast-driver-features" ...) -# -# Install Python SDK to help writing FeatureRow messages to Kafka -cd $FEAST_HOME/sdk/python -pip3 install -e . -pip3 install pendulum - -# Produce FeatureRow messages to Kafka so it will be ingested by Feast -# and written to the corresponding store. -# Make sure the value here is the topic assigned to the feature set -# ... producer.send("feast-test_feature_set-features" ...) -python3 - < Tool Windows > Maven` -1. Drill down to e.g. `Feast Core > Plugins > spring-boot:run`, right-click and `Create 'feast-core [spring-boot'…` -1. In the dialog that pops up, check the `Resolve Workspace artifacts` box -1. Click `OK`. You should now be able to select this run configuration for the Play button in the main toolbar, keyboard shortcuts, etc. - -[idea-boot-main]: https://stackoverflow.com/questions/30237768/run-spring-boots-main-using-ide - -#### Tips for Running Postgres, Redis and Kafka with Docker - -This guide assumes you are running Docker service on a bridge network (which -is usually the case if you're running Linux). Otherwise, you may need to -use different network options than shown below. - -> `--net host` usually only works as expected when you're running Docker -> service in bridge networking mode. - -``` -# Start Postgres -docker run --name postgres --rm -it -d --net host -e POSTGRES_DB=postgres -e POSTGRES_USER=postgres \ --e POSTGRES_PASSWORD=password postgres:12-alpine - -# Start Redis -docker run --name redis --rm -it --net host -d redis:5-alpine - -# Start Zookeeper (needed by Kafka) -docker run --rm \ - --net=host \ - --name=zookeeper \ - --env=ZOOKEEPER_CLIENT_PORT=2181 \ - --detach confluentinc/cp-zookeeper:5.2.1 - -# Start Kafka -docker run --rm \ - --net=host \ - --name=kafka \ - --env=KAFKA_ZOOKEEPER_CONNECT=localhost:2181 \ - --env=KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \ - --env=KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ - --detach confluentinc/cp-kafka:5.2.1 -``` - -## Code reviews - -Code submission to Feast (including submission from project maintainers) requires review and approval. -Please submit a **pull request** to initiate the code review process. We use [prow](https://github.com/kubernetes/test-infra/tree/master/prow) to manage the testing and reviewing of pull requests. Please refer to [config.yaml](../.prow/config.yaml) for details on the test jobs. - -## Code conventions - -### Java - -We conform to the [Google Java Style Guide]. Maven can helpfully take care of -that for you before you commit: - - $ mvn spotless:apply - -Formatting will be checked automatically during the `verify` phase. This can be -skipped temporarily: - - $ mvn spotless:check # Check is automatic upon `mvn verify` - $ mvn verify -Dspotless.check.skip - -If you're using IntelliJ, you can import [these code style settings][G -IntelliJ] if you'd like to use the IDE's reformat function as you work. - -### Go - -Make sure you apply `go fmt`. - -[Google Java Style Guide]: https://google.github.io/styleguide/javaguide.html -[G IntelliJ]: https://github.com/google/styleguide/blob/gh-pages/intellij-java-google-style.xml diff --git a/README.md b/README.md index d9b16748266..ef494274974 100644 --- a/README.md +++ b/README.md @@ -29,11 +29,12 @@ prediction = my_model.predict(fs.get_online_features(customer_features, customer ``` ## Important resources - * [Why Feast?](docs/why-feast.md) - * [Concepts](docs/concepts.md) - * [Installation](docs/getting-started/installing-feast.md) - * [Getting Help](docs/community.md) + * [Why Feast?](https://docs.feast.dev/why-feast) + * [Concepts](https://docs.feast.dev/concepts) + * [Installation](https://docs.feast.dev/getting-started/installing-feast) + * [Getting Help](https://docs.feast.dev/getting-help) + * [Example Notebook](https://github.com/gojek/feast/blob/master/examples/basic/basic.ipynb) ## Notice -Feast is a community project and is still under active development. Your feedback and contributions are important to us. Please have a look at our [contributing guide](CONTRIBUTING.md) for details. +Feast is a community project and is still under active development. Your feedback and contributions are important to us. Please have a look at our [contributing guide](docs/contributing.md) for details. From 6dae18f774356b7d5a0b68390041aea4eb53df9a Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Tue, 7 Jan 2020 20:34:40 +0800 Subject: [PATCH 002/176] Fix missing CI dependency in Python SDK documentation building --- sdk/python/feast/client.py | 2 +- sdk/python/requirements-ci.txt | 14 +++++++++++--- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/sdk/python/feast/client.py b/sdk/python/feast/client.py index a68f0fe2bc5..fb5fe6ffc49 100644 --- a/sdk/python/feast/client.py +++ b/sdk/python/feast/client.py @@ -615,7 +615,7 @@ def ingest( Loads feature data into Feast for a specific feature set. Args: - feature_set (typing.Union[str, FeatureSet]): + feature_set (typing.Union[str, feast.feature_set.FeatureSet]): Feature set object or the string name of the feature set (without a version). diff --git a/sdk/python/requirements-ci.txt b/sdk/python/requirements-ci.txt index f3df60a02ec..d0fdd76e498 100644 --- a/sdk/python/requirements-ci.txt +++ b/sdk/python/requirements-ci.txt @@ -1,3 +1,4 @@ +Click==7.* google-api-core==1.* google-auth==1.* google-cloud-bigquery==1.* @@ -13,10 +14,17 @@ protobuf==3.* pytest pytest-mock pytest-timeout -PyYAML==5.1.2 -fastavro==0.21.* +PyYAML==5.1.* +fastavro==0.* grpcio-testing==1.* pytest-ordering==0.6.* pyarrow Sphinx -sphinx-rtd-theme \ No newline at end of file +sphinx-rtd-theme +toml==0.10.* +tqdm==4.* +confluent_kafka +google +pandavro==1.5.* +kafka-python==1.* +tabulate==0.8.* \ No newline at end of file From d12bcce3f5fb6a55570343bcca669e7ff9805670 Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Tue, 7 Jan 2020 20:42:14 +0800 Subject: [PATCH 003/176] Fix missing CI dependency for Netlify --- sdk/python/docs/requirements.txt | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/sdk/python/docs/requirements.txt b/sdk/python/docs/requirements.txt index a7e825b5d49..3a9bbfee453 100644 --- a/sdk/python/docs/requirements.txt +++ b/sdk/python/docs/requirements.txt @@ -18,6 +18,14 @@ fastavro==0.21.* grpcio-testing==1.* pytest-ordering==0.6.* pyarrow +Click==7.* +toml==0.10.* +tqdm==4.* +confluent_kafka +google +pandavro==1.5.* +kafka-python==1.* +tabulate==0.8.* Sphinx==2.* sphinx-autodoc-napoleon-typehints sphinx-autodoc-typehints @@ -28,4 +36,4 @@ sphinxcontrib-htmlhelp sphinxcontrib-jsmath sphinxcontrib-napoleon sphinxcontrib-qthelp -sphinxcontrib-serializinghtml +sphinxcontrib-serializinghtml \ No newline at end of file From 1a00db17d69f25ad02781b9dfb15f8d1492b7574 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=E5=BC=B5=E6=B3=B0=E7=91=8B=28Chang=20Tai=20Wei=29?= Date: Wed, 8 Jan 2020 14:08:04 +0800 Subject: [PATCH 004/176] (README): add a link for user to view docs on GitBook (#385) Co-authored-by: Willem Pienaar <6728866+woop@users.noreply.github.com> --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index ef494274974..fb36f6cabcb 100644 --- a/README.md +++ b/README.md @@ -29,6 +29,9 @@ prediction = my_model.predict(fs.get_online_features(customer_features, customer ``` ## Important resources + +Please refer to the official docs at + * [Why Feast?](https://docs.feast.dev/why-feast) * [Concepts](https://docs.feast.dev/concepts) * [Installation](https://docs.feast.dev/getting-started/installing-feast) From 1afec9be339aae5ffa925e19377ba7629f8afa46 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Wed, 8 Jan 2020 14:48:50 +0800 Subject: [PATCH 005/176] Increase resource requests to distribute tests (#418) --- .prow/config.yaml | 14 ++++---------- 1 file changed, 4 insertions(+), 10 deletions(-) diff --git a/.prow/config.yaml b/.prow/config.yaml index c63d3dce797..0c0e979b10d 100644 --- a/.prow/config.yaml +++ b/.prow/config.yaml @@ -70,10 +70,8 @@ presubmits: command: [".prow/scripts/test-core-ingestion.sh"] resources: requests: - cpu: "1500m" + cpu: "2000m" memory: "1536Mi" - limit: - memory: "4096Mi" - name: test-serving decorate: true @@ -116,9 +114,7 @@ presubmits: command: [".prow/scripts/test-end-to-end.sh"] resources: requests: - cpu: "3000m" - memory: "4096Mi" - limit: + cpu: "6" memory: "6144Mi" - name: test-end-to-end-batch @@ -134,10 +130,8 @@ presubmits: command: [".prow/scripts/test-end-to-end-batch.sh"] resources: requests: - cpu: "1000m" - memory: "1024Mi" - limit: - memory: "4096Mi" + cpu: "6" + memory: "6144Mi" volumeMounts: - name: service-account mountPath: "/etc/service-account" From c82ba8f1b95987d27f3284867a97647393ab8f0b Mon Sep 17 00:00:00 2001 From: Khor Shu Heng <32997938+khorshuheng@users.noreply.github.com> Date: Wed, 8 Jan 2020 16:00:40 +0800 Subject: [PATCH 006/176] Fix null pointer exception in Dataflow Runner due to unserializable backoff (#417) --- .../java/feast/core/service/SpecService.java | 4 +- .../java/feast/core/util/PackageUtil.java | 6 +- .../ingestion/transform/WriteToStore.java | 15 +- .../transform/fn/ValidateFeatureRowDoFn.java | 7 +- .../java/feast/retry/BackOffExecutor.java | 66 +++++--- .../src/main/java/feast/retry/Retriable.java | 24 ++- .../store/serving/redis/RedisCustomIO.java | 117 ++++++++------- .../transform/ValidateFeatureRowsTest.java | 14 +- .../serving/redis/RedisCustomIOTest.java | 141 ++++++++++-------- 9 files changed, 228 insertions(+), 166 deletions(-) diff --git a/core/src/main/java/feast/core/service/SpecService.java b/core/src/main/java/feast/core/service/SpecService.java index 1d6ce16de54..129fa68a82c 100644 --- a/core/src/main/java/feast/core/service/SpecService.java +++ b/core/src/main/java/feast/core/service/SpecService.java @@ -143,8 +143,8 @@ public GetFeatureSetResponse getFeatureSet(GetFeatureSetRequest request) { * possible if a project name is not set explicitly * *

The version field can be one of - '*' - This will match all versions - 'latest' - This will - * match the latest feature set version - '<number>' - This will match a specific feature set - * version. This property can only be set if both the feature set name and project name are + * match the latest feature set version - '<number>' - This will match a specific feature + * set version. This property can only be set if both the feature set name and project name are * explicitly set. * * @param filter filter containing the desired featureSet name and version filter diff --git a/core/src/main/java/feast/core/util/PackageUtil.java b/core/src/main/java/feast/core/util/PackageUtil.java index 20b2310644b..99c5d73ba78 100644 --- a/core/src/main/java/feast/core/util/PackageUtil.java +++ b/core/src/main/java/feast/core/util/PackageUtil.java @@ -44,9 +44,9 @@ public class PackageUtil { * points to the resource location. Note that the extraction process can take several minutes to * complete. * - *

One use case of this function is to detect the class path of resources to stage when - * using Dataflow runner. The resource URL however is in "jar:file:" format, which cannot be - * handled by default in Apache Beam. + *

One use case of this function is to detect the class path of resources to stage when using + * Dataflow runner. The resource URL however is in "jar:file:" format, which cannot be handled by + * default in Apache Beam. * *

    * 
diff --git a/ingestion/src/main/java/feast/ingestion/transform/WriteToStore.java b/ingestion/src/main/java/feast/ingestion/transform/WriteToStore.java
index 778540595a2..b7901c2f90c 100644
--- a/ingestion/src/main/java/feast/ingestion/transform/WriteToStore.java
+++ b/ingestion/src/main/java/feast/ingestion/transform/WriteToStore.java
@@ -89,15 +89,14 @@ public PDone expand(PCollection input) {
     switch (storeType) {
       case REDIS:
         RedisConfig redisConfig = getStore().getRedisConfig();
-        PCollection redisWriteResult = input
-            .apply(
-                "FeatureRowToRedisMutation",
-                ParDo.of(new FeatureRowToRedisMutationDoFn(getFeatureSets())))
-            .apply(
-                "WriteRedisMutationToRedis",
-                RedisCustomIO.write(redisConfig));
+        PCollection redisWriteResult =
+            input
+                .apply(
+                    "FeatureRowToRedisMutation",
+                    ParDo.of(new FeatureRowToRedisMutationDoFn(getFeatureSets())))
+                .apply("WriteRedisMutationToRedis", RedisCustomIO.write(redisConfig));
         if (options.getDeadLetterTableSpec() != null) {
-            redisWriteResult.apply(
+          redisWriteResult.apply(
               WriteFailedElementToBigQuery.newBuilder()
                   .setTableSpec(options.getDeadLetterTableSpec())
                   .setJsonSchema(ResourceUtil.getDeadletterTableSchemaJson())
diff --git a/ingestion/src/main/java/feast/ingestion/transform/fn/ValidateFeatureRowDoFn.java b/ingestion/src/main/java/feast/ingestion/transform/fn/ValidateFeatureRowDoFn.java
index 7d61a62f3fc..c31d3c535e9 100644
--- a/ingestion/src/main/java/feast/ingestion/transform/fn/ValidateFeatureRowDoFn.java
+++ b/ingestion/src/main/java/feast/ingestion/transform/fn/ValidateFeatureRowDoFn.java
@@ -24,10 +24,8 @@
 import feast.types.FieldProto;
 import feast.types.ValueProto.Value.ValCase;
 import java.util.ArrayList;
-import java.util.HashSet;
 import java.util.List;
 import java.util.Map;
-import java.util.Set;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.values.TupleTag;
 
@@ -111,10 +109,7 @@ public void processElement(ProcessContext context) {
       }
       context.output(getFailureTag(), failedElement.build());
     } else {
-      featureRow = featureRow.toBuilder()
-                    .clearFields()
-                    .addAllFields(fields)
-                    .build();
+      featureRow = featureRow.toBuilder().clearFields().addAllFields(fields).build();
       context.output(getSuccessTag(), featureRow);
     }
   }
diff --git a/ingestion/src/main/java/feast/retry/BackOffExecutor.java b/ingestion/src/main/java/feast/retry/BackOffExecutor.java
index 7e38a3cf706..344c65ac424 100644
--- a/ingestion/src/main/java/feast/retry/BackOffExecutor.java
+++ b/ingestion/src/main/java/feast/retry/BackOffExecutor.java
@@ -1,38 +1,58 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ * Copyright 2018-2020 The Feast Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
 package feast.retry;
 
+import java.io.Serializable;
 import org.apache.beam.sdk.util.BackOff;
 import org.apache.beam.sdk.util.BackOffUtils;
 import org.apache.beam.sdk.util.FluentBackoff;
 import org.apache.beam.sdk.util.Sleeper;
 import org.joda.time.Duration;
 
-import java.io.IOException;
-import java.io.Serializable;
-
 public class BackOffExecutor implements Serializable {
 
-    private static FluentBackoff backoff;
+  private final Integer maxRetries;
+  private final Duration initialBackOff;
 
-    public BackOffExecutor(Integer maxRetries, Duration initialBackOff) {
-        backoff = FluentBackoff.DEFAULT
-                .withMaxRetries(maxRetries)
-                .withInitialBackoff(initialBackOff);
-    }
+  public BackOffExecutor(Integer maxRetries, Duration initialBackOff) {
+    this.maxRetries = maxRetries;
+    this.initialBackOff = initialBackOff;
+  }
+
+  public void execute(Retriable retriable) throws Exception {
+    FluentBackoff backoff =
+        FluentBackoff.DEFAULT.withMaxRetries(maxRetries).withInitialBackoff(initialBackOff);
+    execute(retriable, backoff);
+  }
 
-    public void execute(Retriable retriable) throws Exception {
-        Sleeper sleeper = Sleeper.DEFAULT;
-        BackOff backOff = backoff.backoff();
-        while(true) {
-            try {
-                retriable.execute();
-                break;
-            } catch (Exception e) {
-                if(retriable.isExceptionRetriable(e) && BackOffUtils.next(sleeper, backOff)) {
-                    retriable.cleanUpAfterFailure();
-                } else {
-                    throw e;
-                }
-            }
+  private void execute(Retriable retriable, FluentBackoff backoff) throws Exception {
+    Sleeper sleeper = Sleeper.DEFAULT;
+    BackOff backOff = backoff.backoff();
+    while (true) {
+      try {
+        retriable.execute();
+        break;
+      } catch (Exception e) {
+        if (retriable.isExceptionRetriable(e) && BackOffUtils.next(sleeper, backOff)) {
+          retriable.cleanUpAfterFailure();
+        } else {
+          throw e;
         }
+      }
     }
+  }
 }
diff --git a/ingestion/src/main/java/feast/retry/Retriable.java b/ingestion/src/main/java/feast/retry/Retriable.java
index 8fd76fedbb1..0a788fcdd69 100644
--- a/ingestion/src/main/java/feast/retry/Retriable.java
+++ b/ingestion/src/main/java/feast/retry/Retriable.java
@@ -1,7 +1,25 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ * Copyright 2018-2020 The Feast Authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     https://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
 package feast.retry;
 
 public interface Retriable {
-    void execute();
-    Boolean isExceptionRetriable(Exception e);
-    void cleanUpAfterFailure();
+  void execute();
+
+  Boolean isExceptionRetriable(Exception e);
+
+  void cleanUpAfterFailure();
 }
diff --git a/ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java b/ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java
index 20afc43d76c..8c142b66c93 100644
--- a/ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java
+++ b/ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java
@@ -20,6 +20,9 @@
 import feast.ingestion.values.FailedElement;
 import feast.retry.BackOffExecutor;
 import feast.retry.Retriable;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
 import org.apache.avro.reflect.Nullable;
 import org.apache.beam.sdk.coders.AvroCoder;
 import org.apache.beam.sdk.coders.DefaultCoder;
@@ -38,10 +41,6 @@
 import redis.clients.jedis.Response;
 import redis.clients.jedis.exceptions.JedisConnectionException;
 
-import java.io.IOException;
-import java.util.ArrayList;
-import java.util.List;
-
 public class RedisCustomIO {
 
   private static final int DEFAULT_BATCH_SIZE = 1000;
@@ -164,7 +163,8 @@ public void setScore(@Nullable Long score) {
   }
 
   /** ServingStoreWrite data to a Redis server. */
-  public static class Write extends PTransform, PCollection> {
+  public static class Write
+      extends PTransform, PCollection> {
 
     private WriteDoFn dofn;
 
@@ -202,9 +202,10 @@ public static class WriteDoFn extends DoFn {
       WriteDoFn(StoreProto.Store.RedisConfig redisConfig) {
         this.host = redisConfig.getHost();
         this.port = redisConfig.getPort();
-        long backoffMs = redisConfig.getInitialBackoffMs() > 0 ? redisConfig.getInitialBackoffMs() : 1;
-        this.backOffExecutor = new BackOffExecutor(redisConfig.getMaxRetries(),
-                Duration.millis(backoffMs));
+        long backoffMs =
+            redisConfig.getInitialBackoffMs() > 0 ? redisConfig.getInitialBackoffMs() : 1;
+        this.backOffExecutor =
+            new BackOffExecutor(redisConfig.getMaxRetries(), Duration.millis(backoffMs));
       }
 
       public WriteDoFn withBatchSize(int batchSize) {
@@ -233,47 +234,50 @@ public void startBundle() {
       }
 
       private void executeBatch() throws Exception {
-        backOffExecutor.execute(new Retriable() {
-          @Override
-          public void execute() {
-            pipeline.multi();
-            mutations.forEach(mutation -> {
-              writeRecord(mutation);
-              if (mutation.getExpiryMillis() != null && mutation.getExpiryMillis() > 0) {
-                pipeline.pexpire(mutation.getKey(), mutation.getExpiryMillis());
+        backOffExecutor.execute(
+            new Retriable() {
+              @Override
+              public void execute() {
+                pipeline.multi();
+                mutations.forEach(
+                    mutation -> {
+                      writeRecord(mutation);
+                      if (mutation.getExpiryMillis() != null && mutation.getExpiryMillis() > 0) {
+                        pipeline.pexpire(mutation.getKey(), mutation.getExpiryMillis());
+                      }
+                    });
+                pipeline.exec();
+                pipeline.sync();
+                mutations.clear();
               }
-            });
-            pipeline.exec();
-            pipeline.sync();
-            mutations.clear();
-          }
 
-          @Override
-          public Boolean isExceptionRetriable(Exception e) {
-            return e instanceof JedisConnectionException;
-          }
+              @Override
+              public Boolean isExceptionRetriable(Exception e) {
+                return e instanceof JedisConnectionException;
+              }
 
-          @Override
-          public void cleanUpAfterFailure() {
-            try {
-              pipeline.close();
-            } catch (IOException e) {
-              log.error(String.format("Error while closing pipeline: %s", e.getMessage()));
-            }
-            jedis = new Jedis(host, port, timeout);
-            pipeline = jedis.pipelined();
-          }
-        });
+              @Override
+              public void cleanUpAfterFailure() {
+                try {
+                  pipeline.close();
+                } catch (IOException e) {
+                  log.error(String.format("Error while closing pipeline: %s", e.getMessage()));
+                }
+                jedis = new Jedis(host, port, timeout);
+                pipeline = jedis.pipelined();
+              }
+            });
       }
 
-      private FailedElement toFailedElement(RedisMutation mutation, Exception exception, String jobName) {
+      private FailedElement toFailedElement(
+          RedisMutation mutation, Exception exception, String jobName) {
         return FailedElement.newBuilder()
-          .setJobName(jobName)
-          .setTransformName("RedisCustomIO")
-          .setPayload(mutation.getValue().toString())
-          .setErrorMessage(exception.getMessage())
-          .setStackTrace(ExceptionUtils.getStackTrace(exception))
-          .build();
+            .setJobName(jobName)
+            .setTransformName("RedisCustomIO")
+            .setPayload(mutation.getValue().toString())
+            .setErrorMessage(exception.getMessage())
+            .setStackTrace(ExceptionUtils.getStackTrace(exception))
+            .build();
       }
 
       @ProcessElement
@@ -284,11 +288,12 @@ public void processElement(ProcessContext context) {
           try {
             executeBatch();
           } catch (Exception e) {
-            mutations.forEach(failedMutation -> {
-              FailedElement failedElement = toFailedElement(
-                failedMutation, e, context.getPipelineOptions().getJobName());
-              context.output(failedElement);
-            });
+            mutations.forEach(
+                failedMutation -> {
+                  FailedElement failedElement =
+                      toFailedElement(failedMutation, e, context.getPipelineOptions().getJobName());
+                  context.output(failedElement);
+                });
             mutations.clear();
           }
         }
@@ -315,16 +320,18 @@ private Response writeRecord(RedisMutation mutation) {
       }
 
       @FinishBundle
-      public void finishBundle(FinishBundleContext context) throws IOException, InterruptedException {
-        if(mutations.size() > 0) {
+      public void finishBundle(FinishBundleContext context)
+          throws IOException, InterruptedException {
+        if (mutations.size() > 0) {
           try {
             executeBatch();
           } catch (Exception e) {
-            mutations.forEach(failedMutation -> {
-              FailedElement failedElement = toFailedElement(
-                failedMutation, e, context.getPipelineOptions().getJobName());
-              context.output(failedElement, Instant.now(), GlobalWindow.INSTANCE);
-            });
+            mutations.forEach(
+                failedMutation -> {
+                  FailedElement failedElement =
+                      toFailedElement(failedMutation, e, context.getPipelineOptions().getJobName());
+                  context.output(failedElement, Instant.now(), GlobalWindow.INSTANCE);
+                });
             mutations.clear();
           }
         }
diff --git a/ingestion/src/test/java/feast/ingestion/transform/ValidateFeatureRowsTest.java b/ingestion/src/test/java/feast/ingestion/transform/ValidateFeatureRowsTest.java
index aca39563877..5c9860ed97f 100644
--- a/ingestion/src/test/java/feast/ingestion/transform/ValidateFeatureRowsTest.java
+++ b/ingestion/src/test/java/feast/ingestion/transform/ValidateFeatureRowsTest.java
@@ -180,12 +180,14 @@ public void shouldExcludeUnregisteredFields() {
 
     FeatureRow randomRow = TestUtil.createRandomFeatureRow(fs1);
     expected.add(randomRow);
-    input.add(randomRow.toBuilder()
-        .addFields(Field.newBuilder()
-          .setName("extra")
-          .setValue(Value.newBuilder().setStringVal("hello")))
-        .build()
-    );
+    input.add(
+        randomRow
+            .toBuilder()
+            .addFields(
+                Field.newBuilder()
+                    .setName("extra")
+                    .setValue(Value.newBuilder().setStringVal("hello")))
+            .build());
 
     PCollectionTuple output =
         p.apply(Create.of(input))
diff --git a/ingestion/src/test/java/feast/store/serving/redis/RedisCustomIOTest.java b/ingestion/src/test/java/feast/store/serving/redis/RedisCustomIOTest.java
index 94167059b43..fc17f6207f6 100644
--- a/ingestion/src/test/java/feast/store/serving/redis/RedisCustomIOTest.java
+++ b/ingestion/src/test/java/feast/store/serving/redis/RedisCustomIOTest.java
@@ -16,12 +16,24 @@
  */
 package feast.store.serving.redis;
 
+import static feast.test.TestUtil.field;
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.MatcherAssert.assertThat;
+
 import feast.core.StoreProto;
 import feast.storage.RedisProto.RedisKey;
 import feast.store.serving.redis.RedisCustomIO.Method;
 import feast.store.serving.redis.RedisCustomIO.RedisMutation;
 import feast.types.FeatureRowProto.FeatureRow;
 import feast.types.ValueProto.ValueType.Enum;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.concurrent.ScheduledFuture;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.stream.Collectors;
 import org.apache.beam.sdk.testing.PAssert;
 import org.apache.beam.sdk.testing.TestPipeline;
 import org.apache.beam.sdk.transforms.Count;
@@ -35,29 +47,14 @@
 import redis.embedded.Redis;
 import redis.embedded.RedisServer;
 
-import java.io.IOException;
-import java.util.HashMap;
-import java.util.LinkedHashMap;
-import java.util.List;
-import java.util.concurrent.ScheduledFuture;
-import java.util.concurrent.ScheduledThreadPoolExecutor;
-import java.util.concurrent.TimeUnit;
-import java.util.stream.Collectors;
-
-import static feast.test.TestUtil.field;
-import static org.hamcrest.CoreMatchers.equalTo;
-import static org.hamcrest.MatcherAssert.assertThat;
-
 public class RedisCustomIOTest {
-  @Rule
-  public transient TestPipeline p = TestPipeline.create();
+  @Rule public transient TestPipeline p = TestPipeline.create();
 
   private static String REDIS_HOST = "localhost";
   private static int REDIS_PORT = 51234;
   private Redis redis;
   private Jedis jedis;
 
-
   @Before
   public void setUp() throws IOException {
     redis = new RedisServer(REDIS_PORT);
@@ -72,10 +69,8 @@ public void teardown() {
 
   @Test
   public void shouldWriteToRedis() {
-    StoreProto.Store.RedisConfig redisConfig = StoreProto.Store.RedisConfig.newBuilder()
-            .setHost(REDIS_HOST)
-            .setPort(REDIS_PORT)
-            .build();
+    StoreProto.Store.RedisConfig redisConfig =
+        StoreProto.Store.RedisConfig.newBuilder().setHost(REDIS_HOST).setPort(REDIS_PORT).build();
     HashMap kvs = new LinkedHashMap<>();
     kvs.put(
         RedisKey.newBuilder()
@@ -110,8 +105,7 @@ public void shouldWriteToRedis() {
                         null))
             .collect(Collectors.toList());
 
-    p.apply(Create.of(featureRowWrites))
-        .apply(RedisCustomIO.write(redisConfig));
+    p.apply(Create.of(featureRowWrites)).apply(RedisCustomIO.write(redisConfig));
     p.run();
 
     kvs.forEach(
@@ -123,68 +117,95 @@ public void shouldWriteToRedis() {
 
   @Test(timeout = 10000)
   public void shouldRetryFailConnection() throws InterruptedException {
-    StoreProto.Store.RedisConfig redisConfig = StoreProto.Store.RedisConfig.newBuilder()
+    StoreProto.Store.RedisConfig redisConfig =
+        StoreProto.Store.RedisConfig.newBuilder()
             .setHost(REDIS_HOST)
             .setPort(REDIS_PORT)
             .setMaxRetries(4)
             .setInitialBackoffMs(2000)
             .build();
     HashMap kvs = new LinkedHashMap<>();
-    kvs.put(RedisKey.newBuilder().setFeatureSet("fs:1")
-                    .addEntities(field("entity", 1, Enum.INT64)).build(),
-            FeatureRow.newBuilder().setFeatureSet("fs:1")
-                    .addFields(field("entity", 1, Enum.INT64))
-                    .addFields(field("feature", "one", Enum.STRING)).build());
-
-    List featureRowWrites = kvs.entrySet().stream()
-            .map(kv -> new RedisMutation(Method.SET, kv.getKey().toByteArray(),
-                    kv.getValue().toByteArray(),
-                    null, null)
-            )
+    kvs.put(
+        RedisKey.newBuilder()
+            .setFeatureSet("fs:1")
+            .addEntities(field("entity", 1, Enum.INT64))
+            .build(),
+        FeatureRow.newBuilder()
+            .setFeatureSet("fs:1")
+            .addFields(field("entity", 1, Enum.INT64))
+            .addFields(field("feature", "one", Enum.STRING))
+            .build());
+
+    List featureRowWrites =
+        kvs.entrySet().stream()
+            .map(
+                kv ->
+                    new RedisMutation(
+                        Method.SET,
+                        kv.getKey().toByteArray(),
+                        kv.getValue().toByteArray(),
+                        null,
+                        null))
             .collect(Collectors.toList());
 
-    PCollection failedElementCount = p.apply(Create.of(featureRowWrites))
-        .apply(RedisCustomIO.write(redisConfig))
-        .apply(Count.globally());
+    PCollection failedElementCount =
+        p.apply(Create.of(featureRowWrites))
+            .apply(RedisCustomIO.write(redisConfig))
+            .apply(Count.globally());
 
     redis.stop();
     final ScheduledThreadPoolExecutor redisRestartExecutor = new ScheduledThreadPoolExecutor(1);
-    ScheduledFuture scheduledRedisRestart = redisRestartExecutor.schedule(() -> {
-      redis.start();
-    }, 3, TimeUnit.SECONDS);
+    ScheduledFuture scheduledRedisRestart =
+        redisRestartExecutor.schedule(
+            () -> {
+              redis.start();
+            },
+            3,
+            TimeUnit.SECONDS);
 
     PAssert.that(failedElementCount).containsInAnyOrder(0L);
     p.run();
     scheduledRedisRestart.cancel(true);
 
-    kvs.forEach((key, value) -> {
-      byte[] actual = jedis.get(key.toByteArray());
-      assertThat(actual, equalTo(value.toByteArray()));
-    });
+    kvs.forEach(
+        (key, value) -> {
+          byte[] actual = jedis.get(key.toByteArray());
+          assertThat(actual, equalTo(value.toByteArray()));
+        });
   }
 
   @Test
   public void shouldProduceFailedElementIfRetryExceeded() {
-    StoreProto.Store.RedisConfig redisConfig = StoreProto.Store.RedisConfig.newBuilder()
-        .setHost(REDIS_HOST)
-        .setPort(REDIS_PORT)
-        .build();
+    StoreProto.Store.RedisConfig redisConfig =
+        StoreProto.Store.RedisConfig.newBuilder().setHost(REDIS_HOST).setPort(REDIS_PORT).build();
     HashMap kvs = new LinkedHashMap<>();
-    kvs.put(RedisKey.newBuilder().setFeatureSet("fs:1")
-            .addEntities(field("entity", 1, Enum.INT64)).build(),
-        FeatureRow.newBuilder().setFeatureSet("fs:1")
+    kvs.put(
+        RedisKey.newBuilder()
+            .setFeatureSet("fs:1")
+            .addEntities(field("entity", 1, Enum.INT64))
+            .build(),
+        FeatureRow.newBuilder()
+            .setFeatureSet("fs:1")
             .addFields(field("entity", 1, Enum.INT64))
-            .addFields(field("feature", "one", Enum.STRING)).build());
+            .addFields(field("feature", "one", Enum.STRING))
+            .build());
 
-    List featureRowWrites = kvs.entrySet().stream()
-            .map(kv -> new RedisMutation(Method.SET, kv.getKey().toByteArray(),
-                    kv.getValue().toByteArray(),
-                    null, null)
-            ).collect(Collectors.toList());
+    List featureRowWrites =
+        kvs.entrySet().stream()
+            .map(
+                kv ->
+                    new RedisMutation(
+                        Method.SET,
+                        kv.getKey().toByteArray(),
+                        kv.getValue().toByteArray(),
+                        null,
+                        null))
+            .collect(Collectors.toList());
 
-    PCollection failedElementCount = p.apply(Create.of(featureRowWrites))
-        .apply(RedisCustomIO.write(redisConfig))
-        .apply(Count.globally());
+    PCollection failedElementCount =
+        p.apply(Create.of(featureRowWrites))
+            .apply(RedisCustomIO.write(redisConfig))
+            .apply(Count.globally());
 
     redis.stop();
     PAssert.that(failedElementCount).containsInAnyOrder(1L);

From 8bdc38ccebfa3e3ab4bdf465c7724ff3489f08ee Mon Sep 17 00:00:00 2001
From: Ches Martin 
Date: Wed, 8 Jan 2020 15:36:41 +0700
Subject: [PATCH 007/176] Introduce datatypes/java module for proto generation
 (#391)

Rather than the Maven protobuf plugin running on the same symlinked
definitions in several Java modules, localize this process into one
module that the others depend on.

This provides a single module that can be depended on by third-party
extensions with the bare minimum of dependencies.

Also removes proto files that are no longer used.
---
 core/pom.xml                                  |  4 --
 core/src/main/proto/feast                     |  1 -
 core/src/main/proto/third_party               |  1 -
 datatypes/java/README.md                      | 43 +++++++++++
 datatypes/java/pom.xml                        | 72 +++++++++++++++++++
 {sdk => datatypes}/java/src/main/proto/feast  |  0
 datatypes/java/src/main/proto/third_party     |  1 +
 ingestion/pom.xml                             | 10 +--
 ingestion/src/main/proto/feast                |  1 -
 .../feast_ingestion/types/CoalesceAccum.proto | 35 ---------
 .../feast_ingestion/types/CoalesceKey.proto   | 25 -------
 ingestion/src/main/proto/third_party          |  1 -
 ingestion/src/test/proto/DriverArea.proto     | 10 ---
 ingestion/src/test/proto/Ping.proto           | 12 ----
 pom.xml                                       | 20 +-----
 sdk/java/pom.xml                              | 10 +--
 serving/pom.xml                               | 10 +--
 serving/src/main/proto/feast                  |  1 -
 serving/src/main/proto/third_party            |  1 -
 19 files changed, 135 insertions(+), 123 deletions(-)
 delete mode 120000 core/src/main/proto/feast
 delete mode 120000 core/src/main/proto/third_party
 create mode 100644 datatypes/java/README.md
 create mode 100644 datatypes/java/pom.xml
 rename {sdk => datatypes}/java/src/main/proto/feast (100%)
 create mode 120000 datatypes/java/src/main/proto/third_party
 delete mode 120000 ingestion/src/main/proto/feast
 delete mode 100644 ingestion/src/main/proto/feast_ingestion/types/CoalesceAccum.proto
 delete mode 100644 ingestion/src/main/proto/feast_ingestion/types/CoalesceKey.proto
 delete mode 120000 ingestion/src/main/proto/third_party
 delete mode 100644 ingestion/src/test/proto/DriverArea.proto
 delete mode 100644 ingestion/src/test/proto/Ping.proto
 delete mode 120000 serving/src/main/proto/feast
 delete mode 120000 serving/src/main/proto/third_party

diff --git a/core/pom.xml b/core/pom.xml
index 954c7c00185..e1567ae8fe3 100644
--- a/core/pom.xml
+++ b/core/pom.xml
@@ -39,10 +39,6 @@
                     false
                 
             
-            
-                org.xolstice.maven.plugins
-                protobuf-maven-plugin
-            
         
     
 
diff --git a/core/src/main/proto/feast b/core/src/main/proto/feast
deleted file mode 120000
index d520da9126b..00000000000
--- a/core/src/main/proto/feast
+++ /dev/null
@@ -1 +0,0 @@
-../../../../protos/feast
\ No newline at end of file
diff --git a/core/src/main/proto/third_party b/core/src/main/proto/third_party
deleted file mode 120000
index 363d20598e6..00000000000
--- a/core/src/main/proto/third_party
+++ /dev/null
@@ -1 +0,0 @@
-../../../../protos/third_party
\ No newline at end of file
diff --git a/datatypes/java/README.md b/datatypes/java/README.md
new file mode 100644
index 00000000000..f93bd99aa38
--- /dev/null
+++ b/datatypes/java/README.md
@@ -0,0 +1,43 @@
+Feast Data Types for Java
+=========================
+
+This module produces Java class files for Feast's data type and gRPC service
+definitions, from Protobuf IDL. These are used across Feast components for wire
+interchange, contracts, etc.
+
+End users of Feast will be best served by our Java SDK which adds higher-level
+conveniences, but the data types are published independently for custom needs,
+without any additional dependencies the SDK may add.
+
+Dependency Coordinates
+----------------------
+
+```xml
+
+  dev.feast
+  datatypes-java
+  0.4.0-SNAPSHOT
+
+```
+
+Using the `.proto` Definitions
+------------------------------
+
+The `.proto` definitions are packaged as resources within the Maven artifact,
+which may be useful to `include` them in dependent Protobuf definitions in a
+downstream project, or for other JVM languages to consume from their builds to
+generate more idiomatic bindings.
+
+Google's Gradle plugin, for instance, [can use protos in dependencies][Gradle]
+either for `include` or to compile with a different `protoc` plugin than Java.
+
+[sbt-protoc] offers similar functionality for sbt/Scala.
+
+[Gradle]: https://github.com/google/protobuf-gradle-plugin#protos-in-dependencies
+[sbt-protoc]: https://github.com/thesamet/sbt-protoc
+
+Publishing
+----------
+
+TODO: this module should be published to Maven Central upon Feast releases—this
+needs to be set up in POM configuration and release automation.
diff --git a/datatypes/java/pom.xml b/datatypes/java/pom.xml
new file mode 100644
index 00000000000..a6dfa8e345a
--- /dev/null
+++ b/datatypes/java/pom.xml
@@ -0,0 +1,72 @@
+
+
+
+    4.0.0
+
+    Feast Data Types for Java
+    
+        Data types and service contracts used throughout Feast components and
+        their interchanges. These are generated from Protocol Buffers and gRPC
+        definitions included in the package.
+    
+    datatypes-java
+
+    
+      dev.feast
+      feast-parent
+      ${revision}
+      ../..
+    
+
+    
+      
+        
+          org.xolstice.maven.plugins
+          protobuf-maven-plugin
+          
+            true
+            
+                com.google.protobuf:protoc:${protocVersion}:exe:${os.detected.classifier}
+            
+            grpc-java
+            
+                io.grpc:protoc-gen-grpc-java:${grpcVersion}:exe:${os.detected.classifier}
+            
+          
+          
+            
+              
+                compile
+                compile-custom
+                test-compile
+              
+            
+          
+        
+      
+    
+
+    
+      
+        io.grpc
+        grpc-services
+      
+    
+
diff --git a/sdk/java/src/main/proto/feast b/datatypes/java/src/main/proto/feast
similarity index 100%
rename from sdk/java/src/main/proto/feast
rename to datatypes/java/src/main/proto/feast
diff --git a/datatypes/java/src/main/proto/third_party b/datatypes/java/src/main/proto/third_party
new file mode 120000
index 00000000000..f015f8477d1
--- /dev/null
+++ b/datatypes/java/src/main/proto/third_party
@@ -0,0 +1 @@
+../../../../../protos/third_party
\ No newline at end of file
diff --git a/ingestion/pom.xml b/ingestion/pom.xml
index 4908b546985..2e1dee65536 100644
--- a/ingestion/pom.xml
+++ b/ingestion/pom.xml
@@ -31,10 +31,6 @@
 
   
     
-      
-        org.xolstice.maven.plugins
-        protobuf-maven-plugin
-      
       
         org.apache.maven.plugins
         maven-shade-plugin
@@ -90,6 +86,12 @@
   
 
   
+    
+      dev.feast
+      datatypes-java
+      ${project.version}
+    
+
     
       org.glassfish
       javax.el
diff --git a/ingestion/src/main/proto/feast b/ingestion/src/main/proto/feast
deleted file mode 120000
index d520da9126b..00000000000
--- a/ingestion/src/main/proto/feast
+++ /dev/null
@@ -1 +0,0 @@
-../../../../protos/feast
\ No newline at end of file
diff --git a/ingestion/src/main/proto/feast_ingestion/types/CoalesceAccum.proto b/ingestion/src/main/proto/feast_ingestion/types/CoalesceAccum.proto
deleted file mode 100644
index cb64dd715f6..00000000000
--- a/ingestion/src/main/proto/feast_ingestion/types/CoalesceAccum.proto
+++ /dev/null
@@ -1,35 +0,0 @@
-/*
- * Copyright 2018 The Feast Authors
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     https://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-syntax = "proto3";
-
-import "google/protobuf/timestamp.proto";
-import "feast/types/Field.proto";
-
-option java_package = "feast_ingestion.types";
-option java_outer_classname = "CoalesceAccumProto";
-
-// Accumlator for merging feature rows.
-message CoalesceAccum {
-  string entityKey = 1;
-  google.protobuf.Timestamp eventTimestamp = 3;
-  string entityName = 4;
-
-  map features = 6;
-  // map of features to their counter values when they were last added to accumulator
-  map featureMarks = 7;
-  int64 counter = 8;
-}
\ No newline at end of file
diff --git a/ingestion/src/main/proto/feast_ingestion/types/CoalesceKey.proto b/ingestion/src/main/proto/feast_ingestion/types/CoalesceKey.proto
deleted file mode 100644
index 9730b49ec3b..00000000000
--- a/ingestion/src/main/proto/feast_ingestion/types/CoalesceKey.proto
+++ /dev/null
@@ -1,25 +0,0 @@
-/*
- * Copyright 2018 The Feast Authors
- *
- * Licensed under the Apache License, Version 2.0 (the "License");
- * you may not use this file except in compliance with the License.
- * You may obtain a copy of the License at
- *
- *     https://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-syntax = "proto3";
-
-option java_package = "feast_ingestion.types";
-option java_outer_classname = "CoalesceKeyProto";
-
-message CoalesceKey {
-  string entityName = 1;
-  string entityKey = 2;
-}
\ No newline at end of file
diff --git a/ingestion/src/main/proto/third_party b/ingestion/src/main/proto/third_party
deleted file mode 120000
index 363d20598e6..00000000000
--- a/ingestion/src/main/proto/third_party
+++ /dev/null
@@ -1 +0,0 @@
-../../../../protos/third_party
\ No newline at end of file
diff --git a/ingestion/src/test/proto/DriverArea.proto b/ingestion/src/test/proto/DriverArea.proto
deleted file mode 100644
index fee838b9e17..00000000000
--- a/ingestion/src/test/proto/DriverArea.proto
+++ /dev/null
@@ -1,10 +0,0 @@
-syntax = "proto3";
-
-package feast;
-
-option java_outer_classname = "DriverAreaProto";
-
-message DriverArea {
-  int32 driverId = 1;
-  int32 areaId = 2;
-}
\ No newline at end of file
diff --git a/ingestion/src/test/proto/Ping.proto b/ingestion/src/test/proto/Ping.proto
deleted file mode 100644
index b1069afa5bd..00000000000
--- a/ingestion/src/test/proto/Ping.proto
+++ /dev/null
@@ -1,12 +0,0 @@
-syntax = "proto3";
-
-package feast;
-import "google/protobuf/timestamp.proto";
-
-option java_outer_classname = "PingProto";
-
-message Ping {
-  double lat = 1;
-  double lng = 2;
-  google.protobuf.Timestamp timestamp = 3;
-}
diff --git a/pom.xml b/pom.xml
index 05fb701ac44..939dc8507c7 100644
--- a/pom.xml
+++ b/pom.xml
@@ -28,6 +28,7 @@
     pom
 
     
+        datatypes/java
         ingestion
         core
         serving
@@ -542,25 +543,6 @@
                     org.xolstice.maven.plugins
                     protobuf-maven-plugin
                     0.6.1
-                    
-                        true
-                        
-                            com.google.protobuf:protoc:${protocVersion}:exe:${os.detected.classifier}
-                        
-                        grpc-java
-                        
-                            io.grpc:protoc-gen-grpc-java:${grpcVersion}:exe:${os.detected.classifier}
-                        
-                    
-                    
-                        
-                            
-                                compile
-                                compile-custom
-                                test-compile
-                            
-                        
-                    
                 
             
         
diff --git a/sdk/java/pom.xml b/sdk/java/pom.xml
index 2970dae3ee2..e8a82a485fc 100644
--- a/sdk/java/pom.xml
+++ b/sdk/java/pom.xml
@@ -21,6 +21,12 @@
   
 
   
+    
+      dev.feast
+      datatypes-java
+      ${project.version}
+    
+
     
     
       io.grpc
@@ -79,10 +85,6 @@
 
   
     
-      
-        org.xolstice.maven.plugins
-        protobuf-maven-plugin
-      
       
       
         org.apache.maven.plugins
diff --git a/serving/pom.xml b/serving/pom.xml
index dc3391df62f..c15881030e2 100644
--- a/serving/pom.xml
+++ b/serving/pom.xml
@@ -47,10 +47,6 @@
           false
         
       
-      
-        org.xolstice.maven.plugins
-        protobuf-maven-plugin
-      
       
         org.apache.maven.plugins
         maven-failsafe-plugin
@@ -74,6 +70,12 @@
   
 
   
+    
+      dev.feast
+      datatypes-java
+      ${project.version}
+    
+
     
     
       org.slf4j
diff --git a/serving/src/main/proto/feast b/serving/src/main/proto/feast
deleted file mode 120000
index d520da9126b..00000000000
--- a/serving/src/main/proto/feast
+++ /dev/null
@@ -1 +0,0 @@
-../../../../protos/feast
\ No newline at end of file
diff --git a/serving/src/main/proto/third_party b/serving/src/main/proto/third_party
deleted file mode 120000
index 363d20598e6..00000000000
--- a/serving/src/main/proto/third_party
+++ /dev/null
@@ -1 +0,0 @@
-../../../../protos/third_party
\ No newline at end of file

From 17e7dca8238aae4dcbf0ff9f0db5d80ef8e035cf Mon Sep 17 00:00:00 2001
From: Chen Zhiling 
Date: Wed, 8 Jan 2020 17:13:40 +0800
Subject: [PATCH 008/176] Add PR template (#416)

* Add PR template

* Remove line about not using fixes

* Add notes about CLA and release notes
---
 .github/pull_request_template.md | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
 create mode 100644 .github/pull_request_template.md

diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
new file mode 100644
index 00000000000..b9c8cd6dff8
--- /dev/null
+++ b/.github/pull_request_template.md
@@ -0,0 +1,31 @@
+
+
+**What this PR does / why we need it**:
+
+**Which issue(s) this PR fixes**:
+
+Fixes #
+
+**Does this PR introduce a user-facing change?**:
+
+```release-note
+
+```

From 9912d453ae5a33fec3e1bd3ae905b039571a572e Mon Sep 17 00:00:00 2001
From: Willem Pienaar 
Date: Wed, 8 Jan 2020 14:21:32 +0000
Subject: [PATCH 009/176] GitBook: [master] one page modified

---
 docs/getting-started/installing-feast.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/docs/getting-started/installing-feast.md b/docs/getting-started/installing-feast.md
index 0b8b42f26f6..527f07741fd 100644
--- a/docs/getting-started/installing-feast.md
+++ b/docs/getting-started/installing-feast.md
@@ -306,13 +306,13 @@ kubectl create secret generic feast-gcp-service-account --from-file=key.json
 For this guide we will use `NodePort` for exposing Feast services. In order to do so, we must find an internal IP of at least one GKE node.
 
 ```bash
-export FEAST_IP=$(kubectl describe nodes | grep InternalIP | awk '{print $2}' | head -n 1)
+export FEAST_IP=$(kubectl describe nodes | grep ExternalIP | awk '{print $2}' | head -n 1)
 export FEAST_CORE_URL=${FEAST_IP}:32090
 export FEAST_ONLINE_SERVING_URL=${FEAST_IP}:32091
 export FEAST_BATCH_SERVING_URL=${FEAST_IP}:32092
 ```
 
-Confirm that you are able to access this node:
+Confirm that you are able to access this node \(please make sure that no firewall rules are preventing access to these ports\):
 
 ```bash
 ping $FEAST_IP

From 6681d4f2ee557ac6b7d78153628890797192c559 Mon Sep 17 00:00:00 2001
From: Willem Pienaar <6728866+woop@users.noreply.github.com>
Date: Thu, 9 Jan 2020 09:40:40 +0800
Subject: [PATCH 010/176] Update basic Feast example to Feast 0.4 (#424)

---
 examples/basic/basic.ipynb | 61 +++++++++++++++++++-------------------
 1 file changed, 31 insertions(+), 30 deletions(-)

diff --git a/examples/basic/basic.ipynb b/examples/basic/basic.ipynb
index 6a83e6a08b5..49658b42357 100644
--- a/examples/basic/basic.ipynb
+++ b/examples/basic/basic.ipynb
@@ -2,12 +2,10 @@
  "cells": [
   {
    "cell_type": "markdown",
+   "metadata": {},
    "source": [
     "# Feast Basic Customer Transactions Example"
-   ],
-   "metadata": {
-    "collapsed": false
-   }
+   ]
   },
   {
    "cell_type": "markdown",
@@ -48,7 +46,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 8,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -73,7 +71,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 9,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -84,11 +82,13 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 10,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
-    "client = Client(core_url=CORE_URL, serving_url=BATCH_SERVING_URL) # Connect to Feast Core"
+    "client = Client(core_url=CORE_URL, serving_url=BATCH_SERVING_URL) # Connect to Feast Core\n",
+    "client.create_project('customer_project')\n",
+    "client.set_project('customer_project')"
    ]
   },
   {
@@ -107,7 +107,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 24,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -119,7 +119,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 25,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -154,7 +154,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 13,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -174,7 +174,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 26,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -197,7 +197,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 16,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -213,7 +213,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 17,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -230,7 +230,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 27,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -255,7 +255,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 30,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -280,14 +280,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 32,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "job = client.get_batch_features(\n",
-    "                            feature_ids=[\n",
-    "                                f\"customer_transactions:{customer_fs.version}:daily_transactions\", \n",
-    "                                f\"customer_transactions:{customer_fs.version}:total_transactions\", \n",
+    "                            feature_refs=[\n",
+    "                                f\"daily_transactions\", \n",
+    "                                f\"total_transactions\", \n",
     "                               ],\n",
     "                            entity_rows=entity_rows\n",
     "                         )\n",
@@ -311,11 +311,12 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 36,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
-    "online_client = Client(core_url=CORE_URL, serving_url=ONLINE_SERVING_URL)"
+    "online_client = Client(core_url=CORE_URL, serving_url=ONLINE_SERVING_URL)\n",
+    "online_client.set_project(\"customer_project\")"
    ]
   },
   {
@@ -327,14 +328,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 37,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
     "online_features = online_client.get_online_features(\n",
-    "    feature_ids=[\n",
-    "        f\"customer_transactions:{customer_fs.version}:daily_transactions\",\n",
-    "        f\"customer_transactions:{customer_fs.version}:total_transactions\",\n",
+    "    feature_refs=[\n",
+    "        f\"daily_transactions\",\n",
+    "        f\"total_transactions\",\n",
     "    ],\n",
     "    entity_rows=[\n",
     "        GetOnlineFeaturesRequest.EntityRow(\n",
@@ -373,18 +374,18 @@
    "name": "python",
    "nbconvert_exporter": "python",
    "pygments_lexer": "ipython3",
-   "version": "3.7.4"
+   "version": "3.7.3"
   },
   "pycharm": {
    "stem_cell": {
     "cell_type": "raw",
-    "source": [],
     "metadata": {
      "collapsed": false
-    }
+    },
+    "source": []
    }
   }
  },
  "nbformat": 4,
  "nbformat_minor": 2
-}
\ No newline at end of file
+}

From 0e31aefbbf26022b0f082d07a687f7a9f2d0bedf Mon Sep 17 00:00:00 2001
From: Willem Pienaar <6728866+woop@users.noreply.github.com>
Date: Thu, 9 Jan 2020 11:25:40 +0800
Subject: [PATCH 011/176] Update Changelog (#423)

* Initial commit of changelog up to 0.4.3

* Remove unreleased changes on master

* Add missing changelog manually

Co-authored-by: Khor Shu Heng <32997938+khorshuheng@users.noreply.github.com>
---
 CHANGELOG.md | 139 +++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 139 insertions(+)

diff --git a/CHANGELOG.md b/CHANGELOG.md
index f6ad89e0c0d..98399c61bb5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,144 @@
 # Changelog
 
+## [v0.4.3](https://github.com/gojek/feast/tree/v0.4.3) (2020-01-08)
+
+[Full Changelog](https://github.com/gojek/feast/compare/v0.4.2...v0.4.3)
+
+**Fixed bugs:**
+
+- Bugfix for redis ingestion retries throwing NullPointerException on remote runners [\#417](https://github.com/gojek/feast/pull/417) ([khorshuheng](https://github.com/khorshuheng))
+
+## [v0.4.2](https://github.com/gojek/feast/tree/v0.4.2) (2020-01-07)
+
+[Full Changelog](https://github.com/gojek/feast/compare/v0.4.1...v0.4.2)
+
+**Fixed bugs:**
+
+- Missing argument in error string in ValidateFeatureRowDoFn [\#401](https://github.com/gojek/feast/issues/401)
+
+**Merged pull requests:**
+
+- Define maven revision property when packaging jars in Dockerfile so the images are built successfully [\#410](https://github.com/gojek/feast/pull/410) ([davidheryanto](https://github.com/davidheryanto))
+- Deduplicate rows in subquery [\#409](https://github.com/gojek/feast/pull/409) ([zhilingc](https://github.com/zhilingc))
+- Filter out extra fields, deduplicate fields in ingestion [\#404](https://github.com/gojek/feast/pull/404) ([zhilingc](https://github.com/zhilingc))
+- Automatic documentation generation for gRPC API [\#403](https://github.com/gojek/feast/pull/403) ([woop](https://github.com/woop))
+- Update feast core default values to include hibernate merge strategy [\#400](https://github.com/gojek/feast/pull/400) ([zhilingc](https://github.com/zhilingc))
+- Move cli into feast package [\#398](https://github.com/gojek/feast/pull/398) ([zhilingc](https://github.com/zhilingc))
+- Use Nexus staging plugin for deployment [\#394](https://github.com/gojek/feast/pull/394) ([khorshuheng](https://github.com/khorshuheng))
+- Handle retry for redis io flow [\#274](https://github.com/gojek/feast/pull/274) ([khorshuheng](https://github.com/khorshuheng))
+
+## [v0.4.1](https://github.com/gojek/feast/tree/v0.4.1) (2019-12-30)
+
+[Full Changelog](https://github.com/gojek/feast/compare/v0.4.0...v0.4.1)
+
+**Merged pull requests:**
+
+- Add project-related commands to CLI [\#397](https://github.com/gojek/feast/pull/397) ([zhilingc](https://github.com/zhilingc))
+
+## [v0.4.0](https://github.com/gojek/feast/tree/v0.4.0) (2019-12-28)
+
+[Full Changelog](https://github.com/gojek/feast/compare/v0.3.5...v0.4.0)
+
+**Implemented enhancements:**
+
+- Edit description in feature specification to also reflect in BigQuery schema description. [\#239](https://github.com/gojek/feast/issues/239)
+- Allow for disabling of metrics pushing [\#57](https://github.com/gojek/feast/issues/57)
+
+**Merged pull requests:**
+
+- Java SDK release script [\#406](https://github.com/gojek/feast/pull/406) ([davidheryanto](https://github.com/davidheryanto))
+- Use fixed 'dev' revision for test-e2e-batch [\#395](https://github.com/gojek/feast/pull/395) ([davidheryanto](https://github.com/davidheryanto))
+- Project Namespacing [\#393](https://github.com/gojek/feast/pull/393) ([woop](https://github.com/woop))
+- \\(concepts\): change data types to upper case because lower case … [\#389](https://github.com/gojek/feast/pull/389) ([david30907d](https://github.com/david30907d))
+- Remove alpha v1 from java package name [\#387](https://github.com/gojek/feast/pull/387) ([khorshuheng](https://github.com/khorshuheng))
+- Minor bug fixes for Python SDK [\#383](https://github.com/gojek/feast/pull/383) ([voonhous](https://github.com/voonhous))
+- Allow user to override job options [\#377](https://github.com/gojek/feast/pull/377) ([khorshuheng](https://github.com/khorshuheng))
+- Add documentation to default values.yaml in Feast chart [\#376](https://github.com/gojek/feast/pull/376) ([davidheryanto](https://github.com/davidheryanto))
+- Add support for file paths for providing entity rows during batch retrieval  [\#375](https://github.com/gojek/feast/pull/375) ([voonhous](https://github.com/voonhous))
+- Update sync helm chart script to ensure requirements.lock in in sync with requirements.yaml [\#373](https://github.com/gojek/feast/pull/373) ([davidheryanto](https://github.com/davidheryanto))
+- Catch errors thrown by BQ during entity table loading [\#371](https://github.com/gojek/feast/pull/371) ([zhilingc](https://github.com/zhilingc))
+- Async job management [\#361](https://github.com/gojek/feast/pull/361) ([zhilingc](https://github.com/zhilingc))
+- Infer schema of PyArrow table directly [\#355](https://github.com/gojek/feast/pull/355) ([voonhous](https://github.com/voonhous))
+- Add readiness checks for Feast services in end to end test [\#337](https://github.com/gojek/feast/pull/337) ([davidheryanto](https://github.com/davidheryanto))
+- Create CHANGELOG.md [\#321](https://github.com/gojek/feast/pull/321) ([woop](https://github.com/woop))
+
+## [v0.3.6](https://github.com/gojek/feast/tree/v0.3.6) (2020-01-03)
+
+**Merged pull requests:**
+
+[Full Changelog](https://github.com/gojek/feast/compare/v0.3.5...v0.3.6)
+
+- Add support for file paths for providing entity rows during batch retrieval [\#375](https://github.com/gojek/feast/pull/376) ([voonhous](https://github.com/voonhous))
+
+## [v0.3.5](https://github.com/gojek/feast/tree/v0.3.5) (2019-12-26)
+
+[Full Changelog](https://github.com/gojek/feast/compare/v0.3.4...v0.3.5)
+
+**Merged pull requests:**
+
+- Always set destination table in BigQuery query config in Feast Batch Serving so it can handle large results [\#392](https://github.com/gojek/feast/pull/392) ([davidheryanto](https://github.com/davidheryanto))
+
+## [v0.3.4](https://github.com/gojek/feast/tree/v0.3.4) (2019-12-23)
+
+[Full Changelog](https://github.com/gojek/feast/compare/v0.3.3...v0.3.4)
+
+**Merged pull requests:**
+
+- Make redis key creation more determinisitic [\#380](https://github.com/gojek/feast/pull/380) ([zhilingc](https://github.com/zhilingc))
+
+## [v0.3.3](https://github.com/gojek/feast/tree/v0.3.3) (2019-12-18)
+
+[Full Changelog](https://github.com/gojek/feast/compare/v0.3.2...v0.3.3)
+
+**Implemented enhancements:**
+
+- Added Docker Compose for Feast [\#272](https://github.com/gojek/feast/issues/272)
+- Added ability to check import job status and cancel job through Python SDK [\#194](https://github.com/gojek/feast/issues/194)
+- Added basic customer transactions example [\#354](https://github.com/gojek/feast/pull/354) ([woop](https://github.com/woop))
+
+**Merged pull requests:**
+
+- Added Prow jobs to automate the release of Docker images and Python SDK [\#369](https://github.com/gojek/feast/pull/369) ([davidheryanto](https://github.com/davidheryanto))
+- Fixed installation link in README.md [\#368](https://github.com/gojek/feast/pull/368) ([Jeffwan](https://github.com/Jeffwan))
+- Fixed Java SDK tests not actually running \(missing dependencies\) [\#366](https://github.com/gojek/feast/pull/366) ([woop](https://github.com/woop))
+- Added more batch retrieval tests [\#357](https://github.com/gojek/feast/pull/357) ([zhilingc](https://github.com/zhilingc))
+- Python SDK and Feast Core Bug Fixes [\#353](https://github.com/gojek/feast/pull/353) ([woop](https://github.com/woop))
+- Updated buildFeatureSets method in Golang SDK [\#351](https://github.com/gojek/feast/pull/351) ([davidheryanto](https://github.com/davidheryanto))
+- Python SDK cleanup [\#348](https://github.com/gojek/feast/pull/348) ([woop](https://github.com/woop))
+- Broke up queries for point in time correctness joins [\#347](https://github.com/gojek/feast/pull/347) ([zhilingc](https://github.com/zhilingc))
+- Exports gRPC call metrics and Feast resource metrics in Core [\#345](https://github.com/gojek/feast/pull/345) ([davidheryanto](https://github.com/davidheryanto))
+- Fixed broken Google Group link on Community page [\#343](https://github.com/gojek/feast/pull/343) ([ches](https://github.com/ches))
+- Ensured ImportJobTest is not flaky by checking WriteToStore metric and requesting adequate resources for testing [\#332](https://github.com/gojek/feast/pull/332) ([davidheryanto](https://github.com/davidheryanto))
+- Added docker-compose file with Jupyter notebook [\#328](https://github.com/gojek/feast/pull/328) ([khorshuheng](https://github.com/khorshuheng))
+- Added minimal implementation of ingesting Parquet and CSV files [\#327](https://github.com/gojek/feast/pull/327) ([voonhous](https://github.com/voonhous))
+
+## [v0.3.2](https://github.com/gojek/feast/tree/v0.3.2) (2019-11-29)
+
+[Full Changelog](https://github.com/gojek/feast/compare/v0.3.1...v0.3.2)
+
+**Merged pull requests:**
+
+- Fixed incorrect BigQuery schema creation from FeatureSetSpec [\#340](https://github.com/gojek/feast/pull/340) ([davidheryanto](https://github.com/davidheryanto))
+- Filtered out feature sets that dont share the same source [\#339](https://github.com/gojek/feast/pull/339) ([zhilingc](https://github.com/zhilingc))
+- Changed latency calculation method to not use Timer [\#338](https://github.com/gojek/feast/pull/338) ([zhilingc](https://github.com/zhilingc))
+- Moved Prometheus annotations to pod template for serving [\#336](https://github.com/gojek/feast/pull/336) ([zhilingc](https://github.com/zhilingc))
+- Removed metrics windowing, cleaned up step names for metrics writing [\#334](https://github.com/gojek/feast/pull/334) ([zhilingc](https://github.com/zhilingc))
+- Set BigQuery table time partition inside get table function [\#333](https://github.com/gojek/feast/pull/333) ([zhilingc](https://github.com/zhilingc))
+- Added unit test in Redis to return values with no max age set [\#329](https://github.com/gojek/feast/pull/329) ([smadarasmi](https://github.com/smadarasmi))
+- Consolidated jobs into single steps instead of branching out [\#326](https://github.com/gojek/feast/pull/326) ([zhilingc](https://github.com/zhilingc))
+- Pinned Python SDK to minor versions for dependencies [\#322](https://github.com/gojek/feast/pull/322) ([woop](https://github.com/woop))
+- Added Auto format to Google style with Spotless [\#317](https://github.com/gojek/feast/pull/317) ([ches](https://github.com/ches))
+
+## [v0.3.1](https://github.com/gojek/feast/tree/v0.3.1) (2019-11-25)
+
+[Full Changelog](https://github.com/gojek/feast/compare/v0.3.0...v0.3.1)
+
+**Merged pull requests:**
+
+- Added Prometheus metrics to serving [\#316](https://github.com/gojek/feast/pull/316) ([zhilingc](https://github.com/zhilingc))
+- Changed default job metrics sink to Statsd [\#315](https://github.com/gojek/feast/pull/315) ([zhilingc](https://github.com/zhilingc))
+- Fixed module import error in Feast CLI [\#314](https://github.com/gojek/feast/pull/314) ([davidheryanto](https://github.com/davidheryanto))
+
 ## [v0.3.0](https://github.com/gojek/feast/tree/v0.3.0) (2019-11-19)
 
 [Full Changelog](https://github.com/gojek/feast/compare/v0.1.8...v0.3.0)

From 20ce16e8fa7d914ed8bf04c0988ea467bfafb7e8 Mon Sep 17 00:00:00 2001
From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com>
Date: Thu, 9 Jan 2020 17:17:40 +0800
Subject: [PATCH 012/176] Bump hibernate-validator from 6.0.13.Final to
 6.1.0.Final in /ingestion (#421)

Bumps [hibernate-validator](https://github.com/hibernate/hibernate-validator) from 6.0.13.Final to 6.1.0.Final.
- [Release notes](https://github.com/hibernate/hibernate-validator/releases)
- [Changelog](https://github.com/hibernate/hibernate-validator/blob/master/changelog.txt)
- [Commits](https://github.com/hibernate/hibernate-validator/compare/6.0.13.Final...6.1.0.Final)

Signed-off-by: dependabot[bot] 
---
 ingestion/pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ingestion/pom.xml b/ingestion/pom.xml
index 2e1dee65536..e3961d33855 100644
--- a/ingestion/pom.xml
+++ b/ingestion/pom.xml
@@ -107,7 +107,7 @@
     
       org.hibernate.validator
       hibernate-validator
-      6.0.13.Final
+      6.1.0.Final
     
 
     

From 2a33f7bbbe6bd92f292a128102f864cd67e95660 Mon Sep 17 00:00:00 2001
From: Ches Martin 
Date: Thu, 9 Jan 2020 22:24:41 +0700
Subject: [PATCH 013/176] Publish datatypes/java along with sdk/java (#426)

This forward-ports a straggling commit from #407: it was missed when
initially creating the datatypes module because Sonatype publishing
setup was added concurrently.
---
 .prow/scripts/publish-java-sdk.sh |  2 +-
 datatypes/java/README.md          | 20 ++++++++++++++++----
 2 files changed, 17 insertions(+), 5 deletions(-)

diff --git a/.prow/scripts/publish-java-sdk.sh b/.prow/scripts/publish-java-sdk.sh
index 17513d0eb0d..91123c8d4ee 100755
--- a/.prow/scripts/publish-java-sdk.sh
+++ b/.prow/scripts/publish-java-sdk.sh
@@ -69,4 +69,4 @@ gpg --import --batch --yes $GPG_KEY_IMPORT_DIR/private-key
 echo "============================================================"
 echo "Deploying Java SDK with revision: $REVISION"
 echo "============================================================"
-mvn --projects sdk/java -Drevision=$REVISION --batch-mode clean deploy
+mvn --projects datatypes/java,sdk/java -Drevision=$REVISION --batch-mode clean deploy
diff --git a/datatypes/java/README.md b/datatypes/java/README.md
index f93bd99aa38..535fac73d2e 100644
--- a/datatypes/java/README.md
+++ b/datatypes/java/README.md
@@ -20,6 +20,11 @@ Dependency Coordinates
 
 ```
 
+Use the version corresponding to the Feast release you have deployed in your
+environment—see the [Feast release notes] for details.
+
+[Feast release notes]: ../../CHANGELOG.md
+
 Using the `.proto` Definitions
 ------------------------------
 
@@ -36,8 +41,15 @@ either for `include` or to compile with a different `protoc` plugin than Java.
 [Gradle]: https://github.com/google/protobuf-gradle-plugin#protos-in-dependencies
 [sbt-protoc]: https://github.com/thesamet/sbt-protoc
 
-Publishing
-----------
+Releases
+--------
+
+The module is published to Maven Central upon each release of Feast (since
+v0.3.7).
+
+For developers, the publishing process is automated along with the Java SDK by
+[the `publish-java-sdk` build task in Prow][prow task], where you can see how
+it works. Artifacts are staged to Sonatype where a maintainer needs to take a
+release action for them to go live on Maven Central.
 
-TODO: this module should be published to Maven Central upon Feast releases—this
-needs to be set up in POM configuration and release automation.
+[prow task]: https://github.com/gojek/feast/blob/17e7dca8238aae4dcbf0ff9f0db5d80ef8e035cf/.prow/config.yaml#L166-L192

From 833d49559a4d353ac682f27268725fcc14f93f6e Mon Sep 17 00:00:00 2001
From: Willem Pienaar <6728866+woop@users.noreply.github.com>
Date: Thu, 16 Jan 2020 03:02:42 +0200
Subject: [PATCH 014/176] Remove "resource" concept and the need to specify a
 kind in feature sets (#432)

---
 infra/docker-compose/docker-compose.yml |  5 ++-
 sdk/__init__.py                         |  0
 sdk/python/feast/cli.py                 | 43 ++++++++-----------------
 sdk/python/feast/feature_set.py         |  2 --
 sdk/python/feast/loaders/yaml.py        |  7 ++--
 sdk/python/feast/resource.py            | 10 ------
 6 files changed, 19 insertions(+), 48 deletions(-)
 delete mode 100644 sdk/__init__.py
 delete mode 100644 sdk/python/feast/resource.py

diff --git a/infra/docker-compose/docker-compose.yml b/infra/docker-compose/docker-compose.yml
index a224500ca0a..44750650cec 100644
--- a/infra/docker-compose/docker-compose.yml
+++ b/infra/docker-compose/docker-compose.yml
@@ -59,6 +59,8 @@ services:
 
   redis:
     image: redis:5-alpine
+    ports:
+      - "6379:6379"
 
   kafka:
     image: confluentinc/cp-kafka:5.2.1
@@ -70,7 +72,8 @@ services:
       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
       KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
     ports:
-      - 9094:9092
+      - "9092:9092"
+      - "9094:9094"
 
     depends_on:
       - zookeeper
diff --git a/sdk/__init__.py b/sdk/__init__.py
deleted file mode 100644
index e69de29bb2d..00000000000
diff --git a/sdk/python/feast/cli.py b/sdk/python/feast/cli.py
index 601f41d4b11..8e8f185d038 100644
--- a/sdk/python/feast/cli.py
+++ b/sdk/python/feast/cli.py
@@ -17,7 +17,6 @@
 import click
 from feast import config as feast_config
 from feast.client import Client
-from feast.resource import ResourceFactory
 from feast.feature_set import FeatureSet
 import toml
 import pkg_resources
@@ -147,17 +146,25 @@ def feature_set_list():
     print(tabulate(table, headers=["NAME", "VERSION"], tablefmt="plain"))
 
 
-@feature_set.command("create")
-@click.argument("name")
-def feature_set_create(name):
+@feature_set.command("apply")
+@click.option(
+    "--filename",
+    "-f",
+    help="Path to a feature set configuration file that will be applied",
+    type=click.Path(exists=True),
+)
+def feature_set_create(filename):
     """
-    Create a feature set
+    Create or update a feature set
     """
+
+    feature_sets = [FeatureSet.from_dict(fs_dict) for fs_dict in yaml_loader(filename)]
+
     feast_client = Client(
         core_url=feast_config.get_config_property_or_fail("core_url")
     )  # type: Client
 
-    feast_client.apply(FeatureSet(name=name))
+    feast_client.apply(feature_sets)
 
 
 @feature_set.command("describe")
@@ -264,29 +271,5 @@ def ingest(name, version, filename, file_type):
     feature_set.ingest_file(file_path=filename)
 
 
-@cli.command()
-@click.option(
-    "--filename",
-    "-f",
-    help="Path to the configuration file that will be applied",
-    type=click.Path(exists=True),
-)
-def apply(filename):
-    """
-    Apply a configuration to a resource by filename or stdin
-    """
-
-    resources = [
-        ResourceFactory.get_resource(res_dict["kind"]).from_dict(res_dict)
-        for res_dict in yaml_loader(filename)
-    ]
-
-    feast_client = Client(
-        core_url=feast_config.get_config_property_or_fail("core_url")
-    )  # type: Client
-
-    feast_client.apply(resources)
-
-
 if __name__ == "__main__":
     cli()
diff --git a/sdk/python/feast/feature_set.py b/sdk/python/feast/feature_set.py
index d5576607513..c47c51e5a21 100644
--- a/sdk/python/feast/feature_set.py
+++ b/sdk/python/feast/feature_set.py
@@ -689,8 +689,6 @@ def from_dict(cls, fs_dict):
             Returns a FeatureSet object based on the feature set dict
         """
 
-        if ("kind" not in fs_dict) and (fs_dict["kind"].strip() != "feature_set"):
-            raise Exception(f"Resource kind is not a feature set {str(fs_dict)}")
         feature_set_proto = json_format.ParseDict(
             fs_dict, FeatureSetProto(), ignore_unknown_fields=True
         )
diff --git a/sdk/python/feast/loaders/yaml.py b/sdk/python/feast/loaders/yaml.py
index 4cbe15dfaaf..130a71a3d02 100644
--- a/sdk/python/feast/loaders/yaml.py
+++ b/sdk/python/feast/loaders/yaml.py
@@ -53,7 +53,7 @@ def _get_yaml_contents(yml: str) -> str:
         with open(yml, "r") as f:
             yml_content = f.read()
 
-    elif isinstance(yml, str) and "kind" in yml.lower():
+    elif isinstance(yml, str):
         yml_content = yml
     else:
         raise Exception(
@@ -73,7 +73,4 @@ def _yaml_to_dict(yaml_string):
         Dictionary containing the same object
     """
 
-    yaml_dict = yaml.safe_load(yaml_string)
-    if not isinstance(yaml_dict, dict) or not "kind" in yaml_dict:
-        raise Exception(f"Could not detect YAML kind from resource: ${yaml_string}")
-    return yaml_dict
+    return yaml.safe_load(yaml_string)
diff --git a/sdk/python/feast/resource.py b/sdk/python/feast/resource.py
deleted file mode 100644
index 17a65291667..00000000000
--- a/sdk/python/feast/resource.py
+++ /dev/null
@@ -1,10 +0,0 @@
-from feast.feature_set import FeatureSet
-
-# TODO: This factory adds no value. It should be removed asap.
-class ResourceFactory:
-    @staticmethod
-    def get_resource(kind):
-        if kind == "feature_set":
-            return FeatureSet
-        else:
-            raise ValueError(kind)

From 5fcc30fb882905a5bf1f7c5a80dc6f74d5477d2f Mon Sep 17 00:00:00 2001
From: Lionel Vital 
Date: Sat, 18 Jan 2020 00:18:43 -0800
Subject: [PATCH 015/176] Update GKE installation and chart values to work with
 0.4.3 (#434)

---
 docs/getting-started/installing-feast.md |  15 ++-
 infra/charts/feast/values.yaml           | 131 ++++++++++++++++-------
 2 files changed, 105 insertions(+), 41 deletions(-)

diff --git a/docs/getting-started/installing-feast.md b/docs/getting-started/installing-feast.md
index 527f07741fd..0b212037c12 100644
--- a/docs/getting-started/installing-feast.md
+++ b/docs/getting-started/installing-feast.md
@@ -268,7 +268,7 @@ bq mk ${FEAST_BIGQUERY_DATASET_ID}
 Create the service account that Feast will run as:
 
 ```bash
-gcloud iam service-accounts create ${FEAST_SERVICE_ACCOUNT_NAME}
+gcloud iam service-accounts create ${FEAST_S_ACCOUNT_NAME}
 
 gcloud projects add-iam-policy-binding ${FEAST_GCP_PROJECT_ID} \
   --member serviceAccount:${FEAST_S_ACCOUNT_NAME}@${FEAST_GCP_PROJECT_ID}.iam.gserviceaccount.com \
@@ -324,6 +324,15 @@ PING 10.123.114.11 (10.203.164.22) 56(84) bytes of data.
 64 bytes from 10.123.114.11: icmp_seq=2 ttl=63 time=51.2 ms
 ```
 
+Add firewall rules in gcloud to open up ports:
+```bash
+gcloud compute firewall-rules create feast-core-port --allow tcp:32090
+gcloud compute firewall-rules create feast-online-port --allow tcp:32091
+gcloud compute firewall-rules create feast-batch-port --allow tcp:32092
+gcloud compute firewall-rules create feast-redis-port --allow tcp:32101
+gcloud compute firewall-rules create feast-kafka-ports --allow tcp:31090-31095
+```
+
 ### 3. Set up Helm
 
 Run the following command to provide Tiller with authorization to install Feast:
@@ -377,7 +386,8 @@ cp values.yaml my-feast-values.yaml
 Update `my-feast-values.yaml` based on your GCP and GKE environment.
 
 * Required fields are paired with comments which indicate whether they need to be replaced.
-* All occurrences of `feast.example.com` should be replaced with either your domain name or the IP stored in `$FEAST_IP`.
+* All occurrences of `EXTERNAL_IP` should be replaced with either your domain name or the IP stored in `$FEAST_IP`.
+* Replace all occurrences of `YOUR_BUCKET_NAME` with your bucket name stored in `$FEAST_GCS_BUCKET`
 
 Install the Feast Helm chart:
 
@@ -421,4 +431,3 @@ feast config set serving_url ${FEAST_ONLINE_SERVING_URL}
 ```
 
 That's it! You can now start to use Feast!
-
diff --git a/infra/charts/feast/values.yaml b/infra/charts/feast/values.yaml
index ebc8c802a16..a7d8ce00465 100644
--- a/infra/charts/feast/values.yaml
+++ b/infra/charts/feast/values.yaml
@@ -2,29 +2,29 @@
 # - Feast Core
 # - Feast Serving Online
 # - Feast Serving Batch
-# 
+#
 # The configuration for different components can be referenced from:
 # - charts/feast-core/values.yaml
 # - charts/feast-serving/values.yaml
 #
 # Note that "feast-serving-online" and "feast-serving-batch" are
 # aliases to "feast-serving" chart since in typical scenario two instances
-# of Feast Serving: online and batch will be deployed. Both described 
+# of Feast Serving: online and batch will be deployed. Both described
 # using the same chart "feast-serving".
 #
 # The following are default values for typical Feast deployment, but not
 # for production setting. Refer to "values-production.yaml" for recommended
 # values in production environment.
-# 
-# Note that the import job by default uses DirectRunner 
+#
+# Note that the import job by default uses DirectRunner
 # https://beam.apache.org/documentation/runners/direct/
 # in this configuration since it allows Feast to run in more environments
 # (unlike DataflowRunner which requires Google Cloud services).
-# 
-# A secret containing Google Cloud service account JSON key is required 
-# in this configuration. 
+#
+# A secret containing Google Cloud service account JSON key is required
+# in this configuration.
 # https://cloud.google.com/iam/docs/creating-managing-service-accounts
-# 
+#
 # The Google Cloud service account must have the following roles:
 # - bigquery.dataEditor
 # - bigquery.jobUser
@@ -32,12 +32,13 @@
 # Assuming a service account JSON key file has been downloaded to
 # (please name the file key.json):
 # /home/user/key.json
-# 
+#
 # Run the following command to create the secret in your Kubernetes cluster:
 #
 # kubectl create secret generic feast-gcp-service-account \
 #   --from-file=/home/user/key.json
 #
+# Replace every instance of EXTERNAL_IP with the external IP of your GKE cluster
 
 # ============================================================
 # Feast Core
@@ -51,12 +52,15 @@ feast-core:
   # to the client. These instances of Feast Serving however can still use
   # the same shared Feast Core.
   enabled: true
-  # jvmOptions are options that will be passed to the Java Virtual Machine (JVM) 
+  # Specify what image tag to use. Keep this consistent for all components
+  image:
+    tag: "0.4.3"
+  # jvmOptions are options that will be passed to the Java Virtual Machine (JVM)
   # running Feast Core.
   #
   # For example, it is good practice to set min and max heap size in JVM.
   # https://stackoverflow.com/questions/6902135/side-effect-for-increasing-maxpermsize-and-max-heap-size
-  jvmOptions: 
+  jvmOptions:
   - -Xms1024m
   - -Xmx1024m
   # resources that should be allocated to Feast Core.
@@ -68,18 +72,43 @@ feast-core:
       memory: 2048Mi
   # gcpServiceAccount is the Google service account that Feast Core will use.
   gcpServiceAccount:
-    # useExistingSecret specifies Feast to use an existing secret containing 
+    # useExistingSecret specifies Feast to use an existing secret containing
     # Google Cloud service account JSON key file.
-    # 
+    #
     # This is the only supported option for now to use a service account JSON.
     # Feast admin is expected to create this secret before deploying Feast.
     useExistingSecret: true
     existingSecret:
       # name is the secret name of the existing secret for the service account.
-      name: feast-gcp-service-account 
+      name: feast-gcp-service-account
       # key is the secret key of the existing secret for the service account.
       # key is normally derived from the file name of the JSON key file.
       key: key.json
+  # Setting service.type to NodePort exposes feast-core service at a static port
+  service:
+    type: NodePort
+    grpc:
+      # this is the port that is exposed outside of the cluster
+      nodePort: 32090
+  # Make kafka externally accessible using NodePort
+  # Please set EXTERNAL_IP to your cluster's external IP
+  kafka:
+    external:
+      enabled: true
+      type: NodePort
+      domain: EXTERNAL_IP
+    configurationOverrides:
+      "advertised.listeners": |-
+        EXTERNAL://EXTERNAL_IP:$((31090 + ${KAFKA_BROKER_ID}))
+      "listener.security.protocol.map": |-
+        PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
+  application.yaml:
+    feast:
+      stream:
+        options:
+          # Point to one of your Kafka brokers
+          # Please set EXTERNAL_IP to your cluster's external IP
+          bootstrapServers: EXTERNAL_IP:31090
 
 # ============================================================
 # Feast Serving Online
@@ -88,14 +117,22 @@ feast-core:
 feast-serving-online:
   # enabled specifies whether to install Feast Serving Online component.
   enabled: true
+  # Specify what image tag to use. Keep this consistent for all components
+  image:
+    tag: "0.4.3"
   # redis.enabled specifies whether Redis should be installed as part of Feast Serving.
-  # 
+  #
   # If enabled is set to "false", Feast admin has to ensure there is an
   # existing Redis running outside Feast, that Feast Serving can connect to.
+  # master.service.type set to NodePort exposes Redis to outside of the cluster
   redis:
     enabled: true
+    master:
+      service:
+        nodePort: 32101
+        type: NodePort
   # jvmOptions are options that will be passed to the Feast Serving JVM.
-  jvmOptions: 
+  jvmOptions:
   - -Xms1024m
   - -Xmx1024m
   # resources that should be allocated to Feast Serving.
@@ -105,23 +142,28 @@ feast-serving-online:
       memory: 1024Mi
     limits:
       memory: 2048Mi
+  # Make service accessible to outside of cluster using NodePort
+  service:
+    type: NodePort
+    grpc:
+      nodePort: 32091
   # store.yaml is the configuration for Feast Store.
-  # 
+  #
   # Refer to this link for more description:
   # https://github.com/gojek/feast/blob/79eb4ab5fa3d37102c1dca9968162a98690526ba/protos/feast/core/Store.proto
   store.yaml:
     name: redis
     type: REDIS
     redis_config:
-      # If redis.enabled is set to false, Feast admin should uncomment and 
-      # set the host value to an "existing" Redis instance Feast will use as 
-      # online Store. 
-      # 
-      # Else, if redis.enabled is set to true, no additional configuration is
-      # required.
+      # If redis.enabled is set to false, Feast admin should uncomment and
+      # set the host value to an "existing" Redis instance Feast will use as
+      # online Store. Also use the correct port for that existing instance.
       #
+      # Else, if redis.enabled is set to true, replace EXTERNAL_IP with your
+      # cluster's external IP.
       # host: redis-host
-      port: 6379
+      host: EXTERNAL_IP
+      port: 32101
     subscriptions:
     - name: "*"
       project: "*"
@@ -134,14 +176,17 @@ feast-serving-online:
 feast-serving-batch:
   # enabled specifies whether to install Feast Serving Batch component.
   enabled: true
+  # Specify what image tag to use. Keep this consistent for all components
+  image:
+    tag: "0.4.3"
   # redis.enabled specifies whether Redis should be installed as part of Feast Serving.
-  # 
+  #
   # This is usually set to "false" for Feast Serving Batch because the default
   # store is BigQuery.
   redis:
     enabled: false
   # jvmOptions are options that will be passed to the Feast Serving JVM.
-  jvmOptions: 
+  jvmOptions:
   - -Xms1024m
   - -Xmx1024m
   # resources that should be allocated to Feast Serving.
@@ -151,17 +196,22 @@ feast-serving-batch:
       memory: 1024Mi
     limits:
       memory: 2048Mi
+  # Make service accessible to outside of cluster using NodePort
+  service:
+    type: NodePort
+    grpc:
+      nodePort: 32092
   # gcpServiceAccount is the service account that Feast Serving will use.
   gcpServiceAccount:
-    # useExistingSecret specifies Feast to use an existing secret containing 
+    # useExistingSecret specifies Feast to use an existing secret containing
     # Google Cloud service account JSON key file.
-    # 
+    #
     # This is the only supported option for now to use a service account JSON.
     # Feast admin is expected to create this secret before deploying Feast.
     useExistingSecret: true
     existingSecret:
       # name is the secret name of the existing secret for the service account.
-      name: feast-gcp-service-account 
+      name: feast-gcp-service-account
       # key is the secret key of the existing secret for the service account.
       # key is normally derived from the file name of the JSON key file.
       key: key.json
@@ -172,28 +222,33 @@ feast-serving-batch:
   # for a complete list and description of the configuration.
   application.yaml:
     feast:
-      jobs: 
-        # staging-location specifies the URI to store intermediate files for 
+      jobs:
+        # staging-location specifies the URI to store intermediate files for
         # batch serving (required if using BigQuery as Store).
-        # 
-        # Please set the value to an "existing" Google Cloud Storage URI that 
+        #
+        # Please set the value to an "existing" Google Cloud Storage URI that
         # Feast serving has write access to.
-        staging-location: gs://bucket/path
-        # Type of store to store job metadata. 
+        staging-location: gs://YOUR_BUCKET_NAME/serving/batch
+        # Type of store to store job metadata.
         #
-        # This default configuration assumes that Feast Serving Online is 
+        # This default configuration assumes that Feast Serving Online is
         # enabled as well. So Feast Serving Batch will share the same
         # Redis instance to store job statuses.
         store-type: REDIS
+        store-options:
+          # Use the externally exposed redis instance deployed by Online service
+          # Please set EXTERNAL_IP to your cluster's external IP
+          host: EXTERNAL_IP
+          port: 32101
   # store.yaml is the configuration for Feast Store.
-  # 
+  #
   # Refer to this link for more description:
   # https://github.com/gojek/feast/blob/79eb4ab5fa3d37102c1dca9968162a98690526ba/protos/feast/core/Store.proto
   store.yaml:
     name: bigquery
     type: BIGQUERY
     bigquery_config:
-      # project_id specifies the Google Cloud Project. Please set this to the 
+      # project_id specifies the Google Cloud Project. Please set this to the
       # project id you are using BigQuery in.
       project_id: PROJECT_ID
       # dataset_id specifies an "existing" BigQuery dataset Feast Serving Batch

From 913e7c91ac6c2c26168098788638c5c68a0223fe Mon Sep 17 00:00:00 2001
From: Chen Zhiling 
Date: Sat, 18 Jan 2020 17:07:43 +0800
Subject: [PATCH 016/176] Add documentation for bigquery batch retrieval (#428)

* Add documentation for bigquery batch retrieval

* Fix formatting for multiline comments
---
 .../bigquery/BatchRetrievalQueryRunnable.java | 32 ++++++++++++++++
 .../store/bigquery/SubqueryCallable.java      |  4 +-
 .../resources/templates/join_featuresets.sql  |  3 ++
 .../templates/single_featureset_pit_join.sql  | 37 ++++++++++++++++++-
 4 files changed, 72 insertions(+), 4 deletions(-)

diff --git a/serving/src/main/java/feast/serving/store/bigquery/BatchRetrievalQueryRunnable.java b/serving/src/main/java/feast/serving/store/bigquery/BatchRetrievalQueryRunnable.java
index d437294dfc3..e875de35a80 100644
--- a/serving/src/main/java/feast/serving/store/bigquery/BatchRetrievalQueryRunnable.java
+++ b/serving/src/main/java/feast/serving/store/bigquery/BatchRetrievalQueryRunnable.java
@@ -52,6 +52,27 @@
 import java.util.concurrent.TimeUnit;
 import java.util.concurrent.TimeoutException;
 
+/**
+ * BatchRetrievalQueryRunnable is a Runnable for running a BigQuery Feast batch retrieval job async.
+ *
+ * 

It does the following, in sequence: + * + *

1. Retrieve the temporal bounds of the entity dataset provided. This will be used to filter + * the feature set tables when performing the feature retrieval. + * + *

2. For each of the feature sets requested, generate the subquery for doing a point-in-time + * correctness join of the features in the feature set to the entity table. + * + *

3. Run each of the subqueries in parallel and wait for them to complete. If any of the jobs + * are unsuccessful, the thread running the BatchRetrievalQueryRunnable catches the error and + * updates the job database. + * + *

4. When all the subquery jobs are complete, join the outputs of all the subqueries into a + * single table. + * + *

5. Extract the output of the join to a remote file, and write the location of the remote file + * to the job database, and mark the retrieval job as successful. + */ @AutoValue public abstract class BatchRetrievalQueryRunnable implements Runnable { @@ -109,18 +130,22 @@ public abstract static class Builder { @Override public void run() { + // 1. Retrieve the temporal bounds of the entity dataset provided FieldValueList timestampLimits = getTimestampLimits(entityTableName()); + // 2. Generate the subqueries List featureSetQueries = generateQueries(timestampLimits); QueryJobConfiguration queryConfig; try { + // 3 & 4. Run the subqueries in parallel then collect the outputs Job queryJob = runBatchQuery(featureSetQueries); queryConfig = queryJob.getConfiguration(); String exportTableDestinationUri = String.format("%s/%s/*.avro", jobStagingLocation(), feastJobId()); + // 5. Export the table // Hardcode the format to Avro for now ExtractJobConfiguration extractConfig = ExtractJobConfiguration.of( @@ -141,6 +166,7 @@ public void run() { List fileUris = parseOutputFileURIs(); + // 5. Update the job database jobService() .upsert( ServingAPIProto.Job.newBuilder() @@ -181,6 +207,8 @@ Job runBatchQuery(List featureSetQueries) List featureSetInfos = new ArrayList<>(); + // For each of the feature sets requested, start an async job joining the features in that + // feature set to the provided entity table for (int i = 0; i < featureSetQueries.size(); i++) { QueryJobConfiguration queryJobConfig = QueryJobConfiguration.newBuilder(featureSetQueries.get(i)) @@ -197,6 +225,8 @@ Job runBatchQuery(List featureSetQueries) for (int i = 0; i < featureSetQueries.size(); i++) { try { + // Try to retrieve the outputs of all the jobs. The timeout here is a formality; + // a stricter timeout is implemented in the actual SubqueryCallable. FeatureSetInfo featureSetInfo = executorCompletionService.take().get(SUBQUERY_TIMEOUT_SECS, TimeUnit.SECONDS); featureSetInfos.add(featureSetInfo); @@ -218,6 +248,8 @@ Job runBatchQuery(List featureSetQueries) } } + // Generate and run a join query to collect the outputs of all the + // subqueries into a single table. String joinQuery = QueryTemplater.createJoinQuery( featureSetInfos, entityTableColumnNames(), entityTableName()); diff --git a/serving/src/main/java/feast/serving/store/bigquery/SubqueryCallable.java b/serving/src/main/java/feast/serving/store/bigquery/SubqueryCallable.java index e0b8f457986..14026030b42 100644 --- a/serving/src/main/java/feast/serving/store/bigquery/SubqueryCallable.java +++ b/serving/src/main/java/feast/serving/store/bigquery/SubqueryCallable.java @@ -30,8 +30,8 @@ import java.util.concurrent.Callable; /** - * Waits for a bigquery job to complete; when complete, it updates the feature set info with the - * output table name, as well as increments the completed jobs counter in the query job listener. + * Waits for a point-in-time correctness join to complete. On completion, returns a featureSetInfo + * updated with the reference to the table containing the results of the query. */ @AutoValue public abstract class SubqueryCallable implements Callable { diff --git a/serving/src/main/resources/templates/join_featuresets.sql b/serving/src/main/resources/templates/join_featuresets.sql index e57b0c10314..60b7c7d7a12 100644 --- a/serving/src/main/resources/templates/join_featuresets.sql +++ b/serving/src/main/resources/templates/join_featuresets.sql @@ -1,3 +1,6 @@ +/* + Joins the outputs of multiple point-in-time-correctness joins to a single table. + */ WITH joined as ( SELECT * FROM `{{ leftTableName }}` {% for featureSet in featureSets %} diff --git a/serving/src/main/resources/templates/single_featureset_pit_join.sql b/serving/src/main/resources/templates/single_featureset_pit_join.sql index f6678421851..1f4612b3503 100644 --- a/serving/src/main/resources/templates/single_featureset_pit_join.sql +++ b/serving/src/main/resources/templates/single_featureset_pit_join.sql @@ -1,9 +1,24 @@ -WITH union_features AS (SELECT +/* + This query template performs the point-in-time correctness join for a single feature set table + to the provided entity table. + + 1. Concatenate the timestamp and entities from the feature set table with the entity dataset. + Feature values are joined to this table later for improved efficiency. + featureset_timestamp is equal to null in rows from the entity dataset. + */ +WITH union_features AS ( +SELECT + -- uuid is a unique identifier for each row in the entity dataset. Generated by `QueryTemplater.createEntityTableUUIDQuery` uuid, + -- event_timestamp contains the timestamps to join onto event_timestamp, + -- the feature_timestamp, i.e. the latest occurrence of the requested feature relative to the entity_dataset timestamp NULL as {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, + -- created timestamp of the feature at the corresponding feature_timestamp NULL as created_timestamp, + -- select only entities belonging to this feature set {{ featureSet.entities | join(', ')}}, + -- boolean for filtering the dataset later true AS is_entity_table FROM `{{leftTableName}}` UNION ALL @@ -15,7 +30,18 @@ SELECT {{ featureSet.entities | join(', ')}}, false AS is_entity_table FROM `{{projectId}}.{{datasetId}}.{{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}` WHERE event_timestamp <= '{{maxTimestamp}}' AND event_timestamp >= Timestamp_sub(TIMESTAMP '{{ minTimestamp }}', interval {{ featureSet.maxAge }} second) -), joined AS ( +), +/* + 2. Window the data in the unioned dataset, partitioning by entity and ordering by event_timestamp, as + well as is_entity_table. + Within each window, back-fill the feature_timestamp - as a result of this, the null feature_timestamps + in the rows from the entity table should now contain the latest timestamps relative to the row's + event_timestamp. + + For rows where event_timestamp(provided datetime) - feature_timestamp > max age, set the + feature_timestamp to null. + */ +joined AS ( SELECT uuid, event_timestamp, @@ -34,6 +60,10 @@ SELECT FROM union_features WINDOW w AS (PARTITION BY {{ featureSet.entities | join(', ') }} ORDER BY event_timestamp DESC, is_entity_table DESC, created_timestamp DESC ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) ) +/* + 3. Select only the rows from the entity table, and join the features from the original feature set table + to the dataset using the entity values, feature_timestamp, and created_timestamps. + */ LEFT JOIN ( SELECT event_timestamp as {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, @@ -46,6 +76,9 @@ FROM `{{projectId}}.{{datasetId}}.{{ featureSet.project }}_{{ featureSet.name }} ) USING ({{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, created_timestamp, {{ featureSet.entities | join(', ')}}) WHERE is_entity_table ) +/* + 4. Finally, deduplicate the rows by selecting the first occurrence of each entity table row UUID. + */ SELECT k.* FROM ( From eefdc355e3f993adc75f8416a6becafea7ed3511 Mon Sep 17 00:00:00 2001 From: Iain Rauch Date: Sat, 18 Jan 2020 09:07:51 +0000 Subject: [PATCH 017/176] Fix logging (#430) Allow log level to be set via environmental variable. Add ability to set appender type in serving. Remove logback-classic from ingestion as it is a library so should not bring its own impl. Upgrade log4j to 2.12.1 to support objectMessageAsJsonObject. Fix logger config targeting feast package in serving an add same concept for core. --- core/src/main/resources/log4j2.xml | 12 ++++++++---- ingestion/pom.xml | 9 --------- pom.xml | 22 ++++++++++++++++++++++ serving/src/main/resources/log4j2.xml | 14 +++++++++----- 4 files changed, 39 insertions(+), 18 deletions(-) diff --git a/core/src/main/resources/log4j2.xml b/core/src/main/resources/log4j2.xml index 65b3c5aa4bb..efbf7d1f624 100644 --- a/core/src/main/resources/log4j2.xml +++ b/core/src/main/resources/log4j2.xml @@ -22,9 +22,10 @@ %d{yyyy-MM-dd HH:mm:ss.SSS} %5p ${hostName} --- [%15.15t] %-40.40c{1.} : %m%n%ex ${env:LOG_TYPE:-Console} + ${env:LOG_LEVEL:-info} - + @@ -35,8 +36,11 @@ - - - + + + + + + diff --git a/ingestion/pom.xml b/ingestion/pom.xml index e3961d33855..c829674a64d 100644 --- a/ingestion/pom.xml +++ b/ingestion/pom.xml @@ -225,15 +225,6 @@ slf4j-api - - - - ch.qos.logback - logback-classic - 1.2.3 - runtime - - com.github.kstyrc diff --git a/pom.xml b/pom.xml index 939dc8507c7..821d3b72321 100644 --- a/pom.xml +++ b/pom.xml @@ -55,6 +55,8 @@ 2.28.2 0.21.0 + + 2.12.1 @@ -261,6 +263,26 @@ + + org.apache.logging.log4j + log4j-api + ${log4jVersion} + + + org.apache.logging.log4j + log4j-core + ${log4jVersion} + + + org.apache.logging.log4j + log4j-jul + ${log4jVersion} + + + org.apache.logging.log4j + log4j-slf4j-impl + ${log4jVersion} + @@ -392,6 +403,13 @@ org.apache.maven.plugins maven-enforcer-plugin 3.0.0-M2 + + + org.codehaus.mojo + extra-enforcer-rules + 1.2 + + valid-build-environment @@ -401,10 +419,10 @@ - [3.5,4.0) + [3.6,4.0) - [1.8,1.9) + [1.8,11.1) From 761dfff807398573fbfe1e7148cbf5072e29f763 Mon Sep 17 00:00:00 2001 From: Iain Rauch Date: Tue, 11 Feb 2020 09:07:36 +0000 Subject: [PATCH 038/176] Helm Chart Upgrades (#458) Move prometheus-statsd-exporter to toggleable core dependency (default false). Add ingresses for gRPC and HTTP for both core and serving. Refactor ConfigMaps to user Spring profiles rather than manipulating the base application.yaml. Add ability to define and enable arbitrary Spring profiles. Add toggle to enable prometheus scraping in core. Add parameters to change LOG_LEVEL and LOG_TYPE (#430). Add parameter to specify GOOGLE_CLOUD_PROJECT. Allow jar path to be specified (e.g. if using non-standard image). Add missing documentation for Helm parameters. --- infra/charts/feast/README.md | 80 +++++++++++++++- .../prometheus-statsd-exporter/.helmignore | 0 .../prometheus-statsd-exporter/Chart.yaml | 0 .../prometheus-statsd-exporter/README.md | 0 .../templates/NOTES.txt | 0 .../templates/_helpers.tpl | 0 .../templates/config.yaml | 0 .../templates/deployment.yaml | 0 .../templates/pvc.yaml | 0 .../templates/service.yaml | 0 .../templates/serviceaccount.yaml | 0 .../prometheus-statsd-exporter/values.yaml | 0 .../feast/charts/feast-core/requirements.yaml | 8 +- .../charts/feast-core/templates/_ingress.yaml | 68 +++++++++++++ .../feast-core/templates/configmap.yaml | 45 ++++++--- .../feast-core/templates/deployment.yaml | 42 ++++++-- .../charts/feast-core/templates/ingress.yaml | 33 ++----- .../feast/charts/feast-core/values.yaml | 95 +++++++++++++++---- .../charts/feast-serving/requirements.yaml | 3 + .../feast-serving/templates/_helpers.tpl | 7 ++ .../feast-serving/templates/_ingress.yaml | 68 +++++++++++++ .../feast-serving/templates/configmap.yaml | 36 ++++--- .../feast-serving/templates/deployment.yaml | 30 ++++-- .../feast-serving/templates/ingress.yaml | 31 +----- .../feast/charts/feast-serving/values.yaml | 77 +++++++++++---- infra/charts/feast/requirements.lock | 16 +--- infra/charts/feast/requirements.yaml | 2 +- infra/charts/feast/values-demo.yaml | 17 +++- infra/charts/feast/values.yaml | 12 ++- 29 files changed, 510 insertions(+), 160 deletions(-) rename infra/charts/feast/charts/{ => feast-core/charts}/prometheus-statsd-exporter/.helmignore (100%) rename infra/charts/feast/charts/{ => feast-core/charts}/prometheus-statsd-exporter/Chart.yaml (100%) rename infra/charts/feast/charts/{ => feast-core/charts}/prometheus-statsd-exporter/README.md (100%) rename infra/charts/feast/charts/{ => feast-core/charts}/prometheus-statsd-exporter/templates/NOTES.txt (100%) rename infra/charts/feast/charts/{ => feast-core/charts}/prometheus-statsd-exporter/templates/_helpers.tpl (100%) rename infra/charts/feast/charts/{ => feast-core/charts}/prometheus-statsd-exporter/templates/config.yaml (100%) rename infra/charts/feast/charts/{ => feast-core/charts}/prometheus-statsd-exporter/templates/deployment.yaml (100%) rename infra/charts/feast/charts/{ => feast-core/charts}/prometheus-statsd-exporter/templates/pvc.yaml (100%) rename infra/charts/feast/charts/{ => feast-core/charts}/prometheus-statsd-exporter/templates/service.yaml (100%) rename infra/charts/feast/charts/{ => feast-core/charts}/prometheus-statsd-exporter/templates/serviceaccount.yaml (100%) rename infra/charts/feast/charts/{ => feast-core/charts}/prometheus-statsd-exporter/values.yaml (100%) create mode 100644 infra/charts/feast/charts/feast-core/templates/_ingress.yaml create mode 100644 infra/charts/feast/charts/feast-serving/templates/_ingress.yaml diff --git a/infra/charts/feast/README.md b/infra/charts/feast/README.md index ab5321ca865..e93b687f191 100644 --- a/infra/charts/feast/README.md +++ b/infra/charts/feast/README.md @@ -81,17 +81,26 @@ The following table lists the configurable parameters of the Feast chart and the | `feast-core.kafka.topics[0].name` | Default topic name in Kafka| `feast` | `feast-core.kafka.topics[0].replicationFactor` | No of replication factor for the topic| `1` | `feast-core.kafka.topics[0].partitions` | No of partitions for the topic | `1` +| `feast-core.prometheus-statsd-exporter.enabled` | Flag to install Prometheus StatsD Exporter | `false` +| `feast-core.prometheus-statsd-exporter.*` | Refer to this [link](charts/feast-core/charts/prometheus-statsd-exporter/values.yaml | | `feast-core.replicaCount` | No of pods to create | `1` | `feast-core.image.repository` | Repository for Feast Core Docker image | `gcr.io/kf-feast/feast-core` -| `feast-core.image.tag` | Tag for Feast Core Docker image | `0.3.2` +| `feast-core.image.tag` | Tag for Feast Core Docker image | `0.4.4` | `feast-core.image.pullPolicy` | Image pull policy for Feast Core Docker image | `IfNotPresent` +| `feast-core.prometheus.enabled` | Add annotations to enable Prometheus scraping | `false` | `feast-core.application.yaml` | Configuration for Feast Core application | Refer to this [link](charts/feast-core/values.yaml) | `feast-core.springConfigMountPath` | Directory to mount application.yaml | `/etc/feast/feast-core` | `feast-core.gcpServiceAccount.useExistingSecret` | Flag to use existing secret for GCP service account | `false` | `feast-core.gcpServiceAccount.existingSecret.name` | Secret name for the service account | `feast-gcp-service-account` | `feast-core.gcpServiceAccount.existingSecret.key` | Secret key for the service account | `key.json` | `feast-core.gcpServiceAccount.mountPath` | Directory to mount the JSON key file | `/etc/gcloud/service-accounts` +| `feast-core.gcpProjectId` | Project ID to set `GOOGLE_CLOUD_PROJECT` to change default project used by SDKs | `""` +| `feast-core.jarPath` | Path to Jar file in the Docker image | `/opt/feast/feast-core.jar` | `feast-core.jvmOptions` | Options for the JVM | `[]` +| `feast-core.logLevel` | Application logging level | `warn` +| `feast-core.logType` | Application logging type (`JSON` or `Console`) | `JSON` +| `feast-core.springConfigProfiles` | Map of profile name to file content for additional Spring profiles | `{}` +| `feast-core.springConfigProfilesActive` | CSV of profiles to enable from `springConfigProfiles` | `""` | `feast-core.livenessProbe.enabled` | Flag to enable liveness probe | `true` | `feast-core.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | `60` | `feast-core.livenessProbe.periodSeconds` | How often to perform the probe | `10` @@ -109,6 +118,7 @@ The following table lists the configurable parameters of the Feast chart and the | `feast-core.grpc.port` | Kubernetes Service port for GRPC request| `6565` | `feast-core.grpc.targetPort` | Container port for GRPC request| `6565` | `feast-core.resources` | CPU and memory allocation for the pod | `{}` +| `feast-core.ingress` | See *Ingress Parameters* [below](#ingress-parameters) | `{}` | `feast-serving-online.enabled` | Flag to install Feast Online Serving | `true` | `feast-serving-online.redis.enabled` | Flag to install Redis in Feast Serving | `false` | `feast-serving-online.redis.usePassword` | Flag to use password to access Redis | `false` @@ -116,8 +126,9 @@ The following table lists the configurable parameters of the Feast chart and the | `feast-serving-online.core.enabled` | Flag for Feast Serving to use Feast Core in the same Helm release | `true` | `feast-serving-online.replicaCount` | No of pods to create | `1` | `feast-serving-online.image.repository` | Repository for Feast Serving Docker image | `gcr.io/kf-feast/feast-serving` -| `feast-serving-online.image.tag` | Tag for Feast Serving Docker image | `0.3.2` +| `feast-serving-online.image.tag` | Tag for Feast Serving Docker image | `0.4.4` | `feast-serving-online.image.pullPolicy` | Image pull policy for Feast Serving Docker image | `IfNotPresent` +| `feast-serving-online.prometheus.enabled` | Add annotations to enable Prometheus scraping | `true` | `feast-serving-online.application.yaml` | Application configuration for Feast Serving | Refer to this [link](charts/feast-serving/values.yaml) | `feast-serving-online.store.yaml` | Store configuration for Feast Serving | Refer to this [link](charts/feast-serving/values.yaml) | `feast-serving-online.springConfigMountPath` | Directory to mount application.yaml and store.yaml | `/etc/feast/feast-serving` @@ -125,7 +136,13 @@ The following table lists the configurable parameters of the Feast chart and the | `feast-serving-online.gcpServiceAccount.existingSecret.name` | Secret name for the service account | `feast-gcp-service-account` | `feast-serving-online.gcpServiceAccount.existingSecret.key` | Secret key for the service account | `key.json` | `feast-serving-online.gcpServiceAccount.mountPath` | Directory to mount the JSON key file | `/etc/gcloud/service-accounts` +| `feast-serving-online.gcpProjectId` | Project ID to set `GOOGLE_CLOUD_PROJECT` to change default project used by SDKs | `""` +| `feast-serving-online.jarPath` | Path to Jar file in the Docker image | `/opt/feast/feast-serving.jar` | `feast-serving-online.jvmOptions` | Options for the JVM | `[]` +| `feast-serving-online.logLevel` | Application logging level | `warn` +| `feast-serving-online.logType` | Application logging type (`JSON` or `Console`) | `JSON` +| `feast-serving-online.springConfigProfiles` | Map of profile name to file content for additional Spring profiles | `{}` +| `feast-serving-online.springConfigProfilesActive` | CSV of profiles to enable from `springConfigProfiles` | `""` | `feast-serving-online.livenessProbe.enabled` | Flag to enable liveness probe | `true` | `feast-serving-online.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | `60` | `feast-serving-online.livenessProbe.periodSeconds` | How often to perform the probe | `10` @@ -143,6 +160,7 @@ The following table lists the configurable parameters of the Feast chart and the | `feast-serving-online.grpc.port` | Kubernetes Service port for GRPC request| `6566` | `feast-serving-online.grpc.targetPort` | Container port for GRPC request| `6566` | `feast-serving-online.resources` | CPU and memory allocation for the pod | `{}` +| `feast-serving-online.ingress` | See *Ingress Parameters* [below](#ingress-parameters) | `{}` | `feast-serving-batch.enabled` | Flag to install Feast Batch Serving | `true` | `feast-serving-batch.redis.enabled` | Flag to install Redis in Feast Serving | `false` | `feast-serving-batch.redis.usePassword` | Flag to use password to access Redis | `false` @@ -150,8 +168,9 @@ The following table lists the configurable parameters of the Feast chart and the | `feast-serving-batch.core.enabled` | Flag for Feast Serving to use Feast Core in the same Helm release | `true` | `feast-serving-batch.replicaCount` | No of pods to create | `1` | `feast-serving-batch.image.repository` | Repository for Feast Serving Docker image | `gcr.io/kf-feast/feast-serving` -| `feast-serving-batch.image.tag` | Tag for Feast Serving Docker image | `0.3.2` +| `feast-serving-batch.image.tag` | Tag for Feast Serving Docker image | `0.4.4` | `feast-serving-batch.image.pullPolicy` | Image pull policy for Feast Serving Docker image | `IfNotPresent` +| `feast-serving-batch.prometheus.enabled` | Add annotations to enable Prometheus scraping | `true` | `feast-serving-batch.application.yaml` | Application configuration for Feast Serving | Refer to this [link](charts/feast-serving/values.yaml) | `feast-serving-batch.store.yaml` | Store configuration for Feast Serving | Refer to this [link](charts/feast-serving/values.yaml) | `feast-serving-batch.springConfigMountPath` | Directory to mount application.yaml and store.yaml | `/etc/feast/feast-serving` @@ -159,7 +178,13 @@ The following table lists the configurable parameters of the Feast chart and the | `feast-serving-batch.gcpServiceAccount.existingSecret.name` | Secret name for the service account | `feast-gcp-service-account` | `feast-serving-batch.gcpServiceAccount.existingSecret.key` | Secret key for the service account | `key.json` | `feast-serving-batch.gcpServiceAccount.mountPath` | Directory to mount the JSON key file | `/etc/gcloud/service-accounts` +| `feast-serving-batch.gcpProjectId` | Project ID to set `GOOGLE_CLOUD_PROJECT` to change default project used by SDKs | `""` +| `feast-serving-batch.jarPath` | Path to Jar file in the Docker image | `/opt/feast/feast-serving.jar` | `feast-serving-batch.jvmOptions` | Options for the JVM | `[]` +| `feast-serving-batch.logLevel` | Application logging level | `warn` +| `feast-serving-batch.logType` | Application logging type (`JSON` or `Console`) | `JSON` +| `feast-serving-batch.springConfigProfiles` | Map of profile name to file content for additional Spring profiles | `{}` +| `feast-serving-batch.springConfigProfilesActive` | CSV of profiles to enable from `springConfigProfiles` | `""` | `feast-serving-batch.livenessProbe.enabled` | Flag to enable liveness probe | `true` | `feast-serving-batch.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | `60` | `feast-serving-batch.livenessProbe.periodSeconds` | How often to perform the probe | `10` @@ -176,4 +201,51 @@ The following table lists the configurable parameters of the Feast chart and the | `feast-serving-batch.http.targetPort` | Container port for HTTP request | `8080` | `feast-serving-batch.grpc.port` | Kubernetes Service port for GRPC request| `6566` | `feast-serving-batch.grpc.targetPort` | Container port for GRPC request| `6566` -| `feast-serving-batch.resources` | CPU and memory allocation for the pod | `{}` \ No newline at end of file +| `feast-serving-batch.resources` | CPU and memory allocation for the pod | `{}` +| `feast-serving-batch.ingress` | See *Ingress Parameters* [below](#ingress-parameters) | `{}` + +## Ingress Parameters + +The following table lists the configurable parameters of the ingress section for each Feast module. + +Note, there are two ingresses available for each module - `grpc` and `http`. + +| Parameter | Description | Default +| ----------------------------- | ----------- | ------- +| `ingress.grcp.enabled` | Enables an ingress (endpoint) for the gRPC server | `false` +| `ingress.grcp.*` | See below | +| `ingress.http.enabled` | Enables an ingress (endpoint) for the HTTP server | `false` +| `ingress.http.*` | See below | +| `ingress.*.class` | Value for `kubernetes.io/ingress.class` | `nginx` +| `ingress.*.hosts` | List of host-names for the ingress | `[]` +| `ingress.*.annotations` | Additional ingress annotations | `{}` +| `ingress.*.https.enabled` | Add a tls section to the ingress | `true` +| `ingress.*.https.secretNames` | Map of hostname to TLS secret name | `{}` If not specified, defaults to `domain-tld-tls` e.g. `feast.example.com` uses secret `example-com-tls` +| `ingress.*.auth.enabled` | Enable auth on the ingress (only applicable for `nginx` type | `false` +| `ingress.*.auth.signinHost` | External hostname of the OAuth2 proxy to use | First item in `ingress.hosts`, replacing the sub-domain with 'auth' e.g. `feast.example.com` uses `auth.example.com` +| `ingress.*.auth.authUrl` | Internal URI to internal auth endpoint | `http://auth-server.auth-ns.svc.cluster.local/auth` +| `ingress.*.whitelist` | Subnet masks to whitelist (i.e. value for `nginx.ingress.kubernetes.io/whitelist-source-range`) | `"""` + +To enable all the ingresses will a config like the following (while also adding the hosts etc): + +```yaml +feast-core: + ingress: + grpc: + enabled: true + http: + enabled: true +feast-serving-online: + ingress: + grpc: + enabled: true + http: + enabled: true +feast-serving-batch: + ingress: + grpc: + enabled: true + http: + enabled: true +``` + diff --git a/infra/charts/feast/charts/prometheus-statsd-exporter/.helmignore b/infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/.helmignore similarity index 100% rename from infra/charts/feast/charts/prometheus-statsd-exporter/.helmignore rename to infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/.helmignore diff --git a/infra/charts/feast/charts/prometheus-statsd-exporter/Chart.yaml b/infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/Chart.yaml similarity index 100% rename from infra/charts/feast/charts/prometheus-statsd-exporter/Chart.yaml rename to infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/Chart.yaml diff --git a/infra/charts/feast/charts/prometheus-statsd-exporter/README.md b/infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/README.md similarity index 100% rename from infra/charts/feast/charts/prometheus-statsd-exporter/README.md rename to infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/README.md diff --git a/infra/charts/feast/charts/prometheus-statsd-exporter/templates/NOTES.txt b/infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/templates/NOTES.txt similarity index 100% rename from infra/charts/feast/charts/prometheus-statsd-exporter/templates/NOTES.txt rename to infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/templates/NOTES.txt diff --git a/infra/charts/feast/charts/prometheus-statsd-exporter/templates/_helpers.tpl b/infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/templates/_helpers.tpl similarity index 100% rename from infra/charts/feast/charts/prometheus-statsd-exporter/templates/_helpers.tpl rename to infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/templates/_helpers.tpl diff --git a/infra/charts/feast/charts/prometheus-statsd-exporter/templates/config.yaml b/infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/templates/config.yaml similarity index 100% rename from infra/charts/feast/charts/prometheus-statsd-exporter/templates/config.yaml rename to infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/templates/config.yaml diff --git a/infra/charts/feast/charts/prometheus-statsd-exporter/templates/deployment.yaml b/infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/templates/deployment.yaml similarity index 100% rename from infra/charts/feast/charts/prometheus-statsd-exporter/templates/deployment.yaml rename to infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/templates/deployment.yaml diff --git a/infra/charts/feast/charts/prometheus-statsd-exporter/templates/pvc.yaml b/infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/templates/pvc.yaml similarity index 100% rename from infra/charts/feast/charts/prometheus-statsd-exporter/templates/pvc.yaml rename to infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/templates/pvc.yaml diff --git a/infra/charts/feast/charts/prometheus-statsd-exporter/templates/service.yaml b/infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/templates/service.yaml similarity index 100% rename from infra/charts/feast/charts/prometheus-statsd-exporter/templates/service.yaml rename to infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/templates/service.yaml diff --git a/infra/charts/feast/charts/prometheus-statsd-exporter/templates/serviceaccount.yaml b/infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/templates/serviceaccount.yaml similarity index 100% rename from infra/charts/feast/charts/prometheus-statsd-exporter/templates/serviceaccount.yaml rename to infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/templates/serviceaccount.yaml diff --git a/infra/charts/feast/charts/prometheus-statsd-exporter/values.yaml b/infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/values.yaml similarity index 100% rename from infra/charts/feast/charts/prometheus-statsd-exporter/values.yaml rename to infra/charts/feast/charts/feast-core/charts/prometheus-statsd-exporter/values.yaml diff --git a/infra/charts/feast/charts/feast-core/requirements.yaml b/infra/charts/feast/charts/feast-core/requirements.yaml index efe9fec508a..ef1e39a7d0f 100644 --- a/infra/charts/feast/charts/feast-core/requirements.yaml +++ b/infra/charts/feast/charts/feast-core/requirements.yaml @@ -6,4 +6,10 @@ dependencies: - name: kafka version: 0.20.1 repository: "@incubator" - condition: kafka.enabled \ No newline at end of file + condition: kafka.enabled +- name: common + version: 0.0.5 + repository: "@incubator" +- name: prometheus-statsd-exporter + version: 0.1.2 + condition: prometheus-statsd-exporter.enabled \ No newline at end of file diff --git a/infra/charts/feast/charts/feast-core/templates/_ingress.yaml b/infra/charts/feast/charts/feast-core/templates/_ingress.yaml new file mode 100644 index 00000000000..5bed6df0470 --- /dev/null +++ b/infra/charts/feast/charts/feast-core/templates/_ingress.yaml @@ -0,0 +1,68 @@ +{{- /* +This takes an array of three values: +- the top context +- the feast component +- the service protocol +- the ingress context +*/ -}} +{{- define "feast.ingress" -}} +{{- $top := (index . 0) -}} +{{- $component := (index . 1) -}} +{{- $protocol := (index . 2) -}} +{{- $ingressValues := (index . 3) -}} +apiVersion: extensions/v1beta1 +kind: Ingress +{{ include "feast.ingress.metadata" . }} +spec: + rules: + {{- range $host := $ingressValues.hosts }} + - host: {{ $host }} + http: + paths: + - path: / + backend: + serviceName: {{ include (printf "feast-%s.fullname" $component) $top }} + servicePort: {{ index $top.Values "service" $protocol "port" }} + {{- end }} +{{- if $ingressValues.https.enabled }} + tls: + {{- range $host := $ingressValues.hosts }} + - secretName: {{ index $ingressValues.https.secretNames $host | default (splitList "." $host | rest | join "-" | printf "%s-tls") }} + hosts: + - {{ $host }} + {{- end }} +{{- end -}} +{{- end -}} + +{{- define "feast.ingress.metadata" -}} +{{- $commonMetadata := fromYaml (include "common.metadata" (first .)) }} +{{- $overrides := fromYaml (include "feast.ingress.metadata-overrides" .) -}} +{{- toYaml (merge $overrides $commonMetadata) -}} +{{- end -}} + +{{- define "feast.ingress.metadata-overrides" -}} +{{- $top := (index . 0) -}} +{{- $component := (index . 1) -}} +{{- $protocol := (index . 2) -}} +{{- $ingressValues := (index . 3) -}} +{{- $commonFullname := include "common.fullname" $top }} +metadata: + name: {{ $commonFullname }}-{{ $component }}-{{ $protocol }} + annotations: + kubernetes.io/ingress.class: {{ $ingressValues.class | quote }} + {{- if (and (eq $ingressValues.class "nginx") $ingressValues.auth.enabled) }} + nginx.ingress.kubernetes.io/auth-url: {{ $ingressValues.auth.authUrl | quote }} + nginx.ingress.kubernetes.io/auth-response-headers: "x-auth-request-email, x-auth-request-user" + nginx.ingress.kubernetes.io/auth-signin: "https://{{ $ingressValues.auth.signinHost | default (splitList "." (index $ingressValues.hosts 0) | rest | join "." | printf "auth.%s")}}/oauth2/start?rd=/r/$host/$request_uri" + {{- end }} + {{- if (and (eq $ingressValues.class "nginx") $ingressValues.whitelist) }} + nginx.ingress.kubernetes.io/whitelist-source-range: {{ $ingressValues.whitelist | quote -}} + {{- end }} + {{- if (and (eq $ingressValues.class "nginx") (eq $protocol "grpc") ) }} + # TODO: Allow choice of GRPC/GRPCS + nginx.ingress.kubernetes.io/backend-protocol: "GRPC" + {{- end }} + {{- if $ingressValues.annotations -}} + {{ include "common.annote" $ingressValues.annotations | indent 4 }} + {{- end }} +{{- end -}} diff --git a/infra/charts/feast/charts/feast-core/templates/configmap.yaml b/infra/charts/feast/charts/feast-core/templates/configmap.yaml index 68dc45c0571..da45cad5bdf 100644 --- a/infra/charts/feast/charts/feast-core/templates/configmap.yaml +++ b/infra/charts/feast/charts/feast-core/templates/configmap.yaml @@ -11,22 +11,43 @@ metadata: heritage: {{ .Release.Service }} data: application.yaml: | -{{- $config := index .Values "application.yaml"}} +{{- toYaml (index .Values "application.yaml") | nindent 4 }} {{- if .Values.postgresql.enabled }} -{{- $datasource := dict "url" (printf "jdbc:postgresql://%s:%s/%s" (printf "%s-postgresql" .Release.Name) (.Values.postgresql.service.port | toString) (.Values.postgresql.postgresqlDatabase)) "driverClassName" "org.postgresql.Driver" }} -{{- $newConfig := dict "spring" (dict "datasource" $datasource) }} -{{- $config := mergeOverwrite $config $newConfig }} + application-bundled-postgresql.yaml: | + spring: + datasource: + url: {{ printf "jdbc:postgresql://%s:%s/%s" (printf "%s-postgresql" .Release.Name) (.Values.postgresql.service.port | toString) (.Values.postgresql.postgresqlDatabase) }} + driverClassName: org.postgresql.Driver {{- end }} -{{- if .Values.kafka.enabled }} -{{- $topic := index .Values.kafka.topics 0 }} -{{- $options := dict "topic" $topic.name "replicationFactor" $topic.replicationFactor "partitions" $topic.partitions }} -{{- if not .Values.kafka.external.enabled }} -{{- $_ := set $options "bootstrapServers" (printf "%s:9092" (printf "%s-kafka" .Release.Name)) }} +{{ if .Values.kafka.enabled }} + {{- $topic := index .Values.kafka.topics 0 }} + application-bundled-kafka.yaml: | + feast: + stream: + type: kafka + options: + topic: {{ $topic.name | quote }} + replicationFactor: {{ $topic.replicationFactor }} + partitions: {{ $topic.partitions }} + {{- if not .Values.kafka.external.enabled }} + bootstrapServers: {{ printf "%s:9092" (printf "%s-kafka" .Release.Name) }} + {{- end }} {{- end }} -{{- $newConfig := dict "feast" (dict "stream" (dict "type" "kafka" "options" $options))}} -{{- $config := mergeOverwrite $config $newConfig }} + +{{- if (index .Values "prometheus-statsd-exporter" "enabled" )}} + application-bundled-statsd.yaml: | + feast: + jobs: + metrics: + enabled: true + type: statsd + host: prometheus-statsd-exporter + port: 9125 {{- end }} -{{- toYaml $config | nindent 4 }} +{{- range $name, $content := .Values.springConfigProfiles }} + application-{{ $name }}.yaml: | +{{- toYaml $content | nindent 4 }} +{{- end }} diff --git a/infra/charts/feast/charts/feast-core/templates/deployment.yaml b/infra/charts/feast/charts/feast-core/templates/deployment.yaml index 0671d9574b3..df834b6749e 100644 --- a/infra/charts/feast/charts/feast-core/templates/deployment.yaml +++ b/infra/charts/feast/charts/feast-core/templates/deployment.yaml @@ -18,6 +18,13 @@ spec: release: {{ .Release.Name }} template: metadata: + {{- if .Values.prometheus.enabled }} + annotations: + {{ $config := index .Values "application.yaml" }} + prometheus.io/path: /metrics + prometheus.io/port: "{{ $config.server.port }}" + prometheus.io/scrape: "true" + {{- end }} labels: app: {{ template "feast-core.name" . }} component: core @@ -42,7 +49,7 @@ spec: - name: {{ .Chart.Name }} image: '{{ .Values.image.repository }}:{{ required "No .image.tag found. This must be provided as input." .Values.image.tag }}' imagePullPolicy: {{ .Values.image.pullPolicy }} - + volumeMounts: - name: {{ template "feast-core.fullname" . }}-config mountPath: "{{ .Values.springConfigMountPath }}" @@ -53,31 +60,48 @@ spec: {{- end }} env: + - name: LOG_TYPE + value: {{ .Values.logType | quote }} + - name: LOG_LEVEL + value: {{ .Values.logLevel | quote }} + {{- if .Values.postgresql.enabled }} - name: SPRING_DATASOURCE_USERNAME - value: {{ .Values.postgresql.postgresqlUsername }} + value: {{ .Values.postgresql.postgresqlUsername | quote }} - name: SPRING_DATASOURCE_PASSWORD - value: {{ .Values.postgresql.postgresqlPassword }} + value: {{ .Values.postgresql.postgresqlPassword | quote }} {{- end }} {{- if .Values.gcpServiceAccount.useExistingSecret }} - name: GOOGLE_APPLICATION_CREDENTIALS value: {{ .Values.gcpServiceAccount.mountPath }}/{{ .Values.gcpServiceAccount.existingSecret.key }} {{- end }} + {{- if .Values.gcpProjectId }} + - name: GOOGLE_CLOUD_PROJECT + value: {{ .Values.gcpProjectId | quote }} + {{- end }} command: - java {{- range .Values.jvmOptions }} - - {{ . }} + - {{ . | quote }} + {{- end }} + - -jar + - {{ .Values.jarPath | quote }} + - "--spring.config.location=file:{{ .Values.springConfigMountPath }}/" + {{- $profilesArray := splitList "," .Values.springConfigProfilesActive -}} + {{- $profilesArray = append $profilesArray (.Values.postgresql.enabled | ternary "bundled-postgresql" "") -}} + {{- $profilesArray = append $profilesArray (.Values.kafka.enabled | ternary "bundled-kafka" "") -}} + {{- $profilesArray = append $profilesArray (index .Values "prometheus-statsd-exporter" "enabled" | ternary "bundled-statsd" "") -}} + {{- $profilesArray = compact $profilesArray -}} + {{- if $profilesArray }} + - "--spring.profiles.active={{ join "," $profilesArray }}" {{- end }} - - -jar - - /opt/feast/feast-core.jar - - "--spring.config.location=file:{{ .Values.springConfigMountPath }}/application.yaml" ports: - name: http containerPort: {{ .Values.service.http.targetPort }} - - name: grpc + - name: grpc containerPort: {{ .Values.service.grpc.targetPort }} {{- if .Values.livenessProbe.enabled }} @@ -103,6 +127,6 @@ spec: timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }} failureThreshold: {{ .Values.readinessProbe.failureThreshold }} {{- end }} - + resources: {{- toYaml .Values.resources | nindent 10 }} diff --git a/infra/charts/feast/charts/feast-core/templates/ingress.yaml b/infra/charts/feast/charts/feast-core/templates/ingress.yaml index 86fc2d3f175..7f453e1a75f 100644 --- a/infra/charts/feast/charts/feast-core/templates/ingress.yaml +++ b/infra/charts/feast/charts/feast-core/templates/ingress.yaml @@ -1,28 +1,7 @@ -{{- if .Values.ingress.enabled -}} -{{- $fullName := include "feast-core.fullname" . -}} -apiVersion: extensions/v1beta1 -kind: Ingress -metadata: - name: {{ $fullName }} - labels: - app: {{ template "feast-core.name" . }} - chart: {{ .Chart.Name }}-{{ .Chart.Version }} - component: core - heritage: {{ .Release.Service }} - release: {{ .Release.Name }} - annotations: -{{- with .Values.ingress.annotations }} -{{ toYaml . | indent 4 }} +{{- if .Values.ingress.http.enabled -}} +{{ template "feast.ingress" (list . "core" "http" .Values.ingress.http) }} +{{- end }} +--- +{{ if .Values.ingress.grpc.enabled -}} +{{ template "feast.ingress" (list . "core" "grpc" .Values.ingress.grpc) }} {{- end }} -spec: - rules: - {{- range .Values.ingress.hosts }} - - host: {{ .host | quote }} - http: - paths: - - path: / - backend: - serviceName: {{ $fullName }} - servicePort: {{ .port | quote }} - {{- end }} -{{- end }} \ No newline at end of file diff --git a/infra/charts/feast/charts/feast-core/values.yaml b/infra/charts/feast/charts/feast-core/values.yaml index f746bc96ead..077906dc35d 100644 --- a/infra/charts/feast/charts/feast-core/values.yaml +++ b/infra/charts/feast/charts/feast-core/values.yaml @@ -1,12 +1,15 @@ -# postgresql configures Postgresql that is installed as part of Feast Core. +# ============================================================ +# Bundled PostgreSQL +# ============================================================ + # Refer to https://github.com/helm/charts/tree/c42002a21abf8eff839ff1d2382152bde2bbe596/stable/postgresql # for additional configuration. postgresql: # enabled specifies whether Postgresql should be installed as part of Feast Core. # - # Feast Core requires a database to store data such as the created FeatureSets + # Feast Core requires a database to store data such as the created FeatureSets # and job statuses. If enabled, the database and service port specified below - # will override "spring.datasource.url" value in application.yaml. The + # will override "spring.datasource.url" value in application.yaml. The # username and password will also be set as environment variables that will # override "spring.datasource.username/password" in application.yaml. enabled: true @@ -20,12 +23,15 @@ postgresql: # port is the TCP port that Postgresql will listen to port: 5432 -# kafka configures Kafka that is installed as part of Feast Core. +# ============================================================ +# Bundled Kafka +# ============================================================ + # Refer to https://github.com/helm/charts/tree/c42002a21abf8eff839ff1d2382152bde2bbe596/incubator/kafka # for additional configuration. kafka: # enabled specifies whether Kafka should be installed as part of Feast Core. - # + # # Feast Core requires a Kafka instance to be set as the default source for # FeatureRows. If enabled, "feast.stream" option in application.yaml will # be overridden by this installed Kafka configuration. @@ -36,6 +42,18 @@ kafka: replicationFactor: 1 partitions: 1 + +# ============================================================ +# Bundled Prometheus StatsD Exporter +# ============================================================ + +prometheus-statsd-exporter: + enabled: false + +# ============================================================ +# Feast Core +# ============================================================ + # replicaCount is the number of pods that will be created. replicaCount: 1 @@ -44,13 +62,18 @@ image: repository: gcr.io/kf-feast/feast-core pullPolicy: IfNotPresent +# Add prometheus scraping annotations to the Pod metadata. +# If enabled, you must also ensure server.port is specified under application.yaml +prometheus: + enabled: false + # application.yaml is the main configuration for Feast Core application. -# +# # Feast Core is a Spring Boot app which uses this yaml configuration file. # Refer to https://github.com/gojek/feast/blob/79eb4ab5fa3d37102c1dca9968162a98690526ba/core/src/main/resources/application.yml # for a complete list and description of the configuration. # -# Note that some properties defined in application.yaml may be overriden by +# Note that some properties defined in application.yaml may be overriden by # Helm under certain conditions. For example, if postgresql and kafka dependencies # are enabled. application.yaml: @@ -96,7 +119,14 @@ application.yaml: host: localhost port: 8125 -# springConfigMountPath is the directory path where application.yaml will be +springConfigProfiles: {} +# db: | +# spring: +# datasource: +# driverClassName: org.postgresql.Driver +# url: jdbc:postgresql://${DB_HOST:127.0.0.1}:${DB_PORT:5432}/${DB_DATABASE:postgres} +springConfigProfilesActive: "" +# springConfigMountPath is the directory path where application.yaml will be # mounted in the container. springConfigMountPath: /etc/feast/feast-core @@ -107,7 +137,7 @@ gcpServiceAccount: useExistingSecret: false existingSecret: # name is the secret name of the existing secret for the service account. - name: feast-gcp-service-account + name: feast-gcp-service-account # key is the secret key of the existing secret for the service account. # key is normally derived from the file name of the JSON key file. key: key.json @@ -115,19 +145,29 @@ gcpServiceAccount: # the value of "existingSecret.key" is file name of the service account file. mountPath: /etc/gcloud/service-accounts -# jvmOptions are options that will be passed to the Java Virtual Machine (JVM) +# Project ID picked up by the Cloud SDK (e.g. BigQuery run against this project) +gcpProjectId: "" + +# Path to Jar file in the Docker image. +# If you are using gcr.io/kf-feast/feast-core this should not need to be changed +jarPath: /opt/feast/feast-core.jar + +# jvmOptions are options that will be passed to the Java Virtual Machine (JVM) # running Feast Core. -# +# # For example, it is good practice to set min and max heap size in JVM. # https://stackoverflow.com/questions/6902135/side-effect-for-increasing-maxpermsize-and-max-heap-size # # Refer to https://docs.oracle.com/cd/E22289_01/html/821-1274/configuring-the-default-jvm-and-java-arguments.html # to see other JVM options that can be set. # -# jvmOptions: -# - -Xms1024m +jvmOptions: [] +# - -Xms1024m # - -Xmx1024m +logType: JSON +logLevel: warn + livenessProbe: enabled: true initialDelaySeconds: 60 @@ -162,12 +202,29 @@ service: # nodePort: ingress: - enabled: false - annotations: {} - # kubernetes.io/ingress.class: nginx - hosts: - # - host: chart-example.local - # port: http + grpc: + enabled: false + class: nginx + hosts: [] + annotations: {} + https: + enabled: true + secretNames: {} + whitelist: "" + auth: + enabled: false + http: + enabled: false + class: nginx + hosts: [] + annotations: {} + https: + enabled: true + secretNames: {} + whitelist: "" + auth: + enabled: false + authUrl: http://auth-server.auth-ns.svc.cluster.local/auth resources: {} # We usually recommend not to specify default resources and to leave this as a conscious diff --git a/infra/charts/feast/charts/feast-serving/requirements.yaml b/infra/charts/feast/charts/feast-serving/requirements.yaml index fa4c1df4c10..2cee3f81494 100644 --- a/infra/charts/feast/charts/feast-serving/requirements.yaml +++ b/infra/charts/feast/charts/feast-serving/requirements.yaml @@ -3,3 +3,6 @@ dependencies: version: 9.5.0 repository: "@stable" condition: redis.enabled +- name: common + version: 0.0.5 + repository: "@incubator" diff --git a/infra/charts/feast/charts/feast-serving/templates/_helpers.tpl b/infra/charts/feast/charts/feast-serving/templates/_helpers.tpl index 49abb6b8e50..ab670cc8cc7 100644 --- a/infra/charts/feast/charts/feast-serving/templates/_helpers.tpl +++ b/infra/charts/feast/charts/feast-serving/templates/_helpers.tpl @@ -43,3 +43,10 @@ app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} {{- end }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end -}} + +{{/* +Helpers +*/}} +{{- define "bq_store_and_no_job_options" -}} +{{ and (eq (index .Values "store.yaml" "type") "BIGQUERY") (empty (index .Values "application.yaml" "feast" "jobs" "store-options")) }} +{{- end -}} diff --git a/infra/charts/feast/charts/feast-serving/templates/_ingress.yaml b/infra/charts/feast/charts/feast-serving/templates/_ingress.yaml new file mode 100644 index 00000000000..5bed6df0470 --- /dev/null +++ b/infra/charts/feast/charts/feast-serving/templates/_ingress.yaml @@ -0,0 +1,68 @@ +{{- /* +This takes an array of three values: +- the top context +- the feast component +- the service protocol +- the ingress context +*/ -}} +{{- define "feast.ingress" -}} +{{- $top := (index . 0) -}} +{{- $component := (index . 1) -}} +{{- $protocol := (index . 2) -}} +{{- $ingressValues := (index . 3) -}} +apiVersion: extensions/v1beta1 +kind: Ingress +{{ include "feast.ingress.metadata" . }} +spec: + rules: + {{- range $host := $ingressValues.hosts }} + - host: {{ $host }} + http: + paths: + - path: / + backend: + serviceName: {{ include (printf "feast-%s.fullname" $component) $top }} + servicePort: {{ index $top.Values "service" $protocol "port" }} + {{- end }} +{{- if $ingressValues.https.enabled }} + tls: + {{- range $host := $ingressValues.hosts }} + - secretName: {{ index $ingressValues.https.secretNames $host | default (splitList "." $host | rest | join "-" | printf "%s-tls") }} + hosts: + - {{ $host }} + {{- end }} +{{- end -}} +{{- end -}} + +{{- define "feast.ingress.metadata" -}} +{{- $commonMetadata := fromYaml (include "common.metadata" (first .)) }} +{{- $overrides := fromYaml (include "feast.ingress.metadata-overrides" .) -}} +{{- toYaml (merge $overrides $commonMetadata) -}} +{{- end -}} + +{{- define "feast.ingress.metadata-overrides" -}} +{{- $top := (index . 0) -}} +{{- $component := (index . 1) -}} +{{- $protocol := (index . 2) -}} +{{- $ingressValues := (index . 3) -}} +{{- $commonFullname := include "common.fullname" $top }} +metadata: + name: {{ $commonFullname }}-{{ $component }}-{{ $protocol }} + annotations: + kubernetes.io/ingress.class: {{ $ingressValues.class | quote }} + {{- if (and (eq $ingressValues.class "nginx") $ingressValues.auth.enabled) }} + nginx.ingress.kubernetes.io/auth-url: {{ $ingressValues.auth.authUrl | quote }} + nginx.ingress.kubernetes.io/auth-response-headers: "x-auth-request-email, x-auth-request-user" + nginx.ingress.kubernetes.io/auth-signin: "https://{{ $ingressValues.auth.signinHost | default (splitList "." (index $ingressValues.hosts 0) | rest | join "." | printf "auth.%s")}}/oauth2/start?rd=/r/$host/$request_uri" + {{- end }} + {{- if (and (eq $ingressValues.class "nginx") $ingressValues.whitelist) }} + nginx.ingress.kubernetes.io/whitelist-source-range: {{ $ingressValues.whitelist | quote -}} + {{- end }} + {{- if (and (eq $ingressValues.class "nginx") (eq $protocol "grpc") ) }} + # TODO: Allow choice of GRPC/GRPCS + nginx.ingress.kubernetes.io/backend-protocol: "GRPC" + {{- end }} + {{- if $ingressValues.annotations -}} + {{ include "common.annote" $ingressValues.annotations | indent 4 }} + {{- end }} +{{- end -}} diff --git a/infra/charts/feast/charts/feast-serving/templates/configmap.yaml b/infra/charts/feast/charts/feast-serving/templates/configmap.yaml index 0ec80252c16..934216a9d5f 100644 --- a/infra/charts/feast/charts/feast-serving/templates/configmap.yaml +++ b/infra/charts/feast/charts/feast-serving/templates/configmap.yaml @@ -11,37 +11,43 @@ metadata: heritage: {{ .Release.Service }} data: application.yaml: | -{{- $config := index .Values "application.yaml" }} +{{- toYaml (index .Values "application.yaml") | nindent 4 }} {{- if .Values.core.enabled }} -{{- $newConfig := dict "feast" (dict "core-host" (printf "%s-feast-core" .Release.Name)) }} -{{- $config := mergeOverwrite $config $newConfig }} + application-bundled-core.yaml: | + feast: + core-host: {{ printf "%s-feast-core" .Release.Name }} {{- end }} -{{- $store := index .Values "store.yaml" }} -{{- if and (eq $store.type "BIGQUERY") (not (hasKey $config.feast.jobs "store-options")) }} -{{- $jobStore := dict "host" (printf "%s-redis-headless" .Release.Name) "port" 6379 }} -{{- $newConfig := dict "feast" (dict "jobs" (dict "store-options" $jobStore)) }} -{{- $config := mergeOverwrite $config $newConfig }} +{{- if eq (include "bq_store_and_no_job_options" .) "true" }} + application-bundled-redis.yaml: | + feast: + jobs: + store-options: + host: {{ printf "%s-redis-headless" .Release.Name }} + port: 6379 {{- end }} -{{- toYaml $config | nindent 4 }} - store.yaml: | -{{- $config := index .Values "store.yaml"}} +{{- $store := index .Values "store.yaml"}} -{{- if and .Values.redis.enabled (eq $config.type "REDIS") }} +{{- if and .Values.redis.enabled (eq $store.type "REDIS") }} {{- if eq .Values.redis.master.service.type "ClusterIP" }} {{- $newConfig := dict "redis_config" (dict "host" (printf "%s-redis-headless" .Release.Name) "port" .Values.redis.redisPort) }} -{{- $config := mergeOverwrite $config $newConfig }} +{{- $config := mergeOverwrite $store $newConfig }} {{- end }} {{- if and (eq .Values.redis.master.service.type "LoadBalancer") (not (empty .Values.redis.master.service.loadBalancerIP)) }} {{- $newConfig := dict "redis_config" (dict "host" .Values.redis.master.service.loadBalancerIP "port" .Values.redis.redisPort) }} -{{- $config := mergeOverwrite $config $newConfig }} +{{- $config := mergeOverwrite $store $newConfig }} {{- end }} {{- end }} -{{- toYaml $config | nindent 4 }} +{{- toYaml $store | nindent 4 }} + +{{- range $name, $content := .Values.springConfigProfiles }} + application-{{ $name }}.yaml: | +{{- toYaml $content | nindent 4 }} +{{- end }} diff --git a/infra/charts/feast/charts/feast-serving/templates/deployment.yaml b/infra/charts/feast/charts/feast-serving/templates/deployment.yaml index e6824a23465..64dd3955d0c 100644 --- a/infra/charts/feast/charts/feast-serving/templates/deployment.yaml +++ b/infra/charts/feast/charts/feast-serving/templates/deployment.yaml @@ -49,7 +49,7 @@ spec: - name: {{ .Chart.Name }} image: '{{ .Values.image.repository }}:{{ required "No .image.tag found. This must be provided as input." .Values.image.tag }}' imagePullPolicy: {{ .Values.image.pullPolicy }} - + volumeMounts: - name: {{ template "feast-serving.fullname" . }}-config mountPath: "{{ .Values.springConfigMountPath }}" @@ -60,24 +60,40 @@ spec: {{- end }} env: + - name: LOG_TYPE + value: {{ .Values.logType | quote }} + - name: LOG_LEVEL + value: {{ .Values.logLevel | quote }} + {{- if .Values.gcpServiceAccount.useExistingSecret }} - name: GOOGLE_APPLICATION_CREDENTIALS value: {{ .Values.gcpServiceAccount.mountPath }}/{{ .Values.gcpServiceAccount.existingSecret.key }} {{- end }} + {{- if .Values.gcpProjectId }} + - name: GOOGLE_CLOUD_PROJECT + value: {{ .Values.gcpProjectId | quote }} + {{- end }} command: - java {{- range .Values.jvmOptions }} - - {{ . }} + - {{ . | quote }} + {{- end }} + - -jar + - {{ .Values.jarPath | quote }} + - "--spring.config.location=file:{{ .Values.springConfigMountPath }}/" + {{- $profilesArray := splitList "," .Values.springConfigProfilesActive -}} + {{- $profilesArray = append $profilesArray (.Values.core.enabled | ternary "bundled-core" "") -}} + {{- $profilesArray = append $profilesArray (eq (include "bq_store_and_no_job_options" .) "true" | ternary "bundled-redis" "") -}} + {{- $profilesArray = compact $profilesArray -}} + {{- if $profilesArray }} + - "--spring.profiles.active={{ join "," $profilesArray }}" {{- end }} - - -jar - - /opt/feast/feast-serving.jar - - "--spring.config.location=file:{{ .Values.springConfigMountPath }}/application.yaml" ports: - name: http containerPort: {{ .Values.service.http.targetPort }} - - name: grpc + - name: grpc containerPort: {{ .Values.service.grpc.targetPort }} {{- if .Values.livenessProbe.enabled }} @@ -101,6 +117,6 @@ spec: timeoutSeconds: {{ .Values.readinessProbe.timeoutSeconds }} failureThreshold: {{ .Values.readinessProbe.failureThreshold }} {{- end }} - + resources: {{- toYaml .Values.resources | nindent 10 }} diff --git a/infra/charts/feast/charts/feast-serving/templates/ingress.yaml b/infra/charts/feast/charts/feast-serving/templates/ingress.yaml index c6b4cb07a81..1bcd176147a 100644 --- a/infra/charts/feast/charts/feast-serving/templates/ingress.yaml +++ b/infra/charts/feast/charts/feast-serving/templates/ingress.yaml @@ -1,28 +1,7 @@ -{{- if .Values.ingress.enabled -}} -{{- $fullName := include "feast-serving.fullname" . -}} -apiVersion: extensions/v1beta1 -kind: Ingress -metadata: - name: {{ $fullName }} - labels: - app: {{ template "feast-serving.name" . }} - chart: {{ .Chart.Name }}-{{ .Chart.Version }} - component: serving - heritage: {{ .Release.Service }} - release: {{ .Release.Name }} - annotations: -{{- with .Values.ingress.annotations }} -{{ toYaml . | indent 4 }} +{{- if .Values.ingress.http.enabled -}} +{{ template "feast.ingress" (list . "serving" "http" .Values.ingress.http) }} {{- end }} -spec: - rules: - {{- range .Values.ingress.hosts }} - - host: {{ .host | quote }} - http: - paths: - - path: / - backend: - serviceName: {{ $fullName }} - servicePort: {{ .port | quote }} - {{- end }} +--- +{{ if .Values.ingress.grpc.enabled -}} +{{ template "feast.ingress" (list . "serving" "grpc" .Values.ingress.grpc) }} {{- end }} diff --git a/infra/charts/feast/charts/feast-serving/values.yaml b/infra/charts/feast/charts/feast-serving/values.yaml index d2b3c599479..52d10cd7440 100644 --- a/infra/charts/feast/charts/feast-serving/values.yaml +++ b/infra/charts/feast/charts/feast-serving/values.yaml @@ -3,23 +3,23 @@ # for additional configuration redis: # enabled specifies whether Redis should be installed as part of Feast Serving. - # + # # If enabled, "redis_config" in store.yaml will be overwritten by Helm # to the configuration in this Redis installation. enabled: false # usePassword specifies if password is required to access Redis. Note that # Feast 0.3 does not support Redis with password. - usePassword: false + usePassword: false # cluster configuration for Redis. cluster: # enabled specifies if Redis should be installed in cluster mode. enabled: false -# core configures Feast Core in the same parent feast chart that this Feast +# core configures Feast Core in the same parent feast chart that this Feast # Serving connects to. core: # enabled specifies that Feast Serving will use Feast Core installed - # in the same parent feast chart. If enabled, Helm will overwrite + # in the same parent feast chart. If enabled, Helm will overwrite # "feast.core-host" in application.yaml with the correct value. enabled: true @@ -37,7 +37,7 @@ image: # Refer to https://github.com/gojek/feast/blob/79eb4ab5fa3d37102c1dca9968162a98690526ba/serving/src/main/resources/application.yml # for a complete list and description of the configuration. # -# Note that some properties defined in application.yaml may be overridden by +# Note that some properties defined in application.yaml may be overridden by # Helm under certain conditions. For example, if core is enabled, then # "feast.core-host" will be overridden. Also, if "type: BIGQUERY" is specified # in store.yaml, "feast.jobs.store-options" will be overridden as well with @@ -66,19 +66,19 @@ application.yaml: port: 8080 # store.yaml is the configuration for Feast Store. -# +# # Refer to this link for description: # https://github.com/gojek/feast/blob/79eb4ab5fa3d37102c1dca9968162a98690526ba/protos/feast/core/Store.proto # # Use the correct store configuration depending on whether the installed # Feast Serving is "online" or "batch", by uncommenting the correct store.yaml. # -# Note that if "redis.enabled: true" and "type: REDIS" in store.yaml, +# Note that if "redis.enabled: true" and "type: REDIS" in store.yaml, # Helm will override "redis_config" with configuration of Redis installed # in this chart. -# +# # Note that if "type: BIGQUERY" in store.yaml, Helm assumes Feast Online serving -# is also installed with Redis store. Helm will then override "feast.jobs.store-options" +# is also installed with Redis store. Helm will then override "feast.jobs.store-options" # in application.yaml with the installed Redis store configuration. This is # because in Feast 0.3, Redis job store is required. # @@ -104,7 +104,14 @@ application.yaml: # name: "*" # version: "*" -# springConfigMountPath is the directory path where application.yaml and +springConfigProfiles: {} +# db: | +# spring: +# datasource: +# driverClassName: org.postgresql.Driver +# url: jdbc:postgresql://${DB_HOST:127.0.0.1}:${DB_PORT:5432}/${DB_DATABASE:postgres} +springConfigProfilesActive: "" +# springConfigMountPath is the directory path where application.yaml and # store.yaml will be mounted in the container. springConfigMountPath: /etc/feast/feast-serving @@ -115,7 +122,7 @@ gcpServiceAccount: useExistingSecret: false existingSecret: # name is the secret name of the existing secret for the service account. - name: feast-gcp-service-account + name: feast-gcp-service-account # key is the secret key of the existing secret for the service account. # key is normally derived from the file name of the JSON key file. key: key.json @@ -123,19 +130,29 @@ gcpServiceAccount: # the value of "existingSecret.key" is file name of the service account file. mountPath: /etc/gcloud/service-accounts -# jvmOptions are options that will be passed to the Java Virtual Machine (JVM) +# Project ID picked up by the Cloud SDK (e.g. BigQuery run against this project) +gcpProjectId: "" + +# Path to Jar file in the Docker image. +# If using gcr.io/kf-feast/feast-serving this should not need to be changed. +jarPath: /opt/feast/feast-serving.jar + +# jvmOptions are options that will be passed to the Java Virtual Machine (JVM) # running Feast Core. -# +# # For example, it is good practice to set min and max heap size in JVM. # https://stackoverflow.com/questions/6902135/side-effect-for-increasing-maxpermsize-and-max-heap-size # # Refer to https://docs.oracle.com/cd/E22289_01/html/821-1274/configuring-the-default-jvm-and-java-arguments.html # to see other JVM options that can be set. # -# jvmOptions: -# - -Xms768m +jvmOptions: [] +# - -Xms768m # - -Xmx768m +logType: JSON +logLevel: warn + livenessProbe: enabled: false initialDelaySeconds: 60 @@ -170,12 +187,29 @@ service: # nodePort: ingress: - enabled: false - annotations: {} - # kubernetes.io/ingress.class: nginx - hosts: - # - host: chart-example.local - # port: http + grpc: + enabled: false + class: nginx + hosts: [] + annotations: {} + https: + enabled: true + secretNames: {} + whitelist: "" + auth: + enabled: false + http: + enabled: false + class: nginx + hosts: [] + annotations: {} + https: + enabled: true + secretNames: {} + whitelist: "" + auth: + enabled: false + authUrl: http://auth-server.auth-ns.svc.cluster.local/auth prometheus: enabled: true @@ -185,6 +219,7 @@ resources: {} # choice for the user. This also increases chances charts run on environments with little # resources, such as Minikube. If you do want to specify resources, uncomment the following # lines, adjust them as necessary, and remove the curly braces after 'resources:'. + # # limits: # cpu: 100m # memory: 128Mi diff --git a/infra/charts/feast/requirements.lock b/infra/charts/feast/requirements.lock index 8afd9521573..e441790dc76 100644 --- a/infra/charts/feast/requirements.lock +++ b/infra/charts/feast/requirements.lock @@ -1,12 +1,6 @@ dependencies: -- name: feast-core - repository: "" - version: 0.3.2 -- name: feast-serving - repository: "" - version: 0.3.2 -- name: feast-serving - repository: "" - version: 0.3.2 -digest: sha256:7ee4cd271cbd4ace44817dd12ba65f490a8e3529adf199604a2c2bdad9c2fac3 -generated: "2019-11-27T13:35:41.334054+08:00" +- name: common + repository: https://kubernetes-charts-incubator.storage.googleapis.com + version: 0.0.5 +digest: sha256:935bfb09e9ed90ff800826a7df21adaabe3225511c3ad78df44e1a5a60e93f14 +generated: 2019-12-10T14:47:49.57569Z diff --git a/infra/charts/feast/requirements.yaml b/infra/charts/feast/requirements.yaml index 5416ded3fee..1fa1826965a 100644 --- a/infra/charts/feast/requirements.yaml +++ b/infra/charts/feast/requirements.yaml @@ -9,4 +9,4 @@ dependencies: - name: feast-serving alias: feast-serving-online version: 0.4.4 - condition: feast-serving-online.enabled + condition: feast-serving-online.enabled \ No newline at end of file diff --git a/infra/charts/feast/values-demo.yaml b/infra/charts/feast/values-demo.yaml index fad4bc0afb0..2cb5ccbe741 100644 --- a/infra/charts/feast/values-demo.yaml +++ b/infra/charts/feast/values-demo.yaml @@ -1,7 +1,7 @@ # The following are values for installing Feast for demonstration purpose: # - Persistence is disabled since for demo purpose data is not expected # to be durable -# - Only online serving (no batch serving) is installed to remove dependency +# - Only online serving (no batch serving) is installed to remove dependency # on Google Cloud services. Batch serving requires BigQuery dependency. # - Replace all occurrences of "feast.example.com" with the domain name or # external IP pointing to your cluster @@ -68,4 +68,17 @@ feast-serving-online: version: "*" feast-serving-batch: - enabled: false +# enabled: false + enabled: true + store.yaml: + name: bigquery + type: BIGQUERY + bigquery_config: + project_id: PROJECT_ID + dataset_id: DATASET_ID + subscriptions: + - project: "*" + name: "*" + version: "*" + redis: + enabled: false \ No newline at end of file diff --git a/infra/charts/feast/values.yaml b/infra/charts/feast/values.yaml index f9a0a76dc1b..fde03f9ad71 100644 --- a/infra/charts/feast/values.yaml +++ b/infra/charts/feast/values.yaml @@ -2,10 +2,12 @@ # - Feast Core # - Feast Serving Online # - Feast Serving Batch +# - Prometheus StatsD Exporter # # The configuration for different components can be referenced from: # - charts/feast-core/values.yaml # - charts/feast-serving/values.yaml +# - charts/prometheus-statsd-exporter/values.yaml # # Note that "feast-serving-online" and "feast-serving-batch" are # aliases to "feast-serving" chart since in typical scenario two instances @@ -235,11 +237,11 @@ feast-serving-batch: # enabled as well. So Feast Serving Batch will share the same # Redis instance to store job statuses. store-type: REDIS - store-options: - # Use the externally exposed redis instance deployed by Online service - # Please set EXTERNAL_IP to your cluster's external IP - host: EXTERNAL_IP - port: 32101 + # Default to use the internal hostname of the redis instance deployed by Online service, + # otherwise use externally exposed by setting EXTERNAL_IP to your cluster's external IP + # store-options: + # host: EXTERNAL_IP + # port: 32101 # store.yaml is the configuration for Feast Store. # # Refer to this link for more description: From edfc9f46292f41be5bb45be85f1ca57deb70dd16 Mon Sep 17 00:00:00 2001 From: Shu Heng Date: Thu, 13 Feb 2020 11:23:36 +0800 Subject: [PATCH 039/176] Update v0.4.4 changelog to be consistent with the release --- CHANGELOG.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cc969c34b2a..ee545e3c4d0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,18 +7,14 @@ **Merged pull requests:** - Change RedisBackedJobService to use a connection pool [\#439](https://github.com/gojek/feast/pull/439) ([zhilingc](https://github.com/zhilingc)) -- Update protos with Tensorflow data validation schema [\#438](https://github.com/gojek/feast/pull/438) ([davidheryanto](https://github.com/davidheryanto)) - Update GKE installation and chart values to work with 0.4.3 [\#434](https://github.com/gojek/feast/pull/434) ([lgvital](https://github.com/lgvital)) -- Parameterize end-to-end test scripts [\#433](https://github.com/gojek/feast/pull/433) ([Yanson](https://github.com/Yanson)) - Remove "resource" concept and the need to specify a kind in feature sets [\#432](https://github.com/gojek/feast/pull/432) ([woop](https://github.com/woop)) - Add retry options to BigQuery [\#431](https://github.com/gojek/feast/pull/431) ([Yanson](https://github.com/Yanson)) - Fix logging [\#430](https://github.com/gojek/feast/pull/430) ([Yanson](https://github.com/Yanson)) - Add documentation for bigquery batch retrieval [\#428](https://github.com/gojek/feast/pull/428) ([zhilingc](https://github.com/zhilingc)) - Publish datatypes/java along with sdk/java [\#426](https://github.com/gojek/feast/pull/426) ([ches](https://github.com/ches)) - Update basic Feast example to Feast 0.4 [\#424](https://github.com/gojek/feast/pull/424) ([woop](https://github.com/woop)) -- Unserializable FluentBackoff cause null pointer exception in Dataflow Runner [\#417](https://github.com/gojek/feast/pull/417) ([khorshuheng](https://github.com/khorshuheng)) - Introduce datatypes/java module for proto generation [\#391](https://github.com/gojek/feast/pull/391) ([ches](https://github.com/ches)) -- Allow user to override job options [\#377](https://github.com/gojek/feast/pull/377) ([khorshuheng](https://github.com/khorshuheng)) ## [v0.4.3](https://github.com/gojek/feast/tree/v0.4.3) (2020-01-08) From 177153281fa7697a41314a6e10f46881c0cab66d Mon Sep 17 00:00:00 2001 From: Khor Shu Heng <32997938+khorshuheng@users.noreply.github.com> Date: Thu, 13 Feb 2020 15:39:36 +0800 Subject: [PATCH 040/176] Use Java 8 SDK for branch 3.0 / 4.0 and Java 11 for master (#473) * Use Java 8 SDK for branch 4.0 and Java 11 for master * Add Java 8 tests for branch v3.0 * Semantic versioning for publish java sdk * Fix branch filter refex --- .prow/config.yaml | 113 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 112 insertions(+), 1 deletion(-) diff --git a/.prow/config.yaml b/.prow/config.yaml index e947f440189..3ae9fcbe609 100644 --- a/.prow/config.yaml +++ b/.prow/config.yaml @@ -72,6 +72,22 @@ presubmits: requests: cpu: "2000m" memory: "1536Mi" + skip_branches: + - ^v0\.(3|4)-branch$ + + - name: test-core-and-ingestion-java-8 + decorate: true + always_run: true + spec: + containers: + - image: maven:3.6-jdk-8 + command: [".prow/scripts/test-core-ingestion.sh"] + resources: + requests: + cpu: "2000m" + memory: "1536Mi" + branches: + - ^v0\.(3|4)-branch$ - name: test-serving decorate: true @@ -80,6 +96,18 @@ presubmits: containers: - image: maven:3.6-jdk-11 command: [".prow/scripts/test-serving.sh"] + skip_branches: + - ^v0\.(3|4)-branch$ + + - name: test-serving-java-8 + decorate: true + always_run: true + spec: + containers: + - image: maven:3.6-jdk-8 + command: [".prow/scripts/test-serving.sh"] + branches: + - ^v0\.(3|4)-branch$ - name: test-java-sdk decorate: true @@ -88,6 +116,18 @@ presubmits: containers: - image: maven:3.6-jdk-11 command: [".prow/scripts/test-java-sdk.sh"] + skip_branches: + - ^v0\.(3|4)-branch$ + + - name: test-java-sdk-java-8 + decorate: true + always_run: true + spec: + containers: + - image: maven:3.6-jdk-8 + command: [".prow/scripts/test-java-sdk.sh"] + branches: + - ^v0\.(3|4)-branch$ - name: test-python-sdk decorate: true @@ -116,6 +156,22 @@ presubmits: requests: cpu: "6" memory: "6144Mi" + skip_branches: + - ^v0\.(3|4)-branch$ + + - name: test-end-to-end-java-8 + decorate: true + always_run: true + spec: + containers: + - image: maven:3.6-jdk-8 + command: [".prow/scripts/test-end-to-end.sh"] + resources: + requests: + cpu: "6" + memory: "6144Mi" + branches: + - ^v0\.(3|4)-branch$ - name: test-end-to-end-batch decorate: true @@ -135,6 +191,29 @@ presubmits: volumeMounts: - name: service-account mountPath: "/etc/service-account" + skip_branches: + - ^v0\.(3|4)-branch$ + + - name: test-end-to-end-batch-java-8 + decorate: true + always_run: true + spec: + volumes: + - name: service-account + secret: + secretName: feast-service-account + containers: + - image: maven:3.6-jdk-8 + command: [".prow/scripts/test-end-to-end-batch.sh"] + resources: + requests: + cpu: "6" + memory: "6144Mi" + volumeMounts: + - name: service-account + mountPath: "/etc/service-account" + branches: + - ^v0\.(3|4)-branch$ postsubmits: gojek/feast: @@ -187,10 +266,42 @@ postsubmits: - name: maven-settings secret: secretName: maven-settings + skip_branches: + # Skip version 0.3 and 0.4 + - ^v0\.(3|4)\.(0|[1-9]\d*)(-(0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(\.(0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*)?(\+[0-9a-zA-Z-]+(\.[0-9a-zA-Z-]+)*)?$ + branches: - # Filter on tags with semantic versioning, prefixed with "v" + # Filter on tags with semantic versioning, prefixed with "v". - ^v(0|[1-9]\d*)\.(0|[1-9]\d*)\.(0|[1-9]\d*)(-(0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(\.(0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*)?(\+[0-9a-zA-Z-]+(\.[0-9a-zA-Z-]+)*)?$ + - name: publish-java-8-sdk + decorate: true + spec: + containers: + - image: maven:3.6-jdk-8 + command: + - bash + - -c + - .prow/scripts/publish-java-sdk.sh --revision ${PULL_BASE_REF:1} + volumeMounts: + - name: gpg-keys + mountPath: /etc/gpg + readOnly: true + - name: maven-settings + mountPath: /root/.m2/settings.xml + subPath: settings.xml + readOnly: true + volumes: + - name: gpg-keys + secret: + secretName: gpg-keys + - name: maven-settings + secret: + secretName: maven-settings + branches: + # Filter on tags with semantic versioning, prefixed with "v". v0.3 and v0.4 only. + - ^v0\.(3|4)\.(0|[1-9]\d*)(-(0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*)(\.(0|[1-9]\d*|\d*[a-zA-Z-][0-9a-zA-Z-]*))*)?(\+[0-9a-zA-Z-]+(\.[0-9a-zA-Z-]+)*)?$ + - name: publish-docker-images decorate: true spec: From a7eb4dc130268826277cc8cdabc00f08812d9f21 Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Thu, 13 Feb 2020 17:57:37 +0800 Subject: [PATCH 041/176] Make redis key creation more determinisitic (#380) (#471) * Make redis key creation more determinisitic (#380) * Add documentation to RedisKey in Redis.proto Ensure entities are sorted by the name Co-authored-by: David Heryanto --- .../redis/FeatureRowToRedisMutationDoFn.java | 16 +- .../FeatureRowToRedisMutationDoFnTest.java | 183 ++++++++++++++++++ protos/feast/storage/Redis.proto | 3 +- 3 files changed, 197 insertions(+), 5 deletions(-) create mode 100644 ingestion/src/test/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFnTest.java diff --git a/ingestion/src/main/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFn.java b/ingestion/src/main/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFn.java index 27cca2ffb2e..4b744d0fe6b 100644 --- a/ingestion/src/main/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFn.java +++ b/ingestion/src/main/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFn.java @@ -24,8 +24,9 @@ import feast.store.serving.redis.RedisCustomIO.RedisMutation; import feast.types.FeatureRowProto.FeatureRow; import feast.types.FieldProto.Field; +import java.util.HashMap; +import java.util.List; import java.util.Map; -import java.util.Set; import java.util.stream.Collectors; import org.apache.beam.sdk.transforms.DoFn; import org.slf4j.Logger; @@ -42,17 +43,24 @@ public FeatureRowToRedisMutationDoFn(Map featureSets) { private RedisKey getKey(FeatureRow featureRow) { FeatureSet featureSet = featureSets.get(featureRow.getFeatureSet()); - Set entityNames = + List entityNames = featureSet.getSpec().getEntitiesList().stream() .map(EntitySpec::getName) - .collect(Collectors.toSet()); + .sorted() + .collect(Collectors.toList()); + Map entityFields = new HashMap<>(); Builder redisKeyBuilder = RedisKey.newBuilder().setFeatureSet(featureRow.getFeatureSet()); for (Field field : featureRow.getFieldsList()) { if (entityNames.contains(field.getName())) { - redisKeyBuilder.addEntities(field); + entityFields.putIfAbsent( + field.getName(), + Field.newBuilder().setName(field.getName()).setValue(field.getValue()).build()); } } + for (String entityName : entityNames) { + redisKeyBuilder.addEntities(entityFields.get(entityName)); + } return redisKeyBuilder.build(); } diff --git a/ingestion/src/test/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFnTest.java b/ingestion/src/test/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFnTest.java new file mode 100644 index 00000000000..92bb6e41c38 --- /dev/null +++ b/ingestion/src/test/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFnTest.java @@ -0,0 +1,183 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.store.serving.redis; + +import static org.junit.Assert.*; + +import com.google.protobuf.Timestamp; +import feast.core.FeatureSetProto; +import feast.core.FeatureSetProto.EntitySpec; +import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.core.FeatureSetProto.FeatureSpec; +import feast.storage.RedisProto.RedisKey; +import feast.store.serving.redis.RedisCustomIO.RedisMutation; +import feast.types.FeatureRowProto.FeatureRow; +import feast.types.FieldProto.Field; +import feast.types.ValueProto.Value; +import feast.types.ValueProto.ValueType.Enum; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import org.apache.beam.sdk.extensions.protobuf.ProtoCoder; +import org.apache.beam.sdk.testing.PAssert; +import org.apache.beam.sdk.testing.TestPipeline; +import org.apache.beam.sdk.transforms.Create; +import org.apache.beam.sdk.transforms.ParDo; +import org.apache.beam.sdk.transforms.SerializableFunction; +import org.apache.beam.sdk.values.PCollection; +import org.junit.Rule; +import org.junit.Test; + +public class FeatureRowToRedisMutationDoFnTest { + + @Rule public transient TestPipeline p = TestPipeline.create(); + + private FeatureSetProto.FeatureSet fs = + FeatureSetProto.FeatureSet.newBuilder() + .setSpec( + FeatureSetSpec.newBuilder() + .setName("feature_set") + .setVersion(1) + .addEntities( + EntitySpec.newBuilder() + .setName("entity_id_primary") + .setValueType(Enum.INT32) + .build()) + .addEntities( + EntitySpec.newBuilder() + .setName("entity_id_secondary") + .setValueType(Enum.STRING) + .build()) + .addFeatures( + FeatureSpec.newBuilder() + .setName("feature_1") + .setValueType(Enum.STRING) + .build()) + .addFeatures( + FeatureSpec.newBuilder() + .setName("feature_2") + .setValueType(Enum.INT64) + .build())) + .build(); + + @Test + public void shouldConvertRowWithDuplicateEntitiesToValidKey() { + Map featureSets = new HashMap<>(); + featureSets.put("feature_set", fs); + + FeatureRow offendingRow = + FeatureRow.newBuilder() + .setFeatureSet("feature_set") + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addFields( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addFields( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(2))) + .addFields( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .build(); + + PCollection output = + p.apply(Create.of(Collections.singletonList(offendingRow))) + .setCoder(ProtoCoder.of(FeatureRow.class)) + .apply(ParDo.of(new FeatureRowToRedisMutationDoFn(featureSets))); + + RedisKey expectedKey = + RedisKey.newBuilder() + .setFeatureSet("feature_set") + .addEntities( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addEntities( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .build(); + + PAssert.that(output) + .satisfies( + (SerializableFunction, Void>) + input -> { + input.forEach( + rm -> { + assert (Arrays.equals(rm.getKey(), expectedKey.toByteArray())); + assert (Arrays.equals(rm.getValue(), offendingRow.toByteArray())); + }); + return null; + }); + p.run(); + } + + @Test + public void shouldConvertRowWithOutOfOrderEntitiesToValidKey() { + Map featureSets = new HashMap<>(); + featureSets.put("feature_set", fs); + + FeatureRow offendingRow = + FeatureRow.newBuilder() + .setFeatureSet("feature_set") + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addFields( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .addFields( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .build(); + + PCollection output = + p.apply(Create.of(Collections.singletonList(offendingRow))) + .setCoder(ProtoCoder.of(FeatureRow.class)) + .apply(ParDo.of(new FeatureRowToRedisMutationDoFn(featureSets))); + + RedisKey expectedKey = + RedisKey.newBuilder() + .setFeatureSet("feature_set") + .addEntities( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addEntities( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .build(); + + PAssert.that(output) + .satisfies( + (SerializableFunction, Void>) + input -> { + input.forEach( + rm -> { + assert (Arrays.equals(rm.getKey(), expectedKey.toByteArray())); + assert (Arrays.equals(rm.getValue(), offendingRow.toByteArray())); + }); + return null; + }); + p.run(); + } +} diff --git a/protos/feast/storage/Redis.proto b/protos/feast/storage/Redis.proto index ae287f4e6bf..f58b137e9c1 100644 --- a/protos/feast/storage/Redis.proto +++ b/protos/feast/storage/Redis.proto @@ -32,6 +32,7 @@ message RedisKey { string feature_set = 2; // List of fields containing entity names and their respective values - // contained within this feature row. + // contained within this feature row. The entities should be sorted + // by the entity name alphabetically in ascending order. repeated feast.types.Field entities = 3; } From bbea7c26328a35c352098cfdf97bc929990f3ac2 Mon Sep 17 00:00:00 2001 From: Khor Shu Heng <32997938+khorshuheng@users.noreply.github.com> Date: Fri, 14 Feb 2020 13:56:37 +0800 Subject: [PATCH 042/176] Use bzip2 compressed feature set json as pipeline option (#466) * Use bzip2 compressed feature set json as pipeline option * Make decompressor and compressor more generic and extensible * Avoid code duplication in test --- .../core/job/dataflow/DataflowJobManager.java | 34 ++- .../job/direct/DirectRunnerJobManager.java | 21 +- .../option/FeatureSetJsonByteConverter.java | 47 ++++ .../java/feast/core/model/FeatureSet.java | 16 +- .../src/main/java/feast/core/model/Field.java | 3 +- .../java/feast/core/service/SpecService.java | 9 +- .../job/dataflow/DataflowJobManagerTest.java | 36 ++- .../direct/DirectRunnerJobManagerTest.java | 25 ++- .../FeatureSetJsonByteConverterTest.java | 83 +++++++ .../feast/core/service/SpecServiceTest.java | 209 ++++++++++-------- .../main/java/feast/ingestion/ImportJob.java | 14 +- .../ingestion/options/BZip2Compressor.java | 47 ++++ .../ingestion/options/BZip2Decompressor.java | 38 ++++ .../ingestion/options/ImportOptions.java | 6 +- .../options/InputStreamConverter.java | 31 +++ .../options/OptionByteConverter.java | 30 +++ .../ingestion/options/OptionCompressor.java | 31 +++ .../ingestion/options/OptionDecompressor.java | 30 +++ .../options/StringListStreamConverter.java | 41 ++++ .../java/feast/ingestion/ImportJobTest.java | 17 +- .../options/BZip2CompressorTest.java | 40 ++++ .../options/BZip2DecompressorTest.java | 48 ++++ .../StringListStreamConverterTest.java | 36 +++ .../{util => utils}/DateUtilTest.java | 7 +- .../{util => utils}/JsonUtilTest.java | 3 +- .../{util => utils}/StoreUtilTest.java | 18 +- 26 files changed, 728 insertions(+), 192 deletions(-) create mode 100644 core/src/main/java/feast/core/job/option/FeatureSetJsonByteConverter.java create mode 100644 core/src/test/java/feast/core/job/option/FeatureSetJsonByteConverterTest.java create mode 100644 ingestion/src/main/java/feast/ingestion/options/BZip2Compressor.java create mode 100644 ingestion/src/main/java/feast/ingestion/options/BZip2Decompressor.java create mode 100644 ingestion/src/main/java/feast/ingestion/options/InputStreamConverter.java create mode 100644 ingestion/src/main/java/feast/ingestion/options/OptionByteConverter.java create mode 100644 ingestion/src/main/java/feast/ingestion/options/OptionCompressor.java create mode 100644 ingestion/src/main/java/feast/ingestion/options/OptionDecompressor.java create mode 100644 ingestion/src/main/java/feast/ingestion/options/StringListStreamConverter.java create mode 100644 ingestion/src/test/java/feast/ingestion/options/BZip2CompressorTest.java create mode 100644 ingestion/src/test/java/feast/ingestion/options/BZip2DecompressorTest.java create mode 100644 ingestion/src/test/java/feast/ingestion/options/StringListStreamConverterTest.java rename ingestion/src/test/java/feast/ingestion/{util => utils}/DateUtilTest.java (92%) rename ingestion/src/test/java/feast/ingestion/{util => utils}/JsonUtilTest.java (95%) rename ingestion/src/test/java/feast/ingestion/{util => utils}/StoreUtilTest.java (91%) diff --git a/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java b/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java index 7115ee3f66b..323eb35983e 100644 --- a/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java +++ b/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java @@ -22,7 +22,6 @@ import com.google.common.base.Strings; import com.google.protobuf.InvalidProtocolBufferException; import com.google.protobuf.util.JsonFormat; -import com.google.protobuf.util.JsonFormat.Printer; import feast.core.FeatureSetProto; import feast.core.SourceProto; import feast.core.StoreProto; @@ -30,15 +29,13 @@ import feast.core.exception.JobExecutionException; import feast.core.job.JobManager; import feast.core.job.Runner; -import feast.core.model.FeatureSet; -import feast.core.model.Job; -import feast.core.model.JobStatus; -import feast.core.model.Project; -import feast.core.model.Source; -import feast.core.model.Store; +import feast.core.job.option.FeatureSetJsonByteConverter; +import feast.core.model.*; import feast.core.util.TypeConversion; import feast.ingestion.ImportJob; +import feast.ingestion.options.BZip2Compressor; import feast.ingestion.options.ImportOptions; +import feast.ingestion.options.OptionCompressor; import java.io.IOException; import java.util.ArrayList; import java.util.Collections; @@ -93,7 +90,8 @@ public Job startJob(Job job) { } catch (InvalidProtocolBufferException e) { log.error(e.getMessage()); throw new IllegalArgumentException( - String.format("DataflowJobManager failed to START job with id '%s' because the job" + String.format( + "DataflowJobManager failed to START job with id '%s' because the job" + "has an invalid spec. Please check the FeatureSet, Source and Store specs. Actual error message: %s", job.getId(), e.getMessage())); } @@ -112,12 +110,13 @@ public Job updateJob(Job job) { for (FeatureSet featureSet : job.getFeatureSets()) { featureSetProtos.add(featureSet.toProto()); } - return submitDataflowJob(job.getId(), featureSetProtos, job.getSource().toProto(), - job.getStore().toProto(), true); + return submitDataflowJob( + job.getId(), featureSetProtos, job.getSource().toProto(), job.getStore().toProto(), true); } catch (InvalidProtocolBufferException e) { log.error(e.getMessage()); throw new IllegalArgumentException( - String.format("DataflowJobManager failed to UPDATE job with id '%s' because the job" + String.format( + "DataflowJobManager failed to UPDATE job with id '%s' because the job" + "has an invalid spec. Please check the FeatureSet, Source and Store specs. Actual error message: %s", job.getId(), e.getMessage())); } @@ -221,13 +220,12 @@ private ImportOptions getPipelineOptions( throws IOException { String[] args = TypeConversion.convertMapToArgs(defaultOptions); ImportOptions pipelineOptions = PipelineOptionsFactory.fromArgs(args).as(ImportOptions.class); - Printer printer = JsonFormat.printer(); - List featureSetsJson = new ArrayList<>(); - for (FeatureSetProto.FeatureSet featureSet : featureSets) { - featureSetsJson.add(printer.print(featureSet.getSpec())); - } - pipelineOptions.setFeatureSetJson(featureSetsJson); - pipelineOptions.setStoreJson(Collections.singletonList(printer.print(sink))); + + OptionCompressor> featureSetJsonCompressor = + new BZip2Compressor<>(new FeatureSetJsonByteConverter()); + + pipelineOptions.setFeatureSetJson(featureSetJsonCompressor.compress(featureSets)); + pipelineOptions.setStoreJson(Collections.singletonList(JsonFormat.printer().print(sink))); pipelineOptions.setProject(projectId); pipelineOptions.setUpdate(update); pipelineOptions.setRunner(DataflowRunner.class); diff --git a/core/src/main/java/feast/core/job/direct/DirectRunnerJobManager.java b/core/src/main/java/feast/core/job/direct/DirectRunnerJobManager.java index b01d37d8926..08aeed1cc3a 100644 --- a/core/src/main/java/feast/core/job/direct/DirectRunnerJobManager.java +++ b/core/src/main/java/feast/core/job/direct/DirectRunnerJobManager.java @@ -17,21 +17,22 @@ package feast.core.job.direct; import com.google.common.base.Strings; -import com.google.protobuf.InvalidProtocolBufferException; import com.google.protobuf.util.JsonFormat; -import com.google.protobuf.util.JsonFormat.Printer; import feast.core.FeatureSetProto; import feast.core.StoreProto; import feast.core.config.FeastProperties.MetricsProperties; import feast.core.exception.JobExecutionException; import feast.core.job.JobManager; import feast.core.job.Runner; +import feast.core.job.option.FeatureSetJsonByteConverter; import feast.core.model.FeatureSet; import feast.core.model.Job; import feast.core.model.JobStatus; import feast.core.util.TypeConversion; import feast.ingestion.ImportJob; +import feast.ingestion.options.BZip2Compressor; import feast.ingestion.options.ImportOptions; +import feast.ingestion.options.OptionCompressor; import java.io.IOException; import java.util.ArrayList; import java.util.Collections; @@ -92,17 +93,15 @@ public Job startJob(Job job) { } private ImportOptions getPipelineOptions( - List featureSets, StoreProto.Store sink) - throws InvalidProtocolBufferException { + List featureSets, StoreProto.Store sink) throws IOException { String[] args = TypeConversion.convertMapToArgs(defaultOptions); ImportOptions pipelineOptions = PipelineOptionsFactory.fromArgs(args).as(ImportOptions.class); - Printer printer = JsonFormat.printer(); - List featureSetsJson = new ArrayList<>(); - for (FeatureSetProto.FeatureSet featureSet : featureSets) { - featureSetsJson.add(printer.print(featureSet.getSpec())); - } - pipelineOptions.setFeatureSetJson(featureSetsJson); - pipelineOptions.setStoreJson(Collections.singletonList(printer.print(sink))); + + OptionCompressor> featureSetJsonCompressor = + new BZip2Compressor<>(new FeatureSetJsonByteConverter()); + + pipelineOptions.setFeatureSetJson(featureSetJsonCompressor.compress(featureSets)); + pipelineOptions.setStoreJson(Collections.singletonList(JsonFormat.printer().print(sink))); pipelineOptions.setRunner(DirectRunner.class); pipelineOptions.setProject(""); // set to default value to satisfy validation if (metrics.isEnabled()) { diff --git a/core/src/main/java/feast/core/job/option/FeatureSetJsonByteConverter.java b/core/src/main/java/feast/core/job/option/FeatureSetJsonByteConverter.java new file mode 100644 index 00000000000..dbd04d668fd --- /dev/null +++ b/core/src/main/java/feast/core/job/option/FeatureSetJsonByteConverter.java @@ -0,0 +1,47 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.core.job.option; + +import com.google.protobuf.InvalidProtocolBufferException; +import com.google.protobuf.util.JsonFormat; +import feast.core.FeatureSetProto; +import feast.ingestion.options.OptionByteConverter; +import java.util.ArrayList; +import java.util.List; + +public class FeatureSetJsonByteConverter + implements OptionByteConverter> { + + /** + * Convert list of feature sets to json strings joined by new line, represented as byte arrays + * + * @param featureSets List of feature set protobufs + * @return Byte array representation of the json strings + * @throws InvalidProtocolBufferException + */ + @Override + public byte[] toByte(List featureSets) + throws InvalidProtocolBufferException { + JsonFormat.Printer printer = + JsonFormat.printer().omittingInsignificantWhitespace().printingEnumsAsInts(); + List featureSetsJson = new ArrayList<>(); + for (FeatureSetProto.FeatureSet featureSet : featureSets) { + featureSetsJson.add(printer.print(featureSet.getSpec())); + } + return String.join("\n", featureSetsJson).getBytes(); + } +} diff --git a/core/src/main/java/feast/core/model/FeatureSet.java b/core/src/main/java/feast/core/model/FeatureSet.java index cd6036fe5e5..c593dcd701f 100644 --- a/core/src/main/java/feast/core/model/FeatureSet.java +++ b/core/src/main/java/feast/core/model/FeatureSet.java @@ -264,8 +264,8 @@ private void setEntitySpecFields(EntitySpec.Builder entitySpecBuilder, Field ent if (entityField.getPresence() != null) { entitySpecBuilder.setPresence(FeaturePresence.parseFrom(entityField.getPresence())); } else if (entityField.getGroupPresence() != null) { - entitySpecBuilder - .setGroupPresence(FeaturePresenceWithinGroup.parseFrom(entityField.getGroupPresence())); + entitySpecBuilder.setGroupPresence( + FeaturePresenceWithinGroup.parseFrom(entityField.getGroupPresence())); } if (entityField.getShape() != null) { @@ -298,8 +298,8 @@ private void setEntitySpecFields(EntitySpec.Builder entitySpecBuilder, Field ent } else if (entityField.getTimeDomain() != null) { entitySpecBuilder.setTimeDomain(TimeDomain.parseFrom(entityField.getTimeDomain())); } else if (entityField.getTimeOfDayDomain() != null) { - entitySpecBuilder - .setTimeOfDayDomain(TimeOfDayDomain.parseFrom(entityField.getTimeOfDayDomain())); + entitySpecBuilder.setTimeOfDayDomain( + TimeOfDayDomain.parseFrom(entityField.getTimeOfDayDomain())); } } @@ -314,8 +314,8 @@ private void setFeatureSpecFields(FeatureSpec.Builder featureSpecBuilder, Field if (featureField.getPresence() != null) { featureSpecBuilder.setPresence(FeaturePresence.parseFrom(featureField.getPresence())); } else if (featureField.getGroupPresence() != null) { - featureSpecBuilder - .setGroupPresence(FeaturePresenceWithinGroup.parseFrom(featureField.getGroupPresence())); + featureSpecBuilder.setGroupPresence( + FeaturePresenceWithinGroup.parseFrom(featureField.getGroupPresence())); } if (featureField.getShape() != null) { @@ -348,8 +348,8 @@ private void setFeatureSpecFields(FeatureSpec.Builder featureSpecBuilder, Field } else if (featureField.getTimeDomain() != null) { featureSpecBuilder.setTimeDomain(TimeDomain.parseFrom(featureField.getTimeDomain())); } else if (featureField.getTimeOfDayDomain() != null) { - featureSpecBuilder - .setTimeOfDayDomain(TimeOfDayDomain.parseFrom(featureField.getTimeOfDayDomain())); + featureSpecBuilder.setTimeOfDayDomain( + TimeOfDayDomain.parseFrom(featureField.getTimeOfDayDomain())); } } diff --git a/core/src/main/java/feast/core/model/Field.java b/core/src/main/java/feast/core/model/Field.java index edb0a73acbf..355b673fc84 100644 --- a/core/src/main/java/feast/core/model/Field.java +++ b/core/src/main/java/feast/core/model/Field.java @@ -71,8 +71,7 @@ public class Field { private byte[] timeDomain; private byte[] timeOfDayDomain; - public Field() { - } + public Field() {} public Field(String name, ValueType.Enum type) { this.name = name; diff --git a/core/src/main/java/feast/core/service/SpecService.java b/core/src/main/java/feast/core/service/SpecService.java index 9016b692d1d..5b98d065977 100644 --- a/core/src/main/java/feast/core/service/SpecService.java +++ b/core/src/main/java/feast/core/service/SpecService.java @@ -167,8 +167,7 @@ public ListFeatureSetsResponse listFeatureSets(ListFeatureSetsRequest.Filter fil checkValidCharactersAllowAsterisk(name, "featureSetName"); checkValidCharactersAllowAsterisk(project, "projectName"); - List featureSets = new ArrayList() { - }; + List featureSets = new ArrayList() {}; if (project.equals("*")) { // Matching all projects @@ -277,9 +276,9 @@ public ListStoresResponse listStores(ListStoresRequest.Filter filter) { * Creates or updates a feature set in the repository. If there is a change in the feature set * schema, then the feature set version will be incremented. * - *

This function is idempotent. If no changes are detected in the incoming featureSet's - * schema, this method will update the incoming featureSet spec with the latest version stored in - * the repository, and return that. + *

This function is idempotent. If no changes are detected in the incoming featureSet's schema, + * this method will update the incoming featureSet spec with the latest version stored in the + * repository, and return that. * * @param newFeatureSet Feature set that will be created or updated. */ diff --git a/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java b/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java index c263515ed08..9f26c6919e4 100644 --- a/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java +++ b/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java @@ -19,11 +19,7 @@ import static org.hamcrest.MatcherAssert.assertThat; import static org.hamcrest.Matchers.equalTo; import static org.mockito.ArgumentMatchers.any; -import static org.mockito.Mockito.doReturn; -import static org.mockito.Mockito.spy; -import static org.mockito.Mockito.times; -import static org.mockito.Mockito.verify; -import static org.mockito.Mockito.when; +import static org.mockito.Mockito.*; import static org.mockito.MockitoAnnotations.initMocks; import com.google.api.services.dataflow.Dataflow; @@ -44,14 +40,15 @@ import feast.core.config.FeastProperties.MetricsProperties; import feast.core.exception.JobExecutionException; import feast.core.job.Runner; -import feast.core.model.FeatureSet; -import feast.core.model.Job; -import feast.core.model.JobStatus; -import feast.core.model.Source; -import feast.core.model.Store; +import feast.core.job.option.FeatureSetJsonByteConverter; +import feast.core.model.*; +import feast.ingestion.options.BZip2Compressor; import feast.ingestion.options.ImportOptions; +import feast.ingestion.options.OptionCompressor; import java.io.IOException; +import java.util.Collections; import java.util.HashMap; +import java.util.List; import java.util.Map; import org.apache.beam.runners.dataflow.DataflowPipelineJob; import org.apache.beam.runners.dataflow.DataflowRunner; @@ -131,8 +128,11 @@ public void shouldStartJobWithCorrectPipelineOptions() throws IOException { expectedPipelineOptions.setAppName("DataflowJobManager"); expectedPipelineOptions.setJobName(jobName); expectedPipelineOptions.setStoreJson(Lists.newArrayList(printer.print(store))); + + OptionCompressor> featureSetJsonCompressor = + new BZip2Compressor<>(new FeatureSetJsonByteConverter()); expectedPipelineOptions.setFeatureSetJson( - Lists.newArrayList(printer.print(featureSet.getSpec()))); + featureSetJsonCompressor.compress(Collections.singletonList(featureSet))); ArgumentCaptor captor = ArgumentCaptor.forClass(ImportOptions.class); @@ -170,7 +170,19 @@ public void shouldStartJobWithCorrectPipelineOptions() throws IOException { // Assume the files that are staged are correct expectedPipelineOptions.setFilesToStage(actualPipelineOptions.getFilesToStage()); - assertThat(actualPipelineOptions.toString(), equalTo(expectedPipelineOptions.toString())); + assertThat( + actualPipelineOptions.getFeatureSetJson(), + equalTo(expectedPipelineOptions.getFeatureSetJson())); + assertThat( + actualPipelineOptions.getDeadLetterTableSpec(), + equalTo(expectedPipelineOptions.getDeadLetterTableSpec())); + assertThat( + actualPipelineOptions.getStatsdHost(), equalTo(expectedPipelineOptions.getStatsdHost())); + assertThat( + actualPipelineOptions.getMetricsExporterType(), + equalTo(expectedPipelineOptions.getMetricsExporterType())); + assertThat( + actualPipelineOptions.getStoreJson(), equalTo(expectedPipelineOptions.getStoreJson())); assertThat(actual.getExtId(), equalTo(expectedExtJobId)); } diff --git a/core/src/test/java/feast/core/job/direct/DirectRunnerJobManagerTest.java b/core/src/test/java/feast/core/job/direct/DirectRunnerJobManagerTest.java index 2dd87cfc6e3..64412f4391e 100644 --- a/core/src/test/java/feast/core/job/direct/DirectRunnerJobManagerTest.java +++ b/core/src/test/java/feast/core/job/direct/DirectRunnerJobManagerTest.java @@ -40,14 +40,19 @@ import feast.core.StoreProto.Store.Subscription; import feast.core.config.FeastProperties.MetricsProperties; import feast.core.job.Runner; +import feast.core.job.option.FeatureSetJsonByteConverter; import feast.core.model.FeatureSet; import feast.core.model.Job; import feast.core.model.JobStatus; import feast.core.model.Source; import feast.core.model.Store; +import feast.ingestion.options.BZip2Compressor; import feast.ingestion.options.ImportOptions; +import feast.ingestion.options.OptionCompressor; import java.io.IOException; +import java.util.Collections; import java.util.HashMap; +import java.util.List; import java.util.Map; import org.apache.beam.runners.direct.DirectRunner; import org.apache.beam.sdk.PipelineResult; @@ -121,8 +126,11 @@ public void shouldStartDirectJobAndRegisterPipelineResult() throws IOException { expectedPipelineOptions.setProject(""); expectedPipelineOptions.setStoreJson(Lists.newArrayList(printer.print(store))); expectedPipelineOptions.setProject(""); + + OptionCompressor> featureSetJsonCompressor = + new BZip2Compressor<>(new FeatureSetJsonByteConverter()); expectedPipelineOptions.setFeatureSetJson( - Lists.newArrayList(printer.print(featureSet.getSpec()))); + featureSetJsonCompressor.compress(Collections.singletonList(featureSet))); String expectedJobId = "feast-job-0"; ArgumentCaptor pipelineOptionsCaptor = @@ -150,7 +158,20 @@ public void shouldStartDirectJobAndRegisterPipelineResult() throws IOException { expectedPipelineOptions.setOptionsId( actualPipelineOptions.getOptionsId()); // avoid comparing this value - assertThat(actualPipelineOptions.toString(), equalTo(expectedPipelineOptions.toString())); + assertThat( + actualPipelineOptions.getFeatureSetJson(), + equalTo(expectedPipelineOptions.getFeatureSetJson())); + assertThat( + actualPipelineOptions.getDeadLetterTableSpec(), + equalTo(expectedPipelineOptions.getDeadLetterTableSpec())); + assertThat( + actualPipelineOptions.getStatsdHost(), equalTo(expectedPipelineOptions.getStatsdHost())); + assertThat( + actualPipelineOptions.getMetricsExporterType(), + equalTo(expectedPipelineOptions.getMetricsExporterType())); + assertThat( + actualPipelineOptions.getStoreJson(), equalTo(expectedPipelineOptions.getStoreJson())); + assertThat(jobStarted.getPipelineResult(), equalTo(mockPipelineResult)); assertThat(jobStarted.getJobId(), equalTo(expectedJobId)); assertThat(actual.getExtId(), equalTo(expectedJobId)); diff --git a/core/src/test/java/feast/core/job/option/FeatureSetJsonByteConverterTest.java b/core/src/test/java/feast/core/job/option/FeatureSetJsonByteConverterTest.java new file mode 100644 index 00000000000..2dfeef1d969 --- /dev/null +++ b/core/src/test/java/feast/core/job/option/FeatureSetJsonByteConverterTest.java @@ -0,0 +1,83 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.core.job.option; + +import static org.junit.Assert.*; + +import com.google.protobuf.InvalidProtocolBufferException; +import feast.core.FeatureSetProto; +import feast.core.SourceProto; +import feast.types.ValueProto; +import java.util.List; +import java.util.stream.Collectors; +import java.util.stream.IntStream; +import org.junit.Test; + +public class FeatureSetJsonByteConverterTest { + + private FeatureSetProto.FeatureSet newFeatureSet(Integer version, Integer numberOfFeatures) { + List features = + IntStream.range(1, numberOfFeatures + 1) + .mapToObj( + i -> + FeatureSetProto.FeatureSpec.newBuilder() + .setValueType(ValueProto.ValueType.Enum.FLOAT) + .setName("feature".concat(Integer.toString(i))) + .build()) + .collect(Collectors.toList()); + + return FeatureSetProto.FeatureSet.newBuilder() + .setSpec( + FeatureSetProto.FeatureSetSpec.newBuilder() + .setSource( + SourceProto.Source.newBuilder() + .setType(SourceProto.SourceType.KAFKA) + .setKafkaSourceConfig( + SourceProto.KafkaSourceConfig.newBuilder() + .setBootstrapServers("somebrokers:9092") + .setTopic("sometopic"))) + .addAllFeatures(features) + .setVersion(version) + .addEntities( + FeatureSetProto.EntitySpec.newBuilder() + .setName("entity") + .setValueType(ValueProto.ValueType.Enum.STRING))) + .build(); + } + + @Test + public void shouldConvertFeatureSetsAsJsonStringBytes() throws InvalidProtocolBufferException { + int nrOfFeatureSet = 1; + int nrOfFeatures = 1; + List featureSets = + IntStream.range(1, nrOfFeatureSet + 1) + .mapToObj(i -> newFeatureSet(i, nrOfFeatures)) + .collect(Collectors.toList()); + + String expectedOutputString = + "{\"version\":1," + + "\"entities\":[{\"name\":\"entity\",\"valueType\":2}]," + + "\"features\":[{\"name\":\"feature1\",\"valueType\":6}]," + + "\"source\":{" + + "\"type\":1," + + "\"kafkaSourceConfig\":{" + + "\"bootstrapServers\":\"somebrokers:9092\"," + + "\"topic\":\"sometopic\"}}}"; + FeatureSetJsonByteConverter byteConverter = new FeatureSetJsonByteConverter(); + assertEquals(expectedOutputString, new String(byteConverter.toByte(featureSets))); + } +} diff --git a/core/src/test/java/feast/core/service/SpecServiceTest.java b/core/src/test/java/feast/core/service/SpecServiceTest.java index c533f593e3e..38f7475636d 100644 --- a/core/src/test/java/feast/core/service/SpecServiceTest.java +++ b/core/src/test/java/feast/core/service/SpecServiceTest.java @@ -84,17 +84,13 @@ public class SpecServiceTest { - @Mock - private FeatureSetRepository featureSetRepository; + @Mock private FeatureSetRepository featureSetRepository; - @Mock - private StoreRepository storeRepository; + @Mock private StoreRepository storeRepository; - @Mock - private ProjectRepository projectRepository; + @Mock private ProjectRepository projectRepository; - @Rule - public final ExpectedException expectedException = ExpectedException.none(); + @Rule public final ExpectedException expectedException = ExpectedException.none(); private SpecService specService; private List featureSets; @@ -140,25 +136,25 @@ public void setUp() { when(featureSetRepository.findFeatureSetByNameAndProject_NameAndVersion("f1", "project1", 1)) .thenReturn(featureSets.get(0)); when(featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( - "f1", "project1")) + "f1", "project1")) .thenReturn(featureSets.subList(0, 3)); when(featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( - "f3", "project1")) + "f3", "project1")) .thenReturn(featureSets.subList(4, 5)); when(featureSetRepository.findFirstFeatureSetByNameLikeAndProject_NameOrderByVersionDesc( - "f1", "project1")) + "f1", "project1")) .thenReturn(featureSet1v3); when(featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( - "f1", "project1")) + "f1", "project1")) .thenReturn(featureSets.subList(0, 3)); when(featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( - "asd", "project1")) + "asd", "project1")) .thenReturn(Lists.newArrayList()); when(featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( - "f%", "project1")) + "f%", "project1")) .thenReturn(featureSets); when(featureSetRepository.findAllByNameLikeAndProject_NameLikeOrderByNameAscVersionAsc( - "%", "%")) + "%", "%")) .thenReturn(featureSets); when(projectRepository.findAllByArchivedIsFalse()) @@ -403,7 +399,7 @@ public void applyFeatureSetShouldReturnFeatureSetWithLatestVersionIfFeatureSetHa public void applyFeatureSetShouldApplyFeatureSetWithInitVersionIfNotExists() throws InvalidProtocolBufferException { when(featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( - "f2", "project1")) + "f2", "project1")) .thenReturn(Lists.newArrayList()); FeatureSetProto.FeatureSet incomingFeatureSet = @@ -485,14 +481,14 @@ public void applyFeatureSetShouldNotCreateFeatureSetIfFieldsUnordered() Field f3e1 = new Field("f3e1", Enum.STRING); FeatureSetProto.FeatureSet incomingFeatureSet = (new FeatureSet( - "f3", - "project1", - 5, - 100L, - Arrays.asList(f3e1), - Arrays.asList(f3f2, f3f1), - defaultSource, - FeatureSetStatus.STATUS_READY)) + "f3", + "project1", + 5, + 100L, + Arrays.asList(f3e1), + Arrays.asList(f3f2, f3f1), + defaultSource, + FeatureSetStatus.STATUS_READY)) .toProto(); ApplyFeatureSetResponse applyFeatureSetResponse = @@ -513,78 +509,98 @@ public void applyFeatureSetShouldNotCreateFeatureSetIfFieldsUnordered() public void applyFeatureSetShouldAcceptPresenceShapeAndDomainConstraints() throws InvalidProtocolBufferException { List entitySpecs = new ArrayList<>(); - entitySpecs.add(EntitySpec.newBuilder().setName("entity1") - .setValueType(Enum.INT64) - .setPresence(FeaturePresence.getDefaultInstance()) - .setShape(FixedShape.getDefaultInstance()) - .setDomain("mydomain") - .build()); - entitySpecs.add(EntitySpec.newBuilder().setName("entity2") - .setValueType(Enum.INT64) - .setGroupPresence(FeaturePresenceWithinGroup.getDefaultInstance()) - .setValueCount(ValueCount.getDefaultInstance()) - .setIntDomain(IntDomain.getDefaultInstance()) - .build()); - entitySpecs.add(EntitySpec.newBuilder().setName("entity3") - .setValueType(Enum.FLOAT) - .setPresence(FeaturePresence.getDefaultInstance()) - .setValueCount(ValueCount.getDefaultInstance()) - .setFloatDomain(FloatDomain.getDefaultInstance()) - .build()); - entitySpecs.add(EntitySpec.newBuilder().setName("entity4") - .setValueType(Enum.STRING) - .setPresence(FeaturePresence.getDefaultInstance()) - .setValueCount(ValueCount.getDefaultInstance()) - .setStringDomain(StringDomain.getDefaultInstance()) - .build()); - entitySpecs.add(EntitySpec.newBuilder().setName("entity5") - .setValueType(Enum.BOOL) - .setPresence(FeaturePresence.getDefaultInstance()) - .setValueCount(ValueCount.getDefaultInstance()) - .setBoolDomain(BoolDomain.getDefaultInstance()) - .build()); + entitySpecs.add( + EntitySpec.newBuilder() + .setName("entity1") + .setValueType(Enum.INT64) + .setPresence(FeaturePresence.getDefaultInstance()) + .setShape(FixedShape.getDefaultInstance()) + .setDomain("mydomain") + .build()); + entitySpecs.add( + EntitySpec.newBuilder() + .setName("entity2") + .setValueType(Enum.INT64) + .setGroupPresence(FeaturePresenceWithinGroup.getDefaultInstance()) + .setValueCount(ValueCount.getDefaultInstance()) + .setIntDomain(IntDomain.getDefaultInstance()) + .build()); + entitySpecs.add( + EntitySpec.newBuilder() + .setName("entity3") + .setValueType(Enum.FLOAT) + .setPresence(FeaturePresence.getDefaultInstance()) + .setValueCount(ValueCount.getDefaultInstance()) + .setFloatDomain(FloatDomain.getDefaultInstance()) + .build()); + entitySpecs.add( + EntitySpec.newBuilder() + .setName("entity4") + .setValueType(Enum.STRING) + .setPresence(FeaturePresence.getDefaultInstance()) + .setValueCount(ValueCount.getDefaultInstance()) + .setStringDomain(StringDomain.getDefaultInstance()) + .build()); + entitySpecs.add( + EntitySpec.newBuilder() + .setName("entity5") + .setValueType(Enum.BOOL) + .setPresence(FeaturePresence.getDefaultInstance()) + .setValueCount(ValueCount.getDefaultInstance()) + .setBoolDomain(BoolDomain.getDefaultInstance()) + .build()); List featureSpecs = new ArrayList<>(); - featureSpecs.add(FeatureSpec.newBuilder().setName("feature1") - .setValueType(Enum.INT64) - .setPresence(FeaturePresence.getDefaultInstance()) - .setShape(FixedShape.getDefaultInstance()) - .setDomain("mydomain") - .build()); - featureSpecs.add(FeatureSpec.newBuilder().setName("feature2") - .setValueType(Enum.INT64) - .setGroupPresence(FeaturePresenceWithinGroup.getDefaultInstance()) - .setValueCount(ValueCount.getDefaultInstance()) - .setIntDomain(IntDomain.getDefaultInstance()) - .build()); - featureSpecs.add(FeatureSpec.newBuilder().setName("feature3") - .setValueType(Enum.FLOAT) - .setPresence(FeaturePresence.getDefaultInstance()) - .setValueCount(ValueCount.getDefaultInstance()) - .setFloatDomain(FloatDomain.getDefaultInstance()) - .build()); - featureSpecs.add(FeatureSpec.newBuilder().setName("feature4") - .setValueType(Enum.STRING) - .setPresence(FeaturePresence.getDefaultInstance()) - .setValueCount(ValueCount.getDefaultInstance()) - .setStringDomain(StringDomain.getDefaultInstance()) - .build()); - featureSpecs.add(FeatureSpec.newBuilder().setName("feature5") - .setValueType(Enum.BOOL) - .setPresence(FeaturePresence.getDefaultInstance()) - .setValueCount(ValueCount.getDefaultInstance()) - .setBoolDomain(BoolDomain.getDefaultInstance()) - .build()); - - FeatureSetSpec featureSetSpec = FeatureSetSpec.newBuilder() - .setProject("project1") - .setName("featureSetWithConstraints") - .addAllEntities(entitySpecs) - .addAllFeatures(featureSpecs) - .build(); - FeatureSetProto.FeatureSet featureSet = FeatureSetProto.FeatureSet.newBuilder() - .setSpec(featureSetSpec) - .build(); + featureSpecs.add( + FeatureSpec.newBuilder() + .setName("feature1") + .setValueType(Enum.INT64) + .setPresence(FeaturePresence.getDefaultInstance()) + .setShape(FixedShape.getDefaultInstance()) + .setDomain("mydomain") + .build()); + featureSpecs.add( + FeatureSpec.newBuilder() + .setName("feature2") + .setValueType(Enum.INT64) + .setGroupPresence(FeaturePresenceWithinGroup.getDefaultInstance()) + .setValueCount(ValueCount.getDefaultInstance()) + .setIntDomain(IntDomain.getDefaultInstance()) + .build()); + featureSpecs.add( + FeatureSpec.newBuilder() + .setName("feature3") + .setValueType(Enum.FLOAT) + .setPresence(FeaturePresence.getDefaultInstance()) + .setValueCount(ValueCount.getDefaultInstance()) + .setFloatDomain(FloatDomain.getDefaultInstance()) + .build()); + featureSpecs.add( + FeatureSpec.newBuilder() + .setName("feature4") + .setValueType(Enum.STRING) + .setPresence(FeaturePresence.getDefaultInstance()) + .setValueCount(ValueCount.getDefaultInstance()) + .setStringDomain(StringDomain.getDefaultInstance()) + .build()); + featureSpecs.add( + FeatureSpec.newBuilder() + .setName("feature5") + .setValueType(Enum.BOOL) + .setPresence(FeaturePresence.getDefaultInstance()) + .setValueCount(ValueCount.getDefaultInstance()) + .setBoolDomain(BoolDomain.getDefaultInstance()) + .build()); + + FeatureSetSpec featureSetSpec = + FeatureSetSpec.newBuilder() + .setProject("project1") + .setName("featureSetWithConstraints") + .addAllEntities(entitySpecs) + .addAllFeatures(featureSpecs) + .build(); + FeatureSetProto.FeatureSet featureSet = + FeatureSetProto.FeatureSet.newBuilder().setSpec(featureSetSpec).build(); ApplyFeatureSetResponse applyFeatureSetResponse = specService.applyFeatureSet(featureSet); FeatureSetSpec appliedFeatureSetSpec = applyFeatureSetResponse.getFeatureSet().getSpec(); @@ -596,7 +612,8 @@ public void applyFeatureSetShouldAcceptPresenceShapeAndDomainConstraints() // appliedFeatureSpecs needs to be sorted because the list returned by specService may not // follow the order in the request - List appliedFeatureSpecs = new ArrayList<>(appliedFeatureSetSpec.getFeaturesList()); + List appliedFeatureSpecs = + new ArrayList<>(appliedFeatureSetSpec.getFeaturesList()); appliedFeatureSpecs.sort(Comparator.comparing(FeatureSpec::getName)); assertEquals(appliedEntitySpecs.size(), entitySpecs.size()); @@ -684,6 +701,4 @@ private Store newDummyStore(String name) { store.setConfig(RedisConfig.newBuilder().setPort(6379).build().toByteArray()); return store; } - - } diff --git a/ingestion/src/main/java/feast/ingestion/ImportJob.java b/ingestion/src/main/java/feast/ingestion/ImportJob.java index 41af5f9bb40..c4973ce3cae 100644 --- a/ingestion/src/main/java/feast/ingestion/ImportJob.java +++ b/ingestion/src/main/java/feast/ingestion/ImportJob.java @@ -22,7 +22,9 @@ import feast.core.FeatureSetProto.FeatureSet; import feast.core.SourceProto.Source; import feast.core.StoreProto.Store; +import feast.ingestion.options.BZip2Decompressor; import feast.ingestion.options.ImportOptions; +import feast.ingestion.options.StringListStreamConverter; import feast.ingestion.transform.ReadFromSource; import feast.ingestion.transform.ValidateFeatureRows; import feast.ingestion.transform.WriteFailedElementToBigQuery; @@ -33,6 +35,7 @@ import feast.ingestion.utils.StoreUtil; import feast.ingestion.values.FailedElement; import feast.types.FeatureRowProto.FeatureRow; +import java.io.IOException; import java.util.HashMap; import java.util.List; import java.util.Map; @@ -57,15 +60,14 @@ public class ImportJob { * @param args arguments to be passed to Beam pipeline * @throws InvalidProtocolBufferException if options passed to the pipeline are invalid */ - public static void main(String[] args) throws InvalidProtocolBufferException { + public static void main(String[] args) throws IOException { ImportOptions options = PipelineOptionsFactory.fromArgs(args).withValidation().create().as(ImportOptions.class); runPipeline(options); } @SuppressWarnings("UnusedReturnValue") - public static PipelineResult runPipeline(ImportOptions options) - throws InvalidProtocolBufferException { + public static PipelineResult runPipeline(ImportOptions options) throws IOException { /* * Steps: * 1. Read messages from Feast Source as FeatureRow @@ -80,8 +82,10 @@ public static PipelineResult runPipeline(ImportOptions options) log.info("Starting import job with settings: \n{}", options.toString()); - List featureSets = - SpecUtil.parseFeatureSetSpecJsonList(options.getFeatureSetJson()); + BZip2Decompressor> decompressor = + new BZip2Decompressor<>(new StringListStreamConverter()); + List featureSetJson = decompressor.decompress(options.getFeatureSetJson()); + List featureSets = SpecUtil.parseFeatureSetSpecJsonList(featureSetJson); List stores = SpecUtil.parseStoreJsonList(options.getStoreJson()); for (Store store : stores) { diff --git a/ingestion/src/main/java/feast/ingestion/options/BZip2Compressor.java b/ingestion/src/main/java/feast/ingestion/options/BZip2Compressor.java new file mode 100644 index 00000000000..b7e4e6ee0af --- /dev/null +++ b/ingestion/src/main/java/feast/ingestion/options/BZip2Compressor.java @@ -0,0 +1,47 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.ingestion.options; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import org.apache.commons.compress.compressors.bzip2.BZip2CompressorOutputStream; + +public class BZip2Compressor implements OptionCompressor { + + private final OptionByteConverter byteConverter; + + public BZip2Compressor(OptionByteConverter byteConverter) { + this.byteConverter = byteConverter; + } + /** + * Compress pipeline option using BZip2 + * + * @param option Pipeline option value + * @return BZip2 compressed option value + * @throws IOException + */ + @Override + public byte[] compress(T option) throws IOException { + ByteArrayOutputStream compressedStream = new ByteArrayOutputStream(); + try (BZip2CompressorOutputStream bzip2Output = + new BZip2CompressorOutputStream(compressedStream)) { + bzip2Output.write(byteConverter.toByte(option)); + } + + return compressedStream.toByteArray(); + } +} diff --git a/ingestion/src/main/java/feast/ingestion/options/BZip2Decompressor.java b/ingestion/src/main/java/feast/ingestion/options/BZip2Decompressor.java new file mode 100644 index 00000000000..ce49c1be6e6 --- /dev/null +++ b/ingestion/src/main/java/feast/ingestion/options/BZip2Decompressor.java @@ -0,0 +1,38 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.ingestion.options; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream; + +public class BZip2Decompressor implements OptionDecompressor { + + private final InputStreamConverter inputStreamConverter; + + public BZip2Decompressor(InputStreamConverter inputStreamConverter) { + this.inputStreamConverter = inputStreamConverter; + } + + @Override + public T decompress(byte[] compressed) throws IOException { + try (ByteArrayInputStream inputStream = new ByteArrayInputStream(compressed); + BZip2CompressorInputStream bzip2Input = new BZip2CompressorInputStream(inputStream)) { + return inputStreamConverter.readStream(bzip2Input); + } + } +} diff --git a/ingestion/src/main/java/feast/ingestion/options/ImportOptions.java b/ingestion/src/main/java/feast/ingestion/options/ImportOptions.java index b299bb47e55..6afdd80dd72 100644 --- a/ingestion/src/main/java/feast/ingestion/options/ImportOptions.java +++ b/ingestion/src/main/java/feast/ingestion/options/ImportOptions.java @@ -28,16 +28,16 @@ public interface ImportOptions extends PipelineOptions, DataflowPipelineOptions, DirectOptions { @Required @Description( - "JSON string representation of the FeatureSet that the import job will process." + "JSON string representation of the FeatureSet that the import job will process, in BZip2 binary format." + "FeatureSet follows the format in feast.core.FeatureSet proto." + "Mutliple FeatureSetSpec can be passed by specifying '--featureSet={...}' multiple times" + "The conversion of Proto message to JSON should follow this mapping:" + "https://developers.google.com/protocol-buffers/docs/proto3#json" + "Please minify and remove all insignificant whitespace such as newline in the JSON string" + "to prevent error when parsing the options") - List getFeatureSetJson(); + byte[] getFeatureSetJson(); - void setFeatureSetJson(List featureSetJson); + void setFeatureSetJson(byte[] featureSetJson); @Required @Description( diff --git a/ingestion/src/main/java/feast/ingestion/options/InputStreamConverter.java b/ingestion/src/main/java/feast/ingestion/options/InputStreamConverter.java new file mode 100644 index 00000000000..e2fef732368 --- /dev/null +++ b/ingestion/src/main/java/feast/ingestion/options/InputStreamConverter.java @@ -0,0 +1,31 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.ingestion.options; + +import java.io.IOException; +import java.io.InputStream; + +public interface InputStreamConverter { + + /** + * Used in conjunction with {@link OptionDecompressor} to decompress the pipeline option + * + * @param inputStream Input byte stream in compressed format + * @return Decompressed pipeline option value + */ + T readStream(InputStream inputStream) throws IOException; +} diff --git a/ingestion/src/main/java/feast/ingestion/options/OptionByteConverter.java b/ingestion/src/main/java/feast/ingestion/options/OptionByteConverter.java new file mode 100644 index 00000000000..ff5a41a627d --- /dev/null +++ b/ingestion/src/main/java/feast/ingestion/options/OptionByteConverter.java @@ -0,0 +1,30 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.ingestion.options; + +import java.io.IOException; + +public interface OptionByteConverter { + + /** + * Used in conjunction with {@link OptionCompressor} to compress the pipeline option + * + * @param option Pipeline option value + * @return byte representation of the pipeline option value, without compression. + */ + byte[] toByte(T option) throws IOException; +} diff --git a/ingestion/src/main/java/feast/ingestion/options/OptionCompressor.java b/ingestion/src/main/java/feast/ingestion/options/OptionCompressor.java new file mode 100644 index 00000000000..b2345fc3eb1 --- /dev/null +++ b/ingestion/src/main/java/feast/ingestion/options/OptionCompressor.java @@ -0,0 +1,31 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.ingestion.options; + +import java.io.IOException; + +public interface OptionCompressor { + + /** + * Compress pipeline option into bytes format. This is necessary as some Beam runner has + * limitation in terms of pipeline option size. + * + * @param option Pipeline option value + * @return Compressed values of the option, as byte array + */ + byte[] compress(T option) throws IOException; +} diff --git a/ingestion/src/main/java/feast/ingestion/options/OptionDecompressor.java b/ingestion/src/main/java/feast/ingestion/options/OptionDecompressor.java new file mode 100644 index 00000000000..affeafdaa0b --- /dev/null +++ b/ingestion/src/main/java/feast/ingestion/options/OptionDecompressor.java @@ -0,0 +1,30 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.ingestion.options; + +import java.io.IOException; + +public interface OptionDecompressor { + + /** + * Decompress pipeline option from byte array. + * + * @param compressed Compressed pipeline option value + * @return Decompressed pipeline option + */ + T decompress(byte[] compressed) throws IOException; +} diff --git a/ingestion/src/main/java/feast/ingestion/options/StringListStreamConverter.java b/ingestion/src/main/java/feast/ingestion/options/StringListStreamConverter.java new file mode 100644 index 00000000000..d7277f3c7d6 --- /dev/null +++ b/ingestion/src/main/java/feast/ingestion/options/StringListStreamConverter.java @@ -0,0 +1,41 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.ingestion.options; + +import java.io.BufferedReader; +import java.io.IOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.util.List; +import java.util.stream.Collectors; + +public class StringListStreamConverter implements InputStreamConverter> { + + /** + * Convert Input byte stream to newline separated strings + * + * @param inputStream Input byte stream + * @return List of string + */ + @Override + public List readStream(InputStream inputStream) throws IOException { + BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream)); + List stringList = reader.lines().collect(Collectors.toList()); + reader.close(); + return stringList; + } +} diff --git a/ingestion/src/test/java/feast/ingestion/ImportJobTest.java b/ingestion/src/test/java/feast/ingestion/ImportJobTest.java index 290b38dabee..58ecae8f045 100644 --- a/ingestion/src/test/java/feast/ingestion/ImportJobTest.java +++ b/ingestion/src/test/java/feast/ingestion/ImportJobTest.java @@ -30,13 +30,16 @@ import feast.core.StoreProto.Store.RedisConfig; import feast.core.StoreProto.Store.StoreType; import feast.core.StoreProto.Store.Subscription; +import feast.ingestion.options.BZip2Compressor; import feast.ingestion.options.ImportOptions; +import feast.ingestion.options.OptionByteConverter; import feast.storage.RedisProto.RedisKey; import feast.test.TestUtil; import feast.test.TestUtil.LocalKafka; import feast.test.TestUtil.LocalRedis; import feast.types.FeatureRowProto.FeatureRow; import feast.types.ValueProto.ValueType.Enum; +import java.io.ByteArrayOutputStream; import java.io.IOException; import java.nio.charset.StandardCharsets; import java.util.ArrayList; @@ -48,6 +51,7 @@ import org.apache.beam.sdk.PipelineResult; import org.apache.beam.sdk.PipelineResult.State; import org.apache.beam.sdk.options.PipelineOptionsFactory; +import org.apache.commons.compress.compressors.bzip2.BZip2CompressorOutputStream; import org.apache.kafka.common.serialization.ByteArraySerializer; import org.joda.time.Duration; import org.junit.AfterClass; @@ -162,12 +166,13 @@ public void runPipeline_ShouldWriteToRedisCorrectlyGivenValidSpecAndFeatureRow() .build(); ImportOptions options = PipelineOptionsFactory.create().as(ImportOptions.class); - options.setFeatureSetJson( - Collections.singletonList( - JsonFormat.printer().omittingInsignificantWhitespace().print(featureSet.getSpec()))); - options.setStoreJson( - Collections.singletonList( - JsonFormat.printer().omittingInsignificantWhitespace().print(redis))); + BZip2Compressor compressor = new BZip2Compressor<>(option -> { + JsonFormat.Printer printer = + JsonFormat.printer().omittingInsignificantWhitespace().printingEnumsAsInts(); + return printer.print(option).getBytes(); + }); + options.setFeatureSetJson(compressor.compress(spec)); + options.setStoreJson(Collections.singletonList(JsonFormat.printer().print(redis))); options.setProject(""); options.setBlockOnRun(false); diff --git a/ingestion/src/test/java/feast/ingestion/options/BZip2CompressorTest.java b/ingestion/src/test/java/feast/ingestion/options/BZip2CompressorTest.java new file mode 100644 index 00000000000..cd03b18c793 --- /dev/null +++ b/ingestion/src/test/java/feast/ingestion/options/BZip2CompressorTest.java @@ -0,0 +1,40 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.ingestion.options; + +import java.io.BufferedReader; +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStreamReader; +import org.apache.commons.compress.compressors.bzip2.BZip2CompressorInputStream; +import org.junit.Assert; +import org.junit.Test; + +public class BZip2CompressorTest { + + @Test + public void shouldHavBZip2CompatibleOutput() throws IOException { + BZip2Compressor compressor = new BZip2Compressor<>(String::getBytes); + String origString = "somestring"; + try (ByteArrayInputStream inputStream = + new ByteArrayInputStream(compressor.compress(origString)); + BZip2CompressorInputStream bzip2Input = new BZip2CompressorInputStream(inputStream); + BufferedReader reader = new BufferedReader(new InputStreamReader(bzip2Input))) { + Assert.assertEquals(origString, reader.readLine()); + } + } +} diff --git a/ingestion/src/test/java/feast/ingestion/options/BZip2DecompressorTest.java b/ingestion/src/test/java/feast/ingestion/options/BZip2DecompressorTest.java new file mode 100644 index 00000000000..fe7cc789d86 --- /dev/null +++ b/ingestion/src/test/java/feast/ingestion/options/BZip2DecompressorTest.java @@ -0,0 +1,48 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.ingestion.options; + +import static org.junit.Assert.*; + +import java.io.*; +import org.apache.commons.compress.compressors.bzip2.BZip2CompressorOutputStream; +import org.junit.Test; + +public class BZip2DecompressorTest { + + @Test + public void shouldDecompressBZip2Stream() throws IOException { + BZip2Decompressor decompressor = + new BZip2Decompressor<>( + inputStream -> { + BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream)); + String output = reader.readLine(); + reader.close(); + return output; + }); + + String originalString = "abc"; + ByteArrayOutputStream compressedStream = new ByteArrayOutputStream(); + try (BZip2CompressorOutputStream bzip2Output = + new BZip2CompressorOutputStream(compressedStream)) { + bzip2Output.write(originalString.getBytes()); + } + + String decompressedString = decompressor.decompress(compressedStream.toByteArray()); + assertEquals(originalString, decompressedString); + } +} diff --git a/ingestion/src/test/java/feast/ingestion/options/StringListStreamConverterTest.java b/ingestion/src/test/java/feast/ingestion/options/StringListStreamConverterTest.java new file mode 100644 index 00000000000..5ce9f054bc9 --- /dev/null +++ b/ingestion/src/test/java/feast/ingestion/options/StringListStreamConverterTest.java @@ -0,0 +1,36 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.ingestion.options; + +import static org.junit.Assert.*; + +import java.io.ByteArrayInputStream; +import java.io.IOException; +import java.io.InputStream; +import java.util.Arrays; +import org.junit.Test; + +public class StringListStreamConverterTest { + + @Test + public void shouldReadStreamAsNewlineSeparatedStrings() throws IOException { + StringListStreamConverter converter = new StringListStreamConverter(); + String originalString = "abc\ndef"; + InputStream stringStream = new ByteArrayInputStream(originalString.getBytes()); + assertEquals(Arrays.asList("abc", "def"), converter.readStream(stringStream)); + } +} diff --git a/ingestion/src/test/java/feast/ingestion/util/DateUtilTest.java b/ingestion/src/test/java/feast/ingestion/utils/DateUtilTest.java similarity index 92% rename from ingestion/src/test/java/feast/ingestion/util/DateUtilTest.java rename to ingestion/src/test/java/feast/ingestion/utils/DateUtilTest.java index 71d4e67beaa..151d501a596 100644 --- a/ingestion/src/test/java/feast/ingestion/util/DateUtilTest.java +++ b/ingestion/src/test/java/feast/ingestion/utils/DateUtilTest.java @@ -14,15 +14,12 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.ingestion.util; +package feast.ingestion.utils; import static org.hamcrest.MatcherAssert.assertThat; -import static org.hamcrest.Matchers.equalTo; -import static org.hamcrest.Matchers.is; -import static org.hamcrest.Matchers.not; +import static org.hamcrest.Matchers.*; import com.google.protobuf.Timestamp; -import feast.ingestion.utils.DateUtil; import junit.framework.TestCase; import org.joda.time.DateTime; diff --git a/ingestion/src/test/java/feast/ingestion/util/JsonUtilTest.java b/ingestion/src/test/java/feast/ingestion/utils/JsonUtilTest.java similarity index 95% rename from ingestion/src/test/java/feast/ingestion/util/JsonUtilTest.java rename to ingestion/src/test/java/feast/ingestion/utils/JsonUtilTest.java index 02af4d819f9..62c74dfc345 100644 --- a/ingestion/src/test/java/feast/ingestion/util/JsonUtilTest.java +++ b/ingestion/src/test/java/feast/ingestion/utils/JsonUtilTest.java @@ -14,12 +14,11 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.ingestion.util; +package feast.ingestion.utils; import static org.hamcrest.Matchers.equalTo; import static org.junit.Assert.assertThat; -import feast.ingestion.utils.JsonUtil; import java.util.Collections; import java.util.HashMap; import java.util.Map; diff --git a/ingestion/src/test/java/feast/ingestion/util/StoreUtilTest.java b/ingestion/src/test/java/feast/ingestion/utils/StoreUtilTest.java similarity index 91% rename from ingestion/src/test/java/feast/ingestion/util/StoreUtilTest.java rename to ingestion/src/test/java/feast/ingestion/utils/StoreUtilTest.java index 4e2297e405d..82988121bc8 100644 --- a/ingestion/src/test/java/feast/ingestion/util/StoreUtilTest.java +++ b/ingestion/src/test/java/feast/ingestion/utils/StoreUtilTest.java @@ -14,22 +14,9 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.ingestion.util; +package feast.ingestion.utils; -import static feast.types.ValueProto.ValueType.Enum.BOOL; -import static feast.types.ValueProto.ValueType.Enum.BOOL_LIST; -import static feast.types.ValueProto.ValueType.Enum.BYTES; -import static feast.types.ValueProto.ValueType.Enum.BYTES_LIST; -import static feast.types.ValueProto.ValueType.Enum.DOUBLE; -import static feast.types.ValueProto.ValueType.Enum.DOUBLE_LIST; -import static feast.types.ValueProto.ValueType.Enum.FLOAT; -import static feast.types.ValueProto.ValueType.Enum.FLOAT_LIST; -import static feast.types.ValueProto.ValueType.Enum.INT32; -import static feast.types.ValueProto.ValueType.Enum.INT32_LIST; -import static feast.types.ValueProto.ValueType.Enum.INT64; -import static feast.types.ValueProto.ValueType.Enum.INT64_LIST; -import static feast.types.ValueProto.ValueType.Enum.STRING; -import static feast.types.ValueProto.ValueType.Enum.STRING_LIST; +import static feast.types.ValueProto.ValueType.Enum.*; import com.google.cloud.bigquery.BigQuery; import com.google.cloud.bigquery.Field; @@ -40,7 +27,6 @@ import feast.core.FeatureSetProto.FeatureSet; import feast.core.FeatureSetProto.FeatureSetSpec; import feast.core.FeatureSetProto.FeatureSpec; -import feast.ingestion.utils.StoreUtil; import java.util.Arrays; import org.junit.Assert; import org.junit.Test; From dd59a38a2a1f23d93b5b15ef28ad4917d365aa19 Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Fri, 14 Feb 2020 07:19:56 +0000 Subject: [PATCH 043/176] GitBook: [master] one page modified --- docs/contributing.md | 435 ++++++++++++++++++++++++++++--------------- 1 file changed, 284 insertions(+), 151 deletions(-) diff --git a/docs/contributing.md b/docs/contributing.md index fdac047beba..f3394e12ab9 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -1,78 +1,267 @@ # Contributing -## Getting Started +## 1. Contribution process + +We use [RFCs](https://en.wikipedia.org/wiki/Request_for_Comments) and [GitHub issues](https://github.com/gojek/feast/issues) to communicate development ideas. The simplest way to contribute to Feast is to leave comments in our [RFCs](https://drive.google.com/drive/u/0/folders/1Lj1nIeRB868oZvKTPLYqAvKQ4O0BksjY) in the [Feast Google Drive](https://drive.google.com/drive/u/0/folders/0AAe8j7ZK3sxSUk9PVA) or our GitHub issues. + +Please communicate your ideas through a GitHub issue or through our Slack Channel before starting development. + +Please [submit a PR ](https://github.com/gojek/feast/pulls)to the master branch of the Feast repository once you are ready to submit your contribution. Code submission to Feast \(including submission from project maintainers\) require review and approval from maintainers or code owners. + +PRs that are submitted by the general public need to be identified as `ok-to-test`. Once enabled, [Prow](https://github.com/kubernetes/test-infra/tree/master/prow) will run a range of tests to verify the submission, after which community members will help to review the pull request. + +{% hint style="success" %} +Please sign the [Google CLA](https://cla.developers.google.com/) in order to have your code merged into the Feast repository. +{% endhint %} + +## 2. Development guide + +### 2.1 Overview The following guide will help you quickly run Feast in your local machine. The main components of Feast are: -* **Feast Core** handles FeatureSpec registration, starts and monitors Ingestion +* **Feast Core:** Handles feature registration, starts and manages ingestion jobs and ensures that Feast internal metadata is consistent. +* **Feast Ingestion Jobs:** Subscribes to streams of FeatureRows and writes these as feature - jobs and ensures that Feast internal metadata is consistent. + values to registered databases \(online, historical\) that can be read by Feast Serving. -* **Feast Ingestion** subscribes to streams of FeatureRow and writes the feature +* **Feast Serving:** Service that handles requests for features values, either online or batch. - values to registered Stores. +### 2.**2 Requirements** -* **Feast Serving** handles requests for features values retrieval from the end users. +#### 2.**2.1 Development environment** -**Pre-requisites** +The following software is required for Feast development * Java SE Development Kit 11 * Python version 3.6 \(or above\) and pip -* Access to Postgres database \(version 11 and above\) -* Access to [Redis](https://redis.io/topics/quickstart) instance \(tested on version 5.x\) -* Access to [Kafka](https://kafka.apache.org/) brokers \(tested on version 2.x\) -* [Maven ](https://maven.apache.org/install.html) version 3.6.x -* [grpc\_cli](https://github.com/grpc/grpc/blob/master/doc/command_line_tool.md) is useful for debugging and quick testing -* An overview of Feast specifications and protos - -> **Assumptions:** -> -> 1. Postgres is running in "localhost:5432" and has a database called "postgres" which -> -> can be accessed with credentials user "postgres" and password "password". -> -> To use different database name and credentials, please update -> -> "$FEAST\_HOME/core/src/main/resources/application.yml" -> -> or set these environment variables: DB\_HOST, DB\_USERNAME, DB\_PASSWORD. -> -> 2. Redis is running locally and accessible from "localhost:6379" -> 3. Feast has admin access to BigQuery. +* [Maven ](https://maven.apache.org/install.html)version 3.6.x + +Additionally, [grpc\_cli](https://github.com/grpc/grpc/blob/master/doc/command_line_tool.md) is useful for debugging and quick testing of gRPC endpoints. + +#### 2.**2.2 Services** + +The following components/services are required to develop Feast: + +* **Feast Core:** Requires PostgreSQL \(version 11 and above\) to store state, and requires a Kafka \(tested on version 2.x\) setup to allow for ingestion of FeatureRows. +* **Feast Serving:** Requires Redis \(tested on version 5.x\). + +These services should be running before starting development. The following snippet will start the services using Docker. + +```bash +# Start Postgres +docker run --name postgres --rm -it -d --net host -e POSTGRES_DB=postgres -e POSTGRES_USER=postgres \ +-e POSTGRES_PASSWORD=password postgres:12-alpine + +# Start Redis +docker run --name redis --rm -it --net host -d redis:5-alpine + +# Start Zookeeper (needed by Kafka) +docker run --rm \ + --net=host \ + --name=zookeeper \ + --env=ZOOKEEPER_CLIENT_PORT=2181 \ + --detach confluentinc/cp-zookeeper:5.2.1 + +# Start Kafka +docker run --rm \ + --net=host \ + --name=kafka \ + --env=KAFKA_ZOOKEEPER_CONNECT=localhost:2181 \ + --env=KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \ + --env=KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ + --detach confluentinc/cp-kafka:5.2.1 +``` + +### 2.3 Testing and development + +#### 2.3.1 Running unit tests ```text -# $FEAST_HOME will refer to be the root directory of this Feast Git repository +$ mvn test +``` + +#### 2.3.2 Running integration tests + +_Note: integration suite isn't yet separated from unit._ -git clone https://github.com/gojek/feast -cd feast +```text +$ mvn verify ``` -#### Starting Feast Core +#### 2.3.3 Running components locally + +The `core` and `serving` modules are Spring Boot applications. These may be run as usual for [the Spring Boot Maven plugin](https://docs.spring.io/spring-boot/docs/current/maven-plugin/index.html): ```text -# Please check the default configuration for Feast Core in -# "$FEAST_HOME/core/src/main/resources/application.yml" and update it accordingly. -# -# Start Feast Core GRPC server on localhost:6565 +$ mvn --projects core spring-boot:run + +# Or for short: +$ mvn -pl core spring-boot:run +``` + +Note that you should execute `mvn` from the Feast repository root directory, as there are intermodule dependencies that Maven will not resolve if you `cd` to subdirectories to run. + +#### 2.3.4 Running from IntelliJ + +Compiling and running tests in IntelliJ should work as usual. + +Running the Spring Boot apps may work out of the box in IDEA Ultimate, which has built-in support for Spring Boot projects, but the Community Edition needs a bit of help: + +The Spring Boot Maven plugin automatically puts dependencies with `provided` scope on the runtime classpath when using `spring-boot:run`, such as its embedded Tomcat server. The "Play" buttons in the gutter or right-click menu of a `main()` method [do not do this](https://stackoverflow.com/questions/30237768/run-spring-boots-main-using-ide). + +A solution to this is: + +1. Open `View > Tool Windows > Maven` +2. Drill down to e.g. `Feast Core > Plugins > spring-boot:run`, right-click and `Create 'feast-core [spring-boot'…` +3. In the dialog that pops up, check the `Resolve Workspace artifacts` box +4. Click `OK`. You should now be able to select this run configuration for the Play button in the main toolbar, keyboard shortcuts, etc. + +### 2.**4** Validating your setup + +The following section is a quick walk-through to test whether your local Feast deployment is functional for development purposes. + +**2.4.1 Assumptions** + +* PostgreSQL is running in `localhost:5432` and has a database called `postgres` which + + can be accessed with credentials user `postgres` and password `password`. Different database configurations can be supplied here \(`/core/src/main/resources/application.yml`\) + +* Redis is running locally and accessible from `localhost:6379` +* \(optional\) The local environment has been authentication with Google Cloud Platform and has full access to BigQuery. This is only necessary for BigQuery testing/development. + +#### 2.4.2 Clone Feast + +```bash +git clone https://github.com/gojek/feast.git && cd feast && \ +export FEAST_HOME_DIR=$(pwd) +``` + +#### 2.4.3 Starting Feast Core + +To run Feast Core locally using Maven: + +```bash +# Feast Core can be configured from the following .yml file +# $FEAST_HOME_DIR/core/src/main/resources/application.yml mvn --projects core spring-boot:run +``` -# If Feast Core starts successfully, verify the correct Stores are registered -# correctly, for example by using grpc_cli. +Test whether Feast Core is running + +```text grpc_cli call localhost:6565 ListStores '' +``` + +The output should list **no** stores since no Feast Serving has registered its stores to Feast Core: + +```text +connecting to localhost:6565 + +Rpc succeeded with OK status +``` + +#### 2.4.4 Starting Feast Serving + +Feast Serving is configured through the `$FEAST_HOME_DIR/serving/src/main/resources/application.yml`. Each Serving deployment must be configured with a store. The default store is Redis \(used for online serving\). + +The configuration for this default store is located in a separate `.yml` file. The default location is `$FEAST_HOME_DIR/serving/sample_redis_config.yml`: -# Should return something similar to the following. -# Note that you should change BigQuery projectId and datasetId accordingly -# in "$FEAST_HOME/core/src/main/resources/application.yml" +```text +name: serving +type: REDIS +redis_config: + host: localhost + port: 6379 +subscriptions: + - name: "*" + project: "*" + version: "*" +``` + +Once Feast Serving is started, it will register its store with Feast Core \(by name\) and start to subscribe to a feature sets based on its subscription. + +Start Feast Serving GRPC server on localhost:6566 with store name `serving` + +```text +mvn --projects serving spring-boot:run +``` + +Test connectivity to Feast Serving + +```text +grpc_cli call localhost:6566 GetFeastServingInfo '' +``` +```text +connecting to localhost:6566 +version: "0.4.2-SNAPSHOT" +type: FEAST_SERVING_TYPE_ONLINE + +Rpc succeeded with OK status +``` + +Test Feast Core to see whether it is aware of the Feast Serving deployment + +```text +grpc_cli call localhost:6565 ListStores '' +``` + +```text +connecting to localhost:6565 store { - name: "SERVING" + name: "serving" type: REDIS subscriptions { + name: "*" + version: "*" + project: "*" + } + redis_config { + host: "localhost" + port: 6379 + } +} + +Rpc succeeded with OK status +``` + +In order to use BigQuery as a historical store, it is necessary to start Feast Serving with a different store type. + +Copy `$FEAST_HOME_DIR/serving/sample_redis_config.yml` to the following location `$FEAST_HOME_DIR/serving/my_bigquery_config.yml` and update the configuration as below: + +```text +name: bigquery +type: BIGQUERY +bigquery_config: + project_id: YOUR_GCP_PROJECT_ID + dataset_id: YOUR_GCP_DATASET +subscriptions: + - name: "*" + version: "*" project: "*" +``` + +Then inside `serving/src/main/resources/application.yml` modify the following key `feast.store.config-path` to point to the new store configuration. + +After making these changes, restart Feast Serving: + +```text +mvn --projects serving spring-boot:run +``` + +You should see two stores registered: + +```text +store { + name: "serving" + type: REDIS + subscriptions { name: "*" version: "*" + project: "*" } redis_config { host: "localhost" @@ -80,38 +269,21 @@ store { } } store { - name: "WAREHOUSE" + name: "bigquery" type: BIGQUERY subscriptions { - project: "*" name: "*" version: "*" + project: "*" } bigquery_config { - project_id: "my-google-project-id" - dataset_id: "my-bigquery-dataset-id" + project_id: "my_project" + dataset_id: "my_bq_dataset" } } ``` -#### Starting Feast Serving - -Feast Serving requires administrators to provide an **existing** store name in Feast. An instance of Feast Serving can only retrieve features from a **single** store. - -> In order to retrieve features from multiple stores you must start **multiple** instances of Feast serving. If you start multiple Feast serving on a single host, make sure that they are listening on different ports. - -```text -# Start Feast Serving GRPC server on localhost:6566 with store name "SERVING" -mvn --projects serving spring-boot:run -Dspring-boot.run.arguments='--feast.store-name=SERVING' - -# To verify Feast Serving starts successfully -grpc_cli call localhost:6566 GetFeastServingInfo '' - -# Should return something similar to the following. -type: FEAST_SERVING_TYPE_ONLINE -``` - -#### Registering a FeatureSet +#### 2.4.5 Registering a FeatureSet Before registering a new FeatureSet, a project is required. @@ -121,7 +293,13 @@ grpc_cli call localhost:6565 CreateProject ' ' ``` -Create a new FeatureSet on Feast by sending a request to Feast Core. When a feature set is successfully registered, Feast Core will start an **ingestion** job that listens for new features in the FeatureSet. Note that Feast currently only supports source of type "KAFKA", so you must have access to a running Kafka broker to register a FeatureSet successfully. +When a feature set is successfully registered, Feast Core will start an **ingestion** job that listens for new features in the feature set. + +{% hint style="info" %} +Note that Feast currently only supports source of type `KAFKA`, so you must have access to a running Kafka broker to register a FeatureSet successfully. It is possible to omit the `source` from a Feature Set, but Feast Core will still use Kafka behind the scenes, it is simply abstracted away from the user. +{% endhint %} + +Create a new FeatureSet in Feast by sending a request to Feast Core: ```text # Example of registering a new driver feature set @@ -155,16 +333,22 @@ feature_set { } } ' +``` + +Verify that the FeatureSet has been registered correctly. +```text # To check that the FeatureSet has been registered correctly. # You should also see logs from Feast Core of the ingestion job being started grpc_cli call localhost:6565 GetFeatureSet ' project: "your_project_name" name: "driver" ' +``` -or +Or alternatively, list all feature sets +```text grpc_cli call localhost:6565 ListFeatureSets ' filter { project: "your_project_name" @@ -174,7 +358,7 @@ grpc_cli call localhost:6565 ListFeatureSets ' ' ``` -#### Ingestion and Population of Feature Values +#### 2.4.6 Ingestion and Population of Feature Values ```text # Produce FeatureRow messages to Kafka so it will be ingested by Feast @@ -183,7 +367,7 @@ grpc_cli call localhost:6565 ListFeatureSets ' # ... producer.send("feast-driver-features" ...) # # Install Python SDK to help writing FeatureRow messages to Kafka -cd $FEAST_HOME/sdk/python +cd $FEAST_HOMEDIR/sdk/python pip3 install -e . pip3 install pendulum @@ -226,8 +410,13 @@ producer.send("your-kafka-topic", row.SerializeToString()) producer.flush() logger.info(row) EOF +``` -# Check that the ingested feature rows can be retrieved from Feast serving +#### 2.4.7 Retrieval from Feast Serving + +Ensure that Feast Serving returns results for the feature value for the specific driver + +```text grpc_cli call localhost:6566 GetOnlineFeatures ' features { project: "your_project_name" @@ -248,92 +437,32 @@ entity_rows { ' ``` -## Development - -Notes: - -* Use of Lombok is being phased out, prefer to use [Google Auto](https://github.com/google/auto) in new code. - -### Running Unit Tests - -```text -$ mvn test -``` - -### Running Integration Tests - -_Note: integration suite isn't yet separated from unit._ - -```text -$ mvn verify -``` - -### Running Components Locally - -The `core` and `serving` modules are Spring Boot applications. These may be run as usual for [the Spring Boot Maven plugin](https://docs.spring.io/spring-boot/docs/current/maven-plugin/index.html): - ```text -$ mvn --projects core spring-boot:run - -# Or for short: -$ mvn -pl core spring-boot:run +field_values { + fields { + key: "driver_id" + value { + int64_val: 1234 + } + } + fields { + key: "your_project_name/city:1" + value { + string_val: "JAKARTA" + } + } +} ``` -Note that you should execute `mvn` from the Feast repository root directory, as there are intermodule dependencies that Maven will not resolve if you `cd` to subdirectories to run. - -#### Running From IntelliJ - -Compiling and running tests in IntelliJ should work as usual. - -Running the Spring Boot apps may work out of the box in IDEA Ultimate, which has built-in support for Spring Boot projects, but the Community Edition needs a bit of help: - -The Spring Boot Maven plugin automatically puts dependencies with `provided` scope on the runtime classpath when using `spring-boot:run`, such as its embedded Tomcat server. The "Play" buttons in the gutter or right-click menu of a `main()` method [do not do this](https://stackoverflow.com/questions/30237768/run-spring-boots-main-using-ide). - -A solution to this is: - -1. Open `View > Tool Windows > Maven` -2. Drill down to e.g. `Feast Core > Plugins > spring-boot:run`, right-click and `Create 'feast-core [spring-boot'…` -3. In the dialog that pops up, check the `Resolve Workspace artifacts` box -4. Click `OK`. You should now be able to select this run configuration for the Play button in the main toolbar, keyboard shortcuts, etc. - -#### Tips for Running Postgres, Redis and Kafka with Docker +#### 2.4.8 Summary -This guide assumes you are running Docker service on a bridge network \(which is usually the case if you're running Linux\). Otherwise, you may need to use different network options than shown below. +If you have made it to this point successfully you should have a functioning Feast deployment, at the very least using the Apache Beam DirectRunner for ingestion jobs and Redis for online serving. -> `--net host` usually only works as expected when you're running Docker service in bridge networking mode. - -```text -# Start Postgres -docker run --name postgres --rm -it -d --net host -e POSTGRES_DB=postgres -e POSTGRES_USER=postgres \ --e POSTGRES_PASSWORD=password postgres:12-alpine +It is important to note that most of the functionality demonstrated above is already available in a more abstracted form in the Python SDK \(Feast management, data ingestion, feature retrieval\) and the Java/Go SDKs \(feature retrieval\). However, it is useful to understand these internals from a development standpoint. -# Start Redis -docker run --name redis --rm -it --net host -d redis:5-alpine +## 3. Style guide -# Start Zookeeper (needed by Kafka) -docker run --rm \ - --net=host \ - --name=zookeeper \ - --env=ZOOKEEPER_CLIENT_PORT=2181 \ - --detach confluentinc/cp-zookeeper:5.2.1 - -# Start Kafka -docker run --rm \ - --net=host \ - --name=kafka \ - --env=KAFKA_ZOOKEEPER_CONNECT=localhost:2181 \ - --env=KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \ - --env=KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ - --detach confluentinc/cp-kafka:5.2.1 -``` - -## Code reviews - -Code submission to Feast \(including submission from project maintainers\) requires review and approval. Please submit a **pull request** to initiate the code review process. We use [prow](https://github.com/kubernetes/test-infra/tree/master/prow) to manage the testing and reviewing of pull requests. Please refer to [config.yaml](https://github.com/gojek/feast/tree/4cd928d1d3b7972b15f0c5dd29593fcedecea9f5/.prow/config.yaml) for details on the test jobs. - -## Code conventions - -### Java +### 3.1 Java We conform to the [Google Java Style Guide](https://google.github.io/styleguide/javaguide.html). Maven can helpfully take care of that for you before you commit: @@ -348,13 +477,17 @@ $ mvn spotless:check # Check is automatic upon `mvn verify` $ mvn verify -Dspotless.check.skip ``` -If you're using IntelliJ, you can import [these code style settings](https://github.com/google/styleguide/blob/gh-pages/intellij-java-google-style.xml) if you'd like to use the IDE's reformat function as you work. +If you're using IntelliJ, you can import [these code style settings](https://github.com/google/styleguide/blob/gh-pages/intellij-java-google-style.xml) if you'd like to use the IDE's reformat function as you develop. -### Go +### 3.2 Go Make sure you apply `go fmt`. -## Release process +### 3.3 Python + +We use [Python Black](https://github.com/psf/black) to format our Python code prior to submission. + +## 4. Release process Feast uses [semantic versioning](https://semver.org/). From 98c3d5d7bc3a75a65328707d57f5e91e2db21f89 Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Fri, 14 Feb 2020 08:02:11 +0000 Subject: [PATCH 044/176] GitBook: [master] one page modified --- docs/SUMMARY.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index d2cc03a20bd..5a2ae95dd49 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -5,6 +5,7 @@ * [Concepts](concepts.md) * [Getting Help](getting-help.md) * [Contributing](contributing.md) +* [Roadmap](https://docs.google.com/document/d/1ZZY59j_c2oNN3N6TmavJIyLPMzINdea44CRIe2nhUIo/edit#) ## Installing Feast From 65f2ad7f43450cd8a3f7d64b6a1651a924881008 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Tue, 18 Feb 2020 11:56:40 +0800 Subject: [PATCH 045/176] Update comments on FeatureRow --- protos/feast/types/FeatureRow.proto | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/protos/feast/types/FeatureRow.proto b/protos/feast/types/FeatureRow.proto index 24293c6faa6..c170cd5d502 100644 --- a/protos/feast/types/FeatureRow.proto +++ b/protos/feast/types/FeatureRow.proto @@ -36,7 +36,7 @@ message FeatureRow { google.protobuf.Timestamp event_timestamp = 3; // Complete reference to the featureSet this featureRow belongs to, in the form of - // featureSetName:version. This value will be used by the feast ingestion job to filter + // /:. This value will be used by the feast ingestion job to filter // rows, and write the values to the correct tables. string feature_set = 6; -} \ No newline at end of file +} From fb39c6cc031086f3d84c893e846ace1363c6bff7 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Tue, 18 Feb 2020 22:30:47 +0800 Subject: [PATCH 046/176] Fix time range bug in basic example --- examples/basic/basic.ipynb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/basic/basic.ipynb b/examples/basic/basic.ipynb index 94fc82f2ce9..b9893011d97 100644 --- a/examples/basic/basic.ipynb +++ b/examples/basic/basic.ipynb @@ -203,7 +203,7 @@ "outputs": [], "source": [ "days = [datetime.utcnow().replace(hour=0, minute=0, second=0, microsecond=0).replace(tzinfo=utc) \\\n", - " - timedelta(day) for day in range(31)]\n", + " - timedelta(day) for day in range(3)][::-1]\n", "\n", "customers = [1001, 1002, 1003, 1004, 1005]" ] From 887f9e361c0f021e44bc096a2ba6ada1c6ab1a52 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Thu, 20 Feb 2020 16:47:38 +0800 Subject: [PATCH 047/176] Reduce refresh rate of specification refresh in Serving to 10 seconds (#481) --- .../feast/serving/configuration/SpecServiceConfig.java | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/serving/src/main/java/feast/serving/configuration/SpecServiceConfig.java b/serving/src/main/java/feast/serving/configuration/SpecServiceConfig.java index 3c91c2765aa..0b3a2938b8e 100644 --- a/serving/src/main/java/feast/serving/configuration/SpecServiceConfig.java +++ b/serving/src/main/java/feast/serving/configuration/SpecServiceConfig.java @@ -35,7 +35,7 @@ public class SpecServiceConfig { private static final Logger log = org.slf4j.LoggerFactory.getLogger(SpecServiceConfig.class); private String feastCoreHost; private int feastCorePort; - private static final int CACHE_REFRESH_RATE_MINUTES = 1; + private static final int CACHE_REFRESH_RATE_SECONDS = 10; @Autowired public SpecServiceConfig(FeastProperties feastProperties) { @@ -51,9 +51,9 @@ public ScheduledExecutorService cachedSpecServiceScheduledExecutorService( // reload all specs including new ones periodically scheduledExecutorService.scheduleAtFixedRate( cachedSpecStorage::scheduledPopulateCache, - CACHE_REFRESH_RATE_MINUTES, - CACHE_REFRESH_RATE_MINUTES, - TimeUnit.MINUTES); + CACHE_REFRESH_RATE_SECONDS, + CACHE_REFRESH_RATE_SECONDS, + TimeUnit.SECONDS); return scheduledExecutorService; } From aec7979f2986e56d7be525cc6b27a1eeb786d501 Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Thu, 20 Feb 2020 21:00:51 +0800 Subject: [PATCH 048/176] Expose PosgreSQL port in Docker Compose --- infra/docker-compose/docker-compose.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/infra/docker-compose/docker-compose.yml b/infra/docker-compose/docker-compose.yml index 27d82efc3ca..87d56cbe925 100644 --- a/infra/docker-compose/docker-compose.yml +++ b/infra/docker-compose/docker-compose.yml @@ -106,4 +106,6 @@ services: ZOOKEEPER_CLIENT_PORT: 2181 db: - image: postgres:12-alpine \ No newline at end of file + image: postgres:12-alpine + ports: + - "5432:5342" \ No newline at end of file From 636354092c3967c4b89d4c345ffc281f9f7c592d Mon Sep 17 00:00:00 2001 From: Ches Martin Date: Tue, 25 Feb 2020 06:08:40 +0700 Subject: [PATCH 049/176] Fail Spotless formatting check before tests execute (#487) * Fail formatting check before tests execute By default, the spotless Maven plugin binds its check goal to the verify phase (late in the lifecycle, after integration tests). Because we currently only run `mvn test` for CI, it doesn't proceed as far as verify so missed formatting is not caught by CI. This binds the check to an earlier phase, in between test-compile and test, so that it will fail before `mvn test` but not disrupt your dev workflow of compiling main and test sources as you work. This strikes a good compromise on failing fast for code standards without being _too_ nagging. For the complete lifecycle reference, see: https://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html * Apply spotless formatting --- .../main/java/feast/core/util/PipelineUtil.java | 1 - .../test/java/feast/ingestion/ImportJobTest.java | 15 +++++++-------- pom.xml | 10 ++++++++++ .../feast/serving/specs/CachedSpecService.java | 6 +----- 4 files changed, 18 insertions(+), 14 deletions(-) diff --git a/core/src/main/java/feast/core/util/PipelineUtil.java b/core/src/main/java/feast/core/util/PipelineUtil.java index 71cbc892bd1..8a84caf672c 100644 --- a/core/src/main/java/feast/core/util/PipelineUtil.java +++ b/core/src/main/java/feast/core/util/PipelineUtil.java @@ -72,5 +72,4 @@ private static List getClasspathFiles() { .map(entry -> new File(entry).getPath()) .collect(Collectors.toList()); } - } diff --git a/ingestion/src/test/java/feast/ingestion/ImportJobTest.java b/ingestion/src/test/java/feast/ingestion/ImportJobTest.java index 58ecae8f045..1148fa40422 100644 --- a/ingestion/src/test/java/feast/ingestion/ImportJobTest.java +++ b/ingestion/src/test/java/feast/ingestion/ImportJobTest.java @@ -32,14 +32,12 @@ import feast.core.StoreProto.Store.Subscription; import feast.ingestion.options.BZip2Compressor; import feast.ingestion.options.ImportOptions; -import feast.ingestion.options.OptionByteConverter; import feast.storage.RedisProto.RedisKey; import feast.test.TestUtil; import feast.test.TestUtil.LocalKafka; import feast.test.TestUtil.LocalRedis; import feast.types.FeatureRowProto.FeatureRow; import feast.types.ValueProto.ValueType.Enum; -import java.io.ByteArrayOutputStream; import java.io.IOException; import java.nio.charset.StandardCharsets; import java.util.ArrayList; @@ -51,7 +49,6 @@ import org.apache.beam.sdk.PipelineResult; import org.apache.beam.sdk.PipelineResult.State; import org.apache.beam.sdk.options.PipelineOptionsFactory; -import org.apache.commons.compress.compressors.bzip2.BZip2CompressorOutputStream; import org.apache.kafka.common.serialization.ByteArraySerializer; import org.joda.time.Duration; import org.junit.AfterClass; @@ -166,11 +163,13 @@ public void runPipeline_ShouldWriteToRedisCorrectlyGivenValidSpecAndFeatureRow() .build(); ImportOptions options = PipelineOptionsFactory.create().as(ImportOptions.class); - BZip2Compressor compressor = new BZip2Compressor<>(option -> { - JsonFormat.Printer printer = - JsonFormat.printer().omittingInsignificantWhitespace().printingEnumsAsInts(); - return printer.print(option).getBytes(); - }); + BZip2Compressor compressor = + new BZip2Compressor<>( + option -> { + JsonFormat.Printer printer = + JsonFormat.printer().omittingInsignificantWhitespace().printingEnumsAsInts(); + return printer.print(option).getBytes(); + }); options.setFeatureSetJson(compressor.compress(spec)); options.setStoreJson(Collections.singletonList(JsonFormat.printer().print(redis))); options.setProject(""); diff --git a/pom.xml b/pom.xml index 8e4ed7d459a..d822d367b8d 100644 --- a/pom.xml +++ b/pom.xml @@ -384,6 +384,16 @@ + + + + spotless-check + process-test-classes + + check + + + org.apache.maven.plugins diff --git a/serving/src/main/java/feast/serving/specs/CachedSpecService.java b/serving/src/main/java/feast/serving/specs/CachedSpecService.java index 1184f6da95a..35119589b27 100644 --- a/serving/src/main/java/feast/serving/specs/CachedSpecService.java +++ b/serving/src/main/java/feast/serving/specs/CachedSpecService.java @@ -195,11 +195,7 @@ private Map getFeatureToFeatureSetMapping( HashMap mapping = new HashMap<>(); featureSets.values().stream() - .collect( - groupingBy( - featureSet -> - Pair.of( - featureSet.getProject(), featureSet.getName()))) + .collect(groupingBy(featureSet -> Pair.of(featureSet.getProject(), featureSet.getName()))) .forEach( (group, groupedFeatureSets) -> { groupedFeatureSets = From 7586da644c16c2d96f63eee4783d600b013f502d Mon Sep 17 00:00:00 2001 From: David Heryanto Date: Tue, 25 Feb 2020 10:16:40 +0800 Subject: [PATCH 050/176] Fix fastavro version used in Feast to avoid Timestamp delta error (#490) * Fix fastavro version used in feast to 0.22.9 * Print python packages version used when e2e test fails --- .prow/scripts/test-end-to-end-batch.sh | 3 +++ .prow/scripts/test-end-to-end.sh | 3 +++ sdk/python/setup.py | 4 +++- 3 files changed, 9 insertions(+), 1 deletion(-) diff --git a/.prow/scripts/test-end-to-end-batch.sh b/.prow/scripts/test-end-to-end-batch.sh index 268bd248c17..47da7219daa 100755 --- a/.prow/scripts/test-end-to-end-batch.sh +++ b/.prow/scripts/test-end-to-end-batch.sh @@ -255,6 +255,9 @@ if [[ ${TEST_EXIT_CODE} != 0 ]]; then echo "[DEBUG] Printing logs" ls -ltrh /var/log/feast* cat /var/log/feast-serving-warehouse.log /var/log/feast-core.log + + echo "[DEBUG] Printing Python packages list" + pip list fi cd ${ORIGINAL_DIR} diff --git a/.prow/scripts/test-end-to-end.sh b/.prow/scripts/test-end-to-end.sh index c436d2f6905..97d5d27b5cd 100755 --- a/.prow/scripts/test-end-to-end.sh +++ b/.prow/scripts/test-end-to-end.sh @@ -229,6 +229,9 @@ if [[ ${TEST_EXIT_CODE} != 0 ]]; then echo "[DEBUG] Printing logs" ls -ltrh /var/log/feast* cat /var/log/feast-serving-online.log /var/log/feast-core.log + + echo "[DEBUG] Printing Python packages list" + pip list fi cd ${ORIGINAL_DIR} diff --git a/sdk/python/setup.py b/sdk/python/setup.py index d0b37ad9419..3fc77540c02 100644 --- a/sdk/python/setup.py +++ b/sdk/python/setup.py @@ -37,7 +37,9 @@ "pandavro==1.5.*", "protobuf>=3.10", "PyYAML==5.1.*", - "fastavro==0.*", + # fastavro 0.22.10 and newer will throw this error for e2e batch test: + # TypeError: Timestamp subtraction must have the same timezones or no timezones + "fastavro==0.22.9", "kafka-python==1.*", "tabulate==0.8.*", "toml==0.10.*", From c3591edc37dbb5afe2a5058ffbf0a2cc03f59775 Mon Sep 17 00:00:00 2001 From: Julio Anthony Leonard Date: Tue, 25 Feb 2020 12:04:40 +0700 Subject: [PATCH 051/176] Remove transaction from ingestion redis (#480) --- .../src/main/java/feast/store/serving/redis/RedisCustomIO.java | 2 -- 1 file changed, 2 deletions(-) diff --git a/ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java b/ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java index 8c142b66c93..8541baaffc3 100644 --- a/ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java +++ b/ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java @@ -238,7 +238,6 @@ private void executeBatch() throws Exception { new Retriable() { @Override public void execute() { - pipeline.multi(); mutations.forEach( mutation -> { writeRecord(mutation); @@ -246,7 +245,6 @@ public void execute() { pipeline.pexpire(mutation.getKey(), mutation.getExpiryMillis()); } }); - pipeline.exec(); pipeline.sync(); mutations.clear(); } From 5508c9230c7359ceb761c0b71ac9924e0423fbb4 Mon Sep 17 00:00:00 2001 From: David Heryanto Date: Tue, 25 Feb 2020 15:45:40 +0800 Subject: [PATCH 052/176] Extend WriteMetricsTransform in Ingestion to write feature value stats to StatsD (#486) * Extend WriteMetricsTransform to write feature value stats to StatsD * Apply mvn spotless * Catch all exception not just StatsDClientException during init Since there are other exception like UnknownHostException that can be thrown and we want to know such error. Also change the log level to error because so it's not normal for client to fail to be created" * Change log level due to invalid feature set ref to error (previously warn) On 2nd thought, this should constitute an error not a warning * Apply maven spotless to metric transform codes --- .prow/scripts/test-end-to-end.sh | 36 +- ingestion/pom.xml | 7 + .../ingestion/options/ImportOptions.java | 10 + .../metrics/WriteFeatureValueMetricsDoFn.java | 311 +++++++++++++++++ .../metrics/WriteMetricsTransform.java | 41 +++ .../metrics/WriteRowMetricsDoFn.java | 14 +- .../WriteFeatureValueMetricsDoFnTest.java | 315 ++++++++++++++++++ .../WriteFeatureValueMetricsDoFnTest.README | 9 + .../WriteFeatureValueMetricsDoFnTest.input | 4 + .../WriteFeatureValueMetricsDoFnTest.output | 66 ++++ 10 files changed, 788 insertions(+), 25 deletions(-) create mode 100644 ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java create mode 100644 ingestion/src/test/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFnTest.java create mode 100644 ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.README create mode 100644 ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.input create mode 100644 ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.output diff --git a/.prow/scripts/test-end-to-end.sh b/.prow/scripts/test-end-to-end.sh index 97d5d27b5cd..7709758345d 100755 --- a/.prow/scripts/test-end-to-end.sh +++ b/.prow/scripts/test-end-to-end.sh @@ -67,24 +67,24 @@ tail -n10 /var/log/kafka.log kafkacat -b localhost:9092 -L if [[ ${SKIP_BUILD_JARS} != "true" ]]; then - echo " - ============================================================ - Building jars for Feast - ============================================================ - " - - .prow/scripts/download-maven-cache.sh \ - --archive-uri gs://feast-templocation-kf-feast/.m2.2019-10-24.tar \ - --output-dir /root/ - - # Build jars for Feast - mvn --quiet --batch-mode --define skipTests=true clean package - - ls -lh core/target/*jar - ls -lh serving/target/*jar - else - echo "[DEBUG] Skipping building jars" - fi +echo " +============================================================ +Building jars for Feast +============================================================ +" + +.prow/scripts/download-maven-cache.sh \ + --archive-uri gs://feast-templocation-kf-feast/.m2.2019-10-24.tar \ + --output-dir /root/ + +# Build jars for Feast +mvn --quiet --batch-mode --define skipTests=true clean package + +ls -lh core/target/*jar +ls -lh serving/target/*jar +else + echo "[DEBUG] Skipping building jars" +fi echo " ============================================================ diff --git a/ingestion/pom.xml b/ingestion/pom.xml index c829674a64d..001da1a1453 100644 --- a/ingestion/pom.xml +++ b/ingestion/pom.xml @@ -248,5 +248,12 @@ 2.8.1 + + + org.apache.commons + commons-math3 + 3.6.1 + + diff --git a/ingestion/src/main/java/feast/ingestion/options/ImportOptions.java b/ingestion/src/main/java/feast/ingestion/options/ImportOptions.java index 6afdd80dd72..c1bdcd5fd17 100644 --- a/ingestion/src/main/java/feast/ingestion/options/ImportOptions.java +++ b/ingestion/src/main/java/feast/ingestion/options/ImportOptions.java @@ -26,6 +26,7 @@ /** Options passed to Beam to influence the job's execution environment */ public interface ImportOptions extends PipelineOptions, DataflowPipelineOptions, DirectOptions { + @Required @Description( "JSON string representation of the FeatureSet that the import job will process, in BZip2 binary format." @@ -83,4 +84,13 @@ public interface ImportOptions extends PipelineOptions, DataflowPipelineOptions, int getStatsdPort(); void setStatsdPort(int StatsdPort); + + @Description( + "Fixed window size in seconds (default 30) to apply before aggregation of numerical value of features" + + "and writing the aggregated value to StatsD. Refer to feast.ingestion.transform.metrics.WriteFeatureValueMetricsDoFn" + + "for details on the metric names and types.") + @Default.Integer(30) + int getWindowSizeInSecForFeatureValueMetric(); + + void setWindowSizeInSecForFeatureValueMetric(int seconds); } diff --git a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java new file mode 100644 index 00000000000..8574d2414c3 --- /dev/null +++ b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java @@ -0,0 +1,311 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.ingestion.transform.metrics; + +import static feast.ingestion.transform.metrics.WriteRowMetricsDoFn.FEATURE_SET_NAME_TAG_KEY; +import static feast.ingestion.transform.metrics.WriteRowMetricsDoFn.FEATURE_SET_PROJECT_TAG_KEY; +import static feast.ingestion.transform.metrics.WriteRowMetricsDoFn.FEATURE_SET_VERSION_TAG_KEY; +import static feast.ingestion.transform.metrics.WriteRowMetricsDoFn.FEATURE_TAG_KEY; +import static feast.ingestion.transform.metrics.WriteRowMetricsDoFn.INGESTION_JOB_NAME_KEY; +import static feast.ingestion.transform.metrics.WriteRowMetricsDoFn.METRIC_PREFIX; +import static feast.ingestion.transform.metrics.WriteRowMetricsDoFn.STORE_TAG_KEY; + +import com.google.auto.value.AutoValue; +import com.timgroup.statsd.NonBlockingStatsDClient; +import com.timgroup.statsd.StatsDClient; +import feast.types.FeatureRowProto.FeatureRow; +import feast.types.FieldProto.Field; +import feast.types.ValueProto.Value; +import java.util.ArrayList; +import java.util.DoubleSummaryStatistics; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Map.Entry; +import org.apache.beam.sdk.transforms.DoFn; +import org.apache.beam.sdk.values.KV; +import org.apache.commons.math3.stat.descriptive.rank.Percentile; +import org.slf4j.Logger; + +/** + * WriteFeatureValueMetricsDoFn accepts key value of FeatureSetRef(str) to FeatureRow(List) and + * writes a histogram of the numerical values of each feature to StatsD. + * + *

The histogram of the numerical values is represented as the following in StatsD: + * + *

    + *
  • gauge of feature_value_min + *
  • gauge of feature_value_max + *
  • gauge of feature_value_mean + *
  • gauge of feature_value_percentile_50 + *
  • gauge of feature_value_percentile_90 + *
  • gauge of feature_value_percentile_95 + *
+ * + *

StatsD timing/histogram metric type is not used since it does not support negative values. + */ +@AutoValue +public abstract class WriteFeatureValueMetricsDoFn + extends DoFn>, Void> { + + abstract String getStoreName(); + + abstract String getStatsdHost(); + + abstract int getStatsdPort(); + + static Builder newBuilder() { + return new AutoValue_WriteFeatureValueMetricsDoFn.Builder(); + } + + @AutoValue.Builder + abstract static class Builder { + + abstract Builder setStoreName(String storeName); + + abstract Builder setStatsdHost(String statsdHost); + + abstract Builder setStatsdPort(int statsdPort); + + abstract WriteFeatureValueMetricsDoFn build(); + } + + private static final Logger log = + org.slf4j.LoggerFactory.getLogger(WriteFeatureValueMetricsDoFn.class); + private StatsDClient statsDClient; + public static String GAUGE_NAME_FEATURE_VALUE_MIN = "feature_value_min"; + public static String GAUGE_NAME_FEATURE_VALUE_MAX = "feature_value_max"; + public static String GAUGE_NAME_FEATURE_VALUE_MEAN = "feature_value_mean"; + public static String GAUGE_NAME_FEATURE_VALUE_PERCENTILE_50 = "feature_value_percentile_50"; + public static String GAUGE_NAME_FEATURE_VALUE_PERCENTILE_90 = "feature_value_percentile_90"; + public static String GAUGE_NAME_FEATURE_VALUE_PERCENTILE_95 = "feature_value_percentile_95"; + + @Setup + public void setup() { + // Note that exception may be thrown during StatsD client instantiation but no exception + // will be thrown when sending metrics (mimicking the UDP protocol behaviour). + // https://jar-download.com/artifacts/com.datadoghq/java-dogstatsd-client/2.1.1/documentation + // https://github.com/DataDog/java-dogstatsd-client#unix-domain-socket-support + try { + statsDClient = new NonBlockingStatsDClient(METRIC_PREFIX, getStatsdHost(), getStatsdPort()); + } catch (Exception e) { + log.error("StatsD client cannot be started: " + e.getMessage()); + } + } + + @Teardown + public void tearDown() { + if (statsDClient != null) { + statsDClient.close(); + } + } + + @ProcessElement + public void processElement( + ProcessContext context, + @Element KV> featureSetRefToFeatureRows) { + if (statsDClient == null) { + return; + } + + String featureSetRef = featureSetRefToFeatureRows.getKey(); + if (featureSetRef == null) { + return; + } + String[] colonSplits = featureSetRef.split(":"); + if (colonSplits.length != 2) { + log.error( + "Skip writing feature value metrics because the feature set reference '{}' does not" + + "follow the required format /:", + featureSetRef); + return; + } + String[] slashSplits = colonSplits[0].split("/"); + if (slashSplits.length != 2) { + log.error( + "Skip writing feature value metrics because the feature set reference '{}' does not" + + "follow the required format /:", + featureSetRef); + return; + } + String projectName = slashSplits[0]; + String featureSetName = slashSplits[1]; + String version = colonSplits[1]; + + Map featureNameToStats = new HashMap<>(); + Map> featureNameToValues = new HashMap<>(); + for (FeatureRow featureRow : featureSetRefToFeatureRows.getValue()) { + for (Field field : featureRow.getFieldsList()) { + updateStats(featureNameToStats, featureNameToValues, field); + } + } + + for (Entry entry : featureNameToStats.entrySet()) { + String featureName = entry.getKey(); + DoubleSummaryStatistics stats = entry.getValue(); + String[] tags = { + STORE_TAG_KEY + ":" + getStoreName(), + FEATURE_SET_PROJECT_TAG_KEY + ":" + projectName, + FEATURE_SET_NAME_TAG_KEY + ":" + featureSetName, + FEATURE_SET_VERSION_TAG_KEY + ":" + version, + FEATURE_TAG_KEY + ":" + featureName, + INGESTION_JOB_NAME_KEY + ":" + context.getPipelineOptions().getJobName() + }; + + // stats can return non finite values when there is no element + // or there is an element that is not a number. Metric should only be sent for finite values. + if (Double.isFinite(stats.getMin())) { + if (stats.getMin() < 0) { + // StatsD gauge will asssign a delta instead of the actual value, if there is a sign in + // the value. E.g. if the value is negative, a delta will be assigned. For this reason, + // the gauge value is set to zero beforehand. + // https://github.com/statsd/statsd/blob/master/docs/metric_types.md#gauges + statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_MIN, 0, tags); + } + statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_MIN, stats.getMin(), tags); + } + if (Double.isFinite(stats.getMax())) { + if (stats.getMax() < 0) { + statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_MAX, 0, tags); + } + statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_MAX, stats.getMax(), tags); + } + if (Double.isFinite(stats.getAverage())) { + if (stats.getAverage() < 0) { + statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_MEAN, 0, tags); + } + statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_MEAN, stats.getAverage(), tags); + } + + // For percentile calculation, Percentile class from commons-math3 from Apache is used. + // Percentile requires double[], hence the conversion below. + if (!featureNameToValues.containsKey(featureName)) { + continue; + } + List valueList = featureNameToValues.get(featureName); + if (valueList == null || valueList.size() < 1) { + continue; + } + double[] values = new double[valueList.size()]; + for (int i = 0; i < values.length; i++) { + values[i] = valueList.get(i); + } + + double p50 = new Percentile().evaluate(values, 50); + if (p50 < 0) { + statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_PERCENTILE_50, 0, tags); + } + statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_PERCENTILE_50, p50, tags); + + double p90 = new Percentile().evaluate(values, 90); + if (p90 < 0) { + statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_PERCENTILE_90, 0, tags); + } + statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_PERCENTILE_90, p90, tags); + + double p95 = new Percentile().evaluate(values, 95); + if (p95 < 0) { + statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_PERCENTILE_95, 0, tags); + } + statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_PERCENTILE_95, p95, tags); + } + } + + // Update stats and values array for the feature represented by the field. + // If the field contains non-numerical or non-boolean value, the stats and values array + // won't get updated because we are only concerned with numerical value in metrics data. + // For boolean value, true and false are treated as numerical value of 1 of 0 respectively. + private void updateStats( + Map featureNameToStats, + Map> featureNameToValues, + Field field) { + if (featureNameToStats == null || featureNameToValues == null || field == null) { + return; + } + + String featureName = field.getName(); + if (!featureNameToStats.containsKey(featureName)) { + featureNameToStats.put(featureName, new DoubleSummaryStatistics()); + } + if (!featureNameToValues.containsKey(featureName)) { + featureNameToValues.put(featureName, new ArrayList<>()); + } + + Value value = field.getValue(); + DoubleSummaryStatistics stats = featureNameToStats.get(featureName); + List values = featureNameToValues.get(featureName); + + switch (value.getValCase()) { + case INT32_VAL: + stats.accept(value.getInt32Val()); + values.add(((double) value.getInt32Val())); + break; + case INT64_VAL: + stats.accept(value.getInt64Val()); + values.add((double) value.getInt64Val()); + break; + case DOUBLE_VAL: + stats.accept(value.getDoubleVal()); + values.add(value.getDoubleVal()); + break; + case FLOAT_VAL: + stats.accept(value.getFloatVal()); + values.add((double) value.getFloatVal()); + break; + case BOOL_VAL: + stats.accept(value.getBoolVal() ? 1 : 0); + values.add(value.getBoolVal() ? 1d : 0d); + break; + case INT32_LIST_VAL: + for (Integer val : value.getInt32ListVal().getValList()) { + stats.accept(val); + values.add(((double) val)); + } + break; + case INT64_LIST_VAL: + for (Long val : value.getInt64ListVal().getValList()) { + stats.accept(val); + values.add(((double) val)); + } + break; + case DOUBLE_LIST_VAL: + for (Double val : value.getDoubleListVal().getValList()) { + stats.accept(val); + values.add(val); + } + break; + case FLOAT_LIST_VAL: + for (Float val : value.getFloatListVal().getValList()) { + stats.accept(val); + values.add(((double) val)); + } + break; + case BOOL_LIST_VAL: + for (Boolean val : value.getBoolListVal().getValList()) { + stats.accept(val ? 1 : 0); + values.add(val ? 1d : 0d); + } + break; + case BYTES_VAL: + case BYTES_LIST_VAL: + case STRING_VAL: + case STRING_LIST_VAL: + case VAL_NOT_SET: + default: + } + } +} diff --git a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteMetricsTransform.java b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteMetricsTransform.java index 43f314aa861..10322ac812f 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteMetricsTransform.java +++ b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteMetricsTransform.java @@ -21,11 +21,16 @@ import feast.ingestion.values.FailedElement; import feast.types.FeatureRowProto.FeatureRow; import org.apache.beam.sdk.transforms.DoFn; +import org.apache.beam.sdk.transforms.GroupByKey; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; +import org.apache.beam.sdk.transforms.windowing.FixedWindows; +import org.apache.beam.sdk.transforms.windowing.Window; +import org.apache.beam.sdk.values.KV; import org.apache.beam.sdk.values.PCollectionTuple; import org.apache.beam.sdk.values.PDone; import org.apache.beam.sdk.values.TupleTag; +import org.joda.time.Duration; @AutoValue public abstract class WriteMetricsTransform extends PTransform { @@ -79,6 +84,42 @@ public PDone expand(PCollectionTuple input) { .setStoreName(getStoreName()) .build())); + // 1. Apply a fixed window + // 2. Group feature row by feature set reference + // 3. Calculate min, max, mean, percentiles of numerical values of features in the window + // and + // 4. Send the aggregate value to StatsD metric collector. + // + // NOTE: window is applied here so the metric collector will not be overwhelmed with + // metrics data. And for metric data, only statistic of the values are usually required + // vs the actual values. + input + .get(getSuccessTag()) + .apply( + "FixedWindow", + Window.into( + FixedWindows.of( + Duration.standardSeconds( + options.getWindowSizeInSecForFeatureValueMetric())))) + .apply( + "ConvertTo_FeatureSetRefToFeatureRow", + ParDo.of( + new DoFn>() { + @ProcessElement + public void processElement(ProcessContext c, @Element FeatureRow featureRow) { + c.output(KV.of(featureRow.getFeatureSet(), featureRow)); + } + })) + .apply("GroupByFeatureSetRef", GroupByKey.create()) + .apply( + "WriteFeatureValueMetrics", + ParDo.of( + WriteFeatureValueMetricsDoFn.newBuilder() + .setStatsdHost(options.getStatsdHost()) + .setStatsdPort(options.getStatsdPort()) + .setStoreName(getStoreName()) + .build())); + return PDone.in(input.getPipeline()); case "none": default: diff --git a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteRowMetricsDoFn.java b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteRowMetricsDoFn.java index db2d1acd6d8..2cd1ee94ecc 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteRowMetricsDoFn.java +++ b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteRowMetricsDoFn.java @@ -31,13 +31,13 @@ public abstract class WriteRowMetricsDoFn extends DoFn { private static final Logger log = org.slf4j.LoggerFactory.getLogger(WriteRowMetricsDoFn.class); - private final String METRIC_PREFIX = "feast_ingestion"; - private final String STORE_TAG_KEY = "feast_store"; - private final String FEATURE_SET_PROJECT_TAG_KEY = "feast_project_name"; - private final String FEATURE_SET_NAME_TAG_KEY = "feast_featureSet_name"; - private final String FEATURE_SET_VERSION_TAG_KEY = "feast_featureSet_version"; - private final String FEATURE_TAG_KEY = "feast_feature_name"; - private final String INGESTION_JOB_NAME_KEY = "ingestion_job_name"; + public static final String METRIC_PREFIX = "feast_ingestion"; + public static final String STORE_TAG_KEY = "feast_store"; + public static final String FEATURE_SET_PROJECT_TAG_KEY = "feast_project_name"; + public static final String FEATURE_SET_NAME_TAG_KEY = "feast_featureSet_name"; + public static final String FEATURE_SET_VERSION_TAG_KEY = "feast_featureSet_version"; + public static final String FEATURE_TAG_KEY = "feast_feature_name"; + public static final String INGESTION_JOB_NAME_KEY = "ingestion_job_name"; public abstract String getStoreName(); diff --git a/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFnTest.java b/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFnTest.java new file mode 100644 index 00000000000..8f0adf40168 --- /dev/null +++ b/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFnTest.java @@ -0,0 +1,315 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.ingestion.transform.metrics; + +import static org.junit.Assert.fail; + +import com.google.protobuf.ByteString; +import feast.types.FeatureRowProto.FeatureRow; +import feast.types.FeatureRowProto.FeatureRow.Builder; +import feast.types.FieldProto.Field; +import feast.types.ValueProto.BoolList; +import feast.types.ValueProto.BytesList; +import feast.types.ValueProto.DoubleList; +import feast.types.ValueProto.FloatList; +import feast.types.ValueProto.Int32List; +import feast.types.ValueProto.Int64List; +import feast.types.ValueProto.StringList; +import feast.types.ValueProto.Value; +import java.io.BufferedReader; +import java.io.IOException; +import java.net.DatagramPacket; +import java.net.DatagramSocket; +import java.net.SocketException; +import java.net.URL; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Map.Entry; +import org.apache.beam.sdk.options.PipelineOptions; +import org.apache.beam.sdk.options.PipelineOptionsFactory; +import org.apache.beam.sdk.testing.TestPipeline; +import org.apache.beam.sdk.transforms.Create; +import org.apache.beam.sdk.transforms.ParDo; +import org.junit.Rule; +import org.junit.Test; + +public class WriteFeatureValueMetricsDoFnTest { + + @Rule public final transient TestPipeline pipeline = TestPipeline.create(); + private static final int STATSD_SERVER_PORT = 17254; + private final DummyStatsDServer statsDServer = new DummyStatsDServer(STATSD_SERVER_PORT); + + @Test + public void shouldSendCorrectStatsDMetrics() throws IOException, InterruptedException { + PipelineOptions pipelineOptions = PipelineOptionsFactory.create(); + pipelineOptions.setJobName("job"); + + Map> input = + readTestInput("feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.input"); + List expectedLines = + readTestOutput("feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.output"); + + pipeline + .apply(Create.of(input)) + .apply( + ParDo.of( + WriteFeatureValueMetricsDoFn.newBuilder() + .setStatsdHost("localhost") + .setStatsdPort(STATSD_SERVER_PORT) + .setStoreName("store") + .build())); + pipeline.run(pipelineOptions).waitUntilFinish(); + // Wait until StatsD has finished processed all messages, 3 sec is a reasonable duration + // based on empirical testing. + Thread.sleep(3000); + + List actualLines = statsDServer.messagesReceived(); + for (String expected : expectedLines) { + boolean matched = false; + for (String actual : actualLines) { + if (actual.equals(expected)) { + matched = true; + break; + } + } + if (!matched) { + System.out.println("Print actual metrics output for debugging:"); + for (String line : actualLines) { + System.out.println(line); + } + fail(String.format("Expected StatsD metric not found:\n%s", expected)); + } + } + } + + // Test utility method to read expected StatsD metrics output from a text file. + @SuppressWarnings("SameParameterValue") + private List readTestOutput(String path) throws IOException { + URL url = Thread.currentThread().getContextClassLoader().getResource(path); + if (url == null) { + throw new IllegalArgumentException( + "cannot read test data, path contains null url. Path: " + path); + } + List lines = new ArrayList<>(); + try (BufferedReader reader = Files.newBufferedReader(Paths.get(url.getPath()))) { + String line = reader.readLine(); + while (line != null) { + if (line.trim().length() > 1) { + lines.add(line); + } + line = reader.readLine(); + } + } + return lines; + } + + // Test utility method to create test feature row data from a text file. + @SuppressWarnings("SameParameterValue") + private Map> readTestInput(String path) throws IOException { + Map> data = new HashMap<>(); + URL url = Thread.currentThread().getContextClassLoader().getResource(path); + if (url == null) { + throw new IllegalArgumentException( + "cannot read test data, path contains null url. Path: " + path); + } + List lines = new ArrayList<>(); + try (BufferedReader reader = Files.newBufferedReader(Paths.get(url.getPath()))) { + String line = reader.readLine(); + while (line != null) { + lines.add(line); + line = reader.readLine(); + } + } + List colNames = new ArrayList<>(); + for (String line : lines) { + if (line.strip().length() < 1) { + continue; + } + String[] splits = line.split(","); + colNames.addAll(Arrays.asList(splits)); + + if (line.startsWith("featuresetref")) { + // Header line + colNames.addAll(Arrays.asList(splits).subList(1, splits.length)); + continue; + } + + Builder featureRowBuilder = FeatureRow.newBuilder(); + for (int i = 0; i < splits.length; i++) { + String colVal = splits[i].strip(); + if (i == 0) { + featureRowBuilder.setFeatureSet(colVal); + continue; + } + String colName = colNames.get(i); + Field.Builder fieldBuilder = Field.newBuilder().setName(colName); + if (!colVal.isEmpty()) { + switch (colName) { + case "int32": + fieldBuilder.setValue(Value.newBuilder().setInt32Val((Integer.parseInt(colVal)))); + break; + case "int64": + fieldBuilder.setValue(Value.newBuilder().setInt64Val((Long.parseLong(colVal)))); + break; + case "double": + fieldBuilder.setValue(Value.newBuilder().setDoubleVal((Double.parseDouble(colVal)))); + break; + case "float": + fieldBuilder.setValue(Value.newBuilder().setFloatVal((Float.parseFloat(colVal)))); + break; + case "bool": + fieldBuilder.setValue(Value.newBuilder().setBoolVal((Boolean.parseBoolean(colVal)))); + break; + case "int32list": + List int32List = new ArrayList<>(); + for (String val : colVal.split("\\|")) { + int32List.add(Integer.parseInt(val)); + } + fieldBuilder.setValue( + Value.newBuilder().setInt32ListVal(Int32List.newBuilder().addAllVal(int32List))); + break; + case "int64list": + List int64list = new ArrayList<>(); + for (String val : colVal.split("\\|")) { + int64list.add(Long.parseLong(val)); + } + fieldBuilder.setValue( + Value.newBuilder().setInt64ListVal(Int64List.newBuilder().addAllVal(int64list))); + break; + case "doublelist": + List doubleList = new ArrayList<>(); + for (String val : colVal.split("\\|")) { + doubleList.add(Double.parseDouble(val)); + } + fieldBuilder.setValue( + Value.newBuilder() + .setDoubleListVal(DoubleList.newBuilder().addAllVal(doubleList))); + break; + case "floatlist": + List floatList = new ArrayList<>(); + for (String val : colVal.split("\\|")) { + floatList.add(Float.parseFloat(val)); + } + fieldBuilder.setValue( + Value.newBuilder().setFloatListVal(FloatList.newBuilder().addAllVal(floatList))); + break; + case "boollist": + List boolList = new ArrayList<>(); + for (String val : colVal.split("\\|")) { + boolList.add(Boolean.parseBoolean(val)); + } + fieldBuilder.setValue( + Value.newBuilder().setBoolListVal(BoolList.newBuilder().addAllVal(boolList))); + break; + case "bytes": + fieldBuilder.setValue( + Value.newBuilder().setBytesVal(ByteString.copyFromUtf8("Dummy"))); + break; + case "byteslist": + fieldBuilder.setValue( + Value.newBuilder().setBytesListVal(BytesList.getDefaultInstance())); + break; + case "string": + fieldBuilder.setValue(Value.newBuilder().setStringVal("Dummy")); + break; + case "stringlist": + fieldBuilder.setValue( + Value.newBuilder().setStringListVal(StringList.getDefaultInstance())); + break; + } + } + featureRowBuilder.addFields(fieldBuilder); + } + + if (!data.containsKey(featureRowBuilder.getFeatureSet())) { + data.put(featureRowBuilder.getFeatureSet(), new ArrayList<>()); + } + List featureRowsByFeatureSetRef = data.get(featureRowBuilder.getFeatureSet()); + featureRowsByFeatureSetRef.add(featureRowBuilder.build()); + } + + // Convert List to Iterable to match the function signature in + // WriteFeatureValueMetricsDoFn + Map> dataWithIterable = new HashMap<>(); + for (Entry> entrySet : data.entrySet()) { + String key = entrySet.getKey(); + Iterable value = entrySet.getValue(); + dataWithIterable.put(key, value); + } + return dataWithIterable; + } + + // Modified version of + // https://github.com/tim-group/java-statsd-client/blob/master/src/test/java/com/timgroup/statsd/NonBlockingStatsDClientTest.java + @SuppressWarnings("CatchMayIgnoreException") + private static final class DummyStatsDServer { + + private final List messagesReceived = new ArrayList(); + private final DatagramSocket server; + + public DummyStatsDServer(int port) { + try { + server = new DatagramSocket(port); + } catch (SocketException e) { + throw new IllegalStateException(e); + } + new Thread( + () -> { + try { + while (true) { + final DatagramPacket packet = new DatagramPacket(new byte[65535], 65535); + server.receive(packet); + messagesReceived.add( + new String(packet.getData(), StandardCharsets.UTF_8).trim() + "\n"); + Thread.sleep(50); + } + + } catch (Exception e) { + } + }) + .start(); + } + + public void stop() { + server.close(); + } + + public void waitForMessage() { + while (messagesReceived.isEmpty()) { + try { + Thread.sleep(50L); + } catch (InterruptedException e) { + } + } + } + + public List messagesReceived() { + List out = new ArrayList<>(); + for (String msg : messagesReceived) { + String[] lines = msg.split("\n"); + out.addAll(Arrays.asList(lines)); + } + return out; + } + } +} diff --git a/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.README b/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.README new file mode 100644 index 00000000000..3c8759d1702 --- /dev/null +++ b/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.README @@ -0,0 +1,9 @@ +WriteFeatureValueMetricsDoFnTest.input file contains data that can be read by test utility +into map of FeatureSetRef -> [FeatureRow]. In the first row, the cell value corresponds to the +field name in the FeatureRow. This should not be changed as the test utility derives the value +type from this name. Empty value in the cell is a value that is not set. For list type, the values +of different element is separated by the '|' character. + +WriteFeatureValueMetricsDoFnTest.output file contains lines of expected StatsD metrics that should +be sent when WriteFeatureValueMetricsDoFn runs. It can be checked against the actual outputted +StatsD metrics to test for correctness. diff --git a/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.input b/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.input new file mode 100644 index 00000000000..d2985711cee --- /dev/null +++ b/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.input @@ -0,0 +1,4 @@ +featuresetref,int32,int64,double,float,bool,int32list,int64list,doublelist,floatlist,boollist,bytes,byteslist,string,stringlist +project/featureset:1,1,5,8,5,true,1|4|3,5|1|12,5|7|3,-2.0,true|false,,,, +project/featureset:1,5,-10,8,10.0,true,1|12|5,,,-1.0|-3.0,false|true,,,, +project/featureset:1,6,-4,8,0.0,true,2,2|5,,,true|false,,,, \ No newline at end of file diff --git a/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.output b/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.output new file mode 100644 index 00000000000..63bc7bbfa4e --- /dev/null +++ b/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.output @@ -0,0 +1,66 @@ +feast_ingestion.feature_value_min:1|g|#ingestion_job_name:job,feast_feature_name:int32,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:6|g|#ingestion_job_name:job,feast_feature_name:int32,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:4|g|#ingestion_job_name:job,feast_feature_name:int32,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:5|g|#ingestion_job_name:job,feast_feature_name:int32,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:6|g|#ingestion_job_name:job,feast_feature_name:int32,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store + +feast_ingestion.feature_value_min:0|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_min:-10|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:5|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:0|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:-3|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:-4|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:5|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store + +feast_ingestion.feature_value_min:8|g|#ingestion_job_name:job,feast_feature_name:double,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:8|g|#ingestion_job_name:job,feast_feature_name:double,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:8|g|#ingestion_job_name:job,feast_feature_name:double,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:8|g|#ingestion_job_name:job,feast_feature_name:double,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:8|g|#ingestion_job_name:job,feast_feature_name:double,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store + +feast_ingestion.feature_value_min:0|g|#ingestion_job_name:job,feast_feature_name:float,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:10|g|#ingestion_job_name:job,feast_feature_name:float,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:5|g|#ingestion_job_name:job,feast_feature_name:float,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:5|g|#ingestion_job_name:job,feast_feature_name:float,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:10|g|#ingestion_job_name:job,feast_feature_name:float,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store + +feast_ingestion.feature_value_min:1|g|#ingestion_job_name:job,feast_feature_name:bool,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:1|g|#ingestion_job_name:job,feast_feature_name:bool,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:1|g|#ingestion_job_name:job,feast_feature_name:bool,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:1|g|#ingestion_job_name:job,feast_feature_name:bool,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:1|g|#ingestion_job_name:job,feast_feature_name:bool,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store + +feast_ingestion.feature_value_min:1|g|#ingestion_job_name:job,feast_feature_name:int32list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:12|g|#ingestion_job_name:job,feast_feature_name:int32list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:4|g|#ingestion_job_name:job,feast_feature_name:int32list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:3|g|#ingestion_job_name:job,feast_feature_name:int32list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:12|g|#ingestion_job_name:job,feast_feature_name:int32list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store + +feast_ingestion.feature_value_min:1|g|#ingestion_job_name:job,feast_feature_name:int64list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:12|g|#ingestion_job_name:job,feast_feature_name:int64list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:5|g|#ingestion_job_name:job,feast_feature_name:int64list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:5|g|#ingestion_job_name:job,feast_feature_name:int64list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:12|g|#ingestion_job_name:job,feast_feature_name:int64list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store + +feast_ingestion.feature_value_min:3|g|#ingestion_job_name:job,feast_feature_name:doublelist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:7|g|#ingestion_job_name:job,feast_feature_name:doublelist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:5|g|#ingestion_job_name:job,feast_feature_name:doublelist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:5|g|#ingestion_job_name:job,feast_feature_name:doublelist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:7|g|#ingestion_job_name:job,feast_feature_name:doublelist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store + +feast_ingestion.feature_value_min:0|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_min:-3|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:0|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:-1|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:0|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:-2|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:0|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:-2|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:0|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:-1|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store + +feast_ingestion.feature_value_min:0|g|#ingestion_job_name:job,feast_feature_name:boollist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:1|g|#ingestion_job_name:job,feast_feature_name:boollist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:0.5|g|#ingestion_job_name:job,feast_feature_name:boollist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:0.5|g|#ingestion_job_name:job,feast_feature_name:boollist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:1|g|#ingestion_job_name:job,feast_feature_name:boollist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store \ No newline at end of file From a576a53df4765908bdc817bbaead86abeefb92e8 Mon Sep 17 00:00:00 2001 From: Iain Rauch Date: Tue, 25 Feb 2020 23:08:40 +0000 Subject: [PATCH 053/176] Allow use of secure gRPC in Feast Python client. (#459) * Allow use of secure gRPC in Feast Python client. * Add tests for secure gRPC in Python client. --- sdk/python/feast/client.py | 83 +++++++-- sdk/python/requirements-ci.txt | 1 + sdk/python/tests/data/localhost.crt | 18 ++ sdk/python/tests/data/localhost.key | 28 +++ sdk/python/tests/data/localhost.pem | 18 ++ sdk/python/tests/test_client.py | 273 +++++++++++++++++++--------- 6 files changed, 326 insertions(+), 95 deletions(-) create mode 100644 sdk/python/tests/data/localhost.crt create mode 100644 sdk/python/tests/data/localhost.key create mode 100644 sdk/python/tests/data/localhost.pem diff --git a/sdk/python/feast/client.py b/sdk/python/feast/client.py index fb5fe6ffc49..543f0afeb64 100644 --- a/sdk/python/feast/client.py +++ b/sdk/python/feast/client.py @@ -21,7 +21,6 @@ from collections import OrderedDict from math import ceil from typing import Dict, List, Tuple, Union, Optional -from typing import List from urllib.parse import urlparse import fastavro @@ -29,6 +28,7 @@ import pandas as pd import pyarrow as pa import pyarrow.parquet as pq + from feast.core.CoreService_pb2 import ( GetFeastCoreVersionRequest, ListFeatureSetsResponse, @@ -48,11 +48,11 @@ from feast.core.FeatureSet_pb2 import FeatureSetStatus from feast.feature_set import FeatureSet, Entity from feast.job import Job -from feast.serving.ServingService_pb2 import FeatureReference from feast.loaders.abstract_producer import get_producer from feast.loaders.file import export_source_to_staging_location from feast.loaders.ingest import KAFKA_CHUNK_PRODUCTION_TIMEOUT from feast.loaders.ingest import get_feature_row_chunks +from feast.serving.ServingService_pb2 import FeatureReference from feast.serving.ServingService_pb2 import GetFeastServingInfoResponse from feast.serving.ServingService_pb2 import ( GetOnlineFeaturesRequest, @@ -69,9 +69,11 @@ GRPC_CONNECTION_TIMEOUT_DEFAULT = 3 # type: int GRPC_CONNECTION_TIMEOUT_APPLY = 600 # type: int -FEAST_SERVING_URL_ENV_KEY = "FEAST_SERVING_URL" # type: str -FEAST_CORE_URL_ENV_KEY = "FEAST_CORE_URL" # type: str -FEAST_PROJECT_ENV_KEY = "FEAST_PROJECT" # type: str +FEAST_CORE_URL_ENV_KEY = "FEAST_CORE_URL" +FEAST_SERVING_URL_ENV_KEY = "FEAST_SERVING_URL" +FEAST_PROJECT_ENV_KEY = "FEAST_PROJECT" +FEAST_CORE_SECURE_ENV_KEY = "FEAST_CORE_SECURE" +FEAST_SERVING_SECURE_ENV_KEY = "FEAST_SERVING_SECURE" BATCH_FEATURE_REQUEST_WAIT_TIME_SECONDS = 300 CPU_COUNT = os.cpu_count() # type: int @@ -82,7 +84,8 @@ class Client: """ def __init__( - self, core_url: str = None, serving_url: str = None, project: str = None + self, core_url: str = None, serving_url: str = None, project: str = None, + core_secure: bool = None, serving_secure: bool = None ): """ The Feast Client should be initialized with at least one service url @@ -91,10 +94,14 @@ def __init__( core_url: Feast Core URL. Used to manage features serving_url: Feast Serving URL. Used to retrieve features project: Sets the active project. This field is optional. - """ - self._core_url = core_url - self._serving_url = serving_url - self._project = project + core_secure: Use client-side SSL/TLS for Core gRPC API + serving_secure: Use client-side SSL/TLS for Serving gRPC API + """ + self._core_url: str = core_url + self._serving_url: str = serving_url + self._project: str = project + self._core_secure: bool = core_secure + self._serving_secure: bool = serving_secure self.__core_channel: grpc.Channel = None self.__serving_channel: grpc.Channel = None self._core_service_stub: CoreServiceStub = None @@ -149,6 +156,52 @@ def serving_url(self, value: str): """ self._serving_url = value + @property + def core_secure(self) -> bool: + """ + Retrieve Feast Core client-side SSL/TLS setting + + Returns: + Whether client-side SSL/TLS is enabled + """ + + if self._core_secure is not None: + return self._core_secure + return os.getenv(FEAST_CORE_SECURE_ENV_KEY, "").lower() is "true" + + @core_secure.setter + def core_secure(self, value: bool): + """ + Set the Feast Core client-side SSL/TLS setting + + Args: + value: True to enable client-side SSL/TLS + """ + self._core_secure = value + + @property + def serving_secure(self) -> bool: + """ + Retrieve Feast Serving client-side SSL/TLS setting + + Returns: + Whether client-side SSL/TLS is enabled + """ + + if self._serving_secure is not None: + return self._serving_secure + return os.getenv(FEAST_SERVING_SECURE_ENV_KEY, "").lower() is "true" + + @serving_secure.setter + def serving_secure(self, value: bool): + """ + Set the Feast Serving client-side SSL/TLS setting + + Args: + value: True to enable client-side SSL/TLS + """ + self._serving_secure = value + def version(self): """ Returns version information from Feast Core and Feast Serving @@ -185,7 +238,10 @@ def _connect_core(self, skip_if_connected: bool = True): raise ValueError("Please set Feast Core URL.") if self.__core_channel is None: - self.__core_channel = grpc.insecure_channel(self.core_url) + if self.core_secure or self.core_url.endswith(":443"): + self.__core_channel = grpc.secure_channel(self.core_url, grpc.ssl_channel_credentials()) + else: + self.__core_channel = grpc.insecure_channel(self.core_url) try: grpc.channel_ready_future(self.__core_channel).result( @@ -214,7 +270,10 @@ def _connect_serving(self, skip_if_connected=True): raise ValueError("Please set Feast Serving URL.") if self.__serving_channel is None: - self.__serving_channel = grpc.insecure_channel(self.serving_url) + if self.serving_secure or self.serving_url.endswith(":443"): + self.__serving_channel = grpc.secure_channel(self.serving_url, grpc.ssl_channel_credentials()) + else: + self.__serving_channel = grpc.insecure_channel(self.serving_url) try: grpc.channel_ready_future(self.__serving_channel).result( diff --git a/sdk/python/requirements-ci.txt b/sdk/python/requirements-ci.txt index d0fdd76e498..31818ba7f7b 100644 --- a/sdk/python/requirements-ci.txt +++ b/sdk/python/requirements-ci.txt @@ -12,6 +12,7 @@ mock==2.0.0 pandas==0.* protobuf==3.* pytest +pytest-lazy-fixture==0.6.3 pytest-mock pytest-timeout PyYAML==5.1.* diff --git a/sdk/python/tests/data/localhost.crt b/sdk/python/tests/data/localhost.crt new file mode 100644 index 00000000000..1f471506aab --- /dev/null +++ b/sdk/python/tests/data/localhost.crt @@ -0,0 +1,18 @@ +-----BEGIN CERTIFICATE----- +MIIC5zCCAc+gAwIBAgIJAKzukpnyuwsVMA0GCSqGSIb3DQEBCwUAMBQxEjAQBgNV +BAMMCWxvY2FsaG9zdDAgFw0yMDAyMTcxMTE4NDNaGA8zMDE5MDYyMDExMTg0M1ow +FDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB +CgKCAQEAqoanhiy4EUZjPA/m8IWk50OyTjKAnqZvEW5glqmTHP6lQbfyWQnzj3Ny +c++4Xn901FO2v07h+7lE3BScjgCX6klsLOHRnWcLX8lQygR6zzO+Oey1yXuCebBA +yhrsqgTDC/8zoCxe0W3t0vqvE4AJs3tJHq5Y1ba/X9OiKKsDZuMSSsbdd4qVEL6y +BD8PRNLT/iiD84Kq58GZtOI3fJls8E/bYbvksugcPI3kmlU4Plg3VrVplMl3DcMz +7BbvQP6jmVqdPtUT7+lL0C5CsNqbdDOIwg09+Gwus+A/g8PerBBd+ZCmdvSa9LYJ +OmlJszgZPIL9AagXLfuGQvNN2Y6WowIDAQABozowODAUBgNVHREEDTALgglsb2Nh +bGhvc3QwCwYDVR0PBAQDAgeAMBMGA1UdJQQMMAoGCCsGAQUFBwMBMA0GCSqGSIb3 +DQEBCwUAA4IBAQAuF1/VeQL73Y1FKrBX4bAb/Rdh2+Dadpi+w1pgEOi3P4udmQ+y +Xn9GwwLRQmHRLjyCT5KT8lNHdldPdlBamqPGGku449aCAjA/YHVHhcHaXl0MtPGq +BfKhHYSsvI2sIymlzZIvvIaf04yuJ1g+L0j8Px4Ecor9YwcKDZmpnIXLgdUtUrIQ +5Omrb4jImX6q8jp6Bjplb4H3o4TqKoa74NLOWUiH5/Rix3Lo8MRoEVbX2GhKk+8n +0eD3AuyrI1i+ce7zY8qGJKKFHGLDWPA/+006ZIS4j/Hr2FWo07CPFQ4/3gdJ8Erw +SzgO9vvIhQrBJn2CIH4+P5Cb1ktdobNWW9XK +-----END CERTIFICATE----- diff --git a/sdk/python/tests/data/localhost.key b/sdk/python/tests/data/localhost.key new file mode 100644 index 00000000000..dbd9cda062c --- /dev/null +++ b/sdk/python/tests/data/localhost.key @@ -0,0 +1,28 @@ +-----BEGIN PRIVATE KEY----- +MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCqhqeGLLgRRmM8 +D+bwhaTnQ7JOMoCepm8RbmCWqZMc/qVBt/JZCfOPc3Jz77hef3TUU7a/TuH7uUTc +FJyOAJfqSWws4dGdZwtfyVDKBHrPM7457LXJe4J5sEDKGuyqBMML/zOgLF7Rbe3S ++q8TgAmze0kerljVtr9f06IoqwNm4xJKxt13ipUQvrIEPw9E0tP+KIPzgqrnwZm0 +4jd8mWzwT9thu+Sy6Bw8jeSaVTg+WDdWtWmUyXcNwzPsFu9A/qOZWp0+1RPv6UvQ +LkKw2pt0M4jCDT34bC6z4D+Dw96sEF35kKZ29Jr0tgk6aUmzOBk8gv0BqBct+4ZC +803ZjpajAgMBAAECggEADE4FHphxe8WheX8IQgjSumFXJ29bepc14oMdcyGvXOM/ +F3vnf+dI7Ov+sUD2A9OcoYmc4TcW9WwL/Pl7xn9iduRvatmsn3gFCRdkvf8OwY7R +Riq/f1drNc6zDiJdO3N2g5IZrpAlE2WkSJoQMg8GJC5cO1uHS3yRWJ/Tzq1wZGcW +Dot9hAFgN0qNdP0xFkOsPM5ptC3DjLqsZWboJhIM19hgsIYaWQWHvcYlCcWTVhkj +FYzvLj5GrzAgyE89RpdXus670q5E2R2Rlnja21TfcxK0UOdIrKghZ0jxZMsXEwdB +8V7kIzL5kh//RhT/dIt0mHNMSdLFFx3yMTb2wTzpWQKBgQDRiCRslDSjiNSFySkn +6IivAwJtV2gLSxV05D9u9lrrlskHogrZUJkpVF1VzSnwv/ASaCZX4AGTtNPaz+vy +yDviwfjADsuum8jkzoxKCHnR1HVMyX+vm/g+pE20PMskTUuDE4zROtrqo9Ky0afv +94mJrf93Q815rsbEM5osugaeBQKBgQDQWAPTKy1wcG7edwfu3EaLYHPZ8pW9MldP +FvCLTMwSDkSzU+wA4BGE/5Tuu0WHSAfUc5C1LnMQXKBQXun+YCaBR6GZjUAmntz3 +poBIOYaxe651zqzCmo4ip1h5wIfPvynsyGmhsbpDSNhvXFgH2mF3XSY1nduKSRHu +389cHk3ahwKBgA4gAWSYcRv9I2aJcw7PrDcwGr/IPqlUPHQO1v/h96seFRtAnz6b +IlgY6dnY5NTn+4UiJEOUREbyz71Weu949CCLNvurg6uXsOlLy0VKYPv2OJoek08B +UrDWXq6h0of19fs2HC4Wq59Zv+ByJcIVi94OLsSZe4aSc6/SUrhlKgEJAoGBAIvR +5Y88NNx2uBEYdPx6W+WBr34e7Rrxw+JSFNCHk5SyeqyWr5XOyjMliv/EMl8dmhOc +Ewtkxte+MeB+Mi8CvBSay/rO7rR8fPK+jOzrnldSF7z8HLjlHGppQFlFOl/TfQFp +ZmqbadNp+caShImQp0SCAPiOnh1p+F0FWpYJyFnVAoGAKhSRP0iUmd+tId94px2m +G248BhcM9/0r+Y3yRX1eBx5eBzlzPUPcW1MSbhiZ1DIyLZ/MyObl98A1oNBGun11 +H/7Mq0E8BcJoXmt/6Z+2NhREBV9tDNuINyS/coYBV7H50pnSqyPpREPxNmu3Ukbm +u7ggLRfH+DexDysbpbCZ9l4= +-----END PRIVATE KEY----- diff --git a/sdk/python/tests/data/localhost.pem b/sdk/python/tests/data/localhost.pem new file mode 100644 index 00000000000..1f471506aab --- /dev/null +++ b/sdk/python/tests/data/localhost.pem @@ -0,0 +1,18 @@ +-----BEGIN CERTIFICATE----- +MIIC5zCCAc+gAwIBAgIJAKzukpnyuwsVMA0GCSqGSIb3DQEBCwUAMBQxEjAQBgNV +BAMMCWxvY2FsaG9zdDAgFw0yMDAyMTcxMTE4NDNaGA8zMDE5MDYyMDExMTg0M1ow +FDESMBAGA1UEAwwJbG9jYWxob3N0MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIB +CgKCAQEAqoanhiy4EUZjPA/m8IWk50OyTjKAnqZvEW5glqmTHP6lQbfyWQnzj3Ny +c++4Xn901FO2v07h+7lE3BScjgCX6klsLOHRnWcLX8lQygR6zzO+Oey1yXuCebBA +yhrsqgTDC/8zoCxe0W3t0vqvE4AJs3tJHq5Y1ba/X9OiKKsDZuMSSsbdd4qVEL6y +BD8PRNLT/iiD84Kq58GZtOI3fJls8E/bYbvksugcPI3kmlU4Plg3VrVplMl3DcMz +7BbvQP6jmVqdPtUT7+lL0C5CsNqbdDOIwg09+Gwus+A/g8PerBBd+ZCmdvSa9LYJ +OmlJszgZPIL9AagXLfuGQvNN2Y6WowIDAQABozowODAUBgNVHREEDTALgglsb2Nh +bGhvc3QwCwYDVR0PBAQDAgeAMBMGA1UdJQQMMAoGCCsGAQUFBwMBMA0GCSqGSIb3 +DQEBCwUAA4IBAQAuF1/VeQL73Y1FKrBX4bAb/Rdh2+Dadpi+w1pgEOi3P4udmQ+y +Xn9GwwLRQmHRLjyCT5KT8lNHdldPdlBamqPGGku449aCAjA/YHVHhcHaXl0MtPGq +BfKhHYSsvI2sIymlzZIvvIaf04yuJ1g+L0j8Px4Ecor9YwcKDZmpnIXLgdUtUrIQ +5Omrb4jImX6q8jp6Bjplb4H3o4TqKoa74NLOWUiH5/Rix3Lo8MRoEVbX2GhKk+8n +0eD3AuyrI1i+ce7zY8qGJKKFHGLDWPA/+006ZIS4j/Hr2FWo07CPFQ4/3gdJ8Erw +SzgO9vvIhQrBJn2CIH4+P5Cb1ktdobNWW9XK +-----END CERTIFICATE----- diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index 123cbe47fd6..2724fff52e3 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -11,9 +11,12 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. +import pkgutil from datetime import datetime import tempfile +from unittest import mock + import grpc import pandas as pd from google.protobuf.duration_pb2 import Duration @@ -63,10 +66,38 @@ CORE_URL = "core.feast.example.com" SERVING_URL = "serving.example.com" +_PRIVATE_KEY_RESOURCE_PATH = 'data/localhost.key' +_CERTIFICATE_CHAIN_RESOURCE_PATH = 'data/localhost.pem' +_ROOT_CERTIFICATE_RESOURCE_PATH = 'data/localhost.crt' class TestClient: - @pytest.fixture(scope="function") + + @pytest.fixture + def secure_mock_client(self, mocker): + client = Client(core_url=CORE_URL, serving_url=SERVING_URL, core_secure=True, serving_secure=True) + mocker.patch.object(client, "_connect_core") + mocker.patch.object(client, "_connect_serving") + client._core_url = CORE_URL + client._serving_url = SERVING_URL + return client + + @pytest.fixture + def mock_client(self, mocker): + client = Client(core_url=CORE_URL, serving_url=SERVING_URL) + mocker.patch.object(client, "_connect_core") + mocker.patch.object(client, "_connect_serving") + client._core_url = CORE_URL + client._serving_url = SERVING_URL + return client + + @pytest.fixture + def server_credentials(self): + private_key = pkgutil.get_data(__name__, _PRIVATE_KEY_RESOURCE_PATH) + certificate_chain = pkgutil.get_data(__name__, _CERTIFICATE_CHAIN_RESOURCE_PATH) + return grpc.ssl_server_credentials(((private_key, certificate_chain),)) + + @pytest.fixture def core_server(self): server = grpc.server(futures.ThreadPoolExecutor(max_workers=10)) Core.add_CoreServiceServicer_to_server(CoreServicer(), server) @@ -75,7 +106,7 @@ def core_server(self): yield server server.stop(0) - @pytest.fixture(scope="function") + @pytest.fixture def serving_server(self): server = grpc.server(futures.ThreadPoolExecutor(max_workers=10)) Serving.add_ServingServiceServicer_to_server(ServingServicer(), server) @@ -85,48 +116,73 @@ def serving_server(self): server.stop(0) @pytest.fixture - def mock_client(self, mocker): - client = Client(core_url=CORE_URL, serving_url=SERVING_URL) - mocker.patch.object(client, "_connect_core") - mocker.patch.object(client, "_connect_serving") - client._core_url = CORE_URL - client._serving_url = SERVING_URL - return client + def secure_core_server(self, server_credentials): + server = grpc.server(futures.ThreadPoolExecutor(max_workers=10)) + Core.add_CoreServiceServicer_to_server(CoreServicer(), server) + server.add_secure_port("[::]:50053", server_credentials) + server.start() + yield server + server.stop(0) + + @pytest.fixture + def secure_serving_server(self, server_credentials): + server = grpc.server(futures.ThreadPoolExecutor(max_workers=10)) + Serving.add_ServingServiceServicer_to_server(ServingServicer(), server) + + server.add_secure_port("[::]:50054", server_credentials) + server.start() + yield server + server.stop(0) + + @pytest.fixture + def secure_client(self, secure_core_server, secure_serving_server): + root_certificate_credentials = pkgutil.get_data(__name__, _ROOT_CERTIFICATE_RESOURCE_PATH) + # this is needed to establish a secure connection using self-signed certificates, for the purpose of the test + ssl_channel_credentials = grpc.ssl_channel_credentials(root_certificates=root_certificate_credentials) + with mock.patch("grpc.ssl_channel_credentials", MagicMock(return_value=ssl_channel_credentials)): + yield Client(core_url="localhost:50053", serving_url="localhost:50054", core_secure=True, + serving_secure=True) @pytest.fixture def client(self, core_server, serving_server): return Client(core_url="localhost:50051", serving_url="localhost:50052") - def test_version(self, mock_client, mocker): - mock_client._core_service_stub = Core.CoreServiceStub(grpc.insecure_channel("")) - mock_client._serving_service_stub = Serving.ServingServiceStub( + @pytest.mark.parametrize("mocked_client", [pytest.lazy_fixture("mock_client"), + pytest.lazy_fixture("secure_mock_client") + ]) + def test_version(self, mocked_client, mocker): + mocked_client._core_service_stub = Core.CoreServiceStub(grpc.insecure_channel("")) + mocked_client._serving_service_stub = Serving.ServingServiceStub( grpc.insecure_channel("") ) mocker.patch.object( - mock_client._core_service_stub, + mocked_client._core_service_stub, "GetFeastCoreVersion", return_value=GetFeastCoreVersionResponse(version="0.3.2"), ) mocker.patch.object( - mock_client._serving_service_stub, + mocked_client._serving_service_stub, "GetFeastServingInfo", return_value=GetFeastServingInfoResponse(version="0.3.2"), ) - status = mock_client.version() + status = mocked_client.version() assert ( - status["core"]["url"] == CORE_URL - and status["core"]["version"] == "0.3.2" - and status["serving"]["url"] == SERVING_URL - and status["serving"]["version"] == "0.3.2" + status["core"]["url"] == CORE_URL + and status["core"]["version"] == "0.3.2" + and status["serving"]["url"] == SERVING_URL + and status["serving"]["version"] == "0.3.2" ) - def test_get_online_features(self, mock_client, mocker): + @pytest.mark.parametrize("mocked_client", [pytest.lazy_fixture("mock_client"), + pytest.lazy_fixture("secure_mock_client") + ]) + def test_get_online_features(self, mocked_client, mocker): ROW_COUNT = 300 - mock_client._serving_service_stub = Serving.ServingServiceStub( + mocked_client._serving_service_stub = Serving.ServingServiceStub( grpc.insecure_channel("") ) @@ -148,12 +204,12 @@ def test_get_online_features(self, mock_client, mocker): ) mocker.patch.object( - mock_client._serving_service_stub, + mocked_client._serving_service_stub, "GetOnlineFeatures", return_value=response, ) - response = mock_client.get_online_features( + response = mocked_client.get_online_features( entity_rows=entity_rows, feature_refs=[ "my_project/feature_1:1", @@ -169,17 +225,20 @@ def test_get_online_features(self, mock_client, mocker): ) # type: GetOnlineFeaturesResponse assert ( - response.field_values[0].fields["my_project/feature_1:1"].int64_val == 1 - and response.field_values[0].fields["my_project/feature_9:1"].int64_val == 9 + response.field_values[0].fields["my_project/feature_1:1"].int64_val == 1 + and response.field_values[0].fields["my_project/feature_9:1"].int64_val == 9 ) - def test_get_feature_set(self, mock_client, mocker): - mock_client._core_service_stub = Core.CoreServiceStub(grpc.insecure_channel("")) + @pytest.mark.parametrize("mocked_client", [pytest.lazy_fixture("mock_client"), + pytest.lazy_fixture("secure_mock_client") + ]) + def test_get_feature_set(self, mocked_client, mocker): + mocked_client._core_service_stub = Core.CoreServiceStub(grpc.insecure_channel("")) from google.protobuf.duration_pb2 import Duration mocker.patch.object( - mock_client._core_service_stub, + mocked_client._core_service_stub, "GetFeatureSet", return_value=GetFeatureSetResponse( feature_set=FeatureSetProto( @@ -214,29 +273,32 @@ def test_get_feature_set(self, mock_client, mocker): ) ), ) - mock_client.set_project("my_project") - feature_set = mock_client.get_feature_set("my_feature_set", version=2) + mocked_client.set_project("my_project") + feature_set = mocked_client.get_feature_set("my_feature_set", version=2) assert ( - feature_set.name == "my_feature_set" - and feature_set.version == 2 - and feature_set.fields["my_feature_1"].name == "my_feature_1" - and feature_set.fields["my_feature_1"].dtype == ValueType.FLOAT - and feature_set.fields["my_entity_1"].name == "my_entity_1" - and feature_set.fields["my_entity_1"].dtype == ValueType.INT64 - and len(feature_set.features) == 2 - and len(feature_set.entities) == 1 + feature_set.name == "my_feature_set" + and feature_set.version == 2 + and feature_set.fields["my_feature_1"].name == "my_feature_1" + and feature_set.fields["my_feature_1"].dtype == ValueType.FLOAT + and feature_set.fields["my_entity_1"].name == "my_entity_1" + and feature_set.fields["my_entity_1"].dtype == ValueType.INT64 + and len(feature_set.features) == 2 + and len(feature_set.entities) == 1 ) - def test_get_batch_features(self, mock_client, mocker): + @pytest.mark.parametrize("mocked_client", [pytest.lazy_fixture("mock_client"), + pytest.lazy_fixture("secure_mock_client") + ]) + def test_get_batch_features(self, mocked_client, mocker): - mock_client._serving_service_stub = Serving.ServingServiceStub( + mocked_client._serving_service_stub = Serving.ServingServiceStub( grpc.insecure_channel("") ) - mock_client._core_service_stub = Core.CoreServiceStub(grpc.insecure_channel("")) + mocked_client._core_service_stub = Core.CoreServiceStub(grpc.insecure_channel("")) mocker.patch.object( - mock_client._core_service_stub, + mocked_client._core_service_stub, "GetFeatureSet", return_value=GetFeatureSetResponse( feature_set=FeatureSetProto( @@ -283,7 +345,7 @@ def test_get_batch_features(self, mock_client, mocker): to_avro(file_path_or_buffer=final_results, df=expected_dataframe) mocker.patch.object( - mock_client._serving_service_stub, + mocked_client._serving_service_stub, "GetBatchFeatures", return_value=GetBatchFeaturesResponse( job=BatchFeaturesJob( @@ -297,7 +359,7 @@ def test_get_batch_features(self, mock_client, mocker): ) mocker.patch.object( - mock_client._serving_service_stub, + mocked_client._serving_service_stub, "GetJob", return_value=GetJobResponse( job=BatchFeaturesJob( @@ -311,7 +373,7 @@ def test_get_batch_features(self, mock_client, mocker): ) mocker.patch.object( - mock_client._serving_service_stub, + mocked_client._serving_service_stub, "GetFeastServingInfo", return_value=GetFeastServingInfoResponse( job_staging_location=f"file://{tempfile.mkdtemp()}/", @@ -319,8 +381,8 @@ def test_get_batch_features(self, mock_client, mocker): ), ) - mock_client.set_project("project1") - response = mock_client.get_batch_features( + mocked_client.set_project("project1") + response = mocked_client.get_batch_features( entity_rows=pd.DataFrame( { "datetime": [ @@ -348,9 +410,12 @@ def test_get_batch_features(self, mock_client, mocker): ] ) - def test_apply_feature_set_success(self, client): + @pytest.mark.parametrize("test_client", [pytest.lazy_fixture("client"), + pytest.lazy_fixture("secure_client") + ]) + def test_apply_feature_set_success(self, test_client): - client.set_project("project1") + test_client.set_project("project1") # Create Feature Sets fs1 = FeatureSet("my-feature-set-1") @@ -364,23 +429,24 @@ def test_apply_feature_set_success(self, client): fs2.add(Entity(name="fs2-my-entity-1", dtype=ValueType.INT64)) # Register Feature Set with Core - client.apply(fs1) - client.apply(fs2) + test_client.apply(fs1) + test_client.apply(fs2) - feature_sets = client.list_feature_sets() + feature_sets = test_client.list_feature_sets() # List Feature Sets assert ( - len(feature_sets) == 2 - and feature_sets[0].name == "my-feature-set-1" - and feature_sets[0].features[0].name == "fs1-my-feature-1" - and feature_sets[0].features[0].dtype == ValueType.INT64 - and feature_sets[1].features[1].dtype == ValueType.BYTES_LIST + len(feature_sets) == 2 + and feature_sets[0].name == "my-feature-set-1" + and feature_sets[0].features[0].name == "fs1-my-feature-1" + and feature_sets[0].features[0].dtype == ValueType.INT64 + and feature_sets[1].features[1].dtype == ValueType.BYTES_LIST ) - @pytest.mark.parametrize("dataframe", [dataframes.GOOD]) - def test_feature_set_ingest_success(self, dataframe, client, mocker): - client.set_project("project1") + @pytest.mark.parametrize("dataframe,test_client", [(dataframes.GOOD, pytest.lazy_fixture("client")), + (dataframes.GOOD, pytest.lazy_fixture("secure_client"))]) + def test_feature_set_ingest_success(self, dataframe, test_client, mocker): + test_client.set_project("project1") driver_fs = FeatureSet( "driver-feature-set", source=KafkaSource(brokers="kafka:9092", topic="test") ) @@ -390,12 +456,12 @@ def test_feature_set_ingest_success(self, dataframe, client, mocker): driver_fs.add(Entity(name="entity_id", dtype=ValueType.INT64)) # Register with Feast core - client.apply(driver_fs) + test_client.apply(driver_fs) driver_fs = driver_fs.to_proto() driver_fs.meta.status = FeatureSetStatusProto.STATUS_READY mocker.patch.object( - client._core_service_stub, + test_client._core_service_stub, "GetFeatureSet", return_value=GetFeatureSetResponse(feature_set=driver_fs), ) @@ -403,14 +469,16 @@ def test_feature_set_ingest_success(self, dataframe, client, mocker): # Need to create a mock producer with patch("feast.client.get_producer") as mocked_queue: # Ingest data into Feast - client.ingest("driver-feature-set", dataframe) + test_client.ingest("driver-feature-set", dataframe) - @pytest.mark.parametrize("dataframe,exception", [(dataframes.GOOD, TimeoutError)]) + @pytest.mark.parametrize("dataframe,exception,test_client", + [(dataframes.GOOD, TimeoutError, pytest.lazy_fixture("client")), + (dataframes.GOOD, TimeoutError, pytest.lazy_fixture("secure_client"))]) def test_feature_set_ingest_fail_if_pending( - self, dataframe, exception, client, mocker + self, dataframe, exception, test_client, mocker ): with pytest.raises(exception): - client.set_project("project1") + test_client.set_project("project1") driver_fs = FeatureSet( "driver-feature-set", source=KafkaSource(brokers="kafka:9092", topic="test"), @@ -421,12 +489,12 @@ def test_feature_set_ingest_fail_if_pending( driver_fs.add(Entity(name="entity_id", dtype=ValueType.INT64)) # Register with Feast core - client.apply(driver_fs) + test_client.apply(driver_fs) driver_fs = driver_fs.to_proto() driver_fs.meta.status = FeatureSetStatusProto.STATUS_PENDING mocker.patch.object( - client._core_service_stub, + test_client._core_service_stub, "GetFeatureSet", return_value=GetFeatureSetResponse(feature_set=driver_fs), ) @@ -434,18 +502,22 @@ def test_feature_set_ingest_fail_if_pending( # Need to create a mock producer with patch("feast.client.get_producer") as mocked_queue: # Ingest data into Feast - client.ingest("driver-feature-set", dataframe, timeout=1) + test_client.ingest("driver-feature-set", dataframe, timeout=1) @pytest.mark.parametrize( - "dataframe,exception", + "dataframe,exception,test_client", [ - (dataframes.BAD_NO_DATETIME, Exception), - (dataframes.BAD_INCORRECT_DATETIME_TYPE, Exception), - (dataframes.BAD_NO_ENTITY, Exception), - (dataframes.NO_FEATURES, Exception), + (dataframes.BAD_NO_DATETIME, Exception, pytest.lazy_fixture("client")), + (dataframes.BAD_INCORRECT_DATETIME_TYPE, Exception, pytest.lazy_fixture("client")), + (dataframes.BAD_NO_ENTITY, Exception, pytest.lazy_fixture("client")), + (dataframes.NO_FEATURES, Exception, pytest.lazy_fixture("client")), + (dataframes.BAD_NO_DATETIME, Exception, pytest.lazy_fixture("secure_client")), + (dataframes.BAD_INCORRECT_DATETIME_TYPE, Exception, pytest.lazy_fixture("secure_client")), + (dataframes.BAD_NO_ENTITY, Exception, pytest.lazy_fixture("secure_client")), + (dataframes.NO_FEATURES, Exception, pytest.lazy_fixture("secure_client")), ], ) - def test_feature_set_ingest_failure(self, client, dataframe, exception): + def test_feature_set_ingest_failure(self, test_client, dataframe, exception): with pytest.raises(exception): # Create feature set driver_fs = FeatureSet("driver-feature-set") @@ -454,15 +526,16 @@ def test_feature_set_ingest_failure(self, client, dataframe, exception): driver_fs.infer_fields_from_df(dataframe) # Register with Feast core - client.apply(driver_fs) + test_client.apply(driver_fs) # Ingest data into Feast - client.ingest(driver_fs, dataframe=dataframe) + test_client.ingest(driver_fs, dataframe=dataframe) - @pytest.mark.parametrize("dataframe", [dataframes.ALL_TYPES]) - def test_feature_set_types_success(self, client, dataframe, mocker): + @pytest.mark.parametrize("dataframe,test_client", [(dataframes.ALL_TYPES, pytest.lazy_fixture("client")), + (dataframes.ALL_TYPES, pytest.lazy_fixture("secure_client"))]) + def test_feature_set_types_success(self, test_client, dataframe, mocker): - client.set_project("project1") + test_client.set_project("project1") all_types_fs = FeatureSet( name="all_types", @@ -489,10 +562,10 @@ def test_feature_set_types_success(self, client, dataframe, mocker): ) # Register with Feast core - client.apply(all_types_fs) + test_client.apply(all_types_fs) mocker.patch.object( - client._core_service_stub, + test_client._core_service_stub, "GetFeatureSet", return_value=GetFeatureSetResponse(feature_set=all_types_fs.to_proto()), ) @@ -500,4 +573,38 @@ def test_feature_set_types_success(self, client, dataframe, mocker): # Need to create a mock producer with patch("feast.client.get_producer") as mocked_queue: # Ingest data into Feast - client.ingest(all_types_fs, dataframe) + test_client.ingest(all_types_fs, dataframe) + + @patch("grpc.channel_ready_future") + def test_secure_channel_creation_with_secure_client(self, _mocked_obj): + client = Client(core_url="localhost:50051", serving_url="localhost:50052", serving_secure=True, + core_secure=True) + with mock.patch("grpc.secure_channel") as _grpc_mock, \ + mock.patch("grpc.ssl_channel_credentials", MagicMock(return_value="test")) as _mocked_credentials: + client._connect_serving() + _grpc_mock.assert_called_with(client.serving_url, _mocked_credentials.return_value) + + @mock.patch("grpc.channel_ready_future") + def test_secure_channel_creation_with_secure_serving_url(self, _mocked_obj, ): + client = Client(core_url="localhost:50051", serving_url="localhost:443") + with mock.patch("grpc.secure_channel") as _grpc_mock, \ + mock.patch("grpc.ssl_channel_credentials", MagicMock(return_value="test")) as _mocked_credentials: + client._connect_serving() + _grpc_mock.assert_called_with(client.serving_url, _mocked_credentials.return_value) + + @patch("grpc.channel_ready_future") + def test_secure_channel_creation_with_secure_client(self, _mocked_obj): + client = Client(core_url="localhost:50053", serving_url="localhost:50054", serving_secure=True, + core_secure=True) + with mock.patch("grpc.secure_channel") as _grpc_mock, \ + mock.patch("grpc.ssl_channel_credentials", MagicMock(return_value="test")) as _mocked_credentials: + client._connect_core() + _grpc_mock.assert_called_with(client.core_url, _mocked_credentials.return_value) + + @patch("grpc.channel_ready_future") + def test_secure_channel_creation_with_secure_core_url(self, _mocked_obj): + client = Client(core_url="localhost:443", serving_url="localhost:50054") + with mock.patch("grpc.secure_channel") as _grpc_mock, \ + mock.patch("grpc.ssl_channel_credentials", MagicMock(return_value="test")) as _mocked_credentials: + client._connect_core() + _grpc_mock.assert_called_with(client.core_url, _mocked_credentials.return_value) \ No newline at end of file From 5758d99a7048d166dc011f6a97333e68811c2003 Mon Sep 17 00:00:00 2001 From: David Heryanto Date: Wed, 26 Feb 2020 09:40:40 +0800 Subject: [PATCH 054/176] Rename metric name for request latency in feast serving (#488) So that it is consistent with the actual unit of timing being measured And recommended metric names in Prometheus https://prometheus.io/docs/practices/naming/#metric-names --- .../main/java/feast/serving/service/RedisServingService.java | 2 +- serving/src/main/java/feast/serving/util/Metrics.java | 4 ++-- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/serving/src/main/java/feast/serving/service/RedisServingService.java b/serving/src/main/java/feast/serving/service/RedisServingService.java index 48fc485214d..24c69b9f796 100644 --- a/serving/src/main/java/feast/serving/service/RedisServingService.java +++ b/serving/src/main/java/feast/serving/service/RedisServingService.java @@ -313,7 +313,7 @@ private List sendMultiGet(List keys) { } finally { requestLatency .labels("sendMultiGet") - .observe((System.currentTimeMillis() - startTime) / 1000); + .observe((System.currentTimeMillis() - startTime) / 1000d); } } } diff --git a/serving/src/main/java/feast/serving/util/Metrics.java b/serving/src/main/java/feast/serving/util/Metrics.java index 99f6353e742..05546ec384b 100644 --- a/serving/src/main/java/feast/serving/util/Metrics.java +++ b/serving/src/main/java/feast/serving/util/Metrics.java @@ -24,9 +24,9 @@ public class Metrics { public static final Histogram requestLatency = Histogram.build() .buckets(0.001, 0.002, 0.004, 0.006, 0.008, 0.01, 0.015, 0.02, 0.025, 0.03, 0.035, 0.05) - .name("request_latency_ms") + .name("request_latency_seconds") .subsystem("feast_serving") - .help("Request latency in seconds.") + .help("Request latency in seconds") .labelNames("method") .register(); From 0b31f27d40b4aa141dd982eaf9515cb489729fb5 Mon Sep 17 00:00:00 2001 From: Lavkesh Lahngir Date: Thu, 27 Feb 2020 13:11:40 +0800 Subject: [PATCH 055/176] Replacing Jedis With Lettuce in ingestion and serving (#485) * Replacing Jedis With Lettuce in ingestion and serving * Removing extra lines * Abstacting redis connection based on store * Check the connection before connecting as lettuce does the retry automatically * Running spotless * Throw Exception if the job store config is null * Handle No enum constant RuntimeException --- ingestion/pom.xml | 4 +- .../ingestion/transform/WriteToStore.java | 4 +- .../java/feast/ingestion/utils/StoreUtil.java | 14 +- .../src/main/java/feast/retry/Retriable.java | 2 +- .../store/serving/redis/RedisCustomIO.java | 127 +++++++++--------- .../serving/redis/RedisIngestionClient.java | 49 +++++++ .../redis/RedisStandaloneIngestionClient.java | 119 ++++++++++++++++ .../java/feast/ingestion/ImportJobTest.java | 20 ++- .../serving/redis/RedisCustomIOTest.java | 40 ++++-- serving/pom.xml | 6 +- .../configuration/JobServiceConfig.java | 28 +--- .../configuration/ServingServiceConfig.java | 15 +-- .../configuration/SpecServiceConfig.java | 1 - .../configuration/StoreConfiguration.java | 47 +++++++ .../redis/JobStoreRedisConfig.java | 68 ++++++++++ .../redis/ServingStoreRedisConfig.java | 62 +++++++++ .../service/RedisBackedJobService.java | 35 ++--- .../serving/service/RedisServingService.java | 29 ++-- .../service/RedisBackedJobServiceTest.java | 19 ++- .../service/RedisServingServiceTest.java | 80 ++++++----- 20 files changed, 561 insertions(+), 208 deletions(-) create mode 100644 ingestion/src/main/java/feast/store/serving/redis/RedisIngestionClient.java create mode 100644 ingestion/src/main/java/feast/store/serving/redis/RedisStandaloneIngestionClient.java create mode 100644 serving/src/main/java/feast/serving/configuration/StoreConfiguration.java create mode 100644 serving/src/main/java/feast/serving/configuration/redis/JobStoreRedisConfig.java create mode 100644 serving/src/main/java/feast/serving/configuration/redis/ServingStoreRedisConfig.java diff --git a/ingestion/pom.xml b/ingestion/pom.xml index 001da1a1453..56b5f37c008 100644 --- a/ingestion/pom.xml +++ b/ingestion/pom.xml @@ -216,8 +216,8 @@ - redis.clients - jedis + io.lettuce + lettuce-core diff --git a/ingestion/src/main/java/feast/ingestion/transform/WriteToStore.java b/ingestion/src/main/java/feast/ingestion/transform/WriteToStore.java index b7901c2f90c..4e9082f5554 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/WriteToStore.java +++ b/ingestion/src/main/java/feast/ingestion/transform/WriteToStore.java @@ -22,7 +22,6 @@ import feast.core.FeatureSetProto.FeatureSet; import feast.core.StoreProto.Store; import feast.core.StoreProto.Store.BigQueryConfig; -import feast.core.StoreProto.Store.RedisConfig; import feast.core.StoreProto.Store.StoreType; import feast.ingestion.options.ImportOptions; import feast.ingestion.utils.ResourceUtil; @@ -88,13 +87,12 @@ public PDone expand(PCollection input) { switch (storeType) { case REDIS: - RedisConfig redisConfig = getStore().getRedisConfig(); PCollection redisWriteResult = input .apply( "FeatureRowToRedisMutation", ParDo.of(new FeatureRowToRedisMutationDoFn(getFeatureSets()))) - .apply("WriteRedisMutationToRedis", RedisCustomIO.write(redisConfig)); + .apply("WriteRedisMutationToRedis", RedisCustomIO.write(getStore())); if (options.getDeadLetterTableSpec() != null) { redisWriteResult.apply( WriteFailedElementToBigQuery.newBuilder() diff --git a/ingestion/src/main/java/feast/ingestion/utils/StoreUtil.java b/ingestion/src/main/java/feast/ingestion/utils/StoreUtil.java index 7af98fb8f00..a02b8626945 100644 --- a/ingestion/src/main/java/feast/ingestion/utils/StoreUtil.java +++ b/ingestion/src/main/java/feast/ingestion/utils/StoreUtil.java @@ -43,14 +43,15 @@ import feast.core.StoreProto.Store.RedisConfig; import feast.core.StoreProto.Store.StoreType; import feast.types.ValueProto.ValueType.Enum; +import io.lettuce.core.RedisClient; +import io.lettuce.core.RedisConnectionException; +import io.lettuce.core.RedisURI; import java.util.ArrayList; import java.util.HashMap; import java.util.List; import java.util.Map; import org.apache.commons.lang3.tuple.Pair; import org.slf4j.Logger; -import redis.clients.jedis.JedisPool; -import redis.clients.jedis.exceptions.JedisConnectionException; // TODO: Create partitioned table by default @@ -239,15 +240,16 @@ public static void setupBigQuery( * @param redisConfig Plase refer to feast.core.Store proto */ public static void checkRedisConnection(RedisConfig redisConfig) { - JedisPool jedisPool = new JedisPool(redisConfig.getHost(), redisConfig.getPort()); + RedisClient redisClient = + RedisClient.create(RedisURI.create(redisConfig.getHost(), redisConfig.getPort())); try { - jedisPool.getResource(); - } catch (JedisConnectionException e) { + redisClient.connect(); + } catch (RedisConnectionException e) { throw new RuntimeException( String.format( "Failed to connect to Redis at host: '%s' port: '%d'. Please check that your Redis is running and accessible from Feast.", redisConfig.getHost(), redisConfig.getPort())); } - jedisPool.close(); + redisClient.shutdown(); } } diff --git a/ingestion/src/main/java/feast/retry/Retriable.java b/ingestion/src/main/java/feast/retry/Retriable.java index 0a788fcdd69..30676fe8208 100644 --- a/ingestion/src/main/java/feast/retry/Retriable.java +++ b/ingestion/src/main/java/feast/retry/Retriable.java @@ -17,7 +17,7 @@ package feast.retry; public interface Retriable { - void execute(); + void execute() throws Exception; Boolean isExceptionRetriable(Exception e); diff --git a/ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java b/ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java index 8541baaffc3..633c2eb551d 100644 --- a/ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java +++ b/ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java @@ -18,11 +18,13 @@ import feast.core.StoreProto; import feast.ingestion.values.FailedElement; -import feast.retry.BackOffExecutor; import feast.retry.Retriable; +import io.lettuce.core.RedisConnectionException; import java.io.IOException; import java.util.ArrayList; +import java.util.Arrays; import java.util.List; +import java.util.concurrent.ExecutionException; import org.apache.avro.reflect.Nullable; import org.apache.beam.sdk.coders.AvroCoder; import org.apache.beam.sdk.coders.DefaultCoder; @@ -32,14 +34,9 @@ import org.apache.beam.sdk.transforms.windowing.GlobalWindow; import org.apache.beam.sdk.values.PCollection; import org.apache.commons.lang3.exception.ExceptionUtils; -import org.joda.time.Duration; import org.joda.time.Instant; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import redis.clients.jedis.Jedis; -import redis.clients.jedis.Pipeline; -import redis.clients.jedis.Response; -import redis.clients.jedis.exceptions.JedisConnectionException; public class RedisCustomIO { @@ -50,8 +47,8 @@ public class RedisCustomIO { private RedisCustomIO() {} - public static Write write(StoreProto.Store.RedisConfig redisConfig) { - return new Write(redisConfig); + public static Write write(StoreProto.Store store) { + return new Write(store); } public enum Method { @@ -168,8 +165,8 @@ public static class Write private WriteDoFn dofn; - private Write(StoreProto.Store.RedisConfig redisConfig) { - this.dofn = new WriteDoFn(redisConfig); + private Write(StoreProto.Store store) { + this.dofn = new WriteDoFn(store); } public Write withBatchSize(int batchSize) { @@ -189,23 +186,14 @@ public PCollection expand(PCollection input) { public static class WriteDoFn extends DoFn { - private final String host; - private final int port; - private final BackOffExecutor backOffExecutor; private final List mutations = new ArrayList<>(); - - private Jedis jedis; - private Pipeline pipeline; private int batchSize = DEFAULT_BATCH_SIZE; private int timeout = DEFAULT_TIMEOUT; + private RedisIngestionClient redisIngestionClient; - WriteDoFn(StoreProto.Store.RedisConfig redisConfig) { - this.host = redisConfig.getHost(); - this.port = redisConfig.getPort(); - long backoffMs = - redisConfig.getInitialBackoffMs() > 0 ? redisConfig.getInitialBackoffMs() : 1; - this.backOffExecutor = - new BackOffExecutor(redisConfig.getMaxRetries(), Duration.millis(backoffMs)); + WriteDoFn(StoreProto.Store store) { + if (store.getType() == StoreProto.Store.StoreType.REDIS) + this.redisIngestionClient = new RedisStandaloneIngestionClient(store.getRedisConfig()); } public WriteDoFn withBatchSize(int batchSize) { @@ -224,47 +212,50 @@ public WriteDoFn withTimeout(int timeout) { @Setup public void setup() { - jedis = new Jedis(host, port, timeout); + this.redisIngestionClient.setup(); } @StartBundle public void startBundle() { + try { + redisIngestionClient.connect(); + } catch (RedisConnectionException e) { + log.error("Connection to redis cannot be established ", e); + } mutations.clear(); - pipeline = jedis.pipelined(); } private void executeBatch() throws Exception { - backOffExecutor.execute( - new Retriable() { - @Override - public void execute() { - mutations.forEach( - mutation -> { - writeRecord(mutation); - if (mutation.getExpiryMillis() != null && mutation.getExpiryMillis() > 0) { - pipeline.pexpire(mutation.getKey(), mutation.getExpiryMillis()); - } - }); - pipeline.sync(); - mutations.clear(); - } - - @Override - public Boolean isExceptionRetriable(Exception e) { - return e instanceof JedisConnectionException; - } - - @Override - public void cleanUpAfterFailure() { - try { - pipeline.close(); - } catch (IOException e) { - log.error(String.format("Error while closing pipeline: %s", e.getMessage())); - } - jedis = new Jedis(host, port, timeout); - pipeline = jedis.pipelined(); - } - }); + this.redisIngestionClient + .getBackOffExecutor() + .execute( + new Retriable() { + @Override + public void execute() throws ExecutionException, InterruptedException { + if (!redisIngestionClient.isConnected()) { + redisIngestionClient.connect(); + } + mutations.forEach( + mutation -> { + writeRecord(mutation); + if (mutation.getExpiryMillis() != null + && mutation.getExpiryMillis() > 0) { + redisIngestionClient.pexpire( + mutation.getKey(), mutation.getExpiryMillis()); + } + }); + redisIngestionClient.sync(); + mutations.clear(); + } + + @Override + public Boolean isExceptionRetriable(Exception e) { + return e instanceof RedisConnectionException; + } + + @Override + public void cleanUpAfterFailure() {} + }); } private FailedElement toFailedElement( @@ -272,7 +263,7 @@ private FailedElement toFailedElement( return FailedElement.newBuilder() .setJobName(jobName) .setTransformName("RedisCustomIO") - .setPayload(mutation.getValue().toString()) + .setPayload(Arrays.toString(mutation.getValue())) .setErrorMessage(exception.getMessage()) .setStackTrace(ExceptionUtils.getStackTrace(exception)) .build(); @@ -297,20 +288,26 @@ public void processElement(ProcessContext context) { } } - private Response writeRecord(RedisMutation mutation) { + private void writeRecord(RedisMutation mutation) { switch (mutation.getMethod()) { case APPEND: - return pipeline.append(mutation.getKey(), mutation.getValue()); + redisIngestionClient.append(mutation.getKey(), mutation.getValue()); + return; case SET: - return pipeline.set(mutation.getKey(), mutation.getValue()); + redisIngestionClient.set(mutation.getKey(), mutation.getValue()); + return; case LPUSH: - return pipeline.lpush(mutation.getKey(), mutation.getValue()); + redisIngestionClient.lpush(mutation.getKey(), mutation.getValue()); + return; case RPUSH: - return pipeline.rpush(mutation.getKey(), mutation.getValue()); + redisIngestionClient.rpush(mutation.getKey(), mutation.getValue()); + return; case SADD: - return pipeline.sadd(mutation.getKey(), mutation.getValue()); + redisIngestionClient.sadd(mutation.getKey(), mutation.getValue()); + return; case ZADD: - return pipeline.zadd(mutation.getKey(), mutation.getScore(), mutation.getValue()); + redisIngestionClient.zadd(mutation.getKey(), mutation.getScore(), mutation.getValue()); + return; default: throw new UnsupportedOperationException( String.format("Not implemented writing records for %s", mutation.getMethod())); @@ -337,7 +334,7 @@ public void finishBundle(FinishBundleContext context) @Teardown public void teardown() { - jedis.close(); + redisIngestionClient.shutdown(); } } } diff --git a/ingestion/src/main/java/feast/store/serving/redis/RedisIngestionClient.java b/ingestion/src/main/java/feast/store/serving/redis/RedisIngestionClient.java new file mode 100644 index 00000000000..d51eead53fb --- /dev/null +++ b/ingestion/src/main/java/feast/store/serving/redis/RedisIngestionClient.java @@ -0,0 +1,49 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.store.serving.redis; + +import feast.retry.BackOffExecutor; +import java.io.Serializable; + +public interface RedisIngestionClient extends Serializable { + + void setup(); + + BackOffExecutor getBackOffExecutor(); + + void shutdown(); + + void connect(); + + boolean isConnected(); + + void sync(); + + void pexpire(byte[] key, Long expiryMillis); + + void append(byte[] key, byte[] value); + + void set(byte[] key, byte[] value); + + void lpush(byte[] key, byte[] value); + + void rpush(byte[] key, byte[] value); + + void sadd(byte[] key, byte[] value); + + void zadd(byte[] key, Long score, byte[] value); +} diff --git a/ingestion/src/main/java/feast/store/serving/redis/RedisStandaloneIngestionClient.java b/ingestion/src/main/java/feast/store/serving/redis/RedisStandaloneIngestionClient.java new file mode 100644 index 00000000000..de1f74151ac --- /dev/null +++ b/ingestion/src/main/java/feast/store/serving/redis/RedisStandaloneIngestionClient.java @@ -0,0 +1,119 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.store.serving.redis; + +import com.google.common.collect.Lists; +import feast.core.StoreProto; +import feast.retry.BackOffExecutor; +import io.lettuce.core.*; +import io.lettuce.core.api.StatefulRedisConnection; +import io.lettuce.core.api.async.RedisAsyncCommands; +import io.lettuce.core.codec.ByteArrayCodec; +import java.util.List; +import java.util.concurrent.TimeUnit; +import org.joda.time.Duration; + +public class RedisStandaloneIngestionClient implements RedisIngestionClient { + private final String host; + private final int port; + private final BackOffExecutor backOffExecutor; + private RedisClient redisclient; + private static final int DEFAULT_TIMEOUT = 2000; + private StatefulRedisConnection connection; + private RedisAsyncCommands commands; + private List futures = Lists.newArrayList(); + + public RedisStandaloneIngestionClient(StoreProto.Store.RedisConfig redisConfig) { + this.host = redisConfig.getHost(); + this.port = redisConfig.getPort(); + long backoffMs = redisConfig.getInitialBackoffMs() > 0 ? redisConfig.getInitialBackoffMs() : 1; + this.backOffExecutor = + new BackOffExecutor(redisConfig.getMaxRetries(), Duration.millis(backoffMs)); + } + + @Override + public void setup() { + this.redisclient = + RedisClient.create(new RedisURI(host, port, java.time.Duration.ofMillis(DEFAULT_TIMEOUT))); + } + + @Override + public BackOffExecutor getBackOffExecutor() { + return this.backOffExecutor; + } + + @Override + public void shutdown() { + this.redisclient.shutdown(); + } + + @Override + public void connect() { + if (!isConnected()) { + this.connection = this.redisclient.connect(new ByteArrayCodec()); + this.commands = connection.async(); + } + } + + @Override + public boolean isConnected() { + return connection != null; + } + + @Override + public void sync() { + // Wait for some time for futures to complete + // TODO: should this be configurable? + LettuceFutures.awaitAll(60, TimeUnit.SECONDS, futures.toArray(new RedisFuture[0])); + futures.clear(); + } + + @Override + public void pexpire(byte[] key, Long expiryMillis) { + commands.pexpire(key, expiryMillis); + } + + @Override + public void append(byte[] key, byte[] value) { + futures.add(commands.append(key, value)); + } + + @Override + public void set(byte[] key, byte[] value) { + futures.add(commands.set(key, value)); + } + + @Override + public void lpush(byte[] key, byte[] value) { + futures.add(commands.lpush(key, value)); + } + + @Override + public void rpush(byte[] key, byte[] value) { + futures.add(commands.rpush(key, value)); + } + + @Override + public void sadd(byte[] key, byte[] value) { + futures.add(commands.sadd(key, value)); + } + + @Override + public void zadd(byte[] key, Long score, byte[] value) { + futures.add(commands.zadd(key, score, value)); + } +} diff --git a/ingestion/src/test/java/feast/ingestion/ImportJobTest.java b/ingestion/src/test/java/feast/ingestion/ImportJobTest.java index 1148fa40422..7546d7e36e5 100644 --- a/ingestion/src/test/java/feast/ingestion/ImportJobTest.java +++ b/ingestion/src/test/java/feast/ingestion/ImportJobTest.java @@ -38,6 +38,11 @@ import feast.test.TestUtil.LocalRedis; import feast.types.FeatureRowProto.FeatureRow; import feast.types.ValueProto.ValueType.Enum; +import io.lettuce.core.RedisClient; +import io.lettuce.core.RedisURI; +import io.lettuce.core.api.StatefulRedisConnection; +import io.lettuce.core.api.sync.RedisCommands; +import io.lettuce.core.codec.ByteArrayCodec; import java.io.IOException; import java.nio.charset.StandardCharsets; import java.util.ArrayList; @@ -57,7 +62,6 @@ import org.junit.Test; import org.slf4j.Logger; import org.slf4j.LoggerFactory; -import redis.clients.jedis.Jedis; public class ImportJobTest { @@ -206,21 +210,24 @@ public void runPipeline_ShouldWriteToRedisCorrectlyGivenValidSpecAndFeatureRow() Duration.standardSeconds(IMPORT_JOB_CHECK_INTERVAL_DURATION_SEC)); LOGGER.info("Validating the actual values written to Redis ..."); - Jedis jedis = new Jedis(REDIS_HOST, REDIS_PORT); + RedisClient redisClient = + RedisClient.create(new RedisURI(REDIS_HOST, REDIS_PORT, java.time.Duration.ofMillis(2000))); + StatefulRedisConnection connection = redisClient.connect(new ByteArrayCodec()); + RedisCommands sync = connection.sync(); expected.forEach( (key, expectedValue) -> { // Ensure ingested key exists. - byte[] actualByteValue = jedis.get(key.toByteArray()); + byte[] actualByteValue = sync.get(key.toByteArray()); if (actualByteValue == null) { LOGGER.error("Key not found in Redis: " + key); LOGGER.info("Redis INFO:"); - LOGGER.info(jedis.info()); - String randomKey = jedis.randomKey(); + LOGGER.info(sync.info()); + byte[] randomKey = sync.randomkey(); if (randomKey != null) { LOGGER.info("Sample random key, value (for debugging purpose):"); LOGGER.info("Key: " + randomKey); - LOGGER.info("Value: " + jedis.get(randomKey)); + LOGGER.info("Value: " + sync.get(randomKey)); } Assert.fail("Missing key in Redis."); } @@ -239,5 +246,6 @@ public void runPipeline_ShouldWriteToRedisCorrectlyGivenValidSpecAndFeatureRow() // Ensure the retrieved FeatureRow is equal to the ingested FeatureRow. Assert.assertEquals(expectedValue, actualValue); }); + redisClient.shutdown(); } } diff --git a/ingestion/src/test/java/feast/store/serving/redis/RedisCustomIOTest.java b/ingestion/src/test/java/feast/store/serving/redis/RedisCustomIOTest.java index fc17f6207f6..75663d24a6a 100644 --- a/ingestion/src/test/java/feast/store/serving/redis/RedisCustomIOTest.java +++ b/ingestion/src/test/java/feast/store/serving/redis/RedisCustomIOTest.java @@ -26,6 +26,11 @@ import feast.store.serving.redis.RedisCustomIO.RedisMutation; import feast.types.FeatureRowProto.FeatureRow; import feast.types.ValueProto.ValueType.Enum; +import io.lettuce.core.RedisClient; +import io.lettuce.core.RedisURI; +import io.lettuce.core.api.StatefulRedisConnection; +import io.lettuce.core.api.sync.RedisStringCommands; +import io.lettuce.core.codec.ByteArrayCodec; import java.io.IOException; import java.util.HashMap; import java.util.LinkedHashMap; @@ -43,7 +48,6 @@ import org.junit.Before; import org.junit.Rule; import org.junit.Test; -import redis.clients.jedis.Jedis; import redis.embedded.Redis; import redis.embedded.RedisServer; @@ -53,17 +57,22 @@ public class RedisCustomIOTest { private static String REDIS_HOST = "localhost"; private static int REDIS_PORT = 51234; private Redis redis; - private Jedis jedis; + private RedisClient redisClient; + private RedisStringCommands sync; @Before public void setUp() throws IOException { redis = new RedisServer(REDIS_PORT); redis.start(); - jedis = new Jedis(REDIS_HOST, REDIS_PORT); + redisClient = + RedisClient.create(new RedisURI(REDIS_HOST, REDIS_PORT, java.time.Duration.ofMillis(2000))); + StatefulRedisConnection connection = redisClient.connect(new ByteArrayCodec()); + sync = connection.sync(); } @After public void teardown() { + redisClient.shutdown(); redis.stop(); } @@ -105,12 +114,17 @@ public void shouldWriteToRedis() { null)) .collect(Collectors.toList()); - p.apply(Create.of(featureRowWrites)).apply(RedisCustomIO.write(redisConfig)); + StoreProto.Store store = + StoreProto.Store.newBuilder() + .setRedisConfig(redisConfig) + .setType(StoreProto.Store.StoreType.REDIS) + .build(); + p.apply(Create.of(featureRowWrites)).apply(RedisCustomIO.write(store)); p.run(); kvs.forEach( (key, value) -> { - byte[] actual = jedis.get(key.toByteArray()); + byte[] actual = sync.get(key.toByteArray()); assertThat(actual, equalTo(value.toByteArray())); }); } @@ -148,9 +162,14 @@ public void shouldRetryFailConnection() throws InterruptedException { null)) .collect(Collectors.toList()); + StoreProto.Store store = + StoreProto.Store.newBuilder() + .setRedisConfig(redisConfig) + .setType(StoreProto.Store.StoreType.REDIS) + .build(); PCollection failedElementCount = p.apply(Create.of(featureRowWrites)) - .apply(RedisCustomIO.write(redisConfig)) + .apply(RedisCustomIO.write(store)) .apply(Count.globally()); redis.stop(); @@ -169,7 +188,7 @@ public void shouldRetryFailConnection() throws InterruptedException { kvs.forEach( (key, value) -> { - byte[] actual = jedis.get(key.toByteArray()); + byte[] actual = sync.get(key.toByteArray()); assertThat(actual, equalTo(value.toByteArray())); }); } @@ -202,9 +221,14 @@ public void shouldProduceFailedElementIfRetryExceeded() { null)) .collect(Collectors.toList()); + StoreProto.Store store = + StoreProto.Store.newBuilder() + .setRedisConfig(redisConfig) + .setType(StoreProto.Store.StoreType.REDIS) + .build(); PCollection failedElementCount = p.apply(Create.of(featureRowWrites)) - .apply(RedisCustomIO.write(redisConfig)) + .apply(RedisCustomIO.write(store)) .apply(Count.globally()); redis.stop(); diff --git a/serving/pom.xml b/serving/pom.xml index be573be45c5..17700a351b4 100644 --- a/serving/pom.xml +++ b/serving/pom.xml @@ -138,11 +138,11 @@ 3.1.0 - - redis.clients - jedis + io.lettuce + lettuce-core + com.google.guava diff --git a/serving/src/main/java/feast/serving/configuration/JobServiceConfig.java b/serving/src/main/java/feast/serving/configuration/JobServiceConfig.java index 4c6b652c46e..fa94dab8329 100644 --- a/serving/src/main/java/feast/serving/configuration/JobServiceConfig.java +++ b/serving/src/main/java/feast/serving/configuration/JobServiceConfig.java @@ -22,42 +22,24 @@ import feast.serving.service.NoopJobService; import feast.serving.service.RedisBackedJobService; import feast.serving.specs.CachedSpecService; -import java.util.Map; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; -import redis.clients.jedis.JedisPool; -import redis.clients.jedis.JedisPoolConfig; @Configuration public class JobServiceConfig { - public static final String DEFAULT_REDIS_MAX_CONN = "8"; - public static final String DEFAULT_REDIS_MAX_IDLE = "8"; - public static final String DEFAULT_REDIS_MAX_WAIT_MILLIS = "50"; - @Bean - public JobService jobService(FeastProperties feastProperties, CachedSpecService specService) { + public JobService jobService( + FeastProperties feastProperties, + CachedSpecService specService, + StoreConfiguration storeConfiguration) { if (!specService.getStore().getType().equals(StoreType.BIGQUERY)) { return new NoopJobService(); } StoreType storeType = StoreType.valueOf(feastProperties.getJobs().getStoreType()); - Map storeOptions = feastProperties.getJobs().getStoreOptions(); switch (storeType) { case REDIS: - JedisPoolConfig jedisPoolConfig = new JedisPoolConfig(); - jedisPoolConfig.setMaxTotal( - Integer.parseInt(storeOptions.getOrDefault("max-conn", DEFAULT_REDIS_MAX_CONN))); - jedisPoolConfig.setMaxIdle( - Integer.parseInt(storeOptions.getOrDefault("max-idle", DEFAULT_REDIS_MAX_IDLE))); - jedisPoolConfig.setMaxWaitMillis( - Integer.parseInt( - storeOptions.getOrDefault("max-wait-millis", DEFAULT_REDIS_MAX_WAIT_MILLIS))); - JedisPool jedisPool = - new JedisPool( - jedisPoolConfig, - storeOptions.get("host"), - Integer.parseInt(storeOptions.get("port"))); - return new RedisBackedJobService(jedisPool); + return new RedisBackedJobService(storeConfiguration.getJobStoreRedisConnection()); case INVALID: case BIGQUERY: case CASSANDRA: diff --git a/serving/src/main/java/feast/serving/configuration/ServingServiceConfig.java b/serving/src/main/java/feast/serving/configuration/ServingServiceConfig.java index 3cc115978a3..d0ea058baf4 100644 --- a/serving/src/main/java/feast/serving/configuration/ServingServiceConfig.java +++ b/serving/src/main/java/feast/serving/configuration/ServingServiceConfig.java @@ -36,8 +36,6 @@ import org.slf4j.Logger; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; -import redis.clients.jedis.JedisPool; -import redis.clients.jedis.JedisPoolConfig; @Configuration public class ServingServiceConfig { @@ -74,19 +72,16 @@ public ServingService servingService( FeastProperties feastProperties, CachedSpecService specService, JobService jobService, - Tracer tracer) { + Tracer tracer, + StoreConfiguration storeConfiguration) { ServingService servingService = null; Store store = specService.getStore(); switch (store.getType()) { case REDIS: - RedisConfig redisConfig = store.getRedisConfig(); - JedisPoolConfig poolConfig = new JedisPoolConfig(); - poolConfig.setMaxTotal(feastProperties.getStore().getRedisPoolMaxSize()); - poolConfig.setMaxIdle(feastProperties.getStore().getRedisPoolMaxIdle()); - JedisPool jedisPool = - new JedisPool(poolConfig, redisConfig.getHost(), redisConfig.getPort()); - servingService = new RedisServingService(jedisPool, specService, tracer); + servingService = + new RedisServingService( + storeConfiguration.getServingRedisConnection(), specService, tracer); break; case BIGQUERY: BigQueryConfig bqConfig = store.getBigqueryConfig(); diff --git a/serving/src/main/java/feast/serving/configuration/SpecServiceConfig.java b/serving/src/main/java/feast/serving/configuration/SpecServiceConfig.java index 0b3a2938b8e..26ebfa956ca 100644 --- a/serving/src/main/java/feast/serving/configuration/SpecServiceConfig.java +++ b/serving/src/main/java/feast/serving/configuration/SpecServiceConfig.java @@ -59,7 +59,6 @@ public ScheduledExecutorService cachedSpecServiceScheduledExecutorService( @Bean public CachedSpecService specService(FeastProperties feastProperties) { - CoreSpecService coreService = new CoreSpecService(feastCoreHost, feastCorePort); Path path = Paths.get(feastProperties.getStore().getConfigPath()); CachedSpecService cachedSpecStorage = new CachedSpecService(coreService, path); diff --git a/serving/src/main/java/feast/serving/configuration/StoreConfiguration.java b/serving/src/main/java/feast/serving/configuration/StoreConfiguration.java new file mode 100644 index 00000000000..84dc7b7f8d4 --- /dev/null +++ b/serving/src/main/java/feast/serving/configuration/StoreConfiguration.java @@ -0,0 +1,47 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.serving.configuration; + +import io.lettuce.core.api.StatefulRedisConnection; +import org.springframework.beans.factory.ObjectProvider; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.context.annotation.Configuration; + +@Configuration +public class StoreConfiguration { + + // We can define other store specific beans here + // These beans can be autowired or can be created in this class. + private final StatefulRedisConnection servingRedisConnection; + private final StatefulRedisConnection jobStoreRedisConnection; + + @Autowired + public StoreConfiguration( + ObjectProvider> servingRedisConnection, + ObjectProvider> jobStoreRedisConnection) { + this.servingRedisConnection = servingRedisConnection.getIfAvailable(); + this.jobStoreRedisConnection = jobStoreRedisConnection.getIfAvailable(); + } + + public StatefulRedisConnection getServingRedisConnection() { + return servingRedisConnection; + } + + public StatefulRedisConnection getJobStoreRedisConnection() { + return jobStoreRedisConnection; + } +} diff --git a/serving/src/main/java/feast/serving/configuration/redis/JobStoreRedisConfig.java b/serving/src/main/java/feast/serving/configuration/redis/JobStoreRedisConfig.java new file mode 100644 index 00000000000..77d9262bcb3 --- /dev/null +++ b/serving/src/main/java/feast/serving/configuration/redis/JobStoreRedisConfig.java @@ -0,0 +1,68 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.serving.configuration.redis; + +import com.google.common.base.Enums; +import feast.core.StoreProto; +import feast.serving.FeastProperties; +import io.lettuce.core.RedisClient; +import io.lettuce.core.RedisURI; +import io.lettuce.core.api.StatefulRedisConnection; +import io.lettuce.core.codec.ByteArrayCodec; +import io.lettuce.core.resource.ClientResources; +import io.lettuce.core.resource.DefaultClientResources; +import java.util.Map; +import org.springframework.beans.factory.ObjectProvider; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +@Configuration +public class JobStoreRedisConfig { + + @Bean(destroyMethod = "shutdown") + ClientResources jobStoreClientResources() { + return DefaultClientResources.create(); + } + + @Bean(destroyMethod = "shutdown") + RedisClient jobStoreRedisClient( + ClientResources jobStoreClientResources, FeastProperties feastProperties) { + StoreProto.Store.StoreType storeType = + Enums.getIfPresent( + StoreProto.Store.StoreType.class, feastProperties.getJobs().getStoreType()) + .orNull(); + if (storeType != StoreProto.Store.StoreType.REDIS) return null; + Map jobStoreConf = feastProperties.getJobs().getStoreOptions(); + // If job conf is empty throw StoreException + if (jobStoreConf == null + || jobStoreConf.get("host") == null + || jobStoreConf.get("host").isEmpty() + || jobStoreConf.get("port") == null + || jobStoreConf.get("port").isEmpty()) + throw new IllegalArgumentException("Store Configuration is not set"); + RedisURI uri = + RedisURI.create(jobStoreConf.get("host"), Integer.parseInt(jobStoreConf.get("port"))); + return RedisClient.create(jobStoreClientResources, uri); + } + + @Bean(destroyMethod = "close") + StatefulRedisConnection jobStoreRedisConnection( + ObjectProvider jobStoreRedisClient) { + if (jobStoreRedisClient.getIfAvailable() == null) return null; + return jobStoreRedisClient.getIfAvailable().connect(new ByteArrayCodec()); + } +} diff --git a/serving/src/main/java/feast/serving/configuration/redis/ServingStoreRedisConfig.java b/serving/src/main/java/feast/serving/configuration/redis/ServingStoreRedisConfig.java new file mode 100644 index 00000000000..17a50eef6d6 --- /dev/null +++ b/serving/src/main/java/feast/serving/configuration/redis/ServingStoreRedisConfig.java @@ -0,0 +1,62 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.serving.configuration.redis; + +import feast.core.StoreProto; +import feast.serving.specs.CachedSpecService; +import io.lettuce.core.RedisClient; +import io.lettuce.core.RedisURI; +import io.lettuce.core.api.StatefulRedisConnection; +import io.lettuce.core.codec.ByteArrayCodec; +import io.lettuce.core.resource.ClientResources; +import io.lettuce.core.resource.DefaultClientResources; +import org.springframework.beans.factory.ObjectProvider; +import org.springframework.context.annotation.*; + +@Configuration +public class ServingStoreRedisConfig { + + @Bean + StoreProto.Store.RedisConfig servingStoreRedisConf(CachedSpecService specService) { + if (specService.getStore().getType() != StoreProto.Store.StoreType.REDIS) return null; + return specService.getStore().getRedisConfig(); + } + + @Bean(destroyMethod = "shutdown") + ClientResources servingClientResources() { + return DefaultClientResources.create(); + } + + @Bean(destroyMethod = "shutdown") + RedisClient servingRedisClient( + ClientResources servingClientResources, + ObjectProvider servingStoreRedisConf) { + if (servingStoreRedisConf.getIfAvailable() == null) return null; + RedisURI redisURI = + RedisURI.create( + servingStoreRedisConf.getIfAvailable().getHost(), + servingStoreRedisConf.getIfAvailable().getPort()); + return RedisClient.create(servingClientResources, redisURI); + } + + @Bean(destroyMethod = "close") + StatefulRedisConnection servingRedisConnection( + ObjectProvider servingRedisClient) { + if (servingRedisClient.getIfAvailable() == null) return null; + return servingRedisClient.getIfAvailable().connect(new ByteArrayCodec()); + } +} diff --git a/serving/src/main/java/feast/serving/service/RedisBackedJobService.java b/serving/src/main/java/feast/serving/service/RedisBackedJobService.java index 230e20cd782..0bf53630379 100644 --- a/serving/src/main/java/feast/serving/service/RedisBackedJobService.java +++ b/serving/src/main/java/feast/serving/service/RedisBackedJobService.java @@ -19,12 +19,11 @@ import com.google.protobuf.util.JsonFormat; import feast.serving.ServingAPIProto.Job; import feast.serving.ServingAPIProto.Job.Builder; +import io.lettuce.core.api.StatefulRedisConnection; +import io.lettuce.core.api.sync.RedisCommands; import java.util.Optional; import org.joda.time.Duration; import org.slf4j.Logger; -import redis.clients.jedis.Jedis; -import redis.clients.jedis.JedisPool; -import redis.clients.jedis.exceptions.JedisConnectionException; // TODO: Do rate limiting, currently if clients call get() or upsert() // and an exceedingly high rate e.g. they wrap job reload in a while loop with almost no wait @@ -33,53 +32,41 @@ public class RedisBackedJobService implements JobService { private static final Logger log = org.slf4j.LoggerFactory.getLogger(RedisBackedJobService.class); - private final JedisPool jedisPool; + private final RedisCommands syncCommand; // Remove job state info after "defaultExpirySeconds" to prevent filling up Redis memory // and since users normally don't require info about relatively old jobs. private final int defaultExpirySeconds = (int) Duration.standardDays(1).getStandardSeconds(); - public RedisBackedJobService(JedisPool jedisPool) { - this.jedisPool = jedisPool; + public RedisBackedJobService(StatefulRedisConnection connection) { + this.syncCommand = connection.sync(); } @Override public Optional get(String id) { - Jedis jedis = null; Job job = null; try { - jedis = jedisPool.getResource(); - String json = jedis.get(id); - if (json == null) { + String json = new String(syncCommand.get(id.getBytes())); + if (json.isEmpty()) { return Optional.empty(); } Builder builder = Job.newBuilder(); JsonFormat.parser().merge(json, builder); job = builder.build(); - } catch (JedisConnectionException e) { - log.error(String.format("Failed to connect to the redis instance: %s", e)); } catch (Exception e) { log.error(String.format("Failed to parse JSON for Feast job: %s", e.getMessage())); - } finally { - if (jedis != null) { - jedis.close(); - } } return Optional.ofNullable(job); } @Override public void upsert(Job job) { - Jedis jedis = null; try { - jedis = jedisPool.getResource(); - jedis.set(job.getId(), JsonFormat.printer().omittingInsignificantWhitespace().print(job)); - jedis.expire(job.getId(), defaultExpirySeconds); + syncCommand.set( + job.getId().getBytes(), + JsonFormat.printer().omittingInsignificantWhitespace().print(job).getBytes()); + syncCommand.expire(job.getId().getBytes(), defaultExpirySeconds); } catch (Exception e) { log.error(String.format("Failed to upsert job: %s", e.getMessage())); - } finally { - if (jedis != null) { - jedis.close(); - } } } } diff --git a/serving/src/main/java/feast/serving/service/RedisServingService.java b/serving/src/main/java/feast/serving/service/RedisServingService.java index 24c69b9f796..33030b8eaf9 100644 --- a/serving/src/main/java/feast/serving/service/RedisServingService.java +++ b/serving/src/main/java/feast/serving/service/RedisServingService.java @@ -49,24 +49,27 @@ import feast.types.FieldProto.Field; import feast.types.ValueProto.Value; import io.grpc.Status; +import io.lettuce.core.api.StatefulRedisConnection; +import io.lettuce.core.api.sync.RedisCommands; import io.opentracing.Scope; import io.opentracing.Tracer; import java.util.List; import java.util.Map; import java.util.stream.Collectors; import org.slf4j.Logger; -import redis.clients.jedis.Jedis; -import redis.clients.jedis.JedisPool; public class RedisServingService implements ServingService { private static final Logger log = org.slf4j.LoggerFactory.getLogger(RedisServingService.class); - private final JedisPool jedisPool; private final CachedSpecService specService; private final Tracer tracer; + private final RedisCommands syncCommands; - public RedisServingService(JedisPool jedisPool, CachedSpecService specService, Tracer tracer) { - this.jedisPool = jedisPool; + public RedisServingService( + StatefulRedisConnection connection, + CachedSpecService specService, + Tracer tracer) { + this.syncCommands = connection.sync(); this.specService = specService; this.tracer = tracer; } @@ -194,7 +197,7 @@ private void sendAndProcessMultiGet( FeatureSetRequest featureSetRequest) throws InvalidProtocolBufferException { - List jedisResps = sendMultiGet(redisKeys); + List values = sendMultiGet(redisKeys); long startTime = System.currentTimeMillis(); try (Scope scope = tracer.buildSpan("Redis-processResponse").startActive(true)) { FeatureSetSpec spec = featureSetRequest.getSpec(); @@ -206,12 +209,12 @@ private void sendAndProcessMultiGet( RefUtil::generateFeatureStringRef, featureReference -> Value.newBuilder().build())); - for (int i = 0; i < jedisResps.size(); i++) { + for (int i = 0; i < values.size(); i++) { EntityRow entityRow = entityRows.get(i); Map featureValues = featureValuesMap.get(entityRow); - byte[] jedisResponse = jedisResps.get(i); - if (jedisResponse == null) { + byte[] value = values.get(i); + if (value == null) { featureSetRequest .getFeatureReferences() .parallelStream() @@ -226,7 +229,7 @@ private void sendAndProcessMultiGet( continue; } - FeatureRow featureRow = FeatureRow.parseFrom(jedisResponse); + FeatureRow featureRow = FeatureRow.parseFrom(value); boolean stale = isStale(featureSetRequest, entityRow, featureRow); if (stale) { @@ -298,13 +301,15 @@ private boolean isStale( private List sendMultiGet(List keys) { try (Scope scope = tracer.buildSpan("Redis-sendMultiGet").startActive(true)) { long startTime = System.currentTimeMillis(); - try (Jedis jedis = jedisPool.getResource()) { + try { byte[][] binaryKeys = keys.stream() .map(AbstractMessageLite::toByteArray) .collect(Collectors.toList()) .toArray(new byte[0][0]); - return jedis.mget(binaryKeys); + return syncCommands.mget(binaryKeys).stream() + .map(io.lettuce.core.Value::getValue) + .collect(Collectors.toList()); } catch (Exception e) { throw Status.NOT_FOUND .withDescription("Unable to retrieve feature from Redis") diff --git a/serving/src/test/java/feast/serving/service/RedisBackedJobServiceTest.java b/serving/src/test/java/feast/serving/service/RedisBackedJobServiceTest.java index 9247375f59e..34bc31d2c26 100644 --- a/serving/src/test/java/feast/serving/service/RedisBackedJobServiceTest.java +++ b/serving/src/test/java/feast/serving/service/RedisBackedJobServiceTest.java @@ -16,17 +16,17 @@ */ package feast.serving.service; +import io.lettuce.core.RedisClient; +import io.lettuce.core.RedisURI; +import io.lettuce.core.codec.ByteArrayCodec; import java.io.IOException; import org.junit.After; import org.junit.Before; import org.junit.Test; -import redis.clients.jedis.JedisPool; -import redis.clients.jedis.JedisPoolConfig; import redis.embedded.RedisServer; public class RedisBackedJobServiceTest { - private static String REDIS_HOST = "localhost"; - private static int REDIS_PORT = 51235; + private static Integer REDIS_PORT = 51235; private RedisServer redis; @Before @@ -41,12 +41,10 @@ public void teardown() { } @Test - public void shouldRecoverIfRedisConnectionIsLost() { - JedisPoolConfig jedisPoolConfig = new JedisPoolConfig(); - jedisPoolConfig.setMaxTotal(1); - jedisPoolConfig.setMaxWaitMillis(10); - JedisPool jedisPool = new JedisPool(jedisPoolConfig, REDIS_HOST, REDIS_PORT); - RedisBackedJobService jobService = new RedisBackedJobService(jedisPool); + public void shouldRecoverIfRedisConnectionIsLost() throws IOException { + RedisClient client = RedisClient.create(RedisURI.create("localhost", REDIS_PORT)); + RedisBackedJobService jobService = + new RedisBackedJobService(client.connect(new ByteArrayCodec())); jobService.get("does not exist"); redis.stop(); try { @@ -56,5 +54,6 @@ public void shouldRecoverIfRedisConnectionIsLost() { } redis.start(); jobService.get("does not exist"); + client.shutdown(); } } diff --git a/serving/src/test/java/feast/serving/service/RedisServingServiceTest.java b/serving/src/test/java/feast/serving/service/RedisServingServiceTest.java index 042107e1177..8446218cfff 100644 --- a/serving/src/test/java/feast/serving/service/RedisServingServiceTest.java +++ b/serving/src/test/java/feast/serving/service/RedisServingServiceTest.java @@ -38,38 +38,37 @@ import feast.types.FeatureRowProto.FeatureRow; import feast.types.FieldProto.Field; import feast.types.ValueProto.Value; +import io.lettuce.core.KeyValue; +import io.lettuce.core.api.StatefulRedisConnection; +import io.lettuce.core.api.sync.RedisCommands; import io.opentracing.Tracer; import io.opentracing.Tracer.SpanBuilder; -import java.util.Collections; -import java.util.List; -import java.util.Map; +import java.util.*; import java.util.stream.Collectors; import org.junit.Before; import org.junit.Test; import org.mockito.ArgumentMatchers; import org.mockito.Mock; import org.mockito.Mockito; -import redis.clients.jedis.Jedis; -import redis.clients.jedis.JedisPool; public class RedisServingServiceTest { - @Mock JedisPool jedisPool; - - @Mock Jedis jedis; - @Mock CachedSpecService specService; @Mock Tracer tracer; + @Mock StatefulRedisConnection connection; + + @Mock RedisCommands syncCommands; + private RedisServingService redisServingService; private byte[][] redisKeyList; @Before public void setUp() { initMocks(this); - - redisServingService = new RedisServingService(jedisPool, specService, tracer); + when(connection.sync()).thenReturn(syncCommands); + redisServingService = new RedisServingService(connection, specService, tracer); redisKeyList = Lists.newArrayList( RedisKey.newBuilder() @@ -149,12 +148,14 @@ public void shouldReturnResponseWithValuesIfKeysPresent() { .setSpec(getFeatureSetSpec()) .build(); - List featureRowBytes = - featureRows.stream().map(AbstractMessageLite::toByteArray).collect(Collectors.toList()); + List> featureRowBytes = + featureRows.stream() + .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) + .collect(Collectors.toList()); when(specService.getFeatureSets(request.getFeaturesList())) .thenReturn(Collections.singletonList(featureSetRequest)); - when(jedisPool.getResource()).thenReturn(jedis); - when(jedis.mget(redisKeyList)).thenReturn(featureRowBytes); + when(connection.sync()).thenReturn(syncCommands); + when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); when(tracer.buildSpan(ArgumentMatchers.any())).thenReturn(Mockito.mock(SpanBuilder.class)); GetOnlineFeaturesResponse expected = @@ -234,12 +235,14 @@ public void shouldReturnResponseWithValuesWhenFeatureSetSpecHasUnspecifiedMaxAge .setSpec(getFeatureSetSpecWithNoMaxAge()) .build(); - List featureRowBytes = - featureRows.stream().map(AbstractMessageLite::toByteArray).collect(Collectors.toList()); + List> featureRowBytes = + featureRows.stream() + .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) + .collect(Collectors.toList()); when(specService.getFeatureSets(request.getFeaturesList())) .thenReturn(Collections.singletonList(featureSetRequest)); - when(jedisPool.getResource()).thenReturn(jedis); - when(jedis.mget(redisKeyList)).thenReturn(featureRowBytes); + when(connection.sync()).thenReturn(syncCommands); + when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); when(tracer.buildSpan(ArgumentMatchers.any())).thenReturn(Mockito.mock(SpanBuilder.class)); GetOnlineFeaturesResponse expected = @@ -315,12 +318,14 @@ public void shouldReturnKeysWithoutVersionifNotProvided() { .setSpec(getFeatureSetSpec()) .build(); - List featureRowBytes = - featureRows.stream().map(AbstractMessageLite::toByteArray).collect(Collectors.toList()); + List> featureRowBytes = + featureRows.stream() + .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) + .collect(Collectors.toList()); when(specService.getFeatureSets(request.getFeaturesList())) .thenReturn(Collections.singletonList(featureSetRequest)); - when(jedisPool.getResource()).thenReturn(jedis); - when(jedis.mget(redisKeyList)).thenReturn(featureRowBytes); + when(connection.sync()).thenReturn(syncCommands); + when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); when(tracer.buildSpan(ArgumentMatchers.any())).thenReturn(Mockito.mock(SpanBuilder.class)); GetOnlineFeaturesResponse expected = @@ -401,11 +406,14 @@ public void shouldReturnResponseWithUnsetValuesIfKeysNotPresent() { .setSpec(getFeatureSetSpec()) .build(); - List featureRowBytes = Lists.newArrayList(featureRows.get(0).toByteArray(), null); + List> featureRowBytes = + featureRows.stream() + .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) + .collect(Collectors.toList()); when(specService.getFeatureSets(request.getFeaturesList())) .thenReturn(Collections.singletonList(featureSetRequest)); - when(jedisPool.getResource()).thenReturn(jedis); - when(jedis.mget(redisKeyList)).thenReturn(featureRowBytes); + when(connection.sync()).thenReturn(syncCommands); + when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); when(tracer.buildSpan(ArgumentMatchers.any())).thenReturn(Mockito.mock(SpanBuilder.class)); GetOnlineFeaturesResponse expected = @@ -489,12 +497,14 @@ public void shouldReturnResponseWithUnsetValuesIfMaxAgeIsExceeded() { .setSpec(spec) .build(); - List featureRowBytes = - featureRows.stream().map(AbstractMessageLite::toByteArray).collect(Collectors.toList()); + List> featureRowBytes = + featureRows.stream() + .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) + .collect(Collectors.toList()); when(specService.getFeatureSets(request.getFeaturesList())) .thenReturn(Collections.singletonList(featureSetRequest)); - when(jedisPool.getResource()).thenReturn(jedis); - when(jedis.mget(redisKeyList)).thenReturn(featureRowBytes); + when(connection.sync()).thenReturn(syncCommands); + when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); when(tracer.buildSpan(ArgumentMatchers.any())).thenReturn(Mockito.mock(SpanBuilder.class)); GetOnlineFeaturesResponse expected = @@ -569,12 +579,14 @@ public void shouldFilterOutUndesiredRows() { .setSpec(getFeatureSetSpec()) .build(); - List featureRowBytes = - featureRows.stream().map(AbstractMessageLite::toByteArray).collect(Collectors.toList()); + List> featureRowBytes = + featureRows.stream() + .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) + .collect(Collectors.toList()); when(specService.getFeatureSets(request.getFeaturesList())) .thenReturn(Collections.singletonList(featureSetRequest)); - when(jedisPool.getResource()).thenReturn(jedis); - when(jedis.mget(redisKeyList)).thenReturn(featureRowBytes); + when(connection.sync()).thenReturn(syncCommands); + when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); when(tracer.buildSpan(ArgumentMatchers.any())).thenReturn(Mockito.mock(SpanBuilder.class)); GetOnlineFeaturesResponse expected = From 9e4b4f6ffb3927bc1d069b39af4f1313e606620d Mon Sep 17 00:00:00 2001 From: Iain Rauch Date: Thu, 27 Feb 2020 05:42:40 +0000 Subject: [PATCH 056/176] Add log4j-web jar to core and serving. (#498) --- core/pom.xml | 4 ++++ pom.xml | 5 +++++ serving/pom.xml | 4 ++++ 3 files changed, 13 insertions(+) diff --git a/core/pom.xml b/core/pom.xml index 720f80a8a9b..7961b45074b 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -72,6 +72,10 @@ org.springframework.boot spring-boot-starter-log4j2 + + org.apache.logging.log4j + log4j-web + io.github.lognet diff --git a/pom.xml b/pom.xml index d822d367b8d..c4bf0ea74a8 100644 --- a/pom.xml +++ b/pom.xml @@ -278,6 +278,11 @@ log4j-jul ${log4jVersion} + + org.apache.logging.log4j + log4j-web + ${log4jVersion} + org.apache.logging.log4j log4j-slf4j-impl diff --git a/serving/pom.xml b/serving/pom.xml index 17700a351b4..4cc02dc4510 100644 --- a/serving/pom.xml +++ b/serving/pom.xml @@ -98,6 +98,10 @@ org.springframework.boot spring-boot-starter-log4j2 + + org.apache.logging.log4j + log4j-web + org.springframework.boot From d785b60a4eca69a1056187a8aa264621a89ac482 Mon Sep 17 00:00:00 2001 From: David Heryanto Date: Thu, 27 Feb 2020 14:06:40 +0800 Subject: [PATCH 057/176] Update CHANGELOG for v0.4.5 and v0.4.6 (#497) --- CHANGELOG.md | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index ee545e3c4d0..7758ae3fb97 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,33 @@ # Changelog +## [v0.4.6](https://github.com/gojek/feast/tree/v0.4.6) (2020-02-26) + +[Full Changelog](https://github.com/gojek/feast/compare/v0.4.5...v0.4.6) + +**Merged pull requests:** +- Rename metric name for request latency in feast serving [\#488](https://github.com/gojek/feast/pull/488) ([davidheryanto](https://github.com/davidheryanto)) +- Allow use of secure gRPC in Feast Python client [\#459](https://github.com/gojek/feast/pull/459) ([Yanson](https://github.com/Yanson)) +- Extend WriteMetricsTransform in Ingestion to write feature value stats to StatsD [\#486](https://github.com/gojek/feast/pull/486) ([davidheryanto](https://github.com/davidheryanto)) +- Remove transaction from Ingestion [\#480](https://github.com/gojek/feast/pull/480) ([imjuanleonard](https://github.com/imjuanleonard)) +- Fix fastavro version used in Feast to avoid Timestamp delta error [\#490](https://github.com/gojek/feast/pull/490) ([davidheryanto](https://github.com/davidheryanto)) +- Fail Spotless formatting check before tests execute [\#487](https://github.com/gojek/feast/pull/487) ([ches](https://github.com/ches)) +- Reduce refresh rate of specification refresh in Serving to 10 seconds [\#481](https://github.com/gojek/feast/pull/481) ([woop](https://github.com/woop)) + +## [v0.4.5](https://github.com/gojek/feast/tree/v0.4.5) (2020-02-14) + +[Full Changelog](https://github.com/gojek/feast/compare/v0.4.4...v0.4.5) + +**Merged pull requests:** +- Use bzip2 compressed feature set json as pipeline option [\#466](https://github.com/gojek/feast/pull/466) ([khorshuheng](https://github.com/khorshuheng)) +- Make redis key creation more determinisitic [\#471](https://github.com/gojek/feast/pull/471) ([zhilingc](https://github.com/zhilingc)) +- Helm Chart Upgrades [\#458](https://github.com/gojek/feast/pull/458) ([Yanson](https://github.com/Yanson)) +- Exclude version from grouping [\#441](https://github.com/gojek/feast/pull/441) ([khorshuheng](https://github.com/khorshuheng)) +- Use concrete class for AvroCoder compatibility [\#465](https://github.com/gojek/feast/pull/465) ([zhilingc](https://github.com/zhilingc)) +- Fix typo in split string length check [\#464](https://github.com/gojek/feast/pull/464) ([zhilingc](https://github.com/zhilingc)) +- Update README.md and remove versions from Helm Charts [\#457](https://github.com/gojek/feast/pull/457) ([woop](https://github.com/woop)) +- Deduplicate example notebooks [\#456](https://github.com/gojek/feast/pull/456) ([woop](https://github.com/woop)) +- Allow users not to set max age for batch retrieval [\#446](https://github.com/gojek/feast/pull/446) ([zhilingc](https://github.com/zhilingc)) + ## [v0.4.4](https://github.com/gojek/feast/tree/v0.4.4) (2020-01-28) [Full Changelog](https://github.com/gojek/feast/compare/v0.4.3...v0.4.4) From 50916d5a0390dbe6f76bc40830f38e7ecff97df7 Mon Sep 17 00:00:00 2001 From: Lavkesh Lahngir Date: Fri, 28 Feb 2020 07:50:40 +0800 Subject: [PATCH 058/176] [Bug] Clear all the futures when sync is called. (#501) * Replacing Jedis With Lettuce in ingestion and serving * Removing extra lines * Abstacting redis connection based on store * Check the connection before connecting as lettuce does the retry automatically * Running spotless * Throw Exception if the job store config is null * Handle No enum constant RuntimeException * Future should be cleared everytime sync is called --- .../serving/redis/RedisStandaloneIngestionClient.java | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/ingestion/src/main/java/feast/store/serving/redis/RedisStandaloneIngestionClient.java b/ingestion/src/main/java/feast/store/serving/redis/RedisStandaloneIngestionClient.java index de1f74151ac..d95ebbbf64a 100644 --- a/ingestion/src/main/java/feast/store/serving/redis/RedisStandaloneIngestionClient.java +++ b/ingestion/src/main/java/feast/store/serving/redis/RedisStandaloneIngestionClient.java @@ -78,8 +78,11 @@ public boolean isConnected() { public void sync() { // Wait for some time for futures to complete // TODO: should this be configurable? - LettuceFutures.awaitAll(60, TimeUnit.SECONDS, futures.toArray(new RedisFuture[0])); - futures.clear(); + try { + LettuceFutures.awaitAll(60, TimeUnit.SECONDS, futures.toArray(new RedisFuture[0])); + } finally { + futures.clear(); + } } @Override From a9ff6666604a826af936a1ec1521c22be4a2c37e Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Tue, 3 Mar 2020 12:39:45 +0800 Subject: [PATCH 059/176] Add configuration for multiprocessing for python tests (#506) * Add configuration for multiprocessing so tests can run on mac * Set windows to use spawn as well --- sdk/python/tests/conftest.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 sdk/python/tests/conftest.py diff --git a/sdk/python/tests/conftest.py b/sdk/python/tests/conftest.py new file mode 100644 index 00000000000..b564eeaa5b1 --- /dev/null +++ b/sdk/python/tests/conftest.py @@ -0,0 +1,22 @@ +# Copyright 2019 The Feast Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +from sys import platform +import multiprocessing + + +def pytest_configure(config): + if platform in ["darwin", "windows"]: + multiprocessing.set_start_method("spawn") + else: + multiprocessing.set_start_method("fork") From ca312bec0e87f93900ea6ebb73ceebe6a319c264 Mon Sep 17 00:00:00 2001 From: feast-ci-bot <46292936+feast-ci-bot@users.noreply.github.com> Date: Thu, 5 Mar 2020 13:41:46 +0800 Subject: [PATCH 060/176] Relax fastavro version requirement in Feast (#500) --- sdk/python/setup.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/sdk/python/setup.py b/sdk/python/setup.py index 3fc77540c02..9d8a3786505 100644 --- a/sdk/python/setup.py +++ b/sdk/python/setup.py @@ -37,9 +37,7 @@ "pandavro==1.5.*", "protobuf>=3.10", "PyYAML==5.1.*", - # fastavro 0.22.10 and newer will throw this error for e2e batch test: - # TypeError: Timestamp subtraction must have the same timezones or no timezones - "fastavro==0.22.9", + "fastavro>=0.22.11,<0.23", "kafka-python==1.*", "tabulate==0.8.*", "toml==0.10.*", From 7633912cd8209cece2a21ebe344600342ef1ff9b Mon Sep 17 00:00:00 2001 From: feast-ci-bot <46292936+feast-ci-bot@users.noreply.github.com> Date: Thu, 5 Mar 2020 15:23:46 +0800 Subject: [PATCH 061/176] Send additional 25th and 99th percentile for feature value metrics (#511) --- .../feast/ingestion/options/ImportOptions.java | 12 ++++++------ .../metrics/WriteFeatureValueMetricsDoFn.java | 14 ++++++++++++++ 2 files changed, 20 insertions(+), 6 deletions(-) diff --git a/ingestion/src/main/java/feast/ingestion/options/ImportOptions.java b/ingestion/src/main/java/feast/ingestion/options/ImportOptions.java index c1bdcd5fd17..1fa127d6629 100644 --- a/ingestion/src/main/java/feast/ingestion/options/ImportOptions.java +++ b/ingestion/src/main/java/feast/ingestion/options/ImportOptions.java @@ -65,8 +65,7 @@ public interface ImportOptions extends PipelineOptions, DataflowPipelineOptions, */ void setDeadLetterTableSpec(String deadLetterTableSpec); - // TODO: expound - @Description("MetricsAccumulator exporter type to instantiate.") + @Description("MetricsAccumulator exporter type to instantiate. Supported type: statsd") @Default.String("none") String getMetricsExporterType(); @@ -86,10 +85,11 @@ public interface ImportOptions extends PipelineOptions, DataflowPipelineOptions, void setStatsdPort(int StatsdPort); @Description( - "Fixed window size in seconds (default 30) to apply before aggregation of numerical value of features" - + "and writing the aggregated value to StatsD. Refer to feast.ingestion.transform.metrics.WriteFeatureValueMetricsDoFn" - + "for details on the metric names and types.") - @Default.Integer(30) + "Fixed window size in seconds (default 60) to apply before aggregating the numerical value of " + + "features and exporting the aggregated values as metrics. Refer to " + + "feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java" + + "for the metric nameas and types used.") + @Default.Integer(60) int getWindowSizeInSecForFeatureValueMetric(); void setWindowSizeInSecForFeatureValueMetric(int seconds); diff --git a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java index 8574d2414c3..a4ed07b5052 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java +++ b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java @@ -90,9 +90,11 @@ abstract static class Builder { public static String GAUGE_NAME_FEATURE_VALUE_MIN = "feature_value_min"; public static String GAUGE_NAME_FEATURE_VALUE_MAX = "feature_value_max"; public static String GAUGE_NAME_FEATURE_VALUE_MEAN = "feature_value_mean"; + public static String GAUGE_NAME_FEATURE_VALUE_PERCENTILE_25 = "feature_value_percentile_25"; public static String GAUGE_NAME_FEATURE_VALUE_PERCENTILE_50 = "feature_value_percentile_50"; public static String GAUGE_NAME_FEATURE_VALUE_PERCENTILE_90 = "feature_value_percentile_90"; public static String GAUGE_NAME_FEATURE_VALUE_PERCENTILE_95 = "feature_value_percentile_95"; + public static String GAUGE_NAME_FEATURE_VALUE_PERCENTILE_99 = "feature_value_percentile_99"; @Setup public void setup() { @@ -205,6 +207,12 @@ public void processElement( values[i] = valueList.get(i); } + double p25 = new Percentile().evaluate(values, 25); + if (p25 < 0) { + statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_PERCENTILE_25, 0, tags); + } + statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_PERCENTILE_25, p25, tags); + double p50 = new Percentile().evaluate(values, 50); if (p50 < 0) { statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_PERCENTILE_50, 0, tags); @@ -222,6 +230,12 @@ public void processElement( statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_PERCENTILE_95, 0, tags); } statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_PERCENTILE_95, p95, tags); + + double p99 = new Percentile().evaluate(values, 99); + if (p99 < 0) { + statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_PERCENTILE_99, 0, tags); + } + statsDClient.gauge(GAUGE_NAME_FEATURE_VALUE_PERCENTILE_99, p99, tags); } } From b15a5e63fb5ef67e27c58578bb7d62461942f4ac Mon Sep 17 00:00:00 2001 From: Ches Martin Date: Mon, 9 Mar 2020 08:58:38 +0700 Subject: [PATCH 062/176] Remove unused ingestion deps (#520) * Make dependency:analyze run clean on datatypes-java * Remove stale dependencies from ingestion Unused according to `mvn -pl ingestion dependency:analyze`, and tests. We had a recent bump of hibernate-validator with a CVE fix (#421) that I was looking to backport, and it turns out it's not used anymore anyway. --- datatypes/java/pom.xml | 33 +++++++++++++++++++++++++++++ ingestion/pom.xml | 48 ------------------------------------------ pom.xml | 19 +++++++++++++++++ 3 files changed, 52 insertions(+), 48 deletions(-) diff --git a/datatypes/java/pom.xml b/datatypes/java/pom.xml index efda2e8ba31..5810a6db96a 100644 --- a/datatypes/java/pom.xml +++ b/datatypes/java/pom.xml @@ -37,6 +37,17 @@ + + org.apache.maven.plugins + maven-dependency-plugin + + + + javax.annotation + + + + org.xolstice.maven.plugins protobuf-maven-plugin @@ -64,10 +75,32 @@ + + + com.google.guava + guava + + + com.google.protobuf + protobuf-java + + + + io.grpc + grpc-core + + + io.grpc + grpc-protobuf + io.grpc grpc-services + + io.grpc + grpc-stub + javax.annotation diff --git a/ingestion/pom.xml b/ingestion/pom.xml index 56b5f37c008..ccc8ca04510 100644 --- a/ingestion/pom.xml +++ b/ingestion/pom.xml @@ -92,24 +92,6 @@ ${project.version} - - org.glassfish - javax.el - 3.0.0 - - - - javax.validation - validation-api - 2.0.1.Final - - - - org.hibernate.validator - hibernate-validator - 6.1.0.Final - - com.google.auto.value auto-value-annotations @@ -122,15 +104,6 @@ provided - - io.grpc - grpc-stub - - - - com.google.cloud - google-cloud-storage - com.google.cloud google-cloud-bigquery @@ -150,27 +123,6 @@ mockito-core - - com.fasterxml.jackson.core - jackson-annotations - - - com.fasterxml.jackson.core - jackson-core - - - com.fasterxml.jackson.core - jackson-databind - - - com.fasterxml.jackson.dataformat - jackson-dataformat-yaml - - - com.fasterxml.jackson.module - jackson-module-jsonSchema - - com.google.protobuf protobuf-java diff --git a/pom.xml b/pom.xml index c4bf0ea74a8..37961f0be3e 100644 --- a/pom.xml +++ b/pom.xml @@ -143,6 +143,11 @@ + + io.grpc + grpc-core + ${grpcVersion} + io.grpc grpc-netty @@ -569,6 +574,20 @@ docker-maven-plugin 0.20.1 + + org.apache.maven.plugins + maven-dependency-plugin + 3.1.1 + + + + org.apache.maven.shared + maven-dependency-analyzer + 1.11.1 + + + org.apache.maven.plugins maven-javadoc-plugin From 7881b85c28227a63124056ad393a73ad070a7357 Mon Sep 17 00:00:00 2001 From: Joost Rothweiler Date: Mon, 9 Mar 2020 03:14:37 +0100 Subject: [PATCH 063/176] Fix docker-compose db environment (#519) Co-authored-by: Joost Rothweiler <=> --- infra/docker-compose/docker-compose.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/infra/docker-compose/docker-compose.yml b/infra/docker-compose/docker-compose.yml index 87d56cbe925..a796e5fa44e 100644 --- a/infra/docker-compose/docker-compose.yml +++ b/infra/docker-compose/docker-compose.yml @@ -107,5 +107,7 @@ services: db: image: postgres:12-alpine + environment: + POSTGRES_PASSWORD: password ports: - "5432:5342" \ No newline at end of file From aa9b1a5f2f289a4e928260bcf48f2838557e100a Mon Sep 17 00:00:00 2001 From: David Heryanto Date: Mon, 9 Mar 2020 15:03:37 +0800 Subject: [PATCH 064/176] Remove transaction when listing projects (#522) --- .../main/java/feast/core/service/AccessManagementService.java | 1 - 1 file changed, 1 deletion(-) diff --git a/core/src/main/java/feast/core/service/AccessManagementService.java b/core/src/main/java/feast/core/service/AccessManagementService.java index 6f627df33d6..df92750e94f 100644 --- a/core/src/main/java/feast/core/service/AccessManagementService.java +++ b/core/src/main/java/feast/core/service/AccessManagementService.java @@ -71,7 +71,6 @@ public void archiveProject(String name) { * * @return List of active projects */ - @Transactional public List listProjects() { return projectRepository.findAllByArchivedIsFalse(); } From cdbc1ca74e81483e1e7d9dbfbbf460144258d821 Mon Sep 17 00:00:00 2001 From: David Heryanto Date: Mon, 9 Mar 2020 18:31:37 +0800 Subject: [PATCH 065/176] Update field equality check and add test for apply FeatureSet when constraints are updated (#512) --- .../java/feast/core/model/FeatureSet.java | 3 +- .../src/main/java/feast/core/model/Field.java | 21 ++++- .../feast/core/service/SpecServiceTest.java | 78 +++++++++++++++++++ 3 files changed, 100 insertions(+), 2 deletions(-) diff --git a/core/src/main/java/feast/core/model/FeatureSet.java b/core/src/main/java/feast/core/model/FeatureSet.java index c593dcd701f..232a5f67d14 100644 --- a/core/src/main/java/feast/core/model/FeatureSet.java +++ b/core/src/main/java/feast/core/model/FeatureSet.java @@ -55,6 +55,7 @@ import org.tensorflow.metadata.v0.FloatDomain; import org.tensorflow.metadata.v0.ImageDomain; import org.tensorflow.metadata.v0.IntDomain; +import org.tensorflow.metadata.v0.MIDDomain; import org.tensorflow.metadata.v0.NaturalLanguageDomain; import org.tensorflow.metadata.v0.StringDomain; import org.tensorflow.metadata.v0.StructDomain; @@ -342,7 +343,7 @@ private void setFeatureSpecFields(FeatureSpec.Builder featureSpecBuilder, Field } else if (featureField.getImageDomain() != null) { featureSpecBuilder.setImageDomain(ImageDomain.parseFrom(featureField.getImageDomain())); } else if (featureField.getMidDomain() != null) { - featureSpecBuilder.setIntDomain(IntDomain.parseFrom(featureField.getIntDomain())); + featureSpecBuilder.setMidDomain(MIDDomain.parseFrom(featureField.getMidDomain())); } else if (featureField.getUrlDomain() != null) { featureSpecBuilder.setUrlDomain(URLDomain.parseFrom(featureField.getUrlDomain())); } else if (featureField.getTimeDomain() != null) { diff --git a/core/src/main/java/feast/core/model/Field.java b/core/src/main/java/feast/core/model/Field.java index 355b673fc84..cb23e4eceb7 100644 --- a/core/src/main/java/feast/core/model/Field.java +++ b/core/src/main/java/feast/core/model/Field.java @@ -19,6 +19,7 @@ import feast.core.FeatureSetProto.EntitySpec; import feast.core.FeatureSetProto.FeatureSpec; import feast.types.ValueProto.ValueType; +import java.util.Arrays; import java.util.Objects; import javax.persistence.Column; import javax.persistence.Embeddable; @@ -223,7 +224,25 @@ public boolean equals(Object o) { return false; } Field field = (Field) o; - return name.equals(field.getName()) && type.equals(field.getType()); + return Objects.equals(name, field.name) + && Objects.equals(type, field.type) + && Objects.equals(project, field.project) + && Arrays.equals(presence, field.presence) + && Arrays.equals(groupPresence, field.groupPresence) + && Arrays.equals(shape, field.shape) + && Arrays.equals(valueCount, field.valueCount) + && Objects.equals(domain, field.domain) + && Arrays.equals(intDomain, field.intDomain) + && Arrays.equals(floatDomain, field.floatDomain) + && Arrays.equals(stringDomain, field.stringDomain) + && Arrays.equals(boolDomain, field.boolDomain) + && Arrays.equals(structDomain, field.structDomain) + && Arrays.equals(naturalLanguageDomain, field.naturalLanguageDomain) + && Arrays.equals(imageDomain, field.imageDomain) + && Arrays.equals(midDomain, field.midDomain) + && Arrays.equals(urlDomain, field.urlDomain) + && Arrays.equals(timeDomain, field.timeDomain) + && Arrays.equals(timeOfDayDomain, field.timeOfDayDomain); } @Override diff --git a/core/src/test/java/feast/core/service/SpecServiceTest.java b/core/src/test/java/feast/core/service/SpecServiceTest.java index 38f7475636d..1eb56caac26 100644 --- a/core/src/test/java/feast/core/service/SpecServiceTest.java +++ b/core/src/test/java/feast/core/service/SpecServiceTest.java @@ -63,7 +63,10 @@ import java.util.Arrays; import java.util.Collections; import java.util.Comparator; +import java.util.HashMap; import java.util.List; +import java.util.Map; +import java.util.Map.Entry; import java.util.Optional; import java.util.stream.Collectors; import org.junit.Before; @@ -78,8 +81,15 @@ import org.tensorflow.metadata.v0.FeaturePresenceWithinGroup; import org.tensorflow.metadata.v0.FixedShape; import org.tensorflow.metadata.v0.FloatDomain; +import org.tensorflow.metadata.v0.ImageDomain; import org.tensorflow.metadata.v0.IntDomain; +import org.tensorflow.metadata.v0.MIDDomain; +import org.tensorflow.metadata.v0.NaturalLanguageDomain; import org.tensorflow.metadata.v0.StringDomain; +import org.tensorflow.metadata.v0.StructDomain; +import org.tensorflow.metadata.v0.TimeDomain; +import org.tensorflow.metadata.v0.TimeOfDayDomain; +import org.tensorflow.metadata.v0.URLDomain; import org.tensorflow.metadata.v0.ValueCount; public class SpecServiceTest { @@ -628,6 +638,74 @@ public void applyFeatureSetShouldAcceptPresenceShapeAndDomainConstraints() } } + @Test + public void applyFeatureSetShouldUpdateFeatureSetWhenConstraintsAreUpdated() + throws InvalidProtocolBufferException { + FeatureSetProto.FeatureSet existingFeatureSet = featureSets.get(2).toProto(); + assertThat( + "Existing feature set has version 3", existingFeatureSet.getSpec().getVersion() == 3); + assertThat( + "Existing feature set has at least 1 feature", + existingFeatureSet.getSpec().getFeaturesList().size() > 0); + + // Map of constraint field name -> value, e.g. "shape" -> FixedShape object. + // If any of these fields are updated, SpecService should update the FeatureSet. + Map contraintUpdates = new HashMap<>(); + contraintUpdates.put("presence", FeaturePresence.newBuilder().setMinFraction(0.5).build()); + contraintUpdates.put( + "group_presence", FeaturePresenceWithinGroup.newBuilder().setRequired(true).build()); + contraintUpdates.put("shape", FixedShape.getDefaultInstance()); + contraintUpdates.put("value_count", ValueCount.newBuilder().setMin(2).build()); + contraintUpdates.put("domain", "new_domain"); + contraintUpdates.put("int_domain", IntDomain.newBuilder().setMax(100).build()); + contraintUpdates.put("float_domain", FloatDomain.newBuilder().setMin(-0.5f).build()); + contraintUpdates.put("string_domain", StringDomain.newBuilder().addValue("string1").build()); + contraintUpdates.put("bool_domain", BoolDomain.newBuilder().setFalseValue("falsy").build()); + contraintUpdates.put("struct_domain", StructDomain.getDefaultInstance()); + contraintUpdates.put("natural_language_domain", NaturalLanguageDomain.getDefaultInstance()); + contraintUpdates.put("image_domain", ImageDomain.getDefaultInstance()); + contraintUpdates.put("mid_domain", MIDDomain.getDefaultInstance()); + contraintUpdates.put("url_domain", URLDomain.getDefaultInstance()); + contraintUpdates.put( + "time_domain", TimeDomain.newBuilder().setStringFormat("string_format").build()); + contraintUpdates.put("time_of_day_domain", TimeOfDayDomain.getDefaultInstance()); + + for (Entry constraint : contraintUpdates.entrySet()) { + String name = constraint.getKey(); + Object value = constraint.getValue(); + FeatureSpec newFeatureSpec = + existingFeatureSet + .getSpec() + .getFeatures(0) + .toBuilder() + .setField(FeatureSpec.getDescriptor().findFieldByName(name), value) + .build(); + FeatureSetSpec newFeatureSetSpec = + existingFeatureSet.getSpec().toBuilder().setFeatures(0, newFeatureSpec).build(); + FeatureSetProto.FeatureSet newFeatureSet = + existingFeatureSet.toBuilder().setSpec(newFeatureSetSpec).build(); + + ApplyFeatureSetResponse response = specService.applyFeatureSet(newFeatureSet); + + assertEquals( + "Response should have CREATED status when field '" + name + "' is updated", + Status.CREATED, + response.getStatus()); + assertEquals( + "FeatureSet should have new version when field '" + name + "' is updated", + existingFeatureSet.getSpec().getVersion() + 1, + response.getFeatureSet().getSpec().getVersion()); + assertEquals( + "Feature should have field '" + name + "' set correctly", + constraint.getValue(), + response + .getFeatureSet() + .getSpec() + .getFeatures(0) + .getField(FeatureSpec.getDescriptor().findFieldByName(name))); + } + } + @Test public void shouldUpdateStoreIfConfigChanges() throws InvalidProtocolBufferException { when(storeRepository.findById("SERVING")).thenReturn(Optional.of(stores.get(0))); From fb2430a06a7bbcf7147362b5d0d227dea8c8abe6 Mon Sep 17 00:00:00 2001 From: Ashwin Date: Mon, 9 Mar 2020 21:52:38 +0800 Subject: [PATCH 066/176] Add Feast Serving gRPC call metrics (#509) * add feast serving grpc calls metrics * edited documentation to reflect correct class * separated into two metrics * linting updates --- .../controller/HealthServiceController.java | 3 +- .../ServingServiceGRpcController.java | 3 +- .../GrpcMonitoringInterceptor.java | 55 +++++++++++++++++++ .../service/BigQueryServingService.java | 5 -- .../serving/service/RedisServingService.java | 4 -- .../main/java/feast/serving/util/Metrics.java | 8 +++ 6 files changed, 67 insertions(+), 11 deletions(-) create mode 100644 serving/src/main/java/feast/serving/interceptors/GrpcMonitoringInterceptor.java diff --git a/serving/src/main/java/feast/serving/controller/HealthServiceController.java b/serving/src/main/java/feast/serving/controller/HealthServiceController.java index 53728544656..3d34aea97b5 100644 --- a/serving/src/main/java/feast/serving/controller/HealthServiceController.java +++ b/serving/src/main/java/feast/serving/controller/HealthServiceController.java @@ -18,6 +18,7 @@ import feast.core.StoreProto.Store; import feast.serving.ServingAPIProto.GetFeastServingInfoRequest; +import feast.serving.interceptors.GrpcMonitoringInterceptor; import feast.serving.service.ServingService; import feast.serving.specs.CachedSpecService; import io.grpc.health.v1.HealthGrpc.HealthImplBase; @@ -30,7 +31,7 @@ // Reference: https://github.com/grpc/grpc/blob/master/doc/health-checking.md -@GRpcService +@GRpcService(interceptors = {GrpcMonitoringInterceptor.class}) public class HealthServiceController extends HealthImplBase { private CachedSpecService specService; private ServingService servingService; diff --git a/serving/src/main/java/feast/serving/controller/ServingServiceGRpcController.java b/serving/src/main/java/feast/serving/controller/ServingServiceGRpcController.java index 0eb9d1e3450..cc1f856d728 100644 --- a/serving/src/main/java/feast/serving/controller/ServingServiceGRpcController.java +++ b/serving/src/main/java/feast/serving/controller/ServingServiceGRpcController.java @@ -26,6 +26,7 @@ import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest; import feast.serving.ServingAPIProto.GetOnlineFeaturesResponse; import feast.serving.ServingServiceGrpc.ServingServiceImplBase; +import feast.serving.interceptors.GrpcMonitoringInterceptor; import feast.serving.service.ServingService; import feast.serving.util.RequestHelper; import io.grpc.stub.StreamObserver; @@ -36,7 +37,7 @@ import org.slf4j.Logger; import org.springframework.beans.factory.annotation.Autowired; -@GRpcService +@GRpcService(interceptors = {GrpcMonitoringInterceptor.class}) public class ServingServiceGRpcController extends ServingServiceImplBase { private static final Logger log = diff --git a/serving/src/main/java/feast/serving/interceptors/GrpcMonitoringInterceptor.java b/serving/src/main/java/feast/serving/interceptors/GrpcMonitoringInterceptor.java new file mode 100644 index 00000000000..bc7ed8997e3 --- /dev/null +++ b/serving/src/main/java/feast/serving/interceptors/GrpcMonitoringInterceptor.java @@ -0,0 +1,55 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.serving.interceptors; + +import feast.serving.util.Metrics; +import io.grpc.ForwardingServerCall.SimpleForwardingServerCall; +import io.grpc.Metadata; +import io.grpc.ServerCall; +import io.grpc.ServerCall.Listener; +import io.grpc.ServerCallHandler; +import io.grpc.ServerInterceptor; +import io.grpc.Status; + +/** + * GrpcMonitoringInterceptor intercepts GRPC calls to provide request latency histogram metrics in + * the Prometheus client. + */ +public class GrpcMonitoringInterceptor implements ServerInterceptor { + + @Override + public Listener interceptCall( + ServerCall call, Metadata headers, ServerCallHandler next) { + + long startCallMillis = System.currentTimeMillis(); + String fullMethodName = call.getMethodDescriptor().getFullMethodName(); + String methodName = fullMethodName.substring(fullMethodName.indexOf("/") + 1); + + return next.startCall( + new SimpleForwardingServerCall(call) { + @Override + public void close(Status status, Metadata trailers) { + Metrics.requestLatency + .labels(methodName) + .observe((System.currentTimeMillis() - startCallMillis) / 1000f); + Metrics.grpcRequestCount.labels(methodName, status.getCode().name()).inc(); + super.close(status, trailers); + } + }, + headers); + } +} diff --git a/serving/src/main/java/feast/serving/service/BigQueryServingService.java b/serving/src/main/java/feast/serving/service/BigQueryServingService.java index f23cbbe64ad..8e3b7ae53e4 100644 --- a/serving/src/main/java/feast/serving/service/BigQueryServingService.java +++ b/serving/src/main/java/feast/serving/service/BigQueryServingService.java @@ -18,7 +18,6 @@ import static feast.serving.store.bigquery.QueryTemplater.createEntityTableUUIDQuery; import static feast.serving.store.bigquery.QueryTemplater.generateFullTableName; -import static feast.serving.util.Metrics.requestLatency; import com.google.cloud.RetryOption; import com.google.cloud.bigquery.BigQuery; @@ -116,7 +115,6 @@ public GetOnlineFeaturesResponse getOnlineFeatures(GetOnlineFeaturesRequest getF /** {@inheritDoc} */ @Override public GetBatchFeaturesResponse getBatchFeatures(GetBatchFeaturesRequest getFeaturesRequest) { - long startTime = System.currentTimeMillis(); List featureSetRequests = specService.getFeatureSets(getFeaturesRequest.getFeaturesList()); @@ -168,9 +166,6 @@ public GetBatchFeaturesResponse getBatchFeatures(GetBatchFeaturesRequest getFeat .build()) .start(); - requestLatency - .labels("getBatchFeatures") - .observe((System.currentTimeMillis() - startTime) / 1000); return GetBatchFeaturesResponse.newBuilder().setJob(feastJob).build(); } diff --git a/serving/src/main/java/feast/serving/service/RedisServingService.java b/serving/src/main/java/feast/serving/service/RedisServingService.java index 33030b8eaf9..fad7f5d8cf4 100644 --- a/serving/src/main/java/feast/serving/service/RedisServingService.java +++ b/serving/src/main/java/feast/serving/service/RedisServingService.java @@ -87,7 +87,6 @@ public GetFeastServingInfoResponse getFeastServingInfo( @Override public GetOnlineFeaturesResponse getOnlineFeatures(GetOnlineFeaturesRequest request) { try (Scope scope = tracer.buildSpan("Redis-getOnlineFeatures").startActive(true)) { - long startTime = System.currentTimeMillis(); GetOnlineFeaturesResponse.Builder getOnlineFeaturesResponseBuilder = GetOnlineFeaturesResponse.newBuilder(); @@ -120,9 +119,6 @@ public GetOnlineFeaturesResponse getOnlineFeatures(GetOnlineFeaturesRequest requ featureValuesMap.values().stream() .map(valueMap -> FieldValues.newBuilder().putAllFields(valueMap).build()) .collect(Collectors.toList()); - requestLatency - .labels("getOnlineFeatures") - .observe((System.currentTimeMillis() - startTime) / 1000); return getOnlineFeaturesResponseBuilder.addAllFieldValues(fieldValues).build(); } } diff --git a/serving/src/main/java/feast/serving/util/Metrics.java b/serving/src/main/java/feast/serving/util/Metrics.java index 05546ec384b..a502bb1559c 100644 --- a/serving/src/main/java/feast/serving/util/Metrics.java +++ b/serving/src/main/java/feast/serving/util/Metrics.java @@ -53,4 +53,12 @@ public class Metrics { .help("number requested feature rows that were stale") .labelNames("project", "feature_name") .register(); + + public static final Counter grpcRequestCount = + Counter.build() + .name("grpc_request_count") + .subsystem("feast_serving") + .help("number of grpc requests served") + .labelNames("method", "status_code") + .register(); } From 4f7cf93a26f0853bcc077f0170dd855a99b8915a Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Mon, 9 Mar 2020 13:54:50 +0000 Subject: [PATCH 067/176] GitBook: [master] 2 pages modified --- docs/SUMMARY.md | 2 +- docs/roadmap.md | 44 ++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 45 insertions(+), 1 deletion(-) create mode 100644 docs/roadmap.md diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 5a2ae95dd49..3cab9a11927 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -5,7 +5,7 @@ * [Concepts](concepts.md) * [Getting Help](getting-help.md) * [Contributing](contributing.md) -* [Roadmap](https://docs.google.com/document/d/1ZZY59j_c2oNN3N6TmavJIyLPMzINdea44CRIe2nhUIo/edit#) +* [Roadmap](roadmap.md) ## Installing Feast diff --git a/docs/roadmap.md b/docs/roadmap.md new file mode 100644 index 00000000000..be2fe90c792 --- /dev/null +++ b/docs/roadmap.md @@ -0,0 +1,44 @@ +# Roadmap + +## Feast 0.5 + +#### New functionality + +1. Streaming statistics and validation \(M1 from [Feature Validation RFC](https://docs.google.com/document/d/1TPmd7r4mniL9Y-V_glZaWNo5LMXLshEAUpYsohojZ-8/edit)\) +2. Batch statistics and validation \(M2 from [Feature Validation RFC](https://docs.google.com/document/d/1TPmd7r4mniL9Y-V_glZaWNo5LMXLshEAUpYsohojZ-8/edit)\) +3. Add support for metadata about missing feature values \([\#278](https://github.com/gojek/feast/issues/278), [Missing Features Metadata RFC](https://docs.google.com/document/d/1VQngwBcx-yWgGpAbsFVdth9GnjL8q-ZgUNBGv57R0Fk/edit#)\) +4. User authentication & authorization \([\#504](https://github.com/gojek/feast/issues/504)\) +5. Add feature or feature set descriptions \([\#463](https://github.com/gojek/feast/issues/463)\) +6. Redis Cluster Support \([\#478](https://github.com/gojek/feast/issues/478)\) + +#### Technical debt, refactoring, or housekeeping + +1. Remove feature set versions from API for retrieval only \([\#462](https://github.com/gojek/feast/issues/462)\) +2. Tracking of batch ingestion by with dataset\_id/job\_id \([\#461](https://github.com/gojek/feast/issues/461)\) + +## Feast 0.6 + +#### New functionality + +1. Extended discovery API/SDK \(needs to be scoped + 1. Resource listing + 2. Schemas, statistics, metrics + 3. Entities as a higher-level concept \([\#405](https://github.com/gojek/feast/issues/405)\) + 4. Add support for discovery based on annotations/labels/tags for easier filtering and discovery +2. Add support for default values \(needs to be scoped\) +3. Add support for audit logs \(needs to be scoped\) +4. Support for an open source warehouse store or connector \(needs to be scoped\) + +#### Technical debt, refactoring, or housekeeping + +1. Move all non-registry functionality out of Feast Core and make it optional \(needs to be scoped\) + 1. Allow Feast serving to use its own local feature sets \(files\) + 2. Move job management to Feast serving + 3. Move stream management \(topic generation\) out of Feast core +2. Remove feature set versions from Feast \(not just retrieval API\) \(needs to be scoped\) + 1. Allow for auto-migration of data in Feast + 2. Implement interface for adding a managed data store +3. Multi-store support for serving \(batch and online\) \(needs to be scoped\) + + + From 3a46cf84b8b218a6630411bb9a121457c380b6b0 Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Mon, 9 Mar 2020 14:14:05 +0000 Subject: [PATCH 068/176] GitBook: [master] one page modified --- docs/roadmap.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/roadmap.md b/docs/roadmap.md index be2fe90c792..1bb8101e9bf 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -2,6 +2,8 @@ ## Feast 0.5 +[Discussion](https://github.com/gojek/feast/issues/527) + #### New functionality 1. Streaming statistics and validation \(M1 from [Feature Validation RFC](https://docs.google.com/document/d/1TPmd7r4mniL9Y-V_glZaWNo5LMXLshEAUpYsohojZ-8/edit)\) From 40b67c05a21c5af8cfdeaa1a445f5eb730e5dd48 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Mon, 9 Mar 2020 22:14:42 +0800 Subject: [PATCH 069/176] Update Roadmap URL --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ca94a1f56be..63b1d45d389 100644 --- a/README.md +++ b/README.md @@ -49,7 +49,7 @@ Please refer to the official documentation at * [Concepts](https://docs.feast.dev/concepts) * [Installation](https://docs.feast.dev/installing-feast/overview) * [Examples](https://github.com/gojek/feast/blob/master/examples/) - * [Roadmap](https://docs.google.com/document/d/1ZZY59j_c2oNN3N6TmavJIyLPMzINdea44CRIe2nhUIo/edit#) + * [Roadmap](https://docs.feast.dev/roadmap) * [Change Log](https://github.com/gojek/feast/blob/master/CHANGELOG.md) * [Slack (#Feast)](https://join.slack.com/t/kubeflow/shared_invite/enQtNDg5MTM4NTQyNjczLTdkNTVhMjg1ZTExOWI0N2QyYTQ2MTIzNTJjMWRiOTFjOGRlZWEzODc1NzMwNTMwM2EzNjY1MTFhODczNjk4MTk) From e7a1a39a3756ddf646bef177e76f8d6252cfbc33 Mon Sep 17 00:00:00 2001 From: Khor Shu Heng <32997938+khorshuheng@users.noreply.github.com> Date: Fri, 13 Mar 2020 15:04:38 +0800 Subject: [PATCH 070/176] Encode feature row before storing in Redis (#530) * Encode feature row before storing in Redis * Include encoding as part of RedisMutationDoFn Co-authored-by: Khor Shu Heng --- .../redis/FeatureRowToRedisMutationDoFn.java | 40 +++- .../java/feast/ingestion/ImportJobTest.java | 19 ++ .../FeatureRowToRedisMutationDoFnTest.java | 176 +++++++++++++++++- .../serving/encoding/FeatureRowDecoder.java | 95 ++++++++++ .../serving/service/RedisServingService.java | 28 ++- .../serving/specs/CachedSpecService.java | 5 + .../main/java/feast/serving/util/Metrics.java | 8 + .../encoding/FeatureRowDecoderTest.java | 110 +++++++++++ 8 files changed, 469 insertions(+), 12 deletions(-) create mode 100644 serving/src/main/java/feast/serving/encoding/FeatureRowDecoder.java create mode 100644 serving/src/test/java/feast/serving/encoding/FeatureRowDecoderTest.java diff --git a/ingestion/src/main/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFn.java b/ingestion/src/main/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFn.java index 4b744d0fe6b..ca017c1f756 100644 --- a/ingestion/src/main/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFn.java +++ b/ingestion/src/main/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFn.java @@ -18,12 +18,15 @@ import feast.core.FeatureSetProto.EntitySpec; import feast.core.FeatureSetProto.FeatureSet; +import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.core.FeatureSetProto.FeatureSpec; import feast.storage.RedisProto.RedisKey; import feast.storage.RedisProto.RedisKey.Builder; import feast.store.serving.redis.RedisCustomIO.Method; import feast.store.serving.redis.RedisCustomIO.RedisMutation; import feast.types.FeatureRowProto.FeatureRow; import feast.types.FieldProto.Field; +import feast.types.ValueProto; import java.util.HashMap; import java.util.List; import java.util.Map; @@ -64,14 +67,45 @@ private RedisKey getKey(FeatureRow featureRow) { return redisKeyBuilder.build(); } + private byte[] getValue(FeatureRow featureRow) { + FeatureSetSpec spec = featureSets.get(featureRow.getFeatureSet()).getSpec(); + + List featureNames = + spec.getFeaturesList().stream().map(FeatureSpec::getName).collect(Collectors.toList()); + Map fieldValueOnlyMap = + featureRow.getFieldsList().stream() + .filter(field -> featureNames.contains(field.getName())) + .distinct() + .collect( + Collectors.toMap( + Field::getName, + field -> Field.newBuilder().setValue(field.getValue()).build())); + + List values = + featureNames.stream() + .sorted() + .map( + featureName -> + fieldValueOnlyMap.getOrDefault( + featureName, + Field.newBuilder().setValue(ValueProto.Value.getDefaultInstance()).build())) + .collect(Collectors.toList()); + + return FeatureRow.newBuilder() + .setEventTimestamp(featureRow.getEventTimestamp()) + .addAllFields(values) + .build() + .toByteArray(); + } + /** Output a redis mutation object for every feature in the feature row. */ @ProcessElement public void processElement(ProcessContext context) { FeatureRow featureRow = context.element(); try { - RedisKey key = getKey(featureRow); - RedisMutation redisMutation = - new RedisMutation(Method.SET, key.toByteArray(), featureRow.toByteArray(), null, null); + byte[] key = getKey(featureRow).toByteArray(); + byte[] value = getValue(featureRow); + RedisMutation redisMutation = new RedisMutation(Method.SET, key, value, null, null); context.output(redisMutation); } catch (Exception e) { log.error(e.getMessage(), e); diff --git a/ingestion/src/test/java/feast/ingestion/ImportJobTest.java b/ingestion/src/test/java/feast/ingestion/ImportJobTest.java index 7546d7e36e5..0b000df0f59 100644 --- a/ingestion/src/test/java/feast/ingestion/ImportJobTest.java +++ b/ingestion/src/test/java/feast/ingestion/ImportJobTest.java @@ -37,6 +37,7 @@ import feast.test.TestUtil.LocalKafka; import feast.test.TestUtil.LocalRedis; import feast.types.FeatureRowProto.FeatureRow; +import feast.types.FieldProto; import feast.types.ValueProto.ValueType.Enum; import io.lettuce.core.RedisClient; import io.lettuce.core.RedisURI; @@ -50,6 +51,7 @@ import java.util.HashMap; import java.util.List; import java.util.Map; +import java.util.stream.Collectors; import java.util.stream.IntStream; import org.apache.beam.sdk.PipelineResult; import org.apache.beam.sdk.PipelineResult.State; @@ -189,6 +191,23 @@ public void runPipeline_ShouldWriteToRedisCorrectlyGivenValidSpecAndFeatureRow() FeatureRow randomRow = TestUtil.createRandomFeatureRow(featureSet); RedisKey redisKey = TestUtil.createRedisKey(featureSet, randomRow); input.add(randomRow); + List fields = + randomRow.getFieldsList().stream() + .filter( + field -> + spec.getFeaturesList().stream() + .map(FeatureSpec::getName) + .collect(Collectors.toList()) + .contains(field.getName())) + .map(field -> field.toBuilder().clearName().build()) + .collect(Collectors.toList()); + randomRow = + randomRow + .toBuilder() + .clearFields() + .addAllFields(fields) + .clearFeatureSet() + .build(); expected.put(redisKey, randomRow); }); diff --git a/ingestion/src/test/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFnTest.java b/ingestion/src/test/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFnTest.java index 92bb6e41c38..86b4feae05f 100644 --- a/ingestion/src/test/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFnTest.java +++ b/ingestion/src/test/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFnTest.java @@ -29,10 +29,7 @@ import feast.types.FieldProto.Field; import feast.types.ValueProto.Value; import feast.types.ValueProto.ValueType.Enum; -import java.util.Arrays; -import java.util.Collections; -import java.util.HashMap; -import java.util.Map; +import java.util.*; import org.apache.beam.sdk.extensions.protobuf.ProtoCoder; import org.apache.beam.sdk.testing.PAssert; import org.apache.beam.sdk.testing.TestPipeline; @@ -96,6 +93,14 @@ public void shouldConvertRowWithDuplicateEntitiesToValidKey() { Field.newBuilder() .setName("entity_id_secondary") .setValue(Value.newBuilder().setStringVal("a"))) + .addFields( + Field.newBuilder() + .setName("feature_1") + .setValue(Value.newBuilder().setStringVal("strValue1"))) + .addFields( + Field.newBuilder() + .setName("feature_2") + .setValue(Value.newBuilder().setInt64Val(1001))) .build(); PCollection output = @@ -116,6 +121,13 @@ public void shouldConvertRowWithDuplicateEntitiesToValidKey() { .setValue(Value.newBuilder().setStringVal("a"))) .build(); + FeatureRow expectedValue = + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setStringVal("strValue1"))) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setInt64Val(1001))) + .build(); + PAssert.that(output) .satisfies( (SerializableFunction, Void>) @@ -123,7 +135,7 @@ public void shouldConvertRowWithDuplicateEntitiesToValidKey() { input.forEach( rm -> { assert (Arrays.equals(rm.getKey(), expectedKey.toByteArray())); - assert (Arrays.equals(rm.getValue(), offendingRow.toByteArray())); + assert (Arrays.equals(rm.getValue(), expectedValue.toByteArray())); }); return null; }); @@ -131,7 +143,7 @@ public void shouldConvertRowWithDuplicateEntitiesToValidKey() { } @Test - public void shouldConvertRowWithOutOfOrderEntitiesToValidKey() { + public void shouldConvertRowWithOutOfOrderFieldsToValidKey() { Map featureSets = new HashMap<>(); featureSets.put("feature_set", fs); @@ -147,6 +159,14 @@ public void shouldConvertRowWithOutOfOrderEntitiesToValidKey() { Field.newBuilder() .setName("entity_id_primary") .setValue(Value.newBuilder().setInt32Val(1))) + .addFields( + Field.newBuilder() + .setName("feature_2") + .setValue(Value.newBuilder().setInt64Val(1001))) + .addFields( + Field.newBuilder() + .setName("feature_1") + .setValue(Value.newBuilder().setStringVal("strValue1"))) .build(); PCollection output = @@ -167,6 +187,148 @@ public void shouldConvertRowWithOutOfOrderEntitiesToValidKey() { .setValue(Value.newBuilder().setStringVal("a"))) .build(); + List expectedFields = + Arrays.asList( + Field.newBuilder().setValue(Value.newBuilder().setStringVal("strValue1")).build(), + Field.newBuilder().setValue(Value.newBuilder().setInt64Val(1001)).build()); + FeatureRow expectedValue = + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addAllFields(expectedFields) + .build(); + + PAssert.that(output) + .satisfies( + (SerializableFunction, Void>) + input -> { + input.forEach( + rm -> { + assert (Arrays.equals(rm.getKey(), expectedKey.toByteArray())); + assert (Arrays.equals(rm.getValue(), expectedValue.toByteArray())); + }); + return null; + }); + p.run(); + } + + @Test + public void shouldMergeDuplicateFeatureFields() { + Map featureSets = new HashMap<>(); + featureSets.put("feature_set", fs); + + FeatureRow featureRowWithDuplicatedFeatureFields = + FeatureRow.newBuilder() + .setFeatureSet("feature_set") + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addFields( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addFields( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .addFields( + Field.newBuilder() + .setName("feature_1") + .setValue(Value.newBuilder().setStringVal("strValue1"))) + .addFields( + Field.newBuilder() + .setName("feature_1") + .setValue(Value.newBuilder().setStringVal("strValue1"))) + .addFields( + Field.newBuilder() + .setName("feature_2") + .setValue(Value.newBuilder().setInt64Val(1001))) + .build(); + + PCollection output = + p.apply(Create.of(Collections.singletonList(featureRowWithDuplicatedFeatureFields))) + .setCoder(ProtoCoder.of(FeatureRow.class)) + .apply(ParDo.of(new FeatureRowToRedisMutationDoFn(featureSets))); + + RedisKey expectedKey = + RedisKey.newBuilder() + .setFeatureSet("feature_set") + .addEntities( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addEntities( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .build(); + + FeatureRow expectedValue = + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setStringVal("strValue1"))) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setInt64Val(1001))) + .build(); + + PAssert.that(output) + .satisfies( + (SerializableFunction, Void>) + input -> { + input.forEach( + rm -> { + assert (Arrays.equals(rm.getKey(), expectedKey.toByteArray())); + assert (Arrays.equals(rm.getValue(), expectedValue.toByteArray())); + }); + return null; + }); + p.run(); + } + + @Test + public void shouldPopulateMissingFeatureValuesWithDefaultInstance() { + Map featureSets = new HashMap<>(); + featureSets.put("feature_set", fs); + + FeatureRow featureRowWithDuplicatedFeatureFields = + FeatureRow.newBuilder() + .setFeatureSet("feature_set") + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addFields( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addFields( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .addFields( + Field.newBuilder() + .setName("feature_1") + .setValue(Value.newBuilder().setStringVal("strValue1"))) + .build(); + + PCollection output = + p.apply(Create.of(Collections.singletonList(featureRowWithDuplicatedFeatureFields))) + .setCoder(ProtoCoder.of(FeatureRow.class)) + .apply(ParDo.of(new FeatureRowToRedisMutationDoFn(featureSets))); + + RedisKey expectedKey = + RedisKey.newBuilder() + .setFeatureSet("feature_set") + .addEntities( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addEntities( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .build(); + + FeatureRow expectedValue = + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setStringVal("strValue1"))) + .addFields(Field.newBuilder().setValue(Value.getDefaultInstance())) + .build(); + PAssert.that(output) .satisfies( (SerializableFunction, Void>) @@ -174,7 +336,7 @@ public void shouldConvertRowWithOutOfOrderEntitiesToValidKey() { input.forEach( rm -> { assert (Arrays.equals(rm.getKey(), expectedKey.toByteArray())); - assert (Arrays.equals(rm.getValue(), offendingRow.toByteArray())); + assert (Arrays.equals(rm.getValue(), expectedValue.toByteArray())); }); return null; }); diff --git a/serving/src/main/java/feast/serving/encoding/FeatureRowDecoder.java b/serving/src/main/java/feast/serving/encoding/FeatureRowDecoder.java new file mode 100644 index 00000000000..e70695d8c64 --- /dev/null +++ b/serving/src/main/java/feast/serving/encoding/FeatureRowDecoder.java @@ -0,0 +1,95 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.serving.encoding; + +import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.core.FeatureSetProto.FeatureSpec; +import feast.types.FeatureRowProto.FeatureRow; +import feast.types.FieldProto.Field; +import java.util.Comparator; +import java.util.List; +import java.util.stream.Collectors; +import java.util.stream.IntStream; + +public class FeatureRowDecoder { + + private final String featureSetRef; + private final FeatureSetSpec spec; + + public FeatureRowDecoder(String featureSetRef, FeatureSetSpec spec) { + this.featureSetRef = featureSetRef; + this.spec = spec; + } + + /** + * A feature row is considered encoded if the feature set and field names are not set. This method + * is required for backward compatibility purposes, to allow Feast serving to continue serving non + * encoded Feature Row ingested by an older version of Feast. + * + * @param featureRow Feature row + * @return boolean + */ + public Boolean isEncoded(FeatureRow featureRow) { + return featureRow.getFeatureSet().isEmpty() + && featureRow.getFieldsList().stream().allMatch(field -> field.getName().isEmpty()); + } + + /** + * Validates if an encoded feature row can be decoded without exception. + * + * @param featureRow Feature row + * @return boolean + */ + public Boolean isEncodingValid(FeatureRow featureRow) { + return featureRow.getFieldsList().size() == spec.getFeaturesList().size(); + } + + /** + * Decoding feature row by repopulating the field names based on the corresponding feature set + * spec. + * + * @param encodedFeatureRow Feature row + * @return boolean + */ + public FeatureRow decode(FeatureRow encodedFeatureRow) { + final List fieldsWithoutName = encodedFeatureRow.getFieldsList(); + + List featureNames = + spec.getFeaturesList().stream() + .sorted(Comparator.comparing(FeatureSpec::getName)) + .map(FeatureSpec::getName) + .collect(Collectors.toList()); + List fields = + IntStream.range(0, featureNames.size()) + .mapToObj( + featureNameIndex -> { + String featureName = featureNames.get(featureNameIndex); + return fieldsWithoutName + .get(featureNameIndex) + .toBuilder() + .setName(featureName) + .build(); + }) + .collect(Collectors.toList()); + return encodedFeatureRow + .toBuilder() + .clearFields() + .setFeatureSet(featureSetRef) + .addAllFields(fields) + .build(); + } +} diff --git a/serving/src/main/java/feast/serving/service/RedisServingService.java b/serving/src/main/java/feast/serving/service/RedisServingService.java index fad7f5d8cf4..56ee1e80ec7 100644 --- a/serving/src/main/java/feast/serving/service/RedisServingService.java +++ b/serving/src/main/java/feast/serving/service/RedisServingService.java @@ -16,6 +16,7 @@ */ package feast.serving.service; +import static feast.serving.util.Metrics.invalidEncodingCount; import static feast.serving.util.Metrics.missingKeyCount; import static feast.serving.util.Metrics.requestCount; import static feast.serving.util.Metrics.requestLatency; @@ -41,6 +42,7 @@ import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest.EntityRow; import feast.serving.ServingAPIProto.GetOnlineFeaturesResponse; import feast.serving.ServingAPIProto.GetOnlineFeaturesResponse.FieldValues; +import feast.serving.encoding.FeatureRowDecoder; import feast.serving.specs.CachedSpecService; import feast.serving.specs.FeatureSetRequest; import feast.serving.util.RefUtil; @@ -55,6 +57,7 @@ import io.opentracing.Tracer; import java.util.List; import java.util.Map; +import java.util.concurrent.ExecutionException; import java.util.stream.Collectors; import org.slf4j.Logger; @@ -108,7 +111,7 @@ public GetOnlineFeaturesResponse getOnlineFeatures(GetOnlineFeaturesRequest requ try { sendAndProcessMultiGet(redisKeys, entityRows, featureValuesMap, featureSetRequest); - } catch (InvalidProtocolBufferException e) { + } catch (InvalidProtocolBufferException | ExecutionException e) { throw Status.INTERNAL .withDescription("Unable to parse protobuf while retrieving feature") .withCause(e) @@ -191,7 +194,7 @@ private void sendAndProcessMultiGet( List entityRows, Map> featureValuesMap, FeatureSetRequest featureSetRequest) - throws InvalidProtocolBufferException { + throws InvalidProtocolBufferException, ExecutionException { List values = sendMultiGet(redisKeys); long startTime = System.currentTimeMillis(); @@ -226,6 +229,27 @@ private void sendAndProcessMultiGet( } FeatureRow featureRow = FeatureRow.parseFrom(value); + String featureSetRef = redisKeys.get(i).getFeatureSet(); + FeatureRowDecoder decoder = + new FeatureRowDecoder(featureSetRef, specService.getFeatureSetSpec(featureSetRef)); + if (decoder.isEncoded(featureRow)) { + if (decoder.isEncodingValid(featureRow)) { + featureRow = decoder.decode(featureRow); + } else { + featureSetRequest + .getFeatureReferences() + .parallelStream() + .forEach( + request -> + invalidEncodingCount + .labels( + spec.getProject(), + String.format("%s:%d", request.getName(), request.getVersion())) + .inc()); + featureValues.putAll(nullValues); + continue; + } + } boolean stale = isStale(featureSetRequest, entityRow, featureRow); if (stale) { diff --git a/serving/src/main/java/feast/serving/specs/CachedSpecService.java b/serving/src/main/java/feast/serving/specs/CachedSpecService.java index 35119589b27..12a8242da13 100644 --- a/serving/src/main/java/feast/serving/specs/CachedSpecService.java +++ b/serving/src/main/java/feast/serving/specs/CachedSpecService.java @@ -90,6 +90,7 @@ public CachedSpecService(CoreSpecService coreService, Path configPath) { featureSetCacheLoader = CacheLoader.from(featureSets::get); featureSetCache = CacheBuilder.newBuilder().maximumSize(MAX_SPEC_COUNT).build(featureSetCacheLoader); + featureSetCache.putAll(featureSets); } /** @@ -101,6 +102,10 @@ public Store getStore() { return this.store; } + public FeatureSetSpec getFeatureSetSpec(String featureSetRef) throws ExecutionException { + return featureSetCache.get(featureSetRef); + } + /** * Get FeatureSetSpecs for the given features. * diff --git a/serving/src/main/java/feast/serving/util/Metrics.java b/serving/src/main/java/feast/serving/util/Metrics.java index a502bb1559c..fa66f79a804 100644 --- a/serving/src/main/java/feast/serving/util/Metrics.java +++ b/serving/src/main/java/feast/serving/util/Metrics.java @@ -46,6 +46,14 @@ public class Metrics { .labelNames("project", "feature_name") .register(); + public static final Counter invalidEncodingCount = + Counter.build() + .name("invalid_encoding_feature_count") + .subsystem("feast_serving") + .help("number requested feature rows that were stored with the wrong encoding") + .labelNames("project", "feature_name") + .register(); + public static final Counter staleKeyCount = Counter.build() .name("stale_feature_count") diff --git a/serving/src/test/java/feast/serving/encoding/FeatureRowDecoderTest.java b/serving/src/test/java/feast/serving/encoding/FeatureRowDecoderTest.java new file mode 100644 index 00000000000..8f6c79ad66c --- /dev/null +++ b/serving/src/test/java/feast/serving/encoding/FeatureRowDecoderTest.java @@ -0,0 +1,110 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.serving.encoding; + +import static org.junit.Assert.*; + +import com.google.protobuf.Timestamp; +import feast.core.FeatureSetProto; +import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.types.FeatureRowProto; +import feast.types.FieldProto.Field; +import feast.types.ValueProto.Value; +import feast.types.ValueProto.ValueType; +import java.util.Collections; +import org.junit.Test; + +public class FeatureRowDecoderTest { + + private FeatureSetProto.EntitySpec entity = + FeatureSetProto.EntitySpec.newBuilder().setName("entity1").build(); + + private FeatureSetSpec spec = + FeatureSetSpec.newBuilder() + .addAllEntities(Collections.singletonList(entity)) + .addFeatures( + FeatureSetProto.FeatureSpec.newBuilder() + .setName("feature1") + .setValueType(ValueType.Enum.FLOAT)) + .addFeatures( + FeatureSetProto.FeatureSpec.newBuilder() + .setName("feature2") + .setValueType(ValueType.Enum.INT32)) + .setName("feature_set_name") + .build(); + + @Test + public void featureRowWithFieldNamesIsNotConsideredAsEncoded() { + + FeatureRowDecoder decoder = new FeatureRowDecoder("feature_set_ref", spec); + FeatureRowProto.FeatureRow nonEncodedFeatureRow = + FeatureRowProto.FeatureRow.newBuilder() + .setFeatureSet("feature_set_ref") + .setEventTimestamp(Timestamp.newBuilder().setNanos(1000)) + .addFields( + Field.newBuilder().setName("feature1").setValue(Value.newBuilder().setInt32Val(2))) + .addFields( + Field.newBuilder() + .setName("feature2") + .setValue(Value.newBuilder().setFloatVal(1.0f))) + .build(); + assertFalse(decoder.isEncoded(nonEncodedFeatureRow)); + } + + @Test + public void encodingIsInvalidIfNumberOfFeaturesInSpecDiffersFromFeatureRow() { + + FeatureRowDecoder decoder = new FeatureRowDecoder("feature_set_ref", spec); + + FeatureRowProto.FeatureRow encodedFeatureRow = + FeatureRowProto.FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setNanos(1000)) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setInt32Val(2))) + .build(); + + assertFalse(decoder.isEncodingValid(encodedFeatureRow)); + } + + @Test + public void shouldDecodeValidEncodedFeatureRow() { + + FeatureRowDecoder decoder = new FeatureRowDecoder("feature_set_ref", spec); + + FeatureRowProto.FeatureRow encodedFeatureRow = + FeatureRowProto.FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setNanos(1000)) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setInt32Val(2))) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setFloatVal(1.0f))) + .build(); + + FeatureRowProto.FeatureRow expectedFeatureRow = + FeatureRowProto.FeatureRow.newBuilder() + .setFeatureSet("feature_set_ref") + .setEventTimestamp(Timestamp.newBuilder().setNanos(1000)) + .addFields( + Field.newBuilder().setName("feature1").setValue(Value.newBuilder().setInt32Val(2))) + .addFields( + Field.newBuilder() + .setName("feature2") + .setValue(Value.newBuilder().setFloatVal(1.0f))) + .build(); + + assertTrue(decoder.isEncoded(encodedFeatureRow)); + assertTrue(decoder.isEncodingValid(encodedFeatureRow)); + assertEquals(expectedFeatureRow, decoder.decode(encodedFeatureRow)); + } +} From f50de6186f42e1e4ac1c173f51e06abcd0c5cdd0 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Sat, 14 Mar 2020 16:25:36 +0800 Subject: [PATCH 071/176] Update Feast 0.5 roadmap Please see this issue for more details https://github.com/gojek/feast/issues/527 --- docs/roadmap.md | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-) diff --git a/docs/roadmap.md b/docs/roadmap.md index 1bb8101e9bf..b423a0fe12e 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -8,15 +8,20 @@ 1. Streaming statistics and validation \(M1 from [Feature Validation RFC](https://docs.google.com/document/d/1TPmd7r4mniL9Y-V_glZaWNo5LMXLshEAUpYsohojZ-8/edit)\) 2. Batch statistics and validation \(M2 from [Feature Validation RFC](https://docs.google.com/document/d/1TPmd7r4mniL9Y-V_glZaWNo5LMXLshEAUpYsohojZ-8/edit)\) -3. Add support for metadata about missing feature values \([\#278](https://github.com/gojek/feast/issues/278), [Missing Features Metadata RFC](https://docs.google.com/document/d/1VQngwBcx-yWgGpAbsFVdth9GnjL8q-ZgUNBGv57R0Fk/edit#)\) +3. Support for Redis Clusters \([\#502](https://github.com/gojek/feast/issues/502)\) 4. User authentication & authorization \([\#504](https://github.com/gojek/feast/issues/504)\) 5. Add feature or feature set descriptions \([\#463](https://github.com/gojek/feast/issues/463)\) 6. Redis Cluster Support \([\#478](https://github.com/gojek/feast/issues/478)\) +7. Job management API ([\#302](https://github.com/gojek/feast/issues/302)\) #### Technical debt, refactoring, or housekeeping - -1. Remove feature set versions from API for retrieval only \([\#462](https://github.com/gojek/feast/issues/462)\) -2. Tracking of batch ingestion by with dataset\_id/job\_id \([\#461](https://github.com/gojek/feast/issues/461)\) +1. Clean up and document all configuration options ([\#525](https://github.com/gojek/feast/issues/525)\) +2. Externalize storage interfaces ([\#402](https://github.com/gojek/feast/issues/402)\) +3. Reduce memory usage in Redis \([\#515](https://github.com/gojek/feast/issues/515)\) +4. Support for handling out of order ingestion \([\#273](https://github.com/gojek/feast/issues/273)\) +5. Remove feature versions and enable automatic data migration \([\#386](https://github.com/gojek/feast/issues/386)\) \([\#462](https://github.com/gojek/feast/issues/462)\) +6. Tracking of batch ingestion by with dataset\_id/job\_id \([\#461](https://github.com/gojek/feast/issues/461)\) +7. Write Beam metrics after ingestion to store (not prior) \([\#489](https://github.com/gojek/feast/issues/489)\) ## Feast 0.6 From afe95421d572b66e914cb13238c28f8c3d3b7118 Mon Sep 17 00:00:00 2001 From: David Heryanto Date: Sun, 15 Mar 2020 14:59:39 +0800 Subject: [PATCH 072/176] Reduce sleep interval duration for thread that listens for messages (#534) This seems to make the test pass more deterministically If this value is higher than the one used for sending output (50ms) some messages may be lost leading to failed test --- .../transform/metrics/WriteFeatureValueMetricsDoFnTest.java | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFnTest.java b/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFnTest.java index 8f0adf40168..d2b0275c6fe 100644 --- a/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFnTest.java +++ b/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFnTest.java @@ -281,7 +281,10 @@ public DummyStatsDServer(int port) { server.receive(packet); messagesReceived.add( new String(packet.getData(), StandardCharsets.UTF_8).trim() + "\n"); - Thread.sleep(50); + // The sleep duration here is shorter than that used in waitForMessage() at + // 50ms. + // Otherwise sometimes some messages seem to be lost, leading to flaky tests. + Thread.sleep(15L); } } catch (Exception e) { From fb893ded90cddf7426ea5264d6deba4b772bc2cb Mon Sep 17 00:00:00 2001 From: David Heryanto Date: Mon, 16 Mar 2020 09:45:39 +0800 Subject: [PATCH 073/176] Update base Docker image for building Feast Serving image (#535) * Update base Docker image for building Feast Serving image - Add clean phase before packaging for more deterministic build (in case host directory is dirty) - Move the downloading of grpc-health-probe in Feast Serving to build stage so the production stage does not need extra tools like wget, for slimmer production image. * Update base Docker image for Feast Serving in Dockerfile.dev --- infra/docker/core/Dockerfile | 3 ++- infra/docker/serving/Dockerfile | 19 ++++++++++--------- infra/docker/serving/Dockerfile.dev | 2 +- 3 files changed, 13 insertions(+), 11 deletions(-) diff --git a/infra/docker/core/Dockerfile b/infra/docker/core/Dockerfile index 8f82e69f6af..7e469ed7f61 100644 --- a/infra/docker/core/Dockerfile +++ b/infra/docker/core/Dockerfile @@ -13,7 +13,7 @@ WORKDIR /build # ENV MAVEN_OPTS="-Dmaven.repo.local=/build/.m2/repository -DdependencyLocationsEnabled=false" RUN mvn --also-make --projects core,ingestion -Drevision=$REVISION \ - -DskipTests=true --batch-mode package + -DskipTests=true --batch-mode clean package # # Unpack the jar and copy the files into production Docker image # for faster startup time when starting Dataflow jobs from Feast Core. @@ -32,6 +32,7 @@ RUN apt-get -qq update && apt-get -y install unar && \ FROM openjdk:11-jre as production ARG REVISION=dev COPY --from=builder /build/core/target/feast-core-$REVISION.jar /opt/feast/feast-core.jar +# Required for staging jar dependencies when submitting Dataflow jobs. COPY --from=builder /build/core/target/feast-core-$REVISION /opt/feast/feast-core CMD ["java",\ "-Xms2048m",\ diff --git a/infra/docker/serving/Dockerfile b/infra/docker/serving/Dockerfile index 48ca18462ac..8f2abf5b75c 100644 --- a/infra/docker/serving/Dockerfile +++ b/infra/docker/serving/Dockerfile @@ -2,7 +2,7 @@ # Build stage 1: Builder # ============================================================ -FROM maven:3.6-jdk-11-slim as builder +FROM maven:3.6-jdk-11 as builder ARG REVISION=dev COPY . /build WORKDIR /build @@ -13,14 +13,7 @@ WORKDIR /build # ENV MAVEN_OPTS="-Dmaven.repo.local=/build/.m2/repository -DdependencyLocationsEnabled=false" RUN mvn --also-make --projects serving -Drevision=$REVISION \ - -DskipTests=true --batch-mode package - -# ============================================================ -# Build stage 2: Production -# ============================================================ - -FROM openjdk:11-jre-alpine as production -ARG REVISION=dev + -DskipTests=true --batch-mode clean package # # Download grpc_health_probe to run health check for Feast Serving # https://kubernetes.io/blog/2018/10/01/health-checking-grpc-servers-on-kubernetes/ @@ -28,7 +21,15 @@ ARG REVISION=dev RUN wget -q https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.3.1/grpc_health_probe-linux-amd64 \ -O /usr/bin/grpc-health-probe && \ chmod +x /usr/bin/grpc-health-probe + +# ============================================================ +# Build stage 2: Production +# ============================================================ + +FROM openjdk:11-jre-slim as production +ARG REVISION=dev COPY --from=builder /build/serving/target/feast-serving-$REVISION.jar /opt/feast/feast-serving.jar +COPY --from=builder /usr/bin/grpc-health-probe /usr/bin/grpc-health-probe CMD ["java",\ "-Xms1024m",\ "-Xmx1024m",\ diff --git a/infra/docker/serving/Dockerfile.dev b/infra/docker/serving/Dockerfile.dev index 2075061f98a..469ff1a25bc 100644 --- a/infra/docker/serving/Dockerfile.dev +++ b/infra/docker/serving/Dockerfile.dev @@ -1,4 +1,4 @@ -FROM openjdk:11-jre-alpine as production +FROM openjdk:11-jre as production ARG REVISION=dev # # Download grpc_health_probe to run health check for Feast Serving From 3b15b1488ad129980b787e3707384b0be27f0386 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Thu, 19 Mar 2020 07:59:40 +0800 Subject: [PATCH 074/176] Add Format and Lint to Makefile (#545) * Add Make commands for format, lint, flake8, spotless, isort, black, and refactor * Add mypy test * Add lint tests to CI * Fix broken Python test * Fix broken test for Python * Add black to dependencies * Remove Python Protos * Add automatic local linting * Update precommit names * Add black exclusions * Add tensorflow metadata proto generation * Ignore tf meta directory * Add build essentials to install make in CI * Add exports back to __init__.py * Add __all__ to export * Add white space to export * Add source to export * Fix python export formatting --- .gitignore | 5 + .pre-commit-config.yaml | 13 + .prow/scripts/test-core-ingestion.sh | 5 + .prow/scripts/test-end-to-end-batch.sh | 3 +- .prow/scripts/test-end-to-end.sh | 3 +- .prow/scripts/test-golang-sdk.sh | 2 + .prow/scripts/test-python-sdk.sh | 7 +- Makefile | 113 +- go.mod | 7 +- go.sum | 27 + protos/Makefile | 33 - sdk/go/client.go | 6 +- sdk/go/protos/feast/core/FeatureSet.pb.go | 690 ++++- sdk/go/protos/feast/core/Store.pb.go | 86 +- sdk/go/protos/feast/storage/Redis.pb.go | 3 +- sdk/go/protos/feast/types/FeatureRow.pb.go | 2 +- sdk/go/request.go | 4 +- sdk/go/request_test.go | 11 +- sdk/go/response.go | 11 +- sdk/go/response_test.go | 44 +- sdk/go/types.go | 3 +- sdk/python/Makefile | 20 - sdk/python/docs/conf.py | 2 +- sdk/python/feast/__init__.py | 14 - sdk/python/feast/cli.py | 14 +- sdk/python/feast/client.py | 58 +- sdk/python/feast/config.py | 5 +- sdk/python/feast/core/CoreService_pb2.py | 988 -------- sdk/python/feast/core/CoreService_pb2.pyi | 429 ---- sdk/python/feast/core/CoreService_pb2_grpc.py | 203 -- sdk/python/feast/core/FeatureSet_pb2.py | 344 --- sdk/python/feast/core/FeatureSet_pb2.pyi | 188 -- sdk/python/feast/core/FeatureSet_pb2_grpc.py | 3 - sdk/python/feast/core/Source_pb2.py | 159 -- sdk/python/feast/core/Source_pb2.pyi | 83 - sdk/python/feast/core/Source_pb2_grpc.py | 3 - sdk/python/feast/core/Store_pb2.py | 346 --- sdk/python/feast/core/Store_pb2.pyi | 165 -- sdk/python/feast/core/Store_pb2_grpc.py | 3 - sdk/python/feast/core/__init__.py | 0 sdk/python/feast/entity.py | 4 +- sdk/python/feast/feature.py | 4 +- sdk/python/feast/feature_set.py | 28 +- sdk/python/feast/job.py | 7 +- sdk/python/feast/loaders/abstract_producer.py | 4 +- sdk/python/feast/loaders/file.py | 13 +- sdk/python/feast/loaders/ingest.py | 3 +- .../feast/serving/ServingService_pb2.py | 971 ------- .../feast/serving/ServingService_pb2.pyi | 465 ---- .../feast/serving/ServingService_pb2_grpc.py | 104 - sdk/python/feast/serving/__init__.py | 0 sdk/python/feast/source.py | 8 +- sdk/python/feast/storage/Redis_pb2.py | 81 - sdk/python/feast/storage/Redis_pb2.pyi | 49 - sdk/python/feast/storage/__init__.py | 0 sdk/python/feast/type_map.py | 18 +- .../feast/types/FeatureRowExtended_pb2.py | 198 -- .../feast/types/FeatureRowExtended_pb2.pyi | 102 - sdk/python/feast/types/FeatureRow_pb2.py | 90 - sdk/python/feast/types/FeatureRow_pb2.pyi | 59 - sdk/python/feast/types/Feature_pb2.py | 108 - sdk/python/feast/types/Feature_pb2.pyi | 44 - sdk/python/feast/types/Field_pb2.py | 81 - sdk/python/feast/types/Field_pb2.pyi | 46 - sdk/python/feast/types/Value_pb2.py | 595 ----- sdk/python/feast/types/Value_pb2.pyi | 260 -- sdk/python/pyproject.toml | 26 + sdk/python/requirements-ci.txt | 9 +- sdk/python/setup.cfg | 18 + .../tensorflow_metadata/proto/v0/path_pb2.py | 69 + .../tensorflow_metadata/proto/v0/path_pb2.pyi | 52 + .../proto/v0/schema_pb2.py | 2256 +++++++++++++++++ .../proto/v0/schema_pb2.pyi | 1063 ++++++++ sdk/python/tests/conftest.py | 2 +- sdk/python/tests/dataframes.py | 6 +- sdk/python/tests/feast_core_server.py | 27 +- sdk/python/tests/feast_serving_server.py | 25 +- sdk/python/tests/test_client.py | 280 +- sdk/python/tests/test_feature_set.py | 16 +- 79 files changed, 4685 insertions(+), 6611 deletions(-) create mode 100644 .pre-commit-config.yaml delete mode 100644 protos/Makefile delete mode 100644 sdk/python/Makefile delete mode 100644 sdk/python/feast/core/CoreService_pb2.py delete mode 100644 sdk/python/feast/core/CoreService_pb2.pyi delete mode 100644 sdk/python/feast/core/CoreService_pb2_grpc.py delete mode 100644 sdk/python/feast/core/FeatureSet_pb2.py delete mode 100644 sdk/python/feast/core/FeatureSet_pb2.pyi delete mode 100644 sdk/python/feast/core/FeatureSet_pb2_grpc.py delete mode 100644 sdk/python/feast/core/Source_pb2.py delete mode 100644 sdk/python/feast/core/Source_pb2.pyi delete mode 100644 sdk/python/feast/core/Source_pb2_grpc.py delete mode 100644 sdk/python/feast/core/Store_pb2.py delete mode 100644 sdk/python/feast/core/Store_pb2.pyi delete mode 100644 sdk/python/feast/core/Store_pb2_grpc.py delete mode 100644 sdk/python/feast/core/__init__.py delete mode 100644 sdk/python/feast/serving/ServingService_pb2.py delete mode 100644 sdk/python/feast/serving/ServingService_pb2.pyi delete mode 100644 sdk/python/feast/serving/ServingService_pb2_grpc.py delete mode 100644 sdk/python/feast/serving/__init__.py delete mode 100644 sdk/python/feast/storage/Redis_pb2.py delete mode 100644 sdk/python/feast/storage/Redis_pb2.pyi delete mode 100644 sdk/python/feast/storage/__init__.py delete mode 100644 sdk/python/feast/types/FeatureRowExtended_pb2.py delete mode 100644 sdk/python/feast/types/FeatureRowExtended_pb2.pyi delete mode 100644 sdk/python/feast/types/FeatureRow_pb2.py delete mode 100644 sdk/python/feast/types/FeatureRow_pb2.pyi delete mode 100644 sdk/python/feast/types/Feature_pb2.py delete mode 100644 sdk/python/feast/types/Feature_pb2.pyi delete mode 100644 sdk/python/feast/types/Field_pb2.py delete mode 100644 sdk/python/feast/types/Field_pb2.pyi delete mode 100644 sdk/python/feast/types/Value_pb2.py delete mode 100644 sdk/python/feast/types/Value_pb2.pyi create mode 100644 sdk/python/pyproject.toml create mode 100644 sdk/python/setup.cfg create mode 100644 sdk/python/tensorflow_metadata/proto/v0/path_pb2.py create mode 100644 sdk/python/tensorflow_metadata/proto/v0/path_pb2.pyi create mode 100644 sdk/python/tensorflow_metadata/proto/v0/schema_pb2.py create mode 100644 sdk/python/tensorflow_metadata/proto/v0/schema_pb2.pyi diff --git a/.gitignore b/.gitignore index a8c5e3fe0ba..d034c89dccf 100644 --- a/.gitignore +++ b/.gitignore @@ -179,3 +179,8 @@ dmypy.json .flattened-pom.xml sdk/python/docs/html +sdk/python/feast/core/ +sdk/python/feast/serving/ +sdk/python/feast/storage/ +sdk/python/feast/types/ +sdk/python/tensorflow_metadata \ No newline at end of file diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml new file mode 100644 index 00000000000..251d67a77e4 --- /dev/null +++ b/.pre-commit-config.yaml @@ -0,0 +1,13 @@ +repos: + - repo: local + hooks: + - id: format + name: Format + stages: [commit] + language: system + entry: make format + - id: lint + name: Lint + stages: [commit] + language: system + entry: make lint \ No newline at end of file diff --git a/.prow/scripts/test-core-ingestion.sh b/.prow/scripts/test-core-ingestion.sh index 98a47ca68c9..af91c4c63f8 100755 --- a/.prow/scripts/test-core-ingestion.sh +++ b/.prow/scripts/test-core-ingestion.sh @@ -1,5 +1,10 @@ #!/usr/bin/env bash +apt-get -qq update +apt-get -y install build-essential + +make lint-java + .prow/scripts/download-maven-cache.sh \ --archive-uri gs://feast-templocation-kf-feast/.m2.2019-10-24.tar \ --output-dir /root/ diff --git a/.prow/scripts/test-end-to-end-batch.sh b/.prow/scripts/test-end-to-end-batch.sh index 47da7219daa..8448cb69cc5 100755 --- a/.prow/scripts/test-end-to-end-batch.sh +++ b/.prow/scripts/test-end-to-end-batch.sh @@ -22,7 +22,7 @@ This script will run end-to-end tests for Feast Core and Batch Serving. " apt-get -qq update -apt-get -y install wget netcat kafkacat +apt-get -y install wget netcat kafkacat build-essential echo " @@ -233,6 +233,7 @@ bash /tmp/miniconda.sh -b -p /root/miniconda -f source ~/.bashrc # Install Feast Python SDK and test requirements +make compile-protos-python pip install -qe sdk/python pip install -qr tests/e2e/requirements.txt diff --git a/.prow/scripts/test-end-to-end.sh b/.prow/scripts/test-end-to-end.sh index 7709758345d..de48116a28e 100755 --- a/.prow/scripts/test-end-to-end.sh +++ b/.prow/scripts/test-end-to-end.sh @@ -21,7 +21,7 @@ This script will run end-to-end tests for Feast Core and Online Serving. " apt-get -qq update -apt-get -y install wget netcat kafkacat +apt-get -y install wget netcat kafkacat build-essential echo " ============================================================ @@ -207,6 +207,7 @@ bash /tmp/miniconda.sh -b -p /root/miniconda -f source ~/.bashrc # Install Feast Python SDK and test requirements +make compile-protos-python pip install -qe sdk/python pip install -qr tests/e2e/requirements.txt diff --git a/.prow/scripts/test-golang-sdk.sh b/.prow/scripts/test-golang-sdk.sh index b586927a512..666f6c12d0c 100755 --- a/.prow/scripts/test-golang-sdk.sh +++ b/.prow/scripts/test-golang-sdk.sh @@ -2,6 +2,8 @@ set -o pipefail +make lint-go + cd sdk/go go test -v 2>&1 | tee /tmp/test_output TEST_EXIT_CODE=$? diff --git a/.prow/scripts/test-python-sdk.sh b/.prow/scripts/test-python-sdk.sh index e7e1cc24874..7c264eed642 100755 --- a/.prow/scripts/test-python-sdk.sh +++ b/.prow/scripts/test-python-sdk.sh @@ -5,7 +5,10 @@ set -e # Default artifact location setting in Prow jobs LOGS_ARTIFACT_PATH=/logs/artifacts -cd sdk/python -pip install -r requirements-ci.txt +pip install -r sdk/python/requirements-ci.txt +make compile-protos-python +make lint-python + +cd sdk/python/ pip install -e . pytest --junitxml=${LOGS_ARTIFACT_PATH}/python-sdk-test-report.xml diff --git a/Makefile b/Makefile index de61fe2892f..ee1978ecba4 100644 --- a/Makefile +++ b/Makefile @@ -14,26 +14,73 @@ # limitations under the License. # -PROJECT_ROOT := $(shell git rev-parse --show-toplevel) +ROOT_DIR := $(shell dirname $(realpath $(firstword $(MAKEFILE_LIST)))) +PROTO_TYPE_SUBDIRS = core serving types storage +PROTO_SERVICE_SUBDIRS = core serving -test: - mvn test +# General + +format: format-python lint-go lint-java + +lint: lint-python lint-go lint-java + +test: test-python test-java + +protos: compile-protos-go compile-protos-python compile-protos-docs + +build: protos build-java build-docker build-html -test-integration: - $(MAKE) -C testing/integration test-integration TYPE=$(TYPE) ID=$(ID) +# Java -build-proto: - $(MAKE) -C protos gen-go - $(MAKE) -C protos gen-python - $(MAKE) -C protos gen-docs +format-java: + mvn spotless:apply -build-cli: - $(MAKE) build-proto - $(MAKE) -C cli build-all +lint-java: + mvn spotless:check + +test-java: + mvn test build-java: mvn clean verify +# Python SDK + +install-python-ci-dependencies: + pip install -r sdk/python/requirements-ci.txt + +compile-protos-python: install-python-ci-dependencies + @$(foreach dir,$(PROTO_TYPE_SUBDIRS),cd ${ROOT_DIR}/protos; python -m grpc_tools.protoc -I. --python_out=../sdk/python/ --mypy_out=../sdk/python/ feast/$(dir)/*.proto;) + @$(foreach dir,$(PROTO_SERVICE_SUBDIRS),cd ${ROOT_DIR}/protos; python -m grpc_tools.protoc -I. --grpc_python_out=../sdk/python/ feast/$(dir)/*.proto;) + cd ${ROOT_DIR}/protos; python -m grpc_tools.protoc -I. --python_out=../sdk/python/ --mypy_out=../sdk/python/ tensorflow_metadata/proto/v0/*.proto + +test-python: + pytest --verbose --color=yes sdk/python/tests + +format-python: + cd ${ROOT_DIR}/sdk/python; isort -rc feast tests + cd ${ROOT_DIR}/sdk/python; black --target-version py37 feast tests + +lint-python: + # TODO: This mypy test needs to be re-enabled and all failures fixed + #cd ${ROOT_DIR}/sdk/python; mypy feast/ tests/ + cd ${ROOT_DIR}/sdk/python; flake8 feast/ tests/ + cd ${ROOT_DIR}/sdk/python; black --check feast tests + cd ${ROOT_DIR}/sdk/python; isort -rc feast tests --check-only + +# Go SDK + +compile-protos-go: + @$(foreach dir,$(PROTO_TYPE_SUBDIRS), cd ${ROOT_DIR}/protos; protoc -I/usr/local/include -I. --go_out=plugins=grpc,paths=source_relative:../sdk/go/protos/ feast/$(dir)/*.proto;) + +format-go: + cd ${ROOT_DIR}/sdk/go; gofmt -s -w *.go + +lint-go: + cd ${ROOT_DIR}/sdk/go; go vet; golint *.go + +# Docker + build-docker: docker build -t $(REGISTRY)/feast-core:$(VERSION) -f infra/docker/core/Dockerfile . docker build -t $(REGISTRY)/feast-serving:$(VERSION) -f infra/docker/serving/Dockerfile . @@ -43,13 +90,37 @@ build-push-docker: docker push $(REGISTRY)/feast-core:$(VERSION) docker push $(REGISTRY)/feast-serving:$(VERSION) +# Documentation + +install-dependencies-proto-docs: + # Use the following command to compile dependencies if installing using the below method. + # cd ${ROOT_DIR}/protos; PATH=$$HOME/bin:$$PATH protoc -I $$HOME/include/ \ + # -I . --docs_out=../dist/grpc feast/*/*.proto + cd ${ROOT_DIR}/protos; + mkdir -p $$HOME/bin + mkdir -p $$HOME/include + go get github.com/golang/protobuf/proto && \ + go get github.com/russross/blackfriday/v2 && \ + cd $$(mktemp -d) && \ + git clone https://github.com/istio/tools/ && \ + cd tools/cmd/protoc-gen-docs && \ + go build && \ + cp protoc-gen-docs $$HOME/bin && \ + cd $$HOME && curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v3.11.2/protoc-3.11.2-linux-x86_64.zip && \ + unzip protoc-3.11.2-linux-x86_64.zip -d protoc3 && \ + mv protoc3/bin/* $$HOME/bin/ && \ + chmod +x $$HOME/bin/protoc && \ + mv protoc3/include/* $$HOME/include + +compile-protos-docs: + cd ${ROOT_DIR}/protos; protoc --docs_out=../dist/grpc feast/*/*.proto + clean-html: - rm -rf $(PROJECT_ROOT)/dist - -build-html: - rm -rf $(PROJECT_ROOT)/dist/ - mkdir -p $(PROJECT_ROOT)/dist/python - mkdir -p $(PROJECT_ROOT)/dist/grpc - cd $(PROJECT_ROOT)/protos && $(MAKE) gen-docs - cd $(PROJECT_ROOT)/sdk/python/docs && $(MAKE) html - cp -r $(PROJECT_ROOT)/sdk/python/docs/html/* $(PROJECT_ROOT)/dist/python \ No newline at end of file + rm -rf $(ROOT_DIR)/dist + +build-html: clean-html + mkdir -p $(ROOT_DIR)/dist/python + mkdir -p $(ROOT_DIR)/dist/grpc + cd $(ROOT_DIR)/protos && $(MAKE) gen-docs + cd $(ROOT_DIR)/sdk/python/docs && $(MAKE) html + cp -r $(ROOT_DIR)/sdk/python/docs/html/* $(ROOT_DIR)/dist/python \ No newline at end of file diff --git a/go.mod b/go.mod index 8dc819493e4..45ce654beae 100644 --- a/go.mod +++ b/go.mod @@ -6,9 +6,10 @@ require ( github.com/Masterminds/sprig v2.22.0+incompatible // indirect github.com/ghodss/yaml v1.0.0 github.com/gogo/protobuf v1.3.1 // indirect + github.com/gojek/feast/sdk/go v0.0.0-20200316014539-fb893ded90cd // indirect github.com/golang/mock v1.2.0 - github.com/golang/protobuf v1.3.2 - github.com/google/go-cmp v0.3.0 + github.com/golang/protobuf v1.3.5 + github.com/google/go-cmp v0.3.1 github.com/huandu/xstrings v1.2.0 // indirect github.com/lyft/protoc-gen-validate v0.1.0 // indirect github.com/mitchellh/copystructure v1.0.0 // indirect @@ -20,7 +21,7 @@ require ( github.com/spf13/viper v1.4.0 github.com/woop/protoc-gen-doc v1.3.0 // indirect golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553 - google.golang.org/grpc v1.23.0 + google.golang.org/grpc v1.24.0 gopkg.in/russross/blackfriday.v2 v2.0.0 // indirect gopkg.in/yaml.v2 v2.2.4 istio.io/gogo-genproto v0.0.0-20191212213402-78a529a42cd8 // indirect diff --git a/go.sum b/go.sum index 8ded1a626ec..29b02420e91 100644 --- a/go.sum +++ b/go.sum @@ -116,10 +116,14 @@ github.com/gogo/protobuf v1.3.0 h1:G8O7TerXerS4F6sx9OV7/nRfJdnXgHZu/S/7F2SN+UE= github.com/gogo/protobuf v1.3.0/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= github.com/gogo/protobuf v1.3.1 h1:DqDEcV5aeaTmdFBePNpYsp3FlcVH/2ISVVM9Qf8PSls= github.com/gogo/protobuf v1.3.1/go.mod h1:SlYgWuQ5SjCEi6WLHjHCa1yvBfUnHcTbrrZtXPKa29o= +github.com/gojek/feast/sdk/go v0.0.0-20200316014539-fb893ded90cd h1:CRugphGHc1UqUhN/kamltFrFCFvHIma43XH4t+HMpVI= +github.com/gojek/feast/sdk/go v0.0.0-20200316014539-fb893ded90cd/go.mod h1:68sgjQ6qtzacAt+Yjj8JX0kNM47Nn/PHNiotZwLTtN8= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= +github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 h1:ZgQEtGgCBiWRM39fZuwSd1LwSqqSW0hOdXCYYDX0R3I= +github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc= github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= github.com/golang/mock v1.2.0 h1:28o5sBqPkBsMGnC6b4MvE2TzSr5/AT4c/1fLqVGIwlk= github.com/golang/mock v1.2.0/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A= @@ -129,12 +133,17 @@ github.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs= github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.5 h1:F768QJ1E9tib+q5Sc8MkdJi1RxLTbRcTf8LJV56aRls= +github.com/golang/protobuf v1.3.5/go.mod h1:6O5/vntMXwX2lRkT1hjjk0nAC1IDOTvTlVgjlRvqsdk= github.com/google/btree v0.0.0-20160524151835-7d79101e329e/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/go-cmp v0.2.0 h1:+dTQ8DZQJz0Mb/HjFlkptS1FeQ4cWSnN941F8aEG4SQ= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= +github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY= github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.3.1 h1:Xye71clBPdm5HgqGwUkwhbynsUJZhDbS20FvLhQ2izg= +github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/gofuzz v0.0.0-20161122191042-44d81051d367/go.mod h1:HP5RmnzzSNb993RKQDq4+1A4ia9nllfqcQFTQJedwGI= github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf/go.mod h1:HP5RmnzzSNb993RKQDq4+1A4ia9nllfqcQFTQJedwGI= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= @@ -161,6 +170,7 @@ github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4= github.com/hashicorp/go-multierror v1.0.0/go.mod h1:dHtQlpGsu+cZNNAkkCN/P3hoUDHhCYQXV3UM06sGGrk= github.com/hashicorp/golang-lru v0.5.0/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= +github.com/hashicorp/golang-lru v0.5.1 h1:0hERBMJE1eitiLkihrMvRVBYAkpHzc/J3QdDN+dAcgU= github.com/hashicorp/golang-lru v0.5.1/go.mod h1:/m3WP610KZHVQ1SGc6re/UDhFvYD7pJ4Ao+sR/qLZy8= github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4= github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ= @@ -223,6 +233,8 @@ github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+W github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA= github.com/onsi/gomega v0.0.0-20190113212917-5533ce8a0da3/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY= +github.com/opentracing/opentracing-go v1.1.0 h1:pWlfV3Bxv7k65HYwkikxat0+s3pV4bsqf19k25Ur8rU= +github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o= github.com/pborman/uuid v1.2.0/go.mod h1:X/NO0urCmaxf9VXbdlT7C2Yzkj2IKimNn4k+gtPdI/k= github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc= github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic= @@ -287,6 +299,7 @@ github.com/stretchr/testify v1.2.0/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXf github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w= github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs= github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= +github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U= github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc= @@ -297,7 +310,10 @@ github.com/xiang90/probing v0.0.0-20160813154853-07dd2e8dfe18/go.mod h1:UETIi67q github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU= github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q= go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU= +go.opencensus.io v0.21.0 h1:mU6zScU4U1YAFPHEHYk+3JC4SY7JxgkqS10ZOSyksNg= go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= +go.opencensus.io v0.22.1 h1:8dP3SGL7MPB94crU3bEPplMPe83FI4EouesJUeFHv50= +go.opencensus.io v0.22.1/go.mod h1:Ap50jQcDJrx6rB6VgeeFPtuPIf3wMRvRfrfYDO6+BmA= go.uber.org/atomic v0.0.0-20181018215023-8dc6146f7569/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE= go.uber.org/multierr v0.0.0-20180122172545-ddea229ff1df/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0= @@ -336,6 +352,7 @@ golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg= golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20190812203447-cdfb69ac37fc/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20191004110552-13f9640d40b9 h1:rjwSpXsdiK0dV8/Naq3kAw9ymfAeJIyd0upUIElB+lI= golang.org/x/net v0.0.0-20191004110552-13f9640d40b9/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= @@ -362,14 +379,17 @@ golang.org/x/sys v0.0.0-20190209173611-3b5209105503/go.mod h1:STP8DvDyc/dI5b8T5h golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20190312061237-fead79001313/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190502145724-3ef323f4f1fd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190922100055-0a153f010e69/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe h1:6fAMxZRR6sl1Uq8U61gxU+kPTs2tR8uOySCbBP7BN/M= golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/text v0.0.0-20160726164857-2910a502d2bf/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.1-0.20181227161524-e6919f6577db/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= +golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs= golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk= golang.org/x/time v0.0.0-20161028155119-f51c12702a4d/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ= @@ -401,13 +421,20 @@ google.golang.org/genproto v0.0.0-20170731182057-09f6ed296fc6/go.mod h1:JiN7NxoA google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8 h1:Nw54tB0rB7hY/N0NQvRW8DG4Yk3Q6T9cu9RcFQDu1tc= google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= google.golang.org/genproto v0.0.0-20190307195333-5fe7a883aa19/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7 h1:ZUjXAXmrAyrmmCPHgCA/vChHcpsX27MZ3yBonD/z1KE= google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb h1:i1Ppqkc3WQXikh8bXiwHqAN5Rv3/qDCcRk0/Otx73BY= +google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= google.golang.org/grpc v1.13.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw= google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= google.golang.org/grpc v1.19.1 h1:TrBcJ1yqAl1G++wO39nD/qtgpsW9/1+QGrluyMGEYgM= google.golang.org/grpc v1.19.1/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= +google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= google.golang.org/grpc v1.21.0/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM= +google.golang.org/grpc v1.23.0 h1:AzbTB6ux+okLTzP8Ru1Xs41C303zdcfEht7MQnYJt5A= google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= +google.golang.org/grpc v1.24.0 h1:vb/1TCsVn3DcJlQ0Gs1yB1pKI6Do2/QNwxdKqmc/b0s= +google.golang.org/grpc v1.24.0/go.mod h1:XDChyiUovWa60DnaeDeZmSW86xtLtjtZbwvSiRnRtcA= gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= diff --git a/protos/Makefile b/protos/Makefile deleted file mode 100644 index 5418d85d20f..00000000000 --- a/protos/Makefile +++ /dev/null @@ -1,33 +0,0 @@ -.PHONY: go - -dirs = core serving types storage -service_dirs = core serving - -gen-go: - @$(foreach dir,$(dirs),protoc -I/usr/local/include -I. --go_out=plugins=grpc,paths=source_relative:../sdk/go/protos/ feast/$(dir)/*.proto;) - -gen-python: - pip install grpcio-tools - pip install mypy-protobuf - @$(foreach dir,$(dirs),python -m grpc_tools.protoc -I. --python_out=../sdk/python/ --mypy_out=../sdk/python/ feast/$(dir)/*.proto;) - @$(foreach dir,$(service_dirs),python -m grpc_tools.protoc -I. --grpc_python_out=../sdk/python/ feast/$(dir)/*.proto;) - -install-dependencies-docs: - mkdir -p $$HOME/bin - mkdir -p $$HOME/include - go get github.com/golang/protobuf/proto && \ - go get gopkg.in/russross/blackfriday.v2 && \ - cd $$(mktemp -d) && \ - git clone https://github.com/istio/tools/ && \ - cd tools/cmd/protoc-gen-docs && \ - go build && \ - cp protoc-gen-docs $$HOME/bin && \ - cd $$HOME && curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v3.11.2/protoc-3.11.2-linux-x86_64.zip && \ - unzip protoc-3.11.2-linux-x86_64.zip -d protoc3 && \ - mv protoc3/bin/* $$HOME/bin/ && \ - chmod +x $$HOME/bin/protoc && \ - mv protoc3/include/* $$HOME/include - -gen-docs: - protoc --docs_out=../dist/grpc feast/*/*.proto || \ - $(MAKE) install-dependencies-docs && PATH=$$HOME/bin:$$PATH protoc -I $$HOME/include/ -I . --docs_out=../dist/grpc feast/*/*.proto \ No newline at end of file diff --git a/sdk/go/client.go b/sdk/go/client.go index 0eaf89b04a2..cb42bf08db4 100644 --- a/sdk/go/client.go +++ b/sdk/go/client.go @@ -53,7 +53,7 @@ func (fc *GrpcClient) GetOnlineFeatures(ctx context.Context, req *OnlineFeatures return &OnlineFeaturesResponse{RawResponse: resp}, err } -// GetInfo gets information about the feast serving instance this client is connected to. +// GetFeastServingInfo gets information about the feast serving instance this client is connected to. func (fc *GrpcClient) GetFeastServingInfo(ctx context.Context, in *serving.GetFeastServingInfoRequest) ( *serving.GetFeastServingInfoResponse, error) { span, ctx := opentracing.StartSpanFromContext(ctx, "get_info") @@ -62,7 +62,7 @@ func (fc *GrpcClient) GetFeastServingInfo(ctx context.Context, in *serving.GetFe return fc.cli.GetFeastServingInfo(ctx, in) } -// Closes the grpc connection. +// Close the grpc connection. func (fc *GrpcClient) Close() error { return fc.conn.Close() -} \ No newline at end of file +} diff --git a/sdk/go/protos/feast/core/FeatureSet.pb.go b/sdk/go/protos/feast/core/FeatureSet.pb.go index 26d9d9c4f7e..5f488caee6e 100644 --- a/sdk/go/protos/feast/core/FeatureSet.pb.go +++ b/sdk/go/protos/feast/core/FeatureSet.pb.go @@ -10,6 +10,7 @@ import ( duration "github.com/golang/protobuf/ptypes/duration" timestamp "github.com/golang/protobuf/ptypes/timestamp" math "math" + v0 "tensorflow_metadata/proto/v0" ) // Reference imports to suppress errors if they are not otherwise used. @@ -204,10 +205,37 @@ type EntitySpec struct { // Name of the entity. Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` // Value type of the feature. - ValueType types.ValueType_Enum `protobuf:"varint,2,opt,name=value_type,json=valueType,proto3,enum=feast.types.ValueType_Enum" json:"value_type,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + ValueType types.ValueType_Enum `protobuf:"varint,2,opt,name=value_type,json=valueType,proto3,enum=feast.types.ValueType_Enum" json:"value_type,omitempty"` + // Types that are valid to be assigned to PresenceConstraints: + // *EntitySpec_Presence + // *EntitySpec_GroupPresence + PresenceConstraints isEntitySpec_PresenceConstraints `protobuf_oneof:"presence_constraints"` + // The shape of the feature which governs the number of values that appear in + // each example. + // + // Types that are valid to be assigned to ShapeType: + // *EntitySpec_Shape + // *EntitySpec_ValueCount + ShapeType isEntitySpec_ShapeType `protobuf_oneof:"shape_type"` + // Domain for the values of the feature. + // + // Types that are valid to be assigned to DomainInfo: + // *EntitySpec_Domain + // *EntitySpec_IntDomain + // *EntitySpec_FloatDomain + // *EntitySpec_StringDomain + // *EntitySpec_BoolDomain + // *EntitySpec_StructDomain + // *EntitySpec_NaturalLanguageDomain + // *EntitySpec_ImageDomain + // *EntitySpec_MidDomain + // *EntitySpec_UrlDomain + // *EntitySpec_TimeDomain + // *EntitySpec_TimeOfDayDomain + DomainInfo isEntitySpec_DomainInfo `protobuf_oneof:"domain_info"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` } func (m *EntitySpec) Reset() { *m = EntitySpec{} } @@ -249,14 +277,304 @@ func (m *EntitySpec) GetValueType() types.ValueType_Enum { return types.ValueType_INVALID } +type isEntitySpec_PresenceConstraints interface { + isEntitySpec_PresenceConstraints() +} + +type EntitySpec_Presence struct { + Presence *v0.FeaturePresence `protobuf:"bytes,3,opt,name=presence,proto3,oneof"` +} + +type EntitySpec_GroupPresence struct { + GroupPresence *v0.FeaturePresenceWithinGroup `protobuf:"bytes,4,opt,name=group_presence,json=groupPresence,proto3,oneof"` +} + +func (*EntitySpec_Presence) isEntitySpec_PresenceConstraints() {} + +func (*EntitySpec_GroupPresence) isEntitySpec_PresenceConstraints() {} + +func (m *EntitySpec) GetPresenceConstraints() isEntitySpec_PresenceConstraints { + if m != nil { + return m.PresenceConstraints + } + return nil +} + +func (m *EntitySpec) GetPresence() *v0.FeaturePresence { + if x, ok := m.GetPresenceConstraints().(*EntitySpec_Presence); ok { + return x.Presence + } + return nil +} + +func (m *EntitySpec) GetGroupPresence() *v0.FeaturePresenceWithinGroup { + if x, ok := m.GetPresenceConstraints().(*EntitySpec_GroupPresence); ok { + return x.GroupPresence + } + return nil +} + +type isEntitySpec_ShapeType interface { + isEntitySpec_ShapeType() +} + +type EntitySpec_Shape struct { + Shape *v0.FixedShape `protobuf:"bytes,5,opt,name=shape,proto3,oneof"` +} + +type EntitySpec_ValueCount struct { + ValueCount *v0.ValueCount `protobuf:"bytes,6,opt,name=value_count,json=valueCount,proto3,oneof"` +} + +func (*EntitySpec_Shape) isEntitySpec_ShapeType() {} + +func (*EntitySpec_ValueCount) isEntitySpec_ShapeType() {} + +func (m *EntitySpec) GetShapeType() isEntitySpec_ShapeType { + if m != nil { + return m.ShapeType + } + return nil +} + +func (m *EntitySpec) GetShape() *v0.FixedShape { + if x, ok := m.GetShapeType().(*EntitySpec_Shape); ok { + return x.Shape + } + return nil +} + +func (m *EntitySpec) GetValueCount() *v0.ValueCount { + if x, ok := m.GetShapeType().(*EntitySpec_ValueCount); ok { + return x.ValueCount + } + return nil +} + +type isEntitySpec_DomainInfo interface { + isEntitySpec_DomainInfo() +} + +type EntitySpec_Domain struct { + Domain string `protobuf:"bytes,7,opt,name=domain,proto3,oneof"` +} + +type EntitySpec_IntDomain struct { + IntDomain *v0.IntDomain `protobuf:"bytes,8,opt,name=int_domain,json=intDomain,proto3,oneof"` +} + +type EntitySpec_FloatDomain struct { + FloatDomain *v0.FloatDomain `protobuf:"bytes,9,opt,name=float_domain,json=floatDomain,proto3,oneof"` +} + +type EntitySpec_StringDomain struct { + StringDomain *v0.StringDomain `protobuf:"bytes,10,opt,name=string_domain,json=stringDomain,proto3,oneof"` +} + +type EntitySpec_BoolDomain struct { + BoolDomain *v0.BoolDomain `protobuf:"bytes,11,opt,name=bool_domain,json=boolDomain,proto3,oneof"` +} + +type EntitySpec_StructDomain struct { + StructDomain *v0.StructDomain `protobuf:"bytes,12,opt,name=struct_domain,json=structDomain,proto3,oneof"` +} + +type EntitySpec_NaturalLanguageDomain struct { + NaturalLanguageDomain *v0.NaturalLanguageDomain `protobuf:"bytes,13,opt,name=natural_language_domain,json=naturalLanguageDomain,proto3,oneof"` +} + +type EntitySpec_ImageDomain struct { + ImageDomain *v0.ImageDomain `protobuf:"bytes,14,opt,name=image_domain,json=imageDomain,proto3,oneof"` +} + +type EntitySpec_MidDomain struct { + MidDomain *v0.MIDDomain `protobuf:"bytes,15,opt,name=mid_domain,json=midDomain,proto3,oneof"` +} + +type EntitySpec_UrlDomain struct { + UrlDomain *v0.URLDomain `protobuf:"bytes,16,opt,name=url_domain,json=urlDomain,proto3,oneof"` +} + +type EntitySpec_TimeDomain struct { + TimeDomain *v0.TimeDomain `protobuf:"bytes,17,opt,name=time_domain,json=timeDomain,proto3,oneof"` +} + +type EntitySpec_TimeOfDayDomain struct { + TimeOfDayDomain *v0.TimeOfDayDomain `protobuf:"bytes,18,opt,name=time_of_day_domain,json=timeOfDayDomain,proto3,oneof"` +} + +func (*EntitySpec_Domain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_IntDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_FloatDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_StringDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_BoolDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_StructDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_NaturalLanguageDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_ImageDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_MidDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_UrlDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_TimeDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_TimeOfDayDomain) isEntitySpec_DomainInfo() {} + +func (m *EntitySpec) GetDomainInfo() isEntitySpec_DomainInfo { + if m != nil { + return m.DomainInfo + } + return nil +} + +func (m *EntitySpec) GetDomain() string { + if x, ok := m.GetDomainInfo().(*EntitySpec_Domain); ok { + return x.Domain + } + return "" +} + +func (m *EntitySpec) GetIntDomain() *v0.IntDomain { + if x, ok := m.GetDomainInfo().(*EntitySpec_IntDomain); ok { + return x.IntDomain + } + return nil +} + +func (m *EntitySpec) GetFloatDomain() *v0.FloatDomain { + if x, ok := m.GetDomainInfo().(*EntitySpec_FloatDomain); ok { + return x.FloatDomain + } + return nil +} + +func (m *EntitySpec) GetStringDomain() *v0.StringDomain { + if x, ok := m.GetDomainInfo().(*EntitySpec_StringDomain); ok { + return x.StringDomain + } + return nil +} + +func (m *EntitySpec) GetBoolDomain() *v0.BoolDomain { + if x, ok := m.GetDomainInfo().(*EntitySpec_BoolDomain); ok { + return x.BoolDomain + } + return nil +} + +func (m *EntitySpec) GetStructDomain() *v0.StructDomain { + if x, ok := m.GetDomainInfo().(*EntitySpec_StructDomain); ok { + return x.StructDomain + } + return nil +} + +func (m *EntitySpec) GetNaturalLanguageDomain() *v0.NaturalLanguageDomain { + if x, ok := m.GetDomainInfo().(*EntitySpec_NaturalLanguageDomain); ok { + return x.NaturalLanguageDomain + } + return nil +} + +func (m *EntitySpec) GetImageDomain() *v0.ImageDomain { + if x, ok := m.GetDomainInfo().(*EntitySpec_ImageDomain); ok { + return x.ImageDomain + } + return nil +} + +func (m *EntitySpec) GetMidDomain() *v0.MIDDomain { + if x, ok := m.GetDomainInfo().(*EntitySpec_MidDomain); ok { + return x.MidDomain + } + return nil +} + +func (m *EntitySpec) GetUrlDomain() *v0.URLDomain { + if x, ok := m.GetDomainInfo().(*EntitySpec_UrlDomain); ok { + return x.UrlDomain + } + return nil +} + +func (m *EntitySpec) GetTimeDomain() *v0.TimeDomain { + if x, ok := m.GetDomainInfo().(*EntitySpec_TimeDomain); ok { + return x.TimeDomain + } + return nil +} + +func (m *EntitySpec) GetTimeOfDayDomain() *v0.TimeOfDayDomain { + if x, ok := m.GetDomainInfo().(*EntitySpec_TimeOfDayDomain); ok { + return x.TimeOfDayDomain + } + return nil +} + +// XXX_OneofWrappers is for the internal use of the proto package. +func (*EntitySpec) XXX_OneofWrappers() []interface{} { + return []interface{}{ + (*EntitySpec_Presence)(nil), + (*EntitySpec_GroupPresence)(nil), + (*EntitySpec_Shape)(nil), + (*EntitySpec_ValueCount)(nil), + (*EntitySpec_Domain)(nil), + (*EntitySpec_IntDomain)(nil), + (*EntitySpec_FloatDomain)(nil), + (*EntitySpec_StringDomain)(nil), + (*EntitySpec_BoolDomain)(nil), + (*EntitySpec_StructDomain)(nil), + (*EntitySpec_NaturalLanguageDomain)(nil), + (*EntitySpec_ImageDomain)(nil), + (*EntitySpec_MidDomain)(nil), + (*EntitySpec_UrlDomain)(nil), + (*EntitySpec_TimeDomain)(nil), + (*EntitySpec_TimeOfDayDomain)(nil), + } +} + type FeatureSpec struct { // Name of the feature. Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` // Value type of the feature. - ValueType types.ValueType_Enum `protobuf:"varint,2,opt,name=value_type,json=valueType,proto3,enum=feast.types.ValueType_Enum" json:"value_type,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + ValueType types.ValueType_Enum `protobuf:"varint,2,opt,name=value_type,json=valueType,proto3,enum=feast.types.ValueType_Enum" json:"value_type,omitempty"` + // Types that are valid to be assigned to PresenceConstraints: + // *FeatureSpec_Presence + // *FeatureSpec_GroupPresence + PresenceConstraints isFeatureSpec_PresenceConstraints `protobuf_oneof:"presence_constraints"` + // The shape of the feature which governs the number of values that appear in + // each example. + // + // Types that are valid to be assigned to ShapeType: + // *FeatureSpec_Shape + // *FeatureSpec_ValueCount + ShapeType isFeatureSpec_ShapeType `protobuf_oneof:"shape_type"` + // Domain for the values of the feature. + // + // Types that are valid to be assigned to DomainInfo: + // *FeatureSpec_Domain + // *FeatureSpec_IntDomain + // *FeatureSpec_FloatDomain + // *FeatureSpec_StringDomain + // *FeatureSpec_BoolDomain + // *FeatureSpec_StructDomain + // *FeatureSpec_NaturalLanguageDomain + // *FeatureSpec_ImageDomain + // *FeatureSpec_MidDomain + // *FeatureSpec_UrlDomain + // *FeatureSpec_TimeDomain + // *FeatureSpec_TimeOfDayDomain + DomainInfo isFeatureSpec_DomainInfo `protobuf_oneof:"domain_info"` + XXX_NoUnkeyedLiteral struct{} `json:"-"` + XXX_unrecognized []byte `json:"-"` + XXX_sizecache int32 `json:"-"` } func (m *FeatureSpec) Reset() { *m = FeatureSpec{} } @@ -298,6 +616,269 @@ func (m *FeatureSpec) GetValueType() types.ValueType_Enum { return types.ValueType_INVALID } +type isFeatureSpec_PresenceConstraints interface { + isFeatureSpec_PresenceConstraints() +} + +type FeatureSpec_Presence struct { + Presence *v0.FeaturePresence `protobuf:"bytes,3,opt,name=presence,proto3,oneof"` +} + +type FeatureSpec_GroupPresence struct { + GroupPresence *v0.FeaturePresenceWithinGroup `protobuf:"bytes,4,opt,name=group_presence,json=groupPresence,proto3,oneof"` +} + +func (*FeatureSpec_Presence) isFeatureSpec_PresenceConstraints() {} + +func (*FeatureSpec_GroupPresence) isFeatureSpec_PresenceConstraints() {} + +func (m *FeatureSpec) GetPresenceConstraints() isFeatureSpec_PresenceConstraints { + if m != nil { + return m.PresenceConstraints + } + return nil +} + +func (m *FeatureSpec) GetPresence() *v0.FeaturePresence { + if x, ok := m.GetPresenceConstraints().(*FeatureSpec_Presence); ok { + return x.Presence + } + return nil +} + +func (m *FeatureSpec) GetGroupPresence() *v0.FeaturePresenceWithinGroup { + if x, ok := m.GetPresenceConstraints().(*FeatureSpec_GroupPresence); ok { + return x.GroupPresence + } + return nil +} + +type isFeatureSpec_ShapeType interface { + isFeatureSpec_ShapeType() +} + +type FeatureSpec_Shape struct { + Shape *v0.FixedShape `protobuf:"bytes,5,opt,name=shape,proto3,oneof"` +} + +type FeatureSpec_ValueCount struct { + ValueCount *v0.ValueCount `protobuf:"bytes,6,opt,name=value_count,json=valueCount,proto3,oneof"` +} + +func (*FeatureSpec_Shape) isFeatureSpec_ShapeType() {} + +func (*FeatureSpec_ValueCount) isFeatureSpec_ShapeType() {} + +func (m *FeatureSpec) GetShapeType() isFeatureSpec_ShapeType { + if m != nil { + return m.ShapeType + } + return nil +} + +func (m *FeatureSpec) GetShape() *v0.FixedShape { + if x, ok := m.GetShapeType().(*FeatureSpec_Shape); ok { + return x.Shape + } + return nil +} + +func (m *FeatureSpec) GetValueCount() *v0.ValueCount { + if x, ok := m.GetShapeType().(*FeatureSpec_ValueCount); ok { + return x.ValueCount + } + return nil +} + +type isFeatureSpec_DomainInfo interface { + isFeatureSpec_DomainInfo() +} + +type FeatureSpec_Domain struct { + Domain string `protobuf:"bytes,7,opt,name=domain,proto3,oneof"` +} + +type FeatureSpec_IntDomain struct { + IntDomain *v0.IntDomain `protobuf:"bytes,8,opt,name=int_domain,json=intDomain,proto3,oneof"` +} + +type FeatureSpec_FloatDomain struct { + FloatDomain *v0.FloatDomain `protobuf:"bytes,9,opt,name=float_domain,json=floatDomain,proto3,oneof"` +} + +type FeatureSpec_StringDomain struct { + StringDomain *v0.StringDomain `protobuf:"bytes,10,opt,name=string_domain,json=stringDomain,proto3,oneof"` +} + +type FeatureSpec_BoolDomain struct { + BoolDomain *v0.BoolDomain `protobuf:"bytes,11,opt,name=bool_domain,json=boolDomain,proto3,oneof"` +} + +type FeatureSpec_StructDomain struct { + StructDomain *v0.StructDomain `protobuf:"bytes,12,opt,name=struct_domain,json=structDomain,proto3,oneof"` +} + +type FeatureSpec_NaturalLanguageDomain struct { + NaturalLanguageDomain *v0.NaturalLanguageDomain `protobuf:"bytes,13,opt,name=natural_language_domain,json=naturalLanguageDomain,proto3,oneof"` +} + +type FeatureSpec_ImageDomain struct { + ImageDomain *v0.ImageDomain `protobuf:"bytes,14,opt,name=image_domain,json=imageDomain,proto3,oneof"` +} + +type FeatureSpec_MidDomain struct { + MidDomain *v0.MIDDomain `protobuf:"bytes,15,opt,name=mid_domain,json=midDomain,proto3,oneof"` +} + +type FeatureSpec_UrlDomain struct { + UrlDomain *v0.URLDomain `protobuf:"bytes,16,opt,name=url_domain,json=urlDomain,proto3,oneof"` +} + +type FeatureSpec_TimeDomain struct { + TimeDomain *v0.TimeDomain `protobuf:"bytes,17,opt,name=time_domain,json=timeDomain,proto3,oneof"` +} + +type FeatureSpec_TimeOfDayDomain struct { + TimeOfDayDomain *v0.TimeOfDayDomain `protobuf:"bytes,18,opt,name=time_of_day_domain,json=timeOfDayDomain,proto3,oneof"` +} + +func (*FeatureSpec_Domain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_IntDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_FloatDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_StringDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_BoolDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_StructDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_NaturalLanguageDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_ImageDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_MidDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_UrlDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_TimeDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_TimeOfDayDomain) isFeatureSpec_DomainInfo() {} + +func (m *FeatureSpec) GetDomainInfo() isFeatureSpec_DomainInfo { + if m != nil { + return m.DomainInfo + } + return nil +} + +func (m *FeatureSpec) GetDomain() string { + if x, ok := m.GetDomainInfo().(*FeatureSpec_Domain); ok { + return x.Domain + } + return "" +} + +func (m *FeatureSpec) GetIntDomain() *v0.IntDomain { + if x, ok := m.GetDomainInfo().(*FeatureSpec_IntDomain); ok { + return x.IntDomain + } + return nil +} + +func (m *FeatureSpec) GetFloatDomain() *v0.FloatDomain { + if x, ok := m.GetDomainInfo().(*FeatureSpec_FloatDomain); ok { + return x.FloatDomain + } + return nil +} + +func (m *FeatureSpec) GetStringDomain() *v0.StringDomain { + if x, ok := m.GetDomainInfo().(*FeatureSpec_StringDomain); ok { + return x.StringDomain + } + return nil +} + +func (m *FeatureSpec) GetBoolDomain() *v0.BoolDomain { + if x, ok := m.GetDomainInfo().(*FeatureSpec_BoolDomain); ok { + return x.BoolDomain + } + return nil +} + +func (m *FeatureSpec) GetStructDomain() *v0.StructDomain { + if x, ok := m.GetDomainInfo().(*FeatureSpec_StructDomain); ok { + return x.StructDomain + } + return nil +} + +func (m *FeatureSpec) GetNaturalLanguageDomain() *v0.NaturalLanguageDomain { + if x, ok := m.GetDomainInfo().(*FeatureSpec_NaturalLanguageDomain); ok { + return x.NaturalLanguageDomain + } + return nil +} + +func (m *FeatureSpec) GetImageDomain() *v0.ImageDomain { + if x, ok := m.GetDomainInfo().(*FeatureSpec_ImageDomain); ok { + return x.ImageDomain + } + return nil +} + +func (m *FeatureSpec) GetMidDomain() *v0.MIDDomain { + if x, ok := m.GetDomainInfo().(*FeatureSpec_MidDomain); ok { + return x.MidDomain + } + return nil +} + +func (m *FeatureSpec) GetUrlDomain() *v0.URLDomain { + if x, ok := m.GetDomainInfo().(*FeatureSpec_UrlDomain); ok { + return x.UrlDomain + } + return nil +} + +func (m *FeatureSpec) GetTimeDomain() *v0.TimeDomain { + if x, ok := m.GetDomainInfo().(*FeatureSpec_TimeDomain); ok { + return x.TimeDomain + } + return nil +} + +func (m *FeatureSpec) GetTimeOfDayDomain() *v0.TimeOfDayDomain { + if x, ok := m.GetDomainInfo().(*FeatureSpec_TimeOfDayDomain); ok { + return x.TimeOfDayDomain + } + return nil +} + +// XXX_OneofWrappers is for the internal use of the proto package. +func (*FeatureSpec) XXX_OneofWrappers() []interface{} { + return []interface{}{ + (*FeatureSpec_Presence)(nil), + (*FeatureSpec_GroupPresence)(nil), + (*FeatureSpec_Shape)(nil), + (*FeatureSpec_ValueCount)(nil), + (*FeatureSpec_Domain)(nil), + (*FeatureSpec_IntDomain)(nil), + (*FeatureSpec_FloatDomain)(nil), + (*FeatureSpec_StringDomain)(nil), + (*FeatureSpec_BoolDomain)(nil), + (*FeatureSpec_StructDomain)(nil), + (*FeatureSpec_NaturalLanguageDomain)(nil), + (*FeatureSpec_ImageDomain)(nil), + (*FeatureSpec_MidDomain)(nil), + (*FeatureSpec_UrlDomain)(nil), + (*FeatureSpec_TimeDomain)(nil), + (*FeatureSpec_TimeOfDayDomain)(nil), + } +} + type FeatureSetMeta struct { // Created timestamp of this specific feature set. CreatedTimestamp *timestamp.Timestamp `protobuf:"bytes,1,opt,name=created_timestamp,json=createdTimestamp,proto3" json:"created_timestamp,omitempty"` @@ -364,37 +945,64 @@ func init() { func init() { proto.RegisterFile("feast/core/FeatureSet.proto", fileDescriptor_972fbd278ac19c0c) } var fileDescriptor_972fbd278ac19c0c = []byte{ - // 510 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xac, 0x93, 0x4f, 0x6f, 0xda, 0x30, - 0x18, 0xc6, 0x07, 0xa5, 0x50, 0x5e, 0x26, 0x96, 0xf9, 0xb0, 0x66, 0xed, 0xb4, 0x21, 0x4e, 0xa8, - 0x07, 0x5b, 0x4a, 0x77, 0xda, 0x8d, 0x0a, 0x56, 0x21, 0x75, 0xa8, 0x72, 0x58, 0xa5, 0x4d, 0x9b, - 0x90, 0x09, 0x2f, 0x59, 0x5a, 0x82, 0xa3, 0xd8, 0x41, 0xe5, 0x53, 0xec, 0x33, 0xec, 0x9b, 0x4e, - 0x71, 0x12, 0x92, 0xa1, 0x6e, 0xa7, 0xdd, 0x62, 0x3f, 0x3f, 0xbf, 0x79, 0xde, 0x7f, 0x70, 0xbe, - 0x42, 0xa1, 0x34, 0xf3, 0x64, 0x8c, 0xec, 0x23, 0x0a, 0x9d, 0xc4, 0xe8, 0xa2, 0xa6, 0x51, 0x2c, - 0xb5, 0x24, 0x60, 0x44, 0x9a, 0x8a, 0x67, 0xa7, 0x19, 0xa8, 0x77, 0x11, 0x2a, 0x76, 0x27, 0xd6, - 0x09, 0x66, 0x50, 0x21, 0x98, 0x08, 0xae, 0x4c, 0x62, 0xaf, 0x10, 0xde, 0xfa, 0x52, 0xfa, 0x6b, - 0x64, 0xe6, 0xb4, 0x48, 0x56, 0x6c, 0x99, 0xc4, 0x42, 0x07, 0x72, 0x93, 0xeb, 0xef, 0x0e, 0x75, - 0x1d, 0x84, 0xa8, 0xb4, 0x08, 0xa3, 0x0c, 0xe8, 0xaf, 0x01, 0x4a, 0x4b, 0x84, 0x42, 0x43, 0x45, - 0xe8, 0xd9, 0xb5, 0x5e, 0x6d, 0xd0, 0x71, 0xce, 0x68, 0xe9, 0x8d, 0x96, 0x94, 0x1b, 0xa1, 0xc7, - 0x0d, 0x97, 0xf2, 0x21, 0x6a, 0x61, 0xd7, 0xff, 0xc5, 0x7f, 0x42, 0x2d, 0xb8, 0xe1, 0xfa, 0xbf, - 0xea, 0xd0, 0xfd, 0x33, 0x10, 0xb1, 0xa1, 0x15, 0xc5, 0xf2, 0x1e, 0x3d, 0x6d, 0xb7, 0x7a, 0xb5, - 0x41, 0x9b, 0x17, 0x47, 0x42, 0xa0, 0xb1, 0x11, 0x21, 0x1a, 0x33, 0x6d, 0x6e, 0xbe, 0x53, 0x7a, - 0x8b, 0xb1, 0x0a, 0xe4, 0xc6, 0xfc, 0xf3, 0x98, 0x17, 0x47, 0xe2, 0xc0, 0x09, 0x6e, 0x74, 0xa0, - 0x03, 0x54, 0xf6, 0x51, 0xef, 0x68, 0xd0, 0x71, 0x5e, 0x55, 0xed, 0x8c, 0x53, 0x6d, 0x67, 0xac, - 0xef, 0x39, 0x72, 0x09, 0x27, 0xab, 0xcc, 0x8d, 0xb2, 0x1b, 0xe6, 0xcd, 0xe9, 0x53, 0x29, 0x98, - 0x47, 0x05, 0x48, 0x1c, 0x68, 0x85, 0xe2, 0x71, 0x2e, 0x7c, 0xb4, 0x8f, 0x4d, 0xda, 0xaf, 0x69, - 0x56, 0x64, 0x5a, 0x14, 0x99, 0x8e, 0xf2, 0x26, 0xf0, 0x66, 0x28, 0x1e, 0x87, 0x3e, 0x92, 0x0b, - 0x68, 0x2a, 0xd3, 0x36, 0xbb, 0x69, 0x9e, 0x90, 0xea, 0x6f, 0xb2, 0x86, 0xf2, 0x9c, 0xe8, 0x7f, - 0x03, 0x28, 0xcd, 0x3e, 0x59, 0x84, 0x0f, 0x00, 0xdb, 0x74, 0x38, 0xe6, 0xe9, 0xa0, 0x98, 0x3a, - 0x74, 0x9d, 0xf3, 0x3c, 0xa2, 0x99, 0x1d, 0x6a, 0x66, 0x67, 0xb6, 0x8b, 0xd2, 0xbc, 0x93, 0x90, - 0xb7, 0xb7, 0xc5, 0xb9, 0xff, 0x1d, 0x3a, 0x95, 0xb4, 0xfe, 0x7b, 0xf8, 0x9f, 0xb5, 0x6a, 0x83, - 0xd3, 0xce, 0x93, 0x6b, 0x78, 0xe9, 0xc5, 0x28, 0x34, 0x2e, 0xe7, 0xfb, 0xe1, 0xdb, 0x0f, 0xd8, - 0x61, 0xe5, 0x66, 0x05, 0xc1, 0xad, 0xfc, 0xd1, 0xfe, 0x86, 0xbc, 0x87, 0xa6, 0xd2, 0x42, 0x27, - 0x2a, 0xf7, 0xf4, 0xe6, 0x2f, 0xe3, 0x69, 0x18, 0x9e, 0xb3, 0x17, 0x37, 0x60, 0x1d, 0x6a, 0x84, - 0x40, 0xd7, 0x9d, 0x0d, 0x67, 0x9f, 0xdd, 0xf9, 0x64, 0x7a, 0x37, 0xbc, 0x99, 0x8c, 0xac, 0x67, - 0x95, 0xbb, 0xdb, 0xf1, 0x74, 0x34, 0x99, 0x5e, 0x5b, 0x35, 0x62, 0xc1, 0xf3, 0xfc, 0x8e, 0x8f, - 0x87, 0xa3, 0x2f, 0x56, 0xfd, 0x6a, 0x0a, 0x95, 0x7d, 0xbd, 0x7a, 0x51, 0x46, 0xbe, 0x4d, 0x33, - 0xf8, 0xca, 0xfc, 0x40, 0xff, 0x48, 0x16, 0xd4, 0x93, 0x21, 0xf3, 0xe5, 0x3d, 0x3e, 0xb0, 0x6c, - 0x71, 0xd5, 0xf2, 0x81, 0xf9, 0x32, 0xdb, 0x42, 0xc5, 0xca, 0x65, 0x5e, 0x34, 0xcd, 0xd5, 0xe5, - 0xef, 0x00, 0x00, 0x00, 0xff, 0xff, 0xb8, 0xd5, 0xf0, 0x13, 0x23, 0x04, 0x00, 0x00, + // 938 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xec, 0x97, 0xdf, 0x6e, 0xe2, 0xc6, + 0x17, 0xc7, 0x03, 0x49, 0x48, 0x38, 0x10, 0xc2, 0x8e, 0x7e, 0xbf, 0x8d, 0xbb, 0x5b, 0xb5, 0x29, + 0xad, 0xd4, 0x74, 0xa5, 0xda, 0x2b, 0xb6, 0x57, 0x7b, 0x17, 0x0a, 0x0d, 0xa8, 0x59, 0x1a, 0x19, + 0x36, 0x55, 0xdb, 0x0b, 0x6b, 0x30, 0x83, 0x33, 0xbb, 0xf6, 0x8c, 0xe5, 0x19, 0xd3, 0xf0, 0x14, + 0x7d, 0x80, 0xf6, 0xa6, 0x6f, 0x5a, 0xcd, 0xd8, 0x63, 0x93, 0x28, 0xd0, 0x3e, 0x00, 0x77, 0x39, + 0x3e, 0xdf, 0xf9, 0xf8, 0xfc, 0xcb, 0xc1, 0x03, 0x2f, 0x17, 0x04, 0x0b, 0xe9, 0xf8, 0x3c, 0x21, + 0xce, 0x0f, 0x04, 0xcb, 0x34, 0x21, 0x13, 0x22, 0xed, 0x38, 0xe1, 0x92, 0x23, 0xd0, 0x4e, 0x5b, + 0x39, 0x5f, 0x9c, 0x65, 0x42, 0xb9, 0x8a, 0x89, 0x70, 0x6e, 0x71, 0x98, 0x92, 0x4c, 0x64, 0x1c, + 0x9a, 0x30, 0xe1, 0x69, 0xe2, 0x1b, 0xc7, 0x67, 0x01, 0xe7, 0x41, 0x48, 0x1c, 0x6d, 0xcd, 0xd2, + 0x85, 0x33, 0x4f, 0x13, 0x2c, 0x29, 0x67, 0xb9, 0xff, 0xf3, 0xc7, 0x7e, 0x49, 0x23, 0x22, 0x24, + 0x8e, 0xe2, 0x5c, 0xf0, 0x8d, 0x24, 0x4c, 0xf0, 0x64, 0x11, 0xf2, 0xdf, 0xbd, 0x88, 0x48, 0x3c, + 0xc7, 0x12, 0x67, 0x6a, 0x67, 0xf9, 0xda, 0x11, 0xfe, 0x1d, 0x89, 0x70, 0x26, 0xed, 0x84, 0x00, + 0x65, 0xf4, 0xc8, 0x86, 0x03, 0x11, 0x13, 0xdf, 0xaa, 0x9c, 0x57, 0x2e, 0x1a, 0xdd, 0x17, 0x76, + 0x99, 0x86, 0x5d, 0xaa, 0x26, 0x31, 0xf1, 0x5d, 0xad, 0x53, 0x7a, 0xc5, 0xb7, 0xaa, 0xdb, 0xf4, + 0xef, 0x88, 0xc4, 0xae, 0xd6, 0x75, 0xfe, 0xae, 0x42, 0xeb, 0x21, 0x08, 0x59, 0x70, 0x14, 0x27, + 0xfc, 0x03, 0xf1, 0xa5, 0x75, 0x74, 0x5e, 0xb9, 0xa8, 0xbb, 0xc6, 0x44, 0x08, 0x0e, 0x18, 0x8e, + 0x88, 0x0e, 0xa6, 0xee, 0xea, 0xbf, 0x95, 0x7a, 0x49, 0x12, 0x41, 0x39, 0xd3, 0xef, 0x3c, 0x74, + 0x8d, 0x89, 0xba, 0x70, 0x4c, 0x98, 0xa4, 0x92, 0x12, 0x61, 0xed, 0x9f, 0xef, 0x5f, 0x34, 0xba, + 0xcf, 0xd7, 0xc3, 0x19, 0x28, 0xdf, 0x4a, 0x87, 0x5e, 0xe8, 0xd0, 0x1b, 0x38, 0x5e, 0x64, 0xd1, + 0x08, 0xeb, 0x40, 0x9f, 0x39, 0x7b, 0x2a, 0x05, 0x7d, 0xc8, 0x08, 0x51, 0x17, 0x8e, 0x22, 0x7c, + 0xef, 0xe1, 0x80, 0x58, 0x87, 0x3a, 0xed, 0x4f, 0xec, 0xac, 0x1f, 0xb6, 0xe9, 0x87, 0xdd, 0xcf, + 0xfb, 0xe5, 0xd6, 0x22, 0x7c, 0x7f, 0x19, 0x10, 0xf4, 0x0a, 0x6a, 0x42, 0x77, 0xd8, 0xaa, 0xe9, + 0x23, 0x68, 0xfd, 0x35, 0x59, 0xef, 0xdd, 0x5c, 0xd1, 0xf9, 0x13, 0x00, 0xca, 0x68, 0x9f, 0xac, + 0xc2, 0x5b, 0x80, 0xa5, 0x1a, 0x24, 0x4f, 0x0d, 0x95, 0x2e, 0x44, 0xab, 0xfb, 0x32, 0x47, 0xea, + 0x39, 0xb3, 0xf5, 0x9c, 0x4d, 0x57, 0xb1, 0x4a, 0x3c, 0x8d, 0xdc, 0xfa, 0xd2, 0xd8, 0x68, 0x00, + 0xc7, 0x71, 0x42, 0x04, 0x61, 0x3e, 0xb1, 0xf6, 0x75, 0x30, 0x5f, 0xdb, 0xe5, 0xb8, 0xd8, 0x66, + 0x5c, 0xec, 0xe5, 0x6b, 0x93, 0xff, 0x4d, 0x2e, 0x1f, 0xee, 0xb9, 0xc5, 0x51, 0xf4, 0x1b, 0xb4, + 0x82, 0x84, 0xa7, 0xb1, 0x57, 0xc0, 0x0e, 0x34, 0xac, 0xfb, 0x1f, 0x61, 0x3f, 0x53, 0x79, 0x47, + 0xd9, 0x95, 0x42, 0x0c, 0xf7, 0xdc, 0x13, 0xcd, 0x32, 0x3e, 0xf4, 0x16, 0x0e, 0xc5, 0x1d, 0x8e, + 0x4d, 0x81, 0x3b, 0x1b, 0x99, 0xf4, 0x9e, 0xcc, 0x27, 0x4a, 0x39, 0xac, 0xb8, 0xd9, 0x11, 0x34, + 0x80, 0x46, 0x56, 0x1b, 0x9f, 0xa7, 0x4c, 0xe6, 0xf5, 0xde, 0x48, 0xd0, 0x75, 0xfa, 0x5e, 0x29, + 0x87, 0x15, 0x37, 0x2b, 0xaa, 0xb6, 0x90, 0x05, 0xb5, 0x39, 0x8f, 0x30, 0x65, 0xd9, 0x54, 0x0e, + 0xab, 0x6e, 0x6e, 0xa3, 0x1e, 0x00, 0x65, 0xd2, 0xcb, 0xbd, 0xc7, 0x9a, 0xff, 0xc5, 0x26, 0xfe, + 0x88, 0xc9, 0xbe, 0x16, 0x0e, 0xab, 0x6e, 0x9d, 0x1a, 0x03, 0x0d, 0xa1, 0xb9, 0x08, 0x39, 0x2e, + 0x28, 0x75, 0x4d, 0xf9, 0x72, 0x63, 0x9e, 0x4a, 0x5b, 0x70, 0x1a, 0x8b, 0xd2, 0x44, 0x3f, 0xc2, + 0x89, 0x90, 0x09, 0x65, 0x81, 0x41, 0x81, 0x46, 0x7d, 0xb5, 0x09, 0x35, 0xd1, 0xe2, 0x82, 0xd5, + 0x14, 0x6b, 0xb6, 0xaa, 0xdd, 0x8c, 0xf3, 0xd0, 0xa0, 0x1a, 0xdb, 0x6b, 0xd7, 0xe3, 0x3c, 0x2c, + 0x40, 0x30, 0x2b, 0xac, 0x3c, 0xa6, 0xd4, 0x2f, 0xd2, 0x6b, 0xfe, 0x6b, 0x4c, 0xa9, 0x2f, 0x1f, + 0xc4, 0x54, 0xd8, 0x28, 0x80, 0x33, 0xa6, 0x26, 0x07, 0x87, 0x5e, 0x88, 0x59, 0x90, 0xe2, 0x80, + 0x18, 0xec, 0x89, 0xc6, 0x7e, 0xbb, 0x09, 0x3b, 0xce, 0x8e, 0x5d, 0xe7, 0xa7, 0x0a, 0xfe, 0xff, + 0xd9, 0x53, 0x0e, 0xd5, 0x13, 0x1a, 0xad, 0xd1, 0x5b, 0xdb, 0x7b, 0x32, 0x8a, 0xd6, 0x99, 0x0d, + 0x5a, 0x9a, 0x6a, 0x42, 0x22, 0x3a, 0x37, 0x9c, 0xd3, 0xed, 0x13, 0xf2, 0x6e, 0xd4, 0x2f, 0x27, + 0x24, 0xa2, 0xf3, 0x92, 0x91, 0x26, 0x45, 0x27, 0xda, 0xdb, 0x19, 0xef, 0xdd, 0xeb, 0x92, 0x91, + 0x26, 0x61, 0xd9, 0x4e, 0xf5, 0xcb, 0x60, 0x20, 0xcf, 0xb6, 0xb7, 0x73, 0x4a, 0xa3, 0x32, 0x1f, + 0x90, 0x85, 0x85, 0x6e, 0x01, 0x69, 0x0c, 0x5f, 0x78, 0x73, 0xbc, 0x32, 0x34, 0xb4, 0x7d, 0x77, + 0x28, 0xda, 0x4f, 0x8b, 0x3e, 0x5e, 0x15, 0xc8, 0x53, 0xf9, 0xf0, 0x51, 0xef, 0x39, 0xfc, 0xcf, + 0x2c, 0x0f, 0xcf, 0xe7, 0x4c, 0xc8, 0x04, 0x53, 0x26, 0x45, 0xaf, 0x09, 0xa0, 0xff, 0x95, 0xf5, + 0x76, 0xeb, 0x9d, 0x40, 0x23, 0x7b, 0xa3, 0x47, 0xd9, 0x82, 0x77, 0xfe, 0x02, 0x68, 0xac, 0xed, + 0xe5, 0xdd, 0x7a, 0xdc, 0xad, 0xc7, 0xdd, 0x7a, 0xdc, 0xad, 0xc7, 0xdd, 0x7a, 0xcc, 0xd6, 0xe3, + 0x1f, 0x95, 0xf5, 0x0f, 0x6c, 0xf5, 0xe5, 0x8d, 0xae, 0xe0, 0x99, 0x9f, 0x10, 0x2c, 0xc9, 0xdc, + 0x2b, 0xee, 0x09, 0xc5, 0x07, 0xfe, 0xe3, 0x2f, 0xd7, 0xa9, 0x51, 0xb8, 0xed, 0xfc, 0x50, 0xf1, + 0x04, 0x7d, 0x07, 0x35, 0x21, 0xb1, 0x4c, 0x45, 0xbe, 0x52, 0x3f, 0xdd, 0x70, 0x3d, 0xd0, 0x1a, + 0x37, 0xd7, 0xbe, 0xba, 0x86, 0xf6, 0x63, 0x1f, 0x42, 0xd0, 0x9a, 0x4c, 0x2f, 0xa7, 0xef, 0x27, + 0xde, 0x68, 0x7c, 0x7b, 0x79, 0x3d, 0xea, 0xb7, 0xf7, 0xd6, 0x9e, 0xdd, 0x0c, 0xc6, 0xfd, 0xd1, + 0xf8, 0xaa, 0x5d, 0x41, 0x6d, 0x68, 0xe6, 0xcf, 0xdc, 0xc1, 0x65, 0xff, 0x97, 0x76, 0xb5, 0x37, + 0x86, 0xb5, 0xab, 0x55, 0xef, 0xb4, 0x24, 0xdf, 0xa8, 0x0c, 0x7e, 0x75, 0x02, 0x2a, 0xef, 0xd2, + 0x99, 0xed, 0xf3, 0xc8, 0x09, 0xf8, 0x07, 0xf2, 0xd1, 0xc9, 0xee, 0x58, 0x62, 0xfe, 0xd1, 0x09, + 0x78, 0x76, 0x05, 0x12, 0x4e, 0x79, 0xef, 0x9a, 0xd5, 0xf4, 0xa3, 0x37, 0xff, 0x04, 0x00, 0x00, + 0xff, 0xff, 0x4e, 0x2a, 0x9e, 0xd9, 0xce, 0x0d, 0x00, 0x00, } diff --git a/sdk/go/protos/feast/core/Store.pb.go b/sdk/go/protos/feast/core/Store.pb.go index 9120edcb42c..62bb16d0adb 100644 --- a/sdk/go/protos/feast/core/Store.pb.go +++ b/sdk/go/protos/feast/core/Store.pb.go @@ -250,8 +250,13 @@ func (*Store) XXX_OneofWrappers() []interface{} { } type Store_RedisConfig struct { - Host string `protobuf:"bytes,1,opt,name=host,proto3" json:"host,omitempty"` - Port int32 `protobuf:"varint,2,opt,name=port,proto3" json:"port,omitempty"` + Host string `protobuf:"bytes,1,opt,name=host,proto3" json:"host,omitempty"` + Port int32 `protobuf:"varint,2,opt,name=port,proto3" json:"port,omitempty"` + // Optional. The number of milliseconds to wait before retrying failed Redis connection. + // By default, Feast uses exponential backoff policy and "initial_backoff_ms" sets the initial wait duration. + InitialBackoffMs int32 `protobuf:"varint,3,opt,name=initial_backoff_ms,json=initialBackoffMs,proto3" json:"initial_backoff_ms,omitempty"` + // Optional. Maximum total number of retries for connecting to Redis. Default to zero retries. + MaxRetries int32 `protobuf:"varint,4,opt,name=max_retries,json=maxRetries,proto3" json:"max_retries,omitempty"` XXX_NoUnkeyedLiteral struct{} `json:"-"` XXX_unrecognized []byte `json:"-"` XXX_sizecache int32 `json:"-"` @@ -296,6 +301,20 @@ func (m *Store_RedisConfig) GetPort() int32 { return 0 } +func (m *Store_RedisConfig) GetInitialBackoffMs() int32 { + if m != nil { + return m.InitialBackoffMs + } + return 0 +} + +func (m *Store_RedisConfig) GetMaxRetries() int32 { + if m != nil { + return m.MaxRetries + } + return 0 +} + type Store_BigQueryConfig struct { ProjectId string `protobuf:"bytes,1,opt,name=project_id,json=projectId,proto3" json:"project_id,omitempty"` DatasetId string `protobuf:"bytes,2,opt,name=dataset_id,json=datasetId,proto3" json:"dataset_id,omitempty"` @@ -476,34 +495,37 @@ func init() { func init() { proto.RegisterFile("feast/core/Store.proto", fileDescriptor_4b177bc9ccf64875) } var fileDescriptor_4b177bc9ccf64875 = []byte{ - // 450 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x8c, 0x93, 0x4f, 0x6f, 0xd3, 0x30, - 0x18, 0xc6, 0x97, 0xfe, 0x59, 0x97, 0x37, 0xfd, 0x13, 0xf9, 0x80, 0xa2, 0xa2, 0xa1, 0xb0, 0x53, - 0x4f, 0xb1, 0x54, 0xc4, 0x81, 0x1b, 0x4d, 0x3b, 0x41, 0x04, 0xaa, 0x98, 0x0b, 0x93, 0xe0, 0x32, - 0xa5, 0x89, 0x97, 0x79, 0xd3, 0xe2, 0x60, 0xbb, 0x48, 0xfd, 0xa8, 0x7c, 0x1b, 0x64, 0x27, 0x69, - 0x53, 0xda, 0xc3, 0x2e, 0x91, 0xfd, 0xbc, 0xcf, 0xf3, 0xcb, 0x2b, 0xdb, 0x2f, 0xbc, 0xba, 0xa7, - 0xb1, 0x54, 0x38, 0xe1, 0x82, 0xe2, 0x95, 0xe2, 0x82, 0x06, 0x85, 0xe0, 0x8a, 0x23, 0x30, 0x7a, - 0xa0, 0xf5, 0xab, 0xbf, 0x5d, 0xe8, 0x9a, 0x1a, 0x42, 0xd0, 0xc9, 0xe3, 0x67, 0xea, 0x59, 0xbe, - 0x35, 0xb1, 0x89, 0x59, 0x23, 0x0c, 0x1d, 0xb5, 0x2d, 0xa8, 0xd7, 0xf2, 0xad, 0xc9, 0x70, 0xfa, - 0x3a, 0xd8, 0x07, 0x83, 0x12, 0x68, 0xbe, 0xdf, 0xb7, 0x05, 0x25, 0xc6, 0x88, 0x16, 0x30, 0x90, - 0x9b, 0xb5, 0x4c, 0x04, 0x2b, 0x14, 0xe3, 0xb9, 0xf4, 0x3a, 0x7e, 0x7b, 0xe2, 0x4c, 0xdf, 0x9c, - 0x48, 0x36, 0x6c, 0xe4, 0x30, 0x84, 0x42, 0xe8, 0x0b, 0x9a, 0x32, 0x79, 0x97, 0xf0, 0xfc, 0x9e, - 0x65, 0x9e, 0xe3, 0x5b, 0x13, 0x67, 0x7a, 0x79, 0x0c, 0x21, 0xda, 0x35, 0x37, 0xa6, 0xcf, 0x67, - 0xc4, 0x11, 0xfb, 0x2d, 0xfa, 0x02, 0xa3, 0x35, 0xcb, 0x7e, 0x6f, 0xa8, 0xd8, 0xd6, 0x98, 0xbe, - 0xc1, 0xf8, 0xc7, 0x98, 0x90, 0x65, 0x37, 0xda, 0xb8, 0x23, 0x0d, 0xeb, 0x68, 0x05, 0x5b, 0x82, - 0x9b, 0xc4, 0x52, 0xc6, 0x79, 0x2a, 0xe2, 0x9a, 0x36, 0x30, 0xb4, 0xb7, 0xc7, 0xb4, 0x79, 0xed, - 0xdc, 0xe1, 0x46, 0xc9, 0xa1, 0x34, 0x7e, 0x0f, 0x4e, 0xa3, 0x75, 0x7d, 0xf4, 0x0f, 0x5c, 0xaa, - 0xfa, 0xe8, 0xf5, 0x5a, 0x6b, 0x05, 0x17, 0xca, 0x1c, 0x7d, 0x97, 0x98, 0xf5, 0x78, 0x09, 0xc3, - 0xc3, 0x56, 0xd1, 0x25, 0x40, 0x21, 0xf8, 0x23, 0x4d, 0xd4, 0x1d, 0x4b, 0xab, 0xbc, 0x5d, 0x29, - 0x51, 0xaa, 0xcb, 0x69, 0xac, 0x62, 0x49, 0x4d, 0xb9, 0x55, 0x96, 0x2b, 0x25, 0x4a, 0xc7, 0x1f, - 0x60, 0xf4, 0x5f, 0xb3, 0x2f, 0x6e, 0xe5, 0x16, 0xfa, 0xcd, 0x1b, 0x44, 0x1e, 0xf4, 0xaa, 0xdf, - 0x7a, 0x6d, 0x13, 0xad, 0xb7, 0x27, 0xdf, 0x95, 0x07, 0xbd, 0x3f, 0x54, 0x48, 0xc6, 0xf3, 0xaa, - 0xa9, 0x7a, 0x7b, 0xf5, 0x11, 0xec, 0xdd, 0x9b, 0x42, 0x0e, 0xf4, 0xa2, 0xe5, 0xed, 0xec, 0x6b, - 0xb4, 0x70, 0xcf, 0x90, 0x0d, 0x5d, 0x72, 0xbd, 0x88, 0x56, 0xae, 0x85, 0xfa, 0x70, 0x11, 0x46, - 0x9f, 0x6e, 0x7e, 0x5c, 0x93, 0x9f, 0x6e, 0x0b, 0x0d, 0xc0, 0x9e, 0xcf, 0x56, 0xab, 0xd9, 0x72, - 0x41, 0x66, 0x6e, 0x3b, 0xbc, 0x80, 0xf3, 0xf2, 0x86, 0xc2, 0x08, 0x1a, 0x2f, 0x3d, 0x04, 0xc3, - 0xfd, 0xa6, 0x27, 0xe0, 0x17, 0xce, 0x98, 0x7a, 0xd8, 0xac, 0x83, 0x84, 0x3f, 0xe3, 0x8c, 0x3f, - 0xd2, 0x27, 0x5c, 0x8e, 0x8a, 0x4c, 0x9f, 0x70, 0xc6, 0xb1, 0x19, 0x13, 0x89, 0xf7, 0xe3, 0xb3, - 0x3e, 0x37, 0xd2, 0xbb, 0x7f, 0x01, 0x00, 0x00, 0xff, 0xff, 0x56, 0xfe, 0x58, 0x14, 0x53, 0x03, - 0x00, 0x00, + // 500 bytes of a gzipped FileDescriptorProto + 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x8c, 0x93, 0x4d, 0x8f, 0xd3, 0x3c, + 0x10, 0xc7, 0x37, 0x7d, 0xdd, 0x4c, 0xfa, 0x12, 0xf9, 0xf0, 0x28, 0xea, 0xa3, 0x85, 0xb2, 0xa7, + 0x1e, 0x50, 0x23, 0x95, 0x13, 0x37, 0x9a, 0x76, 0x05, 0x11, 0x50, 0xb1, 0x2e, 0xac, 0x04, 0x97, + 0xca, 0x4d, 0xdc, 0xac, 0xb7, 0x34, 0x0e, 0xb6, 0x8b, 0xb6, 0x77, 0xbe, 0x0c, 0xdf, 0x12, 0xd9, + 0x49, 0xfa, 0x42, 0xf7, 0xc0, 0x25, 0xb2, 0xff, 0xf3, 0x9f, 0x5f, 0x3c, 0xf6, 0x0c, 0xfc, 0xb7, + 0xa2, 0x44, 0x2a, 0x3f, 0xe2, 0x82, 0xfa, 0x73, 0xc5, 0x05, 0x1d, 0x66, 0x82, 0x2b, 0x8e, 0xc0, + 0xe8, 0x43, 0xad, 0x5f, 0xff, 0x6e, 0x40, 0xdd, 0xc4, 0x10, 0x82, 0x5a, 0x4a, 0x36, 0xd4, 0xb3, + 0xfa, 0xd6, 0xc0, 0xc6, 0x66, 0x8d, 0x7c, 0xa8, 0xa9, 0x5d, 0x46, 0xbd, 0x4a, 0xdf, 0x1a, 0x74, + 0x46, 0xff, 0x0f, 0x0f, 0x89, 0xc3, 0x1c, 0x68, 0xbe, 0x9f, 0x77, 0x19, 0xc5, 0xc6, 0x88, 0xa6, + 0xd0, 0x96, 0xdb, 0xa5, 0x8c, 0x04, 0xcb, 0x14, 0xe3, 0xa9, 0xf4, 0x6a, 0xfd, 0xea, 0xc0, 0x19, + 0x3d, 0x7b, 0x22, 0xf3, 0xc8, 0x86, 0x4f, 0x93, 0x50, 0x00, 0x2d, 0x41, 0x63, 0x26, 0x17, 0x11, + 0x4f, 0x57, 0x2c, 0xf1, 0x9c, 0xbe, 0x35, 0x70, 0x46, 0x57, 0xe7, 0x10, 0xac, 0x5d, 0x13, 0x63, + 0x7a, 0x77, 0x81, 0x1d, 0x71, 0xd8, 0xa2, 0xf7, 0xd0, 0x5d, 0xb2, 0xe4, 0xc7, 0x96, 0x8a, 0x5d, + 0x89, 0x69, 0x19, 0x4c, 0xff, 0x1c, 0x13, 0xb0, 0xe4, 0x56, 0x1b, 0xf7, 0xa4, 0x4e, 0x99, 0x5a, + 0xc0, 0x66, 0xe0, 0x46, 0x44, 0x4a, 0x92, 0xc6, 0x82, 0x94, 0xb4, 0xb6, 0xa1, 0xbd, 0x38, 0xa7, + 0x4d, 0x4a, 0xe7, 0x1e, 0xd7, 0x8d, 0x4e, 0xa5, 0xde, 0x2f, 0x0b, 0x9c, 0xa3, 0xb3, 0xeb, 0xbb, + 0xbf, 0xe7, 0x52, 0x95, 0x77, 0xaf, 0xd7, 0x5a, 0xcb, 0xb8, 0x50, 0xe6, 0xee, 0xeb, 0xd8, 0xac, + 0xd1, 0x4b, 0x40, 0x2c, 0x65, 0x8a, 0x91, 0xef, 0x8b, 0x25, 0x89, 0xd6, 0x7c, 0xb5, 0x5a, 0x6c, + 0xa4, 0x57, 0x35, 0x0e, 0xb7, 0x88, 0x04, 0x79, 0xe0, 0xa3, 0x44, 0xcf, 0xc1, 0xd9, 0x90, 0xc7, + 0x85, 0xa0, 0x4a, 0x30, 0xaa, 0x9f, 0x42, 0xdb, 0x60, 0x43, 0x1e, 0x71, 0xae, 0xf4, 0x66, 0xd0, + 0x39, 0x2d, 0x1d, 0x5d, 0x01, 0x64, 0x82, 0x3f, 0xd0, 0x48, 0x2d, 0x58, 0x5c, 0x1c, 0xc7, 0x2e, + 0x94, 0x30, 0xd6, 0xe1, 0x98, 0x28, 0x22, 0xa9, 0x09, 0x57, 0xf2, 0x70, 0xa1, 0x84, 0x71, 0xef, + 0x35, 0x74, 0xff, 0x2a, 0xfe, 0x5f, 0x2b, 0xeb, 0xdd, 0x41, 0xeb, 0xb8, 0x23, 0x90, 0x07, 0xcd, + 0xe2, 0xb7, 0xa6, 0x3c, 0x1b, 0x97, 0xdb, 0x27, 0xfb, 0xd4, 0x83, 0xe6, 0x4f, 0x2a, 0x24, 0xe3, + 0x69, 0x71, 0xa8, 0x72, 0x7b, 0xfd, 0x06, 0xec, 0x7d, 0x8f, 0x22, 0x07, 0x9a, 0xe1, 0xec, 0x6e, + 0xfc, 0x21, 0x9c, 0xba, 0x17, 0xc8, 0x86, 0x3a, 0xbe, 0x99, 0x86, 0x73, 0xd7, 0x42, 0x2d, 0xb8, + 0x0c, 0xc2, 0xb7, 0xb7, 0x5f, 0x6e, 0xf0, 0x57, 0xb7, 0x82, 0xda, 0x60, 0x4f, 0xc6, 0xf3, 0xf9, + 0x78, 0x36, 0xc5, 0x63, 0xb7, 0x1a, 0x5c, 0x42, 0x23, 0x7f, 0xf1, 0x20, 0x84, 0xa3, 0xc9, 0x09, + 0xc0, 0x70, 0x3f, 0xe9, 0x89, 0xfa, 0xe6, 0x27, 0x4c, 0xdd, 0x6f, 0x97, 0xc3, 0x88, 0x6f, 0xfc, + 0x84, 0x3f, 0xd0, 0xb5, 0x9f, 0x8f, 0x9e, 0x8c, 0xd7, 0x7e, 0xc2, 0x7d, 0x33, 0x76, 0xd2, 0x3f, + 0x8c, 0xe3, 0xb2, 0x61, 0xa4, 0x57, 0x7f, 0x02, 0x00, 0x00, 0xff, 0xff, 0x19, 0x4f, 0x37, 0x0a, + 0xa3, 0x03, 0x00, 0x00, } diff --git a/sdk/go/protos/feast/storage/Redis.pb.go b/sdk/go/protos/feast/storage/Redis.pb.go index 55cf42becd8..f5ce0558b55 100644 --- a/sdk/go/protos/feast/storage/Redis.pb.go +++ b/sdk/go/protos/feast/storage/Redis.pb.go @@ -25,7 +25,8 @@ type RedisKey struct { // FeatureSet this row belongs to, this is defined as featureSetName:version. FeatureSet string `protobuf:"bytes,2,opt,name=feature_set,json=featureSet,proto3" json:"feature_set,omitempty"` // List of fields containing entity names and their respective values - // contained within this feature row. + // contained within this feature row. The entities should be sorted + // by the entity name alphabetically in ascending order. Entities []*types.Field `protobuf:"bytes,3,rep,name=entities,proto3" json:"entities,omitempty"` XXX_NoUnkeyedLiteral struct{} `json:"-"` XXX_unrecognized []byte `json:"-"` diff --git a/sdk/go/protos/feast/types/FeatureRow.pb.go b/sdk/go/protos/feast/types/FeatureRow.pb.go index dd9aa6b8c96..75ad87f8d3f 100644 --- a/sdk/go/protos/feast/types/FeatureRow.pb.go +++ b/sdk/go/protos/feast/types/FeatureRow.pb.go @@ -29,7 +29,7 @@ type FeatureRow struct { // will use to perform joins, determine latest values, and coalesce rows. EventTimestamp *timestamp.Timestamp `protobuf:"bytes,3,opt,name=event_timestamp,json=eventTimestamp,proto3" json:"event_timestamp,omitempty"` // Complete reference to the featureSet this featureRow belongs to, in the form of - // featureSetName:version. This value will be used by the feast ingestion job to filter + // /:. This value will be used by the feast ingestion job to filter // rows, and write the values to the correct tables. FeatureSet string `protobuf:"bytes,6,opt,name=feature_set,json=featureSet,proto3" json:"feature_set,omitempty"` XXX_NoUnkeyedLiteral struct{} `json:"-"` diff --git a/sdk/go/request.go b/sdk/go/request.go index 9e97dffb72f..a3a3fe6d71a 100644 --- a/sdk/go/request.go +++ b/sdk/go/request.go @@ -8,7 +8,8 @@ import ( ) var ( - ErrInvalidFeatureName = "invalid feature ids %s provided, feature names must be in the format /:" + // ErrInvalidFeatureName indicates that the user has provided a feature reference with the wrong structure or contents + ErrInvalidFeatureName = "invalid feature references %s provided, feature names must be in the format /:" ) // OnlineFeaturesRequest wrapper on feast.serving.GetOnlineFeaturesRequest. @@ -85,7 +86,6 @@ func buildFeatures(featureReferences []string, defaultProject string) ([]*servin return nil, fmt.Errorf(ErrInvalidFeatureName, featureRef) } - if project == "" || name == "" || version < 0 { return nil, fmt.Errorf(ErrInvalidFeatureName, featureRef) } diff --git a/sdk/go/request_test.go b/sdk/go/request_test.go index 2e403f0bd3e..583a1c8b94f 100644 --- a/sdk/go/request_test.go +++ b/sdk/go/request_test.go @@ -95,8 +95,7 @@ func TestGetOnlineFeaturesRequest(t *testing.T) { Version: 0, }, }, - EntityRows: []*serving.GetOnlineFeaturesRequest_EntityRow{ - }, + EntityRows: []*serving.GetOnlineFeaturesRequest_EntityRow{}, OmitEntitiesInResponse: false, }, wantErr: false, @@ -116,7 +115,7 @@ func TestGetOnlineFeaturesRequest(t *testing.T) { req: OnlineFeaturesRequest{ Features: []string{"fs1:3:feature1"}, Entities: []Row{}, - Project: "my_project", + Project: "my_project", }, wantErr: true, err: fmt.Errorf(ErrInvalidFeatureName, "fs1:3:feature1"), @@ -147,9 +146,9 @@ func TestGetOnlineFeaturesRequest(t *testing.T) { if !cmp.Equal(got, tc.want) { m := json.Marshaler{} - gotJson, _ := m.MarshalToString(got) - wantJson, _ := m.MarshalToString(tc.want) - t.Errorf("got: \n%v\nwant:\n%v", gotJson, wantJson) + gotJSON, _ := m.MarshalToString(got) + wantJSON, _ := m.MarshalToString(tc.want) + t.Errorf("got: \n%v\nwant:\n%v", gotJSON, wantJSON) } }) } diff --git a/sdk/go/response.go b/sdk/go/response.go index 086321cacdf..e6e59268ea0 100644 --- a/sdk/go/response.go +++ b/sdk/go/response.go @@ -7,8 +7,13 @@ import ( ) var ( + // ErrLengthMismatch indicates that the number of values returned is not the same as the number of values requested ErrLengthMismatch = "Length mismatch; number of na values (%d) not equal to number of features requested (%d)." + + // ErrFeatureNotFound indicates that the a requested feature was not found in the response ErrFeatureNotFound = "Feature %s not found in response." + + // ErrTypeMismatch indicates that the there was a type mismatch in the returned values ErrTypeMismatch = "Requested output of type %s does not match type of feature value returned." ) @@ -20,7 +25,7 @@ type OnlineFeaturesResponse struct { // Rows retrieves the result of the request as a list of Rows. func (r OnlineFeaturesResponse) Rows() []Row { rows := make([]Row, len(r.RawResponse.FieldValues)) - for i, val := range r.RawResponse.FieldValues { + for i, val := range r.RawResponse.FieldValues { rows[i] = val.Fields } return rows @@ -33,7 +38,7 @@ func (r OnlineFeaturesResponse) Int64Arrays(order []string, fillNa []int64) ([][ if len(fillNa) != len(order) { return nil, fmt.Errorf(ErrLengthMismatch, len(fillNa), len(order)) } - for i, val := range r.RawResponse.FieldValues { + for i, val := range r.RawResponse.FieldValues { rows[i] = make([]int64, len(order)) for j, fname := range order { fValue, exists := val.Fields[fname] @@ -60,7 +65,7 @@ func (r OnlineFeaturesResponse) Float64Arrays(order []string, fillNa []float64) if len(fillNa) != len(order) { return nil, fmt.Errorf(ErrLengthMismatch, len(fillNa), len(order)) } - for i, val := range r.RawResponse.FieldValues { + for i, val := range r.RawResponse.FieldValues { rows[i] = make([]float64, len(order)) for j, fname := range order { fValue, exists := val.Fields[fname] diff --git a/sdk/go/response_test.go b/sdk/go/response_test.go index 882c1695d59..87bf275da9c 100644 --- a/sdk/go/response_test.go +++ b/sdk/go/response_test.go @@ -9,22 +9,22 @@ import ( ) var response = OnlineFeaturesResponse{ -RawResponse: &serving.GetOnlineFeaturesResponse{ - FieldValues: []*serving.GetOnlineFeaturesResponse_FieldValues{ - { - Fields: map[string]*types.Value{ - "project1/feature1": Int64Val(1), - "project1/feature2": &types.Value{}, + RawResponse: &serving.GetOnlineFeaturesResponse{ + FieldValues: []*serving.GetOnlineFeaturesResponse_FieldValues{ + { + Fields: map[string]*types.Value{ + "project1/feature1": Int64Val(1), + "project1/feature2": {}, + }, }, - }, - { - Fields: map[string]*types.Value{ - "project1/feature1": Int64Val(2), - "project1/feature2": Int64Val(2), + { + Fields: map[string]*types.Value{ + "project1/feature1": Int64Val(2), + "project1/feature2": Int64Val(2), + }, }, }, }, -}, } func TestOnlineFeaturesResponseToRow(t *testing.T) { @@ -40,7 +40,7 @@ func TestOnlineFeaturesResponseToRow(t *testing.T) { func TestOnlineFeaturesResponseToInt64Array(t *testing.T) { type args struct { - order []string + order []string fillNa []int64 } tt := []struct { @@ -53,31 +53,31 @@ func TestOnlineFeaturesResponseToInt64Array(t *testing.T) { { name: "valid", args: args{ - order: []string{"project1/feature2", "project1/feature1" }, + order: []string{"project1/feature2", "project1/feature1"}, fillNa: []int64{-1, -1}, }, - want: [][]int64{{-1, 1}, {2, 2}}, + want: [][]int64{{-1, 1}, {2, 2}}, wantErr: false, }, { name: "length mismatch", args: args{ - order: []string{"fs:1:feature2", "fs:1:feature1"}, + order: []string{"fs:1:feature2", "fs:1:feature1"}, fillNa: []int64{-1}, }, - want: nil, + want: nil, wantErr: true, - err: fmt.Errorf(ErrLengthMismatch, 1, 2), + err: fmt.Errorf(ErrLengthMismatch, 1, 2), }, { name: "length mismatch", args: args{ - order: []string{"project1/feature2", "project1/feature3" }, + order: []string{"project1/feature2", "project1/feature3"}, fillNa: []int64{-1, -1}, }, - want: nil, + want: nil, wantErr: true, - err: fmt.Errorf(ErrFeatureNotFound, "project1/feature3"), + err: fmt.Errorf(ErrFeatureNotFound, "project1/feature3"), }, } for _, tc := range tt { @@ -96,4 +96,4 @@ func TestOnlineFeaturesResponseToInt64Array(t *testing.T) { } }) } -} \ No newline at end of file +} diff --git a/sdk/go/types.go b/sdk/go/types.go index 74606982a4a..92858bd384c 100644 --- a/sdk/go/types.go +++ b/sdk/go/types.go @@ -2,6 +2,7 @@ package feast import "github.com/gojek/feast/sdk/go/protos/feast/types" +// Row map of entity values type Row map[string]*types.Value // StrVal is a int64 type feast value @@ -37,4 +38,4 @@ func BoolVal(val bool) *types.Value { // BytesVal is a bytes type feast value func BytesVal(val []byte) *types.Value { return &types.Value{Val: &types.Value_BytesVal{BytesVal: val}} -} \ No newline at end of file +} diff --git a/sdk/python/Makefile b/sdk/python/Makefile deleted file mode 100644 index f940812ef98..00000000000 --- a/sdk/python/Makefile +++ /dev/null @@ -1,20 +0,0 @@ -# -# Copyright 2019 The Feast Authors -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# https://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. -# - -TEST_PATH=./ - -test: - pytest --verbose --color=yes $(TEST_PATH) diff --git a/sdk/python/docs/conf.py b/sdk/python/docs/conf.py index 999c191ee54..8a72b7456b5 100644 --- a/sdk/python/docs/conf.py +++ b/sdk/python/docs/conf.py @@ -18,8 +18,8 @@ # import os import sys -import sphinx_rtd_theme +import sphinx_rtd_theme sys.path.insert(0, os.path.abspath("../../feast")) sys.path.insert(0, os.path.abspath("../..")) diff --git a/sdk/python/feast/__init__.py b/sdk/python/feast/__init__.py index adcac0cd248..e69de29bb2d 100644 --- a/sdk/python/feast/__init__.py +++ b/sdk/python/feast/__init__.py @@ -1,14 +0,0 @@ -from pkg_resources import get_distribution, DistributionNotFound - -try: - __version__ = get_distribution(__name__).version -except DistributionNotFound: - # package is not installed - pass - -from .client import Client -from .entity import Entity -from .feature_set import FeatureSet -from .feature import Feature -from .source import Source, KafkaSource -from .value_type import ValueType diff --git a/sdk/python/feast/cli.py b/sdk/python/feast/cli.py index 8e8f185d038..27b98b1086e 100644 --- a/sdk/python/feast/cli.py +++ b/sdk/python/feast/cli.py @@ -12,17 +12,19 @@ # See the License for the specific language governing permissions and # limitations under the License. -import sys +import json import logging +import sys + import click +import pkg_resources +import toml +import yaml + from feast import config as feast_config from feast.client import Client from feast.feature_set import FeatureSet -import toml -import pkg_resources from feast.loaders.yaml import yaml_loader -import yaml -import json _logger = logging.getLogger(__name__) @@ -221,7 +223,7 @@ def project_archive(name: str): @project.command(name="list") -def feature_set_list(): +def project_list(): """ List all projects """ diff --git a/sdk/python/feast/client.py b/sdk/python/feast/client.py index 543f0afeb64..ffdb71743d0 100644 --- a/sdk/python/feast/client.py +++ b/sdk/python/feast/client.py @@ -12,7 +12,6 @@ # See the License for the specific language governing permissions and # limitations under the License. -import json import logging import os import shutil @@ -20,48 +19,45 @@ import time from collections import OrderedDict from math import ceil -from typing import Dict, List, Tuple, Union, Optional -from urllib.parse import urlparse +from typing import Dict, List, Optional, Tuple, Union -import fastavro import grpc import pandas as pd import pyarrow as pa import pyarrow.parquet as pq from feast.core.CoreService_pb2 import ( - GetFeastCoreVersionRequest, - ListFeatureSetsResponse, ApplyFeatureSetRequest, - ListFeatureSetsRequest, ApplyFeatureSetResponse, - GetFeatureSetRequest, - GetFeatureSetResponse, - CreateProjectRequest, - CreateProjectResponse, ArchiveProjectRequest, ArchiveProjectResponse, + CreateProjectRequest, + CreateProjectResponse, + GetFeastCoreVersionRequest, + GetFeatureSetRequest, + GetFeatureSetResponse, + ListFeatureSetsRequest, + ListFeatureSetsResponse, ListProjectsRequest, ListProjectsResponse, ) from feast.core.CoreService_pb2_grpc import CoreServiceStub from feast.core.FeatureSet_pb2 import FeatureSetStatus -from feast.feature_set import FeatureSet, Entity +from feast.feature_set import Entity, FeatureSet from feast.job import Job from feast.loaders.abstract_producer import get_producer from feast.loaders.file import export_source_to_staging_location -from feast.loaders.ingest import KAFKA_CHUNK_PRODUCTION_TIMEOUT -from feast.loaders.ingest import get_feature_row_chunks -from feast.serving.ServingService_pb2 import FeatureReference -from feast.serving.ServingService_pb2 import GetFeastServingInfoResponse +from feast.loaders.ingest import KAFKA_CHUNK_PRODUCTION_TIMEOUT, get_feature_row_chunks from feast.serving.ServingService_pb2 import ( - GetOnlineFeaturesRequest, + DataFormat, + DatasetSource, + FeastServingType, + FeatureReference, GetBatchFeaturesRequest, GetFeastServingInfoRequest, + GetFeastServingInfoResponse, + GetOnlineFeaturesRequest, GetOnlineFeaturesResponse, - DatasetSource, - DataFormat, - FeastServingType, ) from feast.serving.ServingService_pb2_grpc import ServingServiceStub @@ -84,8 +80,12 @@ class Client: """ def __init__( - self, core_url: str = None, serving_url: str = None, project: str = None, - core_secure: bool = None, serving_secure: bool = None + self, + core_url: str = None, + serving_url: str = None, + project: str = None, + core_secure: bool = None, + serving_secure: bool = None, ): """ The Feast Client should be initialized with at least one service url @@ -167,7 +167,7 @@ def core_secure(self) -> bool: if self._core_secure is not None: return self._core_secure - return os.getenv(FEAST_CORE_SECURE_ENV_KEY, "").lower() is "true" + return os.getenv(FEAST_CORE_SECURE_ENV_KEY, "").lower() == "true" @core_secure.setter def core_secure(self, value: bool): @@ -190,7 +190,7 @@ def serving_secure(self) -> bool: if self._serving_secure is not None: return self._serving_secure - return os.getenv(FEAST_SERVING_SECURE_ENV_KEY, "").lower() is "true" + return os.getenv(FEAST_SERVING_SECURE_ENV_KEY, "").lower() == "true" @serving_secure.setter def serving_secure(self, value: bool): @@ -239,7 +239,9 @@ def _connect_core(self, skip_if_connected: bool = True): if self.__core_channel is None: if self.core_secure or self.core_url.endswith(":443"): - self.__core_channel = grpc.secure_channel(self.core_url, grpc.ssl_channel_credentials()) + self.__core_channel = grpc.secure_channel( + self.core_url, grpc.ssl_channel_credentials() + ) else: self.__core_channel = grpc.insecure_channel(self.core_url) @@ -271,7 +273,9 @@ def _connect_serving(self, skip_if_connected=True): if self.__serving_channel is None: if self.serving_secure or self.serving_url.endswith(":443"): - self.__serving_channel = grpc.secure_channel(self.serving_url, grpc.ssl_channel_credentials()) + self.__serving_channel = grpc.secure_channel( + self.serving_url, grpc.ssl_channel_credentials() + ) else: self.__serving_channel = grpc.insecure_channel(self.serving_url) @@ -657,7 +661,7 @@ def get_online_features( ), entity_rows=entity_rows, ) - ) # type: GetOnlineFeaturesResponse + ) def ingest( self, diff --git a/sdk/python/feast/config.py b/sdk/python/feast/config.py index 130b76cc86a..061bf24c3b7 100644 --- a/sdk/python/feast/config.py +++ b/sdk/python/feast/config.py @@ -14,13 +14,12 @@ # limitations under the License. # -from os.path import expanduser, join import logging import os import sys +from os.path import expanduser, join from typing import Dict -from urllib.parse import urlparse -from urllib.parse import ParseResult +from urllib.parse import ParseResult, urlparse import toml diff --git a/sdk/python/feast/core/CoreService_pb2.py b/sdk/python/feast/core/CoreService_pb2.py deleted file mode 100644 index 858703d7f3e..00000000000 --- a/sdk/python/feast/core/CoreService_pb2.py +++ /dev/null @@ -1,988 +0,0 @@ -# -*- coding: utf-8 -*- -# Generated by the protocol buffer compiler. DO NOT EDIT! -# source: feast/core/CoreService.proto - -import sys -_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1')) -from google.protobuf import descriptor as _descriptor -from google.protobuf import message as _message -from google.protobuf import reflection as _reflection -from google.protobuf import symbol_database as _symbol_database -# @@protoc_insertion_point(imports) - -_sym_db = _symbol_database.Default() - - -from feast.core import FeatureSet_pb2 as feast_dot_core_dot_FeatureSet__pb2 -from feast.core import Store_pb2 as feast_dot_core_dot_Store__pb2 - - -DESCRIPTOR = _descriptor.FileDescriptor( - name='feast/core/CoreService.proto', - package='feast.core', - syntax='proto3', - serialized_options=_b('\n\nfeast.coreB\020CoreServiceProtoZ/github.com/gojek/feast/sdk/go/protos/feast/core'), - serialized_pb=_b('\n\x1c\x66\x65\x61st/core/CoreService.proto\x12\nfeast.core\x1a\x1b\x66\x65\x61st/core/FeatureSet.proto\x1a\x16\x66\x65\x61st/core/Store.proto\"F\n\x14GetFeatureSetRequest\x12\x0f\n\x07project\x18\x03 \x01(\t\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0f\n\x07version\x18\x02 \x01(\x05\"D\n\x15GetFeatureSetResponse\x12+\n\x0b\x66\x65\x61ture_set\x18\x01 \x01(\x0b\x32\x16.feast.core.FeatureSet\"\xa5\x01\n\x16ListFeatureSetsRequest\x12\x39\n\x06\x66ilter\x18\x01 \x01(\x0b\x32).feast.core.ListFeatureSetsRequest.Filter\x1aP\n\x06\x46ilter\x12\x0f\n\x07project\x18\x03 \x01(\t\x12\x18\n\x10\x66\x65\x61ture_set_name\x18\x01 \x01(\t\x12\x1b\n\x13\x66\x65\x61ture_set_version\x18\x02 \x01(\t\"G\n\x17ListFeatureSetsResponse\x12,\n\x0c\x66\x65\x61ture_sets\x18\x01 \x03(\x0b\x32\x16.feast.core.FeatureSet\"a\n\x11ListStoresRequest\x12\x34\n\x06\x66ilter\x18\x01 \x01(\x0b\x32$.feast.core.ListStoresRequest.Filter\x1a\x16\n\x06\x46ilter\x12\x0c\n\x04name\x18\x01 \x01(\t\"6\n\x12ListStoresResponse\x12 \n\x05store\x18\x01 \x03(\x0b\x32\x11.feast.core.Store\"E\n\x16\x41pplyFeatureSetRequest\x12+\n\x0b\x66\x65\x61ture_set\x18\x01 \x01(\x0b\x32\x16.feast.core.FeatureSet\"\xb3\x01\n\x17\x41pplyFeatureSetResponse\x12+\n\x0b\x66\x65\x61ture_set\x18\x01 \x01(\x0b\x32\x16.feast.core.FeatureSet\x12:\n\x06status\x18\x02 \x01(\x0e\x32*.feast.core.ApplyFeatureSetResponse.Status\"/\n\x06Status\x12\r\n\tNO_CHANGE\x10\x00\x12\x0b\n\x07\x43REATED\x10\x01\x12\t\n\x05\x45RROR\x10\x02\"\x1c\n\x1aGetFeastCoreVersionRequest\".\n\x1bGetFeastCoreVersionResponse\x12\x0f\n\x07version\x18\x01 \x01(\t\"6\n\x12UpdateStoreRequest\x12 \n\x05store\x18\x01 \x01(\x0b\x32\x11.feast.core.Store\"\x95\x01\n\x13UpdateStoreResponse\x12 \n\x05store\x18\x01 \x01(\x0b\x32\x11.feast.core.Store\x12\x36\n\x06status\x18\x02 \x01(\x0e\x32&.feast.core.UpdateStoreResponse.Status\"$\n\x06Status\x12\r\n\tNO_CHANGE\x10\x00\x12\x0b\n\x07UPDATED\x10\x01\"$\n\x14\x43reateProjectRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\"\x17\n\x15\x43reateProjectResponse\"%\n\x15\x41rchiveProjectRequest\x12\x0c\n\x04name\x18\x01 \x01(\t\"\x18\n\x16\x41rchiveProjectResponse\"\x15\n\x13ListProjectsRequest\"(\n\x14ListProjectsResponse\x12\x10\n\x08projects\x18\x01 \x03(\t2\xa2\x06\n\x0b\x43oreService\x12\x66\n\x13GetFeastCoreVersion\x12&.feast.core.GetFeastCoreVersionRequest\x1a\'.feast.core.GetFeastCoreVersionResponse\x12T\n\rGetFeatureSet\x12 .feast.core.GetFeatureSetRequest\x1a!.feast.core.GetFeatureSetResponse\x12Z\n\x0fListFeatureSets\x12\".feast.core.ListFeatureSetsRequest\x1a#.feast.core.ListFeatureSetsResponse\x12K\n\nListStores\x12\x1d.feast.core.ListStoresRequest\x1a\x1e.feast.core.ListStoresResponse\x12Z\n\x0f\x41pplyFeatureSet\x12\".feast.core.ApplyFeatureSetRequest\x1a#.feast.core.ApplyFeatureSetResponse\x12N\n\x0bUpdateStore\x12\x1e.feast.core.UpdateStoreRequest\x1a\x1f.feast.core.UpdateStoreResponse\x12T\n\rCreateProject\x12 .feast.core.CreateProjectRequest\x1a!.feast.core.CreateProjectResponse\x12W\n\x0e\x41rchiveProject\x12!.feast.core.ArchiveProjectRequest\x1a\".feast.core.ArchiveProjectResponse\x12Q\n\x0cListProjects\x12\x1f.feast.core.ListProjectsRequest\x1a .feast.core.ListProjectsResponseBO\n\nfeast.coreB\x10\x43oreServiceProtoZ/github.com/gojek/feast/sdk/go/protos/feast/coreb\x06proto3') - , - dependencies=[feast_dot_core_dot_FeatureSet__pb2.DESCRIPTOR,feast_dot_core_dot_Store__pb2.DESCRIPTOR,]) - - - -_APPLYFEATURESETRESPONSE_STATUS = _descriptor.EnumDescriptor( - name='Status', - full_name='feast.core.ApplyFeatureSetResponse.Status', - filename=None, - file=DESCRIPTOR, - values=[ - _descriptor.EnumValueDescriptor( - name='NO_CHANGE', index=0, number=0, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='CREATED', index=1, number=1, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='ERROR', index=2, number=2, - serialized_options=None, - type=None), - ], - containing_type=None, - serialized_options=None, - serialized_start=839, - serialized_end=886, -) -_sym_db.RegisterEnumDescriptor(_APPLYFEATURESETRESPONSE_STATUS) - -_UPDATESTORERESPONSE_STATUS = _descriptor.EnumDescriptor( - name='Status', - full_name='feast.core.UpdateStoreResponse.Status', - filename=None, - file=DESCRIPTOR, - values=[ - _descriptor.EnumValueDescriptor( - name='NO_CHANGE', index=0, number=0, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='UPDATED', index=1, number=1, - serialized_options=None, - type=None), - ], - containing_type=None, - serialized_options=None, - serialized_start=1136, - serialized_end=1172, -) -_sym_db.RegisterEnumDescriptor(_UPDATESTORERESPONSE_STATUS) - - -_GETFEATURESETREQUEST = _descriptor.Descriptor( - name='GetFeatureSetRequest', - full_name='feast.core.GetFeatureSetRequest', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='project', full_name='feast.core.GetFeatureSetRequest.project', index=0, - number=3, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='name', full_name='feast.core.GetFeatureSetRequest.name', index=1, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='version', full_name='feast.core.GetFeatureSetRequest.version', index=2, - number=2, type=5, cpp_type=1, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=97, - serialized_end=167, -) - - -_GETFEATURESETRESPONSE = _descriptor.Descriptor( - name='GetFeatureSetResponse', - full_name='feast.core.GetFeatureSetResponse', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='feature_set', full_name='feast.core.GetFeatureSetResponse.feature_set', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=169, - serialized_end=237, -) - - -_LISTFEATURESETSREQUEST_FILTER = _descriptor.Descriptor( - name='Filter', - full_name='feast.core.ListFeatureSetsRequest.Filter', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='project', full_name='feast.core.ListFeatureSetsRequest.Filter.project', index=0, - number=3, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='feature_set_name', full_name='feast.core.ListFeatureSetsRequest.Filter.feature_set_name', index=1, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='feature_set_version', full_name='feast.core.ListFeatureSetsRequest.Filter.feature_set_version', index=2, - number=2, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=325, - serialized_end=405, -) - -_LISTFEATURESETSREQUEST = _descriptor.Descriptor( - name='ListFeatureSetsRequest', - full_name='feast.core.ListFeatureSetsRequest', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='filter', full_name='feast.core.ListFeatureSetsRequest.filter', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[_LISTFEATURESETSREQUEST_FILTER, ], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=240, - serialized_end=405, -) - - -_LISTFEATURESETSRESPONSE = _descriptor.Descriptor( - name='ListFeatureSetsResponse', - full_name='feast.core.ListFeatureSetsResponse', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='feature_sets', full_name='feast.core.ListFeatureSetsResponse.feature_sets', index=0, - number=1, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=407, - serialized_end=478, -) - - -_LISTSTORESREQUEST_FILTER = _descriptor.Descriptor( - name='Filter', - full_name='feast.core.ListStoresRequest.Filter', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='name', full_name='feast.core.ListStoresRequest.Filter.name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=555, - serialized_end=577, -) - -_LISTSTORESREQUEST = _descriptor.Descriptor( - name='ListStoresRequest', - full_name='feast.core.ListStoresRequest', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='filter', full_name='feast.core.ListStoresRequest.filter', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[_LISTSTORESREQUEST_FILTER, ], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=480, - serialized_end=577, -) - - -_LISTSTORESRESPONSE = _descriptor.Descriptor( - name='ListStoresResponse', - full_name='feast.core.ListStoresResponse', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='store', full_name='feast.core.ListStoresResponse.store', index=0, - number=1, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=579, - serialized_end=633, -) - - -_APPLYFEATURESETREQUEST = _descriptor.Descriptor( - name='ApplyFeatureSetRequest', - full_name='feast.core.ApplyFeatureSetRequest', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='feature_set', full_name='feast.core.ApplyFeatureSetRequest.feature_set', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=635, - serialized_end=704, -) - - -_APPLYFEATURESETRESPONSE = _descriptor.Descriptor( - name='ApplyFeatureSetResponse', - full_name='feast.core.ApplyFeatureSetResponse', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='feature_set', full_name='feast.core.ApplyFeatureSetResponse.feature_set', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='status', full_name='feast.core.ApplyFeatureSetResponse.status', index=1, - number=2, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - _APPLYFEATURESETRESPONSE_STATUS, - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=707, - serialized_end=886, -) - - -_GETFEASTCOREVERSIONREQUEST = _descriptor.Descriptor( - name='GetFeastCoreVersionRequest', - full_name='feast.core.GetFeastCoreVersionRequest', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=888, - serialized_end=916, -) - - -_GETFEASTCOREVERSIONRESPONSE = _descriptor.Descriptor( - name='GetFeastCoreVersionResponse', - full_name='feast.core.GetFeastCoreVersionResponse', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='version', full_name='feast.core.GetFeastCoreVersionResponse.version', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=918, - serialized_end=964, -) - - -_UPDATESTOREREQUEST = _descriptor.Descriptor( - name='UpdateStoreRequest', - full_name='feast.core.UpdateStoreRequest', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='store', full_name='feast.core.UpdateStoreRequest.store', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=966, - serialized_end=1020, -) - - -_UPDATESTORERESPONSE = _descriptor.Descriptor( - name='UpdateStoreResponse', - full_name='feast.core.UpdateStoreResponse', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='store', full_name='feast.core.UpdateStoreResponse.store', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='status', full_name='feast.core.UpdateStoreResponse.status', index=1, - number=2, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - _UPDATESTORERESPONSE_STATUS, - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=1023, - serialized_end=1172, -) - - -_CREATEPROJECTREQUEST = _descriptor.Descriptor( - name='CreateProjectRequest', - full_name='feast.core.CreateProjectRequest', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='name', full_name='feast.core.CreateProjectRequest.name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=1174, - serialized_end=1210, -) - - -_CREATEPROJECTRESPONSE = _descriptor.Descriptor( - name='CreateProjectResponse', - full_name='feast.core.CreateProjectResponse', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=1212, - serialized_end=1235, -) - - -_ARCHIVEPROJECTREQUEST = _descriptor.Descriptor( - name='ArchiveProjectRequest', - full_name='feast.core.ArchiveProjectRequest', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='name', full_name='feast.core.ArchiveProjectRequest.name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=1237, - serialized_end=1274, -) - - -_ARCHIVEPROJECTRESPONSE = _descriptor.Descriptor( - name='ArchiveProjectResponse', - full_name='feast.core.ArchiveProjectResponse', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=1276, - serialized_end=1300, -) - - -_LISTPROJECTSREQUEST = _descriptor.Descriptor( - name='ListProjectsRequest', - full_name='feast.core.ListProjectsRequest', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=1302, - serialized_end=1323, -) - - -_LISTPROJECTSRESPONSE = _descriptor.Descriptor( - name='ListProjectsResponse', - full_name='feast.core.ListProjectsResponse', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='projects', full_name='feast.core.ListProjectsResponse.projects', index=0, - number=1, type=9, cpp_type=9, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=1325, - serialized_end=1365, -) - -_GETFEATURESETRESPONSE.fields_by_name['feature_set'].message_type = feast_dot_core_dot_FeatureSet__pb2._FEATURESET -_LISTFEATURESETSREQUEST_FILTER.containing_type = _LISTFEATURESETSREQUEST -_LISTFEATURESETSREQUEST.fields_by_name['filter'].message_type = _LISTFEATURESETSREQUEST_FILTER -_LISTFEATURESETSRESPONSE.fields_by_name['feature_sets'].message_type = feast_dot_core_dot_FeatureSet__pb2._FEATURESET -_LISTSTORESREQUEST_FILTER.containing_type = _LISTSTORESREQUEST -_LISTSTORESREQUEST.fields_by_name['filter'].message_type = _LISTSTORESREQUEST_FILTER -_LISTSTORESRESPONSE.fields_by_name['store'].message_type = feast_dot_core_dot_Store__pb2._STORE -_APPLYFEATURESETREQUEST.fields_by_name['feature_set'].message_type = feast_dot_core_dot_FeatureSet__pb2._FEATURESET -_APPLYFEATURESETRESPONSE.fields_by_name['feature_set'].message_type = feast_dot_core_dot_FeatureSet__pb2._FEATURESET -_APPLYFEATURESETRESPONSE.fields_by_name['status'].enum_type = _APPLYFEATURESETRESPONSE_STATUS -_APPLYFEATURESETRESPONSE_STATUS.containing_type = _APPLYFEATURESETRESPONSE -_UPDATESTOREREQUEST.fields_by_name['store'].message_type = feast_dot_core_dot_Store__pb2._STORE -_UPDATESTORERESPONSE.fields_by_name['store'].message_type = feast_dot_core_dot_Store__pb2._STORE -_UPDATESTORERESPONSE.fields_by_name['status'].enum_type = _UPDATESTORERESPONSE_STATUS -_UPDATESTORERESPONSE_STATUS.containing_type = _UPDATESTORERESPONSE -DESCRIPTOR.message_types_by_name['GetFeatureSetRequest'] = _GETFEATURESETREQUEST -DESCRIPTOR.message_types_by_name['GetFeatureSetResponse'] = _GETFEATURESETRESPONSE -DESCRIPTOR.message_types_by_name['ListFeatureSetsRequest'] = _LISTFEATURESETSREQUEST -DESCRIPTOR.message_types_by_name['ListFeatureSetsResponse'] = _LISTFEATURESETSRESPONSE -DESCRIPTOR.message_types_by_name['ListStoresRequest'] = _LISTSTORESREQUEST -DESCRIPTOR.message_types_by_name['ListStoresResponse'] = _LISTSTORESRESPONSE -DESCRIPTOR.message_types_by_name['ApplyFeatureSetRequest'] = _APPLYFEATURESETREQUEST -DESCRIPTOR.message_types_by_name['ApplyFeatureSetResponse'] = _APPLYFEATURESETRESPONSE -DESCRIPTOR.message_types_by_name['GetFeastCoreVersionRequest'] = _GETFEASTCOREVERSIONREQUEST -DESCRIPTOR.message_types_by_name['GetFeastCoreVersionResponse'] = _GETFEASTCOREVERSIONRESPONSE -DESCRIPTOR.message_types_by_name['UpdateStoreRequest'] = _UPDATESTOREREQUEST -DESCRIPTOR.message_types_by_name['UpdateStoreResponse'] = _UPDATESTORERESPONSE -DESCRIPTOR.message_types_by_name['CreateProjectRequest'] = _CREATEPROJECTREQUEST -DESCRIPTOR.message_types_by_name['CreateProjectResponse'] = _CREATEPROJECTRESPONSE -DESCRIPTOR.message_types_by_name['ArchiveProjectRequest'] = _ARCHIVEPROJECTREQUEST -DESCRIPTOR.message_types_by_name['ArchiveProjectResponse'] = _ARCHIVEPROJECTRESPONSE -DESCRIPTOR.message_types_by_name['ListProjectsRequest'] = _LISTPROJECTSREQUEST -DESCRIPTOR.message_types_by_name['ListProjectsResponse'] = _LISTPROJECTSRESPONSE -_sym_db.RegisterFileDescriptor(DESCRIPTOR) - -GetFeatureSetRequest = _reflection.GeneratedProtocolMessageType('GetFeatureSetRequest', (_message.Message,), { - 'DESCRIPTOR' : _GETFEATURESETREQUEST, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.GetFeatureSetRequest) - }) -_sym_db.RegisterMessage(GetFeatureSetRequest) - -GetFeatureSetResponse = _reflection.GeneratedProtocolMessageType('GetFeatureSetResponse', (_message.Message,), { - 'DESCRIPTOR' : _GETFEATURESETRESPONSE, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.GetFeatureSetResponse) - }) -_sym_db.RegisterMessage(GetFeatureSetResponse) - -ListFeatureSetsRequest = _reflection.GeneratedProtocolMessageType('ListFeatureSetsRequest', (_message.Message,), { - - 'Filter' : _reflection.GeneratedProtocolMessageType('Filter', (_message.Message,), { - 'DESCRIPTOR' : _LISTFEATURESETSREQUEST_FILTER, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.ListFeatureSetsRequest.Filter) - }) - , - 'DESCRIPTOR' : _LISTFEATURESETSREQUEST, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.ListFeatureSetsRequest) - }) -_sym_db.RegisterMessage(ListFeatureSetsRequest) -_sym_db.RegisterMessage(ListFeatureSetsRequest.Filter) - -ListFeatureSetsResponse = _reflection.GeneratedProtocolMessageType('ListFeatureSetsResponse', (_message.Message,), { - 'DESCRIPTOR' : _LISTFEATURESETSRESPONSE, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.ListFeatureSetsResponse) - }) -_sym_db.RegisterMessage(ListFeatureSetsResponse) - -ListStoresRequest = _reflection.GeneratedProtocolMessageType('ListStoresRequest', (_message.Message,), { - - 'Filter' : _reflection.GeneratedProtocolMessageType('Filter', (_message.Message,), { - 'DESCRIPTOR' : _LISTSTORESREQUEST_FILTER, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.ListStoresRequest.Filter) - }) - , - 'DESCRIPTOR' : _LISTSTORESREQUEST, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.ListStoresRequest) - }) -_sym_db.RegisterMessage(ListStoresRequest) -_sym_db.RegisterMessage(ListStoresRequest.Filter) - -ListStoresResponse = _reflection.GeneratedProtocolMessageType('ListStoresResponse', (_message.Message,), { - 'DESCRIPTOR' : _LISTSTORESRESPONSE, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.ListStoresResponse) - }) -_sym_db.RegisterMessage(ListStoresResponse) - -ApplyFeatureSetRequest = _reflection.GeneratedProtocolMessageType('ApplyFeatureSetRequest', (_message.Message,), { - 'DESCRIPTOR' : _APPLYFEATURESETREQUEST, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.ApplyFeatureSetRequest) - }) -_sym_db.RegisterMessage(ApplyFeatureSetRequest) - -ApplyFeatureSetResponse = _reflection.GeneratedProtocolMessageType('ApplyFeatureSetResponse', (_message.Message,), { - 'DESCRIPTOR' : _APPLYFEATURESETRESPONSE, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.ApplyFeatureSetResponse) - }) -_sym_db.RegisterMessage(ApplyFeatureSetResponse) - -GetFeastCoreVersionRequest = _reflection.GeneratedProtocolMessageType('GetFeastCoreVersionRequest', (_message.Message,), { - 'DESCRIPTOR' : _GETFEASTCOREVERSIONREQUEST, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.GetFeastCoreVersionRequest) - }) -_sym_db.RegisterMessage(GetFeastCoreVersionRequest) - -GetFeastCoreVersionResponse = _reflection.GeneratedProtocolMessageType('GetFeastCoreVersionResponse', (_message.Message,), { - 'DESCRIPTOR' : _GETFEASTCOREVERSIONRESPONSE, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.GetFeastCoreVersionResponse) - }) -_sym_db.RegisterMessage(GetFeastCoreVersionResponse) - -UpdateStoreRequest = _reflection.GeneratedProtocolMessageType('UpdateStoreRequest', (_message.Message,), { - 'DESCRIPTOR' : _UPDATESTOREREQUEST, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.UpdateStoreRequest) - }) -_sym_db.RegisterMessage(UpdateStoreRequest) - -UpdateStoreResponse = _reflection.GeneratedProtocolMessageType('UpdateStoreResponse', (_message.Message,), { - 'DESCRIPTOR' : _UPDATESTORERESPONSE, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.UpdateStoreResponse) - }) -_sym_db.RegisterMessage(UpdateStoreResponse) - -CreateProjectRequest = _reflection.GeneratedProtocolMessageType('CreateProjectRequest', (_message.Message,), { - 'DESCRIPTOR' : _CREATEPROJECTREQUEST, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.CreateProjectRequest) - }) -_sym_db.RegisterMessage(CreateProjectRequest) - -CreateProjectResponse = _reflection.GeneratedProtocolMessageType('CreateProjectResponse', (_message.Message,), { - 'DESCRIPTOR' : _CREATEPROJECTRESPONSE, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.CreateProjectResponse) - }) -_sym_db.RegisterMessage(CreateProjectResponse) - -ArchiveProjectRequest = _reflection.GeneratedProtocolMessageType('ArchiveProjectRequest', (_message.Message,), { - 'DESCRIPTOR' : _ARCHIVEPROJECTREQUEST, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.ArchiveProjectRequest) - }) -_sym_db.RegisterMessage(ArchiveProjectRequest) - -ArchiveProjectResponse = _reflection.GeneratedProtocolMessageType('ArchiveProjectResponse', (_message.Message,), { - 'DESCRIPTOR' : _ARCHIVEPROJECTRESPONSE, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.ArchiveProjectResponse) - }) -_sym_db.RegisterMessage(ArchiveProjectResponse) - -ListProjectsRequest = _reflection.GeneratedProtocolMessageType('ListProjectsRequest', (_message.Message,), { - 'DESCRIPTOR' : _LISTPROJECTSREQUEST, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.ListProjectsRequest) - }) -_sym_db.RegisterMessage(ListProjectsRequest) - -ListProjectsResponse = _reflection.GeneratedProtocolMessageType('ListProjectsResponse', (_message.Message,), { - 'DESCRIPTOR' : _LISTPROJECTSRESPONSE, - '__module__' : 'feast.core.CoreService_pb2' - # @@protoc_insertion_point(class_scope:feast.core.ListProjectsResponse) - }) -_sym_db.RegisterMessage(ListProjectsResponse) - - -DESCRIPTOR._options = None - -_CORESERVICE = _descriptor.ServiceDescriptor( - name='CoreService', - full_name='feast.core.CoreService', - file=DESCRIPTOR, - index=0, - serialized_options=None, - serialized_start=1368, - serialized_end=2170, - methods=[ - _descriptor.MethodDescriptor( - name='GetFeastCoreVersion', - full_name='feast.core.CoreService.GetFeastCoreVersion', - index=0, - containing_service=None, - input_type=_GETFEASTCOREVERSIONREQUEST, - output_type=_GETFEASTCOREVERSIONRESPONSE, - serialized_options=None, - ), - _descriptor.MethodDescriptor( - name='GetFeatureSet', - full_name='feast.core.CoreService.GetFeatureSet', - index=1, - containing_service=None, - input_type=_GETFEATURESETREQUEST, - output_type=_GETFEATURESETRESPONSE, - serialized_options=None, - ), - _descriptor.MethodDescriptor( - name='ListFeatureSets', - full_name='feast.core.CoreService.ListFeatureSets', - index=2, - containing_service=None, - input_type=_LISTFEATURESETSREQUEST, - output_type=_LISTFEATURESETSRESPONSE, - serialized_options=None, - ), - _descriptor.MethodDescriptor( - name='ListStores', - full_name='feast.core.CoreService.ListStores', - index=3, - containing_service=None, - input_type=_LISTSTORESREQUEST, - output_type=_LISTSTORESRESPONSE, - serialized_options=None, - ), - _descriptor.MethodDescriptor( - name='ApplyFeatureSet', - full_name='feast.core.CoreService.ApplyFeatureSet', - index=4, - containing_service=None, - input_type=_APPLYFEATURESETREQUEST, - output_type=_APPLYFEATURESETRESPONSE, - serialized_options=None, - ), - _descriptor.MethodDescriptor( - name='UpdateStore', - full_name='feast.core.CoreService.UpdateStore', - index=5, - containing_service=None, - input_type=_UPDATESTOREREQUEST, - output_type=_UPDATESTORERESPONSE, - serialized_options=None, - ), - _descriptor.MethodDescriptor( - name='CreateProject', - full_name='feast.core.CoreService.CreateProject', - index=6, - containing_service=None, - input_type=_CREATEPROJECTREQUEST, - output_type=_CREATEPROJECTRESPONSE, - serialized_options=None, - ), - _descriptor.MethodDescriptor( - name='ArchiveProject', - full_name='feast.core.CoreService.ArchiveProject', - index=7, - containing_service=None, - input_type=_ARCHIVEPROJECTREQUEST, - output_type=_ARCHIVEPROJECTRESPONSE, - serialized_options=None, - ), - _descriptor.MethodDescriptor( - name='ListProjects', - full_name='feast.core.CoreService.ListProjects', - index=8, - containing_service=None, - input_type=_LISTPROJECTSREQUEST, - output_type=_LISTPROJECTSRESPONSE, - serialized_options=None, - ), -]) -_sym_db.RegisterServiceDescriptor(_CORESERVICE) - -DESCRIPTOR.services_by_name['CoreService'] = _CORESERVICE - -# @@protoc_insertion_point(module_scope) diff --git a/sdk/python/feast/core/CoreService_pb2.pyi b/sdk/python/feast/core/CoreService_pb2.pyi deleted file mode 100644 index 645226982ad..00000000000 --- a/sdk/python/feast/core/CoreService_pb2.pyi +++ /dev/null @@ -1,429 +0,0 @@ -# @generated by generate_proto_mypy_stubs.py. Do not edit! -import sys -from feast.core.FeatureSet_pb2 import ( - FeatureSet as feast___core___FeatureSet_pb2___FeatureSet, -) - -from feast.core.Store_pb2 import ( - Store as feast___core___Store_pb2___Store, -) - -from google.protobuf.descriptor import ( - Descriptor as google___protobuf___descriptor___Descriptor, - EnumDescriptor as google___protobuf___descriptor___EnumDescriptor, -) - -from google.protobuf.internal.containers import ( - RepeatedCompositeFieldContainer as google___protobuf___internal___containers___RepeatedCompositeFieldContainer, - RepeatedScalarFieldContainer as google___protobuf___internal___containers___RepeatedScalarFieldContainer, -) - -from google.protobuf.message import ( - Message as google___protobuf___message___Message, -) - -from typing import ( - Iterable as typing___Iterable, - List as typing___List, - Optional as typing___Optional, - Text as typing___Text, - Tuple as typing___Tuple, - cast as typing___cast, -) - -from typing_extensions import ( - Literal as typing_extensions___Literal, -) - - -class GetFeatureSetRequest(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - project = ... # type: typing___Text - name = ... # type: typing___Text - version = ... # type: int - - def __init__(self, - *, - project : typing___Optional[typing___Text] = None, - name : typing___Optional[typing___Text] = None, - version : typing___Optional[int] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> GetFeatureSetRequest: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"name",u"project",u"version"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"name",b"name",u"project",b"project",u"version",b"version"]) -> None: ... - -class GetFeatureSetResponse(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - @property - def feature_set(self) -> feast___core___FeatureSet_pb2___FeatureSet: ... - - def __init__(self, - *, - feature_set : typing___Optional[feast___core___FeatureSet_pb2___FeatureSet] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> GetFeatureSetResponse: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"feature_set"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"feature_set"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"feature_set",b"feature_set"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"feature_set",b"feature_set"]) -> None: ... - -class ListFeatureSetsRequest(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class Filter(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - project = ... # type: typing___Text - feature_set_name = ... # type: typing___Text - feature_set_version = ... # type: typing___Text - - def __init__(self, - *, - project : typing___Optional[typing___Text] = None, - feature_set_name : typing___Optional[typing___Text] = None, - feature_set_version : typing___Optional[typing___Text] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> ListFeatureSetsRequest.Filter: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"feature_set_name",u"feature_set_version",u"project"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"feature_set_name",b"feature_set_name",u"feature_set_version",b"feature_set_version",u"project",b"project"]) -> None: ... - - - @property - def filter(self) -> ListFeatureSetsRequest.Filter: ... - - def __init__(self, - *, - filter : typing___Optional[ListFeatureSetsRequest.Filter] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> ListFeatureSetsRequest: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"filter"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"filter"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"filter",b"filter"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"filter",b"filter"]) -> None: ... - -class ListFeatureSetsResponse(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - @property - def feature_sets(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[feast___core___FeatureSet_pb2___FeatureSet]: ... - - def __init__(self, - *, - feature_sets : typing___Optional[typing___Iterable[feast___core___FeatureSet_pb2___FeatureSet]] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> ListFeatureSetsResponse: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"feature_sets"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"feature_sets",b"feature_sets"]) -> None: ... - -class ListStoresRequest(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class Filter(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - name = ... # type: typing___Text - - def __init__(self, - *, - name : typing___Optional[typing___Text] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> ListStoresRequest.Filter: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"name"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"name",b"name"]) -> None: ... - - - @property - def filter(self) -> ListStoresRequest.Filter: ... - - def __init__(self, - *, - filter : typing___Optional[ListStoresRequest.Filter] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> ListStoresRequest: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"filter"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"filter"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"filter",b"filter"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"filter",b"filter"]) -> None: ... - -class ListStoresResponse(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - @property - def store(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[feast___core___Store_pb2___Store]: ... - - def __init__(self, - *, - store : typing___Optional[typing___Iterable[feast___core___Store_pb2___Store]] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> ListStoresResponse: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"store"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"store",b"store"]) -> None: ... - -class ApplyFeatureSetRequest(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - @property - def feature_set(self) -> feast___core___FeatureSet_pb2___FeatureSet: ... - - def __init__(self, - *, - feature_set : typing___Optional[feast___core___FeatureSet_pb2___FeatureSet] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> ApplyFeatureSetRequest: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"feature_set"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"feature_set"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"feature_set",b"feature_set"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"feature_set",b"feature_set"]) -> None: ... - -class ApplyFeatureSetResponse(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class Status(int): - DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... - @classmethod - def Name(cls, number: int) -> str: ... - @classmethod - def Value(cls, name: str) -> ApplyFeatureSetResponse.Status: ... - @classmethod - def keys(cls) -> typing___List[str]: ... - @classmethod - def values(cls) -> typing___List[ApplyFeatureSetResponse.Status]: ... - @classmethod - def items(cls) -> typing___List[typing___Tuple[str, ApplyFeatureSetResponse.Status]]: ... - NO_CHANGE = typing___cast(ApplyFeatureSetResponse.Status, 0) - CREATED = typing___cast(ApplyFeatureSetResponse.Status, 1) - ERROR = typing___cast(ApplyFeatureSetResponse.Status, 2) - NO_CHANGE = typing___cast(ApplyFeatureSetResponse.Status, 0) - CREATED = typing___cast(ApplyFeatureSetResponse.Status, 1) - ERROR = typing___cast(ApplyFeatureSetResponse.Status, 2) - - status = ... # type: ApplyFeatureSetResponse.Status - - @property - def feature_set(self) -> feast___core___FeatureSet_pb2___FeatureSet: ... - - def __init__(self, - *, - feature_set : typing___Optional[feast___core___FeatureSet_pb2___FeatureSet] = None, - status : typing___Optional[ApplyFeatureSetResponse.Status] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> ApplyFeatureSetResponse: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"feature_set"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"feature_set",u"status"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"feature_set",b"feature_set"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"feature_set",b"feature_set",u"status",b"status"]) -> None: ... - -class GetFeastCoreVersionRequest(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - def __init__(self, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> GetFeastCoreVersionRequest: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - -class GetFeastCoreVersionResponse(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - version = ... # type: typing___Text - - def __init__(self, - *, - version : typing___Optional[typing___Text] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> GetFeastCoreVersionResponse: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"version"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"version",b"version"]) -> None: ... - -class UpdateStoreRequest(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - @property - def store(self) -> feast___core___Store_pb2___Store: ... - - def __init__(self, - *, - store : typing___Optional[feast___core___Store_pb2___Store] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> UpdateStoreRequest: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"store"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"store"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"store",b"store"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"store",b"store"]) -> None: ... - -class UpdateStoreResponse(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class Status(int): - DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... - @classmethod - def Name(cls, number: int) -> str: ... - @classmethod - def Value(cls, name: str) -> UpdateStoreResponse.Status: ... - @classmethod - def keys(cls) -> typing___List[str]: ... - @classmethod - def values(cls) -> typing___List[UpdateStoreResponse.Status]: ... - @classmethod - def items(cls) -> typing___List[typing___Tuple[str, UpdateStoreResponse.Status]]: ... - NO_CHANGE = typing___cast(UpdateStoreResponse.Status, 0) - UPDATED = typing___cast(UpdateStoreResponse.Status, 1) - NO_CHANGE = typing___cast(UpdateStoreResponse.Status, 0) - UPDATED = typing___cast(UpdateStoreResponse.Status, 1) - - status = ... # type: UpdateStoreResponse.Status - - @property - def store(self) -> feast___core___Store_pb2___Store: ... - - def __init__(self, - *, - store : typing___Optional[feast___core___Store_pb2___Store] = None, - status : typing___Optional[UpdateStoreResponse.Status] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> UpdateStoreResponse: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"store"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"status",u"store"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"store",b"store"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"status",b"status",u"store",b"store"]) -> None: ... - -class CreateProjectRequest(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - name = ... # type: typing___Text - - def __init__(self, - *, - name : typing___Optional[typing___Text] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> CreateProjectRequest: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"name"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"name",b"name"]) -> None: ... - -class CreateProjectResponse(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - def __init__(self, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> CreateProjectResponse: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - -class ArchiveProjectRequest(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - name = ... # type: typing___Text - - def __init__(self, - *, - name : typing___Optional[typing___Text] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> ArchiveProjectRequest: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"name"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"name",b"name"]) -> None: ... - -class ArchiveProjectResponse(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - def __init__(self, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> ArchiveProjectResponse: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - -class ListProjectsRequest(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - def __init__(self, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> ListProjectsRequest: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - -class ListProjectsResponse(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - projects = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] - - def __init__(self, - *, - projects : typing___Optional[typing___Iterable[typing___Text]] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> ListProjectsResponse: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"projects"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"projects",b"projects"]) -> None: ... diff --git a/sdk/python/feast/core/CoreService_pb2_grpc.py b/sdk/python/feast/core/CoreService_pb2_grpc.py deleted file mode 100644 index 0e17d0552a2..00000000000 --- a/sdk/python/feast/core/CoreService_pb2_grpc.py +++ /dev/null @@ -1,203 +0,0 @@ -# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT! -import grpc - -from feast.core import CoreService_pb2 as feast_dot_core_dot_CoreService__pb2 - - -class CoreServiceStub(object): - # missing associated documentation comment in .proto file - pass - - def __init__(self, channel): - """Constructor. - - Args: - channel: A grpc.Channel. - """ - self.GetFeastCoreVersion = channel.unary_unary( - '/feast.core.CoreService/GetFeastCoreVersion', - request_serializer=feast_dot_core_dot_CoreService__pb2.GetFeastCoreVersionRequest.SerializeToString, - response_deserializer=feast_dot_core_dot_CoreService__pb2.GetFeastCoreVersionResponse.FromString, - ) - self.GetFeatureSet = channel.unary_unary( - '/feast.core.CoreService/GetFeatureSet', - request_serializer=feast_dot_core_dot_CoreService__pb2.GetFeatureSetRequest.SerializeToString, - response_deserializer=feast_dot_core_dot_CoreService__pb2.GetFeatureSetResponse.FromString, - ) - self.ListFeatureSets = channel.unary_unary( - '/feast.core.CoreService/ListFeatureSets', - request_serializer=feast_dot_core_dot_CoreService__pb2.ListFeatureSetsRequest.SerializeToString, - response_deserializer=feast_dot_core_dot_CoreService__pb2.ListFeatureSetsResponse.FromString, - ) - self.ListStores = channel.unary_unary( - '/feast.core.CoreService/ListStores', - request_serializer=feast_dot_core_dot_CoreService__pb2.ListStoresRequest.SerializeToString, - response_deserializer=feast_dot_core_dot_CoreService__pb2.ListStoresResponse.FromString, - ) - self.ApplyFeatureSet = channel.unary_unary( - '/feast.core.CoreService/ApplyFeatureSet', - request_serializer=feast_dot_core_dot_CoreService__pb2.ApplyFeatureSetRequest.SerializeToString, - response_deserializer=feast_dot_core_dot_CoreService__pb2.ApplyFeatureSetResponse.FromString, - ) - self.UpdateStore = channel.unary_unary( - '/feast.core.CoreService/UpdateStore', - request_serializer=feast_dot_core_dot_CoreService__pb2.UpdateStoreRequest.SerializeToString, - response_deserializer=feast_dot_core_dot_CoreService__pb2.UpdateStoreResponse.FromString, - ) - self.CreateProject = channel.unary_unary( - '/feast.core.CoreService/CreateProject', - request_serializer=feast_dot_core_dot_CoreService__pb2.CreateProjectRequest.SerializeToString, - response_deserializer=feast_dot_core_dot_CoreService__pb2.CreateProjectResponse.FromString, - ) - self.ArchiveProject = channel.unary_unary( - '/feast.core.CoreService/ArchiveProject', - request_serializer=feast_dot_core_dot_CoreService__pb2.ArchiveProjectRequest.SerializeToString, - response_deserializer=feast_dot_core_dot_CoreService__pb2.ArchiveProjectResponse.FromString, - ) - self.ListProjects = channel.unary_unary( - '/feast.core.CoreService/ListProjects', - request_serializer=feast_dot_core_dot_CoreService__pb2.ListProjectsRequest.SerializeToString, - response_deserializer=feast_dot_core_dot_CoreService__pb2.ListProjectsResponse.FromString, - ) - - -class CoreServiceServicer(object): - # missing associated documentation comment in .proto file - pass - - def GetFeastCoreVersion(self, request, context): - """Retrieve version information about this Feast deployment - """ - context.set_code(grpc.StatusCode.UNIMPLEMENTED) - context.set_details('Method not implemented!') - raise NotImplementedError('Method not implemented!') - - def GetFeatureSet(self, request, context): - """Returns a specific feature set - """ - context.set_code(grpc.StatusCode.UNIMPLEMENTED) - context.set_details('Method not implemented!') - raise NotImplementedError('Method not implemented!') - - def ListFeatureSets(self, request, context): - """Retrieve feature set details given a filter. - - Returns all feature sets matching that filter. If none are found, - an empty list will be returned. - If no filter is provided in the request, the response will contain all the feature - sets currently stored in the registry. - """ - context.set_code(grpc.StatusCode.UNIMPLEMENTED) - context.set_details('Method not implemented!') - raise NotImplementedError('Method not implemented!') - - def ListStores(self, request, context): - """Retrieve store details given a filter. - - Returns all stores matching that filter. If none are found, an empty list will be returned. - If no filter is provided in the request, the response will contain all the stores currently - stored in the registry. - """ - context.set_code(grpc.StatusCode.UNIMPLEMENTED) - context.set_details('Method not implemented!') - raise NotImplementedError('Method not implemented!') - - def ApplyFeatureSet(self, request, context): - """Create or update and existing feature set. - - This function is idempotent - it will not create a new feature set if schema does not change. - If an existing feature set is updated, core will advance the version number, which will be - returned in response. - """ - context.set_code(grpc.StatusCode.UNIMPLEMENTED) - context.set_details('Method not implemented!') - raise NotImplementedError('Method not implemented!') - - def UpdateStore(self, request, context): - """Updates core with the configuration of the store. - - If the changes are valid, core will return the given store configuration in response, and - start or update the necessary feature population jobs for the updated store. - """ - context.set_code(grpc.StatusCode.UNIMPLEMENTED) - context.set_details('Method not implemented!') - raise NotImplementedError('Method not implemented!') - - def CreateProject(self, request, context): - """Creates a project. Projects serve as namespaces within which resources like features will be - created. Both feature set names as well as field names must be unique within a project. Project - names themselves must be globally unique. - """ - context.set_code(grpc.StatusCode.UNIMPLEMENTED) - context.set_details('Method not implemented!') - raise NotImplementedError('Method not implemented!') - - def ArchiveProject(self, request, context): - """Archives a project. Archived projects will continue to exist and function, but won't be visible - through the Core API. Any existing ingestion or serving requests will continue to function, - but will result in warning messages being logged. It is not possible to unarchive a project - through the Core API - """ - context.set_code(grpc.StatusCode.UNIMPLEMENTED) - context.set_details('Method not implemented!') - raise NotImplementedError('Method not implemented!') - - def ListProjects(self, request, context): - """Lists all projects active projects. - """ - context.set_code(grpc.StatusCode.UNIMPLEMENTED) - context.set_details('Method not implemented!') - raise NotImplementedError('Method not implemented!') - - -def add_CoreServiceServicer_to_server(servicer, server): - rpc_method_handlers = { - 'GetFeastCoreVersion': grpc.unary_unary_rpc_method_handler( - servicer.GetFeastCoreVersion, - request_deserializer=feast_dot_core_dot_CoreService__pb2.GetFeastCoreVersionRequest.FromString, - response_serializer=feast_dot_core_dot_CoreService__pb2.GetFeastCoreVersionResponse.SerializeToString, - ), - 'GetFeatureSet': grpc.unary_unary_rpc_method_handler( - servicer.GetFeatureSet, - request_deserializer=feast_dot_core_dot_CoreService__pb2.GetFeatureSetRequest.FromString, - response_serializer=feast_dot_core_dot_CoreService__pb2.GetFeatureSetResponse.SerializeToString, - ), - 'ListFeatureSets': grpc.unary_unary_rpc_method_handler( - servicer.ListFeatureSets, - request_deserializer=feast_dot_core_dot_CoreService__pb2.ListFeatureSetsRequest.FromString, - response_serializer=feast_dot_core_dot_CoreService__pb2.ListFeatureSetsResponse.SerializeToString, - ), - 'ListStores': grpc.unary_unary_rpc_method_handler( - servicer.ListStores, - request_deserializer=feast_dot_core_dot_CoreService__pb2.ListStoresRequest.FromString, - response_serializer=feast_dot_core_dot_CoreService__pb2.ListStoresResponse.SerializeToString, - ), - 'ApplyFeatureSet': grpc.unary_unary_rpc_method_handler( - servicer.ApplyFeatureSet, - request_deserializer=feast_dot_core_dot_CoreService__pb2.ApplyFeatureSetRequest.FromString, - response_serializer=feast_dot_core_dot_CoreService__pb2.ApplyFeatureSetResponse.SerializeToString, - ), - 'UpdateStore': grpc.unary_unary_rpc_method_handler( - servicer.UpdateStore, - request_deserializer=feast_dot_core_dot_CoreService__pb2.UpdateStoreRequest.FromString, - response_serializer=feast_dot_core_dot_CoreService__pb2.UpdateStoreResponse.SerializeToString, - ), - 'CreateProject': grpc.unary_unary_rpc_method_handler( - servicer.CreateProject, - request_deserializer=feast_dot_core_dot_CoreService__pb2.CreateProjectRequest.FromString, - response_serializer=feast_dot_core_dot_CoreService__pb2.CreateProjectResponse.SerializeToString, - ), - 'ArchiveProject': grpc.unary_unary_rpc_method_handler( - servicer.ArchiveProject, - request_deserializer=feast_dot_core_dot_CoreService__pb2.ArchiveProjectRequest.FromString, - response_serializer=feast_dot_core_dot_CoreService__pb2.ArchiveProjectResponse.SerializeToString, - ), - 'ListProjects': grpc.unary_unary_rpc_method_handler( - servicer.ListProjects, - request_deserializer=feast_dot_core_dot_CoreService__pb2.ListProjectsRequest.FromString, - response_serializer=feast_dot_core_dot_CoreService__pb2.ListProjectsResponse.SerializeToString, - ), - } - generic_handler = grpc.method_handlers_generic_handler( - 'feast.core.CoreService', rpc_method_handlers) - server.add_generic_rpc_handlers((generic_handler,)) diff --git a/sdk/python/feast/core/FeatureSet_pb2.py b/sdk/python/feast/core/FeatureSet_pb2.py deleted file mode 100644 index 991220ccae5..00000000000 --- a/sdk/python/feast/core/FeatureSet_pb2.py +++ /dev/null @@ -1,344 +0,0 @@ -# -*- coding: utf-8 -*- -# Generated by the protocol buffer compiler. DO NOT EDIT! -# source: feast/core/FeatureSet.proto - -import sys -_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1')) -from google.protobuf.internal import enum_type_wrapper -from google.protobuf import descriptor as _descriptor -from google.protobuf import message as _message -from google.protobuf import reflection as _reflection -from google.protobuf import symbol_database as _symbol_database -# @@protoc_insertion_point(imports) - -_sym_db = _symbol_database.Default() - - -from feast.types import Value_pb2 as feast_dot_types_dot_Value__pb2 -from feast.core import Source_pb2 as feast_dot_core_dot_Source__pb2 -from google.protobuf import duration_pb2 as google_dot_protobuf_dot_duration__pb2 -from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2 - - -DESCRIPTOR = _descriptor.FileDescriptor( - name='feast/core/FeatureSet.proto', - package='feast.core', - syntax='proto3', - serialized_options=_b('\n\nfeast.coreB\017FeatureSetProtoZ/github.com/gojek/feast/sdk/go/protos/feast/core'), - serialized_pb=_b('\n\x1b\x66\x65\x61st/core/FeatureSet.proto\x12\nfeast.core\x1a\x17\x66\x65\x61st/types/Value.proto\x1a\x17\x66\x65\x61st/core/Source.proto\x1a\x1egoogle/protobuf/duration.proto\x1a\x1fgoogle/protobuf/timestamp.proto\"`\n\nFeatureSet\x12(\n\x04spec\x18\x01 \x01(\x0b\x32\x1a.feast.core.FeatureSetSpec\x12(\n\x04meta\x18\x02 \x01(\x0b\x32\x1a.feast.core.FeatureSetMeta\"\xe5\x01\n\x0e\x46\x65\x61tureSetSpec\x12\x0f\n\x07project\x18\x07 \x01(\t\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0f\n\x07version\x18\x02 \x01(\x05\x12(\n\x08\x65ntities\x18\x03 \x03(\x0b\x32\x16.feast.core.EntitySpec\x12)\n\x08\x66\x65\x61tures\x18\x04 \x03(\x0b\x32\x17.feast.core.FeatureSpec\x12*\n\x07max_age\x18\x05 \x01(\x0b\x32\x19.google.protobuf.Duration\x12\"\n\x06source\x18\x06 \x01(\x0b\x32\x12.feast.core.Source\"K\n\nEntitySpec\x12\x0c\n\x04name\x18\x01 \x01(\t\x12/\n\nvalue_type\x18\x02 \x01(\x0e\x32\x1b.feast.types.ValueType.Enum\"L\n\x0b\x46\x65\x61tureSpec\x12\x0c\n\x04name\x18\x01 \x01(\t\x12/\n\nvalue_type\x18\x02 \x01(\x0e\x32\x1b.feast.types.ValueType.Enum\"u\n\x0e\x46\x65\x61tureSetMeta\x12\x35\n\x11\x63reated_timestamp\x18\x01 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12,\n\x06status\x18\x02 \x01(\x0e\x32\x1c.feast.core.FeatureSetStatus*L\n\x10\x46\x65\x61tureSetStatus\x12\x12\n\x0eSTATUS_INVALID\x10\x00\x12\x12\n\x0eSTATUS_PENDING\x10\x01\x12\x10\n\x0cSTATUS_READY\x10\x02\x42N\n\nfeast.coreB\x0f\x46\x65\x61tureSetProtoZ/github.com/gojek/feast/sdk/go/protos/feast/coreb\x06proto3') - , - dependencies=[feast_dot_types_dot_Value__pb2.DESCRIPTOR,feast_dot_core_dot_Source__pb2.DESCRIPTOR,google_dot_protobuf_dot_duration__pb2.DESCRIPTOR,google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,]) - -_FEATURESETSTATUS = _descriptor.EnumDescriptor( - name='FeatureSetStatus', - full_name='feast.core.FeatureSetStatus', - filename=None, - file=DESCRIPTOR, - values=[ - _descriptor.EnumValueDescriptor( - name='STATUS_INVALID', index=0, number=0, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='STATUS_PENDING', index=1, number=1, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='STATUS_READY', index=2, number=2, - serialized_options=None, - type=None), - ], - containing_type=None, - serialized_options=None, - serialized_start=762, - serialized_end=838, -) -_sym_db.RegisterEnumDescriptor(_FEATURESETSTATUS) - -FeatureSetStatus = enum_type_wrapper.EnumTypeWrapper(_FEATURESETSTATUS) -STATUS_INVALID = 0 -STATUS_PENDING = 1 -STATUS_READY = 2 - - - -_FEATURESET = _descriptor.Descriptor( - name='FeatureSet', - full_name='feast.core.FeatureSet', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='spec', full_name='feast.core.FeatureSet.spec', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='meta', full_name='feast.core.FeatureSet.meta', index=1, - number=2, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=158, - serialized_end=254, -) - - -_FEATURESETSPEC = _descriptor.Descriptor( - name='FeatureSetSpec', - full_name='feast.core.FeatureSetSpec', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='project', full_name='feast.core.FeatureSetSpec.project', index=0, - number=7, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='name', full_name='feast.core.FeatureSetSpec.name', index=1, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='version', full_name='feast.core.FeatureSetSpec.version', index=2, - number=2, type=5, cpp_type=1, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='entities', full_name='feast.core.FeatureSetSpec.entities', index=3, - number=3, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='features', full_name='feast.core.FeatureSetSpec.features', index=4, - number=4, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='max_age', full_name='feast.core.FeatureSetSpec.max_age', index=5, - number=5, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='source', full_name='feast.core.FeatureSetSpec.source', index=6, - number=6, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=257, - serialized_end=486, -) - - -_ENTITYSPEC = _descriptor.Descriptor( - name='EntitySpec', - full_name='feast.core.EntitySpec', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='name', full_name='feast.core.EntitySpec.name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='value_type', full_name='feast.core.EntitySpec.value_type', index=1, - number=2, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=488, - serialized_end=563, -) - - -_FEATURESPEC = _descriptor.Descriptor( - name='FeatureSpec', - full_name='feast.core.FeatureSpec', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='name', full_name='feast.core.FeatureSpec.name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='value_type', full_name='feast.core.FeatureSpec.value_type', index=1, - number=2, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=565, - serialized_end=641, -) - - -_FEATURESETMETA = _descriptor.Descriptor( - name='FeatureSetMeta', - full_name='feast.core.FeatureSetMeta', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='created_timestamp', full_name='feast.core.FeatureSetMeta.created_timestamp', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='status', full_name='feast.core.FeatureSetMeta.status', index=1, - number=2, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=643, - serialized_end=760, -) - -_FEATURESET.fields_by_name['spec'].message_type = _FEATURESETSPEC -_FEATURESET.fields_by_name['meta'].message_type = _FEATURESETMETA -_FEATURESETSPEC.fields_by_name['entities'].message_type = _ENTITYSPEC -_FEATURESETSPEC.fields_by_name['features'].message_type = _FEATURESPEC -_FEATURESETSPEC.fields_by_name['max_age'].message_type = google_dot_protobuf_dot_duration__pb2._DURATION -_FEATURESETSPEC.fields_by_name['source'].message_type = feast_dot_core_dot_Source__pb2._SOURCE -_ENTITYSPEC.fields_by_name['value_type'].enum_type = feast_dot_types_dot_Value__pb2._VALUETYPE_ENUM -_FEATURESPEC.fields_by_name['value_type'].enum_type = feast_dot_types_dot_Value__pb2._VALUETYPE_ENUM -_FEATURESETMETA.fields_by_name['created_timestamp'].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP -_FEATURESETMETA.fields_by_name['status'].enum_type = _FEATURESETSTATUS -DESCRIPTOR.message_types_by_name['FeatureSet'] = _FEATURESET -DESCRIPTOR.message_types_by_name['FeatureSetSpec'] = _FEATURESETSPEC -DESCRIPTOR.message_types_by_name['EntitySpec'] = _ENTITYSPEC -DESCRIPTOR.message_types_by_name['FeatureSpec'] = _FEATURESPEC -DESCRIPTOR.message_types_by_name['FeatureSetMeta'] = _FEATURESETMETA -DESCRIPTOR.enum_types_by_name['FeatureSetStatus'] = _FEATURESETSTATUS -_sym_db.RegisterFileDescriptor(DESCRIPTOR) - -FeatureSet = _reflection.GeneratedProtocolMessageType('FeatureSet', (_message.Message,), { - 'DESCRIPTOR' : _FEATURESET, - '__module__' : 'feast.core.FeatureSet_pb2' - # @@protoc_insertion_point(class_scope:feast.core.FeatureSet) - }) -_sym_db.RegisterMessage(FeatureSet) - -FeatureSetSpec = _reflection.GeneratedProtocolMessageType('FeatureSetSpec', (_message.Message,), { - 'DESCRIPTOR' : _FEATURESETSPEC, - '__module__' : 'feast.core.FeatureSet_pb2' - # @@protoc_insertion_point(class_scope:feast.core.FeatureSetSpec) - }) -_sym_db.RegisterMessage(FeatureSetSpec) - -EntitySpec = _reflection.GeneratedProtocolMessageType('EntitySpec', (_message.Message,), { - 'DESCRIPTOR' : _ENTITYSPEC, - '__module__' : 'feast.core.FeatureSet_pb2' - # @@protoc_insertion_point(class_scope:feast.core.EntitySpec) - }) -_sym_db.RegisterMessage(EntitySpec) - -FeatureSpec = _reflection.GeneratedProtocolMessageType('FeatureSpec', (_message.Message,), { - 'DESCRIPTOR' : _FEATURESPEC, - '__module__' : 'feast.core.FeatureSet_pb2' - # @@protoc_insertion_point(class_scope:feast.core.FeatureSpec) - }) -_sym_db.RegisterMessage(FeatureSpec) - -FeatureSetMeta = _reflection.GeneratedProtocolMessageType('FeatureSetMeta', (_message.Message,), { - 'DESCRIPTOR' : _FEATURESETMETA, - '__module__' : 'feast.core.FeatureSet_pb2' - # @@protoc_insertion_point(class_scope:feast.core.FeatureSetMeta) - }) -_sym_db.RegisterMessage(FeatureSetMeta) - - -DESCRIPTOR._options = None -# @@protoc_insertion_point(module_scope) diff --git a/sdk/python/feast/core/FeatureSet_pb2.pyi b/sdk/python/feast/core/FeatureSet_pb2.pyi deleted file mode 100644 index c663c70c682..00000000000 --- a/sdk/python/feast/core/FeatureSet_pb2.pyi +++ /dev/null @@ -1,188 +0,0 @@ -# @generated by generate_proto_mypy_stubs.py. Do not edit! -import sys -from feast.core.Source_pb2 import ( - Source as feast___core___Source_pb2___Source, -) - -from feast.types.Value_pb2 import ( - ValueType as feast___types___Value_pb2___ValueType, -) - -from google.protobuf.descriptor import ( - Descriptor as google___protobuf___descriptor___Descriptor, - EnumDescriptor as google___protobuf___descriptor___EnumDescriptor, -) - -from google.protobuf.duration_pb2 import ( - Duration as google___protobuf___duration_pb2___Duration, -) - -from google.protobuf.internal.containers import ( - RepeatedCompositeFieldContainer as google___protobuf___internal___containers___RepeatedCompositeFieldContainer, -) - -from google.protobuf.message import ( - Message as google___protobuf___message___Message, -) - -from google.protobuf.timestamp_pb2 import ( - Timestamp as google___protobuf___timestamp_pb2___Timestamp, -) - -from typing import ( - Iterable as typing___Iterable, - List as typing___List, - Optional as typing___Optional, - Text as typing___Text, - Tuple as typing___Tuple, - cast as typing___cast, -) - -from typing_extensions import ( - Literal as typing_extensions___Literal, -) - - -class FeatureSetStatus(int): - DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... - @classmethod - def Name(cls, number: int) -> str: ... - @classmethod - def Value(cls, name: str) -> FeatureSetStatus: ... - @classmethod - def keys(cls) -> typing___List[str]: ... - @classmethod - def values(cls) -> typing___List[FeatureSetStatus]: ... - @classmethod - def items(cls) -> typing___List[typing___Tuple[str, FeatureSetStatus]]: ... - STATUS_INVALID = typing___cast(FeatureSetStatus, 0) - STATUS_PENDING = typing___cast(FeatureSetStatus, 1) - STATUS_READY = typing___cast(FeatureSetStatus, 2) -STATUS_INVALID = typing___cast(FeatureSetStatus, 0) -STATUS_PENDING = typing___cast(FeatureSetStatus, 1) -STATUS_READY = typing___cast(FeatureSetStatus, 2) - -class FeatureSet(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - @property - def spec(self) -> FeatureSetSpec: ... - - @property - def meta(self) -> FeatureSetMeta: ... - - def __init__(self, - *, - spec : typing___Optional[FeatureSetSpec] = None, - meta : typing___Optional[FeatureSetMeta] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> FeatureSet: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"meta",u"spec"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"meta",u"spec"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"meta",b"meta",u"spec",b"spec"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"meta",b"meta",u"spec",b"spec"]) -> None: ... - -class FeatureSetSpec(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - project = ... # type: typing___Text - name = ... # type: typing___Text - version = ... # type: int - - @property - def entities(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[EntitySpec]: ... - - @property - def features(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[FeatureSpec]: ... - - @property - def max_age(self) -> google___protobuf___duration_pb2___Duration: ... - - @property - def source(self) -> feast___core___Source_pb2___Source: ... - - def __init__(self, - *, - project : typing___Optional[typing___Text] = None, - name : typing___Optional[typing___Text] = None, - version : typing___Optional[int] = None, - entities : typing___Optional[typing___Iterable[EntitySpec]] = None, - features : typing___Optional[typing___Iterable[FeatureSpec]] = None, - max_age : typing___Optional[google___protobuf___duration_pb2___Duration] = None, - source : typing___Optional[feast___core___Source_pb2___Source] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> FeatureSetSpec: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"max_age",u"source"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"entities",u"features",u"max_age",u"name",u"project",u"source",u"version"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"max_age",b"max_age",u"source",b"source"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"entities",b"entities",u"features",b"features",u"max_age",b"max_age",u"name",b"name",u"project",b"project",u"source",b"source",u"version",b"version"]) -> None: ... - -class EntitySpec(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - name = ... # type: typing___Text - value_type = ... # type: feast___types___Value_pb2___ValueType.Enum - - def __init__(self, - *, - name : typing___Optional[typing___Text] = None, - value_type : typing___Optional[feast___types___Value_pb2___ValueType.Enum] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> EntitySpec: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"name",u"value_type"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"name",b"name",u"value_type",b"value_type"]) -> None: ... - -class FeatureSpec(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - name = ... # type: typing___Text - value_type = ... # type: feast___types___Value_pb2___ValueType.Enum - - def __init__(self, - *, - name : typing___Optional[typing___Text] = None, - value_type : typing___Optional[feast___types___Value_pb2___ValueType.Enum] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> FeatureSpec: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"name",u"value_type"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"name",b"name",u"value_type",b"value_type"]) -> None: ... - -class FeatureSetMeta(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - status = ... # type: FeatureSetStatus - - @property - def created_timestamp(self) -> google___protobuf___timestamp_pb2___Timestamp: ... - - def __init__(self, - *, - created_timestamp : typing___Optional[google___protobuf___timestamp_pb2___Timestamp] = None, - status : typing___Optional[FeatureSetStatus] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> FeatureSetMeta: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"created_timestamp"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"created_timestamp",u"status"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"created_timestamp",b"created_timestamp"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"created_timestamp",b"created_timestamp",u"status",b"status"]) -> None: ... diff --git a/sdk/python/feast/core/FeatureSet_pb2_grpc.py b/sdk/python/feast/core/FeatureSet_pb2_grpc.py deleted file mode 100644 index a89435267cb..00000000000 --- a/sdk/python/feast/core/FeatureSet_pb2_grpc.py +++ /dev/null @@ -1,3 +0,0 @@ -# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT! -import grpc - diff --git a/sdk/python/feast/core/Source_pb2.py b/sdk/python/feast/core/Source_pb2.py deleted file mode 100644 index e0d0dd64313..00000000000 --- a/sdk/python/feast/core/Source_pb2.py +++ /dev/null @@ -1,159 +0,0 @@ -# -*- coding: utf-8 -*- -# Generated by the protocol buffer compiler. DO NOT EDIT! -# source: feast/core/Source.proto - -import sys -_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1')) -from google.protobuf.internal import enum_type_wrapper -from google.protobuf import descriptor as _descriptor -from google.protobuf import message as _message -from google.protobuf import reflection as _reflection -from google.protobuf import symbol_database as _symbol_database -# @@protoc_insertion_point(imports) - -_sym_db = _symbol_database.Default() - - - - -DESCRIPTOR = _descriptor.FileDescriptor( - name='feast/core/Source.proto', - package='feast.core', - syntax='proto3', - serialized_options=_b('\n\nfeast.coreB\013SourceProtoZ/github.com/gojek/feast/sdk/go/protos/feast/core'), - serialized_pb=_b('\n\x17\x66\x65\x61st/core/Source.proto\x12\nfeast.core\"}\n\x06Source\x12$\n\x04type\x18\x01 \x01(\x0e\x32\x16.feast.core.SourceType\x12<\n\x13kafka_source_config\x18\x02 \x01(\x0b\x32\x1d.feast.core.KafkaSourceConfigH\x00\x42\x0f\n\rsource_config\"=\n\x11KafkaSourceConfig\x12\x19\n\x11\x62ootstrap_servers\x18\x01 \x01(\t\x12\r\n\x05topic\x18\x02 \x01(\t*$\n\nSourceType\x12\x0b\n\x07INVALID\x10\x00\x12\t\n\x05KAFKA\x10\x01\x42J\n\nfeast.coreB\x0bSourceProtoZ/github.com/gojek/feast/sdk/go/protos/feast/coreb\x06proto3') -) - -_SOURCETYPE = _descriptor.EnumDescriptor( - name='SourceType', - full_name='feast.core.SourceType', - filename=None, - file=DESCRIPTOR, - values=[ - _descriptor.EnumValueDescriptor( - name='INVALID', index=0, number=0, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='KAFKA', index=1, number=1, - serialized_options=None, - type=None), - ], - containing_type=None, - serialized_options=None, - serialized_start=229, - serialized_end=265, -) -_sym_db.RegisterEnumDescriptor(_SOURCETYPE) - -SourceType = enum_type_wrapper.EnumTypeWrapper(_SOURCETYPE) -INVALID = 0 -KAFKA = 1 - - - -_SOURCE = _descriptor.Descriptor( - name='Source', - full_name='feast.core.Source', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='type', full_name='feast.core.Source.type', index=0, - number=1, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='kafka_source_config', full_name='feast.core.Source.kafka_source_config', index=1, - number=2, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - _descriptor.OneofDescriptor( - name='source_config', full_name='feast.core.Source.source_config', - index=0, containing_type=None, fields=[]), - ], - serialized_start=39, - serialized_end=164, -) - - -_KAFKASOURCECONFIG = _descriptor.Descriptor( - name='KafkaSourceConfig', - full_name='feast.core.KafkaSourceConfig', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='bootstrap_servers', full_name='feast.core.KafkaSourceConfig.bootstrap_servers', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='topic', full_name='feast.core.KafkaSourceConfig.topic', index=1, - number=2, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=166, - serialized_end=227, -) - -_SOURCE.fields_by_name['type'].enum_type = _SOURCETYPE -_SOURCE.fields_by_name['kafka_source_config'].message_type = _KAFKASOURCECONFIG -_SOURCE.oneofs_by_name['source_config'].fields.append( - _SOURCE.fields_by_name['kafka_source_config']) -_SOURCE.fields_by_name['kafka_source_config'].containing_oneof = _SOURCE.oneofs_by_name['source_config'] -DESCRIPTOR.message_types_by_name['Source'] = _SOURCE -DESCRIPTOR.message_types_by_name['KafkaSourceConfig'] = _KAFKASOURCECONFIG -DESCRIPTOR.enum_types_by_name['SourceType'] = _SOURCETYPE -_sym_db.RegisterFileDescriptor(DESCRIPTOR) - -Source = _reflection.GeneratedProtocolMessageType('Source', (_message.Message,), { - 'DESCRIPTOR' : _SOURCE, - '__module__' : 'feast.core.Source_pb2' - # @@protoc_insertion_point(class_scope:feast.core.Source) - }) -_sym_db.RegisterMessage(Source) - -KafkaSourceConfig = _reflection.GeneratedProtocolMessageType('KafkaSourceConfig', (_message.Message,), { - 'DESCRIPTOR' : _KAFKASOURCECONFIG, - '__module__' : 'feast.core.Source_pb2' - # @@protoc_insertion_point(class_scope:feast.core.KafkaSourceConfig) - }) -_sym_db.RegisterMessage(KafkaSourceConfig) - - -DESCRIPTOR._options = None -# @@protoc_insertion_point(module_scope) diff --git a/sdk/python/feast/core/Source_pb2.pyi b/sdk/python/feast/core/Source_pb2.pyi deleted file mode 100644 index 0521ac34f80..00000000000 --- a/sdk/python/feast/core/Source_pb2.pyi +++ /dev/null @@ -1,83 +0,0 @@ -# @generated by generate_proto_mypy_stubs.py. Do not edit! -import sys -from google.protobuf.descriptor import ( - Descriptor as google___protobuf___descriptor___Descriptor, - EnumDescriptor as google___protobuf___descriptor___EnumDescriptor, -) - -from google.protobuf.message import ( - Message as google___protobuf___message___Message, -) - -from typing import ( - List as typing___List, - Optional as typing___Optional, - Text as typing___Text, - Tuple as typing___Tuple, - cast as typing___cast, -) - -from typing_extensions import ( - Literal as typing_extensions___Literal, -) - - -class SourceType(int): - DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... - @classmethod - def Name(cls, number: int) -> str: ... - @classmethod - def Value(cls, name: str) -> SourceType: ... - @classmethod - def keys(cls) -> typing___List[str]: ... - @classmethod - def values(cls) -> typing___List[SourceType]: ... - @classmethod - def items(cls) -> typing___List[typing___Tuple[str, SourceType]]: ... - INVALID = typing___cast(SourceType, 0) - KAFKA = typing___cast(SourceType, 1) -INVALID = typing___cast(SourceType, 0) -KAFKA = typing___cast(SourceType, 1) - -class Source(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - type = ... # type: SourceType - - @property - def kafka_source_config(self) -> KafkaSourceConfig: ... - - def __init__(self, - *, - type : typing___Optional[SourceType] = None, - kafka_source_config : typing___Optional[KafkaSourceConfig] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> Source: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"kafka_source_config",u"source_config"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"kafka_source_config",u"source_config",u"type"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"kafka_source_config",b"kafka_source_config",u"source_config",b"source_config"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"kafka_source_config",b"kafka_source_config",u"source_config",b"source_config",u"type",b"type"]) -> None: ... - def WhichOneof(self, oneof_group: typing_extensions___Literal[u"source_config",b"source_config"]) -> typing_extensions___Literal["kafka_source_config"]: ... - -class KafkaSourceConfig(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - bootstrap_servers = ... # type: typing___Text - topic = ... # type: typing___Text - - def __init__(self, - *, - bootstrap_servers : typing___Optional[typing___Text] = None, - topic : typing___Optional[typing___Text] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> KafkaSourceConfig: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"bootstrap_servers",u"topic"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"bootstrap_servers",b"bootstrap_servers",u"topic",b"topic"]) -> None: ... diff --git a/sdk/python/feast/core/Source_pb2_grpc.py b/sdk/python/feast/core/Source_pb2_grpc.py deleted file mode 100644 index a89435267cb..00000000000 --- a/sdk/python/feast/core/Source_pb2_grpc.py +++ /dev/null @@ -1,3 +0,0 @@ -# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT! -import grpc - diff --git a/sdk/python/feast/core/Store_pb2.py b/sdk/python/feast/core/Store_pb2.py deleted file mode 100644 index 716a597b9a3..00000000000 --- a/sdk/python/feast/core/Store_pb2.py +++ /dev/null @@ -1,346 +0,0 @@ -# -*- coding: utf-8 -*- -# Generated by the protocol buffer compiler. DO NOT EDIT! -# source: feast/core/Store.proto - -import sys -_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1')) -from google.protobuf import descriptor as _descriptor -from google.protobuf import message as _message -from google.protobuf import reflection as _reflection -from google.protobuf import symbol_database as _symbol_database -# @@protoc_insertion_point(imports) - -_sym_db = _symbol_database.Default() - - - - -DESCRIPTOR = _descriptor.FileDescriptor( - name='feast/core/Store.proto', - package='feast.core', - syntax='proto3', - serialized_options=_b('\n\nfeast.coreB\nStoreProtoZ/github.com/gojek/feast/sdk/go/protos/feast/core'), - serialized_pb=_b('\n\x16\x66\x65\x61st/core/Store.proto\x12\nfeast.core\"\xca\x04\n\x05Store\x12\x0c\n\x04name\x18\x01 \x01(\t\x12)\n\x04type\x18\x02 \x01(\x0e\x32\x1b.feast.core.Store.StoreType\x12\x35\n\rsubscriptions\x18\x04 \x03(\x0b\x32\x1e.feast.core.Store.Subscription\x12\x35\n\x0credis_config\x18\x0b \x01(\x0b\x32\x1d.feast.core.Store.RedisConfigH\x00\x12;\n\x0f\x62igquery_config\x18\x0c \x01(\x0b\x32 .feast.core.Store.BigQueryConfigH\x00\x12=\n\x10\x63\x61ssandra_config\x18\r \x01(\x0b\x32!.feast.core.Store.CassandraConfigH\x00\x1a)\n\x0bRedisConfig\x12\x0c\n\x04host\x18\x01 \x01(\t\x12\x0c\n\x04port\x18\x02 \x01(\x05\x1a\x38\n\x0e\x42igQueryConfig\x12\x12\n\nproject_id\x18\x01 \x01(\t\x12\x12\n\ndataset_id\x18\x02 \x01(\t\x1a-\n\x0f\x43\x61ssandraConfig\x12\x0c\n\x04host\x18\x01 \x01(\t\x12\x0c\n\x04port\x18\x02 \x01(\x05\x1a>\n\x0cSubscription\x12\x0f\n\x07project\x18\x03 \x01(\t\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0f\n\x07version\x18\x02 \x01(\t\"@\n\tStoreType\x12\x0b\n\x07INVALID\x10\x00\x12\t\n\x05REDIS\x10\x01\x12\x0c\n\x08\x42IGQUERY\x10\x02\x12\r\n\tCASSANDRA\x10\x03\x42\x08\n\x06\x63onfigBI\n\nfeast.coreB\nStoreProtoZ/github.com/gojek/feast/sdk/go/protos/feast/coreb\x06proto3') -) - - - -_STORE_STORETYPE = _descriptor.EnumDescriptor( - name='StoreType', - full_name='feast.core.Store.StoreType', - filename=None, - file=DESCRIPTOR, - values=[ - _descriptor.EnumValueDescriptor( - name='INVALID', index=0, number=0, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='REDIS', index=1, number=1, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='BIGQUERY', index=2, number=2, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='CASSANDRA', index=3, number=3, - serialized_options=None, - type=None), - ], - containing_type=None, - serialized_options=None, - serialized_start=551, - serialized_end=615, -) -_sym_db.RegisterEnumDescriptor(_STORE_STORETYPE) - - -_STORE_REDISCONFIG = _descriptor.Descriptor( - name='RedisConfig', - full_name='feast.core.Store.RedisConfig', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='host', full_name='feast.core.Store.RedisConfig.host', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='port', full_name='feast.core.Store.RedisConfig.port', index=1, - number=2, type=5, cpp_type=1, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=339, - serialized_end=380, -) - -_STORE_BIGQUERYCONFIG = _descriptor.Descriptor( - name='BigQueryConfig', - full_name='feast.core.Store.BigQueryConfig', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='project_id', full_name='feast.core.Store.BigQueryConfig.project_id', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='dataset_id', full_name='feast.core.Store.BigQueryConfig.dataset_id', index=1, - number=2, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=382, - serialized_end=438, -) - -_STORE_CASSANDRACONFIG = _descriptor.Descriptor( - name='CassandraConfig', - full_name='feast.core.Store.CassandraConfig', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='host', full_name='feast.core.Store.CassandraConfig.host', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='port', full_name='feast.core.Store.CassandraConfig.port', index=1, - number=2, type=5, cpp_type=1, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=440, - serialized_end=485, -) - -_STORE_SUBSCRIPTION = _descriptor.Descriptor( - name='Subscription', - full_name='feast.core.Store.Subscription', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='project', full_name='feast.core.Store.Subscription.project', index=0, - number=3, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='name', full_name='feast.core.Store.Subscription.name', index=1, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='version', full_name='feast.core.Store.Subscription.version', index=2, - number=2, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=487, - serialized_end=549, -) - -_STORE = _descriptor.Descriptor( - name='Store', - full_name='feast.core.Store', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='name', full_name='feast.core.Store.name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='type', full_name='feast.core.Store.type', index=1, - number=2, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='subscriptions', full_name='feast.core.Store.subscriptions', index=2, - number=4, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='redis_config', full_name='feast.core.Store.redis_config', index=3, - number=11, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='bigquery_config', full_name='feast.core.Store.bigquery_config', index=4, - number=12, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='cassandra_config', full_name='feast.core.Store.cassandra_config', index=5, - number=13, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[_STORE_REDISCONFIG, _STORE_BIGQUERYCONFIG, _STORE_CASSANDRACONFIG, _STORE_SUBSCRIPTION, ], - enum_types=[ - _STORE_STORETYPE, - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - _descriptor.OneofDescriptor( - name='config', full_name='feast.core.Store.config', - index=0, containing_type=None, fields=[]), - ], - serialized_start=39, - serialized_end=625, -) - -_STORE_REDISCONFIG.containing_type = _STORE -_STORE_BIGQUERYCONFIG.containing_type = _STORE -_STORE_CASSANDRACONFIG.containing_type = _STORE -_STORE_SUBSCRIPTION.containing_type = _STORE -_STORE.fields_by_name['type'].enum_type = _STORE_STORETYPE -_STORE.fields_by_name['subscriptions'].message_type = _STORE_SUBSCRIPTION -_STORE.fields_by_name['redis_config'].message_type = _STORE_REDISCONFIG -_STORE.fields_by_name['bigquery_config'].message_type = _STORE_BIGQUERYCONFIG -_STORE.fields_by_name['cassandra_config'].message_type = _STORE_CASSANDRACONFIG -_STORE_STORETYPE.containing_type = _STORE -_STORE.oneofs_by_name['config'].fields.append( - _STORE.fields_by_name['redis_config']) -_STORE.fields_by_name['redis_config'].containing_oneof = _STORE.oneofs_by_name['config'] -_STORE.oneofs_by_name['config'].fields.append( - _STORE.fields_by_name['bigquery_config']) -_STORE.fields_by_name['bigquery_config'].containing_oneof = _STORE.oneofs_by_name['config'] -_STORE.oneofs_by_name['config'].fields.append( - _STORE.fields_by_name['cassandra_config']) -_STORE.fields_by_name['cassandra_config'].containing_oneof = _STORE.oneofs_by_name['config'] -DESCRIPTOR.message_types_by_name['Store'] = _STORE -_sym_db.RegisterFileDescriptor(DESCRIPTOR) - -Store = _reflection.GeneratedProtocolMessageType('Store', (_message.Message,), { - - 'RedisConfig' : _reflection.GeneratedProtocolMessageType('RedisConfig', (_message.Message,), { - 'DESCRIPTOR' : _STORE_REDISCONFIG, - '__module__' : 'feast.core.Store_pb2' - # @@protoc_insertion_point(class_scope:feast.core.Store.RedisConfig) - }) - , - - 'BigQueryConfig' : _reflection.GeneratedProtocolMessageType('BigQueryConfig', (_message.Message,), { - 'DESCRIPTOR' : _STORE_BIGQUERYCONFIG, - '__module__' : 'feast.core.Store_pb2' - # @@protoc_insertion_point(class_scope:feast.core.Store.BigQueryConfig) - }) - , - - 'CassandraConfig' : _reflection.GeneratedProtocolMessageType('CassandraConfig', (_message.Message,), { - 'DESCRIPTOR' : _STORE_CASSANDRACONFIG, - '__module__' : 'feast.core.Store_pb2' - # @@protoc_insertion_point(class_scope:feast.core.Store.CassandraConfig) - }) - , - - 'Subscription' : _reflection.GeneratedProtocolMessageType('Subscription', (_message.Message,), { - 'DESCRIPTOR' : _STORE_SUBSCRIPTION, - '__module__' : 'feast.core.Store_pb2' - # @@protoc_insertion_point(class_scope:feast.core.Store.Subscription) - }) - , - 'DESCRIPTOR' : _STORE, - '__module__' : 'feast.core.Store_pb2' - # @@protoc_insertion_point(class_scope:feast.core.Store) - }) -_sym_db.RegisterMessage(Store) -_sym_db.RegisterMessage(Store.RedisConfig) -_sym_db.RegisterMessage(Store.BigQueryConfig) -_sym_db.RegisterMessage(Store.CassandraConfig) -_sym_db.RegisterMessage(Store.Subscription) - - -DESCRIPTOR._options = None -# @@protoc_insertion_point(module_scope) diff --git a/sdk/python/feast/core/Store_pb2.pyi b/sdk/python/feast/core/Store_pb2.pyi deleted file mode 100644 index 541bcd329bb..00000000000 --- a/sdk/python/feast/core/Store_pb2.pyi +++ /dev/null @@ -1,165 +0,0 @@ -# @generated by generate_proto_mypy_stubs.py. Do not edit! -import sys -from google.protobuf.descriptor import ( - Descriptor as google___protobuf___descriptor___Descriptor, - EnumDescriptor as google___protobuf___descriptor___EnumDescriptor, -) - -from google.protobuf.internal.containers import ( - RepeatedCompositeFieldContainer as google___protobuf___internal___containers___RepeatedCompositeFieldContainer, -) - -from google.protobuf.message import ( - Message as google___protobuf___message___Message, -) - -from typing import ( - Iterable as typing___Iterable, - List as typing___List, - Optional as typing___Optional, - Text as typing___Text, - Tuple as typing___Tuple, - cast as typing___cast, -) - -from typing_extensions import ( - Literal as typing_extensions___Literal, -) - - -class Store(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class StoreType(int): - DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... - @classmethod - def Name(cls, number: int) -> str: ... - @classmethod - def Value(cls, name: str) -> Store.StoreType: ... - @classmethod - def keys(cls) -> typing___List[str]: ... - @classmethod - def values(cls) -> typing___List[Store.StoreType]: ... - @classmethod - def items(cls) -> typing___List[typing___Tuple[str, Store.StoreType]]: ... - INVALID = typing___cast(Store.StoreType, 0) - REDIS = typing___cast(Store.StoreType, 1) - BIGQUERY = typing___cast(Store.StoreType, 2) - CASSANDRA = typing___cast(Store.StoreType, 3) - INVALID = typing___cast(Store.StoreType, 0) - REDIS = typing___cast(Store.StoreType, 1) - BIGQUERY = typing___cast(Store.StoreType, 2) - CASSANDRA = typing___cast(Store.StoreType, 3) - - class RedisConfig(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - host = ... # type: typing___Text - port = ... # type: int - - def __init__(self, - *, - host : typing___Optional[typing___Text] = None, - port : typing___Optional[int] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> Store.RedisConfig: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"host",u"port"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"host",b"host",u"port",b"port"]) -> None: ... - - class BigQueryConfig(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - project_id = ... # type: typing___Text - dataset_id = ... # type: typing___Text - - def __init__(self, - *, - project_id : typing___Optional[typing___Text] = None, - dataset_id : typing___Optional[typing___Text] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> Store.BigQueryConfig: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"dataset_id",u"project_id"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"dataset_id",b"dataset_id",u"project_id",b"project_id"]) -> None: ... - - class CassandraConfig(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - host = ... # type: typing___Text - port = ... # type: int - - def __init__(self, - *, - host : typing___Optional[typing___Text] = None, - port : typing___Optional[int] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> Store.CassandraConfig: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"host",u"port"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"host",b"host",u"port",b"port"]) -> None: ... - - class Subscription(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - project = ... # type: typing___Text - name = ... # type: typing___Text - version = ... # type: typing___Text - - def __init__(self, - *, - project : typing___Optional[typing___Text] = None, - name : typing___Optional[typing___Text] = None, - version : typing___Optional[typing___Text] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> Store.Subscription: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"name",u"project",u"version"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"name",b"name",u"project",b"project",u"version",b"version"]) -> None: ... - - name = ... # type: typing___Text - type = ... # type: Store.StoreType - - @property - def subscriptions(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[Store.Subscription]: ... - - @property - def redis_config(self) -> Store.RedisConfig: ... - - @property - def bigquery_config(self) -> Store.BigQueryConfig: ... - - @property - def cassandra_config(self) -> Store.CassandraConfig: ... - - def __init__(self, - *, - name : typing___Optional[typing___Text] = None, - type : typing___Optional[Store.StoreType] = None, - subscriptions : typing___Optional[typing___Iterable[Store.Subscription]] = None, - redis_config : typing___Optional[Store.RedisConfig] = None, - bigquery_config : typing___Optional[Store.BigQueryConfig] = None, - cassandra_config : typing___Optional[Store.CassandraConfig] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> Store: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"bigquery_config",u"cassandra_config",u"config",u"redis_config"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"bigquery_config",u"cassandra_config",u"config",u"name",u"redis_config",u"subscriptions",u"type"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"bigquery_config",b"bigquery_config",u"cassandra_config",b"cassandra_config",u"config",b"config",u"redis_config",b"redis_config"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"bigquery_config",b"bigquery_config",u"cassandra_config",b"cassandra_config",u"config",b"config",u"name",b"name",u"redis_config",b"redis_config",u"subscriptions",b"subscriptions",u"type",b"type"]) -> None: ... - def WhichOneof(self, oneof_group: typing_extensions___Literal[u"config",b"config"]) -> typing_extensions___Literal["redis_config","bigquery_config","cassandra_config"]: ... diff --git a/sdk/python/feast/core/Store_pb2_grpc.py b/sdk/python/feast/core/Store_pb2_grpc.py deleted file mode 100644 index a89435267cb..00000000000 --- a/sdk/python/feast/core/Store_pb2_grpc.py +++ /dev/null @@ -1,3 +0,0 @@ -# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT! -import grpc - diff --git a/sdk/python/feast/core/__init__.py b/sdk/python/feast/core/__init__.py deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/sdk/python/feast/entity.py b/sdk/python/feast/entity.py index 795758bc417..5f823a754a0 100644 --- a/sdk/python/feast/entity.py +++ b/sdk/python/feast/entity.py @@ -12,10 +12,10 @@ # See the License for the specific language governing permissions and # limitations under the License. -from feast.value_type import ValueType from feast.core.FeatureSet_pb2 import EntitySpec as EntityProto -from feast.types import Value_pb2 as ValueTypeProto from feast.field import Field +from feast.types import Value_pb2 as ValueTypeProto +from feast.value_type import ValueType class Entity(Field): diff --git a/sdk/python/feast/feature.py b/sdk/python/feast/feature.py index c7e3d7af8b2..c9fc1cbff40 100644 --- a/sdk/python/feast/feature.py +++ b/sdk/python/feast/feature.py @@ -12,10 +12,10 @@ # See the License for the specific language governing permissions and # limitations under the License. -from feast.value_type import ValueType from feast.core.FeatureSet_pb2 import FeatureSpec as FeatureProto -from feast.types import Value_pb2 as ValueTypeProto from feast.field import Field +from feast.types import Value_pb2 as ValueTypeProto +from feast.value_type import ValueType class Feature(Field): diff --git a/sdk/python/feast/feature_set.py b/sdk/python/feast/feature_set.py index c47c51e5a21..4ebfecf1675 100644 --- a/sdk/python/feast/feature_set.py +++ b/sdk/python/feast/feature_set.py @@ -14,11 +14,16 @@ from collections import OrderedDict -from typing import Dict -from typing import List, Optional +from typing import Dict, List, Optional import pandas as pd import pyarrow as pa +from google.protobuf import json_format +from google.protobuf.duration_pb2 import Duration +from google.protobuf.json_format import MessageToJson +from pandas.api.types import is_datetime64_ns_dtype +from pyarrow.lib import TimestampType + from feast.core.FeatureSet_pb2 import FeatureSet as FeatureSetProto from feast.core.FeatureSet_pb2 import FeatureSetMeta as FeatureSetMetaProto from feast.core.FeatureSet_pb2 import FeatureSetSpec as FeatureSetSpecProto @@ -26,18 +31,11 @@ from feast.feature import Feature, Field from feast.loaders import yaml as feast_yaml from feast.source import Source -from feast.type_map import DATETIME_COLUMN -from feast.type_map import pa_to_feast_value_type -from feast.type_map import python_type_to_feast_value_type -from google.protobuf import json_format -from feast.core.FeatureSet_pb2 import FeatureSetSpec as FeatureSetSpecProto -from feast.core.FeatureSet_pb2 import FeatureSetMeta as FeatureSetMetaProto -from feast.core.FeatureSet_pb2 import FeatureSet as FeatureSetProto -from google.protobuf.duration_pb2 import Duration -from feast.type_map import python_type_to_feast_value_type -from google.protobuf.json_format import MessageToJson -from pandas.api.types import is_datetime64_ns_dtype -from pyarrow.lib import TimestampType +from feast.type_map import ( + DATETIME_COLUMN, + pa_to_feast_value_type, + python_type_to_feast_value_type, +) class FeatureSet: @@ -639,7 +637,7 @@ def get_kafka_source_brokers(self) -> str: """ Get the broker list for the source in this feature set """ - if self.source and self.source.source_type is "Kafka": + if self.source and self.source.source_type == "Kafka": return self.source.brokers raise Exception("Source type could not be identified") diff --git a/sdk/python/feast/job.py b/sdk/python/feast/job.py index f849f6630da..ab65da74459 100644 --- a/sdk/python/feast/job.py +++ b/sdk/python/feast/job.py @@ -1,19 +1,18 @@ import tempfile import time from datetime import datetime, timedelta -from typing import Iterable from urllib.parse import urlparse import fastavro import pandas as pd from google.cloud import storage -from feast.serving.ServingService_pb2 import GetJobRequest from feast.serving.ServingService_pb2 import ( - Job as JobProto, - JOB_STATUS_DONE, DATA_FORMAT_AVRO, + JOB_STATUS_DONE, + GetJobRequest, ) +from feast.serving.ServingService_pb2 import Job as JobProto from feast.serving.ServingService_pb2_grpc import ServingServiceStub # Maximum no of seconds to wait until the jobs status is DONE in Feast diff --git a/sdk/python/feast/loaders/abstract_producer.py b/sdk/python/feast/loaders/abstract_producer.py index d0ddabf1e56..6030d14ecc0 100644 --- a/sdk/python/feast/loaders/abstract_producer.py +++ b/sdk/python/feast/loaders/abstract_producer.py @@ -33,7 +33,7 @@ def __init__(self, brokers: str, row_count: int, disable_progress_bar: bool): total=row_count, unit="rows", smoothing=0, disable=disable_progress_bar ) - def produce(self, topic: str, data: str): + def produce(self, topic: str, data: bytes): message = "{} should implement a produce method".format(self.__class__.__name__) raise NotImplementedError(message) @@ -227,6 +227,6 @@ def get_producer( """ try: return ConfluentProducer(brokers, row_count, disable_progress_bar) - except ImportError as e: + except ImportError: print("Unable to import confluent-kafka, falling back to kafka-python") return KafkaPythonProducer(brokers, row_count, disable_progress_bar) diff --git a/sdk/python/feast/loaders/file.py b/sdk/python/feast/loaders/file.py index bb050c07c6d..52cc8ae7dc8 100644 --- a/sdk/python/feast/loaders/file.py +++ b/sdk/python/feast/loaders/file.py @@ -19,7 +19,7 @@ import uuid from datetime import datetime from typing import List, Optional, Tuple, Union -from urllib.parse import urlparse, ParseResult +from urllib.parse import ParseResult, urlparse import pandas as pd from google.cloud import storage @@ -63,13 +63,14 @@ def export_source_to_staging_location( # Prepare Avro file to be exported to staging location if isinstance(source, pd.DataFrame): # DataFrame provided as a source + uri_path = None # type: Optional[str] if uri.scheme == "file": uri_path = uri.path - else: - uri_path = None # Remote gs staging location provided by serving - dir_path, file_name, source_path = export_dataframe_to_local(source, uri_path) + dir_path, file_name, source_path = export_dataframe_to_local( + df=source, dir_path=uri_path + ) elif urlparse(source).scheme in ["", "file"]: # Local file provided as a source dir_path = None @@ -108,7 +109,7 @@ def export_source_to_staging_location( ) # Clean up, remove local staging file - if isinstance(source, pd.DataFrame) and len(str(dir_path)) > 4: + if dir_path and isinstance(source, pd.DataFrame) and len(str(dir_path)) > 4: shutil.rmtree(dir_path) return [staging_location_uri.rstrip("/") + "/" + file_name] @@ -172,7 +173,7 @@ def upload_file_to_gcs(local_path: str, bucket: str, remote_path: str) -> None: remote_path (str): Path within GCS bucket to upload file to, includes file name. - + Returns: None: None diff --git a/sdk/python/feast/loaders/ingest.py b/sdk/python/feast/loaders/ingest.py index 95b699d0005..b4490f025c5 100644 --- a/sdk/python/feast/loaders/ingest.py +++ b/sdk/python/feast/loaders/ingest.py @@ -5,11 +5,12 @@ import pandas as pd import pyarrow.parquet as pq + from feast.constants import DATETIME_COLUMN from feast.feature_set import FeatureSet from feast.type_map import ( - pa_column_to_timestamp_proto_column, pa_column_to_proto_column, + pa_column_to_timestamp_proto_column, ) from feast.types import Field_pb2 as FieldProto from feast.types.FeatureRow_pb2 import FeatureRow diff --git a/sdk/python/feast/serving/ServingService_pb2.py b/sdk/python/feast/serving/ServingService_pb2.py deleted file mode 100644 index 9d0d55f2ab4..00000000000 --- a/sdk/python/feast/serving/ServingService_pb2.py +++ /dev/null @@ -1,971 +0,0 @@ -# -*- coding: utf-8 -*- -# Generated by the protocol buffer compiler. DO NOT EDIT! -# source: feast/serving/ServingService.proto - -import sys -_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1')) -from google.protobuf.internal import enum_type_wrapper -from google.protobuf import descriptor as _descriptor -from google.protobuf import message as _message -from google.protobuf import reflection as _reflection -from google.protobuf import symbol_database as _symbol_database -# @@protoc_insertion_point(imports) - -_sym_db = _symbol_database.Default() - - -from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2 -from google.protobuf import duration_pb2 as google_dot_protobuf_dot_duration__pb2 -from feast.types import Value_pb2 as feast_dot_types_dot_Value__pb2 - - -DESCRIPTOR = _descriptor.FileDescriptor( - name='feast/serving/ServingService.proto', - package='feast.serving', - syntax='proto3', - serialized_options=_b('\n\rfeast.servingB\017ServingAPIProtoZ2github.com/gojek/feast/sdk/go/protos/feast/serving'), - serialized_pb=_b('\n\"feast/serving/ServingService.proto\x12\rfeast.serving\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1egoogle/protobuf/duration.proto\x1a\x17\x66\x65\x61st/types/Value.proto\"\x1c\n\x1aGetFeastServingInfoRequest\"{\n\x1bGetFeastServingInfoResponse\x12\x0f\n\x07version\x18\x01 \x01(\t\x12-\n\x04type\x18\x02 \x01(\x0e\x32\x1f.feast.serving.FeastServingType\x12\x1c\n\x14job_staging_location\x18\n \x01(\t\"n\n\x10\x46\x65\x61tureReference\x12\x0f\n\x07project\x18\x01 \x01(\t\x12\x0c\n\x04name\x18\x02 \x01(\t\x12\x0f\n\x07version\x18\x03 \x01(\x05\x12*\n\x07max_age\x18\x04 \x01(\x0b\x32\x19.google.protobuf.Duration\"\x8e\x03\n\x18GetOnlineFeaturesRequest\x12\x31\n\x08\x66\x65\x61tures\x18\x04 \x03(\x0b\x32\x1f.feast.serving.FeatureReference\x12\x46\n\x0b\x65ntity_rows\x18\x02 \x03(\x0b\x32\x31.feast.serving.GetOnlineFeaturesRequest.EntityRow\x12!\n\x19omit_entities_in_response\x18\x03 \x01(\x08\x1a\xd3\x01\n\tEntityRow\x12\x34\n\x10\x65ntity_timestamp\x18\x01 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12M\n\x06\x66ields\x18\x02 \x03(\x0b\x32=.feast.serving.GetOnlineFeaturesRequest.EntityRow.FieldsEntry\x1a\x41\n\x0b\x46ieldsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12!\n\x05value\x18\x02 \x01(\x0b\x32\x12.feast.types.Value:\x02\x38\x01\"\x82\x01\n\x17GetBatchFeaturesRequest\x12\x31\n\x08\x66\x65\x61tures\x18\x03 \x03(\x0b\x32\x1f.feast.serving.FeatureReference\x12\x34\n\x0e\x64\x61taset_source\x18\x02 \x01(\x0b\x32\x1c.feast.serving.DatasetSource\"\x8c\x02\n\x19GetOnlineFeaturesResponse\x12J\n\x0c\x66ield_values\x18\x01 \x03(\x0b\x32\x34.feast.serving.GetOnlineFeaturesResponse.FieldValues\x1a\xa2\x01\n\x0b\x46ieldValues\x12P\n\x06\x66ields\x18\x01 \x03(\x0b\x32@.feast.serving.GetOnlineFeaturesResponse.FieldValues.FieldsEntry\x1a\x41\n\x0b\x46ieldsEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12!\n\x05value\x18\x02 \x01(\x0b\x32\x12.feast.types.Value:\x02\x38\x01\";\n\x18GetBatchFeaturesResponse\x12\x1f\n\x03job\x18\x01 \x01(\x0b\x32\x12.feast.serving.Job\"0\n\rGetJobRequest\x12\x1f\n\x03job\x18\x01 \x01(\x0b\x32\x12.feast.serving.Job\"1\n\x0eGetJobResponse\x12\x1f\n\x03job\x18\x01 \x01(\x0b\x32\x12.feast.serving.Job\"\xb3\x01\n\x03Job\x12\n\n\x02id\x18\x01 \x01(\t\x12$\n\x04type\x18\x02 \x01(\x0e\x32\x16.feast.serving.JobType\x12(\n\x06status\x18\x03 \x01(\x0e\x32\x18.feast.serving.JobStatus\x12\r\n\x05\x65rror\x18\x04 \x01(\t\x12\x11\n\tfile_uris\x18\x05 \x03(\t\x12.\n\x0b\x64\x61ta_format\x18\x06 \x01(\x0e\x32\x19.feast.serving.DataFormat\"\xb2\x01\n\rDatasetSource\x12>\n\x0b\x66ile_source\x18\x01 \x01(\x0b\x32\'.feast.serving.DatasetSource.FileSourceH\x00\x1aO\n\nFileSource\x12\x11\n\tfile_uris\x18\x01 \x03(\t\x12.\n\x0b\x64\x61ta_format\x18\x02 \x01(\x0e\x32\x19.feast.serving.DataFormatB\x10\n\x0e\x64\x61taset_source*o\n\x10\x46\x65\x61stServingType\x12\x1e\n\x1a\x46\x45\x41ST_SERVING_TYPE_INVALID\x10\x00\x12\x1d\n\x19\x46\x45\x41ST_SERVING_TYPE_ONLINE\x10\x01\x12\x1c\n\x18\x46\x45\x41ST_SERVING_TYPE_BATCH\x10\x02*6\n\x07JobType\x12\x14\n\x10JOB_TYPE_INVALID\x10\x00\x12\x15\n\x11JOB_TYPE_DOWNLOAD\x10\x01*h\n\tJobStatus\x12\x16\n\x12JOB_STATUS_INVALID\x10\x00\x12\x16\n\x12JOB_STATUS_PENDING\x10\x01\x12\x16\n\x12JOB_STATUS_RUNNING\x10\x02\x12\x13\n\x0fJOB_STATUS_DONE\x10\x03*;\n\nDataFormat\x12\x17\n\x13\x44\x41TA_FORMAT_INVALID\x10\x00\x12\x14\n\x10\x44\x41TA_FORMAT_AVRO\x10\x01\x32\x92\x03\n\x0eServingService\x12l\n\x13GetFeastServingInfo\x12).feast.serving.GetFeastServingInfoRequest\x1a*.feast.serving.GetFeastServingInfoResponse\x12\x66\n\x11GetOnlineFeatures\x12\'.feast.serving.GetOnlineFeaturesRequest\x1a(.feast.serving.GetOnlineFeaturesResponse\x12\x63\n\x10GetBatchFeatures\x12&.feast.serving.GetBatchFeaturesRequest\x1a\'.feast.serving.GetBatchFeaturesResponse\x12\x45\n\x06GetJob\x12\x1c.feast.serving.GetJobRequest\x1a\x1d.feast.serving.GetJobResponseBT\n\rfeast.servingB\x0fServingAPIProtoZ2github.com/gojek/feast/sdk/go/protos/feast/servingb\x06proto3') - , - dependencies=[google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,google_dot_protobuf_dot_duration__pb2.DESCRIPTOR,feast_dot_types_dot_Value__pb2.DESCRIPTOR,]) - -_FEASTSERVINGTYPE = _descriptor.EnumDescriptor( - name='FeastServingType', - full_name='feast.serving.FeastServingType', - filename=None, - file=DESCRIPTOR, - values=[ - _descriptor.EnumValueDescriptor( - name='FEAST_SERVING_TYPE_INVALID', index=0, number=0, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='FEAST_SERVING_TYPE_ONLINE', index=1, number=1, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='FEAST_SERVING_TYPE_BATCH', index=2, number=2, - serialized_options=None, - type=None), - ], - containing_type=None, - serialized_options=None, - serialized_start=1740, - serialized_end=1851, -) -_sym_db.RegisterEnumDescriptor(_FEASTSERVINGTYPE) - -FeastServingType = enum_type_wrapper.EnumTypeWrapper(_FEASTSERVINGTYPE) -_JOBTYPE = _descriptor.EnumDescriptor( - name='JobType', - full_name='feast.serving.JobType', - filename=None, - file=DESCRIPTOR, - values=[ - _descriptor.EnumValueDescriptor( - name='JOB_TYPE_INVALID', index=0, number=0, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='JOB_TYPE_DOWNLOAD', index=1, number=1, - serialized_options=None, - type=None), - ], - containing_type=None, - serialized_options=None, - serialized_start=1853, - serialized_end=1907, -) -_sym_db.RegisterEnumDescriptor(_JOBTYPE) - -JobType = enum_type_wrapper.EnumTypeWrapper(_JOBTYPE) -_JOBSTATUS = _descriptor.EnumDescriptor( - name='JobStatus', - full_name='feast.serving.JobStatus', - filename=None, - file=DESCRIPTOR, - values=[ - _descriptor.EnumValueDescriptor( - name='JOB_STATUS_INVALID', index=0, number=0, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='JOB_STATUS_PENDING', index=1, number=1, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='JOB_STATUS_RUNNING', index=2, number=2, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='JOB_STATUS_DONE', index=3, number=3, - serialized_options=None, - type=None), - ], - containing_type=None, - serialized_options=None, - serialized_start=1909, - serialized_end=2013, -) -_sym_db.RegisterEnumDescriptor(_JOBSTATUS) - -JobStatus = enum_type_wrapper.EnumTypeWrapper(_JOBSTATUS) -_DATAFORMAT = _descriptor.EnumDescriptor( - name='DataFormat', - full_name='feast.serving.DataFormat', - filename=None, - file=DESCRIPTOR, - values=[ - _descriptor.EnumValueDescriptor( - name='DATA_FORMAT_INVALID', index=0, number=0, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='DATA_FORMAT_AVRO', index=1, number=1, - serialized_options=None, - type=None), - ], - containing_type=None, - serialized_options=None, - serialized_start=2015, - serialized_end=2074, -) -_sym_db.RegisterEnumDescriptor(_DATAFORMAT) - -DataFormat = enum_type_wrapper.EnumTypeWrapper(_DATAFORMAT) -FEAST_SERVING_TYPE_INVALID = 0 -FEAST_SERVING_TYPE_ONLINE = 1 -FEAST_SERVING_TYPE_BATCH = 2 -JOB_TYPE_INVALID = 0 -JOB_TYPE_DOWNLOAD = 1 -JOB_STATUS_INVALID = 0 -JOB_STATUS_PENDING = 1 -JOB_STATUS_RUNNING = 2 -JOB_STATUS_DONE = 3 -DATA_FORMAT_INVALID = 0 -DATA_FORMAT_AVRO = 1 - - - -_GETFEASTSERVINGINFOREQUEST = _descriptor.Descriptor( - name='GetFeastServingInfoRequest', - full_name='feast.serving.GetFeastServingInfoRequest', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=143, - serialized_end=171, -) - - -_GETFEASTSERVINGINFORESPONSE = _descriptor.Descriptor( - name='GetFeastServingInfoResponse', - full_name='feast.serving.GetFeastServingInfoResponse', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='version', full_name='feast.serving.GetFeastServingInfoResponse.version', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='type', full_name='feast.serving.GetFeastServingInfoResponse.type', index=1, - number=2, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='job_staging_location', full_name='feast.serving.GetFeastServingInfoResponse.job_staging_location', index=2, - number=10, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=173, - serialized_end=296, -) - - -_FEATUREREFERENCE = _descriptor.Descriptor( - name='FeatureReference', - full_name='feast.serving.FeatureReference', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='project', full_name='feast.serving.FeatureReference.project', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='name', full_name='feast.serving.FeatureReference.name', index=1, - number=2, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='version', full_name='feast.serving.FeatureReference.version', index=2, - number=3, type=5, cpp_type=1, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='max_age', full_name='feast.serving.FeatureReference.max_age', index=3, - number=4, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=298, - serialized_end=408, -) - - -_GETONLINEFEATURESREQUEST_ENTITYROW_FIELDSENTRY = _descriptor.Descriptor( - name='FieldsEntry', - full_name='feast.serving.GetOnlineFeaturesRequest.EntityRow.FieldsEntry', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='key', full_name='feast.serving.GetOnlineFeaturesRequest.EntityRow.FieldsEntry.key', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='value', full_name='feast.serving.GetOnlineFeaturesRequest.EntityRow.FieldsEntry.value', index=1, - number=2, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=_b('8\001'), - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=744, - serialized_end=809, -) - -_GETONLINEFEATURESREQUEST_ENTITYROW = _descriptor.Descriptor( - name='EntityRow', - full_name='feast.serving.GetOnlineFeaturesRequest.EntityRow', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='entity_timestamp', full_name='feast.serving.GetOnlineFeaturesRequest.EntityRow.entity_timestamp', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='fields', full_name='feast.serving.GetOnlineFeaturesRequest.EntityRow.fields', index=1, - number=2, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[_GETONLINEFEATURESREQUEST_ENTITYROW_FIELDSENTRY, ], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=598, - serialized_end=809, -) - -_GETONLINEFEATURESREQUEST = _descriptor.Descriptor( - name='GetOnlineFeaturesRequest', - full_name='feast.serving.GetOnlineFeaturesRequest', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='features', full_name='feast.serving.GetOnlineFeaturesRequest.features', index=0, - number=4, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='entity_rows', full_name='feast.serving.GetOnlineFeaturesRequest.entity_rows', index=1, - number=2, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='omit_entities_in_response', full_name='feast.serving.GetOnlineFeaturesRequest.omit_entities_in_response', index=2, - number=3, type=8, cpp_type=7, label=1, - has_default_value=False, default_value=False, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[_GETONLINEFEATURESREQUEST_ENTITYROW, ], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=411, - serialized_end=809, -) - - -_GETBATCHFEATURESREQUEST = _descriptor.Descriptor( - name='GetBatchFeaturesRequest', - full_name='feast.serving.GetBatchFeaturesRequest', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='features', full_name='feast.serving.GetBatchFeaturesRequest.features', index=0, - number=3, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='dataset_source', full_name='feast.serving.GetBatchFeaturesRequest.dataset_source', index=1, - number=2, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=812, - serialized_end=942, -) - - -_GETONLINEFEATURESRESPONSE_FIELDVALUES_FIELDSENTRY = _descriptor.Descriptor( - name='FieldsEntry', - full_name='feast.serving.GetOnlineFeaturesResponse.FieldValues.FieldsEntry', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='key', full_name='feast.serving.GetOnlineFeaturesResponse.FieldValues.FieldsEntry.key', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='value', full_name='feast.serving.GetOnlineFeaturesResponse.FieldValues.FieldsEntry.value', index=1, - number=2, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=_b('8\001'), - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=744, - serialized_end=809, -) - -_GETONLINEFEATURESRESPONSE_FIELDVALUES = _descriptor.Descriptor( - name='FieldValues', - full_name='feast.serving.GetOnlineFeaturesResponse.FieldValues', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='fields', full_name='feast.serving.GetOnlineFeaturesResponse.FieldValues.fields', index=0, - number=1, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[_GETONLINEFEATURESRESPONSE_FIELDVALUES_FIELDSENTRY, ], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=1051, - serialized_end=1213, -) - -_GETONLINEFEATURESRESPONSE = _descriptor.Descriptor( - name='GetOnlineFeaturesResponse', - full_name='feast.serving.GetOnlineFeaturesResponse', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='field_values', full_name='feast.serving.GetOnlineFeaturesResponse.field_values', index=0, - number=1, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[_GETONLINEFEATURESRESPONSE_FIELDVALUES, ], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=945, - serialized_end=1213, -) - - -_GETBATCHFEATURESRESPONSE = _descriptor.Descriptor( - name='GetBatchFeaturesResponse', - full_name='feast.serving.GetBatchFeaturesResponse', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='job', full_name='feast.serving.GetBatchFeaturesResponse.job', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=1215, - serialized_end=1274, -) - - -_GETJOBREQUEST = _descriptor.Descriptor( - name='GetJobRequest', - full_name='feast.serving.GetJobRequest', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='job', full_name='feast.serving.GetJobRequest.job', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=1276, - serialized_end=1324, -) - - -_GETJOBRESPONSE = _descriptor.Descriptor( - name='GetJobResponse', - full_name='feast.serving.GetJobResponse', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='job', full_name='feast.serving.GetJobResponse.job', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=1326, - serialized_end=1375, -) - - -_JOB = _descriptor.Descriptor( - name='Job', - full_name='feast.serving.Job', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='id', full_name='feast.serving.Job.id', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='type', full_name='feast.serving.Job.type', index=1, - number=2, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='status', full_name='feast.serving.Job.status', index=2, - number=3, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='error', full_name='feast.serving.Job.error', index=3, - number=4, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='file_uris', full_name='feast.serving.Job.file_uris', index=4, - number=5, type=9, cpp_type=9, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='data_format', full_name='feast.serving.Job.data_format', index=5, - number=6, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=1378, - serialized_end=1557, -) - - -_DATASETSOURCE_FILESOURCE = _descriptor.Descriptor( - name='FileSource', - full_name='feast.serving.DatasetSource.FileSource', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='file_uris', full_name='feast.serving.DatasetSource.FileSource.file_uris', index=0, - number=1, type=9, cpp_type=9, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='data_format', full_name='feast.serving.DatasetSource.FileSource.data_format', index=1, - number=2, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=1641, - serialized_end=1720, -) - -_DATASETSOURCE = _descriptor.Descriptor( - name='DatasetSource', - full_name='feast.serving.DatasetSource', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='file_source', full_name='feast.serving.DatasetSource.file_source', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[_DATASETSOURCE_FILESOURCE, ], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - _descriptor.OneofDescriptor( - name='dataset_source', full_name='feast.serving.DatasetSource.dataset_source', - index=0, containing_type=None, fields=[]), - ], - serialized_start=1560, - serialized_end=1738, -) - -_GETFEASTSERVINGINFORESPONSE.fields_by_name['type'].enum_type = _FEASTSERVINGTYPE -_FEATUREREFERENCE.fields_by_name['max_age'].message_type = google_dot_protobuf_dot_duration__pb2._DURATION -_GETONLINEFEATURESREQUEST_ENTITYROW_FIELDSENTRY.fields_by_name['value'].message_type = feast_dot_types_dot_Value__pb2._VALUE -_GETONLINEFEATURESREQUEST_ENTITYROW_FIELDSENTRY.containing_type = _GETONLINEFEATURESREQUEST_ENTITYROW -_GETONLINEFEATURESREQUEST_ENTITYROW.fields_by_name['entity_timestamp'].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP -_GETONLINEFEATURESREQUEST_ENTITYROW.fields_by_name['fields'].message_type = _GETONLINEFEATURESREQUEST_ENTITYROW_FIELDSENTRY -_GETONLINEFEATURESREQUEST_ENTITYROW.containing_type = _GETONLINEFEATURESREQUEST -_GETONLINEFEATURESREQUEST.fields_by_name['features'].message_type = _FEATUREREFERENCE -_GETONLINEFEATURESREQUEST.fields_by_name['entity_rows'].message_type = _GETONLINEFEATURESREQUEST_ENTITYROW -_GETBATCHFEATURESREQUEST.fields_by_name['features'].message_type = _FEATUREREFERENCE -_GETBATCHFEATURESREQUEST.fields_by_name['dataset_source'].message_type = _DATASETSOURCE -_GETONLINEFEATURESRESPONSE_FIELDVALUES_FIELDSENTRY.fields_by_name['value'].message_type = feast_dot_types_dot_Value__pb2._VALUE -_GETONLINEFEATURESRESPONSE_FIELDVALUES_FIELDSENTRY.containing_type = _GETONLINEFEATURESRESPONSE_FIELDVALUES -_GETONLINEFEATURESRESPONSE_FIELDVALUES.fields_by_name['fields'].message_type = _GETONLINEFEATURESRESPONSE_FIELDVALUES_FIELDSENTRY -_GETONLINEFEATURESRESPONSE_FIELDVALUES.containing_type = _GETONLINEFEATURESRESPONSE -_GETONLINEFEATURESRESPONSE.fields_by_name['field_values'].message_type = _GETONLINEFEATURESRESPONSE_FIELDVALUES -_GETBATCHFEATURESRESPONSE.fields_by_name['job'].message_type = _JOB -_GETJOBREQUEST.fields_by_name['job'].message_type = _JOB -_GETJOBRESPONSE.fields_by_name['job'].message_type = _JOB -_JOB.fields_by_name['type'].enum_type = _JOBTYPE -_JOB.fields_by_name['status'].enum_type = _JOBSTATUS -_JOB.fields_by_name['data_format'].enum_type = _DATAFORMAT -_DATASETSOURCE_FILESOURCE.fields_by_name['data_format'].enum_type = _DATAFORMAT -_DATASETSOURCE_FILESOURCE.containing_type = _DATASETSOURCE -_DATASETSOURCE.fields_by_name['file_source'].message_type = _DATASETSOURCE_FILESOURCE -_DATASETSOURCE.oneofs_by_name['dataset_source'].fields.append( - _DATASETSOURCE.fields_by_name['file_source']) -_DATASETSOURCE.fields_by_name['file_source'].containing_oneof = _DATASETSOURCE.oneofs_by_name['dataset_source'] -DESCRIPTOR.message_types_by_name['GetFeastServingInfoRequest'] = _GETFEASTSERVINGINFOREQUEST -DESCRIPTOR.message_types_by_name['GetFeastServingInfoResponse'] = _GETFEASTSERVINGINFORESPONSE -DESCRIPTOR.message_types_by_name['FeatureReference'] = _FEATUREREFERENCE -DESCRIPTOR.message_types_by_name['GetOnlineFeaturesRequest'] = _GETONLINEFEATURESREQUEST -DESCRIPTOR.message_types_by_name['GetBatchFeaturesRequest'] = _GETBATCHFEATURESREQUEST -DESCRIPTOR.message_types_by_name['GetOnlineFeaturesResponse'] = _GETONLINEFEATURESRESPONSE -DESCRIPTOR.message_types_by_name['GetBatchFeaturesResponse'] = _GETBATCHFEATURESRESPONSE -DESCRIPTOR.message_types_by_name['GetJobRequest'] = _GETJOBREQUEST -DESCRIPTOR.message_types_by_name['GetJobResponse'] = _GETJOBRESPONSE -DESCRIPTOR.message_types_by_name['Job'] = _JOB -DESCRIPTOR.message_types_by_name['DatasetSource'] = _DATASETSOURCE -DESCRIPTOR.enum_types_by_name['FeastServingType'] = _FEASTSERVINGTYPE -DESCRIPTOR.enum_types_by_name['JobType'] = _JOBTYPE -DESCRIPTOR.enum_types_by_name['JobStatus'] = _JOBSTATUS -DESCRIPTOR.enum_types_by_name['DataFormat'] = _DATAFORMAT -_sym_db.RegisterFileDescriptor(DESCRIPTOR) - -GetFeastServingInfoRequest = _reflection.GeneratedProtocolMessageType('GetFeastServingInfoRequest', (_message.Message,), { - 'DESCRIPTOR' : _GETFEASTSERVINGINFOREQUEST, - '__module__' : 'feast.serving.ServingService_pb2' - # @@protoc_insertion_point(class_scope:feast.serving.GetFeastServingInfoRequest) - }) -_sym_db.RegisterMessage(GetFeastServingInfoRequest) - -GetFeastServingInfoResponse = _reflection.GeneratedProtocolMessageType('GetFeastServingInfoResponse', (_message.Message,), { - 'DESCRIPTOR' : _GETFEASTSERVINGINFORESPONSE, - '__module__' : 'feast.serving.ServingService_pb2' - # @@protoc_insertion_point(class_scope:feast.serving.GetFeastServingInfoResponse) - }) -_sym_db.RegisterMessage(GetFeastServingInfoResponse) - -FeatureReference = _reflection.GeneratedProtocolMessageType('FeatureReference', (_message.Message,), { - 'DESCRIPTOR' : _FEATUREREFERENCE, - '__module__' : 'feast.serving.ServingService_pb2' - # @@protoc_insertion_point(class_scope:feast.serving.FeatureReference) - }) -_sym_db.RegisterMessage(FeatureReference) - -GetOnlineFeaturesRequest = _reflection.GeneratedProtocolMessageType('GetOnlineFeaturesRequest', (_message.Message,), { - - 'EntityRow' : _reflection.GeneratedProtocolMessageType('EntityRow', (_message.Message,), { - - 'FieldsEntry' : _reflection.GeneratedProtocolMessageType('FieldsEntry', (_message.Message,), { - 'DESCRIPTOR' : _GETONLINEFEATURESREQUEST_ENTITYROW_FIELDSENTRY, - '__module__' : 'feast.serving.ServingService_pb2' - # @@protoc_insertion_point(class_scope:feast.serving.GetOnlineFeaturesRequest.EntityRow.FieldsEntry) - }) - , - 'DESCRIPTOR' : _GETONLINEFEATURESREQUEST_ENTITYROW, - '__module__' : 'feast.serving.ServingService_pb2' - # @@protoc_insertion_point(class_scope:feast.serving.GetOnlineFeaturesRequest.EntityRow) - }) - , - 'DESCRIPTOR' : _GETONLINEFEATURESREQUEST, - '__module__' : 'feast.serving.ServingService_pb2' - # @@protoc_insertion_point(class_scope:feast.serving.GetOnlineFeaturesRequest) - }) -_sym_db.RegisterMessage(GetOnlineFeaturesRequest) -_sym_db.RegisterMessage(GetOnlineFeaturesRequest.EntityRow) -_sym_db.RegisterMessage(GetOnlineFeaturesRequest.EntityRow.FieldsEntry) - -GetBatchFeaturesRequest = _reflection.GeneratedProtocolMessageType('GetBatchFeaturesRequest', (_message.Message,), { - 'DESCRIPTOR' : _GETBATCHFEATURESREQUEST, - '__module__' : 'feast.serving.ServingService_pb2' - # @@protoc_insertion_point(class_scope:feast.serving.GetBatchFeaturesRequest) - }) -_sym_db.RegisterMessage(GetBatchFeaturesRequest) - -GetOnlineFeaturesResponse = _reflection.GeneratedProtocolMessageType('GetOnlineFeaturesResponse', (_message.Message,), { - - 'FieldValues' : _reflection.GeneratedProtocolMessageType('FieldValues', (_message.Message,), { - - 'FieldsEntry' : _reflection.GeneratedProtocolMessageType('FieldsEntry', (_message.Message,), { - 'DESCRIPTOR' : _GETONLINEFEATURESRESPONSE_FIELDVALUES_FIELDSENTRY, - '__module__' : 'feast.serving.ServingService_pb2' - # @@protoc_insertion_point(class_scope:feast.serving.GetOnlineFeaturesResponse.FieldValues.FieldsEntry) - }) - , - 'DESCRIPTOR' : _GETONLINEFEATURESRESPONSE_FIELDVALUES, - '__module__' : 'feast.serving.ServingService_pb2' - # @@protoc_insertion_point(class_scope:feast.serving.GetOnlineFeaturesResponse.FieldValues) - }) - , - 'DESCRIPTOR' : _GETONLINEFEATURESRESPONSE, - '__module__' : 'feast.serving.ServingService_pb2' - # @@protoc_insertion_point(class_scope:feast.serving.GetOnlineFeaturesResponse) - }) -_sym_db.RegisterMessage(GetOnlineFeaturesResponse) -_sym_db.RegisterMessage(GetOnlineFeaturesResponse.FieldValues) -_sym_db.RegisterMessage(GetOnlineFeaturesResponse.FieldValues.FieldsEntry) - -GetBatchFeaturesResponse = _reflection.GeneratedProtocolMessageType('GetBatchFeaturesResponse', (_message.Message,), { - 'DESCRIPTOR' : _GETBATCHFEATURESRESPONSE, - '__module__' : 'feast.serving.ServingService_pb2' - # @@protoc_insertion_point(class_scope:feast.serving.GetBatchFeaturesResponse) - }) -_sym_db.RegisterMessage(GetBatchFeaturesResponse) - -GetJobRequest = _reflection.GeneratedProtocolMessageType('GetJobRequest', (_message.Message,), { - 'DESCRIPTOR' : _GETJOBREQUEST, - '__module__' : 'feast.serving.ServingService_pb2' - # @@protoc_insertion_point(class_scope:feast.serving.GetJobRequest) - }) -_sym_db.RegisterMessage(GetJobRequest) - -GetJobResponse = _reflection.GeneratedProtocolMessageType('GetJobResponse', (_message.Message,), { - 'DESCRIPTOR' : _GETJOBRESPONSE, - '__module__' : 'feast.serving.ServingService_pb2' - # @@protoc_insertion_point(class_scope:feast.serving.GetJobResponse) - }) -_sym_db.RegisterMessage(GetJobResponse) - -Job = _reflection.GeneratedProtocolMessageType('Job', (_message.Message,), { - 'DESCRIPTOR' : _JOB, - '__module__' : 'feast.serving.ServingService_pb2' - # @@protoc_insertion_point(class_scope:feast.serving.Job) - }) -_sym_db.RegisterMessage(Job) - -DatasetSource = _reflection.GeneratedProtocolMessageType('DatasetSource', (_message.Message,), { - - 'FileSource' : _reflection.GeneratedProtocolMessageType('FileSource', (_message.Message,), { - 'DESCRIPTOR' : _DATASETSOURCE_FILESOURCE, - '__module__' : 'feast.serving.ServingService_pb2' - # @@protoc_insertion_point(class_scope:feast.serving.DatasetSource.FileSource) - }) - , - 'DESCRIPTOR' : _DATASETSOURCE, - '__module__' : 'feast.serving.ServingService_pb2' - # @@protoc_insertion_point(class_scope:feast.serving.DatasetSource) - }) -_sym_db.RegisterMessage(DatasetSource) -_sym_db.RegisterMessage(DatasetSource.FileSource) - - -DESCRIPTOR._options = None -_GETONLINEFEATURESREQUEST_ENTITYROW_FIELDSENTRY._options = None -_GETONLINEFEATURESRESPONSE_FIELDVALUES_FIELDSENTRY._options = None - -_SERVINGSERVICE = _descriptor.ServiceDescriptor( - name='ServingService', - full_name='feast.serving.ServingService', - file=DESCRIPTOR, - index=0, - serialized_options=None, - serialized_start=2077, - serialized_end=2479, - methods=[ - _descriptor.MethodDescriptor( - name='GetFeastServingInfo', - full_name='feast.serving.ServingService.GetFeastServingInfo', - index=0, - containing_service=None, - input_type=_GETFEASTSERVINGINFOREQUEST, - output_type=_GETFEASTSERVINGINFORESPONSE, - serialized_options=None, - ), - _descriptor.MethodDescriptor( - name='GetOnlineFeatures', - full_name='feast.serving.ServingService.GetOnlineFeatures', - index=1, - containing_service=None, - input_type=_GETONLINEFEATURESREQUEST, - output_type=_GETONLINEFEATURESRESPONSE, - serialized_options=None, - ), - _descriptor.MethodDescriptor( - name='GetBatchFeatures', - full_name='feast.serving.ServingService.GetBatchFeatures', - index=2, - containing_service=None, - input_type=_GETBATCHFEATURESREQUEST, - output_type=_GETBATCHFEATURESRESPONSE, - serialized_options=None, - ), - _descriptor.MethodDescriptor( - name='GetJob', - full_name='feast.serving.ServingService.GetJob', - index=3, - containing_service=None, - input_type=_GETJOBREQUEST, - output_type=_GETJOBRESPONSE, - serialized_options=None, - ), -]) -_sym_db.RegisterServiceDescriptor(_SERVINGSERVICE) - -DESCRIPTOR.services_by_name['ServingService'] = _SERVINGSERVICE - -# @@protoc_insertion_point(module_scope) diff --git a/sdk/python/feast/serving/ServingService_pb2.pyi b/sdk/python/feast/serving/ServingService_pb2.pyi deleted file mode 100644 index e10245d6c7a..00000000000 --- a/sdk/python/feast/serving/ServingService_pb2.pyi +++ /dev/null @@ -1,465 +0,0 @@ -# @generated by generate_proto_mypy_stubs.py. Do not edit! -import sys -from feast.types.Value_pb2 import ( - Value as feast___types___Value_pb2___Value, -) - -from google.protobuf.descriptor import ( - Descriptor as google___protobuf___descriptor___Descriptor, - EnumDescriptor as google___protobuf___descriptor___EnumDescriptor, -) - -from google.protobuf.duration_pb2 import ( - Duration as google___protobuf___duration_pb2___Duration, -) - -from google.protobuf.internal.containers import ( - RepeatedCompositeFieldContainer as google___protobuf___internal___containers___RepeatedCompositeFieldContainer, - RepeatedScalarFieldContainer as google___protobuf___internal___containers___RepeatedScalarFieldContainer, -) - -from google.protobuf.message import ( - Message as google___protobuf___message___Message, -) - -from google.protobuf.timestamp_pb2 import ( - Timestamp as google___protobuf___timestamp_pb2___Timestamp, -) - -from typing import ( - Iterable as typing___Iterable, - List as typing___List, - Mapping as typing___Mapping, - MutableMapping as typing___MutableMapping, - Optional as typing___Optional, - Text as typing___Text, - Tuple as typing___Tuple, - cast as typing___cast, -) - -from typing_extensions import ( - Literal as typing_extensions___Literal, -) - - -class FeastServingType(int): - DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... - @classmethod - def Name(cls, number: int) -> str: ... - @classmethod - def Value(cls, name: str) -> FeastServingType: ... - @classmethod - def keys(cls) -> typing___List[str]: ... - @classmethod - def values(cls) -> typing___List[FeastServingType]: ... - @classmethod - def items(cls) -> typing___List[typing___Tuple[str, FeastServingType]]: ... - FEAST_SERVING_TYPE_INVALID = typing___cast(FeastServingType, 0) - FEAST_SERVING_TYPE_ONLINE = typing___cast(FeastServingType, 1) - FEAST_SERVING_TYPE_BATCH = typing___cast(FeastServingType, 2) -FEAST_SERVING_TYPE_INVALID = typing___cast(FeastServingType, 0) -FEAST_SERVING_TYPE_ONLINE = typing___cast(FeastServingType, 1) -FEAST_SERVING_TYPE_BATCH = typing___cast(FeastServingType, 2) - -class JobType(int): - DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... - @classmethod - def Name(cls, number: int) -> str: ... - @classmethod - def Value(cls, name: str) -> JobType: ... - @classmethod - def keys(cls) -> typing___List[str]: ... - @classmethod - def values(cls) -> typing___List[JobType]: ... - @classmethod - def items(cls) -> typing___List[typing___Tuple[str, JobType]]: ... - JOB_TYPE_INVALID = typing___cast(JobType, 0) - JOB_TYPE_DOWNLOAD = typing___cast(JobType, 1) -JOB_TYPE_INVALID = typing___cast(JobType, 0) -JOB_TYPE_DOWNLOAD = typing___cast(JobType, 1) - -class JobStatus(int): - DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... - @classmethod - def Name(cls, number: int) -> str: ... - @classmethod - def Value(cls, name: str) -> JobStatus: ... - @classmethod - def keys(cls) -> typing___List[str]: ... - @classmethod - def values(cls) -> typing___List[JobStatus]: ... - @classmethod - def items(cls) -> typing___List[typing___Tuple[str, JobStatus]]: ... - JOB_STATUS_INVALID = typing___cast(JobStatus, 0) - JOB_STATUS_PENDING = typing___cast(JobStatus, 1) - JOB_STATUS_RUNNING = typing___cast(JobStatus, 2) - JOB_STATUS_DONE = typing___cast(JobStatus, 3) -JOB_STATUS_INVALID = typing___cast(JobStatus, 0) -JOB_STATUS_PENDING = typing___cast(JobStatus, 1) -JOB_STATUS_RUNNING = typing___cast(JobStatus, 2) -JOB_STATUS_DONE = typing___cast(JobStatus, 3) - -class DataFormat(int): - DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... - @classmethod - def Name(cls, number: int) -> str: ... - @classmethod - def Value(cls, name: str) -> DataFormat: ... - @classmethod - def keys(cls) -> typing___List[str]: ... - @classmethod - def values(cls) -> typing___List[DataFormat]: ... - @classmethod - def items(cls) -> typing___List[typing___Tuple[str, DataFormat]]: ... - DATA_FORMAT_INVALID = typing___cast(DataFormat, 0) - DATA_FORMAT_AVRO = typing___cast(DataFormat, 1) -DATA_FORMAT_INVALID = typing___cast(DataFormat, 0) -DATA_FORMAT_AVRO = typing___cast(DataFormat, 1) - -class GetFeastServingInfoRequest(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - def __init__(self, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> GetFeastServingInfoRequest: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - -class GetFeastServingInfoResponse(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - version = ... # type: typing___Text - type = ... # type: FeastServingType - job_staging_location = ... # type: typing___Text - - def __init__(self, - *, - version : typing___Optional[typing___Text] = None, - type : typing___Optional[FeastServingType] = None, - job_staging_location : typing___Optional[typing___Text] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> GetFeastServingInfoResponse: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"job_staging_location",u"type",u"version"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"job_staging_location",b"job_staging_location",u"type",b"type",u"version",b"version"]) -> None: ... - -class FeatureReference(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - project = ... # type: typing___Text - name = ... # type: typing___Text - version = ... # type: int - - @property - def max_age(self) -> google___protobuf___duration_pb2___Duration: ... - - def __init__(self, - *, - project : typing___Optional[typing___Text] = None, - name : typing___Optional[typing___Text] = None, - version : typing___Optional[int] = None, - max_age : typing___Optional[google___protobuf___duration_pb2___Duration] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> FeatureReference: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"max_age"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"max_age",u"name",u"project",u"version"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"max_age",b"max_age"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"max_age",b"max_age",u"name",b"name",u"project",b"project",u"version",b"version"]) -> None: ... - -class GetOnlineFeaturesRequest(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class EntityRow(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class FieldsEntry(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - key = ... # type: typing___Text - - @property - def value(self) -> feast___types___Value_pb2___Value: ... - - def __init__(self, - *, - key : typing___Optional[typing___Text] = None, - value : typing___Optional[feast___types___Value_pb2___Value] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> GetOnlineFeaturesRequest.EntityRow.FieldsEntry: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"value"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"key",u"value"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"value",b"value"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"key",b"key",u"value",b"value"]) -> None: ... - - - @property - def entity_timestamp(self) -> google___protobuf___timestamp_pb2___Timestamp: ... - - @property - def fields(self) -> typing___MutableMapping[typing___Text, feast___types___Value_pb2___Value]: ... - - def __init__(self, - *, - entity_timestamp : typing___Optional[google___protobuf___timestamp_pb2___Timestamp] = None, - fields : typing___Optional[typing___Mapping[typing___Text, feast___types___Value_pb2___Value]] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> GetOnlineFeaturesRequest.EntityRow: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"entity_timestamp"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"entity_timestamp",u"fields"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"entity_timestamp",b"entity_timestamp"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"entity_timestamp",b"entity_timestamp",u"fields",b"fields"]) -> None: ... - - omit_entities_in_response = ... # type: bool - - @property - def features(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[FeatureReference]: ... - - @property - def entity_rows(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[GetOnlineFeaturesRequest.EntityRow]: ... - - def __init__(self, - *, - features : typing___Optional[typing___Iterable[FeatureReference]] = None, - entity_rows : typing___Optional[typing___Iterable[GetOnlineFeaturesRequest.EntityRow]] = None, - omit_entities_in_response : typing___Optional[bool] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> GetOnlineFeaturesRequest: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"entity_rows",u"features",u"omit_entities_in_response"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"entity_rows",b"entity_rows",u"features",b"features",u"omit_entities_in_response",b"omit_entities_in_response"]) -> None: ... - -class GetBatchFeaturesRequest(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - @property - def features(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[FeatureReference]: ... - - @property - def dataset_source(self) -> DatasetSource: ... - - def __init__(self, - *, - features : typing___Optional[typing___Iterable[FeatureReference]] = None, - dataset_source : typing___Optional[DatasetSource] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> GetBatchFeaturesRequest: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"dataset_source"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"dataset_source",u"features"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"dataset_source",b"dataset_source"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"dataset_source",b"dataset_source",u"features",b"features"]) -> None: ... - -class GetOnlineFeaturesResponse(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class FieldValues(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class FieldsEntry(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - key = ... # type: typing___Text - - @property - def value(self) -> feast___types___Value_pb2___Value: ... - - def __init__(self, - *, - key : typing___Optional[typing___Text] = None, - value : typing___Optional[feast___types___Value_pb2___Value] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> GetOnlineFeaturesResponse.FieldValues.FieldsEntry: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"value"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"key",u"value"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"value",b"value"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"key",b"key",u"value",b"value"]) -> None: ... - - - @property - def fields(self) -> typing___MutableMapping[typing___Text, feast___types___Value_pb2___Value]: ... - - def __init__(self, - *, - fields : typing___Optional[typing___Mapping[typing___Text, feast___types___Value_pb2___Value]] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> GetOnlineFeaturesResponse.FieldValues: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"fields"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"fields",b"fields"]) -> None: ... - - - @property - def field_values(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[GetOnlineFeaturesResponse.FieldValues]: ... - - def __init__(self, - *, - field_values : typing___Optional[typing___Iterable[GetOnlineFeaturesResponse.FieldValues]] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> GetOnlineFeaturesResponse: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"field_values"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"field_values",b"field_values"]) -> None: ... - -class GetBatchFeaturesResponse(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - @property - def job(self) -> Job: ... - - def __init__(self, - *, - job : typing___Optional[Job] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> GetBatchFeaturesResponse: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"job"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"job"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"job",b"job"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"job",b"job"]) -> None: ... - -class GetJobRequest(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - @property - def job(self) -> Job: ... - - def __init__(self, - *, - job : typing___Optional[Job] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> GetJobRequest: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"job"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"job"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"job",b"job"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"job",b"job"]) -> None: ... - -class GetJobResponse(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - @property - def job(self) -> Job: ... - - def __init__(self, - *, - job : typing___Optional[Job] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> GetJobResponse: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"job"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"job"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"job",b"job"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"job",b"job"]) -> None: ... - -class Job(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - id = ... # type: typing___Text - type = ... # type: JobType - status = ... # type: JobStatus - error = ... # type: typing___Text - file_uris = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] - data_format = ... # type: DataFormat - - def __init__(self, - *, - id : typing___Optional[typing___Text] = None, - type : typing___Optional[JobType] = None, - status : typing___Optional[JobStatus] = None, - error : typing___Optional[typing___Text] = None, - file_uris : typing___Optional[typing___Iterable[typing___Text]] = None, - data_format : typing___Optional[DataFormat] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> Job: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"data_format",u"error",u"file_uris",u"id",u"status",u"type"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"data_format",b"data_format",u"error",b"error",u"file_uris",b"file_uris",u"id",b"id",u"status",b"status",u"type",b"type"]) -> None: ... - -class DatasetSource(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class FileSource(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - file_uris = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] - data_format = ... # type: DataFormat - - def __init__(self, - *, - file_uris : typing___Optional[typing___Iterable[typing___Text]] = None, - data_format : typing___Optional[DataFormat] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> DatasetSource.FileSource: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"data_format",u"file_uris"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"data_format",b"data_format",u"file_uris",b"file_uris"]) -> None: ... - - - @property - def file_source(self) -> DatasetSource.FileSource: ... - - def __init__(self, - *, - file_source : typing___Optional[DatasetSource.FileSource] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> DatasetSource: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"dataset_source",u"file_source"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"dataset_source",u"file_source"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"dataset_source",b"dataset_source",u"file_source",b"file_source"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"dataset_source",b"dataset_source",u"file_source",b"file_source"]) -> None: ... - def WhichOneof(self, oneof_group: typing_extensions___Literal[u"dataset_source",b"dataset_source"]) -> typing_extensions___Literal["file_source"]: ... diff --git a/sdk/python/feast/serving/ServingService_pb2_grpc.py b/sdk/python/feast/serving/ServingService_pb2_grpc.py deleted file mode 100644 index c73f9c744a6..00000000000 --- a/sdk/python/feast/serving/ServingService_pb2_grpc.py +++ /dev/null @@ -1,104 +0,0 @@ -# Generated by the gRPC Python protocol compiler plugin. DO NOT EDIT! -import grpc - -from feast.serving import ServingService_pb2 as feast_dot_serving_dot_ServingService__pb2 - - -class ServingServiceStub(object): - # missing associated documentation comment in .proto file - pass - - def __init__(self, channel): - """Constructor. - - Args: - channel: A grpc.Channel. - """ - self.GetFeastServingInfo = channel.unary_unary( - '/feast.serving.ServingService/GetFeastServingInfo', - request_serializer=feast_dot_serving_dot_ServingService__pb2.GetFeastServingInfoRequest.SerializeToString, - response_deserializer=feast_dot_serving_dot_ServingService__pb2.GetFeastServingInfoResponse.FromString, - ) - self.GetOnlineFeatures = channel.unary_unary( - '/feast.serving.ServingService/GetOnlineFeatures', - request_serializer=feast_dot_serving_dot_ServingService__pb2.GetOnlineFeaturesRequest.SerializeToString, - response_deserializer=feast_dot_serving_dot_ServingService__pb2.GetOnlineFeaturesResponse.FromString, - ) - self.GetBatchFeatures = channel.unary_unary( - '/feast.serving.ServingService/GetBatchFeatures', - request_serializer=feast_dot_serving_dot_ServingService__pb2.GetBatchFeaturesRequest.SerializeToString, - response_deserializer=feast_dot_serving_dot_ServingService__pb2.GetBatchFeaturesResponse.FromString, - ) - self.GetJob = channel.unary_unary( - '/feast.serving.ServingService/GetJob', - request_serializer=feast_dot_serving_dot_ServingService__pb2.GetJobRequest.SerializeToString, - response_deserializer=feast_dot_serving_dot_ServingService__pb2.GetJobResponse.FromString, - ) - - -class ServingServiceServicer(object): - # missing associated documentation comment in .proto file - pass - - def GetFeastServingInfo(self, request, context): - """Get information about this Feast serving. - """ - context.set_code(grpc.StatusCode.UNIMPLEMENTED) - context.set_details('Method not implemented!') - raise NotImplementedError('Method not implemented!') - - def GetOnlineFeatures(self, request, context): - """Get online features synchronously. - """ - context.set_code(grpc.StatusCode.UNIMPLEMENTED) - context.set_details('Method not implemented!') - raise NotImplementedError('Method not implemented!') - - def GetBatchFeatures(self, request, context): - """Get batch features asynchronously. - - The client should check the status of the returned job periodically by - calling ReloadJob to determine if the job has completed successfully - or with an error. If the job completes successfully i.e. - status = JOB_STATUS_DONE with no error, then the client can check - the file_uris for the location to download feature values data. - The client is assumed to have access to these file URIs. - """ - context.set_code(grpc.StatusCode.UNIMPLEMENTED) - context.set_details('Method not implemented!') - raise NotImplementedError('Method not implemented!') - - def GetJob(self, request, context): - """Get the latest job status for batch feature retrieval. - """ - context.set_code(grpc.StatusCode.UNIMPLEMENTED) - context.set_details('Method not implemented!') - raise NotImplementedError('Method not implemented!') - - -def add_ServingServiceServicer_to_server(servicer, server): - rpc_method_handlers = { - 'GetFeastServingInfo': grpc.unary_unary_rpc_method_handler( - servicer.GetFeastServingInfo, - request_deserializer=feast_dot_serving_dot_ServingService__pb2.GetFeastServingInfoRequest.FromString, - response_serializer=feast_dot_serving_dot_ServingService__pb2.GetFeastServingInfoResponse.SerializeToString, - ), - 'GetOnlineFeatures': grpc.unary_unary_rpc_method_handler( - servicer.GetOnlineFeatures, - request_deserializer=feast_dot_serving_dot_ServingService__pb2.GetOnlineFeaturesRequest.FromString, - response_serializer=feast_dot_serving_dot_ServingService__pb2.GetOnlineFeaturesResponse.SerializeToString, - ), - 'GetBatchFeatures': grpc.unary_unary_rpc_method_handler( - servicer.GetBatchFeatures, - request_deserializer=feast_dot_serving_dot_ServingService__pb2.GetBatchFeaturesRequest.FromString, - response_serializer=feast_dot_serving_dot_ServingService__pb2.GetBatchFeaturesResponse.SerializeToString, - ), - 'GetJob': grpc.unary_unary_rpc_method_handler( - servicer.GetJob, - request_deserializer=feast_dot_serving_dot_ServingService__pb2.GetJobRequest.FromString, - response_serializer=feast_dot_serving_dot_ServingService__pb2.GetJobResponse.SerializeToString, - ), - } - generic_handler = grpc.method_handlers_generic_handler( - 'feast.serving.ServingService', rpc_method_handlers) - server.add_generic_rpc_handlers((generic_handler,)) diff --git a/sdk/python/feast/serving/__init__.py b/sdk/python/feast/serving/__init__.py deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/sdk/python/feast/source.py b/sdk/python/feast/source.py index 10c21210780..8e388376b3c 100644 --- a/sdk/python/feast/source.py +++ b/sdk/python/feast/source.py @@ -11,11 +11,9 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -from feast.core.Source_pb2 import ( - Source as SourceProto, - KafkaSourceConfig as KafkaSourceConfigProto, - SourceType as SourceTypeProto, -) +from feast.core.Source_pb2 import KafkaSourceConfig as KafkaSourceConfigProto +from feast.core.Source_pb2 import Source as SourceProto +from feast.core.Source_pb2 import SourceType as SourceTypeProto class Source: diff --git a/sdk/python/feast/storage/Redis_pb2.py b/sdk/python/feast/storage/Redis_pb2.py deleted file mode 100644 index 49b0b793781..00000000000 --- a/sdk/python/feast/storage/Redis_pb2.py +++ /dev/null @@ -1,81 +0,0 @@ -# -*- coding: utf-8 -*- -# Generated by the protocol buffer compiler. DO NOT EDIT! -# source: feast/storage/Redis.proto - -import sys -_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1')) -from google.protobuf import descriptor as _descriptor -from google.protobuf import message as _message -from google.protobuf import reflection as _reflection -from google.protobuf import symbol_database as _symbol_database -# @@protoc_insertion_point(imports) - -_sym_db = _symbol_database.Default() - - -from feast.types import Field_pb2 as feast_dot_types_dot_Field__pb2 - - -DESCRIPTOR = _descriptor.FileDescriptor( - name='feast/storage/Redis.proto', - package='feast.storage', - syntax='proto3', - serialized_options=_b('\n\rfeast.storageB\nRedisProtoZ2github.com/gojek/feast/sdk/go/protos/feast/storage'), - serialized_pb=_b('\n\x19\x66\x65\x61st/storage/Redis.proto\x12\rfeast.storage\x1a\x17\x66\x65\x61st/types/Field.proto\"E\n\x08RedisKey\x12\x13\n\x0b\x66\x65\x61ture_set\x18\x02 \x01(\t\x12$\n\x08\x65ntities\x18\x03 \x03(\x0b\x32\x12.feast.types.FieldBO\n\rfeast.storageB\nRedisProtoZ2github.com/gojek/feast/sdk/go/protos/feast/storageb\x06proto3') - , - dependencies=[feast_dot_types_dot_Field__pb2.DESCRIPTOR,]) - - - - -_REDISKEY = _descriptor.Descriptor( - name='RedisKey', - full_name='feast.storage.RedisKey', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='feature_set', full_name='feast.storage.RedisKey.feature_set', index=0, - number=2, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='entities', full_name='feast.storage.RedisKey.entities', index=1, - number=3, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=69, - serialized_end=138, -) - -_REDISKEY.fields_by_name['entities'].message_type = feast_dot_types_dot_Field__pb2._FIELD -DESCRIPTOR.message_types_by_name['RedisKey'] = _REDISKEY -_sym_db.RegisterFileDescriptor(DESCRIPTOR) - -RedisKey = _reflection.GeneratedProtocolMessageType('RedisKey', (_message.Message,), { - 'DESCRIPTOR' : _REDISKEY, - '__module__' : 'feast.storage.Redis_pb2' - # @@protoc_insertion_point(class_scope:feast.storage.RedisKey) - }) -_sym_db.RegisterMessage(RedisKey) - - -DESCRIPTOR._options = None -# @@protoc_insertion_point(module_scope) diff --git a/sdk/python/feast/storage/Redis_pb2.pyi b/sdk/python/feast/storage/Redis_pb2.pyi deleted file mode 100644 index 717aae79db2..00000000000 --- a/sdk/python/feast/storage/Redis_pb2.pyi +++ /dev/null @@ -1,49 +0,0 @@ -# @generated by generate_proto_mypy_stubs.py. Do not edit! -import sys -from feast.types.Field_pb2 import ( - Field as feast___types___Field_pb2___Field, -) - -from google.protobuf.descriptor import ( - Descriptor as google___protobuf___descriptor___Descriptor, -) - -from google.protobuf.internal.containers import ( - RepeatedCompositeFieldContainer as google___protobuf___internal___containers___RepeatedCompositeFieldContainer, -) - -from google.protobuf.message import ( - Message as google___protobuf___message___Message, -) - -from typing import ( - Iterable as typing___Iterable, - Optional as typing___Optional, - Text as typing___Text, -) - -from typing_extensions import ( - Literal as typing_extensions___Literal, -) - - -class RedisKey(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - feature_set = ... # type: typing___Text - - @property - def entities(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[feast___types___Field_pb2___Field]: ... - - def __init__(self, - *, - feature_set : typing___Optional[typing___Text] = None, - entities : typing___Optional[typing___Iterable[feast___types___Field_pb2___Field]] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> RedisKey: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"entities",u"feature_set"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"entities",b"entities",u"feature_set",b"feature_set"]) -> None: ... diff --git a/sdk/python/feast/storage/__init__.py b/sdk/python/feast/storage/__init__.py deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/sdk/python/feast/type_map.py b/sdk/python/feast/type_map.py index af019c3fdbd..8df0499239a 100644 --- a/sdk/python/feast/type_map.py +++ b/sdk/python/feast/type_map.py @@ -18,22 +18,24 @@ import numpy as np import pandas as pd import pyarrow as pa +from google.protobuf.timestamp_pb2 import Timestamp +from pyarrow.lib import TimestampType + from feast.constants import DATETIME_COLUMN -from feast.types import FeatureRow_pb2 as FeatureRowProto, Field_pb2 as FieldProto +from feast.types import FeatureRow_pb2 as FeatureRowProto +from feast.types import Field_pb2 as FieldProto from feast.types.Value_pb2 import ( - Value as ProtoValue, - ValueType as ProtoValueType, - Int64List, - Int32List, BoolList, BytesList, DoubleList, - StringList, FloatList, + Int32List, + Int64List, + StringList, ) +from feast.types.Value_pb2 import Value as ProtoValue +from feast.types.Value_pb2 import ValueType as ProtoValueType from feast.value_type import ValueType -from google.protobuf.timestamp_pb2 import Timestamp -from pyarrow.lib import TimestampType def python_type_to_feast_value_type( diff --git a/sdk/python/feast/types/FeatureRowExtended_pb2.py b/sdk/python/feast/types/FeatureRowExtended_pb2.py deleted file mode 100644 index e7372958168..00000000000 --- a/sdk/python/feast/types/FeatureRowExtended_pb2.py +++ /dev/null @@ -1,198 +0,0 @@ -# -*- coding: utf-8 -*- -# Generated by the protocol buffer compiler. DO NOT EDIT! -# source: feast/types/FeatureRowExtended.proto - -import sys -_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1')) -from google.protobuf import descriptor as _descriptor -from google.protobuf import message as _message -from google.protobuf import reflection as _reflection -from google.protobuf import symbol_database as _symbol_database -# @@protoc_insertion_point(imports) - -_sym_db = _symbol_database.Default() - - -from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2 -from feast.types import FeatureRow_pb2 as feast_dot_types_dot_FeatureRow__pb2 - - -DESCRIPTOR = _descriptor.FileDescriptor( - name='feast/types/FeatureRowExtended.proto', - package='feast.types', - syntax='proto3', - serialized_options=_b('\n\013feast.typesB\027FeatureRowExtendedProtoZ0github.com/gojek/feast/sdk/go/protos/feast/types'), - serialized_pb=_b('\n$feast/types/FeatureRowExtended.proto\x12\x0b\x66\x65\x61st.types\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x1c\x66\x65\x61st/types/FeatureRow.proto\"O\n\x05\x45rror\x12\r\n\x05\x63\x61use\x18\x01 \x01(\t\x12\x11\n\ttransform\x18\x02 \x01(\t\x12\x0f\n\x07message\x18\x03 \x01(\t\x12\x13\n\x0bstack_trace\x18\x04 \x01(\t\">\n\x07\x41ttempt\x12\x10\n\x08\x61ttempts\x18\x01 \x01(\x05\x12!\n\x05\x65rror\x18\x02 \x01(\x0b\x32\x12.feast.types.Error\"\x96\x01\n\x12\x46\x65\x61tureRowExtended\x12$\n\x03row\x18\x01 \x01(\x0b\x32\x17.feast.types.FeatureRow\x12*\n\x0clast_attempt\x18\x02 \x01(\x0b\x32\x14.feast.types.Attempt\x12.\n\nfirst_seen\x18\x03 \x01(\x0b\x32\x1a.google.protobuf.TimestampBX\n\x0b\x66\x65\x61st.typesB\x17\x46\x65\x61tureRowExtendedProtoZ0github.com/gojek/feast/sdk/go/protos/feast/typesb\x06proto3') - , - dependencies=[google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,feast_dot_types_dot_FeatureRow__pb2.DESCRIPTOR,]) - - - - -_ERROR = _descriptor.Descriptor( - name='Error', - full_name='feast.types.Error', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='cause', full_name='feast.types.Error.cause', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='transform', full_name='feast.types.Error.transform', index=1, - number=2, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='message', full_name='feast.types.Error.message', index=2, - number=3, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='stack_trace', full_name='feast.types.Error.stack_trace', index=3, - number=4, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=116, - serialized_end=195, -) - - -_ATTEMPT = _descriptor.Descriptor( - name='Attempt', - full_name='feast.types.Attempt', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='attempts', full_name='feast.types.Attempt.attempts', index=0, - number=1, type=5, cpp_type=1, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='error', full_name='feast.types.Attempt.error', index=1, - number=2, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=197, - serialized_end=259, -) - - -_FEATUREROWEXTENDED = _descriptor.Descriptor( - name='FeatureRowExtended', - full_name='feast.types.FeatureRowExtended', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='row', full_name='feast.types.FeatureRowExtended.row', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='last_attempt', full_name='feast.types.FeatureRowExtended.last_attempt', index=1, - number=2, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='first_seen', full_name='feast.types.FeatureRowExtended.first_seen', index=2, - number=3, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=262, - serialized_end=412, -) - -_ATTEMPT.fields_by_name['error'].message_type = _ERROR -_FEATUREROWEXTENDED.fields_by_name['row'].message_type = feast_dot_types_dot_FeatureRow__pb2._FEATUREROW -_FEATUREROWEXTENDED.fields_by_name['last_attempt'].message_type = _ATTEMPT -_FEATUREROWEXTENDED.fields_by_name['first_seen'].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP -DESCRIPTOR.message_types_by_name['Error'] = _ERROR -DESCRIPTOR.message_types_by_name['Attempt'] = _ATTEMPT -DESCRIPTOR.message_types_by_name['FeatureRowExtended'] = _FEATUREROWEXTENDED -_sym_db.RegisterFileDescriptor(DESCRIPTOR) - -Error = _reflection.GeneratedProtocolMessageType('Error', (_message.Message,), { - 'DESCRIPTOR' : _ERROR, - '__module__' : 'feast.types.FeatureRowExtended_pb2' - # @@protoc_insertion_point(class_scope:feast.types.Error) - }) -_sym_db.RegisterMessage(Error) - -Attempt = _reflection.GeneratedProtocolMessageType('Attempt', (_message.Message,), { - 'DESCRIPTOR' : _ATTEMPT, - '__module__' : 'feast.types.FeatureRowExtended_pb2' - # @@protoc_insertion_point(class_scope:feast.types.Attempt) - }) -_sym_db.RegisterMessage(Attempt) - -FeatureRowExtended = _reflection.GeneratedProtocolMessageType('FeatureRowExtended', (_message.Message,), { - 'DESCRIPTOR' : _FEATUREROWEXTENDED, - '__module__' : 'feast.types.FeatureRowExtended_pb2' - # @@protoc_insertion_point(class_scope:feast.types.FeatureRowExtended) - }) -_sym_db.RegisterMessage(FeatureRowExtended) - - -DESCRIPTOR._options = None -# @@protoc_insertion_point(module_scope) diff --git a/sdk/python/feast/types/FeatureRowExtended_pb2.pyi b/sdk/python/feast/types/FeatureRowExtended_pb2.pyi deleted file mode 100644 index 4f3d02c8ee6..00000000000 --- a/sdk/python/feast/types/FeatureRowExtended_pb2.pyi +++ /dev/null @@ -1,102 +0,0 @@ -# @generated by generate_proto_mypy_stubs.py. Do not edit! -import sys -from feast.types.FeatureRow_pb2 import ( - FeatureRow as feast___types___FeatureRow_pb2___FeatureRow, -) - -from google.protobuf.descriptor import ( - Descriptor as google___protobuf___descriptor___Descriptor, -) - -from google.protobuf.message import ( - Message as google___protobuf___message___Message, -) - -from google.protobuf.timestamp_pb2 import ( - Timestamp as google___protobuf___timestamp_pb2___Timestamp, -) - -from typing import ( - Optional as typing___Optional, - Text as typing___Text, -) - -from typing_extensions import ( - Literal as typing_extensions___Literal, -) - - -class Error(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - cause = ... # type: typing___Text - transform = ... # type: typing___Text - message = ... # type: typing___Text - stack_trace = ... # type: typing___Text - - def __init__(self, - *, - cause : typing___Optional[typing___Text] = None, - transform : typing___Optional[typing___Text] = None, - message : typing___Optional[typing___Text] = None, - stack_trace : typing___Optional[typing___Text] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> Error: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"cause",u"message",u"stack_trace",u"transform"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"cause",b"cause",u"message",b"message",u"stack_trace",b"stack_trace",u"transform",b"transform"]) -> None: ... - -class Attempt(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - attempts = ... # type: int - - @property - def error(self) -> Error: ... - - def __init__(self, - *, - attempts : typing___Optional[int] = None, - error : typing___Optional[Error] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> Attempt: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"error"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"attempts",u"error"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"error",b"error"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"attempts",b"attempts",u"error",b"error"]) -> None: ... - -class FeatureRowExtended(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - @property - def row(self) -> feast___types___FeatureRow_pb2___FeatureRow: ... - - @property - def last_attempt(self) -> Attempt: ... - - @property - def first_seen(self) -> google___protobuf___timestamp_pb2___Timestamp: ... - - def __init__(self, - *, - row : typing___Optional[feast___types___FeatureRow_pb2___FeatureRow] = None, - last_attempt : typing___Optional[Attempt] = None, - first_seen : typing___Optional[google___protobuf___timestamp_pb2___Timestamp] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> FeatureRowExtended: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"first_seen",u"last_attempt",u"row"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"first_seen",u"last_attempt",u"row"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"first_seen",b"first_seen",u"last_attempt",b"last_attempt",u"row",b"row"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"first_seen",b"first_seen",u"last_attempt",b"last_attempt",u"row",b"row"]) -> None: ... diff --git a/sdk/python/feast/types/FeatureRow_pb2.py b/sdk/python/feast/types/FeatureRow_pb2.py deleted file mode 100644 index 1b6c16910f2..00000000000 --- a/sdk/python/feast/types/FeatureRow_pb2.py +++ /dev/null @@ -1,90 +0,0 @@ -# -*- coding: utf-8 -*- -# Generated by the protocol buffer compiler. DO NOT EDIT! -# source: feast/types/FeatureRow.proto - -import sys -_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1')) -from google.protobuf import descriptor as _descriptor -from google.protobuf import message as _message -from google.protobuf import reflection as _reflection -from google.protobuf import symbol_database as _symbol_database -# @@protoc_insertion_point(imports) - -_sym_db = _symbol_database.Default() - - -from google.protobuf import timestamp_pb2 as google_dot_protobuf_dot_timestamp__pb2 -from feast.types import Field_pb2 as feast_dot_types_dot_Field__pb2 - - -DESCRIPTOR = _descriptor.FileDescriptor( - name='feast/types/FeatureRow.proto', - package='feast.types', - syntax='proto3', - serialized_options=_b('\n\013feast.typesB\017FeatureRowProtoZ0github.com/gojek/feast/sdk/go/protos/feast/types'), - serialized_pb=_b('\n\x1c\x66\x65\x61st/types/FeatureRow.proto\x12\x0b\x66\x65\x61st.types\x1a\x1fgoogle/protobuf/timestamp.proto\x1a\x17\x66\x65\x61st/types/Field.proto\"z\n\nFeatureRow\x12\"\n\x06\x66ields\x18\x02 \x03(\x0b\x32\x12.feast.types.Field\x12\x33\n\x0f\x65vent_timestamp\x18\x03 \x01(\x0b\x32\x1a.google.protobuf.Timestamp\x12\x13\n\x0b\x66\x65\x61ture_set\x18\x06 \x01(\tBP\n\x0b\x66\x65\x61st.typesB\x0f\x46\x65\x61tureRowProtoZ0github.com/gojek/feast/sdk/go/protos/feast/typesb\x06proto3') - , - dependencies=[google_dot_protobuf_dot_timestamp__pb2.DESCRIPTOR,feast_dot_types_dot_Field__pb2.DESCRIPTOR,]) - - - - -_FEATUREROW = _descriptor.Descriptor( - name='FeatureRow', - full_name='feast.types.FeatureRow', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='fields', full_name='feast.types.FeatureRow.fields', index=0, - number=2, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='event_timestamp', full_name='feast.types.FeatureRow.event_timestamp', index=1, - number=3, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='feature_set', full_name='feast.types.FeatureRow.feature_set', index=2, - number=6, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=103, - serialized_end=225, -) - -_FEATUREROW.fields_by_name['fields'].message_type = feast_dot_types_dot_Field__pb2._FIELD -_FEATUREROW.fields_by_name['event_timestamp'].message_type = google_dot_protobuf_dot_timestamp__pb2._TIMESTAMP -DESCRIPTOR.message_types_by_name['FeatureRow'] = _FEATUREROW -_sym_db.RegisterFileDescriptor(DESCRIPTOR) - -FeatureRow = _reflection.GeneratedProtocolMessageType('FeatureRow', (_message.Message,), { - 'DESCRIPTOR' : _FEATUREROW, - '__module__' : 'feast.types.FeatureRow_pb2' - # @@protoc_insertion_point(class_scope:feast.types.FeatureRow) - }) -_sym_db.RegisterMessage(FeatureRow) - - -DESCRIPTOR._options = None -# @@protoc_insertion_point(module_scope) diff --git a/sdk/python/feast/types/FeatureRow_pb2.pyi b/sdk/python/feast/types/FeatureRow_pb2.pyi deleted file mode 100644 index 9bf745f9130..00000000000 --- a/sdk/python/feast/types/FeatureRow_pb2.pyi +++ /dev/null @@ -1,59 +0,0 @@ -# @generated by generate_proto_mypy_stubs.py. Do not edit! -import sys -from feast.types.Field_pb2 import ( - Field as feast___types___Field_pb2___Field, -) - -from google.protobuf.descriptor import ( - Descriptor as google___protobuf___descriptor___Descriptor, -) - -from google.protobuf.internal.containers import ( - RepeatedCompositeFieldContainer as google___protobuf___internal___containers___RepeatedCompositeFieldContainer, -) - -from google.protobuf.message import ( - Message as google___protobuf___message___Message, -) - -from google.protobuf.timestamp_pb2 import ( - Timestamp as google___protobuf___timestamp_pb2___Timestamp, -) - -from typing import ( - Iterable as typing___Iterable, - Optional as typing___Optional, - Text as typing___Text, -) - -from typing_extensions import ( - Literal as typing_extensions___Literal, -) - - -class FeatureRow(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - feature_set = ... # type: typing___Text - - @property - def fields(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[feast___types___Field_pb2___Field]: ... - - @property - def event_timestamp(self) -> google___protobuf___timestamp_pb2___Timestamp: ... - - def __init__(self, - *, - fields : typing___Optional[typing___Iterable[feast___types___Field_pb2___Field]] = None, - event_timestamp : typing___Optional[google___protobuf___timestamp_pb2___Timestamp] = None, - feature_set : typing___Optional[typing___Text] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> FeatureRow: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"event_timestamp"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"event_timestamp",u"feature_set",u"fields"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"event_timestamp",b"event_timestamp"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"event_timestamp",b"event_timestamp",u"feature_set",b"feature_set",u"fields",b"fields"]) -> None: ... diff --git a/sdk/python/feast/types/Feature_pb2.py b/sdk/python/feast/types/Feature_pb2.py deleted file mode 100644 index 98d98d88a66..00000000000 --- a/sdk/python/feast/types/Feature_pb2.py +++ /dev/null @@ -1,108 +0,0 @@ -# -*- coding: utf-8 -*- -# Generated by the protocol buffer compiler. DO NOT EDIT! -# source: feast/types/Feature.proto - -import sys - -_b = sys.version_info[0] < 3 and (lambda x: x) or (lambda x: x.encode("latin1")) -from google.protobuf import descriptor as _descriptor -from google.protobuf import message as _message -from google.protobuf import reflection as _reflection -from google.protobuf import symbol_database as _symbol_database - -# @@protoc_insertion_point(imports) - -_sym_db = _symbol_database.Default() - - -from feast.types import Value_pb2 as feast_dot_types_dot_Value__pb2 - - -DESCRIPTOR = _descriptor.FileDescriptor( - name="feast/types/Feature.proto", - package="feast.types", - syntax="proto3", - serialized_options=_b( - "\n\013feast.typesB\014FeatureProtoZ6github.com/gojek/feast/protos/generated/go/feast/types" - ), - serialized_pb=_b( - '\n\x19\x66\x65\x61st/types/Feature.proto\x12\x0b\x66\x65\x61st.types\x1a\x17\x66\x65\x61st/types/Value.proto":\n\x07\x46\x65\x61ture\x12!\n\x05value\x18\x02 \x01(\x0b\x32\x12.feast.types.Value\x12\x0c\n\x04name\x18\x03 \x01(\tBS\n\x0b\x66\x65\x61st.typesB\x0c\x46\x65\x61tureProtoZ6github.com/gojek/feast/protos/generated/go/feast/typesb\x06proto3' - ), - dependencies=[feast_dot_types_dot_Value__pb2.DESCRIPTOR], -) - - -_FEATURE = _descriptor.Descriptor( - name="Feature", - full_name="feast.types.Feature", - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name="value", - full_name="feast.types.Feature.value", - index=0, - number=2, - type=11, - cpp_type=10, - label=1, - has_default_value=False, - default_value=None, - message_type=None, - enum_type=None, - containing_type=None, - is_extension=False, - extension_scope=None, - serialized_options=None, - file=DESCRIPTOR, - ), - _descriptor.FieldDescriptor( - name="name", - full_name="feast.types.Feature.name", - index=1, - number=3, - type=9, - cpp_type=9, - label=1, - has_default_value=False, - default_value=_b("").decode("utf-8"), - message_type=None, - enum_type=None, - containing_type=None, - is_extension=False, - extension_scope=None, - serialized_options=None, - file=DESCRIPTOR, - ), - ], - extensions=[], - nested_types=[], - enum_types=[], - serialized_options=None, - is_extendable=False, - syntax="proto3", - extension_ranges=[], - oneofs=[], - serialized_start=67, - serialized_end=125, -) - -_FEATURE.fields_by_name["value"].message_type = feast_dot_types_dot_Value__pb2._VALUE -DESCRIPTOR.message_types_by_name["Feature"] = _FEATURE -_sym_db.RegisterFileDescriptor(DESCRIPTOR) - -Feature = _reflection.GeneratedProtocolMessageType( - "Feature", - (_message.Message,), - { - "DESCRIPTOR": _FEATURE, - "__module__": "feast.types.Feature_pb2" - # @@protoc_insertion_point(class_scope:feast.types.Feature) - }, -) -_sym_db.RegisterMessage(Feature) - - -DESCRIPTOR._options = None -# @@protoc_insertion_point(module_scope) diff --git a/sdk/python/feast/types/Feature_pb2.pyi b/sdk/python/feast/types/Feature_pb2.pyi deleted file mode 100644 index f31122f1da8..00000000000 --- a/sdk/python/feast/types/Feature_pb2.pyi +++ /dev/null @@ -1,44 +0,0 @@ -# @generated by generate_proto_mypy_stubs.py. Do not edit! -import sys -from feast.types.Value_pb2 import Value as feast___types___Value_pb2___Value - -from google.protobuf.descriptor import ( - Descriptor as google___protobuf___descriptor___Descriptor, -) - -from google.protobuf.message import Message as google___protobuf___message___Message - -from typing import Optional as typing___Optional, Text as typing___Text - -from typing_extensions import Literal as typing_extensions___Literal - -class Feature(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - name = ... # type: typing___Text - @property - def value(self) -> feast___types___Value_pb2___Value: ... - def __init__( - self, - *, - value: typing___Optional[feast___types___Value_pb2___Value] = None, - name: typing___Optional[typing___Text] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> Feature: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField( - self, field_name: typing_extensions___Literal["value"] - ) -> bool: ... - def ClearField( - self, field_name: typing_extensions___Literal["name", "value"] - ) -> None: ... - else: - def HasField( - self, field_name: typing_extensions___Literal["value", b"value"] - ) -> bool: ... - def ClearField( - self, - field_name: typing_extensions___Literal["name", b"name", "value", b"value"], - ) -> None: ... diff --git a/sdk/python/feast/types/Field_pb2.py b/sdk/python/feast/types/Field_pb2.py deleted file mode 100644 index 95bcf38cf9d..00000000000 --- a/sdk/python/feast/types/Field_pb2.py +++ /dev/null @@ -1,81 +0,0 @@ -# -*- coding: utf-8 -*- -# Generated by the protocol buffer compiler. DO NOT EDIT! -# source: feast/types/Field.proto - -import sys -_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1')) -from google.protobuf import descriptor as _descriptor -from google.protobuf import message as _message -from google.protobuf import reflection as _reflection -from google.protobuf import symbol_database as _symbol_database -# @@protoc_insertion_point(imports) - -_sym_db = _symbol_database.Default() - - -from feast.types import Value_pb2 as feast_dot_types_dot_Value__pb2 - - -DESCRIPTOR = _descriptor.FileDescriptor( - name='feast/types/Field.proto', - package='feast.types', - syntax='proto3', - serialized_options=_b('\n\013feast.typesB\nFieldProtoZ0github.com/gojek/feast/sdk/go/protos/feast/types'), - serialized_pb=_b('\n\x17\x66\x65\x61st/types/Field.proto\x12\x0b\x66\x65\x61st.types\x1a\x17\x66\x65\x61st/types/Value.proto\"8\n\x05\x46ield\x12\x0c\n\x04name\x18\x01 \x01(\t\x12!\n\x05value\x18\x02 \x01(\x0b\x32\x12.feast.types.ValueBK\n\x0b\x66\x65\x61st.typesB\nFieldProtoZ0github.com/gojek/feast/sdk/go/protos/feast/typesb\x06proto3') - , - dependencies=[feast_dot_types_dot_Value__pb2.DESCRIPTOR,]) - - - - -_FIELD = _descriptor.Descriptor( - name='Field', - full_name='feast.types.Field', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='name', full_name='feast.types.Field.name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='value', full_name='feast.types.Field.value', index=1, - number=2, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=65, - serialized_end=121, -) - -_FIELD.fields_by_name['value'].message_type = feast_dot_types_dot_Value__pb2._VALUE -DESCRIPTOR.message_types_by_name['Field'] = _FIELD -_sym_db.RegisterFileDescriptor(DESCRIPTOR) - -Field = _reflection.GeneratedProtocolMessageType('Field', (_message.Message,), { - 'DESCRIPTOR' : _FIELD, - '__module__' : 'feast.types.Field_pb2' - # @@protoc_insertion_point(class_scope:feast.types.Field) - }) -_sym_db.RegisterMessage(Field) - - -DESCRIPTOR._options = None -# @@protoc_insertion_point(module_scope) diff --git a/sdk/python/feast/types/Field_pb2.pyi b/sdk/python/feast/types/Field_pb2.pyi deleted file mode 100644 index 1305503fab7..00000000000 --- a/sdk/python/feast/types/Field_pb2.pyi +++ /dev/null @@ -1,46 +0,0 @@ -# @generated by generate_proto_mypy_stubs.py. Do not edit! -import sys -from feast.types.Value_pb2 import ( - Value as feast___types___Value_pb2___Value, -) - -from google.protobuf.descriptor import ( - Descriptor as google___protobuf___descriptor___Descriptor, -) - -from google.protobuf.message import ( - Message as google___protobuf___message___Message, -) - -from typing import ( - Optional as typing___Optional, - Text as typing___Text, -) - -from typing_extensions import ( - Literal as typing_extensions___Literal, -) - - -class Field(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - name = ... # type: typing___Text - - @property - def value(self) -> feast___types___Value_pb2___Value: ... - - def __init__(self, - *, - name : typing___Optional[typing___Text] = None, - value : typing___Optional[feast___types___Value_pb2___Value] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> Field: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"value"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"name",u"value"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"value",b"value"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"name",b"name",u"value",b"value"]) -> None: ... diff --git a/sdk/python/feast/types/Value_pb2.py b/sdk/python/feast/types/Value_pb2.py deleted file mode 100644 index fe2cd125ca5..00000000000 --- a/sdk/python/feast/types/Value_pb2.py +++ /dev/null @@ -1,595 +0,0 @@ -# -*- coding: utf-8 -*- -# Generated by the protocol buffer compiler. DO NOT EDIT! -# source: feast/types/Value.proto - -import sys -_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1')) -from google.protobuf import descriptor as _descriptor -from google.protobuf import message as _message -from google.protobuf import reflection as _reflection -from google.protobuf import symbol_database as _symbol_database -# @@protoc_insertion_point(imports) - -_sym_db = _symbol_database.Default() - - - - -DESCRIPTOR = _descriptor.FileDescriptor( - name='feast/types/Value.proto', - package='feast.types', - syntax='proto3', - serialized_options=_b('\n\013feast.typesB\nValueProtoZ0github.com/gojek/feast/sdk/go/protos/feast/types'), - serialized_pb=_b('\n\x17\x66\x65\x61st/types/Value.proto\x12\x0b\x66\x65\x61st.types\"\xe0\x01\n\tValueType\"\xd2\x01\n\x04\x45num\x12\x0b\n\x07INVALID\x10\x00\x12\t\n\x05\x42YTES\x10\x01\x12\n\n\x06STRING\x10\x02\x12\t\n\x05INT32\x10\x03\x12\t\n\x05INT64\x10\x04\x12\n\n\x06\x44OUBLE\x10\x05\x12\t\n\x05\x46LOAT\x10\x06\x12\x08\n\x04\x42OOL\x10\x07\x12\x0e\n\nBYTES_LIST\x10\x0b\x12\x0f\n\x0bSTRING_LIST\x10\x0c\x12\x0e\n\nINT32_LIST\x10\r\x12\x0e\n\nINT64_LIST\x10\x0e\x12\x0f\n\x0b\x44OUBLE_LIST\x10\x0f\x12\x0e\n\nFLOAT_LIST\x10\x10\x12\r\n\tBOOL_LIST\x10\x11\"\x82\x04\n\x05Value\x12\x13\n\tbytes_val\x18\x01 \x01(\x0cH\x00\x12\x14\n\nstring_val\x18\x02 \x01(\tH\x00\x12\x13\n\tint32_val\x18\x03 \x01(\x05H\x00\x12\x13\n\tint64_val\x18\x04 \x01(\x03H\x00\x12\x14\n\ndouble_val\x18\x05 \x01(\x01H\x00\x12\x13\n\tfloat_val\x18\x06 \x01(\x02H\x00\x12\x12\n\x08\x62ool_val\x18\x07 \x01(\x08H\x00\x12\x30\n\x0e\x62ytes_list_val\x18\x0b \x01(\x0b\x32\x16.feast.types.BytesListH\x00\x12\x32\n\x0fstring_list_val\x18\x0c \x01(\x0b\x32\x17.feast.types.StringListH\x00\x12\x30\n\x0eint32_list_val\x18\r \x01(\x0b\x32\x16.feast.types.Int32ListH\x00\x12\x30\n\x0eint64_list_val\x18\x0e \x01(\x0b\x32\x16.feast.types.Int64ListH\x00\x12\x32\n\x0f\x64ouble_list_val\x18\x0f \x01(\x0b\x32\x17.feast.types.DoubleListH\x00\x12\x30\n\x0e\x66loat_list_val\x18\x10 \x01(\x0b\x32\x16.feast.types.FloatListH\x00\x12.\n\rbool_list_val\x18\x11 \x01(\x0b\x32\x15.feast.types.BoolListH\x00\x42\x05\n\x03val\"\x18\n\tBytesList\x12\x0b\n\x03val\x18\x01 \x03(\x0c\"\x19\n\nStringList\x12\x0b\n\x03val\x18\x01 \x03(\t\"\x18\n\tInt32List\x12\x0b\n\x03val\x18\x01 \x03(\x05\"\x18\n\tInt64List\x12\x0b\n\x03val\x18\x01 \x03(\x03\"\x19\n\nDoubleList\x12\x0b\n\x03val\x18\x01 \x03(\x01\"\x18\n\tFloatList\x12\x0b\n\x03val\x18\x01 \x03(\x02\"\x17\n\x08\x42oolList\x12\x0b\n\x03val\x18\x01 \x03(\x08\x42K\n\x0b\x66\x65\x61st.typesB\nValueProtoZ0github.com/gojek/feast/sdk/go/protos/feast/typesb\x06proto3') -) - - - -_VALUETYPE_ENUM = _descriptor.EnumDescriptor( - name='Enum', - full_name='feast.types.ValueType.Enum', - filename=None, - file=DESCRIPTOR, - values=[ - _descriptor.EnumValueDescriptor( - name='INVALID', index=0, number=0, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='BYTES', index=1, number=1, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='STRING', index=2, number=2, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='INT32', index=3, number=3, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='INT64', index=4, number=4, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='DOUBLE', index=5, number=5, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='FLOAT', index=6, number=6, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='BOOL', index=7, number=7, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='BYTES_LIST', index=8, number=11, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='STRING_LIST', index=9, number=12, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='INT32_LIST', index=10, number=13, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='INT64_LIST', index=11, number=14, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='DOUBLE_LIST', index=12, number=15, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='FLOAT_LIST', index=13, number=16, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='BOOL_LIST', index=14, number=17, - serialized_options=None, - type=None), - ], - containing_type=None, - serialized_options=None, - serialized_start=55, - serialized_end=265, -) -_sym_db.RegisterEnumDescriptor(_VALUETYPE_ENUM) - - -_VALUETYPE = _descriptor.Descriptor( - name='ValueType', - full_name='feast.types.ValueType', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - _VALUETYPE_ENUM, - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=41, - serialized_end=265, -) - - -_VALUE = _descriptor.Descriptor( - name='Value', - full_name='feast.types.Value', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='bytes_val', full_name='feast.types.Value.bytes_val', index=0, - number=1, type=12, cpp_type=9, label=1, - has_default_value=False, default_value=_b(""), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='string_val', full_name='feast.types.Value.string_val', index=1, - number=2, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=_b("").decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='int32_val', full_name='feast.types.Value.int32_val', index=2, - number=3, type=5, cpp_type=1, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='int64_val', full_name='feast.types.Value.int64_val', index=3, - number=4, type=3, cpp_type=2, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='double_val', full_name='feast.types.Value.double_val', index=4, - number=5, type=1, cpp_type=5, label=1, - has_default_value=False, default_value=float(0), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='float_val', full_name='feast.types.Value.float_val', index=5, - number=6, type=2, cpp_type=6, label=1, - has_default_value=False, default_value=float(0), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='bool_val', full_name='feast.types.Value.bool_val', index=6, - number=7, type=8, cpp_type=7, label=1, - has_default_value=False, default_value=False, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='bytes_list_val', full_name='feast.types.Value.bytes_list_val', index=7, - number=11, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='string_list_val', full_name='feast.types.Value.string_list_val', index=8, - number=12, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='int32_list_val', full_name='feast.types.Value.int32_list_val', index=9, - number=13, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='int64_list_val', full_name='feast.types.Value.int64_list_val', index=10, - number=14, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='double_list_val', full_name='feast.types.Value.double_list_val', index=11, - number=15, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='float_list_val', full_name='feast.types.Value.float_list_val', index=12, - number=16, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='bool_list_val', full_name='feast.types.Value.bool_list_val', index=13, - number=17, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - _descriptor.OneofDescriptor( - name='val', full_name='feast.types.Value.val', - index=0, containing_type=None, fields=[]), - ], - serialized_start=268, - serialized_end=782, -) - - -_BYTESLIST = _descriptor.Descriptor( - name='BytesList', - full_name='feast.types.BytesList', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='val', full_name='feast.types.BytesList.val', index=0, - number=1, type=12, cpp_type=9, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=784, - serialized_end=808, -) - - -_STRINGLIST = _descriptor.Descriptor( - name='StringList', - full_name='feast.types.StringList', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='val', full_name='feast.types.StringList.val', index=0, - number=1, type=9, cpp_type=9, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=810, - serialized_end=835, -) - - -_INT32LIST = _descriptor.Descriptor( - name='Int32List', - full_name='feast.types.Int32List', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='val', full_name='feast.types.Int32List.val', index=0, - number=1, type=5, cpp_type=1, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=837, - serialized_end=861, -) - - -_INT64LIST = _descriptor.Descriptor( - name='Int64List', - full_name='feast.types.Int64List', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='val', full_name='feast.types.Int64List.val', index=0, - number=1, type=3, cpp_type=2, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=863, - serialized_end=887, -) - - -_DOUBLELIST = _descriptor.Descriptor( - name='DoubleList', - full_name='feast.types.DoubleList', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='val', full_name='feast.types.DoubleList.val', index=0, - number=1, type=1, cpp_type=5, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=889, - serialized_end=914, -) - - -_FLOATLIST = _descriptor.Descriptor( - name='FloatList', - full_name='feast.types.FloatList', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='val', full_name='feast.types.FloatList.val', index=0, - number=1, type=2, cpp_type=6, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=916, - serialized_end=940, -) - - -_BOOLLIST = _descriptor.Descriptor( - name='BoolList', - full_name='feast.types.BoolList', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='val', full_name='feast.types.BoolList.val', index=0, - number=1, type=8, cpp_type=7, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto3', - extension_ranges=[], - oneofs=[ - ], - serialized_start=942, - serialized_end=965, -) - -_VALUETYPE_ENUM.containing_type = _VALUETYPE -_VALUE.fields_by_name['bytes_list_val'].message_type = _BYTESLIST -_VALUE.fields_by_name['string_list_val'].message_type = _STRINGLIST -_VALUE.fields_by_name['int32_list_val'].message_type = _INT32LIST -_VALUE.fields_by_name['int64_list_val'].message_type = _INT64LIST -_VALUE.fields_by_name['double_list_val'].message_type = _DOUBLELIST -_VALUE.fields_by_name['float_list_val'].message_type = _FLOATLIST -_VALUE.fields_by_name['bool_list_val'].message_type = _BOOLLIST -_VALUE.oneofs_by_name['val'].fields.append( - _VALUE.fields_by_name['bytes_val']) -_VALUE.fields_by_name['bytes_val'].containing_oneof = _VALUE.oneofs_by_name['val'] -_VALUE.oneofs_by_name['val'].fields.append( - _VALUE.fields_by_name['string_val']) -_VALUE.fields_by_name['string_val'].containing_oneof = _VALUE.oneofs_by_name['val'] -_VALUE.oneofs_by_name['val'].fields.append( - _VALUE.fields_by_name['int32_val']) -_VALUE.fields_by_name['int32_val'].containing_oneof = _VALUE.oneofs_by_name['val'] -_VALUE.oneofs_by_name['val'].fields.append( - _VALUE.fields_by_name['int64_val']) -_VALUE.fields_by_name['int64_val'].containing_oneof = _VALUE.oneofs_by_name['val'] -_VALUE.oneofs_by_name['val'].fields.append( - _VALUE.fields_by_name['double_val']) -_VALUE.fields_by_name['double_val'].containing_oneof = _VALUE.oneofs_by_name['val'] -_VALUE.oneofs_by_name['val'].fields.append( - _VALUE.fields_by_name['float_val']) -_VALUE.fields_by_name['float_val'].containing_oneof = _VALUE.oneofs_by_name['val'] -_VALUE.oneofs_by_name['val'].fields.append( - _VALUE.fields_by_name['bool_val']) -_VALUE.fields_by_name['bool_val'].containing_oneof = _VALUE.oneofs_by_name['val'] -_VALUE.oneofs_by_name['val'].fields.append( - _VALUE.fields_by_name['bytes_list_val']) -_VALUE.fields_by_name['bytes_list_val'].containing_oneof = _VALUE.oneofs_by_name['val'] -_VALUE.oneofs_by_name['val'].fields.append( - _VALUE.fields_by_name['string_list_val']) -_VALUE.fields_by_name['string_list_val'].containing_oneof = _VALUE.oneofs_by_name['val'] -_VALUE.oneofs_by_name['val'].fields.append( - _VALUE.fields_by_name['int32_list_val']) -_VALUE.fields_by_name['int32_list_val'].containing_oneof = _VALUE.oneofs_by_name['val'] -_VALUE.oneofs_by_name['val'].fields.append( - _VALUE.fields_by_name['int64_list_val']) -_VALUE.fields_by_name['int64_list_val'].containing_oneof = _VALUE.oneofs_by_name['val'] -_VALUE.oneofs_by_name['val'].fields.append( - _VALUE.fields_by_name['double_list_val']) -_VALUE.fields_by_name['double_list_val'].containing_oneof = _VALUE.oneofs_by_name['val'] -_VALUE.oneofs_by_name['val'].fields.append( - _VALUE.fields_by_name['float_list_val']) -_VALUE.fields_by_name['float_list_val'].containing_oneof = _VALUE.oneofs_by_name['val'] -_VALUE.oneofs_by_name['val'].fields.append( - _VALUE.fields_by_name['bool_list_val']) -_VALUE.fields_by_name['bool_list_val'].containing_oneof = _VALUE.oneofs_by_name['val'] -DESCRIPTOR.message_types_by_name['ValueType'] = _VALUETYPE -DESCRIPTOR.message_types_by_name['Value'] = _VALUE -DESCRIPTOR.message_types_by_name['BytesList'] = _BYTESLIST -DESCRIPTOR.message_types_by_name['StringList'] = _STRINGLIST -DESCRIPTOR.message_types_by_name['Int32List'] = _INT32LIST -DESCRIPTOR.message_types_by_name['Int64List'] = _INT64LIST -DESCRIPTOR.message_types_by_name['DoubleList'] = _DOUBLELIST -DESCRIPTOR.message_types_by_name['FloatList'] = _FLOATLIST -DESCRIPTOR.message_types_by_name['BoolList'] = _BOOLLIST -_sym_db.RegisterFileDescriptor(DESCRIPTOR) - -ValueType = _reflection.GeneratedProtocolMessageType('ValueType', (_message.Message,), { - 'DESCRIPTOR' : _VALUETYPE, - '__module__' : 'feast.types.Value_pb2' - # @@protoc_insertion_point(class_scope:feast.types.ValueType) - }) -_sym_db.RegisterMessage(ValueType) - -Value = _reflection.GeneratedProtocolMessageType('Value', (_message.Message,), { - 'DESCRIPTOR' : _VALUE, - '__module__' : 'feast.types.Value_pb2' - # @@protoc_insertion_point(class_scope:feast.types.Value) - }) -_sym_db.RegisterMessage(Value) - -BytesList = _reflection.GeneratedProtocolMessageType('BytesList', (_message.Message,), { - 'DESCRIPTOR' : _BYTESLIST, - '__module__' : 'feast.types.Value_pb2' - # @@protoc_insertion_point(class_scope:feast.types.BytesList) - }) -_sym_db.RegisterMessage(BytesList) - -StringList = _reflection.GeneratedProtocolMessageType('StringList', (_message.Message,), { - 'DESCRIPTOR' : _STRINGLIST, - '__module__' : 'feast.types.Value_pb2' - # @@protoc_insertion_point(class_scope:feast.types.StringList) - }) -_sym_db.RegisterMessage(StringList) - -Int32List = _reflection.GeneratedProtocolMessageType('Int32List', (_message.Message,), { - 'DESCRIPTOR' : _INT32LIST, - '__module__' : 'feast.types.Value_pb2' - # @@protoc_insertion_point(class_scope:feast.types.Int32List) - }) -_sym_db.RegisterMessage(Int32List) - -Int64List = _reflection.GeneratedProtocolMessageType('Int64List', (_message.Message,), { - 'DESCRIPTOR' : _INT64LIST, - '__module__' : 'feast.types.Value_pb2' - # @@protoc_insertion_point(class_scope:feast.types.Int64List) - }) -_sym_db.RegisterMessage(Int64List) - -DoubleList = _reflection.GeneratedProtocolMessageType('DoubleList', (_message.Message,), { - 'DESCRIPTOR' : _DOUBLELIST, - '__module__' : 'feast.types.Value_pb2' - # @@protoc_insertion_point(class_scope:feast.types.DoubleList) - }) -_sym_db.RegisterMessage(DoubleList) - -FloatList = _reflection.GeneratedProtocolMessageType('FloatList', (_message.Message,), { - 'DESCRIPTOR' : _FLOATLIST, - '__module__' : 'feast.types.Value_pb2' - # @@protoc_insertion_point(class_scope:feast.types.FloatList) - }) -_sym_db.RegisterMessage(FloatList) - -BoolList = _reflection.GeneratedProtocolMessageType('BoolList', (_message.Message,), { - 'DESCRIPTOR' : _BOOLLIST, - '__module__' : 'feast.types.Value_pb2' - # @@protoc_insertion_point(class_scope:feast.types.BoolList) - }) -_sym_db.RegisterMessage(BoolList) - - -DESCRIPTOR._options = None -# @@protoc_insertion_point(module_scope) diff --git a/sdk/python/feast/types/Value_pb2.pyi b/sdk/python/feast/types/Value_pb2.pyi deleted file mode 100644 index d8b8a73dd36..00000000000 --- a/sdk/python/feast/types/Value_pb2.pyi +++ /dev/null @@ -1,260 +0,0 @@ -# @generated by generate_proto_mypy_stubs.py. Do not edit! -import sys -from google.protobuf.descriptor import ( - Descriptor as google___protobuf___descriptor___Descriptor, - EnumDescriptor as google___protobuf___descriptor___EnumDescriptor, -) - -from google.protobuf.internal.containers import ( - RepeatedScalarFieldContainer as google___protobuf___internal___containers___RepeatedScalarFieldContainer, -) - -from google.protobuf.message import ( - Message as google___protobuf___message___Message, -) - -from typing import ( - Iterable as typing___Iterable, - List as typing___List, - Optional as typing___Optional, - Text as typing___Text, - Tuple as typing___Tuple, - cast as typing___cast, -) - -from typing_extensions import ( - Literal as typing_extensions___Literal, -) - - -class ValueType(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class Enum(int): - DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... - @classmethod - def Name(cls, number: int) -> str: ... - @classmethod - def Value(cls, name: str) -> ValueType.Enum: ... - @classmethod - def keys(cls) -> typing___List[str]: ... - @classmethod - def values(cls) -> typing___List[ValueType.Enum]: ... - @classmethod - def items(cls) -> typing___List[typing___Tuple[str, ValueType.Enum]]: ... - INVALID = typing___cast(ValueType.Enum, 0) - BYTES = typing___cast(ValueType.Enum, 1) - STRING = typing___cast(ValueType.Enum, 2) - INT32 = typing___cast(ValueType.Enum, 3) - INT64 = typing___cast(ValueType.Enum, 4) - DOUBLE = typing___cast(ValueType.Enum, 5) - FLOAT = typing___cast(ValueType.Enum, 6) - BOOL = typing___cast(ValueType.Enum, 7) - BYTES_LIST = typing___cast(ValueType.Enum, 11) - STRING_LIST = typing___cast(ValueType.Enum, 12) - INT32_LIST = typing___cast(ValueType.Enum, 13) - INT64_LIST = typing___cast(ValueType.Enum, 14) - DOUBLE_LIST = typing___cast(ValueType.Enum, 15) - FLOAT_LIST = typing___cast(ValueType.Enum, 16) - BOOL_LIST = typing___cast(ValueType.Enum, 17) - INVALID = typing___cast(ValueType.Enum, 0) - BYTES = typing___cast(ValueType.Enum, 1) - STRING = typing___cast(ValueType.Enum, 2) - INT32 = typing___cast(ValueType.Enum, 3) - INT64 = typing___cast(ValueType.Enum, 4) - DOUBLE = typing___cast(ValueType.Enum, 5) - FLOAT = typing___cast(ValueType.Enum, 6) - BOOL = typing___cast(ValueType.Enum, 7) - BYTES_LIST = typing___cast(ValueType.Enum, 11) - STRING_LIST = typing___cast(ValueType.Enum, 12) - INT32_LIST = typing___cast(ValueType.Enum, 13) - INT64_LIST = typing___cast(ValueType.Enum, 14) - DOUBLE_LIST = typing___cast(ValueType.Enum, 15) - FLOAT_LIST = typing___cast(ValueType.Enum, 16) - BOOL_LIST = typing___cast(ValueType.Enum, 17) - - - def __init__(self, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> ValueType: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - -class Value(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - bytes_val = ... # type: bytes - string_val = ... # type: typing___Text - int32_val = ... # type: int - int64_val = ... # type: int - double_val = ... # type: float - float_val = ... # type: float - bool_val = ... # type: bool - - @property - def bytes_list_val(self) -> BytesList: ... - - @property - def string_list_val(self) -> StringList: ... - - @property - def int32_list_val(self) -> Int32List: ... - - @property - def int64_list_val(self) -> Int64List: ... - - @property - def double_list_val(self) -> DoubleList: ... - - @property - def float_list_val(self) -> FloatList: ... - - @property - def bool_list_val(self) -> BoolList: ... - - def __init__(self, - *, - bytes_val : typing___Optional[bytes] = None, - string_val : typing___Optional[typing___Text] = None, - int32_val : typing___Optional[int] = None, - int64_val : typing___Optional[int] = None, - double_val : typing___Optional[float] = None, - float_val : typing___Optional[float] = None, - bool_val : typing___Optional[bool] = None, - bytes_list_val : typing___Optional[BytesList] = None, - string_list_val : typing___Optional[StringList] = None, - int32_list_val : typing___Optional[Int32List] = None, - int64_list_val : typing___Optional[Int64List] = None, - double_list_val : typing___Optional[DoubleList] = None, - float_list_val : typing___Optional[FloatList] = None, - bool_list_val : typing___Optional[BoolList] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> Value: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def HasField(self, field_name: typing_extensions___Literal[u"bool_list_val",u"bool_val",u"bytes_list_val",u"bytes_val",u"double_list_val",u"double_val",u"float_list_val",u"float_val",u"int32_list_val",u"int32_val",u"int64_list_val",u"int64_val",u"string_list_val",u"string_val",u"val"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"bool_list_val",u"bool_val",u"bytes_list_val",u"bytes_val",u"double_list_val",u"double_val",u"float_list_val",u"float_val",u"int32_list_val",u"int32_val",u"int64_list_val",u"int64_val",u"string_list_val",u"string_val",u"val"]) -> None: ... - else: - def HasField(self, field_name: typing_extensions___Literal[u"bool_list_val",b"bool_list_val",u"bool_val",b"bool_val",u"bytes_list_val",b"bytes_list_val",u"bytes_val",b"bytes_val",u"double_list_val",b"double_list_val",u"double_val",b"double_val",u"float_list_val",b"float_list_val",u"float_val",b"float_val",u"int32_list_val",b"int32_list_val",u"int32_val",b"int32_val",u"int64_list_val",b"int64_list_val",u"int64_val",b"int64_val",u"string_list_val",b"string_list_val",u"string_val",b"string_val",u"val",b"val"]) -> bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"bool_list_val",b"bool_list_val",u"bool_val",b"bool_val",u"bytes_list_val",b"bytes_list_val",u"bytes_val",b"bytes_val",u"double_list_val",b"double_list_val",u"double_val",b"double_val",u"float_list_val",b"float_list_val",u"float_val",b"float_val",u"int32_list_val",b"int32_list_val",u"int32_val",b"int32_val",u"int64_list_val",b"int64_list_val",u"int64_val",b"int64_val",u"string_list_val",b"string_list_val",u"string_val",b"string_val",u"val",b"val"]) -> None: ... - def WhichOneof(self, oneof_group: typing_extensions___Literal[u"val",b"val"]) -> typing_extensions___Literal["bytes_val","string_val","int32_val","int64_val","double_val","float_val","bool_val","bytes_list_val","string_list_val","int32_list_val","int64_list_val","double_list_val","float_list_val","bool_list_val"]: ... - -class BytesList(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - val = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[bytes] - - def __init__(self, - *, - val : typing___Optional[typing___Iterable[bytes]] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> BytesList: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"val"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"val",b"val"]) -> None: ... - -class StringList(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - val = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] - - def __init__(self, - *, - val : typing___Optional[typing___Iterable[typing___Text]] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> StringList: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"val"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"val",b"val"]) -> None: ... - -class Int32List(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - val = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[int] - - def __init__(self, - *, - val : typing___Optional[typing___Iterable[int]] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> Int32List: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"val"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"val",b"val"]) -> None: ... - -class Int64List(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - val = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[int] - - def __init__(self, - *, - val : typing___Optional[typing___Iterable[int]] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> Int64List: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"val"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"val",b"val"]) -> None: ... - -class DoubleList(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - val = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[float] - - def __init__(self, - *, - val : typing___Optional[typing___Iterable[float]] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> DoubleList: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"val"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"val",b"val"]) -> None: ... - -class FloatList(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - val = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[float] - - def __init__(self, - *, - val : typing___Optional[typing___Iterable[float]] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> FloatList: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"val"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"val",b"val"]) -> None: ... - -class BoolList(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - val = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[bool] - - def __init__(self, - *, - val : typing___Optional[typing___Iterable[bool]] = None, - ) -> None: ... - @classmethod - def FromString(cls, s: bytes) -> BoolList: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - if sys.version_info >= (3,): - def ClearField(self, field_name: typing_extensions___Literal[u"val"]) -> None: ... - else: - def ClearField(self, field_name: typing_extensions___Literal[u"val",b"val"]) -> None: ... diff --git a/sdk/python/pyproject.toml b/sdk/python/pyproject.toml new file mode 100644 index 00000000000..12073607100 --- /dev/null +++ b/sdk/python/pyproject.toml @@ -0,0 +1,26 @@ +[tool.black] +line-length = 88 +target-version = ['py37'] +include = '\.pyi?$' +exclude = ''' +( + /( + \.eggs # exclude a few common directories in the + | \.git # root of the project + | \.hg + | \.mypy_cache + | \.tox + | \.venv + | _build + | buck-out + | build + | dist + | pb2.py + | \.pyi + | core + | serving + | storage + | types + )/ +) +''' \ No newline at end of file diff --git a/sdk/python/requirements-ci.txt b/sdk/python/requirements-ci.txt index 31818ba7f7b..2975342e24e 100644 --- a/sdk/python/requirements-ci.txt +++ b/sdk/python/requirements-ci.txt @@ -28,4 +28,11 @@ confluent_kafka google pandavro==1.5.* kafka-python==1.* -tabulate==0.8.* \ No newline at end of file +tabulate==0.8.* +isort +grpcio-tools +mypy +mypy-protobuf +pre-commit +flake8 +black \ No newline at end of file diff --git a/sdk/python/setup.cfg b/sdk/python/setup.cfg new file mode 100644 index 00000000000..9ccd3bb57aa --- /dev/null +++ b/sdk/python/setup.cfg @@ -0,0 +1,18 @@ +[isort] +multi_line_output=3 +include_trailing_comma=True +force_grid_wrap=0 +use_parentheses=True +line_length=88 +skip=feast/types,feast/core,feast/serving,feast/storage + +[flake8] +ignore = E203, E266, E501, W503 +max-line-length = 88 +max-complexity = 20 +select = B,C,E,F,W,T4 +exclude = .git,__pycache__,docs/conf.py,dist,feast/core,feast/serving,feast/types,feast/storage + +[mypy] +files=feast,test +ignore_missing_imports=true \ No newline at end of file diff --git a/sdk/python/tensorflow_metadata/proto/v0/path_pb2.py b/sdk/python/tensorflow_metadata/proto/v0/path_pb2.py new file mode 100644 index 00000000000..24850688592 --- /dev/null +++ b/sdk/python/tensorflow_metadata/proto/v0/path_pb2.py @@ -0,0 +1,69 @@ +# -*- coding: utf-8 -*- +# Generated by the protocol buffer compiler. DO NOT EDIT! +# source: tensorflow_metadata/proto/v0/path.proto + +from google.protobuf import descriptor as _descriptor +from google.protobuf import message as _message +from google.protobuf import reflection as _reflection +from google.protobuf import symbol_database as _symbol_database +# @@protoc_insertion_point(imports) + +_sym_db = _symbol_database.Default() + + + + +DESCRIPTOR = _descriptor.FileDescriptor( + name='tensorflow_metadata/proto/v0/path.proto', + package='tensorflow.metadata.v0', + syntax='proto2', + serialized_options=b'\n\032org.tensorflow.metadata.v0P\001\370\001\001', + serialized_pb=b'\n\'tensorflow_metadata/proto/v0/path.proto\x12\x16tensorflow.metadata.v0\"\x14\n\x04Path\x12\x0c\n\x04step\x18\x01 \x03(\tB!\n\x1aorg.tensorflow.metadata.v0P\x01\xf8\x01\x01' +) + + + + +_PATH = _descriptor.Descriptor( + name='Path', + full_name='tensorflow.metadata.v0.Path', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='step', full_name='tensorflow.metadata.v0.Path.step', index=0, + number=1, type=9, cpp_type=9, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=67, + serialized_end=87, +) + +DESCRIPTOR.message_types_by_name['Path'] = _PATH +_sym_db.RegisterFileDescriptor(DESCRIPTOR) + +Path = _reflection.GeneratedProtocolMessageType('Path', (_message.Message,), { + 'DESCRIPTOR' : _PATH, + '__module__' : 'tensorflow_metadata.proto.v0.path_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.Path) + }) +_sym_db.RegisterMessage(Path) + + +DESCRIPTOR._options = None +# @@protoc_insertion_point(module_scope) diff --git a/sdk/python/tensorflow_metadata/proto/v0/path_pb2.pyi b/sdk/python/tensorflow_metadata/proto/v0/path_pb2.pyi new file mode 100644 index 00000000000..caf370bd372 --- /dev/null +++ b/sdk/python/tensorflow_metadata/proto/v0/path_pb2.pyi @@ -0,0 +1,52 @@ +# @generated by generate_proto_mypy_stubs.py. Do not edit! +import sys +from google.protobuf.descriptor import ( + Descriptor as google___protobuf___descriptor___Descriptor, +) + +from google.protobuf.internal.containers import ( + RepeatedScalarFieldContainer as google___protobuf___internal___containers___RepeatedScalarFieldContainer, +) + +from google.protobuf.message import ( + Message as google___protobuf___message___Message, +) + +from typing import ( + Iterable as typing___Iterable, + Optional as typing___Optional, + Text as typing___Text, + Union as typing___Union, +) + +from typing_extensions import ( + Literal as typing_extensions___Literal, +) + + +builtin___bool = bool +builtin___bytes = bytes +builtin___float = float +builtin___int = int +if sys.version_info < (3,): + builtin___buffer = buffer + builtin___unicode = unicode + + +class Path(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + step = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] + + def __init__(self, + *, + step : typing___Optional[typing___Iterable[typing___Text]] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> Path: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> Path: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def ClearField(self, field_name: typing_extensions___Literal[u"step",b"step"]) -> None: ... diff --git a/sdk/python/tensorflow_metadata/proto/v0/schema_pb2.py b/sdk/python/tensorflow_metadata/proto/v0/schema_pb2.py new file mode 100644 index 00000000000..c27579f0e28 --- /dev/null +++ b/sdk/python/tensorflow_metadata/proto/v0/schema_pb2.py @@ -0,0 +1,2256 @@ +# -*- coding: utf-8 -*- +# Generated by the protocol buffer compiler. DO NOT EDIT! +# source: tensorflow_metadata/proto/v0/schema.proto + +from google.protobuf.internal import enum_type_wrapper +from google.protobuf import descriptor as _descriptor +from google.protobuf import message as _message +from google.protobuf import reflection as _reflection +from google.protobuf import symbol_database as _symbol_database +# @@protoc_insertion_point(imports) + +_sym_db = _symbol_database.Default() + + +from google.protobuf import any_pb2 as google_dot_protobuf_dot_any__pb2 +from tensorflow_metadata.proto.v0 import path_pb2 as tensorflow__metadata_dot_proto_dot_v0_dot_path__pb2 + + +DESCRIPTOR = _descriptor.FileDescriptor( + name='tensorflow_metadata/proto/v0/schema.proto', + package='tensorflow.metadata.v0', + syntax='proto2', + serialized_options=b'\n\032org.tensorflow.metadata.v0P\001\370\001\001', + serialized_pb=b'\n)tensorflow_metadata/proto/v0/schema.proto\x12\x16tensorflow.metadata.v0\x1a\x19google/protobuf/any.proto\x1a\'tensorflow_metadata/proto/v0/path.proto\"\xe2\x05\n\x06Schema\x12\x30\n\x07\x66\x65\x61ture\x18\x01 \x03(\x0b\x32\x1f.tensorflow.metadata.v0.Feature\x12=\n\x0esparse_feature\x18\x06 \x03(\x0b\x32%.tensorflow.metadata.v0.SparseFeature\x12\x41\n\x10weighted_feature\x18\x0c \x03(\x0b\x32\'.tensorflow.metadata.v0.WeightedFeature\x12;\n\rstring_domain\x18\x04 \x03(\x0b\x32$.tensorflow.metadata.v0.StringDomain\x12\x39\n\x0c\x66loat_domain\x18\t \x03(\x0b\x32#.tensorflow.metadata.v0.FloatDomain\x12\x35\n\nint_domain\x18\n \x03(\x0b\x32!.tensorflow.metadata.v0.IntDomain\x12\x1b\n\x13\x64\x65\x66\x61ult_environment\x18\x05 \x03(\t\x12\x36\n\nannotation\x18\x08 \x01(\x0b\x32\".tensorflow.metadata.v0.Annotation\x12G\n\x13\x64\x61taset_constraints\x18\x0b \x01(\x0b\x32*.tensorflow.metadata.v0.DatasetConstraints\x12\x62\n\x1btensor_representation_group\x18\r \x03(\x0b\x32=.tensorflow.metadata.v0.Schema.TensorRepresentationGroupEntry\x1as\n\x1eTensorRepresentationGroupEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12@\n\x05value\x18\x02 \x01(\x0b\x32\x31.tensorflow.metadata.v0.TensorRepresentationGroup:\x02\x38\x01\"\xdf\x0b\n\x07\x46\x65\x61ture\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x16\n\ndeprecated\x18\x02 \x01(\x08\x42\x02\x18\x01\x12;\n\x08presence\x18\x0e \x01(\x0b\x32\'.tensorflow.metadata.v0.FeaturePresenceH\x00\x12L\n\x0egroup_presence\x18\x11 \x01(\x0b\x32\x32.tensorflow.metadata.v0.FeaturePresenceWithinGroupH\x00\x12\x33\n\x05shape\x18\x17 \x01(\x0b\x32\".tensorflow.metadata.v0.FixedShapeH\x01\x12\x39\n\x0bvalue_count\x18\x05 \x01(\x0b\x32\".tensorflow.metadata.v0.ValueCountH\x01\x12\x31\n\x04type\x18\x06 \x01(\x0e\x32#.tensorflow.metadata.v0.FeatureType\x12\x10\n\x06\x64omain\x18\x07 \x01(\tH\x02\x12\x37\n\nint_domain\x18\t \x01(\x0b\x32!.tensorflow.metadata.v0.IntDomainH\x02\x12;\n\x0c\x66loat_domain\x18\n \x01(\x0b\x32#.tensorflow.metadata.v0.FloatDomainH\x02\x12=\n\rstring_domain\x18\x0b \x01(\x0b\x32$.tensorflow.metadata.v0.StringDomainH\x02\x12\x39\n\x0b\x62ool_domain\x18\r \x01(\x0b\x32\".tensorflow.metadata.v0.BoolDomainH\x02\x12=\n\rstruct_domain\x18\x1d \x01(\x0b\x32$.tensorflow.metadata.v0.StructDomainH\x02\x12P\n\x17natural_language_domain\x18\x18 \x01(\x0b\x32-.tensorflow.metadata.v0.NaturalLanguageDomainH\x02\x12;\n\x0cimage_domain\x18\x19 \x01(\x0b\x32#.tensorflow.metadata.v0.ImageDomainH\x02\x12\x37\n\nmid_domain\x18\x1a \x01(\x0b\x32!.tensorflow.metadata.v0.MIDDomainH\x02\x12\x37\n\nurl_domain\x18\x1b \x01(\x0b\x32!.tensorflow.metadata.v0.URLDomainH\x02\x12\x39\n\x0btime_domain\x18\x1c \x01(\x0b\x32\".tensorflow.metadata.v0.TimeDomainH\x02\x12\x45\n\x12time_of_day_domain\x18\x1e \x01(\x0b\x32\'.tensorflow.metadata.v0.TimeOfDayDomainH\x02\x12Q\n\x18\x64istribution_constraints\x18\x0f \x01(\x0b\x32/.tensorflow.metadata.v0.DistributionConstraints\x12\x36\n\nannotation\x18\x10 \x01(\x0b\x32\".tensorflow.metadata.v0.Annotation\x12\x42\n\x0fskew_comparator\x18\x12 \x01(\x0b\x32).tensorflow.metadata.v0.FeatureComparator\x12\x43\n\x10\x64rift_comparator\x18\x15 \x01(\x0b\x32).tensorflow.metadata.v0.FeatureComparator\x12\x16\n\x0ein_environment\x18\x14 \x03(\t\x12\x1a\n\x12not_in_environment\x18\x13 \x03(\t\x12?\n\x0flifecycle_stage\x18\x16 \x01(\x0e\x32&.tensorflow.metadata.v0.LifecycleStageB\x16\n\x14presence_constraintsB\x0c\n\nshape_typeB\r\n\x0b\x64omain_info\"X\n\nAnnotation\x12\x0b\n\x03tag\x18\x01 \x03(\t\x12\x0f\n\x07\x63omment\x18\x02 \x03(\t\x12,\n\x0e\x65xtra_metadata\x18\x03 \x03(\x0b\x32\x14.google.protobuf.Any\"X\n\x16NumericValueComparator\x12\x1e\n\x16min_fraction_threshold\x18\x01 \x01(\x01\x12\x1e\n\x16max_fraction_threshold\x18\x02 \x01(\x01\"\xe0\x01\n\x12\x44\x61tasetConstraints\x12U\n\x1dnum_examples_drift_comparator\x18\x01 \x01(\x0b\x32..tensorflow.metadata.v0.NumericValueComparator\x12W\n\x1fnum_examples_version_comparator\x18\x02 \x01(\x0b\x32..tensorflow.metadata.v0.NumericValueComparator\x12\x1a\n\x12min_examples_count\x18\x03 \x01(\x03\"d\n\nFixedShape\x12\x33\n\x03\x64im\x18\x02 \x03(\x0b\x32&.tensorflow.metadata.v0.FixedShape.Dim\x1a!\n\x03\x44im\x12\x0c\n\x04size\x18\x01 \x01(\x03\x12\x0c\n\x04name\x18\x02 \x01(\t\"&\n\nValueCount\x12\x0b\n\x03min\x18\x01 \x01(\x03\x12\x0b\n\x03max\x18\x02 \x01(\x03\"\xc5\x01\n\x0fWeightedFeature\x12\x0c\n\x04name\x18\x01 \x01(\t\x12-\n\x07\x66\x65\x61ture\x18\x02 \x01(\x0b\x32\x1c.tensorflow.metadata.v0.Path\x12\x34\n\x0eweight_feature\x18\x03 \x01(\x0b\x32\x1c.tensorflow.metadata.v0.Path\x12?\n\x0flifecycle_stage\x18\x04 \x01(\x0e\x32&.tensorflow.metadata.v0.LifecycleStage\"\x90\x04\n\rSparseFeature\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x16\n\ndeprecated\x18\x02 \x01(\x08\x42\x02\x18\x01\x12?\n\x0flifecycle_stage\x18\x07 \x01(\x0e\x32&.tensorflow.metadata.v0.LifecycleStage\x12=\n\x08presence\x18\x04 \x01(\x0b\x32\'.tensorflow.metadata.v0.FeaturePresenceB\x02\x18\x01\x12\x37\n\x0b\x64\x65nse_shape\x18\x05 \x01(\x0b\x32\".tensorflow.metadata.v0.FixedShape\x12I\n\rindex_feature\x18\x06 \x03(\x0b\x32\x32.tensorflow.metadata.v0.SparseFeature.IndexFeature\x12\x11\n\tis_sorted\x18\x08 \x01(\x08\x12I\n\rvalue_feature\x18\t \x01(\x0b\x32\x32.tensorflow.metadata.v0.SparseFeature.ValueFeature\x12\x35\n\x04type\x18\n \x01(\x0e\x32#.tensorflow.metadata.v0.FeatureTypeB\x02\x18\x01\x1a\x1c\n\x0cIndexFeature\x12\x0c\n\x04name\x18\x01 \x01(\t\x1a\x1c\n\x0cValueFeature\x12\x0c\n\x04name\x18\x01 \x01(\tJ\x04\x08\x0b\x10\x0c\"5\n\x17\x44istributionConstraints\x12\x1a\n\x0fmin_domain_mass\x18\x01 \x01(\x01:\x01\x31\"K\n\tIntDomain\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0b\n\x03min\x18\x03 \x01(\x03\x12\x0b\n\x03max\x18\x04 \x01(\x03\x12\x16\n\x0eis_categorical\x18\x05 \x01(\x08\"5\n\x0b\x46loatDomain\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0b\n\x03min\x18\x03 \x01(\x02\x12\x0b\n\x03max\x18\x04 \x01(\x02\"\x7f\n\x0cStructDomain\x12\x30\n\x07\x66\x65\x61ture\x18\x01 \x03(\x0b\x32\x1f.tensorflow.metadata.v0.Feature\x12=\n\x0esparse_feature\x18\x02 \x03(\x0b\x32%.tensorflow.metadata.v0.SparseFeature\"+\n\x0cStringDomain\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x03(\t\"C\n\nBoolDomain\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x12\n\ntrue_value\x18\x02 \x01(\t\x12\x13\n\x0b\x66\x61lse_value\x18\x03 \x01(\t\"\x17\n\x15NaturalLanguageDomain\"\r\n\x0bImageDomain\"\x0b\n\tMIDDomain\"\x0b\n\tURLDomain\"\x8e\x02\n\nTimeDomain\x12\x17\n\rstring_format\x18\x01 \x01(\tH\x00\x12N\n\x0einteger_format\x18\x02 \x01(\x0e\x32\x34.tensorflow.metadata.v0.TimeDomain.IntegerTimeFormatH\x00\"\x8c\x01\n\x11IntegerTimeFormat\x12\x12\n\x0e\x46ORMAT_UNKNOWN\x10\x00\x12\r\n\tUNIX_DAYS\x10\x05\x12\x10\n\x0cUNIX_SECONDS\x10\x01\x12\x15\n\x11UNIX_MILLISECONDS\x10\x02\x12\x15\n\x11UNIX_MICROSECONDS\x10\x03\x12\x14\n\x10UNIX_NANOSECONDS\x10\x04\x42\x08\n\x06\x66ormat\"\xd1\x01\n\x0fTimeOfDayDomain\x12\x17\n\rstring_format\x18\x01 \x01(\tH\x00\x12X\n\x0einteger_format\x18\x02 \x01(\x0e\x32>.tensorflow.metadata.v0.TimeOfDayDomain.IntegerTimeOfDayFormatH\x00\"A\n\x16IntegerTimeOfDayFormat\x12\x12\n\x0e\x46ORMAT_UNKNOWN\x10\x00\x12\x13\n\x0fPACKED_64_NANOS\x10\x01\x42\x08\n\x06\x66ormat\":\n\x0f\x46\x65\x61turePresence\x12\x14\n\x0cmin_fraction\x18\x01 \x01(\x01\x12\x11\n\tmin_count\x18\x02 \x01(\x03\".\n\x1a\x46\x65\x61turePresenceWithinGroup\x12\x10\n\x08required\x18\x01 \x01(\x08\"!\n\x0cInfinityNorm\x12\x11\n\tthreshold\x18\x01 \x01(\x01\"P\n\x11\x46\x65\x61tureComparator\x12;\n\rinfinity_norm\x18\x01 \x01(\x0b\x32$.tensorflow.metadata.v0.InfinityNorm\"\xeb\x05\n\x14TensorRepresentation\x12P\n\x0c\x64\x65nse_tensor\x18\x01 \x01(\x0b\x32\x38.tensorflow.metadata.v0.TensorRepresentation.DenseTensorH\x00\x12_\n\x14varlen_sparse_tensor\x18\x02 \x01(\x0b\x32?.tensorflow.metadata.v0.TensorRepresentation.VarLenSparseTensorH\x00\x12R\n\rsparse_tensor\x18\x03 \x01(\x0b\x32\x39.tensorflow.metadata.v0.TensorRepresentation.SparseTensorH\x00\x1ao\n\x0c\x44\x65\x66\x61ultValue\x12\x15\n\x0b\x66loat_value\x18\x01 \x01(\x01H\x00\x12\x13\n\tint_value\x18\x02 \x01(\x03H\x00\x12\x15\n\x0b\x62ytes_value\x18\x03 \x01(\x0cH\x00\x12\x14\n\nuint_value\x18\x04 \x01(\x04H\x00\x42\x06\n\x04kind\x1a\xa7\x01\n\x0b\x44\x65nseTensor\x12\x13\n\x0b\x63olumn_name\x18\x01 \x01(\t\x12\x31\n\x05shape\x18\x02 \x01(\x0b\x32\".tensorflow.metadata.v0.FixedShape\x12P\n\rdefault_value\x18\x03 \x01(\x0b\x32\x39.tensorflow.metadata.v0.TensorRepresentation.DefaultValue\x1a)\n\x12VarLenSparseTensor\x12\x13\n\x0b\x63olumn_name\x18\x01 \x01(\t\x1a~\n\x0cSparseTensor\x12\x37\n\x0b\x64\x65nse_shape\x18\x01 \x01(\x0b\x32\".tensorflow.metadata.v0.FixedShape\x12\x1a\n\x12index_column_names\x18\x02 \x03(\t\x12\x19\n\x11value_column_name\x18\x03 \x01(\tB\x06\n\x04kind\"\xf2\x01\n\x19TensorRepresentationGroup\x12j\n\x15tensor_representation\x18\x01 \x03(\x0b\x32K.tensorflow.metadata.v0.TensorRepresentationGroup.TensorRepresentationEntry\x1ai\n\x19TensorRepresentationEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12;\n\x05value\x18\x02 \x01(\x0b\x32,.tensorflow.metadata.v0.TensorRepresentation:\x02\x38\x01*u\n\x0eLifecycleStage\x12\x11\n\rUNKNOWN_STAGE\x10\x00\x12\x0b\n\x07PLANNED\x10\x01\x12\t\n\x05\x41LPHA\x10\x02\x12\x08\n\x04\x42\x45TA\x10\x03\x12\x0e\n\nPRODUCTION\x10\x04\x12\x0e\n\nDEPRECATED\x10\x05\x12\x0e\n\nDEBUG_ONLY\x10\x06*J\n\x0b\x46\x65\x61tureType\x12\x10\n\x0cTYPE_UNKNOWN\x10\x00\x12\t\n\x05\x42YTES\x10\x01\x12\x07\n\x03INT\x10\x02\x12\t\n\x05\x46LOAT\x10\x03\x12\n\n\x06STRUCT\x10\x04\x42!\n\x1aorg.tensorflow.metadata.v0P\x01\xf8\x01\x01' + , + dependencies=[google_dot_protobuf_dot_any__pb2.DESCRIPTOR,tensorflow__metadata_dot_proto_dot_v0_dot_path__pb2.DESCRIPTOR,]) + +_LIFECYCLESTAGE = _descriptor.EnumDescriptor( + name='LifecycleStage', + full_name='tensorflow.metadata.v0.LifecycleStage', + filename=None, + file=DESCRIPTOR, + values=[ + _descriptor.EnumValueDescriptor( + name='UNKNOWN_STAGE', index=0, number=0, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='PLANNED', index=1, number=1, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='ALPHA', index=2, number=2, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='BETA', index=3, number=3, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='PRODUCTION', index=4, number=4, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='DEPRECATED', index=5, number=5, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='DEBUG_ONLY', index=6, number=6, + serialized_options=None, + type=None), + ], + containing_type=None, + serialized_options=None, + serialized_start=5865, + serialized_end=5982, +) +_sym_db.RegisterEnumDescriptor(_LIFECYCLESTAGE) + +LifecycleStage = enum_type_wrapper.EnumTypeWrapper(_LIFECYCLESTAGE) +_FEATURETYPE = _descriptor.EnumDescriptor( + name='FeatureType', + full_name='tensorflow.metadata.v0.FeatureType', + filename=None, + file=DESCRIPTOR, + values=[ + _descriptor.EnumValueDescriptor( + name='TYPE_UNKNOWN', index=0, number=0, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='BYTES', index=1, number=1, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='INT', index=2, number=2, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='FLOAT', index=3, number=3, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='STRUCT', index=4, number=4, + serialized_options=None, + type=None), + ], + containing_type=None, + serialized_options=None, + serialized_start=5984, + serialized_end=6058, +) +_sym_db.RegisterEnumDescriptor(_FEATURETYPE) + +FeatureType = enum_type_wrapper.EnumTypeWrapper(_FEATURETYPE) +UNKNOWN_STAGE = 0 +PLANNED = 1 +ALPHA = 2 +BETA = 3 +PRODUCTION = 4 +DEPRECATED = 5 +DEBUG_ONLY = 6 +TYPE_UNKNOWN = 0 +BYTES = 1 +INT = 2 +FLOAT = 3 +STRUCT = 4 + + +_TIMEDOMAIN_INTEGERTIMEFORMAT = _descriptor.EnumDescriptor( + name='IntegerTimeFormat', + full_name='tensorflow.metadata.v0.TimeDomain.IntegerTimeFormat', + filename=None, + file=DESCRIPTOR, + values=[ + _descriptor.EnumValueDescriptor( + name='FORMAT_UNKNOWN', index=0, number=0, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='UNIX_DAYS', index=1, number=5, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='UNIX_SECONDS', index=2, number=1, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='UNIX_MILLISECONDS', index=3, number=2, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='UNIX_MICROSECONDS', index=4, number=3, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='UNIX_NANOSECONDS', index=5, number=4, + serialized_options=None, + type=None), + ], + containing_type=None, + serialized_options=None, + serialized_start=4281, + serialized_end=4421, +) +_sym_db.RegisterEnumDescriptor(_TIMEDOMAIN_INTEGERTIMEFORMAT) + +_TIMEOFDAYDOMAIN_INTEGERTIMEOFDAYFORMAT = _descriptor.EnumDescriptor( + name='IntegerTimeOfDayFormat', + full_name='tensorflow.metadata.v0.TimeOfDayDomain.IntegerTimeOfDayFormat', + filename=None, + file=DESCRIPTOR, + values=[ + _descriptor.EnumValueDescriptor( + name='FORMAT_UNKNOWN', index=0, number=0, + serialized_options=None, + type=None), + _descriptor.EnumValueDescriptor( + name='PACKED_64_NANOS', index=1, number=1, + serialized_options=None, + type=None), + ], + containing_type=None, + serialized_options=None, + serialized_start=4568, + serialized_end=4633, +) +_sym_db.RegisterEnumDescriptor(_TIMEOFDAYDOMAIN_INTEGERTIMEOFDAYFORMAT) + + +_SCHEMA_TENSORREPRESENTATIONGROUPENTRY = _descriptor.Descriptor( + name='TensorRepresentationGroupEntry', + full_name='tensorflow.metadata.v0.Schema.TensorRepresentationGroupEntry', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='key', full_name='tensorflow.metadata.v0.Schema.TensorRepresentationGroupEntry.key', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='value', full_name='tensorflow.metadata.v0.Schema.TensorRepresentationGroupEntry.value', index=1, + number=2, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=b'8\001', + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=761, + serialized_end=876, +) + +_SCHEMA = _descriptor.Descriptor( + name='Schema', + full_name='tensorflow.metadata.v0.Schema', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='feature', full_name='tensorflow.metadata.v0.Schema.feature', index=0, + number=1, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='sparse_feature', full_name='tensorflow.metadata.v0.Schema.sparse_feature', index=1, + number=6, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='weighted_feature', full_name='tensorflow.metadata.v0.Schema.weighted_feature', index=2, + number=12, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='string_domain', full_name='tensorflow.metadata.v0.Schema.string_domain', index=3, + number=4, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='float_domain', full_name='tensorflow.metadata.v0.Schema.float_domain', index=4, + number=9, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='int_domain', full_name='tensorflow.metadata.v0.Schema.int_domain', index=5, + number=10, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='default_environment', full_name='tensorflow.metadata.v0.Schema.default_environment', index=6, + number=5, type=9, cpp_type=9, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='annotation', full_name='tensorflow.metadata.v0.Schema.annotation', index=7, + number=8, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='dataset_constraints', full_name='tensorflow.metadata.v0.Schema.dataset_constraints', index=8, + number=11, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='tensor_representation_group', full_name='tensorflow.metadata.v0.Schema.tensor_representation_group', index=9, + number=13, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[_SCHEMA_TENSORREPRESENTATIONGROUPENTRY, ], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=138, + serialized_end=876, +) + + +_FEATURE = _descriptor.Descriptor( + name='Feature', + full_name='tensorflow.metadata.v0.Feature', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='name', full_name='tensorflow.metadata.v0.Feature.name', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='deprecated', full_name='tensorflow.metadata.v0.Feature.deprecated', index=1, + number=2, type=8, cpp_type=7, label=1, + has_default_value=False, default_value=False, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=b'\030\001', file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='presence', full_name='tensorflow.metadata.v0.Feature.presence', index=2, + number=14, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='group_presence', full_name='tensorflow.metadata.v0.Feature.group_presence', index=3, + number=17, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='shape', full_name='tensorflow.metadata.v0.Feature.shape', index=4, + number=23, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='value_count', full_name='tensorflow.metadata.v0.Feature.value_count', index=5, + number=5, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='type', full_name='tensorflow.metadata.v0.Feature.type', index=6, + number=6, type=14, cpp_type=8, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='domain', full_name='tensorflow.metadata.v0.Feature.domain', index=7, + number=7, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='int_domain', full_name='tensorflow.metadata.v0.Feature.int_domain', index=8, + number=9, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='float_domain', full_name='tensorflow.metadata.v0.Feature.float_domain', index=9, + number=10, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='string_domain', full_name='tensorflow.metadata.v0.Feature.string_domain', index=10, + number=11, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='bool_domain', full_name='tensorflow.metadata.v0.Feature.bool_domain', index=11, + number=13, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='struct_domain', full_name='tensorflow.metadata.v0.Feature.struct_domain', index=12, + number=29, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='natural_language_domain', full_name='tensorflow.metadata.v0.Feature.natural_language_domain', index=13, + number=24, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='image_domain', full_name='tensorflow.metadata.v0.Feature.image_domain', index=14, + number=25, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='mid_domain', full_name='tensorflow.metadata.v0.Feature.mid_domain', index=15, + number=26, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='url_domain', full_name='tensorflow.metadata.v0.Feature.url_domain', index=16, + number=27, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='time_domain', full_name='tensorflow.metadata.v0.Feature.time_domain', index=17, + number=28, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='time_of_day_domain', full_name='tensorflow.metadata.v0.Feature.time_of_day_domain', index=18, + number=30, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='distribution_constraints', full_name='tensorflow.metadata.v0.Feature.distribution_constraints', index=19, + number=15, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='annotation', full_name='tensorflow.metadata.v0.Feature.annotation', index=20, + number=16, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='skew_comparator', full_name='tensorflow.metadata.v0.Feature.skew_comparator', index=21, + number=18, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='drift_comparator', full_name='tensorflow.metadata.v0.Feature.drift_comparator', index=22, + number=21, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='in_environment', full_name='tensorflow.metadata.v0.Feature.in_environment', index=23, + number=20, type=9, cpp_type=9, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='not_in_environment', full_name='tensorflow.metadata.v0.Feature.not_in_environment', index=24, + number=19, type=9, cpp_type=9, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='lifecycle_stage', full_name='tensorflow.metadata.v0.Feature.lifecycle_stage', index=25, + number=22, type=14, cpp_type=8, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + _descriptor.OneofDescriptor( + name='presence_constraints', full_name='tensorflow.metadata.v0.Feature.presence_constraints', + index=0, containing_type=None, fields=[]), + _descriptor.OneofDescriptor( + name='shape_type', full_name='tensorflow.metadata.v0.Feature.shape_type', + index=1, containing_type=None, fields=[]), + _descriptor.OneofDescriptor( + name='domain_info', full_name='tensorflow.metadata.v0.Feature.domain_info', + index=2, containing_type=None, fields=[]), + ], + serialized_start=879, + serialized_end=2382, +) + + +_ANNOTATION = _descriptor.Descriptor( + name='Annotation', + full_name='tensorflow.metadata.v0.Annotation', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='tag', full_name='tensorflow.metadata.v0.Annotation.tag', index=0, + number=1, type=9, cpp_type=9, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='comment', full_name='tensorflow.metadata.v0.Annotation.comment', index=1, + number=2, type=9, cpp_type=9, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='extra_metadata', full_name='tensorflow.metadata.v0.Annotation.extra_metadata', index=2, + number=3, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=2384, + serialized_end=2472, +) + + +_NUMERICVALUECOMPARATOR = _descriptor.Descriptor( + name='NumericValueComparator', + full_name='tensorflow.metadata.v0.NumericValueComparator', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='min_fraction_threshold', full_name='tensorflow.metadata.v0.NumericValueComparator.min_fraction_threshold', index=0, + number=1, type=1, cpp_type=5, label=1, + has_default_value=False, default_value=float(0), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='max_fraction_threshold', full_name='tensorflow.metadata.v0.NumericValueComparator.max_fraction_threshold', index=1, + number=2, type=1, cpp_type=5, label=1, + has_default_value=False, default_value=float(0), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=2474, + serialized_end=2562, +) + + +_DATASETCONSTRAINTS = _descriptor.Descriptor( + name='DatasetConstraints', + full_name='tensorflow.metadata.v0.DatasetConstraints', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='num_examples_drift_comparator', full_name='tensorflow.metadata.v0.DatasetConstraints.num_examples_drift_comparator', index=0, + number=1, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='num_examples_version_comparator', full_name='tensorflow.metadata.v0.DatasetConstraints.num_examples_version_comparator', index=1, + number=2, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='min_examples_count', full_name='tensorflow.metadata.v0.DatasetConstraints.min_examples_count', index=2, + number=3, type=3, cpp_type=2, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=2565, + serialized_end=2789, +) + + +_FIXEDSHAPE_DIM = _descriptor.Descriptor( + name='Dim', + full_name='tensorflow.metadata.v0.FixedShape.Dim', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='size', full_name='tensorflow.metadata.v0.FixedShape.Dim.size', index=0, + number=1, type=3, cpp_type=2, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='name', full_name='tensorflow.metadata.v0.FixedShape.Dim.name', index=1, + number=2, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=2858, + serialized_end=2891, +) + +_FIXEDSHAPE = _descriptor.Descriptor( + name='FixedShape', + full_name='tensorflow.metadata.v0.FixedShape', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='dim', full_name='tensorflow.metadata.v0.FixedShape.dim', index=0, + number=2, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[_FIXEDSHAPE_DIM, ], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=2791, + serialized_end=2891, +) + + +_VALUECOUNT = _descriptor.Descriptor( + name='ValueCount', + full_name='tensorflow.metadata.v0.ValueCount', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='min', full_name='tensorflow.metadata.v0.ValueCount.min', index=0, + number=1, type=3, cpp_type=2, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='max', full_name='tensorflow.metadata.v0.ValueCount.max', index=1, + number=2, type=3, cpp_type=2, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=2893, + serialized_end=2931, +) + + +_WEIGHTEDFEATURE = _descriptor.Descriptor( + name='WeightedFeature', + full_name='tensorflow.metadata.v0.WeightedFeature', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='name', full_name='tensorflow.metadata.v0.WeightedFeature.name', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='feature', full_name='tensorflow.metadata.v0.WeightedFeature.feature', index=1, + number=2, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='weight_feature', full_name='tensorflow.metadata.v0.WeightedFeature.weight_feature', index=2, + number=3, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='lifecycle_stage', full_name='tensorflow.metadata.v0.WeightedFeature.lifecycle_stage', index=3, + number=4, type=14, cpp_type=8, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=2934, + serialized_end=3131, +) + + +_SPARSEFEATURE_INDEXFEATURE = _descriptor.Descriptor( + name='IndexFeature', + full_name='tensorflow.metadata.v0.SparseFeature.IndexFeature', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='name', full_name='tensorflow.metadata.v0.SparseFeature.IndexFeature.name', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=3598, + serialized_end=3626, +) + +_SPARSEFEATURE_VALUEFEATURE = _descriptor.Descriptor( + name='ValueFeature', + full_name='tensorflow.metadata.v0.SparseFeature.ValueFeature', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='name', full_name='tensorflow.metadata.v0.SparseFeature.ValueFeature.name', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=3628, + serialized_end=3656, +) + +_SPARSEFEATURE = _descriptor.Descriptor( + name='SparseFeature', + full_name='tensorflow.metadata.v0.SparseFeature', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='name', full_name='tensorflow.metadata.v0.SparseFeature.name', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='deprecated', full_name='tensorflow.metadata.v0.SparseFeature.deprecated', index=1, + number=2, type=8, cpp_type=7, label=1, + has_default_value=False, default_value=False, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=b'\030\001', file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='lifecycle_stage', full_name='tensorflow.metadata.v0.SparseFeature.lifecycle_stage', index=2, + number=7, type=14, cpp_type=8, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='presence', full_name='tensorflow.metadata.v0.SparseFeature.presence', index=3, + number=4, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=b'\030\001', file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='dense_shape', full_name='tensorflow.metadata.v0.SparseFeature.dense_shape', index=4, + number=5, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='index_feature', full_name='tensorflow.metadata.v0.SparseFeature.index_feature', index=5, + number=6, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='is_sorted', full_name='tensorflow.metadata.v0.SparseFeature.is_sorted', index=6, + number=8, type=8, cpp_type=7, label=1, + has_default_value=False, default_value=False, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='value_feature', full_name='tensorflow.metadata.v0.SparseFeature.value_feature', index=7, + number=9, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='type', full_name='tensorflow.metadata.v0.SparseFeature.type', index=8, + number=10, type=14, cpp_type=8, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=b'\030\001', file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[_SPARSEFEATURE_INDEXFEATURE, _SPARSEFEATURE_VALUEFEATURE, ], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=3134, + serialized_end=3662, +) + + +_DISTRIBUTIONCONSTRAINTS = _descriptor.Descriptor( + name='DistributionConstraints', + full_name='tensorflow.metadata.v0.DistributionConstraints', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='min_domain_mass', full_name='tensorflow.metadata.v0.DistributionConstraints.min_domain_mass', index=0, + number=1, type=1, cpp_type=5, label=1, + has_default_value=True, default_value=float(1), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=3664, + serialized_end=3717, +) + + +_INTDOMAIN = _descriptor.Descriptor( + name='IntDomain', + full_name='tensorflow.metadata.v0.IntDomain', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='name', full_name='tensorflow.metadata.v0.IntDomain.name', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='min', full_name='tensorflow.metadata.v0.IntDomain.min', index=1, + number=3, type=3, cpp_type=2, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='max', full_name='tensorflow.metadata.v0.IntDomain.max', index=2, + number=4, type=3, cpp_type=2, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='is_categorical', full_name='tensorflow.metadata.v0.IntDomain.is_categorical', index=3, + number=5, type=8, cpp_type=7, label=1, + has_default_value=False, default_value=False, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=3719, + serialized_end=3794, +) + + +_FLOATDOMAIN = _descriptor.Descriptor( + name='FloatDomain', + full_name='tensorflow.metadata.v0.FloatDomain', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='name', full_name='tensorflow.metadata.v0.FloatDomain.name', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='min', full_name='tensorflow.metadata.v0.FloatDomain.min', index=1, + number=3, type=2, cpp_type=6, label=1, + has_default_value=False, default_value=float(0), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='max', full_name='tensorflow.metadata.v0.FloatDomain.max', index=2, + number=4, type=2, cpp_type=6, label=1, + has_default_value=False, default_value=float(0), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=3796, + serialized_end=3849, +) + + +_STRUCTDOMAIN = _descriptor.Descriptor( + name='StructDomain', + full_name='tensorflow.metadata.v0.StructDomain', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='feature', full_name='tensorflow.metadata.v0.StructDomain.feature', index=0, + number=1, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='sparse_feature', full_name='tensorflow.metadata.v0.StructDomain.sparse_feature', index=1, + number=2, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=3851, + serialized_end=3978, +) + + +_STRINGDOMAIN = _descriptor.Descriptor( + name='StringDomain', + full_name='tensorflow.metadata.v0.StringDomain', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='name', full_name='tensorflow.metadata.v0.StringDomain.name', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='value', full_name='tensorflow.metadata.v0.StringDomain.value', index=1, + number=2, type=9, cpp_type=9, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=3980, + serialized_end=4023, +) + + +_BOOLDOMAIN = _descriptor.Descriptor( + name='BoolDomain', + full_name='tensorflow.metadata.v0.BoolDomain', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='name', full_name='tensorflow.metadata.v0.BoolDomain.name', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='true_value', full_name='tensorflow.metadata.v0.BoolDomain.true_value', index=1, + number=2, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='false_value', full_name='tensorflow.metadata.v0.BoolDomain.false_value', index=2, + number=3, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=4025, + serialized_end=4092, +) + + +_NATURALLANGUAGEDOMAIN = _descriptor.Descriptor( + name='NaturalLanguageDomain', + full_name='tensorflow.metadata.v0.NaturalLanguageDomain', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=4094, + serialized_end=4117, +) + + +_IMAGEDOMAIN = _descriptor.Descriptor( + name='ImageDomain', + full_name='tensorflow.metadata.v0.ImageDomain', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=4119, + serialized_end=4132, +) + + +_MIDDOMAIN = _descriptor.Descriptor( + name='MIDDomain', + full_name='tensorflow.metadata.v0.MIDDomain', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=4134, + serialized_end=4145, +) + + +_URLDOMAIN = _descriptor.Descriptor( + name='URLDomain', + full_name='tensorflow.metadata.v0.URLDomain', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=4147, + serialized_end=4158, +) + + +_TIMEDOMAIN = _descriptor.Descriptor( + name='TimeDomain', + full_name='tensorflow.metadata.v0.TimeDomain', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='string_format', full_name='tensorflow.metadata.v0.TimeDomain.string_format', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='integer_format', full_name='tensorflow.metadata.v0.TimeDomain.integer_format', index=1, + number=2, type=14, cpp_type=8, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + _TIMEDOMAIN_INTEGERTIMEFORMAT, + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + _descriptor.OneofDescriptor( + name='format', full_name='tensorflow.metadata.v0.TimeDomain.format', + index=0, containing_type=None, fields=[]), + ], + serialized_start=4161, + serialized_end=4431, +) + + +_TIMEOFDAYDOMAIN = _descriptor.Descriptor( + name='TimeOfDayDomain', + full_name='tensorflow.metadata.v0.TimeOfDayDomain', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='string_format', full_name='tensorflow.metadata.v0.TimeOfDayDomain.string_format', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='integer_format', full_name='tensorflow.metadata.v0.TimeOfDayDomain.integer_format', index=1, + number=2, type=14, cpp_type=8, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + _TIMEOFDAYDOMAIN_INTEGERTIMEOFDAYFORMAT, + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + _descriptor.OneofDescriptor( + name='format', full_name='tensorflow.metadata.v0.TimeOfDayDomain.format', + index=0, containing_type=None, fields=[]), + ], + serialized_start=4434, + serialized_end=4643, +) + + +_FEATUREPRESENCE = _descriptor.Descriptor( + name='FeaturePresence', + full_name='tensorflow.metadata.v0.FeaturePresence', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='min_fraction', full_name='tensorflow.metadata.v0.FeaturePresence.min_fraction', index=0, + number=1, type=1, cpp_type=5, label=1, + has_default_value=False, default_value=float(0), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='min_count', full_name='tensorflow.metadata.v0.FeaturePresence.min_count', index=1, + number=2, type=3, cpp_type=2, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=4645, + serialized_end=4703, +) + + +_FEATUREPRESENCEWITHINGROUP = _descriptor.Descriptor( + name='FeaturePresenceWithinGroup', + full_name='tensorflow.metadata.v0.FeaturePresenceWithinGroup', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='required', full_name='tensorflow.metadata.v0.FeaturePresenceWithinGroup.required', index=0, + number=1, type=8, cpp_type=7, label=1, + has_default_value=False, default_value=False, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=4705, + serialized_end=4751, +) + + +_INFINITYNORM = _descriptor.Descriptor( + name='InfinityNorm', + full_name='tensorflow.metadata.v0.InfinityNorm', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='threshold', full_name='tensorflow.metadata.v0.InfinityNorm.threshold', index=0, + number=1, type=1, cpp_type=5, label=1, + has_default_value=False, default_value=float(0), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=4753, + serialized_end=4786, +) + + +_FEATURECOMPARATOR = _descriptor.Descriptor( + name='FeatureComparator', + full_name='tensorflow.metadata.v0.FeatureComparator', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='infinity_norm', full_name='tensorflow.metadata.v0.FeatureComparator.infinity_norm', index=0, + number=1, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=4788, + serialized_end=4868, +) + + +_TENSORREPRESENTATION_DEFAULTVALUE = _descriptor.Descriptor( + name='DefaultValue', + full_name='tensorflow.metadata.v0.TensorRepresentation.DefaultValue', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='float_value', full_name='tensorflow.metadata.v0.TensorRepresentation.DefaultValue.float_value', index=0, + number=1, type=1, cpp_type=5, label=1, + has_default_value=False, default_value=float(0), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='int_value', full_name='tensorflow.metadata.v0.TensorRepresentation.DefaultValue.int_value', index=1, + number=2, type=3, cpp_type=2, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='bytes_value', full_name='tensorflow.metadata.v0.TensorRepresentation.DefaultValue.bytes_value', index=2, + number=3, type=12, cpp_type=9, label=1, + has_default_value=False, default_value=b"", + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='uint_value', full_name='tensorflow.metadata.v0.TensorRepresentation.DefaultValue.uint_value', index=3, + number=4, type=4, cpp_type=4, label=1, + has_default_value=False, default_value=0, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + _descriptor.OneofDescriptor( + name='kind', full_name='tensorflow.metadata.v0.TensorRepresentation.DefaultValue.kind', + index=0, containing_type=None, fields=[]), + ], + serialized_start=5158, + serialized_end=5269, +) + +_TENSORREPRESENTATION_DENSETENSOR = _descriptor.Descriptor( + name='DenseTensor', + full_name='tensorflow.metadata.v0.TensorRepresentation.DenseTensor', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='column_name', full_name='tensorflow.metadata.v0.TensorRepresentation.DenseTensor.column_name', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='shape', full_name='tensorflow.metadata.v0.TensorRepresentation.DenseTensor.shape', index=1, + number=2, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='default_value', full_name='tensorflow.metadata.v0.TensorRepresentation.DenseTensor.default_value', index=2, + number=3, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=5272, + serialized_end=5439, +) + +_TENSORREPRESENTATION_VARLENSPARSETENSOR = _descriptor.Descriptor( + name='VarLenSparseTensor', + full_name='tensorflow.metadata.v0.TensorRepresentation.VarLenSparseTensor', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='column_name', full_name='tensorflow.metadata.v0.TensorRepresentation.VarLenSparseTensor.column_name', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=5441, + serialized_end=5482, +) + +_TENSORREPRESENTATION_SPARSETENSOR = _descriptor.Descriptor( + name='SparseTensor', + full_name='tensorflow.metadata.v0.TensorRepresentation.SparseTensor', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='dense_shape', full_name='tensorflow.metadata.v0.TensorRepresentation.SparseTensor.dense_shape', index=0, + number=1, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='index_column_names', full_name='tensorflow.metadata.v0.TensorRepresentation.SparseTensor.index_column_names', index=1, + number=2, type=9, cpp_type=9, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='value_column_name', full_name='tensorflow.metadata.v0.TensorRepresentation.SparseTensor.value_column_name', index=2, + number=3, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=5484, + serialized_end=5610, +) + +_TENSORREPRESENTATION = _descriptor.Descriptor( + name='TensorRepresentation', + full_name='tensorflow.metadata.v0.TensorRepresentation', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='dense_tensor', full_name='tensorflow.metadata.v0.TensorRepresentation.dense_tensor', index=0, + number=1, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='varlen_sparse_tensor', full_name='tensorflow.metadata.v0.TensorRepresentation.varlen_sparse_tensor', index=1, + number=2, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='sparse_tensor', full_name='tensorflow.metadata.v0.TensorRepresentation.sparse_tensor', index=2, + number=3, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[_TENSORREPRESENTATION_DEFAULTVALUE, _TENSORREPRESENTATION_DENSETENSOR, _TENSORREPRESENTATION_VARLENSPARSETENSOR, _TENSORREPRESENTATION_SPARSETENSOR, ], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + _descriptor.OneofDescriptor( + name='kind', full_name='tensorflow.metadata.v0.TensorRepresentation.kind', + index=0, containing_type=None, fields=[]), + ], + serialized_start=4871, + serialized_end=5618, +) + + +_TENSORREPRESENTATIONGROUP_TENSORREPRESENTATIONENTRY = _descriptor.Descriptor( + name='TensorRepresentationEntry', + full_name='tensorflow.metadata.v0.TensorRepresentationGroup.TensorRepresentationEntry', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='key', full_name='tensorflow.metadata.v0.TensorRepresentationGroup.TensorRepresentationEntry.key', index=0, + number=1, type=9, cpp_type=9, label=1, + has_default_value=False, default_value=b"".decode('utf-8'), + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + _descriptor.FieldDescriptor( + name='value', full_name='tensorflow.metadata.v0.TensorRepresentationGroup.TensorRepresentationEntry.value', index=1, + number=2, type=11, cpp_type=10, label=1, + has_default_value=False, default_value=None, + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[], + enum_types=[ + ], + serialized_options=b'8\001', + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=5758, + serialized_end=5863, +) + +_TENSORREPRESENTATIONGROUP = _descriptor.Descriptor( + name='TensorRepresentationGroup', + full_name='tensorflow.metadata.v0.TensorRepresentationGroup', + filename=None, + file=DESCRIPTOR, + containing_type=None, + fields=[ + _descriptor.FieldDescriptor( + name='tensor_representation', full_name='tensorflow.metadata.v0.TensorRepresentationGroup.tensor_representation', index=0, + number=1, type=11, cpp_type=10, label=3, + has_default_value=False, default_value=[], + message_type=None, enum_type=None, containing_type=None, + is_extension=False, extension_scope=None, + serialized_options=None, file=DESCRIPTOR), + ], + extensions=[ + ], + nested_types=[_TENSORREPRESENTATIONGROUP_TENSORREPRESENTATIONENTRY, ], + enum_types=[ + ], + serialized_options=None, + is_extendable=False, + syntax='proto2', + extension_ranges=[], + oneofs=[ + ], + serialized_start=5621, + serialized_end=5863, +) + +_SCHEMA_TENSORREPRESENTATIONGROUPENTRY.fields_by_name['value'].message_type = _TENSORREPRESENTATIONGROUP +_SCHEMA_TENSORREPRESENTATIONGROUPENTRY.containing_type = _SCHEMA +_SCHEMA.fields_by_name['feature'].message_type = _FEATURE +_SCHEMA.fields_by_name['sparse_feature'].message_type = _SPARSEFEATURE +_SCHEMA.fields_by_name['weighted_feature'].message_type = _WEIGHTEDFEATURE +_SCHEMA.fields_by_name['string_domain'].message_type = _STRINGDOMAIN +_SCHEMA.fields_by_name['float_domain'].message_type = _FLOATDOMAIN +_SCHEMA.fields_by_name['int_domain'].message_type = _INTDOMAIN +_SCHEMA.fields_by_name['annotation'].message_type = _ANNOTATION +_SCHEMA.fields_by_name['dataset_constraints'].message_type = _DATASETCONSTRAINTS +_SCHEMA.fields_by_name['tensor_representation_group'].message_type = _SCHEMA_TENSORREPRESENTATIONGROUPENTRY +_FEATURE.fields_by_name['presence'].message_type = _FEATUREPRESENCE +_FEATURE.fields_by_name['group_presence'].message_type = _FEATUREPRESENCEWITHINGROUP +_FEATURE.fields_by_name['shape'].message_type = _FIXEDSHAPE +_FEATURE.fields_by_name['value_count'].message_type = _VALUECOUNT +_FEATURE.fields_by_name['type'].enum_type = _FEATURETYPE +_FEATURE.fields_by_name['int_domain'].message_type = _INTDOMAIN +_FEATURE.fields_by_name['float_domain'].message_type = _FLOATDOMAIN +_FEATURE.fields_by_name['string_domain'].message_type = _STRINGDOMAIN +_FEATURE.fields_by_name['bool_domain'].message_type = _BOOLDOMAIN +_FEATURE.fields_by_name['struct_domain'].message_type = _STRUCTDOMAIN +_FEATURE.fields_by_name['natural_language_domain'].message_type = _NATURALLANGUAGEDOMAIN +_FEATURE.fields_by_name['image_domain'].message_type = _IMAGEDOMAIN +_FEATURE.fields_by_name['mid_domain'].message_type = _MIDDOMAIN +_FEATURE.fields_by_name['url_domain'].message_type = _URLDOMAIN +_FEATURE.fields_by_name['time_domain'].message_type = _TIMEDOMAIN +_FEATURE.fields_by_name['time_of_day_domain'].message_type = _TIMEOFDAYDOMAIN +_FEATURE.fields_by_name['distribution_constraints'].message_type = _DISTRIBUTIONCONSTRAINTS +_FEATURE.fields_by_name['annotation'].message_type = _ANNOTATION +_FEATURE.fields_by_name['skew_comparator'].message_type = _FEATURECOMPARATOR +_FEATURE.fields_by_name['drift_comparator'].message_type = _FEATURECOMPARATOR +_FEATURE.fields_by_name['lifecycle_stage'].enum_type = _LIFECYCLESTAGE +_FEATURE.oneofs_by_name['presence_constraints'].fields.append( + _FEATURE.fields_by_name['presence']) +_FEATURE.fields_by_name['presence'].containing_oneof = _FEATURE.oneofs_by_name['presence_constraints'] +_FEATURE.oneofs_by_name['presence_constraints'].fields.append( + _FEATURE.fields_by_name['group_presence']) +_FEATURE.fields_by_name['group_presence'].containing_oneof = _FEATURE.oneofs_by_name['presence_constraints'] +_FEATURE.oneofs_by_name['shape_type'].fields.append( + _FEATURE.fields_by_name['shape']) +_FEATURE.fields_by_name['shape'].containing_oneof = _FEATURE.oneofs_by_name['shape_type'] +_FEATURE.oneofs_by_name['shape_type'].fields.append( + _FEATURE.fields_by_name['value_count']) +_FEATURE.fields_by_name['value_count'].containing_oneof = _FEATURE.oneofs_by_name['shape_type'] +_FEATURE.oneofs_by_name['domain_info'].fields.append( + _FEATURE.fields_by_name['domain']) +_FEATURE.fields_by_name['domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] +_FEATURE.oneofs_by_name['domain_info'].fields.append( + _FEATURE.fields_by_name['int_domain']) +_FEATURE.fields_by_name['int_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] +_FEATURE.oneofs_by_name['domain_info'].fields.append( + _FEATURE.fields_by_name['float_domain']) +_FEATURE.fields_by_name['float_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] +_FEATURE.oneofs_by_name['domain_info'].fields.append( + _FEATURE.fields_by_name['string_domain']) +_FEATURE.fields_by_name['string_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] +_FEATURE.oneofs_by_name['domain_info'].fields.append( + _FEATURE.fields_by_name['bool_domain']) +_FEATURE.fields_by_name['bool_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] +_FEATURE.oneofs_by_name['domain_info'].fields.append( + _FEATURE.fields_by_name['struct_domain']) +_FEATURE.fields_by_name['struct_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] +_FEATURE.oneofs_by_name['domain_info'].fields.append( + _FEATURE.fields_by_name['natural_language_domain']) +_FEATURE.fields_by_name['natural_language_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] +_FEATURE.oneofs_by_name['domain_info'].fields.append( + _FEATURE.fields_by_name['image_domain']) +_FEATURE.fields_by_name['image_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] +_FEATURE.oneofs_by_name['domain_info'].fields.append( + _FEATURE.fields_by_name['mid_domain']) +_FEATURE.fields_by_name['mid_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] +_FEATURE.oneofs_by_name['domain_info'].fields.append( + _FEATURE.fields_by_name['url_domain']) +_FEATURE.fields_by_name['url_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] +_FEATURE.oneofs_by_name['domain_info'].fields.append( + _FEATURE.fields_by_name['time_domain']) +_FEATURE.fields_by_name['time_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] +_FEATURE.oneofs_by_name['domain_info'].fields.append( + _FEATURE.fields_by_name['time_of_day_domain']) +_FEATURE.fields_by_name['time_of_day_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] +_ANNOTATION.fields_by_name['extra_metadata'].message_type = google_dot_protobuf_dot_any__pb2._ANY +_DATASETCONSTRAINTS.fields_by_name['num_examples_drift_comparator'].message_type = _NUMERICVALUECOMPARATOR +_DATASETCONSTRAINTS.fields_by_name['num_examples_version_comparator'].message_type = _NUMERICVALUECOMPARATOR +_FIXEDSHAPE_DIM.containing_type = _FIXEDSHAPE +_FIXEDSHAPE.fields_by_name['dim'].message_type = _FIXEDSHAPE_DIM +_WEIGHTEDFEATURE.fields_by_name['feature'].message_type = tensorflow__metadata_dot_proto_dot_v0_dot_path__pb2._PATH +_WEIGHTEDFEATURE.fields_by_name['weight_feature'].message_type = tensorflow__metadata_dot_proto_dot_v0_dot_path__pb2._PATH +_WEIGHTEDFEATURE.fields_by_name['lifecycle_stage'].enum_type = _LIFECYCLESTAGE +_SPARSEFEATURE_INDEXFEATURE.containing_type = _SPARSEFEATURE +_SPARSEFEATURE_VALUEFEATURE.containing_type = _SPARSEFEATURE +_SPARSEFEATURE.fields_by_name['lifecycle_stage'].enum_type = _LIFECYCLESTAGE +_SPARSEFEATURE.fields_by_name['presence'].message_type = _FEATUREPRESENCE +_SPARSEFEATURE.fields_by_name['dense_shape'].message_type = _FIXEDSHAPE +_SPARSEFEATURE.fields_by_name['index_feature'].message_type = _SPARSEFEATURE_INDEXFEATURE +_SPARSEFEATURE.fields_by_name['value_feature'].message_type = _SPARSEFEATURE_VALUEFEATURE +_SPARSEFEATURE.fields_by_name['type'].enum_type = _FEATURETYPE +_STRUCTDOMAIN.fields_by_name['feature'].message_type = _FEATURE +_STRUCTDOMAIN.fields_by_name['sparse_feature'].message_type = _SPARSEFEATURE +_TIMEDOMAIN.fields_by_name['integer_format'].enum_type = _TIMEDOMAIN_INTEGERTIMEFORMAT +_TIMEDOMAIN_INTEGERTIMEFORMAT.containing_type = _TIMEDOMAIN +_TIMEDOMAIN.oneofs_by_name['format'].fields.append( + _TIMEDOMAIN.fields_by_name['string_format']) +_TIMEDOMAIN.fields_by_name['string_format'].containing_oneof = _TIMEDOMAIN.oneofs_by_name['format'] +_TIMEDOMAIN.oneofs_by_name['format'].fields.append( + _TIMEDOMAIN.fields_by_name['integer_format']) +_TIMEDOMAIN.fields_by_name['integer_format'].containing_oneof = _TIMEDOMAIN.oneofs_by_name['format'] +_TIMEOFDAYDOMAIN.fields_by_name['integer_format'].enum_type = _TIMEOFDAYDOMAIN_INTEGERTIMEOFDAYFORMAT +_TIMEOFDAYDOMAIN_INTEGERTIMEOFDAYFORMAT.containing_type = _TIMEOFDAYDOMAIN +_TIMEOFDAYDOMAIN.oneofs_by_name['format'].fields.append( + _TIMEOFDAYDOMAIN.fields_by_name['string_format']) +_TIMEOFDAYDOMAIN.fields_by_name['string_format'].containing_oneof = _TIMEOFDAYDOMAIN.oneofs_by_name['format'] +_TIMEOFDAYDOMAIN.oneofs_by_name['format'].fields.append( + _TIMEOFDAYDOMAIN.fields_by_name['integer_format']) +_TIMEOFDAYDOMAIN.fields_by_name['integer_format'].containing_oneof = _TIMEOFDAYDOMAIN.oneofs_by_name['format'] +_FEATURECOMPARATOR.fields_by_name['infinity_norm'].message_type = _INFINITYNORM +_TENSORREPRESENTATION_DEFAULTVALUE.containing_type = _TENSORREPRESENTATION +_TENSORREPRESENTATION_DEFAULTVALUE.oneofs_by_name['kind'].fields.append( + _TENSORREPRESENTATION_DEFAULTVALUE.fields_by_name['float_value']) +_TENSORREPRESENTATION_DEFAULTVALUE.fields_by_name['float_value'].containing_oneof = _TENSORREPRESENTATION_DEFAULTVALUE.oneofs_by_name['kind'] +_TENSORREPRESENTATION_DEFAULTVALUE.oneofs_by_name['kind'].fields.append( + _TENSORREPRESENTATION_DEFAULTVALUE.fields_by_name['int_value']) +_TENSORREPRESENTATION_DEFAULTVALUE.fields_by_name['int_value'].containing_oneof = _TENSORREPRESENTATION_DEFAULTVALUE.oneofs_by_name['kind'] +_TENSORREPRESENTATION_DEFAULTVALUE.oneofs_by_name['kind'].fields.append( + _TENSORREPRESENTATION_DEFAULTVALUE.fields_by_name['bytes_value']) +_TENSORREPRESENTATION_DEFAULTVALUE.fields_by_name['bytes_value'].containing_oneof = _TENSORREPRESENTATION_DEFAULTVALUE.oneofs_by_name['kind'] +_TENSORREPRESENTATION_DEFAULTVALUE.oneofs_by_name['kind'].fields.append( + _TENSORREPRESENTATION_DEFAULTVALUE.fields_by_name['uint_value']) +_TENSORREPRESENTATION_DEFAULTVALUE.fields_by_name['uint_value'].containing_oneof = _TENSORREPRESENTATION_DEFAULTVALUE.oneofs_by_name['kind'] +_TENSORREPRESENTATION_DENSETENSOR.fields_by_name['shape'].message_type = _FIXEDSHAPE +_TENSORREPRESENTATION_DENSETENSOR.fields_by_name['default_value'].message_type = _TENSORREPRESENTATION_DEFAULTVALUE +_TENSORREPRESENTATION_DENSETENSOR.containing_type = _TENSORREPRESENTATION +_TENSORREPRESENTATION_VARLENSPARSETENSOR.containing_type = _TENSORREPRESENTATION +_TENSORREPRESENTATION_SPARSETENSOR.fields_by_name['dense_shape'].message_type = _FIXEDSHAPE +_TENSORREPRESENTATION_SPARSETENSOR.containing_type = _TENSORREPRESENTATION +_TENSORREPRESENTATION.fields_by_name['dense_tensor'].message_type = _TENSORREPRESENTATION_DENSETENSOR +_TENSORREPRESENTATION.fields_by_name['varlen_sparse_tensor'].message_type = _TENSORREPRESENTATION_VARLENSPARSETENSOR +_TENSORREPRESENTATION.fields_by_name['sparse_tensor'].message_type = _TENSORREPRESENTATION_SPARSETENSOR +_TENSORREPRESENTATION.oneofs_by_name['kind'].fields.append( + _TENSORREPRESENTATION.fields_by_name['dense_tensor']) +_TENSORREPRESENTATION.fields_by_name['dense_tensor'].containing_oneof = _TENSORREPRESENTATION.oneofs_by_name['kind'] +_TENSORREPRESENTATION.oneofs_by_name['kind'].fields.append( + _TENSORREPRESENTATION.fields_by_name['varlen_sparse_tensor']) +_TENSORREPRESENTATION.fields_by_name['varlen_sparse_tensor'].containing_oneof = _TENSORREPRESENTATION.oneofs_by_name['kind'] +_TENSORREPRESENTATION.oneofs_by_name['kind'].fields.append( + _TENSORREPRESENTATION.fields_by_name['sparse_tensor']) +_TENSORREPRESENTATION.fields_by_name['sparse_tensor'].containing_oneof = _TENSORREPRESENTATION.oneofs_by_name['kind'] +_TENSORREPRESENTATIONGROUP_TENSORREPRESENTATIONENTRY.fields_by_name['value'].message_type = _TENSORREPRESENTATION +_TENSORREPRESENTATIONGROUP_TENSORREPRESENTATIONENTRY.containing_type = _TENSORREPRESENTATIONGROUP +_TENSORREPRESENTATIONGROUP.fields_by_name['tensor_representation'].message_type = _TENSORREPRESENTATIONGROUP_TENSORREPRESENTATIONENTRY +DESCRIPTOR.message_types_by_name['Schema'] = _SCHEMA +DESCRIPTOR.message_types_by_name['Feature'] = _FEATURE +DESCRIPTOR.message_types_by_name['Annotation'] = _ANNOTATION +DESCRIPTOR.message_types_by_name['NumericValueComparator'] = _NUMERICVALUECOMPARATOR +DESCRIPTOR.message_types_by_name['DatasetConstraints'] = _DATASETCONSTRAINTS +DESCRIPTOR.message_types_by_name['FixedShape'] = _FIXEDSHAPE +DESCRIPTOR.message_types_by_name['ValueCount'] = _VALUECOUNT +DESCRIPTOR.message_types_by_name['WeightedFeature'] = _WEIGHTEDFEATURE +DESCRIPTOR.message_types_by_name['SparseFeature'] = _SPARSEFEATURE +DESCRIPTOR.message_types_by_name['DistributionConstraints'] = _DISTRIBUTIONCONSTRAINTS +DESCRIPTOR.message_types_by_name['IntDomain'] = _INTDOMAIN +DESCRIPTOR.message_types_by_name['FloatDomain'] = _FLOATDOMAIN +DESCRIPTOR.message_types_by_name['StructDomain'] = _STRUCTDOMAIN +DESCRIPTOR.message_types_by_name['StringDomain'] = _STRINGDOMAIN +DESCRIPTOR.message_types_by_name['BoolDomain'] = _BOOLDOMAIN +DESCRIPTOR.message_types_by_name['NaturalLanguageDomain'] = _NATURALLANGUAGEDOMAIN +DESCRIPTOR.message_types_by_name['ImageDomain'] = _IMAGEDOMAIN +DESCRIPTOR.message_types_by_name['MIDDomain'] = _MIDDOMAIN +DESCRIPTOR.message_types_by_name['URLDomain'] = _URLDOMAIN +DESCRIPTOR.message_types_by_name['TimeDomain'] = _TIMEDOMAIN +DESCRIPTOR.message_types_by_name['TimeOfDayDomain'] = _TIMEOFDAYDOMAIN +DESCRIPTOR.message_types_by_name['FeaturePresence'] = _FEATUREPRESENCE +DESCRIPTOR.message_types_by_name['FeaturePresenceWithinGroup'] = _FEATUREPRESENCEWITHINGROUP +DESCRIPTOR.message_types_by_name['InfinityNorm'] = _INFINITYNORM +DESCRIPTOR.message_types_by_name['FeatureComparator'] = _FEATURECOMPARATOR +DESCRIPTOR.message_types_by_name['TensorRepresentation'] = _TENSORREPRESENTATION +DESCRIPTOR.message_types_by_name['TensorRepresentationGroup'] = _TENSORREPRESENTATIONGROUP +DESCRIPTOR.enum_types_by_name['LifecycleStage'] = _LIFECYCLESTAGE +DESCRIPTOR.enum_types_by_name['FeatureType'] = _FEATURETYPE +_sym_db.RegisterFileDescriptor(DESCRIPTOR) + +Schema = _reflection.GeneratedProtocolMessageType('Schema', (_message.Message,), { + + 'TensorRepresentationGroupEntry' : _reflection.GeneratedProtocolMessageType('TensorRepresentationGroupEntry', (_message.Message,), { + 'DESCRIPTOR' : _SCHEMA_TENSORREPRESENTATIONGROUPENTRY, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.Schema.TensorRepresentationGroupEntry) + }) + , + 'DESCRIPTOR' : _SCHEMA, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.Schema) + }) +_sym_db.RegisterMessage(Schema) +_sym_db.RegisterMessage(Schema.TensorRepresentationGroupEntry) + +Feature = _reflection.GeneratedProtocolMessageType('Feature', (_message.Message,), { + 'DESCRIPTOR' : _FEATURE, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.Feature) + }) +_sym_db.RegisterMessage(Feature) + +Annotation = _reflection.GeneratedProtocolMessageType('Annotation', (_message.Message,), { + 'DESCRIPTOR' : _ANNOTATION, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.Annotation) + }) +_sym_db.RegisterMessage(Annotation) + +NumericValueComparator = _reflection.GeneratedProtocolMessageType('NumericValueComparator', (_message.Message,), { + 'DESCRIPTOR' : _NUMERICVALUECOMPARATOR, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.NumericValueComparator) + }) +_sym_db.RegisterMessage(NumericValueComparator) + +DatasetConstraints = _reflection.GeneratedProtocolMessageType('DatasetConstraints', (_message.Message,), { + 'DESCRIPTOR' : _DATASETCONSTRAINTS, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.DatasetConstraints) + }) +_sym_db.RegisterMessage(DatasetConstraints) + +FixedShape = _reflection.GeneratedProtocolMessageType('FixedShape', (_message.Message,), { + + 'Dim' : _reflection.GeneratedProtocolMessageType('Dim', (_message.Message,), { + 'DESCRIPTOR' : _FIXEDSHAPE_DIM, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.FixedShape.Dim) + }) + , + 'DESCRIPTOR' : _FIXEDSHAPE, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.FixedShape) + }) +_sym_db.RegisterMessage(FixedShape) +_sym_db.RegisterMessage(FixedShape.Dim) + +ValueCount = _reflection.GeneratedProtocolMessageType('ValueCount', (_message.Message,), { + 'DESCRIPTOR' : _VALUECOUNT, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.ValueCount) + }) +_sym_db.RegisterMessage(ValueCount) + +WeightedFeature = _reflection.GeneratedProtocolMessageType('WeightedFeature', (_message.Message,), { + 'DESCRIPTOR' : _WEIGHTEDFEATURE, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.WeightedFeature) + }) +_sym_db.RegisterMessage(WeightedFeature) + +SparseFeature = _reflection.GeneratedProtocolMessageType('SparseFeature', (_message.Message,), { + + 'IndexFeature' : _reflection.GeneratedProtocolMessageType('IndexFeature', (_message.Message,), { + 'DESCRIPTOR' : _SPARSEFEATURE_INDEXFEATURE, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.SparseFeature.IndexFeature) + }) + , + + 'ValueFeature' : _reflection.GeneratedProtocolMessageType('ValueFeature', (_message.Message,), { + 'DESCRIPTOR' : _SPARSEFEATURE_VALUEFEATURE, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.SparseFeature.ValueFeature) + }) + , + 'DESCRIPTOR' : _SPARSEFEATURE, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.SparseFeature) + }) +_sym_db.RegisterMessage(SparseFeature) +_sym_db.RegisterMessage(SparseFeature.IndexFeature) +_sym_db.RegisterMessage(SparseFeature.ValueFeature) + +DistributionConstraints = _reflection.GeneratedProtocolMessageType('DistributionConstraints', (_message.Message,), { + 'DESCRIPTOR' : _DISTRIBUTIONCONSTRAINTS, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.DistributionConstraints) + }) +_sym_db.RegisterMessage(DistributionConstraints) + +IntDomain = _reflection.GeneratedProtocolMessageType('IntDomain', (_message.Message,), { + 'DESCRIPTOR' : _INTDOMAIN, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.IntDomain) + }) +_sym_db.RegisterMessage(IntDomain) + +FloatDomain = _reflection.GeneratedProtocolMessageType('FloatDomain', (_message.Message,), { + 'DESCRIPTOR' : _FLOATDOMAIN, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.FloatDomain) + }) +_sym_db.RegisterMessage(FloatDomain) + +StructDomain = _reflection.GeneratedProtocolMessageType('StructDomain', (_message.Message,), { + 'DESCRIPTOR' : _STRUCTDOMAIN, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.StructDomain) + }) +_sym_db.RegisterMessage(StructDomain) + +StringDomain = _reflection.GeneratedProtocolMessageType('StringDomain', (_message.Message,), { + 'DESCRIPTOR' : _STRINGDOMAIN, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.StringDomain) + }) +_sym_db.RegisterMessage(StringDomain) + +BoolDomain = _reflection.GeneratedProtocolMessageType('BoolDomain', (_message.Message,), { + 'DESCRIPTOR' : _BOOLDOMAIN, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.BoolDomain) + }) +_sym_db.RegisterMessage(BoolDomain) + +NaturalLanguageDomain = _reflection.GeneratedProtocolMessageType('NaturalLanguageDomain', (_message.Message,), { + 'DESCRIPTOR' : _NATURALLANGUAGEDOMAIN, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.NaturalLanguageDomain) + }) +_sym_db.RegisterMessage(NaturalLanguageDomain) + +ImageDomain = _reflection.GeneratedProtocolMessageType('ImageDomain', (_message.Message,), { + 'DESCRIPTOR' : _IMAGEDOMAIN, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.ImageDomain) + }) +_sym_db.RegisterMessage(ImageDomain) + +MIDDomain = _reflection.GeneratedProtocolMessageType('MIDDomain', (_message.Message,), { + 'DESCRIPTOR' : _MIDDOMAIN, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.MIDDomain) + }) +_sym_db.RegisterMessage(MIDDomain) + +URLDomain = _reflection.GeneratedProtocolMessageType('URLDomain', (_message.Message,), { + 'DESCRIPTOR' : _URLDOMAIN, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.URLDomain) + }) +_sym_db.RegisterMessage(URLDomain) + +TimeDomain = _reflection.GeneratedProtocolMessageType('TimeDomain', (_message.Message,), { + 'DESCRIPTOR' : _TIMEDOMAIN, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TimeDomain) + }) +_sym_db.RegisterMessage(TimeDomain) + +TimeOfDayDomain = _reflection.GeneratedProtocolMessageType('TimeOfDayDomain', (_message.Message,), { + 'DESCRIPTOR' : _TIMEOFDAYDOMAIN, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TimeOfDayDomain) + }) +_sym_db.RegisterMessage(TimeOfDayDomain) + +FeaturePresence = _reflection.GeneratedProtocolMessageType('FeaturePresence', (_message.Message,), { + 'DESCRIPTOR' : _FEATUREPRESENCE, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.FeaturePresence) + }) +_sym_db.RegisterMessage(FeaturePresence) + +FeaturePresenceWithinGroup = _reflection.GeneratedProtocolMessageType('FeaturePresenceWithinGroup', (_message.Message,), { + 'DESCRIPTOR' : _FEATUREPRESENCEWITHINGROUP, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.FeaturePresenceWithinGroup) + }) +_sym_db.RegisterMessage(FeaturePresenceWithinGroup) + +InfinityNorm = _reflection.GeneratedProtocolMessageType('InfinityNorm', (_message.Message,), { + 'DESCRIPTOR' : _INFINITYNORM, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.InfinityNorm) + }) +_sym_db.RegisterMessage(InfinityNorm) + +FeatureComparator = _reflection.GeneratedProtocolMessageType('FeatureComparator', (_message.Message,), { + 'DESCRIPTOR' : _FEATURECOMPARATOR, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.FeatureComparator) + }) +_sym_db.RegisterMessage(FeatureComparator) + +TensorRepresentation = _reflection.GeneratedProtocolMessageType('TensorRepresentation', (_message.Message,), { + + 'DefaultValue' : _reflection.GeneratedProtocolMessageType('DefaultValue', (_message.Message,), { + 'DESCRIPTOR' : _TENSORREPRESENTATION_DEFAULTVALUE, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TensorRepresentation.DefaultValue) + }) + , + + 'DenseTensor' : _reflection.GeneratedProtocolMessageType('DenseTensor', (_message.Message,), { + 'DESCRIPTOR' : _TENSORREPRESENTATION_DENSETENSOR, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TensorRepresentation.DenseTensor) + }) + , + + 'VarLenSparseTensor' : _reflection.GeneratedProtocolMessageType('VarLenSparseTensor', (_message.Message,), { + 'DESCRIPTOR' : _TENSORREPRESENTATION_VARLENSPARSETENSOR, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TensorRepresentation.VarLenSparseTensor) + }) + , + + 'SparseTensor' : _reflection.GeneratedProtocolMessageType('SparseTensor', (_message.Message,), { + 'DESCRIPTOR' : _TENSORREPRESENTATION_SPARSETENSOR, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TensorRepresentation.SparseTensor) + }) + , + 'DESCRIPTOR' : _TENSORREPRESENTATION, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TensorRepresentation) + }) +_sym_db.RegisterMessage(TensorRepresentation) +_sym_db.RegisterMessage(TensorRepresentation.DefaultValue) +_sym_db.RegisterMessage(TensorRepresentation.DenseTensor) +_sym_db.RegisterMessage(TensorRepresentation.VarLenSparseTensor) +_sym_db.RegisterMessage(TensorRepresentation.SparseTensor) + +TensorRepresentationGroup = _reflection.GeneratedProtocolMessageType('TensorRepresentationGroup', (_message.Message,), { + + 'TensorRepresentationEntry' : _reflection.GeneratedProtocolMessageType('TensorRepresentationEntry', (_message.Message,), { + 'DESCRIPTOR' : _TENSORREPRESENTATIONGROUP_TENSORREPRESENTATIONENTRY, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TensorRepresentationGroup.TensorRepresentationEntry) + }) + , + 'DESCRIPTOR' : _TENSORREPRESENTATIONGROUP, + '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' + # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TensorRepresentationGroup) + }) +_sym_db.RegisterMessage(TensorRepresentationGroup) +_sym_db.RegisterMessage(TensorRepresentationGroup.TensorRepresentationEntry) + + +DESCRIPTOR._options = None +_SCHEMA_TENSORREPRESENTATIONGROUPENTRY._options = None +_FEATURE.fields_by_name['deprecated']._options = None +_SPARSEFEATURE.fields_by_name['deprecated']._options = None +_SPARSEFEATURE.fields_by_name['presence']._options = None +_SPARSEFEATURE.fields_by_name['type']._options = None +_TENSORREPRESENTATIONGROUP_TENSORREPRESENTATIONENTRY._options = None +# @@protoc_insertion_point(module_scope) diff --git a/sdk/python/tensorflow_metadata/proto/v0/schema_pb2.pyi b/sdk/python/tensorflow_metadata/proto/v0/schema_pb2.pyi new file mode 100644 index 00000000000..d684e28c0c2 --- /dev/null +++ b/sdk/python/tensorflow_metadata/proto/v0/schema_pb2.pyi @@ -0,0 +1,1063 @@ +# @generated by generate_proto_mypy_stubs.py. Do not edit! +import sys +from google.protobuf.any_pb2 import ( + Any as google___protobuf___any_pb2___Any, +) + +from google.protobuf.descriptor import ( + Descriptor as google___protobuf___descriptor___Descriptor, + EnumDescriptor as google___protobuf___descriptor___EnumDescriptor, +) + +from google.protobuf.internal.containers import ( + RepeatedCompositeFieldContainer as google___protobuf___internal___containers___RepeatedCompositeFieldContainer, + RepeatedScalarFieldContainer as google___protobuf___internal___containers___RepeatedScalarFieldContainer, +) + +from google.protobuf.message import ( + Message as google___protobuf___message___Message, +) + +from tensorflow_metadata.proto.v0.path_pb2 import ( + Path as tensorflow_metadata___proto___v0___path_pb2___Path, +) + +from typing import ( + Iterable as typing___Iterable, + List as typing___List, + Mapping as typing___Mapping, + MutableMapping as typing___MutableMapping, + Optional as typing___Optional, + Text as typing___Text, + Tuple as typing___Tuple, + Union as typing___Union, + cast as typing___cast, + overload as typing___overload, +) + +from typing_extensions import ( + Literal as typing_extensions___Literal, +) + + +builtin___bool = bool +builtin___bytes = bytes +builtin___float = float +builtin___int = int +builtin___str = str +if sys.version_info < (3,): + builtin___buffer = buffer + builtin___unicode = unicode + + +class LifecycleStage(builtin___int): + DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... + @classmethod + def Name(cls, number: builtin___int) -> builtin___str: ... + @classmethod + def Value(cls, name: builtin___str) -> 'LifecycleStage': ... + @classmethod + def keys(cls) -> typing___List[builtin___str]: ... + @classmethod + def values(cls) -> typing___List['LifecycleStage']: ... + @classmethod + def items(cls) -> typing___List[typing___Tuple[builtin___str, 'LifecycleStage']]: ... + UNKNOWN_STAGE = typing___cast('LifecycleStage', 0) + PLANNED = typing___cast('LifecycleStage', 1) + ALPHA = typing___cast('LifecycleStage', 2) + BETA = typing___cast('LifecycleStage', 3) + PRODUCTION = typing___cast('LifecycleStage', 4) + DEPRECATED = typing___cast('LifecycleStage', 5) + DEBUG_ONLY = typing___cast('LifecycleStage', 6) +UNKNOWN_STAGE = typing___cast('LifecycleStage', 0) +PLANNED = typing___cast('LifecycleStage', 1) +ALPHA = typing___cast('LifecycleStage', 2) +BETA = typing___cast('LifecycleStage', 3) +PRODUCTION = typing___cast('LifecycleStage', 4) +DEPRECATED = typing___cast('LifecycleStage', 5) +DEBUG_ONLY = typing___cast('LifecycleStage', 6) + +class FeatureType(builtin___int): + DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... + @classmethod + def Name(cls, number: builtin___int) -> builtin___str: ... + @classmethod + def Value(cls, name: builtin___str) -> 'FeatureType': ... + @classmethod + def keys(cls) -> typing___List[builtin___str]: ... + @classmethod + def values(cls) -> typing___List['FeatureType']: ... + @classmethod + def items(cls) -> typing___List[typing___Tuple[builtin___str, 'FeatureType']]: ... + TYPE_UNKNOWN = typing___cast('FeatureType', 0) + BYTES = typing___cast('FeatureType', 1) + INT = typing___cast('FeatureType', 2) + FLOAT = typing___cast('FeatureType', 3) + STRUCT = typing___cast('FeatureType', 4) +TYPE_UNKNOWN = typing___cast('FeatureType', 0) +BYTES = typing___cast('FeatureType', 1) +INT = typing___cast('FeatureType', 2) +FLOAT = typing___cast('FeatureType', 3) +STRUCT = typing___cast('FeatureType', 4) + +class Schema(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + class TensorRepresentationGroupEntry(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + key = ... # type: typing___Text + + @property + def value(self) -> TensorRepresentationGroup: ... + + def __init__(self, + *, + key : typing___Optional[typing___Text] = None, + value : typing___Optional[TensorRepresentationGroup] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> Schema.TensorRepresentationGroupEntry: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> Schema.TensorRepresentationGroupEntry: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"key",b"key",u"value",b"value"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"key",b"key",u"value",b"value"]) -> None: ... + + default_environment = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] + + @property + def feature(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[Feature]: ... + + @property + def sparse_feature(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[SparseFeature]: ... + + @property + def weighted_feature(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[WeightedFeature]: ... + + @property + def string_domain(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[StringDomain]: ... + + @property + def float_domain(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[FloatDomain]: ... + + @property + def int_domain(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[IntDomain]: ... + + @property + def annotation(self) -> Annotation: ... + + @property + def dataset_constraints(self) -> DatasetConstraints: ... + + @property + def tensor_representation_group(self) -> typing___MutableMapping[typing___Text, TensorRepresentationGroup]: ... + + def __init__(self, + *, + feature : typing___Optional[typing___Iterable[Feature]] = None, + sparse_feature : typing___Optional[typing___Iterable[SparseFeature]] = None, + weighted_feature : typing___Optional[typing___Iterable[WeightedFeature]] = None, + string_domain : typing___Optional[typing___Iterable[StringDomain]] = None, + float_domain : typing___Optional[typing___Iterable[FloatDomain]] = None, + int_domain : typing___Optional[typing___Iterable[IntDomain]] = None, + default_environment : typing___Optional[typing___Iterable[typing___Text]] = None, + annotation : typing___Optional[Annotation] = None, + dataset_constraints : typing___Optional[DatasetConstraints] = None, + tensor_representation_group : typing___Optional[typing___Mapping[typing___Text, TensorRepresentationGroup]] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> Schema: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> Schema: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"annotation",b"annotation",u"dataset_constraints",b"dataset_constraints"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"annotation",b"annotation",u"dataset_constraints",b"dataset_constraints",u"default_environment",b"default_environment",u"feature",b"feature",u"float_domain",b"float_domain",u"int_domain",b"int_domain",u"sparse_feature",b"sparse_feature",u"string_domain",b"string_domain",u"tensor_representation_group",b"tensor_representation_group",u"weighted_feature",b"weighted_feature"]) -> None: ... + +class Feature(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + name = ... # type: typing___Text + deprecated = ... # type: builtin___bool + type = ... # type: FeatureType + domain = ... # type: typing___Text + in_environment = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] + not_in_environment = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] + lifecycle_stage = ... # type: LifecycleStage + + @property + def presence(self) -> FeaturePresence: ... + + @property + def group_presence(self) -> FeaturePresenceWithinGroup: ... + + @property + def shape(self) -> FixedShape: ... + + @property + def value_count(self) -> ValueCount: ... + + @property + def int_domain(self) -> IntDomain: ... + + @property + def float_domain(self) -> FloatDomain: ... + + @property + def string_domain(self) -> StringDomain: ... + + @property + def bool_domain(self) -> BoolDomain: ... + + @property + def struct_domain(self) -> StructDomain: ... + + @property + def natural_language_domain(self) -> NaturalLanguageDomain: ... + + @property + def image_domain(self) -> ImageDomain: ... + + @property + def mid_domain(self) -> MIDDomain: ... + + @property + def url_domain(self) -> URLDomain: ... + + @property + def time_domain(self) -> TimeDomain: ... + + @property + def time_of_day_domain(self) -> TimeOfDayDomain: ... + + @property + def distribution_constraints(self) -> DistributionConstraints: ... + + @property + def annotation(self) -> Annotation: ... + + @property + def skew_comparator(self) -> FeatureComparator: ... + + @property + def drift_comparator(self) -> FeatureComparator: ... + + def __init__(self, + *, + name : typing___Optional[typing___Text] = None, + deprecated : typing___Optional[builtin___bool] = None, + presence : typing___Optional[FeaturePresence] = None, + group_presence : typing___Optional[FeaturePresenceWithinGroup] = None, + shape : typing___Optional[FixedShape] = None, + value_count : typing___Optional[ValueCount] = None, + type : typing___Optional[FeatureType] = None, + domain : typing___Optional[typing___Text] = None, + int_domain : typing___Optional[IntDomain] = None, + float_domain : typing___Optional[FloatDomain] = None, + string_domain : typing___Optional[StringDomain] = None, + bool_domain : typing___Optional[BoolDomain] = None, + struct_domain : typing___Optional[StructDomain] = None, + natural_language_domain : typing___Optional[NaturalLanguageDomain] = None, + image_domain : typing___Optional[ImageDomain] = None, + mid_domain : typing___Optional[MIDDomain] = None, + url_domain : typing___Optional[URLDomain] = None, + time_domain : typing___Optional[TimeDomain] = None, + time_of_day_domain : typing___Optional[TimeOfDayDomain] = None, + distribution_constraints : typing___Optional[DistributionConstraints] = None, + annotation : typing___Optional[Annotation] = None, + skew_comparator : typing___Optional[FeatureComparator] = None, + drift_comparator : typing___Optional[FeatureComparator] = None, + in_environment : typing___Optional[typing___Iterable[typing___Text]] = None, + not_in_environment : typing___Optional[typing___Iterable[typing___Text]] = None, + lifecycle_stage : typing___Optional[LifecycleStage] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> Feature: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> Feature: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"annotation",b"annotation",u"bool_domain",b"bool_domain",u"deprecated",b"deprecated",u"distribution_constraints",b"distribution_constraints",u"domain",b"domain",u"domain_info",b"domain_info",u"drift_comparator",b"drift_comparator",u"float_domain",b"float_domain",u"group_presence",b"group_presence",u"image_domain",b"image_domain",u"int_domain",b"int_domain",u"lifecycle_stage",b"lifecycle_stage",u"mid_domain",b"mid_domain",u"name",b"name",u"natural_language_domain",b"natural_language_domain",u"presence",b"presence",u"presence_constraints",b"presence_constraints",u"shape",b"shape",u"shape_type",b"shape_type",u"skew_comparator",b"skew_comparator",u"string_domain",b"string_domain",u"struct_domain",b"struct_domain",u"time_domain",b"time_domain",u"time_of_day_domain",b"time_of_day_domain",u"type",b"type",u"url_domain",b"url_domain",u"value_count",b"value_count"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"annotation",b"annotation",u"bool_domain",b"bool_domain",u"deprecated",b"deprecated",u"distribution_constraints",b"distribution_constraints",u"domain",b"domain",u"domain_info",b"domain_info",u"drift_comparator",b"drift_comparator",u"float_domain",b"float_domain",u"group_presence",b"group_presence",u"image_domain",b"image_domain",u"in_environment",b"in_environment",u"int_domain",b"int_domain",u"lifecycle_stage",b"lifecycle_stage",u"mid_domain",b"mid_domain",u"name",b"name",u"natural_language_domain",b"natural_language_domain",u"not_in_environment",b"not_in_environment",u"presence",b"presence",u"presence_constraints",b"presence_constraints",u"shape",b"shape",u"shape_type",b"shape_type",u"skew_comparator",b"skew_comparator",u"string_domain",b"string_domain",u"struct_domain",b"struct_domain",u"time_domain",b"time_domain",u"time_of_day_domain",b"time_of_day_domain",u"type",b"type",u"url_domain",b"url_domain",u"value_count",b"value_count"]) -> None: ... + @typing___overload + def WhichOneof(self, oneof_group: typing_extensions___Literal[u"domain_info",b"domain_info"]) -> typing_extensions___Literal["domain","int_domain","float_domain","string_domain","bool_domain","struct_domain","natural_language_domain","image_domain","mid_domain","url_domain","time_domain","time_of_day_domain"]: ... + @typing___overload + def WhichOneof(self, oneof_group: typing_extensions___Literal[u"presence_constraints",b"presence_constraints"]) -> typing_extensions___Literal["presence","group_presence"]: ... + @typing___overload + def WhichOneof(self, oneof_group: typing_extensions___Literal[u"shape_type",b"shape_type"]) -> typing_extensions___Literal["shape","value_count"]: ... + +class Annotation(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + tag = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] + comment = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] + + @property + def extra_metadata(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[google___protobuf___any_pb2___Any]: ... + + def __init__(self, + *, + tag : typing___Optional[typing___Iterable[typing___Text]] = None, + comment : typing___Optional[typing___Iterable[typing___Text]] = None, + extra_metadata : typing___Optional[typing___Iterable[google___protobuf___any_pb2___Any]] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> Annotation: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> Annotation: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def ClearField(self, field_name: typing_extensions___Literal[u"comment",b"comment",u"extra_metadata",b"extra_metadata",u"tag",b"tag"]) -> None: ... + +class NumericValueComparator(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + min_fraction_threshold = ... # type: builtin___float + max_fraction_threshold = ... # type: builtin___float + + def __init__(self, + *, + min_fraction_threshold : typing___Optional[builtin___float] = None, + max_fraction_threshold : typing___Optional[builtin___float] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> NumericValueComparator: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> NumericValueComparator: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"max_fraction_threshold",b"max_fraction_threshold",u"min_fraction_threshold",b"min_fraction_threshold"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"max_fraction_threshold",b"max_fraction_threshold",u"min_fraction_threshold",b"min_fraction_threshold"]) -> None: ... + +class DatasetConstraints(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + min_examples_count = ... # type: builtin___int + + @property + def num_examples_drift_comparator(self) -> NumericValueComparator: ... + + @property + def num_examples_version_comparator(self) -> NumericValueComparator: ... + + def __init__(self, + *, + num_examples_drift_comparator : typing___Optional[NumericValueComparator] = None, + num_examples_version_comparator : typing___Optional[NumericValueComparator] = None, + min_examples_count : typing___Optional[builtin___int] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> DatasetConstraints: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> DatasetConstraints: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"min_examples_count",b"min_examples_count",u"num_examples_drift_comparator",b"num_examples_drift_comparator",u"num_examples_version_comparator",b"num_examples_version_comparator"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"min_examples_count",b"min_examples_count",u"num_examples_drift_comparator",b"num_examples_drift_comparator",u"num_examples_version_comparator",b"num_examples_version_comparator"]) -> None: ... + +class FixedShape(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + class Dim(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + size = ... # type: builtin___int + name = ... # type: typing___Text + + def __init__(self, + *, + size : typing___Optional[builtin___int] = None, + name : typing___Optional[typing___Text] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> FixedShape.Dim: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> FixedShape.Dim: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"name",b"name",u"size",b"size"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"name",b"name",u"size",b"size"]) -> None: ... + + + @property + def dim(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[FixedShape.Dim]: ... + + def __init__(self, + *, + dim : typing___Optional[typing___Iterable[FixedShape.Dim]] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> FixedShape: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> FixedShape: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def ClearField(self, field_name: typing_extensions___Literal[u"dim",b"dim"]) -> None: ... + +class ValueCount(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + min = ... # type: builtin___int + max = ... # type: builtin___int + + def __init__(self, + *, + min : typing___Optional[builtin___int] = None, + max : typing___Optional[builtin___int] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> ValueCount: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> ValueCount: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"max",b"max",u"min",b"min"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"max",b"max",u"min",b"min"]) -> None: ... + +class WeightedFeature(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + name = ... # type: typing___Text + lifecycle_stage = ... # type: LifecycleStage + + @property + def feature(self) -> tensorflow_metadata___proto___v0___path_pb2___Path: ... + + @property + def weight_feature(self) -> tensorflow_metadata___proto___v0___path_pb2___Path: ... + + def __init__(self, + *, + name : typing___Optional[typing___Text] = None, + feature : typing___Optional[tensorflow_metadata___proto___v0___path_pb2___Path] = None, + weight_feature : typing___Optional[tensorflow_metadata___proto___v0___path_pb2___Path] = None, + lifecycle_stage : typing___Optional[LifecycleStage] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> WeightedFeature: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> WeightedFeature: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"feature",b"feature",u"lifecycle_stage",b"lifecycle_stage",u"name",b"name",u"weight_feature",b"weight_feature"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"feature",b"feature",u"lifecycle_stage",b"lifecycle_stage",u"name",b"name",u"weight_feature",b"weight_feature"]) -> None: ... + +class SparseFeature(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + class IndexFeature(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + name = ... # type: typing___Text + + def __init__(self, + *, + name : typing___Optional[typing___Text] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> SparseFeature.IndexFeature: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> SparseFeature.IndexFeature: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"name",b"name"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"name",b"name"]) -> None: ... + + class ValueFeature(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + name = ... # type: typing___Text + + def __init__(self, + *, + name : typing___Optional[typing___Text] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> SparseFeature.ValueFeature: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> SparseFeature.ValueFeature: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"name",b"name"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"name",b"name"]) -> None: ... + + name = ... # type: typing___Text + deprecated = ... # type: builtin___bool + lifecycle_stage = ... # type: LifecycleStage + is_sorted = ... # type: builtin___bool + type = ... # type: FeatureType + + @property + def presence(self) -> FeaturePresence: ... + + @property + def dense_shape(self) -> FixedShape: ... + + @property + def index_feature(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[SparseFeature.IndexFeature]: ... + + @property + def value_feature(self) -> SparseFeature.ValueFeature: ... + + def __init__(self, + *, + name : typing___Optional[typing___Text] = None, + deprecated : typing___Optional[builtin___bool] = None, + lifecycle_stage : typing___Optional[LifecycleStage] = None, + presence : typing___Optional[FeaturePresence] = None, + dense_shape : typing___Optional[FixedShape] = None, + index_feature : typing___Optional[typing___Iterable[SparseFeature.IndexFeature]] = None, + is_sorted : typing___Optional[builtin___bool] = None, + value_feature : typing___Optional[SparseFeature.ValueFeature] = None, + type : typing___Optional[FeatureType] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> SparseFeature: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> SparseFeature: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"dense_shape",b"dense_shape",u"deprecated",b"deprecated",u"is_sorted",b"is_sorted",u"lifecycle_stage",b"lifecycle_stage",u"name",b"name",u"presence",b"presence",u"type",b"type",u"value_feature",b"value_feature"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"dense_shape",b"dense_shape",u"deprecated",b"deprecated",u"index_feature",b"index_feature",u"is_sorted",b"is_sorted",u"lifecycle_stage",b"lifecycle_stage",u"name",b"name",u"presence",b"presence",u"type",b"type",u"value_feature",b"value_feature"]) -> None: ... + +class DistributionConstraints(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + min_domain_mass = ... # type: builtin___float + + def __init__(self, + *, + min_domain_mass : typing___Optional[builtin___float] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> DistributionConstraints: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> DistributionConstraints: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"min_domain_mass",b"min_domain_mass"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"min_domain_mass",b"min_domain_mass"]) -> None: ... + +class IntDomain(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + name = ... # type: typing___Text + min = ... # type: builtin___int + max = ... # type: builtin___int + is_categorical = ... # type: builtin___bool + + def __init__(self, + *, + name : typing___Optional[typing___Text] = None, + min : typing___Optional[builtin___int] = None, + max : typing___Optional[builtin___int] = None, + is_categorical : typing___Optional[builtin___bool] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> IntDomain: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> IntDomain: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"is_categorical",b"is_categorical",u"max",b"max",u"min",b"min",u"name",b"name"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"is_categorical",b"is_categorical",u"max",b"max",u"min",b"min",u"name",b"name"]) -> None: ... + +class FloatDomain(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + name = ... # type: typing___Text + min = ... # type: builtin___float + max = ... # type: builtin___float + + def __init__(self, + *, + name : typing___Optional[typing___Text] = None, + min : typing___Optional[builtin___float] = None, + max : typing___Optional[builtin___float] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> FloatDomain: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> FloatDomain: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"max",b"max",u"min",b"min",u"name",b"name"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"max",b"max",u"min",b"min",u"name",b"name"]) -> None: ... + +class StructDomain(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + + @property + def feature(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[Feature]: ... + + @property + def sparse_feature(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[SparseFeature]: ... + + def __init__(self, + *, + feature : typing___Optional[typing___Iterable[Feature]] = None, + sparse_feature : typing___Optional[typing___Iterable[SparseFeature]] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> StructDomain: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> StructDomain: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def ClearField(self, field_name: typing_extensions___Literal[u"feature",b"feature",u"sparse_feature",b"sparse_feature"]) -> None: ... + +class StringDomain(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + name = ... # type: typing___Text + value = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] + + def __init__(self, + *, + name : typing___Optional[typing___Text] = None, + value : typing___Optional[typing___Iterable[typing___Text]] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> StringDomain: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> StringDomain: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"name",b"name"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"name",b"name",u"value",b"value"]) -> None: ... + +class BoolDomain(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + name = ... # type: typing___Text + true_value = ... # type: typing___Text + false_value = ... # type: typing___Text + + def __init__(self, + *, + name : typing___Optional[typing___Text] = None, + true_value : typing___Optional[typing___Text] = None, + false_value : typing___Optional[typing___Text] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> BoolDomain: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> BoolDomain: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"false_value",b"false_value",u"name",b"name",u"true_value",b"true_value"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"false_value",b"false_value",u"name",b"name",u"true_value",b"true_value"]) -> None: ... + +class NaturalLanguageDomain(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + + def __init__(self, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> NaturalLanguageDomain: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> NaturalLanguageDomain: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + +class ImageDomain(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + + def __init__(self, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> ImageDomain: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> ImageDomain: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + +class MIDDomain(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + + def __init__(self, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> MIDDomain: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> MIDDomain: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + +class URLDomain(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + + def __init__(self, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> URLDomain: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> URLDomain: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + +class TimeDomain(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + class IntegerTimeFormat(builtin___int): + DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... + @classmethod + def Name(cls, number: builtin___int) -> builtin___str: ... + @classmethod + def Value(cls, name: builtin___str) -> 'TimeDomain.IntegerTimeFormat': ... + @classmethod + def keys(cls) -> typing___List[builtin___str]: ... + @classmethod + def values(cls) -> typing___List['TimeDomain.IntegerTimeFormat']: ... + @classmethod + def items(cls) -> typing___List[typing___Tuple[builtin___str, 'TimeDomain.IntegerTimeFormat']]: ... + FORMAT_UNKNOWN = typing___cast('TimeDomain.IntegerTimeFormat', 0) + UNIX_DAYS = typing___cast('TimeDomain.IntegerTimeFormat', 5) + UNIX_SECONDS = typing___cast('TimeDomain.IntegerTimeFormat', 1) + UNIX_MILLISECONDS = typing___cast('TimeDomain.IntegerTimeFormat', 2) + UNIX_MICROSECONDS = typing___cast('TimeDomain.IntegerTimeFormat', 3) + UNIX_NANOSECONDS = typing___cast('TimeDomain.IntegerTimeFormat', 4) + FORMAT_UNKNOWN = typing___cast('TimeDomain.IntegerTimeFormat', 0) + UNIX_DAYS = typing___cast('TimeDomain.IntegerTimeFormat', 5) + UNIX_SECONDS = typing___cast('TimeDomain.IntegerTimeFormat', 1) + UNIX_MILLISECONDS = typing___cast('TimeDomain.IntegerTimeFormat', 2) + UNIX_MICROSECONDS = typing___cast('TimeDomain.IntegerTimeFormat', 3) + UNIX_NANOSECONDS = typing___cast('TimeDomain.IntegerTimeFormat', 4) + + string_format = ... # type: typing___Text + integer_format = ... # type: TimeDomain.IntegerTimeFormat + + def __init__(self, + *, + string_format : typing___Optional[typing___Text] = None, + integer_format : typing___Optional[TimeDomain.IntegerTimeFormat] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> TimeDomain: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TimeDomain: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"format",b"format",u"integer_format",b"integer_format",u"string_format",b"string_format"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"format",b"format",u"integer_format",b"integer_format",u"string_format",b"string_format"]) -> None: ... + def WhichOneof(self, oneof_group: typing_extensions___Literal[u"format",b"format"]) -> typing_extensions___Literal["string_format","integer_format"]: ... + +class TimeOfDayDomain(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + class IntegerTimeOfDayFormat(builtin___int): + DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... + @classmethod + def Name(cls, number: builtin___int) -> builtin___str: ... + @classmethod + def Value(cls, name: builtin___str) -> 'TimeOfDayDomain.IntegerTimeOfDayFormat': ... + @classmethod + def keys(cls) -> typing___List[builtin___str]: ... + @classmethod + def values(cls) -> typing___List['TimeOfDayDomain.IntegerTimeOfDayFormat']: ... + @classmethod + def items(cls) -> typing___List[typing___Tuple[builtin___str, 'TimeOfDayDomain.IntegerTimeOfDayFormat']]: ... + FORMAT_UNKNOWN = typing___cast('TimeOfDayDomain.IntegerTimeOfDayFormat', 0) + PACKED_64_NANOS = typing___cast('TimeOfDayDomain.IntegerTimeOfDayFormat', 1) + FORMAT_UNKNOWN = typing___cast('TimeOfDayDomain.IntegerTimeOfDayFormat', 0) + PACKED_64_NANOS = typing___cast('TimeOfDayDomain.IntegerTimeOfDayFormat', 1) + + string_format = ... # type: typing___Text + integer_format = ... # type: TimeOfDayDomain.IntegerTimeOfDayFormat + + def __init__(self, + *, + string_format : typing___Optional[typing___Text] = None, + integer_format : typing___Optional[TimeOfDayDomain.IntegerTimeOfDayFormat] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> TimeOfDayDomain: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TimeOfDayDomain: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"format",b"format",u"integer_format",b"integer_format",u"string_format",b"string_format"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"format",b"format",u"integer_format",b"integer_format",u"string_format",b"string_format"]) -> None: ... + def WhichOneof(self, oneof_group: typing_extensions___Literal[u"format",b"format"]) -> typing_extensions___Literal["string_format","integer_format"]: ... + +class FeaturePresence(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + min_fraction = ... # type: builtin___float + min_count = ... # type: builtin___int + + def __init__(self, + *, + min_fraction : typing___Optional[builtin___float] = None, + min_count : typing___Optional[builtin___int] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> FeaturePresence: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> FeaturePresence: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"min_count",b"min_count",u"min_fraction",b"min_fraction"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"min_count",b"min_count",u"min_fraction",b"min_fraction"]) -> None: ... + +class FeaturePresenceWithinGroup(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + required = ... # type: builtin___bool + + def __init__(self, + *, + required : typing___Optional[builtin___bool] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> FeaturePresenceWithinGroup: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> FeaturePresenceWithinGroup: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"required",b"required"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"required",b"required"]) -> None: ... + +class InfinityNorm(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + threshold = ... # type: builtin___float + + def __init__(self, + *, + threshold : typing___Optional[builtin___float] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> InfinityNorm: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> InfinityNorm: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"threshold",b"threshold"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"threshold",b"threshold"]) -> None: ... + +class FeatureComparator(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + + @property + def infinity_norm(self) -> InfinityNorm: ... + + def __init__(self, + *, + infinity_norm : typing___Optional[InfinityNorm] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> FeatureComparator: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> FeatureComparator: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"infinity_norm",b"infinity_norm"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"infinity_norm",b"infinity_norm"]) -> None: ... + +class TensorRepresentation(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + class DefaultValue(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + float_value = ... # type: builtin___float + int_value = ... # type: builtin___int + bytes_value = ... # type: builtin___bytes + uint_value = ... # type: builtin___int + + def __init__(self, + *, + float_value : typing___Optional[builtin___float] = None, + int_value : typing___Optional[builtin___int] = None, + bytes_value : typing___Optional[builtin___bytes] = None, + uint_value : typing___Optional[builtin___int] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> TensorRepresentation.DefaultValue: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TensorRepresentation.DefaultValue: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"bytes_value",b"bytes_value",u"float_value",b"float_value",u"int_value",b"int_value",u"kind",b"kind",u"uint_value",b"uint_value"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"bytes_value",b"bytes_value",u"float_value",b"float_value",u"int_value",b"int_value",u"kind",b"kind",u"uint_value",b"uint_value"]) -> None: ... + def WhichOneof(self, oneof_group: typing_extensions___Literal[u"kind",b"kind"]) -> typing_extensions___Literal["float_value","int_value","bytes_value","uint_value"]: ... + + class DenseTensor(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + column_name = ... # type: typing___Text + + @property + def shape(self) -> FixedShape: ... + + @property + def default_value(self) -> TensorRepresentation.DefaultValue: ... + + def __init__(self, + *, + column_name : typing___Optional[typing___Text] = None, + shape : typing___Optional[FixedShape] = None, + default_value : typing___Optional[TensorRepresentation.DefaultValue] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> TensorRepresentation.DenseTensor: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TensorRepresentation.DenseTensor: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"column_name",b"column_name",u"default_value",b"default_value",u"shape",b"shape"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"column_name",b"column_name",u"default_value",b"default_value",u"shape",b"shape"]) -> None: ... + + class VarLenSparseTensor(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + column_name = ... # type: typing___Text + + def __init__(self, + *, + column_name : typing___Optional[typing___Text] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> TensorRepresentation.VarLenSparseTensor: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TensorRepresentation.VarLenSparseTensor: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"column_name",b"column_name"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"column_name",b"column_name"]) -> None: ... + + class SparseTensor(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + index_column_names = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] + value_column_name = ... # type: typing___Text + + @property + def dense_shape(self) -> FixedShape: ... + + def __init__(self, + *, + dense_shape : typing___Optional[FixedShape] = None, + index_column_names : typing___Optional[typing___Iterable[typing___Text]] = None, + value_column_name : typing___Optional[typing___Text] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> TensorRepresentation.SparseTensor: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TensorRepresentation.SparseTensor: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"dense_shape",b"dense_shape",u"value_column_name",b"value_column_name"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"dense_shape",b"dense_shape",u"index_column_names",b"index_column_names",u"value_column_name",b"value_column_name"]) -> None: ... + + + @property + def dense_tensor(self) -> TensorRepresentation.DenseTensor: ... + + @property + def varlen_sparse_tensor(self) -> TensorRepresentation.VarLenSparseTensor: ... + + @property + def sparse_tensor(self) -> TensorRepresentation.SparseTensor: ... + + def __init__(self, + *, + dense_tensor : typing___Optional[TensorRepresentation.DenseTensor] = None, + varlen_sparse_tensor : typing___Optional[TensorRepresentation.VarLenSparseTensor] = None, + sparse_tensor : typing___Optional[TensorRepresentation.SparseTensor] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> TensorRepresentation: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TensorRepresentation: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"dense_tensor",b"dense_tensor",u"kind",b"kind",u"sparse_tensor",b"sparse_tensor",u"varlen_sparse_tensor",b"varlen_sparse_tensor"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"dense_tensor",b"dense_tensor",u"kind",b"kind",u"sparse_tensor",b"sparse_tensor",u"varlen_sparse_tensor",b"varlen_sparse_tensor"]) -> None: ... + def WhichOneof(self, oneof_group: typing_extensions___Literal[u"kind",b"kind"]) -> typing_extensions___Literal["dense_tensor","varlen_sparse_tensor","sparse_tensor"]: ... + +class TensorRepresentationGroup(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + class TensorRepresentationEntry(google___protobuf___message___Message): + DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... + key = ... # type: typing___Text + + @property + def value(self) -> TensorRepresentation: ... + + def __init__(self, + *, + key : typing___Optional[typing___Text] = None, + value : typing___Optional[TensorRepresentation] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> TensorRepresentationGroup.TensorRepresentationEntry: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TensorRepresentationGroup.TensorRepresentationEntry: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def HasField(self, field_name: typing_extensions___Literal[u"key",b"key",u"value",b"value"]) -> builtin___bool: ... + def ClearField(self, field_name: typing_extensions___Literal[u"key",b"key",u"value",b"value"]) -> None: ... + + + @property + def tensor_representation(self) -> typing___MutableMapping[typing___Text, TensorRepresentation]: ... + + def __init__(self, + *, + tensor_representation : typing___Optional[typing___Mapping[typing___Text, TensorRepresentation]] = None, + ) -> None: ... + if sys.version_info >= (3,): + @classmethod + def FromString(cls, s: builtin___bytes) -> TensorRepresentationGroup: ... + else: + @classmethod + def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TensorRepresentationGroup: ... + def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... + def ClearField(self, field_name: typing_extensions___Literal[u"tensor_representation",b"tensor_representation"]) -> None: ... diff --git a/sdk/python/tests/conftest.py b/sdk/python/tests/conftest.py index b564eeaa5b1..5803f0b0c07 100644 --- a/sdk/python/tests/conftest.py +++ b/sdk/python/tests/conftest.py @@ -11,8 +11,8 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -from sys import platform import multiprocessing +from sys import platform def pytest_configure(config): diff --git a/sdk/python/tests/dataframes.py b/sdk/python/tests/dataframes.py index 28048322062..a50aa89810c 100644 --- a/sdk/python/tests/dataframes.py +++ b/sdk/python/tests/dataframes.py @@ -1,8 +1,8 @@ -import pandas as pd -import pytz -import numpy as np from datetime import datetime +import numpy as np +import pandas as pd +import pytz GOOD = pd.DataFrame( { diff --git a/sdk/python/tests/feast_core_server.py b/sdk/python/tests/feast_core_server.py index 7cc837a4f2d..b6efe2cb6d1 100644 --- a/sdk/python/tests/feast_core_server.py +++ b/sdk/python/tests/feast_core_server.py @@ -1,27 +1,22 @@ -from concurrent import futures -import time import logging +import time +from concurrent import futures + import grpc +from google.protobuf.timestamp_pb2 import Timestamp + import feast.core.CoreService_pb2_grpc as Core from feast.core.CoreService_pb2 import ( - GetFeastCoreVersionResponse, - ApplyFeatureSetResponse, ApplyFeatureSetRequest, - ListFeatureSetsResponse, + ApplyFeatureSetResponse, + GetFeastCoreVersionResponse, ListFeatureSetsRequest, -) -from google.protobuf.timestamp_pb2 import Timestamp -from feast.core.FeatureSet_pb2 import ( - FeatureSetSpec as FeatureSetSpec, - FeatureSetMeta, - FeatureSetStatus, -) -from feast.core.Source_pb2 import ( - SourceType as SourceTypeProto, - KafkaSourceConfig as KafkaSourceConfigProto, + ListFeatureSetsResponse, ) from feast.core.FeatureSet_pb2 import FeatureSet as FeatureSetProto -from typing import List +from feast.core.FeatureSet_pb2 import FeatureSetMeta, FeatureSetStatus +from feast.core.Source_pb2 import KafkaSourceConfig as KafkaSourceConfigProto +from feast.core.Source_pb2 import SourceType as SourceTypeProto _logger = logging.getLogger(__name__) diff --git a/sdk/python/tests/feast_serving_server.py b/sdk/python/tests/feast_serving_server.py index eb46bde1215..364c1907141 100644 --- a/sdk/python/tests/feast_serving_server.py +++ b/sdk/python/tests/feast_serving_server.py @@ -1,24 +1,23 @@ -from concurrent import futures -import time import logging +import time +from concurrent import futures +from typing import Dict import grpc +from google.protobuf.timestamp_pb2 import Timestamp + import feast.serving.ServingService_pb2_grpc as Serving +from feast.core import FeatureSet_pb2 as FeatureSetProto +from feast.core.CoreService_pb2 import ListFeatureSetsResponse +from feast.core.CoreService_pb2_grpc import CoreServiceStub from feast.serving.ServingService_pb2 import ( + GetFeastServingInfoResponse, GetOnlineFeaturesRequest, GetOnlineFeaturesResponse, - GetFeastServingInfoResponse, ) -from typing import Dict -from feast.core.CoreService_pb2_grpc import CoreServiceStub -from feast.core.CoreService_pb2 import ListFeatureSetsResponse -from feast.core import FeatureSet_pb2 as FeatureSetProto -from feast.types import ( - FeatureRow_pb2 as FeatureRowProto, - Field_pb2 as FieldProto, - Value_pb2 as ValueProto, -) -from google.protobuf.timestamp_pb2 import Timestamp +from feast.types import FeatureRow_pb2 as FeatureRowProto +from feast.types import Field_pb2 as FieldProto +from feast.types import Value_pb2 as ValueProto _ONE_DAY_IN_SECONDS = 60 * 60 * 24 diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index 2724fff52e3..1478256c066 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -12,70 +12,70 @@ # See the License for the specific language governing permissions and # limitations under the License. import pkgutil -from datetime import datetime - import tempfile +from concurrent import futures +from datetime import datetime from unittest import mock import grpc import pandas as pd +import pytest from google.protobuf.duration_pb2 import Duration from mock import MagicMock, patch -from pytz import timezone from pandavro import to_avro +from pytz import timezone + +import dataframes import feast.core.CoreService_pb2_grpc as Core import feast.serving.ServingService_pb2_grpc as Serving -from feast.feature_set import FeatureSet -from feast.entity import Entity -from feast.feature_set import Feature -from feast.source import KafkaSource -from feast.core.FeatureSet_pb2 import ( - FeatureSetSpec as FeatureSetSpecProto, - FeatureSpec as FeatureSpecProto, - EntitySpec as EntitySpecProto, - FeatureSetMeta as FeatureSetMetaProto, - FeatureSetStatus as FeatureSetStatusProto, - FeatureSet as FeatureSetProto, -) -from feast.core.Source_pb2 import SourceType, KafkaSourceConfig, Source +from feast.client import Client from feast.core.CoreService_pb2 import ( GetFeastCoreVersionResponse, GetFeatureSetResponse, ) +from feast.core.FeatureSet_pb2 import EntitySpec as EntitySpecProto +from feast.core.FeatureSet_pb2 import FeatureSet as FeatureSetProto +from feast.core.FeatureSet_pb2 import FeatureSetMeta as FeatureSetMetaProto +from feast.core.FeatureSet_pb2 import FeatureSetSpec as FeatureSetSpecProto +from feast.core.FeatureSet_pb2 import FeatureSetStatus as FeatureSetStatusProto +from feast.core.FeatureSet_pb2 import FeatureSpec as FeatureSpecProto +from feast.core.Source_pb2 import KafkaSourceConfig, Source, SourceType +from feast.entity import Entity +from feast.feature_set import Feature, FeatureSet +from feast.job import Job from feast.serving.ServingService_pb2 import ( - GetFeastServingInfoResponse, - GetOnlineFeaturesResponse, - GetOnlineFeaturesRequest, - GetBatchFeaturesResponse, - Job as BatchFeaturesJob, - JobType, - JobStatus, DataFormat, - GetJobResponse, FeastServingType, + GetBatchFeaturesResponse, + GetFeastServingInfoResponse, + GetJobResponse, + GetOnlineFeaturesRequest, + GetOnlineFeaturesResponse, ) -import pytest -from feast.client import Client -from concurrent import futures -from feast_core_server import CoreServicer -from feast_serving_server import ServingServicer +from feast.serving.ServingService_pb2 import Job as BatchFeaturesJob +from feast.serving.ServingService_pb2 import JobStatus, JobType +from feast.source import KafkaSource from feast.types import Value_pb2 as ValueProto from feast.value_type import ValueType -from feast.job import Job -import dataframes +from feast_core_server import CoreServicer +from feast_serving_server import ServingServicer CORE_URL = "core.feast.example.com" SERVING_URL = "serving.example.com" -_PRIVATE_KEY_RESOURCE_PATH = 'data/localhost.key' -_CERTIFICATE_CHAIN_RESOURCE_PATH = 'data/localhost.pem' -_ROOT_CERTIFICATE_RESOURCE_PATH = 'data/localhost.crt' +_PRIVATE_KEY_RESOURCE_PATH = "data/localhost.key" +_CERTIFICATE_CHAIN_RESOURCE_PATH = "data/localhost.pem" +_ROOT_CERTIFICATE_RESOURCE_PATH = "data/localhost.crt" class TestClient: - @pytest.fixture def secure_mock_client(self, mocker): - client = Client(core_url=CORE_URL, serving_url=SERVING_URL, core_secure=True, serving_secure=True) + client = Client( + core_url=CORE_URL, + serving_url=SERVING_URL, + core_secure=True, + serving_secure=True, + ) mocker.patch.object(client, "_connect_core") mocker.patch.object(client, "_connect_serving") client._core_url = CORE_URL @@ -136,22 +136,36 @@ def secure_serving_server(self, server_credentials): @pytest.fixture def secure_client(self, secure_core_server, secure_serving_server): - root_certificate_credentials = pkgutil.get_data(__name__, _ROOT_CERTIFICATE_RESOURCE_PATH) + root_certificate_credentials = pkgutil.get_data( + __name__, _ROOT_CERTIFICATE_RESOURCE_PATH + ) # this is needed to establish a secure connection using self-signed certificates, for the purpose of the test - ssl_channel_credentials = grpc.ssl_channel_credentials(root_certificates=root_certificate_credentials) - with mock.patch("grpc.ssl_channel_credentials", MagicMock(return_value=ssl_channel_credentials)): - yield Client(core_url="localhost:50053", serving_url="localhost:50054", core_secure=True, - serving_secure=True) + ssl_channel_credentials = grpc.ssl_channel_credentials( + root_certificates=root_certificate_credentials + ) + with mock.patch( + "grpc.ssl_channel_credentials", + MagicMock(return_value=ssl_channel_credentials), + ): + yield Client( + core_url="localhost:50053", + serving_url="localhost:50054", + core_secure=True, + serving_secure=True, + ) @pytest.fixture def client(self, core_server, serving_server): return Client(core_url="localhost:50051", serving_url="localhost:50052") - @pytest.mark.parametrize("mocked_client", [pytest.lazy_fixture("mock_client"), - pytest.lazy_fixture("secure_mock_client") - ]) + @pytest.mark.parametrize( + "mocked_client", + [pytest.lazy_fixture("mock_client"), pytest.lazy_fixture("secure_mock_client")], + ) def test_version(self, mocked_client, mocker): - mocked_client._core_service_stub = Core.CoreServiceStub(grpc.insecure_channel("")) + mocked_client._core_service_stub = Core.CoreServiceStub( + grpc.insecure_channel("") + ) mocked_client._serving_service_stub = Serving.ServingServiceStub( grpc.insecure_channel("") ) @@ -170,15 +184,16 @@ def test_version(self, mocked_client, mocker): status = mocked_client.version() assert ( - status["core"]["url"] == CORE_URL - and status["core"]["version"] == "0.3.2" - and status["serving"]["url"] == SERVING_URL - and status["serving"]["version"] == "0.3.2" + status["core"]["url"] == CORE_URL + and status["core"]["version"] == "0.3.2" + and status["serving"]["url"] == SERVING_URL + and status["serving"]["version"] == "0.3.2" ) - @pytest.mark.parametrize("mocked_client", [pytest.lazy_fixture("mock_client"), - pytest.lazy_fixture("secure_mock_client") - ]) + @pytest.mark.parametrize( + "mocked_client", + [pytest.lazy_fixture("mock_client"), pytest.lazy_fixture("secure_mock_client")], + ) def test_get_online_features(self, mocked_client, mocker): ROW_COUNT = 300 @@ -225,15 +240,18 @@ def test_get_online_features(self, mocked_client, mocker): ) # type: GetOnlineFeaturesResponse assert ( - response.field_values[0].fields["my_project/feature_1:1"].int64_val == 1 - and response.field_values[0].fields["my_project/feature_9:1"].int64_val == 9 + response.field_values[0].fields["my_project/feature_1:1"].int64_val == 1 + and response.field_values[0].fields["my_project/feature_9:1"].int64_val == 9 ) - @pytest.mark.parametrize("mocked_client", [pytest.lazy_fixture("mock_client"), - pytest.lazy_fixture("secure_mock_client") - ]) + @pytest.mark.parametrize( + "mocked_client", + [pytest.lazy_fixture("mock_client"), pytest.lazy_fixture("secure_mock_client")], + ) def test_get_feature_set(self, mocked_client, mocker): - mocked_client._core_service_stub = Core.CoreServiceStub(grpc.insecure_channel("")) + mocked_client._core_service_stub = Core.CoreServiceStub( + grpc.insecure_channel("") + ) from google.protobuf.duration_pb2 import Duration @@ -277,25 +295,28 @@ def test_get_feature_set(self, mocked_client, mocker): feature_set = mocked_client.get_feature_set("my_feature_set", version=2) assert ( - feature_set.name == "my_feature_set" - and feature_set.version == 2 - and feature_set.fields["my_feature_1"].name == "my_feature_1" - and feature_set.fields["my_feature_1"].dtype == ValueType.FLOAT - and feature_set.fields["my_entity_1"].name == "my_entity_1" - and feature_set.fields["my_entity_1"].dtype == ValueType.INT64 - and len(feature_set.features) == 2 - and len(feature_set.entities) == 1 + feature_set.name == "my_feature_set" + and feature_set.version == 2 + and feature_set.fields["my_feature_1"].name == "my_feature_1" + and feature_set.fields["my_feature_1"].dtype == ValueType.FLOAT + and feature_set.fields["my_entity_1"].name == "my_entity_1" + and feature_set.fields["my_entity_1"].dtype == ValueType.INT64 + and len(feature_set.features) == 2 + and len(feature_set.entities) == 1 ) - @pytest.mark.parametrize("mocked_client", [pytest.lazy_fixture("mock_client"), - pytest.lazy_fixture("secure_mock_client") - ]) + @pytest.mark.parametrize( + "mocked_client", + [pytest.lazy_fixture("mock_client"), pytest.lazy_fixture("secure_mock_client")], + ) def test_get_batch_features(self, mocked_client, mocker): mocked_client._serving_service_stub = Serving.ServingServiceStub( grpc.insecure_channel("") ) - mocked_client._core_service_stub = Core.CoreServiceStub(grpc.insecure_channel("")) + mocked_client._core_service_stub = Core.CoreServiceStub( + grpc.insecure_channel("") + ) mocker.patch.object( mocked_client._core_service_stub, @@ -410,9 +431,10 @@ def test_get_batch_features(self, mocked_client, mocker): ] ) - @pytest.mark.parametrize("test_client", [pytest.lazy_fixture("client"), - pytest.lazy_fixture("secure_client") - ]) + @pytest.mark.parametrize( + "test_client", + [pytest.lazy_fixture("client"), pytest.lazy_fixture("secure_client")], + ) def test_apply_feature_set_success(self, test_client): test_client.set_project("project1") @@ -436,15 +458,20 @@ def test_apply_feature_set_success(self, test_client): # List Feature Sets assert ( - len(feature_sets) == 2 - and feature_sets[0].name == "my-feature-set-1" - and feature_sets[0].features[0].name == "fs1-my-feature-1" - and feature_sets[0].features[0].dtype == ValueType.INT64 - and feature_sets[1].features[1].dtype == ValueType.BYTES_LIST + len(feature_sets) == 2 + and feature_sets[0].name == "my-feature-set-1" + and feature_sets[0].features[0].name == "fs1-my-feature-1" + and feature_sets[0].features[0].dtype == ValueType.INT64 + and feature_sets[1].features[1].dtype == ValueType.BYTES_LIST ) - @pytest.mark.parametrize("dataframe,test_client", [(dataframes.GOOD, pytest.lazy_fixture("client")), - (dataframes.GOOD, pytest.lazy_fixture("secure_client"))]) + @pytest.mark.parametrize( + "dataframe,test_client", + [ + (dataframes.GOOD, pytest.lazy_fixture("client")), + (dataframes.GOOD, pytest.lazy_fixture("secure_client")), + ], + ) def test_feature_set_ingest_success(self, dataframe, test_client, mocker): test_client.set_project("project1") driver_fs = FeatureSet( @@ -467,15 +494,19 @@ def test_feature_set_ingest_success(self, dataframe, test_client, mocker): ) # Need to create a mock producer - with patch("feast.client.get_producer") as mocked_queue: + with patch("feast.client.get_producer"): # Ingest data into Feast test_client.ingest("driver-feature-set", dataframe) - @pytest.mark.parametrize("dataframe,exception,test_client", - [(dataframes.GOOD, TimeoutError, pytest.lazy_fixture("client")), - (dataframes.GOOD, TimeoutError, pytest.lazy_fixture("secure_client"))]) + @pytest.mark.parametrize( + "dataframe,exception,test_client", + [ + (dataframes.GOOD, TimeoutError, pytest.lazy_fixture("client")), + (dataframes.GOOD, TimeoutError, pytest.lazy_fixture("secure_client")), + ], + ) def test_feature_set_ingest_fail_if_pending( - self, dataframe, exception, test_client, mocker + self, dataframe, exception, test_client, mocker ): with pytest.raises(exception): test_client.set_project("project1") @@ -500,7 +531,7 @@ def test_feature_set_ingest_fail_if_pending( ) # Need to create a mock producer - with patch("feast.client.get_producer") as mocked_queue: + with patch("feast.client.get_producer"): # Ingest data into Feast test_client.ingest("driver-feature-set", dataframe, timeout=1) @@ -508,11 +539,23 @@ def test_feature_set_ingest_fail_if_pending( "dataframe,exception,test_client", [ (dataframes.BAD_NO_DATETIME, Exception, pytest.lazy_fixture("client")), - (dataframes.BAD_INCORRECT_DATETIME_TYPE, Exception, pytest.lazy_fixture("client")), + ( + dataframes.BAD_INCORRECT_DATETIME_TYPE, + Exception, + pytest.lazy_fixture("client"), + ), (dataframes.BAD_NO_ENTITY, Exception, pytest.lazy_fixture("client")), (dataframes.NO_FEATURES, Exception, pytest.lazy_fixture("client")), - (dataframes.BAD_NO_DATETIME, Exception, pytest.lazy_fixture("secure_client")), - (dataframes.BAD_INCORRECT_DATETIME_TYPE, Exception, pytest.lazy_fixture("secure_client")), + ( + dataframes.BAD_NO_DATETIME, + Exception, + pytest.lazy_fixture("secure_client"), + ), + ( + dataframes.BAD_INCORRECT_DATETIME_TYPE, + Exception, + pytest.lazy_fixture("secure_client"), + ), (dataframes.BAD_NO_ENTITY, Exception, pytest.lazy_fixture("secure_client")), (dataframes.NO_FEATURES, Exception, pytest.lazy_fixture("secure_client")), ], @@ -531,8 +574,13 @@ def test_feature_set_ingest_failure(self, test_client, dataframe, exception): # Ingest data into Feast test_client.ingest(driver_fs, dataframe=dataframe) - @pytest.mark.parametrize("dataframe,test_client", [(dataframes.ALL_TYPES, pytest.lazy_fixture("client")), - (dataframes.ALL_TYPES, pytest.lazy_fixture("secure_client"))]) + @pytest.mark.parametrize( + "dataframe,test_client", + [ + (dataframes.ALL_TYPES, pytest.lazy_fixture("client")), + (dataframes.ALL_TYPES, pytest.lazy_fixture("secure_client")), + ], + ) def test_feature_set_types_success(self, test_client, dataframe, mocker): test_client.set_project("project1") @@ -571,40 +619,46 @@ def test_feature_set_types_success(self, test_client, dataframe, mocker): ) # Need to create a mock producer - with patch("feast.client.get_producer") as mocked_queue: + with patch("feast.client.get_producer"): # Ingest data into Feast test_client.ingest(all_types_fs, dataframe) @patch("grpc.channel_ready_future") def test_secure_channel_creation_with_secure_client(self, _mocked_obj): - client = Client(core_url="localhost:50051", serving_url="localhost:50052", serving_secure=True, - core_secure=True) - with mock.patch("grpc.secure_channel") as _grpc_mock, \ - mock.patch("grpc.ssl_channel_credentials", MagicMock(return_value="test")) as _mocked_credentials: + client = Client( + core_url="localhost:50051", + serving_url="localhost:50052", + serving_secure=True, + core_secure=True, + ) + with mock.patch("grpc.secure_channel") as _grpc_mock, mock.patch( + "grpc.ssl_channel_credentials", MagicMock(return_value="test") + ) as _mocked_credentials: client._connect_serving() - _grpc_mock.assert_called_with(client.serving_url, _mocked_credentials.return_value) + _grpc_mock.assert_called_with( + client.serving_url, _mocked_credentials.return_value + ) @mock.patch("grpc.channel_ready_future") - def test_secure_channel_creation_with_secure_serving_url(self, _mocked_obj, ): + def test_secure_channel_creation_with_secure_serving_url( + self, _mocked_obj, + ): client = Client(core_url="localhost:50051", serving_url="localhost:443") - with mock.patch("grpc.secure_channel") as _grpc_mock, \ - mock.patch("grpc.ssl_channel_credentials", MagicMock(return_value="test")) as _mocked_credentials: + with mock.patch("grpc.secure_channel") as _grpc_mock, mock.patch( + "grpc.ssl_channel_credentials", MagicMock(return_value="test") + ) as _mocked_credentials: client._connect_serving() - _grpc_mock.assert_called_with(client.serving_url, _mocked_credentials.return_value) - - @patch("grpc.channel_ready_future") - def test_secure_channel_creation_with_secure_client(self, _mocked_obj): - client = Client(core_url="localhost:50053", serving_url="localhost:50054", serving_secure=True, - core_secure=True) - with mock.patch("grpc.secure_channel") as _grpc_mock, \ - mock.patch("grpc.ssl_channel_credentials", MagicMock(return_value="test")) as _mocked_credentials: - client._connect_core() - _grpc_mock.assert_called_with(client.core_url, _mocked_credentials.return_value) + _grpc_mock.assert_called_with( + client.serving_url, _mocked_credentials.return_value + ) @patch("grpc.channel_ready_future") def test_secure_channel_creation_with_secure_core_url(self, _mocked_obj): client = Client(core_url="localhost:443", serving_url="localhost:50054") - with mock.patch("grpc.secure_channel") as _grpc_mock, \ - mock.patch("grpc.ssl_channel_credentials", MagicMock(return_value="test")) as _mocked_credentials: + with mock.patch("grpc.secure_channel") as _grpc_mock, mock.patch( + "grpc.ssl_channel_credentials", MagicMock(return_value="test") + ) as _mocked_credentials: client._connect_core() - _grpc_mock.assert_called_with(client.core_url, _mocked_credentials.return_value) \ No newline at end of file + _grpc_mock.assert_called_with( + client.core_url, _mocked_credentials.return_value + ) diff --git a/sdk/python/tests/test_feature_set.py b/sdk/python/tests/test_feature_set.py index 57d7a8f8100..2c539ebe0a7 100644 --- a/sdk/python/tests/test_feature_set.py +++ b/sdk/python/tests/test_feature_set.py @@ -11,21 +11,21 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. +from concurrent import futures from datetime import datetime +import grpc +import pandas as pd +import pytest import pytz +import dataframes +import feast.core.CoreService_pb2_grpc as Core +from feast.client import Client from feast.entity import Entity -from feast.feature_set import FeatureSet, Feature +from feast.feature_set import Feature, FeatureSet from feast.value_type import ValueType -from feast.client import Client -import pandas as pd -import pytest -from concurrent import futures -import grpc from feast_core_server import CoreServicer -import feast.core.CoreService_pb2_grpc as Core -import dataframes CORE_URL = "core.feast.local" SERVING_URL = "serving.feast.local" From 28f02d2aeef810ba866a40962d5d81f7d4a0a315 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Thu, 19 Mar 2020 13:24:40 +0800 Subject: [PATCH 075/176] Refactor Feast CLI/SDK configuration (#551) * Create new configuration class * Add new configuration class to CLI and Client class * Add new configuration class to CLI and Client class * Add project key to config in client --- sdk/python/feast/cli.py | 55 ++---- sdk/python/feast/client.py | 103 +++++------- sdk/python/feast/config.py | 288 +++++++++++++++++++------------- sdk/python/feast/constants.py | 33 +++- sdk/python/tests/test_client.py | 2 + sdk/python/tests/test_config.py | 139 +++++++++++++++ 6 files changed, 404 insertions(+), 216 deletions(-) create mode 100644 sdk/python/tests/test_config.py diff --git a/sdk/python/feast/cli.py b/sdk/python/feast/cli.py index 27b98b1086e..dc4784b3025 100644 --- a/sdk/python/feast/cli.py +++ b/sdk/python/feast/cli.py @@ -18,11 +18,10 @@ import click import pkg_resources -import toml import yaml -from feast import config as feast_config from feast.client import Client +from feast.config import Config from feast.feature_set import FeatureSet from feast.loaders.yaml import yaml_loader @@ -64,14 +63,7 @@ def version(client_only: bool, **kwargs): } if not client_only: - feast_client = Client( - core_url=feast_config.get_config_property_or_fail( - "core_url", force_config=kwargs - ), - serving_url=feast_config.get_config_property_or_fail( - "serving_url", force_config=kwargs - ), - ) + feast_client = Client(**kwargs) feast_versions_dict.update(feast_client.version()) print(json.dumps(feast_versions_dict)) @@ -94,13 +86,8 @@ def config_list(): """ List Feast properties for the currently active configuration """ - try: - feast_config_string = toml.dumps(feast_config._get_or_create_config()) - if not feast_config_string.strip(): - print("Configuration has not been set") - else: - print(feast_config_string.replace('""', "").strip()) + print(Config()) except Exception as e: _logger.error("Error occurred when reading Feast configuration file") _logger.exception(e) @@ -115,7 +102,9 @@ def config_set(prop, value): Set a Feast properties for the currently active configuration """ try: - feast_config.set_property(prop.strip(), value.strip()) + conf = Config() + conf.set(option=prop.strip(), value=value.strip()) + conf.save() except Exception as e: _logger.error("Error in reading config file") _logger.exception(e) @@ -135,9 +124,7 @@ def feature_set_list(): """ List all feature sets """ - feast_client = Client( - core_url=feast_config.get_config_property_or_fail("core_url") - ) # type: Client + feast_client = Client() # type: Client table = [] for fs in feast_client.list_feature_sets(): @@ -161,11 +148,7 @@ def feature_set_create(filename): """ feature_sets = [FeatureSet.from_dict(fs_dict) for fs_dict in yaml_loader(filename)] - - feast_client = Client( - core_url=feast_config.get_config_property_or_fail("core_url") - ) # type: Client - + feast_client = Client() # type: Client feast_client.apply(feature_sets) @@ -176,10 +159,7 @@ def feature_set_describe(name: str, version: int): """ Describe a feature set """ - feast_client = Client( - core_url=feast_config.get_config_property_or_fail("core_url") - ) # type: Client - + feast_client = Client() # type: Client fs = feast_client.get_feature_set(name=name, version=version) if not fs: print( @@ -204,9 +184,7 @@ def project_create(name: str): """ Create a project """ - feast_client = Client( - core_url=feast_config.get_config_property_or_fail("core_url") - ) # type: Client + feast_client = Client() # type: Client feast_client.create_project(name) @@ -216,9 +194,7 @@ def project_archive(name: str): """ Archive a project """ - feast_client = Client( - core_url=feast_config.get_config_property_or_fail("core_url") - ) # type: Client + feast_client = Client() # type: Client feast_client.archive_project(name) @@ -227,9 +203,7 @@ def project_list(): """ List all projects """ - feast_client = Client( - core_url=feast_config.get_config_property_or_fail("core_url") - ) # type: Client + feast_client = Client() # type: Client table = [] for project in feast_client.list_projects(): @@ -265,10 +239,7 @@ def ingest(name, version, filename, file_type): Ingest feature data into a feature set """ - feast_client = Client( - core_url=feast_config.get_config_property_or_fail("core_url") - ) # type: Client - + feast_client = Client() # type: Client feature_set = feast_client.get_feature_set(name=name, version=version) feature_set.ingest_file(file_path=filename) diff --git a/sdk/python/feast/client.py b/sdk/python/feast/client.py index ffdb71743d0..2a0b636b373 100644 --- a/sdk/python/feast/client.py +++ b/sdk/python/feast/client.py @@ -12,6 +12,7 @@ # See the License for the specific language governing permissions and # limitations under the License. + import logging import os import shutil @@ -26,6 +27,15 @@ import pyarrow as pa import pyarrow.parquet as pq +from feast.config import Config +from feast.constants import ( + CONFIG_CORE_SECURE_KEY, + CONFIG_CORE_URL_KEY, + CONFIG_GRPC_CONNECTION_TIMEOUT_DEFAULT_KEY, + CONFIG_PROJECT_KEY, + CONFIG_SERVING_SECURE_KEY, + CONFIG_SERVING_URL_KEY, +) from feast.core.CoreService_pb2 import ( ApplyFeatureSetRequest, ApplyFeatureSetResponse, @@ -63,14 +73,6 @@ _logger = logging.getLogger(__name__) -GRPC_CONNECTION_TIMEOUT_DEFAULT = 3 # type: int -GRPC_CONNECTION_TIMEOUT_APPLY = 600 # type: int -FEAST_CORE_URL_ENV_KEY = "FEAST_CORE_URL" -FEAST_SERVING_URL_ENV_KEY = "FEAST_SERVING_URL" -FEAST_PROJECT_ENV_KEY = "FEAST_PROJECT" -FEAST_CORE_SECURE_ENV_KEY = "FEAST_CORE_SECURE" -FEAST_SERVING_SECURE_ENV_KEY = "FEAST_SERVING_SECURE" -BATCH_FEATURE_REQUEST_WAIT_TIME_SECONDS = 300 CPU_COUNT = os.cpu_count() # type: int @@ -79,14 +81,7 @@ class Client: Feast Client: Used for creating, managing, and retrieving features. """ - def __init__( - self, - core_url: str = None, - serving_url: str = None, - project: str = None, - core_secure: bool = None, - serving_secure: bool = None, - ): + def __init__(self, options: Optional[Dict[str, str]] = None, **kwargs): """ The Feast Client should be initialized with at least one service url @@ -96,12 +91,15 @@ def __init__( project: Sets the active project. This field is optional. core_secure: Use client-side SSL/TLS for Core gRPC API serving_secure: Use client-side SSL/TLS for Serving gRPC API + options: Configuration options to initialize client with + **kwargs: Additional keyword arguments that will be used as + configuration options along with "options" """ - self._core_url: str = core_url - self._serving_url: str = serving_url - self._project: str = project - self._core_secure: bool = core_secure - self._serving_secure: bool = serving_secure + + if options is None: + options = dict() + self._config = Config(options={**options, **kwargs}) + self.__core_channel: grpc.Channel = None self.__serving_channel: grpc.Channel = None self._core_service_stub: CoreServiceStub = None @@ -115,12 +113,7 @@ def core_url(self) -> str: Returns: Feast Core URL string """ - - if self._core_url is not None: - return self._core_url - if os.getenv(FEAST_CORE_URL_ENV_KEY) is not None: - return os.getenv(FEAST_CORE_URL_ENV_KEY) - return "" + return self._config.get(CONFIG_CORE_URL_KEY) @core_url.setter def core_url(self, value: str): @@ -130,7 +123,7 @@ def core_url(self, value: str): Args: value: Feast Core URL """ - self._core_url = value + self._config.set(CONFIG_CORE_URL_KEY, value) @property def serving_url(self) -> str: @@ -140,11 +133,7 @@ def serving_url(self) -> str: Returns: Feast Serving URL string """ - if self._serving_url is not None: - return self._serving_url - if os.getenv(FEAST_SERVING_URL_ENV_KEY) is not None: - return os.getenv(FEAST_SERVING_URL_ENV_KEY) - return "" + return self._config.get(CONFIG_SERVING_URL_KEY) @serving_url.setter def serving_url(self, value: str): @@ -154,7 +143,7 @@ def serving_url(self, value: str): Args: value: Feast Serving URL """ - self._serving_url = value + self._config.set(CONFIG_SERVING_URL_KEY, value) @property def core_secure(self) -> bool: @@ -164,10 +153,7 @@ def core_secure(self) -> bool: Returns: Whether client-side SSL/TLS is enabled """ - - if self._core_secure is not None: - return self._core_secure - return os.getenv(FEAST_CORE_SECURE_ENV_KEY, "").lower() == "true" + return self._config.getboolean(CONFIG_CORE_SECURE_KEY) @core_secure.setter def core_secure(self, value: bool): @@ -177,7 +163,7 @@ def core_secure(self, value: bool): Args: value: True to enable client-side SSL/TLS """ - self._core_secure = value + self._config.set(CONFIG_CORE_SECURE_KEY, value) @property def serving_secure(self) -> bool: @@ -187,10 +173,7 @@ def serving_secure(self) -> bool: Returns: Whether client-side SSL/TLS is enabled """ - - if self._serving_secure is not None: - return self._serving_secure - return os.getenv(FEAST_SERVING_SECURE_ENV_KEY, "").lower() == "true" + return self._config.getboolean(CONFIG_SERVING_SECURE_KEY) @serving_secure.setter def serving_secure(self, value: bool): @@ -200,7 +183,7 @@ def serving_secure(self, value: bool): Args: value: True to enable client-side SSL/TLS """ - self._serving_secure = value + self._config.set(CONFIG_SERVING_SECURE_KEY, value) def version(self): """ @@ -211,14 +194,16 @@ def version(self): if self.serving_url: self._connect_serving() serving_version = self._serving_service_stub.GetFeastServingInfo( - GetFeastServingInfoRequest(), timeout=GRPC_CONNECTION_TIMEOUT_DEFAULT + GetFeastServingInfoRequest(), + timeout=self._config.getint(CONFIG_GRPC_CONNECTION_TIMEOUT_DEFAULT_KEY), ).version result["serving"] = {"url": self.serving_url, "version": serving_version} if self.core_url: self._connect_core() core_version = self._core_service_stub.GetFeastCoreVersion( - GetFeastCoreVersionRequest(), timeout=GRPC_CONNECTION_TIMEOUT_DEFAULT + GetFeastCoreVersionRequest(), + timeout=self._config.getint(CONFIG_GRPC_CONNECTION_TIMEOUT_DEFAULT_KEY), ).version result["core"] = {"url": self.core_url, "version": core_version} @@ -247,7 +232,7 @@ def _connect_core(self, skip_if_connected: bool = True): try: grpc.channel_ready_future(self.__core_channel).result( - timeout=GRPC_CONNECTION_TIMEOUT_DEFAULT + timeout=self._config.getint(CONFIG_GRPC_CONNECTION_TIMEOUT_DEFAULT_KEY) ) except grpc.FutureTimeoutError: raise ConnectionError( @@ -281,7 +266,7 @@ def _connect_serving(self, skip_if_connected=True): try: grpc.channel_ready_future(self.__serving_channel).result( - timeout=GRPC_CONNECTION_TIMEOUT_DEFAULT + timeout=self._config.getint(CONFIG_GRPC_CONNECTION_TIMEOUT_DEFAULT_KEY) ) except grpc.FutureTimeoutError: raise ConnectionError( @@ -299,11 +284,7 @@ def project(self) -> Union[str, None]: Returns: Project name """ - if self._project is not None: - return self._project - if os.getenv(FEAST_PROJECT_ENV_KEY) is not None: - return os.getenv(FEAST_PROJECT_ENV_KEY) - return None + return self._config.get(CONFIG_PROJECT_KEY) def set_project(self, project: str): """ @@ -312,7 +293,7 @@ def set_project(self, project: str): Args: project: Project to set as active """ - self._project = project + self._config.set(CONFIG_PROJECT_KEY, project) def list_projects(self) -> List[str]: """ @@ -324,7 +305,8 @@ def list_projects(self) -> List[str]: """ self._connect_core() response = self._core_service_stub.ListProjects( - ListProjectsRequest(), timeout=GRPC_CONNECTION_TIMEOUT_DEFAULT + ListProjectsRequest(), + timeout=self._config.getint(CONFIG_GRPC_CONNECTION_TIMEOUT_DEFAULT_KEY), ) # type: ListProjectsResponse return list(response.projects) @@ -338,7 +320,8 @@ def create_project(self, project: str): self._connect_core() self._core_service_stub.CreateProject( - CreateProjectRequest(name=project), timeout=GRPC_CONNECTION_TIMEOUT_DEFAULT + CreateProjectRequest(name=project), + timeout=self._config.getint(CONFIG_GRPC_CONNECTION_TIMEOUT_DEFAULT_KEY), ) # type: CreateProjectResponse def archive_project(self, project): @@ -353,7 +336,8 @@ def archive_project(self, project): self._connect_core() self._core_service_stub.ArchiveProject( - ArchiveProjectRequest(name=project), timeout=GRPC_CONNECTION_TIMEOUT_DEFAULT + ArchiveProjectRequest(name=project), + timeout=self._config.getint(CONFIG_GRPC_CONNECTION_TIMEOUT_DEFAULT_KEY), ) # type: ArchiveProjectResponse if self._project == project: @@ -402,7 +386,7 @@ def _apply_feature_set(self, feature_set: FeatureSet): try: apply_fs_response = self._core_service_stub.ApplyFeatureSet( ApplyFeatureSetRequest(feature_set=feature_set_proto), - timeout=GRPC_CONNECTION_TIMEOUT_APPLY, + timeout=self._config.getint(CONFIG_GRPC_CONNECTION_TIMEOUT_DEFAULT_KEY), ) # type: ApplyFeatureSetResponse except grpc.RpcError as e: raise grpc.RpcError(e.details()) @@ -573,7 +557,8 @@ def get_batch_features( # Retrieve serving information to determine store type and # staging location serving_info = self._serving_service_stub.GetFeastServingInfo( - GetFeastServingInfoRequest(), timeout=GRPC_CONNECTION_TIMEOUT_DEFAULT + GetFeastServingInfoRequest(), + timeout=self._config.getint(CONFIG_GRPC_CONNECTION_TIMEOUT_DEFAULT_KEY), ) # type: GetFeastServingInfoResponse if serving_info.type != FeastServingType.FEAST_SERVING_TYPE_BATCH: diff --git a/sdk/python/feast/config.py b/sdk/python/feast/config.py index 061bf24c3b7..fd35b6e5d87 100644 --- a/sdk/python/feast/config.py +++ b/sdk/python/feast/config.py @@ -13,152 +13,212 @@ # See the License for the specific language governing permissions and # limitations under the License. # - import logging import os -import sys +from configparser import ConfigParser, NoOptionError from os.path import expanduser, join -from typing import Dict -from urllib.parse import ParseResult, urlparse +from typing import Dict, Optional -import toml +from feast.constants import ( + CONFIG_FEAST_ENV_VAR_PREFIX, + CONFIG_FILE_DEFAULT_DIRECTORY, + CONFIG_FILE_NAME, + CONFIG_FILE_SECTION, + FEAST_CONFIG_FILE_ENV_KEY, +) +from feast.constants import FEAST_DEFAULT_OPTIONS as DEFAULTS _logger = logging.getLogger(__name__) -feast_configuration_properties = {"core_url": "URL", "serving_url": "URL"} -CONFIGURATION_FILE_DIR = os.environ.get("FEAST_CONFIG", ".feast") -CONFIGURATION_FILE_NAME = "config.toml" +def _init_config(path: str): + """ + Returns a ConfigParser that reads in a feast configuration file. If the + file does not exist it will be created. + Args: + path: Optional path to initialize as Feast configuration -def _get_or_create_config() -> Dict: - """Get user configuration file or create it and return""" + Returns: ConfigParser of the Feast configuration file, with defaults + preloaded - user_config_file_dir, user_config_file_path = _get_config_file_locations() - user_config_file_dir = user_config_file_dir.rstrip("/") + "/" - if not os.path.exists(os.path.dirname(user_config_file_dir)): - os.makedirs(os.path.dirname(user_config_file_dir)) + """ + # Create the configuration file directory if needed + config_dir = os.path.dirname(path) + config_dir = config_dir.rstrip("/") + "/" - if not os.path.isfile(user_config_file_path): - _save_config(user_config_file_path, _props_to_dict()) + if not os.path.exists(os.path.dirname(config_dir)): + os.makedirs(os.path.dirname(config_dir)) - try: - return toml.load(user_config_file_path) - except FileNotFoundError: - _logger.error( - "Could not find Feast configuration file " + user_config_file_path - ) - sys.exit(1) - except toml.decoder.TomlDecodeError: - _logger.error( - "Could not decode Feast configuration file " + user_config_file_path - ) - sys.exit(1) - except Exception as e: - _logger.error(e) - sys.exit(1) + # Create the configuration file itself + config = ConfigParser(defaults=DEFAULTS) + if os.path.exists(path): + config.read(path) + # Store all configuration in a single section + if not config.has_section(CONFIG_FILE_SECTION): + config.add_section(CONFIG_FILE_SECTION) -def set_property(prop: str, value: str): - """ - Sets a single property in the Feast users local configuration file + # Save the current configuration + config.write(open(path, "w")) - Args: - prop: Feast property name - value: Feast property value + return config + + +def _get_feast_env_vars(): """ - if _is_valid_property(prop, value): - active_feast_config = _get_or_create_config() - active_feast_config[prop] = value - _, user_config_file_path = _get_config_file_locations() - _save_config(user_config_file_path, active_feast_config) - print("Updated property [%s]" % prop) - else: - _logger.error("Invalid property selected") - sys.exit(1) - - -def get_config_property_or_fail(prop: str, force_config: Dict[str, str] = None) -> str: + Get environmental variables that start with FEAST_ + Returns: Dict of Feast environmental variables (stripped of prefix) """ - Gets a single property in the users configuration + feast_env_vars = {} + for key in os.environ.keys(): + if key.upper().startswith(CONFIG_FEAST_ENV_VAR_PREFIX): + feast_env_vars[key[len(CONFIG_FEAST_ENV_VAR_PREFIX) :]] = os.environ[key] + return feast_env_vars - Args: - prop: Property to retrieve - force_config: Configuration dictionary containing properties that should - be overridden. This will take precedence over file based properties. - Returns: - Returns a string property +class Config: + """ + Maintains and provides access to Feast configuration + + Configuration is stored as key/value pairs. The user can specify options + through either input arguments to this class, environmental variables, or + by setting the config in a configuration file + """ - if ( - isinstance(force_config, dict) - and prop in force_config - and force_config[prop] is not None + + def __init__( + self, options: Optional[Dict[str, str]] = None, path: Optional[str] = None, ): - return force_config[prop] + """ + Configuration options are returned as follows (higher replaces lower) + 1. Initialized options ("options" argument) + 2. Environmental variables (reloaded on every "get") + 3. Configuration file options (loaded once) + 4. Default options (loaded once from memory) + + Args: + options: (optional) A list of initialized/hardcoded options. + path: (optional) File path to configuration file + """ + if not path: + path = join( + expanduser("~"), + os.environ.get( + FEAST_CONFIG_FILE_ENV_KEY, CONFIG_FILE_DEFAULT_DIRECTORY, + ), + CONFIG_FILE_NAME, + ) + + config = _init_config(path) + + self._options = {} + if options and isinstance(options, dict): + self._options = options + + self._config = config # type: ConfigParser + self._path = path # type: str + + def get(self, option): + """ + Returns a single configuration option as a string + + Args: + option: Name of the option + + Returns: String option that is returned + + """ + return self._config.get( + CONFIG_FILE_SECTION, + option, + vars={**_get_feast_env_vars(), **self._options}, + ) - active_feast_config = _get_or_create_config() - if _is_valid_property(prop, active_feast_config[prop]): - return active_feast_config[prop] - _logger.error("Could not load Feast property from configuration: %s" % prop) - sys.exit(1) + def getboolean(self, option): + """ + Returns a single configuration option as a boolean + Args: + option: Name of the option -def _props_to_dict() -> Dict[str, str]: - """Create empty dictionary of all Feast properties""" - prop_dict = {} - for prop in feast_configuration_properties: - prop_dict[prop] = "" - return prop_dict + Returns: Boolean option value that is returned + """ + return self._config.getboolean( + CONFIG_FILE_SECTION, + option, + vars={**_get_feast_env_vars(), **self._options}, + ) -def _is_valid_property(prop: str, value: str) -> bool: - """ - Validates both a Feast property as well as value + def getint(self, option): + """ + Returns a single configuration option as an integer - Args: - prop: Feast property name - value: Feast property value + Args: + option: Name of the option - Returns: - Returns True if property and value are valid - """ - if prop not in feast_configuration_properties: - _logger.error("You are trying to set an invalid property") - sys.exit(1) + Returns: Integer option value that is returned - prop_type = feast_configuration_properties[prop] + """ + return self._config.getint( + CONFIG_FILE_SECTION, + option, + vars={**_get_feast_env_vars(), **self._options}, + ) - if prop_type == "URL": - if "//" not in value: - value = "%s%s" % ("grpc://", value) - parsed_value = urlparse(value) # type: ParseResult - if parsed_value.netloc: - return True + def getfloat(self, option): + """ + Returns a single configuration option as an integer - _logger.error("The property you are trying to set could not be identified") - sys.exit(1) + Args: + option: Name of the option + Returns: Float option value that is returned -def _save_config(user_config_file_path: str, config_string: Dict[str, str]): - """ - Saves Feast configuration + """ + return self._config.getfloat( + CONFIG_FILE_SECTION, + option, + vars={**_get_feast_env_vars(), **self._options}, + ) - Args: - user_config_file_path: Local file system path to save configuration - config_string: Contents in dictionary format to save to path - """ - try: - with open(user_config_file_path, "w+") as f: - toml.dump(config_string, f) - except Exception as e: - _logger.error("Could not update configuration file for Feast") - print(e) - sys.exit(1) - - -def _get_config_file_locations() -> (str, str): - """Gets the local user configuration directory and file path""" - user_config_file_dir = join(expanduser("~"), CONFIGURATION_FILE_DIR) - user_config_file_path = join(user_config_file_dir, CONFIGURATION_FILE_NAME) - return user_config_file_dir, user_config_file_path + def set(self, option, value): + """ + Sets a configuration option. Must be serializable to string + Args: + option: Option name to use as key + value: Value to store under option + """ + self._config.set(CONFIG_FILE_SECTION, option, value=str(value)) + + def exists(self, option): + """ + Tests whether a specific option is available + + Args: + option: Name of the option to check + + Returns: Boolean true/false whether the option is set + + """ + try: + self.get(option=option) + return True + except NoOptionError: + return False + + def save(self): + """ + Save the current configuration to disk. This does not include + environmental variables or initialized options + """ + self._config.write(open(self._path, "w")) + + def __str__(self): + result = "" + for section_name in self._config.sections(): + result += "\n[" + section_name + "]\n" + for name, value in self._config.items(section_name): + result += name + " = " + value + "\n" + return result diff --git a/sdk/python/feast/constants.py b/sdk/python/feast/constants.py index 9b001ac4067..c4bde75404a 100644 --- a/sdk/python/feast/constants.py +++ b/sdk/python/feast/constants.py @@ -14,4 +14,35 @@ # limitations under the License. # -DATETIME_COLUMN = "datetime" # type: str +# General constants +DATETIME_COLUMN = "datetime" +FEAST_CONFIG_FILE_ENV_KEY = "FEAST_CONFIG" +CONFIG_FEAST_ENV_VAR_PREFIX = "FEAST_" +CONFIG_FILE_DEFAULT_DIRECTORY = ".feast" +CONFIG_FILE_NAME = "config" +CONFIG_FILE_SECTION = "general" + + +# Feast configuration options +CONFIG_CORE_URL_KEY = "core_url" +CONFIG_SERVING_URL_KEY = "serving_url" +CONFIG_PROJECT_KEY = "project" +CONFIG_CORE_SECURE_KEY = "core_secure" +CONFIG_SERVING_SECURE_KEY = "serving_secure" +CONFIG_GRPC_CONNECTION_TIMEOUT_DEFAULT_KEY = "grpc_connection_timeout_default" +CONFIG_GRPC_CONNECTION_TIMEOUT_APPLY_KEY = "grpc_connection_timeout_apply_key" +CONFIG_BATCH_FEATURE_REQUEST_WAIT_TIME_SECONDS_KEY = ( + "batch_feature_request_wait_time_seconds" +) + +# Configuration option default values +FEAST_DEFAULT_OPTIONS = { + CONFIG_PROJECT_KEY: "default", + CONFIG_CORE_URL_KEY: "localhost:6565", + CONFIG_CORE_SECURE_KEY: "False", + CONFIG_SERVING_URL_KEY: "localhost:6565", + CONFIG_SERVING_SECURE_KEY: "False", + CONFIG_GRPC_CONNECTION_TIMEOUT_DEFAULT_KEY: "3", + CONFIG_GRPC_CONNECTION_TIMEOUT_APPLY_KEY: "600", + CONFIG_BATCH_FEATURE_REQUEST_WAIT_TIME_SECONDS_KEY: "600", +} diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index 1478256c066..b41500125cd 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -11,6 +11,8 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. + + import pkgutil import tempfile from concurrent import futures diff --git a/sdk/python/tests/test_config.py b/sdk/python/tests/test_config.py new file mode 100644 index 00000000000..9ed34a736a2 --- /dev/null +++ b/sdk/python/tests/test_config.py @@ -0,0 +1,139 @@ +# Copyright 2020 The Feast Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# https://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +from tempfile import mkstemp + +import pytest + +from feast.config import Config + + +class TestConfig: + @pytest.fixture + def normal_config(self): + fd, path = mkstemp() + return Config(path=path) + + def test_init_config_file_with_path(self): + configuration_string = "[general]\nCORE_URL = grpc://127.0.0.1:6565" + + fd, path = mkstemp() + with open(fd, "w") as f: + f.write(configuration_string) + config = Config(path=path) + assert config.get("core_url") == "grpc://127.0.0.1:6565" + + def test_load_environmental_variable(self, normal_config): + import os + + serving_url = "http://196.25.1.1" + os.environ["FEAST_SERVING_URL"] = serving_url + assert normal_config.get("SERVING_URL") == serving_url + del os.environ["FEAST_SERVING_URL"] + + def test_env_var_not_case_sensitive(self, normal_config): + import os + + serving_url = "http://196.25.1.1" + os.environ["FEAST_SerVING_url"] = serving_url + assert normal_config.get("SERVING_URL") == serving_url + + def test_force_options(self): + fd, path = mkstemp() + options = {"feast_config_1": "one", "random_config_two": 2} + config = Config(options, path) + assert config.get("feast_config_1") == "one" + + def test_init_options_precedence(self): + """ + Init options > env var > file options > default options + """ + fd, path = mkstemp() + os.environ["FEAST_CORE_URL"] = "env" + options = {"core_url": "init", "serving_url": "init"} + configuration_string = "[general]\nCORE_URL = file\n" + with open(fd, "w") as f: + f.write(configuration_string) + config = Config(options, path) + assert config.get("core_url") == "init" + del os.environ["FEAST_CORE_URL"] + + def test_env_var_precedence(self): + """ + Env vars > file options > default options + """ + fd, path = mkstemp() + os.environ["FEAST_CORE_URL"] = "env" + configuration_string = "[general]\nCORE_URL = file\n" + with open(fd, "w") as f: + f.write(configuration_string) + config = Config(path=path) + assert config.get("CORE_URL") == "env" + + del os.environ["FEAST_CORE_URL"] + + def test_file_option_precedence(self): + """ + file options > default options + """ + fd, path = mkstemp() + configuration_string = "[general]\nCORE_URL = file\n" + with open(fd, "w") as f: + f.write(configuration_string) + config = Config(path=path) + assert config.get("CORE_URL") == "file" + + def test_default_options(self): + """ + default options + """ + fd, path = mkstemp() + config = Config(path=path) + assert config.get("CORE_URL") == "localhost:6565" + + def test_type_casting(self): + """ + Test type casting of strings to other types + """ + fd, path = mkstemp() + os.environ["FEAST_INT_VAR"] = "1" + os.environ["FEAST_FLOAT_VAR"] = "1.0" + os.environ["FEAST_BOOLEAN_VAR"] = "True" + config = Config(path=path) + + assert config.getint("INT_VAR") == 1 + assert config.getfloat("FLOAT_VAR") == 1.0 + assert config.getboolean("BOOLEAN_VAR") is True + + def test_set_value(self): + """ + Test type casting of strings to other types + """ + fd, path = mkstemp() + config = Config(path=path) + config.set("my_val", 1) + + assert config.getint("my_val") == 1 + + def test_exists(self): + """ + Test type casting of strings to other types + """ + fd, path = mkstemp() + config = Config(path=path) + config.set("my_val_exist", 1) + + assert config.exists("my_val_exist") is True + assert config.exists("my_val_not_exist") is False From a72e2127bd002397106ce00c64ed31c09abeae2f Mon Sep 17 00:00:00 2001 From: Khor Shu Heng Date: Thu, 19 Mar 2020 10:57:33 +0800 Subject: [PATCH 076/176] Serving should not throw exception when key not found in Redis --- .../serving/service/RedisServingService.java | 2 +- .../service/RedisServingServiceTest.java | 42 +++++++------------ 2 files changed, 17 insertions(+), 27 deletions(-) diff --git a/serving/src/main/java/feast/serving/service/RedisServingService.java b/serving/src/main/java/feast/serving/service/RedisServingService.java index 56ee1e80ec7..78d9d9cebe4 100644 --- a/serving/src/main/java/feast/serving/service/RedisServingService.java +++ b/serving/src/main/java/feast/serving/service/RedisServingService.java @@ -328,7 +328,7 @@ private List sendMultiGet(List keys) { .collect(Collectors.toList()) .toArray(new byte[0][0]); return syncCommands.mget(binaryKeys).stream() - .map(io.lettuce.core.Value::getValue) + .map(keyValue -> keyValue.getValueOrElse(null)) .collect(Collectors.toList()); } catch (Exception e) { throw Status.NOT_FOUND diff --git a/serving/src/test/java/feast/serving/service/RedisServingServiceTest.java b/serving/src/test/java/feast/serving/service/RedisServingServiceTest.java index 8446218cfff..05a24d3fe6a 100644 --- a/serving/src/test/java/feast/serving/service/RedisServingServiceTest.java +++ b/serving/src/test/java/feast/serving/service/RedisServingServiceTest.java @@ -377,39 +377,29 @@ public void shouldReturnResponseWithUnsetValuesIfKeysNotPresent() { .putFields("entity2", strValue("b"))) .build(); - List featureRows = - Lists.newArrayList( - FeatureRow.newBuilder() - .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) - .addAllFields( - Lists.newArrayList( - Field.newBuilder().setName("entity1").setValue(intValue(1)).build(), - Field.newBuilder().setName("entity2").setValue(strValue("a")).build(), - Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), - Field.newBuilder().setName("feature2").setValue(intValue(1)).build())) - .setFeatureSet("featureSet:1") - .build(), - FeatureRow.newBuilder() - .setEventTimestamp(Timestamp.newBuilder()) - .addAllFields( - Lists.newArrayList( - Field.newBuilder().setName("entity1").setValue(intValue(2)).build(), - Field.newBuilder().setName("entity2").setValue(strValue("b")).build(), - Field.newBuilder().setName("feature1").build(), - Field.newBuilder().setName("feature2").build())) - .setFeatureSet("featureSet:1") - .build()); - FeatureSetRequest featureSetRequest = FeatureSetRequest.newBuilder() .addAllFeatureReferences(request.getFeaturesList()) .setSpec(getFeatureSetSpec()) .build(); + FeatureRow featureRowPresent = + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setName("entity1").setValue(intValue(1)).build(), + Field.newBuilder().setName("entity2").setValue(strValue("a")).build(), + Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), + Field.newBuilder().setName("feature2").setValue(intValue(1)).build())) + .setFeatureSet("featureSet:1") + .build(); + List> featureRowBytes = - featureRows.stream() - .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) - .collect(Collectors.toList()); + Lists.newArrayList( + KeyValue.from(new byte[1], Optional.of(featureRowPresent.toByteArray())), + KeyValue.from(new byte[1], Optional.empty())); + when(specService.getFeatureSets(request.getFeaturesList())) .thenReturn(Collections.singletonList(featureSetRequest)); when(connection.sync()).thenReturn(syncCommands); From 18235a92260b5952ddd85efe4f83b8c5d4c394c5 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Fri, 20 Mar 2020 13:33:54 +0800 Subject: [PATCH 077/176] Fix lint/format typo in Makefile --- Makefile | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Makefile b/Makefile index ee1978ecba4..47baacab016 100644 --- a/Makefile +++ b/Makefile @@ -20,7 +20,7 @@ PROTO_SERVICE_SUBDIRS = core serving # General -format: format-python lint-go lint-java +format: format-python format-go format-java lint: lint-python lint-go lint-java @@ -123,4 +123,4 @@ build-html: clean-html mkdir -p $(ROOT_DIR)/dist/grpc cd $(ROOT_DIR)/protos && $(MAKE) gen-docs cd $(ROOT_DIR)/sdk/python/docs && $(MAKE) html - cp -r $(ROOT_DIR)/sdk/python/docs/html/* $(ROOT_DIR)/dist/python \ No newline at end of file + cp -r $(ROOT_DIR)/sdk/python/docs/html/* $(ROOT_DIR)/dist/python From 283085450048c7d915fcb92523d0f8aee77322b2 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Sat, 21 Mar 2020 14:58:33 +0800 Subject: [PATCH 078/176] Add Feast CI image and infra housekeeping (#558) --- .prow/config.yaml | 42 +++++++++---------- Makefile | 37 ++++++++++++++-- infra/docker/ci/Dockerfile | 35 ++++++++++++++++ .../scripts/download-maven-cache.sh | 0 .../scripts/install-google-cloud-sdk.sh | 0 .../scripts/publish-docker-image.sh | 0 {.prow => infra}/scripts/publish-java-sdk.sh | 0 .../scripts/publish-python-sdk.sh | 0 {.prow => infra}/scripts/sync-helm-charts.sh | 0 .../scripts/test-core-ingestion.sh | 0 .../scripts/test-end-to-end-batch.sh | 0 {.prow => infra}/scripts/test-end-to-end.sh | 0 {.prow => infra}/scripts/test-golang-sdk.sh | 0 {.prow => infra}/scripts/test-java-sdk.sh | 0 {.prow => infra}/scripts/test-python-sdk.sh | 0 {.prow => infra}/scripts/test-serving.sh | 0 sdk/python/feast/cli.py | 2 +- sdk/python/feast/client.py | 2 +- sdk/python/feast/feature_set.py | 8 ++-- sdk/python/feast/job.py | 4 +- sdk/python/feast/loaders/file.py | 3 +- sdk/python/feast/loaders/ingest.py | 1 - sdk/python/feast/type_map.py | 6 +-- sdk/python/requirements-ci.txt | 39 ++--------------- sdk/python/requirements-dev.txt | 38 +++++++++++++++++ sdk/python/tests/test_client.py | 10 ++--- sdk/python/tests/test_config.py | 1 - sdk/python/tests/test_feature_set.py | 6 +-- 28 files changed, 151 insertions(+), 83 deletions(-) create mode 100644 infra/docker/ci/Dockerfile rename {.prow => infra}/scripts/download-maven-cache.sh (100%) rename {.prow => infra}/scripts/install-google-cloud-sdk.sh (100%) rename {.prow => infra}/scripts/publish-docker-image.sh (100%) rename {.prow => infra}/scripts/publish-java-sdk.sh (100%) rename {.prow => infra}/scripts/publish-python-sdk.sh (100%) rename {.prow => infra}/scripts/sync-helm-charts.sh (100%) rename {.prow => infra}/scripts/test-core-ingestion.sh (100%) rename {.prow => infra}/scripts/test-end-to-end-batch.sh (100%) rename {.prow => infra}/scripts/test-end-to-end.sh (100%) rename {.prow => infra}/scripts/test-golang-sdk.sh (100%) rename {.prow => infra}/scripts/test-java-sdk.sh (100%) rename {.prow => infra}/scripts/test-python-sdk.sh (100%) rename {.prow => infra}/scripts/test-serving.sh (100%) create mode 100644 sdk/python/requirements-dev.txt diff --git a/.prow/config.yaml b/.prow/config.yaml index 3ae9fcbe609..2e10ecfaa7f 100644 --- a/.prow/config.yaml +++ b/.prow/config.yaml @@ -67,7 +67,7 @@ presubmits: spec: containers: - image: maven:3.6-jdk-11 - command: [".prow/scripts/test-core-ingestion.sh"] + command: ["infra/scripts/test-core-ingestion.sh"] resources: requests: cpu: "2000m" @@ -81,7 +81,7 @@ presubmits: spec: containers: - image: maven:3.6-jdk-8 - command: [".prow/scripts/test-core-ingestion.sh"] + command: ["infra/scripts/test-core-ingestion.sh"] resources: requests: cpu: "2000m" @@ -95,7 +95,7 @@ presubmits: spec: containers: - image: maven:3.6-jdk-11 - command: [".prow/scripts/test-serving.sh"] + command: ["infra/scripts/test-serving.sh"] skip_branches: - ^v0\.(3|4)-branch$ @@ -105,7 +105,7 @@ presubmits: spec: containers: - image: maven:3.6-jdk-8 - command: [".prow/scripts/test-serving.sh"] + command: ["infra/scripts/test-serving.sh"] branches: - ^v0\.(3|4)-branch$ @@ -115,7 +115,7 @@ presubmits: spec: containers: - image: maven:3.6-jdk-11 - command: [".prow/scripts/test-java-sdk.sh"] + command: ["infra/scripts/test-java-sdk.sh"] skip_branches: - ^v0\.(3|4)-branch$ @@ -125,7 +125,7 @@ presubmits: spec: containers: - image: maven:3.6-jdk-8 - command: [".prow/scripts/test-java-sdk.sh"] + command: ["infra/scripts/test-java-sdk.sh"] branches: - ^v0\.(3|4)-branch$ @@ -135,7 +135,7 @@ presubmits: spec: containers: - image: python:3.7 - command: [".prow/scripts/test-python-sdk.sh"] + command: ["infra/scripts/test-python-sdk.sh"] - name: test-golang-sdk decorate: true @@ -143,7 +143,7 @@ presubmits: spec: containers: - image: golang:1.13 - command: [".prow/scripts/test-golang-sdk.sh"] + command: ["infra/scripts/test-golang-sdk.sh"] - name: test-end-to-end decorate: true @@ -151,7 +151,7 @@ presubmits: spec: containers: - image: maven:3.6-jdk-11 - command: [".prow/scripts/test-end-to-end.sh"] + command: ["infra/scripts/test-end-to-end.sh"] resources: requests: cpu: "6" @@ -165,7 +165,7 @@ presubmits: spec: containers: - image: maven:3.6-jdk-8 - command: [".prow/scripts/test-end-to-end.sh"] + command: ["infra/scripts/test-end-to-end.sh"] resources: requests: cpu: "6" @@ -183,7 +183,7 @@ presubmits: secretName: feast-service-account containers: - image: maven:3.6-jdk-11 - command: [".prow/scripts/test-end-to-end-batch.sh"] + command: ["infra/scripts/test-end-to-end-batch.sh"] resources: requests: cpu: "6" @@ -204,7 +204,7 @@ presubmits: secretName: feast-service-account containers: - image: maven:3.6-jdk-8 - command: [".prow/scripts/test-end-to-end-batch.sh"] + command: ["infra/scripts/test-end-to-end-batch.sh"] resources: requests: cpu: "6" @@ -226,7 +226,7 @@ postsubmits: - sh - -c - | - .prow/scripts/publish-python-sdk.sh \ + infra/scripts/publish-python-sdk.sh \ --directory-path sdk/python --repository pypi volumeMounts: - name: pypirc @@ -250,7 +250,7 @@ postsubmits: command: - bash - -c - - .prow/scripts/publish-java-sdk.sh --revision ${PULL_BASE_REF:1} + - infra/scripts/publish-java-sdk.sh --revision ${PULL_BASE_REF:1} volumeMounts: - name: gpg-keys mountPath: /etc/gpg @@ -282,7 +282,7 @@ postsubmits: command: - bash - -c - - .prow/scripts/publish-java-sdk.sh --revision ${PULL_BASE_REF:1} + - infra/scripts/publish-java-sdk.sh --revision ${PULL_BASE_REF:1} volumeMounts: - name: gpg-keys mountPath: /etc/gpg @@ -311,19 +311,19 @@ postsubmits: - bash - -c - | - .prow/scripts/download-maven-cache.sh \ + infra/scripts/download-maven-cache.sh \ --archive-uri gs://feast-templocation-kf-feast/.m2.2019-10-24.tar \ --output-dir $PWD/ if [ $PULL_BASE_REF == "master" ]; then - .prow/scripts/publish-docker-image.sh \ + infra/scripts/publish-docker-image.sh \ --repository gcr.io/kf-feast/feast-core \ --tag dev \ --file infra/docker/core/Dockerfile \ --google-service-account-file /etc/gcloud/service-account.json - .prow/scripts/publish-docker-image.sh \ + infra/scripts/publish-docker-image.sh \ --repository gcr.io/kf-feast/feast-serving \ --tag dev \ --file infra/docker/serving/Dockerfile \ @@ -337,13 +337,13 @@ postsubmits: else - .prow/scripts/publish-docker-image.sh \ + infra/scripts/publish-docker-image.sh \ --repository gcr.io/kf-feast/feast-core \ --tag ${PULL_BASE_REF:1} \ --file infra/docker/core/Dockerfile \ --google-service-account-file /etc/gcloud/service-account.json - .prow/scripts/publish-docker-image.sh \ + infra/scripts/publish-docker-image.sh \ --repository gcr.io/kf-feast/feast-serving \ --tag ${PULL_BASE_REF:1} \ --file infra/docker/serving/Dockerfile \ @@ -400,7 +400,7 @@ postsubmits: sed -i "/version: /c\version: ${PULL_BASE_REF:1}" infra/charts/feast/charts/feast-serving/Chart.yaml sed -i "/ tag: /c\ tag: ${PULL_BASE_REF:1}" infra/charts/feast/charts/feast-serving/values.yaml - .prow/scripts/sync-helm-charts.sh + infra/scripts/sync-helm-charts.sh volumeMounts: - name: service-account mountPath: /etc/gcloud/service-account.json diff --git a/Makefile b/Makefile index 47baacab016..2097e8db5a4 100644 --- a/Makefile +++ b/Makefile @@ -30,8 +30,15 @@ protos: compile-protos-go compile-protos-python compile-protos-docs build: protos build-java build-docker build-html +install-ci-dependencies: install-python-ci-dependencies install-go-ci-dependencies + # Java +install-java-ci-dependencies: + cd core; mvn dependency:go-offline + cd serving; mvn dependency:go-offline + cd ingestion; mvn dependency:go-offline + format-java: mvn spotless:apply @@ -70,6 +77,9 @@ lint-python: # Go SDK +install-go-ci-dependencies: + go get -u golang.org/x/lint/golint + compile-protos-go: @$(foreach dir,$(PROTO_TYPE_SUBDIRS), cd ${ROOT_DIR}/protos; protoc -I/usr/local/include -I. --go_out=plugins=grpc,paths=source_relative:../sdk/go/protos/ feast/$(dir)/*.proto;) @@ -81,15 +91,34 @@ lint-go: # Docker -build-docker: - docker build -t $(REGISTRY)/feast-core:$(VERSION) -f infra/docker/core/Dockerfile . - docker build -t $(REGISTRY)/feast-serving:$(VERSION) -f infra/docker/serving/Dockerfile . - build-push-docker: @$(MAKE) build-docker registry=$(REGISTRY) version=$(VERSION) + @$(MAKE) push-core-docker registry=$(REGISTRY) version=$(VERSION) + @$(MAKE) push-serving-docker registry=$(REGISTRY) version=$(VERSION) + @$(MAKE) push-ci-docker registry=$(REGISTRY) + +build-docker: build-core-docker build-serving-docker build-ci-docker + +push-core-docker: docker push $(REGISTRY)/feast-core:$(VERSION) + +push-serving-docker: docker push $(REGISTRY)/feast-serving:$(VERSION) +push-ci-docker: + docker push $(REGISTRY)/feast-ci:latest + +build-core-docker: + docker build -t $(REGISTRY)/feast-core:$(VERSION) -f infra/docker/core/Dockerfile . + +build-serving-docker: + docker build -t $(REGISTRY)/feast-serving:$(VERSION) -f infra/docker/serving/Dockerfile . + +build-ci-docker: + docker build -t $(REGISTRY)/feast-ci:latest -f infra/docker/ci/Dockerfile . + + + # Documentation install-dependencies-proto-docs: diff --git a/infra/docker/ci/Dockerfile b/infra/docker/ci/Dockerfile new file mode 100644 index 00000000000..350f9ddee79 --- /dev/null +++ b/infra/docker/ci/Dockerfile @@ -0,0 +1,35 @@ +FROM maven:3.6-jdk-11 + +ENV PYTHON_VERSION 3.7 +ENV GOLANG_VERSION 1.14.1 + +RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" \ + | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \ + curl https://packages.cloud.google.com/apt/doc/apt-key.gpg \ + | apt-key --keyring /usr/share/keyrings/cloud.google.gpg \ + add - && apt-get update -y && apt-get install google-cloud-sdk -y + +# Update dependencies +RUN apt-get update + +# Install Make and Python +RUN apt-get install -y build-essential curl python${PYTHON_VERSION} \ + python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-distutils && \ + update-alternatives --install /usr/bin/python python /usr/bin/python${PYTHON_VERSION} 1 && \ + update-alternatives --set python /usr/bin/python${PYTHON_VERSION} && \ + curl -s https://bootstrap.pypa.io/get-pip.py -o get-pip.py && \ + python get-pip.py --force-reinstall && \ + rm get-pip.py + + +# Install Go +RUN curl -O https://storage.googleapis.com/golang/go${GOLANG_VERSION}.linux-amd64.tar.gz && \ + tar -xvf go${GOLANG_VERSION}.linux-amd64.tar.gz && chown -R root:root ./go && mv go /usr/local +ENV GOPATH /go +ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH + +# Add contents of local Feast repository to image (execute from Feast root) +COPY . /feast/ + +# Install all dependencies +RUN cd /feast && make install-ci-dependencies \ No newline at end of file diff --git a/.prow/scripts/download-maven-cache.sh b/infra/scripts/download-maven-cache.sh similarity index 100% rename from .prow/scripts/download-maven-cache.sh rename to infra/scripts/download-maven-cache.sh diff --git a/.prow/scripts/install-google-cloud-sdk.sh b/infra/scripts/install-google-cloud-sdk.sh similarity index 100% rename from .prow/scripts/install-google-cloud-sdk.sh rename to infra/scripts/install-google-cloud-sdk.sh diff --git a/.prow/scripts/publish-docker-image.sh b/infra/scripts/publish-docker-image.sh similarity index 100% rename from .prow/scripts/publish-docker-image.sh rename to infra/scripts/publish-docker-image.sh diff --git a/.prow/scripts/publish-java-sdk.sh b/infra/scripts/publish-java-sdk.sh similarity index 100% rename from .prow/scripts/publish-java-sdk.sh rename to infra/scripts/publish-java-sdk.sh diff --git a/.prow/scripts/publish-python-sdk.sh b/infra/scripts/publish-python-sdk.sh similarity index 100% rename from .prow/scripts/publish-python-sdk.sh rename to infra/scripts/publish-python-sdk.sh diff --git a/.prow/scripts/sync-helm-charts.sh b/infra/scripts/sync-helm-charts.sh similarity index 100% rename from .prow/scripts/sync-helm-charts.sh rename to infra/scripts/sync-helm-charts.sh diff --git a/.prow/scripts/test-core-ingestion.sh b/infra/scripts/test-core-ingestion.sh similarity index 100% rename from .prow/scripts/test-core-ingestion.sh rename to infra/scripts/test-core-ingestion.sh diff --git a/.prow/scripts/test-end-to-end-batch.sh b/infra/scripts/test-end-to-end-batch.sh similarity index 100% rename from .prow/scripts/test-end-to-end-batch.sh rename to infra/scripts/test-end-to-end-batch.sh diff --git a/.prow/scripts/test-end-to-end.sh b/infra/scripts/test-end-to-end.sh similarity index 100% rename from .prow/scripts/test-end-to-end.sh rename to infra/scripts/test-end-to-end.sh diff --git a/.prow/scripts/test-golang-sdk.sh b/infra/scripts/test-golang-sdk.sh similarity index 100% rename from .prow/scripts/test-golang-sdk.sh rename to infra/scripts/test-golang-sdk.sh diff --git a/.prow/scripts/test-java-sdk.sh b/infra/scripts/test-java-sdk.sh similarity index 100% rename from .prow/scripts/test-java-sdk.sh rename to infra/scripts/test-java-sdk.sh diff --git a/.prow/scripts/test-python-sdk.sh b/infra/scripts/test-python-sdk.sh similarity index 100% rename from .prow/scripts/test-python-sdk.sh rename to infra/scripts/test-python-sdk.sh diff --git a/.prow/scripts/test-serving.sh b/infra/scripts/test-serving.sh similarity index 100% rename from .prow/scripts/test-serving.sh rename to infra/scripts/test-serving.sh diff --git a/sdk/python/feast/cli.py b/sdk/python/feast/cli.py index dc4784b3025..ec707ae08b7 100644 --- a/sdk/python/feast/cli.py +++ b/sdk/python/feast/cli.py @@ -18,8 +18,8 @@ import click import pkg_resources -import yaml +import yaml from feast.client import Client from feast.config import Config from feast.feature_set import FeatureSet diff --git a/sdk/python/feast/client.py b/sdk/python/feast/client.py index 2a0b636b373..2c8f7a75758 100644 --- a/sdk/python/feast/client.py +++ b/sdk/python/feast/client.py @@ -23,10 +23,10 @@ from typing import Dict, List, Optional, Tuple, Union import grpc + import pandas as pd import pyarrow as pa import pyarrow.parquet as pq - from feast.config import Config from feast.constants import ( CONFIG_CORE_SECURE_KEY, diff --git a/sdk/python/feast/feature_set.py b/sdk/python/feast/feature_set.py index 4ebfecf1675..869b14d0dcc 100644 --- a/sdk/python/feast/feature_set.py +++ b/sdk/python/feast/feature_set.py @@ -16,14 +16,12 @@ from collections import OrderedDict from typing import Dict, List, Optional -import pandas as pd -import pyarrow as pa from google.protobuf import json_format from google.protobuf.duration_pb2 import Duration from google.protobuf.json_format import MessageToJson -from pandas.api.types import is_datetime64_ns_dtype -from pyarrow.lib import TimestampType +import pandas as pd +import pyarrow as pa from feast.core.FeatureSet_pb2 import FeatureSet as FeatureSetProto from feast.core.FeatureSet_pb2 import FeatureSetMeta as FeatureSetMetaProto from feast.core.FeatureSet_pb2 import FeatureSetSpec as FeatureSetSpecProto @@ -36,6 +34,8 @@ pa_to_feast_value_type, python_type_to_feast_value_type, ) +from pandas.api.types import is_datetime64_ns_dtype +from pyarrow.lib import TimestampType class FeatureSet: diff --git a/sdk/python/feast/job.py b/sdk/python/feast/job.py index ab65da74459..4b1c9593730 100644 --- a/sdk/python/feast/job.py +++ b/sdk/python/feast/job.py @@ -3,10 +3,10 @@ from datetime import datetime, timedelta from urllib.parse import urlparse -import fastavro -import pandas as pd from google.cloud import storage +import fastavro +import pandas as pd from feast.serving.ServingService_pb2 import ( DATA_FORMAT_AVRO, JOB_STATUS_DONE, diff --git a/sdk/python/feast/loaders/file.py b/sdk/python/feast/loaders/file.py index 52cc8ae7dc8..4760eac2764 100644 --- a/sdk/python/feast/loaders/file.py +++ b/sdk/python/feast/loaders/file.py @@ -21,8 +21,9 @@ from typing import List, Optional, Tuple, Union from urllib.parse import ParseResult, urlparse -import pandas as pd from google.cloud import storage + +import pandas as pd from pandavro import to_avro diff --git a/sdk/python/feast/loaders/ingest.py b/sdk/python/feast/loaders/ingest.py index b4490f025c5..a5f0332fb91 100644 --- a/sdk/python/feast/loaders/ingest.py +++ b/sdk/python/feast/loaders/ingest.py @@ -5,7 +5,6 @@ import pandas as pd import pyarrow.parquet as pq - from feast.constants import DATETIME_COLUMN from feast.feature_set import FeatureSet from feast.type_map import ( diff --git a/sdk/python/feast/type_map.py b/sdk/python/feast/type_map.py index 8df0499239a..d25b14617b0 100644 --- a/sdk/python/feast/type_map.py +++ b/sdk/python/feast/type_map.py @@ -15,12 +15,11 @@ from datetime import datetime, timezone from typing import List +from google.protobuf.timestamp_pb2 import Timestamp + import numpy as np import pandas as pd import pyarrow as pa -from google.protobuf.timestamp_pb2 import Timestamp -from pyarrow.lib import TimestampType - from feast.constants import DATETIME_COLUMN from feast.types import FeatureRow_pb2 as FeatureRowProto from feast.types import Field_pb2 as FieldProto @@ -36,6 +35,7 @@ from feast.types.Value_pb2 import Value as ProtoValue from feast.types.Value_pb2 import ValueType as ProtoValueType from feast.value_type import ValueType +from pyarrow.lib import TimestampType def python_type_to_feast_value_type( diff --git a/sdk/python/requirements-ci.txt b/sdk/python/requirements-ci.txt index 2975342e24e..745ba25289d 100644 --- a/sdk/python/requirements-ci.txt +++ b/sdk/python/requirements-ci.txt @@ -1,38 +1,5 @@ -Click==7.* -google-api-core==1.* -google-auth==1.* -google-cloud-bigquery==1.* -google-cloud-bigquery-storage==0.* -google-cloud-storage==1.* -google-resumable-media>=0.5 -googleapis-common-protos==1.* -grpcio==1.* -numpy -mock==2.0.0 -pandas==0.* -protobuf==3.* -pytest -pytest-lazy-fixture==0.6.3 -pytest-mock -pytest-timeout -PyYAML==5.1.* -fastavro==0.* -grpcio-testing==1.* -pytest-ordering==0.6.* -pyarrow -Sphinx -sphinx-rtd-theme -toml==0.10.* -tqdm==4.* -confluent_kafka -google -pandavro==1.5.* -kafka-python==1.* -tabulate==0.8.* +flake8 +black isort grpcio-tools -mypy -mypy-protobuf -pre-commit -flake8 -black \ No newline at end of file +mypy-protobuf \ No newline at end of file diff --git a/sdk/python/requirements-dev.txt b/sdk/python/requirements-dev.txt new file mode 100644 index 00000000000..9c2c9d17d21 --- /dev/null +++ b/sdk/python/requirements-dev.txt @@ -0,0 +1,38 @@ +Click==7.* +google-api-core==1.* +google-auth==1.* +google-cloud-bigquery==1.* +google-cloud-bigquery-storage==0.* +google-cloud-storage==1.* +google-resumable-media>=0.5 +googleapis-common-protos==1.* +grpcio==1.* +grpcio-testing==1.* +grpcio-tools +numpy +mock==2.0.0 +pandas==0.* +protobuf==3.* +pytest +pytest-lazy-fixture==0.6.3 +pytest-mock +pytest-timeout +PyYAML==5.1.* +fastavro==0.* +pytest-ordering==0.6.* +pyarrow +Sphinx +sphinx-rtd-theme +toml==0.10.* +tqdm==4.* +confluent_kafka +google +pandavro==1.5.* +kafka-python==1.* +tabulate==0.8.* +isort +mypy +mypy-protobuf +pre-commit +flake8 +black \ No newline at end of file diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index b41500125cd..ac25fecfc2b 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -20,16 +20,13 @@ from unittest import mock import grpc -import pandas as pd -import pytest from google.protobuf.duration_pb2 import Duration -from mock import MagicMock, patch -from pandavro import to_avro -from pytz import timezone import dataframes import feast.core.CoreService_pb2_grpc as Core import feast.serving.ServingService_pb2_grpc as Serving +import pandas as pd +import pytest from feast.client import Client from feast.core.CoreService_pb2 import ( GetFeastCoreVersionResponse, @@ -61,6 +58,9 @@ from feast.value_type import ValueType from feast_core_server import CoreServicer from feast_serving_server import ServingServicer +from mock import MagicMock, patch +from pandavro import to_avro +from pytz import timezone CORE_URL = "core.feast.example.com" SERVING_URL = "serving.example.com" diff --git a/sdk/python/tests/test_config.py b/sdk/python/tests/test_config.py index 9ed34a736a2..8c30f2562cc 100644 --- a/sdk/python/tests/test_config.py +++ b/sdk/python/tests/test_config.py @@ -16,7 +16,6 @@ from tempfile import mkstemp import pytest - from feast.config import Config diff --git a/sdk/python/tests/test_feature_set.py b/sdk/python/tests/test_feature_set.py index 2c539ebe0a7..1793161fdbe 100644 --- a/sdk/python/tests/test_feature_set.py +++ b/sdk/python/tests/test_feature_set.py @@ -15,12 +15,12 @@ from datetime import datetime import grpc -import pandas as pd -import pytest -import pytz import dataframes import feast.core.CoreService_pb2_grpc as Core +import pandas as pd +import pytest +import pytz from feast.client import Client from feast.entity import Entity from feast.feature_set import Feature, FeatureSet From ee66f65988a6ed9ff54e6baa212db30c7fbeba85 Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Sat, 21 Mar 2020 15:05:01 +0800 Subject: [PATCH 079/176] Fix Python Formatting --- sdk/python/feast/cli.py | 2 +- sdk/python/feast/client.py | 2 +- sdk/python/feast/feature_set.py | 8 ++++---- sdk/python/feast/job.py | 4 ++-- sdk/python/feast/loaders/file.py | 3 +-- sdk/python/feast/loaders/ingest.py | 1 + sdk/python/feast/type_map.py | 6 +++--- sdk/python/tests/test_client.py | 10 +++++----- sdk/python/tests/test_config.py | 1 + sdk/python/tests/test_feature_set.py | 6 +++--- 10 files changed, 22 insertions(+), 21 deletions(-) diff --git a/sdk/python/feast/cli.py b/sdk/python/feast/cli.py index ec707ae08b7..dc4784b3025 100644 --- a/sdk/python/feast/cli.py +++ b/sdk/python/feast/cli.py @@ -18,8 +18,8 @@ import click import pkg_resources - import yaml + from feast.client import Client from feast.config import Config from feast.feature_set import FeatureSet diff --git a/sdk/python/feast/client.py b/sdk/python/feast/client.py index 2c8f7a75758..2a0b636b373 100644 --- a/sdk/python/feast/client.py +++ b/sdk/python/feast/client.py @@ -23,10 +23,10 @@ from typing import Dict, List, Optional, Tuple, Union import grpc - import pandas as pd import pyarrow as pa import pyarrow.parquet as pq + from feast.config import Config from feast.constants import ( CONFIG_CORE_SECURE_KEY, diff --git a/sdk/python/feast/feature_set.py b/sdk/python/feast/feature_set.py index 869b14d0dcc..4ebfecf1675 100644 --- a/sdk/python/feast/feature_set.py +++ b/sdk/python/feast/feature_set.py @@ -16,12 +16,14 @@ from collections import OrderedDict from typing import Dict, List, Optional +import pandas as pd +import pyarrow as pa from google.protobuf import json_format from google.protobuf.duration_pb2 import Duration from google.protobuf.json_format import MessageToJson +from pandas.api.types import is_datetime64_ns_dtype +from pyarrow.lib import TimestampType -import pandas as pd -import pyarrow as pa from feast.core.FeatureSet_pb2 import FeatureSet as FeatureSetProto from feast.core.FeatureSet_pb2 import FeatureSetMeta as FeatureSetMetaProto from feast.core.FeatureSet_pb2 import FeatureSetSpec as FeatureSetSpecProto @@ -34,8 +36,6 @@ pa_to_feast_value_type, python_type_to_feast_value_type, ) -from pandas.api.types import is_datetime64_ns_dtype -from pyarrow.lib import TimestampType class FeatureSet: diff --git a/sdk/python/feast/job.py b/sdk/python/feast/job.py index 4b1c9593730..ab65da74459 100644 --- a/sdk/python/feast/job.py +++ b/sdk/python/feast/job.py @@ -3,10 +3,10 @@ from datetime import datetime, timedelta from urllib.parse import urlparse -from google.cloud import storage - import fastavro import pandas as pd +from google.cloud import storage + from feast.serving.ServingService_pb2 import ( DATA_FORMAT_AVRO, JOB_STATUS_DONE, diff --git a/sdk/python/feast/loaders/file.py b/sdk/python/feast/loaders/file.py index 4760eac2764..52cc8ae7dc8 100644 --- a/sdk/python/feast/loaders/file.py +++ b/sdk/python/feast/loaders/file.py @@ -21,9 +21,8 @@ from typing import List, Optional, Tuple, Union from urllib.parse import ParseResult, urlparse -from google.cloud import storage - import pandas as pd +from google.cloud import storage from pandavro import to_avro diff --git a/sdk/python/feast/loaders/ingest.py b/sdk/python/feast/loaders/ingest.py index a5f0332fb91..b4490f025c5 100644 --- a/sdk/python/feast/loaders/ingest.py +++ b/sdk/python/feast/loaders/ingest.py @@ -5,6 +5,7 @@ import pandas as pd import pyarrow.parquet as pq + from feast.constants import DATETIME_COLUMN from feast.feature_set import FeatureSet from feast.type_map import ( diff --git a/sdk/python/feast/type_map.py b/sdk/python/feast/type_map.py index d25b14617b0..8df0499239a 100644 --- a/sdk/python/feast/type_map.py +++ b/sdk/python/feast/type_map.py @@ -15,11 +15,12 @@ from datetime import datetime, timezone from typing import List -from google.protobuf.timestamp_pb2 import Timestamp - import numpy as np import pandas as pd import pyarrow as pa +from google.protobuf.timestamp_pb2 import Timestamp +from pyarrow.lib import TimestampType + from feast.constants import DATETIME_COLUMN from feast.types import FeatureRow_pb2 as FeatureRowProto from feast.types import Field_pb2 as FieldProto @@ -35,7 +36,6 @@ from feast.types.Value_pb2 import Value as ProtoValue from feast.types.Value_pb2 import ValueType as ProtoValueType from feast.value_type import ValueType -from pyarrow.lib import TimestampType def python_type_to_feast_value_type( diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index ac25fecfc2b..b41500125cd 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -20,13 +20,16 @@ from unittest import mock import grpc +import pandas as pd +import pytest from google.protobuf.duration_pb2 import Duration +from mock import MagicMock, patch +from pandavro import to_avro +from pytz import timezone import dataframes import feast.core.CoreService_pb2_grpc as Core import feast.serving.ServingService_pb2_grpc as Serving -import pandas as pd -import pytest from feast.client import Client from feast.core.CoreService_pb2 import ( GetFeastCoreVersionResponse, @@ -58,9 +61,6 @@ from feast.value_type import ValueType from feast_core_server import CoreServicer from feast_serving_server import ServingServicer -from mock import MagicMock, patch -from pandavro import to_avro -from pytz import timezone CORE_URL = "core.feast.example.com" SERVING_URL = "serving.example.com" diff --git a/sdk/python/tests/test_config.py b/sdk/python/tests/test_config.py index 8c30f2562cc..9ed34a736a2 100644 --- a/sdk/python/tests/test_config.py +++ b/sdk/python/tests/test_config.py @@ -16,6 +16,7 @@ from tempfile import mkstemp import pytest + from feast.config import Config diff --git a/sdk/python/tests/test_feature_set.py b/sdk/python/tests/test_feature_set.py index 1793161fdbe..2c539ebe0a7 100644 --- a/sdk/python/tests/test_feature_set.py +++ b/sdk/python/tests/test_feature_set.py @@ -15,12 +15,12 @@ from datetime import datetime import grpc - -import dataframes -import feast.core.CoreService_pb2_grpc as Core import pandas as pd import pytest import pytz + +import dataframes +import feast.core.CoreService_pb2_grpc as Core from feast.client import Client from feast.entity import Entity from feast.feature_set import Feature, FeatureSet From 64e6d63f03f78bcf488050f2bc7acf90cc0e798c Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Sat, 21 Mar 2020 15:33:15 +0800 Subject: [PATCH 080/176] Remove isort checking since it is non-deterministic --- Makefile | 1 - 1 file changed, 1 deletion(-) diff --git a/Makefile b/Makefile index 2097e8db5a4..d6908544248 100644 --- a/Makefile +++ b/Makefile @@ -73,7 +73,6 @@ lint-python: #cd ${ROOT_DIR}/sdk/python; mypy feast/ tests/ cd ${ROOT_DIR}/sdk/python; flake8 feast/ tests/ cd ${ROOT_DIR}/sdk/python; black --check feast tests - cd ${ROOT_DIR}/sdk/python; isort -rc feast tests --check-only # Go SDK From 6a790151bb09a1e2995e4c7b943f9ea66f6007f2 Mon Sep 17 00:00:00 2001 From: Ches Martin Date: Sat, 21 Mar 2020 09:07:46 +0000 Subject: [PATCH 081/176] GitBook: [master] 10 pages and one asset modified --- docs/contributing.md | 55 +++++++++++++++++++++++++++++++++++++------- docs/roadmap.md | 19 ++++++++------- 2 files changed, 56 insertions(+), 18 deletions(-) diff --git a/docs/contributing.md b/docs/contributing.md index f3394e12ab9..b451de39ab7 100644 --- a/docs/contributing.md +++ b/docs/contributing.md @@ -117,7 +117,10 @@ A solution to this is: 1. Open `View > Tool Windows > Maven` 2. Drill down to e.g. `Feast Core > Plugins > spring-boot:run`, right-click and `Create 'feast-core [spring-boot'…` 3. In the dialog that pops up, check the `Resolve Workspace artifacts` box -4. Click `OK`. You should now be able to select this run configuration for the Play button in the main toolbar, keyboard shortcuts, etc. +4. Recommended: add `-Dspring-boot.run.fork=false` to the `Command line` field to get Debug working too +5. Click `OK`. You should now be able to select this run configuration for the Play button in the main toolbar, keyboard shortcuts, etc. + +It is recommend to have IntelliJ delegate building to Maven, if this is not enabled out of the box when you import the project, for greater assurance that build behavior is consistent with CI / production builds. This is set in Preferences at `Build, Execution, Deployment > Build Tools > Maven > Runner > Delegate IDE build/run actions to Maven`. ### 2.**4** Validating your setup @@ -462,30 +465,66 @@ It is important to note that most of the functionality demonstrated above is alr ## 3. Style guide +Coding standards are checked automatically by continuous integration. Checking them during development will streamline your contributing experience. + +For set-and-forget, you may install [pre-commit](https://pre-commit.com/) and run `pre-commit install` in the root of the Feast project. This installs a Git hook that automatically formats code before you commit. You will need to have some development tools for all project languages installed, though. + +If you have GNU Make, convenience targets are available such as `make lint-python`, `make format-java`, and comprehensive `make lint` and `format`, so that you don't need to remember all the incantations for language-specific tools. + +Auto-formatting is a boon to code review and patch management over time, but it isn't perfect. Spot check your changes after formatting, and if something "looks bad", consider whether a small change like introducing a variable could improve readability for real people. + ### 3.1 Java -We conform to the [Google Java Style Guide](https://google.github.io/styleguide/javaguide.html). Maven can helpfully take care of that for you before you commit: +We conform to the [Google Java Style Guide](https://google.github.io/styleguide/javaguide.html). Maven can helpfully take care of that for you: + +```text +$ make format-java +``` + +Or if you don't have Make installed: ```text $ mvn spotless:apply ``` -Formatting will be checked automatically during the `verify` phase. This can be skipped temporarily: +Formatting is checked automatically just prior to the `test` phase of the build lifecycle, in effort to help us remember it before committing without hampering development flow too greatly. This can be skipped temporarily: ```text -$ mvn spotless:check # Check is automatic upon `mvn verify` -$ mvn verify -Dspotless.check.skip +$ mvn spotless:check # Check is automatic upon `mvn test` +$ mvn test -Dspotless.check.skip ``` -If you're using IntelliJ, you can import [these code style settings](https://github.com/google/styleguide/blob/gh-pages/intellij-java-google-style.xml) if you'd like to use the IDE's reformat function as you develop. +If you're using IntelliJ, you can install the [google-java-format plugin](https://plugins.jetbrains.com/plugin/8527-google-java-format) if you'd like to use the IDE's reformat function as you develop. _Note that this uses a built-in version of the formatter that may drift from the Maven plugin over time—_`make lint-java` _is authoritative for CI, so it's still a good idea to check before submitting pull requests._ ### 3.2 Go -Make sure you apply `go fmt`. +`gofmt` style is followed for formatting. `go vet` and `golint` static analysis are also checked by: + +```text +$ make lint-go +``` + +Apply formatting with: + +```text +$ make format-go +``` + +`gofmt` is so ubiquitous in the Go ecosystem, it's almost certain that a plugin exists for your editor of choice to format files as you work. ### 3.3 Python -We use [Python Black](https://github.com/psf/black) to format our Python code prior to submission. +We use [Python Black](https://github.com/psf/black) to format our Python code prior to submission. To check it along with static analysis: + +```text +$ make lint-python +``` + +And to format: + +```text +$ make format-python +``` ## 4. Release process diff --git a/docs/roadmap.md b/docs/roadmap.md index b423a0fe12e..41d8a6b190e 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -4,7 +4,7 @@ [Discussion](https://github.com/gojek/feast/issues/527) -#### New functionality +### New functionality 1. Streaming statistics and validation \(M1 from [Feature Validation RFC](https://docs.google.com/document/d/1TPmd7r4mniL9Y-V_glZaWNo5LMXLshEAUpYsohojZ-8/edit)\) 2. Batch statistics and validation \(M2 from [Feature Validation RFC](https://docs.google.com/document/d/1TPmd7r4mniL9Y-V_glZaWNo5LMXLshEAUpYsohojZ-8/edit)\) @@ -12,20 +12,21 @@ 4. User authentication & authorization \([\#504](https://github.com/gojek/feast/issues/504)\) 5. Add feature or feature set descriptions \([\#463](https://github.com/gojek/feast/issues/463)\) 6. Redis Cluster Support \([\#478](https://github.com/gojek/feast/issues/478)\) -7. Job management API ([\#302](https://github.com/gojek/feast/issues/302)\) +7. Job management API \([\#302](https://github.com/gojek/feast/issues/302)\) -#### Technical debt, refactoring, or housekeeping -1. Clean up and document all configuration options ([\#525](https://github.com/gojek/feast/issues/525)\) -2. Externalize storage interfaces ([\#402](https://github.com/gojek/feast/issues/402)\) +### Technical debt, refactoring, or housekeeping + +1. Clean up and document all configuration options \([\#525](https://github.com/gojek/feast/issues/525)\) +2. Externalize storage interfaces \([\#402](https://github.com/gojek/feast/issues/402)\) 3. Reduce memory usage in Redis \([\#515](https://github.com/gojek/feast/issues/515)\) 4. Support for handling out of order ingestion \([\#273](https://github.com/gojek/feast/issues/273)\) 5. Remove feature versions and enable automatic data migration \([\#386](https://github.com/gojek/feast/issues/386)\) \([\#462](https://github.com/gojek/feast/issues/462)\) 6. Tracking of batch ingestion by with dataset\_id/job\_id \([\#461](https://github.com/gojek/feast/issues/461)\) -7. Write Beam metrics after ingestion to store (not prior) \([\#489](https://github.com/gojek/feast/issues/489)\) +7. Write Beam metrics after ingestion to store \(not prior\) \([\#489](https://github.com/gojek/feast/issues/489)\) ## Feast 0.6 -#### New functionality +### New functionality 1. Extended discovery API/SDK \(needs to be scoped 1. Resource listing @@ -36,7 +37,7 @@ 3. Add support for audit logs \(needs to be scoped\) 4. Support for an open source warehouse store or connector \(needs to be scoped\) -#### Technical debt, refactoring, or housekeeping +### Technical debt, refactoring, or housekeeping 1. Move all non-registry functionality out of Feast Core and make it optional \(needs to be scoped\) 1. Allow Feast serving to use its own local feature sets \(files\) @@ -47,5 +48,3 @@ 2. Implement interface for adding a managed data store 3. Multi-store support for serving \(batch and online\) \(needs to be scoped\) - - From 1c15f65aecdd3319d36686ceaf2dae9e42701d15 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Sat, 21 Mar 2020 22:22:04 +0800 Subject: [PATCH 082/176] Add GitHub Action for Linting (#559) * Add linting workflow to project * Refactor makefile to have java dependency installation and go tests --- .github/workflows/lint.yaml | 33 +++++++++++++++++++++++++++++++++ Makefile | 13 ++++++------- 2 files changed, 39 insertions(+), 7 deletions(-) create mode 100644 .github/workflows/lint.yaml diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml new file mode 100644 index 00000000000..8bd80785130 --- /dev/null +++ b/.github/workflows/lint.yaml @@ -0,0 +1,33 @@ +name: Lint + +on: + push: + branches: [ master ] + pull_request: + branches: [ master ] + +jobs: + lint-java: + container: gcr.io/kf-feast/feast-ci:latest + runs-on: [ubuntu-latest] + steps: + - uses: actions/checkout@v2 + - name: Lint Java + run: make lint-java + + lint-python: + container: gcr.io/kf-feast/feast-ci:latest + runs-on: [ubuntu-latest] + steps: + - uses: actions/checkout@v2 + + - name: Lint Python + run: make lint-python + + lint-go: + container: gcr.io/kf-feast/feast-ci:latest + runs-on: [ubuntu-latest] + steps: + - uses: actions/checkout@v2 + - name: Lint Go + run: make lint-go \ No newline at end of file diff --git a/Makefile b/Makefile index d6908544248..cddee59d8ea 100644 --- a/Makefile +++ b/Makefile @@ -24,20 +24,18 @@ format: format-python format-go format-java lint: lint-python lint-go lint-java -test: test-python test-java +test: test-python test-java test-go protos: compile-protos-go compile-protos-python compile-protos-docs build: protos build-java build-docker build-html -install-ci-dependencies: install-python-ci-dependencies install-go-ci-dependencies +install-ci-dependencies: install-python-ci-dependencies install-go-ci-dependencies install-java-ci-dependencies # Java install-java-ci-dependencies: - cd core; mvn dependency:go-offline - cd serving; mvn dependency:go-offline - cd ingestion; mvn dependency:go-offline + mvn verify clean --fail-never format-java: mvn spotless:apply @@ -82,6 +80,9 @@ install-go-ci-dependencies: compile-protos-go: @$(foreach dir,$(PROTO_TYPE_SUBDIRS), cd ${ROOT_DIR}/protos; protoc -I/usr/local/include -I. --go_out=plugins=grpc,paths=source_relative:../sdk/go/protos/ feast/$(dir)/*.proto;) +test-go: + cd ${ROOT_DIR}/sdk/go; go test ./... + format-go: cd ${ROOT_DIR}/sdk/go; gofmt -s -w *.go @@ -116,8 +117,6 @@ build-serving-docker: build-ci-docker: docker build -t $(REGISTRY)/feast-ci:latest -f infra/docker/ci/Dockerfile . - - # Documentation install-dependencies-proto-docs: From 5ff4556940ec2977ef34a6cd306260ce09c79b2f Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Sun, 22 Mar 2020 12:28:18 +0000 Subject: [PATCH 083/176] GitBook: [master] one page modified --- docs/why-feast.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/why-feast.md b/docs/why-feast.md index 24c7c0c129c..f5551d5f82f 100644 --- a/docs/why-feast.md +++ b/docs/why-feast.md @@ -22,7 +22,7 @@ **Problem:** Teams define features differently and there is no easy access to the documentation of a feature. -**Solution:** Feast becomes the single source of truth for all feature data for all models within an organizations. Teams are able to capture documentation, metadata and metrics about features. This allows teams to communicate clearly about features, test features data, and determine if a feature is useful for a particular model. +**Solution:** Feast becomes the single source of truth for all feature data for all models within an organization. Teams are able to capture documentation, metadata and metrics about features. This allows teams to communicate clearly about features, test features data, and determine if a feature is useful for a particular model. ## **Inconsistency between training and serving** From e01c9a6aaec90d5b4787d87beda4d419d84ea01c Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Sun, 22 Mar 2020 13:36:48 +0000 Subject: [PATCH 084/176] GitBook: [master] 14 pages and 2 assets modified --- docs/.gitbook/assets/image.png | Bin 0 -> 13935 bytes docs/README.md | 2 +- docs/SUMMARY.md | 38 +- docs/concepts/concepts.md | 124 ++++++ docs/contributing/contributing.md | 542 +++++++++++++++++++++++++ docs/contributing/development-guide.md | 446 ++++++++++++++++++++ docs/contributing/release-process.md | 14 + docs/contributing/style-guide.md | 27 ++ docs/installation/docker-compose.md | 112 +++++ docs/installation/gke.md | 211 ++++++++++ docs/installation/overview.md | 14 + docs/installation/troubleshooting.md | 168 ++++++++ docs/introduction/getting-help.md | 36 ++ docs/introduction/why-feast.md | 34 ++ 14 files changed, 1756 insertions(+), 12 deletions(-) create mode 100644 docs/.gitbook/assets/image.png create mode 100644 docs/concepts/concepts.md create mode 100644 docs/contributing/contributing.md create mode 100644 docs/contributing/development-guide.md create mode 100644 docs/contributing/release-process.md create mode 100644 docs/contributing/style-guide.md create mode 100644 docs/installation/docker-compose.md create mode 100644 docs/installation/gke.md create mode 100644 docs/installation/overview.md create mode 100644 docs/installation/troubleshooting.md create mode 100644 docs/introduction/getting-help.md create mode 100644 docs/introduction/why-feast.md diff --git a/docs/.gitbook/assets/image.png b/docs/.gitbook/assets/image.png new file mode 100644 index 0000000000000000000000000000000000000000..273924d8cdbdcc1e0fd5993210d6e2bf1e0ea33f GIT binary patch literal 13935 zcmeHucT|&Ew>JAj;@}lhu4_ zfnw-6B(dT4+@5+RB&PHT-`wB5K_&f?3^_(WZ{x@T?w*$^*S3w{{V^~*5Kpm7>tjA0 zp*cRyzmRXwsw$XScns$s_%Z*77$=zxy{)ZQnCWbl|Rbe zh{GF731b~7*+3K^(IZ*#Q~%{VZ#4GaCdZ{$8wS**q8svj9R633{~sF3IZVM!fAKcO zj9>7`jPE=9DA4Ypd3(}EIVm{zt_=1vZ)N!6wbptw`BbmnWzUYDQ&Kh$eb#vVr=`F9 zwRf5o{u)MiXS_9kp#@`^$=KXUi`$uI1neeOB{}ud;j+83LQlBj@80Zhz#Hg^gQnX+ z(Qn%@=s_CM-jjE~?DFCGm`<25_Adn;65zm^}> zk$!Dw+GR)n)NhuF?2VGKfx5c;S^lgP zSDLi#S=l`p=wfcSmxGDbDU-}EBAMuh6xnG!e+ATYgKneB8z}}1<8r`ji>aaE=aR(d z_|ChUrAQ_2j=)Mfh#F5+Ncii{j1M79qx>#ClX8Apk$MDG@u5DQ~rLpB6 z;n{}U|LF0gj?YOCW+?cXvWAZdmHS(d2rDl)xh1_f5!(*`ih65)8Z1coCe}PlX=z6O zP&6i@>7~+FMY}|1x$jN0Nt zoYv)8e}8S9Hvu%{wU>a*g610ghCvccZxl4RyXF#f2$Lp)x##Yjq&$JSK3q0L1{D~% z%?|smBoPG%t^Fw{62_H9k*L4zf|~B(t5m2hsJ24>bF>|P(3=ZRyNt&vumm~0}W$zjQr3s zGSgUjlPr~!VelX$2wG7!k^6??I`0th=s3D!3uVxvTU$DU_aMfa?B^;m@w6 zLyZGYi|m0F2@g~F=m{iENKrXjgCJtw#4H{c@txA)Hs7RUN4~U;-v^eFnQ`NcCTj&W zanv1+ld>fvI9I|B*mN1vA?(#j`!nP>Mgq9ALzv|Xn@8A)u{*AH1?Aa`zs6ap=F~@4 z)M=xT?EJ%+1Bi;GU3#i7(UbuTR1AreGUxAFP3#+aBm8Y#T~_aD@_i(nAYh}0AMSLT z@4OiZTcPD;X>kU0`hKVH7H{Ddhj%NRn&CZz*G^&$+^s0uJ(WLcwyr)XmxrG+4<0$k zcB}WvK=W@BIM18Ie7OGB`V!W_x&f4*b*tdj;_t0q1E^8s1znwf&S1>cBU-gr=*ihp zV(0+hsX2#8=+GXlGjaWC1}hj}D6*-$;QFAo!{h{qrFXrK-;Uu|dmNFmL8^AGPL$&?7&4s_vgJrY0Bb)$x?J#yhvutTHq>a-}0QEBKsov&rH3 zzHR|oF*F0)JeA}knzP{_zQU^U{oD`6o=PR|a*$N=GpuBzOC5?gfu*7%RE90K1k2+$ zUiuutdMDhewaWpgiYX81nu zOpK^AK8IXEM7ZV1FY>BWd>9V54JK?(_^42a^ZH1{syuNKvQB~PXaAd@{V$vdDH~IN zBU68WhcV`HDxJ_!{h9wgH^_zbEHPLon%4AMb~s_wDJPZ8wDg`M5YdxxnEC?O_~pH_ zFFgg#D$u^7d*`#_N#)`~yGOde$N2I+96km-z`UiT4ZosB$6#=man|O%pRwLEIIFVe zd10%FV{%l-BPW$g6$I9qI5GFo#2x5rAF>oFDqcA%vvDgFn5YPE#sbl^ zVeQ*q0h`Z8S8ZFd;HWbB?#{``MN0{?KBj-7zgu*HKpYC9wTI7K6Mxyq!y|_~qC?FB zX<~wuULkz-FE0A!kw>x%vaKrQOPyj-eYKEj<|qhXJ1&Mgyx8t`@hPq1jNj~DLP9@Z zb9TXCHoxd`pvr__cJ>H~J+rq;xpw3Tt8&mMMIK>drA7(F?_cC7m~`@U zq)?U*XT`!-ElBnjMtn?1L&<44w2_kJ7y%KkINRfT<=L1Q%ufUw9yw$wNL1#GC0gR? zos>zv6~xq!2V(K_cds{>BR_RuV(N=CkDziS;U#U|$3>F}3+o|aHmSl`Vi4OhH(B>o z;K1(-!wr86oZHsYBd)-r;9Ps9h047j=-~+N{AE-OwCr}47IuTZ6V7#ANyJ}LE|%Dd zQE#n7`oX}R;!u5JNC9*s?yi2!Ttn+oBndXBXlxUcikYd5vH-fO4kF;{|-_W!FYtcBW3A8C6Se7iA%01utBC!xjggk};5LxMDVoY)}l07d_Bg9QyH zsF**TR&y7A=-W0p5b{r5{~1O&%U1cZA$Ny)ROYbJZdk1#>6iZ|=U1oy7dg@RA^+Y> zE14=luYR7+#Y=|$r^76ip;hpKZkD2=JC83bb?x5*Hi$n}^{;^A&pCmjqDz(IX4iP~ zf1Qs%qoO~j0k~GjcgM^V{GWachI<57=N|*@J#Z-2fL6l4v>w-#J^u1^SmtXHh$#L+ zLtD0#{G-zEYCDsY9nG+R(a>GkB8z-o%b?A+OYMGt{p%wh^_d^$kAJ7wokD`` zghIKYJx$n+H3?Y(W*~-cP7FTg45Cbw*Flt?x()aoCZFAXnpE@86930&rH^ zuG#)?-P`a>PVvj^VT zNl8iIT_boXv8yfD|B@1#9CSV!SYQV^=(y6UFcqP(rd=!}^O+97X-}GL_?{!eTbhCJ z=mdTuMb+@F`N1tvRQ8_hx)!!@(WX(nJ8kVbtwS{ta&u`rXZu$VU$}9P;%HOnb1LWl z0rNrgss~fb(7K@irlt@_DP8!mf6Zny-F$!>tDluQ?20unxttX2YnF3n{xUOJZOH2R zS||)ZC)B?Y=h183DE3TSd0xx@(l)TdHP9EXe$m{>5AIX}(%ekx9LgDDzcE`16~7p^ z2Z2UnO63~?j%4T#_5>SOGtX~F7k5cBoM&$&;-~i1gq)RyB7G91jIo^z&kvPaKgdau zyp!fPyt3`xskyGffmeH(blE)-ipReeY&f<8M(()1rXk41i;|Q%l4SKRR5tpo=bebh zffiFTOJl=>=?KxdxMSq>Dj+@`--Nltcog_B=^I8~qxBrHLrX3v*O(QR=xhNbQQoNoJfw77zBph%Eu^gCQF&*LPK)M zwKe7rgsUzu6{+YQDc|snfWEn z0pCmJvy2bpeGhv>`xC&2cON(BJa#4|{9{4q)>STug*7r@GrGUSz|r|{7$eG^tpgE!<28jk9Jc#6>}pHU~=#(ym6RBqRu zg1rii6!Fssa_4u@4*ch4?Y%lFUkU0%UIT43BQ%~VpsxO>f|7#XjqGN`9-v+Lx)E>I z%D;ItqZ>^pe_`CaE_z@af`r@g-*kq2}Js@o>|t3GM>R<~tI-tG^dtZ^MkWv<$5dh0JXR;>f= zq0S>eO6`}Ob)L+=csC>6GW5XyO3GGAnYb^~t>R)y2H4Q`-7b0w`U_ykyy2$>6s*-m zRgjC}Z#=L){7au+Ug{;zNb~Vq9!{^e)-CYT!_FUTl3GXsAaRJj-iJ?^Z;ukR!Y|~= zNmPU2;WQI{N?MMUl?}?Jmv?3ufE>oVHbr0?kn)U|;D7$>&eV)jXQjiyop`4v*geSK z?R2sru=cPh`T+zhU=ZHvM-XeDcA5|uM{heudKZc&dg&S~oe3;Cq@;^H>(v3bF&`pE zGp|bKCWegDTpoCIAPC_`g@r81fF$n&g7cPv=%Cd|$YBD@?L6=)-823HXR{mz*rokS(b~$mMhX?@%SpY zM%GCWS_gTue~x-RZyQF({6AUj8#>_K;F zLiQf4kS6{4!$)QJQ#wR+A$qF1ZorGgIhP~zeghsg7Su{@40N!1X$_nE6^RNar}f`! zU8LHFhaX?d5oewr<_~(v&`ZlF#E#P5?4Ia2T;%KEu3IAYvs-=0x=rcl$k!?xqUZBt zIIr_>24*uz({3+$XW6t6(Wkac^)o%4mwNCR$1^zb*z+iBQTuGsszT@O??$v~*x>3C z@yoZ#&5NV^Xo#4Q`a0zpt?BMy`Til5z;L~^Pzx!_h=tQU>l_(ue&|=~?4^JgzIx7K z(5i1At4nt_%y#X!r%e`fH3w?D6!Ti?wSEuk{05ad9eT~Vt*UOXkGfRZKFwDlw&G+K z9;PaukG|^Jvl6Y$q+xb`*Iz)Lf;lq8WZqP`4j% z-iR;=7IgDUG%gB!!xktE8k07k``R`_!Nhj$>pgZOr?c(1oH)rlanb-AIl#!Vs>k%r zRvMu1lsL~8TJ$zcV+NuKs&}d{xb<1!&+;2u)^tEUTLSQ%Op~>POjaj)dnTU^~ zcW)+@W@M0AN)rYM{HMYa?-9_U;EHj#2=X7u>3~Q%;mvqcpO>2zt3HQe#pI5xjZ^L1 zD!fzDN`;@$>bGt7gvP_8jyW`rnD)0v+BD!u1nTB%ER)g3pd3QzHc5mXveC8jiSbU{ z#HdIXyLT@<-j|qttu#jLs5x&IdXDve!2?XVFgwt-FMI!wA)H+Y895!c^uLc^ov0zi z7@YZGXV^d$J~1(NjQ`0;obVyQffH&H9W#_ft@|-?#5}HWyt}&^|1eo5x~p+^Ac&=s zF99NWz*U7RW~{={(}FM7w1X4$#g|W2g|FSI7v}r+4WL+}8+DYx%gB0b{;#`xEeZkY zi7_zq%3AWDFiZkYw@3EbBI6WJaAhb32Os4$1{f1F~Q zW(cke!rEvM&~2bLCR(J3f3!iAs}QQ^`7-o0HaiA~s5EU9H<7GC7>M376N9*xr6w@?mlAW|DCWt+bNCtL{>XPMw#e8cVARFsIO z5QiI%>htjC&(t729u&uH%Uv!V?Gu6Q{KMlUrC` z(w1%MaQfZU@*vAmn&M;l_(#VDQ50=T*eq;WhwgtW7)YCF#HACc7-lPh=rd5{h5-&2 zt~X9#fmm5Z2)kmSW`Y{EO4|Od(yhgLfo0~7bvwjubumA-9-~^pO_Wh=w1^i}yV2Di z;ps*Di6>^Z)mapxn>ZGTz;yN9TH*j^mf0_m38AmHsv_Q+X?ma*LgEt zmCqBIHA(FBbBHYgIVe55>Q!$Jr5v#i-9Dpa2KS4R!_E7P8Sbufa>!Az(y11gzM!SSKVrI??=Qzgh*+!f?SbAY11}2Ty6igh+_`G$s&Kx;3z0?YD0C4Q)?Z zCmNp5o8qV1S{hgcRVBAw$Gi@Ii*HMj_9Kbt>fGvHwly&GqEbPDDfni;a&ElPKI=b+ zZp99b#q`#`o*FMxR)6#uSGJ|F%Vs9BGIm`B#yloz8&ZW-|0*|XR4~FfCWv3`&bdxY z5%q{M2|>I)>O}3kc&;k8PET^p8s$i9OWAg+_sN(m1^#aT#u-9{wc{h^`V#`H8PUWG z*z;Oo9Z45PV{k$8qmj}{RxdKK9^>p-ahG*cpzqGzOqb?#I`RR@Nq!!LFiE<+vDI%t zq`gF$mW!C?-Wr$|xBK7SFUtWO_)O1kg8~C62*Og2wx5-c?BRp8WFWJ%Y8cwaRD{JJ z7>O28Xj~TOaRRv<$Tlpq9%~+z>3?N1koV@xtmac}fv@m_5nd3N>;7?i@keV=z{}{M zFcM1)f$9PusdlOUE$ZAC+f=2s93#dYpKYF=v6uSa2EU#F{jU!HZZ?Q5+~%gHObvf< zR#;%TqbBlNMty|&1IbebSXv(a6$s-U3Tz!=txGIOtfyF^YD@!IxvnfGwM zM3^Zx3!S6p3jt;Fc46SJo`evitRFS2EC?@i$J`uCex%|LLzuIJ3$X7@X1s^DhC++f zH->kPLNx%DFc<>bayz8V)D$i;A?Md2iq?gH09EdeRc@Bsc`t}zTm%mvQYN~QY`p-) z?d(y}(7NY#pE5)o{04}FM~?!dk`DNq(HTCc5CnFs z^@m-`Z^5Y_I1f`iyJ2Uhity3uJ#mp^O|pzGLlRa>YW-Fy7ohr48B^}4KyEu4C5bZ- zk}q7p{AR7;>YybZ^yuoqG}=^!3y0oGu!Ty>k@d>v)Z_qXaQ=#zY9bz9eJs~K>DonR zuq4bqm~*M5dg(kejP1QeKGb@*-kv6xZ z6!F6c$(Y?prb=7$i?dQQG}?9*e9vTndYexa?u)+UjRU(N z^itDHXNv!GnkG%#%?2Uu`J_UxK}t`L8q?oyYBADzQ|3X~rQ+nq{rAoik^7H_bKZZE zd@O0KyfysOgen&|$N0$THpLOMjL__bO<{p!5 zcPAiKLPFFmm}{0^L2qZ(qj1}; zF+|F5KDvZA9TJdeA8@Wf${($)@mQ8Hmf|C4sRH{(#&egG8Y8f*D*{(Xq@nb+R zbZQMr1ggxc`5G4;u?|~md9F_rZ3Ge#a-UE~q5v1ZB#}C1!m$1Psf?)c<)K!Z&O)vKoeo7i zOZt49K2ZPDhDGS8JFc$kUX**xFn3ptpLpQdHulbQ=Y*~AxOBAWSU&na$2=8hnQ-kx zPtU0Y%mA$|h?4Ae#}A{qWf6qG^v&FBFEy?k;Q%*SLb${VEic?;R1;Yv887zQMDxMb zzS#i|Ghmnp==Uurq!$s0(f&;$)TqsO4K369VWcR*-Y~>4xo^7nXeHb-4d2nr(QD!Z zMQVPt(8N#7;!Xd(&_u+ z)qNCU58E&2@w}M7wWI8~x^}p)>*M)VXdQ;B%3J|;E{EPfW`kzGlysnV2LSn#a|v#> z&~VQ04Q2kU*BIW3UA*c>95U4PF@JH0^|SMc&OF=4Cu_e+$g0rocC=|`zOqm((9KP{ zEhPK!bO2qU&Naxa{jsiW{&D>^#Sshdb&oUbhAXX*9*tKpk=Po7sc<5QR8LR|T^rKq zr#Fxh2KDmlGpH4`9W!Z5)%L%-l(m1^>7Q z?$iJ{k+0$ausXr1c6PyH5L|3z)a%v<#?26$!7w2$jvlUZYsR5i*(XKdD4aY$OQhuv zpfxXejAs)n0-C}W7KUraF*7OQoPKtBu=o|*{`|egnO3j1aI@?K16wSG(8`{$7jpet-r1*PP%uzJAj;~CmDZMHAlw_WjLzqE>jmroGrKG?)Zt!`O zVCI%Lul#c)seZrCiDvzESs$hJ%b6Ni2?@OUX#?w8OFdEy`-H2YWeAOr)Uf7~Sp&16BA}*)HJn~408|qR4ElXbZ)ch)>4knr=YI9TZ0+<}FWy5V*}3y^ zL6`c19;R&4lXUas&%dF5jA7$nIwu4e`b@Dmm3W(w6rxiJk*!Yx>qI^)0&hq_l0&$_ zN2335ecU2QJLp?^@O>I(jGCn*%)D3)zf{fyx6PET{c>? zwtGhvN&pg`C|{zPRjC7!RYe~v6z1S0o5>3g!{c4?zJnDLG(-R2UjfaHi&kU=IcCA) zf&^z~1r(C)>RKEnr{{hg&bu$cLfIth`_5HISVL6c~*hVkrff0p)zv`aX1R= zmWJ+PF1{(Beu7r6xsTCTO6)#9Ta;mIKubvN>Ku_o1Ylki75gN`laAshgfu*A_Mw>8 zfu`lAQb(_9tkje)6W_kPQrHT^wzGq#zCBGj>zZRsyPVj#l@V~tUBb^nM2?lUwF}Bq zuEL3x^8zqaIQ|OUESv5MDBj;^He{3BEjdI+N>UA!l*2t~|GLc!mv0meLrQ9=#}~%E z?nMUYbq`qAY?W-Igxh>Q5XMuBIGOraW-BtDToh=UA>Xp(y^5cdcl~yms+o z_N?v?6HPOMAPHAqA%LTJS+JuAU}Su5!`dA8UO$W<*0;^Y!$Ub6CWE zNmKGb2kY2%)|2234g5#~o?lsLp~4pnf`s#(Jmo6)N@tBq$E1fM_Ec(LOH->qxM%r?#r?>DMsU7gT?@HGpi#zB=L(iT;8hsknNC6(qcF zul~G+Gt$y)7vx~BQCXUdcn%Ic~-k1y`^6CVS(cl z-$#sE6s@^6SG7k{X4IG|gRyL&UaWij>)S9VZY2RQR^CW2H0!f=FJVF@lD5!tlzavE zapiy@>V`w3==yBYgigi;xo_2?-p#6Un!plnALvGURyXLV63SX+802M#mQFvk5w))p z%-iRcPm85NLJl2wgpss)}CgL9TSKyE$GrAv(Uo?zD>BX_w+~PwK-9S?8FqUP&bR1{C}us)jcCH8awkB8z>710S$HHx{Ati#!&hUlrmPY%1ZYAEOcEZj)(V z(NUAi;71c<&wZ4m_$(*q!gS=|B%>$H-duKWBo-IZ;nVi8?+{iH<&xQR8BA-R(HR!S zd7FR5A%oP9Ml>wuln0NmUfseXkhBS^&#tslLU`USbHx^YmsC@$+iY>yhbpwsYevK*S$N0Q{Oz+q@nl-)0ySERDb3B0#weAoEqFT58hqdA2tRYJrOTR< zov=7_P)PUG6fSV12SU|GW~Bhcp$$3iP%`f63A6HaCY%J}TzS$)@0ZmBeWLrVa8CBb zY)eTV^P77Fc0e4V+mO5Kw~vK~?=Ca=;S+HYCz95HGowdk=NmaDi@}kJwxjwD4lnOBBLV{R@1;5C-^yd6+t8L0RbH!kdGIGfRBu?Fe^JWJ>@ zHh5R&^-EBO6R7(Y@?Do+Pb=!a2~Ow#1cv`zx}`H)rP7}Bl}A@B%eD-98E(qVGnt5` zihm%{GqGL8kEWg;i_eT zj8*e2YxU1k_^P@k7~N2#(bl+>Qw7HL72E#LS6%okxTa?CN(DfaTAcNq>lqm6vFJZ` zEW-o$J~6!p?swNe5h83-M8`XWmc#q;8!?^6o$n)mw_3Y=N#_F~`ErzQXB#YC*m<*C z5lYv&_^VIHx^vn_3d4fQu!Nk>!Bnf#`lz5 z-FFt}Rxe%LdG_6Zy61D)z`EGOf_j(zGq{*j4sx0P{i09y5r;dW?=~#mG&-OMC6|r$ zu3MJ90^9%~$*Ml^h&oLODbc0g1^0i>ulrB^SY>2yVBfJ_x`lM-v^zsO*T;6j*S1*&+ol6D_pKCU4ia9#Z@eE z%-Pm|pTM!_d0mH?QJ1a3#VSve|JYCY*f;o9>MNG&`Mr&8Sj3tO4;PX8K39N?WKuux MJG!@IkLQ*D0tJl&Q2+n{ literal 0 HcmV?d00001 diff --git a/docs/README.md b/docs/README.md index 62d152ca1e0..9593e9f26e9 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,4 +1,4 @@ -# Overview +# What is Feast? Feast \(**Fea**ture **St**ore\) is a tool for managing and serving machine learning features. diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 3cab9a11927..bdb15821685 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -1,23 +1,32 @@ # Table of contents -* [Overview](README.md) -* [Why Feast?](why-feast.md) -* [Concepts](concepts.md) -* [Getting Help](getting-help.md) -* [Contributing](contributing.md) +* [What is Feast?](README.md) + +## Introduction + +* [Why Feast?](introduction/why-feast.md) +* [Getting Help](introduction/getting-help.md) +* [Changelog](https://github.com/gojek/feast/blob/master/CHANGELOG.md) * [Roadmap](roadmap.md) -## Installing Feast +## Concepts + +* [Concepts](concepts/concepts.md) + +## Installation -* [Overview](installing-feast/overview.md) -* [Docker Compose](installing-feast/docker-compose.md) -* [Google Kubernetes Engine \(GKE\)](installing-feast/gke.md) -* [Troubleshooting](installing-feast/troubleshooting.md) +* [Overview](installation/overview.md) +* [Docker Compose](installation/docker-compose.md) +* [Google Kubernetes Engine \(GKE\)](installation/gke.md) +* [Troubleshooting](installation/troubleshooting.md) -## Using Feast +## Tutorials +* [Basic](https://github.com/gojek/feast/blob/master/examples/basic/basic.ipynb) * [Using Feast](https://github.com/gojek/feast/blob/master/examples/basic/basic.ipynb) +## Administration + ## Reference * [Python SDK](https://api.docs.feast.dev/python/) @@ -26,3 +35,10 @@ * [Core gRPC API](https://api.docs.feast.dev/grpc/feast.core.pb.html) * [Serving gRPC API](https://api.docs.feast.dev/grpc/feast.serving.pb.html) +## Contributing + +* [Contribution Process](contributing/contributing.md) +* [Development Guide](contributing/development-guide.md) +* [Style Guide](contributing/style-guide.md) +* [Release Process](contributing/release-process.md) + diff --git a/docs/concepts/concepts.md b/docs/concepts/concepts.md new file mode 100644 index 00000000000..e54bd488d6a --- /dev/null +++ b/docs/concepts/concepts.md @@ -0,0 +1,124 @@ +# Concepts + +## Architecture + +![Logical diagram of a typical Feast deployment](../.gitbook/assets/basic-architecture-diagram%20%282%29.svg) + +The core components of a Feast deployment are + +* **Feast Core:** Feast Core is a centralized service that acts as the authority on features within an organization. Typically there is only one "Core" deployment per organization, with all feature management happening through it. +* **Feast Ingestion Jobs:** Feast ingestion jobs retrieve feature data from user defined data sources and populate serving stores with this feature data. These jobs are managed by Feast Core. Data can either be sources from existing sources \(like [Kafka](https://kafka.apache.org/)\), or it can be loaded into Feast through its API. +* **Feast Serving:** Feast Serving is the data access layer through which end users and production systems retrieve feature data. Each Serving store is backed by one or more databases. These databases are updated by the Feast ingestion jobs. There are two types of stores: batch and online. Batch stores hold large volumes historical data, while online stores only hold the latest feature values. + +## Data Model + +### Feature Set + +User data is typically in the form of dataframes, tables in data warehouses, or events on a stream. These data sources are loaded into Feast in order to serve features for model training or serving. + +Feature sets allow for groups of fields in these data sources to be ingested and stored together. This allows for efficient storage and logical namespacing of data. + +When data is loaded from these sources, each field in the feature set must be found in every record of the data source. Fields from these data sources must be either a timestamp, an entity, or a feature. + +{% hint style="info" %} +Feature sets are a grouping of feature sets based on how they are loaded into Feast. They ensure that data is efficiently stored during ingestion. Feature sets are not a grouping of features for retrieval of features. During retrieval it is possible to retrieve feature values from any number of feature sets. +{% endhint %} + +#### Customer Transactions Example + +Below is an example of a basic `customer transactions` feature set that has been exported to YAML: + +{% tabs %} +{% tab title="customer\_transactions\_feature\_set.yaml" %} +```yaml +name: customer_transactions +kind: feature_set +entities: +- name: customer_id + valueType: INT64 +features: +- name: daily_transactions + valueType: FLOAT +- name: total_transactions + valueType: FLOAT + maxAge: 3600s +``` +{% endtab %} +{% endtabs %} + +The dataframe below \(`customer_data.csv`\) contains the features and entities of the above feature set + +| datetime | customer\_id | daily\_transactions | total\_tra**nsactions** | +| :--- | :--- | :--- | :--- | +| 2019-01-01 01:00:00 | 20001 | 5.0 | 14.0 | +| 2019-01-01 01:00:00 | 20002 | 2.6 | 43.0 | +| 2019-01-01 01:00:00 | 20003 | 4.1 | 154.0 | +| 2019-01-01 01:00:00 | 20004 | 3.4 | 74.0 | + +In order to ingest feature data into Feast for this specific feature set: + +```python +# Load dataframe +customer_df = pd.read_csv("customer_data.csv") + +# Create feature set from YAML (using YAML is optional) +cust_trans_fs = FeatureSet.from_yaml("customer_transactions_feature_set.yaml") + +# Load feature data into Feast for this specific feature set +client.ingest(cust_trans_fs, customer_data) +``` + +### Feature + +A feature is an individual measurable property or characteristic of a phenomenon being observed. Features are the most important concepts within a feature store. Feature data is used both as input to models during training and when models are served in production. + +In the context of Feast, features are values that are associated with either one or more entities over time. In Feast, these values are either primitives or lists of primitives. Each feature can also have additional information attached to it. For example whether it is a categorical feature or numerical. + +{% hint style="info" %} +Features in Feast are defined within Feature Sets and are not treated as standalone concepts. +{% endhint %} + +### Entity + +An entity type is any object in an organization that needs to be modeled and on which information should be stored. Entity types are usually recognizable concepts, either concrete or abstract, such as persons, places, things, or events which have relevance to the modeled system. + +An entity is an instance of an entity type. + +* Examples of entity types in the context of ride-hailing and food delivery: `customer`, `order`, `driver`, `restaurant`, `dish`, `area`. +* A specific driver, for example a driver with ID `D011234` would be an entity of the entity type `driver` + +An entity is the object on which features are observed. For example we could have a feature `total_trips_24h` on the driver `D01123` with a feature value of `11`. + +In the context of Feast, entities are important because they are used as keys when looking up feature values. Entities are also used when joining feature values between different feature sets in order to build one large data set to train a model, or to serve a model. + +{% hint style="info" %} +Entities in Feast are defined within Feature Sets and are not treated as standalone concepts. +{% endhint %} + +### Types + +Feast supports the following types for feature values + +* BYTES +* STRING +* INT32 +* INT64 +* DOUBLE +* FLOAT +* BOOL +* BYTES\_LIST +* STRING\_LIST +* INT32\_LIST +* INT64\_LIST +* DOUBLE\_LIST +* FLOAT\_LIST +* BOOL\_LIST + +## Glossary + +| Term | Description | +| :--- | :--- | +| Feast deployment | A complete Feast system as it is deployed. Consists out of a single Feast Core deployment and one or more Feast Serving deployments. | +| Feast Core | The centralized service which acts as a registry and authority of features. Organizations should only deploy a single Feast Core instance. Feast Core also manages the ingestion of feature data and population of Feast Serving data stores. | +| Feast Serving | Feast Serving is a service used to access both online and batch feature data. Feast Serving deployments are backed by one or more databases. | + diff --git a/docs/contributing/contributing.md b/docs/contributing/contributing.md new file mode 100644 index 00000000000..0a32ae284a0 --- /dev/null +++ b/docs/contributing/contributing.md @@ -0,0 +1,542 @@ +# Contribution Process + +## 1. Contribution process + +We use [RFCs](https://en.wikipedia.org/wiki/Request_for_Comments) and [GitHub issues](https://github.com/gojek/feast/issues) to communicate development ideas. The simplest way to contribute to Feast is to leave comments in our [RFCs](https://drive.google.com/drive/u/0/folders/1Lj1nIeRB868oZvKTPLYqAvKQ4O0BksjY) in the [Feast Google Drive](https://drive.google.com/drive/u/0/folders/0AAe8j7ZK3sxSUk9PVA) or our GitHub issues. + +Please communicate your ideas through a GitHub issue or through our Slack Channel before starting development. + +Please [submit a PR ](https://github.com/gojek/feast/pulls)to the master branch of the Feast repository once you are ready to submit your contribution. Code submission to Feast \(including submission from project maintainers\) require review and approval from maintainers or code owners. + +PRs that are submitted by the general public need to be identified as `ok-to-test`. Once enabled, [Prow](https://github.com/kubernetes/test-infra/tree/master/prow) will run a range of tests to verify the submission, after which community members will help to review the pull request. + +{% hint style="success" %} +Please sign the [Google CLA](https://cla.developers.google.com/) in order to have your code merged into the Feast repository. +{% endhint %} + +## 2. Development guide + +### 2.1 Overview + +The following guide will help you quickly run Feast in your local machine. + +The main components of Feast are: + +* **Feast Core:** Handles feature registration, starts and manages ingestion jobs and ensures that Feast internal metadata is consistent. +* **Feast Ingestion Jobs:** Subscribes to streams of FeatureRows and writes these as feature + + values to registered databases \(online, historical\) that can be read by Feast Serving. + +* **Feast Serving:** Service that handles requests for features values, either online or batch. + +### 2.**2 Requirements** + +#### 2.**2.1 Development environment** + +The following software is required for Feast development + +* Java SE Development Kit 11 +* Python version 3.6 \(or above\) and pip +* [Maven ](https://maven.apache.org/install.html)version 3.6.x + +Additionally, [grpc\_cli](https://github.com/grpc/grpc/blob/master/doc/command_line_tool.md) is useful for debugging and quick testing of gRPC endpoints. + +#### 2.**2.2 Services** + +The following components/services are required to develop Feast: + +* **Feast Core:** Requires PostgreSQL \(version 11 and above\) to store state, and requires a Kafka \(tested on version 2.x\) setup to allow for ingestion of FeatureRows. +* **Feast Serving:** Requires Redis \(tested on version 5.x\). + +These services should be running before starting development. The following snippet will start the services using Docker. + +```bash +# Start Postgres +docker run --name postgres --rm -it -d --net host -e POSTGRES_DB=postgres -e POSTGRES_USER=postgres \ +-e POSTGRES_PASSWORD=password postgres:12-alpine + +# Start Redis +docker run --name redis --rm -it --net host -d redis:5-alpine + +# Start Zookeeper (needed by Kafka) +docker run --rm \ + --net=host \ + --name=zookeeper \ + --env=ZOOKEEPER_CLIENT_PORT=2181 \ + --detach confluentinc/cp-zookeeper:5.2.1 + +# Start Kafka +docker run --rm \ + --net=host \ + --name=kafka \ + --env=KAFKA_ZOOKEEPER_CONNECT=localhost:2181 \ + --env=KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \ + --env=KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ + --detach confluentinc/cp-kafka:5.2.1 +``` + +### 2.3 Testing and development + +#### 2.3.1 Running unit tests + +```text +$ mvn test +``` + +#### 2.3.2 Running integration tests + +_Note: integration suite isn't yet separated from unit._ + +```text +$ mvn verify +``` + +#### 2.3.3 Running components locally + +The `core` and `serving` modules are Spring Boot applications. These may be run as usual for [the Spring Boot Maven plugin](https://docs.spring.io/spring-boot/docs/current/maven-plugin/index.html): + +```text +$ mvn --projects core spring-boot:run + +# Or for short: +$ mvn -pl core spring-boot:run +``` + +Note that you should execute `mvn` from the Feast repository root directory, as there are intermodule dependencies that Maven will not resolve if you `cd` to subdirectories to run. + +#### 2.3.4 Running from IntelliJ + +Compiling and running tests in IntelliJ should work as usual. + +Running the Spring Boot apps may work out of the box in IDEA Ultimate, which has built-in support for Spring Boot projects, but the Community Edition needs a bit of help: + +The Spring Boot Maven plugin automatically puts dependencies with `provided` scope on the runtime classpath when using `spring-boot:run`, such as its embedded Tomcat server. The "Play" buttons in the gutter or right-click menu of a `main()` method [do not do this](https://stackoverflow.com/questions/30237768/run-spring-boots-main-using-ide). + +A solution to this is: + +1. Open `View > Tool Windows > Maven` +2. Drill down to e.g. `Feast Core > Plugins > spring-boot:run`, right-click and `Create 'feast-core [spring-boot'…` +3. In the dialog that pops up, check the `Resolve Workspace artifacts` box +4. Recommended: add `-Dspring-boot.run.fork=false` to the `Command line` field to get Debug working too +5. Click `OK`. You should now be able to select this run configuration for the Play button in the main toolbar, keyboard shortcuts, etc. + +It is recommend to have IntelliJ delegate building to Maven, if this is not enabled out of the box when you import the project, for greater assurance that build behavior is consistent with CI / production builds. This is set in Preferences at `Build, Execution, Deployment > Build Tools > Maven > Runner > Delegate IDE build/run actions to Maven`. + +### 2.**4** Validating your setup + +The following section is a quick walk-through to test whether your local Feast deployment is functional for development purposes. + +**2.4.1 Assumptions** + +* PostgreSQL is running in `localhost:5432` and has a database called `postgres` which + + can be accessed with credentials user `postgres` and password `password`. Different database configurations can be supplied here \(`/core/src/main/resources/application.yml`\) + +* Redis is running locally and accessible from `localhost:6379` +* \(optional\) The local environment has been authentication with Google Cloud Platform and has full access to BigQuery. This is only necessary for BigQuery testing/development. + +#### 2.4.2 Clone Feast + +```bash +git clone https://github.com/gojek/feast.git && cd feast && \ +export FEAST_HOME_DIR=$(pwd) +``` + +#### 2.4.3 Starting Feast Core + +To run Feast Core locally using Maven: + +```bash +# Feast Core can be configured from the following .yml file +# $FEAST_HOME_DIR/core/src/main/resources/application.yml +mvn --projects core spring-boot:run +``` + +Test whether Feast Core is running + +```text +grpc_cli call localhost:6565 ListStores '' +``` + +The output should list **no** stores since no Feast Serving has registered its stores to Feast Core: + +```text +connecting to localhost:6565 + +Rpc succeeded with OK status +``` + +#### 2.4.4 Starting Feast Serving + +Feast Serving is configured through the `$FEAST_HOME_DIR/serving/src/main/resources/application.yml`. Each Serving deployment must be configured with a store. The default store is Redis \(used for online serving\). + +The configuration for this default store is located in a separate `.yml` file. The default location is `$FEAST_HOME_DIR/serving/sample_redis_config.yml`: + +```text +name: serving +type: REDIS +redis_config: + host: localhost + port: 6379 +subscriptions: + - name: "*" + project: "*" + version: "*" +``` + +Once Feast Serving is started, it will register its store with Feast Core \(by name\) and start to subscribe to a feature sets based on its subscription. + +Start Feast Serving GRPC server on localhost:6566 with store name `serving` + +```text +mvn --projects serving spring-boot:run +``` + +Test connectivity to Feast Serving + +```text +grpc_cli call localhost:6566 GetFeastServingInfo '' +``` + +```text +connecting to localhost:6566 +version: "0.4.2-SNAPSHOT" +type: FEAST_SERVING_TYPE_ONLINE + +Rpc succeeded with OK status +``` + +Test Feast Core to see whether it is aware of the Feast Serving deployment + +```text +grpc_cli call localhost:6565 ListStores '' +``` + +```text +connecting to localhost:6565 +store { + name: "serving" + type: REDIS + subscriptions { + name: "*" + version: "*" + project: "*" + } + redis_config { + host: "localhost" + port: 6379 + } +} + +Rpc succeeded with OK status +``` + +In order to use BigQuery as a historical store, it is necessary to start Feast Serving with a different store type. + +Copy `$FEAST_HOME_DIR/serving/sample_redis_config.yml` to the following location `$FEAST_HOME_DIR/serving/my_bigquery_config.yml` and update the configuration as below: + +```text +name: bigquery +type: BIGQUERY +bigquery_config: + project_id: YOUR_GCP_PROJECT_ID + dataset_id: YOUR_GCP_DATASET +subscriptions: + - name: "*" + version: "*" + project: "*" +``` + +Then inside `serving/src/main/resources/application.yml` modify the following key `feast.store.config-path` to point to the new store configuration. + +After making these changes, restart Feast Serving: + +```text +mvn --projects serving spring-boot:run +``` + +You should see two stores registered: + +```text +store { + name: "serving" + type: REDIS + subscriptions { + name: "*" + version: "*" + project: "*" + } + redis_config { + host: "localhost" + port: 6379 + } +} +store { + name: "bigquery" + type: BIGQUERY + subscriptions { + name: "*" + version: "*" + project: "*" + } + bigquery_config { + project_id: "my_project" + dataset_id: "my_bq_dataset" + } +} +``` + +#### 2.4.5 Registering a FeatureSet + +Before registering a new FeatureSet, a project is required. + +```text +grpc_cli call localhost:6565 CreateProject ' + name: "your_project_name" +' +``` + +When a feature set is successfully registered, Feast Core will start an **ingestion** job that listens for new features in the feature set. + +{% hint style="info" %} +Note that Feast currently only supports source of type `KAFKA`, so you must have access to a running Kafka broker to register a FeatureSet successfully. It is possible to omit the `source` from a Feature Set, but Feast Core will still use Kafka behind the scenes, it is simply abstracted away from the user. +{% endhint %} + +Create a new FeatureSet in Feast by sending a request to Feast Core: + +```text +# Example of registering a new driver feature set +# Note the source value, it assumes that you have access to a Kafka broker +# running on localhost:9092 + +grpc_cli call localhost:6565 ApplyFeatureSet ' +feature_set { + spec { + project: "your_project_name" + name: "driver" + version: 1 + + entities { + name: "driver_id" + value_type: INT64 + } + + features { + name: "city" + value_type: STRING + } + + source { + type: KAFKA + kafka_source_config { + bootstrap_servers: "localhost:9092" + topic: "your-kafka-topic" + } + } + } +} +' +``` + +Verify that the FeatureSet has been registered correctly. + +```text +# To check that the FeatureSet has been registered correctly. +# You should also see logs from Feast Core of the ingestion job being started +grpc_cli call localhost:6565 GetFeatureSet ' + project: "your_project_name" + name: "driver" +' +``` + +Or alternatively, list all feature sets + +```text +grpc_cli call localhost:6565 ListFeatureSets ' + filter { + project: "your_project_name" + feature_set_name: "driver" + feature_set_version: "1" + } +' +``` + +#### 2.4.6 Ingestion and Population of Feature Values + +```text +# Produce FeatureRow messages to Kafka so it will be ingested by Feast +# and written to the registered stores. +# Make sure the value here is the topic assigned to the feature set +# ... producer.send("feast-driver-features" ...) +# +# Install Python SDK to help writing FeatureRow messages to Kafka +cd $FEAST_HOMEDIR/sdk/python +pip3 install -e . +pip3 install pendulum + +# Produce FeatureRow messages to Kafka so it will be ingested by Feast +# and written to the corresponding store. +# Make sure the value here is the topic assigned to the feature set +# ... producer.send("feast-test_feature_set-features" ...) +python3 - < Tool Windows > Maven` +2. Drill down to e.g. `Feast Core > Plugins > spring-boot:run`, right-click and `Create 'feast-core [spring-boot'…` +3. In the dialog that pops up, check the `Resolve Workspace artifacts` box +4. Click `OK`. You should now be able to select this run configuration for the Play button in the main toolbar, keyboard shortcuts, etc. + +### **4** Validating your setup + +The following section is a quick walk-through to test whether your local Feast deployment is functional for development purposes. + +**4.1 Assumptions** + +* PostgreSQL is running in `localhost:5432` and has a database called `postgres` which + + can be accessed with credentials user `postgres` and password `password`. Different database configurations can be supplied here \(`/core/src/main/resources/application.yml`\) + +* Redis is running locally and accessible from `localhost:6379` +* \(optional\) The local environment has been authentication with Google Cloud Platform and has full access to BigQuery. This is only necessary for BigQuery testing/development. + +#### 4.2 Clone Feast + +```bash +git clone https://github.com/gojek/feast.git && cd feast && \ +export FEAST_HOME_DIR=$(pwd) +``` + +#### 4.3 Starting Feast Core + +To run Feast Core locally using Maven: + +```bash +# Feast Core can be configured from the following .yml file +# $FEAST_HOME_DIR/core/src/main/resources/application.yml +mvn --projects core spring-boot:run +``` + +Test whether Feast Core is running + +```text +grpc_cli call localhost:6565 ListStores '' +``` + +The output should list **no** stores since no Feast Serving has registered its stores to Feast Core: + +```text +connecting to localhost:6565 + +Rpc succeeded with OK status +``` + +#### 4.4 Starting Feast Serving + +Feast Serving is configured through the `$FEAST_HOME_DIR/serving/src/main/resources/application.yml`. Each Serving deployment must be configured with a store. The default store is Redis \(used for online serving\). + +The configuration for this default store is located in a separate `.yml` file. The default location is `$FEAST_HOME_DIR/serving/sample_redis_config.yml`: + +```text +name: serving +type: REDIS +redis_config: + host: localhost + port: 6379 +subscriptions: + - name: "*" + project: "*" + version: "*" +``` + +Once Feast Serving is started, it will register its store with Feast Core \(by name\) and start to subscribe to a feature sets based on its subscription. + +Start Feast Serving GRPC server on localhost:6566 with store name `serving` + +```text +mvn --projects serving spring-boot:run +``` + +Test connectivity to Feast Serving + +```text +grpc_cli call localhost:6566 GetFeastServingInfo '' +``` + +```text +connecting to localhost:6566 +version: "0.4.2-SNAPSHOT" +type: FEAST_SERVING_TYPE_ONLINE + +Rpc succeeded with OK status +``` + +Test Feast Core to see whether it is aware of the Feast Serving deployment + +```text +grpc_cli call localhost:6565 ListStores '' +``` + +```text +connecting to localhost:6565 +store { + name: "serving" + type: REDIS + subscriptions { + name: "*" + version: "*" + project: "*" + } + redis_config { + host: "localhost" + port: 6379 + } +} + +Rpc succeeded with OK status +``` + +In order to use BigQuery as a historical store, it is necessary to start Feast Serving with a different store type. + +Copy `$FEAST_HOME_DIR/serving/sample_redis_config.yml` to the following location `$FEAST_HOME_DIR/serving/my_bigquery_config.yml` and update the configuration as below: + +```text +name: bigquery +type: BIGQUERY +bigquery_config: + project_id: YOUR_GCP_PROJECT_ID + dataset_id: YOUR_GCP_DATASET +subscriptions: + - name: "*" + version: "*" + project: "*" +``` + +Then inside `serving/src/main/resources/application.yml` modify the following key `feast.store.config-path` to point to the new store configuration. + +After making these changes, restart Feast Serving: + +```text +mvn --projects serving spring-boot:run +``` + +You should see two stores registered: + +```text +store { + name: "serving" + type: REDIS + subscriptions { + name: "*" + version: "*" + project: "*" + } + redis_config { + host: "localhost" + port: 6379 + } +} +store { + name: "bigquery" + type: BIGQUERY + subscriptions { + name: "*" + version: "*" + project: "*" + } + bigquery_config { + project_id: "my_project" + dataset_id: "my_bq_dataset" + } +} +``` + +#### 4.5 Registering a FeatureSet + +Before registering a new FeatureSet, a project is required. + +```text +grpc_cli call localhost:6565 CreateProject ' + name: "your_project_name" +' +``` + +When a feature set is successfully registered, Feast Core will start an **ingestion** job that listens for new features in the feature set. + +{% hint style="info" %} +Note that Feast currently only supports source of type `KAFKA`, so you must have access to a running Kafka broker to register a FeatureSet successfully. It is possible to omit the `source` from a Feature Set, but Feast Core will still use Kafka behind the scenes, it is simply abstracted away from the user. +{% endhint %} + +Create a new FeatureSet in Feast by sending a request to Feast Core: + +```text +# Example of registering a new driver feature set +# Note the source value, it assumes that you have access to a Kafka broker +# running on localhost:9092 + +grpc_cli call localhost:6565 ApplyFeatureSet ' +feature_set { + spec { + project: "your_project_name" + name: "driver" + version: 1 + + entities { + name: "driver_id" + value_type: INT64 + } + + features { + name: "city" + value_type: STRING + } + + source { + type: KAFKA + kafka_source_config { + bootstrap_servers: "localhost:9092" + topic: "your-kafka-topic" + } + } + } +} +' +``` + +Verify that the FeatureSet has been registered correctly. + +```text +# To check that the FeatureSet has been registered correctly. +# You should also see logs from Feast Core of the ingestion job being started +grpc_cli call localhost:6565 GetFeatureSet ' + project: "your_project_name" + name: "driver" +' +``` + +Or alternatively, list all feature sets + +```text +grpc_cli call localhost:6565 ListFeatureSets ' + filter { + project: "your_project_name" + feature_set_name: "driver" + feature_set_version: "1" + } +' +``` + +#### 4.6 Ingestion and Population of Feature Values + +```text +# Produce FeatureRow messages to Kafka so it will be ingested by Feast +# and written to the registered stores. +# Make sure the value here is the topic assigned to the feature set +# ... producer.send("feast-driver-features" ...) +# +# Install Python SDK to help writing FeatureRow messages to Kafka +cd $FEAST_HOMEDIR/sdk/python +pip3 install -e . +pip3 install pendulum + +# Produce FeatureRow messages to Kafka so it will be ingested by Feast +# and written to the corresponding store. +# Make sure the value here is the topic assigned to the feature set +# ... producer.send("feast-test_feature_set-features" ...) +python3 - < + +This guide will install Feast into a Kubernetes cluster on GCP. It assumes that all of your services will run within a single Kubernetes cluster. Once Feast is installed you will be able to: + +* Define and register features. +* Load feature data from both batch and streaming sources. +* Retrieve features for model training. +* Retrieve features for online serving. + +{% hint style="info" %} +This guide requires [Google Cloud Platform](https://cloud.google.com/) for installation. + +* [BigQuery](https://cloud.google.com/bigquery/) is used for storing historical features. +* [Google Cloud Storage](https://cloud.google.com/storage/) is used for intermediate data storage. +{% endhint %} + +## 0. Requirements + +1. [Google Cloud SDK ](https://cloud.google.com/sdk/install)installed, authenticated, and configured to the project you will use. +2. [Kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) installed. +3. [Helm](https://helm.sh/3) \(2.16.0 or greater\) installed on your local machine with Tiller installed in your cluster. Helm 3 has not been tested yet. + +## 1. Set up GCP + +First define the environmental variables that we will use throughout this installation. Please customize these to reflect your environment. + +```bash +export FEAST_GCP_PROJECT_ID=my-gcp-project +export FEAST_GCP_REGION=us-central1 +export FEAST_GCP_ZONE=us-central1-a +export FEAST_BIGQUERY_DATASET_ID=feast +export FEAST_GCS_BUCKET=${FEAST_GCP_PROJECT_ID}_feast_bucket +export FEAST_GKE_CLUSTER_NAME=feast +export FEAST_SERVICE_ACCOUNT_NAME=feast-sa +``` + +Create a Google Cloud Storage bucket for Feast to stage batch data exports: + +```bash +gsutil mb gs://${FEAST_GCS_BUCKET} +``` + +Create the service account that Feast will run as: + +```bash +gcloud iam service-accounts create ${FEAST_SERVICE_ACCOUNT_NAME} + +gcloud projects add-iam-policy-binding ${FEAST_GCP_PROJECT_ID} \ + --member serviceAccount:${FEAST_SERVICE_ACCOUNT_NAME}@${FEAST_GCP_PROJECT_ID}.iam.gserviceaccount.com \ + --role roles/editor + +gcloud iam service-accounts keys create key.json --iam-account \ +${FEAST_SERVICE_ACCOUNT_NAME}@${FEAST_GCP_PROJECT_ID}.iam.gserviceaccount.com +``` + +## 2. Set up a Kubernetes \(GKE\) cluster + +{% hint style="warning" %} +Provisioning a GKE cluster can expose your services publicly. This guide does not cover securing access to the cluster. +{% endhint %} + +Create a GKE cluster: + +```bash +gcloud container clusters create ${FEAST_GKE_CLUSTER_NAME} \ + --machine-type n1-standard-4 +``` + +Create a secret in the GKE cluster based on your local key `key.json`: + +```bash +kubectl create secret generic feast-gcp-service-account --from-file=key.json +``` + +For this guide we will use `NodePort` for exposing Feast services. In order to do so, we must find an External IP of at least one GKE node. This should be a public IP. + +```bash +export FEAST_IP=$(kubectl describe nodes | grep ExternalIP | awk '{print $2}' | head -n 1) +export FEAST_CORE_URL=${FEAST_IP}:32090 +export FEAST_ONLINE_SERVING_URL=${FEAST_IP}:32091 +export FEAST_BATCH_SERVING_URL=${FEAST_IP}:32092 +``` + +Add firewall rules to open up ports on your Google Cloud Platform project: + +```bash +gcloud compute firewall-rules create feast-core-port --allow tcp:32090 +gcloud compute firewall-rules create feast-online-port --allow tcp:32091 +gcloud compute firewall-rules create feast-batch-port --allow tcp:32092 +gcloud compute firewall-rules create feast-redis-port --allow tcp:32101 +gcloud compute firewall-rules create feast-kafka-ports --allow tcp:31090-31095 +``` + +## 3. Set up Helm + +Run the following command to provide Tiller with authorization to install Feast: + +```bash +kubectl apply -f - <8888/tcp feast_jupyter_1 +8e49dbe81b92 gcr.io/kf-feast/feast-serving:latest "java -Xms1024m -Xmx…" 2 minutes ago Up 5 seconds 0.0.0.0:6567->6567/tcp feast_batch-serving_1 +b859494bd33a gcr.io/kf-feast/feast-serving:latest "java -jar /opt/feas…" 2 minutes ago Up About a minute 0.0.0.0:6566->6566/tcp feast_online-serving_1 +5c4962811767 gcr.io/kf-feast/feast-core:latest "java -jar /opt/feas…" 2 minutes ago Up 2 minutes 0.0.0.0:6565->6565/tcp feast_core_1 +1ba7239e0ae0 confluentinc/cp-kafka:5.2.1 "/etc/confluent/dock…" 2 minutes ago Up 2 minutes 0.0.0.0:9092->9092/tcp, 0.0.0.0:9094->9094/tcp feast_kafka_1 +e2779672735c confluentinc/cp-zookeeper:5.2.1 "/etc/confluent/dock…" 2 minutes ago Up 2 minutes 2181/tcp, 2888/tcp, 3888/tcp feast_zookeeper_1 +39ac26f5c709 postgres:12-alpine "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 5432/tcp feast_db_1 +3c4ee8616096 redis:5-alpine "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 0.0.0.0:6379->6379/tcp feast_redis_1 +``` + +### Google Kubernetes Engine + +All services should either be in a `running` state or `complete`state: + +```text +kubectl get pods +``` + +```text +NAME READY STATUS RESTARTS AGE +feast-feast-core-5ff566f946-4wlbh 1/1 Running 1 32m +feast-feast-serving-batch-848d74587b-96hq6 1/1 Running 2 32m +feast-feast-serving-online-df69755d5-fml8v 1/1 Running 2 32m +feast-kafka-0 1/1 Running 1 32m +feast-kafka-1 1/1 Running 0 30m +feast-kafka-2 1/1 Running 0 29m +feast-kafka-config-3e860262-zkzr8 0/1 Completed 0 32m +feast-postgresql-0 1/1 Running 0 32m +feast-prometheus-statsd-exporter-554db85b8d-r4hb8 1/1 Running 0 32m +feast-redis-master-0 1/1 Running 0 32m +feast-zookeeper-0 1/1 Running 0 32m +feast-zookeeper-1 1/1 Running 0 32m +feast-zookeeper-2 1/1 Running 0 31m +``` + +## How can I verify that I can connect to all services? + +First find the `IP:Port` combination of your services. + +### **Docker Compose \(from inside the docker cluster\)** + +You will probably need to connect using the hostnames of services and standard Feast ports: + +```bash +export FEAST_CORE_URL=core:6565 +export FEAST_ONLINE_SERVING_URL=online-serving:6566 +export FEAST_BATCH_SERVING_URL=batch-serving:6567 +``` + +### **Docker Compose \(from outside the docker cluster\)** + +You will probably need to connect using `localhost` and standard ports: + +```bash +export FEAST_CORE_URL=localhost:6565 +export FEAST_ONLINE_SERVING_URL=localhost:6566 +export FEAST_BATCH_SERVING_URL=localhost:6567 +``` + +### **Google Kubernetes Engine \(GKE\)** + +You will need to find the external IP of one of the nodes as well as the NodePorts. Please make sure that your firewall is open for these ports: + +```bash +export FEAST_IP=$(kubectl describe nodes | grep ExternalIP | awk '{print $2}' | head -n 1) +export FEAST_CORE_URL=${FEAST_IP}:32090 +export FEAST_ONLINE_SERVING_URL=${FEAST_IP}:32091 +export FEAST_BATCH_SERVING_URL=${FEAST_IP}:32092 +``` + +`netcat`, `telnet`, or even `curl` can be used to test whether all services are available and ports are open, but `grpc_cli` is the most powerful. It can be installed from [here](https://github.com/grpc/grpc/blob/master/doc/command_line_tool.md). + +### Testing Feast Core: + +```bash +grpc_cli ls ${FEAST_CORE_URL} feast.core.CoreService +``` + +```text +GetFeastCoreVersion +GetFeatureSet +ListFeatureSets +ListStores +ApplyFeatureSet +UpdateStore +CreateProject +ArchiveProject +ListProjects +``` + +### Testing Feast Batch Serving and Online Serving + +```bash +grpc_cli ls ${FEAST_BATCH_SERVING_URL} feast.serving.ServingService +``` + +```text +GetFeastServingInfo +GetOnlineFeatures +GetBatchFeatures +GetJob +``` + +```bash +grpc_cli ls ${FEAST_ONLINE_SERVING_URL} feast.serving.ServingService +``` + +```text +GetFeastServingInfo +GetOnlineFeatures +GetBatchFeatures +GetJob +``` + +## How can I print logs from the Feast Services? + +Feast will typically have three services that you need to monitor if something goes wrong. + +* Feast Core +* Feast Serving \(Online\) +* Feast Serving \(Batch\) + +In order to print the logs from these services, please run the commands below. + +### Docker Compose + +```text + docker logs -f feast_core_1 +``` + +```text +docker logs -f feast_batch-serving_1 +``` + +```text +docker logs -f feast_online-serving_1 +``` + +### Google Kubernetes Engine + +```text +kubectl logs $(kubectl get pods | grep feast-core | awk '{print $1}') +``` + +```text +kubectl logs $(kubectl get pods | grep feast-serving-batch | awk '{print $1}') +``` + +```text +kubectl logs $(kubectl get pods | grep feast-serving-online | awk '{print $1}') +``` + diff --git a/docs/introduction/getting-help.md b/docs/introduction/getting-help.md new file mode 100644 index 00000000000..597a782d606 --- /dev/null +++ b/docs/introduction/getting-help.md @@ -0,0 +1,36 @@ +# Getting Help + +### Chat + +* Come and say hello in [\#Feast](https://join.slack.com/t/kubeflow/shared_invite/enQtNDg5MTM4NTQyNjczLTdkNTVhMjg1ZTExOWI0N2QyYTQ2MTIzNTJjMWRiOTFjOGRlZWEzODc1NzMwNTMwM2EzNjY1MTFhODczNjk4MTk) over in the Kubeflow Slack. + +### GitHub + +* Feast's GitHub repo can be [found here](https://github.com/gojek/feast/). +* Found a bug or need a feature? [Create an issue on GitHub](https://github.com/gojek/feast/issues/new) + +### Community Call + +We have a community call every 2 weeks. Alternating between two times. + +* 11 am \(UTC + 8\) +* 5 pm \(UTC + 8\) + +Please join the [**feast-dev**](getting-help.md#feast-development) mailing list to receive the the calendar invitation. + +### Mailing list + +#### Feast discussion + +* Google Group: [https://groups.google.com/d/forum/feast-discuss](https://groups.google.com/d/forum/feast-discuss) +* Mailing List: [feast-discuss@googlegroups.com](mailto:feast-discuss@googlegroups.com) + +#### Feast development + +* Google Group: [https://groups.google.com/d/forum/feast-dev](https://groups.google.com/d/forum/feast-dev) +* Mailing List: [feast-dev@googlegroups.com](mailto:feast-dev@googlegroups.com) + +### Google Drive + +The Feast community also maintains a [Google Drive](https://drive.google.com/drive/u/0/folders/0AAe8j7ZK3sxSUk9PVA) with documents like RFCs, meeting notes, or roadmaps. Please join one of the above mailing lists \(feast-dev or feast-discuss\) to gain access to the drive. + diff --git a/docs/introduction/why-feast.md b/docs/introduction/why-feast.md new file mode 100644 index 00000000000..f5551d5f82f --- /dev/null +++ b/docs/introduction/why-feast.md @@ -0,0 +1,34 @@ +# Why Feast? + +## Lack of feature reuse + +**Problem:** The process of engineering features is one of the most time consuming activities in building an end-to-end ML system. Despite this, many teams continue to redevelop the same features from scratch for every new project. Often these features never leaving the notebooks or pipelines they are built in. + +**Solution:** A centralized feature store allows organizations to build up a foundation of features that can be reused across projects. Teams are then able to utilize features developed by other teams, and as more features are added to the store it becomes easier and cheaper to build models. + +## Serving features is hard + +**Problem:** Serving up to date features at scale is hard. Raw data can come from a variety of sources, from data lakes, to even streams, to data warehouse, to simply flat files. Data scientists need the ability to produce massive datasets of features from this data in order to train their models offline. These models then need access to real-time feature data at low latency and high throughput when they are served in production. + +**Solution:** Feast is built to be able to ingest data from a variety of sources, supporting both streaming and batch sources. Once data is loaded into Feast as features, they become available through both a batch serving API as well as an real-time \(online serving\) API. These APIs allows data scientists and ML engineers to easily retrieve feature data for their development, training, or in production. Feast also comes with a Java, Go, and Python SDK to make this experience easy. + +## **Models need point-in-time correctness** + +**Problem:** Most data sources are not built with ML use cases in mind and by extension don't provide point-in-time correct lookups of feature data. One of the reasons why features are often re-engineered is because ML practitioners need to ensure that their models are trained on a dataset that accurately models the state of the world when the model runs in production. + +**Solution:** Feast allows end users to create point-in-time correct datasets across multiple entities. Feast ensures that there is no data leakage, that cross feature set joins are valid, and that models are not fed expired data. + +## Definitions of features vary + +**Problem:** Teams define features differently and there is no easy access to the documentation of a feature. + +**Solution:** Feast becomes the single source of truth for all feature data for all models within an organization. Teams are able to capture documentation, metadata and metrics about features. This allows teams to communicate clearly about features, test features data, and determine if a feature is useful for a particular model. + +## **Inconsistency between training and serving** + +**Problem:** Training requires access to historical data, whereas models that serve predictions need the latest values. Inconsistencies arise when data is siloed into many independent systems requiring separate tooling. Often teams are using Python for creating batch features off line, but these features are redeveloped with different libraries and languages when moving to serving or streaming systems. + +**Solution:** Feast provides consistency by managing and unifying the ingestion of data from batch and streaming sources into both the feature warehouse and feature serving stores. Feast becomes the bridge between your model and your data, both for training and serving. This ensures that there is a consistency in the feature data that your model receives. + +\*\*\*\* + From d9f6875a80037ee8fd31ee184201836abc4db444 Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Sun, 22 Mar 2020 13:37:04 +0000 Subject: [PATCH 085/176] GitBook: [master] 3 pages modified --- docs/SUMMARY.md | 3 ++- docs/{installation => }/troubleshooting.md | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) rename docs/{installation => }/troubleshooting.md (97%) diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index bdb15821685..768c3b20aa0 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -18,7 +18,6 @@ * [Overview](installation/overview.md) * [Docker Compose](installation/docker-compose.md) * [Google Kubernetes Engine \(GKE\)](installation/gke.md) -* [Troubleshooting](installation/troubleshooting.md) ## Tutorials @@ -27,6 +26,8 @@ ## Administration +* [Troubleshooting](troubleshooting.md) + ## Reference * [Python SDK](https://api.docs.feast.dev/python/) diff --git a/docs/installation/troubleshooting.md b/docs/troubleshooting.md similarity index 97% rename from docs/installation/troubleshooting.md rename to docs/troubleshooting.md index d7ef6a3cdd3..6e2e7f78452 100644 --- a/docs/installation/troubleshooting.md +++ b/docs/troubleshooting.md @@ -1,6 +1,6 @@ # Troubleshooting -Please see the [Getting Help](../introduction/getting-help.md) section for reaching out to the Feast community if you need help. +Please see the [Getting Help](introduction/getting-help.md) section for reaching out to the Feast community if you need help. ## How can I verify that all services are operational? From 83bc621c98e9b88e5c8fc067f9658fdefdc9ce75 Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Sun, 22 Mar 2020 13:42:05 +0000 Subject: [PATCH 086/176] GitBook: [master] 14 pages and one asset modified --- docs/SUMMARY.md | 3 +-- docs/introduction/roadmap.md | 50 ++++++++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+), 2 deletions(-) create mode 100644 docs/introduction/roadmap.md diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 768c3b20aa0..885be61a2e8 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -6,8 +6,8 @@ * [Why Feast?](introduction/why-feast.md) * [Getting Help](introduction/getting-help.md) +* [Roadmap](introduction/roadmap.md) * [Changelog](https://github.com/gojek/feast/blob/master/CHANGELOG.md) -* [Roadmap](roadmap.md) ## Concepts @@ -22,7 +22,6 @@ ## Tutorials * [Basic](https://github.com/gojek/feast/blob/master/examples/basic/basic.ipynb) -* [Using Feast](https://github.com/gojek/feast/blob/master/examples/basic/basic.ipynb) ## Administration diff --git a/docs/introduction/roadmap.md b/docs/introduction/roadmap.md new file mode 100644 index 00000000000..41d8a6b190e --- /dev/null +++ b/docs/introduction/roadmap.md @@ -0,0 +1,50 @@ +# Roadmap + +## Feast 0.5 + +[Discussion](https://github.com/gojek/feast/issues/527) + +### New functionality + +1. Streaming statistics and validation \(M1 from [Feature Validation RFC](https://docs.google.com/document/d/1TPmd7r4mniL9Y-V_glZaWNo5LMXLshEAUpYsohojZ-8/edit)\) +2. Batch statistics and validation \(M2 from [Feature Validation RFC](https://docs.google.com/document/d/1TPmd7r4mniL9Y-V_glZaWNo5LMXLshEAUpYsohojZ-8/edit)\) +3. Support for Redis Clusters \([\#502](https://github.com/gojek/feast/issues/502)\) +4. User authentication & authorization \([\#504](https://github.com/gojek/feast/issues/504)\) +5. Add feature or feature set descriptions \([\#463](https://github.com/gojek/feast/issues/463)\) +6. Redis Cluster Support \([\#478](https://github.com/gojek/feast/issues/478)\) +7. Job management API \([\#302](https://github.com/gojek/feast/issues/302)\) + +### Technical debt, refactoring, or housekeeping + +1. Clean up and document all configuration options \([\#525](https://github.com/gojek/feast/issues/525)\) +2. Externalize storage interfaces \([\#402](https://github.com/gojek/feast/issues/402)\) +3. Reduce memory usage in Redis \([\#515](https://github.com/gojek/feast/issues/515)\) +4. Support for handling out of order ingestion \([\#273](https://github.com/gojek/feast/issues/273)\) +5. Remove feature versions and enable automatic data migration \([\#386](https://github.com/gojek/feast/issues/386)\) \([\#462](https://github.com/gojek/feast/issues/462)\) +6. Tracking of batch ingestion by with dataset\_id/job\_id \([\#461](https://github.com/gojek/feast/issues/461)\) +7. Write Beam metrics after ingestion to store \(not prior\) \([\#489](https://github.com/gojek/feast/issues/489)\) + +## Feast 0.6 + +### New functionality + +1. Extended discovery API/SDK \(needs to be scoped + 1. Resource listing + 2. Schemas, statistics, metrics + 3. Entities as a higher-level concept \([\#405](https://github.com/gojek/feast/issues/405)\) + 4. Add support for discovery based on annotations/labels/tags for easier filtering and discovery +2. Add support for default values \(needs to be scoped\) +3. Add support for audit logs \(needs to be scoped\) +4. Support for an open source warehouse store or connector \(needs to be scoped\) + +### Technical debt, refactoring, or housekeeping + +1. Move all non-registry functionality out of Feast Core and make it optional \(needs to be scoped\) + 1. Allow Feast serving to use its own local feature sets \(files\) + 2. Move job management to Feast serving + 3. Move stream management \(topic generation\) out of Feast core +2. Remove feature set versions from Feast \(not just retrieval API\) \(needs to be scoped\) + 1. Allow for auto-migration of data in Feast + 2. Implement interface for adding a managed data store +3. Multi-store support for serving \(batch and online\) \(needs to be scoped\) + From 2d45e9bce4c4f12486acd2ef74781e5028620456 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Mon, 23 Mar 2020 11:55:41 +0800 Subject: [PATCH 087/176] Implement unit tests on GitHub Actions (#560) * Add unit tests to GitHub Actions * Fix linting dependencies * Fix linting dependencies command * Remove GOROOT variable from Makefile * Remove batch retrieval test and fix golang dependencies * Change .prow folder to infra to fix tests * Downgrade protoc version --- .github/workflows/lint.yaml | 13 +- .github/workflows/unit-tests.yml | 45 + Makefile | 13 +- go.mod | 8 +- go.sum | 38 + infra/docker/ci/Dockerfile | 33 +- infra/scripts/test-core-ingestion.sh | 2 +- infra/scripts/test-end-to-end-batch.sh | 2 +- infra/scripts/test-end-to-end.sh | 2 +- infra/scripts/test-serving.sh | 2 +- sdk/go/go.mod | 4 +- sdk/go/go.sum | 17 + sdk/go/protos/feast/core/CoreService.pb.go | 1423 ----------------- sdk/go/protos/feast/core/FeatureSet.pb.go | 1008 ------------ sdk/go/protos/feast/core/Source.pb.go | 201 --- sdk/go/protos/feast/core/Store.pb.go | 531 ------ .../protos/feast/serving/ServingService.pb.go | 12 +- sdk/go/protos/feast/storage/Redis.pb.go | 96 -- sdk/go/protos/feast/types/FeatureRow.pb.go | 4 +- sdk/go/protos/feast/types/Field.pb.go | 4 +- sdk/go/protos/feast/types/Value.pb.go | 4 +- sdk/python/requirements-ci.txt | 10 +- sdk/python/requirements-dev.txt | 2 +- sdk/python/tests/test_client.py | 262 ++- 24 files changed, 299 insertions(+), 3437 deletions(-) create mode 100644 .github/workflows/unit-tests.yml delete mode 100644 sdk/go/protos/feast/core/CoreService.pb.go delete mode 100644 sdk/go/protos/feast/core/FeatureSet.pb.go delete mode 100644 sdk/go/protos/feast/core/Source.pb.go delete mode 100644 sdk/go/protos/feast/core/Store.pb.go delete mode 100644 sdk/go/protos/feast/storage/Redis.pb.go diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml index 8bd80785130..879ffb68ee1 100644 --- a/.github/workflows/lint.yaml +++ b/.github/workflows/lint.yaml @@ -1,4 +1,4 @@ -name: Lint +name: linting on: push: @@ -12,7 +12,7 @@ jobs: runs-on: [ubuntu-latest] steps: - uses: actions/checkout@v2 - - name: Lint Java + - name: lint java run: make lint-java lint-python: @@ -20,8 +20,9 @@ jobs: runs-on: [ubuntu-latest] steps: - uses: actions/checkout@v2 - - - name: Lint Python + - name: install dependencies + run: make install-python-ci-dependencies + - name: lint python run: make lint-python lint-go: @@ -29,5 +30,7 @@ jobs: runs-on: [ubuntu-latest] steps: - uses: actions/checkout@v2 - - name: Lint Go + - name: install dependencies + run: make install-go-ci-dependencies + - name: lint go run: make lint-go \ No newline at end of file diff --git a/.github/workflows/unit-tests.yml b/.github/workflows/unit-tests.yml new file mode 100644 index 00000000000..676b1b50987 --- /dev/null +++ b/.github/workflows/unit-tests.yml @@ -0,0 +1,45 @@ +name: unit tests + +on: + push: + branches: [ master ] + pull_request: + branches: [ master ] + +jobs: + unit-test-java: + runs-on: ubuntu-latest + container: gcr.io/kf-feast/feast-ci:latest + name: unit test java + steps: + - uses: actions/checkout@v1 + - uses: actions/cache@v1 + with: + path: ~/.m2/repository + key: ${{ runner.os }}-maven-jdk11-${{ hashFiles('**/pom.xml') }} + restore-keys: | + ${{ runner.os }}-maven-jdk11- + - name: test java + run: make test-java + + unit-test-python: + runs-on: ubuntu-latest + container: gcr.io/kf-feast/feast-ci:latest + name: unit test python + steps: + - uses: actions/checkout@v1 + - name: install python + run: make install-python + - name: test python + run: make test-python + + unit-test-go: + runs-on: ubuntu-latest + container: gcr.io/kf-feast/feast-ci:latest + name: unit test go + steps: + - uses: actions/checkout@v1 + - name: install dependencies + run: make compile-protos-go + - name: test go + run: make test-go \ No newline at end of file diff --git a/Makefile b/Makefile index cddee59d8ea..51fcab76da6 100644 --- a/Makefile +++ b/Makefile @@ -59,6 +59,9 @@ compile-protos-python: install-python-ci-dependencies @$(foreach dir,$(PROTO_SERVICE_SUBDIRS),cd ${ROOT_DIR}/protos; python -m grpc_tools.protoc -I. --grpc_python_out=../sdk/python/ feast/$(dir)/*.proto;) cd ${ROOT_DIR}/protos; python -m grpc_tools.protoc -I. --python_out=../sdk/python/ --mypy_out=../sdk/python/ tensorflow_metadata/proto/v0/*.proto +install-python: compile-protos-python + pip install -e sdk/python --upgrade + test-python: pytest --verbose --color=yes sdk/python/tests @@ -75,10 +78,11 @@ lint-python: # Go SDK install-go-ci-dependencies: + go get -u github.com/golang/protobuf/protoc-gen-go go get -u golang.org/x/lint/golint -compile-protos-go: - @$(foreach dir,$(PROTO_TYPE_SUBDIRS), cd ${ROOT_DIR}/protos; protoc -I/usr/local/include -I. --go_out=plugins=grpc,paths=source_relative:../sdk/go/protos/ feast/$(dir)/*.proto;) +compile-protos-go: install-go-ci-dependencies + @$(foreach dir,types serving, cd ${ROOT_DIR}/protos; protoc -I/usr/local/include -I. --go_out=plugins=grpc,paths=source_relative:../sdk/go/protos/ feast/$(dir)/*.proto;) test-go: cd ${ROOT_DIR}/sdk/go; go test ./... @@ -87,7 +91,7 @@ format-go: cd ${ROOT_DIR}/sdk/go; gofmt -s -w *.go lint-go: - cd ${ROOT_DIR}/sdk/go; go vet; golint *.go + cd ${ROOT_DIR}/sdk/go; go vet # Docker @@ -120,9 +124,6 @@ build-ci-docker: # Documentation install-dependencies-proto-docs: - # Use the following command to compile dependencies if installing using the below method. - # cd ${ROOT_DIR}/protos; PATH=$$HOME/bin:$$PATH protoc -I $$HOME/include/ \ - # -I . --docs_out=../dist/grpc feast/*/*.proto cd ${ROOT_DIR}/protos; mkdir -p $$HOME/bin mkdir -p $$HOME/include diff --git a/go.mod b/go.mod index 45ce654beae..047f84aa791 100644 --- a/go.mod +++ b/go.mod @@ -20,8 +20,12 @@ require ( github.com/spf13/cobra v0.0.4 github.com/spf13/viper v1.4.0 github.com/woop/protoc-gen-doc v1.3.0 // indirect - golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553 - google.golang.org/grpc v1.24.0 + golang.org/x/lint v0.0.0-20200302205851-738671d3881b // indirect + golang.org/x/net v0.0.0-20200320220750-118fecf932d8 + golang.org/x/sys v0.0.0-20200321134203-328b4cd54aae // indirect + golang.org/x/tools v0.0.0-20200321224714-0d839f3cf2ed // indirect + google.golang.org/genproto v0.0.0-20200319113533-08878b785e9c // indirect + google.golang.org/grpc v1.28.0 gopkg.in/russross/blackfriday.v2 v2.0.0 // indirect gopkg.in/yaml.v2 v2.2.4 istio.io/gogo-genproto v0.0.0-20191212213402-78a529a42cd8 // indirect diff --git a/go.sum b/go.sum index 29b02420e91..e6021308eba 100644 --- a/go.sum +++ b/go.sum @@ -35,10 +35,12 @@ github.com/asaskevich/govalidator v0.0.0-20180720115003-f9ffefc3facf/go.mod h1:l github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q= github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8= github.com/blang/semver v3.5.0+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnwebNt5EWlYSAyrTnjyyk= +github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc= github.com/client9/gospell v0.0.0-20160306015952-90dfc71015df h1:XXCjxndsxMyNjoZtyuyDnzSck+h681QN7vKkK0EIVq0= github.com/client9/gospell v0.0.0-20160306015952-90dfc71015df/go.mod h1:X4IDm8zK6KavjWkfKQCet43DKeLii9nJhUK/seHoSbA= github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= +github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc= github.com/cockroachdb/apd v1.1.0/go.mod h1:8Sl8LxpKi29FqWXR16WEFZRNSz3SoPzUzeMeY4+DwBQ= github.com/cockroachdb/apd/v2 v2.0.1/go.mod h1:DDxRlzC2lo3/vSlmSoS7JkqbbrARPuFOGr0B9pvN3Gw= github.com/coreos/bbolt v1.3.1-coreos.6/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk= @@ -68,6 +70,10 @@ github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3 github.com/elazarl/goproxy v0.0.0-20170405201442-c4fc26588b6e/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc= github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs= github.com/emicklei/proto v1.6.15/go.mod h1:rn1FgRS/FANiZdD2djyH7TMA9jdRDcYQ9IEN9yvjX0A= +github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= +github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= +github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98= +github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= github.com/evanphx/json-patch v0.0.0-20190203023257-5858425f7550/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= github.com/evanphx/json-patch v4.2.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk= github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I= @@ -133,6 +139,7 @@ github.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs= github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= github.com/golang/protobuf v1.3.5 h1:F768QJ1E9tib+q5Sc8MkdJi1RxLTbRcTf8LJV56aRls= github.com/golang/protobuf v1.3.5/go.mod h1:6O5/vntMXwX2lRkT1hjjk0nAC1IDOTvTlVgjlRvqsdk= github.com/google/btree v0.0.0-20160524151835-7d79101e329e/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= @@ -250,6 +257,7 @@ github.com/prometheus/client_golang v0.9.2/go.mod h1:OsXs2jCmiKlQ1lTBmv21f2mNfw4 github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso= github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo= github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= +github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro= github.com/prometheus/common v0.0.0-20181126121408-4724e9255275/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro= github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4= @@ -309,6 +317,7 @@ github.com/woop/protoc-gen-doc v1.3.0/go.mod h1:/cPn1JCjHFIrRBAffIVBmWjOO/h+K5IY github.com/xiang90/probing v0.0.0-20160813154853-07dd2e8dfe18/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU= github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU= github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q= +github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU= go.opencensus.io v0.21.0 h1:mU6zScU4U1YAFPHEHYk+3JC4SY7JxgkqS10ZOSyksNg= go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= @@ -328,6 +337,7 @@ golang.org/x/crypto v0.0.0-20190211182817-74369b46fc67/go.mod h1:6SG95UA2DQfeDnf golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= golang.org/x/crypto v0.0.0-20190611184440-5c40567a22f8/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/crypto v0.0.0-20190923035154-9ee001bba392/go.mod h1:/lpIB1dKB+9EgE3H3cr1v9wB50oz8l4C4h62xy7jSTY= +golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI= golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190125153040-c74c464bbbf2/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA= golang.org/x/exp v0.0.0-20190312203227-4b39c73a6495/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8= @@ -335,8 +345,13 @@ golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMx golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU= golang.org/x/lint v0.0.0-20190301231843-5614ed5bae6f/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE= +golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3 h1:XQyxROzUlZH+WIQwySDgnISgOivlhjIEwaQaJEJrrN0= golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc= +golang.org/x/lint v0.0.0-20200302205851-738671d3881b h1:Wh+f8QHJXR411sJR8/vRBTZ7YapZaRvUcLFFJhusH0k= +golang.org/x/lint v0.0.0-20200302205851-738671d3881b/go.mod h1:3xt1FjdF8hUf6vQPIChWIBhFzV8gjjsPE/fR3IyQdNY= golang.org/x/mobile v0.0.0-20190312151609-d3739f865fa6/go.mod h1:z+o9i4GpDbdi3rU15maQ/Ox0txvL9dWGYEHz965HBQE= +golang.org/x/mod v0.1.1-0.20191105210325-c90efee705ee/go.mod h1:QqPTAvyqsEbceGzBzNggFXnrqF1CaUcvgkdR5Ot7KZg= +golang.org/x/mod v0.2.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA= golang.org/x/net v0.0.0-20170114055629-f2499483f923/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4= golang.org/x/net v0.0.0-20180826012351-8a410e7b638d h1:g9qWBGx4puODJTMVyoPrpoxPFgVGd+z1DZwjfRu4d0I= @@ -358,6 +373,9 @@ golang.org/x/net v0.0.0-20191004110552-13f9640d40b9 h1:rjwSpXsdiK0dV8/Naq3kAw9ym golang.org/x/net v0.0.0-20191004110552-13f9640d40b9/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553 h1:efeOvDhwQ29Dj3SdAV/MJf8oukgn+8D8WgaCaRMchF8= golang.org/x/net v0.0.0-20191209160850-c0dbc17a3553/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200226121028-0de0cce0169b/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20200320220750-118fecf932d8 h1:1+zQlQqEEhUeStBTi653GZAnAuivZq/2hz+Iz+OP7rg= +golang.org/x/net v0.0.0-20200320220750-118fecf932d8/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U= golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= golang.org/x/oauth2 v0.0.0-20190402181905-9f3314589c9a/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw= @@ -367,6 +385,7 @@ golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJ golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= golang.org/x/sys v0.0.0-20170830134202-bb24a47a89ea/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= @@ -384,6 +403,8 @@ golang.org/x/sys v0.0.0-20190616124812-15dcb6c0061f/go.mod h1:h1NjWce9XRLGQEsW7w golang.org/x/sys v0.0.0-20190922100055-0a153f010e69/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe h1:6fAMxZRR6sl1Uq8U61gxU+kPTs2tR8uOySCbBP7BN/M= golang.org/x/sys v0.0.0-20190924154521-2837fb4f24fe/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200321134203-328b4cd54aae h1:3tcmuaB7wwSZtelmiv479UjUB+vviwABz7a133ZwOKQ= +golang.org/x/sys v0.0.0-20200321134203-328b4cd54aae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= golang.org/x/text v0.0.0-20160726164857-2910a502d2bf/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg= golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= @@ -407,8 +428,17 @@ golang.org/x/tools v0.0.0-20190312151545-0bb0c0a6e846/go.mod h1:LCzVGOaR6xXOjkQ3 golang.org/x/tools v0.0.0-20190312170243-e65039ee4138/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= +golang.org/x/tools v0.0.0-20190614205625-5aca471b1d59 h1:QjA/9ArTfVTLfEhClDCG7SGrZkZixxWpwNCDiwJfh88= golang.org/x/tools v0.0.0-20190614205625-5aca471b1d59/go.mod h1:/rFqwRUd4F7ZHNgwSSTFct+R/Kf4OFW1sUzUTQQTgfc= +golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28= +golang.org/x/tools v0.0.0-20200321014904-268ba720d32c h1:Qp5jXmUCqMiVq4676uW7bY2oskIR1ivTboSMn8qgeX0= +golang.org/x/tools v0.0.0-20200321014904-268ba720d32c/go.mod h1:Sl4aGygMT6LrqrWclx+PTx3U+LnKx/seiNR+3G19Ar8= +golang.org/x/tools v0.0.0-20200321224714-0d839f3cf2ed h1:OCZDlBlLYiUK6T33/8+3BnojrS2W+Dg1rKYJhR89xGE= +golang.org/x/tools v0.0.0-20200321224714-0d839f3cf2ed/go.mod h1:Sl4aGygMT6LrqrWclx+PTx3U+LnKx/seiNR+3G19Ar8= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= gonum.org/v1/gonum v0.0.0-20190331200053-3d26580ed485/go.mod h1:2ltnJ7xHfj0zHS40VVPYEAAMTa3ZGguvHGBSJeRWqE0= gonum.org/v1/netlib v0.0.0-20190313105609-8cb42192e0e0/go.mod h1:wa6Ws7BG/ESfp6dHfk7C6KdzKA7wR7u/rKwOGE66zvw= gonum.org/v1/netlib v0.0.0-20190331212654-76723241ea4e/go.mod h1:kS+toOQn6AQKjmKJ7gzohV1XkqsFehRA2FbsbkopSuQ= @@ -425,6 +455,10 @@ google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7 h1:ZUjXAXmrAyrmmCP google.golang.org/genproto v0.0.0-20190418145605-e7d98fc518a7/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb h1:i1Ppqkc3WQXikh8bXiwHqAN5Rv3/qDCcRk0/Otx73BY= google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55 h1:gSJIx1SDwno+2ElGhA4+qG2zF97qiUzTM+rQ0klBOcE= +google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= +google.golang.org/genproto v0.0.0-20200319113533-08878b785e9c h1:5aI3/f/3eCZps9xwoEnmgfDJDhMbnJpfqeGpjVNgVEI= +google.golang.org/genproto v0.0.0-20200319113533-08878b785e9c/go.mod h1:55QSHmfGQM9UVYDPBsyGGes0y52j32PQ3BqQfXhyH3c= google.golang.org/grpc v1.13.0/go.mod h1:yo6s7OP7yaDglbqo1J04qKzAhqBH6lvTonzMVmEdcZw= google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= google.golang.org/grpc v1.19.1 h1:TrBcJ1yqAl1G++wO39nD/qtgpsW9/1+QGrluyMGEYgM= @@ -435,6 +469,10 @@ google.golang.org/grpc v1.23.0 h1:AzbTB6ux+okLTzP8Ru1Xs41C303zdcfEht7MQnYJt5A= google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= google.golang.org/grpc v1.24.0 h1:vb/1TCsVn3DcJlQ0Gs1yB1pKI6Do2/QNwxdKqmc/b0s= google.golang.org/grpc v1.24.0/go.mod h1:XDChyiUovWa60DnaeDeZmSW86xtLtjtZbwvSiRnRtcA= +google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY= +google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= +google.golang.org/grpc v1.28.0 h1:bO/TA4OxCOummhSf10siHuG7vJOiwh7SpRpFZDkOgl4= +google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKal+60= gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= diff --git a/infra/docker/ci/Dockerfile b/infra/docker/ci/Dockerfile index 350f9ddee79..08da02ae202 100644 --- a/infra/docker/ci/Dockerfile +++ b/infra/docker/ci/Dockerfile @@ -1,18 +1,15 @@ FROM maven:3.6-jdk-11 -ENV PYTHON_VERSION 3.7 -ENV GOLANG_VERSION 1.14.1 - +# Install Google Cloud SDK RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" \ | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \ curl https://packages.cloud.google.com/apt/doc/apt-key.gpg \ | apt-key --keyring /usr/share/keyrings/cloud.google.gpg \ add - && apt-get update -y && apt-get install google-cloud-sdk -y -# Update dependencies -RUN apt-get update - # Install Make and Python +ENV PYTHON_VERSION 3.7 + RUN apt-get install -y build-essential curl python${PYTHON_VERSION} \ python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-distutils && \ update-alternatives --install /usr/bin/python python /usr/bin/python${PYTHON_VERSION} 1 && \ @@ -23,13 +20,25 @@ RUN apt-get install -y build-essential curl python${PYTHON_VERSION} \ # Install Go +ENV GOLANG_VERSION 1.14.1 + RUN curl -O https://storage.googleapis.com/golang/go${GOLANG_VERSION}.linux-amd64.tar.gz && \ tar -xvf go${GOLANG_VERSION}.linux-amd64.tar.gz && chown -R root:root ./go && mv go /usr/local + ENV GOPATH /go ENV PATH $GOPATH/bin:/usr/local/go/bin:$PATH - -# Add contents of local Feast repository to image (execute from Feast root) -COPY . /feast/ - -# Install all dependencies -RUN cd /feast && make install-ci-dependencies \ No newline at end of file +ENV PATH="$HOME/bin:${PATH}" + +# Install Protoc and Plugins +ENV PROTOC_VERSION 3.10.0 + +RUN PROTOC_ZIP=protoc-${PROTOC_VERSION}-linux-x86_64.zip && \ + curl -OL https://github.com/protocolbuffers/protobuf/releases/download/v${PROTOC_VERSION}/$PROTOC_ZIP && \ + unzip -o $PROTOC_ZIP -d /usr/local bin/protoc && \ + unzip -o $PROTOC_ZIP -d /usr/local 'include/*' && \ + rm -f $PROTOC_ZIP && \ + go get github.com/golang/protobuf/proto && \ + go get gopkg.in/russross/blackfriday.v2 && \ + git clone https://github.com/istio/tools/ && \ + cd tools/cmd/protoc-gen-docs && \ + go build && mkdir -p $HOME/bin && cp protoc-gen-docs $HOME/bin \ No newline at end of file diff --git a/infra/scripts/test-core-ingestion.sh b/infra/scripts/test-core-ingestion.sh index af91c4c63f8..d3b42926d83 100755 --- a/infra/scripts/test-core-ingestion.sh +++ b/infra/scripts/test-core-ingestion.sh @@ -5,7 +5,7 @@ apt-get -y install build-essential make lint-java -.prow/scripts/download-maven-cache.sh \ +infra/scripts/download-maven-cache.sh \ --archive-uri gs://feast-templocation-kf-feast/.m2.2019-10-24.tar \ --output-dir /root/ diff --git a/infra/scripts/test-end-to-end-batch.sh b/infra/scripts/test-end-to-end-batch.sh index 8448cb69cc5..9bc17c2e757 100755 --- a/infra/scripts/test-end-to-end-batch.sh +++ b/infra/scripts/test-end-to-end-batch.sh @@ -90,7 +90,7 @@ if [[ ${SKIP_BUILD_JARS} != "true" ]]; then ============================================================ " - .prow/scripts/download-maven-cache.sh \ + infra/scripts/download-maven-cache.sh \ --archive-uri gs://feast-templocation-kf-feast/.m2.2019-10-24.tar \ --output-dir /root/ diff --git a/infra/scripts/test-end-to-end.sh b/infra/scripts/test-end-to-end.sh index de48116a28e..3e6a5492a16 100755 --- a/infra/scripts/test-end-to-end.sh +++ b/infra/scripts/test-end-to-end.sh @@ -73,7 +73,7 @@ Building jars for Feast ============================================================ " -.prow/scripts/download-maven-cache.sh \ +infra/scripts/download-maven-cache.sh \ --archive-uri gs://feast-templocation-kf-feast/.m2.2019-10-24.tar \ --output-dir /root/ diff --git a/infra/scripts/test-serving.sh b/infra/scripts/test-serving.sh index b56001619b3..ce9dc0a8162 100755 --- a/infra/scripts/test-serving.sh +++ b/infra/scripts/test-serving.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash -.prow/scripts/download-maven-cache.sh \ +infra/scripts/download-maven-cache.sh \ --archive-uri gs://feast-templocation-kf-feast/.m2.2019-10-24.tar \ --output-dir /root/ diff --git a/sdk/go/go.mod b/sdk/go/go.mod index 0def759a4f2..c5c76a5c884 100644 --- a/sdk/go/go.mod +++ b/sdk/go/go.mod @@ -3,10 +3,10 @@ module github.com/gojek/feast/sdk/go go 1.13 require ( - github.com/golang/protobuf v1.3.2 + github.com/golang/protobuf v1.3.3 github.com/google/go-cmp v0.3.1 github.com/opentracing/opentracing-go v1.1.0 github.com/stretchr/testify v1.4.0 // indirect go.opencensus.io v0.22.1 - google.golang.org/grpc v1.24.0 + google.golang.org/grpc v1.28.0 ) diff --git a/sdk/go/go.sum b/sdk/go/go.sum index 04cf3d8d7f3..c4ff145972e 100644 --- a/sdk/go/go.sum +++ b/sdk/go/go.sum @@ -1,8 +1,14 @@ cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw= github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU= +github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU= github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw= +github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGXZJjfX53e64911xZQV5JYwmTeXPW+k8Sc= github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= +github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= +github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98= +github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q= github.com/golang/groupcache v0.0.0-20190702054246-869f871628b6 h1:ZgQEtGgCBiWRM39fZuwSd1LwSqqSW0hOdXCYYDX0R3I= @@ -12,6 +18,8 @@ github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5y github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs= github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= +github.com/golang/protobuf v1.3.3 h1:gyjaxf+svBWX08ZjK86iN9geUJF0H6gp2IRKX6Nf6/I= +github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= github.com/google/go-cmp v0.2.0 h1:+dTQ8DZQJz0Mb/HjFlkptS1FeQ4cWSnN941F8aEG4SQ= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY= @@ -22,6 +30,7 @@ github.com/opentracing/opentracing-go v1.1.0 h1:pWlfV3Bxv7k65HYwkikxat0+s3pV4bsq github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA= github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= github.com/stretchr/testify v1.4.0 h1:2E4SXV/wtOkTonXsotYi4li6zVWxYlZuYNCXe9XRJyk= github.com/stretchr/testify v1.4.0/go.mod h1:j7eGeouHqKxXV5pUuKE4zz7dFj8WfuZ+81PSLYec5m4= @@ -63,10 +72,18 @@ google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8 h1:Nw54tB0rB7hY/N0 google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc= google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb h1:i1Ppqkc3WQXikh8bXiwHqAN5Rv3/qDCcRk0/Otx73BY= google.golang.org/genproto v0.0.0-20190425155659-357c62f0e4bb/go.mod h1:VzzqZJRnGkLBvHegQrXjBqPurQTc5/KpmUdxsrq26oE= +google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55 h1:gSJIx1SDwno+2ElGhA4+qG2zF97qiUzTM+rQ0klBOcE= +google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc= google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= +google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= google.golang.org/grpc v1.24.0 h1:vb/1TCsVn3DcJlQ0Gs1yB1pKI6Do2/QNwxdKqmc/b0s= google.golang.org/grpc v1.24.0/go.mod h1:XDChyiUovWa60DnaeDeZmSW86xtLtjtZbwvSiRnRtcA= +google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY= +google.golang.org/grpc v1.27.0 h1:rRYRFMVgRv6E0D70Skyfsr28tDXIuuPZyWGMPdMcnXg= +google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= +google.golang.org/grpc v1.28.0 h1:bO/TA4OxCOummhSf10siHuG7vJOiwh7SpRpFZDkOgl4= +google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKal+60= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw= diff --git a/sdk/go/protos/feast/core/CoreService.pb.go b/sdk/go/protos/feast/core/CoreService.pb.go deleted file mode 100644 index 45ad9ed79a4..00000000000 --- a/sdk/go/protos/feast/core/CoreService.pb.go +++ /dev/null @@ -1,1423 +0,0 @@ -// Code generated by protoc-gen-go. DO NOT EDIT. -// source: feast/core/CoreService.proto - -package core - -import ( - context "context" - fmt "fmt" - proto "github.com/golang/protobuf/proto" - grpc "google.golang.org/grpc" - codes "google.golang.org/grpc/codes" - status "google.golang.org/grpc/status" - math "math" -) - -// Reference imports to suppress errors if they are not otherwise used. -var _ = proto.Marshal -var _ = fmt.Errorf -var _ = math.Inf - -// This is a compile-time assertion to ensure that this generated file -// is compatible with the proto package it is being compiled against. -// A compilation error at this line likely means your copy of the -// proto package needs to be updated. -const _ = proto.ProtoPackageIsVersion3 // please upgrade the proto package - -type ApplyFeatureSetResponse_Status int32 - -const ( - // Latest feature set version is consistent with provided feature set - ApplyFeatureSetResponse_NO_CHANGE ApplyFeatureSetResponse_Status = 0 - // New feature set or feature set version created - ApplyFeatureSetResponse_CREATED ApplyFeatureSetResponse_Status = 1 - // Error occurred while trying to apply changes - ApplyFeatureSetResponse_ERROR ApplyFeatureSetResponse_Status = 2 -) - -var ApplyFeatureSetResponse_Status_name = map[int32]string{ - 0: "NO_CHANGE", - 1: "CREATED", - 2: "ERROR", -} - -var ApplyFeatureSetResponse_Status_value = map[string]int32{ - "NO_CHANGE": 0, - "CREATED": 1, - "ERROR": 2, -} - -func (x ApplyFeatureSetResponse_Status) String() string { - return proto.EnumName(ApplyFeatureSetResponse_Status_name, int32(x)) -} - -func (ApplyFeatureSetResponse_Status) EnumDescriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{7, 0} -} - -type UpdateStoreResponse_Status int32 - -const ( - // Existing store config matching the given store id is identical to the given store config. - UpdateStoreResponse_NO_CHANGE UpdateStoreResponse_Status = 0 - // New store created or existing config updated. - UpdateStoreResponse_UPDATED UpdateStoreResponse_Status = 1 -) - -var UpdateStoreResponse_Status_name = map[int32]string{ - 0: "NO_CHANGE", - 1: "UPDATED", -} - -var UpdateStoreResponse_Status_value = map[string]int32{ - "NO_CHANGE": 0, - "UPDATED": 1, -} - -func (x UpdateStoreResponse_Status) String() string { - return proto.EnumName(UpdateStoreResponse_Status_name, int32(x)) -} - -func (UpdateStoreResponse_Status) EnumDescriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{11, 0} -} - -// Request for a single feature set -type GetFeatureSetRequest struct { - // Name of project the feature set belongs to (required) - Project string `protobuf:"bytes,3,opt,name=project,proto3" json:"project,omitempty"` - // Name of feature set (required). - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - // Version of feature set (optional). If omitted then latest feature set will be returned. - Version int32 `protobuf:"varint,2,opt,name=version,proto3" json:"version,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *GetFeatureSetRequest) Reset() { *m = GetFeatureSetRequest{} } -func (m *GetFeatureSetRequest) String() string { return proto.CompactTextString(m) } -func (*GetFeatureSetRequest) ProtoMessage() {} -func (*GetFeatureSetRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{0} -} - -func (m *GetFeatureSetRequest) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_GetFeatureSetRequest.Unmarshal(m, b) -} -func (m *GetFeatureSetRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_GetFeatureSetRequest.Marshal(b, m, deterministic) -} -func (m *GetFeatureSetRequest) XXX_Merge(src proto.Message) { - xxx_messageInfo_GetFeatureSetRequest.Merge(m, src) -} -func (m *GetFeatureSetRequest) XXX_Size() int { - return xxx_messageInfo_GetFeatureSetRequest.Size(m) -} -func (m *GetFeatureSetRequest) XXX_DiscardUnknown() { - xxx_messageInfo_GetFeatureSetRequest.DiscardUnknown(m) -} - -var xxx_messageInfo_GetFeatureSetRequest proto.InternalMessageInfo - -func (m *GetFeatureSetRequest) GetProject() string { - if m != nil { - return m.Project - } - return "" -} - -func (m *GetFeatureSetRequest) GetName() string { - if m != nil { - return m.Name - } - return "" -} - -func (m *GetFeatureSetRequest) GetVersion() int32 { - if m != nil { - return m.Version - } - return 0 -} - -// Response containing a single feature set -type GetFeatureSetResponse struct { - FeatureSet *FeatureSet `protobuf:"bytes,1,opt,name=feature_set,json=featureSet,proto3" json:"feature_set,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *GetFeatureSetResponse) Reset() { *m = GetFeatureSetResponse{} } -func (m *GetFeatureSetResponse) String() string { return proto.CompactTextString(m) } -func (*GetFeatureSetResponse) ProtoMessage() {} -func (*GetFeatureSetResponse) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{1} -} - -func (m *GetFeatureSetResponse) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_GetFeatureSetResponse.Unmarshal(m, b) -} -func (m *GetFeatureSetResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_GetFeatureSetResponse.Marshal(b, m, deterministic) -} -func (m *GetFeatureSetResponse) XXX_Merge(src proto.Message) { - xxx_messageInfo_GetFeatureSetResponse.Merge(m, src) -} -func (m *GetFeatureSetResponse) XXX_Size() int { - return xxx_messageInfo_GetFeatureSetResponse.Size(m) -} -func (m *GetFeatureSetResponse) XXX_DiscardUnknown() { - xxx_messageInfo_GetFeatureSetResponse.DiscardUnknown(m) -} - -var xxx_messageInfo_GetFeatureSetResponse proto.InternalMessageInfo - -func (m *GetFeatureSetResponse) GetFeatureSet() *FeatureSet { - if m != nil { - return m.FeatureSet - } - return nil -} - -// Retrieves details for all versions of a specific feature set -type ListFeatureSetsRequest struct { - Filter *ListFeatureSetsRequest_Filter `protobuf:"bytes,1,opt,name=filter,proto3" json:"filter,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *ListFeatureSetsRequest) Reset() { *m = ListFeatureSetsRequest{} } -func (m *ListFeatureSetsRequest) String() string { return proto.CompactTextString(m) } -func (*ListFeatureSetsRequest) ProtoMessage() {} -func (*ListFeatureSetsRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{2} -} - -func (m *ListFeatureSetsRequest) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_ListFeatureSetsRequest.Unmarshal(m, b) -} -func (m *ListFeatureSetsRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_ListFeatureSetsRequest.Marshal(b, m, deterministic) -} -func (m *ListFeatureSetsRequest) XXX_Merge(src proto.Message) { - xxx_messageInfo_ListFeatureSetsRequest.Merge(m, src) -} -func (m *ListFeatureSetsRequest) XXX_Size() int { - return xxx_messageInfo_ListFeatureSetsRequest.Size(m) -} -func (m *ListFeatureSetsRequest) XXX_DiscardUnknown() { - xxx_messageInfo_ListFeatureSetsRequest.DiscardUnknown(m) -} - -var xxx_messageInfo_ListFeatureSetsRequest proto.InternalMessageInfo - -func (m *ListFeatureSetsRequest) GetFilter() *ListFeatureSetsRequest_Filter { - if m != nil { - return m.Filter - } - return nil -} - -type ListFeatureSetsRequest_Filter struct { - // Name of project that the feature sets belongs to. This can be one of - // - [project_name] - // - * - // If an asterisk is provided, filtering on projects will be disabled. All projects will - // be matched. It is NOT possible to provide an asterisk with a string in order to do - // pattern matching. - Project string `protobuf:"bytes,3,opt,name=project,proto3" json:"project,omitempty"` - // Name of the desired feature set. Asterisks can be used as wildcards in the name. - // Matching on names is only permitted if a specific project is defined. It is disallowed - // If the project name is set to "*" - // e.g. - // - * can be used to match all feature sets - // - my-feature-set* can be used to match all features prefixed by "my-feature-set" - // - my-feature-set-6 can be used to select a single feature set - FeatureSetName string `protobuf:"bytes,1,opt,name=feature_set_name,json=featureSetName,proto3" json:"feature_set_name,omitempty"` - // Versions of the given feature sets that will be returned. - // Valid options for version: - // "latest": only the latest version is returned. - // "*": Subscribe to all versions - // [version number]: pin to a specific version. Project and feature set name must be - // explicitly defined if a specific version is pinned. - FeatureSetVersion string `protobuf:"bytes,2,opt,name=feature_set_version,json=featureSetVersion,proto3" json:"feature_set_version,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *ListFeatureSetsRequest_Filter) Reset() { *m = ListFeatureSetsRequest_Filter{} } -func (m *ListFeatureSetsRequest_Filter) String() string { return proto.CompactTextString(m) } -func (*ListFeatureSetsRequest_Filter) ProtoMessage() {} -func (*ListFeatureSetsRequest_Filter) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{2, 0} -} - -func (m *ListFeatureSetsRequest_Filter) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_ListFeatureSetsRequest_Filter.Unmarshal(m, b) -} -func (m *ListFeatureSetsRequest_Filter) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_ListFeatureSetsRequest_Filter.Marshal(b, m, deterministic) -} -func (m *ListFeatureSetsRequest_Filter) XXX_Merge(src proto.Message) { - xxx_messageInfo_ListFeatureSetsRequest_Filter.Merge(m, src) -} -func (m *ListFeatureSetsRequest_Filter) XXX_Size() int { - return xxx_messageInfo_ListFeatureSetsRequest_Filter.Size(m) -} -func (m *ListFeatureSetsRequest_Filter) XXX_DiscardUnknown() { - xxx_messageInfo_ListFeatureSetsRequest_Filter.DiscardUnknown(m) -} - -var xxx_messageInfo_ListFeatureSetsRequest_Filter proto.InternalMessageInfo - -func (m *ListFeatureSetsRequest_Filter) GetProject() string { - if m != nil { - return m.Project - } - return "" -} - -func (m *ListFeatureSetsRequest_Filter) GetFeatureSetName() string { - if m != nil { - return m.FeatureSetName - } - return "" -} - -func (m *ListFeatureSetsRequest_Filter) GetFeatureSetVersion() string { - if m != nil { - return m.FeatureSetVersion - } - return "" -} - -type ListFeatureSetsResponse struct { - FeatureSets []*FeatureSet `protobuf:"bytes,1,rep,name=feature_sets,json=featureSets,proto3" json:"feature_sets,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *ListFeatureSetsResponse) Reset() { *m = ListFeatureSetsResponse{} } -func (m *ListFeatureSetsResponse) String() string { return proto.CompactTextString(m) } -func (*ListFeatureSetsResponse) ProtoMessage() {} -func (*ListFeatureSetsResponse) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{3} -} - -func (m *ListFeatureSetsResponse) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_ListFeatureSetsResponse.Unmarshal(m, b) -} -func (m *ListFeatureSetsResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_ListFeatureSetsResponse.Marshal(b, m, deterministic) -} -func (m *ListFeatureSetsResponse) XXX_Merge(src proto.Message) { - xxx_messageInfo_ListFeatureSetsResponse.Merge(m, src) -} -func (m *ListFeatureSetsResponse) XXX_Size() int { - return xxx_messageInfo_ListFeatureSetsResponse.Size(m) -} -func (m *ListFeatureSetsResponse) XXX_DiscardUnknown() { - xxx_messageInfo_ListFeatureSetsResponse.DiscardUnknown(m) -} - -var xxx_messageInfo_ListFeatureSetsResponse proto.InternalMessageInfo - -func (m *ListFeatureSetsResponse) GetFeatureSets() []*FeatureSet { - if m != nil { - return m.FeatureSets - } - return nil -} - -type ListStoresRequest struct { - Filter *ListStoresRequest_Filter `protobuf:"bytes,1,opt,name=filter,proto3" json:"filter,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *ListStoresRequest) Reset() { *m = ListStoresRequest{} } -func (m *ListStoresRequest) String() string { return proto.CompactTextString(m) } -func (*ListStoresRequest) ProtoMessage() {} -func (*ListStoresRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{4} -} - -func (m *ListStoresRequest) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_ListStoresRequest.Unmarshal(m, b) -} -func (m *ListStoresRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_ListStoresRequest.Marshal(b, m, deterministic) -} -func (m *ListStoresRequest) XXX_Merge(src proto.Message) { - xxx_messageInfo_ListStoresRequest.Merge(m, src) -} -func (m *ListStoresRequest) XXX_Size() int { - return xxx_messageInfo_ListStoresRequest.Size(m) -} -func (m *ListStoresRequest) XXX_DiscardUnknown() { - xxx_messageInfo_ListStoresRequest.DiscardUnknown(m) -} - -var xxx_messageInfo_ListStoresRequest proto.InternalMessageInfo - -func (m *ListStoresRequest) GetFilter() *ListStoresRequest_Filter { - if m != nil { - return m.Filter - } - return nil -} - -type ListStoresRequest_Filter struct { - // Name of desired store. Regex is not supported in this query. - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *ListStoresRequest_Filter) Reset() { *m = ListStoresRequest_Filter{} } -func (m *ListStoresRequest_Filter) String() string { return proto.CompactTextString(m) } -func (*ListStoresRequest_Filter) ProtoMessage() {} -func (*ListStoresRequest_Filter) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{4, 0} -} - -func (m *ListStoresRequest_Filter) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_ListStoresRequest_Filter.Unmarshal(m, b) -} -func (m *ListStoresRequest_Filter) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_ListStoresRequest_Filter.Marshal(b, m, deterministic) -} -func (m *ListStoresRequest_Filter) XXX_Merge(src proto.Message) { - xxx_messageInfo_ListStoresRequest_Filter.Merge(m, src) -} -func (m *ListStoresRequest_Filter) XXX_Size() int { - return xxx_messageInfo_ListStoresRequest_Filter.Size(m) -} -func (m *ListStoresRequest_Filter) XXX_DiscardUnknown() { - xxx_messageInfo_ListStoresRequest_Filter.DiscardUnknown(m) -} - -var xxx_messageInfo_ListStoresRequest_Filter proto.InternalMessageInfo - -func (m *ListStoresRequest_Filter) GetName() string { - if m != nil { - return m.Name - } - return "" -} - -type ListStoresResponse struct { - Store []*Store `protobuf:"bytes,1,rep,name=store,proto3" json:"store,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *ListStoresResponse) Reset() { *m = ListStoresResponse{} } -func (m *ListStoresResponse) String() string { return proto.CompactTextString(m) } -func (*ListStoresResponse) ProtoMessage() {} -func (*ListStoresResponse) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{5} -} - -func (m *ListStoresResponse) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_ListStoresResponse.Unmarshal(m, b) -} -func (m *ListStoresResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_ListStoresResponse.Marshal(b, m, deterministic) -} -func (m *ListStoresResponse) XXX_Merge(src proto.Message) { - xxx_messageInfo_ListStoresResponse.Merge(m, src) -} -func (m *ListStoresResponse) XXX_Size() int { - return xxx_messageInfo_ListStoresResponse.Size(m) -} -func (m *ListStoresResponse) XXX_DiscardUnknown() { - xxx_messageInfo_ListStoresResponse.DiscardUnknown(m) -} - -var xxx_messageInfo_ListStoresResponse proto.InternalMessageInfo - -func (m *ListStoresResponse) GetStore() []*Store { - if m != nil { - return m.Store - } - return nil -} - -type ApplyFeatureSetRequest struct { - // Feature set version and source will be ignored - FeatureSet *FeatureSet `protobuf:"bytes,1,opt,name=feature_set,json=featureSet,proto3" json:"feature_set,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *ApplyFeatureSetRequest) Reset() { *m = ApplyFeatureSetRequest{} } -func (m *ApplyFeatureSetRequest) String() string { return proto.CompactTextString(m) } -func (*ApplyFeatureSetRequest) ProtoMessage() {} -func (*ApplyFeatureSetRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{6} -} - -func (m *ApplyFeatureSetRequest) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_ApplyFeatureSetRequest.Unmarshal(m, b) -} -func (m *ApplyFeatureSetRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_ApplyFeatureSetRequest.Marshal(b, m, deterministic) -} -func (m *ApplyFeatureSetRequest) XXX_Merge(src proto.Message) { - xxx_messageInfo_ApplyFeatureSetRequest.Merge(m, src) -} -func (m *ApplyFeatureSetRequest) XXX_Size() int { - return xxx_messageInfo_ApplyFeatureSetRequest.Size(m) -} -func (m *ApplyFeatureSetRequest) XXX_DiscardUnknown() { - xxx_messageInfo_ApplyFeatureSetRequest.DiscardUnknown(m) -} - -var xxx_messageInfo_ApplyFeatureSetRequest proto.InternalMessageInfo - -func (m *ApplyFeatureSetRequest) GetFeatureSet() *FeatureSet { - if m != nil { - return m.FeatureSet - } - return nil -} - -type ApplyFeatureSetResponse struct { - // Feature set response has been enriched with version and source information - FeatureSet *FeatureSet `protobuf:"bytes,1,opt,name=feature_set,json=featureSet,proto3" json:"feature_set,omitempty"` - Status ApplyFeatureSetResponse_Status `protobuf:"varint,2,opt,name=status,proto3,enum=feast.core.ApplyFeatureSetResponse_Status" json:"status,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *ApplyFeatureSetResponse) Reset() { *m = ApplyFeatureSetResponse{} } -func (m *ApplyFeatureSetResponse) String() string { return proto.CompactTextString(m) } -func (*ApplyFeatureSetResponse) ProtoMessage() {} -func (*ApplyFeatureSetResponse) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{7} -} - -func (m *ApplyFeatureSetResponse) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_ApplyFeatureSetResponse.Unmarshal(m, b) -} -func (m *ApplyFeatureSetResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_ApplyFeatureSetResponse.Marshal(b, m, deterministic) -} -func (m *ApplyFeatureSetResponse) XXX_Merge(src proto.Message) { - xxx_messageInfo_ApplyFeatureSetResponse.Merge(m, src) -} -func (m *ApplyFeatureSetResponse) XXX_Size() int { - return xxx_messageInfo_ApplyFeatureSetResponse.Size(m) -} -func (m *ApplyFeatureSetResponse) XXX_DiscardUnknown() { - xxx_messageInfo_ApplyFeatureSetResponse.DiscardUnknown(m) -} - -var xxx_messageInfo_ApplyFeatureSetResponse proto.InternalMessageInfo - -func (m *ApplyFeatureSetResponse) GetFeatureSet() *FeatureSet { - if m != nil { - return m.FeatureSet - } - return nil -} - -func (m *ApplyFeatureSetResponse) GetStatus() ApplyFeatureSetResponse_Status { - if m != nil { - return m.Status - } - return ApplyFeatureSetResponse_NO_CHANGE -} - -type GetFeastCoreVersionRequest struct { - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *GetFeastCoreVersionRequest) Reset() { *m = GetFeastCoreVersionRequest{} } -func (m *GetFeastCoreVersionRequest) String() string { return proto.CompactTextString(m) } -func (*GetFeastCoreVersionRequest) ProtoMessage() {} -func (*GetFeastCoreVersionRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{8} -} - -func (m *GetFeastCoreVersionRequest) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_GetFeastCoreVersionRequest.Unmarshal(m, b) -} -func (m *GetFeastCoreVersionRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_GetFeastCoreVersionRequest.Marshal(b, m, deterministic) -} -func (m *GetFeastCoreVersionRequest) XXX_Merge(src proto.Message) { - xxx_messageInfo_GetFeastCoreVersionRequest.Merge(m, src) -} -func (m *GetFeastCoreVersionRequest) XXX_Size() int { - return xxx_messageInfo_GetFeastCoreVersionRequest.Size(m) -} -func (m *GetFeastCoreVersionRequest) XXX_DiscardUnknown() { - xxx_messageInfo_GetFeastCoreVersionRequest.DiscardUnknown(m) -} - -var xxx_messageInfo_GetFeastCoreVersionRequest proto.InternalMessageInfo - -type GetFeastCoreVersionResponse struct { - Version string `protobuf:"bytes,1,opt,name=version,proto3" json:"version,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *GetFeastCoreVersionResponse) Reset() { *m = GetFeastCoreVersionResponse{} } -func (m *GetFeastCoreVersionResponse) String() string { return proto.CompactTextString(m) } -func (*GetFeastCoreVersionResponse) ProtoMessage() {} -func (*GetFeastCoreVersionResponse) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{9} -} - -func (m *GetFeastCoreVersionResponse) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_GetFeastCoreVersionResponse.Unmarshal(m, b) -} -func (m *GetFeastCoreVersionResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_GetFeastCoreVersionResponse.Marshal(b, m, deterministic) -} -func (m *GetFeastCoreVersionResponse) XXX_Merge(src proto.Message) { - xxx_messageInfo_GetFeastCoreVersionResponse.Merge(m, src) -} -func (m *GetFeastCoreVersionResponse) XXX_Size() int { - return xxx_messageInfo_GetFeastCoreVersionResponse.Size(m) -} -func (m *GetFeastCoreVersionResponse) XXX_DiscardUnknown() { - xxx_messageInfo_GetFeastCoreVersionResponse.DiscardUnknown(m) -} - -var xxx_messageInfo_GetFeastCoreVersionResponse proto.InternalMessageInfo - -func (m *GetFeastCoreVersionResponse) GetVersion() string { - if m != nil { - return m.Version - } - return "" -} - -type UpdateStoreRequest struct { - Store *Store `protobuf:"bytes,1,opt,name=store,proto3" json:"store,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *UpdateStoreRequest) Reset() { *m = UpdateStoreRequest{} } -func (m *UpdateStoreRequest) String() string { return proto.CompactTextString(m) } -func (*UpdateStoreRequest) ProtoMessage() {} -func (*UpdateStoreRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{10} -} - -func (m *UpdateStoreRequest) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_UpdateStoreRequest.Unmarshal(m, b) -} -func (m *UpdateStoreRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_UpdateStoreRequest.Marshal(b, m, deterministic) -} -func (m *UpdateStoreRequest) XXX_Merge(src proto.Message) { - xxx_messageInfo_UpdateStoreRequest.Merge(m, src) -} -func (m *UpdateStoreRequest) XXX_Size() int { - return xxx_messageInfo_UpdateStoreRequest.Size(m) -} -func (m *UpdateStoreRequest) XXX_DiscardUnknown() { - xxx_messageInfo_UpdateStoreRequest.DiscardUnknown(m) -} - -var xxx_messageInfo_UpdateStoreRequest proto.InternalMessageInfo - -func (m *UpdateStoreRequest) GetStore() *Store { - if m != nil { - return m.Store - } - return nil -} - -type UpdateStoreResponse struct { - Store *Store `protobuf:"bytes,1,opt,name=store,proto3" json:"store,omitempty"` - Status UpdateStoreResponse_Status `protobuf:"varint,2,opt,name=status,proto3,enum=feast.core.UpdateStoreResponse_Status" json:"status,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *UpdateStoreResponse) Reset() { *m = UpdateStoreResponse{} } -func (m *UpdateStoreResponse) String() string { return proto.CompactTextString(m) } -func (*UpdateStoreResponse) ProtoMessage() {} -func (*UpdateStoreResponse) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{11} -} - -func (m *UpdateStoreResponse) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_UpdateStoreResponse.Unmarshal(m, b) -} -func (m *UpdateStoreResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_UpdateStoreResponse.Marshal(b, m, deterministic) -} -func (m *UpdateStoreResponse) XXX_Merge(src proto.Message) { - xxx_messageInfo_UpdateStoreResponse.Merge(m, src) -} -func (m *UpdateStoreResponse) XXX_Size() int { - return xxx_messageInfo_UpdateStoreResponse.Size(m) -} -func (m *UpdateStoreResponse) XXX_DiscardUnknown() { - xxx_messageInfo_UpdateStoreResponse.DiscardUnknown(m) -} - -var xxx_messageInfo_UpdateStoreResponse proto.InternalMessageInfo - -func (m *UpdateStoreResponse) GetStore() *Store { - if m != nil { - return m.Store - } - return nil -} - -func (m *UpdateStoreResponse) GetStatus() UpdateStoreResponse_Status { - if m != nil { - return m.Status - } - return UpdateStoreResponse_NO_CHANGE -} - -// Request to create a project -type CreateProjectRequest struct { - // Name of project (required) - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *CreateProjectRequest) Reset() { *m = CreateProjectRequest{} } -func (m *CreateProjectRequest) String() string { return proto.CompactTextString(m) } -func (*CreateProjectRequest) ProtoMessage() {} -func (*CreateProjectRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{12} -} - -func (m *CreateProjectRequest) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_CreateProjectRequest.Unmarshal(m, b) -} -func (m *CreateProjectRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_CreateProjectRequest.Marshal(b, m, deterministic) -} -func (m *CreateProjectRequest) XXX_Merge(src proto.Message) { - xxx_messageInfo_CreateProjectRequest.Merge(m, src) -} -func (m *CreateProjectRequest) XXX_Size() int { - return xxx_messageInfo_CreateProjectRequest.Size(m) -} -func (m *CreateProjectRequest) XXX_DiscardUnknown() { - xxx_messageInfo_CreateProjectRequest.DiscardUnknown(m) -} - -var xxx_messageInfo_CreateProjectRequest proto.InternalMessageInfo - -func (m *CreateProjectRequest) GetName() string { - if m != nil { - return m.Name - } - return "" -} - -// Response for creation of a project -type CreateProjectResponse struct { - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *CreateProjectResponse) Reset() { *m = CreateProjectResponse{} } -func (m *CreateProjectResponse) String() string { return proto.CompactTextString(m) } -func (*CreateProjectResponse) ProtoMessage() {} -func (*CreateProjectResponse) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{13} -} - -func (m *CreateProjectResponse) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_CreateProjectResponse.Unmarshal(m, b) -} -func (m *CreateProjectResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_CreateProjectResponse.Marshal(b, m, deterministic) -} -func (m *CreateProjectResponse) XXX_Merge(src proto.Message) { - xxx_messageInfo_CreateProjectResponse.Merge(m, src) -} -func (m *CreateProjectResponse) XXX_Size() int { - return xxx_messageInfo_CreateProjectResponse.Size(m) -} -func (m *CreateProjectResponse) XXX_DiscardUnknown() { - xxx_messageInfo_CreateProjectResponse.DiscardUnknown(m) -} - -var xxx_messageInfo_CreateProjectResponse proto.InternalMessageInfo - -// Request for the archival of a project -type ArchiveProjectRequest struct { - // Name of project to be archived - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *ArchiveProjectRequest) Reset() { *m = ArchiveProjectRequest{} } -func (m *ArchiveProjectRequest) String() string { return proto.CompactTextString(m) } -func (*ArchiveProjectRequest) ProtoMessage() {} -func (*ArchiveProjectRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{14} -} - -func (m *ArchiveProjectRequest) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_ArchiveProjectRequest.Unmarshal(m, b) -} -func (m *ArchiveProjectRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_ArchiveProjectRequest.Marshal(b, m, deterministic) -} -func (m *ArchiveProjectRequest) XXX_Merge(src proto.Message) { - xxx_messageInfo_ArchiveProjectRequest.Merge(m, src) -} -func (m *ArchiveProjectRequest) XXX_Size() int { - return xxx_messageInfo_ArchiveProjectRequest.Size(m) -} -func (m *ArchiveProjectRequest) XXX_DiscardUnknown() { - xxx_messageInfo_ArchiveProjectRequest.DiscardUnknown(m) -} - -var xxx_messageInfo_ArchiveProjectRequest proto.InternalMessageInfo - -func (m *ArchiveProjectRequest) GetName() string { - if m != nil { - return m.Name - } - return "" -} - -// Response for archival of a project -type ArchiveProjectResponse struct { - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *ArchiveProjectResponse) Reset() { *m = ArchiveProjectResponse{} } -func (m *ArchiveProjectResponse) String() string { return proto.CompactTextString(m) } -func (*ArchiveProjectResponse) ProtoMessage() {} -func (*ArchiveProjectResponse) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{15} -} - -func (m *ArchiveProjectResponse) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_ArchiveProjectResponse.Unmarshal(m, b) -} -func (m *ArchiveProjectResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_ArchiveProjectResponse.Marshal(b, m, deterministic) -} -func (m *ArchiveProjectResponse) XXX_Merge(src proto.Message) { - xxx_messageInfo_ArchiveProjectResponse.Merge(m, src) -} -func (m *ArchiveProjectResponse) XXX_Size() int { - return xxx_messageInfo_ArchiveProjectResponse.Size(m) -} -func (m *ArchiveProjectResponse) XXX_DiscardUnknown() { - xxx_messageInfo_ArchiveProjectResponse.DiscardUnknown(m) -} - -var xxx_messageInfo_ArchiveProjectResponse proto.InternalMessageInfo - -// Request for listing of projects -type ListProjectsRequest struct { - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *ListProjectsRequest) Reset() { *m = ListProjectsRequest{} } -func (m *ListProjectsRequest) String() string { return proto.CompactTextString(m) } -func (*ListProjectsRequest) ProtoMessage() {} -func (*ListProjectsRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{16} -} - -func (m *ListProjectsRequest) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_ListProjectsRequest.Unmarshal(m, b) -} -func (m *ListProjectsRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_ListProjectsRequest.Marshal(b, m, deterministic) -} -func (m *ListProjectsRequest) XXX_Merge(src proto.Message) { - xxx_messageInfo_ListProjectsRequest.Merge(m, src) -} -func (m *ListProjectsRequest) XXX_Size() int { - return xxx_messageInfo_ListProjectsRequest.Size(m) -} -func (m *ListProjectsRequest) XXX_DiscardUnknown() { - xxx_messageInfo_ListProjectsRequest.DiscardUnknown(m) -} - -var xxx_messageInfo_ListProjectsRequest proto.InternalMessageInfo - -// Response for listing of projects -type ListProjectsResponse struct { - // List of project names (archived projects are filtered out) - Projects []string `protobuf:"bytes,1,rep,name=projects,proto3" json:"projects,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *ListProjectsResponse) Reset() { *m = ListProjectsResponse{} } -func (m *ListProjectsResponse) String() string { return proto.CompactTextString(m) } -func (*ListProjectsResponse) ProtoMessage() {} -func (*ListProjectsResponse) Descriptor() ([]byte, []int) { - return fileDescriptor_d9be266444105411, []int{17} -} - -func (m *ListProjectsResponse) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_ListProjectsResponse.Unmarshal(m, b) -} -func (m *ListProjectsResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_ListProjectsResponse.Marshal(b, m, deterministic) -} -func (m *ListProjectsResponse) XXX_Merge(src proto.Message) { - xxx_messageInfo_ListProjectsResponse.Merge(m, src) -} -func (m *ListProjectsResponse) XXX_Size() int { - return xxx_messageInfo_ListProjectsResponse.Size(m) -} -func (m *ListProjectsResponse) XXX_DiscardUnknown() { - xxx_messageInfo_ListProjectsResponse.DiscardUnknown(m) -} - -var xxx_messageInfo_ListProjectsResponse proto.InternalMessageInfo - -func (m *ListProjectsResponse) GetProjects() []string { - if m != nil { - return m.Projects - } - return nil -} - -func init() { - proto.RegisterEnum("feast.core.ApplyFeatureSetResponse_Status", ApplyFeatureSetResponse_Status_name, ApplyFeatureSetResponse_Status_value) - proto.RegisterEnum("feast.core.UpdateStoreResponse_Status", UpdateStoreResponse_Status_name, UpdateStoreResponse_Status_value) - proto.RegisterType((*GetFeatureSetRequest)(nil), "feast.core.GetFeatureSetRequest") - proto.RegisterType((*GetFeatureSetResponse)(nil), "feast.core.GetFeatureSetResponse") - proto.RegisterType((*ListFeatureSetsRequest)(nil), "feast.core.ListFeatureSetsRequest") - proto.RegisterType((*ListFeatureSetsRequest_Filter)(nil), "feast.core.ListFeatureSetsRequest.Filter") - proto.RegisterType((*ListFeatureSetsResponse)(nil), "feast.core.ListFeatureSetsResponse") - proto.RegisterType((*ListStoresRequest)(nil), "feast.core.ListStoresRequest") - proto.RegisterType((*ListStoresRequest_Filter)(nil), "feast.core.ListStoresRequest.Filter") - proto.RegisterType((*ListStoresResponse)(nil), "feast.core.ListStoresResponse") - proto.RegisterType((*ApplyFeatureSetRequest)(nil), "feast.core.ApplyFeatureSetRequest") - proto.RegisterType((*ApplyFeatureSetResponse)(nil), "feast.core.ApplyFeatureSetResponse") - proto.RegisterType((*GetFeastCoreVersionRequest)(nil), "feast.core.GetFeastCoreVersionRequest") - proto.RegisterType((*GetFeastCoreVersionResponse)(nil), "feast.core.GetFeastCoreVersionResponse") - proto.RegisterType((*UpdateStoreRequest)(nil), "feast.core.UpdateStoreRequest") - proto.RegisterType((*UpdateStoreResponse)(nil), "feast.core.UpdateStoreResponse") - proto.RegisterType((*CreateProjectRequest)(nil), "feast.core.CreateProjectRequest") - proto.RegisterType((*CreateProjectResponse)(nil), "feast.core.CreateProjectResponse") - proto.RegisterType((*ArchiveProjectRequest)(nil), "feast.core.ArchiveProjectRequest") - proto.RegisterType((*ArchiveProjectResponse)(nil), "feast.core.ArchiveProjectResponse") - proto.RegisterType((*ListProjectsRequest)(nil), "feast.core.ListProjectsRequest") - proto.RegisterType((*ListProjectsResponse)(nil), "feast.core.ListProjectsResponse") -} - -func init() { proto.RegisterFile("feast/core/CoreService.proto", fileDescriptor_d9be266444105411) } - -var fileDescriptor_d9be266444105411 = []byte{ - // 762 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xa4, 0x56, 0xef, 0x4e, 0x13, 0x4f, - 0x14, 0xfd, 0x2d, 0xfc, 0x28, 0xf6, 0x16, 0xb0, 0x4c, 0x69, 0x69, 0x16, 0x84, 0x3a, 0x12, 0x41, - 0x4c, 0x76, 0x93, 0xfa, 0x81, 0x98, 0x88, 0x49, 0x29, 0x05, 0x13, 0x4d, 0x29, 0x0b, 0x68, 0xc2, - 0x07, 0x49, 0x29, 0x53, 0x28, 0xff, 0xa6, 0xee, 0x4c, 0x49, 0x4c, 0x7c, 0x1a, 0xe3, 0xbb, 0xf8, - 0x0c, 0xbe, 0x8d, 0xd9, 0x9d, 0x69, 0x77, 0x66, 0xba, 0xdd, 0x1a, 0xfd, 0xd6, 0xbd, 0x73, 0xee, - 0xd9, 0xbb, 0xe7, 0xde, 0x73, 0x3b, 0xb0, 0xdc, 0x26, 0x4d, 0xc6, 0xdd, 0x16, 0xf5, 0x89, 0x5b, - 0xa5, 0x3e, 0x39, 0x22, 0xfe, 0x43, 0xa7, 0x45, 0x9c, 0xae, 0x4f, 0x39, 0x45, 0x10, 0x9e, 0x3a, - 0xc1, 0xa9, 0xbd, 0xa4, 0x20, 0xf7, 0x48, 0x93, 0xf7, 0x02, 0x30, 0x17, 0x40, 0xbb, 0xa0, 0x1c, - 0x1e, 0x71, 0xea, 0x4b, 0x02, 0xfc, 0x19, 0x16, 0xf6, 0x09, 0x8f, 0xe0, 0x1e, 0xf9, 0xd2, 0x23, - 0x8c, 0xa3, 0x22, 0x4c, 0x77, 0x7d, 0x7a, 0x4d, 0x5a, 0xbc, 0x38, 0x59, 0xb2, 0x36, 0xd2, 0x5e, - 0xff, 0x11, 0x21, 0xf8, 0xff, 0xbe, 0x79, 0x47, 0x8a, 0x56, 0x18, 0x0e, 0x7f, 0x07, 0xe8, 0x07, - 0xe2, 0xb3, 0x0e, 0xbd, 0x2f, 0x4e, 0x94, 0xac, 0x8d, 0x29, 0xaf, 0xff, 0x88, 0x1b, 0x90, 0x37, - 0xf8, 0x59, 0x97, 0xde, 0x33, 0x82, 0xb6, 0x20, 0xd3, 0x16, 0xd1, 0x33, 0x46, 0x78, 0xc8, 0x96, - 0x29, 0x17, 0x9c, 0xe8, 0x7b, 0x1c, 0x25, 0x09, 0xda, 0x83, 0xdf, 0xf8, 0x97, 0x05, 0x85, 0x0f, - 0x1d, 0xa6, 0x70, 0xb2, 0x7e, 0xd1, 0x15, 0x48, 0xb5, 0x3b, 0xb7, 0x9c, 0xf8, 0x92, 0xee, 0x85, - 0x4a, 0x17, 0x9f, 0xe3, 0xec, 0x85, 0x09, 0x9e, 0x4c, 0xb4, 0xbf, 0x41, 0x4a, 0x44, 0x12, 0x14, - 0xd8, 0x80, 0xac, 0x52, 0xfa, 0x99, 0xa2, 0xc6, 0x5c, 0x54, 0x67, 0x3d, 0xd0, 0xc5, 0x81, 0x9c, - 0x8a, 0x54, 0x35, 0x4a, 0x7b, 0xf3, 0x11, 0xf8, 0xa3, 0x54, 0xeb, 0x18, 0x16, 0x87, 0xca, 0x94, - 0x7a, 0xbd, 0x86, 0x19, 0x85, 0x8a, 0x15, 0xad, 0xd2, 0x64, 0x82, 0x60, 0x99, 0x88, 0x9b, 0x61, - 0x0a, 0xf3, 0x01, 0x6b, 0xd8, 0xf6, 0x81, 0x56, 0x6f, 0x0c, 0xad, 0xd6, 0x4c, 0xad, 0x34, 0xb8, - 0x29, 0xd3, 0xf2, 0x40, 0xa6, 0x98, 0x71, 0xc0, 0xdb, 0x80, 0x54, 0x06, 0xf9, 0x05, 0xeb, 0x30, - 0xc5, 0x82, 0x88, 0x2c, 0x7d, 0x5e, 0x7d, 0x61, 0x08, 0xf5, 0xc4, 0x39, 0x3e, 0x84, 0x42, 0xa5, - 0xdb, 0xbd, 0xfd, 0x3a, 0x3c, 0x95, 0x7f, 0x3d, 0x34, 0x3f, 0x2d, 0x58, 0x1c, 0xe2, 0xfc, 0xc7, - 0x49, 0x44, 0x3b, 0x90, 0x62, 0xbc, 0xc9, 0x7b, 0x2c, 0x6c, 0xe8, 0x5c, 0x79, 0x53, 0xcd, 0x19, - 0xf1, 0x36, 0xe7, 0x28, 0xcc, 0xf0, 0x64, 0x26, 0x76, 0x21, 0x25, 0x22, 0x68, 0x16, 0xd2, 0xf5, - 0x83, 0xb3, 0xea, 0xbb, 0x4a, 0x7d, 0xbf, 0x96, 0xfd, 0x0f, 0x65, 0x60, 0xba, 0xea, 0xd5, 0x2a, - 0xc7, 0xb5, 0xdd, 0xac, 0x85, 0xd2, 0x30, 0x55, 0xf3, 0xbc, 0x03, 0x2f, 0x3b, 0x81, 0x97, 0xc1, - 0x16, 0x86, 0x62, 0x3c, 0x58, 0x07, 0x72, 0x72, 0xa4, 0x40, 0x78, 0x0b, 0x96, 0x62, 0x4f, 0xe5, - 0xa7, 0x2a, 0x3e, 0x15, 0xfd, 0x1a, 0xf8, 0x74, 0x1b, 0xd0, 0x49, 0xf7, 0xa2, 0xc9, 0x89, 0xe8, - 0x84, 0xd4, 0x5b, 0x69, 0x99, 0x95, 0xd8, 0xb2, 0x1f, 0x16, 0xe4, 0xb4, 0xfc, 0xe1, 0x9e, 0x27, - 0x12, 0xa0, 0xb7, 0x86, 0x96, 0xcf, 0x55, 0x64, 0x0c, 0xb3, 0xa9, 0xe3, 0x5a, 0x82, 0x8e, 0x27, - 0x8d, 0x5d, 0xa1, 0x23, 0xde, 0x84, 0x85, 0xaa, 0x4f, 0x9a, 0x9c, 0x34, 0x84, 0x95, 0xfb, 0xdf, - 0x19, 0x37, 0xc4, 0x8b, 0x90, 0x37, 0xb0, 0xe2, 0xcd, 0xf8, 0x25, 0xe4, 0x2b, 0x7e, 0xeb, 0xaa, - 0xf3, 0xf0, 0x27, 0x2c, 0x45, 0x28, 0x98, 0x60, 0x49, 0x93, 0x87, 0x5c, 0x60, 0x12, 0x19, 0xee, - 0x1b, 0x0d, 0x97, 0x61, 0x41, 0x0f, 0x4b, 0x25, 0x6d, 0x78, 0x24, 0xf7, 0x8f, 0xf0, 0x7e, 0xda, - 0x1b, 0x3c, 0x97, 0xbf, 0xa7, 0x20, 0xa3, 0xfc, 0x37, 0xa0, 0x36, 0xe4, 0x62, 0xa6, 0x00, 0x69, - 0x9a, 0x8e, 0x1e, 0x22, 0x7b, 0x7d, 0x2c, 0x4e, 0xd6, 0x74, 0x0c, 0xb3, 0xda, 0x72, 0x47, 0xa5, - 0xe1, 0x4c, 0xdd, 0xc1, 0xf6, 0xd3, 0x04, 0x84, 0x64, 0x3d, 0x85, 0xc7, 0xc6, 0x12, 0x44, 0x78, - 0xfc, 0x22, 0xb7, 0x9f, 0x25, 0x62, 0x24, 0xf7, 0x7b, 0x80, 0x68, 0x33, 0xa1, 0x27, 0x89, 0x3b, - 0xcf, 0x5e, 0x19, 0x75, 0x1c, 0x15, 0x6a, 0xb8, 0x5c, 0x2f, 0x34, 0x7e, 0x89, 0xe9, 0x85, 0x8e, - 0x5a, 0x4a, 0x75, 0xc8, 0x28, 0x53, 0x8f, 0x56, 0x46, 0xda, 0x41, 0x70, 0xae, 0x8e, 0xb1, 0x4b, - 0xd0, 0x2a, 0x6d, 0x9a, 0xf5, 0x56, 0xc5, 0x99, 0x42, 0x6f, 0x55, 0xac, 0x15, 0xd0, 0x27, 0x98, - 0xd3, 0xa7, 0x1b, 0x69, 0x49, 0xb1, 0x36, 0xb1, 0x71, 0x12, 0x44, 0x12, 0x1f, 0xc2, 0x8c, 0xea, - 0x02, 0xb4, 0x6a, 0xb6, 0xc2, 0xb0, 0x8d, 0x5d, 0x1a, 0x0d, 0x10, 0x94, 0x3b, 0x07, 0xa0, 0x5c, - 0x96, 0x76, 0xb2, 0x8a, 0x5f, 0x1a, 0xc1, 0x4d, 0xe8, 0xd4, 0xbd, 0xec, 0xf0, 0xab, 0xde, 0xb9, - 0xd3, 0xa2, 0x77, 0xee, 0x25, 0xbd, 0x26, 0x37, 0xae, 0xb8, 0x32, 0xb1, 0x8b, 0x1b, 0xf7, 0x92, - 0xba, 0xe1, 0x75, 0x89, 0xb9, 0xd1, 0x35, 0xea, 0x3c, 0x15, 0x86, 0x5e, 0xfd, 0x0e, 0x00, 0x00, - 0xff, 0xff, 0xe2, 0x7d, 0x9e, 0xca, 0xa2, 0x09, 0x00, 0x00, -} - -// Reference imports to suppress errors if they are not otherwise used. -var _ context.Context -var _ grpc.ClientConn - -// This is a compile-time assertion to ensure that this generated file -// is compatible with the grpc package it is being compiled against. -const _ = grpc.SupportPackageIsVersion4 - -// CoreServiceClient is the client API for CoreService service. -// -// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://godoc.org/google.golang.org/grpc#ClientConn.NewStream. -type CoreServiceClient interface { - // Retrieve version information about this Feast deployment - GetFeastCoreVersion(ctx context.Context, in *GetFeastCoreVersionRequest, opts ...grpc.CallOption) (*GetFeastCoreVersionResponse, error) - // Returns a specific feature set - GetFeatureSet(ctx context.Context, in *GetFeatureSetRequest, opts ...grpc.CallOption) (*GetFeatureSetResponse, error) - // Retrieve feature set details given a filter. - // - // Returns all feature sets matching that filter. If none are found, - // an empty list will be returned. - // If no filter is provided in the request, the response will contain all the feature - // sets currently stored in the registry. - ListFeatureSets(ctx context.Context, in *ListFeatureSetsRequest, opts ...grpc.CallOption) (*ListFeatureSetsResponse, error) - // Retrieve store details given a filter. - // - // Returns all stores matching that filter. If none are found, an empty list will be returned. - // If no filter is provided in the request, the response will contain all the stores currently - // stored in the registry. - ListStores(ctx context.Context, in *ListStoresRequest, opts ...grpc.CallOption) (*ListStoresResponse, error) - // Create or update and existing feature set. - // - // This function is idempotent - it will not create a new feature set if schema does not change. - // If an existing feature set is updated, core will advance the version number, which will be - // returned in response. - ApplyFeatureSet(ctx context.Context, in *ApplyFeatureSetRequest, opts ...grpc.CallOption) (*ApplyFeatureSetResponse, error) - // Updates core with the configuration of the store. - // - // If the changes are valid, core will return the given store configuration in response, and - // start or update the necessary feature population jobs for the updated store. - UpdateStore(ctx context.Context, in *UpdateStoreRequest, opts ...grpc.CallOption) (*UpdateStoreResponse, error) - // Creates a project. Projects serve as namespaces within which resources like features will be - // created. Both feature set names as well as field names must be unique within a project. Project - // names themselves must be globally unique. - CreateProject(ctx context.Context, in *CreateProjectRequest, opts ...grpc.CallOption) (*CreateProjectResponse, error) - // Archives a project. Archived projects will continue to exist and function, but won't be visible - // through the Core API. Any existing ingestion or serving requests will continue to function, - // but will result in warning messages being logged. It is not possible to unarchive a project - // through the Core API - ArchiveProject(ctx context.Context, in *ArchiveProjectRequest, opts ...grpc.CallOption) (*ArchiveProjectResponse, error) - // Lists all projects active projects. - ListProjects(ctx context.Context, in *ListProjectsRequest, opts ...grpc.CallOption) (*ListProjectsResponse, error) -} - -type coreServiceClient struct { - cc *grpc.ClientConn -} - -func NewCoreServiceClient(cc *grpc.ClientConn) CoreServiceClient { - return &coreServiceClient{cc} -} - -func (c *coreServiceClient) GetFeastCoreVersion(ctx context.Context, in *GetFeastCoreVersionRequest, opts ...grpc.CallOption) (*GetFeastCoreVersionResponse, error) { - out := new(GetFeastCoreVersionResponse) - err := c.cc.Invoke(ctx, "/feast.core.CoreService/GetFeastCoreVersion", in, out, opts...) - if err != nil { - return nil, err - } - return out, nil -} - -func (c *coreServiceClient) GetFeatureSet(ctx context.Context, in *GetFeatureSetRequest, opts ...grpc.CallOption) (*GetFeatureSetResponse, error) { - out := new(GetFeatureSetResponse) - err := c.cc.Invoke(ctx, "/feast.core.CoreService/GetFeatureSet", in, out, opts...) - if err != nil { - return nil, err - } - return out, nil -} - -func (c *coreServiceClient) ListFeatureSets(ctx context.Context, in *ListFeatureSetsRequest, opts ...grpc.CallOption) (*ListFeatureSetsResponse, error) { - out := new(ListFeatureSetsResponse) - err := c.cc.Invoke(ctx, "/feast.core.CoreService/ListFeatureSets", in, out, opts...) - if err != nil { - return nil, err - } - return out, nil -} - -func (c *coreServiceClient) ListStores(ctx context.Context, in *ListStoresRequest, opts ...grpc.CallOption) (*ListStoresResponse, error) { - out := new(ListStoresResponse) - err := c.cc.Invoke(ctx, "/feast.core.CoreService/ListStores", in, out, opts...) - if err != nil { - return nil, err - } - return out, nil -} - -func (c *coreServiceClient) ApplyFeatureSet(ctx context.Context, in *ApplyFeatureSetRequest, opts ...grpc.CallOption) (*ApplyFeatureSetResponse, error) { - out := new(ApplyFeatureSetResponse) - err := c.cc.Invoke(ctx, "/feast.core.CoreService/ApplyFeatureSet", in, out, opts...) - if err != nil { - return nil, err - } - return out, nil -} - -func (c *coreServiceClient) UpdateStore(ctx context.Context, in *UpdateStoreRequest, opts ...grpc.CallOption) (*UpdateStoreResponse, error) { - out := new(UpdateStoreResponse) - err := c.cc.Invoke(ctx, "/feast.core.CoreService/UpdateStore", in, out, opts...) - if err != nil { - return nil, err - } - return out, nil -} - -func (c *coreServiceClient) CreateProject(ctx context.Context, in *CreateProjectRequest, opts ...grpc.CallOption) (*CreateProjectResponse, error) { - out := new(CreateProjectResponse) - err := c.cc.Invoke(ctx, "/feast.core.CoreService/CreateProject", in, out, opts...) - if err != nil { - return nil, err - } - return out, nil -} - -func (c *coreServiceClient) ArchiveProject(ctx context.Context, in *ArchiveProjectRequest, opts ...grpc.CallOption) (*ArchiveProjectResponse, error) { - out := new(ArchiveProjectResponse) - err := c.cc.Invoke(ctx, "/feast.core.CoreService/ArchiveProject", in, out, opts...) - if err != nil { - return nil, err - } - return out, nil -} - -func (c *coreServiceClient) ListProjects(ctx context.Context, in *ListProjectsRequest, opts ...grpc.CallOption) (*ListProjectsResponse, error) { - out := new(ListProjectsResponse) - err := c.cc.Invoke(ctx, "/feast.core.CoreService/ListProjects", in, out, opts...) - if err != nil { - return nil, err - } - return out, nil -} - -// CoreServiceServer is the server API for CoreService service. -type CoreServiceServer interface { - // Retrieve version information about this Feast deployment - GetFeastCoreVersion(context.Context, *GetFeastCoreVersionRequest) (*GetFeastCoreVersionResponse, error) - // Returns a specific feature set - GetFeatureSet(context.Context, *GetFeatureSetRequest) (*GetFeatureSetResponse, error) - // Retrieve feature set details given a filter. - // - // Returns all feature sets matching that filter. If none are found, - // an empty list will be returned. - // If no filter is provided in the request, the response will contain all the feature - // sets currently stored in the registry. - ListFeatureSets(context.Context, *ListFeatureSetsRequest) (*ListFeatureSetsResponse, error) - // Retrieve store details given a filter. - // - // Returns all stores matching that filter. If none are found, an empty list will be returned. - // If no filter is provided in the request, the response will contain all the stores currently - // stored in the registry. - ListStores(context.Context, *ListStoresRequest) (*ListStoresResponse, error) - // Create or update and existing feature set. - // - // This function is idempotent - it will not create a new feature set if schema does not change. - // If an existing feature set is updated, core will advance the version number, which will be - // returned in response. - ApplyFeatureSet(context.Context, *ApplyFeatureSetRequest) (*ApplyFeatureSetResponse, error) - // Updates core with the configuration of the store. - // - // If the changes are valid, core will return the given store configuration in response, and - // start or update the necessary feature population jobs for the updated store. - UpdateStore(context.Context, *UpdateStoreRequest) (*UpdateStoreResponse, error) - // Creates a project. Projects serve as namespaces within which resources like features will be - // created. Both feature set names as well as field names must be unique within a project. Project - // names themselves must be globally unique. - CreateProject(context.Context, *CreateProjectRequest) (*CreateProjectResponse, error) - // Archives a project. Archived projects will continue to exist and function, but won't be visible - // through the Core API. Any existing ingestion or serving requests will continue to function, - // but will result in warning messages being logged. It is not possible to unarchive a project - // through the Core API - ArchiveProject(context.Context, *ArchiveProjectRequest) (*ArchiveProjectResponse, error) - // Lists all projects active projects. - ListProjects(context.Context, *ListProjectsRequest) (*ListProjectsResponse, error) -} - -// UnimplementedCoreServiceServer can be embedded to have forward compatible implementations. -type UnimplementedCoreServiceServer struct { -} - -func (*UnimplementedCoreServiceServer) GetFeastCoreVersion(ctx context.Context, req *GetFeastCoreVersionRequest) (*GetFeastCoreVersionResponse, error) { - return nil, status.Errorf(codes.Unimplemented, "method GetFeastCoreVersion not implemented") -} -func (*UnimplementedCoreServiceServer) GetFeatureSet(ctx context.Context, req *GetFeatureSetRequest) (*GetFeatureSetResponse, error) { - return nil, status.Errorf(codes.Unimplemented, "method GetFeatureSet not implemented") -} -func (*UnimplementedCoreServiceServer) ListFeatureSets(ctx context.Context, req *ListFeatureSetsRequest) (*ListFeatureSetsResponse, error) { - return nil, status.Errorf(codes.Unimplemented, "method ListFeatureSets not implemented") -} -func (*UnimplementedCoreServiceServer) ListStores(ctx context.Context, req *ListStoresRequest) (*ListStoresResponse, error) { - return nil, status.Errorf(codes.Unimplemented, "method ListStores not implemented") -} -func (*UnimplementedCoreServiceServer) ApplyFeatureSet(ctx context.Context, req *ApplyFeatureSetRequest) (*ApplyFeatureSetResponse, error) { - return nil, status.Errorf(codes.Unimplemented, "method ApplyFeatureSet not implemented") -} -func (*UnimplementedCoreServiceServer) UpdateStore(ctx context.Context, req *UpdateStoreRequest) (*UpdateStoreResponse, error) { - return nil, status.Errorf(codes.Unimplemented, "method UpdateStore not implemented") -} -func (*UnimplementedCoreServiceServer) CreateProject(ctx context.Context, req *CreateProjectRequest) (*CreateProjectResponse, error) { - return nil, status.Errorf(codes.Unimplemented, "method CreateProject not implemented") -} -func (*UnimplementedCoreServiceServer) ArchiveProject(ctx context.Context, req *ArchiveProjectRequest) (*ArchiveProjectResponse, error) { - return nil, status.Errorf(codes.Unimplemented, "method ArchiveProject not implemented") -} -func (*UnimplementedCoreServiceServer) ListProjects(ctx context.Context, req *ListProjectsRequest) (*ListProjectsResponse, error) { - return nil, status.Errorf(codes.Unimplemented, "method ListProjects not implemented") -} - -func RegisterCoreServiceServer(s *grpc.Server, srv CoreServiceServer) { - s.RegisterService(&_CoreService_serviceDesc, srv) -} - -func _CoreService_GetFeastCoreVersion_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(GetFeastCoreVersionRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(CoreServiceServer).GetFeastCoreVersion(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: "/feast.core.CoreService/GetFeastCoreVersion", - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(CoreServiceServer).GetFeastCoreVersion(ctx, req.(*GetFeastCoreVersionRequest)) - } - return interceptor(ctx, in, info, handler) -} - -func _CoreService_GetFeatureSet_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(GetFeatureSetRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(CoreServiceServer).GetFeatureSet(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: "/feast.core.CoreService/GetFeatureSet", - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(CoreServiceServer).GetFeatureSet(ctx, req.(*GetFeatureSetRequest)) - } - return interceptor(ctx, in, info, handler) -} - -func _CoreService_ListFeatureSets_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(ListFeatureSetsRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(CoreServiceServer).ListFeatureSets(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: "/feast.core.CoreService/ListFeatureSets", - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(CoreServiceServer).ListFeatureSets(ctx, req.(*ListFeatureSetsRequest)) - } - return interceptor(ctx, in, info, handler) -} - -func _CoreService_ListStores_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(ListStoresRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(CoreServiceServer).ListStores(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: "/feast.core.CoreService/ListStores", - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(CoreServiceServer).ListStores(ctx, req.(*ListStoresRequest)) - } - return interceptor(ctx, in, info, handler) -} - -func _CoreService_ApplyFeatureSet_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(ApplyFeatureSetRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(CoreServiceServer).ApplyFeatureSet(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: "/feast.core.CoreService/ApplyFeatureSet", - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(CoreServiceServer).ApplyFeatureSet(ctx, req.(*ApplyFeatureSetRequest)) - } - return interceptor(ctx, in, info, handler) -} - -func _CoreService_UpdateStore_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(UpdateStoreRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(CoreServiceServer).UpdateStore(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: "/feast.core.CoreService/UpdateStore", - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(CoreServiceServer).UpdateStore(ctx, req.(*UpdateStoreRequest)) - } - return interceptor(ctx, in, info, handler) -} - -func _CoreService_CreateProject_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(CreateProjectRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(CoreServiceServer).CreateProject(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: "/feast.core.CoreService/CreateProject", - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(CoreServiceServer).CreateProject(ctx, req.(*CreateProjectRequest)) - } - return interceptor(ctx, in, info, handler) -} - -func _CoreService_ArchiveProject_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(ArchiveProjectRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(CoreServiceServer).ArchiveProject(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: "/feast.core.CoreService/ArchiveProject", - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(CoreServiceServer).ArchiveProject(ctx, req.(*ArchiveProjectRequest)) - } - return interceptor(ctx, in, info, handler) -} - -func _CoreService_ListProjects_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { - in := new(ListProjectsRequest) - if err := dec(in); err != nil { - return nil, err - } - if interceptor == nil { - return srv.(CoreServiceServer).ListProjects(ctx, in) - } - info := &grpc.UnaryServerInfo{ - Server: srv, - FullMethod: "/feast.core.CoreService/ListProjects", - } - handler := func(ctx context.Context, req interface{}) (interface{}, error) { - return srv.(CoreServiceServer).ListProjects(ctx, req.(*ListProjectsRequest)) - } - return interceptor(ctx, in, info, handler) -} - -var _CoreService_serviceDesc = grpc.ServiceDesc{ - ServiceName: "feast.core.CoreService", - HandlerType: (*CoreServiceServer)(nil), - Methods: []grpc.MethodDesc{ - { - MethodName: "GetFeastCoreVersion", - Handler: _CoreService_GetFeastCoreVersion_Handler, - }, - { - MethodName: "GetFeatureSet", - Handler: _CoreService_GetFeatureSet_Handler, - }, - { - MethodName: "ListFeatureSets", - Handler: _CoreService_ListFeatureSets_Handler, - }, - { - MethodName: "ListStores", - Handler: _CoreService_ListStores_Handler, - }, - { - MethodName: "ApplyFeatureSet", - Handler: _CoreService_ApplyFeatureSet_Handler, - }, - { - MethodName: "UpdateStore", - Handler: _CoreService_UpdateStore_Handler, - }, - { - MethodName: "CreateProject", - Handler: _CoreService_CreateProject_Handler, - }, - { - MethodName: "ArchiveProject", - Handler: _CoreService_ArchiveProject_Handler, - }, - { - MethodName: "ListProjects", - Handler: _CoreService_ListProjects_Handler, - }, - }, - Streams: []grpc.StreamDesc{}, - Metadata: "feast/core/CoreService.proto", -} diff --git a/sdk/go/protos/feast/core/FeatureSet.pb.go b/sdk/go/protos/feast/core/FeatureSet.pb.go deleted file mode 100644 index 5f488caee6e..00000000000 --- a/sdk/go/protos/feast/core/FeatureSet.pb.go +++ /dev/null @@ -1,1008 +0,0 @@ -// Code generated by protoc-gen-go. DO NOT EDIT. -// source: feast/core/FeatureSet.proto - -package core - -import ( - fmt "fmt" - types "github.com/gojek/feast/sdk/go/protos/feast/types" - proto "github.com/golang/protobuf/proto" - duration "github.com/golang/protobuf/ptypes/duration" - timestamp "github.com/golang/protobuf/ptypes/timestamp" - math "math" - v0 "tensorflow_metadata/proto/v0" -) - -// Reference imports to suppress errors if they are not otherwise used. -var _ = proto.Marshal -var _ = fmt.Errorf -var _ = math.Inf - -// This is a compile-time assertion to ensure that this generated file -// is compatible with the proto package it is being compiled against. -// A compilation error at this line likely means your copy of the -// proto package needs to be updated. -const _ = proto.ProtoPackageIsVersion3 // please upgrade the proto package - -type FeatureSetStatus int32 - -const ( - FeatureSetStatus_STATUS_INVALID FeatureSetStatus = 0 - FeatureSetStatus_STATUS_PENDING FeatureSetStatus = 1 - FeatureSetStatus_STATUS_READY FeatureSetStatus = 2 -) - -var FeatureSetStatus_name = map[int32]string{ - 0: "STATUS_INVALID", - 1: "STATUS_PENDING", - 2: "STATUS_READY", -} - -var FeatureSetStatus_value = map[string]int32{ - "STATUS_INVALID": 0, - "STATUS_PENDING": 1, - "STATUS_READY": 2, -} - -func (x FeatureSetStatus) String() string { - return proto.EnumName(FeatureSetStatus_name, int32(x)) -} - -func (FeatureSetStatus) EnumDescriptor() ([]byte, []int) { - return fileDescriptor_972fbd278ac19c0c, []int{0} -} - -type FeatureSet struct { - // User-specified specifications of this feature set. - Spec *FeatureSetSpec `protobuf:"bytes,1,opt,name=spec,proto3" json:"spec,omitempty"` - // System-populated metadata for this feature set. - Meta *FeatureSetMeta `protobuf:"bytes,2,opt,name=meta,proto3" json:"meta,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *FeatureSet) Reset() { *m = FeatureSet{} } -func (m *FeatureSet) String() string { return proto.CompactTextString(m) } -func (*FeatureSet) ProtoMessage() {} -func (*FeatureSet) Descriptor() ([]byte, []int) { - return fileDescriptor_972fbd278ac19c0c, []int{0} -} - -func (m *FeatureSet) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_FeatureSet.Unmarshal(m, b) -} -func (m *FeatureSet) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_FeatureSet.Marshal(b, m, deterministic) -} -func (m *FeatureSet) XXX_Merge(src proto.Message) { - xxx_messageInfo_FeatureSet.Merge(m, src) -} -func (m *FeatureSet) XXX_Size() int { - return xxx_messageInfo_FeatureSet.Size(m) -} -func (m *FeatureSet) XXX_DiscardUnknown() { - xxx_messageInfo_FeatureSet.DiscardUnknown(m) -} - -var xxx_messageInfo_FeatureSet proto.InternalMessageInfo - -func (m *FeatureSet) GetSpec() *FeatureSetSpec { - if m != nil { - return m.Spec - } - return nil -} - -func (m *FeatureSet) GetMeta() *FeatureSetMeta { - if m != nil { - return m.Meta - } - return nil -} - -type FeatureSetSpec struct { - // Name of project that this feature set belongs to. - Project string `protobuf:"bytes,7,opt,name=project,proto3" json:"project,omitempty"` - // Name of the feature set. Must be unique. - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - // Feature set version. - Version int32 `protobuf:"varint,2,opt,name=version,proto3" json:"version,omitempty"` - // List of entities contained within this featureSet. - // This allows the feature to be used during joins between feature sets. - // If the featureSet is ingested into a store that supports keys, this value - // will be made a key. - Entities []*EntitySpec `protobuf:"bytes,3,rep,name=entities,proto3" json:"entities,omitempty"` - // List of features contained within this featureSet. - Features []*FeatureSpec `protobuf:"bytes,4,rep,name=features,proto3" json:"features,omitempty"` - // Features in this feature set will only be retrieved if they are found - // after [time - max_age]. Missing or older feature values will be returned - // as nulls and indicated to end user - MaxAge *duration.Duration `protobuf:"bytes,5,opt,name=max_age,json=maxAge,proto3" json:"max_age,omitempty"` - // Optional. Source on which feature rows can be found. - // If not set, source will be set to the default value configured in Feast Core. - Source *Source `protobuf:"bytes,6,opt,name=source,proto3" json:"source,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *FeatureSetSpec) Reset() { *m = FeatureSetSpec{} } -func (m *FeatureSetSpec) String() string { return proto.CompactTextString(m) } -func (*FeatureSetSpec) ProtoMessage() {} -func (*FeatureSetSpec) Descriptor() ([]byte, []int) { - return fileDescriptor_972fbd278ac19c0c, []int{1} -} - -func (m *FeatureSetSpec) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_FeatureSetSpec.Unmarshal(m, b) -} -func (m *FeatureSetSpec) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_FeatureSetSpec.Marshal(b, m, deterministic) -} -func (m *FeatureSetSpec) XXX_Merge(src proto.Message) { - xxx_messageInfo_FeatureSetSpec.Merge(m, src) -} -func (m *FeatureSetSpec) XXX_Size() int { - return xxx_messageInfo_FeatureSetSpec.Size(m) -} -func (m *FeatureSetSpec) XXX_DiscardUnknown() { - xxx_messageInfo_FeatureSetSpec.DiscardUnknown(m) -} - -var xxx_messageInfo_FeatureSetSpec proto.InternalMessageInfo - -func (m *FeatureSetSpec) GetProject() string { - if m != nil { - return m.Project - } - return "" -} - -func (m *FeatureSetSpec) GetName() string { - if m != nil { - return m.Name - } - return "" -} - -func (m *FeatureSetSpec) GetVersion() int32 { - if m != nil { - return m.Version - } - return 0 -} - -func (m *FeatureSetSpec) GetEntities() []*EntitySpec { - if m != nil { - return m.Entities - } - return nil -} - -func (m *FeatureSetSpec) GetFeatures() []*FeatureSpec { - if m != nil { - return m.Features - } - return nil -} - -func (m *FeatureSetSpec) GetMaxAge() *duration.Duration { - if m != nil { - return m.MaxAge - } - return nil -} - -func (m *FeatureSetSpec) GetSource() *Source { - if m != nil { - return m.Source - } - return nil -} - -type EntitySpec struct { - // Name of the entity. - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - // Value type of the feature. - ValueType types.ValueType_Enum `protobuf:"varint,2,opt,name=value_type,json=valueType,proto3,enum=feast.types.ValueType_Enum" json:"value_type,omitempty"` - // Types that are valid to be assigned to PresenceConstraints: - // *EntitySpec_Presence - // *EntitySpec_GroupPresence - PresenceConstraints isEntitySpec_PresenceConstraints `protobuf_oneof:"presence_constraints"` - // The shape of the feature which governs the number of values that appear in - // each example. - // - // Types that are valid to be assigned to ShapeType: - // *EntitySpec_Shape - // *EntitySpec_ValueCount - ShapeType isEntitySpec_ShapeType `protobuf_oneof:"shape_type"` - // Domain for the values of the feature. - // - // Types that are valid to be assigned to DomainInfo: - // *EntitySpec_Domain - // *EntitySpec_IntDomain - // *EntitySpec_FloatDomain - // *EntitySpec_StringDomain - // *EntitySpec_BoolDomain - // *EntitySpec_StructDomain - // *EntitySpec_NaturalLanguageDomain - // *EntitySpec_ImageDomain - // *EntitySpec_MidDomain - // *EntitySpec_UrlDomain - // *EntitySpec_TimeDomain - // *EntitySpec_TimeOfDayDomain - DomainInfo isEntitySpec_DomainInfo `protobuf_oneof:"domain_info"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *EntitySpec) Reset() { *m = EntitySpec{} } -func (m *EntitySpec) String() string { return proto.CompactTextString(m) } -func (*EntitySpec) ProtoMessage() {} -func (*EntitySpec) Descriptor() ([]byte, []int) { - return fileDescriptor_972fbd278ac19c0c, []int{2} -} - -func (m *EntitySpec) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_EntitySpec.Unmarshal(m, b) -} -func (m *EntitySpec) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_EntitySpec.Marshal(b, m, deterministic) -} -func (m *EntitySpec) XXX_Merge(src proto.Message) { - xxx_messageInfo_EntitySpec.Merge(m, src) -} -func (m *EntitySpec) XXX_Size() int { - return xxx_messageInfo_EntitySpec.Size(m) -} -func (m *EntitySpec) XXX_DiscardUnknown() { - xxx_messageInfo_EntitySpec.DiscardUnknown(m) -} - -var xxx_messageInfo_EntitySpec proto.InternalMessageInfo - -func (m *EntitySpec) GetName() string { - if m != nil { - return m.Name - } - return "" -} - -func (m *EntitySpec) GetValueType() types.ValueType_Enum { - if m != nil { - return m.ValueType - } - return types.ValueType_INVALID -} - -type isEntitySpec_PresenceConstraints interface { - isEntitySpec_PresenceConstraints() -} - -type EntitySpec_Presence struct { - Presence *v0.FeaturePresence `protobuf:"bytes,3,opt,name=presence,proto3,oneof"` -} - -type EntitySpec_GroupPresence struct { - GroupPresence *v0.FeaturePresenceWithinGroup `protobuf:"bytes,4,opt,name=group_presence,json=groupPresence,proto3,oneof"` -} - -func (*EntitySpec_Presence) isEntitySpec_PresenceConstraints() {} - -func (*EntitySpec_GroupPresence) isEntitySpec_PresenceConstraints() {} - -func (m *EntitySpec) GetPresenceConstraints() isEntitySpec_PresenceConstraints { - if m != nil { - return m.PresenceConstraints - } - return nil -} - -func (m *EntitySpec) GetPresence() *v0.FeaturePresence { - if x, ok := m.GetPresenceConstraints().(*EntitySpec_Presence); ok { - return x.Presence - } - return nil -} - -func (m *EntitySpec) GetGroupPresence() *v0.FeaturePresenceWithinGroup { - if x, ok := m.GetPresenceConstraints().(*EntitySpec_GroupPresence); ok { - return x.GroupPresence - } - return nil -} - -type isEntitySpec_ShapeType interface { - isEntitySpec_ShapeType() -} - -type EntitySpec_Shape struct { - Shape *v0.FixedShape `protobuf:"bytes,5,opt,name=shape,proto3,oneof"` -} - -type EntitySpec_ValueCount struct { - ValueCount *v0.ValueCount `protobuf:"bytes,6,opt,name=value_count,json=valueCount,proto3,oneof"` -} - -func (*EntitySpec_Shape) isEntitySpec_ShapeType() {} - -func (*EntitySpec_ValueCount) isEntitySpec_ShapeType() {} - -func (m *EntitySpec) GetShapeType() isEntitySpec_ShapeType { - if m != nil { - return m.ShapeType - } - return nil -} - -func (m *EntitySpec) GetShape() *v0.FixedShape { - if x, ok := m.GetShapeType().(*EntitySpec_Shape); ok { - return x.Shape - } - return nil -} - -func (m *EntitySpec) GetValueCount() *v0.ValueCount { - if x, ok := m.GetShapeType().(*EntitySpec_ValueCount); ok { - return x.ValueCount - } - return nil -} - -type isEntitySpec_DomainInfo interface { - isEntitySpec_DomainInfo() -} - -type EntitySpec_Domain struct { - Domain string `protobuf:"bytes,7,opt,name=domain,proto3,oneof"` -} - -type EntitySpec_IntDomain struct { - IntDomain *v0.IntDomain `protobuf:"bytes,8,opt,name=int_domain,json=intDomain,proto3,oneof"` -} - -type EntitySpec_FloatDomain struct { - FloatDomain *v0.FloatDomain `protobuf:"bytes,9,opt,name=float_domain,json=floatDomain,proto3,oneof"` -} - -type EntitySpec_StringDomain struct { - StringDomain *v0.StringDomain `protobuf:"bytes,10,opt,name=string_domain,json=stringDomain,proto3,oneof"` -} - -type EntitySpec_BoolDomain struct { - BoolDomain *v0.BoolDomain `protobuf:"bytes,11,opt,name=bool_domain,json=boolDomain,proto3,oneof"` -} - -type EntitySpec_StructDomain struct { - StructDomain *v0.StructDomain `protobuf:"bytes,12,opt,name=struct_domain,json=structDomain,proto3,oneof"` -} - -type EntitySpec_NaturalLanguageDomain struct { - NaturalLanguageDomain *v0.NaturalLanguageDomain `protobuf:"bytes,13,opt,name=natural_language_domain,json=naturalLanguageDomain,proto3,oneof"` -} - -type EntitySpec_ImageDomain struct { - ImageDomain *v0.ImageDomain `protobuf:"bytes,14,opt,name=image_domain,json=imageDomain,proto3,oneof"` -} - -type EntitySpec_MidDomain struct { - MidDomain *v0.MIDDomain `protobuf:"bytes,15,opt,name=mid_domain,json=midDomain,proto3,oneof"` -} - -type EntitySpec_UrlDomain struct { - UrlDomain *v0.URLDomain `protobuf:"bytes,16,opt,name=url_domain,json=urlDomain,proto3,oneof"` -} - -type EntitySpec_TimeDomain struct { - TimeDomain *v0.TimeDomain `protobuf:"bytes,17,opt,name=time_domain,json=timeDomain,proto3,oneof"` -} - -type EntitySpec_TimeOfDayDomain struct { - TimeOfDayDomain *v0.TimeOfDayDomain `protobuf:"bytes,18,opt,name=time_of_day_domain,json=timeOfDayDomain,proto3,oneof"` -} - -func (*EntitySpec_Domain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_IntDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_FloatDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_StringDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_BoolDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_StructDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_NaturalLanguageDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_ImageDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_MidDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_UrlDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_TimeDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_TimeOfDayDomain) isEntitySpec_DomainInfo() {} - -func (m *EntitySpec) GetDomainInfo() isEntitySpec_DomainInfo { - if m != nil { - return m.DomainInfo - } - return nil -} - -func (m *EntitySpec) GetDomain() string { - if x, ok := m.GetDomainInfo().(*EntitySpec_Domain); ok { - return x.Domain - } - return "" -} - -func (m *EntitySpec) GetIntDomain() *v0.IntDomain { - if x, ok := m.GetDomainInfo().(*EntitySpec_IntDomain); ok { - return x.IntDomain - } - return nil -} - -func (m *EntitySpec) GetFloatDomain() *v0.FloatDomain { - if x, ok := m.GetDomainInfo().(*EntitySpec_FloatDomain); ok { - return x.FloatDomain - } - return nil -} - -func (m *EntitySpec) GetStringDomain() *v0.StringDomain { - if x, ok := m.GetDomainInfo().(*EntitySpec_StringDomain); ok { - return x.StringDomain - } - return nil -} - -func (m *EntitySpec) GetBoolDomain() *v0.BoolDomain { - if x, ok := m.GetDomainInfo().(*EntitySpec_BoolDomain); ok { - return x.BoolDomain - } - return nil -} - -func (m *EntitySpec) GetStructDomain() *v0.StructDomain { - if x, ok := m.GetDomainInfo().(*EntitySpec_StructDomain); ok { - return x.StructDomain - } - return nil -} - -func (m *EntitySpec) GetNaturalLanguageDomain() *v0.NaturalLanguageDomain { - if x, ok := m.GetDomainInfo().(*EntitySpec_NaturalLanguageDomain); ok { - return x.NaturalLanguageDomain - } - return nil -} - -func (m *EntitySpec) GetImageDomain() *v0.ImageDomain { - if x, ok := m.GetDomainInfo().(*EntitySpec_ImageDomain); ok { - return x.ImageDomain - } - return nil -} - -func (m *EntitySpec) GetMidDomain() *v0.MIDDomain { - if x, ok := m.GetDomainInfo().(*EntitySpec_MidDomain); ok { - return x.MidDomain - } - return nil -} - -func (m *EntitySpec) GetUrlDomain() *v0.URLDomain { - if x, ok := m.GetDomainInfo().(*EntitySpec_UrlDomain); ok { - return x.UrlDomain - } - return nil -} - -func (m *EntitySpec) GetTimeDomain() *v0.TimeDomain { - if x, ok := m.GetDomainInfo().(*EntitySpec_TimeDomain); ok { - return x.TimeDomain - } - return nil -} - -func (m *EntitySpec) GetTimeOfDayDomain() *v0.TimeOfDayDomain { - if x, ok := m.GetDomainInfo().(*EntitySpec_TimeOfDayDomain); ok { - return x.TimeOfDayDomain - } - return nil -} - -// XXX_OneofWrappers is for the internal use of the proto package. -func (*EntitySpec) XXX_OneofWrappers() []interface{} { - return []interface{}{ - (*EntitySpec_Presence)(nil), - (*EntitySpec_GroupPresence)(nil), - (*EntitySpec_Shape)(nil), - (*EntitySpec_ValueCount)(nil), - (*EntitySpec_Domain)(nil), - (*EntitySpec_IntDomain)(nil), - (*EntitySpec_FloatDomain)(nil), - (*EntitySpec_StringDomain)(nil), - (*EntitySpec_BoolDomain)(nil), - (*EntitySpec_StructDomain)(nil), - (*EntitySpec_NaturalLanguageDomain)(nil), - (*EntitySpec_ImageDomain)(nil), - (*EntitySpec_MidDomain)(nil), - (*EntitySpec_UrlDomain)(nil), - (*EntitySpec_TimeDomain)(nil), - (*EntitySpec_TimeOfDayDomain)(nil), - } -} - -type FeatureSpec struct { - // Name of the feature. - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - // Value type of the feature. - ValueType types.ValueType_Enum `protobuf:"varint,2,opt,name=value_type,json=valueType,proto3,enum=feast.types.ValueType_Enum" json:"value_type,omitempty"` - // Types that are valid to be assigned to PresenceConstraints: - // *FeatureSpec_Presence - // *FeatureSpec_GroupPresence - PresenceConstraints isFeatureSpec_PresenceConstraints `protobuf_oneof:"presence_constraints"` - // The shape of the feature which governs the number of values that appear in - // each example. - // - // Types that are valid to be assigned to ShapeType: - // *FeatureSpec_Shape - // *FeatureSpec_ValueCount - ShapeType isFeatureSpec_ShapeType `protobuf_oneof:"shape_type"` - // Domain for the values of the feature. - // - // Types that are valid to be assigned to DomainInfo: - // *FeatureSpec_Domain - // *FeatureSpec_IntDomain - // *FeatureSpec_FloatDomain - // *FeatureSpec_StringDomain - // *FeatureSpec_BoolDomain - // *FeatureSpec_StructDomain - // *FeatureSpec_NaturalLanguageDomain - // *FeatureSpec_ImageDomain - // *FeatureSpec_MidDomain - // *FeatureSpec_UrlDomain - // *FeatureSpec_TimeDomain - // *FeatureSpec_TimeOfDayDomain - DomainInfo isFeatureSpec_DomainInfo `protobuf_oneof:"domain_info"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *FeatureSpec) Reset() { *m = FeatureSpec{} } -func (m *FeatureSpec) String() string { return proto.CompactTextString(m) } -func (*FeatureSpec) ProtoMessage() {} -func (*FeatureSpec) Descriptor() ([]byte, []int) { - return fileDescriptor_972fbd278ac19c0c, []int{3} -} - -func (m *FeatureSpec) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_FeatureSpec.Unmarshal(m, b) -} -func (m *FeatureSpec) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_FeatureSpec.Marshal(b, m, deterministic) -} -func (m *FeatureSpec) XXX_Merge(src proto.Message) { - xxx_messageInfo_FeatureSpec.Merge(m, src) -} -func (m *FeatureSpec) XXX_Size() int { - return xxx_messageInfo_FeatureSpec.Size(m) -} -func (m *FeatureSpec) XXX_DiscardUnknown() { - xxx_messageInfo_FeatureSpec.DiscardUnknown(m) -} - -var xxx_messageInfo_FeatureSpec proto.InternalMessageInfo - -func (m *FeatureSpec) GetName() string { - if m != nil { - return m.Name - } - return "" -} - -func (m *FeatureSpec) GetValueType() types.ValueType_Enum { - if m != nil { - return m.ValueType - } - return types.ValueType_INVALID -} - -type isFeatureSpec_PresenceConstraints interface { - isFeatureSpec_PresenceConstraints() -} - -type FeatureSpec_Presence struct { - Presence *v0.FeaturePresence `protobuf:"bytes,3,opt,name=presence,proto3,oneof"` -} - -type FeatureSpec_GroupPresence struct { - GroupPresence *v0.FeaturePresenceWithinGroup `protobuf:"bytes,4,opt,name=group_presence,json=groupPresence,proto3,oneof"` -} - -func (*FeatureSpec_Presence) isFeatureSpec_PresenceConstraints() {} - -func (*FeatureSpec_GroupPresence) isFeatureSpec_PresenceConstraints() {} - -func (m *FeatureSpec) GetPresenceConstraints() isFeatureSpec_PresenceConstraints { - if m != nil { - return m.PresenceConstraints - } - return nil -} - -func (m *FeatureSpec) GetPresence() *v0.FeaturePresence { - if x, ok := m.GetPresenceConstraints().(*FeatureSpec_Presence); ok { - return x.Presence - } - return nil -} - -func (m *FeatureSpec) GetGroupPresence() *v0.FeaturePresenceWithinGroup { - if x, ok := m.GetPresenceConstraints().(*FeatureSpec_GroupPresence); ok { - return x.GroupPresence - } - return nil -} - -type isFeatureSpec_ShapeType interface { - isFeatureSpec_ShapeType() -} - -type FeatureSpec_Shape struct { - Shape *v0.FixedShape `protobuf:"bytes,5,opt,name=shape,proto3,oneof"` -} - -type FeatureSpec_ValueCount struct { - ValueCount *v0.ValueCount `protobuf:"bytes,6,opt,name=value_count,json=valueCount,proto3,oneof"` -} - -func (*FeatureSpec_Shape) isFeatureSpec_ShapeType() {} - -func (*FeatureSpec_ValueCount) isFeatureSpec_ShapeType() {} - -func (m *FeatureSpec) GetShapeType() isFeatureSpec_ShapeType { - if m != nil { - return m.ShapeType - } - return nil -} - -func (m *FeatureSpec) GetShape() *v0.FixedShape { - if x, ok := m.GetShapeType().(*FeatureSpec_Shape); ok { - return x.Shape - } - return nil -} - -func (m *FeatureSpec) GetValueCount() *v0.ValueCount { - if x, ok := m.GetShapeType().(*FeatureSpec_ValueCount); ok { - return x.ValueCount - } - return nil -} - -type isFeatureSpec_DomainInfo interface { - isFeatureSpec_DomainInfo() -} - -type FeatureSpec_Domain struct { - Domain string `protobuf:"bytes,7,opt,name=domain,proto3,oneof"` -} - -type FeatureSpec_IntDomain struct { - IntDomain *v0.IntDomain `protobuf:"bytes,8,opt,name=int_domain,json=intDomain,proto3,oneof"` -} - -type FeatureSpec_FloatDomain struct { - FloatDomain *v0.FloatDomain `protobuf:"bytes,9,opt,name=float_domain,json=floatDomain,proto3,oneof"` -} - -type FeatureSpec_StringDomain struct { - StringDomain *v0.StringDomain `protobuf:"bytes,10,opt,name=string_domain,json=stringDomain,proto3,oneof"` -} - -type FeatureSpec_BoolDomain struct { - BoolDomain *v0.BoolDomain `protobuf:"bytes,11,opt,name=bool_domain,json=boolDomain,proto3,oneof"` -} - -type FeatureSpec_StructDomain struct { - StructDomain *v0.StructDomain `protobuf:"bytes,12,opt,name=struct_domain,json=structDomain,proto3,oneof"` -} - -type FeatureSpec_NaturalLanguageDomain struct { - NaturalLanguageDomain *v0.NaturalLanguageDomain `protobuf:"bytes,13,opt,name=natural_language_domain,json=naturalLanguageDomain,proto3,oneof"` -} - -type FeatureSpec_ImageDomain struct { - ImageDomain *v0.ImageDomain `protobuf:"bytes,14,opt,name=image_domain,json=imageDomain,proto3,oneof"` -} - -type FeatureSpec_MidDomain struct { - MidDomain *v0.MIDDomain `protobuf:"bytes,15,opt,name=mid_domain,json=midDomain,proto3,oneof"` -} - -type FeatureSpec_UrlDomain struct { - UrlDomain *v0.URLDomain `protobuf:"bytes,16,opt,name=url_domain,json=urlDomain,proto3,oneof"` -} - -type FeatureSpec_TimeDomain struct { - TimeDomain *v0.TimeDomain `protobuf:"bytes,17,opt,name=time_domain,json=timeDomain,proto3,oneof"` -} - -type FeatureSpec_TimeOfDayDomain struct { - TimeOfDayDomain *v0.TimeOfDayDomain `protobuf:"bytes,18,opt,name=time_of_day_domain,json=timeOfDayDomain,proto3,oneof"` -} - -func (*FeatureSpec_Domain) isFeatureSpec_DomainInfo() {} - -func (*FeatureSpec_IntDomain) isFeatureSpec_DomainInfo() {} - -func (*FeatureSpec_FloatDomain) isFeatureSpec_DomainInfo() {} - -func (*FeatureSpec_StringDomain) isFeatureSpec_DomainInfo() {} - -func (*FeatureSpec_BoolDomain) isFeatureSpec_DomainInfo() {} - -func (*FeatureSpec_StructDomain) isFeatureSpec_DomainInfo() {} - -func (*FeatureSpec_NaturalLanguageDomain) isFeatureSpec_DomainInfo() {} - -func (*FeatureSpec_ImageDomain) isFeatureSpec_DomainInfo() {} - -func (*FeatureSpec_MidDomain) isFeatureSpec_DomainInfo() {} - -func (*FeatureSpec_UrlDomain) isFeatureSpec_DomainInfo() {} - -func (*FeatureSpec_TimeDomain) isFeatureSpec_DomainInfo() {} - -func (*FeatureSpec_TimeOfDayDomain) isFeatureSpec_DomainInfo() {} - -func (m *FeatureSpec) GetDomainInfo() isFeatureSpec_DomainInfo { - if m != nil { - return m.DomainInfo - } - return nil -} - -func (m *FeatureSpec) GetDomain() string { - if x, ok := m.GetDomainInfo().(*FeatureSpec_Domain); ok { - return x.Domain - } - return "" -} - -func (m *FeatureSpec) GetIntDomain() *v0.IntDomain { - if x, ok := m.GetDomainInfo().(*FeatureSpec_IntDomain); ok { - return x.IntDomain - } - return nil -} - -func (m *FeatureSpec) GetFloatDomain() *v0.FloatDomain { - if x, ok := m.GetDomainInfo().(*FeatureSpec_FloatDomain); ok { - return x.FloatDomain - } - return nil -} - -func (m *FeatureSpec) GetStringDomain() *v0.StringDomain { - if x, ok := m.GetDomainInfo().(*FeatureSpec_StringDomain); ok { - return x.StringDomain - } - return nil -} - -func (m *FeatureSpec) GetBoolDomain() *v0.BoolDomain { - if x, ok := m.GetDomainInfo().(*FeatureSpec_BoolDomain); ok { - return x.BoolDomain - } - return nil -} - -func (m *FeatureSpec) GetStructDomain() *v0.StructDomain { - if x, ok := m.GetDomainInfo().(*FeatureSpec_StructDomain); ok { - return x.StructDomain - } - return nil -} - -func (m *FeatureSpec) GetNaturalLanguageDomain() *v0.NaturalLanguageDomain { - if x, ok := m.GetDomainInfo().(*FeatureSpec_NaturalLanguageDomain); ok { - return x.NaturalLanguageDomain - } - return nil -} - -func (m *FeatureSpec) GetImageDomain() *v0.ImageDomain { - if x, ok := m.GetDomainInfo().(*FeatureSpec_ImageDomain); ok { - return x.ImageDomain - } - return nil -} - -func (m *FeatureSpec) GetMidDomain() *v0.MIDDomain { - if x, ok := m.GetDomainInfo().(*FeatureSpec_MidDomain); ok { - return x.MidDomain - } - return nil -} - -func (m *FeatureSpec) GetUrlDomain() *v0.URLDomain { - if x, ok := m.GetDomainInfo().(*FeatureSpec_UrlDomain); ok { - return x.UrlDomain - } - return nil -} - -func (m *FeatureSpec) GetTimeDomain() *v0.TimeDomain { - if x, ok := m.GetDomainInfo().(*FeatureSpec_TimeDomain); ok { - return x.TimeDomain - } - return nil -} - -func (m *FeatureSpec) GetTimeOfDayDomain() *v0.TimeOfDayDomain { - if x, ok := m.GetDomainInfo().(*FeatureSpec_TimeOfDayDomain); ok { - return x.TimeOfDayDomain - } - return nil -} - -// XXX_OneofWrappers is for the internal use of the proto package. -func (*FeatureSpec) XXX_OneofWrappers() []interface{} { - return []interface{}{ - (*FeatureSpec_Presence)(nil), - (*FeatureSpec_GroupPresence)(nil), - (*FeatureSpec_Shape)(nil), - (*FeatureSpec_ValueCount)(nil), - (*FeatureSpec_Domain)(nil), - (*FeatureSpec_IntDomain)(nil), - (*FeatureSpec_FloatDomain)(nil), - (*FeatureSpec_StringDomain)(nil), - (*FeatureSpec_BoolDomain)(nil), - (*FeatureSpec_StructDomain)(nil), - (*FeatureSpec_NaturalLanguageDomain)(nil), - (*FeatureSpec_ImageDomain)(nil), - (*FeatureSpec_MidDomain)(nil), - (*FeatureSpec_UrlDomain)(nil), - (*FeatureSpec_TimeDomain)(nil), - (*FeatureSpec_TimeOfDayDomain)(nil), - } -} - -type FeatureSetMeta struct { - // Created timestamp of this specific feature set. - CreatedTimestamp *timestamp.Timestamp `protobuf:"bytes,1,opt,name=created_timestamp,json=createdTimestamp,proto3" json:"created_timestamp,omitempty"` - // Status of the feature set. - // Used to indicate whether the feature set is ready for consumption or ingestion. - // Currently supports 2 states: - // 1) STATUS_PENDING - A feature set is in pending state if Feast has not spun up the jobs - // necessary to push rows for this feature set to stores subscribing to this feature set. - // 2) STATUS_READY - Feature set is ready for consumption or ingestion - Status FeatureSetStatus `protobuf:"varint,2,opt,name=status,proto3,enum=feast.core.FeatureSetStatus" json:"status,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *FeatureSetMeta) Reset() { *m = FeatureSetMeta{} } -func (m *FeatureSetMeta) String() string { return proto.CompactTextString(m) } -func (*FeatureSetMeta) ProtoMessage() {} -func (*FeatureSetMeta) Descriptor() ([]byte, []int) { - return fileDescriptor_972fbd278ac19c0c, []int{4} -} - -func (m *FeatureSetMeta) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_FeatureSetMeta.Unmarshal(m, b) -} -func (m *FeatureSetMeta) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_FeatureSetMeta.Marshal(b, m, deterministic) -} -func (m *FeatureSetMeta) XXX_Merge(src proto.Message) { - xxx_messageInfo_FeatureSetMeta.Merge(m, src) -} -func (m *FeatureSetMeta) XXX_Size() int { - return xxx_messageInfo_FeatureSetMeta.Size(m) -} -func (m *FeatureSetMeta) XXX_DiscardUnknown() { - xxx_messageInfo_FeatureSetMeta.DiscardUnknown(m) -} - -var xxx_messageInfo_FeatureSetMeta proto.InternalMessageInfo - -func (m *FeatureSetMeta) GetCreatedTimestamp() *timestamp.Timestamp { - if m != nil { - return m.CreatedTimestamp - } - return nil -} - -func (m *FeatureSetMeta) GetStatus() FeatureSetStatus { - if m != nil { - return m.Status - } - return FeatureSetStatus_STATUS_INVALID -} - -func init() { - proto.RegisterEnum("feast.core.FeatureSetStatus", FeatureSetStatus_name, FeatureSetStatus_value) - proto.RegisterType((*FeatureSet)(nil), "feast.core.FeatureSet") - proto.RegisterType((*FeatureSetSpec)(nil), "feast.core.FeatureSetSpec") - proto.RegisterType((*EntitySpec)(nil), "feast.core.EntitySpec") - proto.RegisterType((*FeatureSpec)(nil), "feast.core.FeatureSpec") - proto.RegisterType((*FeatureSetMeta)(nil), "feast.core.FeatureSetMeta") -} - -func init() { proto.RegisterFile("feast/core/FeatureSet.proto", fileDescriptor_972fbd278ac19c0c) } - -var fileDescriptor_972fbd278ac19c0c = []byte{ - // 938 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xec, 0x97, 0xdf, 0x6e, 0xe2, 0xc6, - 0x17, 0xc7, 0x03, 0x49, 0x48, 0x38, 0x10, 0xc2, 0x8e, 0x7e, 0xbf, 0x8d, 0xbb, 0x5b, 0xb5, 0x29, - 0xad, 0xd4, 0x74, 0xa5, 0xda, 0x2b, 0xb6, 0x57, 0x7b, 0x17, 0x0a, 0x0d, 0xa8, 0x59, 0x1a, 0x19, - 0x36, 0x55, 0xdb, 0x0b, 0x6b, 0x30, 0x83, 0x33, 0xbb, 0xf6, 0x8c, 0xe5, 0x19, 0xd3, 0xf0, 0x14, - 0x7d, 0x80, 0xf6, 0xa6, 0x6f, 0x5a, 0xcd, 0xd8, 0x63, 0x93, 0x28, 0xd0, 0x3e, 0x00, 0x77, 0x39, - 0x3e, 0xdf, 0xf9, 0xf8, 0xfc, 0xcb, 0xc1, 0x03, 0x2f, 0x17, 0x04, 0x0b, 0xe9, 0xf8, 0x3c, 0x21, - 0xce, 0x0f, 0x04, 0xcb, 0x34, 0x21, 0x13, 0x22, 0xed, 0x38, 0xe1, 0x92, 0x23, 0xd0, 0x4e, 0x5b, - 0x39, 0x5f, 0x9c, 0x65, 0x42, 0xb9, 0x8a, 0x89, 0x70, 0x6e, 0x71, 0x98, 0x92, 0x4c, 0x64, 0x1c, - 0x9a, 0x30, 0xe1, 0x69, 0xe2, 0x1b, 0xc7, 0x67, 0x01, 0xe7, 0x41, 0x48, 0x1c, 0x6d, 0xcd, 0xd2, - 0x85, 0x33, 0x4f, 0x13, 0x2c, 0x29, 0x67, 0xb9, 0xff, 0xf3, 0xc7, 0x7e, 0x49, 0x23, 0x22, 0x24, - 0x8e, 0xe2, 0x5c, 0xf0, 0x8d, 0x24, 0x4c, 0xf0, 0x64, 0x11, 0xf2, 0xdf, 0xbd, 0x88, 0x48, 0x3c, - 0xc7, 0x12, 0x67, 0x6a, 0x67, 0xf9, 0xda, 0x11, 0xfe, 0x1d, 0x89, 0x70, 0x26, 0xed, 0x84, 0x00, - 0x65, 0xf4, 0xc8, 0x86, 0x03, 0x11, 0x13, 0xdf, 0xaa, 0x9c, 0x57, 0x2e, 0x1a, 0xdd, 0x17, 0x76, - 0x99, 0x86, 0x5d, 0xaa, 0x26, 0x31, 0xf1, 0x5d, 0xad, 0x53, 0x7a, 0xc5, 0xb7, 0xaa, 0xdb, 0xf4, - 0xef, 0x88, 0xc4, 0xae, 0xd6, 0x75, 0xfe, 0xae, 0x42, 0xeb, 0x21, 0x08, 0x59, 0x70, 0x14, 0x27, - 0xfc, 0x03, 0xf1, 0xa5, 0x75, 0x74, 0x5e, 0xb9, 0xa8, 0xbb, 0xc6, 0x44, 0x08, 0x0e, 0x18, 0x8e, - 0x88, 0x0e, 0xa6, 0xee, 0xea, 0xbf, 0x95, 0x7a, 0x49, 0x12, 0x41, 0x39, 0xd3, 0xef, 0x3c, 0x74, - 0x8d, 0x89, 0xba, 0x70, 0x4c, 0x98, 0xa4, 0x92, 0x12, 0x61, 0xed, 0x9f, 0xef, 0x5f, 0x34, 0xba, - 0xcf, 0xd7, 0xc3, 0x19, 0x28, 0xdf, 0x4a, 0x87, 0x5e, 0xe8, 0xd0, 0x1b, 0x38, 0x5e, 0x64, 0xd1, - 0x08, 0xeb, 0x40, 0x9f, 0x39, 0x7b, 0x2a, 0x05, 0x7d, 0xc8, 0x08, 0x51, 0x17, 0x8e, 0x22, 0x7c, - 0xef, 0xe1, 0x80, 0x58, 0x87, 0x3a, 0xed, 0x4f, 0xec, 0xac, 0x1f, 0xb6, 0xe9, 0x87, 0xdd, 0xcf, - 0xfb, 0xe5, 0xd6, 0x22, 0x7c, 0x7f, 0x19, 0x10, 0xf4, 0x0a, 0x6a, 0x42, 0x77, 0xd8, 0xaa, 0xe9, - 0x23, 0x68, 0xfd, 0x35, 0x59, 0xef, 0xdd, 0x5c, 0xd1, 0xf9, 0x13, 0x00, 0xca, 0x68, 0x9f, 0xac, - 0xc2, 0x5b, 0x80, 0xa5, 0x1a, 0x24, 0x4f, 0x0d, 0x95, 0x2e, 0x44, 0xab, 0xfb, 0x32, 0x47, 0xea, - 0x39, 0xb3, 0xf5, 0x9c, 0x4d, 0x57, 0xb1, 0x4a, 0x3c, 0x8d, 0xdc, 0xfa, 0xd2, 0xd8, 0x68, 0x00, - 0xc7, 0x71, 0x42, 0x04, 0x61, 0x3e, 0xb1, 0xf6, 0x75, 0x30, 0x5f, 0xdb, 0xe5, 0xb8, 0xd8, 0x66, - 0x5c, 0xec, 0xe5, 0x6b, 0x93, 0xff, 0x4d, 0x2e, 0x1f, 0xee, 0xb9, 0xc5, 0x51, 0xf4, 0x1b, 0xb4, - 0x82, 0x84, 0xa7, 0xb1, 0x57, 0xc0, 0x0e, 0x34, 0xac, 0xfb, 0x1f, 0x61, 0x3f, 0x53, 0x79, 0x47, - 0xd9, 0x95, 0x42, 0x0c, 0xf7, 0xdc, 0x13, 0xcd, 0x32, 0x3e, 0xf4, 0x16, 0x0e, 0xc5, 0x1d, 0x8e, - 0x4d, 0x81, 0x3b, 0x1b, 0x99, 0xf4, 0x9e, 0xcc, 0x27, 0x4a, 0x39, 0xac, 0xb8, 0xd9, 0x11, 0x34, - 0x80, 0x46, 0x56, 0x1b, 0x9f, 0xa7, 0x4c, 0xe6, 0xf5, 0xde, 0x48, 0xd0, 0x75, 0xfa, 0x5e, 0x29, - 0x87, 0x15, 0x37, 0x2b, 0xaa, 0xb6, 0x90, 0x05, 0xb5, 0x39, 0x8f, 0x30, 0x65, 0xd9, 0x54, 0x0e, - 0xab, 0x6e, 0x6e, 0xa3, 0x1e, 0x00, 0x65, 0xd2, 0xcb, 0xbd, 0xc7, 0x9a, 0xff, 0xc5, 0x26, 0xfe, - 0x88, 0xc9, 0xbe, 0x16, 0x0e, 0xab, 0x6e, 0x9d, 0x1a, 0x03, 0x0d, 0xa1, 0xb9, 0x08, 0x39, 0x2e, - 0x28, 0x75, 0x4d, 0xf9, 0x72, 0x63, 0x9e, 0x4a, 0x5b, 0x70, 0x1a, 0x8b, 0xd2, 0x44, 0x3f, 0xc2, - 0x89, 0x90, 0x09, 0x65, 0x81, 0x41, 0x81, 0x46, 0x7d, 0xb5, 0x09, 0x35, 0xd1, 0xe2, 0x82, 0xd5, - 0x14, 0x6b, 0xb6, 0xaa, 0xdd, 0x8c, 0xf3, 0xd0, 0xa0, 0x1a, 0xdb, 0x6b, 0xd7, 0xe3, 0x3c, 0x2c, - 0x40, 0x30, 0x2b, 0xac, 0x3c, 0xa6, 0xd4, 0x2f, 0xd2, 0x6b, 0xfe, 0x6b, 0x4c, 0xa9, 0x2f, 0x1f, - 0xc4, 0x54, 0xd8, 0x28, 0x80, 0x33, 0xa6, 0x26, 0x07, 0x87, 0x5e, 0x88, 0x59, 0x90, 0xe2, 0x80, - 0x18, 0xec, 0x89, 0xc6, 0x7e, 0xbb, 0x09, 0x3b, 0xce, 0x8e, 0x5d, 0xe7, 0xa7, 0x0a, 0xfe, 0xff, - 0xd9, 0x53, 0x0e, 0xd5, 0x13, 0x1a, 0xad, 0xd1, 0x5b, 0xdb, 0x7b, 0x32, 0x8a, 0xd6, 0x99, 0x0d, - 0x5a, 0x9a, 0x6a, 0x42, 0x22, 0x3a, 0x37, 0x9c, 0xd3, 0xed, 0x13, 0xf2, 0x6e, 0xd4, 0x2f, 0x27, - 0x24, 0xa2, 0xf3, 0x92, 0x91, 0x26, 0x45, 0x27, 0xda, 0xdb, 0x19, 0xef, 0xdd, 0xeb, 0x92, 0x91, - 0x26, 0x61, 0xd9, 0x4e, 0xf5, 0xcb, 0x60, 0x20, 0xcf, 0xb6, 0xb7, 0x73, 0x4a, 0xa3, 0x32, 0x1f, - 0x90, 0x85, 0x85, 0x6e, 0x01, 0x69, 0x0c, 0x5f, 0x78, 0x73, 0xbc, 0x32, 0x34, 0xb4, 0x7d, 0x77, - 0x28, 0xda, 0x4f, 0x8b, 0x3e, 0x5e, 0x15, 0xc8, 0x53, 0xf9, 0xf0, 0x51, 0xef, 0x39, 0xfc, 0xcf, - 0x2c, 0x0f, 0xcf, 0xe7, 0x4c, 0xc8, 0x04, 0x53, 0x26, 0x45, 0xaf, 0x09, 0xa0, 0xff, 0x95, 0xf5, - 0x76, 0xeb, 0x9d, 0x40, 0x23, 0x7b, 0xa3, 0x47, 0xd9, 0x82, 0x77, 0xfe, 0x02, 0x68, 0xac, 0xed, - 0xe5, 0xdd, 0x7a, 0xdc, 0xad, 0xc7, 0xdd, 0x7a, 0xdc, 0xad, 0xc7, 0xdd, 0x7a, 0xcc, 0xd6, 0xe3, - 0x1f, 0x95, 0xf5, 0x0f, 0x6c, 0xf5, 0xe5, 0x8d, 0xae, 0xe0, 0x99, 0x9f, 0x10, 0x2c, 0xc9, 0xdc, - 0x2b, 0xee, 0x09, 0xc5, 0x07, 0xfe, 0xe3, 0x2f, 0xd7, 0xa9, 0x51, 0xb8, 0xed, 0xfc, 0x50, 0xf1, - 0x04, 0x7d, 0x07, 0x35, 0x21, 0xb1, 0x4c, 0x45, 0xbe, 0x52, 0x3f, 0xdd, 0x70, 0x3d, 0xd0, 0x1a, - 0x37, 0xd7, 0xbe, 0xba, 0x86, 0xf6, 0x63, 0x1f, 0x42, 0xd0, 0x9a, 0x4c, 0x2f, 0xa7, 0xef, 0x27, - 0xde, 0x68, 0x7c, 0x7b, 0x79, 0x3d, 0xea, 0xb7, 0xf7, 0xd6, 0x9e, 0xdd, 0x0c, 0xc6, 0xfd, 0xd1, - 0xf8, 0xaa, 0x5d, 0x41, 0x6d, 0x68, 0xe6, 0xcf, 0xdc, 0xc1, 0x65, 0xff, 0x97, 0x76, 0xb5, 0x37, - 0x86, 0xb5, 0xab, 0x55, 0xef, 0xb4, 0x24, 0xdf, 0xa8, 0x0c, 0x7e, 0x75, 0x02, 0x2a, 0xef, 0xd2, - 0x99, 0xed, 0xf3, 0xc8, 0x09, 0xf8, 0x07, 0xf2, 0xd1, 0xc9, 0xee, 0x58, 0x62, 0xfe, 0xd1, 0x09, - 0x78, 0x76, 0x05, 0x12, 0x4e, 0x79, 0xef, 0x9a, 0xd5, 0xf4, 0xa3, 0x37, 0xff, 0x04, 0x00, 0x00, - 0xff, 0xff, 0x4e, 0x2a, 0x9e, 0xd9, 0xce, 0x0d, 0x00, 0x00, -} diff --git a/sdk/go/protos/feast/core/Source.pb.go b/sdk/go/protos/feast/core/Source.pb.go deleted file mode 100644 index 090210a4961..00000000000 --- a/sdk/go/protos/feast/core/Source.pb.go +++ /dev/null @@ -1,201 +0,0 @@ -// Code generated by protoc-gen-go. DO NOT EDIT. -// source: feast/core/Source.proto - -package core - -import ( - fmt "fmt" - proto "github.com/golang/protobuf/proto" - math "math" -) - -// Reference imports to suppress errors if they are not otherwise used. -var _ = proto.Marshal -var _ = fmt.Errorf -var _ = math.Inf - -// This is a compile-time assertion to ensure that this generated file -// is compatible with the proto package it is being compiled against. -// A compilation error at this line likely means your copy of the -// proto package needs to be updated. -const _ = proto.ProtoPackageIsVersion3 // please upgrade the proto package - -type SourceType int32 - -const ( - SourceType_INVALID SourceType = 0 - SourceType_KAFKA SourceType = 1 -) - -var SourceType_name = map[int32]string{ - 0: "INVALID", - 1: "KAFKA", -} - -var SourceType_value = map[string]int32{ - "INVALID": 0, - "KAFKA": 1, -} - -func (x SourceType) String() string { - return proto.EnumName(SourceType_name, int32(x)) -} - -func (SourceType) EnumDescriptor() ([]byte, []int) { - return fileDescriptor_4d161c4e53091468, []int{0} -} - -type Source struct { - // The kind of data source Feast should connect to in order to retrieve FeatureRow value - Type SourceType `protobuf:"varint,1,opt,name=type,proto3,enum=feast.core.SourceType" json:"type,omitempty"` - // Source specific configuration - // - // Types that are valid to be assigned to SourceConfig: - // *Source_KafkaSourceConfig - SourceConfig isSource_SourceConfig `protobuf_oneof:"source_config"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *Source) Reset() { *m = Source{} } -func (m *Source) String() string { return proto.CompactTextString(m) } -func (*Source) ProtoMessage() {} -func (*Source) Descriptor() ([]byte, []int) { - return fileDescriptor_4d161c4e53091468, []int{0} -} - -func (m *Source) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_Source.Unmarshal(m, b) -} -func (m *Source) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_Source.Marshal(b, m, deterministic) -} -func (m *Source) XXX_Merge(src proto.Message) { - xxx_messageInfo_Source.Merge(m, src) -} -func (m *Source) XXX_Size() int { - return xxx_messageInfo_Source.Size(m) -} -func (m *Source) XXX_DiscardUnknown() { - xxx_messageInfo_Source.DiscardUnknown(m) -} - -var xxx_messageInfo_Source proto.InternalMessageInfo - -func (m *Source) GetType() SourceType { - if m != nil { - return m.Type - } - return SourceType_INVALID -} - -type isSource_SourceConfig interface { - isSource_SourceConfig() -} - -type Source_KafkaSourceConfig struct { - KafkaSourceConfig *KafkaSourceConfig `protobuf:"bytes,2,opt,name=kafka_source_config,json=kafkaSourceConfig,proto3,oneof"` -} - -func (*Source_KafkaSourceConfig) isSource_SourceConfig() {} - -func (m *Source) GetSourceConfig() isSource_SourceConfig { - if m != nil { - return m.SourceConfig - } - return nil -} - -func (m *Source) GetKafkaSourceConfig() *KafkaSourceConfig { - if x, ok := m.GetSourceConfig().(*Source_KafkaSourceConfig); ok { - return x.KafkaSourceConfig - } - return nil -} - -// XXX_OneofWrappers is for the internal use of the proto package. -func (*Source) XXX_OneofWrappers() []interface{} { - return []interface{}{ - (*Source_KafkaSourceConfig)(nil), - } -} - -type KafkaSourceConfig struct { - // - bootstrapServers: [comma delimited value of host[:port]] - BootstrapServers string `protobuf:"bytes,1,opt,name=bootstrap_servers,json=bootstrapServers,proto3" json:"bootstrap_servers,omitempty"` - // - topics: [Kafka topic name. This value is provisioned by core and should not be set by the user.] - Topic string `protobuf:"bytes,2,opt,name=topic,proto3" json:"topic,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *KafkaSourceConfig) Reset() { *m = KafkaSourceConfig{} } -func (m *KafkaSourceConfig) String() string { return proto.CompactTextString(m) } -func (*KafkaSourceConfig) ProtoMessage() {} -func (*KafkaSourceConfig) Descriptor() ([]byte, []int) { - return fileDescriptor_4d161c4e53091468, []int{1} -} - -func (m *KafkaSourceConfig) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_KafkaSourceConfig.Unmarshal(m, b) -} -func (m *KafkaSourceConfig) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_KafkaSourceConfig.Marshal(b, m, deterministic) -} -func (m *KafkaSourceConfig) XXX_Merge(src proto.Message) { - xxx_messageInfo_KafkaSourceConfig.Merge(m, src) -} -func (m *KafkaSourceConfig) XXX_Size() int { - return xxx_messageInfo_KafkaSourceConfig.Size(m) -} -func (m *KafkaSourceConfig) XXX_DiscardUnknown() { - xxx_messageInfo_KafkaSourceConfig.DiscardUnknown(m) -} - -var xxx_messageInfo_KafkaSourceConfig proto.InternalMessageInfo - -func (m *KafkaSourceConfig) GetBootstrapServers() string { - if m != nil { - return m.BootstrapServers - } - return "" -} - -func (m *KafkaSourceConfig) GetTopic() string { - if m != nil { - return m.Topic - } - return "" -} - -func init() { - proto.RegisterEnum("feast.core.SourceType", SourceType_name, SourceType_value) - proto.RegisterType((*Source)(nil), "feast.core.Source") - proto.RegisterType((*KafkaSourceConfig)(nil), "feast.core.KafkaSourceConfig") -} - -func init() { proto.RegisterFile("feast/core/Source.proto", fileDescriptor_4d161c4e53091468) } - -var fileDescriptor_4d161c4e53091468 = []byte{ - // 273 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x64, 0x90, 0xd1, 0x4a, 0xc3, 0x30, - 0x14, 0x86, 0x57, 0x71, 0x93, 0x9e, 0xa2, 0xb6, 0x51, 0x74, 0x37, 0xc2, 0x18, 0x5e, 0x8c, 0x09, - 0x0d, 0xcc, 0x27, 0x68, 0x15, 0x71, 0x56, 0x54, 0x3a, 0xd9, 0x85, 0x37, 0xa5, 0xad, 0x69, 0xad, - 0x41, 0x4f, 0x48, 0x32, 0x61, 0x2f, 0xe2, 0xf3, 0x4a, 0x13, 0xb0, 0xd3, 0x5d, 0xf6, 0xff, 0xbf, - 0xbf, 0x9c, 0x7c, 0x70, 0x5a, 0xb1, 0x5c, 0x69, 0x5a, 0xa2, 0x64, 0x74, 0x81, 0x2b, 0x59, 0xb2, - 0x50, 0x48, 0xd4, 0x48, 0xc0, 0x14, 0x61, 0x5b, 0x8c, 0xbf, 0x1d, 0x18, 0xd8, 0x92, 0x4c, 0x61, - 0x57, 0xaf, 0x05, 0x1b, 0x3a, 0x23, 0x67, 0x72, 0x30, 0x3b, 0x09, 0x3b, 0x2a, 0xb4, 0xc4, 0xf3, - 0x5a, 0xb0, 0xd4, 0x30, 0xe4, 0x11, 0x8e, 0x78, 0x5e, 0xf1, 0x3c, 0x53, 0xa6, 0xc9, 0x4a, 0xfc, - 0xac, 0x9a, 0x7a, 0xb8, 0x33, 0x72, 0x26, 0xde, 0xec, 0x6c, 0x73, 0x9a, 0xb4, 0x98, 0xdd, 0x5f, - 0x19, 0xe8, 0xb6, 0x97, 0x06, 0xfc, 0x7f, 0x18, 0x1f, 0xc2, 0xfe, 0x9f, 0x5f, 0x8d, 0x97, 0x10, - 0x6c, 0x4d, 0xc9, 0x05, 0x04, 0x05, 0xa2, 0x56, 0x5a, 0xe6, 0x22, 0x53, 0x4c, 0x7e, 0x31, 0xa9, - 0xcc, 0xbd, 0x6e, 0xea, 0xff, 0x16, 0x0b, 0x9b, 0x93, 0x63, 0xe8, 0x6b, 0x14, 0x4d, 0x69, 0xae, - 0x72, 0x53, 0xfb, 0x31, 0x3d, 0x07, 0xe8, 0x5e, 0x43, 0x3c, 0xd8, 0x9b, 0x3f, 0x2c, 0xa3, 0xfb, - 0xf9, 0xb5, 0xdf, 0x23, 0x2e, 0xf4, 0x93, 0xe8, 0x26, 0x89, 0x7c, 0x27, 0xbe, 0x83, 0x0d, 0x49, - 0xb1, 0x67, 0x17, 0x4f, 0xad, 0xbd, 0x17, 0x5a, 0x37, 0xfa, 0x6d, 0x55, 0x84, 0x25, 0x7e, 0xd0, - 0x1a, 0xdf, 0x19, 0xa7, 0xd6, 0xb3, 0x7a, 0xe5, 0xb4, 0x46, 0x6a, 0x14, 0x2b, 0xda, 0xb9, 0x2f, - 0x06, 0x26, 0xba, 0xfc, 0x09, 0x00, 0x00, 0xff, 0xff, 0xc3, 0xc9, 0xf2, 0x62, 0x90, 0x01, 0x00, - 0x00, -} diff --git a/sdk/go/protos/feast/core/Store.pb.go b/sdk/go/protos/feast/core/Store.pb.go deleted file mode 100644 index 62bb16d0adb..00000000000 --- a/sdk/go/protos/feast/core/Store.pb.go +++ /dev/null @@ -1,531 +0,0 @@ -// Code generated by protoc-gen-go. DO NOT EDIT. -// source: feast/core/Store.proto - -package core - -import ( - fmt "fmt" - proto "github.com/golang/protobuf/proto" - math "math" -) - -// Reference imports to suppress errors if they are not otherwise used. -var _ = proto.Marshal -var _ = fmt.Errorf -var _ = math.Inf - -// This is a compile-time assertion to ensure that this generated file -// is compatible with the proto package it is being compiled against. -// A compilation error at this line likely means your copy of the -// proto package needs to be updated. -const _ = proto.ProtoPackageIsVersion3 // please upgrade the proto package - -type Store_StoreType int32 - -const ( - Store_INVALID Store_StoreType = 0 - // Redis stores a FeatureRow element as a key, value pair. - // - // The Redis data types used (https://redis.io/topics/data-types): - // - key: STRING - // - value: STRING - // - // Encodings: - // - key: byte array of RedisKey (refer to feast.storage.RedisKey) - // - value: byte array of FeatureRow (refer to feast.types.FeatureRow) - // - Store_REDIS Store_StoreType = 1 - // BigQuery stores a FeatureRow element as a row in a BigQuery table. - // - // Table name is derived from the feature set name and version as: - // [feature_set_name]_v[feature_set_version] - // - // For example: - // A feature row for feature set "driver" and version "1" will be written - // to table "driver_v1". - // - // The entities and features in a FeatureSetSpec corresponds to the - // fields in the BigQuery table (these make up the BigQuery schema). - // The name of the entity spec and feature spec corresponds to the column - // names, and the value_type of entity spec and feature spec corresponds - // to BigQuery standard SQL data type of the column. - // - // The following BigQuery fields are reserved for Feast internal use. - // Ingestion of entity or feature spec with names identical - // to the following field names will raise an exception during ingestion. - // - // column_name | column_data_type | description - // ====================|==================|================================ - // - event_timestamp | TIMESTAMP | event time of the FeatureRow - // - created_timestamp | TIMESTAMP | processing time of the ingestion of the FeatureRow - // - job_id | STRING | identifier for the job that writes the FeatureRow to the corresponding BigQuery table - // - // BigQuery table created will be partitioned by the field "event_timestamp" - // of the FeatureRow (https://cloud.google.com/bigquery/docs/partitioned-tables). - // - // Since newer version of feature set can introduce breaking, non backward- - // compatible BigQuery schema updates, incrementing the version of a - // feature set will result in the creation of a new empty BigQuery table - // with the new schema. - // - // The following table shows how ValueType in Feast is mapped to - // BigQuery Standard SQL data types - // (https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types): - // - // BYTES : BYTES - // STRING : STRING - // INT32 : INT64 - // INT64 : IN64 - // DOUBLE : FLOAT64 - // FLOAT : FLOAT64 - // BOOL : BOOL - // BYTES_LIST : ARRAY - // STRING_LIST : ARRAY - // INT32_LIST : ARRAY - // INT64_LIST : ARRAY - // DOUBLE_LIST : ARRAY - // FLOAT_LIST : ARRAY - // BOOL_LIST : ARRAY - // - // The column mode in BigQuery is set to "Nullable" such that unset Value - // in a FeatureRow corresponds to NULL value in BigQuery. - // - Store_BIGQUERY Store_StoreType = 2 - // Unsupported in Feast 0.3 - Store_CASSANDRA Store_StoreType = 3 -) - -var Store_StoreType_name = map[int32]string{ - 0: "INVALID", - 1: "REDIS", - 2: "BIGQUERY", - 3: "CASSANDRA", -} - -var Store_StoreType_value = map[string]int32{ - "INVALID": 0, - "REDIS": 1, - "BIGQUERY": 2, - "CASSANDRA": 3, -} - -func (x Store_StoreType) String() string { - return proto.EnumName(Store_StoreType_name, int32(x)) -} - -func (Store_StoreType) EnumDescriptor() ([]byte, []int) { - return fileDescriptor_4b177bc9ccf64875, []int{0, 0} -} - -// Store provides a location where Feast reads and writes feature values. -// Feature values will be written to the Store in the form of FeatureRow elements. -// The way FeatureRow is encoded and decoded when it is written to and read from -// the Store depends on the type of the Store. -// -// For example, a FeatureRow will materialize as a row in a table in -// BigQuery but it will materialize as a key, value pair element in Redis. -// -type Store struct { - // Name of the store. - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - // Type of store. - Type Store_StoreType `protobuf:"varint,2,opt,name=type,proto3,enum=feast.core.Store_StoreType" json:"type,omitempty"` - // Feature sets to subscribe to. - Subscriptions []*Store_Subscription `protobuf:"bytes,4,rep,name=subscriptions,proto3" json:"subscriptions,omitempty"` - // Configuration to connect to the store. Required. - // - // Types that are valid to be assigned to Config: - // *Store_RedisConfig_ - // *Store_BigqueryConfig - // *Store_CassandraConfig_ - Config isStore_Config `protobuf_oneof:"config"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *Store) Reset() { *m = Store{} } -func (m *Store) String() string { return proto.CompactTextString(m) } -func (*Store) ProtoMessage() {} -func (*Store) Descriptor() ([]byte, []int) { - return fileDescriptor_4b177bc9ccf64875, []int{0} -} - -func (m *Store) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_Store.Unmarshal(m, b) -} -func (m *Store) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_Store.Marshal(b, m, deterministic) -} -func (m *Store) XXX_Merge(src proto.Message) { - xxx_messageInfo_Store.Merge(m, src) -} -func (m *Store) XXX_Size() int { - return xxx_messageInfo_Store.Size(m) -} -func (m *Store) XXX_DiscardUnknown() { - xxx_messageInfo_Store.DiscardUnknown(m) -} - -var xxx_messageInfo_Store proto.InternalMessageInfo - -func (m *Store) GetName() string { - if m != nil { - return m.Name - } - return "" -} - -func (m *Store) GetType() Store_StoreType { - if m != nil { - return m.Type - } - return Store_INVALID -} - -func (m *Store) GetSubscriptions() []*Store_Subscription { - if m != nil { - return m.Subscriptions - } - return nil -} - -type isStore_Config interface { - isStore_Config() -} - -type Store_RedisConfig_ struct { - RedisConfig *Store_RedisConfig `protobuf:"bytes,11,opt,name=redis_config,json=redisConfig,proto3,oneof"` -} - -type Store_BigqueryConfig struct { - BigqueryConfig *Store_BigQueryConfig `protobuf:"bytes,12,opt,name=bigquery_config,json=bigqueryConfig,proto3,oneof"` -} - -type Store_CassandraConfig_ struct { - CassandraConfig *Store_CassandraConfig `protobuf:"bytes,13,opt,name=cassandra_config,json=cassandraConfig,proto3,oneof"` -} - -func (*Store_RedisConfig_) isStore_Config() {} - -func (*Store_BigqueryConfig) isStore_Config() {} - -func (*Store_CassandraConfig_) isStore_Config() {} - -func (m *Store) GetConfig() isStore_Config { - if m != nil { - return m.Config - } - return nil -} - -func (m *Store) GetRedisConfig() *Store_RedisConfig { - if x, ok := m.GetConfig().(*Store_RedisConfig_); ok { - return x.RedisConfig - } - return nil -} - -func (m *Store) GetBigqueryConfig() *Store_BigQueryConfig { - if x, ok := m.GetConfig().(*Store_BigqueryConfig); ok { - return x.BigqueryConfig - } - return nil -} - -func (m *Store) GetCassandraConfig() *Store_CassandraConfig { - if x, ok := m.GetConfig().(*Store_CassandraConfig_); ok { - return x.CassandraConfig - } - return nil -} - -// XXX_OneofWrappers is for the internal use of the proto package. -func (*Store) XXX_OneofWrappers() []interface{} { - return []interface{}{ - (*Store_RedisConfig_)(nil), - (*Store_BigqueryConfig)(nil), - (*Store_CassandraConfig_)(nil), - } -} - -type Store_RedisConfig struct { - Host string `protobuf:"bytes,1,opt,name=host,proto3" json:"host,omitempty"` - Port int32 `protobuf:"varint,2,opt,name=port,proto3" json:"port,omitempty"` - // Optional. The number of milliseconds to wait before retrying failed Redis connection. - // By default, Feast uses exponential backoff policy and "initial_backoff_ms" sets the initial wait duration. - InitialBackoffMs int32 `protobuf:"varint,3,opt,name=initial_backoff_ms,json=initialBackoffMs,proto3" json:"initial_backoff_ms,omitempty"` - // Optional. Maximum total number of retries for connecting to Redis. Default to zero retries. - MaxRetries int32 `protobuf:"varint,4,opt,name=max_retries,json=maxRetries,proto3" json:"max_retries,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *Store_RedisConfig) Reset() { *m = Store_RedisConfig{} } -func (m *Store_RedisConfig) String() string { return proto.CompactTextString(m) } -func (*Store_RedisConfig) ProtoMessage() {} -func (*Store_RedisConfig) Descriptor() ([]byte, []int) { - return fileDescriptor_4b177bc9ccf64875, []int{0, 0} -} - -func (m *Store_RedisConfig) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_Store_RedisConfig.Unmarshal(m, b) -} -func (m *Store_RedisConfig) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_Store_RedisConfig.Marshal(b, m, deterministic) -} -func (m *Store_RedisConfig) XXX_Merge(src proto.Message) { - xxx_messageInfo_Store_RedisConfig.Merge(m, src) -} -func (m *Store_RedisConfig) XXX_Size() int { - return xxx_messageInfo_Store_RedisConfig.Size(m) -} -func (m *Store_RedisConfig) XXX_DiscardUnknown() { - xxx_messageInfo_Store_RedisConfig.DiscardUnknown(m) -} - -var xxx_messageInfo_Store_RedisConfig proto.InternalMessageInfo - -func (m *Store_RedisConfig) GetHost() string { - if m != nil { - return m.Host - } - return "" -} - -func (m *Store_RedisConfig) GetPort() int32 { - if m != nil { - return m.Port - } - return 0 -} - -func (m *Store_RedisConfig) GetInitialBackoffMs() int32 { - if m != nil { - return m.InitialBackoffMs - } - return 0 -} - -func (m *Store_RedisConfig) GetMaxRetries() int32 { - if m != nil { - return m.MaxRetries - } - return 0 -} - -type Store_BigQueryConfig struct { - ProjectId string `protobuf:"bytes,1,opt,name=project_id,json=projectId,proto3" json:"project_id,omitempty"` - DatasetId string `protobuf:"bytes,2,opt,name=dataset_id,json=datasetId,proto3" json:"dataset_id,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *Store_BigQueryConfig) Reset() { *m = Store_BigQueryConfig{} } -func (m *Store_BigQueryConfig) String() string { return proto.CompactTextString(m) } -func (*Store_BigQueryConfig) ProtoMessage() {} -func (*Store_BigQueryConfig) Descriptor() ([]byte, []int) { - return fileDescriptor_4b177bc9ccf64875, []int{0, 1} -} - -func (m *Store_BigQueryConfig) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_Store_BigQueryConfig.Unmarshal(m, b) -} -func (m *Store_BigQueryConfig) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_Store_BigQueryConfig.Marshal(b, m, deterministic) -} -func (m *Store_BigQueryConfig) XXX_Merge(src proto.Message) { - xxx_messageInfo_Store_BigQueryConfig.Merge(m, src) -} -func (m *Store_BigQueryConfig) XXX_Size() int { - return xxx_messageInfo_Store_BigQueryConfig.Size(m) -} -func (m *Store_BigQueryConfig) XXX_DiscardUnknown() { - xxx_messageInfo_Store_BigQueryConfig.DiscardUnknown(m) -} - -var xxx_messageInfo_Store_BigQueryConfig proto.InternalMessageInfo - -func (m *Store_BigQueryConfig) GetProjectId() string { - if m != nil { - return m.ProjectId - } - return "" -} - -func (m *Store_BigQueryConfig) GetDatasetId() string { - if m != nil { - return m.DatasetId - } - return "" -} - -type Store_CassandraConfig struct { - Host string `protobuf:"bytes,1,opt,name=host,proto3" json:"host,omitempty"` - Port int32 `protobuf:"varint,2,opt,name=port,proto3" json:"port,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *Store_CassandraConfig) Reset() { *m = Store_CassandraConfig{} } -func (m *Store_CassandraConfig) String() string { return proto.CompactTextString(m) } -func (*Store_CassandraConfig) ProtoMessage() {} -func (*Store_CassandraConfig) Descriptor() ([]byte, []int) { - return fileDescriptor_4b177bc9ccf64875, []int{0, 2} -} - -func (m *Store_CassandraConfig) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_Store_CassandraConfig.Unmarshal(m, b) -} -func (m *Store_CassandraConfig) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_Store_CassandraConfig.Marshal(b, m, deterministic) -} -func (m *Store_CassandraConfig) XXX_Merge(src proto.Message) { - xxx_messageInfo_Store_CassandraConfig.Merge(m, src) -} -func (m *Store_CassandraConfig) XXX_Size() int { - return xxx_messageInfo_Store_CassandraConfig.Size(m) -} -func (m *Store_CassandraConfig) XXX_DiscardUnknown() { - xxx_messageInfo_Store_CassandraConfig.DiscardUnknown(m) -} - -var xxx_messageInfo_Store_CassandraConfig proto.InternalMessageInfo - -func (m *Store_CassandraConfig) GetHost() string { - if m != nil { - return m.Host - } - return "" -} - -func (m *Store_CassandraConfig) GetPort() int32 { - if m != nil { - return m.Port - } - return 0 -} - -type Store_Subscription struct { - // Name of project that the feature sets belongs to. This can be one of - // - [project_name] - // - * - // If an asterisk is provided, filtering on projects will be disabled. All projects will - // be matched. It is NOT possible to provide an asterisk with a string in order to do - // pattern matching. - Project string `protobuf:"bytes,3,opt,name=project,proto3" json:"project,omitempty"` - // Name of the desired feature set. Asterisks can be used as wildcards in the name. - // Matching on names is only permitted if a specific project is defined. It is disallowed - // If the project name is set to "*" - // e.g. - // - * can be used to match all feature sets - // - my-feature-set* can be used to match all features prefixed by "my-feature-set" - // - my-feature-set-6 can be used to select a single feature set - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - // Versions of the given feature sets that will be returned. - // Valid options for version: - // "latest": only the latest version is returned. - // "*": Subscribe to all versions - // [version number]: pin to a specific version. Project and feature set name must be - // explicitly defined if a specific version is pinned. - Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *Store_Subscription) Reset() { *m = Store_Subscription{} } -func (m *Store_Subscription) String() string { return proto.CompactTextString(m) } -func (*Store_Subscription) ProtoMessage() {} -func (*Store_Subscription) Descriptor() ([]byte, []int) { - return fileDescriptor_4b177bc9ccf64875, []int{0, 3} -} - -func (m *Store_Subscription) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_Store_Subscription.Unmarshal(m, b) -} -func (m *Store_Subscription) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_Store_Subscription.Marshal(b, m, deterministic) -} -func (m *Store_Subscription) XXX_Merge(src proto.Message) { - xxx_messageInfo_Store_Subscription.Merge(m, src) -} -func (m *Store_Subscription) XXX_Size() int { - return xxx_messageInfo_Store_Subscription.Size(m) -} -func (m *Store_Subscription) XXX_DiscardUnknown() { - xxx_messageInfo_Store_Subscription.DiscardUnknown(m) -} - -var xxx_messageInfo_Store_Subscription proto.InternalMessageInfo - -func (m *Store_Subscription) GetProject() string { - if m != nil { - return m.Project - } - return "" -} - -func (m *Store_Subscription) GetName() string { - if m != nil { - return m.Name - } - return "" -} - -func (m *Store_Subscription) GetVersion() string { - if m != nil { - return m.Version - } - return "" -} - -func init() { - proto.RegisterEnum("feast.core.Store_StoreType", Store_StoreType_name, Store_StoreType_value) - proto.RegisterType((*Store)(nil), "feast.core.Store") - proto.RegisterType((*Store_RedisConfig)(nil), "feast.core.Store.RedisConfig") - proto.RegisterType((*Store_BigQueryConfig)(nil), "feast.core.Store.BigQueryConfig") - proto.RegisterType((*Store_CassandraConfig)(nil), "feast.core.Store.CassandraConfig") - proto.RegisterType((*Store_Subscription)(nil), "feast.core.Store.Subscription") -} - -func init() { proto.RegisterFile("feast/core/Store.proto", fileDescriptor_4b177bc9ccf64875) } - -var fileDescriptor_4b177bc9ccf64875 = []byte{ - // 500 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x8c, 0x93, 0x4d, 0x8f, 0xd3, 0x3c, - 0x10, 0xc7, 0x37, 0x7d, 0xdd, 0x4c, 0xfa, 0x12, 0xf9, 0xf0, 0x28, 0xea, 0xa3, 0x85, 0xb2, 0xa7, - 0x1e, 0x50, 0x23, 0x95, 0x13, 0x37, 0x9a, 0x76, 0x05, 0x11, 0x50, 0xb1, 0x2e, 0xac, 0x04, 0x97, - 0xca, 0x4d, 0xdc, 0xac, 0xb7, 0x34, 0x0e, 0xb6, 0x8b, 0xb6, 0x77, 0xbe, 0x0c, 0xdf, 0x12, 0xd9, - 0x49, 0xfa, 0x42, 0xf7, 0xc0, 0x25, 0xb2, 0xff, 0xf3, 0x9f, 0x5f, 0x3c, 0xf6, 0x0c, 0xfc, 0xb7, - 0xa2, 0x44, 0x2a, 0x3f, 0xe2, 0x82, 0xfa, 0x73, 0xc5, 0x05, 0x1d, 0x66, 0x82, 0x2b, 0x8e, 0xc0, - 0xe8, 0x43, 0xad, 0x5f, 0xff, 0x6e, 0x40, 0xdd, 0xc4, 0x10, 0x82, 0x5a, 0x4a, 0x36, 0xd4, 0xb3, - 0xfa, 0xd6, 0xc0, 0xc6, 0x66, 0x8d, 0x7c, 0xa8, 0xa9, 0x5d, 0x46, 0xbd, 0x4a, 0xdf, 0x1a, 0x74, - 0x46, 0xff, 0x0f, 0x0f, 0x89, 0xc3, 0x1c, 0x68, 0xbe, 0x9f, 0x77, 0x19, 0xc5, 0xc6, 0x88, 0xa6, - 0xd0, 0x96, 0xdb, 0xa5, 0x8c, 0x04, 0xcb, 0x14, 0xe3, 0xa9, 0xf4, 0x6a, 0xfd, 0xea, 0xc0, 0x19, - 0x3d, 0x7b, 0x22, 0xf3, 0xc8, 0x86, 0x4f, 0x93, 0x50, 0x00, 0x2d, 0x41, 0x63, 0x26, 0x17, 0x11, - 0x4f, 0x57, 0x2c, 0xf1, 0x9c, 0xbe, 0x35, 0x70, 0x46, 0x57, 0xe7, 0x10, 0xac, 0x5d, 0x13, 0x63, - 0x7a, 0x77, 0x81, 0x1d, 0x71, 0xd8, 0xa2, 0xf7, 0xd0, 0x5d, 0xb2, 0xe4, 0xc7, 0x96, 0x8a, 0x5d, - 0x89, 0x69, 0x19, 0x4c, 0xff, 0x1c, 0x13, 0xb0, 0xe4, 0x56, 0x1b, 0xf7, 0xa4, 0x4e, 0x99, 0x5a, - 0xc0, 0x66, 0xe0, 0x46, 0x44, 0x4a, 0x92, 0xc6, 0x82, 0x94, 0xb4, 0xb6, 0xa1, 0xbd, 0x38, 0xa7, - 0x4d, 0x4a, 0xe7, 0x1e, 0xd7, 0x8d, 0x4e, 0xa5, 0xde, 0x2f, 0x0b, 0x9c, 0xa3, 0xb3, 0xeb, 0xbb, - 0xbf, 0xe7, 0x52, 0x95, 0x77, 0xaf, 0xd7, 0x5a, 0xcb, 0xb8, 0x50, 0xe6, 0xee, 0xeb, 0xd8, 0xac, - 0xd1, 0x4b, 0x40, 0x2c, 0x65, 0x8a, 0x91, 0xef, 0x8b, 0x25, 0x89, 0xd6, 0x7c, 0xb5, 0x5a, 0x6c, - 0xa4, 0x57, 0x35, 0x0e, 0xb7, 0x88, 0x04, 0x79, 0xe0, 0xa3, 0x44, 0xcf, 0xc1, 0xd9, 0x90, 0xc7, - 0x85, 0xa0, 0x4a, 0x30, 0xaa, 0x9f, 0x42, 0xdb, 0x60, 0x43, 0x1e, 0x71, 0xae, 0xf4, 0x66, 0xd0, - 0x39, 0x2d, 0x1d, 0x5d, 0x01, 0x64, 0x82, 0x3f, 0xd0, 0x48, 0x2d, 0x58, 0x5c, 0x1c, 0xc7, 0x2e, - 0x94, 0x30, 0xd6, 0xe1, 0x98, 0x28, 0x22, 0xa9, 0x09, 0x57, 0xf2, 0x70, 0xa1, 0x84, 0x71, 0xef, - 0x35, 0x74, 0xff, 0x2a, 0xfe, 0x5f, 0x2b, 0xeb, 0xdd, 0x41, 0xeb, 0xb8, 0x23, 0x90, 0x07, 0xcd, - 0xe2, 0xb7, 0xa6, 0x3c, 0x1b, 0x97, 0xdb, 0x27, 0xfb, 0xd4, 0x83, 0xe6, 0x4f, 0x2a, 0x24, 0xe3, - 0x69, 0x71, 0xa8, 0x72, 0x7b, 0xfd, 0x06, 0xec, 0x7d, 0x8f, 0x22, 0x07, 0x9a, 0xe1, 0xec, 0x6e, - 0xfc, 0x21, 0x9c, 0xba, 0x17, 0xc8, 0x86, 0x3a, 0xbe, 0x99, 0x86, 0x73, 0xd7, 0x42, 0x2d, 0xb8, - 0x0c, 0xc2, 0xb7, 0xb7, 0x5f, 0x6e, 0xf0, 0x57, 0xb7, 0x82, 0xda, 0x60, 0x4f, 0xc6, 0xf3, 0xf9, - 0x78, 0x36, 0xc5, 0x63, 0xb7, 0x1a, 0x5c, 0x42, 0x23, 0x7f, 0xf1, 0x20, 0x84, 0xa3, 0xc9, 0x09, - 0xc0, 0x70, 0x3f, 0xe9, 0x89, 0xfa, 0xe6, 0x27, 0x4c, 0xdd, 0x6f, 0x97, 0xc3, 0x88, 0x6f, 0xfc, - 0x84, 0x3f, 0xd0, 0xb5, 0x9f, 0x8f, 0x9e, 0x8c, 0xd7, 0x7e, 0xc2, 0x7d, 0x33, 0x76, 0xd2, 0x3f, - 0x8c, 0xe3, 0xb2, 0x61, 0xa4, 0x57, 0x7f, 0x02, 0x00, 0x00, 0xff, 0xff, 0x19, 0x4f, 0x37, 0x0a, - 0xa3, 0x03, 0x00, 0x00, -} diff --git a/sdk/go/protos/feast/serving/ServingService.pb.go b/sdk/go/protos/feast/serving/ServingService.pb.go index 212e8606ce7..1cde2f358dd 100644 --- a/sdk/go/protos/feast/serving/ServingService.pb.go +++ b/sdk/go/protos/feast/serving/ServingService.pb.go @@ -886,7 +886,9 @@ func init() { proto.RegisterType((*DatasetSource_FileSource)(nil), "feast.serving.DatasetSource.FileSource") } -func init() { proto.RegisterFile("feast/serving/ServingService.proto", fileDescriptor_0c1ba93cf29a8d9d) } +func init() { + proto.RegisterFile("feast/serving/ServingService.proto", fileDescriptor_0c1ba93cf29a8d9d) +} var fileDescriptor_0c1ba93cf29a8d9d = []byte{ // 1101 bytes of a gzipped FileDescriptorProto @@ -963,11 +965,11 @@ var fileDescriptor_0c1ba93cf29a8d9d = []byte{ // Reference imports to suppress errors if they are not otherwise used. var _ context.Context -var _ grpc.ClientConn +var _ grpc.ClientConnInterface // This is a compile-time assertion to ensure that this generated file // is compatible with the grpc package it is being compiled against. -const _ = grpc.SupportPackageIsVersion4 +const _ = grpc.SupportPackageIsVersion6 // ServingServiceClient is the client API for ServingService service. // @@ -991,10 +993,10 @@ type ServingServiceClient interface { } type servingServiceClient struct { - cc *grpc.ClientConn + cc grpc.ClientConnInterface } -func NewServingServiceClient(cc *grpc.ClientConn) ServingServiceClient { +func NewServingServiceClient(cc grpc.ClientConnInterface) ServingServiceClient { return &servingServiceClient{cc} } diff --git a/sdk/go/protos/feast/storage/Redis.pb.go b/sdk/go/protos/feast/storage/Redis.pb.go deleted file mode 100644 index f5ce0558b55..00000000000 --- a/sdk/go/protos/feast/storage/Redis.pb.go +++ /dev/null @@ -1,96 +0,0 @@ -// Code generated by protoc-gen-go. DO NOT EDIT. -// source: feast/storage/Redis.proto - -package storage - -import ( - fmt "fmt" - types "github.com/gojek/feast/sdk/go/protos/feast/types" - proto "github.com/golang/protobuf/proto" - math "math" -) - -// Reference imports to suppress errors if they are not otherwise used. -var _ = proto.Marshal -var _ = fmt.Errorf -var _ = math.Inf - -// This is a compile-time assertion to ensure that this generated file -// is compatible with the proto package it is being compiled against. -// A compilation error at this line likely means your copy of the -// proto package needs to be updated. -const _ = proto.ProtoPackageIsVersion3 // please upgrade the proto package - -type RedisKey struct { - // FeatureSet this row belongs to, this is defined as featureSetName:version. - FeatureSet string `protobuf:"bytes,2,opt,name=feature_set,json=featureSet,proto3" json:"feature_set,omitempty"` - // List of fields containing entity names and their respective values - // contained within this feature row. The entities should be sorted - // by the entity name alphabetically in ascending order. - Entities []*types.Field `protobuf:"bytes,3,rep,name=entities,proto3" json:"entities,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *RedisKey) Reset() { *m = RedisKey{} } -func (m *RedisKey) String() string { return proto.CompactTextString(m) } -func (*RedisKey) ProtoMessage() {} -func (*RedisKey) Descriptor() ([]byte, []int) { - return fileDescriptor_64e898a359fc9e5d, []int{0} -} - -func (m *RedisKey) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_RedisKey.Unmarshal(m, b) -} -func (m *RedisKey) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_RedisKey.Marshal(b, m, deterministic) -} -func (m *RedisKey) XXX_Merge(src proto.Message) { - xxx_messageInfo_RedisKey.Merge(m, src) -} -func (m *RedisKey) XXX_Size() int { - return xxx_messageInfo_RedisKey.Size(m) -} -func (m *RedisKey) XXX_DiscardUnknown() { - xxx_messageInfo_RedisKey.DiscardUnknown(m) -} - -var xxx_messageInfo_RedisKey proto.InternalMessageInfo - -func (m *RedisKey) GetFeatureSet() string { - if m != nil { - return m.FeatureSet - } - return "" -} - -func (m *RedisKey) GetEntities() []*types.Field { - if m != nil { - return m.Entities - } - return nil -} - -func init() { - proto.RegisterType((*RedisKey)(nil), "feast.storage.RedisKey") -} - -func init() { proto.RegisterFile("feast/storage/Redis.proto", fileDescriptor_64e898a359fc9e5d) } - -var fileDescriptor_64e898a359fc9e5d = []byte{ - // 193 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x54, 0x8f, 0x31, 0xaf, 0x82, 0x40, - 0x10, 0x84, 0xf3, 0x1e, 0xc9, 0x0b, 0xef, 0x88, 0xcd, 0x35, 0xa2, 0x8d, 0xc4, 0x8a, 0xea, 0x36, - 0xc1, 0x7f, 0x40, 0x61, 0x63, 0xa1, 0xc1, 0x4e, 0x0b, 0x03, 0xb2, 0x9c, 0x27, 0xea, 0x11, 0x6e, - 0x29, 0xf8, 0xf7, 0xc6, 0x85, 0x98, 0xd0, 0xee, 0xcc, 0xce, 0x37, 0x23, 0x16, 0x15, 0xe6, 0x8e, - 0xc0, 0x91, 0x6d, 0x73, 0x8d, 0x90, 0x61, 0x69, 0x9c, 0x6a, 0x5a, 0x4b, 0x56, 0xce, 0x58, 0x52, - 0xa3, 0xb4, 0x9c, 0x0f, 0x4e, 0xea, 0x1b, 0x74, 0xb0, 0x35, 0xf8, 0x28, 0x07, 0xdf, 0xfa, 0x2c, - 0x7c, 0x7e, 0xdb, 0x61, 0x2f, 0x57, 0x22, 0xa8, 0x30, 0xa7, 0xae, 0xc5, 0x8b, 0x43, 0x0a, 0x7f, - 0xa3, 0x9f, 0xf8, 0x3f, 0x13, 0xe3, 0xe9, 0x88, 0x24, 0x95, 0xf0, 0xf1, 0x45, 0x86, 0x0c, 0xba, - 0xd0, 0x8b, 0xbc, 0x38, 0x48, 0xa4, 0x1a, 0x38, 0x1c, 0xac, 0x38, 0x38, 0xfb, 0x7a, 0xd2, 0xbd, - 0x98, 0xd6, 0x48, 0x05, 0xb3, 0x0e, 0x1f, 0xf2, 0x29, 0xd1, 0x86, 0x6e, 0x5d, 0xa1, 0xae, 0xf6, - 0x09, 0xda, 0xde, 0xb1, 0x86, 0x71, 0x4d, 0x59, 0x83, 0xb6, 0xc0, 0xf5, 0x1c, 0x4c, 0x16, 0x16, - 0x7f, 0x7c, 0xdd, 0xbc, 0x03, 0x00, 0x00, 0xff, 0xff, 0x68, 0x2b, 0x58, 0x50, 0xf9, 0x00, 0x00, - 0x00, -} diff --git a/sdk/go/protos/feast/types/FeatureRow.pb.go b/sdk/go/protos/feast/types/FeatureRow.pb.go index 75ad87f8d3f..26868ebdd0b 100644 --- a/sdk/go/protos/feast/types/FeatureRow.pb.go +++ b/sdk/go/protos/feast/types/FeatureRow.pb.go @@ -87,7 +87,9 @@ func init() { proto.RegisterType((*FeatureRow)(nil), "feast.types.FeatureRow") } -func init() { proto.RegisterFile("feast/types/FeatureRow.proto", fileDescriptor_fbbea9c89787d1c7) } +func init() { + proto.RegisterFile("feast/types/FeatureRow.proto", fileDescriptor_fbbea9c89787d1c7) +} var fileDescriptor_fbbea9c89787d1c7 = []byte{ // 238 bytes of a gzipped FileDescriptorProto diff --git a/sdk/go/protos/feast/types/Field.pb.go b/sdk/go/protos/feast/types/Field.pb.go index 345b5259997..0666d9bf1fb 100644 --- a/sdk/go/protos/feast/types/Field.pb.go +++ b/sdk/go/protos/feast/types/Field.pb.go @@ -71,7 +71,9 @@ func init() { proto.RegisterType((*Field)(nil), "feast.types.Field") } -func init() { proto.RegisterFile("feast/types/Field.proto", fileDescriptor_8c568a78dfaa9ca9) } +func init() { + proto.RegisterFile("feast/types/Field.proto", fileDescriptor_8c568a78dfaa9ca9) +} var fileDescriptor_8c568a78dfaa9ca9 = []byte{ // 165 bytes of a gzipped FileDescriptorProto diff --git a/sdk/go/protos/feast/types/Value.pb.go b/sdk/go/protos/feast/types/Value.pb.go index 3f9808b994f..f6ae73c2de2 100644 --- a/sdk/go/protos/feast/types/Value.pb.go +++ b/sdk/go/protos/feast/types/Value.pb.go @@ -664,7 +664,9 @@ func init() { proto.RegisterType((*BoolList)(nil), "feast.types.BoolList") } -func init() { proto.RegisterFile("feast/types/Value.proto", fileDescriptor_47c504407d284ecc) } +func init() { + proto.RegisterFile("feast/types/Value.proto", fileDescriptor_47c504407d284ecc) +} var fileDescriptor_47c504407d284ecc = []byte{ // 600 bytes of a gzipped FileDescriptorProto diff --git a/sdk/python/requirements-ci.txt b/sdk/python/requirements-ci.txt index 745ba25289d..45aff4788b4 100644 --- a/sdk/python/requirements-ci.txt +++ b/sdk/python/requirements-ci.txt @@ -2,4 +2,12 @@ flake8 black isort grpcio-tools -mypy-protobuf \ No newline at end of file +mypy-protobuf +pytest +pytest-lazy-fixture==0.6.3 +pytest-mock +pytest-timeout +pytest-ordering==0.6.* +pandas==0.* +mock==2.0.0 +pandavro==1.5.* \ No newline at end of file diff --git a/sdk/python/requirements-dev.txt b/sdk/python/requirements-dev.txt index 9c2c9d17d21..f24141fb491 100644 --- a/sdk/python/requirements-dev.txt +++ b/sdk/python/requirements-dev.txt @@ -17,9 +17,9 @@ pytest pytest-lazy-fixture==0.6.3 pytest-mock pytest-timeout +pytest-ordering==0.6.* PyYAML==5.1.* fastavro==0.* -pytest-ordering==0.6.* pyarrow Sphinx sphinx-rtd-theme diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index b41500125cd..3c1e8bef0f0 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -14,18 +14,13 @@ import pkgutil -import tempfile from concurrent import futures -from datetime import datetime from unittest import mock import grpc -import pandas as pd import pytest from google.protobuf.duration_pb2 import Duration from mock import MagicMock, patch -from pandavro import to_avro -from pytz import timezone import dataframes import feast.core.CoreService_pb2_grpc as Core @@ -44,18 +39,11 @@ from feast.core.Source_pb2 import KafkaSourceConfig, Source, SourceType from feast.entity import Entity from feast.feature_set import Feature, FeatureSet -from feast.job import Job from feast.serving.ServingService_pb2 import ( - DataFormat, - FeastServingType, - GetBatchFeaturesResponse, GetFeastServingInfoResponse, - GetJobResponse, GetOnlineFeaturesRequest, GetOnlineFeaturesResponse, ) -from feast.serving.ServingService_pb2 import Job as BatchFeaturesJob -from feast.serving.ServingService_pb2 import JobStatus, JobType from feast.source import KafkaSource from feast.types import Value_pb2 as ValueProto from feast.value_type import ValueType @@ -307,131 +295,131 @@ def test_get_feature_set(self, mocked_client, mocker): and len(feature_set.entities) == 1 ) - @pytest.mark.parametrize( - "mocked_client", - [pytest.lazy_fixture("mock_client"), pytest.lazy_fixture("secure_mock_client")], - ) - def test_get_batch_features(self, mocked_client, mocker): - - mocked_client._serving_service_stub = Serving.ServingServiceStub( - grpc.insecure_channel("") - ) - mocked_client._core_service_stub = Core.CoreServiceStub( - grpc.insecure_channel("") - ) - - mocker.patch.object( - mocked_client._core_service_stub, - "GetFeatureSet", - return_value=GetFeatureSetResponse( - feature_set=FeatureSetProto( - spec=FeatureSetSpecProto( - name="customer_fs", - version=1, - project="my_project", - entities=[ - EntitySpecProto( - name="customer", value_type=ValueProto.ValueType.INT64 - ), - EntitySpecProto( - name="transaction", - value_type=ValueProto.ValueType.INT64, - ), - ], - features=[ - FeatureSpecProto( - name="customer_feature_1", - value_type=ValueProto.ValueType.FLOAT, - ), - FeatureSpecProto( - name="customer_feature_2", - value_type=ValueProto.ValueType.STRING, - ), - ], - ), - meta=FeatureSetMetaProto(status=FeatureSetStatusProto.STATUS_READY), - ) - ), - ) - - expected_dataframe = pd.DataFrame( - { - "datetime": [datetime.utcnow() for _ in range(3)], - "customer": [1001, 1002, 1003], - "transaction": [1001, 1002, 1003], - "my_project/customer_feature_1:1": [1001, 1002, 1003], - "my_project/customer_feature_2:1": [1001, 1002, 1003], - } - ) - - final_results = tempfile.mktemp() - to_avro(file_path_or_buffer=final_results, df=expected_dataframe) - - mocker.patch.object( - mocked_client._serving_service_stub, - "GetBatchFeatures", - return_value=GetBatchFeaturesResponse( - job=BatchFeaturesJob( - id="123", - type=JobType.JOB_TYPE_DOWNLOAD, - status=JobStatus.JOB_STATUS_DONE, - file_uris=[f"file://{final_results}"], - data_format=DataFormat.DATA_FORMAT_AVRO, - ) - ), - ) - - mocker.patch.object( - mocked_client._serving_service_stub, - "GetJob", - return_value=GetJobResponse( - job=BatchFeaturesJob( - id="123", - type=JobType.JOB_TYPE_DOWNLOAD, - status=JobStatus.JOB_STATUS_DONE, - file_uris=[f"file://{final_results}"], - data_format=DataFormat.DATA_FORMAT_AVRO, - ) - ), - ) - - mocker.patch.object( - mocked_client._serving_service_stub, - "GetFeastServingInfo", - return_value=GetFeastServingInfoResponse( - job_staging_location=f"file://{tempfile.mkdtemp()}/", - type=FeastServingType.FEAST_SERVING_TYPE_BATCH, - ), - ) - - mocked_client.set_project("project1") - response = mocked_client.get_batch_features( - entity_rows=pd.DataFrame( - { - "datetime": [ - pd.datetime.now(tz=timezone("Asia/Singapore")) for _ in range(3) - ], - "customer": [1001, 1002, 1003], - "transaction": [1001, 1002, 1003], - } - ), - feature_refs=[ - "my_project/customer_feature_1:1", - "my_project/customer_feature_2:1", - ], - ) # type: Job - - assert response.id == "123" and response.status == JobStatus.JOB_STATUS_DONE - - actual_dataframe = response.to_dataframe() - - assert actual_dataframe[ - ["my_project/customer_feature_1:1", "my_project/customer_feature_2:1"] - ].equals( - expected_dataframe[ - ["my_project/customer_feature_1:1", "my_project/customer_feature_2:1"] - ] - ) + # @pytest.mark.parametrize( + # "mocked_client", + # [pytest.lazy_fixture("mock_client"), pytest.lazy_fixture("secure_mock_client")], + # ) + # def test_get_batch_features(self, mocked_client, mocker): + # + # mocked_client._serving_service_stub = Serving.ServingServiceStub( + # grpc.insecure_channel("") + # ) + # mocked_client._core_service_stub = Core.CoreServiceStub( + # grpc.insecure_channel("") + # ) + # + # mocker.patch.object( + # mocked_client._core_service_stub, + # "GetFeatureSet", + # return_value=GetFeatureSetResponse( + # feature_set=FeatureSetProto( + # spec=FeatureSetSpecProto( + # name="customer_fs", + # version=1, + # project="my_project", + # entities=[ + # EntitySpecProto( + # name="customer", value_type=ValueProto.ValueType.INT64 + # ), + # EntitySpecProto( + # name="transaction", + # value_type=ValueProto.ValueType.INT64, + # ), + # ], + # features=[ + # FeatureSpecProto( + # name="customer_feature_1", + # value_type=ValueProto.ValueType.FLOAT, + # ), + # FeatureSpecProto( + # name="customer_feature_2", + # value_type=ValueProto.ValueType.STRING, + # ), + # ], + # ), + # meta=FeatureSetMetaProto(status=FeatureSetStatusProto.STATUS_READY), + # ) + # ), + # ) + # + # expected_dataframe = pd.DataFrame( + # { + # "datetime": [datetime.utcnow() for _ in range(3)], + # "customer": [1001, 1002, 1003], + # "transaction": [1001, 1002, 1003], + # "my_project/customer_feature_1:1": [1001, 1002, 1003], + # "my_project/customer_feature_2:1": [1001, 1002, 1003], + # } + # ) + # + # final_results = tempfile.mktemp() + # to_avro(file_path_or_buffer=final_results, df=expected_dataframe) + # + # mocker.patch.object( + # mocked_client._serving_service_stub, + # "GetBatchFeatures", + # return_value=GetBatchFeaturesResponse( + # job=BatchFeaturesJob( + # id="123", + # type=JobType.JOB_TYPE_DOWNLOAD, + # status=JobStatus.JOB_STATUS_DONE, + # file_uris=[f"file://{final_results}"], + # data_format=DataFormat.DATA_FORMAT_AVRO, + # ) + # ), + # ) + # + # mocker.patch.object( + # mocked_client._serving_service_stub, + # "GetJob", + # return_value=GetJobResponse( + # job=BatchFeaturesJob( + # id="123", + # type=JobType.JOB_TYPE_DOWNLOAD, + # status=JobStatus.JOB_STATUS_DONE, + # file_uris=[f"file://{final_results}"], + # data_format=DataFormat.DATA_FORMAT_AVRO, + # ) + # ), + # ) + # + # mocker.patch.object( + # mocked_client._serving_service_stub, + # "GetFeastServingInfo", + # return_value=GetFeastServingInfoResponse( + # job_staging_location=f"file://{tempfile.mkdtemp()}/", + # type=FeastServingType.FEAST_SERVING_TYPE_BATCH, + # ), + # ) + # + # mocked_client.set_project("project1") + # response = mocked_client.get_batch_features( + # entity_rows=pd.DataFrame( + # { + # "datetime": [ + # pd.datetime.now(tz=timezone("Asia/Singapore")) for _ in range(3) + # ], + # "customer": [1001, 1002, 1003], + # "transaction": [1001, 1002, 1003], + # } + # ), + # feature_refs=[ + # "my_project/customer_feature_1:1", + # "my_project/customer_feature_2:1", + # ], + # ) # type: Job + # + # assert response.id == "123" and response.status == JobStatus.JOB_STATUS_DONE + # + # actual_dataframe = response.to_dataframe() + # + # assert actual_dataframe[ + # ["my_project/customer_feature_1:1", "my_project/customer_feature_2:1"] + # ].equals( + # expected_dataframe[ + # ["my_project/customer_feature_1:1", "my_project/customer_feature_2:1"] + # ] + # ) @pytest.mark.parametrize( "test_client", From e8d6e9907fd943ac496cb843c29fd04f4a6ced18 Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Wed, 25 Mar 2020 15:37:42 +0800 Subject: [PATCH 088/176] Add index for join table for jobs-featuresets relation (#566) --- core/src/main/java/feast/core/model/Job.java | 23 ++++++++++---------- 1 file changed, 11 insertions(+), 12 deletions(-) diff --git a/core/src/main/java/feast/core/model/Job.java b/core/src/main/java/feast/core/model/Job.java index bbd661309d1..2fd17a58d74 100644 --- a/core/src/main/java/feast/core/model/Job.java +++ b/core/src/main/java/feast/core/model/Job.java @@ -17,17 +17,7 @@ package feast.core.model; import java.util.List; -import javax.persistence.CascadeType; -import javax.persistence.Column; -import javax.persistence.Entity; -import javax.persistence.EnumType; -import javax.persistence.Enumerated; -import javax.persistence.Id; -import javax.persistence.JoinColumn; -import javax.persistence.ManyToMany; -import javax.persistence.ManyToOne; -import javax.persistence.OneToMany; -import javax.persistence.Table; +import javax.persistence.*; import lombok.AllArgsConstructor; import lombok.Getter; import lombok.Setter; @@ -63,7 +53,16 @@ public class Job extends AbstractTimestampEntity { private Store store; // FeatureSets populated by the job - @ManyToMany private List featureSets; + @ManyToMany + @JoinTable( + name = "jobs_feature_sets", + joinColumns = @JoinColumn(name = "feature_sets_id"), + inverseJoinColumns = @JoinColumn(name = "job_id"), + indexes = { + @Index(name = "idx_jobs_feature_sets_job_id", columnList = "job_id"), + @Index(name = "idx_jobs_feature_sets_feature_sets_id", columnList = "feature_sets_id") + }) + private List featureSets; // Job Metrics @OneToMany(mappedBy = "job", cascade = CascadeType.ALL) From 9facd2e5f00c0616da2bb616664e619ee0100de6 Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Wed, 25 Mar 2020 15:51:42 +0800 Subject: [PATCH 089/176] Add pollingInterval config option (#565) * Add pollingInterval config option * Add missing prefix * Increase polling interval --- core/src/main/java/feast/core/config/FeastProperties.java | 1 + .../main/java/feast/core/service/JobCoordinatorService.java | 3 +-- core/src/main/resources/application.yml | 2 ++ infra/scripts/test-end-to-end-batch.sh | 1 + infra/scripts/test-end-to-end.sh | 1 + 5 files changed, 6 insertions(+), 2 deletions(-) diff --git a/core/src/main/java/feast/core/config/FeastProperties.java b/core/src/main/java/feast/core/config/FeastProperties.java index 1887caf5e64..b9c787b6c77 100644 --- a/core/src/main/java/feast/core/config/FeastProperties.java +++ b/core/src/main/java/feast/core/config/FeastProperties.java @@ -45,6 +45,7 @@ public static class JobProperties { public static class JobUpdatesProperties { private long timeoutSeconds; + private long pollingIntervalMillis; } @Getter diff --git a/core/src/main/java/feast/core/service/JobCoordinatorService.java b/core/src/main/java/feast/core/service/JobCoordinatorService.java index 3678135a526..b66d181022e 100644 --- a/core/src/main/java/feast/core/service/JobCoordinatorService.java +++ b/core/src/main/java/feast/core/service/JobCoordinatorService.java @@ -54,7 +54,6 @@ @Service public class JobCoordinatorService { - private final long POLLING_INTERVAL_MILLISECONDS = 60000; // 1 min private JobRepository jobRepository; private FeatureSetRepository featureSetRepository; private SpecService specService; @@ -87,7 +86,7 @@ public JobCoordinatorService( *

4) Updates Feature set statuses */ @Transactional - @Scheduled(fixedDelay = POLLING_INTERVAL_MILLISECONDS) + @Scheduled(fixedDelayString = "${feast.jobs.updates.pollingIntervalMillis}") public void Poll() throws InvalidProtocolBufferException { log.info("Polling for new jobs..."); List jobUpdateTasks = new ArrayList<>(); diff --git a/core/src/main/resources/application.yml b/core/src/main/resources/application.yml index dc78719f22e..ee060fffc95 100644 --- a/core/src/main/resources/application.yml +++ b/core/src/main/resources/application.yml @@ -31,6 +31,8 @@ feast: # Key-value dict of job options to be passed to the population jobs. options: {} updates: + # Job update polling interval in milliseconds: how often Feast checks if new jobs should be sent to the runner. + pollingIntervalMillis: 60000 # Timeout in seconds for each attempt to update or submit a new job to the runner. timeoutSeconds: 240 metrics: diff --git a/infra/scripts/test-end-to-end-batch.sh b/infra/scripts/test-end-to-end-batch.sh index 9bc17c2e757..35553a92814 100755 --- a/infra/scripts/test-end-to-end-batch.sh +++ b/infra/scripts/test-end-to-end-batch.sh @@ -120,6 +120,7 @@ feast: runner: DirectRunner options: {} updates: + pollingIntervalMillis: 30000 timeoutSeconds: 240 metrics: enabled: false diff --git a/infra/scripts/test-end-to-end.sh b/infra/scripts/test-end-to-end.sh index 3e6a5492a16..ee7304f208d 100755 --- a/infra/scripts/test-end-to-end.sh +++ b/infra/scripts/test-end-to-end.sh @@ -103,6 +103,7 @@ feast: runner: DirectRunner options: {} updates: + pollingIntervalMillis: 30000 timeoutSeconds: 240 metrics: enabled: false From 901e260477d6e2b3ccb1bbbe43cd9553cf7a3ad9 Mon Sep 17 00:00:00 2001 From: Zhu Zhan Yan Date: Fri, 27 Mar 2020 13:41:43 +0800 Subject: [PATCH 090/176] Fix runner to string inconsistency (#575) * Changed Runner.getName() to Runner.toString() when passing to Job.runner in DataflowJobManager This is necessary to standardise the use of Runner.toString() when passing to the Job.runner, so that code dependending on Job.runner would know what to expect. * Document how & when Runner.toString() or Runner.getName() should be used * Convert getName() to toString(). Use name() for Job.runner. Use toString() to render human readable strings while using the non overriding name() for code dependencies. Co-authored-by: Zhu Zhanyan --- .../java/feast/core/job/JobUpdateTask.java | 10 +++++----- core/src/main/java/feast/core/job/Runner.java | 10 ++++++++-- .../core/job/dataflow/DataflowJobManager.java | 6 +++--- core/src/main/java/feast/core/model/Job.java | 1 + .../java/feast/core/job/JobUpdateTaskTest.java | 18 +++++++++--------- .../job/dataflow/DataflowJobManagerTest.java | 4 ++-- .../job/direct/DirectRunnerJobManagerTest.java | 2 +- .../service/JobCoordinatorServiceTest.java | 12 ++++++------ 8 files changed, 35 insertions(+), 28 deletions(-) diff --git a/core/src/main/java/feast/core/job/JobUpdateTask.java b/core/src/main/java/feast/core/job/JobUpdateTask.java index 87578cce25a..f3afe84df77 100644 --- a/core/src/main/java/feast/core/job/JobUpdateTask.java +++ b/core/src/main/java/feast/core/job/JobUpdateTask.java @@ -144,7 +144,7 @@ private Job startJob( new Job( jobId, "", - jobManager.getRunnerType().toString(), + jobManager.getRunnerType().name(), Source.fromProto(source), Store.fromProto(sinkSpec), featureSets, @@ -155,7 +155,7 @@ private Job startJob( jobId, Action.SUBMIT, "Building graph and submitting to %s", - jobManager.getRunnerType().getName()); + jobManager.getRunnerType().toString()); job = jobManager.startJob(job); if (job.getExtId().isEmpty()) { @@ -168,7 +168,7 @@ private Job startJob( jobId, Action.STATUS_CHANGE, "Job submitted to runner %s with ext id %s.", - jobManager.getRunnerType().getName(), + jobManager.getRunnerType().toString(), job.getExtId()); return job; @@ -179,7 +179,7 @@ private Job startJob( jobId, Action.STATUS_CHANGE, "Job failed to be submitted to runner %s. Job status changed to ERROR.", - jobManager.getRunnerType().getName()); + jobManager.getRunnerType().toString()); job.setStatus(JobStatus.ERROR); return job; @@ -206,7 +206,7 @@ private Job updateJob( Action.UPDATE, "Updating job %s for runner %s", job.getId(), - jobManager.getRunnerType().getName()); + jobManager.getRunnerType().toString()); return jobManager.updateJob(job); } diff --git a/core/src/main/java/feast/core/job/Runner.java b/core/src/main/java/feast/core/job/Runner.java index 637621be359..4e2033fed69 100644 --- a/core/src/main/java/feast/core/job/Runner.java +++ b/core/src/main/java/feast/core/job/Runner.java @@ -27,13 +27,19 @@ public enum Runner { this.name = name; } - public String getName() { + /** + * Get the human readable name of this runner. Returns a human readable name of the runner that + * can be used for logging/config files/etc. + */ + @Override + public String toString() { return name; } + /** Parses a runner from its human readable name. */ public static Runner fromString(String runner) { for (Runner r : Runner.values()) { - if (r.getName().equals(runner)) { + if (r.toString().equals(runner)) { return r; } } diff --git a/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java b/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java index 323eb35983e..e76568dfb48 100644 --- a/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java +++ b/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java @@ -160,7 +160,7 @@ public void abortJob(String dataflowJobId) { */ @Override public JobStatus getJobStatus(Job job) { - if (!Runner.DATAFLOW.getName().equals(job.getRunner())) { + if (!Runner.DATAFLOW.name().equals(job.getRunner())) { return job.getStatus(); } @@ -191,7 +191,7 @@ private Job submitDataflowJob( .map( fsp -> { FeatureSet featureSet = new FeatureSet(); - featureSet.setName(fsp.getSpec().getName()); + featureSet.setName(fsp.getSpec().toString()); featureSet.setVersion(fsp.getSpec().getVersion()); featureSet.setProject(new Project(fsp.getSpec().getProject())); return featureSet; @@ -201,7 +201,7 @@ private Job submitDataflowJob( return new Job( jobName, jobId, - getRunnerType().getName(), + getRunnerType().name(), Source.fromProto(source), Store.fromProto(sink), featureSets, diff --git a/core/src/main/java/feast/core/model/Job.java b/core/src/main/java/feast/core/model/Job.java index 2fd17a58d74..377f5f70956 100644 --- a/core/src/main/java/feast/core/model/Job.java +++ b/core/src/main/java/feast/core/model/Job.java @@ -39,6 +39,7 @@ public class Job extends AbstractTimestampEntity { private String extId; // Runner type + // Use Runner.name() when converting a Runner to string to assign to this property. @Column(name = "runner") private String runner; diff --git a/core/src/test/java/feast/core/job/JobUpdateTaskTest.java b/core/src/test/java/feast/core/job/JobUpdateTaskTest.java index 19ce0858b20..2a1e80994ae 100644 --- a/core/src/test/java/feast/core/job/JobUpdateTaskTest.java +++ b/core/src/test/java/feast/core/job/JobUpdateTaskTest.java @@ -102,7 +102,7 @@ public void shouldUpdateJobIfPresent() { new Job( "job", "old_ext", - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), @@ -119,7 +119,7 @@ public void shouldUpdateJobIfPresent() { new Job( "job", "old_ext", - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1), FeatureSet.fromProto(featureSet2)), @@ -129,7 +129,7 @@ public void shouldUpdateJobIfPresent() { new Job( "job", "new_ext", - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), Source.fromProto(source), Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1), FeatureSet.fromProto(featureSet2)), @@ -163,7 +163,7 @@ public void shouldCreateJobIfNotPresent() { new Job( "job", "", - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), @@ -173,7 +173,7 @@ public void shouldCreateJobIfNotPresent() { new Job( "job", "ext", - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), @@ -202,7 +202,7 @@ public void shouldUpdateJobStatusIfNotCreateOrUpdate() { new Job( "job", "ext", - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), @@ -216,7 +216,7 @@ public void shouldUpdateJobStatusIfNotCreateOrUpdate() { new Job( "job", "ext", - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), Source.fromProto(source), Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), @@ -248,7 +248,7 @@ public void shouldReturnJobWithErrorStatusIfFailedToSubmit() { new Job( "job", "", - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), @@ -258,7 +258,7 @@ public void shouldReturnJobWithErrorStatusIfFailedToSubmit() { new Job( "job", "", - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), diff --git a/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java b/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java index 9f26c6919e4..2d562d38df2 100644 --- a/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java +++ b/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java @@ -145,7 +145,7 @@ public void shouldStartJobWithCorrectPipelineOptions() throws IOException { new Job( jobName, "", - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), Source.fromProto(source), Store.fromProto(store), Lists.newArrayList(FeatureSet.fromProto(featureSet)), @@ -226,7 +226,7 @@ public void shouldThrowExceptionWhenJobStateTerminal() throws IOException { new Job( "job", "", - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), Source.fromProto(source), Store.fromProto(store), Lists.newArrayList(FeatureSet.fromProto(featureSet)), diff --git a/core/src/test/java/feast/core/job/direct/DirectRunnerJobManagerTest.java b/core/src/test/java/feast/core/job/direct/DirectRunnerJobManagerTest.java index 64412f4391e..76530d9f404 100644 --- a/core/src/test/java/feast/core/job/direct/DirectRunnerJobManagerTest.java +++ b/core/src/test/java/feast/core/job/direct/DirectRunnerJobManagerTest.java @@ -144,7 +144,7 @@ public void shouldStartDirectJobAndRegisterPipelineResult() throws IOException { new Job( expectedJobId, "", - Runner.DIRECT.getName(), + Runner.DIRECT.name(), Source.fromProto(source), Store.fromProto(store), Lists.newArrayList(FeatureSet.fromProto(featureSet)), diff --git a/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java b/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java index 67a87e93167..aa71f201dde 100644 --- a/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java +++ b/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java @@ -161,7 +161,7 @@ public void shouldGenerateAndSubmitJobsIfAny() throws InvalidProtocolBufferExcep new Job( "", "", - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1), FeatureSet.fromProto(featureSet2)), @@ -171,7 +171,7 @@ public void shouldGenerateAndSubmitJobsIfAny() throws InvalidProtocolBufferExcep new Job( "some_id", extId, - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1), FeatureSet.fromProto(featureSet2)), @@ -261,7 +261,7 @@ public void shouldGroupJobsBySource() throws InvalidProtocolBufferException { new Job( "name1", "", - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), feast.core.model.Source.fromProto(source1), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), @@ -271,7 +271,7 @@ public void shouldGroupJobsBySource() throws InvalidProtocolBufferException { new Job( "name1", "extId1", - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), feast.core.model.Source.fromProto(source1), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), @@ -281,7 +281,7 @@ public void shouldGroupJobsBySource() throws InvalidProtocolBufferException { new Job( "", "extId2", - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), feast.core.model.Source.fromProto(source2), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet2)), @@ -291,7 +291,7 @@ public void shouldGroupJobsBySource() throws InvalidProtocolBufferException { new Job( "name2", "extId2", - Runner.DATAFLOW.getName(), + Runner.DATAFLOW.name(), feast.core.model.Source.fromProto(source2), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet2)), From 90debcab2d63f6ad4a5c209064d35fb061ef0598 Mon Sep 17 00:00:00 2001 From: Zhu Zhan Yan Date: Fri, 27 Mar 2020 22:37:42 +0800 Subject: [PATCH 091/176] Fixed bug in featurset proto to model conversion in DataflowJobManager (#578) Bug caused by toString() method being used in the conversion lambda instead of the getName() method. Since lambda duplicates functionality already implemented by Feature.fromProto, removing the lambda block in favor using the method to do the conversion. Co-authored-by: Zhu Zhanyan --- .../feast/core/job/dataflow/DataflowJobManager.java | 11 +---------- 1 file changed, 1 insertion(+), 10 deletions(-) diff --git a/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java b/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java index e76568dfb48..f4df3d352a9 100644 --- a/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java +++ b/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java @@ -187,16 +187,7 @@ private Job submitDataflowJob( ImportOptions pipelineOptions = getPipelineOptions(jobName, featureSetProtos, sink, update); DataflowPipelineJob pipelineResult = runPipeline(pipelineOptions); List featureSets = - featureSetProtos.stream() - .map( - fsp -> { - FeatureSet featureSet = new FeatureSet(); - featureSet.setName(fsp.getSpec().toString()); - featureSet.setVersion(fsp.getSpec().getVersion()); - featureSet.setProject(new Project(fsp.getSpec().getProject())); - return featureSet; - }) - .collect(Collectors.toList()); + featureSetProtos.stream().map(FeatureSet::fromProto).collect(Collectors.toList()); String jobId = waitForJobToRun(pipelineResult); return new Job( jobName, From 8098bcc3b67c44d698945acf83671b70841504ac Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Fri, 27 Mar 2020 22:43:54 +0800 Subject: [PATCH 092/176] Fix slack link. Fix broken Gitbook documentation. --- README.md | 2 +- docs/concepts.md | 124 ------- docs/contributing.md | 542 ------------------------------ docs/getting-help.md | 36 -- docs/introduction/getting-help.md | 2 +- docs/roadmap.md | 50 --- docs/troubleshooting.md | 168 --------- docs/why-feast.md | 34 -- 8 files changed, 2 insertions(+), 956 deletions(-) delete mode 100644 docs/concepts.md delete mode 100644 docs/contributing.md delete mode 100644 docs/getting-help.md delete mode 100644 docs/roadmap.md delete mode 100644 docs/troubleshooting.md delete mode 100644 docs/why-feast.md diff --git a/README.md b/README.md index 63b1d45d389..b95e66cc9fb 100644 --- a/README.md +++ b/README.md @@ -51,7 +51,7 @@ Please refer to the official documentation at * [Examples](https://github.com/gojek/feast/blob/master/examples/) * [Roadmap](https://docs.feast.dev/roadmap) * [Change Log](https://github.com/gojek/feast/blob/master/CHANGELOG.md) - * [Slack (#Feast)](https://join.slack.com/t/kubeflow/shared_invite/enQtNDg5MTM4NTQyNjczLTdkNTVhMjg1ZTExOWI0N2QyYTQ2MTIzNTJjMWRiOTFjOGRlZWEzODc1NzMwNTMwM2EzNjY1MTFhODczNjk4MTk) + * [Slack (#Feast)](https://join.slack.com/t/kubeflow/shared_invite/zt-cpr020z4-PfcAue_2nw67~iIDy7maAQ) ## Notice diff --git a/docs/concepts.md b/docs/concepts.md deleted file mode 100644 index ae158f8f829..00000000000 --- a/docs/concepts.md +++ /dev/null @@ -1,124 +0,0 @@ -# Concepts - -## Architecture - -![Logical diagram of a typical Feast deployment](.gitbook/assets/basic-architecture-diagram%20%282%29.svg) - -The core components of a Feast deployment are - -* **Feast Core:** Feast Core is a centralized service that acts as the authority on features within an organization. Typically there is only one "Core" deployment per organization, with all feature management happening through it. -* **Feast Ingestion Jobs:** Feast ingestion jobs retrieve feature data from user defined data sources and populate serving stores with this feature data. These jobs are managed by Feast Core. Data can either be sources from existing sources \(like [Kafka](https://kafka.apache.org/)\), or it can be loaded into Feast through its API. -* **Feast Serving:** Feast Serving is the data access layer through which end users and production systems retrieve feature data. Each Serving store is backed by one or more databases. These databases are updated by the Feast ingestion jobs. There are two types of stores: batch and online. Batch stores hold large volumes historical data, while online stores only hold the latest feature values. - -## Data Model - -### Feature Set - -User data is typically in the form of dataframes, tables in data warehouses, or events on a stream. These data sources are loaded into Feast in order to serve features for model training or serving. - -Feature sets allow for groups of fields in these data sources to be ingested and stored together. This allows for efficient storage and logical namespacing of data. - -When data is loaded from these sources, each field in the feature set must be found in every record of the data source. Fields from these data sources must be either a timestamp, an entity, or a feature. - -{% hint style="info" %} -Feature sets are a grouping of feature sets based on how they are loaded into Feast. They ensure that data is efficiently stored during ingestion. Feature sets are not a grouping of features for retrieval of features. During retrieval it is possible to retrieve feature values from any number of feature sets. -{% endhint %} - -#### Customer Transactions Example - -Below is an example of a basic `customer transactions` feature set that has been exported to YAML: - -{% tabs %} -{% tab title="customer\_transactions\_feature\_set.yaml" %} -```yaml -name: customer_transactions -kind: feature_set -entities: -- name: customer_id - valueType: INT64 -features: -- name: daily_transactions - valueType: FLOAT -- name: total_transactions - valueType: FLOAT - maxAge: 3600s -``` -{% endtab %} -{% endtabs %} - -The dataframe below \(`customer_data.csv`\) contains the features and entities of the above feature set - -| datetime | customer\_id | daily\_transactions | total\_tra**nsactions** | -| :--- | :--- | :--- | :--- | -| 2019-01-01 01:00:00 | 20001 | 5.0 | 14.0 | -| 2019-01-01 01:00:00 | 20002 | 2.6 | 43.0 | -| 2019-01-01 01:00:00 | 20003 | 4.1 | 154.0 | -| 2019-01-01 01:00:00 | 20004 | 3.4 | 74.0 | - -In order to ingest feature data into Feast for this specific feature set: - -```python -# Load dataframe -customer_df = pd.read_csv("customer_data.csv") - -# Create feature set from YAML (using YAML is optional) -cust_trans_fs = FeatureSet.from_yaml("customer_transactions_feature_set.yaml") - -# Load feature data into Feast for this specific feature set -client.ingest(cust_trans_fs, customer_data) -``` - -### Feature - -A feature is an individual measurable property or characteristic of a phenomenon being observed. Features are the most important concepts within a feature store. Feature data is used both as input to models during training and when models are served in production. - -In the context of Feast, features are values that are associated with either one or more entities over time. In Feast, these values are either primitives or lists of primitives. Each feature can also have additional information attached to it. For example whether it is a categorical feature or numerical. - -{% hint style="info" %} -Features in Feast are defined within Feature Sets and are not treated as standalone concepts. -{% endhint %} - -### Entity - -An entity type is any object in an organization that needs to be modeled and on which information should be stored. Entity types are usually recognizable concepts, either concrete or abstract, such as persons, places, things, or events which have relevance to the modeled system. - -An entity is an instance of an entity type. - -* Examples of entity types in the context of ride-hailing and food delivery: `customer`, `order`, `driver`, `restaurant`, `dish`, `area`. -* A specific driver, for example a driver with ID `D011234` would be an entity of the entity type `driver` - -An entity is the object on which features are observed. For example we could have a feature `total_trips_24h` on the driver `D01123` with a feature value of `11`. - -In the context of Feast, entities are important because they are used as keys when looking up feature values. Entities are also used when joining feature values between different feature sets in order to build one large data set to train a model, or to serve a model. - -{% hint style="info" %} -Entities in Feast are defined within Feature Sets and are not treated as standalone concepts. -{% endhint %} - -### Types - -Feast supports the following types for feature values - -* BYTES -* STRING -* INT32 -* INT64 -* DOUBLE -* FLOAT -* BOOL -* BYTES\_LIST -* STRING\_LIST -* INT32\_LIST -* INT64\_LIST -* DOUBLE\_LIST -* FLOAT\_LIST -* BOOL\_LIST - -## Glossary - -| Term | Description | -| :--- | :--- | -| Feast deployment | A complete Feast system as it is deployed. Consists out of a single Feast Core deployment and one or more Feast Serving deployments. | -| Feast Core | The centralized service which acts as a registry and authority of features. Organizations should only deploy a single Feast Core instance. Feast Core also manages the ingestion of feature data and population of Feast Serving data stores. | -| Feast Serving | Feast Serving is a service used to access both online and batch feature data. Feast Serving deployments are backed by one or more databases. | - diff --git a/docs/contributing.md b/docs/contributing.md deleted file mode 100644 index b451de39ab7..00000000000 --- a/docs/contributing.md +++ /dev/null @@ -1,542 +0,0 @@ -# Contributing - -## 1. Contribution process - -We use [RFCs](https://en.wikipedia.org/wiki/Request_for_Comments) and [GitHub issues](https://github.com/gojek/feast/issues) to communicate development ideas. The simplest way to contribute to Feast is to leave comments in our [RFCs](https://drive.google.com/drive/u/0/folders/1Lj1nIeRB868oZvKTPLYqAvKQ4O0BksjY) in the [Feast Google Drive](https://drive.google.com/drive/u/0/folders/0AAe8j7ZK3sxSUk9PVA) or our GitHub issues. - -Please communicate your ideas through a GitHub issue or through our Slack Channel before starting development. - -Please [submit a PR ](https://github.com/gojek/feast/pulls)to the master branch of the Feast repository once you are ready to submit your contribution. Code submission to Feast \(including submission from project maintainers\) require review and approval from maintainers or code owners. - -PRs that are submitted by the general public need to be identified as `ok-to-test`. Once enabled, [Prow](https://github.com/kubernetes/test-infra/tree/master/prow) will run a range of tests to verify the submission, after which community members will help to review the pull request. - -{% hint style="success" %} -Please sign the [Google CLA](https://cla.developers.google.com/) in order to have your code merged into the Feast repository. -{% endhint %} - -## 2. Development guide - -### 2.1 Overview - -The following guide will help you quickly run Feast in your local machine. - -The main components of Feast are: - -* **Feast Core:** Handles feature registration, starts and manages ingestion jobs and ensures that Feast internal metadata is consistent. -* **Feast Ingestion Jobs:** Subscribes to streams of FeatureRows and writes these as feature - - values to registered databases \(online, historical\) that can be read by Feast Serving. - -* **Feast Serving:** Service that handles requests for features values, either online or batch. - -### 2.**2 Requirements** - -#### 2.**2.1 Development environment** - -The following software is required for Feast development - -* Java SE Development Kit 11 -* Python version 3.6 \(or above\) and pip -* [Maven ](https://maven.apache.org/install.html)version 3.6.x - -Additionally, [grpc\_cli](https://github.com/grpc/grpc/blob/master/doc/command_line_tool.md) is useful for debugging and quick testing of gRPC endpoints. - -#### 2.**2.2 Services** - -The following components/services are required to develop Feast: - -* **Feast Core:** Requires PostgreSQL \(version 11 and above\) to store state, and requires a Kafka \(tested on version 2.x\) setup to allow for ingestion of FeatureRows. -* **Feast Serving:** Requires Redis \(tested on version 5.x\). - -These services should be running before starting development. The following snippet will start the services using Docker. - -```bash -# Start Postgres -docker run --name postgres --rm -it -d --net host -e POSTGRES_DB=postgres -e POSTGRES_USER=postgres \ --e POSTGRES_PASSWORD=password postgres:12-alpine - -# Start Redis -docker run --name redis --rm -it --net host -d redis:5-alpine - -# Start Zookeeper (needed by Kafka) -docker run --rm \ - --net=host \ - --name=zookeeper \ - --env=ZOOKEEPER_CLIENT_PORT=2181 \ - --detach confluentinc/cp-zookeeper:5.2.1 - -# Start Kafka -docker run --rm \ - --net=host \ - --name=kafka \ - --env=KAFKA_ZOOKEEPER_CONNECT=localhost:2181 \ - --env=KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092 \ - --env=KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \ - --detach confluentinc/cp-kafka:5.2.1 -``` - -### 2.3 Testing and development - -#### 2.3.1 Running unit tests - -```text -$ mvn test -``` - -#### 2.3.2 Running integration tests - -_Note: integration suite isn't yet separated from unit._ - -```text -$ mvn verify -``` - -#### 2.3.3 Running components locally - -The `core` and `serving` modules are Spring Boot applications. These may be run as usual for [the Spring Boot Maven plugin](https://docs.spring.io/spring-boot/docs/current/maven-plugin/index.html): - -```text -$ mvn --projects core spring-boot:run - -# Or for short: -$ mvn -pl core spring-boot:run -``` - -Note that you should execute `mvn` from the Feast repository root directory, as there are intermodule dependencies that Maven will not resolve if you `cd` to subdirectories to run. - -#### 2.3.4 Running from IntelliJ - -Compiling and running tests in IntelliJ should work as usual. - -Running the Spring Boot apps may work out of the box in IDEA Ultimate, which has built-in support for Spring Boot projects, but the Community Edition needs a bit of help: - -The Spring Boot Maven plugin automatically puts dependencies with `provided` scope on the runtime classpath when using `spring-boot:run`, such as its embedded Tomcat server. The "Play" buttons in the gutter or right-click menu of a `main()` method [do not do this](https://stackoverflow.com/questions/30237768/run-spring-boots-main-using-ide). - -A solution to this is: - -1. Open `View > Tool Windows > Maven` -2. Drill down to e.g. `Feast Core > Plugins > spring-boot:run`, right-click and `Create 'feast-core [spring-boot'…` -3. In the dialog that pops up, check the `Resolve Workspace artifacts` box -4. Recommended: add `-Dspring-boot.run.fork=false` to the `Command line` field to get Debug working too -5. Click `OK`. You should now be able to select this run configuration for the Play button in the main toolbar, keyboard shortcuts, etc. - -It is recommend to have IntelliJ delegate building to Maven, if this is not enabled out of the box when you import the project, for greater assurance that build behavior is consistent with CI / production builds. This is set in Preferences at `Build, Execution, Deployment > Build Tools > Maven > Runner > Delegate IDE build/run actions to Maven`. - -### 2.**4** Validating your setup - -The following section is a quick walk-through to test whether your local Feast deployment is functional for development purposes. - -**2.4.1 Assumptions** - -* PostgreSQL is running in `localhost:5432` and has a database called `postgres` which - - can be accessed with credentials user `postgres` and password `password`. Different database configurations can be supplied here \(`/core/src/main/resources/application.yml`\) - -* Redis is running locally and accessible from `localhost:6379` -* \(optional\) The local environment has been authentication with Google Cloud Platform and has full access to BigQuery. This is only necessary for BigQuery testing/development. - -#### 2.4.2 Clone Feast - -```bash -git clone https://github.com/gojek/feast.git && cd feast && \ -export FEAST_HOME_DIR=$(pwd) -``` - -#### 2.4.3 Starting Feast Core - -To run Feast Core locally using Maven: - -```bash -# Feast Core can be configured from the following .yml file -# $FEAST_HOME_DIR/core/src/main/resources/application.yml -mvn --projects core spring-boot:run -``` - -Test whether Feast Core is running - -```text -grpc_cli call localhost:6565 ListStores '' -``` - -The output should list **no** stores since no Feast Serving has registered its stores to Feast Core: - -```text -connecting to localhost:6565 - -Rpc succeeded with OK status -``` - -#### 2.4.4 Starting Feast Serving - -Feast Serving is configured through the `$FEAST_HOME_DIR/serving/src/main/resources/application.yml`. Each Serving deployment must be configured with a store. The default store is Redis \(used for online serving\). - -The configuration for this default store is located in a separate `.yml` file. The default location is `$FEAST_HOME_DIR/serving/sample_redis_config.yml`: - -```text -name: serving -type: REDIS -redis_config: - host: localhost - port: 6379 -subscriptions: - - name: "*" - project: "*" - version: "*" -``` - -Once Feast Serving is started, it will register its store with Feast Core \(by name\) and start to subscribe to a feature sets based on its subscription. - -Start Feast Serving GRPC server on localhost:6566 with store name `serving` - -```text -mvn --projects serving spring-boot:run -``` - -Test connectivity to Feast Serving - -```text -grpc_cli call localhost:6566 GetFeastServingInfo '' -``` - -```text -connecting to localhost:6566 -version: "0.4.2-SNAPSHOT" -type: FEAST_SERVING_TYPE_ONLINE - -Rpc succeeded with OK status -``` - -Test Feast Core to see whether it is aware of the Feast Serving deployment - -```text -grpc_cli call localhost:6565 ListStores '' -``` - -```text -connecting to localhost:6565 -store { - name: "serving" - type: REDIS - subscriptions { - name: "*" - version: "*" - project: "*" - } - redis_config { - host: "localhost" - port: 6379 - } -} - -Rpc succeeded with OK status -``` - -In order to use BigQuery as a historical store, it is necessary to start Feast Serving with a different store type. - -Copy `$FEAST_HOME_DIR/serving/sample_redis_config.yml` to the following location `$FEAST_HOME_DIR/serving/my_bigquery_config.yml` and update the configuration as below: - -```text -name: bigquery -type: BIGQUERY -bigquery_config: - project_id: YOUR_GCP_PROJECT_ID - dataset_id: YOUR_GCP_DATASET -subscriptions: - - name: "*" - version: "*" - project: "*" -``` - -Then inside `serving/src/main/resources/application.yml` modify the following key `feast.store.config-path` to point to the new store configuration. - -After making these changes, restart Feast Serving: - -```text -mvn --projects serving spring-boot:run -``` - -You should see two stores registered: - -```text -store { - name: "serving" - type: REDIS - subscriptions { - name: "*" - version: "*" - project: "*" - } - redis_config { - host: "localhost" - port: 6379 - } -} -store { - name: "bigquery" - type: BIGQUERY - subscriptions { - name: "*" - version: "*" - project: "*" - } - bigquery_config { - project_id: "my_project" - dataset_id: "my_bq_dataset" - } -} -``` - -#### 2.4.5 Registering a FeatureSet - -Before registering a new FeatureSet, a project is required. - -```text -grpc_cli call localhost:6565 CreateProject ' - name: "your_project_name" -' -``` - -When a feature set is successfully registered, Feast Core will start an **ingestion** job that listens for new features in the feature set. - -{% hint style="info" %} -Note that Feast currently only supports source of type `KAFKA`, so you must have access to a running Kafka broker to register a FeatureSet successfully. It is possible to omit the `source` from a Feature Set, but Feast Core will still use Kafka behind the scenes, it is simply abstracted away from the user. -{% endhint %} - -Create a new FeatureSet in Feast by sending a request to Feast Core: - -```text -# Example of registering a new driver feature set -# Note the source value, it assumes that you have access to a Kafka broker -# running on localhost:9092 - -grpc_cli call localhost:6565 ApplyFeatureSet ' -feature_set { - spec { - project: "your_project_name" - name: "driver" - version: 1 - - entities { - name: "driver_id" - value_type: INT64 - } - - features { - name: "city" - value_type: STRING - } - - source { - type: KAFKA - kafka_source_config { - bootstrap_servers: "localhost:9092" - topic: "your-kafka-topic" - } - } - } -} -' -``` - -Verify that the FeatureSet has been registered correctly. - -```text -# To check that the FeatureSet has been registered correctly. -# You should also see logs from Feast Core of the ingestion job being started -grpc_cli call localhost:6565 GetFeatureSet ' - project: "your_project_name" - name: "driver" -' -``` - -Or alternatively, list all feature sets - -```text -grpc_cli call localhost:6565 ListFeatureSets ' - filter { - project: "your_project_name" - feature_set_name: "driver" - feature_set_version: "1" - } -' -``` - -#### 2.4.6 Ingestion and Population of Feature Values - -```text -# Produce FeatureRow messages to Kafka so it will be ingested by Feast -# and written to the registered stores. -# Make sure the value here is the topic assigned to the feature set -# ... producer.send("feast-driver-features" ...) -# -# Install Python SDK to help writing FeatureRow messages to Kafka -cd $FEAST_HOMEDIR/sdk/python -pip3 install -e . -pip3 install pendulum - -# Produce FeatureRow messages to Kafka so it will be ingested by Feast -# and written to the corresponding store. -# Make sure the value here is the topic assigned to the feature set -# ... producer.send("feast-test_feature_set-features" ...) -python3 - <8888/tcp feast_jupyter_1 -8e49dbe81b92 gcr.io/kf-feast/feast-serving:latest "java -Xms1024m -Xmx…" 2 minutes ago Up 5 seconds 0.0.0.0:6567->6567/tcp feast_batch-serving_1 -b859494bd33a gcr.io/kf-feast/feast-serving:latest "java -jar /opt/feas…" 2 minutes ago Up About a minute 0.0.0.0:6566->6566/tcp feast_online-serving_1 -5c4962811767 gcr.io/kf-feast/feast-core:latest "java -jar /opt/feas…" 2 minutes ago Up 2 minutes 0.0.0.0:6565->6565/tcp feast_core_1 -1ba7239e0ae0 confluentinc/cp-kafka:5.2.1 "/etc/confluent/dock…" 2 minutes ago Up 2 minutes 0.0.0.0:9092->9092/tcp, 0.0.0.0:9094->9094/tcp feast_kafka_1 -e2779672735c confluentinc/cp-zookeeper:5.2.1 "/etc/confluent/dock…" 2 minutes ago Up 2 minutes 2181/tcp, 2888/tcp, 3888/tcp feast_zookeeper_1 -39ac26f5c709 postgres:12-alpine "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 5432/tcp feast_db_1 -3c4ee8616096 redis:5-alpine "docker-entrypoint.s…" 2 minutes ago Up 2 minutes 0.0.0.0:6379->6379/tcp feast_redis_1 -``` - -### Google Kubernetes Engine - -All services should either be in a `running` state or `complete`state: - -```text -kubectl get pods -``` - -```text -NAME READY STATUS RESTARTS AGE -feast-feast-core-5ff566f946-4wlbh 1/1 Running 1 32m -feast-feast-serving-batch-848d74587b-96hq6 1/1 Running 2 32m -feast-feast-serving-online-df69755d5-fml8v 1/1 Running 2 32m -feast-kafka-0 1/1 Running 1 32m -feast-kafka-1 1/1 Running 0 30m -feast-kafka-2 1/1 Running 0 29m -feast-kafka-config-3e860262-zkzr8 0/1 Completed 0 32m -feast-postgresql-0 1/1 Running 0 32m -feast-prometheus-statsd-exporter-554db85b8d-r4hb8 1/1 Running 0 32m -feast-redis-master-0 1/1 Running 0 32m -feast-zookeeper-0 1/1 Running 0 32m -feast-zookeeper-1 1/1 Running 0 32m -feast-zookeeper-2 1/1 Running 0 31m -``` - -## How can I verify that I can connect to all services? - -First find the `IP:Port` combination of your services. - -### **Docker Compose \(from inside the docker cluster\)** - -You will probably need to connect using the hostnames of services and standard Feast ports: - -```bash -export FEAST_CORE_URL=core:6565 -export FEAST_ONLINE_SERVING_URL=online-serving:6566 -export FEAST_BATCH_SERVING_URL=batch-serving:6567 -``` - -### **Docker Compose \(from outside the docker cluster\)** - -You will probably need to connect using `localhost` and standard ports: - -```bash -export FEAST_CORE_URL=localhost:6565 -export FEAST_ONLINE_SERVING_URL=localhost:6566 -export FEAST_BATCH_SERVING_URL=localhost:6567 -``` - -### **Google Kubernetes Engine \(GKE\)** - -You will need to find the external IP of one of the nodes as well as the NodePorts. Please make sure that your firewall is open for these ports: - -```bash -export FEAST_IP=$(kubectl describe nodes | grep ExternalIP | awk '{print $2}' | head -n 1) -export FEAST_CORE_URL=${FEAST_IP}:32090 -export FEAST_ONLINE_SERVING_URL=${FEAST_IP}:32091 -export FEAST_BATCH_SERVING_URL=${FEAST_IP}:32092 -``` - -`netcat`, `telnet`, or even `curl` can be used to test whether all services are available and ports are open, but `grpc_cli` is the most powerful. It can be installed from [here](https://github.com/grpc/grpc/blob/master/doc/command_line_tool.md). - -### Testing Feast Core: - -```bash -grpc_cli ls ${FEAST_CORE_URL} feast.core.CoreService -``` - -```text -GetFeastCoreVersion -GetFeatureSet -ListFeatureSets -ListStores -ApplyFeatureSet -UpdateStore -CreateProject -ArchiveProject -ListProjects -``` - -### Testing Feast Batch Serving and Online Serving - -```bash -grpc_cli ls ${FEAST_BATCH_SERVING_URL} feast.serving.ServingService -``` - -```text -GetFeastServingInfo -GetOnlineFeatures -GetBatchFeatures -GetJob -``` - -```bash -grpc_cli ls ${FEAST_ONLINE_SERVING_URL} feast.serving.ServingService -``` - -```text -GetFeastServingInfo -GetOnlineFeatures -GetBatchFeatures -GetJob -``` - -## How can I print logs from the Feast Services? - -Feast will typically have three services that you need to monitor if something goes wrong. - -* Feast Core -* Feast Serving \(Online\) -* Feast Serving \(Batch\) - -In order to print the logs from these services, please run the commands below. - -### Docker Compose - -```text - docker logs -f feast_core_1 -``` - -```text -docker logs -f feast_batch-serving_1 -``` - -```text -docker logs -f feast_online-serving_1 -``` - -### Google Kubernetes Engine - -```text -kubectl logs $(kubectl get pods | grep feast-core | awk '{print $1}') -``` - -```text -kubectl logs $(kubectl get pods | grep feast-serving-batch | awk '{print $1}') -``` - -```text -kubectl logs $(kubectl get pods | grep feast-serving-online | awk '{print $1}') -``` - diff --git a/docs/why-feast.md b/docs/why-feast.md deleted file mode 100644 index f5551d5f82f..00000000000 --- a/docs/why-feast.md +++ /dev/null @@ -1,34 +0,0 @@ -# Why Feast? - -## Lack of feature reuse - -**Problem:** The process of engineering features is one of the most time consuming activities in building an end-to-end ML system. Despite this, many teams continue to redevelop the same features from scratch for every new project. Often these features never leaving the notebooks or pipelines they are built in. - -**Solution:** A centralized feature store allows organizations to build up a foundation of features that can be reused across projects. Teams are then able to utilize features developed by other teams, and as more features are added to the store it becomes easier and cheaper to build models. - -## Serving features is hard - -**Problem:** Serving up to date features at scale is hard. Raw data can come from a variety of sources, from data lakes, to even streams, to data warehouse, to simply flat files. Data scientists need the ability to produce massive datasets of features from this data in order to train their models offline. These models then need access to real-time feature data at low latency and high throughput when they are served in production. - -**Solution:** Feast is built to be able to ingest data from a variety of sources, supporting both streaming and batch sources. Once data is loaded into Feast as features, they become available through both a batch serving API as well as an real-time \(online serving\) API. These APIs allows data scientists and ML engineers to easily retrieve feature data for their development, training, or in production. Feast also comes with a Java, Go, and Python SDK to make this experience easy. - -## **Models need point-in-time correctness** - -**Problem:** Most data sources are not built with ML use cases in mind and by extension don't provide point-in-time correct lookups of feature data. One of the reasons why features are often re-engineered is because ML practitioners need to ensure that their models are trained on a dataset that accurately models the state of the world when the model runs in production. - -**Solution:** Feast allows end users to create point-in-time correct datasets across multiple entities. Feast ensures that there is no data leakage, that cross feature set joins are valid, and that models are not fed expired data. - -## Definitions of features vary - -**Problem:** Teams define features differently and there is no easy access to the documentation of a feature. - -**Solution:** Feast becomes the single source of truth for all feature data for all models within an organization. Teams are able to capture documentation, metadata and metrics about features. This allows teams to communicate clearly about features, test features data, and determine if a feature is useful for a particular model. - -## **Inconsistency between training and serving** - -**Problem:** Training requires access to historical data, whereas models that serve predictions need the latest values. Inconsistencies arise when data is siloed into many independent systems requiring separate tooling. Often teams are using Python for creating batch features off line, but these features are redeveloped with different libraries and languages when moving to serving or streaming systems. - -**Solution:** Feast provides consistency by managing and unifying the ingestion of data from batch and streaming sources into both the feature warehouse and feature serving stores. Feast becomes the bridge between your model and your data, both for training and serving. This ensures that there is a consistency in the feature data that your model receives. - -\*\*\*\* - From 75f3b783e5a7c5e0217a3020422548fb0d0ce0bf Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Fri, 27 Mar 2020 22:54:35 +0800 Subject: [PATCH 093/176] Fix folders/naming for documentation --- README.md | 4 +- docs/SUMMARY.md | 2 +- .../troubleshooting.md | 0 docs/installing-feast/docker-compose.md | 112 ---------- docs/installing-feast/gke.md | 211 ------------------ docs/installing-feast/overview.md | 14 -- 6 files changed, 3 insertions(+), 340 deletions(-) rename docs/{installing-feast => administration}/troubleshooting.md (100%) delete mode 100644 docs/installing-feast/docker-compose.md delete mode 100644 docs/installing-feast/gke.md delete mode 100644 docs/installing-feast/overview.md diff --git a/README.md b/README.md index b95e66cc9fb..3c9ec929b79 100644 --- a/README.md +++ b/README.md @@ -47,7 +47,7 @@ Please refer to the official documentation at * [Why Feast?](https://docs.feast.dev/why-feast) * [Concepts](https://docs.feast.dev/concepts) - * [Installation](https://docs.feast.dev/installing-feast/overview) + * [Installation](https://docs.feast.dev/installation/overview) * [Examples](https://github.com/gojek/feast/blob/master/examples/) * [Roadmap](https://docs.feast.dev/roadmap) * [Change Log](https://github.com/gojek/feast/blob/master/CHANGELOG.md) @@ -55,4 +55,4 @@ Please refer to the official documentation at ## Notice -Feast is a community project and is still under active development. Your feedback and contributions are important to us. Please have a look at our [contributing guide](docs/contributing.md) for details. +Feast is a community project and is still under active development. Your feedback and contributions are important to us. Please have a look at our [contributing guide](docs/contributing/contributing.md) for details. diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index 885be61a2e8..d5c844a1810 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -25,7 +25,7 @@ ## Administration -* [Troubleshooting](troubleshooting.md) +* [Troubleshooting](administration/troubleshooting.md) ## Reference diff --git a/docs/installing-feast/troubleshooting.md b/docs/administration/troubleshooting.md similarity index 100% rename from docs/installing-feast/troubleshooting.md rename to docs/administration/troubleshooting.md diff --git a/docs/installing-feast/docker-compose.md b/docs/installing-feast/docker-compose.md deleted file mode 100644 index fbc491b4376..00000000000 --- a/docs/installing-feast/docker-compose.md +++ /dev/null @@ -1,112 +0,0 @@ -# Docker Compose - -### Overview - -This guide will bring Feast up using Docker Compose. This will allow you to: - -* Create, register, and manage feature sets -* Ingest feature data into Feast -* Retrieve features for online serving -* Retrieve features for batch serving \(only if using Google Cloud Platform\) - -This guide is split into three parts: - -1. Setting up your environment -2. Starting Feast with **online serving support only** \(does not require GCP\). -3. Starting Feast with support for **both online and batch** serving \(requires GCP\) - -{% hint style="info" %} -The docker compose setup uses Direct Runner for the Apache Beam jobs that populate data stores. Running Beam with the Direct Runner means it does not need a dedicated runner like Flink or Dataflow, but this comes at the cost of performance. We recommend the use of a dedicated runner when running Feast with very large workloads. -{% endhint %} - -### 0. Requirements - -* [Docker compose](https://docs.docker.com/compose/install/) must be installed. -* The following list of TCP ports must be free: - * 6565, 6566, 8888, and 9094. - * Alternatively it is possible to modify port mappings in `/docker-compose/docker-compose.yml`. -* \(for batch serving only\) For batch serving you will also need a [GCP service account key](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) that has access to [Google Cloud Storage](https://cloud.google.com/storage) and [BigQuery](https://cloud.google.com/bigquery). -* \(for batch serving only\) [Google Cloud SDK ](https://cloud.google.com/sdk/install)installed, authenticated, and configured to the project you will use. - -## 1. Set up environment - -Clone the [Feast repository](https://github.com/gojek/feast/) and navigate to the `docker-compose` sub-directory: - -```bash -git clone https://github.com/gojek/feast.git && \ -cd feast && export FEAST_HOME_DIR=$(pwd) && \ -cd infra/docker-compose -``` - -Make a copy of the `.env.sample` file: - -```bash -cp .env.sample .env -``` - -## 2. Docker Compose for Online Serving Only - -### 2.1 Start Feast \(without batch retrieval support\) - -If you do not require batch serving, then its possible to simply bring up Feast: - -```javascript -docker-compose up -d -``` - -A Jupyter Notebook environment is now available to use Feast: - -[http://localhost:8888/tree/feast/examples](http://localhost:8888/tree/feast/examples) - -## 3. Docker Compose for Online and Batch Serving - -{% hint style="info" %} -Batch serving requires Google Cloud Storage to function, specifically Google Cloud Storage \(GCP\) and BigQuery. -{% endhint %} - -### 3.1 Set up Google Cloud Platform - -Create a [service account ](https://cloud.google.com/iam/docs/creating-managing-service-accounts)from the GCP console and copy it to the `infra/docker-compose/gcp-service-accounts` folder: - -```javascript -cp my-service-account.json ${FEAST_HOME_DIR}/infra/docker-compose/gcp-service-accounts -``` - -Create a Google Cloud Storage bucket. Make sure that your service account above has read/write permissions to this bucket: - -```bash -gsutil mb gs://my-feast-staging-bucket -``` - -### 3.2 Configure .env - -Configure the `.env` file based on your environment. At the very least you have to modify: - -| Parameter | Description | -| :--- | :--- | -| FEAST\_CORE\_GCP\_SERVICE\_ACCOUNT\_KEY | This should be your service account file name, for example `key.json`. | -| FEAST\_BATCH\_SERVING\_GCP\_SERVICE\_ACCOUNT\_KEY | This should be your service account file name, for example `key.json` | -| FEAST\_JUPYTER\_GCP\_SERVICE\_ACCOUNT\_KEY | This should be your service account file name, for example `key.json` | -| FEAST\_JOB\_STAGING\_LOCATION | Google Cloud Storage bucket that Feast will use to stage data exports and batch retrieval requests, for example `gs://your-gcs-bucket/staging` | - -### 3.3 Configure .bq-store.yml - -We will also need to configure the `bq-store.yml` file inside `infra/docker-compose/serving/` to configure the BigQuery storage configuration as well as the feature sets that the store subscribes to. At a minimum you will need to set: - -| Parameter | Description | -| :--- | :--- | -| bigquery\_config.project\_id | This is you [GCP project Id](https://cloud.google.com/resource-manager/docs/creating-managing-projects). | -| bigquery\_config.dataset\_id | This is the name of the BigQuery dataset that tables will be created in. Each feature set will have one table in BigQuery. | - -### 3.4 Start Feast \(with batch retrieval support\) - -Start Feast: - -```javascript -docker-compose up -d -``` - -A Jupyter Notebook environment is now available to use Feast: - -[http://localhost:8888/tree/feast/examples](http://localhost:8888/tree/feast/examples) - diff --git a/docs/installing-feast/gke.md b/docs/installing-feast/gke.md deleted file mode 100644 index 162f0a26064..00000000000 --- a/docs/installing-feast/gke.md +++ /dev/null @@ -1,211 +0,0 @@ -# Google Kubernetes Engine \(GKE\) - -### Overview - -This guide will install Feast into a Kubernetes cluster on GCP. It assumes that all of your services will run within a single Kubernetes cluster. Once Feast is installed you will be able to: - -* Define and register features. -* Load feature data from both batch and streaming sources. -* Retrieve features for model training. -* Retrieve features for online serving. - -{% hint style="info" %} -This guide requires [Google Cloud Platform](https://cloud.google.com/) for installation. - -* [BigQuery](https://cloud.google.com/bigquery/) is used for storing historical features. -* [Google Cloud Storage](https://cloud.google.com/storage/) is used for intermediate data storage. -{% endhint %} - -## 0. Requirements - -1. [Google Cloud SDK ](https://cloud.google.com/sdk/install)installed, authenticated, and configured to the project you will use. -2. [Kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) installed. -3. [Helm](https://helm.sh/3) \(2.16.0 or greater\) installed on your local machine with Tiller installed in your cluster. Helm 3 has not been tested yet. - -## 1. Set up GCP - -First define the environmental variables that we will use throughout this installation. Please customize these to reflect your environment. - -```bash -export FEAST_GCP_PROJECT_ID=my-gcp-project -export FEAST_GCP_REGION=us-central1 -export FEAST_GCP_ZONE=us-central1-a -export FEAST_BIGQUERY_DATASET_ID=feast -export FEAST_GCS_BUCKET=${FEAST_GCP_PROJECT_ID}_feast_bucket -export FEAST_GKE_CLUSTER_NAME=feast -export FEAST_SERVICE_ACCOUNT_NAME=feast-sa -``` - -Create a Google Cloud Storage bucket for Feast to stage batch data exports: - -```bash -gsutil mb gs://${FEAST_GCS_BUCKET} -``` - -Create the service account that Feast will run as: - -```bash -gcloud iam service-accounts create ${FEAST_SERVICE_ACCOUNT_NAME} - -gcloud projects add-iam-policy-binding ${FEAST_GCP_PROJECT_ID} \ - --member serviceAccount:${FEAST_SERVICE_ACCOUNT_NAME}@${FEAST_GCP_PROJECT_ID}.iam.gserviceaccount.com \ - --role roles/editor - -gcloud iam service-accounts keys create key.json --iam-account \ -${FEAST_SERVICE_ACCOUNT_NAME}@${FEAST_GCP_PROJECT_ID}.iam.gserviceaccount.com -``` - -## 2. Set up a Kubernetes \(GKE\) cluster - -{% hint style="warning" %} -Provisioning a GKE cluster can expose your services publicly. This guide does not cover securing access to the cluster. -{% endhint %} - -Create a GKE cluster: - -```bash -gcloud container clusters create ${FEAST_GKE_CLUSTER_NAME} \ - --machine-type n1-standard-4 -``` - -Create a secret in the GKE cluster based on your local key `key.json`: - -```bash -kubectl create secret generic feast-gcp-service-account --from-file=key.json -``` - -For this guide we will use `NodePort` for exposing Feast services. In order to do so, we must find an External IP of at least one GKE node. This should be a public IP. - -```bash -export FEAST_IP=$(kubectl describe nodes | grep ExternalIP | awk '{print $2}' | head -n 1) -export FEAST_CORE_URL=${FEAST_IP}:32090 -export FEAST_ONLINE_SERVING_URL=${FEAST_IP}:32091 -export FEAST_BATCH_SERVING_URL=${FEAST_IP}:32092 -``` - -Add firewall rules to open up ports on your Google Cloud Platform project: - -```bash -gcloud compute firewall-rules create feast-core-port --allow tcp:32090 -gcloud compute firewall-rules create feast-online-port --allow tcp:32091 -gcloud compute firewall-rules create feast-batch-port --allow tcp:32092 -gcloud compute firewall-rules create feast-redis-port --allow tcp:32101 -gcloud compute firewall-rules create feast-kafka-ports --allow tcp:31090-31095 -``` - -## 3. Set up Helm - -Run the following command to provide Tiller with authorization to install Feast: - -```bash -kubectl apply -f - < Date: Fri, 27 Mar 2020 23:41:43 +0800 Subject: [PATCH 094/176] Changelog v0.4.7 (#579) Co-authored-by: Khor Shu Heng --- CHANGELOG.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7758ae3fb97..4276858ca0a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,18 @@ # Changelog +## [v0.4.7](https://github.com/gojek/feast/tree/v0.4.7) (2020-03-17) + +[Full Changelog](https://github.com/gojek/feast/compare/v0.4.6...v0.4.7) + +**Merged pull requests:** +- Add log4j-web jar to core and serving. [\#498](https://github.com/gojek/feast/pull/498) ([Yanson](https://github.com/Yanson)) +- Clear all the futures when sync is called. [\#501](https://github.com/gojek/feast/pull/501) ([lavkesh](https://github.com/lavkesh)) +- Encode feature row before storing in Redis [\#530](https://github.com/gojek/feast/pull/530) ([khorshuheng](https://github.com/khorshuheng)) +- Remove transaction when listing projects [\#522](https://github.com/gojek/feast/pull/522) ([davidheryanto](https://github.com/davidheryanto)) +- Remove unused ingestion deps [\#520](https://github.com/gojek/feast/pull/520) ([ches](https://github.com/ches)) +- Parameterize end to end test scripts. [\#433](https://github.com/gojek/feast/pull/433) ([Yanson](https://github.com/Yanson)) +- Replacing Jedis With Lettuce in ingestion and serving [\#485](https://github.com/gojek/feast/pull/485) ([lavkesh](https://github.com/lavkesh)) + ## [v0.4.6](https://github.com/gojek/feast/tree/v0.4.6) (2020-02-26) [Full Changelog](https://github.com/gojek/feast/compare/v0.4.5...v0.4.6) From 8b4509aeda0ebae0a584ecda4e3e27dfd30eee66 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Sat, 28 Mar 2020 09:29:50 +0800 Subject: [PATCH 095/176] Add badges and rename linting to code standards (#581) * Add badges and rename linting to code standards * Fix code standards workflow link --- .github/workflows/{lint.yaml => code_standards.yaml} | 2 +- README.md | 5 +++++ 2 files changed, 6 insertions(+), 1 deletion(-) rename .github/workflows/{lint.yaml => code_standards.yaml} (97%) diff --git a/.github/workflows/lint.yaml b/.github/workflows/code_standards.yaml similarity index 97% rename from .github/workflows/lint.yaml rename to .github/workflows/code_standards.yaml index 879ffb68ee1..684f1efaec4 100644 --- a/.github/workflows/lint.yaml +++ b/.github/workflows/code_standards.yaml @@ -1,4 +1,4 @@ -name: linting +name: code standards on: push: diff --git a/README.md b/README.md index 3c9ec929b79..881ccea49b1 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,10 @@ # Feast - Feature Store for Machine Learning +[![Unit Tests](https://github.com/gojek/feast/workflows/unit%20tests/badge.svg?branch=master)](https://github.com/gojek/feast/actions?query=workflow%3A%22unit+tests%22+branch%3Amaster) +[![Code Standards](https://github.com/gojek/feast/workflows/code%20standards/badge.svg?branch=master)](https://github.com/gojek/feast/actions?query=workflow%3A%22code+standards%22+branch%3Amaster) +[![Docs latest](https://img.shields.io/badge/Docs-latest-blue.svg)](https://docs.feast.dev/) +[![GitHub Release](https://img.shields.io/github/release/gojek/feast.svg?style=flat)](https://github.com/gojek/feast/releases) + ## Overview Feast (Feature Store) is a tool for managing and serving machine learning features. Feast is the bridge between models and data. From 2ee238a01882c8300f339753afaf7041d1d12088 Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Sat, 28 Mar 2020 11:07:43 +0800 Subject: [PATCH 096/176] Fix for extra keys throwing an error (#570) --- .../redis/FeatureRowToRedisMutationDoFn.java | 4 +- .../FeatureRowToRedisMutationDoFnTest.java | 70 +++++++++++++++++++ 2 files changed, 73 insertions(+), 1 deletion(-) diff --git a/ingestion/src/main/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFn.java b/ingestion/src/main/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFn.java index ca017c1f756..1f5a0f19677 100644 --- a/ingestion/src/main/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFn.java +++ b/ingestion/src/main/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFn.java @@ -62,7 +62,9 @@ private RedisKey getKey(FeatureRow featureRow) { } } for (String entityName : entityNames) { - redisKeyBuilder.addEntities(entityFields.get(entityName)); + if (entityFields.containsKey(entityName)) { + redisKeyBuilder.addEntities(entityFields.get(entityName)); + } } return redisKeyBuilder.build(); } diff --git a/ingestion/src/test/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFnTest.java b/ingestion/src/test/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFnTest.java index 86b4feae05f..7db5e28ecb8 100644 --- a/ingestion/src/test/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFnTest.java +++ b/ingestion/src/test/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFnTest.java @@ -142,6 +142,76 @@ public void shouldConvertRowWithDuplicateEntitiesToValidKey() { p.run(); } + @Test + public void shouldConvertRowWithExtraEntitiesToValidKey() { + Map featureSets = new HashMap<>(); + featureSets.put("feature_set", fs); + + FeatureRow offendingRow = + FeatureRow.newBuilder() + .setFeatureSet("feature_set") + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addFields( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addFields( + Field.newBuilder() + .setName("entity_id_invalid") + .setValue(Value.newBuilder().setInt32Val(2))) + .addFields( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .addFields( + Field.newBuilder() + .setName("feature_1") + .setValue(Value.newBuilder().setStringVal("strValue1"))) + .addFields( + Field.newBuilder() + .setName("feature_2") + .setValue(Value.newBuilder().setInt64Val(1001))) + .build(); + + PCollection output = + p.apply(Create.of(Collections.singletonList(offendingRow))) + .setCoder(ProtoCoder.of(FeatureRow.class)) + .apply(ParDo.of(new FeatureRowToRedisMutationDoFn(featureSets))); + + RedisKey expectedKey = + RedisKey.newBuilder() + .setFeatureSet("feature_set") + .addEntities( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addEntities( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .build(); + + FeatureRow expectedValue = + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setStringVal("strValue1"))) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setInt64Val(1001))) + .build(); + + PAssert.that(output) + .satisfies( + (SerializableFunction, Void>) + input -> { + input.forEach( + rm -> { + assert (Arrays.equals(rm.getKey(), expectedKey.toByteArray())); + assert (Arrays.equals(rm.getValue(), expectedValue.toByteArray())); + }); + return null; + }); + p.run(); + } + @Test public void shouldConvertRowWithOutOfOrderFieldsToValidKey() { Map featureSets = new HashMap<>(); From 33cfbf93fb429d50bcc9bb895779fb8d8bb01490 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Sat, 28 Mar 2020 12:14:30 +0800 Subject: [PATCH 097/176] Add Telco Churn Tutorial with XGBoost and Feast (#561) * Add Telco Churn Tutorial * Remove telco_customer_churn.csv --- ... Prediction (with Feast and XGBoost).ipynb | 6840 +++++++++++++++++ 1 file changed, 6840 insertions(+) create mode 100644 examples/feast-xgboost-churn-prediction-tutorial/Telecom Customer Churn Prediction (with Feast and XGBoost).ipynb diff --git a/examples/feast-xgboost-churn-prediction-tutorial/Telecom Customer Churn Prediction (with Feast and XGBoost).ipynb b/examples/feast-xgboost-churn-prediction-tutorial/Telecom Customer Churn Prediction (with Feast and XGBoost).ipynb new file mode 100644 index 00000000000..e88fe970d54 --- /dev/null +++ b/examples/feast-xgboost-churn-prediction-tutorial/Telecom Customer Churn Prediction (with Feast and XGBoost).ipynb @@ -0,0 +1,6840 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Telecom Customer Churn Prediction (with Feast and XGBoost)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This tutorial will demonstrate the use of Feast in productionising a churn model. The tutorial is broken down into two sections\n", + "\n", + "1. Churn Modelling (without Feast): In this section we explore the data, refine it, train a model, and evaluate its performance.\n", + "2. Churn Modelling (with Feast): In this section we introduce Feast for feature storage, management, as well as serving.\n", + "\n", + "\n", + "This tutorial is an extension of [this](https://www.kaggle.com/pavanraj159/telecom-customer-churn-prediction/comments#6.-Model-Performances) Kaggle notebook" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "_uuid": "fa7e507381a982dfb9bcba253537c50ecd956230" + }, + "source": [ + "## 1. Churn Modelling (without Feast)" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19", + "_kg_hide-input": false, + "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5" + }, + "outputs": [], + "source": [ + "import numpy as np\n", + "import pandas as pd\n", + "import os\n", + "import matplotlib.pyplot as plt\n", + "from PIL import Image\n", + "%matplotlib inline\n", + "import pandas as pd\n", + "import seaborn as sns\n", + "import itertools\n", + "import warnings\n", + "warnings.filterwarnings(\"ignore\")\n", + "import io\n", + "import plotly.offline as py\n", + "py.init_notebook_mode(connected=True)\n", + "import plotly.graph_objs as go\n", + "import plotly.tools as tls\n", + "import plotly.figure_factory as ff\n", + "import statsmodels, yellowbrick\n", + "import sklearn # Tested with 0.22.1\n", + "import imblearn\n", + "from slugify import slugify" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1.1 Data" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0", + "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a" + }, + "outputs": [ + { + "data": { + "text/html": [ + "

\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
customer_idgenderSeniorCitizenPartnerDependentstenurePhoneServiceMultipleLinesInternetServiceOnlineSecurity...DeviceProtectionTechSupportStreamingTVStreamingMoviesContractPaperlessBillingPaymentMethodMonthlyChargesTotalChargesChurn
07590-VHVEGFemale0YesNo1NoNo phone serviceDSLNo...NoNoNoNoMonth-to-monthYesElectronic check29.8529.85No
15575-GNVDEMale0NoNo34YesNoDSLYes...YesNoNoNoOne yearNoMailed check56.951889.5No
23668-QPYBKMale0NoNo2YesNoDSLYes...NoNoNoNoMonth-to-monthYesMailed check53.85108.15Yes
37795-CFOCWMale0NoNo45NoNo phone serviceDSLYes...YesYesNoNoOne yearNoBank transfer (automatic)42.301840.75No
49237-HQITUFemale0NoNo2YesNoFiber opticNo...NoNoNoNoMonth-to-monthYesElectronic check70.70151.65Yes
\n", + "

5 rows × 21 columns

\n", + "
" + ], + "text/plain": [ + " customer_id gender SeniorCitizen Partner Dependents tenure PhoneService \\\n", + "0 7590-VHVEG Female 0 Yes No 1 No \n", + "1 5575-GNVDE Male 0 No No 34 Yes \n", + "2 3668-QPYBK Male 0 No No 2 Yes \n", + "3 7795-CFOCW Male 0 No No 45 No \n", + "4 9237-HQITU Female 0 No No 2 Yes \n", + "\n", + " MultipleLines InternetService OnlineSecurity ... DeviceProtection \\\n", + "0 No phone service DSL No ... No \n", + "1 No DSL Yes ... Yes \n", + "2 No DSL Yes ... No \n", + "3 No phone service DSL Yes ... Yes \n", + "4 No Fiber optic No ... No \n", + "\n", + " TechSupport StreamingTV StreamingMovies Contract PaperlessBilling \\\n", + "0 No No No Month-to-month Yes \n", + "1 No No No One year No \n", + "2 No No No Month-to-month Yes \n", + "3 Yes No No One year No \n", + "4 No No No Month-to-month Yes \n", + "\n", + " PaymentMethod MonthlyCharges TotalCharges Churn \n", + "0 Electronic check 29.85 29.85 No \n", + "1 Mailed check 56.95 1889.5 No \n", + "2 Mailed check 53.85 108.15 Yes \n", + "3 Bank transfer (automatic) 42.30 1840.75 No \n", + "4 Electronic check 70.70 151.65 Yes \n", + "\n", + "[5 rows x 21 columns]" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "telcom = pd.read_csv('http://feast-examples.storage.googleapis.com/telco-churn-xgboost/telco_customer_churn.csv')\n", + "telcom.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "_uuid": "c8c010f36e29c116c6662301b08b0b0019d6e22e" + }, + "source": [ + "### 1.2 Data Manipulation" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "_uuid": "8b10c13086dff7182e399b849e31bc03df54a14e" + }, + "outputs": [], + "source": [ + "# Replacing spaces with null values in total charges column\n", + "telcom['TotalCharges'] = telcom[\"TotalCharges\"].replace(\" \",np.nan)\n", + "\n", + "# Dropping null values from total charges column which contain .15% missing data \n", + "telcom = telcom[telcom[\"TotalCharges\"].notnull()]\n", + "telcom = telcom.reset_index()[telcom.columns]\n", + "\n", + "# Convert to float type\n", + "telcom[\"TotalCharges\"] = telcom[\"TotalCharges\"].astype(float)\n", + "\n", + "# Replace 'No internet service' to No for the following columns\n", + "replace_cols = [ 'OnlineSecurity', 'OnlineBackup', 'DeviceProtection',\n", + " 'TechSupport','StreamingTV', 'StreamingMovies']\n", + "for i in replace_cols : \n", + " telcom[i] = telcom[i].replace({'No internet service' : 'No'})\n", + " \n", + "# Replace binary values with strings\n", + "telcom[\"SeniorCitizen\"] = telcom[\"SeniorCitizen\"].replace({1:\"Yes\",0:\"No\"})\n", + "\n", + "# Tenure to categorical column\n", + "def tenure_lab(telcom) :\n", + " \n", + " if telcom[\"tenure\"] <= 12 :\n", + " return \"Tenure_0-12\"\n", + " elif (telcom[\"tenure\"] > 12) & (telcom[\"tenure\"] <= 24 ):\n", + " return \"Tenure_12-24\"\n", + " elif (telcom[\"tenure\"] > 24) & (telcom[\"tenure\"] <= 48) :\n", + " return \"Tenure_24-48\"\n", + " elif (telcom[\"tenure\"] > 48) & (telcom[\"tenure\"] <= 60) :\n", + " return \"Tenure_48-60\"\n", + " elif telcom[\"tenure\"] > 60 :\n", + " return \"Tenure_gt_60\"\n", + "telcom[\"tenure_group\"] = telcom.apply(lambda telcom:tenure_lab(telcom),\n", + " axis = 1)\n", + "\n", + "# Separating churn and non churn customers\n", + "churn = telcom[telcom[\"Churn\"] == \"Yes\"]\n", + "not_churn = telcom[telcom[\"Churn\"] == \"No\"]\n", + "\n", + "# Separating catagorical and numerical columns\n", + "Id_col = ['customerID']\n", + "target_col = [\"Churn\"]\n", + "cat_cols = telcom.nunique()[telcom.nunique() < 6].keys().tolist()\n", + "cat_cols = [x for x in cat_cols if x not in target_col]\n", + "num_cols = [x for x in telcom.columns if x not in cat_cols + target_col + Id_col]\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "_uuid": "6dfa77b43fe1a1a301bab65186c2a9f90245ab7d" + }, + "source": [ + "### 1.3 Data Processing" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "_uuid": "8921591320c5e336ec5a2e1efc5ed3cb0f9ec1b2" + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
customer_idgenderseniorcitizenpartnerdependentsphoneserviceonlinesecurityonlinebackupdeviceprotectiontechsupport...paymentmethod_electronic_checkpaymentmethod_mailed_checktenure_group_tenure_0_12tenure_group_tenure_12_24tenure_group_tenure_24_48tenure_group_tenure_48_60tenure_group_tenure_gt_60tenuremonthlychargestotalcharges
07590-VHVEG001000100...1010000-1.280248-1.161694-0.994194
15575-GNVDE100011010...01001000.064303-0.260878-0.173740
23668-QPYBK100011100...0110000-1.239504-0.363923-0.959649
37795-CFOCW100001011...00001000.512486-0.747850-0.195248
49237-HQITU000010000...1010000-1.2395040.196178-0.940457
\n", + "

5 rows × 35 columns

\n", + "
" + ], + "text/plain": [ + " customer_id gender seniorcitizen partner dependents phoneservice \\\n", + "0 7590-VHVEG 0 0 1 0 0 \n", + "1 5575-GNVDE 1 0 0 0 1 \n", + "2 3668-QPYBK 1 0 0 0 1 \n", + "3 7795-CFOCW 1 0 0 0 0 \n", + "4 9237-HQITU 0 0 0 0 1 \n", + "\n", + " onlinesecurity onlinebackup deviceprotection techsupport ... \\\n", + "0 0 1 0 0 ... \n", + "1 1 0 1 0 ... \n", + "2 1 1 0 0 ... \n", + "3 1 0 1 1 ... \n", + "4 0 0 0 0 ... \n", + "\n", + " paymentmethod_electronic_check paymentmethod_mailed_check \\\n", + "0 1 0 \n", + "1 0 1 \n", + "2 0 1 \n", + "3 0 0 \n", + "4 1 0 \n", + "\n", + " tenure_group_tenure_0_12 tenure_group_tenure_12_24 \\\n", + "0 1 0 \n", + "1 0 0 \n", + "2 1 0 \n", + "3 0 0 \n", + "4 1 0 \n", + "\n", + " tenure_group_tenure_24_48 tenure_group_tenure_48_60 \\\n", + "0 0 0 \n", + "1 1 0 \n", + "2 0 0 \n", + "3 1 0 \n", + "4 0 0 \n", + "\n", + " tenure_group_tenure_gt_60 tenure monthlycharges totalcharges \n", + "0 0 -1.280248 -1.161694 -0.994194 \n", + "1 0 0.064303 -0.260878 -0.173740 \n", + "2 0 -1.239504 -0.363923 -0.959649 \n", + "3 0 0.512486 -0.747850 -0.195248 \n", + "4 0 -1.239504 0.196178 -0.940457 \n", + "\n", + "[5 rows x 35 columns]" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "from sklearn.preprocessing import LabelEncoder\n", + "from sklearn.preprocessing import StandardScaler\n", + "\n", + "# Customer id col\n", + "Id_col = ['customer_id']\n", + "\n", + "# Target columns\n", + "target_col = [\"churn\"]\n", + "\n", + "# Categorical columns\n", + "cat_cols = telcom.nunique()[telcom.nunique() < 6].keys().tolist()\n", + "cat_cols = [x for x in cat_cols if x not in target_col]\n", + "\n", + "# Numerical columns\n", + "num_cols = [x for x in telcom.columns if x not in cat_cols + target_col + Id_col]\n", + "\n", + "# Binary columns with 2 values\n", + "bin_cols = telcom.nunique()[telcom.nunique() == 2].keys().tolist()\n", + "\n", + "# Columns more than 2 values\n", + "multi_cols = [i for i in cat_cols if i not in bin_cols]\n", + "\n", + "# Label encoding Binary columns\n", + "le = LabelEncoder()\n", + "for i in bin_cols :\n", + " telcom[i] = le.fit_transform(telcom[i])\n", + " \n", + "# Duplicating columns for multi value columns\n", + "telcom = pd.get_dummies(data = telcom,columns = multi_cols )\n", + "\n", + "# Scaling Numerical columns\n", + "std = StandardScaler()\n", + "scaled = std.fit_transform(telcom[num_cols])\n", + "scaled = pd.DataFrame(scaled,columns=num_cols)\n", + "\n", + "# Dropping original values merging scaled values for numerical columns\n", + "df_telcom_og = telcom.copy()\n", + "telcom = telcom.drop(columns = num_cols,axis = 1)\n", + "telcom = telcom.merge(scaled,left_index=True,right_index=True,how = \"left\")\n", + "\n", + "# Clean up column names\n", + "telcom.columns = [slugify(col, lowercase=True, separator='_') for col in telcom.columns]\n", + "telcom.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "_uuid": "9ec25cff71c0eb0f0c839a726cb06cb43462a53f" + }, + "source": [ + "### 1.4 Descriptive Statistics" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
countuniquetopfreqmeanstdmin25%50%75%max
customer_id703270320835-DUUIQ1NaNNaNNaNNaNNaNNaNNaN
gender7032NaNNaNNaN0.5046930.50001400111
seniorcitizen7032NaNNaNNaN0.16240.36884400001
partner7032NaNNaNNaN0.4825090.49972900011
dependents7032NaNNaNNaN0.2984930.45762900011
phoneservice7032NaNNaNNaN0.9032990.29557101111
onlinesecurity7032NaNNaNNaN0.2865470.4521800011
onlinebackup7032NaNNaNNaN0.3448520.47535400011
deviceprotection7032NaNNaNNaN0.3438570.47502800011
techsupport7032NaNNaNNaN0.2901020.45384200011
streamingtv7032NaNNaNNaN0.3843860.48648400011
streamingmovies7032NaNNaNNaN0.3883670.48741400011
paperlessbilling7032NaNNaNNaN0.5927190.49136300111
churn7032NaNNaNNaN0.2657850.44178200011
multiplelines_no7032NaNNaNNaN0.4813710.49968800011
multiplelines_no_phone_service7032NaNNaNNaN0.09670080.29557100001
multiplelines_yes7032NaNNaNNaN0.4219280.49390200011
internetservice_dsl7032NaNNaNNaN0.3435720.47493400011
internetservice_fiber_optic7032NaNNaNNaN0.4402730.49645500011
internetservice_no7032NaNNaNNaN0.2161550.4116500001
contract_month_to_month7032NaNNaNNaN0.5510520.49742200111
contract_one_year7032NaNNaNNaN0.2093290.40685800001
contract_two_year7032NaNNaNNaN0.2396190.42688100001
paymentmethod_bank_transfer_automatic7032NaNNaNNaN0.2192830.4137900001
paymentmethod_credit_card_automatic7032NaNNaNNaN0.2162970.41174800001
paymentmethod_electronic_check7032NaNNaNNaN0.336320.47248300011
paymentmethod_mailed_check7032NaNNaNNaN0.22810.41963700001
tenure_group_tenure_0_127032NaNNaNNaN0.30930.46223800011
tenure_group_tenure_12_247032NaNNaNNaN0.145620.3527500001
tenure_group_tenure_24_487032NaNNaNNaN0.2266780.41871200001
tenure_group_tenure_48_607032NaNNaNNaN0.1183160.32300500001
tenure_group_tenure_gt_607032NaNNaNNaN0.2000850.40009200001
tenure7032NaNNaNNaN-1.12664e-161.00007-1.28025-0.954296-0.1394170.9199261.61257
monthlycharges7032NaNNaNNaN6.06265e-171.00007-1.54728-0.9709770.1845440.8331481.79338
totalcharges7032NaNNaNNaN-1.11906e-161.00007-0.999069-0.830249-0.3908150.6668272.82426
\n", + "
" + ], + "text/plain": [ + " count unique top freq \\\n", + "customer_id 7032 7032 0835-DUUIQ 1 \n", + "gender 7032 NaN NaN NaN \n", + "seniorcitizen 7032 NaN NaN NaN \n", + "partner 7032 NaN NaN NaN \n", + "dependents 7032 NaN NaN NaN \n", + "phoneservice 7032 NaN NaN NaN \n", + "onlinesecurity 7032 NaN NaN NaN \n", + "onlinebackup 7032 NaN NaN NaN \n", + "deviceprotection 7032 NaN NaN NaN \n", + "techsupport 7032 NaN NaN NaN \n", + "streamingtv 7032 NaN NaN NaN \n", + "streamingmovies 7032 NaN NaN NaN \n", + "paperlessbilling 7032 NaN NaN NaN \n", + "churn 7032 NaN NaN NaN \n", + "multiplelines_no 7032 NaN NaN NaN \n", + "multiplelines_no_phone_service 7032 NaN NaN NaN \n", + "multiplelines_yes 7032 NaN NaN NaN \n", + "internetservice_dsl 7032 NaN NaN NaN \n", + "internetservice_fiber_optic 7032 NaN NaN NaN \n", + "internetservice_no 7032 NaN NaN NaN \n", + "contract_month_to_month 7032 NaN NaN NaN \n", + "contract_one_year 7032 NaN NaN NaN \n", + "contract_two_year 7032 NaN NaN NaN \n", + "paymentmethod_bank_transfer_automatic 7032 NaN NaN NaN \n", + "paymentmethod_credit_card_automatic 7032 NaN NaN NaN \n", + "paymentmethod_electronic_check 7032 NaN NaN NaN \n", + "paymentmethod_mailed_check 7032 NaN NaN NaN \n", + "tenure_group_tenure_0_12 7032 NaN NaN NaN \n", + "tenure_group_tenure_12_24 7032 NaN NaN NaN \n", + "tenure_group_tenure_24_48 7032 NaN NaN NaN \n", + "tenure_group_tenure_48_60 7032 NaN NaN NaN \n", + "tenure_group_tenure_gt_60 7032 NaN NaN NaN \n", + "tenure 7032 NaN NaN NaN \n", + "monthlycharges 7032 NaN NaN NaN \n", + "totalcharges 7032 NaN NaN NaN \n", + "\n", + " mean std min \\\n", + "customer_id NaN NaN NaN \n", + "gender 0.504693 0.500014 0 \n", + "seniorcitizen 0.1624 0.368844 0 \n", + "partner 0.482509 0.499729 0 \n", + "dependents 0.298493 0.457629 0 \n", + "phoneservice 0.903299 0.295571 0 \n", + "onlinesecurity 0.286547 0.45218 0 \n", + "onlinebackup 0.344852 0.475354 0 \n", + "deviceprotection 0.343857 0.475028 0 \n", + "techsupport 0.290102 0.453842 0 \n", + "streamingtv 0.384386 0.486484 0 \n", + "streamingmovies 0.388367 0.487414 0 \n", + "paperlessbilling 0.592719 0.491363 0 \n", + "churn 0.265785 0.441782 0 \n", + "multiplelines_no 0.481371 0.499688 0 \n", + "multiplelines_no_phone_service 0.0967008 0.295571 0 \n", + "multiplelines_yes 0.421928 0.493902 0 \n", + "internetservice_dsl 0.343572 0.474934 0 \n", + "internetservice_fiber_optic 0.440273 0.496455 0 \n", + "internetservice_no 0.216155 0.41165 0 \n", + "contract_month_to_month 0.551052 0.497422 0 \n", + "contract_one_year 0.209329 0.406858 0 \n", + "contract_two_year 0.239619 0.426881 0 \n", + "paymentmethod_bank_transfer_automatic 0.219283 0.41379 0 \n", + "paymentmethod_credit_card_automatic 0.216297 0.411748 0 \n", + "paymentmethod_electronic_check 0.33632 0.472483 0 \n", + "paymentmethod_mailed_check 0.2281 0.419637 0 \n", + "tenure_group_tenure_0_12 0.3093 0.462238 0 \n", + "tenure_group_tenure_12_24 0.14562 0.35275 0 \n", + "tenure_group_tenure_24_48 0.226678 0.418712 0 \n", + "tenure_group_tenure_48_60 0.118316 0.323005 0 \n", + "tenure_group_tenure_gt_60 0.200085 0.400092 0 \n", + "tenure -1.12664e-16 1.00007 -1.28025 \n", + "monthlycharges 6.06265e-17 1.00007 -1.54728 \n", + "totalcharges -1.11906e-16 1.00007 -0.999069 \n", + "\n", + " 25% 50% 75% max \n", + "customer_id NaN NaN NaN NaN \n", + "gender 0 1 1 1 \n", + "seniorcitizen 0 0 0 1 \n", + "partner 0 0 1 1 \n", + "dependents 0 0 1 1 \n", + "phoneservice 1 1 1 1 \n", + "onlinesecurity 0 0 1 1 \n", + "onlinebackup 0 0 1 1 \n", + "deviceprotection 0 0 1 1 \n", + "techsupport 0 0 1 1 \n", + "streamingtv 0 0 1 1 \n", + "streamingmovies 0 0 1 1 \n", + "paperlessbilling 0 1 1 1 \n", + "churn 0 0 1 1 \n", + "multiplelines_no 0 0 1 1 \n", + "multiplelines_no_phone_service 0 0 0 1 \n", + "multiplelines_yes 0 0 1 1 \n", + "internetservice_dsl 0 0 1 1 \n", + "internetservice_fiber_optic 0 0 1 1 \n", + "internetservice_no 0 0 0 1 \n", + "contract_month_to_month 0 1 1 1 \n", + "contract_one_year 0 0 0 1 \n", + "contract_two_year 0 0 0 1 \n", + "paymentmethod_bank_transfer_automatic 0 0 0 1 \n", + "paymentmethod_credit_card_automatic 0 0 0 1 \n", + "paymentmethod_electronic_check 0 0 1 1 \n", + "paymentmethod_mailed_check 0 0 0 1 \n", + "tenure_group_tenure_0_12 0 0 1 1 \n", + "tenure_group_tenure_12_24 0 0 0 1 \n", + "tenure_group_tenure_24_48 0 0 0 1 \n", + "tenure_group_tenure_48_60 0 0 0 1 \n", + "tenure_group_tenure_gt_60 0 0 0 1 \n", + "tenure -0.954296 -0.139417 0.919926 1.61257 \n", + "monthlycharges -0.970977 0.184544 0.833148 1.79338 \n", + "totalcharges -0.830249 -0.390815 0.666827 2.82426 " + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "telcom.describe(include='all').T" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "_uuid": "82a7617e37906622dbe00a1783a13cbf382d2513" + }, + "source": [ + "### 1.5 Correlation Matrix" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "_uuid": "b52cf9c7f402ed706e82221e3f8601fdeea9ab27" + }, + "outputs": [ + { + "data": { + "application/vnd.plotly.v1+json": { + "config": { + "linkText": "Export to plot.ly", + "plotlyServerURL": "https://plot.ly", + "showLink": false + }, + "data": [ + { + "colorbar": { + "title": { + "side": "right", + "text": "Pearson Correlation coefficient" + } + }, + "colorscale": [ + [ + 0, + "#440154" + ], + [ + 0.1111111111111111, + "#482878" + ], + [ + 0.2222222222222222, + "#3e4989" + ], + [ + 0.3333333333333333, + "#31688e" + ], + [ + 0.4444444444444444, + "#26828e" + ], + [ + 0.5555555555555556, + "#1f9e89" + ], + [ + 0.6666666666666666, + "#35b779" + ], + [ + 0.7777777777777778, + "#6ece58" + ], + [ + 0.8888888888888888, + "#b5de2b" + ], + [ + 1, + "#fde725" + ] + ], + "type": "heatmap", + "x": [ + "gender", + "seniorcitizen", + "partner", + "dependents", + "phoneservice", + "onlinesecurity", + "onlinebackup", + "deviceprotection", + "techsupport", + "streamingtv", + "streamingmovies", + "paperlessbilling", + "churn", + "multiplelines_no", + "multiplelines_no_phone_service", + "multiplelines_yes", + "internetservice_dsl", + "internetservice_fiber_optic", + "internetservice_no", + "contract_month_to_month", + "contract_one_year", + "contract_two_year", + "paymentmethod_bank_transfer_automatic", + "paymentmethod_credit_card_automatic", + "paymentmethod_electronic_check", + "paymentmethod_mailed_check", + "tenure_group_tenure_0_12", + "tenure_group_tenure_12_24", + "tenure_group_tenure_24_48", + "tenure_group_tenure_48_60", + "tenure_group_tenure_gt_60", + "tenure", + "monthlycharges", + "totalcharges" + ], + "y": [ + "gender", + "seniorcitizen", + "partner", + "dependents", + "phoneservice", + "onlinesecurity", + "onlinebackup", + "deviceprotection", + "techsupport", + "streamingtv", + "streamingmovies", + "paperlessbilling", + "churn", + "multiplelines_no", + "multiplelines_no_phone_service", + "multiplelines_yes", + "internetservice_dsl", + "internetservice_fiber_optic", + "internetservice_no", + "contract_month_to_month", + "contract_one_year", + "contract_two_year", + "paymentmethod_bank_transfer_automatic", + "paymentmethod_credit_card_automatic", + "paymentmethod_electronic_check", + "paymentmethod_mailed_check", + "tenure_group_tenure_0_12", + "tenure_group_tenure_12_24", + "tenure_group_tenure_24_48", + "tenure_group_tenure_48_60", + "tenure_group_tenure_gt_60", + "tenure", + "monthlycharges", + "totalcharges" + ], + "z": [ + [ + 1, + -0.001819390613419179, + -0.0013790513218356025, + 0.010348917127614397, + -0.007514979909200033, + -0.01632782307070617, + -0.013092839264555001, + -0.0008067457759124324, + -0.008507162405232782, + -0.0071243969867245535, + -0.010105418366566195, + -0.011901894766838502, + -0.008544643224947218, + 0.0043346914502916285, + 0.0075149799091999425, + -0.008882737146286056, + 0.00758357610303961, + -0.011189259276385864, + 0.004744965758849955, + -0.0032507651194551004, + 0.0077548529142672145, + -0.003603167413572989, + -0.015973079031173835, + 0.001631872518613598, + 0.0008437084888753327, + 0.01319936726545174, + -0.0010503798619876774, + -0.0006494991207074467, + -0.010516394125351592, + -0.004318975744551275, + 0.01627881894278637, + 0.005285371870295646, + -0.013779327268354416, + 4.783950839776602e-05 + ], + [ + -0.001819390613419179, + 1, + 0.01695661453202187, + -0.21055006112684216, + 0.008391611911217043, + -0.0385763901686064, + 0.06666279065142021, + 0.059513871482029225, + -0.060576839406188035, + 0.10544501753678828, + 0.11984236746151568, + 0.15625775052783097, + 0.1505410534156757, + -0.13637672686229402, + -0.008391611911217034, + 0.14299625086621018, + -0.10827563872848943, + 0.25492331502717946, + -0.18251949495535458, + 0.13775207088551514, + -0.0464907545657889, + -0.11620511425710835, + -0.01623474200582214, + -0.024359419683712323, + 0.17132216591713703, + -0.15298719260173027, + -0.02771322371899571, + 0.0018604411671567017, + 0.02038346087056509, + 0.01418568213097567, + -0.0024069937431968856, + 0.01568347989913396, + 0.21987422950593646, + 0.10241060539532633 + ], + [ + -0.0013790513218356025, + 0.01695661453202187, + 1, + 0.45226888584550023, + 0.018397189302703662, + 0.14334606167364233, + 0.14184917072520303, + 0.1535564364182745, + 0.12020601780298455, + 0.12448262672518244, + 0.11810820943292764, + -0.013956696136191696, + -0.14998192562006138, + -0.13002839561671578, + -0.018397189302703666, + 0.1425612874681736, + -0.0010430787434336079, + 0.0012346095228208073, + -0.0002855204740384597, + -0.2802019157901561, + 0.08306706395255747, + 0.24733370647615796, + 0.11140561212215645, + 0.08232738919649572, + -0.08320661736633733, + -0.09694798339506473, + -0.30506147447521575, + -0.048481275955609554, + 0.028467762106169584, + 0.10534126166196492, + 0.28035324824529767, + 0.38191150910757077, + 0.09782497186892049, + 0.31907236323857324 + ], + [ + 0.010348917127614397, + -0.21055006112684216, + 0.45226888584550023, + 1, + -0.001077812708067608, + 0.08078553224088346, + 0.023638813060609963, + 0.013899668260943368, + 0.06305315799997827, + -0.01649868035801052, + -0.03837492560091315, + -0.11013068597336993, + -0.16312843938822, + 0.023387669506115295, + 0.0010778127080674446, + -0.024306661314620996, + 0.05159321756471237, + -0.16410089031864167, + 0.13838288994798562, + -0.2297147909213749, + 0.06922205672629726, + 0.2016993304297039, + 0.05236890928127545, + 0.06113408322897806, + -0.1492739811934336, + 0.05644841359590588, + -0.1453791450700764, + -0.0014594010172321779, + 0.02464494581436856, + 0.03141940765317757, + 0.1180897178470907, + 0.16338596691556453, + -0.11234295350128225, + 0.0646532494217739 + ], + [ + -0.007514979909200033, + 0.008391611911217043, + 0.018397189302703662, + -0.001077812708067608, + 1, + -0.09167570469500017, + -0.05213341919796151, + -0.07007561533228862, + -0.09513849428922488, + -0.021382711870578185, + -0.03347749718236395, + 0.016696123642784135, + 0.011691398865422323, + 0.31521775126801543, + -1, + 0.2795295400049995, + -0.45225528090657086, + 0.29018311793843365, + 0.1718171065699321, + -0.0012425134067734023, + -0.0031417807184111624, + 0.0044422513152849235, + 0.008271245210923577, + -0.006916252198127548, + 0.0027471183312986857, + -0.004462839400732194, + -0.00694955039725527, + 0.012306612692869211, + -0.014777815951004124, + -0.009750281447949313, + 0.020515848357693, + 0.007877333295041818, + 0.2480330664757158, + 0.11300826095473893 + ], + [ + -0.01632782307070617, + -0.0385763901686064, + 0.14334606167364233, + 0.08078553224088346, + -0.09167570469500017, + 1, + 0.28328454262626757, + 0.274875003842449, + 0.35445796164509147, + 0.1755144708953688, + 0.18742584957299618, + -0.004051250607988492, + -0.17126992353351678, + -0.15167751168612081, + 0.09167570469500017, + 0.0985919934252315, + 0.3203433737595294, + -0.03050626904076453, + -0.33279949932167546, + -0.24684428487400414, + 0.10065777311969464, + 0.19169819815673583, + 0.09436639279979232, + 0.11547320256635303, + -0.11229466175861408, + -0.07991768713640306, + -0.24240887436050382, + -0.0556619409198579, + 0.0076950987178344265, + 0.0716632710172793, + 0.26322845571231857, + 0.32829748818662485, + 0.2964469592375873, + 0.41261876950713655 + ], + [ + -0.013092839264555001, + 0.06666279065142021, + 0.14184917072520303, + 0.023638813060609963, + -0.05213341919796151, + 0.28328454262626757, + 1, + 0.30305766643807824, + 0.29370469187781045, + 0.2816010622259753, + 0.27452301070772545, + 0.12705603268686044, + -0.08230696876508349, + -0.2307241996493635, + 0.05213341919796151, + 0.2022283972825372, + 0.15676460995441888, + 0.16594028590307677, + -0.3809903317320751, + -0.16439302987919688, + 0.08411316021806066, + 0.11139068731904943, + 0.0869415675760231, + 0.09045518641091457, + -0.00036426636786500763, + -0.17407470231312427, + -0.26736609150996915, + -0.08408097138202993, + 0.023085000444599418, + 0.09919210454041022, + 0.2787875817043761, + 0.36113847824658735, + 0.4415290881871007, + 0.510100290145439 + ], + [ + -0.0008067457759124324, + 0.059513871482029225, + 0.1535564364182745, + 0.013899668260943368, + -0.07007561533228862, + 0.274875003842449, + 0.30305766643807824, + 1, + 0.33285005080469243, + 0.3899237975094597, + 0.4023088228216018, + 0.10407904724402045, + -0.06619251684228997, + -0.24084736328107104, + 0.07007561533228862, + 0.201732824517757, + 0.14514955473692903, + 0.17635617323471664, + -0.3801513548956378, + -0.22598757731112262, + 0.10291089629343353, + 0.16524753554250074, + 0.08304690185342707, + 0.11125168129784246, + -0.003308493511411254, + -0.18732483013668594, + -0.2739200544387702, + -0.07733200729381666, + 0.0449724925036831, + 0.07685425917962475, + 0.2755370112516221, + 0.3615199952862194, + 0.48260691224313995, + 0.5228814865154369 + ], + [ + -0.008507162405232782, + -0.060576839406188035, + 0.12020601780298455, + 0.06305315799997827, + -0.09513849428922488, + 0.35445796164509147, + 0.29370469187781045, + 0.33285005080469243, + 1, + 0.277548599200549, + 0.2801552432906342, + 0.03753587307318976, + -0.16471590834411207, + -0.15553386914722026, + 0.09513849428922488, + 0.10042125595413272, + 0.3121832985222757, + -0.020298967520709605, + -0.33569508671869736, + -0.28549086901033593, + 0.09625836225380952, + 0.24092408252256528, + 0.10047200087443556, + 0.11702370730984143, + -0.11480726996437085, + -0.08463055196615278, + -0.23862820953475447, + -0.07201850701116608, + 0.022136746625123563, + 0.06270980841796413, + 0.26539627802277554, + 0.325288454100009, + 0.33830139143424953, + 0.4328683682410939 + ], + [ + -0.0071243969867245535, + 0.10544501753678828, + 0.12448262672518244, + -0.01649868035801052, + -0.021382711870578185, + 0.1755144708953688, + 0.2816010622259753, + 0.3899237975094597, + 0.277548599200549, + 1, + 0.5333800979319763, + 0.22424119793848596, + 0.06325398027519404, + -0.26746641467807136, + 0.021382711870577976, + 0.25780350066730157, + 0.014973379079172382, + 0.3297441152730512, + -0.41495062156578044, + -0.11254989712217289, + 0.061929689963200855, + 0.07212357537070835, + 0.04612070051242859, + 0.040010276337768505, + 0.1447470086556032, + -0.2477115493728633, + -0.22076087097198827, + -0.050234026791169964, + 0.02882937792536747, + 0.08796930347195206, + 0.1981501311372027, + 0.280263628074823, + 0.6296678921767406, + 0.5157090769923935 + ], + [ + -0.010105418366566195, + 0.11984236746151568, + 0.11810820943292764, + -0.03837492560091315, + -0.03347749718236395, + 0.18742584957299618, + 0.27452301070772545, + 0.4023088228216018, + 0.2801552432906342, + 0.5333800979319763, + 1, + 0.21158250423808916, + 0.06085993668146301, + -0.2759953197780627, + 0.03347749718236395, + 0.2591943175468362, + 0.02562310861129719, + 0.3224574540559222, + -0.41844975538334045, + -0.11786687989290552, + 0.06477997824381859, + 0.07560257919382919, + 0.04875484714119087, + 0.048398314068082315, + 0.13742008269944622, + -0.2502897149395328, + -0.2213881765552116, + -0.054338325985925064, + 0.02504810987080452, + 0.08661557153323526, + 0.20754366554274195, + 0.2854022671060787, + 0.6272347301103788, + 0.5198665357835017 + ], + [ + -0.011901894766838502, + 0.15625775052783097, + -0.013956696136191696, + -0.11013068597336993, + 0.016696123642784135, + -0.004051250607988492, + 0.12705603268686044, + 0.10407904724402045, + 0.03753587307318976, + 0.22424119793848596, + 0.21158250423808916, + 1, + 0.19145432108006671, + -0.1519737778021082, + -0.016696123642784142, + 0.1637457730112591, + -0.06338966821876392, + 0.32647017160380964, + -0.3205922451174622, + 0.16829626845602835, + -0.052278164693773076, + -0.1462807050684952, + -0.017468900682392235, + -0.013726285095880284, + 0.20842668228002995, + -0.20398064814312206, + -0.003859801258113621, + 0.003328006906514837, + -0.005388027856510724, + 0.010626123397145876, + -0.001414839543993747, + 0.004823156615386272, + 0.3519304153712528, + 0.15782978286591698 + ], + [ + -0.008544643224947218, + 0.1505410534156757, + -0.14998192562006138, + -0.16312843938822, + 0.011691398865422323, + -0.17126992353351678, + -0.08230696876508349, + -0.06619251684228997, + -0.16471590834411207, + 0.06325398027519404, + 0.06085993668146301, + 0.19145432108006671, + 1, + -0.03265360299730401, + -0.01169139886542221, + 0.040032739872523634, + -0.12414142842590645, + 0.30746259069818205, + -0.22757762044656818, + 0.40456455007784087, + -0.17822502328994053, + -0.30155233962397837, + -0.1181359978280296, + -0.1346868372340906, + 0.30145463790858057, + -0.09077284582582087, + 0.3196275743451608, + 0.019928968647699895, + -0.07585881574899747, + -0.10079964105628228, + -0.2260781096915624, + -0.3540493589532626, + 0.1928582184700881, + -0.1994840835675715 + ], + [ + 0.0043346914502916285, + -0.13637672686229402, + -0.13002839561671578, + 0.023387669506115295, + 0.31521775126801543, + -0.15167751168612081, + -0.2307241996493635, + -0.24084736328107104, + -0.15553386914722026, + -0.26746641467807136, + -0.2759953197780627, + -0.1519737778021082, + -0.03265360299730401, + 1, + -0.3152177512680155, + -0.8230760279128654, + -0.06951498798055156, + -0.19053099387424138, + 0.3099843327828031, + 0.08679791861868207, + 0.0016944436455045328, + -0.1027560050227385, + -0.06966277653167001, + -0.06371157499118037, + -0.0809902106781048, + 0.22239547449983668, + 0.25617128287621094, + 0.05170274575687212, + -0.03215707580783382, + -0.08063023581160907, + -0.2427981631428492, + -0.323890766966285, + -0.3385136039067215, + -0.39676537714107735 + ], + [ + 0.0075149799091999425, + -0.008391611911217034, + -0.018397189302703666, + 0.0010778127080674446, + -1, + 0.09167570469500017, + 0.05213341919796151, + 0.07007561533228862, + 0.09513849428922488, + 0.021382711870577976, + 0.03347749718236395, + -0.016696123642784142, + -0.01169139886542221, + -0.3152177512680155, + 1, + -0.2795295400049995, + 0.45225528090657086, + -0.29018311793843365, + -0.1718171065699321, + 0.0012425134067734103, + 0.0031417807184113207, + -0.004442251315284906, + -0.008271245210923579, + 0.006916252198127564, + -0.0027471183312986857, + 0.004462839400732508, + 0.006949550397255344, + -0.012306612692869208, + 0.014777815951004133, + 0.009750281447949528, + -0.02051584835769357, + -0.007877333295041807, + -0.2480330664757158, + -0.1130082609547389 + ], + [ + -0.008882737146286056, + 0.14299625086621018, + 0.1425612874681736, + -0.024306661314620996, + 0.2795295400049995, + 0.0985919934252315, + 0.2022283972825372, + 0.201732824517757, + 0.10042125595413272, + 0.25780350066730157, + 0.2591943175468362, + 0.1637457730112591, + 0.040032739872523634, + -0.8230760279128654, + -0.2795295400049995, + 1, + -0.2003183214156725, + 0.3664202566051166, + -0.21079354712189471, + -0.08855832218643561, + -0.003594461398434266, + 0.10661820819152797, + 0.07542871730303179, + 0.06031899094084278, + 0.08358299305536028, + -0.22767156949803466, + -0.26333120743722876, + -0.04494367118136859, + 0.023690157563696343, + 0.07573985650934924, + 0.25792003827196397, + 0.33239924473562554, + 0.4909121973267493, + 0.4690421356971826 + ], + [ + 0.00758357610303961, + -0.10827563872848943, + -0.0010430787434336079, + 0.05159321756471237, + -0.45225528090657086, + 0.3203433737595294, + 0.15676460995441888, + 0.14514955473692903, + 0.3121832985222757, + 0.014973379079172382, + 0.02562310861129719, + -0.06338966821876392, + -0.12414142842590645, + -0.06951498798055156, + 0.45225528090657086, + -0.2003183214156725, + 1, + -0.6416356650534906, + -0.3799117751052334, + -0.06522632996531619, + 0.04729967349374622, + 0.030923714855574744, + 0.02475954021490298, + 0.05122176476699011, + -0.10429333541089097, + 0.04275388869901973, + -0.0014704079413737543, + -0.013428683581908657, + -0.0004678565419111244, + -0.00079001062411148, + 0.014665913237832425, + 0.013786269825793156, + -0.16136793538251534, + -0.052189866637419764 + ], + [ + -0.011189259276385864, + 0.25492331502717946, + 0.0012346095228208073, + -0.16410089031864167, + 0.29018311793843365, + -0.03050626904076453, + 0.16594028590307677, + 0.17635617323471664, + -0.020298967520709605, + 0.3297441152730512, + 0.3224574540559222, + 0.32647017160380964, + 0.30746259069818205, + -0.19053099387424138, + -0.29018311793843365, + 0.3664202566051166, + -0.6416356650534906, + 1, + -0.4657363343235562, + 0.24301351814831268, + -0.07680902975096314, + -0.20996452908904545, + -0.022778855291876635, + -0.050551991558036385, + 0.3357634768102139, + -0.3059839224841771, + -0.021440560786684968, + -0.0014940221824271288, + 0.005613761063888361, + 0.01746632456448475, + 0.0061120739544805805, + 0.017929529906438778, + 0.7871948529419658, + 0.3607687920142481 + ], + [ + 0.004744965758849955, + -0.18251949495535458, + -0.0002855204740384597, + 0.13838288994798562, + 0.1718171065699321, + -0.33279949932167546, + -0.3809903317320751, + -0.3801513548956378, + -0.33569508671869736, + -0.41495062156578044, + -0.41844975538334045, + -0.3205922451174622, + -0.22757762044656818, + 0.3099843327828031, + -0.1718171065699321, + -0.21079354712189471, + -0.3799117751052334, + -0.4657363343235562, + 1, + -0.217823505545489, + 0.038061459724733744, + 0.21754205606911817, + -0.0010943032992050863, + 0.0018701145623306124, + -0.2846082097225956, + 0.3196937439459745, + 0.027554030004254233, + 0.01729491162511594, + -0.006230482005525746, + -0.020153136101165186, + -0.024291768931540957, + -0.037528889581619734, + -0.7631910615169571, + -0.37487836259896 + ], + [ + -0.0032507651194551004, + 0.13775207088551514, + -0.2802019157901561, + -0.2297147909213749, + -0.0012425134067734023, + -0.24684428487400414, + -0.16439302987919688, + -0.22598757731112262, + -0.28549086901033593, + -0.11254989712217289, + -0.11786687989290552, + 0.16829626845602835, + 0.40456455007784087, + 0.08679791861868207, + 0.0012425134067734103, + -0.08855832218643561, + -0.06522632996531619, + 0.24301351814831268, + -0.217823505545489, + 1, + -0.5700527848944215, + -0.6219327447713561, + -0.18015909738683936, + -0.20496021669474843, + 0.33087881370583405, + 0.006208692442055045, + 0.49205202568054424, + 0.14000368638519403, + -0.05215631834144274, + -0.19870886170688193, + -0.47691223651608966, + -0.649345648869048, + 0.0589334582292538, + -0.4467758743246882 + ], + [ + 0.0077548529142672145, + -0.0464907545657889, + 0.08306706395255747, + 0.06922205672629726, + -0.0031417807184111624, + 0.10065777311969464, + 0.08411316021806066, + 0.10291089629343353, + 0.09625836225380952, + 0.061929689963200855, + 0.06477997824381859, + -0.052278164693773076, + -0.17822502328994053, + 0.0016944436455045328, + 0.0031417807184113207, + -0.003594461398434266, + 0.04729967349374622, + -0.07680902975096314, + 0.038061459724733744, + -0.5700527848944215, + 1, + -0.28884268256780254, + 0.05762874929825631, + 0.06758968145792751, + -0.10954646682258033, + 0.0001971241890499786, + -0.2512993032332009, + -0.017196458507178478, + 0.15389327309228798, + 0.15891681528381144, + 0.01614171428694319, + 0.2023384083649546, + 0.004809615120440435, + 0.17056928655718714 + ], + [ + -0.003603167413572989, + -0.11620511425710835, + 0.24733370647615796, + 0.2016993304297039, + 0.0044422513152849235, + 0.19169819815673583, + 0.11139068731904943, + 0.16524753554250074, + 0.24092408252256528, + 0.07212357537070835, + 0.07560257919382919, + -0.1462807050684952, + -0.30155233962397837, + -0.1027560050227385, + -0.004442251315284906, + 0.10661820819152797, + 0.030923714855574744, + -0.20996452908904545, + 0.21754205606911817, + -0.6219327447713561, + -0.28884268256780254, + 1, + 0.1550042180767056, + 0.17440993895950943, + -0.28114743393395547, + -0.007422540118618717, + -0.33385014178843664, + -0.14674905438081498, + -0.08589992270093438, + 0.08008211690632902, + 0.5403360993105428, + 0.5638005002286687, + -0.07325607300641665, + 0.35803561609140894 + ], + [ + -0.015973079031173835, + -0.01623474200582214, + 0.11140561212215645, + 0.05236890928127545, + 0.008271245210923577, + 0.09436639279979232, + 0.0869415675760231, + 0.08304690185342707, + 0.10047200087443556, + 0.04612070051242859, + 0.04875484714119087, + -0.017468900682392235, + -0.1181359978280296, + -0.06966277653167001, + -0.008271245210923579, + 0.07542871730303179, + 0.02475954021490298, + -0.022778855291876635, + -0.0010943032992050863, + -0.18015909738683936, + 0.05762874929825631, + 0.1550042180767056, + 1, + -0.27842319712065355, + -0.3772703602158491, + -0.28809669563791657, + -0.18585512574019983, + -0.04632871286727803, + 0.02172285552026243, + 0.060183035947993645, + 0.1842491799195877, + 0.24382246495742915, + 0.04240972759459293, + 0.18611944938877964 + ], + [ + 0.001631872518613598, + -0.024359419683712323, + 0.08232738919649572, + 0.06113408322897806, + -0.006916252198127548, + 0.11547320256635303, + 0.09045518641091457, + 0.11125168129784246, + 0.11702370730984143, + 0.040010276337768505, + 0.048398314068082315, + -0.013726285095880284, + -0.1346868372340906, + -0.06371157499118037, + 0.006916252198127564, + 0.06031899094084278, + 0.05122176476699011, + -0.050551991558036385, + 0.0018701145623306124, + -0.20496021669474843, + 0.06758968145792751, + 0.17440993895950943, + -0.27842319712065355, + 1, + -0.37397801626928234, + -0.28558254792863036, + -0.18416488323856553, + -0.03964706284937272, + 0.028232511385228868, + 0.048166948994686994, + 0.17929367300430438, + 0.23280034413314615, + 0.030054608328366407, + 0.1826633671546002 + ], + [ + 0.0008437084888753327, + 0.17132216591713703, + -0.08320661736633733, + -0.1492739811934336, + 0.0027471183312986857, + -0.11229466175861408, + -0.00036426636786500763, + -0.003308493511411254, + -0.11480726996437085, + 0.1447470086556032, + 0.13742008269944622, + 0.20842668228002995, + 0.30145463790858057, + -0.0809902106781048, + -0.0027471183312986857, + 0.08358299305536028, + -0.10429333541089097, + 0.3357634768102139, + -0.2846082097225956, + 0.33087881370583405, + -0.10954646682258033, + -0.28114743393395547, + -0.3772703602158491, + -0.37397801626928234, + 1, + -0.3869714587097063, + 0.1605298863682866, + 0.030386796069204806, + -0.011569992641797876, + -0.030584308144429393, + -0.1754558932547761, + -0.21019749610817867, + 0.27111737739356384, + -0.060436278689198375 + ], + [ + 0.01319936726545174, + -0.15298719260173027, + -0.09694798339506473, + 0.05644841359590588, + -0.004462839400732194, + -0.07991768713640306, + -0.17407470231312427, + -0.18732483013668594, + -0.08463055196615278, + -0.2477115493728633, + -0.2502897149395328, + -0.20398064814312206, + -0.09077284582582087, + 0.22239547449983668, + 0.004462839400732508, + -0.22767156949803466, + 0.04275388869901973, + -0.3059839224841771, + 0.3196937439459745, + 0.006208692442055045, + 0.0001971241890499786, + -0.007422540118618717, + -0.28809669563791657, + -0.28558254792863036, + -0.3869714587097063, + 1, + 0.18322236745325193, + 0.05037142820607642, + -0.03609490261093368, + -0.07217004405828796, + -0.16005348763614577, + -0.23218078070201725, + -0.37656828808083825, + -0.29470837017550655 + ], + [ + -0.0010503798619876774, + -0.02771322371899571, + -0.30506147447521575, + -0.1453791450700764, + -0.00694955039725527, + -0.24240887436050382, + -0.26736609150996915, + -0.2739200544387702, + -0.23862820953475447, + -0.22076087097198827, + -0.2213881765552116, + -0.003859801258113621, + 0.3196275743451608, + 0.25617128287621094, + 0.006949550397255344, + -0.26333120743722876, + -0.0014704079413737543, + -0.021440560786684968, + 0.027554030004254233, + 0.49205202568054424, + -0.2512993032332009, + -0.33385014178843664, + -0.18585512574019983, + -0.18416488323856553, + 0.1605298863682866, + 0.18322236745325193, + 1, + -0.2762680073398803, + -0.3623015030946614, + -0.2451384283025972, + -0.33468123603042815, + -0.7543297340826929, + -0.19188064290675352, + -0.5924430690900127 + ], + [ + -0.0006494991207074467, + 0.0018604411671567017, + -0.048481275955609554, + -0.0014594010172321779, + 0.012306612692869211, + -0.0556619409198579, + -0.08408097138202993, + -0.07733200729381666, + -0.07201850701116608, + -0.050234026791169964, + -0.054338325985925064, + 0.003328006906514837, + 0.019928968647699895, + 0.05170274575687212, + -0.012306612692869208, + -0.04494367118136859, + -0.013428683581908657, + -0.0014940221824271288, + 0.01729491162511594, + 0.14000368638519403, + -0.017196458507178478, + -0.14674905438081498, + -0.04632871286727803, + -0.03964706284937272, + 0.030386796069204806, + 0.05037142820607642, + -0.2762680073398803, + 1, + -0.2235164922454485, + -0.15123448603095185, + -0.20647658168386077, + -0.23667330012633156, + -0.04722022100509185, + -0.21074518016919763 + ], + [ + -0.010516394125351592, + 0.02038346087056509, + 0.028467762106169584, + 0.02464494581436856, + -0.014777815951004124, + 0.0076950987178344265, + 0.023085000444599418, + 0.0449724925036831, + 0.022136746625123563, + 0.02882937792536747, + 0.02504810987080452, + -0.005388027856510724, + -0.07585881574899747, + -0.03215707580783382, + 0.014777815951004133, + 0.023690157563696343, + -0.0004678565419111244, + 0.005613761063888361, + -0.006230482005525746, + -0.05215631834144274, + 0.15389327309228798, + -0.08589992270093438, + 0.02172285552026243, + 0.028232511385228868, + -0.011569992641797876, + -0.03609490261093368, + -0.3623015030946614, + -0.2235164922454485, + 1, + -0.19833089664036088, + -0.270776108381858, + 0.08315302530583908, + 0.020378382125023714, + 0.025594453789595965 + ], + [ + -0.004318975744551275, + 0.01418568213097567, + 0.10534126166196492, + 0.03141940765317757, + -0.009750281447949313, + 0.0716632710172793, + 0.09919210454041022, + 0.07685425917962475, + 0.06270980841796413, + 0.08796930347195206, + 0.08661557153323526, + 0.010626123397145876, + -0.10079964105628228, + -0.08063023581160907, + 0.009750281447949528, + 0.07573985650934924, + -0.00079001062411148, + 0.01746632456448475, + -0.020153136101165186, + -0.19870886170688193, + 0.15891681528381144, + 0.08008211690632902, + 0.060183035947993645, + 0.048166948994686994, + -0.030584308144429393, + -0.07217004405828796, + -0.2451384283025972, + -0.15123448603095185, + -0.19833089664036088, + 1, + -0.18321102469526726, + 0.3293670640558551, + 0.07004784345008332, + 0.2529046056409522 + ], + [ + 0.01627881894278637, + -0.0024069937431968856, + 0.28035324824529767, + 0.1180897178470907, + 0.020515848357693, + 0.26322845571231857, + 0.2787875817043761, + 0.2755370112516221, + 0.26539627802277554, + 0.1981501311372027, + 0.20754366554274195, + -0.001414839543993747, + -0.2260781096915624, + -0.2427981631428492, + -0.02051584835769357, + 0.25792003827196397, + 0.014665913237832425, + 0.0061120739544805805, + -0.024291768931540957, + -0.47691223651608966, + 0.01614171428694319, + 0.5403360993105428, + 0.1842491799195877, + 0.17929367300430438, + -0.1754558932547761, + -0.16005348763614577, + -0.33468123603042815, + -0.20647658168386077, + -0.270776108381858, + -0.18321102469526726, + 1, + 0.7272367846079387, + 0.18543957041906836, + 0.6393119430257832 + ], + [ + 0.005285371870295646, + 0.01568347989913396, + 0.38191150910757077, + 0.16338596691556453, + 0.007877333295041818, + 0.32829748818662485, + 0.36113847824658735, + 0.3615199952862194, + 0.325288454100009, + 0.280263628074823, + 0.2854022671060787, + 0.004823156615386272, + -0.3540493589532626, + -0.323890766966285, + -0.007877333295041807, + 0.33239924473562554, + 0.013786269825793156, + 0.017929529906438778, + -0.037528889581619734, + -0.649345648869048, + 0.2023384083649546, + 0.5638005002286687, + 0.24382246495742915, + 0.23280034413314615, + -0.21019749610817867, + -0.23218078070201725, + -0.7543297340826929, + -0.23667330012633156, + 0.08315302530583908, + 0.3293670640558551, + 0.7272367846079387, + 1, + 0.2468617666408947, + 0.8258804609332017 + ], + [ + -0.013779327268354416, + 0.21987422950593646, + 0.09782497186892049, + -0.11234295350128225, + 0.2480330664757158, + 0.2964469592375873, + 0.4415290881871007, + 0.48260691224313995, + 0.33830139143424953, + 0.6296678921767406, + 0.6272347301103788, + 0.3519304153712528, + 0.1928582184700881, + -0.3385136039067215, + -0.2480330664757158, + 0.4909121973267493, + -0.16136793538251534, + 0.7871948529419658, + -0.7631910615169571, + 0.0589334582292538, + 0.004809615120440435, + -0.07325607300641665, + 0.04240972759459293, + 0.030054608328366407, + 0.27111737739356384, + -0.37656828808083825, + -0.19188064290675352, + -0.04722022100509185, + 0.020378382125023714, + 0.07004784345008332, + 0.18543957041906836, + 0.2468617666408947, + 1, + 0.6510648032262032 + ], + [ + 4.783950839776602e-05, + 0.10241060539532633, + 0.31907236323857324, + 0.0646532494217739, + 0.11300826095473893, + 0.41261876950713655, + 0.510100290145439, + 0.5228814865154369, + 0.4328683682410939, + 0.5157090769923935, + 0.5198665357835017, + 0.15782978286591698, + -0.1994840835675715, + -0.39676537714107735, + -0.1130082609547389, + 0.4690421356971826, + -0.052189866637419764, + 0.3607687920142481, + -0.37487836259896, + -0.4467758743246882, + 0.17056928655718714, + 0.35803561609140894, + 0.18611944938877964, + 0.1826633671546002, + -0.060436278689198375, + -0.29470837017550655, + -0.5924430690900127, + -0.21074518016919763, + 0.025594453789595965, + 0.2529046056409522, + 0.6393119430257832, + 0.8258804609332017, + 0.6510648032262032, + 1 + ] + ] + } + ], + "layout": { + "autosize": false, + "height": 720, + "margin": { + "b": 210, + "l": 210, + "r": 0, + "t": 25 + }, + "template": { + "data": { + "bar": [ + { + "error_x": { + "color": "#2a3f5f" + }, + "error_y": { + "color": "#2a3f5f" + }, + "marker": { + "line": { + "color": "#E5ECF6", + "width": 0.5 + } + }, + "type": "bar" + } + ], + "barpolar": [ + { + "marker": { + "line": { + "color": "#E5ECF6", + "width": 0.5 + } + }, + "type": "barpolar" + } + ], + "carpet": [ + { + "aaxis": { + "endlinecolor": "#2a3f5f", + "gridcolor": "white", + "linecolor": "white", + "minorgridcolor": "white", + "startlinecolor": "#2a3f5f" + }, + "baxis": { + "endlinecolor": "#2a3f5f", + "gridcolor": "white", + "linecolor": "white", + "minorgridcolor": "white", + "startlinecolor": "#2a3f5f" + }, + "type": "carpet" + } + ], + "choropleth": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "type": "choropleth" + } + ], + "contour": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "contour" + } + ], + "contourcarpet": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "type": "contourcarpet" + } + ], + "heatmap": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "heatmap" + } + ], + "heatmapgl": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "heatmapgl" + } + ], + "histogram": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "histogram" + } + ], + "histogram2d": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "histogram2d" + } + ], + "histogram2dcontour": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "histogram2dcontour" + } + ], + "mesh3d": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "type": "mesh3d" + } + ], + "parcoords": [ + { + "line": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "parcoords" + } + ], + "pie": [ + { + "automargin": true, + "type": "pie" + } + ], + "scatter": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatter" + } + ], + "scatter3d": [ + { + "line": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatter3d" + } + ], + "scattercarpet": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattercarpet" + } + ], + "scattergeo": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattergeo" + } + ], + "scattergl": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattergl" + } + ], + "scattermapbox": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattermapbox" + } + ], + "scatterpolar": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatterpolar" + } + ], + "scatterpolargl": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatterpolargl" + } + ], + "scatterternary": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatterternary" + } + ], + "surface": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "surface" + } + ], + "table": [ + { + "cells": { + "fill": { + "color": "#EBF0F8" + }, + "line": { + "color": "white" + } + }, + "header": { + "fill": { + "color": "#C8D4E3" + }, + "line": { + "color": "white" + } + }, + "type": "table" + } + ] + }, + "layout": { + "annotationdefaults": { + "arrowcolor": "#2a3f5f", + "arrowhead": 0, + "arrowwidth": 1 + }, + "coloraxis": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "colorscale": { + "diverging": [ + [ + 0, + "#8e0152" + ], + [ + 0.1, + "#c51b7d" + ], + [ + 0.2, + "#de77ae" + ], + [ + 0.3, + "#f1b6da" + ], + [ + 0.4, + "#fde0ef" + ], + [ + 0.5, + "#f7f7f7" + ], + [ + 0.6, + "#e6f5d0" + ], + [ + 0.7, + "#b8e186" + ], + [ + 0.8, + "#7fbc41" + ], + [ + 0.9, + "#4d9221" + ], + [ + 1, + "#276419" + ] + ], + "sequential": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "sequentialminus": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ] + }, + "colorway": [ + "#636efa", + "#EF553B", + "#00cc96", + "#ab63fa", + "#FFA15A", + "#19d3f3", + "#FF6692", + "#B6E880", + "#FF97FF", + "#FECB52" + ], + "font": { + "color": "#2a3f5f" + }, + "geo": { + "bgcolor": "white", + "lakecolor": "white", + "landcolor": "#E5ECF6", + "showlakes": true, + "showland": true, + "subunitcolor": "white" + }, + "hoverlabel": { + "align": "left" + }, + "hovermode": "closest", + "mapbox": { + "style": "light" + }, + "paper_bgcolor": "white", + "plot_bgcolor": "#E5ECF6", + "polar": { + "angularaxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + }, + "bgcolor": "#E5ECF6", + "radialaxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + } + }, + "scene": { + "xaxis": { + "backgroundcolor": "#E5ECF6", + "gridcolor": "white", + "gridwidth": 2, + "linecolor": "white", + "showbackground": true, + "ticks": "", + "zerolinecolor": "white" + }, + "yaxis": { + "backgroundcolor": "#E5ECF6", + "gridcolor": "white", + "gridwidth": 2, + "linecolor": "white", + "showbackground": true, + "ticks": "", + "zerolinecolor": "white" + }, + "zaxis": { + "backgroundcolor": "#E5ECF6", + "gridcolor": "white", + "gridwidth": 2, + "linecolor": "white", + "showbackground": true, + "ticks": "", + "zerolinecolor": "white" + } + }, + "shapedefaults": { + "line": { + "color": "#2a3f5f" + } + }, + "ternary": { + "aaxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + }, + "baxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + }, + "bgcolor": "#E5ECF6", + "caxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + } + }, + "title": { + "x": 0.05 + }, + "xaxis": { + "automargin": true, + "gridcolor": "white", + "linecolor": "white", + "ticks": "", + "title": { + "standoff": 15 + }, + "zerolinecolor": "white", + "zerolinewidth": 2 + }, + "yaxis": { + "automargin": true, + "gridcolor": "white", + "linecolor": "white", + "ticks": "", + "title": { + "standoff": 15 + }, + "zerolinecolor": "white", + "zerolinewidth": 2 + } + } + }, + "title": { + "text": "Correlation Matrix for variables" + }, + "width": 800, + "xaxis": { + "tickfont": { + "size": 9 + } + }, + "yaxis": { + "tickfont": { + "size": 9 + } + } + } + }, + "text/html": [ + "
\n", + " \n", + " \n", + "
\n", + " \n", + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "correlation = telcom.corr()\n", + "matrix_cols = correlation.columns.tolist()\n", + "corr_array = np.array(correlation)\n", + "trace = go.Heatmap(z=corr_array,\n", + " x=matrix_cols,\n", + " y=matrix_cols,\n", + " colorscale=\"Viridis\",\n", + " colorbar=dict(title=\"Pearson Correlation coefficient\",\n", + " titleside=\"right\"\n", + " ),\n", + " )\n", + "layout = go.Layout(dict(title=\"Correlation Matrix for variables\",\n", + " autosize=False,\n", + " height=720,\n", + " width=800,\n", + " margin=dict(r=0, l=210,\n", + " t=25, b=210,\n", + " ),\n", + " yaxis=dict(tickfont=dict(size=9)),\n", + " xaxis=dict(tickfont=dict(size=9))\n", + " )\n", + " )\n", + "fig = go.Figure(data=[trace], layout=layout)\n", + "py.iplot(fig)\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "_uuid": "f944336cbe67efb3422b79864d9478e2cfbdc860" + }, + "source": [ + "### 1.6 Data Preparation for Training" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": {}, + "outputs": [], + "source": [ + "from sklearn.model_selection import train_test_split\n", + "from sklearn.linear_model import LogisticRegression\n", + "from sklearn.metrics import confusion_matrix,accuracy_score,classification_report\n", + "from sklearn.metrics import roc_auc_score,roc_curve,scorer\n", + "from sklearn.metrics import f1_score\n", + "import statsmodels.api as sm\n", + "from sklearn.metrics import precision_score,recall_score\n", + "from yellowbrick.classifier import DiscriminationThreshold\n", + "\n", + "# Split into a train and test set\n", + "train, test = train_test_split(telcom,test_size = .25 ,random_state = 111)\n", + " \n", + "# Seperating dependent and independent variables\n", + "cols = [i for i in telcom.columns if i not in Id_col + target_col]\n", + "training_x = train[cols]\n", + "training_y = train[target_col]\n", + "testing_x = test[cols]\n", + "testing_y = test[target_col]" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1.7 Training" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "scrolled": true + }, + "outputs": [], + "source": [ + "from xgboost import XGBClassifier\n", + "\n", + "model = XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n", + " colsample_bytree=1, gamma=0, learning_rate=0.9, max_delta_step=0,\n", + " max_depth=7, min_child_weight=1, missing=None, n_estimators=100,\n", + " n_jobs=1, nthread=None, objective='binary:logistic', random_state=0,\n", + " reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,\n", + " silent=True, subsample=1)\n", + "\n", + "# Train model\n", + "model.fit(training_x, training_y)\n", + "predictions = model.predict(testing_x)\n", + "probabilities = model.predict_proba(testing_x)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 1.8 Analysis" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,\n", + " colsample_bynode=1, colsample_bytree=1, gamma=0, gpu_id=-1,\n", + " importance_type='gain', interaction_constraints=None,\n", + " learning_rate=0.9, max_delta_step=0, max_depth=7,\n", + " min_child_weight=1, missing=nan, monotone_constraints=None,\n", + " n_estimators=100, n_jobs=1, nthread=1, num_parallel_tree=1,\n", + " objective='binary:logistic', random_state=0, reg_alpha=0,\n", + " reg_lambda=1, scale_pos_weight=1, seed=0, silent=True,\n", + " subsample=1, tree_method=None, validate_parameters=False,\n", + " verbosity=None)\n", + "\n", + " Classification report : \n", + " precision recall f1-score support\n", + "\n", + " 0 0.81 0.86 0.83 1268\n", + " 1 0.56 0.47 0.51 490\n", + "\n", + " accuracy 0.75 1758\n", + " macro avg 0.69 0.67 0.67 1758\n", + "weighted avg 0.74 0.75 0.74 1758\n", + "\n", + "Accuracy Score : 0.7514220705346986\n", + "Area under curve : 0.6655250112663361 \n", + "\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/home/willem/.pyenv/versions/3.7.4/lib/python3.7/site-packages/plotly/tools.py:465: DeprecationWarning:\n", + "\n", + "plotly.tools.make_subplots is deprecated, please use plotly.subplots.make_subplots instead\n", + "\n" + ] + }, + { + "data": { + "application/vnd.plotly.v1+json": { + "config": { + "linkText": "Export to plot.ly", + "plotlyServerURL": "https://plot.ly", + "showLink": false + }, + "data": [ + { + "colorscale": [ + [ + 0, + "rgb(0,0,255)" + ], + [ + 0.1, + "rgb(51,153,255)" + ], + [ + 0.2, + "rgb(102,204,255)" + ], + [ + 0.3, + "rgb(153,204,255)" + ], + [ + 0.4, + "rgb(204,204,255)" + ], + [ + 0.5, + "rgb(255,255,255)" + ], + [ + 0.6, + "rgb(255,204,255)" + ], + [ + 0.7, + "rgb(255,153,255)" + ], + [ + 0.8, + "rgb(255,102,204)" + ], + [ + 0.9, + "rgb(255,102,102)" + ], + [ + 1, + "rgb(255,0,0)" + ] + ], + "name": "matrix", + "showscale": false, + "type": "heatmap", + "x": [ + "Not churn", + "Churn" + ], + "xaxis": "x", + "y": [ + "Not churn", + "Churn" + ], + "yaxis": "y", + "z": [ + [ + 1090, + 178 + ], + [ + 259, + 231 + ] + ] + }, + { + "line": { + "color": "rgb(22, 96, 167)", + "width": 2 + }, + "name": "Roc : 0.6655250112663361", + "type": "scatter", + "x": [ + 0, + 0, + 0, + 0.0015772870662460567, + 0.0015772870662460567, + 0.0031545741324921135, + 0.0031545741324921135, + 0.003943217665615142, + 0.003943217665615142, + 0.003943217665615142, + 0.003943217665615142, + 0.00473186119873817, + 0.00473186119873817, + 0.005520504731861199, + 0.005520504731861199, + 0.006309148264984227, + 0.006309148264984227, + 0.007097791798107256, + 0.007097791798107256, + 0.008675078864353312, + 0.008675078864353312, + 0.00946372239747634, + 0.00946372239747634, + 0.01025236593059937, + 0.01025236593059937, + 0.011041009463722398, + 0.011041009463722398, + 0.011829652996845425, + 0.011829652996845425, + 0.013406940063091483, + 0.013406940063091483, + 0.017350157728706624, + 0.017350157728706624, + 0.018138801261829655, + 0.018138801261829655, + 0.01892744479495268, + 0.01892744479495268, + 0.01971608832807571, + 0.01971608832807571, + 0.02050473186119874, + 0.02050473186119874, + 0.021293375394321766, + 0.021293375394321766, + 0.02444794952681388, + 0.02444794952681388, + 0.025236593059936908, + 0.025236593059936908, + 0.026813880126182965, + 0.026813880126182965, + 0.027602523659305992, + 0.027602523659305992, + 0.028391167192429023, + 0.028391167192429023, + 0.02917981072555205, + 0.02917981072555205, + 0.030757097791798107, + 0.030757097791798107, + 0.031545741324921134, + 0.031545741324921134, + 0.03391167192429022, + 0.03391167192429022, + 0.03470031545741325, + 0.03470031545741325, + 0.03548895899053628, + 0.03548895899053628, + 0.03627760252365931, + 0.03627760252365931, + 0.03785488958990536, + 0.03785488958990536, + 0.04022082018927445, + 0.04022082018927445, + 0.04100946372239748, + 0.04100946372239748, + 0.0417981072555205, + 0.0417981072555205, + 0.04258675078864353, + 0.04258675078864353, + 0.04416403785488959, + 0.04416403785488959, + 0.044952681388012616, + 0.044952681388012616, + 0.04574132492113565, + 0.04574132492113565, + 0.04652996845425868, + 0.04652996845425868, + 0.050473186119873815, + 0.050473186119873815, + 0.051261829652996846, + 0.051261829652996846, + 0.05362776025236593, + 0.05362776025236593, + 0.05441640378548896, + 0.05441640378548896, + 0.055205047318611984, + 0.055205047318611984, + 0.055993690851735015, + 0.055993690851735015, + 0.056782334384858045, + 0.056782334384858045, + 0.057570977917981075, + 0.057570977917981075, + 0.0583596214511041, + 0.0583596214511041, + 0.061514195583596214, + 0.061514195583596214, + 0.062302839116719244, + 0.062302839116719244, + 0.0638801261829653, + 0.0638801261829653, + 0.06466876971608833, + 0.06466876971608833, + 0.06545741324921135, + 0.06545741324921135, + 0.06782334384858044, + 0.06782334384858044, + 0.07018927444794952, + 0.07018927444794952, + 0.07176656151419558, + 0.07176656151419558, + 0.07334384858044164, + 0.07334384858044164, + 0.07413249211356467, + 0.07413249211356467, + 0.07570977917981073, + 0.07570977917981073, + 0.07649842271293375, + 0.07649842271293375, + 0.07807570977917981, + 0.07807570977917981, + 0.08201892744479496, + 0.08201892744479496, + 0.083596214511041, + 0.083596214511041, + 0.08517350157728706, + 0.08517350157728706, + 0.0859621451104101, + 0.0859621451104101, + 0.08675078864353312, + 0.08675078864353312, + 0.08753943217665615, + 0.08753943217665615, + 0.08990536277602523, + 0.08990536277602523, + 0.09069400630914827, + 0.09069400630914827, + 0.0914826498422713, + 0.0914826498422713, + 0.09542586750788644, + 0.09542586750788644, + 0.09779179810725552, + 0.09779179810725552, + 0.09936908517350158, + 0.09936908517350158, + 0.10252365930599369, + 0.10252365930599369, + 0.10331230283911672, + 0.10331230283911672, + 0.10488958990536278, + 0.10488958990536278, + 0.10646687697160884, + 0.10646687697160884, + 0.10725552050473186, + 0.10725552050473186, + 0.10962145110410094, + 0.10962145110410094, + 0.11041009463722397, + 0.11041009463722397, + 0.11277602523659307, + 0.11277602523659307, + 0.11356466876971609, + 0.11356466876971609, + 0.11593059936908517, + 0.11593059936908517, + 0.11829652996845426, + 0.11829652996845426, + 0.12066246056782334, + 0.12066246056782334, + 0.12145110410094637, + 0.12145110410094637, + 0.1222397476340694, + 0.1222397476340694, + 0.12460567823343849, + 0.12460567823343849, + 0.12618296529968454, + 0.12618296529968454, + 0.12854889589905363, + 0.12854889589905363, + 0.12933753943217666, + 0.12933753943217666, + 0.13249211356466878, + 0.13249211356466878, + 0.13485804416403785, + 0.13485804416403785, + 0.13564668769716087, + 0.13564668769716087, + 0.13643533123028392, + 0.13643533123028392, + 0.138801261829653, + 0.138801261829653, + 0.14274447949526814, + 0.14274447949526814, + 0.1443217665615142, + 0.1443217665615142, + 0.14511041009463724, + 0.14511041009463724, + 0.14589905362776026, + 0.14589905362776026, + 0.14668769716088328, + 0.14668769716088328, + 0.14826498422712933, + 0.14826498422712933, + 0.15063091482649843, + 0.15063091482649843, + 0.1529968454258675, + 0.1529968454258675, + 0.15378548895899052, + 0.15378548895899052, + 0.1553627760252366, + 0.1553627760252366, + 0.15694006309148265, + 0.15694006309148265, + 0.15851735015772872, + 0.16009463722397477, + 0.16009463722397477, + 0.1640378548895899, + 0.1640378548895899, + 0.16561514195583596, + 0.16561514195583596, + 0.16640378548895898, + 0.16640378548895898, + 0.16876971608832808, + 0.16876971608832808, + 0.17034700315457413, + 0.17034700315457413, + 0.17113564668769715, + 0.17113564668769715, + 0.17271293375394323, + 0.17271293375394323, + 0.17429022082018927, + 0.17429022082018927, + 0.17586750788643532, + 0.17586750788643532, + 0.17665615141955837, + 0.17665615141955837, + 0.17823343848580442, + 0.17823343848580442, + 0.17902208201892744, + 0.17902208201892744, + 0.17981072555205047, + 0.1805993690851735, + 0.1829652996845426, + 0.1829652996845426, + 0.18454258675078863, + 0.18454258675078863, + 0.18533123028391169, + 0.18533123028391169, + 0.18848580441640378, + 0.18848580441640378, + 0.1892744479495268, + 0.1892744479495268, + 0.19085173501577288, + 0.19085173501577288, + 0.1916403785488959, + 0.1916403785488959, + 0.19400630914826497, + 0.19400630914826497, + 0.19479495268138802, + 0.19479495268138802, + 0.19558359621451105, + 0.19558359621451105, + 0.19873817034700317, + 0.19873817034700317, + 0.1995268138801262, + 0.1995268138801262, + 0.20189274447949526, + 0.20189274447949526, + 0.20268138801261829, + 0.20268138801261829, + 0.20347003154574134, + 0.20347003154574134, + 0.2058359621451104, + 0.2058359621451104, + 0.2082018927444795, + 0.2082018927444795, + 0.2113564668769716, + 0.2113564668769716, + 0.21293375394321767, + 0.21293375394321767, + 0.21845425867507887, + 0.21845425867507887, + 0.2200315457413249, + 0.2200315457413249, + 0.221608832807571, + 0.221608832807571, + 0.222397476340694, + 0.222397476340694, + 0.22318611987381703, + 0.22318611987381703, + 0.22555205047318613, + 0.22555205047318613, + 0.22712933753943218, + 0.22712933753943218, + 0.2279179810725552, + 0.2279179810725552, + 0.2334384858044164, + 0.2334384858044164, + 0.2358044164037855, + 0.2358044164037855, + 0.23738170347003154, + 0.23738170347003154, + 0.23817034700315456, + 0.23817034700315456, + 0.24290220820189273, + 0.24290220820189273, + 0.24369085173501578, + 0.24369085173501578, + 0.2444794952681388, + 0.2444794952681388, + 0.24526813880126183, + 0.24526813880126183, + 0.24684542586750788, + 0.24684542586750788, + 0.250788643533123, + 0.250788643533123, + 0.25236593059936907, + 0.25236593059936907, + 0.2539432176656151, + 0.2539432176656151, + 0.2547318611987382, + 0.2547318611987382, + 0.2586750788643533, + 0.2586750788643533, + 0.26025236593059936, + 0.26025236593059936, + 0.2610410094637224, + 0.2610410094637224, + 0.26498422712933756, + 0.26498422712933756, + 0.2665615141955836, + 0.2665615141955836, + 0.26735015772870663, + 0.26735015772870663, + 0.26813880126182965, + 0.26813880126182965, + 0.2689274447949527, + 0.2689274447949527, + 0.27208201892744477, + 0.27208201892744477, + 0.27602523659305994, + 0.27602523659305994, + 0.27996845425867506, + 0.27996845425867506, + 0.2823343848580442, + 0.2823343848580442, + 0.2831230283911672, + 0.2831230283911672, + 0.28785488958990535, + 0.28785488958990535, + 0.29652996845425866, + 0.29652996845425866, + 0.2996845425867508, + 0.2996845425867508, + 0.30126182965299686, + 0.30126182965299686, + 0.3020504731861199, + 0.3020504731861199, + 0.305993690851735, + 0.305993690851735, + 0.30757097791798105, + 0.30757097791798105, + 0.3194006309148265, + 0.3194006309148265, + 0.32097791798107256, + 0.32097791798107256, + 0.3225552050473186, + 0.3225552050473186, + 0.32413249211356465, + 0.32413249211356465, + 0.3249211356466877, + 0.3249211356466877, + 0.32965299684542587, + 0.32965299684542587, + 0.3312302839116719, + 0.3312302839116719, + 0.334384858044164, + 0.334384858044164, + 0.3351735015772871, + 0.3351735015772871, + 0.33753943217665616, + 0.33753943217665616, + 0.3383280757097792, + 0.3383280757097792, + 0.3501577287066246, + 0.3501577287066246, + 0.35331230283911674, + 0.35331230283911674, + 0.35646687697160884, + 0.35646687697160884, + 0.3588328075709779, + 0.3588328075709779, + 0.36198738170347006, + 0.36198738170347006, + 0.3635646687697161, + 0.3635646687697161, + 0.3659305993690852, + 0.3659305993690852, + 0.36829652996845424, + 0.36829652996845424, + 0.37302839116719244, + 0.37302839116719244, + 0.3777602523659306, + 0.3777602523659306, + 0.3824921135646688, + 0.3824921135646688, + 0.3840694006309148, + 0.3840694006309148, + 0.3864353312302839, + 0.3864353312302839, + 0.388801261829653, + 0.388801261829653, + 0.38958990536277605, + 0.38958990536277605, + 0.3943217665615142, + 0.3943217665615142, + 0.39668769716088326, + 0.39668769716088326, + 0.3998422712933754, + 0.3998422712933754, + 0.40536277602523657, + 0.40536277602523657, + 0.4061514195583596, + 0.4061514195583596, + 0.4069400630914827, + 0.4069400630914827, + 0.4235015772870662, + 0.4235015772870662, + 0.42586750788643535, + 0.42586750788643535, + 0.42665615141955837, + 0.42665615141955837, + 0.4274447949526814, + 0.4274447949526814, + 0.4282334384858044, + 0.4282334384858044, + 0.43217665615141954, + 0.43217665615141954, + 0.43454258675078866, + 0.43454258675078866, + 0.43769716088328076, + 0.43769716088328076, + 0.444006309148265, + 0.444006309148265, + 0.444794952681388, + 0.444794952681388, + 0.44558359621451105, + 0.44558359621451105, + 0.44637223974763407, + 0.44637223974763407, + 0.44873817034700314, + 0.44873817034700314, + 0.4503154574132492, + 0.4503154574132492, + 0.4518927444794953, + 0.4518927444794953, + 0.45347003154574134, + 0.45347003154574134, + 0.45425867507886436, + 0.45425867507886436, + 0.4558359621451104, + 0.4558359621451104, + 0.47003154574132494, + 0.47003154574132494, + 0.47318611987381703, + 0.47318611987381703, + 0.47870662460567825, + 0.47870662460567825, + 0.48580441640378547, + 0.48580441640378547, + 0.49290220820189273, + 0.49290220820189273, + 0.4960567823343849, + 0.4960567823343849, + 0.49842271293375395, + 0.49842271293375395, + 0.5023659305993691, + 0.5023659305993691, + 0.5063091482649842, + 0.5063091482649842, + 0.5126182965299685, + 0.5126182965299685, + 0.5134069400630915, + 0.5134069400630915, + 0.5165615141955836, + 0.5165615141955836, + 0.5205047318611987, + 0.5205047318611987, + 0.5331230283911672, + 0.5331230283911672, + 0.5528391167192429, + 0.5528391167192429, + 0.554416403785489, + 0.554416403785489, + 0.5662460567823344, + 0.5662460567823344, + 0.5717665615141956, + 0.5717665615141956, + 0.5741324921135647, + 0.5741324921135647, + 0.5757097791798107, + 0.5757097791798107, + 0.5891167192429022, + 0.5906940063091483, + 0.5954258675078864, + 0.5954258675078864, + 0.6041009463722398, + 0.6041009463722398, + 0.6072555205047319, + 0.6072555205047319, + 0.6088328075709779, + 0.6088328075709779, + 0.612776025236593, + 0.612776025236593, + 0.6151419558359621, + 0.6151419558359621, + 0.6238170347003155, + 0.6238170347003155, + 0.6246056782334385, + 0.6246056782334385, + 0.6348580441640379, + 0.6348580441640379, + 0.63801261829653, + 0.63801261829653, + 0.6482649842271293, + 0.6482649842271293, + 0.6529968454258676, + 0.6529968454258676, + 0.6648264984227129, + 0.6648264984227129, + 0.6758675078864353, + 0.6758675078864353, + 0.6892744479495269, + 0.6892744479495269, + 0.7011041009463722, + 0.7011041009463722, + 0.7089905362776026, + 0.7089905362776026, + 0.7342271293375394, + 0.7342271293375394, + 0.7570977917981072, + 0.7570977917981072, + 0.7941640378548895, + 0.7941640378548895, + 0.8138801261829653, + 0.8138801261829653, + 0.8422712933753943, + 0.8422712933753943, + 0.8698738170347003, + 0.8698738170347003, + 0.8738170347003155, + 0.8738170347003155, + 0.8832807570977917, + 0.8832807570977917, + 0.8848580441640379, + 0.8848580441640379, + 0.9069400630914827, + 0.9069400630914827, + 0.9353312302839116, + 0.9353312302839116, + 1 + ], + "xaxis": "x2", + "y": [ + 0, + 0.0020408163265306124, + 0.006122448979591836, + 0.006122448979591836, + 0.01020408163265306, + 0.01020408163265306, + 0.0163265306122449, + 0.0163265306122449, + 0.024489795918367346, + 0.02857142857142857, + 0.03877551020408163, + 0.03877551020408163, + 0.04081632653061224, + 0.04081632653061224, + 0.05714285714285714, + 0.05714285714285714, + 0.05918367346938776, + 0.05918367346938776, + 0.07142857142857142, + 0.07142857142857142, + 0.07551020408163266, + 0.07551020408163266, + 0.07755102040816327, + 0.07755102040816327, + 0.08571428571428572, + 0.08571428571428572, + 0.09183673469387756, + 0.09183673469387756, + 0.09387755102040816, + 0.09387755102040816, + 0.09591836734693877, + 0.09591836734693877, + 0.09795918367346938, + 0.09795918367346938, + 0.10408163265306122, + 0.10408163265306122, + 0.11632653061224489, + 0.11632653061224489, + 0.11836734693877551, + 0.11836734693877551, + 0.1326530612244898, + 0.1326530612244898, + 0.1346938775510204, + 0.1346938775510204, + 0.13877551020408163, + 0.13877551020408163, + 0.1489795918367347, + 0.1489795918367347, + 0.1510204081632653, + 0.1510204081632653, + 0.16938775510204082, + 0.17142857142857143, + 0.17346938775510204, + 0.17346938775510204, + 0.17959183673469387, + 0.17959183673469387, + 0.1816326530612245, + 0.1816326530612245, + 0.1836734693877551, + 0.1836734693877551, + 0.19183673469387755, + 0.19183673469387755, + 0.20408163265306123, + 0.20408163265306123, + 0.20612244897959184, + 0.20612244897959184, + 0.21224489795918366, + 0.21224489795918366, + 0.21428571428571427, + 0.21428571428571427, + 0.21836734693877552, + 0.21836734693877552, + 0.22448979591836735, + 0.22448979591836735, + 0.22857142857142856, + 0.22857142857142856, + 0.23469387755102042, + 0.23469387755102042, + 0.23877551020408164, + 0.23877551020408164, + 0.24081632653061225, + 0.24081632653061225, + 0.24897959183673468, + 0.24897959183673468, + 0.2510204081632653, + 0.2510204081632653, + 0.25918367346938775, + 0.25918367346938775, + 0.26326530612244897, + 0.26326530612244897, + 0.2857142857142857, + 0.2857142857142857, + 0.28775510204081634, + 0.28775510204081634, + 0.2897959183673469, + 0.2897959183673469, + 0.29591836734693877, + 0.29591836734693877, + 0.2979591836734694, + 0.2979591836734694, + 0.3040816326530612, + 0.3040816326530612, + 0.31020408163265306, + 0.31020408163265306, + 0.3122448979591837, + 0.3122448979591837, + 0.3142857142857143, + 0.3142857142857143, + 0.32040816326530613, + 0.32040816326530613, + 0.32653061224489793, + 0.32653061224489793, + 0.3306122448979592, + 0.3306122448979592, + 0.3326530612244898, + 0.3326530612244898, + 0.3346938775510204, + 0.3346938775510204, + 0.336734693877551, + 0.336734693877551, + 0.33877551020408164, + 0.33877551020408164, + 0.3408163265306122, + 0.3408163265306122, + 0.3469387755102041, + 0.3469387755102041, + 0.35306122448979593, + 0.35306122448979593, + 0.3551020408163265, + 0.3551020408163265, + 0.35918367346938773, + 0.35918367346938773, + 0.36122448979591837, + 0.36122448979591837, + 0.363265306122449, + 0.363265306122449, + 0.3653061224489796, + 0.3653061224489796, + 0.3693877551020408, + 0.3693877551020408, + 0.37142857142857144, + 0.37142857142857144, + 0.373469387755102, + 0.373469387755102, + 0.37551020408163266, + 0.37551020408163266, + 0.3816326530612245, + 0.3816326530612245, + 0.38571428571428573, + 0.38571428571428573, + 0.3877551020408163, + 0.3877551020408163, + 0.38979591836734695, + 0.38979591836734695, + 0.39183673469387753, + 0.39183673469387753, + 0.39387755102040817, + 0.39387755102040817, + 0.3979591836734694, + 0.3979591836734694, + 0.4, + 0.4, + 0.4020408163265306, + 0.4020408163265306, + 0.41020408163265304, + 0.41020408163265304, + 0.4122448979591837, + 0.4122448979591837, + 0.41836734693877553, + 0.41836734693877553, + 0.4204081632653061, + 0.4204081632653061, + 0.42244897959183675, + 0.42244897959183675, + 0.42448979591836733, + 0.42448979591836733, + 0.42653061224489797, + 0.42653061224489797, + 0.42857142857142855, + 0.42857142857142855, + 0.4326530612244898, + 0.4326530612244898, + 0.43673469387755104, + 0.43673469387755104, + 0.4387755102040816, + 0.4387755102040816, + 0.4448979591836735, + 0.4448979591836735, + 0.45102040816326533, + 0.45102040816326533, + 0.4530612244897959, + 0.4530612244897959, + 0.45510204081632655, + 0.45510204081632655, + 0.45714285714285713, + 0.45714285714285713, + 0.45918367346938777, + 0.45918367346938777, + 0.4714285714285714, + 0.4714285714285714, + 0.47346938775510206, + 0.47346938775510206, + 0.4816326530612245, + 0.4816326530612245, + 0.48367346938775513, + 0.48367346938775513, + 0.4857142857142857, + 0.4857142857142857, + 0.5020408163265306, + 0.5020408163265306, + 0.5040816326530613, + 0.5040816326530613, + 0.5061224489795918, + 0.5061224489795918, + 0.5102040816326531, + 0.5102040816326531, + 0.5122448979591837, + 0.5122448979591837, + 0.5224489795918368, + 0.5224489795918368, + 0.5244897959183673, + 0.5244897959183673, + 0.5244897959183673, + 0.5265306122448979, + 0.5265306122448979, + 0.5306122448979592, + 0.5306122448979592, + 0.536734693877551, + 0.536734693877551, + 0.5387755102040817, + 0.5387755102040817, + 0.5408163265306123, + 0.5408163265306123, + 0.5428571428571428, + 0.5428571428571428, + 0.5469387755102041, + 0.5469387755102041, + 0.5530612244897959, + 0.5530612244897959, + 0.5551020408163265, + 0.5551020408163265, + 0.5653061224489796, + 0.5653061224489796, + 0.5673469387755102, + 0.5673469387755102, + 0.5714285714285714, + 0.5714285714285714, + 0.573469387755102, + 0.573469387755102, + 0.5755102040816327, + 0.5755102040816327, + 0.5775510204081633, + 0.5775510204081633, + 0.5795918367346938, + 0.5795918367346938, + 0.5836734693877551, + 0.5836734693877551, + 0.5897959183673469, + 0.5897959183673469, + 0.5918367346938775, + 0.5918367346938775, + 0.5938775510204082, + 0.5938775510204082, + 0.5959183673469388, + 0.5959183673469388, + 0.5979591836734693, + 0.5979591836734693, + 0.6020408163265306, + 0.6020408163265306, + 0.6040816326530613, + 0.6040816326530613, + 0.6061224489795919, + 0.6061224489795919, + 0.6122448979591837, + 0.6122448979591837, + 0.6163265306122448, + 0.6163265306122448, + 0.6183673469387755, + 0.6183673469387755, + 0.6204081632653061, + 0.6204081632653061, + 0.6224489795918368, + 0.6224489795918368, + 0.6244897959183674, + 0.6244897959183674, + 0.6285714285714286, + 0.6285714285714286, + 0.6306122448979592, + 0.6306122448979592, + 0.6326530612244898, + 0.6326530612244898, + 0.6346938775510204, + 0.6346938775510204, + 0.636734693877551, + 0.636734693877551, + 0.6387755102040816, + 0.6387755102040816, + 0.6428571428571429, + 0.6428571428571429, + 0.6489795918367347, + 0.6489795918367347, + 0.6530612244897959, + 0.6530612244897959, + 0.6551020408163265, + 0.6551020408163265, + 0.6571428571428571, + 0.6571428571428571, + 0.6632653061224489, + 0.6632653061224489, + 0.6673469387755102, + 0.6673469387755102, + 0.6693877551020408, + 0.6693877551020408, + 0.6755102040816326, + 0.6755102040816326, + 0.6775510204081633, + 0.6775510204081633, + 0.6795918367346939, + 0.6795918367346939, + 0.6836734693877551, + 0.6836734693877551, + 0.6857142857142857, + 0.6857142857142857, + 0.6877551020408164, + 0.6877551020408164, + 0.689795918367347, + 0.689795918367347, + 0.6918367346938775, + 0.6918367346938775, + 0.6938775510204082, + 0.6938775510204082, + 0.6979591836734694, + 0.6979591836734694, + 0.7020408163265306, + 0.7020408163265306, + 0.7040816326530612, + 0.7040816326530612, + 0.7061224489795919, + 0.7061224489795919, + 0.7081632653061225, + 0.7081632653061225, + 0.710204081632653, + 0.710204081632653, + 0.7122448979591837, + 0.7122448979591837, + 0.7163265306122449, + 0.7163265306122449, + 0.7204081632653061, + 0.7204081632653061, + 0.7244897959183674, + 0.7244897959183674, + 0.7326530612244898, + 0.7326530612244898, + 0.7346938775510204, + 0.7346938775510204, + 0.736734693877551, + 0.736734693877551, + 0.7408163265306122, + 0.7408163265306122, + 0.7428571428571429, + 0.7428571428571429, + 0.7448979591836735, + 0.7448979591836735, + 0.746938775510204, + 0.746938775510204, + 0.7510204081632653, + 0.7510204081632653, + 0.7591836734693878, + 0.7591836734693878, + 0.7653061224489796, + 0.7653061224489796, + 0.7673469387755102, + 0.7673469387755102, + 0.7693877551020408, + 0.7693877551020408, + 0.773469387755102, + 0.773469387755102, + 0.7775510204081633, + 0.7775510204081633, + 0.7795918367346939, + 0.7795918367346939, + 0.7816326530612245, + 0.7816326530612245, + 0.7836734693877551, + 0.7836734693877551, + 0.7857142857142857, + 0.7857142857142857, + 0.789795918367347, + 0.789795918367347, + 0.7918367346938775, + 0.7918367346938775, + 0.7938775510204081, + 0.7938775510204081, + 0.7959183673469388, + 0.7959183673469388, + 0.7979591836734694, + 0.7979591836734694, + 0.8, + 0.8, + 0.8020408163265306, + 0.8020408163265306, + 0.8040816326530612, + 0.8040816326530612, + 0.8081632653061225, + 0.8081632653061225, + 0.810204081632653, + 0.810204081632653, + 0.8142857142857143, + 0.8142857142857143, + 0.8163265306122449, + 0.8163265306122449, + 0.8183673469387756, + 0.8183673469387756, + 0.8204081632653061, + 0.8204081632653061, + 0.8224489795918367, + 0.8224489795918367, + 0.8244897959183674, + 0.8244897959183674, + 0.826530612244898, + 0.826530612244898, + 0.8346938775510204, + 0.8346938775510204, + 0.8367346938775511, + 0.8367346938775511, + 0.8428571428571429, + 0.8428571428571429, + 0.8469387755102041, + 0.8469387755102041, + 0.8489795918367347, + 0.8489795918367347, + 0.8510204081632653, + 0.8510204081632653, + 0.8530612244897959, + 0.8530612244897959, + 0.8551020408163266, + 0.8551020408163266, + 0.8571428571428571, + 0.8571428571428571, + 0.8591836734693877, + 0.8591836734693877, + 0.8612244897959184, + 0.8612244897959184, + 0.863265306122449, + 0.863265306122449, + 0.8653061224489796, + 0.8653061224489796, + 0.8693877551020408, + 0.8693877551020408, + 0.8714285714285714, + 0.8714285714285714, + 0.8734693877551021, + 0.8734693877551021, + 0.8755102040816326, + 0.8755102040816326, + 0.8775510204081632, + 0.8775510204081632, + 0.8795918367346939, + 0.8795918367346939, + 0.8816326530612245, + 0.8816326530612245, + 0.8857142857142857, + 0.8857142857142857, + 0.8877551020408163, + 0.8877551020408163, + 0.889795918367347, + 0.889795918367347, + 0.8918367346938776, + 0.8918367346938776, + 0.8938775510204081, + 0.8938775510204081, + 0.8959183673469387, + 0.8959183673469387, + 0.9, + 0.9, + 0.9020408163265307, + 0.9020408163265307, + 0.9040816326530612, + 0.9040816326530612, + 0.9081632653061225, + 0.9081632653061225, + 0.9102040816326531, + 0.9102040816326531, + 0.9122448979591836, + 0.9122448979591836, + 0.9142857142857143, + 0.9142857142857143, + 0.9163265306122449, + 0.9163265306122449, + 0.9183673469387755, + 0.9183673469387755, + 0.9204081632653062, + 0.9204081632653062, + 0.9224489795918367, + 0.9224489795918367, + 0.9244897959183673, + 0.9244897959183673, + 0.926530612244898, + 0.926530612244898, + 0.9285714285714286, + 0.9285714285714286, + 0.9306122448979591, + 0.9306122448979591, + 0.9346938775510204, + 0.9346938775510204, + 0.936734693877551, + 0.936734693877551, + 0.9387755102040817, + 0.9387755102040817, + 0.9408163265306122, + 0.9408163265306122, + 0.9408163265306122, + 0.9408163265306122, + 0.9428571428571428, + 0.9428571428571428, + 0.9448979591836735, + 0.9448979591836735, + 0.9469387755102041, + 0.9469387755102041, + 0.9489795918367347, + 0.9489795918367347, + 0.9510204081632653, + 0.9510204081632653, + 0.9530612244897959, + 0.9530612244897959, + 0.9551020408163265, + 0.9551020408163265, + 0.9571428571428572, + 0.9571428571428572, + 0.9591836734693877, + 0.9591836734693877, + 0.963265306122449, + 0.963265306122449, + 0.9653061224489796, + 0.9653061224489796, + 0.9673469387755103, + 0.9673469387755103, + 0.9693877551020408, + 0.9693877551020408, + 0.9714285714285714, + 0.9714285714285714, + 0.9734693877551021, + 0.9734693877551021, + 0.9755102040816327, + 0.9755102040816327, + 0.9775510204081632, + 0.9775510204081632, + 0.9795918367346939, + 0.9795918367346939, + 0.9816326530612245, + 0.9816326530612245, + 0.9836734693877551, + 0.9836734693877551, + 0.9857142857142858, + 0.9857142857142858, + 0.9877551020408163, + 0.9877551020408163, + 0.9897959183673469, + 0.9897959183673469, + 0.9918367346938776, + 0.9918367346938776, + 0.9938775510204082, + 0.9938775510204082, + 0.9959183673469387, + 0.9959183673469387, + 0.9979591836734694, + 0.9979591836734694, + 1, + 1 + ], + "yaxis": "y2" + }, + { + "line": { + "color": "rgb(205, 12, 24)", + "dash": "dot", + "width": 2 + }, + "type": "scatter", + "x": [ + 0, + 1 + ], + "xaxis": "x2", + "y": [ + 0, + 1 + ], + "yaxis": "y2" + }, + { + "marker": { + "color": [ + 0.40381771326065063, + 0.28717586398124695, + 0.019390283152461052, + 0.018444739282131195, + 0.01538677979260683, + 0.014990396797657013, + 0.014325669966638088, + 0.013916175812482834, + 0.012448116205632687, + 0.01242363452911377, + 0.012042813934385777, + 0.012016979977488518, + 0.011734840460121632, + 0.011441107839345932, + 0.01139832753688097, + 0.009455341845750809, + 0.009003291837871075, + 0.008989614434540272, + 0.00892818532884121, + 0.008804365992546082, + 0.00837039016187191, + 0.008362136781215668, + 0.008307449519634247, + 0.008072943426668644, + 0.00748491520062089, + 0.007470999378710985, + 0.006756791844964027, + 0.006560647860169411, + 0.006558740511536598, + 0.006327662151306868, + 0.0055321562103927135, + 0.00406095152720809, + 0 + ], + "colorscale": [ + [ + 0, + "rgb(0,0,255)" + ], + [ + 0.1, + "rgb(51,153,255)" + ], + [ + 0.2, + "rgb(102,204,255)" + ], + [ + 0.3, + "rgb(153,204,255)" + ], + [ + 0.4, + "rgb(204,204,255)" + ], + [ + 0.5, + "rgb(255,255,255)" + ], + [ + 0.6, + "rgb(255,204,255)" + ], + [ + 0.7, + "rgb(255,153,255)" + ], + [ + 0.8, + "rgb(255,102,204)" + ], + [ + 0.9, + "rgb(255,102,102)" + ], + [ + 1, + "rgb(255,0,0)" + ] + ], + "line": { + "color": "black", + "width": 0.6 + } + }, + "name": "coefficients", + "type": "bar", + "x": [ + "internetservice_fiber_optic", + "contract_month_to_month", + "internetservice_dsl", + "tenure_group_tenure_0_12", + "phoneservice", + "contract_one_year", + "tenure", + "tenure_group_tenure_48_60", + "internetservice_no", + "seniorcitizen", + "onlinesecurity", + "techsupport", + "streamingmovies", + "multiplelines_no", + "contract_two_year", + "paymentmethod_electronic_check", + "onlinebackup", + "totalcharges", + "paymentmethod_bank_transfer_automatic", + "tenure_group_tenure_24_48", + "monthlycharges", + "paperlessbilling", + "streamingtv", + "multiplelines_yes", + "paymentmethod_mailed_check", + "gender", + "dependents", + "partner", + "tenure_group_tenure_12_24", + "deviceprotection", + "paymentmethod_credit_card_automatic", + "tenure_group_tenure_gt_60", + "multiplelines_no_phone_service" + ], + "xaxis": "x3", + "y": [ + 0.40381771326065063, + 0.28717586398124695, + 0.019390283152461052, + 0.018444739282131195, + 0.01538677979260683, + 0.014990396797657013, + 0.014325669966638088, + 0.013916175812482834, + 0.012448116205632687, + 0.01242363452911377, + 0.012042813934385777, + 0.012016979977488518, + 0.011734840460121632, + 0.011441107839345932, + 0.01139832753688097, + 0.009455341845750809, + 0.009003291837871075, + 0.008989614434540272, + 0.00892818532884121, + 0.008804365992546082, + 0.00837039016187191, + 0.008362136781215668, + 0.008307449519634247, + 0.008072943426668644, + 0.00748491520062089, + 0.007470999378710985, + 0.006756791844964027, + 0.006560647860169411, + 0.006558740511536598, + 0.006327662151306868, + 0.0055321562103927135, + 0.00406095152720809, + 0 + ], + "yaxis": "y3" + } + ], + "layout": { + "annotations": [ + { + "font": { + "size": 16 + }, + "showarrow": false, + "text": "Confusion Matrix", + "x": 0.225, + "xanchor": "center", + "xref": "paper", + "y": 1, + "yanchor": "bottom", + "yref": "paper" + }, + { + "font": { + "size": 16 + }, + "showarrow": false, + "text": "Receiver operating characteristic", + "x": 0.775, + "xanchor": "center", + "xref": "paper", + "y": 1, + "yanchor": "bottom", + "yref": "paper" + }, + { + "font": { + "size": 16 + }, + "showarrow": false, + "text": "Feature Importances", + "x": 0.5, + "xanchor": "center", + "xref": "paper", + "y": 0.375, + "yanchor": "bottom", + "yref": "paper" + } + ], + "autosize": false, + "height": 900, + "margin": { + "b": 195 + }, + "paper_bgcolor": "rgba(240,240,240, 0.95)", + "plot_bgcolor": "rgba(240,240,240, 0.95)", + "showlegend": false, + "template": { + "data": { + "bar": [ + { + "error_x": { + "color": "#2a3f5f" + }, + "error_y": { + "color": "#2a3f5f" + }, + "marker": { + "line": { + "color": "#E5ECF6", + "width": 0.5 + } + }, + "type": "bar" + } + ], + "barpolar": [ + { + "marker": { + "line": { + "color": "#E5ECF6", + "width": 0.5 + } + }, + "type": "barpolar" + } + ], + "carpet": [ + { + "aaxis": { + "endlinecolor": "#2a3f5f", + "gridcolor": "white", + "linecolor": "white", + "minorgridcolor": "white", + "startlinecolor": "#2a3f5f" + }, + "baxis": { + "endlinecolor": "#2a3f5f", + "gridcolor": "white", + "linecolor": "white", + "minorgridcolor": "white", + "startlinecolor": "#2a3f5f" + }, + "type": "carpet" + } + ], + "choropleth": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "type": "choropleth" + } + ], + "contour": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "contour" + } + ], + "contourcarpet": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "type": "contourcarpet" + } + ], + "heatmap": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "heatmap" + } + ], + "heatmapgl": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "heatmapgl" + } + ], + "histogram": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "histogram" + } + ], + "histogram2d": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "histogram2d" + } + ], + "histogram2dcontour": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "histogram2dcontour" + } + ], + "mesh3d": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "type": "mesh3d" + } + ], + "parcoords": [ + { + "line": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "parcoords" + } + ], + "pie": [ + { + "automargin": true, + "type": "pie" + } + ], + "scatter": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatter" + } + ], + "scatter3d": [ + { + "line": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatter3d" + } + ], + "scattercarpet": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattercarpet" + } + ], + "scattergeo": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattergeo" + } + ], + "scattergl": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattergl" + } + ], + "scattermapbox": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scattermapbox" + } + ], + "scatterpolar": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatterpolar" + } + ], + "scatterpolargl": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatterpolargl" + } + ], + "scatterternary": [ + { + "marker": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "type": "scatterternary" + } + ], + "surface": [ + { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + }, + "colorscale": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "type": "surface" + } + ], + "table": [ + { + "cells": { + "fill": { + "color": "#EBF0F8" + }, + "line": { + "color": "white" + } + }, + "header": { + "fill": { + "color": "#C8D4E3" + }, + "line": { + "color": "white" + } + }, + "type": "table" + } + ] + }, + "layout": { + "annotationdefaults": { + "arrowcolor": "#2a3f5f", + "arrowhead": 0, + "arrowwidth": 1 + }, + "coloraxis": { + "colorbar": { + "outlinewidth": 0, + "ticks": "" + } + }, + "colorscale": { + "diverging": [ + [ + 0, + "#8e0152" + ], + [ + 0.1, + "#c51b7d" + ], + [ + 0.2, + "#de77ae" + ], + [ + 0.3, + "#f1b6da" + ], + [ + 0.4, + "#fde0ef" + ], + [ + 0.5, + "#f7f7f7" + ], + [ + 0.6, + "#e6f5d0" + ], + [ + 0.7, + "#b8e186" + ], + [ + 0.8, + "#7fbc41" + ], + [ + 0.9, + "#4d9221" + ], + [ + 1, + "#276419" + ] + ], + "sequential": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ], + "sequentialminus": [ + [ + 0, + "#0d0887" + ], + [ + 0.1111111111111111, + "#46039f" + ], + [ + 0.2222222222222222, + "#7201a8" + ], + [ + 0.3333333333333333, + "#9c179e" + ], + [ + 0.4444444444444444, + "#bd3786" + ], + [ + 0.5555555555555556, + "#d8576b" + ], + [ + 0.6666666666666666, + "#ed7953" + ], + [ + 0.7777777777777778, + "#fb9f3a" + ], + [ + 0.8888888888888888, + "#fdca26" + ], + [ + 1, + "#f0f921" + ] + ] + }, + "colorway": [ + "#636efa", + "#EF553B", + "#00cc96", + "#ab63fa", + "#FFA15A", + "#19d3f3", + "#FF6692", + "#B6E880", + "#FF97FF", + "#FECB52" + ], + "font": { + "color": "#2a3f5f" + }, + "geo": { + "bgcolor": "white", + "lakecolor": "white", + "landcolor": "#E5ECF6", + "showlakes": true, + "showland": true, + "subunitcolor": "white" + }, + "hoverlabel": { + "align": "left" + }, + "hovermode": "closest", + "mapbox": { + "style": "light" + }, + "paper_bgcolor": "white", + "plot_bgcolor": "#E5ECF6", + "polar": { + "angularaxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + }, + "bgcolor": "#E5ECF6", + "radialaxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + } + }, + "scene": { + "xaxis": { + "backgroundcolor": "#E5ECF6", + "gridcolor": "white", + "gridwidth": 2, + "linecolor": "white", + "showbackground": true, + "ticks": "", + "zerolinecolor": "white" + }, + "yaxis": { + "backgroundcolor": "#E5ECF6", + "gridcolor": "white", + "gridwidth": 2, + "linecolor": "white", + "showbackground": true, + "ticks": "", + "zerolinecolor": "white" + }, + "zaxis": { + "backgroundcolor": "#E5ECF6", + "gridcolor": "white", + "gridwidth": 2, + "linecolor": "white", + "showbackground": true, + "ticks": "", + "zerolinecolor": "white" + } + }, + "shapedefaults": { + "line": { + "color": "#2a3f5f" + } + }, + "ternary": { + "aaxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + }, + "baxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + }, + "bgcolor": "#E5ECF6", + "caxis": { + "gridcolor": "white", + "linecolor": "white", + "ticks": "" + } + }, + "title": { + "x": 0.05 + }, + "xaxis": { + "automargin": true, + "gridcolor": "white", + "linecolor": "white", + "ticks": "", + "title": { + "standoff": 15 + }, + "zerolinecolor": "white", + "zerolinewidth": 2 + }, + "yaxis": { + "automargin": true, + "gridcolor": "white", + "linecolor": "white", + "ticks": "", + "title": { + "standoff": 15 + }, + "zerolinecolor": "white", + "zerolinewidth": 2 + } + } + }, + "title": { + "text": "Model performance" + }, + "width": 800, + "xaxis": { + "anchor": "y", + "domain": [ + 0, + 0.45 + ] + }, + "xaxis2": { + "anchor": "y2", + "domain": [ + 0.55, + 1 + ], + "title": { + "text": "false positive rate" + } + }, + "xaxis3": { + "anchor": "y3", + "domain": [ + 0, + 1 + ], + "showgrid": true, + "tickangle": 90, + "tickfont": { + "size": 10 + } + }, + "yaxis": { + "anchor": "x", + "domain": [ + 0.625, + 1 + ] + }, + "yaxis2": { + "anchor": "x2", + "domain": [ + 0.625, + 1 + ], + "title": { + "text": "true positive rate" + } + }, + "yaxis3": { + "anchor": "x3", + "domain": [ + 0, + 0.375 + ] + } + } + }, + "text/html": [ + "
\n", + " \n", + " \n", + "
\n", + " \n", + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "coefficients = pd.DataFrame(model.feature_importances_)\n", + "column_df = pd.DataFrame(cols)\n", + "coef_sumry = (pd.merge(coefficients, column_df, left_index=True,\n", + " right_index=True, how=\"left\"))\n", + "coef_sumry.columns = [\"coefficients\", \"features\"]\n", + "coef_sumry = coef_sumry.sort_values(by=\"coefficients\", ascending=False)\n", + "\n", + "print(model)\n", + "print(\"\\n Classification report : \\n\", classification_report(testing_y, predictions))\n", + "print(\"Accuracy Score : \", accuracy_score(testing_y, predictions))\n", + "# confusion matrix\n", + "conf_matrix = confusion_matrix(testing_y, predictions)\n", + "# roc_auc_score\n", + "model_roc_auc = roc_auc_score(testing_y, predictions)\n", + "print(\"Area under curve : \", model_roc_auc, \"\\n\")\n", + "fpr, tpr, thresholds = roc_curve(testing_y, probabilities[:, 1])\n", + "\n", + "# plot confusion matrix\n", + "trace1 = go.Heatmap(z=conf_matrix,\n", + " x=[\"Not churn\", \"Churn\"],\n", + " y=[\"Not churn\", \"Churn\"],\n", + " showscale=False, colorscale=\"Picnic\",\n", + " name=\"matrix\")\n", + "\n", + "# plot roc curve\n", + "trace2 = go.Scatter(x=fpr, y=tpr,\n", + " name=\"Roc : \" + str(model_roc_auc),\n", + " line=dict(color=('rgb(22, 96, 167)'), width=2))\n", + "trace3 = go.Scatter(x=[0, 1], y=[0, 1],\n", + " line=dict(color=('rgb(205, 12, 24)'), width=2,\n", + " dash='dot'))\n", + "\n", + "# plot coeffs\n", + "trace4 = go.Bar(x=coef_sumry[\"features\"], y=coef_sumry[\"coefficients\"],\n", + " name=\"coefficients\",\n", + " marker=dict(color=coef_sumry[\"coefficients\"],\n", + " colorscale=\"Picnic\",\n", + " line=dict(width=.6, color=\"black\")))\n", + "\n", + "# subplots\n", + "fig = tls.make_subplots(rows=2, cols=2, specs=[[{}, {}], [{'colspan': 2}, None]],\n", + " subplot_titles=('Confusion Matrix',\n", + " 'Receiver operating characteristic',\n", + " 'Feature Importances'))\n", + "\n", + "fig.append_trace(trace1, 1, 1)\n", + "fig.append_trace(trace2, 1, 2)\n", + "fig.append_trace(trace3, 1, 2)\n", + "fig.append_trace(trace4, 2, 1)\n", + "\n", + "fig['layout'].update(showlegend=False, title=\"Model performance\",\n", + " autosize=False, height=900, width=800,\n", + " plot_bgcolor='rgba(240,240,240, 0.95)',\n", + " paper_bgcolor='rgba(240,240,240, 0.95)',\n", + " margin=dict(b=195))\n", + "fig[\"layout\"][\"xaxis2\"].update(dict(title=\"false positive rate\"))\n", + "fig[\"layout\"][\"yaxis2\"].update(dict(title=\"true positive rate\"))\n", + "fig[\"layout\"][\"xaxis3\"].update(dict(showgrid=True, tickfont=dict(size=10),\n", + " tickangle=90))\n", + "py.iplot(fig)\n", + "\n", + "visualizer = DiscriminationThreshold(model)\n", + "visualizer.fit(training_x, training_y)\n", + "visualizer.poof()" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "## 2. Churn Modelling (with Feast)" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "1. Make sure your features are registered with Feast and that data is being published to stores\n", + "2. Train your model\n", + "3. Serve your model" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.1 Configure Feast" + ] + }, + { + "cell_type": "code", + "execution_count": 540, + "metadata": {}, + "outputs": [], + "source": [ + "os.environ['FEAST_CORE_URL'] = 'localhost:6565'\n", + "os.environ['FEAST_ONLINE_URL'] = 'localhost:6566'\n", + "os.environ['FEAST_BATCH_URL'] = 'localhost:6567'\n", + "os.environ['FEAST_PROJECT'] = 'default'" + ] + }, + { + "cell_type": "code", + "execution_count": 527, + "metadata": {}, + "outputs": [], + "source": [ + "from feast import Client, FeatureSet, Entity, ValueType" + ] + }, + { + "cell_type": "code", + "execution_count": 528, + "metadata": {}, + "outputs": [], + "source": [ + "client = Client(core_url=os.environ['FEAST_CORE_URL'])\n", + "client.set_project(os.environ['FEAST_PROJECT'])" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.2 Create a Feature Set" + ] + }, + { + "cell_type": "code", + "execution_count": 529, + "metadata": {}, + "outputs": [], + "source": [ + "customer_churn_fs = FeatureSet('customer_churn')" + ] + }, + { + "cell_type": "code", + "execution_count": 530, + "metadata": {}, + "outputs": [], + "source": [ + "# Add a datetime column to todays date\n", + "telcom['datetime'] = pd.Series([dt.datetime.now()] * len(entity_df))" + ] + }, + { + "cell_type": "code", + "execution_count": 278, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Entity customer_id(ValueType.STRING) manually updated (replacing an existing field).\n", + "Feature gender (ValueType.INT64) added from dataframe.\n", + "Feature seniorcitizen (ValueType.INT64) added from dataframe.\n", + "Feature partner (ValueType.INT64) added from dataframe.\n", + "Feature dependents (ValueType.INT64) added from dataframe.\n", + "Feature phoneservice (ValueType.INT64) added from dataframe.\n", + "Feature onlinesecurity (ValueType.INT64) added from dataframe.\n", + "Feature onlinebackup (ValueType.INT64) added from dataframe.\n", + "Feature deviceprotection (ValueType.INT64) added from dataframe.\n", + "Feature techsupport (ValueType.INT64) added from dataframe.\n", + "Feature streamingtv (ValueType.INT64) added from dataframe.\n", + "Feature streamingmovies (ValueType.INT64) added from dataframe.\n", + "Feature paperlessbilling (ValueType.INT64) added from dataframe.\n", + "Feature churn (ValueType.INT64) added from dataframe.\n", + "Feature multiplelines_no (ValueType.INT64) added from dataframe.\n", + "Feature multiplelines_no_phone_service (ValueType.INT64) added from dataframe.\n", + "Feature multiplelines_yes (ValueType.INT64) added from dataframe.\n", + "Feature internetservice_dsl (ValueType.INT64) added from dataframe.\n", + "Feature internetservice_fiber_optic (ValueType.INT64) added from dataframe.\n", + "Feature internetservice_no (ValueType.INT64) added from dataframe.\n", + "Feature contract_month_to_month (ValueType.INT64) added from dataframe.\n", + "Feature contract_one_year (ValueType.INT64) added from dataframe.\n", + "Feature contract_two_year (ValueType.INT64) added from dataframe.\n", + "Feature paymentmethod_bank_transfer_automatic (ValueType.INT64) added from dataframe.\n", + "Feature paymentmethod_credit_card_automatic (ValueType.INT64) added from dataframe.\n", + "Feature paymentmethod_electronic_check (ValueType.INT64) added from dataframe.\n", + "Feature paymentmethod_mailed_check (ValueType.INT64) added from dataframe.\n", + "Feature tenure_group_tenure_0_12 (ValueType.INT64) added from dataframe.\n", + "Feature tenure_group_tenure_12_24 (ValueType.INT64) added from dataframe.\n", + "Feature tenure_group_tenure_24_48 (ValueType.INT64) added from dataframe.\n", + "Feature tenure_group_tenure_48_60 (ValueType.INT64) added from dataframe.\n", + "Feature tenure_group_tenure_gt_60 (ValueType.INT64) added from dataframe.\n", + "Feature tenure (ValueType.DOUBLE) added from dataframe.\n", + "Feature monthlycharges (ValueType.DOUBLE) added from dataframe.\n", + "Feature totalcharges (ValueType.DOUBLE) added from dataframe.\n", + "\n" + ] + } + ], + "source": [ + "customer_churn_fs.infer_fields_from_df(telcom, entities=[Entity(name='customer_id', dtype=ValueType.STRING)])" + ] + }, + { + "cell_type": "code", + "execution_count": 281, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Feature set updated/created: \"customer_churn:1\"\n" + ] + } + ], + "source": [ + "client.apply(customer_churn_fs)" + ] + }, + { + "cell_type": "code", + "execution_count": 285, + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{\n", + " \"spec\": {\n", + " \"name\": \"customer_churn\",\n", + " \"version\": 1,\n", + " \"entities\": [\n", + " {\n", + " \"name\": \"customer_id\",\n", + " \"valueType\": \"STRING\"\n", + " }\n", + " ],\n", + " \"features\": [\n", + " {\n", + " \"name\": \"churn\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"seniorcitizen\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"contract_month_to_month\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"streamingmovies\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"dependents\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"paymentmethod_credit_card_automatic\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"multiplelines_no_phone_service\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"techsupport\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"internetservice_no\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"tenure_group_tenure_gt_60\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"phoneservice\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"deviceprotection\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"tenure\",\n", + " \"valueType\": \"DOUBLE\"\n", + " },\n", + " {\n", + " \"name\": \"internetservice_fiber_optic\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"multiplelines_no\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"monthlycharges\",\n", + " \"valueType\": \"DOUBLE\"\n", + " },\n", + " {\n", + " \"name\": \"gender\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"internetservice_dsl\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"onlinebackup\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"paymentmethod_electronic_check\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"tenure_group_tenure_48_60\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"paymentmethod_bank_transfer_automatic\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"totalcharges\",\n", + " \"valueType\": \"DOUBLE\"\n", + " },\n", + " {\n", + " \"name\": \"paymentmethod_mailed_check\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"tenure_group_tenure_0_12\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"tenure_group_tenure_12_24\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"multiplelines_yes\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"tenure_group_tenure_24_48\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"streamingtv\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"onlinesecurity\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"contract_one_year\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"paperlessbilling\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"contract_two_year\",\n", + " \"valueType\": \"INT64\"\n", + " },\n", + " {\n", + " \"name\": \"partner\",\n", + " \"valueType\": \"INT64\"\n", + " }\n", + " ],\n", + " \"maxAge\": \"0s\",\n", + " \"source\": {\n", + " \"type\": \"KAFKA\",\n", + " \"kafkaSourceConfig\": {\n", + " \"bootstrapServers\": \"10.202.250.99:31190\",\n", + " \"topic\": \"feast\"\n", + " }\n", + " },\n", + " \"project\": \"default\"\n", + " },\n", + " \"meta\": {\n", + " \"createdTimestamp\": \"2020-03-15T07:47:52Z\",\n", + " \"status\": \"STATUS_READY\"\n", + " }\n", + "}\n" + ] + } + ], + "source": [ + "customer_churn_fs = client.get_feature_set('customer_churn')\n", + "print(client.get_feature_set('customer_churn'))" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "### 2.3 Load Features Into Feast" + ] + }, + { + "cell_type": "code", + "execution_count": 286, + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\r", + " 0%| | 0/7032 [00:00 Date: Sat, 28 Mar 2020 04:20:39 +0000 Subject: [PATCH 098/176] GitBook: [master] 3 pages modified --- docs/SUMMARY.md | 1 + docs/administration/troubleshooting.md | 2 +- docs/introduction/getting-help.md | 14 +++++++------- 3 files changed, 9 insertions(+), 8 deletions(-) diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md index d5c844a1810..f74a691970c 100644 --- a/docs/SUMMARY.md +++ b/docs/SUMMARY.md @@ -22,6 +22,7 @@ ## Tutorials * [Basic](https://github.com/gojek/feast/blob/master/examples/basic/basic.ipynb) +* [Churn Prediction \(XGBoost\)](https://github.com/gojek/feast/blob/master/examples/feast-xgboost-churn-prediction-tutorial/Telecom%20Customer%20Churn%20Prediction%20%28with%20Feast%20and%20XGBoost%29.ipynb) ## Administration diff --git a/docs/administration/troubleshooting.md b/docs/administration/troubleshooting.md index bcd4c9bdf93..a16a74d5d98 100644 --- a/docs/administration/troubleshooting.md +++ b/docs/administration/troubleshooting.md @@ -1,6 +1,6 @@ # Troubleshooting -If at any point in time you cannot resolve a problem, please see the [Getting Help](../getting-help.md) section for reaching out to the Feast community. +If at any point in time you cannot resolve a problem, please see the [Getting Help](https://github.com/gojek/feast/tree/75f3b783e5a7c5e0217a3020422548fb0d0ce0bf/docs/getting-help.md) section for reaching out to the Feast community. ## How can I verify that all services are operational? diff --git a/docs/introduction/getting-help.md b/docs/introduction/getting-help.md index 887d0950d09..5e060bb805d 100644 --- a/docs/introduction/getting-help.md +++ b/docs/introduction/getting-help.md @@ -1,15 +1,15 @@ # Getting Help -### Chat +## Chat * Come and say hello in [\#Feast](https://join.slack.com/t/kubeflow/shared_invite/zt-cpr020z4-PfcAue_2nw67~iIDy7maAQ) over in the Kubeflow Slack. -### GitHub +## GitHub * Feast's GitHub repo can be [found here](https://github.com/gojek/feast/). * Found a bug or need a feature? [Create an issue on GitHub](https://github.com/gojek/feast/issues/new) -### Community Call +## Community Call We have a community call every 2 weeks. Alternating between two times. @@ -18,19 +18,19 @@ We have a community call every 2 weeks. Alternating between two times. Please join the [**feast-dev**](getting-help.md#feast-development) mailing list to receive the the calendar invitation. -### Mailing list +## Mailing list -#### Feast discussion +### Feast discussion * Google Group: [https://groups.google.com/d/forum/feast-discuss](https://groups.google.com/d/forum/feast-discuss) * Mailing List: [feast-discuss@googlegroups.com](mailto:feast-discuss@googlegroups.com) -#### Feast development +### Feast development * Google Group: [https://groups.google.com/d/forum/feast-dev](https://groups.google.com/d/forum/feast-dev) * Mailing List: [feast-dev@googlegroups.com](mailto:feast-dev@googlegroups.com) -### Google Drive +## Google Drive The Feast community also maintains a [Google Drive](https://drive.google.com/drive/u/0/folders/0AAe8j7ZK3sxSUk9PVA) with documents like RFCs, meeting notes, or roadmaps. Please join one of the above mailing lists \(feast-dev or feast-discuss\) to gain access to the drive. From e4da59d04a17dea1008cd0c8a58c22de59e8fbc3 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Sat, 28 Mar 2020 12:57:51 +0800 Subject: [PATCH 099/176] Disable tests that should be manual or that exist in GitHub Actions (#583) --- .prow/config.yaml | 7 ------- 1 file changed, 7 deletions(-) diff --git a/.prow/config.yaml b/.prow/config.yaml index 2e10ecfaa7f..085cfe85423 100644 --- a/.prow/config.yaml +++ b/.prow/config.yaml @@ -63,7 +63,6 @@ presubmits: gojek/feast: - name: test-core-and-ingestion decorate: true - always_run: true spec: containers: - image: maven:3.6-jdk-11 @@ -91,7 +90,6 @@ presubmits: - name: test-serving decorate: true - always_run: true spec: containers: - image: maven:3.6-jdk-11 @@ -111,7 +109,6 @@ presubmits: - name: test-java-sdk decorate: true - always_run: true spec: containers: - image: maven:3.6-jdk-11 @@ -131,7 +128,6 @@ presubmits: - name: test-python-sdk decorate: true - always_run: true spec: containers: - image: python:3.7 @@ -139,7 +135,6 @@ presubmits: - name: test-golang-sdk decorate: true - always_run: true spec: containers: - image: golang:1.13 @@ -147,7 +142,6 @@ presubmits: - name: test-end-to-end decorate: true - always_run: true spec: containers: - image: maven:3.6-jdk-11 @@ -175,7 +169,6 @@ presubmits: - name: test-end-to-end-batch decorate: true - always_run: true spec: volumes: - name: service-account From de3fd2b4269428da6b53ee00701c1e4eb0cbf423 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Sat, 28 Mar 2020 13:41:51 +0800 Subject: [PATCH 100/176] Fix caching for maven in unit-tests.yml We are getting cache misses between builds. This can be seen here: https://github.com/gojek/feast/runs/541172041?check_suite_focus=true This is a common problem with GitHub Actions, and apparently its necessary to change the cache key in order to resolve it. --- .github/workflows/unit-tests.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/unit-tests.yml b/.github/workflows/unit-tests.yml index 676b1b50987..6fd5bd51cc0 100644 --- a/.github/workflows/unit-tests.yml +++ b/.github/workflows/unit-tests.yml @@ -16,9 +16,9 @@ jobs: - uses: actions/cache@v1 with: path: ~/.m2/repository - key: ${{ runner.os }}-maven-jdk11-${{ hashFiles('**/pom.xml') }} + key: ${{ runner.os }}-maven-${{ hashFiles('**/pom.xml') }} restore-keys: | - ${{ runner.os }}-maven-jdk11- + ${{ runner.os }}-maven- - name: test java run: make test-java @@ -42,4 +42,4 @@ jobs: - name: install dependencies run: make compile-protos-go - name: test go - run: make test-go \ No newline at end of file + run: make test-go From 880269b2d30e15b27e1140bf41e9bd2540bcdac0 Mon Sep 17 00:00:00 2001 From: Ches Martin Date: Sat, 28 Mar 2020 13:13:43 +0700 Subject: [PATCH 101/176] Enforce JDK 11 for development, but build Ingestion to target Java 8 (#518) * Require Java 11 with Enforcer We build for Java 11 now, so the build will fail with older JDKs. Have the Enforcer plugin do that for a more lucid error message. This reverts 190e605 which was squash-merged in a larger commit. Partially addresses #517 * Build Ingestion to target Java 8, for Beam compat As well as datatypes-java since ingestion depends on it. Java 11 is desirable for the other components, but for Beam it may impose limitations on what runners Feast can support, if it is even safe to run on an 11 JRE now. https://issues.apache.org/jira/browse/BEAM-2530 References #517 --- datatypes/java/pom.xml | 9 +++++ docs/contributing/development-guide.md | 34 ++++++++++++++++++- ingestion/pom.xml | 9 +++++ .../WriteFeatureValueMetricsDoFnTest.java | 4 +-- pom.xml | 23 +++++++------ 5 files changed, 66 insertions(+), 13 deletions(-) diff --git a/datatypes/java/pom.xml b/datatypes/java/pom.xml index 5810a6db96a..a127853258e 100644 --- a/datatypes/java/pom.xml +++ b/datatypes/java/pom.xml @@ -37,6 +37,15 @@ + + org.apache.maven.plugins + maven-compiler-plugin + + + 8 + + + org.apache.maven.plugins maven-dependency-plugin diff --git a/docs/contributing/development-guide.md b/docs/contributing/development-guide.md index c81a7050c9d..395ef5de8e8 100644 --- a/docs/contributing/development-guide.md +++ b/docs/contributing/development-guide.md @@ -21,7 +21,7 @@ The following software is required for Feast development * Java SE Development Kit 11 * Python version 3.6 \(or above\) and pip -* [Maven ](https://maven.apache.org/install.html)version 3.6.x +* [Maven](https://maven.apache.org/install.html) version 3.6.x Additionally, [grpc\_cli](https://github.com/grpc/grpc/blob/master/doc/command_line_tool.md) is useful for debugging and quick testing of gRPC endpoints. @@ -444,3 +444,35 @@ If you have made it to this point successfully you should have a functioning Fea It is important to note that most of the functionality demonstrated above is already available in a more abstracted form in the Python SDK \(Feast management, data ingestion, feature retrieval\) and the Java/Go SDKs \(feature retrieval\). However, it is useful to understand these internals from a development standpoint. +### 5 Appendix + +#### 5.1 Java / JDK Versions + +Feast requires a Java 11 or greater JDK for building the project. This is checked by Maven so you'll be informed if you try to use an older version. + +Leaf application modules of the build such as the Core and Serving APIs compile with [the `javac --release` flag] set for 11, and Ingestion and shared library modules that it uses target release 8. Here's why. + +While we want to take advantage of advancements in the (long-term supported) language and platform, and for contributors to enjoy those as well, Apache Beam forms a major part of Feast's Ingestion component and fully validating Beam on Java 11 [is an open issue][BEAM-2530]. Moreover, Beam runners other than the DirectRunner may lag behind further still—Spark does not _build or run_ on Java 11 until its version 3.0 which is still in preview. Presumably Beam's SparkRunner will have to wait for Spark, and for its implementation to update to Spark 3. + +To have confidence in Beam stability and our users' ability to deploy Feast on a range of Beam runners, we will continue to target Java 8 bytecode and Platform version for Feast Ingestion, until the ecosystem moves forward. + +You do _not_ need a Java 8 SDK installed for development. Newer JDKs can build for the older platform, and Feast's Maven build does this automatically. + +See [Feast issue #517][\#517] for discussion. + +[the `javac --release` flag]: https://stackoverflow.com/questions/43102787/what-is-the-release-flag-in-the-java-9-compiler +[BEAM-2530]: https://issues.apache.org/jira/browse/BEAM-2530 +[\#517]: https://github.com/gojek/feast/issues/517 + +#### 5.2 IntelliJ Tips and Troubleshooting + +For IntelliJ users, this section collects notes and setup recommendations for working comfortably on the Feast project, especially to coexist as peacefully as possible with the Maven build. + +##### Language Level + +IntelliJ uses a notion of "Language Level" to drive assistance features according to the target Java version of a project. It often infers this appropriately from a Maven import, but it's wise to check, especially for a multi-module project with differing target Java releases across modules, like ours. Language level [can be set per module][module lang level]—if IntelliJ is suggesting things that turn out to fail when building with `mvn`, make sure the language level of the module in question corresponds to the `` set for `maven-compiler-plugin` in the module's `pom.xml` (11 is project default if the module doesn't set one). + +Ensure that the Project _SDK_ (not language level) is set to a Java 11 JDK, and that all modules use the Project SDK (see [the Dependencies tab] in the Modules view). + +[module lang level]: https://www.jetbrains.com/help/idea/sources-tab.html#module_language_level +[the Dependencies tab]: https://www.jetbrains.com/help/idea/dependencies.html diff --git a/ingestion/pom.xml b/ingestion/pom.xml index ccc8ca04510..47204c33f29 100644 --- a/ingestion/pom.xml +++ b/ingestion/pom.xml @@ -31,6 +31,15 @@ + + org.apache.maven.plugins + maven-compiler-plugin + + + 8 + + + org.apache.maven.plugins maven-shade-plugin diff --git a/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFnTest.java b/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFnTest.java index d2b0275c6fe..88e1bf8088d 100644 --- a/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFnTest.java +++ b/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFnTest.java @@ -142,7 +142,7 @@ private Map> readTestInput(String path) throws IOEx } List colNames = new ArrayList<>(); for (String line : lines) { - if (line.strip().length() < 1) { + if (line.trim().length() < 1) { continue; } String[] splits = line.split(","); @@ -156,7 +156,7 @@ private Map> readTestInput(String path) throws IOEx Builder featureRowBuilder = FeatureRow.newBuilder(); for (int i = 0; i < splits.length; i++) { - String colVal = splits[i].strip(); + String colVal = splits[i].trim(); if (i == 0) { featureRowBuilder.setFeatureSet(colVal); continue; diff --git a/pom.xml b/pom.xml index 37961f0be3e..3abb0eb9ace 100644 --- a/pom.xml +++ b/pom.xml @@ -349,7 +349,6 @@ org.apache.maven.plugins maven-javadoc-plugin - 3.1.1 attach-javadocs @@ -408,15 +407,8 @@ org.apache.maven.plugins maven-compiler-plugin - 3.8.1 11 - - -Xlint:all - - - -Xdoclint:-syntax - @@ -442,7 +434,7 @@ [3.6,4.0) - [1.8,11.1) + [11.0,) @@ -574,6 +566,17 @@ docker-maven-plugin 0.20.1 + + org.apache.maven.plugins + maven-compiler-plugin + 3.8.1 + + + -Xlint:all + -Xdoclint:all + + + org.apache.maven.plugins maven-dependency-plugin @@ -591,7 +594,7 @@ org.apache.maven.plugins maven-javadoc-plugin - 3.1.0 + 3.1.1 org.codehaus.mojo From 314369d3415c41c92def7a0958d07579eb1ef9ab Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Sun, 29 Mar 2020 12:01:43 +0800 Subject: [PATCH 102/176] Add __all__ exports to Python SDK (#587) --- sdk/python/feast/__init__.py | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/sdk/python/feast/__init__.py b/sdk/python/feast/__init__.py index e69de29bb2d..8342de4c9b0 100644 --- a/sdk/python/feast/__init__.py +++ b/sdk/python/feast/__init__.py @@ -0,0 +1,24 @@ +from pkg_resources import DistributionNotFound, get_distribution + +from .client import Client +from .entity import Entity +from .feature import Feature +from .feature_set import FeatureSet +from .source import KafkaSource, Source +from .value_type import ValueType + +try: + __version__ = get_distribution(__name__).version +except DistributionNotFound: + # package is not installed + pass + +__all__ = [ + "Client", + "Entity", + "Feature", + "FeatureSet", + "Source", + "KafkaSource", + "ValueType", +] From dad20ff058ae194e22e5fdcf48960d8a30154834 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Sun, 29 Mar 2020 12:07:43 +0800 Subject: [PATCH 103/176] Allow tests to run on non-master branches (#588) * Allow tests on all branches * Allow unit tests to run on all branches --- .github/workflows/code_standards.yaml | 8 ++------ .github/workflows/unit-tests.yml | 6 +----- 2 files changed, 3 insertions(+), 11 deletions(-) diff --git a/.github/workflows/code_standards.yaml b/.github/workflows/code_standards.yaml index 684f1efaec4..2077d751824 100644 --- a/.github/workflows/code_standards.yaml +++ b/.github/workflows/code_standards.yaml @@ -1,10 +1,6 @@ name: code standards -on: - push: - branches: [ master ] - pull_request: - branches: [ master ] +on: [push, pull_request] jobs: lint-java: @@ -33,4 +29,4 @@ jobs: - name: install dependencies run: make install-go-ci-dependencies - name: lint go - run: make lint-go \ No newline at end of file + run: make lint-go diff --git a/.github/workflows/unit-tests.yml b/.github/workflows/unit-tests.yml index 6fd5bd51cc0..b84cae395e0 100644 --- a/.github/workflows/unit-tests.yml +++ b/.github/workflows/unit-tests.yml @@ -1,10 +1,6 @@ name: unit tests -on: - push: - branches: [ master ] - pull_request: - branches: [ master ] +on: [push, pull_request] jobs: unit-test-java: From e8a12f418ed66393d5c98641e44266afc497fdf5 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Sun, 29 Mar 2020 17:35:10 +0800 Subject: [PATCH 104/176] Add stale bot to Feast project https://github.com/apps/stale --- .github/stale.yml | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 .github/stale.yml diff --git a/.github/stale.yml b/.github/stale.yml new file mode 100644 index 00000000000..4e8f2b1d889 --- /dev/null +++ b/.github/stale.yml @@ -0,0 +1,18 @@ +# Number of days of inactivity before an issue becomes stale +daysUntilStale: 60 +# Number of days of inactivity before a stale issue is closed +daysUntilClose: 7 +# Issues with these labels will never be considered stale +exemptLabels: + - pinned + - security + - keep-open +# Label to use when marking an issue as stale +staleLabel: wontfix +# Comment to post when marking an issue as stale. Set to `false` to disable +markComment: > + This issue has been automatically marked as stale because it has not had + recent activity. It will be closed if no further activity occurs. Thank you + for your contributions. +# Comment to post when closing a stale issue. Set to `false` to disable +closeComment: false From 698281f84d137a3643cb62abc5ba1b3a8b8cb4cb Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Tue, 31 Mar 2020 05:15:47 +0000 Subject: [PATCH 105/176] GitBook: [master] 13 pages and 3 assets modified --- docs/.gitbook/assets/image (1).png | Bin 0 -> 56975 bytes docs/contributing/development-guide.md | 76 ++++++++---------- .../assets/feast-docs-overview-diagram-2.svg | 1 + 3 files changed, 36 insertions(+), 41 deletions(-) create mode 100644 docs/.gitbook/assets/image (1).png create mode 100644 docs/docs/.gitbook/assets/feast-docs-overview-diagram-2.svg diff --git a/docs/.gitbook/assets/image (1).png b/docs/.gitbook/assets/image (1).png new file mode 100644 index 0000000000000000000000000000000000000000..090fdabf79221bb802f2d724874590b9e1e97ee4 GIT binary patch literal 56975 zcmeFZRa9Kvwk`@42n7TvJV6V0NPyt(PVmAlgb;!R3GNUG1a}DT?(QBeNN{&|cg`gL zT5F%ZZoBXI;kGuZ2U^Y1$LOQW*R8(4Q;tkQd*%xV2l@kBTwoLTRD2 z@~V(~269?)p)hLVOrJ9XXAw_?E5SowZjM-4n^>LcS(i6f$Q@L14)Kc$#Uw#{gW4pvHgg+>{`bYVG9b`` zo4n&W$^K3A-^5QbQ=#mOKT9S5XTg7nFX4i4eD&G}62st-|JUPr4Soknuu_w(!G`?*gS;z1evp%95A%O71&8eD2ZxxZ@%ohdzsVa20OU6ku`dGt`vMz9 zJiUh*;+(MmoBV51fc)tovWwXN=AIBpk4#8-3wNLq`M=4>g#zTSH+=d3@9X$`m=Z`T z^bF<0g71HmACLsdyV|ysg#R~RGyuLhBeGeE{5Sax3L)YDw`=ikw88%u5h%J>Zmu>K zrjux~xL9vh0Acn0lyTW%iB1zUrSW zoCv)pR?|LcJlp(Y&zHuY;}@?+yO-NJ|8o-yQ~=j>7a9qX!#u^2g+3;6*$amxHA0v1 zi~FuoC)ZrB)R0>~&V}-SWxfmS?3}yc{?6-g@M|#9QT(=$rL9dED}k-TcjFhMA$Nro z>?2;4+Tr2;CVD^p_31{K2%<=|^|_|0lkvZeikpg6OGfV znx`yXTw17{X%L6y--WUnu_A6Gdz@aErFz_`4AuXgnXu&aABp1ylo)wv=zSN`DWi4PUB)FgTs8g{O-L$rS6MerdV%;W01)R98J&cj!m3_b{8zbiOUHkEGR3Ej_69woPW^RbMQtN1L;BBy}+R z$9MWNvF<{BF-#0^%_pMh)$)h)H~pNXdYP6E>MdqDG#kDa?akIGUL4F*6lj!$&(>JU z_Hfv*2Vtbd5>2yZt>HDsJZY$$BA}ILD(~yjl_3e1GN61>9AEgi^BW|e&$F$5ZCLo5 z48*+|ZE|t&(xwS|#O89jD3nX$QCo1owa8N|$}D_b)eS}8If<6))w|;kUGNERNNKH( zCLImjb1(cnA_*M84PhbFs-Mr-ozH2H2hfOHu+WS2)`U(Yc@QB7>Csgy9?Rm!vCGl( z11bnsM6%IvlhiZ;@VyjFgD54z#MfUZT@g5}1Q`>niDUklvc#9zx-Vn^`qDRjs*uN0W>ckUT5PavxmS5=1$ep={4(?W!4yJhIuesb z+73e|b|r;+O)~}L9@F_rpI#gHghUJbJg+k~_}oQ-(0sLmx;@`eb7Oh@3vG(_$@X0# z14W~Fj*eO}Q$A}E%8V4OSXVxxvid+e7Dpt6UJGYZc39#a&Ulr@&~ z`8qE)2jVF_9lkm3Cp!=Xq@kwHq-hy;Qx89C5 zhCc(j*L^$sRdzaBIp1Kj)x!2kib=aVYNabkw)*ol(Ti|@9V+YV6O(v;muj6xM$FehcHdpg&wA37U$?l}3~j%~=P81>IV#eukicWoqAXA^ir{y- z$o=Vz988l}i7h3XrlR%GTT-=OwQYvNu;IFgA7)?gU4k<57v4k`%vt!jtoaw z&aS7*jJ3DnI4rNU^ZeBPK=nw%nSk1UI%T@dH-Ai=8?Ka99T4aygvusMWdO zJA3zT1xxQNZ)|^7`o3+ut~@c-?uRN{X4Ex?Do7B;tgB^^cWyFrkPgXU+@LP~vMO=T z&8*4C(v@156WBxnk97nBM2SUcgQ-v$u7y_c;mlv=g?fRy{KL&IOK;qpjJMBCy_Gzz z5;-i^I=^E_ywqQVpKElg;5!=@JsA>wlu9fn?eEUYu)u>Vq9b}gW0{+67Gg|0{1u*q zoxfYuG>+Os=6lp$C&Ua{n6b&PkuayfFjPXk6!AluW_17XzS?dbeVnO2DB;W3m5c8O z>0g|Vh>^pD46sw7FZQlCw{)Sezs!)9n-9H-jhX2 zfGt#9i$x^^7Q;#xx|xqE>sX-ntn#7dcRmYa)GT+ry@+X91Y~UecM|VO zJ1R@A{fA<7F7t`PlR4X=y#D4Z=kEov6#~eaajk=iBPW;N+~`X&5e);sI5)x*zx%d} z#2=Et3EayOHsH71S(Wy(t-wpdRy%v^JHjO%j}Oh;zJ7;+ReUNr3i5CCYk#K8C6}qb zHf5w-k_RzN+%dd)Q#0VYCQ^ukEACeU` zb7>&Hw1DN5JI*~LV$v+*@3JU^$FlbGRFE3F^C^CJDzD1~-e%q*d}w31@M{IN>DEZFcF`t1jLY?on7_G9|$p#C@_2ZAYq`8 z|M0dN&sU3_{jtwe`{SFRU+;rY&f3&>>)9^{VV*R|TU+V!3GCsisZhm*K#QC(xD!z$ zG{fpbGUYtiWt{5nerX`QZRG*z6tTq;eNRNbtJn|wR0cReOUw8{uhtl*vWZC~QrBT7 zp)f}xbABZbOYW^0j%hG5rMSU#zHUCsym%z}`t9kA?%7CowA()y>d8t2{CdFR;v?z#aCfV#u&&v;&=-LR{b?R)Sy8?0L-#o;# znf7@+-fxz&fE!gaB+#60_iF&?4_k2BBv!+)j4INuiE%pJkoHF>CeqtIL^@n(V&EZj z>-SeVNIy8_oAP+VvabQDWFK7;Kz<1(x??fHYqx5?mPmHmc(py27da5mCJC4$MgXHk zi3j2J0xN!Y2>kIQ3V?~fvSedF_kaw0D)8F>czK2ypS^)+lfX}UWBl#&Y_&W9<8VSl z^iYFIa{y()J8X=`_&ilAe59+LFDB578#CS~x<2(3B(^?5MP`asL@$x0y=#U~xxkj? zC&C0D?`+M+a>HGLLsjn{Gw``5amK9o$I9B-Em5iXtAVHh+$=#vH#2FQwaF|Vd~Adz z`$8SKTv7*Cx2D++l%nM=?X%t z^-%%$B1vgxeM!zg7P>8*4k0^7FOh3~JZ*$^s#M-)*&5Vm=vq}C1GM^pKqe{h0M(ql zb1a-fa%{0FqbkDV{ycUG7aKdHnE^tZ?|y%?@G|hv|$SG8)!QnFS7|K#bVehyBlq}N_MlDZ31tvo|-W5R6kve90+%0KgzQ4VU zO?YeGhsx3jMMT9@-E4URo8twsTCrCr{Ve_GyOUX#^Yt=91c#RkE{A!>fR!>7Pqg0| z!%V^L6& zUFzCI&I;;Dg1%!;0iI4wV&R=&sR3G=AZ{UXr-OvD$@$KBgrJA}D=zC)*ZcFy(Lzl& zv&mvzqR{7#eC(g6-aTBlJoW;7V$P5V#uEGRt-nzdrSlN5z=+`iiElHZ?EQ_UqyQ4# zAQM7ObK3CcfAh*K;1^-OWIU^hjQS9?lYrBb5l{kL7zDa!r-d*NC8>noaab)dcKBmV z`mcUAEb6$yH$K9>(^@Ri7u0NVZ-PFO^sR;p7K<5P9?T=|o`%s9b}jH1y4_yL*eU$v zQX+9$!&H-ejAYX0xCX>*%m+IpnpR1K#Qn<9o?w^M-s^}_y)bSdnP1!aVlNM)p|a`g zenNgpku$UIPa%`}iSu08n-rn9B-CCw*slv3J_aIvE%AE7wuk;^f3l@FeZoXw`}7 zll$zAzYx3Di=ACa)7jPWYF}SY=(N@H39>Mfl9~RJOrZXV{Rjxp=npIqhc2C*ff{FU z7{;0A1`}7ZTx?Ey7yX{ppErdQWo}!&Sch(NC?(W^*VWQmK~klv1kMXB?Qq|l6zgc( zQ2LcuR>9ZY=LT=twA&Ur+zrY-+TPAjv)XOFquQx_*=avzwa{26mm;A1_L*=%%VMGC zA(EN!nX=eB4iejo9oD2*Vnte2D{8YX9xYmmA0HX+#tSsm zbon6dCb)=g+8aYDB@+-50asWNu{R>-gwU<`0|9C$Md1Axq7Lr;3$w=VX*ZajA+bEN z{;f+IQ`A5_>r~3gFBh;P%nOL2Ua`O+ozTJpK)O{9sxOE?PM2rNs2u_NHGo!QM%(g+bJmNzCG@1#%Ol*5KMi6n8tE-n*5{!G(fWIf!hefO-u+?Bp^8oK~ebf z7&4JAj+&=ll%Qqx%q!rn^KTqU<^t=Qfp7|c(Hcg+PP5BV0Bz6;d+P*R%v3719IVFP zUM_m*YhvvmrU--_WMs1$+!XLZFqr_)iXGG|f9K!bT>9%Eu`&GeZ!;#j)TmQq?QHtV`cSPQ~jvFJbI z!}~+tBW&~kDDFbmjhy;2=h72G465n)@ro@UDqgSOy?Z2tv8$5I=al&_O@ywwq-=lf zAaPokz0R0b&;Jl*0C@tH`kdHjw^V|l#C~_8H_@s&&poUigWUPPtjxmd(A;5!`n+HE z-MhlX`kLF;+ox=%=QMKdqp3$lXxu#oJPq<%FKz?#b2{QVwbF&>%hUFhN|mS#SR~b$ z$V1cKAJf1$o(N;#abal?-7(td$Gd_E=_RtwuGgygVJIEC;x*Bn4z3O`hxv|X;sBlj zVv=w6(@iclSZ`pmh_T9G5o0f?*!w;pf0x??`JBpaM7!TEJ*)W`AZNTp4(_5! zaaCT=^IhtySNp+GmfKWK=;+w918=f~=~<^&{{FixWtjp2yiTb)L0jp-yDjEV-!u(K z0r)bY006~3w$6v^A40t=WysrQdPCzNLvN(7cUl6*-7lgq;4w6=aEURRe(~p`lBAcB zAYoj9Z{RN5Gf_w;pd3f2d^Si%O%QUpb_+~(I$g*E@itYMKV%n`jB-gizbmeIGl+>i zF5Q4XqaiT4AfoG}wxE}C)+^V`WcVc`cmfj!gZ_kOzrKXr*SCIx$2Zb_O$srAnBlYN z;eciMR7LOFL_RQHVZt4B$C0GD;kP zNEr%{NJhg?QAHr|CPzX+`h~&F?xoJq%7(AIodiUtD4;H(*FRpq5&4c<>(JJ2Y=_x- zE8unm){!jx3L!7In638ymDS8_tt58*1NsB^bm1ihU|?M&guxC<0rmrk6P89zxHgq( zt>W0b6+aHrt)qR#5y??wxL{zx`!f|2Cw8qtYkuWmXdk05$wYX&+e?#QVPreWSA1JK z(O`1#A$$R!ZV>$QH((`{N!{9Pa(NWSsjGnTL1R!-+;qWn1g)E}UG8+B8 zku=e*p67}snq36iV!y_j%fW%ZI&o5F!}a#&rT%(bLWoOsZ<* zXA&K_j?#dNHv6~!(~l|T*cPz%wUb}YS0h&TKMX1ftm zP;W3HXA~@f$3DN^$A1gR571*GzA2u>AYk&92q`TUV*lpvU(n%Puq)Im{GI^Sm&pB{ zXSAla)tOUm^2}qd*HDO3SUNL#?@H|m6}F(>lHP@|pFyvWAO2<%T1e)t&k%u*8X363 zem6TJc~(2pz^^k7>KMG*W{PQ!f1G+)P;O^Z2xT|6Q+n-{^)}Btw^jn*-c=$@1t%G) z+xrl2fQ513%Eh1R)h^euF#?KC#IcjGGcl|x{2O8l{vDSehnw!vwV2tA=ZLr)KYpVi zU={10f@2r=DPB^nT(!3oSzmw*oWP!+y;TvZfZ?%?q!(;>_v({6-9ZR-lI61%6m^Y4 z3_n@Q8<2S+b7{I<0=u+kw}G@TVfLGSq{5dl4WxcT^04;#sjJrF*|rI#GYQz>JAvCh zwsY2r9)XEG03oDgq`v0Lo_|L6tJHm1uWxG7d{LpJD(%hp zL{ssttgf}mKCL~&;9KHoE;VWuQEHYWTPNR;_XHz6^Fk5^m-(ELOQ0hAqRokgKSuyf zeN6HZ$2=_Kum++f`V*Q$Bjh@9k$2Phu+ev6uZfJt9uS$kjjWE@lK%TxWB?>3qq30qcfT&rd=kWKbm+P%zTFMkZ}!PSFb9+A zEC==*>4C0(eiXifRnTwj(*n6Jz=BXi0U+ZoZGb-@o4VbkAA=d^*L9CjijLX^01&ZG ze*#BAM+8Q5h<)k16~v&_GJ|_uF$n0f6>*FfOa={ihlkg#^m{Qm)8IrfTd~+PqngPs zyoB$72mTrWy9Mo1&2fCu!-vI}W)Cz|R!iW_8EAr@6=IHU!_MlZD()c}^Wwp$rh)kzgU-;T<% zAWtxTs?pTNkHNZ?%DT}})`lHJFvk~)9QLNlWbVk6bD4qIUER1xw3fM)y!r9r_NHj2 zeJ7`l{`l4p3f3J#ws1_$l`O+C?o zJ1_Z9I(4V0k+0~yHstB0V+LNS<+`!?DPOAmJgyIJrMH&#EijR}BDbL`l;bWpjrbs- z#WL3DWLt%Y9wW4dRLKJI)pt9O7D6Dzo}ze*II&8W3jy65Fj^CL5y%qz{b^>BJk)pD ze_Qn&Q%Q(S3bPiGjuP^=5-8vu`^IN*?1hr0f(t{y)(ABkXV}F>;2%D}s(9?d#k
    HucQ~yybg_<9o2&s}4B+LU|xAG9O}*j1>WG zk0SQV&L`vx5&jlClwIBj9oT>=VAYUc2xfQ@+7M}6i^j2{c!S@rO3;`vYC`}wqFZ4o zu>rLmVRuerVID{TV*p2?kC^An*IsWiQ@Hm9zMId~Se5w39y>Q-#L9!IU~ACw%e`re z(itx_^3)ygF}%_{Zu(rLv=59x`XNn*Bjm07M{IJ4+1s$yVVJ>)kAY~qQ*C^@Ae-rL z)pm3`reS;>gaP3rA_NX$I~g$?@lH*I*^zKlC5-l0o8hF#C+q$6Fbqcj(mO}(7wLk# zoWxC`K$fka>VmBPPb}%dp(58%d>~ zyeF-@a^#vn9J0g3D9kuP0r$l-V(u_e$=D;xVId{w#F?7%rbu#7P}2APS*h}5v7Yk8 zJ;|j%>LU@Ri#(_pbNDgkJ2>T3R}{ar%8cyARNiK564kq0pQEEA|YARaRQXMJ&xY(fr7r$d-7QULH^2HmZ< z>YQSJY&u3oC8J&+nC@53zkmaZCNaa3!&i&-8KU9WasrNG zV}36?wN~oi%EmX=?2-JvMIWI3fqTQN>F62A&4oRdCP5=s)E_;M&!+q6#(miBk3*S$ zi~DGrC=#wJqNUUKV?P%lw;;a`l$%BmjIg!K!Q)ecS!pd-)k&Sr58gVBMQk}Fm;Y#IX@sbknw;FE>?ijQZn#aZ7_8w z{y>3RL3SZr@J@8(Va|od2*Ugy)M^pXF*2ZiSD4;ia|m`1!unY$Z#aoBLZ0frMA%CN zB0V~eW4s7=AJEb;d#VCx^A>GtKQG#g5lj5$^j(p>fs2X44hRW}+5VGL!1vW`G&B6E zoESpHt*R%G`5p#T4eOAOt2hB|<3pl+X4pda23)*ZZ%4;GYl=HQ{E7V^6O##|lo z@CS0*fNhsSo}MiYpk{|?nZf&TBSjy@^nRnb+=_DP4d~{Pt(ffWrBVOefyZe^&Mcy^ z9OU=Eql71z3FuZ^bQnqH@5ZU$@wqJueI_+WxjbAf`PK~7;;O_?uF3ATq!;lMWWT*I zdxnYHvnLLc)#!4DXa6)7U^E;w7ir6|*6aIn05>Mu=ruQ-6BYdGC_U;0zPg^o(z`Fuvr<52z599B}Oak#rP2yC0dbVxruGKVGqJaL>J zH~Ld_VBO4%-AJKwWJcXOQC(Swk@pm5Gk!qYH2%ke`LNSdn~y9cwLh0Cd&CN9IhHxY zU3PZ(xVuQAZ@nPAd}Y&tQd|AoFcQJOQAZ{AH;h2PuaUc~C=N=s!R1#NUfbBt(lrXf zcR3we4e#uPosx-X#6ma^=TkO zG88Yf(xB`)`~a15Lh4wt-e{KaKEHK@(J+U5Z?kc!M-* zS}66`R&6$uyD0IKJry*Q?#@`ZF;v_6i>p$K zYO+9Wg52zn&0u=n2PR7^%Y{ZdK&>L5Lg!|;+xLeH{N>K)1ksD-q6Ivx>#HOY{De#^ zU-G)#a+pvZIySrc+Z+IScNZ0UB>p3^#i_D!8lnujaYX^uKLZpX;en#(x9mv1+t_?f z@s$w02JK2la@=8c=}(2ECY_A|eF}|Yy&C)1KmuPTnw|=mUiBUM8z7JDtpS6@tqrB< z1KBciKz?h+H$!fb=W4%*hLG^n|GPN@-#b9)j_lR-LX!&>HFft_V+5#kzP#@C^=Fkl zSaIn-mje-Yt7(hODw%gx7lX&EjasZwz=g*p1KPix^UIDHNPx~qA*lKTT@GCHGcw>e zzp8wXX~=nQstcz}HSn0`>7VQgKazrsSZDAR?jdm-K^33ViPT8CIF&}Rjts2`BY}Qe zTG|f3$m*Y&m#FCEEPQ-Azf%ND^t#=MI&Po3Sf0|-9Bs&u@TWeO_=s!7YdTU3ElrZb zb58^$Hf|@QfghXo=hq@1+*<;ZpGnZp(ZCB*0u4z1ksGVbDv%pT5a{j#VC5a0PBRu= zuvW0@H4Yt0QX*K4civ?wib%jl9i5b)#y>O}7p5JW^dgqcl!bTlSCiKEXx39LLpED3 zer%%FiVqK{j_)kRElg_P8h`u~xeny_u`n^o!o$PARA2NI>ow6&mgvU<7$eVlinxkm)w|x@x0(+O$IfB? zbsUMmcIQ`M^U(g5n>1t}Xs&Yf&`&~bp$7_#_qM5)2lm$1VZIv!fBUIEh74fgB~UH6 zACwl#?BUWwWP(x>uQRv<+PaG*QJOJsE>%jg zRpK1K{vi}bb4sJ zG3ztrG@1!b3GN0!8SGZBuEqo6O(BrL!q%*vNANp=D&qy|obdC;vGyiTBXdBz1t?Nm z7U{rl9Ek1?#wa2!iLA2rao)q7f4(Dir{G&e=;QThLF_aPg*|!cbYvM~SqeqQc5UZw zFyQZ70ZA6LjhRbc@lV#rvx<1NSQHWM@n~-TTwORo_<2|v*)lpZ%p2H#4z#k<&e;U= zPxB?Ekf--+_}3qJe)vn*Mhg?I7><7A9=m5$ri~4}1E&S}%Zhn*I#c4EVhv-w0HiD_ zBkJr$j1XWC20zujt3(Qd7T)7{)1~#BOSjG;oL%xFrqyls$~9#`+5db-(9@>{-&_Aa zPW|pb5bbdoL(wxd>pj9n5QrBT&43g0SyKdI7puagSE_6a91|P5jr1I7`S7_M-am5q z?izp9pz4v>^Si4~r9g7|LRa^@a544|A8P1=PCXIb>?1#cRUr`bL^sw|^zi3h z(E{l4C7*M}B@l6D4!TSO%^H<7D>;Y9uj?%)!>1;HXUNc6F4|L$j4Y*9+FX-IFqunK zjmLD!tfR7(il?v5*(wXZeg~=d&*9V-P6)`LX;-TU-^~ldwMSSnmLjwlC6W(IQCVNc z7R}HN&NLBX2qm!RjfWz1=$W(N_TdFOEKEOZxJ(N3zP3e42UlDr|K<1Dz0Q>^2EYlE z?80e96cO{^1s1B1z*6!R_=}Tqsq>}4NS-dgqI91mn7PrOa~JFw2CHjW=92A0-U2w% z9#wIsih9wm_{hXvufObVxZ{soYq4tOYi3NOgfeHW1u4F^$Q&zB(@M{<>d#}tH@R$% zWOeK$;=WpZb?yGi16PfQJ1S6pzo588tw_vdZ>K7VqDFa=~WIE!g5>O-P0<^{v~%>~1;x=uQG z2gl8oY?h7y-^kiBwTZ6Vvh8_866z4X+?vGeU^1C(40Lt|t@W?|45Am?-PreE?;iXp z9m!Lr%BUfAPy6xXM77JYvhSV4-cPNTWH!kUv&T^fsf$Nn0fVXk^9ukeRpt-dyTEx$ zS+(zfg>fttSpp`IxFJ+J?T(vD12*BfIl}Rf(m;SoRr))ME_%_h$r1kqs8o6a4ht!@ z9585JnaEoovm%6!Ue&gx2|@IlZFK80R=lorovj6ONQ8*Ki`S=94<^5Dloj%@Jbng7 zNIkAdf+_C#k-CG^)>+;d@x>B>tO@T=BJs0e=gm*Ok=F)Of2uz;Cs&Ak7`&AQ>MATF zBX7Lx0~B~IbQGnd>7!kb&#_(|O~|RWTu@j|7dN;axJW1QL}S^dC}>dz#aiJ8aNyKt zyI-0J(p?G(YrEYh$d%|GbB8gu&f)YVZ{j}cGI1CZ8ctdWYRV@c8_oG>pV%v`!a+()^h7tT( zt+(Wm2g5b1@2;EI(EIa2D|a_NAM!rJHm^2!bVu)c->{xVSF?KDFnrp-CiSq4P4>x_ z$qqX^>CG)hu1*>IF5X|=w@7%|yVR^G)s&F+o)X^`UU3{IZqq+U=U7 zjjuG!ljR8qc=H@egDAtuL3ZNum;!n~Q)!6UCL8-9m|g~G zTzfyaCVmsvu|Kl9Jo@lmD?-o;-M(2X%g8&ntd$knVbVdbVX+6j=u`i=`D8Th-gr@b z6l3zFBO76~x!r_}13_Fp_AsE>Cf(nyWC zke92>I@>G_zj}4i#^-c77_!3)m&`oQfM200z04elkBWR12xw`^uhVA0W0H_9(P^j> z=RVu5(&E>er*JdPXr@Zl| zIbMz-efX_Ql_j%Q@m<)h5eUgYT>|aqG7xe!xx+U7-o!(9ntT1vWOcS(`o_5>>oEw8 z!a!w$Fv!C1Pcs5h@ zr#2bGvZuKI>rP|B%#P^-W`!x`5|P(4|Dn!aX>c3I{VeXuYJ2cgpHG!LVLAj+-O{wL zn+eHeXHU?iW8;|(&@?3c8qXAEjz0)zfXZ5tnEPw<3+KA57p~iHW(oN2KKi6H1$1)& zcW{P}gpu-wxn&Z7izDxm6u*=8ng0BskkVAR&>@pRp&ii3vqyMs zcS26(9DYN=o^uOY>8yLR7$$@TSb<2m&Og8dBv&O;5HQAfPu%)bW&+K?DcN<2M`3-l zMg$AylT(1Mkp;SCjY?uHnUWgQbJTG(JE>z`0 zr{e`>)#MX}Grv?Xn$IY7i9@0^bNI@yUpN&czYwxu)?a4o+^7D8!k(iiQE^&H$fj%> zQ#iKsa2d`Vp4|NjHZdrf+KPV(uPod;s*yHuPkE82lB4ePM@isZDrZoy*t=oirm+Ug z@rFx>C5cID^QfIoK1#~@ZW}}OcPE;xglrjk9pVa(`Kw^_t6}ER+Ef_3?xm9$sM!pxGAS)aPn%U#bEOpvP-hn8~%$aB&kEwm>h^h)GaSODmp z={WEQkO6-Axcx{tWEc?NfZOQtoxiDz$UIeVQjLHkKBE4`>2Vt#Y9~ zx~yc;tI5-msy~=yBotmSP>-Tsn8rdf)hT{q>_HNtY?qCn#Yc!3mD`QMtjk`b;KQb) zWyx~0ZK^wmh(dL+C zc>G|xLQ*l$L;)7|nyeq5vJ9glP%YDb|ALtr|2XXm_mX}7uSZSy=Rc#rgiu{Pw-A`A z40L@M$D(?-tYo0Cw|5ivxI-#Rh;si+YRnw+ltco^?T#KsNelIGjY$08Y z^lSiZcYP4$Q{fB@1C@f%9rOI0;X$2g?q-w9OyuDmK@+r_g7lC{83g7Bngt%g{T1Zo z(knyRk$pbxi5Gkzx5c>PQAucN|Mx%mYkPGJJ66w~z?l3h_~59VS;1UT!$+@jqe07P zd*=KH zd;0DBI|}6IGt&81e2O4CvL&uY6&bCyD#|53_-OG9U}k2s#6TYFhyqmlQTXNWEB9yv z-9+YJY?J$W-$q+DGX$_-^K5xdKRi?t=2q8|aN{}5lct{vU(sR0+Ui>I5_oGUA-M=p z&oF}pn6flIF52mcKfNq4X0%FIg<(#JKrA)~OjcyVmgiWFq>$EG11V1d@vhYM$;|*v z1|?w)5mv3KJ6m38tBK8S$YT^P205B~bMg>J^i5TNZap~Nd+j2m&q}uX*Ucc3>iZ!8 z!{wW&IF&csSza16?p(Ef>*10h1rGJ4{JU{!qjBS`p`f>mYZhMYlUz#lf zCP1~h625EW*J#)x7_v?IJL(2yhQGg85Plhpx=1Al1fPZ+e5+1C5;xg&fHy7-PHd=_ zF6gtms6JTqzI91f7<|cGD@_IsEi*uaGUir-2=c7|uJiMsSUG96ON*WAlx;~HSfY7- zsNpsa4q@a7!{g}t>oJTM3wXXfuLq7WHiJfpRZEI|al>}u-0s@e_JP*kyM%wDvt6^Z zPn%|BXY`f}10evkAci!f$$^lOI}CkhU0wvZ8yv+z#jIbhVfxtP0DkizHK287KTkq# zXt^x@L&uZH?KSOTPysqOUa!BH_VxoCFcC9;v(69`K8goYm2{KR z4j3weuN}1@B6}fM*3DX^xjmk=e%8qQ6M5kRolH+@^DxguFu;2h5l(nC&kwZVX9f}S zd}1gf*ox>48@wQh()^99A*#T$<+JSu-T57vJkpnZKnJBIkg_mC#qzc+*Q758+wjRvs9%asWwjsn)=MLVaA*NjBqj=-CU6>K!5=AF|0+ zrVfYgnN_5{$nx3Cbk2kcxPt>MUx@c=4x{>r$~l2BLf|XT8%>lpxQ9ENEr59oG=D?U zrlAG>Ln+~{5;oIdcNK`1F$p&c3qz)+XCZL9;g#4MrMNycs@@~#;2YBeIA}>Kp!`L@B`xx zQ26L}!`(m(1SLMQ;Tbz(bJ?o{X+B7bmgmLcw70sjV*7{Lf4#Vb2y9r-b3psA3<34i z`Y?2rbcnENY4JPBE4$14${_kqn!YpSrkBEF^I_9}Yc#-jub$#wjMUSZzpP>gNhsy| z4AvG{%%gXvq8mL6qGXJe@@V_d*y++E;J5ER(78bXzu{=$lpNphikZ;)(vC4F9kix* zL1)Bql*D+t?qN?{{QDF>L0W>-UrGQq1TJ`lL@(GMa}=b;=##CC&T_Q9_mw z2*w%AWu@O(Z!t!93UEs;eMR9^f)Rz^h5 z#fDT^M;u-}PBSjIF^!~8>%wAy1DWK+JOfk!EC#xDAP#@)aYs=Q`#GX%7NhP5bms=G z0trU;?_L1z#>+T%mOr{X97XivdDD3RqRRl~)rrj$(}3qS18{{EzhB&nKS_TtG%QCbxAAzl1dL!ibL-FF@MzVGqIaY0fo>X_zxxMval{6^;Lc&V$NY2~T(YU^ zt9>VqoBjwRlDc$R3^>swCH8jfSQGyU%F_!fPcQI%+50ONT%ra`4f9lrUaw8_^Le(} zvLi2>o#jUnvcB99sQ;DGCq5nN1MfWWI_z;j5uucZigur|+Kj?U_gSc>cv#3`^lH#* zMC_M?{w-;qZvkxi*OLAf&oCiBf~+jrLv`fj9^~3+@-DHUkYKE$p2+_&c7hz}Xeqj0 z`}m}^tz=%wB@PiQCMdjo{yuf5{;XYM-gb`O|JG<(;1cR>`@{b>b_T;uAcQ=e;36v! zzUR$$P|i_c%E&=C3PeWp)&AEtgQ39o;n}B&=>Lw<(dw|aG@f<;IfmL1PAT-MF}*tE z!T`?o&jWkX(DThWiod5P^ip7{ilVpgpI#6J3kc@=?DISPV^@GrP>_)8Vp=PKprkA) z8*05gXTt4#l-bdosSq8A+)f;^+_nZ0VpkF)w$L~=vFFwiUuNz5k12!10jBJk?E?3w zNK9CbYtRppE58msM8vG{aheStzRf`&rAez_KUwgk(u|zem#fd4oXi*c2N6VUO=Dvh zwhC>~$m3r8<7R=a3oShrOy}x0e@VAefy^JKDD{JrXzUZ!v!HfD#R|z6=phM@!#ScY z93C`i{5o!TF32bscOJSAJSopr-OCyd23%XVXm|l{@icM0bp5DSBo}H-0gnNB$$N{I zH(98a&ZHk25V22q(YNf@L#fgHE>FGW=Nh;l?=Ybsn3}&pb6Gn(KBcdJKQfA#rCh`s zK`pO?-)0_5@&4M|9?Sh8;uD42H#_CJudnFzi=v!8csHsU@7(=nTAb-bstfNm_BUC6 zYG*OLv%13gw-i0Wk;SL#P6M5>|Fd|w2>(a+RgC!8Mlo}ZU;4k1%T~*M(uEauM_crK zkiSYP-;Kp7 z8K8r{3?gi7VY>(eX)idh$3Y!(ZY$*)(l(SAXj zkz3ktc3gW%LZyRmaL4OT0b5&JV>-1X?|!JA_;PP<7uo5=4^IXmTQ)^~M^IU}SaKn3 z4pBs0u+^kDSyMbuEK{afu)=T+{ePHqMFp<~zPFi~GGoAwV|NZ2h_7tr=-fH)>$mmc zk&?6fAxCmr#G_Yb$$fp7gZb%kCR)lVy)VdS^+x{syg^W{6!P=qRC>%^xk;e!`k?dMiD>m zsi^nl5H^tqlN2z`BUMH!;|9*mZ2n&y3G;jlG&E}2>T%NxQXg3-B4rsIyH;c3FzCM9 zPvQQu@b&ZT4ZnA+Tb<2u0*|k~4{X!_++8}_@_x3rSpL>zHGe;Ur*Clr6&MSxiCM66 zklwLL{*R@7vV+D?wiB3=uYZdS?Mca?%>bzARYj8X^$tFE+oTg-bj>a0THQDDn zzy53X|7BqQXa27X8HY(%XfzuY5B3;B%tHl}jC(iTuv)r5mimT8>iyw0 z>#RAm9v<&`o;&xx_O-8ll%zR&hI&O+Mai=TH@H14CwKm;w(#;7HR<`rtu?N~H(Ib9 zj3=x^HR3NiraASxJ^y={L>R#^bwqX)-y%k`Sb8Ga1sO}HZFFn2Z#ua>quM_IU{Ewa zgvw-c@z&3Tb@$aqp2?7n+{ek|?gth6HC+qxtl}t@=k6$Nc_lP9=-ViZ7&$(@P zU|{`$luIc~4Mkg^Zf{BLvBQ2s^1V$ozrh8TC|NBhOQVl%|DG@41X$7EZ=b!%>-z$A zvH(5(>IGjK6fT}twvuEK=4gFiMBz#|dDsfFo1=TYRQ>C;-afZ=Ea5YClU5dPzKw5& zYxbh*dz+2~K|LwccW25*MlY~*Q~=}F@4wZ8TD-vwIo+NvTPCJeD)3Nty&Pj23&JGs zNu2NeaMNh*<#v}JUysQ^wZC6yrnHoDv87DaR;h6v?b_sn=#ZX?W^Z_NpKf|#VCj<3 zPVkP7Y2S@3xuMJ%ww7#9no2@8efQfZQl*x4J$l`#qX z=n~bpqw+e*_Ym1XRj$Ixs`sb=F)5K#OazjDUedEs_8NVr+R{9k}cFf z+$Sjh^3Y2*4Q%KI+mlbML_RRc#&qI}oWnr|Z;PL)T$f>KC`0u3cIn~vw+1#VaWS48H9CQN+GM5rC zsJcLU+{K&L^OJp(ukT24&OTDFP^K)1#$p5n2FhNX9-8dTf419OkU{1^;`?h?TZb>F zxGOC6HKaO^E`D`I`D=Tlf84S!Qo(3vpYlh(hJ)P~B+p2L1kqoPJ0N4?o%6d}5sLF5 zEqWMa?;x`;kcnQywYwKp4d#b{1Qk<`!ZI!Q8eZsKYqM0 zBt;!f#G>A*?|s1lk~52Kl1}(p4)`pWF_!K(|7C%UyPjG@Cr?0PrjG2A8^+ zZF9P=0!V;7X}UPlu5(?^QEU8i0Jf z^Mzqrvd*tI~-DR0v@L2rhNrV zwLnJT)!OI!T0p*XJz0z$TV8C~Zi)oAb+-7Tjy=k2eBOcMxQ$dk;WGz`WKgu-Avlh7 z)Ab|_A(*deYG;aT1J*nTI|_PcULNzwX3gBf$C4&|!NFXaNJ)^gcNe6mD;dcgQ~BYo zcfvcVJa$CC*S^Xu)NN2ll6i222ot@ajx>ZRZfkd|^vdMoA4IS`V{=`Mer61`Ki0CR z+w zsxT|~YZgAqoeqE=LpzYXa=;r3Ybrv(ns&dEKWQ{d0&^q^Og&`gJrnpFN$vQXH*Y4_ zZnfe}+qJc}E_^pjb+oKKJKBr{$-4!0_w<_@Cyg#m`x3I3^mxi$_f1J3n!ZCyDDXS3 z(SI*DQQbc|ntJ*1Ui0~0Naf2pymLI(o!-}dYIyB3g+ z163#0Fn?g$o9A}5)v%38HEAfp0lRLK>p)v!9jdbwh%E?lOa*KYl1ks*rq7W>RQ*;;C^< zyULQG^HGA@XEL?IM<9ne{ZtvDtp*a7SWD+y+es)K`JA_s4z^~rHBHvXi$3DA7=A0( z6*%N`Uru(0T0I^B6sY?ns5fsb{Tx=07x-hz$1r|4xDt+`625iq(KIpKIBo&+pYuP9~6H~nfwZ1gNdG)bIG9z6ymn|FM) zT117p*j~Lh>5qqmuuQw;%EjY%d^#$snZ?%#_&lb;F;U7*-VX2oWv^0eYe>p%9f$AF zdy|DBfL}Q5h^^}@cb(!j6av!PD`8qz!hB@lxD=sNDmD1*!WN-H@CNdc26I~4P$;0P zLj#9exDa|r2SpjagJf4-T^y&{%%m3M{y<@&zzn!EEkZ+OAQbSCS^T*j)_D2nz7g~I zVH+9*rj9s`B`1#V#su+C-mmW+U;C=fGluxJaP||HzNvLy7{-|RLG69csrL|Fx3^NiLa+$O*H-<~9%=*zahFnIk2BOJa$5|GKYQgCa>iJ>g==+( z=4Kfip65jS_gCwu>N7@L@GaOzdwc|;RZ<{U2`4N1eeWZ*LnWya+(#*a0tL#AR5o7d3|FZ%S0;iR4`<2T(+Hw^k^cDgDBGHmr-lle zIzN=w7q&4`l8UL|pS5ySoG(*_nqFtzg&#Op%7*azhgio-i@K* z=yiFr%xeS_YfIqJ@W;nD!@#DVfZH)wOIxL54U4wa$ExM)eX6&6DG{J_=GW*}Dm?tt zSaIpny|C&_!Y81uqF03a$(!+#O5?duKHY?t_ z8f~B-QAQQ5Zh)Su%dfB4M0o1WQVODlq!MX&C@Ik8Ta)iti6@(2lPmxxbwOvAv}XoZ z03+>S`5#9Yii3jL+7fRS8U;M=FpQQA`_0KmRbO#^wZK7`MQd-wN`;@e4pDxihX0MW zgBrgt6Gug%WswlA_%%AhFDQ*SjO%8n*)3V3VbvAC^VhHdSg~T^H+%yFkb@>rcRPz z{zsOhN_C9kInY%)j*@=D+8x?shOLl+ThoC*-oAU3%1>7GaI^G}(eHCbAia1cHjjQD z=kvg38r%8&!)nJbyKjeT>6Bx;j%G^&f&>1S@8 zq#Nnt8H3|h1rSiG7;pRG=F`LQ)I@gL+F>#D(AT~a%OHcGdj5gA6);_%+@SRp%us3d zGm~R^wb)H$ViK^3SDXP-m?I+?WXT%RkA7qTyFeuxgmlno4(7=)pQ*SXi@&2>Rxi{@ zr`4ckqJrj@wG3B*1DL`1!}6d8TqB&nEiUwNYT((u+Ct&WPf#>>uM>Bb{rt1VEmn|p zR{qx2pSLZkWpVrt4E-4D-76x@q_yK;Q+%)}&-5a0tdapas|+2<_ZL9M{NX{_K3H+Sh#TvjrBt_-AD3;x)_Y62OMni6pQER3?th?KxbFf$;^c51>0K{U zJw~6E2=sG5!07k~_NfiIG4Xda^xA*@$4L?a5{D~h1Yft1sX_n!-+u>C|L^`2&~m_= zH{Bm-#HPn6rgmw3!2<3E7NEyB)Y*h<6s$!b*#+ypWpkhlP!tUgR)+in4mdeEQ3hH` zZg`#VN_Bmp@uvX!`7>TXh{jTwgURKG4v>7v0**qt$BA8-({3QO*9R&gx6X92KtVTZ zGqO1#en)opviV|3&T;&YPqM>_vP5&R3*9;TbH0O=b-=^A7Mo*c2eUc}}rV6|gaZ{0)c^C1&j zk*X|wE<0)9q-fKXulUO=#D%539z<38PLJ&L-&nqB@JqJ1q`K3C5yi!$&8a<*57yq$ z2r>YN^OWfh{a{I477ymeq6lffk$Kuc6!MQYJgWt6c}oi-I+f zXrA;b;P8)^?IG@?2oZ)1OwBR@oxxuhJk!ARmFD!_*|8;kWEpi|w_XjHO>7KLqCVgX z|3+x$Z;&ejfkveH4T)_7HamI%TA>6;Rx(T~0Je#m-xBbm*IWR%9ba(H`|4Ac0;NVX zE1<7G3vL95w={UXU$#$H&Z$PpqpbXn7!5yL~D@N|A5eOQj&!a- zAvHAW9oM6Jm~~8;6jSb(H156id|LBq`HK35Sl1iSTaY+}!38^23qfBxpiMw;`T}(m zY2a%{eVj!K3W$q>iMCkQK@|y9Q@;4$pxFiD9J0W4MEi-e<)7-Y7mSZm{{V{hr487+ zY)HYqLXEP=^Tff2uY4JAel`X>EEr893*eJTMlBoxUV8+)eiQFui$_-k!!VE@`Rpcu z&R?rZR%8i#rYN`@fN$5}yd9(4;FQmg%zSCSy41$_%=>=U-GAV7#egsiP-IbFC*d>$ zIz-AMI4xC31A;q*VOro7RSBT@e@pPcQKZ{YJ-`1DsNcA*<>y985PE*{I4Lipaso0Y z={iOr71vZiui49!&vxF&=6&`i^e^yNP6A}fsmeA2G$ahkF90!@q#=xHgGJQ@@Nm@C9j2DCxC{DiYO5`a) ziU?_X7;GBiAzi7~oq2y;#$Z$wZ?B8PXAHE>Crjkoo*L6$R~OE{oh-WbmHe|=G6bTC z7+=&|1qiOVhZ-*o%lxB(+}!5=fXK7Q)J7kHMol&MdjVKcsd4KmIR%BaKz5NY5PEsK z{0brhZ)qQELMB=7JVyYh!zNeh7`S=rc=8}=@#)lVNz>_*BEya0?2yekJ@#M&fur^N zHqk1z1k6|7Sug^2Tn>oGEP5%1bDOONjRUGUHvmEr5L37(tbts` zmKg$d_;U*&?;(GC?N=$((DOU6Y}0X&Mlco)w_!Tw0pwT#>9h%ma~ibVU?{a5&7%Pl zP%$96otZ?nR%z{l^Og?MHbXXGyBcsP-{9Jc#syMHPAezh1O%Yp4*)R6t5EW}l{9X2 zj|$+${kofbZ205hTTvZ!=TYF0cLiV&So`637>-$w*P%|Hb)TcgeNAzebI z;T-vog+m%6e}m9^5C_sn+ta0H$gO!EAEg#O1Ac1qSYl#md2??4*Eg~Ongm8YjgcdO zbKEsV?;)ASL@jE;bOP%P8i;L})lMdkH~kW*dYLfvz9XSQ?*TA@m7wv|1hKG&3EsP# z5Zn@=xx?H8_d7SBG%!NjN^}7t1M*)(DOtcmSIxkFk=0L_`7yr?@T*$i6R9g`aOKJ; z=8Q{mTaEW>$a{JjXmV>||5BSL|8tb8A~$rxkroByA&Th6Fzi}sQ_R?BW1Ac9TvM;9+8n^Mp(TB`bRSfz9qJb7b zxJp?+9_^iu_h=VimeCvk3KaJuV@i$oZ7J6=C0QsYA#+-xH2RKVM* z+F2Sztn#+)lJ2#^!vf7wNWK@O{vCi_uBWTHV`Oa5z!RWGwRmwn7YhKE_SIWDSM&8j z238&`9H$5IyQ*@l61H|rytw|G;})3I9j6#Q35%RfhAM1ND}aS!Z*l4 zf1~ENwc&esbtvZjzTdA}FQw3>99`f2G1Z)*Kl#tkim4mz4e*ulI3NsRQTYP>q}Kpf8%$;qEpA^SAweR+&B~9UD9e2k?X4c54`y zL6~nxiGqIS`}eM#A%Z@mxd8p&2S~s0+ed9Ei5q}tMiV5@S~xp}fK&&|q8PyAO7u^q z-Ys}hubp5-U5WfeSpYUpfRER#&ogh@F<`k{Lj|vz3pUmNf}*AGqSQJ(z71axxzB-I z^A?ef!lcmo2rN-vmmPx>pwc8*9)n+;X?qxuSnOE`GcFgVcP6dGfymc2IgKeU}R3;oRn5Cl7zJQa`Lb@W}B#wRG$rBm{YOcsq# zB2CCk^cEAsb=0bW8Mi%;*UJLlhU}`Kdu@T8}2?M^5jbP%GdQa4{r$hZatY)xXnnbwahao_yUm!j=iL`;=W*VP8;1 z4ItTZeI}#Y(=<@LY{K=&;R?_f6&hl{fy`dPN~W~Tk3K&H7tmZ&^^ALD6{B?)mvlB_ z!o@f0f{no|<-?ZQ|H`Q747&2bEfwwfdbu7zOJCgXYSD!F-|#D+H;E~CKYl^g8NA#(ub>K=>_UqWu|EHBdSe-E=AD&ccA0C(O4t}H6SGV z&wlXFdmyDR{R$rv{&^}7MT6fvKbp1!(A;g8`zcf3z1?3W*l4_a{WCr!z!7r8<9-Py zk%1?KPV!ypB^=c}AZg%6f;CPwas@)702j(fK`>AroRfNxf#%aJck%;UtB|E)HD~jo zvhcv|h1n(5+}-{sx4aV-k5x>O8FAo0Uvc@ zAQmGuBsfJ6I6R~5gUVz2o<_y86X4zBd=bfln@@Qli&wZ0In1cf1Hn3j;R!FF4OE^~ z=2pT^2}NFBd%aCGFarB29-bxLe_mfsqD7#ICx`5Tz_p|LuO7YuM(Uj6U&D$#NPUoj z|H0tp(tjHHyBqyJr2fyb!$(&Ft)TGlQv5GckP>X~;h%B*uTPXGfJ}%K)BRtw0Q0fe z&A)%>dYeOHZwN3JH&?#cfBV;iBij@09Qbe7`T9E`SYfu_NOr+V{JVi`XaKYdrf2B$h5=93eSm4M{Nkr^NPB! z|Ap`Z<3Qmc2Tf>D1VRBcfq=994lLJ!!TqoFfTL0{&Po!8%JV&s50^0!<`nqXPyx4D z)WidtQu7b5= z@8Y@4#y|_@4Ejl$rAH4FU%komkCP)8VDH}rT>5OQX6{#^yVw{S@i(r73k4e}$T7}S zKQ-K_C@@Fmy@WJ4+@G15VVy~Lc#AI0{Dj$%C7;$W0DcpW?ROw0^VTQ8!P(h)Uub*2 zZo1*q#_SCF)xMyIyHS4cP#B^`2yNxTTFStC6}tIh2+?{JrsK)IaaPi`N_z1Y z0w}m~hp_Gz@qF>KEXmDnat%s|8)@brdIIz9;U#u4yP z$&#{P(Oqf~?feY#v4d%Zl|AO>O@y}`h0pM8%}~|?1w@3pLvKLPU1Dyaq?-vTZrwzQ z#TzV6n`Ge1lgLe%j^p)G3j7}CBZUQ?;B?o{9@(%qvLDBOWt5`V#`;wIL8}IJxtK|h z+(T|0wrQtu9S4I3YpX$nRwC3-0)4cx)QYwF2g4;nOjqO0tg{sqGYkQPjE2DtzP$&x z!eRLkXh3*(Ups7v4P_HwE6*Y222XOci-Jr-D|?HV6CB7`kK(>`o1A6=chA>@yNl1P znMpzNmJ;8GaPcaW05lY+cpkL}CJb={1)8DdJI+Z!l`SseCw&_?Z+OQHON zlIYzRx3{QJyutH(!>y5njxJu{qnM@lK7_`{=NV?!lT;l$qTYLose(khW}-KINx%b= z5Es&1-`ND3TS&B*fHtb6dGz7yxz0^5%x^Vn!*9@h0)vx|u`AQxX*51#$(i&p0O z^>vmoY7v}Yn)J?(z=X0-K|gHcD_(eueACyUn1FJ6|06u)s9X;X@`D8V%)WsLynn7B z|K^(qF2BQcEJn^Ayj%m1fY@T~<6OnKdT8YSqz zF2nDE%bAapt*-n3pSIbM%MXIz`DnK_KoBt?3Y;KwIf;f3%_@qtg_8cJy+~ z;dfA+@ieNf zh7*zx@xWr3eieW)e49XyFM>`A=VT#LaRG!vF=f@WB>s%C9GJ_7KWRB3voxh3?ncT7 z%9!kRvs9CN?acck!@mI7U<8rt4}c)@J@mDcMDxz+>FJaGr9Lrt%vG2zhA)6UrIYy` z7eFLS<&&FZSNy}=FJaXBPTRAr?I9#QsjIucIuZw*#ff>{9QgOdvzT13Cn=N=4!>>Q zN`+iR`c5EFy=joH%Z){Jj~+mJ(EvN0`JrLX%>^$6K(cwn%>+iRN^{>WGbTtsq08@` z^pok?Mpsp!aKQv>bvV~1jT2nxFv%yV0lLMO?S=`D(U%}mlGWJuUxH9t>G|{3NV^{g zGIurB+QTyOq8@#R&#QnJXFKF&z6~j6O`CZHR3Yg>M`9}}+1&kJ_B;?7VJSBoV0&7i z7A_u)AMbGb+U@fEWDoE?x&R!%urJ3Uj&;xj}x-o`bpAG1dN65yq$$&aDay*Ry zuAdC;cuOSvV!Dc|6Cq=qK zw{WTMBavTv#gu76L!cGC_j~twzJ7Dls$bDIMvioNt4FbH^gSXY&Z@-9n+Ya8F^?>+ zPL`=hXumgkx~nPzpulo1?q(iPU5{z;;`m-=6?+XL1G#f#ogNU|lLr2xSR*%v;IsLK zAE@O5ntdJd`@g4d0K8PqY1bhE3uZFPHq4WbUj!J=hJUy zUe6HK0|HAD%^YUEamx7g(grufhkT8Zm0YH(kaE*NTrtLF_h&p}4}?^CTvsna0fcYE zWqt^8Ou0nvK93~;C6gOhYu&nuO-2u7S7HG!9_K&?qvEwiaKD&lKGp1tWl#MQN~z87 zymkLQ*`o-sT+rfEsRjp{b7~-uJxiYXD|p71xgT#WQHqI(h>S#QdsCc%HPha)+jvWecv6J z`$^7{vv60Zm5;=>OlN>XgjVW!GsdoJtxlth83%%brgttP)$)8p6s$FC$Ygh}zGW4f zqV|&w5}Q>3|Hf++Q2`v2a~;qqI4r=0yQhlU9DsXb+|)9edIb>KZlFXir?oD73}{uT zl`bBD&{J+cM4PXa?#HLAEp(t;T+)20<#n=1a&})+3f>BJ+M2$LO~!@J-5hmS^$92d zBEMsX7U1UR#59It#HQECj6uN2<+>6^mtYFQ4>?#)rKD#9E?d)lVE+ih-~#>nGIR}> z{Zud0>DLQ@+J1LF|NV;KHi#&7FqV$i=dx}C2>H>(__ze|Hu1_MmrCOvRroNhGP_7^ zk!*_y_U6~4*AfpIyW1b;qNCGo5`2B#N(Z(@DxrRf7JszLv3_q7E}Pfdp!Sg%k0Qg4 z$X=e){^+=f%`TS35St@#-#CgG)j9H(&3gi`7C{G8qxrjmiC~I`+?;85N}wgf7-^i`f)2$e#v zYXDSIt%WhG3yl{SBK7G#4u+&MQ0L8Ags~u>F$7)DHI81)(7RW^h#<5WmI=mu%HWmZ zb7eEGzJ4)P(^=d{PA~sPPNh=E1ejR9tX4LZ)2GMxWKtO#tfzKjzo*ja(*(!D)4N1` z^^NR%o&18P8@--}ZcN!|J^1zIy%`SW_Zr!s4AS)Z!zLeOm-{={uKHp&T(M-*hZn7L z^%^{s$FJ(7sf0WX`um>d#cgWXvcwLgMi;G5be^a((W`Ips$}aC1e3BjeaNK{sa3qZ zaL=aSk)?et8ymZjuirRpyYyRGlXKfyCid>fO|KD?{(D`jpQJO7YZgnQ?Kbz&x-`@B z7FdYwc=q5}ZQz)?3x-Akp&$jU*GlaiC)WyM&tpYxxC8U~<;7VK*q_WPA9z9W_59LxxKx zp3R?Prg;jg*5;*8%eO}ovq?VQT}yKzCwr1J*3$Qb@$S>QoP*z{ zS^hmuHriRo|T%KIW}1KLtlJfbF@MEk+Di(Y`fjh zSleiS7GwV5Sbcy!%kemja_^`)UDP1<&C|uxG1(=p*#6W<+c@Kvr*E#Z%3qB=IJt~0 zw!7#y>H5H$o5XLYL?v9yPK|S+S-@jW1MUq^r*;lsk7e)jcRyaJNk^5V!k%iPBeLZA z;O@NLnY)?L8R|VwHgV>>9fYH&nql;aRM6GSXT1#7w-agc+%_rp^&!VxX^`gEcyp0~ z+Py-emrO3-5VYpQ!M|q&&_bZ~aVqgt8|zT_@w$-c**>cF#8!5_xs|?jh*zDdDEtAM z!{n_^OgA8(31k1Ls%mbPeDU+$fVKIaASjP;QoiAxWx$%~R9EdiG1$~tNN0d?@Z#wJ zRR1|~adcdnQG85=Gr3w!Q?hoC;?>#I_+C9g4fvN&y7{&byuM|ta zs_mTnQI?t)S)z$G*}K}?nz1hV&ED*~%+4?8B1*nq(Y)?a<1iU{H_on4PuH|Ulet_H zGyOW{%7u67qoT=%^98-xXIhnVpiW#eu|31Xpw}Ak;Ar4|#2|aH%yF|tb63r$*|No# z$w{2+T+n>b4Q#g>D3UN_#+Vu?V?)ao@9MtO3r@%vGcL< zety=;C#jr~HWLZEBetY7S9C)$YLZ-z^G0+L!1|-ZNa%sj--Yleq~pLVoVWD|U~74A zC&}{s+gC`;tcHdpiwp)PNxsc-Sq#U`04oN2gX0VqWakTFnZf;i`K+MC6VdO4=`+0ABgA1~Q7cHoC|B=IynD;p=ArNVoBs8{ z$RDb##$G$Awpi>*I6FCD;lm=Z-Ic<|J+E5#;};z(-r>+18s#Xvq52JhWr9L-SD zo#0Wcna{}WP=OlvL-f9BcWeGH|J&%tHgosE#ujziELVYM?)D;?*v9z#Mt+X$$BWmM z4F#7K#MnEP46kgrofmgArGEZex1z+{D{+#l(kE;D-UST+^-gHvm4Svr!^m!VpMXU( z4>n^4H4-Z}ZpvZ3TajhBHxFw=&sV*N)G;oGt((nRGM&;#V&1*MCZD7>qC|;>nU57j zud)@PR<4{CUy^Lzn*_1Zt{B!tQlZhS-MnvXfFHTCl6Q(jKTok~N5M|79lmC4 z|2)CHS)#+s#P4RgIFaye_IpL~W-M1-l7M6D|VR`fnrr5bjr0S<<< zi1!wiuM&ni4A@1&$l@5JWlFh0r8ldjfvirvzLGY&;~*&|{SqM1K45(_E*LN_ae!43 z%2=RZSkyi-@eEWD5klJB@oqn(pQR|?D9n7YcJ|u$6T3-rk3N)EY<}#S{-b@AH8V9S z%(1(Sg%;!vmwkYV(9$UIapYs|lV`n13P+g8B{Qp5C5%#e)bjgi%x`2YT+e;8sx$2u z8QXo>G;=0Ne^Qh-Pw~{OcSSQcWlKBH(!RPFrQFi;OW{~x?{!CqEX#k0YZ*}NPZ4A* zYu&jl^1P%e$~|W{uTpgW(R;U|&Z*1ed$r;F3z}d8_7vCSA;a96uVb<#z3Q!BP;!le zujX{>ow_6%A{5SEho0#dK5@{wn)H&1 zkSSNQF#g>;FZDKMugxHy(=_h(Z^t)L`RX0S-#qq3rM{Q#ZrjeRfb$t5G)MMm;?f|4 zeqc`2J99lyrR>Eww*-2bqsFPi(9?K&*C%Jzi~W6xI8%ue_j$I7oX%^Txi$x%dOZKi z)v>JbTc+u%vO#RJg_}y)1^!8UB6gJYL09GBS#M`ig2G&IVRTQ~*$ac^3S1O30HW~A z#(3b_K1Z=B`}l0T-MIeLyM=!xqGMQ4GKb;Z#*_>{TOCC;h!n32_aHiHgQD4X$pq zm>5bUPM%FyTHVW_^PFUNuidS+}Yr2I3>?riDs##xoDm3eGse7W>b z?FkJA+$2YVIW0F zEV@cz2!HhF)jowr^2iWklO!37Gw%fEZz(eo(+ZsTDYG_y2o2`wi=!Mlip%#NL;2Yr zLN9XYWx~6YGF2I^HJUW;HIIL@?s!i<2j!h~RFhg56MgD{`IaE?NH*bJyv@WAwOr*> z>$5k_Sr#3|)fe>4;VQb4@K+PM`h4MTN`@mu#WTBq*X&?M)M|(i%aI?O7@DC1(5`!jb=-eLG4n3}ioe_YWZf<8Nl$J-`O+(+SQEqk6FYxM_d&ndr@cge#Iv#yD4Wa*`g)t*3K= zqY}#;NnCaxiqH?WoTEgR~pW?7e<0_31%O^uAeFvNn&zzWH!^h@d-h>EWob$T*&D4uZ?fRACl90KOv^ zR>J4W9$=jEs`ha`?~){UQYsM*j~f5jzP`C*t4mA}$>8S~4n9d$xj6udYfhGEEVZ}K z^5UJ~DOe9Y^!T)RB|^JrVJSO3gzC=l>~ltjDkHRJ#odwr=Sm$x$KvcK0ek@koe?1} zv2l{3Ieyd89C`n-pl{?3_sloX9S43E1r(qHTr3k~?;@?e~V7Sd`9ziDr&=KTXZ$D9K5HNaB|rGH4{ zj~%wTH6d>bN?I!eAP?1;c#6 zg?InD`hWFiA8qt$Bw_O6SRYA!6m&aa?(X&gbiH^`Q7B3`6^jlQ2i}K+=;#_NbBO>C z$|NMeh33nFzA@AAtl;TK#$bGg2oO@10kH}Oy+#+)IRL}7eo*~FPYH;Nm6l^qKnWdu zPSOZ0$PXAmeo@0zui@0`CoLUCPh<`85b(#)6k*SMfC3Q(2vXfZ44592WjT<@#J7stU8@Y88Xqe!l!w;bb!U7r9wTg3pa{*2oI~CXyZ$5XI+tp+mq0&VFSoL6>>kf}2s`bd<0j z%6R?z_iyQ3g=BhAqiGmq!~|B3O9GB%7U&?;eZU5M$Qm=wdGT>#B+U`vh9EhaEHW2J zB3>L#*{u#UoMqX8a{J@Qf&j%Y6~H09>DU;C+Lf5c3vb zPNf`)*HJ?s(0grNQ7cq|WoKT5NBBd7vjpJ`t2=<;FR?cnNCmNzHm+d7k|K6=ni7 ze)j;B+XK+mML?~sWccK~ouURRmm&*F0Xwe&kk5vqR8hsSxg15XAn8P4sMX&(TqDvc za6lqbba^|C3Na)MB47%|NJ`+c0J8OjoUAS&BP@#Wt_tsh0|=@QIiI;5Z?Pgv7Hooq zqLD1XDSBYmPhkOQr8ZYVcyz{qBMRyQqD=|g@KJzTbai^5AhPutlUy7e)Ud#36|?OD zJmQ6wp-fP~bl4(Rm@>c=h&h{1089ocs{R0QEbT?=D*!p0(FbVXHIVXl@W(cz2hbxf z&}D*t{drz`Aa(usG&g@Hk6nFwxNfF}@zA&vPRZ}^0i?-&vqPebI-lIeDq)jOZ{ySv zoA~vB@^>}^KIGGnM3a%XiF*ib)e@LaCL;X!tKP+1N0E6jGKP> z+ZPr)HMTz{1S0Dn=XSjK< zVB&UuLmuO}GwgzWTsmvKF&GJqXGSUj;#Q1WPZNMWZ=e*gqOgFY0(1vTP}_BPP$4;B zVJ`2cp^zUWgKTM_fHDuAC&&#f83vA%b`C$$Yp2Zp=By{t-G;FxB_(auRq6h{4y(iM zxA`--A;6E|mwpFAd@My*&79A_9yGJGRNs6*`MwHBf1AR{Y=ou(I6PXSuP+SZ0ykE_ z`yw=uVL&1}q3c!MfDTr5Ph>_Cy11W@)Ane%6yW1p7n@f7wrF{phVYGL^Igi<6#;-I z>SM}BY7M!TAD@e}g8BOJb0H2bfryt!SHh!@fch<*QO-mSl{+?DyG8vcz@kV1MAvhL zl(%mA+M=(ZCxb%V5x`-N*EuKwZu%V#M2e6*acEMz1MUn+!z2xoYzSQ=#KWsw$;&MH zbd)e!q&*>LBFb?mynZ1YM9741{h2<@$mt{ebPK~51_fRxV^1u5EZ6|PtE`D1PDWQ4 zw!LA@8%E{#)(7R@<*jA`B`h1@=cZ{fGY!s($NO(?lhOjBcj+SDG>|HPD#tSfIJEJ_ z8IxqB;UPEu=REe`12eYY_@nL`NvtlK~U{`G$S&k5qw zg9#Mad*l|=BAcj1yrvxp#*Q}+bUymj=#;4RrKSNQ$UEe!7%5|-O#D-up)a0)L5w#kEHqQ!~4 z4_hZ`_sRPQ-hLIWou#RHttPwpUN z!C2EE4G<*IS>KGp*pMvNZ)OA4hp;zbXRsX?5D)ASF?s<5ix9o$hKPG$;)mNH9(fPV zL0oxrsbEIg(W@0iGJ^UQYXKw^50Z}JuWZ3NsH`bo{2md70S=dUDOkwU{skSF6H|is zFhN{V6az#)+Iso&wz1Y62>9hCt1QQQbppVy!j&0z(~<2AZ^r@ra;}Ks7B2FaQ$ZB+ z$Nn#CD9jec7gW0I0=t`T8a*Kj$^=20SMBV>ba626d{{6*eDjH-fBAn53*@(OW&o}R zf77`R8UGW;v}HhnJbvaz76k&&{}sjz5=ViwN`F~Wxt_>>e}JaJ5*!{$yZ`rRf7+&11|~UvSsfvAQU7Dn%s&CASn&ZDGuFQ@ z!-B!(@Swiz|NML8w`U*%Cq;+Z(TSMB!vNSu~WF*HFN!3;*B6O`z0}g z4iZkgqdt%mZkHTX-&uX=E~oywXW@O`eL9BJ70wk+>brw@9(Zd~Q2(3Zl>da#U(mz) z-giGy7E+v#pB+vZITqaRhty?auKtYs1m-j5Lf-PWrAA{#UDx+SDfi1hyFl~+WuH?# z?$Y6WvP6PLU|_fAh*`cQDQyC2Qgi#CF4|z-XSuIAqLk>xZustC0N{wzgVdHALUkWJ z^?YDY5pt!ou_5%CNxTt+$O(fSgg+m)NLECJaTm$o7q0(g`wEQx|A%h|?uv z?`xheD|ruSRqe>xn`GGKev7#K*P?PsN4$1;RoJlP-o)^E&H%82$nN9b`Zud#fCxla zr_FkjuTPGDmgr}&#&7ntiT?Mr0(>DfV2~Vtw=?_iwd{i^g#cF_+42sB??;k>o?yCL zHavvDcb7?~iKIV}HJT#PWbtNAVJZ9dB@&;t+hby1+cFCJ&@4DlIn3K z!}Gn#a8p*dtYf{TJN}nctHV|<-Pf-9yZdbpIKg;9w}Fdc%xsT+-2?yGfr7++pl;ZiQGL#$yRP$cY*zHmlP09Rhjoop4xfwU~puRD5{ z7BD_o6%)?H;piyT`uZ}_Y?(>IPTy$S!#x|aywYzE7kHqos?{Wo0vm=A=jv;W=@L6# zwpj?j=~>Ylg9KA+QN&%}?_R&>eZS9sTBMHWGWx)EFF|6qK>5j%+@0s4-m~d-jr*2p z+IsbS^Or&=X|6UF8eAcK6HaTCmbWh%Np&e&8S&q)a6 z@9nP|YuA;xIag+jfGpx(C#T=6Sn$Yv%XZr9Iby=e?nkc-=>x~~bJvCX```=V3)C1uL?p3<8ZYM363uzB7LsRo$*dm_s*9$op^3(hf`Frt00 zSf()Q9;>Tp@1+vDzKa9WhZ5z>Ec8Q}Qn3^#ShBy4w!02E=MpP%NK1VYaNs^7l)k-2 zZWr_6V6Co&heanohMW6-ZLi>?7)5Na&s}~Hlz8&(=%seEWcflrlK@Xq#pVbRHMM@7 z$7SK_EQtA11U~n*QOncSH|hu`|3(w&sXbA|Et)1C!g#8eQMD3lr`w+_$oTq6Y$T|E z)Rk(VBij*nrY*DJJ;D?;D^`A`#ap#OTE56t6kIJ%yLYgvx{vd#PPKzU*HjAi%%RY| z=EkA_>5rL5J`I4k_7bpj4!-903%OSz<~UX9m4}V@Rz9N+IiE!2DzIdK9JS*Zy0P&k z{UWC!x6Sv8yI8N$V2R`QM%;z4Cl4oA&+8E;i8+Cd*(}Mlt`Z%=mq%7LLv;k++^lcN z;A`%nIJZqCyH}inz&2p!n8K6N_J+#|V3V~PyG)3SG_T)h(xpg>lnby4(*Ib3QofB6 zvI%gdF z+7u+RK21V?hc$`^B)UBMob>#}gXO=UU2J@-8jbNGg8BiD9Mr#! z=6cTwBb=vv9Oh9tO$bRRFdr{V%<7AhxOup;zyUE^N^)J8FD{}n#b-VBzQQH0IZ0faipN=^P+5^e-_kiNieqPoizMiC|QHMHEMfc1?4j}UeINcDa* z(V6S5qHr0-z*k0YSFY-XBTD!T3irf>*No~zC#fft#T^IlSmh7ZMhkkKOPjvG`0QX+ zG*ejR*m-XA?TB*7g6S)+>uySV^!%E)u40kRB`quoKu)}rp?@H!w>NYpg%)M(sL>m< z5M_4J+17{kd+YPUDpg*pG7+@VtA3hI>OvBDVJ`_AK3huG)0juJeJX4ff~BFwS$Yc9nW*Kzr!rwSypjkC#2VRw?TUJb-} z8}}a(FdB@$D`Lw4Q7ZHLur>6IBiU*}v@GQYXOq^_l0N2wlaN^Mk!KX&@{8Umnw;;k z9(4iYiv~C=N(Cvzkrvp_4d1~}^S@Adx>;+p5Y((sr*MVsyINhf(laCo2tYw0-OH1p z6epgz8uPoslsv^>g2@$CatDk<)x3(zzz{#(ny#nG4W)=m-ay3VoZHb>@+ za%7G)!~3abt3=7Rm^dP+xPy=O+O0$XdLJJt^cCEkD^|}UBIIZxD(7+RZlf?^367jk zBsDEB^nAGX+^@G8N(j;L_ng;1-J6^R_iX1Fwqb9sohW1v{cWfngGKHzuq>H>J@Nj& z>}v~5f&V|;42M9+?DSUuh&&0&2|Wi!jFO zJ+XP`F`c9QA$DeeTXJvPH@v62?VPM~eH@%b zcJMR2eBh|-E4*F#Al17Fb(^Q?E^0${WN=$k4BeR%-={(EcjT;Bg&KRq6MVH+iE0Be z8r8+x7sIZ$w0{i2Bj8mXkrmR-(?HBZsiSohCvLg&@D%HYBZ(?AQ#IBWZ(}~*Z|{DJ zHy6igbN0C6!CsFZ2sVEnr1#9{db;)~92^dfqjgCE2m{Eee`m6GO%{lgMU=Q$0AnY# zH~Du(7LJT6(tDGt5`p1-l>Z!LZn_ zf~9ibm)3;k?Il(Z(&~ukSvAWaPt(*;z*3zg-u8gy9MDU`xgda16H!g=VJdsN@%`cI zg{06BHLoSlOFD4aZEj}=I=zf_>>=)s%vH2D8I1m4Tyj4@F61BgpT2-wh>^<(`I?k4Z zl^Yt1&64WuUvU*t#LE7Wq(6q$>RIQnz%XRwXLxTW1)u4K;MAiRy3LFgU*AV8CD$j$ zkZr2uvW9}KhJg=@F#Pq%SJ}qj7}>)gLv;-fg$fV7*&_CH6{UMM${3u}jfPi>oNZj! zt9XjF?vcl^#{}YT$!yuh(rWoh3DMHNYPkybP>j`8$at`~I$L<3mAZ72%dQ+&))Fb5hvq-*FJ;@pGp?~RZDv({PbeE9oe)`FvSn>|E3h4wp5o}XCng+5Z1d0j;J z`t}+A>hHiiRn}h_RV|E9&!sX|mX}7y=do)`{_chjzWxQ5SRlRJ|NskBS}wyFj2cyXQvxK{{aUqm4WCuvxDxa#OURf8ir>BX^9+9-<0XH z_QdIH_z)fDdL5d-;sxa))9%RW7>o51y63DTf6jk4>-#&gdH1pc-drT_)v!i-cheCA zh{eO0m~n`{`1^h2_c>e?(7r6$uuu=IP+Mo~4lRz)oo^~#9=AvEXj<9mhg7+njs^%( zi%&T@o_*szJHN!)zwFQ>9eQtWvYO`%p5r&+Xh5#(8X-iS_#REcysu-M$1{QfNkMA= z5d#w-m2fkqod?R~L&`h%8`!Cx+vOJHsT;gTFPReAHu!=rYHDDTJ&^tBliM_)`j=O{ z5fR?lsWFR*Nt!W%+)MeDfRGi}t6=5Xc4&V1; zeOLc??A=u{9o6p9BHBHIv&35pI&+SX=*{^IZcC8XQ@t8XVRP7mKlr|S=4)N1*J|0C zas{a`cOu_yVQMKuCQQSSvvbcDl4Ha?kMDVo9O+RihxNEt<1z8#b;YDY9DS(VrDyn> zfsK}+#fL-qkVM$lEFD+Gx~h^AfI4=l_a%KL{_0u@OcI|SYLHLI8Rj$A#bsz_$9P27(U3*~fc zX4U5{uGmz3)~2UNjvP4+zT}$SfMIQ4&@3qrZ}1>Pf_l%@S|@uIs$afBz6bA_^pqGr zJ)n)8<}UB)qPjy8MJ=B=WyDtz+Gi8jvlocyjaOxOGUD_pIyb@tlz%UzVLSkx?QrtL zMX-GFUoJ*GmcN!a?g*u|_dS2DB;D(SFjR7VNUztwHO`#Jh)FV2qt}j0O7W+4lh%9A zjwUqNESAZaDL!pJEK0NT&y$R7glo?f%|WhJhh43q5^lBowJKD%DMDKoM02D8BC`hd zPQ1gv;$wZOjV_RcZL!&+ij%Rx@&((h2(@E$-2=&J-gr^%s;#TL}3Xja@-dJ1Gd zSMI4TM+ww2QN14=-3AT@dK-lZlTHk6?}*7QIPLY)A@uV23d0PGgdmn0HrwZgJ<*ck zq(fT7V|=w^wO33!En!IP0mh)Vd2SMN4f(xEJd9R#<@;msvT>%#Q{w>*KCg>IdSUeR zSb3z$ofC+z;=DM&Z!ozsiPGP7B?m&`?_@9P0)@q}EJHdLx(>txI}(M$BWg#3WsfA7 za`gsSs9!fO1U18;2YGZKMu&_I@_6EL7jf7%#GMoN2$^LKV%eyNh4((R z@)gK#o7>AIeHOVR4vFXmY?1D9*rm}%FinHyIht0@F0KuGn?WhDERG<&mT19{dZ%K; zIJfaWjqfM6NH;U)zbP#?YV)f#Y(;-IiUkbR$^FMM;UBK3$IJEOOzx!N%9zNLXR0br zu%OGrj6ZZ6G?}Jd+I?wt+2dzN&AJwTDg2GX<>3o0_5M_d(I)DK(B#+I=_OD&Q^|Dk z4tfQV2jU13tY*YAh5N;6{2V-31;V~)RZP)+Qb^+Wnq^MI0tl;}<1TZe?Qi#x=P@IUgSkj6qMx2(*cc%~Np zo=cr(vmYv75y&lRNOO%H9(>{)R*kOO4tdDvUR0e$(7l;r$*|J*aQkUKQ=qEPlQ-IO zPD?-HQ%`kt96M-n?+8!g6*;k=HscRh!vg#k$>`dc_&NYInQT|qEea`jO!QY=qiM%OP4Jyut%gH zeAmfU`&!YP1`=E|s+;l*hD_Ot?CsZ2=6@VNP`M!B~DXOz-t&#bGdS0&Rb^J);^FYANpkrO4%V#B%9G#o*y7_ zK9bowS+W@z%hZ&y^HRGj-QU$}g>L2n>UsxS1PE4N zqL{gjRy*tIkQ}v0Y^JvFt`Nj- zJNQ}Ik)>`ehnJG>@t~XdS4_;7#h1|No9_tVY2?llNwdv6{+m>Ez&&mT>u*Z(&p$s0 zg(udxK%n-Yv=HP4B%o69|L0#$XU%>75|jy2+W<3nx%X4He;gl>FoBuSDjZ%(fMO2{ zP;TEvm^f?(k1X^PtV8ACD0sSVi+-Z01uW6`=kUMIHOj zNBUdxqgAQ}4p0ajw(PnYmNt2i!pts-D9fRJW2iX%_2?5Rt)1H!tfRBrjkVDrGopX$ zR7J6{0q|lCU(uMGQV=kI0DB^%*1mX~Jo?UT3l*ZyK9aA_4#V@#34Av8HeVew??cVu zgI*MLQMLIyY`N{9R)Ro8o@@?APJzTrkeE1NebAzWt%voj4giaN8?)&TuqD9D+6PXb z!o!sW{tE^|or2*_V82fXDQm2bB^_fvQC>Y*T|dk)u)IPyTO| z9b=HFJx}#+8!u5)&|-sDF(OQgI2cl8AtUruLWkS7e6HVve-7n{xmc;@7F2uDq;s&u z%HqIht@z3qz6)IoKYi3gKbxc$zawug)nrYxbmVxpG2MEb4Bs2lU z#f_gE4~Db1keWy!N6G~X4Y~NC^sAG)9GQ6Kxc($YD%cjyTLS6`7XiUKJ!ze3`VlJ% zxG$$J68@6CPIv|s1au8|AI0a9Hp$FAqglT6|Gu0|mYOq6dz;$z_u>jD8eEE%6O5TH zUnGLIBCutUj>?pY{-oAhdGkC38B(^{*-MYN&P#)6V0oy4J!@8tb1k{mC;f-w5$LH0`9v#`f)gSxXyGftDNd(ONw{=$ z-CXoWsCJUB({2(9OfFu1`EKXde7n@99eLLoJ8fTOlDxJw@C}r|rBk+IlXa~L3O^Y> zNzATph@|6?+n^Q!vZJ*B1G6_@GtggGEHP9$jXfO9 zJX5vkG~KSf9F2>j>8y@4%~!=irWuyFR=iI|6xQXNpReaZQ`T z{gYNXUdKC~f3~{*n5VaHKAE_|9Qd?~IM`e!C5VnE5&Qkcj{v0NICS=US=3x!bQO=!z7af5gMvr$atd>=;JkrD_5Y(?xWbbH=eqDq^dfY^+hT`AGR% z&2F0@6ZWMSN)y%e8=juO^>$_(2rM593bvvgONQ@x@s~&!*U5f(!+V>gZax!E;=^s` zPcjXYG{4~2o%5+_Rx6klZoPOL1&jdt1nD;QLBSd9W_um!IslkwAvBCpX1TuQfWRye z4)c1y^lOKd(ijPCxnC>XaRUn1M>>SV!yVx~Da1_Y2iMi7>2qJX&E-DfQ+-(b7t zV`+e%bS*LTA!ai$c;6QvXnC+rWdZH(UEWt%c+v~%g8cu!YNMuVs ztj59UYX2YcyITwrj^23d$d1{Z;-^3~Sh3B==Hg&5S#Mpq20Wppdb_^gq4`iA{`tbB zp{rgZd)%zwUPuOjFf{5ZJm%Oz(ULead)Ti@nMqhgMCsJasy<)VmUl@u*WLaEEExx)*Cp1MS#{QyN*j?qWFC1_yyz z@p!DWEMd2zfiE+X$SyYtO)SyqO{cSBGmitn+J|G6oC8O2lEnMosz=frfvteC+v(+7 z&oZW8k|E>AfdjftF?RZmAwUZVU=8s=UYm`NZFbUuML-pZJRsz3GSi}8NUMPI>z%=M zI;X)l#rdw`$1Tx8I8xnCL`w&e1k_qZBY+}rLZ{vu%w%)Fv)Xg*iuAg>MwW0ncEh=k z?{fq8<5*CW>Uz(f-|Jkp($AuI?nCR2O&AZvGSZ>#<;e#e*)0IKALBpkoB2@t`C3Oh zh4KA}-JEtBzYilajA!o%OF*%cQ7bTCksiqyWzk}jPT^<(^l0HQ(g}uD#dKCGy5n<~ z>a{aQOeLWL@)1-te)$C;X)D#)uqoXo^j|=GqD%=Y3%oiDiTZ|p>wfTPUc|Sh@Ee2> zytrlE)XAp-T76R|cMD?GB*Xb%)qNTR*pC_p;`naPiGLt$v;a^(zZVx2_^)^dNge8@4Bx59sJM~ISbZ)i zI_U^kbQx850Q zaMuf5$fPJ)Qd|Vzf!x0W&D}_PQxG-gp??PYxBWz1*{O{3PW-=7alm6R$)-L~VN&cy zp)P;>wLutH&7wu3@Tx5r?dEs2;~S;1|E5}8Al|k+D0uM#b%zR1C^G$Bp4qEmOYO&? z=QW@RA`eug^MWpzO}~!P@oB``n9xOu@fw%=lr52h+)GSaihShxiz$`>2vP5pqwrBN zC+@z>E7q|bnpo|Epr$>;OB(R89C8c3b8-V7;5mbjRJIt1#ox8`k7)cks5g>LF2qe3 z0Lpo26gYn`mIN-AqYR})YgVMFw|o1Q^gxhqXMz|#p!ha;FTt=wbQ44V+uEdApQ4`G z-d|n`eq?ydEf%bgVlr-SBul5h1gwu9!Kc_Cz-bV4PEkekJ;3<8cP z;=8(!QS9+wX#C-vE|mZWV97N63#lnO3d_Ta^1CCrEEl&NdXN{si)5#^y~@DEbgS}n zxnI*5D(HFKTQaOS5}nR-^7t``DK)F5ZdgS?kc=v7Q4mE7EJ`pZT>ZZoxJv!NK-H%= zG@#uYZ;1(ZiYvwihK8vZWJ&97bNo7`}MZ=qOVp7Kzs z?jviBTj*O&$l=!i;EeX;bG?moah}iar<=X$;~Q|b&c~}Bm~Mr)^nN!VeuYkFJ^@_;4M@Iq?-vUy%um_5k_eQj;2PqsBvJJqZU|@OA6Gh0A;S0XZ#gK02+Nx2!RN1 zq`U(vL)h#?a)bGWD2HBj5NAdnD1nzhy|0n`oCP*(!Dn{ou7BO<->Hk|U7D*o z;y@Fn%77v&#hU#9Sr`pN94rJ67_rhtw3ot>Ee0 zyhq4M{r|!PMg$w_YYS{Jo-XZW-~hX7Nni=W3VT2*51(sBv#levM_~|O7_e)Q%^(2NJysK_q;)xY?!6goD+b2fD@xO~wz)tC z=TBooj!K54KNfU!_1z5S1YnU`27RRmQ6``_Yk+nI7VgJeS{WojEf51drDcKe3{`+^ zF7>7H4@~>VDdE7TseoJ|o5ln=@cu6VjgB>Vp@9*#jD8m|#Cw3tE+x`r8At^LgkSms zhlurGfE}LjlB(^cO$QKGn1N-2*t?NQDD1rOCNf2L=8(RKM(-Z*;g((LO_V1CqJy_m0~o-)2CO&L_~sj6 z;y){bnK8q_Et%WiuF>-fn48u{ZB*j`XPHQs?Wsg>Tynu`(&1QObaP|C0Q^y0Eq@nl zl&4|M0O~%0&N-;WhF~}X_l$_O!Srj3P9Q9<=Ce-WdHZJF$qdLa@?_ULBdKVBZe~Rj zOo3&jAVyWrGG6sD+B1-9;cE@)-a#&4iYj1Z`Sj**-1xs(n)eoeMgw6=Fa{vPJG0Y_ zyJC2Id5PWT*So00vt}*mKpN33C?KtKrbG)$gMhb446-{?7zyu>fq?bSgtM5k zw|xbWM`X>4AI*x`b`Af$evt!VmD^wH>U(*89sNuk`-^N-#Kv0AR4#{f6!lefqV2Uv z)(Qfc<{ao}T@%o&6&Rw*T_YwX03tJjYta?;cXw3-8Kp!l|G0RYIC=?l3g`nB16D!e zSF;B+GStLeZ}3t%1Pf?@m}tejtu4%;(Wz@|?80f#pVuiPq3yA^{h*Ih;yog^gh4Uq zjHf{Ns0~JORMBtq?_8_C1;>-9esM;*umU9V{o2*B4&aX2-b}F40gd}oZ{je78uUN< zQo0#THg1;#oNb^J<-|xL`q$hriXvGcQR*wm0@Qt|lqcfY7AX>_wB9rUx|m*)ck0iz%Ro~vt>g=F ziXr52qtK6jGjUUp7FcL%RlxTS&B@~vu#!(06ad}4h!$;`^qVJo84~DU)6=Qse6(~O z0Dd+JXBf_Go;HIgXwE-WY`sUimVwO53L^~J-6yf*mrTAdc+&STs;3JTqzr(hie2D6 zjzTsH&0SPwut|QTcQ3m+$nhdk!@OIcv03bkQkeQ%B$@WDds608i{J9cUQbgTSnzd- zzlHUx;Sbo|_6LNn6VwI0Eq-YOFL$Gr>)~4iK7+2<(KNEp!Twe^Z#(9^fBw2yJ<(91 zDQH9N#GeUlf2>fvjkcFS1A>>@v3t7r$8mFviM_JrlK0&1ckm+yu|NrLLS>!%MlWZu@r3%lxGJ5X52>5rq@$CEYwH_!^;yz zqza>jX!GvsZ;zs=DSNpPjOFDnSWmmRrJKCWCV9rPzZRUd3TZe?_z3R zUs6%r3(s;*l?9&A9&@V^-T~=55_d%@`~cDpsS@ zaU3Y=H*gCq;80>=)I{cJGq3&JF=vmu+$9?N_}`_yWl(k{?0vOH;%Sq{xw@oK76~4$ zLm0vA>w*1CUu(++XH1)GKKX&Jr2EA zbgq+>1KxAA5`Vqtgf-Dl;RYbwD@OI|`3=R3Ma_G^JDGlvRdvtKbK7!sBeAN<_--e_ zQq6#jZ3whEh~L+%F!Tr7M9`A&#jeV#2%efwYPL>?8!euNcm)X_Nxee&kqYnO0-lTJePX`3O zlk|&37iIywXtpf;44>qbne(3U+Km`FEeL-@&dw;hL(6eOGH3qBov?C>GGAI`vh4UoQKV6)@& zfA8|z^WwA;Z0};=_@J0Vf88`}C0uBbvK{|5@J=#xYuitKaytJEh+(H=)`4!51OC=(Gdm6nPTyLmvEbG~&0LvB(%^*l z+o_z2<*bUAxWtgoO-si>ly@lpp~}&;aoLIMVi@a8Mc_W87P9XZ6GAjv5)S@i1UqvS z=w$KfsxwO@+sB2R-{Ell5~yG$|D}Tc=n^`1J_)kApI+`Qz%3k@iIIOzD>kS&x?6>__0s4Y+5x&7h?4gE`QQmVg=2_t zAa#uQINIp)L%A(?3%q%K24Y(A9^iKGn_P9rlaiK%pKtc%M9*|<$WmX7zRz`u>pO!| zi|F&Al0}9GLO2Hd29Gs_=BGTamg5bQ$R8nH5lz*Bi#Xo)vr>@^pMD?-W-_Cl3A>SKk);Oc6>(oujQB%9rx@y~+!5{SpJMYSBV&@b zoLF{)ELA|+T1i6(nievGu7Dh@)u`VtAUbOgY;cHHYQMr(rf2i^d*_? zNDkM4R*W18NENZwdR;%je!wQ^cY=j>z1@T0uLlXysiJUG!F%&Te{N)=qK*Gb#-Pc? z7SthzZHhlIpZCYrUZK}n67nI$n_7V{nJh9ym@Z|CxVqD^cW}TyyQJ?jqN=l^pijhR zQzoZVqYfrmd^6uZtyYzLu~njXg=HL8{nO(!?Z@_W^x1tcC>JFtmf3uTPBClq5{o(YeUrv{r}|P&d0O4Lnr&kV^LIn;s<<_ zji=-3rH!?C7NB%V0>!rJWlm%-C46)%CTdx{gO`SYQ(UdBtNviwucvRUC<^ohN>7ZD zZD9cM!BYoFqAl-M*Wv-AlBiN4yEP8{W0cOXN}^qm&dQhGoDN zkjo7%uwW_c;5SiHJ`}@ zv=6-{Y?YmkmJf4#hH|d!REph{8#rvDF4z}!h9UV(PG@WuF<324U4?}v%NKBh9M#mM zxt=Gdw{DT)NQ;T6v6-&uo*m3@Q_iV}r?+=C3Ls2-3Mlt2yWT*3 ztgrGF)477GiwkN~Hm@S$GB)0&S&W8&mb1y^kE>&6w5zv52w8COGPFC+{-RF;ppm4< zz?T&Te9UNJLjnib(U?x&^kjw6cVyz4Wx&CyS7j>VDo_ox1gFC`_xEP9q=TBf=`NAO zWtkA)6pRn&IHSPf>4|at7%O2X7Oif*i{8|D5wjlg2xkMWY|Jm0Sbtlc$uVa$sr*_t zyDrB`kJAD*2JOz@3D=i81l%l-H|G`G<)g}u zHf4gQSXOc_DmVGq5O819i7oVfdBoWMX&1pdpH*YCenWG!UR_>;+9KLsuIuoJs)Mz+ ziB-m-*F8;_wJs>5C#udAIiQd3X!9ML*%$@!%vrfh7RYYYWTjQ@(X*n!M>`FlYT&~%+qaaf6)jFTcb7SYy+JR{28$byT=sJgG zt9`T2c85^GCcrj3<&<%swECNN^`mdrFDsA5mkqXy@@EmCZIDr`smk5enRhU}ibAzHsbW4#anREF!V)#` zALUR;yOao1t-tvY@wvPyXbhQ3|I_Qd-8uzY#Ui`OEDfa3C&oK!&k?CYaP+eXhi>g& zl6Q1pbJ@8I`pNchEp8ILDw}U5D00l%J-Ev@JDfkzTS-8eruh0-p8>bZ=8t=HX|Uov zltSjh=;yUKlUL7Apch^fIBsphI2Tz$$cDH5Goe?b>cXUh>US)$6mR9vz5)$s%`vhD z%d+tW1cDXD_or;irDq!lbkYPC-a6d9i8}F-fUQV`X`o;JT}zvQ+UE6m&_fQM_Ux`egx`F|0oy)6mna>9x0hwU>%u9Y^eqil=SN;H*0 zwqP#634PeIsmZdTB;fhLp^vj5#A6L0V+G=?vR2l?z@VcW7priYMok2Mjkcg#`jM!EOL+hAsqD&tMN!#o5^~ZW13q8xc_3yDc*Ew|NX391n3u;Txa5+yh znaQGgUwK&Rmj-_}sQ%RLPz5!%bjc+d}SD&3>j`KshfQ#^1GWK;=rayIEl1tM8##3-3G->8t^^$_`8 z=LA-R@ujL!lT39&V82}&tFrRif1&dSl$y3MWREawNX#KTf=L9NWtjC_pM{KYf|qG5;uy_e66u+m%f)Ntn)R)Xkdik~NEErOQ!YcC zA~7Cy_q^|kHycb6TKbyx-fY$XRUOolom?eQUS?mS!t1-QD01v+(QM9=dXxBz@P+B-jl4sq(V1U(VgbS1r+GG#m3?^Eln zY_3z692BZ73a$?~zMucycrfRy(cW*s*fVY)MPM-?v+bChI}c?tTaL4R5Jew5`tH;w zdpcVF$5Wh*!WYj@mK(-N2GXWPgL3SMg=D^&_eO3baHUQov!7F57Kd(hNZBgC(7@3W zbn^S0BB1C(3SrFMUyZh(U$;a3<{u3skT)lv-rxJo%VyN&Xr-N-Rb8kf+x?gxZ~A!w zoF;=Ic!O1gG^~hW$9f>M>|V*bW_6wJ0Xaa7hd1^3k%baGDH+tOF=i>5@qwD>C4O2{ ztkzvy?*u|91U#xB^1X?hzeKv7QBntZ%zK{@pji)me2%FI5-;g}fWz5;ap*Xj;+`8y zd3!$Bb2m@6D8|dx#iX#_GZ+4&>p3iQ)cA^IQrvFGJEtsEmD`MD!9? ze#Xm*HxjE-vqtg*AYI94AI5F&)c%A^8Y+y4Qc$9mz z*h0R%4vksxL&t5dpcMdP>lVc^;!|r7*X0)$huKY#k~w7OKkG+V)7aF*#mV->v55&I zQ}52=K*3kR`U_Zv1Ns zJOs`$yo?|kthGPqpat;JwC`Nb*6p=K8#iBqBEl3@{iC-^SHFSU3?7PJ-+^{aXHCH- z@2%VJ>91*DNqwwITq65o+WTVE1b@k_F))VUshbY;9_Ndkf682j0ATg!RlbXY7FEy| z;P0(zcE|;9S)sl#T0@|~bTo)kTQtBc%w!Kli5hDsyd;loOJ)E?g zOVw+&VWXy(>m>8Pw`)~SAx0)H!oNTz6Qi$mQ4A!C{yB8QcnKVEK8?Q(E`-1b@BQgj zzFA|&luCI@DADSoK>=}0Y7~#*IiOl}bKM^jVD)ink09segG}LJ!P~h$@ArqU_B<*Z z8<~Fl_S{i5@AMp~d$9epX*f!#L*T(!W-3RiinLSmKd@mxC^tgvdvAIxz)*$kFIDHR zE*PYrcbTfg>clm=4>dqH@8_YOFcy457MWG^&>St2@~j5y9xgf$duHFD*7er!w2Ivt zvhlLDb*z`R;7g(xCm(4NydB<#T>d40cHk?WqMLP$_IJ>MLz9zkzCpUnRUxoMU zBk#+<9vr@(W#y+tDNxXW&s6*^$>oT*GH zMo(VOMP6S!hhEYQ9M4OJceygCcLj&)^i~YZ{0!+T4&C(Buj49{kmzpf8a5qHYsuP< z>rK}^6d%TGWBhV1fgYGW3O-J`8K zE{Xm$7T|iu#SpHIIIPqze4x^#O9Xrj@+|85UyaQ892+VbY;ES}yH6ZWs#FOJDzX+F zMbRoUms$%Ldtl?HopuZ}6aQU{6nHaW#$udY{@DWi4L+uBNHClG35O+<~)}fxBFd^Mq`ZerT!lzryI`zHDv69GWq+7OdV}<`#2>p z4(cauurDaj>oenkO<9D(y~=cjV_HC6GL4kK7fm;lXt<|3UA^ZWw?22d_$Rm+Y^j>NcvJeT=SZ84EwkCN2qWSz1}WK#=qg4OVi{&!6$x7 z*I{$B%{8C5hm@yAN|z7Kh6GXfo`ZbCeuM-Q`OsK03Tdl3%GI?XSKKXLT=tok1A0<` z=_WQRv%-0KYef%xxtA_UZO&lQb*p}90BpP%M@RgVJ6tV3bI#7o%J&(!b2Zau9=!b> zz=}Ii_KgcwTE}qUHdjIk`^k>`unU$OMia%s7xd*!abg57GEQF(oq=n*htJ3lnLTp${x4~m{IB(}9 ze!9XuQDuLu9?Sg7c+RuReah@MSX{M!VdlG6SyILL=NcsFRB8@41zZr&5?Gsz27Zfo3 zh(%GeGvrL1bJ>yYy^rT&d6<9(sL1Mb~mI2=?6A4aK+gN%65ERJ5H?gjAbfy6@`>F- zpnRHpujA_)xStbZ z3ktnx6gdBwH#O)DX%;q(-C_NX?f`0R%{eEaOOy7QobGBR{cB*WJsboC6JU>)!eR!Tt>PXET`LN<z>;}c9B=P z=1US`InTEG(pwXr#BLCl&P$bhcZu78aa}<^KHEThJznKLJ^zbBl{~z@{35HjJ{OjT zpV5n*(tSR?E_72yhSHPHyA>R3lk*leOYE!tvas^j*Z^K=!f4F)?tJ(piih@3v@7@mEQH{UC=rjh+o|7(r_c9YoT)HTX%sErHR)e^DkI#Cvn zlKU>4O=k*tL7x26$sLUuEY1X~{TmHvK#sP~AR@#6E4LE!0wU$>Nm zI@eG!uL49i`8oU;BU!@LSoDTy+rDf}kRQE@$B`Xv8#It6T=ZR5@pYaAE?HlhGHggU zBjmS@h`|Be%~-49nId zei;pa#?VgMXy>c>@%@C5+ZVRg5!`pYQR>?rc6rB4X?jfXtKple$_?weF-e(ua;hgI z;}C8eYNFv>&X#C(J#V?lXYcRXjkmYTrWL-P<)44MVEFlxUxcC+t<2Yqz(8_|=U+!7 zs4a`oA|ouY<~T&;cS^K+J5jbp!44^3_)2i?rqNQm>d}e_Pv1q%c}s?eI1BhNR1knhqk!ShV_ zpgE9V9?l9M=R7V_X>fZve86Mld}!VCQvc%Q)2NR7diNt=mX3-1Gs4xYizc`6(Vz8P z+gK-wI#2%e@r!`)8}RfR@Kccc{0&JUA_3iLFF|BGEjm1}@3Fk4@79PF@5O+0bg{!> z%Gv%oK`1{%xb=Bzdn|)gykbYl$^(Aiy{JU)3C9Jc!9(2Rth-iLx6US;#q&;-1KUHJ zz6-1~Z8<|tv0}NLZ4wXhqvOk2&#s+X7Z20bllk9iYY@)K#IogD4P5E_PJ5@>8lm=n zXu;@Q*=-&d=5=+&}xk`uc z5iBoPD6fb*cUqj}C&>(`EB>tC{bb)_7gaazksUu--r|?^!FYnfHuvP6z3~X?SZitZ z)5FLm+fg7HKl5Y}($~-)D|rsol25aDI!N_nfDl-!*q_|}*rq#ey~R_ilD~mQw*B+| z*AUyK(zgVrGZ#Pe6n@OmMe!%v3kA2@4^QB)Xp^n|8odV{<&L3`m|ud{lvi!=J|V`e zm!nAR=?@x~)|V?V|y`<`={+?~{$n>bfNh z;KTa8rM~cJU9B$((4#tvuomz9+UQX$W6r)+FWG>OWb9q?|NFYaZAV~!m%+qSDE^t>Hy2;{3_g#&tQy&u@LzBM zcvr8Fq;J=KB>4B9a&z$%eDHbgvFiW(apOqU%+=$!0snOY@U7Xw=k?bQ{gcE0z4(F$ z_|oI^&lHf}r+>!*IT#Kg!T&|1q5XFn+zbzR`M-nwpE3IXaF8Qu|J=?0efQ5C{jW*` set for `maven-compiler-plugin` in the module's `pom.xml` (11 is project default if the module doesn't set one). +IntelliJ uses a notion of "Language Level" to drive assistance features according to the target Java version of a project. It often infers this appropriately from a Maven import, but it's wise to check, especially for a multi-module project with differing target Java releases across modules, like ours. Language level [can be set per module](https://www.jetbrains.com/help/idea/sources-tab.html#module_language_level)—if IntelliJ is suggesting things that turn out to fail when building with `mvn`, make sure the language level of the module in question corresponds to the `` set for `maven-compiler-plugin` in the module's `pom.xml` \(11 is project default if the module doesn't set one\). -Ensure that the Project _SDK_ (not language level) is set to a Java 11 JDK, and that all modules use the Project SDK (see [the Dependencies tab] in the Modules view). +Ensure that the Project _SDK_ \(not language level\) is set to a Java 11 JDK, and that all modules use the Project SDK \(see [the Dependencies tab](https://www.jetbrains.com/help/idea/dependencies.html) in the Modules view\). -[module lang level]: https://www.jetbrains.com/help/idea/sources-tab.html#module_language_level -[the Dependencies tab]: https://www.jetbrains.com/help/idea/dependencies.html diff --git a/docs/docs/.gitbook/assets/feast-docs-overview-diagram-2.svg b/docs/docs/.gitbook/assets/feast-docs-overview-diagram-2.svg new file mode 100644 index 00000000000..7f30963ec78 --- /dev/null +++ b/docs/docs/.gitbook/assets/feast-docs-overview-diagram-2.svg @@ -0,0 +1 @@ + \ No newline at end of file From 2fa30ac1883ad6fdc42168487a0fbafd38020b16 Mon Sep 17 00:00:00 2001 From: David Heryanto Date: Tue, 31 Mar 2020 15:30:44 +0800 Subject: [PATCH 106/176] Apply a fixed window before writing row metrics (#590) --- .../metrics/WriteFeatureValueMetricsDoFn.java | 3 + .../metrics/WriteMetricsTransform.java | 86 ++++--- .../metrics/WriteRowMetricsDoFn.java | 225 +++++++++++++----- .../WriteFeatureValueMetricsDoFnTest.java | 90 +++---- .../metrics/WriteRowMetricsDoFnTest.java | 88 +++++++ .../src/test/java/feast/test/TestUtil.java | 63 +++++ .../transform/WriteRowMetricsDoFnTest.input | 4 + .../transform/WriteRowMetricsDoFnTest.output | 23 ++ 8 files changed, 413 insertions(+), 169 deletions(-) create mode 100644 ingestion/src/test/java/feast/ingestion/transform/metrics/WriteRowMetricsDoFnTest.java create mode 100644 ingestion/src/test/resources/feast/ingestion/transform/WriteRowMetricsDoFnTest.input create mode 100644 ingestion/src/test/resources/feast/ingestion/transform/WriteRowMetricsDoFnTest.output diff --git a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java index a4ed07b5052..cfecb858dcf 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java +++ b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java @@ -121,11 +121,14 @@ public void processElement( ProcessContext context, @Element KV> featureSetRefToFeatureRows) { if (statsDClient == null) { + log.error("StatsD client is null, likely because it encounters an error during setup"); return; } String featureSetRef = featureSetRefToFeatureRows.getKey(); if (featureSetRef == null) { + log.error( + "Feature set reference in the feature row is null. Please check the input feature rows from previous steps"); return; } String[] colonSplits = featureSetRef.split(":"); diff --git a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteMetricsTransform.java b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteMetricsTransform.java index 10322ac812f..8a5869d78ec 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteMetricsTransform.java +++ b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteMetricsTransform.java @@ -27,6 +27,7 @@ import org.apache.beam.sdk.transforms.windowing.FixedWindows; import org.apache.beam.sdk.transforms.windowing.Window; import org.apache.beam.sdk.values.KV; +import org.apache.beam.sdk.values.PCollection; import org.apache.beam.sdk.values.PCollectionTuple; import org.apache.beam.sdk.values.PDone; import org.apache.beam.sdk.values.TupleTag; @@ -73,52 +74,47 @@ public PDone expand(PCollectionTuple input) { .setStoreName(getStoreName()) .build())); - input - .get(getSuccessTag()) - .apply( - "WriteRowMetrics", - ParDo.of( - WriteRowMetricsDoFn.newBuilder() - .setStatsdHost(options.getStatsdHost()) - .setStatsdPort(options.getStatsdPort()) - .setStoreName(getStoreName()) - .build())); + // Fixed window is applied so the metric collector will not be overwhelmed with the metrics + // data. For validation, only summaries of the values are usually required vs the actual + // values. + PCollection>> validRowsGroupedByRef = + input + .get(getSuccessTag()) + .apply( + "FixedWindow", + Window.into( + FixedWindows.of( + Duration.standardSeconds( + options.getWindowSizeInSecForFeatureValueMetric())))) + .apply( + "ConvertToKV_FeatureSetRefToFeatureRow", + ParDo.of( + new DoFn>() { + @ProcessElement + public void processElement( + ProcessContext c, @Element FeatureRow featureRow) { + c.output(KV.of(featureRow.getFeatureSet(), featureRow)); + } + })) + .apply("GroupByFeatureSetRef", GroupByKey.create()); - // 1. Apply a fixed window - // 2. Group feature row by feature set reference - // 3. Calculate min, max, mean, percentiles of numerical values of features in the window - // and - // 4. Send the aggregate value to StatsD metric collector. - // - // NOTE: window is applied here so the metric collector will not be overwhelmed with - // metrics data. And for metric data, only statistic of the values are usually required - // vs the actual values. - input - .get(getSuccessTag()) - .apply( - "FixedWindow", - Window.into( - FixedWindows.of( - Duration.standardSeconds( - options.getWindowSizeInSecForFeatureValueMetric())))) - .apply( - "ConvertTo_FeatureSetRefToFeatureRow", - ParDo.of( - new DoFn>() { - @ProcessElement - public void processElement(ProcessContext c, @Element FeatureRow featureRow) { - c.output(KV.of(featureRow.getFeatureSet(), featureRow)); - } - })) - .apply("GroupByFeatureSetRef", GroupByKey.create()) - .apply( - "WriteFeatureValueMetrics", - ParDo.of( - WriteFeatureValueMetricsDoFn.newBuilder() - .setStatsdHost(options.getStatsdHost()) - .setStatsdPort(options.getStatsdPort()) - .setStoreName(getStoreName()) - .build())); + validRowsGroupedByRef.apply( + "WriteRowMetrics", + ParDo.of( + WriteRowMetricsDoFn.newBuilder() + .setStatsdHost(options.getStatsdHost()) + .setStatsdPort(options.getStatsdPort()) + .setStoreName(getStoreName()) + .build())); + + validRowsGroupedByRef.apply( + "WriteFeatureValueMetrics", + ParDo.of( + WriteFeatureValueMetricsDoFn.newBuilder() + .setStatsdHost(options.getStatsdHost()) + .setStatsdPort(options.getStatsdPort()) + .setStoreName(getStoreName()) + .build())); return PDone.in(input.getPipeline()); case "none": diff --git a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteRowMetricsDoFn.java b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteRowMetricsDoFn.java index 2cd1ee94ecc..2fe1f2e7f01 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteRowMetricsDoFn.java +++ b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteRowMetricsDoFn.java @@ -17,17 +17,26 @@ package feast.ingestion.transform.metrics; import com.google.auto.value.AutoValue; +import com.google.protobuf.util.Timestamps; import com.timgroup.statsd.NonBlockingStatsDClient; import com.timgroup.statsd.StatsDClient; -import com.timgroup.statsd.StatsDClientException; import feast.types.FeatureRowProto.FeatureRow; import feast.types.FieldProto.Field; +import feast.types.ValueProto.Value; import feast.types.ValueProto.Value.ValCase; +import java.time.Clock; +import java.util.HashMap; +import java.util.Map; +import java.util.Map.Entry; +import javax.annotation.Nullable; import org.apache.beam.sdk.transforms.DoFn; +import org.apache.beam.sdk.values.KV; +import org.apache.commons.lang3.ArrayUtils; +import org.apache.commons.math3.stat.descriptive.DescriptiveStatistics; import org.slf4j.Logger; @AutoValue -public abstract class WriteRowMetricsDoFn extends DoFn { +public abstract class WriteRowMetricsDoFn extends DoFn>, Void> { private static final Logger log = org.slf4j.LoggerFactory.getLogger(WriteRowMetricsDoFn.class); @@ -39,12 +48,38 @@ public abstract class WriteRowMetricsDoFn extends DoFn { public static final String FEATURE_TAG_KEY = "feast_feature_name"; public static final String INGESTION_JOB_NAME_KEY = "ingestion_job_name"; + public static final String GAUGE_NAME_FEATURE_ROW_LAG_MS_MIN = "feature_row_lag_ms_min"; + public static final String GAUGE_NAME_FEATURE_ROW_LAG_MS_MAX = "feature_row_lag_ms_max"; + public static final String GAUGE_NAME_FEATURE_ROW_LAG_MS_MEAN = "feature_row_lag_ms_mean"; + public static final String GAUGE_NAME_FEATURE_ROW_LAG_MS_PERCENTILE_90 = + "feature_row_lag_ms_percentile_90"; + public static final String GAUGE_NAME_FEATURE_ROW_LAG_MS_PERCENTILE_95 = + "feature_row_lag_ms_percentile_95"; + public static final String GAUGE_NAME_FEATURE_ROW_LAG_MS_PERCENTILE_99 = + "feature_row_lag_ms_percentile_99"; + + public static final String GAUGE_NAME_FEATURE_VALUE_LAG_MS_MIN = "feature_value_lag_ms_min"; + public static final String GAUGE_NAME_FEATURE_VALUE_LAG_MS_MAX = "feature_value_lag_ms_max"; + public static final String GAUGE_NAME_FEATURE_VALUE_LAG_MS_MEAN = "feature_value_lag_ms_mean"; + public static final String GAUGE_NAME_FEATURE_VALUE_LAG_MS_PERCENTILE_90 = + "feature_value_lag_ms_percentile_90"; + public static final String GAUGE_NAME_FEATURE_VALUE_LAG_MS_PERCENTILE_95 = + "feature_value_lag_ms_percentile_95"; + public static final String GAUGE_NAME_FEATURE_VALUE_LAG_MS_PERCENTILE_99 = + "feature_value_lag_ms_percentile_99"; + + public static final String COUNT_NAME_FEATURE_ROW_INGESTED = "feature_row_ingested_count"; + public static final String COUNT_NAME_FEATURE_VALUE_MISSING = "feature_value_missing_count"; + public abstract String getStoreName(); public abstract String getStatsdHost(); public abstract int getStatsdPort(); + @Nullable + public abstract Clock getClock(); + public static WriteRowMetricsDoFn create( String newStoreName, String newStatsdHost, int newStatsdPort) { return newBuilder() @@ -69,79 +104,147 @@ public abstract static class Builder { public abstract Builder setStatsdPort(int statsdPort); + /** + * setClock will override the default system clock used to calculate feature row lag. + * + * @param clock Clock instance + */ + public abstract Builder setClock(Clock clock); + public abstract WriteRowMetricsDoFn build(); } @Setup public void setup() { - statsd = new NonBlockingStatsDClient(METRIC_PREFIX, getStatsdHost(), getStatsdPort()); + // Note that exception may be thrown during StatsD client instantiation but no exception + // will be thrown when sending metrics (mimicking the UDP protocol behaviour). + // https://jar-download.com/artifacts/com.datadoghq/java-dogstatsd-client/2.1.1/documentation + // https://github.com/DataDog/java-dogstatsd-client#unix-domain-socket-support + try { + statsd = new NonBlockingStatsDClient(METRIC_PREFIX, getStatsdHost(), getStatsdPort()); + } catch (Exception e) { + log.error("StatsD client cannot be started: " + e.getMessage()); + } } + @SuppressWarnings("DuplicatedCode") @ProcessElement - public void processElement(ProcessContext c) { + public void processElement( + ProcessContext c, @Element KV> featureSetRefToFeatureRows) { + if (statsd == null) { + log.error("StatsD client is null, likely because it encounters an error during setup"); + return; + } - try { - FeatureRow row = c.element(); - long eventTimestamp = com.google.protobuf.util.Timestamps.toMillis(row.getEventTimestamp()); - - String[] split = row.getFeatureSet().split(":"); - String featureSetProject = split[0].split("/")[0]; - String featureSetName = split[0].split("/")[1]; - String featureSetVersion = split[1]; - - statsd.histogram( - "feature_row_lag_ms", - System.currentTimeMillis() - eventTimestamp, - STORE_TAG_KEY + ":" + getStoreName(), - FEATURE_SET_PROJECT_TAG_KEY + ":" + featureSetProject, - FEATURE_SET_NAME_TAG_KEY + ":" + featureSetName, - FEATURE_SET_VERSION_TAG_KEY + ":" + featureSetVersion, - INGESTION_JOB_NAME_KEY + ":" + c.getPipelineOptions().getJobName()); - - statsd.histogram( - "feature_row_event_time_epoch_ms", - eventTimestamp, - STORE_TAG_KEY + ":" + getStoreName(), - FEATURE_SET_PROJECT_TAG_KEY + ":" + featureSetProject, - FEATURE_SET_NAME_TAG_KEY + ":" + featureSetName, - FEATURE_SET_VERSION_TAG_KEY + ":" + featureSetVersion, - INGESTION_JOB_NAME_KEY + ":" + c.getPipelineOptions().getJobName()); - - for (Field field : row.getFieldsList()) { - if (!field.getValue().getValCase().equals(ValCase.VAL_NOT_SET)) { - statsd.histogram( - "feature_value_lag_ms", - System.currentTimeMillis() - eventTimestamp, - STORE_TAG_KEY + ":" + getStoreName(), - FEATURE_SET_PROJECT_TAG_KEY + ":" + featureSetProject, - FEATURE_SET_NAME_TAG_KEY + ":" + featureSetName, - FEATURE_SET_VERSION_TAG_KEY + ":" + featureSetVersion, - FEATURE_TAG_KEY + ":" + field.getName(), - INGESTION_JOB_NAME_KEY + ":" + c.getPipelineOptions().getJobName()); + String featureSetRef = featureSetRefToFeatureRows.getKey(); + if (featureSetRef == null) { + log.error( + "Feature set reference in the feature row is null. Please check the input feature rows from previous steps"); + return; + } + String[] colonSplits = featureSetRef.split(":"); + if (colonSplits.length != 2) { + log.error( + "Skip writing feature row metrics because the feature set reference '{}' does not" + + "follow the required format /:", + featureSetRef); + return; + } + String[] slashSplits = colonSplits[0].split("/"); + if (slashSplits.length != 2) { + log.error( + "Skip writing feature row metrics because the feature set reference '{}' does not" + + "follow the required format /:", + featureSetRef); + return; + } + + String featureSetProject = slashSplits[0]; + String featureSetName = slashSplits[1]; + String featureSetVersion = colonSplits[1]; + + // featureRowLagStats is stats for feature row lag for feature set "featureSetName" + DescriptiveStatistics featureRowLagStats = new DescriptiveStatistics(); + // featureNameToLagStats is stats for feature lag for all features in feature set + // "featureSetName" + Map featureNameToLagStats = new HashMap<>(); + // featureNameToMissingCount is count for "value_not_set" for all features in feature set + // "featureSetName" + Map featureNameToMissingCount = new HashMap<>(); + + for (FeatureRow featureRow : featureSetRefToFeatureRows.getValue()) { + long currentTime = getClock() == null ? System.currentTimeMillis() : getClock().millis(); + long featureRowLag = currentTime - Timestamps.toMillis(featureRow.getEventTimestamp()); + featureRowLagStats.addValue(featureRowLag); + + for (Field field : featureRow.getFieldsList()) { + String featureName = field.getName(); + Value featureValue = field.getValue(); + if (!featureNameToLagStats.containsKey(featureName)) { + // Ensure map contains the "featureName" key + featureNameToLagStats.put(featureName, new DescriptiveStatistics()); + } + if (!featureNameToMissingCount.containsKey(featureName)) { + // Ensure map contains the "featureName" key + featureNameToMissingCount.put(featureName, 0L); + } + if (featureValue.getValCase().equals(ValCase.VAL_NOT_SET)) { + featureNameToMissingCount.put( + featureName, featureNameToMissingCount.get(featureName) + 1); } else { - statsd.count( - "feature_value_missing_count", - 1, - STORE_TAG_KEY + ":" + getStoreName(), - FEATURE_SET_PROJECT_TAG_KEY + ":" + featureSetProject, - FEATURE_SET_NAME_TAG_KEY + ":" + featureSetName, - FEATURE_SET_VERSION_TAG_KEY + ":" + featureSetVersion, - FEATURE_TAG_KEY + ":" + field.getName(), - INGESTION_JOB_NAME_KEY + ":" + c.getPipelineOptions().getJobName()); + featureNameToLagStats.get(featureName).addValue(featureRowLag); } } + } + + String[] tags = { + STORE_TAG_KEY + ":" + getStoreName(), + FEATURE_SET_PROJECT_TAG_KEY + ":" + featureSetProject, + FEATURE_SET_NAME_TAG_KEY + ":" + featureSetName, + FEATURE_SET_VERSION_TAG_KEY + ":" + featureSetVersion, + INGESTION_JOB_NAME_KEY + ":" + c.getPipelineOptions().getJobName(), + }; + + statsd.count(COUNT_NAME_FEATURE_ROW_INGESTED, featureRowLagStats.getN(), tags); + // DescriptiveStatistics returns invalid NaN value for getMin(), getMax(), ... when there is no + // items in the stats. + if (featureRowLagStats.getN() > 0) { + statsd.gauge(GAUGE_NAME_FEATURE_ROW_LAG_MS_MIN, featureRowLagStats.getMin(), tags); + statsd.gauge(GAUGE_NAME_FEATURE_ROW_LAG_MS_MAX, featureRowLagStats.getMax(), tags); + statsd.gauge(GAUGE_NAME_FEATURE_ROW_LAG_MS_MEAN, featureRowLagStats.getMean(), tags); + statsd.gauge( + GAUGE_NAME_FEATURE_ROW_LAG_MS_PERCENTILE_90, featureRowLagStats.getPercentile(90), tags); + statsd.gauge( + GAUGE_NAME_FEATURE_ROW_LAG_MS_PERCENTILE_95, featureRowLagStats.getPercentile(95), tags); + statsd.gauge( + GAUGE_NAME_FEATURE_ROW_LAG_MS_PERCENTILE_99, featureRowLagStats.getPercentile(99), tags); + } + for (Entry entry : featureNameToLagStats.entrySet()) { + String featureName = entry.getKey(); + String[] tagsWithFeatureName = ArrayUtils.add(tags, FEATURE_TAG_KEY + ":" + featureName); + DescriptiveStatistics stats = entry.getValue(); + if (stats.getN() > 0) { + statsd.gauge(GAUGE_NAME_FEATURE_VALUE_LAG_MS_MIN, stats.getMin(), tagsWithFeatureName); + statsd.gauge(GAUGE_NAME_FEATURE_VALUE_LAG_MS_MAX, stats.getMax(), tagsWithFeatureName); + statsd.gauge(GAUGE_NAME_FEATURE_VALUE_LAG_MS_MEAN, stats.getMean(), tagsWithFeatureName); + statsd.gauge( + GAUGE_NAME_FEATURE_VALUE_LAG_MS_PERCENTILE_90, + stats.getPercentile(90), + tagsWithFeatureName); + statsd.gauge( + GAUGE_NAME_FEATURE_VALUE_LAG_MS_PERCENTILE_95, + stats.getPercentile(95), + tagsWithFeatureName); + statsd.gauge( + GAUGE_NAME_FEATURE_VALUE_LAG_MS_PERCENTILE_99, + stats.getPercentile(99), + tagsWithFeatureName); + } statsd.count( - "feature_row_ingested_count", - 1, - STORE_TAG_KEY + ":" + getStoreName(), - FEATURE_SET_PROJECT_TAG_KEY + ":" + featureSetProject, - FEATURE_SET_NAME_TAG_KEY + ":" + featureSetName, - FEATURE_SET_VERSION_TAG_KEY + ":" + featureSetVersion, - INGESTION_JOB_NAME_KEY + ":" + c.getPipelineOptions().getJobName()); - - } catch (StatsDClientException e) { - log.warn("Unable to push metrics to server", e); + COUNT_NAME_FEATURE_VALUE_MISSING, + featureNameToMissingCount.get(featureName), + tagsWithFeatureName); } } } diff --git a/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFnTest.java b/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFnTest.java index 88e1bf8088d..cc65f2cff96 100644 --- a/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFnTest.java +++ b/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFnTest.java @@ -19,6 +19,9 @@ import static org.junit.Assert.fail; import com.google.protobuf.ByteString; +import com.google.protobuf.Timestamp; +import com.google.protobuf.util.Timestamps; +import feast.test.TestUtil.DummyStatsDServer; import feast.types.FeatureRowProto.FeatureRow; import feast.types.FeatureRowProto.FeatureRow.Builder; import feast.types.FieldProto.Field; @@ -32,13 +35,10 @@ import feast.types.ValueProto.Value; import java.io.BufferedReader; import java.io.IOException; -import java.net.DatagramPacket; -import java.net.DatagramSocket; -import java.net.SocketException; import java.net.URL; -import java.nio.charset.StandardCharsets; import java.nio.file.Files; import java.nio.file.Paths; +import java.time.Instant; import java.util.ArrayList; import java.util.Arrays; import java.util.HashMap; @@ -100,11 +100,12 @@ public void shouldSendCorrectStatsDMetrics() throws IOException, InterruptedExce fail(String.format("Expected StatsD metric not found:\n%s", expected)); } } + statsDServer.stop(); } // Test utility method to read expected StatsD metrics output from a text file. @SuppressWarnings("SameParameterValue") - private List readTestOutput(String path) throws IOException { + public static List readTestOutput(String path) throws IOException { URL url = Thread.currentThread().getContextClassLoader().getResource(path); if (url == null) { throw new IllegalArgumentException( @@ -123,9 +124,19 @@ private List readTestOutput(String path) throws IOException { return lines; } + public static Map> readTestInput(String path) throws IOException { + return readTestInput(path, null); + } + // Test utility method to create test feature row data from a text file. + // If tsOverride is not null, all the feature row will have the same timestamp "tsOverride". + // Else if there exist a "timestamp" column with RFC3339 format, the feature row will be assigned + // that timestamp. + // Else no timestamp will be assigned (the feature row will have the default proto Timestamp + // object). @SuppressWarnings("SameParameterValue") - private Map> readTestInput(String path) throws IOException { + public static Map> readTestInput(String path, Timestamp tsOverride) + throws IOException { Map> data = new HashMap<>(); URL url = Thread.currentThread().getContextClassLoader().getResource(path); if (url == null) { @@ -162,6 +173,13 @@ private Map> readTestInput(String path) throws IOEx continue; } String colName = colNames.get(i); + if (colName.equals("timestamp")) { + Instant instant = Instant.parse(colVal); + featureRowBuilder.setEventTimestamp( + Timestamps.fromNanos(instant.getEpochSecond() * 1_000_000_000 + instant.getNano())); + continue; + } + Field.Builder fieldBuilder = Field.newBuilder().setName(colName); if (!colVal.isEmpty()) { switch (colName) { @@ -245,6 +263,9 @@ private Map> readTestInput(String path) throws IOEx data.put(featureRowBuilder.getFeatureSet(), new ArrayList<>()); } List featureRowsByFeatureSetRef = data.get(featureRowBuilder.getFeatureSet()); + if (tsOverride != null) { + featureRowBuilder.setEventTimestamp(tsOverride); + } featureRowsByFeatureSetRef.add(featureRowBuilder.build()); } @@ -258,61 +279,4 @@ private Map> readTestInput(String path) throws IOEx } return dataWithIterable; } - - // Modified version of - // https://github.com/tim-group/java-statsd-client/blob/master/src/test/java/com/timgroup/statsd/NonBlockingStatsDClientTest.java - @SuppressWarnings("CatchMayIgnoreException") - private static final class DummyStatsDServer { - - private final List messagesReceived = new ArrayList(); - private final DatagramSocket server; - - public DummyStatsDServer(int port) { - try { - server = new DatagramSocket(port); - } catch (SocketException e) { - throw new IllegalStateException(e); - } - new Thread( - () -> { - try { - while (true) { - final DatagramPacket packet = new DatagramPacket(new byte[65535], 65535); - server.receive(packet); - messagesReceived.add( - new String(packet.getData(), StandardCharsets.UTF_8).trim() + "\n"); - // The sleep duration here is shorter than that used in waitForMessage() at - // 50ms. - // Otherwise sometimes some messages seem to be lost, leading to flaky tests. - Thread.sleep(15L); - } - - } catch (Exception e) { - } - }) - .start(); - } - - public void stop() { - server.close(); - } - - public void waitForMessage() { - while (messagesReceived.isEmpty()) { - try { - Thread.sleep(50L); - } catch (InterruptedException e) { - } - } - } - - public List messagesReceived() { - List out = new ArrayList<>(); - for (String msg : messagesReceived) { - String[] lines = msg.split("\n"); - out.addAll(Arrays.asList(lines)); - } - return out; - } - } } diff --git a/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteRowMetricsDoFnTest.java b/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteRowMetricsDoFnTest.java new file mode 100644 index 00000000000..6e3caff56b9 --- /dev/null +++ b/ingestion/src/test/java/feast/ingestion/transform/metrics/WriteRowMetricsDoFnTest.java @@ -0,0 +1,88 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.ingestion.transform.metrics; + +import static feast.ingestion.transform.metrics.WriteFeatureValueMetricsDoFnTest.readTestInput; +import static feast.ingestion.transform.metrics.WriteFeatureValueMetricsDoFnTest.readTestOutput; +import static org.junit.Assert.fail; + +import feast.test.TestUtil.DummyStatsDServer; +import feast.types.FeatureRowProto.FeatureRow; +import java.io.IOException; +import java.time.Clock; +import java.time.Instant; +import java.time.ZoneId; +import java.util.List; +import java.util.Map; +import org.apache.beam.sdk.options.PipelineOptions; +import org.apache.beam.sdk.options.PipelineOptionsFactory; +import org.apache.beam.sdk.testing.TestPipeline; +import org.apache.beam.sdk.transforms.Create; +import org.apache.beam.sdk.transforms.ParDo; +import org.junit.Rule; +import org.junit.Test; + +public class WriteRowMetricsDoFnTest { + + @Rule public final transient TestPipeline pipeline = TestPipeline.create(); + private static final int STATSD_SERVER_PORT = 17255; + private final DummyStatsDServer statsDServer = new DummyStatsDServer(STATSD_SERVER_PORT); + + @Test + public void shouldSendCorrectStatsDMetrics() throws IOException, InterruptedException { + PipelineOptions pipelineOptions = PipelineOptionsFactory.create(); + pipelineOptions.setJobName("job"); + Map> input = + readTestInput("feast/ingestion/transform/WriteRowMetricsDoFnTest.input"); + List expectedLines = + readTestOutput("feast/ingestion/transform/WriteRowMetricsDoFnTest.output"); + + pipeline + .apply(Create.of(input)) + .apply( + ParDo.of( + WriteRowMetricsDoFn.newBuilder() + .setStatsdHost("localhost") + .setStatsdPort(STATSD_SERVER_PORT) + .setStoreName("store") + .setClock(Clock.fixed(Instant.ofEpochSecond(1585548645), ZoneId.of("UTC"))) + .build())); + pipeline.run(pipelineOptions).waitUntilFinish(); + // Wait until StatsD has finished processed all messages, 3 sec is a reasonable duration + // based on empirical testing. + Thread.sleep(3000); + + List actualLines = statsDServer.messagesReceived(); + for (String expected : expectedLines) { + boolean matched = false; + for (String actual : actualLines) { + if (actual.equals(expected)) { + matched = true; + break; + } + } + if (!matched) { + System.out.println("Print actual metrics output for debugging:"); + for (String line : actualLines) { + System.out.println(line); + } + fail(String.format("Expected StatsD metric not found:\n%s", expected)); + } + } + statsDServer.stop(); + } +} diff --git a/ingestion/src/test/java/feast/test/TestUtil.java b/ingestion/src/test/java/feast/test/TestUtil.java index 5c16d7e9e31..3cad39e3ec5 100644 --- a/ingestion/src/test/java/feast/test/TestUtil.java +++ b/ingestion/src/test/java/feast/test/TestUtil.java @@ -36,6 +36,12 @@ import feast.types.ValueProto.Value; import feast.types.ValueProto.ValueType; import java.io.IOException; +import java.net.DatagramPacket; +import java.net.DatagramSocket; +import java.net.SocketException; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Arrays; import java.util.List; import java.util.Properties; import java.util.concurrent.ExecutionException; @@ -332,6 +338,63 @@ static void start(int zookeeperPort, String zookeeperDataDir) { } } + // Modified version of + // https://github.com/tim-group/java-statsd-client/blob/master/src/test/java/com/timgroup/statsd/NonBlockingStatsDClientTest.java + @SuppressWarnings("CatchMayIgnoreException") + public static class DummyStatsDServer { + + private final List messagesReceived = new ArrayList(); + private final DatagramSocket server; + + public DummyStatsDServer(int port) { + try { + server = new DatagramSocket(port); + } catch (SocketException e) { + throw new IllegalStateException(e); + } + new Thread( + () -> { + try { + while (true) { + final DatagramPacket packet = new DatagramPacket(new byte[65535], 65535); + server.receive(packet); + messagesReceived.add( + new String(packet.getData(), StandardCharsets.UTF_8).trim() + "\n"); + // The sleep duration here is shorter than that used in waitForMessage() at + // 50ms. + // Otherwise sometimes some messages seem to be lost, leading to flaky tests. + Thread.sleep(15L); + } + + } catch (Exception e) { + } + }) + .start(); + } + + public void stop() { + server.close(); + } + + public void waitForMessage() { + while (messagesReceived.isEmpty()) { + try { + Thread.sleep(50L); + } catch (InterruptedException e) { + } + } + } + + public List messagesReceived() { + List out = new ArrayList<>(); + for (String msg : messagesReceived) { + String[] lines = msg.split("\n"); + out.addAll(Arrays.asList(lines)); + } + return out; + } + } + /** * Create a field object with given name and type. * diff --git a/ingestion/src/test/resources/feast/ingestion/transform/WriteRowMetricsDoFnTest.input b/ingestion/src/test/resources/feast/ingestion/transform/WriteRowMetricsDoFnTest.input new file mode 100644 index 00000000000..4d42f5bc4c4 --- /dev/null +++ b/ingestion/src/test/resources/feast/ingestion/transform/WriteRowMetricsDoFnTest.input @@ -0,0 +1,4 @@ +featuresetref,int32,int64,timestamp +project/featureset:1,1,5,2020-03-30T06:10:38Z +project/featureset:1,5,8,2020-03-30T06:10:43Z +project/featureset:1,6,,2020-03-30T06:10:42Z \ No newline at end of file diff --git a/ingestion/src/test/resources/feast/ingestion/transform/WriteRowMetricsDoFnTest.output b/ingestion/src/test/resources/feast/ingestion/transform/WriteRowMetricsDoFnTest.output new file mode 100644 index 00000000000..318ce8eb08b --- /dev/null +++ b/ingestion/src/test/resources/feast/ingestion/transform/WriteRowMetricsDoFnTest.output @@ -0,0 +1,23 @@ +feast_ingestion.feature_row_ingested_count:3|c|#ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_row_lag_ms_min:2000|g|#ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_row_lag_ms_max:7000|g|#ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_row_lag_ms_mean:4000|g|#ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_row_lag_ms_percentile_90:7000|g|#ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_row_lag_ms_percentile_95:7000|g|#ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_row_lag_ms_percentile_99:7000|g|#ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store + +feast_ingestion.feature_value_lag_ms_min:2000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_max:7000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_mean:4000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_percentile_90:7000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_percentile_95:7000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_percentile_99:7000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_missing_count:0|c|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store + +feast_ingestion.feature_value_lag_ms_min:2000|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_max:7000|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_mean:4500|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_percentile_90:7000|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_percentile_95:7000|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_percentile_99:7000|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_missing_count:1|c|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store \ No newline at end of file From 49b7cd1c97a42512bacf4fea8e4f6171021e59c7 Mon Sep 17 00:00:00 2001 From: Joost Rothweiler Date: Fri, 3 Apr 2020 02:33:44 +0200 Subject: [PATCH 107/176] Create project if not exists on applyFeatureSet (#596) * Create project if not exists on apply * Update comments to reflect new situation * Fixed typo in comment Co-authored-by: Joost Rothweiler <=> Co-authored-by: Willem Pienaar <6728866+woop@users.noreply.github.com> --- .../java/feast/core/service/SpecService.java | 13 ++--- .../feast/core/service/SpecServiceTest.java | 53 +++++++++++++++++++ 2 files changed, 57 insertions(+), 9 deletions(-) diff --git a/core/src/main/java/feast/core/service/SpecService.java b/core/src/main/java/feast/core/service/SpecService.java index 5b98d065977..8fec6ac5112 100644 --- a/core/src/main/java/feast/core/service/SpecService.java +++ b/core/src/main/java/feast/core/service/SpecService.java @@ -288,19 +288,14 @@ public ApplyFeatureSetResponse applyFeatureSet(FeatureSetProto.FeatureSet newFea // Validate incoming feature set FeatureSetValidator.validateSpec(newFeatureSet); - // Ensure that the project already exists + // Find project or create new one if it does not exist String project_name = newFeatureSet.getSpec().getProject(); Project project = projectRepository .findById(newFeatureSet.getSpec().getProject()) - .orElseThrow( - () -> - new IllegalArgumentException( - String.format( - "Project name does not exist. Please create a project first: %s", - project_name))); - - // Ensure that the project is not archived + .orElse(new Project(project_name)); + + // Ensure that the project retrieved from repository is not archived if (project.isArchived()) { throw new IllegalArgumentException(String.format("Project is archived: %s", project_name)); } diff --git a/core/src/test/java/feast/core/service/SpecServiceTest.java b/core/src/test/java/feast/core/service/SpecServiceTest.java index 1eb56caac26..43a66135dce 100644 --- a/core/src/test/java/feast/core/service/SpecServiceTest.java +++ b/core/src/test/java/feast/core/service/SpecServiceTest.java @@ -170,6 +170,10 @@ public void setUp() { when(projectRepository.findAllByArchivedIsFalse()) .thenReturn(Collections.singletonList(new Project("project1"))); when(projectRepository.findById("project1")).thenReturn(Optional.of(new Project("project1"))); + Project archivedProject = new Project("archivedproject"); + archivedProject.setArchived(true); + when(projectRepository.findById(archivedProject.getName())) + .thenReturn(Optional.of(archivedProject)); Store store1 = newDummyStore("SERVING"); Store store2 = newDummyStore("WAREHOUSE"); @@ -706,6 +710,55 @@ public void applyFeatureSetShouldUpdateFeatureSetWhenConstraintsAreUpdated() } } + @Test + public void applyFeatureSetShouldCreateProjectWhenNotAlreadyExists() + throws InvalidProtocolBufferException { + Field f3f1 = new Field("f3f1", Enum.INT64); + Field f3f2 = new Field("f3f2", Enum.INT64); + Field f3e1 = new Field("f3e1", Enum.STRING); + FeatureSetProto.FeatureSet incomingFeatureSet = + (new FeatureSet( + "f3", + "newproject", + 5, + 100L, + Arrays.asList(f3e1), + Arrays.asList(f3f2, f3f1), + defaultSource, + FeatureSetStatus.STATUS_READY)) + .toProto(); + + ApplyFeatureSetResponse applyFeatureSetResponse = + specService.applyFeatureSet(incomingFeatureSet); + assertThat(applyFeatureSetResponse.getStatus(), equalTo(Status.CREATED)); + assertThat( + applyFeatureSetResponse.getFeatureSet().getSpec().getProject(), + equalTo(incomingFeatureSet.getSpec().getProject())); + } + + @Test + public void applyFeatureSetShouldFailWhenProjectIsArchived() + throws InvalidProtocolBufferException { + Field f3f1 = new Field("f3f1", Enum.INT64); + Field f3f2 = new Field("f3f2", Enum.INT64); + Field f3e1 = new Field("f3e1", Enum.STRING); + FeatureSetProto.FeatureSet incomingFeatureSet = + (new FeatureSet( + "f3", + "archivedproject", + 5, + 100L, + Arrays.asList(f3e1), + Arrays.asList(f3f2, f3f1), + defaultSource, + FeatureSetStatus.STATUS_READY)) + .toProto(); + + expectedException.expect(IllegalArgumentException.class); + expectedException.expectMessage("Project is archived"); + specService.applyFeatureSet(incomingFeatureSet); + } + @Test public void shouldUpdateStoreIfConfigChanges() throws InvalidProtocolBufferException { when(storeRepository.findById("SERVING")).thenReturn(Optional.of(stores.get(0))); From 1decc2a1d9cc9584eb16d73eee0d6b2e6593a42e Mon Sep 17 00:00:00 2001 From: Julio Anthony Leonard Date: Fri, 3 Apr 2020 12:45:44 +0700 Subject: [PATCH 108/176] Pin Jupyter Notebook version (#597) * Pin jupyter notebook image to 63d0df23b673 for panda 0.25.3 backward compatibility * Explicitly set python interpreter used when using pip inside jupyter --- examples/basic/basic.ipynb | 2 +- infra/docker-compose/docker-compose.yml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/examples/basic/basic.ipynb b/examples/basic/basic.ipynb index b9893011d97..a6feb0ef13a 100644 --- a/examples/basic/basic.ipynb +++ b/examples/basic/basic.ipynb @@ -67,7 +67,7 @@ "metadata": {}, "outputs": [], "source": [ - "!pip install --ignore-installed --upgrade feast" + "!python -m pip install --ignore-installed --upgrade feast" ] }, { diff --git a/infra/docker-compose/docker-compose.yml b/infra/docker-compose/docker-compose.yml index a796e5fa44e..38234cff22d 100644 --- a/infra/docker-compose/docker-compose.yml +++ b/infra/docker-compose/docker-compose.yml @@ -62,7 +62,7 @@ services: - "--spring.config.location=classpath:/application.yml,file:/etc/feast/application.yml" jupyter: - image: jupyter/datascience-notebook:latest + image: jupyter/datascience-notebook:63d0df23b673 volumes: - ../../:/home/jovyan/feast - ./gcp-service-accounts/${FEAST_JUPYTER_GCP_SERVICE_ACCOUNT_KEY}:/etc/gcloud/service-accounts/key.json From 9917b630046d5da10709dac18e3888f628862386 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Sat, 4 Apr 2020 13:17:44 +0800 Subject: [PATCH 109/176] Fix doc building (#603) * Fix documentation building * Add comments to documentation building * Test protoc build * Add protoc building * Add path to protoc building * Add path to protoc building * Build Python Protos --- Makefile | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/Makefile b/Makefile index 51fcab76da6..e780864ddca 100644 --- a/Makefile +++ b/Makefile @@ -141,7 +141,8 @@ install-dependencies-proto-docs: mv protoc3/include/* $$HOME/include compile-protos-docs: - cd ${ROOT_DIR}/protos; protoc --docs_out=../dist/grpc feast/*/*.proto + cd ${ROOT_DIR}/protos; protoc --docs_out=../dist/grpc feast/*/*.proto || \ + cd ${ROOT_DIR}; $(MAKE) install-dependencies-proto-docs && cd ${ROOT_DIR}/protos; PATH=$$HOME/bin:$$PATH protoc -I $$HOME/include/ -I . --docs_out=../dist/grpc feast/*/*.proto clean-html: rm -rf $(ROOT_DIR)/dist @@ -149,6 +150,11 @@ clean-html: build-html: clean-html mkdir -p $(ROOT_DIR)/dist/python mkdir -p $(ROOT_DIR)/dist/grpc - cd $(ROOT_DIR)/protos && $(MAKE) gen-docs + + # Build Protobuf documentation + $(MAKE) compile-protos-docs + + # Build Python SDK documentation + $(MAKE) compile-protos-python cd $(ROOT_DIR)/sdk/python/docs && $(MAKE) html - cp -r $(ROOT_DIR)/sdk/python/docs/html/* $(ROOT_DIR)/dist/python + cp -r $(ROOT_DIR)/sdk/python/docs/html/* $(ROOT_DIR)/dist/python \ No newline at end of file From 9139fe317eea713b2d6bdb83334483be3c2188ef Mon Sep 17 00:00:00 2001 From: Zhu Zhan Yan Date: Tue, 7 Apr 2020 15:00:45 +0800 Subject: [PATCH 110/176] Add Ingestion Job management API for Feast Core (#548) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Added protobuf definitions for a Job Management API * Added query methods to JobRepository to query jobs by store and featureset * Added hashCode() & equals() to Job model compare and hash jobs This would allow jobs to be used as elements in HashSets and keys in HashMaps * Added toIngestionProto() to Job object to convert Job model to ingestion job proto * Added query methods to FeatureSetRepository to query by exact (name, project) or (name,version) * Added listJobs() to JobService to handle request list ingestion jobs requests * Added missing filter field in ListIngestionJobsRequest protobuf * Added code to setup mockups for testing JobService * Revert "Added hashCode() & equals() to Job model compare and hash jobs" This reverts commit ba995bb712cbd38f0f4bef47efbe433d5ec07521 as it caused core tests to fail * Added JobServiceTest unit tests for JobService's listJobs() * Added stopJobs() to JobService to handle requests to stop jobs * Added getTransitionalStates() to JobStatus to return collection of transitional states * Changed stopJobs() to throw unsupported error on transitional job statuses * Moved conversion of JobStatus to IngestionJobStatus proto to JobStatus. * Make findFeatureSet() match only one featureset. Limit findFeatureSets() as feature set references as composite keys should match one and only one featureset. * Added matchFeatureSets() to SpecService to match Feature Sets from References This commit is adds temporary support for FeatureSetReference to SpecService via listFeatureSets(). In the future, this should merged together with listFeatureSets() as their functionality is almost the same. * Refactor JobService listJobs() to use SpecService to provide featureset matching * Revert findBy methods added to FeatureSetRepository as no longer used. JobService no longer depends on FeatureSetRepository directly, instead via SpecService * Added restartJob() to JobManagers to restart ingestion/import jobs * Added restartJob() to JobService to restart ingestion jobs * Update job model (due to new extId) when restartJob() in JobService * Use assertThat() & equalTo() instead of assertEquals() in JobService * Revert "Use assertThat() & equalTo() instead of assertEquals() in JobService" due to failed tests This reverts commit bb3cf0ae1e7cfbde7a2f997cacc661d040c0a35f. * Use hamcrest’s assertThat() and equalTo() instead of Junit's assertEquals * Hook up JobService list, stop, restart job methods to CoreServiceImpl. * Throw InvalidArgumentException instead when calling listJobs() with invalid FeatureSetReference Throw InvalidArgumentException instead of UnsupportedOperationException for better semantics: UnsupportedOperationException should be reserved for operations that fail due to failed preconditions * Fixed typos in javadocs: InvalidArgumentException should be IllegalArgumentException * Fixed in findByFeatureSetsIn() query in JobRepository * Renamed toIngestionProto() methods to toProto() to follow code convention * Make JobService's listJobs() a transaction to prevent DB data race conditions * Moved matchFeatureSets() in SpecService to private method in JobService * Make the map that maps between JobStatus and IngestionJobStatus static * Make JobService's listJobs() to return all ingestion jobs on empty filter * Fixed issue where the jobManager map that JobService built used wrong keys * Fix issue where actual Job Status is not synced with database. Issue occurs when the job is aborted/restarted, but the JobStatus has not yet been updated by JobUpdateTask. Hence another call to abortJob() & restartJob() that should be rejected due invalid status is allowed through * Log stopJob() & restartJob() operations to make debugging easier * Use Runner.name() instead of runner.toString() to build JobManager map * Move documentation on JobService operations to CoreService protobuf definition * Added IngestJob to python sdk as native representation of IngestionJob proto * Make empty filter on JobService's listJobs() select all ingestion jobs * Added bindings for Job management API to python sdk client. * Fixed __connect_core() to connect to Feast CoreService when calling on Job API calls * Auto reload IngestJob.status and IngestJob.external_id on get property. * Added IngestJob.wait() to wait for job status to transtion * Added basic job api e2e test to exercise job api * Reorder the operations e2etest to make sure that jobs are running after test * Added e2e test for all types exercising job api * Fixed typo in function arguments * Added unit tests for Ingestion Job API additions in python sdk * Rename "ingestion" to "ingest" for more consistent naming * Disable support for restarting Job in a terminal state due to possible race conditions * Added __str__ and __repr__ to IngestJob to render ingestjob in human readable string * Added FeatureSetRef to represent references to featursets * Admend client's list_ingest_jobs() to accept feature references directly * Fixed typo in IngestJob.store property * Fixed issue with FeatureSetRef.from_str not converting version to int * Make the grpc error message more apparent on stop_ingest_job() and restart_ingest_job() * Added feast ingest-job list, describe, stop, restart to CLI * Rename Job to RetrievalJob to prevent confusion with IngestJob * Updated e2e tests to use FeatureSetRef in list_ingest_jobs() * Fixed due e2e tests to cater to new limitations on stop_ingest_job() * Increase timeout on test_all_types_ingest_jobs() e2e test. * Configure IngestJob.wait() to backoff with a exponentially larger wait duration * Added print statements to debug e2e test failure. * Revert "Added print statements to debug e2e test failure." This reverts commit 146fb2bc327c427167b133dbae5742ed39e3477e. * Fixed issue of test waiting for aborted job to become running causing timeout. Co-authored-by: Zhu Zhanyan --- .../java/feast/core/dao/JobRepository.java | 7 + .../java/feast/core/grpc/CoreServiceImpl.java | 81 +++- .../main/java/feast/core/job/JobManager.java | 10 + .../core/job/dataflow/DataflowJobManager.java | 20 + .../job/direct/DirectRunnerJobManager.java | 20 + core/src/main/java/feast/core/model/Job.java | 45 +- .../main/java/feast/core/model/JobStatus.java | 37 ++ .../java/feast/core/service/JobService.java | 284 +++++++++++ .../feast/core/service/JobServiceTest.java | 444 ++++++++++++++++++ protos/feast/core/CoreService.proto | 60 ++- protos/feast/core/FeatureSetReference.proto | 33 ++ protos/feast/core/IngestionJob.proto | 65 +++ sdk/python/feast/cli.py | 117 ++++- sdk/python/feast/client.py | 83 +++- sdk/python/feast/feature_set.py | 110 ++++- sdk/python/feast/job.py | 118 ++++- sdk/python/tests/test_client.py | 113 ++++- sdk/python/tests/test_feature_set.py | 19 +- tests/e2e/basic-ingest-redis-serving.py | 48 +- 19 files changed, 1684 insertions(+), 30 deletions(-) create mode 100644 core/src/main/java/feast/core/service/JobService.java create mode 100644 core/src/test/java/feast/core/service/JobServiceTest.java create mode 100644 protos/feast/core/FeatureSetReference.proto create mode 100644 protos/feast/core/IngestionJob.proto diff --git a/core/src/main/java/feast/core/dao/JobRepository.java b/core/src/main/java/feast/core/dao/JobRepository.java index 98da76912e7..c61f3eacc05 100644 --- a/core/src/main/java/feast/core/dao/JobRepository.java +++ b/core/src/main/java/feast/core/dao/JobRepository.java @@ -16,6 +16,7 @@ */ package feast.core.dao; +import feast.core.model.FeatureSet; import feast.core.model.Job; import feast.core.model.JobStatus; import java.util.Collection; @@ -29,4 +30,10 @@ public interface JobRepository extends JpaRepository { List findByStatusNotIn(Collection statuses); List findBySourceIdAndStoreNameOrderByLastUpdatedDesc(String sourceId, String storeName); + + // find jobs by feast store name + List findByStoreName(String storeName); + + // find jobs by featureset + List findByFeatureSetsIn(List featureSets); } diff --git a/core/src/main/java/feast/core/grpc/CoreServiceImpl.java b/core/src/main/java/feast/core/grpc/CoreServiceImpl.java index 661bbe24039..42bc0ba23de 100644 --- a/core/src/main/java/feast/core/grpc/CoreServiceImpl.java +++ b/core/src/main/java/feast/core/grpc/CoreServiceImpl.java @@ -16,6 +16,7 @@ */ package feast.core.grpc; +import com.google.api.gax.rpc.InvalidArgumentException; import com.google.protobuf.InvalidProtocolBufferException; import feast.core.CoreServiceGrpc.CoreServiceImplBase; import feast.core.CoreServiceProto.ApplyFeatureSetRequest; @@ -30,21 +31,29 @@ import feast.core.CoreServiceProto.GetFeatureSetResponse; import feast.core.CoreServiceProto.ListFeatureSetsRequest; import feast.core.CoreServiceProto.ListFeatureSetsResponse; +import feast.core.CoreServiceProto.ListIngestionJobsRequest; +import feast.core.CoreServiceProto.ListIngestionJobsResponse; import feast.core.CoreServiceProto.ListProjectsRequest; import feast.core.CoreServiceProto.ListProjectsResponse; import feast.core.CoreServiceProto.ListStoresRequest; import feast.core.CoreServiceProto.ListStoresResponse; +import feast.core.CoreServiceProto.RestartIngestionJobRequest; +import feast.core.CoreServiceProto.RestartIngestionJobResponse; +import feast.core.CoreServiceProto.StopIngestionJobRequest; +import feast.core.CoreServiceProto.StopIngestionJobResponse; import feast.core.CoreServiceProto.UpdateStoreRequest; import feast.core.CoreServiceProto.UpdateStoreResponse; import feast.core.exception.RetrievalException; import feast.core.grpc.interceptors.MonitoringInterceptor; import feast.core.model.Project; import feast.core.service.AccessManagementService; +import feast.core.service.JobService; import feast.core.service.SpecService; import io.grpc.Status; import io.grpc.StatusRuntimeException; import io.grpc.stub.StreamObserver; import java.util.List; +import java.util.NoSuchElementException; import java.util.stream.Collectors; import lombok.extern.slf4j.Slf4j; import org.lognet.springboot.grpc.GRpcService; @@ -57,11 +66,16 @@ public class CoreServiceImpl extends CoreServiceImplBase { private SpecService specService; private AccessManagementService accessManagementService; + private JobService jobService; @Autowired - public CoreServiceImpl(SpecService specService, AccessManagementService accessManagementService) { + public CoreServiceImpl( + SpecService specService, + AccessManagementService accessManagementService, + JobService jobService) { this.specService = specService; this.accessManagementService = accessManagementService; + this.jobService = jobService; } @Override @@ -192,4 +206,69 @@ public void listProjects( Status.INTERNAL.withDescription(e.getMessage()).withCause(e).asRuntimeException()); } } + + @Override + public void listIngestionJobs( + ListIngestionJobsRequest request, + StreamObserver responseObserver) { + try { + ListIngestionJobsResponse response = this.jobService.listJobs(request); + responseObserver.onNext(response); + responseObserver.onCompleted(); + } catch (InvalidArgumentException e) { + log.error("Recieved an invalid request on calling listIngestionJobs method:", e); + responseObserver.onError( + Status.INVALID_ARGUMENT.withDescription(e.getMessage()).withCause(e).asException()); + } catch (Exception e) { + log.error("Unexpected exception on calling listIngestionJobs method:", e); + responseObserver.onError( + Status.INTERNAL.withDescription(e.getMessage()).withCause(e).asRuntimeException()); + } + } + + @Override + public void restartIngestionJob( + RestartIngestionJobRequest request, + StreamObserver responseObserver) { + try { + RestartIngestionJobResponse response = this.jobService.restartJob(request); + responseObserver.onNext(response); + responseObserver.onCompleted(); + } catch (NoSuchElementException e) { + log.error( + "Attempted to restart an nonexistent job on calling restartIngestionJob method:", e); + responseObserver.onError( + Status.NOT_FOUND.withDescription(e.getMessage()).withCause(e).asException()); + } catch (UnsupportedOperationException e) { + log.error("Recieved an unsupported request on calling restartIngestionJob method:", e); + responseObserver.onError( + Status.FAILED_PRECONDITION.withDescription(e.getMessage()).withCause(e).asException()); + } catch (Exception e) { + log.error("Unexpected exception on calling restartIngestionJob method:", e); + responseObserver.onError( + Status.INTERNAL.withDescription(e.getMessage()).withCause(e).asRuntimeException()); + } + } + + @Override + public void stopIngestionJob( + StopIngestionJobRequest request, StreamObserver responseObserver) { + try { + StopIngestionJobResponse response = this.jobService.stopJob(request); + responseObserver.onNext(response); + responseObserver.onCompleted(); + } catch (NoSuchElementException e) { + log.error("Attempted to stop an nonexistent job on calling stopIngestionJob method:", e); + responseObserver.onError( + Status.NOT_FOUND.withDescription(e.getMessage()).withCause(e).asException()); + } catch (UnsupportedOperationException e) { + log.error("Recieved an unsupported request on calling stopIngestionJob method:", e); + responseObserver.onError( + Status.FAILED_PRECONDITION.withDescription(e.getMessage()).withCause(e).asException()); + } catch (Exception e) { + log.error("Unexpected exception on calling stopIngestionJob method:", e); + responseObserver.onError( + Status.INTERNAL.withDescription(e.getMessage()).withCause(e).asRuntimeException()); + } + } } diff --git a/core/src/main/java/feast/core/job/JobManager.java b/core/src/main/java/feast/core/job/JobManager.java index 99880cdb764..eda211b7574 100644 --- a/core/src/main/java/feast/core/job/JobManager.java +++ b/core/src/main/java/feast/core/job/JobManager.java @@ -51,6 +51,16 @@ public interface JobManager { */ void abortJob(String extId); + /** + * Restart an job. If job is an terminated state, will simply start the job. Might cause data to + * be lost during when restarting running jobs in some implementations. Refer to on docs the + * specific implementation. + * + * @param job job to restart + * @return the restarted job + */ + Job restartJob(Job job); + /** * Get status of a job given runner-specific job ID. * diff --git a/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java b/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java index f4df3d352a9..c2313d75ecc 100644 --- a/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java +++ b/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java @@ -152,6 +152,26 @@ public void abortJob(String dataflowJobId) { } } + /** + * Restart a restart dataflow job. Dataflow should ensure continuity between during the restart, + * so no data should be lost during the restart operation. + * + * @param job job to restart + * @return the restarted job + */ + @Override + public Job restartJob(Job job) { + JobStatus status = job.getStatus(); + if (JobStatus.getTerminalState().contains(status)) { + // job yet not running: just start job + return this.startJob(job); + } else { + // job is running - updating the job without changing the job has + // the effect of restarting the job + return this.updateJob(job); + } + } + /** * Get status of a dataflow job with given id and try to map it into Feast's JobStatus. * diff --git a/core/src/main/java/feast/core/job/direct/DirectRunnerJobManager.java b/core/src/main/java/feast/core/job/direct/DirectRunnerJobManager.java index 08aeed1cc3a..9b3a8473e47 100644 --- a/core/src/main/java/feast/core/job/direct/DirectRunnerJobManager.java +++ b/core/src/main/java/feast/core/job/direct/DirectRunnerJobManager.java @@ -157,6 +157,26 @@ public PipelineResult runPipeline(ImportOptions pipelineOptions) throws IOExcept return ImportJob.runPipeline(pipelineOptions); } + /** + * Restart a direct runner job. Note that some data will be temporarily lost during when + * restarting running direct runner jobs. See {#link {@link #updateJob(Job)} for more info. + * + * @param job job to restart + * @return the restarted job + */ + @Override + public Job restartJob(Job job) { + JobStatus status = job.getStatus(); + if (JobStatus.getTerminalState().contains(status)) { + // job yet not running: just start job + return this.startJob(job); + } else { + // job is running - updating the job without changing the job has + // the effect of restarting the job. + return this.updateJob(job); + } + } + /** * Gets the state of the direct runner job. Direct runner jobs only have 2 states: RUNNING and * ABORTED. diff --git a/core/src/main/java/feast/core/model/Job.java b/core/src/main/java/feast/core/model/Job.java index 377f5f70956..738a16db2d1 100644 --- a/core/src/main/java/feast/core/model/Job.java +++ b/core/src/main/java/feast/core/model/Job.java @@ -16,8 +16,24 @@ */ package feast.core.model; +import com.google.protobuf.InvalidProtocolBufferException; +import feast.core.FeatureSetProto; +import feast.core.IngestionJobProto; +import java.util.ArrayList; import java.util.List; -import javax.persistence.*; +import javax.persistence.CascadeType; +import javax.persistence.Column; +import javax.persistence.Entity; +import javax.persistence.EnumType; +import javax.persistence.Enumerated; +import javax.persistence.Id; +import javax.persistence.Index; +import javax.persistence.JoinColumn; +import javax.persistence.JoinTable; +import javax.persistence.ManyToMany; +import javax.persistence.ManyToOne; +import javax.persistence.OneToMany; +import javax.persistence.Table; import lombok.AllArgsConstructor; import lombok.Getter; import lombok.Setter; @@ -102,4 +118,31 @@ public void updateMetrics(List newMetrics) { public String getSinkName() { return store.getName(); } + + /** + * Convert a job model to ingestion job proto + * + * @return Ingestion Job proto derieved from the given job + */ + public IngestionJobProto.IngestionJob toProto() throws InvalidProtocolBufferException { + + // convert featuresets of job to protos + List featureSetProtos = new ArrayList<>(); + for (FeatureSet featureSet : this.getFeatureSets()) { + featureSetProtos.add(featureSet.toProto()); + } + + // build ingestion job proto with job data + IngestionJobProto.IngestionJob ingestJob = + IngestionJobProto.IngestionJob.newBuilder() + .setId(this.getId()) + .setExternalId(this.getExtId()) + .setStatus(this.getStatus().toProto()) + .addAllFeatureSets(featureSetProtos) + .setSource(this.getSource().toProto()) + .setStore(this.getStore().toProto()) + .build(); + + return ingestJob; + } } diff --git a/core/src/main/java/feast/core/model/JobStatus.java b/core/src/main/java/feast/core/model/JobStatus.java index 123b57a21b1..86aa512933c 100644 --- a/core/src/main/java/feast/core/model/JobStatus.java +++ b/core/src/main/java/feast/core/model/JobStatus.java @@ -16,9 +16,11 @@ */ package feast.core.model; +import feast.core.IngestionJobProto.IngestionJobStatus; import java.util.Arrays; import java.util.Collection; import java.util.Collections; +import java.util.Map; public enum JobStatus { /** Job status is not known. */ @@ -64,4 +66,39 @@ public enum JobStatus { public static Collection getTerminalState() { return TERMINAL_STATE; } + + private static final Collection TRANSITIONAL_STATES = + Collections.unmodifiableList(Arrays.asList(PENDING, ABORTING, SUSPENDING)); + + /** + * Get Transitional Job Status states. Transitionals states are assigned to jobs that + * transitioning to a more stable state (ie SUSPENDED, ABORTED etc.) + * + * @return Collection of transitional Job Status states. + */ + public static final Collection getTransitionalStates() { + return TRANSITIONAL_STATES; + } + + private static final Map INGESTION_JOB_STATUS_MAP = + Map.of( + JobStatus.UNKNOWN, IngestionJobStatus.UNKNOWN, + JobStatus.PENDING, IngestionJobStatus.PENDING, + JobStatus.RUNNING, IngestionJobStatus.RUNNING, + JobStatus.COMPLETED, IngestionJobStatus.COMPLETED, + JobStatus.ABORTING, IngestionJobStatus.ABORTING, + JobStatus.ABORTED, IngestionJobStatus.ABORTED, + JobStatus.ERROR, IngestionJobStatus.ERROR, + JobStatus.SUSPENDING, IngestionJobStatus.SUSPENDING, + JobStatus.SUSPENDED, IngestionJobStatus.SUSPENDED); + + /** + * Convert a Job Status to Ingestion Job Status proto + * + * @return IngestionJobStatus proto derieved from this job status + */ + public IngestionJobStatus toProto() { + // maps job models job status to ingestion job status + return INGESTION_JOB_STATUS_MAP.get(this); + } } diff --git a/core/src/main/java/feast/core/service/JobService.java b/core/src/main/java/feast/core/service/JobService.java new file mode 100644 index 00000000000..bf74b90e80c --- /dev/null +++ b/core/src/main/java/feast/core/service/JobService.java @@ -0,0 +1,284 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.core.service; + +import com.google.protobuf.InvalidProtocolBufferException; +import feast.core.CoreServiceProto.ListFeatureSetsRequest; +import feast.core.CoreServiceProto.ListFeatureSetsResponse; +import feast.core.CoreServiceProto.ListIngestionJobsRequest; +import feast.core.CoreServiceProto.ListIngestionJobsResponse; +import feast.core.CoreServiceProto.RestartIngestionJobRequest; +import feast.core.CoreServiceProto.RestartIngestionJobResponse; +import feast.core.CoreServiceProto.StopIngestionJobRequest; +import feast.core.CoreServiceProto.StopIngestionJobResponse; +import feast.core.FeatureSetReferenceProto.FeatureSetReference; +import feast.core.IngestionJobProto; +import feast.core.dao.JobRepository; +import feast.core.job.JobManager; +import feast.core.log.Action; +import feast.core.log.AuditLogger; +import feast.core.log.Resource; +import feast.core.model.FeatureSet; +import feast.core.model.Job; +import feast.core.model.JobStatus; +import java.util.ArrayList; +import java.util.Collection; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.NoSuchElementException; +import java.util.Optional; +import java.util.Set; +import java.util.stream.Collectors; +import lombok.extern.slf4j.Slf4j; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.stereotype.Service; +import org.springframework.transaction.annotation.Transactional; + +/** Defines a Job Managemenent Service that allows users to manage feast ingestion jobs. */ +@Slf4j +@Service +public class JobService { + private JobRepository jobRepository; + private SpecService specService; + private Map jobManagers; + + @Autowired + public JobService( + JobRepository jobRepository, SpecService specService, List jobManagerList) { + this.jobRepository = jobRepository; + this.specService = specService; + + this.jobManagers = new HashMap<>(); + for (JobManager manager : jobManagerList) { + this.jobManagers.put(manager.getRunnerType().name(), manager); + } + } + + /* Job Service API */ + /** + * List Ingestion Jobs in feast matching the given request. See CoreService protobuf documentation + * for more detailed documentation. + * + * @param request list ingestion jobs request specifying which jobs to include + * @throws IllegalArgumentException when given filter in a unsupported configuration + * @throws InvalidProtocolBufferException on error when constructing response protobuf + * @return list ingestion jobs response + */ + @Transactional(readOnly = true) + public ListIngestionJobsResponse listJobs(ListIngestionJobsRequest request) + throws InvalidProtocolBufferException { + Set matchingJobIds = new HashSet<>(); + + // check that filter specified and not empty + if (request.hasFilter() + && !(request.getFilter().getId() == "" + && request.getFilter().getStoreName() == "" + && request.getFilter().hasFeatureSetReference() == false)) { + // filter jobs based on request filter + ListIngestionJobsRequest.Filter filter = request.getFilter(); + + // for proto3, default value for missing values: + // - numeric values (ie int) is zero + // - strings is empty string + if (filter.getId() != "") { + // get by id: no more filters required: found job + Optional job = this.jobRepository.findById(filter.getId()); + if (job.isPresent()) { + matchingJobIds.add(filter.getId()); + } + } else { + // multiple filters can apply together in an 'and' operation + if (filter.getStoreName() != "") { + // find jobs by name + List jobs = this.jobRepository.findByStoreName(filter.getStoreName()); + Set jobIds = jobs.stream().map(Job::getId).collect(Collectors.toSet()); + matchingJobIds = this.mergeResults(matchingJobIds, jobIds); + } + if (filter.hasFeatureSetReference()) { + // find a matching featuresets for reference + FeatureSetReference fsReference = filter.getFeatureSetReference(); + ListFeatureSetsResponse response = + this.specService.listFeatureSets(this.toListFeatureSetFilter(fsReference)); + List featureSets = + response.getFeatureSetsList().stream() + .map(FeatureSet::fromProto) + .collect(Collectors.toList()); + + // find jobs for the matching featuresets + Collection matchingJobs = this.jobRepository.findByFeatureSetsIn(featureSets); + List jobIds = matchingJobs.stream().map(Job::getId).collect(Collectors.toList()); + matchingJobIds = this.mergeResults(matchingJobIds, jobIds); + } + } + } else { + // no or empty filter: match all jobs + matchingJobIds = + this.jobRepository.findAll().stream().map(Job::getId).collect(Collectors.toSet()); + } + + // convert matching job models to ingestion job protos + List ingestJobs = new ArrayList<>(); + for (String jobId : matchingJobIds) { + Job job = this.jobRepository.findById(jobId).get(); + ingestJobs.add(job.toProto()); + } + + // pack jobs into response + return ListIngestionJobsResponse.newBuilder().addAllJobs(ingestJobs).build(); + } + + /** + * Restart (Aborts) the ingestion job matching the given restart request. See CoreService protobuf + * documentation for more detailed documentation. + * + * @param request restart ingestion job request specifying which job to stop + * @throws NoSuchElementException when restart job request requests to restart a nonexistent job. + * @throws UnsupportedOperationException when job to be restarted is in an unsupported status + * @throws InvalidProtocolBufferException on error when constructing response protobuf + */ + @Transactional + public RestartIngestionJobResponse restartJob(RestartIngestionJobRequest request) + throws InvalidProtocolBufferException { + // check job exists + Optional getJob = this.jobRepository.findById(request.getId()); + if (getJob.isEmpty()) { + throw new NoSuchElementException( + "Attempted to stop nonexistent job with id: " + getJob.get().getId()); + } + + // check job status is valid for restarting + Job job = getJob.get(); + JobStatus status = job.getStatus(); + if (JobStatus.getTransitionalStates().contains(status) + || JobStatus.getTerminalState().contains(status) + || status.equals(JobStatus.UNKNOWN)) { + throw new UnsupportedOperationException( + "Restarting a job with a transitional, terminal or unknown status is unsupported"); + } + + // restart job with job manager + JobManager jobManager = this.jobManagers.get(job.getRunner()); + job = jobManager.restartJob(job); + log.info( + String.format( + "Restarted job (id: %s, extId: %s runner: %s)", + job.getId(), job.getExtId(), job.getRunner())); + // sync job status & update job model in job repository + job = this.syncJobStatus(jobManager, job); + this.jobRepository.saveAndFlush(job); + + return RestartIngestionJobResponse.newBuilder().build(); + } + + /** + * Stops (Aborts) the ingestion job matching the given stop request. See CoreService protobuf + * documentation for more detailed documentation. + * + * @param request stop ingestion job request specifying which job to stop + * @throws NoSuchElementException when stop job request requests to stop a nonexistent job. + * @throws UnsupportedOperationException when job to be stopped is in an unsupported status + * @throws InvalidProtocolBufferException on error when constructing response protobuf + */ + @Transactional + public StopIngestionJobResponse stopJob(StopIngestionJobRequest request) + throws InvalidProtocolBufferException { + // check job exists + Optional getJob = this.jobRepository.findById(request.getId()); + if (getJob.isEmpty()) { + throw new NoSuchElementException( + "Attempted to stop nonexistent job with id: " + getJob.get().getId()); + } + + // check job status is valid for stopping + Job job = getJob.get(); + JobStatus status = job.getStatus(); + if (JobStatus.getTerminalState().contains(status)) { + // do nothing - job is already stopped + return StopIngestionJobResponse.newBuilder().build(); + } else if (JobStatus.getTransitionalStates().contains(status) + || status.equals(JobStatus.UNKNOWN)) { + throw new UnsupportedOperationException( + "Stopping a job with a transitional or unknown status is unsupported"); + } + + // stop job with job manager + JobManager jobManager = this.jobManagers.get(job.getRunner()); + jobManager.abortJob(job.getExtId()); + log.info( + String.format( + "Restarted job (id: %s, extId: %s runner: %s)", + job.getId(), job.getExtId(), job.getRunner())); + + // sync job status & update job model in job repository + job = this.syncJobStatus(jobManager, job); + this.jobRepository.saveAndFlush(job); + + return StopIngestionJobResponse.newBuilder().build(); + } + + /* Private Utility Methods */ + private Set mergeResults(Set results, Collection newResults) { + if (results.size() <= 0) { + // no existing results: copy over new results + results.addAll(newResults); + } else { + // and operation: keep results that exist in both existing and new results + results.retainAll(newResults); + } + return results; + } + + // converts feature set reference to a list feature set filter + private ListFeatureSetsRequest.Filter toListFeatureSetFilter(FeatureSetReference fsReference) { + // match featuresets using contents of featureset reference + String fsName = fsReference.getName(); + String fsProject = fsReference.getProject(); + Integer fsVersion = fsReference.getVersion(); + + // construct list featureset request filter using feature set reference + // for proto3, default value for missing values: + // - numeric values (ie int) is zero + // - strings is empty string + ListFeatureSetsRequest.Filter filter = + ListFeatureSetsRequest.Filter.newBuilder() + .setFeatureSetName((fsName != "") ? fsName : "*") + .setProject((fsProject != "") ? fsProject : "*") + .setFeatureSetVersion((fsVersion != 0) ? fsVersion.toString() : "*") + .build(); + + return filter; + } + + // sync job status using job manager + private Job syncJobStatus(JobManager jobManager, Job job) { + JobStatus newStatus = jobManager.getJobStatus(job); + // log job status transition + if (newStatus != job.getStatus()) { + AuditLogger.log( + Resource.JOB, + job.getId(), + Action.STATUS_CHANGE, + "Job status transition: changed from %s to %s", + job.getStatus(), + newStatus); + job.setStatus(newStatus); + } + return job; + } +} diff --git a/core/src/test/java/feast/core/service/JobServiceTest.java b/core/src/test/java/feast/core/service/JobServiceTest.java new file mode 100644 index 00000000000..c0e90ca43f4 --- /dev/null +++ b/core/src/test/java/feast/core/service/JobServiceTest.java @@ -0,0 +1,444 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.core.service; + +import static org.hamcrest.CoreMatchers.equalTo; +import static org.hamcrest.MatcherAssert.assertThat; +import static org.junit.Assert.fail; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; +import static org.mockito.Mockito.when; +import static org.mockito.MockitoAnnotations.initMocks; + +import com.google.protobuf.InvalidProtocolBufferException; +import feast.core.CoreServiceProto.ListFeatureSetsRequest; +import feast.core.CoreServiceProto.ListFeatureSetsResponse; +import feast.core.CoreServiceProto.ListIngestionJobsRequest; +import feast.core.CoreServiceProto.ListIngestionJobsResponse; +import feast.core.CoreServiceProto.RestartIngestionJobRequest; +import feast.core.CoreServiceProto.RestartIngestionJobResponse; +import feast.core.CoreServiceProto.StopIngestionJobRequest; +import feast.core.CoreServiceProto.StopIngestionJobResponse; +import feast.core.FeatureSetProto.FeatureSetStatus; +import feast.core.FeatureSetReferenceProto.FeatureSetReference; +import feast.core.IngestionJobProto.IngestionJob; +import feast.core.SourceProto.KafkaSourceConfig; +import feast.core.SourceProto.SourceType; +import feast.core.StoreProto.Store.RedisConfig; +import feast.core.StoreProto.Store.StoreType; +import feast.core.dao.JobRepository; +import feast.core.job.JobManager; +import feast.core.job.Runner; +import feast.core.model.FeatureSet; +import feast.core.model.Field; +import feast.core.model.Job; +import feast.core.model.JobStatus; +import feast.core.model.Source; +import feast.core.model.Store; +import feast.types.ValueProto.ValueType.Enum; +import java.time.Instant; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Date; +import java.util.List; +import java.util.Optional; +import org.junit.Before; +import org.junit.Test; +import org.mockito.Mock; + +public class JobServiceTest { + // mocks + @Mock private JobRepository jobRepository; + @Mock private JobManager jobManager; + @Mock private SpecService specService; + // fake models + private Source dataSource; + private Store dataStore; + private FeatureSet featureSet; + private List fsReferences; + private List listFilters; + private Job job; + private IngestionJob ingestionJob; + // test target + public JobService jobService; + + /* unit test setup */ + @Before + public void setup() { + initMocks(this); + + // create mock objects for testing + // fake data source + this.dataSource = + new Source( + SourceType.KAFKA, + KafkaSourceConfig.newBuilder() + .setBootstrapServers("kafka:9092") + .setTopic("my-topic") + .build(), + true); + // fake data store + this.dataStore = + new Store( + "feast-redis", + StoreType.REDIS.toString(), + RedisConfig.newBuilder().setPort(6379).build().toByteArray(), + "*:*:*"); + + // fake featureset & job + this.featureSet = this.newDummyFeatureSet("food", 2, "hunger"); + this.job = this.newDummyJob("kafka-to-redis", "job-1111", JobStatus.PENDING); + try { + this.ingestionJob = this.job.toProto(); + } catch (InvalidProtocolBufferException e) { + e.printStackTrace(); + } + + this.fsReferences = this.newDummyFeatureSetReferences(); + this.listFilters = this.newDummyListRequestFilters(); + + // setup mock objects + this.setupSpecService(); + this.setupJobRepository(); + this.setupJobManager(); + + // create test target + this.jobService = + new JobService(this.jobRepository, this.specService, Arrays.asList(this.jobManager)); + } + + // setup fake spec service + public void setupSpecService() { + try { + ListFeatureSetsResponse response = + ListFeatureSetsResponse.newBuilder().addFeatureSets(this.featureSet.toProto()).build(); + + when(this.specService.listFeatureSets(this.listFilters.get(0))).thenReturn(response); + + when(this.specService.listFeatureSets(this.listFilters.get(1))).thenReturn(response); + + when(this.specService.listFeatureSets(this.listFilters.get(2))).thenReturn(response); + } catch (InvalidProtocolBufferException e) { + e.printStackTrace(); + fail("Unexpected exception"); + } + } + + // setup fake job repository + public void setupJobRepository() { + when(this.jobRepository.findById(this.job.getId())).thenReturn(Optional.of(this.job)); + when(this.jobRepository.findByStoreName(this.dataStore.getName())) + .thenReturn(Arrays.asList(this.job)); + when(this.jobRepository.findByFeatureSetsIn(Arrays.asList(this.featureSet))) + .thenReturn(Arrays.asList(this.job)); + when(this.jobRepository.findAll()).thenReturn(Arrays.asList(this.job)); + } + + // TODO: setup fake job manager + public void setupJobManager() { + when(this.jobManager.getRunnerType()).thenReturn(Runner.DATAFLOW); + when(this.jobManager.restartJob(this.job)) + .thenReturn(this.newDummyJob(this.job.getId(), this.job.getExtId(), JobStatus.PENDING)); + } + + // dummy model constructorss + private FeatureSet newDummyFeatureSet(String name, int version, String project) { + Field feature = new Field(name + "_feature", Enum.INT64); + Field entity = new Field(name + "_entity", Enum.STRING); + + FeatureSet fs = + new FeatureSet( + name, + project, + version, + 100L, + Arrays.asList(entity), + Arrays.asList(feature), + this.dataSource, + FeatureSetStatus.STATUS_READY); + fs.setCreated(Date.from(Instant.ofEpochSecond(10L))); + return fs; + } + + private Job newDummyJob(String id, String extId, JobStatus status) { + return new Job( + id, + extId, + Runner.DATAFLOW.name(), + this.dataSource, + this.dataStore, + Arrays.asList(this.featureSet), + status); + } + + private List newDummyFeatureSetReferences() { + return Arrays.asList( + // all provided: name, version and project + FeatureSetReference.newBuilder() + .setVersion(this.featureSet.getVersion()) + .setName(this.featureSet.getName()) + .setProject(this.featureSet.getProject().toString()) + .build(), + + // name and project + FeatureSetReference.newBuilder() + .setName(this.featureSet.getName()) + .setProject(this.featureSet.getProject().toString()) + .build(), + + // name and version + FeatureSetReference.newBuilder() + .setName(this.featureSet.getName()) + .setVersion(this.featureSet.getVersion()) + .build()); + } + + private List newDummyListRequestFilters() { + return Arrays.asList( + // all provided: name, version and project + ListFeatureSetsRequest.Filter.newBuilder() + .setFeatureSetName(this.featureSet.getName()) + .setProject(this.featureSet.getProject().toString()) + .setFeatureSetVersion(String.valueOf(this.featureSet.getVersion())) + .build(), + + // name and project + ListFeatureSetsRequest.Filter.newBuilder() + .setFeatureSetName(this.featureSet.getName()) + .setProject(this.featureSet.getProject().toString()) + .setFeatureSetVersion("*") + .build(), + + // name and project + ListFeatureSetsRequest.Filter.newBuilder() + .setFeatureSetName(this.featureSet.getName()) + .setProject("*") + .setFeatureSetVersion(String.valueOf(this.featureSet.getVersion())) + .build()); + } + + /* unit tests */ + private ListIngestionJobsResponse tryListJobs(ListIngestionJobsRequest request) { + ListIngestionJobsResponse response = null; + try { + response = this.jobService.listJobs(request); + } catch (InvalidProtocolBufferException e) { + e.printStackTrace(); + fail("Caught Unexpected exception"); + } + + return response; + } + + // list jobs + @Test + public void testListJobsById() { + ListIngestionJobsRequest.Filter filter = + ListIngestionJobsRequest.Filter.newBuilder().setId(this.job.getId()).build(); + ListIngestionJobsRequest request = + ListIngestionJobsRequest.newBuilder().setFilter(filter).build(); + assertThat(this.tryListJobs(request).getJobs(0), equalTo(this.ingestionJob)); + + // list with no filter + request = ListIngestionJobsRequest.newBuilder().build(); + assertThat(this.tryListJobs(request).getJobs(0), equalTo(this.ingestionJob)); + + // list with empty filter + filter = ListIngestionJobsRequest.Filter.newBuilder().build(); + request = ListIngestionJobsRequest.newBuilder().setFilter(filter).build(); + assertThat(this.tryListJobs(request).getJobs(0), equalTo(this.ingestionJob)); + } + + @Test + public void testListJobsByStoreName() { + ListIngestionJobsRequest.Filter filter = + ListIngestionJobsRequest.Filter.newBuilder().setStoreName(this.dataStore.getName()).build(); + ListIngestionJobsRequest request = + ListIngestionJobsRequest.newBuilder().setFilter(filter).build(); + assertThat(this.tryListJobs(request).getJobs(0), equalTo(this.ingestionJob)); + } + + @Test + public void testListIngestionJobByFeatureSetReference() { + // list job by feature set reference: name and version and project + ListIngestionJobsRequest.Filter filter = + ListIngestionJobsRequest.Filter.newBuilder() + .setFeatureSetReference(this.fsReferences.get(0)) + .setId(this.job.getId()) + .build(); + ListIngestionJobsRequest request = + ListIngestionJobsRequest.newBuilder().setFilter(filter).build(); + assertThat(this.tryListJobs(request).getJobs(0), equalTo(this.ingestionJob)); + + // list job by feature set reference: name and version + filter = + ListIngestionJobsRequest.Filter.newBuilder() + .setFeatureSetReference(this.fsReferences.get(1)) + .setId(this.job.getId()) + .build(); + request = ListIngestionJobsRequest.newBuilder().setFilter(filter).build(); + assertThat(this.tryListJobs(request).getJobs(0), equalTo(this.ingestionJob)); + + // list job by feature set reference: name and project + filter = + ListIngestionJobsRequest.Filter.newBuilder() + .setFeatureSetReference(this.fsReferences.get(2)) + .setId(this.job.getId()) + .build(); + request = ListIngestionJobsRequest.newBuilder().setFilter(filter).build(); + assertThat(this.tryListJobs(request).getJobs(0), equalTo(this.ingestionJob)); + } + + // stop jobs + private StopIngestionJobResponse tryStopJob( + StopIngestionJobRequest request, boolean expectError) { + StopIngestionJobResponse response = null; + try { + response = this.jobService.stopJob(request); + // expected exception, but none was thrown + if (expectError) { + fail("Expected exception, but none was thrown"); + } + } catch (Exception e) { + if (expectError != true) { + // unexpected exception + e.printStackTrace(); + fail("Caught Unexpected exception trying to restart job"); + } + } + + return response; + } + + @Test + public void testStopJobForId() { + JobStatus prevStatus = this.job.getStatus(); + this.job.setStatus(JobStatus.RUNNING); + + StopIngestionJobRequest request = + StopIngestionJobRequest.newBuilder().setId(this.job.getId()).build(); + this.tryStopJob(request, false); + verify(this.jobManager).abortJob(this.job.getExtId()); + + // TODO: check that for job status change in featureset source + + this.job.setStatus(prevStatus); + } + + @Test + public void testStopAlreadyStop() { + // check that stop jobs does not trying to stop jobs that are not already stopped + List doNothingStatuses = new ArrayList<>(); + doNothingStatuses.addAll(JobStatus.getTerminalState()); + + JobStatus prevStatus = this.job.getStatus(); + for (JobStatus status : doNothingStatuses) { + this.job.setStatus(status); + + StopIngestionJobRequest request = + StopIngestionJobRequest.newBuilder().setId(this.job.getId()).build(); + this.tryStopJob(request, false); + + verify(this.jobManager, never()).abortJob(this.job.getExtId()); + } + + this.job.setStatus(prevStatus); + } + + @Test + public void testStopUnsupportedError() { + // check for UnsupportedOperationException when trying to stop jobs are + // in an in unknown or in a transitional state + JobStatus prevStatus = this.job.getStatus(); + List unsupportedStatuses = new ArrayList<>(); + unsupportedStatuses.addAll(JobStatus.getTransitionalStates()); + unsupportedStatuses.add(JobStatus.UNKNOWN); + + for (JobStatus status : unsupportedStatuses) { + this.job.setStatus(status); + + StopIngestionJobRequest request = + StopIngestionJobRequest.newBuilder().setId(this.job.getId()).build(); + this.tryStopJob(request, true); + } + + this.job.setStatus(prevStatus); + } + + // restart jobs + private RestartIngestionJobResponse tryRestartJob( + RestartIngestionJobRequest request, boolean expectError) { + RestartIngestionJobResponse response = null; + try { + response = this.jobService.restartJob(request); + // expected exception, but none was thrown + if (expectError) { + fail("Expected exception, but none was thrown"); + } + } catch (Exception e) { + if (expectError != true) { + // unexpected exception + e.printStackTrace(); + fail("Caught Unexpected exception trying to stop job"); + } + } + + return response; + } + + @Test + public void testRestartJobForId() { + JobStatus prevStatus = this.job.getStatus(); + + // restart running job + this.job.setStatus(JobStatus.RUNNING); + RestartIngestionJobRequest request = + RestartIngestionJobRequest.newBuilder().setId(this.job.getId()).build(); + this.tryRestartJob(request, false); + + // restart terminated job + this.job.setStatus(JobStatus.SUSPENDED); + request = RestartIngestionJobRequest.newBuilder().setId(this.job.getId()).build(); + this.tryRestartJob(request, false); + + verify(this.jobManager, times(2)).restartJob(this.job); + verify(this.jobRepository, times(2)).saveAndFlush(this.job); + + this.job.setStatus(prevStatus); + } + + @Test + public void testRestartUnsupportedError() { + // check for UnsupportedOperationException when trying to restart jobs are + // in an in unknown or in a transitional state + JobStatus prevStatus = this.job.getStatus(); + List unsupportedStatuses = new ArrayList<>(); + unsupportedStatuses.addAll(JobStatus.getTransitionalStates()); + unsupportedStatuses.add(JobStatus.UNKNOWN); + + for (JobStatus status : unsupportedStatuses) { + this.job.setStatus(status); + + RestartIngestionJobRequest request = + RestartIngestionJobRequest.newBuilder().setId(this.job.getId()).build(); + this.tryRestartJob(request, true); + } + + this.job.setStatus(prevStatus); + } +} diff --git a/protos/feast/core/CoreService.proto b/protos/feast/core/CoreService.proto index 35b96e17895..b7760d0b9aa 100644 --- a/protos/feast/core/CoreService.proto +++ b/protos/feast/core/CoreService.proto @@ -24,6 +24,8 @@ option java_package = "feast.core"; import "feast/core/FeatureSet.proto"; import "feast/core/Store.proto"; +import "feast/core/FeatureSetReference.proto"; +import "feast/core/IngestionJob.proto"; service CoreService { // Retrieve version information about this Feast deployment @@ -73,6 +75,24 @@ service CoreService { // Lists all projects active projects. rpc ListProjects (ListProjectsRequest) returns (ListProjectsResponse); + + // List Ingestion Jobs given an optional filter. + // Returns allow ingestions matching the given request filter. + // Returns all ingestion jobs if no filter is provided. + // Returns an empty list if no ingestion jobs match the filter. + rpc ListIngestionJobs(ListIngestionJobsRequest) returns (ListIngestionJobsResponse); + + // Restart an Ingestion Job. Restarts the ingestion job with the given job id. + // NOTE: Data might be lost during the restart for some job runners. + // Does not support stopping a job in a transitional (ie pending, suspending, aborting), + // terminal state (ie suspended or aborted) or unknown status + rpc RestartIngestionJob(RestartIngestionJobRequest) returns (RestartIngestionJobResponse); + + // Stop an Ingestion Job. Stop (Aborts) the ingestion job with the given job id. + // Does nothing if the target job if already in a terminal state (ie suspended or aborted). + // Does not support stopping a job in a transitional (ie pending, suspending, aborting) or unknown status + rpc StopIngestionJob(StopIngestionJobRequest) returns (StopIngestionJobResponse); + } // Request for a single feature set @@ -215,4 +235,42 @@ message ListProjectsRequest { message ListProjectsResponse { // List of project names (archived projects are filtered out) repeated string projects = 1; -} \ No newline at end of file +} + +// Request for listing ingestion jobs +message ListIngestionJobsRequest { + Filter filter = 1; + + message Filter { + // Filter by Job ID assigned by Feast + string id = 1; + // Filter by ingestion job target feature set. + FeatureSetReference feature_set_reference = 2; + // Filter by Name of store + string store_name = 3; + } +} + +// Response from listing ingestion jobs +message ListIngestionJobsResponse { + repeated IngestionJob jobs = 1; +} + +// Request to restart ingestion job +message RestartIngestionJobRequest { + // Job ID assigned by Feast + string id = 1; +} + +// Response from restartingan injestion job +message RestartIngestionJobResponse {} + + +// Request to stop ingestion job +message StopIngestionJobRequest { + // Job ID assigned by Feast + string id = 1; +} + +// Request from stopping an ingestion job +message StopIngestionJobResponse {} diff --git a/protos/feast/core/FeatureSetReference.proto b/protos/feast/core/FeatureSetReference.proto new file mode 100644 index 00000000000..2501ec0931c --- /dev/null +++ b/protos/feast/core/FeatureSetReference.proto @@ -0,0 +1,33 @@ +// +// Copyright 2020 The Feast Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +syntax = "proto3"; + +package feast.core; + +option go_package = "github.com/gojek/feast/sdk/go/protos/feast/core"; +option java_outer_classname = "FeatureSetReferenceProto"; +option java_package = "feast.core"; + +// Defines a composite key that refers to a unique FeatureSet +message FeatureSetReference { + // Name of the project + string project = 1; + // Name of the FeatureSet + string name = 2; + // Version no. of the FeatureSet + int32 version = 3; +} diff --git a/protos/feast/core/IngestionJob.proto b/protos/feast/core/IngestionJob.proto new file mode 100644 index 00000000000..68af28c0763 --- /dev/null +++ b/protos/feast/core/IngestionJob.proto @@ -0,0 +1,65 @@ +// +// Copyright 2020 The Feast Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +syntax = "proto3"; + +package feast.core; + +option go_package = "github.com/gojek/feast/sdk/go/protos/feast/core"; +option java_outer_classname = "IngestionJobProto"; +option java_package = "feast.core"; + +import "feast/core/FeatureSet.proto"; +import "feast/core/Store.proto"; +import "feast/core/Source.proto"; + +// Represents Feast Injestion Job +message IngestionJob { + // Job ID assigned by Feast + string id = 1; + // External job ID specific to the runner. + // For DirectRunner jobs, this is identical to id. For DataflowRunner jobs, this refers to the Dataflow job ID. + string external_id = 2; + IngestionJobStatus status = 3; + // List of feature sets whose features are populated by this job. + repeated feast.core.FeatureSet feature_sets = 4; + // Source this job is reading from. + feast.core.Source source = 5; + // Store this job is writing to. + feast.core.Store store = 6; +} + +// Status of a Feast Ingestion Job +enum IngestionJobStatus { + // Job status is not known. + UNKNOWN = 0; + // Import job is submitted to runner and currently pending for executing + PENDING = 1; + // Import job is currently running in the runner + RUNNING = 2; + // Runner's reported the import job has completed (applicable to batch job) + COMPLETED = 3; + // When user sent abort command, but it's still running + ABORTING = 4; + // User initiated abort job + ABORTED = 5; + // Runner's reported that the import job failed to run or there is a failure during job + ERROR = 6; + // job has been suspended and waiting for cleanup + SUSPENDING = 7; + // job has been suspended + SUSPENDED = 8; +} diff --git a/sdk/python/feast/cli.py b/sdk/python/feast/cli.py index dc4784b3025..cd1146b4810 100644 --- a/sdk/python/feast/cli.py +++ b/sdk/python/feast/cli.py @@ -22,8 +22,9 @@ from feast.client import Client from feast.config import Config -from feast.feature_set import FeatureSet +from feast.feature_set import FeatureSet, FeatureSetRef from feast.loaders.yaml import yaml_loader +from feast.core.IngestionJob_pb2 import IngestionJobStatus _logger = logging.getLogger(__name__) @@ -128,11 +129,11 @@ def feature_set_list(): table = [] for fs in feast_client.list_feature_sets(): - table.append([fs.name, fs.version]) + table.append([fs.name, fs.version, repr(fs)]) from tabulate import tabulate - print(tabulate(table, headers=["NAME", "VERSION"], tablefmt="plain")) + print(tabulate(table, headers=["NAME", "VERSION", "REFERENCE"], tablefmt="plain")) @feature_set.command("apply") @@ -214,6 +215,116 @@ def project_list(): print(tabulate(table, headers=["NAME"], tablefmt="plain")) +@cli.group(name="ingest-jobs") +def ingest_job(): + """ + Manage ingestion jobs + """ + pass + + +@ingest_job.command("list") +@click.option("--job-id", "-i", help="Show only ingestion jobs with the given job id") +@click.option( + "--feature-set-ref", + "-f", + help="Show only ingestion job targeting the feature set with the given reference", +) +@click.option( + "--store-name", + "-s", + help="List only ingestion job that ingest into feast store with given name", +) +# TODO: types +def ingest_job_list(job_id, feature_set_ref, store_name): + """ + List ingestion jobs + """ + # parse feature set reference + if feature_set_ref is not None: + feature_set_ref = FeatureSetRef.from_str(feature_set_ref) + + # pull & render ingestion jobs as a table + feast_client = Client() + table = [] + for ingest_job in feast_client.list_ingest_jobs( + job_id=job_id, feature_set_ref=feature_set_ref, store_name=store_name + ): + table.append([ingest_job.id, IngestionJobStatus.Name(ingest_job.status)]) + + from tabulate import tabulate + + print(tabulate(table, headers=["ID", "STATUS"], tablefmt="plain")) + + +@ingest_job.command("describe") +@click.argument("job_id") +def ingest_job_describe(job_id: str): + """ + Describe the ingestion job with the given id. + """ + # find ingestion job for id + feast_client = Client() + jobs = feast_client.list_ingest_jobs(job_id=job_id) + if len(jobs) < 1: + print(f"Ingestion Job with id {job_id} could not be found") + sys.exit(1) + job = jobs[0] + + # pretty render ingestion job as yaml + print( + yaml.dump(yaml.safe_load(str(job)), default_flow_style=False, sort_keys=False) + ) + + +@ingest_job.command("stop") +@click.option( + "--wait", "-w", is_flag=True, help="Wait for the ingestion job to fully stop." +) +@click.option( + "--timeout", + "-t", + default=600, + help="Timeout in seconds to wait for the job to stop.", +) +@click.argument("job_id") +def ingest_job_stop(wait: bool, timeout: int, job_id: str): + """ + Stop ingestion job for id. + """ + # find ingestion job for id + feast_client = Client() + jobs = feast_client.list_ingest_jobs(job_id=job_id) + if len(jobs) < 1: + print(f"Ingestion Job with id {job_id} could not be found") + sys.exit(1) + job = jobs[0] + + feast_client.stop_ingest_job(job) + + # wait for ingestion job to stop + if wait: + job.wait(IngestionJobStatus.ABORTED, timeout=timeout) + + +@ingest_job.command("restart") +@click.argument("job_id") +def ingest_job_restart(job_id: str): + """ + Restart job for id. + Waits for the job to fully restart. + """ + # find ingestion job for id + feast_client = Client() + jobs = feast_client.list_ingest_jobs(job_id=job_id) + if len(jobs) < 1: + print(f"Ingestion Job with id {job_id} could not be found") + sys.exit(1) + job = jobs[0] + + feast_client.restart_ingest_job(job) + + @cli.command() @click.option( "--name", "-n", help="Feature set name to ingest data into", required=True diff --git a/sdk/python/feast/client.py b/sdk/python/feast/client.py index 2a0b636b373..f5aed118cfd 100644 --- a/sdk/python/feast/client.py +++ b/sdk/python/feast/client.py @@ -50,11 +50,14 @@ ListFeatureSetsResponse, ListProjectsRequest, ListProjectsResponse, + ListIngestionJobsRequest, + RestartIngestionJobRequest, + StopIngestionJobRequest, ) from feast.core.CoreService_pb2_grpc import CoreServiceStub from feast.core.FeatureSet_pb2 import FeatureSetStatus -from feast.feature_set import Entity, FeatureSet -from feast.job import Job +from feast.feature_set import Entity, FeatureSet, FeatureSetRef +from feast.job import RetrievalJob, IngestJob from feast.loaders.abstract_producer import get_producer from feast.loaders.file import export_source_to_staging_location from feast.loaders.ingest import KAFKA_CHUNK_PRODUCTION_TIMEOUT, get_feature_row_chunks @@ -416,7 +419,7 @@ def list_feature_sets( Args: project: Filter feature sets based on project name name: Filter feature sets based on feature set name - version: Filter feature sets based on version number + version: Filter feature sets based on version numbf, Returns: List of feature sets @@ -507,7 +510,7 @@ def get_batch_features( feature_refs: List[str], entity_rows: Union[pd.DataFrame, str], default_project: str = None, - ) -> Job: + ) -> RetrievalJob: """ Retrieves historical features from a Feast Serving deployment. @@ -525,8 +528,8 @@ def get_batch_features( default_project: Default project where feature values will be found. Returns: - feast.job.Job: - Returns a job object that can be used to monitor retrieval + feast.job.RetrievalJob: + Returns a retrival job object that can be used to monitor retrieval progress asynchronously, and can be used to materialize the results. @@ -606,7 +609,7 @@ def get_batch_features( # Retrieve Feast Job object to manage life cycle of retrieval response = self._serving_service_stub.GetBatchFeatures(request) - return Job(response.job, self._serving_service_stub) + return RetrievalJob(response.job, self._serving_service_stub) def get_online_features( self, @@ -648,6 +651,72 @@ def get_online_features( ) ) + def list_ingest_jobs( + self, + job_id: str = None, + feature_set_ref: FeatureSetRef = None, + store_name: str = None, + ): + """ + List the ingestion jobs currently registered in Feast, with optional filters. + Provides detailed metadata about each ingestion job. + + Args: + job_id: Select specific ingestion job with the given job_id + feature_set_ref: Filter ingestion jobs by target feature set (via reference) + store_name: Filter ingestion jobs by target feast store's name + + Returns: + List of IngestJobs matching the given filters + """ + self._connect_core() + # construct list request + feature_set_ref = None + list_filter = ListIngestionJobsRequest.Filter( + id=job_id, feature_set_reference=feature_set_ref, store_name=store_name, + ) + request = ListIngestionJobsRequest(filter=list_filter) + # make list request & unpack response + response = self._core_service_stub.ListIngestionJobs(request) + ingest_jobs = [ + IngestJob(proto, self._core_service_stub) for proto in response.jobs + ] + return ingest_jobs + + def restart_ingest_job(self, job: IngestJob): + """ + Restart ingestion job currently registered in Feast. + NOTE: Data might be lost during the restart for some job runners. + Does not support stopping a job in a transitional (ie pending, suspending, aborting), + terminal state (ie suspended or aborted) or unknown status + + Args: + job: IngestJob to restart + """ + self._connect_core() + request = RestartIngestionJobRequest(id=job.id) + try: + self._core_service_stub.RestartIngestionJob(request) + except grpc.RpcError as e: + raise grpc.RpcError(e.details()) + + def stop_ingest_job(self, job: IngestJob): + """ + Stop ingestion job currently resgistered in Feast + Does nothing if the target job if already in a terminal state (ie suspended or aborted). + Does not support stopping a job in a transitional (ie pending, suspending, aborting) + or in a unknown status + + Args: + job: IngestJob to restart + """ + self._connect_core() + request = StopIngestionJobRequest(id=job.id) + try: + self._core_service_stub.StopIngestionJob(request) + except grpc.RpcError as e: + raise grpc.RpcError(e.details()) + def ingest( self, feature_set: Union[str, FeatureSet], diff --git a/sdk/python/feast/feature_set.py b/sdk/python/feast/feature_set.py index 4ebfecf1675..c4cedaf6b2a 100644 --- a/sdk/python/feast/feature_set.py +++ b/sdk/python/feast/feature_set.py @@ -27,6 +27,9 @@ from feast.core.FeatureSet_pb2 import FeatureSet as FeatureSetProto from feast.core.FeatureSet_pb2 import FeatureSetMeta as FeatureSetMetaProto from feast.core.FeatureSet_pb2 import FeatureSetSpec as FeatureSetSpecProto +from feast.core.FeatureSetReference_pb2 import ( + FeatureSetReference as FeatureSetReferenceProto, +) from feast.entity import Entity from feast.feature import Feature, Field from feast.loaders import yaml as feast_yaml @@ -88,14 +91,7 @@ def __str__(self): return str(MessageToJson(self.to_proto())) def __repr__(self): - ref = "" - if self.project: - ref += self.project + "/" - if self.name: - ref += self.name - if self.version: - ref += ":" + str(self.version).strip() - return ref + return FeatureSetRef.from_feature_set(self).__repr__() @property def fields(self) -> Dict[str, Field]: @@ -761,6 +757,104 @@ def to_proto(self) -> FeatureSetProto: return FeatureSetProto(spec=spec, meta=meta) +class FeatureSetRef: + """ + Represents a reference to a featureset + """ + + def __init__(self, project: str = None, name: str = None, version: int = None): + self.proto = FeatureSetReferenceProto( + project=project, name=name, version=version + ) + + @property + def project(self) -> str: + """ + Get the project of feature set referenced by this reference + """ + return self.proto.project + + @property + def name(self) -> str: + """ + Get the name of feature set referenced by this reference + """ + return self.proto.name + + @property + def version(self) -> int: + """ + Get the version of feature set referenced by this reference + """ + return self.proto.version + + @classmethod + def from_feature_set(cls, feature_set: FeatureSet): + """ + Construct a feature set reference that refers to the given feature set. + + Args: + feature_set: Feature set to create reference from. + + Returns: + FeatureSetRef that refers to the given feature set + """ + return cls(feature_set.project, feature_set.name, feature_set.version) + + @classmethod + def from_str(cls, ref_str: str): + """ + Parse a feature reference from string representation. + (as defined by __repr__()) + + Args: + ref_str: string representation of the reference. + + Returns: + FeatureSetRef constructed from the string + """ + if "/" in ref_str: + project, ref_str = ref_str.split("/") + if ":" in ref_str: + ref_str, version_str = ref_str.split(":") + name = ref_str + + return cls(project, name, int(version_str)) + + def to_proto(self, arg1) -> FeatureSetReferenceProto: + """ + Convert and return this feature set reference to protobuf. + + Returns: + Protobuf version of this feature set reference. + """ + return self.proto + + def __str__(self): + # human readable string of the reference + return f"FeatureSetRef<{self.__repr__()}>" + + def __repr__(self): + # return string representation of the reference + # [project/]name[:version] + ref_str = "" + if self.proto.project: + ref_str += self.proto.project + "/" + if self.proto.name: + ref_str += self.proto.name + if self.proto.version: + ref_str += ":" + str(self.proto.version).strip() + return ref_str + + def __eq__(self, other): + # compare with other feature set + return hash(self) == hash(other) + + def __hash__(self): + # hash this reference + return hash(repr(self)) + + def _infer_pd_column_type(column, series, rows_to_sample): dtype = None sample_count = 0 diff --git a/sdk/python/feast/job.py b/sdk/python/feast/job.py index ab65da74459..3576bc1b385 100644 --- a/sdk/python/feast/job.py +++ b/sdk/python/feast/job.py @@ -2,11 +2,15 @@ import time from datetime import datetime, timedelta from urllib.parse import urlparse +from typing import List import fastavro import pandas as pd from google.cloud import storage +from google.protobuf.json_format import MessageToJson +from feast.feature_set import FeatureSet +from feast.source import Source from feast.serving.ServingService_pb2 import ( DATA_FORMAT_AVRO, JOB_STATUS_DONE, @@ -14,8 +18,13 @@ ) from feast.serving.ServingService_pb2 import Job as JobProto from feast.serving.ServingService_pb2_grpc import ServingServiceStub +from feast.core.Store_pb2 import Store +from feast.core.IngestionJob_pb2 import IngestionJob as IngestJobProto +from feast.core.IngestionJob_pb2 import IngestionJobStatus +from feast.core.CoreService_pb2_grpc import CoreServiceStub +from feast.core.CoreService_pb2 import ListIngestionJobsRequest -# Maximum no of seconds to wait until the jobs status is DONE in Feast +# Maximum no of seconds to wait until the retrieval jobs status is DONE in Feast # Currently set to the maximum query execution time limit in BigQuery DEFAULT_TIMEOUT_SEC: int = 21600 @@ -23,7 +32,7 @@ MAX_WAIT_INTERVAL_SEC: int = 60 -class Job: +class RetrievalJob: """ A class representing a job for feature retrieval in Feast. """ @@ -33,7 +42,6 @@ def __init__(self, job_proto: JobProto, serving_stub: ServingServiceStub): Args: job_proto: Job proto object (wrapped by this job object) serving_stub: Stub for Feast serving service - storage_client: Google Cloud Storage client """ self.job_proto = job_proto self.serving_stub = serving_stub @@ -187,3 +195,107 @@ def to_chunked_dataframe( def __iter__(self): return iter(self.result()) + + +class IngestJob: + """ + Defines a job for feature ingestion in feast. + """ + + def __init__(self, job_proto: IngestJobProto, core_stub: CoreServiceStub): + """ + Construct a native ingest job from its protobuf version. + + Args: + job_proto: Job proto object to construct from. + core_stub: stub for Feast CoreService + """ + self.proto = job_proto + self.core_svc = core_stub + + def reload(self): + """ + Update this IngestJob with the latest info from Feast + """ + # pull latest proto from feast core + response = self.core_svc.ListIngestionJobs( + ListIngestionJobsRequest(filter=ListIngestionJobsRequest.Filter(id=self.id)) + ) + self.proto = response.jobs[0] + + @property + def id(self) -> str: + """ + Getter for IngestJob's job id. + """ + return self.proto.id + + @property + def external_id(self) -> str: + """ + Getter for IngestJob's external job id. + """ + self.reload() + return self.proto.external_id + + @property + def status(self) -> IngestionJobStatus: + """ + Getter for IngestJob's status + """ + self.reload() + return self.proto.status + + @property + def feature_sets(self) -> List[FeatureSet]: + """ + Getter for the IngestJob's feature sets + """ + # convert featureset protos to native objects + return [FeatureSet.from_proto(fs) for fs in self.proto.feature_sets] + + @property + def source(self) -> Source: + """ + Getter for the IngestJob's data source. + """ + return Source.from_proto(self.proto.source) + + @property + def store(self) -> Store: + """ + Getter for the IngestJob's target feast store. + """ + return self.proto.store + + def wait(self, status: IngestionJobStatus, timeout_secs: float = 300): + """ + Wait for this IngestJob to transtion to the given status. + Raises TimeoutError if the wait operation times out. + + Args: + status: The IngestionJobStatus to wait for. + timeout_secs: Maximum seconds to wait before timing out. + """ + # poll & wait for job status to transition + wait_begin = time.time() + wait_secs = 2 + elapsed_secs = 0 + while self.status != status and elapsed_secs <= timeout_secs: + time.sleep(wait_secs) + # back off wait duration exponentially, capped at MAX_WAIT_INTERVAL_SEC + wait_secs = min(wait_secs * 2, MAX_WAIT_INTERVAL_SEC) + elapsed_secs = time.time() - wait_begin + + # raise error if timeout + if elapsed_secs > timeout_secs: + raise TimeoutError("Wait for IngestJob's status to transition timed out") + + def __str__(self): + # render the contents of ingest job as human readable string + self.reload() + return str(MessageToJson(self.proto)) + + def __repr__(self): + # render the ingest job as human readable string + return f"IngestJob<{self.id}>" diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index 3c1e8bef0f0..f7f5676ced5 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -29,6 +29,12 @@ from feast.core.CoreService_pb2 import ( GetFeastCoreVersionResponse, GetFeatureSetResponse, + ListIngestionJobsResponse, +) +from feast.core.Store_pb2 import Store +from feast.core.IngestionJob_pb2 import ( + IngestionJob as IngestJobProto, + IngestionJobStatus, ) from feast.core.FeatureSet_pb2 import EntitySpec as EntitySpecProto from feast.core.FeatureSet_pb2 import FeatureSet as FeatureSetProto @@ -38,7 +44,8 @@ from feast.core.FeatureSet_pb2 import FeatureSpec as FeatureSpecProto from feast.core.Source_pb2 import KafkaSourceConfig, Source, SourceType from feast.entity import Entity -from feast.feature_set import Feature, FeatureSet +from feast.feature_set import Feature, FeatureSet, FeatureSetRef +from feast.job import IngestJob from feast.serving.ServingService_pb2 import ( GetFeastServingInfoResponse, GetOnlineFeaturesRequest, @@ -295,7 +302,109 @@ def test_get_feature_set(self, mocked_client, mocker): and len(feature_set.entities) == 1 ) - # @pytest.mark.parametrize( + @pytest.mark.parametrize( + "mocked_client", + [pytest.lazy_fixture("mock_client"), pytest.lazy_fixture("secure_mock_client")], + ) + def test_list_ingest_jobs(self, mocked_client, mocker): + mocker.patch.object( + mocked_client, + "_core_service_stub", + return_value=Core.CoreServiceStub(grpc.insecure_channel("")), + ) + + feature_set_proto = FeatureSetProto( + spec=FeatureSetSpecProto( + project="test", name="driver", max_age=Duration(seconds=3600), + ) + ) + + mocker.patch.object( + mocked_client._core_service_stub, + "ListIngestionJobs", + return_value=ListIngestionJobsResponse( + jobs=[ + IngestJobProto( + id="kafka-to-redis", + external_id="job-2222", + status=IngestionJobStatus.RUNNING, + feature_sets=[feature_set_proto], + source=Source( + type=SourceType.KAFKA, + kafka_source_config=KafkaSourceConfig( + bootstrap_servers="localhost:9092", topic="topic" + ), + ), + store=Store(name="redis"), + ) + ] + ), + ) + + # list ingestion jobs by target feature set reference + ingest_jobs = mocked_client.list_ingest_jobs( + feature_set_ref=FeatureSetRef.from_feature_set( + FeatureSet.from_proto(feature_set_proto) + ) + ) + assert len(ingest_jobs) >= 1 + + ingest_job = ingest_jobs[0] + assert ( + ingest_job.status == IngestionJobStatus.RUNNING + and ingest_job.id == "kafka-to-redis" + and ingest_job.external_id == "job-2222" + and ingest_job.feature_sets[0].name == "driver" + and ingest_job.source.source_type == "Kafka" + ) + + @pytest.mark.parametrize( + "mocked_client", + [pytest.lazy_fixture("mock_client"), pytest.lazy_fixture("secure_mock_client")], + ) + def test_restart_ingest_job(self, mocked_client, mocker): + mocker.patch.object( + mocked_client, + "_core_service_stub", + return_value=Core.CoreServiceStub(grpc.insecure_channel("")), + ) + + ingest_job = IngestJob( + job_proto=IngestJobProto( + id="kafka-to-redis", + external_id="job#2222", + status=IngestionJobStatus.ERROR, + ), + core_stub=mocked_client._core_service_stub, + ) + + mocked_client.restart_ingest_job(ingest_job) + assert mocked_client._core_service_stub.RestartIngestionJob.called + + @pytest.mark.parametrize( + "mocked_client", + [pytest.lazy_fixture("mock_client"), pytest.lazy_fixture("secure_mock_client")], + ) + def test_stop_ingest_job(self, mocked_client, mocker): + mocker.patch.object( + mocked_client, + "_core_service_stub", + return_value=Core.CoreServiceStub(grpc.insecure_channel("")), + ) + + ingest_job = IngestJob( + job_proto=IngestJobProto( + id="kafka-to-redis", + external_id="job#2222", + status=IngestionJobStatus.RUNNING, + ), + core_stub=mocked_client._core_service_stub, + ) + + mocked_client.stop_ingest_job(ingest_job) + assert mocked_client._core_service_stub.StopIngestionJob.called + + # @pytest.mark.parametrize # "mocked_client", # [pytest.lazy_fixture("mock_client"), pytest.lazy_fixture("secure_mock_client")], # ) diff --git a/sdk/python/tests/test_feature_set.py b/sdk/python/tests/test_feature_set.py index 2c539ebe0a7..bd31d712bb3 100644 --- a/sdk/python/tests/test_feature_set.py +++ b/sdk/python/tests/test_feature_set.py @@ -23,7 +23,7 @@ import feast.core.CoreService_pb2_grpc as Core from feast.client import Client from feast.entity import Entity -from feast.feature_set import Feature, FeatureSet +from feast.feature_set import Feature, FeatureSet, FeatureSetRef from feast.value_type import ValueType from feast_core_server import CoreServicer @@ -167,3 +167,20 @@ def test_add_features_from_df_success( ) assert len(my_feature_set.features) == feature_count assert len(my_feature_set.entities) == entity_count + + +class TestFeatureSetRef: + def test_from_feature_set(self): + feature_set = FeatureSet("test", "test") + feature_set.version = 2 + ref = FeatureSetRef.from_feature_set(feature_set) + + assert ref.name == "test" + assert ref.project == "test" + assert ref.version == 2 + + def test_str_ref(self): + original_ref = FeatureSetRef(project="test", name="test", version=2) + ref_str = repr(original_ref) + parsed_ref = FeatureSetRef.from_str(ref_str) + assert original_ref == parsed_ref diff --git a/tests/e2e/basic-ingest-redis-serving.py b/tests/e2e/basic-ingest-redis-serving.py index 1aeccfa5a3a..8e40794344e 100644 --- a/tests/e2e/basic-ingest-redis-serving.py +++ b/tests/e2e/basic-ingest-redis-serving.py @@ -7,9 +7,10 @@ GetOnlineFeaturesRequest, GetOnlineFeaturesResponse, ) +from feast.core.IngestionJob_pb2 import IngestionJobStatus from feast.types.Value_pb2 import Value as Value from feast.client import Client -from feast.feature_set import FeatureSet +from feast.feature_set import FeatureSet, FeatureSetRef from feast.type_map import ValueType from google.protobuf.duration_pb2 import Duration from datetime import datetime @@ -108,7 +109,6 @@ def test_basic_ingest_success(client, basic_dataframe): client.ingest(cust_trans_fs, basic_dataframe) time.sleep(5) - @pytest.mark.timeout(45) @pytest.mark.run(order=12) def test_basic_retrieve_online_success(client, basic_dataframe): @@ -152,6 +152,28 @@ def test_basic_retrieve_online_success(client, basic_dataframe): ): break +@pytest.mark.timeout(300) +@pytest.mark.run(order=19) +def test_basic_ingest_jobs(client, basic_dataframe): + # list ingestion jobs given featureset + cust_trans_fs = client.get_feature_set(name="customer_transactions") + ingest_jobs = client.list_ingest_jobs( + feature_set_ref=FeatureSetRef.from_feature_set(cust_trans_fs)) + # filter ingestion jobs to only those that are running + ingest_jobs = [job for job in ingest_jobs if job.status == IngestionJobStatus.RUNNING] + assert len(ingest_jobs) >= 1 + + for ingest_job in ingest_jobs: + # restart ingestion ingest_job + client.restart_ingest_job(ingest_job) + ingest_job.wait(IngestionJobStatus.RUNNING) + assert ingest_job.status == IngestionJobStatus.RUNNING + + # stop ingestion ingest_job + client.stop_ingest_job(ingest_job) + ingest_job.wait(IngestionJobStatus.ABORTED) + assert ingest_job.status == IngestionJobStatus.ABORTED + @pytest.fixture(scope='module') def all_types_dataframe(): @@ -311,6 +333,27 @@ def test_all_types_retrieve_online_success(client, all_types_dataframe): ): break +@pytest.mark.timeout(300) +@pytest.mark.run(order=29) +def test_all_types_ingest_jobs(client, all_types_dataframe): + # list ingestion jobs given featureset + all_types_fs = client.get_feature_set(name="all_types") + ingest_jobs = client.list_ingest_jobs( + feature_set_ref=FeatureSetRef.from_feature_set(all_types_fs)) + # filter ingestion jobs to only those that are running + ingest_jobs = [job for job in ingest_jobs if job.status == IngestionJobStatus.RUNNING] + assert len(ingest_jobs) >= 1 + + for ingest_job in ingest_jobs: + # restart ingestion ingest_job + client.restart_ingest_job(ingest_job) + ingest_job.wait(IngestionJobStatus.RUNNING) + assert ingest_job.status == IngestionJobStatus.RUNNING + + # stop ingestion ingest_job + client.stop_ingest_job(ingest_job) + ingest_job.wait(IngestionJobStatus.ABORTED) + assert ingest_job.status == IngestionJobStatus.ABORTED @pytest.fixture(scope='module') def large_volume_dataframe(): @@ -466,7 +509,6 @@ def all_types_parquet_file(): df.to_parquet(file_path, allow_truncated_timestamps=True) return file_path - @pytest.mark.timeout(300) @pytest.mark.run(order=40) def test_all_types_parquet_register_feature_set_success(client): From e4f8fe905d3c182416e1b69bf60bfbc0e8348f8a Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Tue, 7 Apr 2020 18:45:45 +0800 Subject: [PATCH 111/176] Add general storage API and refactor existing store implementations (#567) * Add storage interfaces, basic file structure (#529) * Add storage interfaces, basic file structure * Apply spotless, add comments * Move parseResponse and isEmpty to response object * Make changes to write interface to be more beam-like * Pass feature specs to the retriever * Pass feature specs to online retriever * Add FeatureSetRequest * Add mistakenly removed TestUtil * Add mistakenly removed TestUtil * Add BigQuery storage (#546) * Add Redis storage implementation (#547) * Add Redis storage * Remove staleness check; can be checked at the service level * Remove staleness related tests * Add dependencies to top level pom * Clean up code * Change serving and ingestion to use storage API (#553) * Change serving and ingestion to use storage API * Remove extra exclusion clause * Storage refactor API and docstring tweaks (#569) * API and docstring tweaks * Fix javadoc linting errors * Apply spotless * Fix javadoc formatting * Drop result from HistoricalRetrievalResult constructors * Change pipeline to use DeadletterSink API (#586) * Add better code docs to storage refactor (#601) * Add better code documentation, make GetFeastServingInfo independent of retriever * Make getStagingLocation method of historical retriever * Apply spotless * Clean up dependencies, remove exclusions at serving (#607) * Clean up OnlineServingService code (#605) * Clean up OnlineServingService code to be more readable * Revert Metrics * Rename storage API packages to nouns --- ingestion/pom.xml | 18 + .../main/java/feast/ingestion/ImportJob.java | 78 ++-- .../ingestion/transform/ReadFromSource.java | 2 +- .../transform/ValidateFeatureRows.java | 9 +- .../WriteFailedElementToBigQuery.java | 2 +- .../ingestion/transform/WriteToStore.java | 168 ------- .../fn/KafkaRecordToFeatureRowDoFn.java | 3 +- .../transform/fn/ValidateFeatureRowDoFn.java | 2 +- .../WriteDeadletterRowMetricsDoFn.java | 2 +- .../metrics/WriteFailureMetricsTransform.java | 52 +++ ...java => WriteSuccessMetricsTransform.java} | 78 ++-- .../java/feast/ingestion/utils/SpecUtil.java | 7 +- .../java/feast/ingestion/utils/StoreUtil.java | 181 +------- .../feast/ingestion/values/FeatureSet.java | 6 +- .../redis/FeatureRowToRedisMutationDoFn.java | 116 ----- .../store/serving/redis/RedisCustomIO.java | 341 -------------- .../java/feast/ingestion/ImportJobTest.java | 4 +- .../transform/ValidateFeatureRowsTest.java | 159 +++---- .../feast/ingestion/utils/StoreUtilTest.java | 211 --------- .../serving/redis/RedisCustomIOTest.java | 238 ---------- .../src/test/java/feast/test/TestUtil.java | 48 +- pom.xml | 2 + serving/pom.xml | 43 +- .../configuration/ServingServiceConfig.java | 41 +- .../service/BigQueryServingService.java | 282 ------------ .../service/HistoricalServingService.java | 119 +++++ .../serving/service/OnlineServingService.java | 176 ++++++++ .../serving/service/RedisServingService.java | 345 -------------- .../feast/serving/service/ServingService.java | 63 +++ .../serving/specs/CachedSpecService.java | 1 + .../main/java/feast/serving/util/RefUtil.java | 8 + .../service/CachedSpecServiceTest.java | 2 +- ...est.java => OnlineServingServiceTest.java} | 215 ++------- storage/api/pom.xml | 72 +++ .../api/retriever}/FeatureSetRequest.java | 9 +- .../retriever/HistoricalRetrievalResult.java | 100 ++++ .../api/retriever/HistoricalRetriever.java | 49 ++ .../api/retriever/OnlineRetriever.java | 40 ++ .../storage/api/writer/DeadletterSink.java | 38 ++ .../storage/api/writer/FailedElement.java | 83 ++++ .../feast/storage/api/writer/FeatureSink.java | 54 +++ .../feast/storage/api/writer/WriteResult.java | 97 ++++ .../common}/retry/BackOffExecutor.java | 2 +- .../storage/common}/retry/Retriable.java | 2 +- .../storage/common/testing/TestUtil.java | 188 ++++++++ storage/connectors/bigquery/pom.xml | 94 ++++ .../connectors/bigquery/common/TypeUtil.java | 66 +++ .../BigQueryHistoricalRetriever.java | 426 ++++++++++-------- .../retriever/FeatureSetQueryInfo.java | 8 +- .../bigquery/retriever}/QueryTemplater.java | 20 +- .../bigquery/retriever}/SubqueryCallable.java | 24 +- .../writer/BigQueryDeadletterSink.java | 133 ++++++ .../bigquery/writer/BigQueryFeatureSink.java | 188 ++++++++ .../bigquery/writer/BigQueryWrite.java | 107 +++++ .../writer}/FeatureRowToTableRow.java | 2 +- .../bigquery/writer}/GetTableDestination.java | 2 +- .../schemas/deadletter_table_schema.json | 34 ++ .../resources/templates/join_featuresets.sql | 24 + .../templates/single_featureset_pit_join.sql | 90 ++++ storage/connectors/pom.xml | 51 +++ storage/connectors/redis/pom.xml | 82 ++++ .../redis/retriever}/FeatureRowDecoder.java | 2 +- .../redis/retriever/RedisOnlineRetriever.java | 204 +++++++++ .../redis/writer/RedisCustomIO.java | 292 ++++++++++++ .../redis/writer/RedisFeatureSink.java | 74 +++ .../redis/writer}/RedisIngestionClient.java | 4 +- .../RedisStandaloneIngestionClient.java | 9 +- .../retriever}/FeatureRowDecoderTest.java | 2 +- .../retriever/RedisOnlineRetrieverTest.java | 262 +++++++++++ .../connectors/redis/test/TestUtil.java | 44 ++ .../redis/writer/RedisFeatureSinkTest.java | 422 +++++++++-------- 71 files changed, 3711 insertions(+), 2711 deletions(-) delete mode 100644 ingestion/src/main/java/feast/ingestion/transform/WriteToStore.java create mode 100644 ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFailureMetricsTransform.java rename ingestion/src/main/java/feast/ingestion/transform/metrics/{WriteMetricsTransform.java => WriteSuccessMetricsTransform.java} (65%) delete mode 100644 ingestion/src/main/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFn.java delete mode 100644 ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java delete mode 100644 ingestion/src/test/java/feast/ingestion/utils/StoreUtilTest.java delete mode 100644 ingestion/src/test/java/feast/store/serving/redis/RedisCustomIOTest.java delete mode 100644 serving/src/main/java/feast/serving/service/BigQueryServingService.java create mode 100644 serving/src/main/java/feast/serving/service/HistoricalServingService.java create mode 100644 serving/src/main/java/feast/serving/service/OnlineServingService.java delete mode 100644 serving/src/main/java/feast/serving/service/RedisServingService.java rename serving/src/test/java/feast/serving/service/{RedisServingServiceTest.java => OnlineServingServiceTest.java} (72%) create mode 100644 storage/api/pom.xml rename {serving/src/main/java/feast/serving/specs => storage/api/src/main/java/feast/storage/api/retriever}/FeatureSetRequest.java (84%) create mode 100644 storage/api/src/main/java/feast/storage/api/retriever/HistoricalRetrievalResult.java create mode 100644 storage/api/src/main/java/feast/storage/api/retriever/HistoricalRetriever.java create mode 100644 storage/api/src/main/java/feast/storage/api/retriever/OnlineRetriever.java create mode 100644 storage/api/src/main/java/feast/storage/api/writer/DeadletterSink.java create mode 100644 storage/api/src/main/java/feast/storage/api/writer/FailedElement.java create mode 100644 storage/api/src/main/java/feast/storage/api/writer/FeatureSink.java create mode 100644 storage/api/src/main/java/feast/storage/api/writer/WriteResult.java rename {ingestion/src/main/java/feast => storage/api/src/main/java/feast/storage/common}/retry/BackOffExecutor.java (98%) rename {ingestion/src/main/java/feast => storage/api/src/main/java/feast/storage/common}/retry/Retriable.java (95%) create mode 100644 storage/api/src/main/java/feast/storage/common/testing/TestUtil.java create mode 100644 storage/connectors/bigquery/pom.xml create mode 100644 storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/common/TypeUtil.java rename serving/src/main/java/feast/serving/store/bigquery/BatchRetrievalQueryRunnable.java => storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/BigQueryHistoricalRetriever.java (52%) rename serving/src/main/java/feast/serving/store/bigquery/model/FeatureSetInfo.java => storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/FeatureSetQueryInfo.java (90%) rename {serving/src/main/java/feast/serving/store/bigquery => storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever}/QueryTemplater.java (90%) rename {serving/src/main/java/feast/serving/store/bigquery => storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever}/SubqueryCallable.java (70%) create mode 100644 storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryDeadletterSink.java create mode 100644 storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryFeatureSink.java create mode 100644 storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryWrite.java rename {ingestion/src/main/java/feast/store/serving/bigquery => storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer}/FeatureRowToTableRow.java (98%) rename {ingestion/src/main/java/feast/store/serving/bigquery => storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer}/GetTableDestination.java (97%) create mode 100644 storage/connectors/bigquery/src/main/resources/schemas/deadletter_table_schema.json create mode 100644 storage/connectors/bigquery/src/main/resources/templates/join_featuresets.sql create mode 100644 storage/connectors/bigquery/src/main/resources/templates/single_featureset_pit_join.sql create mode 100644 storage/connectors/pom.xml create mode 100644 storage/connectors/redis/pom.xml rename {serving/src/main/java/feast/serving/encoding => storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever}/FeatureRowDecoder.java (98%) create mode 100644 storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever/RedisOnlineRetriever.java create mode 100644 storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisCustomIO.java create mode 100644 storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisFeatureSink.java rename {ingestion/src/main/java/feast/store/serving/redis => storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer}/RedisIngestionClient.java (92%) rename {ingestion/src/main/java/feast/store/serving/redis => storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer}/RedisStandaloneIngestionClient.java (93%) rename {serving/src/test/java/feast/serving/encoding => storage/connectors/redis/src/test/java/feast/storage/connectors/redis/retriever}/FeatureRowDecoderTest.java (98%) create mode 100644 storage/connectors/redis/src/test/java/feast/storage/connectors/redis/retriever/RedisOnlineRetrieverTest.java create mode 100644 storage/connectors/redis/src/test/java/feast/storage/connectors/redis/test/TestUtil.java rename ingestion/src/test/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFnTest.java => storage/connectors/redis/src/test/java/feast/storage/connectors/redis/writer/RedisFeatureSinkTest.java (50%) diff --git a/ingestion/pom.xml b/ingestion/pom.xml index 47204c33f29..9386d066bfd 100644 --- a/ingestion/pom.xml +++ b/ingestion/pom.xml @@ -101,6 +101,24 @@ ${project.version} + + dev.feast + feast-storage-api + ${project.version} + + + + dev.feast + feast-storage-connector-redis + ${project.version} + + + + dev.feast + feast-storage-connector-bigquery + ${project.version} + + com.google.auto.value auto-value-annotations diff --git a/ingestion/src/main/java/feast/ingestion/ImportJob.java b/ingestion/src/main/java/feast/ingestion/ImportJob.java index c4973ce3cae..ef6039e5536 100644 --- a/ingestion/src/main/java/feast/ingestion/ImportJob.java +++ b/ingestion/src/main/java/feast/ingestion/ImportJob.java @@ -17,9 +17,11 @@ package feast.ingestion; import static feast.ingestion.utils.SpecUtil.getFeatureSetReference; +import static feast.ingestion.utils.StoreUtil.getFeatureSink; import com.google.protobuf.InvalidProtocolBufferException; import feast.core.FeatureSetProto.FeatureSet; +import feast.core.FeatureSetProto.FeatureSetSpec; import feast.core.SourceProto.Source; import feast.core.StoreProto.Store; import feast.ingestion.options.BZip2Decompressor; @@ -27,13 +29,14 @@ import feast.ingestion.options.StringListStreamConverter; import feast.ingestion.transform.ReadFromSource; import feast.ingestion.transform.ValidateFeatureRows; -import feast.ingestion.transform.WriteFailedElementToBigQuery; -import feast.ingestion.transform.WriteToStore; -import feast.ingestion.transform.metrics.WriteMetricsTransform; -import feast.ingestion.utils.ResourceUtil; +import feast.ingestion.transform.metrics.WriteFailureMetricsTransform; +import feast.ingestion.transform.metrics.WriteSuccessMetricsTransform; import feast.ingestion.utils.SpecUtil; -import feast.ingestion.utils.StoreUtil; -import feast.ingestion.values.FailedElement; +import feast.storage.api.writer.DeadletterSink; +import feast.storage.api.writer.FailedElement; +import feast.storage.api.writer.FeatureSink; +import feast.storage.api.writer.WriteResult; +import feast.storage.connectors.bigquery.writer.BigQueryDeadletterSink; import feast.types.FeatureRowProto.FeatureRow; import java.io.IOException; import java.util.HashMap; @@ -93,17 +96,24 @@ public static PipelineResult runPipeline(ImportOptions options) throws IOExcepti SpecUtil.getSubscribedFeatureSets(store.getSubscriptionsList(), featureSets); // Generate tags by key - Map featureSetsByKey = new HashMap<>(); + Map featureSetSpecsByKey = new HashMap<>(); subscribedFeatureSets.stream() .forEach( fs -> { - String ref = getFeatureSetReference(fs); - featureSetsByKey.put(ref, fs); + String ref = getFeatureSetReference(fs.getSpec()); + featureSetSpecsByKey.put(ref, fs.getSpec()); }); + FeatureSink featureSink = getFeatureSink(store, featureSetSpecsByKey); + // TODO: make the source part of the job initialisation options Source source = subscribedFeatureSets.get(0).getSpec().getSource(); + for (FeatureSet featureSet : subscribedFeatureSets) { + // Ensure Store has valid configuration and Feast can access it. + featureSink.prepareWrite(featureSet); + } + // Step 1. Read messages from Feast Source as FeatureRow. PCollectionTuple convertedFeatureRows = pipeline.apply( @@ -114,58 +124,48 @@ public static PipelineResult runPipeline(ImportOptions options) throws IOExcepti .setFailureTag(DEADLETTER_OUT) .build()); - for (FeatureSet featureSet : subscribedFeatureSets) { - // Ensure Store has valid configuration and Feast can access it. - StoreUtil.setupStore(store, featureSet); - } - // Step 2. Validate incoming FeatureRows PCollectionTuple validatedRows = convertedFeatureRows .get(FEATURE_ROW_OUT) .apply( ValidateFeatureRows.newBuilder() - .setFeatureSets(featureSetsByKey) + .setFeatureSetSpecs(featureSetSpecsByKey) .setSuccessTag(FEATURE_ROW_OUT) .setFailureTag(DEADLETTER_OUT) .build()); // Step 3. Write FeatureRow to the corresponding Store. - validatedRows - .get(FEATURE_ROW_OUT) - .apply( - "WriteFeatureRowToStore", - WriteToStore.newBuilder().setFeatureSets(featureSetsByKey).setStore(store).build()); + WriteResult writeFeatureRows = + validatedRows.get(FEATURE_ROW_OUT).apply("WriteFeatureRowToStore", featureSink.writer()); // Step 4. Write FailedElements to a dead letter table in BigQuery. if (options.getDeadLetterTableSpec() != null) { + // TODO: make deadletter destination type configurable + DeadletterSink deadletterSink = + new BigQueryDeadletterSink(options.getDeadLetterTableSpec()); + convertedFeatureRows .get(DEADLETTER_OUT) - .apply( - "WriteFailedElements_ReadFromSource", - WriteFailedElementToBigQuery.newBuilder() - .setJsonSchema(ResourceUtil.getDeadletterTableSchemaJson()) - .setTableSpec(options.getDeadLetterTableSpec()) - .build()); + .apply("WriteFailedElements_ReadFromSource", deadletterSink.write()); validatedRows .get(DEADLETTER_OUT) - .apply( - "WriteFailedElements_ValidateRows", - WriteFailedElementToBigQuery.newBuilder() - .setJsonSchema(ResourceUtil.getDeadletterTableSchemaJson()) - .setTableSpec(options.getDeadLetterTableSpec()) - .build()); + .apply("WriteFailedElements_ValidateRows", deadletterSink.write()); + + writeFeatureRows + .getFailedInserts() + .apply("WriteFailedElements_WriteFeatureRowToStore", deadletterSink.write()); } // Step 5. Write metrics to a metrics sink. - validatedRows.apply( - "WriteMetrics", - WriteMetricsTransform.newBuilder() - .setStoreName(store.getName()) - .setSuccessTag(FEATURE_ROW_OUT) - .setFailureTag(DEADLETTER_OUT) - .build()); + writeFeatureRows + .getSuccessfulInserts() + .apply("WriteSuccessMetrics", WriteSuccessMetricsTransform.create(store.getName())); + + writeFeatureRows + .getFailedInserts() + .apply("WriteFailureMetrics", WriteFailureMetricsTransform.create(store.getName())); } return pipeline.run(); diff --git a/ingestion/src/main/java/feast/ingestion/transform/ReadFromSource.java b/ingestion/src/main/java/feast/ingestion/transform/ReadFromSource.java index 65e95b287dc..fb013d0375f 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/ReadFromSource.java +++ b/ingestion/src/main/java/feast/ingestion/transform/ReadFromSource.java @@ -21,7 +21,7 @@ import feast.core.SourceProto.Source; import feast.core.SourceProto.SourceType; import feast.ingestion.transform.fn.KafkaRecordToFeatureRowDoFn; -import feast.ingestion.values.FailedElement; +import feast.storage.api.writer.FailedElement; import feast.types.FeatureRowProto.FeatureRow; import org.apache.beam.sdk.io.kafka.KafkaIO; import org.apache.beam.sdk.transforms.PTransform; diff --git a/ingestion/src/main/java/feast/ingestion/transform/ValidateFeatureRows.java b/ingestion/src/main/java/feast/ingestion/transform/ValidateFeatureRows.java index 5ca6a710f62..06df06c074c 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/ValidateFeatureRows.java +++ b/ingestion/src/main/java/feast/ingestion/transform/ValidateFeatureRows.java @@ -19,8 +19,8 @@ import com.google.auto.value.AutoValue; import feast.core.FeatureSetProto; import feast.ingestion.transform.fn.ValidateFeatureRowDoFn; -import feast.ingestion.values.FailedElement; import feast.ingestion.values.FeatureSet; +import feast.storage.api.writer.FailedElement; import feast.types.FeatureRowProto.FeatureRow; import java.util.Map; import java.util.stream.Collectors; @@ -36,7 +36,7 @@ public abstract class ValidateFeatureRows extends PTransform, PCollectionTuple> { - public abstract Map getFeatureSets(); + public abstract Map getFeatureSetSpecs(); public abstract TupleTag getSuccessTag(); @@ -49,7 +49,8 @@ public static Builder newBuilder() { @AutoValue.Builder public abstract static class Builder { - public abstract Builder setFeatureSets(Map featureSets); + public abstract Builder setFeatureSetSpecs( + Map featureSets); public abstract Builder setSuccessTag(TupleTag successTag); @@ -62,7 +63,7 @@ public abstract static class Builder { public PCollectionTuple expand(PCollection input) { Map featureSets = - getFeatureSets().entrySet().stream() + getFeatureSetSpecs().entrySet().stream() .map(e -> Pair.of(e.getKey(), new FeatureSet(e.getValue()))) .collect(Collectors.toMap(Pair::getLeft, Pair::getRight)); diff --git a/ingestion/src/main/java/feast/ingestion/transform/WriteFailedElementToBigQuery.java b/ingestion/src/main/java/feast/ingestion/transform/WriteFailedElementToBigQuery.java index cda590b21aa..0da281790c5 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/WriteFailedElementToBigQuery.java +++ b/ingestion/src/main/java/feast/ingestion/transform/WriteFailedElementToBigQuery.java @@ -18,7 +18,7 @@ import com.google.api.services.bigquery.model.TableRow; import com.google.auto.value.AutoValue; -import feast.ingestion.values.FailedElement; +import feast.storage.api.writer.FailedElement; import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO; import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.CreateDisposition; import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.WriteDisposition; diff --git a/ingestion/src/main/java/feast/ingestion/transform/WriteToStore.java b/ingestion/src/main/java/feast/ingestion/transform/WriteToStore.java deleted file mode 100644 index 4e9082f5554..00000000000 --- a/ingestion/src/main/java/feast/ingestion/transform/WriteToStore.java +++ /dev/null @@ -1,168 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.ingestion.transform; - -import com.google.api.services.bigquery.model.TableDataInsertAllResponse.InsertErrors; -import com.google.api.services.bigquery.model.TableRow; -import com.google.auto.value.AutoValue; -import feast.core.FeatureSetProto.FeatureSet; -import feast.core.StoreProto.Store; -import feast.core.StoreProto.Store.BigQueryConfig; -import feast.core.StoreProto.Store.StoreType; -import feast.ingestion.options.ImportOptions; -import feast.ingestion.utils.ResourceUtil; -import feast.ingestion.values.FailedElement; -import feast.store.serving.bigquery.FeatureRowToTableRow; -import feast.store.serving.bigquery.GetTableDestination; -import feast.store.serving.redis.FeatureRowToRedisMutationDoFn; -import feast.store.serving.redis.RedisCustomIO; -import feast.types.FeatureRowProto.FeatureRow; -import java.io.IOException; -import java.util.Map; -import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO; -import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.CreateDisposition; -import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.Method; -import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.WriteDisposition; -import org.apache.beam.sdk.io.gcp.bigquery.BigQueryInsertError; -import org.apache.beam.sdk.io.gcp.bigquery.InsertRetryPolicy; -import org.apache.beam.sdk.io.gcp.bigquery.WriteResult; -import org.apache.beam.sdk.metrics.Counter; -import org.apache.beam.sdk.metrics.Metrics; -import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.MapElements; -import org.apache.beam.sdk.transforms.PTransform; -import org.apache.beam.sdk.transforms.ParDo; -import org.apache.beam.sdk.values.PCollection; -import org.apache.beam.sdk.values.PDone; -import org.apache.beam.sdk.values.TypeDescriptors; -import org.slf4j.Logger; - -@AutoValue -public abstract class WriteToStore extends PTransform, PDone> { - - private static final Logger log = org.slf4j.LoggerFactory.getLogger(WriteToStore.class); - - public static final String METRIC_NAMESPACE = "WriteToStore"; - public static final String ELEMENTS_WRITTEN_METRIC = "elements_written"; - - private static final Counter elementsWritten = - Metrics.counter(METRIC_NAMESPACE, ELEMENTS_WRITTEN_METRIC); - - public abstract Store getStore(); - - public abstract Map getFeatureSets(); - - public static Builder newBuilder() { - return new AutoValue_WriteToStore.Builder(); - } - - @AutoValue.Builder - public abstract static class Builder { - - public abstract Builder setStore(Store store); - - public abstract Builder setFeatureSets(Map featureSets); - - public abstract WriteToStore build(); - } - - @Override - public PDone expand(PCollection input) { - ImportOptions options = input.getPipeline().getOptions().as(ImportOptions.class); - StoreType storeType = getStore().getType(); - - switch (storeType) { - case REDIS: - PCollection redisWriteResult = - input - .apply( - "FeatureRowToRedisMutation", - ParDo.of(new FeatureRowToRedisMutationDoFn(getFeatureSets()))) - .apply("WriteRedisMutationToRedis", RedisCustomIO.write(getStore())); - if (options.getDeadLetterTableSpec() != null) { - redisWriteResult.apply( - WriteFailedElementToBigQuery.newBuilder() - .setTableSpec(options.getDeadLetterTableSpec()) - .setJsonSchema(ResourceUtil.getDeadletterTableSchemaJson()) - .build()); - } - break; - case BIGQUERY: - BigQueryConfig bigqueryConfig = getStore().getBigqueryConfig(); - - WriteResult bigqueryWriteResult = - input.apply( - "WriteTableRowToBigQuery", - BigQueryIO.write() - .to( - new GetTableDestination( - bigqueryConfig.getProjectId(), bigqueryConfig.getDatasetId())) - .withFormatFunction(new FeatureRowToTableRow(options.getJobName())) - .withCreateDisposition(CreateDisposition.CREATE_NEVER) - .withWriteDisposition(WriteDisposition.WRITE_APPEND) - .withExtendedErrorInfo() - .withMethod(Method.STREAMING_INSERTS) - .withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors())); - - if (options.getDeadLetterTableSpec() != null) { - bigqueryWriteResult - .getFailedInsertsWithErr() - .apply( - "WrapBigQueryInsertionError", - ParDo.of( - new DoFn() { - @ProcessElement - public void processElement(ProcessContext context) { - InsertErrors error = context.element().getError(); - TableRow row = context.element().getRow(); - try { - context.output( - FailedElement.newBuilder() - .setErrorMessage(error.toPrettyString()) - .setPayload(row.toPrettyString()) - .setJobName(context.getPipelineOptions().getJobName()) - .setTransformName("WriteTableRowToBigQuery") - .build()); - } catch (IOException e) { - log.error(e.getMessage()); - } - } - })) - .apply( - WriteFailedElementToBigQuery.newBuilder() - .setTableSpec(options.getDeadLetterTableSpec()) - .setJsonSchema(ResourceUtil.getDeadletterTableSchemaJson()) - .build()); - } - break; - default: - log.error("Store type '{}' is not supported. No Feature Row will be written.", storeType); - break; - } - - input.apply( - "IncrementWriteToStoreElementsWrittenCounter", - MapElements.into(TypeDescriptors.booleans()) - .via( - (FeatureRow row) -> { - elementsWritten.inc(); - return true; - })); - - return PDone.in(input.getPipeline()); - } -} diff --git a/ingestion/src/main/java/feast/ingestion/transform/fn/KafkaRecordToFeatureRowDoFn.java b/ingestion/src/main/java/feast/ingestion/transform/fn/KafkaRecordToFeatureRowDoFn.java index 25aafd6ee71..b332c0ca09a 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/fn/KafkaRecordToFeatureRowDoFn.java +++ b/ingestion/src/main/java/feast/ingestion/transform/fn/KafkaRecordToFeatureRowDoFn.java @@ -18,8 +18,7 @@ import com.google.auto.value.AutoValue; import com.google.protobuf.InvalidProtocolBufferException; -import feast.ingestion.transform.ReadFromSource.Builder; -import feast.ingestion.values.FailedElement; +import feast.storage.api.writer.FailedElement; import feast.types.FeatureRowProto.FeatureRow; import java.util.Base64; import org.apache.beam.sdk.io.kafka.KafkaRecord; diff --git a/ingestion/src/main/java/feast/ingestion/transform/fn/ValidateFeatureRowDoFn.java b/ingestion/src/main/java/feast/ingestion/transform/fn/ValidateFeatureRowDoFn.java index c31d3c535e9..85ac3c86faa 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/fn/ValidateFeatureRowDoFn.java +++ b/ingestion/src/main/java/feast/ingestion/transform/fn/ValidateFeatureRowDoFn.java @@ -17,9 +17,9 @@ package feast.ingestion.transform.fn; import com.google.auto.value.AutoValue; -import feast.ingestion.values.FailedElement; import feast.ingestion.values.FeatureSet; import feast.ingestion.values.Field; +import feast.storage.api.writer.FailedElement; import feast.types.FeatureRowProto.FeatureRow; import feast.types.FieldProto; import feast.types.ValueProto.Value.ValCase; diff --git a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteDeadletterRowMetricsDoFn.java b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteDeadletterRowMetricsDoFn.java index 687670c5cf0..b4338cda09b 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteDeadletterRowMetricsDoFn.java +++ b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteDeadletterRowMetricsDoFn.java @@ -20,7 +20,7 @@ import com.timgroup.statsd.NonBlockingStatsDClient; import com.timgroup.statsd.StatsDClient; import com.timgroup.statsd.StatsDClientException; -import feast.ingestion.values.FailedElement; +import feast.storage.api.writer.FailedElement; import org.apache.beam.sdk.transforms.DoFn; import org.slf4j.Logger; diff --git a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFailureMetricsTransform.java b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFailureMetricsTransform.java new file mode 100644 index 00000000000..65a27fa8bf4 --- /dev/null +++ b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFailureMetricsTransform.java @@ -0,0 +1,52 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2019 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.ingestion.transform.metrics; + +import com.google.auto.value.AutoValue; +import feast.ingestion.options.ImportOptions; +import feast.storage.api.writer.FailedElement; +import org.apache.beam.sdk.transforms.PTransform; +import org.apache.beam.sdk.transforms.ParDo; +import org.apache.beam.sdk.values.PCollection; +import org.apache.beam.sdk.values.PDone; + +@AutoValue +public abstract class WriteFailureMetricsTransform + extends PTransform, PDone> { + + public abstract String getStoreName(); + + public static WriteFailureMetricsTransform create(String storeName) { + return new AutoValue_WriteFailureMetricsTransform(storeName); + } + + @Override + public PDone expand(PCollection input) { + ImportOptions options = input.getPipeline().getOptions().as(ImportOptions.class); + if ("statsd".equals(options.getMetricsExporterType())) { + input.apply( + "WriteDeadletterMetrics", + ParDo.of( + WriteDeadletterRowMetricsDoFn.newBuilder() + .setStatsdHost(options.getStatsdHost()) + .setStatsdPort(options.getStatsdPort()) + .setStoreName(getStoreName()) + .build())); + } + return PDone.in(input.getPipeline()); + } +} diff --git a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteMetricsTransform.java b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteSuccessMetricsTransform.java similarity index 65% rename from ingestion/src/main/java/feast/ingestion/transform/metrics/WriteMetricsTransform.java rename to ingestion/src/main/java/feast/ingestion/transform/metrics/WriteSuccessMetricsTransform.java index 8a5869d78ec..37eed7455a9 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteMetricsTransform.java +++ b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteSuccessMetricsTransform.java @@ -18,68 +18,54 @@ import com.google.auto.value.AutoValue; import feast.ingestion.options.ImportOptions; -import feast.ingestion.values.FailedElement; import feast.types.FeatureRowProto.FeatureRow; -import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.GroupByKey; -import org.apache.beam.sdk.transforms.PTransform; -import org.apache.beam.sdk.transforms.ParDo; +import org.apache.beam.sdk.metrics.Counter; +import org.apache.beam.sdk.metrics.Metrics; +import org.apache.beam.sdk.transforms.*; import org.apache.beam.sdk.transforms.windowing.FixedWindows; import org.apache.beam.sdk.transforms.windowing.Window; import org.apache.beam.sdk.values.KV; import org.apache.beam.sdk.values.PCollection; -import org.apache.beam.sdk.values.PCollectionTuple; import org.apache.beam.sdk.values.PDone; -import org.apache.beam.sdk.values.TupleTag; +import org.apache.beam.sdk.values.TypeDescriptors; import org.joda.time.Duration; @AutoValue -public abstract class WriteMetricsTransform extends PTransform { +public abstract class WriteSuccessMetricsTransform + extends PTransform, PDone> { - public abstract String getStoreName(); - - public abstract TupleTag getSuccessTag(); - - public abstract TupleTag getFailureTag(); - - public static Builder newBuilder() { - return new AutoValue_WriteMetricsTransform.Builder(); - } - - @AutoValue.Builder - public abstract static class Builder { + public static final String METRIC_NAMESPACE = "WriteToStoreSuccess"; + public static final String ELEMENTS_WRITTEN_METRIC = "elements_written"; + private static final Counter elementsWritten = + Metrics.counter(METRIC_NAMESPACE, ELEMENTS_WRITTEN_METRIC); - public abstract Builder setStoreName(String storeName); - - public abstract Builder setSuccessTag(TupleTag successTag); - - public abstract Builder setFailureTag(TupleTag failureTag); + public abstract String getStoreName(); - public abstract WriteMetricsTransform build(); + public static WriteSuccessMetricsTransform create(String storeName) { + return new AutoValue_WriteSuccessMetricsTransform(storeName); } @Override - public PDone expand(PCollectionTuple input) { + public PDone expand(PCollection input) { ImportOptions options = input.getPipeline().getOptions().as(ImportOptions.class); + + input.apply( + "IncrementSuccessfulWriteToStoreElementsWrittenCounter", + MapElements.into(TypeDescriptors.booleans()) + .via( + (FeatureRow row) -> { + elementsWritten.inc(); + return true; + })); + switch (options.getMetricsExporterType()) { case "statsd": - input - .get(getFailureTag()) - .apply( - "WriteDeadletterMetrics", - ParDo.of( - WriteDeadletterRowMetricsDoFn.newBuilder() - .setStatsdHost(options.getStatsdHost()) - .setStatsdPort(options.getStatsdPort()) - .setStoreName(getStoreName()) - .build())); // Fixed window is applied so the metric collector will not be overwhelmed with the metrics // data. For validation, only summaries of the values are usually required vs the actual // values. PCollection>> validRowsGroupedByRef = input - .get(getSuccessTag()) .apply( "FixedWindow", Window.into( @@ -119,15 +105,13 @@ public void processElement( return PDone.in(input.getPipeline()); case "none": default: - input - .get(getSuccessTag()) - .apply( - "Noop", - ParDo.of( - new DoFn() { - @ProcessElement - public void processElement(ProcessContext c) {} - })); + input.apply( + "Noop", + ParDo.of( + new DoFn() { + @ProcessElement + public void processElement(ProcessContext c) {} + })); return PDone.in(input.getPipeline()); } } diff --git a/ingestion/src/main/java/feast/ingestion/utils/SpecUtil.java b/ingestion/src/main/java/feast/ingestion/utils/SpecUtil.java index 9163c5b2d6f..f28dfc9ee39 100644 --- a/ingestion/src/main/java/feast/ingestion/utils/SpecUtil.java +++ b/ingestion/src/main/java/feast/ingestion/utils/SpecUtil.java @@ -33,9 +33,10 @@ public class SpecUtil { - public static String getFeatureSetReference(FeatureSet featureSet) { - FeatureSetSpec spec = featureSet.getSpec(); - return String.format("%s/%s:%d", spec.getProject(), spec.getName(), spec.getVersion()); + public static String getFeatureSetReference(FeatureSetSpec featureSetSpec) { + return String.format( + "%s/%s:%d", + featureSetSpec.getProject(), featureSetSpec.getName(), featureSetSpec.getVersion()); } /** Get only feature set specs that matches the subscription */ diff --git a/ingestion/src/main/java/feast/ingestion/utils/StoreUtil.java b/ingestion/src/main/java/feast/ingestion/utils/StoreUtil.java index a02b8626945..1b884333818 100644 --- a/ingestion/src/main/java/feast/ingestion/utils/StoreUtil.java +++ b/ingestion/src/main/java/feast/ingestion/utils/StoreUtil.java @@ -18,39 +18,16 @@ import static feast.types.ValueProto.ValueType; -import com.google.cloud.bigquery.BigQuery; -import com.google.cloud.bigquery.BigQueryOptions; -import com.google.cloud.bigquery.DatasetId; -import com.google.cloud.bigquery.DatasetInfo; -import com.google.cloud.bigquery.Field; -import com.google.cloud.bigquery.Field.Builder; -import com.google.cloud.bigquery.Field.Mode; -import com.google.cloud.bigquery.Schema; import com.google.cloud.bigquery.StandardSQLTypeName; -import com.google.cloud.bigquery.StandardTableDefinition; -import com.google.cloud.bigquery.Table; -import com.google.cloud.bigquery.TableDefinition; -import com.google.cloud.bigquery.TableId; -import com.google.cloud.bigquery.TableInfo; -import com.google.cloud.bigquery.TimePartitioning; -import com.google.cloud.bigquery.TimePartitioning.Type; -import com.google.common.collect.ImmutableMap; -import feast.core.FeatureSetProto.EntitySpec; -import feast.core.FeatureSetProto.FeatureSet; import feast.core.FeatureSetProto.FeatureSetSpec; -import feast.core.FeatureSetProto.FeatureSpec; import feast.core.StoreProto.Store; -import feast.core.StoreProto.Store.RedisConfig; import feast.core.StoreProto.Store.StoreType; +import feast.storage.api.writer.FeatureSink; +import feast.storage.connectors.bigquery.writer.BigQueryFeatureSink; +import feast.storage.connectors.redis.writer.RedisFeatureSink; import feast.types.ValueProto.ValueType.Enum; -import io.lettuce.core.RedisClient; -import io.lettuce.core.RedisConnectionException; -import io.lettuce.core.RedisURI; -import java.util.ArrayList; import java.util.HashMap; -import java.util.List; import java.util.Map; -import org.apache.commons.lang3.tuple.Pair; import org.slf4j.Logger; // TODO: Create partitioned table by default @@ -101,155 +78,19 @@ public class StoreUtil { VALUE_TYPE_TO_STANDARD_SQL_TYPE.put(Enum.BOOL_LIST, StandardSQLTypeName.BOOL); } - public static void setupStore(Store store, FeatureSet featureSet) { + public static FeatureSink getFeatureSink( + Store store, Map featureSetSpecs) { StoreType storeType = store.getType(); switch (storeType) { case REDIS: - StoreUtil.checkRedisConnection(store.getRedisConfig()); - break; + return RedisFeatureSink.builder() + .setRedisConfig(store.getRedisConfig()) + .setFeatureSetSpecs(featureSetSpecs) + .build(); case BIGQUERY: - StoreUtil.setupBigQuery( - featureSet, - store.getBigqueryConfig().getProjectId(), - store.getBigqueryConfig().getDatasetId(), - BigQueryOptions.getDefaultInstance().getService()); - break; + return BigQueryFeatureSink.fromConfig(store.getBigqueryConfig()); default: - log.warn("Store type '{}' is unsupported", storeType); - break; + throw new RuntimeException(String.format("Store type '{}' is unsupported", storeType)); } } - - @SuppressWarnings("DuplicatedCode") - public static TableDefinition createBigQueryTableDefinition(FeatureSetSpec featureSetSpec) { - List fields = new ArrayList<>(); - log.info("Table will have the following fields:"); - - for (EntitySpec entitySpec : featureSetSpec.getEntitiesList()) { - Builder builder = - Field.newBuilder( - entitySpec.getName(), VALUE_TYPE_TO_STANDARD_SQL_TYPE.get(entitySpec.getValueType())); - if (entitySpec.getValueType().name().toLowerCase().endsWith("_list")) { - builder.setMode(Mode.REPEATED); - } - Field field = builder.build(); - log.info("- {}", field.toString()); - fields.add(field); - } - for (FeatureSpec featureSpec : featureSetSpec.getFeaturesList()) { - Builder builder = - Field.newBuilder( - featureSpec.getName(), - VALUE_TYPE_TO_STANDARD_SQL_TYPE.get(featureSpec.getValueType())); - if (featureSpec.getValueType().name().toLowerCase().endsWith("_list")) { - builder.setMode(Mode.REPEATED); - } - Field field = builder.build(); - log.info("- {}", field.toString()); - fields.add(field); - } - - // Refer to protos/feast/core/Store.proto for reserved fields in BigQuery. - Map> - reservedFieldNameToPairOfStandardSQLTypeAndDescription = - ImmutableMap.of( - "event_timestamp", - Pair.of(StandardSQLTypeName.TIMESTAMP, BIGQUERY_EVENT_TIMESTAMP_FIELD_DESCRIPTION), - "created_timestamp", - Pair.of( - StandardSQLTypeName.TIMESTAMP, BIGQUERY_CREATED_TIMESTAMP_FIELD_DESCRIPTION), - "job_id", - Pair.of(StandardSQLTypeName.STRING, BIGQUERY_JOB_ID_FIELD_DESCRIPTION)); - for (Map.Entry> entry : - reservedFieldNameToPairOfStandardSQLTypeAndDescription.entrySet()) { - Field field = - Field.newBuilder(entry.getKey(), entry.getValue().getLeft()) - .setDescription(entry.getValue().getRight()) - .build(); - log.info("- {}", field.toString()); - fields.add(field); - } - - TimePartitioning timePartitioning = - TimePartitioning.newBuilder(Type.DAY).setField("event_timestamp").build(); - log.info("Table partitioning: " + timePartitioning.toString()); - - return StandardTableDefinition.newBuilder() - .setTimePartitioning(timePartitioning) - .setSchema(Schema.of(fields)) - .build(); - } - - /** - * This method ensures that, given a FeatureSetSpec object, the relevant BigQuery table is created - * with the correct schema. - * - *

    Refer to protos/feast/core/Store.proto for the derivation of the table name and schema from - * a FeatureSetSpec object. - * - * @param featureSet FeatureSet object - * @param bigqueryProjectId BigQuery project id - * @param bigqueryDatasetId BigQuery dataset id - * @param bigquery BigQuery service object - */ - public static void setupBigQuery( - FeatureSet featureSet, - String bigqueryProjectId, - String bigqueryDatasetId, - BigQuery bigquery) { - - FeatureSetSpec featureSetSpec = featureSet.getSpec(); - // Ensure BigQuery dataset exists. - DatasetId datasetId = DatasetId.of(bigqueryProjectId, bigqueryDatasetId); - if (bigquery.getDataset(datasetId) == null) { - log.info("Creating dataset '{}' in project '{}'", datasetId.getDataset(), bigqueryProjectId); - bigquery.create(DatasetInfo.of(datasetId)); - } - - String tableName = - String.format( - "%s_%s_v%d", - featureSetSpec.getProject(), featureSetSpec.getName(), featureSetSpec.getVersion()) - .replaceAll("-", "_"); - TableId tableId = TableId.of(bigqueryProjectId, datasetId.getDataset(), tableName); - - // Return if there is an existing table - Table table = bigquery.getTable(tableId); - if (table != null) { - log.info( - "Writing to existing BigQuery table '{}:{}.{}'", - bigqueryProjectId, - datasetId.getDataset(), - tableName); - return; - } - - log.info( - "Creating table '{}' in dataset '{}' in project '{}'", - tableId.getTable(), - datasetId.getDataset(), - bigqueryProjectId); - TableDefinition tableDefinition = createBigQueryTableDefinition(featureSet.getSpec()); - TableInfo tableInfo = TableInfo.of(tableId, tableDefinition); - bigquery.create(tableInfo); - } - - /** - * Ensure Redis is accessible, else throw a RuntimeException. - * - * @param redisConfig Plase refer to feast.core.Store proto - */ - public static void checkRedisConnection(RedisConfig redisConfig) { - RedisClient redisClient = - RedisClient.create(RedisURI.create(redisConfig.getHost(), redisConfig.getPort())); - try { - redisClient.connect(); - } catch (RedisConnectionException e) { - throw new RuntimeException( - String.format( - "Failed to connect to Redis at host: '%s' port: '%d'. Please check that your Redis is running and accessible from Feast.", - redisConfig.getHost(), redisConfig.getPort())); - } - redisClient.shutdown(); - } } diff --git a/ingestion/src/main/java/feast/ingestion/values/FeatureSet.java b/ingestion/src/main/java/feast/ingestion/values/FeatureSet.java index bf07bcec966..758fbd0ba31 100644 --- a/ingestion/src/main/java/feast/ingestion/values/FeatureSet.java +++ b/ingestion/src/main/java/feast/ingestion/values/FeatureSet.java @@ -34,9 +34,9 @@ public class FeatureSet implements Serializable { private final Map fields; - public FeatureSet(FeatureSetProto.FeatureSet featureSet) { - this.reference = getFeatureSetReference(featureSet); - this.fields = getFieldsByName(featureSet.getSpec()); + public FeatureSet(FeatureSetProto.FeatureSetSpec featureSetSpec) { + this.reference = getFeatureSetReference(featureSetSpec); + this.fields = getFieldsByName(featureSetSpec); } public String getReference() { diff --git a/ingestion/src/main/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFn.java b/ingestion/src/main/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFn.java deleted file mode 100644 index 1f5a0f19677..00000000000 --- a/ingestion/src/main/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFn.java +++ /dev/null @@ -1,116 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.store.serving.redis; - -import feast.core.FeatureSetProto.EntitySpec; -import feast.core.FeatureSetProto.FeatureSet; -import feast.core.FeatureSetProto.FeatureSetSpec; -import feast.core.FeatureSetProto.FeatureSpec; -import feast.storage.RedisProto.RedisKey; -import feast.storage.RedisProto.RedisKey.Builder; -import feast.store.serving.redis.RedisCustomIO.Method; -import feast.store.serving.redis.RedisCustomIO.RedisMutation; -import feast.types.FeatureRowProto.FeatureRow; -import feast.types.FieldProto.Field; -import feast.types.ValueProto; -import java.util.HashMap; -import java.util.List; -import java.util.Map; -import java.util.stream.Collectors; -import org.apache.beam.sdk.transforms.DoFn; -import org.slf4j.Logger; - -public class FeatureRowToRedisMutationDoFn extends DoFn { - - private static final Logger log = - org.slf4j.LoggerFactory.getLogger(FeatureRowToRedisMutationDoFn.class); - private Map featureSets; - - public FeatureRowToRedisMutationDoFn(Map featureSets) { - this.featureSets = featureSets; - } - - private RedisKey getKey(FeatureRow featureRow) { - FeatureSet featureSet = featureSets.get(featureRow.getFeatureSet()); - List entityNames = - featureSet.getSpec().getEntitiesList().stream() - .map(EntitySpec::getName) - .sorted() - .collect(Collectors.toList()); - - Map entityFields = new HashMap<>(); - Builder redisKeyBuilder = RedisKey.newBuilder().setFeatureSet(featureRow.getFeatureSet()); - for (Field field : featureRow.getFieldsList()) { - if (entityNames.contains(field.getName())) { - entityFields.putIfAbsent( - field.getName(), - Field.newBuilder().setName(field.getName()).setValue(field.getValue()).build()); - } - } - for (String entityName : entityNames) { - if (entityFields.containsKey(entityName)) { - redisKeyBuilder.addEntities(entityFields.get(entityName)); - } - } - return redisKeyBuilder.build(); - } - - private byte[] getValue(FeatureRow featureRow) { - FeatureSetSpec spec = featureSets.get(featureRow.getFeatureSet()).getSpec(); - - List featureNames = - spec.getFeaturesList().stream().map(FeatureSpec::getName).collect(Collectors.toList()); - Map fieldValueOnlyMap = - featureRow.getFieldsList().stream() - .filter(field -> featureNames.contains(field.getName())) - .distinct() - .collect( - Collectors.toMap( - Field::getName, - field -> Field.newBuilder().setValue(field.getValue()).build())); - - List values = - featureNames.stream() - .sorted() - .map( - featureName -> - fieldValueOnlyMap.getOrDefault( - featureName, - Field.newBuilder().setValue(ValueProto.Value.getDefaultInstance()).build())) - .collect(Collectors.toList()); - - return FeatureRow.newBuilder() - .setEventTimestamp(featureRow.getEventTimestamp()) - .addAllFields(values) - .build() - .toByteArray(); - } - - /** Output a redis mutation object for every feature in the feature row. */ - @ProcessElement - public void processElement(ProcessContext context) { - FeatureRow featureRow = context.element(); - try { - byte[] key = getKey(featureRow).toByteArray(); - byte[] value = getValue(featureRow); - RedisMutation redisMutation = new RedisMutation(Method.SET, key, value, null, null); - context.output(redisMutation); - } catch (Exception e) { - log.error(e.getMessage(), e); - } - } -} diff --git a/ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java b/ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java deleted file mode 100644 index 633c2eb551d..00000000000 --- a/ingestion/src/main/java/feast/store/serving/redis/RedisCustomIO.java +++ /dev/null @@ -1,341 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.store.serving.redis; - -import feast.core.StoreProto; -import feast.ingestion.values.FailedElement; -import feast.retry.Retriable; -import io.lettuce.core.RedisConnectionException; -import java.io.IOException; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.List; -import java.util.concurrent.ExecutionException; -import org.apache.avro.reflect.Nullable; -import org.apache.beam.sdk.coders.AvroCoder; -import org.apache.beam.sdk.coders.DefaultCoder; -import org.apache.beam.sdk.transforms.DoFn; -import org.apache.beam.sdk.transforms.PTransform; -import org.apache.beam.sdk.transforms.ParDo; -import org.apache.beam.sdk.transforms.windowing.GlobalWindow; -import org.apache.beam.sdk.values.PCollection; -import org.apache.commons.lang3.exception.ExceptionUtils; -import org.joda.time.Instant; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -public class RedisCustomIO { - - private static final int DEFAULT_BATCH_SIZE = 1000; - private static final int DEFAULT_TIMEOUT = 2000; - - private static final Logger log = LoggerFactory.getLogger(RedisCustomIO.class); - - private RedisCustomIO() {} - - public static Write write(StoreProto.Store store) { - return new Write(store); - } - - public enum Method { - - /** - * Use APPEND command. If key already exists and is a string, this command appends the value at - * the end of the string. - */ - APPEND, - - /** Use SET command. If key already holds a value, it is overwritten. */ - SET, - - /** - * Use LPUSH command. Insert value at the head of the list stored at key. If key does not exist, - * it is created as empty list before performing the push operations. When key holds a value - * that is not a list, an error is returned. - */ - LPUSH, - - /** - * Use RPUSH command. Insert value at the tail of the list stored at key. If key does not exist, - * it is created as empty list before performing the push operations. When key holds a value - * that is not a list, an error is returned. - */ - RPUSH, - - /** - * Use SADD command. Insert value into a set with a defined key. If key does not exist, it is - * created as empty set before performing the add operations. When key holds a value that is not - * a set, an error is returned. - */ - SADD, - - /** - * Use ZADD command. Adds all the specified members with the specified scores to the sorted set - * stored at key. It is possible to specify multiple score / member pairs. If a specified member - * is already a member of the sorted set, the score is updated and the element reinserted at the - * right position to ensure the correct ordering. - */ - ZADD - } - - @DefaultCoder(AvroCoder.class) - public static class RedisMutation { - - private Method method; - private byte[] key; - private byte[] value; - @Nullable private Long expiryMillis; - @Nullable private Long score; - - public RedisMutation() {} - - public RedisMutation( - Method method, - byte[] key, - byte[] value, - @Nullable Long expiryMillis, - @Nullable Long score) { - this.method = method; - this.key = key; - this.value = value; - this.expiryMillis = expiryMillis; - this.score = score; - } - - public Method getMethod() { - return method; - } - - public void setMethod(Method method) { - this.method = method; - } - - public byte[] getKey() { - return key; - } - - public void setKey(byte[] key) { - this.key = key; - } - - public byte[] getValue() { - return value; - } - - public void setValue(byte[] value) { - this.value = value; - } - - @Nullable - public Long getExpiryMillis() { - return expiryMillis; - } - - public void setExpiryMillis(@Nullable Long expiryMillis) { - this.expiryMillis = expiryMillis; - } - - @Nullable - public Long getScore() { - return score; - } - - public void setScore(@Nullable Long score) { - this.score = score; - } - } - - /** ServingStoreWrite data to a Redis server. */ - public static class Write - extends PTransform, PCollection> { - - private WriteDoFn dofn; - - private Write(StoreProto.Store store) { - this.dofn = new WriteDoFn(store); - } - - public Write withBatchSize(int batchSize) { - this.dofn.withBatchSize(batchSize); - return this; - } - - public Write withTimeout(int timeout) { - this.dofn.withTimeout(timeout); - return this; - } - - @Override - public PCollection expand(PCollection input) { - return input.apply(ParDo.of(dofn)); - } - - public static class WriteDoFn extends DoFn { - - private final List mutations = new ArrayList<>(); - private int batchSize = DEFAULT_BATCH_SIZE; - private int timeout = DEFAULT_TIMEOUT; - private RedisIngestionClient redisIngestionClient; - - WriteDoFn(StoreProto.Store store) { - if (store.getType() == StoreProto.Store.StoreType.REDIS) - this.redisIngestionClient = new RedisStandaloneIngestionClient(store.getRedisConfig()); - } - - public WriteDoFn withBatchSize(int batchSize) { - if (batchSize > 0) { - this.batchSize = batchSize; - } - return this; - } - - public WriteDoFn withTimeout(int timeout) { - if (timeout > 0) { - this.timeout = timeout; - } - return this; - } - - @Setup - public void setup() { - this.redisIngestionClient.setup(); - } - - @StartBundle - public void startBundle() { - try { - redisIngestionClient.connect(); - } catch (RedisConnectionException e) { - log.error("Connection to redis cannot be established ", e); - } - mutations.clear(); - } - - private void executeBatch() throws Exception { - this.redisIngestionClient - .getBackOffExecutor() - .execute( - new Retriable() { - @Override - public void execute() throws ExecutionException, InterruptedException { - if (!redisIngestionClient.isConnected()) { - redisIngestionClient.connect(); - } - mutations.forEach( - mutation -> { - writeRecord(mutation); - if (mutation.getExpiryMillis() != null - && mutation.getExpiryMillis() > 0) { - redisIngestionClient.pexpire( - mutation.getKey(), mutation.getExpiryMillis()); - } - }); - redisIngestionClient.sync(); - mutations.clear(); - } - - @Override - public Boolean isExceptionRetriable(Exception e) { - return e instanceof RedisConnectionException; - } - - @Override - public void cleanUpAfterFailure() {} - }); - } - - private FailedElement toFailedElement( - RedisMutation mutation, Exception exception, String jobName) { - return FailedElement.newBuilder() - .setJobName(jobName) - .setTransformName("RedisCustomIO") - .setPayload(Arrays.toString(mutation.getValue())) - .setErrorMessage(exception.getMessage()) - .setStackTrace(ExceptionUtils.getStackTrace(exception)) - .build(); - } - - @ProcessElement - public void processElement(ProcessContext context) { - RedisMutation mutation = context.element(); - mutations.add(mutation); - if (mutations.size() >= batchSize) { - try { - executeBatch(); - } catch (Exception e) { - mutations.forEach( - failedMutation -> { - FailedElement failedElement = - toFailedElement(failedMutation, e, context.getPipelineOptions().getJobName()); - context.output(failedElement); - }); - mutations.clear(); - } - } - } - - private void writeRecord(RedisMutation mutation) { - switch (mutation.getMethod()) { - case APPEND: - redisIngestionClient.append(mutation.getKey(), mutation.getValue()); - return; - case SET: - redisIngestionClient.set(mutation.getKey(), mutation.getValue()); - return; - case LPUSH: - redisIngestionClient.lpush(mutation.getKey(), mutation.getValue()); - return; - case RPUSH: - redisIngestionClient.rpush(mutation.getKey(), mutation.getValue()); - return; - case SADD: - redisIngestionClient.sadd(mutation.getKey(), mutation.getValue()); - return; - case ZADD: - redisIngestionClient.zadd(mutation.getKey(), mutation.getScore(), mutation.getValue()); - return; - default: - throw new UnsupportedOperationException( - String.format("Not implemented writing records for %s", mutation.getMethod())); - } - } - - @FinishBundle - public void finishBundle(FinishBundleContext context) - throws IOException, InterruptedException { - if (mutations.size() > 0) { - try { - executeBatch(); - } catch (Exception e) { - mutations.forEach( - failedMutation -> { - FailedElement failedElement = - toFailedElement(failedMutation, e, context.getPipelineOptions().getJobName()); - context.output(failedElement, Instant.now(), GlobalWindow.INSTANCE); - }); - mutations.clear(); - } - } - } - - @Teardown - public void teardown() { - redisIngestionClient.shutdown(); - } - } - } -} diff --git a/ingestion/src/test/java/feast/ingestion/ImportJobTest.java b/ingestion/src/test/java/feast/ingestion/ImportJobTest.java index 0b000df0f59..13df73e96a4 100644 --- a/ingestion/src/test/java/feast/ingestion/ImportJobTest.java +++ b/ingestion/src/test/java/feast/ingestion/ImportJobTest.java @@ -188,8 +188,8 @@ public void runPipeline_ShouldWriteToRedisCorrectlyGivenValidSpecAndFeatureRow() IntStream.range(0, IMPORT_JOB_SAMPLE_FEATURE_ROW_SIZE) .forEach( i -> { - FeatureRow randomRow = TestUtil.createRandomFeatureRow(featureSet); - RedisKey redisKey = TestUtil.createRedisKey(featureSet, randomRow); + FeatureRow randomRow = TestUtil.createRandomFeatureRow(featureSet.getSpec()); + RedisKey redisKey = TestUtil.createRedisKey(featureSet.getSpec(), randomRow); input.add(randomRow); List fields = randomRow.getFieldsList().stream() diff --git a/ingestion/src/test/java/feast/ingestion/transform/ValidateFeatureRowsTest.java b/ingestion/src/test/java/feast/ingestion/transform/ValidateFeatureRowsTest.java index 5c9860ed97f..3737a736168 100644 --- a/ingestion/src/test/java/feast/ingestion/transform/ValidateFeatureRowsTest.java +++ b/ingestion/src/test/java/feast/ingestion/transform/ValidateFeatureRowsTest.java @@ -16,13 +16,10 @@ */ package feast.ingestion.transform; -import static org.junit.Assert.*; - import feast.core.FeatureSetProto.EntitySpec; -import feast.core.FeatureSetProto.FeatureSet; import feast.core.FeatureSetProto.FeatureSetSpec; import feast.core.FeatureSetProto.FeatureSpec; -import feast.ingestion.values.FailedElement; +import feast.storage.api.writer.FailedElement; import feast.test.TestUtil; import feast.types.FeatureRowProto.FeatureRow; import feast.types.FieldProto.Field; @@ -52,73 +49,57 @@ public class ValidateFeatureRowsTest { @Test public void shouldWriteSuccessAndFailureTagsCorrectly() { - FeatureSet fs1 = - FeatureSet.newBuilder() - .setSpec( - FeatureSetSpec.newBuilder() - .setName("feature_set") - .setVersion(1) - .setProject("myproject") - .addEntities( - EntitySpec.newBuilder() - .setName("entity_id_primary") - .setValueType(Enum.INT32) - .build()) - .addEntities( - EntitySpec.newBuilder() - .setName("entity_id_secondary") - .setValueType(Enum.STRING) - .build()) - .addFeatures( - FeatureSpec.newBuilder() - .setName("feature_1") - .setValueType(Enum.STRING) - .build()) - .addFeatures( - FeatureSpec.newBuilder() - .setName("feature_2") - .setValueType(Enum.INT64) - .build())) + FeatureSetSpec fs1 = + FeatureSetSpec.newBuilder() + .setName("feature_set") + .setVersion(1) + .setProject("myproject") + .addEntities( + EntitySpec.newBuilder() + .setName("entity_id_primary") + .setValueType(Enum.INT32) + .build()) + .addEntities( + EntitySpec.newBuilder() + .setName("entity_id_secondary") + .setValueType(Enum.STRING) + .build()) + .addFeatures( + FeatureSpec.newBuilder().setName("feature_1").setValueType(Enum.STRING).build()) + .addFeatures( + FeatureSpec.newBuilder().setName("feature_2").setValueType(Enum.INT64).build()) .build(); - FeatureSet fs2 = - FeatureSet.newBuilder() - .setSpec( - FeatureSetSpec.newBuilder() - .setName("feature_set") - .setVersion(2) - .setProject("myproject") - .addEntities( - EntitySpec.newBuilder() - .setName("entity_id_primary") - .setValueType(Enum.INT32) - .build()) - .addEntities( - EntitySpec.newBuilder() - .setName("entity_id_secondary") - .setValueType(Enum.STRING) - .build()) - .addFeatures( - FeatureSpec.newBuilder() - .setName("feature_1") - .setValueType(Enum.STRING) - .build()) - .addFeatures( - FeatureSpec.newBuilder() - .setName("feature_2") - .setValueType(Enum.INT64) - .build())) + FeatureSetSpec fs2 = + FeatureSetSpec.newBuilder() + .setName("feature_set") + .setVersion(2) + .setProject("myproject") + .addEntities( + EntitySpec.newBuilder() + .setName("entity_id_primary") + .setValueType(Enum.INT32) + .build()) + .addEntities( + EntitySpec.newBuilder() + .setName("entity_id_secondary") + .setValueType(Enum.STRING) + .build()) + .addFeatures( + FeatureSpec.newBuilder().setName("feature_1").setValueType(Enum.STRING).build()) + .addFeatures( + FeatureSpec.newBuilder().setName("feature_2").setValueType(Enum.INT64).build()) .build(); - Map featureSets = new HashMap<>(); - featureSets.put("myproject/feature_set:1", fs1); - featureSets.put("myproject/feature_set:2", fs2); + Map featureSetSpecs = new HashMap<>(); + featureSetSpecs.put("myproject/feature_set:1", fs1); + featureSetSpecs.put("myproject/feature_set:2", fs2); List input = new ArrayList<>(); List expected = new ArrayList<>(); - for (FeatureSet featureSet : featureSets.values()) { - FeatureRow randomRow = TestUtil.createRandomFeatureRow(featureSet); + for (FeatureSetSpec featureSetSpec : featureSetSpecs.values()) { + FeatureRow randomRow = TestUtil.createRandomFeatureRow(featureSetSpec); input.add(randomRow); expected.add(randomRow); } @@ -132,7 +113,7 @@ public void shouldWriteSuccessAndFailureTagsCorrectly() { ValidateFeatureRows.newBuilder() .setFailureTag(FAILURE_TAG) .setSuccessTag(SUCCESS_TAG) - .setFeatureSets(featureSets) + .setFeatureSetSpecs(featureSetSpecs) .build()); PAssert.that(output.get(SUCCESS_TAG)).containsInAnyOrder(expected); @@ -143,36 +124,28 @@ public void shouldWriteSuccessAndFailureTagsCorrectly() { @Test public void shouldExcludeUnregisteredFields() { - FeatureSet fs1 = - FeatureSet.newBuilder() - .setSpec( - FeatureSetSpec.newBuilder() - .setName("feature_set") - .setVersion(1) - .setProject("myproject") - .addEntities( - EntitySpec.newBuilder() - .setName("entity_id_primary") - .setValueType(Enum.INT32) - .build()) - .addEntities( - EntitySpec.newBuilder() - .setName("entity_id_secondary") - .setValueType(Enum.STRING) - .build()) - .addFeatures( - FeatureSpec.newBuilder() - .setName("feature_1") - .setValueType(Enum.STRING) - .build()) - .addFeatures( - FeatureSpec.newBuilder() - .setName("feature_2") - .setValueType(Enum.INT64) - .build())) + FeatureSetSpec fs1 = + FeatureSetSpec.newBuilder() + .setName("feature_set") + .setVersion(1) + .setProject("myproject") + .addEntities( + EntitySpec.newBuilder() + .setName("entity_id_primary") + .setValueType(Enum.INT32) + .build()) + .addEntities( + EntitySpec.newBuilder() + .setName("entity_id_secondary") + .setValueType(Enum.STRING) + .build()) + .addFeatures( + FeatureSpec.newBuilder().setName("feature_1").setValueType(Enum.STRING).build()) + .addFeatures( + FeatureSpec.newBuilder().setName("feature_2").setValueType(Enum.INT64).build()) .build(); - Map featureSets = new HashMap<>(); + Map featureSets = new HashMap<>(); featureSets.put("myproject/feature_set:1", fs1); List input = new ArrayList<>(); @@ -196,7 +169,7 @@ public void shouldExcludeUnregisteredFields() { ValidateFeatureRows.newBuilder() .setFailureTag(FAILURE_TAG) .setSuccessTag(SUCCESS_TAG) - .setFeatureSets(featureSets) + .setFeatureSetSpecs(featureSets) .build()); PAssert.that(output.get(SUCCESS_TAG)).containsInAnyOrder(expected); diff --git a/ingestion/src/test/java/feast/ingestion/utils/StoreUtilTest.java b/ingestion/src/test/java/feast/ingestion/utils/StoreUtilTest.java deleted file mode 100644 index 82988121bc8..00000000000 --- a/ingestion/src/test/java/feast/ingestion/utils/StoreUtilTest.java +++ /dev/null @@ -1,211 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.ingestion.utils; - -import static feast.types.ValueProto.ValueType.Enum.*; - -import com.google.cloud.bigquery.BigQuery; -import com.google.cloud.bigquery.Field; -import com.google.cloud.bigquery.Field.Mode; -import com.google.cloud.bigquery.Schema; -import com.google.cloud.bigquery.StandardSQLTypeName; -import feast.core.FeatureSetProto.EntitySpec; -import feast.core.FeatureSetProto.FeatureSet; -import feast.core.FeatureSetProto.FeatureSetSpec; -import feast.core.FeatureSetProto.FeatureSpec; -import java.util.Arrays; -import org.junit.Assert; -import org.junit.Test; -import org.mockito.Mockito; - -public class StoreUtilTest { - - @Test - public void setupBigQuery_shouldCreateTable_givenValidFeatureSetSpec() { - FeatureSet featureSet = - FeatureSet.newBuilder() - .setSpec( - FeatureSetSpec.newBuilder() - .setName("feature_set_1") - .setVersion(1) - .setProject("feast-project") - .addEntities(EntitySpec.newBuilder().setName("entity_1").setValueType(INT32)) - .addFeatures(FeatureSpec.newBuilder().setName("feature_1").setValueType(INT32)) - .addFeatures( - FeatureSpec.newBuilder().setName("feature_2").setValueType(STRING_LIST))) - .build(); - BigQuery mockedBigquery = Mockito.mock(BigQuery.class); - StoreUtil.setupBigQuery(featureSet, "project-1", "dataset_1", mockedBigquery); - } - - @Test - public void createBigQueryTableDefinition_shouldCreateCorrectSchema_givenValidFeatureSetSpec() { - FeatureSetSpec input = - FeatureSetSpec.newBuilder() - .addAllEntities( - Arrays.asList( - EntitySpec.newBuilder().setName("bytes_entity").setValueType(BYTES).build(), - EntitySpec.newBuilder().setName("string_entity").setValueType(STRING).build(), - EntitySpec.newBuilder().setName("int32_entity").setValueType(INT32).build(), - EntitySpec.newBuilder().setName("int64_entity").setValueType(INT64).build(), - EntitySpec.newBuilder().setName("double_entity").setValueType(DOUBLE).build(), - EntitySpec.newBuilder().setName("float_entity").setValueType(FLOAT).build(), - EntitySpec.newBuilder().setName("bool_entity").setValueType(BOOL).build(), - EntitySpec.newBuilder() - .setName("bytes_list_entity") - .setValueType(BYTES_LIST) - .build(), - EntitySpec.newBuilder() - .setName("string_list_entity") - .setValueType(STRING_LIST) - .build(), - EntitySpec.newBuilder() - .setName("int32_list_entity") - .setValueType(INT32_LIST) - .build(), - EntitySpec.newBuilder() - .setName("int64_list_entity") - .setValueType(INT64_LIST) - .build(), - EntitySpec.newBuilder() - .setName("double_list_entity") - .setValueType(DOUBLE_LIST) - .build(), - EntitySpec.newBuilder() - .setName("float_list_entity") - .setValueType(FLOAT_LIST) - .build(), - EntitySpec.newBuilder() - .setName("bool_list_entity") - .setValueType(BOOL_LIST) - .build())) - .addAllFeatures( - Arrays.asList( - FeatureSpec.newBuilder().setName("bytes_feature").setValueType(BYTES).build(), - FeatureSpec.newBuilder().setName("string_feature").setValueType(STRING).build(), - FeatureSpec.newBuilder().setName("int32_feature").setValueType(INT32).build(), - FeatureSpec.newBuilder().setName("int64_feature").setValueType(INT64).build(), - FeatureSpec.newBuilder().setName("double_feature").setValueType(DOUBLE).build(), - FeatureSpec.newBuilder().setName("float_feature").setValueType(FLOAT).build(), - FeatureSpec.newBuilder().setName("bool_feature").setValueType(BOOL).build(), - FeatureSpec.newBuilder() - .setName("bytes_list_feature") - .setValueType(BYTES_LIST) - .build(), - FeatureSpec.newBuilder() - .setName("string_list_feature") - .setValueType(STRING_LIST) - .build(), - FeatureSpec.newBuilder() - .setName("int32_list_feature") - .setValueType(INT32_LIST) - .build(), - FeatureSpec.newBuilder() - .setName("int64_list_feature") - .setValueType(INT64_LIST) - .build(), - FeatureSpec.newBuilder() - .setName("double_list_feature") - .setValueType(DOUBLE_LIST) - .build(), - FeatureSpec.newBuilder() - .setName("float_list_feature") - .setValueType(FLOAT_LIST) - .build(), - FeatureSpec.newBuilder() - .setName("bool_list_feature") - .setValueType(BOOL_LIST) - .build())) - .build(); - - Schema actual = StoreUtil.createBigQueryTableDefinition(input).getSchema(); - - Schema expected = - Schema.of( - Arrays.asList( - // Fields from entity - Field.newBuilder("bytes_entity", StandardSQLTypeName.BYTES).build(), - Field.newBuilder("string_entity", StandardSQLTypeName.STRING).build(), - Field.newBuilder("int32_entity", StandardSQLTypeName.INT64).build(), - Field.newBuilder("int64_entity", StandardSQLTypeName.INT64).build(), - Field.newBuilder("double_entity", StandardSQLTypeName.FLOAT64).build(), - Field.newBuilder("float_entity", StandardSQLTypeName.FLOAT64).build(), - Field.newBuilder("bool_entity", StandardSQLTypeName.BOOL).build(), - Field.newBuilder("bytes_list_entity", StandardSQLTypeName.BYTES) - .setMode(Mode.REPEATED) - .build(), - Field.newBuilder("string_list_entity", StandardSQLTypeName.STRING) - .setMode(Mode.REPEATED) - .build(), - Field.newBuilder("int32_list_entity", StandardSQLTypeName.INT64) - .setMode(Mode.REPEATED) - .build(), - Field.newBuilder("int64_list_entity", StandardSQLTypeName.INT64) - .setMode(Mode.REPEATED) - .build(), - Field.newBuilder("double_list_entity", StandardSQLTypeName.FLOAT64) - .setMode(Mode.REPEATED) - .build(), - Field.newBuilder("float_list_entity", StandardSQLTypeName.FLOAT64) - .setMode(Mode.REPEATED) - .build(), - Field.newBuilder("bool_list_entity", StandardSQLTypeName.BOOL) - .setMode(Mode.REPEATED) - .build(), - // Fields from feature - Field.newBuilder("bytes_feature", StandardSQLTypeName.BYTES).build(), - Field.newBuilder("string_feature", StandardSQLTypeName.STRING).build(), - Field.newBuilder("int32_feature", StandardSQLTypeName.INT64).build(), - Field.newBuilder("int64_feature", StandardSQLTypeName.INT64).build(), - Field.newBuilder("double_feature", StandardSQLTypeName.FLOAT64).build(), - Field.newBuilder("float_feature", StandardSQLTypeName.FLOAT64).build(), - Field.newBuilder("bool_feature", StandardSQLTypeName.BOOL).build(), - Field.newBuilder("bytes_list_feature", StandardSQLTypeName.BYTES) - .setMode(Mode.REPEATED) - .build(), - Field.newBuilder("string_list_feature", StandardSQLTypeName.STRING) - .setMode(Mode.REPEATED) - .build(), - Field.newBuilder("int32_list_feature", StandardSQLTypeName.INT64) - .setMode(Mode.REPEATED) - .build(), - Field.newBuilder("int64_list_feature", StandardSQLTypeName.INT64) - .setMode(Mode.REPEATED) - .build(), - Field.newBuilder("double_list_feature", StandardSQLTypeName.FLOAT64) - .setMode(Mode.REPEATED) - .build(), - Field.newBuilder("float_list_feature", StandardSQLTypeName.FLOAT64) - .setMode(Mode.REPEATED) - .build(), - Field.newBuilder("bool_list_feature", StandardSQLTypeName.BOOL) - .setMode(Mode.REPEATED) - .build(), - // Reserved fields - Field.newBuilder("event_timestamp", StandardSQLTypeName.TIMESTAMP) - .setDescription(StoreUtil.BIGQUERY_EVENT_TIMESTAMP_FIELD_DESCRIPTION) - .build(), - Field.newBuilder("created_timestamp", StandardSQLTypeName.TIMESTAMP) - .setDescription(StoreUtil.BIGQUERY_CREATED_TIMESTAMP_FIELD_DESCRIPTION) - .build(), - Field.newBuilder("job_id", StandardSQLTypeName.STRING) - .setDescription(StoreUtil.BIGQUERY_JOB_ID_FIELD_DESCRIPTION) - .build())); - - Assert.assertEquals(expected, actual); - } -} diff --git a/ingestion/src/test/java/feast/store/serving/redis/RedisCustomIOTest.java b/ingestion/src/test/java/feast/store/serving/redis/RedisCustomIOTest.java deleted file mode 100644 index 75663d24a6a..00000000000 --- a/ingestion/src/test/java/feast/store/serving/redis/RedisCustomIOTest.java +++ /dev/null @@ -1,238 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.store.serving.redis; - -import static feast.test.TestUtil.field; -import static org.hamcrest.CoreMatchers.equalTo; -import static org.hamcrest.MatcherAssert.assertThat; - -import feast.core.StoreProto; -import feast.storage.RedisProto.RedisKey; -import feast.store.serving.redis.RedisCustomIO.Method; -import feast.store.serving.redis.RedisCustomIO.RedisMutation; -import feast.types.FeatureRowProto.FeatureRow; -import feast.types.ValueProto.ValueType.Enum; -import io.lettuce.core.RedisClient; -import io.lettuce.core.RedisURI; -import io.lettuce.core.api.StatefulRedisConnection; -import io.lettuce.core.api.sync.RedisStringCommands; -import io.lettuce.core.codec.ByteArrayCodec; -import java.io.IOException; -import java.util.HashMap; -import java.util.LinkedHashMap; -import java.util.List; -import java.util.concurrent.ScheduledFuture; -import java.util.concurrent.ScheduledThreadPoolExecutor; -import java.util.concurrent.TimeUnit; -import java.util.stream.Collectors; -import org.apache.beam.sdk.testing.PAssert; -import org.apache.beam.sdk.testing.TestPipeline; -import org.apache.beam.sdk.transforms.Count; -import org.apache.beam.sdk.transforms.Create; -import org.apache.beam.sdk.values.PCollection; -import org.junit.After; -import org.junit.Before; -import org.junit.Rule; -import org.junit.Test; -import redis.embedded.Redis; -import redis.embedded.RedisServer; - -public class RedisCustomIOTest { - @Rule public transient TestPipeline p = TestPipeline.create(); - - private static String REDIS_HOST = "localhost"; - private static int REDIS_PORT = 51234; - private Redis redis; - private RedisClient redisClient; - private RedisStringCommands sync; - - @Before - public void setUp() throws IOException { - redis = new RedisServer(REDIS_PORT); - redis.start(); - redisClient = - RedisClient.create(new RedisURI(REDIS_HOST, REDIS_PORT, java.time.Duration.ofMillis(2000))); - StatefulRedisConnection connection = redisClient.connect(new ByteArrayCodec()); - sync = connection.sync(); - } - - @After - public void teardown() { - redisClient.shutdown(); - redis.stop(); - } - - @Test - public void shouldWriteToRedis() { - StoreProto.Store.RedisConfig redisConfig = - StoreProto.Store.RedisConfig.newBuilder().setHost(REDIS_HOST).setPort(REDIS_PORT).build(); - HashMap kvs = new LinkedHashMap<>(); - kvs.put( - RedisKey.newBuilder() - .setFeatureSet("fs:1") - .addEntities(field("entity", 1, Enum.INT64)) - .build(), - FeatureRow.newBuilder() - .setFeatureSet("fs:1") - .addFields(field("entity", 1, Enum.INT64)) - .addFields(field("feature", "one", Enum.STRING)) - .build()); - kvs.put( - RedisKey.newBuilder() - .setFeatureSet("fs:1") - .addEntities(field("entity", 2, Enum.INT64)) - .build(), - FeatureRow.newBuilder() - .setFeatureSet("fs:1") - .addFields(field("entity", 2, Enum.INT64)) - .addFields(field("feature", "two", Enum.STRING)) - .build()); - - List featureRowWrites = - kvs.entrySet().stream() - .map( - kv -> - new RedisMutation( - Method.SET, - kv.getKey().toByteArray(), - kv.getValue().toByteArray(), - null, - null)) - .collect(Collectors.toList()); - - StoreProto.Store store = - StoreProto.Store.newBuilder() - .setRedisConfig(redisConfig) - .setType(StoreProto.Store.StoreType.REDIS) - .build(); - p.apply(Create.of(featureRowWrites)).apply(RedisCustomIO.write(store)); - p.run(); - - kvs.forEach( - (key, value) -> { - byte[] actual = sync.get(key.toByteArray()); - assertThat(actual, equalTo(value.toByteArray())); - }); - } - - @Test(timeout = 10000) - public void shouldRetryFailConnection() throws InterruptedException { - StoreProto.Store.RedisConfig redisConfig = - StoreProto.Store.RedisConfig.newBuilder() - .setHost(REDIS_HOST) - .setPort(REDIS_PORT) - .setMaxRetries(4) - .setInitialBackoffMs(2000) - .build(); - HashMap kvs = new LinkedHashMap<>(); - kvs.put( - RedisKey.newBuilder() - .setFeatureSet("fs:1") - .addEntities(field("entity", 1, Enum.INT64)) - .build(), - FeatureRow.newBuilder() - .setFeatureSet("fs:1") - .addFields(field("entity", 1, Enum.INT64)) - .addFields(field("feature", "one", Enum.STRING)) - .build()); - - List featureRowWrites = - kvs.entrySet().stream() - .map( - kv -> - new RedisMutation( - Method.SET, - kv.getKey().toByteArray(), - kv.getValue().toByteArray(), - null, - null)) - .collect(Collectors.toList()); - - StoreProto.Store store = - StoreProto.Store.newBuilder() - .setRedisConfig(redisConfig) - .setType(StoreProto.Store.StoreType.REDIS) - .build(); - PCollection failedElementCount = - p.apply(Create.of(featureRowWrites)) - .apply(RedisCustomIO.write(store)) - .apply(Count.globally()); - - redis.stop(); - final ScheduledThreadPoolExecutor redisRestartExecutor = new ScheduledThreadPoolExecutor(1); - ScheduledFuture scheduledRedisRestart = - redisRestartExecutor.schedule( - () -> { - redis.start(); - }, - 3, - TimeUnit.SECONDS); - - PAssert.that(failedElementCount).containsInAnyOrder(0L); - p.run(); - scheduledRedisRestart.cancel(true); - - kvs.forEach( - (key, value) -> { - byte[] actual = sync.get(key.toByteArray()); - assertThat(actual, equalTo(value.toByteArray())); - }); - } - - @Test - public void shouldProduceFailedElementIfRetryExceeded() { - StoreProto.Store.RedisConfig redisConfig = - StoreProto.Store.RedisConfig.newBuilder().setHost(REDIS_HOST).setPort(REDIS_PORT).build(); - HashMap kvs = new LinkedHashMap<>(); - kvs.put( - RedisKey.newBuilder() - .setFeatureSet("fs:1") - .addEntities(field("entity", 1, Enum.INT64)) - .build(), - FeatureRow.newBuilder() - .setFeatureSet("fs:1") - .addFields(field("entity", 1, Enum.INT64)) - .addFields(field("feature", "one", Enum.STRING)) - .build()); - - List featureRowWrites = - kvs.entrySet().stream() - .map( - kv -> - new RedisMutation( - Method.SET, - kv.getKey().toByteArray(), - kv.getValue().toByteArray(), - null, - null)) - .collect(Collectors.toList()); - - StoreProto.Store store = - StoreProto.Store.newBuilder() - .setRedisConfig(redisConfig) - .setType(StoreProto.Store.StoreType.REDIS) - .build(); - PCollection failedElementCount = - p.apply(Create.of(featureRowWrites)) - .apply(RedisCustomIO.write(store)) - .apply(Count.globally()); - - redis.stop(); - PAssert.that(failedElementCount).containsInAnyOrder(1L); - p.run(); - } -} diff --git a/ingestion/src/test/java/feast/test/TestUtil.java b/ingestion/src/test/java/feast/test/TestUtil.java index 3cad39e3ec5..2cd3242fb00 100644 --- a/ingestion/src/test/java/feast/test/TestUtil.java +++ b/ingestion/src/test/java/feast/test/TestUtil.java @@ -21,20 +21,13 @@ import com.google.protobuf.ByteString; import com.google.protobuf.util.Timestamps; import feast.core.FeatureSetProto.FeatureSet; -import feast.ingestion.transform.WriteToStore; +import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.ingestion.transform.metrics.WriteSuccessMetricsTransform; import feast.storage.RedisProto.RedisKey; import feast.types.FeatureRowProto.FeatureRow; import feast.types.FeatureRowProto.FeatureRow.Builder; import feast.types.FieldProto.Field; -import feast.types.ValueProto.BoolList; -import feast.types.ValueProto.BytesList; -import feast.types.ValueProto.DoubleList; -import feast.types.ValueProto.FloatList; -import feast.types.ValueProto.Int32List; -import feast.types.ValueProto.Int64List; -import feast.types.ValueProto.StringList; -import feast.types.ValueProto.Value; -import feast.types.ValueProto.ValueType; +import feast.types.ValueProto.*; import java.io.IOException; import java.net.DatagramPacket; import java.net.DatagramSocket; @@ -174,12 +167,15 @@ public static void publishFeatureRowsToKafka( /** * Create a Feature Row with random value according to the FeatureSetSpec * - *

    See {@link #createRandomFeatureRow(FeatureSet, int)} + *

    See {@link #createRandomFeatureRow(FeatureSetSpec, int)} + * + * @param featureSetSpec {@link FeatureSetSpec} + * @return {@link FeatureRow} */ - public static FeatureRow createRandomFeatureRow(FeatureSet featureSet) { + public static FeatureRow createRandomFeatureRow(FeatureSetSpec featureSetSpec) { ThreadLocalRandom random = ThreadLocalRandom.current(); int randomStringSizeMaxSize = 12; - return createRandomFeatureRow(featureSet, random.nextInt(0, randomStringSizeMaxSize) + 4); + return createRandomFeatureRow(featureSetSpec, random.nextInt(0, randomStringSizeMaxSize) + 4); } /** @@ -188,18 +184,18 @@ public static FeatureRow createRandomFeatureRow(FeatureSet featureSet) { *

    The Feature Row created contains fields according to the entities and features defined in * FeatureSet, matching the value type of the field, with randomized value for testing. * - * @param featureSet {@link FeatureSet} + * @param featureSetSpec {@link FeatureSetSpec} * @param randomStringSize number of characters for the generated random string * @return {@link FeatureRow} */ - public static FeatureRow createRandomFeatureRow(FeatureSet featureSet, int randomStringSize) { + public static FeatureRow createRandomFeatureRow( + FeatureSetSpec featureSetSpec, int randomStringSize) { Builder builder = FeatureRow.newBuilder() - .setFeatureSet(getFeatureSetReference(featureSet)) + .setFeatureSet(getFeatureSetReference(featureSetSpec)) .setEventTimestamp(Timestamps.fromMillis(System.currentTimeMillis())); - featureSet - .getSpec() + featureSetSpec .getEntitiesList() .forEach( field -> { @@ -210,8 +206,7 @@ public static FeatureRow createRandomFeatureRow(FeatureSet featureSet, int rando .build()); }); - featureSet - .getSpec() + featureSetSpec .getFeaturesList() .forEach( field -> { @@ -301,15 +296,14 @@ public static Value createRandomValue(ValueType.Enum type, int randomStringSize) *

    The entities in the created {@link RedisKey} will contain the value with matching field name * in the {@link FeatureRow} * - * @param featureSet {@link FeatureSet} + * @param featureSetSpec {@link FeatureSetSpec} * @param row {@link FeatureSet} * @return {@link RedisKey} */ - public static RedisKey createRedisKey(FeatureSet featureSet, FeatureRow row) { + public static RedisKey createRedisKey(FeatureSetSpec featureSetSpec, FeatureRow row) { RedisKey.Builder builder = - RedisKey.newBuilder().setFeatureSet(getFeatureSetReference(featureSet)); - featureSet - .getSpec() + RedisKey.newBuilder().setFeatureSet(getFeatureSetReference(featureSetSpec)); + featureSetSpec .getEntitiesList() .forEach( entityField -> @@ -452,7 +446,9 @@ public static void waitUntilAllElementsAreWrittenToStore( } String writeToStoreMetric = - WriteToStore.METRIC_NAMESPACE + ":" + WriteToStore.ELEMENTS_WRITTEN_METRIC; + WriteSuccessMetricsTransform.METRIC_NAMESPACE + + ":" + + WriteSuccessMetricsTransform.ELEMENTS_WRITTEN_METRIC; long committed = 0; long maxSystemTimeMillis = System.currentTimeMillis() + maxWaitDuration.getMillis(); diff --git a/pom.xml b/pom.xml index 3abb0eb9ace..649ef01865b 100644 --- a/pom.xml +++ b/pom.xml @@ -29,6 +29,8 @@ datatypes/java + storage/api + storage/connectors ingestion core serving diff --git a/serving/pom.xml b/serving/pom.xml index 4cc02dc4510..1390bfdc80c 100644 --- a/serving/pom.xml +++ b/serving/pom.xml @@ -76,6 +76,24 @@ ${project.version} + + dev.feast + feast-storage-api + ${project.version} + + + + dev.feast + feast-storage-connector-redis + ${project.version} + + + + dev.feast + feast-storage-connector-bigquery + ${project.version} + + org.slf4j @@ -114,6 +132,7 @@ io.github.lognet grpc-spring-boot-starter + org.springframework.boot @@ -136,17 +155,6 @@ protobuf-java-util - - io.pebbletemplates - pebble - 3.1.0 - - - - io.lettuce - lettuce-core - - com.google.guava @@ -180,12 +188,14 @@ simpleclient 0.8.0 + io.prometheus simpleclient_hotspot 0.8.0 + io.prometheus @@ -198,17 +208,6 @@ 0.8.0 - - - com.google.cloud - google-cloud-bigquery - - - - com.google.cloud - google-cloud-storage - - com.google.auto.value auto-value-annotations diff --git a/serving/src/main/java/feast/serving/configuration/ServingServiceConfig.java b/serving/src/main/java/feast/serving/configuration/ServingServiceConfig.java index d0ea058baf4..28df853e224 100644 --- a/serving/src/main/java/feast/serving/configuration/ServingServiceConfig.java +++ b/serving/src/main/java/feast/serving/configuration/ServingServiceConfig.java @@ -25,12 +25,12 @@ import feast.core.StoreProto.Store.RedisConfig; import feast.core.StoreProto.Store.Subscription; import feast.serving.FeastProperties; -import feast.serving.service.BigQueryServingService; -import feast.serving.service.JobService; -import feast.serving.service.NoopJobService; -import feast.serving.service.RedisServingService; -import feast.serving.service.ServingService; +import feast.serving.service.*; import feast.serving.specs.CachedSpecService; +import feast.storage.api.retriever.HistoricalRetriever; +import feast.storage.api.retriever.OnlineRetriever; +import feast.storage.connectors.bigquery.retriever.BigQueryHistoricalRetriever; +import feast.storage.connectors.redis.retriever.RedisOnlineRetriever; import io.opentracing.Tracer; import java.util.Map; import org.slf4j.Logger; @@ -79,9 +79,9 @@ public ServingService servingService( switch (store.getType()) { case REDIS: - servingService = - new RedisServingService( - storeConfiguration.getServingRedisConnection(), specService, tracer); + OnlineRetriever redisRetriever = + new RedisOnlineRetriever(storeConfiguration.getServingRedisConnection()); + servingService = new OnlineServingService(redisRetriever, specService, tracer); break; case BIGQUERY: BigQueryConfig bqConfig = store.getBigqueryConfig(); @@ -104,17 +104,20 @@ public ServingService servingService( throw new IllegalArgumentException( "Unable to instantiate jobService for BigQuery store."); } - servingService = - new BigQueryServingService( - bigquery, - bqConfig.getProjectId(), - bqConfig.getDatasetId(), - specService, - jobService, - jobStagingLocation, - feastProperties.getJobs().getBigqueryInitialRetryDelaySecs(), - feastProperties.getJobs().getBigqueryTotalTimeoutSecs(), - storage); + + HistoricalRetriever bqRetriever = + BigQueryHistoricalRetriever.builder() + .setBigquery(bigquery) + .setDatasetId(bqConfig.getDatasetId()) + .setProjectId(bqConfig.getProjectId()) + .setJobStagingLocation(jobStagingLocation) + .setInitialRetryDelaySecs( + feastProperties.getJobs().getBigqueryInitialRetryDelaySecs()) + .setTotalTimeoutSecs(feastProperties.getJobs().getBigqueryTotalTimeoutSecs()) + .setStorage(storage) + .build(); + + servingService = new HistoricalServingService(bqRetriever, specService, jobService); break; case CASSANDRA: case UNRECOGNIZED: diff --git a/serving/src/main/java/feast/serving/service/BigQueryServingService.java b/serving/src/main/java/feast/serving/service/BigQueryServingService.java deleted file mode 100644 index 8e3b7ae53e4..00000000000 --- a/serving/src/main/java/feast/serving/service/BigQueryServingService.java +++ /dev/null @@ -1,282 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.serving.service; - -import static feast.serving.store.bigquery.QueryTemplater.createEntityTableUUIDQuery; -import static feast.serving.store.bigquery.QueryTemplater.generateFullTableName; - -import com.google.cloud.RetryOption; -import com.google.cloud.bigquery.BigQuery; -import com.google.cloud.bigquery.BigQueryException; -import com.google.cloud.bigquery.Field; -import com.google.cloud.bigquery.FormatOptions; -import com.google.cloud.bigquery.Job; -import com.google.cloud.bigquery.JobInfo; -import com.google.cloud.bigquery.LoadJobConfiguration; -import com.google.cloud.bigquery.QueryJobConfiguration; -import com.google.cloud.bigquery.Schema; -import com.google.cloud.bigquery.Table; -import com.google.cloud.bigquery.TableId; -import com.google.cloud.bigquery.TableInfo; -import com.google.cloud.storage.Storage; -import feast.serving.ServingAPIProto; -import feast.serving.ServingAPIProto.DataFormat; -import feast.serving.ServingAPIProto.DatasetSource; -import feast.serving.ServingAPIProto.FeastServingType; -import feast.serving.ServingAPIProto.GetBatchFeaturesRequest; -import feast.serving.ServingAPIProto.GetBatchFeaturesResponse; -import feast.serving.ServingAPIProto.GetFeastServingInfoRequest; -import feast.serving.ServingAPIProto.GetFeastServingInfoResponse; -import feast.serving.ServingAPIProto.GetJobRequest; -import feast.serving.ServingAPIProto.GetJobResponse; -import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest; -import feast.serving.ServingAPIProto.GetOnlineFeaturesResponse; -import feast.serving.ServingAPIProto.JobStatus; -import feast.serving.ServingAPIProto.JobType; -import feast.serving.specs.CachedSpecService; -import feast.serving.specs.FeatureSetRequest; -import feast.serving.store.bigquery.BatchRetrievalQueryRunnable; -import feast.serving.store.bigquery.QueryTemplater; -import feast.serving.store.bigquery.model.FeatureSetInfo; -import io.grpc.Status; -import java.util.List; -import java.util.Optional; -import java.util.UUID; -import java.util.stream.Collectors; -import org.slf4j.Logger; -import org.threeten.bp.Duration; - -public class BigQueryServingService implements ServingService { - - public static final long TEMP_TABLE_EXPIRY_DURATION_MS = Duration.ofDays(1).toMillis(); - private static final Logger log = org.slf4j.LoggerFactory.getLogger(BigQueryServingService.class); - - private final BigQuery bigquery; - private final String projectId; - private final String datasetId; - private final CachedSpecService specService; - private final JobService jobService; - private final String jobStagingLocation; - private final int initialRetryDelaySecs; - private final int totalTimeoutSecs; - private final Storage storage; - - public BigQueryServingService( - BigQuery bigquery, - String projectId, - String datasetId, - CachedSpecService specService, - JobService jobService, - String jobStagingLocation, - int initialRetryDelaySecs, - int totalTimeoutSecs, - Storage storage) { - this.bigquery = bigquery; - this.projectId = projectId; - this.datasetId = datasetId; - this.specService = specService; - this.jobService = jobService; - this.jobStagingLocation = jobStagingLocation; - this.initialRetryDelaySecs = initialRetryDelaySecs; - this.totalTimeoutSecs = totalTimeoutSecs; - this.storage = storage; - } - - /** {@inheritDoc} */ - @Override - public GetFeastServingInfoResponse getFeastServingInfo( - GetFeastServingInfoRequest getFeastServingInfoRequest) { - return GetFeastServingInfoResponse.newBuilder() - .setType(FeastServingType.FEAST_SERVING_TYPE_BATCH) - .setJobStagingLocation(jobStagingLocation) - .build(); - } - - /** {@inheritDoc} */ - @Override - public GetOnlineFeaturesResponse getOnlineFeatures(GetOnlineFeaturesRequest getFeaturesRequest) { - throw Status.UNIMPLEMENTED.withDescription("Method not implemented").asRuntimeException(); - } - - /** {@inheritDoc} */ - @Override - public GetBatchFeaturesResponse getBatchFeatures(GetBatchFeaturesRequest getFeaturesRequest) { - List featureSetRequests = - specService.getFeatureSets(getFeaturesRequest.getFeaturesList()); - - Table entityTable; - String entityTableName; - try { - entityTable = loadEntities(getFeaturesRequest.getDatasetSource()); - - TableId entityTableWithUUIDs = generateUUIDs(entityTable); - entityTableName = generateFullTableName(entityTableWithUUIDs); - } catch (Exception e) { - throw Status.INTERNAL - .withDescription("Unable to load entity dataset to Bigquery") - .asRuntimeException(); - } - - Schema entityTableSchema = entityTable.getDefinition().getSchema(); - List entityNames = - entityTableSchema.getFields().stream() - .map(Field::getName) - .filter(name -> !name.equals("event_timestamp")) - .collect(Collectors.toList()); - - List featureSetInfos = QueryTemplater.getFeatureSetInfos(featureSetRequests); - - String feastJobId = UUID.randomUUID().toString(); - ServingAPIProto.Job feastJob = - ServingAPIProto.Job.newBuilder() - .setId(feastJobId) - .setType(JobType.JOB_TYPE_DOWNLOAD) - .setStatus(JobStatus.JOB_STATUS_PENDING) - .build(); - jobService.upsert(feastJob); - - new Thread( - BatchRetrievalQueryRunnable.builder() - .setEntityTableName(entityTableName) - .setBigquery(bigquery) - .setStorage(storage) - .setJobService(jobService) - .setProjectId(projectId) - .setDatasetId(datasetId) - .setFeastJobId(feastJobId) - .setEntityTableColumnNames(entityNames) - .setFeatureSetInfos(featureSetInfos) - .setJobStagingLocation(jobStagingLocation) - .setInitialRetryDelaySecs(initialRetryDelaySecs) - .setTotalTimeoutSecs(totalTimeoutSecs) - .build()) - .start(); - - return GetBatchFeaturesResponse.newBuilder().setJob(feastJob).build(); - } - - /** {@inheritDoc} */ - @Override - public GetJobResponse getJob(GetJobRequest getJobRequest) { - Optional job = jobService.get(getJobRequest.getJob().getId()); - if (!job.isPresent()) { - throw Status.NOT_FOUND - .withDescription(String.format("Job not found: %s", getJobRequest.getJob().getId())) - .asRuntimeException(); - } - return GetJobResponse.newBuilder().setJob(job.get()).build(); - } - - private Table loadEntities(DatasetSource datasetSource) { - Table loadedEntityTable; - switch (datasetSource.getDatasetSourceCase()) { - case FILE_SOURCE: - try { - // Currently only AVRO format is supported - - if (datasetSource.getFileSource().getDataFormat() != DataFormat.DATA_FORMAT_AVRO) { - throw Status.INVALID_ARGUMENT - .withDescription("Invalid file format, only AVRO is supported.") - .asRuntimeException(); - } - - TableId tableId = TableId.of(projectId, datasetId, createTempTableName()); - log.info("Loading entity rows to: {}.{}.{}", projectId, datasetId, tableId.getTable()); - - LoadJobConfiguration loadJobConfiguration = - LoadJobConfiguration.of( - tableId, datasetSource.getFileSource().getFileUrisList(), FormatOptions.avro()); - loadJobConfiguration = - loadJobConfiguration.toBuilder().setUseAvroLogicalTypes(true).build(); - Job job = bigquery.create(JobInfo.of(loadJobConfiguration)); - waitForJob(job); - - TableInfo expiry = - bigquery - .getTable(tableId) - .toBuilder() - .setExpirationTime(System.currentTimeMillis() + TEMP_TABLE_EXPIRY_DURATION_MS) - .build(); - bigquery.update(expiry); - - loadedEntityTable = bigquery.getTable(tableId); - if (!loadedEntityTable.exists()) { - throw new RuntimeException( - "Unable to create entity dataset table, table already exists"); - } - return loadedEntityTable; - } catch (Exception e) { - log.error("Exception has occurred in loadEntities method: ", e); - throw Status.INTERNAL - .withDescription("Failed to load entity dataset into store: " + e.toString()) - .withCause(e) - .asRuntimeException(); - } - case DATASETSOURCE_NOT_SET: - default: - throw Status.INVALID_ARGUMENT - .withDescription("Data source must be set.") - .asRuntimeException(); - } - } - - private TableId generateUUIDs(Table loadedEntityTable) { - try { - String uuidQuery = - createEntityTableUUIDQuery(generateFullTableName(loadedEntityTable.getTableId())); - QueryJobConfiguration queryJobConfig = - QueryJobConfiguration.newBuilder(uuidQuery) - .setDestinationTable(TableId.of(projectId, datasetId, createTempTableName())) - .build(); - Job queryJob = bigquery.create(JobInfo.of(queryJobConfig)); - Job completedJob = waitForJob(queryJob); - TableInfo expiry = - bigquery - .getTable(queryJobConfig.getDestinationTable()) - .toBuilder() - .setExpirationTime(System.currentTimeMillis() + TEMP_TABLE_EXPIRY_DURATION_MS) - .build(); - bigquery.update(expiry); - queryJobConfig = completedJob.getConfiguration(); - return queryJobConfig.getDestinationTable(); - } catch (InterruptedException | BigQueryException e) { - throw Status.INTERNAL - .withDescription("Failed to load entity dataset into store") - .withCause(e) - .asRuntimeException(); - } - } - - private Job waitForJob(Job queryJob) throws InterruptedException { - Job completedJob = - queryJob.waitFor( - RetryOption.initialRetryDelay(Duration.ofSeconds(initialRetryDelaySecs)), - RetryOption.totalTimeout(Duration.ofSeconds(totalTimeoutSecs))); - if (completedJob == null) { - throw Status.INTERNAL.withDescription("Job no longer exists").asRuntimeException(); - } else if (completedJob.getStatus().getError() != null) { - throw Status.INTERNAL - .withDescription("Job failed: " + completedJob.getStatus().getError()) - .asRuntimeException(); - } - return completedJob; - } - - public static String createTempTableName() { - return "_" + UUID.randomUUID().toString().replace("-", ""); - } -} diff --git a/serving/src/main/java/feast/serving/service/HistoricalServingService.java b/serving/src/main/java/feast/serving/service/HistoricalServingService.java new file mode 100644 index 00000000000..cc6df1b6b5a --- /dev/null +++ b/serving/src/main/java/feast/serving/service/HistoricalServingService.java @@ -0,0 +1,119 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2019 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.serving.service; + +import feast.serving.ServingAPIProto; +import feast.serving.ServingAPIProto.*; +import feast.serving.ServingAPIProto.Job.Builder; +import feast.serving.specs.CachedSpecService; +import feast.storage.api.retriever.FeatureSetRequest; +import feast.storage.api.retriever.HistoricalRetrievalResult; +import feast.storage.api.retriever.HistoricalRetriever; +import io.grpc.Status; +import java.util.List; +import java.util.Optional; +import java.util.UUID; +import org.slf4j.Logger; + +public class HistoricalServingService implements ServingService { + + private static final Logger log = + org.slf4j.LoggerFactory.getLogger(HistoricalServingService.class); + + private final HistoricalRetriever retriever; + private final CachedSpecService specService; + private final JobService jobService; + + public HistoricalServingService( + HistoricalRetriever retriever, CachedSpecService specService, JobService jobService) { + this.retriever = retriever; + this.specService = specService; + this.jobService = jobService; + } + + /** {@inheritDoc} */ + @Override + public GetFeastServingInfoResponse getFeastServingInfo( + GetFeastServingInfoRequest getFeastServingInfoRequest) { + return GetFeastServingInfoResponse.newBuilder() + .setType(FeastServingType.FEAST_SERVING_TYPE_BATCH) + .setJobStagingLocation(retriever.getStagingLocation()) + .build(); + } + + /** {@inheritDoc} */ + @Override + public GetOnlineFeaturesResponse getOnlineFeatures(GetOnlineFeaturesRequest getFeaturesRequest) { + throw Status.UNIMPLEMENTED.withDescription("Method not implemented").asRuntimeException(); + } + + /** {@inheritDoc} */ + @Override + public GetBatchFeaturesResponse getBatchFeatures(GetBatchFeaturesRequest getFeaturesRequest) { + List featureSetRequests = + specService.getFeatureSets(getFeaturesRequest.getFeaturesList()); + String retrievalId = UUID.randomUUID().toString(); + Job runningJob = + Job.newBuilder() + .setId(retrievalId) + .setType(JobType.JOB_TYPE_DOWNLOAD) + .setStatus(JobStatus.JOB_STATUS_RUNNING) + .build(); + jobService.upsert(runningJob); + Thread thread = + new Thread( + new Runnable() { + @Override + public void run() { + HistoricalRetrievalResult result = + retriever.getHistoricalFeatures( + retrievalId, getFeaturesRequest.getDatasetSource(), featureSetRequests); + jobService.upsert(resultToJob(result)); + } + }); + thread.start(); + + return GetBatchFeaturesResponse.newBuilder().setJob(runningJob).build(); + } + + /** {@inheritDoc} */ + @Override + public GetJobResponse getJob(GetJobRequest getJobRequest) { + Optional job = jobService.get(getJobRequest.getJob().getId()); + if (!job.isPresent()) { + throw Status.NOT_FOUND + .withDescription(String.format("Job not found: %s", getJobRequest.getJob().getId())) + .asRuntimeException(); + } + return GetJobResponse.newBuilder().setJob(job.get()).build(); + } + + private Job resultToJob(HistoricalRetrievalResult result) { + Builder builder = + Job.newBuilder() + .setId(result.getId()) + .setType(JobType.JOB_TYPE_DOWNLOAD) + .setStatus(result.getStatus()); + if (result.hasError()) { + return builder.setError(result.getError()).build(); + } + return builder + .addAllFileUris(result.getFileUris()) + .setDataFormat(result.getDataFormat()) + .build(); + } +} diff --git a/serving/src/main/java/feast/serving/service/OnlineServingService.java b/serving/src/main/java/feast/serving/service/OnlineServingService.java new file mode 100644 index 00000000000..30addd2b9f2 --- /dev/null +++ b/serving/src/main/java/feast/serving/service/OnlineServingService.java @@ -0,0 +1,176 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2019 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.serving.service; + +import com.google.common.collect.Maps; +import com.google.protobuf.Duration; +import feast.serving.ServingAPIProto.*; +import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest.EntityRow; +import feast.serving.ServingAPIProto.GetOnlineFeaturesResponse.FieldValues; +import feast.serving.specs.CachedSpecService; +import feast.serving.util.Metrics; +import feast.serving.util.RefUtil; +import feast.storage.api.retriever.FeatureSetRequest; +import feast.storage.api.retriever.OnlineRetriever; +import feast.types.FeatureRowProto.FeatureRow; +import feast.types.ValueProto.Value; +import io.grpc.Status; +import io.opentracing.Scope; +import io.opentracing.Tracer; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; +import org.slf4j.Logger; + +public class OnlineServingService implements ServingService { + + private static final Logger log = org.slf4j.LoggerFactory.getLogger(OnlineServingService.class); + private final CachedSpecService specService; + private final Tracer tracer; + private final OnlineRetriever retriever; + + public OnlineServingService( + OnlineRetriever retriever, CachedSpecService specService, Tracer tracer) { + this.retriever = retriever; + this.specService = specService; + this.tracer = tracer; + } + + /** {@inheritDoc} */ + @Override + public GetFeastServingInfoResponse getFeastServingInfo( + GetFeastServingInfoRequest getFeastServingInfoRequest) { + return GetFeastServingInfoResponse.newBuilder() + .setType(FeastServingType.FEAST_SERVING_TYPE_ONLINE) + .build(); + } + + /** {@inheritDoc} */ + @Override + public GetOnlineFeaturesResponse getOnlineFeatures(GetOnlineFeaturesRequest request) { + try (Scope scope = tracer.buildSpan("getOnlineFeatures").startActive(true)) { + GetOnlineFeaturesResponse.Builder getOnlineFeaturesResponseBuilder = + GetOnlineFeaturesResponse.newBuilder(); + List featureSetRequests = + specService.getFeatureSets(request.getFeaturesList()); + List entityRows = request.getEntityRowsList(); + Map> featureValuesMap = + entityRows.stream() + .collect(Collectors.toMap(row -> row, row -> Maps.newHashMap(row.getFieldsMap()))); + // Get all feature rows from the retriever. Each feature row list corresponds to a single + // feature set request. + List> featureRows = + retriever.getOnlineFeatures(entityRows, featureSetRequests); + + // For each feature set request, read the feature rows returned by the retriever, and + // populate the featureValuesMap with the feature values corresponding to that entity row. + for (var fsIdx = 0; fsIdx < featureRows.size(); fsIdx++) { + List featureRowsForFs = featureRows.get(fsIdx); + FeatureSetRequest featureSetRequest = featureSetRequests.get(fsIdx); + + String project = featureSetRequest.getSpec().getProject(); + + // In order to return values containing the same feature references provided by the user, + // we reuse the feature references in the request as the keys in the featureValuesMap + Map refsByName = featureSetRequest.getFeatureRefsByName(); + + // Each feature row returned (per feature set request) corresponds to a given entity row. + // For each feature row, update the featureValuesMap. + for (var entityRowIdx = 0; entityRowIdx < entityRows.size(); entityRowIdx++) { + FeatureRow featureRow = featureRowsForFs.get(entityRowIdx); + EntityRow entityRow = entityRows.get(entityRowIdx); + + // If the row is stale, put an empty value into the featureValuesMap. + if (isStale(featureSetRequest, entityRow, featureRow)) { + featureSetRequest + .getFeatureReferences() + .parallelStream() + .forEach( + ref -> { + populateStaleKeyCountMetrics(project, ref); + featureValuesMap + .get(entityRow) + .put(RefUtil.generateFeatureStringRef(ref), Value.newBuilder().build()); + }); + + } else { + populateRequestCountMetrics(featureSetRequest); + + // Else populate the featureValueMap at this entityRow with the values in the feature + // row. + featureRow.getFieldsList().stream() + .filter(field -> refsByName.containsKey(field.getName())) + .forEach( + field -> { + FeatureReference ref = refsByName.get(field.getName()); + String id = RefUtil.generateFeatureStringRef(ref); + featureValuesMap.get(entityRow).put(id, field.getValue()); + }); + } + } + } + + List fieldValues = + featureValuesMap.values().stream() + .map(valueMap -> FieldValues.newBuilder().putAllFields(valueMap).build()) + .collect(Collectors.toList()); + return getOnlineFeaturesResponseBuilder.addAllFieldValues(fieldValues).build(); + } + } + + private void populateStaleKeyCountMetrics(String project, FeatureReference ref) { + Metrics.staleKeyCount + .labels(project, RefUtil.generateFeatureStringRefWithoutProject(ref)) + .inc(); + } + + private void populateRequestCountMetrics(FeatureSetRequest featureSetRequest) { + String project = featureSetRequest.getSpec().getProject(); + featureSetRequest + .getFeatureReferences() + .parallelStream() + .forEach( + ref -> + Metrics.requestCount + .labels(project, RefUtil.generateFeatureStringRefWithoutProject(ref)) + .inc()); + } + + @Override + public GetBatchFeaturesResponse getBatchFeatures(GetBatchFeaturesRequest getFeaturesRequest) { + throw Status.UNIMPLEMENTED.withDescription("Method not implemented").asRuntimeException(); + } + + @Override + public GetJobResponse getJob(GetJobRequest getJobRequest) { + throw Status.UNIMPLEMENTED.withDescription("Method not implemented").asRuntimeException(); + } + + private boolean isStale( + FeatureSetRequest featureSetRequest, EntityRow entityRow, FeatureRow featureRow) { + Duration maxAge = featureSetRequest.getSpec().getMaxAge(); + if (maxAge.equals(Duration.getDefaultInstance())) { + return false; + } + long givenTimestamp = entityRow.getEntityTimestamp().getSeconds(); + if (givenTimestamp == 0) { + givenTimestamp = System.currentTimeMillis() / 1000; + } + long timeDifference = givenTimestamp - featureRow.getEventTimestamp().getSeconds(); + return timeDifference > maxAge.getSeconds(); + } +} diff --git a/serving/src/main/java/feast/serving/service/RedisServingService.java b/serving/src/main/java/feast/serving/service/RedisServingService.java deleted file mode 100644 index 78d9d9cebe4..00000000000 --- a/serving/src/main/java/feast/serving/service/RedisServingService.java +++ /dev/null @@ -1,345 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.serving.service; - -import static feast.serving.util.Metrics.invalidEncodingCount; -import static feast.serving.util.Metrics.missingKeyCount; -import static feast.serving.util.Metrics.requestCount; -import static feast.serving.util.Metrics.requestLatency; -import static feast.serving.util.Metrics.staleKeyCount; -import static feast.serving.util.RefUtil.generateFeatureSetStringRef; -import static feast.serving.util.RefUtil.generateFeatureStringRef; - -import com.google.common.collect.Maps; -import com.google.protobuf.AbstractMessageLite; -import com.google.protobuf.Duration; -import com.google.protobuf.InvalidProtocolBufferException; -import feast.core.FeatureSetProto.EntitySpec; -import feast.core.FeatureSetProto.FeatureSetSpec; -import feast.serving.ServingAPIProto.FeastServingType; -import feast.serving.ServingAPIProto.FeatureReference; -import feast.serving.ServingAPIProto.GetBatchFeaturesRequest; -import feast.serving.ServingAPIProto.GetBatchFeaturesResponse; -import feast.serving.ServingAPIProto.GetFeastServingInfoRequest; -import feast.serving.ServingAPIProto.GetFeastServingInfoResponse; -import feast.serving.ServingAPIProto.GetJobRequest; -import feast.serving.ServingAPIProto.GetJobResponse; -import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest; -import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest.EntityRow; -import feast.serving.ServingAPIProto.GetOnlineFeaturesResponse; -import feast.serving.ServingAPIProto.GetOnlineFeaturesResponse.FieldValues; -import feast.serving.encoding.FeatureRowDecoder; -import feast.serving.specs.CachedSpecService; -import feast.serving.specs.FeatureSetRequest; -import feast.serving.util.RefUtil; -import feast.storage.RedisProto.RedisKey; -import feast.types.FeatureRowProto.FeatureRow; -import feast.types.FieldProto.Field; -import feast.types.ValueProto.Value; -import io.grpc.Status; -import io.lettuce.core.api.StatefulRedisConnection; -import io.lettuce.core.api.sync.RedisCommands; -import io.opentracing.Scope; -import io.opentracing.Tracer; -import java.util.List; -import java.util.Map; -import java.util.concurrent.ExecutionException; -import java.util.stream.Collectors; -import org.slf4j.Logger; - -public class RedisServingService implements ServingService { - - private static final Logger log = org.slf4j.LoggerFactory.getLogger(RedisServingService.class); - private final CachedSpecService specService; - private final Tracer tracer; - private final RedisCommands syncCommands; - - public RedisServingService( - StatefulRedisConnection connection, - CachedSpecService specService, - Tracer tracer) { - this.syncCommands = connection.sync(); - this.specService = specService; - this.tracer = tracer; - } - - /** {@inheritDoc} */ - @Override - public GetFeastServingInfoResponse getFeastServingInfo( - GetFeastServingInfoRequest getFeastServingInfoRequest) { - return GetFeastServingInfoResponse.newBuilder() - .setType(FeastServingType.FEAST_SERVING_TYPE_ONLINE) - .build(); - } - - /** {@inheritDoc} */ - @Override - public GetOnlineFeaturesResponse getOnlineFeatures(GetOnlineFeaturesRequest request) { - try (Scope scope = tracer.buildSpan("Redis-getOnlineFeatures").startActive(true)) { - GetOnlineFeaturesResponse.Builder getOnlineFeaturesResponseBuilder = - GetOnlineFeaturesResponse.newBuilder(); - - List entityRows = request.getEntityRowsList(); - Map> featureValuesMap = - entityRows.stream() - .collect(Collectors.toMap(row -> row, row -> Maps.newHashMap(row.getFieldsMap()))); - List featureSetRequests = - specService.getFeatureSets(request.getFeaturesList()); - for (FeatureSetRequest featureSetRequest : featureSetRequests) { - - List featureSetEntityNames = - featureSetRequest.getSpec().getEntitiesList().stream() - .map(EntitySpec::getName) - .collect(Collectors.toList()); - - List redisKeys = - getRedisKeys(featureSetEntityNames, entityRows, featureSetRequest.getSpec()); - - try { - sendAndProcessMultiGet(redisKeys, entityRows, featureValuesMap, featureSetRequest); - } catch (InvalidProtocolBufferException | ExecutionException e) { - throw Status.INTERNAL - .withDescription("Unable to parse protobuf while retrieving feature") - .withCause(e) - .asRuntimeException(); - } - } - List fieldValues = - featureValuesMap.values().stream() - .map(valueMap -> FieldValues.newBuilder().putAllFields(valueMap).build()) - .collect(Collectors.toList()); - return getOnlineFeaturesResponseBuilder.addAllFieldValues(fieldValues).build(); - } - } - - @Override - public GetBatchFeaturesResponse getBatchFeatures(GetBatchFeaturesRequest getFeaturesRequest) { - throw Status.UNIMPLEMENTED.withDescription("Method not implemented").asRuntimeException(); - } - - @Override - public GetJobResponse getJob(GetJobRequest getJobRequest) { - throw Status.UNIMPLEMENTED.withDescription("Method not implemented").asRuntimeException(); - } - - /** - * Build the redis keys for retrieval from the store. - * - * @param featureSetEntityNames entity names that actually belong to the featureSet - * @param entityRows entity values to retrieve for - * @param featureSetSpec featureSetSpec of the features to retrieve - * @return list of RedisKeys - */ - private List getRedisKeys( - List featureSetEntityNames, - List entityRows, - FeatureSetSpec featureSetSpec) { - try (Scope scope = tracer.buildSpan("Redis-makeRedisKeys").startActive(true)) { - String featureSetRef = generateFeatureSetStringRef(featureSetSpec); - List redisKeys = - entityRows.stream() - .map(row -> makeRedisKey(featureSetRef, featureSetEntityNames, row)) - .collect(Collectors.toList()); - return redisKeys; - } - } - - /** - * Create {@link RedisKey} - * - * @param featureSet featureSet reference of the feature. E.g. feature_set_1:1 - * @param featureSetEntityNames entity names that belong to the featureSet - * @param entityRow entityRow to build the key from - * @return {@link RedisKey} - */ - private RedisKey makeRedisKey( - String featureSet, List featureSetEntityNames, EntityRow entityRow) { - RedisKey.Builder builder = RedisKey.newBuilder().setFeatureSet(featureSet); - Map fieldsMap = entityRow.getFieldsMap(); - featureSetEntityNames.sort(String::compareTo); - for (int i = 0; i < featureSetEntityNames.size(); i++) { - String entityName = featureSetEntityNames.get(i); - - if (!fieldsMap.containsKey(entityName)) { - throw Status.INVALID_ARGUMENT - .withDescription( - String.format( - "Entity row fields \"%s\" does not contain required entity field \"%s\"", - fieldsMap.keySet().toString(), entityName)) - .asRuntimeException(); - } - - builder.addEntities( - Field.newBuilder().setName(entityName).setValue(fieldsMap.get(entityName))); - } - return builder.build(); - } - - private void sendAndProcessMultiGet( - List redisKeys, - List entityRows, - Map> featureValuesMap, - FeatureSetRequest featureSetRequest) - throws InvalidProtocolBufferException, ExecutionException { - - List values = sendMultiGet(redisKeys); - long startTime = System.currentTimeMillis(); - try (Scope scope = tracer.buildSpan("Redis-processResponse").startActive(true)) { - FeatureSetSpec spec = featureSetRequest.getSpec(); - - Map nullValues = - featureSetRequest.getFeatureReferences().stream() - .collect( - Collectors.toMap( - RefUtil::generateFeatureStringRef, - featureReference -> Value.newBuilder().build())); - - for (int i = 0; i < values.size(); i++) { - EntityRow entityRow = entityRows.get(i); - Map featureValues = featureValuesMap.get(entityRow); - - byte[] value = values.get(i); - if (value == null) { - featureSetRequest - .getFeatureReferences() - .parallelStream() - .forEach( - request -> - missingKeyCount - .labels( - spec.getProject(), - String.format("%s:%d", request.getName(), request.getVersion())) - .inc()); - featureValues.putAll(nullValues); - continue; - } - - FeatureRow featureRow = FeatureRow.parseFrom(value); - String featureSetRef = redisKeys.get(i).getFeatureSet(); - FeatureRowDecoder decoder = - new FeatureRowDecoder(featureSetRef, specService.getFeatureSetSpec(featureSetRef)); - if (decoder.isEncoded(featureRow)) { - if (decoder.isEncodingValid(featureRow)) { - featureRow = decoder.decode(featureRow); - } else { - featureSetRequest - .getFeatureReferences() - .parallelStream() - .forEach( - request -> - invalidEncodingCount - .labels( - spec.getProject(), - String.format("%s:%d", request.getName(), request.getVersion())) - .inc()); - featureValues.putAll(nullValues); - continue; - } - } - - boolean stale = isStale(featureSetRequest, entityRow, featureRow); - if (stale) { - featureSetRequest - .getFeatureReferences() - .parallelStream() - .forEach( - request -> - staleKeyCount - .labels( - spec.getProject(), - String.format("%s:%d", request.getName(), request.getVersion())) - .inc()); - featureValues.putAll(nullValues); - continue; - } - - featureSetRequest - .getFeatureReferences() - .parallelStream() - .forEach( - request -> - requestCount - .labels( - spec.getProject(), - String.format("%s:%d", request.getName(), request.getVersion())) - .inc()); - - Map featureNames = - featureSetRequest.getFeatureReferences().stream() - .collect( - Collectors.toMap( - FeatureReference::getName, featureReference -> featureReference)); - featureRow.getFieldsList().stream() - .filter(field -> featureNames.keySet().contains(field.getName())) - .forEach( - field -> { - FeatureReference ref = featureNames.get(field.getName()); - String id = generateFeatureStringRef(ref); - featureValues.put(id, field.getValue()); - }); - } - } finally { - requestLatency - .labels("processResponse") - .observe((System.currentTimeMillis() - startTime) / 1000); - } - } - - private boolean isStale( - FeatureSetRequest featureSetRequest, EntityRow entityRow, FeatureRow featureRow) { - if (featureSetRequest.getSpec().getMaxAge().equals(Duration.getDefaultInstance())) { - return false; - } - long givenTimestamp = entityRow.getEntityTimestamp().getSeconds(); - if (givenTimestamp == 0) { - givenTimestamp = System.currentTimeMillis() / 1000; - } - long timeDifference = givenTimestamp - featureRow.getEventTimestamp().getSeconds(); - return timeDifference > featureSetRequest.getSpec().getMaxAge().getSeconds(); - } - - /** - * Send a list of get request as an mget - * - * @param keys list of {@link RedisKey} - * @return list of {@link FeatureRow} in primitive byte representation for each {@link RedisKey} - */ - private List sendMultiGet(List keys) { - try (Scope scope = tracer.buildSpan("Redis-sendMultiGet").startActive(true)) { - long startTime = System.currentTimeMillis(); - try { - byte[][] binaryKeys = - keys.stream() - .map(AbstractMessageLite::toByteArray) - .collect(Collectors.toList()) - .toArray(new byte[0][0]); - return syncCommands.mget(binaryKeys).stream() - .map(keyValue -> keyValue.getValueOrElse(null)) - .collect(Collectors.toList()); - } catch (Exception e) { - throw Status.NOT_FOUND - .withDescription("Unable to retrieve feature from Redis") - .withCause(e) - .asRuntimeException(); - } finally { - requestLatency - .labels("sendMultiGet") - .observe((System.currentTimeMillis() - startTime) / 1000d); - } - } - } -} diff --git a/serving/src/main/java/feast/serving/service/ServingService.java b/serving/src/main/java/feast/serving/service/ServingService.java index 83adcb73ba8..5e662229eeb 100644 --- a/serving/src/main/java/feast/serving/service/ServingService.java +++ b/serving/src/main/java/feast/serving/service/ServingService.java @@ -26,12 +26,75 @@ import feast.serving.ServingAPIProto.GetOnlineFeaturesResponse; public interface ServingService { + /** + * Get information about the Feast serving deployment. + * + *

    For Bigquery deployments, this includes the default job staging location to load + * intermediate files to. Otherwise, this method only returns the current Feast Serving backing + * store type. + * + * @param getFeastServingInfoRequest {@link GetFeastServingInfoRequest} + * @return {@link GetFeastServingInfoResponse} + */ GetFeastServingInfoResponse getFeastServingInfo( GetFeastServingInfoRequest getFeastServingInfoRequest); + /** + * Get features from an online serving store, given a list of {@link + * feast.serving.ServingAPIProto.FeatureReference}s to retrieve, and list of {@link + * feast.serving.ServingAPIProto.GetOnlineFeaturesRequest.EntityRow}s to join the retrieved values + * to. + * + *

    Features can be queried across feature sets, but each {@link + * feast.serving.ServingAPIProto.GetOnlineFeaturesRequest.EntityRow} must contain all entities for + * all feature sets included in the request. + * + *

    This request is fulfilled synchronously. + * + * @param getFeaturesRequest {@link GetOnlineFeaturesRequest} containing list of {@link + * feast.serving.ServingAPIProto.FeatureReference}s to retrieve and list of {@link + * feast.serving.ServingAPIProto.GetOnlineFeaturesRequest.EntityRow}s to join the retrieved + * values to. + * @return {@link GetOnlineFeaturesResponse} with list of {@link + * feast.serving.ServingAPIProto.GetOnlineFeaturesResponse.FieldValues} for each {@link + * feast.serving.ServingAPIProto.GetOnlineFeaturesRequest.EntityRow} supplied. + */ GetOnlineFeaturesResponse getOnlineFeatures(GetOnlineFeaturesRequest getFeaturesRequest); + /** + * Get features from a batch serving store, given a list of {@link + * feast.serving.ServingAPIProto.FeatureReference}s to retrieve, and {@link + * feast.serving.ServingAPIProto.DatasetSource} pointing to remote location of dataset to join + * retrieved features to. All columns in the provided dataset will be preserved in the output + * dataset. + * + *

    Due to the potential size of batch retrieval requests, this request is fulfilled + * asynchronously, and returns a retrieval job id, which when supplied to {@link + * #getJob(GetJobRequest)} will return the status of the retrieval job. + * + * @param getFeaturesRequest {@link GetBatchFeaturesRequest} containing a list of {@link + * feast.serving.ServingAPIProto.FeatureReference}s to retrieve, and {@link + * feast.serving.ServingAPIProto.DatasetSource} pointing to remote location of dataset to join + * retrieved features to. + * @return {@link GetBatchFeaturesResponse} containing reference to a retrieval {@link + * feast.serving.ServingAPIProto.Job}. + */ GetBatchFeaturesResponse getBatchFeatures(GetBatchFeaturesRequest getFeaturesRequest); + /** + * Get the status of a retrieval job from a batch serving store. + * + *

    The client should check the status of the returned job periodically by calling ReloadJob to + * determine if the job has completed successfully or with an error. If the job completes + * successfully i.e. status = JOB_STATUS_DONE with no error, then the client can check the + * file_uris for the location to download feature values data. The client is assumed to have + * access to these file URIs. + * + *

    If an error occurred during retrieval, the {@link GetJobResponse} will also contain the + * error that resulted in termination. + * + * @param getJobRequest {@link GetJobRequest} containing reference to a retrieval job + * @return {@link GetJobResponse} + */ GetJobResponse getJob(GetJobRequest getJobRequest); } diff --git a/serving/src/main/java/feast/serving/specs/CachedSpecService.java b/serving/src/main/java/feast/serving/specs/CachedSpecService.java index 12a8242da13..47f4934d52c 100644 --- a/serving/src/main/java/feast/serving/specs/CachedSpecService.java +++ b/serving/src/main/java/feast/serving/specs/CachedSpecService.java @@ -36,6 +36,7 @@ import feast.core.StoreProto.Store.Subscription; import feast.serving.ServingAPIProto.FeatureReference; import feast.serving.exception.SpecRetrievalException; +import feast.storage.api.retriever.FeatureSetRequest; import io.grpc.StatusRuntimeException; import io.prometheus.client.Gauge; import java.io.IOException; diff --git a/serving/src/main/java/feast/serving/util/RefUtil.java b/serving/src/main/java/feast/serving/util/RefUtil.java index 74de3e65620..c3bcb0827a2 100644 --- a/serving/src/main/java/feast/serving/util/RefUtil.java +++ b/serving/src/main/java/feast/serving/util/RefUtil.java @@ -28,6 +28,14 @@ public static String generateFeatureStringRef(FeatureReference featureReference) return ref; } + public static String generateFeatureStringRefWithoutProject(FeatureReference featureReference) { + String ref = String.format("%s", featureReference.getName()); + if (featureReference.getVersion() > 0) { + return ref + String.format(":%d", featureReference.getVersion()); + } + return ref; + } + public static String generateFeatureSetStringRef(FeatureSetSpec featureSetSpec) { String ref = String.format("%s/%s", featureSetSpec.getProject(), featureSetSpec.getName()); if (featureSetSpec.getVersion() > 0) { diff --git a/serving/src/test/java/feast/serving/service/CachedSpecServiceTest.java b/serving/src/test/java/feast/serving/service/CachedSpecServiceTest.java index abeb44bd731..01c9304bda0 100644 --- a/serving/src/test/java/feast/serving/service/CachedSpecServiceTest.java +++ b/serving/src/test/java/feast/serving/service/CachedSpecServiceTest.java @@ -37,7 +37,7 @@ import feast.serving.ServingAPIProto.FeatureReference; import feast.serving.specs.CachedSpecService; import feast.serving.specs.CoreSpecService; -import feast.serving.specs.FeatureSetRequest; +import feast.storage.api.retriever.FeatureSetRequest; import java.io.BufferedWriter; import java.io.File; import java.io.FileWriter; diff --git a/serving/src/test/java/feast/serving/service/RedisServingServiceTest.java b/serving/src/test/java/feast/serving/service/OnlineServingServiceTest.java similarity index 72% rename from serving/src/test/java/feast/serving/service/RedisServingServiceTest.java rename to serving/src/test/java/feast/serving/service/OnlineServingServiceTest.java index 05a24d3fe6a..b78fcb69170 100644 --- a/serving/src/test/java/feast/serving/service/RedisServingServiceTest.java +++ b/serving/src/test/java/feast/serving/service/OnlineServingServiceTest.java @@ -22,7 +22,6 @@ import static org.mockito.MockitoAnnotations.initMocks; import com.google.common.collect.Lists; -import com.google.protobuf.AbstractMessageLite; import com.google.protobuf.Duration; import com.google.protobuf.Timestamp; import feast.core.FeatureSetProto.EntitySpec; @@ -33,17 +32,16 @@ import feast.serving.ServingAPIProto.GetOnlineFeaturesResponse; import feast.serving.ServingAPIProto.GetOnlineFeaturesResponse.FieldValues; import feast.serving.specs.CachedSpecService; -import feast.serving.specs.FeatureSetRequest; -import feast.storage.RedisProto.RedisKey; +import feast.storage.api.retriever.FeatureSetRequest; +import feast.storage.connectors.redis.retriever.RedisOnlineRetriever; import feast.types.FeatureRowProto.FeatureRow; import feast.types.FieldProto.Field; import feast.types.ValueProto.Value; -import io.lettuce.core.KeyValue; -import io.lettuce.core.api.StatefulRedisConnection; -import io.lettuce.core.api.sync.RedisCommands; import io.opentracing.Tracer; import io.opentracing.Tracer.SpanBuilder; -import java.util.*; +import java.util.Collections; +import java.util.List; +import java.util.Map; import java.util.stream.Collectors; import org.junit.Before; import org.junit.Test; @@ -51,44 +49,20 @@ import org.mockito.Mock; import org.mockito.Mockito; -public class RedisServingServiceTest { +public class OnlineServingServiceTest { @Mock CachedSpecService specService; @Mock Tracer tracer; - @Mock StatefulRedisConnection connection; + @Mock RedisOnlineRetriever retriever; - @Mock RedisCommands syncCommands; - - private RedisServingService redisServingService; - private byte[][] redisKeyList; + private OnlineServingService onlineServingService; @Before public void setUp() { initMocks(this); - when(connection.sync()).thenReturn(syncCommands); - redisServingService = new RedisServingService(connection, specService, tracer); - redisKeyList = - Lists.newArrayList( - RedisKey.newBuilder() - .setFeatureSet("project/featureSet:1") - .addAllEntities( - Lists.newArrayList( - Field.newBuilder().setName("entity1").setValue(intValue(1)).build(), - Field.newBuilder().setName("entity2").setValue(strValue("a")).build())) - .build(), - RedisKey.newBuilder() - .setFeatureSet("project/featureSet:1") - .addAllEntities( - Lists.newArrayList( - Field.newBuilder().setName("entity1").setValue(intValue(2)).build(), - Field.newBuilder().setName("entity2").setValue(strValue("b")).build())) - .build()) - .stream() - .map(AbstractMessageLite::toByteArray) - .collect(Collectors.toList()) - .toArray(new byte[0][0]); + onlineServingService = new OnlineServingService(retriever, specService, tracer); } @Test @@ -148,14 +122,11 @@ public void shouldReturnResponseWithValuesIfKeysPresent() { .setSpec(getFeatureSetSpec()) .build(); - List> featureRowBytes = - featureRows.stream() - .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) - .collect(Collectors.toList()); when(specService.getFeatureSets(request.getFeaturesList())) .thenReturn(Collections.singletonList(featureSetRequest)); - when(connection.sync()).thenReturn(syncCommands); - when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); + when(retriever.getOnlineFeatures( + request.getEntityRowsList(), Collections.singletonList(featureSetRequest))) + .thenReturn(Collections.singletonList(featureRows)); when(tracer.buildSpan(ArgumentMatchers.any())).thenReturn(Mockito.mock(SpanBuilder.class)); GetOnlineFeaturesResponse expected = @@ -173,100 +144,13 @@ public void shouldReturnResponseWithValuesIfKeysPresent() { .putFields("project/feature1:1", intValue(2)) .putFields("project/feature2:1", intValue(2))) .build(); - GetOnlineFeaturesResponse actual = redisServingService.getOnlineFeatures(request); + GetOnlineFeaturesResponse actual = onlineServingService.getOnlineFeatures(request); assertThat( responseToMapList(actual), containsInAnyOrder(responseToMapList(expected).toArray())); } @Test - public void shouldReturnResponseWithValuesWhenFeatureSetSpecHasUnspecifiedMaxAge() { - GetOnlineFeaturesRequest request = - GetOnlineFeaturesRequest.newBuilder() - .addFeatures( - FeatureReference.newBuilder() - .setName("feature1") - .setVersion(1) - .setProject("project") - .build()) - .addFeatures( - FeatureReference.newBuilder() - .setName("feature2") - .setVersion(1) - .setProject("project") - .build()) - .addEntityRows( - EntityRow.newBuilder() - .setEntityTimestamp(Timestamp.newBuilder().setSeconds(100)) - .putFields("entity1", intValue(1)) - .putFields("entity2", strValue("a"))) - .addEntityRows( - EntityRow.newBuilder() - .setEntityTimestamp(Timestamp.newBuilder().setSeconds(100)) - .putFields("entity1", intValue(2)) - .putFields("entity2", strValue("b"))) - .build(); - - List featureRows = - Lists.newArrayList( - FeatureRow.newBuilder() - .setEventTimestamp(Timestamp.newBuilder().setSeconds(2)) // much older timestamp - .addAllFields( - Lists.newArrayList( - Field.newBuilder().setName("entity1").setValue(intValue(1)).build(), - Field.newBuilder().setName("entity2").setValue(strValue("a")).build(), - Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), - Field.newBuilder().setName("feature2").setValue(intValue(1)).build())) - .setFeatureSet("featureSet:1") - .build(), - FeatureRow.newBuilder() - .setEventTimestamp(Timestamp.newBuilder().setSeconds(15)) // much older timestamp - .addAllFields( - Lists.newArrayList( - Field.newBuilder().setName("entity1").setValue(intValue(2)).build(), - Field.newBuilder().setName("entity2").setValue(strValue("b")).build(), - Field.newBuilder().setName("feature1").setValue(intValue(2)).build(), - Field.newBuilder().setName("feature2").setValue(intValue(2)).build())) - .setFeatureSet("featureSet:1") - .build()); - - FeatureSetRequest featureSetRequest = - FeatureSetRequest.newBuilder() - .addAllFeatureReferences(request.getFeaturesList()) - .setSpec(getFeatureSetSpecWithNoMaxAge()) - .build(); - - List> featureRowBytes = - featureRows.stream() - .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) - .collect(Collectors.toList()); - when(specService.getFeatureSets(request.getFeaturesList())) - .thenReturn(Collections.singletonList(featureSetRequest)); - when(connection.sync()).thenReturn(syncCommands); - when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); - when(tracer.buildSpan(ArgumentMatchers.any())).thenReturn(Mockito.mock(SpanBuilder.class)); - - GetOnlineFeaturesResponse expected = - GetOnlineFeaturesResponse.newBuilder() - .addFieldValues( - FieldValues.newBuilder() - .putFields("entity1", intValue(1)) - .putFields("entity2", strValue("a")) - .putFields("project/feature1:1", intValue(1)) - .putFields("project/feature2:1", intValue(1))) - .addFieldValues( - FieldValues.newBuilder() - .putFields("entity1", intValue(2)) - .putFields("entity2", strValue("b")) - .putFields("project/feature1:1", intValue(2)) - .putFields("project/feature2:1", intValue(2))) - .build(); - GetOnlineFeaturesResponse actual = redisServingService.getOnlineFeatures(request); - assertThat( - responseToMapList(actual), containsInAnyOrder(responseToMapList(expected).toArray())); - } - - @Test - public void shouldReturnKeysWithoutVersionifNotProvided() { + public void shouldReturnKeysWithoutVersionIfNotProvided() { GetOnlineFeaturesRequest request = GetOnlineFeaturesRequest.newBuilder() .addFeatures( @@ -318,14 +202,11 @@ public void shouldReturnKeysWithoutVersionifNotProvided() { .setSpec(getFeatureSetSpec()) .build(); - List> featureRowBytes = - featureRows.stream() - .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) - .collect(Collectors.toList()); when(specService.getFeatureSets(request.getFeaturesList())) .thenReturn(Collections.singletonList(featureSetRequest)); - when(connection.sync()).thenReturn(syncCommands); - when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); + when(retriever.getOnlineFeatures( + request.getEntityRowsList(), Collections.singletonList(featureSetRequest))) + .thenReturn(Collections.singletonList(featureRows)); when(tracer.buildSpan(ArgumentMatchers.any())).thenReturn(Mockito.mock(SpanBuilder.class)); GetOnlineFeaturesResponse expected = @@ -343,7 +224,7 @@ public void shouldReturnKeysWithoutVersionifNotProvided() { .putFields("project/feature1:1", intValue(2)) .putFields("project/feature2", intValue(2))) .build(); - GetOnlineFeaturesResponse actual = redisServingService.getOnlineFeatures(request); + GetOnlineFeaturesResponse actual = onlineServingService.getOnlineFeatures(request); assertThat( responseToMapList(actual), containsInAnyOrder(responseToMapList(expected).toArray())); } @@ -383,27 +264,29 @@ public void shouldReturnResponseWithUnsetValuesIfKeysNotPresent() { .setSpec(getFeatureSetSpec()) .build(); - FeatureRow featureRowPresent = - FeatureRow.newBuilder() - .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) - .addAllFields( - Lists.newArrayList( - Field.newBuilder().setName("entity1").setValue(intValue(1)).build(), - Field.newBuilder().setName("entity2").setValue(strValue("a")).build(), - Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), - Field.newBuilder().setName("feature2").setValue(intValue(1)).build())) - .setFeatureSet("featureSet:1") - .build(); - - List> featureRowBytes = + List featureRows = Lists.newArrayList( - KeyValue.from(new byte[1], Optional.of(featureRowPresent.toByteArray())), - KeyValue.from(new byte[1], Optional.empty())); + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) + .setFeatureSet("project/featureSet:1") + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), + Field.newBuilder().setName("feature2").setValue(intValue(1)).build())) + .build(), + FeatureRow.newBuilder() + .setFeatureSet("project/featureSet:1") + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setName("feature1").build(), + Field.newBuilder().setName("feature2").build())) + .build()); when(specService.getFeatureSets(request.getFeaturesList())) .thenReturn(Collections.singletonList(featureSetRequest)); - when(connection.sync()).thenReturn(syncCommands); - when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); + when(retriever.getOnlineFeatures( + request.getEntityRowsList(), Collections.singletonList(featureSetRequest))) + .thenReturn(Collections.singletonList(featureRows)); when(tracer.buildSpan(ArgumentMatchers.any())).thenReturn(Mockito.mock(SpanBuilder.class)); GetOnlineFeaturesResponse expected = @@ -421,7 +304,7 @@ public void shouldReturnResponseWithUnsetValuesIfKeysNotPresent() { .putFields("project/feature1:1", Value.newBuilder().build()) .putFields("project/feature2:1", Value.newBuilder().build())) .build(); - GetOnlineFeaturesResponse actual = redisServingService.getOnlineFeatures(request); + GetOnlineFeaturesResponse actual = onlineServingService.getOnlineFeatures(request); assertThat( responseToMapList(actual), containsInAnyOrder(responseToMapList(expected).toArray())); } @@ -487,14 +370,11 @@ public void shouldReturnResponseWithUnsetValuesIfMaxAgeIsExceeded() { .setSpec(spec) .build(); - List> featureRowBytes = - featureRows.stream() - .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) - .collect(Collectors.toList()); when(specService.getFeatureSets(request.getFeaturesList())) .thenReturn(Collections.singletonList(featureSetRequest)); - when(connection.sync()).thenReturn(syncCommands); - when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); + when(retriever.getOnlineFeatures( + request.getEntityRowsList(), Collections.singletonList(featureSetRequest))) + .thenReturn(Collections.singletonList(featureRows)); when(tracer.buildSpan(ArgumentMatchers.any())).thenReturn(Mockito.mock(SpanBuilder.class)); GetOnlineFeaturesResponse expected = @@ -512,7 +392,7 @@ public void shouldReturnResponseWithUnsetValuesIfMaxAgeIsExceeded() { .putFields("project/feature1:1", Value.newBuilder().build()) .putFields("project/feature2:1", Value.newBuilder().build())) .build(); - GetOnlineFeaturesResponse actual = redisServingService.getOnlineFeatures(request); + GetOnlineFeaturesResponse actual = onlineServingService.getOnlineFeatures(request); assertThat( responseToMapList(actual), containsInAnyOrder(responseToMapList(expected).toArray())); } @@ -569,14 +449,11 @@ public void shouldFilterOutUndesiredRows() { .setSpec(getFeatureSetSpec()) .build(); - List> featureRowBytes = - featureRows.stream() - .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) - .collect(Collectors.toList()); when(specService.getFeatureSets(request.getFeaturesList())) .thenReturn(Collections.singletonList(featureSetRequest)); - when(connection.sync()).thenReturn(syncCommands); - when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); + when(retriever.getOnlineFeatures( + request.getEntityRowsList(), Collections.singletonList(featureSetRequest))) + .thenReturn(Collections.singletonList(featureRows)); when(tracer.buildSpan(ArgumentMatchers.any())).thenReturn(Mockito.mock(SpanBuilder.class)); GetOnlineFeaturesResponse expected = @@ -592,7 +469,7 @@ public void shouldFilterOutUndesiredRows() { .putFields("entity2", strValue("b")) .putFields("project/feature1:1", intValue(2))) .build(); - GetOnlineFeaturesResponse actual = redisServingService.getOnlineFeatures(request); + GetOnlineFeaturesResponse actual = onlineServingService.getOnlineFeatures(request); assertThat( responseToMapList(actual), containsInAnyOrder(responseToMapList(expected).toArray())); } diff --git a/storage/api/pom.xml b/storage/api/pom.xml new file mode 100644 index 00000000000..c1648c7cfa1 --- /dev/null +++ b/storage/api/pom.xml @@ -0,0 +1,72 @@ + + + + dev.feast + feast-parent + ${revision} + ../.. + + + 4.0.0 + feast-storage-api + + Feast Storage API + + + + + org.apache.maven.plugins + maven-dependency-plugin + + + + javax.annotation + + + + + + + + + + dev.feast + datatypes-java + ${project.version} + + + + org.apache.beam + beam-sdks-java-core + ${org.apache.beam.version} + + + + com.google.auto.value + auto-value-annotations + 1.6.6 + + + + com.google.auto.value + auto-value + 1.6.6 + provided + + + + org.apache.commons + commons-lang3 + 3.9 + + + + junit + junit + 4.12 + test + + + + diff --git a/serving/src/main/java/feast/serving/specs/FeatureSetRequest.java b/storage/api/src/main/java/feast/storage/api/retriever/FeatureSetRequest.java similarity index 84% rename from serving/src/main/java/feast/serving/specs/FeatureSetRequest.java rename to storage/api/src/main/java/feast/storage/api/retriever/FeatureSetRequest.java index 904630659d7..d181abfbe63 100644 --- a/serving/src/main/java/feast/serving/specs/FeatureSetRequest.java +++ b/storage/api/src/main/java/feast/storage/api/retriever/FeatureSetRequest.java @@ -14,13 +14,15 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.serving.specs; +package feast.storage.api.retriever; import com.google.auto.value.AutoValue; import com.google.common.collect.ImmutableSet; import feast.core.FeatureSetProto.FeatureSetSpec; import feast.serving.ServingAPIProto.FeatureReference; import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; @AutoValue public abstract class FeatureSetRequest { @@ -50,4 +52,9 @@ public Builder addFeatureReference(FeatureReference featureReference) { public abstract FeatureSetRequest build(); } + + public Map getFeatureRefsByName() { + return getFeatureReferences().stream() + .collect(Collectors.toMap(FeatureReference::getName, featureReference -> featureReference)); + } } diff --git a/storage/api/src/main/java/feast/storage/api/retriever/HistoricalRetrievalResult.java b/storage/api/src/main/java/feast/storage/api/retriever/HistoricalRetrievalResult.java new file mode 100644 index 00000000000..a81ce776254 --- /dev/null +++ b/storage/api/src/main/java/feast/storage/api/retriever/HistoricalRetrievalResult.java @@ -0,0 +1,100 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.api.retriever; + +import com.google.auto.value.AutoValue; +import feast.serving.ServingAPIProto.DataFormat; +import feast.serving.ServingAPIProto.JobStatus; +import java.io.Serializable; +import java.util.List; +import javax.annotation.Nullable; + +/** Result of a historical feature retrieval request. */ +@AutoValue +public abstract class HistoricalRetrievalResult implements Serializable { + + public abstract String getId(); + + public abstract JobStatus getStatus(); + + @Nullable + public abstract String getError(); + + @Nullable + public abstract List getFileUris(); + + @Nullable + public abstract DataFormat getDataFormat(); + + /** + * Instantiates a {@link HistoricalRetrievalResult} indicating that the retrieval was a failure, + * together with its associated error. + * + * @param id retrieval id identifying the retrieval request. + * @param error error that occurred + * @return {@link HistoricalRetrievalResult} + */ + public static HistoricalRetrievalResult error(String id, Exception error) { + return newBuilder() + .setId(id) + .setStatus(JobStatus.JOB_STATUS_DONE) + .setError(error.getMessage()) + .build(); + } + + /** + * Instantiates a {@link HistoricalRetrievalResult} indicating that the retrieval was a success, + * together with the location of the output. + * + * @param id retrieval id identifying the retrieval request + * @param fileUris list of output file URIs + * @param dataFormat data format of the output files + * @return + */ + public static HistoricalRetrievalResult success( + String id, List fileUris, DataFormat dataFormat) { + return newBuilder() + .setId(id) + .setStatus(JobStatus.JOB_STATUS_DONE) + .setFileUris(fileUris) + .setDataFormat(dataFormat) + .build(); + } + + static Builder newBuilder() { + return new AutoValue_HistoricalRetrievalResult.Builder(); + } + + @AutoValue.Builder + abstract static class Builder { + abstract Builder setId(String id); + + abstract Builder setStatus(JobStatus jobStatus); + + abstract Builder setError(String error); + + abstract Builder setFileUris(List fileUris); + + abstract Builder setDataFormat(DataFormat dataFormat); + + abstract HistoricalRetrievalResult build(); + } + + public boolean hasError() { + return getError() != null; + } +} diff --git a/storage/api/src/main/java/feast/storage/api/retriever/HistoricalRetriever.java b/storage/api/src/main/java/feast/storage/api/retriever/HistoricalRetriever.java new file mode 100644 index 00000000000..95a89c1a3cb --- /dev/null +++ b/storage/api/src/main/java/feast/storage/api/retriever/HistoricalRetriever.java @@ -0,0 +1,49 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.api.retriever; + +import feast.serving.ServingAPIProto.DatasetSource; +import java.util.List; + +/** + * A historical retriever is a feature retriever that retrieves feature data corresponding to + * provided entities over a given period of time. + */ +public interface HistoricalRetriever { + + /** + * Get temporary staging location if applicable. If not applicable to this store, returns an empty + * string. + * + * @return staging location uri + */ + String getStagingLocation(); + + /** + * Get all features corresponding to the provided batch features request. + * + * @param retrievalId String that uniquely identifies this retrieval request. + * @param datasetSource {@link DatasetSource} containing source to load the dataset containing + * entity columns. + * @param featureSetRequests List of {@link FeatureSetRequest} to feature references in the + * request tied to that feature set. + * @return {@link HistoricalRetrievalResult} if successful, contains the location of the results, + * else contains the error to be returned to the user. + */ + HistoricalRetrievalResult getHistoricalFeatures( + String retrievalId, DatasetSource datasetSource, List featureSetRequests); +} diff --git a/storage/api/src/main/java/feast/storage/api/retriever/OnlineRetriever.java b/storage/api/src/main/java/feast/storage/api/retriever/OnlineRetriever.java new file mode 100644 index 00000000000..5eb27b995ea --- /dev/null +++ b/storage/api/src/main/java/feast/storage/api/retriever/OnlineRetriever.java @@ -0,0 +1,40 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.api.retriever; + +import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest.EntityRow; +import feast.types.FeatureRowProto.FeatureRow; +import java.util.List; + +/** + * An online retriever is a feature retriever that retrieves the latest feature data corresponding + * to provided entities. + */ +public interface OnlineRetriever { + + /** + * Get all values corresponding to the request. + * + * @param entityRows list of entity rows in the feature request + * @param featureSetRequests List of {@link FeatureSetRequest} to feature references in the + * request tied to that feature set. + * @return list of lists of {@link FeatureRow}s corresponding to each feature set request and + * entity row. + */ + List> getOnlineFeatures( + List entityRows, List featureSetRequests); +} diff --git a/storage/api/src/main/java/feast/storage/api/writer/DeadletterSink.java b/storage/api/src/main/java/feast/storage/api/writer/DeadletterSink.java new file mode 100644 index 00000000000..a07254bddb0 --- /dev/null +++ b/storage/api/src/main/java/feast/storage/api/writer/DeadletterSink.java @@ -0,0 +1,38 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.api.writer; + +import org.apache.beam.sdk.transforms.PTransform; +import org.apache.beam.sdk.values.PCollection; +import org.apache.beam.sdk.values.PDone; + +/** Interface for for implementing user defined deadletter sinks to write failed elements to. */ +public interface DeadletterSink { + + /** + * Set up the deadletter sink for writes. This method will be called once during pipeline + * initialisation. + */ + void prepareWrite(); + + /** + * Get a {@link PTransform} that writes a collection of FailedElements to the deadletter sink. + * + * @return {@link PTransform} + */ + PTransform, PDone> write(); +} diff --git a/storage/api/src/main/java/feast/storage/api/writer/FailedElement.java b/storage/api/src/main/java/feast/storage/api/writer/FailedElement.java new file mode 100644 index 00000000000..d5823414772 --- /dev/null +++ b/storage/api/src/main/java/feast/storage/api/writer/FailedElement.java @@ -0,0 +1,83 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2019 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.api.writer; + +import com.google.auto.value.AutoValue; +import javax.annotation.Nullable; +import org.apache.beam.sdk.schemas.AutoValueSchema; +import org.apache.beam.sdk.schemas.annotations.DefaultSchema; +import org.joda.time.Instant; + +@AutoValue +// Use DefaultSchema annotation so this AutoValue class can be serialized by Beam +// https://issues.apache.org/jira/browse/BEAM-1891 +// https://github.com/apache/beam/pull/7334 +@DefaultSchema(AutoValueSchema.class) +public abstract class FailedElement { + public abstract Instant getTimestamp(); + + @Nullable + public abstract String getJobName(); + + @Nullable + public abstract String getProjectName(); + + @Nullable + public abstract String getFeatureSetName(); + + @Nullable + public abstract String getFeatureSetVersion(); + + @Nullable + public abstract String getTransformName(); + + @Nullable + public abstract String getPayload(); + + @Nullable + public abstract String getErrorMessage(); + + @Nullable + public abstract String getStackTrace(); + + public static Builder newBuilder() { + return new AutoValue_FailedElement.Builder().setTimestamp(Instant.now()); + } + + @AutoValue.Builder + public abstract static class Builder { + public abstract Builder setTimestamp(Instant timestamp); + + public abstract Builder setProjectName(String projectName); + + public abstract Builder setFeatureSetName(String featureSetName); + + public abstract Builder setFeatureSetVersion(String featureSetVersion); + + public abstract Builder setJobName(String jobName); + + public abstract Builder setTransformName(String transformName); + + public abstract Builder setPayload(String payload); + + public abstract Builder setErrorMessage(String errorMessage); + + public abstract Builder setStackTrace(String stackTrace); + + public abstract FailedElement build(); + } +} diff --git a/storage/api/src/main/java/feast/storage/api/writer/FeatureSink.java b/storage/api/src/main/java/feast/storage/api/writer/FeatureSink.java new file mode 100644 index 00000000000..3dfe7e8f103 --- /dev/null +++ b/storage/api/src/main/java/feast/storage/api/writer/FeatureSink.java @@ -0,0 +1,54 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.api.writer; + +import feast.core.FeatureSetProto; +import feast.types.FeatureRowProto.FeatureRow; +import java.io.Serializable; +import org.apache.beam.sdk.transforms.PTransform; +import org.apache.beam.sdk.values.PCollection; + +/** Interface for implementing user defined feature sink functionality. */ +public interface FeatureSink extends Serializable { + + /** + * Set up storage backend for write. This method will be called once during pipeline + * initialisation. + * + *

    Examples when schemas need to be updated: + * + *

      + *
    • when a new entity is registered, a table usually needs to be created + *
    • when a new feature is registered, a column with appropriate data type usually needs to be + * created + *
    + * + *

    If the storage backend is a key-value or a schema-less database, however, there may not be a + * need to manage any schemas. + * + * @param featureSet Feature set to be written + */ + void prepareWrite(FeatureSetProto.FeatureSet featureSet); + + /** + * Get a {@link PTransform} that writes feature rows to the store, and returns a {@link + * WriteResult} that splits successful and failed inserts to be separately logged. + * + * @return {@link PTransform} + */ + PTransform, WriteResult> writer(); +} diff --git a/storage/api/src/main/java/feast/storage/api/writer/WriteResult.java b/storage/api/src/main/java/feast/storage/api/writer/WriteResult.java new file mode 100644 index 00000000000..e378c2b46a4 --- /dev/null +++ b/storage/api/src/main/java/feast/storage/api/writer/WriteResult.java @@ -0,0 +1,97 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.api.writer; + +import com.google.common.collect.ImmutableMap; +import feast.types.FeatureRowProto.FeatureRow; +import java.io.Serializable; +import java.util.Map; +import org.apache.beam.sdk.Pipeline; +import org.apache.beam.sdk.transforms.PTransform; +import org.apache.beam.sdk.values.*; + +/** The result of a write transform. */ +public final class WriteResult implements Serializable, POutput { + + private final Pipeline pipeline; + private final PCollection successfulInserts; + private final PCollection failedInserts; + + private static TupleTag successfulInsertsTag = new TupleTag<>("successfulInserts"); + private static TupleTag failedInsertsTupleTag = new TupleTag<>("failedInserts"); + + /** + * Creates a {@link WriteResult} in the given {@link Pipeline}. + * + * @param pipeline {@link Pipeline} + * @param successfulInserts {@link PCollection} of {@link FeatureRow}s successfully inserted into + * the store + * @param failedInserts {@link PCollection} of {@link FailedElement}s + * @return {@link WriteResult} + */ + public static WriteResult in( + Pipeline pipeline, + PCollection successfulInserts, + PCollection failedInserts) { + return new WriteResult(pipeline, successfulInserts, failedInserts); + } + + private WriteResult( + Pipeline pipeline, + PCollection successfulInserts, + PCollection failedInserts) { + + this.pipeline = pipeline; + this.successfulInserts = successfulInserts; + this.failedInserts = failedInserts; + } + + /** + * Gets set of feature rows that were unsuccessfully written to the store. The failed feature rows + * are wrapped in FailedElement objects so implementations of WriteResult can be flexible in how + * errors are stored. + * + * @return FailedElements of unsuccessfully written feature rows + */ + public PCollection getFailedInserts() { + return failedInserts; + } + + /** + * Gets set of successfully written feature rows. + * + * @return PCollection of feature rows successfully written to the store + */ + public PCollection getSuccessfulInserts() { + return successfulInserts; + } + + @Override + public Pipeline getPipeline() { + return pipeline; + } + + @Override + public Map, PValue> expand() { + return ImmutableMap.of( + successfulInsertsTag, successfulInserts, failedInsertsTupleTag, failedInserts); + } + + @Override + public void finishSpecifyingOutput( + String transformName, PInput input, PTransform transform) {} +} diff --git a/ingestion/src/main/java/feast/retry/BackOffExecutor.java b/storage/api/src/main/java/feast/storage/common/retry/BackOffExecutor.java similarity index 98% rename from ingestion/src/main/java/feast/retry/BackOffExecutor.java rename to storage/api/src/main/java/feast/storage/common/retry/BackOffExecutor.java index 344c65ac424..296582f8b35 100644 --- a/ingestion/src/main/java/feast/retry/BackOffExecutor.java +++ b/storage/api/src/main/java/feast/storage/common/retry/BackOffExecutor.java @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.retry; +package feast.storage.common.retry; import java.io.Serializable; import org.apache.beam.sdk.util.BackOff; diff --git a/ingestion/src/main/java/feast/retry/Retriable.java b/storage/api/src/main/java/feast/storage/common/retry/Retriable.java similarity index 95% rename from ingestion/src/main/java/feast/retry/Retriable.java rename to storage/api/src/main/java/feast/storage/common/retry/Retriable.java index 30676fe8208..2c92c851758 100644 --- a/ingestion/src/main/java/feast/retry/Retriable.java +++ b/storage/api/src/main/java/feast/storage/common/retry/Retriable.java @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.retry; +package feast.storage.common.retry; public interface Retriable { void execute() throws Exception; diff --git a/storage/api/src/main/java/feast/storage/common/testing/TestUtil.java b/storage/api/src/main/java/feast/storage/common/testing/TestUtil.java new file mode 100644 index 00000000000..6047a93dc17 --- /dev/null +++ b/storage/api/src/main/java/feast/storage/common/testing/TestUtil.java @@ -0,0 +1,188 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2019 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.common.testing; + +import com.google.protobuf.ByteString; +import com.google.protobuf.util.Timestamps; +import feast.core.FeatureSetProto.FeatureSet; +import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.types.FeatureRowProto.FeatureRow; +import feast.types.FeatureRowProto.FeatureRow.Builder; +import feast.types.FieldProto.Field; +import feast.types.ValueProto.*; +import java.util.concurrent.ThreadLocalRandom; +import org.apache.commons.lang3.RandomStringUtils; + +@SuppressWarnings("WeakerAccess") +public class TestUtil { + + /** + * Create a Feature Row with random value according to the FeatureSetSpec + * + * @param featureSet {@link FeatureSet} + * @return {@link FeatureRow} + */ + public static FeatureRow createRandomFeatureRow(FeatureSet featureSet) { + ThreadLocalRandom random = ThreadLocalRandom.current(); + int randomStringSizeMaxSize = 12; + return createRandomFeatureRow(featureSet, random.nextInt(0, randomStringSizeMaxSize) + 4); + } + + /** + * Create a Feature Row with random value according to the FeatureSet. + * + *

    The Feature Row created contains fields according to the entities and features defined in + * FeatureSet, matching the value type of the field, with randomized value for testing. + * + * @param featureSet {@link FeatureSet} + * @param randomStringSize number of characters for the generated random string + * @return {@link FeatureRow} + */ + public static FeatureRow createRandomFeatureRow(FeatureSet featureSet, int randomStringSize) { + Builder builder = + FeatureRow.newBuilder() + .setFeatureSet(getFeatureSetReference(featureSet)) + .setEventTimestamp(Timestamps.fromMillis(System.currentTimeMillis())); + + featureSet + .getSpec() + .getEntitiesList() + .forEach( + field -> { + builder.addFields( + Field.newBuilder() + .setName(field.getName()) + .setValue(createRandomValue(field.getValueType(), randomStringSize)) + .build()); + }); + + featureSet + .getSpec() + .getFeaturesList() + .forEach( + field -> { + builder.addFields( + Field.newBuilder() + .setName(field.getName()) + .setValue(createRandomValue(field.getValueType(), randomStringSize)) + .build()); + }); + + return builder.build(); + } + + private static String getFeatureSetReference(FeatureSet featureSet) { + FeatureSetSpec spec = featureSet.getSpec(); + return String.format("%s/%s:%d", spec.getProject(), spec.getName(), spec.getVersion()); + } + + /** + * Create a random Feast {@link Value} of {@link ValueType.Enum}. + * + * @param type {@link ValueType.Enum} + * @param randomStringSize number of characters for the generated random string + * @return {@link Value} + */ + public static Value createRandomValue(ValueType.Enum type, int randomStringSize) { + Value.Builder builder = Value.newBuilder(); + ThreadLocalRandom random = ThreadLocalRandom.current(); + + switch (type) { + case INVALID: + case UNRECOGNIZED: + throw new IllegalArgumentException("Invalid ValueType: " + type); + case BYTES: + builder.setBytesVal( + ByteString.copyFrom(RandomStringUtils.randomAlphanumeric(randomStringSize).getBytes())); + break; + case STRING: + builder.setStringVal(RandomStringUtils.randomAlphanumeric(randomStringSize)); + break; + case INT32: + builder.setInt32Val(random.nextInt()); + break; + case INT64: + builder.setInt64Val(random.nextLong()); + break; + case DOUBLE: + builder.setDoubleVal(random.nextDouble()); + break; + case FLOAT: + builder.setFloatVal(random.nextFloat()); + break; + case BOOL: + builder.setBoolVal(random.nextBoolean()); + break; + case BYTES_LIST: + builder.setBytesListVal( + BytesList.newBuilder() + .addVal( + ByteString.copyFrom( + RandomStringUtils.randomAlphanumeric(randomStringSize).getBytes())) + .build()); + break; + case STRING_LIST: + builder.setStringListVal( + StringList.newBuilder() + .addVal(RandomStringUtils.randomAlphanumeric(randomStringSize)) + .build()); + break; + case INT32_LIST: + builder.setInt32ListVal(Int32List.newBuilder().addVal(random.nextInt()).build()); + break; + case INT64_LIST: + builder.setInt64ListVal(Int64List.newBuilder().addVal(random.nextLong()).build()); + break; + case DOUBLE_LIST: + builder.setDoubleListVal(DoubleList.newBuilder().addVal(random.nextDouble()).build()); + break; + case FLOAT_LIST: + builder.setFloatListVal(FloatList.newBuilder().addVal(random.nextFloat()).build()); + break; + case BOOL_LIST: + builder.setBoolListVal(BoolList.newBuilder().addVal(random.nextBoolean()).build()); + break; + } + return builder.build(); + } + + /** + * Create a field object with given name and type. + * + * @param name of the field. + * @param value of the field. Should be compatible with the valuetype given. + * @param valueType type of the field. + * @return Field object + */ + public static Field field(String name, Object value, ValueType.Enum valueType) { + Field.Builder fieldBuilder = Field.newBuilder().setName(name); + switch (valueType) { + case INT32: + return fieldBuilder.setValue(Value.newBuilder().setInt32Val((int) value)).build(); + case INT64: + return fieldBuilder.setValue(Value.newBuilder().setInt64Val((int) value)).build(); + case FLOAT: + return fieldBuilder.setValue(Value.newBuilder().setFloatVal((float) value)).build(); + case DOUBLE: + return fieldBuilder.setValue(Value.newBuilder().setDoubleVal((double) value)).build(); + case STRING: + return fieldBuilder.setValue(Value.newBuilder().setStringVal((String) value)).build(); + default: + throw new IllegalStateException("Unexpected valueType: " + value.getClass()); + } + } +} diff --git a/storage/connectors/bigquery/pom.xml b/storage/connectors/bigquery/pom.xml new file mode 100644 index 00000000000..fab3739c43c --- /dev/null +++ b/storage/connectors/bigquery/pom.xml @@ -0,0 +1,94 @@ + + + + dev.feast + feast-storage-connectors + ${revision} + + + 4.0.0 + feast-storage-connector-bigquery + + Feast Storage Connector for BigQuery + + + + io.pebbletemplates + pebble + 3.1.0 + + + + + com.google.cloud + google-cloud-bigquery + + + + com.google.cloud + google-cloud-storage + + + + org.apache.beam + beam-sdks-java-io-google-cloud-platform + ${org.apache.beam.version} + + + com.google.cloud + google-cloud-spanner + + + com.google.cloud.bigtable + bigtable-client-core + + + + + + io.opencensus + opencensus-contrib-http-util + 0.21.0 + + + + com.google.auto.value + auto-value-annotations + 1.6.6 + + + + com.google.auto.value + auto-value + 1.6.6 + provided + + + + junit + junit + 4.12 + test + + + + org.apache.beam + beam-runners-direct-java + ${org.apache.beam.version} + test + + + + org.hamcrest + hamcrest-core + test + + + org.hamcrest + hamcrest-library + test + + + + diff --git a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/common/TypeUtil.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/common/TypeUtil.java new file mode 100644 index 00000000000..dcd13093177 --- /dev/null +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/common/TypeUtil.java @@ -0,0 +1,66 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.bigquery.common; + +import com.google.cloud.bigquery.StandardSQLTypeName; +import feast.types.ValueProto; +import java.util.HashMap; +import java.util.Map; + +public class TypeUtil { + + private static final Map + VALUE_TYPE_TO_STANDARD_SQL_TYPE = new HashMap<>(); + + static { + VALUE_TYPE_TO_STANDARD_SQL_TYPE.put(ValueProto.ValueType.Enum.BYTES, StandardSQLTypeName.BYTES); + VALUE_TYPE_TO_STANDARD_SQL_TYPE.put( + ValueProto.ValueType.Enum.STRING, StandardSQLTypeName.STRING); + VALUE_TYPE_TO_STANDARD_SQL_TYPE.put(ValueProto.ValueType.Enum.INT32, StandardSQLTypeName.INT64); + VALUE_TYPE_TO_STANDARD_SQL_TYPE.put(ValueProto.ValueType.Enum.INT64, StandardSQLTypeName.INT64); + VALUE_TYPE_TO_STANDARD_SQL_TYPE.put( + ValueProto.ValueType.Enum.DOUBLE, StandardSQLTypeName.FLOAT64); + VALUE_TYPE_TO_STANDARD_SQL_TYPE.put( + ValueProto.ValueType.Enum.FLOAT, StandardSQLTypeName.FLOAT64); + VALUE_TYPE_TO_STANDARD_SQL_TYPE.put(ValueProto.ValueType.Enum.BOOL, StandardSQLTypeName.BOOL); + VALUE_TYPE_TO_STANDARD_SQL_TYPE.put( + ValueProto.ValueType.Enum.BYTES_LIST, StandardSQLTypeName.BYTES); + VALUE_TYPE_TO_STANDARD_SQL_TYPE.put( + ValueProto.ValueType.Enum.STRING_LIST, StandardSQLTypeName.STRING); + VALUE_TYPE_TO_STANDARD_SQL_TYPE.put( + ValueProto.ValueType.Enum.INT32_LIST, StandardSQLTypeName.INT64); + VALUE_TYPE_TO_STANDARD_SQL_TYPE.put( + ValueProto.ValueType.Enum.INT64_LIST, StandardSQLTypeName.INT64); + VALUE_TYPE_TO_STANDARD_SQL_TYPE.put( + ValueProto.ValueType.Enum.DOUBLE_LIST, StandardSQLTypeName.FLOAT64); + VALUE_TYPE_TO_STANDARD_SQL_TYPE.put( + ValueProto.ValueType.Enum.FLOAT_LIST, StandardSQLTypeName.FLOAT64); + VALUE_TYPE_TO_STANDARD_SQL_TYPE.put( + ValueProto.ValueType.Enum.BOOL_LIST, StandardSQLTypeName.BOOL); + } + + /** + * Converts {@link feast.types.ValueProto.ValueType} to its corresponding {@link + * StandardSQLTypeName} + * + * @param valueType value type to convert + * @return {@link StandardSQLTypeName} + */ + public static StandardSQLTypeName toStandardSqlType(ValueProto.ValueType.Enum valueType) { + return VALUE_TYPE_TO_STANDARD_SQL_TYPE.get(valueType); + } +} diff --git a/serving/src/main/java/feast/serving/store/bigquery/BatchRetrievalQueryRunnable.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/BigQueryHistoricalRetriever.java similarity index 52% rename from serving/src/main/java/feast/serving/store/bigquery/BatchRetrievalQueryRunnable.java rename to storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/BigQueryHistoricalRetriever.java index 61103af1092..27ba07e82ec 100644 --- a/serving/src/main/java/feast/serving/store/bigquery/BatchRetrievalQueryRunnable.java +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/BigQueryHistoricalRetriever.java @@ -1,6 +1,6 @@ /* * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors + * Copyright 2018-2020 The Feast Authors * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -14,88 +14,46 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.serving.store.bigquery; +package feast.storage.connectors.bigquery.retriever; -import static feast.serving.service.BigQueryServingService.TEMP_TABLE_EXPIRY_DURATION_MS; -import static feast.serving.service.BigQueryServingService.createTempTableName; -import static feast.serving.store.bigquery.QueryTemplater.createTimestampLimitQuery; +import static feast.storage.connectors.bigquery.retriever.QueryTemplater.createEntityTableUUIDQuery; +import static feast.storage.connectors.bigquery.retriever.QueryTemplater.createTimestampLimitQuery; import com.google.auto.value.AutoValue; import com.google.cloud.RetryOption; -import com.google.cloud.bigquery.BigQuery; -import com.google.cloud.bigquery.BigQueryException; -import com.google.cloud.bigquery.DatasetId; -import com.google.cloud.bigquery.ExtractJobConfiguration; -import com.google.cloud.bigquery.FieldValueList; -import com.google.cloud.bigquery.Job; -import com.google.cloud.bigquery.JobInfo; -import com.google.cloud.bigquery.QueryJobConfiguration; -import com.google.cloud.bigquery.TableId; -import com.google.cloud.bigquery.TableInfo; -import com.google.cloud.bigquery.TableResult; +import com.google.cloud.bigquery.*; import com.google.cloud.storage.Blob; import com.google.cloud.storage.Storage; -import com.google.cloud.storage.Storage.BlobListOption; import feast.serving.ServingAPIProto; -import feast.serving.ServingAPIProto.DataFormat; -import feast.serving.ServingAPIProto.JobStatus; -import feast.serving.ServingAPIProto.JobType; -import feast.serving.service.JobService; -import feast.serving.store.bigquery.model.FeatureSetInfo; +import feast.serving.ServingAPIProto.DatasetSource; +import feast.storage.api.retriever.FeatureSetRequest; +import feast.storage.api.retriever.HistoricalRetrievalResult; +import feast.storage.api.retriever.HistoricalRetriever; import io.grpc.Status; import java.io.IOException; import java.util.ArrayList; import java.util.List; -import java.util.concurrent.ExecutionException; -import java.util.concurrent.ExecutorCompletionService; -import java.util.concurrent.ExecutorService; -import java.util.concurrent.Executors; -import java.util.concurrent.TimeUnit; -import java.util.concurrent.TimeoutException; +import java.util.UUID; +import java.util.concurrent.*; +import java.util.stream.Collectors; +import org.slf4j.Logger; import org.threeten.bp.Duration; -/** - * BatchRetrievalQueryRunnable is a Runnable for running a BigQuery Feast batch retrieval job async. - * - *

    It does the following, in sequence: - * - *

    1. Retrieve the temporal bounds of the entity dataset provided. This will be used to filter - * the feature set tables when performing the feature retrieval. - * - *

    2. For each of the feature sets requested, generate the subquery for doing a point-in-time - * correctness join of the features in the feature set to the entity table. - * - *

    3. Run each of the subqueries in parallel and wait for them to complete. If any of the jobs - * are unsuccessful, the thread running the BatchRetrievalQueryRunnable catches the error and - * updates the job database. - * - *

    4. When all the subquery jobs are complete, join the outputs of all the subqueries into a - * single table. - * - *

    5. Extract the output of the join to a remote file, and write the location of the remote file - * to the job database, and mark the retrieval job as successful. - */ @AutoValue -public abstract class BatchRetrievalQueryRunnable implements Runnable { +public abstract class BigQueryHistoricalRetriever implements HistoricalRetriever { - private static final long SUBQUERY_TIMEOUT_SECS = 900; // 15 minutes + private static final Logger log = + org.slf4j.LoggerFactory.getLogger(BigQueryHistoricalRetriever.class); - public abstract JobService jobService(); + public static final long TEMP_TABLE_EXPIRY_DURATION_MS = Duration.ofDays(1).toMillis(); + private static final long SUBQUERY_TIMEOUT_SECS = 900; // 15 minutes public abstract String projectId(); public abstract String datasetId(); - public abstract String feastJobId(); - public abstract BigQuery bigquery(); - public abstract List entityTableColumnNames(); - - public abstract List featureSetInfos(); - - public abstract String entityTableName(); - public abstract String jobStagingLocation(); public abstract int initialRetryDelaySecs(); @@ -105,55 +63,78 @@ public abstract class BatchRetrievalQueryRunnable implements Runnable { public abstract Storage storage(); public static Builder builder() { - return new AutoValue_BatchRetrievalQueryRunnable.Builder(); + return new AutoValue_BigQueryHistoricalRetriever.Builder(); } @AutoValue.Builder public abstract static class Builder { - public abstract Builder setJobService(JobService jobService); - public abstract Builder setProjectId(String projectId); public abstract Builder setDatasetId(String datasetId); - public abstract Builder setFeastJobId(String feastJobId); + public abstract Builder setJobStagingLocation(String jobStagingLocation); public abstract Builder setBigquery(BigQuery bigquery); - public abstract Builder setEntityTableColumnNames(List entityTableColumnNames); - - public abstract Builder setFeatureSetInfos(List featureSetInfos); - - public abstract Builder setEntityTableName(String entityTableName); - - public abstract Builder setJobStagingLocation(String jobStagingLocation); - public abstract Builder setInitialRetryDelaySecs(int initialRetryDelaySecs); public abstract Builder setTotalTimeoutSecs(int totalTimeoutSecs); public abstract Builder setStorage(Storage storage); - public abstract BatchRetrievalQueryRunnable build(); + public abstract BigQueryHistoricalRetriever build(); } @Override - public void run() { + public String getStagingLocation() { + return jobStagingLocation(); + } - // 1. Retrieve the temporal bounds of the entity dataset provided - FieldValueList timestampLimits = getTimestampLimits(entityTableName()); + @Override + public HistoricalRetrievalResult getHistoricalFeatures( + String retrievalId, DatasetSource datasetSource, List featureSetRequests) { + List featureSetQueryInfos = + QueryTemplater.getFeatureSetInfos(featureSetRequests); + + // 1. load entity table + Table entityTable; + String entityTableName; + try { + entityTable = loadEntities(datasetSource); + + TableId entityTableWithUUIDs = generateUUIDs(entityTable); + entityTableName = generateFullTableName(entityTableWithUUIDs); + } catch (Exception e) { + return HistoricalRetrievalResult.error( + retrievalId, + new RuntimeException( + String.format("Unable to load entity table to BigQuery: %s", e.toString()))); + } + + Schema entityTableSchema = entityTable.getDefinition().getSchema(); + List entityTableColumnNames = + entityTableSchema.getFields().stream() + .map(Field::getName) + .filter(name -> !name.equals("event_timestamp")) + .collect(Collectors.toList()); + + // 2. Retrieve the temporal bounds of the entity dataset provided + FieldValueList timestampLimits = getTimestampLimits(entityTableName); - // 2. Generate the subqueries - List featureSetQueries = generateQueries(timestampLimits); + // 3. Generate the subqueries + List featureSetQueries = + generateQueries(entityTableName, timestampLimits, featureSetQueryInfos); QueryJobConfiguration queryConfig; try { - // 3 & 4. Run the subqueries in parallel then collect the outputs - Job queryJob = runBatchQuery(featureSetQueries); + // 4. Run the subqueries in parallel then collect the outputs + Job queryJob = + runBatchQuery( + entityTableName, entityTableColumnNames, featureSetQueryInfos, featureSetQueries); queryConfig = queryJob.getConfiguration(); String exportTableDestinationUri = - String.format("%s/%s/*.avro", jobStagingLocation(), feastJobId()); + String.format("%s/%s/*.avro", jobStagingLocation(), retrievalId); // 5. Export the table // Hardcode the format to Avro for now @@ -162,60 +143,166 @@ public void run() { queryConfig.getDestinationTable(), exportTableDestinationUri, "Avro"); Job extractJob = bigquery().create(JobInfo.of(extractConfig)); waitForJob(extractJob); + } catch (BigQueryException | InterruptedException | IOException e) { - jobService() - .upsert( - ServingAPIProto.Job.newBuilder() - .setId(feastJobId()) - .setType(JobType.JOB_TYPE_DOWNLOAD) - .setStatus(JobStatus.JOB_STATUS_DONE) - .setError(e.getMessage()) - .build()); - return; + return HistoricalRetrievalResult.error(retrievalId, e); } - List fileUris = parseOutputFileURIs(); - - // 5. Update the job database - jobService() - .upsert( - ServingAPIProto.Job.newBuilder() - .setId(feastJobId()) - .setType(JobType.JOB_TYPE_DOWNLOAD) - .setStatus(JobStatus.JOB_STATUS_DONE) - .addAllFileUris(fileUris) - .setDataFormat(DataFormat.DATA_FORMAT_AVRO) - .build()); + List fileUris = parseOutputFileURIs(retrievalId); + + return HistoricalRetrievalResult.success( + retrievalId, fileUris, ServingAPIProto.DataFormat.DATA_FORMAT_AVRO); } - private List parseOutputFileURIs() { - String scheme = jobStagingLocation().substring(0, jobStagingLocation().indexOf("://")); - String stagingLocationNoScheme = - jobStagingLocation().substring(jobStagingLocation().indexOf("://") + 3); - String bucket = stagingLocationNoScheme.split("/")[0]; - List prefixParts = new ArrayList<>(); - prefixParts.add( - stagingLocationNoScheme.contains("/") && !stagingLocationNoScheme.endsWith("/") - ? stagingLocationNoScheme.substring(stagingLocationNoScheme.indexOf("/") + 1) - : ""); - prefixParts.add(feastJobId()); - String prefix = String.join("/", prefixParts) + "/"; + private TableId generateUUIDs(Table loadedEntityTable) { + try { + String uuidQuery = + createEntityTableUUIDQuery(generateFullTableName(loadedEntityTable.getTableId())); + QueryJobConfiguration queryJobConfig = + QueryJobConfiguration.newBuilder(uuidQuery) + .setDestinationTable(TableId.of(projectId(), datasetId(), createTempTableName())) + .build(); + Job queryJob = bigquery().create(JobInfo.of(queryJobConfig)); + Job completedJob = waitForJob(queryJob); + TableInfo expiry = + bigquery() + .getTable(queryJobConfig.getDestinationTable()) + .toBuilder() + .setExpirationTime(System.currentTimeMillis() + TEMP_TABLE_EXPIRY_DURATION_MS) + .build(); + bigquery().update(expiry); + queryJobConfig = completedJob.getConfiguration(); + return queryJobConfig.getDestinationTable(); + } catch (InterruptedException | BigQueryException e) { + throw Status.INTERNAL + .withDescription("Failed to load entity dataset into store") + .withCause(e) + .asRuntimeException(); + } + } - List fileUris = new ArrayList<>(); - for (Blob blob : storage().list(bucket, BlobListOption.prefix(prefix)).iterateAll()) { - fileUris.add(String.format("%s://%s/%s", scheme, blob.getBucket(), blob.getName())); + private FieldValueList getTimestampLimits(String entityTableName) { + QueryJobConfiguration getTimestampLimitsQuery = + QueryJobConfiguration.newBuilder(createTimestampLimitQuery(entityTableName)) + .setDefaultDataset(DatasetId.of(projectId(), datasetId())) + .setDestinationTable(TableId.of(projectId(), datasetId(), createTempTableName())) + .build(); + try { + Job job = bigquery().create(JobInfo.of(getTimestampLimitsQuery)); + TableResult getTimestampLimitsQueryResult = waitForJob(job).getQueryResults(); + TableInfo expiry = + bigquery() + .getTable(getTimestampLimitsQuery.getDestinationTable()) + .toBuilder() + .setExpirationTime(System.currentTimeMillis() + TEMP_TABLE_EXPIRY_DURATION_MS) + .build(); + bigquery().update(expiry); + FieldValueList result = null; + for (FieldValueList fields : getTimestampLimitsQueryResult.getValues()) { + result = fields; + } + if (result == null || result.get("min").isNull() || result.get("max").isNull()) { + throw new RuntimeException("query returned insufficient values"); + } + return result; + } catch (InterruptedException e) { + throw Status.INTERNAL + .withDescription("Unable to extract min and max timestamps from query") + .withCause(e) + .asRuntimeException(); + } + } + + private Table loadEntities(ServingAPIProto.DatasetSource datasetSource) { + Table loadedEntityTable; + switch (datasetSource.getDatasetSourceCase()) { + case FILE_SOURCE: + try { + // Currently only AVRO format is supported + if (datasetSource.getFileSource().getDataFormat() + != ServingAPIProto.DataFormat.DATA_FORMAT_AVRO) { + throw Status.INVALID_ARGUMENT + .withDescription("Invalid file format, only AVRO is supported.") + .asRuntimeException(); + } + + TableId tableId = TableId.of(projectId(), datasetId(), createTempTableName()); + log.info( + "Loading entity rows to: {}.{}.{}", projectId(), datasetId(), tableId.getTable()); + + LoadJobConfiguration loadJobConfiguration = + LoadJobConfiguration.of( + tableId, datasetSource.getFileSource().getFileUrisList(), FormatOptions.avro()); + loadJobConfiguration = + loadJobConfiguration.toBuilder().setUseAvroLogicalTypes(true).build(); + Job job = bigquery().create(JobInfo.of(loadJobConfiguration)); + waitForJob(job); + + TableInfo expiry = + bigquery() + .getTable(tableId) + .toBuilder() + .setExpirationTime(System.currentTimeMillis() + TEMP_TABLE_EXPIRY_DURATION_MS) + .build(); + bigquery().update(expiry); + + loadedEntityTable = bigquery().getTable(tableId); + if (!loadedEntityTable.exists()) { + throw new RuntimeException( + "Unable to create entity dataset table, table already exists"); + } + return loadedEntityTable; + } catch (Exception e) { + log.error("Exception has occurred in loadEntities method: ", e); + throw Status.INTERNAL + .withDescription("Failed to load entity dataset into store: " + e.toString()) + .withCause(e) + .asRuntimeException(); + } + case DATASETSOURCE_NOT_SET: + default: + throw Status.INVALID_ARGUMENT + .withDescription("Data source must be set.") + .asRuntimeException(); } - return fileUris; } - Job runBatchQuery(List featureSetQueries) + private List generateQueries( + String entityTableName, + FieldValueList timestampLimits, + List featureSetQueryInfos) { + List featureSetQueries = new ArrayList<>(); + try { + for (FeatureSetQueryInfo featureSetInfo : featureSetQueryInfos) { + String query = + QueryTemplater.createFeatureSetPointInTimeQuery( + featureSetInfo, + projectId(), + datasetId(), + entityTableName, + timestampLimits.get("min").getStringValue(), + timestampLimits.get("max").getStringValue()); + featureSetQueries.add(query); + } + } catch (IOException e) { + throw Status.INTERNAL + .withDescription("Unable to generate query for batch retrieval") + .withCause(e) + .asRuntimeException(); + } + return featureSetQueries; + } + + Job runBatchQuery( + String entityTableName, + List entityTableColumnNames, + List featureSetQueryInfos, + List featureSetQueries) throws BigQueryException, InterruptedException, IOException { ExecutorService executorService = Executors.newFixedThreadPool(featureSetQueries.size()); - ExecutorCompletionService executorCompletionService = + ExecutorCompletionService executorCompletionService = new ExecutorCompletionService<>(executorService); - List featureSetInfos = new ArrayList<>(); - // For each of the feature sets requested, start an async job joining the features in that // feature set to the provided entity table for (int i = 0; i < featureSetQueries.size(); i++) { @@ -227,28 +314,21 @@ Job runBatchQuery(List featureSetQueries) executorCompletionService.submit( SubqueryCallable.builder() .setBigquery(bigquery()) - .setFeatureSetInfo(featureSetInfos().get(i)) + .setFeatureSetInfo(featureSetQueryInfos.get(i)) .setSubqueryJob(subqueryJob) .build()); } + List completedFeatureSetQueryInfos = new ArrayList<>(); + for (int i = 0; i < featureSetQueries.size(); i++) { try { // Try to retrieve the outputs of all the jobs. The timeout here is a formality; // a stricter timeout is implemented in the actual SubqueryCallable. - FeatureSetInfo featureSetInfo = + FeatureSetQueryInfo featureSetInfo = executorCompletionService.take().get(SUBQUERY_TIMEOUT_SECS, TimeUnit.SECONDS); - featureSetInfos.add(featureSetInfo); + completedFeatureSetQueryInfos.add(featureSetInfo); } catch (InterruptedException | ExecutionException | TimeoutException e) { - jobService() - .upsert( - ServingAPIProto.Job.newBuilder() - .setId(feastJobId()) - .setType(JobType.JOB_TYPE_DOWNLOAD) - .setStatus(JobStatus.JOB_STATUS_DONE) - .setError(e.getMessage()) - .build()); - executorService.shutdownNow(); throw Status.INTERNAL .withDescription("Error running batch query") @@ -261,7 +341,7 @@ Job runBatchQuery(List featureSetQueries) // subqueries into a single table. String joinQuery = QueryTemplater.createJoinQuery( - featureSetInfos, entityTableColumnNames(), entityTableName()); + completedFeatureSetQueryInfos, entityTableColumnNames, entityTableName); QueryJobConfiguration queryJobConfig = QueryJobConfiguration.newBuilder(joinQuery) .setDestinationTable(TableId.of(projectId(), datasetId(), createTempTableName())) @@ -280,59 +360,24 @@ Job runBatchQuery(List featureSetQueries) return completedQueryJob; } - private List generateQueries(FieldValueList timestampLimits) { - List featureSetQueries = new ArrayList<>(); - try { - for (FeatureSetInfo featureSetInfo : featureSetInfos()) { - String query = - QueryTemplater.createFeatureSetPointInTimeQuery( - featureSetInfo, - projectId(), - datasetId(), - entityTableName(), - timestampLimits.get("min").getStringValue(), - timestampLimits.get("max").getStringValue()); - featureSetQueries.add(query); - } - } catch (IOException e) { - throw Status.INTERNAL - .withDescription("Unable to generate query for batch retrieval") - .withCause(e) - .asRuntimeException(); - } - return featureSetQueries; - } + private List parseOutputFileURIs(String feastJobId) { + String scheme = jobStagingLocation().substring(0, jobStagingLocation().indexOf("://")); + String stagingLocationNoScheme = + jobStagingLocation().substring(jobStagingLocation().indexOf("://") + 3); + String bucket = stagingLocationNoScheme.split("/")[0]; + List prefixParts = new ArrayList<>(); + prefixParts.add( + stagingLocationNoScheme.contains("/") && !stagingLocationNoScheme.endsWith("/") + ? stagingLocationNoScheme.substring(stagingLocationNoScheme.indexOf("/") + 1) + : ""); + prefixParts.add(feastJobId); + String prefix = String.join("/", prefixParts) + "/"; - private FieldValueList getTimestampLimits(String entityTableName) { - QueryJobConfiguration getTimestampLimitsQuery = - QueryJobConfiguration.newBuilder(createTimestampLimitQuery(entityTableName)) - .setDefaultDataset(DatasetId.of(projectId(), datasetId())) - .setDestinationTable(TableId.of(projectId(), datasetId(), createTempTableName())) - .build(); - try { - Job job = bigquery().create(JobInfo.of(getTimestampLimitsQuery)); - TableResult getTimestampLimitsQueryResult = waitForJob(job).getQueryResults(); - TableInfo expiry = - bigquery() - .getTable(getTimestampLimitsQuery.getDestinationTable()) - .toBuilder() - .setExpirationTime(System.currentTimeMillis() + TEMP_TABLE_EXPIRY_DURATION_MS) - .build(); - bigquery().update(expiry); - FieldValueList result = null; - for (FieldValueList fields : getTimestampLimitsQueryResult.getValues()) { - result = fields; - } - if (result == null || result.get("min").isNull() || result.get("max").isNull()) { - throw new RuntimeException("query returned insufficient values"); - } - return result; - } catch (InterruptedException e) { - throw Status.INTERNAL - .withDescription("Unable to extract min and max timestamps from query") - .withCause(e) - .asRuntimeException(); + List fileUris = new ArrayList<>(); + for (Blob blob : storage().list(bucket, Storage.BlobListOption.prefix(prefix)).iterateAll()) { + fileUris.add(String.format("%s://%s/%s", scheme, blob.getBucket(), blob.getName())); } + return fileUris; } private Job waitForJob(Job queryJob) throws InterruptedException { @@ -349,4 +394,13 @@ private Job waitForJob(Job queryJob) throws InterruptedException { } return completedJob; } + + public String generateFullTableName(TableId tableId) { + return String.format( + "%s.%s.%s", tableId.getProject(), tableId.getDataset(), tableId.getTable()); + } + + public String createTempTableName() { + return "_" + UUID.randomUUID().toString().replace("-", ""); + } } diff --git a/serving/src/main/java/feast/serving/store/bigquery/model/FeatureSetInfo.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/FeatureSetQueryInfo.java similarity index 90% rename from serving/src/main/java/feast/serving/store/bigquery/model/FeatureSetInfo.java rename to storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/FeatureSetQueryInfo.java index 77c80ead0ea..5a7d56e9844 100644 --- a/serving/src/main/java/feast/serving/store/bigquery/model/FeatureSetInfo.java +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/FeatureSetQueryInfo.java @@ -14,11 +14,11 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.serving.store.bigquery.model; +package feast.storage.connectors.bigquery.retriever; import java.util.List; -public class FeatureSetInfo { +public class FeatureSetQueryInfo { private final String project; private final String name; @@ -28,7 +28,7 @@ public class FeatureSetInfo { private final List features; private final String table; - public FeatureSetInfo( + public FeatureSetQueryInfo( String project, String name, int version, @@ -45,7 +45,7 @@ public FeatureSetInfo( this.table = table; } - public FeatureSetInfo(FeatureSetInfo featureSetInfo, String table) { + public FeatureSetQueryInfo(FeatureSetQueryInfo featureSetInfo, String table) { this.project = featureSetInfo.getProject(); this.name = featureSetInfo.getName(); diff --git a/serving/src/main/java/feast/serving/store/bigquery/QueryTemplater.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/QueryTemplater.java similarity index 90% rename from serving/src/main/java/feast/serving/store/bigquery/QueryTemplater.java rename to storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/QueryTemplater.java index e3f1138db89..cba997b6ab0 100644 --- a/serving/src/main/java/feast/serving/store/bigquery/QueryTemplater.java +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/QueryTemplater.java @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.serving.store.bigquery; +package feast.storage.connectors.bigquery.retriever; import com.google.cloud.bigquery.TableId; import com.google.protobuf.Duration; @@ -23,8 +23,7 @@ import feast.core.FeatureSetProto.EntitySpec; import feast.core.FeatureSetProto.FeatureSetSpec; import feast.serving.ServingAPIProto.FeatureReference; -import feast.serving.specs.FeatureSetRequest; -import feast.serving.store.bigquery.model.FeatureSetInfo; +import feast.storage.api.retriever.FeatureSetRequest; import java.io.IOException; import java.io.StringWriter; import java.io.Writer; @@ -67,13 +66,14 @@ public static String createEntityTableUUIDQuery(String leftTableName) { * Generate the information necessary for the sql templating for point in time correctness join to * the entity dataset for each feature set requested. * - * @param featureSetRequests List of feature sets requested + * @param featureSetRequests List of {@link FeatureSetRequest} containing a {@link FeatureSetSpec} + * and its corresponding {@link FeatureReference}s provided by the user. * @return List of FeatureSetInfos */ - public static List getFeatureSetInfos(List featureSetRequests) - throws IllegalArgumentException { + public static List getFeatureSetInfos( + List featureSetRequests) throws IllegalArgumentException { - List featureSetInfos = new ArrayList<>(); + List featureSetInfos = new ArrayList<>(); for (FeatureSetRequest featureSetRequest : featureSetRequests) { FeatureSetSpec spec = featureSetRequest.getSpec(); Duration maxAge = spec.getMaxAge(); @@ -84,7 +84,7 @@ public static List getFeatureSetInfos(List fe .map(FeatureReference::getName) .collect(Collectors.toList()); featureSetInfos.add( - new FeatureSetInfo( + new FeatureSetQueryInfo( spec.getProject(), spec.getName(), spec.getVersion(), @@ -109,7 +109,7 @@ public static List getFeatureSetInfos(List fe * @return point in time correctness join BQ SQL query */ public static String createFeatureSetPointInTimeQuery( - FeatureSetInfo featureSetInfo, + FeatureSetQueryInfo featureSetInfo, String projectId, String datasetId, String leftTableName, @@ -139,7 +139,7 @@ public static String createFeatureSetPointInTimeQuery( * @return query to join temporary feature set tables to the entity table */ public static String createJoinQuery( - List featureSetInfos, + List featureSetInfos, List entityTableColumnNames, String leftTableName) throws IOException { diff --git a/serving/src/main/java/feast/serving/store/bigquery/SubqueryCallable.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/SubqueryCallable.java similarity index 70% rename from serving/src/main/java/feast/serving/store/bigquery/SubqueryCallable.java rename to storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/SubqueryCallable.java index 14026030b42..43a32cef504 100644 --- a/serving/src/main/java/feast/serving/store/bigquery/SubqueryCallable.java +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/SubqueryCallable.java @@ -14,19 +14,13 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.serving.store.bigquery; +package feast.storage.connectors.bigquery.retriever; -import static feast.serving.service.BigQueryServingService.TEMP_TABLE_EXPIRY_DURATION_MS; -import static feast.serving.store.bigquery.QueryTemplater.generateFullTableName; +import static feast.storage.connectors.bigquery.retriever.BigQueryHistoricalRetriever.TEMP_TABLE_EXPIRY_DURATION_MS; +import static feast.storage.connectors.bigquery.retriever.QueryTemplater.generateFullTableName; import com.google.auto.value.AutoValue; -import com.google.cloud.bigquery.BigQuery; -import com.google.cloud.bigquery.BigQueryException; -import com.google.cloud.bigquery.Job; -import com.google.cloud.bigquery.QueryJobConfiguration; -import com.google.cloud.bigquery.TableId; -import com.google.cloud.bigquery.TableInfo; -import feast.serving.store.bigquery.model.FeatureSetInfo; +import com.google.cloud.bigquery.*; import java.util.concurrent.Callable; /** @@ -34,11 +28,11 @@ * updated with the reference to the table containing the results of the query. */ @AutoValue -public abstract class SubqueryCallable implements Callable { +public abstract class SubqueryCallable implements Callable { public abstract BigQuery bigquery(); - public abstract FeatureSetInfo featureSetInfo(); + public abstract FeatureSetQueryInfo featureSetInfo(); public abstract Job subqueryJob(); @@ -51,7 +45,7 @@ public abstract static class Builder { public abstract Builder setBigquery(BigQuery bigquery); - public abstract Builder setFeatureSetInfo(FeatureSetInfo featureSetInfo); + public abstract Builder setFeatureSetInfo(FeatureSetQueryInfo featureSetInfo); public abstract Builder setSubqueryJob(Job subqueryJob); @@ -59,7 +53,7 @@ public abstract static class Builder { } @Override - public FeatureSetInfo call() throws BigQueryException, InterruptedException { + public FeatureSetQueryInfo call() throws BigQueryException, InterruptedException { QueryJobConfiguration subqueryConfig; subqueryJob().waitFor(); subqueryConfig = subqueryJob().getConfiguration(); @@ -75,6 +69,6 @@ public FeatureSetInfo call() throws BigQueryException, InterruptedException { String fullTablePath = generateFullTableName(destinationTable); - return new FeatureSetInfo(featureSetInfo(), fullTablePath); + return new FeatureSetQueryInfo(featureSetInfo(), fullTablePath); } } diff --git a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryDeadletterSink.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryDeadletterSink.java new file mode 100644 index 00000000000..96364c96c78 --- /dev/null +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryDeadletterSink.java @@ -0,0 +1,133 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.bigquery.writer; + +import com.google.api.services.bigquery.model.TableRow; +import com.google.api.services.bigquery.model.TimePartitioning; +import com.google.auto.value.AutoValue; +import com.google.common.io.Resources; +import feast.storage.api.writer.DeadletterSink; +import feast.storage.api.writer.FailedElement; +import java.nio.charset.StandardCharsets; +import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO; +import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.CreateDisposition; +import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO.Write.WriteDisposition; +import org.apache.beam.sdk.transforms.DoFn; +import org.apache.beam.sdk.transforms.PTransform; +import org.apache.beam.sdk.transforms.ParDo; +import org.apache.beam.sdk.values.PCollection; +import org.apache.beam.sdk.values.PDone; +import org.slf4j.Logger; + +public class BigQueryDeadletterSink implements DeadletterSink { + + private static final String DEADLETTER_SCHEMA_FILE_PATH = "schemas/deadletter_table_schema.json"; + private static final Logger log = org.slf4j.LoggerFactory.getLogger(BigQueryDeadletterSink.class); + private static final String TIMESTAMP_COLUMN = "timestamp"; + + private final String tableSpec; + private String jsonSchema; + + public BigQueryDeadletterSink(String tableSpec) { + + this.tableSpec = tableSpec; + try { + jsonSchema = + Resources.toString( + Resources.getResource(DEADLETTER_SCHEMA_FILE_PATH), StandardCharsets.UTF_8); + } catch (Exception e) { + log.error( + "Unable to read {} file from the resources folder!", DEADLETTER_SCHEMA_FILE_PATH, e); + } + } + + @Override + public void prepareWrite() {} + + @Override + public PTransform, PDone> write() { + return WriteFailedElement.newBuilder() + .setJsonSchema(jsonSchema) + .setTableSpec(tableSpec) + .build(); + } + + @AutoValue + public abstract static class WriteFailedElement + extends PTransform, PDone> { + + public abstract String getTableSpec(); + + public abstract String getJsonSchema(); + + public static Builder newBuilder() { + return new AutoValue_BigQueryDeadletterSink_WriteFailedElement.Builder(); + } + + @AutoValue.Builder + public abstract static class Builder { + + /** + * @param tableSpec Table spec should follow the format "PROJECT_ID:DATASET_ID.TABLE_ID". + * Table will be created if not exists. + */ + public abstract Builder setTableSpec(String tableSpec); + + /** + * @param jsonSchema JSON string describing the schema + * of the table. + */ + public abstract Builder setJsonSchema(String jsonSchema); + + public abstract WriteFailedElement build(); + } + + @Override + public PDone expand(PCollection input) { + TimePartitioning partition = new TimePartitioning().setType("DAY"); + partition.setField(TIMESTAMP_COLUMN); + input + .apply("FailedElementToTableRow", ParDo.of(new FailedElementToTableRowFn())) + .apply( + "WriteFailedElementsToBigQuery", + BigQueryIO.writeTableRows() + .to(getTableSpec()) + .withJsonSchema(getJsonSchema()) + .withTimePartitioning(partition) + .withCreateDisposition(CreateDisposition.CREATE_IF_NEEDED) + .withWriteDisposition(WriteDisposition.WRITE_APPEND)); + return PDone.in(input.getPipeline()); + } + } + + public static class FailedElementToTableRowFn extends DoFn { + @ProcessElement + public void processElement(ProcessContext context) { + final FailedElement element = context.element(); + final TableRow tableRow = + new TableRow() + .set(TIMESTAMP_COLUMN, element.getTimestamp().toString()) + .set("job_name", element.getJobName()) + .set("transform_name", element.getTransformName()) + .set("payload", element.getPayload()) + .set("error_message", element.getErrorMessage()) + .set("stack_trace", element.getStackTrace()); + context.output(tableRow); + } + } +} diff --git a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryFeatureSink.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryFeatureSink.java new file mode 100644 index 00000000000..8860db2622a --- /dev/null +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryFeatureSink.java @@ -0,0 +1,188 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.bigquery.writer; + +import com.google.auto.value.AutoValue; +import com.google.cloud.bigquery.*; +import com.google.common.collect.ImmutableMap; +import feast.core.FeatureSetProto; +import feast.core.StoreProto.Store.BigQueryConfig; +import feast.storage.api.writer.FeatureSink; +import feast.storage.api.writer.WriteResult; +import feast.storage.connectors.bigquery.common.TypeUtil; +import feast.types.FeatureRowProto; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import org.apache.beam.repackaged.core.org.apache.commons.lang3.tuple.Pair; +import org.apache.beam.sdk.transforms.PTransform; +import org.apache.beam.sdk.values.PCollection; +import org.slf4j.Logger; + +@AutoValue +public abstract class BigQueryFeatureSink implements FeatureSink { + private static final Logger log = org.slf4j.LoggerFactory.getLogger(BigQueryFeatureSink.class); + + // Column description for reserved fields + public static final String BIGQUERY_EVENT_TIMESTAMP_FIELD_DESCRIPTION = + "Event time for the FeatureRow"; + public static final String BIGQUERY_CREATED_TIMESTAMP_FIELD_DESCRIPTION = + "Processing time of the FeatureRow ingestion in Feast\""; + public static final String BIGQUERY_JOB_ID_FIELD_DESCRIPTION = + "Feast import job ID for the FeatureRow"; + + public abstract String getProjectId(); + + public abstract String getDatasetId(); + + public abstract BigQuery getBigQuery(); + + /** + * Initialize a {@link BigQueryFeatureSink.Builder} from a {@link BigQueryConfig}. This method + * initializes a {@link BigQuery} client with default options. Use the builder method to inject + * your own client. + * + * @param config {@link BigQueryConfig} + * @return {@link BigQueryFeatureSink.Builder} + */ + public static BigQueryFeatureSink fromConfig(BigQueryConfig config) { + return builder() + .setDatasetId(config.getDatasetId()) + .setProjectId(config.getProjectId()) + .setBigQuery(BigQueryOptions.getDefaultInstance().getService()) + .build(); + } + + public static Builder builder() { + return new AutoValue_BigQueryFeatureSink.Builder(); + } + + @AutoValue.Builder + public abstract static class Builder { + + public abstract Builder setProjectId(String projectId); + + public abstract Builder setDatasetId(String datasetId); + + public abstract Builder setBigQuery(BigQuery bigQuery); + + public abstract BigQueryFeatureSink build(); + } + + /** @param featureSet Feature set to be written */ + @Override + public void prepareWrite(FeatureSetProto.FeatureSet featureSet) { + BigQuery bigquery = getBigQuery(); + FeatureSetProto.FeatureSetSpec featureSetSpec = featureSet.getSpec(); + + DatasetId datasetId = DatasetId.of(getProjectId(), getDatasetId()); + if (bigquery.getDataset(datasetId) == null) { + log.info( + "Creating dataset '{}' in project '{}'", datasetId.getDataset(), datasetId.getProject()); + bigquery.create(DatasetInfo.of(datasetId)); + } + String tableName = + String.format( + "%s_%s_v%d", + featureSetSpec.getProject(), featureSetSpec.getName(), featureSetSpec.getVersion()) + .replaceAll("-", "_"); + TableId tableId = TableId.of(datasetId.getProject(), datasetId.getDataset(), tableName); + + // Return if there is an existing table + Table table = bigquery.getTable(tableId); + if (table != null) { + log.info( + "Writing to existing BigQuery table '{}:{}.{}'", + getProjectId(), + datasetId.getDataset(), + tableName); + return; + } + + log.info( + "Creating table '{}' in dataset '{}' in project '{}'", + tableId.getTable(), + datasetId.getDataset(), + datasetId.getProject()); + TableDefinition tableDefinition = createBigQueryTableDefinition(featureSet.getSpec()); + TableInfo tableInfo = TableInfo.of(tableId, tableDefinition); + bigquery.create(tableInfo); + } + + @Override + public PTransform, WriteResult> writer() { + return new BigQueryWrite(DatasetId.of(getProjectId(), getDatasetId())); + } + + private TableDefinition createBigQueryTableDefinition(FeatureSetProto.FeatureSetSpec spec) { + List fields = new ArrayList<>(); + log.info("Table will have the following fields:"); + + for (FeatureSetProto.EntitySpec entitySpec : spec.getEntitiesList()) { + Field.Builder builder = + Field.newBuilder( + entitySpec.getName(), TypeUtil.toStandardSqlType(entitySpec.getValueType())); + if (entitySpec.getValueType().name().toLowerCase().endsWith("_list")) { + builder.setMode(Field.Mode.REPEATED); + } + Field field = builder.build(); + log.info("- {}", field.toString()); + fields.add(field); + } + for (FeatureSetProto.FeatureSpec featureSpec : spec.getFeaturesList()) { + Field.Builder builder = + Field.newBuilder( + featureSpec.getName(), TypeUtil.toStandardSqlType(featureSpec.getValueType())); + if (featureSpec.getValueType().name().toLowerCase().endsWith("_list")) { + builder.setMode(Field.Mode.REPEATED); + } + Field field = builder.build(); + log.info("- {}", field.toString()); + fields.add(field); + } + + // Refer to protos/feast/core/Store.proto for reserved fields in BigQuery. + Map> + reservedFieldNameToPairOfStandardSQLTypeAndDescription = + ImmutableMap.of( + "event_timestamp", + Pair.of(StandardSQLTypeName.TIMESTAMP, BIGQUERY_EVENT_TIMESTAMP_FIELD_DESCRIPTION), + "created_timestamp", + Pair.of( + StandardSQLTypeName.TIMESTAMP, BIGQUERY_CREATED_TIMESTAMP_FIELD_DESCRIPTION), + "job_id", + Pair.of(StandardSQLTypeName.STRING, BIGQUERY_JOB_ID_FIELD_DESCRIPTION)); + for (Map.Entry> entry : + reservedFieldNameToPairOfStandardSQLTypeAndDescription.entrySet()) { + Field field = + Field.newBuilder(entry.getKey(), entry.getValue().getLeft()) + .setDescription(entry.getValue().getRight()) + .build(); + log.info("- {}", field.toString()); + fields.add(field); + } + + TimePartitioning timePartitioning = + TimePartitioning.newBuilder(TimePartitioning.Type.DAY).setField("event_timestamp").build(); + log.info("Table partitioning: " + timePartitioning.toString()); + + return StandardTableDefinition.newBuilder() + .setTimePartitioning(timePartitioning) + .setSchema(Schema.of(fields)) + .build(); + } +} diff --git a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryWrite.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryWrite.java new file mode 100644 index 00000000000..e3f5e5ae713 --- /dev/null +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryWrite.java @@ -0,0 +1,107 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.bigquery.writer; + +import com.google.api.services.bigquery.model.TableDataInsertAllResponse; +import com.google.api.services.bigquery.model.TableRow; +import com.google.cloud.bigquery.DatasetId; +import feast.storage.api.writer.FailedElement; +import feast.storage.api.writer.WriteResult; +import feast.types.FeatureRowProto; +import java.io.IOException; +import org.apache.beam.sdk.io.gcp.bigquery.BigQueryIO; +import org.apache.beam.sdk.io.gcp.bigquery.BigQueryInsertError; +import org.apache.beam.sdk.io.gcp.bigquery.InsertRetryPolicy; +import org.apache.beam.sdk.transforms.DoFn; +import org.apache.beam.sdk.transforms.PTransform; +import org.apache.beam.sdk.transforms.ParDo; +import org.apache.beam.sdk.values.PCollection; +import org.slf4j.Logger; + +/** + * A {@link PTransform} that writes {@link FeatureRowProto FeatureRows} to the specified BigQuery + * dataset, and returns a {@link WriteResult} containing the unsuccessful writes. Since Bigquery + * does not output successful writes, we cannot emit those, and so no success metrics will be + * captured if this sink is used. + */ +public class BigQueryWrite + extends PTransform, WriteResult> { + private static final Logger log = org.slf4j.LoggerFactory.getLogger(BigQueryWrite.class); + + // Destination dataset + private DatasetId destination; + + public BigQueryWrite(DatasetId destination) { + this.destination = destination; + } + + @Override + public WriteResult expand(PCollection input) { + String jobName = input.getPipeline().getOptions().getJobName(); + org.apache.beam.sdk.io.gcp.bigquery.WriteResult bigqueryWriteResult = + input.apply( + "WriteTableRowToBigQuery", + BigQueryIO.write() + .to(new GetTableDestination(destination.getProject(), destination.getDataset())) + .withFormatFunction(new FeatureRowToTableRow(jobName)) + .withCreateDisposition(BigQueryIO.Write.CreateDisposition.CREATE_NEVER) + .withWriteDisposition(BigQueryIO.Write.WriteDisposition.WRITE_APPEND) + .withExtendedErrorInfo() + .withMethod(BigQueryIO.Write.Method.STREAMING_INSERTS) + .withFailedInsertRetryPolicy(InsertRetryPolicy.retryTransientErrors())); + + PCollection failedElements = + bigqueryWriteResult + .getFailedInsertsWithErr() + .apply( + "WrapBigQueryInsertionError", + ParDo.of( + new DoFn() { + @ProcessElement + public void processElement(ProcessContext context) { + TableDataInsertAllResponse.InsertErrors error = + context.element().getError(); + TableRow row = context.element().getRow(); + try { + context.output( + FailedElement.newBuilder() + .setErrorMessage(error.toPrettyString()) + .setPayload(row.toPrettyString()) + .setJobName(context.getPipelineOptions().getJobName()) + .setTransformName("WriteTableRowToBigQuery") + .build()); + } catch (IOException e) { + log.error(e.getMessage()); + } + } + })); + + // Since BigQueryIO does not support emitting successful writes, we set successfulInserts to + // an empty stream, + // and no metrics will be collected. + PCollection successfulInserts = + input.apply( + "dummy", + ParDo.of( + new DoFn() { + @ProcessElement + public void processElement(ProcessContext context) {} + })); + + return WriteResult.in(input.getPipeline(), successfulInserts, failedElements); + } +} diff --git a/ingestion/src/main/java/feast/store/serving/bigquery/FeatureRowToTableRow.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/FeatureRowToTableRow.java similarity index 98% rename from ingestion/src/main/java/feast/store/serving/bigquery/FeatureRowToTableRow.java rename to storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/FeatureRowToTableRow.java index b89cf832910..12833b31b85 100644 --- a/ingestion/src/main/java/feast/store/serving/bigquery/FeatureRowToTableRow.java +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/FeatureRowToTableRow.java @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.store.serving.bigquery; +package feast.storage.connectors.bigquery.writer; import com.google.api.services.bigquery.model.TableRow; import com.google.protobuf.util.Timestamps; diff --git a/ingestion/src/main/java/feast/store/serving/bigquery/GetTableDestination.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/GetTableDestination.java similarity index 97% rename from ingestion/src/main/java/feast/store/serving/bigquery/GetTableDestination.java rename to storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/GetTableDestination.java index eb37db94498..5903d36b858 100644 --- a/ingestion/src/main/java/feast/store/serving/bigquery/GetTableDestination.java +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/GetTableDestination.java @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.store.serving.bigquery; +package feast.storage.connectors.bigquery.writer; import com.google.api.services.bigquery.model.TimePartitioning; import feast.types.FeatureRowProto.FeatureRow; diff --git a/storage/connectors/bigquery/src/main/resources/schemas/deadletter_table_schema.json b/storage/connectors/bigquery/src/main/resources/schemas/deadletter_table_schema.json new file mode 100644 index 00000000000..92381189073 --- /dev/null +++ b/storage/connectors/bigquery/src/main/resources/schemas/deadletter_table_schema.json @@ -0,0 +1,34 @@ +{ + "fields": [ + { + "name": "timestamp", + "type": "TIMESTAMP", + "mode": "REQUIRED" + }, + { + "name": "job_name", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "transform_name", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "payload", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "error_message", + "type": "STRING", + "mode": "NULLABLE" + }, + { + "name": "stack_trace", + "type": "STRING", + "mode": "NULLABLE" + } + ] +} \ No newline at end of file diff --git a/storage/connectors/bigquery/src/main/resources/templates/join_featuresets.sql b/storage/connectors/bigquery/src/main/resources/templates/join_featuresets.sql new file mode 100644 index 00000000000..60b7c7d7a12 --- /dev/null +++ b/storage/connectors/bigquery/src/main/resources/templates/join_featuresets.sql @@ -0,0 +1,24 @@ +/* + Joins the outputs of multiple point-in-time-correctness joins to a single table. + */ +WITH joined as ( +SELECT * FROM `{{ leftTableName }}` +{% for featureSet in featureSets %} +LEFT JOIN ( + SELECT + uuid, + {% for featureName in featureSet.features %} + {{ featureSet.project }}_{{ featureName }}_v{{ featureSet.version }}{% if loop.last %}{% else %}, {% endif %} + {% endfor %} + FROM `{{ featureSet.table }}` +) USING (uuid) +{% endfor %} +) SELECT + event_timestamp, + {{ entities | join(', ') }} + {% for featureSet in featureSets %} + {% for featureName in featureSet.features %} + ,{{ featureSet.project }}_{{ featureName }}_v{{ featureSet.version }} as {{ featureName }} + {% endfor %} + {% endfor %} +FROM joined \ No newline at end of file diff --git a/storage/connectors/bigquery/src/main/resources/templates/single_featureset_pit_join.sql b/storage/connectors/bigquery/src/main/resources/templates/single_featureset_pit_join.sql new file mode 100644 index 00000000000..fb4c555b529 --- /dev/null +++ b/storage/connectors/bigquery/src/main/resources/templates/single_featureset_pit_join.sql @@ -0,0 +1,90 @@ +/* + This query template performs the point-in-time correctness join for a single feature set table + to the provided entity table. + + 1. Concatenate the timestamp and entities from the feature set table with the entity dataset. + Feature values are joined to this table later for improved efficiency. + featureset_timestamp is equal to null in rows from the entity dataset. + */ +WITH union_features AS ( +SELECT + -- uuid is a unique identifier for each row in the entity dataset. Generated by `QueryTemplater.createEntityTableUUIDQuery` + uuid, + -- event_timestamp contains the timestamps to join onto + event_timestamp, + -- the feature_timestamp, i.e. the latest occurrence of the requested feature relative to the entity_dataset timestamp + NULL as {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, + -- created timestamp of the feature at the corresponding feature_timestamp + NULL as created_timestamp, + -- select only entities belonging to this feature set + {{ featureSet.entities | join(', ')}}, + -- boolean for filtering the dataset later + true AS is_entity_table +FROM `{{leftTableName}}` +UNION ALL +SELECT + NULL as uuid, + event_timestamp, + event_timestamp as {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, + created_timestamp, + {{ featureSet.entities | join(', ')}}, + false AS is_entity_table +FROM `{{projectId}}.{{datasetId}}.{{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}` WHERE event_timestamp <= '{{maxTimestamp}}' +{% if featureSet.maxAge == 0 %}{% else %}AND event_timestamp >= Timestamp_sub(TIMESTAMP '{{ minTimestamp }}', interval {{ featureSet.maxAge }} second){% endif %} +), +/* + 2. Window the data in the unioned dataset, partitioning by entity and ordering by event_timestamp, as + well as is_entity_table. + Within each window, back-fill the feature_timestamp - as a result of this, the null feature_timestamps + in the rows from the entity table should now contain the latest timestamps relative to the row's + event_timestamp. + + For rows where event_timestamp(provided datetime) - feature_timestamp > max age, set the + feature_timestamp to null. + */ +joined AS ( +SELECT + uuid, + event_timestamp, + {{ featureSet.entities | join(', ')}}, + {% for featureName in featureSet.features %} + IF(event_timestamp >= {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp {% if featureSet.maxAge == 0 %}{% else %}AND Timestamp_sub(event_timestamp, interval {{ featureSet.maxAge }} second) < {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp{% endif %}, {{ featureSet.project }}_{{ featureName }}_v{{ featureSet.version }}, NULL) as {{ featureSet.project }}_{{ featureName }}_v{{ featureSet.version }}{% if loop.last %}{% else %}, {% endif %} + {% endfor %} +FROM ( +SELECT + uuid, + event_timestamp, + {{ featureSet.entities | join(', ')}}, + FIRST_VALUE(created_timestamp IGNORE NULLS) over w AS created_timestamp, + FIRST_VALUE({{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp IGNORE NULLS) over w AS {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, + is_entity_table +FROM union_features +WINDOW w AS (PARTITION BY {{ featureSet.entities | join(', ') }} ORDER BY event_timestamp DESC, is_entity_table DESC, created_timestamp DESC ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) +) +/* + 3. Select only the rows from the entity table, and join the features from the original feature set table + to the dataset using the entity values, feature_timestamp, and created_timestamps. + */ +LEFT JOIN ( +SELECT + event_timestamp as {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, + created_timestamp, + {{ featureSet.entities | join(', ')}}, + {% for featureName in featureSet.features %} + {{ featureName }} as {{ featureSet.project }}_{{ featureName }}_v{{ featureSet.version }}{% if loop.last %}{% else %}, {% endif %} + {% endfor %} +FROM `{{ projectId }}.{{ datasetId }}.{{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}` WHERE event_timestamp <= '{{maxTimestamp}}' +{% if featureSet.maxAge == 0 %}{% else %}AND event_timestamp >= Timestamp_sub(TIMESTAMP '{{ minTimestamp }}', interval {{ featureSet.maxAge }} second){% endif %} +) USING ({{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, created_timestamp, {{ featureSet.entities | join(', ')}}) +WHERE is_entity_table +) +/* + 4. Finally, deduplicate the rows by selecting the first occurrence of each entity table row UUID. + */ +SELECT + k.* +FROM ( + SELECT ARRAY_AGG(row LIMIT 1)[OFFSET(0)] k + FROM joined row + GROUP BY uuid +) \ No newline at end of file diff --git a/storage/connectors/pom.xml b/storage/connectors/pom.xml new file mode 100644 index 00000000000..b52668a31a4 --- /dev/null +++ b/storage/connectors/pom.xml @@ -0,0 +1,51 @@ + + + + dev.feast + feast-parent + ${revision} + ../.. + + + 4.0.0 + feast-storage-connectors + pom + + Feast Storage Connectors + + + redis + bigquery + + + + + + org.apache.maven.plugins + maven-dependency-plugin + + + + javax.annotation + + + + + + + + + dev.feast + datatypes-java + ${project.version} + + + + dev.feast + feast-storage-api + ${project.version} + + + + diff --git a/storage/connectors/redis/pom.xml b/storage/connectors/redis/pom.xml new file mode 100644 index 00000000000..6c50895bd20 --- /dev/null +++ b/storage/connectors/redis/pom.xml @@ -0,0 +1,82 @@ + + + + dev.feast + feast-storage-connectors + ${revision} + + + 4.0.0 + feast-storage-connector-redis + + Feast Storage Connector for Redis + + + + io.lettuce + lettuce-core + + + + org.apache.commons + commons-lang3 + 3.9 + + + + com.google.auto.value + auto-value-annotations + 1.6.6 + + + + com.google.auto.value + auto-value + 1.6.6 + provided + + + + org.mockito + mockito-core + 2.23.0 + test + + + + + com.github.kstyrc + embedded-redis + test + + + + org.apache.beam + beam-runners-direct-java + ${org.apache.beam.version} + test + + + + org.hamcrest + hamcrest-core + test + + + + org.hamcrest + hamcrest-library + test + + + + + junit + junit + 4.12 + test + + + + diff --git a/serving/src/main/java/feast/serving/encoding/FeatureRowDecoder.java b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever/FeatureRowDecoder.java similarity index 98% rename from serving/src/main/java/feast/serving/encoding/FeatureRowDecoder.java rename to storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever/FeatureRowDecoder.java index e70695d8c64..a5506028cbf 100644 --- a/serving/src/main/java/feast/serving/encoding/FeatureRowDecoder.java +++ b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever/FeatureRowDecoder.java @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.serving.encoding; +package feast.storage.connectors.redis.retriever; import feast.core.FeatureSetProto.FeatureSetSpec; import feast.core.FeatureSetProto.FeatureSpec; diff --git a/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever/RedisOnlineRetriever.java b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever/RedisOnlineRetriever.java new file mode 100644 index 00000000000..c8bb33de5fd --- /dev/null +++ b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever/RedisOnlineRetriever.java @@ -0,0 +1,204 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.redis.retriever; + +import com.google.protobuf.AbstractMessageLite; +import com.google.protobuf.InvalidProtocolBufferException; +import feast.core.FeatureSetProto.EntitySpec; +import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.serving.ServingAPIProto.FeatureReference; +import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest.EntityRow; +import feast.storage.RedisProto.RedisKey; +import feast.storage.api.retriever.FeatureSetRequest; +import feast.storage.api.retriever.OnlineRetriever; +import feast.types.FeatureRowProto.FeatureRow; +import feast.types.FieldProto.Field; +import feast.types.ValueProto.Value; +import io.grpc.Status; +import io.lettuce.core.api.StatefulRedisConnection; +import io.lettuce.core.api.sync.RedisCommands; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ExecutionException; +import java.util.stream.Collectors; + +public class RedisOnlineRetriever implements OnlineRetriever { + + private final RedisCommands syncCommands; + + public RedisOnlineRetriever(StatefulRedisConnection connection) { + this.syncCommands = connection.sync(); + } + + /** + * Gets online features from redis. This method returns a list of {@link FeatureRow}s + * corresponding to each feature set spec. Each feature row in the list then corresponds to an + * {@link EntityRow} provided by the user. + * + * @param entityRows list of entity rows in the feature request + * @param featureSetRequests Map of {@link feast.core.FeatureSetProto.FeatureSetSpec} to feature + * references in the request tied to that feature set. + * @return List of List of {@link FeatureRow} + */ + @Override + public List> getOnlineFeatures( + List entityRows, List featureSetRequests) { + + List> featureRows = new ArrayList<>(); + for (FeatureSetRequest featureSetRequest : featureSetRequests) { + List redisKeys = buildRedisKeys(entityRows, featureSetRequest.getSpec()); + try { + List featureRowsForFeatureSet = + sendAndProcessMultiGet( + redisKeys, + featureSetRequest.getSpec(), + featureSetRequest.getFeatureReferences().asList()); + featureRows.add(featureRowsForFeatureSet); + } catch (InvalidProtocolBufferException | ExecutionException e) { + throw Status.INTERNAL + .withDescription("Unable to parse protobuf while retrieving feature") + .withCause(e) + .asRuntimeException(); + } + } + return featureRows; + } + + private List buildRedisKeys(List entityRows, FeatureSetSpec featureSetSpec) { + String featureSetRef = generateFeatureSetStringRef(featureSetSpec); + List featureSetEntityNames = + featureSetSpec.getEntitiesList().stream() + .map(EntitySpec::getName) + .collect(Collectors.toList()); + List redisKeys = + entityRows.stream() + .map(row -> makeRedisKey(featureSetRef, featureSetEntityNames, row)) + .collect(Collectors.toList()); + return redisKeys; + } + + /** + * Create {@link RedisKey} + * + * @param featureSet featureSet reference of the feature. E.g. feature_set_1:1 + * @param featureSetEntityNames entity names that belong to the featureSet + * @param entityRow entityRow to build the key from + * @return {@link RedisKey} + */ + private RedisKey makeRedisKey( + String featureSet, List featureSetEntityNames, EntityRow entityRow) { + RedisKey.Builder builder = RedisKey.newBuilder().setFeatureSet(featureSet); + Map fieldsMap = entityRow.getFieldsMap(); + featureSetEntityNames.sort(String::compareTo); + for (int i = 0; i < featureSetEntityNames.size(); i++) { + String entityName = featureSetEntityNames.get(i); + + if (!fieldsMap.containsKey(entityName)) { + throw Status.INVALID_ARGUMENT + .withDescription( + String.format( + "Entity row fields \"%s\" does not contain required entity field \"%s\"", + fieldsMap.keySet().toString(), entityName)) + .asRuntimeException(); + } + + builder.addEntities( + Field.newBuilder().setName(entityName).setValue(fieldsMap.get(entityName))); + } + return builder.build(); + } + + private List sendAndProcessMultiGet( + List redisKeys, + FeatureSetSpec featureSetSpec, + List featureReferences) + throws InvalidProtocolBufferException, ExecutionException { + + List values = sendMultiGet(redisKeys); + List featureRows = new ArrayList<>(); + + FeatureRow.Builder nullFeatureRowBuilder = + FeatureRow.newBuilder().setFeatureSet(generateFeatureSetStringRef(featureSetSpec)); + for (FeatureReference featureReference : featureReferences) { + nullFeatureRowBuilder.addFields(Field.newBuilder().setName(featureReference.getName())); + } + + for (int i = 0; i < values.size(); i++) { + + byte[] value = values.get(i); + if (value == null) { + featureRows.add(nullFeatureRowBuilder.build()); + continue; + } + + FeatureRow featureRow = FeatureRow.parseFrom(value); + String featureSetRef = redisKeys.get(i).getFeatureSet(); + FeatureRowDecoder decoder = new FeatureRowDecoder(featureSetRef, featureSetSpec); + if (decoder.isEncoded(featureRow)) { + if (decoder.isEncodingValid(featureRow)) { + featureRow = decoder.decode(featureRow); + } else { + featureRows.add(nullFeatureRowBuilder.build()); + continue; + } + } + + featureRows.add(featureRow); + } + return featureRows; + } + + /** + * Send a list of get request as an mget + * + * @param keys list of {@link RedisKey} + * @return list of {@link FeatureRow} in primitive byte representation for each {@link RedisKey} + */ + private List sendMultiGet(List keys) { + try { + byte[][] binaryKeys = + keys.stream() + .map(AbstractMessageLite::toByteArray) + .collect(Collectors.toList()) + .toArray(new byte[0][0]); + return syncCommands.mget(binaryKeys).stream() + .map( + keyValue -> { + if (keyValue == null) { + return null; + } + return keyValue.getValueOrElse(null); + }) + .collect(Collectors.toList()); + } catch (Exception e) { + throw Status.NOT_FOUND + .withDescription("Unable to retrieve feature from Redis") + .withCause(e) + .asRuntimeException(); + } + } + + // TODO: Refactor this out to common package? + private static String generateFeatureSetStringRef(FeatureSetSpec featureSetSpec) { + String ref = String.format("%s/%s", featureSetSpec.getProject(), featureSetSpec.getName()); + if (featureSetSpec.getVersion() > 0) { + return ref + String.format(":%d", featureSetSpec.getVersion()); + } + return ref; + } +} diff --git a/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisCustomIO.java b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisCustomIO.java new file mode 100644 index 00000000000..cfe7771b324 --- /dev/null +++ b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisCustomIO.java @@ -0,0 +1,292 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2019 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.redis.writer; + +import feast.core.FeatureSetProto.EntitySpec; +import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.core.FeatureSetProto.FeatureSpec; +import feast.core.StoreProto.Store.RedisConfig; +import feast.storage.RedisProto.RedisKey; +import feast.storage.RedisProto.RedisKey.Builder; +import feast.storage.api.writer.FailedElement; +import feast.storage.api.writer.WriteResult; +import feast.storage.common.retry.Retriable; +import feast.types.FeatureRowProto.FeatureRow; +import feast.types.FieldProto.Field; +import feast.types.ValueProto; +import io.lettuce.core.RedisConnectionException; +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ExecutionException; +import java.util.stream.Collectors; +import org.apache.beam.sdk.transforms.DoFn; +import org.apache.beam.sdk.transforms.PTransform; +import org.apache.beam.sdk.transforms.ParDo; +import org.apache.beam.sdk.transforms.windowing.GlobalWindow; +import org.apache.beam.sdk.values.PCollection; +import org.apache.beam.sdk.values.PCollectionTuple; +import org.apache.beam.sdk.values.TupleTag; +import org.apache.beam.sdk.values.TupleTagList; +import org.apache.commons.lang3.exception.ExceptionUtils; +import org.joda.time.Instant; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class RedisCustomIO { + + private static final int DEFAULT_BATCH_SIZE = 1000; + private static final int DEFAULT_TIMEOUT = 2000; + + private static TupleTag successfulInsertsTag = new TupleTag<>("successfulInserts") {}; + private static TupleTag failedInsertsTupleTag = new TupleTag<>("failedInserts") {}; + + private static final Logger log = LoggerFactory.getLogger(RedisCustomIO.class); + + private RedisCustomIO() {} + + public static Write write(RedisConfig redisConfig, Map featureSetSpecs) { + return new Write(redisConfig, featureSetSpecs); + } + + /** ServingStoreWrite data to a Redis server. */ + public static class Write extends PTransform, WriteResult> { + + private Map featureSetSpecs; + private RedisConfig redisConfig; + private int batchSize; + private int timeout; + + public Write(RedisConfig redisConfig, Map featureSetSpecs) { + + this.redisConfig = redisConfig; + this.featureSetSpecs = featureSetSpecs; + } + + public Write withBatchSize(int batchSize) { + this.batchSize = batchSize; + return this; + } + + public Write withTimeout(int timeout) { + this.timeout = timeout; + return this; + } + + @Override + public WriteResult expand(PCollection input) { + PCollectionTuple redisWrite = + input.apply( + ParDo.of(new WriteDoFn(redisConfig, featureSetSpecs)) + .withOutputTags(successfulInsertsTag, TupleTagList.of(failedInsertsTupleTag))); + return WriteResult.in( + input.getPipeline(), + redisWrite.get(successfulInsertsTag), + redisWrite.get(failedInsertsTupleTag)); + } + + public static class WriteDoFn extends DoFn { + + private final List featureRows = new ArrayList<>(); + private Map featureSetSpecs; + private int batchSize = DEFAULT_BATCH_SIZE; + private int timeout = DEFAULT_TIMEOUT; + private RedisIngestionClient redisIngestionClient; + + WriteDoFn(RedisConfig config, Map featureSetSpecs) { + + this.redisIngestionClient = new RedisStandaloneIngestionClient(config); + this.featureSetSpecs = featureSetSpecs; + } + + public WriteDoFn withBatchSize(int batchSize) { + if (batchSize > 0) { + this.batchSize = batchSize; + } + return this; + } + + public WriteDoFn withTimeout(int timeout) { + if (timeout > 0) { + this.timeout = timeout; + } + return this; + } + + @Setup + public void setup() { + this.redisIngestionClient.setup(); + } + + @StartBundle + public void startBundle() { + try { + redisIngestionClient.connect(); + } catch (RedisConnectionException e) { + log.error("Connection to redis cannot be established ", e); + } + featureRows.clear(); + } + + private void executeBatch() throws Exception { + this.redisIngestionClient + .getBackOffExecutor() + .execute( + new Retriable() { + @Override + public void execute() throws ExecutionException, InterruptedException { + if (!redisIngestionClient.isConnected()) { + redisIngestionClient.connect(); + } + featureRows.forEach( + row -> { + redisIngestionClient.set(getKey(row), getValue(row)); + }); + redisIngestionClient.sync(); + } + + @Override + public Boolean isExceptionRetriable(Exception e) { + return e instanceof RedisConnectionException; + } + + @Override + public void cleanUpAfterFailure() {} + }); + } + + private FailedElement toFailedElement( + FeatureRow featureRow, Exception exception, String jobName) { + return FailedElement.newBuilder() + .setJobName(jobName) + .setTransformName("RedisCustomIO") + .setPayload(featureRow.toString()) + .setErrorMessage(exception.getMessage()) + .setStackTrace(ExceptionUtils.getStackTrace(exception)) + .build(); + } + + private byte[] getKey(FeatureRow featureRow) { + FeatureSetSpec featureSetSpec = featureSetSpecs.get(featureRow.getFeatureSet()); + List entityNames = + featureSetSpec.getEntitiesList().stream() + .map(EntitySpec::getName) + .sorted() + .collect(Collectors.toList()); + + Map entityFields = new HashMap<>(); + Builder redisKeyBuilder = RedisKey.newBuilder().setFeatureSet(featureRow.getFeatureSet()); + for (Field field : featureRow.getFieldsList()) { + if (entityNames.contains(field.getName())) { + entityFields.putIfAbsent( + field.getName(), + Field.newBuilder().setName(field.getName()).setValue(field.getValue()).build()); + } + } + for (String entityName : entityNames) { + redisKeyBuilder.addEntities(entityFields.get(entityName)); + } + return redisKeyBuilder.build().toByteArray(); + } + + private byte[] getValue(FeatureRow featureRow) { + FeatureSetSpec spec = featureSetSpecs.get(featureRow.getFeatureSet()); + + List featureNames = + spec.getFeaturesList().stream().map(FeatureSpec::getName).collect(Collectors.toList()); + Map fieldValueOnlyMap = + featureRow.getFieldsList().stream() + .filter(field -> featureNames.contains(field.getName())) + .distinct() + .collect( + Collectors.toMap( + Field::getName, + field -> Field.newBuilder().setValue(field.getValue()).build())); + + List values = + featureNames.stream() + .sorted() + .map( + featureName -> + fieldValueOnlyMap.getOrDefault( + featureName, + Field.newBuilder() + .setValue(ValueProto.Value.getDefaultInstance()) + .build())) + .collect(Collectors.toList()); + + return FeatureRow.newBuilder() + .setEventTimestamp(featureRow.getEventTimestamp()) + .addAllFields(values) + .build() + .toByteArray(); + } + + @ProcessElement + public void processElement(ProcessContext context) { + FeatureRow featureRow = context.element(); + featureRows.add(featureRow); + if (featureRows.size() >= batchSize) { + try { + executeBatch(); + featureRows.forEach(row -> context.output(successfulInsertsTag, row)); + featureRows.clear(); + } catch (Exception e) { + featureRows.forEach( + failedMutation -> { + FailedElement failedElement = + toFailedElement(failedMutation, e, context.getPipelineOptions().getJobName()); + context.output(failedInsertsTupleTag, failedElement); + }); + featureRows.clear(); + } + } + } + + @FinishBundle + public void finishBundle(FinishBundleContext context) + throws IOException, InterruptedException { + if (featureRows.size() > 0) { + try { + executeBatch(); + featureRows.forEach( + row -> + context.output( + successfulInsertsTag, row, Instant.now(), GlobalWindow.INSTANCE)); + featureRows.clear(); + } catch (Exception e) { + featureRows.forEach( + failedMutation -> { + FailedElement failedElement = + toFailedElement(failedMutation, e, context.getPipelineOptions().getJobName()); + context.output( + failedInsertsTupleTag, failedElement, Instant.now(), GlobalWindow.INSTANCE); + }); + featureRows.clear(); + } + } + } + + @Teardown + public void teardown() { + redisIngestionClient.shutdown(); + } + } + } +} diff --git a/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisFeatureSink.java b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisFeatureSink.java new file mode 100644 index 00000000000..63c8c68d9bb --- /dev/null +++ b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisFeatureSink.java @@ -0,0 +1,74 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.redis.writer; + +import com.google.auto.value.AutoValue; +import feast.core.FeatureSetProto.FeatureSet; +import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.core.StoreProto.Store.RedisConfig; +import feast.storage.api.writer.FeatureSink; +import feast.storage.api.writer.WriteResult; +import feast.types.FeatureRowProto.FeatureRow; +import io.lettuce.core.RedisClient; +import io.lettuce.core.RedisConnectionException; +import io.lettuce.core.RedisURI; +import java.util.Map; +import org.apache.beam.sdk.transforms.PTransform; +import org.apache.beam.sdk.values.PCollection; + +@AutoValue +public abstract class RedisFeatureSink implements FeatureSink { + + public abstract RedisConfig getRedisConfig(); + + public abstract Map getFeatureSetSpecs(); + + public abstract Builder toBuilder(); + + public static Builder builder() { + return new AutoValue_RedisFeatureSink.Builder(); + } + + @AutoValue.Builder + public abstract static class Builder { + public abstract Builder setRedisConfig(RedisConfig redisConfig); + + public abstract Builder setFeatureSetSpecs(Map featureSetSpecs); + + public abstract RedisFeatureSink build(); + } + + @Override + public void prepareWrite(FeatureSet featureSet) { + RedisClient redisClient = + RedisClient.create(RedisURI.create(getRedisConfig().getHost(), getRedisConfig().getPort())); + try { + redisClient.connect(); + } catch (RedisConnectionException e) { + throw new RuntimeException( + String.format( + "Failed to connect to Redis at host: '%s' port: '%d'. Please check that your Redis is running and accessible from Feast.", + getRedisConfig().getHost(), getRedisConfig().getPort())); + } + redisClient.shutdown(); + } + + @Override + public PTransform, WriteResult> writer() { + return new RedisCustomIO.Write(getRedisConfig(), getFeatureSetSpecs()); + } +} diff --git a/ingestion/src/main/java/feast/store/serving/redis/RedisIngestionClient.java b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisIngestionClient.java similarity index 92% rename from ingestion/src/main/java/feast/store/serving/redis/RedisIngestionClient.java rename to storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisIngestionClient.java index d51eead53fb..6616a79aaca 100644 --- a/ingestion/src/main/java/feast/store/serving/redis/RedisIngestionClient.java +++ b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisIngestionClient.java @@ -14,9 +14,9 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.store.serving.redis; +package feast.storage.connectors.redis.writer; -import feast.retry.BackOffExecutor; +import feast.storage.common.retry.BackOffExecutor; import java.io.Serializable; public interface RedisIngestionClient extends Serializable { diff --git a/ingestion/src/main/java/feast/store/serving/redis/RedisStandaloneIngestionClient.java b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisStandaloneIngestionClient.java similarity index 93% rename from ingestion/src/main/java/feast/store/serving/redis/RedisStandaloneIngestionClient.java rename to storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisStandaloneIngestionClient.java index d95ebbbf64a..95bd7ad1516 100644 --- a/ingestion/src/main/java/feast/store/serving/redis/RedisStandaloneIngestionClient.java +++ b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisStandaloneIngestionClient.java @@ -14,12 +14,15 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.store.serving.redis; +package feast.storage.connectors.redis.writer; import com.google.common.collect.Lists; import feast.core.StoreProto; -import feast.retry.BackOffExecutor; -import io.lettuce.core.*; +import feast.storage.common.retry.BackOffExecutor; +import io.lettuce.core.LettuceFutures; +import io.lettuce.core.RedisClient; +import io.lettuce.core.RedisFuture; +import io.lettuce.core.RedisURI; import io.lettuce.core.api.StatefulRedisConnection; import io.lettuce.core.api.async.RedisAsyncCommands; import io.lettuce.core.codec.ByteArrayCodec; diff --git a/serving/src/test/java/feast/serving/encoding/FeatureRowDecoderTest.java b/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/retriever/FeatureRowDecoderTest.java similarity index 98% rename from serving/src/test/java/feast/serving/encoding/FeatureRowDecoderTest.java rename to storage/connectors/redis/src/test/java/feast/storage/connectors/redis/retriever/FeatureRowDecoderTest.java index 8f6c79ad66c..0f37e68941c 100644 --- a/serving/src/test/java/feast/serving/encoding/FeatureRowDecoderTest.java +++ b/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/retriever/FeatureRowDecoderTest.java @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.serving.encoding; +package feast.storage.connectors.redis.retriever; import static org.junit.Assert.*; diff --git a/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/retriever/RedisOnlineRetrieverTest.java b/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/retriever/RedisOnlineRetrieverTest.java new file mode 100644 index 00000000000..11c216c5a0c --- /dev/null +++ b/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/retriever/RedisOnlineRetrieverTest.java @@ -0,0 +1,262 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.redis.retriever; + +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.equalTo; +import static org.mockito.Mockito.when; +import static org.mockito.MockitoAnnotations.initMocks; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.Lists; +import com.google.protobuf.AbstractMessageLite; +import com.google.protobuf.Duration; +import com.google.protobuf.Timestamp; +import feast.core.FeatureSetProto.EntitySpec; +import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.core.FeatureSetProto.FeatureSpec; +import feast.serving.ServingAPIProto.FeatureReference; +import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest.EntityRow; +import feast.storage.RedisProto.RedisKey; +import feast.storage.api.retriever.FeatureSetRequest; +import feast.types.FeatureRowProto.FeatureRow; +import feast.types.FieldProto.Field; +import feast.types.ValueProto.Value; +import io.lettuce.core.KeyValue; +import io.lettuce.core.api.StatefulRedisConnection; +import io.lettuce.core.api.sync.RedisCommands; +import java.util.List; +import java.util.Optional; +import java.util.stream.Collectors; +import org.junit.Before; +import org.junit.Test; +import org.mockito.Mock; + +public class RedisOnlineRetrieverTest { + + @Mock StatefulRedisConnection connection; + + @Mock RedisCommands syncCommands; + + private RedisOnlineRetriever redisOnlineRetriever; + private byte[][] redisKeyList; + + @Before + public void setUp() { + initMocks(this); + when(connection.sync()).thenReturn(syncCommands); + redisOnlineRetriever = new RedisOnlineRetriever(connection); + redisKeyList = + Lists.newArrayList( + RedisKey.newBuilder() + .setFeatureSet("project/featureSet:1") + .addAllEntities( + Lists.newArrayList( + Field.newBuilder().setName("entity1").setValue(intValue(1)).build(), + Field.newBuilder().setName("entity2").setValue(strValue("a")).build())) + .build(), + RedisKey.newBuilder() + .setFeatureSet("project/featureSet:1") + .addAllEntities( + Lists.newArrayList( + Field.newBuilder().setName("entity1").setValue(intValue(2)).build(), + Field.newBuilder().setName("entity2").setValue(strValue("b")).build())) + .build()) + .stream() + .map(AbstractMessageLite::toByteArray) + .collect(Collectors.toList()) + .toArray(new byte[0][0]); + } + + @Test + public void shouldReturnResponseWithValuesIfKeysPresent() { + FeatureSetRequest featureSetRequest = + FeatureSetRequest.newBuilder() + .setSpec(getFeatureSetSpec()) + .addFeatureReference( + FeatureReference.newBuilder() + .setName("feature1") + .setVersion(1) + .setProject("project") + .build()) + .addFeatureReference( + FeatureReference.newBuilder() + .setName("feature2") + .setVersion(1) + .setProject("project") + .build()) + .build(); + List entityRows = + ImmutableList.of( + EntityRow.newBuilder() + .setEntityTimestamp(Timestamp.newBuilder().setSeconds(100)) + .putFields("entity1", intValue(1)) + .putFields("entity2", strValue("a")) + .build(), + EntityRow.newBuilder() + .setEntityTimestamp(Timestamp.newBuilder().setSeconds(100)) + .putFields("entity1", intValue(2)) + .putFields("entity2", strValue("b")) + .build()); + + List featureRows = + Lists.newArrayList( + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setValue(intValue(1)).build(), + Field.newBuilder().setValue(intValue(1)).build())) + .build(), + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setValue(intValue(2)).build(), + Field.newBuilder().setValue(intValue(2)).build())) + .build()); + + List> featureRowBytes = + featureRows.stream() + .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) + .collect(Collectors.toList()); + + redisOnlineRetriever = new RedisOnlineRetriever(connection); + when(connection.sync()).thenReturn(syncCommands); + when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); + + List> expected = + List.of( + Lists.newArrayList( + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) + .setFeatureSet("project/featureSet:1") + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), + Field.newBuilder().setName("feature2").setValue(intValue(1)).build())) + .build(), + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) + .setFeatureSet("project/featureSet:1") + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setName("feature1").setValue(intValue(2)).build(), + Field.newBuilder().setName("feature2").setValue(intValue(2)).build())) + .build())); + + List> actual = + redisOnlineRetriever.getOnlineFeatures(entityRows, List.of(featureSetRequest)); + assertThat(actual, equalTo(expected)); + } + + @Test + public void shouldReturnResponseWithUnsetValuesIfKeysNotPresent() { + FeatureSetRequest featureSetRequest = + FeatureSetRequest.newBuilder() + .setSpec(getFeatureSetSpec()) + .addFeatureReference( + FeatureReference.newBuilder() + .setName("feature1") + .setVersion(1) + .setProject("project") + .build()) + .addFeatureReference( + FeatureReference.newBuilder() + .setName("feature2") + .setVersion(1) + .setProject("project") + .build()) + .build(); + List entityRows = + ImmutableList.of( + EntityRow.newBuilder() + .setEntityTimestamp(Timestamp.newBuilder().setSeconds(100)) + .putFields("entity1", intValue(1)) + .putFields("entity2", strValue("a")) + .build(), + EntityRow.newBuilder() + .setEntityTimestamp(Timestamp.newBuilder().setSeconds(100)) + .putFields("entity1", intValue(2)) + .putFields("entity2", strValue("b")) + .build()); + + List featureRows = + Lists.newArrayList( + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setValue(intValue(1)).build(), + Field.newBuilder().setValue(intValue(1)).build())) + .build()); + + List> featureRowBytes = + featureRows.stream() + .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) + .collect(Collectors.toList()); + featureRowBytes.add(null); + + redisOnlineRetriever = new RedisOnlineRetriever(connection); + when(connection.sync()).thenReturn(syncCommands); + when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); + + List> expected = + List.of( + Lists.newArrayList( + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) + .setFeatureSet("project/featureSet:1") + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), + Field.newBuilder().setName("feature2").setValue(intValue(1)).build())) + .build(), + FeatureRow.newBuilder() + .setFeatureSet("project/featureSet:1") + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setName("feature1").build(), + Field.newBuilder().setName("feature2").build())) + .build())); + + List> actual = + redisOnlineRetriever.getOnlineFeatures(entityRows, List.of(featureSetRequest)); + assertThat(actual, equalTo(expected)); + } + + private Value intValue(int val) { + return Value.newBuilder().setInt64Val(val).build(); + } + + private Value strValue(String val) { + return Value.newBuilder().setStringVal(val).build(); + } + + private FeatureSetSpec getFeatureSetSpec() { + return FeatureSetSpec.newBuilder() + .setProject("project") + .setName("featureSet") + .setVersion(1) + .addEntities(EntitySpec.newBuilder().setName("entity1")) + .addEntities(EntitySpec.newBuilder().setName("entity2")) + .addFeatures(FeatureSpec.newBuilder().setName("feature1")) + .addFeatures(FeatureSpec.newBuilder().setName("feature2")) + .setMaxAge(Duration.newBuilder().setSeconds(30)) // default + .build(); + } +} diff --git a/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/test/TestUtil.java b/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/test/TestUtil.java new file mode 100644 index 00000000000..66aba44bc20 --- /dev/null +++ b/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/test/TestUtil.java @@ -0,0 +1,44 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.redis.test; + +import java.io.IOException; +import redis.embedded.RedisServer; + +public class TestUtil { + public static class LocalRedis { + + private static RedisServer server; + + /** + * Start local Redis for used in testing at "localhost" + * + * @param port port number + * @throws IOException if Redis failed to start + */ + public static void start(int port) throws IOException { + server = new RedisServer(port); + server.start(); + } + + public static void stop() { + if (server != null) { + server.stop(); + } + } + } +} diff --git a/ingestion/src/test/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFnTest.java b/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/writer/RedisFeatureSinkTest.java similarity index 50% rename from ingestion/src/test/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFnTest.java rename to storage/connectors/redis/src/test/java/feast/storage/connectors/redis/writer/RedisFeatureSinkTest.java index 7db5e28ecb8..beeabc2c884 100644 --- a/ingestion/src/test/java/feast/store/serving/redis/FeatureRowToRedisMutationDoFnTest.java +++ b/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/writer/RedisFeatureSinkTest.java @@ -1,6 +1,6 @@ /* * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2020 The Feast Authors + * Copyright 2018-2019 The Feast Authors * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. @@ -14,142 +14,259 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.store.serving.redis; +package feast.storage.connectors.redis.writer; -import static org.junit.Assert.*; +import static feast.storage.common.testing.TestUtil.field; +import static org.hamcrest.CoreMatchers.equalTo; +import static org.hamcrest.MatcherAssert.assertThat; +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; import com.google.protobuf.Timestamp; -import feast.core.FeatureSetProto; import feast.core.FeatureSetProto.EntitySpec; import feast.core.FeatureSetProto.FeatureSetSpec; import feast.core.FeatureSetProto.FeatureSpec; +import feast.core.StoreProto; +import feast.core.StoreProto.Store.RedisConfig; import feast.storage.RedisProto.RedisKey; -import feast.store.serving.redis.RedisCustomIO.RedisMutation; import feast.types.FeatureRowProto.FeatureRow; import feast.types.FieldProto.Field; import feast.types.ValueProto.Value; import feast.types.ValueProto.ValueType.Enum; +import io.lettuce.core.RedisClient; +import io.lettuce.core.RedisURI; +import io.lettuce.core.api.StatefulRedisConnection; +import io.lettuce.core.api.sync.RedisStringCommands; +import io.lettuce.core.codec.ByteArrayCodec; +import java.io.IOException; import java.util.*; -import org.apache.beam.sdk.extensions.protobuf.ProtoCoder; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.ScheduledThreadPoolExecutor; +import java.util.concurrent.TimeUnit; import org.apache.beam.sdk.testing.PAssert; import org.apache.beam.sdk.testing.TestPipeline; +import org.apache.beam.sdk.transforms.Count; import org.apache.beam.sdk.transforms.Create; -import org.apache.beam.sdk.transforms.ParDo; -import org.apache.beam.sdk.transforms.SerializableFunction; import org.apache.beam.sdk.values.PCollection; +import org.junit.After; +import org.junit.Before; import org.junit.Rule; import org.junit.Test; +import redis.embedded.Redis; +import redis.embedded.RedisServer; -public class FeatureRowToRedisMutationDoFnTest { - +public class RedisFeatureSinkTest { @Rule public transient TestPipeline p = TestPipeline.create(); - private FeatureSetProto.FeatureSet fs = - FeatureSetProto.FeatureSet.newBuilder() - .setSpec( - FeatureSetSpec.newBuilder() - .setName("feature_set") - .setVersion(1) - .addEntities( - EntitySpec.newBuilder() - .setName("entity_id_primary") - .setValueType(Enum.INT32) - .build()) - .addEntities( - EntitySpec.newBuilder() - .setName("entity_id_secondary") - .setValueType(Enum.STRING) - .build()) - .addFeatures( - FeatureSpec.newBuilder() - .setName("feature_1") - .setValueType(Enum.STRING) - .build()) - .addFeatures( - FeatureSpec.newBuilder() - .setName("feature_2") - .setValueType(Enum.INT64) - .build())) - .build(); - - @Test - public void shouldConvertRowWithDuplicateEntitiesToValidKey() { - Map featureSets = new HashMap<>(); - featureSets.put("feature_set", fs); - - FeatureRow offendingRow = - FeatureRow.newBuilder() - .setFeatureSet("feature_set") - .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) - .addFields( - Field.newBuilder() - .setName("entity_id_primary") - .setValue(Value.newBuilder().setInt32Val(1))) - .addFields( - Field.newBuilder() - .setName("entity_id_primary") - .setValue(Value.newBuilder().setInt32Val(2))) - .addFields( - Field.newBuilder() - .setName("entity_id_secondary") - .setValue(Value.newBuilder().setStringVal("a"))) - .addFields( - Field.newBuilder() - .setName("feature_1") - .setValue(Value.newBuilder().setStringVal("strValue1"))) - .addFields( - Field.newBuilder() - .setName("feature_2") - .setValue(Value.newBuilder().setInt64Val(1001))) + private static String REDIS_HOST = "localhost"; + private static int REDIS_PORT = 51234; + private Redis redis; + private RedisClient redisClient; + private RedisStringCommands sync; + + private RedisFeatureSink redisFeatureSink; + + @Before + public void setUp() throws IOException { + redis = new RedisServer(REDIS_PORT); + redis.start(); + redisClient = + RedisClient.create(new RedisURI(REDIS_HOST, REDIS_PORT, java.time.Duration.ofMillis(2000))); + StatefulRedisConnection connection = redisClient.connect(new ByteArrayCodec()); + sync = connection.sync(); + + FeatureSetSpec spec1 = + FeatureSetSpec.newBuilder() + .setName("fs") + .setVersion(1) + .setProject("myproject") + .addEntities(EntitySpec.newBuilder().setName("entity").setValueType(Enum.INT64).build()) + .addFeatures( + FeatureSpec.newBuilder().setName("feature").setValueType(Enum.STRING).build()) .build(); - PCollection output = - p.apply(Create.of(Collections.singletonList(offendingRow))) - .setCoder(ProtoCoder.of(FeatureRow.class)) - .apply(ParDo.of(new FeatureRowToRedisMutationDoFn(featureSets))); - - RedisKey expectedKey = - RedisKey.newBuilder() - .setFeatureSet("feature_set") + FeatureSetSpec spec2 = + FeatureSetSpec.newBuilder() + .setName("feature_set") + .setProject("myproject") + .setVersion(1) .addEntities( - Field.newBuilder() + EntitySpec.newBuilder() .setName("entity_id_primary") - .setValue(Value.newBuilder().setInt32Val(1))) + .setValueType(Enum.INT32) + .build()) .addEntities( - Field.newBuilder() + EntitySpec.newBuilder() .setName("entity_id_secondary") - .setValue(Value.newBuilder().setStringVal("a"))) + .setValueType(Enum.STRING) + .build()) + .addFeatures( + FeatureSpec.newBuilder().setName("feature_1").setValueType(Enum.STRING).build()) + .addFeatures( + FeatureSpec.newBuilder().setName("feature_2").setValueType(Enum.INT64).build()) .build(); - FeatureRow expectedValue = + Map specMap = + ImmutableMap.of("myproject/fs:1", spec1, "myproject/feature_set:1", spec2); + StoreProto.Store.RedisConfig redisConfig = + StoreProto.Store.RedisConfig.newBuilder().setHost(REDIS_HOST).setPort(REDIS_PORT).build(); + + redisFeatureSink = + RedisFeatureSink.builder().setFeatureSetSpecs(specMap).setRedisConfig(redisConfig).build(); + } + + @After + public void teardown() { + redisClient.shutdown(); + redis.stop(); + } + + @Test + public void shouldWriteToRedis() { + + HashMap kvs = new LinkedHashMap<>(); + kvs.put( + RedisKey.newBuilder() + .setFeatureSet("myproject/fs:1") + .addEntities(field("entity", 1, Enum.INT64)) + .build(), FeatureRow.newBuilder() - .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) - .addFields(Field.newBuilder().setValue(Value.newBuilder().setStringVal("strValue1"))) - .addFields(Field.newBuilder().setValue(Value.newBuilder().setInt64Val(1001))) + .setEventTimestamp(Timestamp.getDefaultInstance()) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setStringVal("one"))) + .build()); + kvs.put( + RedisKey.newBuilder() + .setFeatureSet("myproject/fs:1") + .addEntities(field("entity", 2, Enum.INT64)) + .build(), + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.getDefaultInstance()) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setStringVal("two"))) + .build()); + + List featureRows = + ImmutableList.of( + FeatureRow.newBuilder() + .setFeatureSet("myproject/fs:1") + .addFields(field("entity", 1, Enum.INT64)) + .addFields(field("feature", "one", Enum.STRING)) + .build(), + FeatureRow.newBuilder() + .setFeatureSet("myproject/fs:1") + .addFields(field("entity", 2, Enum.INT64)) + .addFields(field("feature", "two", Enum.STRING)) + .build()); + + p.apply(Create.of(featureRows)).apply(redisFeatureSink.writer()); + p.run(); + + kvs.forEach( + (key, value) -> { + byte[] actual = sync.get(key.toByteArray()); + assertThat(actual, equalTo(value.toByteArray())); + }); + } + + @Test(timeout = 10000) + public void shouldRetryFailConnection() throws InterruptedException { + RedisConfig redisConfig = + RedisConfig.newBuilder() + .setHost(REDIS_HOST) + .setPort(REDIS_PORT) + .setMaxRetries(4) + .setInitialBackoffMs(2000) .build(); + redisFeatureSink = redisFeatureSink.toBuilder().setRedisConfig(redisConfig).build(); + + HashMap kvs = new LinkedHashMap<>(); + kvs.put( + RedisKey.newBuilder() + .setFeatureSet("myproject/fs:1") + .addEntities(field("entity", 1, Enum.INT64)) + .build(), + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.getDefaultInstance()) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setStringVal("one"))) + .build()); + + List featureRows = + ImmutableList.of( + FeatureRow.newBuilder() + .setFeatureSet("myproject/fs:1") + .addFields(field("entity", 1, Enum.INT64)) + .addFields(field("feature", "one", Enum.STRING)) + .build()); + + PCollection failedElementCount = + p.apply(Create.of(featureRows)) + .apply(redisFeatureSink.writer()) + .getFailedInserts() + .apply(Count.globally()); + + redis.stop(); + final ScheduledThreadPoolExecutor redisRestartExecutor = new ScheduledThreadPoolExecutor(1); + ScheduledFuture scheduledRedisRestart = + redisRestartExecutor.schedule( + () -> { + redis.start(); + }, + 3, + TimeUnit.SECONDS); + + PAssert.that(failedElementCount).containsInAnyOrder(0L); + p.run(); + scheduledRedisRestart.cancel(true); + + kvs.forEach( + (key, value) -> { + byte[] actual = sync.get(key.toByteArray()); + assertThat(actual, equalTo(value.toByteArray())); + }); + } + + @Test + public void shouldProduceFailedElementIfRetryExceeded() { - PAssert.that(output) - .satisfies( - (SerializableFunction, Void>) - input -> { - input.forEach( - rm -> { - assert (Arrays.equals(rm.getKey(), expectedKey.toByteArray())); - assert (Arrays.equals(rm.getValue(), expectedValue.toByteArray())); - }); - return null; - }); + RedisConfig redisConfig = + RedisConfig.newBuilder().setHost(REDIS_HOST).setPort(REDIS_PORT + 1).build(); + redisFeatureSink = redisFeatureSink.toBuilder().setRedisConfig(redisConfig).build(); + + HashMap kvs = new LinkedHashMap<>(); + kvs.put( + RedisKey.newBuilder() + .setFeatureSet("myproject/fs:1") + .addEntities(field("entity", 1, Enum.INT64)) + .build(), + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.getDefaultInstance()) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setStringVal("one"))) + .build()); + + List featureRows = + ImmutableList.of( + FeatureRow.newBuilder() + .setFeatureSet("myproject/fs:1") + .addFields(field("entity", 1, Enum.INT64)) + .addFields(field("feature", "one", Enum.STRING)) + .build()); + + PCollection failedElementCount = + p.apply(Create.of(featureRows)) + .apply(redisFeatureSink.writer()) + .getFailedInserts() + .apply(Count.globally()); + + redis.stop(); + PAssert.that(failedElementCount).containsInAnyOrder(1L); p.run(); } @Test - public void shouldConvertRowWithExtraEntitiesToValidKey() { - Map featureSets = new HashMap<>(); - featureSets.put("feature_set", fs); + public void shouldConvertRowWithDuplicateEntitiesToValidKey() { FeatureRow offendingRow = FeatureRow.newBuilder() - .setFeatureSet("feature_set") + .setFeatureSet("myproject/feature_set:1") .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) .addFields( Field.newBuilder() @@ -157,7 +274,7 @@ public void shouldConvertRowWithExtraEntitiesToValidKey() { .setValue(Value.newBuilder().setInt32Val(1))) .addFields( Field.newBuilder() - .setName("entity_id_invalid") + .setName("entity_id_primary") .setValue(Value.newBuilder().setInt32Val(2))) .addFields( Field.newBuilder() @@ -173,14 +290,9 @@ public void shouldConvertRowWithExtraEntitiesToValidKey() { .setValue(Value.newBuilder().setInt64Val(1001))) .build(); - PCollection output = - p.apply(Create.of(Collections.singletonList(offendingRow))) - .setCoder(ProtoCoder.of(FeatureRow.class)) - .apply(ParDo.of(new FeatureRowToRedisMutationDoFn(featureSets))); - RedisKey expectedKey = RedisKey.newBuilder() - .setFeatureSet("feature_set") + .setFeatureSet("myproject/feature_set:1") .addEntities( Field.newBuilder() .setName("entity_id_primary") @@ -198,28 +310,19 @@ public void shouldConvertRowWithExtraEntitiesToValidKey() { .addFields(Field.newBuilder().setValue(Value.newBuilder().setInt64Val(1001))) .build(); - PAssert.that(output) - .satisfies( - (SerializableFunction, Void>) - input -> { - input.forEach( - rm -> { - assert (Arrays.equals(rm.getKey(), expectedKey.toByteArray())); - assert (Arrays.equals(rm.getValue(), expectedValue.toByteArray())); - }); - return null; - }); + p.apply(Create.of(offendingRow)).apply(redisFeatureSink.writer()); + p.run(); + + byte[] actual = sync.get(expectedKey.toByteArray()); + assertThat(actual, equalTo(expectedValue.toByteArray())); } @Test public void shouldConvertRowWithOutOfOrderFieldsToValidKey() { - Map featureSets = new HashMap<>(); - featureSets.put("feature_set", fs); - FeatureRow offendingRow = FeatureRow.newBuilder() - .setFeatureSet("feature_set") + .setFeatureSet("myproject/feature_set:1") .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) .addFields( Field.newBuilder() @@ -239,14 +342,9 @@ public void shouldConvertRowWithOutOfOrderFieldsToValidKey() { .setValue(Value.newBuilder().setStringVal("strValue1"))) .build(); - PCollection output = - p.apply(Create.of(Collections.singletonList(offendingRow))) - .setCoder(ProtoCoder.of(FeatureRow.class)) - .apply(ParDo.of(new FeatureRowToRedisMutationDoFn(featureSets))); - RedisKey expectedKey = RedisKey.newBuilder() - .setFeatureSet("feature_set") + .setFeatureSet("myproject/feature_set:1") .addEntities( Field.newBuilder() .setName("entity_id_primary") @@ -267,28 +365,19 @@ public void shouldConvertRowWithOutOfOrderFieldsToValidKey() { .addAllFields(expectedFields) .build(); - PAssert.that(output) - .satisfies( - (SerializableFunction, Void>) - input -> { - input.forEach( - rm -> { - assert (Arrays.equals(rm.getKey(), expectedKey.toByteArray())); - assert (Arrays.equals(rm.getValue(), expectedValue.toByteArray())); - }); - return null; - }); + p.apply(Create.of(offendingRow)).apply(redisFeatureSink.writer()); + p.run(); + + byte[] actual = sync.get(expectedKey.toByteArray()); + assertThat(actual, equalTo(expectedValue.toByteArray())); } @Test public void shouldMergeDuplicateFeatureFields() { - Map featureSets = new HashMap<>(); - featureSets.put("feature_set", fs); - FeatureRow featureRowWithDuplicatedFeatureFields = FeatureRow.newBuilder() - .setFeatureSet("feature_set") + .setFeatureSet("myproject/feature_set:1") .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) .addFields( Field.newBuilder() @@ -312,14 +401,9 @@ public void shouldMergeDuplicateFeatureFields() { .setValue(Value.newBuilder().setInt64Val(1001))) .build(); - PCollection output = - p.apply(Create.of(Collections.singletonList(featureRowWithDuplicatedFeatureFields))) - .setCoder(ProtoCoder.of(FeatureRow.class)) - .apply(ParDo.of(new FeatureRowToRedisMutationDoFn(featureSets))); - RedisKey expectedKey = RedisKey.newBuilder() - .setFeatureSet("feature_set") + .setFeatureSet("myproject/feature_set:1") .addEntities( Field.newBuilder() .setName("entity_id_primary") @@ -337,28 +421,19 @@ public void shouldMergeDuplicateFeatureFields() { .addFields(Field.newBuilder().setValue(Value.newBuilder().setInt64Val(1001))) .build(); - PAssert.that(output) - .satisfies( - (SerializableFunction, Void>) - input -> { - input.forEach( - rm -> { - assert (Arrays.equals(rm.getKey(), expectedKey.toByteArray())); - assert (Arrays.equals(rm.getValue(), expectedValue.toByteArray())); - }); - return null; - }); + p.apply(Create.of(featureRowWithDuplicatedFeatureFields)).apply(redisFeatureSink.writer()); + p.run(); + + byte[] actual = sync.get(expectedKey.toByteArray()); + assertThat(actual, equalTo(expectedValue.toByteArray())); } @Test public void shouldPopulateMissingFeatureValuesWithDefaultInstance() { - Map featureSets = new HashMap<>(); - featureSets.put("feature_set", fs); - FeatureRow featureRowWithDuplicatedFeatureFields = FeatureRow.newBuilder() - .setFeatureSet("feature_set") + .setFeatureSet("myproject/feature_set:1") .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) .addFields( Field.newBuilder() @@ -374,14 +449,9 @@ public void shouldPopulateMissingFeatureValuesWithDefaultInstance() { .setValue(Value.newBuilder().setStringVal("strValue1"))) .build(); - PCollection output = - p.apply(Create.of(Collections.singletonList(featureRowWithDuplicatedFeatureFields))) - .setCoder(ProtoCoder.of(FeatureRow.class)) - .apply(ParDo.of(new FeatureRowToRedisMutationDoFn(featureSets))); - RedisKey expectedKey = RedisKey.newBuilder() - .setFeatureSet("feature_set") + .setFeatureSet("myproject/feature_set:1") .addEntities( Field.newBuilder() .setName("entity_id_primary") @@ -399,17 +469,11 @@ public void shouldPopulateMissingFeatureValuesWithDefaultInstance() { .addFields(Field.newBuilder().setValue(Value.getDefaultInstance())) .build(); - PAssert.that(output) - .satisfies( - (SerializableFunction, Void>) - input -> { - input.forEach( - rm -> { - assert (Arrays.equals(rm.getKey(), expectedKey.toByteArray())); - assert (Arrays.equals(rm.getValue(), expectedValue.toByteArray())); - }); - return null; - }); + p.apply(Create.of(featureRowWithDuplicatedFeatureFields)).apply(redisFeatureSink.writer()); + p.run(); + + byte[] actual = sync.get(expectedKey.toByteArray()); + assertThat(actual, equalTo(expectedValue.toByteArray())); } } From e7482afcae03f8c4d0c3672d2ceb1a642d748d0b Mon Sep 17 00:00:00 2001 From: David Heryanto Date: Fri, 10 Apr 2020 17:31:46 +0800 Subject: [PATCH 112/176] Update Python SDK so FeatureSet can import Schema from Tensorflow metadata (#450) * Add skeleton for update/get schema in FeatureSet * Add update_schema method to FeatureSet - Update Field, Feature and Entity class with fields from presence_constraints, shape_type and domain_info * Update error message when domain ref is missing from top level schema * Add more assertion in test_update_schema before updating schema * Fix conflicting versions in package requirements * Add export_schema method to export schema from FeatureSet * Add exporting of Tensorflow metadata schema from FeatureSet. - Update documentation for properties in Field - Deduplication refactoring in FeatureSet * Remove changes to mypy generated codes * Revert changes to packages version in requirements-ci and setup.py They are not necessary for now and to avoid unexpected breaking changes. * Remove 'schema' param in 'from_proto' method in Entity and Feature. In import_tfx_schema method, the domain info is first made inline so there is no need to have schema level domain info when updating Feast Entity and Feature. Also added documentation to setter property methods in Field.py * Fix rebase errors, apply black * Remove unnecessary imports Co-authored-by: zhilingc --- sdk/python/feast/entity.py | 27 +- sdk/python/feast/feature.py | 38 +- sdk/python/feast/feature_set.py | 129 +- sdk/python/feast/field.py | 389 +++ sdk/python/feast/loaders/yaml.py | 3 +- sdk/python/feast/value_type.py | 23 + .../tensorflow_metadata/proto/v0/path_pb2.py | 69 - .../tensorflow_metadata/proto/v0/path_pb2.pyi | 52 - .../proto/v0/schema_pb2.py | 2256 ----------------- .../proto/v0/schema_pb2.pyi | 1063 -------- .../bikeshare_feature_set.yaml | 81 + .../tensorflow_metadata/bikeshare_schema.json | 136 + sdk/python/tests/test_feature_set.py | 101 +- 13 files changed, 917 insertions(+), 3450 deletions(-) delete mode 100644 sdk/python/tensorflow_metadata/proto/v0/path_pb2.py delete mode 100644 sdk/python/tensorflow_metadata/proto/v0/path_pb2.pyi delete mode 100644 sdk/python/tensorflow_metadata/proto/v0/schema_pb2.py delete mode 100644 sdk/python/tensorflow_metadata/proto/v0/schema_pb2.pyi create mode 100644 sdk/python/tests/data/tensorflow_metadata/bikeshare_feature_set.yaml create mode 100644 sdk/python/tests/data/tensorflow_metadata/bikeshare_schema.json diff --git a/sdk/python/feast/entity.py b/sdk/python/feast/entity.py index 5f823a754a0..9c5a027b974 100644 --- a/sdk/python/feast/entity.py +++ b/sdk/python/feast/entity.py @@ -29,7 +29,26 @@ def to_proto(self) -> EntityProto: Returns EntitySpec object """ value_type = ValueTypeProto.ValueType.Enum.Value(self.dtype.name) - return EntityProto(name=self.name, value_type=value_type) + return EntityProto( + name=self.name, + value_type=value_type, + presence=self.presence, + group_presence=self.group_presence, + shape=self.shape, + value_count=self.value_count, + domain=self.domain, + int_domain=self.int_domain, + float_domain=self.float_domain, + string_domain=self.string_domain, + bool_domain=self.bool_domain, + struct_domain=self.struct_domain, + natural_language_domain=self.natural_language_domain, + image_domain=self.image_domain, + mid_domain=self.mid_domain, + url_domain=self.url_domain, + time_domain=self.time_domain, + time_of_day_domain=self.time_of_day_domain, + ) @classmethod def from_proto(cls, entity_proto: EntityProto): @@ -42,4 +61,8 @@ def from_proto(cls, entity_proto: EntityProto): Returns: Entity object """ - return cls(name=entity_proto.name, dtype=ValueType(entity_proto.value_type)) + entity = cls(name=entity_proto.name, dtype=ValueType(entity_proto.value_type)) + entity.update_presence_constraints(entity_proto) + entity.update_shape_type(entity_proto) + entity.update_domain_info(entity_proto) + return entity diff --git a/sdk/python/feast/feature.py b/sdk/python/feast/feature.py index c9fc1cbff40..9c7ff20f9e2 100644 --- a/sdk/python/feast/feature.py +++ b/sdk/python/feast/feature.py @@ -24,9 +24,41 @@ class Feature(Field): def to_proto(self) -> FeatureProto: """Converts Feature object to its Protocol Buffer representation""" value_type = ValueTypeProto.ValueType.Enum.Value(self.dtype.name) - return FeatureProto(name=self.name, value_type=value_type) + return FeatureProto( + name=self.name, + value_type=value_type, + presence=self.presence, + group_presence=self.group_presence, + shape=self.shape, + value_count=self.value_count, + domain=self.domain, + int_domain=self.int_domain, + float_domain=self.float_domain, + string_domain=self.string_domain, + bool_domain=self.bool_domain, + struct_domain=self.struct_domain, + natural_language_domain=self.natural_language_domain, + image_domain=self.image_domain, + mid_domain=self.mid_domain, + url_domain=self.url_domain, + time_domain=self.time_domain, + time_of_day_domain=self.time_of_day_domain, + ) @classmethod def from_proto(cls, feature_proto: FeatureProto): - """Converts Protobuf Feature to its SDK equivalent""" - return cls(name=feature_proto.name, dtype=ValueType(feature_proto.value_type)) + """ + + Args: + feature_proto: FeatureSpec protobuf object + + Returns: + Feature object + """ + feature = cls( + name=feature_proto.name, dtype=ValueType(feature_proto.value_type) + ) + feature.update_presence_constraints(feature_proto) + feature.update_shape_type(feature_proto) + feature.update_domain_info(feature_proto) + return feature diff --git a/sdk/python/feast/feature_set.py b/sdk/python/feast/feature_set.py index c4cedaf6b2a..c6104f47a08 100644 --- a/sdk/python/feast/feature_set.py +++ b/sdk/python/feast/feature_set.py @@ -11,18 +11,20 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. - - +import warnings from collections import OrderedDict -from typing import Dict, List, Optional +from typing import Dict +from typing import List, Optional import pandas as pd import pyarrow as pa from google.protobuf import json_format from google.protobuf.duration_pb2 import Duration from google.protobuf.json_format import MessageToJson +from google.protobuf.message import Message from pandas.api.types import is_datetime64_ns_dtype from pyarrow.lib import TimestampType +from tensorflow_metadata.proto.v0 import schema_pb2 from feast.core.FeatureSet_pb2 import FeatureSet as FeatureSetProto from feast.core.FeatureSet_pb2 import FeatureSetMeta as FeatureSetMetaProto @@ -657,6 +659,93 @@ def is_valid(self): if len(self.entities) == 0: raise ValueError(f"No entities found in feature set {self.name}") + def import_tfx_schema(self, schema: schema_pb2.Schema): + """ + Updates presence_constraints, shape_type and domain_info for all fields + (features and entities) in the FeatureSet from schema in the Tensorflow metadata. + + Args: + schema: Schema from Tensorflow metadata + + Returns: + None + + """ + _make_tfx_schema_domain_info_inline(schema) + for feature_from_tfx_schema in schema.feature: + if feature_from_tfx_schema.name in self._fields.keys(): + field = self._fields[feature_from_tfx_schema.name] + field.update_presence_constraints(feature_from_tfx_schema) + field.update_shape_type(feature_from_tfx_schema) + field.update_domain_info(feature_from_tfx_schema) + else: + warnings.warn( + f"The provided schema contains feature name '{feature_from_tfx_schema.name}' " + f"that does not exist in the FeatureSet '{self.name}' in Feast" + ) + + def export_tfx_schema(self) -> schema_pb2.Schema: + """ + Create a Tensorflow metadata schema from a FeatureSet. + + Returns: + Tensorflow metadata schema. + + """ + schema = schema_pb2.Schema() + + # List of attributes to copy from fields in the FeatureSet to feature in + # Tensorflow metadata schema where the attribute name is the same. + attributes_to_copy_from_field_to_feature = [ + "name", + "presence", + "group_presence", + "shape", + "value_count", + "domain", + "int_domain", + "float_domain", + "string_domain", + "bool_domain", + "struct_domain", + "_natural_language_domain", + "image_domain", + "mid_domain", + "url_domain", + "time_domain", + "time_of_day_domain", + ] + + for _, field in self._fields.items(): + feature = schema_pb2.Feature() + for attr in attributes_to_copy_from_field_to_feature: + if getattr(field, attr) is None: + # This corresponds to an unset member in the proto Oneof field. + continue + if issubclass(type(getattr(feature, attr)), Message): + # Proto message field to copy is an "embedded" field, so MergeFrom() + # method must be used. + getattr(feature, attr).MergeFrom(getattr(field, attr)) + elif issubclass(type(getattr(feature, attr)), (int, str, bool)): + # Proto message field is a simple Python type, so setattr() + # can be used. + setattr(feature, attr, getattr(field, attr)) + else: + warnings.warn( + f"Attribute '{attr}' cannot be copied from Field " + f"'{field.name}' in FeatureSet '{self.name}' to a " + f"Feature in the Tensorflow metadata schema, because" + f"the type is neither a Protobuf message or Python " + f"int, str and bool" + ) + # "type" attr is handled separately because the attribute name is different + # ("dtype" in field and "type" in Feature) and "type" in Feature is only + # a subset of "dtype". + feature.type = field.dtype.to_tfx_schema_feature_type() + schema.feature.append(feature) + + return schema + @classmethod def from_yaml(cls, yml: str): """ @@ -855,6 +944,40 @@ def __hash__(self): return hash(repr(self)) +def _make_tfx_schema_domain_info_inline(schema: schema_pb2.Schema) -> None: + """ + Copy top level domain info defined at schema level into inline definition. + One use case is when importing domain info from Tensorflow metadata schema + into Feast features. Feast features do not have access to schema level information + so the domain info needs to be inline. + + Args: + schema: Tensorflow metadata schema + + Returns: None + """ + # Reference to domains defined at schema level + domain_ref_to_string_domain = {d.name: d for d in schema.string_domain} + domain_ref_to_float_domain = {d.name: d for d in schema.float_domain} + domain_ref_to_int_domain = {d.name: d for d in schema.int_domain} + + # With the reference, it is safe to remove the domains defined at schema level + del schema.string_domain[:] + del schema.float_domain[:] + del schema.int_domain[:] + + for feature in schema.feature: + domain_info_case = feature.WhichOneof("domain_info") + if domain_info_case == "domain": + domain_ref = feature.domain + if domain_ref in domain_ref_to_string_domain: + feature.string_domain.MergeFrom(domain_ref_to_string_domain[domain_ref]) + elif domain_ref in domain_ref_to_float_domain: + feature.float_domain.MergeFrom(domain_ref_to_float_domain[domain_ref]) + elif domain_ref in domain_ref_to_int_domain: + feature.int_domain.MergeFrom(domain_ref_to_int_domain[domain_ref]) + + def _infer_pd_column_type(column, series, rows_to_sample): dtype = None sample_count = 0 diff --git a/sdk/python/feast/field.py b/sdk/python/feast/field.py index 2efd4587ff0..be56823489b 100644 --- a/sdk/python/feast/field.py +++ b/sdk/python/feast/field.py @@ -11,8 +11,11 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. +from typing import Union +from feast.core.FeatureSet_pb2 import EntitySpec, FeatureSpec from feast.value_type import ValueType +from tensorflow_metadata.proto.v0 import schema_pb2 class Field: @@ -26,6 +29,22 @@ def __init__(self, name: str, dtype: ValueType): if not isinstance(dtype, ValueType): raise ValueError("dtype is not a valid ValueType") self._dtype = dtype + self._presence = None + self._group_presence = None + self._shape = None + self._value_count = None + self._domain = None + self._int_domain = None + self._float_domain = None + self._string_domain = None + self._bool_domain = None + self._struct_domain = None + self._natural_language_domain = None + self._image_domain = None + self._mid_domain = None + self._url_domain = None + self._time_domain = None + self._time_of_day_domain = None def __eq__(self, other): if self.name != other.name or self.dtype != other.dtype: @@ -46,6 +65,354 @@ def dtype(self) -> ValueType: """ return self._dtype + @property + def presence(self) -> schema_pb2.FeaturePresence: + """ + Getter for presence of this field + """ + return self._presence + + @presence.setter + def presence(self, presence: schema_pb2.FeaturePresence): + """ + Setter for presence of this field + """ + if not isinstance(presence, schema_pb2.FeaturePresence): + raise TypeError("presence must be of FeaturePresence type") + self._clear_presence_constraints() + self._presence = presence + + @property + def group_presence(self) -> schema_pb2.FeaturePresenceWithinGroup: + """ + Getter for group_presence of this field + """ + return self._group_presence + + @group_presence.setter + def group_presence(self, group_presence: schema_pb2.FeaturePresenceWithinGroup): + """ + Setter for group_presence of this field + """ + if not isinstance(group_presence, schema_pb2.FeaturePresenceWithinGroup): + raise TypeError("group_presence must be of FeaturePresenceWithinGroup type") + self._clear_presence_constraints() + self._group_presence = group_presence + + @property + def shape(self) -> schema_pb2.FixedShape: + """ + Getter for shape of this field + """ + return self._shape + + @shape.setter + def shape(self, shape: schema_pb2.FixedShape): + """ + Setter for shape of this field + """ + if not isinstance(shape, schema_pb2.FixedShape): + raise TypeError("shape must be of FixedShape type") + self._clear_shape_type() + self._shape = shape + + @property + def value_count(self) -> schema_pb2.ValueCount: + """ + Getter for value_count of this field + """ + return self._value_count + + @value_count.setter + def value_count(self, value_count: schema_pb2.ValueCount): + """ + Setter for value_count of this field + """ + if not isinstance(value_count, schema_pb2.ValueCount): + raise TypeError("value_count must be of ValueCount type") + self._clear_shape_type() + self._value_count = value_count + + @property + def domain(self) -> str: + """ + Getter for domain of this field + """ + return self._domain + + @domain.setter + def domain(self, domain: str): + """ + Setter for domain of this field + """ + if not isinstance(domain, str): + raise TypeError("domain must be of str type") + self._clear_domain_info() + self._domain = domain + + @property + def int_domain(self) -> schema_pb2.IntDomain: + """ + Getter for int_domain of this field + """ + return self._int_domain + + @int_domain.setter + def int_domain(self, int_domain: schema_pb2.IntDomain): + """ + Setter for int_domain of this field + """ + if not isinstance(int_domain, schema_pb2.IntDomain): + raise TypeError("int_domain must be of IntDomain type") + self._clear_domain_info() + self._int_domain = int_domain + + @property + def float_domain(self) -> schema_pb2.FloatDomain: + """ + Getter for float_domain of this field + """ + return self._float_domain + + @float_domain.setter + def float_domain(self, float_domain: schema_pb2.FloatDomain): + """ + Setter for float_domain of this field + """ + if not isinstance(float_domain, schema_pb2.FloatDomain): + raise TypeError("float_domain must be of FloatDomain type") + self._clear_domain_info() + self._float_domain = float_domain + + @property + def string_domain(self) -> schema_pb2.StringDomain: + """ + Getter for string_domain of this field + """ + return self._string_domain + + @string_domain.setter + def string_domain(self, string_domain: schema_pb2.StringDomain): + """ + Setter for string_domain of this field + """ + if not isinstance(string_domain, schema_pb2.StringDomain): + raise TypeError("string_domain must be of StringDomain type") + self._clear_domain_info() + self._string_domain = string_domain + + @property + def bool_domain(self) -> schema_pb2.BoolDomain: + """ + Getter for bool_domain of this field + """ + return self._bool_domain + + @bool_domain.setter + def bool_domain(self, bool_domain: schema_pb2.BoolDomain): + """ + Setter for bool_domain of this field + """ + if not isinstance(bool_domain, schema_pb2.BoolDomain): + raise TypeError("bool_domain must be of BoolDomain type") + self._clear_domain_info() + self._bool_domain = bool_domain + + @property + def struct_domain(self) -> schema_pb2.StructDomain: + """ + Getter for struct_domain of this field + """ + return self._struct_domain + + @struct_domain.setter + def struct_domain(self, struct_domain: schema_pb2.StructDomain): + """ + Setter for struct_domain of this field + """ + if not isinstance(struct_domain, schema_pb2.StructDomain): + raise TypeError("struct_domain must be of StructDomain type") + self._clear_domain_info() + self._struct_domain = struct_domain + + @property + def natural_language_domain(self) -> schema_pb2.NaturalLanguageDomain: + """ + Getter for natural_language_domain of this field + """ + return self._natural_language_domain + + @natural_language_domain.setter + def natural_language_domain( + self, natural_language_domain: schema_pb2.NaturalLanguageDomain + ): + """ + Setter for natural_language_domin of this field + """ + if not isinstance(natural_language_domain, schema_pb2.NaturalLanguageDomain): + raise TypeError( + "natural_language_domain must be of NaturalLanguageDomain type" + ) + self._clear_domain_info() + self._natural_language_domain = natural_language_domain + + @property + def image_domain(self) -> schema_pb2.ImageDomain: + """ + Getter for image_domain of this field + """ + return self._image_domain + + @image_domain.setter + def image_domain(self, image_domain: schema_pb2.ImageDomain): + """ + Setter for image_domain of this field + """ + if not isinstance(image_domain, schema_pb2.ImageDomain): + raise TypeError("image_domain must be of ImageDomain type") + self._clear_domain_info() + self._image_domain = image_domain + + @property + def mid_domain(self) -> schema_pb2.MIDDomain: + """ + Getter for mid_domain of this field + """ + return self._mid_domain + + @mid_domain.setter + def mid_domain(self, mid_domain: schema_pb2.MIDDomain): + """ + Setter for mid_domain of this field + """ + if not isinstance(mid_domain, schema_pb2.MIDDomain): + raise TypeError("mid_domain must be of MIDDomain type") + self._clear_domain_info() + self._mid_domain = mid_domain + + @property + def url_domain(self) -> schema_pb2.URLDomain: + """ + Getter for url_domain of this field + """ + return self._url_domain + + @url_domain.setter + def url_domain(self, url_domain: schema_pb2.URLDomain): + """ + Setter for url_domain of this field + """ + if not isinstance(url_domain, schema_pb2.URLDomain): + raise TypeError("url_domain must be of URLDomain type") + self._clear_domain_info() + self.url_domain = url_domain + + @property + def time_domain(self) -> schema_pb2.TimeDomain: + """ + Getter for time_domain of this field + """ + return self._time_domain + + @time_domain.setter + def time_domain(self, time_domain: schema_pb2.TimeDomain): + """ + Setter for time_domain of this field + """ + if not isinstance(time_domain, schema_pb2.TimeDomain): + raise TypeError("time_domain must be of TimeDomain type") + self._clear_domain_info() + self._time_domain = time_domain + + @property + def time_of_day_domain(self) -> schema_pb2.TimeOfDayDomain: + """ + Getter for time_of_day_domain of this field + """ + return self._time_of_day_domain + + @time_of_day_domain.setter + def time_of_day_domain(self, time_of_day_domain): + """ + Setter for time_of_day_domain of this field + """ + if not isinstance(time_of_day_domain, schema_pb2.TimeOfDayDomain): + raise TypeError("time_of_day_domain must be of TimeOfDayDomain type") + self._clear_domain_info() + self._time_of_day_domain = time_of_day_domain + + def update_presence_constraints( + self, feature: Union[schema_pb2.Feature, EntitySpec, FeatureSpec] + ) -> None: + """ + Update the presence constraints in this field from Tensorflow Feature, + Feast EntitySpec or FeatureSpec + + Args: + feature: Tensorflow Feature, Feast EntitySpec or FeatureSpec + + Returns: None + """ + presence_constraints_case = feature.WhichOneof("presence_constraints") + if presence_constraints_case == "presence": + self.presence = feature.presence + elif presence_constraints_case == "group_presence": + self.group_presence = feature.group_presence + + def update_shape_type( + self, feature: Union[schema_pb2.Feature, EntitySpec, FeatureSpec] + ) -> None: + """ + Update the shape type in this field from Tensorflow Feature, + Feast EntitySpec or FeatureSpec + + Args: + feature: Tensorflow Feature, Feast EntitySpec or FeatureSpec + + Returns: None + """ + shape_type_case = feature.WhichOneof("shape_type") + if shape_type_case == "shape": + self.shape = feature.shape + elif shape_type_case == "value_count": + self.value_count = feature.value_count + + def update_domain_info( + self, feature: Union[schema_pb2.Feature, EntitySpec, FeatureSpec] + ) -> None: + """ + Update the domain info in this field from Tensorflow Feature, Feast EntitySpec + or FeatureSpec + + Args: + feature: Tensorflow Feature, Feast EntitySpec or FeatureSpec + + Returns: None + """ + domain_info_case = feature.WhichOneof("domain_info") + if domain_info_case == "int_domain": + self.int_domain = feature.int_domain + elif domain_info_case == "float_domain": + self.float_domain = feature.float_domain + elif domain_info_case == "string_domain": + self.string_domain = feature.string_domain + elif domain_info_case == "bool_domain": + self.bool_domain = feature.bool_domain + elif domain_info_case == "struct_domain": + self.struct_domain = feature.struct_domain + elif domain_info_case == "natural_language_domain": + self.natural_language_domain = feature.natural_language_domain + elif domain_info_case == "image_domain": + self.image_domain = feature.image_domain + elif domain_info_case == "mid_domain": + self.mid_domain = feature.mid_domain + elif domain_info_case == "url_domain": + self.url_domain = feature.url_domain + elif domain_info_case == "time_domain": + self.time_domain = feature.time_domain + elif domain_info_case == "time_of_day_domain": + self.time_of_day_domain = feature.time_of_day_domain + def to_proto(self): """ Unimplemented to_proto method for a field. This should be extended. @@ -57,3 +424,25 @@ def from_proto(self, proto): Unimplemented from_proto method for a field. This should be extended. """ pass + + def _clear_presence_constraints(self): + self._presence = None + self._group_presence = None + + def _clear_shape_type(self): + self._shape = None + self._value_count = None + + def _clear_domain_info(self): + self._domain = None + self._int_domain = None + self._float_domain = None + self._string_domain = None + self._bool_domain = None + self._struct_domain = None + self._natural_language_domain = None + self._image_domain = None + self._mid_domain = None + self._url_domain = None + self._time_domain = None + self._time_of_day_domain = None diff --git a/sdk/python/feast/loaders/yaml.py b/sdk/python/feast/loaders/yaml.py index 130a71a3d02..624bc47d49c 100644 --- a/sdk/python/feast/loaders/yaml.py +++ b/sdk/python/feast/loaders/yaml.py @@ -57,7 +57,8 @@ def _get_yaml_contents(yml: str) -> str: yml_content = yml else: raise Exception( - f"Invalid YAML provided. Please provide either a file path or YAML string: ${yml}" + f"Invalid YAML provided. Please provide either a file path or YAML string.\n" + f"Provided YAML: {yml}" ) return yml_content diff --git a/sdk/python/feast/value_type.py b/sdk/python/feast/value_type.py index df315480ce7..687dccc7b7f 100644 --- a/sdk/python/feast/value_type.py +++ b/sdk/python/feast/value_type.py @@ -14,6 +14,8 @@ import enum +from tensorflow_metadata.proto.v0 import schema_pb2 + class ValueType(enum.Enum): """ @@ -35,3 +37,24 @@ class ValueType(enum.Enum): DOUBLE_LIST = 15 FLOAT_LIST = 16 BOOL_LIST = 17 + + def to_tfx_schema_feature_type(self) -> schema_pb2.FeatureType: + if self.value in [ + ValueType.BYTES.value, + ValueType.STRING.value, + ValueType.BOOL.value, + ValueType.BYTES_LIST.value, + ValueType.STRING_LIST.value, + ValueType.INT32_LIST.value, + ValueType.INT64_LIST.value, + ValueType.DOUBLE_LIST.value, + ValueType.FLOAT_LIST.value, + ValueType.BOOL_LIST.value, + ]: + return schema_pb2.FeatureType.BYTES + elif self.value in [ValueType.INT32.value, ValueType.INT64.value]: + return schema_pb2.FeatureType.INT + elif self.value in [ValueType.DOUBLE.value, ValueType.FLOAT.value]: + return schema_pb2.FeatureType.FLOAT + else: + return schema_pb2.FeatureType.TYPE_UNKNOWN diff --git a/sdk/python/tensorflow_metadata/proto/v0/path_pb2.py b/sdk/python/tensorflow_metadata/proto/v0/path_pb2.py deleted file mode 100644 index 24850688592..00000000000 --- a/sdk/python/tensorflow_metadata/proto/v0/path_pb2.py +++ /dev/null @@ -1,69 +0,0 @@ -# -*- coding: utf-8 -*- -# Generated by the protocol buffer compiler. DO NOT EDIT! -# source: tensorflow_metadata/proto/v0/path.proto - -from google.protobuf import descriptor as _descriptor -from google.protobuf import message as _message -from google.protobuf import reflection as _reflection -from google.protobuf import symbol_database as _symbol_database -# @@protoc_insertion_point(imports) - -_sym_db = _symbol_database.Default() - - - - -DESCRIPTOR = _descriptor.FileDescriptor( - name='tensorflow_metadata/proto/v0/path.proto', - package='tensorflow.metadata.v0', - syntax='proto2', - serialized_options=b'\n\032org.tensorflow.metadata.v0P\001\370\001\001', - serialized_pb=b'\n\'tensorflow_metadata/proto/v0/path.proto\x12\x16tensorflow.metadata.v0\"\x14\n\x04Path\x12\x0c\n\x04step\x18\x01 \x03(\tB!\n\x1aorg.tensorflow.metadata.v0P\x01\xf8\x01\x01' -) - - - - -_PATH = _descriptor.Descriptor( - name='Path', - full_name='tensorflow.metadata.v0.Path', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='step', full_name='tensorflow.metadata.v0.Path.step', index=0, - number=1, type=9, cpp_type=9, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=67, - serialized_end=87, -) - -DESCRIPTOR.message_types_by_name['Path'] = _PATH -_sym_db.RegisterFileDescriptor(DESCRIPTOR) - -Path = _reflection.GeneratedProtocolMessageType('Path', (_message.Message,), { - 'DESCRIPTOR' : _PATH, - '__module__' : 'tensorflow_metadata.proto.v0.path_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.Path) - }) -_sym_db.RegisterMessage(Path) - - -DESCRIPTOR._options = None -# @@protoc_insertion_point(module_scope) diff --git a/sdk/python/tensorflow_metadata/proto/v0/path_pb2.pyi b/sdk/python/tensorflow_metadata/proto/v0/path_pb2.pyi deleted file mode 100644 index caf370bd372..00000000000 --- a/sdk/python/tensorflow_metadata/proto/v0/path_pb2.pyi +++ /dev/null @@ -1,52 +0,0 @@ -# @generated by generate_proto_mypy_stubs.py. Do not edit! -import sys -from google.protobuf.descriptor import ( - Descriptor as google___protobuf___descriptor___Descriptor, -) - -from google.protobuf.internal.containers import ( - RepeatedScalarFieldContainer as google___protobuf___internal___containers___RepeatedScalarFieldContainer, -) - -from google.protobuf.message import ( - Message as google___protobuf___message___Message, -) - -from typing import ( - Iterable as typing___Iterable, - Optional as typing___Optional, - Text as typing___Text, - Union as typing___Union, -) - -from typing_extensions import ( - Literal as typing_extensions___Literal, -) - - -builtin___bool = bool -builtin___bytes = bytes -builtin___float = float -builtin___int = int -if sys.version_info < (3,): - builtin___buffer = buffer - builtin___unicode = unicode - - -class Path(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - step = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] - - def __init__(self, - *, - step : typing___Optional[typing___Iterable[typing___Text]] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> Path: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> Path: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def ClearField(self, field_name: typing_extensions___Literal[u"step",b"step"]) -> None: ... diff --git a/sdk/python/tensorflow_metadata/proto/v0/schema_pb2.py b/sdk/python/tensorflow_metadata/proto/v0/schema_pb2.py deleted file mode 100644 index c27579f0e28..00000000000 --- a/sdk/python/tensorflow_metadata/proto/v0/schema_pb2.py +++ /dev/null @@ -1,2256 +0,0 @@ -# -*- coding: utf-8 -*- -# Generated by the protocol buffer compiler. DO NOT EDIT! -# source: tensorflow_metadata/proto/v0/schema.proto - -from google.protobuf.internal import enum_type_wrapper -from google.protobuf import descriptor as _descriptor -from google.protobuf import message as _message -from google.protobuf import reflection as _reflection -from google.protobuf import symbol_database as _symbol_database -# @@protoc_insertion_point(imports) - -_sym_db = _symbol_database.Default() - - -from google.protobuf import any_pb2 as google_dot_protobuf_dot_any__pb2 -from tensorflow_metadata.proto.v0 import path_pb2 as tensorflow__metadata_dot_proto_dot_v0_dot_path__pb2 - - -DESCRIPTOR = _descriptor.FileDescriptor( - name='tensorflow_metadata/proto/v0/schema.proto', - package='tensorflow.metadata.v0', - syntax='proto2', - serialized_options=b'\n\032org.tensorflow.metadata.v0P\001\370\001\001', - serialized_pb=b'\n)tensorflow_metadata/proto/v0/schema.proto\x12\x16tensorflow.metadata.v0\x1a\x19google/protobuf/any.proto\x1a\'tensorflow_metadata/proto/v0/path.proto\"\xe2\x05\n\x06Schema\x12\x30\n\x07\x66\x65\x61ture\x18\x01 \x03(\x0b\x32\x1f.tensorflow.metadata.v0.Feature\x12=\n\x0esparse_feature\x18\x06 \x03(\x0b\x32%.tensorflow.metadata.v0.SparseFeature\x12\x41\n\x10weighted_feature\x18\x0c \x03(\x0b\x32\'.tensorflow.metadata.v0.WeightedFeature\x12;\n\rstring_domain\x18\x04 \x03(\x0b\x32$.tensorflow.metadata.v0.StringDomain\x12\x39\n\x0c\x66loat_domain\x18\t \x03(\x0b\x32#.tensorflow.metadata.v0.FloatDomain\x12\x35\n\nint_domain\x18\n \x03(\x0b\x32!.tensorflow.metadata.v0.IntDomain\x12\x1b\n\x13\x64\x65\x66\x61ult_environment\x18\x05 \x03(\t\x12\x36\n\nannotation\x18\x08 \x01(\x0b\x32\".tensorflow.metadata.v0.Annotation\x12G\n\x13\x64\x61taset_constraints\x18\x0b \x01(\x0b\x32*.tensorflow.metadata.v0.DatasetConstraints\x12\x62\n\x1btensor_representation_group\x18\r \x03(\x0b\x32=.tensorflow.metadata.v0.Schema.TensorRepresentationGroupEntry\x1as\n\x1eTensorRepresentationGroupEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12@\n\x05value\x18\x02 \x01(\x0b\x32\x31.tensorflow.metadata.v0.TensorRepresentationGroup:\x02\x38\x01\"\xdf\x0b\n\x07\x46\x65\x61ture\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x16\n\ndeprecated\x18\x02 \x01(\x08\x42\x02\x18\x01\x12;\n\x08presence\x18\x0e \x01(\x0b\x32\'.tensorflow.metadata.v0.FeaturePresenceH\x00\x12L\n\x0egroup_presence\x18\x11 \x01(\x0b\x32\x32.tensorflow.metadata.v0.FeaturePresenceWithinGroupH\x00\x12\x33\n\x05shape\x18\x17 \x01(\x0b\x32\".tensorflow.metadata.v0.FixedShapeH\x01\x12\x39\n\x0bvalue_count\x18\x05 \x01(\x0b\x32\".tensorflow.metadata.v0.ValueCountH\x01\x12\x31\n\x04type\x18\x06 \x01(\x0e\x32#.tensorflow.metadata.v0.FeatureType\x12\x10\n\x06\x64omain\x18\x07 \x01(\tH\x02\x12\x37\n\nint_domain\x18\t \x01(\x0b\x32!.tensorflow.metadata.v0.IntDomainH\x02\x12;\n\x0c\x66loat_domain\x18\n \x01(\x0b\x32#.tensorflow.metadata.v0.FloatDomainH\x02\x12=\n\rstring_domain\x18\x0b \x01(\x0b\x32$.tensorflow.metadata.v0.StringDomainH\x02\x12\x39\n\x0b\x62ool_domain\x18\r \x01(\x0b\x32\".tensorflow.metadata.v0.BoolDomainH\x02\x12=\n\rstruct_domain\x18\x1d \x01(\x0b\x32$.tensorflow.metadata.v0.StructDomainH\x02\x12P\n\x17natural_language_domain\x18\x18 \x01(\x0b\x32-.tensorflow.metadata.v0.NaturalLanguageDomainH\x02\x12;\n\x0cimage_domain\x18\x19 \x01(\x0b\x32#.tensorflow.metadata.v0.ImageDomainH\x02\x12\x37\n\nmid_domain\x18\x1a \x01(\x0b\x32!.tensorflow.metadata.v0.MIDDomainH\x02\x12\x37\n\nurl_domain\x18\x1b \x01(\x0b\x32!.tensorflow.metadata.v0.URLDomainH\x02\x12\x39\n\x0btime_domain\x18\x1c \x01(\x0b\x32\".tensorflow.metadata.v0.TimeDomainH\x02\x12\x45\n\x12time_of_day_domain\x18\x1e \x01(\x0b\x32\'.tensorflow.metadata.v0.TimeOfDayDomainH\x02\x12Q\n\x18\x64istribution_constraints\x18\x0f \x01(\x0b\x32/.tensorflow.metadata.v0.DistributionConstraints\x12\x36\n\nannotation\x18\x10 \x01(\x0b\x32\".tensorflow.metadata.v0.Annotation\x12\x42\n\x0fskew_comparator\x18\x12 \x01(\x0b\x32).tensorflow.metadata.v0.FeatureComparator\x12\x43\n\x10\x64rift_comparator\x18\x15 \x01(\x0b\x32).tensorflow.metadata.v0.FeatureComparator\x12\x16\n\x0ein_environment\x18\x14 \x03(\t\x12\x1a\n\x12not_in_environment\x18\x13 \x03(\t\x12?\n\x0flifecycle_stage\x18\x16 \x01(\x0e\x32&.tensorflow.metadata.v0.LifecycleStageB\x16\n\x14presence_constraintsB\x0c\n\nshape_typeB\r\n\x0b\x64omain_info\"X\n\nAnnotation\x12\x0b\n\x03tag\x18\x01 \x03(\t\x12\x0f\n\x07\x63omment\x18\x02 \x03(\t\x12,\n\x0e\x65xtra_metadata\x18\x03 \x03(\x0b\x32\x14.google.protobuf.Any\"X\n\x16NumericValueComparator\x12\x1e\n\x16min_fraction_threshold\x18\x01 \x01(\x01\x12\x1e\n\x16max_fraction_threshold\x18\x02 \x01(\x01\"\xe0\x01\n\x12\x44\x61tasetConstraints\x12U\n\x1dnum_examples_drift_comparator\x18\x01 \x01(\x0b\x32..tensorflow.metadata.v0.NumericValueComparator\x12W\n\x1fnum_examples_version_comparator\x18\x02 \x01(\x0b\x32..tensorflow.metadata.v0.NumericValueComparator\x12\x1a\n\x12min_examples_count\x18\x03 \x01(\x03\"d\n\nFixedShape\x12\x33\n\x03\x64im\x18\x02 \x03(\x0b\x32&.tensorflow.metadata.v0.FixedShape.Dim\x1a!\n\x03\x44im\x12\x0c\n\x04size\x18\x01 \x01(\x03\x12\x0c\n\x04name\x18\x02 \x01(\t\"&\n\nValueCount\x12\x0b\n\x03min\x18\x01 \x01(\x03\x12\x0b\n\x03max\x18\x02 \x01(\x03\"\xc5\x01\n\x0fWeightedFeature\x12\x0c\n\x04name\x18\x01 \x01(\t\x12-\n\x07\x66\x65\x61ture\x18\x02 \x01(\x0b\x32\x1c.tensorflow.metadata.v0.Path\x12\x34\n\x0eweight_feature\x18\x03 \x01(\x0b\x32\x1c.tensorflow.metadata.v0.Path\x12?\n\x0flifecycle_stage\x18\x04 \x01(\x0e\x32&.tensorflow.metadata.v0.LifecycleStage\"\x90\x04\n\rSparseFeature\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x16\n\ndeprecated\x18\x02 \x01(\x08\x42\x02\x18\x01\x12?\n\x0flifecycle_stage\x18\x07 \x01(\x0e\x32&.tensorflow.metadata.v0.LifecycleStage\x12=\n\x08presence\x18\x04 \x01(\x0b\x32\'.tensorflow.metadata.v0.FeaturePresenceB\x02\x18\x01\x12\x37\n\x0b\x64\x65nse_shape\x18\x05 \x01(\x0b\x32\".tensorflow.metadata.v0.FixedShape\x12I\n\rindex_feature\x18\x06 \x03(\x0b\x32\x32.tensorflow.metadata.v0.SparseFeature.IndexFeature\x12\x11\n\tis_sorted\x18\x08 \x01(\x08\x12I\n\rvalue_feature\x18\t \x01(\x0b\x32\x32.tensorflow.metadata.v0.SparseFeature.ValueFeature\x12\x35\n\x04type\x18\n \x01(\x0e\x32#.tensorflow.metadata.v0.FeatureTypeB\x02\x18\x01\x1a\x1c\n\x0cIndexFeature\x12\x0c\n\x04name\x18\x01 \x01(\t\x1a\x1c\n\x0cValueFeature\x12\x0c\n\x04name\x18\x01 \x01(\tJ\x04\x08\x0b\x10\x0c\"5\n\x17\x44istributionConstraints\x12\x1a\n\x0fmin_domain_mass\x18\x01 \x01(\x01:\x01\x31\"K\n\tIntDomain\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0b\n\x03min\x18\x03 \x01(\x03\x12\x0b\n\x03max\x18\x04 \x01(\x03\x12\x16\n\x0eis_categorical\x18\x05 \x01(\x08\"5\n\x0b\x46loatDomain\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x0b\n\x03min\x18\x03 \x01(\x02\x12\x0b\n\x03max\x18\x04 \x01(\x02\"\x7f\n\x0cStructDomain\x12\x30\n\x07\x66\x65\x61ture\x18\x01 \x03(\x0b\x32\x1f.tensorflow.metadata.v0.Feature\x12=\n\x0esparse_feature\x18\x02 \x03(\x0b\x32%.tensorflow.metadata.v0.SparseFeature\"+\n\x0cStringDomain\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x03(\t\"C\n\nBoolDomain\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x12\n\ntrue_value\x18\x02 \x01(\t\x12\x13\n\x0b\x66\x61lse_value\x18\x03 \x01(\t\"\x17\n\x15NaturalLanguageDomain\"\r\n\x0bImageDomain\"\x0b\n\tMIDDomain\"\x0b\n\tURLDomain\"\x8e\x02\n\nTimeDomain\x12\x17\n\rstring_format\x18\x01 \x01(\tH\x00\x12N\n\x0einteger_format\x18\x02 \x01(\x0e\x32\x34.tensorflow.metadata.v0.TimeDomain.IntegerTimeFormatH\x00\"\x8c\x01\n\x11IntegerTimeFormat\x12\x12\n\x0e\x46ORMAT_UNKNOWN\x10\x00\x12\r\n\tUNIX_DAYS\x10\x05\x12\x10\n\x0cUNIX_SECONDS\x10\x01\x12\x15\n\x11UNIX_MILLISECONDS\x10\x02\x12\x15\n\x11UNIX_MICROSECONDS\x10\x03\x12\x14\n\x10UNIX_NANOSECONDS\x10\x04\x42\x08\n\x06\x66ormat\"\xd1\x01\n\x0fTimeOfDayDomain\x12\x17\n\rstring_format\x18\x01 \x01(\tH\x00\x12X\n\x0einteger_format\x18\x02 \x01(\x0e\x32>.tensorflow.metadata.v0.TimeOfDayDomain.IntegerTimeOfDayFormatH\x00\"A\n\x16IntegerTimeOfDayFormat\x12\x12\n\x0e\x46ORMAT_UNKNOWN\x10\x00\x12\x13\n\x0fPACKED_64_NANOS\x10\x01\x42\x08\n\x06\x66ormat\":\n\x0f\x46\x65\x61turePresence\x12\x14\n\x0cmin_fraction\x18\x01 \x01(\x01\x12\x11\n\tmin_count\x18\x02 \x01(\x03\".\n\x1a\x46\x65\x61turePresenceWithinGroup\x12\x10\n\x08required\x18\x01 \x01(\x08\"!\n\x0cInfinityNorm\x12\x11\n\tthreshold\x18\x01 \x01(\x01\"P\n\x11\x46\x65\x61tureComparator\x12;\n\rinfinity_norm\x18\x01 \x01(\x0b\x32$.tensorflow.metadata.v0.InfinityNorm\"\xeb\x05\n\x14TensorRepresentation\x12P\n\x0c\x64\x65nse_tensor\x18\x01 \x01(\x0b\x32\x38.tensorflow.metadata.v0.TensorRepresentation.DenseTensorH\x00\x12_\n\x14varlen_sparse_tensor\x18\x02 \x01(\x0b\x32?.tensorflow.metadata.v0.TensorRepresentation.VarLenSparseTensorH\x00\x12R\n\rsparse_tensor\x18\x03 \x01(\x0b\x32\x39.tensorflow.metadata.v0.TensorRepresentation.SparseTensorH\x00\x1ao\n\x0c\x44\x65\x66\x61ultValue\x12\x15\n\x0b\x66loat_value\x18\x01 \x01(\x01H\x00\x12\x13\n\tint_value\x18\x02 \x01(\x03H\x00\x12\x15\n\x0b\x62ytes_value\x18\x03 \x01(\x0cH\x00\x12\x14\n\nuint_value\x18\x04 \x01(\x04H\x00\x42\x06\n\x04kind\x1a\xa7\x01\n\x0b\x44\x65nseTensor\x12\x13\n\x0b\x63olumn_name\x18\x01 \x01(\t\x12\x31\n\x05shape\x18\x02 \x01(\x0b\x32\".tensorflow.metadata.v0.FixedShape\x12P\n\rdefault_value\x18\x03 \x01(\x0b\x32\x39.tensorflow.metadata.v0.TensorRepresentation.DefaultValue\x1a)\n\x12VarLenSparseTensor\x12\x13\n\x0b\x63olumn_name\x18\x01 \x01(\t\x1a~\n\x0cSparseTensor\x12\x37\n\x0b\x64\x65nse_shape\x18\x01 \x01(\x0b\x32\".tensorflow.metadata.v0.FixedShape\x12\x1a\n\x12index_column_names\x18\x02 \x03(\t\x12\x19\n\x11value_column_name\x18\x03 \x01(\tB\x06\n\x04kind\"\xf2\x01\n\x19TensorRepresentationGroup\x12j\n\x15tensor_representation\x18\x01 \x03(\x0b\x32K.tensorflow.metadata.v0.TensorRepresentationGroup.TensorRepresentationEntry\x1ai\n\x19TensorRepresentationEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12;\n\x05value\x18\x02 \x01(\x0b\x32,.tensorflow.metadata.v0.TensorRepresentation:\x02\x38\x01*u\n\x0eLifecycleStage\x12\x11\n\rUNKNOWN_STAGE\x10\x00\x12\x0b\n\x07PLANNED\x10\x01\x12\t\n\x05\x41LPHA\x10\x02\x12\x08\n\x04\x42\x45TA\x10\x03\x12\x0e\n\nPRODUCTION\x10\x04\x12\x0e\n\nDEPRECATED\x10\x05\x12\x0e\n\nDEBUG_ONLY\x10\x06*J\n\x0b\x46\x65\x61tureType\x12\x10\n\x0cTYPE_UNKNOWN\x10\x00\x12\t\n\x05\x42YTES\x10\x01\x12\x07\n\x03INT\x10\x02\x12\t\n\x05\x46LOAT\x10\x03\x12\n\n\x06STRUCT\x10\x04\x42!\n\x1aorg.tensorflow.metadata.v0P\x01\xf8\x01\x01' - , - dependencies=[google_dot_protobuf_dot_any__pb2.DESCRIPTOR,tensorflow__metadata_dot_proto_dot_v0_dot_path__pb2.DESCRIPTOR,]) - -_LIFECYCLESTAGE = _descriptor.EnumDescriptor( - name='LifecycleStage', - full_name='tensorflow.metadata.v0.LifecycleStage', - filename=None, - file=DESCRIPTOR, - values=[ - _descriptor.EnumValueDescriptor( - name='UNKNOWN_STAGE', index=0, number=0, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='PLANNED', index=1, number=1, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='ALPHA', index=2, number=2, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='BETA', index=3, number=3, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='PRODUCTION', index=4, number=4, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='DEPRECATED', index=5, number=5, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='DEBUG_ONLY', index=6, number=6, - serialized_options=None, - type=None), - ], - containing_type=None, - serialized_options=None, - serialized_start=5865, - serialized_end=5982, -) -_sym_db.RegisterEnumDescriptor(_LIFECYCLESTAGE) - -LifecycleStage = enum_type_wrapper.EnumTypeWrapper(_LIFECYCLESTAGE) -_FEATURETYPE = _descriptor.EnumDescriptor( - name='FeatureType', - full_name='tensorflow.metadata.v0.FeatureType', - filename=None, - file=DESCRIPTOR, - values=[ - _descriptor.EnumValueDescriptor( - name='TYPE_UNKNOWN', index=0, number=0, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='BYTES', index=1, number=1, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='INT', index=2, number=2, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='FLOAT', index=3, number=3, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='STRUCT', index=4, number=4, - serialized_options=None, - type=None), - ], - containing_type=None, - serialized_options=None, - serialized_start=5984, - serialized_end=6058, -) -_sym_db.RegisterEnumDescriptor(_FEATURETYPE) - -FeatureType = enum_type_wrapper.EnumTypeWrapper(_FEATURETYPE) -UNKNOWN_STAGE = 0 -PLANNED = 1 -ALPHA = 2 -BETA = 3 -PRODUCTION = 4 -DEPRECATED = 5 -DEBUG_ONLY = 6 -TYPE_UNKNOWN = 0 -BYTES = 1 -INT = 2 -FLOAT = 3 -STRUCT = 4 - - -_TIMEDOMAIN_INTEGERTIMEFORMAT = _descriptor.EnumDescriptor( - name='IntegerTimeFormat', - full_name='tensorflow.metadata.v0.TimeDomain.IntegerTimeFormat', - filename=None, - file=DESCRIPTOR, - values=[ - _descriptor.EnumValueDescriptor( - name='FORMAT_UNKNOWN', index=0, number=0, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='UNIX_DAYS', index=1, number=5, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='UNIX_SECONDS', index=2, number=1, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='UNIX_MILLISECONDS', index=3, number=2, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='UNIX_MICROSECONDS', index=4, number=3, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='UNIX_NANOSECONDS', index=5, number=4, - serialized_options=None, - type=None), - ], - containing_type=None, - serialized_options=None, - serialized_start=4281, - serialized_end=4421, -) -_sym_db.RegisterEnumDescriptor(_TIMEDOMAIN_INTEGERTIMEFORMAT) - -_TIMEOFDAYDOMAIN_INTEGERTIMEOFDAYFORMAT = _descriptor.EnumDescriptor( - name='IntegerTimeOfDayFormat', - full_name='tensorflow.metadata.v0.TimeOfDayDomain.IntegerTimeOfDayFormat', - filename=None, - file=DESCRIPTOR, - values=[ - _descriptor.EnumValueDescriptor( - name='FORMAT_UNKNOWN', index=0, number=0, - serialized_options=None, - type=None), - _descriptor.EnumValueDescriptor( - name='PACKED_64_NANOS', index=1, number=1, - serialized_options=None, - type=None), - ], - containing_type=None, - serialized_options=None, - serialized_start=4568, - serialized_end=4633, -) -_sym_db.RegisterEnumDescriptor(_TIMEOFDAYDOMAIN_INTEGERTIMEOFDAYFORMAT) - - -_SCHEMA_TENSORREPRESENTATIONGROUPENTRY = _descriptor.Descriptor( - name='TensorRepresentationGroupEntry', - full_name='tensorflow.metadata.v0.Schema.TensorRepresentationGroupEntry', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='key', full_name='tensorflow.metadata.v0.Schema.TensorRepresentationGroupEntry.key', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='value', full_name='tensorflow.metadata.v0.Schema.TensorRepresentationGroupEntry.value', index=1, - number=2, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=b'8\001', - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=761, - serialized_end=876, -) - -_SCHEMA = _descriptor.Descriptor( - name='Schema', - full_name='tensorflow.metadata.v0.Schema', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='feature', full_name='tensorflow.metadata.v0.Schema.feature', index=0, - number=1, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='sparse_feature', full_name='tensorflow.metadata.v0.Schema.sparse_feature', index=1, - number=6, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='weighted_feature', full_name='tensorflow.metadata.v0.Schema.weighted_feature', index=2, - number=12, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='string_domain', full_name='tensorflow.metadata.v0.Schema.string_domain', index=3, - number=4, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='float_domain', full_name='tensorflow.metadata.v0.Schema.float_domain', index=4, - number=9, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='int_domain', full_name='tensorflow.metadata.v0.Schema.int_domain', index=5, - number=10, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='default_environment', full_name='tensorflow.metadata.v0.Schema.default_environment', index=6, - number=5, type=9, cpp_type=9, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='annotation', full_name='tensorflow.metadata.v0.Schema.annotation', index=7, - number=8, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='dataset_constraints', full_name='tensorflow.metadata.v0.Schema.dataset_constraints', index=8, - number=11, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='tensor_representation_group', full_name='tensorflow.metadata.v0.Schema.tensor_representation_group', index=9, - number=13, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[_SCHEMA_TENSORREPRESENTATIONGROUPENTRY, ], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=138, - serialized_end=876, -) - - -_FEATURE = _descriptor.Descriptor( - name='Feature', - full_name='tensorflow.metadata.v0.Feature', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='name', full_name='tensorflow.metadata.v0.Feature.name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='deprecated', full_name='tensorflow.metadata.v0.Feature.deprecated', index=1, - number=2, type=8, cpp_type=7, label=1, - has_default_value=False, default_value=False, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=b'\030\001', file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='presence', full_name='tensorflow.metadata.v0.Feature.presence', index=2, - number=14, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='group_presence', full_name='tensorflow.metadata.v0.Feature.group_presence', index=3, - number=17, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='shape', full_name='tensorflow.metadata.v0.Feature.shape', index=4, - number=23, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='value_count', full_name='tensorflow.metadata.v0.Feature.value_count', index=5, - number=5, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='type', full_name='tensorflow.metadata.v0.Feature.type', index=6, - number=6, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='domain', full_name='tensorflow.metadata.v0.Feature.domain', index=7, - number=7, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='int_domain', full_name='tensorflow.metadata.v0.Feature.int_domain', index=8, - number=9, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='float_domain', full_name='tensorflow.metadata.v0.Feature.float_domain', index=9, - number=10, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='string_domain', full_name='tensorflow.metadata.v0.Feature.string_domain', index=10, - number=11, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='bool_domain', full_name='tensorflow.metadata.v0.Feature.bool_domain', index=11, - number=13, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='struct_domain', full_name='tensorflow.metadata.v0.Feature.struct_domain', index=12, - number=29, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='natural_language_domain', full_name='tensorflow.metadata.v0.Feature.natural_language_domain', index=13, - number=24, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='image_domain', full_name='tensorflow.metadata.v0.Feature.image_domain', index=14, - number=25, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='mid_domain', full_name='tensorflow.metadata.v0.Feature.mid_domain', index=15, - number=26, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='url_domain', full_name='tensorflow.metadata.v0.Feature.url_domain', index=16, - number=27, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='time_domain', full_name='tensorflow.metadata.v0.Feature.time_domain', index=17, - number=28, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='time_of_day_domain', full_name='tensorflow.metadata.v0.Feature.time_of_day_domain', index=18, - number=30, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='distribution_constraints', full_name='tensorflow.metadata.v0.Feature.distribution_constraints', index=19, - number=15, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='annotation', full_name='tensorflow.metadata.v0.Feature.annotation', index=20, - number=16, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='skew_comparator', full_name='tensorflow.metadata.v0.Feature.skew_comparator', index=21, - number=18, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='drift_comparator', full_name='tensorflow.metadata.v0.Feature.drift_comparator', index=22, - number=21, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='in_environment', full_name='tensorflow.metadata.v0.Feature.in_environment', index=23, - number=20, type=9, cpp_type=9, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='not_in_environment', full_name='tensorflow.metadata.v0.Feature.not_in_environment', index=24, - number=19, type=9, cpp_type=9, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='lifecycle_stage', full_name='tensorflow.metadata.v0.Feature.lifecycle_stage', index=25, - number=22, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - _descriptor.OneofDescriptor( - name='presence_constraints', full_name='tensorflow.metadata.v0.Feature.presence_constraints', - index=0, containing_type=None, fields=[]), - _descriptor.OneofDescriptor( - name='shape_type', full_name='tensorflow.metadata.v0.Feature.shape_type', - index=1, containing_type=None, fields=[]), - _descriptor.OneofDescriptor( - name='domain_info', full_name='tensorflow.metadata.v0.Feature.domain_info', - index=2, containing_type=None, fields=[]), - ], - serialized_start=879, - serialized_end=2382, -) - - -_ANNOTATION = _descriptor.Descriptor( - name='Annotation', - full_name='tensorflow.metadata.v0.Annotation', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='tag', full_name='tensorflow.metadata.v0.Annotation.tag', index=0, - number=1, type=9, cpp_type=9, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='comment', full_name='tensorflow.metadata.v0.Annotation.comment', index=1, - number=2, type=9, cpp_type=9, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='extra_metadata', full_name='tensorflow.metadata.v0.Annotation.extra_metadata', index=2, - number=3, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=2384, - serialized_end=2472, -) - - -_NUMERICVALUECOMPARATOR = _descriptor.Descriptor( - name='NumericValueComparator', - full_name='tensorflow.metadata.v0.NumericValueComparator', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='min_fraction_threshold', full_name='tensorflow.metadata.v0.NumericValueComparator.min_fraction_threshold', index=0, - number=1, type=1, cpp_type=5, label=1, - has_default_value=False, default_value=float(0), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='max_fraction_threshold', full_name='tensorflow.metadata.v0.NumericValueComparator.max_fraction_threshold', index=1, - number=2, type=1, cpp_type=5, label=1, - has_default_value=False, default_value=float(0), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=2474, - serialized_end=2562, -) - - -_DATASETCONSTRAINTS = _descriptor.Descriptor( - name='DatasetConstraints', - full_name='tensorflow.metadata.v0.DatasetConstraints', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='num_examples_drift_comparator', full_name='tensorflow.metadata.v0.DatasetConstraints.num_examples_drift_comparator', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='num_examples_version_comparator', full_name='tensorflow.metadata.v0.DatasetConstraints.num_examples_version_comparator', index=1, - number=2, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='min_examples_count', full_name='tensorflow.metadata.v0.DatasetConstraints.min_examples_count', index=2, - number=3, type=3, cpp_type=2, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=2565, - serialized_end=2789, -) - - -_FIXEDSHAPE_DIM = _descriptor.Descriptor( - name='Dim', - full_name='tensorflow.metadata.v0.FixedShape.Dim', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='size', full_name='tensorflow.metadata.v0.FixedShape.Dim.size', index=0, - number=1, type=3, cpp_type=2, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='name', full_name='tensorflow.metadata.v0.FixedShape.Dim.name', index=1, - number=2, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=2858, - serialized_end=2891, -) - -_FIXEDSHAPE = _descriptor.Descriptor( - name='FixedShape', - full_name='tensorflow.metadata.v0.FixedShape', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='dim', full_name='tensorflow.metadata.v0.FixedShape.dim', index=0, - number=2, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[_FIXEDSHAPE_DIM, ], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=2791, - serialized_end=2891, -) - - -_VALUECOUNT = _descriptor.Descriptor( - name='ValueCount', - full_name='tensorflow.metadata.v0.ValueCount', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='min', full_name='tensorflow.metadata.v0.ValueCount.min', index=0, - number=1, type=3, cpp_type=2, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='max', full_name='tensorflow.metadata.v0.ValueCount.max', index=1, - number=2, type=3, cpp_type=2, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=2893, - serialized_end=2931, -) - - -_WEIGHTEDFEATURE = _descriptor.Descriptor( - name='WeightedFeature', - full_name='tensorflow.metadata.v0.WeightedFeature', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='name', full_name='tensorflow.metadata.v0.WeightedFeature.name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='feature', full_name='tensorflow.metadata.v0.WeightedFeature.feature', index=1, - number=2, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='weight_feature', full_name='tensorflow.metadata.v0.WeightedFeature.weight_feature', index=2, - number=3, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='lifecycle_stage', full_name='tensorflow.metadata.v0.WeightedFeature.lifecycle_stage', index=3, - number=4, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=2934, - serialized_end=3131, -) - - -_SPARSEFEATURE_INDEXFEATURE = _descriptor.Descriptor( - name='IndexFeature', - full_name='tensorflow.metadata.v0.SparseFeature.IndexFeature', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='name', full_name='tensorflow.metadata.v0.SparseFeature.IndexFeature.name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=3598, - serialized_end=3626, -) - -_SPARSEFEATURE_VALUEFEATURE = _descriptor.Descriptor( - name='ValueFeature', - full_name='tensorflow.metadata.v0.SparseFeature.ValueFeature', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='name', full_name='tensorflow.metadata.v0.SparseFeature.ValueFeature.name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=3628, - serialized_end=3656, -) - -_SPARSEFEATURE = _descriptor.Descriptor( - name='SparseFeature', - full_name='tensorflow.metadata.v0.SparseFeature', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='name', full_name='tensorflow.metadata.v0.SparseFeature.name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='deprecated', full_name='tensorflow.metadata.v0.SparseFeature.deprecated', index=1, - number=2, type=8, cpp_type=7, label=1, - has_default_value=False, default_value=False, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=b'\030\001', file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='lifecycle_stage', full_name='tensorflow.metadata.v0.SparseFeature.lifecycle_stage', index=2, - number=7, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='presence', full_name='tensorflow.metadata.v0.SparseFeature.presence', index=3, - number=4, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=b'\030\001', file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='dense_shape', full_name='tensorflow.metadata.v0.SparseFeature.dense_shape', index=4, - number=5, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='index_feature', full_name='tensorflow.metadata.v0.SparseFeature.index_feature', index=5, - number=6, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='is_sorted', full_name='tensorflow.metadata.v0.SparseFeature.is_sorted', index=6, - number=8, type=8, cpp_type=7, label=1, - has_default_value=False, default_value=False, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='value_feature', full_name='tensorflow.metadata.v0.SparseFeature.value_feature', index=7, - number=9, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='type', full_name='tensorflow.metadata.v0.SparseFeature.type', index=8, - number=10, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=b'\030\001', file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[_SPARSEFEATURE_INDEXFEATURE, _SPARSEFEATURE_VALUEFEATURE, ], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=3134, - serialized_end=3662, -) - - -_DISTRIBUTIONCONSTRAINTS = _descriptor.Descriptor( - name='DistributionConstraints', - full_name='tensorflow.metadata.v0.DistributionConstraints', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='min_domain_mass', full_name='tensorflow.metadata.v0.DistributionConstraints.min_domain_mass', index=0, - number=1, type=1, cpp_type=5, label=1, - has_default_value=True, default_value=float(1), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=3664, - serialized_end=3717, -) - - -_INTDOMAIN = _descriptor.Descriptor( - name='IntDomain', - full_name='tensorflow.metadata.v0.IntDomain', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='name', full_name='tensorflow.metadata.v0.IntDomain.name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='min', full_name='tensorflow.metadata.v0.IntDomain.min', index=1, - number=3, type=3, cpp_type=2, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='max', full_name='tensorflow.metadata.v0.IntDomain.max', index=2, - number=4, type=3, cpp_type=2, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='is_categorical', full_name='tensorflow.metadata.v0.IntDomain.is_categorical', index=3, - number=5, type=8, cpp_type=7, label=1, - has_default_value=False, default_value=False, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=3719, - serialized_end=3794, -) - - -_FLOATDOMAIN = _descriptor.Descriptor( - name='FloatDomain', - full_name='tensorflow.metadata.v0.FloatDomain', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='name', full_name='tensorflow.metadata.v0.FloatDomain.name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='min', full_name='tensorflow.metadata.v0.FloatDomain.min', index=1, - number=3, type=2, cpp_type=6, label=1, - has_default_value=False, default_value=float(0), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='max', full_name='tensorflow.metadata.v0.FloatDomain.max', index=2, - number=4, type=2, cpp_type=6, label=1, - has_default_value=False, default_value=float(0), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=3796, - serialized_end=3849, -) - - -_STRUCTDOMAIN = _descriptor.Descriptor( - name='StructDomain', - full_name='tensorflow.metadata.v0.StructDomain', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='feature', full_name='tensorflow.metadata.v0.StructDomain.feature', index=0, - number=1, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='sparse_feature', full_name='tensorflow.metadata.v0.StructDomain.sparse_feature', index=1, - number=2, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=3851, - serialized_end=3978, -) - - -_STRINGDOMAIN = _descriptor.Descriptor( - name='StringDomain', - full_name='tensorflow.metadata.v0.StringDomain', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='name', full_name='tensorflow.metadata.v0.StringDomain.name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='value', full_name='tensorflow.metadata.v0.StringDomain.value', index=1, - number=2, type=9, cpp_type=9, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=3980, - serialized_end=4023, -) - - -_BOOLDOMAIN = _descriptor.Descriptor( - name='BoolDomain', - full_name='tensorflow.metadata.v0.BoolDomain', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='name', full_name='tensorflow.metadata.v0.BoolDomain.name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='true_value', full_name='tensorflow.metadata.v0.BoolDomain.true_value', index=1, - number=2, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='false_value', full_name='tensorflow.metadata.v0.BoolDomain.false_value', index=2, - number=3, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=4025, - serialized_end=4092, -) - - -_NATURALLANGUAGEDOMAIN = _descriptor.Descriptor( - name='NaturalLanguageDomain', - full_name='tensorflow.metadata.v0.NaturalLanguageDomain', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=4094, - serialized_end=4117, -) - - -_IMAGEDOMAIN = _descriptor.Descriptor( - name='ImageDomain', - full_name='tensorflow.metadata.v0.ImageDomain', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=4119, - serialized_end=4132, -) - - -_MIDDOMAIN = _descriptor.Descriptor( - name='MIDDomain', - full_name='tensorflow.metadata.v0.MIDDomain', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=4134, - serialized_end=4145, -) - - -_URLDOMAIN = _descriptor.Descriptor( - name='URLDomain', - full_name='tensorflow.metadata.v0.URLDomain', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=4147, - serialized_end=4158, -) - - -_TIMEDOMAIN = _descriptor.Descriptor( - name='TimeDomain', - full_name='tensorflow.metadata.v0.TimeDomain', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='string_format', full_name='tensorflow.metadata.v0.TimeDomain.string_format', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='integer_format', full_name='tensorflow.metadata.v0.TimeDomain.integer_format', index=1, - number=2, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - _TIMEDOMAIN_INTEGERTIMEFORMAT, - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - _descriptor.OneofDescriptor( - name='format', full_name='tensorflow.metadata.v0.TimeDomain.format', - index=0, containing_type=None, fields=[]), - ], - serialized_start=4161, - serialized_end=4431, -) - - -_TIMEOFDAYDOMAIN = _descriptor.Descriptor( - name='TimeOfDayDomain', - full_name='tensorflow.metadata.v0.TimeOfDayDomain', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='string_format', full_name='tensorflow.metadata.v0.TimeOfDayDomain.string_format', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='integer_format', full_name='tensorflow.metadata.v0.TimeOfDayDomain.integer_format', index=1, - number=2, type=14, cpp_type=8, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - _TIMEOFDAYDOMAIN_INTEGERTIMEOFDAYFORMAT, - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - _descriptor.OneofDescriptor( - name='format', full_name='tensorflow.metadata.v0.TimeOfDayDomain.format', - index=0, containing_type=None, fields=[]), - ], - serialized_start=4434, - serialized_end=4643, -) - - -_FEATUREPRESENCE = _descriptor.Descriptor( - name='FeaturePresence', - full_name='tensorflow.metadata.v0.FeaturePresence', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='min_fraction', full_name='tensorflow.metadata.v0.FeaturePresence.min_fraction', index=0, - number=1, type=1, cpp_type=5, label=1, - has_default_value=False, default_value=float(0), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='min_count', full_name='tensorflow.metadata.v0.FeaturePresence.min_count', index=1, - number=2, type=3, cpp_type=2, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=4645, - serialized_end=4703, -) - - -_FEATUREPRESENCEWITHINGROUP = _descriptor.Descriptor( - name='FeaturePresenceWithinGroup', - full_name='tensorflow.metadata.v0.FeaturePresenceWithinGroup', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='required', full_name='tensorflow.metadata.v0.FeaturePresenceWithinGroup.required', index=0, - number=1, type=8, cpp_type=7, label=1, - has_default_value=False, default_value=False, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=4705, - serialized_end=4751, -) - - -_INFINITYNORM = _descriptor.Descriptor( - name='InfinityNorm', - full_name='tensorflow.metadata.v0.InfinityNorm', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='threshold', full_name='tensorflow.metadata.v0.InfinityNorm.threshold', index=0, - number=1, type=1, cpp_type=5, label=1, - has_default_value=False, default_value=float(0), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=4753, - serialized_end=4786, -) - - -_FEATURECOMPARATOR = _descriptor.Descriptor( - name='FeatureComparator', - full_name='tensorflow.metadata.v0.FeatureComparator', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='infinity_norm', full_name='tensorflow.metadata.v0.FeatureComparator.infinity_norm', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=4788, - serialized_end=4868, -) - - -_TENSORREPRESENTATION_DEFAULTVALUE = _descriptor.Descriptor( - name='DefaultValue', - full_name='tensorflow.metadata.v0.TensorRepresentation.DefaultValue', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='float_value', full_name='tensorflow.metadata.v0.TensorRepresentation.DefaultValue.float_value', index=0, - number=1, type=1, cpp_type=5, label=1, - has_default_value=False, default_value=float(0), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='int_value', full_name='tensorflow.metadata.v0.TensorRepresentation.DefaultValue.int_value', index=1, - number=2, type=3, cpp_type=2, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='bytes_value', full_name='tensorflow.metadata.v0.TensorRepresentation.DefaultValue.bytes_value', index=2, - number=3, type=12, cpp_type=9, label=1, - has_default_value=False, default_value=b"", - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='uint_value', full_name='tensorflow.metadata.v0.TensorRepresentation.DefaultValue.uint_value', index=3, - number=4, type=4, cpp_type=4, label=1, - has_default_value=False, default_value=0, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - _descriptor.OneofDescriptor( - name='kind', full_name='tensorflow.metadata.v0.TensorRepresentation.DefaultValue.kind', - index=0, containing_type=None, fields=[]), - ], - serialized_start=5158, - serialized_end=5269, -) - -_TENSORREPRESENTATION_DENSETENSOR = _descriptor.Descriptor( - name='DenseTensor', - full_name='tensorflow.metadata.v0.TensorRepresentation.DenseTensor', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='column_name', full_name='tensorflow.metadata.v0.TensorRepresentation.DenseTensor.column_name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='shape', full_name='tensorflow.metadata.v0.TensorRepresentation.DenseTensor.shape', index=1, - number=2, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='default_value', full_name='tensorflow.metadata.v0.TensorRepresentation.DenseTensor.default_value', index=2, - number=3, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=5272, - serialized_end=5439, -) - -_TENSORREPRESENTATION_VARLENSPARSETENSOR = _descriptor.Descriptor( - name='VarLenSparseTensor', - full_name='tensorflow.metadata.v0.TensorRepresentation.VarLenSparseTensor', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='column_name', full_name='tensorflow.metadata.v0.TensorRepresentation.VarLenSparseTensor.column_name', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=5441, - serialized_end=5482, -) - -_TENSORREPRESENTATION_SPARSETENSOR = _descriptor.Descriptor( - name='SparseTensor', - full_name='tensorflow.metadata.v0.TensorRepresentation.SparseTensor', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='dense_shape', full_name='tensorflow.metadata.v0.TensorRepresentation.SparseTensor.dense_shape', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='index_column_names', full_name='tensorflow.metadata.v0.TensorRepresentation.SparseTensor.index_column_names', index=1, - number=2, type=9, cpp_type=9, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='value_column_name', full_name='tensorflow.metadata.v0.TensorRepresentation.SparseTensor.value_column_name', index=2, - number=3, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=5484, - serialized_end=5610, -) - -_TENSORREPRESENTATION = _descriptor.Descriptor( - name='TensorRepresentation', - full_name='tensorflow.metadata.v0.TensorRepresentation', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='dense_tensor', full_name='tensorflow.metadata.v0.TensorRepresentation.dense_tensor', index=0, - number=1, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='varlen_sparse_tensor', full_name='tensorflow.metadata.v0.TensorRepresentation.varlen_sparse_tensor', index=1, - number=2, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='sparse_tensor', full_name='tensorflow.metadata.v0.TensorRepresentation.sparse_tensor', index=2, - number=3, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[_TENSORREPRESENTATION_DEFAULTVALUE, _TENSORREPRESENTATION_DENSETENSOR, _TENSORREPRESENTATION_VARLENSPARSETENSOR, _TENSORREPRESENTATION_SPARSETENSOR, ], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - _descriptor.OneofDescriptor( - name='kind', full_name='tensorflow.metadata.v0.TensorRepresentation.kind', - index=0, containing_type=None, fields=[]), - ], - serialized_start=4871, - serialized_end=5618, -) - - -_TENSORREPRESENTATIONGROUP_TENSORREPRESENTATIONENTRY = _descriptor.Descriptor( - name='TensorRepresentationEntry', - full_name='tensorflow.metadata.v0.TensorRepresentationGroup.TensorRepresentationEntry', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='key', full_name='tensorflow.metadata.v0.TensorRepresentationGroup.TensorRepresentationEntry.key', index=0, - number=1, type=9, cpp_type=9, label=1, - has_default_value=False, default_value=b"".decode('utf-8'), - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - _descriptor.FieldDescriptor( - name='value', full_name='tensorflow.metadata.v0.TensorRepresentationGroup.TensorRepresentationEntry.value', index=1, - number=2, type=11, cpp_type=10, label=1, - has_default_value=False, default_value=None, - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[], - enum_types=[ - ], - serialized_options=b'8\001', - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=5758, - serialized_end=5863, -) - -_TENSORREPRESENTATIONGROUP = _descriptor.Descriptor( - name='TensorRepresentationGroup', - full_name='tensorflow.metadata.v0.TensorRepresentationGroup', - filename=None, - file=DESCRIPTOR, - containing_type=None, - fields=[ - _descriptor.FieldDescriptor( - name='tensor_representation', full_name='tensorflow.metadata.v0.TensorRepresentationGroup.tensor_representation', index=0, - number=1, type=11, cpp_type=10, label=3, - has_default_value=False, default_value=[], - message_type=None, enum_type=None, containing_type=None, - is_extension=False, extension_scope=None, - serialized_options=None, file=DESCRIPTOR), - ], - extensions=[ - ], - nested_types=[_TENSORREPRESENTATIONGROUP_TENSORREPRESENTATIONENTRY, ], - enum_types=[ - ], - serialized_options=None, - is_extendable=False, - syntax='proto2', - extension_ranges=[], - oneofs=[ - ], - serialized_start=5621, - serialized_end=5863, -) - -_SCHEMA_TENSORREPRESENTATIONGROUPENTRY.fields_by_name['value'].message_type = _TENSORREPRESENTATIONGROUP -_SCHEMA_TENSORREPRESENTATIONGROUPENTRY.containing_type = _SCHEMA -_SCHEMA.fields_by_name['feature'].message_type = _FEATURE -_SCHEMA.fields_by_name['sparse_feature'].message_type = _SPARSEFEATURE -_SCHEMA.fields_by_name['weighted_feature'].message_type = _WEIGHTEDFEATURE -_SCHEMA.fields_by_name['string_domain'].message_type = _STRINGDOMAIN -_SCHEMA.fields_by_name['float_domain'].message_type = _FLOATDOMAIN -_SCHEMA.fields_by_name['int_domain'].message_type = _INTDOMAIN -_SCHEMA.fields_by_name['annotation'].message_type = _ANNOTATION -_SCHEMA.fields_by_name['dataset_constraints'].message_type = _DATASETCONSTRAINTS -_SCHEMA.fields_by_name['tensor_representation_group'].message_type = _SCHEMA_TENSORREPRESENTATIONGROUPENTRY -_FEATURE.fields_by_name['presence'].message_type = _FEATUREPRESENCE -_FEATURE.fields_by_name['group_presence'].message_type = _FEATUREPRESENCEWITHINGROUP -_FEATURE.fields_by_name['shape'].message_type = _FIXEDSHAPE -_FEATURE.fields_by_name['value_count'].message_type = _VALUECOUNT -_FEATURE.fields_by_name['type'].enum_type = _FEATURETYPE -_FEATURE.fields_by_name['int_domain'].message_type = _INTDOMAIN -_FEATURE.fields_by_name['float_domain'].message_type = _FLOATDOMAIN -_FEATURE.fields_by_name['string_domain'].message_type = _STRINGDOMAIN -_FEATURE.fields_by_name['bool_domain'].message_type = _BOOLDOMAIN -_FEATURE.fields_by_name['struct_domain'].message_type = _STRUCTDOMAIN -_FEATURE.fields_by_name['natural_language_domain'].message_type = _NATURALLANGUAGEDOMAIN -_FEATURE.fields_by_name['image_domain'].message_type = _IMAGEDOMAIN -_FEATURE.fields_by_name['mid_domain'].message_type = _MIDDOMAIN -_FEATURE.fields_by_name['url_domain'].message_type = _URLDOMAIN -_FEATURE.fields_by_name['time_domain'].message_type = _TIMEDOMAIN -_FEATURE.fields_by_name['time_of_day_domain'].message_type = _TIMEOFDAYDOMAIN -_FEATURE.fields_by_name['distribution_constraints'].message_type = _DISTRIBUTIONCONSTRAINTS -_FEATURE.fields_by_name['annotation'].message_type = _ANNOTATION -_FEATURE.fields_by_name['skew_comparator'].message_type = _FEATURECOMPARATOR -_FEATURE.fields_by_name['drift_comparator'].message_type = _FEATURECOMPARATOR -_FEATURE.fields_by_name['lifecycle_stage'].enum_type = _LIFECYCLESTAGE -_FEATURE.oneofs_by_name['presence_constraints'].fields.append( - _FEATURE.fields_by_name['presence']) -_FEATURE.fields_by_name['presence'].containing_oneof = _FEATURE.oneofs_by_name['presence_constraints'] -_FEATURE.oneofs_by_name['presence_constraints'].fields.append( - _FEATURE.fields_by_name['group_presence']) -_FEATURE.fields_by_name['group_presence'].containing_oneof = _FEATURE.oneofs_by_name['presence_constraints'] -_FEATURE.oneofs_by_name['shape_type'].fields.append( - _FEATURE.fields_by_name['shape']) -_FEATURE.fields_by_name['shape'].containing_oneof = _FEATURE.oneofs_by_name['shape_type'] -_FEATURE.oneofs_by_name['shape_type'].fields.append( - _FEATURE.fields_by_name['value_count']) -_FEATURE.fields_by_name['value_count'].containing_oneof = _FEATURE.oneofs_by_name['shape_type'] -_FEATURE.oneofs_by_name['domain_info'].fields.append( - _FEATURE.fields_by_name['domain']) -_FEATURE.fields_by_name['domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] -_FEATURE.oneofs_by_name['domain_info'].fields.append( - _FEATURE.fields_by_name['int_domain']) -_FEATURE.fields_by_name['int_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] -_FEATURE.oneofs_by_name['domain_info'].fields.append( - _FEATURE.fields_by_name['float_domain']) -_FEATURE.fields_by_name['float_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] -_FEATURE.oneofs_by_name['domain_info'].fields.append( - _FEATURE.fields_by_name['string_domain']) -_FEATURE.fields_by_name['string_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] -_FEATURE.oneofs_by_name['domain_info'].fields.append( - _FEATURE.fields_by_name['bool_domain']) -_FEATURE.fields_by_name['bool_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] -_FEATURE.oneofs_by_name['domain_info'].fields.append( - _FEATURE.fields_by_name['struct_domain']) -_FEATURE.fields_by_name['struct_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] -_FEATURE.oneofs_by_name['domain_info'].fields.append( - _FEATURE.fields_by_name['natural_language_domain']) -_FEATURE.fields_by_name['natural_language_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] -_FEATURE.oneofs_by_name['domain_info'].fields.append( - _FEATURE.fields_by_name['image_domain']) -_FEATURE.fields_by_name['image_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] -_FEATURE.oneofs_by_name['domain_info'].fields.append( - _FEATURE.fields_by_name['mid_domain']) -_FEATURE.fields_by_name['mid_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] -_FEATURE.oneofs_by_name['domain_info'].fields.append( - _FEATURE.fields_by_name['url_domain']) -_FEATURE.fields_by_name['url_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] -_FEATURE.oneofs_by_name['domain_info'].fields.append( - _FEATURE.fields_by_name['time_domain']) -_FEATURE.fields_by_name['time_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] -_FEATURE.oneofs_by_name['domain_info'].fields.append( - _FEATURE.fields_by_name['time_of_day_domain']) -_FEATURE.fields_by_name['time_of_day_domain'].containing_oneof = _FEATURE.oneofs_by_name['domain_info'] -_ANNOTATION.fields_by_name['extra_metadata'].message_type = google_dot_protobuf_dot_any__pb2._ANY -_DATASETCONSTRAINTS.fields_by_name['num_examples_drift_comparator'].message_type = _NUMERICVALUECOMPARATOR -_DATASETCONSTRAINTS.fields_by_name['num_examples_version_comparator'].message_type = _NUMERICVALUECOMPARATOR -_FIXEDSHAPE_DIM.containing_type = _FIXEDSHAPE -_FIXEDSHAPE.fields_by_name['dim'].message_type = _FIXEDSHAPE_DIM -_WEIGHTEDFEATURE.fields_by_name['feature'].message_type = tensorflow__metadata_dot_proto_dot_v0_dot_path__pb2._PATH -_WEIGHTEDFEATURE.fields_by_name['weight_feature'].message_type = tensorflow__metadata_dot_proto_dot_v0_dot_path__pb2._PATH -_WEIGHTEDFEATURE.fields_by_name['lifecycle_stage'].enum_type = _LIFECYCLESTAGE -_SPARSEFEATURE_INDEXFEATURE.containing_type = _SPARSEFEATURE -_SPARSEFEATURE_VALUEFEATURE.containing_type = _SPARSEFEATURE -_SPARSEFEATURE.fields_by_name['lifecycle_stage'].enum_type = _LIFECYCLESTAGE -_SPARSEFEATURE.fields_by_name['presence'].message_type = _FEATUREPRESENCE -_SPARSEFEATURE.fields_by_name['dense_shape'].message_type = _FIXEDSHAPE -_SPARSEFEATURE.fields_by_name['index_feature'].message_type = _SPARSEFEATURE_INDEXFEATURE -_SPARSEFEATURE.fields_by_name['value_feature'].message_type = _SPARSEFEATURE_VALUEFEATURE -_SPARSEFEATURE.fields_by_name['type'].enum_type = _FEATURETYPE -_STRUCTDOMAIN.fields_by_name['feature'].message_type = _FEATURE -_STRUCTDOMAIN.fields_by_name['sparse_feature'].message_type = _SPARSEFEATURE -_TIMEDOMAIN.fields_by_name['integer_format'].enum_type = _TIMEDOMAIN_INTEGERTIMEFORMAT -_TIMEDOMAIN_INTEGERTIMEFORMAT.containing_type = _TIMEDOMAIN -_TIMEDOMAIN.oneofs_by_name['format'].fields.append( - _TIMEDOMAIN.fields_by_name['string_format']) -_TIMEDOMAIN.fields_by_name['string_format'].containing_oneof = _TIMEDOMAIN.oneofs_by_name['format'] -_TIMEDOMAIN.oneofs_by_name['format'].fields.append( - _TIMEDOMAIN.fields_by_name['integer_format']) -_TIMEDOMAIN.fields_by_name['integer_format'].containing_oneof = _TIMEDOMAIN.oneofs_by_name['format'] -_TIMEOFDAYDOMAIN.fields_by_name['integer_format'].enum_type = _TIMEOFDAYDOMAIN_INTEGERTIMEOFDAYFORMAT -_TIMEOFDAYDOMAIN_INTEGERTIMEOFDAYFORMAT.containing_type = _TIMEOFDAYDOMAIN -_TIMEOFDAYDOMAIN.oneofs_by_name['format'].fields.append( - _TIMEOFDAYDOMAIN.fields_by_name['string_format']) -_TIMEOFDAYDOMAIN.fields_by_name['string_format'].containing_oneof = _TIMEOFDAYDOMAIN.oneofs_by_name['format'] -_TIMEOFDAYDOMAIN.oneofs_by_name['format'].fields.append( - _TIMEOFDAYDOMAIN.fields_by_name['integer_format']) -_TIMEOFDAYDOMAIN.fields_by_name['integer_format'].containing_oneof = _TIMEOFDAYDOMAIN.oneofs_by_name['format'] -_FEATURECOMPARATOR.fields_by_name['infinity_norm'].message_type = _INFINITYNORM -_TENSORREPRESENTATION_DEFAULTVALUE.containing_type = _TENSORREPRESENTATION -_TENSORREPRESENTATION_DEFAULTVALUE.oneofs_by_name['kind'].fields.append( - _TENSORREPRESENTATION_DEFAULTVALUE.fields_by_name['float_value']) -_TENSORREPRESENTATION_DEFAULTVALUE.fields_by_name['float_value'].containing_oneof = _TENSORREPRESENTATION_DEFAULTVALUE.oneofs_by_name['kind'] -_TENSORREPRESENTATION_DEFAULTVALUE.oneofs_by_name['kind'].fields.append( - _TENSORREPRESENTATION_DEFAULTVALUE.fields_by_name['int_value']) -_TENSORREPRESENTATION_DEFAULTVALUE.fields_by_name['int_value'].containing_oneof = _TENSORREPRESENTATION_DEFAULTVALUE.oneofs_by_name['kind'] -_TENSORREPRESENTATION_DEFAULTVALUE.oneofs_by_name['kind'].fields.append( - _TENSORREPRESENTATION_DEFAULTVALUE.fields_by_name['bytes_value']) -_TENSORREPRESENTATION_DEFAULTVALUE.fields_by_name['bytes_value'].containing_oneof = _TENSORREPRESENTATION_DEFAULTVALUE.oneofs_by_name['kind'] -_TENSORREPRESENTATION_DEFAULTVALUE.oneofs_by_name['kind'].fields.append( - _TENSORREPRESENTATION_DEFAULTVALUE.fields_by_name['uint_value']) -_TENSORREPRESENTATION_DEFAULTVALUE.fields_by_name['uint_value'].containing_oneof = _TENSORREPRESENTATION_DEFAULTVALUE.oneofs_by_name['kind'] -_TENSORREPRESENTATION_DENSETENSOR.fields_by_name['shape'].message_type = _FIXEDSHAPE -_TENSORREPRESENTATION_DENSETENSOR.fields_by_name['default_value'].message_type = _TENSORREPRESENTATION_DEFAULTVALUE -_TENSORREPRESENTATION_DENSETENSOR.containing_type = _TENSORREPRESENTATION -_TENSORREPRESENTATION_VARLENSPARSETENSOR.containing_type = _TENSORREPRESENTATION -_TENSORREPRESENTATION_SPARSETENSOR.fields_by_name['dense_shape'].message_type = _FIXEDSHAPE -_TENSORREPRESENTATION_SPARSETENSOR.containing_type = _TENSORREPRESENTATION -_TENSORREPRESENTATION.fields_by_name['dense_tensor'].message_type = _TENSORREPRESENTATION_DENSETENSOR -_TENSORREPRESENTATION.fields_by_name['varlen_sparse_tensor'].message_type = _TENSORREPRESENTATION_VARLENSPARSETENSOR -_TENSORREPRESENTATION.fields_by_name['sparse_tensor'].message_type = _TENSORREPRESENTATION_SPARSETENSOR -_TENSORREPRESENTATION.oneofs_by_name['kind'].fields.append( - _TENSORREPRESENTATION.fields_by_name['dense_tensor']) -_TENSORREPRESENTATION.fields_by_name['dense_tensor'].containing_oneof = _TENSORREPRESENTATION.oneofs_by_name['kind'] -_TENSORREPRESENTATION.oneofs_by_name['kind'].fields.append( - _TENSORREPRESENTATION.fields_by_name['varlen_sparse_tensor']) -_TENSORREPRESENTATION.fields_by_name['varlen_sparse_tensor'].containing_oneof = _TENSORREPRESENTATION.oneofs_by_name['kind'] -_TENSORREPRESENTATION.oneofs_by_name['kind'].fields.append( - _TENSORREPRESENTATION.fields_by_name['sparse_tensor']) -_TENSORREPRESENTATION.fields_by_name['sparse_tensor'].containing_oneof = _TENSORREPRESENTATION.oneofs_by_name['kind'] -_TENSORREPRESENTATIONGROUP_TENSORREPRESENTATIONENTRY.fields_by_name['value'].message_type = _TENSORREPRESENTATION -_TENSORREPRESENTATIONGROUP_TENSORREPRESENTATIONENTRY.containing_type = _TENSORREPRESENTATIONGROUP -_TENSORREPRESENTATIONGROUP.fields_by_name['tensor_representation'].message_type = _TENSORREPRESENTATIONGROUP_TENSORREPRESENTATIONENTRY -DESCRIPTOR.message_types_by_name['Schema'] = _SCHEMA -DESCRIPTOR.message_types_by_name['Feature'] = _FEATURE -DESCRIPTOR.message_types_by_name['Annotation'] = _ANNOTATION -DESCRIPTOR.message_types_by_name['NumericValueComparator'] = _NUMERICVALUECOMPARATOR -DESCRIPTOR.message_types_by_name['DatasetConstraints'] = _DATASETCONSTRAINTS -DESCRIPTOR.message_types_by_name['FixedShape'] = _FIXEDSHAPE -DESCRIPTOR.message_types_by_name['ValueCount'] = _VALUECOUNT -DESCRIPTOR.message_types_by_name['WeightedFeature'] = _WEIGHTEDFEATURE -DESCRIPTOR.message_types_by_name['SparseFeature'] = _SPARSEFEATURE -DESCRIPTOR.message_types_by_name['DistributionConstraints'] = _DISTRIBUTIONCONSTRAINTS -DESCRIPTOR.message_types_by_name['IntDomain'] = _INTDOMAIN -DESCRIPTOR.message_types_by_name['FloatDomain'] = _FLOATDOMAIN -DESCRIPTOR.message_types_by_name['StructDomain'] = _STRUCTDOMAIN -DESCRIPTOR.message_types_by_name['StringDomain'] = _STRINGDOMAIN -DESCRIPTOR.message_types_by_name['BoolDomain'] = _BOOLDOMAIN -DESCRIPTOR.message_types_by_name['NaturalLanguageDomain'] = _NATURALLANGUAGEDOMAIN -DESCRIPTOR.message_types_by_name['ImageDomain'] = _IMAGEDOMAIN -DESCRIPTOR.message_types_by_name['MIDDomain'] = _MIDDOMAIN -DESCRIPTOR.message_types_by_name['URLDomain'] = _URLDOMAIN -DESCRIPTOR.message_types_by_name['TimeDomain'] = _TIMEDOMAIN -DESCRIPTOR.message_types_by_name['TimeOfDayDomain'] = _TIMEOFDAYDOMAIN -DESCRIPTOR.message_types_by_name['FeaturePresence'] = _FEATUREPRESENCE -DESCRIPTOR.message_types_by_name['FeaturePresenceWithinGroup'] = _FEATUREPRESENCEWITHINGROUP -DESCRIPTOR.message_types_by_name['InfinityNorm'] = _INFINITYNORM -DESCRIPTOR.message_types_by_name['FeatureComparator'] = _FEATURECOMPARATOR -DESCRIPTOR.message_types_by_name['TensorRepresentation'] = _TENSORREPRESENTATION -DESCRIPTOR.message_types_by_name['TensorRepresentationGroup'] = _TENSORREPRESENTATIONGROUP -DESCRIPTOR.enum_types_by_name['LifecycleStage'] = _LIFECYCLESTAGE -DESCRIPTOR.enum_types_by_name['FeatureType'] = _FEATURETYPE -_sym_db.RegisterFileDescriptor(DESCRIPTOR) - -Schema = _reflection.GeneratedProtocolMessageType('Schema', (_message.Message,), { - - 'TensorRepresentationGroupEntry' : _reflection.GeneratedProtocolMessageType('TensorRepresentationGroupEntry', (_message.Message,), { - 'DESCRIPTOR' : _SCHEMA_TENSORREPRESENTATIONGROUPENTRY, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.Schema.TensorRepresentationGroupEntry) - }) - , - 'DESCRIPTOR' : _SCHEMA, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.Schema) - }) -_sym_db.RegisterMessage(Schema) -_sym_db.RegisterMessage(Schema.TensorRepresentationGroupEntry) - -Feature = _reflection.GeneratedProtocolMessageType('Feature', (_message.Message,), { - 'DESCRIPTOR' : _FEATURE, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.Feature) - }) -_sym_db.RegisterMessage(Feature) - -Annotation = _reflection.GeneratedProtocolMessageType('Annotation', (_message.Message,), { - 'DESCRIPTOR' : _ANNOTATION, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.Annotation) - }) -_sym_db.RegisterMessage(Annotation) - -NumericValueComparator = _reflection.GeneratedProtocolMessageType('NumericValueComparator', (_message.Message,), { - 'DESCRIPTOR' : _NUMERICVALUECOMPARATOR, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.NumericValueComparator) - }) -_sym_db.RegisterMessage(NumericValueComparator) - -DatasetConstraints = _reflection.GeneratedProtocolMessageType('DatasetConstraints', (_message.Message,), { - 'DESCRIPTOR' : _DATASETCONSTRAINTS, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.DatasetConstraints) - }) -_sym_db.RegisterMessage(DatasetConstraints) - -FixedShape = _reflection.GeneratedProtocolMessageType('FixedShape', (_message.Message,), { - - 'Dim' : _reflection.GeneratedProtocolMessageType('Dim', (_message.Message,), { - 'DESCRIPTOR' : _FIXEDSHAPE_DIM, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.FixedShape.Dim) - }) - , - 'DESCRIPTOR' : _FIXEDSHAPE, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.FixedShape) - }) -_sym_db.RegisterMessage(FixedShape) -_sym_db.RegisterMessage(FixedShape.Dim) - -ValueCount = _reflection.GeneratedProtocolMessageType('ValueCount', (_message.Message,), { - 'DESCRIPTOR' : _VALUECOUNT, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.ValueCount) - }) -_sym_db.RegisterMessage(ValueCount) - -WeightedFeature = _reflection.GeneratedProtocolMessageType('WeightedFeature', (_message.Message,), { - 'DESCRIPTOR' : _WEIGHTEDFEATURE, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.WeightedFeature) - }) -_sym_db.RegisterMessage(WeightedFeature) - -SparseFeature = _reflection.GeneratedProtocolMessageType('SparseFeature', (_message.Message,), { - - 'IndexFeature' : _reflection.GeneratedProtocolMessageType('IndexFeature', (_message.Message,), { - 'DESCRIPTOR' : _SPARSEFEATURE_INDEXFEATURE, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.SparseFeature.IndexFeature) - }) - , - - 'ValueFeature' : _reflection.GeneratedProtocolMessageType('ValueFeature', (_message.Message,), { - 'DESCRIPTOR' : _SPARSEFEATURE_VALUEFEATURE, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.SparseFeature.ValueFeature) - }) - , - 'DESCRIPTOR' : _SPARSEFEATURE, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.SparseFeature) - }) -_sym_db.RegisterMessage(SparseFeature) -_sym_db.RegisterMessage(SparseFeature.IndexFeature) -_sym_db.RegisterMessage(SparseFeature.ValueFeature) - -DistributionConstraints = _reflection.GeneratedProtocolMessageType('DistributionConstraints', (_message.Message,), { - 'DESCRIPTOR' : _DISTRIBUTIONCONSTRAINTS, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.DistributionConstraints) - }) -_sym_db.RegisterMessage(DistributionConstraints) - -IntDomain = _reflection.GeneratedProtocolMessageType('IntDomain', (_message.Message,), { - 'DESCRIPTOR' : _INTDOMAIN, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.IntDomain) - }) -_sym_db.RegisterMessage(IntDomain) - -FloatDomain = _reflection.GeneratedProtocolMessageType('FloatDomain', (_message.Message,), { - 'DESCRIPTOR' : _FLOATDOMAIN, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.FloatDomain) - }) -_sym_db.RegisterMessage(FloatDomain) - -StructDomain = _reflection.GeneratedProtocolMessageType('StructDomain', (_message.Message,), { - 'DESCRIPTOR' : _STRUCTDOMAIN, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.StructDomain) - }) -_sym_db.RegisterMessage(StructDomain) - -StringDomain = _reflection.GeneratedProtocolMessageType('StringDomain', (_message.Message,), { - 'DESCRIPTOR' : _STRINGDOMAIN, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.StringDomain) - }) -_sym_db.RegisterMessage(StringDomain) - -BoolDomain = _reflection.GeneratedProtocolMessageType('BoolDomain', (_message.Message,), { - 'DESCRIPTOR' : _BOOLDOMAIN, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.BoolDomain) - }) -_sym_db.RegisterMessage(BoolDomain) - -NaturalLanguageDomain = _reflection.GeneratedProtocolMessageType('NaturalLanguageDomain', (_message.Message,), { - 'DESCRIPTOR' : _NATURALLANGUAGEDOMAIN, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.NaturalLanguageDomain) - }) -_sym_db.RegisterMessage(NaturalLanguageDomain) - -ImageDomain = _reflection.GeneratedProtocolMessageType('ImageDomain', (_message.Message,), { - 'DESCRIPTOR' : _IMAGEDOMAIN, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.ImageDomain) - }) -_sym_db.RegisterMessage(ImageDomain) - -MIDDomain = _reflection.GeneratedProtocolMessageType('MIDDomain', (_message.Message,), { - 'DESCRIPTOR' : _MIDDOMAIN, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.MIDDomain) - }) -_sym_db.RegisterMessage(MIDDomain) - -URLDomain = _reflection.GeneratedProtocolMessageType('URLDomain', (_message.Message,), { - 'DESCRIPTOR' : _URLDOMAIN, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.URLDomain) - }) -_sym_db.RegisterMessage(URLDomain) - -TimeDomain = _reflection.GeneratedProtocolMessageType('TimeDomain', (_message.Message,), { - 'DESCRIPTOR' : _TIMEDOMAIN, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TimeDomain) - }) -_sym_db.RegisterMessage(TimeDomain) - -TimeOfDayDomain = _reflection.GeneratedProtocolMessageType('TimeOfDayDomain', (_message.Message,), { - 'DESCRIPTOR' : _TIMEOFDAYDOMAIN, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TimeOfDayDomain) - }) -_sym_db.RegisterMessage(TimeOfDayDomain) - -FeaturePresence = _reflection.GeneratedProtocolMessageType('FeaturePresence', (_message.Message,), { - 'DESCRIPTOR' : _FEATUREPRESENCE, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.FeaturePresence) - }) -_sym_db.RegisterMessage(FeaturePresence) - -FeaturePresenceWithinGroup = _reflection.GeneratedProtocolMessageType('FeaturePresenceWithinGroup', (_message.Message,), { - 'DESCRIPTOR' : _FEATUREPRESENCEWITHINGROUP, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.FeaturePresenceWithinGroup) - }) -_sym_db.RegisterMessage(FeaturePresenceWithinGroup) - -InfinityNorm = _reflection.GeneratedProtocolMessageType('InfinityNorm', (_message.Message,), { - 'DESCRIPTOR' : _INFINITYNORM, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.InfinityNorm) - }) -_sym_db.RegisterMessage(InfinityNorm) - -FeatureComparator = _reflection.GeneratedProtocolMessageType('FeatureComparator', (_message.Message,), { - 'DESCRIPTOR' : _FEATURECOMPARATOR, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.FeatureComparator) - }) -_sym_db.RegisterMessage(FeatureComparator) - -TensorRepresentation = _reflection.GeneratedProtocolMessageType('TensorRepresentation', (_message.Message,), { - - 'DefaultValue' : _reflection.GeneratedProtocolMessageType('DefaultValue', (_message.Message,), { - 'DESCRIPTOR' : _TENSORREPRESENTATION_DEFAULTVALUE, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TensorRepresentation.DefaultValue) - }) - , - - 'DenseTensor' : _reflection.GeneratedProtocolMessageType('DenseTensor', (_message.Message,), { - 'DESCRIPTOR' : _TENSORREPRESENTATION_DENSETENSOR, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TensorRepresentation.DenseTensor) - }) - , - - 'VarLenSparseTensor' : _reflection.GeneratedProtocolMessageType('VarLenSparseTensor', (_message.Message,), { - 'DESCRIPTOR' : _TENSORREPRESENTATION_VARLENSPARSETENSOR, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TensorRepresentation.VarLenSparseTensor) - }) - , - - 'SparseTensor' : _reflection.GeneratedProtocolMessageType('SparseTensor', (_message.Message,), { - 'DESCRIPTOR' : _TENSORREPRESENTATION_SPARSETENSOR, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TensorRepresentation.SparseTensor) - }) - , - 'DESCRIPTOR' : _TENSORREPRESENTATION, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TensorRepresentation) - }) -_sym_db.RegisterMessage(TensorRepresentation) -_sym_db.RegisterMessage(TensorRepresentation.DefaultValue) -_sym_db.RegisterMessage(TensorRepresentation.DenseTensor) -_sym_db.RegisterMessage(TensorRepresentation.VarLenSparseTensor) -_sym_db.RegisterMessage(TensorRepresentation.SparseTensor) - -TensorRepresentationGroup = _reflection.GeneratedProtocolMessageType('TensorRepresentationGroup', (_message.Message,), { - - 'TensorRepresentationEntry' : _reflection.GeneratedProtocolMessageType('TensorRepresentationEntry', (_message.Message,), { - 'DESCRIPTOR' : _TENSORREPRESENTATIONGROUP_TENSORREPRESENTATIONENTRY, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TensorRepresentationGroup.TensorRepresentationEntry) - }) - , - 'DESCRIPTOR' : _TENSORREPRESENTATIONGROUP, - '__module__' : 'tensorflow_metadata.proto.v0.schema_pb2' - # @@protoc_insertion_point(class_scope:tensorflow.metadata.v0.TensorRepresentationGroup) - }) -_sym_db.RegisterMessage(TensorRepresentationGroup) -_sym_db.RegisterMessage(TensorRepresentationGroup.TensorRepresentationEntry) - - -DESCRIPTOR._options = None -_SCHEMA_TENSORREPRESENTATIONGROUPENTRY._options = None -_FEATURE.fields_by_name['deprecated']._options = None -_SPARSEFEATURE.fields_by_name['deprecated']._options = None -_SPARSEFEATURE.fields_by_name['presence']._options = None -_SPARSEFEATURE.fields_by_name['type']._options = None -_TENSORREPRESENTATIONGROUP_TENSORREPRESENTATIONENTRY._options = None -# @@protoc_insertion_point(module_scope) diff --git a/sdk/python/tensorflow_metadata/proto/v0/schema_pb2.pyi b/sdk/python/tensorflow_metadata/proto/v0/schema_pb2.pyi deleted file mode 100644 index d684e28c0c2..00000000000 --- a/sdk/python/tensorflow_metadata/proto/v0/schema_pb2.pyi +++ /dev/null @@ -1,1063 +0,0 @@ -# @generated by generate_proto_mypy_stubs.py. Do not edit! -import sys -from google.protobuf.any_pb2 import ( - Any as google___protobuf___any_pb2___Any, -) - -from google.protobuf.descriptor import ( - Descriptor as google___protobuf___descriptor___Descriptor, - EnumDescriptor as google___protobuf___descriptor___EnumDescriptor, -) - -from google.protobuf.internal.containers import ( - RepeatedCompositeFieldContainer as google___protobuf___internal___containers___RepeatedCompositeFieldContainer, - RepeatedScalarFieldContainer as google___protobuf___internal___containers___RepeatedScalarFieldContainer, -) - -from google.protobuf.message import ( - Message as google___protobuf___message___Message, -) - -from tensorflow_metadata.proto.v0.path_pb2 import ( - Path as tensorflow_metadata___proto___v0___path_pb2___Path, -) - -from typing import ( - Iterable as typing___Iterable, - List as typing___List, - Mapping as typing___Mapping, - MutableMapping as typing___MutableMapping, - Optional as typing___Optional, - Text as typing___Text, - Tuple as typing___Tuple, - Union as typing___Union, - cast as typing___cast, - overload as typing___overload, -) - -from typing_extensions import ( - Literal as typing_extensions___Literal, -) - - -builtin___bool = bool -builtin___bytes = bytes -builtin___float = float -builtin___int = int -builtin___str = str -if sys.version_info < (3,): - builtin___buffer = buffer - builtin___unicode = unicode - - -class LifecycleStage(builtin___int): - DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... - @classmethod - def Name(cls, number: builtin___int) -> builtin___str: ... - @classmethod - def Value(cls, name: builtin___str) -> 'LifecycleStage': ... - @classmethod - def keys(cls) -> typing___List[builtin___str]: ... - @classmethod - def values(cls) -> typing___List['LifecycleStage']: ... - @classmethod - def items(cls) -> typing___List[typing___Tuple[builtin___str, 'LifecycleStage']]: ... - UNKNOWN_STAGE = typing___cast('LifecycleStage', 0) - PLANNED = typing___cast('LifecycleStage', 1) - ALPHA = typing___cast('LifecycleStage', 2) - BETA = typing___cast('LifecycleStage', 3) - PRODUCTION = typing___cast('LifecycleStage', 4) - DEPRECATED = typing___cast('LifecycleStage', 5) - DEBUG_ONLY = typing___cast('LifecycleStage', 6) -UNKNOWN_STAGE = typing___cast('LifecycleStage', 0) -PLANNED = typing___cast('LifecycleStage', 1) -ALPHA = typing___cast('LifecycleStage', 2) -BETA = typing___cast('LifecycleStage', 3) -PRODUCTION = typing___cast('LifecycleStage', 4) -DEPRECATED = typing___cast('LifecycleStage', 5) -DEBUG_ONLY = typing___cast('LifecycleStage', 6) - -class FeatureType(builtin___int): - DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... - @classmethod - def Name(cls, number: builtin___int) -> builtin___str: ... - @classmethod - def Value(cls, name: builtin___str) -> 'FeatureType': ... - @classmethod - def keys(cls) -> typing___List[builtin___str]: ... - @classmethod - def values(cls) -> typing___List['FeatureType']: ... - @classmethod - def items(cls) -> typing___List[typing___Tuple[builtin___str, 'FeatureType']]: ... - TYPE_UNKNOWN = typing___cast('FeatureType', 0) - BYTES = typing___cast('FeatureType', 1) - INT = typing___cast('FeatureType', 2) - FLOAT = typing___cast('FeatureType', 3) - STRUCT = typing___cast('FeatureType', 4) -TYPE_UNKNOWN = typing___cast('FeatureType', 0) -BYTES = typing___cast('FeatureType', 1) -INT = typing___cast('FeatureType', 2) -FLOAT = typing___cast('FeatureType', 3) -STRUCT = typing___cast('FeatureType', 4) - -class Schema(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class TensorRepresentationGroupEntry(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - key = ... # type: typing___Text - - @property - def value(self) -> TensorRepresentationGroup: ... - - def __init__(self, - *, - key : typing___Optional[typing___Text] = None, - value : typing___Optional[TensorRepresentationGroup] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> Schema.TensorRepresentationGroupEntry: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> Schema.TensorRepresentationGroupEntry: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"key",b"key",u"value",b"value"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"key",b"key",u"value",b"value"]) -> None: ... - - default_environment = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] - - @property - def feature(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[Feature]: ... - - @property - def sparse_feature(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[SparseFeature]: ... - - @property - def weighted_feature(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[WeightedFeature]: ... - - @property - def string_domain(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[StringDomain]: ... - - @property - def float_domain(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[FloatDomain]: ... - - @property - def int_domain(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[IntDomain]: ... - - @property - def annotation(self) -> Annotation: ... - - @property - def dataset_constraints(self) -> DatasetConstraints: ... - - @property - def tensor_representation_group(self) -> typing___MutableMapping[typing___Text, TensorRepresentationGroup]: ... - - def __init__(self, - *, - feature : typing___Optional[typing___Iterable[Feature]] = None, - sparse_feature : typing___Optional[typing___Iterable[SparseFeature]] = None, - weighted_feature : typing___Optional[typing___Iterable[WeightedFeature]] = None, - string_domain : typing___Optional[typing___Iterable[StringDomain]] = None, - float_domain : typing___Optional[typing___Iterable[FloatDomain]] = None, - int_domain : typing___Optional[typing___Iterable[IntDomain]] = None, - default_environment : typing___Optional[typing___Iterable[typing___Text]] = None, - annotation : typing___Optional[Annotation] = None, - dataset_constraints : typing___Optional[DatasetConstraints] = None, - tensor_representation_group : typing___Optional[typing___Mapping[typing___Text, TensorRepresentationGroup]] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> Schema: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> Schema: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"annotation",b"annotation",u"dataset_constraints",b"dataset_constraints"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"annotation",b"annotation",u"dataset_constraints",b"dataset_constraints",u"default_environment",b"default_environment",u"feature",b"feature",u"float_domain",b"float_domain",u"int_domain",b"int_domain",u"sparse_feature",b"sparse_feature",u"string_domain",b"string_domain",u"tensor_representation_group",b"tensor_representation_group",u"weighted_feature",b"weighted_feature"]) -> None: ... - -class Feature(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - name = ... # type: typing___Text - deprecated = ... # type: builtin___bool - type = ... # type: FeatureType - domain = ... # type: typing___Text - in_environment = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] - not_in_environment = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] - lifecycle_stage = ... # type: LifecycleStage - - @property - def presence(self) -> FeaturePresence: ... - - @property - def group_presence(self) -> FeaturePresenceWithinGroup: ... - - @property - def shape(self) -> FixedShape: ... - - @property - def value_count(self) -> ValueCount: ... - - @property - def int_domain(self) -> IntDomain: ... - - @property - def float_domain(self) -> FloatDomain: ... - - @property - def string_domain(self) -> StringDomain: ... - - @property - def bool_domain(self) -> BoolDomain: ... - - @property - def struct_domain(self) -> StructDomain: ... - - @property - def natural_language_domain(self) -> NaturalLanguageDomain: ... - - @property - def image_domain(self) -> ImageDomain: ... - - @property - def mid_domain(self) -> MIDDomain: ... - - @property - def url_domain(self) -> URLDomain: ... - - @property - def time_domain(self) -> TimeDomain: ... - - @property - def time_of_day_domain(self) -> TimeOfDayDomain: ... - - @property - def distribution_constraints(self) -> DistributionConstraints: ... - - @property - def annotation(self) -> Annotation: ... - - @property - def skew_comparator(self) -> FeatureComparator: ... - - @property - def drift_comparator(self) -> FeatureComparator: ... - - def __init__(self, - *, - name : typing___Optional[typing___Text] = None, - deprecated : typing___Optional[builtin___bool] = None, - presence : typing___Optional[FeaturePresence] = None, - group_presence : typing___Optional[FeaturePresenceWithinGroup] = None, - shape : typing___Optional[FixedShape] = None, - value_count : typing___Optional[ValueCount] = None, - type : typing___Optional[FeatureType] = None, - domain : typing___Optional[typing___Text] = None, - int_domain : typing___Optional[IntDomain] = None, - float_domain : typing___Optional[FloatDomain] = None, - string_domain : typing___Optional[StringDomain] = None, - bool_domain : typing___Optional[BoolDomain] = None, - struct_domain : typing___Optional[StructDomain] = None, - natural_language_domain : typing___Optional[NaturalLanguageDomain] = None, - image_domain : typing___Optional[ImageDomain] = None, - mid_domain : typing___Optional[MIDDomain] = None, - url_domain : typing___Optional[URLDomain] = None, - time_domain : typing___Optional[TimeDomain] = None, - time_of_day_domain : typing___Optional[TimeOfDayDomain] = None, - distribution_constraints : typing___Optional[DistributionConstraints] = None, - annotation : typing___Optional[Annotation] = None, - skew_comparator : typing___Optional[FeatureComparator] = None, - drift_comparator : typing___Optional[FeatureComparator] = None, - in_environment : typing___Optional[typing___Iterable[typing___Text]] = None, - not_in_environment : typing___Optional[typing___Iterable[typing___Text]] = None, - lifecycle_stage : typing___Optional[LifecycleStage] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> Feature: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> Feature: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"annotation",b"annotation",u"bool_domain",b"bool_domain",u"deprecated",b"deprecated",u"distribution_constraints",b"distribution_constraints",u"domain",b"domain",u"domain_info",b"domain_info",u"drift_comparator",b"drift_comparator",u"float_domain",b"float_domain",u"group_presence",b"group_presence",u"image_domain",b"image_domain",u"int_domain",b"int_domain",u"lifecycle_stage",b"lifecycle_stage",u"mid_domain",b"mid_domain",u"name",b"name",u"natural_language_domain",b"natural_language_domain",u"presence",b"presence",u"presence_constraints",b"presence_constraints",u"shape",b"shape",u"shape_type",b"shape_type",u"skew_comparator",b"skew_comparator",u"string_domain",b"string_domain",u"struct_domain",b"struct_domain",u"time_domain",b"time_domain",u"time_of_day_domain",b"time_of_day_domain",u"type",b"type",u"url_domain",b"url_domain",u"value_count",b"value_count"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"annotation",b"annotation",u"bool_domain",b"bool_domain",u"deprecated",b"deprecated",u"distribution_constraints",b"distribution_constraints",u"domain",b"domain",u"domain_info",b"domain_info",u"drift_comparator",b"drift_comparator",u"float_domain",b"float_domain",u"group_presence",b"group_presence",u"image_domain",b"image_domain",u"in_environment",b"in_environment",u"int_domain",b"int_domain",u"lifecycle_stage",b"lifecycle_stage",u"mid_domain",b"mid_domain",u"name",b"name",u"natural_language_domain",b"natural_language_domain",u"not_in_environment",b"not_in_environment",u"presence",b"presence",u"presence_constraints",b"presence_constraints",u"shape",b"shape",u"shape_type",b"shape_type",u"skew_comparator",b"skew_comparator",u"string_domain",b"string_domain",u"struct_domain",b"struct_domain",u"time_domain",b"time_domain",u"time_of_day_domain",b"time_of_day_domain",u"type",b"type",u"url_domain",b"url_domain",u"value_count",b"value_count"]) -> None: ... - @typing___overload - def WhichOneof(self, oneof_group: typing_extensions___Literal[u"domain_info",b"domain_info"]) -> typing_extensions___Literal["domain","int_domain","float_domain","string_domain","bool_domain","struct_domain","natural_language_domain","image_domain","mid_domain","url_domain","time_domain","time_of_day_domain"]: ... - @typing___overload - def WhichOneof(self, oneof_group: typing_extensions___Literal[u"presence_constraints",b"presence_constraints"]) -> typing_extensions___Literal["presence","group_presence"]: ... - @typing___overload - def WhichOneof(self, oneof_group: typing_extensions___Literal[u"shape_type",b"shape_type"]) -> typing_extensions___Literal["shape","value_count"]: ... - -class Annotation(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - tag = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] - comment = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] - - @property - def extra_metadata(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[google___protobuf___any_pb2___Any]: ... - - def __init__(self, - *, - tag : typing___Optional[typing___Iterable[typing___Text]] = None, - comment : typing___Optional[typing___Iterable[typing___Text]] = None, - extra_metadata : typing___Optional[typing___Iterable[google___protobuf___any_pb2___Any]] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> Annotation: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> Annotation: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def ClearField(self, field_name: typing_extensions___Literal[u"comment",b"comment",u"extra_metadata",b"extra_metadata",u"tag",b"tag"]) -> None: ... - -class NumericValueComparator(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - min_fraction_threshold = ... # type: builtin___float - max_fraction_threshold = ... # type: builtin___float - - def __init__(self, - *, - min_fraction_threshold : typing___Optional[builtin___float] = None, - max_fraction_threshold : typing___Optional[builtin___float] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> NumericValueComparator: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> NumericValueComparator: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"max_fraction_threshold",b"max_fraction_threshold",u"min_fraction_threshold",b"min_fraction_threshold"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"max_fraction_threshold",b"max_fraction_threshold",u"min_fraction_threshold",b"min_fraction_threshold"]) -> None: ... - -class DatasetConstraints(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - min_examples_count = ... # type: builtin___int - - @property - def num_examples_drift_comparator(self) -> NumericValueComparator: ... - - @property - def num_examples_version_comparator(self) -> NumericValueComparator: ... - - def __init__(self, - *, - num_examples_drift_comparator : typing___Optional[NumericValueComparator] = None, - num_examples_version_comparator : typing___Optional[NumericValueComparator] = None, - min_examples_count : typing___Optional[builtin___int] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> DatasetConstraints: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> DatasetConstraints: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"min_examples_count",b"min_examples_count",u"num_examples_drift_comparator",b"num_examples_drift_comparator",u"num_examples_version_comparator",b"num_examples_version_comparator"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"min_examples_count",b"min_examples_count",u"num_examples_drift_comparator",b"num_examples_drift_comparator",u"num_examples_version_comparator",b"num_examples_version_comparator"]) -> None: ... - -class FixedShape(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class Dim(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - size = ... # type: builtin___int - name = ... # type: typing___Text - - def __init__(self, - *, - size : typing___Optional[builtin___int] = None, - name : typing___Optional[typing___Text] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> FixedShape.Dim: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> FixedShape.Dim: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"name",b"name",u"size",b"size"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"name",b"name",u"size",b"size"]) -> None: ... - - - @property - def dim(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[FixedShape.Dim]: ... - - def __init__(self, - *, - dim : typing___Optional[typing___Iterable[FixedShape.Dim]] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> FixedShape: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> FixedShape: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def ClearField(self, field_name: typing_extensions___Literal[u"dim",b"dim"]) -> None: ... - -class ValueCount(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - min = ... # type: builtin___int - max = ... # type: builtin___int - - def __init__(self, - *, - min : typing___Optional[builtin___int] = None, - max : typing___Optional[builtin___int] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> ValueCount: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> ValueCount: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"max",b"max",u"min",b"min"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"max",b"max",u"min",b"min"]) -> None: ... - -class WeightedFeature(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - name = ... # type: typing___Text - lifecycle_stage = ... # type: LifecycleStage - - @property - def feature(self) -> tensorflow_metadata___proto___v0___path_pb2___Path: ... - - @property - def weight_feature(self) -> tensorflow_metadata___proto___v0___path_pb2___Path: ... - - def __init__(self, - *, - name : typing___Optional[typing___Text] = None, - feature : typing___Optional[tensorflow_metadata___proto___v0___path_pb2___Path] = None, - weight_feature : typing___Optional[tensorflow_metadata___proto___v0___path_pb2___Path] = None, - lifecycle_stage : typing___Optional[LifecycleStage] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> WeightedFeature: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> WeightedFeature: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"feature",b"feature",u"lifecycle_stage",b"lifecycle_stage",u"name",b"name",u"weight_feature",b"weight_feature"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"feature",b"feature",u"lifecycle_stage",b"lifecycle_stage",u"name",b"name",u"weight_feature",b"weight_feature"]) -> None: ... - -class SparseFeature(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class IndexFeature(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - name = ... # type: typing___Text - - def __init__(self, - *, - name : typing___Optional[typing___Text] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> SparseFeature.IndexFeature: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> SparseFeature.IndexFeature: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"name",b"name"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"name",b"name"]) -> None: ... - - class ValueFeature(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - name = ... # type: typing___Text - - def __init__(self, - *, - name : typing___Optional[typing___Text] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> SparseFeature.ValueFeature: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> SparseFeature.ValueFeature: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"name",b"name"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"name",b"name"]) -> None: ... - - name = ... # type: typing___Text - deprecated = ... # type: builtin___bool - lifecycle_stage = ... # type: LifecycleStage - is_sorted = ... # type: builtin___bool - type = ... # type: FeatureType - - @property - def presence(self) -> FeaturePresence: ... - - @property - def dense_shape(self) -> FixedShape: ... - - @property - def index_feature(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[SparseFeature.IndexFeature]: ... - - @property - def value_feature(self) -> SparseFeature.ValueFeature: ... - - def __init__(self, - *, - name : typing___Optional[typing___Text] = None, - deprecated : typing___Optional[builtin___bool] = None, - lifecycle_stage : typing___Optional[LifecycleStage] = None, - presence : typing___Optional[FeaturePresence] = None, - dense_shape : typing___Optional[FixedShape] = None, - index_feature : typing___Optional[typing___Iterable[SparseFeature.IndexFeature]] = None, - is_sorted : typing___Optional[builtin___bool] = None, - value_feature : typing___Optional[SparseFeature.ValueFeature] = None, - type : typing___Optional[FeatureType] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> SparseFeature: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> SparseFeature: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"dense_shape",b"dense_shape",u"deprecated",b"deprecated",u"is_sorted",b"is_sorted",u"lifecycle_stage",b"lifecycle_stage",u"name",b"name",u"presence",b"presence",u"type",b"type",u"value_feature",b"value_feature"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"dense_shape",b"dense_shape",u"deprecated",b"deprecated",u"index_feature",b"index_feature",u"is_sorted",b"is_sorted",u"lifecycle_stage",b"lifecycle_stage",u"name",b"name",u"presence",b"presence",u"type",b"type",u"value_feature",b"value_feature"]) -> None: ... - -class DistributionConstraints(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - min_domain_mass = ... # type: builtin___float - - def __init__(self, - *, - min_domain_mass : typing___Optional[builtin___float] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> DistributionConstraints: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> DistributionConstraints: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"min_domain_mass",b"min_domain_mass"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"min_domain_mass",b"min_domain_mass"]) -> None: ... - -class IntDomain(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - name = ... # type: typing___Text - min = ... # type: builtin___int - max = ... # type: builtin___int - is_categorical = ... # type: builtin___bool - - def __init__(self, - *, - name : typing___Optional[typing___Text] = None, - min : typing___Optional[builtin___int] = None, - max : typing___Optional[builtin___int] = None, - is_categorical : typing___Optional[builtin___bool] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> IntDomain: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> IntDomain: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"is_categorical",b"is_categorical",u"max",b"max",u"min",b"min",u"name",b"name"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"is_categorical",b"is_categorical",u"max",b"max",u"min",b"min",u"name",b"name"]) -> None: ... - -class FloatDomain(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - name = ... # type: typing___Text - min = ... # type: builtin___float - max = ... # type: builtin___float - - def __init__(self, - *, - name : typing___Optional[typing___Text] = None, - min : typing___Optional[builtin___float] = None, - max : typing___Optional[builtin___float] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> FloatDomain: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> FloatDomain: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"max",b"max",u"min",b"min",u"name",b"name"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"max",b"max",u"min",b"min",u"name",b"name"]) -> None: ... - -class StructDomain(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - @property - def feature(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[Feature]: ... - - @property - def sparse_feature(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[SparseFeature]: ... - - def __init__(self, - *, - feature : typing___Optional[typing___Iterable[Feature]] = None, - sparse_feature : typing___Optional[typing___Iterable[SparseFeature]] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> StructDomain: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> StructDomain: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def ClearField(self, field_name: typing_extensions___Literal[u"feature",b"feature",u"sparse_feature",b"sparse_feature"]) -> None: ... - -class StringDomain(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - name = ... # type: typing___Text - value = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] - - def __init__(self, - *, - name : typing___Optional[typing___Text] = None, - value : typing___Optional[typing___Iterable[typing___Text]] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> StringDomain: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> StringDomain: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"name",b"name"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"name",b"name",u"value",b"value"]) -> None: ... - -class BoolDomain(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - name = ... # type: typing___Text - true_value = ... # type: typing___Text - false_value = ... # type: typing___Text - - def __init__(self, - *, - name : typing___Optional[typing___Text] = None, - true_value : typing___Optional[typing___Text] = None, - false_value : typing___Optional[typing___Text] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> BoolDomain: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> BoolDomain: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"false_value",b"false_value",u"name",b"name",u"true_value",b"true_value"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"false_value",b"false_value",u"name",b"name",u"true_value",b"true_value"]) -> None: ... - -class NaturalLanguageDomain(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - def __init__(self, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> NaturalLanguageDomain: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> NaturalLanguageDomain: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - -class ImageDomain(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - def __init__(self, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> ImageDomain: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> ImageDomain: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - -class MIDDomain(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - def __init__(self, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> MIDDomain: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> MIDDomain: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - -class URLDomain(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - def __init__(self, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> URLDomain: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> URLDomain: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - -class TimeDomain(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class IntegerTimeFormat(builtin___int): - DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... - @classmethod - def Name(cls, number: builtin___int) -> builtin___str: ... - @classmethod - def Value(cls, name: builtin___str) -> 'TimeDomain.IntegerTimeFormat': ... - @classmethod - def keys(cls) -> typing___List[builtin___str]: ... - @classmethod - def values(cls) -> typing___List['TimeDomain.IntegerTimeFormat']: ... - @classmethod - def items(cls) -> typing___List[typing___Tuple[builtin___str, 'TimeDomain.IntegerTimeFormat']]: ... - FORMAT_UNKNOWN = typing___cast('TimeDomain.IntegerTimeFormat', 0) - UNIX_DAYS = typing___cast('TimeDomain.IntegerTimeFormat', 5) - UNIX_SECONDS = typing___cast('TimeDomain.IntegerTimeFormat', 1) - UNIX_MILLISECONDS = typing___cast('TimeDomain.IntegerTimeFormat', 2) - UNIX_MICROSECONDS = typing___cast('TimeDomain.IntegerTimeFormat', 3) - UNIX_NANOSECONDS = typing___cast('TimeDomain.IntegerTimeFormat', 4) - FORMAT_UNKNOWN = typing___cast('TimeDomain.IntegerTimeFormat', 0) - UNIX_DAYS = typing___cast('TimeDomain.IntegerTimeFormat', 5) - UNIX_SECONDS = typing___cast('TimeDomain.IntegerTimeFormat', 1) - UNIX_MILLISECONDS = typing___cast('TimeDomain.IntegerTimeFormat', 2) - UNIX_MICROSECONDS = typing___cast('TimeDomain.IntegerTimeFormat', 3) - UNIX_NANOSECONDS = typing___cast('TimeDomain.IntegerTimeFormat', 4) - - string_format = ... # type: typing___Text - integer_format = ... # type: TimeDomain.IntegerTimeFormat - - def __init__(self, - *, - string_format : typing___Optional[typing___Text] = None, - integer_format : typing___Optional[TimeDomain.IntegerTimeFormat] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> TimeDomain: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TimeDomain: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"format",b"format",u"integer_format",b"integer_format",u"string_format",b"string_format"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"format",b"format",u"integer_format",b"integer_format",u"string_format",b"string_format"]) -> None: ... - def WhichOneof(self, oneof_group: typing_extensions___Literal[u"format",b"format"]) -> typing_extensions___Literal["string_format","integer_format"]: ... - -class TimeOfDayDomain(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class IntegerTimeOfDayFormat(builtin___int): - DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ... - @classmethod - def Name(cls, number: builtin___int) -> builtin___str: ... - @classmethod - def Value(cls, name: builtin___str) -> 'TimeOfDayDomain.IntegerTimeOfDayFormat': ... - @classmethod - def keys(cls) -> typing___List[builtin___str]: ... - @classmethod - def values(cls) -> typing___List['TimeOfDayDomain.IntegerTimeOfDayFormat']: ... - @classmethod - def items(cls) -> typing___List[typing___Tuple[builtin___str, 'TimeOfDayDomain.IntegerTimeOfDayFormat']]: ... - FORMAT_UNKNOWN = typing___cast('TimeOfDayDomain.IntegerTimeOfDayFormat', 0) - PACKED_64_NANOS = typing___cast('TimeOfDayDomain.IntegerTimeOfDayFormat', 1) - FORMAT_UNKNOWN = typing___cast('TimeOfDayDomain.IntegerTimeOfDayFormat', 0) - PACKED_64_NANOS = typing___cast('TimeOfDayDomain.IntegerTimeOfDayFormat', 1) - - string_format = ... # type: typing___Text - integer_format = ... # type: TimeOfDayDomain.IntegerTimeOfDayFormat - - def __init__(self, - *, - string_format : typing___Optional[typing___Text] = None, - integer_format : typing___Optional[TimeOfDayDomain.IntegerTimeOfDayFormat] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> TimeOfDayDomain: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TimeOfDayDomain: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"format",b"format",u"integer_format",b"integer_format",u"string_format",b"string_format"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"format",b"format",u"integer_format",b"integer_format",u"string_format",b"string_format"]) -> None: ... - def WhichOneof(self, oneof_group: typing_extensions___Literal[u"format",b"format"]) -> typing_extensions___Literal["string_format","integer_format"]: ... - -class FeaturePresence(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - min_fraction = ... # type: builtin___float - min_count = ... # type: builtin___int - - def __init__(self, - *, - min_fraction : typing___Optional[builtin___float] = None, - min_count : typing___Optional[builtin___int] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> FeaturePresence: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> FeaturePresence: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"min_count",b"min_count",u"min_fraction",b"min_fraction"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"min_count",b"min_count",u"min_fraction",b"min_fraction"]) -> None: ... - -class FeaturePresenceWithinGroup(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - required = ... # type: builtin___bool - - def __init__(self, - *, - required : typing___Optional[builtin___bool] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> FeaturePresenceWithinGroup: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> FeaturePresenceWithinGroup: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"required",b"required"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"required",b"required"]) -> None: ... - -class InfinityNorm(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - threshold = ... # type: builtin___float - - def __init__(self, - *, - threshold : typing___Optional[builtin___float] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> InfinityNorm: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> InfinityNorm: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"threshold",b"threshold"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"threshold",b"threshold"]) -> None: ... - -class FeatureComparator(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - - @property - def infinity_norm(self) -> InfinityNorm: ... - - def __init__(self, - *, - infinity_norm : typing___Optional[InfinityNorm] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> FeatureComparator: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> FeatureComparator: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"infinity_norm",b"infinity_norm"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"infinity_norm",b"infinity_norm"]) -> None: ... - -class TensorRepresentation(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class DefaultValue(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - float_value = ... # type: builtin___float - int_value = ... # type: builtin___int - bytes_value = ... # type: builtin___bytes - uint_value = ... # type: builtin___int - - def __init__(self, - *, - float_value : typing___Optional[builtin___float] = None, - int_value : typing___Optional[builtin___int] = None, - bytes_value : typing___Optional[builtin___bytes] = None, - uint_value : typing___Optional[builtin___int] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> TensorRepresentation.DefaultValue: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TensorRepresentation.DefaultValue: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"bytes_value",b"bytes_value",u"float_value",b"float_value",u"int_value",b"int_value",u"kind",b"kind",u"uint_value",b"uint_value"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"bytes_value",b"bytes_value",u"float_value",b"float_value",u"int_value",b"int_value",u"kind",b"kind",u"uint_value",b"uint_value"]) -> None: ... - def WhichOneof(self, oneof_group: typing_extensions___Literal[u"kind",b"kind"]) -> typing_extensions___Literal["float_value","int_value","bytes_value","uint_value"]: ... - - class DenseTensor(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - column_name = ... # type: typing___Text - - @property - def shape(self) -> FixedShape: ... - - @property - def default_value(self) -> TensorRepresentation.DefaultValue: ... - - def __init__(self, - *, - column_name : typing___Optional[typing___Text] = None, - shape : typing___Optional[FixedShape] = None, - default_value : typing___Optional[TensorRepresentation.DefaultValue] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> TensorRepresentation.DenseTensor: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TensorRepresentation.DenseTensor: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"column_name",b"column_name",u"default_value",b"default_value",u"shape",b"shape"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"column_name",b"column_name",u"default_value",b"default_value",u"shape",b"shape"]) -> None: ... - - class VarLenSparseTensor(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - column_name = ... # type: typing___Text - - def __init__(self, - *, - column_name : typing___Optional[typing___Text] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> TensorRepresentation.VarLenSparseTensor: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TensorRepresentation.VarLenSparseTensor: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"column_name",b"column_name"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"column_name",b"column_name"]) -> None: ... - - class SparseTensor(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - index_column_names = ... # type: google___protobuf___internal___containers___RepeatedScalarFieldContainer[typing___Text] - value_column_name = ... # type: typing___Text - - @property - def dense_shape(self) -> FixedShape: ... - - def __init__(self, - *, - dense_shape : typing___Optional[FixedShape] = None, - index_column_names : typing___Optional[typing___Iterable[typing___Text]] = None, - value_column_name : typing___Optional[typing___Text] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> TensorRepresentation.SparseTensor: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TensorRepresentation.SparseTensor: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"dense_shape",b"dense_shape",u"value_column_name",b"value_column_name"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"dense_shape",b"dense_shape",u"index_column_names",b"index_column_names",u"value_column_name",b"value_column_name"]) -> None: ... - - - @property - def dense_tensor(self) -> TensorRepresentation.DenseTensor: ... - - @property - def varlen_sparse_tensor(self) -> TensorRepresentation.VarLenSparseTensor: ... - - @property - def sparse_tensor(self) -> TensorRepresentation.SparseTensor: ... - - def __init__(self, - *, - dense_tensor : typing___Optional[TensorRepresentation.DenseTensor] = None, - varlen_sparse_tensor : typing___Optional[TensorRepresentation.VarLenSparseTensor] = None, - sparse_tensor : typing___Optional[TensorRepresentation.SparseTensor] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> TensorRepresentation: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TensorRepresentation: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"dense_tensor",b"dense_tensor",u"kind",b"kind",u"sparse_tensor",b"sparse_tensor",u"varlen_sparse_tensor",b"varlen_sparse_tensor"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"dense_tensor",b"dense_tensor",u"kind",b"kind",u"sparse_tensor",b"sparse_tensor",u"varlen_sparse_tensor",b"varlen_sparse_tensor"]) -> None: ... - def WhichOneof(self, oneof_group: typing_extensions___Literal[u"kind",b"kind"]) -> typing_extensions___Literal["dense_tensor","varlen_sparse_tensor","sparse_tensor"]: ... - -class TensorRepresentationGroup(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - class TensorRepresentationEntry(google___protobuf___message___Message): - DESCRIPTOR: google___protobuf___descriptor___Descriptor = ... - key = ... # type: typing___Text - - @property - def value(self) -> TensorRepresentation: ... - - def __init__(self, - *, - key : typing___Optional[typing___Text] = None, - value : typing___Optional[TensorRepresentation] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> TensorRepresentationGroup.TensorRepresentationEntry: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TensorRepresentationGroup.TensorRepresentationEntry: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def HasField(self, field_name: typing_extensions___Literal[u"key",b"key",u"value",b"value"]) -> builtin___bool: ... - def ClearField(self, field_name: typing_extensions___Literal[u"key",b"key",u"value",b"value"]) -> None: ... - - - @property - def tensor_representation(self) -> typing___MutableMapping[typing___Text, TensorRepresentation]: ... - - def __init__(self, - *, - tensor_representation : typing___Optional[typing___Mapping[typing___Text, TensorRepresentation]] = None, - ) -> None: ... - if sys.version_info >= (3,): - @classmethod - def FromString(cls, s: builtin___bytes) -> TensorRepresentationGroup: ... - else: - @classmethod - def FromString(cls, s: typing___Union[builtin___bytes, builtin___buffer, builtin___unicode]) -> TensorRepresentationGroup: ... - def MergeFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def CopyFrom(self, other_msg: google___protobuf___message___Message) -> None: ... - def ClearField(self, field_name: typing_extensions___Literal[u"tensor_representation",b"tensor_representation"]) -> None: ... diff --git a/sdk/python/tests/data/tensorflow_metadata/bikeshare_feature_set.yaml b/sdk/python/tests/data/tensorflow_metadata/bikeshare_feature_set.yaml new file mode 100644 index 00000000000..daa0a35f0ab --- /dev/null +++ b/sdk/python/tests/data/tensorflow_metadata/bikeshare_feature_set.yaml @@ -0,0 +1,81 @@ +spec: + name: bikeshare + entities: + - name: station_id + valueType: INT64 + intDomain: + min: 1 + max: 5000 + presence: + minFraction: 1.0 + minCount: 1 + shape: + dim: + - size: 1 + features: + - name: location + valueType: STRING + stringDomain: + name: location + value: + - (30.24258, -97.71726) + - (30.24472, -97.72336) + - (30.24891, -97.75019) + presence: + minFraction: 1.0 + minCount: 1 + shape: + dim: + - size: 1 + - name: name + valueType: STRING + stringDomain: + name: name + value: + - 10th & Red River + - 11th & Salina + - 11th & San Jacinto + - 13th & San Antonio + - 17th & Guadalupe + presence: + minFraction: 1.0 + minCount: 1 + shape: + dim: + - size: 1 + - name: status + valueType: STRING + stringDomain: + name: status + value: + - "active" + - "closed" + presence: + minFraction: 1.0 + minCount: 1 + shape: + dim: + - size: 1 + - name: latitude + valueType: DOUBLE + floatDomain: + min: 100.0 + max: 105.0 + presence: + minFraction: 1.0 + minCount: 1 + shape: + dim: + - size: 1 + - name: longitude + valueType: DOUBLE + floatDomain: + min: 102.0 + max: 105.0 + presence: + minFraction: 1.0 + minCount: 1 + shape: + dim: + - size: 1 + maxAge: 3600s diff --git a/sdk/python/tests/data/tensorflow_metadata/bikeshare_schema.json b/sdk/python/tests/data/tensorflow_metadata/bikeshare_schema.json new file mode 100644 index 00000000000..e7a886053c1 --- /dev/null +++ b/sdk/python/tests/data/tensorflow_metadata/bikeshare_schema.json @@ -0,0 +1,136 @@ +{ + "feature": [ + { + "name": "location", + "type": "BYTES", + "domain": "location", + "presence": { + "minFraction": 1.0, + "minCount": "1" + }, + "shape": { + "dim": [ + { + "size": "1" + } + ] + } + }, + { + "name": "name", + "type": "BYTES", + "domain": "name", + "presence": { + "minFraction": 1.0, + "minCount": "1" + }, + "shape": { + "dim": [ + { + "size": "1" + } + ] + } + }, + { + "name": "status", + "type": "BYTES", + "domain": "status", + "presence": { + "minFraction": 1.0, + "minCount": "1" + }, + "shape": { + "dim": [ + { + "size": "1" + } + ] + } + }, + { + "name": "latitude", + "type": "FLOAT", + "float_domain": { + "min": 100.0, + "max": 105.0 + }, + "presence": { + "minFraction": 1.0, + "minCount": "1" + }, + "shape": { + "dim": [ + { + "size": "1" + } + ] + } + }, + { + "name": "longitude", + "type": "FLOAT", + "presence": { + "minFraction": 1.0, + "minCount": "1" + }, + "float_domain": { + "min": 102.0, + "max": 105.0 + }, + "shape": { + "dim": [ + { + "size": "1" + } + ] + } + }, + { + "name": "station_id", + "type": "INT", + "presence": { + "minFraction": 1.0, + "minCount": "1" + }, + "int_domain": { + "min": 1, + "max": 5000 + }, + "shape": { + "dim": [ + { + "size": "1" + } + ] + } + } + ], + "stringDomain": [ + { + "name": "location", + "value": [ + "(30.24258, -97.71726)", + "(30.24472, -97.72336)", + "(30.24891, -97.75019)" + ] + }, + { + "name": "name", + "value": [ + "10th & Red River", + "11th & Salina", + "11th & San Jacinto", + "13th & San Antonio", + "17th & Guadalupe" + ] + }, + { + "name": "status", + "value": [ + "active", + "closed" + ] + } + ] +} \ No newline at end of file diff --git a/sdk/python/tests/test_feature_set.py b/sdk/python/tests/test_feature_set.py index bd31d712bb3..6f087d98bbf 100644 --- a/sdk/python/tests/test_feature_set.py +++ b/sdk/python/tests/test_feature_set.py @@ -11,6 +11,7 @@ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. +import pathlib from concurrent import futures from datetime import datetime @@ -18,12 +19,19 @@ import pandas as pd import pytest import pytz +from google.protobuf import json_format +from tensorflow_metadata.proto.v0 import schema_pb2 import dataframes import feast.core.CoreService_pb2_grpc as Core from feast.client import Client from feast.entity import Entity -from feast.feature_set import Feature, FeatureSet, FeatureSetRef +from feast.feature_set import ( + Feature, + FeatureSet, + FeatureSetRef, + _make_tfx_schema_domain_info_inline, +) from feast.value_type import ValueType from feast_core_server import CoreServicer @@ -168,6 +176,97 @@ def test_add_features_from_df_success( assert len(my_feature_set.features) == feature_count assert len(my_feature_set.entities) == entity_count + def test_import_tfx_schema(self): + tests_folder = pathlib.Path(__file__).parent + test_input_schema_json = open( + tests_folder / "data" / "tensorflow_metadata" / "bikeshare_schema.json" + ).read() + test_input_schema = schema_pb2.Schema() + json_format.Parse(test_input_schema_json, test_input_schema) + + feature_set = FeatureSet( + name="bikeshare", + entities=[Entity(name="station_id", dtype=ValueType.INT64)], + features=[ + Feature(name="name", dtype=ValueType.STRING), + Feature(name="status", dtype=ValueType.STRING), + Feature(name="latitude", dtype=ValueType.FLOAT), + Feature(name="longitude", dtype=ValueType.FLOAT), + Feature(name="location", dtype=ValueType.STRING), + ], + ) + + # Before update + for entity in feature_set.entities: + assert entity.presence is None + assert entity.shape is None + for feature in feature_set.features: + assert feature.presence is None + assert feature.shape is None + assert feature.string_domain is None + assert feature.float_domain is None + assert feature.int_domain is None + + feature_set.import_tfx_schema(test_input_schema) + + # After update + for entity in feature_set.entities: + assert entity.presence is not None + assert entity.shape is not None + for feature in feature_set.features: + assert feature.presence is not None + assert feature.shape is not None + if feature.name in ["location", "name", "status"]: + assert feature.string_domain is not None + elif feature.name in ["latitude", "longitude"]: + assert feature.float_domain is not None + elif feature.name in ["station_id"]: + assert feature.int_domain is not None + + def test_export_tfx_schema(self): + tests_folder = pathlib.Path(__file__).parent + test_input_feature_set = FeatureSet.from_yaml( + str( + tests_folder + / "data" + / "tensorflow_metadata" + / "bikeshare_feature_set.yaml" + ) + ) + + expected_schema_json = open( + tests_folder / "data" / "tensorflow_metadata" / "bikeshare_schema.json" + ).read() + expected_schema = schema_pb2.Schema() + json_format.Parse(expected_schema_json, expected_schema) + _make_tfx_schema_domain_info_inline(expected_schema) + + actual_schema = test_input_feature_set.export_tfx_schema() + + assert len(actual_schema.feature) == len(expected_schema.feature) + for actual, expected in zip(actual_schema.feature, expected_schema.feature): + assert actual.SerializeToString() == expected.SerializeToString() + + +def make_tfx_schema_domain_info_inline(schema): + # Copy top-level domain info defined in the schema to inline definition. + # One use case is in FeatureSet which does not have access to the top-level domain + # info. + domain_ref_to_string_domain = {d.name: d for d in schema.string_domain} + domain_ref_to_float_domain = {d.name: d for d in schema.float_domain} + domain_ref_to_int_domain = {d.name: d for d in schema.int_domain} + + for feature in schema.feature: + domain_info_case = feature.WhichOneof("domain_info") + if domain_info_case == "domain": + domain_ref = feature.domain + if domain_ref in domain_ref_to_string_domain: + feature.string_domain.MergeFrom(domain_ref_to_string_domain[domain_ref]) + elif domain_ref in domain_ref_to_float_domain: + feature.float_domain.MergeFrom(domain_ref_to_float_domain[domain_ref]) + elif domain_ref in domain_ref_to_int_domain: + feature.int_domain.MergeFrom(domain_ref_to_int_domain[domain_ref]) + class TestFeatureSetRef: def test_from_feature_set(self): From 42c3c6e6e32ae873746a8629fc7a95b10b0e8dd6 Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Sun, 12 Apr 2020 16:08:42 +0800 Subject: [PATCH 113/176] Housekeeping commit * Remove format from pre-commit. Formatting should be run manually or during build process instead. Otherwise this causes failed commits because new unstaged files appear. * Bump Feast version to 0.5-SNAPSHOT --- .pre-commit-config.yaml | 5 ----- pom.xml | 2 +- 2 files changed, 1 insertion(+), 6 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 251d67a77e4..f7cf514e580 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -1,11 +1,6 @@ repos: - repo: local hooks: - - id: format - name: Format - stages: [commit] - language: system - entry: make format - id: lint name: Lint stages: [commit] diff --git a/pom.xml b/pom.xml index 649ef01865b..5a9ab5292ab 100644 --- a/pom.xml +++ b/pom.xml @@ -38,7 +38,7 @@ - 0.4.2-SNAPSHOT + 0.5-SNAPSHOT https://github.com/gojek/feast UTF-8 From 6c2cd01a45edb678489bb8a7532b4f7f07e67024 Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Tue, 14 Apr 2020 15:26:47 +0800 Subject: [PATCH 114/176] Regenerate golang code, fix proto comparisons (#616) * Regenerate golang code, fix proto comparisons * Upgrade to protoc v3.10 --- go.mod | 6 +- go.sum | 16 + sdk/go/go.mod | 5 +- sdk/go/go.sum | 20 +- .../protos/feast/serving/ServingService.pb.go | 1758 +++++++++++------ sdk/go/protos/feast/types/FeatureRow.pb.go | 224 ++- .../feast/types/FeatureRowExtended.pb.go | 443 +++-- sdk/go/protos/feast/types/Field.pb.go | 206 +- sdk/go/protos/feast/types/Value.pb.go | 1182 +++++++---- sdk/go/request_test.go | 4 +- sdk/go/response_test.go | 7 +- sdk/go/types.go | 18 +- 12 files changed, 2574 insertions(+), 1315 deletions(-) diff --git a/go.mod b/go.mod index 047f84aa791..15160e57786 100644 --- a/go.mod +++ b/go.mod @@ -8,8 +8,8 @@ require ( github.com/gogo/protobuf v1.3.1 // indirect github.com/gojek/feast/sdk/go v0.0.0-20200316014539-fb893ded90cd // indirect github.com/golang/mock v1.2.0 - github.com/golang/protobuf v1.3.5 - github.com/google/go-cmp v0.3.1 + github.com/golang/protobuf v1.4.0 + github.com/google/go-cmp v0.4.0 github.com/huandu/xstrings v1.2.0 // indirect github.com/lyft/protoc-gen-validate v0.1.0 // indirect github.com/mitchellh/copystructure v1.0.0 // indirect @@ -23,7 +23,7 @@ require ( golang.org/x/lint v0.0.0-20200302205851-738671d3881b // indirect golang.org/x/net v0.0.0-20200320220750-118fecf932d8 golang.org/x/sys v0.0.0-20200321134203-328b4cd54aae // indirect - golang.org/x/tools v0.0.0-20200321224714-0d839f3cf2ed // indirect + golang.org/x/tools v0.0.0-20200414032229-332987a829c3 // indirect google.golang.org/genproto v0.0.0-20200319113533-08878b785e9c // indirect google.golang.org/grpc v1.28.0 gopkg.in/russross/blackfriday.v2 v2.0.0 // indirect diff --git a/go.sum b/go.sum index e6021308eba..1b53b39cf40 100644 --- a/go.sum +++ b/go.sum @@ -142,6 +142,12 @@ github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5y github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= github.com/golang/protobuf v1.3.5 h1:F768QJ1E9tib+q5Sc8MkdJi1RxLTbRcTf8LJV56aRls= github.com/golang/protobuf v1.3.5/go.mod h1:6O5/vntMXwX2lRkT1hjjk0nAC1IDOTvTlVgjlRvqsdk= +github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= +github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= +github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= +github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= +github.com/golang/protobuf v1.4.0 h1:oOuy+ugB+P/kBdUnG5QaMXSIyJ1q38wWSojYCb3z5VQ= +github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0= github.com/google/btree v0.0.0-20160524151835-7d79101e329e/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ= @@ -151,6 +157,7 @@ github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY= github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/go-cmp v0.3.1 h1:Xye71clBPdm5HgqGwUkwhbynsUJZhDbS20FvLhQ2izg= github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/google/gofuzz v0.0.0-20161122191042-44d81051d367/go.mod h1:HP5RmnzzSNb993RKQDq4+1A4ia9nllfqcQFTQJedwGI= github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf/go.mod h1:HP5RmnzzSNb993RKQDq4+1A4ia9nllfqcQFTQJedwGI= github.com/google/gofuzz v1.0.0/go.mod h1:dBl0BpW6vV/+mYPU4Po3pmUjxk6FQPldtuIdl/M65Eg= @@ -318,6 +325,7 @@ github.com/xiang90/probing v0.0.0-20160813154853-07dd2e8dfe18/go.mod h1:UETIi67q github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU= github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q= github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= +github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74= go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU= go.opencensus.io v0.21.0 h1:mU6zScU4U1YAFPHEHYk+3JC4SY7JxgkqS10ZOSyksNg= go.opencensus.io v0.21.0/go.mod h1:mSImk1erAIZhrmZN+AvHh14ztQfjbGwt4TtuofqLduU= @@ -436,6 +444,8 @@ golang.org/x/tools v0.0.0-20200321014904-268ba720d32c h1:Qp5jXmUCqMiVq4676uW7bY2 golang.org/x/tools v0.0.0-20200321014904-268ba720d32c/go.mod h1:Sl4aGygMT6LrqrWclx+PTx3U+LnKx/seiNR+3G19Ar8= golang.org/x/tools v0.0.0-20200321224714-0d839f3cf2ed h1:OCZDlBlLYiUK6T33/8+3BnojrS2W+Dg1rKYJhR89xGE= golang.org/x/tools v0.0.0-20200321224714-0d839f3cf2ed/go.mod h1:Sl4aGygMT6LrqrWclx+PTx3U+LnKx/seiNR+3G19Ar8= +golang.org/x/tools v0.0.0-20200414032229-332987a829c3 h1:Z68UA+HA9shnGhQbAFXKqL1Rk/tfiTHJ57bNm/MUL/A= +golang.org/x/tools v0.0.0-20200414032229-332987a829c3/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= @@ -473,6 +483,12 @@ google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQ google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= google.golang.org/grpc v1.28.0 h1:bO/TA4OxCOummhSf10siHuG7vJOiwh7SpRpFZDkOgl4= google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKal+60= +google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= +google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= +google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= +google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= +google.golang.org/protobuf v1.21.0 h1:qdOKuR/EIArgaWNjetjgTzgVTAZ+S/WXVrq9HW9zimw= +google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= diff --git a/sdk/go/go.mod b/sdk/go/go.mod index c5c76a5c884..58998bc5b54 100644 --- a/sdk/go/go.mod +++ b/sdk/go/go.mod @@ -3,10 +3,11 @@ module github.com/gojek/feast/sdk/go go 1.13 require ( - github.com/golang/protobuf v1.3.3 - github.com/google/go-cmp v0.3.1 + github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0 + github.com/google/go-cmp v0.4.0 github.com/opentracing/opentracing-go v1.1.0 github.com/stretchr/testify v1.4.0 // indirect go.opencensus.io v0.22.1 google.golang.org/grpc v1.28.0 + google.golang.org/protobuf v1.21.0 ) diff --git a/sdk/go/go.sum b/sdk/go/go.sum index c4ff145972e..f08f3026170 100644 --- a/sdk/go/go.sum +++ b/sdk/go/go.sum @@ -6,7 +6,6 @@ github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGX github.com/davecgh/go-spew v1.1.0 h1:ZDRjVQ15GmhC3fiQ8ni8+OwkZQO4DARzQgrnXU1Liz8= github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/envoyproxy/go-control-plane v0.9.0/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= -github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4= github.com/envoyproxy/go-control-plane v0.9.4/go.mod h1:6rpuAdCZL397s3pYoYcLgu1mIlRU8Am5FuJP05cCM98= github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c= github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58= @@ -20,12 +19,19 @@ github.com/golang/protobuf v1.3.2 h1:6nsPYzhq5kReh6QImI3k5qWzO4PEbvbIW2cwSfR/6xs github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U= github.com/golang/protobuf v1.3.3 h1:gyjaxf+svBWX08ZjK86iN9geUJF0H6gp2IRKX6Nf6/I= github.com/golang/protobuf v1.3.3/go.mod h1:vzj43D7+SQXF/4pzW/hwtAqwc6iTitCiVSaWz5lYuqw= +github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8= +github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA= +github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs= +github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0 h1:aRz0NBceriICVtjhCgKkDvl+RudKu1CT6h0ZvUTrNfE= +github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w= github.com/google/go-cmp v0.2.0 h1:+dTQ8DZQJz0Mb/HjFlkptS1FeQ4cWSnN941F8aEG4SQ= github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M= github.com/google/go-cmp v0.3.0 h1:crn/baboCvb5fXaQ0IJ1SGTsTVrWpDsCWC8EGETZijY= github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= github.com/google/go-cmp v0.3.1 h1:Xye71clBPdm5HgqGwUkwhbynsUJZhDbS20FvLhQ2izg= github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU= +github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4= +github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE= github.com/opentracing/opentracing-go v1.1.0 h1:pWlfV3Bxv7k65HYwkikxat0+s3pV4bsqf19k25Ur8rU= github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o= github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= @@ -66,6 +72,8 @@ golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGm golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY= golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs= golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4= +golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM= google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4= google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8 h1:Nw54tB0rB7hY/N0NQvRW8DG4Yk3Q6T9cu9RcFQDu1tc= @@ -77,13 +85,15 @@ google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98 google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c= google.golang.org/grpc v1.20.1/go.mod h1:10oTOabMzJvdu6/UiuZezV6QK5dSlG84ov/aaiqXj38= google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg= -google.golang.org/grpc v1.24.0 h1:vb/1TCsVn3DcJlQ0Gs1yB1pKI6Do2/QNwxdKqmc/b0s= -google.golang.org/grpc v1.24.0/go.mod h1:XDChyiUovWa60DnaeDeZmSW86xtLtjtZbwvSiRnRtcA= google.golang.org/grpc v1.25.1/go.mod h1:c3i+UQWmh7LiEpx4sFZnkU36qjEYZ0imhYfXVyQciAY= -google.golang.org/grpc v1.27.0 h1:rRYRFMVgRv6E0D70Skyfsr28tDXIuuPZyWGMPdMcnXg= -google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk= google.golang.org/grpc v1.28.0 h1:bO/TA4OxCOummhSf10siHuG7vJOiwh7SpRpFZDkOgl4= google.golang.org/grpc v1.28.0/go.mod h1:rpkK4SK4GF4Ach/+MFLZUBavHOvF2JJB5uozKKal+60= +google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8= +google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0= +google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM= +google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE= +google.golang.org/protobuf v1.21.0 h1:qdOKuR/EIArgaWNjetjgTzgVTAZ+S/WXVrq9HW9zimw= +google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw= diff --git a/sdk/go/protos/feast/serving/ServingService.pb.go b/sdk/go/protos/feast/serving/ServingService.pb.go index 1cde2f358dd..6954b1f4f61 100644 --- a/sdk/go/protos/feast/serving/ServingService.pb.go +++ b/sdk/go/protos/feast/serving/ServingService.pb.go @@ -1,11 +1,28 @@ +// +// Copyright 2018 The Feast Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + // Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.21.0 +// protoc v3.10.1 // source: feast/serving/ServingService.proto package serving import ( context "context" - fmt "fmt" types "github.com/gojek/feast/sdk/go/protos/feast/types" proto "github.com/golang/protobuf/proto" duration "github.com/golang/protobuf/ptypes/duration" @@ -13,19 +30,22 @@ import ( grpc "google.golang.org/grpc" codes "google.golang.org/grpc/codes" status "google.golang.org/grpc/status" - math "math" + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" ) -// Reference imports to suppress errors if they are not otherwise used. -var _ = proto.Marshal -var _ = fmt.Errorf -var _ = math.Inf +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) -// This is a compile-time assertion to ensure that this generated file -// is compatible with the proto package it is being compiled against. -// A compilation error at this line likely means your copy of the -// proto package needs to be updated. -const _ = proto.ProtoPackageIsVersion3 // please upgrade the proto package +// This is a compile-time assertion that a sufficiently up-to-date version +// of the legacy proto package is being used. +const _ = proto.ProtoPackageIsVersion4 type FeastServingType int32 @@ -39,24 +59,45 @@ const ( FeastServingType_FEAST_SERVING_TYPE_BATCH FeastServingType = 2 ) -var FeastServingType_name = map[int32]string{ - 0: "FEAST_SERVING_TYPE_INVALID", - 1: "FEAST_SERVING_TYPE_ONLINE", - 2: "FEAST_SERVING_TYPE_BATCH", -} +// Enum value maps for FeastServingType. +var ( + FeastServingType_name = map[int32]string{ + 0: "FEAST_SERVING_TYPE_INVALID", + 1: "FEAST_SERVING_TYPE_ONLINE", + 2: "FEAST_SERVING_TYPE_BATCH", + } + FeastServingType_value = map[string]int32{ + "FEAST_SERVING_TYPE_INVALID": 0, + "FEAST_SERVING_TYPE_ONLINE": 1, + "FEAST_SERVING_TYPE_BATCH": 2, + } +) -var FeastServingType_value = map[string]int32{ - "FEAST_SERVING_TYPE_INVALID": 0, - "FEAST_SERVING_TYPE_ONLINE": 1, - "FEAST_SERVING_TYPE_BATCH": 2, +func (x FeastServingType) Enum() *FeastServingType { + p := new(FeastServingType) + *p = x + return p } func (x FeastServingType) String() string { - return proto.EnumName(FeastServingType_name, int32(x)) + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) +} + +func (FeastServingType) Descriptor() protoreflect.EnumDescriptor { + return file_feast_serving_ServingService_proto_enumTypes[0].Descriptor() +} + +func (FeastServingType) Type() protoreflect.EnumType { + return &file_feast_serving_ServingService_proto_enumTypes[0] +} + +func (x FeastServingType) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) } +// Deprecated: Use FeastServingType.Descriptor instead. func (FeastServingType) EnumDescriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{0} + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{0} } type JobType int32 @@ -66,22 +107,43 @@ const ( JobType_JOB_TYPE_DOWNLOAD JobType = 1 ) -var JobType_name = map[int32]string{ - 0: "JOB_TYPE_INVALID", - 1: "JOB_TYPE_DOWNLOAD", -} +// Enum value maps for JobType. +var ( + JobType_name = map[int32]string{ + 0: "JOB_TYPE_INVALID", + 1: "JOB_TYPE_DOWNLOAD", + } + JobType_value = map[string]int32{ + "JOB_TYPE_INVALID": 0, + "JOB_TYPE_DOWNLOAD": 1, + } +) -var JobType_value = map[string]int32{ - "JOB_TYPE_INVALID": 0, - "JOB_TYPE_DOWNLOAD": 1, +func (x JobType) Enum() *JobType { + p := new(JobType) + *p = x + return p } func (x JobType) String() string { - return proto.EnumName(JobType_name, int32(x)) + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) +} + +func (JobType) Descriptor() protoreflect.EnumDescriptor { + return file_feast_serving_ServingService_proto_enumTypes[1].Descriptor() +} + +func (JobType) Type() protoreflect.EnumType { + return &file_feast_serving_ServingService_proto_enumTypes[1] +} + +func (x JobType) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) } +// Deprecated: Use JobType.Descriptor instead. func (JobType) EnumDescriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{1} + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{1} } type JobStatus int32 @@ -93,26 +155,47 @@ const ( JobStatus_JOB_STATUS_DONE JobStatus = 3 ) -var JobStatus_name = map[int32]string{ - 0: "JOB_STATUS_INVALID", - 1: "JOB_STATUS_PENDING", - 2: "JOB_STATUS_RUNNING", - 3: "JOB_STATUS_DONE", -} +// Enum value maps for JobStatus. +var ( + JobStatus_name = map[int32]string{ + 0: "JOB_STATUS_INVALID", + 1: "JOB_STATUS_PENDING", + 2: "JOB_STATUS_RUNNING", + 3: "JOB_STATUS_DONE", + } + JobStatus_value = map[string]int32{ + "JOB_STATUS_INVALID": 0, + "JOB_STATUS_PENDING": 1, + "JOB_STATUS_RUNNING": 2, + "JOB_STATUS_DONE": 3, + } +) -var JobStatus_value = map[string]int32{ - "JOB_STATUS_INVALID": 0, - "JOB_STATUS_PENDING": 1, - "JOB_STATUS_RUNNING": 2, - "JOB_STATUS_DONE": 3, +func (x JobStatus) Enum() *JobStatus { + p := new(JobStatus) + *p = x + return p } func (x JobStatus) String() string { - return proto.EnumName(JobStatus_name, int32(x)) + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) } +func (JobStatus) Descriptor() protoreflect.EnumDescriptor { + return file_feast_serving_ServingService_proto_enumTypes[2].Descriptor() +} + +func (JobStatus) Type() protoreflect.EnumType { + return &file_feast_serving_ServingService_proto_enumTypes[2] +} + +func (x JobStatus) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) +} + +// Deprecated: Use JobStatus.Descriptor instead. func (JobStatus) EnumDescriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{2} + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{2} } type DataFormat int32 @@ -122,56 +205,88 @@ const ( DataFormat_DATA_FORMAT_AVRO DataFormat = 1 ) -var DataFormat_name = map[int32]string{ - 0: "DATA_FORMAT_INVALID", - 1: "DATA_FORMAT_AVRO", -} +// Enum value maps for DataFormat. +var ( + DataFormat_name = map[int32]string{ + 0: "DATA_FORMAT_INVALID", + 1: "DATA_FORMAT_AVRO", + } + DataFormat_value = map[string]int32{ + "DATA_FORMAT_INVALID": 0, + "DATA_FORMAT_AVRO": 1, + } +) -var DataFormat_value = map[string]int32{ - "DATA_FORMAT_INVALID": 0, - "DATA_FORMAT_AVRO": 1, +func (x DataFormat) Enum() *DataFormat { + p := new(DataFormat) + *p = x + return p } func (x DataFormat) String() string { - return proto.EnumName(DataFormat_name, int32(x)) + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) } -func (DataFormat) EnumDescriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{3} +func (DataFormat) Descriptor() protoreflect.EnumDescriptor { + return file_feast_serving_ServingService_proto_enumTypes[3].Descriptor() } -type GetFeastServingInfoRequest struct { - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` +func (DataFormat) Type() protoreflect.EnumType { + return &file_feast_serving_ServingService_proto_enumTypes[3] } -func (m *GetFeastServingInfoRequest) Reset() { *m = GetFeastServingInfoRequest{} } -func (m *GetFeastServingInfoRequest) String() string { return proto.CompactTextString(m) } -func (*GetFeastServingInfoRequest) ProtoMessage() {} -func (*GetFeastServingInfoRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{0} +func (x DataFormat) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) } -func (m *GetFeastServingInfoRequest) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_GetFeastServingInfoRequest.Unmarshal(m, b) +// Deprecated: Use DataFormat.Descriptor instead. +func (DataFormat) EnumDescriptor() ([]byte, []int) { + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{3} } -func (m *GetFeastServingInfoRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_GetFeastServingInfoRequest.Marshal(b, m, deterministic) + +type GetFeastServingInfoRequest struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields } -func (m *GetFeastServingInfoRequest) XXX_Merge(src proto.Message) { - xxx_messageInfo_GetFeastServingInfoRequest.Merge(m, src) + +func (x *GetFeastServingInfoRequest) Reset() { + *x = GetFeastServingInfoRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_serving_ServingService_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *GetFeastServingInfoRequest) XXX_Size() int { - return xxx_messageInfo_GetFeastServingInfoRequest.Size(m) + +func (x *GetFeastServingInfoRequest) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *GetFeastServingInfoRequest) XXX_DiscardUnknown() { - xxx_messageInfo_GetFeastServingInfoRequest.DiscardUnknown(m) + +func (*GetFeastServingInfoRequest) ProtoMessage() {} + +func (x *GetFeastServingInfoRequest) ProtoReflect() protoreflect.Message { + mi := &file_feast_serving_ServingService_proto_msgTypes[0] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_GetFeastServingInfoRequest proto.InternalMessageInfo +// Deprecated: Use GetFeastServingInfoRequest.ProtoReflect.Descriptor instead. +func (*GetFeastServingInfoRequest) Descriptor() ([]byte, []int) { + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{0} +} type GetFeastServingInfoResponse struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + // Feast version of this serving deployment. Version string `protobuf:"bytes,1,opt,name=version,proto3" json:"version,omitempty"` // Type of serving deployment, either ONLINE or BATCH. Different store types support different @@ -179,59 +294,67 @@ type GetFeastServingInfoResponse struct { Type FeastServingType `protobuf:"varint,2,opt,name=type,proto3,enum=feast.serving.FeastServingType" json:"type,omitempty"` // Note: Batch specific options start from 10. // Staging location for this serving store, if any. - JobStagingLocation string `protobuf:"bytes,10,opt,name=job_staging_location,json=jobStagingLocation,proto3" json:"job_staging_location,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + JobStagingLocation string `protobuf:"bytes,10,opt,name=job_staging_location,json=jobStagingLocation,proto3" json:"job_staging_location,omitempty"` } -func (m *GetFeastServingInfoResponse) Reset() { *m = GetFeastServingInfoResponse{} } -func (m *GetFeastServingInfoResponse) String() string { return proto.CompactTextString(m) } -func (*GetFeastServingInfoResponse) ProtoMessage() {} -func (*GetFeastServingInfoResponse) Descriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{1} +func (x *GetFeastServingInfoResponse) Reset() { + *x = GetFeastServingInfoResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_serving_ServingService_proto_msgTypes[1] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *GetFeastServingInfoResponse) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_GetFeastServingInfoResponse.Unmarshal(m, b) -} -func (m *GetFeastServingInfoResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_GetFeastServingInfoResponse.Marshal(b, m, deterministic) -} -func (m *GetFeastServingInfoResponse) XXX_Merge(src proto.Message) { - xxx_messageInfo_GetFeastServingInfoResponse.Merge(m, src) +func (x *GetFeastServingInfoResponse) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *GetFeastServingInfoResponse) XXX_Size() int { - return xxx_messageInfo_GetFeastServingInfoResponse.Size(m) -} -func (m *GetFeastServingInfoResponse) XXX_DiscardUnknown() { - xxx_messageInfo_GetFeastServingInfoResponse.DiscardUnknown(m) + +func (*GetFeastServingInfoResponse) ProtoMessage() {} + +func (x *GetFeastServingInfoResponse) ProtoReflect() protoreflect.Message { + mi := &file_feast_serving_ServingService_proto_msgTypes[1] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_GetFeastServingInfoResponse proto.InternalMessageInfo +// Deprecated: Use GetFeastServingInfoResponse.ProtoReflect.Descriptor instead. +func (*GetFeastServingInfoResponse) Descriptor() ([]byte, []int) { + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{1} +} -func (m *GetFeastServingInfoResponse) GetVersion() string { - if m != nil { - return m.Version +func (x *GetFeastServingInfoResponse) GetVersion() string { + if x != nil { + return x.Version } return "" } -func (m *GetFeastServingInfoResponse) GetType() FeastServingType { - if m != nil { - return m.Type +func (x *GetFeastServingInfoResponse) GetType() FeastServingType { + if x != nil { + return x.Type } return FeastServingType_FEAST_SERVING_TYPE_INVALID } -func (m *GetFeastServingInfoResponse) GetJobStagingLocation() string { - if m != nil { - return m.JobStagingLocation +func (x *GetFeastServingInfoResponse) GetJobStagingLocation() string { + if x != nil { + return x.JobStagingLocation } return "" } type FeatureReference struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + // Project name Project string `protobuf:"bytes,1,opt,name=project,proto3" json:"project,omitempty"` // Feature name @@ -243,66 +366,74 @@ type FeatureReference struct { // // If unspecified the default max_age specified in FeatureSetSpec will // be used. - MaxAge *duration.Duration `protobuf:"bytes,4,opt,name=max_age,json=maxAge,proto3" json:"max_age,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + MaxAge *duration.Duration `protobuf:"bytes,4,opt,name=max_age,json=maxAge,proto3" json:"max_age,omitempty"` } -func (m *FeatureReference) Reset() { *m = FeatureReference{} } -func (m *FeatureReference) String() string { return proto.CompactTextString(m) } -func (*FeatureReference) ProtoMessage() {} -func (*FeatureReference) Descriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{2} +func (x *FeatureReference) Reset() { + *x = FeatureReference{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_serving_ServingService_proto_msgTypes[2] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *FeatureReference) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_FeatureReference.Unmarshal(m, b) -} -func (m *FeatureReference) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_FeatureReference.Marshal(b, m, deterministic) +func (x *FeatureReference) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *FeatureReference) XXX_Merge(src proto.Message) { - xxx_messageInfo_FeatureReference.Merge(m, src) -} -func (m *FeatureReference) XXX_Size() int { - return xxx_messageInfo_FeatureReference.Size(m) -} -func (m *FeatureReference) XXX_DiscardUnknown() { - xxx_messageInfo_FeatureReference.DiscardUnknown(m) + +func (*FeatureReference) ProtoMessage() {} + +func (x *FeatureReference) ProtoReflect() protoreflect.Message { + mi := &file_feast_serving_ServingService_proto_msgTypes[2] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_FeatureReference proto.InternalMessageInfo +// Deprecated: Use FeatureReference.ProtoReflect.Descriptor instead. +func (*FeatureReference) Descriptor() ([]byte, []int) { + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{2} +} -func (m *FeatureReference) GetProject() string { - if m != nil { - return m.Project +func (x *FeatureReference) GetProject() string { + if x != nil { + return x.Project } return "" } -func (m *FeatureReference) GetName() string { - if m != nil { - return m.Name +func (x *FeatureReference) GetName() string { + if x != nil { + return x.Name } return "" } -func (m *FeatureReference) GetVersion() int32 { - if m != nil { - return m.Version +func (x *FeatureReference) GetVersion() int32 { + if x != nil { + return x.Version } return 0 } -func (m *FeatureReference) GetMaxAge() *duration.Duration { - if m != nil { - return m.MaxAge +func (x *FeatureReference) GetMaxAge() *duration.Duration { + if x != nil { + return x.MaxAge } return nil } type GetOnlineFeaturesRequest struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + // List of features that are being retrieved Features []*FeatureReference `protobuf:"bytes,4,rep,name=features,proto3" json:"features,omitempty"` // List of entity rows, containing entity id and timestamp data. @@ -311,357 +442,314 @@ type GetOnlineFeaturesRequest struct { EntityRows []*GetOnlineFeaturesRequest_EntityRow `protobuf:"bytes,2,rep,name=entity_rows,json=entityRows,proto3" json:"entity_rows,omitempty"` // Option to omit entities from the response. If true, only feature // values will be returned. - OmitEntitiesInResponse bool `protobuf:"varint,3,opt,name=omit_entities_in_response,json=omitEntitiesInResponse,proto3" json:"omit_entities_in_response,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + OmitEntitiesInResponse bool `protobuf:"varint,3,opt,name=omit_entities_in_response,json=omitEntitiesInResponse,proto3" json:"omit_entities_in_response,omitempty"` } -func (m *GetOnlineFeaturesRequest) Reset() { *m = GetOnlineFeaturesRequest{} } -func (m *GetOnlineFeaturesRequest) String() string { return proto.CompactTextString(m) } -func (*GetOnlineFeaturesRequest) ProtoMessage() {} -func (*GetOnlineFeaturesRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{3} +func (x *GetOnlineFeaturesRequest) Reset() { + *x = GetOnlineFeaturesRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_serving_ServingService_proto_msgTypes[3] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *GetOnlineFeaturesRequest) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_GetOnlineFeaturesRequest.Unmarshal(m, b) -} -func (m *GetOnlineFeaturesRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_GetOnlineFeaturesRequest.Marshal(b, m, deterministic) -} -func (m *GetOnlineFeaturesRequest) XXX_Merge(src proto.Message) { - xxx_messageInfo_GetOnlineFeaturesRequest.Merge(m, src) -} -func (m *GetOnlineFeaturesRequest) XXX_Size() int { - return xxx_messageInfo_GetOnlineFeaturesRequest.Size(m) -} -func (m *GetOnlineFeaturesRequest) XXX_DiscardUnknown() { - xxx_messageInfo_GetOnlineFeaturesRequest.DiscardUnknown(m) +func (x *GetOnlineFeaturesRequest) String() string { + return protoimpl.X.MessageStringOf(x) } -var xxx_messageInfo_GetOnlineFeaturesRequest proto.InternalMessageInfo +func (*GetOnlineFeaturesRequest) ProtoMessage() {} -func (m *GetOnlineFeaturesRequest) GetFeatures() []*FeatureReference { - if m != nil { - return m.Features +func (x *GetOnlineFeaturesRequest) ProtoReflect() protoreflect.Message { + mi := &file_feast_serving_ServingService_proto_msgTypes[3] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms } - return nil + return mi.MessageOf(x) } -func (m *GetOnlineFeaturesRequest) GetEntityRows() []*GetOnlineFeaturesRequest_EntityRow { - if m != nil { - return m.EntityRows - } - return nil +// Deprecated: Use GetOnlineFeaturesRequest.ProtoReflect.Descriptor instead. +func (*GetOnlineFeaturesRequest) Descriptor() ([]byte, []int) { + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{3} } -func (m *GetOnlineFeaturesRequest) GetOmitEntitiesInResponse() bool { - if m != nil { - return m.OmitEntitiesInResponse +func (x *GetOnlineFeaturesRequest) GetFeatures() []*FeatureReference { + if x != nil { + return x.Features } - return false -} - -type GetOnlineFeaturesRequest_EntityRow struct { - // Request timestamp of this row. This value will be used, together with maxAge, - // to determine feature staleness. - EntityTimestamp *timestamp.Timestamp `protobuf:"bytes,1,opt,name=entity_timestamp,json=entityTimestamp,proto3" json:"entity_timestamp,omitempty"` - // Map containing mapping of entity name to entity value. - Fields map[string]*types.Value `protobuf:"bytes,2,rep,name=fields,proto3" json:"fields,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *GetOnlineFeaturesRequest_EntityRow) Reset() { *m = GetOnlineFeaturesRequest_EntityRow{} } -func (m *GetOnlineFeaturesRequest_EntityRow) String() string { return proto.CompactTextString(m) } -func (*GetOnlineFeaturesRequest_EntityRow) ProtoMessage() {} -func (*GetOnlineFeaturesRequest_EntityRow) Descriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{3, 0} -} - -func (m *GetOnlineFeaturesRequest_EntityRow) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_GetOnlineFeaturesRequest_EntityRow.Unmarshal(m, b) -} -func (m *GetOnlineFeaturesRequest_EntityRow) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_GetOnlineFeaturesRequest_EntityRow.Marshal(b, m, deterministic) -} -func (m *GetOnlineFeaturesRequest_EntityRow) XXX_Merge(src proto.Message) { - xxx_messageInfo_GetOnlineFeaturesRequest_EntityRow.Merge(m, src) -} -func (m *GetOnlineFeaturesRequest_EntityRow) XXX_Size() int { - return xxx_messageInfo_GetOnlineFeaturesRequest_EntityRow.Size(m) -} -func (m *GetOnlineFeaturesRequest_EntityRow) XXX_DiscardUnknown() { - xxx_messageInfo_GetOnlineFeaturesRequest_EntityRow.DiscardUnknown(m) + return nil } -var xxx_messageInfo_GetOnlineFeaturesRequest_EntityRow proto.InternalMessageInfo - -func (m *GetOnlineFeaturesRequest_EntityRow) GetEntityTimestamp() *timestamp.Timestamp { - if m != nil { - return m.EntityTimestamp +func (x *GetOnlineFeaturesRequest) GetEntityRows() []*GetOnlineFeaturesRequest_EntityRow { + if x != nil { + return x.EntityRows } return nil } -func (m *GetOnlineFeaturesRequest_EntityRow) GetFields() map[string]*types.Value { - if m != nil { - return m.Fields +func (x *GetOnlineFeaturesRequest) GetOmitEntitiesInResponse() bool { + if x != nil { + return x.OmitEntitiesInResponse } - return nil + return false } type GetBatchFeaturesRequest struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + // List of features that are being retrieved Features []*FeatureReference `protobuf:"bytes,3,rep,name=features,proto3" json:"features,omitempty"` // Source of the entity dataset containing the timestamps and entity keys to retrieve // features for. - DatasetSource *DatasetSource `protobuf:"bytes,2,opt,name=dataset_source,json=datasetSource,proto3" json:"dataset_source,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + DatasetSource *DatasetSource `protobuf:"bytes,2,opt,name=dataset_source,json=datasetSource,proto3" json:"dataset_source,omitempty"` } -func (m *GetBatchFeaturesRequest) Reset() { *m = GetBatchFeaturesRequest{} } -func (m *GetBatchFeaturesRequest) String() string { return proto.CompactTextString(m) } -func (*GetBatchFeaturesRequest) ProtoMessage() {} -func (*GetBatchFeaturesRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{4} +func (x *GetBatchFeaturesRequest) Reset() { + *x = GetBatchFeaturesRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_serving_ServingService_proto_msgTypes[4] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *GetBatchFeaturesRequest) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_GetBatchFeaturesRequest.Unmarshal(m, b) -} -func (m *GetBatchFeaturesRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_GetBatchFeaturesRequest.Marshal(b, m, deterministic) +func (x *GetBatchFeaturesRequest) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *GetBatchFeaturesRequest) XXX_Merge(src proto.Message) { - xxx_messageInfo_GetBatchFeaturesRequest.Merge(m, src) -} -func (m *GetBatchFeaturesRequest) XXX_Size() int { - return xxx_messageInfo_GetBatchFeaturesRequest.Size(m) -} -func (m *GetBatchFeaturesRequest) XXX_DiscardUnknown() { - xxx_messageInfo_GetBatchFeaturesRequest.DiscardUnknown(m) + +func (*GetBatchFeaturesRequest) ProtoMessage() {} + +func (x *GetBatchFeaturesRequest) ProtoReflect() protoreflect.Message { + mi := &file_feast_serving_ServingService_proto_msgTypes[4] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_GetBatchFeaturesRequest proto.InternalMessageInfo +// Deprecated: Use GetBatchFeaturesRequest.ProtoReflect.Descriptor instead. +func (*GetBatchFeaturesRequest) Descriptor() ([]byte, []int) { + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{4} +} -func (m *GetBatchFeaturesRequest) GetFeatures() []*FeatureReference { - if m != nil { - return m.Features +func (x *GetBatchFeaturesRequest) GetFeatures() []*FeatureReference { + if x != nil { + return x.Features } return nil } -func (m *GetBatchFeaturesRequest) GetDatasetSource() *DatasetSource { - if m != nil { - return m.DatasetSource +func (x *GetBatchFeaturesRequest) GetDatasetSource() *DatasetSource { + if x != nil { + return x.DatasetSource } return nil } type GetOnlineFeaturesResponse struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + // Feature values retrieved from feast. - FieldValues []*GetOnlineFeaturesResponse_FieldValues `protobuf:"bytes,1,rep,name=field_values,json=fieldValues,proto3" json:"field_values,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + FieldValues []*GetOnlineFeaturesResponse_FieldValues `protobuf:"bytes,1,rep,name=field_values,json=fieldValues,proto3" json:"field_values,omitempty"` } -func (m *GetOnlineFeaturesResponse) Reset() { *m = GetOnlineFeaturesResponse{} } -func (m *GetOnlineFeaturesResponse) String() string { return proto.CompactTextString(m) } -func (*GetOnlineFeaturesResponse) ProtoMessage() {} -func (*GetOnlineFeaturesResponse) Descriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{5} +func (x *GetOnlineFeaturesResponse) Reset() { + *x = GetOnlineFeaturesResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_serving_ServingService_proto_msgTypes[5] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *GetOnlineFeaturesResponse) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_GetOnlineFeaturesResponse.Unmarshal(m, b) -} -func (m *GetOnlineFeaturesResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_GetOnlineFeaturesResponse.Marshal(b, m, deterministic) -} -func (m *GetOnlineFeaturesResponse) XXX_Merge(src proto.Message) { - xxx_messageInfo_GetOnlineFeaturesResponse.Merge(m, src) -} -func (m *GetOnlineFeaturesResponse) XXX_Size() int { - return xxx_messageInfo_GetOnlineFeaturesResponse.Size(m) -} -func (m *GetOnlineFeaturesResponse) XXX_DiscardUnknown() { - xxx_messageInfo_GetOnlineFeaturesResponse.DiscardUnknown(m) +func (x *GetOnlineFeaturesResponse) String() string { + return protoimpl.X.MessageStringOf(x) } -var xxx_messageInfo_GetOnlineFeaturesResponse proto.InternalMessageInfo +func (*GetOnlineFeaturesResponse) ProtoMessage() {} -func (m *GetOnlineFeaturesResponse) GetFieldValues() []*GetOnlineFeaturesResponse_FieldValues { - if m != nil { - return m.FieldValues +func (x *GetOnlineFeaturesResponse) ProtoReflect() protoreflect.Message { + mi := &file_feast_serving_ServingService_proto_msgTypes[5] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms } - return nil + return mi.MessageOf(x) } -type GetOnlineFeaturesResponse_FieldValues struct { - // Map of feature or entity name to feature/entity values. - // Timestamps are not returned in this response. - Fields map[string]*types.Value `protobuf:"bytes,1,rep,name=fields,proto3" json:"fields,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *GetOnlineFeaturesResponse_FieldValues) Reset() { *m = GetOnlineFeaturesResponse_FieldValues{} } -func (m *GetOnlineFeaturesResponse_FieldValues) String() string { return proto.CompactTextString(m) } -func (*GetOnlineFeaturesResponse_FieldValues) ProtoMessage() {} -func (*GetOnlineFeaturesResponse_FieldValues) Descriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{5, 0} -} - -func (m *GetOnlineFeaturesResponse_FieldValues) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_GetOnlineFeaturesResponse_FieldValues.Unmarshal(m, b) -} -func (m *GetOnlineFeaturesResponse_FieldValues) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_GetOnlineFeaturesResponse_FieldValues.Marshal(b, m, deterministic) -} -func (m *GetOnlineFeaturesResponse_FieldValues) XXX_Merge(src proto.Message) { - xxx_messageInfo_GetOnlineFeaturesResponse_FieldValues.Merge(m, src) -} -func (m *GetOnlineFeaturesResponse_FieldValues) XXX_Size() int { - return xxx_messageInfo_GetOnlineFeaturesResponse_FieldValues.Size(m) -} -func (m *GetOnlineFeaturesResponse_FieldValues) XXX_DiscardUnknown() { - xxx_messageInfo_GetOnlineFeaturesResponse_FieldValues.DiscardUnknown(m) +// Deprecated: Use GetOnlineFeaturesResponse.ProtoReflect.Descriptor instead. +func (*GetOnlineFeaturesResponse) Descriptor() ([]byte, []int) { + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{5} } -var xxx_messageInfo_GetOnlineFeaturesResponse_FieldValues proto.InternalMessageInfo - -func (m *GetOnlineFeaturesResponse_FieldValues) GetFields() map[string]*types.Value { - if m != nil { - return m.Fields +func (x *GetOnlineFeaturesResponse) GetFieldValues() []*GetOnlineFeaturesResponse_FieldValues { + if x != nil { + return x.FieldValues } return nil } type GetBatchFeaturesResponse struct { - Job *Job `protobuf:"bytes,1,opt,name=job,proto3" json:"job,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields -func (m *GetBatchFeaturesResponse) Reset() { *m = GetBatchFeaturesResponse{} } -func (m *GetBatchFeaturesResponse) String() string { return proto.CompactTextString(m) } -func (*GetBatchFeaturesResponse) ProtoMessage() {} -func (*GetBatchFeaturesResponse) Descriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{6} + Job *Job `protobuf:"bytes,1,opt,name=job,proto3" json:"job,omitempty"` } -func (m *GetBatchFeaturesResponse) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_GetBatchFeaturesResponse.Unmarshal(m, b) -} -func (m *GetBatchFeaturesResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_GetBatchFeaturesResponse.Marshal(b, m, deterministic) -} -func (m *GetBatchFeaturesResponse) XXX_Merge(src proto.Message) { - xxx_messageInfo_GetBatchFeaturesResponse.Merge(m, src) +func (x *GetBatchFeaturesResponse) Reset() { + *x = GetBatchFeaturesResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_serving_ServingService_proto_msgTypes[6] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *GetBatchFeaturesResponse) XXX_Size() int { - return xxx_messageInfo_GetBatchFeaturesResponse.Size(m) + +func (x *GetBatchFeaturesResponse) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *GetBatchFeaturesResponse) XXX_DiscardUnknown() { - xxx_messageInfo_GetBatchFeaturesResponse.DiscardUnknown(m) + +func (*GetBatchFeaturesResponse) ProtoMessage() {} + +func (x *GetBatchFeaturesResponse) ProtoReflect() protoreflect.Message { + mi := &file_feast_serving_ServingService_proto_msgTypes[6] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_GetBatchFeaturesResponse proto.InternalMessageInfo +// Deprecated: Use GetBatchFeaturesResponse.ProtoReflect.Descriptor instead. +func (*GetBatchFeaturesResponse) Descriptor() ([]byte, []int) { + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{6} +} -func (m *GetBatchFeaturesResponse) GetJob() *Job { - if m != nil { - return m.Job +func (x *GetBatchFeaturesResponse) GetJob() *Job { + if x != nil { + return x.Job } return nil } type GetJobRequest struct { - Job *Job `protobuf:"bytes,1,opt,name=job,proto3" json:"job,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields -func (m *GetJobRequest) Reset() { *m = GetJobRequest{} } -func (m *GetJobRequest) String() string { return proto.CompactTextString(m) } -func (*GetJobRequest) ProtoMessage() {} -func (*GetJobRequest) Descriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{7} + Job *Job `protobuf:"bytes,1,opt,name=job,proto3" json:"job,omitempty"` } -func (m *GetJobRequest) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_GetJobRequest.Unmarshal(m, b) -} -func (m *GetJobRequest) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_GetJobRequest.Marshal(b, m, deterministic) -} -func (m *GetJobRequest) XXX_Merge(src proto.Message) { - xxx_messageInfo_GetJobRequest.Merge(m, src) +func (x *GetJobRequest) Reset() { + *x = GetJobRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_serving_ServingService_proto_msgTypes[7] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *GetJobRequest) XXX_Size() int { - return xxx_messageInfo_GetJobRequest.Size(m) + +func (x *GetJobRequest) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *GetJobRequest) XXX_DiscardUnknown() { - xxx_messageInfo_GetJobRequest.DiscardUnknown(m) + +func (*GetJobRequest) ProtoMessage() {} + +func (x *GetJobRequest) ProtoReflect() protoreflect.Message { + mi := &file_feast_serving_ServingService_proto_msgTypes[7] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_GetJobRequest proto.InternalMessageInfo +// Deprecated: Use GetJobRequest.ProtoReflect.Descriptor instead. +func (*GetJobRequest) Descriptor() ([]byte, []int) { + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{7} +} -func (m *GetJobRequest) GetJob() *Job { - if m != nil { - return m.Job +func (x *GetJobRequest) GetJob() *Job { + if x != nil { + return x.Job } return nil } type GetJobResponse struct { - Job *Job `protobuf:"bytes,1,opt,name=job,proto3" json:"job,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields -func (m *GetJobResponse) Reset() { *m = GetJobResponse{} } -func (m *GetJobResponse) String() string { return proto.CompactTextString(m) } -func (*GetJobResponse) ProtoMessage() {} -func (*GetJobResponse) Descriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{8} + Job *Job `protobuf:"bytes,1,opt,name=job,proto3" json:"job,omitempty"` } -func (m *GetJobResponse) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_GetJobResponse.Unmarshal(m, b) -} -func (m *GetJobResponse) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_GetJobResponse.Marshal(b, m, deterministic) -} -func (m *GetJobResponse) XXX_Merge(src proto.Message) { - xxx_messageInfo_GetJobResponse.Merge(m, src) +func (x *GetJobResponse) Reset() { + *x = GetJobResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_serving_ServingService_proto_msgTypes[8] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *GetJobResponse) XXX_Size() int { - return xxx_messageInfo_GetJobResponse.Size(m) + +func (x *GetJobResponse) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *GetJobResponse) XXX_DiscardUnknown() { - xxx_messageInfo_GetJobResponse.DiscardUnknown(m) + +func (*GetJobResponse) ProtoMessage() {} + +func (x *GetJobResponse) ProtoReflect() protoreflect.Message { + mi := &file_feast_serving_ServingService_proto_msgTypes[8] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_GetJobResponse proto.InternalMessageInfo +// Deprecated: Use GetJobResponse.ProtoReflect.Descriptor instead. +func (*GetJobResponse) Descriptor() ([]byte, []int) { + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{8} +} -func (m *GetJobResponse) GetJob() *Job { - if m != nil { - return m.Job +func (x *GetJobResponse) GetJob() *Job { + if x != nil { + return x.Job } return nil } type Job struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` // Output only. The type of the job. Type JobType `protobuf:"varint,2,opt,name=type,proto3,enum=feast.serving.JobType" json:"type,omitempty"` @@ -674,293 +762,773 @@ type Job struct { FileUris []string `protobuf:"bytes,5,rep,name=file_uris,json=fileUris,proto3" json:"file_uris,omitempty"` // Output only. The data format for all the files. // For CSV format, the files contain both feature values and a column header. - DataFormat DataFormat `protobuf:"varint,6,opt,name=data_format,json=dataFormat,proto3,enum=feast.serving.DataFormat" json:"data_format,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + DataFormat DataFormat `protobuf:"varint,6,opt,name=data_format,json=dataFormat,proto3,enum=feast.serving.DataFormat" json:"data_format,omitempty"` } -func (m *Job) Reset() { *m = Job{} } -func (m *Job) String() string { return proto.CompactTextString(m) } -func (*Job) ProtoMessage() {} -func (*Job) Descriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{9} +func (x *Job) Reset() { + *x = Job{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_serving_ServingService_proto_msgTypes[9] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *Job) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_Job.Unmarshal(m, b) -} -func (m *Job) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_Job.Marshal(b, m, deterministic) +func (x *Job) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *Job) XXX_Merge(src proto.Message) { - xxx_messageInfo_Job.Merge(m, src) -} -func (m *Job) XXX_Size() int { - return xxx_messageInfo_Job.Size(m) -} -func (m *Job) XXX_DiscardUnknown() { - xxx_messageInfo_Job.DiscardUnknown(m) + +func (*Job) ProtoMessage() {} + +func (x *Job) ProtoReflect() protoreflect.Message { + mi := &file_feast_serving_ServingService_proto_msgTypes[9] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_Job proto.InternalMessageInfo +// Deprecated: Use Job.ProtoReflect.Descriptor instead. +func (*Job) Descriptor() ([]byte, []int) { + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{9} +} -func (m *Job) GetId() string { - if m != nil { - return m.Id +func (x *Job) GetId() string { + if x != nil { + return x.Id } return "" } -func (m *Job) GetType() JobType { - if m != nil { - return m.Type +func (x *Job) GetType() JobType { + if x != nil { + return x.Type } return JobType_JOB_TYPE_INVALID } -func (m *Job) GetStatus() JobStatus { - if m != nil { - return m.Status +func (x *Job) GetStatus() JobStatus { + if x != nil { + return x.Status } return JobStatus_JOB_STATUS_INVALID } -func (m *Job) GetError() string { - if m != nil { - return m.Error +func (x *Job) GetError() string { + if x != nil { + return x.Error } return "" } -func (m *Job) GetFileUris() []string { - if m != nil { - return m.FileUris +func (x *Job) GetFileUris() []string { + if x != nil { + return x.FileUris } return nil } -func (m *Job) GetDataFormat() DataFormat { - if m != nil { - return m.DataFormat +func (x *Job) GetDataFormat() DataFormat { + if x != nil { + return x.DataFormat } return DataFormat_DATA_FORMAT_INVALID } type DatasetSource struct { - // Types that are valid to be assigned to DatasetSource: + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Types that are assignable to DatasetSource: // *DatasetSource_FileSource_ - DatasetSource isDatasetSource_DatasetSource `protobuf_oneof:"dataset_source"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + DatasetSource isDatasetSource_DatasetSource `protobuf_oneof:"dataset_source"` } -func (m *DatasetSource) Reset() { *m = DatasetSource{} } -func (m *DatasetSource) String() string { return proto.CompactTextString(m) } -func (*DatasetSource) ProtoMessage() {} -func (*DatasetSource) Descriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{10} +func (x *DatasetSource) Reset() { + *x = DatasetSource{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_serving_ServingService_proto_msgTypes[10] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *DatasetSource) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_DatasetSource.Unmarshal(m, b) +func (x *DatasetSource) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *DatasetSource) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_DatasetSource.Marshal(b, m, deterministic) -} -func (m *DatasetSource) XXX_Merge(src proto.Message) { - xxx_messageInfo_DatasetSource.Merge(m, src) + +func (*DatasetSource) ProtoMessage() {} + +func (x *DatasetSource) ProtoReflect() protoreflect.Message { + mi := &file_feast_serving_ServingService_proto_msgTypes[10] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -func (m *DatasetSource) XXX_Size() int { - return xxx_messageInfo_DatasetSource.Size(m) + +// Deprecated: Use DatasetSource.ProtoReflect.Descriptor instead. +func (*DatasetSource) Descriptor() ([]byte, []int) { + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{10} } -func (m *DatasetSource) XXX_DiscardUnknown() { - xxx_messageInfo_DatasetSource.DiscardUnknown(m) + +func (m *DatasetSource) GetDatasetSource() isDatasetSource_DatasetSource { + if m != nil { + return m.DatasetSource + } + return nil } -var xxx_messageInfo_DatasetSource proto.InternalMessageInfo +func (x *DatasetSource) GetFileSource() *DatasetSource_FileSource { + if x, ok := x.GetDatasetSource().(*DatasetSource_FileSource_); ok { + return x.FileSource + } + return nil +} type isDatasetSource_DatasetSource interface { isDatasetSource_DatasetSource() } type DatasetSource_FileSource_ struct { + // File source to load the dataset from. FileSource *DatasetSource_FileSource `protobuf:"bytes,1,opt,name=file_source,json=fileSource,proto3,oneof"` } func (*DatasetSource_FileSource_) isDatasetSource_DatasetSource() {} -func (m *DatasetSource) GetDatasetSource() isDatasetSource_DatasetSource { - if m != nil { - return m.DatasetSource +type GetOnlineFeaturesRequest_EntityRow struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Request timestamp of this row. This value will be used, together with maxAge, + // to determine feature staleness. + EntityTimestamp *timestamp.Timestamp `protobuf:"bytes,1,opt,name=entity_timestamp,json=entityTimestamp,proto3" json:"entity_timestamp,omitempty"` + // Map containing mapping of entity name to entity value. + Fields map[string]*types.Value `protobuf:"bytes,2,rep,name=fields,proto3" json:"fields,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` +} + +func (x *GetOnlineFeaturesRequest_EntityRow) Reset() { + *x = GetOnlineFeaturesRequest_EntityRow{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_serving_ServingService_proto_msgTypes[11] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *GetOnlineFeaturesRequest_EntityRow) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*GetOnlineFeaturesRequest_EntityRow) ProtoMessage() {} + +func (x *GetOnlineFeaturesRequest_EntityRow) ProtoReflect() protoreflect.Message { + mi := &file_feast_serving_ServingService_proto_msgTypes[11] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use GetOnlineFeaturesRequest_EntityRow.ProtoReflect.Descriptor instead. +func (*GetOnlineFeaturesRequest_EntityRow) Descriptor() ([]byte, []int) { + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{3, 0} +} + +func (x *GetOnlineFeaturesRequest_EntityRow) GetEntityTimestamp() *timestamp.Timestamp { + if x != nil { + return x.EntityTimestamp } return nil } -func (m *DatasetSource) GetFileSource() *DatasetSource_FileSource { - if x, ok := m.GetDatasetSource().(*DatasetSource_FileSource_); ok { - return x.FileSource +func (x *GetOnlineFeaturesRequest_EntityRow) GetFields() map[string]*types.Value { + if x != nil { + return x.Fields } return nil } -// XXX_OneofWrappers is for the internal use of the proto package. -func (*DatasetSource) XXX_OneofWrappers() []interface{} { - return []interface{}{ - (*DatasetSource_FileSource_)(nil), +type GetOnlineFeaturesResponse_FieldValues struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Map of feature or entity name to feature/entity values. + // Timestamps are not returned in this response. + Fields map[string]*types.Value `protobuf:"bytes,1,rep,name=fields,proto3" json:"fields,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` +} + +func (x *GetOnlineFeaturesResponse_FieldValues) Reset() { + *x = GetOnlineFeaturesResponse_FieldValues{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_serving_ServingService_proto_msgTypes[13] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *GetOnlineFeaturesResponse_FieldValues) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*GetOnlineFeaturesResponse_FieldValues) ProtoMessage() {} + +func (x *GetOnlineFeaturesResponse_FieldValues) ProtoReflect() protoreflect.Message { + mi := &file_feast_serving_ServingService_proto_msgTypes[13] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms } + return mi.MessageOf(x) +} + +// Deprecated: Use GetOnlineFeaturesResponse_FieldValues.ProtoReflect.Descriptor instead. +func (*GetOnlineFeaturesResponse_FieldValues) Descriptor() ([]byte, []int) { + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{5, 0} +} + +func (x *GetOnlineFeaturesResponse_FieldValues) GetFields() map[string]*types.Value { + if x != nil { + return x.Fields + } + return nil } type DatasetSource_FileSource struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + // URIs to retrieve the dataset from, e.g. gs://bucket/directory/object.csv. Wildcards are // supported. This data must be compatible to be uploaded to the serving store, and also be // accessible by this serving instance. FileUris []string `protobuf:"bytes,1,rep,name=file_uris,json=fileUris,proto3" json:"file_uris,omitempty"` // Format of the data. Currently only avro is supported. - DataFormat DataFormat `protobuf:"varint,2,opt,name=data_format,json=dataFormat,proto3,enum=feast.serving.DataFormat" json:"data_format,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + DataFormat DataFormat `protobuf:"varint,2,opt,name=data_format,json=dataFormat,proto3,enum=feast.serving.DataFormat" json:"data_format,omitempty"` } -func (m *DatasetSource_FileSource) Reset() { *m = DatasetSource_FileSource{} } -func (m *DatasetSource_FileSource) String() string { return proto.CompactTextString(m) } -func (*DatasetSource_FileSource) ProtoMessage() {} -func (*DatasetSource_FileSource) Descriptor() ([]byte, []int) { - return fileDescriptor_0c1ba93cf29a8d9d, []int{10, 0} +func (x *DatasetSource_FileSource) Reset() { + *x = DatasetSource_FileSource{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_serving_ServingService_proto_msgTypes[15] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *DatasetSource_FileSource) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_DatasetSource_FileSource.Unmarshal(m, b) -} -func (m *DatasetSource_FileSource) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_DatasetSource_FileSource.Marshal(b, m, deterministic) -} -func (m *DatasetSource_FileSource) XXX_Merge(src proto.Message) { - xxx_messageInfo_DatasetSource_FileSource.Merge(m, src) +func (x *DatasetSource_FileSource) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *DatasetSource_FileSource) XXX_Size() int { - return xxx_messageInfo_DatasetSource_FileSource.Size(m) -} -func (m *DatasetSource_FileSource) XXX_DiscardUnknown() { - xxx_messageInfo_DatasetSource_FileSource.DiscardUnknown(m) + +func (*DatasetSource_FileSource) ProtoMessage() {} + +func (x *DatasetSource_FileSource) ProtoReflect() protoreflect.Message { + mi := &file_feast_serving_ServingService_proto_msgTypes[15] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_DatasetSource_FileSource proto.InternalMessageInfo +// Deprecated: Use DatasetSource_FileSource.ProtoReflect.Descriptor instead. +func (*DatasetSource_FileSource) Descriptor() ([]byte, []int) { + return file_feast_serving_ServingService_proto_rawDescGZIP(), []int{10, 0} +} -func (m *DatasetSource_FileSource) GetFileUris() []string { - if m != nil { - return m.FileUris +func (x *DatasetSource_FileSource) GetFileUris() []string { + if x != nil { + return x.FileUris } return nil } -func (m *DatasetSource_FileSource) GetDataFormat() DataFormat { - if m != nil { - return m.DataFormat +func (x *DatasetSource_FileSource) GetDataFormat() DataFormat { + if x != nil { + return x.DataFormat } return DataFormat_DATA_FORMAT_INVALID } -func init() { - proto.RegisterEnum("feast.serving.FeastServingType", FeastServingType_name, FeastServingType_value) - proto.RegisterEnum("feast.serving.JobType", JobType_name, JobType_value) - proto.RegisterEnum("feast.serving.JobStatus", JobStatus_name, JobStatus_value) - proto.RegisterEnum("feast.serving.DataFormat", DataFormat_name, DataFormat_value) - proto.RegisterType((*GetFeastServingInfoRequest)(nil), "feast.serving.GetFeastServingInfoRequest") - proto.RegisterType((*GetFeastServingInfoResponse)(nil), "feast.serving.GetFeastServingInfoResponse") - proto.RegisterType((*FeatureReference)(nil), "feast.serving.FeatureReference") - proto.RegisterType((*GetOnlineFeaturesRequest)(nil), "feast.serving.GetOnlineFeaturesRequest") - proto.RegisterType((*GetOnlineFeaturesRequest_EntityRow)(nil), "feast.serving.GetOnlineFeaturesRequest.EntityRow") - proto.RegisterMapType((map[string]*types.Value)(nil), "feast.serving.GetOnlineFeaturesRequest.EntityRow.FieldsEntry") - proto.RegisterType((*GetBatchFeaturesRequest)(nil), "feast.serving.GetBatchFeaturesRequest") - proto.RegisterType((*GetOnlineFeaturesResponse)(nil), "feast.serving.GetOnlineFeaturesResponse") - proto.RegisterType((*GetOnlineFeaturesResponse_FieldValues)(nil), "feast.serving.GetOnlineFeaturesResponse.FieldValues") - proto.RegisterMapType((map[string]*types.Value)(nil), "feast.serving.GetOnlineFeaturesResponse.FieldValues.FieldsEntry") - proto.RegisterType((*GetBatchFeaturesResponse)(nil), "feast.serving.GetBatchFeaturesResponse") - proto.RegisterType((*GetJobRequest)(nil), "feast.serving.GetJobRequest") - proto.RegisterType((*GetJobResponse)(nil), "feast.serving.GetJobResponse") - proto.RegisterType((*Job)(nil), "feast.serving.Job") - proto.RegisterType((*DatasetSource)(nil), "feast.serving.DatasetSource") - proto.RegisterType((*DatasetSource_FileSource)(nil), "feast.serving.DatasetSource.FileSource") -} - -func init() { - proto.RegisterFile("feast/serving/ServingService.proto", fileDescriptor_0c1ba93cf29a8d9d) -} - -var fileDescriptor_0c1ba93cf29a8d9d = []byte{ - // 1101 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xbc, 0x56, 0x51, 0x73, 0xda, 0xc6, - 0x13, 0x8f, 0xc0, 0xc6, 0x66, 0xf9, 0x1b, 0x2b, 0x67, 0xff, 0x6d, 0x59, 0x71, 0x12, 0x86, 0xe9, - 0xd4, 0x94, 0x07, 0xd1, 0x92, 0x36, 0xd3, 0x26, 0xd3, 0x99, 0x40, 0x10, 0x04, 0x8f, 0x03, 0x9e, - 0x03, 0x3b, 0x6d, 0x5f, 0x34, 0x02, 0x4e, 0x58, 0x36, 0xe8, 0xa8, 0xee, 0x70, 0xe2, 0x2f, 0xd1, - 0x87, 0xbe, 0x76, 0xa6, 0xdf, 0xa0, 0xaf, 0xfd, 0x24, 0xfd, 0x02, 0xed, 0xa7, 0xe8, 0x63, 0x47, - 0xa7, 0x03, 0x23, 0xc0, 0x8e, 0xdd, 0x87, 0x3e, 0xe9, 0x6e, 0xf7, 0xb7, 0xb7, 0xb7, 0xbf, 0xdb, - 0x5d, 0x2d, 0x64, 0x1d, 0x62, 0x33, 0x5e, 0x60, 0xc4, 0xbf, 0x74, 0xbd, 0x7e, 0xa1, 0x15, 0x7e, - 0xc5, 0xa7, 0x4b, 0x8c, 0x91, 0x4f, 0x39, 0x45, 0x1b, 0x02, 0x63, 0x48, 0x8c, 0xfe, 0xb4, 0x4f, - 0x69, 0x7f, 0x40, 0x0a, 0x42, 0xd9, 0x19, 0x3b, 0x05, 0xee, 0x0e, 0x09, 0xe3, 0xf6, 0x70, 0x14, - 0xe2, 0xf5, 0x27, 0xf3, 0x80, 0xde, 0xd8, 0xb7, 0xb9, 0x4b, 0x3d, 0xa9, 0xdf, 0x0d, 0x7d, 0xf2, - 0xab, 0x11, 0x61, 0x85, 0x53, 0x7b, 0x30, 0x96, 0x8e, 0xb2, 0xfb, 0xa0, 0xd7, 0x08, 0xaf, 0x06, - 0x5a, 0x79, 0x91, 0xba, 0xe7, 0x50, 0x4c, 0x7e, 0x1c, 0x13, 0xc6, 0xb3, 0xbf, 0x2a, 0xf0, 0x68, - 0xa9, 0x9a, 0x8d, 0xa8, 0xc7, 0x08, 0xd2, 0x60, 0xed, 0x92, 0xf8, 0xcc, 0xa5, 0x9e, 0xa6, 0x64, - 0x94, 0x5c, 0x12, 0x4f, 0xb6, 0xe8, 0x19, 0xac, 0x04, 0xce, 0xb4, 0x58, 0x46, 0xc9, 0xa5, 0x8b, - 0x4f, 0x8d, 0x48, 0x3c, 0xc6, 0xec, 0x81, 0xed, 0xab, 0x11, 0xc1, 0x02, 0x8c, 0x3e, 0x87, 0xed, - 0x73, 0xda, 0xb1, 0x18, 0xb7, 0xfb, 0xae, 0xd7, 0xb7, 0x06, 0xb4, 0x2b, 0x62, 0xd0, 0x40, 0x9c, - 0x8d, 0xce, 0x69, 0xa7, 0x15, 0xaa, 0x8e, 0xa4, 0x26, 0xfb, 0x93, 0x02, 0x6a, 0x95, 0xd8, 0x7c, - 0xec, 0x13, 0x4c, 0x1c, 0xe2, 0x13, 0xaf, 0x2b, 0x6e, 0x35, 0xf2, 0xe9, 0x39, 0xe9, 0xf2, 0xc9, - 0xad, 0xe4, 0x16, 0x21, 0x58, 0xf1, 0xec, 0x61, 0x78, 0xab, 0x24, 0x16, 0xeb, 0xd9, 0x18, 0xe2, - 0x19, 0x25, 0xb7, 0x7a, 0x1d, 0x43, 0x11, 0xd6, 0x86, 0xf6, 0x07, 0xcb, 0xee, 0x13, 0x6d, 0x25, - 0xa3, 0xe4, 0x52, 0xc5, 0x3d, 0x23, 0xa4, 0xd9, 0x98, 0xd0, 0x6c, 0x54, 0x24, 0xcd, 0x38, 0x31, - 0xb4, 0x3f, 0x94, 0xfa, 0x24, 0xfb, 0x67, 0x1c, 0xb4, 0x1a, 0xe1, 0x4d, 0x6f, 0xe0, 0x7a, 0x44, - 0xde, 0x8c, 0x49, 0x3a, 0xd1, 0x4b, 0x58, 0x77, 0xa4, 0x48, 0x5b, 0xc9, 0xc4, 0x73, 0xa9, 0x65, - 0xc4, 0x44, 0x62, 0xc1, 0x53, 0x03, 0x84, 0x21, 0x45, 0x3c, 0xee, 0xf2, 0x2b, 0xcb, 0xa7, 0xef, - 0x99, 0x16, 0x13, 0xf6, 0x5f, 0xcc, 0xd9, 0xdf, 0xe4, 0xda, 0x30, 0x85, 0x29, 0xa6, 0xef, 0x31, - 0x90, 0xc9, 0x92, 0xa1, 0x6f, 0x60, 0x8f, 0x0e, 0x5d, 0x6e, 0x09, 0x91, 0x4b, 0x98, 0xe5, 0x7a, - 0x96, 0x2f, 0x1f, 0x57, 0xb0, 0xb1, 0x8e, 0x77, 0x02, 0x80, 0x29, 0xf5, 0x75, 0x6f, 0xf2, 0xf4, - 0xfa, 0xdf, 0x0a, 0x24, 0xa7, 0x87, 0x22, 0x13, 0x54, 0x79, 0xb9, 0x69, 0x66, 0x0a, 0xee, 0x53, - 0x45, 0x7d, 0x81, 0xb3, 0xf6, 0x04, 0x81, 0x37, 0x43, 0x9b, 0xa9, 0x00, 0x9d, 0x40, 0xc2, 0x71, - 0xc9, 0xa0, 0x37, 0x09, 0xef, 0xdb, 0x7b, 0x87, 0x67, 0x54, 0x85, 0xbd, 0xe9, 0x71, 0xff, 0x0a, - 0xcb, 0xc3, 0xf4, 0xb7, 0x90, 0x9a, 0x11, 0x23, 0x15, 0xe2, 0x17, 0xe4, 0x4a, 0xe6, 0x46, 0xb0, - 0x44, 0x39, 0x58, 0xbd, 0x0c, 0x8a, 0x42, 0x24, 0x46, 0xaa, 0x88, 0xa4, 0x5b, 0x51, 0x2e, 0x86, - 0x28, 0x17, 0x1c, 0x02, 0x5e, 0xc4, 0xbe, 0x56, 0xb2, 0xbf, 0x28, 0xb0, 0x5b, 0x23, 0xbc, 0x6c, - 0xf3, 0xee, 0xd9, 0x6d, 0x4f, 0x1c, 0xbf, 0xef, 0x13, 0xbf, 0x86, 0x74, 0xcf, 0xe6, 0x36, 0x23, - 0xdc, 0x62, 0x74, 0xec, 0x77, 0x27, 0xf7, 0xd9, 0x9f, 0x3b, 0xa2, 0x12, 0x82, 0x5a, 0x02, 0x83, - 0x37, 0x7a, 0xb3, 0xdb, 0xec, 0x6f, 0x31, 0xd8, 0x5b, 0xc2, 0x93, 0xac, 0xd8, 0x77, 0xf0, 0x3f, - 0x41, 0x8a, 0x25, 0xc2, 0x61, 0x9a, 0x22, 0xee, 0xf8, 0xe5, 0xc7, 0x79, 0x0e, 0xed, 0x43, 0x7a, - 0x05, 0x23, 0x0c, 0xa7, 0x9c, 0xeb, 0x8d, 0xfe, 0xbb, 0x22, 0x49, 0x0e, 0xf7, 0xe8, 0xbb, 0xe9, - 0x53, 0x86, 0x2e, 0x5e, 0xfd, 0x1b, 0x17, 0xff, 0xc5, 0x6b, 0xbe, 0x12, 0x05, 0x3b, 0xf7, 0x98, - 0x92, 0xad, 0x4f, 0x20, 0x7e, 0x4e, 0x3b, 0x32, 0x93, 0xd1, 0x5c, 0x04, 0x87, 0xb4, 0x83, 0x03, - 0x75, 0xf6, 0x2b, 0xd8, 0xa8, 0x11, 0x1e, 0x6c, 0x65, 0x12, 0xdc, 0xcd, 0xec, 0x39, 0xa4, 0x27, - 0x66, 0xf7, 0x72, 0xf7, 0x97, 0x02, 0xf1, 0x43, 0xda, 0x41, 0x69, 0x88, 0xb9, 0x3d, 0x19, 0x77, - 0xcc, 0xed, 0xa1, 0x7c, 0xa4, 0xe5, 0xee, 0x2c, 0x9a, 0x47, 0x3a, 0x6d, 0x82, 0x71, 0x9b, 0x8f, - 0x99, 0xa8, 0xf2, 0x74, 0x51, 0x5b, 0x44, 0xb7, 0x84, 0x1e, 0x4b, 0x1c, 0xda, 0x86, 0x55, 0xe2, - 0xfb, 0xd4, 0x17, 0xad, 0x30, 0x89, 0xc3, 0x0d, 0x7a, 0x04, 0x49, 0xc7, 0x1d, 0x10, 0x6b, 0xec, - 0xbb, 0x4c, 0x5b, 0xcd, 0xc4, 0x73, 0x49, 0xbc, 0x1e, 0x08, 0x4e, 0x7c, 0x97, 0xa1, 0x17, 0x90, - 0x0a, 0x52, 0xd3, 0x72, 0xa8, 0x3f, 0xb4, 0xb9, 0x96, 0x10, 0x9e, 0xf6, 0x96, 0xe4, 0x72, 0x55, - 0x00, 0x30, 0xf4, 0xa6, 0xeb, 0xec, 0x1f, 0x0a, 0x6c, 0x44, 0xd2, 0x1c, 0x1d, 0x42, 0x4a, 0xb8, - 0x92, 0x95, 0x11, 0x92, 0x74, 0x70, 0x5b, 0x65, 0x18, 0x55, 0x77, 0x40, 0xc2, 0xe5, 0x9b, 0x07, - 0x18, 0x9c, 0xe9, 0x4e, 0x27, 0x00, 0xd7, 0xba, 0x68, 0x10, 0xca, 0xed, 0x41, 0xc4, 0xee, 0x11, - 0x44, 0x59, 0x9d, 0xaf, 0xe7, 0x3c, 0x15, 0xbf, 0xab, 0xc8, 0xbf, 0x0f, 0x3d, 0x01, 0xbd, 0x6a, - 0x96, 0x5a, 0x6d, 0xab, 0x65, 0xe2, 0xd3, 0x7a, 0xa3, 0x66, 0xb5, 0xbf, 0x3f, 0x36, 0xad, 0x7a, - 0xe3, 0xb4, 0x74, 0x54, 0xaf, 0xa8, 0x0f, 0xd0, 0x63, 0xd8, 0x5b, 0xa2, 0x6f, 0x36, 0x8e, 0xea, - 0x0d, 0x53, 0x55, 0xd0, 0x3e, 0x68, 0x4b, 0xd4, 0xe5, 0x52, 0xfb, 0xf5, 0x1b, 0x35, 0x96, 0x7f, - 0x0e, 0x6b, 0xf2, 0xe5, 0xd1, 0x36, 0xa8, 0x87, 0xcd, 0xf2, 0xfc, 0xe9, 0xff, 0x87, 0x87, 0x53, - 0x69, 0xa5, 0xf9, 0xae, 0x71, 0xd4, 0x2c, 0x55, 0x54, 0x25, 0x7f, 0x06, 0xc9, 0x69, 0x0e, 0xa0, - 0x1d, 0x40, 0x01, 0xa6, 0xd5, 0x2e, 0xb5, 0x4f, 0x5a, 0x33, 0xb6, 0x51, 0xf9, 0xb1, 0xd9, 0xa8, - 0xd4, 0x1b, 0x35, 0x55, 0x99, 0x93, 0xe3, 0x93, 0x46, 0x23, 0x90, 0xc7, 0xd0, 0x16, 0x6c, 0xce, - 0xc8, 0x2b, 0xcd, 0x86, 0xa9, 0xc6, 0xf3, 0x2f, 0x01, 0xae, 0xe9, 0x43, 0xbb, 0xb0, 0x55, 0x29, - 0xb5, 0x4b, 0x56, 0xb5, 0x89, 0xdf, 0x96, 0xda, 0x33, 0xbe, 0xb6, 0x41, 0x9d, 0x55, 0x94, 0x4e, - 0x71, 0x53, 0x55, 0x8a, 0x3f, 0xc7, 0x21, 0x1d, 0x1d, 0xa0, 0xd0, 0x00, 0xb6, 0x96, 0x8c, 0x2c, - 0xe8, 0xb3, 0xc5, 0xfe, 0x73, 0xc3, 0xd4, 0xa3, 0xe7, 0xef, 0x02, 0x95, 0x25, 0xeb, 0xc0, 0xc3, - 0x85, 0x4e, 0x86, 0x0e, 0xee, 0xf8, 0xdb, 0xd2, 0x73, 0x77, 0x6d, 0x8a, 0xa8, 0x0b, 0xea, 0x7c, - 0x97, 0x42, 0x9f, 0x2e, 0x5a, 0x2f, 0xfb, 0x27, 0xe9, 0x07, 0x1f, 0xc5, 0x49, 0x27, 0x26, 0x24, - 0xc2, 0x8e, 0x84, 0xf6, 0x17, 0x4d, 0xae, 0xfb, 0x9b, 0xfe, 0xf8, 0x06, 0x6d, 0x78, 0x4c, 0xb9, - 0x0d, 0xd1, 0xf1, 0xb5, 0xbc, 0x29, 0x99, 0x2b, 0x1d, 0xd7, 0x8f, 0x83, 0x29, 0xe0, 0x87, 0x62, - 0xdf, 0xe5, 0x67, 0xe3, 0x8e, 0xd1, 0xa5, 0xc3, 0x42, 0x9f, 0x9e, 0x93, 0x8b, 0x82, 0x9c, 0x89, - 0x7b, 0x17, 0x85, 0x3e, 0x0d, 0xa7, 0x58, 0x56, 0x88, 0xcc, 0xc9, 0x9d, 0x84, 0x90, 0x3e, 0xfb, - 0x27, 0x00, 0x00, 0xff, 0xff, 0x36, 0xb8, 0x08, 0x14, 0x3f, 0x0b, 0x00, 0x00, +var File_feast_serving_ServingService_proto protoreflect.FileDescriptor + +var file_feast_serving_ServingService_proto_rawDesc = []byte{ + 0x0a, 0x22, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2f, + 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x2e, 0x70, + 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0d, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, + 0x69, 0x6e, 0x67, 0x1a, 0x1f, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74, + 0x6f, 0x62, 0x75, 0x66, 0x2f, 0x74, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x2e, 0x70, + 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x1e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, + 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2f, 0x64, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2e, 0x70, + 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x17, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x74, 0x79, 0x70, 0x65, + 0x73, 0x2f, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x1c, 0x0a, + 0x1a, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, + 0x49, 0x6e, 0x66, 0x6f, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x22, 0x9e, 0x01, 0x0a, 0x1b, + 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x49, + 0x6e, 0x66, 0x6f, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, + 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x76, 0x65, + 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x12, 0x33, 0x0a, 0x04, 0x74, 0x79, 0x70, 0x65, 0x18, 0x02, 0x20, + 0x01, 0x28, 0x0e, 0x32, 0x1f, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, + 0x69, 0x6e, 0x67, 0x2e, 0x46, 0x65, 0x61, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, + 0x54, 0x79, 0x70, 0x65, 0x52, 0x04, 0x74, 0x79, 0x70, 0x65, 0x12, 0x30, 0x0a, 0x14, 0x6a, 0x6f, + 0x62, 0x5f, 0x73, 0x74, 0x61, 0x67, 0x69, 0x6e, 0x67, 0x5f, 0x6c, 0x6f, 0x63, 0x61, 0x74, 0x69, + 0x6f, 0x6e, 0x18, 0x0a, 0x20, 0x01, 0x28, 0x09, 0x52, 0x12, 0x6a, 0x6f, 0x62, 0x53, 0x74, 0x61, + 0x67, 0x69, 0x6e, 0x67, 0x4c, 0x6f, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x22, 0x8e, 0x01, 0x0a, + 0x10, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, + 0x65, 0x12, 0x18, 0x0a, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x18, 0x01, 0x20, 0x01, + 0x28, 0x09, 0x52, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, + 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, + 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x03, 0x20, 0x01, 0x28, 0x05, + 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x12, 0x32, 0x0a, 0x07, 0x6d, 0x61, 0x78, + 0x5f, 0x61, 0x67, 0x65, 0x18, 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x67, 0x6f, 0x6f, + 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x44, 0x75, 0x72, + 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x06, 0x6d, 0x61, 0x78, 0x41, 0x67, 0x65, 0x22, 0xe1, 0x03, + 0x0a, 0x18, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, + 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x3b, 0x0a, 0x08, 0x66, 0x65, + 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x1f, 0x2e, 0x66, + 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x46, 0x65, 0x61, + 0x74, 0x75, 0x72, 0x65, 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x52, 0x08, 0x66, + 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x12, 0x52, 0x0a, 0x0b, 0x65, 0x6e, 0x74, 0x69, 0x74, + 0x79, 0x5f, 0x72, 0x6f, 0x77, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x31, 0x2e, 0x66, + 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, + 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, + 0x71, 0x75, 0x65, 0x73, 0x74, 0x2e, 0x45, 0x6e, 0x74, 0x69, 0x74, 0x79, 0x52, 0x6f, 0x77, 0x52, + 0x0a, 0x65, 0x6e, 0x74, 0x69, 0x74, 0x79, 0x52, 0x6f, 0x77, 0x73, 0x12, 0x39, 0x0a, 0x19, 0x6f, + 0x6d, 0x69, 0x74, 0x5f, 0x65, 0x6e, 0x74, 0x69, 0x74, 0x69, 0x65, 0x73, 0x5f, 0x69, 0x6e, 0x5f, + 0x72, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x08, 0x52, 0x16, + 0x6f, 0x6d, 0x69, 0x74, 0x45, 0x6e, 0x74, 0x69, 0x74, 0x69, 0x65, 0x73, 0x49, 0x6e, 0x52, 0x65, + 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x1a, 0xf8, 0x01, 0x0a, 0x09, 0x45, 0x6e, 0x74, 0x69, 0x74, + 0x79, 0x52, 0x6f, 0x77, 0x12, 0x45, 0x0a, 0x10, 0x65, 0x6e, 0x74, 0x69, 0x74, 0x79, 0x5f, 0x74, + 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, + 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, + 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x0f, 0x65, 0x6e, 0x74, 0x69, + 0x74, 0x79, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x12, 0x55, 0x0a, 0x06, 0x66, + 0x69, 0x65, 0x6c, 0x64, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x3d, 0x2e, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4f, + 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, + 0x75, 0x65, 0x73, 0x74, 0x2e, 0x45, 0x6e, 0x74, 0x69, 0x74, 0x79, 0x52, 0x6f, 0x77, 0x2e, 0x46, + 0x69, 0x65, 0x6c, 0x64, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x06, 0x66, 0x69, 0x65, 0x6c, + 0x64, 0x73, 0x1a, 0x4d, 0x0a, 0x0b, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x73, 0x45, 0x6e, 0x74, 0x72, + 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, + 0x6b, 0x65, 0x79, 0x12, 0x28, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, + 0x2e, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, + 0x01, 0x22, 0x9b, 0x01, 0x0a, 0x17, 0x47, 0x65, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x46, 0x65, + 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x3b, 0x0a, + 0x08, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x18, 0x03, 0x20, 0x03, 0x28, 0x0b, 0x32, + 0x1f, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, + 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, + 0x52, 0x08, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x12, 0x43, 0x0a, 0x0e, 0x64, 0x61, + 0x74, 0x61, 0x73, 0x65, 0x74, 0x5f, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x02, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x1c, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, + 0x6e, 0x67, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, + 0x52, 0x0d, 0x64, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x22, + 0xad, 0x02, 0x0a, 0x19, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, + 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x57, 0x0a, + 0x0c, 0x66, 0x69, 0x65, 0x6c, 0x64, 0x5f, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x73, 0x18, 0x01, 0x20, + 0x03, 0x28, 0x0b, 0x32, 0x34, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, + 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, + 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x46, 0x69, + 0x65, 0x6c, 0x64, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x73, 0x52, 0x0b, 0x66, 0x69, 0x65, 0x6c, 0x64, + 0x56, 0x61, 0x6c, 0x75, 0x65, 0x73, 0x1a, 0xb6, 0x01, 0x0a, 0x0b, 0x46, 0x69, 0x65, 0x6c, 0x64, + 0x56, 0x61, 0x6c, 0x75, 0x65, 0x73, 0x12, 0x58, 0x0a, 0x06, 0x66, 0x69, 0x65, 0x6c, 0x64, 0x73, + 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x40, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, + 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, + 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, + 0x2e, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x73, 0x2e, 0x46, 0x69, 0x65, + 0x6c, 0x64, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x06, 0x66, 0x69, 0x65, 0x6c, 0x64, 0x73, + 0x1a, 0x4d, 0x0a, 0x0b, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, + 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, + 0x79, 0x12, 0x28, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, + 0x32, 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x56, + 0x61, 0x6c, 0x75, 0x65, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x22, + 0x40, 0x0a, 0x18, 0x47, 0x65, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x46, 0x65, 0x61, 0x74, 0x75, + 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x24, 0x0a, 0x03, 0x6a, + 0x6f, 0x62, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x4a, 0x6f, 0x62, 0x52, 0x03, 0x6a, 0x6f, + 0x62, 0x22, 0x35, 0x0a, 0x0d, 0x47, 0x65, 0x74, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x71, 0x75, 0x65, + 0x73, 0x74, 0x12, 0x24, 0x0a, 0x03, 0x6a, 0x6f, 0x62, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, + 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, + 0x4a, 0x6f, 0x62, 0x52, 0x03, 0x6a, 0x6f, 0x62, 0x22, 0x36, 0x0a, 0x0e, 0x47, 0x65, 0x74, 0x4a, + 0x6f, 0x62, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x24, 0x0a, 0x03, 0x6a, 0x6f, + 0x62, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x4a, 0x6f, 0x62, 0x52, 0x03, 0x6a, 0x6f, 0x62, + 0x22, 0xe2, 0x01, 0x0a, 0x03, 0x4a, 0x6f, 0x62, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, + 0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x12, 0x2a, 0x0a, 0x04, 0x74, 0x79, 0x70, 0x65, + 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, + 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x4a, 0x6f, 0x62, 0x54, 0x79, 0x70, 0x65, 0x52, 0x04, + 0x74, 0x79, 0x70, 0x65, 0x12, 0x30, 0x0a, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x18, 0x03, + 0x20, 0x01, 0x28, 0x0e, 0x32, 0x18, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, + 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x4a, 0x6f, 0x62, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x06, + 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x14, 0x0a, 0x05, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x18, + 0x04, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x12, 0x1b, 0x0a, 0x09, + 0x66, 0x69, 0x6c, 0x65, 0x5f, 0x75, 0x72, 0x69, 0x73, 0x18, 0x05, 0x20, 0x03, 0x28, 0x09, 0x52, + 0x08, 0x66, 0x69, 0x6c, 0x65, 0x55, 0x72, 0x69, 0x73, 0x12, 0x3a, 0x0a, 0x0b, 0x64, 0x61, 0x74, + 0x61, 0x5f, 0x66, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x18, 0x06, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x19, + 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x44, + 0x61, 0x74, 0x61, 0x46, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x52, 0x0a, 0x64, 0x61, 0x74, 0x61, 0x46, + 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x22, 0xd4, 0x01, 0x0a, 0x0d, 0x44, 0x61, 0x74, 0x61, 0x73, 0x65, + 0x74, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x12, 0x4a, 0x0a, 0x0b, 0x66, 0x69, 0x6c, 0x65, 0x5f, + 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x27, 0x2e, 0x66, + 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x44, 0x61, 0x74, + 0x61, 0x73, 0x65, 0x74, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x2e, 0x46, 0x69, 0x6c, 0x65, 0x53, + 0x6f, 0x75, 0x72, 0x63, 0x65, 0x48, 0x00, 0x52, 0x0a, 0x66, 0x69, 0x6c, 0x65, 0x53, 0x6f, 0x75, + 0x72, 0x63, 0x65, 0x1a, 0x65, 0x0a, 0x0a, 0x46, 0x69, 0x6c, 0x65, 0x53, 0x6f, 0x75, 0x72, 0x63, + 0x65, 0x12, 0x1b, 0x0a, 0x09, 0x66, 0x69, 0x6c, 0x65, 0x5f, 0x75, 0x72, 0x69, 0x73, 0x18, 0x01, + 0x20, 0x03, 0x28, 0x09, 0x52, 0x08, 0x66, 0x69, 0x6c, 0x65, 0x55, 0x72, 0x69, 0x73, 0x12, 0x3a, + 0x0a, 0x0b, 0x64, 0x61, 0x74, 0x61, 0x5f, 0x66, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x18, 0x02, 0x20, + 0x01, 0x28, 0x0e, 0x32, 0x19, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, + 0x69, 0x6e, 0x67, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x46, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x52, 0x0a, + 0x64, 0x61, 0x74, 0x61, 0x46, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x42, 0x10, 0x0a, 0x0e, 0x64, 0x61, + 0x74, 0x61, 0x73, 0x65, 0x74, 0x5f, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x2a, 0x6f, 0x0a, 0x10, + 0x46, 0x65, 0x61, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x54, 0x79, 0x70, 0x65, + 0x12, 0x1e, 0x0a, 0x1a, 0x46, 0x45, 0x41, 0x53, 0x54, 0x5f, 0x53, 0x45, 0x52, 0x56, 0x49, 0x4e, + 0x47, 0x5f, 0x54, 0x59, 0x50, 0x45, 0x5f, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, + 0x12, 0x1d, 0x0a, 0x19, 0x46, 0x45, 0x41, 0x53, 0x54, 0x5f, 0x53, 0x45, 0x52, 0x56, 0x49, 0x4e, + 0x47, 0x5f, 0x54, 0x59, 0x50, 0x45, 0x5f, 0x4f, 0x4e, 0x4c, 0x49, 0x4e, 0x45, 0x10, 0x01, 0x12, + 0x1c, 0x0a, 0x18, 0x46, 0x45, 0x41, 0x53, 0x54, 0x5f, 0x53, 0x45, 0x52, 0x56, 0x49, 0x4e, 0x47, + 0x5f, 0x54, 0x59, 0x50, 0x45, 0x5f, 0x42, 0x41, 0x54, 0x43, 0x48, 0x10, 0x02, 0x2a, 0x36, 0x0a, + 0x07, 0x4a, 0x6f, 0x62, 0x54, 0x79, 0x70, 0x65, 0x12, 0x14, 0x0a, 0x10, 0x4a, 0x4f, 0x42, 0x5f, + 0x54, 0x59, 0x50, 0x45, 0x5f, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, 0x12, 0x15, + 0x0a, 0x11, 0x4a, 0x4f, 0x42, 0x5f, 0x54, 0x59, 0x50, 0x45, 0x5f, 0x44, 0x4f, 0x57, 0x4e, 0x4c, + 0x4f, 0x41, 0x44, 0x10, 0x01, 0x2a, 0x68, 0x0a, 0x09, 0x4a, 0x6f, 0x62, 0x53, 0x74, 0x61, 0x74, + 0x75, 0x73, 0x12, 0x16, 0x0a, 0x12, 0x4a, 0x4f, 0x42, 0x5f, 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, + 0x5f, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, 0x12, 0x16, 0x0a, 0x12, 0x4a, 0x4f, + 0x42, 0x5f, 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, 0x5f, 0x50, 0x45, 0x4e, 0x44, 0x49, 0x4e, 0x47, + 0x10, 0x01, 0x12, 0x16, 0x0a, 0x12, 0x4a, 0x4f, 0x42, 0x5f, 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, + 0x5f, 0x52, 0x55, 0x4e, 0x4e, 0x49, 0x4e, 0x47, 0x10, 0x02, 0x12, 0x13, 0x0a, 0x0f, 0x4a, 0x4f, + 0x42, 0x5f, 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, 0x5f, 0x44, 0x4f, 0x4e, 0x45, 0x10, 0x03, 0x2a, + 0x3b, 0x0a, 0x0a, 0x44, 0x61, 0x74, 0x61, 0x46, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x12, 0x17, 0x0a, + 0x13, 0x44, 0x41, 0x54, 0x41, 0x5f, 0x46, 0x4f, 0x52, 0x4d, 0x41, 0x54, 0x5f, 0x49, 0x4e, 0x56, + 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, 0x12, 0x14, 0x0a, 0x10, 0x44, 0x41, 0x54, 0x41, 0x5f, 0x46, + 0x4f, 0x52, 0x4d, 0x41, 0x54, 0x5f, 0x41, 0x56, 0x52, 0x4f, 0x10, 0x01, 0x32, 0x92, 0x03, 0x0a, + 0x0e, 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, + 0x6c, 0x0a, 0x13, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, + 0x6e, 0x67, 0x49, 0x6e, 0x66, 0x6f, 0x12, 0x29, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, + 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, 0x53, + 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x49, 0x6e, 0x66, 0x6f, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, + 0x74, 0x1a, 0x2a, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, + 0x67, 0x2e, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, + 0x67, 0x49, 0x6e, 0x66, 0x6f, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x66, 0x0a, + 0x11, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, + 0x65, 0x73, 0x12, 0x27, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, + 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, + 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x28, 0x2e, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4f, + 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, + 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x63, 0x0a, 0x10, 0x47, 0x65, 0x74, 0x42, 0x61, 0x74, 0x63, + 0x68, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x12, 0x26, 0x2e, 0x66, 0x65, 0x61, 0x73, + 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x42, 0x61, 0x74, + 0x63, 0x68, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, + 0x74, 0x1a, 0x27, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, + 0x67, 0x2e, 0x47, 0x65, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, + 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x45, 0x0a, 0x06, 0x47, 0x65, + 0x74, 0x4a, 0x6f, 0x62, 0x12, 0x1c, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, + 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x71, 0x75, 0x65, + 0x73, 0x74, 0x1a, 0x1d, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, + 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, + 0x65, 0x42, 0x54, 0x0a, 0x0d, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, + 0x6e, 0x67, 0x42, 0x0f, 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x41, 0x50, 0x49, 0x50, 0x72, + 0x6f, 0x74, 0x6f, 0x5a, 0x32, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, + 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, + 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, + 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, +} + +var ( + file_feast_serving_ServingService_proto_rawDescOnce sync.Once + file_feast_serving_ServingService_proto_rawDescData = file_feast_serving_ServingService_proto_rawDesc +) + +func file_feast_serving_ServingService_proto_rawDescGZIP() []byte { + file_feast_serving_ServingService_proto_rawDescOnce.Do(func() { + file_feast_serving_ServingService_proto_rawDescData = protoimpl.X.CompressGZIP(file_feast_serving_ServingService_proto_rawDescData) + }) + return file_feast_serving_ServingService_proto_rawDescData +} + +var file_feast_serving_ServingService_proto_enumTypes = make([]protoimpl.EnumInfo, 4) +var file_feast_serving_ServingService_proto_msgTypes = make([]protoimpl.MessageInfo, 16) +var file_feast_serving_ServingService_proto_goTypes = []interface{}{ + (FeastServingType)(0), // 0: feast.serving.FeastServingType + (JobType)(0), // 1: feast.serving.JobType + (JobStatus)(0), // 2: feast.serving.JobStatus + (DataFormat)(0), // 3: feast.serving.DataFormat + (*GetFeastServingInfoRequest)(nil), // 4: feast.serving.GetFeastServingInfoRequest + (*GetFeastServingInfoResponse)(nil), // 5: feast.serving.GetFeastServingInfoResponse + (*FeatureReference)(nil), // 6: feast.serving.FeatureReference + (*GetOnlineFeaturesRequest)(nil), // 7: feast.serving.GetOnlineFeaturesRequest + (*GetBatchFeaturesRequest)(nil), // 8: feast.serving.GetBatchFeaturesRequest + (*GetOnlineFeaturesResponse)(nil), // 9: feast.serving.GetOnlineFeaturesResponse + (*GetBatchFeaturesResponse)(nil), // 10: feast.serving.GetBatchFeaturesResponse + (*GetJobRequest)(nil), // 11: feast.serving.GetJobRequest + (*GetJobResponse)(nil), // 12: feast.serving.GetJobResponse + (*Job)(nil), // 13: feast.serving.Job + (*DatasetSource)(nil), // 14: feast.serving.DatasetSource + (*GetOnlineFeaturesRequest_EntityRow)(nil), // 15: feast.serving.GetOnlineFeaturesRequest.EntityRow + nil, // 16: feast.serving.GetOnlineFeaturesRequest.EntityRow.FieldsEntry + (*GetOnlineFeaturesResponse_FieldValues)(nil), // 17: feast.serving.GetOnlineFeaturesResponse.FieldValues + nil, // 18: feast.serving.GetOnlineFeaturesResponse.FieldValues.FieldsEntry + (*DatasetSource_FileSource)(nil), // 19: feast.serving.DatasetSource.FileSource + (*duration.Duration)(nil), // 20: google.protobuf.Duration + (*timestamp.Timestamp)(nil), // 21: google.protobuf.Timestamp + (*types.Value)(nil), // 22: feast.types.Value +} +var file_feast_serving_ServingService_proto_depIdxs = []int32{ + 0, // 0: feast.serving.GetFeastServingInfoResponse.type:type_name -> feast.serving.FeastServingType + 20, // 1: feast.serving.FeatureReference.max_age:type_name -> google.protobuf.Duration + 6, // 2: feast.serving.GetOnlineFeaturesRequest.features:type_name -> feast.serving.FeatureReference + 15, // 3: feast.serving.GetOnlineFeaturesRequest.entity_rows:type_name -> feast.serving.GetOnlineFeaturesRequest.EntityRow + 6, // 4: feast.serving.GetBatchFeaturesRequest.features:type_name -> feast.serving.FeatureReference + 14, // 5: feast.serving.GetBatchFeaturesRequest.dataset_source:type_name -> feast.serving.DatasetSource + 17, // 6: feast.serving.GetOnlineFeaturesResponse.field_values:type_name -> feast.serving.GetOnlineFeaturesResponse.FieldValues + 13, // 7: feast.serving.GetBatchFeaturesResponse.job:type_name -> feast.serving.Job + 13, // 8: feast.serving.GetJobRequest.job:type_name -> feast.serving.Job + 13, // 9: feast.serving.GetJobResponse.job:type_name -> feast.serving.Job + 1, // 10: feast.serving.Job.type:type_name -> feast.serving.JobType + 2, // 11: feast.serving.Job.status:type_name -> feast.serving.JobStatus + 3, // 12: feast.serving.Job.data_format:type_name -> feast.serving.DataFormat + 19, // 13: feast.serving.DatasetSource.file_source:type_name -> feast.serving.DatasetSource.FileSource + 21, // 14: feast.serving.GetOnlineFeaturesRequest.EntityRow.entity_timestamp:type_name -> google.protobuf.Timestamp + 16, // 15: feast.serving.GetOnlineFeaturesRequest.EntityRow.fields:type_name -> feast.serving.GetOnlineFeaturesRequest.EntityRow.FieldsEntry + 22, // 16: feast.serving.GetOnlineFeaturesRequest.EntityRow.FieldsEntry.value:type_name -> feast.types.Value + 18, // 17: feast.serving.GetOnlineFeaturesResponse.FieldValues.fields:type_name -> feast.serving.GetOnlineFeaturesResponse.FieldValues.FieldsEntry + 22, // 18: feast.serving.GetOnlineFeaturesResponse.FieldValues.FieldsEntry.value:type_name -> feast.types.Value + 3, // 19: feast.serving.DatasetSource.FileSource.data_format:type_name -> feast.serving.DataFormat + 4, // 20: feast.serving.ServingService.GetFeastServingInfo:input_type -> feast.serving.GetFeastServingInfoRequest + 7, // 21: feast.serving.ServingService.GetOnlineFeatures:input_type -> feast.serving.GetOnlineFeaturesRequest + 8, // 22: feast.serving.ServingService.GetBatchFeatures:input_type -> feast.serving.GetBatchFeaturesRequest + 11, // 23: feast.serving.ServingService.GetJob:input_type -> feast.serving.GetJobRequest + 5, // 24: feast.serving.ServingService.GetFeastServingInfo:output_type -> feast.serving.GetFeastServingInfoResponse + 9, // 25: feast.serving.ServingService.GetOnlineFeatures:output_type -> feast.serving.GetOnlineFeaturesResponse + 10, // 26: feast.serving.ServingService.GetBatchFeatures:output_type -> feast.serving.GetBatchFeaturesResponse + 12, // 27: feast.serving.ServingService.GetJob:output_type -> feast.serving.GetJobResponse + 24, // [24:28] is the sub-list for method output_type + 20, // [20:24] is the sub-list for method input_type + 20, // [20:20] is the sub-list for extension type_name + 20, // [20:20] is the sub-list for extension extendee + 0, // [0:20] is the sub-list for field type_name +} + +func init() { file_feast_serving_ServingService_proto_init() } +func file_feast_serving_ServingService_proto_init() { + if File_feast_serving_ServingService_proto != nil { + return + } + if !protoimpl.UnsafeEnabled { + file_feast_serving_ServingService_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetFeastServingInfoRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_serving_ServingService_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetFeastServingInfoResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_serving_ServingService_proto_msgTypes[2].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*FeatureReference); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_serving_ServingService_proto_msgTypes[3].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetOnlineFeaturesRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_serving_ServingService_proto_msgTypes[4].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetBatchFeaturesRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_serving_ServingService_proto_msgTypes[5].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetOnlineFeaturesResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_serving_ServingService_proto_msgTypes[6].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetBatchFeaturesResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_serving_ServingService_proto_msgTypes[7].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetJobRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_serving_ServingService_proto_msgTypes[8].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetJobResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_serving_ServingService_proto_msgTypes[9].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Job); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_serving_ServingService_proto_msgTypes[10].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*DatasetSource); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_serving_ServingService_proto_msgTypes[11].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetOnlineFeaturesRequest_EntityRow); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_serving_ServingService_proto_msgTypes[13].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetOnlineFeaturesResponse_FieldValues); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_serving_ServingService_proto_msgTypes[15].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*DatasetSource_FileSource); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + } + file_feast_serving_ServingService_proto_msgTypes[10].OneofWrappers = []interface{}{ + (*DatasetSource_FileSource_)(nil), + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_feast_serving_ServingService_proto_rawDesc, + NumEnums: 4, + NumMessages: 16, + NumExtensions: 0, + NumServices: 1, + }, + GoTypes: file_feast_serving_ServingService_proto_goTypes, + DependencyIndexes: file_feast_serving_ServingService_proto_depIdxs, + EnumInfos: file_feast_serving_ServingService_proto_enumTypes, + MessageInfos: file_feast_serving_ServingService_proto_msgTypes, + }.Build() + File_feast_serving_ServingService_proto = out.File + file_feast_serving_ServingService_proto_rawDesc = nil + file_feast_serving_ServingService_proto_goTypes = nil + file_feast_serving_ServingService_proto_depIdxs = nil } // Reference imports to suppress errors if they are not otherwise used. @@ -1059,16 +1627,16 @@ type ServingServiceServer interface { type UnimplementedServingServiceServer struct { } -func (*UnimplementedServingServiceServer) GetFeastServingInfo(ctx context.Context, req *GetFeastServingInfoRequest) (*GetFeastServingInfoResponse, error) { +func (*UnimplementedServingServiceServer) GetFeastServingInfo(context.Context, *GetFeastServingInfoRequest) (*GetFeastServingInfoResponse, error) { return nil, status.Errorf(codes.Unimplemented, "method GetFeastServingInfo not implemented") } -func (*UnimplementedServingServiceServer) GetOnlineFeatures(ctx context.Context, req *GetOnlineFeaturesRequest) (*GetOnlineFeaturesResponse, error) { +func (*UnimplementedServingServiceServer) GetOnlineFeatures(context.Context, *GetOnlineFeaturesRequest) (*GetOnlineFeaturesResponse, error) { return nil, status.Errorf(codes.Unimplemented, "method GetOnlineFeatures not implemented") } -func (*UnimplementedServingServiceServer) GetBatchFeatures(ctx context.Context, req *GetBatchFeaturesRequest) (*GetBatchFeaturesResponse, error) { +func (*UnimplementedServingServiceServer) GetBatchFeatures(context.Context, *GetBatchFeaturesRequest) (*GetBatchFeaturesResponse, error) { return nil, status.Errorf(codes.Unimplemented, "method GetBatchFeatures not implemented") } -func (*UnimplementedServingServiceServer) GetJob(ctx context.Context, req *GetJobRequest) (*GetJobResponse, error) { +func (*UnimplementedServingServiceServer) GetJob(context.Context, *GetJobRequest) (*GetJobResponse, error) { return nil, status.Errorf(codes.Unimplemented, "method GetJob not implemented") } diff --git a/sdk/go/protos/feast/types/FeatureRow.pb.go b/sdk/go/protos/feast/types/FeatureRow.pb.go index 26868ebdd0b..e6358b13003 100644 --- a/sdk/go/protos/feast/types/FeatureRow.pb.go +++ b/sdk/go/protos/feast/types/FeatureRow.pb.go @@ -1,27 +1,51 @@ +// +// Copyright 2018 The Feast Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + // Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.21.0 +// protoc v3.10.1 // source: feast/types/FeatureRow.proto package types import ( - fmt "fmt" proto "github.com/golang/protobuf/proto" timestamp "github.com/golang/protobuf/ptypes/timestamp" - math "math" + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" ) -// Reference imports to suppress errors if they are not otherwise used. -var _ = proto.Marshal -var _ = fmt.Errorf -var _ = math.Inf +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) -// This is a compile-time assertion to ensure that this generated file -// is compatible with the proto package it is being compiled against. -// A compilation error at this line likely means your copy of the -// proto package needs to be updated. -const _ = proto.ProtoPackageIsVersion3 // please upgrade the proto package +// This is a compile-time assertion that a sufficiently up-to-date version +// of the legacy proto package is being used. +const _ = proto.ProtoPackageIsVersion4 type FeatureRow struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + // Fields in the feature row. Fields []*Field `protobuf:"bytes,2,rep,name=fields,proto3" json:"fields,omitempty"` // Timestamp of the feature row. While the actual definition of this timestamp may vary @@ -31,81 +55,153 @@ type FeatureRow struct { // Complete reference to the featureSet this featureRow belongs to, in the form of // /:. This value will be used by the feast ingestion job to filter // rows, and write the values to the correct tables. - FeatureSet string `protobuf:"bytes,6,opt,name=feature_set,json=featureSet,proto3" json:"feature_set,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + FeatureSet string `protobuf:"bytes,6,opt,name=feature_set,json=featureSet,proto3" json:"feature_set,omitempty"` } -func (m *FeatureRow) Reset() { *m = FeatureRow{} } -func (m *FeatureRow) String() string { return proto.CompactTextString(m) } -func (*FeatureRow) ProtoMessage() {} -func (*FeatureRow) Descriptor() ([]byte, []int) { - return fileDescriptor_fbbea9c89787d1c7, []int{0} +func (x *FeatureRow) Reset() { + *x = FeatureRow{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_types_FeatureRow_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *FeatureRow) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_FeatureRow.Unmarshal(m, b) -} -func (m *FeatureRow) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_FeatureRow.Marshal(b, m, deterministic) +func (x *FeatureRow) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *FeatureRow) XXX_Merge(src proto.Message) { - xxx_messageInfo_FeatureRow.Merge(m, src) -} -func (m *FeatureRow) XXX_Size() int { - return xxx_messageInfo_FeatureRow.Size(m) -} -func (m *FeatureRow) XXX_DiscardUnknown() { - xxx_messageInfo_FeatureRow.DiscardUnknown(m) + +func (*FeatureRow) ProtoMessage() {} + +func (x *FeatureRow) ProtoReflect() protoreflect.Message { + mi := &file_feast_types_FeatureRow_proto_msgTypes[0] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_FeatureRow proto.InternalMessageInfo +// Deprecated: Use FeatureRow.ProtoReflect.Descriptor instead. +func (*FeatureRow) Descriptor() ([]byte, []int) { + return file_feast_types_FeatureRow_proto_rawDescGZIP(), []int{0} +} -func (m *FeatureRow) GetFields() []*Field { - if m != nil { - return m.Fields +func (x *FeatureRow) GetFields() []*Field { + if x != nil { + return x.Fields } return nil } -func (m *FeatureRow) GetEventTimestamp() *timestamp.Timestamp { - if m != nil { - return m.EventTimestamp +func (x *FeatureRow) GetEventTimestamp() *timestamp.Timestamp { + if x != nil { + return x.EventTimestamp } return nil } -func (m *FeatureRow) GetFeatureSet() string { - if m != nil { - return m.FeatureSet +func (x *FeatureRow) GetFeatureSet() string { + if x != nil { + return x.FeatureSet } return "" } -func init() { - proto.RegisterType((*FeatureRow)(nil), "feast.types.FeatureRow") +var File_feast_types_FeatureRow_proto protoreflect.FileDescriptor + +var file_feast_types_FeatureRow_proto_rawDesc = []byte{ + 0x0a, 0x1c, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2f, 0x46, 0x65, + 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x6f, 0x77, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0b, + 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x1a, 0x1f, 0x67, 0x6f, 0x6f, + 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2f, 0x74, 0x69, 0x6d, + 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x17, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2f, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2f, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x2e, + 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x9e, 0x01, 0x0a, 0x0a, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, + 0x65, 0x52, 0x6f, 0x77, 0x12, 0x2a, 0x0a, 0x06, 0x66, 0x69, 0x65, 0x6c, 0x64, 0x73, 0x18, 0x02, + 0x20, 0x03, 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, + 0x65, 0x73, 0x2e, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x52, 0x06, 0x66, 0x69, 0x65, 0x6c, 0x64, 0x73, + 0x12, 0x43, 0x0a, 0x0f, 0x65, 0x76, 0x65, 0x6e, 0x74, 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x73, 0x74, + 0x61, 0x6d, 0x70, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, + 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, + 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x0e, 0x65, 0x76, 0x65, 0x6e, 0x74, 0x54, 0x69, 0x6d, 0x65, + 0x73, 0x74, 0x61, 0x6d, 0x70, 0x12, 0x1f, 0x0a, 0x0b, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, + 0x5f, 0x73, 0x65, 0x74, 0x18, 0x06, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0a, 0x66, 0x65, 0x61, 0x74, + 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x42, 0x50, 0x0a, 0x0b, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x74, 0x79, 0x70, 0x65, 0x73, 0x42, 0x0f, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x6f, + 0x77, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x30, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, + 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, + 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, 0x61, + 0x73, 0x74, 0x2f, 0x74, 0x79, 0x70, 0x65, 0x73, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, +} + +var ( + file_feast_types_FeatureRow_proto_rawDescOnce sync.Once + file_feast_types_FeatureRow_proto_rawDescData = file_feast_types_FeatureRow_proto_rawDesc +) + +func file_feast_types_FeatureRow_proto_rawDescGZIP() []byte { + file_feast_types_FeatureRow_proto_rawDescOnce.Do(func() { + file_feast_types_FeatureRow_proto_rawDescData = protoimpl.X.CompressGZIP(file_feast_types_FeatureRow_proto_rawDescData) + }) + return file_feast_types_FeatureRow_proto_rawDescData } -func init() { - proto.RegisterFile("feast/types/FeatureRow.proto", fileDescriptor_fbbea9c89787d1c7) +var file_feast_types_FeatureRow_proto_msgTypes = make([]protoimpl.MessageInfo, 1) +var file_feast_types_FeatureRow_proto_goTypes = []interface{}{ + (*FeatureRow)(nil), // 0: feast.types.FeatureRow + (*Field)(nil), // 1: feast.types.Field + (*timestamp.Timestamp)(nil), // 2: google.protobuf.Timestamp +} +var file_feast_types_FeatureRow_proto_depIdxs = []int32{ + 1, // 0: feast.types.FeatureRow.fields:type_name -> feast.types.Field + 2, // 1: feast.types.FeatureRow.event_timestamp:type_name -> google.protobuf.Timestamp + 2, // [2:2] is the sub-list for method output_type + 2, // [2:2] is the sub-list for method input_type + 2, // [2:2] is the sub-list for extension type_name + 2, // [2:2] is the sub-list for extension extendee + 0, // [0:2] is the sub-list for field type_name } -var fileDescriptor_fbbea9c89787d1c7 = []byte{ - // 238 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x54, 0x90, 0xcd, 0x4a, 0xc3, 0x40, - 0x10, 0xc7, 0x89, 0x85, 0x80, 0x1b, 0xb0, 0xb0, 0x17, 0x43, 0x10, 0x1a, 0x3c, 0x05, 0x0f, 0x33, - 0x52, 0xdf, 0xa0, 0x82, 0xe7, 0x12, 0x3d, 0x79, 0x29, 0x89, 0x9d, 0xac, 0xb1, 0x8d, 0x13, 0xba, - 0x13, 0xc5, 0x97, 0xf1, 0x59, 0x65, 0x77, 0xdb, 0x26, 0x1e, 0x77, 0x7e, 0x33, 0xff, 0x8f, 0x55, - 0x37, 0x0d, 0x55, 0x56, 0x50, 0x7e, 0x7a, 0xb2, 0xf8, 0x44, 0x95, 0x0c, 0x07, 0x2a, 0xf9, 0x1b, - 0xfa, 0x03, 0x0b, 0xeb, 0xc4, 0x53, 0xf0, 0x34, 0x5b, 0x18, 0x66, 0xb3, 0x27, 0xf4, 0xa8, 0x1e, - 0x1a, 0x94, 0xb6, 0x23, 0x2b, 0x55, 0xd7, 0x87, 0xed, 0xec, 0xfa, 0x9f, 0x56, 0x4b, 0xfb, 0x6d, - 0x00, 0xb7, 0xbf, 0x91, 0x52, 0xa3, 0xb6, 0xbe, 0x53, 0x71, 0xe3, 0xa8, 0x4d, 0x2f, 0xf2, 0x59, - 0x91, 0x2c, 0x35, 0x4c, 0x6c, 0xc0, 0x1f, 0x96, 0xc7, 0x0d, 0xfd, 0xa8, 0xe6, 0xf4, 0x45, 0x9f, - 0xb2, 0x39, 0x9b, 0xa5, 0xb3, 0x3c, 0x2a, 0x92, 0x65, 0x06, 0x21, 0x0e, 0x9c, 0xe2, 0xc0, 0xcb, - 0x69, 0xa3, 0xbc, 0xf2, 0x27, 0xe7, 0xb7, 0x5e, 0x28, 0x57, 0xc4, 0xd9, 0x6f, 0x2c, 0x49, 0x1a, - 0xe7, 0x51, 0x71, 0x59, 0xaa, 0xe3, 0xe8, 0x99, 0x64, 0xb5, 0x56, 0xd3, 0xa6, 0xab, 0xf9, 0x18, - 0x76, 0xed, 0xd4, 0x5f, 0xef, 0x4d, 0x2b, 0xef, 0x43, 0x0d, 0x6f, 0xdc, 0xa1, 0xe1, 0x0f, 0xda, - 0x61, 0xa8, 0x6a, 0xb7, 0x3b, 0x34, 0x1c, 0x7e, 0xc4, 0xe2, 0xa4, 0x7e, 0x1d, 0xfb, 0xd9, 0xc3, - 0x5f, 0x00, 0x00, 0x00, 0xff, 0xff, 0x2f, 0x88, 0x1b, 0x19, 0x60, 0x01, 0x00, 0x00, +func init() { file_feast_types_FeatureRow_proto_init() } +func file_feast_types_FeatureRow_proto_init() { + if File_feast_types_FeatureRow_proto != nil { + return + } + file_feast_types_Field_proto_init() + if !protoimpl.UnsafeEnabled { + file_feast_types_FeatureRow_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*FeatureRow); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_feast_types_FeatureRow_proto_rawDesc, + NumEnums: 0, + NumMessages: 1, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_feast_types_FeatureRow_proto_goTypes, + DependencyIndexes: file_feast_types_FeatureRow_proto_depIdxs, + MessageInfos: file_feast_types_FeatureRow_proto_msgTypes, + }.Build() + File_feast_types_FeatureRow_proto = out.File + file_feast_types_FeatureRow_proto_rawDesc = nil + file_feast_types_FeatureRow_proto_goTypes = nil + file_feast_types_FeatureRow_proto_depIdxs = nil } diff --git a/sdk/go/protos/feast/types/FeatureRowExtended.pb.go b/sdk/go/protos/feast/types/FeatureRowExtended.pb.go index 734c98687a0..c18628a2f34 100644 --- a/sdk/go/protos/feast/types/FeatureRowExtended.pb.go +++ b/sdk/go/protos/feast/types/FeatureRowExtended.pb.go @@ -1,223 +1,370 @@ +// +// Copyright 2018 The Feast Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + // Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.21.0 +// protoc v3.10.1 // source: feast/types/FeatureRowExtended.proto package types import ( - fmt "fmt" proto "github.com/golang/protobuf/proto" timestamp "github.com/golang/protobuf/ptypes/timestamp" - math "math" + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" ) -// Reference imports to suppress errors if they are not otherwise used. -var _ = proto.Marshal -var _ = fmt.Errorf -var _ = math.Inf +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) -// This is a compile-time assertion to ensure that this generated file -// is compatible with the proto package it is being compiled against. -// A compilation error at this line likely means your copy of the -// proto package needs to be updated. -const _ = proto.ProtoPackageIsVersion3 // please upgrade the proto package +// This is a compile-time assertion that a sufficiently up-to-date version +// of the legacy proto package is being used. +const _ = proto.ProtoPackageIsVersion4 type Error struct { - Cause string `protobuf:"bytes,1,opt,name=cause,proto3" json:"cause,omitempty"` - Transform string `protobuf:"bytes,2,opt,name=transform,proto3" json:"transform,omitempty"` - Message string `protobuf:"bytes,3,opt,name=message,proto3" json:"message,omitempty"` - StackTrace string `protobuf:"bytes,4,opt,name=stack_trace,json=stackTrace,proto3" json:"stack_trace,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} - -func (m *Error) Reset() { *m = Error{} } -func (m *Error) String() string { return proto.CompactTextString(m) } -func (*Error) ProtoMessage() {} -func (*Error) Descriptor() ([]byte, []int) { - return fileDescriptor_7823aa2c72575793, []int{0} -} + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields -func (m *Error) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_Error.Unmarshal(m, b) -} -func (m *Error) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_Error.Marshal(b, m, deterministic) + Cause string `protobuf:"bytes,1,opt,name=cause,proto3" json:"cause,omitempty"` // exception class name + Transform string `protobuf:"bytes,2,opt,name=transform,proto3" json:"transform,omitempty"` // name of transform where the error occurred + Message string `protobuf:"bytes,3,opt,name=message,proto3" json:"message,omitempty"` + StackTrace string `protobuf:"bytes,4,opt,name=stack_trace,json=stackTrace,proto3" json:"stack_trace,omitempty"` } -func (m *Error) XXX_Merge(src proto.Message) { - xxx_messageInfo_Error.Merge(m, src) + +func (x *Error) Reset() { + *x = Error{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_types_FeatureRowExtended_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *Error) XXX_Size() int { - return xxx_messageInfo_Error.Size(m) + +func (x *Error) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *Error) XXX_DiscardUnknown() { - xxx_messageInfo_Error.DiscardUnknown(m) + +func (*Error) ProtoMessage() {} + +func (x *Error) ProtoReflect() protoreflect.Message { + mi := &file_feast_types_FeatureRowExtended_proto_msgTypes[0] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_Error proto.InternalMessageInfo +// Deprecated: Use Error.ProtoReflect.Descriptor instead. +func (*Error) Descriptor() ([]byte, []int) { + return file_feast_types_FeatureRowExtended_proto_rawDescGZIP(), []int{0} +} -func (m *Error) GetCause() string { - if m != nil { - return m.Cause +func (x *Error) GetCause() string { + if x != nil { + return x.Cause } return "" } -func (m *Error) GetTransform() string { - if m != nil { - return m.Transform +func (x *Error) GetTransform() string { + if x != nil { + return x.Transform } return "" } -func (m *Error) GetMessage() string { - if m != nil { - return m.Message +func (x *Error) GetMessage() string { + if x != nil { + return x.Message } return "" } -func (m *Error) GetStackTrace() string { - if m != nil { - return m.StackTrace +func (x *Error) GetStackTrace() string { + if x != nil { + return x.StackTrace } return "" } type Attempt struct { - Attempts int32 `protobuf:"varint,1,opt,name=attempts,proto3" json:"attempts,omitempty"` - Error *Error `protobuf:"bytes,2,opt,name=error,proto3" json:"error,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields -func (m *Attempt) Reset() { *m = Attempt{} } -func (m *Attempt) String() string { return proto.CompactTextString(m) } -func (*Attempt) ProtoMessage() {} -func (*Attempt) Descriptor() ([]byte, []int) { - return fileDescriptor_7823aa2c72575793, []int{1} + Attempts int32 `protobuf:"varint,1,opt,name=attempts,proto3" json:"attempts,omitempty"` + Error *Error `protobuf:"bytes,2,opt,name=error,proto3" json:"error,omitempty"` } -func (m *Attempt) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_Attempt.Unmarshal(m, b) -} -func (m *Attempt) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_Attempt.Marshal(b, m, deterministic) -} -func (m *Attempt) XXX_Merge(src proto.Message) { - xxx_messageInfo_Attempt.Merge(m, src) +func (x *Attempt) Reset() { + *x = Attempt{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_types_FeatureRowExtended_proto_msgTypes[1] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *Attempt) XXX_Size() int { - return xxx_messageInfo_Attempt.Size(m) + +func (x *Attempt) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *Attempt) XXX_DiscardUnknown() { - xxx_messageInfo_Attempt.DiscardUnknown(m) + +func (*Attempt) ProtoMessage() {} + +func (x *Attempt) ProtoReflect() protoreflect.Message { + mi := &file_feast_types_FeatureRowExtended_proto_msgTypes[1] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_Attempt proto.InternalMessageInfo +// Deprecated: Use Attempt.ProtoReflect.Descriptor instead. +func (*Attempt) Descriptor() ([]byte, []int) { + return file_feast_types_FeatureRowExtended_proto_rawDescGZIP(), []int{1} +} -func (m *Attempt) GetAttempts() int32 { - if m != nil { - return m.Attempts +func (x *Attempt) GetAttempts() int32 { + if x != nil { + return x.Attempts } return 0 } -func (m *Attempt) GetError() *Error { - if m != nil { - return m.Error +func (x *Attempt) GetError() *Error { + if x != nil { + return x.Error } return nil } type FeatureRowExtended struct { - Row *FeatureRow `protobuf:"bytes,1,opt,name=row,proto3" json:"row,omitempty"` - LastAttempt *Attempt `protobuf:"bytes,2,opt,name=last_attempt,json=lastAttempt,proto3" json:"last_attempt,omitempty"` - FirstSeen *timestamp.Timestamp `protobuf:"bytes,3,opt,name=first_seen,json=firstSeen,proto3" json:"first_seen,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields -func (m *FeatureRowExtended) Reset() { *m = FeatureRowExtended{} } -func (m *FeatureRowExtended) String() string { return proto.CompactTextString(m) } -func (*FeatureRowExtended) ProtoMessage() {} -func (*FeatureRowExtended) Descriptor() ([]byte, []int) { - return fileDescriptor_7823aa2c72575793, []int{2} + Row *FeatureRow `protobuf:"bytes,1,opt,name=row,proto3" json:"row,omitempty"` + LastAttempt *Attempt `protobuf:"bytes,2,opt,name=last_attempt,json=lastAttempt,proto3" json:"last_attempt,omitempty"` + FirstSeen *timestamp.Timestamp `protobuf:"bytes,3,opt,name=first_seen,json=firstSeen,proto3" json:"first_seen,omitempty"` } -func (m *FeatureRowExtended) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_FeatureRowExtended.Unmarshal(m, b) -} -func (m *FeatureRowExtended) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_FeatureRowExtended.Marshal(b, m, deterministic) -} -func (m *FeatureRowExtended) XXX_Merge(src proto.Message) { - xxx_messageInfo_FeatureRowExtended.Merge(m, src) +func (x *FeatureRowExtended) Reset() { + *x = FeatureRowExtended{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_types_FeatureRowExtended_proto_msgTypes[2] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *FeatureRowExtended) XXX_Size() int { - return xxx_messageInfo_FeatureRowExtended.Size(m) + +func (x *FeatureRowExtended) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *FeatureRowExtended) XXX_DiscardUnknown() { - xxx_messageInfo_FeatureRowExtended.DiscardUnknown(m) + +func (*FeatureRowExtended) ProtoMessage() {} + +func (x *FeatureRowExtended) ProtoReflect() protoreflect.Message { + mi := &file_feast_types_FeatureRowExtended_proto_msgTypes[2] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_FeatureRowExtended proto.InternalMessageInfo +// Deprecated: Use FeatureRowExtended.ProtoReflect.Descriptor instead. +func (*FeatureRowExtended) Descriptor() ([]byte, []int) { + return file_feast_types_FeatureRowExtended_proto_rawDescGZIP(), []int{2} +} -func (m *FeatureRowExtended) GetRow() *FeatureRow { - if m != nil { - return m.Row +func (x *FeatureRowExtended) GetRow() *FeatureRow { + if x != nil { + return x.Row } return nil } -func (m *FeatureRowExtended) GetLastAttempt() *Attempt { - if m != nil { - return m.LastAttempt +func (x *FeatureRowExtended) GetLastAttempt() *Attempt { + if x != nil { + return x.LastAttempt } return nil } -func (m *FeatureRowExtended) GetFirstSeen() *timestamp.Timestamp { - if m != nil { - return m.FirstSeen +func (x *FeatureRowExtended) GetFirstSeen() *timestamp.Timestamp { + if x != nil { + return x.FirstSeen } return nil } -func init() { - proto.RegisterType((*Error)(nil), "feast.types.Error") - proto.RegisterType((*Attempt)(nil), "feast.types.Attempt") - proto.RegisterType((*FeatureRowExtended)(nil), "feast.types.FeatureRowExtended") -} - -func init() { - proto.RegisterFile("feast/types/FeatureRowExtended.proto", fileDescriptor_7823aa2c72575793) -} - -var fileDescriptor_7823aa2c72575793 = []byte{ - // 345 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x6c, 0x91, 0xcf, 0x4e, 0xc2, 0x40, - 0x10, 0xc6, 0x83, 0x58, 0x91, 0xa9, 0xa7, 0x0d, 0x09, 0x4d, 0x43, 0x82, 0x21, 0x1e, 0xf0, 0xb2, - 0x6b, 0xf0, 0x60, 0x3c, 0x4a, 0x82, 0x57, 0x4d, 0xe5, 0x60, 0xbc, 0x90, 0xa5, 0x4c, 0x2b, 0x42, - 0xbb, 0xcd, 0xee, 0x54, 0xf4, 0xb9, 0x7c, 0x41, 0xd3, 0x5d, 0x2a, 0x10, 0xbc, 0xed, 0xcc, 0xf7, - 0x9b, 0x9d, 0x3f, 0x1f, 0x5c, 0x25, 0x28, 0x0d, 0x09, 0xfa, 0x2e, 0xd0, 0x88, 0x47, 0x94, 0x54, - 0x6a, 0x8c, 0xd4, 0x66, 0xf2, 0x45, 0x98, 0x2f, 0x70, 0xc1, 0x0b, 0xad, 0x48, 0x31, 0xdf, 0x52, - 0xdc, 0x52, 0x61, 0x3f, 0x55, 0x2a, 0x5d, 0xa3, 0xb0, 0xd2, 0xbc, 0x4c, 0x04, 0x2d, 0x33, 0x34, - 0x24, 0xb3, 0xc2, 0xd1, 0x61, 0xef, 0xff, 0x3f, 0x9d, 0x3a, 0xf8, 0x04, 0x6f, 0xa2, 0xb5, 0xd2, - 0xac, 0x03, 0x5e, 0x2c, 0x4b, 0x83, 0x41, 0xe3, 0xb2, 0x31, 0x6c, 0x47, 0x2e, 0x60, 0x3d, 0x68, - 0x93, 0x96, 0xb9, 0x49, 0x94, 0xce, 0x82, 0x13, 0xab, 0xec, 0x12, 0x2c, 0x80, 0x56, 0x86, 0xc6, - 0xc8, 0x14, 0x83, 0xa6, 0xd5, 0xea, 0x90, 0xf5, 0xc1, 0x37, 0x24, 0xe3, 0xd5, 0x8c, 0xb4, 0x8c, - 0x31, 0x38, 0xb5, 0x2a, 0xd8, 0xd4, 0xb4, 0xca, 0x0c, 0x9e, 0xa0, 0xf5, 0x40, 0x84, 0x59, 0x41, - 0x2c, 0x84, 0x73, 0xe9, 0x9e, 0xc6, 0x36, 0xf7, 0xa2, 0xbf, 0x98, 0x0d, 0xc1, 0xc3, 0x6a, 0x3c, - 0xdb, 0xdb, 0x1f, 0x31, 0xbe, 0xb7, 0x3a, 0xb7, 0x83, 0x47, 0x0e, 0x18, 0xfc, 0x34, 0x80, 0x1d, - 0x5f, 0x8c, 0x5d, 0x43, 0x53, 0xab, 0x8d, 0xfd, 0xd7, 0x1f, 0x75, 0x0f, 0xca, 0x77, 0x74, 0x54, - 0x31, 0xec, 0x0e, 0x2e, 0xd6, 0xd2, 0xd0, 0x6c, 0xdb, 0x7c, 0xdb, 0xb2, 0x73, 0x50, 0xb3, 0x9d, - 0x39, 0xf2, 0x2b, 0xb2, 0x5e, 0xe0, 0x1e, 0x20, 0x59, 0x6a, 0x43, 0x33, 0x83, 0x98, 0xdb, 0x4b, - 0xf8, 0xa3, 0x90, 0x3b, 0x5f, 0x78, 0xed, 0x0b, 0x9f, 0xd6, 0xbe, 0x44, 0x6d, 0x4b, 0xbf, 0x20, - 0xe6, 0xe3, 0x57, 0xd8, 0x37, 0x73, 0xdc, 0x3d, 0xde, 0xe0, 0xb9, 0xaa, 0x7f, 0xbb, 0x49, 0x97, - 0xf4, 0x5e, 0xce, 0x79, 0xac, 0x32, 0x91, 0xaa, 0x0f, 0x5c, 0x09, 0xe7, 0xaa, 0x59, 0xac, 0x44, - 0xaa, 0x9c, 0xf9, 0x46, 0xec, 0x39, 0x3d, 0x3f, 0xb3, 0xb9, 0xdb, 0xdf, 0x00, 0x00, 0x00, 0xff, - 0xff, 0x9d, 0x45, 0x64, 0x53, 0x53, 0x02, 0x00, 0x00, +var File_feast_types_FeatureRowExtended_proto protoreflect.FileDescriptor + +var file_feast_types_FeatureRowExtended_proto_rawDesc = []byte{ + 0x0a, 0x24, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2f, 0x46, 0x65, + 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x6f, 0x77, 0x45, 0x78, 0x74, 0x65, 0x6e, 0x64, 0x65, 0x64, + 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0b, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, + 0x70, 0x65, 0x73, 0x1a, 0x1f, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74, + 0x6f, 0x62, 0x75, 0x66, 0x2f, 0x74, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x2e, 0x70, + 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x1c, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x74, 0x79, 0x70, 0x65, + 0x73, 0x2f, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x6f, 0x77, 0x2e, 0x70, 0x72, 0x6f, + 0x74, 0x6f, 0x22, 0x76, 0x0a, 0x05, 0x45, 0x72, 0x72, 0x6f, 0x72, 0x12, 0x14, 0x0a, 0x05, 0x63, + 0x61, 0x75, 0x73, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x63, 0x61, 0x75, 0x73, + 0x65, 0x12, 0x1c, 0x0a, 0x09, 0x74, 0x72, 0x61, 0x6e, 0x73, 0x66, 0x6f, 0x72, 0x6d, 0x18, 0x02, + 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x74, 0x72, 0x61, 0x6e, 0x73, 0x66, 0x6f, 0x72, 0x6d, 0x12, + 0x18, 0x0a, 0x07, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, + 0x52, 0x07, 0x6d, 0x65, 0x73, 0x73, 0x61, 0x67, 0x65, 0x12, 0x1f, 0x0a, 0x0b, 0x73, 0x74, 0x61, + 0x63, 0x6b, 0x5f, 0x74, 0x72, 0x61, 0x63, 0x65, 0x18, 0x04, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0a, + 0x73, 0x74, 0x61, 0x63, 0x6b, 0x54, 0x72, 0x61, 0x63, 0x65, 0x22, 0x4f, 0x0a, 0x07, 0x41, 0x74, + 0x74, 0x65, 0x6d, 0x70, 0x74, 0x12, 0x1a, 0x0a, 0x08, 0x61, 0x74, 0x74, 0x65, 0x6d, 0x70, 0x74, + 0x73, 0x18, 0x01, 0x20, 0x01, 0x28, 0x05, 0x52, 0x08, 0x61, 0x74, 0x74, 0x65, 0x6d, 0x70, 0x74, + 0x73, 0x12, 0x28, 0x0a, 0x05, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, + 0x32, 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x45, + 0x72, 0x72, 0x6f, 0x72, 0x52, 0x05, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x22, 0xb3, 0x01, 0x0a, 0x12, + 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x6f, 0x77, 0x45, 0x78, 0x74, 0x65, 0x6e, 0x64, + 0x65, 0x64, 0x12, 0x29, 0x0a, 0x03, 0x72, 0x6f, 0x77, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, + 0x17, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x46, 0x65, + 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x6f, 0x77, 0x52, 0x03, 0x72, 0x6f, 0x77, 0x12, 0x37, 0x0a, + 0x0c, 0x6c, 0x61, 0x73, 0x74, 0x5f, 0x61, 0x74, 0x74, 0x65, 0x6d, 0x70, 0x74, 0x18, 0x02, 0x20, + 0x01, 0x28, 0x0b, 0x32, 0x14, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, + 0x73, 0x2e, 0x41, 0x74, 0x74, 0x65, 0x6d, 0x70, 0x74, 0x52, 0x0b, 0x6c, 0x61, 0x73, 0x74, 0x41, + 0x74, 0x74, 0x65, 0x6d, 0x70, 0x74, 0x12, 0x39, 0x0a, 0x0a, 0x66, 0x69, 0x72, 0x73, 0x74, 0x5f, + 0x73, 0x65, 0x65, 0x6e, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, + 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, + 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x09, 0x66, 0x69, 0x72, 0x73, 0x74, 0x53, 0x65, 0x65, + 0x6e, 0x42, 0x58, 0x0a, 0x0b, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, + 0x42, 0x17, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x6f, 0x77, 0x45, 0x78, 0x74, 0x65, + 0x6e, 0x64, 0x65, 0x64, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x30, 0x67, 0x69, 0x74, 0x68, 0x75, + 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, + 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, + 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x74, 0x79, 0x70, 0x65, 0x73, 0x62, 0x06, 0x70, 0x72, 0x6f, + 0x74, 0x6f, 0x33, +} + +var ( + file_feast_types_FeatureRowExtended_proto_rawDescOnce sync.Once + file_feast_types_FeatureRowExtended_proto_rawDescData = file_feast_types_FeatureRowExtended_proto_rawDesc +) + +func file_feast_types_FeatureRowExtended_proto_rawDescGZIP() []byte { + file_feast_types_FeatureRowExtended_proto_rawDescOnce.Do(func() { + file_feast_types_FeatureRowExtended_proto_rawDescData = protoimpl.X.CompressGZIP(file_feast_types_FeatureRowExtended_proto_rawDescData) + }) + return file_feast_types_FeatureRowExtended_proto_rawDescData +} + +var file_feast_types_FeatureRowExtended_proto_msgTypes = make([]protoimpl.MessageInfo, 3) +var file_feast_types_FeatureRowExtended_proto_goTypes = []interface{}{ + (*Error)(nil), // 0: feast.types.Error + (*Attempt)(nil), // 1: feast.types.Attempt + (*FeatureRowExtended)(nil), // 2: feast.types.FeatureRowExtended + (*FeatureRow)(nil), // 3: feast.types.FeatureRow + (*timestamp.Timestamp)(nil), // 4: google.protobuf.Timestamp +} +var file_feast_types_FeatureRowExtended_proto_depIdxs = []int32{ + 0, // 0: feast.types.Attempt.error:type_name -> feast.types.Error + 3, // 1: feast.types.FeatureRowExtended.row:type_name -> feast.types.FeatureRow + 1, // 2: feast.types.FeatureRowExtended.last_attempt:type_name -> feast.types.Attempt + 4, // 3: feast.types.FeatureRowExtended.first_seen:type_name -> google.protobuf.Timestamp + 4, // [4:4] is the sub-list for method output_type + 4, // [4:4] is the sub-list for method input_type + 4, // [4:4] is the sub-list for extension type_name + 4, // [4:4] is the sub-list for extension extendee + 0, // [0:4] is the sub-list for field type_name +} + +func init() { file_feast_types_FeatureRowExtended_proto_init() } +func file_feast_types_FeatureRowExtended_proto_init() { + if File_feast_types_FeatureRowExtended_proto != nil { + return + } + file_feast_types_FeatureRow_proto_init() + if !protoimpl.UnsafeEnabled { + file_feast_types_FeatureRowExtended_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Error); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_types_FeatureRowExtended_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Attempt); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_types_FeatureRowExtended_proto_msgTypes[2].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*FeatureRowExtended); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_feast_types_FeatureRowExtended_proto_rawDesc, + NumEnums: 0, + NumMessages: 3, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_feast_types_FeatureRowExtended_proto_goTypes, + DependencyIndexes: file_feast_types_FeatureRowExtended_proto_depIdxs, + MessageInfos: file_feast_types_FeatureRowExtended_proto_msgTypes, + }.Build() + File_feast_types_FeatureRowExtended_proto = out.File + file_feast_types_FeatureRowExtended_proto_rawDesc = nil + file_feast_types_FeatureRowExtended_proto_goTypes = nil + file_feast_types_FeatureRowExtended_proto_depIdxs = nil } diff --git a/sdk/go/protos/feast/types/Field.pb.go b/sdk/go/protos/feast/types/Field.pb.go index 0666d9bf1fb..bcd8505d6bd 100644 --- a/sdk/go/protos/feast/types/Field.pb.go +++ b/sdk/go/protos/feast/types/Field.pb.go @@ -1,91 +1,181 @@ +// +// Copyright 2018 The Feast Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + // Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.21.0 +// protoc v3.10.1 // source: feast/types/Field.proto package types import ( - fmt "fmt" proto "github.com/golang/protobuf/proto" - math "math" + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" ) -// Reference imports to suppress errors if they are not otherwise used. -var _ = proto.Marshal -var _ = fmt.Errorf -var _ = math.Inf +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) -// This is a compile-time assertion to ensure that this generated file -// is compatible with the proto package it is being compiled against. -// A compilation error at this line likely means your copy of the -// proto package needs to be updated. -const _ = proto.ProtoPackageIsVersion3 // please upgrade the proto package +// This is a compile-time assertion that a sufficiently up-to-date version +// of the legacy proto package is being used. +const _ = proto.ProtoPackageIsVersion4 type Field struct { - Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - Value *Value `protobuf:"bytes,2,opt,name=value,proto3" json:"value,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields -func (m *Field) Reset() { *m = Field{} } -func (m *Field) String() string { return proto.CompactTextString(m) } -func (*Field) ProtoMessage() {} -func (*Field) Descriptor() ([]byte, []int) { - return fileDescriptor_8c568a78dfaa9ca9, []int{0} + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` + Value *Value `protobuf:"bytes,2,opt,name=value,proto3" json:"value,omitempty"` } -func (m *Field) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_Field.Unmarshal(m, b) -} -func (m *Field) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_Field.Marshal(b, m, deterministic) -} -func (m *Field) XXX_Merge(src proto.Message) { - xxx_messageInfo_Field.Merge(m, src) +func (x *Field) Reset() { + *x = Field{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_types_Field_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *Field) XXX_Size() int { - return xxx_messageInfo_Field.Size(m) + +func (x *Field) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *Field) XXX_DiscardUnknown() { - xxx_messageInfo_Field.DiscardUnknown(m) + +func (*Field) ProtoMessage() {} + +func (x *Field) ProtoReflect() protoreflect.Message { + mi := &file_feast_types_Field_proto_msgTypes[0] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_Field proto.InternalMessageInfo +// Deprecated: Use Field.ProtoReflect.Descriptor instead. +func (*Field) Descriptor() ([]byte, []int) { + return file_feast_types_Field_proto_rawDescGZIP(), []int{0} +} -func (m *Field) GetName() string { - if m != nil { - return m.Name +func (x *Field) GetName() string { + if x != nil { + return x.Name } return "" } -func (m *Field) GetValue() *Value { - if m != nil { - return m.Value +func (x *Field) GetValue() *Value { + if x != nil { + return x.Value } return nil } -func init() { - proto.RegisterType((*Field)(nil), "feast.types.Field") +var File_feast_types_Field_proto protoreflect.FileDescriptor + +var file_feast_types_Field_proto_rawDesc = []byte{ + 0x0a, 0x17, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2f, 0x46, 0x69, + 0x65, 0x6c, 0x64, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0b, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x1a, 0x17, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x74, 0x79, + 0x70, 0x65, 0x73, 0x2f, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, + 0x45, 0x0a, 0x05, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, + 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x28, 0x0a, 0x05, + 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x52, + 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x42, 0x4b, 0x0a, 0x0b, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x74, 0x79, 0x70, 0x65, 0x73, 0x42, 0x0a, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x50, 0x72, 0x6f, 0x74, + 0x6f, 0x5a, 0x30, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, + 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, + 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x74, 0x79, + 0x70, 0x65, 0x73, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, +} + +var ( + file_feast_types_Field_proto_rawDescOnce sync.Once + file_feast_types_Field_proto_rawDescData = file_feast_types_Field_proto_rawDesc +) + +func file_feast_types_Field_proto_rawDescGZIP() []byte { + file_feast_types_Field_proto_rawDescOnce.Do(func() { + file_feast_types_Field_proto_rawDescData = protoimpl.X.CompressGZIP(file_feast_types_Field_proto_rawDescData) + }) + return file_feast_types_Field_proto_rawDescData } -func init() { - proto.RegisterFile("feast/types/Field.proto", fileDescriptor_8c568a78dfaa9ca9) +var file_feast_types_Field_proto_msgTypes = make([]protoimpl.MessageInfo, 1) +var file_feast_types_Field_proto_goTypes = []interface{}{ + (*Field)(nil), // 0: feast.types.Field + (*Value)(nil), // 1: feast.types.Value +} +var file_feast_types_Field_proto_depIdxs = []int32{ + 1, // 0: feast.types.Field.value:type_name -> feast.types.Value + 1, // [1:1] is the sub-list for method output_type + 1, // [1:1] is the sub-list for method input_type + 1, // [1:1] is the sub-list for extension type_name + 1, // [1:1] is the sub-list for extension extendee + 0, // [0:1] is the sub-list for field type_name } -var fileDescriptor_8c568a78dfaa9ca9 = []byte{ - // 165 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0xe2, 0x12, 0x4f, 0x4b, 0x4d, 0x2c, - 0x2e, 0xd1, 0x2f, 0xa9, 0x2c, 0x48, 0x2d, 0xd6, 0x77, 0xcb, 0x4c, 0xcd, 0x49, 0xd1, 0x2b, 0x28, - 0xca, 0x2f, 0xc9, 0x17, 0xe2, 0x06, 0x4b, 0xe8, 0x81, 0x25, 0xa4, 0x50, 0x54, 0x85, 0x25, 0xe6, - 0x94, 0xa6, 0x42, 0x54, 0x29, 0xb9, 0x72, 0xb1, 0x82, 0x35, 0x09, 0x09, 0x71, 0xb1, 0xe4, 0x25, - 0xe6, 0xa6, 0x4a, 0x30, 0x2a, 0x30, 0x6a, 0x70, 0x06, 0x81, 0xd9, 0x42, 0x1a, 0x5c, 0xac, 0x65, - 0x20, 0xb5, 0x12, 0x4c, 0x0a, 0x8c, 0x1a, 0xdc, 0x46, 0x42, 0x7a, 0x48, 0x46, 0xea, 0x81, 0x4d, - 0x09, 0x82, 0x28, 0x70, 0xf2, 0xe6, 0x42, 0xb6, 0xce, 0x89, 0x0b, 0x6c, 0x66, 0x00, 0xc8, 0x86, - 0x28, 0x83, 0xf4, 0xcc, 0x92, 0x8c, 0xd2, 0x24, 0xbd, 0xe4, 0xfc, 0x5c, 0xfd, 0xf4, 0xfc, 0xac, - 0xd4, 0x6c, 0x7d, 0x88, 0x5b, 0x8a, 0x53, 0xb2, 0xf5, 0xd3, 0xf3, 0xf5, 0xc1, 0xce, 0x28, 0xd6, - 0x47, 0x72, 0x5f, 0x12, 0x1b, 0x58, 0xcc, 0x18, 0x10, 0x00, 0x00, 0xff, 0xff, 0xef, 0xe8, 0xff, - 0x05, 0xdb, 0x00, 0x00, 0x00, +func init() { file_feast_types_Field_proto_init() } +func file_feast_types_Field_proto_init() { + if File_feast_types_Field_proto != nil { + return + } + file_feast_types_Value_proto_init() + if !protoimpl.UnsafeEnabled { + file_feast_types_Field_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Field); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_feast_types_Field_proto_rawDesc, + NumEnums: 0, + NumMessages: 1, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_feast_types_Field_proto_goTypes, + DependencyIndexes: file_feast_types_Field_proto_depIdxs, + MessageInfos: file_feast_types_Field_proto_msgTypes, + }.Build() + File_feast_types_Field_proto = out.File + file_feast_types_Field_proto_rawDesc = nil + file_feast_types_Field_proto_goTypes = nil + file_feast_types_Field_proto_depIdxs = nil } diff --git a/sdk/go/protos/feast/types/Value.pb.go b/sdk/go/protos/feast/types/Value.pb.go index f6ae73c2de2..ad574405aeb 100644 --- a/sdk/go/protos/feast/types/Value.pb.go +++ b/sdk/go/protos/feast/types/Value.pb.go @@ -1,24 +1,44 @@ +// +// Copyright 2018 The Feast Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + // Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.21.0 +// protoc v3.10.1 // source: feast/types/Value.proto package types import ( - fmt "fmt" proto "github.com/golang/protobuf/proto" - math "math" + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" ) -// Reference imports to suppress errors if they are not otherwise used. -var _ = proto.Marshal -var _ = fmt.Errorf -var _ = math.Inf +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) -// This is a compile-time assertion to ensure that this generated file -// is compatible with the proto package it is being compiled against. -// A compilation error at this line likely means your copy of the -// proto package needs to be updated. -const _ = proto.ProtoPackageIsVersion3 // please upgrade the proto package +// This is a compile-time assertion that a sufficiently up-to-date version +// of the legacy proto package is being used. +const _ = proto.ProtoPackageIsVersion4 type ValueType_Enum int32 @@ -40,86 +60,118 @@ const ( ValueType_BOOL_LIST ValueType_Enum = 17 ) -var ValueType_Enum_name = map[int32]string{ - 0: "INVALID", - 1: "BYTES", - 2: "STRING", - 3: "INT32", - 4: "INT64", - 5: "DOUBLE", - 6: "FLOAT", - 7: "BOOL", - 11: "BYTES_LIST", - 12: "STRING_LIST", - 13: "INT32_LIST", - 14: "INT64_LIST", - 15: "DOUBLE_LIST", - 16: "FLOAT_LIST", - 17: "BOOL_LIST", -} - -var ValueType_Enum_value = map[string]int32{ - "INVALID": 0, - "BYTES": 1, - "STRING": 2, - "INT32": 3, - "INT64": 4, - "DOUBLE": 5, - "FLOAT": 6, - "BOOL": 7, - "BYTES_LIST": 11, - "STRING_LIST": 12, - "INT32_LIST": 13, - "INT64_LIST": 14, - "DOUBLE_LIST": 15, - "FLOAT_LIST": 16, - "BOOL_LIST": 17, +// Enum value maps for ValueType_Enum. +var ( + ValueType_Enum_name = map[int32]string{ + 0: "INVALID", + 1: "BYTES", + 2: "STRING", + 3: "INT32", + 4: "INT64", + 5: "DOUBLE", + 6: "FLOAT", + 7: "BOOL", + 11: "BYTES_LIST", + 12: "STRING_LIST", + 13: "INT32_LIST", + 14: "INT64_LIST", + 15: "DOUBLE_LIST", + 16: "FLOAT_LIST", + 17: "BOOL_LIST", + } + ValueType_Enum_value = map[string]int32{ + "INVALID": 0, + "BYTES": 1, + "STRING": 2, + "INT32": 3, + "INT64": 4, + "DOUBLE": 5, + "FLOAT": 6, + "BOOL": 7, + "BYTES_LIST": 11, + "STRING_LIST": 12, + "INT32_LIST": 13, + "INT64_LIST": 14, + "DOUBLE_LIST": 15, + "FLOAT_LIST": 16, + "BOOL_LIST": 17, + } +) + +func (x ValueType_Enum) Enum() *ValueType_Enum { + p := new(ValueType_Enum) + *p = x + return p } func (x ValueType_Enum) String() string { - return proto.EnumName(ValueType_Enum_name, int32(x)) + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) } -func (ValueType_Enum) EnumDescriptor() ([]byte, []int) { - return fileDescriptor_47c504407d284ecc, []int{0, 0} +func (ValueType_Enum) Descriptor() protoreflect.EnumDescriptor { + return file_feast_types_Value_proto_enumTypes[0].Descriptor() } -type ValueType struct { - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` +func (ValueType_Enum) Type() protoreflect.EnumType { + return &file_feast_types_Value_proto_enumTypes[0] } -func (m *ValueType) Reset() { *m = ValueType{} } -func (m *ValueType) String() string { return proto.CompactTextString(m) } -func (*ValueType) ProtoMessage() {} -func (*ValueType) Descriptor() ([]byte, []int) { - return fileDescriptor_47c504407d284ecc, []int{0} +func (x ValueType_Enum) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) } -func (m *ValueType) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_ValueType.Unmarshal(m, b) +// Deprecated: Use ValueType_Enum.Descriptor instead. +func (ValueType_Enum) EnumDescriptor() ([]byte, []int) { + return file_feast_types_Value_proto_rawDescGZIP(), []int{0, 0} } -func (m *ValueType) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_ValueType.Marshal(b, m, deterministic) + +type ValueType struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields } -func (m *ValueType) XXX_Merge(src proto.Message) { - xxx_messageInfo_ValueType.Merge(m, src) + +func (x *ValueType) Reset() { + *x = ValueType{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_types_Value_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *ValueType) XXX_Size() int { - return xxx_messageInfo_ValueType.Size(m) + +func (x *ValueType) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *ValueType) XXX_DiscardUnknown() { - xxx_messageInfo_ValueType.DiscardUnknown(m) + +func (*ValueType) ProtoMessage() {} + +func (x *ValueType) ProtoReflect() protoreflect.Message { + mi := &file_feast_types_Value_proto_msgTypes[0] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_ValueType proto.InternalMessageInfo +// Deprecated: Use ValueType.ProtoReflect.Descriptor instead. +func (*ValueType) Descriptor() ([]byte, []int) { + return file_feast_types_Value_proto_rawDescGZIP(), []int{0} +} type Value struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + // ValueType is referenced by the metadata types, FeatureInfo and EntityInfo. // The enum values do not have to match the oneof val field ids, but they should. // - // Types that are valid to be assigned to Val: + // Types that are assignable to Val: // *Value_BytesVal // *Value_StringVal // *Value_Int32Val @@ -134,36 +186,145 @@ type Value struct { // *Value_DoubleListVal // *Value_FloatListVal // *Value_BoolListVal - Val isValue_Val `protobuf_oneof:"val"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + Val isValue_Val `protobuf_oneof:"val"` +} + +func (x *Value) Reset() { + *x = Value{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_types_Value_proto_msgTypes[1] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *Value) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *Value) Reset() { *m = Value{} } -func (m *Value) String() string { return proto.CompactTextString(m) } -func (*Value) ProtoMessage() {} +func (*Value) ProtoMessage() {} + +func (x *Value) ProtoReflect() protoreflect.Message { + mi := &file_feast_types_Value_proto_msgTypes[1] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use Value.ProtoReflect.Descriptor instead. func (*Value) Descriptor() ([]byte, []int) { - return fileDescriptor_47c504407d284ecc, []int{1} + return file_feast_types_Value_proto_rawDescGZIP(), []int{1} } -func (m *Value) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_Value.Unmarshal(m, b) +func (m *Value) GetVal() isValue_Val { + if m != nil { + return m.Val + } + return nil } -func (m *Value) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_Value.Marshal(b, m, deterministic) + +func (x *Value) GetBytesVal() []byte { + if x, ok := x.GetVal().(*Value_BytesVal); ok { + return x.BytesVal + } + return nil } -func (m *Value) XXX_Merge(src proto.Message) { - xxx_messageInfo_Value.Merge(m, src) + +func (x *Value) GetStringVal() string { + if x, ok := x.GetVal().(*Value_StringVal); ok { + return x.StringVal + } + return "" +} + +func (x *Value) GetInt32Val() int32 { + if x, ok := x.GetVal().(*Value_Int32Val); ok { + return x.Int32Val + } + return 0 +} + +func (x *Value) GetInt64Val() int64 { + if x, ok := x.GetVal().(*Value_Int64Val); ok { + return x.Int64Val + } + return 0 } -func (m *Value) XXX_Size() int { - return xxx_messageInfo_Value.Size(m) + +func (x *Value) GetDoubleVal() float64 { + if x, ok := x.GetVal().(*Value_DoubleVal); ok { + return x.DoubleVal + } + return 0 } -func (m *Value) XXX_DiscardUnknown() { - xxx_messageInfo_Value.DiscardUnknown(m) + +func (x *Value) GetFloatVal() float32 { + if x, ok := x.GetVal().(*Value_FloatVal); ok { + return x.FloatVal + } + return 0 } -var xxx_messageInfo_Value proto.InternalMessageInfo +func (x *Value) GetBoolVal() bool { + if x, ok := x.GetVal().(*Value_BoolVal); ok { + return x.BoolVal + } + return false +} + +func (x *Value) GetBytesListVal() *BytesList { + if x, ok := x.GetVal().(*Value_BytesListVal); ok { + return x.BytesListVal + } + return nil +} + +func (x *Value) GetStringListVal() *StringList { + if x, ok := x.GetVal().(*Value_StringListVal); ok { + return x.StringListVal + } + return nil +} + +func (x *Value) GetInt32ListVal() *Int32List { + if x, ok := x.GetVal().(*Value_Int32ListVal); ok { + return x.Int32ListVal + } + return nil +} + +func (x *Value) GetInt64ListVal() *Int64List { + if x, ok := x.GetVal().(*Value_Int64ListVal); ok { + return x.Int64ListVal + } + return nil +} + +func (x *Value) GetDoubleListVal() *DoubleList { + if x, ok := x.GetVal().(*Value_DoubleListVal); ok { + return x.DoubleListVal + } + return nil +} + +func (x *Value) GetFloatListVal() *FloatList { + if x, ok := x.GetVal().(*Value_FloatListVal); ok { + return x.FloatListVal + } + return nil +} + +func (x *Value) GetBoolListVal() *BoolList { + if x, ok := x.GetVal().(*Value_BoolListVal); ok { + return x.BoolListVal + } + return nil +} type isValue_Val interface { isValue_Val() @@ -253,459 +414,608 @@ func (*Value_FloatListVal) isValue_Val() {} func (*Value_BoolListVal) isValue_Val() {} -func (m *Value) GetVal() isValue_Val { - if m != nil { - return m.Val - } - return nil -} - -func (m *Value) GetBytesVal() []byte { - if x, ok := m.GetVal().(*Value_BytesVal); ok { - return x.BytesVal - } - return nil -} +type BytesList struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields -func (m *Value) GetStringVal() string { - if x, ok := m.GetVal().(*Value_StringVal); ok { - return x.StringVal - } - return "" + Val [][]byte `protobuf:"bytes,1,rep,name=val,proto3" json:"val,omitempty"` } -func (m *Value) GetInt32Val() int32 { - if x, ok := m.GetVal().(*Value_Int32Val); ok { - return x.Int32Val +func (x *BytesList) Reset() { + *x = BytesList{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_types_Value_proto_msgTypes[2] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) } - return 0 } -func (m *Value) GetInt64Val() int64 { - if x, ok := m.GetVal().(*Value_Int64Val); ok { - return x.Int64Val - } - return 0 +func (x *BytesList) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *Value) GetDoubleVal() float64 { - if x, ok := m.GetVal().(*Value_DoubleVal); ok { - return x.DoubleVal - } - return 0 -} +func (*BytesList) ProtoMessage() {} -func (m *Value) GetFloatVal() float32 { - if x, ok := m.GetVal().(*Value_FloatVal); ok { - return x.FloatVal +func (x *BytesList) ProtoReflect() protoreflect.Message { + mi := &file_feast_types_Value_proto_msgTypes[2] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms } - return 0 + return mi.MessageOf(x) } -func (m *Value) GetBoolVal() bool { - if x, ok := m.GetVal().(*Value_BoolVal); ok { - return x.BoolVal - } - return false +// Deprecated: Use BytesList.ProtoReflect.Descriptor instead. +func (*BytesList) Descriptor() ([]byte, []int) { + return file_feast_types_Value_proto_rawDescGZIP(), []int{2} } -func (m *Value) GetBytesListVal() *BytesList { - if x, ok := m.GetVal().(*Value_BytesListVal); ok { - return x.BytesListVal +func (x *BytesList) GetVal() [][]byte { + if x != nil { + return x.Val } return nil } -func (m *Value) GetStringListVal() *StringList { - if x, ok := m.GetVal().(*Value_StringListVal); ok { - return x.StringListVal - } - return nil +type StringList struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Val []string `protobuf:"bytes,1,rep,name=val,proto3" json:"val,omitempty"` } -func (m *Value) GetInt32ListVal() *Int32List { - if x, ok := m.GetVal().(*Value_Int32ListVal); ok { - return x.Int32ListVal +func (x *StringList) Reset() { + *x = StringList{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_types_Value_proto_msgTypes[3] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) } - return nil } -func (m *Value) GetInt64ListVal() *Int64List { - if x, ok := m.GetVal().(*Value_Int64ListVal); ok { - return x.Int64ListVal - } - return nil +func (x *StringList) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *Value) GetDoubleListVal() *DoubleList { - if x, ok := m.GetVal().(*Value_DoubleListVal); ok { - return x.DoubleListVal +func (*StringList) ProtoMessage() {} + +func (x *StringList) ProtoReflect() protoreflect.Message { + mi := &file_feast_types_Value_proto_msgTypes[3] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms } - return nil + return mi.MessageOf(x) } -func (m *Value) GetFloatListVal() *FloatList { - if x, ok := m.GetVal().(*Value_FloatListVal); ok { - return x.FloatListVal - } - return nil +// Deprecated: Use StringList.ProtoReflect.Descriptor instead. +func (*StringList) Descriptor() ([]byte, []int) { + return file_feast_types_Value_proto_rawDescGZIP(), []int{3} } -func (m *Value) GetBoolListVal() *BoolList { - if x, ok := m.GetVal().(*Value_BoolListVal); ok { - return x.BoolListVal +func (x *StringList) GetVal() []string { + if x != nil { + return x.Val } return nil } -// XXX_OneofWrappers is for the internal use of the proto package. -func (*Value) XXX_OneofWrappers() []interface{} { - return []interface{}{ - (*Value_BytesVal)(nil), - (*Value_StringVal)(nil), - (*Value_Int32Val)(nil), - (*Value_Int64Val)(nil), - (*Value_DoubleVal)(nil), - (*Value_FloatVal)(nil), - (*Value_BoolVal)(nil), - (*Value_BytesListVal)(nil), - (*Value_StringListVal)(nil), - (*Value_Int32ListVal)(nil), - (*Value_Int64ListVal)(nil), - (*Value_DoubleListVal)(nil), - (*Value_FloatListVal)(nil), - (*Value_BoolListVal)(nil), - } -} +type Int32List struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields -type BytesList struct { - Val [][]byte `protobuf:"bytes,1,rep,name=val,proto3" json:"val,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + Val []int32 `protobuf:"varint,1,rep,packed,name=val,proto3" json:"val,omitempty"` } -func (m *BytesList) Reset() { *m = BytesList{} } -func (m *BytesList) String() string { return proto.CompactTextString(m) } -func (*BytesList) ProtoMessage() {} -func (*BytesList) Descriptor() ([]byte, []int) { - return fileDescriptor_47c504407d284ecc, []int{2} +func (x *Int32List) Reset() { + *x = Int32List{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_types_Value_proto_msgTypes[4] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *BytesList) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_BytesList.Unmarshal(m, b) -} -func (m *BytesList) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_BytesList.Marshal(b, m, deterministic) -} -func (m *BytesList) XXX_Merge(src proto.Message) { - xxx_messageInfo_BytesList.Merge(m, src) -} -func (m *BytesList) XXX_Size() int { - return xxx_messageInfo_BytesList.Size(m) -} -func (m *BytesList) XXX_DiscardUnknown() { - xxx_messageInfo_BytesList.DiscardUnknown(m) +func (x *Int32List) String() string { + return protoimpl.X.MessageStringOf(x) } -var xxx_messageInfo_BytesList proto.InternalMessageInfo +func (*Int32List) ProtoMessage() {} -func (m *BytesList) GetVal() [][]byte { - if m != nil { - return m.Val +func (x *Int32List) ProtoReflect() protoreflect.Message { + mi := &file_feast_types_Value_proto_msgTypes[4] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms } - return nil -} - -type StringList struct { - Val []string `protobuf:"bytes,1,rep,name=val,proto3" json:"val,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + return mi.MessageOf(x) } -func (m *StringList) Reset() { *m = StringList{} } -func (m *StringList) String() string { return proto.CompactTextString(m) } -func (*StringList) ProtoMessage() {} -func (*StringList) Descriptor() ([]byte, []int) { - return fileDescriptor_47c504407d284ecc, []int{3} -} - -func (m *StringList) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_StringList.Unmarshal(m, b) -} -func (m *StringList) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_StringList.Marshal(b, m, deterministic) -} -func (m *StringList) XXX_Merge(src proto.Message) { - xxx_messageInfo_StringList.Merge(m, src) -} -func (m *StringList) XXX_Size() int { - return xxx_messageInfo_StringList.Size(m) -} -func (m *StringList) XXX_DiscardUnknown() { - xxx_messageInfo_StringList.DiscardUnknown(m) +// Deprecated: Use Int32List.ProtoReflect.Descriptor instead. +func (*Int32List) Descriptor() ([]byte, []int) { + return file_feast_types_Value_proto_rawDescGZIP(), []int{4} } -var xxx_messageInfo_StringList proto.InternalMessageInfo - -func (m *StringList) GetVal() []string { - if m != nil { - return m.Val +func (x *Int32List) GetVal() []int32 { + if x != nil { + return x.Val } return nil } -type Int32List struct { - Val []int32 `protobuf:"varint,1,rep,packed,name=val,proto3" json:"val,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} +type Int64List struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields -func (m *Int32List) Reset() { *m = Int32List{} } -func (m *Int32List) String() string { return proto.CompactTextString(m) } -func (*Int32List) ProtoMessage() {} -func (*Int32List) Descriptor() ([]byte, []int) { - return fileDescriptor_47c504407d284ecc, []int{4} + Val []int64 `protobuf:"varint,1,rep,packed,name=val,proto3" json:"val,omitempty"` } -func (m *Int32List) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_Int32List.Unmarshal(m, b) -} -func (m *Int32List) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_Int32List.Marshal(b, m, deterministic) -} -func (m *Int32List) XXX_Merge(src proto.Message) { - xxx_messageInfo_Int32List.Merge(m, src) -} -func (m *Int32List) XXX_Size() int { - return xxx_messageInfo_Int32List.Size(m) +func (x *Int64List) Reset() { + *x = Int64List{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_types_Value_proto_msgTypes[5] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *Int32List) XXX_DiscardUnknown() { - xxx_messageInfo_Int32List.DiscardUnknown(m) + +func (x *Int64List) String() string { + return protoimpl.X.MessageStringOf(x) } -var xxx_messageInfo_Int32List proto.InternalMessageInfo +func (*Int64List) ProtoMessage() {} -func (m *Int32List) GetVal() []int32 { - if m != nil { - return m.Val +func (x *Int64List) ProtoReflect() protoreflect.Message { + mi := &file_feast_types_Value_proto_msgTypes[5] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms } - return nil -} - -type Int64List struct { - Val []int64 `protobuf:"varint,1,rep,packed,name=val,proto3" json:"val,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` + return mi.MessageOf(x) } -func (m *Int64List) Reset() { *m = Int64List{} } -func (m *Int64List) String() string { return proto.CompactTextString(m) } -func (*Int64List) ProtoMessage() {} +// Deprecated: Use Int64List.ProtoReflect.Descriptor instead. func (*Int64List) Descriptor() ([]byte, []int) { - return fileDescriptor_47c504407d284ecc, []int{5} + return file_feast_types_Value_proto_rawDescGZIP(), []int{5} } -func (m *Int64List) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_Int64List.Unmarshal(m, b) -} -func (m *Int64List) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_Int64List.Marshal(b, m, deterministic) -} -func (m *Int64List) XXX_Merge(src proto.Message) { - xxx_messageInfo_Int64List.Merge(m, src) -} -func (m *Int64List) XXX_Size() int { - return xxx_messageInfo_Int64List.Size(m) -} -func (m *Int64List) XXX_DiscardUnknown() { - xxx_messageInfo_Int64List.DiscardUnknown(m) -} - -var xxx_messageInfo_Int64List proto.InternalMessageInfo - -func (m *Int64List) GetVal() []int64 { - if m != nil { - return m.Val +func (x *Int64List) GetVal() []int64 { + if x != nil { + return x.Val } return nil } type DoubleList struct { - Val []float64 `protobuf:"fixed64,1,rep,packed,name=val,proto3" json:"val,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields -func (m *DoubleList) Reset() { *m = DoubleList{} } -func (m *DoubleList) String() string { return proto.CompactTextString(m) } -func (*DoubleList) ProtoMessage() {} -func (*DoubleList) Descriptor() ([]byte, []int) { - return fileDescriptor_47c504407d284ecc, []int{6} + Val []float64 `protobuf:"fixed64,1,rep,packed,name=val,proto3" json:"val,omitempty"` } -func (m *DoubleList) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_DoubleList.Unmarshal(m, b) -} -func (m *DoubleList) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_DoubleList.Marshal(b, m, deterministic) -} -func (m *DoubleList) XXX_Merge(src proto.Message) { - xxx_messageInfo_DoubleList.Merge(m, src) +func (x *DoubleList) Reset() { + *x = DoubleList{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_types_Value_proto_msgTypes[6] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *DoubleList) XXX_Size() int { - return xxx_messageInfo_DoubleList.Size(m) + +func (x *DoubleList) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *DoubleList) XXX_DiscardUnknown() { - xxx_messageInfo_DoubleList.DiscardUnknown(m) + +func (*DoubleList) ProtoMessage() {} + +func (x *DoubleList) ProtoReflect() protoreflect.Message { + mi := &file_feast_types_Value_proto_msgTypes[6] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_DoubleList proto.InternalMessageInfo +// Deprecated: Use DoubleList.ProtoReflect.Descriptor instead. +func (*DoubleList) Descriptor() ([]byte, []int) { + return file_feast_types_Value_proto_rawDescGZIP(), []int{6} +} -func (m *DoubleList) GetVal() []float64 { - if m != nil { - return m.Val +func (x *DoubleList) GetVal() []float64 { + if x != nil { + return x.Val } return nil } type FloatList struct { - Val []float32 `protobuf:"fixed32,1,rep,packed,name=val,proto3" json:"val,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields -func (m *FloatList) Reset() { *m = FloatList{} } -func (m *FloatList) String() string { return proto.CompactTextString(m) } -func (*FloatList) ProtoMessage() {} -func (*FloatList) Descriptor() ([]byte, []int) { - return fileDescriptor_47c504407d284ecc, []int{7} + Val []float32 `protobuf:"fixed32,1,rep,packed,name=val,proto3" json:"val,omitempty"` } -func (m *FloatList) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_FloatList.Unmarshal(m, b) -} -func (m *FloatList) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_FloatList.Marshal(b, m, deterministic) -} -func (m *FloatList) XXX_Merge(src proto.Message) { - xxx_messageInfo_FloatList.Merge(m, src) +func (x *FloatList) Reset() { + *x = FloatList{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_types_Value_proto_msgTypes[7] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *FloatList) XXX_Size() int { - return xxx_messageInfo_FloatList.Size(m) + +func (x *FloatList) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *FloatList) XXX_DiscardUnknown() { - xxx_messageInfo_FloatList.DiscardUnknown(m) + +func (*FloatList) ProtoMessage() {} + +func (x *FloatList) ProtoReflect() protoreflect.Message { + mi := &file_feast_types_Value_proto_msgTypes[7] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_FloatList proto.InternalMessageInfo +// Deprecated: Use FloatList.ProtoReflect.Descriptor instead. +func (*FloatList) Descriptor() ([]byte, []int) { + return file_feast_types_Value_proto_rawDescGZIP(), []int{7} +} -func (m *FloatList) GetVal() []float32 { - if m != nil { - return m.Val +func (x *FloatList) GetVal() []float32 { + if x != nil { + return x.Val } return nil } type BoolList struct { - Val []bool `protobuf:"varint,1,rep,packed,name=val,proto3" json:"val,omitempty"` - XXX_NoUnkeyedLiteral struct{} `json:"-"` - XXX_unrecognized []byte `json:"-"` - XXX_sizecache int32 `json:"-"` -} + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields -func (m *BoolList) Reset() { *m = BoolList{} } -func (m *BoolList) String() string { return proto.CompactTextString(m) } -func (*BoolList) ProtoMessage() {} -func (*BoolList) Descriptor() ([]byte, []int) { - return fileDescriptor_47c504407d284ecc, []int{8} + Val []bool `protobuf:"varint,1,rep,packed,name=val,proto3" json:"val,omitempty"` } -func (m *BoolList) XXX_Unmarshal(b []byte) error { - return xxx_messageInfo_BoolList.Unmarshal(m, b) -} -func (m *BoolList) XXX_Marshal(b []byte, deterministic bool) ([]byte, error) { - return xxx_messageInfo_BoolList.Marshal(b, m, deterministic) -} -func (m *BoolList) XXX_Merge(src proto.Message) { - xxx_messageInfo_BoolList.Merge(m, src) +func (x *BoolList) Reset() { + *x = BoolList{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_types_Value_proto_msgTypes[8] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } } -func (m *BoolList) XXX_Size() int { - return xxx_messageInfo_BoolList.Size(m) + +func (x *BoolList) String() string { + return protoimpl.X.MessageStringOf(x) } -func (m *BoolList) XXX_DiscardUnknown() { - xxx_messageInfo_BoolList.DiscardUnknown(m) + +func (*BoolList) ProtoMessage() {} + +func (x *BoolList) ProtoReflect() protoreflect.Message { + mi := &file_feast_types_Value_proto_msgTypes[8] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) } -var xxx_messageInfo_BoolList proto.InternalMessageInfo +// Deprecated: Use BoolList.ProtoReflect.Descriptor instead. +func (*BoolList) Descriptor() ([]byte, []int) { + return file_feast_types_Value_proto_rawDescGZIP(), []int{8} +} -func (m *BoolList) GetVal() []bool { - if m != nil { - return m.Val +func (x *BoolList) GetVal() []bool { + if x != nil { + return x.Val } return nil } -func init() { - proto.RegisterEnum("feast.types.ValueType_Enum", ValueType_Enum_name, ValueType_Enum_value) - proto.RegisterType((*ValueType)(nil), "feast.types.ValueType") - proto.RegisterType((*Value)(nil), "feast.types.Value") - proto.RegisterType((*BytesList)(nil), "feast.types.BytesList") - proto.RegisterType((*StringList)(nil), "feast.types.StringList") - proto.RegisterType((*Int32List)(nil), "feast.types.Int32List") - proto.RegisterType((*Int64List)(nil), "feast.types.Int64List") - proto.RegisterType((*DoubleList)(nil), "feast.types.DoubleList") - proto.RegisterType((*FloatList)(nil), "feast.types.FloatList") - proto.RegisterType((*BoolList)(nil), "feast.types.BoolList") -} - -func init() { - proto.RegisterFile("feast/types/Value.proto", fileDescriptor_47c504407d284ecc) -} - -var fileDescriptor_47c504407d284ecc = []byte{ - // 600 bytes of a gzipped FileDescriptorProto - 0x1f, 0x8b, 0x08, 0x00, 0x00, 0x00, 0x00, 0x00, 0x02, 0xff, 0x6c, 0x94, 0xcf, 0x6e, 0x9b, 0x40, - 0x10, 0xc6, 0xbd, 0xc6, 0xd8, 0x30, 0xf8, 0xcf, 0x06, 0xa9, 0x4d, 0xa4, 0x36, 0x2d, 0xf2, 0x89, - 0x93, 0xa9, 0x12, 0xc4, 0xa5, 0x52, 0xa5, 0xa0, 0x24, 0x35, 0x2a, 0x8a, 0x2b, 0x4c, 0x2d, 0xb5, - 0x97, 0x08, 0x1a, 0xe2, 0xd2, 0x90, 0x10, 0x05, 0x1c, 0xc9, 0xef, 0xd4, 0xa7, 0xe9, 0x13, 0xf4, - 0x51, 0xaa, 0x9d, 0x5d, 0xd6, 0x44, 0xf2, 0xcd, 0xf3, 0xfd, 0xe6, 0xfb, 0xd8, 0xd9, 0x91, 0x17, - 0x0e, 0x6f, 0xb3, 0xa4, 0xaa, 0x9d, 0x7a, 0xfb, 0x98, 0x55, 0xce, 0x2a, 0x29, 0x36, 0xd9, 0xec, - 0xf1, 0xa9, 0xac, 0x4b, 0xd3, 0x40, 0x30, 0x43, 0x30, 0xfd, 0x47, 0x40, 0x47, 0x18, 0x6f, 0x1f, - 0xb3, 0xe9, 0x5f, 0x02, 0xbd, 0x8b, 0x87, 0xcd, 0xbd, 0x69, 0xc0, 0x20, 0xb8, 0x5a, 0x9d, 0x85, - 0xc1, 0x39, 0xed, 0x98, 0x3a, 0xa8, 0xfe, 0xf7, 0xf8, 0x62, 0x49, 0x89, 0x09, 0xd0, 0x5f, 0xc6, - 0x51, 0x70, 0xf5, 0x99, 0x76, 0x99, 0x1c, 0x5c, 0xc5, 0xa7, 0x27, 0x54, 0x11, 0x3f, 0x3d, 0x97, - 0xf6, 0x58, 0xc7, 0xf9, 0xe2, 0x9b, 0x1f, 0x5e, 0x50, 0x95, 0xc9, 0x97, 0xe1, 0xe2, 0x2c, 0xa6, - 0x7d, 0x53, 0x83, 0x9e, 0xbf, 0x58, 0x84, 0x74, 0x60, 0x8e, 0x01, 0x30, 0xed, 0x3a, 0x0c, 0x96, - 0x31, 0x35, 0xcc, 0x09, 0x18, 0x3c, 0x92, 0x0b, 0x43, 0xd6, 0x80, 0xb9, 0xbc, 0x1e, 0x89, 0xda, - 0x73, 0x79, 0x3d, 0x66, 0x06, 0xfe, 0x05, 0x2e, 0x4c, 0x58, 0x03, 0x7e, 0x86, 0xd7, 0xd4, 0x1c, - 0x81, 0xce, 0xbe, 0xc5, 0xcb, 0x83, 0xe9, 0x1f, 0x15, 0x54, 0x1c, 0xd1, 0x3c, 0x06, 0x3d, 0xdd, - 0xd6, 0x59, 0x75, 0xfd, 0x9c, 0x14, 0x47, 0xc4, 0x22, 0xf6, 0x70, 0xde, 0x89, 0x34, 0x94, 0x56, - 0x49, 0x61, 0xbe, 0x07, 0xa8, 0xea, 0xa7, 0xfc, 0x61, 0x8d, 0xbc, 0x6b, 0x11, 0x5b, 0x9f, 0x77, - 0x22, 0x9d, 0x6b, 0xac, 0xe1, 0x18, 0xf4, 0xfc, 0xa1, 0x3e, 0x3d, 0x41, 0xae, 0x58, 0xc4, 0x56, - 0x99, 0x1f, 0xa5, 0x1d, 0xf6, 0x5c, 0xc4, 0x3d, 0x8b, 0xd8, 0x8a, 0xc0, 0x9e, 0x2b, 0xe2, 0x6f, - 0xca, 0x4d, 0x5a, 0x64, 0xc8, 0x55, 0x8b, 0xd8, 0x84, 0xc5, 0x73, 0x4d, 0xf8, 0x6f, 0x8b, 0x32, - 0xa9, 0x91, 0xf7, 0x2d, 0x62, 0x77, 0x99, 0x1f, 0x25, 0x86, 0xdf, 0x80, 0x96, 0x96, 0x65, 0x81, - 0x74, 0x60, 0x11, 0x5b, 0x9b, 0x77, 0xa2, 0x01, 0x53, 0x18, 0xfc, 0x04, 0x63, 0x3e, 0x5a, 0x91, - 0x57, 0x3c, 0xc0, 0xb0, 0x88, 0x6d, 0x9c, 0xbc, 0x9e, 0xb5, 0xb6, 0x3d, 0xf3, 0x59, 0x4b, 0x98, - 0x57, 0xf5, 0xbc, 0x13, 0x0d, 0xd3, 0xa6, 0x60, 0xfe, 0x33, 0x98, 0x88, 0xd9, 0x65, 0xc0, 0x10, - 0x03, 0x0e, 0x5f, 0x04, 0x2c, 0xb1, 0x47, 0x24, 0x8c, 0x2a, 0x59, 0x89, 0x23, 0xf0, 0xdb, 0x91, - 0x09, 0xa3, 0x3d, 0x47, 0x08, 0x58, 0x4b, 0x73, 0x84, 0xbc, 0x29, 0x76, 0x7e, 0xcf, 0xdd, 0xf9, - 0xc7, 0xfb, 0xfd, 0x9e, 0xdb, 0xf2, 0xf3, 0x42, 0x8c, 0x20, 0xee, 0x57, 0x06, 0x4c, 0xf6, 0x8c, - 0x70, 0x8e, 0x3d, 0xcd, 0x08, 0x37, 0xb2, 0x12, 0x47, 0xe0, 0x1b, 0x90, 0x09, 0x74, 0xcf, 0x11, - 0x2e, 0x59, 0x4b, 0x73, 0x84, 0xdb, 0xa6, 0x60, 0xfe, 0x8f, 0x30, 0xc2, 0x15, 0x49, 0xfb, 0x01, - 0xda, 0x5f, 0xbd, 0x5c, 0x42, 0x59, 0x16, 0xc2, 0x6d, 0xa4, 0xe2, 0xf7, 0x2a, 0x29, 0x7c, 0x15, - 0x94, 0xe7, 0xa4, 0x98, 0x1e, 0x83, 0x2e, 0xd7, 0x64, 0x52, 0xd4, 0x8e, 0x88, 0xa5, 0xd8, 0xc3, - 0x08, 0xf1, 0x3b, 0x80, 0xdd, 0x12, 0xda, 0x5c, 0x8f, 0x1a, 0xbb, 0xbc, 0xe2, 0x36, 0x56, 0xdb, - 0x98, 0x5f, 0x5a, 0x1b, 0x2b, 0x32, 0x7d, 0x77, 0x3f, 0x6d, 0x4e, 0xa4, 0x5d, 0x4e, 0xdf, 0xc6, - 0x5d, 0x8e, 0xdf, 0x82, 0xd6, 0x4c, 0xd7, 0xa6, 0x1a, 0x52, 0xff, 0x0b, 0xb4, 0x9f, 0x1e, 0x1f, - 0xf0, 0x4f, 0xf9, 0x95, 0xbd, 0x49, 0x3f, 0x3e, 0xac, 0xf3, 0xfa, 0xd7, 0x26, 0x9d, 0xfd, 0x2c, - 0xef, 0x9d, 0x75, 0xf9, 0x3b, 0xbb, 0x73, 0xf8, 0xeb, 0x55, 0xdd, 0xdc, 0x39, 0xeb, 0xd2, 0xc1, - 0x87, 0xab, 0x72, 0x5a, 0x2f, 0x5a, 0xda, 0x47, 0xed, 0xf4, 0x7f, 0x00, 0x00, 0x00, 0xff, 0xff, - 0xa3, 0x3f, 0x67, 0x48, 0xe7, 0x04, 0x00, 0x00, +var File_feast_types_Value_proto protoreflect.FileDescriptor + +var file_feast_types_Value_proto_rawDesc = []byte{ + 0x0a, 0x17, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2f, 0x56, 0x61, + 0x6c, 0x75, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0b, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x22, 0xe0, 0x01, 0x0a, 0x09, 0x56, 0x61, 0x6c, 0x75, 0x65, + 0x54, 0x79, 0x70, 0x65, 0x22, 0xd2, 0x01, 0x0a, 0x04, 0x45, 0x6e, 0x75, 0x6d, 0x12, 0x0b, 0x0a, + 0x07, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, 0x12, 0x09, 0x0a, 0x05, 0x42, 0x59, + 0x54, 0x45, 0x53, 0x10, 0x01, 0x12, 0x0a, 0x0a, 0x06, 0x53, 0x54, 0x52, 0x49, 0x4e, 0x47, 0x10, + 0x02, 0x12, 0x09, 0x0a, 0x05, 0x49, 0x4e, 0x54, 0x33, 0x32, 0x10, 0x03, 0x12, 0x09, 0x0a, 0x05, + 0x49, 0x4e, 0x54, 0x36, 0x34, 0x10, 0x04, 0x12, 0x0a, 0x0a, 0x06, 0x44, 0x4f, 0x55, 0x42, 0x4c, + 0x45, 0x10, 0x05, 0x12, 0x09, 0x0a, 0x05, 0x46, 0x4c, 0x4f, 0x41, 0x54, 0x10, 0x06, 0x12, 0x08, + 0x0a, 0x04, 0x42, 0x4f, 0x4f, 0x4c, 0x10, 0x07, 0x12, 0x0e, 0x0a, 0x0a, 0x42, 0x59, 0x54, 0x45, + 0x53, 0x5f, 0x4c, 0x49, 0x53, 0x54, 0x10, 0x0b, 0x12, 0x0f, 0x0a, 0x0b, 0x53, 0x54, 0x52, 0x49, + 0x4e, 0x47, 0x5f, 0x4c, 0x49, 0x53, 0x54, 0x10, 0x0c, 0x12, 0x0e, 0x0a, 0x0a, 0x49, 0x4e, 0x54, + 0x33, 0x32, 0x5f, 0x4c, 0x49, 0x53, 0x54, 0x10, 0x0d, 0x12, 0x0e, 0x0a, 0x0a, 0x49, 0x4e, 0x54, + 0x36, 0x34, 0x5f, 0x4c, 0x49, 0x53, 0x54, 0x10, 0x0e, 0x12, 0x0f, 0x0a, 0x0b, 0x44, 0x4f, 0x55, + 0x42, 0x4c, 0x45, 0x5f, 0x4c, 0x49, 0x53, 0x54, 0x10, 0x0f, 0x12, 0x0e, 0x0a, 0x0a, 0x46, 0x4c, + 0x4f, 0x41, 0x54, 0x5f, 0x4c, 0x49, 0x53, 0x54, 0x10, 0x10, 0x12, 0x0d, 0x0a, 0x09, 0x42, 0x4f, + 0x4f, 0x4c, 0x5f, 0x4c, 0x49, 0x53, 0x54, 0x10, 0x11, 0x22, 0xac, 0x05, 0x0a, 0x05, 0x56, 0x61, + 0x6c, 0x75, 0x65, 0x12, 0x1d, 0x0a, 0x09, 0x62, 0x79, 0x74, 0x65, 0x73, 0x5f, 0x76, 0x61, 0x6c, + 0x18, 0x01, 0x20, 0x01, 0x28, 0x0c, 0x48, 0x00, 0x52, 0x08, 0x62, 0x79, 0x74, 0x65, 0x73, 0x56, + 0x61, 0x6c, 0x12, 0x1f, 0x0a, 0x0a, 0x73, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x5f, 0x76, 0x61, 0x6c, + 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x48, 0x00, 0x52, 0x09, 0x73, 0x74, 0x72, 0x69, 0x6e, 0x67, + 0x56, 0x61, 0x6c, 0x12, 0x1d, 0x0a, 0x09, 0x69, 0x6e, 0x74, 0x33, 0x32, 0x5f, 0x76, 0x61, 0x6c, + 0x18, 0x03, 0x20, 0x01, 0x28, 0x05, 0x48, 0x00, 0x52, 0x08, 0x69, 0x6e, 0x74, 0x33, 0x32, 0x56, + 0x61, 0x6c, 0x12, 0x1d, 0x0a, 0x09, 0x69, 0x6e, 0x74, 0x36, 0x34, 0x5f, 0x76, 0x61, 0x6c, 0x18, + 0x04, 0x20, 0x01, 0x28, 0x03, 0x48, 0x00, 0x52, 0x08, 0x69, 0x6e, 0x74, 0x36, 0x34, 0x56, 0x61, + 0x6c, 0x12, 0x1f, 0x0a, 0x0a, 0x64, 0x6f, 0x75, 0x62, 0x6c, 0x65, 0x5f, 0x76, 0x61, 0x6c, 0x18, + 0x05, 0x20, 0x01, 0x28, 0x01, 0x48, 0x00, 0x52, 0x09, 0x64, 0x6f, 0x75, 0x62, 0x6c, 0x65, 0x56, + 0x61, 0x6c, 0x12, 0x1d, 0x0a, 0x09, 0x66, 0x6c, 0x6f, 0x61, 0x74, 0x5f, 0x76, 0x61, 0x6c, 0x18, + 0x06, 0x20, 0x01, 0x28, 0x02, 0x48, 0x00, 0x52, 0x08, 0x66, 0x6c, 0x6f, 0x61, 0x74, 0x56, 0x61, + 0x6c, 0x12, 0x1b, 0x0a, 0x08, 0x62, 0x6f, 0x6f, 0x6c, 0x5f, 0x76, 0x61, 0x6c, 0x18, 0x07, 0x20, + 0x01, 0x28, 0x08, 0x48, 0x00, 0x52, 0x07, 0x62, 0x6f, 0x6f, 0x6c, 0x56, 0x61, 0x6c, 0x12, 0x3e, + 0x0a, 0x0e, 0x62, 0x79, 0x74, 0x65, 0x73, 0x5f, 0x6c, 0x69, 0x73, 0x74, 0x5f, 0x76, 0x61, 0x6c, + 0x18, 0x0b, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, + 0x79, 0x70, 0x65, 0x73, 0x2e, 0x42, 0x79, 0x74, 0x65, 0x73, 0x4c, 0x69, 0x73, 0x74, 0x48, 0x00, + 0x52, 0x0c, 0x62, 0x79, 0x74, 0x65, 0x73, 0x4c, 0x69, 0x73, 0x74, 0x56, 0x61, 0x6c, 0x12, 0x41, + 0x0a, 0x0f, 0x73, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x5f, 0x6c, 0x69, 0x73, 0x74, 0x5f, 0x76, 0x61, + 0x6c, 0x18, 0x0c, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x17, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x53, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x4c, 0x69, 0x73, 0x74, + 0x48, 0x00, 0x52, 0x0d, 0x73, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x4c, 0x69, 0x73, 0x74, 0x56, 0x61, + 0x6c, 0x12, 0x3e, 0x0a, 0x0e, 0x69, 0x6e, 0x74, 0x33, 0x32, 0x5f, 0x6c, 0x69, 0x73, 0x74, 0x5f, + 0x76, 0x61, 0x6c, 0x18, 0x0d, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, + 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x49, 0x6e, 0x74, 0x33, 0x32, 0x4c, 0x69, 0x73, + 0x74, 0x48, 0x00, 0x52, 0x0c, 0x69, 0x6e, 0x74, 0x33, 0x32, 0x4c, 0x69, 0x73, 0x74, 0x56, 0x61, + 0x6c, 0x12, 0x3e, 0x0a, 0x0e, 0x69, 0x6e, 0x74, 0x36, 0x34, 0x5f, 0x6c, 0x69, 0x73, 0x74, 0x5f, + 0x76, 0x61, 0x6c, 0x18, 0x0e, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, + 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x49, 0x6e, 0x74, 0x36, 0x34, 0x4c, 0x69, 0x73, + 0x74, 0x48, 0x00, 0x52, 0x0c, 0x69, 0x6e, 0x74, 0x36, 0x34, 0x4c, 0x69, 0x73, 0x74, 0x56, 0x61, + 0x6c, 0x12, 0x41, 0x0a, 0x0f, 0x64, 0x6f, 0x75, 0x62, 0x6c, 0x65, 0x5f, 0x6c, 0x69, 0x73, 0x74, + 0x5f, 0x76, 0x61, 0x6c, 0x18, 0x0f, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x17, 0x2e, 0x66, 0x65, 0x61, + 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x44, 0x6f, 0x75, 0x62, 0x6c, 0x65, 0x4c, + 0x69, 0x73, 0x74, 0x48, 0x00, 0x52, 0x0d, 0x64, 0x6f, 0x75, 0x62, 0x6c, 0x65, 0x4c, 0x69, 0x73, + 0x74, 0x56, 0x61, 0x6c, 0x12, 0x3e, 0x0a, 0x0e, 0x66, 0x6c, 0x6f, 0x61, 0x74, 0x5f, 0x6c, 0x69, + 0x73, 0x74, 0x5f, 0x76, 0x61, 0x6c, 0x18, 0x10, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, + 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x46, 0x6c, 0x6f, 0x61, 0x74, + 0x4c, 0x69, 0x73, 0x74, 0x48, 0x00, 0x52, 0x0c, 0x66, 0x6c, 0x6f, 0x61, 0x74, 0x4c, 0x69, 0x73, + 0x74, 0x56, 0x61, 0x6c, 0x12, 0x3b, 0x0a, 0x0d, 0x62, 0x6f, 0x6f, 0x6c, 0x5f, 0x6c, 0x69, 0x73, + 0x74, 0x5f, 0x76, 0x61, 0x6c, 0x18, 0x11, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x15, 0x2e, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x42, 0x6f, 0x6f, 0x6c, 0x4c, 0x69, + 0x73, 0x74, 0x48, 0x00, 0x52, 0x0b, 0x62, 0x6f, 0x6f, 0x6c, 0x4c, 0x69, 0x73, 0x74, 0x56, 0x61, + 0x6c, 0x42, 0x05, 0x0a, 0x03, 0x76, 0x61, 0x6c, 0x22, 0x1d, 0x0a, 0x09, 0x42, 0x79, 0x74, 0x65, + 0x73, 0x4c, 0x69, 0x73, 0x74, 0x12, 0x10, 0x0a, 0x03, 0x76, 0x61, 0x6c, 0x18, 0x01, 0x20, 0x03, + 0x28, 0x0c, 0x52, 0x03, 0x76, 0x61, 0x6c, 0x22, 0x1e, 0x0a, 0x0a, 0x53, 0x74, 0x72, 0x69, 0x6e, + 0x67, 0x4c, 0x69, 0x73, 0x74, 0x12, 0x10, 0x0a, 0x03, 0x76, 0x61, 0x6c, 0x18, 0x01, 0x20, 0x03, + 0x28, 0x09, 0x52, 0x03, 0x76, 0x61, 0x6c, 0x22, 0x1d, 0x0a, 0x09, 0x49, 0x6e, 0x74, 0x33, 0x32, + 0x4c, 0x69, 0x73, 0x74, 0x12, 0x10, 0x0a, 0x03, 0x76, 0x61, 0x6c, 0x18, 0x01, 0x20, 0x03, 0x28, + 0x05, 0x52, 0x03, 0x76, 0x61, 0x6c, 0x22, 0x1d, 0x0a, 0x09, 0x49, 0x6e, 0x74, 0x36, 0x34, 0x4c, + 0x69, 0x73, 0x74, 0x12, 0x10, 0x0a, 0x03, 0x76, 0x61, 0x6c, 0x18, 0x01, 0x20, 0x03, 0x28, 0x03, + 0x52, 0x03, 0x76, 0x61, 0x6c, 0x22, 0x1e, 0x0a, 0x0a, 0x44, 0x6f, 0x75, 0x62, 0x6c, 0x65, 0x4c, + 0x69, 0x73, 0x74, 0x12, 0x10, 0x0a, 0x03, 0x76, 0x61, 0x6c, 0x18, 0x01, 0x20, 0x03, 0x28, 0x01, + 0x52, 0x03, 0x76, 0x61, 0x6c, 0x22, 0x1d, 0x0a, 0x09, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x4c, 0x69, + 0x73, 0x74, 0x12, 0x10, 0x0a, 0x03, 0x76, 0x61, 0x6c, 0x18, 0x01, 0x20, 0x03, 0x28, 0x02, 0x52, + 0x03, 0x76, 0x61, 0x6c, 0x22, 0x1c, 0x0a, 0x08, 0x42, 0x6f, 0x6f, 0x6c, 0x4c, 0x69, 0x73, 0x74, + 0x12, 0x10, 0x0a, 0x03, 0x76, 0x61, 0x6c, 0x18, 0x01, 0x20, 0x03, 0x28, 0x08, 0x52, 0x03, 0x76, + 0x61, 0x6c, 0x42, 0x4b, 0x0a, 0x0b, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, + 0x73, 0x42, 0x0a, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x30, 0x67, + 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, + 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, + 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x74, 0x79, 0x70, 0x65, 0x73, 0x62, + 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, +} + +var ( + file_feast_types_Value_proto_rawDescOnce sync.Once + file_feast_types_Value_proto_rawDescData = file_feast_types_Value_proto_rawDesc +) + +func file_feast_types_Value_proto_rawDescGZIP() []byte { + file_feast_types_Value_proto_rawDescOnce.Do(func() { + file_feast_types_Value_proto_rawDescData = protoimpl.X.CompressGZIP(file_feast_types_Value_proto_rawDescData) + }) + return file_feast_types_Value_proto_rawDescData +} + +var file_feast_types_Value_proto_enumTypes = make([]protoimpl.EnumInfo, 1) +var file_feast_types_Value_proto_msgTypes = make([]protoimpl.MessageInfo, 9) +var file_feast_types_Value_proto_goTypes = []interface{}{ + (ValueType_Enum)(0), // 0: feast.types.ValueType.Enum + (*ValueType)(nil), // 1: feast.types.ValueType + (*Value)(nil), // 2: feast.types.Value + (*BytesList)(nil), // 3: feast.types.BytesList + (*StringList)(nil), // 4: feast.types.StringList + (*Int32List)(nil), // 5: feast.types.Int32List + (*Int64List)(nil), // 6: feast.types.Int64List + (*DoubleList)(nil), // 7: feast.types.DoubleList + (*FloatList)(nil), // 8: feast.types.FloatList + (*BoolList)(nil), // 9: feast.types.BoolList +} +var file_feast_types_Value_proto_depIdxs = []int32{ + 3, // 0: feast.types.Value.bytes_list_val:type_name -> feast.types.BytesList + 4, // 1: feast.types.Value.string_list_val:type_name -> feast.types.StringList + 5, // 2: feast.types.Value.int32_list_val:type_name -> feast.types.Int32List + 6, // 3: feast.types.Value.int64_list_val:type_name -> feast.types.Int64List + 7, // 4: feast.types.Value.double_list_val:type_name -> feast.types.DoubleList + 8, // 5: feast.types.Value.float_list_val:type_name -> feast.types.FloatList + 9, // 6: feast.types.Value.bool_list_val:type_name -> feast.types.BoolList + 7, // [7:7] is the sub-list for method output_type + 7, // [7:7] is the sub-list for method input_type + 7, // [7:7] is the sub-list for extension type_name + 7, // [7:7] is the sub-list for extension extendee + 0, // [0:7] is the sub-list for field type_name +} + +func init() { file_feast_types_Value_proto_init() } +func file_feast_types_Value_proto_init() { + if File_feast_types_Value_proto != nil { + return + } + if !protoimpl.UnsafeEnabled { + file_feast_types_Value_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ValueType); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_types_Value_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Value); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_types_Value_proto_msgTypes[2].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*BytesList); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_types_Value_proto_msgTypes[3].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*StringList); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_types_Value_proto_msgTypes[4].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Int32List); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_types_Value_proto_msgTypes[5].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Int64List); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_types_Value_proto_msgTypes[6].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*DoubleList); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_types_Value_proto_msgTypes[7].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*FloatList); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_types_Value_proto_msgTypes[8].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*BoolList); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + } + file_feast_types_Value_proto_msgTypes[1].OneofWrappers = []interface{}{ + (*Value_BytesVal)(nil), + (*Value_StringVal)(nil), + (*Value_Int32Val)(nil), + (*Value_Int64Val)(nil), + (*Value_DoubleVal)(nil), + (*Value_FloatVal)(nil), + (*Value_BoolVal)(nil), + (*Value_BytesListVal)(nil), + (*Value_StringListVal)(nil), + (*Value_Int32ListVal)(nil), + (*Value_Int64ListVal)(nil), + (*Value_DoubleListVal)(nil), + (*Value_FloatListVal)(nil), + (*Value_BoolListVal)(nil), + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_feast_types_Value_proto_rawDesc, + NumEnums: 1, + NumMessages: 9, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_feast_types_Value_proto_goTypes, + DependencyIndexes: file_feast_types_Value_proto_depIdxs, + EnumInfos: file_feast_types_Value_proto_enumTypes, + MessageInfos: file_feast_types_Value_proto_msgTypes, + }.Build() + File_feast_types_Value_proto = out.File + file_feast_types_Value_proto_rawDesc = nil + file_feast_types_Value_proto_goTypes = nil + file_feast_types_Value_proto_depIdxs = nil } diff --git a/sdk/go/request_test.go b/sdk/go/request_test.go index 583a1c8b94f..b6866638670 100644 --- a/sdk/go/request_test.go +++ b/sdk/go/request_test.go @@ -5,7 +5,7 @@ import ( "github.com/gojek/feast/sdk/go/protos/feast/serving" "github.com/gojek/feast/sdk/go/protos/feast/types" json "github.com/golang/protobuf/jsonpb" - "github.com/google/go-cmp/cmp" + "github.com/golang/protobuf/proto" "testing" ) @@ -144,7 +144,7 @@ func TestGetOnlineFeaturesRequest(t *testing.T) { return } - if !cmp.Equal(got, tc.want) { + if !proto.Equal(got, tc.want) { m := json.Marshaler{} gotJSON, _ := m.MarshalToString(got) wantJSON, _ := m.MarshalToString(tc.want) diff --git a/sdk/go/response_test.go b/sdk/go/response_test.go index 87bf275da9c..5aa2c276d61 100644 --- a/sdk/go/response_test.go +++ b/sdk/go/response_test.go @@ -33,9 +33,14 @@ func TestOnlineFeaturesResponseToRow(t *testing.T) { {"project1/feature1": Int64Val(1), "project1/feature2": &types.Value{}}, {"project1/feature1": Int64Val(2), "project1/feature2": Int64Val(2)}, } - if !cmp.Equal(actual, expected) { + if len(expected) != len(actual) { t.Errorf("expected: %v, got: %v", expected, actual) } + for i := range expected { + if !expected[i].equalTo(actual[i]) { + t.Errorf("expected: %v, got: %v", expected, actual) + } + } } func TestOnlineFeaturesResponseToInt64Array(t *testing.T) { diff --git a/sdk/go/types.go b/sdk/go/types.go index 92858bd384c..9af888e3557 100644 --- a/sdk/go/types.go +++ b/sdk/go/types.go @@ -1,10 +1,26 @@ package feast -import "github.com/gojek/feast/sdk/go/protos/feast/types" +import ( + "github.com/gojek/feast/sdk/go/protos/feast/types" + "github.com/golang/protobuf/proto" +) // Row map of entity values type Row map[string]*types.Value +func (r Row) equalTo(other Row) bool { + for k, v := range r { + if otherV, ok := other[k]; !ok { + return false + } else { + if !proto.Equal(v, otherV) { + return false + } + } + } + return true +} + // StrVal is a int64 type feast value func StrVal(val string) *types.Value { return &types.Value{Val: &types.Value_StringVal{StringVal: val}} From 4f01f17dc63260b8565ec66ed3806dfa068e5543 Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Tue, 14 Apr 2020 21:43:47 +0800 Subject: [PATCH 115/176] Generate golang code for non-serving protos (#618) * Generate golang code for non-serving protos as well * Generate code with protoc 3.10.0 * Unnest from third_party dir --- Makefile | 3 +- .../tensorflow_metadata/proto/v0/path.proto | 1 + .../tensorflow_metadata/proto/v0/schema.proto | 1 + sdk/go/protos/feast/core/CoreService.pb.go | 2693 +++++++++++ sdk/go/protos/feast/core/FeatureSet.pb.go | 1430 ++++++ .../feast/core/FeatureSetReference.pb.go | 193 + sdk/go/protos/feast/core/IngestionJob.pb.go | 333 ++ sdk/go/protos/feast/core/Source.pb.go | 336 ++ sdk/go/protos/feast/core/Store.pb.go | 749 +++ .../protos/feast/serving/ServingService.pb.go | 2 +- sdk/go/protos/feast/storage/Redis.pb.go | 187 + sdk/go/protos/feast/types/FeatureRow.pb.go | 2 +- .../feast/types/FeatureRowExtended.pb.go | 2 +- sdk/go/protos/feast/types/Field.pb.go | 2 +- sdk/go/protos/feast/types/Value.pb.go | 2 +- .../tensorflow_metadata/proto/v0/path.pb.go | 186 + .../tensorflow_metadata/proto/v0/schema.pb.go | 4084 +++++++++++++++++ 17 files changed, 10200 insertions(+), 6 deletions(-) create mode 100644 sdk/go/protos/feast/core/CoreService.pb.go create mode 100644 sdk/go/protos/feast/core/FeatureSet.pb.go create mode 100644 sdk/go/protos/feast/core/FeatureSetReference.pb.go create mode 100644 sdk/go/protos/feast/core/IngestionJob.pb.go create mode 100644 sdk/go/protos/feast/core/Source.pb.go create mode 100644 sdk/go/protos/feast/core/Store.pb.go create mode 100644 sdk/go/protos/feast/storage/Redis.pb.go create mode 100644 sdk/go/protos/tensorflow_metadata/proto/v0/path.pb.go create mode 100644 sdk/go/protos/tensorflow_metadata/proto/v0/schema.pb.go diff --git a/Makefile b/Makefile index e780864ddca..5cf5ae78ced 100644 --- a/Makefile +++ b/Makefile @@ -82,7 +82,8 @@ install-go-ci-dependencies: go get -u golang.org/x/lint/golint compile-protos-go: install-go-ci-dependencies - @$(foreach dir,types serving, cd ${ROOT_DIR}/protos; protoc -I/usr/local/include -I. --go_out=plugins=grpc,paths=source_relative:../sdk/go/protos/ feast/$(dir)/*.proto;) + cd ${ROOT_DIR}/protos; protoc -I/usr/local/include -I. --go_out=plugins=grpc,paths=source_relative:../sdk/go/protos/ tensorflow_metadata/proto/v0/*.proto + $(foreach dir,types serving core storage,cd ${ROOT_DIR}/protos; protoc -I/usr/local/include -I. --go_out=plugins=grpc,paths=source_relative:../sdk/go/protos feast/$(dir)/*.proto;) test-go: cd ${ROOT_DIR}/sdk/go; go test ./... diff --git a/protos/tensorflow_metadata/proto/v0/path.proto b/protos/tensorflow_metadata/proto/v0/path.proto index cac09b7a086..2d45e1326ea 100644 --- a/protos/tensorflow_metadata/proto/v0/path.proto +++ b/protos/tensorflow_metadata/proto/v0/path.proto @@ -20,6 +20,7 @@ package tensorflow.metadata.v0; option java_package = "org.tensorflow.metadata.v0"; option java_multiple_files = true; +option go_package = "github.com/gojek/feast/sdk/go/protos/tensorflow_metadata/proto/v0"; // A path is a more general substitute for the name of a field or feature that // can be used for flat examples as well as structured data. For example, if diff --git a/protos/tensorflow_metadata/proto/v0/schema.proto b/protos/tensorflow_metadata/proto/v0/schema.proto index ce30515c69d..8d4da75e160 100644 --- a/protos/tensorflow_metadata/proto/v0/schema.proto +++ b/protos/tensorflow_metadata/proto/v0/schema.proto @@ -23,6 +23,7 @@ import "tensorflow_metadata/proto/v0/path.proto"; option cc_enable_arenas = true; option java_package = "org.tensorflow.metadata.v0"; option java_multiple_files = true; +option go_package = "github.com/gojek/feast/sdk/go/protos/tensorflow_metadata/proto/v0"; // LifecycleStage. Only UNKNOWN_STAGE, BETA, and PRODUCTION features are // actually validated. diff --git a/sdk/go/protos/feast/core/CoreService.pb.go b/sdk/go/protos/feast/core/CoreService.pb.go new file mode 100644 index 00000000000..90e5f7d2408 --- /dev/null +++ b/sdk/go/protos/feast/core/CoreService.pb.go @@ -0,0 +1,2693 @@ +// +// Copyright 2018 The Feast Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.21.0 +// protoc v3.10.0 +// source: feast/core/CoreService.proto + +package core + +import ( + context "context" + proto "github.com/golang/protobuf/proto" + grpc "google.golang.org/grpc" + codes "google.golang.org/grpc/codes" + status "google.golang.org/grpc/status" + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +// This is a compile-time assertion that a sufficiently up-to-date version +// of the legacy proto package is being used. +const _ = proto.ProtoPackageIsVersion4 + +type ApplyFeatureSetResponse_Status int32 + +const ( + // Latest feature set version is consistent with provided feature set + ApplyFeatureSetResponse_NO_CHANGE ApplyFeatureSetResponse_Status = 0 + // New feature set or feature set version created + ApplyFeatureSetResponse_CREATED ApplyFeatureSetResponse_Status = 1 + // Error occurred while trying to apply changes + ApplyFeatureSetResponse_ERROR ApplyFeatureSetResponse_Status = 2 +) + +// Enum value maps for ApplyFeatureSetResponse_Status. +var ( + ApplyFeatureSetResponse_Status_name = map[int32]string{ + 0: "NO_CHANGE", + 1: "CREATED", + 2: "ERROR", + } + ApplyFeatureSetResponse_Status_value = map[string]int32{ + "NO_CHANGE": 0, + "CREATED": 1, + "ERROR": 2, + } +) + +func (x ApplyFeatureSetResponse_Status) Enum() *ApplyFeatureSetResponse_Status { + p := new(ApplyFeatureSetResponse_Status) + *p = x + return p +} + +func (x ApplyFeatureSetResponse_Status) String() string { + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) +} + +func (ApplyFeatureSetResponse_Status) Descriptor() protoreflect.EnumDescriptor { + return file_feast_core_CoreService_proto_enumTypes[0].Descriptor() +} + +func (ApplyFeatureSetResponse_Status) Type() protoreflect.EnumType { + return &file_feast_core_CoreService_proto_enumTypes[0] +} + +func (x ApplyFeatureSetResponse_Status) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) +} + +// Deprecated: Use ApplyFeatureSetResponse_Status.Descriptor instead. +func (ApplyFeatureSetResponse_Status) EnumDescriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{7, 0} +} + +type UpdateStoreResponse_Status int32 + +const ( + // Existing store config matching the given store id is identical to the given store config. + UpdateStoreResponse_NO_CHANGE UpdateStoreResponse_Status = 0 + // New store created or existing config updated. + UpdateStoreResponse_UPDATED UpdateStoreResponse_Status = 1 +) + +// Enum value maps for UpdateStoreResponse_Status. +var ( + UpdateStoreResponse_Status_name = map[int32]string{ + 0: "NO_CHANGE", + 1: "UPDATED", + } + UpdateStoreResponse_Status_value = map[string]int32{ + "NO_CHANGE": 0, + "UPDATED": 1, + } +) + +func (x UpdateStoreResponse_Status) Enum() *UpdateStoreResponse_Status { + p := new(UpdateStoreResponse_Status) + *p = x + return p +} + +func (x UpdateStoreResponse_Status) String() string { + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) +} + +func (UpdateStoreResponse_Status) Descriptor() protoreflect.EnumDescriptor { + return file_feast_core_CoreService_proto_enumTypes[1].Descriptor() +} + +func (UpdateStoreResponse_Status) Type() protoreflect.EnumType { + return &file_feast_core_CoreService_proto_enumTypes[1] +} + +func (x UpdateStoreResponse_Status) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) +} + +// Deprecated: Use UpdateStoreResponse_Status.Descriptor instead. +func (UpdateStoreResponse_Status) EnumDescriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{11, 0} +} + +// Request for a single feature set +type GetFeatureSetRequest struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Name of project the feature set belongs to (required) + Project string `protobuf:"bytes,3,opt,name=project,proto3" json:"project,omitempty"` + // Name of feature set (required). + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` + // Version of feature set (optional). If omitted then latest feature set will be returned. + Version int32 `protobuf:"varint,2,opt,name=version,proto3" json:"version,omitempty"` +} + +func (x *GetFeatureSetRequest) Reset() { + *x = GetFeatureSetRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *GetFeatureSetRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*GetFeatureSetRequest) ProtoMessage() {} + +func (x *GetFeatureSetRequest) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[0] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use GetFeatureSetRequest.ProtoReflect.Descriptor instead. +func (*GetFeatureSetRequest) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{0} +} + +func (x *GetFeatureSetRequest) GetProject() string { + if x != nil { + return x.Project + } + return "" +} + +func (x *GetFeatureSetRequest) GetName() string { + if x != nil { + return x.Name + } + return "" +} + +func (x *GetFeatureSetRequest) GetVersion() int32 { + if x != nil { + return x.Version + } + return 0 +} + +// Response containing a single feature set +type GetFeatureSetResponse struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + FeatureSet *FeatureSet `protobuf:"bytes,1,opt,name=feature_set,json=featureSet,proto3" json:"feature_set,omitempty"` +} + +func (x *GetFeatureSetResponse) Reset() { + *x = GetFeatureSetResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[1] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *GetFeatureSetResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*GetFeatureSetResponse) ProtoMessage() {} + +func (x *GetFeatureSetResponse) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[1] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use GetFeatureSetResponse.ProtoReflect.Descriptor instead. +func (*GetFeatureSetResponse) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{1} +} + +func (x *GetFeatureSetResponse) GetFeatureSet() *FeatureSet { + if x != nil { + return x.FeatureSet + } + return nil +} + +// Retrieves details for all versions of a specific feature set +type ListFeatureSetsRequest struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Filter *ListFeatureSetsRequest_Filter `protobuf:"bytes,1,opt,name=filter,proto3" json:"filter,omitempty"` +} + +func (x *ListFeatureSetsRequest) Reset() { + *x = ListFeatureSetsRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[2] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ListFeatureSetsRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ListFeatureSetsRequest) ProtoMessage() {} + +func (x *ListFeatureSetsRequest) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[2] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ListFeatureSetsRequest.ProtoReflect.Descriptor instead. +func (*ListFeatureSetsRequest) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{2} +} + +func (x *ListFeatureSetsRequest) GetFilter() *ListFeatureSetsRequest_Filter { + if x != nil { + return x.Filter + } + return nil +} + +type ListFeatureSetsResponse struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + FeatureSets []*FeatureSet `protobuf:"bytes,1,rep,name=feature_sets,json=featureSets,proto3" json:"feature_sets,omitempty"` +} + +func (x *ListFeatureSetsResponse) Reset() { + *x = ListFeatureSetsResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[3] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ListFeatureSetsResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ListFeatureSetsResponse) ProtoMessage() {} + +func (x *ListFeatureSetsResponse) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[3] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ListFeatureSetsResponse.ProtoReflect.Descriptor instead. +func (*ListFeatureSetsResponse) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{3} +} + +func (x *ListFeatureSetsResponse) GetFeatureSets() []*FeatureSet { + if x != nil { + return x.FeatureSets + } + return nil +} + +type ListStoresRequest struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Filter *ListStoresRequest_Filter `protobuf:"bytes,1,opt,name=filter,proto3" json:"filter,omitempty"` +} + +func (x *ListStoresRequest) Reset() { + *x = ListStoresRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[4] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ListStoresRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ListStoresRequest) ProtoMessage() {} + +func (x *ListStoresRequest) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[4] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ListStoresRequest.ProtoReflect.Descriptor instead. +func (*ListStoresRequest) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{4} +} + +func (x *ListStoresRequest) GetFilter() *ListStoresRequest_Filter { + if x != nil { + return x.Filter + } + return nil +} + +type ListStoresResponse struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Store []*Store `protobuf:"bytes,1,rep,name=store,proto3" json:"store,omitempty"` +} + +func (x *ListStoresResponse) Reset() { + *x = ListStoresResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[5] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ListStoresResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ListStoresResponse) ProtoMessage() {} + +func (x *ListStoresResponse) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[5] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ListStoresResponse.ProtoReflect.Descriptor instead. +func (*ListStoresResponse) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{5} +} + +func (x *ListStoresResponse) GetStore() []*Store { + if x != nil { + return x.Store + } + return nil +} + +type ApplyFeatureSetRequest struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Feature set version and source will be ignored + FeatureSet *FeatureSet `protobuf:"bytes,1,opt,name=feature_set,json=featureSet,proto3" json:"feature_set,omitempty"` +} + +func (x *ApplyFeatureSetRequest) Reset() { + *x = ApplyFeatureSetRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[6] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ApplyFeatureSetRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ApplyFeatureSetRequest) ProtoMessage() {} + +func (x *ApplyFeatureSetRequest) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[6] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ApplyFeatureSetRequest.ProtoReflect.Descriptor instead. +func (*ApplyFeatureSetRequest) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{6} +} + +func (x *ApplyFeatureSetRequest) GetFeatureSet() *FeatureSet { + if x != nil { + return x.FeatureSet + } + return nil +} + +type ApplyFeatureSetResponse struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Feature set response has been enriched with version and source information + FeatureSet *FeatureSet `protobuf:"bytes,1,opt,name=feature_set,json=featureSet,proto3" json:"feature_set,omitempty"` + Status ApplyFeatureSetResponse_Status `protobuf:"varint,2,opt,name=status,proto3,enum=feast.core.ApplyFeatureSetResponse_Status" json:"status,omitempty"` +} + +func (x *ApplyFeatureSetResponse) Reset() { + *x = ApplyFeatureSetResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[7] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ApplyFeatureSetResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ApplyFeatureSetResponse) ProtoMessage() {} + +func (x *ApplyFeatureSetResponse) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[7] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ApplyFeatureSetResponse.ProtoReflect.Descriptor instead. +func (*ApplyFeatureSetResponse) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{7} +} + +func (x *ApplyFeatureSetResponse) GetFeatureSet() *FeatureSet { + if x != nil { + return x.FeatureSet + } + return nil +} + +func (x *ApplyFeatureSetResponse) GetStatus() ApplyFeatureSetResponse_Status { + if x != nil { + return x.Status + } + return ApplyFeatureSetResponse_NO_CHANGE +} + +type GetFeastCoreVersionRequest struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields +} + +func (x *GetFeastCoreVersionRequest) Reset() { + *x = GetFeastCoreVersionRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[8] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *GetFeastCoreVersionRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*GetFeastCoreVersionRequest) ProtoMessage() {} + +func (x *GetFeastCoreVersionRequest) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[8] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use GetFeastCoreVersionRequest.ProtoReflect.Descriptor instead. +func (*GetFeastCoreVersionRequest) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{8} +} + +type GetFeastCoreVersionResponse struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Version string `protobuf:"bytes,1,opt,name=version,proto3" json:"version,omitempty"` +} + +func (x *GetFeastCoreVersionResponse) Reset() { + *x = GetFeastCoreVersionResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[9] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *GetFeastCoreVersionResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*GetFeastCoreVersionResponse) ProtoMessage() {} + +func (x *GetFeastCoreVersionResponse) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[9] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use GetFeastCoreVersionResponse.ProtoReflect.Descriptor instead. +func (*GetFeastCoreVersionResponse) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{9} +} + +func (x *GetFeastCoreVersionResponse) GetVersion() string { + if x != nil { + return x.Version + } + return "" +} + +type UpdateStoreRequest struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Store *Store `protobuf:"bytes,1,opt,name=store,proto3" json:"store,omitempty"` +} + +func (x *UpdateStoreRequest) Reset() { + *x = UpdateStoreRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[10] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *UpdateStoreRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*UpdateStoreRequest) ProtoMessage() {} + +func (x *UpdateStoreRequest) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[10] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use UpdateStoreRequest.ProtoReflect.Descriptor instead. +func (*UpdateStoreRequest) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{10} +} + +func (x *UpdateStoreRequest) GetStore() *Store { + if x != nil { + return x.Store + } + return nil +} + +type UpdateStoreResponse struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Store *Store `protobuf:"bytes,1,opt,name=store,proto3" json:"store,omitempty"` + Status UpdateStoreResponse_Status `protobuf:"varint,2,opt,name=status,proto3,enum=feast.core.UpdateStoreResponse_Status" json:"status,omitempty"` +} + +func (x *UpdateStoreResponse) Reset() { + *x = UpdateStoreResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[11] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *UpdateStoreResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*UpdateStoreResponse) ProtoMessage() {} + +func (x *UpdateStoreResponse) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[11] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use UpdateStoreResponse.ProtoReflect.Descriptor instead. +func (*UpdateStoreResponse) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{11} +} + +func (x *UpdateStoreResponse) GetStore() *Store { + if x != nil { + return x.Store + } + return nil +} + +func (x *UpdateStoreResponse) GetStatus() UpdateStoreResponse_Status { + if x != nil { + return x.Status + } + return UpdateStoreResponse_NO_CHANGE +} + +// Request to create a project +type CreateProjectRequest struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Name of project (required) + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` +} + +func (x *CreateProjectRequest) Reset() { + *x = CreateProjectRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[12] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *CreateProjectRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*CreateProjectRequest) ProtoMessage() {} + +func (x *CreateProjectRequest) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[12] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use CreateProjectRequest.ProtoReflect.Descriptor instead. +func (*CreateProjectRequest) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{12} +} + +func (x *CreateProjectRequest) GetName() string { + if x != nil { + return x.Name + } + return "" +} + +// Response for creation of a project +type CreateProjectResponse struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields +} + +func (x *CreateProjectResponse) Reset() { + *x = CreateProjectResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[13] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *CreateProjectResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*CreateProjectResponse) ProtoMessage() {} + +func (x *CreateProjectResponse) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[13] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use CreateProjectResponse.ProtoReflect.Descriptor instead. +func (*CreateProjectResponse) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{13} +} + +// Request for the archival of a project +type ArchiveProjectRequest struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Name of project to be archived + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` +} + +func (x *ArchiveProjectRequest) Reset() { + *x = ArchiveProjectRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[14] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ArchiveProjectRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ArchiveProjectRequest) ProtoMessage() {} + +func (x *ArchiveProjectRequest) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[14] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ArchiveProjectRequest.ProtoReflect.Descriptor instead. +func (*ArchiveProjectRequest) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{14} +} + +func (x *ArchiveProjectRequest) GetName() string { + if x != nil { + return x.Name + } + return "" +} + +// Response for archival of a project +type ArchiveProjectResponse struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields +} + +func (x *ArchiveProjectResponse) Reset() { + *x = ArchiveProjectResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[15] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ArchiveProjectResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ArchiveProjectResponse) ProtoMessage() {} + +func (x *ArchiveProjectResponse) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[15] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ArchiveProjectResponse.ProtoReflect.Descriptor instead. +func (*ArchiveProjectResponse) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{15} +} + +// Request for listing of projects +type ListProjectsRequest struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields +} + +func (x *ListProjectsRequest) Reset() { + *x = ListProjectsRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[16] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ListProjectsRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ListProjectsRequest) ProtoMessage() {} + +func (x *ListProjectsRequest) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[16] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ListProjectsRequest.ProtoReflect.Descriptor instead. +func (*ListProjectsRequest) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{16} +} + +// Response for listing of projects +type ListProjectsResponse struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // List of project names (archived projects are filtered out) + Projects []string `protobuf:"bytes,1,rep,name=projects,proto3" json:"projects,omitempty"` +} + +func (x *ListProjectsResponse) Reset() { + *x = ListProjectsResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[17] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ListProjectsResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ListProjectsResponse) ProtoMessage() {} + +func (x *ListProjectsResponse) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[17] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ListProjectsResponse.ProtoReflect.Descriptor instead. +func (*ListProjectsResponse) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{17} +} + +func (x *ListProjectsResponse) GetProjects() []string { + if x != nil { + return x.Projects + } + return nil +} + +// Request for listing ingestion jobs +type ListIngestionJobsRequest struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Filter *ListIngestionJobsRequest_Filter `protobuf:"bytes,1,opt,name=filter,proto3" json:"filter,omitempty"` +} + +func (x *ListIngestionJobsRequest) Reset() { + *x = ListIngestionJobsRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[18] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ListIngestionJobsRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ListIngestionJobsRequest) ProtoMessage() {} + +func (x *ListIngestionJobsRequest) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[18] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ListIngestionJobsRequest.ProtoReflect.Descriptor instead. +func (*ListIngestionJobsRequest) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{18} +} + +func (x *ListIngestionJobsRequest) GetFilter() *ListIngestionJobsRequest_Filter { + if x != nil { + return x.Filter + } + return nil +} + +// Response from listing ingestion jobs +type ListIngestionJobsResponse struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Jobs []*IngestionJob `protobuf:"bytes,1,rep,name=jobs,proto3" json:"jobs,omitempty"` +} + +func (x *ListIngestionJobsResponse) Reset() { + *x = ListIngestionJobsResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[19] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ListIngestionJobsResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ListIngestionJobsResponse) ProtoMessage() {} + +func (x *ListIngestionJobsResponse) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[19] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ListIngestionJobsResponse.ProtoReflect.Descriptor instead. +func (*ListIngestionJobsResponse) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{19} +} + +func (x *ListIngestionJobsResponse) GetJobs() []*IngestionJob { + if x != nil { + return x.Jobs + } + return nil +} + +// Request to restart ingestion job +type RestartIngestionJobRequest struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Job ID assigned by Feast + Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` +} + +func (x *RestartIngestionJobRequest) Reset() { + *x = RestartIngestionJobRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[20] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *RestartIngestionJobRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*RestartIngestionJobRequest) ProtoMessage() {} + +func (x *RestartIngestionJobRequest) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[20] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use RestartIngestionJobRequest.ProtoReflect.Descriptor instead. +func (*RestartIngestionJobRequest) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{20} +} + +func (x *RestartIngestionJobRequest) GetId() string { + if x != nil { + return x.Id + } + return "" +} + +// Response from restartingan injestion job +type RestartIngestionJobResponse struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields +} + +func (x *RestartIngestionJobResponse) Reset() { + *x = RestartIngestionJobResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[21] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *RestartIngestionJobResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*RestartIngestionJobResponse) ProtoMessage() {} + +func (x *RestartIngestionJobResponse) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[21] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use RestartIngestionJobResponse.ProtoReflect.Descriptor instead. +func (*RestartIngestionJobResponse) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{21} +} + +// Request to stop ingestion job +type StopIngestionJobRequest struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Job ID assigned by Feast + Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` +} + +func (x *StopIngestionJobRequest) Reset() { + *x = StopIngestionJobRequest{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[22] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *StopIngestionJobRequest) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*StopIngestionJobRequest) ProtoMessage() {} + +func (x *StopIngestionJobRequest) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[22] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use StopIngestionJobRequest.ProtoReflect.Descriptor instead. +func (*StopIngestionJobRequest) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{22} +} + +func (x *StopIngestionJobRequest) GetId() string { + if x != nil { + return x.Id + } + return "" +} + +// Request from stopping an ingestion job +type StopIngestionJobResponse struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields +} + +func (x *StopIngestionJobResponse) Reset() { + *x = StopIngestionJobResponse{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[23] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *StopIngestionJobResponse) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*StopIngestionJobResponse) ProtoMessage() {} + +func (x *StopIngestionJobResponse) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[23] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use StopIngestionJobResponse.ProtoReflect.Descriptor instead. +func (*StopIngestionJobResponse) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{23} +} + +type ListFeatureSetsRequest_Filter struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Name of project that the feature sets belongs to. This can be one of + // - [project_name] + // - * + // If an asterisk is provided, filtering on projects will be disabled. All projects will + // be matched. It is NOT possible to provide an asterisk with a string in order to do + // pattern matching. + Project string `protobuf:"bytes,3,opt,name=project,proto3" json:"project,omitempty"` + // Name of the desired feature set. Asterisks can be used as wildcards in the name. + // Matching on names is only permitted if a specific project is defined. It is disallowed + // If the project name is set to "*" + // e.g. + // - * can be used to match all feature sets + // - my-feature-set* can be used to match all features prefixed by "my-feature-set" + // - my-feature-set-6 can be used to select a single feature set + FeatureSetName string `protobuf:"bytes,1,opt,name=feature_set_name,json=featureSetName,proto3" json:"feature_set_name,omitempty"` + // Versions of the given feature sets that will be returned. + // Valid options for version: + // "latest": only the latest version is returned. + // "*": Subscribe to all versions + // [version number]: pin to a specific version. Project and feature set name must be + // explicitly defined if a specific version is pinned. + FeatureSetVersion string `protobuf:"bytes,2,opt,name=feature_set_version,json=featureSetVersion,proto3" json:"feature_set_version,omitempty"` +} + +func (x *ListFeatureSetsRequest_Filter) Reset() { + *x = ListFeatureSetsRequest_Filter{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[24] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ListFeatureSetsRequest_Filter) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ListFeatureSetsRequest_Filter) ProtoMessage() {} + +func (x *ListFeatureSetsRequest_Filter) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[24] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ListFeatureSetsRequest_Filter.ProtoReflect.Descriptor instead. +func (*ListFeatureSetsRequest_Filter) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{2, 0} +} + +func (x *ListFeatureSetsRequest_Filter) GetProject() string { + if x != nil { + return x.Project + } + return "" +} + +func (x *ListFeatureSetsRequest_Filter) GetFeatureSetName() string { + if x != nil { + return x.FeatureSetName + } + return "" +} + +func (x *ListFeatureSetsRequest_Filter) GetFeatureSetVersion() string { + if x != nil { + return x.FeatureSetVersion + } + return "" +} + +type ListStoresRequest_Filter struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Name of desired store. Regex is not supported in this query. + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` +} + +func (x *ListStoresRequest_Filter) Reset() { + *x = ListStoresRequest_Filter{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[25] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ListStoresRequest_Filter) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ListStoresRequest_Filter) ProtoMessage() {} + +func (x *ListStoresRequest_Filter) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[25] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ListStoresRequest_Filter.ProtoReflect.Descriptor instead. +func (*ListStoresRequest_Filter) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{4, 0} +} + +func (x *ListStoresRequest_Filter) GetName() string { + if x != nil { + return x.Name + } + return "" +} + +type ListIngestionJobsRequest_Filter struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Filter by Job ID assigned by Feast + Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` + // Filter by ingestion job target feature set. + FeatureSetReference *FeatureSetReference `protobuf:"bytes,2,opt,name=feature_set_reference,json=featureSetReference,proto3" json:"feature_set_reference,omitempty"` + // Filter by Name of store + StoreName string `protobuf:"bytes,3,opt,name=store_name,json=storeName,proto3" json:"store_name,omitempty"` +} + +func (x *ListIngestionJobsRequest_Filter) Reset() { + *x = ListIngestionJobsRequest_Filter{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_CoreService_proto_msgTypes[26] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ListIngestionJobsRequest_Filter) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ListIngestionJobsRequest_Filter) ProtoMessage() {} + +func (x *ListIngestionJobsRequest_Filter) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_CoreService_proto_msgTypes[26] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ListIngestionJobsRequest_Filter.ProtoReflect.Descriptor instead. +func (*ListIngestionJobsRequest_Filter) Descriptor() ([]byte, []int) { + return file_feast_core_CoreService_proto_rawDescGZIP(), []int{18, 0} +} + +func (x *ListIngestionJobsRequest_Filter) GetId() string { + if x != nil { + return x.Id + } + return "" +} + +func (x *ListIngestionJobsRequest_Filter) GetFeatureSetReference() *FeatureSetReference { + if x != nil { + return x.FeatureSetReference + } + return nil +} + +func (x *ListIngestionJobsRequest_Filter) GetStoreName() string { + if x != nil { + return x.StoreName + } + return "" +} + +var File_feast_core_CoreService_proto protoreflect.FileDescriptor + +var file_feast_core_CoreService_proto_rawDesc = []byte{ + 0x0a, 0x1c, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x43, 0x6f, 0x72, + 0x65, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0a, + 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x1a, 0x1b, 0x66, 0x65, 0x61, 0x73, + 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, + 0x74, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x16, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, + 0x6f, 0x72, 0x65, 0x2f, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, + 0x24, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x46, 0x65, 0x61, 0x74, + 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x2e, + 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x1d, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, + 0x65, 0x2f, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x2e, 0x70, + 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x5e, 0x0a, 0x14, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, + 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x18, 0x0a, 0x07, + 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x70, + 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, + 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, + 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, 0x07, 0x76, 0x65, 0x72, + 0x73, 0x69, 0x6f, 0x6e, 0x22, 0x50, 0x0a, 0x15, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, + 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x37, 0x0a, + 0x0b, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x5f, 0x73, 0x65, 0x74, 0x18, 0x01, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, + 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x0a, 0x66, 0x65, 0x61, 0x74, + 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x22, 0xd9, 0x01, 0x0a, 0x16, 0x4c, 0x69, 0x73, 0x74, 0x46, + 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, + 0x74, 0x12, 0x41, 0x0a, 0x06, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, + 0x0b, 0x32, 0x29, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, + 0x69, 0x73, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, 0x52, 0x65, + 0x71, 0x75, 0x65, 0x73, 0x74, 0x2e, 0x46, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x52, 0x06, 0x66, 0x69, + 0x6c, 0x74, 0x65, 0x72, 0x1a, 0x7c, 0x0a, 0x06, 0x46, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x12, 0x18, + 0x0a, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x12, 0x28, 0x0a, 0x10, 0x66, 0x65, 0x61, 0x74, + 0x75, 0x72, 0x65, 0x5f, 0x73, 0x65, 0x74, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, + 0x28, 0x09, 0x52, 0x0e, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x4e, 0x61, + 0x6d, 0x65, 0x12, 0x2e, 0x0a, 0x13, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x5f, 0x73, 0x65, + 0x74, 0x5f, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x11, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x56, 0x65, 0x72, 0x73, 0x69, + 0x6f, 0x6e, 0x22, 0x54, 0x0a, 0x17, 0x4c, 0x69, 0x73, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, + 0x65, 0x53, 0x65, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x39, 0x0a, + 0x0c, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x5f, 0x73, 0x65, 0x74, 0x73, 0x18, 0x01, 0x20, + 0x03, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, + 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x0b, 0x66, 0x65, 0x61, + 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, 0x22, 0x6f, 0x0a, 0x11, 0x4c, 0x69, 0x73, 0x74, + 0x53, 0x74, 0x6f, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x3c, 0x0a, + 0x06, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x24, 0x2e, + 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x53, + 0x74, 0x6f, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x2e, 0x46, 0x69, 0x6c, + 0x74, 0x65, 0x72, 0x52, 0x06, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x1a, 0x1c, 0x0a, 0x06, 0x46, + 0x69, 0x6c, 0x74, 0x65, 0x72, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, + 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x3d, 0x0a, 0x12, 0x4c, 0x69, 0x73, + 0x74, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, + 0x27, 0x0a, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x11, + 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x74, 0x6f, 0x72, + 0x65, 0x52, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x22, 0x51, 0x0a, 0x16, 0x41, 0x70, 0x70, 0x6c, + 0x79, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, + 0x73, 0x74, 0x12, 0x37, 0x0a, 0x0b, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x5f, 0x73, 0x65, + 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, + 0x0a, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x22, 0xc7, 0x01, 0x0a, 0x17, + 0x41, 0x70, 0x70, 0x6c, 0x79, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, + 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x37, 0x0a, 0x0b, 0x66, 0x65, 0x61, 0x74, 0x75, + 0x72, 0x65, 0x5f, 0x73, 0x65, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, + 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, + 0x65, 0x53, 0x65, 0x74, 0x52, 0x0a, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, + 0x12, 0x42, 0x0a, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, + 0x32, 0x2a, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x70, + 0x70, 0x6c, 0x79, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x73, + 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x06, 0x73, 0x74, + 0x61, 0x74, 0x75, 0x73, 0x22, 0x2f, 0x0a, 0x06, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x0d, + 0x0a, 0x09, 0x4e, 0x4f, 0x5f, 0x43, 0x48, 0x41, 0x4e, 0x47, 0x45, 0x10, 0x00, 0x12, 0x0b, 0x0a, + 0x07, 0x43, 0x52, 0x45, 0x41, 0x54, 0x45, 0x44, 0x10, 0x01, 0x12, 0x09, 0x0a, 0x05, 0x45, 0x52, + 0x52, 0x4f, 0x52, 0x10, 0x02, 0x22, 0x1c, 0x0a, 0x1a, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, + 0x74, 0x43, 0x6f, 0x72, 0x65, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x52, 0x65, 0x71, 0x75, + 0x65, 0x73, 0x74, 0x22, 0x37, 0x0a, 0x1b, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, 0x43, + 0x6f, 0x72, 0x65, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, + 0x73, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x01, 0x20, + 0x01, 0x28, 0x09, 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x22, 0x3d, 0x0a, 0x12, + 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, + 0x73, 0x74, 0x12, 0x27, 0x0a, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, + 0x0b, 0x32, 0x11, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, + 0x74, 0x6f, 0x72, 0x65, 0x52, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x22, 0xa4, 0x01, 0x0a, 0x13, + 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, + 0x6e, 0x73, 0x65, 0x12, 0x27, 0x0a, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x18, 0x01, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x11, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, + 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x12, 0x3e, 0x0a, 0x06, + 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x26, 0x2e, 0x66, + 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, + 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x53, 0x74, + 0x61, 0x74, 0x75, 0x73, 0x52, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x22, 0x24, 0x0a, 0x06, + 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x0d, 0x0a, 0x09, 0x4e, 0x4f, 0x5f, 0x43, 0x48, 0x41, + 0x4e, 0x47, 0x45, 0x10, 0x00, 0x12, 0x0b, 0x0a, 0x07, 0x55, 0x50, 0x44, 0x41, 0x54, 0x45, 0x44, + 0x10, 0x01, 0x22, 0x2a, 0x0a, 0x14, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x50, 0x72, 0x6f, 0x6a, + 0x65, 0x63, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, + 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x17, + 0x0a, 0x15, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x52, + 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x2b, 0x0a, 0x15, 0x41, 0x72, 0x63, 0x68, 0x69, + 0x76, 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, + 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, + 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x18, 0x0a, 0x16, 0x41, 0x72, 0x63, 0x68, 0x69, 0x76, 0x65, 0x50, + 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x15, + 0x0a, 0x13, 0x4c, 0x69, 0x73, 0x74, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x52, 0x65, + 0x71, 0x75, 0x65, 0x73, 0x74, 0x22, 0x32, 0x0a, 0x14, 0x4c, 0x69, 0x73, 0x74, 0x50, 0x72, 0x6f, + 0x6a, 0x65, 0x63, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x1a, 0x0a, + 0x08, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x09, 0x52, + 0x08, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x22, 0xee, 0x01, 0x0a, 0x18, 0x4c, 0x69, + 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x73, 0x52, + 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x43, 0x0a, 0x06, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, + 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2b, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, + 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, + 0x6e, 0x4a, 0x6f, 0x62, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x2e, 0x46, 0x69, 0x6c, + 0x74, 0x65, 0x72, 0x52, 0x06, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x1a, 0x8c, 0x01, 0x0a, 0x06, + 0x46, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, + 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x12, 0x53, 0x0a, 0x15, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, + 0x65, 0x5f, 0x73, 0x65, 0x74, 0x5f, 0x72, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x18, + 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1f, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, + 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x66, + 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x52, 0x13, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, + 0x65, 0x74, 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x12, 0x1d, 0x0a, 0x0a, 0x73, + 0x74, 0x6f, 0x72, 0x65, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x09, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x4e, 0x61, 0x6d, 0x65, 0x22, 0x49, 0x0a, 0x19, 0x4c, 0x69, + 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x73, 0x52, + 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x2c, 0x0a, 0x04, 0x6a, 0x6f, 0x62, 0x73, 0x18, + 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x18, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, + 0x72, 0x65, 0x2e, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, + 0x04, 0x6a, 0x6f, 0x62, 0x73, 0x22, 0x2c, 0x0a, 0x1a, 0x52, 0x65, 0x73, 0x74, 0x61, 0x72, 0x74, + 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x71, 0x75, + 0x65, 0x73, 0x74, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x02, 0x69, 0x64, 0x22, 0x1d, 0x0a, 0x1b, 0x52, 0x65, 0x73, 0x74, 0x61, 0x72, 0x74, 0x49, 0x6e, + 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, + 0x73, 0x65, 0x22, 0x29, 0x0a, 0x17, 0x53, 0x74, 0x6f, 0x70, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, + 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x0e, 0x0a, + 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x22, 0x1a, 0x0a, + 0x18, 0x53, 0x74, 0x6f, 0x70, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, + 0x62, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x32, 0xcb, 0x08, 0x0a, 0x0b, 0x43, 0x6f, + 0x72, 0x65, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x66, 0x0a, 0x13, 0x47, 0x65, 0x74, + 0x46, 0x65, 0x61, 0x73, 0x74, 0x43, 0x6f, 0x72, 0x65, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, + 0x12, 0x26, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x47, 0x65, + 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, 0x43, 0x6f, 0x72, 0x65, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, + 0x6e, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x27, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, 0x43, 0x6f, + 0x72, 0x65, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, + 0x65, 0x12, 0x54, 0x0a, 0x0d, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, + 0x65, 0x74, 0x12, 0x20, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, + 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x71, + 0x75, 0x65, 0x73, 0x74, 0x1a, 0x21, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, + 0x65, 0x2e, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, + 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x5a, 0x0a, 0x0f, 0x4c, 0x69, 0x73, 0x74, 0x46, + 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, 0x12, 0x22, 0x2e, 0x66, 0x65, 0x61, + 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x46, 0x65, 0x61, 0x74, + 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x23, + 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, + 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, + 0x6e, 0x73, 0x65, 0x12, 0x4b, 0x0a, 0x0a, 0x4c, 0x69, 0x73, 0x74, 0x53, 0x74, 0x6f, 0x72, 0x65, + 0x73, 0x12, 0x1d, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, + 0x69, 0x73, 0x74, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, + 0x1a, 0x1e, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, + 0x73, 0x74, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, + 0x12, 0x5a, 0x0a, 0x0f, 0x41, 0x70, 0x70, 0x6c, 0x79, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, + 0x53, 0x65, 0x74, 0x12, 0x22, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, + 0x2e, 0x41, 0x70, 0x70, 0x6c, 0x79, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, + 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x23, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x70, 0x70, 0x6c, 0x79, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, + 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x4e, 0x0a, 0x0b, + 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x12, 0x1e, 0x2e, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x53, + 0x74, 0x6f, 0x72, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x1f, 0x2e, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x53, + 0x74, 0x6f, 0x72, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x54, 0x0a, 0x0d, + 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x12, 0x20, 0x2e, + 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x43, 0x72, 0x65, 0x61, 0x74, + 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, + 0x21, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x43, 0x72, 0x65, + 0x61, 0x74, 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, + 0x73, 0x65, 0x12, 0x57, 0x0a, 0x0e, 0x41, 0x72, 0x63, 0x68, 0x69, 0x76, 0x65, 0x50, 0x72, 0x6f, + 0x6a, 0x65, 0x63, 0x74, 0x12, 0x21, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, + 0x65, 0x2e, 0x41, 0x72, 0x63, 0x68, 0x69, 0x76, 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, + 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x22, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x72, 0x63, 0x68, 0x69, 0x76, 0x65, 0x50, 0x72, 0x6f, 0x6a, + 0x65, 0x63, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x51, 0x0a, 0x0c, 0x4c, + 0x69, 0x73, 0x74, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x12, 0x1f, 0x2e, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x50, 0x72, 0x6f, + 0x6a, 0x65, 0x63, 0x74, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x20, 0x2e, 0x66, + 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x50, 0x72, + 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x60, + 0x0a, 0x11, 0x4c, 0x69, 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, + 0x6f, 0x62, 0x73, 0x12, 0x24, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, + 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, + 0x62, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x25, 0x2e, 0x66, 0x65, 0x61, 0x73, + 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, + 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, + 0x12, 0x66, 0x0a, 0x13, 0x52, 0x65, 0x73, 0x74, 0x61, 0x72, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, + 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x12, 0x26, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x52, 0x65, 0x73, 0x74, 0x61, 0x72, 0x74, 0x49, 0x6e, 0x67, 0x65, + 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, + 0x27, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x52, 0x65, 0x73, + 0x74, 0x61, 0x72, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, + 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x5d, 0x0a, 0x10, 0x53, 0x74, 0x6f, 0x70, + 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x12, 0x23, 0x2e, 0x66, + 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x74, 0x6f, 0x70, 0x49, 0x6e, + 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, + 0x74, 0x1a, 0x24, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, + 0x74, 0x6f, 0x70, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, + 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x42, 0x4f, 0x0a, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x42, 0x10, 0x43, 0x6f, 0x72, 0x65, 0x53, 0x65, 0x72, 0x76, 0x69, + 0x63, 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x2f, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, + 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, + 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, +} + +var ( + file_feast_core_CoreService_proto_rawDescOnce sync.Once + file_feast_core_CoreService_proto_rawDescData = file_feast_core_CoreService_proto_rawDesc +) + +func file_feast_core_CoreService_proto_rawDescGZIP() []byte { + file_feast_core_CoreService_proto_rawDescOnce.Do(func() { + file_feast_core_CoreService_proto_rawDescData = protoimpl.X.CompressGZIP(file_feast_core_CoreService_proto_rawDescData) + }) + return file_feast_core_CoreService_proto_rawDescData +} + +var file_feast_core_CoreService_proto_enumTypes = make([]protoimpl.EnumInfo, 2) +var file_feast_core_CoreService_proto_msgTypes = make([]protoimpl.MessageInfo, 27) +var file_feast_core_CoreService_proto_goTypes = []interface{}{ + (ApplyFeatureSetResponse_Status)(0), // 0: feast.core.ApplyFeatureSetResponse.Status + (UpdateStoreResponse_Status)(0), // 1: feast.core.UpdateStoreResponse.Status + (*GetFeatureSetRequest)(nil), // 2: feast.core.GetFeatureSetRequest + (*GetFeatureSetResponse)(nil), // 3: feast.core.GetFeatureSetResponse + (*ListFeatureSetsRequest)(nil), // 4: feast.core.ListFeatureSetsRequest + (*ListFeatureSetsResponse)(nil), // 5: feast.core.ListFeatureSetsResponse + (*ListStoresRequest)(nil), // 6: feast.core.ListStoresRequest + (*ListStoresResponse)(nil), // 7: feast.core.ListStoresResponse + (*ApplyFeatureSetRequest)(nil), // 8: feast.core.ApplyFeatureSetRequest + (*ApplyFeatureSetResponse)(nil), // 9: feast.core.ApplyFeatureSetResponse + (*GetFeastCoreVersionRequest)(nil), // 10: feast.core.GetFeastCoreVersionRequest + (*GetFeastCoreVersionResponse)(nil), // 11: feast.core.GetFeastCoreVersionResponse + (*UpdateStoreRequest)(nil), // 12: feast.core.UpdateStoreRequest + (*UpdateStoreResponse)(nil), // 13: feast.core.UpdateStoreResponse + (*CreateProjectRequest)(nil), // 14: feast.core.CreateProjectRequest + (*CreateProjectResponse)(nil), // 15: feast.core.CreateProjectResponse + (*ArchiveProjectRequest)(nil), // 16: feast.core.ArchiveProjectRequest + (*ArchiveProjectResponse)(nil), // 17: feast.core.ArchiveProjectResponse + (*ListProjectsRequest)(nil), // 18: feast.core.ListProjectsRequest + (*ListProjectsResponse)(nil), // 19: feast.core.ListProjectsResponse + (*ListIngestionJobsRequest)(nil), // 20: feast.core.ListIngestionJobsRequest + (*ListIngestionJobsResponse)(nil), // 21: feast.core.ListIngestionJobsResponse + (*RestartIngestionJobRequest)(nil), // 22: feast.core.RestartIngestionJobRequest + (*RestartIngestionJobResponse)(nil), // 23: feast.core.RestartIngestionJobResponse + (*StopIngestionJobRequest)(nil), // 24: feast.core.StopIngestionJobRequest + (*StopIngestionJobResponse)(nil), // 25: feast.core.StopIngestionJobResponse + (*ListFeatureSetsRequest_Filter)(nil), // 26: feast.core.ListFeatureSetsRequest.Filter + (*ListStoresRequest_Filter)(nil), // 27: feast.core.ListStoresRequest.Filter + (*ListIngestionJobsRequest_Filter)(nil), // 28: feast.core.ListIngestionJobsRequest.Filter + (*FeatureSet)(nil), // 29: feast.core.FeatureSet + (*Store)(nil), // 30: feast.core.Store + (*IngestionJob)(nil), // 31: feast.core.IngestionJob + (*FeatureSetReference)(nil), // 32: feast.core.FeatureSetReference +} +var file_feast_core_CoreService_proto_depIdxs = []int32{ + 29, // 0: feast.core.GetFeatureSetResponse.feature_set:type_name -> feast.core.FeatureSet + 26, // 1: feast.core.ListFeatureSetsRequest.filter:type_name -> feast.core.ListFeatureSetsRequest.Filter + 29, // 2: feast.core.ListFeatureSetsResponse.feature_sets:type_name -> feast.core.FeatureSet + 27, // 3: feast.core.ListStoresRequest.filter:type_name -> feast.core.ListStoresRequest.Filter + 30, // 4: feast.core.ListStoresResponse.store:type_name -> feast.core.Store + 29, // 5: feast.core.ApplyFeatureSetRequest.feature_set:type_name -> feast.core.FeatureSet + 29, // 6: feast.core.ApplyFeatureSetResponse.feature_set:type_name -> feast.core.FeatureSet + 0, // 7: feast.core.ApplyFeatureSetResponse.status:type_name -> feast.core.ApplyFeatureSetResponse.Status + 30, // 8: feast.core.UpdateStoreRequest.store:type_name -> feast.core.Store + 30, // 9: feast.core.UpdateStoreResponse.store:type_name -> feast.core.Store + 1, // 10: feast.core.UpdateStoreResponse.status:type_name -> feast.core.UpdateStoreResponse.Status + 28, // 11: feast.core.ListIngestionJobsRequest.filter:type_name -> feast.core.ListIngestionJobsRequest.Filter + 31, // 12: feast.core.ListIngestionJobsResponse.jobs:type_name -> feast.core.IngestionJob + 32, // 13: feast.core.ListIngestionJobsRequest.Filter.feature_set_reference:type_name -> feast.core.FeatureSetReference + 10, // 14: feast.core.CoreService.GetFeastCoreVersion:input_type -> feast.core.GetFeastCoreVersionRequest + 2, // 15: feast.core.CoreService.GetFeatureSet:input_type -> feast.core.GetFeatureSetRequest + 4, // 16: feast.core.CoreService.ListFeatureSets:input_type -> feast.core.ListFeatureSetsRequest + 6, // 17: feast.core.CoreService.ListStores:input_type -> feast.core.ListStoresRequest + 8, // 18: feast.core.CoreService.ApplyFeatureSet:input_type -> feast.core.ApplyFeatureSetRequest + 12, // 19: feast.core.CoreService.UpdateStore:input_type -> feast.core.UpdateStoreRequest + 14, // 20: feast.core.CoreService.CreateProject:input_type -> feast.core.CreateProjectRequest + 16, // 21: feast.core.CoreService.ArchiveProject:input_type -> feast.core.ArchiveProjectRequest + 18, // 22: feast.core.CoreService.ListProjects:input_type -> feast.core.ListProjectsRequest + 20, // 23: feast.core.CoreService.ListIngestionJobs:input_type -> feast.core.ListIngestionJobsRequest + 22, // 24: feast.core.CoreService.RestartIngestionJob:input_type -> feast.core.RestartIngestionJobRequest + 24, // 25: feast.core.CoreService.StopIngestionJob:input_type -> feast.core.StopIngestionJobRequest + 11, // 26: feast.core.CoreService.GetFeastCoreVersion:output_type -> feast.core.GetFeastCoreVersionResponse + 3, // 27: feast.core.CoreService.GetFeatureSet:output_type -> feast.core.GetFeatureSetResponse + 5, // 28: feast.core.CoreService.ListFeatureSets:output_type -> feast.core.ListFeatureSetsResponse + 7, // 29: feast.core.CoreService.ListStores:output_type -> feast.core.ListStoresResponse + 9, // 30: feast.core.CoreService.ApplyFeatureSet:output_type -> feast.core.ApplyFeatureSetResponse + 13, // 31: feast.core.CoreService.UpdateStore:output_type -> feast.core.UpdateStoreResponse + 15, // 32: feast.core.CoreService.CreateProject:output_type -> feast.core.CreateProjectResponse + 17, // 33: feast.core.CoreService.ArchiveProject:output_type -> feast.core.ArchiveProjectResponse + 19, // 34: feast.core.CoreService.ListProjects:output_type -> feast.core.ListProjectsResponse + 21, // 35: feast.core.CoreService.ListIngestionJobs:output_type -> feast.core.ListIngestionJobsResponse + 23, // 36: feast.core.CoreService.RestartIngestionJob:output_type -> feast.core.RestartIngestionJobResponse + 25, // 37: feast.core.CoreService.StopIngestionJob:output_type -> feast.core.StopIngestionJobResponse + 26, // [26:38] is the sub-list for method output_type + 14, // [14:26] is the sub-list for method input_type + 14, // [14:14] is the sub-list for extension type_name + 14, // [14:14] is the sub-list for extension extendee + 0, // [0:14] is the sub-list for field type_name +} + +func init() { file_feast_core_CoreService_proto_init() } +func file_feast_core_CoreService_proto_init() { + if File_feast_core_CoreService_proto != nil { + return + } + file_feast_core_FeatureSet_proto_init() + file_feast_core_Store_proto_init() + file_feast_core_FeatureSetReference_proto_init() + file_feast_core_IngestionJob_proto_init() + if !protoimpl.UnsafeEnabled { + file_feast_core_CoreService_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetFeatureSetRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetFeatureSetResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[2].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ListFeatureSetsRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[3].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ListFeatureSetsResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[4].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ListStoresRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[5].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ListStoresResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[6].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ApplyFeatureSetRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[7].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ApplyFeatureSetResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[8].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetFeastCoreVersionRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[9].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*GetFeastCoreVersionResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[10].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*UpdateStoreRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[11].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*UpdateStoreResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[12].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*CreateProjectRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[13].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*CreateProjectResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[14].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ArchiveProjectRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[15].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ArchiveProjectResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[16].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ListProjectsRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[17].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ListProjectsResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[18].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ListIngestionJobsRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[19].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ListIngestionJobsResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[20].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*RestartIngestionJobRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[21].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*RestartIngestionJobResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[22].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*StopIngestionJobRequest); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[23].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*StopIngestionJobResponse); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[24].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ListFeatureSetsRequest_Filter); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[25].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ListStoresRequest_Filter); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_CoreService_proto_msgTypes[26].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ListIngestionJobsRequest_Filter); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_feast_core_CoreService_proto_rawDesc, + NumEnums: 2, + NumMessages: 27, + NumExtensions: 0, + NumServices: 1, + }, + GoTypes: file_feast_core_CoreService_proto_goTypes, + DependencyIndexes: file_feast_core_CoreService_proto_depIdxs, + EnumInfos: file_feast_core_CoreService_proto_enumTypes, + MessageInfos: file_feast_core_CoreService_proto_msgTypes, + }.Build() + File_feast_core_CoreService_proto = out.File + file_feast_core_CoreService_proto_rawDesc = nil + file_feast_core_CoreService_proto_goTypes = nil + file_feast_core_CoreService_proto_depIdxs = nil +} + +// Reference imports to suppress errors if they are not otherwise used. +var _ context.Context +var _ grpc.ClientConnInterface + +// This is a compile-time assertion to ensure that this generated file +// is compatible with the grpc package it is being compiled against. +const _ = grpc.SupportPackageIsVersion6 + +// CoreServiceClient is the client API for CoreService service. +// +// For semantics around ctx use and closing/ending streaming RPCs, please refer to https://godoc.org/google.golang.org/grpc#ClientConn.NewStream. +type CoreServiceClient interface { + // Retrieve version information about this Feast deployment + GetFeastCoreVersion(ctx context.Context, in *GetFeastCoreVersionRequest, opts ...grpc.CallOption) (*GetFeastCoreVersionResponse, error) + // Returns a specific feature set + GetFeatureSet(ctx context.Context, in *GetFeatureSetRequest, opts ...grpc.CallOption) (*GetFeatureSetResponse, error) + // Retrieve feature set details given a filter. + // + // Returns all feature sets matching that filter. If none are found, + // an empty list will be returned. + // If no filter is provided in the request, the response will contain all the feature + // sets currently stored in the registry. + ListFeatureSets(ctx context.Context, in *ListFeatureSetsRequest, opts ...grpc.CallOption) (*ListFeatureSetsResponse, error) + // Retrieve store details given a filter. + // + // Returns all stores matching that filter. If none are found, an empty list will be returned. + // If no filter is provided in the request, the response will contain all the stores currently + // stored in the registry. + ListStores(ctx context.Context, in *ListStoresRequest, opts ...grpc.CallOption) (*ListStoresResponse, error) + // Create or update and existing feature set. + // + // This function is idempotent - it will not create a new feature set if schema does not change. + // If an existing feature set is updated, core will advance the version number, which will be + // returned in response. + ApplyFeatureSet(ctx context.Context, in *ApplyFeatureSetRequest, opts ...grpc.CallOption) (*ApplyFeatureSetResponse, error) + // Updates core with the configuration of the store. + // + // If the changes are valid, core will return the given store configuration in response, and + // start or update the necessary feature population jobs for the updated store. + UpdateStore(ctx context.Context, in *UpdateStoreRequest, opts ...grpc.CallOption) (*UpdateStoreResponse, error) + // Creates a project. Projects serve as namespaces within which resources like features will be + // created. Both feature set names as well as field names must be unique within a project. Project + // names themselves must be globally unique. + CreateProject(ctx context.Context, in *CreateProjectRequest, opts ...grpc.CallOption) (*CreateProjectResponse, error) + // Archives a project. Archived projects will continue to exist and function, but won't be visible + // through the Core API. Any existing ingestion or serving requests will continue to function, + // but will result in warning messages being logged. It is not possible to unarchive a project + // through the Core API + ArchiveProject(ctx context.Context, in *ArchiveProjectRequest, opts ...grpc.CallOption) (*ArchiveProjectResponse, error) + // Lists all projects active projects. + ListProjects(ctx context.Context, in *ListProjectsRequest, opts ...grpc.CallOption) (*ListProjectsResponse, error) + // List Ingestion Jobs given an optional filter. + // Returns allow ingestions matching the given request filter. + // Returns all ingestion jobs if no filter is provided. + // Returns an empty list if no ingestion jobs match the filter. + ListIngestionJobs(ctx context.Context, in *ListIngestionJobsRequest, opts ...grpc.CallOption) (*ListIngestionJobsResponse, error) + // Restart an Ingestion Job. Restarts the ingestion job with the given job id. + // NOTE: Data might be lost during the restart for some job runners. + // Does not support stopping a job in a transitional (ie pending, suspending, aborting), + // terminal state (ie suspended or aborted) or unknown status + RestartIngestionJob(ctx context.Context, in *RestartIngestionJobRequest, opts ...grpc.CallOption) (*RestartIngestionJobResponse, error) + // Stop an Ingestion Job. Stop (Aborts) the ingestion job with the given job id. + // Does nothing if the target job if already in a terminal state (ie suspended or aborted). + // Does not support stopping a job in a transitional (ie pending, suspending, aborting) or unknown status + StopIngestionJob(ctx context.Context, in *StopIngestionJobRequest, opts ...grpc.CallOption) (*StopIngestionJobResponse, error) +} + +type coreServiceClient struct { + cc grpc.ClientConnInterface +} + +func NewCoreServiceClient(cc grpc.ClientConnInterface) CoreServiceClient { + return &coreServiceClient{cc} +} + +func (c *coreServiceClient) GetFeastCoreVersion(ctx context.Context, in *GetFeastCoreVersionRequest, opts ...grpc.CallOption) (*GetFeastCoreVersionResponse, error) { + out := new(GetFeastCoreVersionResponse) + err := c.cc.Invoke(ctx, "/feast.core.CoreService/GetFeastCoreVersion", in, out, opts...) + if err != nil { + return nil, err + } + return out, nil +} + +func (c *coreServiceClient) GetFeatureSet(ctx context.Context, in *GetFeatureSetRequest, opts ...grpc.CallOption) (*GetFeatureSetResponse, error) { + out := new(GetFeatureSetResponse) + err := c.cc.Invoke(ctx, "/feast.core.CoreService/GetFeatureSet", in, out, opts...) + if err != nil { + return nil, err + } + return out, nil +} + +func (c *coreServiceClient) ListFeatureSets(ctx context.Context, in *ListFeatureSetsRequest, opts ...grpc.CallOption) (*ListFeatureSetsResponse, error) { + out := new(ListFeatureSetsResponse) + err := c.cc.Invoke(ctx, "/feast.core.CoreService/ListFeatureSets", in, out, opts...) + if err != nil { + return nil, err + } + return out, nil +} + +func (c *coreServiceClient) ListStores(ctx context.Context, in *ListStoresRequest, opts ...grpc.CallOption) (*ListStoresResponse, error) { + out := new(ListStoresResponse) + err := c.cc.Invoke(ctx, "/feast.core.CoreService/ListStores", in, out, opts...) + if err != nil { + return nil, err + } + return out, nil +} + +func (c *coreServiceClient) ApplyFeatureSet(ctx context.Context, in *ApplyFeatureSetRequest, opts ...grpc.CallOption) (*ApplyFeatureSetResponse, error) { + out := new(ApplyFeatureSetResponse) + err := c.cc.Invoke(ctx, "/feast.core.CoreService/ApplyFeatureSet", in, out, opts...) + if err != nil { + return nil, err + } + return out, nil +} + +func (c *coreServiceClient) UpdateStore(ctx context.Context, in *UpdateStoreRequest, opts ...grpc.CallOption) (*UpdateStoreResponse, error) { + out := new(UpdateStoreResponse) + err := c.cc.Invoke(ctx, "/feast.core.CoreService/UpdateStore", in, out, opts...) + if err != nil { + return nil, err + } + return out, nil +} + +func (c *coreServiceClient) CreateProject(ctx context.Context, in *CreateProjectRequest, opts ...grpc.CallOption) (*CreateProjectResponse, error) { + out := new(CreateProjectResponse) + err := c.cc.Invoke(ctx, "/feast.core.CoreService/CreateProject", in, out, opts...) + if err != nil { + return nil, err + } + return out, nil +} + +func (c *coreServiceClient) ArchiveProject(ctx context.Context, in *ArchiveProjectRequest, opts ...grpc.CallOption) (*ArchiveProjectResponse, error) { + out := new(ArchiveProjectResponse) + err := c.cc.Invoke(ctx, "/feast.core.CoreService/ArchiveProject", in, out, opts...) + if err != nil { + return nil, err + } + return out, nil +} + +func (c *coreServiceClient) ListProjects(ctx context.Context, in *ListProjectsRequest, opts ...grpc.CallOption) (*ListProjectsResponse, error) { + out := new(ListProjectsResponse) + err := c.cc.Invoke(ctx, "/feast.core.CoreService/ListProjects", in, out, opts...) + if err != nil { + return nil, err + } + return out, nil +} + +func (c *coreServiceClient) ListIngestionJobs(ctx context.Context, in *ListIngestionJobsRequest, opts ...grpc.CallOption) (*ListIngestionJobsResponse, error) { + out := new(ListIngestionJobsResponse) + err := c.cc.Invoke(ctx, "/feast.core.CoreService/ListIngestionJobs", in, out, opts...) + if err != nil { + return nil, err + } + return out, nil +} + +func (c *coreServiceClient) RestartIngestionJob(ctx context.Context, in *RestartIngestionJobRequest, opts ...grpc.CallOption) (*RestartIngestionJobResponse, error) { + out := new(RestartIngestionJobResponse) + err := c.cc.Invoke(ctx, "/feast.core.CoreService/RestartIngestionJob", in, out, opts...) + if err != nil { + return nil, err + } + return out, nil +} + +func (c *coreServiceClient) StopIngestionJob(ctx context.Context, in *StopIngestionJobRequest, opts ...grpc.CallOption) (*StopIngestionJobResponse, error) { + out := new(StopIngestionJobResponse) + err := c.cc.Invoke(ctx, "/feast.core.CoreService/StopIngestionJob", in, out, opts...) + if err != nil { + return nil, err + } + return out, nil +} + +// CoreServiceServer is the server API for CoreService service. +type CoreServiceServer interface { + // Retrieve version information about this Feast deployment + GetFeastCoreVersion(context.Context, *GetFeastCoreVersionRequest) (*GetFeastCoreVersionResponse, error) + // Returns a specific feature set + GetFeatureSet(context.Context, *GetFeatureSetRequest) (*GetFeatureSetResponse, error) + // Retrieve feature set details given a filter. + // + // Returns all feature sets matching that filter. If none are found, + // an empty list will be returned. + // If no filter is provided in the request, the response will contain all the feature + // sets currently stored in the registry. + ListFeatureSets(context.Context, *ListFeatureSetsRequest) (*ListFeatureSetsResponse, error) + // Retrieve store details given a filter. + // + // Returns all stores matching that filter. If none are found, an empty list will be returned. + // If no filter is provided in the request, the response will contain all the stores currently + // stored in the registry. + ListStores(context.Context, *ListStoresRequest) (*ListStoresResponse, error) + // Create or update and existing feature set. + // + // This function is idempotent - it will not create a new feature set if schema does not change. + // If an existing feature set is updated, core will advance the version number, which will be + // returned in response. + ApplyFeatureSet(context.Context, *ApplyFeatureSetRequest) (*ApplyFeatureSetResponse, error) + // Updates core with the configuration of the store. + // + // If the changes are valid, core will return the given store configuration in response, and + // start or update the necessary feature population jobs for the updated store. + UpdateStore(context.Context, *UpdateStoreRequest) (*UpdateStoreResponse, error) + // Creates a project. Projects serve as namespaces within which resources like features will be + // created. Both feature set names as well as field names must be unique within a project. Project + // names themselves must be globally unique. + CreateProject(context.Context, *CreateProjectRequest) (*CreateProjectResponse, error) + // Archives a project. Archived projects will continue to exist and function, but won't be visible + // through the Core API. Any existing ingestion or serving requests will continue to function, + // but will result in warning messages being logged. It is not possible to unarchive a project + // through the Core API + ArchiveProject(context.Context, *ArchiveProjectRequest) (*ArchiveProjectResponse, error) + // Lists all projects active projects. + ListProjects(context.Context, *ListProjectsRequest) (*ListProjectsResponse, error) + // List Ingestion Jobs given an optional filter. + // Returns allow ingestions matching the given request filter. + // Returns all ingestion jobs if no filter is provided. + // Returns an empty list if no ingestion jobs match the filter. + ListIngestionJobs(context.Context, *ListIngestionJobsRequest) (*ListIngestionJobsResponse, error) + // Restart an Ingestion Job. Restarts the ingestion job with the given job id. + // NOTE: Data might be lost during the restart for some job runners. + // Does not support stopping a job in a transitional (ie pending, suspending, aborting), + // terminal state (ie suspended or aborted) or unknown status + RestartIngestionJob(context.Context, *RestartIngestionJobRequest) (*RestartIngestionJobResponse, error) + // Stop an Ingestion Job. Stop (Aborts) the ingestion job with the given job id. + // Does nothing if the target job if already in a terminal state (ie suspended or aborted). + // Does not support stopping a job in a transitional (ie pending, suspending, aborting) or unknown status + StopIngestionJob(context.Context, *StopIngestionJobRequest) (*StopIngestionJobResponse, error) +} + +// UnimplementedCoreServiceServer can be embedded to have forward compatible implementations. +type UnimplementedCoreServiceServer struct { +} + +func (*UnimplementedCoreServiceServer) GetFeastCoreVersion(context.Context, *GetFeastCoreVersionRequest) (*GetFeastCoreVersionResponse, error) { + return nil, status.Errorf(codes.Unimplemented, "method GetFeastCoreVersion not implemented") +} +func (*UnimplementedCoreServiceServer) GetFeatureSet(context.Context, *GetFeatureSetRequest) (*GetFeatureSetResponse, error) { + return nil, status.Errorf(codes.Unimplemented, "method GetFeatureSet not implemented") +} +func (*UnimplementedCoreServiceServer) ListFeatureSets(context.Context, *ListFeatureSetsRequest) (*ListFeatureSetsResponse, error) { + return nil, status.Errorf(codes.Unimplemented, "method ListFeatureSets not implemented") +} +func (*UnimplementedCoreServiceServer) ListStores(context.Context, *ListStoresRequest) (*ListStoresResponse, error) { + return nil, status.Errorf(codes.Unimplemented, "method ListStores not implemented") +} +func (*UnimplementedCoreServiceServer) ApplyFeatureSet(context.Context, *ApplyFeatureSetRequest) (*ApplyFeatureSetResponse, error) { + return nil, status.Errorf(codes.Unimplemented, "method ApplyFeatureSet not implemented") +} +func (*UnimplementedCoreServiceServer) UpdateStore(context.Context, *UpdateStoreRequest) (*UpdateStoreResponse, error) { + return nil, status.Errorf(codes.Unimplemented, "method UpdateStore not implemented") +} +func (*UnimplementedCoreServiceServer) CreateProject(context.Context, *CreateProjectRequest) (*CreateProjectResponse, error) { + return nil, status.Errorf(codes.Unimplemented, "method CreateProject not implemented") +} +func (*UnimplementedCoreServiceServer) ArchiveProject(context.Context, *ArchiveProjectRequest) (*ArchiveProjectResponse, error) { + return nil, status.Errorf(codes.Unimplemented, "method ArchiveProject not implemented") +} +func (*UnimplementedCoreServiceServer) ListProjects(context.Context, *ListProjectsRequest) (*ListProjectsResponse, error) { + return nil, status.Errorf(codes.Unimplemented, "method ListProjects not implemented") +} +func (*UnimplementedCoreServiceServer) ListIngestionJobs(context.Context, *ListIngestionJobsRequest) (*ListIngestionJobsResponse, error) { + return nil, status.Errorf(codes.Unimplemented, "method ListIngestionJobs not implemented") +} +func (*UnimplementedCoreServiceServer) RestartIngestionJob(context.Context, *RestartIngestionJobRequest) (*RestartIngestionJobResponse, error) { + return nil, status.Errorf(codes.Unimplemented, "method RestartIngestionJob not implemented") +} +func (*UnimplementedCoreServiceServer) StopIngestionJob(context.Context, *StopIngestionJobRequest) (*StopIngestionJobResponse, error) { + return nil, status.Errorf(codes.Unimplemented, "method StopIngestionJob not implemented") +} + +func RegisterCoreServiceServer(s *grpc.Server, srv CoreServiceServer) { + s.RegisterService(&_CoreService_serviceDesc, srv) +} + +func _CoreService_GetFeastCoreVersion_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(GetFeastCoreVersionRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(CoreServiceServer).GetFeastCoreVersion(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: "/feast.core.CoreService/GetFeastCoreVersion", + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(CoreServiceServer).GetFeastCoreVersion(ctx, req.(*GetFeastCoreVersionRequest)) + } + return interceptor(ctx, in, info, handler) +} + +func _CoreService_GetFeatureSet_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(GetFeatureSetRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(CoreServiceServer).GetFeatureSet(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: "/feast.core.CoreService/GetFeatureSet", + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(CoreServiceServer).GetFeatureSet(ctx, req.(*GetFeatureSetRequest)) + } + return interceptor(ctx, in, info, handler) +} + +func _CoreService_ListFeatureSets_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(ListFeatureSetsRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(CoreServiceServer).ListFeatureSets(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: "/feast.core.CoreService/ListFeatureSets", + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(CoreServiceServer).ListFeatureSets(ctx, req.(*ListFeatureSetsRequest)) + } + return interceptor(ctx, in, info, handler) +} + +func _CoreService_ListStores_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(ListStoresRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(CoreServiceServer).ListStores(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: "/feast.core.CoreService/ListStores", + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(CoreServiceServer).ListStores(ctx, req.(*ListStoresRequest)) + } + return interceptor(ctx, in, info, handler) +} + +func _CoreService_ApplyFeatureSet_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(ApplyFeatureSetRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(CoreServiceServer).ApplyFeatureSet(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: "/feast.core.CoreService/ApplyFeatureSet", + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(CoreServiceServer).ApplyFeatureSet(ctx, req.(*ApplyFeatureSetRequest)) + } + return interceptor(ctx, in, info, handler) +} + +func _CoreService_UpdateStore_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(UpdateStoreRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(CoreServiceServer).UpdateStore(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: "/feast.core.CoreService/UpdateStore", + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(CoreServiceServer).UpdateStore(ctx, req.(*UpdateStoreRequest)) + } + return interceptor(ctx, in, info, handler) +} + +func _CoreService_CreateProject_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(CreateProjectRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(CoreServiceServer).CreateProject(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: "/feast.core.CoreService/CreateProject", + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(CoreServiceServer).CreateProject(ctx, req.(*CreateProjectRequest)) + } + return interceptor(ctx, in, info, handler) +} + +func _CoreService_ArchiveProject_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(ArchiveProjectRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(CoreServiceServer).ArchiveProject(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: "/feast.core.CoreService/ArchiveProject", + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(CoreServiceServer).ArchiveProject(ctx, req.(*ArchiveProjectRequest)) + } + return interceptor(ctx, in, info, handler) +} + +func _CoreService_ListProjects_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(ListProjectsRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(CoreServiceServer).ListProjects(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: "/feast.core.CoreService/ListProjects", + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(CoreServiceServer).ListProjects(ctx, req.(*ListProjectsRequest)) + } + return interceptor(ctx, in, info, handler) +} + +func _CoreService_ListIngestionJobs_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(ListIngestionJobsRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(CoreServiceServer).ListIngestionJobs(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: "/feast.core.CoreService/ListIngestionJobs", + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(CoreServiceServer).ListIngestionJobs(ctx, req.(*ListIngestionJobsRequest)) + } + return interceptor(ctx, in, info, handler) +} + +func _CoreService_RestartIngestionJob_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(RestartIngestionJobRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(CoreServiceServer).RestartIngestionJob(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: "/feast.core.CoreService/RestartIngestionJob", + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(CoreServiceServer).RestartIngestionJob(ctx, req.(*RestartIngestionJobRequest)) + } + return interceptor(ctx, in, info, handler) +} + +func _CoreService_StopIngestionJob_Handler(srv interface{}, ctx context.Context, dec func(interface{}) error, interceptor grpc.UnaryServerInterceptor) (interface{}, error) { + in := new(StopIngestionJobRequest) + if err := dec(in); err != nil { + return nil, err + } + if interceptor == nil { + return srv.(CoreServiceServer).StopIngestionJob(ctx, in) + } + info := &grpc.UnaryServerInfo{ + Server: srv, + FullMethod: "/feast.core.CoreService/StopIngestionJob", + } + handler := func(ctx context.Context, req interface{}) (interface{}, error) { + return srv.(CoreServiceServer).StopIngestionJob(ctx, req.(*StopIngestionJobRequest)) + } + return interceptor(ctx, in, info, handler) +} + +var _CoreService_serviceDesc = grpc.ServiceDesc{ + ServiceName: "feast.core.CoreService", + HandlerType: (*CoreServiceServer)(nil), + Methods: []grpc.MethodDesc{ + { + MethodName: "GetFeastCoreVersion", + Handler: _CoreService_GetFeastCoreVersion_Handler, + }, + { + MethodName: "GetFeatureSet", + Handler: _CoreService_GetFeatureSet_Handler, + }, + { + MethodName: "ListFeatureSets", + Handler: _CoreService_ListFeatureSets_Handler, + }, + { + MethodName: "ListStores", + Handler: _CoreService_ListStores_Handler, + }, + { + MethodName: "ApplyFeatureSet", + Handler: _CoreService_ApplyFeatureSet_Handler, + }, + { + MethodName: "UpdateStore", + Handler: _CoreService_UpdateStore_Handler, + }, + { + MethodName: "CreateProject", + Handler: _CoreService_CreateProject_Handler, + }, + { + MethodName: "ArchiveProject", + Handler: _CoreService_ArchiveProject_Handler, + }, + { + MethodName: "ListProjects", + Handler: _CoreService_ListProjects_Handler, + }, + { + MethodName: "ListIngestionJobs", + Handler: _CoreService_ListIngestionJobs_Handler, + }, + { + MethodName: "RestartIngestionJob", + Handler: _CoreService_RestartIngestionJob_Handler, + }, + { + MethodName: "StopIngestionJob", + Handler: _CoreService_StopIngestionJob_Handler, + }, + }, + Streams: []grpc.StreamDesc{}, + Metadata: "feast/core/CoreService.proto", +} diff --git a/sdk/go/protos/feast/core/FeatureSet.pb.go b/sdk/go/protos/feast/core/FeatureSet.pb.go new file mode 100644 index 00000000000..bbf79e7d2a4 --- /dev/null +++ b/sdk/go/protos/feast/core/FeatureSet.pb.go @@ -0,0 +1,1430 @@ +// +// * Copyright 2019 The Feast Authors +// * +// * Licensed under the Apache License, Version 2.0 (the "License"); +// * you may not use this file except in compliance with the License. +// * You may obtain a copy of the License at +// * +// * https://www.apache.org/licenses/LICENSE-2.0 +// * +// * Unless required by applicable law or agreed to in writing, software +// * distributed under the License is distributed on an "AS IS" BASIS, +// * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// * See the License for the specific language governing permissions and +// * limitations under the License. +// + +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.21.0 +// protoc v3.10.1 +// source: feast/core/FeatureSet.proto + +package core + +import ( + types "github.com/gojek/feast/sdk/go/protos/feast/types" + v0 "github.com/gojek/feast/sdk/go/protos/tensorflow_metadata/proto/v0" + proto "github.com/golang/protobuf/proto" + duration "github.com/golang/protobuf/ptypes/duration" + timestamp "github.com/golang/protobuf/ptypes/timestamp" + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +// This is a compile-time assertion that a sufficiently up-to-date version +// of the legacy proto package is being used. +const _ = proto.ProtoPackageIsVersion4 + +type FeatureSetStatus int32 + +const ( + FeatureSetStatus_STATUS_INVALID FeatureSetStatus = 0 + FeatureSetStatus_STATUS_PENDING FeatureSetStatus = 1 + FeatureSetStatus_STATUS_READY FeatureSetStatus = 2 +) + +// Enum value maps for FeatureSetStatus. +var ( + FeatureSetStatus_name = map[int32]string{ + 0: "STATUS_INVALID", + 1: "STATUS_PENDING", + 2: "STATUS_READY", + } + FeatureSetStatus_value = map[string]int32{ + "STATUS_INVALID": 0, + "STATUS_PENDING": 1, + "STATUS_READY": 2, + } +) + +func (x FeatureSetStatus) Enum() *FeatureSetStatus { + p := new(FeatureSetStatus) + *p = x + return p +} + +func (x FeatureSetStatus) String() string { + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) +} + +func (FeatureSetStatus) Descriptor() protoreflect.EnumDescriptor { + return file_feast_core_FeatureSet_proto_enumTypes[0].Descriptor() +} + +func (FeatureSetStatus) Type() protoreflect.EnumType { + return &file_feast_core_FeatureSet_proto_enumTypes[0] +} + +func (x FeatureSetStatus) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) +} + +// Deprecated: Use FeatureSetStatus.Descriptor instead. +func (FeatureSetStatus) EnumDescriptor() ([]byte, []int) { + return file_feast_core_FeatureSet_proto_rawDescGZIP(), []int{0} +} + +type FeatureSet struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // User-specified specifications of this feature set. + Spec *FeatureSetSpec `protobuf:"bytes,1,opt,name=spec,proto3" json:"spec,omitempty"` + // System-populated metadata for this feature set. + Meta *FeatureSetMeta `protobuf:"bytes,2,opt,name=meta,proto3" json:"meta,omitempty"` +} + +func (x *FeatureSet) Reset() { + *x = FeatureSet{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_FeatureSet_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *FeatureSet) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*FeatureSet) ProtoMessage() {} + +func (x *FeatureSet) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_FeatureSet_proto_msgTypes[0] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use FeatureSet.ProtoReflect.Descriptor instead. +func (*FeatureSet) Descriptor() ([]byte, []int) { + return file_feast_core_FeatureSet_proto_rawDescGZIP(), []int{0} +} + +func (x *FeatureSet) GetSpec() *FeatureSetSpec { + if x != nil { + return x.Spec + } + return nil +} + +func (x *FeatureSet) GetMeta() *FeatureSetMeta { + if x != nil { + return x.Meta + } + return nil +} + +type FeatureSetSpec struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Name of project that this feature set belongs to. + Project string `protobuf:"bytes,7,opt,name=project,proto3" json:"project,omitempty"` + // Name of the feature set. Must be unique. + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` + // Feature set version. + Version int32 `protobuf:"varint,2,opt,name=version,proto3" json:"version,omitempty"` + // List of entities contained within this featureSet. + // This allows the feature to be used during joins between feature sets. + // If the featureSet is ingested into a store that supports keys, this value + // will be made a key. + Entities []*EntitySpec `protobuf:"bytes,3,rep,name=entities,proto3" json:"entities,omitempty"` + // List of features contained within this featureSet. + Features []*FeatureSpec `protobuf:"bytes,4,rep,name=features,proto3" json:"features,omitempty"` + // Features in this feature set will only be retrieved if they are found + // after [time - max_age]. Missing or older feature values will be returned + // as nulls and indicated to end user + MaxAge *duration.Duration `protobuf:"bytes,5,opt,name=max_age,json=maxAge,proto3" json:"max_age,omitempty"` + // Optional. Source on which feature rows can be found. + // If not set, source will be set to the default value configured in Feast Core. + Source *Source `protobuf:"bytes,6,opt,name=source,proto3" json:"source,omitempty"` +} + +func (x *FeatureSetSpec) Reset() { + *x = FeatureSetSpec{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_FeatureSet_proto_msgTypes[1] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *FeatureSetSpec) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*FeatureSetSpec) ProtoMessage() {} + +func (x *FeatureSetSpec) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_FeatureSet_proto_msgTypes[1] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use FeatureSetSpec.ProtoReflect.Descriptor instead. +func (*FeatureSetSpec) Descriptor() ([]byte, []int) { + return file_feast_core_FeatureSet_proto_rawDescGZIP(), []int{1} +} + +func (x *FeatureSetSpec) GetProject() string { + if x != nil { + return x.Project + } + return "" +} + +func (x *FeatureSetSpec) GetName() string { + if x != nil { + return x.Name + } + return "" +} + +func (x *FeatureSetSpec) GetVersion() int32 { + if x != nil { + return x.Version + } + return 0 +} + +func (x *FeatureSetSpec) GetEntities() []*EntitySpec { + if x != nil { + return x.Entities + } + return nil +} + +func (x *FeatureSetSpec) GetFeatures() []*FeatureSpec { + if x != nil { + return x.Features + } + return nil +} + +func (x *FeatureSetSpec) GetMaxAge() *duration.Duration { + if x != nil { + return x.MaxAge + } + return nil +} + +func (x *FeatureSetSpec) GetSource() *Source { + if x != nil { + return x.Source + } + return nil +} + +type EntitySpec struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Name of the entity. + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` + // Value type of the feature. + ValueType types.ValueType_Enum `protobuf:"varint,2,opt,name=value_type,json=valueType,proto3,enum=feast.types.ValueType_Enum" json:"value_type,omitempty"` + // Types that are assignable to PresenceConstraints: + // *EntitySpec_Presence + // *EntitySpec_GroupPresence + PresenceConstraints isEntitySpec_PresenceConstraints `protobuf_oneof:"presence_constraints"` + // The shape of the feature which governs the number of values that appear in + // each example. + // + // Types that are assignable to ShapeType: + // *EntitySpec_Shape + // *EntitySpec_ValueCount + ShapeType isEntitySpec_ShapeType `protobuf_oneof:"shape_type"` + // Domain for the values of the feature. + // + // Types that are assignable to DomainInfo: + // *EntitySpec_Domain + // *EntitySpec_IntDomain + // *EntitySpec_FloatDomain + // *EntitySpec_StringDomain + // *EntitySpec_BoolDomain + // *EntitySpec_StructDomain + // *EntitySpec_NaturalLanguageDomain + // *EntitySpec_ImageDomain + // *EntitySpec_MidDomain + // *EntitySpec_UrlDomain + // *EntitySpec_TimeDomain + // *EntitySpec_TimeOfDayDomain + DomainInfo isEntitySpec_DomainInfo `protobuf_oneof:"domain_info"` +} + +func (x *EntitySpec) Reset() { + *x = EntitySpec{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_FeatureSet_proto_msgTypes[2] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *EntitySpec) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*EntitySpec) ProtoMessage() {} + +func (x *EntitySpec) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_FeatureSet_proto_msgTypes[2] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use EntitySpec.ProtoReflect.Descriptor instead. +func (*EntitySpec) Descriptor() ([]byte, []int) { + return file_feast_core_FeatureSet_proto_rawDescGZIP(), []int{2} +} + +func (x *EntitySpec) GetName() string { + if x != nil { + return x.Name + } + return "" +} + +func (x *EntitySpec) GetValueType() types.ValueType_Enum { + if x != nil { + return x.ValueType + } + return types.ValueType_INVALID +} + +func (m *EntitySpec) GetPresenceConstraints() isEntitySpec_PresenceConstraints { + if m != nil { + return m.PresenceConstraints + } + return nil +} + +func (x *EntitySpec) GetPresence() *v0.FeaturePresence { + if x, ok := x.GetPresenceConstraints().(*EntitySpec_Presence); ok { + return x.Presence + } + return nil +} + +func (x *EntitySpec) GetGroupPresence() *v0.FeaturePresenceWithinGroup { + if x, ok := x.GetPresenceConstraints().(*EntitySpec_GroupPresence); ok { + return x.GroupPresence + } + return nil +} + +func (m *EntitySpec) GetShapeType() isEntitySpec_ShapeType { + if m != nil { + return m.ShapeType + } + return nil +} + +func (x *EntitySpec) GetShape() *v0.FixedShape { + if x, ok := x.GetShapeType().(*EntitySpec_Shape); ok { + return x.Shape + } + return nil +} + +func (x *EntitySpec) GetValueCount() *v0.ValueCount { + if x, ok := x.GetShapeType().(*EntitySpec_ValueCount); ok { + return x.ValueCount + } + return nil +} + +func (m *EntitySpec) GetDomainInfo() isEntitySpec_DomainInfo { + if m != nil { + return m.DomainInfo + } + return nil +} + +func (x *EntitySpec) GetDomain() string { + if x, ok := x.GetDomainInfo().(*EntitySpec_Domain); ok { + return x.Domain + } + return "" +} + +func (x *EntitySpec) GetIntDomain() *v0.IntDomain { + if x, ok := x.GetDomainInfo().(*EntitySpec_IntDomain); ok { + return x.IntDomain + } + return nil +} + +func (x *EntitySpec) GetFloatDomain() *v0.FloatDomain { + if x, ok := x.GetDomainInfo().(*EntitySpec_FloatDomain); ok { + return x.FloatDomain + } + return nil +} + +func (x *EntitySpec) GetStringDomain() *v0.StringDomain { + if x, ok := x.GetDomainInfo().(*EntitySpec_StringDomain); ok { + return x.StringDomain + } + return nil +} + +func (x *EntitySpec) GetBoolDomain() *v0.BoolDomain { + if x, ok := x.GetDomainInfo().(*EntitySpec_BoolDomain); ok { + return x.BoolDomain + } + return nil +} + +func (x *EntitySpec) GetStructDomain() *v0.StructDomain { + if x, ok := x.GetDomainInfo().(*EntitySpec_StructDomain); ok { + return x.StructDomain + } + return nil +} + +func (x *EntitySpec) GetNaturalLanguageDomain() *v0.NaturalLanguageDomain { + if x, ok := x.GetDomainInfo().(*EntitySpec_NaturalLanguageDomain); ok { + return x.NaturalLanguageDomain + } + return nil +} + +func (x *EntitySpec) GetImageDomain() *v0.ImageDomain { + if x, ok := x.GetDomainInfo().(*EntitySpec_ImageDomain); ok { + return x.ImageDomain + } + return nil +} + +func (x *EntitySpec) GetMidDomain() *v0.MIDDomain { + if x, ok := x.GetDomainInfo().(*EntitySpec_MidDomain); ok { + return x.MidDomain + } + return nil +} + +func (x *EntitySpec) GetUrlDomain() *v0.URLDomain { + if x, ok := x.GetDomainInfo().(*EntitySpec_UrlDomain); ok { + return x.UrlDomain + } + return nil +} + +func (x *EntitySpec) GetTimeDomain() *v0.TimeDomain { + if x, ok := x.GetDomainInfo().(*EntitySpec_TimeDomain); ok { + return x.TimeDomain + } + return nil +} + +func (x *EntitySpec) GetTimeOfDayDomain() *v0.TimeOfDayDomain { + if x, ok := x.GetDomainInfo().(*EntitySpec_TimeOfDayDomain); ok { + return x.TimeOfDayDomain + } + return nil +} + +type isEntitySpec_PresenceConstraints interface { + isEntitySpec_PresenceConstraints() +} + +type EntitySpec_Presence struct { + // Constraints on the presence of this feature in the examples. + Presence *v0.FeaturePresence `protobuf:"bytes,3,opt,name=presence,proto3,oneof"` +} + +type EntitySpec_GroupPresence struct { + // Only used in the context of a "group" context, e.g., inside a sequence. + GroupPresence *v0.FeaturePresenceWithinGroup `protobuf:"bytes,4,opt,name=group_presence,json=groupPresence,proto3,oneof"` +} + +func (*EntitySpec_Presence) isEntitySpec_PresenceConstraints() {} + +func (*EntitySpec_GroupPresence) isEntitySpec_PresenceConstraints() {} + +type isEntitySpec_ShapeType interface { + isEntitySpec_ShapeType() +} + +type EntitySpec_Shape struct { + // The feature has a fixed shape corresponding to a multi-dimensional + // tensor. + Shape *v0.FixedShape `protobuf:"bytes,5,opt,name=shape,proto3,oneof"` +} + +type EntitySpec_ValueCount struct { + // The feature doesn't have a well defined shape. All we know are limits on + // the minimum and maximum number of values. + ValueCount *v0.ValueCount `protobuf:"bytes,6,opt,name=value_count,json=valueCount,proto3,oneof"` +} + +func (*EntitySpec_Shape) isEntitySpec_ShapeType() {} + +func (*EntitySpec_ValueCount) isEntitySpec_ShapeType() {} + +type isEntitySpec_DomainInfo interface { + isEntitySpec_DomainInfo() +} + +type EntitySpec_Domain struct { + // Reference to a domain defined at the schema level. + Domain string `protobuf:"bytes,7,opt,name=domain,proto3,oneof"` +} + +type EntitySpec_IntDomain struct { + // Inline definitions of domains. + IntDomain *v0.IntDomain `protobuf:"bytes,8,opt,name=int_domain,json=intDomain,proto3,oneof"` +} + +type EntitySpec_FloatDomain struct { + FloatDomain *v0.FloatDomain `protobuf:"bytes,9,opt,name=float_domain,json=floatDomain,proto3,oneof"` +} + +type EntitySpec_StringDomain struct { + StringDomain *v0.StringDomain `protobuf:"bytes,10,opt,name=string_domain,json=stringDomain,proto3,oneof"` +} + +type EntitySpec_BoolDomain struct { + BoolDomain *v0.BoolDomain `protobuf:"bytes,11,opt,name=bool_domain,json=boolDomain,proto3,oneof"` +} + +type EntitySpec_StructDomain struct { + StructDomain *v0.StructDomain `protobuf:"bytes,12,opt,name=struct_domain,json=structDomain,proto3,oneof"` +} + +type EntitySpec_NaturalLanguageDomain struct { + // Supported semantic domains. + NaturalLanguageDomain *v0.NaturalLanguageDomain `protobuf:"bytes,13,opt,name=natural_language_domain,json=naturalLanguageDomain,proto3,oneof"` +} + +type EntitySpec_ImageDomain struct { + ImageDomain *v0.ImageDomain `protobuf:"bytes,14,opt,name=image_domain,json=imageDomain,proto3,oneof"` +} + +type EntitySpec_MidDomain struct { + MidDomain *v0.MIDDomain `protobuf:"bytes,15,opt,name=mid_domain,json=midDomain,proto3,oneof"` +} + +type EntitySpec_UrlDomain struct { + UrlDomain *v0.URLDomain `protobuf:"bytes,16,opt,name=url_domain,json=urlDomain,proto3,oneof"` +} + +type EntitySpec_TimeDomain struct { + TimeDomain *v0.TimeDomain `protobuf:"bytes,17,opt,name=time_domain,json=timeDomain,proto3,oneof"` +} + +type EntitySpec_TimeOfDayDomain struct { + TimeOfDayDomain *v0.TimeOfDayDomain `protobuf:"bytes,18,opt,name=time_of_day_domain,json=timeOfDayDomain,proto3,oneof"` +} + +func (*EntitySpec_Domain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_IntDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_FloatDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_StringDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_BoolDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_StructDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_NaturalLanguageDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_ImageDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_MidDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_UrlDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_TimeDomain) isEntitySpec_DomainInfo() {} + +func (*EntitySpec_TimeOfDayDomain) isEntitySpec_DomainInfo() {} + +type FeatureSpec struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Name of the feature. + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` + // Value type of the feature. + ValueType types.ValueType_Enum `protobuf:"varint,2,opt,name=value_type,json=valueType,proto3,enum=feast.types.ValueType_Enum" json:"value_type,omitempty"` + // Types that are assignable to PresenceConstraints: + // *FeatureSpec_Presence + // *FeatureSpec_GroupPresence + PresenceConstraints isFeatureSpec_PresenceConstraints `protobuf_oneof:"presence_constraints"` + // The shape of the feature which governs the number of values that appear in + // each example. + // + // Types that are assignable to ShapeType: + // *FeatureSpec_Shape + // *FeatureSpec_ValueCount + ShapeType isFeatureSpec_ShapeType `protobuf_oneof:"shape_type"` + // Domain for the values of the feature. + // + // Types that are assignable to DomainInfo: + // *FeatureSpec_Domain + // *FeatureSpec_IntDomain + // *FeatureSpec_FloatDomain + // *FeatureSpec_StringDomain + // *FeatureSpec_BoolDomain + // *FeatureSpec_StructDomain + // *FeatureSpec_NaturalLanguageDomain + // *FeatureSpec_ImageDomain + // *FeatureSpec_MidDomain + // *FeatureSpec_UrlDomain + // *FeatureSpec_TimeDomain + // *FeatureSpec_TimeOfDayDomain + DomainInfo isFeatureSpec_DomainInfo `protobuf_oneof:"domain_info"` +} + +func (x *FeatureSpec) Reset() { + *x = FeatureSpec{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_FeatureSet_proto_msgTypes[3] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *FeatureSpec) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*FeatureSpec) ProtoMessage() {} + +func (x *FeatureSpec) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_FeatureSet_proto_msgTypes[3] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use FeatureSpec.ProtoReflect.Descriptor instead. +func (*FeatureSpec) Descriptor() ([]byte, []int) { + return file_feast_core_FeatureSet_proto_rawDescGZIP(), []int{3} +} + +func (x *FeatureSpec) GetName() string { + if x != nil { + return x.Name + } + return "" +} + +func (x *FeatureSpec) GetValueType() types.ValueType_Enum { + if x != nil { + return x.ValueType + } + return types.ValueType_INVALID +} + +func (m *FeatureSpec) GetPresenceConstraints() isFeatureSpec_PresenceConstraints { + if m != nil { + return m.PresenceConstraints + } + return nil +} + +func (x *FeatureSpec) GetPresence() *v0.FeaturePresence { + if x, ok := x.GetPresenceConstraints().(*FeatureSpec_Presence); ok { + return x.Presence + } + return nil +} + +func (x *FeatureSpec) GetGroupPresence() *v0.FeaturePresenceWithinGroup { + if x, ok := x.GetPresenceConstraints().(*FeatureSpec_GroupPresence); ok { + return x.GroupPresence + } + return nil +} + +func (m *FeatureSpec) GetShapeType() isFeatureSpec_ShapeType { + if m != nil { + return m.ShapeType + } + return nil +} + +func (x *FeatureSpec) GetShape() *v0.FixedShape { + if x, ok := x.GetShapeType().(*FeatureSpec_Shape); ok { + return x.Shape + } + return nil +} + +func (x *FeatureSpec) GetValueCount() *v0.ValueCount { + if x, ok := x.GetShapeType().(*FeatureSpec_ValueCount); ok { + return x.ValueCount + } + return nil +} + +func (m *FeatureSpec) GetDomainInfo() isFeatureSpec_DomainInfo { + if m != nil { + return m.DomainInfo + } + return nil +} + +func (x *FeatureSpec) GetDomain() string { + if x, ok := x.GetDomainInfo().(*FeatureSpec_Domain); ok { + return x.Domain + } + return "" +} + +func (x *FeatureSpec) GetIntDomain() *v0.IntDomain { + if x, ok := x.GetDomainInfo().(*FeatureSpec_IntDomain); ok { + return x.IntDomain + } + return nil +} + +func (x *FeatureSpec) GetFloatDomain() *v0.FloatDomain { + if x, ok := x.GetDomainInfo().(*FeatureSpec_FloatDomain); ok { + return x.FloatDomain + } + return nil +} + +func (x *FeatureSpec) GetStringDomain() *v0.StringDomain { + if x, ok := x.GetDomainInfo().(*FeatureSpec_StringDomain); ok { + return x.StringDomain + } + return nil +} + +func (x *FeatureSpec) GetBoolDomain() *v0.BoolDomain { + if x, ok := x.GetDomainInfo().(*FeatureSpec_BoolDomain); ok { + return x.BoolDomain + } + return nil +} + +func (x *FeatureSpec) GetStructDomain() *v0.StructDomain { + if x, ok := x.GetDomainInfo().(*FeatureSpec_StructDomain); ok { + return x.StructDomain + } + return nil +} + +func (x *FeatureSpec) GetNaturalLanguageDomain() *v0.NaturalLanguageDomain { + if x, ok := x.GetDomainInfo().(*FeatureSpec_NaturalLanguageDomain); ok { + return x.NaturalLanguageDomain + } + return nil +} + +func (x *FeatureSpec) GetImageDomain() *v0.ImageDomain { + if x, ok := x.GetDomainInfo().(*FeatureSpec_ImageDomain); ok { + return x.ImageDomain + } + return nil +} + +func (x *FeatureSpec) GetMidDomain() *v0.MIDDomain { + if x, ok := x.GetDomainInfo().(*FeatureSpec_MidDomain); ok { + return x.MidDomain + } + return nil +} + +func (x *FeatureSpec) GetUrlDomain() *v0.URLDomain { + if x, ok := x.GetDomainInfo().(*FeatureSpec_UrlDomain); ok { + return x.UrlDomain + } + return nil +} + +func (x *FeatureSpec) GetTimeDomain() *v0.TimeDomain { + if x, ok := x.GetDomainInfo().(*FeatureSpec_TimeDomain); ok { + return x.TimeDomain + } + return nil +} + +func (x *FeatureSpec) GetTimeOfDayDomain() *v0.TimeOfDayDomain { + if x, ok := x.GetDomainInfo().(*FeatureSpec_TimeOfDayDomain); ok { + return x.TimeOfDayDomain + } + return nil +} + +type isFeatureSpec_PresenceConstraints interface { + isFeatureSpec_PresenceConstraints() +} + +type FeatureSpec_Presence struct { + // Constraints on the presence of this feature in the examples. + Presence *v0.FeaturePresence `protobuf:"bytes,3,opt,name=presence,proto3,oneof"` +} + +type FeatureSpec_GroupPresence struct { + // Only used in the context of a "group" context, e.g., inside a sequence. + GroupPresence *v0.FeaturePresenceWithinGroup `protobuf:"bytes,4,opt,name=group_presence,json=groupPresence,proto3,oneof"` +} + +func (*FeatureSpec_Presence) isFeatureSpec_PresenceConstraints() {} + +func (*FeatureSpec_GroupPresence) isFeatureSpec_PresenceConstraints() {} + +type isFeatureSpec_ShapeType interface { + isFeatureSpec_ShapeType() +} + +type FeatureSpec_Shape struct { + // The feature has a fixed shape corresponding to a multi-dimensional + // tensor. + Shape *v0.FixedShape `protobuf:"bytes,5,opt,name=shape,proto3,oneof"` +} + +type FeatureSpec_ValueCount struct { + // The feature doesn't have a well defined shape. All we know are limits on + // the minimum and maximum number of values. + ValueCount *v0.ValueCount `protobuf:"bytes,6,opt,name=value_count,json=valueCount,proto3,oneof"` +} + +func (*FeatureSpec_Shape) isFeatureSpec_ShapeType() {} + +func (*FeatureSpec_ValueCount) isFeatureSpec_ShapeType() {} + +type isFeatureSpec_DomainInfo interface { + isFeatureSpec_DomainInfo() +} + +type FeatureSpec_Domain struct { + // Reference to a domain defined at the schema level. + Domain string `protobuf:"bytes,7,opt,name=domain,proto3,oneof"` +} + +type FeatureSpec_IntDomain struct { + // Inline definitions of domains. + IntDomain *v0.IntDomain `protobuf:"bytes,8,opt,name=int_domain,json=intDomain,proto3,oneof"` +} + +type FeatureSpec_FloatDomain struct { + FloatDomain *v0.FloatDomain `protobuf:"bytes,9,opt,name=float_domain,json=floatDomain,proto3,oneof"` +} + +type FeatureSpec_StringDomain struct { + StringDomain *v0.StringDomain `protobuf:"bytes,10,opt,name=string_domain,json=stringDomain,proto3,oneof"` +} + +type FeatureSpec_BoolDomain struct { + BoolDomain *v0.BoolDomain `protobuf:"bytes,11,opt,name=bool_domain,json=boolDomain,proto3,oneof"` +} + +type FeatureSpec_StructDomain struct { + StructDomain *v0.StructDomain `protobuf:"bytes,12,opt,name=struct_domain,json=structDomain,proto3,oneof"` +} + +type FeatureSpec_NaturalLanguageDomain struct { + // Supported semantic domains. + NaturalLanguageDomain *v0.NaturalLanguageDomain `protobuf:"bytes,13,opt,name=natural_language_domain,json=naturalLanguageDomain,proto3,oneof"` +} + +type FeatureSpec_ImageDomain struct { + ImageDomain *v0.ImageDomain `protobuf:"bytes,14,opt,name=image_domain,json=imageDomain,proto3,oneof"` +} + +type FeatureSpec_MidDomain struct { + MidDomain *v0.MIDDomain `protobuf:"bytes,15,opt,name=mid_domain,json=midDomain,proto3,oneof"` +} + +type FeatureSpec_UrlDomain struct { + UrlDomain *v0.URLDomain `protobuf:"bytes,16,opt,name=url_domain,json=urlDomain,proto3,oneof"` +} + +type FeatureSpec_TimeDomain struct { + TimeDomain *v0.TimeDomain `protobuf:"bytes,17,opt,name=time_domain,json=timeDomain,proto3,oneof"` +} + +type FeatureSpec_TimeOfDayDomain struct { + TimeOfDayDomain *v0.TimeOfDayDomain `protobuf:"bytes,18,opt,name=time_of_day_domain,json=timeOfDayDomain,proto3,oneof"` +} + +func (*FeatureSpec_Domain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_IntDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_FloatDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_StringDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_BoolDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_StructDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_NaturalLanguageDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_ImageDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_MidDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_UrlDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_TimeDomain) isFeatureSpec_DomainInfo() {} + +func (*FeatureSpec_TimeOfDayDomain) isFeatureSpec_DomainInfo() {} + +type FeatureSetMeta struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Created timestamp of this specific feature set. + CreatedTimestamp *timestamp.Timestamp `protobuf:"bytes,1,opt,name=created_timestamp,json=createdTimestamp,proto3" json:"created_timestamp,omitempty"` + // Status of the feature set. + // Used to indicate whether the feature set is ready for consumption or ingestion. + // Currently supports 2 states: + // 1) STATUS_PENDING - A feature set is in pending state if Feast has not spun up the jobs + // necessary to push rows for this feature set to stores subscribing to this feature set. + // 2) STATUS_READY - Feature set is ready for consumption or ingestion + Status FeatureSetStatus `protobuf:"varint,2,opt,name=status,proto3,enum=feast.core.FeatureSetStatus" json:"status,omitempty"` +} + +func (x *FeatureSetMeta) Reset() { + *x = FeatureSetMeta{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_FeatureSet_proto_msgTypes[4] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *FeatureSetMeta) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*FeatureSetMeta) ProtoMessage() {} + +func (x *FeatureSetMeta) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_FeatureSet_proto_msgTypes[4] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use FeatureSetMeta.ProtoReflect.Descriptor instead. +func (*FeatureSetMeta) Descriptor() ([]byte, []int) { + return file_feast_core_FeatureSet_proto_rawDescGZIP(), []int{4} +} + +func (x *FeatureSetMeta) GetCreatedTimestamp() *timestamp.Timestamp { + if x != nil { + return x.CreatedTimestamp + } + return nil +} + +func (x *FeatureSetMeta) GetStatus() FeatureSetStatus { + if x != nil { + return x.Status + } + return FeatureSetStatus_STATUS_INVALID +} + +var File_feast_core_FeatureSet_proto protoreflect.FileDescriptor + +var file_feast_core_FeatureSet_proto_rawDesc = []byte{ + 0x0a, 0x1b, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x46, 0x65, 0x61, + 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0a, 0x66, + 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x1a, 0x17, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2f, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2f, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x2e, 0x70, 0x72, 0x6f, + 0x74, 0x6f, 0x1a, 0x17, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x53, + 0x6f, 0x75, 0x72, 0x63, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x1e, 0x67, 0x6f, 0x6f, + 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2f, 0x64, 0x75, 0x72, + 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x1f, 0x67, 0x6f, 0x6f, + 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2f, 0x74, 0x69, 0x6d, + 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x29, 0x74, 0x65, + 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, + 0x61, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2f, 0x76, 0x30, 0x2f, 0x73, 0x63, 0x68, 0x65, 0x6d, + 0x61, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x6c, 0x0a, 0x0a, 0x46, 0x65, 0x61, 0x74, 0x75, + 0x72, 0x65, 0x53, 0x65, 0x74, 0x12, 0x2e, 0x0a, 0x04, 0x73, 0x70, 0x65, 0x63, 0x18, 0x01, 0x20, + 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, + 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x53, 0x70, 0x65, 0x63, 0x52, + 0x04, 0x73, 0x70, 0x65, 0x63, 0x12, 0x2e, 0x0a, 0x04, 0x6d, 0x65, 0x74, 0x61, 0x18, 0x02, 0x20, + 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, + 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x4d, 0x65, 0x74, 0x61, 0x52, + 0x04, 0x6d, 0x65, 0x74, 0x61, 0x22, 0xa1, 0x02, 0x0a, 0x0e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, + 0x65, 0x53, 0x65, 0x74, 0x53, 0x70, 0x65, 0x63, 0x12, 0x18, 0x0a, 0x07, 0x70, 0x72, 0x6f, 0x6a, + 0x65, 0x63, 0x74, 0x18, 0x07, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, + 0x63, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, + 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, + 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, + 0x12, 0x32, 0x0a, 0x08, 0x65, 0x6e, 0x74, 0x69, 0x74, 0x69, 0x65, 0x73, 0x18, 0x03, 0x20, 0x03, + 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, + 0x45, 0x6e, 0x74, 0x69, 0x74, 0x79, 0x53, 0x70, 0x65, 0x63, 0x52, 0x08, 0x65, 0x6e, 0x74, 0x69, + 0x74, 0x69, 0x65, 0x73, 0x12, 0x33, 0x0a, 0x08, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, + 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x17, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, + 0x6f, 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x70, 0x65, 0x63, 0x52, + 0x08, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x12, 0x32, 0x0a, 0x07, 0x6d, 0x61, 0x78, + 0x5f, 0x61, 0x67, 0x65, 0x18, 0x05, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x67, 0x6f, 0x6f, + 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x44, 0x75, 0x72, + 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x06, 0x6d, 0x61, 0x78, 0x41, 0x67, 0x65, 0x12, 0x2a, 0x0a, + 0x06, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x06, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x12, 0x2e, + 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x6f, 0x75, 0x72, 0x63, + 0x65, 0x52, 0x06, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x22, 0x9b, 0x0a, 0x0a, 0x0a, 0x45, 0x6e, + 0x74, 0x69, 0x74, 0x79, 0x53, 0x70, 0x65, 0x63, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, + 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x3a, 0x0a, 0x0a, + 0x76, 0x61, 0x6c, 0x75, 0x65, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, + 0x32, 0x1b, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x56, + 0x61, 0x6c, 0x75, 0x65, 0x54, 0x79, 0x70, 0x65, 0x2e, 0x45, 0x6e, 0x75, 0x6d, 0x52, 0x09, 0x76, + 0x61, 0x6c, 0x75, 0x65, 0x54, 0x79, 0x70, 0x65, 0x12, 0x45, 0x0a, 0x08, 0x70, 0x72, 0x65, 0x73, + 0x65, 0x6e, 0x63, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x27, 0x2e, 0x74, 0x65, 0x6e, + 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, + 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x50, 0x72, 0x65, 0x73, 0x65, + 0x6e, 0x63, 0x65, 0x48, 0x00, 0x52, 0x08, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x12, + 0x5b, 0x0a, 0x0e, 0x67, 0x72, 0x6f, 0x75, 0x70, 0x5f, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, + 0x65, 0x18, 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x32, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, + 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, + 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x50, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, + 0x57, 0x69, 0x74, 0x68, 0x69, 0x6e, 0x47, 0x72, 0x6f, 0x75, 0x70, 0x48, 0x00, 0x52, 0x0d, 0x67, + 0x72, 0x6f, 0x75, 0x70, 0x50, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x12, 0x3a, 0x0a, 0x05, + 0x73, 0x68, 0x61, 0x70, 0x65, 0x18, 0x05, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, 0x65, + 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, + 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x69, 0x78, 0x65, 0x64, 0x53, 0x68, 0x61, 0x70, 0x65, 0x48, + 0x01, 0x52, 0x05, 0x73, 0x68, 0x61, 0x70, 0x65, 0x12, 0x45, 0x0a, 0x0b, 0x76, 0x61, 0x6c, 0x75, + 0x65, 0x5f, 0x63, 0x6f, 0x75, 0x6e, 0x74, 0x18, 0x06, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, + 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, + 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x43, 0x6f, 0x75, 0x6e, + 0x74, 0x48, 0x01, 0x52, 0x0a, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x12, + 0x18, 0x0a, 0x06, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x07, 0x20, 0x01, 0x28, 0x09, 0x48, + 0x02, 0x52, 0x06, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x42, 0x0a, 0x0a, 0x69, 0x6e, 0x74, + 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x08, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x21, 0x2e, + 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, + 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x49, 0x6e, 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, + 0x48, 0x02, 0x52, 0x09, 0x69, 0x6e, 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x48, 0x0a, + 0x0c, 0x66, 0x6c, 0x6f, 0x61, 0x74, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x09, 0x20, + 0x01, 0x28, 0x0b, 0x32, 0x23, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, + 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x6c, 0x6f, + 0x61, 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0b, 0x66, 0x6c, 0x6f, 0x61, + 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x4b, 0x0a, 0x0d, 0x73, 0x74, 0x72, 0x69, 0x6e, + 0x67, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x0a, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x24, + 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, + 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x53, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x44, 0x6f, + 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0c, 0x73, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x44, 0x6f, + 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x45, 0x0a, 0x0b, 0x62, 0x6f, 0x6f, 0x6c, 0x5f, 0x64, 0x6f, 0x6d, + 0x61, 0x69, 0x6e, 0x18, 0x0b, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, 0x65, 0x6e, 0x73, + 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, + 0x76, 0x30, 0x2e, 0x42, 0x6f, 0x6f, 0x6c, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, + 0x0a, 0x62, 0x6f, 0x6f, 0x6c, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x4b, 0x0a, 0x0d, 0x73, + 0x74, 0x72, 0x75, 0x63, 0x74, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x0c, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x24, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, + 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x53, 0x74, 0x72, 0x75, + 0x63, 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0c, 0x73, 0x74, 0x72, 0x75, + 0x63, 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x67, 0x0a, 0x17, 0x6e, 0x61, 0x74, 0x75, + 0x72, 0x61, 0x6c, 0x5f, 0x6c, 0x61, 0x6e, 0x67, 0x75, 0x61, 0x67, 0x65, 0x5f, 0x64, 0x6f, 0x6d, + 0x61, 0x69, 0x6e, 0x18, 0x0d, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2d, 0x2e, 0x74, 0x65, 0x6e, 0x73, + 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, + 0x76, 0x30, 0x2e, 0x4e, 0x61, 0x74, 0x75, 0x72, 0x61, 0x6c, 0x4c, 0x61, 0x6e, 0x67, 0x75, 0x61, + 0x67, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x15, 0x6e, 0x61, 0x74, 0x75, + 0x72, 0x61, 0x6c, 0x4c, 0x61, 0x6e, 0x67, 0x75, 0x61, 0x67, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, + 0x6e, 0x12, 0x48, 0x0a, 0x0c, 0x69, 0x6d, 0x61, 0x67, 0x65, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, + 0x6e, 0x18, 0x0e, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x23, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, + 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, + 0x2e, 0x49, 0x6d, 0x61, 0x67, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0b, + 0x69, 0x6d, 0x61, 0x67, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x42, 0x0a, 0x0a, 0x6d, + 0x69, 0x64, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x0f, 0x20, 0x01, 0x28, 0x0b, 0x32, + 0x21, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, + 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x4d, 0x49, 0x44, 0x44, 0x6f, 0x6d, 0x61, + 0x69, 0x6e, 0x48, 0x02, 0x52, 0x09, 0x6d, 0x69, 0x64, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, + 0x42, 0x0a, 0x0a, 0x75, 0x72, 0x6c, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x10, 0x20, + 0x01, 0x28, 0x0b, 0x32, 0x21, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, + 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x55, 0x52, 0x4c, + 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x09, 0x75, 0x72, 0x6c, 0x44, 0x6f, 0x6d, + 0x61, 0x69, 0x6e, 0x12, 0x45, 0x0a, 0x0b, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x64, 0x6f, 0x6d, 0x61, + 0x69, 0x6e, 0x18, 0x11, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, + 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, + 0x30, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0a, + 0x74, 0x69, 0x6d, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x56, 0x0a, 0x12, 0x74, 0x69, + 0x6d, 0x65, 0x5f, 0x6f, 0x66, 0x5f, 0x64, 0x61, 0x79, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, + 0x18, 0x12, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x27, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, + 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, + 0x54, 0x69, 0x6d, 0x65, 0x4f, 0x66, 0x44, 0x61, 0x79, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, + 0x02, 0x52, 0x0f, 0x74, 0x69, 0x6d, 0x65, 0x4f, 0x66, 0x44, 0x61, 0x79, 0x44, 0x6f, 0x6d, 0x61, + 0x69, 0x6e, 0x42, 0x16, 0x0a, 0x14, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x5f, 0x63, + 0x6f, 0x6e, 0x73, 0x74, 0x72, 0x61, 0x69, 0x6e, 0x74, 0x73, 0x42, 0x0c, 0x0a, 0x0a, 0x73, 0x68, + 0x61, 0x70, 0x65, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x42, 0x0d, 0x0a, 0x0b, 0x64, 0x6f, 0x6d, 0x61, + 0x69, 0x6e, 0x5f, 0x69, 0x6e, 0x66, 0x6f, 0x22, 0x9c, 0x0a, 0x0a, 0x0b, 0x46, 0x65, 0x61, 0x74, + 0x75, 0x72, 0x65, 0x53, 0x70, 0x65, 0x63, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, + 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x3a, 0x0a, 0x0a, 0x76, + 0x61, 0x6c, 0x75, 0x65, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, + 0x1b, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x56, 0x61, + 0x6c, 0x75, 0x65, 0x54, 0x79, 0x70, 0x65, 0x2e, 0x45, 0x6e, 0x75, 0x6d, 0x52, 0x09, 0x76, 0x61, + 0x6c, 0x75, 0x65, 0x54, 0x79, 0x70, 0x65, 0x12, 0x45, 0x0a, 0x08, 0x70, 0x72, 0x65, 0x73, 0x65, + 0x6e, 0x63, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x27, 0x2e, 0x74, 0x65, 0x6e, 0x73, + 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, + 0x76, 0x30, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x50, 0x72, 0x65, 0x73, 0x65, 0x6e, + 0x63, 0x65, 0x48, 0x00, 0x52, 0x08, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x12, 0x5b, + 0x0a, 0x0e, 0x67, 0x72, 0x6f, 0x75, 0x70, 0x5f, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, + 0x18, 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x32, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, + 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, + 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x50, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x57, + 0x69, 0x74, 0x68, 0x69, 0x6e, 0x47, 0x72, 0x6f, 0x75, 0x70, 0x48, 0x00, 0x52, 0x0d, 0x67, 0x72, + 0x6f, 0x75, 0x70, 0x50, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x12, 0x3a, 0x0a, 0x05, 0x73, + 0x68, 0x61, 0x70, 0x65, 0x18, 0x05, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, 0x65, 0x6e, + 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, + 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x69, 0x78, 0x65, 0x64, 0x53, 0x68, 0x61, 0x70, 0x65, 0x48, 0x01, + 0x52, 0x05, 0x73, 0x68, 0x61, 0x70, 0x65, 0x12, 0x45, 0x0a, 0x0b, 0x76, 0x61, 0x6c, 0x75, 0x65, + 0x5f, 0x63, 0x6f, 0x75, 0x6e, 0x74, 0x18, 0x06, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, + 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, + 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, + 0x48, 0x01, 0x52, 0x0a, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x12, 0x18, + 0x0a, 0x06, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x07, 0x20, 0x01, 0x28, 0x09, 0x48, 0x02, + 0x52, 0x06, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x42, 0x0a, 0x0a, 0x69, 0x6e, 0x74, 0x5f, + 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x08, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x21, 0x2e, 0x74, + 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, + 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x49, 0x6e, 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, + 0x02, 0x52, 0x09, 0x69, 0x6e, 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x48, 0x0a, 0x0c, + 0x66, 0x6c, 0x6f, 0x61, 0x74, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x09, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x23, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, + 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x6c, 0x6f, 0x61, + 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0b, 0x66, 0x6c, 0x6f, 0x61, 0x74, + 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x4b, 0x0a, 0x0d, 0x73, 0x74, 0x72, 0x69, 0x6e, 0x67, + 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x0a, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x24, 0x2e, + 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, + 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x53, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x44, 0x6f, 0x6d, + 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0c, 0x73, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x44, 0x6f, 0x6d, + 0x61, 0x69, 0x6e, 0x12, 0x45, 0x0a, 0x0b, 0x62, 0x6f, 0x6f, 0x6c, 0x5f, 0x64, 0x6f, 0x6d, 0x61, + 0x69, 0x6e, 0x18, 0x0b, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, + 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, + 0x30, 0x2e, 0x42, 0x6f, 0x6f, 0x6c, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0a, + 0x62, 0x6f, 0x6f, 0x6c, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x4b, 0x0a, 0x0d, 0x73, 0x74, + 0x72, 0x75, 0x63, 0x74, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x0c, 0x20, 0x01, 0x28, + 0x0b, 0x32, 0x24, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, + 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x53, 0x74, 0x72, 0x75, 0x63, + 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0c, 0x73, 0x74, 0x72, 0x75, 0x63, + 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x67, 0x0a, 0x17, 0x6e, 0x61, 0x74, 0x75, 0x72, + 0x61, 0x6c, 0x5f, 0x6c, 0x61, 0x6e, 0x67, 0x75, 0x61, 0x67, 0x65, 0x5f, 0x64, 0x6f, 0x6d, 0x61, + 0x69, 0x6e, 0x18, 0x0d, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2d, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, + 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, + 0x30, 0x2e, 0x4e, 0x61, 0x74, 0x75, 0x72, 0x61, 0x6c, 0x4c, 0x61, 0x6e, 0x67, 0x75, 0x61, 0x67, + 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x15, 0x6e, 0x61, 0x74, 0x75, 0x72, + 0x61, 0x6c, 0x4c, 0x61, 0x6e, 0x67, 0x75, 0x61, 0x67, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, + 0x12, 0x48, 0x0a, 0x0c, 0x69, 0x6d, 0x61, 0x67, 0x65, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, + 0x18, 0x0e, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x23, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, + 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, + 0x49, 0x6d, 0x61, 0x67, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0b, 0x69, + 0x6d, 0x61, 0x67, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x42, 0x0a, 0x0a, 0x6d, 0x69, + 0x64, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x0f, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x21, + 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, + 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x4d, 0x49, 0x44, 0x44, 0x6f, 0x6d, 0x61, 0x69, + 0x6e, 0x48, 0x02, 0x52, 0x09, 0x6d, 0x69, 0x64, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x42, + 0x0a, 0x0a, 0x75, 0x72, 0x6c, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x10, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x21, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, + 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x55, 0x52, 0x4c, 0x44, + 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x09, 0x75, 0x72, 0x6c, 0x44, 0x6f, 0x6d, 0x61, + 0x69, 0x6e, 0x12, 0x45, 0x0a, 0x0b, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, + 0x6e, 0x18, 0x11, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, + 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, + 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0a, 0x74, + 0x69, 0x6d, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x56, 0x0a, 0x12, 0x74, 0x69, 0x6d, + 0x65, 0x5f, 0x6f, 0x66, 0x5f, 0x64, 0x61, 0x79, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, + 0x12, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x27, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, + 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x54, + 0x69, 0x6d, 0x65, 0x4f, 0x66, 0x44, 0x61, 0x79, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, + 0x52, 0x0f, 0x74, 0x69, 0x6d, 0x65, 0x4f, 0x66, 0x44, 0x61, 0x79, 0x44, 0x6f, 0x6d, 0x61, 0x69, + 0x6e, 0x42, 0x16, 0x0a, 0x14, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x5f, 0x63, 0x6f, + 0x6e, 0x73, 0x74, 0x72, 0x61, 0x69, 0x6e, 0x74, 0x73, 0x42, 0x0c, 0x0a, 0x0a, 0x73, 0x68, 0x61, + 0x70, 0x65, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x42, 0x0d, 0x0a, 0x0b, 0x64, 0x6f, 0x6d, 0x61, 0x69, + 0x6e, 0x5f, 0x69, 0x6e, 0x66, 0x6f, 0x22, 0x8f, 0x01, 0x0a, 0x0e, 0x46, 0x65, 0x61, 0x74, 0x75, + 0x72, 0x65, 0x53, 0x65, 0x74, 0x4d, 0x65, 0x74, 0x61, 0x12, 0x47, 0x0a, 0x11, 0x63, 0x72, 0x65, + 0x61, 0x74, 0x65, 0x64, 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x18, 0x01, + 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, + 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, + 0x52, 0x10, 0x63, 0x72, 0x65, 0x61, 0x74, 0x65, 0x64, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, + 0x6d, 0x70, 0x12, 0x34, 0x0a, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x18, 0x02, 0x20, 0x01, + 0x28, 0x0e, 0x32, 0x1c, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, + 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, + 0x52, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x2a, 0x4c, 0x0a, 0x10, 0x46, 0x65, 0x61, 0x74, + 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x12, 0x0a, 0x0e, + 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, 0x5f, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, + 0x12, 0x12, 0x0a, 0x0e, 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, 0x5f, 0x50, 0x45, 0x4e, 0x44, 0x49, + 0x4e, 0x47, 0x10, 0x01, 0x12, 0x10, 0x0a, 0x0c, 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, 0x5f, 0x52, + 0x45, 0x41, 0x44, 0x59, 0x10, 0x02, 0x42, 0x4e, 0x0a, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x63, 0x6f, 0x72, 0x65, 0x42, 0x0f, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, + 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x2f, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, + 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x64, + 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, 0x61, 0x73, + 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, +} + +var ( + file_feast_core_FeatureSet_proto_rawDescOnce sync.Once + file_feast_core_FeatureSet_proto_rawDescData = file_feast_core_FeatureSet_proto_rawDesc +) + +func file_feast_core_FeatureSet_proto_rawDescGZIP() []byte { + file_feast_core_FeatureSet_proto_rawDescOnce.Do(func() { + file_feast_core_FeatureSet_proto_rawDescData = protoimpl.X.CompressGZIP(file_feast_core_FeatureSet_proto_rawDescData) + }) + return file_feast_core_FeatureSet_proto_rawDescData +} + +var file_feast_core_FeatureSet_proto_enumTypes = make([]protoimpl.EnumInfo, 1) +var file_feast_core_FeatureSet_proto_msgTypes = make([]protoimpl.MessageInfo, 5) +var file_feast_core_FeatureSet_proto_goTypes = []interface{}{ + (FeatureSetStatus)(0), // 0: feast.core.FeatureSetStatus + (*FeatureSet)(nil), // 1: feast.core.FeatureSet + (*FeatureSetSpec)(nil), // 2: feast.core.FeatureSetSpec + (*EntitySpec)(nil), // 3: feast.core.EntitySpec + (*FeatureSpec)(nil), // 4: feast.core.FeatureSpec + (*FeatureSetMeta)(nil), // 5: feast.core.FeatureSetMeta + (*duration.Duration)(nil), // 6: google.protobuf.Duration + (*Source)(nil), // 7: feast.core.Source + (types.ValueType_Enum)(0), // 8: feast.types.ValueType.Enum + (*v0.FeaturePresence)(nil), // 9: tensorflow.metadata.v0.FeaturePresence + (*v0.FeaturePresenceWithinGroup)(nil), // 10: tensorflow.metadata.v0.FeaturePresenceWithinGroup + (*v0.FixedShape)(nil), // 11: tensorflow.metadata.v0.FixedShape + (*v0.ValueCount)(nil), // 12: tensorflow.metadata.v0.ValueCount + (*v0.IntDomain)(nil), // 13: tensorflow.metadata.v0.IntDomain + (*v0.FloatDomain)(nil), // 14: tensorflow.metadata.v0.FloatDomain + (*v0.StringDomain)(nil), // 15: tensorflow.metadata.v0.StringDomain + (*v0.BoolDomain)(nil), // 16: tensorflow.metadata.v0.BoolDomain + (*v0.StructDomain)(nil), // 17: tensorflow.metadata.v0.StructDomain + (*v0.NaturalLanguageDomain)(nil), // 18: tensorflow.metadata.v0.NaturalLanguageDomain + (*v0.ImageDomain)(nil), // 19: tensorflow.metadata.v0.ImageDomain + (*v0.MIDDomain)(nil), // 20: tensorflow.metadata.v0.MIDDomain + (*v0.URLDomain)(nil), // 21: tensorflow.metadata.v0.URLDomain + (*v0.TimeDomain)(nil), // 22: tensorflow.metadata.v0.TimeDomain + (*v0.TimeOfDayDomain)(nil), // 23: tensorflow.metadata.v0.TimeOfDayDomain + (*timestamp.Timestamp)(nil), // 24: google.protobuf.Timestamp +} +var file_feast_core_FeatureSet_proto_depIdxs = []int32{ + 2, // 0: feast.core.FeatureSet.spec:type_name -> feast.core.FeatureSetSpec + 5, // 1: feast.core.FeatureSet.meta:type_name -> feast.core.FeatureSetMeta + 3, // 2: feast.core.FeatureSetSpec.entities:type_name -> feast.core.EntitySpec + 4, // 3: feast.core.FeatureSetSpec.features:type_name -> feast.core.FeatureSpec + 6, // 4: feast.core.FeatureSetSpec.max_age:type_name -> google.protobuf.Duration + 7, // 5: feast.core.FeatureSetSpec.source:type_name -> feast.core.Source + 8, // 6: feast.core.EntitySpec.value_type:type_name -> feast.types.ValueType.Enum + 9, // 7: feast.core.EntitySpec.presence:type_name -> tensorflow.metadata.v0.FeaturePresence + 10, // 8: feast.core.EntitySpec.group_presence:type_name -> tensorflow.metadata.v0.FeaturePresenceWithinGroup + 11, // 9: feast.core.EntitySpec.shape:type_name -> tensorflow.metadata.v0.FixedShape + 12, // 10: feast.core.EntitySpec.value_count:type_name -> tensorflow.metadata.v0.ValueCount + 13, // 11: feast.core.EntitySpec.int_domain:type_name -> tensorflow.metadata.v0.IntDomain + 14, // 12: feast.core.EntitySpec.float_domain:type_name -> tensorflow.metadata.v0.FloatDomain + 15, // 13: feast.core.EntitySpec.string_domain:type_name -> tensorflow.metadata.v0.StringDomain + 16, // 14: feast.core.EntitySpec.bool_domain:type_name -> tensorflow.metadata.v0.BoolDomain + 17, // 15: feast.core.EntitySpec.struct_domain:type_name -> tensorflow.metadata.v0.StructDomain + 18, // 16: feast.core.EntitySpec.natural_language_domain:type_name -> tensorflow.metadata.v0.NaturalLanguageDomain + 19, // 17: feast.core.EntitySpec.image_domain:type_name -> tensorflow.metadata.v0.ImageDomain + 20, // 18: feast.core.EntitySpec.mid_domain:type_name -> tensorflow.metadata.v0.MIDDomain + 21, // 19: feast.core.EntitySpec.url_domain:type_name -> tensorflow.metadata.v0.URLDomain + 22, // 20: feast.core.EntitySpec.time_domain:type_name -> tensorflow.metadata.v0.TimeDomain + 23, // 21: feast.core.EntitySpec.time_of_day_domain:type_name -> tensorflow.metadata.v0.TimeOfDayDomain + 8, // 22: feast.core.FeatureSpec.value_type:type_name -> feast.types.ValueType.Enum + 9, // 23: feast.core.FeatureSpec.presence:type_name -> tensorflow.metadata.v0.FeaturePresence + 10, // 24: feast.core.FeatureSpec.group_presence:type_name -> tensorflow.metadata.v0.FeaturePresenceWithinGroup + 11, // 25: feast.core.FeatureSpec.shape:type_name -> tensorflow.metadata.v0.FixedShape + 12, // 26: feast.core.FeatureSpec.value_count:type_name -> tensorflow.metadata.v0.ValueCount + 13, // 27: feast.core.FeatureSpec.int_domain:type_name -> tensorflow.metadata.v0.IntDomain + 14, // 28: feast.core.FeatureSpec.float_domain:type_name -> tensorflow.metadata.v0.FloatDomain + 15, // 29: feast.core.FeatureSpec.string_domain:type_name -> tensorflow.metadata.v0.StringDomain + 16, // 30: feast.core.FeatureSpec.bool_domain:type_name -> tensorflow.metadata.v0.BoolDomain + 17, // 31: feast.core.FeatureSpec.struct_domain:type_name -> tensorflow.metadata.v0.StructDomain + 18, // 32: feast.core.FeatureSpec.natural_language_domain:type_name -> tensorflow.metadata.v0.NaturalLanguageDomain + 19, // 33: feast.core.FeatureSpec.image_domain:type_name -> tensorflow.metadata.v0.ImageDomain + 20, // 34: feast.core.FeatureSpec.mid_domain:type_name -> tensorflow.metadata.v0.MIDDomain + 21, // 35: feast.core.FeatureSpec.url_domain:type_name -> tensorflow.metadata.v0.URLDomain + 22, // 36: feast.core.FeatureSpec.time_domain:type_name -> tensorflow.metadata.v0.TimeDomain + 23, // 37: feast.core.FeatureSpec.time_of_day_domain:type_name -> tensorflow.metadata.v0.TimeOfDayDomain + 24, // 38: feast.core.FeatureSetMeta.created_timestamp:type_name -> google.protobuf.Timestamp + 0, // 39: feast.core.FeatureSetMeta.status:type_name -> feast.core.FeatureSetStatus + 40, // [40:40] is the sub-list for method output_type + 40, // [40:40] is the sub-list for method input_type + 40, // [40:40] is the sub-list for extension type_name + 40, // [40:40] is the sub-list for extension extendee + 0, // [0:40] is the sub-list for field type_name +} + +func init() { file_feast_core_FeatureSet_proto_init() } +func file_feast_core_FeatureSet_proto_init() { + if File_feast_core_FeatureSet_proto != nil { + return + } + file_feast_core_Source_proto_init() + if !protoimpl.UnsafeEnabled { + file_feast_core_FeatureSet_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*FeatureSet); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_FeatureSet_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*FeatureSetSpec); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_FeatureSet_proto_msgTypes[2].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*EntitySpec); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_FeatureSet_proto_msgTypes[3].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*FeatureSpec); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_FeatureSet_proto_msgTypes[4].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*FeatureSetMeta); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + } + file_feast_core_FeatureSet_proto_msgTypes[2].OneofWrappers = []interface{}{ + (*EntitySpec_Presence)(nil), + (*EntitySpec_GroupPresence)(nil), + (*EntitySpec_Shape)(nil), + (*EntitySpec_ValueCount)(nil), + (*EntitySpec_Domain)(nil), + (*EntitySpec_IntDomain)(nil), + (*EntitySpec_FloatDomain)(nil), + (*EntitySpec_StringDomain)(nil), + (*EntitySpec_BoolDomain)(nil), + (*EntitySpec_StructDomain)(nil), + (*EntitySpec_NaturalLanguageDomain)(nil), + (*EntitySpec_ImageDomain)(nil), + (*EntitySpec_MidDomain)(nil), + (*EntitySpec_UrlDomain)(nil), + (*EntitySpec_TimeDomain)(nil), + (*EntitySpec_TimeOfDayDomain)(nil), + } + file_feast_core_FeatureSet_proto_msgTypes[3].OneofWrappers = []interface{}{ + (*FeatureSpec_Presence)(nil), + (*FeatureSpec_GroupPresence)(nil), + (*FeatureSpec_Shape)(nil), + (*FeatureSpec_ValueCount)(nil), + (*FeatureSpec_Domain)(nil), + (*FeatureSpec_IntDomain)(nil), + (*FeatureSpec_FloatDomain)(nil), + (*FeatureSpec_StringDomain)(nil), + (*FeatureSpec_BoolDomain)(nil), + (*FeatureSpec_StructDomain)(nil), + (*FeatureSpec_NaturalLanguageDomain)(nil), + (*FeatureSpec_ImageDomain)(nil), + (*FeatureSpec_MidDomain)(nil), + (*FeatureSpec_UrlDomain)(nil), + (*FeatureSpec_TimeDomain)(nil), + (*FeatureSpec_TimeOfDayDomain)(nil), + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_feast_core_FeatureSet_proto_rawDesc, + NumEnums: 1, + NumMessages: 5, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_feast_core_FeatureSet_proto_goTypes, + DependencyIndexes: file_feast_core_FeatureSet_proto_depIdxs, + EnumInfos: file_feast_core_FeatureSet_proto_enumTypes, + MessageInfos: file_feast_core_FeatureSet_proto_msgTypes, + }.Build() + File_feast_core_FeatureSet_proto = out.File + file_feast_core_FeatureSet_proto_rawDesc = nil + file_feast_core_FeatureSet_proto_goTypes = nil + file_feast_core_FeatureSet_proto_depIdxs = nil +} diff --git a/sdk/go/protos/feast/core/FeatureSetReference.pb.go b/sdk/go/protos/feast/core/FeatureSetReference.pb.go new file mode 100644 index 00000000000..52a63b6a8c1 --- /dev/null +++ b/sdk/go/protos/feast/core/FeatureSetReference.pb.go @@ -0,0 +1,193 @@ +// +// Copyright 2020 The Feast Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.21.0 +// protoc v3.10.0 +// source: feast/core/FeatureSetReference.proto + +package core + +import ( + proto "github.com/golang/protobuf/proto" + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +// This is a compile-time assertion that a sufficiently up-to-date version +// of the legacy proto package is being used. +const _ = proto.ProtoPackageIsVersion4 + +// Defines a composite key that refers to a unique FeatureSet +type FeatureSetReference struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Name of the project + Project string `protobuf:"bytes,1,opt,name=project,proto3" json:"project,omitempty"` + // Name of the FeatureSet + Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"` + // Version no. of the FeatureSet + Version int32 `protobuf:"varint,3,opt,name=version,proto3" json:"version,omitempty"` +} + +func (x *FeatureSetReference) Reset() { + *x = FeatureSetReference{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_FeatureSetReference_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *FeatureSetReference) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*FeatureSetReference) ProtoMessage() {} + +func (x *FeatureSetReference) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_FeatureSetReference_proto_msgTypes[0] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use FeatureSetReference.ProtoReflect.Descriptor instead. +func (*FeatureSetReference) Descriptor() ([]byte, []int) { + return file_feast_core_FeatureSetReference_proto_rawDescGZIP(), []int{0} +} + +func (x *FeatureSetReference) GetProject() string { + if x != nil { + return x.Project + } + return "" +} + +func (x *FeatureSetReference) GetName() string { + if x != nil { + return x.Name + } + return "" +} + +func (x *FeatureSetReference) GetVersion() int32 { + if x != nil { + return x.Version + } + return 0 +} + +var File_feast_core_FeatureSetReference_proto protoreflect.FileDescriptor + +var file_feast_core_FeatureSetReference_proto_rawDesc = []byte{ + 0x0a, 0x24, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x46, 0x65, 0x61, + 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, + 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, + 0x72, 0x65, 0x22, 0x5d, 0x0a, 0x13, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, + 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x70, 0x72, 0x6f, + 0x6a, 0x65, 0x63, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x70, 0x72, 0x6f, 0x6a, + 0x65, 0x63, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, + 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, + 0x6f, 0x6e, 0x18, 0x03, 0x20, 0x01, 0x28, 0x05, 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, + 0x6e, 0x42, 0x57, 0x0a, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x42, + 0x18, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x66, 0x65, 0x72, + 0x65, 0x6e, 0x63, 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x2f, 0x67, 0x69, 0x74, 0x68, 0x75, + 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, + 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, + 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, + 0x6f, 0x33, +} + +var ( + file_feast_core_FeatureSetReference_proto_rawDescOnce sync.Once + file_feast_core_FeatureSetReference_proto_rawDescData = file_feast_core_FeatureSetReference_proto_rawDesc +) + +func file_feast_core_FeatureSetReference_proto_rawDescGZIP() []byte { + file_feast_core_FeatureSetReference_proto_rawDescOnce.Do(func() { + file_feast_core_FeatureSetReference_proto_rawDescData = protoimpl.X.CompressGZIP(file_feast_core_FeatureSetReference_proto_rawDescData) + }) + return file_feast_core_FeatureSetReference_proto_rawDescData +} + +var file_feast_core_FeatureSetReference_proto_msgTypes = make([]protoimpl.MessageInfo, 1) +var file_feast_core_FeatureSetReference_proto_goTypes = []interface{}{ + (*FeatureSetReference)(nil), // 0: feast.core.FeatureSetReference +} +var file_feast_core_FeatureSetReference_proto_depIdxs = []int32{ + 0, // [0:0] is the sub-list for method output_type + 0, // [0:0] is the sub-list for method input_type + 0, // [0:0] is the sub-list for extension type_name + 0, // [0:0] is the sub-list for extension extendee + 0, // [0:0] is the sub-list for field type_name +} + +func init() { file_feast_core_FeatureSetReference_proto_init() } +func file_feast_core_FeatureSetReference_proto_init() { + if File_feast_core_FeatureSetReference_proto != nil { + return + } + if !protoimpl.UnsafeEnabled { + file_feast_core_FeatureSetReference_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*FeatureSetReference); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_feast_core_FeatureSetReference_proto_rawDesc, + NumEnums: 0, + NumMessages: 1, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_feast_core_FeatureSetReference_proto_goTypes, + DependencyIndexes: file_feast_core_FeatureSetReference_proto_depIdxs, + MessageInfos: file_feast_core_FeatureSetReference_proto_msgTypes, + }.Build() + File_feast_core_FeatureSetReference_proto = out.File + file_feast_core_FeatureSetReference_proto_rawDesc = nil + file_feast_core_FeatureSetReference_proto_goTypes = nil + file_feast_core_FeatureSetReference_proto_depIdxs = nil +} diff --git a/sdk/go/protos/feast/core/IngestionJob.pb.go b/sdk/go/protos/feast/core/IngestionJob.pb.go new file mode 100644 index 00000000000..3623d95d885 --- /dev/null +++ b/sdk/go/protos/feast/core/IngestionJob.pb.go @@ -0,0 +1,333 @@ +// +// Copyright 2020 The Feast Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// + +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.21.0 +// protoc v3.10.0 +// source: feast/core/IngestionJob.proto + +package core + +import ( + proto "github.com/golang/protobuf/proto" + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +// This is a compile-time assertion that a sufficiently up-to-date version +// of the legacy proto package is being used. +const _ = proto.ProtoPackageIsVersion4 + +// Status of a Feast Ingestion Job +type IngestionJobStatus int32 + +const ( + // Job status is not known. + IngestionJobStatus_UNKNOWN IngestionJobStatus = 0 + // Import job is submitted to runner and currently pending for executing + IngestionJobStatus_PENDING IngestionJobStatus = 1 + // Import job is currently running in the runner + IngestionJobStatus_RUNNING IngestionJobStatus = 2 + // Runner's reported the import job has completed (applicable to batch job) + IngestionJobStatus_COMPLETED IngestionJobStatus = 3 + // When user sent abort command, but it's still running + IngestionJobStatus_ABORTING IngestionJobStatus = 4 + // User initiated abort job + IngestionJobStatus_ABORTED IngestionJobStatus = 5 + // Runner's reported that the import job failed to run or there is a failure during job + IngestionJobStatus_ERROR IngestionJobStatus = 6 + // job has been suspended and waiting for cleanup + IngestionJobStatus_SUSPENDING IngestionJobStatus = 7 + // job has been suspended + IngestionJobStatus_SUSPENDED IngestionJobStatus = 8 +) + +// Enum value maps for IngestionJobStatus. +var ( + IngestionJobStatus_name = map[int32]string{ + 0: "UNKNOWN", + 1: "PENDING", + 2: "RUNNING", + 3: "COMPLETED", + 4: "ABORTING", + 5: "ABORTED", + 6: "ERROR", + 7: "SUSPENDING", + 8: "SUSPENDED", + } + IngestionJobStatus_value = map[string]int32{ + "UNKNOWN": 0, + "PENDING": 1, + "RUNNING": 2, + "COMPLETED": 3, + "ABORTING": 4, + "ABORTED": 5, + "ERROR": 6, + "SUSPENDING": 7, + "SUSPENDED": 8, + } +) + +func (x IngestionJobStatus) Enum() *IngestionJobStatus { + p := new(IngestionJobStatus) + *p = x + return p +} + +func (x IngestionJobStatus) String() string { + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) +} + +func (IngestionJobStatus) Descriptor() protoreflect.EnumDescriptor { + return file_feast_core_IngestionJob_proto_enumTypes[0].Descriptor() +} + +func (IngestionJobStatus) Type() protoreflect.EnumType { + return &file_feast_core_IngestionJob_proto_enumTypes[0] +} + +func (x IngestionJobStatus) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) +} + +// Deprecated: Use IngestionJobStatus.Descriptor instead. +func (IngestionJobStatus) EnumDescriptor() ([]byte, []int) { + return file_feast_core_IngestionJob_proto_rawDescGZIP(), []int{0} +} + +// Represents Feast Injestion Job +type IngestionJob struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Job ID assigned by Feast + Id string `protobuf:"bytes,1,opt,name=id,proto3" json:"id,omitempty"` + // External job ID specific to the runner. + // For DirectRunner jobs, this is identical to id. For DataflowRunner jobs, this refers to the Dataflow job ID. + ExternalId string `protobuf:"bytes,2,opt,name=external_id,json=externalId,proto3" json:"external_id,omitempty"` + Status IngestionJobStatus `protobuf:"varint,3,opt,name=status,proto3,enum=feast.core.IngestionJobStatus" json:"status,omitempty"` + // List of feature sets whose features are populated by this job. + FeatureSets []*FeatureSet `protobuf:"bytes,4,rep,name=feature_sets,json=featureSets,proto3" json:"feature_sets,omitempty"` + // Source this job is reading from. + Source *Source `protobuf:"bytes,5,opt,name=source,proto3" json:"source,omitempty"` + // Store this job is writing to. + Store *Store `protobuf:"bytes,6,opt,name=store,proto3" json:"store,omitempty"` +} + +func (x *IngestionJob) Reset() { + *x = IngestionJob{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_IngestionJob_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *IngestionJob) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*IngestionJob) ProtoMessage() {} + +func (x *IngestionJob) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_IngestionJob_proto_msgTypes[0] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use IngestionJob.ProtoReflect.Descriptor instead. +func (*IngestionJob) Descriptor() ([]byte, []int) { + return file_feast_core_IngestionJob_proto_rawDescGZIP(), []int{0} +} + +func (x *IngestionJob) GetId() string { + if x != nil { + return x.Id + } + return "" +} + +func (x *IngestionJob) GetExternalId() string { + if x != nil { + return x.ExternalId + } + return "" +} + +func (x *IngestionJob) GetStatus() IngestionJobStatus { + if x != nil { + return x.Status + } + return IngestionJobStatus_UNKNOWN +} + +func (x *IngestionJob) GetFeatureSets() []*FeatureSet { + if x != nil { + return x.FeatureSets + } + return nil +} + +func (x *IngestionJob) GetSource() *Source { + if x != nil { + return x.Source + } + return nil +} + +func (x *IngestionJob) GetStore() *Store { + if x != nil { + return x.Store + } + return nil +} + +var File_feast_core_IngestionJob_proto protoreflect.FileDescriptor + +var file_feast_core_IngestionJob_proto_rawDesc = []byte{ + 0x0a, 0x1d, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x49, 0x6e, 0x67, + 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, + 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x1a, 0x1b, 0x66, 0x65, 0x61, + 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, + 0x65, 0x74, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x16, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, + 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, + 0x1a, 0x17, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x53, 0x6f, 0x75, + 0x72, 0x63, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x87, 0x02, 0x0a, 0x0c, 0x49, 0x6e, + 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, + 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x12, 0x1f, 0x0a, 0x0b, 0x65, 0x78, + 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x5f, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x0a, 0x65, 0x78, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x49, 0x64, 0x12, 0x36, 0x0a, 0x06, 0x73, + 0x74, 0x61, 0x74, 0x75, 0x73, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x1e, 0x2e, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, + 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x06, 0x73, 0x74, 0x61, + 0x74, 0x75, 0x73, 0x12, 0x39, 0x0a, 0x0c, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x5f, 0x73, + 0x65, 0x74, 0x73, 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, + 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, + 0x74, 0x52, 0x0b, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, 0x12, 0x2a, + 0x0a, 0x06, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x05, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x12, + 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x6f, 0x75, 0x72, + 0x63, 0x65, 0x52, 0x06, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x12, 0x27, 0x0a, 0x05, 0x73, 0x74, + 0x6f, 0x72, 0x65, 0x18, 0x06, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x11, 0x2e, 0x66, 0x65, 0x61, 0x73, + 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x05, 0x73, 0x74, + 0x6f, 0x72, 0x65, 0x2a, 0x8f, 0x01, 0x0a, 0x12, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, + 0x6e, 0x4a, 0x6f, 0x62, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x0b, 0x0a, 0x07, 0x55, 0x4e, + 0x4b, 0x4e, 0x4f, 0x57, 0x4e, 0x10, 0x00, 0x12, 0x0b, 0x0a, 0x07, 0x50, 0x45, 0x4e, 0x44, 0x49, + 0x4e, 0x47, 0x10, 0x01, 0x12, 0x0b, 0x0a, 0x07, 0x52, 0x55, 0x4e, 0x4e, 0x49, 0x4e, 0x47, 0x10, + 0x02, 0x12, 0x0d, 0x0a, 0x09, 0x43, 0x4f, 0x4d, 0x50, 0x4c, 0x45, 0x54, 0x45, 0x44, 0x10, 0x03, + 0x12, 0x0c, 0x0a, 0x08, 0x41, 0x42, 0x4f, 0x52, 0x54, 0x49, 0x4e, 0x47, 0x10, 0x04, 0x12, 0x0b, + 0x0a, 0x07, 0x41, 0x42, 0x4f, 0x52, 0x54, 0x45, 0x44, 0x10, 0x05, 0x12, 0x09, 0x0a, 0x05, 0x45, + 0x52, 0x52, 0x4f, 0x52, 0x10, 0x06, 0x12, 0x0e, 0x0a, 0x0a, 0x53, 0x55, 0x53, 0x50, 0x45, 0x4e, + 0x44, 0x49, 0x4e, 0x47, 0x10, 0x07, 0x12, 0x0d, 0x0a, 0x09, 0x53, 0x55, 0x53, 0x50, 0x45, 0x4e, + 0x44, 0x45, 0x44, 0x10, 0x08, 0x42, 0x50, 0x0a, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, + 0x6f, 0x72, 0x65, 0x42, 0x11, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, + 0x62, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x2f, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, + 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, + 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, 0x61, + 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, +} + +var ( + file_feast_core_IngestionJob_proto_rawDescOnce sync.Once + file_feast_core_IngestionJob_proto_rawDescData = file_feast_core_IngestionJob_proto_rawDesc +) + +func file_feast_core_IngestionJob_proto_rawDescGZIP() []byte { + file_feast_core_IngestionJob_proto_rawDescOnce.Do(func() { + file_feast_core_IngestionJob_proto_rawDescData = protoimpl.X.CompressGZIP(file_feast_core_IngestionJob_proto_rawDescData) + }) + return file_feast_core_IngestionJob_proto_rawDescData +} + +var file_feast_core_IngestionJob_proto_enumTypes = make([]protoimpl.EnumInfo, 1) +var file_feast_core_IngestionJob_proto_msgTypes = make([]protoimpl.MessageInfo, 1) +var file_feast_core_IngestionJob_proto_goTypes = []interface{}{ + (IngestionJobStatus)(0), // 0: feast.core.IngestionJobStatus + (*IngestionJob)(nil), // 1: feast.core.IngestionJob + (*FeatureSet)(nil), // 2: feast.core.FeatureSet + (*Source)(nil), // 3: feast.core.Source + (*Store)(nil), // 4: feast.core.Store +} +var file_feast_core_IngestionJob_proto_depIdxs = []int32{ + 0, // 0: feast.core.IngestionJob.status:type_name -> feast.core.IngestionJobStatus + 2, // 1: feast.core.IngestionJob.feature_sets:type_name -> feast.core.FeatureSet + 3, // 2: feast.core.IngestionJob.source:type_name -> feast.core.Source + 4, // 3: feast.core.IngestionJob.store:type_name -> feast.core.Store + 4, // [4:4] is the sub-list for method output_type + 4, // [4:4] is the sub-list for method input_type + 4, // [4:4] is the sub-list for extension type_name + 4, // [4:4] is the sub-list for extension extendee + 0, // [0:4] is the sub-list for field type_name +} + +func init() { file_feast_core_IngestionJob_proto_init() } +func file_feast_core_IngestionJob_proto_init() { + if File_feast_core_IngestionJob_proto != nil { + return + } + file_feast_core_FeatureSet_proto_init() + file_feast_core_Store_proto_init() + file_feast_core_Source_proto_init() + if !protoimpl.UnsafeEnabled { + file_feast_core_IngestionJob_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*IngestionJob); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_feast_core_IngestionJob_proto_rawDesc, + NumEnums: 1, + NumMessages: 1, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_feast_core_IngestionJob_proto_goTypes, + DependencyIndexes: file_feast_core_IngestionJob_proto_depIdxs, + EnumInfos: file_feast_core_IngestionJob_proto_enumTypes, + MessageInfos: file_feast_core_IngestionJob_proto_msgTypes, + }.Build() + File_feast_core_IngestionJob_proto = out.File + file_feast_core_IngestionJob_proto_rawDesc = nil + file_feast_core_IngestionJob_proto_goTypes = nil + file_feast_core_IngestionJob_proto_depIdxs = nil +} diff --git a/sdk/go/protos/feast/core/Source.pb.go b/sdk/go/protos/feast/core/Source.pb.go new file mode 100644 index 00000000000..30bd2362723 --- /dev/null +++ b/sdk/go/protos/feast/core/Source.pb.go @@ -0,0 +1,336 @@ +// +// * Copyright 2019 The Feast Authors +// * +// * Licensed under the Apache License, Version 2.0 (the "License"); +// * you may not use this file except in compliance with the License. +// * You may obtain a copy of the License at +// * +// * https://www.apache.org/licenses/LICENSE-2.0 +// * +// * Unless required by applicable law or agreed to in writing, software +// * distributed under the License is distributed on an "AS IS" BASIS, +// * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// * See the License for the specific language governing permissions and +// * limitations under the License. +// + +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.21.0 +// protoc v3.10.0 +// source: feast/core/Source.proto + +package core + +import ( + proto "github.com/golang/protobuf/proto" + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +// This is a compile-time assertion that a sufficiently up-to-date version +// of the legacy proto package is being used. +const _ = proto.ProtoPackageIsVersion4 + +type SourceType int32 + +const ( + SourceType_INVALID SourceType = 0 + SourceType_KAFKA SourceType = 1 +) + +// Enum value maps for SourceType. +var ( + SourceType_name = map[int32]string{ + 0: "INVALID", + 1: "KAFKA", + } + SourceType_value = map[string]int32{ + "INVALID": 0, + "KAFKA": 1, + } +) + +func (x SourceType) Enum() *SourceType { + p := new(SourceType) + *p = x + return p +} + +func (x SourceType) String() string { + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) +} + +func (SourceType) Descriptor() protoreflect.EnumDescriptor { + return file_feast_core_Source_proto_enumTypes[0].Descriptor() +} + +func (SourceType) Type() protoreflect.EnumType { + return &file_feast_core_Source_proto_enumTypes[0] +} + +func (x SourceType) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) +} + +// Deprecated: Use SourceType.Descriptor instead. +func (SourceType) EnumDescriptor() ([]byte, []int) { + return file_feast_core_Source_proto_rawDescGZIP(), []int{0} +} + +type Source struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // The kind of data source Feast should connect to in order to retrieve FeatureRow value + Type SourceType `protobuf:"varint,1,opt,name=type,proto3,enum=feast.core.SourceType" json:"type,omitempty"` + // Source specific configuration + // + // Types that are assignable to SourceConfig: + // *Source_KafkaSourceConfig + SourceConfig isSource_SourceConfig `protobuf_oneof:"source_config"` +} + +func (x *Source) Reset() { + *x = Source{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_Source_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *Source) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*Source) ProtoMessage() {} + +func (x *Source) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_Source_proto_msgTypes[0] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use Source.ProtoReflect.Descriptor instead. +func (*Source) Descriptor() ([]byte, []int) { + return file_feast_core_Source_proto_rawDescGZIP(), []int{0} +} + +func (x *Source) GetType() SourceType { + if x != nil { + return x.Type + } + return SourceType_INVALID +} + +func (m *Source) GetSourceConfig() isSource_SourceConfig { + if m != nil { + return m.SourceConfig + } + return nil +} + +func (x *Source) GetKafkaSourceConfig() *KafkaSourceConfig { + if x, ok := x.GetSourceConfig().(*Source_KafkaSourceConfig); ok { + return x.KafkaSourceConfig + } + return nil +} + +type isSource_SourceConfig interface { + isSource_SourceConfig() +} + +type Source_KafkaSourceConfig struct { + KafkaSourceConfig *KafkaSourceConfig `protobuf:"bytes,2,opt,name=kafka_source_config,json=kafkaSourceConfig,proto3,oneof"` +} + +func (*Source_KafkaSourceConfig) isSource_SourceConfig() {} + +type KafkaSourceConfig struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // - bootstrapServers: [comma delimited value of host[:port]] + BootstrapServers string `protobuf:"bytes,1,opt,name=bootstrap_servers,json=bootstrapServers,proto3" json:"bootstrap_servers,omitempty"` + // - topics: [Kafka topic name. This value is provisioned by core and should not be set by the user.] + Topic string `protobuf:"bytes,2,opt,name=topic,proto3" json:"topic,omitempty"` +} + +func (x *KafkaSourceConfig) Reset() { + *x = KafkaSourceConfig{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_Source_proto_msgTypes[1] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *KafkaSourceConfig) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*KafkaSourceConfig) ProtoMessage() {} + +func (x *KafkaSourceConfig) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_Source_proto_msgTypes[1] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use KafkaSourceConfig.ProtoReflect.Descriptor instead. +func (*KafkaSourceConfig) Descriptor() ([]byte, []int) { + return file_feast_core_Source_proto_rawDescGZIP(), []int{1} +} + +func (x *KafkaSourceConfig) GetBootstrapServers() string { + if x != nil { + return x.BootstrapServers + } + return "" +} + +func (x *KafkaSourceConfig) GetTopic() string { + if x != nil { + return x.Topic + } + return "" +} + +var File_feast_core_Source_proto protoreflect.FileDescriptor + +var file_feast_core_Source_proto_rawDesc = []byte{ + 0x0a, 0x17, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x53, 0x6f, 0x75, + 0x72, 0x63, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x22, 0x96, 0x01, 0x0a, 0x06, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, + 0x12, 0x2a, 0x0a, 0x04, 0x74, 0x79, 0x70, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x16, + 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x6f, 0x75, 0x72, + 0x63, 0x65, 0x54, 0x79, 0x70, 0x65, 0x52, 0x04, 0x74, 0x79, 0x70, 0x65, 0x12, 0x4f, 0x0a, 0x13, + 0x6b, 0x61, 0x66, 0x6b, 0x61, 0x5f, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x5f, 0x63, 0x6f, 0x6e, + 0x66, 0x69, 0x67, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1d, 0x2e, 0x66, 0x65, 0x61, 0x73, + 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4b, 0x61, 0x66, 0x6b, 0x61, 0x53, 0x6f, 0x75, 0x72, + 0x63, 0x65, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x48, 0x00, 0x52, 0x11, 0x6b, 0x61, 0x66, 0x6b, + 0x61, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x42, 0x0f, 0x0a, + 0x0d, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x5f, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x22, 0x56, + 0x0a, 0x11, 0x4b, 0x61, 0x66, 0x6b, 0x61, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x43, 0x6f, 0x6e, + 0x66, 0x69, 0x67, 0x12, 0x2b, 0x0a, 0x11, 0x62, 0x6f, 0x6f, 0x74, 0x73, 0x74, 0x72, 0x61, 0x70, + 0x5f, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x73, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x10, + 0x62, 0x6f, 0x6f, 0x74, 0x73, 0x74, 0x72, 0x61, 0x70, 0x53, 0x65, 0x72, 0x76, 0x65, 0x72, 0x73, + 0x12, 0x14, 0x0a, 0x05, 0x74, 0x6f, 0x70, 0x69, 0x63, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x05, 0x74, 0x6f, 0x70, 0x69, 0x63, 0x2a, 0x24, 0x0a, 0x0a, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, + 0x54, 0x79, 0x70, 0x65, 0x12, 0x0b, 0x0a, 0x07, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, + 0x00, 0x12, 0x09, 0x0a, 0x05, 0x4b, 0x41, 0x46, 0x4b, 0x41, 0x10, 0x01, 0x42, 0x4a, 0x0a, 0x0a, + 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x42, 0x0b, 0x53, 0x6f, 0x75, 0x72, + 0x63, 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x2f, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, + 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, + 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, +} + +var ( + file_feast_core_Source_proto_rawDescOnce sync.Once + file_feast_core_Source_proto_rawDescData = file_feast_core_Source_proto_rawDesc +) + +func file_feast_core_Source_proto_rawDescGZIP() []byte { + file_feast_core_Source_proto_rawDescOnce.Do(func() { + file_feast_core_Source_proto_rawDescData = protoimpl.X.CompressGZIP(file_feast_core_Source_proto_rawDescData) + }) + return file_feast_core_Source_proto_rawDescData +} + +var file_feast_core_Source_proto_enumTypes = make([]protoimpl.EnumInfo, 1) +var file_feast_core_Source_proto_msgTypes = make([]protoimpl.MessageInfo, 2) +var file_feast_core_Source_proto_goTypes = []interface{}{ + (SourceType)(0), // 0: feast.core.SourceType + (*Source)(nil), // 1: feast.core.Source + (*KafkaSourceConfig)(nil), // 2: feast.core.KafkaSourceConfig +} +var file_feast_core_Source_proto_depIdxs = []int32{ + 0, // 0: feast.core.Source.type:type_name -> feast.core.SourceType + 2, // 1: feast.core.Source.kafka_source_config:type_name -> feast.core.KafkaSourceConfig + 2, // [2:2] is the sub-list for method output_type + 2, // [2:2] is the sub-list for method input_type + 2, // [2:2] is the sub-list for extension type_name + 2, // [2:2] is the sub-list for extension extendee + 0, // [0:2] is the sub-list for field type_name +} + +func init() { file_feast_core_Source_proto_init() } +func file_feast_core_Source_proto_init() { + if File_feast_core_Source_proto != nil { + return + } + if !protoimpl.UnsafeEnabled { + file_feast_core_Source_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Source); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_Source_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*KafkaSourceConfig); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + } + file_feast_core_Source_proto_msgTypes[0].OneofWrappers = []interface{}{ + (*Source_KafkaSourceConfig)(nil), + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_feast_core_Source_proto_rawDesc, + NumEnums: 1, + NumMessages: 2, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_feast_core_Source_proto_goTypes, + DependencyIndexes: file_feast_core_Source_proto_depIdxs, + EnumInfos: file_feast_core_Source_proto_enumTypes, + MessageInfos: file_feast_core_Source_proto_msgTypes, + }.Build() + File_feast_core_Source_proto = out.File + file_feast_core_Source_proto_rawDesc = nil + file_feast_core_Source_proto_goTypes = nil + file_feast_core_Source_proto_depIdxs = nil +} diff --git a/sdk/go/protos/feast/core/Store.pb.go b/sdk/go/protos/feast/core/Store.pb.go new file mode 100644 index 00000000000..55c699d788b --- /dev/null +++ b/sdk/go/protos/feast/core/Store.pb.go @@ -0,0 +1,749 @@ +// +// * Copyright 2019 The Feast Authors +// * +// * Licensed under the Apache License, Version 2.0 (the "License"); +// * you may not use this file except in compliance with the License. +// * You may obtain a copy of the License at +// * +// * https://www.apache.org/licenses/LICENSE-2.0 +// * +// * Unless required by applicable law or agreed to in writing, software +// * distributed under the License is distributed on an "AS IS" BASIS, +// * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// * See the License for the specific language governing permissions and +// * limitations under the License. +// + +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.21.0 +// protoc v3.10.0 +// source: feast/core/Store.proto + +package core + +import ( + proto "github.com/golang/protobuf/proto" + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +// This is a compile-time assertion that a sufficiently up-to-date version +// of the legacy proto package is being used. +const _ = proto.ProtoPackageIsVersion4 + +type Store_StoreType int32 + +const ( + Store_INVALID Store_StoreType = 0 + // Redis stores a FeatureRow element as a key, value pair. + // + // The Redis data types used (https://redis.io/topics/data-types): + // - key: STRING + // - value: STRING + // + // Encodings: + // - key: byte array of RedisKey (refer to feast.storage.RedisKey) + // - value: byte array of FeatureRow (refer to feast.types.FeatureRow) + // + Store_REDIS Store_StoreType = 1 + // BigQuery stores a FeatureRow element as a row in a BigQuery table. + // + // Table name is derived from the feature set name and version as: + // [feature_set_name]_v[feature_set_version] + // + // For example: + // A feature row for feature set "driver" and version "1" will be written + // to table "driver_v1". + // + // The entities and features in a FeatureSetSpec corresponds to the + // fields in the BigQuery table (these make up the BigQuery schema). + // The name of the entity spec and feature spec corresponds to the column + // names, and the value_type of entity spec and feature spec corresponds + // to BigQuery standard SQL data type of the column. + // + // The following BigQuery fields are reserved for Feast internal use. + // Ingestion of entity or feature spec with names identical + // to the following field names will raise an exception during ingestion. + // + // column_name | column_data_type | description + // ====================|==================|================================ + // - event_timestamp | TIMESTAMP | event time of the FeatureRow + // - created_timestamp | TIMESTAMP | processing time of the ingestion of the FeatureRow + // - job_id | STRING | identifier for the job that writes the FeatureRow to the corresponding BigQuery table + // + // BigQuery table created will be partitioned by the field "event_timestamp" + // of the FeatureRow (https://cloud.google.com/bigquery/docs/partitioned-tables). + // + // Since newer version of feature set can introduce breaking, non backward- + // compatible BigQuery schema updates, incrementing the version of a + // feature set will result in the creation of a new empty BigQuery table + // with the new schema. + // + // The following table shows how ValueType in Feast is mapped to + // BigQuery Standard SQL data types + // (https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types): + // + // BYTES : BYTES + // STRING : STRING + // INT32 : INT64 + // INT64 : IN64 + // DOUBLE : FLOAT64 + // FLOAT : FLOAT64 + // BOOL : BOOL + // BYTES_LIST : ARRAY + // STRING_LIST : ARRAY + // INT32_LIST : ARRAY + // INT64_LIST : ARRAY + // DOUBLE_LIST : ARRAY + // FLOAT_LIST : ARRAY + // BOOL_LIST : ARRAY + // + // The column mode in BigQuery is set to "Nullable" such that unset Value + // in a FeatureRow corresponds to NULL value in BigQuery. + // + Store_BIGQUERY Store_StoreType = 2 + // Unsupported in Feast 0.3 + Store_CASSANDRA Store_StoreType = 3 +) + +// Enum value maps for Store_StoreType. +var ( + Store_StoreType_name = map[int32]string{ + 0: "INVALID", + 1: "REDIS", + 2: "BIGQUERY", + 3: "CASSANDRA", + } + Store_StoreType_value = map[string]int32{ + "INVALID": 0, + "REDIS": 1, + "BIGQUERY": 2, + "CASSANDRA": 3, + } +) + +func (x Store_StoreType) Enum() *Store_StoreType { + p := new(Store_StoreType) + *p = x + return p +} + +func (x Store_StoreType) String() string { + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) +} + +func (Store_StoreType) Descriptor() protoreflect.EnumDescriptor { + return file_feast_core_Store_proto_enumTypes[0].Descriptor() +} + +func (Store_StoreType) Type() protoreflect.EnumType { + return &file_feast_core_Store_proto_enumTypes[0] +} + +func (x Store_StoreType) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) +} + +// Deprecated: Use Store_StoreType.Descriptor instead. +func (Store_StoreType) EnumDescriptor() ([]byte, []int) { + return file_feast_core_Store_proto_rawDescGZIP(), []int{0, 0} +} + +// Store provides a location where Feast reads and writes feature values. +// Feature values will be written to the Store in the form of FeatureRow elements. +// The way FeatureRow is encoded and decoded when it is written to and read from +// the Store depends on the type of the Store. +// +// For example, a FeatureRow will materialize as a row in a table in +// BigQuery but it will materialize as a key, value pair element in Redis. +// +type Store struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Name of the store. + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` + // Type of store. + Type Store_StoreType `protobuf:"varint,2,opt,name=type,proto3,enum=feast.core.Store_StoreType" json:"type,omitempty"` + // Feature sets to subscribe to. + Subscriptions []*Store_Subscription `protobuf:"bytes,4,rep,name=subscriptions,proto3" json:"subscriptions,omitempty"` + // Configuration to connect to the store. Required. + // + // Types that are assignable to Config: + // *Store_RedisConfig_ + // *Store_BigqueryConfig + // *Store_CassandraConfig_ + Config isStore_Config `protobuf_oneof:"config"` +} + +func (x *Store) Reset() { + *x = Store{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_Store_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *Store) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*Store) ProtoMessage() {} + +func (x *Store) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_Store_proto_msgTypes[0] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use Store.ProtoReflect.Descriptor instead. +func (*Store) Descriptor() ([]byte, []int) { + return file_feast_core_Store_proto_rawDescGZIP(), []int{0} +} + +func (x *Store) GetName() string { + if x != nil { + return x.Name + } + return "" +} + +func (x *Store) GetType() Store_StoreType { + if x != nil { + return x.Type + } + return Store_INVALID +} + +func (x *Store) GetSubscriptions() []*Store_Subscription { + if x != nil { + return x.Subscriptions + } + return nil +} + +func (m *Store) GetConfig() isStore_Config { + if m != nil { + return m.Config + } + return nil +} + +func (x *Store) GetRedisConfig() *Store_RedisConfig { + if x, ok := x.GetConfig().(*Store_RedisConfig_); ok { + return x.RedisConfig + } + return nil +} + +func (x *Store) GetBigqueryConfig() *Store_BigQueryConfig { + if x, ok := x.GetConfig().(*Store_BigqueryConfig); ok { + return x.BigqueryConfig + } + return nil +} + +func (x *Store) GetCassandraConfig() *Store_CassandraConfig { + if x, ok := x.GetConfig().(*Store_CassandraConfig_); ok { + return x.CassandraConfig + } + return nil +} + +type isStore_Config interface { + isStore_Config() +} + +type Store_RedisConfig_ struct { + RedisConfig *Store_RedisConfig `protobuf:"bytes,11,opt,name=redis_config,json=redisConfig,proto3,oneof"` +} + +type Store_BigqueryConfig struct { + BigqueryConfig *Store_BigQueryConfig `protobuf:"bytes,12,opt,name=bigquery_config,json=bigqueryConfig,proto3,oneof"` +} + +type Store_CassandraConfig_ struct { + CassandraConfig *Store_CassandraConfig `protobuf:"bytes,13,opt,name=cassandra_config,json=cassandraConfig,proto3,oneof"` +} + +func (*Store_RedisConfig_) isStore_Config() {} + +func (*Store_BigqueryConfig) isStore_Config() {} + +func (*Store_CassandraConfig_) isStore_Config() {} + +type Store_RedisConfig struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Host string `protobuf:"bytes,1,opt,name=host,proto3" json:"host,omitempty"` + Port int32 `protobuf:"varint,2,opt,name=port,proto3" json:"port,omitempty"` + // Optional. The number of milliseconds to wait before retrying failed Redis connection. + // By default, Feast uses exponential backoff policy and "initial_backoff_ms" sets the initial wait duration. + InitialBackoffMs int32 `protobuf:"varint,3,opt,name=initial_backoff_ms,json=initialBackoffMs,proto3" json:"initial_backoff_ms,omitempty"` + // Optional. Maximum total number of retries for connecting to Redis. Default to zero retries. + MaxRetries int32 `protobuf:"varint,4,opt,name=max_retries,json=maxRetries,proto3" json:"max_retries,omitempty"` +} + +func (x *Store_RedisConfig) Reset() { + *x = Store_RedisConfig{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_Store_proto_msgTypes[1] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *Store_RedisConfig) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*Store_RedisConfig) ProtoMessage() {} + +func (x *Store_RedisConfig) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_Store_proto_msgTypes[1] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use Store_RedisConfig.ProtoReflect.Descriptor instead. +func (*Store_RedisConfig) Descriptor() ([]byte, []int) { + return file_feast_core_Store_proto_rawDescGZIP(), []int{0, 0} +} + +func (x *Store_RedisConfig) GetHost() string { + if x != nil { + return x.Host + } + return "" +} + +func (x *Store_RedisConfig) GetPort() int32 { + if x != nil { + return x.Port + } + return 0 +} + +func (x *Store_RedisConfig) GetInitialBackoffMs() int32 { + if x != nil { + return x.InitialBackoffMs + } + return 0 +} + +func (x *Store_RedisConfig) GetMaxRetries() int32 { + if x != nil { + return x.MaxRetries + } + return 0 +} + +type Store_BigQueryConfig struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + ProjectId string `protobuf:"bytes,1,opt,name=project_id,json=projectId,proto3" json:"project_id,omitempty"` + DatasetId string `protobuf:"bytes,2,opt,name=dataset_id,json=datasetId,proto3" json:"dataset_id,omitempty"` +} + +func (x *Store_BigQueryConfig) Reset() { + *x = Store_BigQueryConfig{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_Store_proto_msgTypes[2] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *Store_BigQueryConfig) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*Store_BigQueryConfig) ProtoMessage() {} + +func (x *Store_BigQueryConfig) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_Store_proto_msgTypes[2] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use Store_BigQueryConfig.ProtoReflect.Descriptor instead. +func (*Store_BigQueryConfig) Descriptor() ([]byte, []int) { + return file_feast_core_Store_proto_rawDescGZIP(), []int{0, 1} +} + +func (x *Store_BigQueryConfig) GetProjectId() string { + if x != nil { + return x.ProjectId + } + return "" +} + +func (x *Store_BigQueryConfig) GetDatasetId() string { + if x != nil { + return x.DatasetId + } + return "" +} + +type Store_CassandraConfig struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Host string `protobuf:"bytes,1,opt,name=host,proto3" json:"host,omitempty"` + Port int32 `protobuf:"varint,2,opt,name=port,proto3" json:"port,omitempty"` +} + +func (x *Store_CassandraConfig) Reset() { + *x = Store_CassandraConfig{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_Store_proto_msgTypes[3] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *Store_CassandraConfig) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*Store_CassandraConfig) ProtoMessage() {} + +func (x *Store_CassandraConfig) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_Store_proto_msgTypes[3] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use Store_CassandraConfig.ProtoReflect.Descriptor instead. +func (*Store_CassandraConfig) Descriptor() ([]byte, []int) { + return file_feast_core_Store_proto_rawDescGZIP(), []int{0, 2} +} + +func (x *Store_CassandraConfig) GetHost() string { + if x != nil { + return x.Host + } + return "" +} + +func (x *Store_CassandraConfig) GetPort() int32 { + if x != nil { + return x.Port + } + return 0 +} + +type Store_Subscription struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Name of project that the feature sets belongs to. This can be one of + // - [project_name] + // - * + // If an asterisk is provided, filtering on projects will be disabled. All projects will + // be matched. It is NOT possible to provide an asterisk with a string in order to do + // pattern matching. + Project string `protobuf:"bytes,3,opt,name=project,proto3" json:"project,omitempty"` + // Name of the desired feature set. Asterisks can be used as wildcards in the name. + // Matching on names is only permitted if a specific project is defined. It is disallowed + // If the project name is set to "*" + // e.g. + // - * can be used to match all feature sets + // - my-feature-set* can be used to match all features prefixed by "my-feature-set" + // - my-feature-set-6 can be used to select a single feature set + Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` + // Versions of the given feature sets that will be returned. + // Valid options for version: + // "latest": only the latest version is returned. + // "*": Subscribe to all versions + // [version number]: pin to a specific version. Project and feature set name must be + // explicitly defined if a specific version is pinned. + Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"` +} + +func (x *Store_Subscription) Reset() { + *x = Store_Subscription{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_Store_proto_msgTypes[4] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *Store_Subscription) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*Store_Subscription) ProtoMessage() {} + +func (x *Store_Subscription) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_Store_proto_msgTypes[4] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use Store_Subscription.ProtoReflect.Descriptor instead. +func (*Store_Subscription) Descriptor() ([]byte, []int) { + return file_feast_core_Store_proto_rawDescGZIP(), []int{0, 3} +} + +func (x *Store_Subscription) GetProject() string { + if x != nil { + return x.Project + } + return "" +} + +func (x *Store_Subscription) GetName() string { + if x != nil { + return x.Name + } + return "" +} + +func (x *Store_Subscription) GetVersion() string { + if x != nil { + return x.Version + } + return "" +} + +var File_feast_core_Store_proto protoreflect.FileDescriptor + +var file_feast_core_Store_proto_rawDesc = []byte{ + 0x0a, 0x16, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x53, 0x74, 0x6f, + 0x72, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x63, 0x6f, 0x72, 0x65, 0x22, 0xa9, 0x06, 0x0a, 0x05, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x12, 0x12, + 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, + 0x6d, 0x65, 0x12, 0x2f, 0x0a, 0x04, 0x74, 0x79, 0x70, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, + 0x32, 0x1b, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x74, + 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x54, 0x79, 0x70, 0x65, 0x52, 0x04, 0x74, + 0x79, 0x70, 0x65, 0x12, 0x44, 0x0a, 0x0d, 0x73, 0x75, 0x62, 0x73, 0x63, 0x72, 0x69, 0x70, 0x74, + 0x69, 0x6f, 0x6e, 0x73, 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x1e, 0x2e, 0x66, 0x65, 0x61, + 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x75, + 0x62, 0x73, 0x63, 0x72, 0x69, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x0d, 0x73, 0x75, 0x62, 0x73, + 0x63, 0x72, 0x69, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x12, 0x42, 0x0a, 0x0c, 0x72, 0x65, 0x64, + 0x69, 0x73, 0x5f, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x18, 0x0b, 0x20, 0x01, 0x28, 0x0b, 0x32, + 0x1d, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x74, 0x6f, + 0x72, 0x65, 0x2e, 0x52, 0x65, 0x64, 0x69, 0x73, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x48, 0x00, + 0x52, 0x0b, 0x72, 0x65, 0x64, 0x69, 0x73, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x12, 0x4b, 0x0a, + 0x0f, 0x62, 0x69, 0x67, 0x71, 0x75, 0x65, 0x72, 0x79, 0x5f, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, + 0x18, 0x0c, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x20, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, + 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x2e, 0x42, 0x69, 0x67, 0x51, 0x75, 0x65, + 0x72, 0x79, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x48, 0x00, 0x52, 0x0e, 0x62, 0x69, 0x67, 0x71, + 0x75, 0x65, 0x72, 0x79, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x12, 0x4e, 0x0a, 0x10, 0x63, 0x61, + 0x73, 0x73, 0x61, 0x6e, 0x64, 0x72, 0x61, 0x5f, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x18, 0x0d, + 0x20, 0x01, 0x28, 0x0b, 0x32, 0x21, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, + 0x65, 0x2e, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x2e, 0x43, 0x61, 0x73, 0x73, 0x61, 0x6e, 0x64, 0x72, + 0x61, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x48, 0x00, 0x52, 0x0f, 0x63, 0x61, 0x73, 0x73, 0x61, + 0x6e, 0x64, 0x72, 0x61, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x1a, 0x84, 0x01, 0x0a, 0x0b, 0x52, + 0x65, 0x64, 0x69, 0x73, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x12, 0x12, 0x0a, 0x04, 0x68, 0x6f, + 0x73, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x68, 0x6f, 0x73, 0x74, 0x12, 0x12, + 0x0a, 0x04, 0x70, 0x6f, 0x72, 0x74, 0x18, 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, 0x04, 0x70, 0x6f, + 0x72, 0x74, 0x12, 0x2c, 0x0a, 0x12, 0x69, 0x6e, 0x69, 0x74, 0x69, 0x61, 0x6c, 0x5f, 0x62, 0x61, + 0x63, 0x6b, 0x6f, 0x66, 0x66, 0x5f, 0x6d, 0x73, 0x18, 0x03, 0x20, 0x01, 0x28, 0x05, 0x52, 0x10, + 0x69, 0x6e, 0x69, 0x74, 0x69, 0x61, 0x6c, 0x42, 0x61, 0x63, 0x6b, 0x6f, 0x66, 0x66, 0x4d, 0x73, + 0x12, 0x1f, 0x0a, 0x0b, 0x6d, 0x61, 0x78, 0x5f, 0x72, 0x65, 0x74, 0x72, 0x69, 0x65, 0x73, 0x18, + 0x04, 0x20, 0x01, 0x28, 0x05, 0x52, 0x0a, 0x6d, 0x61, 0x78, 0x52, 0x65, 0x74, 0x72, 0x69, 0x65, + 0x73, 0x1a, 0x4e, 0x0a, 0x0e, 0x42, 0x69, 0x67, 0x51, 0x75, 0x65, 0x72, 0x79, 0x43, 0x6f, 0x6e, + 0x66, 0x69, 0x67, 0x12, 0x1d, 0x0a, 0x0a, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x5f, 0x69, + 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, + 0x49, 0x64, 0x12, 0x1d, 0x0a, 0x0a, 0x64, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x5f, 0x69, 0x64, + 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x64, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x49, + 0x64, 0x1a, 0x39, 0x0a, 0x0f, 0x43, 0x61, 0x73, 0x73, 0x61, 0x6e, 0x64, 0x72, 0x61, 0x43, 0x6f, + 0x6e, 0x66, 0x69, 0x67, 0x12, 0x12, 0x0a, 0x04, 0x68, 0x6f, 0x73, 0x74, 0x18, 0x01, 0x20, 0x01, + 0x28, 0x09, 0x52, 0x04, 0x68, 0x6f, 0x73, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x70, 0x6f, 0x72, 0x74, + 0x18, 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, 0x04, 0x70, 0x6f, 0x72, 0x74, 0x1a, 0x56, 0x0a, 0x0c, + 0x53, 0x75, 0x62, 0x73, 0x63, 0x72, 0x69, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x18, 0x0a, 0x07, + 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x70, + 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, + 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, + 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x76, 0x65, 0x72, + 0x73, 0x69, 0x6f, 0x6e, 0x22, 0x40, 0x0a, 0x09, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x54, 0x79, 0x70, + 0x65, 0x12, 0x0b, 0x0a, 0x07, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, 0x12, 0x09, + 0x0a, 0x05, 0x52, 0x45, 0x44, 0x49, 0x53, 0x10, 0x01, 0x12, 0x0c, 0x0a, 0x08, 0x42, 0x49, 0x47, + 0x51, 0x55, 0x45, 0x52, 0x59, 0x10, 0x02, 0x12, 0x0d, 0x0a, 0x09, 0x43, 0x41, 0x53, 0x53, 0x41, + 0x4e, 0x44, 0x52, 0x41, 0x10, 0x03, 0x42, 0x08, 0x0a, 0x06, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, + 0x42, 0x49, 0x0a, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x42, 0x0a, + 0x53, 0x74, 0x6f, 0x72, 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x2f, 0x67, 0x69, 0x74, 0x68, + 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, + 0x73, 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, + 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, + 0x74, 0x6f, 0x33, +} + +var ( + file_feast_core_Store_proto_rawDescOnce sync.Once + file_feast_core_Store_proto_rawDescData = file_feast_core_Store_proto_rawDesc +) + +func file_feast_core_Store_proto_rawDescGZIP() []byte { + file_feast_core_Store_proto_rawDescOnce.Do(func() { + file_feast_core_Store_proto_rawDescData = protoimpl.X.CompressGZIP(file_feast_core_Store_proto_rawDescData) + }) + return file_feast_core_Store_proto_rawDescData +} + +var file_feast_core_Store_proto_enumTypes = make([]protoimpl.EnumInfo, 1) +var file_feast_core_Store_proto_msgTypes = make([]protoimpl.MessageInfo, 5) +var file_feast_core_Store_proto_goTypes = []interface{}{ + (Store_StoreType)(0), // 0: feast.core.Store.StoreType + (*Store)(nil), // 1: feast.core.Store + (*Store_RedisConfig)(nil), // 2: feast.core.Store.RedisConfig + (*Store_BigQueryConfig)(nil), // 3: feast.core.Store.BigQueryConfig + (*Store_CassandraConfig)(nil), // 4: feast.core.Store.CassandraConfig + (*Store_Subscription)(nil), // 5: feast.core.Store.Subscription +} +var file_feast_core_Store_proto_depIdxs = []int32{ + 0, // 0: feast.core.Store.type:type_name -> feast.core.Store.StoreType + 5, // 1: feast.core.Store.subscriptions:type_name -> feast.core.Store.Subscription + 2, // 2: feast.core.Store.redis_config:type_name -> feast.core.Store.RedisConfig + 3, // 3: feast.core.Store.bigquery_config:type_name -> feast.core.Store.BigQueryConfig + 4, // 4: feast.core.Store.cassandra_config:type_name -> feast.core.Store.CassandraConfig + 5, // [5:5] is the sub-list for method output_type + 5, // [5:5] is the sub-list for method input_type + 5, // [5:5] is the sub-list for extension type_name + 5, // [5:5] is the sub-list for extension extendee + 0, // [0:5] is the sub-list for field type_name +} + +func init() { file_feast_core_Store_proto_init() } +func file_feast_core_Store_proto_init() { + if File_feast_core_Store_proto != nil { + return + } + if !protoimpl.UnsafeEnabled { + file_feast_core_Store_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Store); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_Store_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Store_RedisConfig); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_Store_proto_msgTypes[2].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Store_BigQueryConfig); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_Store_proto_msgTypes[3].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Store_CassandraConfig); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_Store_proto_msgTypes[4].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Store_Subscription); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + } + file_feast_core_Store_proto_msgTypes[0].OneofWrappers = []interface{}{ + (*Store_RedisConfig_)(nil), + (*Store_BigqueryConfig)(nil), + (*Store_CassandraConfig_)(nil), + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_feast_core_Store_proto_rawDesc, + NumEnums: 1, + NumMessages: 5, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_feast_core_Store_proto_goTypes, + DependencyIndexes: file_feast_core_Store_proto_depIdxs, + EnumInfos: file_feast_core_Store_proto_enumTypes, + MessageInfos: file_feast_core_Store_proto_msgTypes, + }.Build() + File_feast_core_Store_proto = out.File + file_feast_core_Store_proto_rawDesc = nil + file_feast_core_Store_proto_goTypes = nil + file_feast_core_Store_proto_depIdxs = nil +} diff --git a/sdk/go/protos/feast/serving/ServingService.pb.go b/sdk/go/protos/feast/serving/ServingService.pb.go index 6954b1f4f61..2485687d81e 100644 --- a/sdk/go/protos/feast/serving/ServingService.pb.go +++ b/sdk/go/protos/feast/serving/ServingService.pb.go @@ -16,7 +16,7 @@ // Code generated by protoc-gen-go. DO NOT EDIT. // versions: // protoc-gen-go v1.21.0 -// protoc v3.10.1 +// protoc v3.10.0 // source: feast/serving/ServingService.proto package serving diff --git a/sdk/go/protos/feast/storage/Redis.pb.go b/sdk/go/protos/feast/storage/Redis.pb.go new file mode 100644 index 00000000000..1dca28e26af --- /dev/null +++ b/sdk/go/protos/feast/storage/Redis.pb.go @@ -0,0 +1,187 @@ +// +// Copyright 2019 The Feast Authors +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// https://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.21.0 +// protoc v3.10.0 +// source: feast/storage/Redis.proto + +package storage + +import ( + types "github.com/gojek/feast/sdk/go/protos/feast/types" + proto "github.com/golang/protobuf/proto" + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +// This is a compile-time assertion that a sufficiently up-to-date version +// of the legacy proto package is being used. +const _ = proto.ProtoPackageIsVersion4 + +type RedisKey struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // FeatureSet this row belongs to, this is defined as featureSetName:version. + FeatureSet string `protobuf:"bytes,2,opt,name=feature_set,json=featureSet,proto3" json:"feature_set,omitempty"` + // List of fields containing entity names and their respective values + // contained within this feature row. The entities should be sorted + // by the entity name alphabetically in ascending order. + Entities []*types.Field `protobuf:"bytes,3,rep,name=entities,proto3" json:"entities,omitempty"` +} + +func (x *RedisKey) Reset() { + *x = RedisKey{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_storage_Redis_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *RedisKey) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*RedisKey) ProtoMessage() {} + +func (x *RedisKey) ProtoReflect() protoreflect.Message { + mi := &file_feast_storage_Redis_proto_msgTypes[0] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use RedisKey.ProtoReflect.Descriptor instead. +func (*RedisKey) Descriptor() ([]byte, []int) { + return file_feast_storage_Redis_proto_rawDescGZIP(), []int{0} +} + +func (x *RedisKey) GetFeatureSet() string { + if x != nil { + return x.FeatureSet + } + return "" +} + +func (x *RedisKey) GetEntities() []*types.Field { + if x != nil { + return x.Entities + } + return nil +} + +var File_feast_storage_Redis_proto protoreflect.FileDescriptor + +var file_feast_storage_Redis_proto_rawDesc = []byte{ + 0x0a, 0x19, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x74, 0x6f, 0x72, 0x61, 0x67, 0x65, 0x2f, + 0x52, 0x65, 0x64, 0x69, 0x73, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0d, 0x66, 0x65, 0x61, + 0x73, 0x74, 0x2e, 0x73, 0x74, 0x6f, 0x72, 0x61, 0x67, 0x65, 0x1a, 0x17, 0x66, 0x65, 0x61, 0x73, + 0x74, 0x2f, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2f, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x2e, 0x70, 0x72, + 0x6f, 0x74, 0x6f, 0x22, 0x5b, 0x0a, 0x08, 0x52, 0x65, 0x64, 0x69, 0x73, 0x4b, 0x65, 0x79, 0x12, + 0x1f, 0x0a, 0x0b, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x5f, 0x73, 0x65, 0x74, 0x18, 0x02, + 0x20, 0x01, 0x28, 0x09, 0x52, 0x0a, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, + 0x12, 0x2e, 0x0a, 0x08, 0x65, 0x6e, 0x74, 0x69, 0x74, 0x69, 0x65, 0x73, 0x18, 0x03, 0x20, 0x03, + 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, + 0x2e, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x52, 0x08, 0x65, 0x6e, 0x74, 0x69, 0x74, 0x69, 0x65, 0x73, + 0x42, 0x4f, 0x0a, 0x0d, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x74, 0x6f, 0x72, 0x61, 0x67, + 0x65, 0x42, 0x0a, 0x52, 0x65, 0x64, 0x69, 0x73, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x32, 0x67, + 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, + 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, + 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x74, 0x6f, 0x72, 0x61, 0x67, + 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, +} + +var ( + file_feast_storage_Redis_proto_rawDescOnce sync.Once + file_feast_storage_Redis_proto_rawDescData = file_feast_storage_Redis_proto_rawDesc +) + +func file_feast_storage_Redis_proto_rawDescGZIP() []byte { + file_feast_storage_Redis_proto_rawDescOnce.Do(func() { + file_feast_storage_Redis_proto_rawDescData = protoimpl.X.CompressGZIP(file_feast_storage_Redis_proto_rawDescData) + }) + return file_feast_storage_Redis_proto_rawDescData +} + +var file_feast_storage_Redis_proto_msgTypes = make([]protoimpl.MessageInfo, 1) +var file_feast_storage_Redis_proto_goTypes = []interface{}{ + (*RedisKey)(nil), // 0: feast.storage.RedisKey + (*types.Field)(nil), // 1: feast.types.Field +} +var file_feast_storage_Redis_proto_depIdxs = []int32{ + 1, // 0: feast.storage.RedisKey.entities:type_name -> feast.types.Field + 1, // [1:1] is the sub-list for method output_type + 1, // [1:1] is the sub-list for method input_type + 1, // [1:1] is the sub-list for extension type_name + 1, // [1:1] is the sub-list for extension extendee + 0, // [0:1] is the sub-list for field type_name +} + +func init() { file_feast_storage_Redis_proto_init() } +func file_feast_storage_Redis_proto_init() { + if File_feast_storage_Redis_proto != nil { + return + } + if !protoimpl.UnsafeEnabled { + file_feast_storage_Redis_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*RedisKey); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_feast_storage_Redis_proto_rawDesc, + NumEnums: 0, + NumMessages: 1, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_feast_storage_Redis_proto_goTypes, + DependencyIndexes: file_feast_storage_Redis_proto_depIdxs, + MessageInfos: file_feast_storage_Redis_proto_msgTypes, + }.Build() + File_feast_storage_Redis_proto = out.File + file_feast_storage_Redis_proto_rawDesc = nil + file_feast_storage_Redis_proto_goTypes = nil + file_feast_storage_Redis_proto_depIdxs = nil +} diff --git a/sdk/go/protos/feast/types/FeatureRow.pb.go b/sdk/go/protos/feast/types/FeatureRow.pb.go index e6358b13003..769f219d32e 100644 --- a/sdk/go/protos/feast/types/FeatureRow.pb.go +++ b/sdk/go/protos/feast/types/FeatureRow.pb.go @@ -16,7 +16,7 @@ // Code generated by protoc-gen-go. DO NOT EDIT. // versions: // protoc-gen-go v1.21.0 -// protoc v3.10.1 +// protoc v3.10.0 // source: feast/types/FeatureRow.proto package types diff --git a/sdk/go/protos/feast/types/FeatureRowExtended.pb.go b/sdk/go/protos/feast/types/FeatureRowExtended.pb.go index c18628a2f34..8ca9ee1bc90 100644 --- a/sdk/go/protos/feast/types/FeatureRowExtended.pb.go +++ b/sdk/go/protos/feast/types/FeatureRowExtended.pb.go @@ -16,7 +16,7 @@ // Code generated by protoc-gen-go. DO NOT EDIT. // versions: // protoc-gen-go v1.21.0 -// protoc v3.10.1 +// protoc v3.10.0 // source: feast/types/FeatureRowExtended.proto package types diff --git a/sdk/go/protos/feast/types/Field.pb.go b/sdk/go/protos/feast/types/Field.pb.go index bcd8505d6bd..c7b5193db5d 100644 --- a/sdk/go/protos/feast/types/Field.pb.go +++ b/sdk/go/protos/feast/types/Field.pb.go @@ -16,7 +16,7 @@ // Code generated by protoc-gen-go. DO NOT EDIT. // versions: // protoc-gen-go v1.21.0 -// protoc v3.10.1 +// protoc v3.10.0 // source: feast/types/Field.proto package types diff --git a/sdk/go/protos/feast/types/Value.pb.go b/sdk/go/protos/feast/types/Value.pb.go index ad574405aeb..ba530b2dac2 100644 --- a/sdk/go/protos/feast/types/Value.pb.go +++ b/sdk/go/protos/feast/types/Value.pb.go @@ -16,7 +16,7 @@ // Code generated by protoc-gen-go. DO NOT EDIT. // versions: // protoc-gen-go v1.21.0 -// protoc v3.10.1 +// protoc v3.10.0 // source: feast/types/Value.proto package types diff --git a/sdk/go/protos/tensorflow_metadata/proto/v0/path.pb.go b/sdk/go/protos/tensorflow_metadata/proto/v0/path.pb.go new file mode 100644 index 00000000000..5b55e5bca5a --- /dev/null +++ b/sdk/go/protos/tensorflow_metadata/proto/v0/path.pb.go @@ -0,0 +1,186 @@ +// Copyright 2018 The TensorFlow Authors. All Rights Reserved. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ============================================================================= + +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.21.0 +// protoc v3.10.0 +// source: tensorflow_metadata/proto/v0/path.proto + +package v0 + +import ( + proto "github.com/golang/protobuf/proto" + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +// This is a compile-time assertion that a sufficiently up-to-date version +// of the legacy proto package is being used. +const _ = proto.ProtoPackageIsVersion4 + +// A path is a more general substitute for the name of a field or feature that +// can be used for flat examples as well as structured data. For example, if +// we had data in a protocol buffer: +// message Person { +// int age = 1; +// optional string gender = 2; +// repeated Person parent = 3; +// } +// Thus, here the path {step:["parent", "age"]} in statistics would refer to the +// age of a parent, and {step:["parent", "parent", "age"]} would refer to the +// age of a grandparent. This allows us to distinguish between the statistics +// of parents' ages and grandparents' ages. In general, repeated messages are +// to be preferred to linked lists of arbitrary length. +// For SequenceExample, if we have a feature list "foo", this is represented +// by {step:["##SEQUENCE##", "foo"]}. +type Path struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Any string is a valid step. + // However, whenever possible have a step be [A-Za-z0-9_]+. + Step []string `protobuf:"bytes,1,rep,name=step" json:"step,omitempty"` +} + +func (x *Path) Reset() { + *x = Path{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_path_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *Path) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*Path) ProtoMessage() {} + +func (x *Path) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_path_proto_msgTypes[0] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use Path.ProtoReflect.Descriptor instead. +func (*Path) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_path_proto_rawDescGZIP(), []int{0} +} + +func (x *Path) GetStep() []string { + if x != nil { + return x.Step + } + return nil +} + +var File_tensorflow_metadata_proto_v0_path_proto protoreflect.FileDescriptor + +var file_tensorflow_metadata_proto_v0_path_proto_rawDesc = []byte{ + 0x0a, 0x27, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x5f, 0x6d, 0x65, 0x74, + 0x61, 0x64, 0x61, 0x74, 0x61, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2f, 0x76, 0x30, 0x2f, 0x70, + 0x61, 0x74, 0x68, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x16, 0x74, 0x65, 0x6e, 0x73, 0x6f, + 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, + 0x30, 0x22, 0x1a, 0x0a, 0x04, 0x50, 0x61, 0x74, 0x68, 0x12, 0x12, 0x0a, 0x04, 0x73, 0x74, 0x65, + 0x70, 0x18, 0x01, 0x20, 0x03, 0x28, 0x09, 0x52, 0x04, 0x73, 0x74, 0x65, 0x70, 0x42, 0x70, 0x0a, + 0x1a, 0x6f, 0x72, 0x67, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, + 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x50, 0x01, 0x5a, 0x4d, 0x67, + 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, + 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, + 0x74, 0x6f, 0x73, 0x2f, 0x74, 0x68, 0x69, 0x72, 0x64, 0x5f, 0x70, 0x61, 0x72, 0x74, 0x79, 0x2f, + 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x64, + 0x61, 0x74, 0x61, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2f, 0x76, 0x30, 0xf8, 0x01, 0x01, +} + +var ( + file_tensorflow_metadata_proto_v0_path_proto_rawDescOnce sync.Once + file_tensorflow_metadata_proto_v0_path_proto_rawDescData = file_tensorflow_metadata_proto_v0_path_proto_rawDesc +) + +func file_tensorflow_metadata_proto_v0_path_proto_rawDescGZIP() []byte { + file_tensorflow_metadata_proto_v0_path_proto_rawDescOnce.Do(func() { + file_tensorflow_metadata_proto_v0_path_proto_rawDescData = protoimpl.X.CompressGZIP(file_tensorflow_metadata_proto_v0_path_proto_rawDescData) + }) + return file_tensorflow_metadata_proto_v0_path_proto_rawDescData +} + +var file_tensorflow_metadata_proto_v0_path_proto_msgTypes = make([]protoimpl.MessageInfo, 1) +var file_tensorflow_metadata_proto_v0_path_proto_goTypes = []interface{}{ + (*Path)(nil), // 0: tensorflow.metadata.v0.Path +} +var file_tensorflow_metadata_proto_v0_path_proto_depIdxs = []int32{ + 0, // [0:0] is the sub-list for method output_type + 0, // [0:0] is the sub-list for method input_type + 0, // [0:0] is the sub-list for extension type_name + 0, // [0:0] is the sub-list for extension extendee + 0, // [0:0] is the sub-list for field type_name +} + +func init() { file_tensorflow_metadata_proto_v0_path_proto_init() } +func file_tensorflow_metadata_proto_v0_path_proto_init() { + if File_tensorflow_metadata_proto_v0_path_proto != nil { + return + } + if !protoimpl.UnsafeEnabled { + file_tensorflow_metadata_proto_v0_path_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Path); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_tensorflow_metadata_proto_v0_path_proto_rawDesc, + NumEnums: 0, + NumMessages: 1, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_tensorflow_metadata_proto_v0_path_proto_goTypes, + DependencyIndexes: file_tensorflow_metadata_proto_v0_path_proto_depIdxs, + MessageInfos: file_tensorflow_metadata_proto_v0_path_proto_msgTypes, + }.Build() + File_tensorflow_metadata_proto_v0_path_proto = out.File + file_tensorflow_metadata_proto_v0_path_proto_rawDesc = nil + file_tensorflow_metadata_proto_v0_path_proto_goTypes = nil + file_tensorflow_metadata_proto_v0_path_proto_depIdxs = nil +} diff --git a/sdk/go/protos/tensorflow_metadata/proto/v0/schema.pb.go b/sdk/go/protos/tensorflow_metadata/proto/v0/schema.pb.go new file mode 100644 index 00000000000..5eec2f259d0 --- /dev/null +++ b/sdk/go/protos/tensorflow_metadata/proto/v0/schema.pb.go @@ -0,0 +1,4084 @@ +// Copyright 2017 The TensorFlow Authors. All Rights Reserved. +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ============================================================================= + +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.21.0 +// protoc v3.10.0 +// source: tensorflow_metadata/proto/v0/schema.proto + +package v0 + +import ( + proto "github.com/golang/protobuf/proto" + any "github.com/golang/protobuf/ptypes/any" + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +// This is a compile-time assertion that a sufficiently up-to-date version +// of the legacy proto package is being used. +const _ = proto.ProtoPackageIsVersion4 + +// LifecycleStage. Only UNKNOWN_STAGE, BETA, and PRODUCTION features are +// actually validated. +// PLANNED, ALPHA, and DEBUG are treated as DEPRECATED. +type LifecycleStage int32 + +const ( + LifecycleStage_UNKNOWN_STAGE LifecycleStage = 0 // Unknown stage. + LifecycleStage_PLANNED LifecycleStage = 1 // Planned feature, may not be created yet. + LifecycleStage_ALPHA LifecycleStage = 2 // Prototype feature, not used in experiments yet. + LifecycleStage_BETA LifecycleStage = 3 // Used in user-facing experiments. + LifecycleStage_PRODUCTION LifecycleStage = 4 // Used in a significant fraction of user traffic. + LifecycleStage_DEPRECATED LifecycleStage = 5 // No longer supported: do not use in new models. + LifecycleStage_DEBUG_ONLY LifecycleStage = 6 // Only exists for debugging purposes. +) + +// Enum value maps for LifecycleStage. +var ( + LifecycleStage_name = map[int32]string{ + 0: "UNKNOWN_STAGE", + 1: "PLANNED", + 2: "ALPHA", + 3: "BETA", + 4: "PRODUCTION", + 5: "DEPRECATED", + 6: "DEBUG_ONLY", + } + LifecycleStage_value = map[string]int32{ + "UNKNOWN_STAGE": 0, + "PLANNED": 1, + "ALPHA": 2, + "BETA": 3, + "PRODUCTION": 4, + "DEPRECATED": 5, + "DEBUG_ONLY": 6, + } +) + +func (x LifecycleStage) Enum() *LifecycleStage { + p := new(LifecycleStage) + *p = x + return p +} + +func (x LifecycleStage) String() string { + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) +} + +func (LifecycleStage) Descriptor() protoreflect.EnumDescriptor { + return file_tensorflow_metadata_proto_v0_schema_proto_enumTypes[0].Descriptor() +} + +func (LifecycleStage) Type() protoreflect.EnumType { + return &file_tensorflow_metadata_proto_v0_schema_proto_enumTypes[0] +} + +func (x LifecycleStage) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) +} + +// Deprecated: Do not use. +func (x *LifecycleStage) UnmarshalJSON(b []byte) error { + num, err := protoimpl.X.UnmarshalJSONEnum(x.Descriptor(), b) + if err != nil { + return err + } + *x = LifecycleStage(num) + return nil +} + +// Deprecated: Use LifecycleStage.Descriptor instead. +func (LifecycleStage) EnumDescriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{0} +} + +// Describes the physical representation of a feature. +// It may be different than the logical representation, which +// is represented as a Domain. +type FeatureType int32 + +const ( + FeatureType_TYPE_UNKNOWN FeatureType = 0 + FeatureType_BYTES FeatureType = 1 + FeatureType_INT FeatureType = 2 + FeatureType_FLOAT FeatureType = 3 + FeatureType_STRUCT FeatureType = 4 +) + +// Enum value maps for FeatureType. +var ( + FeatureType_name = map[int32]string{ + 0: "TYPE_UNKNOWN", + 1: "BYTES", + 2: "INT", + 3: "FLOAT", + 4: "STRUCT", + } + FeatureType_value = map[string]int32{ + "TYPE_UNKNOWN": 0, + "BYTES": 1, + "INT": 2, + "FLOAT": 3, + "STRUCT": 4, + } +) + +func (x FeatureType) Enum() *FeatureType { + p := new(FeatureType) + *p = x + return p +} + +func (x FeatureType) String() string { + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) +} + +func (FeatureType) Descriptor() protoreflect.EnumDescriptor { + return file_tensorflow_metadata_proto_v0_schema_proto_enumTypes[1].Descriptor() +} + +func (FeatureType) Type() protoreflect.EnumType { + return &file_tensorflow_metadata_proto_v0_schema_proto_enumTypes[1] +} + +func (x FeatureType) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) +} + +// Deprecated: Do not use. +func (x *FeatureType) UnmarshalJSON(b []byte) error { + num, err := protoimpl.X.UnmarshalJSONEnum(x.Descriptor(), b) + if err != nil { + return err + } + *x = FeatureType(num) + return nil +} + +// Deprecated: Use FeatureType.Descriptor instead. +func (FeatureType) EnumDescriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{1} +} + +type TimeDomain_IntegerTimeFormat int32 + +const ( + TimeDomain_FORMAT_UNKNOWN TimeDomain_IntegerTimeFormat = 0 + TimeDomain_UNIX_DAYS TimeDomain_IntegerTimeFormat = 5 // Number of days since 1970-01-01. + TimeDomain_UNIX_SECONDS TimeDomain_IntegerTimeFormat = 1 + TimeDomain_UNIX_MILLISECONDS TimeDomain_IntegerTimeFormat = 2 + TimeDomain_UNIX_MICROSECONDS TimeDomain_IntegerTimeFormat = 3 + TimeDomain_UNIX_NANOSECONDS TimeDomain_IntegerTimeFormat = 4 +) + +// Enum value maps for TimeDomain_IntegerTimeFormat. +var ( + TimeDomain_IntegerTimeFormat_name = map[int32]string{ + 0: "FORMAT_UNKNOWN", + 5: "UNIX_DAYS", + 1: "UNIX_SECONDS", + 2: "UNIX_MILLISECONDS", + 3: "UNIX_MICROSECONDS", + 4: "UNIX_NANOSECONDS", + } + TimeDomain_IntegerTimeFormat_value = map[string]int32{ + "FORMAT_UNKNOWN": 0, + "UNIX_DAYS": 5, + "UNIX_SECONDS": 1, + "UNIX_MILLISECONDS": 2, + "UNIX_MICROSECONDS": 3, + "UNIX_NANOSECONDS": 4, + } +) + +func (x TimeDomain_IntegerTimeFormat) Enum() *TimeDomain_IntegerTimeFormat { + p := new(TimeDomain_IntegerTimeFormat) + *p = x + return p +} + +func (x TimeDomain_IntegerTimeFormat) String() string { + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) +} + +func (TimeDomain_IntegerTimeFormat) Descriptor() protoreflect.EnumDescriptor { + return file_tensorflow_metadata_proto_v0_schema_proto_enumTypes[2].Descriptor() +} + +func (TimeDomain_IntegerTimeFormat) Type() protoreflect.EnumType { + return &file_tensorflow_metadata_proto_v0_schema_proto_enumTypes[2] +} + +func (x TimeDomain_IntegerTimeFormat) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) +} + +// Deprecated: Do not use. +func (x *TimeDomain_IntegerTimeFormat) UnmarshalJSON(b []byte) error { + num, err := protoimpl.X.UnmarshalJSONEnum(x.Descriptor(), b) + if err != nil { + return err + } + *x = TimeDomain_IntegerTimeFormat(num) + return nil +} + +// Deprecated: Use TimeDomain_IntegerTimeFormat.Descriptor instead. +func (TimeDomain_IntegerTimeFormat) EnumDescriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{19, 0} +} + +type TimeOfDayDomain_IntegerTimeOfDayFormat int32 + +const ( + TimeOfDayDomain_FORMAT_UNKNOWN TimeOfDayDomain_IntegerTimeOfDayFormat = 0 + // Time values, containing hour/minute/second/nanos, encoded into 8-byte + // bit fields following the ZetaSQL convention: + // 6 5 4 3 2 1 + // MSB 3210987654321098765432109876543210987654321098765432109876543210 LSB + // | H || M || S ||---------- nanos -----------| + TimeOfDayDomain_PACKED_64_NANOS TimeOfDayDomain_IntegerTimeOfDayFormat = 1 +) + +// Enum value maps for TimeOfDayDomain_IntegerTimeOfDayFormat. +var ( + TimeOfDayDomain_IntegerTimeOfDayFormat_name = map[int32]string{ + 0: "FORMAT_UNKNOWN", + 1: "PACKED_64_NANOS", + } + TimeOfDayDomain_IntegerTimeOfDayFormat_value = map[string]int32{ + "FORMAT_UNKNOWN": 0, + "PACKED_64_NANOS": 1, + } +) + +func (x TimeOfDayDomain_IntegerTimeOfDayFormat) Enum() *TimeOfDayDomain_IntegerTimeOfDayFormat { + p := new(TimeOfDayDomain_IntegerTimeOfDayFormat) + *p = x + return p +} + +func (x TimeOfDayDomain_IntegerTimeOfDayFormat) String() string { + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) +} + +func (TimeOfDayDomain_IntegerTimeOfDayFormat) Descriptor() protoreflect.EnumDescriptor { + return file_tensorflow_metadata_proto_v0_schema_proto_enumTypes[3].Descriptor() +} + +func (TimeOfDayDomain_IntegerTimeOfDayFormat) Type() protoreflect.EnumType { + return &file_tensorflow_metadata_proto_v0_schema_proto_enumTypes[3] +} + +func (x TimeOfDayDomain_IntegerTimeOfDayFormat) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) +} + +// Deprecated: Do not use. +func (x *TimeOfDayDomain_IntegerTimeOfDayFormat) UnmarshalJSON(b []byte) error { + num, err := protoimpl.X.UnmarshalJSONEnum(x.Descriptor(), b) + if err != nil { + return err + } + *x = TimeOfDayDomain_IntegerTimeOfDayFormat(num) + return nil +} + +// Deprecated: Use TimeOfDayDomain_IntegerTimeOfDayFormat.Descriptor instead. +func (TimeOfDayDomain_IntegerTimeOfDayFormat) EnumDescriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{20, 0} +} + +// +// Message to represent schema information. +// NextID: 14 +type Schema struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Features described in this schema. + Feature []*Feature `protobuf:"bytes,1,rep,name=feature" json:"feature,omitempty"` + // Sparse features described in this schema. + SparseFeature []*SparseFeature `protobuf:"bytes,6,rep,name=sparse_feature,json=sparseFeature" json:"sparse_feature,omitempty"` + // Weighted features described in this schema. + WeightedFeature []*WeightedFeature `protobuf:"bytes,12,rep,name=weighted_feature,json=weightedFeature" json:"weighted_feature,omitempty"` + // declared as top-level features in . + // String domains referenced in the features. + StringDomain []*StringDomain `protobuf:"bytes,4,rep,name=string_domain,json=stringDomain" json:"string_domain,omitempty"` + // top level float domains that can be reused by features + FloatDomain []*FloatDomain `protobuf:"bytes,9,rep,name=float_domain,json=floatDomain" json:"float_domain,omitempty"` + // top level int domains that can be reused by features + IntDomain []*IntDomain `protobuf:"bytes,10,rep,name=int_domain,json=intDomain" json:"int_domain,omitempty"` + // Default environments for each feature. + // An environment represents both a type of location (e.g. a server or phone) + // and a time (e.g. right before model X is run). In the standard scenario, + // 99% of the features should be in the default environments TRAINING, + // SERVING, and the LABEL (or labels) AND WEIGHT is only available at TRAINING + // (not at serving). + // Other possible variations: + // 1. There may be TRAINING_MOBILE, SERVING_MOBILE, TRAINING_SERVICE, + // and SERVING_SERVICE. + // 2. If one is ensembling three models, where the predictions of the first + // three models are available for the ensemble model, there may be + // TRAINING, SERVING_INITIAL, SERVING_ENSEMBLE. + // See FeatureProto::not_in_environment and FeatureProto::in_environment. + DefaultEnvironment []string `protobuf:"bytes,5,rep,name=default_environment,json=defaultEnvironment" json:"default_environment,omitempty"` + // Additional information about the schema as a whole. Features may also + // be annotated individually. + Annotation *Annotation `protobuf:"bytes,8,opt,name=annotation" json:"annotation,omitempty"` + // Dataset-level constraints. This is currently used for specifying + // information about changes in num_examples. + DatasetConstraints *DatasetConstraints `protobuf:"bytes,11,opt,name=dataset_constraints,json=datasetConstraints" json:"dataset_constraints,omitempty"` + // TensorRepresentation groups. The keys are the names of the groups. + // Key "" (empty string) denotes the "default" group, which is what should + // be used when a group name is not provided. + // See the documentation at TensorRepresentationGroup for more info. + // Under development. DO NOT USE. + TensorRepresentationGroup map[string]*TensorRepresentationGroup `protobuf:"bytes,13,rep,name=tensor_representation_group,json=tensorRepresentationGroup" json:"tensor_representation_group,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` +} + +func (x *Schema) Reset() { + *x = Schema{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *Schema) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*Schema) ProtoMessage() {} + +func (x *Schema) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[0] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use Schema.ProtoReflect.Descriptor instead. +func (*Schema) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{0} +} + +func (x *Schema) GetFeature() []*Feature { + if x != nil { + return x.Feature + } + return nil +} + +func (x *Schema) GetSparseFeature() []*SparseFeature { + if x != nil { + return x.SparseFeature + } + return nil +} + +func (x *Schema) GetWeightedFeature() []*WeightedFeature { + if x != nil { + return x.WeightedFeature + } + return nil +} + +func (x *Schema) GetStringDomain() []*StringDomain { + if x != nil { + return x.StringDomain + } + return nil +} + +func (x *Schema) GetFloatDomain() []*FloatDomain { + if x != nil { + return x.FloatDomain + } + return nil +} + +func (x *Schema) GetIntDomain() []*IntDomain { + if x != nil { + return x.IntDomain + } + return nil +} + +func (x *Schema) GetDefaultEnvironment() []string { + if x != nil { + return x.DefaultEnvironment + } + return nil +} + +func (x *Schema) GetAnnotation() *Annotation { + if x != nil { + return x.Annotation + } + return nil +} + +func (x *Schema) GetDatasetConstraints() *DatasetConstraints { + if x != nil { + return x.DatasetConstraints + } + return nil +} + +func (x *Schema) GetTensorRepresentationGroup() map[string]*TensorRepresentationGroup { + if x != nil { + return x.TensorRepresentationGroup + } + return nil +} + +// Describes schema-level information about a specific feature. +// NextID: 31 +type Feature struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // The name of the feature. + Name *string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"` // required + // This field is no longer supported. Instead, use: + // lifecycle_stage: DEPRECATED + // TODO(b/111450258): remove this. + // + // Deprecated: Do not use. + Deprecated *bool `protobuf:"varint,2,opt,name=deprecated" json:"deprecated,omitempty"` + // Types that are assignable to PresenceConstraints: + // *Feature_Presence + // *Feature_GroupPresence + PresenceConstraints isFeature_PresenceConstraints `protobuf_oneof:"presence_constraints"` + // The shape of the feature which governs the number of values that appear in + // each example. + // + // Types that are assignable to ShapeType: + // *Feature_Shape + // *Feature_ValueCount + ShapeType isFeature_ShapeType `protobuf_oneof:"shape_type"` + // Physical type of the feature's values. + // Note that you can have: + // type: BYTES + // int_domain: { + // min: 0 + // max: 3 + // } + // This would be a field that is syntactically BYTES (i.e. strings), but + // semantically an int, i.e. it would be "0", "1", "2", or "3". + Type *FeatureType `protobuf:"varint,6,opt,name=type,enum=tensorflow.metadata.v0.FeatureType" json:"type,omitempty"` + // Domain for the values of the feature. + // + // Types that are assignable to DomainInfo: + // *Feature_Domain + // *Feature_IntDomain + // *Feature_FloatDomain + // *Feature_StringDomain + // *Feature_BoolDomain + // *Feature_StructDomain + // *Feature_NaturalLanguageDomain + // *Feature_ImageDomain + // *Feature_MidDomain + // *Feature_UrlDomain + // *Feature_TimeDomain + // *Feature_TimeOfDayDomain + DomainInfo isFeature_DomainInfo `protobuf_oneof:"domain_info"` + // Constraints on the distribution of the feature values. + // Currently only supported for StringDomains. + // TODO(b/69473628): Extend functionality to other domain types. + DistributionConstraints *DistributionConstraints `protobuf:"bytes,15,opt,name=distribution_constraints,json=distributionConstraints" json:"distribution_constraints,omitempty"` + // Additional information about the feature for documentation purpose. + Annotation *Annotation `protobuf:"bytes,16,opt,name=annotation" json:"annotation,omitempty"` + // Tests comparing the distribution to the associated serving data. + SkewComparator *FeatureComparator `protobuf:"bytes,18,opt,name=skew_comparator,json=skewComparator" json:"skew_comparator,omitempty"` + // Tests comparing the distribution between two consecutive spans (e.g. days). + DriftComparator *FeatureComparator `protobuf:"bytes,21,opt,name=drift_comparator,json=driftComparator" json:"drift_comparator,omitempty"` + // List of environments this feature is present in. + // Should be disjoint from not_in_environment. + // This feature is in environment "foo" if: + // ("foo" is in in_environment or default_environments) AND + // "foo" is not in not_in_environment. + // See Schema::default_environments. + InEnvironment []string `protobuf:"bytes,20,rep,name=in_environment,json=inEnvironment" json:"in_environment,omitempty"` + // List of environments this feature is not present in. + // Should be disjoint from of in_environment. + // See Schema::default_environments and in_environment. + NotInEnvironment []string `protobuf:"bytes,19,rep,name=not_in_environment,json=notInEnvironment" json:"not_in_environment,omitempty"` + // The lifecycle stage of a feature. It can also apply to its descendants. + // i.e., if a struct is DEPRECATED, its children are implicitly deprecated. + LifecycleStage *LifecycleStage `protobuf:"varint,22,opt,name=lifecycle_stage,json=lifecycleStage,enum=tensorflow.metadata.v0.LifecycleStage" json:"lifecycle_stage,omitempty"` +} + +func (x *Feature) Reset() { + *x = Feature{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[1] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *Feature) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*Feature) ProtoMessage() {} + +func (x *Feature) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[1] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use Feature.ProtoReflect.Descriptor instead. +func (*Feature) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{1} +} + +func (x *Feature) GetName() string { + if x != nil && x.Name != nil { + return *x.Name + } + return "" +} + +// Deprecated: Do not use. +func (x *Feature) GetDeprecated() bool { + if x != nil && x.Deprecated != nil { + return *x.Deprecated + } + return false +} + +func (m *Feature) GetPresenceConstraints() isFeature_PresenceConstraints { + if m != nil { + return m.PresenceConstraints + } + return nil +} + +func (x *Feature) GetPresence() *FeaturePresence { + if x, ok := x.GetPresenceConstraints().(*Feature_Presence); ok { + return x.Presence + } + return nil +} + +func (x *Feature) GetGroupPresence() *FeaturePresenceWithinGroup { + if x, ok := x.GetPresenceConstraints().(*Feature_GroupPresence); ok { + return x.GroupPresence + } + return nil +} + +func (m *Feature) GetShapeType() isFeature_ShapeType { + if m != nil { + return m.ShapeType + } + return nil +} + +func (x *Feature) GetShape() *FixedShape { + if x, ok := x.GetShapeType().(*Feature_Shape); ok { + return x.Shape + } + return nil +} + +func (x *Feature) GetValueCount() *ValueCount { + if x, ok := x.GetShapeType().(*Feature_ValueCount); ok { + return x.ValueCount + } + return nil +} + +func (x *Feature) GetType() FeatureType { + if x != nil && x.Type != nil { + return *x.Type + } + return FeatureType_TYPE_UNKNOWN +} + +func (m *Feature) GetDomainInfo() isFeature_DomainInfo { + if m != nil { + return m.DomainInfo + } + return nil +} + +func (x *Feature) GetDomain() string { + if x, ok := x.GetDomainInfo().(*Feature_Domain); ok { + return x.Domain + } + return "" +} + +func (x *Feature) GetIntDomain() *IntDomain { + if x, ok := x.GetDomainInfo().(*Feature_IntDomain); ok { + return x.IntDomain + } + return nil +} + +func (x *Feature) GetFloatDomain() *FloatDomain { + if x, ok := x.GetDomainInfo().(*Feature_FloatDomain); ok { + return x.FloatDomain + } + return nil +} + +func (x *Feature) GetStringDomain() *StringDomain { + if x, ok := x.GetDomainInfo().(*Feature_StringDomain); ok { + return x.StringDomain + } + return nil +} + +func (x *Feature) GetBoolDomain() *BoolDomain { + if x, ok := x.GetDomainInfo().(*Feature_BoolDomain); ok { + return x.BoolDomain + } + return nil +} + +func (x *Feature) GetStructDomain() *StructDomain { + if x, ok := x.GetDomainInfo().(*Feature_StructDomain); ok { + return x.StructDomain + } + return nil +} + +func (x *Feature) GetNaturalLanguageDomain() *NaturalLanguageDomain { + if x, ok := x.GetDomainInfo().(*Feature_NaturalLanguageDomain); ok { + return x.NaturalLanguageDomain + } + return nil +} + +func (x *Feature) GetImageDomain() *ImageDomain { + if x, ok := x.GetDomainInfo().(*Feature_ImageDomain); ok { + return x.ImageDomain + } + return nil +} + +func (x *Feature) GetMidDomain() *MIDDomain { + if x, ok := x.GetDomainInfo().(*Feature_MidDomain); ok { + return x.MidDomain + } + return nil +} + +func (x *Feature) GetUrlDomain() *URLDomain { + if x, ok := x.GetDomainInfo().(*Feature_UrlDomain); ok { + return x.UrlDomain + } + return nil +} + +func (x *Feature) GetTimeDomain() *TimeDomain { + if x, ok := x.GetDomainInfo().(*Feature_TimeDomain); ok { + return x.TimeDomain + } + return nil +} + +func (x *Feature) GetTimeOfDayDomain() *TimeOfDayDomain { + if x, ok := x.GetDomainInfo().(*Feature_TimeOfDayDomain); ok { + return x.TimeOfDayDomain + } + return nil +} + +func (x *Feature) GetDistributionConstraints() *DistributionConstraints { + if x != nil { + return x.DistributionConstraints + } + return nil +} + +func (x *Feature) GetAnnotation() *Annotation { + if x != nil { + return x.Annotation + } + return nil +} + +func (x *Feature) GetSkewComparator() *FeatureComparator { + if x != nil { + return x.SkewComparator + } + return nil +} + +func (x *Feature) GetDriftComparator() *FeatureComparator { + if x != nil { + return x.DriftComparator + } + return nil +} + +func (x *Feature) GetInEnvironment() []string { + if x != nil { + return x.InEnvironment + } + return nil +} + +func (x *Feature) GetNotInEnvironment() []string { + if x != nil { + return x.NotInEnvironment + } + return nil +} + +func (x *Feature) GetLifecycleStage() LifecycleStage { + if x != nil && x.LifecycleStage != nil { + return *x.LifecycleStage + } + return LifecycleStage_UNKNOWN_STAGE +} + +type isFeature_PresenceConstraints interface { + isFeature_PresenceConstraints() +} + +type Feature_Presence struct { + // Constraints on the presence of this feature in the examples. + Presence *FeaturePresence `protobuf:"bytes,14,opt,name=presence,oneof"` +} + +type Feature_GroupPresence struct { + // Only used in the context of a "group" context, e.g., inside a sequence. + GroupPresence *FeaturePresenceWithinGroup `protobuf:"bytes,17,opt,name=group_presence,json=groupPresence,oneof"` +} + +func (*Feature_Presence) isFeature_PresenceConstraints() {} + +func (*Feature_GroupPresence) isFeature_PresenceConstraints() {} + +type isFeature_ShapeType interface { + isFeature_ShapeType() +} + +type Feature_Shape struct { + // The feature has a fixed shape corresponding to a multi-dimensional + // tensor. + Shape *FixedShape `protobuf:"bytes,23,opt,name=shape,oneof"` +} + +type Feature_ValueCount struct { + // The feature doesn't have a well defined shape. All we know are limits on + // the minimum and maximum number of values. + ValueCount *ValueCount `protobuf:"bytes,5,opt,name=value_count,json=valueCount,oneof"` +} + +func (*Feature_Shape) isFeature_ShapeType() {} + +func (*Feature_ValueCount) isFeature_ShapeType() {} + +type isFeature_DomainInfo interface { + isFeature_DomainInfo() +} + +type Feature_Domain struct { + // Reference to a domain defined at the schema level. + Domain string `protobuf:"bytes,7,opt,name=domain,oneof"` +} + +type Feature_IntDomain struct { + // Inline definitions of domains. + IntDomain *IntDomain `protobuf:"bytes,9,opt,name=int_domain,json=intDomain,oneof"` +} + +type Feature_FloatDomain struct { + FloatDomain *FloatDomain `protobuf:"bytes,10,opt,name=float_domain,json=floatDomain,oneof"` +} + +type Feature_StringDomain struct { + StringDomain *StringDomain `protobuf:"bytes,11,opt,name=string_domain,json=stringDomain,oneof"` +} + +type Feature_BoolDomain struct { + BoolDomain *BoolDomain `protobuf:"bytes,13,opt,name=bool_domain,json=boolDomain,oneof"` +} + +type Feature_StructDomain struct { + StructDomain *StructDomain `protobuf:"bytes,29,opt,name=struct_domain,json=structDomain,oneof"` +} + +type Feature_NaturalLanguageDomain struct { + // Supported semantic domains. + NaturalLanguageDomain *NaturalLanguageDomain `protobuf:"bytes,24,opt,name=natural_language_domain,json=naturalLanguageDomain,oneof"` +} + +type Feature_ImageDomain struct { + ImageDomain *ImageDomain `protobuf:"bytes,25,opt,name=image_domain,json=imageDomain,oneof"` +} + +type Feature_MidDomain struct { + MidDomain *MIDDomain `protobuf:"bytes,26,opt,name=mid_domain,json=midDomain,oneof"` +} + +type Feature_UrlDomain struct { + UrlDomain *URLDomain `protobuf:"bytes,27,opt,name=url_domain,json=urlDomain,oneof"` +} + +type Feature_TimeDomain struct { + TimeDomain *TimeDomain `protobuf:"bytes,28,opt,name=time_domain,json=timeDomain,oneof"` +} + +type Feature_TimeOfDayDomain struct { + TimeOfDayDomain *TimeOfDayDomain `protobuf:"bytes,30,opt,name=time_of_day_domain,json=timeOfDayDomain,oneof"` +} + +func (*Feature_Domain) isFeature_DomainInfo() {} + +func (*Feature_IntDomain) isFeature_DomainInfo() {} + +func (*Feature_FloatDomain) isFeature_DomainInfo() {} + +func (*Feature_StringDomain) isFeature_DomainInfo() {} + +func (*Feature_BoolDomain) isFeature_DomainInfo() {} + +func (*Feature_StructDomain) isFeature_DomainInfo() {} + +func (*Feature_NaturalLanguageDomain) isFeature_DomainInfo() {} + +func (*Feature_ImageDomain) isFeature_DomainInfo() {} + +func (*Feature_MidDomain) isFeature_DomainInfo() {} + +func (*Feature_UrlDomain) isFeature_DomainInfo() {} + +func (*Feature_TimeDomain) isFeature_DomainInfo() {} + +func (*Feature_TimeOfDayDomain) isFeature_DomainInfo() {} + +// Additional information about the schema or about a feature. +type Annotation struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Tags can be used to mark features. For example, tag on user_age feature can + // be `user_feature`, tag on user_country feature can be `location_feature`, + // `user_feature`. + Tag []string `protobuf:"bytes,1,rep,name=tag" json:"tag,omitempty"` + // Free-text comments. This can be used as a description of the feature, + // developer notes etc. + Comment []string `protobuf:"bytes,2,rep,name=comment" json:"comment,omitempty"` + // Application-specific metadata may be attached here. + ExtraMetadata []*any.Any `protobuf:"bytes,3,rep,name=extra_metadata,json=extraMetadata" json:"extra_metadata,omitempty"` +} + +func (x *Annotation) Reset() { + *x = Annotation{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[2] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *Annotation) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*Annotation) ProtoMessage() {} + +func (x *Annotation) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[2] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use Annotation.ProtoReflect.Descriptor instead. +func (*Annotation) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{2} +} + +func (x *Annotation) GetTag() []string { + if x != nil { + return x.Tag + } + return nil +} + +func (x *Annotation) GetComment() []string { + if x != nil { + return x.Comment + } + return nil +} + +func (x *Annotation) GetExtraMetadata() []*any.Any { + if x != nil { + return x.ExtraMetadata + } + return nil +} + +// Checks that the ratio of the current value to the previous value is not below +// the min_fraction_threshold or above the max_fraction_threshold. That is, +// previous value * min_fraction_threshold <= current value <= +// previous value * max_fraction_threshold. +// To specify that the value cannot change, set both min_fraction_threshold and +// max_fraction_threshold to 1.0. +type NumericValueComparator struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + MinFractionThreshold *float64 `protobuf:"fixed64,1,opt,name=min_fraction_threshold,json=minFractionThreshold" json:"min_fraction_threshold,omitempty"` + MaxFractionThreshold *float64 `protobuf:"fixed64,2,opt,name=max_fraction_threshold,json=maxFractionThreshold" json:"max_fraction_threshold,omitempty"` +} + +func (x *NumericValueComparator) Reset() { + *x = NumericValueComparator{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[3] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *NumericValueComparator) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*NumericValueComparator) ProtoMessage() {} + +func (x *NumericValueComparator) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[3] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use NumericValueComparator.ProtoReflect.Descriptor instead. +func (*NumericValueComparator) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{3} +} + +func (x *NumericValueComparator) GetMinFractionThreshold() float64 { + if x != nil && x.MinFractionThreshold != nil { + return *x.MinFractionThreshold + } + return 0 +} + +func (x *NumericValueComparator) GetMaxFractionThreshold() float64 { + if x != nil && x.MaxFractionThreshold != nil { + return *x.MaxFractionThreshold + } + return 0 +} + +// Constraints on the entire dataset. +type DatasetConstraints struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Tests differences in number of examples between the current data and the + // previous span. + NumExamplesDriftComparator *NumericValueComparator `protobuf:"bytes,1,opt,name=num_examples_drift_comparator,json=numExamplesDriftComparator" json:"num_examples_drift_comparator,omitempty"` + // Tests comparisions in number of examples between the current data and the + // previous version of that data. + NumExamplesVersionComparator *NumericValueComparator `protobuf:"bytes,2,opt,name=num_examples_version_comparator,json=numExamplesVersionComparator" json:"num_examples_version_comparator,omitempty"` + // Minimum number of examples in the dataset. + MinExamplesCount *int64 `protobuf:"varint,3,opt,name=min_examples_count,json=minExamplesCount" json:"min_examples_count,omitempty"` +} + +func (x *DatasetConstraints) Reset() { + *x = DatasetConstraints{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[4] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *DatasetConstraints) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*DatasetConstraints) ProtoMessage() {} + +func (x *DatasetConstraints) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[4] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use DatasetConstraints.ProtoReflect.Descriptor instead. +func (*DatasetConstraints) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{4} +} + +func (x *DatasetConstraints) GetNumExamplesDriftComparator() *NumericValueComparator { + if x != nil { + return x.NumExamplesDriftComparator + } + return nil +} + +func (x *DatasetConstraints) GetNumExamplesVersionComparator() *NumericValueComparator { + if x != nil { + return x.NumExamplesVersionComparator + } + return nil +} + +func (x *DatasetConstraints) GetMinExamplesCount() int64 { + if x != nil && x.MinExamplesCount != nil { + return *x.MinExamplesCount + } + return 0 +} + +// Specifies a fixed shape for the feature's values. The immediate implication +// is that each feature has a fixed number of values. Moreover, these values +// can be parsed in a multi-dimensional tensor using the specified axis sizes. +// The FixedShape defines a lexicographical ordering of the data. For instance, +// if there is a FixedShape { +// dim {size:3} dim {size:2} +// } +// then tensor[0][0]=field[0] +// then tensor[0][1]=field[1] +// then tensor[1][0]=field[2] +// then tensor[1][1]=field[3] +// then tensor[2][0]=field[4] +// then tensor[2][1]=field[5] +// +// The FixedShape message is identical with the TensorFlow TensorShape proto +// message. +type FixedShape struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // The dimensions that define the shape. The total number of values in each + // example is the product of sizes of each dimension. + Dim []*FixedShape_Dim `protobuf:"bytes,2,rep,name=dim" json:"dim,omitempty"` +} + +func (x *FixedShape) Reset() { + *x = FixedShape{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[5] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *FixedShape) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*FixedShape) ProtoMessage() {} + +func (x *FixedShape) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[5] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use FixedShape.ProtoReflect.Descriptor instead. +func (*FixedShape) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{5} +} + +func (x *FixedShape) GetDim() []*FixedShape_Dim { + if x != nil { + return x.Dim + } + return nil +} + +// Limits on maximum and minimum number of values in a +// single example (when the feature is present). Use this when the minimum +// value count can be different than the maximum value count. Otherwise prefer +// FixedShape. +type ValueCount struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Min *int64 `protobuf:"varint,1,opt,name=min" json:"min,omitempty"` + Max *int64 `protobuf:"varint,2,opt,name=max" json:"max,omitempty"` +} + +func (x *ValueCount) Reset() { + *x = ValueCount{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[6] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ValueCount) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ValueCount) ProtoMessage() {} + +func (x *ValueCount) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[6] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ValueCount.ProtoReflect.Descriptor instead. +func (*ValueCount) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{6} +} + +func (x *ValueCount) GetMin() int64 { + if x != nil && x.Min != nil { + return *x.Min + } + return 0 +} + +func (x *ValueCount) GetMax() int64 { + if x != nil && x.Max != nil { + return *x.Max + } + return 0 +} + +// Represents a weighted feature that is encoded as a combination of raw base +// features. The `weight_feature` should be a float feature with identical +// shape as the `feature`. This is useful for representing weights associated +// with categorical tokens (e.g. a TFIDF weight associated with each token). +// TODO(b/142122960): Handle WeightedCategorical end to end in TFX (validation, +// TFX Unit Testing, etc) +type WeightedFeature struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Name for the weighted feature. This should not clash with other features in + // the same schema. + Name *string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"` // required + // Path of a base feature to be weighted. Required. + Feature *Path `protobuf:"bytes,2,opt,name=feature" json:"feature,omitempty"` + // Path of weight feature to associate with the base feature. Must be same + // shape as feature. Required. + WeightFeature *Path `protobuf:"bytes,3,opt,name=weight_feature,json=weightFeature" json:"weight_feature,omitempty"` + // The lifecycle_stage determines where a feature is expected to be used, + // and therefore how important issues with it are. + LifecycleStage *LifecycleStage `protobuf:"varint,4,opt,name=lifecycle_stage,json=lifecycleStage,enum=tensorflow.metadata.v0.LifecycleStage" json:"lifecycle_stage,omitempty"` +} + +func (x *WeightedFeature) Reset() { + *x = WeightedFeature{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[7] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *WeightedFeature) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*WeightedFeature) ProtoMessage() {} + +func (x *WeightedFeature) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[7] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use WeightedFeature.ProtoReflect.Descriptor instead. +func (*WeightedFeature) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{7} +} + +func (x *WeightedFeature) GetName() string { + if x != nil && x.Name != nil { + return *x.Name + } + return "" +} + +func (x *WeightedFeature) GetFeature() *Path { + if x != nil { + return x.Feature + } + return nil +} + +func (x *WeightedFeature) GetWeightFeature() *Path { + if x != nil { + return x.WeightFeature + } + return nil +} + +func (x *WeightedFeature) GetLifecycleStage() LifecycleStage { + if x != nil && x.LifecycleStage != nil { + return *x.LifecycleStage + } + return LifecycleStage_UNKNOWN_STAGE +} + +// A sparse feature represents a sparse tensor that is encoded with a +// combination of raw features, namely index features and a value feature. Each +// index feature defines a list of indices in a different dimension. +type SparseFeature struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Name for the sparse feature. This should not clash with other features in + // the same schema. + Name *string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"` // required + // This field is no longer supported. Instead, use: + // lifecycle_stage: DEPRECATED + // TODO(b/111450258): remove this. + // + // Deprecated: Do not use. + Deprecated *bool `protobuf:"varint,2,opt,name=deprecated" json:"deprecated,omitempty"` + // The lifecycle_stage determines where a feature is expected to be used, + // and therefore how important issues with it are. + LifecycleStage *LifecycleStage `protobuf:"varint,7,opt,name=lifecycle_stage,json=lifecycleStage,enum=tensorflow.metadata.v0.LifecycleStage" json:"lifecycle_stage,omitempty"` + // Constraints on the presence of this feature in examples. + // Deprecated, this is inferred by the referred features. + // + // Deprecated: Do not use. + Presence *FeaturePresence `protobuf:"bytes,4,opt,name=presence" json:"presence,omitempty"` + // Shape of the sparse tensor that this SparseFeature represents. + // Currently not supported. + // TODO(b/109669962): Consider deriving this from the referred features. + DenseShape *FixedShape `protobuf:"bytes,5,opt,name=dense_shape,json=denseShape" json:"dense_shape,omitempty"` + // Features that represent indexes. Should be integers >= 0. + IndexFeature []*SparseFeature_IndexFeature `protobuf:"bytes,6,rep,name=index_feature,json=indexFeature" json:"index_feature,omitempty"` // at least one + // If true then the index values are already sorted lexicographically. + IsSorted *bool `protobuf:"varint,8,opt,name=is_sorted,json=isSorted" json:"is_sorted,omitempty"` + ValueFeature *SparseFeature_ValueFeature `protobuf:"bytes,9,opt,name=value_feature,json=valueFeature" json:"value_feature,omitempty"` // required + // Type of value feature. + // Deprecated, this is inferred by the referred features. + // + // Deprecated: Do not use. + Type *FeatureType `protobuf:"varint,10,opt,name=type,enum=tensorflow.metadata.v0.FeatureType" json:"type,omitempty"` +} + +func (x *SparseFeature) Reset() { + *x = SparseFeature{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[8] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *SparseFeature) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*SparseFeature) ProtoMessage() {} + +func (x *SparseFeature) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[8] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use SparseFeature.ProtoReflect.Descriptor instead. +func (*SparseFeature) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{8} +} + +func (x *SparseFeature) GetName() string { + if x != nil && x.Name != nil { + return *x.Name + } + return "" +} + +// Deprecated: Do not use. +func (x *SparseFeature) GetDeprecated() bool { + if x != nil && x.Deprecated != nil { + return *x.Deprecated + } + return false +} + +func (x *SparseFeature) GetLifecycleStage() LifecycleStage { + if x != nil && x.LifecycleStage != nil { + return *x.LifecycleStage + } + return LifecycleStage_UNKNOWN_STAGE +} + +// Deprecated: Do not use. +func (x *SparseFeature) GetPresence() *FeaturePresence { + if x != nil { + return x.Presence + } + return nil +} + +func (x *SparseFeature) GetDenseShape() *FixedShape { + if x != nil { + return x.DenseShape + } + return nil +} + +func (x *SparseFeature) GetIndexFeature() []*SparseFeature_IndexFeature { + if x != nil { + return x.IndexFeature + } + return nil +} + +func (x *SparseFeature) GetIsSorted() bool { + if x != nil && x.IsSorted != nil { + return *x.IsSorted + } + return false +} + +func (x *SparseFeature) GetValueFeature() *SparseFeature_ValueFeature { + if x != nil { + return x.ValueFeature + } + return nil +} + +// Deprecated: Do not use. +func (x *SparseFeature) GetType() FeatureType { + if x != nil && x.Type != nil { + return *x.Type + } + return FeatureType_TYPE_UNKNOWN +} + +// Models constraints on the distribution of a feature's values. +// TODO(martinz): replace min_domain_mass with max_off_domain (but slowly). +type DistributionConstraints struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // The minimum fraction (in [0,1]) of values across all examples that + // should come from the feature's domain, e.g.: + // 1.0 => All values must come from the domain. + // .9 => At least 90% of the values must come from the domain. + MinDomainMass *float64 `protobuf:"fixed64,1,opt,name=min_domain_mass,json=minDomainMass,def=1" json:"min_domain_mass,omitempty"` +} + +// Default values for DistributionConstraints fields. +const ( + Default_DistributionConstraints_MinDomainMass = float64(1) +) + +func (x *DistributionConstraints) Reset() { + *x = DistributionConstraints{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[9] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *DistributionConstraints) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*DistributionConstraints) ProtoMessage() {} + +func (x *DistributionConstraints) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[9] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use DistributionConstraints.ProtoReflect.Descriptor instead. +func (*DistributionConstraints) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{9} +} + +func (x *DistributionConstraints) GetMinDomainMass() float64 { + if x != nil && x.MinDomainMass != nil { + return *x.MinDomainMass + } + return Default_DistributionConstraints_MinDomainMass +} + +// Encodes information for domains of integer values. +// Note that FeatureType could be either INT or BYTES. +type IntDomain struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Id of the domain. Required if the domain is defined at the schema level. If + // so, then the name must be unique within the schema. + Name *string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"` + // Min and max values for the domain. + Min *int64 `protobuf:"varint,3,opt,name=min" json:"min,omitempty"` + Max *int64 `protobuf:"varint,4,opt,name=max" json:"max,omitempty"` + // If true then the domain encodes categorical values (i.e., ids) rather than + // ordinal values. + IsCategorical *bool `protobuf:"varint,5,opt,name=is_categorical,json=isCategorical" json:"is_categorical,omitempty"` +} + +func (x *IntDomain) Reset() { + *x = IntDomain{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[10] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *IntDomain) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*IntDomain) ProtoMessage() {} + +func (x *IntDomain) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[10] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use IntDomain.ProtoReflect.Descriptor instead. +func (*IntDomain) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{10} +} + +func (x *IntDomain) GetName() string { + if x != nil && x.Name != nil { + return *x.Name + } + return "" +} + +func (x *IntDomain) GetMin() int64 { + if x != nil && x.Min != nil { + return *x.Min + } + return 0 +} + +func (x *IntDomain) GetMax() int64 { + if x != nil && x.Max != nil { + return *x.Max + } + return 0 +} + +func (x *IntDomain) GetIsCategorical() bool { + if x != nil && x.IsCategorical != nil { + return *x.IsCategorical + } + return false +} + +// Encodes information for domains of float values. +// Note that FeatureType could be either INT or BYTES. +type FloatDomain struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Id of the domain. Required if the domain is defined at the schema level. If + // so, then the name must be unique within the schema. + Name *string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"` + // Min and max values of the domain. + Min *float32 `protobuf:"fixed32,3,opt,name=min" json:"min,omitempty"` + Max *float32 `protobuf:"fixed32,4,opt,name=max" json:"max,omitempty"` +} + +func (x *FloatDomain) Reset() { + *x = FloatDomain{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[11] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *FloatDomain) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*FloatDomain) ProtoMessage() {} + +func (x *FloatDomain) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[11] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use FloatDomain.ProtoReflect.Descriptor instead. +func (*FloatDomain) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{11} +} + +func (x *FloatDomain) GetName() string { + if x != nil && x.Name != nil { + return *x.Name + } + return "" +} + +func (x *FloatDomain) GetMin() float32 { + if x != nil && x.Min != nil { + return *x.Min + } + return 0 +} + +func (x *FloatDomain) GetMax() float32 { + if x != nil && x.Max != nil { + return *x.Max + } + return 0 +} + +// Domain for a recursive struct. +// NOTE: If a feature with a StructDomain is deprecated, then all the +// child features (features and sparse_features of the StructDomain) are also +// considered to be deprecated. Similarly child features can only be in +// environments of the parent feature. +type StructDomain struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Feature []*Feature `protobuf:"bytes,1,rep,name=feature" json:"feature,omitempty"` + SparseFeature []*SparseFeature `protobuf:"bytes,2,rep,name=sparse_feature,json=sparseFeature" json:"sparse_feature,omitempty"` +} + +func (x *StructDomain) Reset() { + *x = StructDomain{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[12] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *StructDomain) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*StructDomain) ProtoMessage() {} + +func (x *StructDomain) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[12] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use StructDomain.ProtoReflect.Descriptor instead. +func (*StructDomain) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{12} +} + +func (x *StructDomain) GetFeature() []*Feature { + if x != nil { + return x.Feature + } + return nil +} + +func (x *StructDomain) GetSparseFeature() []*SparseFeature { + if x != nil { + return x.SparseFeature + } + return nil +} + +// Encodes information for domains of string values. +type StringDomain struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Id of the domain. Required if the domain is defined at the schema level. If + // so, then the name must be unique within the schema. + Name *string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"` + // The values appearing in the domain. + Value []string `protobuf:"bytes,2,rep,name=value" json:"value,omitempty"` +} + +func (x *StringDomain) Reset() { + *x = StringDomain{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[13] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *StringDomain) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*StringDomain) ProtoMessage() {} + +func (x *StringDomain) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[13] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use StringDomain.ProtoReflect.Descriptor instead. +func (*StringDomain) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{13} +} + +func (x *StringDomain) GetName() string { + if x != nil && x.Name != nil { + return *x.Name + } + return "" +} + +func (x *StringDomain) GetValue() []string { + if x != nil { + return x.Value + } + return nil +} + +// Encodes information about the domain of a boolean attribute that encodes its +// TRUE/FALSE values as strings, or 0=false, 1=true. +// Note that FeatureType could be either INT or BYTES. +type BoolDomain struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Id of the domain. Required if the domain is defined at the schema level. If + // so, then the name must be unique within the schema. + Name *string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"` + // Strings values for TRUE/FALSE. + TrueValue *string `protobuf:"bytes,2,opt,name=true_value,json=trueValue" json:"true_value,omitempty"` + FalseValue *string `protobuf:"bytes,3,opt,name=false_value,json=falseValue" json:"false_value,omitempty"` +} + +func (x *BoolDomain) Reset() { + *x = BoolDomain{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[14] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *BoolDomain) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*BoolDomain) ProtoMessage() {} + +func (x *BoolDomain) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[14] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use BoolDomain.ProtoReflect.Descriptor instead. +func (*BoolDomain) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{14} +} + +func (x *BoolDomain) GetName() string { + if x != nil && x.Name != nil { + return *x.Name + } + return "" +} + +func (x *BoolDomain) GetTrueValue() string { + if x != nil && x.TrueValue != nil { + return *x.TrueValue + } + return "" +} + +func (x *BoolDomain) GetFalseValue() string { + if x != nil && x.FalseValue != nil { + return *x.FalseValue + } + return "" +} + +// Natural language text. +type NaturalLanguageDomain struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields +} + +func (x *NaturalLanguageDomain) Reset() { + *x = NaturalLanguageDomain{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[15] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *NaturalLanguageDomain) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*NaturalLanguageDomain) ProtoMessage() {} + +func (x *NaturalLanguageDomain) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[15] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use NaturalLanguageDomain.ProtoReflect.Descriptor instead. +func (*NaturalLanguageDomain) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{15} +} + +// Image data. +type ImageDomain struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields +} + +func (x *ImageDomain) Reset() { + *x = ImageDomain{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[16] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *ImageDomain) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*ImageDomain) ProtoMessage() {} + +func (x *ImageDomain) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[16] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use ImageDomain.ProtoReflect.Descriptor instead. +func (*ImageDomain) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{16} +} + +// Knowledge graph ID, see: https://www.wikidata.org/wiki/Property:P646 +type MIDDomain struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields +} + +func (x *MIDDomain) Reset() { + *x = MIDDomain{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[17] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *MIDDomain) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*MIDDomain) ProtoMessage() {} + +func (x *MIDDomain) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[17] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use MIDDomain.ProtoReflect.Descriptor instead. +func (*MIDDomain) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{17} +} + +// A URL, see: https://en.wikipedia.org/wiki/URL +type URLDomain struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields +} + +func (x *URLDomain) Reset() { + *x = URLDomain{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[18] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *URLDomain) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*URLDomain) ProtoMessage() {} + +func (x *URLDomain) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[18] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use URLDomain.ProtoReflect.Descriptor instead. +func (*URLDomain) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{18} +} + +// Time or date representation. +type TimeDomain struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Types that are assignable to Format: + // *TimeDomain_StringFormat + // *TimeDomain_IntegerFormat + Format isTimeDomain_Format `protobuf_oneof:"format"` +} + +func (x *TimeDomain) Reset() { + *x = TimeDomain{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[19] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *TimeDomain) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*TimeDomain) ProtoMessage() {} + +func (x *TimeDomain) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[19] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use TimeDomain.ProtoReflect.Descriptor instead. +func (*TimeDomain) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{19} +} + +func (m *TimeDomain) GetFormat() isTimeDomain_Format { + if m != nil { + return m.Format + } + return nil +} + +func (x *TimeDomain) GetStringFormat() string { + if x, ok := x.GetFormat().(*TimeDomain_StringFormat); ok { + return x.StringFormat + } + return "" +} + +func (x *TimeDomain) GetIntegerFormat() TimeDomain_IntegerTimeFormat { + if x, ok := x.GetFormat().(*TimeDomain_IntegerFormat); ok { + return x.IntegerFormat + } + return TimeDomain_FORMAT_UNKNOWN +} + +type isTimeDomain_Format interface { + isTimeDomain_Format() +} + +type TimeDomain_StringFormat struct { + // Expected format that contains a combination of regular characters and + // special format specifiers. Format specifiers are a subset of the + // strptime standard. + StringFormat string `protobuf:"bytes,1,opt,name=string_format,json=stringFormat,oneof"` +} + +type TimeDomain_IntegerFormat struct { + // Expected format of integer times. + IntegerFormat TimeDomain_IntegerTimeFormat `protobuf:"varint,2,opt,name=integer_format,json=integerFormat,enum=tensorflow.metadata.v0.TimeDomain_IntegerTimeFormat,oneof"` +} + +func (*TimeDomain_StringFormat) isTimeDomain_Format() {} + +func (*TimeDomain_IntegerFormat) isTimeDomain_Format() {} + +// Time of day, without a particular date. +type TimeOfDayDomain struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Types that are assignable to Format: + // *TimeOfDayDomain_StringFormat + // *TimeOfDayDomain_IntegerFormat + Format isTimeOfDayDomain_Format `protobuf_oneof:"format"` +} + +func (x *TimeOfDayDomain) Reset() { + *x = TimeOfDayDomain{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[20] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *TimeOfDayDomain) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*TimeOfDayDomain) ProtoMessage() {} + +func (x *TimeOfDayDomain) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[20] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use TimeOfDayDomain.ProtoReflect.Descriptor instead. +func (*TimeOfDayDomain) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{20} +} + +func (m *TimeOfDayDomain) GetFormat() isTimeOfDayDomain_Format { + if m != nil { + return m.Format + } + return nil +} + +func (x *TimeOfDayDomain) GetStringFormat() string { + if x, ok := x.GetFormat().(*TimeOfDayDomain_StringFormat); ok { + return x.StringFormat + } + return "" +} + +func (x *TimeOfDayDomain) GetIntegerFormat() TimeOfDayDomain_IntegerTimeOfDayFormat { + if x, ok := x.GetFormat().(*TimeOfDayDomain_IntegerFormat); ok { + return x.IntegerFormat + } + return TimeOfDayDomain_FORMAT_UNKNOWN +} + +type isTimeOfDayDomain_Format interface { + isTimeOfDayDomain_Format() +} + +type TimeOfDayDomain_StringFormat struct { + // Expected format that contains a combination of regular characters and + // special format specifiers. Format specifiers are a subset of the + // strptime standard. + StringFormat string `protobuf:"bytes,1,opt,name=string_format,json=stringFormat,oneof"` +} + +type TimeOfDayDomain_IntegerFormat struct { + // Expected format of integer times. + IntegerFormat TimeOfDayDomain_IntegerTimeOfDayFormat `protobuf:"varint,2,opt,name=integer_format,json=integerFormat,enum=tensorflow.metadata.v0.TimeOfDayDomain_IntegerTimeOfDayFormat,oneof"` +} + +func (*TimeOfDayDomain_StringFormat) isTimeOfDayDomain_Format() {} + +func (*TimeOfDayDomain_IntegerFormat) isTimeOfDayDomain_Format() {} + +// Describes constraints on the presence of the feature in the data. +type FeaturePresence struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Minimum fraction of examples that have this feature. + MinFraction *float64 `protobuf:"fixed64,1,opt,name=min_fraction,json=minFraction" json:"min_fraction,omitempty"` + // Minimum number of examples that have this feature. + MinCount *int64 `protobuf:"varint,2,opt,name=min_count,json=minCount" json:"min_count,omitempty"` +} + +func (x *FeaturePresence) Reset() { + *x = FeaturePresence{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[21] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *FeaturePresence) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*FeaturePresence) ProtoMessage() {} + +func (x *FeaturePresence) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[21] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use FeaturePresence.ProtoReflect.Descriptor instead. +func (*FeaturePresence) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{21} +} + +func (x *FeaturePresence) GetMinFraction() float64 { + if x != nil && x.MinFraction != nil { + return *x.MinFraction + } + return 0 +} + +func (x *FeaturePresence) GetMinCount() int64 { + if x != nil && x.MinCount != nil { + return *x.MinCount + } + return 0 +} + +// Records constraints on the presence of a feature inside a "group" context +// (e.g., .presence inside a group of features that define a sequence). +type FeaturePresenceWithinGroup struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Required *bool `protobuf:"varint,1,opt,name=required" json:"required,omitempty"` +} + +func (x *FeaturePresenceWithinGroup) Reset() { + *x = FeaturePresenceWithinGroup{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[22] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *FeaturePresenceWithinGroup) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*FeaturePresenceWithinGroup) ProtoMessage() {} + +func (x *FeaturePresenceWithinGroup) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[22] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use FeaturePresenceWithinGroup.ProtoReflect.Descriptor instead. +func (*FeaturePresenceWithinGroup) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{22} +} + +func (x *FeaturePresenceWithinGroup) GetRequired() bool { + if x != nil && x.Required != nil { + return *x.Required + } + return false +} + +// Checks that the L-infinity norm is below a certain threshold between the +// two discrete distributions. Since this is applied to a FeatureNameStatistics, +// it only considers the top k. +// L_infty(p,q) = max_i |p_i-q_i| +type InfinityNorm struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // The InfinityNorm is in the interval [0.0, 1.0] so sensible bounds should + // be in the interval [0.0, 1.0). + Threshold *float64 `protobuf:"fixed64,1,opt,name=threshold" json:"threshold,omitempty"` +} + +func (x *InfinityNorm) Reset() { + *x = InfinityNorm{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[23] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *InfinityNorm) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*InfinityNorm) ProtoMessage() {} + +func (x *InfinityNorm) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[23] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use InfinityNorm.ProtoReflect.Descriptor instead. +func (*InfinityNorm) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{23} +} + +func (x *InfinityNorm) GetThreshold() float64 { + if x != nil && x.Threshold != nil { + return *x.Threshold + } + return 0 +} + +type FeatureComparator struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + InfinityNorm *InfinityNorm `protobuf:"bytes,1,opt,name=infinity_norm,json=infinityNorm" json:"infinity_norm,omitempty"` +} + +func (x *FeatureComparator) Reset() { + *x = FeatureComparator{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[24] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *FeatureComparator) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*FeatureComparator) ProtoMessage() {} + +func (x *FeatureComparator) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[24] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use FeatureComparator.ProtoReflect.Descriptor instead. +func (*FeatureComparator) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{24} +} + +func (x *FeatureComparator) GetInfinityNorm() *InfinityNorm { + if x != nil { + return x.InfinityNorm + } + return nil +} + +// A TensorRepresentation captures the intent for converting columns in a +// dataset to TensorFlow Tensors (or more generally, tf.CompositeTensors). +// Note that one tf.CompositeTensor may consist of data from multiple columns, +// for example, a N-dimensional tf.SparseTensor may need N + 1 columns to +// provide the sparse indices and values. +// Note that the "column name" that a TensorRepresentation needs is a +// string, not a Path -- it means that the column name identifies a top-level +// Feature in the schema (i.e. you cannot specify a Feature nested in a STRUCT +// Feature). +type TensorRepresentation struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Types that are assignable to Kind: + // *TensorRepresentation_DenseTensor_ + // *TensorRepresentation_VarlenSparseTensor + // *TensorRepresentation_SparseTensor_ + Kind isTensorRepresentation_Kind `protobuf_oneof:"kind"` +} + +func (x *TensorRepresentation) Reset() { + *x = TensorRepresentation{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[25] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *TensorRepresentation) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*TensorRepresentation) ProtoMessage() {} + +func (x *TensorRepresentation) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[25] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use TensorRepresentation.ProtoReflect.Descriptor instead. +func (*TensorRepresentation) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{25} +} + +func (m *TensorRepresentation) GetKind() isTensorRepresentation_Kind { + if m != nil { + return m.Kind + } + return nil +} + +func (x *TensorRepresentation) GetDenseTensor() *TensorRepresentation_DenseTensor { + if x, ok := x.GetKind().(*TensorRepresentation_DenseTensor_); ok { + return x.DenseTensor + } + return nil +} + +func (x *TensorRepresentation) GetVarlenSparseTensor() *TensorRepresentation_VarLenSparseTensor { + if x, ok := x.GetKind().(*TensorRepresentation_VarlenSparseTensor); ok { + return x.VarlenSparseTensor + } + return nil +} + +func (x *TensorRepresentation) GetSparseTensor() *TensorRepresentation_SparseTensor { + if x, ok := x.GetKind().(*TensorRepresentation_SparseTensor_); ok { + return x.SparseTensor + } + return nil +} + +type isTensorRepresentation_Kind interface { + isTensorRepresentation_Kind() +} + +type TensorRepresentation_DenseTensor_ struct { + DenseTensor *TensorRepresentation_DenseTensor `protobuf:"bytes,1,opt,name=dense_tensor,json=denseTensor,oneof"` +} + +type TensorRepresentation_VarlenSparseTensor struct { + VarlenSparseTensor *TensorRepresentation_VarLenSparseTensor `protobuf:"bytes,2,opt,name=varlen_sparse_tensor,json=varlenSparseTensor,oneof"` +} + +type TensorRepresentation_SparseTensor_ struct { + SparseTensor *TensorRepresentation_SparseTensor `protobuf:"bytes,3,opt,name=sparse_tensor,json=sparseTensor,oneof"` +} + +func (*TensorRepresentation_DenseTensor_) isTensorRepresentation_Kind() {} + +func (*TensorRepresentation_VarlenSparseTensor) isTensorRepresentation_Kind() {} + +func (*TensorRepresentation_SparseTensor_) isTensorRepresentation_Kind() {} + +// A TensorRepresentationGroup is a collection of TensorRepresentations with +// names. These names may serve as identifiers when converting the dataset +// to a collection of Tensors or tf.CompositeTensors. +// For example, given the following group: +// { +// key: "dense_tensor" +// tensor_representation { +// dense_tensor { +// column_name: "univalent_feature" +// shape { +// dim { +// size: 1 +// } +// } +// default_value { +// float_value: 0 +// } +// } +// } +// } +// { +// key: "varlen_sparse_tensor" +// tensor_representation { +// varlen_sparse_tensor { +// column_name: "multivalent_feature" +// } +// } +// } +// +// Then the schema is expected to have feature "univalent_feature" and +// "multivalent_feature", and when a batch of data is converted to Tensors using +// this TensorRepresentationGroup, the result may be the following dict: +// { +// "dense_tensor": tf.Tensor(...), +// "varlen_sparse_tensor": tf.SparseTensor(...), +// } +type TensorRepresentationGroup struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + TensorRepresentation map[string]*TensorRepresentation `protobuf:"bytes,1,rep,name=tensor_representation,json=tensorRepresentation" json:"tensor_representation,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` +} + +func (x *TensorRepresentationGroup) Reset() { + *x = TensorRepresentationGroup{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[26] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *TensorRepresentationGroup) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*TensorRepresentationGroup) ProtoMessage() {} + +func (x *TensorRepresentationGroup) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[26] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use TensorRepresentationGroup.ProtoReflect.Descriptor instead. +func (*TensorRepresentationGroup) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{26} +} + +func (x *TensorRepresentationGroup) GetTensorRepresentation() map[string]*TensorRepresentation { + if x != nil { + return x.TensorRepresentation + } + return nil +} + +// An axis in a multi-dimensional feature representation. +type FixedShape_Dim struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + Size *int64 `protobuf:"varint,1,opt,name=size" json:"size,omitempty"` + // Optional name of the tensor dimension. + Name *string `protobuf:"bytes,2,opt,name=name" json:"name,omitempty"` +} + +func (x *FixedShape_Dim) Reset() { + *x = FixedShape_Dim{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[28] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *FixedShape_Dim) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*FixedShape_Dim) ProtoMessage() {} + +func (x *FixedShape_Dim) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[28] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use FixedShape_Dim.ProtoReflect.Descriptor instead. +func (*FixedShape_Dim) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{5, 0} +} + +func (x *FixedShape_Dim) GetSize() int64 { + if x != nil && x.Size != nil { + return *x.Size + } + return 0 +} + +func (x *FixedShape_Dim) GetName() string { + if x != nil && x.Name != nil { + return *x.Name + } + return "" +} + +type SparseFeature_IndexFeature struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Name of the index-feature. This should be a reference to an existing + // feature in the schema. + Name *string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"` +} + +func (x *SparseFeature_IndexFeature) Reset() { + *x = SparseFeature_IndexFeature{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[29] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *SparseFeature_IndexFeature) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*SparseFeature_IndexFeature) ProtoMessage() {} + +func (x *SparseFeature_IndexFeature) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[29] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use SparseFeature_IndexFeature.ProtoReflect.Descriptor instead. +func (*SparseFeature_IndexFeature) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{8, 0} +} + +func (x *SparseFeature_IndexFeature) GetName() string { + if x != nil && x.Name != nil { + return *x.Name + } + return "" +} + +type SparseFeature_ValueFeature struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Name of the value-feature. This should be a reference to an existing + // feature in the schema. + Name *string `protobuf:"bytes,1,opt,name=name" json:"name,omitempty"` +} + +func (x *SparseFeature_ValueFeature) Reset() { + *x = SparseFeature_ValueFeature{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[30] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *SparseFeature_ValueFeature) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*SparseFeature_ValueFeature) ProtoMessage() {} + +func (x *SparseFeature_ValueFeature) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[30] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use SparseFeature_ValueFeature.ProtoReflect.Descriptor instead. +func (*SparseFeature_ValueFeature) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{8, 1} +} + +func (x *SparseFeature_ValueFeature) GetName() string { + if x != nil && x.Name != nil { + return *x.Name + } + return "" +} + +type TensorRepresentation_DefaultValue struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Types that are assignable to Kind: + // *TensorRepresentation_DefaultValue_FloatValue + // *TensorRepresentation_DefaultValue_IntValue + // *TensorRepresentation_DefaultValue_BytesValue + // *TensorRepresentation_DefaultValue_UintValue + Kind isTensorRepresentation_DefaultValue_Kind `protobuf_oneof:"kind"` +} + +func (x *TensorRepresentation_DefaultValue) Reset() { + *x = TensorRepresentation_DefaultValue{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[31] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *TensorRepresentation_DefaultValue) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*TensorRepresentation_DefaultValue) ProtoMessage() {} + +func (x *TensorRepresentation_DefaultValue) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[31] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use TensorRepresentation_DefaultValue.ProtoReflect.Descriptor instead. +func (*TensorRepresentation_DefaultValue) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{25, 0} +} + +func (m *TensorRepresentation_DefaultValue) GetKind() isTensorRepresentation_DefaultValue_Kind { + if m != nil { + return m.Kind + } + return nil +} + +func (x *TensorRepresentation_DefaultValue) GetFloatValue() float64 { + if x, ok := x.GetKind().(*TensorRepresentation_DefaultValue_FloatValue); ok { + return x.FloatValue + } + return 0 +} + +func (x *TensorRepresentation_DefaultValue) GetIntValue() int64 { + if x, ok := x.GetKind().(*TensorRepresentation_DefaultValue_IntValue); ok { + return x.IntValue + } + return 0 +} + +func (x *TensorRepresentation_DefaultValue) GetBytesValue() []byte { + if x, ok := x.GetKind().(*TensorRepresentation_DefaultValue_BytesValue); ok { + return x.BytesValue + } + return nil +} + +func (x *TensorRepresentation_DefaultValue) GetUintValue() uint64 { + if x, ok := x.GetKind().(*TensorRepresentation_DefaultValue_UintValue); ok { + return x.UintValue + } + return 0 +} + +type isTensorRepresentation_DefaultValue_Kind interface { + isTensorRepresentation_DefaultValue_Kind() +} + +type TensorRepresentation_DefaultValue_FloatValue struct { + FloatValue float64 `protobuf:"fixed64,1,opt,name=float_value,json=floatValue,oneof"` +} + +type TensorRepresentation_DefaultValue_IntValue struct { + // Note that the data column might be of a shorter integral type. It's the + // user's responsitiblity to make sure the default value fits that type. + IntValue int64 `protobuf:"varint,2,opt,name=int_value,json=intValue,oneof"` +} + +type TensorRepresentation_DefaultValue_BytesValue struct { + BytesValue []byte `protobuf:"bytes,3,opt,name=bytes_value,json=bytesValue,oneof"` +} + +type TensorRepresentation_DefaultValue_UintValue struct { + // uint_value should only be used if the default value can't fit in a + // int64 (`int_value`). + UintValue uint64 `protobuf:"varint,4,opt,name=uint_value,json=uintValue,oneof"` +} + +func (*TensorRepresentation_DefaultValue_FloatValue) isTensorRepresentation_DefaultValue_Kind() {} + +func (*TensorRepresentation_DefaultValue_IntValue) isTensorRepresentation_DefaultValue_Kind() {} + +func (*TensorRepresentation_DefaultValue_BytesValue) isTensorRepresentation_DefaultValue_Kind() {} + +func (*TensorRepresentation_DefaultValue_UintValue) isTensorRepresentation_DefaultValue_Kind() {} + +// A tf.Tensor +type TensorRepresentation_DenseTensor struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Identifies the column in the dataset that provides the values of this + // Tensor. + ColumnName *string `protobuf:"bytes,1,opt,name=column_name,json=columnName" json:"column_name,omitempty"` + // The shape of each row of the data (i.e. does not include the batch + // dimension) + Shape *FixedShape `protobuf:"bytes,2,opt,name=shape" json:"shape,omitempty"` + // If this column is missing values in a row, the default_value will be + // used to fill that row. + DefaultValue *TensorRepresentation_DefaultValue `protobuf:"bytes,3,opt,name=default_value,json=defaultValue" json:"default_value,omitempty"` +} + +func (x *TensorRepresentation_DenseTensor) Reset() { + *x = TensorRepresentation_DenseTensor{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[32] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *TensorRepresentation_DenseTensor) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*TensorRepresentation_DenseTensor) ProtoMessage() {} + +func (x *TensorRepresentation_DenseTensor) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[32] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use TensorRepresentation_DenseTensor.ProtoReflect.Descriptor instead. +func (*TensorRepresentation_DenseTensor) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{25, 1} +} + +func (x *TensorRepresentation_DenseTensor) GetColumnName() string { + if x != nil && x.ColumnName != nil { + return *x.ColumnName + } + return "" +} + +func (x *TensorRepresentation_DenseTensor) GetShape() *FixedShape { + if x != nil { + return x.Shape + } + return nil +} + +func (x *TensorRepresentation_DenseTensor) GetDefaultValue() *TensorRepresentation_DefaultValue { + if x != nil { + return x.DefaultValue + } + return nil +} + +// A ragged tf.SparseTensor that models nested lists. +type TensorRepresentation_VarLenSparseTensor struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Identifies the column in the dataset that should be converted to the + // VarLenSparseTensor. + ColumnName *string `protobuf:"bytes,1,opt,name=column_name,json=columnName" json:"column_name,omitempty"` +} + +func (x *TensorRepresentation_VarLenSparseTensor) Reset() { + *x = TensorRepresentation_VarLenSparseTensor{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[33] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *TensorRepresentation_VarLenSparseTensor) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*TensorRepresentation_VarLenSparseTensor) ProtoMessage() {} + +func (x *TensorRepresentation_VarLenSparseTensor) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[33] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use TensorRepresentation_VarLenSparseTensor.ProtoReflect.Descriptor instead. +func (*TensorRepresentation_VarLenSparseTensor) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{25, 2} +} + +func (x *TensorRepresentation_VarLenSparseTensor) GetColumnName() string { + if x != nil && x.ColumnName != nil { + return *x.ColumnName + } + return "" +} + +// A tf.SparseTensor whose indices and values come from separate data columns. +// This will replace Schema.sparse_feature eventually. +// The index columns must be of INT type, and all the columns must co-occur +// and have the same valency at the same row. +type TensorRepresentation_SparseTensor struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // The dense shape of the resulting SparseTensor (does not include the batch + // dimension). + DenseShape *FixedShape `protobuf:"bytes,1,opt,name=dense_shape,json=denseShape" json:"dense_shape,omitempty"` + // The columns constitute the coordinates of the values. + // indices_column[i][j] contains the coordinate of the i-th dimension of the + // j-th value. + IndexColumnNames []string `protobuf:"bytes,2,rep,name=index_column_names,json=indexColumnNames" json:"index_column_names,omitempty"` + // The column that contains the values. + ValueColumnName *string `protobuf:"bytes,3,opt,name=value_column_name,json=valueColumnName" json:"value_column_name,omitempty"` +} + +func (x *TensorRepresentation_SparseTensor) Reset() { + *x = TensorRepresentation_SparseTensor{} + if protoimpl.UnsafeEnabled { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[34] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *TensorRepresentation_SparseTensor) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*TensorRepresentation_SparseTensor) ProtoMessage() {} + +func (x *TensorRepresentation_SparseTensor) ProtoReflect() protoreflect.Message { + mi := &file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[34] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use TensorRepresentation_SparseTensor.ProtoReflect.Descriptor instead. +func (*TensorRepresentation_SparseTensor) Descriptor() ([]byte, []int) { + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP(), []int{25, 3} +} + +func (x *TensorRepresentation_SparseTensor) GetDenseShape() *FixedShape { + if x != nil { + return x.DenseShape + } + return nil +} + +func (x *TensorRepresentation_SparseTensor) GetIndexColumnNames() []string { + if x != nil { + return x.IndexColumnNames + } + return nil +} + +func (x *TensorRepresentation_SparseTensor) GetValueColumnName() string { + if x != nil && x.ValueColumnName != nil { + return *x.ValueColumnName + } + return "" +} + +var File_tensorflow_metadata_proto_v0_schema_proto protoreflect.FileDescriptor + +var file_tensorflow_metadata_proto_v0_schema_proto_rawDesc = []byte{ + 0x0a, 0x29, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x5f, 0x6d, 0x65, 0x74, + 0x61, 0x64, 0x61, 0x74, 0x61, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2f, 0x76, 0x30, 0x2f, 0x73, + 0x63, 0x68, 0x65, 0x6d, 0x61, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x16, 0x74, 0x65, 0x6e, + 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, + 0x2e, 0x76, 0x30, 0x1a, 0x19, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f, 0x74, + 0x6f, 0x62, 0x75, 0x66, 0x2f, 0x61, 0x6e, 0x79, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x27, + 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x64, + 0x61, 0x74, 0x61, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2f, 0x76, 0x30, 0x2f, 0x70, 0x61, 0x74, + 0x68, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x8c, 0x07, 0x0a, 0x06, 0x53, 0x63, 0x68, 0x65, + 0x6d, 0x61, 0x12, 0x39, 0x0a, 0x07, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x18, 0x01, 0x20, + 0x03, 0x28, 0x0b, 0x32, 0x1f, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, + 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x65, 0x61, + 0x74, 0x75, 0x72, 0x65, 0x52, 0x07, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x12, 0x4c, 0x0a, + 0x0e, 0x73, 0x70, 0x61, 0x72, 0x73, 0x65, 0x5f, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x18, + 0x06, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x25, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, + 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x53, + 0x70, 0x61, 0x72, 0x73, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x0d, 0x73, 0x70, + 0x61, 0x72, 0x73, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x12, 0x52, 0x0a, 0x10, 0x77, + 0x65, 0x69, 0x67, 0x68, 0x74, 0x65, 0x64, 0x5f, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x18, + 0x0c, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x27, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, + 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x57, + 0x65, 0x69, 0x67, 0x68, 0x74, 0x65, 0x64, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x0f, + 0x77, 0x65, 0x69, 0x67, 0x68, 0x74, 0x65, 0x64, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x12, + 0x49, 0x0a, 0x0d, 0x73, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, + 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x24, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, + 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, + 0x53, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x52, 0x0c, 0x73, 0x74, + 0x72, 0x69, 0x6e, 0x67, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x46, 0x0a, 0x0c, 0x66, 0x6c, + 0x6f, 0x61, 0x74, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x09, 0x20, 0x03, 0x28, 0x0b, + 0x32, 0x23, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, + 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x44, + 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x52, 0x0b, 0x66, 0x6c, 0x6f, 0x61, 0x74, 0x44, 0x6f, 0x6d, 0x61, + 0x69, 0x6e, 0x12, 0x40, 0x0a, 0x0a, 0x69, 0x6e, 0x74, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, + 0x18, 0x0a, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x21, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, + 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, + 0x49, 0x6e, 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x52, 0x09, 0x69, 0x6e, 0x74, 0x44, 0x6f, + 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x2f, 0x0a, 0x13, 0x64, 0x65, 0x66, 0x61, 0x75, 0x6c, 0x74, 0x5f, + 0x65, 0x6e, 0x76, 0x69, 0x72, 0x6f, 0x6e, 0x6d, 0x65, 0x6e, 0x74, 0x18, 0x05, 0x20, 0x03, 0x28, + 0x09, 0x52, 0x12, 0x64, 0x65, 0x66, 0x61, 0x75, 0x6c, 0x74, 0x45, 0x6e, 0x76, 0x69, 0x72, 0x6f, + 0x6e, 0x6d, 0x65, 0x6e, 0x74, 0x12, 0x42, 0x0a, 0x0a, 0x61, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, + 0x69, 0x6f, 0x6e, 0x18, 0x08, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, 0x65, 0x6e, 0x73, + 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, + 0x76, 0x30, 0x2e, 0x41, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x0a, 0x61, + 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x5b, 0x0a, 0x13, 0x64, 0x61, 0x74, + 0x61, 0x73, 0x65, 0x74, 0x5f, 0x63, 0x6f, 0x6e, 0x73, 0x74, 0x72, 0x61, 0x69, 0x6e, 0x74, 0x73, + 0x18, 0x0b, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2a, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, + 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, + 0x44, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x43, 0x6f, 0x6e, 0x73, 0x74, 0x72, 0x61, 0x69, 0x6e, + 0x74, 0x73, 0x52, 0x12, 0x64, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x43, 0x6f, 0x6e, 0x73, 0x74, + 0x72, 0x61, 0x69, 0x6e, 0x74, 0x73, 0x12, 0x7d, 0x0a, 0x1b, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, + 0x5f, 0x72, 0x65, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x5f, + 0x67, 0x72, 0x6f, 0x75, 0x70, 0x18, 0x0d, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x3d, 0x2e, 0x74, 0x65, + 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, + 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x53, 0x63, 0x68, 0x65, 0x6d, 0x61, 0x2e, 0x54, 0x65, 0x6e, 0x73, + 0x6f, 0x72, 0x52, 0x65, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, + 0x47, 0x72, 0x6f, 0x75, 0x70, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x19, 0x74, 0x65, 0x6e, 0x73, + 0x6f, 0x72, 0x52, 0x65, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, + 0x47, 0x72, 0x6f, 0x75, 0x70, 0x1a, 0x7f, 0x0a, 0x1e, 0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x52, + 0x65, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x47, 0x72, 0x6f, + 0x75, 0x70, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, + 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x47, 0x0a, 0x05, 0x76, 0x61, 0x6c, + 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x31, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, + 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, + 0x30, 0x2e, 0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x52, 0x65, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, + 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x47, 0x72, 0x6f, 0x75, 0x70, 0x52, 0x05, 0x76, 0x61, 0x6c, + 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x22, 0xb9, 0x0e, 0x0a, 0x07, 0x46, 0x65, 0x61, 0x74, 0x75, + 0x72, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, + 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x22, 0x0a, 0x0a, 0x64, 0x65, 0x70, 0x72, 0x65, 0x63, + 0x61, 0x74, 0x65, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, 0x08, 0x42, 0x02, 0x18, 0x01, 0x52, 0x0a, + 0x64, 0x65, 0x70, 0x72, 0x65, 0x63, 0x61, 0x74, 0x65, 0x64, 0x12, 0x45, 0x0a, 0x08, 0x70, 0x72, + 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x18, 0x0e, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x27, 0x2e, 0x74, + 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, + 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x50, 0x72, 0x65, + 0x73, 0x65, 0x6e, 0x63, 0x65, 0x48, 0x00, 0x52, 0x08, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, + 0x65, 0x12, 0x5b, 0x0a, 0x0e, 0x67, 0x72, 0x6f, 0x75, 0x70, 0x5f, 0x70, 0x72, 0x65, 0x73, 0x65, + 0x6e, 0x63, 0x65, 0x18, 0x11, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x32, 0x2e, 0x74, 0x65, 0x6e, 0x73, + 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, + 0x76, 0x30, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x50, 0x72, 0x65, 0x73, 0x65, 0x6e, + 0x63, 0x65, 0x57, 0x69, 0x74, 0x68, 0x69, 0x6e, 0x47, 0x72, 0x6f, 0x75, 0x70, 0x48, 0x00, 0x52, + 0x0d, 0x67, 0x72, 0x6f, 0x75, 0x70, 0x50, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x12, 0x3a, + 0x0a, 0x05, 0x73, 0x68, 0x61, 0x70, 0x65, 0x18, 0x17, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, + 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, + 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x69, 0x78, 0x65, 0x64, 0x53, 0x68, 0x61, 0x70, + 0x65, 0x48, 0x01, 0x52, 0x05, 0x73, 0x68, 0x61, 0x70, 0x65, 0x12, 0x45, 0x0a, 0x0b, 0x76, 0x61, + 0x6c, 0x75, 0x65, 0x5f, 0x63, 0x6f, 0x75, 0x6e, 0x74, 0x18, 0x05, 0x20, 0x01, 0x28, 0x0b, 0x32, + 0x22, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, + 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x43, 0x6f, + 0x75, 0x6e, 0x74, 0x48, 0x01, 0x52, 0x0a, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x43, 0x6f, 0x75, 0x6e, + 0x74, 0x12, 0x37, 0x0a, 0x04, 0x74, 0x79, 0x70, 0x65, 0x18, 0x06, 0x20, 0x01, 0x28, 0x0e, 0x32, + 0x23, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, + 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, + 0x54, 0x79, 0x70, 0x65, 0x52, 0x04, 0x74, 0x79, 0x70, 0x65, 0x12, 0x18, 0x0a, 0x06, 0x64, 0x6f, + 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x07, 0x20, 0x01, 0x28, 0x09, 0x48, 0x02, 0x52, 0x06, 0x64, 0x6f, + 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x42, 0x0a, 0x0a, 0x69, 0x6e, 0x74, 0x5f, 0x64, 0x6f, 0x6d, 0x61, + 0x69, 0x6e, 0x18, 0x09, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x21, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, + 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, + 0x30, 0x2e, 0x49, 0x6e, 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x09, 0x69, + 0x6e, 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x48, 0x0a, 0x0c, 0x66, 0x6c, 0x6f, 0x61, + 0x74, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x0a, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x23, + 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, + 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x44, 0x6f, 0x6d, + 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0b, 0x66, 0x6c, 0x6f, 0x61, 0x74, 0x44, 0x6f, 0x6d, 0x61, + 0x69, 0x6e, 0x12, 0x4b, 0x0a, 0x0d, 0x73, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x5f, 0x64, 0x6f, 0x6d, + 0x61, 0x69, 0x6e, 0x18, 0x0b, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x24, 0x2e, 0x74, 0x65, 0x6e, 0x73, + 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, + 0x76, 0x30, 0x2e, 0x53, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, + 0x02, 0x52, 0x0c, 0x73, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, + 0x45, 0x0a, 0x0b, 0x62, 0x6f, 0x6f, 0x6c, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x0d, + 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, + 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x42, 0x6f, + 0x6f, 0x6c, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0a, 0x62, 0x6f, 0x6f, 0x6c, + 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x4b, 0x0a, 0x0d, 0x73, 0x74, 0x72, 0x75, 0x63, 0x74, + 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x1d, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x24, 0x2e, + 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, + 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x53, 0x74, 0x72, 0x75, 0x63, 0x74, 0x44, 0x6f, 0x6d, + 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0c, 0x73, 0x74, 0x72, 0x75, 0x63, 0x74, 0x44, 0x6f, 0x6d, + 0x61, 0x69, 0x6e, 0x12, 0x67, 0x0a, 0x17, 0x6e, 0x61, 0x74, 0x75, 0x72, 0x61, 0x6c, 0x5f, 0x6c, + 0x61, 0x6e, 0x67, 0x75, 0x61, 0x67, 0x65, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x18, + 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2d, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, + 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x4e, 0x61, + 0x74, 0x75, 0x72, 0x61, 0x6c, 0x4c, 0x61, 0x6e, 0x67, 0x75, 0x61, 0x67, 0x65, 0x44, 0x6f, 0x6d, + 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x15, 0x6e, 0x61, 0x74, 0x75, 0x72, 0x61, 0x6c, 0x4c, 0x61, + 0x6e, 0x67, 0x75, 0x61, 0x67, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x48, 0x0a, 0x0c, + 0x69, 0x6d, 0x61, 0x67, 0x65, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x19, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x23, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, + 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x49, 0x6d, 0x61, 0x67, + 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0b, 0x69, 0x6d, 0x61, 0x67, 0x65, + 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x42, 0x0a, 0x0a, 0x6d, 0x69, 0x64, 0x5f, 0x64, 0x6f, + 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x1a, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x21, 0x2e, 0x74, 0x65, 0x6e, + 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, + 0x2e, 0x76, 0x30, 0x2e, 0x4d, 0x49, 0x44, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, + 0x09, 0x6d, 0x69, 0x64, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x42, 0x0a, 0x0a, 0x75, 0x72, + 0x6c, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x1b, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x21, + 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, + 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x55, 0x52, 0x4c, 0x44, 0x6f, 0x6d, 0x61, 0x69, + 0x6e, 0x48, 0x02, 0x52, 0x09, 0x75, 0x72, 0x6c, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x45, + 0x0a, 0x0b, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x1c, 0x20, + 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, + 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x54, 0x69, 0x6d, + 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0a, 0x74, 0x69, 0x6d, 0x65, 0x44, + 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x56, 0x0a, 0x12, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x6f, 0x66, + 0x5f, 0x64, 0x61, 0x79, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x1e, 0x20, 0x01, 0x28, + 0x0b, 0x32, 0x27, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, + 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x4f, + 0x66, 0x44, 0x61, 0x79, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0f, 0x74, 0x69, + 0x6d, 0x65, 0x4f, 0x66, 0x44, 0x61, 0x79, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x6a, 0x0a, + 0x18, 0x64, 0x69, 0x73, 0x74, 0x72, 0x69, 0x62, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x63, 0x6f, + 0x6e, 0x73, 0x74, 0x72, 0x61, 0x69, 0x6e, 0x74, 0x73, 0x18, 0x0f, 0x20, 0x01, 0x28, 0x0b, 0x32, + 0x2f, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, + 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x44, 0x69, 0x73, 0x74, 0x72, 0x69, 0x62, + 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x43, 0x6f, 0x6e, 0x73, 0x74, 0x72, 0x61, 0x69, 0x6e, 0x74, 0x73, + 0x52, 0x17, 0x64, 0x69, 0x73, 0x74, 0x72, 0x69, 0x62, 0x75, 0x74, 0x69, 0x6f, 0x6e, 0x43, 0x6f, + 0x6e, 0x73, 0x74, 0x72, 0x61, 0x69, 0x6e, 0x74, 0x73, 0x12, 0x42, 0x0a, 0x0a, 0x61, 0x6e, 0x6e, + 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x10, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, + 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, + 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x41, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, + 0x6e, 0x52, 0x0a, 0x61, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x52, 0x0a, + 0x0f, 0x73, 0x6b, 0x65, 0x77, 0x5f, 0x63, 0x6f, 0x6d, 0x70, 0x61, 0x72, 0x61, 0x74, 0x6f, 0x72, + 0x18, 0x12, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x29, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, + 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, + 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x43, 0x6f, 0x6d, 0x70, 0x61, 0x72, 0x61, 0x74, 0x6f, + 0x72, 0x52, 0x0e, 0x73, 0x6b, 0x65, 0x77, 0x43, 0x6f, 0x6d, 0x70, 0x61, 0x72, 0x61, 0x74, 0x6f, + 0x72, 0x12, 0x54, 0x0a, 0x10, 0x64, 0x72, 0x69, 0x66, 0x74, 0x5f, 0x63, 0x6f, 0x6d, 0x70, 0x61, + 0x72, 0x61, 0x74, 0x6f, 0x72, 0x18, 0x15, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x29, 0x2e, 0x74, 0x65, + 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, + 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x43, 0x6f, 0x6d, 0x70, + 0x61, 0x72, 0x61, 0x74, 0x6f, 0x72, 0x52, 0x0f, 0x64, 0x72, 0x69, 0x66, 0x74, 0x43, 0x6f, 0x6d, + 0x70, 0x61, 0x72, 0x61, 0x74, 0x6f, 0x72, 0x12, 0x25, 0x0a, 0x0e, 0x69, 0x6e, 0x5f, 0x65, 0x6e, + 0x76, 0x69, 0x72, 0x6f, 0x6e, 0x6d, 0x65, 0x6e, 0x74, 0x18, 0x14, 0x20, 0x03, 0x28, 0x09, 0x52, + 0x0d, 0x69, 0x6e, 0x45, 0x6e, 0x76, 0x69, 0x72, 0x6f, 0x6e, 0x6d, 0x65, 0x6e, 0x74, 0x12, 0x2c, + 0x0a, 0x12, 0x6e, 0x6f, 0x74, 0x5f, 0x69, 0x6e, 0x5f, 0x65, 0x6e, 0x76, 0x69, 0x72, 0x6f, 0x6e, + 0x6d, 0x65, 0x6e, 0x74, 0x18, 0x13, 0x20, 0x03, 0x28, 0x09, 0x52, 0x10, 0x6e, 0x6f, 0x74, 0x49, + 0x6e, 0x45, 0x6e, 0x76, 0x69, 0x72, 0x6f, 0x6e, 0x6d, 0x65, 0x6e, 0x74, 0x12, 0x4f, 0x0a, 0x0f, + 0x6c, 0x69, 0x66, 0x65, 0x63, 0x79, 0x63, 0x6c, 0x65, 0x5f, 0x73, 0x74, 0x61, 0x67, 0x65, 0x18, + 0x16, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x26, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, + 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x4c, + 0x69, 0x66, 0x65, 0x63, 0x79, 0x63, 0x6c, 0x65, 0x53, 0x74, 0x61, 0x67, 0x65, 0x52, 0x0e, 0x6c, + 0x69, 0x66, 0x65, 0x63, 0x79, 0x63, 0x6c, 0x65, 0x53, 0x74, 0x61, 0x67, 0x65, 0x42, 0x16, 0x0a, + 0x14, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x5f, 0x63, 0x6f, 0x6e, 0x73, 0x74, 0x72, + 0x61, 0x69, 0x6e, 0x74, 0x73, 0x42, 0x0c, 0x0a, 0x0a, 0x73, 0x68, 0x61, 0x70, 0x65, 0x5f, 0x74, + 0x79, 0x70, 0x65, 0x42, 0x0d, 0x0a, 0x0b, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x5f, 0x69, 0x6e, + 0x66, 0x6f, 0x22, 0x75, 0x0a, 0x0a, 0x41, 0x6e, 0x6e, 0x6f, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, + 0x12, 0x10, 0x0a, 0x03, 0x74, 0x61, 0x67, 0x18, 0x01, 0x20, 0x03, 0x28, 0x09, 0x52, 0x03, 0x74, + 0x61, 0x67, 0x12, 0x18, 0x0a, 0x07, 0x63, 0x6f, 0x6d, 0x6d, 0x65, 0x6e, 0x74, 0x18, 0x02, 0x20, + 0x03, 0x28, 0x09, 0x52, 0x07, 0x63, 0x6f, 0x6d, 0x6d, 0x65, 0x6e, 0x74, 0x12, 0x3b, 0x0a, 0x0e, + 0x65, 0x78, 0x74, 0x72, 0x61, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x18, 0x03, + 0x20, 0x03, 0x28, 0x0b, 0x32, 0x14, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, + 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x41, 0x6e, 0x79, 0x52, 0x0d, 0x65, 0x78, 0x74, 0x72, + 0x61, 0x4d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x22, 0x84, 0x01, 0x0a, 0x16, 0x4e, 0x75, + 0x6d, 0x65, 0x72, 0x69, 0x63, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x43, 0x6f, 0x6d, 0x70, 0x61, 0x72, + 0x61, 0x74, 0x6f, 0x72, 0x12, 0x34, 0x0a, 0x16, 0x6d, 0x69, 0x6e, 0x5f, 0x66, 0x72, 0x61, 0x63, + 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x74, 0x68, 0x72, 0x65, 0x73, 0x68, 0x6f, 0x6c, 0x64, 0x18, 0x01, + 0x20, 0x01, 0x28, 0x01, 0x52, 0x14, 0x6d, 0x69, 0x6e, 0x46, 0x72, 0x61, 0x63, 0x74, 0x69, 0x6f, + 0x6e, 0x54, 0x68, 0x72, 0x65, 0x73, 0x68, 0x6f, 0x6c, 0x64, 0x12, 0x34, 0x0a, 0x16, 0x6d, 0x61, + 0x78, 0x5f, 0x66, 0x72, 0x61, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x74, 0x68, 0x72, 0x65, 0x73, + 0x68, 0x6f, 0x6c, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, 0x01, 0x52, 0x14, 0x6d, 0x61, 0x78, 0x46, + 0x72, 0x61, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x54, 0x68, 0x72, 0x65, 0x73, 0x68, 0x6f, 0x6c, 0x64, + 0x22, 0xac, 0x02, 0x0a, 0x12, 0x44, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x43, 0x6f, 0x6e, 0x73, + 0x74, 0x72, 0x61, 0x69, 0x6e, 0x74, 0x73, 0x12, 0x71, 0x0a, 0x1d, 0x6e, 0x75, 0x6d, 0x5f, 0x65, + 0x78, 0x61, 0x6d, 0x70, 0x6c, 0x65, 0x73, 0x5f, 0x64, 0x72, 0x69, 0x66, 0x74, 0x5f, 0x63, 0x6f, + 0x6d, 0x70, 0x61, 0x72, 0x61, 0x74, 0x6f, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2e, + 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, + 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x4e, 0x75, 0x6d, 0x65, 0x72, 0x69, 0x63, 0x56, + 0x61, 0x6c, 0x75, 0x65, 0x43, 0x6f, 0x6d, 0x70, 0x61, 0x72, 0x61, 0x74, 0x6f, 0x72, 0x52, 0x1a, + 0x6e, 0x75, 0x6d, 0x45, 0x78, 0x61, 0x6d, 0x70, 0x6c, 0x65, 0x73, 0x44, 0x72, 0x69, 0x66, 0x74, + 0x43, 0x6f, 0x6d, 0x70, 0x61, 0x72, 0x61, 0x74, 0x6f, 0x72, 0x12, 0x75, 0x0a, 0x1f, 0x6e, 0x75, + 0x6d, 0x5f, 0x65, 0x78, 0x61, 0x6d, 0x70, 0x6c, 0x65, 0x73, 0x5f, 0x76, 0x65, 0x72, 0x73, 0x69, + 0x6f, 0x6e, 0x5f, 0x63, 0x6f, 0x6d, 0x70, 0x61, 0x72, 0x61, 0x74, 0x6f, 0x72, 0x18, 0x02, 0x20, + 0x01, 0x28, 0x0b, 0x32, 0x2e, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, + 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x4e, 0x75, 0x6d, + 0x65, 0x72, 0x69, 0x63, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x43, 0x6f, 0x6d, 0x70, 0x61, 0x72, 0x61, + 0x74, 0x6f, 0x72, 0x52, 0x1c, 0x6e, 0x75, 0x6d, 0x45, 0x78, 0x61, 0x6d, 0x70, 0x6c, 0x65, 0x73, + 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x43, 0x6f, 0x6d, 0x70, 0x61, 0x72, 0x61, 0x74, 0x6f, + 0x72, 0x12, 0x2c, 0x0a, 0x12, 0x6d, 0x69, 0x6e, 0x5f, 0x65, 0x78, 0x61, 0x6d, 0x70, 0x6c, 0x65, + 0x73, 0x5f, 0x63, 0x6f, 0x75, 0x6e, 0x74, 0x18, 0x03, 0x20, 0x01, 0x28, 0x03, 0x52, 0x10, 0x6d, + 0x69, 0x6e, 0x45, 0x78, 0x61, 0x6d, 0x70, 0x6c, 0x65, 0x73, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x22, + 0x75, 0x0a, 0x0a, 0x46, 0x69, 0x78, 0x65, 0x64, 0x53, 0x68, 0x61, 0x70, 0x65, 0x12, 0x38, 0x0a, + 0x03, 0x64, 0x69, 0x6d, 0x18, 0x02, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x26, 0x2e, 0x74, 0x65, 0x6e, + 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, + 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x69, 0x78, 0x65, 0x64, 0x53, 0x68, 0x61, 0x70, 0x65, 0x2e, 0x44, + 0x69, 0x6d, 0x52, 0x03, 0x64, 0x69, 0x6d, 0x1a, 0x2d, 0x0a, 0x03, 0x44, 0x69, 0x6d, 0x12, 0x12, + 0x0a, 0x04, 0x73, 0x69, 0x7a, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x03, 0x52, 0x04, 0x73, 0x69, + 0x7a, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, + 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x30, 0x0a, 0x0a, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x43, + 0x6f, 0x75, 0x6e, 0x74, 0x12, 0x10, 0x0a, 0x03, 0x6d, 0x69, 0x6e, 0x18, 0x01, 0x20, 0x01, 0x28, + 0x03, 0x52, 0x03, 0x6d, 0x69, 0x6e, 0x12, 0x10, 0x0a, 0x03, 0x6d, 0x61, 0x78, 0x18, 0x02, 0x20, + 0x01, 0x28, 0x03, 0x52, 0x03, 0x6d, 0x61, 0x78, 0x22, 0xf3, 0x01, 0x0a, 0x0f, 0x57, 0x65, 0x69, + 0x67, 0x68, 0x74, 0x65, 0x64, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x12, 0x12, 0x0a, 0x04, + 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, + 0x12, 0x36, 0x0a, 0x07, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, + 0x0b, 0x32, 0x1c, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, + 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x50, 0x61, 0x74, 0x68, 0x52, + 0x07, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x12, 0x43, 0x0a, 0x0e, 0x77, 0x65, 0x69, 0x67, + 0x68, 0x74, 0x5f, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, + 0x32, 0x1c, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, + 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x50, 0x61, 0x74, 0x68, 0x52, 0x0d, + 0x77, 0x65, 0x69, 0x67, 0x68, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x12, 0x4f, 0x0a, + 0x0f, 0x6c, 0x69, 0x66, 0x65, 0x63, 0x79, 0x63, 0x6c, 0x65, 0x5f, 0x73, 0x74, 0x61, 0x67, 0x65, + 0x18, 0x04, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x26, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, + 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, + 0x4c, 0x69, 0x66, 0x65, 0x63, 0x79, 0x63, 0x6c, 0x65, 0x53, 0x74, 0x61, 0x67, 0x65, 0x52, 0x0e, + 0x6c, 0x69, 0x66, 0x65, 0x63, 0x79, 0x63, 0x6c, 0x65, 0x53, 0x74, 0x61, 0x67, 0x65, 0x22, 0x80, + 0x05, 0x0a, 0x0d, 0x53, 0x70, 0x61, 0x72, 0x73, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, + 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, + 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x22, 0x0a, 0x0a, 0x64, 0x65, 0x70, 0x72, 0x65, 0x63, 0x61, 0x74, + 0x65, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, 0x08, 0x42, 0x02, 0x18, 0x01, 0x52, 0x0a, 0x64, 0x65, + 0x70, 0x72, 0x65, 0x63, 0x61, 0x74, 0x65, 0x64, 0x12, 0x4f, 0x0a, 0x0f, 0x6c, 0x69, 0x66, 0x65, + 0x63, 0x79, 0x63, 0x6c, 0x65, 0x5f, 0x73, 0x74, 0x61, 0x67, 0x65, 0x18, 0x07, 0x20, 0x01, 0x28, + 0x0e, 0x32, 0x26, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, + 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x4c, 0x69, 0x66, 0x65, 0x63, + 0x79, 0x63, 0x6c, 0x65, 0x53, 0x74, 0x61, 0x67, 0x65, 0x52, 0x0e, 0x6c, 0x69, 0x66, 0x65, 0x63, + 0x79, 0x63, 0x6c, 0x65, 0x53, 0x74, 0x61, 0x67, 0x65, 0x12, 0x47, 0x0a, 0x08, 0x70, 0x72, 0x65, + 0x73, 0x65, 0x6e, 0x63, 0x65, 0x18, 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x27, 0x2e, 0x74, 0x65, + 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, + 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x50, 0x72, 0x65, 0x73, + 0x65, 0x6e, 0x63, 0x65, 0x42, 0x02, 0x18, 0x01, 0x52, 0x08, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, + 0x63, 0x65, 0x12, 0x43, 0x0a, 0x0b, 0x64, 0x65, 0x6e, 0x73, 0x65, 0x5f, 0x73, 0x68, 0x61, 0x70, + 0x65, 0x18, 0x05, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, + 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, + 0x2e, 0x46, 0x69, 0x78, 0x65, 0x64, 0x53, 0x68, 0x61, 0x70, 0x65, 0x52, 0x0a, 0x64, 0x65, 0x6e, + 0x73, 0x65, 0x53, 0x68, 0x61, 0x70, 0x65, 0x12, 0x57, 0x0a, 0x0d, 0x69, 0x6e, 0x64, 0x65, 0x78, + 0x5f, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x18, 0x06, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x32, + 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, + 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x53, 0x70, 0x61, 0x72, 0x73, 0x65, 0x46, 0x65, + 0x61, 0x74, 0x75, 0x72, 0x65, 0x2e, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x46, 0x65, 0x61, 0x74, 0x75, + 0x72, 0x65, 0x52, 0x0c, 0x69, 0x6e, 0x64, 0x65, 0x78, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, + 0x12, 0x1b, 0x0a, 0x09, 0x69, 0x73, 0x5f, 0x73, 0x6f, 0x72, 0x74, 0x65, 0x64, 0x18, 0x08, 0x20, + 0x01, 0x28, 0x08, 0x52, 0x08, 0x69, 0x73, 0x53, 0x6f, 0x72, 0x74, 0x65, 0x64, 0x12, 0x57, 0x0a, + 0x0d, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x5f, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x18, 0x09, + 0x20, 0x01, 0x28, 0x0b, 0x32, 0x32, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, + 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x53, 0x70, + 0x61, 0x72, 0x73, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x2e, 0x56, 0x61, 0x6c, 0x75, + 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x0c, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x46, + 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x12, 0x3b, 0x0a, 0x04, 0x74, 0x79, 0x70, 0x65, 0x18, 0x0a, + 0x20, 0x01, 0x28, 0x0e, 0x32, 0x23, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, + 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x65, + 0x61, 0x74, 0x75, 0x72, 0x65, 0x54, 0x79, 0x70, 0x65, 0x42, 0x02, 0x18, 0x01, 0x52, 0x04, 0x74, + 0x79, 0x70, 0x65, 0x1a, 0x22, 0x0a, 0x0c, 0x49, 0x6e, 0x64, 0x65, 0x78, 0x46, 0x65, 0x61, 0x74, + 0x75, 0x72, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, + 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x1a, 0x22, 0x0a, 0x0c, 0x56, 0x61, 0x6c, 0x75, 0x65, + 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, + 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x4a, 0x04, 0x08, 0x0b, 0x10, + 0x0c, 0x22, 0x44, 0x0a, 0x17, 0x44, 0x69, 0x73, 0x74, 0x72, 0x69, 0x62, 0x75, 0x74, 0x69, 0x6f, + 0x6e, 0x43, 0x6f, 0x6e, 0x73, 0x74, 0x72, 0x61, 0x69, 0x6e, 0x74, 0x73, 0x12, 0x29, 0x0a, 0x0f, + 0x6d, 0x69, 0x6e, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x5f, 0x6d, 0x61, 0x73, 0x73, 0x18, + 0x01, 0x20, 0x01, 0x28, 0x01, 0x3a, 0x01, 0x31, 0x52, 0x0d, 0x6d, 0x69, 0x6e, 0x44, 0x6f, 0x6d, + 0x61, 0x69, 0x6e, 0x4d, 0x61, 0x73, 0x73, 0x22, 0x6a, 0x0a, 0x09, 0x49, 0x6e, 0x74, 0x44, 0x6f, + 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, + 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x10, 0x0a, 0x03, 0x6d, 0x69, 0x6e, 0x18, + 0x03, 0x20, 0x01, 0x28, 0x03, 0x52, 0x03, 0x6d, 0x69, 0x6e, 0x12, 0x10, 0x0a, 0x03, 0x6d, 0x61, + 0x78, 0x18, 0x04, 0x20, 0x01, 0x28, 0x03, 0x52, 0x03, 0x6d, 0x61, 0x78, 0x12, 0x25, 0x0a, 0x0e, + 0x69, 0x73, 0x5f, 0x63, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x69, 0x63, 0x61, 0x6c, 0x18, 0x05, + 0x20, 0x01, 0x28, 0x08, 0x52, 0x0d, 0x69, 0x73, 0x43, 0x61, 0x74, 0x65, 0x67, 0x6f, 0x72, 0x69, + 0x63, 0x61, 0x6c, 0x22, 0x45, 0x0a, 0x0b, 0x46, 0x6c, 0x6f, 0x61, 0x74, 0x44, 0x6f, 0x6d, 0x61, + 0x69, 0x6e, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, + 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x10, 0x0a, 0x03, 0x6d, 0x69, 0x6e, 0x18, 0x03, 0x20, + 0x01, 0x28, 0x02, 0x52, 0x03, 0x6d, 0x69, 0x6e, 0x12, 0x10, 0x0a, 0x03, 0x6d, 0x61, 0x78, 0x18, + 0x04, 0x20, 0x01, 0x28, 0x02, 0x52, 0x03, 0x6d, 0x61, 0x78, 0x22, 0x97, 0x01, 0x0a, 0x0c, 0x53, + 0x74, 0x72, 0x75, 0x63, 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x39, 0x0a, 0x07, 0x66, + 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x1f, 0x2e, 0x74, + 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, + 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x07, 0x66, + 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x12, 0x4c, 0x0a, 0x0e, 0x73, 0x70, 0x61, 0x72, 0x73, 0x65, + 0x5f, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x18, 0x02, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x25, + 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, + 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x53, 0x70, 0x61, 0x72, 0x73, 0x65, 0x46, 0x65, + 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x0d, 0x73, 0x70, 0x61, 0x72, 0x73, 0x65, 0x46, 0x65, 0x61, + 0x74, 0x75, 0x72, 0x65, 0x22, 0x38, 0x0a, 0x0c, 0x53, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x44, 0x6f, + 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, + 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, + 0x65, 0x18, 0x02, 0x20, 0x03, 0x28, 0x09, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x22, 0x60, + 0x0a, 0x0a, 0x42, 0x6f, 0x6f, 0x6c, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x12, 0x0a, 0x04, + 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, + 0x12, 0x1d, 0x0a, 0x0a, 0x74, 0x72, 0x75, 0x65, 0x5f, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, + 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x74, 0x72, 0x75, 0x65, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x12, + 0x1f, 0x0a, 0x0b, 0x66, 0x61, 0x6c, 0x73, 0x65, 0x5f, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x03, + 0x20, 0x01, 0x28, 0x09, 0x52, 0x0a, 0x66, 0x61, 0x6c, 0x73, 0x65, 0x56, 0x61, 0x6c, 0x75, 0x65, + 0x22, 0x17, 0x0a, 0x15, 0x4e, 0x61, 0x74, 0x75, 0x72, 0x61, 0x6c, 0x4c, 0x61, 0x6e, 0x67, 0x75, + 0x61, 0x67, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x22, 0x0d, 0x0a, 0x0b, 0x49, 0x6d, 0x61, + 0x67, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x22, 0x0b, 0x0a, 0x09, 0x4d, 0x49, 0x44, 0x44, + 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x22, 0x0b, 0x0a, 0x09, 0x55, 0x52, 0x4c, 0x44, 0x6f, 0x6d, 0x61, + 0x69, 0x6e, 0x22, 0xab, 0x02, 0x0a, 0x0a, 0x54, 0x69, 0x6d, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, + 0x6e, 0x12, 0x25, 0x0a, 0x0d, 0x73, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x5f, 0x66, 0x6f, 0x72, 0x6d, + 0x61, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x48, 0x00, 0x52, 0x0c, 0x73, 0x74, 0x72, 0x69, + 0x6e, 0x67, 0x46, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x12, 0x5d, 0x0a, 0x0e, 0x69, 0x6e, 0x74, 0x65, + 0x67, 0x65, 0x72, 0x5f, 0x66, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, + 0x32, 0x34, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, + 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x44, 0x6f, + 0x6d, 0x61, 0x69, 0x6e, 0x2e, 0x49, 0x6e, 0x74, 0x65, 0x67, 0x65, 0x72, 0x54, 0x69, 0x6d, 0x65, + 0x46, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x48, 0x00, 0x52, 0x0d, 0x69, 0x6e, 0x74, 0x65, 0x67, 0x65, + 0x72, 0x46, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x22, 0x8c, 0x01, 0x0a, 0x11, 0x49, 0x6e, 0x74, 0x65, + 0x67, 0x65, 0x72, 0x54, 0x69, 0x6d, 0x65, 0x46, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x12, 0x12, 0x0a, + 0x0e, 0x46, 0x4f, 0x52, 0x4d, 0x41, 0x54, 0x5f, 0x55, 0x4e, 0x4b, 0x4e, 0x4f, 0x57, 0x4e, 0x10, + 0x00, 0x12, 0x0d, 0x0a, 0x09, 0x55, 0x4e, 0x49, 0x58, 0x5f, 0x44, 0x41, 0x59, 0x53, 0x10, 0x05, + 0x12, 0x10, 0x0a, 0x0c, 0x55, 0x4e, 0x49, 0x58, 0x5f, 0x53, 0x45, 0x43, 0x4f, 0x4e, 0x44, 0x53, + 0x10, 0x01, 0x12, 0x15, 0x0a, 0x11, 0x55, 0x4e, 0x49, 0x58, 0x5f, 0x4d, 0x49, 0x4c, 0x4c, 0x49, + 0x53, 0x45, 0x43, 0x4f, 0x4e, 0x44, 0x53, 0x10, 0x02, 0x12, 0x15, 0x0a, 0x11, 0x55, 0x4e, 0x49, + 0x58, 0x5f, 0x4d, 0x49, 0x43, 0x52, 0x4f, 0x53, 0x45, 0x43, 0x4f, 0x4e, 0x44, 0x53, 0x10, 0x03, + 0x12, 0x14, 0x0a, 0x10, 0x55, 0x4e, 0x49, 0x58, 0x5f, 0x4e, 0x41, 0x4e, 0x4f, 0x53, 0x45, 0x43, + 0x4f, 0x4e, 0x44, 0x53, 0x10, 0x04, 0x42, 0x08, 0x0a, 0x06, 0x66, 0x6f, 0x72, 0x6d, 0x61, 0x74, + 0x22, 0xee, 0x01, 0x0a, 0x0f, 0x54, 0x69, 0x6d, 0x65, 0x4f, 0x66, 0x44, 0x61, 0x79, 0x44, 0x6f, + 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x25, 0x0a, 0x0d, 0x73, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x5f, 0x66, + 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x48, 0x00, 0x52, 0x0c, 0x73, + 0x74, 0x72, 0x69, 0x6e, 0x67, 0x46, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x12, 0x67, 0x0a, 0x0e, 0x69, + 0x6e, 0x74, 0x65, 0x67, 0x65, 0x72, 0x5f, 0x66, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x18, 0x02, 0x20, + 0x01, 0x28, 0x0e, 0x32, 0x3e, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, + 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x54, 0x69, 0x6d, + 0x65, 0x4f, 0x66, 0x44, 0x61, 0x79, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x2e, 0x49, 0x6e, 0x74, + 0x65, 0x67, 0x65, 0x72, 0x54, 0x69, 0x6d, 0x65, 0x4f, 0x66, 0x44, 0x61, 0x79, 0x46, 0x6f, 0x72, + 0x6d, 0x61, 0x74, 0x48, 0x00, 0x52, 0x0d, 0x69, 0x6e, 0x74, 0x65, 0x67, 0x65, 0x72, 0x46, 0x6f, + 0x72, 0x6d, 0x61, 0x74, 0x22, 0x41, 0x0a, 0x16, 0x49, 0x6e, 0x74, 0x65, 0x67, 0x65, 0x72, 0x54, + 0x69, 0x6d, 0x65, 0x4f, 0x66, 0x44, 0x61, 0x79, 0x46, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x12, 0x12, + 0x0a, 0x0e, 0x46, 0x4f, 0x52, 0x4d, 0x41, 0x54, 0x5f, 0x55, 0x4e, 0x4b, 0x4e, 0x4f, 0x57, 0x4e, + 0x10, 0x00, 0x12, 0x13, 0x0a, 0x0f, 0x50, 0x41, 0x43, 0x4b, 0x45, 0x44, 0x5f, 0x36, 0x34, 0x5f, + 0x4e, 0x41, 0x4e, 0x4f, 0x53, 0x10, 0x01, 0x42, 0x08, 0x0a, 0x06, 0x66, 0x6f, 0x72, 0x6d, 0x61, + 0x74, 0x22, 0x51, 0x0a, 0x0f, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x50, 0x72, 0x65, 0x73, + 0x65, 0x6e, 0x63, 0x65, 0x12, 0x21, 0x0a, 0x0c, 0x6d, 0x69, 0x6e, 0x5f, 0x66, 0x72, 0x61, 0x63, + 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x01, 0x20, 0x01, 0x28, 0x01, 0x52, 0x0b, 0x6d, 0x69, 0x6e, 0x46, + 0x72, 0x61, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x1b, 0x0a, 0x09, 0x6d, 0x69, 0x6e, 0x5f, 0x63, + 0x6f, 0x75, 0x6e, 0x74, 0x18, 0x02, 0x20, 0x01, 0x28, 0x03, 0x52, 0x08, 0x6d, 0x69, 0x6e, 0x43, + 0x6f, 0x75, 0x6e, 0x74, 0x22, 0x38, 0x0a, 0x1a, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x50, + 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x57, 0x69, 0x74, 0x68, 0x69, 0x6e, 0x47, 0x72, 0x6f, + 0x75, 0x70, 0x12, 0x1a, 0x0a, 0x08, 0x72, 0x65, 0x71, 0x75, 0x69, 0x72, 0x65, 0x64, 0x18, 0x01, + 0x20, 0x01, 0x28, 0x08, 0x52, 0x08, 0x72, 0x65, 0x71, 0x75, 0x69, 0x72, 0x65, 0x64, 0x22, 0x2c, + 0x0a, 0x0c, 0x49, 0x6e, 0x66, 0x69, 0x6e, 0x69, 0x74, 0x79, 0x4e, 0x6f, 0x72, 0x6d, 0x12, 0x1c, + 0x0a, 0x09, 0x74, 0x68, 0x72, 0x65, 0x73, 0x68, 0x6f, 0x6c, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, + 0x01, 0x52, 0x09, 0x74, 0x68, 0x72, 0x65, 0x73, 0x68, 0x6f, 0x6c, 0x64, 0x22, 0x5e, 0x0a, 0x11, + 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x43, 0x6f, 0x6d, 0x70, 0x61, 0x72, 0x61, 0x74, 0x6f, + 0x72, 0x12, 0x49, 0x0a, 0x0d, 0x69, 0x6e, 0x66, 0x69, 0x6e, 0x69, 0x74, 0x79, 0x5f, 0x6e, 0x6f, + 0x72, 0x6d, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x24, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, + 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, + 0x30, 0x2e, 0x49, 0x6e, 0x66, 0x69, 0x6e, 0x69, 0x74, 0x79, 0x4e, 0x6f, 0x72, 0x6d, 0x52, 0x0c, + 0x69, 0x6e, 0x66, 0x69, 0x6e, 0x69, 0x74, 0x79, 0x4e, 0x6f, 0x72, 0x6d, 0x22, 0xa5, 0x07, 0x0a, + 0x14, 0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x52, 0x65, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x74, + 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x5d, 0x0a, 0x0c, 0x64, 0x65, 0x6e, 0x73, 0x65, 0x5f, 0x74, + 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x38, 0x2e, 0x74, 0x65, + 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, + 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x52, 0x65, 0x70, 0x72, 0x65, + 0x73, 0x65, 0x6e, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2e, 0x44, 0x65, 0x6e, 0x73, 0x65, 0x54, + 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x48, 0x00, 0x52, 0x0b, 0x64, 0x65, 0x6e, 0x73, 0x65, 0x54, 0x65, + 0x6e, 0x73, 0x6f, 0x72, 0x12, 0x73, 0x0a, 0x14, 0x76, 0x61, 0x72, 0x6c, 0x65, 0x6e, 0x5f, 0x73, + 0x70, 0x61, 0x72, 0x73, 0x65, 0x5f, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x18, 0x02, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x3f, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, + 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x54, 0x65, 0x6e, 0x73, + 0x6f, 0x72, 0x52, 0x65, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, + 0x2e, 0x56, 0x61, 0x72, 0x4c, 0x65, 0x6e, 0x53, 0x70, 0x61, 0x72, 0x73, 0x65, 0x54, 0x65, 0x6e, + 0x73, 0x6f, 0x72, 0x48, 0x00, 0x52, 0x12, 0x76, 0x61, 0x72, 0x6c, 0x65, 0x6e, 0x53, 0x70, 0x61, + 0x72, 0x73, 0x65, 0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x12, 0x60, 0x0a, 0x0d, 0x73, 0x70, 0x61, + 0x72, 0x73, 0x65, 0x5f, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, + 0x32, 0x39, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, + 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, + 0x52, 0x65, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2e, 0x53, + 0x70, 0x61, 0x72, 0x73, 0x65, 0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x48, 0x00, 0x52, 0x0c, 0x73, + 0x70, 0x61, 0x72, 0x73, 0x65, 0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x1a, 0x9c, 0x01, 0x0a, 0x0c, + 0x44, 0x65, 0x66, 0x61, 0x75, 0x6c, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x12, 0x21, 0x0a, 0x0b, + 0x66, 0x6c, 0x6f, 0x61, 0x74, 0x5f, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, + 0x01, 0x48, 0x00, 0x52, 0x0a, 0x66, 0x6c, 0x6f, 0x61, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x12, + 0x1d, 0x0a, 0x09, 0x69, 0x6e, 0x74, 0x5f, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, + 0x28, 0x03, 0x48, 0x00, 0x52, 0x08, 0x69, 0x6e, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x12, 0x21, + 0x0a, 0x0b, 0x62, 0x79, 0x74, 0x65, 0x73, 0x5f, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x03, 0x20, + 0x01, 0x28, 0x0c, 0x48, 0x00, 0x52, 0x0a, 0x62, 0x79, 0x74, 0x65, 0x73, 0x56, 0x61, 0x6c, 0x75, + 0x65, 0x12, 0x1f, 0x0a, 0x0a, 0x75, 0x69, 0x6e, 0x74, 0x5f, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, + 0x04, 0x20, 0x01, 0x28, 0x04, 0x48, 0x00, 0x52, 0x09, 0x75, 0x69, 0x6e, 0x74, 0x56, 0x61, 0x6c, + 0x75, 0x65, 0x42, 0x06, 0x0a, 0x04, 0x6b, 0x69, 0x6e, 0x64, 0x1a, 0xc8, 0x01, 0x0a, 0x0b, 0x44, + 0x65, 0x6e, 0x73, 0x65, 0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x12, 0x1f, 0x0a, 0x0b, 0x63, 0x6f, + 0x6c, 0x75, 0x6d, 0x6e, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x0a, 0x63, 0x6f, 0x6c, 0x75, 0x6d, 0x6e, 0x4e, 0x61, 0x6d, 0x65, 0x12, 0x38, 0x0a, 0x05, 0x73, + 0x68, 0x61, 0x70, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, 0x65, 0x6e, + 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, + 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x69, 0x78, 0x65, 0x64, 0x53, 0x68, 0x61, 0x70, 0x65, 0x52, 0x05, + 0x73, 0x68, 0x61, 0x70, 0x65, 0x12, 0x5e, 0x0a, 0x0d, 0x64, 0x65, 0x66, 0x61, 0x75, 0x6c, 0x74, + 0x5f, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x39, 0x2e, 0x74, + 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, + 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x52, 0x65, 0x70, 0x72, + 0x65, 0x73, 0x65, 0x6e, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x2e, 0x44, 0x65, 0x66, 0x61, 0x75, + 0x6c, 0x74, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x52, 0x0c, 0x64, 0x65, 0x66, 0x61, 0x75, 0x6c, 0x74, + 0x56, 0x61, 0x6c, 0x75, 0x65, 0x1a, 0x35, 0x0a, 0x12, 0x56, 0x61, 0x72, 0x4c, 0x65, 0x6e, 0x53, + 0x70, 0x61, 0x72, 0x73, 0x65, 0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x12, 0x1f, 0x0a, 0x0b, 0x63, + 0x6f, 0x6c, 0x75, 0x6d, 0x6e, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, + 0x52, 0x0a, 0x63, 0x6f, 0x6c, 0x75, 0x6d, 0x6e, 0x4e, 0x61, 0x6d, 0x65, 0x1a, 0xad, 0x01, 0x0a, + 0x0c, 0x53, 0x70, 0x61, 0x72, 0x73, 0x65, 0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x12, 0x43, 0x0a, + 0x0b, 0x64, 0x65, 0x6e, 0x73, 0x65, 0x5f, 0x73, 0x68, 0x61, 0x70, 0x65, 0x18, 0x01, 0x20, 0x01, + 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, + 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x69, 0x78, 0x65, + 0x64, 0x53, 0x68, 0x61, 0x70, 0x65, 0x52, 0x0a, 0x64, 0x65, 0x6e, 0x73, 0x65, 0x53, 0x68, 0x61, + 0x70, 0x65, 0x12, 0x2c, 0x0a, 0x12, 0x69, 0x6e, 0x64, 0x65, 0x78, 0x5f, 0x63, 0x6f, 0x6c, 0x75, + 0x6d, 0x6e, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x09, 0x52, 0x10, + 0x69, 0x6e, 0x64, 0x65, 0x78, 0x43, 0x6f, 0x6c, 0x75, 0x6d, 0x6e, 0x4e, 0x61, 0x6d, 0x65, 0x73, + 0x12, 0x2a, 0x0a, 0x11, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x5f, 0x63, 0x6f, 0x6c, 0x75, 0x6d, 0x6e, + 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0f, 0x76, 0x61, 0x6c, + 0x75, 0x65, 0x43, 0x6f, 0x6c, 0x75, 0x6d, 0x6e, 0x4e, 0x61, 0x6d, 0x65, 0x42, 0x06, 0x0a, 0x04, + 0x6b, 0x69, 0x6e, 0x64, 0x22, 0x95, 0x02, 0x0a, 0x19, 0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x52, + 0x65, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x47, 0x72, 0x6f, + 0x75, 0x70, 0x12, 0x80, 0x01, 0x0a, 0x15, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x5f, 0x72, 0x65, + 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x01, 0x20, 0x03, + 0x28, 0x0b, 0x32, 0x4b, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, + 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x54, 0x65, 0x6e, 0x73, + 0x6f, 0x72, 0x52, 0x65, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, + 0x47, 0x72, 0x6f, 0x75, 0x70, 0x2e, 0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x52, 0x65, 0x70, 0x72, + 0x65, 0x73, 0x65, 0x6e, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, + 0x14, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x52, 0x65, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x74, + 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x1a, 0x75, 0x0a, 0x19, 0x54, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x52, + 0x65, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x74, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x45, 0x6e, 0x74, + 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x03, 0x6b, 0x65, 0x79, 0x12, 0x42, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, + 0x01, 0x28, 0x0b, 0x32, 0x2c, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, + 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x54, 0x65, 0x6e, + 0x73, 0x6f, 0x72, 0x52, 0x65, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x74, 0x61, 0x74, 0x69, 0x6f, + 0x6e, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x2a, 0x75, 0x0a, 0x0e, + 0x4c, 0x69, 0x66, 0x65, 0x63, 0x79, 0x63, 0x6c, 0x65, 0x53, 0x74, 0x61, 0x67, 0x65, 0x12, 0x11, + 0x0a, 0x0d, 0x55, 0x4e, 0x4b, 0x4e, 0x4f, 0x57, 0x4e, 0x5f, 0x53, 0x54, 0x41, 0x47, 0x45, 0x10, + 0x00, 0x12, 0x0b, 0x0a, 0x07, 0x50, 0x4c, 0x41, 0x4e, 0x4e, 0x45, 0x44, 0x10, 0x01, 0x12, 0x09, + 0x0a, 0x05, 0x41, 0x4c, 0x50, 0x48, 0x41, 0x10, 0x02, 0x12, 0x08, 0x0a, 0x04, 0x42, 0x45, 0x54, + 0x41, 0x10, 0x03, 0x12, 0x0e, 0x0a, 0x0a, 0x50, 0x52, 0x4f, 0x44, 0x55, 0x43, 0x54, 0x49, 0x4f, + 0x4e, 0x10, 0x04, 0x12, 0x0e, 0x0a, 0x0a, 0x44, 0x45, 0x50, 0x52, 0x45, 0x43, 0x41, 0x54, 0x45, + 0x44, 0x10, 0x05, 0x12, 0x0e, 0x0a, 0x0a, 0x44, 0x45, 0x42, 0x55, 0x47, 0x5f, 0x4f, 0x4e, 0x4c, + 0x59, 0x10, 0x06, 0x2a, 0x4a, 0x0a, 0x0b, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x54, 0x79, + 0x70, 0x65, 0x12, 0x10, 0x0a, 0x0c, 0x54, 0x59, 0x50, 0x45, 0x5f, 0x55, 0x4e, 0x4b, 0x4e, 0x4f, + 0x57, 0x4e, 0x10, 0x00, 0x12, 0x09, 0x0a, 0x05, 0x42, 0x59, 0x54, 0x45, 0x53, 0x10, 0x01, 0x12, + 0x07, 0x0a, 0x03, 0x49, 0x4e, 0x54, 0x10, 0x02, 0x12, 0x09, 0x0a, 0x05, 0x46, 0x4c, 0x4f, 0x41, + 0x54, 0x10, 0x03, 0x12, 0x0a, 0x0a, 0x06, 0x53, 0x54, 0x52, 0x55, 0x43, 0x54, 0x10, 0x04, 0x42, + 0x70, 0x0a, 0x1a, 0x6f, 0x72, 0x67, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, + 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x50, 0x01, 0x5a, + 0x4d, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, + 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, + 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x74, 0x68, 0x69, 0x72, 0x64, 0x5f, 0x70, 0x61, 0x72, 0x74, + 0x79, 0x2f, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x5f, 0x6d, 0x65, 0x74, + 0x61, 0x64, 0x61, 0x74, 0x61, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2f, 0x76, 0x30, 0xf8, 0x01, + 0x01, +} + +var ( + file_tensorflow_metadata_proto_v0_schema_proto_rawDescOnce sync.Once + file_tensorflow_metadata_proto_v0_schema_proto_rawDescData = file_tensorflow_metadata_proto_v0_schema_proto_rawDesc +) + +func file_tensorflow_metadata_proto_v0_schema_proto_rawDescGZIP() []byte { + file_tensorflow_metadata_proto_v0_schema_proto_rawDescOnce.Do(func() { + file_tensorflow_metadata_proto_v0_schema_proto_rawDescData = protoimpl.X.CompressGZIP(file_tensorflow_metadata_proto_v0_schema_proto_rawDescData) + }) + return file_tensorflow_metadata_proto_v0_schema_proto_rawDescData +} + +var file_tensorflow_metadata_proto_v0_schema_proto_enumTypes = make([]protoimpl.EnumInfo, 4) +var file_tensorflow_metadata_proto_v0_schema_proto_msgTypes = make([]protoimpl.MessageInfo, 36) +var file_tensorflow_metadata_proto_v0_schema_proto_goTypes = []interface{}{ + (LifecycleStage)(0), // 0: tensorflow.metadata.v0.LifecycleStage + (FeatureType)(0), // 1: tensorflow.metadata.v0.FeatureType + (TimeDomain_IntegerTimeFormat)(0), // 2: tensorflow.metadata.v0.TimeDomain.IntegerTimeFormat + (TimeOfDayDomain_IntegerTimeOfDayFormat)(0), // 3: tensorflow.metadata.v0.TimeOfDayDomain.IntegerTimeOfDayFormat + (*Schema)(nil), // 4: tensorflow.metadata.v0.Schema + (*Feature)(nil), // 5: tensorflow.metadata.v0.Feature + (*Annotation)(nil), // 6: tensorflow.metadata.v0.Annotation + (*NumericValueComparator)(nil), // 7: tensorflow.metadata.v0.NumericValueComparator + (*DatasetConstraints)(nil), // 8: tensorflow.metadata.v0.DatasetConstraints + (*FixedShape)(nil), // 9: tensorflow.metadata.v0.FixedShape + (*ValueCount)(nil), // 10: tensorflow.metadata.v0.ValueCount + (*WeightedFeature)(nil), // 11: tensorflow.metadata.v0.WeightedFeature + (*SparseFeature)(nil), // 12: tensorflow.metadata.v0.SparseFeature + (*DistributionConstraints)(nil), // 13: tensorflow.metadata.v0.DistributionConstraints + (*IntDomain)(nil), // 14: tensorflow.metadata.v0.IntDomain + (*FloatDomain)(nil), // 15: tensorflow.metadata.v0.FloatDomain + (*StructDomain)(nil), // 16: tensorflow.metadata.v0.StructDomain + (*StringDomain)(nil), // 17: tensorflow.metadata.v0.StringDomain + (*BoolDomain)(nil), // 18: tensorflow.metadata.v0.BoolDomain + (*NaturalLanguageDomain)(nil), // 19: tensorflow.metadata.v0.NaturalLanguageDomain + (*ImageDomain)(nil), // 20: tensorflow.metadata.v0.ImageDomain + (*MIDDomain)(nil), // 21: tensorflow.metadata.v0.MIDDomain + (*URLDomain)(nil), // 22: tensorflow.metadata.v0.URLDomain + (*TimeDomain)(nil), // 23: tensorflow.metadata.v0.TimeDomain + (*TimeOfDayDomain)(nil), // 24: tensorflow.metadata.v0.TimeOfDayDomain + (*FeaturePresence)(nil), // 25: tensorflow.metadata.v0.FeaturePresence + (*FeaturePresenceWithinGroup)(nil), // 26: tensorflow.metadata.v0.FeaturePresenceWithinGroup + (*InfinityNorm)(nil), // 27: tensorflow.metadata.v0.InfinityNorm + (*FeatureComparator)(nil), // 28: tensorflow.metadata.v0.FeatureComparator + (*TensorRepresentation)(nil), // 29: tensorflow.metadata.v0.TensorRepresentation + (*TensorRepresentationGroup)(nil), // 30: tensorflow.metadata.v0.TensorRepresentationGroup + nil, // 31: tensorflow.metadata.v0.Schema.TensorRepresentationGroupEntry + (*FixedShape_Dim)(nil), // 32: tensorflow.metadata.v0.FixedShape.Dim + (*SparseFeature_IndexFeature)(nil), // 33: tensorflow.metadata.v0.SparseFeature.IndexFeature + (*SparseFeature_ValueFeature)(nil), // 34: tensorflow.metadata.v0.SparseFeature.ValueFeature + (*TensorRepresentation_DefaultValue)(nil), // 35: tensorflow.metadata.v0.TensorRepresentation.DefaultValue + (*TensorRepresentation_DenseTensor)(nil), // 36: tensorflow.metadata.v0.TensorRepresentation.DenseTensor + (*TensorRepresentation_VarLenSparseTensor)(nil), // 37: tensorflow.metadata.v0.TensorRepresentation.VarLenSparseTensor + (*TensorRepresentation_SparseTensor)(nil), // 38: tensorflow.metadata.v0.TensorRepresentation.SparseTensor + nil, // 39: tensorflow.metadata.v0.TensorRepresentationGroup.TensorRepresentationEntry + (*any.Any)(nil), // 40: google.protobuf.Any + (*Path)(nil), // 41: tensorflow.metadata.v0.Path +} +var file_tensorflow_metadata_proto_v0_schema_proto_depIdxs = []int32{ + 5, // 0: tensorflow.metadata.v0.Schema.feature:type_name -> tensorflow.metadata.v0.Feature + 12, // 1: tensorflow.metadata.v0.Schema.sparse_feature:type_name -> tensorflow.metadata.v0.SparseFeature + 11, // 2: tensorflow.metadata.v0.Schema.weighted_feature:type_name -> tensorflow.metadata.v0.WeightedFeature + 17, // 3: tensorflow.metadata.v0.Schema.string_domain:type_name -> tensorflow.metadata.v0.StringDomain + 15, // 4: tensorflow.metadata.v0.Schema.float_domain:type_name -> tensorflow.metadata.v0.FloatDomain + 14, // 5: tensorflow.metadata.v0.Schema.int_domain:type_name -> tensorflow.metadata.v0.IntDomain + 6, // 6: tensorflow.metadata.v0.Schema.annotation:type_name -> tensorflow.metadata.v0.Annotation + 8, // 7: tensorflow.metadata.v0.Schema.dataset_constraints:type_name -> tensorflow.metadata.v0.DatasetConstraints + 31, // 8: tensorflow.metadata.v0.Schema.tensor_representation_group:type_name -> tensorflow.metadata.v0.Schema.TensorRepresentationGroupEntry + 25, // 9: tensorflow.metadata.v0.Feature.presence:type_name -> tensorflow.metadata.v0.FeaturePresence + 26, // 10: tensorflow.metadata.v0.Feature.group_presence:type_name -> tensorflow.metadata.v0.FeaturePresenceWithinGroup + 9, // 11: tensorflow.metadata.v0.Feature.shape:type_name -> tensorflow.metadata.v0.FixedShape + 10, // 12: tensorflow.metadata.v0.Feature.value_count:type_name -> tensorflow.metadata.v0.ValueCount + 1, // 13: tensorflow.metadata.v0.Feature.type:type_name -> tensorflow.metadata.v0.FeatureType + 14, // 14: tensorflow.metadata.v0.Feature.int_domain:type_name -> tensorflow.metadata.v0.IntDomain + 15, // 15: tensorflow.metadata.v0.Feature.float_domain:type_name -> tensorflow.metadata.v0.FloatDomain + 17, // 16: tensorflow.metadata.v0.Feature.string_domain:type_name -> tensorflow.metadata.v0.StringDomain + 18, // 17: tensorflow.metadata.v0.Feature.bool_domain:type_name -> tensorflow.metadata.v0.BoolDomain + 16, // 18: tensorflow.metadata.v0.Feature.struct_domain:type_name -> tensorflow.metadata.v0.StructDomain + 19, // 19: tensorflow.metadata.v0.Feature.natural_language_domain:type_name -> tensorflow.metadata.v0.NaturalLanguageDomain + 20, // 20: tensorflow.metadata.v0.Feature.image_domain:type_name -> tensorflow.metadata.v0.ImageDomain + 21, // 21: tensorflow.metadata.v0.Feature.mid_domain:type_name -> tensorflow.metadata.v0.MIDDomain + 22, // 22: tensorflow.metadata.v0.Feature.url_domain:type_name -> tensorflow.metadata.v0.URLDomain + 23, // 23: tensorflow.metadata.v0.Feature.time_domain:type_name -> tensorflow.metadata.v0.TimeDomain + 24, // 24: tensorflow.metadata.v0.Feature.time_of_day_domain:type_name -> tensorflow.metadata.v0.TimeOfDayDomain + 13, // 25: tensorflow.metadata.v0.Feature.distribution_constraints:type_name -> tensorflow.metadata.v0.DistributionConstraints + 6, // 26: tensorflow.metadata.v0.Feature.annotation:type_name -> tensorflow.metadata.v0.Annotation + 28, // 27: tensorflow.metadata.v0.Feature.skew_comparator:type_name -> tensorflow.metadata.v0.FeatureComparator + 28, // 28: tensorflow.metadata.v0.Feature.drift_comparator:type_name -> tensorflow.metadata.v0.FeatureComparator + 0, // 29: tensorflow.metadata.v0.Feature.lifecycle_stage:type_name -> tensorflow.metadata.v0.LifecycleStage + 40, // 30: tensorflow.metadata.v0.Annotation.extra_metadata:type_name -> google.protobuf.Any + 7, // 31: tensorflow.metadata.v0.DatasetConstraints.num_examples_drift_comparator:type_name -> tensorflow.metadata.v0.NumericValueComparator + 7, // 32: tensorflow.metadata.v0.DatasetConstraints.num_examples_version_comparator:type_name -> tensorflow.metadata.v0.NumericValueComparator + 32, // 33: tensorflow.metadata.v0.FixedShape.dim:type_name -> tensorflow.metadata.v0.FixedShape.Dim + 41, // 34: tensorflow.metadata.v0.WeightedFeature.feature:type_name -> tensorflow.metadata.v0.Path + 41, // 35: tensorflow.metadata.v0.WeightedFeature.weight_feature:type_name -> tensorflow.metadata.v0.Path + 0, // 36: tensorflow.metadata.v0.WeightedFeature.lifecycle_stage:type_name -> tensorflow.metadata.v0.LifecycleStage + 0, // 37: tensorflow.metadata.v0.SparseFeature.lifecycle_stage:type_name -> tensorflow.metadata.v0.LifecycleStage + 25, // 38: tensorflow.metadata.v0.SparseFeature.presence:type_name -> tensorflow.metadata.v0.FeaturePresence + 9, // 39: tensorflow.metadata.v0.SparseFeature.dense_shape:type_name -> tensorflow.metadata.v0.FixedShape + 33, // 40: tensorflow.metadata.v0.SparseFeature.index_feature:type_name -> tensorflow.metadata.v0.SparseFeature.IndexFeature + 34, // 41: tensorflow.metadata.v0.SparseFeature.value_feature:type_name -> tensorflow.metadata.v0.SparseFeature.ValueFeature + 1, // 42: tensorflow.metadata.v0.SparseFeature.type:type_name -> tensorflow.metadata.v0.FeatureType + 5, // 43: tensorflow.metadata.v0.StructDomain.feature:type_name -> tensorflow.metadata.v0.Feature + 12, // 44: tensorflow.metadata.v0.StructDomain.sparse_feature:type_name -> tensorflow.metadata.v0.SparseFeature + 2, // 45: tensorflow.metadata.v0.TimeDomain.integer_format:type_name -> tensorflow.metadata.v0.TimeDomain.IntegerTimeFormat + 3, // 46: tensorflow.metadata.v0.TimeOfDayDomain.integer_format:type_name -> tensorflow.metadata.v0.TimeOfDayDomain.IntegerTimeOfDayFormat + 27, // 47: tensorflow.metadata.v0.FeatureComparator.infinity_norm:type_name -> tensorflow.metadata.v0.InfinityNorm + 36, // 48: tensorflow.metadata.v0.TensorRepresentation.dense_tensor:type_name -> tensorflow.metadata.v0.TensorRepresentation.DenseTensor + 37, // 49: tensorflow.metadata.v0.TensorRepresentation.varlen_sparse_tensor:type_name -> tensorflow.metadata.v0.TensorRepresentation.VarLenSparseTensor + 38, // 50: tensorflow.metadata.v0.TensorRepresentation.sparse_tensor:type_name -> tensorflow.metadata.v0.TensorRepresentation.SparseTensor + 39, // 51: tensorflow.metadata.v0.TensorRepresentationGroup.tensor_representation:type_name -> tensorflow.metadata.v0.TensorRepresentationGroup.TensorRepresentationEntry + 30, // 52: tensorflow.metadata.v0.Schema.TensorRepresentationGroupEntry.value:type_name -> tensorflow.metadata.v0.TensorRepresentationGroup + 9, // 53: tensorflow.metadata.v0.TensorRepresentation.DenseTensor.shape:type_name -> tensorflow.metadata.v0.FixedShape + 35, // 54: tensorflow.metadata.v0.TensorRepresentation.DenseTensor.default_value:type_name -> tensorflow.metadata.v0.TensorRepresentation.DefaultValue + 9, // 55: tensorflow.metadata.v0.TensorRepresentation.SparseTensor.dense_shape:type_name -> tensorflow.metadata.v0.FixedShape + 29, // 56: tensorflow.metadata.v0.TensorRepresentationGroup.TensorRepresentationEntry.value:type_name -> tensorflow.metadata.v0.TensorRepresentation + 57, // [57:57] is the sub-list for method output_type + 57, // [57:57] is the sub-list for method input_type + 57, // [57:57] is the sub-list for extension type_name + 57, // [57:57] is the sub-list for extension extendee + 0, // [0:57] is the sub-list for field type_name +} + +func init() { file_tensorflow_metadata_proto_v0_schema_proto_init() } +func file_tensorflow_metadata_proto_v0_schema_proto_init() { + if File_tensorflow_metadata_proto_v0_schema_proto != nil { + return + } + file_tensorflow_metadata_proto_v0_path_proto_init() + if !protoimpl.UnsafeEnabled { + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Schema); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Feature); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[2].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Annotation); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[3].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*NumericValueComparator); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[4].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*DatasetConstraints); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[5].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*FixedShape); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[6].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ValueCount); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[7].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*WeightedFeature); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[8].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*SparseFeature); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[9].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*DistributionConstraints); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[10].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*IntDomain); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[11].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*FloatDomain); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[12].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*StructDomain); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[13].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*StringDomain); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[14].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*BoolDomain); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[15].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*NaturalLanguageDomain); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[16].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*ImageDomain); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[17].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*MIDDomain); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[18].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*URLDomain); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[19].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*TimeDomain); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[20].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*TimeOfDayDomain); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[21].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*FeaturePresence); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[22].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*FeaturePresenceWithinGroup); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[23].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*InfinityNorm); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[24].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*FeatureComparator); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[25].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*TensorRepresentation); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[26].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*TensorRepresentationGroup); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[28].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*FixedShape_Dim); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[29].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*SparseFeature_IndexFeature); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[30].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*SparseFeature_ValueFeature); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[31].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*TensorRepresentation_DefaultValue); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[32].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*TensorRepresentation_DenseTensor); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[33].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*TensorRepresentation_VarLenSparseTensor); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[34].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*TensorRepresentation_SparseTensor); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[1].OneofWrappers = []interface{}{ + (*Feature_Presence)(nil), + (*Feature_GroupPresence)(nil), + (*Feature_Shape)(nil), + (*Feature_ValueCount)(nil), + (*Feature_Domain)(nil), + (*Feature_IntDomain)(nil), + (*Feature_FloatDomain)(nil), + (*Feature_StringDomain)(nil), + (*Feature_BoolDomain)(nil), + (*Feature_StructDomain)(nil), + (*Feature_NaturalLanguageDomain)(nil), + (*Feature_ImageDomain)(nil), + (*Feature_MidDomain)(nil), + (*Feature_UrlDomain)(nil), + (*Feature_TimeDomain)(nil), + (*Feature_TimeOfDayDomain)(nil), + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[19].OneofWrappers = []interface{}{ + (*TimeDomain_StringFormat)(nil), + (*TimeDomain_IntegerFormat)(nil), + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[20].OneofWrappers = []interface{}{ + (*TimeOfDayDomain_StringFormat)(nil), + (*TimeOfDayDomain_IntegerFormat)(nil), + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[25].OneofWrappers = []interface{}{ + (*TensorRepresentation_DenseTensor_)(nil), + (*TensorRepresentation_VarlenSparseTensor)(nil), + (*TensorRepresentation_SparseTensor_)(nil), + } + file_tensorflow_metadata_proto_v0_schema_proto_msgTypes[31].OneofWrappers = []interface{}{ + (*TensorRepresentation_DefaultValue_FloatValue)(nil), + (*TensorRepresentation_DefaultValue_IntValue)(nil), + (*TensorRepresentation_DefaultValue_BytesValue)(nil), + (*TensorRepresentation_DefaultValue_UintValue)(nil), + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_tensorflow_metadata_proto_v0_schema_proto_rawDesc, + NumEnums: 4, + NumMessages: 36, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_tensorflow_metadata_proto_v0_schema_proto_goTypes, + DependencyIndexes: file_tensorflow_metadata_proto_v0_schema_proto_depIdxs, + EnumInfos: file_tensorflow_metadata_proto_v0_schema_proto_enumTypes, + MessageInfos: file_tensorflow_metadata_proto_v0_schema_proto_msgTypes, + }.Build() + File_tensorflow_metadata_proto_v0_schema_proto = out.File + file_tensorflow_metadata_proto_v0_schema_proto_rawDesc = nil + file_tensorflow_metadata_proto_v0_schema_proto_goTypes = nil + file_tensorflow_metadata_proto_v0_schema_proto_depIdxs = nil +} From eb04874982d858ac724921ef974e25c3db801653 Mon Sep 17 00:00:00 2001 From: Anders Eriksson Date: Thu, 16 Apr 2020 03:01:48 +0200 Subject: [PATCH 116/176] Correct links to why-feast and concepts doc site (#627) --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 881ccea49b1..dd44db66ebf 100644 --- a/README.md +++ b/README.md @@ -50,8 +50,8 @@ Please see the links below to set up Feast for batch/historical serving with Big Please refer to the official documentation at - * [Why Feast?](https://docs.feast.dev/why-feast) - * [Concepts](https://docs.feast.dev/concepts) + * [Why Feast?](https://docs.feast.dev/introduction/why-feast) + * [Concepts](https://docs.feast.dev/concepts/concepts) * [Installation](https://docs.feast.dev/installation/overview) * [Examples](https://github.com/gojek/feast/blob/master/examples/) * [Roadmap](https://docs.feast.dev/roadmap) From e461cde7a752ec2c8ad4979dd2610daff3f47934 Mon Sep 17 00:00:00 2001 From: Zhu Zhan Yan Date: Thu, 16 Apr 2020 11:02:48 +0800 Subject: [PATCH 117/176] Make error on retrieval of nonexistent feature humanly readable (#625) * Make ServingServiceGRpcController return NOT_FOUND status code on spec retrieval error. * Format python sdk with isort * Improve error message on spec retrieval exception for added clarity to the user. * Added Grpc error handlers to python client to show feature retrieval errors from feast serving. * Fixed typo in client: self._serving_url should be self.serving_url * Fixed misplaced try catch block in python Client's get_batch_features() * Remove max age part in the error message thrown on invalid feature ref. * Change redudant .getOrDefault() in CachedSpecService to .get() Co-authored-by: Zhu Zhanyan --- sdk/python/feast/cli.py | 2 +- sdk/python/feast/client.py | 38 +++++++++++-------- sdk/python/feast/feature_set.py | 5 +-- sdk/python/feast/job.py | 14 +++---- sdk/python/tests/test_client.py | 8 ++-- sdk/python/tests/test_feature_set.py | 2 +- .../ServingServiceGRpcController.java | 10 +++++ .../serving/specs/CachedSpecService.java | 14 ++++--- 8 files changed, 56 insertions(+), 37 deletions(-) diff --git a/sdk/python/feast/cli.py b/sdk/python/feast/cli.py index cd1146b4810..489d28fe833 100644 --- a/sdk/python/feast/cli.py +++ b/sdk/python/feast/cli.py @@ -22,9 +22,9 @@ from feast.client import Client from feast.config import Config +from feast.core.IngestionJob_pb2 import IngestionJobStatus from feast.feature_set import FeatureSet, FeatureSetRef from feast.loaders.yaml import yaml_loader -from feast.core.IngestionJob_pb2 import IngestionJobStatus _logger = logging.getLogger(__name__) diff --git a/sdk/python/feast/client.py b/sdk/python/feast/client.py index f5aed118cfd..0a38236a510 100644 --- a/sdk/python/feast/client.py +++ b/sdk/python/feast/client.py @@ -48,16 +48,16 @@ GetFeatureSetResponse, ListFeatureSetsRequest, ListFeatureSetsResponse, + ListIngestionJobsRequest, ListProjectsRequest, ListProjectsResponse, - ListIngestionJobsRequest, RestartIngestionJobRequest, StopIngestionJobRequest, ) from feast.core.CoreService_pb2_grpc import CoreServiceStub from feast.core.FeatureSet_pb2 import FeatureSetStatus from feast.feature_set import Entity, FeatureSet, FeatureSetRef -from feast.job import RetrievalJob, IngestJob +from feast.job import IngestJob, RetrievalJob from feast.loaders.abstract_producer import get_producer from feast.loaders.file import export_source_to_staging_location from feast.loaders.ingest import KAFKA_CHUNK_PRODUCTION_TIMEOUT, get_feature_row_chunks @@ -566,8 +566,8 @@ def get_batch_features( if serving_info.type != FeastServingType.FEAST_SERVING_TYPE_BATCH: raise Exception( - f'You are connected to a store "{self._serving_url}" which ' - f"does not support batch retrieval " + f'You are connected to a store "{self.serving_url}" which ' + f"does not support batch retrieval" ) if isinstance(entity_rows, pd.DataFrame): @@ -597,7 +597,6 @@ def get_batch_features( staged_files = export_source_to_staging_location( entity_rows, serving_info.job_staging_location ) # type: List[str] - request = GetBatchFeaturesRequest( features=feature_references, dataset_source=DatasetSource( @@ -608,7 +607,11 @@ def get_batch_features( ) # Retrieve Feast Job object to manage life cycle of retrieval - response = self._serving_service_stub.GetBatchFeatures(request) + try: + response = self._serving_service_stub.GetBatchFeatures(request) + except grpc.RpcError as e: + raise grpc.RpcError(e.details()) + return RetrievalJob(response.job, self._serving_service_stub) def get_online_features( @@ -639,17 +642,22 @@ def get_online_features( """ self._connect_serving() - return self._serving_service_stub.GetOnlineFeatures( - GetOnlineFeaturesRequest( - features=_build_feature_references( - feature_refs=feature_refs, - default_project=( - default_project if not self.project else self.project + try: + response = self._serving_service_stub.GetOnlineFeatures( + GetOnlineFeaturesRequest( + features=_build_feature_references( + feature_refs=feature_refs, + default_project=( + default_project if not self.project else self.project + ), ), - ), - entity_rows=entity_rows, + entity_rows=entity_rows, + ) ) - ) + except grpc.RpcError as e: + raise grpc.RpcError(e.details()) + + return response def list_ingest_jobs( self, diff --git a/sdk/python/feast/feature_set.py b/sdk/python/feast/feature_set.py index c6104f47a08..760e947318f 100644 --- a/sdk/python/feast/feature_set.py +++ b/sdk/python/feast/feature_set.py @@ -13,8 +13,7 @@ # limitations under the License. import warnings from collections import OrderedDict -from typing import Dict -from typing import List, Optional +from typing import Dict, List, Optional import pandas as pd import pyarrow as pa @@ -24,7 +23,6 @@ from google.protobuf.message import Message from pandas.api.types import is_datetime64_ns_dtype from pyarrow.lib import TimestampType -from tensorflow_metadata.proto.v0 import schema_pb2 from feast.core.FeatureSet_pb2 import FeatureSet as FeatureSetProto from feast.core.FeatureSet_pb2 import FeatureSetMeta as FeatureSetMetaProto @@ -41,6 +39,7 @@ pa_to_feast_value_type, python_type_to_feast_value_type, ) +from tensorflow_metadata.proto.v0 import schema_pb2 class FeatureSet: diff --git a/sdk/python/feast/job.py b/sdk/python/feast/job.py index 3576bc1b385..f2e31c709e2 100644 --- a/sdk/python/feast/job.py +++ b/sdk/python/feast/job.py @@ -1,16 +1,20 @@ import tempfile import time from datetime import datetime, timedelta -from urllib.parse import urlparse from typing import List +from urllib.parse import urlparse import fastavro import pandas as pd from google.cloud import storage from google.protobuf.json_format import MessageToJson +from feast.core.CoreService_pb2 import ListIngestionJobsRequest +from feast.core.CoreService_pb2_grpc import CoreServiceStub +from feast.core.IngestionJob_pb2 import IngestionJob as IngestJobProto +from feast.core.IngestionJob_pb2 import IngestionJobStatus +from feast.core.Store_pb2 import Store from feast.feature_set import FeatureSet -from feast.source import Source from feast.serving.ServingService_pb2 import ( DATA_FORMAT_AVRO, JOB_STATUS_DONE, @@ -18,11 +22,7 @@ ) from feast.serving.ServingService_pb2 import Job as JobProto from feast.serving.ServingService_pb2_grpc import ServingServiceStub -from feast.core.Store_pb2 import Store -from feast.core.IngestionJob_pb2 import IngestionJob as IngestJobProto -from feast.core.IngestionJob_pb2 import IngestionJobStatus -from feast.core.CoreService_pb2_grpc import CoreServiceStub -from feast.core.CoreService_pb2 import ListIngestionJobsRequest +from feast.source import Source # Maximum no of seconds to wait until the retrieval jobs status is DONE in Feast # Currently set to the maximum query execution time limit in BigQuery diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index f7f5676ced5..ed0426b2f6a 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -31,18 +31,16 @@ GetFeatureSetResponse, ListIngestionJobsResponse, ) -from feast.core.Store_pb2 import Store -from feast.core.IngestionJob_pb2 import ( - IngestionJob as IngestJobProto, - IngestionJobStatus, -) from feast.core.FeatureSet_pb2 import EntitySpec as EntitySpecProto from feast.core.FeatureSet_pb2 import FeatureSet as FeatureSetProto from feast.core.FeatureSet_pb2 import FeatureSetMeta as FeatureSetMetaProto from feast.core.FeatureSet_pb2 import FeatureSetSpec as FeatureSetSpecProto from feast.core.FeatureSet_pb2 import FeatureSetStatus as FeatureSetStatusProto from feast.core.FeatureSet_pb2 import FeatureSpec as FeatureSpecProto +from feast.core.IngestionJob_pb2 import IngestionJob as IngestJobProto +from feast.core.IngestionJob_pb2 import IngestionJobStatus from feast.core.Source_pb2 import KafkaSourceConfig, Source, SourceType +from feast.core.Store_pb2 import Store from feast.entity import Entity from feast.feature_set import Feature, FeatureSet, FeatureSetRef from feast.job import IngestJob diff --git a/sdk/python/tests/test_feature_set.py b/sdk/python/tests/test_feature_set.py index 6f087d98bbf..0a7d1ebabea 100644 --- a/sdk/python/tests/test_feature_set.py +++ b/sdk/python/tests/test_feature_set.py @@ -20,7 +20,6 @@ import pytest import pytz from google.protobuf import json_format -from tensorflow_metadata.proto.v0 import schema_pb2 import dataframes import feast.core.CoreService_pb2_grpc as Core @@ -34,6 +33,7 @@ ) from feast.value_type import ValueType from feast_core_server import CoreServicer +from tensorflow_metadata.proto.v0 import schema_pb2 CORE_URL = "core.feast.local" SERVING_URL = "serving.feast.local" diff --git a/serving/src/main/java/feast/serving/controller/ServingServiceGRpcController.java b/serving/src/main/java/feast/serving/controller/ServingServiceGRpcController.java index cc1f856d728..0eba67d4b4e 100644 --- a/serving/src/main/java/feast/serving/controller/ServingServiceGRpcController.java +++ b/serving/src/main/java/feast/serving/controller/ServingServiceGRpcController.java @@ -26,9 +26,11 @@ import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest; import feast.serving.ServingAPIProto.GetOnlineFeaturesResponse; import feast.serving.ServingServiceGrpc.ServingServiceImplBase; +import feast.serving.exception.SpecRetrievalException; import feast.serving.interceptors.GrpcMonitoringInterceptor; import feast.serving.service.ServingService; import feast.serving.util.RequestHelper; +import io.grpc.Status; import io.grpc.stub.StreamObserver; import io.opentracing.Scope; import io.opentracing.Span; @@ -74,6 +76,10 @@ public void getOnlineFeatures( GetOnlineFeaturesResponse onlineFeatures = servingService.getOnlineFeatures(request); responseObserver.onNext(onlineFeatures); responseObserver.onCompleted(); + } catch (SpecRetrievalException e) { + log.error("Failed to retrieve specs in SpecService", e); + responseObserver.onError( + Status.NOT_FOUND.withDescription(e.getMessage()).withCause(e).asException()); } catch (Exception e) { log.warn("Failed to get Online Features", e); responseObserver.onError(e); @@ -89,6 +95,10 @@ public void getBatchFeatures( GetBatchFeaturesResponse batchFeatures = servingService.getBatchFeatures(request); responseObserver.onNext(batchFeatures); responseObserver.onCompleted(); + } catch (SpecRetrievalException e) { + log.error("Failed to retrieve specs in SpecService", e); + responseObserver.onError( + Status.NOT_FOUND.withDescription(e.getMessage()).withCause(e).asException()); } catch (Exception e) { log.warn("Failed to get Batch Features", e); responseObserver.onError(e); diff --git a/serving/src/main/java/feast/serving/specs/CachedSpecService.java b/serving/src/main/java/feast/serving/specs/CachedSpecService.java index 47f4934d52c..246be8c5fdd 100644 --- a/serving/src/main/java/feast/serving/specs/CachedSpecService.java +++ b/serving/src/main/java/feast/serving/specs/CachedSpecService.java @@ -118,11 +118,15 @@ public List getFeatureSets(List featureRefe .map( featureReference -> { String featureSet = - featureToFeatureSetMapping.getOrDefault( - generateFeatureStringRef(featureReference), ""); - if (featureSet.isEmpty()) { + featureToFeatureSetMapping.get(generateFeatureStringRef(featureReference)); + if (featureSet == null) { throw new SpecRetrievalException( - String.format("Unable to retrieve feature %s", featureReference)); + String.format( + "Unable to find feature set for feature ref: " + + "(project: %s, name: %s, version: %d)", + featureReference.getProject(), + featureReference.getName(), + featureReference.getVersion())); } return Pair.of(featureSet, featureReference); }) @@ -141,7 +145,7 @@ public List getFeatureSets(List featureRefe featureSetRequests.add(featureSetRequest); } catch (ExecutionException e) { throw new SpecRetrievalException( - String.format("Unable to retrieve featureSet with id %s", fsName), e); + String.format("Unable to find featureSet with name: %s", fsName), e); } }); return featureSetRequests; From 3fd6c7abfa625aefb6cddc55b5270c27714d62c1 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Thu, 16 Apr 2020 15:25:48 +0800 Subject: [PATCH 118/176] Clean up Feast configuration (#611) * Add validation to Core configuration and fix version loading Refactor, document, and validate Feast Core Properties Refactor FeastProperties to support nested store configuration Localize all store configuration in Serving in Spring configuration Various configuration updates * Allow Feast Serving to use types properties instead of maps * Reuse Feast Core Store model in serving * Remove redundant config classes for Redis * Update Serving Beans and Config classes to use ne1w configuration getters * Remove hot-loading from store configuration. This reduces a bit of flexibility, but simplifies the code and configuration * Set default build version in Feast Core "version" field in Feast Properties * Ensure FeatureSink creation is consistent for both Redis and BigQuery * Move BigQueryHistoricalRetriever configuration into Retriever from ServingServiceConfig * Allow a list of stores to be configured for forward compability * Remove Lombok from Serving configuration * Update Store configuration loading in serving to use a store model * Update RedisBackedJobService to instantiate its own Redis Client * Update comments in FeastProperties * Fix broken default application.yml and add comments in Serving * Refactored and cleaned up Feast Core configuration for job runners. * Remove commented out DataflowRunnerConfig setters * Clean up getJobManager and simplify field mapping in DataflowRunnerConfig * Add static factory methods to retrievers * Remove runner specific comment typo * Add oneOfStrings validator annotation for configuration validation * Fix broken Dataflow unit test that depends on GOOGLE_APPLICATION_CREDENTIALS --- core/pom.xml | 24 + .../feast/core/config/FeastProperties.java | 188 +++++- .../core/config/FeatureStreamConfig.java | 9 +- .../java/feast/core/config/JobConfig.java | 70 +-- .../core/job/dataflow/DataflowJobConfig.java | 25 - .../core/job/dataflow/DataflowJobManager.java | 48 +- .../job/dataflow/DataflowRunnerConfig.java | 113 ++++ .../core/service/JobCoordinatorService.java | 13 +- .../core/validators/OneOfStringValidator.java | 51 ++ .../feast/core/validators/OneOfStrings.java | 49 ++ core/src/main/resources/application.yml | 46 +- .../job/dataflow/DataflowJobManagerTest.java | 15 +- .../service/JobCoordinatorServiceTest.java | 19 +- .../java/feast/ingestion/utils/StoreUtil.java | 7 +- pom.xml | 8 + protos/feast/core/Runner.proto | 73 +++ protos/feast/core/Source.proto | 10 +- protos/feast/core/Store.proto | 3 + serving/lombok.config | 1 - serving/pom.xml | 11 +- serving/sample_redis_config.yml | 9 - .../java/feast/serving/FeastProperties.java | 191 ------- .../feast/serving/ServingApplication.java | 11 +- .../ContextClosedHandler.java | 2 +- .../feast/serving/config/FeastProperties.java | 541 ++++++++++++++++++ .../InstrumentationConfig.java | 3 +- .../JobServiceConfig.java | 27 +- .../ServingApiConfiguration.java | 2 +- .../serving/config/ServingServiceConfig.java | 79 +++ .../SpecServiceConfig.java | 15 +- .../configuration/ServingServiceConfig.java | 138 ----- .../configuration/StoreConfiguration.java | 47 -- .../redis/JobStoreRedisConfig.java | 68 --- .../redis/ServingStoreRedisConfig.java | 62 -- .../ServingServiceGRpcController.java | 2 +- .../ServingServiceRestController.java | 2 +- .../service/RedisBackedJobService.java | 15 + .../serving/specs/CachedSpecService.java | 39 +- .../util/mappers/YamlToProtoMapper.java | 37 -- serving/src/main/resources/application.yml | 87 +-- .../ServingServiceGRpcControllerTest.java | 2 +- .../service/CachedSpecServiceTest.java | 33 +- .../service/RedisBackedJobServiceTest.java | 3 +- .../util/mappers/YamlToProtoMapperTest.java | 54 -- .../BigQueryHistoricalRetriever.java | 32 ++ .../bigquery/writer/BigQueryFeatureSink.java | 4 +- .../redis/retriever/RedisOnlineRetriever.java | 19 +- .../redis/writer/RedisFeatureSink.java | 14 + .../retriever/RedisOnlineRetrieverTest.java | 9 +- 49 files changed, 1432 insertions(+), 898 deletions(-) delete mode 100644 core/src/main/java/feast/core/job/dataflow/DataflowJobConfig.java create mode 100644 core/src/main/java/feast/core/job/dataflow/DataflowRunnerConfig.java create mode 100644 core/src/main/java/feast/core/validators/OneOfStringValidator.java create mode 100644 core/src/main/java/feast/core/validators/OneOfStrings.java create mode 100644 protos/feast/core/Runner.proto delete mode 100644 serving/lombok.config delete mode 100644 serving/sample_redis_config.yml delete mode 100644 serving/src/main/java/feast/serving/FeastProperties.java rename serving/src/main/java/feast/serving/{configuration => config}/ContextClosedHandler.java (96%) create mode 100644 serving/src/main/java/feast/serving/config/FeastProperties.java rename serving/src/main/java/feast/serving/{configuration => config}/InstrumentationConfig.java (96%) rename serving/src/main/java/feast/serving/{configuration => config}/JobServiceConfig.java (56%) rename serving/src/main/java/feast/serving/{configuration => config}/ServingApiConfiguration.java (97%) create mode 100644 serving/src/main/java/feast/serving/config/ServingServiceConfig.java rename serving/src/main/java/feast/serving/{configuration => config}/SpecServiceConfig.java (87%) delete mode 100644 serving/src/main/java/feast/serving/configuration/ServingServiceConfig.java delete mode 100644 serving/src/main/java/feast/serving/configuration/StoreConfiguration.java delete mode 100644 serving/src/main/java/feast/serving/configuration/redis/JobStoreRedisConfig.java delete mode 100644 serving/src/main/java/feast/serving/configuration/redis/ServingStoreRedisConfig.java delete mode 100644 serving/src/main/java/feast/serving/util/mappers/YamlToProtoMapper.java delete mode 100644 serving/src/test/java/feast/serving/util/mappers/YamlToProtoMapperTest.java diff --git a/core/pom.xml b/core/pom.xml index 7961b45074b..f4fb6c659c0 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -38,6 +38,14 @@ false + + + build-info + + build-info + + + @@ -207,5 +215,21 @@ jaxb-api + + javax.validation + validation-api + 2.0.0.Final + + + org.hibernate.validator + hibernate-validator + 6.1.2.Final + + + org.hibernate.validator + hibernate-validator-annotation-processor + 6.1.2.Final + + diff --git a/core/src/main/java/feast/core/config/FeastProperties.java b/core/src/main/java/feast/core/config/FeastProperties.java index b9c787b6c77..941d51f68c9 100644 --- a/core/src/main/java/feast/core/config/FeastProperties.java +++ b/core/src/main/java/feast/core/config/FeastProperties.java @@ -16,53 +16,211 @@ */ package feast.core.config; -import java.util.Map; +import feast.core.config.FeastProperties.StreamProperties.FeatureStreamOptions; +import feast.core.validators.OneOfStrings; +import java.util.*; +import javax.annotation.PostConstruct; +import javax.validation.*; +import javax.validation.constraints.NotBlank; +import javax.validation.constraints.NotNull; +import javax.validation.constraints.Positive; import lombok.Getter; import lombok.Setter; +import org.hibernate.validator.constraints.URL; +import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.context.properties.ConfigurationProperties; +import org.springframework.boot.info.BuildProperties; @Getter @Setter @ConfigurationProperties(prefix = "feast", ignoreInvalidFields = true) public class FeastProperties { - private String version; - private JobProperties jobs; + /** + * Instantiates a new Feast properties. + * + * @param buildProperties Feast build properties + */ + @Autowired + public FeastProperties(BuildProperties buildProperties) { + setVersion(buildProperties.getVersion()); + } + + /** Instantiates a new Feast properties. */ + public FeastProperties() {} + + /* Feast Core Build Version */ + @NotBlank private String version = "unknown"; + + /* Population job properties */ + @NotNull private JobProperties jobs; + + @NotNull + /* Feast Kafka stream properties */ private StreamProperties stream; + /** Feast job properties. These properties are used for ingestion jobs. */ @Getter @Setter public static class JobProperties { - private String runner; - private Map options; + @NotBlank + /* The active Apache Beam runner name. This name references one instance of the Runner class */ + private String activeRunner; + + /** List of configured job runners. */ + private List runners = new ArrayList<>(); + + /** + * Gets a {@link Runner} instance of the active runner + * + * @return the active runner + */ + public Runner getActiveRunner() { + for (Runner runner : getRunners()) { + if (activeRunner.equals(runner.getName())) { + return runner; + } + } + throw new RuntimeException( + String.format( + "Active runner is misconfigured. Could not find runner: %s.", activeRunner)); + } + + /** Job Runner class. */ + @Getter + @Setter + public static class Runner { + /** Job runner name. This must be unique. */ + String name; + + /** Job runner type DirectRunner, DataflowRunner currently supported */ + String type; + + /** + * Job runner configuration options. See the following for options + * https://api.docs.feast.dev/grpc/feast.core.pb.html#Runner + */ + Map options = new HashMap<>(); + + /** + * Gets the job runner type as an enum. + * + * @return Returns the job runner type as {@link feast.core.job.Runner} + */ + public feast.core.job.Runner getType() { + return feast.core.job.Runner.fromString(type); + } + } + + @NotNull + /* Population job metric properties */ private MetricsProperties metrics; - private JobUpdatesProperties updates; - } - @Getter - @Setter - public static class JobUpdatesProperties { + /* Timeout in seconds for each attempt to update or submit a new job to the runner */ + @Positive private long jobUpdateTimeoutSeconds; - private long timeoutSeconds; - private long pollingIntervalMillis; + /* Job update polling interval in millisecond. How frequently Feast will update running jobs. */ + @Positive private long pollingIntervalMilliseconds; } + /** Properties used to configure Feast's managed Kafka feature stream. */ @Getter @Setter public static class StreamProperties { + /* Feature stream type. Only "kafka" is supported. */ + @OneOfStrings({"kafka"}) + @NotBlank private String type; - private Map options; + + /* Feature stream options */ + @NotNull private FeatureStreamOptions options; + + /** Feature stream options */ + @Getter + @Setter + public static class FeatureStreamOptions { + + /* Kafka topic to use for feature sets without source topics. */ + @NotBlank private String topic = "feast-features"; + + /** + * Comma separated list of Kafka bootstrap servers. Used for feature sets without a defined + * source. + */ + @NotBlank private String bootstrapServers = "localhost:9092"; + + /* Defines the number of copies of managed feature stream Kafka. */ + @Positive private short replicationFactor = 1; + + /* Number of Kafka partitions to to use for managed feature stream. */ + @Positive private int partitions = 1; + } } + /** Feast population job metrics */ @Getter @Setter public static class MetricsProperties { + /* Population job metrics enabled */ private boolean enabled; + + /* Metric type. Possible options: statsd */ + @OneOfStrings({"statsd"}) + @NotBlank private String type; - private String host; - private int port; + + /* Host of metric sink */ + @URL private String host; + + /* Port of metric sink */ + @Positive private int port; + } + + /** + * Validates all FeastProperties. This method runs after properties have been initialized and + * individually and conditionally validates each class. + */ + @PostConstruct + public void validate() { + ValidatorFactory factory = Validation.buildDefaultValidatorFactory(); + Validator validator = factory.getValidator(); + + // Validate root fields in FeastProperties + Set> violations = validator.validate(this); + if (!violations.isEmpty()) { + throw new ConstraintViolationException(violations); + } + + // Validate Stream properties + Set> streamPropertyViolations = + validator.validate(getStream()); + if (!streamPropertyViolations.isEmpty()) { + throw new ConstraintViolationException(streamPropertyViolations); + } + + // Validate Stream Options + Set> featureStreamOptionsViolations = + validator.validate(getStream().getOptions()); + if (!featureStreamOptionsViolations.isEmpty()) { + throw new ConstraintViolationException(featureStreamOptionsViolations); + } + + // Validate JobProperties + Set> jobPropertiesViolations = validator.validate(getJobs()); + if (!jobPropertiesViolations.isEmpty()) { + throw new ConstraintViolationException(jobPropertiesViolations); + } + + // Validate MetricsProperties + if (getJobs().getMetrics().isEnabled()) { + Set> jobMetricViolations = + validator.validate(getJobs().getMetrics()); + if (!jobMetricViolations.isEmpty()) { + throw new ConstraintViolationException(jobMetricViolations); + } + } } } diff --git a/core/src/main/java/feast/core/config/FeatureStreamConfig.java b/core/src/main/java/feast/core/config/FeatureStreamConfig.java index 45de359ac76..44f0e0e0993 100644 --- a/core/src/main/java/feast/core/config/FeatureStreamConfig.java +++ b/core/src/main/java/feast/core/config/FeatureStreamConfig.java @@ -48,8 +48,8 @@ public Source getDefaultSource(FeastProperties feastProperties) { SourceType featureStreamType = SourceType.valueOf(streamProperties.getType().toUpperCase()); switch (featureStreamType) { case KAFKA: - String bootstrapServers = streamProperties.getOptions().get("bootstrapServers"); - String topicName = streamProperties.getOptions().get("topic"); + String bootstrapServers = streamProperties.getOptions().getBootstrapServers(); + String topicName = streamProperties.getOptions().getTopic(); Map map = new HashMap<>(); map.put(AdminClientConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers); map.put( @@ -59,9 +59,8 @@ public Source getDefaultSource(FeastProperties feastProperties) { NewTopic newTopic = new NewTopic( topicName, - Integer.valueOf(streamProperties.getOptions().getOrDefault("numPartitions", "1")), - Short.valueOf( - streamProperties.getOptions().getOrDefault("replicationFactor", "1"))); + streamProperties.getOptions().getPartitions(), + streamProperties.getOptions().getReplicationFactor()); CreateTopicsResult createTopicsResult = client.createTopics(Collections.singleton(newTopic)); try { diff --git a/core/src/main/java/feast/core/config/JobConfig.java b/core/src/main/java/feast/core/config/JobConfig.java index 728fc0545bf..69636963bea 100644 --- a/core/src/main/java/feast/core/config/JobConfig.java +++ b/core/src/main/java/feast/core/config/JobConfig.java @@ -16,22 +16,11 @@ */ package feast.core.config; -import com.google.api.client.googleapis.auth.oauth2.GoogleCredential; -import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport; -import com.google.api.client.json.jackson2.JacksonFactory; -import com.google.api.services.dataflow.Dataflow; -import com.google.api.services.dataflow.DataflowScopes; -import com.google.common.base.Strings; import feast.core.config.FeastProperties.JobProperties; -import feast.core.config.FeastProperties.JobUpdatesProperties; import feast.core.job.JobManager; -import feast.core.job.Runner; import feast.core.job.dataflow.DataflowJobManager; import feast.core.job.direct.DirectJobRegistry; import feast.core.job.direct.DirectRunnerJobManager; -import java.io.IOException; -import java.security.GeneralSecurityException; -import java.util.HashMap; import java.util.Map; import lombok.extern.slf4j.Slf4j; import org.springframework.beans.factory.annotation.Autowired; @@ -44,65 +33,26 @@ public class JobConfig { /** - * Get a JobManager according to the runner type and dataflow configuration. + * Get a JobManager according to the runner type and Dataflow configuration. * * @param feastProperties feast config properties */ @Bean @Autowired - public JobManager getJobManager( - FeastProperties feastProperties, DirectJobRegistry directJobRegistry) { + public JobManager getJobManager(FeastProperties feastProperties) { JobProperties jobProperties = feastProperties.getJobs(); - Runner runner = Runner.fromString(jobProperties.getRunner()); - if (jobProperties.getOptions() == null) { - jobProperties.setOptions(new HashMap<>()); - } - Map jobOptions = jobProperties.getOptions(); - switch (runner) { - case DATAFLOW: - if (Strings.isNullOrEmpty(jobOptions.getOrDefault("region", null)) - || Strings.isNullOrEmpty(jobOptions.getOrDefault("project", null))) { - log.error("Project and location of the Dataflow runner is not configured"); - throw new IllegalStateException( - "Project and location of Dataflow runner must be specified for jobs to be run on Dataflow runner."); - } - try { - GoogleCredential credential = - GoogleCredential.getApplicationDefault().createScoped(DataflowScopes.all()); - Dataflow dataflow = - new Dataflow( - GoogleNetHttpTransport.newTrustedTransport(), - JacksonFactory.getDefaultInstance(), - credential); + FeastProperties.JobProperties.Runner runner = jobProperties.getActiveRunner(); + Map runnerConfigOptions = runner.getOptions(); + FeastProperties.MetricsProperties metrics = jobProperties.getMetrics(); - return new DataflowJobManager( - dataflow, jobProperties.getOptions(), jobProperties.getMetrics()); - } catch (IOException e) { - throw new IllegalStateException( - "Unable to find credential required for Dataflow monitoring API", e); - } catch (GeneralSecurityException e) { - throw new IllegalStateException("Security exception while connecting to Dataflow API", e); - } catch (Exception e) { - throw new IllegalStateException("Unable to initialize DataflowJobManager", e); - } + switch (runner.getType()) { + case DATAFLOW: + return new DataflowJobManager(runnerConfigOptions, metrics); case DIRECT: - return new DirectRunnerJobManager( - jobProperties.getOptions(), directJobRegistry, jobProperties.getMetrics()); + return new DirectRunnerJobManager(runnerConfigOptions, new DirectJobRegistry(), metrics); default: - throw new IllegalArgumentException("Unsupported runner: " + jobProperties.getRunner()); + throw new IllegalArgumentException("Unsupported runner: " + runner); } } - - /** Get a direct job registry */ - @Bean - public DirectJobRegistry directJobRegistry() { - return new DirectJobRegistry(); - } - - /** Extracts job update options from feast core options. */ - @Bean - public JobUpdatesProperties jobUpdatesProperties(FeastProperties feastProperties) { - return feastProperties.getJobs().getUpdates(); - } } diff --git a/core/src/main/java/feast/core/job/dataflow/DataflowJobConfig.java b/core/src/main/java/feast/core/job/dataflow/DataflowJobConfig.java deleted file mode 100644 index a9bbf345d19..00000000000 --- a/core/src/main/java/feast/core/job/dataflow/DataflowJobConfig.java +++ /dev/null @@ -1,25 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.core.job.dataflow; - -import lombok.Value; - -@Value -public class DataflowJobConfig { - private String projectId; - private String location; -} diff --git a/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java b/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java index c2313d75ecc..6002133e828 100644 --- a/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java +++ b/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java @@ -18,7 +18,12 @@ import static feast.core.util.PipelineUtil.detectClassPathResourcesToStage; +import com.google.api.client.auth.oauth2.Credential; +import com.google.api.client.googleapis.auth.oauth2.GoogleCredential; +import com.google.api.client.googleapis.javanet.GoogleNetHttpTransport; +import com.google.api.client.json.jackson2.JacksonFactory; import com.google.api.services.dataflow.Dataflow; +import com.google.api.services.dataflow.DataflowScopes; import com.google.common.base.Strings; import com.google.protobuf.InvalidProtocolBufferException; import com.google.protobuf.util.JsonFormat; @@ -37,6 +42,7 @@ import feast.ingestion.options.ImportOptions; import feast.ingestion.options.OptionCompressor; import java.io.IOException; +import java.security.GeneralSecurityException; import java.util.ArrayList; import java.util.Collections; import java.util.List; @@ -60,12 +66,46 @@ public class DataflowJobManager implements JobManager { private final MetricsProperties metrics; public DataflowJobManager( - Dataflow dataflow, Map defaultOptions, MetricsProperties metricsProperties) { - this.defaultOptions = defaultOptions; + Map runnerConfigOptions, MetricsProperties metricsProperties) { + this(runnerConfigOptions, metricsProperties, getGoogleCredential()); + } + + public DataflowJobManager( + Map runnerConfigOptions, + MetricsProperties metricsProperties, + Credential credential) { + + DataflowRunnerConfig config = new DataflowRunnerConfig(runnerConfigOptions); + + Dataflow dataflow = null; + try { + dataflow = + new Dataflow( + GoogleNetHttpTransport.newTrustedTransport(), + JacksonFactory.getDefaultInstance(), + credential); + } catch (GeneralSecurityException e) { + throw new IllegalStateException("Security exception while connecting to Dataflow API", e); + } catch (IOException e) { + throw new IllegalStateException("Unable to initialize DataflowJobManager", e); + } + + this.defaultOptions = runnerConfigOptions; this.dataflow = dataflow; this.metrics = metricsProperties; - this.projectId = defaultOptions.get("project"); - this.location = defaultOptions.get("region"); + this.projectId = config.getProject(); + this.location = config.getRegion(); + } + + private static Credential getGoogleCredential() { + GoogleCredential credential = null; + try { + credential = GoogleCredential.getApplicationDefault().createScoped(DataflowScopes.all()); + } catch (IOException e) { + throw new IllegalStateException( + "Unable to find credential required for Dataflow monitoring API", e); + } + return credential; } @Override diff --git a/core/src/main/java/feast/core/job/dataflow/DataflowRunnerConfig.java b/core/src/main/java/feast/core/job/dataflow/DataflowRunnerConfig.java new file mode 100644 index 00000000000..6fe93ca80cd --- /dev/null +++ b/core/src/main/java/feast/core/job/dataflow/DataflowRunnerConfig.java @@ -0,0 +1,113 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2019 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.core.job.dataflow; + +import java.lang.reflect.Field; +import java.util.Map; +import java.util.Set; +import javax.validation.*; +import javax.validation.constraints.NotBlank; +import lombok.Getter; +import lombok.Setter; + +/** DataflowRunnerConfig contains configuration fields for the Dataflow job runner. */ +@Getter +@Setter +public class DataflowRunnerConfig { + + public DataflowRunnerConfig(Map runnerConfigOptions) { + + // Try to find all fields in DataflowRunnerConfig inside the runnerConfigOptions and map it into + // this object + for (Field field : DataflowRunnerConfig.class.getFields()) { + String fieldName = field.getName(); + try { + if (!runnerConfigOptions.containsKey(fieldName)) { + continue; + } + String value = runnerConfigOptions.get(fieldName); + + if (Boolean.class.equals(field.getType())) { + field.set(this, Boolean.valueOf(value)); + continue; + } + if (field.getType() == Integer.class) { + field.set(this, Integer.valueOf(value)); + continue; + } + field.set(this, value); + } catch (IllegalAccessException e) { + throw new RuntimeException( + String.format( + "Could not successfully convert DataflowRunnerConfig for key: %s", fieldName), + e); + } + } + validate(); + } + + /* Project id to use when launching jobs. */ + @NotBlank public String project; + + /* The Google Compute Engine region for creating Dataflow jobs. */ + @NotBlank public String region; + + /* GCP availability zone for operations. */ + @NotBlank public String zone; + + /* Run the job as a specific service account, instead of the default GCE robot. */ + public String serviceAccount; + + /* GCE network for launching workers. */ + @NotBlank public String network; + + /* GCE subnetwork for launching workers. */ + @NotBlank public String subnetwork; + + /* Machine type to create Dataflow worker VMs as. */ + public String workerMachineType; + + /* The autoscaling algorithm to use for the workerpool. */ + public String autoscalingAlgorithm; + + /* Specifies whether worker pools should be started with public IP addresses. */ + public Boolean usePublicIps; + + /** + * A pipeline level default location for storing temporary files. Support Google Cloud Storage + * locations, e.g. gs://bucket/object + */ + @NotBlank public String tempLocation; + + /* The maximum number of workers to use for the workerpool. */ + public Integer maxNumWorkers; + + /* BigQuery table specification, e.g. PROJECT_ID:DATASET_ID.PROJECT_ID */ + public String deadLetterTableSpec; + + /** Validates Dataflow runner configuration options */ + public void validate() { + ValidatorFactory factory = Validation.buildDefaultValidatorFactory(); + Validator validator = factory.getValidator(); + + Set> dataflowRunnerConfigViolation = + validator.validate(this); + if (!dataflowRunnerConfigViolation.isEmpty()) { + throw new ConstraintViolationException(dataflowRunnerConfigViolation); + } + } +} diff --git a/core/src/main/java/feast/core/service/JobCoordinatorService.java b/core/src/main/java/feast/core/service/JobCoordinatorService.java index b66d181022e..b4ed341edc6 100644 --- a/core/src/main/java/feast/core/service/JobCoordinatorService.java +++ b/core/src/main/java/feast/core/service/JobCoordinatorService.java @@ -24,7 +24,8 @@ import feast.core.FeatureSetProto.FeatureSetStatus; import feast.core.StoreProto; import feast.core.StoreProto.Store.Subscription; -import feast.core.config.FeastProperties.JobUpdatesProperties; +import feast.core.config.FeastProperties; +import feast.core.config.FeastProperties.JobProperties; import feast.core.dao.FeatureSetRepository; import feast.core.dao.JobRepository; import feast.core.job.JobManager; @@ -58,7 +59,7 @@ public class JobCoordinatorService { private FeatureSetRepository featureSetRepository; private SpecService specService; private JobManager jobManager; - private JobUpdatesProperties jobUpdatesProperties; + private JobProperties jobProperties; @Autowired public JobCoordinatorService( @@ -66,12 +67,12 @@ public JobCoordinatorService( FeatureSetRepository featureSetRepository, SpecService specService, JobManager jobManager, - JobUpdatesProperties jobUpdatesProperties) { + FeastProperties feastProperties) { this.jobRepository = jobRepository; this.featureSetRepository = featureSetRepository; this.specService = specService; this.jobManager = jobManager; - this.jobUpdatesProperties = jobUpdatesProperties; + this.jobProperties = feastProperties.getJobs(); } /** @@ -86,7 +87,7 @@ public JobCoordinatorService( *

    4) Updates Feature set statuses */ @Transactional - @Scheduled(fixedDelayString = "${feast.jobs.updates.pollingIntervalMillis}") + @Scheduled(fixedDelayString = "${feast.jobs.polling_interval_milliseconds}") public void Poll() throws InvalidProtocolBufferException { log.info("Polling for new jobs..."); List jobUpdateTasks = new ArrayList<>(); @@ -121,7 +122,7 @@ public void Poll() throws InvalidProtocolBufferException { store, originalJob, jobManager, - jobUpdatesProperties.getTimeoutSeconds())); + jobProperties.getJobUpdateTimeoutSeconds())); }); } } diff --git a/core/src/main/java/feast/core/validators/OneOfStringValidator.java b/core/src/main/java/feast/core/validators/OneOfStringValidator.java new file mode 100644 index 00000000000..6b84e44b01c --- /dev/null +++ b/core/src/main/java/feast/core/validators/OneOfStringValidator.java @@ -0,0 +1,51 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.core.validators; + +import java.util.Arrays; +import javax.validation.ConstraintValidator; +import javax.validation.ConstraintValidatorContext; + +/** Validates whether a string value is found within a collection. */ +public class OneOfStringValidator implements ConstraintValidator { + + /** Values that are permitted for a specific instance of this validator */ + String[] allowedValues; + + /** + * Initialize the OneOfStringValidator with a collection of allowed String values. + * + * @param constraintAnnotation + */ + @Override + public void initialize(OneOfStrings constraintAnnotation) { + allowedValues = constraintAnnotation.value(); + } + + /** + * Validates whether a string value is found within the collection defined in the annotation. + * + * @param value String value that should be validated + * @param context Provides contextual data and operation when applying a given constraint + * validator + * @return Boolean value indicating whether the string is found within the allowed values. + */ + @Override + public boolean isValid(String value, ConstraintValidatorContext context) { + return Arrays.asList(allowedValues).contains(value); + } +} diff --git a/core/src/main/java/feast/core/validators/OneOfStrings.java b/core/src/main/java/feast/core/validators/OneOfStrings.java new file mode 100644 index 00000000000..dba290438c8 --- /dev/null +++ b/core/src/main/java/feast/core/validators/OneOfStrings.java @@ -0,0 +1,49 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.core.validators; + +import java.lang.annotation.*; +import javax.validation.Constraint; +import javax.validation.Payload; + +/** + * Annotation for String "one of" validation. Allows for the definition of a collection through an + * annotation. The collection is used to test values defined in the object. + */ +@Target({ + ElementType.METHOD, + ElementType.FIELD, + ElementType.ANNOTATION_TYPE, + ElementType.CONSTRUCTOR, + ElementType.PARAMETER +}) +@Retention(RetentionPolicy.RUNTIME) +@Documented +@Constraint(validatedBy = OneOfStringValidator.class) +public @interface OneOfStrings { + /** @return Default error message that is returned if the incorrect value is set */ + String message() default "Field value must be one of the following: {value}"; + + /** Allows for the specification of validation groups to which this constraint belongs. */ + Class[] groups() default {}; + + /** An attribute payload that can be used to assign custom payload objects to a constraint. */ + Class[] payload() default {}; + + /** @return Default value that is returned if no allowed values are configured */ + String[] value() default {}; +} diff --git a/core/src/main/resources/application.yml b/core/src/main/resources/application.yml index ee060fffc95..51395cf6449 100644 --- a/core/src/main/resources/application.yml +++ b/core/src/main/resources/application.yml @@ -23,18 +23,39 @@ grpc: enable-reflection: true feast: -# version: @project.version@ jobs: - # Runner type for feature population jobs. Currently supported runner types are - # DirectRunner and DataflowRunner. - runner: DirectRunner - # Key-value dict of job options to be passed to the population jobs. - options: {} - updates: - # Job update polling interval in milliseconds: how often Feast checks if new jobs should be sent to the runner. - pollingIntervalMillis: 60000 - # Timeout in seconds for each attempt to update or submit a new job to the runner. - timeoutSeconds: 240 + # Job update polling interval in milliseconds: how often Feast checks if new jobs should be sent to the runner. + polling_interval_milliseconds: 60000 + + # Timeout in seconds for each attempt to update or submit a new job to the runner. + job_update_timeout_seconds: 240 + + # Name of the active runner in "runners" that should be used. Only a single runner can be active at one time. + active_runner: direct + + # List of runner configurations. Please see protos/feast/core/Runner.proto for more details + # Alternatively see the following for options https://api.docs.feast.dev/grpc/feast.core.pb.html#Runner + runners: + - name: direct + type: DirectRunner + options: {} + + - name: dataflow + type: DataflowRunner + options: + project: my_gcp_project + region: asia-east1 + zone: asia-east1-a + tempLocation: gs://bucket/tempLocation + network: default + subnetwork: regions/asia-east1/subnetworks/mysubnetwork + maxNumWorkers: 1 + autoscalingAlgorithm: THROUGHPUT_BASED + usePublicIps: false + workerMachineType: n1-standard-1 + deadLetterTableSpec: project_id:dataset_id.table_id + + # Configuration options for metric collection for all ingestion jobs metrics: # Enable metrics pushing for all ingestion jobs. enabled: false @@ -49,9 +70,10 @@ feast: # Feature stream type. Only kafka is supported. type: kafka # Feature stream options. + # See the following for options https://api.docs.feast.dev/grpc/feast.core.pb.html#KafkaSourceConfig options: topic: feast-features - bootstrapServers: kafka:9092 + bootstrapServers: localhost:9092 replicationFactor: 1 partitions: 1 diff --git a/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java b/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java index 2d562d38df2..e610f393732 100644 --- a/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java +++ b/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java @@ -22,6 +22,8 @@ import static org.mockito.Mockito.*; import static org.mockito.MockitoAnnotations.initMocks; +import com.google.api.client.auth.oauth2.Credential; +import com.google.api.client.googleapis.testing.auth.oauth2.MockGoogleCredential; import com.google.api.services.dataflow.Dataflow; import com.google.common.collect.Lists; import com.google.protobuf.Duration; @@ -77,9 +79,20 @@ public void setUp() { defaults = new HashMap<>(); defaults.put("project", "project"); defaults.put("region", "region"); + defaults.put("zone", "zone"); + defaults.put("tempLocation", "tempLocation"); + defaults.put("network", "network"); + defaults.put("subnetwork", "subnetwork"); MetricsProperties metricsProperties = new MetricsProperties(); metricsProperties.setEnabled(false); - dfJobManager = new DataflowJobManager(dataflow, defaults, metricsProperties); + Credential credential = null; + try { + credential = MockGoogleCredential.getApplicationDefault(); + } catch (IOException e) { + e.printStackTrace(); + } + + dfJobManager = new DataflowJobManager(defaults, metricsProperties, credential); dfJobManager = spy(dfJobManager); } diff --git a/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java b/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java index aa71f201dde..52e838c3d9d 100644 --- a/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java +++ b/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java @@ -39,7 +39,8 @@ import feast.core.StoreProto.Store.RedisConfig; import feast.core.StoreProto.Store.StoreType; import feast.core.StoreProto.Store.Subscription; -import feast.core.config.FeastProperties.JobUpdatesProperties; +import feast.core.config.FeastProperties; +import feast.core.config.FeastProperties.JobProperties; import feast.core.dao.FeatureSetRepository; import feast.core.dao.JobRepository; import feast.core.job.JobManager; @@ -65,13 +66,15 @@ public class JobCoordinatorServiceTest { @Mock SpecService specService; @Mock FeatureSetRepository featureSetRepository; - private JobUpdatesProperties jobUpdatesProperties; + private FeastProperties feastProperties; @Before public void setUp() { initMocks(this); - jobUpdatesProperties = new JobUpdatesProperties(); - jobUpdatesProperties.setTimeoutSeconds(5); + feastProperties = new FeastProperties(); + JobProperties jobProperties = new JobProperties(); + jobProperties.setJobUpdateTimeoutSeconds(5); + feastProperties.setJobs(jobProperties); } @Test @@ -79,7 +82,7 @@ public void shouldDoNothingIfNoStoresFound() throws InvalidProtocolBufferExcepti when(specService.listStores(any())).thenReturn(ListStoresResponse.newBuilder().build()); JobCoordinatorService jcs = new JobCoordinatorService( - jobRepository, featureSetRepository, specService, jobManager, jobUpdatesProperties); + jobRepository, featureSetRepository, specService, jobManager, feastProperties); jcs.Poll(); verify(jobRepository, times(0)).saveAndFlush(any()); } @@ -105,7 +108,7 @@ public void shouldDoNothingIfNoMatchingFeatureSetsFound() throws InvalidProtocol .thenReturn(ListFeatureSetsResponse.newBuilder().build()); JobCoordinatorService jcs = new JobCoordinatorService( - jobRepository, featureSetRepository, specService, jobManager, jobUpdatesProperties); + jobRepository, featureSetRepository, specService, jobManager, feastProperties); jcs.Poll(); verify(jobRepository, times(0)).saveAndFlush(any()); } @@ -196,7 +199,7 @@ public void shouldGenerateAndSubmitJobsIfAny() throws InvalidProtocolBufferExcep JobCoordinatorService jcs = new JobCoordinatorService( - jobRepository, featureSetRepository, specService, jobManager, jobUpdatesProperties); + jobRepository, featureSetRepository, specService, jobManager, feastProperties); jcs.Poll(); verify(jobRepository, times(1)).saveAndFlush(jobArgCaptor.capture()); Job actual = jobArgCaptor.getValue(); @@ -318,7 +321,7 @@ public void shouldGroupJobsBySource() throws InvalidProtocolBufferException { JobCoordinatorService jcs = new JobCoordinatorService( - jobRepository, featureSetRepository, specService, jobManager, jobUpdatesProperties); + jobRepository, featureSetRepository, specService, jobManager, feastProperties); jcs.Poll(); verify(jobRepository, times(2)).saveAndFlush(jobArgCaptor.capture()); diff --git a/ingestion/src/main/java/feast/ingestion/utils/StoreUtil.java b/ingestion/src/main/java/feast/ingestion/utils/StoreUtil.java index 1b884333818..b62f83f0f30 100644 --- a/ingestion/src/main/java/feast/ingestion/utils/StoreUtil.java +++ b/ingestion/src/main/java/feast/ingestion/utils/StoreUtil.java @@ -83,12 +83,9 @@ public static FeatureSink getFeatureSink( StoreType storeType = store.getType(); switch (storeType) { case REDIS: - return RedisFeatureSink.builder() - .setRedisConfig(store.getRedisConfig()) - .setFeatureSetSpecs(featureSetSpecs) - .build(); + return RedisFeatureSink.fromConfig(store.getRedisConfig(), featureSetSpecs); case BIGQUERY: - return BigQueryFeatureSink.fromConfig(store.getBigqueryConfig()); + return BigQueryFeatureSink.fromConfig(store.getBigqueryConfig(), featureSetSpecs); default: throw new RuntimeException(String.format("Store type '{}' is unsupported", storeType)); } diff --git a/pom.xml b/pom.xml index 5a9ab5292ab..7b7cd1d0fed 100644 --- a/pom.xml +++ b/pom.xml @@ -486,6 +486,14 @@ true + + + build-info + + build-info + + + diff --git a/protos/feast/core/Runner.proto b/protos/feast/core/Runner.proto new file mode 100644 index 00000000000..779f4d44bea --- /dev/null +++ b/protos/feast/core/Runner.proto @@ -0,0 +1,73 @@ +// +// * Copyright 2020 The Feast Authors +// * +// * Licensed under the Apache License, Version 2.0 (the "License"); +// * you may not use this file except in compliance with the License. +// * You may obtain a copy of the License at +// * +// * https://www.apache.org/licenses/LICENSE-2.0 +// * +// * Unless required by applicable law or agreed to in writing, software +// * distributed under the License is distributed on an "AS IS" BASIS, +// * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// * See the License for the specific language governing permissions and +// * limitations under the License. +// + +syntax = "proto3"; +package feast.core; + +option java_package = "feast.core"; +option java_outer_classname = "RunnerProto"; +option go_package = "github.com/gojek/feast/sdk/go/protos/feast/core"; + +message DirectRunnerConfigOptions { + /** + * Controls the amount of target parallelism the DirectRunner will use. + * Defaults to the greater of the number of available processors and 3. Must be a value + * greater than zero. + */ + int32 targetParallelism = 1; + + /* BigQuery table specification, e.g. PROJECT_ID:DATASET_ID.PROJECT_ID */ + string deadLetterTableSpec = 2; +} + +message DataflowRunnerConfigOptions { + /* Project id to use when launching jobs. */ + string project = 1; + + /* The Google Compute Engine region for creating Dataflow jobs. */ + string region = 2; + + /* GCP availability zone for operations. */ + string zone = 3; + + /* Run the job as a specific service account, instead of the default GCE robot. */ + string serviceAccount = 4; + + /* GCE network for launching workers. */ + string network = 5; + + /* GCE subnetwork for launching workers. e.g. regions/asia-east1/subnetworks/mysubnetwork */ + string subnetwork = 6; + + /* Machine type to create Dataflow worker VMs as. */ + string workerMachineType = 7; + + /* The autoscaling algorithm to use for the workerpool. */ + string autoscalingAlgorithm = 8; + + /* Specifies whether worker pools should be started with public IP addresses. */ + bool usePublicIps = 9; + + // A pipeline level default location for storing temporary files. Support Google Cloud Storage locations, + // e.g. gs://bucket/object + string tempLocation = 10; + + /* The maximum number of workers to use for the workerpool. */ + int32 maxNumWorkers = 11; + + /* BigQuery table specification, e.g. PROJECT_ID:DATASET_ID.PROJECT_ID */ + string deadLetterTableSpec = 12; +} \ No newline at end of file diff --git a/protos/feast/core/Source.proto b/protos/feast/core/Source.proto index 8a6cbd415a2..b9e6227199b 100644 --- a/protos/feast/core/Source.proto +++ b/protos/feast/core/Source.proto @@ -39,9 +39,15 @@ enum SourceType { } message KafkaSourceConfig { - // - bootstrapServers: [comma delimited value of host[:port]] + // Comma separated list of Kafka bootstrap servers. Used for feature sets without a defined source host[:port]] string bootstrap_servers = 1; - // - topics: [Kafka topic name. This value is provisioned by core and should not be set by the user.] + // Kafka topic to use for feature sets without user defined topics string topic = 2; + + // Number of Kafka partitions to to use for managed feature stream. + int32 partitions = 3; + + // Defines the number of copies of managed feature stream Kafka. + int32 replicationFactor = 4; } \ No newline at end of file diff --git a/protos/feast/core/Store.proto b/protos/feast/core/Store.proto index 931a9d46b69..de9af0a99fe 100644 --- a/protos/feast/core/Store.proto +++ b/protos/feast/core/Store.proto @@ -120,6 +120,9 @@ message Store { message BigQueryConfig { string project_id = 1; string dataset_id = 2; + string staging_location = 3; + int32 initial_retry_delay_seconds = 4; + int32 total_timeout_seconds = 5; } message CassandraConfig { diff --git a/serving/lombok.config b/serving/lombok.config deleted file mode 100644 index 8f7e8aa1ac9..00000000000 --- a/serving/lombok.config +++ /dev/null @@ -1 +0,0 @@ -lombok.addLombokGeneratedAnnotation = true \ No newline at end of file diff --git a/serving/pom.xml b/serving/pom.xml index 1390bfdc80c..bbb694011a3 100644 --- a/serving/pom.xml +++ b/serving/pom.xml @@ -34,7 +34,7 @@ spring-plugins Spring Plugins - http://repo.spring.io/plugins-release + https://repo.spring.io/plugins-release @@ -46,6 +46,14 @@ false + + + build-info + + build-info + + + org.apache.maven.plugins @@ -76,6 +84,7 @@ ${project.version} + dev.feast feast-storage-api diff --git a/serving/sample_redis_config.yml b/serving/sample_redis_config.yml deleted file mode 100644 index b3461649a1d..00000000000 --- a/serving/sample_redis_config.yml +++ /dev/null @@ -1,9 +0,0 @@ -name: serving -type: REDIS -redis_config: - host: localhost - port: 6379 -subscriptions: - - name: "*" - project: "*" - version: "*" diff --git a/serving/src/main/java/feast/serving/FeastProperties.java b/serving/src/main/java/feast/serving/FeastProperties.java deleted file mode 100644 index 505d7d03301..00000000000 --- a/serving/src/main/java/feast/serving/FeastProperties.java +++ /dev/null @@ -1,191 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.serving; - -// Feast configuration properties that maps Feast configuration from default application.yml file to -// a Java object. -// https://www.baeldung.com/configuration-properties-in-spring-boot -// https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html#boot-features-external-config-typesafe-configuration-properties - -import java.util.Map; -import org.springframework.boot.context.properties.ConfigurationProperties; - -@ConfigurationProperties(prefix = "feast") -public class FeastProperties { - private String version; - private String coreHost; - private int coreGrpcPort; - private StoreProperties store; - private JobProperties jobs; - private TracingProperties tracing; - - public String getVersion() { - return this.version; - } - - public String getCoreHost() { - return this.coreHost; - } - - public int getCoreGrpcPort() { - return this.coreGrpcPort; - } - - public StoreProperties getStore() { - return this.store; - } - - public JobProperties getJobs() { - return this.jobs; - } - - public TracingProperties getTracing() { - return this.tracing; - } - - public void setVersion(String version) { - this.version = version; - } - - public void setCoreHost(String coreHost) { - this.coreHost = coreHost; - } - - public void setCoreGrpcPort(int coreGrpcPort) { - this.coreGrpcPort = coreGrpcPort; - } - - public void setStore(StoreProperties store) { - this.store = store; - } - - public void setJobs(JobProperties jobs) { - this.jobs = jobs; - } - - public void setTracing(TracingProperties tracing) { - this.tracing = tracing; - } - - public static class StoreProperties { - private String configPath; - private int redisPoolMaxSize; - private int redisPoolMaxIdle; - - public String getConfigPath() { - return this.configPath; - } - - public int getRedisPoolMaxSize() { - return this.redisPoolMaxSize; - } - - public int getRedisPoolMaxIdle() { - return this.redisPoolMaxIdle; - } - - public void setConfigPath(String configPath) { - this.configPath = configPath; - } - - public void setRedisPoolMaxSize(int redisPoolMaxSize) { - this.redisPoolMaxSize = redisPoolMaxSize; - } - - public void setRedisPoolMaxIdle(int redisPoolMaxIdle) { - this.redisPoolMaxIdle = redisPoolMaxIdle; - } - } - - public static class JobProperties { - private String stagingLocation; - private int bigqueryInitialRetryDelaySecs; - private int bigqueryTotalTimeoutSecs; - private String storeType; - private Map storeOptions; - - public String getStagingLocation() { - return this.stagingLocation; - } - - public int getBigqueryInitialRetryDelaySecs() { - return bigqueryInitialRetryDelaySecs; - } - - public int getBigqueryTotalTimeoutSecs() { - return bigqueryTotalTimeoutSecs; - } - - public String getStoreType() { - return this.storeType; - } - - public Map getStoreOptions() { - return this.storeOptions; - } - - public void setStagingLocation(String stagingLocation) { - this.stagingLocation = stagingLocation; - } - - public void setBigqueryInitialRetryDelaySecs(int bigqueryInitialRetryDelaySecs) { - this.bigqueryInitialRetryDelaySecs = bigqueryInitialRetryDelaySecs; - } - - public void setBigqueryTotalTimeoutSecs(int bigqueryTotalTimeoutSecs) { - this.bigqueryTotalTimeoutSecs = bigqueryTotalTimeoutSecs; - } - - public void setStoreType(String storeType) { - this.storeType = storeType; - } - - public void setStoreOptions(Map storeOptions) { - this.storeOptions = storeOptions; - } - } - - public static class TracingProperties { - private boolean enabled; - private String tracerName; - private String serviceName; - - public boolean isEnabled() { - return this.enabled; - } - - public String getTracerName() { - return this.tracerName; - } - - public String getServiceName() { - return this.serviceName; - } - - public void setEnabled(boolean enabled) { - this.enabled = enabled; - } - - public void setTracerName(String tracerName) { - this.tracerName = tracerName; - } - - public void setServiceName(String serviceName) { - this.serviceName = serviceName; - } - } -} diff --git a/serving/src/main/java/feast/serving/ServingApplication.java b/serving/src/main/java/feast/serving/ServingApplication.java index ae9bb87a0b5..ab036d04d18 100644 --- a/serving/src/main/java/feast/serving/ServingApplication.java +++ b/serving/src/main/java/feast/serving/ServingApplication.java @@ -16,11 +16,20 @@ */ package feast.serving; +import feast.serving.config.FeastProperties; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; +import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration; +import org.springframework.boot.autoconfigure.jdbc.DataSourceTransactionManagerAutoConfiguration; +import org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaAutoConfiguration; import org.springframework.boot.context.properties.EnableConfigurationProperties; -@SpringBootApplication +@SpringBootApplication( + exclude = { + DataSourceAutoConfiguration.class, + DataSourceTransactionManagerAutoConfiguration.class, + HibernateJpaAutoConfiguration.class + }) @EnableConfigurationProperties(FeastProperties.class) public class ServingApplication { public static void main(String[] args) { diff --git a/serving/src/main/java/feast/serving/configuration/ContextClosedHandler.java b/serving/src/main/java/feast/serving/config/ContextClosedHandler.java similarity index 96% rename from serving/src/main/java/feast/serving/configuration/ContextClosedHandler.java rename to serving/src/main/java/feast/serving/config/ContextClosedHandler.java index a4f6d64d84f..2bc97439f38 100644 --- a/serving/src/main/java/feast/serving/configuration/ContextClosedHandler.java +++ b/serving/src/main/java/feast/serving/config/ContextClosedHandler.java @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.serving.configuration; +package feast.serving.config; import java.util.concurrent.ScheduledExecutorService; import org.springframework.beans.factory.annotation.Autowired; diff --git a/serving/src/main/java/feast/serving/config/FeastProperties.java b/serving/src/main/java/feast/serving/config/FeastProperties.java new file mode 100644 index 00000000000..bf3387728a7 --- /dev/null +++ b/serving/src/main/java/feast/serving/config/FeastProperties.java @@ -0,0 +1,541 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2019 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.serving.config; + +// Feast configuration properties that maps Feast configuration from default application.yml file to +// a Java object. +// https://www.baeldung.com/configuration-properties-in-spring-boot +// https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html#boot-features-external-config-typesafe-configuration-properties + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.ObjectMapper; +import com.google.protobuf.InvalidProtocolBufferException; +import com.google.protobuf.util.JsonFormat; +import feast.core.StoreProto; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; +import javax.validation.constraints.NotBlank; +import javax.validation.constraints.Positive; +import org.apache.logging.log4j.core.config.plugins.validation.constraints.ValidHost; +import org.springframework.beans.factory.annotation.Autowired; +import org.springframework.boot.context.properties.ConfigurationProperties; +import org.springframework.boot.info.BuildProperties; + +/** Feast Serving properties. */ +@ConfigurationProperties(prefix = "feast", ignoreInvalidFields = true) +public class FeastProperties { + + /** + * Instantiates a new Feast Serving properties. + * + * @param buildProperties the build properties + */ + @Autowired + public FeastProperties(BuildProperties buildProperties) { + setVersion(buildProperties.getVersion()); + } + + /** Instantiates a new Feast class. */ + public FeastProperties() {} + + /* Feast Serving build version */ + @NotBlank private String version = "unknown"; + + /* Feast Core host to connect to. */ + @ValidHost @NotBlank private String coreHost; + + /* Feast Core port to connect to. */ + @Positive private int coreGrpcPort; + + /** + * Finds and returns the active store + * + * @return Returns the {@link Store} model object + */ + public Store getActiveStore() { + for (Store store : getStores()) { + if (activeStore.equals(store.getName())) { + return store; + } + } + throw new RuntimeException( + String.format("Active store is misconfigured. Could not find store: %s.", activeStore)); + } + + /** + * Set the name of the active store found in the "stores" configuration list + * + * @param activeStore String name to active store + */ + public void setActiveStore(String activeStore) { + this.activeStore = activeStore; + } + + /** Name of the active store configuration (only one store can be active at a time). */ + @NotBlank private String activeStore; + + /** + * Collection of store configurations. The active store is selected by the "activeStore" field. + */ + private List stores = new ArrayList<>(); + + /* Job Store properties to retain state of async jobs. */ + private JobStoreProperties jobStore; + + /* Metric tracing properties. */ + private TracingProperties tracing; + + /** + * Gets Serving store configuration as a list of {@link Store}. + * + * @return List of stores objects + */ + public List getStores() { + return stores; + } + + /** + * Gets Feast Serving build version. + * + * @return the build version + */ + public String getVersion() { + return version; + } + + /** + * Sets build version + * + * @param version the build version + */ + public void setVersion(String version) { + this.version = version; + } + + /** + * Gets Feast Core host. + * + * @return Feast Core host + */ + public String getCoreHost() { + return coreHost; + } + + /** + * Sets Feast Core host to connect to. + * + * @param coreHost Feast Core host + */ + public void setCoreHost(String coreHost) { + this.coreHost = coreHost; + } + + /** + * Gets Feast Core gRPC port. + * + * @return Feast Core gRPC port + */ + public int getCoreGrpcPort() { + return coreGrpcPort; + } + + /** + * Sets Feast Core gRPC port. + * + * @param coreGrpcPort gRPC port of Feast Core + */ + public void setCoreGrpcPort(int coreGrpcPort) { + this.coreGrpcPort = coreGrpcPort; + } + + /** + * Sets the collection of configured stores. + * + * @param stores List of {@link Store} + */ + public void setStores(List stores) { + this.stores = stores; + } + + /** Store configuration class for database that this Feast Serving uses. */ + public static class Store { + + private String name; + + private String type; + + private Map config = new HashMap<>(); + + private List subscriptions = new ArrayList<>(); + + /** + * Gets name of this store. This is unique to this specific instance. + * + * @return the name of the store + */ + public String getName() { + return name; + } + + /** + * Sets the name of this store. + * + * @param name the name of the store + */ + public void setName(String name) { + this.name = name; + } + + /** + * Gets the store type. Example are REDIS or BIGQUERY + * + * @return the store type as a String. + */ + public String getType() { + return type; + } + + /** + * Sets the store type + * + * @param type the type + */ + public void setType(String type) { + this.type = type; + } + + /** + * Converts this {@link Store} to a {@StoreProto.Store} + * + * @return {@StoreProto.Store} with configuration set + * @throws InvalidProtocolBufferException the invalid protocol buffer exception + * @throws JsonProcessingException the json processing exception + */ + public StoreProto.Store toProto() + throws InvalidProtocolBufferException, JsonProcessingException { + List subscriptions = getSubscriptions(); + List subscriptionProtos = + subscriptions.stream().map(Subscription::toProto).collect(Collectors.toList()); + + StoreProto.Store.Builder storeProtoBuilder = + StoreProto.Store.newBuilder() + .setName(name) + .setType(StoreProto.Store.StoreType.valueOf(type)) + .addAllSubscriptions(subscriptionProtos); + + ObjectMapper jsonWriter = new ObjectMapper(); + + // TODO: All of this logic should be moved to the store layer. Only a Map + // should be sent to a store and it should do its own validation. + switch (StoreProto.Store.StoreType.valueOf(type)) { + case REDIS: + StoreProto.Store.RedisConfig.Builder redisConfig = + StoreProto.Store.RedisConfig.newBuilder(); + JsonFormat.parser().merge(jsonWriter.writeValueAsString(config), redisConfig); + return storeProtoBuilder.setRedisConfig(redisConfig.build()).build(); + case BIGQUERY: + StoreProto.Store.BigQueryConfig.Builder bqConfig = + StoreProto.Store.BigQueryConfig.newBuilder(); + JsonFormat.parser().merge(jsonWriter.writeValueAsString(config), bqConfig); + return storeProtoBuilder.setBigqueryConfig(bqConfig.build()).build(); + case CASSANDRA: + StoreProto.Store.CassandraConfig.Builder cassandraConfig = + StoreProto.Store.CassandraConfig.newBuilder(); + JsonFormat.parser().merge(jsonWriter.writeValueAsString(config), cassandraConfig); + return storeProtoBuilder.setCassandraConfig(cassandraConfig.build()).build(); + default: + throw new InvalidProtocolBufferException("Invalid store set"); + } + } + + /** + * Get the subscriptions to this specific store. The subscriptions indicate which feature sets a + * store subscribes to. + * + * @return List of subscriptions. + */ + public List getSubscriptions() { + return subscriptions; + } + + /** + * Sets the store specific configuration. See getSubscriptions() for more details. + * + * @param subscriptions the subscriptions list + */ + public void setSubscriptions(List subscriptions) { + this.subscriptions = subscriptions; + } + + /** + * Gets the configuration to this specific store. This is a map of strings. These options are + * unique to the store. Please see protos/feast/core/Store.proto for the store specific + * configuration options + * + * @return Returns the store specific configuration + */ + public Map getConfig() { + return config; + } + + /** + * Sets the store config. Please protos/feast/core/Store.proto for the specific options for each + * store. + * + * @param config the config map + */ + public void setConfig(Map config) { + this.config = config; + } + + /** + * The Subscription type. + * + *

    Note: Please see protos/feast/core/CoreService.proto for details on how to subscribe to + * feature sets. + */ + public class Subscription { + /** Feast project to subscribe to. */ + String project; + + /** Feature set to subscribe to. */ + String name; + + /** Feature set versions to subscribe to. */ + String version; + + /** + * Gets Feast project subscribed to. + * + * @return the project string + */ + public String getProject() { + return project; + } + + /** + * Sets Feast project to subscribe to for this store. + * + * @param project the project + */ + public void setProject(String project) { + this.project = project; + } + + /** + * Gets the feature set name to subscribe to. + * + * @return the name + */ + public String getName() { + return name; + } + + /** + * Sets the feature set name to subscribe to. + * + * @param name the name + */ + public void setName(String name) { + this.name = name; + } + + /** + * Gets the feature set version that is being subscribed to by this store. + * + * @return the version + */ + public String getVersion() { + return version; + } + + /** + * Sets the feature set version that is being subscribed to by this store. + * + * @param version the version + */ + public void setVersion(String version) { + this.version = version; + } + + /** + * Convert this {@link Subscription} to a {@link StoreProto.Store.Subscription}. + * + * @return the store proto . store . subscription + */ + public StoreProto.Store.Subscription toProto() { + return StoreProto.Store.Subscription.newBuilder() + .setName(getName()) + .setProject(getProject()) + .setVersion(getVersion()) + .build(); + } + } + } + + /** + * Gets job store properties + * + * @return the job store properties + */ + public JobStoreProperties getJobStore() { + return jobStore; + } + + /** + * Set job store properties + * + * @param jobStore Job store properties to set + */ + public void setJobStore(JobStoreProperties jobStore) { + this.jobStore = jobStore; + } + + /** + * Gets tracing properties + * + * @return tracing properties + */ + public TracingProperties getTracing() { + return tracing; + } + + /** + * Sets the tracing configuration. + * + * @param tracing the tracing + */ + public void setTracing(TracingProperties tracing) { + this.tracing = tracing; + } + + /** The type Job store properties. */ + public static class JobStoreProperties { + + /** Job Store Redis Host */ + private String redisHost; + + /** Job Store Redis Host */ + private int redisPort; + + /** + * Gets redis host. + * + * @return the redis host + */ + public String getRedisHost() { + return redisHost; + } + + /** + * Sets redis host. + * + * @param redisHost the redis host + */ + public void setRedisHost(String redisHost) { + this.redisHost = redisHost; + } + + /** + * Gets redis port. + * + * @return the redis port + */ + public int getRedisPort() { + return redisPort; + } + + /** + * Sets redis port. + * + * @param redisPort the redis port + */ + public void setRedisPort(int redisPort) { + this.redisPort = redisPort; + } + } + + /** Trace metric collection properties */ + public static class TracingProperties { + + /** Tracing enabled/disabled */ + private boolean enabled; + + /** Name of tracer to use (only "jaeger") */ + private String tracerName; + + /** Service name uniquely identifies this Feast Serving deployment */ + private String serviceName; + + /** + * Is tracing enabled + * + * @return boolean flag + */ + public boolean isEnabled() { + return enabled; + } + + /** + * Sets tracing enabled or disabled. + * + * @param enabled flag + */ + public void setEnabled(boolean enabled) { + this.enabled = enabled; + } + + /** + * Gets tracer name ('jaeger') + * + * @return the tracer name + */ + public String getTracerName() { + return tracerName; + } + + /** + * Sets tracer name. + * + * @param tracerName the tracer name + */ + public void setTracerName(String tracerName) { + this.tracerName = tracerName; + } + + /** + * Gets the service name. The service name uniquely identifies this Feast serving instance. + * + * @return the service name + */ + public String getServiceName() { + return serviceName; + } + + /** + * Sets service name. + * + * @param serviceName the service name + */ + public void setServiceName(String serviceName) { + this.serviceName = serviceName; + } + } +} diff --git a/serving/src/main/java/feast/serving/configuration/InstrumentationConfig.java b/serving/src/main/java/feast/serving/config/InstrumentationConfig.java similarity index 96% rename from serving/src/main/java/feast/serving/configuration/InstrumentationConfig.java rename to serving/src/main/java/feast/serving/config/InstrumentationConfig.java index 2cd284829c4..30269c5d0ec 100644 --- a/serving/src/main/java/feast/serving/configuration/InstrumentationConfig.java +++ b/serving/src/main/java/feast/serving/config/InstrumentationConfig.java @@ -14,9 +14,8 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.serving.configuration; +package feast.serving.config; -import feast.serving.FeastProperties; import io.opentracing.Tracer; import io.opentracing.noop.NoopTracerFactory; import io.prometheus.client.exporter.MetricsServlet; diff --git a/serving/src/main/java/feast/serving/configuration/JobServiceConfig.java b/serving/src/main/java/feast/serving/config/JobServiceConfig.java similarity index 56% rename from serving/src/main/java/feast/serving/configuration/JobServiceConfig.java rename to serving/src/main/java/feast/serving/config/JobServiceConfig.java index fa94dab8329..fa2272e5cd0 100644 --- a/serving/src/main/java/feast/serving/configuration/JobServiceConfig.java +++ b/serving/src/main/java/feast/serving/config/JobServiceConfig.java @@ -14,14 +14,14 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.serving.configuration; +package feast.serving.config; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.google.protobuf.InvalidProtocolBufferException; import feast.core.StoreProto.Store.StoreType; -import feast.serving.FeastProperties; import feast.serving.service.JobService; import feast.serving.service.NoopJobService; import feast.serving.service.RedisBackedJobService; -import feast.serving.specs.CachedSpecService; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; @@ -29,24 +29,11 @@ public class JobServiceConfig { @Bean - public JobService jobService( - FeastProperties feastProperties, - CachedSpecService specService, - StoreConfiguration storeConfiguration) { - if (!specService.getStore().getType().equals(StoreType.BIGQUERY)) { + public JobService jobService(FeastProperties feastProperties) + throws InvalidProtocolBufferException, JsonProcessingException { + if (!feastProperties.getActiveStore().toProto().getType().equals(StoreType.BIGQUERY)) { return new NoopJobService(); } - StoreType storeType = StoreType.valueOf(feastProperties.getJobs().getStoreType()); - switch (storeType) { - case REDIS: - return new RedisBackedJobService(storeConfiguration.getJobStoreRedisConnection()); - case INVALID: - case BIGQUERY: - case CASSANDRA: - case UNRECOGNIZED: - default: - throw new IllegalArgumentException( - String.format("Unsupported store type '%s' for job store", storeType)); - } + return new RedisBackedJobService(feastProperties.getJobStore()); } } diff --git a/serving/src/main/java/feast/serving/configuration/ServingApiConfiguration.java b/serving/src/main/java/feast/serving/config/ServingApiConfiguration.java similarity index 97% rename from serving/src/main/java/feast/serving/configuration/ServingApiConfiguration.java rename to serving/src/main/java/feast/serving/config/ServingApiConfiguration.java index 539b25a0fcd..ce4fe134373 100644 --- a/serving/src/main/java/feast/serving/configuration/ServingApiConfiguration.java +++ b/serving/src/main/java/feast/serving/config/ServingApiConfiguration.java @@ -14,7 +14,7 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.serving.configuration; +package feast.serving.config; import java.util.List; import org.springframework.beans.factory.annotation.Autowired; diff --git a/serving/src/main/java/feast/serving/config/ServingServiceConfig.java b/serving/src/main/java/feast/serving/config/ServingServiceConfig.java new file mode 100644 index 00000000000..ec84e6c4fef --- /dev/null +++ b/serving/src/main/java/feast/serving/config/ServingServiceConfig.java @@ -0,0 +1,79 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2019 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.serving.config; + +import com.fasterxml.jackson.core.JsonProcessingException; +import com.google.protobuf.InvalidProtocolBufferException; +import feast.core.StoreProto; +import feast.serving.service.HistoricalServingService; +import feast.serving.service.JobService; +import feast.serving.service.NoopJobService; +import feast.serving.service.OnlineServingService; +import feast.serving.service.ServingService; +import feast.serving.specs.CachedSpecService; +import feast.storage.api.retriever.HistoricalRetriever; +import feast.storage.api.retriever.OnlineRetriever; +import feast.storage.connectors.bigquery.retriever.BigQueryHistoricalRetriever; +import feast.storage.connectors.redis.retriever.RedisOnlineRetriever; +import io.opentracing.Tracer; +import java.util.Map; +import org.slf4j.Logger; +import org.springframework.context.annotation.Bean; +import org.springframework.context.annotation.Configuration; + +@Configuration +public class ServingServiceConfig { + + private static final Logger log = org.slf4j.LoggerFactory.getLogger(ServingServiceConfig.class); + + @Bean + public ServingService servingService( + FeastProperties feastProperties, + CachedSpecService specService, + JobService jobService, + Tracer tracer) + throws InvalidProtocolBufferException, JsonProcessingException { + ServingService servingService = null; + FeastProperties.Store store = feastProperties.getActiveStore(); + StoreProto.Store.StoreType storeType = store.toProto().getType(); + Map config = store.getConfig(); + + switch (storeType) { + case REDIS: + OnlineRetriever redisRetriever = RedisOnlineRetriever.create(config); + servingService = new OnlineServingService(redisRetriever, specService, tracer); + break; + case BIGQUERY: + if (jobService.getClass() == NoopJobService.class) { + throw new IllegalArgumentException( + "Unable to instantiate JobService which is required by BigQueryHistoricalRetriever."); + } + HistoricalRetriever bqRetriever = BigQueryHistoricalRetriever.create(config); + servingService = new HistoricalServingService(bqRetriever, specService, jobService); + break; + case CASSANDRA: + case UNRECOGNIZED: + case INVALID: + throw new IllegalArgumentException( + String.format( + "Unsupported store type '%s' for store name '%s'", + store.getType(), store.getName())); + } + + return servingService; + } +} diff --git a/serving/src/main/java/feast/serving/configuration/SpecServiceConfig.java b/serving/src/main/java/feast/serving/config/SpecServiceConfig.java similarity index 87% rename from serving/src/main/java/feast/serving/configuration/SpecServiceConfig.java rename to serving/src/main/java/feast/serving/config/SpecServiceConfig.java index 26ebfa956ca..dbe2de665ee 100644 --- a/serving/src/main/java/feast/serving/configuration/SpecServiceConfig.java +++ b/serving/src/main/java/feast/serving/config/SpecServiceConfig.java @@ -14,13 +14,13 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -package feast.serving.configuration; +package feast.serving.config; -import feast.serving.FeastProperties; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.google.protobuf.InvalidProtocolBufferException; +import feast.core.StoreProto; import feast.serving.specs.CachedSpecService; import feast.serving.specs.CoreSpecService; -import java.nio.file.Path; -import java.nio.file.Paths; import java.util.concurrent.Executors; import java.util.concurrent.ScheduledExecutorService; import java.util.concurrent.TimeUnit; @@ -58,10 +58,11 @@ public ScheduledExecutorService cachedSpecServiceScheduledExecutorService( } @Bean - public CachedSpecService specService(FeastProperties feastProperties) { + public CachedSpecService specService(FeastProperties feastProperties) + throws InvalidProtocolBufferException, JsonProcessingException { CoreSpecService coreService = new CoreSpecService(feastCoreHost, feastCorePort); - Path path = Paths.get(feastProperties.getStore().getConfigPath()); - CachedSpecService cachedSpecStorage = new CachedSpecService(coreService, path); + StoreProto.Store storeProto = feastProperties.getActiveStore().toProto(); + CachedSpecService cachedSpecStorage = new CachedSpecService(coreService, storeProto); try { cachedSpecStorage.populateCache(); } catch (Exception e) { diff --git a/serving/src/main/java/feast/serving/configuration/ServingServiceConfig.java b/serving/src/main/java/feast/serving/configuration/ServingServiceConfig.java deleted file mode 100644 index 28df853e224..00000000000 --- a/serving/src/main/java/feast/serving/configuration/ServingServiceConfig.java +++ /dev/null @@ -1,138 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.serving.configuration; - -import com.google.cloud.bigquery.BigQuery; -import com.google.cloud.bigquery.BigQueryOptions; -import com.google.cloud.storage.Storage; -import com.google.cloud.storage.StorageOptions; -import feast.core.StoreProto.Store; -import feast.core.StoreProto.Store.BigQueryConfig; -import feast.core.StoreProto.Store.RedisConfig; -import feast.core.StoreProto.Store.Subscription; -import feast.serving.FeastProperties; -import feast.serving.service.*; -import feast.serving.specs.CachedSpecService; -import feast.storage.api.retriever.HistoricalRetriever; -import feast.storage.api.retriever.OnlineRetriever; -import feast.storage.connectors.bigquery.retriever.BigQueryHistoricalRetriever; -import feast.storage.connectors.redis.retriever.RedisOnlineRetriever; -import io.opentracing.Tracer; -import java.util.Map; -import org.slf4j.Logger; -import org.springframework.context.annotation.Bean; -import org.springframework.context.annotation.Configuration; - -@Configuration -public class ServingServiceConfig { - - private static final Logger log = org.slf4j.LoggerFactory.getLogger(ServingServiceConfig.class); - - private Store setStoreConfig(Store.Builder builder, Map options) { - switch (builder.getType()) { - case REDIS: - RedisConfig redisConfig = - RedisConfig.newBuilder() - .setHost(options.get("host")) - .setPort(Integer.parseInt(options.get("port"))) - .build(); - return builder.setRedisConfig(redisConfig).build(); - case BIGQUERY: - BigQueryConfig bqConfig = - BigQueryConfig.newBuilder() - .setProjectId(options.get("projectId")) - .setDatasetId(options.get("datasetId")) - .build(); - return builder.setBigqueryConfig(bqConfig).build(); - case CASSANDRA: - default: - throw new IllegalArgumentException( - String.format( - "Unsupported store %s provided, only REDIS or BIGQUERY are currently supported.", - builder.getType())); - } - } - - @Bean - public ServingService servingService( - FeastProperties feastProperties, - CachedSpecService specService, - JobService jobService, - Tracer tracer, - StoreConfiguration storeConfiguration) { - ServingService servingService = null; - Store store = specService.getStore(); - - switch (store.getType()) { - case REDIS: - OnlineRetriever redisRetriever = - new RedisOnlineRetriever(storeConfiguration.getServingRedisConnection()); - servingService = new OnlineServingService(redisRetriever, specService, tracer); - break; - case BIGQUERY: - BigQueryConfig bqConfig = store.getBigqueryConfig(); - BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService(); - Storage storage = StorageOptions.getDefaultInstance().getService(); - String jobStagingLocation = feastProperties.getJobs().getStagingLocation(); - if (!jobStagingLocation.contains("://")) { - throw new IllegalArgumentException( - String.format("jobStagingLocation is not a valid URI: %s", jobStagingLocation)); - } - if (jobStagingLocation.endsWith("/")) { - jobStagingLocation = jobStagingLocation.substring(0, jobStagingLocation.length() - 1); - } - if (!jobStagingLocation.startsWith("gs://")) { - throw new IllegalArgumentException( - "Store type BIGQUERY requires job staging location to be a valid and existing Google Cloud Storage URI. Invalid staging location: " - + jobStagingLocation); - } - if (jobService.getClass() == NoopJobService.class) { - throw new IllegalArgumentException( - "Unable to instantiate jobService for BigQuery store."); - } - - HistoricalRetriever bqRetriever = - BigQueryHistoricalRetriever.builder() - .setBigquery(bigquery) - .setDatasetId(bqConfig.getDatasetId()) - .setProjectId(bqConfig.getProjectId()) - .setJobStagingLocation(jobStagingLocation) - .setInitialRetryDelaySecs( - feastProperties.getJobs().getBigqueryInitialRetryDelaySecs()) - .setTotalTimeoutSecs(feastProperties.getJobs().getBigqueryTotalTimeoutSecs()) - .setStorage(storage) - .build(); - - servingService = new HistoricalServingService(bqRetriever, specService, jobService); - break; - case CASSANDRA: - case UNRECOGNIZED: - case INVALID: - throw new IllegalArgumentException( - String.format( - "Unsupported store type '%s' for store name '%s'", - store.getType(), store.getName())); - } - - return servingService; - } - - private Subscription parseSubscription(String subscription) { - String[] split = subscription.split(":"); - return Subscription.newBuilder().setName(split[0]).setVersion(split[1]).build(); - } -} diff --git a/serving/src/main/java/feast/serving/configuration/StoreConfiguration.java b/serving/src/main/java/feast/serving/configuration/StoreConfiguration.java deleted file mode 100644 index 84dc7b7f8d4..00000000000 --- a/serving/src/main/java/feast/serving/configuration/StoreConfiguration.java +++ /dev/null @@ -1,47 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2020 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.serving.configuration; - -import io.lettuce.core.api.StatefulRedisConnection; -import org.springframework.beans.factory.ObjectProvider; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.context.annotation.Configuration; - -@Configuration -public class StoreConfiguration { - - // We can define other store specific beans here - // These beans can be autowired or can be created in this class. - private final StatefulRedisConnection servingRedisConnection; - private final StatefulRedisConnection jobStoreRedisConnection; - - @Autowired - public StoreConfiguration( - ObjectProvider> servingRedisConnection, - ObjectProvider> jobStoreRedisConnection) { - this.servingRedisConnection = servingRedisConnection.getIfAvailable(); - this.jobStoreRedisConnection = jobStoreRedisConnection.getIfAvailable(); - } - - public StatefulRedisConnection getServingRedisConnection() { - return servingRedisConnection; - } - - public StatefulRedisConnection getJobStoreRedisConnection() { - return jobStoreRedisConnection; - } -} diff --git a/serving/src/main/java/feast/serving/configuration/redis/JobStoreRedisConfig.java b/serving/src/main/java/feast/serving/configuration/redis/JobStoreRedisConfig.java deleted file mode 100644 index 77d9262bcb3..00000000000 --- a/serving/src/main/java/feast/serving/configuration/redis/JobStoreRedisConfig.java +++ /dev/null @@ -1,68 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2020 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.serving.configuration.redis; - -import com.google.common.base.Enums; -import feast.core.StoreProto; -import feast.serving.FeastProperties; -import io.lettuce.core.RedisClient; -import io.lettuce.core.RedisURI; -import io.lettuce.core.api.StatefulRedisConnection; -import io.lettuce.core.codec.ByteArrayCodec; -import io.lettuce.core.resource.ClientResources; -import io.lettuce.core.resource.DefaultClientResources; -import java.util.Map; -import org.springframework.beans.factory.ObjectProvider; -import org.springframework.context.annotation.Bean; -import org.springframework.context.annotation.Configuration; - -@Configuration -public class JobStoreRedisConfig { - - @Bean(destroyMethod = "shutdown") - ClientResources jobStoreClientResources() { - return DefaultClientResources.create(); - } - - @Bean(destroyMethod = "shutdown") - RedisClient jobStoreRedisClient( - ClientResources jobStoreClientResources, FeastProperties feastProperties) { - StoreProto.Store.StoreType storeType = - Enums.getIfPresent( - StoreProto.Store.StoreType.class, feastProperties.getJobs().getStoreType()) - .orNull(); - if (storeType != StoreProto.Store.StoreType.REDIS) return null; - Map jobStoreConf = feastProperties.getJobs().getStoreOptions(); - // If job conf is empty throw StoreException - if (jobStoreConf == null - || jobStoreConf.get("host") == null - || jobStoreConf.get("host").isEmpty() - || jobStoreConf.get("port") == null - || jobStoreConf.get("port").isEmpty()) - throw new IllegalArgumentException("Store Configuration is not set"); - RedisURI uri = - RedisURI.create(jobStoreConf.get("host"), Integer.parseInt(jobStoreConf.get("port"))); - return RedisClient.create(jobStoreClientResources, uri); - } - - @Bean(destroyMethod = "close") - StatefulRedisConnection jobStoreRedisConnection( - ObjectProvider jobStoreRedisClient) { - if (jobStoreRedisClient.getIfAvailable() == null) return null; - return jobStoreRedisClient.getIfAvailable().connect(new ByteArrayCodec()); - } -} diff --git a/serving/src/main/java/feast/serving/configuration/redis/ServingStoreRedisConfig.java b/serving/src/main/java/feast/serving/configuration/redis/ServingStoreRedisConfig.java deleted file mode 100644 index 17a50eef6d6..00000000000 --- a/serving/src/main/java/feast/serving/configuration/redis/ServingStoreRedisConfig.java +++ /dev/null @@ -1,62 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2020 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.serving.configuration.redis; - -import feast.core.StoreProto; -import feast.serving.specs.CachedSpecService; -import io.lettuce.core.RedisClient; -import io.lettuce.core.RedisURI; -import io.lettuce.core.api.StatefulRedisConnection; -import io.lettuce.core.codec.ByteArrayCodec; -import io.lettuce.core.resource.ClientResources; -import io.lettuce.core.resource.DefaultClientResources; -import org.springframework.beans.factory.ObjectProvider; -import org.springframework.context.annotation.*; - -@Configuration -public class ServingStoreRedisConfig { - - @Bean - StoreProto.Store.RedisConfig servingStoreRedisConf(CachedSpecService specService) { - if (specService.getStore().getType() != StoreProto.Store.StoreType.REDIS) return null; - return specService.getStore().getRedisConfig(); - } - - @Bean(destroyMethod = "shutdown") - ClientResources servingClientResources() { - return DefaultClientResources.create(); - } - - @Bean(destroyMethod = "shutdown") - RedisClient servingRedisClient( - ClientResources servingClientResources, - ObjectProvider servingStoreRedisConf) { - if (servingStoreRedisConf.getIfAvailable() == null) return null; - RedisURI redisURI = - RedisURI.create( - servingStoreRedisConf.getIfAvailable().getHost(), - servingStoreRedisConf.getIfAvailable().getPort()); - return RedisClient.create(servingClientResources, redisURI); - } - - @Bean(destroyMethod = "close") - StatefulRedisConnection servingRedisConnection( - ObjectProvider servingRedisClient) { - if (servingRedisClient.getIfAvailable() == null) return null; - return servingRedisClient.getIfAvailable().connect(new ByteArrayCodec()); - } -} diff --git a/serving/src/main/java/feast/serving/controller/ServingServiceGRpcController.java b/serving/src/main/java/feast/serving/controller/ServingServiceGRpcController.java index 0eba67d4b4e..91f38e2bd41 100644 --- a/serving/src/main/java/feast/serving/controller/ServingServiceGRpcController.java +++ b/serving/src/main/java/feast/serving/controller/ServingServiceGRpcController.java @@ -16,7 +16,6 @@ */ package feast.serving.controller; -import feast.serving.FeastProperties; import feast.serving.ServingAPIProto.GetBatchFeaturesRequest; import feast.serving.ServingAPIProto.GetBatchFeaturesResponse; import feast.serving.ServingAPIProto.GetFeastServingInfoRequest; @@ -26,6 +25,7 @@ import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest; import feast.serving.ServingAPIProto.GetOnlineFeaturesResponse; import feast.serving.ServingServiceGrpc.ServingServiceImplBase; +import feast.serving.config.FeastProperties; import feast.serving.exception.SpecRetrievalException; import feast.serving.interceptors.GrpcMonitoringInterceptor; import feast.serving.service.ServingService; diff --git a/serving/src/main/java/feast/serving/controller/ServingServiceRestController.java b/serving/src/main/java/feast/serving/controller/ServingServiceRestController.java index b0e349fd6b0..344ab7cf3ae 100644 --- a/serving/src/main/java/feast/serving/controller/ServingServiceRestController.java +++ b/serving/src/main/java/feast/serving/controller/ServingServiceRestController.java @@ -18,11 +18,11 @@ import static feast.serving.util.mappers.ResponseJSONMapper.mapGetOnlineFeaturesResponse; -import feast.serving.FeastProperties; import feast.serving.ServingAPIProto.GetFeastServingInfoRequest; import feast.serving.ServingAPIProto.GetFeastServingInfoResponse; import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest; import feast.serving.ServingAPIProto.GetOnlineFeaturesResponse; +import feast.serving.config.FeastProperties; import feast.serving.service.ServingService; import feast.serving.util.RequestHelper; import io.opentracing.Tracer; diff --git a/serving/src/main/java/feast/serving/service/RedisBackedJobService.java b/serving/src/main/java/feast/serving/service/RedisBackedJobService.java index 0bf53630379..dd010e58970 100644 --- a/serving/src/main/java/feast/serving/service/RedisBackedJobService.java +++ b/serving/src/main/java/feast/serving/service/RedisBackedJobService.java @@ -19,8 +19,13 @@ import com.google.protobuf.util.JsonFormat; import feast.serving.ServingAPIProto.Job; import feast.serving.ServingAPIProto.Job.Builder; +import feast.serving.config.FeastProperties; +import io.lettuce.core.RedisClient; +import io.lettuce.core.RedisURI; import io.lettuce.core.api.StatefulRedisConnection; import io.lettuce.core.api.sync.RedisCommands; +import io.lettuce.core.codec.ByteArrayCodec; +import io.lettuce.core.resource.DefaultClientResources; import java.util.Optional; import org.joda.time.Duration; import org.slf4j.Logger; @@ -37,6 +42,16 @@ public class RedisBackedJobService implements JobService { // and since users normally don't require info about relatively old jobs. private final int defaultExpirySeconds = (int) Duration.standardDays(1).getStandardSeconds(); + public RedisBackedJobService(FeastProperties.JobStoreProperties jobStoreProperties) { + RedisURI uri = + RedisURI.create(jobStoreProperties.getRedisHost(), jobStoreProperties.getRedisPort()); + + this.syncCommand = + RedisClient.create(DefaultClientResources.create(), uri) + .connect(new ByteArrayCodec()) + .sync(); + } + public RedisBackedJobService(StatefulRedisConnection connection) { this.syncCommand = connection.sync(); } diff --git a/serving/src/main/java/feast/serving/specs/CachedSpecService.java b/serving/src/main/java/feast/serving/specs/CachedSpecService.java index 246be8c5fdd..2f68711bf20 100644 --- a/serving/src/main/java/feast/serving/specs/CachedSpecService.java +++ b/serving/src/main/java/feast/serving/specs/CachedSpecService.java @@ -18,7 +18,6 @@ import static feast.serving.util.RefUtil.generateFeatureSetStringRef; import static feast.serving.util.RefUtil.generateFeatureStringRef; -import static feast.serving.util.mappers.YamlToProtoMapper.yamlToStoreProto; import static java.util.Comparator.comparingInt; import static java.util.stream.Collectors.groupingBy; @@ -27,11 +26,10 @@ import com.google.common.cache.LoadingCache; import feast.core.CoreServiceProto.ListFeatureSetsRequest; import feast.core.CoreServiceProto.ListFeatureSetsResponse; -import feast.core.CoreServiceProto.UpdateStoreRequest; -import feast.core.CoreServiceProto.UpdateStoreResponse; import feast.core.FeatureSetProto.FeatureSet; import feast.core.FeatureSetProto.FeatureSetSpec; import feast.core.FeatureSetProto.FeatureSpec; +import feast.core.StoreProto; import feast.core.StoreProto.Store; import feast.core.StoreProto.Store.Subscription; import feast.serving.ServingAPIProto.FeatureReference; @@ -39,9 +37,6 @@ import feast.storage.api.retriever.FeatureSetRequest; import io.grpc.StatusRuntimeException; import io.prometheus.client.Gauge; -import java.io.IOException; -import java.nio.file.Files; -import java.nio.file.Path; import java.util.ArrayList; import java.util.HashMap; import java.util.List; @@ -59,7 +54,6 @@ public class CachedSpecService { private static final Logger log = org.slf4j.LoggerFactory.getLogger(CachedSpecService.class); private final CoreSpecService coreService; - private final Path configPath; private final Map featureToFeatureSetMapping; @@ -80,10 +74,9 @@ public class CachedSpecService { .help("epoch time of the last time the cache was updated") .register(); - public CachedSpecService(CoreSpecService coreService, Path configPath) { - this.configPath = configPath; + public CachedSpecService(CoreSpecService coreService, StoreProto.Store store) { this.coreService = coreService; - this.store = updateStore(readConfig(configPath)); + this.store = store; Map featureSets = getFeatureSetMap(); featureToFeatureSetMapping = @@ -156,7 +149,6 @@ public List getFeatureSets(List featureRefe * from core to preload the cache. */ public void populateCache() { - this.store = updateStore(readConfig(configPath)); Map featureSetMap = getFeatureSetMap(); featureSetCache.putAll(featureSetMap); featureToFeatureSetMapping.putAll(getFeatureToFeatureSetMapping(featureSetMap)); @@ -239,29 +231,4 @@ private Map getFeatureToFeatureSetMapping( }); return mapping; } - - private Store readConfig(Path path) { - try { - List fileContents = Files.readAllLines(path); - String yaml = fileContents.stream().reduce("", (l1, l2) -> l1 + "\n" + l2); - log.info("loaded store config at {}: \n{}", path.toString(), yaml); - return yamlToStoreProto(yaml); - } catch (IOException e) { - throw new RuntimeException( - String.format("Unable to read store config at %s", path.toAbsolutePath()), e); - } - } - - private Store updateStore(Store store) { - UpdateStoreRequest request = UpdateStoreRequest.newBuilder().setStore(store).build(); - try { - UpdateStoreResponse updateStoreResponse = coreService.updateStore(request); - if (!updateStoreResponse.getStore().equals(store)) { - throw new RuntimeException("Core store config not matching current store config"); - } - return updateStoreResponse.getStore(); - } catch (Exception e) { - throw new RuntimeException("Unable to update store configuration", e); - } - } } diff --git a/serving/src/main/java/feast/serving/util/mappers/YamlToProtoMapper.java b/serving/src/main/java/feast/serving/util/mappers/YamlToProtoMapper.java deleted file mode 100644 index 00ad1fabb1c..00000000000 --- a/serving/src/main/java/feast/serving/util/mappers/YamlToProtoMapper.java +++ /dev/null @@ -1,37 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.serving.util.mappers; - -import com.fasterxml.jackson.databind.ObjectMapper; -import com.fasterxml.jackson.dataformat.yaml.YAMLFactory; -import com.google.protobuf.util.JsonFormat; -import feast.core.StoreProto.Store; -import feast.core.StoreProto.Store.Builder; -import java.io.IOException; - -public class YamlToProtoMapper { - private static final ObjectMapper yamlReader = new ObjectMapper(new YAMLFactory()); - private static final ObjectMapper jsonWriter = new ObjectMapper(); - - public static Store yamlToStoreProto(String yaml) throws IOException { - Object obj = yamlReader.readValue(yaml, Object.class); - String jsonString = jsonWriter.writeValueAsString(obj); - Builder builder = Store.newBuilder(); - JsonFormat.parser().merge(jsonString, builder); - return builder.build(); - } -} diff --git a/serving/src/main/resources/application.yml b/serving/src/main/resources/application.yml index 96713c80287..053fddfafff 100644 --- a/serving/src/main/resources/application.yml +++ b/serving/src/main/resources/application.yml @@ -1,12 +1,51 @@ feast: - # This value is retrieved from project.version properties in pom.xml - # https://docs.spring.io/spring-boot/docs/current/reference/html/ - version: @project.version@ # GRPC service address for Feast Core # Feast Serving requires connection to Feast Core to retrieve and reload Feast metadata (e.g. FeatureSpecs, Store information) core-host: ${FEAST_CORE_HOST:localhost} core-grpc-port: ${FEAST_CORE_GRPC_PORT:6565} + # Indicates the active store. Only a single store in the last can be active at one time. In the future this key + # will be deprecated in order to allow multiple stores to be served from a single serving instance + active_store: online + + # List of store configurations + stores: + # Below are two store configurations. One for Redis and one for BigQuery. + # Please see https://api.docs.feast.dev/grpc/feast.core.pb.html#Store for configuration options + - name: online # Name of the store (referenced by active_store) + type: REDIS # Type of the store. REDIS, BIGQUERY are available options + config: # Store specific configuration. See + host: localhost + port: 6379 + # Subscriptions indicate which feature sets needs to be retrieved and used to populate this store + subscriptions: + # Wildcards match all options. No filtering is done. + - name: "*" + project: "*" + version: "*" + + - name: historical + type: BIGQUERY + config: # Store specific configuration. + # GCP Project + project_id: my_project + # BigQuery Dataset Id + dataset_id: my_dataset + # staging-location specifies the URI to store intermediate files for batch serving. + # Feast Serving client is expected to have read access to this staging location + # to download the batch features. + # For example: gs://mybucket/myprefix + # Please omit the trailing slash in the URI. + staging-location: gs://mybucket/myprefix + # Retry options for BigQuery retrieval jobs + bigquery-initial-retry-delay-secs: 1 + # BigQuery timeout for retrieval jobs + bigquery-total-timeout-secs: 21600 + subscriptions: + - name: "*" + project: "*" + version: "*" + tracing: # If true, Feast will provide tracing data (using OpenTracing API) for various RPC method calls # which can be useful to debug performance issues and perform benchmarking @@ -17,41 +56,13 @@ feast: # The service name identifier for the tracing data service-name: feast_serving - store: - # Path containing the store configuration for this serving store. - config-path: ${FEAST_STORE_CONFIG_PATH:serving/sample_redis_config.yml} - # If serving redis, the redis pool max size - redis-pool-max-size: ${FEAST_REDIS_POOL_MAX_SIZE:128} - # If serving redis, the redis pool max idle conns - redis-pool-max-idle: ${FEAST_REDIS_POOL_MAX_IDLE:16} - - jobs: - # staging-location specifies the URI to store intermediate files for batch serving. - # Feast Serving client is expected to have read access to this staging location - # to download the batch features. - # - # For example: gs://mybucket/myprefix - # Please omit the trailing slash in the URI. - staging-location: ${FEAST_JOB_STAGING_LOCATION:} - # - # Retry options for BigQuery jobs: - bigquery-initial-retry-delay-secs: 1 - bigquery-total-timeout-secs: 21600 - # - # Type of store to store job metadata. This only needs to be set if the - # serving store type is Bigquery. - store-type: ${FEAST_JOB_STORE_TYPE:} - # - # Job store connection options. If the job store is redis, the following items are required: - # - # store-options: - # host: localhost - # port: 6379 - # Optionally, you can configure the connection pool with the following items: - # max-conn: 8 - # max-idle: 8 - # max-wait-millis: 50 - store-options: {} + # The job store is used to maintain job management state for Feast Serving. This is required when using certain + # historical stores like BigQuery. Only Redis is supported as a job store. + job_store: + # Redis host to connect to + redis_host: localhost + # Redis port to connect to + redis_port: 6379 grpc: # The port number Feast Serving GRPC service should listen on diff --git a/serving/src/test/java/feast/serving/controller/ServingServiceGRpcControllerTest.java b/serving/src/test/java/feast/serving/controller/ServingServiceGRpcControllerTest.java index f2c51bc7dde..d23f9da1d25 100644 --- a/serving/src/test/java/feast/serving/controller/ServingServiceGRpcControllerTest.java +++ b/serving/src/test/java/feast/serving/controller/ServingServiceGRpcControllerTest.java @@ -19,11 +19,11 @@ import static org.mockito.MockitoAnnotations.initMocks; import com.google.protobuf.Timestamp; -import feast.serving.FeastProperties; import feast.serving.ServingAPIProto.FeatureReference; import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest; import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest.EntityRow; import feast.serving.ServingAPIProto.GetOnlineFeaturesResponse; +import feast.serving.config.FeastProperties; import feast.serving.service.ServingService; import feast.types.ValueProto.Value; import io.grpc.StatusRuntimeException; diff --git a/serving/src/test/java/feast/serving/service/CachedSpecServiceTest.java b/serving/src/test/java/feast/serving/service/CachedSpecServiceTest.java index 01c9304bda0..f4f795ed32f 100644 --- a/serving/src/test/java/feast/serving/service/CachedSpecServiceTest.java +++ b/serving/src/test/java/feast/serving/service/CachedSpecServiceTest.java @@ -38,15 +38,10 @@ import feast.serving.specs.CachedSpecService; import feast.serving.specs.CoreSpecService; import feast.storage.api.retriever.FeatureSetRequest; -import java.io.BufferedWriter; -import java.io.File; -import java.io.FileWriter; -import java.io.IOException; import java.util.Collections; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; -import org.junit.After; import org.junit.Before; import org.junit.Rule; import org.junit.Test; @@ -55,7 +50,6 @@ public class CachedSpecServiceTest { - private File configFile; private Store store; @Rule public final ExpectedException expectedException = ExpectedException.none(); @@ -66,27 +60,9 @@ public class CachedSpecServiceTest { private CachedSpecService cachedSpecService; @Before - public void setUp() throws IOException { + public void setUp() { initMocks(this); - configFile = File.createTempFile("serving", ".yml"); - String yamlString = - "name: SERVING\n" - + "type: REDIS\n" - + "redis_config:\n" - + " host: localhost\n" - + " port: 6379\n" - + "subscriptions:\n" - + "- project: project\n" - + " name: fs1\n" - + " version: \"*\"\n" - + "- project: project\n" - + " name: fs2\n" - + " version: \"*\""; - BufferedWriter writer = new BufferedWriter(new FileWriter(configFile)); - writer.write(yamlString); - writer.close(); - store = Store.newBuilder() .setName("SERVING") @@ -164,12 +140,7 @@ public void setUp() throws IOException { .build())) .thenReturn(ListFeatureSetsResponse.newBuilder().addAllFeatureSets(fs2FeatureSets).build()); - cachedSpecService = new CachedSpecService(coreService, configFile.toPath()); - } - - @After - public void tearDown() { - configFile.delete(); + cachedSpecService = new CachedSpecService(coreService, store); } @Test diff --git a/serving/src/test/java/feast/serving/service/RedisBackedJobServiceTest.java b/serving/src/test/java/feast/serving/service/RedisBackedJobServiceTest.java index 34bc31d2c26..23626c2cb85 100644 --- a/serving/src/test/java/feast/serving/service/RedisBackedJobServiceTest.java +++ b/serving/src/test/java/feast/serving/service/RedisBackedJobServiceTest.java @@ -26,6 +26,7 @@ import redis.embedded.RedisServer; public class RedisBackedJobServiceTest { + private static Integer REDIS_PORT = 51235; private RedisServer redis; @@ -41,7 +42,7 @@ public void teardown() { } @Test - public void shouldRecoverIfRedisConnectionIsLost() throws IOException { + public void shouldRecoverIfRedisConnectionIsLost() { RedisClient client = RedisClient.create(RedisURI.create("localhost", REDIS_PORT)); RedisBackedJobService jobService = new RedisBackedJobService(client.connect(new ByteArrayCodec())); diff --git a/serving/src/test/java/feast/serving/util/mappers/YamlToProtoMapperTest.java b/serving/src/test/java/feast/serving/util/mappers/YamlToProtoMapperTest.java deleted file mode 100644 index 6f95f5307b2..00000000000 --- a/serving/src/test/java/feast/serving/util/mappers/YamlToProtoMapperTest.java +++ /dev/null @@ -1,54 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.serving.util.mappers; - -import static org.hamcrest.core.IsEqual.equalTo; -import static org.junit.Assert.*; - -import feast.core.StoreProto.Store; -import feast.core.StoreProto.Store.RedisConfig; -import feast.core.StoreProto.Store.StoreType; -import feast.core.StoreProto.Store.Subscription; -import java.io.IOException; -import org.junit.Test; - -public class YamlToProtoMapperTest { - - @Test - public void shouldConvertYamlToProto() throws IOException { - String yaml = - "name: test\n" - + "type: REDIS\n" - + "redis_config:\n" - + " host: localhost\n" - + " port: 6379\n" - + "subscriptions:\n" - + "- project: \"*\"\n" - + " name: \"*\"\n" - + " version: \"*\"\n"; - Store store = YamlToProtoMapper.yamlToStoreProto(yaml); - Store expected = - Store.newBuilder() - .setName("test") - .setType(StoreType.REDIS) - .setRedisConfig(RedisConfig.newBuilder().setHost("localhost").setPort(6379)) - .addSubscriptions( - Subscription.newBuilder().setProject("*").setName("*").setVersion("*")) - .build(); - assertThat(store, equalTo(expected)); - } -} diff --git a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/BigQueryHistoricalRetriever.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/BigQueryHistoricalRetriever.java index 27ba07e82ec..cd372511c0a 100644 --- a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/BigQueryHistoricalRetriever.java +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/BigQueryHistoricalRetriever.java @@ -24,6 +24,7 @@ import com.google.cloud.bigquery.*; import com.google.cloud.storage.Blob; import com.google.cloud.storage.Storage; +import com.google.cloud.storage.StorageOptions; import feast.serving.ServingAPIProto; import feast.serving.ServingAPIProto.DatasetSource; import feast.storage.api.retriever.FeatureSetRequest; @@ -33,6 +34,7 @@ import java.io.IOException; import java.util.ArrayList; import java.util.List; +import java.util.Map; import java.util.UUID; import java.util.concurrent.*; import java.util.stream.Collectors; @@ -48,6 +50,36 @@ public abstract class BigQueryHistoricalRetriever implements HistoricalRetriever public static final long TEMP_TABLE_EXPIRY_DURATION_MS = Duration.ofDays(1).toMillis(); private static final long SUBQUERY_TIMEOUT_SECS = 900; // 15 minutes + public static HistoricalRetriever create(Map config) { + + BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService(); + Storage storage = StorageOptions.getDefaultInstance().getService(); + + String jobStagingLocation = config.get("staging-location"); + if (!jobStagingLocation.contains("://")) { + throw new IllegalArgumentException( + String.format("jobStagingLocation is not a valid URI: %s", jobStagingLocation)); + } + if (jobStagingLocation.endsWith("/")) { + jobStagingLocation = jobStagingLocation.substring(0, jobStagingLocation.length() - 1); + } + if (!jobStagingLocation.startsWith("gs://")) { + throw new IllegalArgumentException( + "Store type BIGQUERY requires job staging location to be a valid and existing Google Cloud Storage URI. Invalid staging location: " + + jobStagingLocation); + } + + return builder() + .setBigquery(bigquery) + .setDatasetId(config.get("dataset_id")) + .setProjectId(config.get("project_id")) + .setJobStagingLocation(config.get("staging-location")) + .setInitialRetryDelaySecs(Integer.parseInt(config.get("bigquery-initial-retry-delay-secs"))) + .setTotalTimeoutSecs(Integer.parseInt(config.get("bigquery-total-timeout-secs"))) + .setStorage(storage) + .build(); + } + public abstract String projectId(); public abstract String datasetId(); diff --git a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryFeatureSink.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryFeatureSink.java index 8860db2622a..d155d3f1f50 100644 --- a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryFeatureSink.java +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryFeatureSink.java @@ -57,9 +57,11 @@ public abstract class BigQueryFeatureSink implements FeatureSink { * your own client. * * @param config {@link BigQueryConfig} + * @param featureSetSpecs * @return {@link BigQueryFeatureSink.Builder} */ - public static BigQueryFeatureSink fromConfig(BigQueryConfig config) { + public static FeatureSink fromConfig( + BigQueryConfig config, Map featureSetSpecs) { return builder() .setDatasetId(config.getDatasetId()) .setProjectId(config.getProjectId()) diff --git a/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever/RedisOnlineRetriever.java b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever/RedisOnlineRetriever.java index c8bb33de5fd..0963731988c 100644 --- a/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever/RedisOnlineRetriever.java +++ b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever/RedisOnlineRetriever.java @@ -29,8 +29,11 @@ import feast.types.FieldProto.Field; import feast.types.ValueProto.Value; import io.grpc.Status; +import io.lettuce.core.RedisClient; +import io.lettuce.core.RedisURI; import io.lettuce.core.api.StatefulRedisConnection; import io.lettuce.core.api.sync.RedisCommands; +import io.lettuce.core.codec.ByteArrayCodec; import java.util.ArrayList; import java.util.List; import java.util.Map; @@ -41,10 +44,24 @@ public class RedisOnlineRetriever implements OnlineRetriever { private final RedisCommands syncCommands; - public RedisOnlineRetriever(StatefulRedisConnection connection) { + private RedisOnlineRetriever(StatefulRedisConnection connection) { this.syncCommands = connection.sync(); } + public static OnlineRetriever create(Map config) { + + StatefulRedisConnection connection = + RedisClient.create( + RedisURI.create(config.get("host"), Integer.parseInt(config.get("port")))) + .connect(new ByteArrayCodec()); + + return new RedisOnlineRetriever(connection); + } + + public static OnlineRetriever create(StatefulRedisConnection connection) { + return new RedisOnlineRetriever(connection); + } + /** * Gets online features from redis. This method returns a list of {@link FeatureRow}s * corresponding to each feature set spec. Each feature row in the list then corresponds to an diff --git a/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisFeatureSink.java b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisFeatureSink.java index 63c8c68d9bb..8801460231e 100644 --- a/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisFeatureSink.java +++ b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/writer/RedisFeatureSink.java @@ -19,6 +19,7 @@ import com.google.auto.value.AutoValue; import feast.core.FeatureSetProto.FeatureSet; import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.core.StoreProto; import feast.core.StoreProto.Store.RedisConfig; import feast.storage.api.writer.FeatureSink; import feast.storage.api.writer.WriteResult; @@ -33,6 +34,18 @@ @AutoValue public abstract class RedisFeatureSink implements FeatureSink { + /** + * Initialize a {@link RedisFeatureSink.Builder} from a {@link StoreProto.Store.RedisConfig}. + * + * @param redisConfig {@link RedisConfig} + * @param featureSetSpecs + * @return {@link RedisFeatureSink.Builder} + */ + public static FeatureSink fromConfig( + RedisConfig redisConfig, Map featureSetSpecs) { + return builder().setFeatureSetSpecs(featureSetSpecs).setRedisConfig(redisConfig).build(); + } + public abstract RedisConfig getRedisConfig(); public abstract Map getFeatureSetSpecs(); @@ -54,6 +67,7 @@ public abstract static class Builder { @Override public void prepareWrite(FeatureSet featureSet) { + RedisClient redisClient = RedisClient.create(RedisURI.create(getRedisConfig().getHost(), getRedisConfig().getPort())); try { diff --git a/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/retriever/RedisOnlineRetrieverTest.java b/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/retriever/RedisOnlineRetrieverTest.java index 11c216c5a0c..41bbfaa74c4 100644 --- a/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/retriever/RedisOnlineRetrieverTest.java +++ b/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/retriever/RedisOnlineRetrieverTest.java @@ -33,6 +33,7 @@ import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest.EntityRow; import feast.storage.RedisProto.RedisKey; import feast.storage.api.retriever.FeatureSetRequest; +import feast.storage.api.retriever.OnlineRetriever; import feast.types.FeatureRowProto.FeatureRow; import feast.types.FieldProto.Field; import feast.types.ValueProto.Value; @@ -52,14 +53,14 @@ public class RedisOnlineRetrieverTest { @Mock RedisCommands syncCommands; - private RedisOnlineRetriever redisOnlineRetriever; + private OnlineRetriever redisOnlineRetriever; private byte[][] redisKeyList; @Before public void setUp() { initMocks(this); when(connection.sync()).thenReturn(syncCommands); - redisOnlineRetriever = new RedisOnlineRetriever(connection); + redisOnlineRetriever = RedisOnlineRetriever.create(connection); redisKeyList = Lists.newArrayList( RedisKey.newBuilder() @@ -135,7 +136,7 @@ public void shouldReturnResponseWithValuesIfKeysPresent() { .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) .collect(Collectors.toList()); - redisOnlineRetriever = new RedisOnlineRetriever(connection); + redisOnlineRetriever = RedisOnlineRetriever.create(connection); when(connection.sync()).thenReturn(syncCommands); when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); @@ -211,7 +212,7 @@ public void shouldReturnResponseWithUnsetValuesIfKeysNotPresent() { .collect(Collectors.toList()); featureRowBytes.add(null); - redisOnlineRetriever = new RedisOnlineRetriever(connection); + redisOnlineRetriever = RedisOnlineRetriever.create(connection); when(connection.sync()).thenReturn(syncCommands); when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); From 0816cd4918bd679e49678fbf041d402715fff511 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Mon, 20 Apr 2020 15:36:53 +0800 Subject: [PATCH 119/176] Fix subscription config and doctests (#634) * Fix bug in configuration loading for subscriptions * Fix javadoc generation for StoreProto --- .../src/main/java/feast/serving/config/FeastProperties.java | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/serving/src/main/java/feast/serving/config/FeastProperties.java b/serving/src/main/java/feast/serving/config/FeastProperties.java index bf3387728a7..e884c0c0c92 100644 --- a/serving/src/main/java/feast/serving/config/FeastProperties.java +++ b/serving/src/main/java/feast/serving/config/FeastProperties.java @@ -222,9 +222,9 @@ public void setType(String type) { } /** - * Converts this {@link Store} to a {@StoreProto.Store} + * Converts this {@link Store} to a {@link StoreProto.Store} * - * @return {@StoreProto.Store} with configuration set + * @return {@link StoreProto.Store} with configuration set * @throws InvalidProtocolBufferException the invalid protocol buffer exception * @throws JsonProcessingException the json processing exception */ @@ -311,7 +311,7 @@ public void setConfig(Map config) { *

    Note: Please see protos/feast/core/CoreService.proto for details on how to subscribe to * feature sets. */ - public class Subscription { + public static class Subscription { /** Feast project to subscribe to. */ String project; From ffdd1f1a14da2ec7c685d4e17d53575d79d60455 Mon Sep 17 00:00:00 2001 From: Zhu Zhan Yan Date: Wed, 22 Apr 2020 10:01:51 +0800 Subject: [PATCH 120/176] Fix Feast Serving not registering its store in Feast Core (#641) * Fixed missing call in serving's CachedSpecService to register store in core. * Add unit test to check that cachedSpecService will register store with Core. Co-authored-by: Zhu Zhanyan --- .../serving/specs/CachedSpecService.java | 4 ++-- .../feast/serving/specs/CoreSpecService.java | 23 ++++++++++++++++++- .../service/CachedSpecServiceTest.java | 12 ++++++---- 3 files changed, 32 insertions(+), 7 deletions(-) diff --git a/serving/src/main/java/feast/serving/specs/CachedSpecService.java b/serving/src/main/java/feast/serving/specs/CachedSpecService.java index 2f68711bf20..a117754e844 100644 --- a/serving/src/main/java/feast/serving/specs/CachedSpecService.java +++ b/serving/src/main/java/feast/serving/specs/CachedSpecService.java @@ -47,7 +47,7 @@ import org.apache.commons.lang3.tuple.Pair; import org.slf4j.Logger; -/** In-memory cache of specs. */ +/** In-memory cache of specs hosted in Feast Core. */ public class CachedSpecService { private static final int MAX_SPEC_COUNT = 1000; @@ -76,7 +76,7 @@ public class CachedSpecService { public CachedSpecService(CoreSpecService coreService, StoreProto.Store store) { this.coreService = coreService; - this.store = store; + this.store = coreService.registerStore(store); Map featureSets = getFeatureSetMap(); featureToFeatureSetMapping = diff --git a/serving/src/main/java/feast/serving/specs/CoreSpecService.java b/serving/src/main/java/feast/serving/specs/CoreSpecService.java index 2f5cef342e0..259aa3f3f0c 100644 --- a/serving/src/main/java/feast/serving/specs/CoreSpecService.java +++ b/serving/src/main/java/feast/serving/specs/CoreSpecService.java @@ -23,11 +23,12 @@ import feast.core.CoreServiceProto.ListFeatureSetsResponse; import feast.core.CoreServiceProto.UpdateStoreRequest; import feast.core.CoreServiceProto.UpdateStoreResponse; +import feast.core.StoreProto.Store; import io.grpc.ManagedChannel; import io.grpc.ManagedChannelBuilder; import org.slf4j.Logger; -/** Client for spec retrieval from core. */ +/** Client for interfacing with specs in Feast Core. */ public class CoreSpecService { private static final Logger log = org.slf4j.LoggerFactory.getLogger(CoreSpecService.class); @@ -50,4 +51,24 @@ public ListFeatureSetsResponse listFeatureSets(ListFeatureSetsRequest ListFeatur public UpdateStoreResponse updateStore(UpdateStoreRequest updateStoreRequest) { return blockingStub.updateStore(updateStoreRequest); } + + /** + * Register the given store entry in Feast Core. If store already exists in Feast Core, updates + * the store entry in feast core. + * + * @param store entry to register/update in Feast Core. + * @return The register/updated store entry + */ + public Store registerStore(Store store) { + UpdateStoreRequest request = UpdateStoreRequest.newBuilder().setStore(store).build(); + try { + UpdateStoreResponse updateStoreResponse = this.updateStore(request); + if (!updateStoreResponse.getStore().equals(store)) { + throw new RuntimeException("Core store config not matching current store config"); + } + return updateStoreResponse.getStore(); + } catch (Exception e) { + throw new RuntimeException("Unable to update store configuration", e); + } + } } diff --git a/serving/src/test/java/feast/serving/service/CachedSpecServiceTest.java b/serving/src/test/java/feast/serving/service/CachedSpecServiceTest.java index f4f795ed32f..144b967c9f5 100644 --- a/serving/src/test/java/feast/serving/service/CachedSpecServiceTest.java +++ b/serving/src/test/java/feast/serving/service/CachedSpecServiceTest.java @@ -19,14 +19,14 @@ import static org.hamcrest.CoreMatchers.equalTo; import static org.hamcrest.Matchers.containsInAnyOrder; import static org.junit.Assert.assertThat; +import static org.mockito.Mockito.times; +import static org.mockito.Mockito.verify; import static org.mockito.Mockito.when; import static org.mockito.MockitoAnnotations.initMocks; import com.google.common.collect.Lists; import feast.core.CoreServiceProto.ListFeatureSetsRequest; import feast.core.CoreServiceProto.ListFeatureSetsResponse; -import feast.core.CoreServiceProto.UpdateStoreRequest; -import feast.core.CoreServiceProto.UpdateStoreResponse; import feast.core.FeatureSetProto; import feast.core.FeatureSetProto.FeatureSetSpec; import feast.core.FeatureSetProto.FeatureSpec; @@ -82,8 +82,7 @@ public void setUp() { .build()) .build(); - when(coreService.updateStore(UpdateStoreRequest.newBuilder().setStore(store).build())) - .thenReturn(UpdateStoreResponse.newBuilder().setStore(store).build()); + when(coreService.registerStore(store)).thenReturn(store); featureSetSpecs = new LinkedHashMap<>(); featureSetSpecs.put( @@ -143,6 +142,11 @@ public void setUp() { cachedSpecService = new CachedSpecService(coreService, store); } + @Test + public void shouldRegisterStoreWithCore() { + verify(coreService, times(1)).registerStore(cachedSpecService.getStore()); + } + @Test public void shouldPopulateAndReturnStore() { cachedSpecService.populateCache(); From 9a4b44c94d8219f282eb5e592eb7958e4895b252 Mon Sep 17 00:00:00 2001 From: zhilingc Date: Wed, 22 Apr 2020 15:32:32 +0800 Subject: [PATCH 121/176] Change hyphens to underscores to follow proto --- serving/src/main/resources/application.yml | 6 +++--- .../bigquery/retriever/BigQueryHistoricalRetriever.java | 6 +++--- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/serving/src/main/resources/application.yml b/serving/src/main/resources/application.yml index 053fddfafff..0e5ff3a405d 100644 --- a/serving/src/main/resources/application.yml +++ b/serving/src/main/resources/application.yml @@ -36,11 +36,11 @@ feast: # to download the batch features. # For example: gs://mybucket/myprefix # Please omit the trailing slash in the URI. - staging-location: gs://mybucket/myprefix + staging_location: gs://mybucket/myprefix # Retry options for BigQuery retrieval jobs - bigquery-initial-retry-delay-secs: 1 + bigquery_initial_retry_delay_secs: 1 # BigQuery timeout for retrieval jobs - bigquery-total-timeout-secs: 21600 + bigquery_total_timeout_secs: 21600 subscriptions: - name: "*" project: "*" diff --git a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/BigQueryHistoricalRetriever.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/BigQueryHistoricalRetriever.java index cd372511c0a..cc7a2f26959 100644 --- a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/BigQueryHistoricalRetriever.java +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/BigQueryHistoricalRetriever.java @@ -73,9 +73,9 @@ public static HistoricalRetriever create(Map config) { .setBigquery(bigquery) .setDatasetId(config.get("dataset_id")) .setProjectId(config.get("project_id")) - .setJobStagingLocation(config.get("staging-location")) - .setInitialRetryDelaySecs(Integer.parseInt(config.get("bigquery-initial-retry-delay-secs"))) - .setTotalTimeoutSecs(Integer.parseInt(config.get("bigquery-total-timeout-secs"))) + .setJobStagingLocation(config.get("staging_location")) + .setInitialRetryDelaySecs(Integer.parseInt(config.get("bigquery_initial_retry_delay_secs"))) + .setTotalTimeoutSecs(Integer.parseInt(config.get("bigquery_total_timeout_secs"))) .setStorage(storage) .build(); } From dad2cd1a100fac918b170c05a188865d6507036e Mon Sep 17 00:00:00 2001 From: zhilingc Date: Wed, 22 Apr 2020 16:53:44 +0800 Subject: [PATCH 122/176] Remove bigquery prefix --- serving/src/main/resources/application.yml | 4 ++-- .../bigquery/retriever/BigQueryHistoricalRetriever.java | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/serving/src/main/resources/application.yml b/serving/src/main/resources/application.yml index 0e5ff3a405d..f6eaccf3cd4 100644 --- a/serving/src/main/resources/application.yml +++ b/serving/src/main/resources/application.yml @@ -38,9 +38,9 @@ feast: # Please omit the trailing slash in the URI. staging_location: gs://mybucket/myprefix # Retry options for BigQuery retrieval jobs - bigquery_initial_retry_delay_secs: 1 + initial_retry_delay_seconds: 1 # BigQuery timeout for retrieval jobs - bigquery_total_timeout_secs: 21600 + total_timeout_seconds: 21600 subscriptions: - name: "*" project: "*" diff --git a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/BigQueryHistoricalRetriever.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/BigQueryHistoricalRetriever.java index cc7a2f26959..0edcf67806b 100644 --- a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/BigQueryHistoricalRetriever.java +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/BigQueryHistoricalRetriever.java @@ -55,7 +55,7 @@ public static HistoricalRetriever create(Map config) { BigQuery bigquery = BigQueryOptions.getDefaultInstance().getService(); Storage storage = StorageOptions.getDefaultInstance().getService(); - String jobStagingLocation = config.get("staging-location"); + String jobStagingLocation = config.get("staging_location"); if (!jobStagingLocation.contains("://")) { throw new IllegalArgumentException( String.format("jobStagingLocation is not a valid URI: %s", jobStagingLocation)); @@ -74,8 +74,8 @@ public static HistoricalRetriever create(Map config) { .setDatasetId(config.get("dataset_id")) .setProjectId(config.get("project_id")) .setJobStagingLocation(config.get("staging_location")) - .setInitialRetryDelaySecs(Integer.parseInt(config.get("bigquery_initial_retry_delay_secs"))) - .setTotalTimeoutSecs(Integer.parseInt(config.get("bigquery_total_timeout_secs"))) + .setInitialRetryDelaySecs(Integer.parseInt(config.get("initial_retry_delay_seconds"))) + .setTotalTimeoutSecs(Integer.parseInt(config.get("total_timeout_seconds"))) .setStorage(storage) .build(); } From 196f7112f0c4ba58f2417e0c9856dac07435e0be Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Thu, 23 Apr 2020 16:09:52 +0800 Subject: [PATCH 123/176] Update end-to-end test config (#645) * Update end-to-end test config * Change http port for batch serving * Change serving port to 8081 to avoid clashing with core * Decrease job polling interval * Set google cloud project in batch store definition --- infra/scripts/test-end-to-end-batch.sh | 83 +++++++++++++------------- infra/scripts/test-end-to-end.sh | 68 ++++++++++----------- 2 files changed, 72 insertions(+), 79 deletions(-) diff --git a/infra/scripts/test-end-to-end-batch.sh b/infra/scripts/test-end-to-end-batch.sh index 35553a92814..0e7bfe8bf8d 100755 --- a/infra/scripts/test-end-to-end-batch.sh +++ b/infra/scripts/test-end-to-end-batch.sh @@ -115,13 +115,17 @@ grpc: enable-reflection: true feast: - version: 0.3 jobs: - runner: DirectRunner - options: {} - updates: - pollingIntervalMillis: 30000 - timeoutSeconds: 240 + polling_interval_milliseconds: 10000 + job_update_timeout_seconds: 240 + + active_runner: direct + + runners: + - name: direct + type: DirectRunner + options: {} + metrics: enabled: false @@ -137,21 +141,15 @@ spring: jpa: properties.hibernate: format_sql: true - event.merge.entity_copy_observer: allow + event: + merge: + entity_copy_observer: allow hibernate.naming.physical-strategy=org.hibernate.boot.model.naming: PhysicalNamingStrategyStandardImpl hibernate.ddl-auto: update datasource: url: jdbc:postgresql://localhost:5432/postgres username: postgres password: password - -management: - metrics: - export: - simple: - enabled: false - statsd: - enabled: false EOF nohup java -jar core/target/feast-core-*${JAR_VERSION_SUFFIX}.jar \ @@ -175,42 +173,45 @@ bq --location=US --project_id=${GOOGLE_CLOUD_PROJECT} mk \ ${GOOGLE_CLOUD_PROJECT}:${DATASET_NAME} # Start Feast Online Serving in background -cat < /tmp/serving.store.bigquery.yml -name: warehouse -type: BIGQUERY -bigquery_config: - projectId: ${GOOGLE_CLOUD_PROJECT} - datasetId: ${DATASET_NAME} -subscriptions: - - name: "*" - version: "*" - project: "*" -EOF - cat < /tmp/serving.warehouse.application.yml feast: - version: 0.3 + # GRPC service address for Feast Core + # Feast Serving requires connection to Feast Core to retrieve and reload Feast metadata (e.g. FeatureSpecs, Store information) core-host: localhost core-grpc-port: 6565 + + # Indicates the active store. Only a single store in the last can be active at one time. In the future this key + # will be deprecated in order to allow multiple stores to be served from a single serving instance + active_store: historical + + # List of store configurations + stores: + - name: historical + type: BIGQUERY + config: + project_id: ${GOOGLE_CLOUD_PROJECT} + dataset_id: ${DATASET_NAME} + staging_location: ${JOBS_STAGING_LOCATION} + initial_retry_delay_seconds: 1 + total_timeout_seconds: 21600 + subscriptions: + - name: "*" + project: "*" + version: "*" + + job_store: + redis_host: localhost + redis_port: 6379 + tracing: enabled: false - store: - config-path: /tmp/serving.store.bigquery.yml - jobs: - staging-location: ${JOBS_STAGING_LOCATION} - store-type: REDIS - bigquery-initial-retry-delay-secs: 1 - bigquery-total-timeout-secs: 900 - store-options: - host: localhost - port: 6379 + grpc: port: 6566 enable-reflection: true -spring: - main: - web-environment: false +server: + port: 8081 EOF diff --git a/infra/scripts/test-end-to-end.sh b/infra/scripts/test-end-to-end.sh index ee7304f208d..c33dadc5413 100755 --- a/infra/scripts/test-end-to-end.sh +++ b/infra/scripts/test-end-to-end.sh @@ -98,13 +98,17 @@ grpc: enable-reflection: true feast: - version: 0.3 jobs: - runner: DirectRunner - options: {} - updates: - pollingIntervalMillis: 30000 - timeoutSeconds: 240 + polling_interval_milliseconds: 30000 + job_update_timeout_seconds: 240 + + active_runner: direct + + runners: + - name: direct + type: DirectRunner + options: {} + metrics: enabled: false @@ -120,7 +124,9 @@ spring: jpa: properties.hibernate: format_sql: true - event.merge.entity_copy_observer: allow + event: + merge: + entity_copy_observer: allow hibernate.naming.physical-strategy=org.hibernate.boot.model.naming: PhysicalNamingStrategyStandardImpl hibernate.ddl-auto: update datasource: @@ -128,13 +134,6 @@ spring: username: postgres password: password -management: - metrics: - export: - simple: - enabled: false - statsd: - enabled: false EOF nohup java -jar core/target/feast-core-*${JAR_VERSION_SUFFIX}.jar \ @@ -149,42 +148,35 @@ echo " Starting Feast Online Serving ============================================================ " -# Start Feast Online Serving in background -cat < /tmp/serving.store.redis.yml -name: serving -type: REDIS -redis_config: - host: localhost - port: 6379 -subscriptions: - - name: "*" - version: "*" - project: "*" -EOF cat < /tmp/serving.online.application.yml feast: - version: 0.3 core-host: localhost core-grpc-port: 6565 + + active_store: serving + + # List of store configurations + stores: + - name: serving + type: REDIS # Type of the store. REDIS, BIGQUERY are available options + config: + host: localhost + port: 6379 + subscriptions: + - name: "*" + project: "*" + version: "*" + tracing: enabled: false - store: - config-path: /tmp/serving.store.redis.yml - redis-pool-max-size: 128 - redis-pool-max-idle: 16 - jobs: - staging-location: ${JOBS_STAGING_LOCATION} - store-type: - store-options: {} grpc: port: 6566 enable-reflection: true -spring: - main: - web-environment: false +server: + port: 8081 EOF From 20ffa82670916121fdecf499bcd9e05d7415a049 Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Fri, 24 Apr 2020 14:31:12 +0800 Subject: [PATCH 124/176] Swap joincolumns (#647) --- core/src/main/java/feast/core/model/Job.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/core/src/main/java/feast/core/model/Job.java b/core/src/main/java/feast/core/model/Job.java index 738a16db2d1..060edefc3b9 100644 --- a/core/src/main/java/feast/core/model/Job.java +++ b/core/src/main/java/feast/core/model/Job.java @@ -73,8 +73,8 @@ public class Job extends AbstractTimestampEntity { @ManyToMany @JoinTable( name = "jobs_feature_sets", - joinColumns = @JoinColumn(name = "feature_sets_id"), - inverseJoinColumns = @JoinColumn(name = "job_id"), + joinColumns = @JoinColumn(name = "job_id"), + inverseJoinColumns = @JoinColumn(name = "feature_sets_id"), indexes = { @Index(name = "idx_jobs_feature_sets_job_id", columnList = "job_id"), @Index(name = "idx_jobs_feature_sets_feature_sets_id", columnList = "feature_sets_id") From e5bc18c6833f742a954c1325764721b96f31faa1 Mon Sep 17 00:00:00 2001 From: Khor Shu Heng <32997938+khorshuheng@users.noreply.github.com> Date: Fri, 24 Apr 2020 14:42:12 +0800 Subject: [PATCH 125/176] Update approvers list (#648) Co-authored-by: Khor Shu Heng --- OWNERS | 1 + 1 file changed, 1 insertion(+) diff --git a/OWNERS b/OWNERS index cfc3fe4ee89..408d4a0d51a 100644 --- a/OWNERS +++ b/OWNERS @@ -4,6 +4,7 @@ approvers: - woop - thirteen37 - davidheryanto + - khorshuheng reviewers: - zhilingc - woop From 78b60247d8cf9faad0f230580f8107754ad5f666 Mon Sep 17 00:00:00 2001 From: Ches Martin Date: Mon, 27 Apr 2020 12:55:13 +0700 Subject: [PATCH 126/176] core: Use Runner enum type instead of string for Job model (#651) #575 sought to clear up inconsistencies between uses of `Runner#name()` (the standard final method of `java.lang.Enum` that returns the value's enum constant name) and the riskily-named `Runner#getName()` defined in Feast for human-readable Beam Runner names. The latter is used as runner name users can set in config. The former is used for values of the runner column of the jobs table in SQL (as it should be). But it relied on careful coding to use the right one when constructing `Job` instances. This is error prone, as #578 demonstrates. There is a more robust way: use the enum instead of stringly-typed programming. It's one of the reasons we have enums :-) This also renames the internal identifier in the Runner definition to `humanName`, to distinguish it further from `Enum#name()`. --- .../java/feast/core/job/JobUpdateTask.java | 2 +- core/src/main/java/feast/core/job/Runner.java | 26 +++++++----- .../core/job/dataflow/DataflowJobManager.java | 4 +- core/src/main/java/feast/core/model/Job.java | 7 ++-- .../java/feast/core/service/JobService.java | 13 +++--- .../feast/core/job/JobUpdateTaskTest.java | 18 ++++---- .../test/java/feast/core/job/RunnerTest.java | 42 +++++++++++++++++++ .../job/dataflow/DataflowJobManagerTest.java | 4 +- .../direct/DirectRunnerJobManagerTest.java | 2 +- .../service/JobCoordinatorServiceTest.java | 12 +++--- .../feast/core/service/JobServiceTest.java | 2 +- 11 files changed, 90 insertions(+), 42 deletions(-) create mode 100644 core/src/test/java/feast/core/job/RunnerTest.java diff --git a/core/src/main/java/feast/core/job/JobUpdateTask.java b/core/src/main/java/feast/core/job/JobUpdateTask.java index f3afe84df77..04aab0cff68 100644 --- a/core/src/main/java/feast/core/job/JobUpdateTask.java +++ b/core/src/main/java/feast/core/job/JobUpdateTask.java @@ -144,7 +144,7 @@ private Job startJob( new Job( jobId, "", - jobManager.getRunnerType().name(), + jobManager.getRunnerType(), Source.fromProto(source), Store.fromProto(sinkSpec), featureSets, diff --git a/core/src/main/java/feast/core/job/Runner.java b/core/src/main/java/feast/core/job/Runner.java index 4e2033fed69..acccb70c8b2 100644 --- a/core/src/main/java/feast/core/job/Runner.java +++ b/core/src/main/java/feast/core/job/Runner.java @@ -16,33 +16,37 @@ */ package feast.core.job; +import java.util.NoSuchElementException; + +/** + * An Apache Beam Runner, for which Feast Core supports managing ingestion jobs. + * + * @see Beam Runners + */ public enum Runner { DATAFLOW("DataflowRunner"), FLINK("FlinkRunner"), DIRECT("DirectRunner"); - private final String name; + private final String humanName; - Runner(String name) { - this.name = name; + Runner(String humanName) { + this.humanName = humanName; } - /** - * Get the human readable name of this runner. Returns a human readable name of the runner that - * can be used for logging/config files/etc. - */ + /** Returns the human readable name of this runner, usable in logging, config files, etc. */ @Override public String toString() { - return name; + return humanName; } /** Parses a runner from its human readable name. */ - public static Runner fromString(String runner) { + public static Runner fromString(String humanName) { for (Runner r : Runner.values()) { - if (r.toString().equals(runner)) { + if (r.toString().equals(humanName)) { return r; } } - throw new IllegalArgumentException("Unknown value: " + runner); + throw new NoSuchElementException("Unknown Runner value: " + humanName); } } diff --git a/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java b/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java index 6002133e828..880dd6c146b 100644 --- a/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java +++ b/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java @@ -220,7 +220,7 @@ public Job restartJob(Job job) { */ @Override public JobStatus getJobStatus(Job job) { - if (!Runner.DATAFLOW.name().equals(job.getRunner())) { + if (job.getRunner() != RUNNER_TYPE) { return job.getStatus(); } @@ -252,7 +252,7 @@ private Job submitDataflowJob( return new Job( jobName, jobId, - getRunnerType().name(), + getRunnerType(), Source.fromProto(source), Store.fromProto(sink), featureSets, diff --git a/core/src/main/java/feast/core/model/Job.java b/core/src/main/java/feast/core/model/Job.java index 060edefc3b9..95bcd79e6c0 100644 --- a/core/src/main/java/feast/core/model/Job.java +++ b/core/src/main/java/feast/core/model/Job.java @@ -19,6 +19,7 @@ import com.google.protobuf.InvalidProtocolBufferException; import feast.core.FeatureSetProto; import feast.core.IngestionJobProto; +import feast.core.job.Runner; import java.util.ArrayList; import java.util.List; import javax.persistence.CascadeType; @@ -55,9 +56,9 @@ public class Job extends AbstractTimestampEntity { private String extId; // Runner type - // Use Runner.name() when converting a Runner to string to assign to this property. + @Enumerated(EnumType.STRING) @Column(name = "runner") - private String runner; + private Runner runner; // Source id @ManyToOne @@ -96,7 +97,7 @@ public Job() { public Job( String id, String extId, - String runner, + Runner runner, Source source, Store sink, List featureSets, diff --git a/core/src/main/java/feast/core/service/JobService.java b/core/src/main/java/feast/core/service/JobService.java index bf74b90e80c..c8fc5caf5e1 100644 --- a/core/src/main/java/feast/core/service/JobService.java +++ b/core/src/main/java/feast/core/service/JobService.java @@ -29,6 +29,7 @@ import feast.core.IngestionJobProto; import feast.core.dao.JobRepository; import feast.core.job.JobManager; +import feast.core.job.Runner; import feast.core.log.Action; import feast.core.log.AuditLogger; import feast.core.log.Resource; @@ -50,13 +51,13 @@ import org.springframework.stereotype.Service; import org.springframework.transaction.annotation.Transactional; -/** Defines a Job Managemenent Service that allows users to manage feast ingestion jobs. */ +/** A Job Management Service that allows users to manage Feast ingestion jobs. */ @Slf4j @Service public class JobService { - private JobRepository jobRepository; - private SpecService specService; - private Map jobManagers; + private final JobRepository jobRepository; + private final SpecService specService; + private final Map jobManagers; @Autowired public JobService( @@ -66,13 +67,13 @@ public JobService( this.jobManagers = new HashMap<>(); for (JobManager manager : jobManagerList) { - this.jobManagers.put(manager.getRunnerType().name(), manager); + this.jobManagers.put(manager.getRunnerType(), manager); } } /* Job Service API */ /** - * List Ingestion Jobs in feast matching the given request. See CoreService protobuf documentation + * List Ingestion Jobs in Feast matching the given request. See CoreService protobuf documentation * for more detailed documentation. * * @param request list ingestion jobs request specifying which jobs to include diff --git a/core/src/test/java/feast/core/job/JobUpdateTaskTest.java b/core/src/test/java/feast/core/job/JobUpdateTaskTest.java index 2a1e80994ae..5faf446a948 100644 --- a/core/src/test/java/feast/core/job/JobUpdateTaskTest.java +++ b/core/src/test/java/feast/core/job/JobUpdateTaskTest.java @@ -102,7 +102,7 @@ public void shouldUpdateJobIfPresent() { new Job( "job", "old_ext", - Runner.DATAFLOW.name(), + Runner.DATAFLOW, feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), @@ -119,7 +119,7 @@ public void shouldUpdateJobIfPresent() { new Job( "job", "old_ext", - Runner.DATAFLOW.name(), + Runner.DATAFLOW, feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1), FeatureSet.fromProto(featureSet2)), @@ -129,7 +129,7 @@ public void shouldUpdateJobIfPresent() { new Job( "job", "new_ext", - Runner.DATAFLOW.name(), + Runner.DATAFLOW, Source.fromProto(source), Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1), FeatureSet.fromProto(featureSet2)), @@ -163,7 +163,7 @@ public void shouldCreateJobIfNotPresent() { new Job( "job", "", - Runner.DATAFLOW.name(), + Runner.DATAFLOW, feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), @@ -173,7 +173,7 @@ public void shouldCreateJobIfNotPresent() { new Job( "job", "ext", - Runner.DATAFLOW.name(), + Runner.DATAFLOW, feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), @@ -202,7 +202,7 @@ public void shouldUpdateJobStatusIfNotCreateOrUpdate() { new Job( "job", "ext", - Runner.DATAFLOW.name(), + Runner.DATAFLOW, feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), @@ -216,7 +216,7 @@ public void shouldUpdateJobStatusIfNotCreateOrUpdate() { new Job( "job", "ext", - Runner.DATAFLOW.name(), + Runner.DATAFLOW, Source.fromProto(source), Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), @@ -248,7 +248,7 @@ public void shouldReturnJobWithErrorStatusIfFailedToSubmit() { new Job( "job", "", - Runner.DATAFLOW.name(), + Runner.DATAFLOW, feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), @@ -258,7 +258,7 @@ public void shouldReturnJobWithErrorStatusIfFailedToSubmit() { new Job( "job", "", - Runner.DATAFLOW.name(), + Runner.DATAFLOW, feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), diff --git a/core/src/test/java/feast/core/job/RunnerTest.java b/core/src/test/java/feast/core/job/RunnerTest.java new file mode 100644 index 00000000000..ce1700acbe9 --- /dev/null +++ b/core/src/test/java/feast/core/job/RunnerTest.java @@ -0,0 +1,42 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.core.job; + +import static org.hamcrest.Matchers.is; +import static org.junit.Assert.assertThat; + +import java.util.NoSuchElementException; +import org.junit.Test; + +public class RunnerTest { + + @Test + public void toStringReturnsHumanReadableName() { + assertThat(Runner.DATAFLOW.toString(), is("DataflowRunner")); + } + + @Test + public void fromStringLoadsValueFromHumanReadableName() { + var humanName = Runner.DATAFLOW.toString(); + assertThat(Runner.fromString(humanName), is(Runner.DATAFLOW)); + } + + @Test(expected = NoSuchElementException.class) + public void fromStringThrowsNoSuchElementExceptionForUnknownValue() { + Runner.fromString("this is not a valid Runner"); + } +} diff --git a/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java b/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java index e610f393732..72b921ef694 100644 --- a/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java +++ b/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java @@ -158,7 +158,7 @@ public void shouldStartJobWithCorrectPipelineOptions() throws IOException { new Job( jobName, "", - Runner.DATAFLOW.name(), + Runner.DATAFLOW, Source.fromProto(source), Store.fromProto(store), Lists.newArrayList(FeatureSet.fromProto(featureSet)), @@ -239,7 +239,7 @@ public void shouldThrowExceptionWhenJobStateTerminal() throws IOException { new Job( "job", "", - Runner.DATAFLOW.name(), + Runner.DATAFLOW, Source.fromProto(source), Store.fromProto(store), Lists.newArrayList(FeatureSet.fromProto(featureSet)), diff --git a/core/src/test/java/feast/core/job/direct/DirectRunnerJobManagerTest.java b/core/src/test/java/feast/core/job/direct/DirectRunnerJobManagerTest.java index 76530d9f404..6980450ca4d 100644 --- a/core/src/test/java/feast/core/job/direct/DirectRunnerJobManagerTest.java +++ b/core/src/test/java/feast/core/job/direct/DirectRunnerJobManagerTest.java @@ -144,7 +144,7 @@ public void shouldStartDirectJobAndRegisterPipelineResult() throws IOException { new Job( expectedJobId, "", - Runner.DIRECT.name(), + Runner.DIRECT, Source.fromProto(source), Store.fromProto(store), Lists.newArrayList(FeatureSet.fromProto(featureSet)), diff --git a/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java b/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java index 52e838c3d9d..59fdc32b20f 100644 --- a/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java +++ b/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java @@ -164,7 +164,7 @@ public void shouldGenerateAndSubmitJobsIfAny() throws InvalidProtocolBufferExcep new Job( "", "", - Runner.DATAFLOW.name(), + Runner.DATAFLOW, feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1), FeatureSet.fromProto(featureSet2)), @@ -174,7 +174,7 @@ public void shouldGenerateAndSubmitJobsIfAny() throws InvalidProtocolBufferExcep new Job( "some_id", extId, - Runner.DATAFLOW.name(), + Runner.DATAFLOW, feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1), FeatureSet.fromProto(featureSet2)), @@ -264,7 +264,7 @@ public void shouldGroupJobsBySource() throws InvalidProtocolBufferException { new Job( "name1", "", - Runner.DATAFLOW.name(), + Runner.DATAFLOW, feast.core.model.Source.fromProto(source1), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), @@ -274,7 +274,7 @@ public void shouldGroupJobsBySource() throws InvalidProtocolBufferException { new Job( "name1", "extId1", - Runner.DATAFLOW.name(), + Runner.DATAFLOW, feast.core.model.Source.fromProto(source1), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet1)), @@ -284,7 +284,7 @@ public void shouldGroupJobsBySource() throws InvalidProtocolBufferException { new Job( "", "extId2", - Runner.DATAFLOW.name(), + Runner.DATAFLOW, feast.core.model.Source.fromProto(source2), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet2)), @@ -294,7 +294,7 @@ public void shouldGroupJobsBySource() throws InvalidProtocolBufferException { new Job( "name2", "extId2", - Runner.DATAFLOW.name(), + Runner.DATAFLOW, feast.core.model.Source.fromProto(source2), feast.core.model.Store.fromProto(store), Arrays.asList(FeatureSet.fromProto(featureSet2)), diff --git a/core/src/test/java/feast/core/service/JobServiceTest.java b/core/src/test/java/feast/core/service/JobServiceTest.java index c0e90ca43f4..6f34205bbfd 100644 --- a/core/src/test/java/feast/core/service/JobServiceTest.java +++ b/core/src/test/java/feast/core/service/JobServiceTest.java @@ -179,7 +179,7 @@ private Job newDummyJob(String id, String extId, JobStatus status) { return new Job( id, extId, - Runner.DATAFLOW.name(), + Runner.DATAFLOW, this.dataSource, this.dataStore, Arrays.asList(this.featureSet), From 11151e86981dc27dd494b028e7a4b62b87e035e9 Mon Sep 17 00:00:00 2001 From: Lavkesh Lahngir Date: Mon, 27 Apr 2020 17:26:13 +0800 Subject: [PATCH 127/176] Add Redis Cluster Support, (#502) Co-authored-by: Khor Shu Heng Co-authored-by: Lavkesh Lahngir Co-authored-by: Khor Shu Heng --- .prow/config.yaml | 13 + .../src/main/java/feast/core/model/Store.java | 7 + infra/scripts/setup-redis-cluster.sh | 16 + .../scripts/test-end-to-end-redis-cluster.sh | 248 +++++++++ ingestion/pom.xml | 6 + .../java/feast/ingestion/utils/StoreUtil.java | 5 +- protos/feast/core/Store.proto | 10 + serving/pom.xml | 6 + .../feast/serving/config/FeastProperties.java | 10 +- .../serving/config/ServingServiceConfig.java | 5 + storage/connectors/pom.xml | 1 + storage/connectors/rediscluster/pom.xml | 81 +++ .../retriever/FeatureRowDecoder.java | 82 +++ .../RedisClusterOnlineRetriever.java | 226 ++++++++ .../writer/RedisClusterCustomIO.java | 294 ++++++++++ .../writer/RedisClusterFeatureSink.java | 75 +++ .../writer/RedisClusterIngestionClient.java | 132 +++++ .../writer/RedisIngestionClient.java | 49 ++ .../RedisClusterOnlineRetrieverTest.java | 263 +++++++++ .../writer/RedisClusterFeatureSinkTest.java | 506 ++++++++++++++++++ 20 files changed, 2030 insertions(+), 5 deletions(-) create mode 100755 infra/scripts/setup-redis-cluster.sh create mode 100755 infra/scripts/test-end-to-end-redis-cluster.sh create mode 100644 storage/connectors/rediscluster/pom.xml create mode 100644 storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/retriever/FeatureRowDecoder.java create mode 100644 storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetriever.java create mode 100644 storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisClusterCustomIO.java create mode 100644 storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisClusterFeatureSink.java create mode 100644 storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisClusterIngestionClient.java create mode 100644 storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisIngestionClient.java create mode 100644 storage/connectors/rediscluster/src/test/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetrieverTest.java create mode 100644 storage/connectors/rediscluster/src/test/java/feast/storage/connectors/rediscluster/writer/RedisClusterFeatureSinkTest.java diff --git a/.prow/config.yaml b/.prow/config.yaml index 085cfe85423..5b039ff6616 100644 --- a/.prow/config.yaml +++ b/.prow/config.yaml @@ -153,6 +153,19 @@ presubmits: skip_branches: - ^v0\.(3|4)-branch$ + - name: test-end-to-end-redis-cluster + decorate: true + spec: + containers: + - image: maven:3.6-jdk-11 + command: ["infra/scripts/test-end-to-end-redis-cluster.sh"] + resources: + requests: + cpu: "6" + memory: "6144Mi" + skip_branches: + - ^v0\.(3|4)-branch$ + - name: test-end-to-end-java-8 decorate: true always_run: true diff --git a/core/src/main/java/feast/core/model/Store.java b/core/src/main/java/feast/core/model/Store.java index 9dc44bdc73a..debf211ec8d 100644 --- a/core/src/main/java/feast/core/model/Store.java +++ b/core/src/main/java/feast/core/model/Store.java @@ -21,6 +21,7 @@ import feast.core.StoreProto.Store.BigQueryConfig; import feast.core.StoreProto.Store.Builder; import feast.core.StoreProto.Store.CassandraConfig; +import feast.core.StoreProto.Store.RedisClusterConfig; import feast.core.StoreProto.Store.RedisConfig; import feast.core.StoreProto.Store.StoreType; import feast.core.StoreProto.Store.Subscription; @@ -82,6 +83,9 @@ public static Store fromProto(StoreProto.Store storeProto) throws IllegalArgumen case CASSANDRA: config = storeProto.getCassandraConfig().toByteArray(); break; + case REDIS_CLUSTER: + config = storeProto.getRedisClusterConfig().toByteArray(); + break; default: throw new IllegalArgumentException("Invalid store provided"); } @@ -106,6 +110,9 @@ public StoreProto.Store toProto() throws InvalidProtocolBufferException { case CASSANDRA: CassandraConfig cassConfig = CassandraConfig.parseFrom(config); return storeProtoBuilder.setCassandraConfig(cassConfig).build(); + case REDIS_CLUSTER: + RedisClusterConfig redisClusterConfig = RedisClusterConfig.parseFrom(config); + return storeProtoBuilder.setRedisClusterConfig(redisClusterConfig).build(); default: throw new InvalidProtocolBufferException("Invalid store set"); } diff --git a/infra/scripts/setup-redis-cluster.sh b/infra/scripts/setup-redis-cluster.sh new file mode 100755 index 00000000000..a1939705318 --- /dev/null +++ b/infra/scripts/setup-redis-cluster.sh @@ -0,0 +1,16 @@ +#!/usr/bin/env bash + +apt-get -y install redis-server > /var/log/redis.install.log + +mkdir 7000 7001 7002 7003 7004 7005 +for i in {0..5} ; do +echo "port 700$i +cluster-enabled yes +cluster-config-file nodes-$i.conf +cluster-node-timeout 5000 +appendonly yes" > 700$i/redis.conf +redis-server 700$i/redis.conf --daemonize yes +done +echo yes | redis-cli --cluster create 127.0.0.1:7000 127.0.0.1:7001 \ +127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005 \ +--cluster-replicas 1 diff --git a/infra/scripts/test-end-to-end-redis-cluster.sh b/infra/scripts/test-end-to-end-redis-cluster.sh new file mode 100755 index 00000000000..7f0d47fc92b --- /dev/null +++ b/infra/scripts/test-end-to-end-redis-cluster.sh @@ -0,0 +1,248 @@ +#!/usr/bin/env bash + +set -e +set -o pipefail + +test -z ${GOOGLE_APPLICATION_CREDENTIALS} && GOOGLE_APPLICATION_CREDENTIALS="/etc/service-account/service-account.json" +test -z ${SKIP_BUILD_JARS} && SKIP_BUILD_JARS="false" +test -z ${GOOGLE_CLOUD_PROJECT} && GOOGLE_CLOUD_PROJECT="kf-feast" +test -z ${TEMP_BUCKET} && TEMP_BUCKET="feast-templocation-kf-feast" +test -z ${JOBS_STAGING_LOCATION} && JOBS_STAGING_LOCATION="gs://${TEMP_BUCKET}/staging-location" +test -z ${JAR_VERSION_SUFFIX} && JAR_VERSION_SUFFIX="-SNAPSHOT" + +echo " +This script will run end-to-end tests for Feast Core and Online Serving. + +1. Install Redis as the store for Feast Online Serving. +2. Install Postgres for persisting Feast metadata. +3. Install Kafka and Zookeeper as the Source in Feast. +4. Install Python 3.7.4, Feast Python SDK and run end-to-end tests from + tests/e2e via pytest. +" + +apt-get -qq update +apt-get -y install wget netcat kafkacat + +echo " +============================================================ +Installing Redis at localhost:6379 +============================================================ +" +# Allow starting serving in this Maven Docker image. Default set to not allowed. +echo "exit 0" > /usr/sbin/policy-rc.d +infra/scripts/setup-redis-cluster.sh +redis-cli -c -p 7000 ping + +echo " +============================================================ +Installing Postgres at localhost:5432 +============================================================ +" +apt-get -y install postgresql > /var/log/postgresql.install.log +service postgresql start +# Initialize with database: 'postgres', user: 'postgres', password: 'password' +cat < /tmp/update-postgres-role.sh +psql -c "ALTER USER postgres PASSWORD 'password';" +EOF +chmod +x /tmp/update-postgres-role.sh +su -s /bin/bash -c /tmp/update-postgres-role.sh postgres +export PGPASSWORD=password +pg_isready + +echo " +============================================================ +Installing Zookeeper at localhost:2181 +Installing Kafka at localhost:9092 +============================================================ +" +wget -qO- https://www-eu.apache.org/dist/kafka/2.3.0/kafka_2.12-2.3.0.tgz | tar xz +mv kafka_2.12-2.3.0/ /tmp/kafka +nohup /tmp/kafka/bin/zookeeper-server-start.sh /tmp/kafka/config/zookeeper.properties &> /var/log/zookeeper.log 2>&1 & +sleep 5 +tail -n10 /var/log/zookeeper.log +nohup /tmp/kafka/bin/kafka-server-start.sh /tmp/kafka/config/server.properties &> /var/log/kafka.log 2>&1 & +sleep 20 +tail -n10 /var/log/kafka.log +kafkacat -b localhost:9092 -L + +if [[ ${SKIP_BUILD_JARS} != "true" ]]; then +echo " +============================================================ +Building jars for Feast +============================================================ +" + +.prow/scripts/download-maven-cache.sh \ + --archive-uri gs://feast-templocation-kf-feast/.m2.2019-10-24.tar \ + --output-dir /root/ + +# Build jars for Feast +mvn --quiet --batch-mode --define skipTests=true clean package + +ls -lh core/target/*jar +ls -lh serving/target/*jar +else + echo "[DEBUG] Skipping building jars" +fi + +echo " +============================================================ +Starting Feast Core +============================================================ +" +# Start Feast Core in background +cat < /tmp/core.application.yml +grpc: + port: 6565 + enable-reflection: true + +feast: + version: 0.3 + jobs: + runner: DirectRunner + options: {} + updates: + timeoutSeconds: 240 + metrics: + enabled: false + + stream: + type: kafka + options: + topic: feast-features + bootstrapServers: localhost:9092 + replicationFactor: 1 + partitions: 1 + +spring: + jpa: + properties.hibernate: + format_sql: true + event.merge.entity_copy_observer: allow + hibernate.naming.physical-strategy=org.hibernate.boot.model.naming: PhysicalNamingStrategyStandardImpl + hibernate.ddl-auto: update + datasource: + url: jdbc:postgresql://localhost:5432/postgres + username: postgres + password: password + +management: + metrics: + export: + simple: + enabled: false + statsd: + enabled: false +EOF + +nohup java -jar core/target/feast-core-*${JAR_VERSION_SUFFIX}.jar \ + --spring.config.location=file:///tmp/core.application.yml \ + &> /var/log/feast-core.log & +sleep 35 +tail -n10 /var/log/feast-core.log +nc -w2 localhost 6565 < /dev/null + +echo " +============================================================ +Starting Feast Online Serving +============================================================ +" +# Start Feast Online Serving in background +cat < /tmp/serving.store.redis.cluster.yml +name: serving +type: REDIS_CLUSTER +redis_cluster_config: + nodes: + - host: localhost + port: 7000 + - host: localhost + port: 7001 + - host: localhost + port: 7002 + - host: localhost + port: 7003 + - host: localhost + port: 7004 + - host: localhost + port: 7005 +subscriptions: + - name: "*" + version: "*" + project: "*" +EOF + +cat < /tmp/serving.online.application.yml +feast: + version: 0.3 + core-host: localhost + core-grpc-port: 6565 + tracing: + enabled: false + store: + config-path: /tmp/serving.store.redis.cluster.yml + redis-pool-max-size: 128 + redis-pool-max-idle: 16 + jobs: + staging-location: ${JOBS_STAGING_LOCATION} + store-type: + store-options: {} + +grpc: + port: 6566 + enable-reflection: true + +spring: + main: + web-environment: false + +EOF + +nohup java -jar serving/target/feast-serving-*${JAR_VERSION_SUFFIX}.jar \ + --spring.config.location=file:///tmp/serving.online.application.yml \ + &> /var/log/feast-serving-online.log & +sleep 15 +tail -n100 /var/log/feast-serving-online.log +nc -w2 localhost 6566 < /dev/null + +echo " +============================================================ +Installing Python 3.7 with Miniconda and Feast SDK +============================================================ +" +# Install Python 3.7 with Miniconda +wget -q https://repo.continuum.io/miniconda/Miniconda3-4.7.12-Linux-x86_64.sh \ + -O /tmp/miniconda.sh +bash /tmp/miniconda.sh -b -p /root/miniconda -f +/root/miniconda/bin/conda init +source ~/.bashrc + +# Install Feast Python SDK and test requirements +pip install -qe sdk/python +pip install -qr tests/e2e/requirements.txt + +echo " +============================================================ +Running end-to-end tests with pytest at 'tests/e2e' +============================================================ +" +# Default artifact location setting in Prow jobs +LOGS_ARTIFACT_PATH=/logs/artifacts + +ORIGINAL_DIR=$(pwd) +cd tests/e2e + +set +e +pytest basic-ingest-redis-serving.py --junitxml=${LOGS_ARTIFACT_PATH}/python-sdk-test-report.xml +TEST_EXIT_CODE=$? + +if [[ ${TEST_EXIT_CODE} != 0 ]]; then + echo "[DEBUG] Printing logs" + ls -ltrh /var/log/feast* + cat /var/log/feast-serving-online.log /var/log/feast-core.log + + echo "[DEBUG] Printing Python packages list" + pip list +fi + +cd ${ORIGINAL_DIR} +exit ${TEST_EXIT_CODE} diff --git a/ingestion/pom.xml b/ingestion/pom.xml index 9386d066bfd..64d5a41f86f 100644 --- a/ingestion/pom.xml +++ b/ingestion/pom.xml @@ -113,6 +113,12 @@ ${project.version} + + dev.feast + feast-storage-connector-redis-cluster + ${project.version} + + dev.feast feast-storage-connector-bigquery diff --git a/ingestion/src/main/java/feast/ingestion/utils/StoreUtil.java b/ingestion/src/main/java/feast/ingestion/utils/StoreUtil.java index b62f83f0f30..566124b3075 100644 --- a/ingestion/src/main/java/feast/ingestion/utils/StoreUtil.java +++ b/ingestion/src/main/java/feast/ingestion/utils/StoreUtil.java @@ -25,6 +25,7 @@ import feast.storage.api.writer.FeatureSink; import feast.storage.connectors.bigquery.writer.BigQueryFeatureSink; import feast.storage.connectors.redis.writer.RedisFeatureSink; +import feast.storage.connectors.rediscluster.writer.RedisClusterFeatureSink; import feast.types.ValueProto.ValueType.Enum; import java.util.HashMap; import java.util.Map; @@ -82,12 +83,14 @@ public static FeatureSink getFeatureSink( Store store, Map featureSetSpecs) { StoreType storeType = store.getType(); switch (storeType) { + case REDIS_CLUSTER: + return RedisClusterFeatureSink.fromConfig(store.getRedisClusterConfig(), featureSetSpecs); case REDIS: return RedisFeatureSink.fromConfig(store.getRedisConfig(), featureSetSpecs); case BIGQUERY: return BigQueryFeatureSink.fromConfig(store.getBigqueryConfig(), featureSetSpecs); default: - throw new RuntimeException(String.format("Store type '{}' is unsupported", storeType)); + throw new RuntimeException(String.format("Store type '%s' is unsupported", storeType)); } } } diff --git a/protos/feast/core/Store.proto b/protos/feast/core/Store.proto index de9af0a99fe..0aa4c8cd420 100644 --- a/protos/feast/core/Store.proto +++ b/protos/feast/core/Store.proto @@ -105,6 +105,8 @@ message Store { // Unsupported in Feast 0.3 CASSANDRA = 3; + + REDIS_CLUSTER = 4; } message RedisConfig { @@ -130,6 +132,13 @@ message Store { int32 port = 2; } + message RedisClusterConfig { + // List of Redis Uri for all the nodes in Redis Cluster, comma separated. Eg. host1:6379, host2:6379 + string connection_string = 1; + int32 initial_backoff_ms = 2; + int32 max_retries = 3; + } + message Subscription { // Name of project that the feature sets belongs to. This can be one of // - [project_name] @@ -172,5 +181,6 @@ message Store { RedisConfig redis_config = 11; BigQueryConfig bigquery_config = 12; CassandraConfig cassandra_config = 13; + RedisClusterConfig redis_cluster_config = 14; } } diff --git a/serving/pom.xml b/serving/pom.xml index bbb694011a3..d3d7ae212fd 100644 --- a/serving/pom.xml +++ b/serving/pom.xml @@ -97,6 +97,12 @@ ${project.version} + + dev.feast + feast-storage-connector-redis-cluster + ${project.version} + + dev.feast feast-storage-connector-bigquery diff --git a/serving/src/main/java/feast/serving/config/FeastProperties.java b/serving/src/main/java/feast/serving/config/FeastProperties.java index e884c0c0c92..b7fd0a9fed7 100644 --- a/serving/src/main/java/feast/serving/config/FeastProperties.java +++ b/serving/src/main/java/feast/serving/config/FeastProperties.java @@ -26,10 +26,7 @@ import com.google.protobuf.InvalidProtocolBufferException; import com.google.protobuf.util.JsonFormat; import feast.core.StoreProto; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.List; -import java.util.Map; +import java.util.*; import java.util.stream.Collectors; import javax.validation.constraints.NotBlank; import javax.validation.constraints.Positive; @@ -245,6 +242,11 @@ public StoreProto.Store toProto() // TODO: All of this logic should be moved to the store layer. Only a Map // should be sent to a store and it should do its own validation. switch (StoreProto.Store.StoreType.valueOf(type)) { + case REDIS_CLUSTER: + StoreProto.Store.RedisClusterConfig.Builder redisClusterConfig = + StoreProto.Store.RedisClusterConfig.newBuilder(); + JsonFormat.parser().merge(jsonWriter.writeValueAsString(config), redisClusterConfig); + return storeProtoBuilder.setRedisClusterConfig(redisClusterConfig.build()).build(); case REDIS: StoreProto.Store.RedisConfig.Builder redisConfig = StoreProto.Store.RedisConfig.newBuilder(); diff --git a/serving/src/main/java/feast/serving/config/ServingServiceConfig.java b/serving/src/main/java/feast/serving/config/ServingServiceConfig.java index ec84e6c4fef..a1dbc1db604 100644 --- a/serving/src/main/java/feast/serving/config/ServingServiceConfig.java +++ b/serving/src/main/java/feast/serving/config/ServingServiceConfig.java @@ -29,6 +29,7 @@ import feast.storage.api.retriever.OnlineRetriever; import feast.storage.connectors.bigquery.retriever.BigQueryHistoricalRetriever; import feast.storage.connectors.redis.retriever.RedisOnlineRetriever; +import feast.storage.connectors.rediscluster.retriever.RedisClusterOnlineRetriever; import io.opentracing.Tracer; import java.util.Map; import org.slf4j.Logger; @@ -53,6 +54,10 @@ public ServingService servingService( Map config = store.getConfig(); switch (storeType) { + case REDIS_CLUSTER: + OnlineRetriever redisClusterRetriever = RedisClusterOnlineRetriever.create(config); + servingService = new OnlineServingService(redisClusterRetriever, specService, tracer); + break; case REDIS: OnlineRetriever redisRetriever = RedisOnlineRetriever.create(config); servingService = new OnlineServingService(redisRetriever, specService, tracer); diff --git a/storage/connectors/pom.xml b/storage/connectors/pom.xml index b52668a31a4..b57fe98cd25 100644 --- a/storage/connectors/pom.xml +++ b/storage/connectors/pom.xml @@ -16,6 +16,7 @@ redis + rediscluster bigquery diff --git a/storage/connectors/rediscluster/pom.xml b/storage/connectors/rediscluster/pom.xml new file mode 100644 index 00000000000..5c3cb6e42d3 --- /dev/null +++ b/storage/connectors/rediscluster/pom.xml @@ -0,0 +1,81 @@ + + + + dev.feast + feast-storage-connectors + ${revision} + + + 4.0.0 + feast-storage-connector-redis-cluster + + Feast Storage Connector for Redis Cluster + + + + io.lettuce + lettuce-core + + + + org.apache.commons + commons-lang3 + 3.9 + + + + com.google.auto.value + auto-value-annotations + 1.6.6 + + + + com.google.auto.value + auto-value + 1.6.6 + provided + + + + org.mockito + mockito-core + 2.23.0 + test + + + + org.apache.beam + beam-runners-direct-java + ${org.apache.beam.version} + test + + + + org.hamcrest + hamcrest-core + test + + + + org.hamcrest + hamcrest-library + test + + + + net.ishiis.redis + redis-unit + 1.0.3 + test + + + + junit + junit + 4.12 + test + + + + diff --git a/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/retriever/FeatureRowDecoder.java b/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/retriever/FeatureRowDecoder.java new file mode 100644 index 00000000000..d6312c6b6ab --- /dev/null +++ b/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/retriever/FeatureRowDecoder.java @@ -0,0 +1,82 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.rediscluster.retriever; + +import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.core.FeatureSetProto.FeatureSpec; +import feast.types.FeatureRowProto.FeatureRow; +import feast.types.FieldProto.Field; +import java.util.Comparator; +import java.util.List; +import java.util.stream.Collectors; +import java.util.stream.IntStream; + +public class FeatureRowDecoder { + + private final String featureSetRef; + private final FeatureSetSpec spec; + + public FeatureRowDecoder(String featureSetRef, FeatureSetSpec spec) { + this.featureSetRef = featureSetRef; + this.spec = spec; + } + + /** + * Validates if an encoded feature row can be decoded without exception. + * + * @param featureRow Feature row + * @return boolean + */ + public Boolean isEncodingValid(FeatureRow featureRow) { + return featureRow.getFieldsList().size() == spec.getFeaturesList().size(); + } + + /** + * Decoding feature row by repopulating the field names based on the corresponding feature set + * spec. + * + * @param encodedFeatureRow Feature row + * @return boolean + */ + public FeatureRow decode(FeatureRow encodedFeatureRow) { + final List fieldsWithoutName = encodedFeatureRow.getFieldsList(); + + List featureNames = + spec.getFeaturesList().stream() + .sorted(Comparator.comparing(FeatureSpec::getName)) + .map(FeatureSpec::getName) + .collect(Collectors.toList()); + List fields = + IntStream.range(0, featureNames.size()) + .mapToObj( + featureNameIndex -> { + String featureName = featureNames.get(featureNameIndex); + return fieldsWithoutName + .get(featureNameIndex) + .toBuilder() + .setName(featureName) + .build(); + }) + .collect(Collectors.toList()); + return encodedFeatureRow + .toBuilder() + .clearFields() + .setFeatureSet(featureSetRef) + .addAllFields(fields) + .build(); + } +} diff --git a/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetriever.java b/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetriever.java new file mode 100644 index 00000000000..713b6897b2d --- /dev/null +++ b/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetriever.java @@ -0,0 +1,226 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.rediscluster.retriever; + +import com.google.protobuf.AbstractMessageLite; +import com.google.protobuf.InvalidProtocolBufferException; +import feast.core.FeatureSetProto.EntitySpec; +import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.serving.ServingAPIProto.FeatureReference; +import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest.EntityRow; +import feast.storage.RedisProto.RedisKey; +import feast.storage.api.retriever.FeatureSetRequest; +import feast.storage.api.retriever.OnlineRetriever; +import feast.types.FeatureRowProto.FeatureRow; +import feast.types.FieldProto.Field; +import feast.types.ValueProto.Value; +import io.grpc.Status; +import io.lettuce.core.RedisURI; +import io.lettuce.core.cluster.RedisClusterClient; +import io.lettuce.core.cluster.api.StatefulRedisClusterConnection; +import io.lettuce.core.cluster.api.sync.RedisAdvancedClusterCommands; +import io.lettuce.core.codec.ByteArrayCodec; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ExecutionException; +import java.util.stream.Collectors; + +public class RedisClusterOnlineRetriever implements OnlineRetriever { + + private final RedisAdvancedClusterCommands syncCommands; + + private RedisClusterOnlineRetriever(StatefulRedisClusterConnection connection) { + this.syncCommands = connection.sync(); + } + + public static OnlineRetriever create(Map config) { + List redisURIList = + Arrays.stream(config.get("connection_string").split(",")) + .map( + hostPort -> { + String[] hostPortSplit = hostPort.trim().split(":"); + return RedisURI.create(hostPortSplit[0], Integer.parseInt(hostPortSplit[1])); + }) + .collect(Collectors.toList()); + + StatefulRedisClusterConnection connection = + RedisClusterClient.create(redisURIList).connect(new ByteArrayCodec()); + + return new RedisClusterOnlineRetriever(connection); + } + + public static OnlineRetriever create(StatefulRedisClusterConnection connection) { + return new RedisClusterOnlineRetriever(connection); + } + + /** + * Gets online features from redis. This method returns a list of {@link FeatureRow}s + * corresponding to each feature set spec. Each feature row in the list then corresponds to an + * {@link EntityRow} provided by the user. + * + * @param entityRows list of entity rows in the feature request + * @param featureSetRequests Map of {@link FeatureSetSpec} to feature references in the request + * tied to that feature set. + * @return List of List of {@link FeatureRow} + */ + @Override + public List> getOnlineFeatures( + List entityRows, List featureSetRequests) { + + List> featureRows = new ArrayList<>(); + for (FeatureSetRequest featureSetRequest : featureSetRequests) { + List redisKeys = buildRedisKeys(entityRows, featureSetRequest.getSpec()); + try { + List featureRowsForFeatureSet = + sendAndProcessMultiGet( + redisKeys, + featureSetRequest.getSpec(), + featureSetRequest.getFeatureReferences().asList()); + featureRows.add(featureRowsForFeatureSet); + } catch (InvalidProtocolBufferException | ExecutionException e) { + throw Status.INTERNAL + .withDescription("Unable to parse protobuf while retrieving feature") + .withCause(e) + .asRuntimeException(); + } + } + return featureRows; + } + + private List buildRedisKeys(List entityRows, FeatureSetSpec featureSetSpec) { + String featureSetRef = generateFeatureSetStringRef(featureSetSpec); + List featureSetEntityNames = + featureSetSpec.getEntitiesList().stream() + .map(EntitySpec::getName) + .collect(Collectors.toList()); + List redisKeys = + entityRows.stream() + .map(row -> makeRedisKey(featureSetRef, featureSetEntityNames, row)) + .collect(Collectors.toList()); + return redisKeys; + } + + /** + * Create {@link RedisKey} + * + * @param featureSet featureSet reference of the feature. E.g. feature_set_1:1 + * @param featureSetEntityNames entity names that belong to the featureSet + * @param entityRow entityRow to build the key from + * @return {@link RedisKey} + */ + private RedisKey makeRedisKey( + String featureSet, List featureSetEntityNames, EntityRow entityRow) { + RedisKey.Builder builder = RedisKey.newBuilder().setFeatureSet(featureSet); + Map fieldsMap = entityRow.getFieldsMap(); + featureSetEntityNames.sort(String::compareTo); + for (int i = 0; i < featureSetEntityNames.size(); i++) { + String entityName = featureSetEntityNames.get(i); + + if (!fieldsMap.containsKey(entityName)) { + throw Status.INVALID_ARGUMENT + .withDescription( + String.format( + "Entity row fields \"%s\" does not contain required entity field \"%s\"", + fieldsMap.keySet().toString(), entityName)) + .asRuntimeException(); + } + + builder.addEntities( + Field.newBuilder().setName(entityName).setValue(fieldsMap.get(entityName))); + } + return builder.build(); + } + + private List sendAndProcessMultiGet( + List redisKeys, + FeatureSetSpec featureSetSpec, + List featureReferences) + throws InvalidProtocolBufferException, ExecutionException { + + List values = sendMultiGet(redisKeys); + List featureRows = new ArrayList<>(); + + FeatureRow.Builder nullFeatureRowBuilder = + FeatureRow.newBuilder().setFeatureSet(generateFeatureSetStringRef(featureSetSpec)); + for (FeatureReference featureReference : featureReferences) { + nullFeatureRowBuilder.addFields(Field.newBuilder().setName(featureReference.getName())); + } + + for (int i = 0; i < values.size(); i++) { + + byte[] value = values.get(i); + if (value == null) { + featureRows.add(nullFeatureRowBuilder.build()); + continue; + } + + FeatureRow featureRow = FeatureRow.parseFrom(value); + String featureSetRef = redisKeys.get(i).getFeatureSet(); + FeatureRowDecoder decoder = new FeatureRowDecoder(featureSetRef, featureSetSpec); + if (decoder.isEncodingValid(featureRow)) { + featureRow = decoder.decode(featureRow); + } else { + featureRows.add(nullFeatureRowBuilder.build()); + continue; + } + + featureRows.add(featureRow); + } + return featureRows; + } + + /** + * Send a list of get request as an mget + * + * @param keys list of {@link RedisKey} + * @return list of {@link FeatureRow} in primitive byte representation for each {@link RedisKey} + */ + private List sendMultiGet(List keys) { + try { + byte[][] binaryKeys = + keys.stream() + .map(AbstractMessageLite::toByteArray) + .collect(Collectors.toList()) + .toArray(new byte[0][0]); + return syncCommands.mget(binaryKeys).stream() + .map( + keyValue -> { + if (keyValue == null) { + return null; + } + return keyValue.getValueOrElse(null); + }) + .collect(Collectors.toList()); + } catch (Exception e) { + throw Status.NOT_FOUND + .withDescription("Unable to retrieve feature from Redis") + .withCause(e) + .asRuntimeException(); + } + } + + // TODO: Refactor this out to common package? + private static String generateFeatureSetStringRef(FeatureSetSpec featureSetSpec) { + String ref = String.format("%s/%s", featureSetSpec.getProject(), featureSetSpec.getName()); + if (featureSetSpec.getVersion() > 0) { + return ref + String.format(":%d", featureSetSpec.getVersion()); + } + return ref; + } +} diff --git a/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisClusterCustomIO.java b/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisClusterCustomIO.java new file mode 100644 index 00000000000..0a7634c5c5f --- /dev/null +++ b/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisClusterCustomIO.java @@ -0,0 +1,294 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2019 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.rediscluster.writer; + +import feast.core.FeatureSetProto.EntitySpec; +import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.core.FeatureSetProto.FeatureSpec; +import feast.core.StoreProto.Store.RedisClusterConfig; +import feast.storage.RedisProto.RedisKey; +import feast.storage.RedisProto.RedisKey.Builder; +import feast.storage.api.writer.FailedElement; +import feast.storage.api.writer.WriteResult; +import feast.storage.common.retry.Retriable; +import feast.types.FeatureRowProto.FeatureRow; +import feast.types.FieldProto.Field; +import feast.types.ValueProto; +import io.lettuce.core.RedisException; +import java.io.IOException; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ExecutionException; +import java.util.stream.Collectors; +import org.apache.beam.sdk.transforms.DoFn; +import org.apache.beam.sdk.transforms.PTransform; +import org.apache.beam.sdk.transforms.ParDo; +import org.apache.beam.sdk.transforms.windowing.GlobalWindow; +import org.apache.beam.sdk.values.PCollection; +import org.apache.beam.sdk.values.PCollectionTuple; +import org.apache.beam.sdk.values.TupleTag; +import org.apache.beam.sdk.values.TupleTagList; +import org.apache.commons.lang3.exception.ExceptionUtils; +import org.joda.time.Instant; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class RedisClusterCustomIO { + + private static final int DEFAULT_BATCH_SIZE = 1000; + private static final int DEFAULT_TIMEOUT = 2000; + + private static TupleTag successfulInsertsTag = new TupleTag<>("successfulInserts") {}; + private static TupleTag failedInsertsTupleTag = new TupleTag<>("failedInserts") {}; + + private static final Logger log = LoggerFactory.getLogger(RedisClusterCustomIO.class); + + private RedisClusterCustomIO() {} + + public static Write write( + RedisClusterConfig redisClusterConfig, Map featureSetSpecs) { + return new Write(redisClusterConfig, featureSetSpecs); + } + + /** ServingStoreWrite data to a Redis server. */ + public static class Write extends PTransform, WriteResult> { + + private Map featureSetSpecs; + private RedisClusterConfig redisClusterConfig; + private int batchSize; + private int timeout; + + public Write( + RedisClusterConfig redisClusterConfig, Map featureSetSpecs) { + + this.redisClusterConfig = redisClusterConfig; + this.featureSetSpecs = featureSetSpecs; + } + + public Write withBatchSize(int batchSize) { + this.batchSize = batchSize; + return this; + } + + public Write withTimeout(int timeout) { + this.timeout = timeout; + return this; + } + + @Override + public WriteResult expand(PCollection input) { + PCollectionTuple redisWrite = + input.apply( + ParDo.of(new WriteDoFn(redisClusterConfig, featureSetSpecs)) + .withOutputTags(successfulInsertsTag, TupleTagList.of(failedInsertsTupleTag))); + return WriteResult.in( + input.getPipeline(), + redisWrite.get(successfulInsertsTag), + redisWrite.get(failedInsertsTupleTag)); + } + + public static class WriteDoFn extends DoFn { + + private final List featureRows = new ArrayList<>(); + private Map featureSetSpecs; + private int batchSize = DEFAULT_BATCH_SIZE; + private int timeout = DEFAULT_TIMEOUT; + private RedisIngestionClient redisIngestionClient; + + WriteDoFn(RedisClusterConfig config, Map featureSetSpecs) { + + this.redisIngestionClient = new RedisClusterIngestionClient(config); + this.featureSetSpecs = featureSetSpecs; + } + + public WriteDoFn withBatchSize(int batchSize) { + if (batchSize > 0) { + this.batchSize = batchSize; + } + return this; + } + + public WriteDoFn withTimeout(int timeout) { + if (timeout > 0) { + this.timeout = timeout; + } + return this; + } + + @Setup + public void setup() { + this.redisIngestionClient.setup(); + } + + @StartBundle + public void startBundle() { + try { + redisIngestionClient.connect(); + } catch (RedisException e) { + log.error("Connection to redis cannot be established ", e); + } + featureRows.clear(); + } + + private void executeBatch() throws Exception { + this.redisIngestionClient + .getBackOffExecutor() + .execute( + new Retriable() { + @Override + public void execute() throws ExecutionException, InterruptedException { + if (!redisIngestionClient.isConnected()) { + redisIngestionClient.connect(); + } + featureRows.forEach( + row -> { + redisIngestionClient.set(getKey(row), getValue(row)); + }); + redisIngestionClient.sync(); + } + + @Override + public Boolean isExceptionRetriable(Exception e) { + return e instanceof RedisException; + } + + @Override + public void cleanUpAfterFailure() {} + }); + } + + private FailedElement toFailedElement( + FeatureRow featureRow, Exception exception, String jobName) { + return FailedElement.newBuilder() + .setJobName(jobName) + .setTransformName("RedisClusterCustomIO") + .setPayload(featureRow.toString()) + .setErrorMessage(exception.getMessage()) + .setStackTrace(ExceptionUtils.getStackTrace(exception)) + .build(); + } + + private byte[] getKey(FeatureRow featureRow) { + FeatureSetSpec featureSetSpec = featureSetSpecs.get(featureRow.getFeatureSet()); + List entityNames = + featureSetSpec.getEntitiesList().stream() + .map(EntitySpec::getName) + .sorted() + .collect(Collectors.toList()); + + Map entityFields = new HashMap<>(); + Builder redisKeyBuilder = RedisKey.newBuilder().setFeatureSet(featureRow.getFeatureSet()); + for (Field field : featureRow.getFieldsList()) { + if (entityNames.contains(field.getName())) { + entityFields.putIfAbsent( + field.getName(), + Field.newBuilder().setName(field.getName()).setValue(field.getValue()).build()); + } + } + for (String entityName : entityNames) { + redisKeyBuilder.addEntities(entityFields.get(entityName)); + } + return redisKeyBuilder.build().toByteArray(); + } + + private byte[] getValue(FeatureRow featureRow) { + FeatureSetSpec spec = featureSetSpecs.get(featureRow.getFeatureSet()); + + List featureNames = + spec.getFeaturesList().stream().map(FeatureSpec::getName).collect(Collectors.toList()); + Map fieldValueOnlyMap = + featureRow.getFieldsList().stream() + .filter(field -> featureNames.contains(field.getName())) + .distinct() + .collect( + Collectors.toMap( + Field::getName, + field -> Field.newBuilder().setValue(field.getValue()).build())); + + List values = + featureNames.stream() + .sorted() + .map( + featureName -> + fieldValueOnlyMap.getOrDefault( + featureName, + Field.newBuilder() + .setValue(ValueProto.Value.getDefaultInstance()) + .build())) + .collect(Collectors.toList()); + + return FeatureRow.newBuilder() + .setEventTimestamp(featureRow.getEventTimestamp()) + .addAllFields(values) + .build() + .toByteArray(); + } + + @ProcessElement + public void processElement(ProcessContext context) { + FeatureRow featureRow = context.element(); + featureRows.add(featureRow); + if (featureRows.size() >= batchSize) { + try { + executeBatch(); + featureRows.forEach(row -> context.output(successfulInsertsTag, row)); + featureRows.clear(); + } catch (Exception e) { + featureRows.forEach( + failedMutation -> { + FailedElement failedElement = + toFailedElement(failedMutation, e, context.getPipelineOptions().getJobName()); + context.output(failedInsertsTupleTag, failedElement); + }); + featureRows.clear(); + } + } + } + + @FinishBundle + public void finishBundle(FinishBundleContext context) + throws IOException, InterruptedException { + if (featureRows.size() > 0) { + try { + executeBatch(); + featureRows.forEach( + row -> + context.output( + successfulInsertsTag, row, Instant.now(), GlobalWindow.INSTANCE)); + featureRows.clear(); + } catch (Exception e) { + featureRows.forEach( + failedMutation -> { + FailedElement failedElement = + toFailedElement(failedMutation, e, context.getPipelineOptions().getJobName()); + context.output( + failedInsertsTupleTag, failedElement, Instant.now(), GlobalWindow.INSTANCE); + }); + featureRows.clear(); + } + } + } + + @Teardown + public void teardown() { + redisIngestionClient.shutdown(); + } + } + } +} diff --git a/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisClusterFeatureSink.java b/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisClusterFeatureSink.java new file mode 100644 index 00000000000..c8126c77930 --- /dev/null +++ b/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisClusterFeatureSink.java @@ -0,0 +1,75 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.rediscluster.writer; + +import com.google.auto.value.AutoValue; +import feast.core.FeatureSetProto; +import feast.core.StoreProto.Store.RedisClusterConfig; +import feast.storage.api.writer.FeatureSink; +import feast.storage.api.writer.WriteResult; +import feast.types.FeatureRowProto; +import java.util.Map; +import org.apache.beam.sdk.transforms.PTransform; +import org.apache.beam.sdk.values.PCollection; + +@AutoValue +public abstract class RedisClusterFeatureSink implements FeatureSink { + + /** + * Initialize a {@link RedisClusterFeatureSink.Builder} from a {@link RedisClusterConfig}. + * + * @param redisClusterConfig {@link RedisClusterConfig} + * @param featureSetSpecs + * @return {@link RedisClusterFeatureSink.Builder} + */ + public static FeatureSink fromConfig( + RedisClusterConfig redisClusterConfig, + Map featureSetSpecs) { + return builder() + .setFeatureSetSpecs(featureSetSpecs) + .setRedisClusterConfig(redisClusterConfig) + .build(); + } + + public abstract RedisClusterConfig getRedisClusterConfig(); + + public abstract Map getFeatureSetSpecs(); + + public abstract Builder toBuilder(); + + public static Builder builder() { + return new AutoValue_RedisClusterFeatureSink.Builder(); + } + + @AutoValue.Builder + public abstract static class Builder { + public abstract Builder setRedisClusterConfig(RedisClusterConfig redisClusterConfig); + + public abstract Builder setFeatureSetSpecs( + Map featureSetSpecs); + + public abstract RedisClusterFeatureSink build(); + } + + @Override + public void prepareWrite(FeatureSetProto.FeatureSet featureSet) {} + + @Override + public PTransform, WriteResult> writer() { + return new RedisClusterCustomIO.Write(getRedisClusterConfig(), getFeatureSetSpecs()); + } +} diff --git a/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisClusterIngestionClient.java b/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisClusterIngestionClient.java new file mode 100644 index 00000000000..1f395f02e56 --- /dev/null +++ b/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisClusterIngestionClient.java @@ -0,0 +1,132 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.rediscluster.writer; + +import com.google.common.collect.Lists; +import feast.core.StoreProto; +import feast.storage.common.retry.BackOffExecutor; +import io.lettuce.core.LettuceFutures; +import io.lettuce.core.RedisFuture; +import io.lettuce.core.RedisURI; +import io.lettuce.core.cluster.RedisClusterClient; +import io.lettuce.core.cluster.api.StatefulRedisClusterConnection; +import io.lettuce.core.cluster.api.async.RedisAdvancedClusterAsyncCommands; +import io.lettuce.core.codec.ByteArrayCodec; +import java.util.Arrays; +import java.util.List; +import java.util.concurrent.TimeUnit; +import java.util.stream.Collectors; +import org.joda.time.Duration; + +public class RedisClusterIngestionClient implements RedisIngestionClient { + + private final BackOffExecutor backOffExecutor; + private final List uriList; + private transient RedisClusterClient clusterClient; + private StatefulRedisClusterConnection connection; + private RedisAdvancedClusterAsyncCommands commands; + private List futures = Lists.newArrayList(); + + public RedisClusterIngestionClient(StoreProto.Store.RedisClusterConfig redisClusterConfig) { + this.uriList = + Arrays.stream(redisClusterConfig.getConnectionString().split(",")) + .map( + hostPort -> { + String[] hostPortSplit = hostPort.trim().split(":"); + return RedisURI.create(hostPortSplit[0], Integer.parseInt(hostPortSplit[1])); + }) + .collect(Collectors.toList()); + + long backoffMs = + redisClusterConfig.getInitialBackoffMs() > 0 ? redisClusterConfig.getInitialBackoffMs() : 1; + this.backOffExecutor = + new BackOffExecutor(redisClusterConfig.getMaxRetries(), Duration.millis(backoffMs)); + this.clusterClient = RedisClusterClient.create(uriList); + } + + @Override + public void setup() { + this.clusterClient = RedisClusterClient.create(this.uriList); + } + + @Override + public BackOffExecutor getBackOffExecutor() { + return this.backOffExecutor; + } + + @Override + public void shutdown() { + this.clusterClient.shutdown(); + } + + @Override + public void connect() { + if (!isConnected()) { + this.connection = clusterClient.connect(new ByteArrayCodec()); + this.commands = connection.async(); + } + } + + @Override + public boolean isConnected() { + return this.connection != null; + } + + @Override + public void sync() { + try { + LettuceFutures.awaitAll(60, TimeUnit.SECONDS, futures.toArray(new RedisFuture[0])); + } finally { + futures.clear(); + } + } + + @Override + public void pexpire(byte[] key, Long expiryMillis) { + futures.add(commands.pexpire(key, expiryMillis)); + } + + @Override + public void append(byte[] key, byte[] value) { + futures.add(commands.append(key, value)); + } + + @Override + public void set(byte[] key, byte[] value) { + futures.add(commands.set(key, value)); + } + + @Override + public void lpush(byte[] key, byte[] value) { + futures.add(commands.lpush(key, value)); + } + + @Override + public void rpush(byte[] key, byte[] value) { + futures.add(commands.rpush(key, value)); + } + + @Override + public void sadd(byte[] key, byte[] value) { + futures.add(commands.sadd(key, value)); + } + + @Override + public void zadd(byte[] key, Long score, byte[] value) { + futures.add(commands.zadd(key, score, value)); + } +} diff --git a/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisIngestionClient.java b/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisIngestionClient.java new file mode 100644 index 00000000000..5a0b54e6970 --- /dev/null +++ b/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/writer/RedisIngestionClient.java @@ -0,0 +1,49 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.rediscluster.writer; + +import feast.storage.common.retry.BackOffExecutor; +import java.io.Serializable; + +public interface RedisIngestionClient extends Serializable { + + void setup(); + + BackOffExecutor getBackOffExecutor(); + + void shutdown(); + + void connect(); + + boolean isConnected(); + + void sync(); + + void pexpire(byte[] key, Long expiryMillis); + + void append(byte[] key, byte[] value); + + void set(byte[] key, byte[] value); + + void lpush(byte[] key, byte[] value); + + void rpush(byte[] key, byte[] value); + + void sadd(byte[] key, byte[] value); + + void zadd(byte[] key, Long score, byte[] value); +} diff --git a/storage/connectors/rediscluster/src/test/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetrieverTest.java b/storage/connectors/rediscluster/src/test/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetrieverTest.java new file mode 100644 index 00000000000..567e92a3d41 --- /dev/null +++ b/storage/connectors/rediscluster/src/test/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetrieverTest.java @@ -0,0 +1,263 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.rediscluster.retriever; + +import static org.hamcrest.MatcherAssert.assertThat; +import static org.hamcrest.Matchers.equalTo; +import static org.mockito.Mockito.when; +import static org.mockito.MockitoAnnotations.initMocks; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.Lists; +import com.google.protobuf.AbstractMessageLite; +import com.google.protobuf.Duration; +import com.google.protobuf.Timestamp; +import feast.core.FeatureSetProto.EntitySpec; +import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.core.FeatureSetProto.FeatureSpec; +import feast.serving.ServingAPIProto.FeatureReference; +import feast.serving.ServingAPIProto.GetOnlineFeaturesRequest.EntityRow; +import feast.storage.RedisProto.RedisKey; +import feast.storage.api.retriever.FeatureSetRequest; +import feast.storage.api.retriever.OnlineRetriever; +import feast.types.FeatureRowProto.FeatureRow; +import feast.types.FieldProto.Field; +import feast.types.ValueProto.Value; +import io.lettuce.core.KeyValue; +import io.lettuce.core.cluster.api.StatefulRedisClusterConnection; +import io.lettuce.core.cluster.api.sync.RedisAdvancedClusterCommands; +import java.util.List; +import java.util.Optional; +import java.util.stream.Collectors; +import org.junit.Before; +import org.junit.Test; +import org.mockito.Mock; + +public class RedisClusterOnlineRetrieverTest { + + @Mock StatefulRedisClusterConnection connection; + + @Mock RedisAdvancedClusterCommands syncCommands; + + private OnlineRetriever redisClusterOnlineRetriever; + private byte[][] redisKeyList; + + @Before + public void setUp() { + initMocks(this); + when(connection.sync()).thenReturn(syncCommands); + redisClusterOnlineRetriever = RedisClusterOnlineRetriever.create(connection); + redisKeyList = + Lists.newArrayList( + RedisKey.newBuilder() + .setFeatureSet("project/featureSet:1") + .addAllEntities( + Lists.newArrayList( + Field.newBuilder().setName("entity1").setValue(intValue(1)).build(), + Field.newBuilder().setName("entity2").setValue(strValue("a")).build())) + .build(), + RedisKey.newBuilder() + .setFeatureSet("project/featureSet:1") + .addAllEntities( + Lists.newArrayList( + Field.newBuilder().setName("entity1").setValue(intValue(2)).build(), + Field.newBuilder().setName("entity2").setValue(strValue("b")).build())) + .build()) + .stream() + .map(AbstractMessageLite::toByteArray) + .collect(Collectors.toList()) + .toArray(new byte[0][0]); + } + + @Test + public void shouldReturnResponseWithValuesIfKeysPresent() { + FeatureSetRequest featureSetRequest = + FeatureSetRequest.newBuilder() + .setSpec(getFeatureSetSpec()) + .addFeatureReference( + FeatureReference.newBuilder() + .setName("feature1") + .setVersion(1) + .setProject("project") + .build()) + .addFeatureReference( + FeatureReference.newBuilder() + .setName("feature2") + .setVersion(1) + .setProject("project") + .build()) + .build(); + List entityRows = + ImmutableList.of( + EntityRow.newBuilder() + .setEntityTimestamp(Timestamp.newBuilder().setSeconds(100)) + .putFields("entity1", intValue(1)) + .putFields("entity2", strValue("a")) + .build(), + EntityRow.newBuilder() + .setEntityTimestamp(Timestamp.newBuilder().setSeconds(100)) + .putFields("entity1", intValue(2)) + .putFields("entity2", strValue("b")) + .build()); + + List featureRows = + Lists.newArrayList( + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setValue(intValue(1)).build(), + Field.newBuilder().setValue(intValue(1)).build())) + .build(), + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setValue(intValue(2)).build(), + Field.newBuilder().setValue(intValue(2)).build())) + .build()); + + List> featureRowBytes = + featureRows.stream() + .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) + .collect(Collectors.toList()); + + redisClusterOnlineRetriever = RedisClusterOnlineRetriever.create(connection); + when(connection.sync()).thenReturn(syncCommands); + when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); + + List> expected = + List.of( + Lists.newArrayList( + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) + .setFeatureSet("project/featureSet:1") + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), + Field.newBuilder().setName("feature2").setValue(intValue(1)).build())) + .build(), + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) + .setFeatureSet("project/featureSet:1") + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setName("feature1").setValue(intValue(2)).build(), + Field.newBuilder().setName("feature2").setValue(intValue(2)).build())) + .build())); + + List> actual = + redisClusterOnlineRetriever.getOnlineFeatures(entityRows, List.of(featureSetRequest)); + assertThat(actual, equalTo(expected)); + } + + @Test + public void shouldReturnResponseWithUnsetValuesIfKeysNotPresent() { + FeatureSetRequest featureSetRequest = + FeatureSetRequest.newBuilder() + .setSpec(getFeatureSetSpec()) + .addFeatureReference( + FeatureReference.newBuilder() + .setName("feature1") + .setVersion(1) + .setProject("project") + .build()) + .addFeatureReference( + FeatureReference.newBuilder() + .setName("feature2") + .setVersion(1) + .setProject("project") + .build()) + .build(); + List entityRows = + ImmutableList.of( + EntityRow.newBuilder() + .setEntityTimestamp(Timestamp.newBuilder().setSeconds(100)) + .putFields("entity1", intValue(1)) + .putFields("entity2", strValue("a")) + .build(), + EntityRow.newBuilder() + .setEntityTimestamp(Timestamp.newBuilder().setSeconds(100)) + .putFields("entity1", intValue(2)) + .putFields("entity2", strValue("b")) + .build()); + + List featureRows = + Lists.newArrayList( + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setValue(intValue(1)).build(), + Field.newBuilder().setValue(intValue(1)).build())) + .build()); + + List> featureRowBytes = + featureRows.stream() + .map(x -> KeyValue.from(new byte[1], Optional.of(x.toByteArray()))) + .collect(Collectors.toList()); + featureRowBytes.add(null); + + redisClusterOnlineRetriever = RedisClusterOnlineRetriever.create(connection); + when(connection.sync()).thenReturn(syncCommands); + when(syncCommands.mget(redisKeyList)).thenReturn(featureRowBytes); + + List> expected = + List.of( + Lists.newArrayList( + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) + .setFeatureSet("project/featureSet:1") + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), + Field.newBuilder().setName("feature2").setValue(intValue(1)).build())) + .build(), + FeatureRow.newBuilder() + .setFeatureSet("project/featureSet:1") + .addAllFields( + Lists.newArrayList( + Field.newBuilder().setName("feature1").build(), + Field.newBuilder().setName("feature2").build())) + .build())); + + List> actual = + redisClusterOnlineRetriever.getOnlineFeatures(entityRows, List.of(featureSetRequest)); + assertThat(actual, equalTo(expected)); + } + + private Value intValue(int val) { + return Value.newBuilder().setInt64Val(val).build(); + } + + private Value strValue(String val) { + return Value.newBuilder().setStringVal(val).build(); + } + + private FeatureSetSpec getFeatureSetSpec() { + return FeatureSetSpec.newBuilder() + .setProject("project") + .setName("featureSet") + .setVersion(1) + .addEntities(EntitySpec.newBuilder().setName("entity1")) + .addEntities(EntitySpec.newBuilder().setName("entity2")) + .addFeatures(FeatureSpec.newBuilder().setName("feature1")) + .addFeatures(FeatureSpec.newBuilder().setName("feature2")) + .setMaxAge(Duration.newBuilder().setSeconds(30)) // default + .build(); + } +} diff --git a/storage/connectors/rediscluster/src/test/java/feast/storage/connectors/rediscluster/writer/RedisClusterFeatureSinkTest.java b/storage/connectors/rediscluster/src/test/java/feast/storage/connectors/rediscluster/writer/RedisClusterFeatureSinkTest.java new file mode 100644 index 00000000000..cc1993636ee --- /dev/null +++ b/storage/connectors/rediscluster/src/test/java/feast/storage/connectors/rediscluster/writer/RedisClusterFeatureSinkTest.java @@ -0,0 +1,506 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2019 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.storage.connectors.rediscluster.writer; + +import static feast.storage.common.testing.TestUtil.field; +import static org.hamcrest.CoreMatchers.equalTo; +import static org.hamcrest.MatcherAssert.assertThat; + +import com.google.common.collect.ImmutableList; +import com.google.common.collect.ImmutableMap; +import com.google.protobuf.Timestamp; +import feast.core.FeatureSetProto.EntitySpec; +import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.core.FeatureSetProto.FeatureSpec; +import feast.core.StoreProto.Store.RedisClusterConfig; +import feast.storage.RedisProto.RedisKey; +import feast.types.FeatureRowProto.FeatureRow; +import feast.types.FieldProto.Field; +import feast.types.ValueProto.Value; +import feast.types.ValueProto.ValueType.Enum; +import io.lettuce.core.RedisURI; +import io.lettuce.core.cluster.RedisClusterClient; +import io.lettuce.core.cluster.api.StatefulRedisClusterConnection; +import io.lettuce.core.cluster.api.sync.RedisClusterCommands; +import io.lettuce.core.codec.ByteArrayCodec; +import java.io.File; +import java.io.IOException; +import java.nio.file.Paths; +import java.util.*; +import java.util.concurrent.ScheduledFuture; +import java.util.concurrent.ScheduledThreadPoolExecutor; +import java.util.concurrent.TimeUnit; +import net.ishiis.redis.unit.RedisCluster; +import org.apache.beam.sdk.testing.PAssert; +import org.apache.beam.sdk.testing.TestPipeline; +import org.apache.beam.sdk.transforms.Count; +import org.apache.beam.sdk.transforms.Create; +import org.apache.beam.sdk.values.PCollection; +import org.junit.After; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; + +public class RedisClusterFeatureSinkTest { + @Rule public transient TestPipeline p = TestPipeline.create(); + + private static String REDIS_CLUSTER_HOST = "localhost"; + private static int REDIS_CLUSTER_PORT1 = 6380; + private static int REDIS_CLUSTER_PORT2 = 6381; + private static int REDIS_CLUSTER_PORT3 = 6382; + private static String CONNECTION_STRING = "localhost:6380,localhost:6381,localhost:6382"; + private RedisCluster redisCluster; + private RedisClusterClient redisClusterClient; + private RedisClusterCommands redisClusterCommands; + + private RedisClusterFeatureSink redisClusterFeatureSink; + + @Before + public void setUp() throws IOException { + redisCluster = new RedisCluster(REDIS_CLUSTER_PORT1, REDIS_CLUSTER_PORT2, REDIS_CLUSTER_PORT3); + redisCluster.start(); + redisClusterClient = + RedisClusterClient.create( + Arrays.asList( + RedisURI.create(REDIS_CLUSTER_HOST, REDIS_CLUSTER_PORT1), + RedisURI.create(REDIS_CLUSTER_HOST, REDIS_CLUSTER_PORT2), + RedisURI.create(REDIS_CLUSTER_HOST, REDIS_CLUSTER_PORT3))); + StatefulRedisClusterConnection connection = + redisClusterClient.connect(new ByteArrayCodec()); + redisClusterCommands = connection.sync(); + redisClusterCommands.setTimeout(java.time.Duration.ofMillis(600000)); + + FeatureSetSpec spec1 = + FeatureSetSpec.newBuilder() + .setName("fs") + .setVersion(1) + .setProject("myproject") + .addEntities(EntitySpec.newBuilder().setName("entity").setValueType(Enum.INT64).build()) + .addFeatures( + FeatureSpec.newBuilder().setName("feature").setValueType(Enum.STRING).build()) + .build(); + + FeatureSetSpec spec2 = + FeatureSetSpec.newBuilder() + .setName("feature_set") + .setProject("myproject") + .setVersion(1) + .addEntities( + EntitySpec.newBuilder() + .setName("entity_id_primary") + .setValueType(Enum.INT32) + .build()) + .addEntities( + EntitySpec.newBuilder() + .setName("entity_id_secondary") + .setValueType(Enum.STRING) + .build()) + .addFeatures( + FeatureSpec.newBuilder().setName("feature_1").setValueType(Enum.STRING).build()) + .addFeatures( + FeatureSpec.newBuilder().setName("feature_2").setValueType(Enum.INT64).build()) + .build(); + + Map specMap = + ImmutableMap.of("myproject/fs:1", spec1, "myproject/feature_set:1", spec2); + RedisClusterConfig redisClusterConfig = + RedisClusterConfig.newBuilder() + .setConnectionString(CONNECTION_STRING) + .setInitialBackoffMs(2000) + .setMaxRetries(4) + .build(); + + redisClusterFeatureSink = + RedisClusterFeatureSink.builder() + .setFeatureSetSpecs(specMap) + .setRedisClusterConfig(redisClusterConfig) + .build(); + } + + static boolean deleteDirectory(File directoryToBeDeleted) { + File[] allContents = directoryToBeDeleted.listFiles(); + if (allContents != null) { + for (File file : allContents) { + deleteDirectory(file); + } + } + return directoryToBeDeleted.delete(); + } + + @After + public void teardown() { + redisClusterClient.shutdown(); + redisCluster.stop(); + deleteDirectory(new File(String.valueOf(Paths.get(System.getProperty("user.dir"), ".redis")))); + } + + @Test + public void shouldWriteToRedis() { + + HashMap kvs = new LinkedHashMap<>(); + kvs.put( + RedisKey.newBuilder() + .setFeatureSet("myproject/fs:1") + .addEntities(field("entity", 1, Enum.INT64)) + .build(), + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.getDefaultInstance()) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setStringVal("one"))) + .build()); + kvs.put( + RedisKey.newBuilder() + .setFeatureSet("myproject/fs:1") + .addEntities(field("entity", 2, Enum.INT64)) + .build(), + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.getDefaultInstance()) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setStringVal("two"))) + .build()); + + List featureRows = + ImmutableList.of( + FeatureRow.newBuilder() + .setFeatureSet("myproject/fs:1") + .addFields(field("entity", 1, Enum.INT64)) + .addFields(field("feature", "one", Enum.STRING)) + .build(), + FeatureRow.newBuilder() + .setFeatureSet("myproject/fs:1") + .addFields(field("entity", 2, Enum.INT64)) + .addFields(field("feature", "two", Enum.STRING)) + .build()); + + p.apply(Create.of(featureRows)).apply(redisClusterFeatureSink.writer()); + p.run(); + + kvs.forEach( + (key, value) -> { + byte[] actual = redisClusterCommands.get(key.toByteArray()); + assertThat(actual, equalTo(value.toByteArray())); + }); + } + + @Test(timeout = 15000) + public void shouldRetryFailConnection() throws InterruptedException { + HashMap kvs = new LinkedHashMap<>(); + kvs.put( + RedisKey.newBuilder() + .setFeatureSet("myproject/fs:1") + .addEntities(field("entity", 1, Enum.INT64)) + .build(), + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.getDefaultInstance()) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setStringVal("one"))) + .build()); + + List featureRows = + ImmutableList.of( + FeatureRow.newBuilder() + .setFeatureSet("myproject/fs:1") + .addFields(field("entity", 1, Enum.INT64)) + .addFields(field("feature", "one", Enum.STRING)) + .build()); + + PCollection failedElementCount = + p.apply(Create.of(featureRows)) + .apply(redisClusterFeatureSink.writer()) + .getFailedInserts() + .apply(Count.globally()); + + redisCluster.stop(); + final ScheduledThreadPoolExecutor redisRestartExecutor = new ScheduledThreadPoolExecutor(1); + ScheduledFuture scheduledRedisRestart = + redisRestartExecutor.schedule( + () -> { + redisCluster.start(); + }, + 3, + TimeUnit.SECONDS); + + PAssert.that(failedElementCount).containsInAnyOrder(0L); + p.run(); + scheduledRedisRestart.cancel(true); + + kvs.forEach( + (key, value) -> { + byte[] actual = redisClusterCommands.get(key.toByteArray()); + assertThat(actual, equalTo(value.toByteArray())); + }); + } + + @Test + public void shouldProduceFailedElementIfRetryExceeded() { + RedisClusterConfig redisClusterConfig = + RedisClusterConfig.newBuilder() + .setConnectionString(CONNECTION_STRING) + .setInitialBackoffMs(2000) + .setMaxRetries(1) + .build(); + + FeatureSetSpec spec1 = + FeatureSetSpec.newBuilder() + .setName("fs") + .setVersion(1) + .setProject("myproject") + .addEntities(EntitySpec.newBuilder().setName("entity").setValueType(Enum.INT64).build()) + .addFeatures( + FeatureSpec.newBuilder().setName("feature").setValueType(Enum.STRING).build()) + .build(); + Map specMap = ImmutableMap.of("myproject/fs:1", spec1); + redisClusterFeatureSink = + RedisClusterFeatureSink.builder() + .setFeatureSetSpecs(specMap) + .setRedisClusterConfig(redisClusterConfig) + .build(); + redisCluster.stop(); + + List featureRows = + ImmutableList.of( + FeatureRow.newBuilder() + .setFeatureSet("myproject/fs:1") + .addFields(field("entity", 1, Enum.INT64)) + .addFields(field("feature", "one", Enum.STRING)) + .build()); + + PCollection failedElementCount = + p.apply(Create.of(featureRows)) + .apply(redisClusterFeatureSink.writer()) + .getFailedInserts() + .apply(Count.globally()); + + PAssert.that(failedElementCount).containsInAnyOrder(1L); + p.run(); + } + + @Test + public void shouldConvertRowWithDuplicateEntitiesToValidKey() { + + FeatureRow offendingRow = + FeatureRow.newBuilder() + .setFeatureSet("myproject/feature_set:1") + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addFields( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addFields( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(2))) + .addFields( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .addFields( + Field.newBuilder() + .setName("feature_1") + .setValue(Value.newBuilder().setStringVal("strValue1"))) + .addFields( + Field.newBuilder() + .setName("feature_2") + .setValue(Value.newBuilder().setInt64Val(1001))) + .build(); + + RedisKey expectedKey = + RedisKey.newBuilder() + .setFeatureSet("myproject/feature_set:1") + .addEntities( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addEntities( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .build(); + + FeatureRow expectedValue = + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setStringVal("strValue1"))) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setInt64Val(1001))) + .build(); + + p.apply(Create.of(offendingRow)).apply(redisClusterFeatureSink.writer()); + + p.run(); + + byte[] actual = redisClusterCommands.get(expectedKey.toByteArray()); + assertThat(actual, equalTo(expectedValue.toByteArray())); + } + + @Test + public void shouldConvertRowWithOutOfOrderFieldsToValidKey() { + FeatureRow offendingRow = + FeatureRow.newBuilder() + .setFeatureSet("myproject/feature_set:1") + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addFields( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .addFields( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addFields( + Field.newBuilder() + .setName("feature_2") + .setValue(Value.newBuilder().setInt64Val(1001))) + .addFields( + Field.newBuilder() + .setName("feature_1") + .setValue(Value.newBuilder().setStringVal("strValue1"))) + .build(); + + RedisKey expectedKey = + RedisKey.newBuilder() + .setFeatureSet("myproject/feature_set:1") + .addEntities( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addEntities( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .build(); + + List expectedFields = + Arrays.asList( + Field.newBuilder().setValue(Value.newBuilder().setStringVal("strValue1")).build(), + Field.newBuilder().setValue(Value.newBuilder().setInt64Val(1001)).build()); + FeatureRow expectedValue = + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addAllFields(expectedFields) + .build(); + + p.apply(Create.of(offendingRow)).apply(redisClusterFeatureSink.writer()); + + p.run(); + + byte[] actual = redisClusterCommands.get(expectedKey.toByteArray()); + assertThat(actual, equalTo(expectedValue.toByteArray())); + } + + @Test + public void shouldMergeDuplicateFeatureFields() { + FeatureRow featureRowWithDuplicatedFeatureFields = + FeatureRow.newBuilder() + .setFeatureSet("myproject/feature_set:1") + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addFields( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addFields( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .addFields( + Field.newBuilder() + .setName("feature_1") + .setValue(Value.newBuilder().setStringVal("strValue1"))) + .addFields( + Field.newBuilder() + .setName("feature_1") + .setValue(Value.newBuilder().setStringVal("strValue1"))) + .addFields( + Field.newBuilder() + .setName("feature_2") + .setValue(Value.newBuilder().setInt64Val(1001))) + .build(); + + RedisKey expectedKey = + RedisKey.newBuilder() + .setFeatureSet("myproject/feature_set:1") + .addEntities( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addEntities( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .build(); + + FeatureRow expectedValue = + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setStringVal("strValue1"))) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setInt64Val(1001))) + .build(); + + p.apply(Create.of(featureRowWithDuplicatedFeatureFields)) + .apply(redisClusterFeatureSink.writer()); + + p.run(); + + byte[] actual = redisClusterCommands.get(expectedKey.toByteArray()); + assertThat(actual, equalTo(expectedValue.toByteArray())); + } + + @Test + public void shouldPopulateMissingFeatureValuesWithDefaultInstance() { + FeatureRow featureRowWithDuplicatedFeatureFields = + FeatureRow.newBuilder() + .setFeatureSet("myproject/feature_set:1") + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addFields( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addFields( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .addFields( + Field.newBuilder() + .setName("feature_1") + .setValue(Value.newBuilder().setStringVal("strValue1"))) + .build(); + + RedisKey expectedKey = + RedisKey.newBuilder() + .setFeatureSet("myproject/feature_set:1") + .addEntities( + Field.newBuilder() + .setName("entity_id_primary") + .setValue(Value.newBuilder().setInt32Val(1))) + .addEntities( + Field.newBuilder() + .setName("entity_id_secondary") + .setValue(Value.newBuilder().setStringVal("a"))) + .build(); + + FeatureRow expectedValue = + FeatureRow.newBuilder() + .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) + .addFields(Field.newBuilder().setValue(Value.newBuilder().setStringVal("strValue1"))) + .addFields(Field.newBuilder().setValue(Value.getDefaultInstance())) + .build(); + + p.apply(Create.of(featureRowWithDuplicatedFeatureFields)) + .apply(redisClusterFeatureSink.writer()); + + p.run(); + + byte[] actual = redisClusterCommands.get(expectedKey.toByteArray()); + assertThat(actual, equalTo(expectedValue.toByteArray())); + } +} From 45e6500ca5dfa78c36be95069863c515d762dc1b Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Mon, 27 Apr 2020 17:32:13 +0800 Subject: [PATCH 128/176] Add DoFn to transform (#643) --- .../transform/metrics/WriteFailureMetricsTransform.java | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFailureMetricsTransform.java b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFailureMetricsTransform.java index 65a27fa8bf4..778515ae8a8 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFailureMetricsTransform.java +++ b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFailureMetricsTransform.java @@ -19,6 +19,7 @@ import com.google.auto.value.AutoValue; import feast.ingestion.options.ImportOptions; import feast.storage.api.writer.FailedElement; +import org.apache.beam.sdk.transforms.DoFn; import org.apache.beam.sdk.transforms.PTransform; import org.apache.beam.sdk.transforms.ParDo; import org.apache.beam.sdk.values.PCollection; @@ -46,6 +47,14 @@ public PDone expand(PCollection input) { .setStatsdPort(options.getStatsdPort()) .setStoreName(getStoreName()) .build())); + } else { + input.apply( + "Noop", + ParDo.of( + new DoFn() { + @ProcessElement + public void processElement(ProcessContext c) {} + })); } return PDone.in(input.getPipeline()); } From 8f94f21c3bb74ad2eb4ed58aa181af0126293446 Mon Sep 17 00:00:00 2001 From: Khor Shu Heng Date: Tue, 28 Apr 2020 17:09:29 +0800 Subject: [PATCH 129/176] Upgrade github checkout action to v2 --- .github/workflows/unit-tests.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.github/workflows/unit-tests.yml b/.github/workflows/unit-tests.yml index b84cae395e0..2044396a84c 100644 --- a/.github/workflows/unit-tests.yml +++ b/.github/workflows/unit-tests.yml @@ -8,7 +8,7 @@ jobs: container: gcr.io/kf-feast/feast-ci:latest name: unit test java steps: - - uses: actions/checkout@v1 + - uses: actions/checkout@v2 - uses: actions/cache@v1 with: path: ~/.m2/repository @@ -23,7 +23,7 @@ jobs: container: gcr.io/kf-feast/feast-ci:latest name: unit test python steps: - - uses: actions/checkout@v1 + - uses: actions/checkout@v2 - name: install python run: make install-python - name: test python @@ -34,7 +34,7 @@ jobs: container: gcr.io/kf-feast/feast-ci:latest name: unit test go steps: - - uses: actions/checkout@v1 + - uses: actions/checkout@v2 - name: install dependencies run: make compile-protos-go - name: test go From 93a14a822e58fd8307018074e0ff27e0d3071714 Mon Sep 17 00:00:00 2001 From: Terence Lim Date: Tue, 28 Apr 2020 17:30:13 +0800 Subject: [PATCH 130/176] Fix redis cluster e2e (#659) * Fix redis cluster e2e * Fix wrong directory for maven-cache --- .../scripts/test-end-to-end-redis-cluster.sh | 46 ++++++++++++------- 1 file changed, 30 insertions(+), 16 deletions(-) diff --git a/infra/scripts/test-end-to-end-redis-cluster.sh b/infra/scripts/test-end-to-end-redis-cluster.sh index 7f0d47fc92b..e17eeef381c 100755 --- a/infra/scripts/test-end-to-end-redis-cluster.sh +++ b/infra/scripts/test-end-to-end-redis-cluster.sh @@ -21,7 +21,7 @@ This script will run end-to-end tests for Feast Core and Online Serving. " apt-get -qq update -apt-get -y install wget netcat kafkacat +apt-get -y install wget netcat kafkacat build-essential echo " ============================================================ @@ -72,7 +72,7 @@ Building jars for Feast ============================================================ " -.prow/scripts/download-maven-cache.sh \ +infra/scripts/download-maven-cache.sh \ --archive-uri gs://feast-templocation-kf-feast/.m2.2019-10-24.tar \ --output-dir /root/ @@ -97,12 +97,17 @@ grpc: enable-reflection: true feast: - version: 0.3 jobs: - runner: DirectRunner - options: {} - updates: - timeoutSeconds: 240 + polling_interval_milliseconds: 30000 + job_update_timeout_seconds: 240 + + active_runner: direct + + runners: + - name: direct + type: DirectRunner + options: {} + metrics: enabled: false @@ -173,19 +178,27 @@ EOF cat < /tmp/serving.online.application.yml feast: - version: 0.3 core-host: localhost core-grpc-port: 6565 + + active_store: online + + # List of store configurations + stores: + - name: online # Name of the store (referenced by active_store) + type: REDIS_CLUSTER # Type of the store. REDIS, BIGQUERY are available options + config: + # Connection string specifies the IP and ports of Redis instances in Redis cluster + connection_string: "localhost:7000,localhost:7001,localhost:7002,localhost:7003,localhost:7004,localhost:7005" + # Subscriptions indicate which feature sets needs to be retrieved and used to populate this store + subscriptions: + # Wildcards match all options. No filtering is done. + - name: "*" + project: "*" + version: "*" + tracing: enabled: false - store: - config-path: /tmp/serving.store.redis.cluster.yml - redis-pool-max-size: 128 - redis-pool-max-idle: 16 - jobs: - staging-location: ${JOBS_STAGING_LOCATION} - store-type: - store-options: {} grpc: port: 6566 @@ -217,6 +230,7 @@ bash /tmp/miniconda.sh -b -p /root/miniconda -f source ~/.bashrc # Install Feast Python SDK and test requirements +make compile-protos-python pip install -qe sdk/python pip install -qr tests/e2e/requirements.txt From 0e30ffeab9998d1e1a5aa78477882c053c4ded65 Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Wed, 29 Apr 2020 09:26:25 +0000 Subject: [PATCH 131/176] GitBook: [master] 13 pages and 5 assets modified --- docs/.gitbook/assets/blank-diagram-4.svg | 1 + docs/.gitbook/assets/image (2).png | Bin 0 -> 149255 bytes docs/.gitbook/assets/image (3).png | Bin 0 -> 17434 bytes 3 files changed, 1 insertion(+) create mode 100644 docs/.gitbook/assets/blank-diagram-4.svg create mode 100644 docs/.gitbook/assets/image (2).png create mode 100644 docs/.gitbook/assets/image (3).png diff --git a/docs/.gitbook/assets/blank-diagram-4.svg b/docs/.gitbook/assets/blank-diagram-4.svg new file mode 100644 index 00000000000..fb5e0659e55 --- /dev/null +++ b/docs/.gitbook/assets/blank-diagram-4.svg @@ -0,0 +1 @@ + \ No newline at end of file diff --git a/docs/.gitbook/assets/image (2).png b/docs/.gitbook/assets/image (2).png new file mode 100644 index 0000000000000000000000000000000000000000..d3b359a5988f3b0d75afe44b0ae2cd7d9ef0aab1 GIT binary patch literal 149255 zcmeFZbySsa*DeZ+1uUc%T}nzQDJdP&Ac81ef`qhyfOI#4K}m`zs30IET_Q?KgGh>$ zD4h~}E_r|7-urvsG0s2dpL52?U^s>kYdvw_Gp>2fYd+E1S}Me6=+9tbVG*mTDqhFJ z!j;6rLYAWtaD=&9<30R~rKTuNacGTGJclbcz)Qd8~&n>7nvL}Y8ENI9&3`;`koMja|BC^BOq{{0t0DanD23c~vD zzn>mrQP!R<3rYPCC&Y?5=l<`P!9vcdV#Sy~FqGH)_scM2wLK93PuF)lkJZN4UdYUW z#=`#Zzd@GmXaD^u7PdHfkY$RM9-896-!wdiT&Wewzdub-&{+bx(#}iVQ3U_CJXpw_ z(M$OM;W6bNz+Zl(Qhh-Fhjr6lNyYgOkIAfp&|Y!tI%A3Wx2M6v=JM?N4@c37Z15Lx z8{A95|F#Q3Sd^0U|KTX=|I6F+>4BV)krCoJI7o>g@8!#2GV+x5{=@$u|?_y}tDl6yag^)&~` zP2QT-*C^pf_7r8xz0oUD7YH=(^H}PC@vqZRaB9_AKAjcBoQ5j~PLtbjeXd!d)cC*| z|A>OiQxic&5oD=<*X@$90*bQs-Zx&$C(IT~U;#y3W9oDlbyu33n+d_ex>R!lPmZ?& zMu=i21_lP=*ska^uLUAAGBQvs^ExYR%5hj8XaUY(=ZTs=*=g++_3EiB8DJ@RV60;@{`ddgu$Xd79RO;2zkz#YEPx4J&$Zr!9-7iNI&Pm$L_dS#3<}7iY zQQ-3A&c>mYK@}vyeH#hAnZ7Gu%6<9%8PanxrZWVJd&S^g$7kf;7v2WrQ}1E4#K%T| zQ2}3MI%iUb0^^FcP{~JRm0FP(WWpJ~3ONpa54EgX-m+R_&asE@yw)lf#K&)Y+g109 z#a_xv{|fabya{;pdu7d3ZJ1^8J97yOyz?C9Sde_h4{$T%2lW-ZL~YK$gyepGkF?bH zjVSW*(_Ycb3?O1e-na@^A6mLOKbPkT{~1LKQrPn}0pdg%4QZ%N}SrhK)Cr%!bwGEWR^5DMVRY(Ti1) zz!YQpi>`8cUB1-#-oYhrw6+MW&&_=Ow9rsXeVfTavBMijvf!e# zx>49*H;b^&Z*hoT^TOIrxUCe`+j4n2WMkVi$^JhsDwn6oZ8EDbm}L=TFwv2Mi?-#X zaUi;Vk%wdWv^R_&z=eK{8vFKYuMErJ49Wy>c~;ntzHY~Fa~@i1pO%*!g4Ms7X=Y7A z5@VWnU-!zO_KN){){i;q@I`0Q_qpD%lQGv=@F2|Qgu*crfr84jaG}o9s-3K7qecFB z1^$Qdo!*k038XO-uLlyELxWz!j=Yhm?>KvwvQ}{OWkkWV(V;}8IC(Mn;)J8C(mGs} zDqDdF;=LR`96M*B?IaEtO15(OC4HsT_z#ixpFQ|a)~z>;Oq8`Z^O-{Nv2S8*=AKWI z_+>6n>-nq1I#){Hu*A}aI>Q%h#-?Lt;iB9!;j|Fhf=J+)L{rXJH@FaQ`{Y;O2+OJ+ zx}2SL_>Od|p0o^?=Nm0qJW|AK%#JK?U37QVUg_vcr_YVB{D0F>pS&0?h6v)fzk!W| zSME6WPOr#F#D36jut29bOFI8rcEHwR-{GO3HAl1h^9V~Cb{TGNhtI=Zo9^G0f3qrtu2Mvkjb zx0VKMcUH#owKDtdD_DK~o(RDTi`(LtLMC_;hDGVCw)m__4aKEYda}QeZCc||T3X6d zb8@rL;O_cto4B|*2MccgM_8Ahs0P1%H#pUg&Xf`9+UU$o(`*_4M~@#Ha#P;c!ivdc zUxaHNA0PX?6l2Nr*_7zB!U;LCiqPnenaxpy`fddJ7c z`iv#Kymp7)=*tEkXIwLJcK+=(nxUysL2{qLoKrGbq-fKE4yk;roF^7DO~TJmy)~_U zX?w`6|MvE--=#~Je0c7vk${oo%10$6*gp3B7)R%`;r4U5=&6Xq_3vN}i=`jf(;#lq zAfMg+{?Vg* zy*({Y-0OEnfR#zbW!U{EPbfvL;Qn9BvKbXs-rt-bJg(oJ9B_$yLk&jkyiUoC$pRD* zsKv+(-t)>}r*ZGJCoq*UG+4DisCUayQ&XD?`17l{@%Ui2jZp1-S63Hoj@RnsL}zF0 zG4=HF^4F4EtqC#*1#9D<23xbFeHBhNU5*a6AIad#U-n#~tuUs+ z%3Hu=@ym#rZ+CO^%x16b0`Hyu*|ylnoYx?`ZaWm$>w%SNWlHdxykDDXjSS~CysD+u z^D&;mcW1@$)pd4!e0)jC+88SC$~yxbf`WqV?BQvjOTdayEr~2w7KuoSi2SS{!16{A zGdwsa>ACyc)bt~GBfXG~W+)!<;xWmC2M^55%={0wq8~g+K2&06V++;l`|{-r=e4Xw zH%Xr@JNuo*KDJ4lZ}0DxXZ_xzJ$LTc&z~BS-CbRCytwk8Iyw%P-xT}6dEe;q^79wN zttHT2n$6D+AWw^y;o#uVN)>npal^w@ZtR`4!^m4{%MVd(Di5M~e(aFN9QDec+zxEw zH>=z0KxRtacLRT-;WPPC5p3t<11lVhgFu-1{;IhCLY+*ZBZ(VqzHDvH)zrm8NkwJg zLXA~Bi7EEVWPSQqh)H5~-QXN5KX+HB_)#tK3}Oe{KNo)g{yjIh)kb5!mR#<4e7H|Q z!6EE2`E6jJ45A;z8^db19|QT<2#w!8jf;z;;@0!{{cWtXQ)4qb5S9g2EG{Odf7AbV zH?ILCZ)QIaVg$3rb+9##Ho}EA7H0Rhtdx|6>1nyu{rT>6{R}cUrw5GE%fEhg|7KRg zg}o^$DQO{`w(Uxl=M4=F6}LUnHL3HyRIxOWPa1d&qG@@++^2_k?%Y`^vQhX15!SmC zA~6Z0RO(cAboj#T?7Lg7c$29~fyes`^dj~G8amBwZA!hxL|xG@&R{Iw?_eok*dZ$` zD}2H1;q^^evj+b^G4og_rDpX$;ep9M2U{1)Ahrgc_vPW@vJeFvw%{~cd8;>DHe1M+ zM2Icy_U+qi$&y~b3k^zHU79+AgM)LK5O%GA_gJn(EQ+(_LKHj*?489C2&9WSIzIZt z)0r;nbn(_1&y`X6hl1ID&6AS>5ShbkX)jqKMgA;4PdH=#LWC*s@KObxq$iJykpF>) zk*xcCSGt%>!_HTS!gj+loAqSar`54C&e{$dDQRhNp>Bq<4Y+p?{|hJb{0jCP%cypMD30z7h3)B@bG%lglr)jgyNI(8CUW+ zaP9L%k@jD==JJ$BS2|C02wH!tAvOK%v%kUK=%o)K(6%R|8eIB?MhXe7Kn}q6@{}`F z7Y~nir|KSOWSpfA;%m#w%R_D}MZ#;sk#m{4t4J|7E?muWw{TgchXZy|o!00S1Qj zmd2WZlcQb7V@YxG?cWo=;CDi{Ju6dNQ&gn*2rd;Dg697cI3;Bg1 zg|m(>81l;>2xp@c2ALA~9zA*#b7=+qq2c)8=ioK4x-;A-)}6_vW;?&e+r3I}eU)kq<=GlRT;2d{!^p^ZZ+TGMb0t;p4Ha?*K*OM*m{=85QeAO@*|M^- z+0s?zrKRE?i)i<(q@tGHf3J<*NCx@HvrZJiN+qu;r zdi2hUyX8R+R#g1(y*&YT|LI8B8N>u{3FeOW&%60;B!e;q-79X4=|K z=F)z90*^xge;u;#KY0dub#JAtcku&Tn6SgJ=2qy#;LkPg3nY@cwTMSnN0%U%+ z4G-@e?r+u^?;L(ZxAmMPUPQHYr3u|x155REb+sEV92+aMedWF*-TmoPfo?vD2!puW z`No~RoSfgrDUu1fn14GrDgAf}w0 zWCdHtR9ygDY@ljz#{0CGYloaG9xZ$K?y2w{g;i4&i>Rn5gxLMrSboLcgDq0Yix*>j zT;1J$*QSH>*72GAWKHh3%gs1K?pD-)io~(rx1INZ6^WqRjap3sSfhe|z-$JhpzZ zStj~=W5Cg)%a`sLC#ygGa`EPCzm3nF>+9<^$s5!p^RVK4e0=5muvCN$sF1av#f|Sq zIJCk_}v#1`N6>mYj z2V_?i2yX6hbMMuKihXeCrRS=?zJDNSUW(kfAXiEll9-&VS7CoMc^1+b#vQ;&S>s>J zIykPYR@e`nXBynhKMw|-vb!@A$<#l&&9wtgex2c=WwwX|HD)cqJg;86wjezqI6iB( zaig!UPHN}r=%7TO0k$e`D@*?i;r~R2yF;4>KO%8C}3! zVz@W=c*wB#xgkRR{#M<(gMR5HuoWmREe)anEFgVD*^4;XBLH%C#(ion<^lWcMVw(_k&&@Jc}4|*w+59*H2NexSG<2!jL?fZ ze&T=RGG58+Byy`Iti}0GH<&o8j#E76I3R+}SNN`>YLY52;^%B*z5saJme0^gPZ zBm`gzQwZcnb`=&eqsA+po?kUFF*(0YB=Y;qvnw;=kF&CR4u)QUE4X@i8sdo%P$r)FG{ zVOd0RiW*%^CdVw;-pU2xbjPqnc$9mqlK_X7B1z8mKejH?>!kMOH?Gvwcia6nu97J! zA+a$Pa;r6>&qq4Oe|hlLU9{o%^%5ff)sXT`p6EhWRkUJgG9t= zuWYf8L&RGDSu++-$vZkaDtxE^{Kbo}OB_FSb=`JT?Y-1!RFph+7!q1;S--?`!HWQ*A^I9H^KEN!u zQVP#8Sxn}&X2{un#pbdcabtxRxKMb&a~>W4S@q*KVia>eD?)K(asJ(HZXTmb$IpE0 zzM)GxD=UUIcI~V1;C*kEgtGT-g-TTUeM-dA0H1y~q2hy>pS^S3Be_c1V82e!<(V7; zh*>F?dTESfpr_|8JRO6W^W4&mxr#3NTAnnJEa-Fb_x#%KHD)boNay_TuE`??;VXPKUj z@@65BKowZ9V&EEru&RwShqA3l*?WKE2hdXqbV4ryhd5^2d+uIAy}JIQ15$cwJu?b| zyA@uQ%hWu7bjOO0pQ0kNRyg3$`>RQU<sd<~NKQPSr@MnPj9Kh6{gB)Rtw^7`!E5 zf&2~$ol?5cdF1W6ioVSj+{R8*fFp(@V$Ro!T8-l~+UC@Wu2hZ;l31+K%Z1a%|3x!Wyb zQxlWDh=b0%ckh-<#w)WO@Ws-i6S5NEuAl29@tGR!uJE<*)VtriH}t*xdP>gd_T%2p z&RXyFS=bC|?=_1`G8~pXh43=~H>{n;jx(K|oeQtOYkJxJt=c_J-ok=8Q^SW+@{gpH z$D;PIHX2#^qM*2#-@La2uvu;oRLxx%%?&Ns-j$a23DtKdakuxFkG{YAoqOWx7_e_$ zO?~G88I1vVm`*GrA^0RHmr7Fh6Oa>NrtgkQ_a?qK9PJqU=>C|VZW6m-60ker^@{xj zkAM83R)VFitw;1u?Sts(XC#2@%-&1FIwQQjyu8$KKC4$cr%8y-1wt8kaj7FDNqa>p zB=BJCPUTc>d^~ZZugMGbq{XuafM*!90eji#hM#-!vC`#B@#xscm0XI6F}5vf-yK!Y zZj$8jy8C5$!ME9c4LT6R<`9yfY*~$|b9uJz%lhub@&0BGTMKj?Ysdnc>C0;3QAwK% zeSfKyip2dbE>8{_rF>sLgOVWN9V&}%k2qrcJy(uDPkf%6hSw-BP===A^~oFFvDjDK zbmib8sWtq2!1`igyJKi(PNywvH1-ger-V_Z#@WUd*WKR}TX>5}y`>O%)Ohhp8N1d> ztrHKgp08CWbgcK0lfw#(uHTw`f7hxZE6LE&y5q^Mq+*_uDP|N4q+>#n=_&9@^QGY~ zJXj9gfQ6oOmEBY)2hNb77wb$*EyEeJfd{ZvXXfRNH`?Il*{Y-J9WITf?iXU`uf7_} zjN++;V73%})#LN{czVRwy8G@pHeQRsXDBhwt&zpl7Z_U9ez+q{@0m9SzHLMV==g?jsz-!x`IphD%B5g`-wOqOTukkW2=(>E&ykYcLB;JHnfpp0@b3MM6wG zRE86K+@R@|)EsOncJ8(}9yYq<9|HUURiIg#pPygWzOo`R0kQlc0oCWWIU@9UP-Y-^ z)`a3R#A}j5-F&SshDJAo6w3v#ETKbT{z`5^7R8Mw78$&;e*Ac>>PpIA3dUt2W=F;Eg{3X`RcaoC98S?b_8rmY@;d{HDU@57 zhy4uPQsb!OZG8NaDQWZlGLG>v=gGSJg+KXBYqYb$&pEUG6rA>|-XP^Ia&M$q+Di6f z@aFbjM@!AUpLSkaT3W@z$wkc4UNx0G=DlTr=uY3$HAglspzK)0q#+eG4;H*%pv`gGbB=U}*%gUX?A{`<%H$ByhMgzh$z-x{_{nfJQo zSH1FY(PI!yBI%!8ys3Wj*1Is883nM!lFeshjv!D?9w0a_JK?hsEEHulaH$Vnn@7(g z=L$zLIUT5wDoD#==!aTg!a4Z)69_Dh(-_D5e1Ord`8q#0S5j6%a`tjlmkm%8t8W2( zHCMrl(C8e`O% z_N*sQJ}mDe#$N`bAEl;#GPl%Nyw`2EL!-QPK(IO&$rNA!Y%k>3o-ApNpRKK}ODPH? zofrJv$C8UoO3mN9J_GU;xEiS7?n5oXa%Z-e{&wJ)S)F(D11Es98GN!AE?n^FK`ZQ4 zFtkRJ=5Mbyo;2Jx1~3=kO+L2hb3bPH$_e3%1=;J6HKOC_BnpufJ88DGScIIh^>%&H6fL$ zvfto&U!C`Q$S}mP^vTU73HcrzPpM=iz@J7p8fNIU9-EvBQI*kj!D820c>m@r&*3vY?}D^NF1?rO-`Dn zI66!=SECbFQ`&7VYNk`|jIY3*_$2O~@^ayLZiv+lI3U&Q5_U%0VH)c21T;6pTp<2zznfuJEs zv6QuMGZ&EZw7>&7y>;;Fg^kFf_xAQKIf)T*nat3O(F$7d1y(oy!Gqxnz3{4XRcMZ} zj`m%E+((DWeb|CET=dxD+@8m-=M;dJxsqy@bat-VePPu2A=E`bR=)#lukMlnH3U1; z$in=5rQ_J6uTeLojg9VDS-J0TSnG`7;^IpCY~=t(%Z<#?{N?NG3m_%|_B=N>?A+P2 ziYm4`=}`RZoWQ3nib$Abx+H!~p)Bx#FH!qe>16Dg|B1n%va4mh=H8nnR!li6JvZiI zN;$p>#x?f98AAs{R^EEw(bHoi6pVi7G%h+hIx>>dgWtD)qbuSl^+lhZ;_C*WKg*r$ z?6}Ho!}s>QcS0#y6t@x|OIv(v@RxL8y;xGB5>L-wa~{`hIm-!%6erwySs-TKY{xxtI9Y$JoMJ^T#R`0 z=FQyK__#Qt%H3eqAtrIRFs(c&b!f~2pmEZAyg{LGu)Es3+o|R6Fj*&V-kSn;pm7AW z1XaKZh9k**)f3n_N=}8reN=Jcwv9AfKSvU7cH{%WoNdBP4%zrSCL8lz^AMYrlY;#Sz7r{=#>I#5rL z##mWaMo2;;Uf4yzpH&;|hDK`Wt{qk0XPC5`to)_Wk>Eh;sF!hn@V50=?wF^7A==nU?AyuZJXq41%1py37me9Yn4J7~{lmOp1!o=9n(5az8&OrBNg^sy))7{mz0N6&*EZ7egH0yZ`yn&sD z+r1b!4l#U?FGZ$OU|D7bif5zAmQ1>k|(R_?F>Xsb6; zL#=OZt@J%_haiH$!QsA9&{P*sdnu%mhElwAT7}E=jI4x&k#Hmd3#1lkk6MHCrR7)? z_%CF7L2Km<=Yt}f{De$Q|L572$epEuNNGIIFsN=q19!tDJ_DV0{l%G0b6CQlpdi=` zoo2(gCe@uO{9TkQR|uPGj$0xT+SyZHaURrU_ab0ls`|D(RUl$&v?Of%5S7 zPI-y_&_xa-@Fif!gzUcL04e^?iH)nE#ib$Y&|sO3vX&VI{37Z2>(K6C@aI2!d*wjX zfj_KNOVlsV%ru|ae|Gcw(Lq&-3`O@tQvpbOsJco3Xgem=rl*gCiF7?JL7WV^y18L` zm;jIe^vZjC??GvE)tBbT@dKsrz{{&?RW0Fvc2~=smI+t@-C`Py%R_J40}7yK8GC>C zdBF@og(VHv@}B@{cbA78)~1`mU2Tdd(F$8kSS%ZNdZ0p_&x>z>9NMJ@RZT{DUy3}2 zCB&8gVqXCq*{{)3w%VD;JWnp@+CU@3>-V=lk&Q@FCSb{fr28qw?|+XM18SLfOz+Lm zJKo}bJt&xAd%JskZ9AR8sJeYE7Vg}?e}8>Dyb}_EW$*nxJ`@Ys7o=RiZ;I$J=<0wS zZnk8({QUj90otxq0}A3bn!#ZUL*n7R z%*&&6W<|BYzKxFeyF>2jIH@zZoD_xzTKbyI{&`;B ztNITBBaamsi+!#POC&ibu_$qw4KPp&^7)Jy8InarBq^|3ZtzQIXJ=j@1N#-hI`WtaSG^J>Hd>n++|>F< z=f&KcVV1UOP;q{Q3XDTD6-`SEoWBb665#&;{yFo2i$qYQW_i{C(jW;Zsoh*+It~Q@VC^ zkdJ8uXliQ4MMW(_9||6Qyux7@Xtb`UA{yleSKiG);EW_@7zPuA2j5qYM*30c&*_B7f6-K|HZ6$NGH2@4~C4`A9ktA0lh{m8mwXu1bC;g_*^Oq51 zmLO~#r~%{}%V7D&%WM;lpRX9WYfyn?MTzkUDvVT&Ht7{XYv4lyE{#%}``&iUR=}$X zYLO^F?&tDklM5rP#z+PnY|u0y8BoJW2A(i~Wxxos`X07C3x@U`F%nlO96qFPH;b-h z!5nyoWB?SwjyYyIe1j20OdS+F-)H1YJ1>i)x>1;x9~K$koI2p|w`uYwK#{ToMQV0E z#LmS7F0ZU|Ch{NM871aJ&!Vr4MqudfFmgu-I!`cqhiF6H`(4^A7q6AXMdG*deV=0% zX2X00q4++L8*aVY2JbFLkMabgMf)9mrklbQWfCFgt z5=eI1tZdK7vBeD7=MR|M8UW11^GD8`CdlXOG%Z z#%K&Y@?7nh;mbJk&6>~PLsU;MhdE(N_$yq^c%VUwhGR*oB1!E;ZErV_o!;L6v&jFm z$p2%>|9=S=LTvy-Vki%Bu!YDC%4~@Ddm91XKK<f6? zNl4H>L8^RddZ6|q8IlWolgsl99<~OMXn}1`)O?{J5{`$++%IT@yAC4ZWj_2PK?qYs zxM!9@h4!`~!%`8W;pBdQ^8;GGsRpBm)o-A>^!3As4`h4#CJ2y;R2**>0|aZ^4)$aV z*#AK@;BWX+GY#>fjYAuHuK;cL4-Tluap-~MZ34wJ^edT791+lO&>BBE^aX)H>@Os7 zu}QUM4IN1@JOi^u0Nla6?E^;nrZ6OjR)#8*2oaOax3GyO5tKnxP*D*tOCCrksF#^0 zLV^egx-unGjZsvXdYBuOcP|;MPhRh`k7nD{$$RPg0pvY z(y@xBHf4yrFEY74ZkV@ny(|Er2)>($3y*7huCQVJu6H3A z+NJ{xGV#j5y*}^Pu{6+Ap9bF2!-qnFQO3{i1*L%FHn|1ELz4zS!}{kRK0(5R-xy`A zn0mW93u1&Irz`OK9Cx5o&AMqv#t1ZF69$xS$$0%XdHA)37CCp#;&5->BLceaSC0Ew z6;7XG6O$cDZMP5z8&+3Pu5;W0`Iya1vp!-}YIqY=9AjIP@2q1D4-`3>Eo2bSRT5qb z9Rf_7|4Ai%D)Ik>k?huOhAztGL?HnXk#gJt<;>>Z`gc}|AaT^El&0zC(6Oy!&@;?2 zaIxY9TawX=I?`s&7;NeA1lawZM_7iziMKs{uA zpdTb}x}u9tWC{j_#`QKcP>mR+UcQ@q*rKS&tbH2+wNk!r4PeZ2ShamtUwIsK;-z4~ zi)f5^jq^J#8U5c|3#tZ$k)D3}00&!_f(wX3mOMbEAXN)X1c?AU2glPIU1)abQqV%z zm1^02q31<7G_g|u9(5;QTVH1hgF1I2BQux(bjQ-b?o#fDpka&Cf>KgwSmvM@hHerY zm8gu2Oj=r+l4lqmGcyAMRF6<6ZmnAx89h~g&hl4DB8tt5MhhUYU0hrgR3Ws)PZRQ; zn$jQu9#h`550~Yp8AX=L-7?eF<(xjjtCXWU@tBwK% zZ*{;?%?4g*lspKk4=S>a)#B69lZA^`QOJ{DSaEps##hiR&*y#m=;l7vPwkIrt> zB?&4Di{cNfQ;p{6Fko*a-oOqO&FWa$=_v*MQrG|lY%tVx$>8*%j{qrcZ#LhAyTI;q zZUx0+UNL(7>sKhyt2!7+Su1@IV`F1f7jNX)GR@7;p9xu88524Xeg}2a=*{tE&M0*( zDZjnMV-gYQ&?ONOwEm4CXaPJ31XkK%r07J7VEItIzVQY?_5JTbO`zf+Opg&6a^_o* z5u)mzHbE7!h_Ua4G!e<_1F-2ra$K}FUodDJzT7`v%Fn*&MHzMW??g`1BHV8L14)v9{7yY(9pC;Zom!n`+x`$qasMr z#={;Xw-9^@B~mW4aiybLzQ^_Jj~2;gLV{*dj-w^PJquuk7x=u9e=`cRg#^UQmsW(m z%ocQrDgWJb^}md3J&U+Ppq}X02utjh`9nk)q~y1zcIL70%uWkXL}n6h6Vm-M2HU7FHn#xM1gwC%q;62t4jIligRFu8rSPwNf6Y` z+S=N33GM@BZ2E99Aan`_@4C81fo{~kf;x)IG$hFRZ^G8~0Q$P72>IF_#R9sfa7G~$ zAWbPClCg(*-@pG^N-GTi?;Mte4J^D3-hgn(A$9NbO8A~3h z{eS^VVDyDkxRa}DX;oHLafZEr|2{7-Z*l1WLUb2-F~v{pa1A$(oFRJ|M&z(0V245_IVq<$Bo-O7ribmpCkuk0Gh+7(w&ckkUh+UVq8FEJ&(WFkl!3`8e5 zPG9z8Ou^Zq1;xEqJm}g(2oBXv1QJ<-VS{Kll%t8*R8&u|@DgPE&h{;9q_9M2YHDSd zV^HxG>w9LWrjCLV-u@a2LlwBWZEzbmV<9zFg3j4budcd$ipB%4g+eyp-5W$;FA0gV zCA`tg`h5rI%`aNCW#-Y) znubUL!iD@YF%FKGswB|PR%E~|THkXOC3`|EbO}!|Fg{^DlMp2gM9H^E{_Z-n80@-R zCu$ZOn+^T`-cRcKeFhmn?}y*0C@Bj~YlkN$GDSJV!0e~z1B6j)z2Fa72JaS6M6m#@j!j4q zvngD&-bT*%WNL640Q;)m;PMTWEIEC4f3=DXqK+j#HCW7E2ndlcHUmM^?3O^aeu`*P zF+9fD!{=Af0))XJF7G#Pg8syly4J9;FtAO*J}uOp{ya5JO`_y8DH*&{Jb>7>c;l7b z7r(GP#{xDv7Q_{%w1=QhV`$^d2nvcu@tD=62H*nqW1Gu2P5n1?$yI})C8bL$k{}NL zM#f2mR>ozWY*U@L=cIkgA*R+jV`er$B%hj4&;x;HlICrN2l5Q2te zz~LS^16>uHx2|+>I3Z1c^ldAv%^xY|^!|0wVYJPamXP?mcszqf#wp}L3oR@WRL!() z3>kkB7^)x+7OjIkW+teq5eZ!>6bm%=`OO;+tAMiLHcZ(3MblxF6zoaU4m zz*KK8SJOVNphy+Cg&4XBlIu%kv4ewy(3^Y#aO|$OH z-(a>tBA}6xD;5Hbr(lTW1RqF@8et*=n8A&jiH{#K}Izt=h3aBP1l${pCUp7WNesMy~2%a|{za5u>fZ3&Dk(zGA8{ zrocER*#@W=>pb}N-@>TIcOWHS4B2;6;Q2h{4ThB# zwfhpJMG9Oqfc2-mgJ4LdN@u}mf!SjtfKHk|C@Bv6;eZ0M+@@f+jdRNBEpxa}=#WTIz8zOX6Nfl=IA)QzyR)PCoSH zY8)=|0}@!-mzekicRlN1bCQviM8&``4zmX;Dk?8i7E4{hkfza!sj7`{6=0)CJI(0%o~%)B=1gW)Lw3 z%;ybP8toaBDp}6Y&rtV3Mapw;0&x$7&lBZ$bb*;py`77g{P+>0j09k>sjl84dXHK) zPN%l3rzZ>CYT{2K`Z_Eu?EO%I^pn9c*pG|)Z#5TSdI9w9$Q!z9B)h>E|Iq@Z<@)QO zz%IkJfDt&LdU4M2)Po9RAu!zg!SK970RbM7FBT5k;l%MZCfR_q6ciL_bd$$9;v~jp zMsjj;7$|u7kfo}Gte2Z0L)v$^y*<*}{X-NPObvNkb8j+80RiAL)xgsO27ChHb)FPQ z+K4k4I-UL1yF>cshMd@O^rsGlhK-KU4k1Qyr8WVgR(3gqPejW+ehV7#zF@%+BS9l? zu@OxT!!IyHSOL8esOdUI1CSq4P(dgwEB9B>hM_@k3|fu|(j{Iljg1ZH zobBuiQc_Y>YbX@_{QO#4TPeuN*Ln_5EFX8~1Fln!r!V~gNd6Zn)!YcQT+YJ&uJ7*J zKCH4IT80AI$9S3*^(3kZJbdw84gij%Ph|AK^T4(LH5DDV9vt*{VCY7SA4D7V868k* zf`%E=q!`49B`!T^VRk_J^xj*$wSR*5jB=ViZWFZfWMpLEL&7Q`;D)(M{b5pWXoy2! z8q*NJ^2JUVMidO5=(I!!&)1*|jjJB?ltcGr`JKG&QC8(Z!Dpx6X_P*S|_@JLG zfPtl}KdKfzZ2}zn>TtBeA?1wR+`>XO%-VoZA3`wYIWm<<9yAhf>IC4xW=b@c&Ac^>&j;G;pVDRQJyVb{m{ z<+O^$!v5doF~Q$@l;H!kA(M!f8e2ibsstA1?ft&7QFc5xgCzOYF0`{gefk9K9+X%o zm8~Z{SMw6ljeT;|B)@+yUu9&uilWec$IyxiXt> z6h_aCDNMduDfyjN0m+yuV2LOv5G_Cx3^Sl;1pwwrj~-dRmFb?irmT#2b2T6!0KTSa zR~iCYSa(%D@c#HWSKMcCx8Pmy>A3vJ$aa9aLr0RT5YQIc_hHTnrvAnc5Br`=pw4r0 z4!CGT7klUuYaHU#(ep4BU<^%xUXT{$^ z>rc~t0~banVLzR}y(eRcqtd!(X=w=x7dk#D$3SB+S?!)*@?zNmWCZ zwW|%juzy3OXK{q7gF%A_S}cXAH*dH}@`b5y?)}Ya_b@5cE8vm@dXEVK!wk=Ae!_J0 ziF@-?nQx7OvM`q;EL;xpL^FH?S6^R0Fz{q;Z4IhNXg(THFsvK&W=hgfQhq>QWXS{g z2VfIDZaB6838CC+Jh__=?hX_MFv@I9heuR(8M*U&Lc_KP3WwMMMyFHes1Qb`Yqx)# zWvjGAz*+OeVsY?@N-T*^NRD79&HcA;g4==6I2DV6>h;Yx!%&PtoPqE!#m4r1>f>v^ zOmTPRydef*yOO|@fSaT>cA)^^T?Pxt(s|+bVbsNbBDmPj&Y~z?4*Xy-Lrq=1-zl8? z)aaPk_<^hoG|WLFUUNeI3!PsLRG*zNNVRVsdB+Mjfxg5S;umChFyZgn7lqW6QGfv* zMoVU(CWSH6$Em3W9n!7WG9{q=`jAiqBdjpteEEFwn#V;@F2%q}F`5z9ZjfjGZGKU-i^K~~&yHYP*fmVF80H$h8Ohi*rQIV6+e*fOO zm0AEb2zaD|fWc1A|w7ViLzeA+(L+uh!dkB@Jw^gGC>9r^`j@j||eoLm!tsco?j zHyzl}^z`bSpFzCn;AGYAeVSckK(1Iny7&c+q*R-R8-%bOhQ4;K7X}TL9yG7WLpFgR zs08U67x%um_gYIFzoew(pFe*h!^1&&$mMX;9c~6xpZldSEb9jZ`j_R1-U{#$03(=8 z(LM^oL(mb^{YEBVXj^gRtg7-24V&1qcVM5kbsX6)j-qaVu67a2v%v?5a~Mc#R>DJIXD!nR61F zAR#RpshgXdODPVibObqohzdqwU{4`;0rrCQR(J5D^F|YlL2#Tuf7`}}etJBNeerBJG3OeL z!GHRs3RJZW8O*lEL&+#}JHqNl?hnvN*!52Zp7^tWOh`TT10D@%#=kr4lSCQBH39pw zboDmG3LfgPdtQD=2lpzmvI4O>xg``(jiIIw_2jRxcDM%2;8W6ZwB@k?5e5O0OaTA_ zBxQ^KhUrrpS1t)s%gr=BqBBS$)-UkBgo`A1v6AHLH1BKy1_Kfa6w~%l9RTePNsV3ItxV2#E) zpm7BMcR~@tT=KCx5fT}r$;`+1U{W1s@}GZ-?`wL31ZCPHVh^GTj4qo>j>6k0I@;Sg zFJDg0Vu5H^57WRfRu`=Q$Ky>IV6AjPvpqfz?9cI7zzJ6Zm;%UUDWN1JPG&bzBXT5@ z1)Pr7T^4GL`(k0Y0jr{~`5MiIqRj;K7Q~AH$tN@8B6XGtTEN&i^wyGm&{F!WfTf9Vr3P>H$6a!ghiYw|+s;2VBbM!=^3U-3=BD>!(BYq+MJsNdd`E7s?a`W3p z(IVrta)9RB>e+x+oU#{5q%uolX688xiUBD9%o@kW$Kj08K&RBPM_tcauOz(4WaEOc3bxy~sk4No*<{yGJ ze0|0pZk?K%IwBM+?$a38Ti`C?B`kMd!Z?8gWwz_7rCC(Kju(k;(_Tg$k_JNr?fcUS zN$RK<*RKQ}41NKr2oh9)X-!d)6TBP-0s&;3(%BlXRWqp702v=Yo;$-I+Kyvp^`gmD z)+U`QA`;ozf`mY%e3Ai}sCMn<;3*FlB=-<@fQEo3PF}782`PVx*9InDZ%C?!6UE6~ zErJF_H|ogzKzJL#(Mv{Ta&!9A5}iZyYBzQ%DHD&u?5Zj+P-jAY8rP4TL75;2kQr=b z@l_fI$gF4Nz)M>G7J35UJ*ZJ0f znm3fj$)cxS4MoNqT7^{lxhw=AKvc96!m1xD(bz;&jX;`LIlR_J_}ps!Zq|KYWtY>u z`d^J%W^Ez_^sa^K`?fYWTa&HYunbl>9;sA_OG?5U0Xlxsol9<}k?iDf;<_J#MG_`- z!Ch?u>IIkvV-6b~8#Ar**6GJfIz1_~Hs(b*N-elnXl?Oez%%JFJF5+Uw0>MkYF|4a zP91n7U8-e}1~<9+=X0w`he!A=Nxa4YV=3`l(m1IA;5`pixT2uYHNfU`O6zD#1Ns=s zr^P`Av!X8t9!pIu6v8|!%mOIc5e(#_xps7%l$JgSRw|j|F#*j0?=GUD*#%;;Q?and z>_>H-!o5^>JWNoBj!_@}3bcHaOXU7pXiA`ief!d$;lfM1ctTcSXm?AI8E1>Fa_!r@zn6x>_3Q2xyEikF}t^Up!1I^aP}^m9PNmOkAj)cPPgT^))X+8f%;ps3mo zVg6#h2Hpj4e}kNLIJ4( zR7)BvDC~ZKMOv>Z{fNVviBa!D|AD;W?c+m^At6`4Ra>c2jQ-1gp#N|x9z>V$A;1_J zsZ_x(hebBX6ZGU&Dw4blIlO|b*Ir)Xxvzyy62{HA2B>#HHkE=ylPe3q{GV49n!;J` zJp@+5KqMF(r}oCH>tTsdeZa6$|7$~9DQ{KgP#vlP0hpFD$6KA5Vewh(3G{Gx-`w3L zB_xcEj#kss`Y7nFw&)XJ(Gu+X=))O(8Q+~Q=**z6*LW;J*OE$9j`&}$_|NODcu@su za`1wJAz3g|-ObAyQlfP>CYE7sm^*H>ZLj#iNKJ{pmV4+DJp`1v19rk;XaGMe8|>nQ9_NOr85slM-8-K-3WP5NgA_h2EUciQ0Kzg) zC_kRPGz0sr+-0>a=%s?Sv4uV1HQs@+`bf0kRPurq{0K5UKY?^PE_Mb@6YcMX-ES*sbtnHc@MTwAOC;T1Gn33EV=5WC=N{Obkyva&vrncVE3m(bDyRxH%!Xn)Q_S>^2l32<+Mf6|A<|Kr>~Lc-~-If<~1YbE$Hf4 zv`ckE`P4BJwSkPb-A`Y1;%)yEPZYqH*pX55%sFldNDWW`aD?3`(8dgOTx4>3Hgk*f zmxAh`iO!1k8g!KG=QpNGXN1E}9_=-rGzOkDz)Mb8@&MGpOw4yoC;0TV%1~1{E2g>v zls~`Lr^3{?pbGq@{sQWSGVWbgrM8-7#m?!aYx-NY+IxZZb#+iPu|hJNogJ)KV+i<| zYaUusA|OG`%EngoGd8&Iw2_&b0*iS6{&O+3HghmEEP!nxSAc$N|G)s%1lBvTac@zvhH_xG&z-0OLsKc4rW_g?GXYiH-WzTeMyp2v9{ z#|ihBB)Mvz=dGmZKL^KQBzyy%A;TLcs>Mu({67`10JW<)JMQs`2nGc3 zasdBuKY!j}B<7Xkt2a!e1#w|^&5c8kzIuaJEdfOp`3hitwztw9kC_LET^m5SO+Nnc z$$Cs!pJ%oH-5L$n!_jEZ=`vr~UukNF;1|3(4ic0c26pypK0^L|C%RTX^5A3JoB_1R2*cuyqw(xLk`t362!{5GD14JcKVRqKq#+WrY{gH&hFouW6FqKRz zKhLx-+E3sYWa9GYdfE^c`Prdfl&s*A)b6naE}JJiO!j%$V^Tl+*AWg#G6k-@084I@}hGLx^n& zO-v+W8vGA2jjU|6WN<4oggLS5J{$=w0w>}QL?y^Y*rQ;6c8J;O)B z1$PJW5n*$U0Jf=_M$9g44rzfvC|M1@Pjwto_dRxz>g&7S z>}onSJ^dA81)NIA3~pD)US6<0Y0YO8>QZn@?mtDVBQ?$16C1CLPQ z8h0LEa#=&|VHW|*a2$p4A7~D*Vh-BajDkYAP&M-Ao|Gz>DI@twcO@K1Hk?Q;<^UKI zB2*<02Mdc7U?MuDB{#9YaKR!EB2sBB@?FXd@=*vZIDIt`SS(ME96I2=$B+nvqa6Ue zjBv19{lReyxiB0jZribNldfr7Ab*wlw7eK-ub1E|Ms6r2 zCKjb}DUI*N&5cKbk&R>#)S%Up+cucAwBI2}Sf%WLf=V^bMynnr`vXX+|H?iW z$SrmsXe6{nHbKIM`6o@S55U7bE!8d4rbpMRoegbvW&}5jNgIyu199X z&n@BUbvq`_`d9wxYDs&-#^_c(Ae@7y+FDxQV}LN+m4!cF_a(e62)knTh@xMcWoBd? zhzGEE#qb@Xg)I`O#J^Vd`Til480m1k&<5-sQ3;$ZP0CHJt@nr_)zQ@*kpNRh)cz-A zJ;aE>R_u`J`)9rhrqEM8fAL~*X{n=R!$JY0g!kpk2D5qzU_hF;52MVtkdK0du)X}5 zR~+vTbC9sw4m5&VBi#rvj9{GHY<~a#{lLILeZ4ibR87~`#>2ku3sGlzdiLD8yay&Y zLx>jun?=I@_44zE7cbg-4x$*^vs+qP|5G8d!q8^a2nfR`vkiOPG~4s=qo;Ag3>fk| zrTvdxHo9E?@SO2L2_kFa(Sfz^Z{IC1AMh#J z{rhk;w9&iITzN2p#P4z=f>N{gJm5~mheJzstMnI?UY8nyY#fTe?Cni_xX;Yn{kLOq zJ&j6i+~*@^KRP^oD>=E<3>m*kyBqF^r$F^uMTrOAEhBTe5$uP9rh0(N6uf^6StkEbIz(Y7>e!+!^F zp$}7v1C1!lMLx!IDS6NfBd;3#d}w@phkYD`{btUC&&=ENrdW+ZiV&3mgx;XsUwN_M zfeGBHumbDA8Q5>*MpdW_W^gY&Yqi{s`R{W{(+eQe))eN5rFd{hHX>=8>Uh6DpT4#J5hAoow70O za9o|T&##4|2mFw0Z@dQJ&bWOVUTGhMz^OVA&q>x?sa*h1V~Yf6L(R&a4rRA(LZT`! zew&T~Y3Wv}AcVSjZR%sWo0dRBm?q|KEq~ihB9Fn(gYqZR7uK7=pe1r_)8o)tS0J8R z@-yjU+H=qnN6UJn>Z!$ff>`XgPe2tI-3MnaU5szCbp^|_!tWq3)m*Fsu2?6(aYUMH;^G$Sn|BEqSVe|1{FTk;eT|fUHKNB%HYS^q?z&a|DUO|da_@T5 z_=@IlXzvue!b`$`$;_D3eRSlj1&8CYv2?J!3f9!QFd@ zaYl#}VY-aXnm2-DO0yYt@(A%}gA%t%kxADLul`sT2MMNz6T{f*Bl| ztR2MsYBlpP(XMmFj!ug?S^Wc8TN;KPBu04~gc;dey1#*t!6Uah#C1mo?i;<%+v68U zM@FtTnnfviHHP-atjHuhWBVU7>tNMnx-c6$LpWQh7NySG6lsR_ty^Ui`r zn{9%W%><_sq{>bfhJ-pZpXv3c`mlSW+RSL5&QG`TIuO-jXa zoEFm`T#e}CN<0WU=c#~*;LYg$5z7&ygKS)W6!K?HiI>;czl_hw%8De0W6xP!qg-Wz zVBoKpR#rCayv9+dasW)J+u0dk146R-MMU@PiEH+PJTWZ>^qI4mME&`yYp`f4Ce}D7 zYj$pUVq&6}1UMP5=uSSpRFoZ;yWzy+q*s`h-T>eLT*O~NK7^;aIa{N!Q<3lkF8m2Q zFFo>^>nIr?dFKW+kYb?oa9@^VTMCunS#DJF&(Z)!yQjir-}U(daH-JrbDMCivR(oO zb587m%HF-k7@>VDzuXnyY(0^G^0g_5HWB#i3JMCu8kKssKz)OYj3L}ZL*tZY)h%PJ zPXR+0)|gIRxxDvk9~`*j)396~-Uh}HPs#N0s$XEhZNm~f=`;Q5YIYn9n-pH`b9}ec z;E)~*Vw)@R(6qRze1!i*geepc_d@R^CQ3Mrj*mB9KdYssb=A~Owc>)0ap&zNTGRpV zX7?f(?crhyA`%IML(Z&!#m4d(l%Rg>Mg^LP>nJnXgS0b+_@_6-2aO9K>}{# zv;0g%G$ia02QL{@=fD5D9)BDTkQ0PipBqgzN+3L}uOx<_!M?zt1lL2+&-U;WX)%aO zI7!36LoRrcK{M29CX>nK0jQR9mVHXvPL#jmXCd-u=HuepA~68atuIPSoRhL<{{kMX*SQEJ?rO#eOG1|N;V`r|3XWiMq`hY0LS zJW`c7VqJ~3rKZJz;$tk)D>;Q|ANp>VBPq{<~aD%g!6#{<>XU^TJf30i>vaJsdD; zuG!_u?QCaA+1h`}_<}iu=+>?6@1K6<)-|uwDe?tLfUuvuesaf-hgdM~+`Aq;*V>3* zCV{0ggg_F9AVFpahe@qeBMpo&kPBdrCX;yDTHgj9L#XQ-i1El?1eXm#*ZD(Se5Gq- z6@3-$?CoE(vI~efHc>OKK23x$l_5o|(BOjDQ8Kx%^rCG<` z+!6R^11Al-WY<@^U4i|b=8)yCTLFw5uNoVbFHfI=yAJ#uev?Tjy+TkyOMuw};e6eH z?sAn81!)9cCO8i`Z25syuhv_v3lK*wzDmKZMDaVk`Ok>-a>au7f}^W^9ffs2&7ie{ zpxL!+q=-FK#{7|Hx0}_vz~-AJP#6tef_pNK0P#GKZaZsZ(HhQ6NMI4h)z#GjC?S2| z=R(y669duE(24dBK|1dd^q(%Ypt|-GyBdQNXO^Jm=GuIN&^dK|TI>Fn61xA^8ZI?z zoi&VOk!!>~eQ!Ni)TyDY5(z!e9PgHGSB}KK$0is0qz+W{yjSjf5o&pl|BIwMIK%eJ0<^uLjtK_dGge>mKU2Vm`l;Yl&MM~`4*;AnPm51v3zhC)E zBp||^O=U)ldQ`6lW^Va>t-iu)H&0?EA(3JlED%4IdF_1I%&ajjODHK~MpK~^b)uU;E*g_*Vm2S=54WmA^A!(j`^j*eWe4*@EKy zi;J$V;GM?(ishrib7*kuE7-mVO#wrIfMjIfS*b^R0Zt%EcnH>el~(i)BWKvfw3F?p zzW)7K>bgI?Nmy+zG5lHcO%ViV5pE1X>n!h^rsJBn{Z)~W6lhhRrYpQ{+e-jo97mCM zsKm&ss`}$dor*Yz^*7jkF?QB?J0_v&x*BvOV07XlKBOh}%dGg!Yq0eQYm@1XPRTGn zKu#rFoRM)BoVN|OuRw0XBi-OQ3JZOeFuBJaYsMo}<}_n_G6h% zsBj{t*tws*8aEa)p1?vRiye)8@P5U`lWIv2Ps_fJg%e+^M%@sFW!~XzXh_K%@H6%r zVBBe1dd1TL49}cOg>UCUR#@n-NKdyuc~a0N@Nevk*qy{u*NfX7`g@Y5${S`x_Mm;c zB`r#rSMb=pf+fr=M%oJmg~;?NND)GwR{tC@t6#K_UcV^8bIJ0H$2QlGFIf9>&^6-S zz>?|)EM=gFnQu`P36r|99M0w}9Qc8(mSdQtc-{(4GWvIWfE9?y1X`PmetyqFDnKEA z3KPs3x}#u7Vh0~!sH-4YVjOAuV&FNoOum%;vy)@@V1;G+Qm0rcOw84{kL%%6373%& z4iMM|s#ZR8a&yyYoPiixzRzmI$_}bbjEuA-QHh0jHRK*ITUu_2J1rIf_XDZ`Nvwy2 zo`Zwe2)zzuYe;%!lyrAZDc=bGu3i!gM*5F*>H1EGu_JUc5onn?92sy5bAvC|KgmeEXN=arUul8FywtH0 zT^bmvhH~L7l_0N6CS6dc)=9avbP%^ue?5D^)x4>F}jvWML_(R-|bhgzoU`hiy$T zshzv}C@(K9J>6qk%bNIBl|Wox&~Ve9iW3Rx>FWy%34x9Gp@fRoIGO#a`fXhwA)Uy) z&fEh<94A@)L>_~;bl{)2)Cu4M#|R#*JH*7K**L83pOaoke567k)X69s3JR8<{`3=- z<8-%y_V%OGm6;6*;_uin9qJr_B!r;qC_Zz*>!omrv8gFw@_9fmq0=Vit|Gh>APWFI z+|ijNfFNJL0D>NeFF`hc<}!f9qKSh!u*S+*)Z!HB+0vwPU);M)b_oWMgLY1En34spnXg^rh^^v-3D4mQV>2^?{#CL|MF7yY`PL&z zdDm6?gjcU!EO$ZdkudAXTekxEIiVm-6FTXn-3XctX`|bnzr>>XZ#Cp@q7tp)c=h_V z2{_Q<;q(vxUYZ@x`gzP%#-=G{4&6KeEm?fHx(&(Zp^3*Z_HhpM{`Y3%BQ8_xTQ7W~ z)QYyc!dk4aR!G75{%FMe3XjHyhERczZ>Hi+h7B;Pnv9@OPBQrp*0&?ar zTq;@6T-zb#4j;tX$(m%f{s(+s$Bs4TMEKKs$ z(;mgUWIRL5f`@_%O#{l}0XHrJmb4Rd!)ms_`4g40#@DakT3iiZ-9~*Jh+LcydV_{M zHV0H2NsM-v!j2u|gfqTPYHDgO-G!S+a~^CAap^0~QCi=B^Uum0J-nuR(x&gx&4_{0 zzu6B-1*<_6#2|u2g@tZaL78^DHL} z+OwSoX(W;7eSLr>)VtRFiEqzvl_NIu z8Bn~=;6(B2_{|R+{`saq=L+$K#Hp#iCaeuAZTM3?Mlf@-+D~Qs_Ae;FAUyy_H6w8d z=>Mx%B=gn97SAAd_f2eUEVxpE&g_hguiw0}@(p7XI(QR}YHVL7NVeKO-ecX1!5zW80BYA`i6fgFp#SZ@gOgSY`4930yG z_be(lh$rKhXF87j`0=Bxk@RSFR=F7A4fhli;mVu~=f)+QCBPOIjE4&?xq-W617w7Z z%+-}37G?or@L&ElHZ}%)>CjG&3X#=)skUoV0A_YQHy-El#+Xx%)ZqA8DVK+4Vm$0^ zZRd&FVpf)8BBKB{X8`E|H!^KooZ(*vUPHnL&s5oWh^oFgOohHLR}9A0c}R*C84NMZ zYUdGc4wRa4{nuv`hUkmMrp~5jW)OCWqj(Gm*7S2&*y`5UzUm^euHk_~HUPL2xrPTC z6Yl1kpu1BvIfebt-s6!XX0B2N@yFiLfx&^wi@KFtjlD!uZmH?%-GC{i!_5;}DkyHC zy4o4bzCn`+v|kbFR5i}FL}%T!7XZu`E_<3t1hx?0iV(m8(FFi$w9CmjFCKz~fX%jv zmXSw;8b9Qm=ys@%<(V1_{#jPi%y_!y zw78<%IyD?59L&0^x1Y~Cmi1ro^z>VpN)=%W+qJreqH73aMUU~!5#0m9abUh<_e6>R zd%S$a@#5?=8oQZGY#-@@x&Ms`4w#YTF9g)Y3*-KM&VVwNETmi5T~!l0dLQH6wel_( zDXtEp3^AYA1lB){AkFAmz~-;rtAJ_8D^TLrCWjHdz1|+UUU(| z5qqi3yOYPU@NN}wujRlmTM-wu6sV!chLoH>#Je|qGOTgHU? zPI0?Fjo5!~}lOTK(E>|9q_>b$oa7 zF)<#*%vqg1e**Ki^?^7a?ur?Ze{gGi@pRwsSbu>fggA!9PT_qMJiXKYM6->;DVD#V z09$&^a4mmzjrnpZA{U3qn2C2{!IIgR{L+~Cd7u>$OV}&eAk1E@tVX-NKM~uQj@rPY zJN6{JI+W-%+*ikxiV>Eesi^G6YbodNv8|!2Xy|34X))^3d#E9v%sKHWUg<4SvduVZ?{>G2`cKNK_8kLQwNSSG z9RuT21?t@KR@|&#tM1atXcI?*ohWp+=>*OnF?Zz(!}a`2SVgO+R=_O=gtuIFO>~U0 zG=Vrz#d(;pNP4-4vTpFe$yHSH~Wif@jUB9=Z%jE4?2 zLgPFX?>_3i!J>I}VViQ)6ja(%sI_SzT|Ue=g&D*-l_KbM+Kar#<>{dGZ}Y(v$*FS$Kn+pKHepNSDn5qk&(go z7Z(E12GiQYDAv14few(5G`|%|RQNjLBl~`1DMQLmRzO2$ z)co-~jkZ@YTpHH`NTM8gHY$P7K@o{GK?!~MQ$FjKIQi_Jv(!B2r7q2Hb6a!Kp12%5 z@3%2AjwNGYe;zi#2VCKd3LxP_x2SzQsbA!-=6Y;B^=MAd-GgV<)dP>_bpAUs%iE~^ z1}0P$MV(Tt?uD=>Mg?rZ(Q%KR^FNR8KwVuB2Nxe-HtVLJc#;vBUq?t-gS8^|G`y9f z=f*-vGitx4gbhuuDm8Qo1aJ(=Bec~gZ8z^FuiFJY?R#6Truhy7&PMa>mOJOIh=KC`$d8-ue0 zF?sUQXZrAIj8PZ&H?(lS^O}&;+_(SV-4KWeU~E%dXGjs` z=U9dFu<`{R!0sT zxC~AS&c;%8(Lua8wwZb>K6{*B49B=Bv5(}$%fhmIy3%TCbZVlAQOJrUMnUEpcHh7& z*6a93&92HiKrB-CIZcJB>Q^Uc$NzeAI0j~qV6NbcuPK`H=b_urghrTi3+)Xfu@La9 zPbu;Ivh@tvJ=fdD@Q*e8C2hocGH@wXj;n1WF+e8&TLT1uTGxO6-u?Sc8|$4Wh_!Wt z+f;<=H(pROdHa8O)|$P%OK8`II@2>iOHLfGA7h@9hy6FQ9dUw5qTZfon5}z{spm__hlUj(znjnraQp8wGF6V^ZtF^mNqNiIENe|89M4vlH9) zFBjndXsul|r0nG!cx+HK-om0172rqg+SdPcfMDsbKe*couSR-RnQry@>e#dE5#|?d z#sWz*tB353|GUQqGXKTJfk9-+|A!lcosEs~M?$py=--3yIfK_$yicpzEo{l=~Zl~GqG=9 zFh9B#u)I1c$YTGex8{GFlq-{qxb@6e4)5B;)pi4qH=$@>43y?7+XfA4I>e*q`%M3N zC%lc_1zhf7dm-ywy66wKV7I$n31fJ~z`1yV9j2d%?;I5UPxFZ(0UQQ?58&!||NTC? z`TPH`_tE>Tm#9|zU!K2+Sb5&$Mp`_(t6>sG)dGi0TB=T)vVJ>JSlp<7s)b&;SBbS+e zcw*;f!-prXKV33It;+tmGz6vxnv)SC7@5Q7X?rrd!+hP5IsAdx)PIeM5v@nOXbv4b zc>8pL6+c=?#+*t*4y*8l2^C4h89H>gbocE#3xlBOn;;WDvl5VG!x}%QODhf^rhFrBkeh zC}TqBM!4W4zvz%hbe|Z)ZU`;C@^ZUUgfFVsGyM;d337(dJ&Ukyj}Yp+SLut2+=0`3 zF(x{}h!WPdZ`$xul!qYsJaK>4rt9MJZIj;n2a#B;49(%##8EIIOIn!x22kUlcynEC zCcvyO?;iaPnyz7^JlMgzzAXc{%=Ri3`(}_uoZ&OZ6nLMvwU7|89+h~L3VfjK9VB>P zfFi+$1w@eNcPBs;n3g*$$ug?hl0<;)m+WRTm7T|XiYn~af5PLbuGPZ2n zIbI0~r2zNK9+RC7icY2x18Bi~)6*Zfq};%XO0+SpVmSh*D_KWN;*NI;{O@=Ct+x_{ z9sGx^o;?wO8?#g+up~#>UiXGPrgOb!&Zf;ht%au$a0E;+(=KTfk@&G6d>hkd5MJEE zMP2I5Xb`E&$V%+*I7$HxJ2TYC)P8t`$B@##W-U9?gobpX$m0^k*p7yLu537WE%75u z!~n{U%A&R%=&$LSjw7_zj{GdUC(_g!2uy1p&nlYGMS_ByM+hZq5s44y@d34Tm@b|w z76bh2yIrQY^o!QB%Go@SpgLwifpgkb?|J%%(b#QpPT3Bs3H)2oq8k#2Us&UEMoYlZ zQA1;8W*+Y?A+%x0-5DQ7!z<+W3#7XFc2Q7;pc;ny1R)8BZXG@|0&r8oiiJv5+<{&B z?!gi8eh7~+qURC*5-1sjQ~u*~z0+t}UI>HqaJGhq22kQ|EJG;!{{GtS@CVqjBeD;L z)y=>QA?;VZSn(4T5*(a200@AF2)}=8flx$aVssP({IJTtXM{ zU%!8&UpGi%>GR#A>O@)?aRMC}@W?1pUxz^z_>p*IWU=GGB!RPsg^luf`}r+{hXl?E zSZn<-cO>ezAh?6E_SH>x+VA9!Q~O-+=u;pFpW%YvAlIX5uwij9trrmJgKdw${|dH3 zfoYkzf`SnZF+Sk?z*hGOY7&_0;o)IuNjf1+jeIc(@0AJ$Y#s>h(YHYtt+XKPN7*(R zd@K|=(^68@5}5~9;Hd%#7@fk=+({X9D%dt6*;8^KW%+y;b{^38d^zyVg0L;}24ZpG zI;`sUyPO8QWvJis2X+_AhOkGH+d}xHp_z~m9u|W~yI%e~s2FW0!Livt4;2pV%|PcY z#dKg$P-7#EocjBXPw%nkvj!u%qCyc>3Yt;;M}UziK7f-2BFf$DMj$eZ4a;EHqYY~Y zKADP&3d?|rnOPaOI;px_1x|ibKL6;_Pn~AnA_!2>gXgLL$})!K7s}KCOjJ;AM8O?q zMeCS;pp;4!t_QdEnJ ze6X-)cB4wNb-*1JadM>XVVe8AE~Gvj4Ju1#qYr+zLkQXp*4(x^NrLYP zm_ZN6b@R<0x761MyQd}qF(7QnKzb6X1f%eZDHmDIr9RK)Z9yfREKn9dAbSOMPq4Rs z)lriG z5m|ow_N#C9v<#rr`S%YE=T3dwfuM5VWvokxE@ngpcUgr9GhlxQK2GVq)A91TSi94ETy-ju8u6LhqD%&JI%BOnCG!25N|*f z#4MxqlB1n%QZfqM>G;5wZ&VHY0F8AC!(W1MkW?DQ@0p>=W3JJK3$hY>gsfMvTv2B} zKHiQ^@#)i<;}2kg@sE7 zgQzFWd>c$&Zpp3i_VU7N;t>SpFU3J)9-m0p6hW@r%j zHC?SDY_&Es+k*$+?BD3CNf23bapMAnH!;uB7c)!_v$3`HhdrUIl&8BpSSIG!zJ&7_ z!}jC3Sj1TZ3eJaF4V6PdHnpnMg_$xsjO4(iWl5$D9d^3D7l8;*&V(leIC0=BJ%kVm z)<%^1>1m>yp3}>k8Z~T6RFJ$nQO^B2NU&)2*Jtni+bA=fEPd@XY4dpHWH*Xp8tXi9zKGhLpp)`qtAgh2Z zc<%1V62Na{8r!R)a=dbA%I6H4%FGPYK01W%8&y?Rz1uj`RBZD3!rl{5Md1OlIdB6z zet04f!F@B+@9E(IGlui1T%cFdaGrBl?p^UagyZAAV5oL6HVLx!R!&a33`AG-le!== z4EKXv^A^ML@!$N)Ybq<30QBzX`1D7MJzj-bckW*2*{_uvM zmsiUj<{`Zm9y=t=fP`ZJaXA{I2rqUsIXQX5{w(#Pn}U08?`X%$w~W2PFXKnrQ z^XG2#H(994RH?wIJlq6I<^?x5>eF~R_*~&Z5x&%#zinyJJ3dwI^YgavnbV-|fgwdl zODhL+D_>V>%Q{m07buXv{AQXKHz^h!;O60ZU|Kj|{&Fv`veP71%g3nw#PTIVUOCB=vINCHWagLRHWb2?Q@`6F@Lhdsoi_40;Ek8G1XlPuw&181_?(?olY9n>7B#2OjPh&{&kt}u?uWG zKJU29ElVd7;&f1*1ERP!P4J}b{ka0J8sx6hVWZ!^weqeJRnZLz2`S`ug|mg~&rjT@ zCg`6HmT$|zs|4b$6&M@#p%a+bJ2c+u$NGSb49lKsb^*LLEFJ?IvJRxtg9C(d!Xgjwpq)4X>0G0?Vm`Fkd;9uY;$*@TS>aeBaw*ybYC9wY@8-dt zO1**7NWXO(?9X5daAQMRQITb2bW)Pue2Gl3aGp{_zM(i_X)V+Umw9O1XavrJp;0y^BIa>doGY1w1XIW;|9gui# z9(Vi&koRX(vu08KMBSS=4hV}te4x7kCMddOretB4t1P)6af0Yb_omn7%jf4`po(9c zrv&+ubIcMtG38j}f*_djb1@+1+<_@?o8%pGYQQ=NyBkwiI0mi5ZN!-v!4^&BeQ*!y z*agoOulNWnspGjkRX@4NpkFRaEUzqwE3Cl^goc8K9{NmLD-RqTg&Vn0vL5@lep(O9b9^L}y>Q`j;1&>gBT&=0J2%*< zlY$kIb%R3xI%E4BCMP(oxU`g~SyD=h{gePfm>_9>vgAn)Ek>rXw0EE>}&wn$y;7Mavpc0{(4*ixBIyY!HfJTl>s7pDqmOZr_$DA)1rK76_mxbT!CI{4SLn-h$Q#*kqG;RJ ztwG&L5CVRppk|?ZFJR*Wn7~Eh{R0E3PkQOa@|p=2^z|{8Gw<5l?2&IXCWKgp^S6XV z!BX$)J1;`WUrOW|j%ue34dx#lEdqrgdmXh$! zs@=bl3gOYA7Aw>+Yv)lG>p{v;JYw5{G$h>v8d%HZ^$2qXa|BpIy6673TOxku#y~rb{TG)!^nU*nq-Ho0F?`Yd`=+bJX7F6ycb#D)hFyD4E z?Zs$HkLB&if`S4u5_-|^^F7!tE{;9$_vV8K55`Mdl3iV0@x@&#$poRK$^iz0Eib~K z1fC0zl0$QeFBlr+aW7fjYGy;NDLeD$le8O_-zTV@pu!Tbn{Vkg&Y_y82g$DxtkUjB6 z)}mFPNortI*#X$33&cv|jl=vTxe;3;aW)l0kvO42Y!CYZ*n(J?6{wb^9T0a=IdIw+EaBf;GUC zoZ9V)HggyWQo)I!>vrTIt8k;^Pu^~jT*kd(zX$fZJ#}h>-(jaZbkgEZ{@k_-Q?8z| zXbWEDN<%rr(?8ZMAni}bW?*531NG*7G=o=XQD_D+;%3D6LVr++O@ zNjM|i>LHlZibh?&m#}ki!ReP^fP=mv{i!t?5aX5$Ga-YMM0K&8>x)1}gq@}%b13TRqt&!i=tEG=uFvD4B837>;t#wK;x>~>z2yuCgJ={XJI#8P7@ z4mBk8LaxLc<$^TxgVEL?>8u_mYQCCy$+c1jm)&3 zVQZaS$0szTOPtYFAL|XV1#veg-k?o{Os?}rAkZR~EaoU6-j!DVWh zb)WRGr}t)+|Id+orINVjx;c_3MoN-x!}*tU$E7t&qoaA^X+=H(3n(01z%4r+N1SBG zC18x$dHgcsubEPUZP!y{=t^d&^xDGL_M9+ybpgwJxD68W;k|gKs>Suj3gFy0ylxSP zoole%c1D5~2$n@WEKGr0pgSz%6IVvYh~GFj_4T@>ObWkSO0KEwS>N==^7GchQ|NMv zY_zyd+CBV7`%}h% z%ByADcxfGp>YdTbr#nY#ZXYU)NHib4Bhz8e%|pPjHMh(?ZeN?>Ih~&-wHtK}APd+9@TEC$7 z)-g(uFr;B>H-kJ%3k%U5gh&bI69xKG5OXK@gk3>H5+arG47w}g78r}{SuOBt8o36S ziqF#t=zG5Q1X{7Pv(qPFiM1)0g{d2|?CC?h+3qXkC_uBqM(F$0P z9Q6A}>Ldo4=w!bGmJfKnZ(&V?WW`pAS>NA_atm(BbVF{tX0{>@4gow59$aja+t+7; zA`H}Go=CK7sD-?mJDyldcwvi7g=#4lcDva{T1Uif$Ocif9VIzw%?~~fVHSg;hE37ijKI{uHl+G z6JyT8&f^WM*_xwrRFNgF$#nWsee0$K87`B!EqvOd5y?U**!& znw^$8ijc&{y*2GPVXfX<(70{54pHXO>eMvaf#YK>%X#l>BQ`3LNb zF3pD!O+8-ukxhcEY;7AHV_l>e$k3oY;#{I)GyZt2K0xMLk(r6f)f2jPdioT=S)_}e zC!UVZ+aQcOG+x(oF8s&y+FB~B+^ux%8xH#-89jbez=<6TCp)srO7-HTV1KBr&ns=A z*NFR$UW5wHQ#Q0L+~9&ogwQnry}m#nksld8GUYvZ*4nzBmxWiEL*_GzChpF^H<@0% zbBsa>5P!+Tvjl6UA&&lLxu)W0pjr1n>%Hr3$+dMUbCYJwopQJz;ApnXeE04h#qJt{ zt8u;a9k0u|>S7bfsh4KI@(R*65cKIfJxMBS3S}KdaebnEsxQkLepF0jt(*Yw%0v;!zg)gv|Tvn>pCyAG4xN zJyWu-pVl?5=G`!{uG;>yn>LfwuVtbi8{{V zS%^sdcW%WO&ZNIbAG;)12w?V;8;H)$%NB(@7Tq`@(PdB-r*t>^L5Eb4 zoap%G=Rx$s{joJK=5)fLf4Km$#i&bN*^J%IbuE6|fx2lUxZDI&Na_KticVmJ542{g z7nS%J(@0hUnF4-;(=*rX*@nf;e$$hMro@z$$5OW5e@xf{9X7q>{o!H3GpmW-nOmFp z+m-b7^o%)1>|B3(3HkVhNlpnSfNe*F{lW`TG0!>YJMr;MkGHb^P={7bw;A~weF}7b zmYmmdF4p<^Zb2Jk`^=60zji&3##VKMgm5GIJY&XDMk9qv?NKnZ?RS+cGmf)GB#zVz zxPHMUiZUh99{9}b+Z6HjdHlabols^QU~de9fqLQ66WOQv4J~vL5fMS7@1~>)^PbLg z$U#pUTRVeRa-_0nhbNBfV)H_aK)vTJ$*S!4f-7uqKm!}@7d;n|xYincM%jJyTqxKj z#^;m^q+3blOE}ieQjfrYJ$~NoI!ZpZ199Utj6rKoPgr%keP&{bfHuiLTqA|W{N%}U zg%`DS&(`c`u=F?Vp^{`x7J38U_ok*0J6YI>5G|TLs#F|kN+z{bbDyisXGj*#eZgyr z>=(;I=fW^8{nYq={k^X6>J5!OiGId0pI@T@VCM+g&IulGV`Dx!i*4m-wFUs8rYA9s zN?~DYgMw6NR~N5eaBr&jUrIzYUq%44m4gdJhPWD6eG4pceB$mw-*5^tzc~)Edhks> zRUZJWKHO?c29uGQVw-iVuV6pk+X&SzOp zskfS}t#sX*)q@&xc0Q-t<;nL|>(7A^&&m|#yI+U}t+m+_w~-khzfe3}Zq=%YZS^V^NF^i2`_*v%cZD_+9Dp=6$Jzjo)wDue~MP?%fh*NQ*GfF|70R z*ipY_%a+>oF(TW(&eBhKB_JP8=}~a%dyaw8J9v;9N*V))R-#;3{7hOSV!fEPlUh-U zDt{p|4+a7x^y0D6LdFm$|c^NeCDTs*48KcQVw$8Z6rzn3Bm|>Omgi5R?w~ z&zxtp+Y1l7xsk1Jud!4>=i&3TExy5HWDmAtIxkZwdEiYgT-4kyT!0?~Gb3Z*ygE=S z4JmH&m9gS-aQ8#xf*D1C-3PDbNBMUmFxz{23j?)}@*Uq%F@_*$V4IEVm1CpcdLM}W zoVNh-HVj1~Jg>37Wx|p;%)6Clql%YdYijdW!3$qX1(oFl`I@aMur|t^*Ei_WJw*HF zj@cT)eGgAnT(F2Ya3Q z&m$wX)D)BubEQ(m@4f*D^Hu;bYw>B*u$%>SM$|VCXN=yX{G!G4GIV-|;ct!|Jvzkf zn(RBr|Mz|#QYyNLl#2FuF#YvNy~t5Dw*z>pz`MRaFP!3SOYj#H@lksZG<#0Nvuy;x zC?%TSfoQTAn?Oi%ScmF^eqn{8yo?%aSfT890A+1QdwY5crE1)5Ui1>hjS?)J1skH% zTi0=Z9_1{Z-Yx6B<|sc$o7Z2NLLyDTy&Tko7S^_%%t2Af$?OGYMrdtfW^sTc1ZtqV zkygDblQIY1s>27P#W;n8ng@1eZ|?t~eRBh=)=}}hGkIG>2=?$?M~G5GQbGcAt2VHp z%*Eq#sFrp74LE3MadL|Lg~}s~61s*Ky&+px!wyL30=iG4Kg7^S0l*l9rhK1D6( znxQy;Tlw!SHnmQr|l%&n6`a;B-4r1kn}7Tl-FQG~8kIg4y|`)yjHa^b_n1na)y`bUB6amC|j)jp*~NgrvrF=#Et^f_r}H&`?+lc$f2 zP4b*?V3VSE;G@S|JxYm*jb+d%4v9js9_Y3hx<9~%fGM=@?c49DMy1RchWiXRxUIHs z1Iwx!HwhLWirf4rZXOf5zXECr_UW)fI_hpn--z9M_;3?4%AqaVWQHXPCAb&ez4Op8 z2bkIr(%O2RVx; z|l48Se4 ze&oZbOyFEyQ8-cy9U~cjl8=E3{`J=+@*5~R&z++{`(Zhg#IYOwKAN- zEbyE*GCHeIK^XDjT=veaZGS~Z9)iYK`-WgI+At}lKR^Y5dF{@W{l~+75Qtmu(Z*b$ zh6D$3IEA$&C=|eVCuOfAd>n6xLRk)7W8LrPjg5}TU$AtRaruH3Pn2{oX%M}i@7FlP zip|4Nj0d%LGzlIfKxM}VdPN=lg&?fyzl3MA6e>{lPv+Z^qMmKw*m2s4natTr8<+ip^PMn~rrG{10P zv}sLZJzXFglAT{hHDQdteM-%>ct#wDpdFb2FSN50+iT(pqix@N>gMJwMMOfK3nY|Q zy&TqHd+m#hi@)|1eT2whMEp+jX2k)g3h!B9t!PxS2nkJ(cH!LgQ#gzb{H_XorsRHhDFbIYMB}Ft- z_0hu88voBtG>`AbZ@}UP78)Md=kOd}TwFvlyiQSHC+*1EG27N>IQ_83`t1KIEW~M! zs@LC`?c>TF1@u5LFUDF9r+}MHHlzPoN*eSLspx795r{xmtI<;YF2D4nDLl<0K?K(1>v1Jqs z=F12X$IrC%9=JptC}<{pX=WEu>H1>Nzjo~lp`d1j2?hy3mTUFo2@}kXehVUEVq<_7 z96{wG`cD)bs5hPYdLkBrdW6{HXeJ0o#FZ%wCfw^25^g<5@KT2d-+x^Id5T1=fCfRf zFoa(A@ZoC}FHfPC3GNXZF~r2gfN+7SPqRdrbdVEAvK#-~W&}n}$*R@(UDtYmrJATv z;C3nKOUROhlS*2Dr-Zi%Lmj9M%w?PsH-#A>h=x`ee!wa1Rhn11P#=axd{KoN$Me_C z&E|idGeOz-@GnTw3yx*Apk`}%Lxr#7IVQ(`&lhG|a5r)C*!LE15h{8NZ)n zlRht;k_Uh9r8A0)IO}GLyQbdXmlwkZMtlL-%^bGB<;mrh2q)hAzPurnTku}33GV-( za%pro!gv>$9;IdhBn6b>2f(2~#RVnCx!Gj=#xlrCKv2(SR0aoYATGy-%GV1g8qPwh zj=rkbLPNFS=!fNCJT}XgAxEa>t>hFYJWf`RGSE>at0mnwwb_;;M(u0z^_rbL(oKOG z3fP&K0gqt$j(^?JyiY~NJn`VYwfdPxCr-pvNcyGT0!Ayk#1w??!zq@{J!E%S?M`cdnAv>t8+kj(|2D!-sv#8$T2^tH2q-4G~%X;q;_OxXVy)(Z6AkL z4n!H)CBdb0%xk`5`{vDjtD+-6;1+1hxg`e zm(hhaZ+d%8Jf0N8$xW$da%d>wVVyu5Z0-y=#5z zyI1RftLyR`&S9T>AN$zHe)ar@$QsdT8u6(b2KLjB7Qur~W&_NWz$NRn z6Fi15yoGSOfd5IPa-e2LeX%HRBUTKG;{Hoj+5x6WTO?-xEUdRYyZl1)CUlU^JDC5f zM)}6lHLYnNRre(Q;Dc*`BNK-}OYfa>Uod?w4m1C_>A8NzZ~`ehP$ESKF6UW6fX!?( zp{%EsJ(RhgR?Y;YNSG9Yf;Al-96PcZHh{s0bFOYM*;m!I)!K*@r7U*^o?;q#>CzYm zkKm3IN6E2%32u-^@PduTE>y1hqK11hS)13XOX)LwiQjYkoRye`coW$A$dO(@1asE_s-d+%a(EH z)}rtwNe)A*NiIza@8q$Dvb`KDjVX8Kw_JoSGVf}&0< z?rS^H#hMga9Kj7q@yLG^TKvDLq9iE|QL>6iS=?U%t3nQOzpKk~E{C*{H@w7+=lCDC z7G9AVHQRqHMaYA2E52O?fMiMg&T1Djrg|g~jdCwkcQag!A>4Vq`SM8ekC^$JWaVC5 z|1B&sEj!e8;{h@ngIRcvf2&SGDFS@Rl_#eCkt|aW-f)G-=q?3-QV|n0&XKV9334Uc&JXP>rf)Y(|j$8i9Q-ci&SMZI3c7%+BEVm zzM?i1xt|jgLSqZ-r0r~|;(H6lOVMGt%M>Urh+c}uJ!BWSxe!_t(tWx{=a;Tp*C1W| zWk)Bp^#*}&9C>(eOL*L4h>1hl>Ku!zY6Z0XmspTFFRnGB5Dn8!n;;k}1(6P1be#1d zy#*=Iv{2^sg%@}BV{M-)1E)1*8-jR9KMS0Tm$Fq66Wis(3TEc#nK64DJqqhnKAIR) zkx&*>8GP*7e+8db&J|p} zgAvh~vfLxawS4^i$6Q@^F+iJB%=ghS+!X4)cA^JzPr{T{li>ZCJJ}1tqmm3@bj7^E zL4)ITYiGg-%`jj9jJj>`BB7yKiK1LvjaLpCoj|U>jEofZ-^dC_CY9p;e(TJ$EoTJa zHDC1P#&Pktz;~Gi71Yx##c!YmthFwD_Dm;CTZ<5nv+YnZAoaSbbhRc0dWj{&@u?P6 zxGeJnJ@)WDU2gzCG~ZWaRFaVJ1`bh_;SA&6IFM){B)~$tP}o(S%AY20r*{OfQx~aJ#mXJSye}FX#f62ljiuRSX=>pN+TDy~jO>9}ZSOr2z@J=BG(B;( zzFu86RL(qaZk95&*kXJTdgVI{c=-5kPfIS|;#H(Ncn%vko~ta!x2xvHn_6322`!6T zJ`bI%Y=(#;IkC4eT1t>H``5wpRBQ^|Ktz?SqtGHh)2?}a!D9T+T22lj3tNQks$jDp2N5{aiUo0kCo!bd=l) zhd}s%Xb8gw+#U3$5m4NI+CsWLvucwQO(a^jKOA0LVO-0k8|Hs<#ReFyGxN1dKLD$% zI300!4}wQL|CSi&^kt@0)M1n!2ekGBtO@O=6pkApXh+@jUt&y! z#^Xt5DUk!2k!j0_EEHOHxF;KF%;*@#Y#KnVhJJzCfod9H2hB4G{D&1wW_09iYc)`@ zaMF%dJI65X)`REZnUW6SBcRT#{=H(!OJOlSy_aSb=)GGMSt}?hbt+v%xoS)5k3fI` z+&d}TgFCQ{6cp~Yde^M?2M7YSaD>RYivB1vW2_?1>f_+=o{H+~#H~jSN3yEF2L{I5 zyW~8Br>L?r=@ zQV|q%fNfFv0d?W$P|aQ^Wy8tG2NG>63Qk}Mx6j@>l$?~btK|&5@ssk>(qpKr0+0pL zH_XYcF;6It^QxV|QB;eRb1!Jz_&@WfHGG_hI%eIt^Ezou2F``9odtR3%VrlmU)a^6 zjErhjFm>^(`0ivmhfGfUOK>+`Jdx>PR6qXi5(8{EbeE0;*2CDNXT+A&H{4)(4?qC7 z5Y`<56NU$Y4Z4DKoL*<(;W3K+g2;}CgTo(*D+CN}k6_jcEE!h=69+&hhYf}`3`?)u zl&FV{*<--YIKQBYh710_^LW8DQOxP)hRkf@{Ue=KYKtwaK);rLgRrnOCIak^R0X%U zBn%zavoO8+$*H8Drw@=7J^`1Z086=JaW$r(s+MIEC3fi9I8 zE1`*Iq2w@DzhvUSi7tcx7FtX!_0JcJ?ie-{(P_K?8AI-BwlkLKiL=vhAWT$^*WVcj z@G{A(Hf%(O1UYU=$rnh9`>oN2I2dru+=JQ#TMY^dvf|iDy0A_?Rxm$^nE`T1$MMkE z3U8~+o0*KVKNk9fpmA;ayo42f@9>njrRtFu1atmHTN^8N1L^?I;`A)=d0lKDNN`$^ z^Ic-V61eN~0-}KMZva?wuUxi*pZ_}evfE!rjlyR?g^a%1S{l7Yl(=7JIR{z&ue}kP zfZGb*+@NLQ=DoMO0kpd2ZbboT?)B5&etw6}CW@LkpfJy7U#A*Ms1(p)y>;xLwYp*L zEH-REeHFS+WjZer+Yca8C6CE|IrUX`)Us(1D%#AV{0{y>Ulafq*VN^{tQSS~*4Hx# zD813;baGy(nS!)8dZmU`6}p^5^CS=w@+2>!qo^#7<>=_u%}Z}VsG1Rc@NK8N;UWx| zo7v;o1x3q^B^G@3(0Z<;p`aB5a4cOrXY-_VFBJ>$W)c#|o>~Xw&y_or9OR=;Se0 z`hzfFgbY*ZM;1ewJ9;@dzv%;DnK1=1vKvC`o)TGs`#83D2Xy|bGavytG%x-YfG-jA z+ox)t@y*mwu8>_&?)%gOiN&@=8oc?hH# zhrr#CJH?%Eq}o@WB)yQGpLw8>=H+h>&Nz@xJ2-)?9Jkt{4N03qfk*G7U5Pag7avR_ zdnaIR&8T{BFBOh;X#D`J*ew}|c7E{>d06526j5rDw@?%5@sPF}n@v$I#Bt(=E9yod%U1jb`DtX5VH`;T*U2{Z>d@mb zVMz6UwHaPOhV+a4g9~*Xc9f@q>cIDg8h0pah24_Kh2+}s_Ehw}LywzcI~>a0SJpzI z{!JG|s*Ue}!3p~5j!9QZ+d0xQ6f+IhwS!M+DZq=bPJ-6?f9iW5guFBV8GPzx&^1;z z@6GY&&fR1PNYD`$kDnB09kkdLxAXFDF7I7yyg8nhUzt7LJ?8&OWAv=d?N1tsKZ8r= zf;WFF^f`3>?zD1q(nMPBP*6wjY{hA}qL!}1?>6$sKs0wpFLd34J|yL78iOC_OOdOw=>vlAg)0uiJ-;N5oE& zEA{v6u|-7@n7BIVVea`}ZxS@Ng_ht2A$stz($xaSQ5%Mu>T(=OmPOVkoj7(I!{QtX zQ4?J~QDOloGWpuI6L9}QbtL}W?eC~`mO4C;iayzEH(Q&>UxUYjmL#rt59}PU)A7(7 zIvP2BzUvrX5ZiiDbqXdkY9(|h-`zIx#}k85$5INsmF!wDg^^F(2bodzp@Y%yG_D)L z`71*ha1bTHHKbt@>d*#+e3N=#Z*oA21Zj3rw1IGqQ1ioN&Q_^l!V2t zcqMLD);8p&>h}HZr3;?^3=WaaHs#+@KEQDX#dEKF2PJKdIA^|wt}F2q;i*^`Wu}T* zsJ-xv^X`z^Aor$Cn<8iRf!yL0yt&s88fj!L?bSm$*zDvNA1KXw(ILVFAtzx}?Z{9D(qKmBs#~@cBd|e6rUIrB zu^S>`9l1G}Q^{@XHq|40{RO!$8E{Tf+lg=O-n9!IXGn_~Fy4>Gh9 zC0Yvrf0_mI0#^gW)2p?Lya&2*1{`-70k;{r3Hx8hZ2#b~DjvR71UO;tA$1P#n2H8S~`A}loBS|pSB8+ zIRX0p#^U?u2XQ-{#{(_b?ZtXtY~rclIw^QZl}B~6`0Z%Ut}aL#>`7gZP9HZU1%kgF zn16sI{Eh+k53zSE*ArytVb&zdo|}F^4bxy~1eY4vF1Y7$KHLOnk2ySX!v=`3YCjJg z?N$~fTd3XSh)H$Yw)c8=*tu8@(Jm%<4za9j+D5nU!HAT78iA2^t#X|NL zWg|mF57K*KKg`X5DyyKizW&O}EY!9wc^|9~e6c|dirAL7pTZ9qwx@uHJ`=mMy-_ln z0%WF%j{Ee8N%sD2(CWd_Mp(IbeVRW64Dq(zMv;`QvQCQpX_dfjiGrP*hV6MU`(cir zm!wtve)FQA+KU5nybZX6i?uQEuA>Ra7UWL5#QR^vHkN9hgD|%`vP!752pwsGCAmAg zKH#X^wDqr8~dJh?=hk_taB-CckZ%W0GD)(bUJFgta`N!hu0nj)>Boz<=Tllk8 zr^_-)3o!a<3RtNlamSDV{+OIDGxc2LdV>~fZ*6YqTtHK)z^J@7N4CKy4Y0c4T5gJb zTcwhY6HnS+V*{s>rvSUSs^6dorS)cR?$GlqdQR~@pMdFnSvgB|n?v+J0tQ+eK!{y~ zv1j)@baIN`1K*S~sR)@-B)DLcE=yTJWT5KzSBdwlMnSg|)QyaIg~-d}j?(W_1>=1b zQvRc#0D|3-wi{tvrPX^|u83R`%rnc45lxiIvHclh7=jgsFhT5{@(|D-c2;W8@i72C z=I5qw$fa4;+nl?#!qvQ@oyfD;{x3psV7*(h9}jGn)! z*c-d!PA#I%Ymj3{VzC#?B-v?y;XauWy6Z5RYZRg(8l8AIUK^Ux8)H#a-#EP=3x5th z?+|M$K@ZhW`znLBzg;}-DNO7Qo4S2C_Y&494}oR6;Yr|FgUi`gBj{~K5Da;u1UH4e#Up&7wwz$t*}_pn!AGY z@;yTXKWgLB8$4f;$K=q`2l6=9BCif1&C4bMU%c>T2Xfxs>u5}!0D%RVkP8mc65WmU zc2M#U%-;6%wV7afw-8R8wW?2_K6U5+VkVlrajH?`PHVFPTZtn_w#y$RR)3-S*jp)@ zr^>m5@6a5`{kpvgFBsiYXSU5!@0Pajbij@mbkx)gDJhh(t0gLrej2)i(fX+;%T)>Japx<&49xvkQ$xeMg zDuSywhf=q?CuiiA>sjBrg(cn`0)|@#N@M-iRceI zcE=aJMQoWj6;`FrKN2p=$dptlK`Qyy_job0JdOC+rCr*P4GHQu3*_o#)6Demd=CpJiP7G5a_YA` zPCI@9055Ncj@wVSR&;LXxQuk3q58ua%)IU#&a0*;(5@AkQVqP6#8YuLO8eCuGOgyK zd+zQ%R2QeY5iXsm_*KPYcEnJejY*SPhyJZ_0#JO{mCSaM9TMp2p2n8wQ*@jf4-XmFjL|KTH5g)2|DNykoOV<>k>tW`TF8W{^H3dZ}S&pEg}g&r%otbxpjg zmF8c5-QRReYeE&Le};LqPkL;uz^2>p_Xo!u`T=)OOxmrX&1Dbc@~qUt+(|gDB8FWs zQ|-<0f-a;*Q*g$=S|IE0{xmw&AvG|(JUuyCf7*`s!*>vl(^fr}C69g;tq@VM0#xID z8D)|w{?i-bz{%kaM}`kcHK+0=!r+r>&wh;Td}y{Q*fAl!Qm(Ev zil-tF{<@U^*tAhSyq897-I1E>qclwQqfECVTq8pUW7O5v(|T{eHq?kO*v%m`^=O^C z{M*~x{6P|aXgY%t8*K0<9k zdj_oa_xIfPB2E7lZc-Cz!N{b0R|@sI2OppL!^4Up%$aZgj^h;MPTo4#isJC_4AS^Il`IKQcsuvSa|psHA8-Mc+Xeupo- z`?}emTKaY`VELm_SEflPD64L`%gA*mGMgm7WgdG|z%g=@esj1lnrWM|wVEV^TD_1%_U-QGvA-av~ zKjXI_a0^c~bLpm9xW^m9kEd>`Ux3N#Ry{d?muQp&hEo9E;~~6!`ErS4DeaYpb)Asn z8O%688N?F!=MZ8c@vU3S0QF+su?MvQ7erR5O=3-?BkI2I3SevKv~vwi!K9coPv>5t_lha z3_CJ?#c{7)Ktk<6=V0F?r&a#7vwLbynsh~7-%;M*rjd?9Oti;682fxUGSz3{!=Hr? zE$7nk`1LCA%4Z+~@5;*UzK&w%MIFc(*Y1fUMWKww~Xyn10_++tj>Gl z<1?nyk{@HKovttLSI!J(Fs1rb@tOa@9iQ&LZ5()zB+AeeIZN9@Ar@xN(0R;l(k}PXAVcm!-Cvz ztb_DJKKEhTKyTomBG?)!t z!LcJ9Z^KnM7Zm@55=f)LQ?Xnii(!bN^-W3Dd+a_jeCGRelwWwH|JA!|V4>eYT2qyzd`(#nHcyZKvaDae;f> zF=@u-sm0xtKV!jv*oKNfamfQ0Iysyf9({eJ>;Ac6UH?Fd>J^fc*5PwrPsmDKV;oI^ z>nD?7kel z#YzO8f=y(TevQS^b24X7lM`oQ`gozVf`SDX4W9dVm)jlG5#9teN)Q9Y|0Fm+BN3Bk zXIiY-a)zg(M4pB>caeTUCY?pWhV-65*{_SX$Z4*p%v(d?qe)n4CZ z)y0FmdV2FE&dkAM`A9Fd_cl)$Pz}^gjznrpMXPny^5*ME@2aK*W(f6ls?%`kD(O{w z*PjMB_)xaBDnajOmIUXk!^y^@H90vsrt6mTUE_1uQsEg}k9IzWSMQahoIW8IT#?djP=5i?m^kCc`qAnoKFwd! z<4>`LHh)C{5Q07J+4FLhG}5Y55FWafd-;RvnpZYDCG#4F&)XR<|qO*46x1%YDIs$R*nk(wddvxzRu$M8c)Iz`1AM zXLx3DT8WS4Lat!M>&p6_{-&a5+*TY>IOw}MX4}xXZhaLIw9}-%{IaDp*Vp6yZ1bwx z%!5{^>kL0wMMOt$%ue;3ZV*Zwd@he2=kk=Yf3-3s%Ri<)hBs%$({-%IYP1t>pNo9Q z>8&;y*&MHHMM~`be(CWqx3u$!Zz`F;wdH5#9G_~??F$RE5fKrsc2w_Xm^QJl8%lo3 zO<$=a`(S8BrDGiv_inureiXWC;`f?vox2;8cC{()jsR$pLi-*Rulo_fDx}kkQ3J;2 z_FlLp*o;SYPguhwSA`n0rBoQtEIJPfjU~35N8V+V7!c6sj)nyK*J} zJaA~AGRtQV0M@-eBFs?qEK`ZlbRutHKJ%PdQwEOk0az>2^rKE$*5uC8H5(STUKRDL zp{Ry9evg$_nL+P(NEh@IP)_gL3YZyLD&ive$%q7&nel+7GWqVu!u5 z(|5WdYxCBdCc0?LivsJI%pnWuA)K)E~zcCp6&=r225b!|sIE!13=JFMc)>j|l?4BY5Z2gTk_x8|p_7 zC_6l?Lhm1G-3+sNN_e_2aImhv zzCG+SV4#ub&lZr<0ge`BW^OebYx!UmxbbP9wv)t>L`i#~sBirh89$V=nl@elgWgQo zxxQIB_t=p`zm=Jl1(f{;-4aXA^PnLm{Rk@9v-Y|+Y-~4~Sb1I6u^9p=4p|baMR_nH)(m zJ?_92$@T?$9=!nMZVKESTCSdU_-c(2oM`rw#bZ#>ES z3q@%sxl&3Z=z}~j%j76cNk5)5jW0O0K2`8$b~X-_441YWu+LDD%I!45oIJIgQJ(@n zZY)1@KJjd}pi`w8HY36IB5))(qSB zBP$vxs-BKmeXfC=bE4U86UmzConE7m8BKLfOwLP5!%vMTh19dw*Pza*SRbE$O+L zv3`_WQ8^J?xYvhi<;qtvhJf-eq3u;}H|9}rZ=D@bd6bewu?oZ8PxLZcXrT~kXPA^_Ug&oFThzO;K@xBmE>l{z}sheD%5F}{g%&bAM>vK`F6j# z%#{^NephIehbF&W{qsPP|Lqc1jajOkN;cBYKJS?iHXH}*LQ(yXsCk|-nqKi{b44+$ zeGjQ*UpIU|Nj=c-B*kRrP-Z3T(;^|}do@pSF=m?Ze}lpap=|q%+nvfo`=7!pCodnY zJ|)hD@6=ko-KL>qn+3WUloWNp~6Vf@U@Kqs%wFrt$~%^-{Hd$tRd zFWC6T&(w4cyZE+Wg7qH9(nMys0xzr)&MQluz=B~XxiV5W@jibS+1~0AC^Iq5kK<|F z>i&#k57Arf7~kouFpfJfCnt+H3)k%nTgiU(<@Oyr>VS1`PX68?bgA{R0*a~NH`L$m7QU>2Fky|t-rlBIg=Qd)X~KsJ zrfuut-5(G5(TBP^h(dKwX-y;J0uJ*|{xd6>nfKaV1$A%h#jVFGf3BM^jIcnh)E-YP z-4>T^LpJH}{GqTj!;;;jGHxu!X9(AtMhTukY=7s=CC9@ZbC;^|pXj4~pD6eIn`|BR z<%ZW~1xR6iOy0hLa^f^ zk?}$T6bW!WxUF1{z^rfN@<&sE1Mvgjuv3cESh7&Pgc(JG0RTM^rhv+wZ(C88C*o+3 zy!`rIuOY047`@CzsiJxI4=kg&g;2U|X{Wbqo#LKQ~<7J~O%O;x;HNen5F41{X@w6wm$| zXfrvrq^HSjg;d6PH6qtYl=ki1DPU9%-}Cprx>eGTp!0YRk>F+|h~b#X(<*>$IUb=G zqYlW>Zub{1csXG#()-JzEt}vMc9NVGRyqvB1$L(|OxobI=ImY~ntjC)T6o1H-@#M6 zhmxUJMLTF8A(c|rdjJsnN0H~868nk|0I{vI!h>;!I6;AwCS9*#TXkI7~Cw7^;+P72LQ|*ZVt|7ebSYm13o9=R7Qexo0u5McoRPFSi@QgO9SOUgFn~0AETIPA&r#Bz$`*S zPM&|D{fAE}sF`RJAi9XzalV;o73Vw{`)SWf&&_wXiS&wa_#`-2?CdZS{PAkHbX&G~ zg0@?_ghO4YnUT8i?OR{0ja|qCy#qs4o2h#%og)>r6G^m>rwAjHPz+T zRpK(V{@5d}c7^;~F$7yvVp@Zfn385G5Q!ZBCiNd>9R0x2N^J`cfrRXPO!xUUIG0|B zb8$k9Z?InYS~=azHjCtPp!@FU6bYj@I}I>;UB?j~apFHMiOqp5!%Qm9DAsEc)y^dBshRz6=ZUA z8;E9=Zj~K*fy6s9zzLU=R$5aW!l*K$9Kt2YzrHyAk?;~k--b*rp^pA^S9$Z1I~;lh@X;a9;*d-l1dGtt+r)n%HOT85TxYnGYyntt;@moaP*8MF zLP*FFxg7avQ}YaLQ$jqO(vR=~gUTiOhV&#TdxY{~)rd6CDRc-fHMGMna*lx@E;)C&a(ET3sGnAo!5Mf&e#Hl;b0kQefmJ$-Bmx8FlEk z!HTXC7UyV(nl@lqNHK#MXpN<>Kudcn}x@tPF4-B>^#&Fnzr%RlpD!P>1LXcYAUz~-PU+5mC) z^70fGG&Hi6&|2T#V5sCL6wl+<-QRg7F7BD!bx6q$W`85hYcrxHI@RnH$i4ulQrGTx zy7J4O6VOCd9s-Wy-kK3Q_Y@Z6B*fhZb@>PSfFB(N;jED3?4?r@2T|}Nuc)D>X7j+M z7kqQX2<#vDKo&vqXm)?<1aPCs-#8Y`Z=X+%iHc(54z$0w)4+hF28XXoF*0E^Jlb;@ z_oQBSJe2J5I`6Wmxz=SA&;ras8X2RQk7E^w!3bFYr8#@01eKKu*>ZA$>CF~+IEYCp zvhAU=b}+N3d}F!0UUWBn)jXx zp8*R|@8@MUa*ez=**1f>{{csL@K(4q1NVG9H@-W$+i++b!3ha`2OPg~vT-Ar9*_Ub zOTWCCB>-_(YR8@HBWxVa0qBRRGy&SCY>;t(%w@eVdoH=x;3QWdC;1-`c5nGF9Q_b< zul#1=`sO_`gdObA900P1Yrmks@y+Mvz9uQ@>0t5`ekW+B@cQc;EPetGM;SHm`t`7J z$*@OVZ;9MHFd45cBS!!u0c4E1c+s}h6 z-;d3OVBo`vPljZA}|28Rcv^dKCYji#(!#h4#b z;{CS_eq`o3^u++hz7MPtKYa^ROxU)7+NjST6*xFKpWxYnBGL(=xGDEJK%TtBhnZ}S>*`tsdIE|Q zcmxAuQxPVz z-29b>%(&?BvYtt!dW66~;S6*I-fz1a_Z&oqO|*g)Kk2)HIQwg%INBko^3OIwWxvCZ zMW_DIe*YuT7g3GTc$ShEO@WD5jZ+C;9n%wV=m?Xd@)QvH$iNu*ppBFHh1McZ-w$-A z&EL4zzeZrvpAH-}R{t;rn9iG5fykzI_nM<3Gd*iG%Xg!Ck8{ys+)93{4hqIA(031Y zf}|w*c?|-{XF-?VxoYHA-r?GRc7@4Y0*!5gb;ak@>J{}K3#Hyy9EVaL&wNg)`Kt75 zUDos8?NPQ@jXrF#l5uL`rU#-V_?E)Vz)*~iqqO05JDh)x6^1s;6@m7%dPvf8B|>7A zwJ|+QVolEZkeecwV*LqjlFHuAPmi2@+TAsC`g~LP76~>bU@<4$HPELHC&*Prc7)KT z)5=ru>2!5;w$$CYabtB=d;d|Nlh6S4SSe~h=e?C_QXoTE^LiGk(kP{qQ-!%4%TK?&jLqMcC;^Hvzl7>9ydmYmZzL%R~7tX=w1tC2x%Hw@t(S9 zl-M$?q2!l!llpD(IidKqHs&`D-n;foJ1)fz5kK~uNh}+^FFky;jlJ9w=6R4{8}eOa zHakqMmGetfuAx*!H(!^-uFaSRh2Sr=U4$iH1^>dU%^18T1C9yb+h6@(PwKL~Z)teI z!f2%#!a#am)D%^YH%fG|-`x4d8wnm~-}tto7+>_;rl^vaaYN2glo z3d97j*@>=X_)RWHP;kJ*_g_wr88`nQUq5OC#L7!vq-`ry`7CSYuj)xL$BYkpfJAc6?;yNnrBoHeWiX$D!K$MwVYlc?RS6jAfN=4nC=B4#g?V4Qzw10f5 z@C+ru)@rLs?clwf2l-hl5v;2v5Z~>cY0%!#uNp z63_oxPRsHuk%njXv82tO0KAE{!oZTyLIawZo>OE0x(1xc)-QxQa30$W%Uf-`4h?Mw z!@s>YH@(^p{D#^Q%*pEcEp%ib@c!d=lB+~82=sB&D*1JkKY!5I>CC=1e`z3NsE1%#zYuZxQZsZiVi)C=S7;oc{)$3EaOeB zYDZk)WftDErIlHTByHOJJ2#(41z!IH{{_gfAMkQ{ykdF%sG{w~^Cx-9^Vrh;>)xx# zy+d-GLLK*p9B5}C>j3Z{Qa7U@#6}Cq-U-E3Sy&}E;hAKN&n=;SieF&NDv>znct2{{ za?R>Aa_{!AzxV!?>p1PH_fN)_h9~fIbOsIWG)o9RCf0&X0SB1^nmy0y>TlCL$zi1< z{$A2+j4FKu^(|Y7uBg zV@%SlrX%mM;eY2nJgMTWL-&DB96rw@aDb6KKF@#i_;|;2al06KDt3h&@l*0kh?scq zNo|(f&csrDo;-2PfAhqcIPK>*m&NpK)KpOAsnCBm*(R0i;G?W z(*SzNDxL~D`jj1*;`NzMrb~!NpUfL$EBtY8D^JD5i`XUfg&XiJ=|W_6N6^qoVQCP0 zz)~cK&g>DwTDoW~n~yaI+rw0UdyUs=P`m;THPF(%U0g)0|I0;`cZeUMVi&;3uI+W|Yn zQF7m{qnSZ)0()*^_1J}wG0KOtPqP`7{1S^durcf-J3~$9zgaynBj{e`6b&V^`v1HH zP5o(Rro^qVS2gDi1P;)W_0IDDulGaYj(aaCJqFOPCO1Gs>qZW!A;i*)N6UX*<|ITk zIMUa-htB*ifosMW)5iJr#qQBC12Trjp&>tsL6;*2orP7idMVb7l%`opHGrRD*K;Ak zDpQcfCKN=t3p-enI>@GqU|2^CH>YS#XvjhQ!g-T->@}44x@VUN^cfO!_T^uO8|Qjp zc^2HR)jYM%hBPtiK0AyX@!XUA-&ilipWR$J;s528@K4UNi)6Xitq^A^PA7&k=f7X~ zRmhnj7)Akq&wno3kUJGU|998@KI|F8i2En%{I7s@cB?J4~(W zbo=v`D`wt)-R+zPO4NsK3mo*C)xyBw|`(duS8%`5sk(>`UJJ*BMY=gFj}^@ow~xM;9r`U$1tQ>u=^-|r8*IA_5f zJ)NxOx*Na#S;YArlagPPaik_DBNR0X)Th^5*8D<*5!pKfv!BwkwdRAw>5tD=Ol)&{ ze9?2Dfi-wr=&7=Kpk{1OI|#$=?nTYoY1_xAdydufUtz3Y%6WR*AqHe9>jt=nx4)B- z=nYrdZDyh5$GDa{E*ROo@RG7Z&L2pie*X@d6U-_g&g*h-JsU_IF1CNK+V_1Fqimv6W#tT@^9;94A*C zr>A9UCG)KLM2ZX=4effmJCDZo0hg&B|LWxPY_n=#=jOV|F(hf%_MalI?wy~UUsury zc*5gXHC+1lmxSoQtHYQ*+PkEFv@+;ya=1Zu*E_82+=c zePj;|mEuJfTI_~VW(?!(Q{L<|WPt8qpHJW8tGHj~iQZk+S3i)yS4iod)eD_8_FqQw>I9p>mp`OtV@VUz^oc%({%R$I~6IwbM-@Pf&f54!mtBhrWjD zS|M{$RKytNTiyYFJV623sKVi=Fk#b42&95WY1T3?F2D>dirZOdg7C?044r=h&p38* z_hBrp=?6=U)&8!h7(8Um$|-zOu8(=O@NQU+=e>@*e;8uY{6W?M4IOdGRm3~-NuM}g ze(&=QxOH_9-uLu%Rhrtf7i{rn64$x3H%71{~G3M`y!eJ)#NRg&$)K8lE=nS#)kH2}#bJCZr#Pk-v zy5h2&+@(Ab-}r1-H5vmNvPVtC*dPX=^3hpib<98@ejd5}#9dIDq#_C&7mDThSGZ(?Px`JQA|{LqZ;7FHI+=?@-keXu=L_@wSazhW886v_u4v&VG{B zorZRdSSaA~Jl$hX~} zWxl7l2QMP8O^=Z*--hpvG5g1_BzyT^LqYBRjbiM8*0jfv9S^*8U={t`8C==kSB@J? z<1&_pCDo#Iaf1|!Y=LSk=nXH^)!+YaeHeiu`#)(i{?FO5$O}QJg$*2?BOZYcr_)ke zdu*xE0`>5dXG1=a0h%?Vp;;V%mQeJIg3twzs^5#)u2KdJR!!iNzHc7?i!%e?7#;jJ zI_yau-4y-Hj<`Whstj+iWsCLjGanZfkQN3#7AvGl{*?5dm`c!_+S=M6tX=Txyt2<{ zoo@@!y%&{$;9j1Y?)0|~qpM!cLoZWuEdbS8TEY!-amM#cqJ64d+60SGc#%*tjx@Tp z6%;9<%2#Lu{M+)u(RZ>|SCf-(CFttueJJk!YW3}h^KU@INV|Evyr}LGjzG6)b43X3>8b`{e#2Ynp*6-kd zk{7HFp|u`_GL`cnu5+&heU%#Uf@qU_uMs(whDK!C-JDUXm4G24DjEe~2i+{}zI*tR z-e0}7W-l75s!FxR~rt7-QNr9&?`i^f|sx0jIrSigh#5V8?oRU8j%n_;tX;Txi4Zsmjq}}#wH-k)7FUY8j`p# zGBvH@r1(n$f(-XcP)yV)+x(z%RWTGo8XA$kab2=7-Xu+C1~qL(-f;O!#=!tt9LpU$ z&KHvLvgG365NU#u$zNQeXt{0)wv9~z8E47eyLY#fu}tt+gIagC1;9;XrUqr``CYOk zb7e(lt>`FH(zr=4X0vDx-V^w@wX>;dTwymt1xw@!i{X{P z%VW{Di-V*hAOSf4M#4oNWm8-Mim!PVSockh;I_CpW1h*%h7f&`H>go1;~?KzzXM~C zY#)#{IU2#XT%?n&4NcjKIK-iNptJVzZ|r~bl`hE1D&3sB;~jPqjVyHqwt|b55Q0H$ z7e}G0lmv3)`3W+0uiD$4oq^)tYa}eM5|D4*oTDj73qEfIb;LD+D2zzrzmSh*x%S;r>;c5ZJ()g@WL$O^ZKClt&fT5@wHnT|EZ$c$PKqU)aXAd=*z zB`A}{+6q95+OWVJV< zS`Mq=5lCm&&Zv)aB1-`yVYYe-9K=kE(@27QS0mlx;v6kAs9}rWxd)zER|Bn?nb|$r zx@+G1i!hef&PZeqgrrWOW}?XS0RuXZ-?w~wa~GTN0Wjz+^KIl9d6bML0>2gAj}Jk0 z_ZxBcZ|DHKrHJ}wue}AG28s@?2QhNagD9#hEkW4}HhBOzZ>5dMH#anr6;cQrj%92I z4HVpBn=E>w77?vY{c1>pxF!IGxtxI1@Y_bnXl=l#!Ow3MPpG(o4D#DXzJLK%;RE1W zVGcHZMgehY3Cy`T?T3@!HW!%&tOMtRuSgO(X8mfYDYztnI&^i>&RThT7Gsek^Jr5* zMR;3iU|{gNk<+n}Ee6Bjk^ph>_$79aPMAnIMvF0T?7DyKjs;YVESX62z@liGoX!rU znOzbHV`+(ZGa0gvTR{ZNx4^HPuei2(|5m(>>zh0F+lmSy#G_xGmnT}}4bEDLH_W!R z73S0#6NC)8^&kwb^B@d=X$c{rVL$N)Qf2V#cX`;@vJzn1ELQx2yugtJtn$PJVt*RN z&GQ5>7+}!87Ec)31$KXHqkf`P5q%*P2e3qgps3YOwDEK33oQz+;$I|D8g$muGDDMA z_yE?4lQ2(3P5t4m{hj?tczIIgQ=d>sn(fP}5JwR8u7t`gs|SZ?XbHF}JTYm_%awqr zt8j5TdQu@y@}n{YDe(GT!Hq9On`dxNWqPfXahB%&qnQ6< zumXqt7r2&i4Lv)u-yn8}TC3r2D~uKZ*>JEMF?y-#v8EGXI4c zNExpipRy#!#6JWvT7P?kZ>zm{y3j1Kp5_t zgBP5kZDv_^{F~*2F{L4L=8Grfb3uQkwGs01h4d(`tFX*hJ5uiZCC)O(;Ba5+SoZ_7 z0`nYt&@*)<%sg6rn!!vjvJf{LL!H{K1hG3WJe#3}fxB*P#Pi&f9elEB=oGS+cr{;g z@>wU1GcgU#0cNo#2h!kZMbK=S+$EtgWhsPzIvV2x zxl$*1(5P@{vYRFPd}a!lh2XB!oG1gGL(HLl&zJOiHQm%LPCJL3|6+OoiD zj~O*zPuiSE41HCrwme|>^73kQqFv5$0Pj*xBNSJLpURzPTdsIp?z;za0sF3Jz<_>* z(^#fI4InXP7Jcz}>Lov(TZ|+suu6f)|8~^Al!;!AL=NrGO!yE~=#m(W?!p!l0@2V} z0!FwWx5)wVMm6OV54_y(CJ5Oyy1eqXl5on&odj?d!EGum-{lcQ8nNjG;Mdk6KD^Jc zu?FA3pMASNCvNObjobx=<09)NhC$<|=dq#6T;T0ZYI zM=APTBr`XFv%Z(~gl?j0aG@Uun@w3bGzmsuzSQa=Aq`@9fHPBtU3XnL7|PSCLE9!w zZL z8$W;UTq^4EKm5w5#2DG_?t}BXEL^&7A0#0km2Joa1omGBdwR_^$Phf)-HX*)abG-w ziA4ozd=kag&W0gQ6;t{LTRrG2)0%(;4k0eGqAb55}k@?NCS~3hFa`fPwd_3`M z(ZN1)OKw9Z(1yjamB5nTzl)8cS-ow=YtXEp2p`kbtcKv&+?CbV*nfS4^f-AxANb`E zO>%BcDIo+a!bSgmN708Oj#4QfD1sJ_h%;=2gb5wGej>rpTerF3;S{WF&YUs2#; z1>loZjff8p=7{b1)VGe7>-`xccpp2%3`77^=fS4+#=+Cn^zG9B$oJ3{|q96~;<`2}XM}uUY=Co+h$-#%-gQoe2qNAmurY!YDd&RyW@o2^&d6%SI2^d(~RejA+8n zmSbIG_0Q7b#?pD~sk0U+J75zKGiT!N;}AuzAA_Kwm4U-({CfgU0=RC!bzYk6TAYv2 zLYP!_w`pnX<#c?)#*jOhiSq|@nr|~-*dlgQKz~)zN|4zedL^C;YY0rcxS;;SKJ^XJ z|DGN*N_+&_r|&-8E*cn7X4aihFb4hm;sHb?lbMZKxBgCq)io^Db)z=&j412mq)Q7W0c?QM1?f#xdhb0EkkARehe&UMK!899gtKt( z@7(iz+w=E6_nznc12ik^U2Cp6#~gF4`Q8RYC9H#UqismEA1Ucsw|^UyywVpy`~f*Xh|61*ao_6ydA9%O7_P9s0Uk2f1Vu$& z41N2XeA7A%fDjNHmStCd`u&^-UjfKhdr+Y|dPsryCZH!&KW?o%|J6YY;>9bF>46(f zoVl`p$H2>B+y|kj#eQ@7!5(-ikMAm&{j2@sHI$WVq}+$G!4aU+h9~ zw_K0kp8ym4kJ=@U>kKQafc8@3K3L8%ro+Ou8{sV`g$kBF#1>wHN`A^?U=DXK3kh{5 z`(G|$2Yze`m@mfDYjOR?)uyEUu7*aH=tdtfBFD8eXU_b%1d3&1K_{Cy&YgR<(sFhC zpfUkHdUBd#ndpPQ#8w@9_LK*B@R~4i*qLAfD4maN@bp7I=){^g`{2=KA6ym}1a#o^ z89_GqC;xSDulb)LzE&f_l~AYcyRd^p0ybeKac^?4^euENM(*C7%U=MuE2Xdvq;>5n zDVKj!S@E36Gl#&}L%<5bEm7O9xj>L}-dF7(S=2jWwrbn2VaE=If(MdPKwJUz>T9=y zf=#FQ+LAu#E6|<n{vMN@rFN>el1K z!rn##w_uI;P0#On*F)$J+YQWYHmaU16 z<>cW*2Q>)t1l?J%Z8Z_#XE#!y$>7NlBatUVNB>z=upezaC;uZv`@+9}Z0GAUoyjJtpmhij^N1^xzrW5$1rRKJDcleQnPBI%;UY zH-nGp&G|@J*4=w@5!*1A%U9p4_CfEr)E$l^92Q6 zx-gs{%bw+JSNNJ59p7BvB7BUi*|^KhRMe)|6T!od6EUL%$j3)h5iX;@3_hNp{`PPb zc)owzFgXFrDQcs*PQpd|AU($q`_|VNi=Y9GFYorFMz^;H#7b0kRtYldxjN z`wv3nCj^AVE$nuCwW#0BuMl)mQ3%XJ{Fx^#8Hn(jv zV#?_Bq&_-jPd}inFUOwLQf71Ui=hP@d@=T;5z6Wkqi5)oXn*-fgBkjsgevN@jD!kr zr`<^_qbSvV7HOs_4>?%HZlv(>ew|W4Wd3fwk1@QMsN$M*-QZ3P zXU_#e;(Uy`0UL#zDP+VMFZ8|&rH2+AJr~IJ~BV|Hy`#?uWpQ)4nV$gx*bcnsY_2Z>HoVcxSz7xsTRQwoNqWoQol7S#y zI-=jP5rzMK^xyyf22iDNmXd=na9%y)9%mP zdpZKzr!59v;oCxuQvKfh`Iawto9I2%c+66c7!7m~zntXY(>D@j43?*^aP);p1?c;? z`)jE@L~a&SHd%fGv@3Kb$&n7pDSZF5+%cFWR|?O_twSQr+J6{x&VJaP!wR5vku}nG zI`nw3&9_LPqZm;39E9orrC2=1{cA6z){B6f?^!H-s-ZTD7<$95@4c(v7NXE2Fs#TS zT={%on%2{j;ZwJkc9OGX=UzWUkkRdj5UOaOqxm>>-sdjcxpP@L52Y;KzTOVD#D_4C zK!7UE`!dO(iQ2!v7{CddOVEYX>VEP+TD$*@FA*k*9?0uW+N6W~`W>Tl#BuJcP!zu! zCkDI%x7cQzM1BJaeh|-%``I(v6^_<`rb4xIE@dAcI+i}mn1>$2sXO#%`Zw2KvLW@m zk#@std^+hbDDC;gS3%_6Xq4}IfB!N>*hY_5| zs9&}}3pMnsw+8v>su@Db9_=M{z0sYcaP8BPR-TJ#_q@;;Hd(Jsn9S|57FH3PDRX&} zSuEqU@*!Y(Pi_fFlQzC9Hc3X8QS~FQAaB>=ba&q=cvogqV=R zYoj-k2DrWL_Jf=wF3c@F7u#(AS-K8ykMCP(9;iEQHgfMH+|S;4XMJF%JNn!M8=5kR z#>x(~5uc~|pfp}I3tJScz_rpXyO(B|*Abvl?Qm=Fh{#g@=?fjkU&$2V123d4Zc;sv zFA(>Z{OyzssZ9x;6FUecy?00pwnDj}w_-_gPrHIOeHuF!$}X3!FKnB(UMck8Gu;>= z@Bd8R)bR(?Nf=eQD^UE3*P`*?9t`YQMENOj_1SLZ+&g<~@|uk~zc%BbS*6pQp3D5( zghFsgrJZJ5KZy+vnHx;0k8eX^hBMKN;9Z?P5^PZXn&=EQ0j1u_-9uudG=$10$%!hF zn%Te#e|DZh+$_JYa;Z64NyCd6`^f1IXw1C7?u6gja}tuSUHwI7WUmTnnR7zZIw{uf ztk9zAQEbJ-I{P(naP8MHr=kXD>^Tr}zBPr2(ZPHY({V4uUsZqjGUla4NJ8ajgL5JdAFW&>8r&vAKFRx*WQ-pS&?({2vRwW z%x8s?(O0hA!DwBl8W2K-ILTY*S5zz!oRI=o`E~9;|Ah^OOd~E=53O4j>pz+f4)n>Q zJfbnb=z`Xz`a5~8d(;N{EAe>(RVlh6U$Ml3(EqC?LZ^L2zv8+5lWf1u6(N*ViHS-# zBU}JuBrBU`Ip-G0R88qnSyK$<$3(`+PO8AA=k(*#_eqy4 z;G##y7K0iXPo;Z(x@8ZJxO^B!NjnP)YzvsOCzV(4=Q)fbGv}jK_o zG-6<-bn~mzxb4|bwGYrE7A`gE>p#w3_*ts6mwcTfuK4&IQSH65%@(7Ow&B_9@q|)) zO9O+&EyuRziS$cM)_eV*;=yPnt>`s6majab^8 zBRO2Uo{xMGf?eAhitgCGBkM6RPz+ooUlY1@6IAUnoNRi9HJ-g{Ez}hQE!*_`>e(Sp zjOuspNjuJX0oPu85Ww=Dg-ZJ1w?ms4_fr#t$Exl$N==G=s8w|H5U}6>&N+w{P4dv=qdI5*qXig!Qib0x z!`)0b93V~&HY*CKq{(S7@|`6u&DUK?F&)0ZO`fCQ z_Mo%WTQ2ot!Zlork5jCJIsMho?01I~=}5`nG7n7*Uz2}HSzh(9v9N`(RD17ZplMp( z3ntM$v;HqrKPYhsT)FDT$#IRND>74;-0v2r?NdtaQ}chnoxHuj|5A=i9KRe&9kkq4 zt(%MJlv}^(JuQ?R)^(?sQDCjs?ELSby%e&yC4b}RN0u{Zy1w5jB}YKmN2!$9+6F1X z#s1BBKZcnWz7Y8*HNTd7_qX$U*BV|Zm9dhS(qAY*ZC|;p4}Dq&e;6&8#|eYldRM^F z$q7=O8j5Fm-+8VT2~MI$@)v(|mt+-LsF*)g<$#<%=fZQj&)ui1nW2>JlG_!8SI&aVAOd>t9#I z#_~3ogvUJgQXapdWkqStymQMA-QCm-*tZ*%;-k_Sx=&l zk+nH%RP_W^A9Js9RFfS~{Z8l)gtt1bwEXJ~ha6ydIcmcF=xf9Xt#`2Oo$O6T5(w$h z#TSH)wDAll=-c)ldyksWDeSHAUO9g9bW-tBpPWZ>O}vhtN65Y z+BGi2($(i@z5#qtGqAl9-dM@mn$Px?8+X-7_NMKM^~`+zySbTTLJHn$XK#%o^3Vc5 z%RrieQjIB|{OQ`HXB#>|Av9h603aJct5-rEOgCqGh2yq+g?%p@+@CJFal&c$YADW2 zsoXX-3!Ivlyco@0;F)~dLn4Zs5^Dm*t7be_T^1;?%NvG=CQ5m=e>eVmnOnZtds>bq~)+;na^kc{a^m3jdZNfIVt z;fIpTbMBdX5T~u6BY``g;YH0yq>E}dcbs)XLnCNpJ&NR-ykE?P(SBPIf)M9kYkW3b zVj`V9Q3)5@7)*}q?gmaRth}#tk_sH!*GDu&MLy>#BN#qHX^b6l32K^hr50&pTYXx# zzsvZ=UcK?x7bE%Ab_QqFFzCi<7 zRuEo)j-YZ9=`QyfN<&b3;{7?&F#^?GVz?6K^3705v*0tW|v$1Le zs+{n_LNPRl&}YiXiT*suD?F!EEo32$q%t#IbbSvs>*jDlUMJ<&a>A|Mzz_I+1X^Le&;L%ysQ&OBSB)3AcfF0W)z-* zN^glc6CrLIYd`oqi#S6yRuYC|LAPAm*%dgt>5AP7yet%D6--C65)8VXXZ{2PZ+!%(m_2Rq$i4d7SaH38GGp z(BWko{Aw;xx*+$87(id-O4%-848l5e^D}mhUuh(@IU7p4A7NC7BwqtQ8{QIXPh>RrRm>M*7k|cVv z!updiE~&K%rZ(S`H_&`Y{>`DpjPQ%bW5yQ5)OlM0Iraj({xisH%vN^^0fSAxD33Fu z1#Z`>@EC@?+}yn9NI`akL0y^jre!f_vK-Nz4o#&#(H#-@%fG;+y41ck*`^bU`MyUi z@6}-M^;?X_EVa0q&bTo&JgsC!_DBrBmVkTP@KGdxcJwb0;5<bq8N7Gkv7aOkT!eY8oO+hSw6azdT|4yQ`hwa`fJOLLmpye2@D@xEP9Q$=<%PjuCrm z2^yJe+;n%>9IjD#%cCYh_a&)RvTi6+18FTql94!7zWZHm3T>iTE~w5Jw3kfbjMf@u zlj*vfaMGM=aO1^eXX`T4#(Z%Gysp^iMeh)^9wwWnilQfCxLBW-#t*0;W zf}N%3#b`IJ5)?Ens>G5{Ctq}tiPzVvB}*K>euxg;iDP;-5r*(k2>238mo9v5JDtNF zT=!YL!^z}5@%XP9tD0BroG3%ShQ(aoE%oW8$Y|3)odGz+q#>($LDWM1y=M2O<*)>4 zSMgDSb7Gm-u~<5lP>;4eBi@lA{|H_VzQ z<9e`$2%E)kD>eFsgsl`Y#=WjojfJez`9g#YUwfEz_FC~gV^FRBJAxiAYm>#`;qyDw zcSahsu0>~Z=bjz{OP@7OH)G@7CKHwYWRNZ4b=QrJ+~9~n0iR?V&P9PJFm+zschK{^ zQgm}=H&*rQqSIfDk;P*&o-b@U9~4CYbj2Qe0z#Nf3mH^v|@o9!mk(%ttp zZdV#uVt3i;d(q4TQpyn{k)qRdCAYui=Gr&Kr9iIv#Zq)~3MM+@ChmBVkGHnngw6E+ zE^mBL{tGQMTk%px;8SqzTt*AtwxxTR`>cMydGoCStel!Y5?=QrRaeeOE#Bb_`*{uY z_!o0pZ_T^G*v1>&qtU)Itt~@j?%ss~uK{e_zV~=ze?5Z{9%9deO{C%k=3jWQFa{+e z9k=r~F$g9->5cMrmzTTF)DoHJgIBzZgz(Um$<)kmku-EtnD{ZpbI~k?)$|CHQ>3yr3N|Ekw_3xQTy;-UhZxh>c&dyh%FjQQSOI z9J-zkn=I7TOcKld*7Epc-ups<W zrukCq3->)t3As!+s(lPUX7&E)MyS%ESB$D;+j14Ryuq@1G{iz&j_-OBehyC;Id_Y! z?dqJkQ0G)P%}OvQbFC@;_S(#ss^x`te{$?=%SX<)m{FyMMSN==%6E1rU+q-9qC+Fy z8XhiEH+Ejo$q%Hf$PI-197m|cm|A@-u^MaNl(53Dr(GIn`y_IY-Z zxtL2?J>>MbudF~+he>vp+w!7foCLp8zORNTH2oQaE%Uvpa9+y?-oa9IXb$mOI>^@z zE6u0ZBYbyomsZG&eQ7)rF(Opnfm+*^P?e2tZ7;*^3oCcrI1+t}Y^6Qr^(R~ZXsUAf zUox0@&-nE-P(sBVvORQ36Rq zqB^;QL-j_u-84C=>xvV7An)!>w0 zr|$ho#2XH~48!s(40U%swmMMT$FF|=?G~m@@v5=oXg`h-N&D6qNwvjjM`wu|i?@61 z{W(u#e3av9pbBL+ht=XmEa&vXM5@rn`Pz=w56Yb_+rp#x&1oXgNa?;;klH}!lX~zr zPDq(#&P`$Er{J73IBba-vtY$+WMGe#9q>C9tG{Bg#~0SH=jUGm+Z^&49AeiT(u7CQ zRG~Xj77kH9@Uo`G$Qo7BSIaYOnnUr`oW|Nd>wUHnDd&t%Ky~E3ewS)vjy}h~2nkFp zJ~uq{4LaFt-rlZmC@B2H!q!A(E8)9y&e`H~4gElPGLFagvayTiPt8(HXQs zdrr2PRifZIemBn2@++?<)6`+PrcTsOV33?jnmF1>H&_|;o^fYMwcab5E@WcrIzMGJ z(l1)Br2k|;`x0f{z)#BUVR(s)I>V!$w}phDOAo|Vkyk?Bb~)qUQ6$qIog-#lJo$v% zL|1Zf)k!|F6TMA$8aWd!&;Xpz?=tmeon~FY-41bSD68B`SjQBWud~;0sbv&arz^Mh#9S;43~c~L=#MeJyGgGPd(N(+ zm8l$=oLxZOu-i|y;T1H7n*aFZ9NjxSmsaeS8BGHvY-djl7&-^m7GI8t%D4Qqyv)8G zJ=D$Z;tcJpNSsLPeKC{$SUyYO$ANon=^u)Hx#uAf6Q!JQP<&|^eBOtZwbgi@TRFV4 zaa=Fq(agyL6!HpE?K+Xve@`z}$t_O0~afiToG{?&e1AUenQOQswo#wd;Ov z;@)|4L;J|`$yFFikGh;|awXKhEm=J4B6}bWG*lWAI;+|yE^4t&kk4T@;v*&~v+HdI zn6GPcoBSetFJMb1Yn>=L{1~bcIq*ts?sH`P_APK=y$m1U>fe{KIt_t3JE~X_eW&|GQ+ye*l)`2` zr|GirIgPzjylZ7J*lz6>h=Ty++6t=+L7mkRo9DnedEw+!dhaw!Mi$}f9P*K+j@Ef_6Lz0+BD+!ke~ zOh>Y2diM@jlTP#LCe5y#3Jv+lDK1fPCy)eX@>4Y4Mrn=lg*ck-WZ4Pka49QnvRaSb z)-GP#y5u=*&|MU{ooj|im!}b`)&$KL#ch>#YmXqdG9dY~>(iEY-W)draQCy1kT}GA z^V5k@3M=Wa#AW9N;CJ7Q+;95H5X!wl&m5w3-sC#_xWy z*_xO*s#;FU;jE)RAWpYGJd9?oUgOr*FBfz>%|KmnTg(F>bnsne)h&5c>1WAheVgrb z9sP};fPZyHtGDH@sSK}qS*0)ZrkSZ!_|k5gE2Z}iH){_<*^{xl;S$9Y>fqSiWg<-+ z;hFTp6FB^4+U>U$K3p|Uiej3BQWnPsR`b-9!}HkU@01fa6(ygM>UxHou>PcvPQ!!-`~v$c(s^cBvwO<@W$@Xlp9fEhad=<)Jw(^3)$ zM${S>+n#T6Qj#^W6sv}!zF%7HPOPk~q`6`^_EnapZ$;akk{a@=NnC5$;#57SiQZ?3 zKkd@^Hz+%@ZpHN&c-4u<&?t7paROs6m=Zn8Y_h$#ZvBcX>*uPxXWJAQ5(zRB0Auyrp^Qe1GP=L_sa*1jPPScHY&`ktR$e?gg?4aE~9x6vHz z=Q$lGWo{^+YF}=9lC<75Lry{h%8EGkCG(BgHCUd!C-=rL&f>vkW*M;x&xSM@ z%#T%=TiJRfg*Mq3_!HEc*!73i$am3rX%Vp(ye;b>*v)tKn77Mbe_l7czNPmbn3cHo zXzb^Euxt0g2PU4*=uWG@LpQ@7ab6St>1noU)?qj{R?Vp*@h;%q|(tzvf6lO#Zhp-^!jTZMDZ@$hww*?IQ%Yso)x>`$Flwh0+ z>Hc25m7l8&aklk7t&QRQVC|v6%gRsN(jregY*FUCKLEo8Ic5CaW7^jvLhW2#9cvI5 z1_5f2Jdodgxd?KT+@O+x5Y3ARMv2;UGJF@sZo7O6!`WjM&~lf0m3v#qx8+oD6m$72 zjax?0=oYiP!E%X8JW(mOwVv|aJ^Iu<_kqjPKPpv|q_?4}6Z2*cgdc;SGj!F;N;o*j z3_bQ1MlT{WF-?V!VJ=Q9@?$t%PpQP(HXoJxtJ^zj>4|v&3~Ey>4HAv8X77do1!b*4 zqtD(|A3?`yeA;a_w0+olEtYkiPdl$@GxXHs!^)bkL2AXXWp@xWj~JlCDw(Md5}ukp zoKxq&t{kpzDF!w~)Ra~FhyQ*@G#8&MKNGT}%Nw^&kH_1ZI|Qw_$F0<-{DlmZ7m~I9 zX4>%(Qhn3}YMD~ymeE1b(S|v@?%Z$^MD>E9yWY)&a}AFyKD~w5{rc)>zC8HG$Tzju zLlP5n)h!#6u01R%MEP7`yjgZbu0^A+tAa9qI$qZjlcXAsp_CA`r!Aw_y z&O_|{R@#Z2_%YPT#-wJM$_TYQR_e4r?3XKJ(VP&epp;VdEy6k0y0nSLm$jN(_&GPf zHq^)rqrPs}JXC#(xoMDhXlNe}!N$;}{7xlT%A0?x9s0!d!eEQ#w!_D7?;^SQA}mAX zp+|BIcQ#s zJbQBnE3x~re< z)lnNSUlQea=heRH#$PyXVZvoU!2)S>9v4|ec<)8%sxY>iN?!rs45b)yW753B z#Lv^5_^<=@$gcg%ta=D1D(BftydZ{m@NIgK)s#_lguQTjZSmVc=YTZ&wiSH=pOx2s zQ^UdBrP)jq@HKR&{Z(E0d|qtdkMj1gPOth^MxNwx6Y*~XHVayp1(w?kO{ZE9aR(r3 zECXN@)AOsEf!65qrTD;zt^;oLzde6dP~x<;$42a0T{f+HEY9_V2_SZr+Vv#i*rde4 zq{m0-eC?t+?BsoX*2xsck$9V8^6uVdx0sDoQA4826oM^DPaeCb2v%{AHHt+dKfCIc^Z&kO2IVK(pCc6^HS_Gv0 zsIc+40!SW}`ApT!V2d;0)M{QWU5W6kSs#f2gx9;YV=!jXJVJ#Pzg;TARLSJ+Mn$%f)|iqJ8-<;^^n-cWY-l{ z&yXeaUqlr{MHY5*W++OgN^T)0lI2aGMeNcjr3S?n$(0)!do8u+nbemi6~4~6pkVNU z(C1R_>l80u zia*f5eSNzKpoxEF@ox)#9vr%N{mjENAYCQ4FrkwK?at70%OCvqU#565NbXL6FZ%K- zGjJ^+hr9_}>p#uRqXOc=znH*F6w3=Y-yeQ+^z4D>MVR7SdX70ho;-i>+ka79bwlUB z>%KhopZU^3nM{_$O#k1R?4{6Q%73%~e^Z$MKbiRd>66XX$*Cj6Ml+7 zFD?(bT}U_g)!)>m_STmZ^|bORMu1m7$tZ9Um#`G=WCXV_(zXT>?HjT zOfMp}kf5jV)vVi)J8*WfD{BW37|Bn@%knw&)9xpUauc@^IxAv+ir49mp0;Dv$r|$S zwq5gT1TjVTWoDk*pKQB&GjFIyQ+NTF47h|R`C3XsZp@-9>E8DYU<3!E2?Te=3wdN@|rxnSqXy7ubki5vb@m^{dL!?{vb6D?+_#V3~Y zN=d)@ETR}G&_6v@eden2)7hX*!mfO@pgjm$)NA@hKKBsur|{U9fZ=c0@gV@W^!-UL zBWAIT&+J!lL~&+qOMUC@2CgV69i9k!c^&iiVGrOa@nmN#XZ*7aquPfcaPvs6Vx`bA*4EUT|0V;}% z63|VUwnK-HYgyTNY8yk0tixci<%KJJbbw>A@CZ45YGR9083ZU%?;heNi2EQtZ6}mY zEhg_8KL&TQe(zCiVH3{iNuNM9m$FP^$BJu%haOi{{q%_lGq1go~92xNg z6itrl2*%@_vY2}+0i2%9k5{=~IBn#* z0@73evS3Jw-Pl5gGTPwcfNU+Y!Z>$Q@DLSs`ZO$0CFqxaQx_Drue`Q~N+8s&p>c%i?Dz%_my2Pun+j z;SaKx&M-m-`pS1 zzO5Z+3ShDiPD+n)y3Gsm%JfbhLhmJEg)fof=1>5~uBHz%6(^!`Mr$DD%7l)%GZDXA zfTss_B$T1{UBx7sGG z=X5iM-yKW_splNz{=hHraIv^R_O|BAHq<;4v6oiN?h&22s_6E8@z*~*G3liFsq!B= z`qMUoPXOr@jmumEE+OJU=?fpva4y%ET8WZg-Sq&2$7!8%^HK@lfJU4Q6{3k9odSRJ z(Da7ADu8;(KvHkHJrAfvw`5Y2DOWt4EX`{Yn6Y(L@DT3`?oZb^9*K~E+sos;X9{gKlkO@GwF`! z{JFj*teHDtdbh{lb|!_piusSO;Av`xQ#a`v{-^+OV%2#Hsh<3R76eeARyC>V#iC0U z@ZnFdBhQp9znX#>6-_tWIS?)@hXd@s=&~TtZ*|KBeYK5*Bu&#NhscRpv~@=LsKoNS z7811{v>Ug=J_7=FbaQoBWgEO$cQDnu7Ok};kyL#pAm4JvBc4Qd!RuwXTWyJ3fjn?* zOmk+9pJ=x(33$L2BstmvlNm82e!m?7+`wR;xnDHk6vRa4;M;Qf6YlVwV9Q0o>H62rylRp+{l1e z^)hq^UKMb*HUk-knB@I@0$9K#Jq1~sYsyc7`A-tOU)pvkv}%Tb2kFSF(SXo0hBPC! zp#|ob=^5O4_DQYJD3MF=Ifzqz3 ztl;bLjWo;NIg7J96XbPp3~wY1_}Z~|EkQ;CTk1ctd*_(P&uY1k0?s$U#frY=z5|PS zQNBL^9|+%@u(qE0-a=kTzfE6RaZA+4n665c8Hjg zQM0(pP0D?fn~2@fEYIMex%xL<&HzOL+lIfzc6LgVCJ#q+TGSTyR{sPv$aOk%)7)(- zhPyK5xr#m`h`Q>qIp7RAhD80e7~8jeqE`lmmNkM0g*AS3j015R9!_ z3yQPB;svt;8k)C$2vlVnUW^P6vzkEDwN~T>Z%0CsHJXz4c-rw@KJJ#FB>EW zGyx=k?dqywiIyBGK&hT;(<-so)=LWnTAp$Puc&ui%BcY}apGD1L=U<25G&2hG5;BU zXY;M&dR_4n*H;qXF^sOyRprP>C@q5R*;Y#y+*%rcz3J-gI+?GCAv$0dZ(O3ZpYW6H z7w~9-xxn@>kksMI;UFZi_6doYx(6yBQcH(1*&#hSQlnS#H z0bx}~IL~f{FT2bKDY{4w1D7D0Y*MqN z8-k$Gdv~&Qv1P}5bgbZvW84NuQQ#`v0cAU$If&xTC*@GUhHM|Ky)L!U#ORH~%H>IN# zD99vs&UTeKie`#g;^$7Qb}!iKD&g!m%|mrHwrqHnpGK)#>ArJHB@#`C>86L@IwNW& zW||A?1k<>bc}d%vOjX|Ybwe$S+1l6dvjd*O0eVk`3DZ$;fr+B3ZtUNP*~jdKf%I{h zoj#mvIp`94%c#8lrnwfM$xR(t=$N$ZSILlRtL@O2Em!B-5H_pBfIdruIPwHro7=vg4N6?`7yhh}-9m>_D9u1qqd<9vxhMkU z;*wm-qLoThY};}E(Tq=r!f0x(&KoI|KgX8@7yvy-<~MmE1Yl>^2p*IcCleKrw{UpFTMv-jhuN8P8X!zQ(MWBbk;fht z@o=$;IQ#6$AF*46JTntZsfa8u*=Q*_xV(sEDcEa@AX3|;>ZrERsXm)vXgJ7L-RhqR zf}LH@f=iJaqS(HHr@*3SH57nvextE5wY)GU_VfEo)0H{FT*`pTnTE!RlLl%}@gi~J z0wqeat@iEofE6UF9YcFhP228>k=LyYqz$e?H9$}ksHWY=$fb%Y78Dj{ z8s@#Wy&@H0)j4FfP{FsKSMgvu^VR0I>WJKtj}dhFUa zP={=rV`gLqAN~tfJsaFs{5eia2s5}oQ(l-dT=d;M&=Z}Is1!d>tl&}x751;5$Y1M5 z(u^p#IR~<*wzr36YC_dEe3mM18iKR1g|> z4U%XB@vFal$)=*R|Gd_X$Y`2)qRv%(q!Ip-{!w>mND$dB+QcTg+Pdtdm@=ZjG&`|gqKlIQqrMsktl z#kSe>5TK^ouKu*ZQoVeSd#Xk}F_}V~BQ0dU z_j7#|hd(G{Mo+Zlt;X93Tgd$pu+=dDEn-{e@f$U@2~x0u?TG!3D1rPISG=`^WWx6W z>)m(0IRi;Gb+C!aS@n*WwN*0mxfxHjhdh=~%t*epzI?DaO2!{jA%|ULjA}VYJB-Vl zh}AROF*&NI%n1=<_L+6ROL$Vum*8C+LU@Ma3>_Gq{R4DTgP0d6au@l4B>bwTdH14^ z=3A6!`_*`H(mR8up7vLSnuWG{kTH>N&`Y7C`9Ap*gR0r?djUD2(&hjYkX2 z9+*`yQX*%?%5)gfDr+-aQnbRqbNo+gem>I5Yp!A&%| zFNEKN1QQffwem&_md>Dnnk5!;M*fheI~B-0$)2CVfBPGnKF8Y{Nhyb$4F;GKAE4zy zLTP*6s|IdQ9cy8i>tRj%V5`^D`W^eda6N8^*eM#oBM`(f6x4BKSBeeZ zDgTK*+O$lh;M;z<3~4vDE?ul`SM@fck6Uf?b78(usODqc&HHg-blpxj5j+PGlY z+gcCm$htr3Q%{9zm#vVqeT}}a->muK{$tt$Ar>(>r1uU4kVnsQ9`Mc}US#&BI$&-7 z^0N6s``rmh26Ur2uJJtOJ{j~k)HwyD$@8Ijhv|503zOxvP`YyTQ0!)-(dn^^-5%Xl=^Qd_KT}rCMp{G^SPf{HxsGM(X_oRWoLL<-1(DSmpg;ZJ79WNHv zt*>$SM$v^9Fj&-UO}Gf%9|>kk2nlm<{ar-@`E|a(9-Ah{}f%tBN zM3gmFG-GIpr#vjf8>Jl{RSdL8&Q2;d{p>xf=YSW~|G8!Zq)>fpR*G(wy=OgUJIg^9 z{WrPl#I*u|Hny%at^%R3)XdKhQAF0>JxPfdz);y3*U#&p^NLYJKIj8 zo+m2REup)TqKhhi;MmY=fm*ta)@B)x50Wl&!Ol9I3o9=cy;xnE-|<5KCTaBxGKbzL zn(FY0ldwwYOPd9RygWJ7)XAJDcuRN<(UVm-fgMMg%S%qe=%Cgb(%TR>Yj8AKy0a^`iRL6>6HRXJ1T_mvg{ zmkuNMkGb=fFkv{aJKJtP?k;Cw!tM2XtJDFHpATLGD%{o^-qk_1I z5ThF$B3tCt?94}kQn`7HsrytRmofksmSaKvUBO58_AFy9m2X)@{!92HIcJ*!`w>8p@;#vUm&OTl3OMU4g@n%JqDi{YAs*0vIVS4_#X)YYB z6QnG!Zm0FsFaXus4s}Cw8WJ^a-<}m3URZVM;~IDZbiPj_GR9jE=koKmHB~^c3K|0a z3NQjRzB@p{xsYwVTHK;YhhH--f>yORE>oK3L%+= z_)53Mj8HmIjlJjHrQ_>ZA71XJ4#sNz&qvW#Tt0#>RJR;=t+slG1mrU2n5d*$&Xtjm4Ec7+a)n2xif!!~+PoIL zsqM^~2KIbJ?LXmt>|3Fz9NP*ibRxuFAQ%vgNX*b*FnLQT^!%BsMt2m_yI1es=geL% z7iAJq78D>oPF@rVX}9LdANn!!Fr2G;GVwSR=#+3SBq3*Y<#6!#=QTLM%CIHU)7`-Y z4Oxxdo|5>@8j0S775xONQ~jGIN0}+qj&BR0a-O}{Pulbhd9-1fMS@t?eF5G%$l+tq zk_Lr26M@oOpF0^7k{Q5C5xPOuK1|*6T6J(jQmpV3uJjxxuN(lF+LQ+Tk|sDMzA#Ia z72UR}PgR>DwFrGXb}4F-Rc^H*gnutS6u4Wu?5)C>;@#2Wb~5j(LM5b|<)HrZFMgBp zd*Kt-2BYZU5t!s)yC)O}r*P*oM7M}F0r7>Ihz>jQd+)+f?ATB9B{PEsXjfUoH49-Z zc=&#hy=u_mT4exZBhi#cjL$Fxda9gY2{dvhlyaUie?k1!2H>tQ)K%P;ss#YlqDBHJ zdPbl>KQ5u^#f=Y4mCSPligF}QE!qG{n`vW&;XwZ8{TBK zA&lnAOU%;QFm8QAnM32T)uvoG$i`{+Ic!#fO=Qw(T^GhvBvF8rx`j=&-(CL8#$#!N z&A_0+makeW#mB9oR)J>Sys%&#zS?dN*i|^tEfV5_upRw9t!aH7sd0h2~Uv84lR@?Oh>0$GdBNeRN( z=q(FheLu0v1oFx5EGI-UD7;>{uKzTgD=~K&WQi|(!9AM!Ja~k#yFtm+aU+Iku1s(# zr%y_TXMCH5QlINTR0ua~u~07?OstBn?|!AZ1pc=DI*08{l`oB>v&Kc&xk$$&!GnKm zGk*->(0*wyV!Q0@Ba)p71$tI^ADn^6qz)&^jaNAw!GBjS935Kb(G_jD`_EJzoz zO$yP@tg33fwIdp@n0Q`_S`6dLd`!*R_4U##iFS*kU)q;bY>O8-zn^&62C+4Uka-V) zSL@Xp_@xKs^D5y#QFiL!?#0@wcsSW=YNTbSC=^Kk!E1%&iqis7)o(}edJ1os2kYuW z_-|Zb=@Y6O@xlFAZTi1WxeD$eRi8(mKK%hjJQ$&zq(a4d}f$>oFMWrPDb-+%)xFAXR}eFi0wcrkHN5)gL*l30LJ zjxff66)7cVe!h}8VU7>yGJiOCkV(mZk%pq&ypce99d)l0&kUH=`aE?M{8AaTqYF_f zCY@G5*y)wZxdXIg;uGQ1VjW3djnTVgIwL_2wcEc4?#mO8ylyiJMgxSu`19e z=I)2g*lS;f(zL}-)0JFj8L97#$b}GK2FikS;Vo$Y(0K?(Ecb`Gq;uK^kfN@r-*DXs(o7@^(CiVqonn z^&b&x*DEpxo!yJ?6vBtUeb+CxPDS>S)nOHX(tx?=!M!*Scr4FTT%Q-oQ1M9nFC3TG z+>egQy;&QAtnX=PX++n@bolnqzuRayyDVD&o<`Jz`EYJGn|%1gr-W5{RF4#%4;A=& z*IFZi1Mh}U#tZ{6Yc>=!)0f02vdHsR=To~Z55F*UUGLF z0d-brtRPKfraE~wkF}k#&(d`>J}&$7ga>z~zSBQQ{tg6V6^xS`RRc}py+jsHbu%Rs z@1?WaXf+V+~kwvkG>Mf0he`4vh*%~MORe|+iy0MPhblZW&_={25pmbGsT%zPh( zP-lshEhHhAR~sswg$44$!#|836ds745^lWx!~tGhq<8PMNa2jRxIl*~0ol(fGt{f7 zo)s1960a_Nn^Fl6H1??+Fxt8Kn>QQd*T3pxFjQrZt{)5pxg)ru?o~0`ekZAQx0ts) zAQ;1(UFqWOJ(#!Y!mM}j!>dZQ$T9VLM74zZW2dkab} z^y}W~-Ra)_kIBen(67@!jm(Td(@vHx_q~`6Jn16SYS6gxlTaQ&ndPCj^;Nz4ZMpo5 zzKb{C4xUV3|F;%kP-H`-aAA@6oz{AqR%C2R-1AA18-FulkY!+YPmxF>-|?1t$K4vK;5(D#^=@!8aI^3-*U zr2yH&$!84Gm9@{Bq|@>U69PGRO=k zB*M-&Mwd(+J(TD_v$>j(hS0Gw(hx1|^Lu}p_z5EAIZP*jh4 zO5J{EKN|V&PSc63aSi|=lJ7=xe1|q#_*mmp70&iX%&31D(Y8<~h z2wC&#Jniy}7pAwDDE=PT61;IOsm9j(N=VrB>JJTydu68AC8`IcgsrbJI^W3k3KkEsB0*?W(` z#d>;yKahEujm{`N?&UpKw`!mLY&XLj9Y}W{uM{=-x4iUERul2a0Q@ z7jv(=o->~ge2}E%uw0pFeUS&8;p~A8Rp&!lME}H}Y{;Ka>3oDJ;tCow$vSL1mKj;t zGpf%}Da2LW-SYU)s-eg<%@}|J+p7)B(*e7c+y$Ykd|OxhnZN{Cqi` zOBDDiFkreG*E$IyK71;?XfO%jWncIS(a*|>n0 z?&|fq%@JEfhk~ zKKbH(l}u=BysbvYrPG`2^NG_OrZoS!{g||hIiV4T+z;{wg}Bhfo$0w8Z%K=Pd~a$o zoLKKtdA^g;?Qi(S;T=@CP-^QMZ1MjpzmhgZNESrDk{x>Q3`A1+2G|eTv+Gf9o>Fy4+I zy&9AEx7I%w=$3Jc{6_gY6GnfMopd_y}1?H#V`6%XA>3l-nD zgX|ADKS{`FlY(O34%|!>z1{Fs(bPbDgO$V%TsOm5e`Bi2HbTUDQKi)AjZ>)9T@??^ zI&SRm7Ek>JF*NImhcAxxle{8lHVV{Q&2?4ohdl0*2d-efwfdFD@gL!y`$wQW9UuaJ zkMHlpk1poi(d%$&lWPKQYg=kzFEVR^u6#W@ul=n$x3JQmDP1Orpgq^;3{ZV~8`)1d zqe=PNck0zA5_!hmw@f z9Jed15{mp>U&1|=@YMd5HOPk;k+kL_y0k&c26j*5+Xe$yVjN5s2|AdY1J(^Y_C4GE zPfQ^93BKRkh*kFST8a?AN2DEPMVZBX}s& z`s@(J{XgoO|3hvUNquPLrZ456t#>R~@4NaEDY3lt|6l|QDIZTxZX+@NpFCK!+x{~* zB|m}4`3J$@|8oT57Qc6U5dCLr|1-7!vD*K)90CpD$bFW;5(DAw&f&qnEgd-If{ruh z?&KQIGw-_+5;RU(X_5c^t0Zbu5HtQj(^a20KmIg%Fof&4C&2JlCcpDxLDA+!oe$m~ z9sUSjt5NHHu=w5EyDQMgjF3IEo}^yEJFH3N@DhM8kIltdsYMfMgZ1_$FIIqFZYF^Y^r>wRMi=Ua5(@n|P(R zZc^a>!#AJ zT0r!-kUd^XNkI{V2Oj=-)|Tn<8O+Ok2n|18{=F2vC78a3@J?m?n;7o}Ars<|+exCO z1b_SOmGn5%V;g+gCQKrff$+L@5>`yjjjQ($l4eBU=Tf)cF2*xlom`^-^X5wmiu?HV z<=L|eOpgt5-GRLf8as$=Agi|WSiAoF&y#Urc#pT* zf98RSi3y+j-ntnfDOpDtB!gHAIdKo$7NgFErJ)~qVo6CgJl@$I+b#Tj6)>C&BKC50 zxOaW%;J+Fr5D^A>cz+5BF&D1Wp8dk?MDgJ?e)f3-qwJr4h&KtdEpl9rj6@5=$@}Cd zP3RhO+_h-1?||IdCT`Xezrt>L&*Klk`mV7gWp+QrpFq%Fp~W6 zCoHLN!H~(n_z@j@sJV70m(k0?8P9d7l+~ zV>>CD$~CNvH8nLUZk7A@&5mGF?g??d-HDq$Mgzf%iE|>NqBWk&8oUhwV(<;&;(i{B zQ)4$oxXqsJJ1WEC!6YO}*sWWL{&nUJ?Xkz1o*6ZebGN#9d3v^8mwQVIk8+ZP>q3_C z$nJ&J=^CrRUF7=-zD!b*2Ya}nerva{Z#gO2rLpOi-{^>+3k(d*n)q;^7Y%|6*^i@P zpjBEv;jbS1|9!J-*zp1Mv@F|kvk%o?tE!1}X>t#U$?CF=gdVVxn%{kiCVSS_)@Ouo z?O_nRq^M<|^LNYtJGO2o(UQH27I;ZwgEn2ylE*{Ic49}E`$>*{N|g}4MUNhYWE}Rv z|GJ1UE5dweb*m-)137^B3+q3g_M=ucv-`UMMrFzuLpIC_U9O)`Jk zLU6%bNY25t9&)>JpN=%+)=RYM;f+8O=(tZA2dwY8y=QNu4wRxDoMMEFh3-w9U;Ic7p#VDTXfhxrX5_{jj z4|xOu>7JO#+dX*5$2vu(Y^rWHHa2+E(FI3pIV_i?2MM2krMbgOI7=qFYW58I9sS#) zoZ=oFbPuUuX5pA(cnYB}-tf%zNB{i++s<7@ADKeU0d%Fy z8ryQ88~Y(}Cti*!d-@h$K_^`+X8+^wHBS(5?(tK;-AdA?f>-3SlarILaZcm++r8D^ zqV2fkQIhSusb8uZSbR!Z#@0RMFe#+;cjX_zM9(wO4{s;tIyysW2%TQpnQx%oO~j;! zFXU1RISi9Zc$L@<>Swdq_x-)yf3Z$wCJv4}DsI?=x#LD>=kWK~BQxk0I@w+HdG~X4 ziBqOOLw@k@s$asYU%S2iuAlD;>6^N`?(XgaO??>TJpC2|4Ek`~4_l;gtf{$~_skjn zT9$u59WlYRoT8T7MiNfs7b{?e89_TG_FJ{Dm=U7QZ@d2Jo1^)e9#<`pY9x!62*bg@Ri$~~Zr<0FZF zwddVI=J`LJj|8o|9uKTNST^ukpND5Yx4Hi0=zoDG=HjUrvM&~t_{)W7y3Qk-ihK7IDMRU;w-GbV7=3(l zzPt3mKcId71@x{oPuLd9ND?09`}FD2U_E9BQ&@;=kmSJCnJ_yWm3X(Je2 z39LEWB;`~Dd$6ALtq|!0f8;y9pg2yRBcYHZA)@v;@;I3cyMYUC>%s}89tFz!yxiIP zk3r|~ES{%8s?Z!ECS|#~C0KmlhY#DZbO-QUS=4MctUHZ5J6wm%&;L=I(}FVFy^n~5 z^6ZxSL{jq~XuR}z7gqDa?uXCA{R^zDDz6tGAuuG2d#EcM0&%weE?}}M$*CLY?4-60 z^T-t_e_8%@SBb!YknRJFCNgY$u%58=xh;O@@D%LCwB<0-3=Bl?Ky6Ac%eBcKTD!x! zav04!*{$_?@ythjC6C}bDxy7l@TOWANM6vjPZz@h_(KFkFS1}XBTtEpv|A|lNiyxj zb$5LO`LGaZ&t#-scie(o9&omvPL`VzqnZ1Nv6|f>Wx7vFe0<@Yr1T+7=LN!a8a^tH z#96J&Lm^KO+{I{?&Hpo<|C!GJA6w_ad2{sN_F5WDY6xSz7dtfCN`~#k`DgQ~$)PJ& zsImc?P^(ibobSIj$wkYPf`0Y)!A2~Qe3cqXA#CGa?+i>CXJ==@2cUQTuPh%Q=jj12 zpsiy?*~b_JaP_Ae2_OXnPqWvP1joWba4g=rr07Z(5*n&sWZIl{A-f~@vJ2iAi~gRT zxyBKbE(qDyEANsdYQlxa$HA<3x-wtJE}TG|%{2>%l0Wxlf_61g+~X7b+USR5Y58-q zZ~H5~*WJ+o9o_mA9vs;wsyw_IhgRB2-YcAtzP;^hX4_^YcyBu$59Jc!bRXZ|NWS7s1yZ<)7diAQ7)_wyEuVplO3)ppFh2@BRc=hu4I8m3W zON`4C&#locZwzxZ)wk?)QvO}l5N(A+k3nZwQzW-pGvm?0G~E)*cVR15=Uo@4^uE+c zAYN%7ad1Tk-^XWfO-@d_GO;yfYNW=Zi|i5S*$d_8WMm4*vH{@?SpD(~HLrjL>MGDg z5Mv=M4)NQr^RG$o7)|QrYzTWU*}o>&Na;|HI}L4v*XqJuxpB0%#T=zj_2GF{V^3M2 zQyu`9BE{2XvOL)#uwChEQU4`M4-yLnw)IWi^;o)wY!J>G_}z#ClnE_7dgV#HSG%?&SSw%9vXldFrky;+@*qn)2PKbCBw zfmDK>)n<|mSez4<`!M|Ug2@5DVdKfxB#FU76JRIDdZY(N^})}c9iLzvu6Cav$7Vp! z=;6^(LBLFOYrOKp{uS*_Y%ny-Cd0pW@7}$0C!&5UUbh@9TL0$Fl?vyOJhkPer7Nq` zqi8T%|1-o2Eq)tXTyRzoJ(A7v{x8iJ_Fuo%-P@e0-HSG%;Wkgg2+1l|dF$4#*!dDR zJ;j7uReQ!fr#;lAj-ULAzUdtk^V5ShVL?H$CyZq9;_LO*#ZA%2Rv1gY*D`I;ubHpX z7dO#<^pTGbFjrkEHlPTZ;_Wp;pUD0SK@JXcw7tiQ^*pzsQw*GD<*xDB+(1)y^iRiQ zT&Z#w_!<8t;KRSQ09yH-5}r%<>s0$3%L?gPe;*ANPks~J%Z?MWXg2yORS}JRMXtT} zKY#Dw&+Z}s@Qo8^jE7>EmX^?8yCp%ib8v95wbkzX`xqFm=f+yub;Y4r5T9sZF3!Qh zAtZzzz$HFwvkmynq#5Y2aYG47LA_}*2-%CZn*xpQjb}{XKYB2qAawnS^~2eQRK;j_ zclQn~37pmMQoFuMq;Kf*8jo(epr8P|AdcVU9efjm znCr_>bW8Q{JT}B%SLXbaHm=G5^OTm4LvYpW=t!^g9^H1bYOW8-oICd>f>U#CeI36E z*Q5~l{s4o7cO}wqpgxjq+vr7Z{J`|*i~#LkT0?uwn2FVq@oSR?p}a=`kv;X`lJlDw zJp|7(xlynF>?L%-w5fD1#|colM2yYu+uP0To=Ufq34w)F3M+PKAn!ccHESTfyor9} z1$q_EQ&UrIiD((Gb*cOYXM*468~*B}s3Yt&-nMo7?q@SCMKI{a$?ng?jD-N!U|_l# zLr;vB+6}fKoESHT*NZAga$N{x5O;l8{Op`LOg;EkT#kIw8S&oE6#0irr{&a4&}P2T z4bPXVnXZCw@`n%1g!>gRp79jIbFS?SnOv){uQ$b= z*v=|FI{vD@K5#mYO_}z)dd??&j>fa~UPq4}O-M+{d3p_bS--)rz%+a&zrk2K~M1h0bya`2feNy z9)lGw`G{*dXI))|<`oKwfiEme=vFZd^pI`ZjOyv_ZDLFHT%L=mobi|*5brKkS5Z+Z zYfm|MC|@m^_}-PAODAHfIa5gv1d`Lg zW4BwK-fQrXSvn9@Lz5vor3RFM!)d{PmRKhy+OrQ)Q5jiTHPL?XYdZ2Q3@w+hHJe`( zbF61TcTe4g$!;{IdH6eFTlVl~Z|-HmAuo^;@*S+0b{_cz6+!swj0}#0DUyD~_5C$B z_5`mhQpupNtzSOs78@mRjcQDHX7qPW_gBc8WiT)>IF7bdx&2Dh&-cMkBW7f2WxiH1 z2V2#VqjU7BJ$lfW_kBcYhYND{nXhn}Ms_NzourW>$6T>{V`f|7lbUHt7YB+}`!i?e zMifPwFlE0M7i(N+pI|5mYI%AUs8XF9eNb=U_#sGba>1LOJsl`)b$NLY-RVj6tiQ2h zZe_(?AsVre*Es>V)8af^yH`T3O4x|~9<{i8F^fEQ-@Q9`TxY)c>F3v=fqYGkM1pgv z#pl2$1ybhb9Q2Ct{n*P5&(>^-uc?RX zReIPuBBaOaf=$O4gh#ZgHBQ0&Pg*P8Y_3w#(<=`S zL;nr3XI2zASs9yEXA%9ax9!+bM7IG`uQF|pweBt)yHW1FzEZFqvplrQrc6_xcq+aC zsi@TxuHkF8Y|= zTE;Nq&Di!yOskqwm41!+$0HpddwY9R?CknpJkVnW7CGr-*z;6&Z^wy!+^z8b@Z-;> z@cxd93IK$}GVY{a_NE(dT!2!5`loFr&=xwXGA=Gosd8-XQ&ST+Cg#TLcQ-42HWS}? zyt=uq{s9~yD5f!w+pu0sY5LmNuQN3AJ3l8Z;N>KkXu&jb+diTCH{IPaHATM>vk<9_ z;s|RUMtl!Qz+OK5hv9du#?m_B@$=$~vO?Cud=Ix{!H=h1hmr8;6h1Q;g6o{ylXs;| zs-|i!iLxX{%#DBI$Cocm9%<x%VmNe2cF?fcsE(c};!)j-4Xd zJz$aW03)E5H6<8E-AzRBz?*R=7sfuP7uDk#pFe+IU;F#_m}Ys<8RyAMyD~Z_v~w=K zTIuQO0XDcln9@0co5lfLt!(4C{5&V;ETyM^0XvcLBzUqf*?_h_2PV5RS&8xPMcFWh zk3ahm5Cz=gvB>#anUtdQ5Bik`#(gFeTs*bTA6jG&@c-ap_vE*}l&r4g1GXdRue{A- zAw2uokJBSd2)Qaj_CE0dysof%CSo$6=pX07mW z@mbyRK0$Qk{i5-7%e2bGCxT-}lwC$sZl_;`F9;NRtZ%(X07bEgdR5(5BnL_fB2H-B zJpT64t)O*;3$b^~ymRyOxjIGPXTH{I=UvGZ<&FQD{1PSzr>yFhTcWGbIDPh1?IhY( zr&dbc!G61-`69w$Y36Ihmy%BXqDt=6VX8#eogA!1Xwp-m+y_o$)_AUTe zSaXHtWwWuevuAVPcA4x-9&l2j4(h+&5_oZPoH|qB?hBjV(<_R3mkOQ)ar5&R!gXL9 z!2*&-$T(vKYA<`3KTjH#myDwNo*A@t2YE29fI`uM)LTJ%3N-uny|UxTP%m>BaUV@^ zNmEU#cA3_1w%S%wa5z!KiOYmX?Xhpk$od)si)`rO@ylXP*FJLRS3Wrtley;aPoh6j zjIZZu?fEbmMQcQL#Uqcsud#y!JVmwUE#tQf1z^SLKmQ8NXH%e{$+m=QHX;gr<+TNTregD|~ z-l1cOvHEPIPHF{;oQ&e`Z8bhV%{SeqO0he#k?ElixmF2xCtqq`fzb|Da#GS~9Xin( zQ-d{0;THNAW)#|jX-32~nU?FRY^ zPX&D~nF7-rALw$su;8^Lz`fQB3aO(OFXC_Pkt`4m?fejwusP|oGR98F@h*~E)+b}! z*8cRTU%XM#WQX{M!_>G~zU=HQMc;5nU~{CQxi-r! zRGkv5lPBpq3j6wzJFpMAFiPFYui1zlYJVv@T`A3mrtESd=`(k!*nikcO8S^5i2RRo*8^0d+To6C;!UrZ?xfRQ6(+i}^KN)@S~axxD?oyiBk7ii|9PB0ytNeFxR0Erf|Uz58qAZ9vAt2k++&PV-(#k=r*v zg}__(v>SL4=|F~yjeqymRnt1h#%Mmh736~eeQ&Eop5NXI=q2p_o5L)5i)uvf#oXBM zQSQsbjZ|FhRvkIxGkQq3IjfO7S-K0nUu|qfC-;zO4HT6mP!CWL@?Lx49-Vs6^r9!4 z_i?NvNH?9!UroCyyDTN*ZI3%uU&X3B+i%IMV~&-79@_U zPo{_Ofp52{wTN;m&IEkDG1V8dp%#72pyfg4&Y6o3VlQxco|CE%rokF#dpFqZ&!)9* z$Z_QZY+GI&^zoUlT3ou^Y9LKM{Pw|6uU$=h<-t>M7X<-7L)cwid@=hDRpTOSuxR^A z+AsZMK?iv)?*>GF!-iCEhUskn{;b+|#K2$>?Z~opgx1wzfQ>XHs@_QckpoVla@JtRcrb&ujxSZq#k}d!iJ{AORRX9fjV>4(L>QXM z#ww*DMQMpSY4+u%A25R0u+$~uty=|Fe|`+LDjo@Uy80Dy+W6Q^cb&$&z&&&to|_x% z-P@JFG%)(4oI*J5==&BK1CAl~a-T~lhqL~0+0|FX$_J~5_sPx2o9W+K{V2}&cQ5eQPDqm61(q*iGP zoNm!ksPNkBr*C`Ql@LgKz;6qwPz+qyL%*lxqg3{cb&1EE7@bxOkW7^Ah)bl zik&I94LjQ$u!JBQGf}3s)>63@H5hYB2dD_^AtE+9H`dLR#O zADPuqunHn<>NK9yY^()_i%25rfGZl(EWL|_LL$OPm5`dLgt!nQn{d23p;Ys0Cwy0u zu%#zC(W3ffRIjf@!wpWItm~uS%ZM(>-M)Q0TUQ*`-W)41fxQ55v11SI{(}cQs=Yk8 zK4=s?v;mE`d(WPcm|n}D!j9jI;}0ed1SQ$Znm;gQ=r}XQbP4WhWb&3 zobCsqP3e-^7jrqfQTbcCC?!R_v(4An7s%E@Y+DYeGsAS@bt(Q$nqUU;*n1=EPXt7d zI}-1j5piplGEgqOQSNW(ayTcBW8qqDU}_03PrJ1bQF{dC8E@gV9s6o)AbcwQv2aygyP z=()l3SRSx^bfBW#G&1^gZ`v zDnLzXCA~JP0|!i&<%~%!efLU9I8RbH{Q5|VOz4oT#>-g4&5>w>3>hmkvy29_jOl%@ z!c`Qnd)PcR6gu7mTYY{s?5~*DUq7M6{>YDup!fI^03_q{RZji!Q&&93 z9q5psjKD&zAUWh~K&qxm+Z&|Uyo$AkH!mQ6|ByXQ#N}m7P{M6ne+sd*{vtC~OLc6p z3q_hVrgbe|=KaCQNYb_yW1@%U_lQ;2*!GJwUBTW@mI`=#)k>6C(IhLteSk8~(x`DA zK0Cv{-=a23rh0}N`~Cr~b^!r(Z>}X)Yzy4phB?ytFg|6eGUn!9W>c4U(0gs^`o}M(u)Jbfv)xMb&3cXH9CB=dM`Z)Su5*YN6JNu4uyh@h7&5?#g zAWA2Pa{-LJdLEb^-S1DRz16_WMt^>m`Mk4WFLH@@$1Xi%8gIm7%DQFvf;V0lw}bGg+gLn6YxP9>)OK$d0eC)>IW zOd0~;213R2^rxFXpHtx%kJ23T7jl_8+#kztvg`aoE-g9b_;>b*y&M%iVV%uQN7ukn zKGYZ}b7V&gd(vxVBoT#|H76@&I(r6MvJABw8a;mhIGZ+3KBk#EuA&8)`N2l6Tt@I*&uzilQ*oXr=8Lho4?*Mgp9}mq$bR z!RHMkao%7APpx}0j|H!Kupy3LZ* ztB~B6yf;)XXpK2}+X1<3C;y?kXNJdgi#gKEZj4cy?3c=z2p1yS=IWz-;iuK7ltI@0 z9xDas8&TNJW8G3WrMr=#ciT7lfGA@z$eqXC!%FFyn{*e-98AV3y_QF!H$6r-^;lV1 zXCtij>hiihSmtxol9e!_x-bJyp4^R(l6Z``9tZ|R0I_wGr|lHPl;H^6*q?)#dbpI_|dsOZZ3 z8(^_GIXUa@9XcfSW9rp4qtP-1+VHrni@BCM+c1rzrCT-L4G$aJZh)b^@oSDDN$L*% z*La$IryR$^jk+n73;_XzfG?Ovc=trwrCMqIys zy>cv~T=}|!!Y)R<hn|Om{Fqm;{iWFTOuAzc>jb^9gU2O-o6M;82$$eSm`1n>TOn z-=`Qa)8!*d4aUX?o6YewNL}*~xATQ|T=*=e4$Uq7#asGnNryVS8nD$!5Yc2q{kL1|Vx{+&~ZYD$1!m zCoC1N*5wH&hp2YA&bMm|==^Yad!Ko75W~|2* zz@TgQyViRkinzrH_6x){1B1@Z&SqZBr9Ch;J?%5vLSOEIIvDLSd6*afrd|>JW+_^k-tk&0&e6*guD3u znx9{X8j6REkOL*%%#)w#vC3C3Fz}oWFp*K-ZI^qPfkDN}mY^V$uZ`~5FWTtM>_1Y! z8XtXN6*Di)&&nE5+Rp`Qq?{-+Vh#`iGY)USo-}1@!*q{KDGZ=H6w_KWZq!92ju9P{bsZrDfcIm7L1_^kzBKL*@Hft35uFqJNB<4?a!=7 zEiQYk{W$l6h|_q+tXPN?DS=zfC{2j~;)b6Y;glQKB2|9(yTad4RpL1Z4FE|A0gyB? zmP}9_FE>(S zFg=Vqi{gn!ssf0w_%wKK(4$ojBS8@ksK5990@Uk4au#-2*a2laB)8|zpU0$!akmGj zZZ7oN#m{0v^^q;Oy1rOpINKn-8REY3JM%lq5+2lRq1y~W@)zRXX(;-zDEZYZTtY&R zvMHo~B-(W3>4~aP(b5ul&91I4ctb+Q_R^P##}F)L>Oy2{gthO_Jzb5%7^gZfph3EC z$oe>-RcL7e1#ANO-=^Zk=0**wF$bwZ7t)>aT*8xqRhtDm`Q&~ZwZT~>f8gdUw*;y{ zLznoX(+sM5jc}Q$*P$MG12q#M(1tf} z)_?&k4N-CYfq;+|h9ZshHdyoGZAR}qckalox6oqNM|gz;D0B*q$GQrJFqCK>9ZDLS zGK4_XDV!%dLKqK8-GP;$(w4bhjuQnuRfEIK=fD-SE63jW{o^C5o}hN#qoj2)SJ(I& zND$Ksg?W1+I=W|$PRYs1TKi)njv1V>5*b7!a-~}hd}g?pocw!MYJ_P~8BnpYFU2@a9~*wt+?XtO>lsSaAx_fny5A6I2x=7C{`0RK>QFST2PK4#+7f zY0=O8_;^%zZzh}K`Yy|djlbtR4SuOb>fSMVarM>B;)yuREdfblZl4fLn|LghfE!QM zD{{K+v*0@P6p30w6IwsNxIPq^dSZX=yDKMG6WGa8OoP2}kAV;%1d)(is13i`$#EahMVXc^8o*c^b*@vaKlE3Y@UH4E^^>c2 z6$g-Np^E2pQo|VDmN#T{eIS6MqgW6aUn^{oN-E>m{x;*&9R5wgJ=--so7t@G(Aq!@EeOZtV-3BvXxp4*!>d3kw3ra`@LUowqeIuA^6n2)Yvdq&9`SfEWOyB6Sr z?9x+T5kJ)xnyq_LeNyv0cg~-U%m98N?o-aCg$od!LqjTm?yOe9&7;Ehpq)@gkK#~2 za{Tyl4vq}$4loo$d$N~*zW|HlaFD=1(98bRts8)Bb)}aMRjH#kckkVsfcl_D;@h+l zaDhxP^);KKM$9U>X5Qn+?|&v^w(Q7R2BGup1OzAS=cMl7kzsM+w!nhT(6gdcDHd`E z3VZQY?AeS8F?|JL@b`&BK`7imK!0EPZb7kT|T)StYOW%T;pH$8-a zvwbha`_HetmoVHt)W^TT6bnv$#~vTfA>gYs@d7%DsIAO39dkTL*R%npj7+0tXI}2X zLO|xA(gG%P&?XUSC<0mP??Nrl7_1PXDw`k*P^b#Cr4_VYgG)N&Hg`^;akL5?SUta; zdxK;_y5O}49LV{?MXWZNcO=X`m(o7hu+%cDN)Q3x5rh|J0POu&2&Uk`l$7P+$*{e2 zfyG|k+QAzguE43xrs?UqhJ=|%@nnn=QXCz?aaY$RjQq51pX>big(l0|V4I-+uh~f#BRPB{T;GHwuFHf`UYRHY%sG#Z@ds z?SWLFq99bAvF%_I>zGGW{;Poqvz%p9*v42%NUqB`|@L7cTC@?e{4EcK$2DTd=@eNG)aP5}fv# zLvLu4wj?naRC~HLb#DL+xcCuo^QVBQqE+uyoXT+wcE|tz{iWZoz+mz;NFI{B7HF#M zt$Uy6Do)-ublK^w1@+$-5?GgtZ!p&3@i=^ zauobr2diBMy*e-0+(axEBcwX&H$5mxckX0Xo|>3goo!%5rCbnD!RGp0su+3#T~&zwNC^O9(Ph)9Lh9k`K8^e#&5535JMYg z)n6{K_qk)fAS&n5fxE98=X*DtKw92Hfn^NjcxAIw%zMohiDA_3uAR)l@y~Hi{*W&D zpNYni3?@A!@#)<9`)&84s^F2q9QKivg@xsutn9JN&w0eeJmFvg(NeaC==G9=h*&0& zI25cego%o}JyDSdifi^>&_Q;}rMjWm>3sIjkC9`>r7+`s{$W}3xWR1e)q2?@osTd)WE5BDMF zeo29=_IzcV<_A4pu@=%%PDS|9D$F_zZ40q8g|A`hR@y>@e(IaJin(uC4;+9;bj?Ry zmm@H*7lN=GvpbZMFUc`JXG!UU;O?bym7srIWs@9-^Yomc?t}{NLch5E&&m^%cAZ5& zNpAiY67j2~xGK>8p$pB=0t4O|Mu;P zwB)zQ5>R8x2mC}p;f9@~u7skEE(9j{rzi4D2z}%f6bc{ozq%}7*~WC#XJgr{Bm+fv z%{5qjo9pQ9>AtdG@3`|xnNc-@gEbN+AH{Yd-YbUP!FPN4Uae27B%MjqGU{jR0bt zg>WN>$B3{_d+m?J;?Tuo#VtX4hsciT$>YDHH&CNe^~ynQcZ}`6MxxO5Bd1Qab#_K; z=&qJtDli1P_rZy~5{JWgU3^dw3JwiL`IACwKi`!l!12)H&e}PIo+-6f%}JC8b2ZaL zDPA`;m;qvNJy{&^2unN()t%aqjsLHaCRv^8jTI{a9e0ielrrDc*CT>!<-Xl{O;CH1 z?O~>EfBEn>)mTfCx+(t}8!aMiT1u=1->Xms8`x@TiX)C@ynHFieD1-K^OTg7JXCU= zU;~hH=V$#!1-5!^t_ejYXfO$g-d&MY9dsR%uOA0TZoX70S!c^1^JCw&pZn4?SIVMse2cXRSa!4T z&9T#GH8dzoMiA_eJs&QPN)xT;OBM6k@KjT~PjTe(b8Dn-s=D`_T7Scg#tvP=fs`*o z1gAjr#Q%^IrhMxuR@PK3PuqO(ojV`;Qk#xdFWsahW%l&)0%f?5h9)y+-5Y=tWC3g}_Q+PgmydyWe?0ht8t&krkpTZV|MmxinKgbgq6ad?~y8#rC=yi~K z(79&CeoXs$1dZ?ujcElv=h2F&si3Q0za{5|og>U9teoj??6D z1A!9j<1X!g*EXlxA_Hz|kI)ND+f{Rdb8}8IqupDVw;})N*+>r8q5Oe!0VpX|`PapO1ky7ly;tx>SvAAl^?2Hkd3(C$N9>uiOhUtNV(Neio? z%F4mw$bcb55{_PF%3eTL(tn+Ri(_CrNa=FFm+VtfP{`6q-DSdqikqp8o4L7ph*{F3 zN7S4q(8{3_Qe@Ipe_(la6&f5UYdQ*dSp`+W%tn&DHkO9=3TWdAGWlr7B~T6lc9R#> zZ~_sBDc=AfZ@XgIWqlf?`(ycsY{-wUqQDPMe{(!L-+D2bpjUXOk*_aaqLjY}A$7`K zRb#Th-0A7Dw}4}rir=?!EzpZPe>nOLDQ-y)HyhjcU#${|={e<>WwD)mO6_Xg=Pl=V zI)8adoDo}rLX{7sVd0RT5VUAIux5o9SxjvQfO z@h`gHjE6+|y|3IUTldUu8}0>IgS`b1p5Q3IIl|fF&|BR=1+8GiH?EwM`(h2V^tgzS;6bzFAzl!9y}0s7*?w&I?Sh)aerjMsbhad z$>;tTHIE)W60#c*>v^4!{Y4U@2$aQ)yYqnR0lAVM>TCypU~Kf7phMRgg%+-((9`ao zAH38M!_zJ*DykymMyWwOvC3Gun_p0Wz5XohZg@I##N`U9I+EX|CD%fGqzUB~{uE#j zE?h2foi)5TU#`QYm2q|X&SlivSZzD=`oRyoO?X@==pEoDFx?=0I&wcYG^9i89c8x* zgK8^xvg+qKENuRPsl;jj^IVj1Jk2voqWPmt%5AVud&{>663W&Dg8QuR+l_C)Rwp$d zHoiI9i#`3_l^ok3y43;c5p6>{+YCfpi7FEIUnB(-K{R9@dI2FZDA?82Rm7|cqz?Tc zYZ1{1LRqhK%6l&<#9XXv5&sVA42nvVmiRIV`w%($@HJfL)+EmSMsU#cDJ^FzJYtqU zUdE68_8CVv4ZN34V8PJYAP^Z95Y`=2WIi`%0vTK6@oTS1LVx!VkW$_vbzQY}H~)&I zR)*Ru+Jo#I&ZnhG30hL%2Cy=#kV$c0%vt(!YcIc9i;~Rt9Xmerq=0k_g`BVS0+rNm zY=yO=R)|txFvCTDy*=7=tVZnmw>LOWEh8&y*Y)fgxVhr6a*8LAC*Hk#H(pVp%(9&; zs{l9zT5AfarPWo4m<4V79xZ+^0ND$tA!=r+rfN*>YLVkpLvV%Nd9DDpJkWycCl;=#!?($jvDacg@8Dwl_AV^Tb z_@=AJp@-%oU~J0$;lgnDqD2Iy%zSF0apRMUO9)FL5t5{{Aaoq+wYGy;l9&BuQOM z*#72w=lO?Vng}|n`7@A1bw9`ER7;_d`laRgU7U*_*y@ z%YjN^*(^WsqH40dC$F7<YYGffJZoNak=2v0wY+h3a{1EqZ(T)8_y%_Y}+c}jpWw*$Vf{9p?+Y=w+nSmRMk=| z2~pmw2T}hPC9Ko;UV@+aBS?qh;!IZlwfWI zGPg#Hy4W@yh}3d`wt=G=lzMylr>riwsdCf!q$bFyyuH1(qRXx57pHICya_qz%P|2z zDBvEUTyW(d1mpMO{HFyftkeSOhc*Ak~9V>u{WF_Ah`B1$b1{CUf%c9pH{yS$}WC;at=mFJN`ivLDBJas-qWbSj)} z6i%DIr&f6mX#ks18TIY(F)ruCmZN_PkwZ~eLBCGSjTE*EJ5a8=O#cD ze(A3O*0HIVpxs-1VaDMc6%C2n^H=@{2%3fgrT*JH@&t~Si#g}UoG89Arm?b%jQQrx zn`c&jJOxQM@0MznxTns0_S<>B8DyK~-JIcqAJtMPq5Bf5C|8l?mDMiT1mPI3zG&}J zLPGcTFM*eHrHuc`5kvTquaMm6YMv|-7YOn~regdkm@c84~Uya=*EF}lWtFQePr>N2i`Miiog5z#5*(f*s^Fc_S^vc%*rooLJ7 zMS**{7eKU{+v-#p+r%g!hg(}33KJNdg6R4b*vL0`_Nl9@myh=c8fur;xopjuYFsw}{d`GzG1mfE#0Ptz^1INaUYkRxH8;#w-q-(pt>n6eO_>Mibk0NM zuD1+FC5~5Y#p5muuYFQH)-Cz>H9)JybK|1qE|;G@b=`2m zMo*ofph}4tzeWXo^5iY{#pR~#>@FSZ(WdBDU73E@hmCDj9xl&gh=Cq?alQ)<%}Q)_ z=KVI&{i{gkGv9+<9izP(0hZ?f3=CfqkEC(@qMe2^X2L-NLK2rDtWauPy$nkjua%Zy zev(14`GbbvN0(BwRn zuy>mlTL{R-dvSMNOL{2IOZ080f94Xl!+iB4T69mDF!8tbSMJSC`e1FU;qlzt+bR#cZi}QrG%7# zigXAHNOQ*Hde`;tecpGUYo8DM!@16XUH=bD{nhh4_dV}9#~fqK{PagtfC*DHPBnDp zneI5>qwSxZ+2z{fm3lF%T{r&VaGQWH6(9|$QHaja@lFNa%5$3C30 zo14$m1S-?6<4T&787wG3i4D?{OE3N0xZ|F2_4RM0;sZWZ=psevrXB}7TSmScy`!o^ zXo~%~KoG)DD)U=Ixu9=1-kFQ@mlveqrjG`sj43~kkkU$XC_4{2Oj=vFq+2|5HOlf8*pG8ZK!yU&7(upc z2oBNXi@~%j^hUQ{JyDa_tWFmTj1&EW{beZv4jR$67Fh?z&&q7%J|+0LfLA5;;(m?~ zZ%f-2!mLa5aX^(fo}^hpuV_r+3CN-cFX}(A8(#w`TZr6i}*{MVz?)FU6l8hR3DEtx*o zn&m@2EcCfmuVMNM;N!=l?YKf@5~Kd?J|&)OWrAEhT9axYpf%<%Joj*~b=ofM-X9*E z9oc-syjvR0PX5eGXcqDd2uL2cA8t&quZ>u}Nr5J;EW0Ve#`=lQb%)DtC|zD6M5k;WE}V^j~gUU{gr5rbu29{jWwqbSYFw#5x_|ur0Nc32diaX&emu(-0a0~ z`EQH4HiCj@lnrPOY*1NQ*~^NYKIZ)uOXx3ghsd!b*+;|TLx`*|Rhe$Y$Q{eaMrsWv zbrM(K?i7^BMs}NBRd|MVUd*X!J7#A6-aX5XCLbQY#!0qAM~)cblH`-d-rIjCI6!wu zw@_S9?%axv=UhpowH(rYlG3kPD``caUi$>fr$Hr0$|`cRQO+!$AXhMdlqGuduLZuH z1Qc^IieL2wN@B}td3Go%0eLyf&rFy9%-BvO3A5;iSXLY`GkvlDYPI78-4yHi;9ydf zS8@fwnc+Su2%O}ejO*M)GwYHS>Iy(j_wJz|J;N6j{ZO^#(j@vg6$14=e2LNWeG7p4 zZ45TQ2Bv<9=nyeVnRv}2Ttr!5hdAzn{$mC}^yA{40Jl?7;f89He;XO;L`j)@^&K?I zIB2e+IEN+K0paVH2@#3_j@ICs-#b-m0H?zCU7Uksp zL?(n*_XiuX>t9KfV~EVe%texe!_TTfNWzYQ5?x&g={>4zn5Gql{sLcyAZ9p|A@v%* z`@PBRq$G(y4STKKk-gN^IIJL#cV&75_WdHQC6JHyBmN^C3qYCG9iZFB7w^0CzY0xS zH}Z$SI3-v%M6HcqT)0wJ+x^ps;)(HQU(ePHz z{>zoZf+bu?MQ;VwQT$k>t=u|!$JmuQN~L`%%9v-`sXXg0UyTvBq7vg}0G9gHEx>qW zYyhQo(Q(B#JzOP9$=9Nad4g|HdU{l~XZIMB+36 z*%mi!I$>S0SRu!B4jyN)^T=io4|Mo5Ki~1?y8^OM(7d$=S;1@sZ;-JgzC!G++f{* za{)l%cJ^bH%W&p?jpGP#mxgG`mq$rb_>G>6-?mx(R!FZwwWt0aJs2I$GkjTHf_kTN zNj?Wu{?;x_J&CJP%}~g%%hl!ObW${wr>3VTNYT^P8B-8;HlS$1kdFDU8=8RQ+RlZ10XC> z_iaT4ge46KYmeLe7#{58OIe`6Br0j+jspZs?$6r;I`b^fS%Q*f|KmMD5CeY@2QYD4 zI0C&y-50Fq@&iw+Yp6qUF>?*xSdXRPjBttqV+0Gc_=>@S3ngb!0Jyd z4yaL3D2M+Wg;NSuR0K6gsd=?5zjsvvT z#RvNb^c;+#Swad^B*2aphJOcYus)gJvi|$&26PB+Ky$`59iq7?8N?Vx!DLG;t&0Fi z^-SP%^VOuN8DL$2I~u+4@-uh0u7T||p6tQnw;2Hy8jvY1BaaUBX29ldjycuxi)9dd z2`2@iw#FI#ar*(lTD&?~Gq2CzU=$Jx_cX#WW*tjjGu9`Dm^!@sjw>KnC*Ui^2M_b7 z(jyDo)r|yKJ#8uNsb}k#+7xCoESN|*xEMENo;}R|ar56k)H%S;xRxZzhd2?ssd68m zJ-eNU=_B-fZX`VtkKeEqb*r-fcimfq!Ji;(w4RU!^wVw-Ay9ukN|O{%II-Ung?w9d zr+OO!YwO_i$CW0L+m;SqSGlu7^&l`X?Wa4uRman0O9I!z!=-Zd_|NP_3V97(+&?=d z-849#b#eC*!q1SRL!wSuLk4fU&JY%cHyMJft2dBIOtHOU&L_gPCZ+l$=pU)+Y=G84 zCv%LYl)p|*3h{ix)39%^@0UN6+Z~PGGL9Tr-(J^4ZEZVo;HThDzrNN#csTFT_o6xq z_;94{l=FlOoOg>P7+l&fy{ov8>Xj8L4z~G_KQvHc zgrX_cw({2p&T)(DJf|A>Y{o&HaK8eNVH$0#zPQT4l%g}a6Q?~9J#W@1BijK5{bm~@ ze;C=KT^5QP?{w+A##$C#^b2A{PIF2hz0q9C9rmS^w*lJGaB5y8e*)z+s;-j2E4y_` z|Lb)GbhP6)jaSml=}K6(vmF>4?=8;h_jr@ySA$~--HYQsY}zj`>+9=h{Blt|3P{6X z3Jp`_%fPPbqyaU8Xepw!ou$N2%HIALcqvMfEJ%Xv!}gD<@`A6Raz2W<9w}VU;R=$2 z>mZ=4YY#sNJrL6uuld9 zfNA{yjwbLA_u~Hm_u`+M`p-@M-(d;ql(c^Xw)3YZwI$8)Li&gbRkF&If#vOnH0zqr+6*wdr|(FbSvA@>pFp@hKpR*)HkJ zd+cO)VlCgR`IJcizFfzNu!WtUGt*WEkMj`$V{ z9Xm$mb@nZbRA3JN{b2F-m#b)f@y>}pswMPtHFb!a29qQjl%xz7o&lv7`1m9!=<5~5 zvkv%Yv6vbZUt_cLV`)#_oD@Et2H6V#Z3esKR8an94)T58Qplh<#8yDJ93UC4Jx9D? zk@q&bO09!ds4q4iz23mJ5dVEOMm*aExJUIh04k(s3Kh0K;4|N0;v+?Q$NZO`05g2ZbWhE)W%kyQ}X z$Uln*xc|-!_#b~D{(e&|88tAtIf9?0dgc&e!TO)KoPkU%0I#_(LesQq(@)~05wE(S zfR{_vNa5b>`oR@RLAsToaFMXSBCbyyU+Showe7!N6yAGf_jw1Y8h3n?dtlBdHR5)h zpu!QgS!74*9q&!}`-3%dC-%VBlu(f{(7quaKhC^J{K@~Y_xPEb*Tj1x&we^YHN${! z`sddCb8G%tHUDg=|9b%aC)WHEhW|OC{yDt=$!q@KrF?1Kn*jF3&P+OhyeV*PePAaj z$O(z?JzcGK{xU~+^#K4mfIosX_rkDpJXN!nD*~N9XLe5w=prr?Du|WtwNuxK?u9jI z48I42PpuK|hSbL|E+SEmHG6va@cYov(C5#eZ^z%oi!A{Fy&j$ZoE$l2HhD>$F;ug* zC1EW{$l-Byh?;s(_s-zEK>8Mm`@0X@e+-6utGX(1EvJ`ktS0dEM7yQC{xZ`W-s;uw zKJu!*--v|ZU;H3%{9grj<+#&=Kf6ajelOZM0r2+%JBTpv+0lMeTJwQ*)AeNctq}LI z`Md`azpXLm#1+z4BH)2qawsY(c}`>(mkeBbu9DFHL;3(R))wZkPJZG5ELW}b=zy&o znVXNIkE{Qka)pKfj(GG-p7C`_`H8c$r$apX;|Z`iuml`IUyUA49vDJ*0z*9rhd<9t z&wkLKC-GZ5|KywqvUj4S0iOam(P*(JD2+u+UZDSn+JjRS`^%SY0iBRKu-suK>8}{* zy~4QwvhAL4kdXoXU4kQp0_E<{9|$K$j1sFi$qOf$B|en-YpS34i4!k>Y;=WCmw#4_^Hs)>yvMR5yal79s;wrE~TaobVoN{Z95`7AeK=H zB6M@k0m0P&V5?!&m3Is-jV2Q@V{D1oY|Hn(cXf?(=H1x~KT2j4gNxnpN)6lvW7yF2Px@kaqN z|ApJJys{GF4Hu0h+YfX81!r@in$jV^lHS$U@}A!C7kFIT7r|mSa{1Q6JriSBwMSYi zSCT$d^ca5KHr|@8cxZ%jMf642RE=#^3P58}mBQy!oiFOr65d2URA3TXx*asrsj4^m z0MkX9a+dvXyuB63_g&0AnFRKyFBx>k+#;tv@uB?JmD zmGkbix#juJmk;93?85Oo{j7_y4L)2k$8fcHHX=x=7-TaznL@ERRoK~D&v3AgpjK6& z*gF_{eFs_Wg_`F?Bd0A>rjPsV#p8szZaIiW=pI1$OZY}Y_J*I4pL>86+C30v@K-NX zleU_Ub><8Jvga0ma=fkzi{jAdlz^>5GoQ)+;R7nsbaL$8Wqt;Uxrly{Dcgle*+hlcnm;A!#~Tjqo?KxZWeg5 z`Dl`$^#$?b6>z}`!@rm=enQm^V|lxZ*{$N`0Pf}KEEg2J;c<(u1$mX^&_&S@Zs!_6V@F$C8)QNsD(tV+j8ir} z#|VmUVguM;nL5H^@O^?=x1)yvf{rO{5rXSYAkk67(t@fKHi>f0EGcj{o+R1`Af4g? zYJw00p9zA%4@33n3%g*`2MWnIyQ(EKn4(BaOJg941ne`x+tyMX6u$!L)U2Q`1z-S_ z;5X_#zrWc&-bGl^k3W@>m;ZpfgW%Kyz4mhUD%ibd&9Ujz74Bf%;)kv88SAeUT{9Ud zb&%m6hY3CKepfUDi0z2whxp(Hkv5R4= zI0=U#s5O0y_z5~Uy8Oi;>EZwV>@EI@eg=1l>P|<-Ufl0|GsWJ+?=?6U@K)8=7Uctb zS>!*m{9#D%#})m7jg}Px3|fycA{_vo06_Yv{j<5_Ar=TWgnJvB^S;-PjQ4{yb>5~2 zjhEWmT7th@S%phgpMp3iOwIO7iE1aihBVyu`zp}~j}ZCmn*I`?2!d{`@SXz)@UT0u zP3%e8Q|e80^GqJ54luyXG0sqsZ{8Yh1@vuxp0aEo1&!YZKS*1<3%tcELNS8uH@1_; zpTUMO_{$vB?N6G0gN4GOnI%hkps5<`ph$m*iBJsaP7`Rp%Uz7X({8=B(U#mrkQW<_ zU5k|);zoePv#o*aSBk}o7LG+89Ndqwvs)1SAcx`hF9ge*)5|Om<^^tr336OM<{+}8 zb0eHmCVBHj5Yhh>zW`)u8!~uc*dIjbQT5Y4LDH(v@FMtRs3!ijlHl7 z4hL7zV2qyYzg{1g4bMtDbKV|uphA2ASq^JWmWj52&7O~qI<9=|K00NflSsBc{fI{* zVz_I@aIU_-{^_k>4>2RsEut1GQlMUKT=AgZS5hTH`+xxZg&u2u@1<)tBc(Wk(1C8t z^Mhea)9w=JR6v{id^!ldwDy~f>gin@VAD19_)>jw4<(vAQqGxX@F9RAUWcL&LFiJt z+WF!xF#f<+46d$2M$oEwLA;Sr9!=+|3nzvCg26ya`8eaY?6)ADb)(zTJ&f;acEPTP z8evou@;2H+whwt%UT>W@J`E zOM@r#Lbq@n0WDR=l2CA*@&YgFEB13NdO|p7z@I4vNdiZG+g)PxjwNlZgkW0vVKJ8z zVq&rJ!Dx%$wCzJ?kuIfxv-X%M&eX&E_Gu$jZP`HwvbYy$d63>6bw{QX0p>J1Ge}|_ z&;$fiBIi%deoDI*%L8gzB@X+OULjOxc18P*l3m+dCC*oj+3~a6! z>a8j`*uh^%Wf?ssl@75AWy#($zqkPpxLf#I4Zw?&b{+cCYo>4xgXLetrq8vFhDBOE(wZEK=8{ky#5+z%O>n7VH%C80_eQvc&Z6n3 z+U9;1E_Vkj0)*~(%Mh9&*1G1Ld2nTsmDS%$SNb;>;1;4B6x_;D{Jy8{r5_;5qp1Zd zCj&$CNYt()60x3ht`y(UXO*wcCY}`DP6R;oz!BNA!Rn{kKwg;P6{XP*j$k`f)J*5_ zi;x5eLMbAjl-aehJi#0NSqq(g+y`syN63<~-4oX@fN#(q@hPhYq6%S*H9_L>^t7Y= zl$7V6KIaVa*!Xh3af98g$B(PMmWv4I)OyTQ`lke^LLMt375JR~0Rdl-iC{QFN>#M_yxQ1dk;y-UAAsY;#k0KZtDcUej$T( zG=a9DgK!-qB%XmGAgt|GBFNx}?u&{QDYN6#NUCAR555C36&b0Is zv^c%bwNT3>_kaMAnR-Q}52}%-@+t7t%^JcL$rHM*yyN@8pkMthwFQ)?OE2_w$iKnA z2s2Y&3?XONC>*Ajl^+`!0gLd`i5qUL3&WFFgsjsQ-^66`##J=M(Z$Dz%PJI7tAM6Ah_SDKf|FLYKnCdtp{z+GuwE} zUXVR}s2~|qIjp$fRSLK9e01XV?W7l2FU!_dfo9Jj3-gD~D{wyLLqz?n%zqvzt26_l z<#r}ePtIiCNDc)hW1o<><4EsB^|_6E+fj{pYq0!(2hLv_fP69 zTTKG1<7~=ofP@lPiWqJqcr#p9s%sT}9x>kqeC@^?z{5gi-*uda&>ezK&g`_UY^~I|kG6wJ8hRd8k|H(l2 zl*}xcEy?e(IBv-A*5+{P@947kj73F&(HFbg)wd&ECHyj=&amBf46jp| zNJ!tZps1@vA%nV@_%Ow~sDF(=kqOv0+wcB!_7DdX;Ul9lCpO~Rnk;82b{{3^dF5Sw z+(W|qw`tenN&}deijT#?->Vj+KZi$$K=V#zGH;k47qde0FcInrWlpV>VZ=W8D)iof#E?gTy#5=zA;|D ziSlWDpXi!?>abC--x}D80!Olu#*CAlw({cs($LE0q&pr9o^T)2{`~HXD+L|t>h~2= zH(g^ohjA7%_BC)S{d%A3PPtmFw!ABN$~fMh^9j|8Nv=X&6FZ4w8p+HP_4dv~`o5ur zrd|5M(er-Kp9s|oWIscqXnDZO&jS^fM)|Zi*;CHs-QI+vE)T`Sg=}egqA)J2yUG+z zhO8r*Ozc&N(SB`P?&NxZ1*(~2q_!DdV=ESe0|?~M^%>@;yn`n-8-#qTdo%xBk;~y^ zB&rQ;WTfvoI{C-Hci%rntNy3|$3XB+wMkUr(&SCY*p_3G(hNZ#;@Tjyh=vv%w5Dk% z92@3`ByMR_Uf{|eqLZHCux&6FxuBH`|A5CC*u{~yv@nnfy$n8?KFv1NkRZ5MqHhV3 z?B46^zKh75gro3?2H)=D_!4c*3*U6AM4WW`G?L30w)OhSp=W$c$N==L{*I2yJ9G`g zSC)fFEeMrkgU2qBlt0KwE8lgRvy;2*1K4vZR|}-m(k9BlXI+|L(HYVhWNGK(oQ8dB z{ad5mfATcy)*;o5zWsbZg=Ry-_5}ok5!vaw^t`B_x*Sf?Ht~=NDTRFSHr*ENMbV*I zK~<-8m$2;nyv%`=cDtfo`(eu}1X`qbNLDVay?|pE*1~os!jnNLeC2BNA5sCGj-J!R zgsE|w^urAx)~T7m%2d}kG*~o|NKb8c+wE)gKcEyZR@7zT`AtXa>A)@dh@0LNG1k_22s-Jnv z9Vb%5v5LY^^}F0nl^!#bdt^Ma#}oQ##Bk{c*h?{)!#WP(X{BN=Gg6&<8{Q)^v%gIc z=H{njvt*wf#Ri5|=bs)`=jqhvy}wWJ{hdTwV}nk4rXOLvRU}^bn+l>$NW@*#1(r*> zR1?oZ!;1ehyLW2$QEGRTArVUsupE)U%FtAWwBXxA`&IYk?!h>IJeHpN3jCDl;B14` z)XnKd-?3v*L0*q|?c{y`y1B&~fiJ*@$Vvx#x209aK0c>uI@mqjNqAxmQfguBp;ycc zpR>NZIG*!>Sh2vBx^LIcomrUq0&eF~W@hfp%O?TLoTa|5UiU)O0#CPTZs-=I0XGwV zA7EX-r?(^EkVvpR++j%HPvK7;PQsW!JBpkSrA+BpDl$2iq!ci^LXCrN@?Z(o?}(K9UeiD$Ro1i zT?l_11kkyOPF^mqNq^od5ZyS36dxHAQX7_{)UGjFWt83}mWFg^((>&qLW5an^bX`a zB<#ITMmG&3t;KkX$3@+PZ?JHYg^qaMjjqdo{pIW(<3H9iNsf$PU9oN2_7K>z)kvixcARG}lt(zGcB5 z1eSJM(NFk6f)n;jRbrN2+ z442ZN-i2*QS*GnnnUh>N`Cp9+OHnUocjvsddfZTHKO_%?1$9rmKtW&_MJ%T@30Z(r zjVxilU|eHAAqWdqBF6B9J&uA()1wzZJ-sR*d9q?G#iov7|9+9T-;-?hf{)iJIplYo ztS{ll12|>d$am;+U9*4AXqqhfjo-r?7C)N?9W(kjHXOG;{uaq`mM!61pO~EavOs#j zg;K~xQ;rnX(8T9KI?rWk)MTQ@4CME4BpfyQ5ELaWd~Yz;;pf0$=kl!=f0n1+M+^RV z3|zBYab17z__OkJ;fwi-X?MMx%2b2krW;2SjUyt`_Y_M$)b4~vMsNm(L_WNnL5Hm< zlDY^=wBmX7Nw|d3uLKfz=0*02|ppuD)CXb zc6iFEBmiztSVH)BUar~`*Dx zCf!p*wXUYtIdI%$bqwXv?x#r^j&@A6<&)7gs<{p0?I@viD#{>Lo~Ns`Kfjlsa6h?O zx)`_@s~{Q%V!-$2tGwOZ$AAuPFf*In^Zv7a@nY`Jonjr98GdJ-pKrgEGbg@r&V*OT)8eK+;O;W#+98XUhFAGlZn&;2JwcIaHrd@FziKF*a;9Lj|rExvfkg4x5Fh6t0 zw_>kqE}n;h9SI?iqH9%xzpg;*gM)9^R|YmdJMT#9A}G_&M@omn^=hpm3}zbz?d-qz z5#i8(^wFAi+N=B5IDgRUa1xfKGj$p;Tc}K2)+vb^U?P3gY&F$rn)EHZe>IPc!#c1yh8h19DU>jD+r61HG zo`#0;hIem%f5>{_P!}OEg^~XlqFj9d`;djNszwib1^k9Rh3XLs2sU-Ho6T)?ac`C%wa%|*D=p44;4As>`M z>C=>!!F_j$58Kvj#*dzt;f7~BVTM#>BMQJ!L`sQf1@NlkO1P;9((?55^ynpgEw8g_ z&LS#&ZJzNrQP<+rLr>4QtG|Bjj2ShS>c#zTnW4+TTITB>j68UsxB6ft6DfQ$&N{ow z`S8Y~13(uL{{itIU1;**+7Bx7ZP&F(WjyR9sx~R|Lz#GV4^<{1!7Kqv2t0&#){42@ zw?Wpi!0rKiyX5{xrv zP-2Iv5u*Fz0%OdqxadN8uI!>bySi3`iae9i(1b+B)C{9YS`h^OLag)eSfAd?Lr8Ph89kkH?U(7FM*%d)z0c+@HrKvTOY=rNoKfdf|oWQ}j(~$Va&u{^X^)v2DGg zLafqn{SCEpz`i7K(j|X4o@})P@=O>KdJTnXT@baE`sscod%(&eGLUL!&!!D_rv^fG zoHCGjDNFlP(FRd5&bKX9txV_J2*6Jw7&rw-%8SzB0}q2nPxr<@NA1I_llQ}y-tc*L z)Y+;}>)d-{zHN_+FlA7&P&&d-TOR+1_C~ud!dY8Up>eU|Y0EDqxeFUm*uQiYgl(KS z?G)bovd?c(b$#gxGoND-BnYD%7nT#qn{8nWv(qx%so*Eui{W zL%N}zbMtBkf)D&n^I&@;B=dBh+Ft>a^OTJqt!yQ%b-qhc?PXhFJQpP-{WTJOEM*_# z&J8s<((-e1HY^l&SGpH}u0+-m`fwu8J*xCbkf{#(8=oeQ!nc_xX43L5PTwrU$}+VK zYz8x}WTcwQM?AXyn{P7e=jw&0+#iCj)0QELn!@+A^K_PCB@U9F2eAn&IBKHJB$r2- z^EVUznxzNn>li3}iL5qMUP0lcds1XU<{ z3dwWOu3Cu}2?oG#q&$2Gdc0Gtq5$sJ?(Ez9ZRW1kqw1z@XG46h(rKxS9O< zWlmE)ycILG*vkNCza-01uQ7vGU%?(zy*$pjHd_Dwy$cab&}D+kD7z3qDA)blaW`A# z0JpU>Pwl25j9Chc0k7LYJcs`_5Od7^2*_DJ{h)0L)9OSDtvj;n*LD{3<}o)5P}4?5 zTmpm(UD0C(@iO4&od(Wvw)4op4&UIHE?G18>&xRZC{)XUtXo-pzTN$2W_6*KeDf9f{d{kDdwV;)-U9aVB|@cv%-T_2ULFW2BvEPiyD^2h^IsUHK7FXi zPk6>YbNHOk~=p zQ_|7~99qPIp+k>D%&jV5PiapIl;HS`o6VH=Dc*g$6#hP_EG7}#;0VOp=cppzZFo_( z`>OD5W667W71|iDTT@HrvP$Q|iRMgT5=RP8>5pKv9s{x7kjvmgHGs{|OMV#UH(QD* z!>*;2SiFodTD|yv$6DPAd}rqmt^!n-`1PKdd41V^GWXq_{3I5sV!T-Qs^14ylRZXp zIHf>408AIHmI3P*2v|MgK7v|1tK3&U9kwh+7Z}Z2;<*pMhp=^Yck=X$F9^$fM1yXq zPP%=S|7ZXm6EF;T&)&o%`ccx2+D-R(h;DcXEc%*C5oLfLBm6Gn>M<&0sGy(#90YA4 z>*}%8n10uh!<7cPT-;o!3LvXQ3yXzXAtlBG0H8Bc48%lq<2)Bn|-tb$&NYY`D7>bCk{VDpWV_E`Rsu{otr`-~=>l+{yFVDAeRm5_0r4<o> z3Ol-sh=O%3UyrFwzIrIg?OVfl0cjz5ovqGifK%tgqBc~aiG}@opZ-xWpO9`u(`g9@=tK9VBH1^W6nc@UuS^jS!~NHiuZ(qA5Ri_v-hQF75{1e6vT>)X`LPMXspu^P5hz)YtEc;$A}MJ+qVE8eYB0Axgm$ zZn?=KB&l}rCRSJPE3gQ${(UT^Sz=`p=LLLQ42<~bL;CF-xaHFW0(Nds|3H@v1hYu89llMXl0L6bc z@&bmltGXQ7^%G_pLf;4Jpa()4?!`48mb@Xryk`+C2|IPf6(p(A3gu@LJ`xRYcopGj z8Gp5e=;raO^QlPbd$SX(gZLbPkMlU8#b7arE=7W8BwPv3#RS)14mZVa6v0U&#Hn|1 z6|>Xw^73$L`PWw@*XBY}uf&{s;jj+ld9)+Euz^u-5ZXXAiC+NC`XU>^3H!{C3#fw* z^YsGx2huut?Wso*5;!1U@E<|*F+Xvts0;#Wya|Zz0&=;@_VhMX^)@~Fpr|0T#bi}0 ziCM%zBKn~Th$M}0@itq>{EX_E+8ww9HfYu&aGS<05q2(NVWytqXfP(}G2IW1b~+aK z^r<9}R4qeSVrL*X8imy{%=fPPxZ}Yh^4}t=AR@o9t0H`wkaPPzG2%FVoSJRnDu0#$ z1?d?^_!g?E{JDtRx>s}*O{R|`Lr)DF@{lfP@s5}Jr266vp2J0p^QPA1}490zfpw{?tH_`3{tA>b$$V<;9KYMl= zy}}l;-;EF=i$XD%zk)th0?dMoaV}7b%}7P?7-0C~i+3mS%;@^H(8@E^)C{*7VV{-X zi%x3cd~cJ|Gn2N5U%q_7*H_{xAzv*2cJJVH=}0_Q<=fGbUccXaNUNB?iquVu-Vxrp zi98Zn?(to%RNmf!sHi(k+Lm(eG?rn&A1;gsDF0xFqO9yDN%Ei(1uWX z<0W6iLB-%c^WiKB>G4f8<#Klg@OR9lY)(hE?9gH&*+fouAFd+8xv42svRgONxbw5A zZzE$2BEbifN!1-AW5i2JSilid2>OL@mxG3(m2uZ5^2G=ET$xw3m&m0^NMA8|8&fJ} zS+J7@pFsfH`JsK&Ch}w6_~1<{eA}T|0C>3lSy$iQWHB`irXesI6Uw2JK|m2v9^>+G%iY$;3eLA_}Rr|?3Dl9j(={)|JPQB zf~bJ%zqtVaY@C1M$UkBApHt?4@6aXZx=mOgsCNy-IrqB`8F{rxtYl3MgFJRvTT|c= zDb>sY!Xm3DUR7Ca*2nW!AByNAJ)n{pkCELO?JfB?51=B$iT)QVVPqmhW)>SAok6kI ziA3CI@wO^)LT|)M0TriMn^ObZyRp80X(UXA6vyuWm_#tXCj;`MvdHrj)#;r^zapIC z@kdXf(NQZ|HC^#rNm(TDPjLkG!$E+_S)Qsd{y=<025jcjm2KpE^t7_M#Xr} z5XbM|*2d<&=F!`4pW72e9dLEd?WyZ<)VWK62E^y`kP844eVebq9u^;8W!1?;FQ`KE zA9vFasS~XP1@2wADe+8m%PYWDA)z6XEy}%-!Z(a3-)VtGAwDYfAK&$#$+?D02L0$P9{<*%zKQseX=H?f zqj_LX=8Liq>6bqIy&)uI-_PQ0M;>e6ma5r~s+Gk!f|!mC@M+Wiv%m2Sc&UY9L=5gA zv&1XRnwd0hncekYzYo~+(FJEzx-m0l>3yY;IPp}d$VrjoQYBtI$P~1bl-Nkw4RBmv zLMm?3u88CrlLM(R@XrpkNgL&t*j8mHXV+{IXZcke1QD+k3EqmuRWe~juEg3uJB9Wj+^XlMG9V0w%9Ap7y4s8 z3=kv#&&!{LG+GKYFOZ#YM*L}0mfW&~c+f}o5OfL8TX7D}A``WG?F#0Qr!zCM{yC`E*?UL7KmJQ06P@do6jCZjH|IgD3vJ{x7!w&eFt!p3Lu_l%b&^7_i`q83}Z# za}qD0C)>Jn-*ph}%TGxC`ys`m&KTTD(@t<|LKbpx>9=2kdmI4h7@yn?4dDlFfVtQG|@Ee6MMzB16wo*vEuHT?PUsy zU65D|onOUXlRFmldIYqP=8WSB<1yLGAx%EA_0YM~nZS3LWM_BSiJ@-c; zne+#%pSaYk)N}@B6o5ylXv*F1X?uHbMj5v4BO57-z&rSL^j9dtL)aEB*TsBxdx$@K z2td`b-*^-0Zd}G;sU&i&^Bs7Lkq^7JQexL#T07kd7X10oC-@ayd$5IE7$Ml1(-70;8>wn%KXG8=PZ|gRmEoRR((u2+T!FXvoy`TOaqB37% z*VU8OePxB~f`u4I;?!I_-~nj5pb5_Jv{j)I4%1z^XNzUo3o(#|-xvSidwN4|{yu@s?#||Fzfz z_shV{ujud84IeVh#Vns{VDkj_530}ZL^Y0+Cx1schFLga< zvG7~FchENIehXYHDU|Jy&a2I!(%Gjxjo4IA&L#K1eQVHnEfU4$w>*#$~><% zqTTiIMW*@G$+y$-UafFDb1PeYV~I}7Vrd)t3WYQp1a+sx&hZquhp`iT_mK3ZVpSZHYx;wT|g>)4O!T3&FO`b;X>WlCHVXJv8NMwYO1R zNgX0m9CB*)StgYqewBp4WwHtGfjyJsEJnHhWspl+-x@97chv|J*^tkX$%6RT1Ed>} zH)tG``}vc>@s2QXiV_+3tY?Wh6ms(9Ank*rxp$4711n`|%GE3}bquPz+VHd2cy=nkDkX7}zQTVUKhngrtxshcMgIjYSAj1n zAk#=<_{ZrOHIB!3t-m?msM`eOdb4@aQBZWCX;V^AP?x7wBSBk0af4*&RoD-{1=e^M zY0(+<3VJ_^tS>6lx`WE06e_u&$%C zv#WcNiF?^IH8(eh-bh_@W1k7o?eerZlR8fnVtvf>;5|bT^PVQ4gH=^0LKJS6;OGhuu1v@RlkaHN!MZC?h1yaOc)YUt*g)AkuHpmWtiHf3=NU% zU%h$uD&`&$zWxl#RLryc*!`n=Y-|h>1tp)?;iN?TaH~y>De|FgpP*yyv#XkGRhXa0 z_P&2m1TA$OIOtGaWY44~v~FKYW})?a?Jta(LwJAZTdVfxdp>1N8Ba=>+S*Q|jcCGp zYgQ7E6sibf?z#2nxp!DYnu`-Z!Lk;`Xez8S)9%$fTXsNO`uwL-6hDjc4UbtoUe>sq z!OiRwq?nEyw$wQQZ_*?AlQ-TSPIIt~62}bKhYuFNAcpU|f07{I?m8&co*(V_BwEl4 zqy&wF6+f$13f7=}jF~^q^T^n_MW`OrGSFCqwRrp?gwdp~b;+5U&C6Crcv|>L<2TRM z8LFt9z(nI6ycH^*=YE9|J=Xy+8=#9Zcg)t#O~jDsVap}pMpikMPOt842{$E#MKP8T z{+!zZRcOg~KhDM@zG@uY+*28uFZ`S;jDOL(gTJ?KqX|+~UKYq5`VLKkWd^wAMNUna zrVHQ%Y?R0o?GFKH6xt1Vf;U#Q8g7c>S3%MNz)Mry2;Q48kqxlHR3|~j9D;A07iAVA zf`WwvEx$Y$Oe0sM^+zEe0Ez!8 za3QlN5HsGwf-AEicxEq*!At-4=usyvjeCWx@iqeaLxku&dPh=o561%9neBO5UTYIZ zf7u_B7fwh(-_H#Gbf;{45=4s^*kJmRfd4QkVc%bPf*c?Zj z(?+}V)3i_g=1c+S=`qCH4^z=Ak9LXtNaoik-hYg90f%x!9^rP%p3>UZ1`QrV_V(`X zZZIud)xUE$1dp68F$Vvj6ucDc0lD6Hci$U9nTyj1>nfjX0($R` zK7W+`EV}#Pt1b~IqiowbHp91oMtc26H#0LcpeuCXgW=Dh|0u->HBC}e?mJPV*$IG( z)qc8m2X8DSDVU~$tDgm`gN=|5u25Q>Q#702{#~ChfR@G4Hw$VHICw7jN01E+;7^YD zVLEwQP;egaDeRcA)~qPqG!Gu&Xo+5UOblo57-&)$-)x4o0`FJ@wQ}q3LoPiQX%_7k z78a1o6uRr;$8!8DXp4fGd6rtdGxr+Z#rx!ZV%2fUSnCr2CdrXis>I1dQ2&QqX{ z=e0M80c!FfU}S?70oCgi(T=a9H+t`qsy2=ij80n6;yX zMKpgUmTlEbTIG*2KmlGNQ@9rInu;|5BjJHwFo4cm)gqtG{tbKYzPv0VczErk??-qm zuL=teCYA!e&J8a|i#`sbDUg|mtT4>0xOcp#+N;{yTnjU~aXMo5O&GxF5#&St7-ek7>|>;0&j)VoC`Ew(B15XN0ByMi^->&5?g?v&gw3bq@PH) z)yBU#w0M9@3;pE^z6z0f$ZLJXEBP(>q%qGD*`1bt#o{!@X%L_`9<0FQijsj2gG8~u zZvl@SZfEKhV|HluQ5(O#jRDrq{U9v>nj@wFic{ObW&OLFG(B5FSJU;7dn9C!0o*KH zWsiO*tLLYmyMa8lOko-E`hrPxT%19kiCl`tqXU8z6chx*q@4Mi-x^-hSqu+{m&SDz zF@-8@bM5NNV>_dwqHtcHJgM(KRk*B)fJ97c09V!u{@95lVQNKu*Tza{;9Nh<;^&LI zC9<~^oGGEfW*U%`hEFu0w&24VJ(F-^mPuFIFv7&dq%m5sA$$s0J*uD2+DalrG%+)D z9`1FtS|(;@o;Z8VW)WM6^fBU*kO?7MYizt2_b%4!pki)=KqnGVf1Wb7Tj|%Z1PYNc zXGAsA(9n>N5D@og|E7k9a?p$CBa?Xy-`_279%7Ol0j5V&)`H z!$!SuQyVaE5p+-ARm(sIWw!PA;UP%9P5N8MaIK@P4wjbV%93j)X0lbr$QFX`>EEcM zO^dbvfb?21;-+runLejpo1#5-4S5*(I$?e+iG2Z=D_^;P|9*mci9%89@#rE08@-w@R+%}~}N=uV>~=7suTX6d~V-SbjjD=MB!rH)r3LzUfj@ZkJ@jV>OKp#v z#2T^BBlvyD#@MJ3rPFfd?O>tT%I$_YU2P;{z#nUjaSBNa{iAyq&$HK13jUi5K!_C@#oXkc zsExyeUMNRF_w5qa|2#WC>)(xhteX2%p_SH}|I0@$tVX%_rJ0ttbEZ@wYmUhbH-+}T z1WQ#nq}R|!)~q{v9n9o#e?GIin2d6$M)pMX2H;#+N$u?N*U6uRDalaTl5hrK-zmlC z%|e)5VWVDUw_Nt%2f|o)zu5RrSQ_k6f#N<$p85&ZH8AUI{1wTio{^*E^rw=zBy(KF z=jx=r*dn;EIimYPHzjQyr-On!m%Z61+vk@)srFDP})h3`I`?Jpp2D zGjx5{Ny}<07~|pi1Y2Yc*|vROcu>kPv9Mhg-tI*GzdyHa2`8q-3gTl+QB z_DL|cP7e2(`x$JFz!7x%WOefIU%!4d`HKwaq~&v@%pEnUw>$o$%Q8iypf9ay^@dzh zSKn3CW|C6{Z3Qg^NzbSVNxnn{ZQejk#lsL)b@g0goZtnItJZ!?Pd8R&Hq?mpMsft% z?Wn(e`UAUNTn!kmY6LK#`7?OdvnNlT($cKH&)f0DnjoM{mJL1@ZT?~|zC_TUEvzRM zl|f)b2aZm)NjIFOWdj{tt!PY1&(STEA1{1(i+vq*UIhk+v|@9=HA2a;>Q?6y8T+lTxHiUS&Se8$oJgNR>2^%HNrYBV} zIt3Exl;3PA8289i{Nq_6f~G9bJysKbFC69?5LscoC1|Fkq=Z2!g`11GcjENQ zwhL2pEArJaIsO;}AZ*%Vx81-2Htt`PAf5jebjeiwlt@~;;8~Z`s2p&%&@#HYRDg|K zw1xdw6AU&AcE0%FNtbF&&u$3Ct@+Wmye!ncOZH$}2KH=n-L9RX!5*{udqQdP&}K?8 z#!hepxwbMtKB%DN%6F|1QHUlge1U7Ilx=XgXD>%V#6;AI|_An#Zb2FueRUll!jJy`99a-0od9QPiEyX#%AaA zZdFdEh&%D_)Q_ye^64^#h(hjpfd=4-`?KAU3jFwtc%IyVQ6+YobCP3*oGFbwdG2*Q z@qh=+CcAQxzgK{5hy*6JGI%2%r2nG2=SZPCoOAHhM<2IDgCpD=9vy9|WEk`$cIHv~ z8^YIYXS{sDov!V zDWQtyclezCxP)YNAx-3#m^e%orUs&FxIdIu-E_kTk9NQN1hJJ|)bC3+gG(w zqhp_ z;wlzwu*|qP3eSq76R)q(6>upqWMeaj#8@vK_Bxj%69sgBA={b zacrtaoxpfQyA97(;~`}2B7LA2LBY_lgT`@u>e z`fWRj$M89eT>`j*loxB0Oq8h(AgyQXJW=u}4||pq)4PmZhiE-!k zCwPe3aExZYp?YqN|BXcU1r(1KF{lkv51+K*SO7jUsy8f_vrpljX=#Ys2eEDa1xF$T z;F_?H4}Hb|?0TMm-qp0LUOM3;ds=KF<^JFx*_PL0a@v}9Jv8M3N%x*kvTWsU=)n}} zxf*1(&+QC=e=i%qdwjU5v8dqC;XlCW>bbj33>(*NYGaL}h3j63b0K${N(xmQ?swt; zGFQA?;U0R`a<7M^${+ML%L(oGJGGJrZ;DRiLX3JE{-A8Y5e=E@7E|Mw?MJt(P67{) z@JD{wsz@%`q5MlboMqw%t75c?p|bt~^eJsHiVnZkc=-#G=oAf9#ip*M?-!BceJDHF zkO!-^^lG0};|Mu6_~4vqx|e3ER%sozgyO~WZV_ujcGjV1mz5-f;|~8wTPxn~Alh&sS18rO^1QYg|1w3^o>kz4NsZMo|S6M?!a zllr|j`4GqoZ44!)cMaXR*1Q?RbDo$Ku{64xeDhDRj_SL0F1I}4|7!2b1F1~kwwg*Z z6Gf?}l$|I=*$Kl@gzQPEV=p@)qE$GQNMs!;N@M99hHNRN#X3@!V@qV~jIwWWzUwtJ z-z#Ipf)=$+x->iR4jl*J}Nz51;cX$P2?IllOt5$H}tFH2}%OJbqkYOL3Ws)1ATy8v^ zU^%HF?$hxYas@p_F$W{4N2S>bVcN(JA9dR>JM*}so?!*@z{)o~#&GB6#s8#lXE=<0 zn2lC4FkL|5BO`R4T~eH4d6|Dl_1d4gY!p=0)Xvw))uQRYFey|VQ`(rA(r@k^pCw|* z(5E|_Yh<8V3q*&E)o!mUv2}g*)-p*%23uRx?X=P6EEL}pmIfWv>MFr)M%1anMlD5i zsM=fhoi3U~(G_a8IIHRK`l#fR#{R!~hOPO&$w?R)37<&LLuFXpjoCeVLGJkhrwp=e zm+(zy(@UQfv3B#2Ezm>Y+I4dw(aDHfo#p364U9NVk~*#{_H*EzPowjhM$yUX(Px{$ zhHnA;{ps0K{RJW#`*=m&wKK=MY?#jm2X7H;a_CU1VrIdu?X9a`W)K4LtP-`p6*}F= zW%Qe(S_fz4T+?G}h4TD@f*yIZ)*9H!`wbF!OMLmY!Cuixz(!h659sJzzj1};&n#$j z6~9wxFB90F|7(__^hLc-Ab2AE3EQE%q(!anJ{03@lB0!`j)HAn`P%?&Tk$n6f!xj6 zUZ%0Hm)*eH{<6j$gWt1Tl#ChmG|zr?yS#3mkO{#VZGD%xQRkdwGa^+r+(7ZZ7jt^T zOjArq@}=!}rL{s9i#af}iI9^O_MYdoxoHMmmZCvrUlRYlofiWu^z9Bp;++3fGUasN z#^k%#D1pEP6v_>&3Uahg$gV4yk>SG|_p+-3*d0+%%d!1ONcL6*3;N3dgCB028KbyNjSHEuevKQ&3* zVUTud{Bf_Xq=jsqgt%g-SUNNS{DrnLj0owvYvIBAaz=~jGnC`qB5pA6GnjS9qo*Aj+xiA2cU))DFUGzQl=+ej>U46QruEwXNg0TfkM4d5y6v+eHJ3Y#9Ct+o z$RlTivb+dCJ<193H}B=-J=u`l+OqexVFL|Kml7$KlO(Ey&O)Z4cDxM(tuk*vH#Rm3 z$ULJ-;1jvpXcFc>?gQ}ZX!qesPRx<=pul|R;ap_P($CNi^G+AD+U>c}FIMZiHv~+Z zyTah1*)pEk>A9M*Wz{ZRhrS5nkkif-LWwu)kG;`mJaD2tAl(=`KB3Pb5gP5#)+Ws3 z!nPzmCORw=cbRz%9F%=?m8lY4i1)8~8z@WaG((yY zy7x%z1k8v|gM&kL@yW4cyTsDD0N>o$nHU=@OxI7jRPfE?8XCmBaq()t9jD^j2_)&w z$wzI`-%`1-)!6mC3JsA&;^s7n-Dh(HlMrKPH1QZx6D3bP^=<#<=bIGE=TB`!37YlQ zZDIO1?Zyl+0yp|z8p+bTKli1iYG*>Gy}$LL$f2)u{c|e~yw;DLbrab(fmt@Rpba}s z=RWvhnjz^;N3yJ(-ptztj;cV(^H4VU22_(coR99Aui}WEHbQQ&Xo~)@b?S=DIh)}u zS68)QVM!by|=Ly?3(Qgt1#D ze0`-WueXE(qF3i1(FldasGC3DWII65bE?d7+e)ZVU zPq|l3KjBzDc%$IN^=)mr8Q?^nl{WqwoVdV*9y28YXenx%V?d>Fkp;) z2-}O;tBBa!8MZv-i{yELzkNiT(YW#Iv(%+!HcH*5?MCLTMv{LJ7ES-Zr_D8r&iliH%nbT<7C;bx+`9i|!p>eG#A}s5!b-iL(gjxf(fgLg^Ovp(t)p=6kn-N+nuN+o$0&XP4;R zR&+5^X*6`F1&k=%m5a{Ic+O`W>@PwAeXs{a+YL3Rv2}NP7Gi=WGeg(QM2h*x~8!30Gx@B)6R74 znM`XjGBQdBG>tTpC+rBOODzG+wbIftQhn80V@i4!_ETFC4v7Xg&d-FTO15XMV=mPzPqNRc@JgZ10-IPE`H2Ed5t;s+u)_QgTB0MK_V2qz~rnS-};)}9l z(4e|NDF1-s0~uq2#3pE^0|$!UU10A`l^V+eqUY)5CHe5((vBTaANB&fEk)rO_@EI0 z3L=ytFt$LXgT)0-*Xp?EAR*J+H}CCYBxV9L$T@&#z(I)KUpON@M4k^i>dHpxK6nN+ zia$T4=T+!aI|e52$Y)V7D%;-zBt(nGkOjtVD`wKl4vic)#$ zcsAl<6YBFKUX>4rw@339Nld*h1Rk#Y#01+Z@Jb0Bpoa|}7i-dL;sDoI|LKr+Wy{+G z+bKFkO}GJ>+fT4s9rp6^DafLkYHKES?G(`k_+V!))ozI!hhVz8W{5EG9qzDr6pJJ- zGJ6QA`YVo7V8K8EI)N2I@>qz`hQJ&hBn9Y&AYQi6N3aJ4?;}-a?|l(P>RsU8tv>Hv z_3Bl;oJZQqcOXOCVujbe+EBuhq;#5=%{O~{v(BxU6Kw^*;wr#y0PbNw&f01kXXT&C z=*^9h5~C4E!?~F)d3kw4`be_j3Pj!L;_z@$%HC>YB6bSi7xjR;5F6kG z$$*V4eu900aCmi$VYju6M|~iU;UgLvS(dn=TOdGs=lRL|E7z5{w39+^rKYNCSb}?a zYLn;Kz~9lqW(HRh@MPMPwoR=d8i7dO3AqzPk?}3VZfY+hZ3)>W+?%RmN4rmW;f-ni zr=CQVFB}vA+jV^U*hA^coPadnc8`$Ki@U^E>BAji6y^@Q^h4G^K{#VVm%VeAz@Sy>KwPq~MaMR~y5(h_`2aFT!P(Uo}nr)*%c*t$o5Ej+VTn z&KE^$FeSBo@n){ULmiJ^7l5Y~n3brQMiF`SrrPDur<%rkdUYmdmUcmWIx(Vr;yDu& z*b;*4^9v=aV;nM;ryIi?8XPfN!n1Niwz5UV#fe_?78m<$A(J(5N)wo)?jf6FLwll$Ba z4)N`s;8#O8YOsZfBWwaUM~gM*$e;n5lVB>E3-7Qr^J@_4Ti&o5Yc!w?*gl%b3GVL^&zf!GXYgY&{m!nIc!Pyww}Q}!;1NKEYL znIXb{PhL%|fDG^NySu1cnni9TPy%tQo*l-0H0#u$YzY~=^*3W)vu(67(+=y8Ci+US zRT56aEn;vAAn-2V`_`p?)JIlSvZh#c=rW4(8e=LNL{7Qj1<;SDMg0E?7W(SsQ(Mf zjV}L>*?`e;;%cW-c0XuRIu+3?@!Mc^h`t-4n`ZrX!!H;?Sd$XRIqddt1K`^_(`-Hc zer7<0^e_*dbvV$0FtmpKI|;@MfWt=sXKQ^VlSG-#qT{GN!5rRJW`aHk;o7dbW3&Zi>`zF@$aB~lQTTm$w?EoxH;fIJWV^Qu|ECqaSBR&Z| zi+DcbYy~nODGhpOh1pCOUV0HQxun4OZlx4LV+i>Lg=?-T-QEtjj5ujSm32JeoJP>hC%XZeQ~&7)stC8_b3T^y{OG2aRG&895)Was z6i*KC56i%G1`!=fHZOdjccDj2AuS$s%~9E-BA=~AE$NRyOA9A^5T85fMl@c1yH({2 zG~VGdJ9L`1R>~O(?EF=e`eL~N388?42nD-D)qL41dB@#UchM#9PIaJ+gnXueMA%-f z4~DvAs-PrnE)-W?zZgQks1vp!o^Pzt9#zqi0++U+`x;74LQ3U{fJbD<7bOI^-Hz%8 zwtH@nN3BpfR?7<=JFhw#lkIDwJ&x$~;bd3ge*mN3w$25xlq$gEKU}$EI0Pbnrfq(v zU3TMFF>+NU@4#o)tPvEJFa*tL{rLt#@RT8)5gfx+S+6c$b%7r|<}D;fRZAvDIgPaK zY;C>a|2KJj^T-Jain5g;^&8wQdTKjPu6eTcdCSb|W#Q*KgI&?_hAQ;qcykxJGnfs# z{XBYu$-W$1zm4<$cla%kYn*@V2Yo-1<#BYDIoJ^$K;lj=YE+T$_O2tO8R#_CmvX;9 z7qB#f%2XEcRNb~4$^CHA;*c&~wi&TS@f>Lll71fi{?SUK2T2-KR`JgKCLQ8+x6dy= z1gAR`Si>xvG%+!(@s>IMgHJryR`O=~H6D?K>ZBxc1uCB(Bvc>&AMK0>xC0)%*F9OL zflH`tB7k2i%Mw;xeRZnAGD1Lvq*bzlTSCNu@q{9qJ+T)5pK}D@K-BU7-XI!6{~bXr z&wuCO|HmRZedT&23(NA!0~%_^|E}o&{GO;uC5(7}9A$-+FG>|D+zrF#qh7iXcojTc zvP~K2*E&dUzqe{B=QQCRGn8A7u>Nd9_R4NTUaE;C!|K_R)LHkXX)P1`%D(U{<=ig? zG3T^b=4Z2fvcBibd#jgn5<=yQeYr1@p8iEa=QHSY?VM^9cKedkeQ7P>y*fJS#7n;t z^SwqxofdI}+*{5=!YVNk$;b0649>|7!B-QIr$nJn^&Mf)SQK5>BU%LP6qH+r{v1L~ z3)Sgw%1_P*66}?A&42$L@#lA%_)JOC@*n?r;h*mj^?CHoUoY|7UnGglW%u^!uRlVS zI=1D{cmMHDzbFLXaDyiz|GXpU`;tHXd4qqv^Km$0msP|nmEUE?f4Q$ij5Dl%`S0&P zS`o5Gvhim^_xktCog?=v{{HxX`O9a1L_nU~7xe%6a+W3Bid|m~+_`f+Xoo4=un7Wk literal 0 HcmV?d00001 diff --git a/docs/.gitbook/assets/image (3).png b/docs/.gitbook/assets/image (3).png new file mode 100644 index 0000000000000000000000000000000000000000..2442410112fc9413bf4969dddf296cb06f201d3a GIT binary patch literal 17434 zcmeIa2{e@b|1fMzX%&SQTO`R|b_S)0P*L_JOJWSkzLUEm%Y>rrcZ#f&eK%8CLKzBE zGq&uE!C>se^Bu*wfB$EB&v~Ecyze>Bede5To$GsDpYLa1zb5Ruj{1SU$M@3E&>Xm` zp>~sohJK8OhBk5!Be?T$mkB=&4G+y#HC25tT0&3wi$S&T;+A$a$5a@f(b`b{ID*eJ z+C=cNLB-PsGX5pLkrkX!R;n&zVN;gDoqWErRMg*_KfY4p#{JK8m(c zPwUp63%dSBbH^(G*r!ZenmudhcvL+vQUx|+NXt|?89@_lN6WNg9a4a0+Loq9+i5$7 zJsNGv`>29-YiI5 zJVmMZ+Jb`^%pP0`3Sy`UvCL(n61)YXbq5pVnV+R;weA7mp3D(a!P~mG1)`^ds*ftW z=~&#Fj?vPw%pZJMAh1o{HV6fF>b)@)m4~8oeBirRRlJLV%Dz7!+(zlB_c#G>esf4+ zPeTIL1hzm-nNshi(F5LWa>!@*2SZjWL$^TuKV$NWWz$cYKEJtYv~K2iOC?N?RIseT z`YP)&s??ah01y2|p!RVaYjfyVb=M#P%fs=yeF3Y4?~8qb8}sYr)l4|Ivh-=ig|`gk zf^{W~sA`}J!|l^l!JXn!RX%TRtktF!GZ%vjTwfm5<%aml7G?O~-XN6qrMx~w#Ng*! z#E^=CWm2WKr&~+qVZhzeTdeck0ZYG5|NQz!h8wv$Rh5Z`tWKW3 z3LhG60kM?EN+P}ME8)xjgd?Ob_Xj;*YXCAka(ex{u5wKWCN2@d_cIbl z9ITovne^f!bo;QQ^cB)Krh99Ar|i1e;Xm^YQ`|GO6ifB|;vp*u_z~-3Ic%+aU-59% zn*4$IOOM~~r0MibQh!9!^CaA&7R&s*bbjSj{bOM^qT!=1kCZGLtrOGA%Dyzp6FhsY zd%G8VFah;SEPcis-E$SQ>8?cHOTwu*=@Io)=7<%Hhc+z!-VG1_NrJo{V>|nIk6^1x zgX$Zi+ia($+ee2EYhB>2mO^N^@kQtAX}1+kiq$R+$VtF$iSc@-6|MY2&GrZz^^dEa z1{O8BtOjp!Sppvkg+CxI@Wy+({B?4d{Y%ysK82Vf0zO}F2`wC;e|#-t(81Fex^i>% z_1cA}3zwA$3+)XvEdlyS$J9pq;0q?9>}-dr9^rjD%JtNFB~uws%`|~flUQVep*o#$ zreHe%6*SX*?Wg!kDPpccDqg+-W`@wVjA!YubR3^>)&7Q?#UEX&V5sfg#f<#RA`|;q zUOCmSr@gV_EcSVZ9{0^JFqxIL3pj~h!Gvu|(**CLsG8Bw3ML=^aFTB)Yyc-P<{IV+ zKs(?>JNV^de#w75jgrcWfUmlKKM1$+F_S`{|XrY{40&xzzv9*8F@h1Rd(YTgr1A85D5$6!}#rYa6GIv5mk!d|&D-6voW$n}*q+Dm?) z#TVm|XA8#fZ53GDcZKT-X2KELdk2Hr(JB&(?Lhdei-pvxL zSnn7#U7e^Tg#=-{$_YvVN)Q&0@5^-cvj_iHKe|Khi-FrdKld`>uZBeg-v_6Sky>@g ztz3WY>K)y);jYw?Xf-j*Y3A7E17) z7-DCzM#G@fqcwJ?_KkAi{-i@%4zKtiXzP)KF8{#z+9upOr|8dJLS+IaB(`}G?)^4} zN12?HM_Br?i1J5kF=ttZbIZu?3wgs|OPuiw?T=Ud$y&IAlG?Ij^Qo#Z**k1IY|#@$ zs?=9^I@LV1URU2<>td)0n64ya?!q`rzDTPk117)`W^>3-b8&MkgD$Yu0r&Yc0gr0P zrxz*~v|mWbAOfzxUzq+e(#+vf%RoMO*kA{L*+6WzyK4p4Y#S_HxRy2$-x8nUQ&L0b zS}>(Mc@@^n#Z~sW5npjOt{(Ypb%{8bbIYMet8Bo|jDSo@gj(nyDa%p6i0s-hlu;Jk zbN;(@uwpISz@y7M1UM-OoN8QEbuQ^0gVJ$kgHPwf5~>a;#r#<7!xI+K=@D;%relcE zF;_j9x-Yxxs_IAm*A@qGRkRMPSO#S8WoB;XFWPsW=@2%8Q*jMzne0WJgORUy#5==H z{=DiOD%93N3r~2Hd=z}1*Ic;#F~J~(k2aEDKHq9|LkRvxDt68g(pj58Q|5kI(hQ+E zy1{{-9qfxA-cW|mjygR;TyY8aUbi6qfHZ#@lACZ~U8;n29K|(Q2;fWnD-?0;aFhMm z0ADgfWh(xW*PX*VM5M6yw=U7iy`V`o>UL#G4r@1l$YtBu`Kw$q1$Ry93QJ!vHiclU z*=C)r@e(huH{4iVF{FskG8oKk4waW2Z-osMkeN@FY2)u%kPnxe_Cg+i=r&_4fuJ|e zT;K1xU+$>@HJPx~QF*wIT(kLJq;Z5VItrP^;Ht-K`&N*zrG>v(IK}^XlO2C{^w^b; zTyUbi(=lg+Cg!5Utl4N8I$b2%1~WUjEJu!#9Pn5&zEW+jmGp_fEF3oI4K>5XhuQKE z7py0!;tUWQM>QHNEbl;ekhu!T+}+Z8D)_6K|4#5_{%mb6YWR{0G6+=*U~);OC*zovCvV3fMn|E_ z9i^_ZNBaI&k-M@GDX)2E2lH-%DO+UY_Aq zLZ`O37e5QkL8Rw~c(65i>WuZ9S5#Y>5&&t-?SpSi7(77y=o6XB4&8R_s5bz;GB@=P z3{j_fR8u3O@3L$&y*v?QDTZ>-yI6i`Qy5pCbBdaz#sc(uzH#nimHn5%|Jx*}cxKGj za*8xw7xZDQ5oPmUJa${3XoCHBUC~vWw~(CT@bce3dBXj<&bH=LE{f=0y=l$2q*imk z_4_>%=wYC>6t!fcsA2FeU?4)#N2&HfSx2n{-6KbKZY%%qi`%>S$8yweZ>#U`i`9J^ zZHqFu>9@^j^TP1xvor*Y5w*hGBw$hu7*U_$0jlXxHnzN_@~~FZW*1fWDeM0tyd{+X zQyji;61 zuPN;D3%K%-3!kt>!5?`CCo`-&Uy4~D)p0);N7d*}g;~<=?|BS>N&FXJd;KbSRpIW} z{;4!Us8%|yvWiAY)_*D<&;8xOZ*ko{y#Y@+)As&TrI;yiRMqrIP$Sl+=}ZZVyC+uR zaMR`V`j`JnX)9(sUO4cFT{ii^<1ewc9t7^!> z3M!pDx#cC)r@XM{W`#mY@Q6~w56=uUZByu99iUrv4<7u4f$s%%jl-CFM1t*mev9~JYHc0uwe08AtuwcA)g63o zK$R~SX!`u_>;-Bp-8xI%S{L~1-(9@ahRcZm*MjJJivnFJ@{z!dRi-4J%u%nIXBuyI zh>>BP^I4zRuJZg0P@Uq(an8t^vFXgzfXe&_CeuPQl&%pOG=p?40+wwB!o=XIpgr}w zvGwcI6EmE+&o34?yB(c7_)*!8qw^XAnk+6|bI!HP8Mw7``A3j}8|&j`eF6B=NMET* z0K=r4|<&%gyWMyS^%>|>j^q2Nk9#8F*^wz_R>O{^9 zCnuz-3~t=3c;=MMmkJM&Y;#q(+Em^(;;#>(Om`+fUE7OXnaGKMRR_oeCx(=yv5^g?g}xk6EQoHGX-(@tx@-ZvX-1r>1L|v0Cg)4?pnuJpau6N{!IV^ z!C=FgPB_H0QhtU`X*($%yJWeg@7JoOF&$g}Z2xLGuBdzrH4)T!eI8<&%k$A85osAiv`) z3zJ`{{FqAWTak!d*`sZ&x~=P$iov4{<6ZRRn&$92;n+}DDs*2!6(1S+Ggc$M1^=oSl*Ma@&klxaoiz zM9aSTYv^o#%3aB$Oosd^#B|$=R{c}Q@ts48G5yuqCtFU9uT2)A&71Ej>xZhztrtd> zzfPJf6p8-aS?6-L02(_}>14*{A1R++>t&AP#Hm#kX}7OytQA=hIq;Z(%ZXyI91;hz z{RnfXhwG*L{1```hJEyuoP`kUjv3w(u1p1eBJtsgs`;MYa64#;OW$Q66_QKn^2hOguMZlF3~q zh!yhd>4r`#hdu-WZp!8UMVoXfDocK()OToY661B}I-Kf8=Xu3#9-7CFRJ@U8eVZ^} z5_X&aPV{7bpXsR+!!|M8Ol=*{!`e+vN(IJB^0b(II%MJ*)VYh4T)E|KxE)?A7?|fP zEE=+sb7i;&KSeAVtME@!I@FGZW-WOJjkt9TX~4Uev)%kM-1cWw1v$DU9x@dA;PnAl zZkA%g_%HW`od0Bxtm8v20~nt-ZpDDL6ttRdjh386%;ulAaUJep@D++i*WDB4&BdWR z56`8;jcya1g>#zbO_d7FYy7K3((@EnVn{F-$1*e~x?8SC`8rWKq|+qhaO( z%6g>X!ivAd;~NLR6%XmL&Sy#6RFvq-Au=0%&iRsWtzj4C8uQ2+)2_{7YKoAhd(*9< z(_hh?>?Sx@1gWquq+X?m8tB_1QO}rsOufI_41MEXmOsN=Ev?#@Z~?<#O{Pf~HPLK? zn)qr9yY|&v>qxmMpDAc1;H~2V?d76#G5)1)jTiG4+tEdRJ!)qT@NOp$;x8#M$wwEZ zqk9tgJ2zZwZeOX{Q+&BYeuTb1Ve1`b)%&$+A z8~PU5()G{AYkOC^vP#ABZ(_3IDm?7egg2Im);(*(-tsyG_?YWD6d8ZuZi?`W^F>XU z?CgB2!gP*P6K_lLU|!bw&kr3W%iQ)oFhiJ=^O>AYnNkw0lFJhk4Lta0vu%f@Uvbng zzR1i-7($=Mw6_}TR++<`t8(*-dQ+4-dkvK#%%KZcaOu6S&QIWnjyw4BU9~vJr}RCC z%v+6ZTLrhe+LWmcm3K4&v;odqZmZfQCmicQVr~a<89bmPs&uf+9jg&tnk*YNy`Qho1LRU-9_>Bhr z)q%axT=td$1UbsBn0G`y<_P!kYdt=kI2+)&6%jDgStH{(e|zq9XW|Ey(sL7E)-T5- z8ARd;qTZZ@@H+X4>{Z^?V_bKRz_60G-ks-6)7M0X)nZa$O$4k@W>li1^Smw81owcO zB#&M1v*2#u=}h+v+<(VFz7ZDg!xRB#<=nz6`0eqUiYKUc4oiyp^S=i}p_*l_z+1Q*m4a zy%}DYg1y0q!BWWOK&m{)K(iNeapWzWK^oLJRpIF4YFinf_)9>}r2(s>bbMfGclbKh z*xHi8(qCV+xS1R&rvTVfrM7#emg0b%$945W3|GL?AvQAAd!;FWr9H3WAuK;ACm2K_ zQ0A>P9Ayc}IS>|pYtO#|1KRWd8-an8zh?iJ8b~sSI0u~mwLYLcC7k_3;^qY6xz?p$*pT&D8|8e)rvOQzzATWNCyk0p)lNAoUqI)*4g9tgJ!!v6f9?fUo9(xcAJID*btFz2qi zQBIem+JC~x*bNlT`^K8r+F-Alb5l{<2;K)C|CdL_K~|po%(TR19tF+wTM@7f;86YI z%>xiD@0)TR2mseDX1vvM{GpAaxi4e93vjbv2%skfIkCmqs56!{Ue*2?^$dF=czGc3f>ct> zAA!xBvZPtM_=^butYl$Mt-^91zw`&c9~J=M@3Brl3joL>h1(LxL16%}KRI)pVNbO^ zSkj(0Pu}81HJdn%AI$0?fat@_oa*~yA9kx6`Ej+MJnptqvAjCyCSPx%plqH<$vnXL zcVNyxX?xf^cBOxCe10|%J=grD`_FJ7colGH(dG8OKO6#iH8)zL&Dw0UN8hck;#UhV zGTzE#dUj3;{tI|b%NE#kC+BI0c>DddbRht6$mG2q{y$j0LB-(&dwNrhxd6xSD+B5R zx*WEW7B}9R;m4F7d~bdJw>?1GtU%hhU)wpzp5P4=jc?YkHtm6nj|ZaMV3zkM@d|#z zRBZxK{vRz0_gp#Fve}TI;Q+fzOgU`nxS~E76=;@~DZ9nLO9FyJ(+w2SZ_U3zh1WqL zhkKHz4P=IuKrO1eMUHQ)h3@Znr$_T_K;GyG)U!;}2)-r$#|#scmjf>;+&0+5EL#oC zZprK&{T9k|zn7KKtypmDhn|XGub#ort<{6OHea5;zuTULmhRKuQ1quMAI?85#4Bu} z6h1;`52sBt?3owhfh<A)$+Chkq=!$n`k$oa^b?cHjf>+}uaj(DPIH}#>~h4zf!89$1H zQey1&`k`=b5lbE@S~fu!Ugb<69m9T_)&2bgMT_82z<_Hi(I^KjzE2FIO^a2g1=e2m z=ts_8w$vIKCSWv(z3I~6YzK-6-b!9$Vg}Cdyhk{?H<1A-tAB%C9?(aW6L?E!ona3L z%@mszLdei>p_g9;*nv<4meP}KWEH1zn?0if93A?H2*lT!g$Ib>SZWC+^{Hawppjo@ z1-`S~B(x^mlK9gX%z=r^+2Rh?KnN#w0m)6^%p*MtVqLaHu$^C*5LlcP*e2MCcOm=X z7Qs0m-$hbPMv+QzM9;qj|F0#y8?wAXY~)6-Zn()Sk-><;PRnoZtBe>fwWi=eC2Qzl zKN2~3xv=b~6|thqGmP=DS47a#nsi7tWuRrLiK3Zd9K~)AMwT`QBdX_G#V$S!sQqcu z`;q3pkQ&4-WvP1Rad^46eK#2#*JaZz4OIyQe)p1fK{G{a20+#)lAb_co9MhomztP2 zz8I^?;0OhW3fIS{ksATGF8jS_c5m$K2$CudgJs5Dz9XU1O0YQU)bNX;xxfd*^QO>< zmgxP<>_|c8$s#)?1}LpaLuF)guHQMvEzh&95f#69rdG+%FMLe0>RXx$ZF~| z@&4auWz3$FCn_Rk2=6~iGlio6RF|8Ek*E{da;}r1rtI8#@0&=Q_v{9rt$mqRX4eDH zY0Vk_6u}BEnNraG`XbEnWLbP&*`wFom19L{h;Q$-bC&L*UH*J8!qKCTb@7yVS`JK2 zM%KGeL6veaIb~pBz9g#@AM?TdV36~xG-vxoW2A7OI-gz1b+z6}EoOA2z{ed(PUJ2Od84`uS!ljrzK4KTRh3k=@t4 zu$jF4w8sAP0W@cw1L45~aOL$SNw1bz66LVAL)sBsUFqgTX@#g5c>V0|Wa7)Y!jv(u zu4H#V(DfF>cXAi!D1W=Kn=PdL-eNqMihYX{mciEAGAEFm2(uC^Lsbc`v!iHTOrrX+ zUFN_wi=AxnyBt{`D-t%d!>Wt6?K+;=0uLBvLEa@9w?b1l3QCTSlejcOCO8I zSbdF?d>tiSkze4IeV`<;!jE98=Z|e2E%b``@yd3@+AI)yGD^*ND%;18`I)VlQfu-) z2j46H=H0!1{ohrAi;EDQ@%wg?^p|H&0Oms{Hq9QI>@68ZYq5XtU}Nv88)(|=ifHbj zaCc};flhJOLAF3}nts$tH^%KvZo0G7Y_A_7l*Gp9|?Pq9D%72w$iKXH|2ttL>O}U8;1Y9VIFtGI-pCBJ4ZK}8|`YOcsD7s zQLsov1{3KsXZmcoewn8XK1>XDP`I3nmM#-5aPk#>w*WKZL+X5r8i>uHWOXPVI`o;= zXkDMOF<g(`69`^FY%T87EysCp<>Hb=JW@Xw#MeA7~?u*PNeDD^rn;#wbFXmZz zj5`Mrt3C@|s}NumJ{=Duk@k8fepa_S?&2mCcUP`JyB3`>Fk$>9>cNB6w?0-i6S1UG zYcuK2C{t&x2g&%6ig()Yp=NnTPAA|lQUUhfW}bDU6@7j8tG`Q8lAI+FU}qTZRuabI zV#iuk9ALfrtP|3DEw1LAYsDcQB_GUTiyf9?IB}Vv9Cth4fdcc+6rr;G6KLepK)78_ zX-X0|AyiU(b??wR8)wyFW?%-(#{&ndTH*r)ja!Vgn4w=B2MHJTl#QNG7fJWlYhoHE zjc*VPfPUZ$7oK{W$>1s%RH2S~IP;cwbN-Fh%}I-ji{SMw$dbiCp2^Fe!E1UTp7!9*dG<$y-I2v4{=4m_ERPQaBa=fnd zKpdkn=HNmX_BuWvmQA{Q%!C^%N38C>Y1|TuV5vhYJym@(E=^az=oLK)D>d{pmo0{_ zoo$@biYdQ?Kmwrny}I`Uf~b9U6q7^bX8(Y z5j9RYb7YHo+1J1VfjiD53#6!%^Lgy}<{6G^khs-C4y7n%9<<3|gxRqh*4oKRh2{)* z2H#c?tn_#vPyet&x}y@BII+l96p;rnN%G?&KUJ$SAI~1XTqt@&H?M^|d|WGNX6SDa z$3Olu9n?Ls$|mhw^6unfV$7;3GLl59nk>-i)jo>vM0N)V<(r=En0rS>G+^mZ#i%%_@5!IH?J1qR+kSH{~eR)+8uyy+VroIX@SNlxl^e zV~HI+`Ef?OjR|RE*blwNCmec8w4va?UC{nqVP5+Ou!m>|N}Eci;VCAVd-&1yUggPm zchZAE`XP`@%#8Hkmsn%0AGK=GQrh}%G-!#(WnAWbVQtorS~dl!5_B|S;`@TNy-~gi zKEVN+@7ZKsHzA&|((evt1(m(c(%#*U5uCRv#bz))NDO-WC(U+CLPIk0I)|IQV3zv3 zjoYK-%@lr^KriL_=JzTgLe&b*5e>yiIt$#;a;3`DG<{=HS(h8|O7HWw(V70@U2=ag z!bf5G=El}OH52Un0)sC2Tmb|5epKZ z&zp%pTb*=&DL$(DM|fx{CLfW^$iMe>1{U6f9+Q%8kR}wf6{Y(xLz1zOmK<(|x)p*x z4kwy}nEMo7PWGG5TE|4y;N`8BEBd&Pg#SMH<#_}&DL%Za6bRD`n>u}KsL9KonQd6~ zKyeqNFO*;o{bbrBA^RoRY;Di=>Xbw)qqUDN3gS_v^=nJKBMl>IZA%tnVhF&+XUrz~bk&${uWKy3J_V_yy(8Pp`2KICM&>(|dQ9b1ky$vh4n zjaduTy`gJeHxD64^SMqA*jjx{J9i;breFwERHHKiZR}GZ;mAbA=V^IGCC5K z{Ra1{AG`~#%4B@u&chLsv!;yt*($Hn>b=ZCpI_18IJVgnk!6ozs_!3XDPe; zYk|%h{&hE`IjXd-lhh6AGOM~63!9S_c^0>i`0@2BD6qdjnV9STlkk+kYiRc3R&xyF zU@Z{h8Wg&(?Sor!IF(Q!JyP#;*z^2ygscUQv&E|Bnkn>J>x0+p#unNGc=^NJ={*u+ zR?g(MG=d-OQdO?MnX+RZ>8*Cs2}zG@^}mb_WYtpBb=yY;`KB~9QU&cMOgl>L>oE3y zeavs9Da3pI1&ZARt@G~JZ|S@i3yBk%!>V3&Vhu@rn{_ZCqbuj?gSc1jogSmr##cD^ zjFb{zemCxspvl2^NKVX2%b{njXW|SJCsqh^97non9i66$iopXbp@wf-F%(D@RyJ@Mq>FnNZhCEiNACeh9TC#amU@F~c!tQ5I$uElK( z7U?=3KHTBC3`VyKGf2aN1PU5Ux@WNeX*xUu4;(<%zG*(Rp@Y3p<+#C_|4x#G8xcC zJvls?pUQBXLL`?3Lgc!ix!he50dded^@GWlC;_h?#RvM5%|FeQnQ}oK@?bBoAAMon zWkr_`8Voz@sPv>BQY`8zkhc?PZam!aRQ%%`I7%1S3z+ZrQ4|I9Hd9RuoS zQ*3r(A|R@6qh!xf!bw5t%+9Ag*ikz+`xhm@|4+Cyt^}0=KNQeJ`CWj2h4`m~zc6e|+&9A0Q zyRIl7Bv}Z1?5~C}avguJPd#G$XD0C<2j3~a{7>8d-wR4){-1F}{OrsI?Sl8U{sO4X@eE8o0r3@a9 literal 0 HcmV?d00001 From d3ba7d83643d06f46b7583a7f5087fae0f818127 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Wed, 29 Apr 2020 20:47:14 +0800 Subject: [PATCH 132/176] Fix typos in Docker Compose configuration (#661) --- infra/docker-compose/docker-compose.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/infra/docker-compose/docker-compose.yml b/infra/docker-compose/docker-compose.yml index 38234cff22d..b44212d0d32 100644 --- a/infra/docker-compose/docker-compose.yml +++ b/infra/docker-compose/docker-compose.yml @@ -19,7 +19,7 @@ services: - java - -jar - /opt/feast/feast-core.jar - - --spring.config.location=classpath:/application.yml,file:/etc/feast/application.yaml + - --spring.config.location=classpath:/application.yml,file:/etc/feast/application.yml online-serving: image: ${FEAST_SERVING_IMAGE}:${FEAST_VERSION} @@ -110,4 +110,4 @@ services: environment: POSTGRES_PASSWORD: password ports: - - "5432:5342" \ No newline at end of file + - "5432:5432" \ No newline at end of file From 65c27896ffeb16edf9dbfddbc6893a92e4ee9c2e Mon Sep 17 00:00:00 2001 From: David Heryanto Date: Wed, 29 Apr 2020 21:51:14 +0800 Subject: [PATCH 133/176] Fix config validation for feast.jobs.metrics.host (#662) * Update config validation for feast.jobs.metrics.host `host` will be used as the parameter to connect to a metrics server e.g StatsD Server. It's not a URL but rather an IP address or hostname. For example StatsD UDP server expects a DNS hostname or IP address and not a URL such as http://10.23.40.1. The UDP server does not understand http application layer protocol. * Typo in error message --- .../feast/core/config/FeastProperties.java | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/core/src/main/java/feast/core/config/FeastProperties.java b/core/src/main/java/feast/core/config/FeastProperties.java index 941d51f68c9..eb50728baf5 100644 --- a/core/src/main/java/feast/core/config/FeastProperties.java +++ b/core/src/main/java/feast/core/config/FeastProperties.java @@ -18,6 +18,8 @@ import feast.core.config.FeastProperties.StreamProperties.FeatureStreamOptions; import feast.core.validators.OneOfStrings; +import java.net.InetAddress; +import java.net.UnknownHostException; import java.util.*; import javax.annotation.PostConstruct; import javax.validation.*; @@ -26,7 +28,6 @@ import javax.validation.constraints.Positive; import lombok.Getter; import lombok.Setter; -import org.hibernate.validator.constraints.URL; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.boot.context.properties.ConfigurationProperties; import org.springframework.boot.info.BuildProperties; @@ -91,6 +92,7 @@ public Runner getActiveRunner() { @Getter @Setter public static class Runner { + /** Job runner name. This must be unique. */ String name; @@ -173,7 +175,7 @@ public static class MetricsProperties { private String type; /* Host of metric sink */ - @URL private String host; + private String host; /* Port of metric sink */ @Positive private int port; @@ -221,6 +223,18 @@ public void validate() { if (!jobMetricViolations.isEmpty()) { throw new ConstraintViolationException(jobMetricViolations); } + // Additional custom check for hostname value because there is no built-in Spring annotation + // to validate the value is a DNS resolvable hostname or an IP address. + try { + //noinspection ResultOfMethodCallIgnored + InetAddress.getByName(getJobs().getMetrics().getHost()); + } catch (UnknownHostException e) { + throw new IllegalArgumentException( + "Invalid config value for feast.jobs.metrics.host: " + + getJobs().getMetrics().getHost() + + ". Make sure it is a valid IP address or DNS hostname e.g. localhost or 10.128.10.40. Error detail: " + + e.getMessage()); + } } } } From 497b08d039b0cd3b03bdb95c01f95ba261632a10 Mon Sep 17 00:00:00 2001 From: Ches Martin Date: Thu, 30 Apr 2020 07:33:14 +0700 Subject: [PATCH 134/176] JobUpdateTask cleanups (#650) * core: Clean up repetition in JobUpdateTask Logic is a bit complicated here and trimming excess noise starts to help make it quicker to grok. We were doing the equivalent of Set#equals by hand. * core: Reduce noise of AuditLogger in JobUpdateTask * core: Simplify JobUpdateTask internal helper calls The helper methods like startJob and updateJob had parameters that are instance state, so there seemed to be no reason to pass around arguments for them. * core: Make isTerminal an instance property of JobStatus * core: Remove test setup duplication in JobUpdateTaskTest * core: Add unit tests for JobStatus * core: Refactor JobUpdateTask to use domain model types And factor out things that don't differ between its test cases. JobCoordinatorService unmarshals protos to model types eagerly as they come off the wire, so we're left dealing with models everywhere else. --- .../java/feast/core/job/JobUpdateTask.java | 175 +++++------- .../core/job/dataflow/DataflowJobManager.java | 7 +- .../job/direct/DirectRunnerJobManager.java | 3 +- core/src/main/java/feast/core/model/Job.java | 8 + .../main/java/feast/core/model/JobStatus.java | 38 +-- .../core/service/JobCoordinatorService.java | 124 ++++----- .../java/feast/core/service/JobService.java | 10 +- .../feast/core/job/JobUpdateTaskTest.java | 254 +++++------------- .../java/feast/core/model/JobStatusTest.java | 45 ++++ .../feast/core/service/JobServiceTest.java | 5 +- 10 files changed, 282 insertions(+), 387 deletions(-) create mode 100644 core/src/test/java/feast/core/model/JobStatusTest.java diff --git a/core/src/main/java/feast/core/job/JobUpdateTask.java b/core/src/main/java/feast/core/job/JobUpdateTask.java index 04aab0cff68..25ce386d40c 100644 --- a/core/src/main/java/feast/core/job/JobUpdateTask.java +++ b/core/src/main/java/feast/core/job/JobUpdateTask.java @@ -16,9 +16,6 @@ */ package feast.core.job; -import feast.core.FeatureSetProto; -import feast.core.SourceProto; -import feast.core.StoreProto; import feast.core.log.Action; import feast.core.log.AuditLogger; import feast.core.log.Resource; @@ -52,134 +49,96 @@ @Getter public class JobUpdateTask implements Callable { - private final List featureSets; - private final SourceProto.Source sourceSpec; - private final StoreProto.Store store; + private final List featureSets; + private final Source source; + private final Store store; private final Optional currentJob; - private JobManager jobManager; - private long jobUpdateTimeoutSeconds; + private final JobManager jobManager; + private final long jobUpdateTimeoutSeconds; + private final String runnerName; public JobUpdateTask( - List featureSets, - SourceProto.Source sourceSpec, - StoreProto.Store store, + List featureSets, + Source source, + Store store, Optional currentJob, JobManager jobManager, long jobUpdateTimeoutSeconds) { this.featureSets = featureSets; - this.sourceSpec = sourceSpec; + this.source = source; this.store = store; this.currentJob = currentJob; this.jobManager = jobManager; this.jobUpdateTimeoutSeconds = jobUpdateTimeoutSeconds; + this.runnerName = jobManager.getRunnerType().toString(); } @Override public Job call() { ExecutorService executorService = Executors.newSingleThreadExecutor(); - Source source = Source.fromProto(sourceSpec); Future submittedJob; - if (currentJob.isPresent()) { - Set existingFeatureSetsPopulatedByJob = - currentJob.get().getFeatureSets().stream() - .map(FeatureSet::getId) - .collect(Collectors.toSet()); - Set newFeatureSetsPopulatedByJob = - featureSets.stream() - .map(fs -> FeatureSet.fromProto(fs).getId()) - .collect(Collectors.toSet()); - if (existingFeatureSetsPopulatedByJob.size() == newFeatureSetsPopulatedByJob.size() - && existingFeatureSetsPopulatedByJob.containsAll(newFeatureSetsPopulatedByJob)) { - Job job = currentJob.get(); - JobStatus newJobStatus = jobManager.getJobStatus(job); - if (newJobStatus != job.getStatus()) { - AuditLogger.log( - Resource.JOB, - job.getId(), - Action.STATUS_CHANGE, - "Job status updated: changed from %s to %s", - job.getStatus(), - newJobStatus); - } - job.setStatus(newJobStatus); - return job; + + if (currentJob.isEmpty()) { + submittedJob = executorService.submit(this::createJob); + } else { + Job job = currentJob.get(); + + if (featureSetsChangedFor(job)) { + submittedJob = executorService.submit(() -> updateJob(job)); } else { - submittedJob = - executorService.submit(() -> updateJob(currentJob.get(), featureSets, store)); + return updateStatus(job); } - } else { - String jobId = createJobId(source.getId(), store.getName()); - submittedJob = executorService.submit(() -> startJob(jobId, featureSets, sourceSpec, store)); } - Job job = null; try { - job = submittedJob.get(getJobUpdateTimeoutSeconds(), TimeUnit.SECONDS); + return submittedJob.get(getJobUpdateTimeoutSeconds(), TimeUnit.SECONDS); } catch (InterruptedException | ExecutionException | TimeoutException e) { log.warn("Unable to start job for source {} and sink {}: {}", source, store, e.getMessage()); + return null; + } finally { executorService.shutdownNow(); } - return job; + } + + boolean featureSetsChangedFor(Job job) { + Set existingFeatureSetsPopulatedByJob = + job.getFeatureSets().stream().map(FeatureSet::getId).collect(Collectors.toSet()); + Set newFeatureSetsPopulatedByJob = + featureSets.stream().map(FeatureSet::getId).collect(Collectors.toSet()); + + return !newFeatureSetsPopulatedByJob.equals(existingFeatureSetsPopulatedByJob); + } + + private Job createJob() { + String jobId = createJobId(source.getId(), store.getName()); + return startJob(jobId); } /** Start or update the job to ingest data to the sink. */ - private Job startJob( - String jobId, - List featureSetProtos, - SourceProto.Source source, - StoreProto.Store sinkSpec) { - - List featureSets = - featureSetProtos.stream() - .map( - fsp -> - FeatureSet.fromProto( - FeatureSetProto.FeatureSet.newBuilder() - .setSpec(fsp.getSpec()) - .setMeta(fsp.getMeta()) - .build())) - .collect(Collectors.toList()); + private Job startJob(String jobId) { + Job job = new Job( - jobId, - "", - jobManager.getRunnerType(), - Source.fromProto(source), - Store.fromProto(sinkSpec), - featureSets, - JobStatus.PENDING); + jobId, "", jobManager.getRunnerType(), source, store, featureSets, JobStatus.PENDING); try { - AuditLogger.log( - Resource.JOB, - jobId, - Action.SUBMIT, - "Building graph and submitting to %s", - jobManager.getRunnerType().toString()); + logAudit(Action.SUBMIT, job, "Building graph and submitting to %s", runnerName); job = jobManager.startJob(job); - if (job.getExtId().isEmpty()) { + var extId = job.getExtId(); + if (extId.isEmpty()) { throw new RuntimeException( String.format("Could not submit job: \n%s", "unable to retrieve job external id")); } - AuditLogger.log( - Resource.JOB, - jobId, - Action.STATUS_CHANGE, - "Job submitted to runner %s with ext id %s.", - jobManager.getRunnerType().toString(), - job.getExtId()); + var auditMessage = "Job submitted to runner %s with ext id %s."; + logAudit(Action.STATUS_CHANGE, job, auditMessage, runnerName, extId); return job; } catch (Exception e) { log.error(e.getMessage()); - AuditLogger.log( - Resource.JOB, - jobId, - Action.STATUS_CHANGE, - "Job failed to be submitted to runner %s. Job status changed to ERROR.", - jobManager.getRunnerType().toString()); + var auditMessage = "Job failed to be submitted to runner %s. Job status changed to ERROR."; + logAudit(Action.STATUS_CHANGE, job, auditMessage, runnerName); job.setStatus(JobStatus.ERROR); return job; @@ -187,33 +146,33 @@ private Job startJob( } /** Update the given job */ - private Job updateJob( - Job job, List featureSets, StoreProto.Store store) { - job.setFeatureSets( - featureSets.stream() - .map( - fs -> - FeatureSet.fromProto( - FeatureSetProto.FeatureSet.newBuilder() - .setSpec(fs.getSpec()) - .setMeta(fs.getMeta()) - .build())) - .collect(Collectors.toList())); - job.setStore(feast.core.model.Store.fromProto(store)); - AuditLogger.log( - Resource.JOB, - job.getId(), - Action.UPDATE, - "Updating job %s for runner %s", - job.getId(), - jobManager.getRunnerType().toString()); + private Job updateJob(Job job) { + job.setFeatureSets(featureSets); + job.setStore(store); + logAudit(Action.UPDATE, job, "Updating job %s for runner %s", job.getId(), runnerName); return jobManager.updateJob(job); } + private Job updateStatus(Job job) { + JobStatus currentStatus = job.getStatus(); + JobStatus newStatus = jobManager.getJobStatus(job); + if (newStatus != currentStatus) { + var auditMessage = "Job status updated: changed from %s to %s"; + logAudit(Action.STATUS_CHANGE, job, auditMessage, currentStatus, newStatus); + } + + job.setStatus(newStatus); + return job; + } + String createJobId(String sourceId, String storeName) { String dateSuffix = String.valueOf(Instant.now().toEpochMilli()); String sourceIdTrunc = sourceId.split("/")[0].toLowerCase(); String jobId = String.format("%s-to-%s", sourceIdTrunc, storeName) + dateSuffix; return jobId.replaceAll("_", "-"); } + + private void logAudit(Action action, Job job, String detail, Object... args) { + AuditLogger.log(Resource.JOB, job.getId(), action, detail, args); + } } diff --git a/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java b/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java index 880dd6c146b..db9a7f90707 100644 --- a/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java +++ b/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java @@ -193,16 +193,15 @@ public void abortJob(String dataflowJobId) { } /** - * Restart a restart dataflow job. Dataflow should ensure continuity between during the restart, - * so no data should be lost during the restart operation. + * Restart a Dataflow job. Dataflow should ensure continuity such that no data should be lost + * during the restart operation. * * @param job job to restart * @return the restarted job */ @Override public Job restartJob(Job job) { - JobStatus status = job.getStatus(); - if (JobStatus.getTerminalState().contains(status)) { + if (job.getStatus().isTerminal()) { // job yet not running: just start job return this.startJob(job); } else { diff --git a/core/src/main/java/feast/core/job/direct/DirectRunnerJobManager.java b/core/src/main/java/feast/core/job/direct/DirectRunnerJobManager.java index 9b3a8473e47..2adedbefd9f 100644 --- a/core/src/main/java/feast/core/job/direct/DirectRunnerJobManager.java +++ b/core/src/main/java/feast/core/job/direct/DirectRunnerJobManager.java @@ -166,8 +166,7 @@ public PipelineResult runPipeline(ImportOptions pipelineOptions) throws IOExcept */ @Override public Job restartJob(Job job) { - JobStatus status = job.getStatus(); - if (JobStatus.getTerminalState().contains(status)) { + if (job.getStatus().isTerminal()) { // job yet not running: just start job return this.startJob(job); } else { diff --git a/core/src/main/java/feast/core/model/Job.java b/core/src/main/java/feast/core/model/Job.java index 95bcd79e6c0..fc801f76a44 100644 --- a/core/src/main/java/feast/core/model/Job.java +++ b/core/src/main/java/feast/core/model/Job.java @@ -111,6 +111,14 @@ public Job( this.status = jobStatus; } + public boolean hasTerminated() { + return getStatus().isTerminal(); + } + + public boolean isRunning() { + return getStatus() == JobStatus.RUNNING; + } + public void updateMetrics(List newMetrics) { metrics.clear(); metrics.addAll(newMetrics); diff --git a/core/src/main/java/feast/core/model/JobStatus.java b/core/src/main/java/feast/core/model/JobStatus.java index 86aa512933c..1d86900e2c5 100644 --- a/core/src/main/java/feast/core/model/JobStatus.java +++ b/core/src/main/java/feast/core/model/JobStatus.java @@ -17,10 +17,8 @@ package feast.core.model; import feast.core.IngestionJobProto.IngestionJobStatus; -import java.util.Arrays; -import java.util.Collection; -import java.util.Collections; import java.util.Map; +import java.util.Set; public enum JobStatus { /** Job status is not known. */ @@ -53,33 +51,41 @@ public enum JobStatus { /** job has been suspended */ SUSPENDED; - private static final Collection TERMINAL_STATE = - Collections.unmodifiableList(Arrays.asList(COMPLETED, ABORTED, ERROR)); + private static final Set TERMINAL_STATES = Set.of(COMPLETED, ABORTED, ERROR); /** - * Get a collection of terminal job state. + * Get the set of terminal job states. * - *

    Terminal job state is final and will not change to any other state. + *

    A terminal job state is final and will not change to any other state. * - * @return collection of terminal job state. + * @return set of terminal job states. */ - public static Collection getTerminalState() { - return TERMINAL_STATE; + public static Set getTerminalStates() { + return TERMINAL_STATES; } - private static final Collection TRANSITIONAL_STATES = - Collections.unmodifiableList(Arrays.asList(PENDING, ABORTING, SUSPENDING)); + private static final Set TRANSITIONAL_STATES = Set.of(PENDING, ABORTING, SUSPENDING); /** - * Get Transitional Job Status states. Transitionals states are assigned to jobs that + * Get Transitional Job Status states. Transitional states are assigned to jobs that are * transitioning to a more stable state (ie SUSPENDED, ABORTED etc.) * - * @return Collection of transitional Job Status states. + * @return set of transitional Job Status states. */ - public static final Collection getTransitionalStates() { + public static Set getTransitionalStates() { return TRANSITIONAL_STATES; } + /** @return true if this {@code JobStatus} is a terminal state. */ + public boolean isTerminal() { + return getTerminalStates().contains(this); + } + + /** @return true if this {@code JobStatus} is a transitional state. */ + public boolean isTransitional() { + return getTransitionalStates().contains(this); + } + private static final Map INGESTION_JOB_STATUS_MAP = Map.of( JobStatus.UNKNOWN, IngestionJobStatus.UNKNOWN, @@ -95,7 +101,7 @@ public static final Collection getTransitionalStates() { /** * Convert a Job Status to Ingestion Job Status proto * - * @return IngestionJobStatus proto derieved from this job status + * @return IngestionJobStatus proto derived from this job status */ public IngestionJobStatus toProto() { // maps job models job status to ingestion job status diff --git a/core/src/main/java/feast/core/service/JobCoordinatorService.java b/core/src/main/java/feast/core/service/JobCoordinatorService.java index b4ed341edc6..6f366be5083 100644 --- a/core/src/main/java/feast/core/service/JobCoordinatorService.java +++ b/core/src/main/java/feast/core/service/JobCoordinatorService.java @@ -32,7 +32,6 @@ import feast.core.job.JobUpdateTask; import feast.core.model.FeatureSet; import feast.core.model.Job; -import feast.core.model.JobStatus; import feast.core.model.Source; import feast.core.model.Store; import java.util.ArrayList; @@ -45,6 +44,7 @@ import java.util.concurrent.ExecutorService; import java.util.concurrent.Executors; import java.util.stream.Collectors; +import javax.validation.constraints.Positive; import lombok.extern.slf4j.Slf4j; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.scheduling.annotation.Scheduled; @@ -55,11 +55,11 @@ @Service public class JobCoordinatorService { - private JobRepository jobRepository; - private FeatureSetRepository featureSetRepository; - private SpecService specService; - private JobManager jobManager; - private JobProperties jobProperties; + private final JobRepository jobRepository; + private final FeatureSetRepository featureSetRepository; + private final SpecService specService; + private final JobManager jobManager; + private final JobProperties jobProperties; @Autowired public JobCoordinatorService( @@ -90,54 +90,56 @@ public JobCoordinatorService( @Scheduled(fixedDelayString = "${feast.jobs.polling_interval_milliseconds}") public void Poll() throws InvalidProtocolBufferException { log.info("Polling for new jobs..."); + @Positive long updateTimeout = jobProperties.getJobUpdateTimeoutSeconds(); List jobUpdateTasks = new ArrayList<>(); ListStoresResponse listStoresResponse = specService.listStores(Filter.newBuilder().build()); - for (StoreProto.Store store : listStoresResponse.getStoreList()) { - Set featureSets = new HashSet<>(); - for (Subscription subscription : store.getSubscriptionsList()) { - featureSets.addAll( - new ArrayList<>( - specService - .listFeatureSets( - ListFeatureSetsRequest.Filter.newBuilder() - .setFeatureSetName(subscription.getName()) - .setFeatureSetVersion(subscription.getVersion()) - .setProject(subscription.getProject()) - .build()) - .getFeatureSetsList())); - } - if (!featureSets.isEmpty()) { - featureSets.stream() - .collect(Collectors.groupingBy(fs -> fs.getSpec().getSource())) - .entrySet() - .stream() - .forEach( - kv -> { - Optional originalJob = - getJob(Source.fromProto(kv.getKey()), Store.fromProto(store)); - jobUpdateTasks.add( - new JobUpdateTask( - kv.getValue(), - kv.getKey(), - store, - originalJob, - jobManager, - jobProperties.getJobUpdateTimeoutSeconds())); - }); + + for (StoreProto.Store storeSpec : listStoresResponse.getStoreList()) { + Set featureSets = new HashSet<>(); + Store store = Store.fromProto(storeSpec); + + for (Subscription subscription : store.getSubscriptions()) { + var featureSetSpecs = + specService + .listFeatureSets( + ListFeatureSetsRequest.Filter.newBuilder() + .setFeatureSetName(subscription.getName()) + .setFeatureSetVersion(subscription.getVersion()) + .setProject(subscription.getProject()) + .build()) + .getFeatureSetsList(); + featureSets.addAll(featureSetsFromProto(featureSetSpecs)); } + + featureSets.stream() + .collect(Collectors.groupingBy(FeatureSet::getSource)) + .forEach( + (source, setsForSource) -> { + Optional originalJob = getJob(source, store); + jobUpdateTasks.add( + new JobUpdateTask( + setsForSource, source, store, originalJob, jobManager, updateTimeout)); + }); } - if (jobUpdateTasks.size() == 0) { + if (jobUpdateTasks.isEmpty()) { log.info("No jobs found."); return; } log.info("Creating/Updating {} jobs...", jobUpdateTasks.size()); - ExecutorService executorService = Executors.newFixedThreadPool(jobUpdateTasks.size()); + startOrUpdateJobs(jobUpdateTasks); + + log.info("Updating feature set status"); + updateFeatureSetStatuses(jobUpdateTasks); + } + + void startOrUpdateJobs(List tasks) { + ExecutorService executorService = Executors.newFixedThreadPool(tasks.size()); ExecutorCompletionService ecs = new ExecutorCompletionService<>(executorService); - jobUpdateTasks.forEach(ecs::submit); + tasks.forEach(ecs::submit); int completedTasks = 0; - while (completedTasks < jobUpdateTasks.size()) { + while (completedTasks < tasks.size()) { try { Job job = ecs.take().get(); if (job != null) { @@ -148,27 +150,23 @@ public void Poll() throws InvalidProtocolBufferException { } completedTasks++; } - - log.info("Updating feature set status"); - updateFeatureSetStatuses(jobUpdateTasks); + executorService.shutdown(); } // TODO: make this more efficient private void updateFeatureSetStatuses(List jobUpdateTasks) { Set ready = new HashSet<>(); Set pending = new HashSet<>(); - for (JobUpdateTask jobUpdateTask : jobUpdateTasks) { - Optional job = - getJob( - Source.fromProto(jobUpdateTask.getSourceSpec()), - Store.fromProto(jobUpdateTask.getStore())); - if (job.isPresent()) { - if (job.get().getStatus() == JobStatus.RUNNING) { - ready.addAll(job.get().getFeatureSets()); - } else { - pending.addAll(job.get().getFeatureSets()); - } - } + for (JobUpdateTask task : jobUpdateTasks) { + getJob(task.getSource(), task.getStore()) + .ifPresent( + job -> { + if (job.isRunning()) { + ready.addAll(job.getFeatureSets()); + } else { + pending.addAll(job.getFeatureSets()); + } + }); } ready.removeAll(pending); ready.forEach( @@ -189,14 +187,16 @@ public Optional getJob(Source source, Store store) { List jobs = jobRepository.findBySourceIdAndStoreNameOrderByLastUpdatedDesc( source.getId(), store.getName()); - jobs = - jobs.stream() - .filter(job -> !JobStatus.getTerminalState().contains(job.getStatus())) - .collect(Collectors.toList()); - if (jobs.size() == 0) { + jobs = jobs.stream().filter(job -> !job.hasTerminated()).collect(Collectors.toList()); + if (jobs.isEmpty()) { return Optional.empty(); } // return the latest return Optional.of(jobs.get(0)); } + + // TODO: Put in a util somewhere? + private static List featureSetsFromProto(List protos) { + return protos.stream().map(FeatureSet::fromProto).collect(Collectors.toList()); + } } diff --git a/core/src/main/java/feast/core/service/JobService.java b/core/src/main/java/feast/core/service/JobService.java index c8fc5caf5e1..33c118999cc 100644 --- a/core/src/main/java/feast/core/service/JobService.java +++ b/core/src/main/java/feast/core/service/JobService.java @@ -159,6 +159,7 @@ public RestartIngestionJobResponse restartJob(RestartIngestionJobRequest request // check job exists Optional getJob = this.jobRepository.findById(request.getId()); if (getJob.isEmpty()) { + // FIXME: if getJob.isEmpty then constructing this error message will always throw an error... throw new NoSuchElementException( "Attempted to stop nonexistent job with id: " + getJob.get().getId()); } @@ -166,9 +167,7 @@ public RestartIngestionJobResponse restartJob(RestartIngestionJobRequest request // check job status is valid for restarting Job job = getJob.get(); JobStatus status = job.getStatus(); - if (JobStatus.getTransitionalStates().contains(status) - || JobStatus.getTerminalState().contains(status) - || status.equals(JobStatus.UNKNOWN)) { + if (status.isTransitional() || status.isTerminal() || status == JobStatus.UNKNOWN) { throw new UnsupportedOperationException( "Restarting a job with a transitional, terminal or unknown status is unsupported"); } @@ -209,11 +208,10 @@ public StopIngestionJobResponse stopJob(StopIngestionJobRequest request) // check job status is valid for stopping Job job = getJob.get(); JobStatus status = job.getStatus(); - if (JobStatus.getTerminalState().contains(status)) { + if (status.isTerminal()) { // do nothing - job is already stopped return StopIngestionJobResponse.newBuilder().build(); - } else if (JobStatus.getTransitionalStates().contains(status) - || status.equals(JobStatus.UNKNOWN)) { + } else if (status.isTransitional() || status == JobStatus.UNKNOWN) { throw new UnsupportedOperationException( "Stopping a job with a transitional or unknown status is unsupported"); } diff --git a/core/src/test/java/feast/core/job/JobUpdateTaskTest.java b/core/src/test/java/feast/core/job/JobUpdateTaskTest.java index 5faf446a948..8d179baebb1 100644 --- a/core/src/test/java/feast/core/job/JobUpdateTaskTest.java +++ b/core/src/test/java/feast/core/job/JobUpdateTaskTest.java @@ -40,6 +40,8 @@ import feast.core.model.Source; import feast.core.model.Store; import java.util.Arrays; +import java.util.Collections; +import java.util.List; import java.util.Optional; import org.hamcrest.core.IsNull; import org.junit.Before; @@ -47,95 +49,71 @@ import org.mockito.Mock; public class JobUpdateTaskTest { + private static final Runner RUNNER = Runner.DATAFLOW; + + private static final FeatureSetProto.FeatureSet.Builder fsBuilder = + FeatureSetProto.FeatureSet.newBuilder().setMeta(FeatureSetMeta.newBuilder()); + private static final FeatureSetSpec.Builder specBuilder = + FeatureSetSpec.newBuilder().setProject("project1").setVersion(1); @Mock private JobManager jobManager; - private StoreProto.Store store; - private SourceProto.Source source; + private Store store; + private Source source; + private FeatureSet featureSet1; @Before public void setUp() { initMocks(this); + when(jobManager.getRunnerType()).thenReturn(RUNNER); + store = - StoreProto.Store.newBuilder() - .setName("test") - .setType(StoreType.REDIS) - .setRedisConfig(RedisConfig.newBuilder().build()) - .addSubscriptions( - Subscription.newBuilder().setProject("*").setName("*").setVersion("*").build()) - .build(); + Store.fromProto( + StoreProto.Store.newBuilder() + .setName("test") + .setType(StoreType.REDIS) + .setRedisConfig(RedisConfig.newBuilder().build()) + .addSubscriptions( + Subscription.newBuilder().setProject("*").setName("*").setVersion("*").build()) + .build()); source = - SourceProto.Source.newBuilder() - .setType(SourceType.KAFKA) - .setKafkaSourceConfig( - KafkaSourceConfig.newBuilder() - .setTopic("topic") - .setBootstrapServers("servers:9092") - .build()) - .build(); + Source.fromProto( + SourceProto.Source.newBuilder() + .setType(SourceType.KAFKA) + .setKafkaSourceConfig( + KafkaSourceConfig.newBuilder() + .setTopic("topic") + .setBootstrapServers("servers:9092") + .build()) + .build()); + + featureSet1 = + FeatureSet.fromProto(fsBuilder.setSpec(specBuilder.setName("featureSet1")).build()); + featureSet1.setSource(source); + } + + Job makeJob(String extId, List featureSets, JobStatus status) { + return new Job("job", extId, RUNNER, source, store, featureSets, status); + } + + JobUpdateTask makeTask(List featureSets, Optional currentJob) { + return new JobUpdateTask(featureSets, source, store, currentJob, jobManager, 100L); } @Test public void shouldUpdateJobIfPresent() { - FeatureSetProto.FeatureSet featureSet1 = - FeatureSetProto.FeatureSet.newBuilder() - .setSpec( - FeatureSetSpec.newBuilder() - .setSource(source) - .setProject("project1") - .setName("featureSet1") - .setVersion(1)) - .setMeta(FeatureSetMeta.newBuilder()) - .build(); - FeatureSetProto.FeatureSet featureSet2 = - FeatureSetProto.FeatureSet.newBuilder() - .setSpec( - FeatureSetSpec.newBuilder() - .setSource(source) - .setProject("project1") - .setName("featureSet2") - .setVersion(1)) - .setMeta(FeatureSetMeta.newBuilder()) - .build(); - Job originalJob = - new Job( - "job", - "old_ext", - Runner.DATAFLOW, - feast.core.model.Source.fromProto(source), - feast.core.model.Store.fromProto(store), - Arrays.asList(FeatureSet.fromProto(featureSet1)), - JobStatus.RUNNING); - JobUpdateTask jobUpdateTask = - new JobUpdateTask( - Arrays.asList(featureSet1, featureSet2), - source, - store, - Optional.of(originalJob), - jobManager, - 100L); - Job submittedJob = - new Job( - "job", - "old_ext", - Runner.DATAFLOW, - feast.core.model.Source.fromProto(source), - feast.core.model.Store.fromProto(store), - Arrays.asList(FeatureSet.fromProto(featureSet1), FeatureSet.fromProto(featureSet2)), - JobStatus.RUNNING); + FeatureSet featureSet2 = + FeatureSet.fromProto(fsBuilder.setSpec(specBuilder.setName("featureSet2")).build()); + List existingFeatureSetsPopulatedByJob = Collections.singletonList(featureSet1); + List newFeatureSetsPopulatedByJob = Arrays.asList(featureSet1, featureSet2); + + Job originalJob = makeJob("old_ext", existingFeatureSetsPopulatedByJob, JobStatus.RUNNING); + JobUpdateTask jobUpdateTask = makeTask(newFeatureSetsPopulatedByJob, Optional.of(originalJob)); + Job submittedJob = makeJob("old_ext", newFeatureSetsPopulatedByJob, JobStatus.RUNNING); - Job expected = - new Job( - "job", - "new_ext", - Runner.DATAFLOW, - Source.fromProto(source), - Store.fromProto(store), - Arrays.asList(FeatureSet.fromProto(featureSet1), FeatureSet.fromProto(featureSet2)), - JobStatus.PENDING); + Job expected = makeJob("new_ext", newFeatureSetsPopulatedByJob, JobStatus.PENDING); when(jobManager.updateJob(submittedJob)).thenReturn(expected); - when(jobManager.getRunnerType()).thenReturn(Runner.DATAFLOW); Job actual = jobUpdateTask.call(); assertThat(actual, equalTo(expected)); @@ -143,43 +121,13 @@ public void shouldUpdateJobIfPresent() { @Test public void shouldCreateJobIfNotPresent() { - FeatureSetProto.FeatureSet featureSet1 = - FeatureSetProto.FeatureSet.newBuilder() - .setSpec( - FeatureSetSpec.newBuilder() - .setSource(source) - .setProject("project1") - .setName("featureSet1") - .setVersion(1)) - .setMeta(FeatureSetMeta.newBuilder()) - .build(); - JobUpdateTask jobUpdateTask = - spy( - new JobUpdateTask( - Arrays.asList(featureSet1), source, store, Optional.empty(), jobManager, 100L)); + var featureSets = Collections.singletonList(featureSet1); + JobUpdateTask jobUpdateTask = spy(makeTask(featureSets, Optional.empty())); doReturn("job").when(jobUpdateTask).createJobId("KAFKA/servers:9092/topic", "test"); - Job expectedInput = - new Job( - "job", - "", - Runner.DATAFLOW, - feast.core.model.Source.fromProto(source), - feast.core.model.Store.fromProto(store), - Arrays.asList(FeatureSet.fromProto(featureSet1)), - JobStatus.PENDING); + Job expectedInput = makeJob("", featureSets, JobStatus.PENDING); + Job expected = makeJob("ext", featureSets, JobStatus.PENDING); - Job expected = - new Job( - "job", - "ext", - Runner.DATAFLOW, - feast.core.model.Source.fromProto(source), - feast.core.model.Store.fromProto(store), - Arrays.asList(FeatureSet.fromProto(featureSet1)), - JobStatus.RUNNING); - - when(jobManager.getRunnerType()).thenReturn(Runner.DATAFLOW); when(jobManager.startJob(expectedInput)).thenReturn(expected); Job actual = jobUpdateTask.call(); @@ -188,83 +136,25 @@ public void shouldCreateJobIfNotPresent() { @Test public void shouldUpdateJobStatusIfNotCreateOrUpdate() { - FeatureSetProto.FeatureSet featureSet1 = - FeatureSetProto.FeatureSet.newBuilder() - .setSpec( - FeatureSetSpec.newBuilder() - .setSource(source) - .setProject("project1") - .setName("featureSet1") - .setVersion(1)) - .setMeta(FeatureSetMeta.newBuilder()) - .build(); - Job originalJob = - new Job( - "job", - "ext", - Runner.DATAFLOW, - feast.core.model.Source.fromProto(source), - feast.core.model.Store.fromProto(store), - Arrays.asList(FeatureSet.fromProto(featureSet1)), - JobStatus.RUNNING); - JobUpdateTask jobUpdateTask = - new JobUpdateTask( - Arrays.asList(featureSet1), source, store, Optional.of(originalJob), jobManager, 100L); + var featureSets = Collections.singletonList(featureSet1); + Job originalJob = makeJob("ext", featureSets, JobStatus.RUNNING); + JobUpdateTask jobUpdateTask = makeTask(featureSets, Optional.of(originalJob)); when(jobManager.getJobStatus(originalJob)).thenReturn(JobStatus.ABORTING); - Job expected = - new Job( - "job", - "ext", - Runner.DATAFLOW, - Source.fromProto(source), - Store.fromProto(store), - Arrays.asList(FeatureSet.fromProto(featureSet1)), - JobStatus.ABORTING); - Job actual = jobUpdateTask.call(); + Job updated = jobUpdateTask.call(); - assertThat(actual, equalTo(expected)); + assertThat(updated.getStatus(), equalTo(JobStatus.ABORTING)); } @Test public void shouldReturnJobWithErrorStatusIfFailedToSubmit() { - FeatureSetProto.FeatureSet featureSet1 = - FeatureSetProto.FeatureSet.newBuilder() - .setSpec( - FeatureSetSpec.newBuilder() - .setSource(source) - .setProject("project1") - .setName("featureSet1") - .setVersion(1)) - .setMeta(FeatureSetMeta.newBuilder()) - .build(); - JobUpdateTask jobUpdateTask = - spy( - new JobUpdateTask( - Arrays.asList(featureSet1), source, store, Optional.empty(), jobManager, 100L)); + var featureSets = Collections.singletonList(featureSet1); + JobUpdateTask jobUpdateTask = spy(makeTask(featureSets, Optional.empty())); doReturn("job").when(jobUpdateTask).createJobId("KAFKA/servers:9092/topic", "test"); - Job expectedInput = - new Job( - "job", - "", - Runner.DATAFLOW, - feast.core.model.Source.fromProto(source), - feast.core.model.Store.fromProto(store), - Arrays.asList(FeatureSet.fromProto(featureSet1)), - JobStatus.PENDING); - - Job expected = - new Job( - "job", - "", - Runner.DATAFLOW, - feast.core.model.Source.fromProto(source), - feast.core.model.Store.fromProto(store), - Arrays.asList(FeatureSet.fromProto(featureSet1)), - JobStatus.ERROR); + Job expectedInput = makeJob("", featureSets, JobStatus.PENDING); + Job expected = makeJob("", featureSets, JobStatus.ERROR); - when(jobManager.getRunnerType()).thenReturn(Runner.DATAFLOW); when(jobManager.startJob(expectedInput)) .thenThrow(new RuntimeException("Something went wrong")); @@ -274,21 +164,13 @@ public void shouldReturnJobWithErrorStatusIfFailedToSubmit() { @Test public void shouldTimeout() { - FeatureSetProto.FeatureSet featureSet1 = - FeatureSetProto.FeatureSet.newBuilder() - .setSpec( - FeatureSetSpec.newBuilder() - .setSource(source) - .setProject("project1") - .setName("featureSet1") - .setVersion(1)) - .setMeta(FeatureSetMeta.newBuilder()) - .build(); - + var featureSets = Collections.singletonList(featureSet1); + var timeoutSeconds = 0L; JobUpdateTask jobUpdateTask = spy( new JobUpdateTask( - Arrays.asList(featureSet1), source, store, Optional.empty(), jobManager, 0L)); + featureSets, source, store, Optional.empty(), jobManager, timeoutSeconds)); + Job actual = jobUpdateTask.call(); assertThat(actual, is(IsNull.nullValue())); } diff --git a/core/src/test/java/feast/core/model/JobStatusTest.java b/core/src/test/java/feast/core/model/JobStatusTest.java new file mode 100644 index 00000000000..f5c8839386c --- /dev/null +++ b/core/src/test/java/feast/core/model/JobStatusTest.java @@ -0,0 +1,45 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.core.model; + +import static org.hamcrest.Matchers.is; +import static org.junit.Assert.assertThat; + +import org.junit.Test; + +public class JobStatusTest { + + @Test + public void isTerminalReturnsTrueForJobStatusWithTerminalState() { + JobStatus.getTerminalStates() + .forEach( + status -> { + assertThat(status.isTerminal(), is(true)); + assertThat(status.isTransitional(), is(false)); + }); + } + + @Test + public void isTransitionalReturnsTrueForJobStatusWithTransitionalState() { + JobStatus.getTransitionalStates() + .forEach( + status -> { + assertThat(status.isTransitional(), is(true)); + assertThat(status.isTerminal(), is(false)); + }); + } +} diff --git a/core/src/test/java/feast/core/service/JobServiceTest.java b/core/src/test/java/feast/core/service/JobServiceTest.java index 6f34205bbfd..3d527a90a88 100644 --- a/core/src/test/java/feast/core/service/JobServiceTest.java +++ b/core/src/test/java/feast/core/service/JobServiceTest.java @@ -341,10 +341,9 @@ public void testStopJobForId() { } @Test - public void testStopAlreadyStop() { + public void testStopAlreadyStopped() { // check that stop jobs does not trying to stop jobs that are not already stopped - List doNothingStatuses = new ArrayList<>(); - doNothingStatuses.addAll(JobStatus.getTerminalState()); + List doNothingStatuses = new ArrayList<>(JobStatus.getTerminalStates()); JobStatus prevStatus = this.job.getStatus(); for (JobStatus status : doNothingStatuses) { From 20f491c0ce207dbaaddfa1900a859a317b67de90 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Thu, 30 Apr 2020 11:12:11 +0800 Subject: [PATCH 135/176] Add label checking to Prow (#665) * Add label checking to Prow * Add plugin to list of enabled plugins --- .prow/plugins.yaml | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/.prow/plugins.yaml b/.prow/plugins.yaml index 80195222903..ae8a1d0f135 100644 --- a/.prow/plugins.yaml +++ b/.prow/plugins.yaml @@ -12,6 +12,7 @@ plugins: - wip - trigger - config-updater + - require-matching-label config_updater: maps: @@ -23,3 +24,10 @@ external_plugins: - name: needs-rebase events: - pull_request + +require_matching_label: +- missing_label: needs-kind + org: gojek + repo: feast + prs: true + regexp: ^kind/ From ea4e2b0c9d7ca3c2eda9f8a464555fb9bb1a9f46 Mon Sep 17 00:00:00 2001 From: Julio Anthony Leonard Date: Thu, 30 Apr 2020 15:57:31 +0700 Subject: [PATCH 136/176] Add feature and feature set labels, for metadata (#536) * Add labels column to feature field * Add labels to feature spec proto * Implement labels on feature model * Implement list labels * Implement set_label and remove_label from feature client * Refactor ingest to only accept featureset and dtype * Add equality on labels field * Add tests for labels apply * Add Optional type hint to labels * Add empty labels key validation * Add label when generating feature spec for specServiceTest to test equality * corrected convention (push test) * corrected review comments * corrected lint-python check * corrected lint-python 2 * back out python SDK changes * Implemented labels on a feature set level * added empty keys validation * corrected review comments (storing empty json for features if labels map is empty) * Updated the comment to match the logic * added e2e tests for feature and feature set labels * moved e2e tests for feature and feature set labels * changed tests ordering Co-authored-by: Krzysztof Suwinski Co-authored-by: suwik --- .../java/feast/core/model/FeatureSet.java | 13 ++ .../src/main/java/feast/core/model/Field.java | 20 +- .../java/feast/core/util/TypeConversion.java | 3 - .../core/validators/FeatureSetValidator.java | 7 + .../feast/core/service/JobServiceTest.java | 27 +-- .../feast/core/service/SpecServiceTest.java | 209 +++++++++++------- .../feast/core/service/TestObjectFactory.java | 62 ++++++ .../feast/core/util/TypeConversionTest.java | 9 +- .../validators/FeatureSetValidatorTest.java | 87 ++++++++ protos/feast/core/FeatureSet.proto | 6 + sdk/python/feast/feature.py | 2 +- sdk/python/feast/loaders/ingest.py | 23 +- sdk/python/tests/test_client.py | 9 + tests/e2e/basic-ingest-redis-serving.py | 112 ++++++++-- 14 files changed, 451 insertions(+), 138 deletions(-) create mode 100644 core/src/test/java/feast/core/service/TestObjectFactory.java create mode 100644 core/src/test/java/feast/core/validators/FeatureSetValidatorTest.java diff --git a/core/src/main/java/feast/core/model/FeatureSet.java b/core/src/main/java/feast/core/model/FeatureSet.java index 232a5f67d14..ec8da77c5f9 100644 --- a/core/src/main/java/feast/core/model/FeatureSet.java +++ b/core/src/main/java/feast/core/model/FeatureSet.java @@ -25,6 +25,7 @@ import feast.core.FeatureSetProto.FeatureSetSpec; import feast.core.FeatureSetProto.FeatureSetStatus; import feast.core.FeatureSetProto.FeatureSpec; +import feast.core.util.TypeConversion; import feast.types.ValueProto.ValueType.Enum; import java.util.ArrayList; import java.util.HashMap; @@ -116,6 +117,10 @@ public class FeatureSet extends AbstractTimestampEntity implements Comparable entities, List features, Source source, + Map labels, FeatureSetStatus status) { this.maxAgeSeconds = maxAgeSeconds; this.source = source; @@ -137,6 +143,7 @@ public FeatureSet( this.name = name; this.project = new Project(project); this.version = version; + this.labels = TypeConversion.convertMapToJsonString(labels); this.setId(project, name, version); addEntities(entities); addFeatures(features); @@ -191,6 +198,7 @@ public static FeatureSet fromProto(FeatureSetProto.FeatureSet featureSetProto) { entitySpecs, featureSpecs, source, + featureSetProto.getSpec().getLabelsMap(), featureSetProto.getMeta().getStatus()); } @@ -247,6 +255,7 @@ public FeatureSetProto.FeatureSet toProto() throws InvalidProtocolBufferExceptio .setMaxAge(Duration.newBuilder().setSeconds(maxAgeSeconds)) .addAllEntities(entitySpecs) .addAllFeatures(featureSpecs) + .putAllLabels(TypeConversion.convertJsonStringToMap(labels)) .setSource(source.toProto()); return FeatureSetProto.FeatureSet.newBuilder().setMeta(meta).setSpec(spec).build(); @@ -352,6 +361,10 @@ private void setFeatureSpecFields(FeatureSpec.Builder featureSpecBuilder, Field featureSpecBuilder.setTimeOfDayDomain( TimeOfDayDomain.parseFrom(featureField.getTimeOfDayDomain())); } + + if (featureField.getLabels() != null) { + featureSpecBuilder.putAllLabels(featureField.getLabels()); + } } /** diff --git a/core/src/main/java/feast/core/model/Field.java b/core/src/main/java/feast/core/model/Field.java index cb23e4eceb7..213c17f954a 100644 --- a/core/src/main/java/feast/core/model/Field.java +++ b/core/src/main/java/feast/core/model/Field.java @@ -18,8 +18,9 @@ import feast.core.FeatureSetProto.EntitySpec; import feast.core.FeatureSetProto.FeatureSpec; -import feast.types.ValueProto.ValueType; +import feast.core.util.TypeConversion; import java.util.Arrays; +import java.util.Map; import java.util.Objects; import javax.persistence.Column; import javax.persistence.Embeddable; @@ -47,6 +48,10 @@ public class Field { @Column(name = "project") private String project; + // Labels that this field belongs to + @Column(name = "labels", columnDefinition = "text") + private String labels; + // Presence constraints (refer to proto feast.core.FeatureSet.FeatureSpec) // Only one of them can be set. private byte[] presence; @@ -74,14 +79,10 @@ public class Field { public Field() {} - public Field(String name, ValueType.Enum type) { - this.name = name; - this.type = type.toString(); - } - public Field(FeatureSpec featureSpec) { this.name = featureSpec.getName(); this.type = featureSpec.getValueType().toString(); + this.labels = TypeConversion.convertMapToJsonString(featureSpec.getLabelsMap()); switch (featureSpec.getPresenceConstraintsCase()) { case PRESENCE: @@ -215,6 +216,10 @@ public Field(EntitySpec entitySpec) { } } + public Map getLabels() { + return TypeConversion.convertJsonStringToMap(this.labels); + } + @Override public boolean equals(Object o) { if (this == o) { @@ -227,6 +232,7 @@ public boolean equals(Object o) { return Objects.equals(name, field.name) && Objects.equals(type, field.type) && Objects.equals(project, field.project) + && Objects.equals(labels, field.labels) && Arrays.equals(presence, field.presence) && Arrays.equals(groupPresence, field.groupPresence) && Arrays.equals(shape, field.shape) @@ -247,6 +253,6 @@ public boolean equals(Object o) { @Override public int hashCode() { - return Objects.hash(super.hashCode(), name, type); + return Objects.hash(super.hashCode(), name, type, project, labels); } } diff --git a/core/src/main/java/feast/core/util/TypeConversion.java b/core/src/main/java/feast/core/util/TypeConversion.java index e01a5511359..6ee990fc1c5 100644 --- a/core/src/main/java/feast/core/util/TypeConversion.java +++ b/core/src/main/java/feast/core/util/TypeConversion.java @@ -70,9 +70,6 @@ public static Map convertJsonStringToMap(String jsonString) { * @return json string corresponding to given map */ public static String convertMapToJsonString(Map map) { - if (map.isEmpty()) { - return "{}"; - } return gson.toJson(map); } diff --git a/core/src/main/java/feast/core/validators/FeatureSetValidator.java b/core/src/main/java/feast/core/validators/FeatureSetValidator.java index 213e3898d51..ca0f1ec035d 100644 --- a/core/src/main/java/feast/core/validators/FeatureSetValidator.java +++ b/core/src/main/java/feast/core/validators/FeatureSetValidator.java @@ -27,6 +27,7 @@ import java.util.stream.Collectors; public class FeatureSetValidator { + public static void validateSpec(FeatureSet featureSet) { if (featureSet.getSpec().getProject().isEmpty()) { throw new IllegalArgumentException("Project name must be provided"); @@ -34,6 +35,9 @@ public static void validateSpec(FeatureSet featureSet) { if (featureSet.getSpec().getName().isEmpty()) { throw new IllegalArgumentException("Feature set name must be provided"); } + if (featureSet.getSpec().getLabelsMap().containsKey("")) { + throw new IllegalArgumentException("Feature set label keys must not be empty"); + } checkValidCharacters(featureSet.getSpec().getProject(), "project"); checkValidCharacters(featureSet.getSpec().getName(), "name"); @@ -44,6 +48,9 @@ public static void validateSpec(FeatureSet featureSet) { } for (FeatureSpec featureSpec : featureSet.getSpec().getFeaturesList()) { checkValidCharacters(featureSpec.getName(), "features::name"); + if (featureSpec.getLabelsMap().containsKey("")) { + throw new IllegalArgumentException("Feature label keys must not be empty"); + } } } diff --git a/core/src/test/java/feast/core/service/JobServiceTest.java b/core/src/test/java/feast/core/service/JobServiceTest.java index 3d527a90a88..b649181afbf 100644 --- a/core/src/test/java/feast/core/service/JobServiceTest.java +++ b/core/src/test/java/feast/core/service/JobServiceTest.java @@ -34,11 +34,8 @@ import feast.core.CoreServiceProto.RestartIngestionJobResponse; import feast.core.CoreServiceProto.StopIngestionJobRequest; import feast.core.CoreServiceProto.StopIngestionJobResponse; -import feast.core.FeatureSetProto.FeatureSetStatus; import feast.core.FeatureSetReferenceProto.FeatureSetReference; import feast.core.IngestionJobProto.IngestionJob; -import feast.core.SourceProto.KafkaSourceConfig; -import feast.core.SourceProto.SourceType; import feast.core.StoreProto.Store.RedisConfig; import feast.core.StoreProto.Store.StoreType; import feast.core.dao.JobRepository; @@ -84,14 +81,7 @@ public void setup() { // create mock objects for testing // fake data source - this.dataSource = - new Source( - SourceType.KAFKA, - KafkaSourceConfig.newBuilder() - .setBootstrapServers("kafka:9092") - .setTopic("my-topic") - .build(), - true); + this.dataSource = TestObjectFactory.defaultSource; // fake data store this.dataStore = new Store( @@ -158,19 +148,12 @@ public void setupJobManager() { // dummy model constructorss private FeatureSet newDummyFeatureSet(String name, int version, String project) { - Field feature = new Field(name + "_feature", Enum.INT64); - Field entity = new Field(name + "_entity", Enum.STRING); + Field feature = TestObjectFactory.CreateFeatureField(name + "_feature", Enum.INT64); + Field entity = TestObjectFactory.CreateEntityField(name + "_entity", Enum.STRING); FeatureSet fs = - new FeatureSet( - name, - project, - version, - 100L, - Arrays.asList(entity), - Arrays.asList(feature), - this.dataSource, - FeatureSetStatus.STATUS_READY); + TestObjectFactory.CreateFeatureSet( + name, project, version, Arrays.asList(entity), Arrays.asList(feature)); fs.setCreated(Date.from(Instant.ofEpochSecond(10L))); return fs; } diff --git a/core/src/test/java/feast/core/service/SpecServiceTest.java b/core/src/test/java/feast/core/service/SpecServiceTest.java index 43a66135dce..bb9f832bd7f 100644 --- a/core/src/test/java/feast/core/service/SpecServiceTest.java +++ b/core/src/test/java/feast/core/service/SpecServiceTest.java @@ -39,10 +39,7 @@ import feast.core.FeatureSetProto; import feast.core.FeatureSetProto.EntitySpec; import feast.core.FeatureSetProto.FeatureSetSpec; -import feast.core.FeatureSetProto.FeatureSetStatus; import feast.core.FeatureSetProto.FeatureSpec; -import feast.core.SourceProto.KafkaSourceConfig; -import feast.core.SourceProto.SourceType; import feast.core.StoreProto; import feast.core.StoreProto.Store.RedisConfig; import feast.core.StoreProto.Store.StoreType; @@ -110,39 +107,24 @@ public class SpecServiceTest { @Before public void setUp() { initMocks(this); - defaultSource = - new Source( - SourceType.KAFKA, - KafkaSourceConfig.newBuilder() - .setBootstrapServers("kafka:9092") - .setTopic("my-topic") - .build(), - true); + defaultSource = TestObjectFactory.defaultSource; FeatureSet featureSet1v1 = newDummyFeatureSet("f1", 1, "project1"); FeatureSet featureSet1v2 = newDummyFeatureSet("f1", 2, "project1"); FeatureSet featureSet1v3 = newDummyFeatureSet("f1", 3, "project1"); FeatureSet featureSet2v1 = newDummyFeatureSet("f2", 1, "project1"); - Field f3f1 = new Field("f3f1", Enum.INT64); - Field f3f2 = new Field("f3f2", Enum.INT64); - Field f3e1 = new Field("f3e1", Enum.STRING); + Field f3f1 = TestObjectFactory.CreateFeatureField("f3f1", Enum.INT64); + Field f3f2 = TestObjectFactory.CreateFeatureField("f3f2", Enum.INT64); + Field f3e1 = TestObjectFactory.CreateEntityField("f3e1", Enum.STRING); FeatureSet featureSet3v1 = - new FeatureSet( - "f3", - "project1", - 1, - 100L, - Arrays.asList(f3e1), - Arrays.asList(f3f2, f3f1), - defaultSource, - FeatureSetStatus.STATUS_READY); + TestObjectFactory.CreateFeatureSet( + "f3", "project1", 1, Arrays.asList(f3e1), Arrays.asList(f3f2, f3f1)); featureSets = Arrays.asList(featureSet1v1, featureSet1v2, featureSet1v3, featureSet2v1, featureSet3v1); when(featureSetRepository.findAll()).thenReturn(featureSets); when(featureSetRepository.findAllByOrderByNameAscVersionAsc()).thenReturn(featureSets); - when(featureSetRepository.findFeatureSetByNameAndProject_NameAndVersion("f1", "project1", 1)) .thenReturn(featureSets.get(0)); when(featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( @@ -490,19 +472,12 @@ public void applyFeatureSetShouldIncrementFeatureSetVersionIfAlreadyExists() public void applyFeatureSetShouldNotCreateFeatureSetIfFieldsUnordered() throws InvalidProtocolBufferException { - Field f3f1 = new Field("f3f1", Enum.INT64); - Field f3f2 = new Field("f3f2", Enum.INT64); - Field f3e1 = new Field("f3e1", Enum.STRING); + Field f3f1 = TestObjectFactory.CreateFeatureField("f3f1", Enum.INT64); + Field f3f2 = TestObjectFactory.CreateFeatureField("f3f2", Enum.INT64); + Field f3e1 = TestObjectFactory.CreateEntityField("f3e1", Enum.STRING); FeatureSetProto.FeatureSet incomingFeatureSet = - (new FeatureSet( - "f3", - "project1", - 5, - 100L, - Arrays.asList(f3e1), - Arrays.asList(f3f2, f3f1), - defaultSource, - FeatureSetStatus.STATUS_READY)) + (TestObjectFactory.CreateFeatureSet( + "f3", "project1", 5, Arrays.asList(f3e1), Arrays.asList(f3f2, f3f1))) .toProto(); ApplyFeatureSetResponse applyFeatureSetResponse = @@ -630,16 +605,8 @@ public void applyFeatureSetShouldAcceptPresenceShapeAndDomainConstraints() new ArrayList<>(appliedFeatureSetSpec.getFeaturesList()); appliedFeatureSpecs.sort(Comparator.comparing(FeatureSpec::getName)); - assertEquals(appliedEntitySpecs.size(), entitySpecs.size()); - assertEquals(appliedFeatureSpecs.size(), featureSpecs.size()); - - for (int i = 0; i < appliedEntitySpecs.size(); i++) { - assertEquals(entitySpecs.get(i), appliedEntitySpecs.get(i)); - } - - for (int i = 0; i < appliedFeatureSpecs.size(); i++) { - assertEquals(featureSpecs.get(i), appliedFeatureSpecs.get(i)); - } + assertEquals(appliedEntitySpecs, entitySpecs); + assertEquals(appliedFeatureSpecs, featureSpecs); } @Test @@ -713,19 +680,12 @@ public void applyFeatureSetShouldUpdateFeatureSetWhenConstraintsAreUpdated() @Test public void applyFeatureSetShouldCreateProjectWhenNotAlreadyExists() throws InvalidProtocolBufferException { - Field f3f1 = new Field("f3f1", Enum.INT64); - Field f3f2 = new Field("f3f2", Enum.INT64); - Field f3e1 = new Field("f3e1", Enum.STRING); + Field f3f1 = TestObjectFactory.CreateFeatureField("f3f1", Enum.INT64); + Field f3f2 = TestObjectFactory.CreateFeatureField("f3f2", Enum.INT64); + Field f3e1 = TestObjectFactory.CreateEntityField("f3e1", Enum.STRING); FeatureSetProto.FeatureSet incomingFeatureSet = - (new FeatureSet( - "f3", - "newproject", - 5, - 100L, - Arrays.asList(f3e1), - Arrays.asList(f3f2, f3f1), - defaultSource, - FeatureSetStatus.STATUS_READY)) + (TestObjectFactory.CreateFeatureSet( + "f3", "newproject", 5, Arrays.asList(f3e1), Arrays.asList(f3f2, f3f1))) .toProto(); ApplyFeatureSetResponse applyFeatureSetResponse = @@ -739,19 +699,12 @@ public void applyFeatureSetShouldCreateProjectWhenNotAlreadyExists() @Test public void applyFeatureSetShouldFailWhenProjectIsArchived() throws InvalidProtocolBufferException { - Field f3f1 = new Field("f3f1", Enum.INT64); - Field f3f2 = new Field("f3f2", Enum.INT64); - Field f3e1 = new Field("f3e1", Enum.STRING); + Field f3f1 = TestObjectFactory.CreateFeatureField("f3f1", Enum.INT64); + Field f3f2 = TestObjectFactory.CreateFeatureField("f3f2", Enum.INT64); + Field f3e1 = TestObjectFactory.CreateEntityField("f3e1", Enum.STRING); FeatureSetProto.FeatureSet incomingFeatureSet = - (new FeatureSet( - "f3", - "archivedproject", - 5, - 100L, - Arrays.asList(f3e1), - Arrays.asList(f3f2, f3f1), - defaultSource, - FeatureSetStatus.STATUS_READY)) + (TestObjectFactory.CreateFeatureSet( + "f3", "archivedproject", 5, Arrays.asList(f3e1), Arrays.asList(f3f2, f3f1))) .toProto(); expectedException.expect(IllegalArgumentException.class); @@ -759,6 +712,101 @@ public void applyFeatureSetShouldFailWhenProjectIsArchived() specService.applyFeatureSet(incomingFeatureSet); } + @Test + public void applyFeatureSetShouldAcceptFeatureLabels() throws InvalidProtocolBufferException { + List entitySpecs = new ArrayList<>(); + entitySpecs.add(EntitySpec.newBuilder().setName("entity1").setValueType(Enum.INT64).build()); + + Map featureLabels0 = + new HashMap<>() { + { + put("label1", "feast1"); + } + }; + + Map featureLabels1 = + new HashMap<>() { + { + put("label1", "feast1"); + put("label2", "feast2"); + } + }; + + List> featureLabels = new ArrayList<>(); + featureLabels.add(featureLabels0); + featureLabels.add(featureLabels1); + + List featureSpecs = new ArrayList<>(); + featureSpecs.add( + FeatureSpec.newBuilder() + .setName("feature1") + .setValueType(Enum.INT64) + .putAllLabels(featureLabels.get(0)) + .build()); + featureSpecs.add( + FeatureSpec.newBuilder() + .setName("feature2") + .setValueType(Enum.INT64) + .putAllLabels(featureLabels.get(1)) + .build()); + + FeatureSetSpec featureSetSpec = + FeatureSetSpec.newBuilder() + .setProject("project1") + .setName("featureSetWithConstraints") + .addAllEntities(entitySpecs) + .addAllFeatures(featureSpecs) + .build(); + FeatureSetProto.FeatureSet featureSet = + FeatureSetProto.FeatureSet.newBuilder().setSpec(featureSetSpec).build(); + + ApplyFeatureSetResponse applyFeatureSetResponse = specService.applyFeatureSet(featureSet); + FeatureSetSpec appliedFeatureSetSpec = applyFeatureSetResponse.getFeatureSet().getSpec(); + + // appliedEntitySpecs needs to be sorted because the list returned by specService may not + // follow the order in the request + List appliedEntitySpecs = new ArrayList<>(appliedFeatureSetSpec.getEntitiesList()); + appliedEntitySpecs.sort(Comparator.comparing(EntitySpec::getName)); + + // appliedFeatureSpecs needs to be sorted because the list returned by specService may not + // follow the order in the request + List appliedFeatureSpecs = + new ArrayList<>(appliedFeatureSetSpec.getFeaturesList()); + appliedFeatureSpecs.sort(Comparator.comparing(FeatureSpec::getName)); + + var featureSpecsLabels = + featureSpecs.stream().map(e -> e.getLabelsMap()).collect(Collectors.toList()); + assertEquals(appliedEntitySpecs, entitySpecs); + assertEquals(appliedFeatureSpecs, featureSpecs); + assertEquals(featureSpecsLabels, featureLabels); + } + + @Test + public void applyFeatureSetShouldAcceptFeatureSetLabels() throws InvalidProtocolBufferException { + Map featureSetLabels = + new HashMap<>() { + { + put("description", "My precious feature set"); + } + }; + + FeatureSetSpec featureSetSpec = + FeatureSetSpec.newBuilder() + .setProject("project1") + .setName("preciousFeatureSet") + .putAllLabels(featureSetLabels) + .build(); + FeatureSetProto.FeatureSet featureSet = + FeatureSetProto.FeatureSet.newBuilder().setSpec(featureSetSpec).build(); + + ApplyFeatureSetResponse applyFeatureSetResponse = specService.applyFeatureSet(featureSet); + FeatureSetSpec appliedFeatureSetSpec = applyFeatureSetResponse.getFeatureSet().getSpec(); + + var appliedLabels = appliedFeatureSetSpec.getLabelsMap(); + + assertEquals(featureSetLabels, appliedLabels); + } + @Test public void shouldUpdateStoreIfConfigChanges() throws InvalidProtocolBufferException { when(storeRepository.findById("SERVING")).thenReturn(Optional.of(stores.get(0))); @@ -806,19 +854,18 @@ public void shouldFailIfGetFeatureSetWithoutProject() throws InvalidProtocolBuff } private FeatureSet newDummyFeatureSet(String name, int version, String project) { - Field feature = new Field("feature", Enum.INT64); - Field entity = new Field("entity", Enum.STRING); + FeatureSpec f1 = + FeatureSpec.newBuilder() + .setName("feature") + .setValueType(Enum.STRING) + .putLabels("key", "value") + .build(); + Field feature = new Field(f1); + Field entity = TestObjectFactory.CreateEntityField("entity", Enum.STRING); FeatureSet fs = - new FeatureSet( - name, - project, - version, - 100L, - Arrays.asList(entity), - Arrays.asList(feature), - defaultSource, - FeatureSetStatus.STATUS_READY); + TestObjectFactory.CreateFeatureSet( + name, project, version, Arrays.asList(entity), Arrays.asList(feature)); fs.setCreated(Date.from(Instant.ofEpochSecond(10L))); return fs; } diff --git a/core/src/test/java/feast/core/service/TestObjectFactory.java b/core/src/test/java/feast/core/service/TestObjectFactory.java new file mode 100644 index 00000000000..966cb8d8163 --- /dev/null +++ b/core/src/test/java/feast/core/service/TestObjectFactory.java @@ -0,0 +1,62 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.core.service; + +import feast.core.FeatureSetProto; +import feast.core.SourceProto; +import feast.core.model.FeatureSet; +import feast.core.model.Field; +import feast.core.model.Source; +import feast.types.ValueProto; +import java.util.HashMap; +import java.util.List; + +public class TestObjectFactory { + + public static Source defaultSource = + new Source( + SourceProto.SourceType.KAFKA, + SourceProto.KafkaSourceConfig.newBuilder() + .setBootstrapServers("kafka:9092") + .setTopic("my-topic") + .build(), + true); + + public static FeatureSet CreateFeatureSet( + String name, String project, int version, List entities, List features) { + return new FeatureSet( + name, + project, + version, + 100L, + entities, + features, + defaultSource, + new HashMap<>(), + FeatureSetProto.FeatureSetStatus.STATUS_READY); + } + + public static Field CreateFeatureField(String name, ValueProto.ValueType.Enum valueType) { + return new Field( + FeatureSetProto.FeatureSpec.newBuilder().setName(name).setValueType(valueType).build()); + } + + public static Field CreateEntityField(String name, ValueProto.ValueType.Enum valueType) { + return new Field( + FeatureSetProto.EntitySpec.newBuilder().setName(name).setValueType(valueType).build()); + } +} diff --git a/core/src/test/java/feast/core/util/TypeConversionTest.java b/core/src/test/java/feast/core/util/TypeConversionTest.java index 75548f34653..02f0a7cee45 100644 --- a/core/src/test/java/feast/core/util/TypeConversionTest.java +++ b/core/src/test/java/feast/core/util/TypeConversionTest.java @@ -18,8 +18,7 @@ import static com.jayway.jsonpath.matchers.JsonPathMatchers.hasJsonPath; import static org.hamcrest.Matchers.equalTo; -import static org.junit.Assert.assertThat; -import static org.junit.Assert.assertTrue; +import static org.junit.Assert.*; import com.google.protobuf.Timestamp; import java.util.*; @@ -70,6 +69,12 @@ public void convertMapToJsonStringShouldReturnJsonStringForGivenMap() { TypeConversion.convertMapToJsonString(input), hasJsonPath("$.key", equalTo("value"))); } + @Test + public void convertMapToJsonStringShouldReturnEmptyJsonForAnEmptyMap() { + Map input = new HashMap<>(); + assertThat(TypeConversion.convertMapToJsonString(input), equalTo("{}")); + } + @Test public void convertJsonStringToArgsShouldReturnCorrectListOfArgs() { Map input = new HashMap<>(); diff --git a/core/src/test/java/feast/core/validators/FeatureSetValidatorTest.java b/core/src/test/java/feast/core/validators/FeatureSetValidatorTest.java new file mode 100644 index 00000000000..2e1e4e381ae --- /dev/null +++ b/core/src/test/java/feast/core/validators/FeatureSetValidatorTest.java @@ -0,0 +1,87 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.core.validators; + +import feast.core.FeatureSetProto; +import feast.types.ValueProto; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.ExpectedException; + +public class FeatureSetValidatorTest { + + @Rule public final ExpectedException expectedException = ExpectedException.none(); + + @Test + public void shouldThrowExceptionForFeatureLabelsWithAnEmptyKey() { + Map featureLabels = + new HashMap<>() { + { + put("", "empty_key"); + } + }; + + List featureSpecs = new ArrayList<>(); + featureSpecs.add( + FeatureSetProto.FeatureSpec.newBuilder() + .setName("feature1") + .setValueType(ValueProto.ValueType.Enum.INT64) + .putAllLabels(featureLabels) + .build()); + + FeatureSetProto.FeatureSetSpec featureSetSpec = + FeatureSetProto.FeatureSetSpec.newBuilder() + .setProject("project1") + .setName("featureSetWithConstraints") + .addAllFeatures(featureSpecs) + .build(); + FeatureSetProto.FeatureSet featureSet = + FeatureSetProto.FeatureSet.newBuilder().setSpec(featureSetSpec).build(); + + expectedException.expect(IllegalArgumentException.class); + expectedException.expectMessage("Feature label keys must not be empty"); + FeatureSetValidator.validateSpec(featureSet); + } + + @Test + public void shouldThrowExceptionForFeatureSetLabelsWithAnEmptyKey() { + + Map featureSetLabels = + new HashMap<>() { + { + put("", "empty_key"); + } + }; + + FeatureSetProto.FeatureSetSpec featureSetSpec = + FeatureSetProto.FeatureSetSpec.newBuilder() + .setProject("project1") + .setName("featureSetWithConstraints") + .putAllLabels(featureSetLabels) + .build(); + FeatureSetProto.FeatureSet featureSet = + FeatureSetProto.FeatureSet.newBuilder().setSpec(featureSetSpec).build(); + + expectedException.expect(IllegalArgumentException.class); + expectedException.expectMessage("Feature set label keys must not be empty"); + FeatureSetValidator.validateSpec(featureSet); + } +} diff --git a/protos/feast/core/FeatureSet.proto b/protos/feast/core/FeatureSet.proto index 429d99c8547..73173f09914 100644 --- a/protos/feast/core/FeatureSet.proto +++ b/protos/feast/core/FeatureSet.proto @@ -60,6 +60,9 @@ message FeatureSetSpec { // Optional. Source on which feature rows can be found. // If not set, source will be set to the default value configured in Feast Core. Source source = 6; + + // User defined metadata + map labels = 8; } message EntitySpec { @@ -156,6 +159,9 @@ message FeatureSpec { tensorflow.metadata.v0.TimeDomain time_domain = 17; tensorflow.metadata.v0.TimeOfDayDomain time_of_day_domain = 18; } + + // Labels for user defined metadata on a feature + map labels = 19; } message FeatureSetMeta { diff --git a/sdk/python/feast/feature.py b/sdk/python/feast/feature.py index 9c7ff20f9e2..f5c07070b09 100644 --- a/sdk/python/feast/feature.py +++ b/sdk/python/feast/feature.py @@ -56,7 +56,7 @@ def from_proto(cls, feature_proto: FeatureProto): Feature object """ feature = cls( - name=feature_proto.name, dtype=ValueType(feature_proto.value_type) + name=feature_proto.name, dtype=ValueType(feature_proto.value_type), ) feature.update_presence_constraints(feature_proto) feature.update_shape_type(feature_proto) diff --git a/sdk/python/feast/loaders/ingest.py b/sdk/python/feast/loaders/ingest.py index b4490f025c5..4d215cc9901 100644 --- a/sdk/python/feast/loaders/ingest.py +++ b/sdk/python/feast/loaders/ingest.py @@ -25,7 +25,9 @@ KAFKA_CHUNK_PRODUCTION_TIMEOUT = 120 # type: int -def _encode_pa_tables(file: str, fs: FeatureSet, row_group_idx: int) -> List[bytes]: +def _encode_pa_tables( + file: str, feature_set: str, fields: dict, row_group_idx: int +) -> List[bytes]: """ Helper function to encode a PyArrow table(s) read from parquet file(s) into FeatureRows. @@ -41,8 +43,11 @@ def _encode_pa_tables(file: str, fs: FeatureSet, row_group_idx: int) -> List[byt File directory of all the parquet file to encode. Parquet file must have more than one row group. - fs (feast.feature_set.FeatureSet): - FeatureSet describing parquet files. + feature_set (str): + Feature set reference in the format f"{project}/{name}:{version}". + + fields (dict[str, enum.Enum.ValueType]): + A mapping of field names to their value types. row_group_idx(int): Row group index to read and encode into byte like FeatureRow @@ -61,12 +66,10 @@ def _encode_pa_tables(file: str, fs: FeatureSet, row_group_idx: int) -> List[byt # Preprocess the columns by converting all its values to Proto values proto_columns = { - field_name: pa_column_to_proto_column(field.dtype, table.column(field_name)) - for field_name, field in fs.fields.items() + field_name: pa_column_to_proto_column(dtype, table.column(field_name)) + for field_name, dtype in fields.items() } - feature_set = f"{fs.project}/{fs.name}:{fs.version}" - # List to store result feature_rows = [] @@ -120,8 +123,12 @@ def get_feature_row_chunks( Iterable list of byte encoded FeatureRow(s). """ + feature_set = f"{fs.project}/{fs.name}:{fs.version}" + + field_map = {field.name: field.dtype for field in fs.fields.values()} + pool = Pool(max_workers) - func = partial(_encode_pa_tables, file, fs) + func = partial(_encode_pa_tables, file, feature_set, field_map) for chunk in pool.imap(func, row_groups): yield chunk return diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index ed0426b2f6a..3082265eccf 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -559,7 +559,16 @@ def test_apply_feature_set_success(self, test_client): and feature_sets[0].name == "my-feature-set-1" and feature_sets[0].features[0].name == "fs1-my-feature-1" and feature_sets[0].features[0].dtype == ValueType.INT64 + and feature_sets[0].features[1].name == "fs1-my-feature-2" + and feature_sets[0].features[1].dtype == ValueType.STRING + and feature_sets[0].entities[0].name == "fs1-my-entity-1" + and feature_sets[0].entities[0].dtype == ValueType.INT64 + and feature_sets[1].features[0].name == "fs2-my-feature-1" + and feature_sets[1].features[0].dtype == ValueType.STRING_LIST + and feature_sets[1].features[1].name == "fs2-my-feature-2" and feature_sets[1].features[1].dtype == ValueType.BYTES_LIST + and feature_sets[1].entities[0].name == "fs2-my-entity-1" + and feature_sets[1].entities[0].dtype == ValueType.INT64 ) @pytest.mark.parametrize( diff --git a/tests/e2e/basic-ingest-redis-serving.py b/tests/e2e/basic-ingest-redis-serving.py index 8e40794344e..e80c5b7af61 100644 --- a/tests/e2e/basic-ingest-redis-serving.py +++ b/tests/e2e/basic-ingest-redis-serving.py @@ -2,12 +2,15 @@ import math import random import time +import grpc from feast.entity import Entity from feast.serving.ServingService_pb2 import ( GetOnlineFeaturesRequest, GetOnlineFeaturesResponse, ) from feast.core.IngestionJob_pb2 import IngestionJobStatus +from feast.core.CoreService_pb2_grpc import CoreServiceStub +from feast.core import CoreService_pb2 from feast.types.Value_pb2 import Value as Value from feast.client import Client from feast.feature_set import FeatureSet, FeatureSetRef @@ -26,6 +29,7 @@ FLOAT_TOLERANCE = 0.00001 PROJECT_NAME = 'basic_' + uuid.uuid4().hex.upper()[0:6] + @pytest.fixture(scope='module') def core_url(pytestconfig): return pytestconfig.getoption("core_url") @@ -109,6 +113,7 @@ def test_basic_ingest_success(client, basic_dataframe): client.ingest(cust_trans_fs, basic_dataframe) time.sleep(5) + @pytest.mark.timeout(45) @pytest.mark.run(order=12) def test_basic_retrieve_online_success(client, basic_dataframe): @@ -146,12 +151,13 @@ def test_basic_retrieve_online_success(client, basic_dataframe): basic_dataframe.iloc[0]["daily_transactions"]) if math.isclose( - sent_daily_transactions, - returned_daily_transactions, - abs_tol=FLOAT_TOLERANCE, + sent_daily_transactions, + returned_daily_transactions, + abs_tol=FLOAT_TOLERANCE, ): break + @pytest.mark.timeout(300) @pytest.mark.run(order=19) def test_basic_ingest_jobs(client, basic_dataframe): @@ -319,20 +325,20 @@ def test_all_types_retrieve_online_success(client, all_types_dataframe): if response is None: continue - returned_float_list = ( response.field_values[0] - .fields[PROJECT_NAME+"/float_list_feature"] + .fields[PROJECT_NAME + "/float_list_feature"] .float_list_val.val ) sent_float_list = all_types_dataframe.iloc[0]["float_list_feature"] if math.isclose( - returned_float_list[0], sent_float_list[0], abs_tol=FLOAT_TOLERANCE + returned_float_list[0], sent_float_list[0], abs_tol=FLOAT_TOLERANCE ): break + @pytest.mark.timeout(300) @pytest.mark.run(order=29) def test_all_types_ingest_jobs(client, all_types_dataframe): @@ -355,6 +361,7 @@ def test_all_types_ingest_jobs(client, all_types_dataframe): ingest_job.wait(IngestionJobStatus.ABORTED) assert ingest_job.status == IngestionJobStatus.ABORTED + @pytest.fixture(scope='module') def large_volume_dataframe(): ROW_COUNT = 100000 @@ -445,9 +452,9 @@ def test_large_volume_retrieve_online_success(client, large_volume_dataframe): large_volume_dataframe.iloc[0]["daily_transactions_large"]) if math.isclose( - sent_daily_transactions, - returned_daily_transactions, - abs_tol=FLOAT_TOLERANCE, + sent_daily_transactions, + returned_daily_transactions, + abs_tol=FLOAT_TOLERANCE, ): break @@ -462,14 +469,14 @@ def all_types_parquet_file(): "customer_id": [np.int32(random.randint(0, 10000)) for _ in range(COUNT)], "int32_feature_parquet": [np.int32(random.randint(0, 10000)) for _ in - range(COUNT)], + range(COUNT)], "int64_feature_parquet": [np.int64(random.randint(0, 10000)) for _ in - range(COUNT)], + range(COUNT)], "float_feature_parquet": [np.float(random.random()) for _ in range(COUNT)], "double_feature_parquet": [np.float64(random.random()) for _ in - range(COUNT)], + range(COUNT)], "string_feature_parquet": ["one" + str(random.random()) for _ in - range(COUNT)], + range(COUNT)], "bytes_feature_parquet": [b"one" for _ in range(COUNT)], "int32_list_feature_parquet": [ np.array([1, 2, 3, random.randint(0, 10000)], dtype=np.int32) @@ -509,6 +516,7 @@ def all_types_parquet_file(): df.to_parquet(file_path, allow_truncated_timestamps=True) return file_path + @pytest.mark.timeout(300) @pytest.mark.run(order=40) def test_all_types_parquet_register_feature_set_success(client): @@ -539,10 +547,86 @@ def test_all_types_parquet_register_feature_set_success(client): @pytest.mark.timeout(600) @pytest.mark.run(order=41) def test_all_types_infer_register_ingest_file_success(client, - all_types_parquet_file): + all_types_parquet_file): # Get feature set all_types_fs = client.get_feature_set(name="all_types_parquet") # Ingest user embedding data client.ingest(feature_set=all_types_fs, source=all_types_parquet_file, force_update=True) + + +# TODO: rewrite these using python SDK once the labels are implemented there +class TestsBasedOnGrpc: + LAST_VERSION = 0 + GRPC_CONNECTION_TIMEOUT = 3 + LABEL_KEY = "my" + LABEL_VALUE = "label" + + @pytest.fixture(scope="module") + def core_service_stub(self, core_url): + if core_url.endswith(":443"): + core_channel = grpc.secure_channel( + core_url, grpc.ssl_channel_credentials() + ) + else: + core_channel = grpc.insecure_channel(core_url) + + try: + grpc.channel_ready_future(core_channel).result(timeout=self.GRPC_CONNECTION_TIMEOUT) + except grpc.FutureTimeoutError: + raise ConnectionError( + f"Connection timed out while attempting to connect to Feast " + f"Core gRPC server {core_url} " + ) + core_service_stub = CoreServiceStub(core_channel) + return core_service_stub + + def apply_feature_set(self, core_service_stub, feature_set_proto): + try: + apply_fs_response = core_service_stub.ApplyFeatureSet( + CoreService_pb2.ApplyFeatureSetRequest(feature_set=feature_set_proto), + timeout=self.GRPC_CONNECTION_TIMEOUT, + ) # type: ApplyFeatureSetResponse + except grpc.RpcError as e: + raise grpc.RpcError(e.details()) + return apply_fs_response.feature_set + + def get_feature_set(self, core_service_stub, name, project): + try: + get_feature_set_response = core_service_stub.GetFeatureSet( + CoreService_pb2.GetFeatureSetRequest( + project=project, name=name.strip(), version=self.LAST_VERSION + ) + ) # type: GetFeatureSetResponse + except grpc.RpcError as e: + raise grpc.RpcError(e.details()) + return get_feature_set_response.feature_set + + @pytest.mark.timeout(45) + @pytest.mark.run(order=51) + def test_register_feature_set_with_labels(self, core_service_stub): + feature_set_name = "test_feature_set_labels" + feature_set_proto = FeatureSet(feature_set_name, PROJECT_NAME).to_proto() + feature_set_proto.spec.labels[self.LABEL_KEY] = self.LABEL_VALUE + self.apply_feature_set(core_service_stub, feature_set_proto) + + retrieved_feature_set = self.get_feature_set(core_service_stub, feature_set_name, PROJECT_NAME) + + assert self.LABEL_KEY in retrieved_feature_set.spec.labels + assert retrieved_feature_set.spec.labels[self.LABEL_KEY] == self.LABEL_VALUE + + @pytest.mark.timeout(45) + @pytest.mark.run(order=52) + def test_register_feature_with_labels(self, core_service_stub): + feature_set_name = "test_feature_labels" + feature_set_proto = FeatureSet(feature_set_name, PROJECT_NAME, features=[Feature("rating", ValueType.INT64)]) \ + .to_proto() + feature_set_proto.spec.features[0].labels[self.LABEL_KEY] = self.LABEL_VALUE + self.apply_feature_set(core_service_stub, feature_set_proto) + + retrieved_feature_set = self.get_feature_set(core_service_stub, feature_set_name, PROJECT_NAME) + retrieved_feature = retrieved_feature_set.spec.features[0] + + assert self.LABEL_KEY in retrieved_feature.labels + assert retrieved_feature.labels[self.LABEL_KEY] == self.LABEL_VALUE From b636e6c688c46f63de73a35d68cc72113bc08d87 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Fri, 1 May 2020 14:16:31 +0800 Subject: [PATCH 137/176] Enable Prow e2e tests by default (#666) --- .prow/config.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.prow/config.yaml b/.prow/config.yaml index 5b039ff6616..fc397cc5544 100644 --- a/.prow/config.yaml +++ b/.prow/config.yaml @@ -142,6 +142,7 @@ presubmits: - name: test-end-to-end decorate: true + always_run: true spec: containers: - image: maven:3.6-jdk-11 @@ -155,6 +156,7 @@ presubmits: - name: test-end-to-end-redis-cluster decorate: true + always_run: true spec: containers: - image: maven:3.6-jdk-11 From e014733d347a89c5e941c79829cea5b0e9075f9e Mon Sep 17 00:00:00 2001 From: Ches Martin Date: Fri, 1 May 2020 14:58:31 +0700 Subject: [PATCH 138/176] Move TFDV stats to higher-numbered protobuf fields (#669) Within the Feast v0.5 release cycle these fields have not been in a release yet, so it's safe to renumber them. The motivation is reserving the lower-numbered fields for optimal encoding, referenced in the patch. Closes #667 --- protos/feast/core/FeatureSet.proto | 45 +++++++++++++++++------------- 1 file changed, 26 insertions(+), 19 deletions(-) diff --git a/protos/feast/core/FeatureSet.proto b/protos/feast/core/FeatureSet.proto index 73173f09914..9b60270a87a 100644 --- a/protos/feast/core/FeatureSet.proto +++ b/protos/feast/core/FeatureSet.proto @@ -120,14 +120,24 @@ message FeatureSpec { // Value type of the feature. feast.types.ValueType.Enum value_type = 2; + // Reserve field numbers 15 and below for fields that will almost always be set + // https://developers.google.com/protocol-buffers/docs/proto3#assigning-field-numbers + reserved 3 to 15; + + // Labels for user defined metadata on a feature + map labels = 16; + + // Reserved for fundamental future additions less noisy in the schema that TFDV stats fields + reserved 17 to 29; + // presence_constraints, shape_type and domain_info are referenced from: // https://github.com/tensorflow/metadata/blob/36f65d1268cbc92cdbcf812ee03dcf47fb53b91e/tensorflow_metadata/proto/v0/schema.proto#L107 oneof presence_constraints { // Constraints on the presence of this feature in the examples. - tensorflow.metadata.v0.FeaturePresence presence = 3; + tensorflow.metadata.v0.FeaturePresence presence = 30; // Only used in the context of a "group" context, e.g., inside a sequence. - tensorflow.metadata.v0.FeaturePresenceWithinGroup group_presence = 4; + tensorflow.metadata.v0.FeaturePresenceWithinGroup group_presence = 31; } // The shape of the feature which governs the number of values that appear in @@ -135,33 +145,30 @@ message FeatureSpec { oneof shape_type { // The feature has a fixed shape corresponding to a multi-dimensional // tensor. - tensorflow.metadata.v0.FixedShape shape = 5; + tensorflow.metadata.v0.FixedShape shape = 32; // The feature doesn't have a well defined shape. All we know are limits on // the minimum and maximum number of values. - tensorflow.metadata.v0.ValueCount value_count = 6; + tensorflow.metadata.v0.ValueCount value_count = 33; } // Domain for the values of the feature. oneof domain_info { // Reference to a domain defined at the schema level. - string domain = 7; + string domain = 34; // Inline definitions of domains. - tensorflow.metadata.v0.IntDomain int_domain = 8; - tensorflow.metadata.v0.FloatDomain float_domain = 9; - tensorflow.metadata.v0.StringDomain string_domain = 10; - tensorflow.metadata.v0.BoolDomain bool_domain = 11; - tensorflow.metadata.v0.StructDomain struct_domain = 12; + tensorflow.metadata.v0.IntDomain int_domain = 35; + tensorflow.metadata.v0.FloatDomain float_domain = 36; + tensorflow.metadata.v0.StringDomain string_domain = 37; + tensorflow.metadata.v0.BoolDomain bool_domain = 38; + tensorflow.metadata.v0.StructDomain struct_domain = 39; // Supported semantic domains. - tensorflow.metadata.v0.NaturalLanguageDomain natural_language_domain = 13; - tensorflow.metadata.v0.ImageDomain image_domain = 14; - tensorflow.metadata.v0.MIDDomain mid_domain = 15; - tensorflow.metadata.v0.URLDomain url_domain = 16; - tensorflow.metadata.v0.TimeDomain time_domain = 17; - tensorflow.metadata.v0.TimeOfDayDomain time_of_day_domain = 18; + tensorflow.metadata.v0.NaturalLanguageDomain natural_language_domain = 40; + tensorflow.metadata.v0.ImageDomain image_domain = 41; + tensorflow.metadata.v0.MIDDomain mid_domain = 42; + tensorflow.metadata.v0.URLDomain url_domain = 43; + tensorflow.metadata.v0.TimeDomain time_domain = 44; + tensorflow.metadata.v0.TimeOfDayDomain time_of_day_domain = 45; } - - // Labels for user defined metadata on a feature - map labels = 19; } message FeatureSetMeta { From 9bc04034a387479b4048cf5c9782532dae43432a Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Fri, 1 May 2020 21:44:08 +0800 Subject: [PATCH 139/176] Add v0.3.7 to change log --- CHANGELOG.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4276858ca0a..66352435381 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -119,13 +119,22 @@ - Add readiness checks for Feast services in end to end test [\#337](https://github.com/gojek/feast/pull/337) ([davidheryanto](https://github.com/davidheryanto)) - Create CHANGELOG.md [\#321](https://github.com/gojek/feast/pull/321) ([woop](https://github.com/woop)) +## [v0.3.7](https://github.com/gojek/feast/tree/v0.3.7) (2020-05-01) + +[Full Changelog](https://github.com/gojek/feast/compare/v0.3.6...v0.3.7) + +**Merged pull requests:** + +- Moved end-to-end test scripts from .prow to infra [\#657](https://github.com/gojek/feast/pull/657) ([khorshuheng](https://github.com/khorshuheng)) +- Backported \#566 & \#647 to v0.3 [\#654](https://github.com/gojek/feast/pull/654) ([ches](https://github.com/ches)) + ## [v0.3.6](https://github.com/gojek/feast/tree/v0.3.6) (2020-01-03) **Merged pull requests:** [Full Changelog](https://github.com/gojek/feast/compare/v0.3.5...v0.3.6) -- Add support for file paths for providing entity rows during batch retrieval [\#375](https://github.com/gojek/feast/pull/376) ([voonhous](https://github.com/voonhous)) +- Add support for file paths for providing entity rows during batch retrieval [\#375](https://github.com/gojek/feast/pull/375) ([voonhous](https://github.com/voonhous)) ## [v0.3.5](https://github.com/gojek/feast/tree/v0.3.5) (2019-12-26) From 68a0d92fe8fefd9b20d5917c72ef34d5e67495cc Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Sat, 2 May 2020 09:02:34 +0800 Subject: [PATCH 140/176] Prevent Prow merge when needs-kind label is present --- .prow/config.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/.prow/config.yaml b/.prow/config.yaml index fc397cc5544..357c29cbc22 100644 --- a/.prow/config.yaml +++ b/.prow/config.yaml @@ -51,6 +51,7 @@ tide: - do-not-merge/invalid-owners-file - do-not-merge/work-in-progress - needs-rebase + - needs-kind merge_method: gojek/feast: squash blocker_label: merge-blocker From 6dd9a862d461258a8388d9febcd686423448ef9f Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Sat, 2 May 2020 10:52:15 +0800 Subject: [PATCH 141/176] Enabled always_run for test-end-to-end-batch --- .prow/config.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/.prow/config.yaml b/.prow/config.yaml index 357c29cbc22..af41aab759f 100644 --- a/.prow/config.yaml +++ b/.prow/config.yaml @@ -185,6 +185,7 @@ presubmits: - name: test-end-to-end-batch decorate: true + always_run: true spec: volumes: - name: service-account From 07b8bdf8c645a3f01c3ee0d98b25bd92dbed6054 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Sat, 2 May 2020 12:35:13 +0800 Subject: [PATCH 142/176] Update e2e tests to allow non-snapshot testing (#672) --- infra/scripts/test-end-to-end-batch.sh | 9 ++++++--- infra/scripts/test-end-to-end-redis-cluster.sh | 9 ++++++--- infra/scripts/test-end-to-end.sh | 9 ++++++--- 3 files changed, 18 insertions(+), 9 deletions(-) diff --git a/infra/scripts/test-end-to-end-batch.sh b/infra/scripts/test-end-to-end-batch.sh index 0e7bfe8bf8d..a5d8d9556b6 100755 --- a/infra/scripts/test-end-to-end-batch.sh +++ b/infra/scripts/test-end-to-end-batch.sh @@ -8,7 +8,10 @@ test -z ${SKIP_BUILD_JARS} && SKIP_BUILD_JARS="false" test -z ${GOOGLE_CLOUD_PROJECT} && GOOGLE_CLOUD_PROJECT="kf-feast" test -z ${TEMP_BUCKET} && TEMP_BUCKET="feast-templocation-kf-feast" test -z ${JOBS_STAGING_LOCATION} && JOBS_STAGING_LOCATION="gs://${TEMP_BUCKET}/staging-location" -test -z ${JAR_VERSION_SUFFIX} && JAR_VERSION_SUFFIX="-SNAPSHOT" + +# Get the current build version using maven (and pom.xml) +FEAST_BUILD_VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout) +echo Building version: $FEAST_BUILD_VERSION echo " This script will run end-to-end tests for Feast Core and Batch Serving. @@ -152,7 +155,7 @@ spring: password: password EOF -nohup java -jar core/target/feast-core-*${JAR_VERSION_SUFFIX}.jar \ +nohup java -jar core/target/feast-core-${FEAST_BUILD_VERSION}.jar \ --spring.config.location=file:///tmp/core.application.yml \ &> /var/log/feast-core.log & sleep 35 @@ -215,7 +218,7 @@ server: EOF -nohup java -jar serving/target/feast-serving-*${JAR_VERSION_SUFFIX}.jar \ +nohup java -jar serving/target/feast-serving-${FEAST_BUILD_VERSION}.jar \ --spring.config.location=file:///tmp/serving.warehouse.application.yml \ &> /var/log/feast-serving-warehouse.log & sleep 15 diff --git a/infra/scripts/test-end-to-end-redis-cluster.sh b/infra/scripts/test-end-to-end-redis-cluster.sh index e17eeef381c..de41edf4741 100755 --- a/infra/scripts/test-end-to-end-redis-cluster.sh +++ b/infra/scripts/test-end-to-end-redis-cluster.sh @@ -8,7 +8,10 @@ test -z ${SKIP_BUILD_JARS} && SKIP_BUILD_JARS="false" test -z ${GOOGLE_CLOUD_PROJECT} && GOOGLE_CLOUD_PROJECT="kf-feast" test -z ${TEMP_BUCKET} && TEMP_BUCKET="feast-templocation-kf-feast" test -z ${JOBS_STAGING_LOCATION} && JOBS_STAGING_LOCATION="gs://${TEMP_BUCKET}/staging-location" -test -z ${JAR_VERSION_SUFFIX} && JAR_VERSION_SUFFIX="-SNAPSHOT" + +# Get the current build version using maven (and pom.xml) +FEAST_BUILD_VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout) +echo Building version: $FEAST_BUILD_VERSION echo " This script will run end-to-end tests for Feast Core and Online Serving. @@ -140,7 +143,7 @@ management: enabled: false EOF -nohup java -jar core/target/feast-core-*${JAR_VERSION_SUFFIX}.jar \ +nohup java -jar core/target/feast-core-${FEAST_BUILD_VERSION}.jar \ --spring.config.location=file:///tmp/core.application.yml \ &> /var/log/feast-core.log & sleep 35 @@ -210,7 +213,7 @@ spring: EOF -nohup java -jar serving/target/feast-serving-*${JAR_VERSION_SUFFIX}.jar \ +nohup java -jar serving/target/feast-serving-${FEAST_BUILD_VERSION}.jar \ --spring.config.location=file:///tmp/serving.online.application.yml \ &> /var/log/feast-serving-online.log & sleep 15 diff --git a/infra/scripts/test-end-to-end.sh b/infra/scripts/test-end-to-end.sh index c33dadc5413..452d646f2f2 100755 --- a/infra/scripts/test-end-to-end.sh +++ b/infra/scripts/test-end-to-end.sh @@ -8,7 +8,10 @@ test -z ${SKIP_BUILD_JARS} && SKIP_BUILD_JARS="false" test -z ${GOOGLE_CLOUD_PROJECT} && GOOGLE_CLOUD_PROJECT="kf-feast" test -z ${TEMP_BUCKET} && TEMP_BUCKET="feast-templocation-kf-feast" test -z ${JOBS_STAGING_LOCATION} && JOBS_STAGING_LOCATION="gs://${TEMP_BUCKET}/staging-location" -test -z ${JAR_VERSION_SUFFIX} && JAR_VERSION_SUFFIX="-SNAPSHOT" + +# Get the current build version using maven (and pom.xml) +FEAST_BUILD_VERSION=$(mvn help:evaluate -Dexpression=project.version -q -DforceStdout) +echo Building version: $FEAST_BUILD_VERSION echo " This script will run end-to-end tests for Feast Core and Online Serving. @@ -136,7 +139,7 @@ spring: EOF -nohup java -jar core/target/feast-core-*${JAR_VERSION_SUFFIX}.jar \ +nohup java -jar core/target/feast-core-$FEAST_BUILD_VERSION.jar \ --spring.config.location=file:///tmp/core.application.yml \ &> /var/log/feast-core.log & sleep 35 @@ -180,7 +183,7 @@ server: EOF -nohup java -jar serving/target/feast-serving-*${JAR_VERSION_SUFFIX}.jar \ +nohup java -jar serving/target/feast-serving-${FEAST_BUILD_VERSION}.jar \ --spring.config.location=file:///tmp/serving.online.application.yml \ &> /var/log/feast-serving-online.log & sleep 15 From 7a0ff91839757233d3af957b84096f9e2836386b Mon Sep 17 00:00:00 2001 From: David Heryanto Date: Sat, 2 May 2020 23:25:14 +0800 Subject: [PATCH 143/176] Refactor Feast Helm charts for better end user install experience (#533) * Refactor Feast Helm charts 1. Use --spring.config.additional-location to specify Spring application config so the default application.yaml bundled in the jar can be used, and users only need to override required config. This is to prevent chart users being overwhelmed by the amount of config to set. 2. Use less templating, if else, and functions in configmap.yaml in Feast charts. Instead set a reasonable default and allow users to override in application-override.yaml. This makes it easier to manage when application.yaml structure changes and documentation for application.yaml can be delegated to: https://github.com/gojek/feast/blob/master/core/src/main/resources/application.yml https://github.com/gojek/feast/blob/master/serving/src/main/resources/application.yml 3. Remove double nesting of charts. Previously, Postgresql and Kafka subcharts are bundled in Feast Core suchart. Now all 3 subcharts are sibling charts. This reduces coupling between charts and it's easier to configure chart values one at a time than one big values yaml. Also, documentation for third party subcharts can be delegated to the original authors. 4. Use helm-docs to generate README for Feast charts so the documentation is reproducible and easier to keep up to date when values.yaml changes. It is also easier to follow best practices when writing values.yaml. https://github.com/norwoodj/helm-docs 5. Add prometheus and grafana charts to support out of the box metrics collector and visualization when users install Feast with this chart. 6. Add tests for users to verify installation: 7. Add examples to install Feast with 3 different profiles: - online serving only with direct runner - online and batch serving with DirectRunner - online and batch serving with DataflowRunner * Update docs * Remove unneeded OWNERS file for 3rd party charts * Move .helmdocsignore to correct location helm-docs look for it in repo root folder * Add test prefix for projects created in helm test * Add missing kafka port in example feast stream config Incorrectly swapped resources requests and limit * Typo * Add application-secret.yaml, make it optional to use feast-core.postgresql.existingSecret, use JAVA_TOOL_OPTIONS instead of JAVA_OPTS - Also adjust CPU request and memory limit for test pods to fit the actual usage * Update README and configuration so it works Feast v0.5, but not Feast v0.4 * Typo * Add 4 different application config with flag toggle for Feast Core and Serving - application.yaml: default application.yaml bundled in the jar - application-generated.yaml: additional config generated by Helm that is valid when the dependencies like Postgres, Kafka and Redis are installed with default config - application-secret.yaml: config to override default and Helm generated config - application-override.yaml: same as application-secret.yaml but config is created as ConfigMap vs Secret and has a higher precendence than application-secret.yaml * Update apiVersion for deployment in prometheus-statsd-exporter so it works with latest Kubernetes version * Cleanup unsued template function and add checksum to configmap and secret So that deployment will be updated when configmap and secret are updated * Remove 3rd party charts from charts folder * Update example values so it's compatible with Feast v0.5. Add helm documentation for .Values.gcpProjectId --- .helmdocsignore | 6 + infra/charts/feast/Chart.yaml | 4 +- infra/charts/feast/README.md | 584 +++++++++++------- infra/charts/feast/README.md.gotmpl | 354 +++++++++++ .../feast/charts/feast-core/.helmignore | 22 - .../charts/feast/charts/feast-core/Chart.yaml | 4 +- .../charts/feast/charts/feast-core/README.md | 70 +++ .../charts/feast-core/charts/kafka-0.20.1.tgz | Bin 30761 -> 0 bytes .../feast-core/charts/postgresql-6.5.5.tgz | Bin 23599 -> 0 bytes .../feast/charts/feast-core/requirements.yaml | 15 - .../feast-core/templates/configmap.yaml | 42 +- .../feast-core/templates/deployment.yaml | 81 ++- .../charts/feast-core/templates/secret.yaml | 15 + .../feast/charts/feast-core/values.yaml | 265 +++----- .../feast/charts/feast-serving/.helmignore | 22 - .../feast/charts/feast-serving/Chart.yaml | 4 +- .../feast/charts/feast-serving/README.md | 69 +++ .../feast-serving/charts/redis-9.5.0.tgz | Bin 27574 -> 0 bytes .../charts/feast-serving/requirements.yaml | 8 - .../feast-serving/templates/_helpers.tpl | 7 - .../feast-serving/templates/configmap.yaml | 56 +- .../feast-serving/templates/deployment.yaml | 66 +- .../feast-serving/templates/secret.yaml | 15 + .../feast/charts/feast-serving/values.yaml | 250 +++----- infra/charts/feast/charts/grafana-5.0.5.tgz | Bin 0 -> 19271 bytes infra/charts/feast/charts/kafka-0.20.8.tgz | Bin 0 -> 31688 bytes .../charts/feast/charts/postgresql-8.6.1.tgz | Bin 0 -> 31082 bytes .../charts/feast/charts/prometheus-11.0.2.tgz | Bin 0 -> 32813 bytes .../prometheus-statsd-exporter/.helmignore | 0 .../prometheus-statsd-exporter/Chart.yaml | 0 .../prometheus-statsd-exporter/README.md | 3 +- .../templates/NOTES.txt | 0 .../templates/_helpers.tpl | 0 .../templates/config.yaml | 0 .../templates/deployment.yaml | 4 +- .../templates/pvc.yaml | 0 .../templates/service.yaml | 40 +- .../templates/serviceaccount.yaml | 0 .../prometheus-statsd-exporter/values.yaml | 3 + infra/charts/feast/charts/redis-10.5.6.tgz | Bin 0 -> 31002 bytes .../charts/feast/files/img/dataflow-jobs.png | Bin 0 -> 29728 bytes .../feast/files/img/prometheus-server.png | Bin 0 -> 84787 bytes infra/charts/feast/requirements.lock | 34 +- infra/charts/feast/requirements.yaml | 37 +- .../tests/test-feast-batch-serving.yaml | 116 ++++ .../tests/test-feast-online-serving.yaml | 105 ++++ infra/charts/feast/values-batch-serving.yaml | 29 + .../charts/feast/values-dataflow-runner.yaml | 113 ++++ infra/charts/feast/values-demo.yaml | 84 --- infra/charts/feast/values-external-store.yaml | 5 - infra/charts/feast/values-production.yaml | 4 - infra/charts/feast/values.yaml | 273 +------- 52 files changed, 1667 insertions(+), 1142 deletions(-) create mode 100644 .helmdocsignore create mode 100644 infra/charts/feast/README.md.gotmpl delete mode 100644 infra/charts/feast/charts/feast-core/.helmignore create mode 100644 infra/charts/feast/charts/feast-core/README.md delete mode 100644 infra/charts/feast/charts/feast-core/charts/kafka-0.20.1.tgz delete mode 100644 infra/charts/feast/charts/feast-core/charts/postgresql-6.5.5.tgz delete mode 100644 infra/charts/feast/charts/feast-core/requirements.yaml create mode 100644 infra/charts/feast/charts/feast-core/templates/secret.yaml delete mode 100644 infra/charts/feast/charts/feast-serving/.helmignore create mode 100644 infra/charts/feast/charts/feast-serving/README.md delete mode 100644 infra/charts/feast/charts/feast-serving/charts/redis-9.5.0.tgz delete mode 100644 infra/charts/feast/charts/feast-serving/requirements.yaml create mode 100644 infra/charts/feast/charts/feast-serving/templates/secret.yaml create mode 100644 infra/charts/feast/charts/grafana-5.0.5.tgz create mode 100644 infra/charts/feast/charts/kafka-0.20.8.tgz create mode 100644 infra/charts/feast/charts/postgresql-8.6.1.tgz create mode 100644 infra/charts/feast/charts/prometheus-11.0.2.tgz rename infra/charts/feast/charts/{feast-core/charts => }/prometheus-statsd-exporter/.helmignore (100%) rename infra/charts/feast/charts/{feast-core/charts => }/prometheus-statsd-exporter/Chart.yaml (100%) rename infra/charts/feast/charts/{feast-core/charts => }/prometheus-statsd-exporter/README.md (91%) rename infra/charts/feast/charts/{feast-core/charts => }/prometheus-statsd-exporter/templates/NOTES.txt (100%) rename infra/charts/feast/charts/{feast-core/charts => }/prometheus-statsd-exporter/templates/_helpers.tpl (100%) rename infra/charts/feast/charts/{feast-core/charts => }/prometheus-statsd-exporter/templates/config.yaml (100%) rename infra/charts/feast/charts/{feast-core/charts => }/prometheus-statsd-exporter/templates/deployment.yaml (96%) rename infra/charts/feast/charts/{feast-core/charts => }/prometheus-statsd-exporter/templates/pvc.yaml (100%) rename infra/charts/feast/charts/{feast-core/charts => }/prometheus-statsd-exporter/templates/service.yaml (69%) rename infra/charts/feast/charts/{feast-core/charts => }/prometheus-statsd-exporter/templates/serviceaccount.yaml (100%) rename infra/charts/feast/charts/{feast-core/charts => }/prometheus-statsd-exporter/values.yaml (96%) create mode 100644 infra/charts/feast/charts/redis-10.5.6.tgz create mode 100644 infra/charts/feast/files/img/dataflow-jobs.png create mode 100644 infra/charts/feast/files/img/prometheus-server.png create mode 100644 infra/charts/feast/templates/tests/test-feast-batch-serving.yaml create mode 100644 infra/charts/feast/templates/tests/test-feast-online-serving.yaml create mode 100644 infra/charts/feast/values-batch-serving.yaml create mode 100644 infra/charts/feast/values-dataflow-runner.yaml delete mode 100644 infra/charts/feast/values-demo.yaml delete mode 100644 infra/charts/feast/values-external-store.yaml delete mode 100644 infra/charts/feast/values-production.yaml diff --git a/.helmdocsignore b/.helmdocsignore new file mode 100644 index 00000000000..8f246bffece --- /dev/null +++ b/.helmdocsignore @@ -0,0 +1,6 @@ +infra/charts/feast/charts/postgresql +infra/charts/feast/charts/kafka +infra/charts/feast/charts/redis +infra/charts/feast/charts/prometheus-statsd-exporter +infra/charts/feast/charts/prometheus +infra/charts/feast/charts/grafana diff --git a/infra/charts/feast/Chart.yaml b/infra/charts/feast/Chart.yaml index c8f328548a9..8ce82e4a01d 100644 --- a/infra/charts/feast/Chart.yaml +++ b/infra/charts/feast/Chart.yaml @@ -1,4 +1,4 @@ apiVersion: v1 -description: A Helm chart to install Feast on kubernetes +description: Feature store for machine learning. name: feast -version: 0.4.4 +version: 0.5.0-alpha.1 diff --git a/infra/charts/feast/README.md b/infra/charts/feast/README.md index e93b687f191..3d868b2fd13 100644 --- a/infra/charts/feast/README.md +++ b/infra/charts/feast/README.md @@ -1,251 +1,379 @@ -# Feast Chart +feast +===== -This directory provides the Helm chart for Feast installation. +Feature store for machine learning. Current chart version is `0.5.0-alpha.1` -This chart installs Feast Core and Feast Serving components of Feast, along with -the required and optional dependencies. Components and dependencies can be -enabled or disabled by changing the corresponding `enabled` flag. Feast Core and -Feast Serving are subcharts of this parent Feast chart. The structure of the charts -are as follows: +## TL;DR; +```bash +# Add Feast Helm chart +helm repo add feast-charts https://feast-charts.storage.googleapis.com +helm repo update + +# Create secret for Feast database, replace with the desired value +kubectl create secret generic feast-postgresql \ + --from-literal=postgresql-password= + +# Install Feast with Online Serving and Beam DirectRunner +helm install --name myrelease feast-charts/feast \ + --set feast-core.postgresql.existingSecret=feast-postgresql \ + --set postgresql.existingSecret=feast-postgresql ``` -feast // top level feast chart -│ -├── feast-core // feast-core subchart -│ ├── postgresql // Postgresql dependency for feast-core (Feast database) -│ └── kafka // Kafka dependency for feast-core (default stream source) -│ -├── feast-serving-online // feast-serving subchart -│ └── redis // Redis dependency for installation of store together with feast-serving -│ -└── feast-serving-batch // feast-serving subchart -``` + +## Introduction +This chart install Feast deployment on a Kubernetes cluster using the [Helm](https://v2.helm.sh/docs/using_helm/#installing-helm) package manager. ## Prerequisites -- Kubernetes 1.13 or newer cluster -- Helm 2.15.2 or newer +- Kubernetes 1.12+ +- Helm 2.15+ (not tested with Helm 3) +- Persistent Volume support on the underlying infrastructure + +## Chart Requirements + +| Repository | Name | Version | +|------------|------|---------| +| | feast-core | 0.5.0-alpha.1 | +| | feast-serving | 0.5.0-alpha.1 | +| | feast-serving | 0.5.0-alpha.1 | +| | prometheus-statsd-exporter | 0.1.2 | +| https://kubernetes-charts-incubator.storage.googleapis.com/ | kafka | 0.20.8 | +| https://kubernetes-charts.storage.googleapis.com/ | grafana | 5.0.5 | +| https://kubernetes-charts.storage.googleapis.com/ | postgresql | 8.6.1 | +| https://kubernetes-charts.storage.googleapis.com/ | prometheus | 11.0.2 | +| https://kubernetes-charts.storage.googleapis.com/ | redis | 10.5.6 | -## Resources Required -The chart deploys pods that consume minimum resources as specified in the resources configuration parameter. +## Chart Values -## Installing the Chart +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| feast-batch-serving.enabled | bool | `false` | Flag to install Feast Batch Serving | +| feast-core.enabled | bool | `true` | Flag to install Feast Core | +| feast-online-serving.enabled | bool | `true` | Flag to install Feast Online Serving | +| grafana.enabled | bool | `true` | Flag to install Grafana | +| kafka.enabled | bool | `true` | Flag to install Kafka | +| postgresql.enabled | bool | `true` | Flag to install Postgresql | +| prometheus-statsd-exporter.enabled | bool | `true` | Flag to install StatsD to Prometheus Exporter | +| prometheus.enabled | bool | `true` | Flag to install Prometheus | +| redis.enabled | bool | `true` | Flag to install Redis | + +## Configuration and installation details + +The default configuration will install Feast with Online Serving. Ingestion +of features will use Beam [DirectRunner](https://beam.apache.org/documentation/runners/direct/) +that runs on the same container where Feast Core is running. -Add repository for Feast chart: ```bash -helm repo add feast-charts https://feast-charts.storage.googleapis.com -helm repo update +# Create secret for Feast database, replace accordingly +kubectl create secret generic feast-postgresql \ + --from-literal=postgresql-password= + +# Install Feast with Online Serving and Beam DirectRunner +helm install --name myrelease feast-charts/feast \ + --set feast-core.postgresql.existingSecret=feast-postgresql \ + --set postgresql.existingSecret=feast-postgresql +``` + +In order to test that the installation is successful: +```bash +helm test myrelease + +# If the installation is successful, the following should be printed +RUNNING: myrelease-feast-online-serving-test +PASSED: myrelease-feast-online-serving-test +RUNNING: myrelease-grafana-test +PASSED: myrelease-grafana-test +RUNNING: myrelease-test-topic-create-consume-produce +PASSED: myrelease-test-topic-create-consume-produce + +# Once the test completes, to check the logs +kubectl logs myrelease-feast-online-serving-test +``` + +> The test pods can be safely deleted after the test finishes. +> Check the yaml files in `templates/tests/` folder to see the processes +> the test pods execute. + +### Feast metrics + +Feast default installation includes Grafana, StatsD exporter and Prometheus. Request +metrics from Feast Core and Feast Serving, as well as ingestion statistic from +Feast Ingestion are accessible from Prometheus and Grafana dashboard. The following +show a quick example how to access the metrics. + +``` +# Forwards local port 9090 to the Prometheus server pod +kubectl port-forward svc/myrelease-prometheus-server 9090:80 +``` + +Visit http://localhost:9090 to access the Prometheus server: + +![Prometheus Server](files/img/prometheus-server.png?raw=true) + +### Enable Batch Serving + +To install Feast Batch Serving for retrieval of historical features in offline +training, access to BigQuery is required. First, create a [service account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) key that +will provide the credentials to access BigQuery. Grant the service account `editor` +role so it has write permissions to BigQuery and Cloud Storage. + +> In production, it is advised to give only the required [permissions](foo-feast-batch-serving-test) for the +> the service account, versus `editor` role which is very permissive. + +Create a Kubernetes secret for the service account JSON file: +```bash +# By default Feast expects the secret to be named "feast-gcp-service-account" +# and the JSON file to be named "credentials.json" +kubectl create secret generic feast-gcp-service-account --from-file=credentials.json +``` + +Create a new Cloud Storage bucket (if not exists) and make sure the service +account has write access to the bucket: +```bash +gsutil mb ``` -Install Feast release with minimal features, without batch serving and persistence: +Use the following Helm values to enable Batch Serving: +```yaml +# values-batch-serving.yaml +feast-core: + gcpServiceAccount: + enabled: true + postgresql: + existingSecret: feast-postgresql + +feast-batch-serving: + enabled: true + gcpServiceAccount: + enabled: true + application-override.yaml: + feast: + active_store: historical + stores: + - name: historical + type: BIGQUERY + config: + project_id: + dataset_id: + staging_location: gs:///feast-staging-location + initial_retry_delay_seconds: 3 + total_timeout_seconds: 21600 + subscriptions: + - name: "*" + project: "*" + version: "*" + +postgresql: + existingSecret: feast-postgresql +``` + +> To delete the previous release, run `helm delete --purge myrelease` +> Note this will not delete the persistent volume that has been claimed (PVC). +> In a test cluster, run `kubectl delete pvc --all` to delete all claimed PVCs. + ```bash -RELEASE_NAME=demo -helm install feast-charts/feast --name $RELEASE_NAME -f values-demo.yaml +# Install a new release +helm install --name myrelease -f values-batch-serving.yaml feast-charts/feast + +# Wait until all pods are created and running/completed (can take about 5m) +kubectl get pods + +# Batch Serving is installed so `helm test` will also test for batch retrieval +helm test myrelease ``` -Install Feast release for typical use cases, with batch and online serving: +### Use DataflowRunner for ingestion + +Apache Beam [DirectRunner](https://beam.apache.org/documentation/runners/direct/) +is not suitable for production use case because it is not easy to scale the +number of workers and there is no convenient API to monitor and manage the +workers. Feast supports [DataflowRunner](https://beam.apache.org/documentation/runners/dataflow/) which is a managed service on Google Cloud. + +> Make sure `feast-gcp-service-account` Kubernetes secret containing the +> service account has been created and the service account has permissions +> to manage Dataflow jobs. + +Since Dataflow workers run outside the Kube cluster and they will need to interact +with Kafka brokers, Redis stores and StatsD server installed in the cluster, +these services need to be exposed for access outside the cluster by setting +`service.type: LoadBalancer`. + +In a typical use case, 5 `LoadBalancer` (internal) IP addresses are required by +Feast when running with `DataflowRunner`. In Google Cloud, these (internal) IP +addresses should be reserved first: ```bash -# To install Feast Batch serving, BigQuery and Google Cloud service account -# is required. The service account needs to have these roles: -# - bigquery.dataEditor -# - bigquery.jobUser -# -# Assuming a service account JSON file has been downloaded to /home/user/key.json, -# run the following command to create a secret in Kubernetes -# (make sure the file name is called key.json): -kubectl create secret generic feast-gcp-service-account --from-file=/home/user/key.json - -# Set these required configuration in Feast Batch Serving -STAGING_LOCATION=gs://bucket/path -PROJECT_ID=google-cloud-project-id -DATASET_ID=bigquery-dataset-id - -# Install the Helm release using default values.yaml -helm install feast-charts/feast --name feast \ - --set feast-serving-batch."application\.yaml".feast.jobs.staging-location=$STAGING_LOCATION \ - --set feast-serving-batch."store\.yaml".bigquery_config.project_id=$PROJECT_ID \ - --set feast-serving-batch."store\.yaml".bigquery_config.dataset_id=$DATASET_ID +# Check with your network configuration which IP addresses are available for use +gcloud compute addresses create \ + feast-kafka-1 feast-kafka-2 feast-kafka-3 feast-redis feast-statsd \ + --region --subnet \ + --addresses 10.128.0.11,10.128.0.12,10.128.0.13,10.128.0.14,10.128.0.15 ``` -## Parameters - -The following table lists the configurable parameters of the Feast chart and their default values. - -| Parameter | Description | Default -| --------- | ----------- | ------- -| `feast-core.enabled` | Flag to install Feast Core | `true` -| `feast-core.postgresql.enabled` | Flag to install Postgresql as Feast database | `true` -| `feast-core.postgresql.postgresqlDatabase` | Name of the database used by Feast Core | `feast` -| `feast-core.postgresql.postgresqlUsername` | Username to authenticate to Feast database | `postgres` -| `feast-core.postgresql.postgresqlPassword` | Passsword to authenticate to Feast database | `password` -| `feast-core.kafka.enabled` | Flag to install Kafka as the default source for Feast | `true` -| `feast-core.kafka.topics[0].name` | Default topic name in Kafka| `feast` -| `feast-core.kafka.topics[0].replicationFactor` | No of replication factor for the topic| `1` -| `feast-core.kafka.topics[0].partitions` | No of partitions for the topic | `1` -| `feast-core.prometheus-statsd-exporter.enabled` | Flag to install Prometheus StatsD Exporter | `false` -| `feast-core.prometheus-statsd-exporter.*` | Refer to this [link](charts/feast-core/charts/prometheus-statsd-exporter/values.yaml | -| `feast-core.replicaCount` | No of pods to create | `1` -| `feast-core.image.repository` | Repository for Feast Core Docker image | `gcr.io/kf-feast/feast-core` -| `feast-core.image.tag` | Tag for Feast Core Docker image | `0.4.4` -| `feast-core.image.pullPolicy` | Image pull policy for Feast Core Docker image | `IfNotPresent` -| `feast-core.prometheus.enabled` | Add annotations to enable Prometheus scraping | `false` -| `feast-core.application.yaml` | Configuration for Feast Core application | Refer to this [link](charts/feast-core/values.yaml) -| `feast-core.springConfigMountPath` | Directory to mount application.yaml | `/etc/feast/feast-core` -| `feast-core.gcpServiceAccount.useExistingSecret` | Flag to use existing secret for GCP service account | `false` -| `feast-core.gcpServiceAccount.existingSecret.name` | Secret name for the service account | `feast-gcp-service-account` -| `feast-core.gcpServiceAccount.existingSecret.key` | Secret key for the service account | `key.json` -| `feast-core.gcpServiceAccount.mountPath` | Directory to mount the JSON key file | `/etc/gcloud/service-accounts` -| `feast-core.gcpProjectId` | Project ID to set `GOOGLE_CLOUD_PROJECT` to change default project used by SDKs | `""` -| `feast-core.jarPath` | Path to Jar file in the Docker image | `/opt/feast/feast-core.jar` -| `feast-core.jvmOptions` | Options for the JVM | `[]` -| `feast-core.logLevel` | Application logging level | `warn` -| `feast-core.logType` | Application logging type (`JSON` or `Console`) | `JSON` -| `feast-core.springConfigProfiles` | Map of profile name to file content for additional Spring profiles | `{}` -| `feast-core.springConfigProfilesActive` | CSV of profiles to enable from `springConfigProfiles` | `""` -| `feast-core.livenessProbe.enabled` | Flag to enable liveness probe | `true` -| `feast-core.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | `60` -| `feast-core.livenessProbe.periodSeconds` | How often to perform the probe | `10` -| `feast-core.livenessProbe.timeoutSeconds` | Timeout duration for the probe | `5` -| `feast-core.livenessProbe.successThreshold` | Minimum no of consecutive successes for the probe to be considered successful | `1` -| `feast-core.livenessProbe.failureThreshold` | Minimum no of consecutive failures for the probe to be considered failed | `5` -| `feast-core.readinessProbe.enabled` | Flag to enable readiness probe | `true` -| `feast-core.readinessProbe.initialDelaySeconds` | Delay before readiness probe is initiated | `30` -| `feast-core.readinessProbe.periodSeconds` | How often to perform the probe | `10` -| `feast-core.readinessProbe.timeoutSeconds` | Timeout duration for the probe | `10` -| `feast-core.readinessProbe.successThreshold` | Minimum no of consecutive successes for the probe to be considered successful | `1` -| `feast-core.service.type` | Kubernetes Service Type | `ClusterIP` -| `feast-core.http.port` | Kubernetes Service port for HTTP request| `80` -| `feast-core.http.targetPort` | Container port for HTTP request | `8080` -| `feast-core.grpc.port` | Kubernetes Service port for GRPC request| `6565` -| `feast-core.grpc.targetPort` | Container port for GRPC request| `6565` -| `feast-core.resources` | CPU and memory allocation for the pod | `{}` -| `feast-core.ingress` | See *Ingress Parameters* [below](#ingress-parameters) | `{}` -| `feast-serving-online.enabled` | Flag to install Feast Online Serving | `true` -| `feast-serving-online.redis.enabled` | Flag to install Redis in Feast Serving | `false` -| `feast-serving-online.redis.usePassword` | Flag to use password to access Redis | `false` -| `feast-serving-online.redis.cluster.enabled` | Flag to enable Redis cluster | `false` -| `feast-serving-online.core.enabled` | Flag for Feast Serving to use Feast Core in the same Helm release | `true` -| `feast-serving-online.replicaCount` | No of pods to create | `1` -| `feast-serving-online.image.repository` | Repository for Feast Serving Docker image | `gcr.io/kf-feast/feast-serving` -| `feast-serving-online.image.tag` | Tag for Feast Serving Docker image | `0.4.4` -| `feast-serving-online.image.pullPolicy` | Image pull policy for Feast Serving Docker image | `IfNotPresent` -| `feast-serving-online.prometheus.enabled` | Add annotations to enable Prometheus scraping | `true` -| `feast-serving-online.application.yaml` | Application configuration for Feast Serving | Refer to this [link](charts/feast-serving/values.yaml) -| `feast-serving-online.store.yaml` | Store configuration for Feast Serving | Refer to this [link](charts/feast-serving/values.yaml) -| `feast-serving-online.springConfigMountPath` | Directory to mount application.yaml and store.yaml | `/etc/feast/feast-serving` -| `feast-serving-online.gcpServiceAccount.useExistingSecret` | Flag to use existing secret for GCP service account | `false` -| `feast-serving-online.gcpServiceAccount.existingSecret.name` | Secret name for the service account | `feast-gcp-service-account` -| `feast-serving-online.gcpServiceAccount.existingSecret.key` | Secret key for the service account | `key.json` -| `feast-serving-online.gcpServiceAccount.mountPath` | Directory to mount the JSON key file | `/etc/gcloud/service-accounts` -| `feast-serving-online.gcpProjectId` | Project ID to set `GOOGLE_CLOUD_PROJECT` to change default project used by SDKs | `""` -| `feast-serving-online.jarPath` | Path to Jar file in the Docker image | `/opt/feast/feast-serving.jar` -| `feast-serving-online.jvmOptions` | Options for the JVM | `[]` -| `feast-serving-online.logLevel` | Application logging level | `warn` -| `feast-serving-online.logType` | Application logging type (`JSON` or `Console`) | `JSON` -| `feast-serving-online.springConfigProfiles` | Map of profile name to file content for additional Spring profiles | `{}` -| `feast-serving-online.springConfigProfilesActive` | CSV of profiles to enable from `springConfigProfiles` | `""` -| `feast-serving-online.livenessProbe.enabled` | Flag to enable liveness probe | `true` -| `feast-serving-online.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | `60` -| `feast-serving-online.livenessProbe.periodSeconds` | How often to perform the probe | `10` -| `feast-serving-online.livenessProbe.timeoutSeconds` | Timeout duration for the probe | `5` -| `feast-serving-online.livenessProbe.successThreshold` | Minimum no of consecutive successes for the probe to be considered successful | `1` -| `feast-serving-online.livenessProbe.failureThreshold` | Minimum no of consecutive failures for the probe to be considered failed | `5` -| `feast-serving-online.readinessProbe.enabled` | Flag to enable readiness probe | `true` -| `feast-serving-online.readinessProbe.initialDelaySeconds` | Delay before readiness probe is initiated | `30` -| `feast-serving-online.readinessProbe.periodSeconds` | How often to perform the probe | `10` -| `feast-serving-online.readinessProbe.timeoutSeconds` | Timeout duration for the probe | `10` -| `feast-serving-online.readinessProbe.successThreshold` | Minimum no of consecutive successes for the probe to be considered successful | `1` -| `feast-serving-online.service.type` | Kubernetes Service Type | `ClusterIP` -| `feast-serving-online.http.port` | Kubernetes Service port for HTTP request| `80` -| `feast-serving-online.http.targetPort` | Container port for HTTP request | `8080` -| `feast-serving-online.grpc.port` | Kubernetes Service port for GRPC request| `6566` -| `feast-serving-online.grpc.targetPort` | Container port for GRPC request| `6566` -| `feast-serving-online.resources` | CPU and memory allocation for the pod | `{}` -| `feast-serving-online.ingress` | See *Ingress Parameters* [below](#ingress-parameters) | `{}` -| `feast-serving-batch.enabled` | Flag to install Feast Batch Serving | `true` -| `feast-serving-batch.redis.enabled` | Flag to install Redis in Feast Serving | `false` -| `feast-serving-batch.redis.usePassword` | Flag to use password to access Redis | `false` -| `feast-serving-batch.redis.cluster.enabled` | Flag to enable Redis cluster | `false` -| `feast-serving-batch.core.enabled` | Flag for Feast Serving to use Feast Core in the same Helm release | `true` -| `feast-serving-batch.replicaCount` | No of pods to create | `1` -| `feast-serving-batch.image.repository` | Repository for Feast Serving Docker image | `gcr.io/kf-feast/feast-serving` -| `feast-serving-batch.image.tag` | Tag for Feast Serving Docker image | `0.4.4` -| `feast-serving-batch.image.pullPolicy` | Image pull policy for Feast Serving Docker image | `IfNotPresent` -| `feast-serving-batch.prometheus.enabled` | Add annotations to enable Prometheus scraping | `true` -| `feast-serving-batch.application.yaml` | Application configuration for Feast Serving | Refer to this [link](charts/feast-serving/values.yaml) -| `feast-serving-batch.store.yaml` | Store configuration for Feast Serving | Refer to this [link](charts/feast-serving/values.yaml) -| `feast-serving-batch.springConfigMountPath` | Directory to mount application.yaml and store.yaml | `/etc/feast/feast-serving` -| `feast-serving-batch.gcpServiceAccount.useExistingSecret` | Flag to use existing secret for GCP service account | `false` -| `feast-serving-batch.gcpServiceAccount.existingSecret.name` | Secret name for the service account | `feast-gcp-service-account` -| `feast-serving-batch.gcpServiceAccount.existingSecret.key` | Secret key for the service account | `key.json` -| `feast-serving-batch.gcpServiceAccount.mountPath` | Directory to mount the JSON key file | `/etc/gcloud/service-accounts` -| `feast-serving-batch.gcpProjectId` | Project ID to set `GOOGLE_CLOUD_PROJECT` to change default project used by SDKs | `""` -| `feast-serving-batch.jarPath` | Path to Jar file in the Docker image | `/opt/feast/feast-serving.jar` -| `feast-serving-batch.jvmOptions` | Options for the JVM | `[]` -| `feast-serving-batch.logLevel` | Application logging level | `warn` -| `feast-serving-batch.logType` | Application logging type (`JSON` or `Console`) | `JSON` -| `feast-serving-batch.springConfigProfiles` | Map of profile name to file content for additional Spring profiles | `{}` -| `feast-serving-batch.springConfigProfilesActive` | CSV of profiles to enable from `springConfigProfiles` | `""` -| `feast-serving-batch.livenessProbe.enabled` | Flag to enable liveness probe | `true` -| `feast-serving-batch.livenessProbe.initialDelaySeconds` | Delay before liveness probe is initiated | `60` -| `feast-serving-batch.livenessProbe.periodSeconds` | How often to perform the probe | `10` -| `feast-serving-batch.livenessProbe.timeoutSeconds` | Timeout duration for the probe | `5` -| `feast-serving-batch.livenessProbe.successThreshold` | Minimum no of consecutive successes for the probe to be considered successful | `1` -| `feast-serving-batch.livenessProbe.failureThreshold` | Minimum no of consecutive failures for the probe to be considered failed | `5` -| `feast-serving-batch.readinessProbe.enabled` | Flag to enable readiness probe | `true` -| `feast-serving-batch.readinessProbe.initialDelaySeconds` | Delay before readiness probe is initiated | `30` -| `feast-serving-batch.readinessProbe.periodSeconds` | How often to perform the probe | `10` -| `feast-serving-batch.readinessProbe.timeoutSeconds` | Timeout duration for the probe | `10` -| `feast-serving-batch.readinessProbe.successThreshold` | Minimum no of consecutive successes for the probe to be considered successful | `1` -| `feast-serving-batch.service.type` | Kubernetes Service Type | `ClusterIP` -| `feast-serving-batch.http.port` | Kubernetes Service port for HTTP request| `80` -| `feast-serving-batch.http.targetPort` | Container port for HTTP request | `8080` -| `feast-serving-batch.grpc.port` | Kubernetes Service port for GRPC request| `6566` -| `feast-serving-batch.grpc.targetPort` | Container port for GRPC request| `6566` -| `feast-serving-batch.resources` | CPU and memory allocation for the pod | `{}` -| `feast-serving-batch.ingress` | See *Ingress Parameters* [below](#ingress-parameters) | `{}` - -## Ingress Parameters - -The following table lists the configurable parameters of the ingress section for each Feast module. - -Note, there are two ingresses available for each module - `grpc` and `http`. - -| Parameter | Description | Default -| ----------------------------- | ----------- | ------- -| `ingress.grcp.enabled` | Enables an ingress (endpoint) for the gRPC server | `false` -| `ingress.grcp.*` | See below | -| `ingress.http.enabled` | Enables an ingress (endpoint) for the HTTP server | `false` -| `ingress.http.*` | See below | -| `ingress.*.class` | Value for `kubernetes.io/ingress.class` | `nginx` -| `ingress.*.hosts` | List of host-names for the ingress | `[]` -| `ingress.*.annotations` | Additional ingress annotations | `{}` -| `ingress.*.https.enabled` | Add a tls section to the ingress | `true` -| `ingress.*.https.secretNames` | Map of hostname to TLS secret name | `{}` If not specified, defaults to `domain-tld-tls` e.g. `feast.example.com` uses secret `example-com-tls` -| `ingress.*.auth.enabled` | Enable auth on the ingress (only applicable for `nginx` type | `false` -| `ingress.*.auth.signinHost` | External hostname of the OAuth2 proxy to use | First item in `ingress.hosts`, replacing the sub-domain with 'auth' e.g. `feast.example.com` uses `auth.example.com` -| `ingress.*.auth.authUrl` | Internal URI to internal auth endpoint | `http://auth-server.auth-ns.svc.cluster.local/auth` -| `ingress.*.whitelist` | Subnet masks to whitelist (i.e. value for `nginx.ingress.kubernetes.io/whitelist-source-range`) | `"""` - -To enable all the ingresses will a config like the following (while also adding the hosts etc): +Use the following Helm values to enable DataflowRuner (and Batch Serving), +replacing the `<*load_balancer_ip*>` tags with the ip addresses reserved above: ```yaml +# values-dataflow-runner.yaml feast-core: - ingress: - grpc: - enabled: true - http: - enabled: true -feast-serving-online: - ingress: - grpc: - enabled: true - http: - enabled: true -feast-serving-batch: - ingress: - grpc: - enabled: true - http: - enabled: true + gcpServiceAccount: + enabled: true + postgresql: + existingSecret: feast-postgresql + application-override.yaml: + feast: + stream: + options: + bootstrapServers: + jobs: + active_runner: dataflow + metrics: + host: + runners: + - name: dataflow + type: DataflowRunner + options: + project: + region: + zone: + tempLocation: + network: + subnetwork: + maxNumWorkers: 1 + autoscalingAlgorithm: THROUGHPUT_BASED + usePublicIps: false + workerMachineType: n1-standard-1 + deadLetterTableSpec: + +feast-online-serving: + application-override.yaml: + feast: + stores: + - name: online + type: REDIS + config: + host: + port: 6379 + subscriptions: + - name: "*" + project: "*" + version: "*" + +feast-batch-serving: + enabled: true + gcpServiceAccount: + enabled: true + application-override.yaml: + feast: + active_store: historical + stores: + - name: historical + type: BIGQUERY + config: + project_id: + dataset_id: + staging_location: gs:///feast-staging-location + initial_retry_delay_seconds: 3 + total_timeout_seconds: 21600 + subscriptions: + - name: "*" + project: "*" + version: "*" + +postgresql: + existingSecret: feast-postgresql + +kafka: + external: + enabled: true + type: LoadBalancer + annotations: + cloud.google.com/load-balancer-type: Internal + loadBalancerSourceRanges: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + firstListenerPort: 31090 + loadBalancerIP: + - + - + - + configurationOverrides: + "advertised.listeners": |- + EXTERNAL://${LOAD_BALANCER_IP}:31090 + "listener.security.protocol.map": |- + PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT + "log.retention.hours": 1 + +redis: + master: + service: + type: LoadBalancer + loadBalancerIP: + annotations: + cloud.google.com/load-balancer-type: Internal + loadBalancerSourceRanges: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + +prometheus-statsd-exporter: + service: + type: LoadBalancer + annotations: + cloud.google.com/load-balancer-type: Internal + loadBalancerSourceRanges: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + loadBalancerIP: +``` + +```bash +# Install a new release +helm install --name myrelease -f values-dataflow-runner.yaml feast-charts/feast + +# Wait until all pods are created and running/completed (can take about 5m) +kubectl get pods + +# Test the installation +helm test myrelease ``` +If the tests are successful, Dataflow jobs should appear in Google Cloud console +running features ingestion: https://console.cloud.google.com/dataflow + +![Dataflow Jobs](files/img/dataflow-jobs.png) + +### Production configuration + +#### Resources requests + +The `resources` field in the deployment spec is left empty in the examples. In +production these should be set according to the load each services are expected +to handle and the service level objectives (SLO). Also Feast Core and Serving +is Java application and it is [good practice](https://stackoverflow.com/a/6916718/3949303) +to set the minimum and maximum heap. This is an example reasonable value to set for Feast Serving: + +```yaml +feast-online-serving: + javaOpts: "-Xms2048m -Xmx2048m" + resources: + limits: + memory: "2048Mi" + requests: + memory: "2048Mi" + cpu: "1" +``` + +#### High availability + +Default Feast installation only configures a single instance of Redis +server. If due to network failures or out of memory error Redis is down, +Feast serving will fail to respond to requests. Soon, Feast will support +highly available Redis via [Redis cluster](https://redis.io/topics/cluster-tutorial), +sentinel or additional proxies. + +### Documentation development + +This `README.md` is generated using [helm-docs](https://github.com/norwoodj/helm-docs/). +Please run `helm-docs` to regenerate the `README.md` every time `README.md.gotmpl` +or `values.yaml` are updated. diff --git a/infra/charts/feast/README.md.gotmpl b/infra/charts/feast/README.md.gotmpl new file mode 100644 index 00000000000..69d40fbb25e --- /dev/null +++ b/infra/charts/feast/README.md.gotmpl @@ -0,0 +1,354 @@ +{{ template "chart.header" . }} + +{{ template "chart.description" . }} {{ template "chart.versionLine" . }} + +## TL;DR; + +```bash +# Add Feast Helm chart +helm repo add feast-charts https://feast-charts.storage.googleapis.com +helm repo update + +# Create secret for Feast database, replace with the desired value +kubectl create secret generic feast-postgresql \ + --from-literal=postgresql-password= + +# Install Feast with Online Serving and Beam DirectRunner +helm install --name myrelease feast-charts/feast \ + --set feast-core.postgresql.existingSecret=feast-postgresql \ + --set postgresql.existingSecret=feast-postgresql +``` + +## Introduction +This chart install Feast deployment on a Kubernetes cluster using the [Helm](https://v2.helm.sh/docs/using_helm/#installing-helm) package manager. + +## Prerequisites +- Kubernetes 1.12+ +- Helm 2.15+ (not tested with Helm 3) +- Persistent Volume support on the underlying infrastructure + +{{ template "chart.requirementsSection" . }} + +{{ template "chart.valuesSection" . }} + +## Configuration and installation details + +The default configuration will install Feast with Online Serving. Ingestion +of features will use Beam [DirectRunner](https://beam.apache.org/documentation/runners/direct/) +that runs on the same container where Feast Core is running. + +```bash +# Create secret for Feast database, replace accordingly +kubectl create secret generic feast-postgresql \ + --from-literal=postgresql-password= + +# Install Feast with Online Serving and Beam DirectRunner +helm install --name myrelease feast-charts/feast \ + --set feast-core.postgresql.existingSecret=feast-postgresql \ + --set postgresql.existingSecret=feast-postgresql +``` + +In order to test that the installation is successful: +```bash +helm test myrelease + +# If the installation is successful, the following should be printed +RUNNING: myrelease-feast-online-serving-test +PASSED: myrelease-feast-online-serving-test +RUNNING: myrelease-grafana-test +PASSED: myrelease-grafana-test +RUNNING: myrelease-test-topic-create-consume-produce +PASSED: myrelease-test-topic-create-consume-produce + +# Once the test completes, to check the logs +kubectl logs myrelease-feast-online-serving-test +``` + +> The test pods can be safely deleted after the test finishes. +> Check the yaml files in `templates/tests/` folder to see the processes +> the test pods execute. + +### Feast metrics + +Feast default installation includes Grafana, StatsD exporter and Prometheus. Request +metrics from Feast Core and Feast Serving, as well as ingestion statistic from +Feast Ingestion are accessible from Prometheus and Grafana dashboard. The following +show a quick example how to access the metrics. + +``` +# Forwards local port 9090 to the Prometheus server pod +kubectl port-forward svc/myrelease-prometheus-server 9090:80 +``` + +Visit http://localhost:9090 to access the Prometheus server: + +![Prometheus Server](files/img/prometheus-server.png?raw=true) + +### Enable Batch Serving + +To install Feast Batch Serving for retrieval of historical features in offline +training, access to BigQuery is required. First, create a [service account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) key that +will provide the credentials to access BigQuery. Grant the service account `editor` +role so it has write permissions to BigQuery and Cloud Storage. + +> In production, it is advised to give only the required [permissions](foo-feast-batch-serving-test) for the +> the service account, versus `editor` role which is very permissive. + +Create a Kubernetes secret for the service account JSON file: +```bash +# By default Feast expects the secret to be named "feast-gcp-service-account" +# and the JSON file to be named "credentials.json" +kubectl create secret generic feast-gcp-service-account --from-file=credentials.json +``` + +Create a new Cloud Storage bucket (if not exists) and make sure the service +account has write access to the bucket: +```bash +gsutil mb +``` + +Use the following Helm values to enable Batch Serving: +```yaml +# values-batch-serving.yaml +feast-core: + gcpServiceAccount: + enabled: true + postgresql: + existingSecret: feast-postgresql + +feast-batch-serving: + enabled: true + gcpServiceAccount: + enabled: true + application-override.yaml: + feast: + active_store: historical + stores: + - name: historical + type: BIGQUERY + config: + project_id: + dataset_id: + staging_location: gs:///feast-staging-location + initial_retry_delay_seconds: 3 + total_timeout_seconds: 21600 + subscriptions: + - name: "*" + project: "*" + version: "*" + +postgresql: + existingSecret: feast-postgresql +``` + +> To delete the previous release, run `helm delete --purge myrelease` +> Note this will not delete the persistent volume that has been claimed (PVC). +> In a test cluster, run `kubectl delete pvc --all` to delete all claimed PVCs. + +```bash +# Install a new release +helm install --name myrelease -f values-batch-serving.yaml feast-charts/feast + +# Wait until all pods are created and running/completed (can take about 5m) +kubectl get pods + +# Batch Serving is installed so `helm test` will also test for batch retrieval +helm test myrelease +``` + +### Use DataflowRunner for ingestion + +Apache Beam [DirectRunner](https://beam.apache.org/documentation/runners/direct/) +is not suitable for production use case because it is not easy to scale the +number of workers and there is no convenient API to monitor and manage the +workers. Feast supports [DataflowRunner](https://beam.apache.org/documentation/runners/dataflow/) which is a managed service on Google Cloud. + +> Make sure `feast-gcp-service-account` Kubernetes secret containing the +> service account has been created and the service account has permissions +> to manage Dataflow jobs. + +Since Dataflow workers run outside the Kube cluster and they will need to interact +with Kafka brokers, Redis stores and StatsD server installed in the cluster, +these services need to be exposed for access outside the cluster by setting +`service.type: LoadBalancer`. + +In a typical use case, 5 `LoadBalancer` (internal) IP addresses are required by +Feast when running with `DataflowRunner`. In Google Cloud, these (internal) IP +addresses should be reserved first: +```bash +# Check with your network configuration which IP addresses are available for use +gcloud compute addresses create \ + feast-kafka-1 feast-kafka-2 feast-kafka-3 feast-redis feast-statsd \ + --region --subnet \ + --addresses 10.128.0.11,10.128.0.12,10.128.0.13,10.128.0.14,10.128.0.15 +``` + +Use the following Helm values to enable DataflowRuner (and Batch Serving), +replacing the `<*load_balancer_ip*>` tags with the ip addresses reserved above: + +```yaml +# values-dataflow-runner.yaml +feast-core: + gcpServiceAccount: + enabled: true + postgresql: + existingSecret: feast-postgresql + application-override.yaml: + feast: + stream: + options: + bootstrapServers: + jobs: + active_runner: dataflow + metrics: + host: + runners: + - name: dataflow + type: DataflowRunner + options: + project: + region: + zone: + tempLocation: + network: + subnetwork: + maxNumWorkers: 1 + autoscalingAlgorithm: THROUGHPUT_BASED + usePublicIps: false + workerMachineType: n1-standard-1 + deadLetterTableSpec: + +feast-online-serving: + application-override.yaml: + feast: + stores: + - name: online + type: REDIS + config: + host: + port: 6379 + subscriptions: + - name: "*" + project: "*" + version: "*" + +feast-batch-serving: + enabled: true + gcpServiceAccount: + enabled: true + application-override.yaml: + feast: + active_store: historical + stores: + - name: historical + type: BIGQUERY + config: + project_id: + dataset_id: + staging_location: gs:///feast-staging-location + initial_retry_delay_seconds: 3 + total_timeout_seconds: 21600 + subscriptions: + - name: "*" + project: "*" + version: "*" + +postgresql: + existingSecret: feast-postgresql + +kafka: + external: + enabled: true + type: LoadBalancer + annotations: + cloud.google.com/load-balancer-type: Internal + loadBalancerSourceRanges: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + firstListenerPort: 31090 + loadBalancerIP: + - + - + - + configurationOverrides: + "advertised.listeners": |- + EXTERNAL://${LOAD_BALANCER_IP}:31090 + "listener.security.protocol.map": |- + PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT + "log.retention.hours": 1 + +redis: + master: + service: + type: LoadBalancer + loadBalancerIP: + annotations: + cloud.google.com/load-balancer-type: Internal + loadBalancerSourceRanges: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + +prometheus-statsd-exporter: + service: + type: LoadBalancer + annotations: + cloud.google.com/load-balancer-type: Internal + loadBalancerSourceRanges: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + loadBalancerIP: +``` + +```bash +# Install a new release +helm install --name myrelease -f values-dataflow-runner.yaml feast-charts/feast + +# Wait until all pods are created and running/completed (can take about 5m) +kubectl get pods + +# Test the installation +helm test myrelease +``` + +If the tests are successful, Dataflow jobs should appear in Google Cloud console +running features ingestion: https://console.cloud.google.com/dataflow + +![Dataflow Jobs](files/img/dataflow-jobs.png) + +### Production configuration + +#### Resources requests + +The `resources` field in the deployment spec is left empty in the examples. In +production these should be set according to the load each services are expected +to handle and the service level objectives (SLO). Also Feast Core and Serving +is Java application and it is [good practice](https://stackoverflow.com/a/6916718/3949303) +to set the minimum and maximum heap. This is an example reasonable value to set for Feast Serving: + +```yaml +feast-online-serving: + javaOpts: "-Xms2048m -Xmx2048m" + resources: + limits: + memory: "2048Mi" + requests: + memory: "2048Mi" + cpu: "1" +``` + +#### High availability + +Default Feast installation only configures a single instance of Redis +server. If due to network failures or out of memory error Redis is down, +Feast serving will fail to respond to requests. Soon, Feast will support +highly available Redis via [Redis cluster](https://redis.io/topics/cluster-tutorial), +sentinel or additional proxies. + +### Documentation development + +This `README.md` is generated using [helm-docs](https://github.com/norwoodj/helm-docs/). +Please run `helm-docs` to regenerate the `README.md` every time `README.md.gotmpl` +or `values.yaml` are updated. diff --git a/infra/charts/feast/charts/feast-core/.helmignore b/infra/charts/feast/charts/feast-core/.helmignore deleted file mode 100644 index 50af0317254..00000000000 --- a/infra/charts/feast/charts/feast-core/.helmignore +++ /dev/null @@ -1,22 +0,0 @@ -# Patterns to ignore when building packages. -# This supports shell glob matching, relative path matching, and -# negation (prefixed with !). Only one pattern per line. -.DS_Store -# Common VCS dirs -.git/ -.gitignore -.bzr/ -.bzrignore -.hg/ -.hgignore -.svn/ -# Common backup files -*.swp -*.bak -*.tmp -*~ -# Various IDEs -.project -.idea/ -*.tmproj -.vscode/ diff --git a/infra/charts/feast/charts/feast-core/Chart.yaml b/infra/charts/feast/charts/feast-core/Chart.yaml index 86d0699b9ac..5b832943cfe 100644 --- a/infra/charts/feast/charts/feast-core/Chart.yaml +++ b/infra/charts/feast/charts/feast-core/Chart.yaml @@ -1,4 +1,4 @@ apiVersion: v1 -description: A Helm chart for core component of Feast +description: Feast Core registers feature specifications and manage ingestion jobs. name: feast-core -version: 0.4.4 +version: 0.5.0-alpha.1 diff --git a/infra/charts/feast/charts/feast-core/README.md b/infra/charts/feast/charts/feast-core/README.md new file mode 100644 index 00000000000..4bf4578eb75 --- /dev/null +++ b/infra/charts/feast/charts/feast-core/README.md @@ -0,0 +1,70 @@ +feast-core +========== +Feast Core registers feature specifications and manage ingestion jobs. + +Current chart version is `0.5.0-alpha.1` + + + + + +## Chart Values + +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| "application-generated.yaml".enabled | bool | `true` | Flag to include Helm generated configuration for Feast database URL, Kafka bootstrap servers and jobs metrics host. This is useful for deployment that uses default configuration for Kafka, Postgres and StatsD exporter. Please set `application-override.yaml` to override this configuration. | +| "application-override.yaml" | object | `{"enabled":true}` | Configuration to override the default [application.yaml](https://github.com/gojek/feast/blob/master/core/src/main/resources/application.yml). Will be created as a ConfigMap. `application-override.yaml` has a higher precedence than `application-secret.yaml` | +| "application-secret.yaml" | object | `{"enabled":true}` | Configuration to override the default [application.yaml](https://github.com/gojek/feast/blob/master/core/src/main/resources/application.yml). Will be created as a Secret. `application-override.yaml` has a higher precedence than `application-secret.yaml`. It is recommended to either set `application-override.yaml` or `application-secret.yaml` only to simplify config management. | +| "application.yaml".enabled | bool | `true` | Flag to include the default [configuration](https://github.com/gojek/feast/blob/master/core/src/main/resources/application.yml). Please set `application-override.yaml` to override this configuration. | +| envOverrides | object | `{}` | Extra environment variables to set | +| gcpProjectId | string | `""` | Project ID to use when using Google Cloud services such as BigQuery, Cloud Storage and Dataflow | +| gcpServiceAccount.enabled | bool | `false` | Flag to use [service account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) JSON key | +| gcpServiceAccount.existingSecret.key | string | `"credentials.json"` | Key in the secret data (file name of the service account) | +| gcpServiceAccount.existingSecret.name | string | `"feast-gcp-service-account"` | Name of the existing secret containing the service account | +| image.pullPolicy | string | `"IfNotPresent"` | Image pull policy | +| image.repository | string | `"gcr.io/kf-feast/feast-core"` | Docker image repository | +| image.tag | string | `"dev"` | Image tag | +| ingress.grpc.annotations | object | `{}` | Extra annotations for the ingress | +| ingress.grpc.auth.enabled | bool | `false` | Flag to enable auth | +| ingress.grpc.class | string | `"nginx"` | Which ingress controller to use | +| ingress.grpc.enabled | bool | `false` | Flag to create an ingress resource for the service | +| ingress.grpc.hosts | list | `[]` | List of hostnames to match when routing requests | +| ingress.grpc.https.enabled | bool | `true` | Flag to enable HTTPS | +| ingress.grpc.https.secretNames | object | `{}` | Map of hostname to TLS secret name | +| ingress.grpc.whitelist | string | `""` | Allowed client IP source ranges | +| ingress.http.annotations | object | `{}` | Extra annotations for the ingress | +| ingress.http.auth.authUrl | string | `"http://auth-server.auth-ns.svc.cluster.local/auth"` | URL to an existing authentication service | +| ingress.http.auth.enabled | bool | `false` | Flag to enable auth | +| ingress.http.class | string | `"nginx"` | Which ingress controller to use | +| ingress.http.enabled | bool | `false` | Flag to create an ingress resource for the service | +| ingress.http.hosts | list | `[]` | List of hostnames to match when routing requests | +| ingress.http.https.enabled | bool | `true` | Flag to enable HTTPS | +| ingress.http.https.secretNames | object | `{}` | Map of hostname to TLS secret name | +| ingress.http.whitelist | string | `""` | Allowed client IP source ranges | +| javaOpts | string | `nil` | [JVM options](https://docs.oracle.com/cd/E22289_01/html/821-1274/configuring-the-default-jvm-and-java-arguments.html). For better performance, it is advised to set the min and max heap:
    `-Xms2048m -Xmx2048m` | +| livenessProbe.enabled | bool | `true` | Flag to enabled the probe | +| livenessProbe.failureThreshold | int | `5` | Min consecutive failures for the probe to be considered failed | +| livenessProbe.initialDelaySeconds | int | `60` | Delay before the probe is initiated | +| livenessProbe.periodSeconds | int | `10` | How often to perform the probe | +| livenessProbe.successThreshold | int | `1` | Min consecutive success for the probe to be considered successful | +| livenessProbe.timeoutSeconds | int | `5` | When the probe times out | +| logLevel | string | `"WARN"` | Default log level, use either one of `DEBUG`, `INFO`, `WARN` or `ERROR` | +| logType | string | `"Console"` | Log format, either `JSON` or `Console` | +| nodeSelector | object | `{}` | Node labels for pod assignment | +| postgresql.existingSecret | string | `""` | Existing secret to use for authenticating to Postgres | +| prometheus.enabled | bool | `true` | Flag to enable scraping of Feast Core metrics | +| readinessProbe.enabled | bool | `true` | Flag to enabled the probe | +| readinessProbe.failureThreshold | int | `5` | Min consecutive failures for the probe to be considered failed | +| readinessProbe.initialDelaySeconds | int | `20` | Delay before the probe is initiated | +| readinessProbe.periodSeconds | int | `10` | How often to perform the probe | +| readinessProbe.successThreshold | int | `1` | Min consecutive success for the probe to be considered successful | +| readinessProbe.timeoutSeconds | int | `10` | When the probe times out | +| replicaCount | int | `1` | Number of pods that will be created | +| resources | object | `{}` | CPU/memory [resource requests/limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container) | +| service.grpc.nodePort | string | `nil` | Port number that each cluster node will listen to | +| service.grpc.port | int | `6565` | Service port for GRPC requests | +| service.grpc.targetPort | int | `6565` | Container port serving GRPC requests | +| service.http.nodePort | string | `nil` | Port number that each cluster node will listen to | +| service.http.port | int | `80` | Service port for HTTP requests | +| service.http.targetPort | int | `8080` | Container port serving HTTP requests | +| service.type | string | `"ClusterIP"` | Kubernetes service type | diff --git a/infra/charts/feast/charts/feast-core/charts/kafka-0.20.1.tgz b/infra/charts/feast/charts/feast-core/charts/kafka-0.20.1.tgz deleted file mode 100644 index 76a2247577d43a5c7b7da189eab9ca2bb90e30ab..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 30761 zcmV)yK$5>7iwG0|00000|0w_~VMtOiV@ORlOnEsqVl!4SWK%V1T2nbTPgYhoO;>Dc zVQyr3R8em|NM&qo0PMZ%cH20zFuH&9De9B6ciSD2l6*^|Z@SO86L*g%7yHES%S=x% z8zLbIF-foq(6&0xx6W&v*E>&g7773fQj{!TlFr@}uGJlx#HCOu6sihUg>y1KC%xxW zlFML5lK4M9!qe~f`%iXu@ZWyF-~4y5xw&h9>-U4r!PeG)z{h}P?kR*M`G55L_id*+ zxc?>($=FXc7mTMvxEOdO%gR4@f_^aYA}Ye1WfEWQWh9(Zc&Ug6n1HMp#Y{|nQH)fn z5ru>bK_)DnKu+b97n7+3B7i`|gv{BfkTmiDLY^c{Ld++DH{}T(!c@vk40}B!$beu+ z0naBrkARVN{N7+kP8CLzxy z`Rhcz4nm%kwWfVJVtG_m8PW8wBI9yOb6zVsqUk#};c+yg`NY%j^mqg>@{l5|7Aq4b zr^QIs@0}MTnx|A!(F@hG7Co6$+DnL#H1DxAEJj4~yyxPirI~;5n3B|K&9}&GDID^2 z92Yc|EDd{M=0`j{r@61EroD=uK|P(`g{8TEu&Idd&t4V(K9A=A7bGsI_*fgja{j;F z-`d%3^8c+TyI=VKr+C)Df6&KU)4Rjad+?UMwKX`IQaCoO`q39KY z=KiE$5w$ztT9#8kE8>`?6U%ZN?GB-HFn+`3QBDO-r5=)klue3Up+avdkkj7_Dx?6C zMi8@vNpU-nLY`!Wq`qxY^t|%t5Z-_A6pD1iL;MX8X2lRT`~AfHozR5mvmp$2wsv1L zd}+a|RT;EY>2GeoW**Md-zdorQEU-I==i@RSA!qG|0NNFAN26`M-nSM=-nTVX;|b; z&LHHeq*u}t=EZaUH;$l<41dfbBGs#$NIIFp<&=d}Gwqiwj-e0~E~hkwa7xk%6+m(d zB+Ft(qo9Uj(T?Mgry^cxGFLZh5z zisBE`7c{1lYLpyHl1rR>A$g`(6ieZVCN!0Prq@))qu1q2yB^$e;Mji@svTfsD1;JUNjihD5RxZA3OVID`-A4H;1Lms z6~Pknaq7IVF;I9WGx(LPR7{dVug^J~=K%dMq9Freo(AND3sD3dfRy2&XhEVw!h>29p3f z3cT~qYV=9Ox#*E7VW|Q)_rthA5j(Ve)lZ68vMi>4*^lUXYio)y?29*QwiWwNPSD;lb;&v^;JGny-{bO;A&?Ss}# zhP8KAuea|c&v?uyvzMynPMu<=Tu8+cJ0%7hqt|qM>*0eBmy}JWatJ&93Oc8Q6PhPA zxEJE<29zzv+A_o~z@2HXQvRKftKm^m=^GwVJsyDqu3{v5{i!|nAr~?71%oqB{ZS&& z%3^8@SBp+SmUCPcs=-^Z@QPonZp$sqc^|V2s(SEq+n0}hX>MM=JW}tJaHZy-k@y9T z$?TYhJdKnV()R!|nlm1i?*{6fWC`VkEMM+;0HO#(D#Xb&r((+EXb1xjU`$wC-cH;DpCi)1jtnq&<*R zB4J7{l+MReiF{j5X%cv@&^Y4-T#{6(Nr-5qq*Q7f;P9?c!f%n5jO+F*13TB4E5Yn} z)o(F`Uw`w~ze5FO?lzpdG?;G$SfW zP@fl`(Nsthp8G7-VEIY~^GBwOA#`M}@CFd+_CU$l2c~cZIKH;S9fGQaogY}O!w%a* zqX+5^2PK5ESd@w`iY*ZnlBFTipK2CHH0GB9wvK|fmYtPm4||Z)kmnH+IVIuLwT3Hx z(9=sQfPe4;#FQ6tq)0x6gk1QW$RmC&v} z)W*V6wQzEH@#YvuOmhXX9ZQ8bgAJq~7$D(*Rw-Yhsz7u7X=~7brdVpk6T;FV7?C_s zVp8k@#7^f2N4R1%x7KrJ?LhG*kH@$+Lm8h&G>iF+=AEjHHv05~ZRg!mYYXg^^k#%N z5=9hQb|uz%D4G?Sv23?<1BB6>k%#RG(=>b+nhFVGP3 zT@g*FEL%v()sHE;AS_mEZwLeR@)a6xV9GBcQTnkXktSrOz$>ac5EWDbdcor&l_a03 zc{QDhpk{&6csabFdCnp#g33Tvb|R>hES(6=Xptggc21G@HA8j{GOBUZJ(;H2G0zhs zyFi2y(IYT=D<6+p$ka;Wp=c2rnp;q9|LN8D9>uq6OuHPBBCpE27>FWMEHFr@%vmS` z&EPtQ-W8@_Fri-Ic4o|py0Fr{6NW+}&QpgJk_Ry*5zT{~8je)8Sh5&hP;-Gm@~qZx zRjO_KT(MX}u7X?vNks?EbDk<|4Q-;-Gc?<&^L)5qq{g82Jm_D3X?wh3HmtesRL??1 z{I(4>rQszzXKo`F#UbrQOk^>c^vp<8Q&Hp;7l5|sApvVvr~5k?FW5ksK;&j-|6N-E(GaeEV^X#J zyTp2pl@1@RuOqL8$MEPqV(av~w}&tH-<}@4_^`2YGdOcwTDgHKwK^~89pUsJt*^I1 z)PUdaA@_k27whf8eDwa+;oghW@Ah8py?L&{JNhuR`_|eT{FsJ3Q8cY3NyV({8>-#$ zl)58%$YTwG?KZHBAh6$qgk&x(6_u60x+Q$&D`e1$yV5I}6WD+AldzV=N=~HdMaAQA zyFyC|S|%uJ07^O_QN)a`T~Q#?I=#xcHUa5XDqvkRX-SnT390%ct_+GO+096z6qA}A zOyNeKwCE;K5KF>@?A|H zXntjXP%Cwnu*jdV9cvOu4Ueq!HY1D)ZRJXfWJGf-2r8_` zxY;K~IE7eA_9(U?*Hf?!p)>e_bsUHh9@Xw*i?J~lCUl7xk7HZGXj=UF-_TPsp4zpNwk*A~nTrSq7M(jtz#kn*|C zKx8xoX2zmqYT1Qw$~h_?PHR=qN(F%RNw%rh>E6%BY6K*Wh!Xl!+s+R!_q*yN{1+T% zG(A>&(nh%n*1#Bg-n-<=?CNieSS!aQ%i@{U8;;853V(n73+!Jhj)8FC>C)wt#fqGi z8a_s~MQGi8L}?0POo_scR*@7ph`PEnsrTBPf{>cD@F&sC2g2gjk-7t^WB-86sX=ziEOAU`u z04bDMe#T_U;GCa(g~oH*R_h78oV*tsbQvELZ&+7V=G#wB%uKuBx?Tg zuuLCQPGSsnfx&WdZnaN zc0o75ZpXZkTB(LJZuUH_zntk4Qz~s|XO~kZsoIbaNAGBe$1Z4|Il~h5+D7BTs_-tk z`Z14(rT(T?OP;V{uh)B*My#iQ4>q3!{h%KVhC4euJH2|2jXouyWK4fag&eI`ug0DN}1RacZ(lnAbB5;SO(6QV|OhH`aKL$zhLhVi2?p&EaeT$R+I zNZqV%wX*a{50rBvoD65Z9Z6@t_+RSzVm z=~ogeISv_AC4hmDvzTtc`e0-E8c{uNmMf1&xc111U(mU=r{=l6DOl6y9dlX~RPGYP zubtOg(lM`;Kp4a%oz#k?R9^D@y!I~Ustihl)%YyR1@z#x!JiWA$`WSlTd>IuEB>z7 zK6!_St%~?jQW4Iam~C{VfUxLR%@YHBh71;i}xxWn(H+{*$mUjmcXR96ZW!&5@qmQU_{r|hU~_kCGu-S`vOA(rcAq}o ziN@h(Ka8G@$J_n!Xmji7_U_Kk6Z*8`I7%hIATbbWmXm{I5#}VTGy}cs41udoc3y-pqN9?H2MPy;YC8n(3k5Eix{5GD?b>V@lO7$4!f# zKA<#=%1mI>JIm}4rH7XOyY8iB-<57xKUBMq>#9|&J6KS+-+#@#xn-e6^ZA7l@+6BH zntbw^J4AOKd7u~RM_3^dNg0iQ7w^m(vGedStKni0YzF;*N1CoyB#!E z#Yzyp5{EpA9ohmA8?dwVJ<;eR&6;o2qKv@5XuG;(Js3 zI_H1W_}b=!Ggrj%!`A$Y)fnK zLN7oh`pYSiLT_z`@s2qf5uWp$9v;K5PHzXgEBrUtMpW_JdTXduPPyng!nMUvElz7n z<77?7QNB^Z=JsdNtWSq;+tVoKBey4~P*2h%i;1LeO^gg$d$qgcLi4g1)e!!`ASuFh zbPzX2Y3-`-K*BpdsvGgEQ{2Dvkx|&JSsx7&+8WNs zU=BO2BUQ*#!6K>%%51-3)ddmgy`}ACYpZR=?XLds_(=G9k@}MRC=OG7_^HMr)Pcy8$w}RhV8I)|H3WN5F(3UZ6o-l_o zX3$e2j8PWD0JO`u4+mrTDkHhXgB9@=_WG3x)lsj$Cu%Og(xzI(I#U}CHU2Bu1KKrV z9d{w$XJbF*(x+EUND*wbcGGR|L-(!u+K~%XvvXSryRX^>$yIkORlo@ioYN1Cu_HWG zN&x{xOQ-^W0Xmqdcl;H)gMZZ^ysAc^oqO;!WNi%ui)kw3nYKuhF^y+Pc8Y&t6ZEpQ znlh38fB(<_D?unpTGi1fIZ6~Yt`Ttr>nxxF5O{NVcob~F3+;n)sSuS^LB0!rpt=fhw#+Ao5P59n$@40 z?Ko~L3;JH6&#LAddWn^OW^!7?&5D3-nNKys9YROY2@?EhCaLJ!zY@{;7^vS1<5fov zK*G3;ytC0mLxt$bt3=gmaSq{E^vQ}w{_l{1F5B|B2?hdGfHT*YPJeSM9h<0w^V>tS$a&IU# zZ0osE3+b`m8*c6jtpFLXi_OaGq6#stnrH?a-*OU`gd1D?D1x)h5w@h`8s%d88MSd5 zCx{3L{P^}25=_PwQRlRBAJhkDDGk-=i#*1IJ!=Ns96TMAr+gpBD)gyrLMvE=6vADD z5C)(t_LwsQBO8aTNC%AqzBw{v*5T2`j}qMes6~)z@A0{)F}tu>b9l0U9LOvAfPLJM|7mxyJ!s_r*xKFt zlKJm$}AW<$+&-({2(22ot;HtuC26>U9lK=t1(B|yySudXo z8;xNcLbTp0r=|CgjRPn@qc>3_K^4F`OQWG;?jArwC5ebMCw*GFl~l+uX4c;682sM9 zhv2QrKv05ym)wN_}Zth41|pla_YZI#kTejh#zo9|A@ME9WEP7;zv zm8HrDF?Ig%L+6hUy`mvVrM>%O=tcCRmlko{@w`LDxo5mE2S!S%EXS-(zUV4prlX5R z)tsR#ijo`kSd4zI<^VBm;$6T`7h(!DOjU2bFCmXqHnTen%a^Qm zXdzDpk176JBs2%#A8{^)%t_{JpBB|JOT)N`D0H;f%D9MQ++HD2#QLoNY;$h7YUTSQ zI$=mXI3UahqUg*znS;=0B!<%P$Yco5Wgb6%ZffXEBKoMAgy>6UkYW6ct5${&iIB2! z-`9?hNMbkQ!Y>0D=aEvUPdf_3*fa*rNV=Ny&2#{PKJwS4QIu6^f(}(E>8LiT_MDnS z2*`X^q0>0i2_w;WN^%Vh6YgOCht|NOE@WDiRm7^$*tvNDbTnI5fT6sol$4fEyTEgO z8qMNZ`^M%u&jZg>xE_~m&<0nSFrrAY!=<|pFJ$YsgWS$GqM&QW5REC3g|$ptXWGE? zzW(~>y|-@;-uy89`fFFD#zM9d(LSZdD;;Sm4Oszf#)6R zNf4AekbNcK?f%}2*ZV=DW1`S0IiiZ?Yh_vKT^P<)62T=CQ%3<>fH^bgWdOVOxh&D8 z4Fh(Jl0(K>ihdsotNM&fO^b=l2*7}EJZND|FCk%RA&rLyiDQol!9#|+YpIv(gX54V zRH@amoX(U+@%}x)BenRD`i9>uUzb?HhetM>m#SbXt9*B@*o;%y2^V$Cmz3y!$GT2G zoa;JOb2+9ldc?Z$2;nVK$zlg0v%O<=+l3pJs_Kv^(rwSdG8e@PkC@ve8!xC>h9Z6V zpjKR#vs8|u^Pj^1&nWm$G5pV{V`|ox`y;lfJq#qPBTpeGQfzk91&PLj&h@&i%ccFb zqHICOX;e*%`NR3yU_8$8<-6p{Wj`kzE*QN;r?I;1xttRcZta2ViLA%4%8sX#ua-$I5|X(>=S@Ale_u}3eY#t|Z&sdI?UV9n zD>z@CrB74;Kb_JzQ%qlG50(n_2L1oZpug47|F?HGzv%y;VoNr@R4Y$yo!mqxe27bMn4u_ViMgp>C( z)vaTBsrpc35K2}dg%Z=0#Nvp2INC9zairNko?+M%U8b$c3KK8h9ILWQbW+0lMxdlN zYfHvPjB&xndEq;I#5|(f)d;g?E8q>Z3lLkeVT%uE@39|CeA+S#3#^22wVBvI9;HTn zTX{|$I7>X}R_fwH32W``tpRieBGGBIru)4Ol`ZD~0|q_c1azqLJ!neqt+p6nWgSa@eD+p$1jOGA(LLek{>zZQ1l&@p&G* zLxtV-hLqSwUcrU|rgg^48pbf7KyGw04^@%J^)~04Daisg&4^yvR&+v^d6P+})xy%) zz~-S{-0_Wc^a5jpfse<6N}-dp1deqvFxEwuI?B1t#R)u*Xg*wM|FLT7B0ezRPenO% z?kjge=1s$z!|j}UzXXnD%-GpumE2W#$l?mjme6q1amG?iwG+u-L(!qdPxE!CZoENf&s(`Oy?N}pqp znXiQWEp0ntX_=ozuu(}|(55s_0x{KYDQys$IavUb&H^3zqgfrOW5Z!@hRdWtTLEX~ z1eil2V|VsiU~ZX*V0Bnx2aLsyJBT!GH%qn+Z51#~X3e{xL|1)4lo(hZ3MFkrTLlag zLvSaYn7R*$6bsEmVWo9+bAbHWD(si%9#7N$Q(FGal6{CfVQ#Sh?Cd^i-v70`HTZJ> z*C%-z^1eMWFQw`8+6hfV!w3wkt9fW>|BNz4hv)ie#HFhi_S?#X(|$?z7Cj<|M6q^_8ZuR4IRo9apV0#!zqW( z&xA?qmscv7K@fCo+*3?x2Ah3TH>D1sX$bxwHzNvD@EV9rFw^QV;VI&c@aTGbSr*Um zQaC*ojZ+sL3b?vEO{ab`mv-TiKCXAPT!Z=gNt)6FbkV?Z=6E!xnrJPf`p}Fiu+4@< z!e9OZ`-k7dw@3rJLTb*{Z5&67t4Q!pa2u2OzGHNg&T!C$P6b(K_-pA9q*eI}g5ftx z=a%!s*{J-DTFy@0I&%Y_t7?xvony~z4heP;*l1_inZrY;TaBrsrW8*&yB!NxXZWkr z*}L{LR#ojC{);o2V##$;jGR&fHGpn)_*DJ+&8^2wJ=dS7QL~1H+Nr?Vzm|RmkX+E5jc4^E;@dAcgXLx3TP5_RTUIRq z5PW~k^N^MvJaeO|J91T@YlPkI-c=P|2)aA?wuV<9A4dK?)Nrn^tJGc%lD19;x>_wP zsSfX*_C50wfo+4mIZ7SIWFu$azlW6TL)93-M+DN!BQ1y` zVeIgBhclBdJR)JN_;YJf>j8m_o`i8w2%76(RXkRGY}kPBS2t|=S+2@TMg`27VZU*Q z?VuNt9`j2-%UK$-jKuJxpgC?eB^-J?FheQf!NJTy2!0i9is-BU`tZS-r6tRjc zk4R1uv2{9A|ZaW{(l#=pvUYF zx?!{B`Yifp{a?}Jjc@<*?K;A^;UthTJ)KrPojw|zKH5Bev~5aN^r|1->M4tc(0R1g zc__e7xFm4}{VjprS`iG?^eb4(-h*4CXVo67&W29j~XzKavoA4cm>))6w@P`hcuPyoi1G# zyBStf<0fEjvCOtSgeG1Wfc#cF(wbbTCEVth;r8YUUJ+u`-aMSG^N%y^ zOg4&bby%(sQ~DZR-ro{=k9pS-K&yd;*T>Q5(2AZ%MLr>t=d(JrQKQ&xNp#CXdmAKG zh{0ZNgC41P*2-(s^!2;qalTMY?uG$1+bmQ1E*J;?fBj#VkLwMqYNx6iwz`dIx^>$( zwmaV_w6yy5#<%U5ig|m_E(v$5oE{b`oU#@)#(d)7@FwB~H(kgO9&LMH9KiqkJx%?u zO1NbFQN!<{>TOr+A$DA*Bw9Z=94JO zsSw{Rh1x*U6r-MtU9e#nUq@08gsR)_|DR(G3dAW4$oC|l0#R$*D^-0 z(7W46+5gi9ZQcY)nf$l}qspkW0t{vKI|JZm%B_H?<}dpy5$)V(2FLjtF{0Q-5K$zhAN12X7ow5cbSv57?|jrERCHE#0`lL#uZ=?U{d@T#+0Ck+Q`h0Q zW%l5}EwYNpaN};;?x%Xgj+#?U$6<4_9j4#w^uXq`zJO4;S!pzq4F>ghXRmRt`vT{= zrFWKF)tu_S<`Er2|H=0DLRp}K?hYNbh!HDz8dbyA*?ZvE4*G`SO`<(bTfL@_nXy{C z7=F!*w7zNIjA|qj_a?(;$DMng2R8YC#;aSJM|dUl{r71wpY%6(zVQD~ z@iYaM`76Pjs-Mm~+q)Qys3ew4AI;ah!-3yToVgJPv5+(0UgK*IiD)Q%Edp5!y6UMb zK6gu4bN!nWNE5>%{4!m~J-EP4=l?r{&8GdQKX~$G{eOz5vHshVH5o6`&nIP%1Jhjc zoVL32rK%R9R2=1W%&zL;rp<8S&p1KQqnwJ2r-F@Qdc>ppIQ;*NI!0VK<0|KIquGv4 zDAa}zY{x-XqiXEjlYVgYilSRpOw$EJsuxE#>HB`W1Br>)tL-K=%NpD!QIaEAFD#yl zpb*sF@o#IX*CfuSWG?tom(yZp6Nw)_1nTXw1W1*T+3FE?6P){3bvW&Je-Xeo#qPrc z-1KVsxQH%k@xK9%9|YC~-mR@|IV+?tx!hd5?x{4_aVg2QB>MXB!MEQ1rQmQ&nNIW9 zxyT>9-OF@aXLpWq+KPX2mboA{2aArf*s5biwTf6dwI+zSMkFMSV4eQHtliElhqrY$ zv?+f%UvjMYr9r-BXRG^()|OdSZEl-fM)P+W&dNiPB%e@uv;t&KsK7@>ZWE$gp}1MO zbsxVG<2MY?=8){vO~QBXHOsl=A&-Y}^8CoV*SxFZ^SgX0;}|s8OBjVOqF1ZQ0mwMO zYYe2W%&n-qEo39k-3bkzjam6C=C+W_{Le~scZ2@-Acp`q+5dN+H1ogjZuY;}|3AsI zvdsNyEjZ=QZ1-hnJ2vBGwa0WxA$pZa>lL)Wx;Uo-uBv9tg{)ncxcZ7?A%Pa zkm=pQT!pw)D%EY=4pv0Q15?bydww+0xB@H-Z);k0H%gh1+RuL2LU3 zi5KBjk%lc(Sw$A#zM%b)Pgp{EQSMWd8|UMUOff5f%&XYXHMp8Gv>@}FZL ze<%m=gYw)}fk|))5<)VEsrfz*4 zZdcAA^xAIpZ!SHjdF9%tWi01)JxZWiqqAg6%xd2YF~@78i@9$#GH9>5EUZl50r5O0 zLezVj`_zn{oUgfTP>|9fx~vf65i85};@T-AIF3kF?koK) z4}nE>e0czDS)>`+a^tPk^vb=L`q~LkkvYkrL$8>Geg{tW-@g9e@Gs~4<&?z~hW{HP zUVHzJT#(ue|Ce8ekAF;K9-i-CWi)4MM@^0^@%9+F~fZD@QR_s&0}o+XkvAa zvF(RdV~SO?3nPkcW91S?f+o3uO+`jT@IQP!xK<~ZF)ZF0DNlr8He~05D zLrX~X(!nZJ49}INY9QAMNHK6mVoY{)OhcYVVhCG(m&s^TM)`54(O2D3Z(*Abc5C1S zEv4bQ5_jj+w?yNpOQQ$?}C)1pY zDUTbVG`}tZW_d%c16>N(SO>H&Xs`Ko31G_`S_oLE{vtu5fQ(0egfx!+a{L~0%2`}cQ@L2?mNde7T1faU9C!K+(H%gd#gWg^SFM&XF27sLDDPvhh7uYWl`I(&QLAnHQ3QI|^`jCPid$hLZP z_~P{7sM+TCIZx^g5N|epOUF$~1>b8aPDmmPGypgay8XP_d%gc4Jy{DD3VsW_IzHNa z{t^1aGnTndU+#VXa_{tS`+G;Hhes#J^ONMv5ppG^2U*>zWnOf9hu?pHynk|hdUAMl z@ci`c{?V(0=X)myhi^{5-+O*?__np1hH%_=p<__@m#?VAh=uD@B3{zj4MyX(dpI{1 z59XhThcEZ{kM`f5K0kc(X8(B$CA-kiU1hWY%t=hz}=b_46tQT4$yjh7AG z$!A6-2(6j}Q_=z(8}@=P62+|@MQM~z?rEX-D2;gK|h%Q`GyllMeUUIakv+)dw zH7>yAl!a5LWxR?*Kp|9dy!;}R2(aq%;00bDRtPeoKm@)=0|I=9rWH7N5jbrfQ#u5+ zu?>5@36s-egsw7`_pket2~kkd+Zt>Ro|cW=!@OCff=a_76nB9Cs$1Q(FNi7d-P^;L z`)^MVUVQWDy_FHx*1q;XG;39{fUCsb)hWsJ`08-)#p!o@ulC+NS6v=`^JpC}TKmdj zR*R$Uze3+-@FV!@*Wc71VIlD^_zL_nbRMm*KWdB#9>ZW`qqEUAmPM-i;sLhAInJST z@5N91Z%+=6_g|d8IygSrf3yGg_?x3wdk1e$_J27U_Ieu22S+-;qW0H1XK8dGpd;`; z^Um^_Ud z#05=dFXG`j&3k!~u4v{}B67s?b{ns2vu>4(7O>nqZPP1l(N4MMzC73c>ihUYx_D%x z?-%Z}b!@ebq1U?q3N1!1*qw7qqQf+vIX8#gKn(5czjiX&eTo# z6ErOFh@7XAUP<>5zxJWcX49OoA5Y*F3^Y zi=gwClIZ80NqU%uv;#J^qhq$SXfqd&)UrGZ$#Xm=T(Cz8_Q!n6oYsQ6A@tdptFTUg zM=kEAijEP&eQjJsYN=;xi?5~dqVd$hT$*{O?Si{M+a>bf>S_A_N=3c+|5MBbe5$Wm zJq>v(iiDnKIgg5v-pV)lhWM||el!2`=GJC^^NauQr+8Z9e~);yAm{R2z9&T~`VLHU zA+K*J#W$4FX$$4;Olh11V%nQ>e%=`Zwqg$YN`KtCE0wdqwF6=L#rBO~#_Dbv9yBcl zrIo-3KQuqAZD0+a>&u7b)Lgl0J}kS~cd_N!xkpst0r!Xsy>I0GCsm!l+&_!gIw) z0?!;snBxTjVoKu}CNUrBa2b|Px{%YDNOnOXQ~GK3o}`ht1}UBB!-(|^Gh)&RE}5Ld z|Jn%PFpX!xQ(E4Ir*_Vmr8Mw@7ssc^lIPS@Tsuj43O_wRhKS|D3nonV@c$ZoFBtui z_wfJr#dOkB|FeIJi?mlYG$P@7k-?b7RCr$p;xhBT4o2kM`#O+G=6(HtyfyfVr z4qogFFUWHKj)u|;SVT!rm&^IP|Bmcaum8iJ-|W9V{#bq7wEp|M&HO)u!SIuA{g0pIS%bZdgykXp-)Nk`^C`)t=UL|ik^)Pz7=ui$BY`RI1n$C^7ik1j zn$w}@!Pjl4_W$59C8={_Zz3|hHKk5ccuRjTm|$qN^}dFeWjPoGTUt)Q6 z9O`gJC{}8(GR)XEiK25Zr!-&CaODJ}XsmKuDH|B^3nZZZtBec#)#jP3GK#ruV)riFyVX>Q<5>!@WJsB5aE|KZJQQY zA2Tb11*z(yrx}0(!#fz8nI#a>h=oMbsLEEdp3{&giCSFIh8j-_2|1@Aid@^_D(6{a zN%XEzSO~+(e z+GHs(Dd5cEK*)g3U^KI25x_}FQ6WiTL?Og@Z1x9@%*4nk^~#kCr8}rvQ5nfeLM2Lz z-ZdQAKVhZk8eUL+pP6E;pY@|a)qzv@X~etsZ~5H1!k_m%2&mWIHJqV^{%rNB(MpqQ? zD5`>KZ)k8HMOhKY=Gwda!1D0VHJlxc-*9=9Q$bUCH%uXlY>3$Hs7Sl)iaT}tiJogX z+qxG8fkjcslZ>aUqS>P8{HZ7t3&A=4%m@gdYdE_-@0n{vQ;~6qxfP0Rtg3NO9S@EyInml*Pp&%>dp8Nar?QHvsdatwd}=`~C# z$$*>Sx6kOghBN<{|SHWGYG;$5c#RyMh(DDIitsc^6a5<&9RY_t# z5n5v(QNU1_D7X{6w;UagZN+V9VI#QE+>}i>ws#HaOTx#l_Fpyu>)J9Pr=+GzZ`=#a zJ3iNNma_PsE6wntX1b|mt+u16)N4innFLhF{Okqb

    Aw_*qVAg6yrgW?!PQ^G`vMO~SRR z==g$;SxSXX?4>4N?e`>=%y-_6c@7zm1Q4N?8%t>fEFInfcSHy_NqxNEi$}D#W+C=T z8u@O&Gsto}rnzE_;sJLwCllHV?OnqwCZwVw1+JTAtu#F<#N369ln^0t*j1~&`-~3G ze*Nu1P}FwU#^Q!66uDP(SnSxg${Ryg!jv_|`YDx{JU?gY#J=mT*(UwcVEZScsJgym zNxRo@g2gbiiDJPc3WadY*YjAi2R} z9|=WUgZ{Jr1ENUZ4t}G(=!L#SoTUaw3KKSv+pzAr0gF$e+0TP|W{xL=n1_ov$4!mY zP$XyBDw@8zT?I~N*+&&??)_ZDnWfab^P>h8nQO6cP-9=1V#tzdDs!umQ4&tgd3OYR zkkgRopTsb<5Yz|X(NVdN?(9K}L}UZ6k`%gNB80a8w2c%`ytF zIElcv5WoxcUl72xbuG*J1&e5oNknXJ{<%T@`r8>=XlmK*ZyM*dD|~3dUvD9jTaO(c zeI7>Rhnu%DXa0)EGA6V&eYNtrF$B~?Cd0;DKsWz(aDw?P36EB2qH>db_kvC z)o##%n}%ynbldXTEvk79XL?%C;MZDB5IT4KZ@+4#Sbr410WZ^v-ExTceGo;R(!nG! z?x}&vieb>z83OK|OOa&{EpsYYJ(anJBO(NlbOttoh2j0@B?}2)Ur7=s?M`{lM1&I2 z(H9e_Lo64DSmylIOamcFfy4qC79MD6mPJgqG8XD!BsMOsT0xB%%l6ST4|ASPlw^!z z4iQ5*`%9G&oqxkMf6*|NUQ>dXsDNn7@ zrE$!nFsr?Itck|VGpxt#oI1(*yFg_aY=E5RyqHXZ;6)5%qENt6At{N_rV4^ZG$hNM z_a2U?+?>Vwtk&-vt>W>qntaR=GNNOi<2XlDDTTTT`Dsop7eCiCqXk0niEouw8{Xe!8Pt~iesvqpci0(1>$gO7rudh+qP9@?mgj3v#j)@;#(`OXFK z;+_V9yFb@(w(~F!OgH3JKWp~;nx!l$5(s%JXjouOr>R7pqbQ9gWWB1S65MHI3o0h1 z5j!OpES+G!_J>}Wii$o8ikb=I=TNkVL#xoBDhQBMCLmG!A5u>j?NfG-OGtiBBluC@ zQ~#-`>fzkrmZocJS!h2KCZWi*2w?GI*PJ4v(sf625Ys|J#4l5rsJ6#N9M8J7mC2;y z@aHUUQ3~%lIyl>X7+zGrTkmF-)QvLHbwSvR`6P!zd@g1xlq;12KZMR<`hvz(t{`hS z8!A^qfk2O(NE9KKCdcchO@g5=HomMcf``n zqIT0+<}v#Ny>k#NI$7MkfcR*S+R=Vs58}Yo?&`O&UrfHLwWZG+QP*(R8tQv5zU;SB+}D?dV`#Pzq;yt?K~YThD;; zT5i8yb43)D0XZfHM=@+gWY#KJMx#8$sBbsAhdcCX&5pg;_W$#gVh8}wam7@mqR7H7 zG|8RCb5AJm`aIl;c-!5<%TB~=3nXj=n0`*rHJk|^3&GP~^J-oHU2^49Gl;*6DcRiF zU9f5vp=fa|?hPpVoMLdV;f!o<^>;V7p6!Ow4jB(f7>%Fo?hH0}w>HDgJ|(*&`egU% z)17D>ZuZ0I*?7F&ACES-o^J2%>^z}Q&)QLxN}69RboaT215*Ixlt_0u=}k)1795^` z=-dQO$wPBp|3^pBpnuOurB)Q_fu3j!cAMyEKK=V*=mrU?vCfVv*{FFy#!?q6=1dlI z2q{!5qCqiAU??!RT|?Hr1h0lYUu$j0LSTZY~`DB^~RE)gEDF^~(`xXKCF#)i9qo|yLX zvp?neD3W7|BIDTqnx{LpP*xlun7b| za_DWM2f)%5Q;4-RvLcFX9;7pHvgj=yH>^MBIXyfEH}hi`Muh}1Hn3Ug@3d$SJ1V3*eYia7^?ogNp2WeLu)txneanLH177F(fcm5gN94njWjTx1U?7r9h)u|m+K<1J6((qCTKSS@t7 zuh3Zi`s9y7(LZ0F{%wEn==AvDpZi1T41Va`#YfGFG0`3$H!QSj&_~vzL|W$=yyNk6 zd9LAX^I^P;Dioo;Cs>|j(?!u|o?Y|dGHcduds|_l{rL8k+Rz+KnIz8ZMVeBomKqyd zUakay1~>m?ImsT*>7hc=Dy|PEYfCGRKC?$Hw87JR`ldHfv`V_Ie?Lv%^Nx*tS0y4h z3hJ5fD<$Ea&b~ph?bmMk^S3h~`J})QFv;P}P1}Cvd3!O&J4<#!;~5b6$KLB###3ub zq>U}30(D^Hp;ej*Dlkko^>U}gpe#9MLMNvm5iz|nW&F4%a<=A+Ut7Zq4O1FQqY7!G zA?5{Ps4zHb={)Zryg<-GU8kxrnw1(V0fAS4m<$^_w)LqfFFh!SVlL-^dSGN@qG@qtSnuh7?_irsTq03|fMZ=)(&lEM~h{RG}ZQx>R$nAvTJPy}O{i zW57>M^dth|OXfE3`w;X-%(^q^HUcqquORi3nQwgEtCvH?p1{ZZHqSioXG{QkR+T&h zCTf0nGhXEOE(Ps3kC%SzUrCw@eFumpqA6ZXkfVh<Cn1hQ54VahsYaTws=1l^sd1p~@7byWvWC52t2qZhA7&1)Dj^sQdvOq;SqN zVN%~_IlCZ|cGW86F`vv7IT|gytNGlR5?QFC#VBTCO7-M z97}0+dqzqUmQFmJ9jb$wn<%h-A>mio=n#Qx-y#m-=PB&{d~7-z$0SW@3{@h-z&jXg zKSw5}&ZIPZENP-LQDXaDO_XKs=}FFu%;X1FOg^4rvR9DY!;}&43SmRA)p#N0sqdsj zRT6@Lbs9_p7!je!2BT9g6w`&NinOC^PST0uJc{wQ+y@;vcRV-pC&7mg2Odc+hy)j_@9)twl$!27YJSL?mMG*y=wyb%GMy999k zD)K$sI_uWy-)o1p!ZpumYSMKzhC75aOs;j-)jtOI$L3i8dv>Fsf`YxJRB;GJn8+EB z@R8?{bOuu{Q1qz)aJ{8RG$e&^xuRO)>L0{niD7I`QOgWz+ml>v`Mqk4ntl~lO^zF( zNCAmiF9S1@jllC1D_D>{j!4~clK^biJWLHI3e2pa-;{ac81BcIDNgcBl1H8bgP>Jf z1F5?(vuNhjUf+uM2dOPqZ5|iYNlc|#7p_Q!_zI7DDWSdIGfWy2i-xkmJwl^I@nqcH zo%Av-Bnf*up~Qq;(6oxD)olPb7sq-ybi^u=KrsBaTZPYSBXkSDbMM zf{bZhn#BziWgWeV+07}5>qKfXkBN>fI&e}S<7Kv*!b%RJ13n|$9Ip7Xv7zS!l*vu- zFw1-Fypef<_qDI--i)R}8G3e=)X8&9xMcG!sR}xXvxE+tsNK^4*S~^4c+&3pgmZbok=*;7H%? z-tBaOrg*`Z+Ph3>EV3nJkfy_f@dD&Cfg@D(fQVdM0Mihzef=F^|Cf{fw{P}dDdiW> zi)%^hhL6_Qk@vu3c=Z0|-uEx}PD@_?VPiv0&XPVJ!~J`>dCcxyBXhOe8w&GzLG^qs z0~Fek;XhhmZv(0^1k4V3UqF%A?*ggw^U?cPhkGwhzuSAY_vX0*_UOY9k!yhaG1Ui) zxC3bW2a=a<0$!D3hq=ekkdXx^#n!dazNB%u}C^t?w} zX$*-3e*|shrfJQ8<-IlvhmOgQV|%2wj(eMtYWl;)jr95(Ll3rWk{UI2rq-rnxQps- zR^VZ zKLDkZGXX47m@<)LDVN4BxU|WLQ=MR4n~5-RQON=ZZBFjrYAUqQVl9h?u(~kimr9CJ z!dh9>4pgaKp)qtUl3LX0bjoE}WJ_e28o||c^>RXtB=R{4IsTlG^Vq&=3PTOiM{g-M zZztaRTepKN)kH`Vp8Hnu_tj$a?P423N5R}tEk&g;we_?+=)+_4?_dAwj30X}uRmR3 z7HWBmqg0@fTWz0;#Pfo}h{pUf(9P@XMA7<%HVd1)wS9#u?L~9j0RB-~S&FX@pZ9*)_jJ##k2$}!O%3|N z*0W%9dk}06o&dmKQl8Rpo$_k(dMD41S{vI;ww?*yo8R|K0-+@iNZ@T_lp+ujbgw}Zj%COEzAJZ%}G+uP!xju ztfHINu~KdjbGm6A<;`-HZrTRNO}c420#G-doYNC3WTiK5YK;IjMDy_$Z9ZSQP3v6n z@pT9-=6z8j?yN*KQF(j$7#7LLZGv!pg?DQ_r(0ttUqt55R%HHnw~>Akj_)TNBdvTg z;n?=}X@ujlw=cr+ClHS7vP+%NZhubER{sVuur~iC6LbZ@8a*~k^qn?p-7b;(+DCb3 zfv>%}Tg!UzGUb;kV0fAqXgu$0I85Uin5!fNn3#z2o^>>RqfvWA!gFe~J7;sD5MfMM zB4FM4FeO|_JiUj#pN}`ZuR-r}ieZ-4Y%q8<@;XysuVz|Ryed43KA9lEsl)rdloR=xy8T0qY5wFCEA;CYR^ zKpdB99jKj=a7qK7PkIp#(F0lgQLokN;bb@wjmMHmmcL)mmjT8@R#`j{HOVGKa^M->47$-N9QxAn|O<{N)UKuX2AYK_VbkPDq zwRrPu8GC9jC~x-;4S7zv=s5|*Z18K%O>|1*WCKx~a7B=*&dZ?i&2ulOBr5OZa62Ex zd;%F45?CtnSa8V*D)=-|2%l=9C#P<_Y6)R5l}WszsmV>4q>m31@(wQ+$J@+NI1!3O zg|rzZOMiBoDG>t<9Oo(Z9!KK<>-N)%&(Lj=Y4- z@Q81Ol`SO(aKd3q<4h4`POHgOR97{jPfl2l4%QbWmFNS_Q<_QvmsGRZk|9D^=v=X7 zMoc(sPmXDi4;2P2Rs#=LcU4zGj#ubJ`Y@L(FcN;@%%+KRXu3w7p!X(E5{^zf3T7hN zh~fG!Rd|OIOumw?>)L8=CsE20NkzFv&|5qwmu_OS0KP|K16m}yb_+Jw_b8UFrGn}{ zAC`tZ&p2XBaR+@srCZbP?amE2BO3BV2SCJRL}Qn=D3mi^;7rAoQ>J4v_;HuGb}norzp1?8tNp}TKs+@S zp7T@`3C-aL^i~eM|M>FEJ(?)YtypSmL=f}v{6YKZ_xt@PJ3IJqzu#~E+uz!KQvSyG zgRR{`|3BbCfmrkuN?iPpe*eDhGza(JDc6K+To$+Yr z+0*g%&a-DvpAI(1WU#wSpKXuB!H8^clg;syCr`Gc@w3f-CTGqeX2_CeMo7Vs4?&kdZ-~O`xKgDzN`fse- zhhzvLPxTIN-Ad5%|&i3Z!m-YWC zo<=w}$+Gg#tzakE@gge1oMn2IKBuY7NzDGBQO(0QAq+iwmC95P0e4ItT@&-z^}xvR zs(8eN%-N_=O01Zr;WXzd`@>XFnp{;BqYiDqBJieM@hZ!LoD>O_am`~PPj4uLhmj8P zpNfRUaWG+H&yEX?#Xiu4uy_bD38yhh1A0MI!PCDcEX=v!V`*FlH4jy#7=5}`)#H~m z73Y7|e6UvboL_FG!fP^LleoL19!44z^9z~8d)JFe$YioFPGv$J&0lelp}y|8=^03#^CFc)*aBFC7gVquu|iU5Y~lap{G~p8)tO** z`U(v^i7&|vPm0zD-L0;HeZ%uI9=%{9FK`?Eu81ZSJz)}Z^%QP|p)G4s*_8-9Z|Q}tLl`{m_i?P%j$K`eTCGgl0PReN`4Eq^fus{1hkzk%SElM(3)>KyJUK@g&y9{GX zumQ}p7UoRSoRP&%c}BvjCeS(UUqd^rpu(xkQy|t|tn_AsgChl%>l6p?KVZm6K4LN_ z7%QLpETXAoS zZ_0&iNMXU8BOI(PbL=CRqVCD(E9p@9fAEyLA6Po^OZUXe*YCECTYNih;Q6ffw9^TK zY-S^jghpQs)r}1q%&PPfh|V%+Q7!{hQIdQ@rFIlFpPdHuSKZEn2CS3T5YXkL(YgIP zUt6u}zgf4Fk`C6^JnM@xoTo9xZVdw5;n(sUR9HA_jJ{<_%nxX8I7jXXR%ikiw93M2Qm(~J%9vu? zB`95;0)z;>!e23v^87O5)_Dg?=AU6Xq% z;U@c8Smf~xMoK;q=ECi-s_FYGFpVl%XNbQc%!(ll`u#-zsi$oSgUzR}nHiwY$-7E< zFUuMFFW^1+s{_8qfRmIn=biDl;l=!7z_ z)OS(;05+qXYXI?ZDEF5j9NbiC>KX6p3uc_HvsoY`ayo=g&m2~_7D2Yyc{bP{bP!-O zIs@xVI@8)TWQ66$Pfn>_?k2%U&t(3od5H=SGI3_4|{juP3lH?)!& z=@ssHV}a-1xJ1xwU!;5DM?rJ*Ms2M>3MKk>fsJdopeliiq(DsmOR5FXz;hBM`hW=s zF?l8CNxY(=ey#0)zP1{QUS6b5SJk}{O0@|KA$bC%kW-$sKa}vfwgyK;=wv^FhB=j` z}*R$GoRLSy%Lfs|LBYd$2Jwa~r#&vt|I#+M2UiaHTuRwtvzdruKq%Wc{8E z+c5cUyWrmOt~Z9ixUaq~YnW^i9lz67snh8iZXGe9lc7hmlq8B!a(=-Coz2Yo(J$Q zxzf^VBLob}U?b0NGL;H1L`c*Ih@Mc6gQRV`RDLEy=m%RSTkK*@+ltF$HnH8s{F3Hs zr5+YCE2O5J3N)ODMK-Mku8=kBg;9e~jlZySz=6qj&6tbWgh`dlQ7X!vr987VFAE>O_YykoOfA+q;J#CzM^nZVf`4M_H z$<_vwYw6PG@B(e=637ACz4X^75m%j4B)pW-h3f zklyH=eaJBXlR6Yna8(V-Vi`XPErR@52F&=285YObM3Y6axfP)ueEw(V_VyTgmWu<) zlsc32SF9GPVmRYV?9rND>B8|1C-9x9Zm9c8$;#K)*BY_63}-s~Bf475*uIa0o;^q|62Da@gk4xB z8T5VquayiE72nHrdJOsS9d_%{rNmaaf6j5HoNnmqFubvln=Dt6^4clcgtEm84JU?@ zt%jk)_)Ly``Nf)?U%bLb-h|)UqRVhwN|=Z}wu2aNY}?<-!MY{qhKS^mj9z3OOkP5v zi0mg z$_IOCGHr|oNcIRZHKoT-QAbL{!_M(Wmhc1EU@ z&+^FtjAx$X2Z{Y5SV;6Bt9e1-h4999zd ze=6nj^ZlQvdG78?yKrUBtMm|ZVp=%3b#ZM@*fO5sD(sf-?-c^2v~FAx5MQJN4C5JS z3Wrlgj5tZYFxjF;@9BzKV|UEx`KFLJ1X2X{gRc;H3wsQQd-P9pK3Gg9<_#3o2pP5w z`ZuGn!E(Q&unp>&41X+;#o}2Y*QK;;$}gBGd}swJeKx}_IP4q%U~z= z)N_fgiKwsFHV<;)fCwgeRLo=^&&lZM+Z}f?n7cBHd}q zek?%Ytnh$2HB3zR$opcJ^2=S)2LYVfjp*gJX^*Xik!POlJZcB{!V)f=(F*=kxJ=T+ zcaKW>)5-sp`k%XVUCeL71D8#&g{@xyCGP)MDwV_M`~Oe!Bos#IRij@G$(ytCUdn?=i67k~*RAHQPolL6+o>UgH5Zr*%Md#BD$O8>0?e>y1~eS8 z3?c3(+}|srUu-65w{uHG6{guEd1&Rzkyh5qnl3M&uzjWREnd-DrbAFH2dMl3c#|2G z!tOEOo>dqq7?C^9Z)k48S1@gZtPBqY9^bd*J+%yXh*eFbs(>Wao>ZniKkGm1%75n0 z`>+ea%jLhWmk;(5@?Y002hZ~VDIQf-A0*pX?<4(l1iUyApg|U^e9I~n`vA^m6}}38 z&Htfie$?xo?)Hyo4tNXa625{xj?@4Y@ARr#(U0|eCG>5$eGEJcJB5XM;R{az9Dpj0 zDEhtLuvXlr$oWC06wi5gM^yU#=d+K4R=-xf<9}6E>3YF^1l8%iJL|MQwK@zm{8?2C z?e5#24BD&@>u1e2M=E|*)k2AIHU&vg#i4nLs8O<%NU@h+hPnh^!bSJNF}M zm;*-zg0~^?pw<7>>Px^M~$-*vQF^4jE+nOv6dL zdp2kd2Bh1hH@%OXn2Z-y6^h&2a^H6=J39#qvXP}kbt(F>HdZ0dX9^|z{I|&_P#eDGhN)q+Tq2`-buaPt(}W

    pjz zRIk@~`>w{Dj}_8cCAG?5e)x3qwo`vMs1?*-=Ql-p6oLBd+>ezMglwswo_?tJg%m0h zkyP5fK%OA6(Wwsxr}g3cTA_GH-rkoib5z=`7VlWq`)X{4-&aG_<^(=}h9c=Z)cy_I z+U}0J{kg2Z`tsvW5x#tRO%oP|$F%_6I-Pm{st?|u^;;+PVdMR0{mZ=s(d##p0F@ZP z$!}-vX05or-Rn1h)OSQLaOBb-iu1x-WLj*G5Z?qI8VBHbhDqaJd(Irfk3xGOyD8%v zGXUZLH!($j=a+`*f$D-h(sJK%`6(-yBBdVQyn!MgwTk?@z)pC`iiUmvNKO`LrT9RO zRV5<2X?iu8w6!XAoN25cCwpbxz#Lx=%{itu!}-6mnrWK>^#!2yBC)!%qJL|Tql%H3 zU0HFCZW@%Y4g1FSY3-yuU5&%Exkm(d4rFN3;&mNrvM{A(>q<;$5jgI`n_|tkk=w@s zGHn_?g)t~8VK=}>|7~;sNQx~kd{?5{x2`pQS<%WxoYcPNO5yH7{IH{S=k2#BGtw>~yTHM|?RefhiD>4aOEha39V$xJt(kH{Q zv7pkMgF1m?QcJ|AabzrejuU8>Go4~K!$O1YB{X{7L9fFnObTB%y0K&Xj)k>AxVXlV zZiLHnrYrHhBkZDx2I-F+M`h&7kPYaWm=>hnEZ~lpEm4m~Re?0`l2#qfn zu4xM-r5V}u07}- zAC~od_)Kio>Wg3R$p80WK!q3TfeW8s1l1RfJN^FuzUa`x`3pOT63HF|)rLYba(69& zaG^`sV$?8|`W^9?3sV-0O#6TIzf+986J*-i<`%_OP@7|CuK5dFo74Otb8H9N#r(#@ zb0@&C{55rJsTIX`?FwMDAiW9b)i+wH{&DzzRVClIDC}U+0ks+R_hNp-zdFB3Ora=~ zVPggrQdT3~5x5bIyTN0)KvxLVu?vM_Xnhx;DC~PE5`LzX(cisQ_5Az=&&vJZ3uhz_ zIUgDha>e=I{(dt4Z>4;wA3g8?KE;zb|GRKTzBuoekN>Er2PuVWjcvWsk_%@<3o>da z4-H2s7|%-K_VNd3wEo6&c*xE-eUq71p)kYNT=Qq8nd4j*sx)Vk%AHwX(jNLCNC zp;Z?9vQUMB$}bkj-y7J%0ajh=o?WN{Eu>eyut9Gi{WEvD$+x0q7tW|ETp<+S#bX*$ zE!(I5?v8$QE%P;NB<^l6D#)b_F6M$Sz@$P36?32$P*Nd-5)5SDKps|*aYKBAZ?qQY z^B9|AE>*^y!H{Z+Lbi-WsQoHU zsvkvjPgz>YH3`eg&3m4-vZ58D!x>ELYq6+95tJP7J&huaMiin61GvXKl@yg5jD)4% z_V1oF|3S~X^8Z^>0j-e#6+M~%r$Us%v;2RG=W*o!KiAGn2zHf4)lcs8B=mt~eXW-0 zEQLzOrF!y$?{bRhaQVyW%$=tt(q3BSinVh5%7;Yhx_5vw=u0Z0c-`+sC8c_cZDU!# zQ%OfR?tO&eJLkc(ivI6@*46(!uHJ*rfLG}My`ysK{?Bv%=O=o&3`DlELjFf*NMW$S zc(};C+oDlKMs4hP*kRe9lePv&l5KJwIi&a zjNO@Bd|_zHxFI2|2W%UzW7;|9!A9!W&Cwe)nkp&I!GeNW1RX^!&99N@p{NoeNrX9F z8&j;FjE)6I-2e}%knsW6-r36-k(+Mpg#&4jZj?ZUZK^2*;rNq?S5;L}md@t1Rr6iX zY5Au3^W#3zF_e=O7}zbxi-rnWi@sK7ZYbB0w^Ib6U~G zAreaMJ9EPey9JQ8A4HNqx$f2$G*~{mF#BTU$G3W8E)!>@iC4I-q?}e#1u37`tX|cV z=T=!=lQ3=8zZ~q3)=Aknaz)m1X*Ml|%-G(HMV^n_6H!3=m`7u>Muy8|D*s}7@+3f_ z*>n7N`*^0DJfkT35I%pur)O$+o;!YKdxH|JY(#4*Pcmj!t?fRR=~4EeW7#C5vp4K1 zipU!+d9VwlSP$>dI_<%;FI@e!3sDL=@3D)6LvVedv85wo*t)Rn%F*`q~A&= zd#ue=3&1kDX6z!YVBLB?x6#PSal4iJO0&71kg?@w4^?7bCdii)%%)yHv&6v7?k44FmK{0R~S24z?+#qSs3!JyyQC!Y;YowH44`=Mshx9!W>CY z`ZzH$@O&|lROGAB#^Lkv<|@EJtN*FpXwj$~D>YySe>Iw-GI55VtA=70ZE_T+G~*I= z`s7OV(R4xb`#pkY$cS3JM7TI^5Kpljg?i1Cn-YZ7;f-BN;#BA6wuDs@Dnh@uoJv-C zsI*Gr$&*>go2EXNIx=lTMTX(kRuRdjmXzHw&q*f4&GSR5E(&I~RoFB1xe8IN#G2)d zku`JtK&CTx{cjaHmNQi?e1)x5w1h!_SxV9jR5UTE>ewnf8(9-!ES>4M>yw;AsN|S_ zO06Qr(RqB`B89Dan1)$nDU>NLzLVwAnj&0g(~6sQ_dKL&N)6hM5gv_2M98JpY#+>H z0iEbUss%Kekmb1~qD0BJVh$V&d&x~wsc=L5LfK_p#ZKAGV-d19E@n47tBD?-(Bs50 za23i22L~Cu#Y+}Xs}?sIs~UzSPCYA&;-n{Q$a)H@^`7c$SY>RLJXJoZLeGAWRLC=a~KfB~On3hc?63+??8uhrhjz75=~F z13hv7Q$IMYJjZ`}l4lEeL_nH!n0;}$o?#nC3)3=8dkQWZU!p1YHDwEiGt&ou;WCro zpJB^_spX8A&w^=BU$QLQ<`t%1XHq$`4P^^#JY_lRw_OiU%o}Vl^XY$fH0asZEjTu< zNk{@>)LN#EHAQO<&IWt1{>#W$w9%hliT*1JX44Y+ zNBs1!>{5h?1>2;#?|fxf^RHcHR~w;AWmgO4uCn`YWeYwb&vX_(w3{to(Ol2Dz~eyC zOar45gY%q=f10IyGzT)946JO`P{=2Dve!lg|ng9J(y?N5o=Eh^RvBLjz?_mEZIscCipY#7b z$+HEfJzk7Z;CRvJR}`w}i4Dk;R1zU!8rTP9gE@CC8rqgAa-LQ(O8NX*avJ@zJqrT2 zS}I+JsZ=%7DH+a~W|kUbH}FenekKEogv2i4@kDIVuGn#-%y(1MI~c4Oz0IDazIsoO+n7HBH|Ph_N73 zqHBjgD!Xth7GS|Ax`-Cm5;!*H)512eXWbIR%CsjQ@&j)%4i+9(cA*Z&t&Qg9nDN^; z9UFTzFg>Z6#O%`=6Ert16M1HE%PgtSbs8>M`XGDSfp!zIA~*HAM*t5>JG7@XlPEEf zm_wz{h6Ec8G*_+RP^u=5WjWWTJ*_IrE`%9)nQ2E?*byPv_VIjVVOaR2|L89pDw<5n z=Zp|W5jQeg3y^;)5}z$Rth$b&hP(TT@+dZz%7SB+cEERlU>ye*7#=ch#xD>1F3pub z!@xo|GxqTLBX|=h{k_Ps%H5Qg(j9V%yDIytje~2)yEN^oC@byoyzPW@v|+=U6B<*~ zIFlgZEgdq!NTh)$II{A<(_pOj7>j8jA>c{s>5XM&7kZSN$g*w$xi0yaenF_)fvMK{ zF70}J5RFhlRYJaRPHp=fWAm^~@a4 zc#$vF6JdF`u{PDomO7l(^1C%8;#e{srhAe#r-xfZGb0J-(I5;OmlGx}#2<4O53^9Zb@qUbH zcP~72eXkTj!bB0M7MXNWvInJGKWS9~B$wb$eK3S(>$KDRt<|i;FW7*(>w&I=epD^L zs_q{{rK}&*nuF7Nqg53r3CdttAATHE!`h0{Zw-1M`;FG1s?=(4fL}f-Wd)rh`1tOtBG zLwZRkxvs%^wv~7ercY;})N?LU<5}M|io|uIs_IYWulm=@n{z-OX7-+A_{qu=UJY7A z2ou#fLp~rNHkE}VAq?Kawwaw3DJq^ig7x8QTuP6zun$SwgXOMISP$I7Si_1)wMZL^ zOp0KJ0xIH0ILwmIOanh*V<0=z#E5k*7PTVPX1;P`>XZ3ZFgD1?c))#1!kfEG8w;yF zpw)~~m};Av>9UCvY6+ousG4?>hvt}IU}iT#f3B)N4&V~s)~FxA=a=E1U&siYF319p zF#BBAgy))~)U5z}Hr?X2ZeId|-|8nFx_in#A#2;@{h%oQa19gE;)_qsSC$6hffcGQ zK_{Cm@RpbyH;61eUuG%)grc6`s=|m+C!7R&c{N{gZ*4)7X(?bQX^NtPVCTfN4XPd& zKK9hmswBhD$3bA)Q~xC)*Yt#{^Qmb!cII@Q?A-zyISXq06Jp`BMAMgT=?7nCHZgXg zX@LwZ%m&I8kpnxi0-Xs z66P&zP*oHJWD<`?&JCz4{qVKX!oOv|d9Hw37ZWC^D)=){$M6H2Oo6w+uZb~AzNSeA zUKw<#&>|ZA`_D_YcxT=NPtHk{n$Hpt|3sHbXpY}d2<<2Fja3m4mcFbKb3lX{KXDAa z^IAl>e@*k&v44Mfttfw9suf_)By7(Bd!($KQKs`U_7yX0Qg$|JY7V9-$$Yy;W)O;+ zm16>Ddn~+?+4KQh$K!r0wFk=}aKhYw~eb$F^=0=10F(Z~jLAVg@iP z8w~6H;UHz|l>wk!64qy*t8jpFgm3S_PgKyCRyZij@OrF)Q*0Ya_{Rfp2wQ}1AZ$=E zs~h^OB>eJj@5k_0HtWYXfp5_5%7)#ptnRk%v)R#@;qfKYx*onVorQ1R5>1IcVo7s&ECQO@$u2|!7)BAm-h}{f{mw6U{ceS-(rIr?3K&qvZDM_A2!~% z`qe5Ih}dXd5o?3m)YZYQJ#H{#k5s;Oaka}2g=bZJVOh!uMM`@xMnvc7iwG0|00000|0w_~VMtOiV@ORlOnEsqVl!4SWK%V1T2nbTPgYhoO;>Dc zVQyr3R8em|NM&qo0POvHcN;g7Fb?nE{1o+2*^T9>NlLyYeAe?mvgLSm#uu$*$2%vJ zR|301660p01E3{yEPwakK>_GS-$;t0Fe+fEg^ zZ{#5<`-KY0crt|Z-4039{O9g&?@@2iiKq+(OBKFD9)U3zaAX+g z2pSVzlEe@ZMMgwYn304`=!_-`j2z)dwlTh^~p9|Fg4Zn=`5WR$t3Uu=qE)J z$wn+@Y7Qic^h>Eo5)l!}Amp<&W+Vw|uS1gwOXv_LT%~bm%4c*4Q>9Wl?DsD&E_wxf z_qdq!J1o>#xDq3#5;9}Gkk9%=N~&aEDiVH_eQ#Q@*~I%9V1ZzFckpR<_en2JCY_IT ze!)c~hn)b5g5Hl#kQ1X5fcYk98nX~-)d?VsGpVTP%m_;qVF?x38l4dq4SkI~OboKblL3ShDPe=vCX;9*t%Kib>7lmEBz zY;D2In2$&d2RM;|&B%m8N(7lvMTP8aZNX7YiKIJ_a795)i2^lc5-wOA13#xiut>kB zwaPodk}%F9mP{a`DNQ1pgpA4^Ez`zqk_j4t;*d#Ge_}vEPShPg&xFu=gPao*3e@9WKU02G-xn;?kjh>;}tDG-kb9e}MZI7w;9 z#&aMAagKP*!nyf;K*uDDmDGd%{eFBw=JI=BV~d|66JQd)kBOpEeZK=VmJ}paSp0DO zhN~k%B~8@#rn#I7!+!r`HliY-ipn12{fLLMpGhi$NyZ}D#|~tFD-|?Ivp8nSBrxx= zCCvdjs6+6(GCjRbjX4v~K{^x@AVQEi@Ntb3dX^gI7tAuLV1y=6hHZp`5=A3m3G%zv zy)q@*k}2aQn-Nn)ew6)u?gS~1g4{ajYfKDsK`PieQ8aL4-FMh-A-*}^>KBXn)}ezV zp+rXYpv`#(7d(q2h}lP~zeq~8)g}nAWX#13?HX;h#e8B24TlO;CK3tH_D}wb6Z6gf z>qCg>D4R^Qd|(O8Si&MY@@TQh0{#}r#Wd&BKH#&o=bMp#FT$<7`HcsF*RSrH4f8J((Cu%P$ftGzaS+sg^HN*(xyO zS)x+T5*+p8EZLXuB^AT5d?|RA>h9}}cK~RjMRYVoM(J2FsXw&IoMx$0aDf#%QVFWU zDZW@DYBnxxHYslG6fBDz1E1?@4RN!+z`1#mlVe;HJIe%*kr^cp*GOhUFA1OXMA1)L zLQDI~zOl@fb1V#S_Pv)B$rvjt5*kN$UQNTc5`;vWP4%KJ zu!}d3MzbtdERAW9_hVG>R?)FW(ujs59@R0s_P{v8;4At`VgC)(s?J$E$A_>x80>aR zw8S?B#NwQV6!ybVON;9RT)da2>42D~`yKnjc2$Df$aG{DO;J5ZaG(Vx(H!U}CY6@E zwgs?F_OwRB4`|1txd{=Ix>1Sqgk*|qEf!X*nhC{tf*RW_s5>2RHQxwch6L25PW7MK z>L5~Ha1q(J?x$wyn4HsN)1Vx}ZlQwZJPD_QCp?o7^4W|jnDK}XfhRk_$KxFkbjB4u zC21PZ^IsQ&DT*35<{^n2ER6%pm zFMfIPN`qYh_Z(pxLYI%nUDMsqg3pWq%4Q>@c+N_Qog7DU`JuVuB=VqpZrmuUa zIiRN{%s7kDF@|_(-f~jFhYM0I_8F=JuN9OAFq@xx@9b2|mjWnpkdtbY?JDk>{wU_7 z{*0i=I)1T#@cKn>7Hvs0Ap$j}f#6&Pjw7{2HD*GpKx9d$Xh+M0vg@b{aL&_T4_uVF zMPo;9(zSsKnbafa$qlWLbYb_(sHma)qCdSxx!|>Cz5$UyV9a6)ERl+mNHekS3A;$n zncxX}MbC*~T3~dx^zb?$e`YcDjn?O=$$=EU#2cWik#5N(Ac5;^tp&CVXZHU-S z@T!KP48Z{tLugw3YAB5TZH+0nNMR}(0TB~g@g#Y7HEun-6{srU@VG71MR zKD5m9=%@rqBnk0z%qO=7?gtWUs#b^9X=x*&JY=kzC>QXSBxW{5EL1yol7(bO~mL6iJ;N5Y&@nyc8Ab?G#I>o*8QTt?pyPXol~vd^eQwY38b0Qe+z#prJ;Z< ziKwBco`kZm*^!2Ntx=<5n*t1t|EVvPwgoD>5cOb{de43QYvjY7jfh5dI_^uS_(qum ztp~8+DXzPx6qlS0wU&_NelsQX%U)C=$T}RgMdFxW1Xdc$K=EKigP4;VDttfXQsu9)5WcAt zn9YkFx4G1I0_^^@*V}#kV{gzK^!FaZEP7-#3HJV)q-OVR(eySJ)VsR3w}s=;%&eo> z*(x`kLoi$dP}_vbKj^BJ)~s|lyOz@D0xf>@@><|x+7{3=Y(%vH1EK&m9gH|vtBks9 zqB`PsqR>Q%IXY4T15__6&AE;P_QbSf{K&KUseXHE8t=*J5H78Q8MgjUf5ei$oVph@ zoO0+MaFBdP)s!WZUbl&+Dtkt{#QX44_jg1Esx`-ww^f=7>u@f&JoYp6b=A{%7bGJBLTO&<)+MA-v%U#hnwOhVba&gS}39 z3jkMbJfcTjD7&_HhRrKZqGu!~NvKy*Nd#|PIUfYMqBlGzrmZZ4gepx~mQ38c{+4br z(C}!03pcofaA=`F68u~Kz0$LMrU_LGPyoDi^GVoT) z4Ia2K=2;Y^W=w@%zt9U3$Neoqr4lSufo*2Y1%mD%ciyLF3{DWzW0Fj$-8>0kchEEc z(>k_s8FXaa>)QVK#dz55)*qO?zpX7e96SHVPHCUeI0pXPfAtqz-g>ONLAR+&x4UEc zHTP>pb8W*A( z=cU%zh3nhI9Yqb~*OKRXOxVo2nLXkk{qj6_qC4Mt-r|Ukxu9;F1eN57!iZ-{bZses zJ3lK30n3%nAAxANX{EbQiJBU*Qic*Ybu=uv07Gxp#gry?F%~hs;>veDt2@wEB6q+v zb41j%al!4uA+u3#>bQL-3zFid8Bjcjg=xgkeYc2}2PXWB!qG2Cv)rlXq*4x<%ti*T zZWqC*?6-G=t`?y<>tFMT+FghMjwy*g2&U-U1gE`Z|Dr?q@g-}JxSBn)G$M+gC_xmR z%qy{QaH5FP2x+m5(FaNzkE8TI)!h!O*$Uc95nB^JL<=YDjtX$}f@fXurv{*e}eWI)nV;?G@*- z6zEt7MBTB zDrxrL*o;+V|I&zpkk8T#x7Dqgj61pcFB8yZ`Y;-8MrRmKvG?flYld&aG}Au^vrfmF z@HY6sVEqEEsC;8nE9*CYu^gE~*H$(-3$1T!%y>j&t>;TtLju zX`&I+>>?N&POy<5@ z0w68BDt7fd#b%UeDu4N?1285m&IElo6;w`n91Yr``p_p4%qIo`=CPz$JQg)0FFcs+gqZzV+3#v;ab z)BS1p$0w#hb(NmO@tP-0%XBPN10Ot3Nl93{&hhF*hnJu4mzZ0r*l zxZj!4&|@Rl4vTw1dO+A~{Dn_O;Ju+{WXd6tE5|*?;r)-HVf+`lL3~$CC4(Paf}9 z&woCA@?>y#{^vHH`}ghX9<7{GqR{BRn6lV}iHCUJlSvp+dxj_Ga~i?@`<;%zZ%h(h z-x%dIf^$ZUP5$|F5j)@YA}sXBF7)8b7vM$#j=46MbteaJPK-AcEpTmcv&3w-T0cL! zJE6607cg(y?|uH9J5qaZoR-xe zR_wu-FIRvG70A^AGi#;_Tr|PX-*K4OouE9JO>3EM8$_9n-!l+(wdS;RH8v9*y*+vN z^7zHc>Cyhl$%nVc2T#Adr1#qu}=4TMT)G4{kh17FWr2TY{ik^b~=oW;hs6UTVZc2 ze|7O~fG=M>8PshSLY5Ky$^D%rKLi1i;=7y{UuQxacEV#|C7$v_Di-ope$HS<=Ag(& z3IxXV0%k196qU{4FNa4qFe*RX+_TOafDc54#?5FL3McUpzAx$D^KeCGxma=dzc0;^ zlNZOo96oQDCM$Qmst^x~QlCF-l_Kz%;yso`^b@!_<@>05EYWRA7;NK6_VoYR(d>S4 zGi_x?UYCyx5;<00!QTBtv3TxVTp~B!_-j0ZF&7s^M5{4@JP-RTGQx7=4OgJ=f8EO1 z@9sV64SIv#?jPL-DRY1`ND2;cF4KYm(A4Y|ZdGfa+$(=qWCE93rBfeslQdWgdfoM{y&nXsBoe_dM*YAlRX=L-6Ix z_70@>0Ct|8g5i=?D#4RUJO@Fwcd1PG*pN0~Wq;W>|bogY{- z=D8O!;-T#M>4S=M*!?NL2o&#QOfZHv1-6$!_Y!y|OXKr(*8h$EpVKLgQz~RnrJG0v zv)KQ$_xR!CYW|l;j~_p|^Z(q&^Z9fCKAf}J5VvQ?ET&rSKb>i^gi|_%`~5FpI=WQn z#ita*C|t0b4eCM{QLkgm1vZqfB$14a4~ha95oki%P+Js>A1~2?7B3=8LU{ZD|75e1 zY&>S4pc@p8>B$oq>DIsBDbC2*K1{ZS-!l@kF{2TXG&LtSJ0GZN9m^}-hekxgh=wGS z6yxdPZ~10{^++)qM-qsjW-s3e6kG@08=1&w>%GF$gDDNS7gMItfDR!~PgFBM5HWdKvb5;cbIf6CxLWw+9r>GvvZ(*F0i z$UP=3uYrlyAJ*^kEF@m-d&@^*dpXxG@u0ShcmYt?pxeC;6Z!3}%QuEuj;YFoaVe5C z6+9ITnIX@oh^s#9%t?(iXR(kh`+f3uN}@s1**O)@`7G63)cwoT-QL4sumk_S+j|lW zy3l)0Qi6#S8I?WFfVP$1OQsxPy1lMVvd9zJKi?fuMRvQTHA1DQ|NoUcC{7FXKK_WC zt>>92rD$Rv%x&J|M%JzI+@iv_!_BE_E?=5L3`B_7?^3z_Z@ zUFe>!$a-yjqPc{MdU`L7=t2an^EaQP!b?b|=V{e!w%O0T^3ITMtC1Vpc^z$C*9umQ z{#+SA>%u;kjnTe5yRbpZe8yc@o@O(2iPy2wzcOHfgGUO@U>5MW3w(*~k z7xsF)1Bf`4$@j{222AaM+Hvp@95$o(amUq zO+c>)Vq|?M#AEWWxr^e*6JZh$h?2za;}>J*i8h;{F5eGo_}D8ask230@q#`3|0##a zp^Iu)2Nwq}@Vd_>;o?rpOtrjKe-8UST&wk!L~c`xiAA%gSnoGzB1_ZK^0}HW!XL~l z)hiLG(E(5@TJ2+5i+&wJl4w6pr{qmGgWbVe)GXyGS%tE`8owfyy{cCub5q4OIyb@B zNa@ysH%4o>{ko98#UAYyIoI<&U5Rt^dabxG@N+FB!zw(t5>5P}x*~nNsv77k^BY!L zN|`Ev>j^e4)MBB=eUoM(+SZ|q$81+Z*{`q^UFR-NOQo-Xstbh7)7mUT+ZDR0*~EYmy7^s8H@&5JyDC}73Ko-Mv~LgwC?ZXOR`3`nVwE6vSn&DI&2cW)A@w)K5C zG4Cek*PECxt1*!#;&8FQ$d5`~$;>Dx2ZTnnYGcw~zs=NWD7L7dMXk2gZQL;Tw&}O8 zAPjA`aN;6G%@r{qrpMhV;3cXhDa ztVw&_$st;hxgNLeV7S%t{R8iGV)v_>VQ3L;!VbfRZpac_V1;>v)Ftbe!B`*RgsxH@ zAh6eReg15MwmugzLB8VRqOuv|9o%fOy_-t3Euhn@W7Sx3DL&;k+5x_LTPvcroZ?kO z^^W>qi~5ZsymDC9oeNmvHujIKJ~uUt91P&nP4`AcQ}`Tv_5e&eIc3sZF<_!Uo2vN- ztS;gAj4Ln9b9qO8m8lDe0&brr!cg@a=0WAd_FAq00ApZ)%82UU_jdxnj(ta;j zRpwFSH#yRXDf*p zd4;Y@>T47E_Ste}ycQB}f#kz5^g<9#iI;5IN%CIjy~IQ-ExJLaF@oELdzeHUy65I+ zjs;QEp5i?UBUgv!IvQWYn%~u9r%~N_PH3%(SV6a<>|9`yuX!kDu z|27`)kTMr=ulF;HsqDRs`3Slumwm-W+1+j~Ros8j-EP?RF5eQ6f3MvXP&o|U?6^dM zQn|VOBTJ%T;rZ&!s3O{+9_H6xuX_5qlrufgQ)tp%rD=ORMH?pT-joQ{UfKqw=Edzb zlEd0bI?ou#f>SD((!#n_(%r}A6*kyAmAKN#qM%!?Y;WIs^xzU$5^1Y*ug%xq#s+3_ zQf_%@S7T;z`Q1l4-+}Lpp!PN$Q3WWoG^R zz0Y#GBY%(ef4#}SzIf`PwB=pV3eAB5H@d9zBMQD{ZLmlWEF38_LZ}@^EFofkkn0sI z##5rEJMi6TuBi0n_~*}S5kCA zS<{toj5F;RFCgxIEokOXrj;7yzuguNYz5Q$;@EcdElc7`E`CpfREX)0NcndOmOV2vrD1VJljqZ^HsO+TK`Ff_{e5wT zawfa&*23j$08BrXs{s}P*HqmTndZvK{N63YE(IUx$6QSC)9#N?>h2?EhPND)PTEL9OP0 zBlb)AW=oo(b^f(xHNiQ#uKxIz^@bDnDUVJ{f^0@qIDO@zbv49RhSes|bhmx%$1%SE zmf$^>{uM}8Py{i59 zWbg4^{O_$i-u!Rd188yg+N)vPwhD?nUwa`JluJyLUzG^9k!I6z$*J!~>!=0D#JiHh zc46L;#j|6FzB=ZZ{wQ2m?cbk87AGE_OA0f z>vyYoOhuIwyqE2r`!(0;Ur{k*2{!OjkdPix!FY5+L!LxFEQ^YjgyD(}E~N_%Gr`pS zxjB(uy=PoY)lz{%3XSDU!LzgkRJAo*<|^PcxdeS^{O`GT$uAIEwv^vqdT5d(lqXtI=@Jqny60RUHcUkphLQ*;9s#|&;bE$q9^DK$Fa0&mYKmhQGh>3)7$}bWKju%aF zktO@mUUM+KiYS0Z!Ueg-wOkT0b=F)2k=LN9_*xTCR$0Z&xrVXG={;O!L@JeKHdC8lO0!NzD5vfa^B2{&ko@Ag~CTtBpWKomJ(qK&GJV_m1i z>G6xBSBKB{-yObvbNc%2!HX7BDZd_;V?C)IA2aztQ@1cMF8v zRa9JiCrd?6TiBEpI5#sHHar2Y$@JdF&&}qpL-pL!A?1c24s& zu+7#+sYn@8`l)a;6~HyDa-emk15K+P56~4;&{iANkbMJrX`7I4OH8>5q&5U$T|(Ow z2?sPL^CmY?na>(ZR~t-QJxb-K(q?~>d6d~0UfxQ%V6_>Ug&~#l-BeIH<#Dv2p=!}; zQ)9y7Owd)DsusQaVxpYnxp{V%%iP}d4j`b-bH35^Ys^;N^A}r>uga1Y>RgK_D{ZaI zl$B<#!IhPkuEmy>w$|XwcAv(2hRHHbTXKjjGfY-7Ko_jr8fVoX$XVkMSb)|>5?w91 zA!hkVRx(vB7|@2n)O8%dR~XPgM2^oT**#Yv-MSQ8gZ0fe6aAzF`Nf;nO|=>~sOmM_ z9cB$M8A;*^k<|srYQn=vg|d(;765B35*o<*4Kd$h`qlU@boI&vT)e)j-Sl-m`rVY+ zx{}CRix92XcdbYe|2)DH71g4i=qy$9115&>d6f=Zr?#vXUM35CG4Nw6wpkhxMNgC< zicaQ+w6z=|N-Ln>Gb*cwsl9uk-OghFYK$3bS&)}6O=-L5 z=Puy)Yd_24e{Vkj*Mmn-9zU+;|9U*QJO6Vl&$^MlZ7SVlPWlra@UE#Ji{W)6du=m+ z#L(W|Kf2}d^<#={g0Q=w;#Gr+y^Ly2(_jN(#YIbZam5vytQT0^RC%2cTPNwR6k1%Z zbd8SJ2`+A^dKX;`f6D0MKmF5dOxLUaJ5|BTT_<)I3f&Q@ns+F1x zz_y)S5AZ)=DEDfI*%uOa#o%wVKDdkizKj0$o=ru6d%K%{YX8zP`*{*h1y6V;pYz#_ zDd_U?xVycKf+v4}^Zc*JZ{NIqe{%Z#?d#Wv?-p=*+Y$d--z(}Ftzb$J*zsI0cm3{@ zY<5yWkf$2B=An<946t7nhAYd6o0kiBf$%pL2)};Bca1Iw&sy2$DgoVVs920>c#)hp z7F&FgeB(Xxd3&Yn%L;O>yYztJIs8@$i62pDbr z+-9+anN)mMy^GU-{2=ZHfUaSxlmIl_s8;|pFt*@&&^FXvD@_02;DEy2!I&FA7;|?% zWRvGZY^1com<1e@F_-FGeINh9wxv=LK8={v*drt@Q5$xRw7u;SbEBRuAD5&3lasIN*s22aC-G}F^35XOW+RaP zYWwD#EmNh^m(}xc0I5+E)wd04=*_ciS>z>aHo3%ZY4f^Vw7DgBON$%fqksEz{tMjXO$=J|k$V?)H{e{5d5r*|dgYY@fmS|0 z{VgnX8VvZmv&etcv&i4#f&4#+hj2CT;98?v2I=afxx{IBC5Pd<1KR+@mGP*%8kU_{ zZ>q8U$a+J>e3Of;@!KaR6-y>~P`ruA6$e^gSN&2PFdx>>1+CNvGMIi@3`H74559wj zgUfnFxB2V?e$&h7cdxZ?A3?jIzdL__cm5vQ4&67O|2_}%lc5_p0KVk@$KAo+ZuR{4 zqbGyA^WV4eEV{0B?F+sAW5wU>!EYx*%&F(+ai;a*;Za8`3j6mE-tdHWN(-shIHUAx zVMU6F2~}pDRcfnz^!X|zU5Q3tn<4L>TYXyPzn~=AMDmXn zv)Yk~o6!=Q#lB??QL~Kg>EcZ$^{D0CaD5cXHX7Dku29TL^o+zL3AMT^ohB%?(|zR? z?AmohxpE^AmG*ORax-{!#kc{@2(2f}F^Pe$QMBUTG~!h+z99R^##JyhT*Y@&*|OT? z|As_AK1<~P-h;Q}eDvf_|KG;bn=sXR`{B)ta<`M>+%(St|T{I7eD_wMHZtvp+BWH{jDe_nMuzZO4!yEj!TmBW7j z;^M-OE#_j<-v%auB)~`iq@fA~jnNJ!aS^*bBMF(%nKrQGTq-()dvA{qp1nTVHf9@1 zfX>n>kxT+l0K+dN8?l(FIT~9$0jWq55fRBCn z$P>w9+W8K&{sNXrMdDc39mUiO=+R}d?qQ+?kFroh-08fVGHF&FFydTEB}giPKqDz8 zOii;)O%D70kNK+}KH4{k z(Kd{^@PXE&;0+@mgN6m3oAz1app`xFs5m7OMwBL?s8lAj{ep{+28K5;58wPR944AX zatA^xRA4dKfpMJCL`B%|k>E3`rZhwDP(0P$BR!t|?dZ6SK`MCEW4ym@c;iS=48E64 zQQ7GPP>>yVd%Jr-bOM9OUT=5z2jIf|de9pLBdW*){r%_{NCiJSv&LD& zkR=fn@mvoPOU8moC9+Uuf?EDU4s-*EL83Kr9YPT|)hBFKztF8J-$9gnkWfG zN**$z+ypHc!wgi66+Hg1{GF50bUNm7%r6j8J^Iq4cs^jGxI}(CmpmRQ`c+fPL_Xx_ zW--_4To(PpD$gYxP!kHJncx;5-LZjvt=gSS2(I$L70Ykzxn9SJ=$Sn^glDvl>4Zro z=4b7MeTl{40973v>RKRdwd)MovN3_PgvG6JEvxMzy6}k9S<#<2F^a=j{G1BGA}Zmm zq4M@0AJO;rL`^fI?z~M4qIu;%ETS&<&4fsvmHR`#5FdjpWFs%Nb&^SnmXAjZ7nUna z#He(z3L$QIL|vqD^N5mY#*!SDmT}FSxZb$l2%^@7c*KROwp>CKZ8IM|=X=y2Ho~xe z$2Wkex9wgX(n~llKH?;89#MbiKHbEaZVu6E^7`gBtLi0JEOw@9;iGLPy>cW%kd0}a zOE~-Ww==jGaVjyWmqd{{+6-x?SCR&5Eh>yo!OY7K}gvOx}A(DB*lcl$3fHRG2n52=S}k%-``BZ{&hH8~7!?zx1s z!|@xgjs%r7sj0nXwabVul((+G6PtJ7urBd#@^QjwJ+bQ{im9%b)2Tp|jlrbHHfvju zQ=Y{Uj3`LD-lB0CqL`w4J-LgBYGLx$s`{nPT*8S$FN)%N$*&lR<2g7>0UClfirrS@ znFVPYxOc%c%L|%P;}x2Gom#XeDHQM0LgWkX#4U>m%BB(?+9ay@bPrsp9ah zJ<$`SMRhxIHmqJZphw#gZP2W%Bf4Sr>gmxLOISolS3!3nqK%q$bwoF;UN7NHDnV5^ zU59-aBHE}~S4VWi>h%)N_TvjOU#R@9ifC)Tpfk7x%o}ugfOfo=m8^{TO+A-zHdt2_ zSwt&GmKV2n34bN$#%b;}H0C6r5w6W(%wmcTd`cp8xFqLrPQ>+%i4Ew{hUoT1%d{6J zS(8`m8sJ|6)XgHAA%8CHujXO6oyP)tx)j3Z+Eql;sxz$KU%M;iMjwahQV5%CR}sai zaF_VLsjW+Rlg&m{==q0*Wq!No#&Bu|70!0ADU250G*I^fo=^o|G2DS;3T8Z_J=l+9 zenBH^=nvtHCucjliy2q+l%#1q_kLdprl|fM^N_@6xEXfF$Kz%Sw|!X)(dIi1N_w;y z(R?Uz6BTXrsTRpT8T`iK4Oess&oN$EFLNe%8>`k*Bw55+z0IhX1ve!$Sw=RL7QCWp z?)9{W9cM8rSQ4F+B&1PEI7_M&OD5~__$8dp=BHk#SHc<*U9bVs)Ze;W64!h-uzFdr zrPS14ZA;>Ibc_e+8rW74t?jfl_1C`la-+{BoC=aed~#kBq?Q>b9|n=o zm??O>ifD0F*#gVRL)7G(0nGdYL` zquN&AG2H~BwTy5Jr7pT1Odw`D`A43DZcPYQ2Ne;8dl5%By}8RDyc~Gj3uUNAv^l?B z>xdS)!}7Q|KUA%hLJu>R!98Q+zHN$ds1O*<`Uc8xn}9|mT$u!_|020x!?}-pZ3Ayj zz=lM2ZRBZ0)U~h#<5huOFrwGn2= z!M4t-Wm>Mxhc}7nvL%dOc;jLPv8-Tdf_shU6C_O}Q|w&RVa&zM@USI9Ek?bS++3_p zCyzFWsDE;Eh21nScGn#j#xtKc{9M9WeyDU!=U6EUYI%RvfT~-1F5&Fa!v}k7@@@%H zchq!=yR%_L^FyjP_-tT~Hk?IYG)MQnBkL7+f^BIBCDfW~q`l+MU)IF3ifH)^daKl} z993syho70MOv0Ie4t;%xR<+}ejcD7c^~y%@eiR8Rt%xOp0LlE^Nuz<%#Q>=XA#ANf6n81cGQ%_^=y+;X~QF$yIIuQPQ~g9+jWk5> z(c0ykEhgZ86fv9P#2N-Ob64lh;(Q6$kwz_uE*85rOmB;KuZ<~2FRf@WT+L#@mN(>O z*Z}VuL{~Eyu5K}G3fYYzx|+eTj>WJUR5xHT*b!aHV7Q{iAUB8c#zu5CgJE@xVS_l@ zjyHtpY6inv7Q;q(caw;&U@%)84TC37_N!)#`Nfl2E$qw!-jbG+L$UT zW-P(^_EM0L9{Km4HU*oT3rsCecy%}8>e`_&(ao6Ch#Px0;0MevqFSYuOL%_tzWrlE z3-r^C?|pAz_oa&HSiTfIYY9xb1pMe9PrLb(X{O_|-N9hpGa-$LdWY_s`l}wpyB$0? zgD7qV?}wq*6HE3EJ=QfF-NaOLku@%`!M0|OiSQ(z!-xXO6rT~rLOdF70{S9g;~ESm zw4a&t`I4&bjhs#^I`MmHtI>2$qKxv*JR6X@jaSpRdyPI|9)dZ~AZ8yakn+9tTF;Cq zxLbKP_-sHLB~)E-@zI?BU5=^c-8X)%>tL`ds2fPdP(l==FJ637R3xNcb$@^57r@7gCYWD?iZK^6V-O;)H}zb??q(4!-^pK(W`++_ns|um z@)CG<)3(tiJi1QpI-=EU{m-iXy=DoUWive)wXK1%s#eSu{8ltw+a!hT(aTb@cwG>#0=SO1W4yW%#;>t4M0W$brn3@)Moi{--q6X3&5!A8o-oBl zi*z!nVbqkNgWeiuio4^hC#5z_?5B6-NJ<6K@7F_g!{W#xN^&Okx7td~>$cUr>oxFj zo#a)F(IGgZ5HT6@bJR7Xxle;t5WRK~PBWsI>NzDLt(<;&n_4vU0>Bcun6hx%KD5_= zZjNhl*xzU#Y-mPyNlxr>bywPl`7=n}mG)u2WvPoH*e{avl?2))>?d=ZwUZ|2OmK7F<(vp+?uFIz z*hprS%6LDFwD&8!nC3N-k8QvYn57X>^h61w=w$hUruSGJq%D8*D^b_+*=IFJ=ln6p ztA!09-IcJv>~!4nl9F&*T*vEO_I(xvk}CK}=TC76e%;Cc{Ov5y#flDX|0d|O9{kJ& z&`)HR#&qXucSirC17NzXXFGms7M?DR@lHKW@YC5m$Ws98reM(})MUiZi(8WM;7%pE zqnEPTg?Hvjlr`FAfX;fJ^E;jcbLlBHv*uuFj3d0_*}XdJbvpYohV5x_5YKmjz~A>@ zzcSZXp!;0nU@?0Gv%M=Smlj2CQR`+;O?sOME~f4V4Db4Byk7Y_qZEwYRm^iTYrWmx zyD9tcJVVejNlSNl|LQgK+dc2cHeN1W%87}&GX2ZAA)<<~Sn8r%@T;8#@XHAt&Sn|H z1d2?&8|EV(%AWPi;Vtv|0$lc|{31}ikGHAgRsVr74F}k=^Ub~S1Q4iNjwKZ zH5cVMT{V`h1Ol+cY(?9f{(YHf@2jVYL}fSUS?S*U+zS&D4 zih;=8EETcwmmzwMtTVVp%c-_8s(1qIEKug=Vzx|mWyQ3YwP95X%t`K){^xnLA8lF^!}4JO2U zmJ-2ap|}K{WHAvA1k-P?G6$HfT~6aq44&wb3L0*G8S2TU2elJG!xb_^K%Z9UhvtuZ zMPg=W(0(mPTNkAZ8HTW1>7?>`&-Jt5R*cr3s&{P+q07hPZnejzVoA4-;1zGFXalSH zW{e>WDsZ)x+s#SR9-QL$T?u(9#wPSoj$HWtd6Q61`9)$BwG|J~jTH1# zBN<_Yn0u|%OPH-WmUzU-;gr@GBB+vBR9tvgXpd<&>e>A(tvf_Rg0sFB_Wf<&i1M~@ za>UBI*DLdMcds|-4T6Y{7)gSk0x>ux3d`ULMpDvF$>+86Kz0^g=H}bUUE%eIi6KG?yS=}93x>O zJmGRd%(BE+USB2Jo>vyk>^N1Gr(Bbi}|77_M-lxdC_?{ z<&xT(SDhJA#F9y0lm0>*Sat{-FBB*VCSxp}MogSpP}sR0Q9Mc{H#b1-lkh5RY}^S~ zXXchX_*dcO3LjfGJ~WP4EkX&51(j2?KuJPsmYBafTf1PqLZ3{S(keyk5~oS*zkyLR zK|#iCtB~x#XzJO8hA)`&oafv+dmBOP!C?YiM0(^n&`&hXl)XdI)W>L-Im|Mt%&>>% zNGAHBiWQ{0d*;lrxrLb~O3YKv5*3&f>AmO-BsWdk<*8n1W;5r<$%kaQsIK5y;^xv7 z03}(hCGm~O)tt!o$}iLV5^iiuDt~R!3PuQu7O<8i!IxP$MNuPYL=#cKXlWlYa=kV-u4%NTijfdgMmVW~K0}wHWu*hb&><(CdV=Aq`j`7GYHBb<}n3a29Y(N0%K8KgxKHwsz} z|DI97v;;q)hCt8tNPjQOt7mh~UgAt<90wF4v2v6#Fz7kwVbMs53&Arg6B_51>#3{0 zWt3@<(2KxA9h@@~kiZZ)o!n(t97?06VF(VvgS$)6U_pbsyIbS#E(spogS!))#+ruU z9z3{P<8DEE*n6FK)|pvnzRa(vT2;?|O%*w0{ueNNRKOEZ*(e zrdpT^en&JDhCH~3VW^ss)UFq2r#NVciya@mp4mm(ZK{Bw7-G8Vx{I$WGNVF zul){=p81!;lgI7K4%KY3YS#YE^+3Cl* zdtPXEBnk$f7*hdzpaRx}2_1ePP@%DrL4-uc4Fb1iJ+QysceVnC12YZ^77eA>^j;?YZv#{MaGTwBXG`h-F z+f^3M4|S(aVFA)cS`mV<*dz&h83R>FPs{1!HF}^0TQ5&K9=owtU@WKVJXzN9I3>uH zhfbUBwP?muzi`b@Y+sNEvYM6pmfROBWYO6m8gzS#z^dZ)swELTN8pRAyNcP9c=@oF z@nE3(%ZBhwXPwi^e$NsRy7Kg_n`NpEg-P*!e<6+x*eZmRpLCYj0MwW*6mF{fovAu= z7SO@63Efc)Ks2RLY4sP|=7`_mAAfaha!nsL2P- zmR;V*}p?2+-!t z+O3dfWOCBPeGbZ%cK(ZU;>{Lr><{tld7b{g&)d5J5kUjDeC zxR?qB{y<)%_qXBLJg;TSTG!kTbtA3Hv&|XvPf{@s5jFE6qVZOFG zsla3|qR?@VNI;1@;dEGh=BKo`KZW|s zQ^ObiKYFlJ#?r5v4y!yk?+N{{Qt9$0M(y_`D@kEe-m_zn4 z_gb^%v6g*vn5d&;&yxqWtfXMB#4R{1%nDh|A|eu!Q)PQIl^_9pNpdP2OU{Qm!%A@kNm%nVla*e zZh^AoLuNi@*q!qb;~Q%DNX)DfFIYl$ESsCA&o5beV$#$$qOfO7BVyV@t<+-O$kRp<0WL zKJe3a<`Kh<_?>s~^PyRHeYiGbr{j2KDbHYPg`#}`-D0^`2cyoS7d^`(0|8rwa{6Hl zAadl4sc^I59{g=7vZtWe;YP*&J8{vW9lhP?C%aFsTzxC4NvT;_yv*vNS;;R$7)}D0+`GsXEnLs?2 z-0Rp$Kh|#L*}z$=gD^YL@St8iQ=ByIGx7l0wB|7MAUw)TAksM~2<_tL&}8#95VuV$ zlOef^CH(1L(6`5=`4=6p@Xs}IYZM>1&8`+EAS;u@Q~Ct@j0ys1t|bSbZtYLP;8i0~3pHuaDv0Xj+(TzB|KNU~O}om3a$ux84X^ z(HTt_9OdgkB^E_ZU%<7_$*Ny7{;1IndUz%lx7y3*nYH%Vs|UDrr$mmO+;~oYTSR&lKU29oD()qyM6h>ge=}O#KaJd zp`Uab$)J~>{vmplg*|mL;!6m-kS>x2uNJ>|o==TN+b<((?CswR7Y93PE9%2@FUH0> zA;(H(l-gk!wmI+!XFEpoE-VJ$9-T(c64WKEzX&@&9j>mYd^2uxb9&d*B0=t9AFZaC zobY<}Hfw%K!Ghd!lIS0B?^j0jvc|M*e&geD^G6vO)3hx-)utH`Lxif=kh$=wWR|+Q z3Z^WT4oxBMVElRi9izwiU`oJa=ma;5XmGPW?}N(8;_>L99%**xg$8kX7VRB`NIfTk zky%3Ls3vZX^OGrB9*X>wo>SbWGQ(G{F#U;1ms_DCZdkF2Y03c-r>*`{t-Z3EJg0IG zviYJGc?9tO^Zlav<7lIlQo4elW4eg0THB90aW~G*eSNm#NklO8^yQo=7G6#fzoyA3 zKx0Nhdc+i=>GmU@5|AGawsXKT+2}*|oGZko&D-m0`{7650Xv7@M5Z3zu66raE;!Ir zHLBi`&pSq{-5MpkPXjowVk=kHW>mj4VY8C(^0I#lB47}{9Vts=hP9*Fz1my1!Jt;a z5juPq7Zl7Yw(A>Z!edXSuQMjRgCdIKe9*y^kY1O6kHfw9C)p$j{^!6Jwhh;$?~GAC zQ^IDRL4}V;4=)=0*0_wE%3RZF%LK+-mx;xJ;P+WAxcznGsMLC#0hgOSptRPLpSXj+ zdykk-=fMj5Ctea;Pzzvk5gUPE?$Sheki)rTlvZoeu^3CS$66YBo#3@T^u_z>4f;9x z(Jg8+G1FySXkp9Fa3zxWf~Tj>-wT<*c1LtcXC|RnaiedJ{~N+!NrYB>qhB7ehRey1 zGVzUZEngfQWiQKH?3J*_BH;dnjl#0jJw8HD%7E@l{?}rvpNjbpFfZH^vzg!F8jlT8 z&rVF0rj0z>>*&>+%k%h-Lm$7bA=C@ARRNONzn*cvNVyQ=F`)WDe+8bYy$(islbnqU8nRkNm3df7o6T&d7eHZ*5alx@XdX?`==-`la zb=pwv+Bu2#$bH)7Vj{O)dtdaMo>p&c2^b`I$0`hbbWTuqRDaKs4OJn>V7`VYDL;%E+?Kw=oC8V(XtQWg}fzs>uJ1{^f4j;96TIQ5WEraJk(+HX0p)VleZ zKLvZYw44ydnIUx|Ok#-yA3DIMW)t>0oV^A~ST!?l@$mPz!hF+taQ1NO+6Ovpj$CKR zR%Q!d1_9<&?MtezDV-g^73@$M@(V}*mjecTwu1>crrpLm`d0ya|Ie{gk;<}nSpZiL zOtV&IKyW?rMS6CiS(+~R8}y}%(*04m-ZkkOAtUS@P*kD0aSjtl$0OtQIpF-@D|DUv zheqKew*-A9GCfg@4TWIhOO56f<%FUa=QW1G3~6Gh0DSky+1ZutXZO5%dRS`sO#3wL zy|BE}h&T+7gGX!Hkx_3}9c98BW*=6evZTpe%00Ct`kFdU&S8W~(*w!rUvZurW*~&h z$=1IE)Lots!UfQ91$na{xtyF)6*4pcwDimy79kOC%}*<|^5yZP@;Kj99VS&#vBRgH z#2@gOyOJKe)8=ZcNvve1Lyg^~*_b{!u~9jQz92G=z&1r?Kb%}fv$ats>}A=ggK*Mo zMW0r%u7k+BL?#~%%Sfx{>Rl{bTMff9{)0hspL{~c1fD-voYwv4hMst(@GWb)%Sjq1 z$0g+5Vruw6$S3|nBnsh@Ka5SBlBETfxSl^hlY;vHO85_So^`GJAQj{gk42AAAqMe| z{r?GFS;feAzod`~+~E5@8T@YG;DFs~3r@*L=ke!zgSpI7Bs_smuZySi+$UP`qj=m| zmJQ~%ilLnp#UZWH@jS`W+@zNk@k>CYL2UYZK7Z5yN#n9Y3f!Qz1&jO|@CWfzEI!x0 z#0Q#zn8h3|1U|)-F_NRq!m%=b-pNY;ZBq*qXBkyW7~RzD{*t@FQ7H( zw(Nr*$|P45gfNud#XpeYORbTnYDTR+&>}gVYQ4*!bIwb~wPK&$^2!Tl`C8hXWXO2# zC)gjC+jmV+_?3{8@3%RL4-YTWV$OR;q;w>ZmlYI#BRB$)6sFm-M=!}hFS+NuEV@?lIfAK2>3%jUo|52%U zzak1$HdnE;@hm|D=}3#AuMxZ)^2mp)e{OoU>YSesXzD0$nT^Y374F0s&eu~6Ou6;t zQN5#w*jbQtj2ZHwF&)hi%`qMoRXW4SqD!3GEMaA(cM+ok0zsX}Uf+8)x!+OKUqx(Q z+j>NSPtk;4)EbRz67bx??UJ-XZGM!3a%=oyR#H5&BVj zFc$~3hMbtbjt+8r+LmQ`*K$_3d5RBd-oAB}2tE==Dy9D=o<h>Lx|T|aP_wky zpfTM#E>MsgEt`hz7XB`#+P8|JdOsrj;A}M5a6jT~0jmx=6!<1qWyqCjti$}$B1wQ? z6O7jW%r)CFp>-DeRhxquVmM8ZhdZmCE|x?)0zP&=#D_yqKx}gf5k!u}|4T!1DW2S( z&(FYle0k>I*vP0K~hp{Pm~tK(FpUwB3`&2BZ0Jmr(RWbj0=1DGz0J1iSPrt z&&NGPsKHx(gzE4iW$Ku~Pi(ubo327LB%HZ^BDP(0ifs4y{akDhinw1aL-hC`IT%oA z-edlshPbf;KV>e^C6d*FvppKd&~Ipi#xfmu{5R;G$;&``>+6xzlw;M*X-8;fyUxcf zdm1+G2BTzGp$+rpnZIK`CTG4qiYr-94t#R>LxPDH`Z|zO#0cH5?JdGNZfw;|)7Qq8 z5ZKl169248PF#}pXNfSYfZ5Tp?o3+83lsW*Sn4gARo%|>#n`;aKv@C&@;@CB92){LF}B?McG zwjSlPfU1DEJ+KRI==Fvfb8BHkz%=q#TCE;BQ-bj$ojUm`Unc&*fFJt%tC&`e92xZj z|A_#xx(YZO;9yo3C+pnOO+P6Mn=8}rCP8{IKz)cjdrPgmCY&Y9+F-gG4lw6OQst?$v&wk>Yq=|b=0-bm47kr(7JzAM3*yHs z#9ZW=AM)F-lTP6<8M3cOWb+a}vAjl);s;bGUIKdx`8tRP(-Wc^#8XG5j5dOq-f_oi z=5~7D_SZTm-j#9>h7va+jv2aQP>>%;)2#kiw+np@en>xx@qsayedz z@X@K?bhQfk+;#&qRe`sf9UN=TNJQB%D=t~cvD%mTKrP{#T#Cvt@=g~Tj(S&;pC&82CGc)8|OJjjV+Z*f5+ z!&O1$^jT$c7YKRLxU!H4WZ{S2UtFdz0I$14!Jz9et(O63Fdp4pfORtQg{Minr~zg!a;v$xTyO&c)?__!~u!5OqTDX|`% z0AREFYp_q;k7?pnMb^Pl_0cm!r^@qyx}SLs?-f!OBy^Rr?TNGHKk_Q2;Jj?c+UNgk zSLX9HP+@~?3Oj}3*aSg|nz{ZoLyFuHo#o+-N1;Pv(ebaih^HEeH^4uiqjV*3q~mdj zoeXLs)3$nM6z;&^G2}Ig9)Er0OO>L4g!hxJ!<5OG!cfh**0JJBT0f%t-BVG^MYdke z!aBRvSDcHoZ~&YRc(HeyE#;o41-?cpO&2hANAw7AzPK1le-|%j?63F^!G5>2j~j%c z*UP;Euarjt-oO$=bVMK(v%>T9$a;0kR(`aCFymHjJ{9TDI(+O+#~ja%89Rss!mX4I zMO|kkmq_L4sm!^;>C&|+G1LrT9&(tY9I{dxuEhZ~92prvt i*B<|RzG0Q-cE2j@J*gxFHTlN|hvt}vxWI_MgZW=6jEw^T diff --git a/infra/charts/feast/charts/feast-core/requirements.yaml b/infra/charts/feast/charts/feast-core/requirements.yaml deleted file mode 100644 index ef1e39a7d0f..00000000000 --- a/infra/charts/feast/charts/feast-core/requirements.yaml +++ /dev/null @@ -1,15 +0,0 @@ -dependencies: -- name: postgresql - version: 6.5.5 - repository: "@stable" - condition: postgresql.enabled -- name: kafka - version: 0.20.1 - repository: "@incubator" - condition: kafka.enabled -- name: common - version: 0.0.5 - repository: "@incubator" -- name: prometheus-statsd-exporter - version: 0.1.2 - condition: prometheus-statsd-exporter.enabled \ No newline at end of file diff --git a/infra/charts/feast/charts/feast-core/templates/configmap.yaml b/infra/charts/feast/charts/feast-core/templates/configmap.yaml index da45cad5bdf..bce32ef33a6 100644 --- a/infra/charts/feast/charts/feast-core/templates/configmap.yaml +++ b/infra/charts/feast/charts/feast-core/templates/configmap.yaml @@ -10,44 +10,26 @@ metadata: release: {{ .Release.Name }} heritage: {{ .Release.Service }} data: - application.yaml: | -{{- toYaml (index .Values "application.yaml") | nindent 4 }} - -{{- if .Values.postgresql.enabled }} - application-bundled-postgresql.yaml: | + application-generated.yaml: | +{{- if index .Values "application-generated.yaml" "enabled" }} spring: datasource: - url: {{ printf "jdbc:postgresql://%s:%s/%s" (printf "%s-postgresql" .Release.Name) (.Values.postgresql.service.port | toString) (.Values.postgresql.postgresqlDatabase) }} - driverClassName: org.postgresql.Driver -{{- end }} - -{{ if .Values.kafka.enabled }} - {{- $topic := index .Values.kafka.topics 0 }} - application-bundled-kafka.yaml: | + url: jdbc:postgresql://{{ .Release.Name }}-postgresql:5432/postgres feast: stream: type: kafka - options: - topic: {{ $topic.name | quote }} - replicationFactor: {{ $topic.replicationFactor }} - partitions: {{ $topic.partitions }} - {{- if not .Values.kafka.external.enabled }} - bootstrapServers: {{ printf "%s:9092" (printf "%s-kafka" .Release.Name) }} - {{- end }} -{{- end }} - -{{- if (index .Values "prometheus-statsd-exporter" "enabled" )}} - application-bundled-statsd.yaml: | - feast: + options: + bootstrapServers: {{ .Release.Name }}-kafka:9092 + topic: feast jobs: metrics: - enabled: true + enabled: true type: statsd - host: prometheus-statsd-exporter + host: {{ .Release.Name }}-prometheus-statsd-exporter-udp port: 9125 {{- end }} -{{- range $name, $content := .Values.springConfigProfiles }} - application-{{ $name }}.yaml: | -{{- toYaml $content | nindent 4 }} -{{- end }} + application-override.yaml: | +{{- if index .Values "application-override.yaml" "enabled" }} +{{- toYaml (index .Values "application-override.yaml") | nindent 4 }} +{{- end }} \ No newline at end of file diff --git a/infra/charts/feast/charts/feast-core/templates/deployment.yaml b/infra/charts/feast/charts/feast-core/templates/deployment.yaml index df834b6749e..1f4fd996efa 100644 --- a/infra/charts/feast/charts/feast-core/templates/deployment.yaml +++ b/infra/charts/feast/charts/feast-core/templates/deployment.yaml @@ -18,11 +18,12 @@ spec: release: {{ .Release.Name }} template: metadata: - {{- if .Values.prometheus.enabled }} annotations: - {{ $config := index .Values "application.yaml" }} + checksum/configmap: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} + checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }} + {{- if .Values.prometheus.enabled }} prometheus.io/path: /metrics - prometheus.io/port: "{{ $config.server.port }}" + prometheus.io/port: "{{ .Values.service.http.targetPort }}" prometheus.io/scrape: "true" {{- end }} labels: @@ -39,23 +40,29 @@ spec: - name: {{ template "feast-core.fullname" . }}-config configMap: name: {{ template "feast-core.fullname" . }} - {{- if .Values.gcpServiceAccount.useExistingSecret }} - - name: {{ template "feast-core.fullname" . }}-gcpserviceaccount + - name: {{ template "feast-core.fullname" . }}-secret + secret: + secretName: {{ template "feast-core.fullname" . }} + {{- if .Values.gcpServiceAccount.enabled }} + - name: {{ template "feast-core.fullname" . }}-gcp-service-account secret: secretName: {{ .Values.gcpServiceAccount.existingSecret.name }} {{- end }} containers: - name: {{ .Chart.Name }} - image: '{{ .Values.image.repository }}:{{ required "No .image.tag found. This must be provided as input." .Values.image.tag }}' + image: {{ .Values.image.repository }}:{{ .Values.image.tag }} imagePullPolicy: {{ .Values.image.pullPolicy }} volumeMounts: - name: {{ template "feast-core.fullname" . }}-config - mountPath: "{{ .Values.springConfigMountPath }}" - {{- if .Values.gcpServiceAccount.useExistingSecret }} - - name: {{ template "feast-core.fullname" . }}-gcpserviceaccount - mountPath: {{ .Values.gcpServiceAccount.mountPath }} + mountPath: /etc/feast + - name: {{ template "feast-core.fullname" . }}-secret + mountPath: /etc/secrets/feast + readOnly: true + {{- if .Values.gcpServiceAccount.enabled }} + - name: {{ template "feast-core.fullname" . }}-gcp-service-account + mountPath: /etc/secrets/google readOnly: true {{- end }} @@ -64,40 +71,52 @@ spec: value: {{ .Values.logType | quote }} - name: LOG_LEVEL value: {{ .Values.logLevel | quote }} - - {{- if .Values.postgresql.enabled }} - - name: SPRING_DATASOURCE_USERNAME - value: {{ .Values.postgresql.postgresqlUsername | quote }} + + {{- if .Values.postgresql.existingSecret }} - name: SPRING_DATASOURCE_PASSWORD - value: {{ .Values.postgresql.postgresqlPassword | quote }} + valueFrom: + secretKeyRef: + name: {{ .Values.postgresql.existingSecret }} + key: postgresql-password {{- end }} - {{- if .Values.gcpServiceAccount.useExistingSecret }} + {{- if .Values.gcpServiceAccount.enabled }} - name: GOOGLE_APPLICATION_CREDENTIALS - value: {{ .Values.gcpServiceAccount.mountPath }}/{{ .Values.gcpServiceAccount.existingSecret.key }} + value: /etc/secrets/google/{{ .Values.gcpServiceAccount.existingSecret.key }} {{- end }} + {{- if .Values.gcpProjectId }} - name: GOOGLE_CLOUD_PROJECT value: {{ .Values.gcpProjectId | quote }} {{- end }} - command: - - java - {{- range .Values.jvmOptions }} - - {{ . | quote }} + {{- if .Values.javaOpts }} + - name: JAVA_TOOL_OPTIONS + value: {{ .Values.javaOpts }} {{- end }} - - -jar - - {{ .Values.jarPath | quote }} - - "--spring.config.location=file:{{ .Values.springConfigMountPath }}/" - {{- $profilesArray := splitList "," .Values.springConfigProfilesActive -}} - {{- $profilesArray = append $profilesArray (.Values.postgresql.enabled | ternary "bundled-postgresql" "") -}} - {{- $profilesArray = append $profilesArray (.Values.kafka.enabled | ternary "bundled-kafka" "") -}} - {{- $profilesArray = append $profilesArray (index .Values "prometheus-statsd-exporter" "enabled" | ternary "bundled-statsd" "") -}} - {{- $profilesArray = compact $profilesArray -}} - {{- if $profilesArray }} - - "--spring.profiles.active={{ join "," $profilesArray }}" + + {{- range $key, $value := .Values.envOverrides }} + - name: {{ printf "%s" $key | replace "." "_" | upper | quote }} + value: {{ $value | quote }} {{- end }} + command: + - java + - -jar + - /opt/feast/feast-core.jar + - --spring.config.location= + {{- if index .Values "application.yaml" "enabled" -}} + classpath:/application.yml + {{- end }} + {{- if index .Values "application-generated.yaml" "enabled" -}} + ,file:/etc/feast/application-generated.yaml + {{- end }} + {{- if index .Values "application-secret.yaml" "enabled" -}} + ,file:/etc/secrets/feast/application-secret.yaml + {{- end }} + {{- if index .Values "application-override.yaml" "enabled" -}} + ,file:/etc/feast/application-override.yaml + {{- end }} ports: - name: http containerPort: {{ .Values.service.http.targetPort }} diff --git a/infra/charts/feast/charts/feast-core/templates/secret.yaml b/infra/charts/feast/charts/feast-core/templates/secret.yaml new file mode 100644 index 00000000000..dd33e2dd487 --- /dev/null +++ b/infra/charts/feast/charts/feast-core/templates/secret.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: Secret +metadata: + name: {{ template "feast-core.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + app: {{ template "feast-core.name" . }} + component: core + chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} + release: {{ .Release.Name }} + heritage: {{ .Release.Service }} +type: Opaque +stringData: + application-secret.yaml: | +{{- toYaml (index .Values "application-secret.yaml") | nindent 4 }} diff --git a/infra/charts/feast/charts/feast-core/values.yaml b/infra/charts/feast/charts/feast-core/values.yaml index 077906dc35d..5032e8d87ae 100644 --- a/infra/charts/feast/charts/feast-core/values.yaml +++ b/infra/charts/feast/charts/feast-core/values.yaml @@ -1,246 +1,151 @@ -# ============================================================ -# Bundled PostgreSQL -# ============================================================ - -# Refer to https://github.com/helm/charts/tree/c42002a21abf8eff839ff1d2382152bde2bbe596/stable/postgresql -# for additional configuration. -postgresql: - # enabled specifies whether Postgresql should be installed as part of Feast Core. - # - # Feast Core requires a database to store data such as the created FeatureSets - # and job statuses. If enabled, the database and service port specified below - # will override "spring.datasource.url" value in application.yaml. The - # username and password will also be set as environment variables that will - # override "spring.datasource.username/password" in application.yaml. - enabled: true - # postgresqlDatabase is the name of the database used by Feast Core. - postgresqlDatabase: feast - # postgresqlUsername is the username to authenticate to the database. - postgresqlUsername: postgres - # postgresqlPassword is the password to authenticate to the database. - postgresqlPassword: password - service: - # port is the TCP port that Postgresql will listen to - port: 5432 - -# ============================================================ -# Bundled Kafka -# ============================================================ - -# Refer to https://github.com/helm/charts/tree/c42002a21abf8eff839ff1d2382152bde2bbe596/incubator/kafka -# for additional configuration. -kafka: - # enabled specifies whether Kafka should be installed as part of Feast Core. - # - # Feast Core requires a Kafka instance to be set as the default source for - # FeatureRows. If enabled, "feast.stream" option in application.yaml will - # be overridden by this installed Kafka configuration. - enabled: true - topics: - # topic that will be used as default in Feast Core for the default Kafka source. - - name: feast - replicationFactor: 1 - partitions: 1 - - -# ============================================================ -# Bundled Prometheus StatsD Exporter -# ============================================================ - -prometheus-statsd-exporter: - enabled: false - -# ============================================================ -# Feast Core -# ============================================================ - -# replicaCount is the number of pods that will be created. +# replicaCount -- Number of pods that will be created replicaCount: 1 -# image configures the Docker image for Feast Core image: + # image.repository -- Docker image repository repository: gcr.io/kf-feast/feast-core + # image.tag -- Image tag + tag: dev + # image.pullPolicy -- Image pull policy pullPolicy: IfNotPresent -# Add prometheus scraping annotations to the Pod metadata. -# If enabled, you must also ensure server.port is specified under application.yaml -prometheus: - enabled: false - -# application.yaml is the main configuration for Feast Core application. -# -# Feast Core is a Spring Boot app which uses this yaml configuration file. -# Refer to https://github.com/gojek/feast/blob/79eb4ab5fa3d37102c1dca9968162a98690526ba/core/src/main/resources/application.yml -# for a complete list and description of the configuration. -# -# Note that some properties defined in application.yaml may be overriden by -# Helm under certain conditions. For example, if postgresql and kafka dependencies -# are enabled. application.yaml: - grpc: - port: 6565 - enable-reflection: true - feast: - jobs: - runner: DirectRunner - options: {} - updates: - timeoutSeconds: 240 - metrics: - enabled: false - type: statsd - host: localhost - port: 9125 - stream: - type: kafka - options: - topic: TOPIC - bootstrapServers: HOST:PORT - replicationFactor: 1 - partitions: 1 - spring: - jpa: - properties.hibernate.format_sql: true - properties.hibernate.event.merge.entity_copy_observer: allow - hibernate.naming.physical-strategy=org.hibernate.boot.model.naming: PhysicalNamingStrategyStandardImpl - hibernate.ddl-auto: update - datasource: - driverClassName: org.postgresql.Driver - url: jdbc:postgresql://HOST:PORT/DATABASE - username: USERNAME - password: PASSWORD - management: - metrics: - export: - simple: - enabled: false - statsd: - enabled: false - host: localhost - port: 8125 - -springConfigProfiles: {} -# db: | -# spring: -# datasource: -# driverClassName: org.postgresql.Driver -# url: jdbc:postgresql://${DB_HOST:127.0.0.1}:${DB_PORT:5432}/${DB_DATABASE:postgres} -springConfigProfilesActive: "" -# springConfigMountPath is the directory path where application.yaml will be -# mounted in the container. -springConfigMountPath: /etc/feast/feast-core - -# gcpServiceAccount is the service account that Feast Core will use. + # "application.yaml".enabled -- Flag to include the default [configuration](https://github.com/gojek/feast/blob/master/core/src/main/resources/application.yml). Please set `application-override.yaml` to override this configuration. + enabled: true + +application-generated.yaml: + # "application-generated.yaml".enabled -- Flag to include Helm generated configuration for Feast database URL, Kafka bootstrap servers and jobs metrics host. This is useful for deployment that uses default configuration for Kafka, Postgres and StatsD exporter. Please set `application-override.yaml` to override this configuration. + enabled: true + +# "application-secret.yaml" -- Configuration to override the default [application.yaml](https://github.com/gojek/feast/blob/master/core/src/main/resources/application.yml). Will be created as a Secret. `application-override.yaml` has a higher precedence than `application-secret.yaml`. It is recommended to either set `application-override.yaml` or `application-secret.yaml` only to simplify config management. +application-secret.yaml: + enabled: true + +# "application-override.yaml" -- Configuration to override the default [application.yaml](https://github.com/gojek/feast/blob/master/core/src/main/resources/application.yml). Will be created as a ConfigMap. `application-override.yaml` has a higher precedence than `application-secret.yaml` +application-override.yaml: + enabled: true + gcpServiceAccount: - # useExistingSecret specifies Feast to use an existing secret containing Google - # Cloud service account JSON key file. - useExistingSecret: false + # gcpServiceAccount.enabled -- Flag to use [service account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) JSON key + enabled: false existingSecret: - # name is the secret name of the existing secret for the service account. + # gcpServiceAccount.existingSecret.name -- Name of the existing secret containing the service account name: feast-gcp-service-account - # key is the secret key of the existing secret for the service account. - # key is normally derived from the file name of the JSON key file. - key: key.json - # mountPath is the directory path where the JSON key file will be mounted. - # the value of "existingSecret.key" is file name of the service account file. - mountPath: /etc/gcloud/service-accounts - -# Project ID picked up by the Cloud SDK (e.g. BigQuery run against this project) + # gcpServiceAccount.existingSecret.key -- Key in the secret data (file name of the service account) + key: credentials.json + +postgresql: + # postgresql.existingSecret -- Existing secret to use for authenticating to Postgres + existingSecret: "" + +# gcpProjectId -- Project ID to use when using Google Cloud services such as BigQuery, Cloud Storage and Dataflow gcpProjectId: "" -# Path to Jar file in the Docker image. -# If you are using gcr.io/kf-feast/feast-core this should not need to be changed -jarPath: /opt/feast/feast-core.jar - -# jvmOptions are options that will be passed to the Java Virtual Machine (JVM) -# running Feast Core. -# -# For example, it is good practice to set min and max heap size in JVM. -# https://stackoverflow.com/questions/6902135/side-effect-for-increasing-maxpermsize-and-max-heap-size -# -# Refer to https://docs.oracle.com/cd/E22289_01/html/821-1274/configuring-the-default-jvm-and-java-arguments.html -# to see other JVM options that can be set. -# -jvmOptions: [] -# - -Xms1024m -# - -Xmx1024m - -logType: JSON -logLevel: warn +# javaOpts -- [JVM options](https://docs.oracle.com/cd/E22289_01/html/821-1274/configuring-the-default-jvm-and-java-arguments.html). For better performance, it is advised to set the min and max heap:
    `-Xms2048m -Xmx2048m` +javaOpts: + +# logType -- Log format, either `JSON` or `Console` +logType: Console +# logLevel -- Default log level, use either one of `DEBUG`, `INFO`, `WARN` or `ERROR` +logLevel: WARN + +prometheus: + # prometheus.enabled -- Flag to enable scraping of Feast Core metrics + enabled: true livenessProbe: + # livenessProbe.enabled -- Flag to enabled the probe enabled: true + # livenessProbe.initialDelaySeconds -- Delay before the probe is initiated initialDelaySeconds: 60 + # livenessProbe.periodSeconds -- How often to perform the probe periodSeconds: 10 + # livenessProbe.timeoutSeconds -- When the probe times out timeoutSeconds: 5 + # livenessProbe.successThreshold -- Min consecutive success for the probe to be considered successful successThreshold: 1 + # livenessProbe.failureThreshold -- Min consecutive failures for the probe to be considered failed failureThreshold: 5 readinessProbe: + # readinessProbe.enabled -- Flag to enabled the probe enabled: true - initialDelaySeconds: 15 + # readinessProbe.initialDelaySeconds -- Delay before the probe is initiated + initialDelaySeconds: 20 + # readinessProbe.periodSeconds -- How often to perform the probe periodSeconds: 10 + # readinessProbe.timeoutSeconds -- When the probe times out timeoutSeconds: 10 + # readinessProbe.successThreshold -- Min consecutive success for the probe to be considered successful successThreshold: 1 + # readinessProbe.failureThreshold -- Min consecutive failures for the probe to be considered failed failureThreshold: 5 service: + # service.type -- Kubernetes service type type: ClusterIP http: + # service.http.port -- Service port for HTTP requests port: 80 + # service.http.targetPort -- Container port serving HTTP requests targetPort: 8080 - # nodePort is the port number that each cluster node will listen to - # https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport - # - # nodePort: + # service.http.nodePort -- Port number that each cluster node will listen to + nodePort: grpc: + # service.grpc.port -- Service port for GRPC requests port: 6565 + # service.grpc.targetPort -- Container port serving GRPC requests targetPort: 6565 - # nodePort is the port number that each cluster node will listen to - # https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport - # - # nodePort: + # service.grpc.nodePort -- Port number that each cluster node will listen to + nodePort: ingress: grpc: + # ingress.grpc.enabled -- Flag to create an ingress resource for the service enabled: false + # ingress.grpc.class -- Which ingress controller to use class: nginx + # ingress.grpc.hosts -- List of hostnames to match when routing requests hosts: [] + # ingress.grpc.annotations -- Extra annotations for the ingress annotations: {} https: + # ingress.grpc.https.enabled -- Flag to enable HTTPS enabled: true + # ingress.grpc.https.secretNames -- Map of hostname to TLS secret name secretNames: {} + # ingress.grpc.whitelist -- Allowed client IP source ranges whitelist: "" auth: + # ingress.grpc.auth.enabled -- Flag to enable auth enabled: false http: + # ingress.http.enabled -- Flag to create an ingress resource for the service enabled: false + # ingress.http.class -- Which ingress controller to use class: nginx + # ingress.http.hosts -- List of hostnames to match when routing requests hosts: [] + # ingress.http.annotations -- Extra annotations for the ingress annotations: {} https: + # ingress.http.https.enabled -- Flag to enable HTTPS enabled: true + # ingress.http.https.secretNames -- Map of hostname to TLS secret name secretNames: {} + # ingress.http.whitelist -- Allowed client IP source ranges whitelist: "" auth: + # ingress.http.auth.enabled -- Flag to enable auth enabled: false + # ingress.http.auth.authUrl -- URL to an existing authentication service authUrl: http://auth-server.auth-ns.svc.cluster.local/auth +# resources -- CPU/memory [resource requests/limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container) resources: {} - # We usually recommend not to specify default resources and to leave this as a conscious - # choice for the user. This also increases chances charts run on environments with little - # resources, such as Minikube. If you do want to specify resources, uncomment the following - # lines, adjust them as necessary, and remove the curly braces after 'resources:'. - # - # limits: - # cpu: 100m - # memory: 128Mi - # requests: - # cpu: 100m - # memory: 128Mi +# nodeSelector -- Node labels for pod assignment nodeSelector: {} -tolerations: [] - -affinity: {} +# envOverrides -- Extra environment variables to set +envOverrides: {} \ No newline at end of file diff --git a/infra/charts/feast/charts/feast-serving/.helmignore b/infra/charts/feast/charts/feast-serving/.helmignore deleted file mode 100644 index 50af0317254..00000000000 --- a/infra/charts/feast/charts/feast-serving/.helmignore +++ /dev/null @@ -1,22 +0,0 @@ -# Patterns to ignore when building packages. -# This supports shell glob matching, relative path matching, and -# negation (prefixed with !). Only one pattern per line. -.DS_Store -# Common VCS dirs -.git/ -.gitignore -.bzr/ -.bzrignore -.hg/ -.hgignore -.svn/ -# Common backup files -*.swp -*.bak -*.tmp -*~ -# Various IDEs -.project -.idea/ -*.tmproj -.vscode/ diff --git a/infra/charts/feast/charts/feast-serving/Chart.yaml b/infra/charts/feast/charts/feast-serving/Chart.yaml index 2e9cf89243d..7c8e6131cf7 100644 --- a/infra/charts/feast/charts/feast-serving/Chart.yaml +++ b/infra/charts/feast/charts/feast-serving/Chart.yaml @@ -1,4 +1,4 @@ apiVersion: v1 -description: A Helm chart for serving component of Feast +description: Feast Serving serves low-latency latest features and historical batch features. name: feast-serving -version: 0.4.4 +version: 0.5.0-alpha.1 diff --git a/infra/charts/feast/charts/feast-serving/README.md b/infra/charts/feast/charts/feast-serving/README.md new file mode 100644 index 00000000000..7882463977a --- /dev/null +++ b/infra/charts/feast/charts/feast-serving/README.md @@ -0,0 +1,69 @@ +feast-serving +============= +Feast Serving serves low-latency latest features and historical batch features. + +Current chart version is `0.5.0-alpha.1` + + + + + +## Chart Values + +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| "application-generated.yaml".enabled | bool | `true` | Flag to include Helm generated configuration for Feast Core host, Redis store and job store. This is useful for deployment that uses default configuration for Redis. Please set `application-override.yaml` to override this configuration. | +| "application-override.yaml" | object | `{"enabled":true}` | Configuration to override the default [application.yaml](https://github.com/gojek/feast/blob/master/serving/src/main/resources/application.yml). Will be created as a ConfigMap. `application-override.yaml` has a higher precedence than `application-secret.yaml` | +| "application-secret.yaml" | object | `{"enabled":true}` | Configuration to override the default [application.yaml](https://github.com/gojek/feast/blob/master/serving/src/main/resources/application.yml). Will be created as a Secret. `application-override.yaml` has a higher precedence than `application-secret.yaml`. It is recommended to either set `application-override.yaml` or `application-secret.yaml` only to simplify config management. | +| "application.yaml".enabled | bool | `true` | Flag to include the default [configuration](https://github.com/gojek/feast/blob/master/serving/src/main/resources/application.yml). Please set `application-override.yaml` to override this configuration. | +| envOverrides | object | `{}` | Extra environment variables to set | +| gcpProjectId | string | `""` | Project ID to use when using Google Cloud services such as BigQuery, Cloud Storage and Dataflow | +| gcpServiceAccount.enabled | bool | `false` | Flag to use [service account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) JSON key | +| gcpServiceAccount.existingSecret.key | string | `"credentials.json"` | Key in the secret data (file name of the service account) | +| gcpServiceAccount.existingSecret.name | string | `"feast-gcp-service-account"` | Name of the existing secret containing the service account | +| image.pullPolicy | string | `"IfNotPresent"` | Image pull policy | +| image.repository | string | `"gcr.io/kf-feast/feast-serving"` | Docker image repository | +| image.tag | string | `"dev"` | Image tag | +| ingress.grpc.annotations | object | `{}` | Extra annotations for the ingress | +| ingress.grpc.auth.enabled | bool | `false` | Flag to enable auth | +| ingress.grpc.class | string | `"nginx"` | Which ingress controller to use | +| ingress.grpc.enabled | bool | `false` | Flag to create an ingress resource for the service | +| ingress.grpc.hosts | list | `[]` | List of hostnames to match when routing requests | +| ingress.grpc.https.enabled | bool | `true` | Flag to enable HTTPS | +| ingress.grpc.https.secretNames | object | `{}` | Map of hostname to TLS secret name | +| ingress.grpc.whitelist | string | `""` | Allowed client IP source ranges | +| ingress.http.annotations | object | `{}` | Extra annotations for the ingress | +| ingress.http.auth.authUrl | string | `"http://auth-server.auth-ns.svc.cluster.local/auth"` | URL to an existing authentication service | +| ingress.http.auth.enabled | bool | `false` | Flag to enable auth | +| ingress.http.class | string | `"nginx"` | Which ingress controller to use | +| ingress.http.enabled | bool | `false` | Flag to create an ingress resource for the service | +| ingress.http.hosts | list | `[]` | List of hostnames to match when routing requests | +| ingress.http.https.enabled | bool | `true` | Flag to enable HTTPS | +| ingress.http.https.secretNames | object | `{}` | Map of hostname to TLS secret name | +| ingress.http.whitelist | string | `""` | Allowed client IP source ranges | +| javaOpts | string | `nil` | [JVM options](https://docs.oracle.com/cd/E22289_01/html/821-1274/configuring-the-default-jvm-and-java-arguments.html). For better performance, it is advised to set the min and max heap:
    `-Xms2048m -Xmx2048m` | +| livenessProbe.enabled | bool | `true` | Flag to enabled the probe | +| livenessProbe.failureThreshold | int | `5` | Min consecutive failures for the probe to be considered failed | +| livenessProbe.initialDelaySeconds | int | `60` | Delay before the probe is initiated | +| livenessProbe.periodSeconds | int | `10` | How often to perform the probe | +| livenessProbe.successThreshold | int | `1` | Min consecutive success for the probe to be considered successful | +| livenessProbe.timeoutSeconds | int | `5` | When the probe times out | +| logLevel | string | `"WARN"` | Default log level, use either one of `DEBUG`, `INFO`, `WARN` or `ERROR` | +| logType | string | `"Console"` | Log format, either `JSON` or `Console` | +| nodeSelector | object | `{}` | Node labels for pod assignment | +| prometheus.enabled | bool | `true` | Flag to enable scraping of Feast Core metrics | +| readinessProbe.enabled | bool | `true` | Flag to enabled the probe | +| readinessProbe.failureThreshold | int | `5` | Min consecutive failures for the probe to be considered failed | +| readinessProbe.initialDelaySeconds | int | `15` | Delay before the probe is initiated | +| readinessProbe.periodSeconds | int | `10` | How often to perform the probe | +| readinessProbe.successThreshold | int | `1` | Min consecutive success for the probe to be considered successful | +| readinessProbe.timeoutSeconds | int | `10` | When the probe times out | +| replicaCount | int | `1` | Number of pods that will be created | +| resources | object | `{}` | CPU/memory [resource requests/limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container) | +| service.grpc.nodePort | string | `nil` | Port number that each cluster node will listen to | +| service.grpc.port | int | `6566` | Service port for GRPC requests | +| service.grpc.targetPort | int | `6566` | Container port serving GRPC requests | +| service.http.nodePort | string | `nil` | Port number that each cluster node will listen to | +| service.http.port | int | `80` | Service port for HTTP requests | +| service.http.targetPort | int | `8080` | Container port serving HTTP requests | +| service.type | string | `"ClusterIP"` | Kubernetes service type | diff --git a/infra/charts/feast/charts/feast-serving/charts/redis-9.5.0.tgz b/infra/charts/feast/charts/feast-serving/charts/redis-9.5.0.tgz deleted file mode 100644 index 962893a825d6723e5831ad5528d82c00caaa57f1..0000000000000000000000000000000000000000 GIT binary patch literal 0 HcmV?d00001 literal 27574 zcmV)pK%2iGiwG0|00000|0w_~VMtOiV@ORlOnEsqVl!4SWK%V1T2nbTPgYhoO;>Dc zVQyr3R8em|NM&qo0PMZ{ciT3W06ahIued8`H+ILAZ0FT#HuJ6PB&|;q$Di%C-=4H5 zhDb=lm?BsJw4;ghzkdfW5`2oXWZB7@GpDgg-~za~xG!8>FdPuxKbRmUdNVW)|8@({ z&d$!xi)YW&zdJiS#ea8qp6}Ve_50nu-RFCMgWKp!=}EXi>~A|e%fcxR?kjmvOn$?R z6B-TRa<_xxIQ{vox6^yp2{88=iG_OgKE@GnnlK-40}3utphF%@yxnb0X7z(k06(C;f+?-APX z5MPc)zUG5|e@H}xrljZ7X&-S3rq2cPFSxGJC)2V1%Tuqty}g}JdrzMB;%MBtz_TmL z0zT+?pvrVSIR=G}j;97YL>zafh^X;kri$VzBH_Re5;OikPA5755t9A>QNMG_F!OZPOG^+x^q%#0 zI(JR656>zeP2q}M`E3b+X8!+l@9B%DMgITv$yLF;H22)tfP;iF90@>|h=gburX^}RJJ4@V@R)GHW=cM&H^)gB zo?@S2AqUjiFmMK)4S;-|0ZeE1?;|wDSqTS*t_R>bg@3^_{WYy6r6C&QgAj2p!Jaz5 zJ92o2zr%@Kgk~^?+se-jk1`=8S+!=%ies~sXX_O#(3tn-&=izG2Y{smGSvuGZ)3`d zpz?hVLI5N%N;l-IG%hOC?5)BV{pKfg2@ZD8~WV8S+Y5T^+M)V0S4>=DdTKgB|76Z(f-1`&~r@?e7fi!(CCG!X-M)&ck{p)8q776TB_ zt7v~DFndSBkZ|nNDBuHlE_KxozzC60a=*0x?)UnG)UCo0ho`g1=L6W4uPBpz>-m!x zMjY=40nu}0R*yvul5b%|S)RE>G%yk|kLuJu7c?fm#jsuN`qvc216}pN;veuQ!4ObI zCtNk>vL75mNG@@Nx%L+@i~^#5#f%OsJF@VcXYXxOUfPVOo9S;jmDQUqJ2G46SiaVj zP?Q+yEWg4bl0>1W@0ngfm=PKjUJ5;vg||HMea!jUgke6RVW4MJ&ws+O^ZuDq7R?xL zwP(9G=-I|bagG_*E)HP$uLN`B916*lh~gB_Wn4a;#)(?D#%-cRPTEAO%GjptZ-6PD zDp|Al?D;#AzV+i|0J}Ro)5@No8gU#m8e=AiB1t2o2bL$*RT8sU&w~78V6~9LV~tZ& z`Z%{@K}&W0QoSn*6U7#3fMpSFjHQeCEk(hPD3scO);W%BIqvj`x8|ijjeHyn-nUwd z7h!Ql*#(Kl_FaEN!t`X1B=5;HQ9DN456el_7_*nQKgWz-N(~Q1Ge8;NLQzBo(hTHk zLi`C#(M(DWImLuwt9|(>l5Z2P>{bg@$;%>gTzp7rDit9n+oDl8TQPntY%ioJ@P?)f zE~1nMmMWx~mQbhHnf^76&{dxfA zXD_5a3OgOsdn52&b51sLHNj#c2@8(ga=<4v2?L3FBfKr^Ea;#b-q{3$sW_zp4wdUU zj>B0GexKmTsyC|2wwey9UvsSw%W?XYG3*%bR4yRORyk#pEYOlh{!M!INqYj-gsE|@#8Jucl}Jdb ze*=Dvd^(-VwX3yHRTYLfq*t0B%V{%?#|Vp_PZ{oUIz^x4Iy6qmbTR237*X$;{g!ey zBlA_|0d#kze%(z?{1|b5MOgqS5fdr63|S-`ADd4GFBw{GmCYd)lhUBKIt3dK0Ygzh zr?9)@`4hyDugZt<1VzadGvb>qHE9I8E6?eJU4A7eWNW9NMz$v=*wfp4o*jorBX2|) z7amI@DKG4YfkhJjM7WwNu52IJB;$xOEcJDKNobPcWUI50mez)n0+J5n0?#B$lo%I` z;7c56zItN(AhL>f>g2*2Q&mV@s!4>Tqv;vlVc?HIu*kbDkDmxR$s(c2|rL=ri7 zC-w~qGfpCVi5Vwcs4YT0aj|}*AkZ+GV%XFjeWxaUtCJS+v60%+`pPWZDT>e-2Ntvh z^qU4GC-@*lB-KEbR>Q%8*efnf&71=~A`uR@;rO?M)Rwj;blUO|k0`^o+e)SlF$`%E zDPEeQSM@=nG-Vo0bV}-|oDo|>k3GL3(b215;HuMdt;EQxx2(p|{*HOqE2malHe(Oz={8HH%P=-^eI>Va=; zX#4t&EMhtU9&|l#iau#`$BVVHbxA`cNQgZl!meAA;W+e$$iE2b*yH5i7r^w@2PP!yBsYl5*mYZqC?TjO|ez?|CTBs^VYGso&b2*KyJN>|xu~aOr z!4L}^&MdS5hO(CAd(jv$@+WEmM}ViXm>K?BZ zdUW_N>_-<~Q_c!Hv_GVZ&(J3aNk z{`cKK4RyX&+{`r_j5nriawt`QVL6@IK%O=xGO6ZTKQ*k?_?}{9ykrG?WGbyV8~_sA za*jty2v-vvnIQyZG{Vf;d33gd4N*(L1L&I&*0epI32@8l!W0nM6}eLvSomzBT^y2+u%uW_K1Mj`E;@od{NB7=bwsy5F}DawFGudp;pq#a^LXf(?vl`IjI5frNEW+5rj zLvb7$A5?#cIPtyo6UU;zk;oRMUTtZ3EwsRyYF;#1TALB7W+b4E-kj%!5ysZyK*90Ke_6DE1km~Z!3*sBj@oc=pYX&$b$;f zS_Lt_TjTLp^N}N^d*ScX-CL!*Pn+!4xmV>id)vZGb~6vssoh?=p-s~wR7bV&&H5dl z>{OZeyF2&R^H9?@^ZKT~2fZpzaJHqr3I|-dg+=`utMUVUMZUm17Y!hD$10p@Z|&lq z+YI1BO4NgG`CwZ<*p?5r<+o&8YExi}#RMlDUVl;t02uV)9TtrEdLLW=S#g?T+2ak^ zw5P>58V_K%_oTP$93s$W*F*Gu>*)K94LfyUvAc#X%8i1WV(P^XRv0&0 z2j6GeZa(-E>|b~nc&3gi{(@)Q)7g75!R!}2GvOPyO4fV{7jx7IS8dRwm#z+p4em1( zOD-=3ZnvU~Rz6@ZhA*P~A0t<)8zes~fGqzzxOcI6C z5;BniDO6>ZU!qWI!l|foqrt8wNId4+64nSY5sGj?8Ge5X9~>Ta7;%BZQ2omi9f$ho zW(D3RIGpzV31Wix1;eUh8S+W z6Lw{A-LSPvG%2~{JpjoWyzE8O5?e&EE;#OL9CN?&R0({W^Uz2NGK>vS8p*08H1Xl^ zc-umtl3834jv>U-02qcSx~OpPSf|e%Uy*UD9!5ksDKgx1-tx27f{!tq63#WKg9(bp zI--+aMVO^tR1<4w6lbM1$CT(;W_FA(wDcP0X8A$T`MG}M)mc6NE9&pLX$W%wiqf1HxxI#=? zXMC%p;T_tgQ(9E1>s2{6+ov(eR#jn5Rj#Oqpn`{NLKzI#v!w74T?| z=j5E{rIprXs_Br^zC@A@qBLQsDhozjY^vd)OSxMu&x71`?v_y`NKwk{~+~MI-3iAKlD4=&gQ=C9&ctuykAg(~`3#LVN1f zkf%fNwuC*m>=rS~r}ockXD<%mk|2Gq^z-L(ZA)0`85;QVMIuI>j#QSuMBS^SQ+qE%9bdlm_@&=-I+r^EEc-K} zOzlwWv$m-s$vd4l=1c#I5rLgg+n^4KrN?ViT2D{Fn^x?e0O3xSFcQtjn0q*0b}~Cv z$|a@(-Ft;KG^Cffrzcq@NLgsE+D#2#YHmEkXO^C#(nPdqTW$I+-?Uexj^g^dE!5vB z<~xO9x;&yF5YH%#vCutCnIKCd6MLk+L~!!@)#2&KxbV3Z55Wp9Xj< zH|8^H557w=phlO8H8lylJvd1sK+Z|yR18TnNI2HHW%XHojSo9)mP9umZ(*uUDge)8 zQ}8??X5bO|3rihRynK9wFEM)zf5_&WW)eIPN0+6U%KwHxK7Y12Qt-S~9h#>K)vHS- zb;$Fy-|c0$V)&(@Yq>0Y<~d#Sv|(C>K=r}WiYMt{(&_B=;J{3H#&q@69=tsqv=OEm zLYm~MO?4uB58h<{41*W9ma@$gjG!=`>14u&+O#dn!j*TnAmEnhZz^*&v$fc_{CMG> z8=iU79^IfpC8CyTuc-Fj)zHhrV$~d>qa&72RXXl`8WzH>x2}{nMRAc9($eK49UD8N zH|=s_Kv&Li?bS#Mn_yeZaGD5ibI}^E0r(U46M+k|-%leTOLlB#yEK}#U%uqT?$P^K zuRk6h7v!yq0Bt9J3Zj$ z_Ec_2*`AGZ{?g5|OSUETEwc^ctdd6tS(1Z2N_$=8=nJfhj81YBnK!iOPv+74v)2PS z6e=`ZpbHELM)(S*BuWJ4)lukrn@m!%H9vZ0gAd?$q>Ni>Q>a-~Yh?hBb1K=T-*Seo z=HvWvjwVlEpZs=sP(hZh+iW3(ysW_I&r*wEY7cIcL;?N;mVDh^z#fUll6+xjOUb34 z5>9$5yI(9(=P4!|N~{;u&DC2Loz((N zX1pijm0bnU?El|;@nm6N=gd{#SxR71RYNj#W>4o!_xWfGRH}8f0HM46%s!FWt7=jNE>=_O_d zbQ()q?EdrRZttnLvkm{f+k4^d$d!X)G$bJr1oNIGV*{o46A{HVq>u#NEE{6~a(9RY z+HFE=+EEgXdl%oU0D{ZiW|S?0rc%s61*%Fh2n(QV_vuaXWP%`C;iwU(Z7`(I2Vw1yC#WZ_r9z!}Ney_71a;kP|eI&ssH#l$-jYDO) zSYZ*Of6oBRPV4wB!qv7rip)XGY`Pr&Y%bc-KCYtfuRr$km+hFMws^+e`v1uXioZ40 zE>5nRIL~Q5CE(^p@3rj)2*#b)4tW7H^b@R4eO!*26t*TPhxy!3i<6)gW zth4*H&a#CCn`vrJ6J)Qx)tUmPRO?ljE5Fd9E7qL5=JFLDmfXXVyYEYmxXSG2~l)NMxOeBvk#v{h35Ie2jL>*vrRIZ@3xt6w{S7C>56txEjmL)jK^ zj+ir3ilx+|LY0{-x*W>22cR6<&CXISX4&QH)yl23X>*vWy5R@u^&q|Olgeo>hgQT1 zDM}0T&pIlg z7*$~6H7F~1d!18}HAZt)375lFeXrByb-k^eb}_*Wt4(f`dI`_AAbaYe{{_!Xz$tcL3c|9+?w-)X=W>RD|3Ja_5}`L(}~9_ zIU4d|URejeeA(KD*lg=lW!xbtLnL=kkMuFDFGvftJkjU$bwqvhqq#oJp}`(|7bI^vV;opF{y>DleL6TG5wcJ0*)$(PnaL_bQ6B)VxKp z(&?(-KgrKqMVG#<39QuET$L@CMbNq3uht!J!Y-XOd0(-v)}Uc|jV6OT{*pFNMzD$h zQjsDJ)wXi=in@9^3*#T%Kf2vgTh4iimT)Ren8wjq5^6MH91Ix>aWA;5h2My?Ie^Oe zyBQXnak(f)Z#H@t$F+EGGVyMR|5agr{gk`tC>yak&%|wJnCggAt*=QP(S2Z-5A3oq zw&wr#j?0PdaUC{}U z=JOvrPoBLf=6`y&^Wyo#`HyuxSxE8#E_XW@Bnk%RzK(Y&?o6>j0TO6nFV>%bBCwOD z^33sWUZWuDYpLV|;Q^16XK`y7WA)q8LaBnqT9^R(dd0%PW8osbO$gjJkybXljm zgrKYhIlvlwZVnDM51<6BVI0-{C|ovE)SAz(Od-89?QCBDL$@!c zvCi!^BvAn7h9{lut1f9%tk0n8`LSWrCXse!1x;3`ShUORqKkVwz4=8a7zH8b{N@Rd z>JQkK$MIJ}Su!mRJ)l>Sheoo`Qxb-R>l=Dnp`uhk0^66c-npQPO^y%==_O{KxfJMT zZCNO121v1$7%>!vIP~}|@^3gE3*%IQXrS2n3phL%J>${uf94Q$l#TIq>A#=h)aIV-+z{~o{* zjj*|4(A<`%5urMj%B(>hPgf-bVq+}SEs>^PD-oJ&%Clvv+pH9nrFJtQIh84YF+t0j zYQ+CtQ}x5s_Nn0ip_q6pbpy=d|1b7-_6q#}`IEgD5Bz@}&+6@ePm|Es;|7RHH2#=g zp!qSRJ_>t$QsonHa^AclVcLuh5dMggPcP4XB;e8Kd@h`iZw}wS{&ManXDViYbb;jg zvr5Sy(}?4jk2Z~$)GFZ~hace4uFg4`OY8GTL+4#LZaHxecwR(3l!%FEZf!8YoF1u~ z6Hv&SP%hFAfJ*9R0KPvtb3VlO$|IfNiHzU_bRXHVb>Tl=LihOn(NA6Y^B*84I5IV{ zKcQ|x`}z|R_To`u=h5EL`Y?_6VH)AT{;T5S+xPpg4v&7aX#=>rA}GC75edYx01}xy zC2m9wtrRRpFj?Bn2U1y{R8C(XogE&%e!K3aj~K`4RPr9Wx=4LaeG>QK^PBg-9=-Z_ zc>Ho#k49Z5m%dj?6YUq4L?lQ*o&0)q_{x6!k3Na7WzB0_$aIK|U^8vvBs(ALf%qtBq&fbo!D9p>sO!{v&;HGR;qZ7{^=%s-?bs3ORttkH_-G3f*9LQ~W&yt2KRtVWGJhQw%H4)mCMsM@c!iN z*vsyt&44g$gz-J!;s6spr=tNLUEtX^JkoKz6$yk2_yzk;Jc-L-4>PxRh}X)~=ee+ung*S4kX?(6weSghNy{V_$tpS*RZwe`f#ov@UU zPRlK-00RBntIZ333lWcX{C|bpzw`P3^B2Ya-zP5~_J7y%)T}+nPqy+M;%w=ECET-~ zH}&)g+&nXa{x6z_^D10JzDdc!Nc_)c|D^|FFvpr zU*W%+ix%$Y@N_6o6Q=Vw(`zgh?v`)rV4=S4Y98a+uhNFUE3mW(`(bx)x3tfm6s9yU>6Lsoi!=4|P+z}ee!aFou=+z6+JOBqq*p)_F{DWp z*n78>zz`Iux;x!ou@%$6LQDB(_Om%Ss_jL7)!-`Qo(lTCV$sy?Q(=bmQ!!5q&Q|L= zOS@(+LdxTuL?x8pZ{&V1U*+9!$6Qu##OUgS-572L;R=~K%MgK}Q9n>xFUG7lzjpP|^?-{#`6W<{ln-F*$(Ug1;J|6;z%lYjb| zqyO#gJSqBrpS;+8(Erx*6xLMv&JvTMwcrSFZ*FV8=cJcukEKgf?xQBSd|aW`1eMe~ zX5^BDc#L0jABFn(2#iq3CDR-6pBPPIJrb5ke)#xZ!o^7v?eoX_EsDn2;ZRc&6=bwV z=mCuc{v>(^q|0ol=u`VD6UxQm@j=!{`NF9E`IpCsud)}h9cGZdV;BYRqi}XYsdz&| z%x7HSsr&t}gfJYuVl;Lyy0L2N{@b^*l*Gla9J5toDFwO+o(h#XeoLa{Q?6U8OZ(AG z_DUThQb{aOs1VHW+X!^hwiMrwzkB@6in)iU<|)d59m-;kLvn!x9wi~iVi~2NS^htH zQS$%n$?^~Ke;to2)KMIBXLr_2{wWqWbMxy2ZeNY#W^LPmj!kD#WSrDTi=VNxB~7oL zd6G|9u;&)A<=Rg{YpX9e=$dq4G|))bHdkpIqgX0EWcxi-9#VG`$`|d$;`@1Rmx|Rk zZjkM4$v9}m{sm8_=Aczhlit+V@;&OE8I1RSL>wO@F@bLN!L06%6@LE%bl?XufyV`GrvdGWm|Mfb12oV{8bL+ zbmh*m^XCmtz`R;mauO8^WYlX7mIW=4IqmB-#N4f6UJWYnSXa9Wk@I~@DerYIE*U_# zg6&mZa3XnNonz4uAR|=u)@H@Z8ZLd(?Y02WZ6e((A|^#3Xfz@b5zegw_D!10VAG;` z3mz@BP6|y$G)UuHva;^ms!6S?yhJAi4KZ`K$3g7@q%Rw~;8tCt2i%|FAPF(c&w1{3 z<$M<_*3Na7Za1?U+xt-TqZmjvxtnWATnabkCBxDFn;aj=5`OEC#mKOmFro%lP`BatN{=DLlX6eh)Z0tqT(qqb!-Cu%!lafXR-5c_GxqOEi0`;{queN5Aa6Nr?KeU zgx}g;#lr79`Ke@|OI`Nks+v>Dym8&<~_~wjNP!K4uH8$)OBg$NKCP~T>7mN$ zmGR@bsHuHfm26I@4J#C_F9wf3%U*r>pFc}2mS)&yEn%0KNgA;gyUl&Y(WT4BwN%vf zb@KZ7?cu@x+2Q-6kMG{UdhGyG@yw1}l?hNy8QxgTuy+1iC1F4L_sL+0Lo6S|Jg!GdtcE>H^JL;CDJ(P9K=nz*a zR=EQ1+)A^6pzswA(d<;`f*VvU=zIeemGUzmV@A~WUpsiY;(18s5N2yo`SN*KJn?g2R_c|8Hlj{95g>u8 z!7plq@?_YiqA6C@BkhiEJy^$dsu`S^#$Lf#Y6dA)_`JWzbq^P#?b<|J>h9#S`azvX zCVxNFB2JmY%~Jnoy)$3E%Og$DqDV29aJIcuAp_!i7xKUsYC6|sTfaeN78;fDLSkut z0-C}UcHy7@T(Z|CQ#Us+I!VP(SO^l0fkY$9ruwQ>G^B|DKVj<98WUq_mT$hA^8 z;3t&`Bde_L%(nMZXO(h7R)5SA8@+tAnae>^xze@PqwY)0vMf6}N<5FDEAS@s{lwrQ z=$6p!#g*;agI4y1&1!8^LS;joJH=Zi>^~~wq6^o6t}fJau_{D44HR61z*r7sK^m#^ zLM*m4J+H*!bIyqe@4|z3A;nxSd0L~JA%8MsL03bWB%6E=HHSdzPnO)C=b@&FCCt1U zh&QwIYJlC?(5r#Bgr!#l@or2#C-LG^Se);FNI~AB2coWxRS@S&ybyKJmM72Ec_QjS zescuc=!S5x@ZgB}njI09=2i)&Ewr#xBCmILO6Y8V(y^T305VrZg))`9gT5-Hq{bUj zzPrBO8L=XZe*}+fblS(2W`B+SzA5eh8bDihhNCL>QA2cg6cHQqyamuMJ(Jqw-Dkxu zUJpDkOU?8$%gofwf*STn_un4wpH}LhEyIc`Tx70gv+14`F0h)d{{x0aHQT-=7Tkv6 z)>sQghEOsS(u1-v*R^hFxK^Z9hSfcaFb?={3SGyTO$p~JE^qKMM_ZT00x6H%RC(?4 zp}b{=pO$v%GFQMj9}Yr9rdzNz&qcabiVmc#b}%U{6~}t0SYwGtOEvPK6!2$&*Fi3E zc#SA8&dN0Voi&A*(Yvr4x&}l`5da^$iuJ2lxb8M|y%YK0mTu)it61d%zQ2^)xZ0qJFr2t3W@_wa74*r?f4 zlvLU8(#+J@P|Hk39?!1yZjkb&4s|n5nMpG|jm7L0VFUP_qqUqrxR^YlPGxA>W?Elp zS+453Xz}HXblMkacK*6qnEBK)8I>DA(nY68C;hBgEp^oF&(}=H%J<=0x14sq>a^JM zDy9N`nNfi~L6;uZFP`*-dZ}ML1iD~KpOy)ITIJ2KOMV*DNTp`V1>4$_xTcfJjVPIF zrM29EY^P??R4xQUH;Fz+u)E0kE5Te8QY3LfSvvFv)rkDNlrLE~UOd=m&jM+<>gZQ4 z;dR>8Bo2_kr-C7Y$1~lla;{DbEj!JhuV*_`bCk)c$7mRmX#A_HevFtPYA-uqe-r-& z-4m=c$8_6wBQ=|HtqdSa!Z3yJa^QTqj^#C`SMzW%e=LREQ{Q4@jOP3gt4RVqFaOii z=TA%bKkhz%$p5g8r$TqCl9sDa0{xX_hiG7nty4f(mY@o>A0dz(AeE#RHT4YsZZEi~XJim6G@^%XyK2`d|iOTF$?z;PR(5XMW-3GEV z7skHGmDYu>G+r{@V#!1_!0s^?`@5;lVx7;odxJGp~)T z|IV`)Mfv}1_r;S3`M-{*!hB^z`i2@RhSHK45t^XG37ko~IzF z-L=2S8XsbT+=c$5CZ2C}z0(kz$O04-lM$&FI~@Rr0k%g2Plo@(zEsgX$k|&a15p#G zx?DwN_EWw7O?{;6FUtbvt^cPlb_(nN#op7shxNaXrxYsd5>6c_D0%=aMYdY)wX}Kf z*ZqBy-K%(N)_;gD@v73l&RPFYcb@DN_5T+yc6J`t|2m$}bMK9~y$rGT?uUE6?49(B z4x^NPuiQrA>1Iqt%tTeK28Rv9aY%fWkL0&=q<|erCTg7^H%%gY2dQuE+}~RM+E&|^ z3nsa(ys@N26m#;(77N{CPSR!4+17OAA*t=EliH^AFgJs2Wyyz>wykm%JfyT;KBcXp zzZ++?t*_sh(6-cro28hosC$$At{Tiu63mus-n{+A)5})Zt3iBeFG|;5K8^5gX0dHf zVw*4hki*vgxYHcAh9i8-DQvSozS}=NdwsH8SF2z&^|(-eRk?SKU9wzGTL(moW|ryf zSCxJ@ODd~Y2OT-3&2;|!{`Bli`L%@g@#xogKfXS>M;Uz!B>1f*_H}8r3F(^EwNLEZ zT>cjG`s#+~rS&bAYRt}iFY@}jc6+h5v#!b&+Xl0FJH7T^mtN5=!DFP{O+9|;_sZU$ zS9FR<;8XxxCjCtXxMGDIw3TJPsY{qtK@zZFHt9tXUZ*n5Y4szDf{!+2 zbQK+_Soiu>^OCAgF_qt#xN(lDsuq@BRjbJ^RRwCvb+y90QdRKQlvHZDrPZ7d`_|T0 zq1p8{&tlk!6*<%dkl51Eu7GATEL{N@PsA(QedY{iF6%9-BTDPgmCnEfG<@;{yvR8S-%4KO+JMb(F)ebN=+r7rs9f(MDe^;7?TQCuCx!N*$c;|KEHq-FlWF*yDc)!tneCvK1^M9-+{qNlT zzt47di}}BH_ntlE|5(Scqf{<^|5fnwu85qIg}B-2$2xOA0){a+{pME9vc$8G+jazU_p-`JODC4>(`FRXX72 zDr?IItU1%SqkL4$e@S)err6SYzVh~`OZWB&$T=)!qZnR-M>z`o`x{wWp%w| zuc^==+$&^z5y-;gy$Qe#BzzH^E97OzbD1Nm%I0OJ)Mrf1dc5TYwl`PY&jDb2Yt^Z| zakWmNkNJMFS|x zesB||Ms3dfBLZg*YWhmM07Ub{qIMwPflS( zLJT9y;Fn~GS%d}V9sTL+mU0hI-J{q4$?N@B?_T$&!L78>xc+yaKP&tHcb`4@|JU<$ zHsC~=C7loI-#<4eLd1i<$&5v4e+vi)6afk@r3n?l1)h02FxE%@1S{Qv3(9a04uvdC zM*>H{@CY-80}vDt2Lw`30v9atMZz$cQ-GXAK9(>!_$ac0k+K|##(Wzlh)*!zhLCW< zw}E5zU&;gy%#R@vQxtR6kI5{?ETrR*j(M-s+1P-yxBqx`@{dmE{QO)+Ja51mR$7b+ zvPg6?&dwu{G7w0_1qwsp0u4i~u^{WZ0FD_A5?^gowuS+o#u?Y4w%m(TAWGTRZES47 zVIJhpHWvcLm z>Q3k(=oubjNH`9(XInLo;@BC=g?Xa}v!$ew(`h-$8oVCE2xMR2K*J}I?Je(tgS!dh zFvK_lfw|Bmc}3ZU?!(bfhe!V(97bb?Io}2!GvS#LY{Mu_a3liN?lGfNEG9VN5_f{e zvU!D%AAdPM$&WxzuUb8t3Xd69DfKv!BDLc=0v2|AyL*4{czPgvz1`iv17-TxlirRu z!~#8$-;aNTn9)ljm2J6jl34lONu>IaL;+^uOimGrMhvA3%}|vhNjrfGU7gMuO=(rW z)wohPkw1;DtW6tOw zn8e(wlM{{M!!crLisj7zxw#RiKem8ly;KzgQDCyKASOsys&*^bLtZ$?260?K~GSqDme6D2s*`~ZgfP$kY1@#$_aO{olcf~ z?V#SQ9X<6~i(-NaOJ`a$%U-8*owW#Uo@;o8xz9+f=Kj7svR8Uw3)k#igSW^73zlEc zbMsy)44jWcIz%Rj`2>#%7i@N3kL{mSad@Rfpbm2eW~wbamp6T`;XEQ?ZKs=R7BS%D z$*+c2#6UWB?dDUgoh$7u0>l_v4^pn-{KKE;uo+OyRrDqb0y8;rBK0OcBq^me3&a7$ zG~in~47eeA_1H`?V2Xn;%WJJ-CRGzZ6RMJXt>;;i28fBBpF{t)EzM#N?*;W|y*k&5 zX^+sw@Q}kmOgRyB4i54dP#VfTc&_0*cL=pT790!+G;UKbj{$+!Md!`WHJpDw+yCj~ z(f+&FUlt+)2LstblsPnTm@$}GA6&Wn=NirrM@Li~Gt6-$7QjIP1C3I^wsjg~pNwXZ zT7>sN7%a&NWuM<)V#Y{-=K{HgQ{`2Vf<=WVqcEHSHPs=%ACYY(D76?Fj|6O95lKM| z$5?BtIXh`=5hJKc4dsWjwiNXK`{53b={ zuw*_3SJZSdklhH^0O=ZzlIajLX@IE;e5tOl=(&dT-J4K=!9ad%a;`?CHvH%aY&FXQ zsk(9(Hl=!)?#rP>OmHNKulxWvSm74DP zPVMekg<%C)x`tDXt?O@wreXVkZfR|%LG zv<)LHt|+_EX3;#o=o((D0&wKkTG>!;_$@)$s2NUvUh5|~9*xV(l}GEo7o*ceXGj296bY7XeLxR+bcXn=~7*}E-O&(FL6RN~8_X`25#(=q5 zwT=c(ts5jpJ`U`at#^x%CAH%G9j`ro7`$EZCSttlNN)NpNdx($nR9l}tZkT%^Aol zYv$SR&G-J7F)(NMtOf&jynVLhu325`*cux626xTM7--l%`zCkIN{MirSHWvp3=kV9 zQx!61)!`wD5S#r2V#+y@)}x7V{V>4{SH`B&kX&N6v-7;4PwBfc3ek8Wo%tF*f4RZ% zix`M$@K&8SYQ})$B{Ro(B@W(cAQ^v%=^8!BtHZ#4b|$!18afzoj}YG}8dijXRIi=4 z3e50q`7{ApKO|GK(w&gD&o#WmQ_5xoINy8r{2e*phJ)i@2g+uiHtz(x7%&#nI~qxK zphiW!1~M`ZnQfyNb3krT^!r$vD3n=~$-sk66mn|4YrH}g)tgNk_JW4GwOa(KLiWmJ(SqiVlkzaRpoHe{+Qx53nK zz7!>yi%`GNjzUvqHJDnFG|ZG~8j-|CIUdE z{H?~oF{76xz|!_h<6@322=QB%F?iWwEi#TJX`pm*=h8fob&Gukll?I*q`%a!~IR9{hQSdt>0>6)Z+LrC0*aayD<~vH8 zJ*9Vsh9xnOW;I;oteHwqut_3Np)$oFnKioyV)QKeajPjB2A;e-L<(i}R_)7Ri3QJaBul#XAQSErga2x}s(Zz)KPsQc#a zuctr1`mwtW`p^BhZ@Y6fBNqc|+we_@=ItBiRsqTfnoG(LNrXY|{H=JHjlKHt=M8H* z7?AXOTInCox0S=d5e={_6r}(_oRhIWU9qm`8a{W2SfE}m%}*aOf6^U57fplbPrH@U z!o`4~A!f~@P)F0Hs+QEp|-SECx)kmfJDOV<2wS4l6OB6~w#6 zzzTH2;*}J2{(%Y%q_$o^#Vy?X^93h>O)=pj9AMQ7b+;{YklNuoM8axz&GHzKiZ4OoD;%QP zDfX$gm#;x}ePwXn%CfEFEQt|!S*^G=mH7lz2EpX# z)t3LwU>*Njy-g54%9b7=Pa5+k8YhET%r;G8f#^{xr8F+(hl6a@rwI6hD<%44mOOlz zNtjH>aGNeYya)hMF9SC~&-_z*M$8Y#%;pN7`tPuuQlvk|EZ6`Qkz=rsCT8LNMI36S zI3_xi0x2SVeIC~ zP)OGjzhtSoB9Kt-zj+5IfOCujCE3ZD&AWZ!o)RI^>`t^1DlyMB;{^>a4&q2=e*4a! zSpsN%qG5blfMm^@AoRWF4|Sx(2w4dtNdI;Kt1Y+?bBbY%r9F|uObyK2hP+|NKM1Y1 z^lbZG{YG_@a2RvStYH78jURf^9Ua8&%14e0cV+iWqn-y*B}dRE*58>>?9=KnU}L^d z?ZBKcDY*Yzm*oj}`f-4kO`Axyp-MB_pRORp2XnCZS=Ur%Jm7}lB-(St(Pav~k=+)D zzQa<@JJQ`6Q7V$Zpb2OI@)G4nWV6LtxWL(uk+1qGix2}$uE=wrFq_W#t-Ebdgl)FL zlhfnsrSA1Jy=(Ui5}Qp%AA2YLc}BL%4X4JS`wysIJ!BGE4?Z*EC zg+OXQ_CIRh%oiwJi1X$cS|g!f&)Kzs$)f=-+9P6`3jD84kwL3@{>Hmu-NKr)Hm85z z#{-gqTXMT$7vh+Wt32Jhnsz`MEx2!HsbNLjXfh_KH6KAkH}!eCBq_G6Xfiku_u(4d zQ?hgz*rh#xA2TNhm4*eq|J6Yu%~KdYaj@25+qzeX(rubhq_9Arpw)S0rfW>nm?ei9 z%Tc7Lq_$ma*gVi>$PbsL4+utz(pj_At%2iWPEqd8JE;j(1TJkJj9FS@HF8_1Ei<-K zN=&IVZw%SkNLrd;o2B{?MV2P5`0A76);_`R@w65o_^AT~W*JI3iv0gL)oljmtrRi) zkqVLEH@8oZw5`i8jrWgVTwNR%+1|5Y&z{{iUN0U-p`FN1K_eXtt#-_XO_#8)heE~2 zjrd*15X#p2+81{jsdgK_iP69G047ZyaG7^EY9;GX(HG4Fyw2t7H%0o@L#mDW$uJ>& z$&<*ZMx3^B-}P0ck!1HT;$a29Zqv%Zx?brFJZ;c0{Z~T%)lIQMIP@1fC44T|cK|YA z(L#L50^jc4%hdD`dU}ozT0AW14$#n!0Y;99ZQ3U8Ukr=uAKVa8*Mtv~b?MXd1#vL! z7t`SOG2PASZGv&j$LdY5F_dRmzYbsLYb_?uNJ94xIt0A~dhxTLu^COuRrR+q7#mE5 zv=PC~pXCqnDwtxYu*F)>Im&_l9NYW1zzL+rrss*&XnID&JnGI)U8|?|x>^fdGkC_D zPZ^D{rMH?fRwEo^E}FKFgqsFCbVkC`;z)|A97%{G7#qB)ATaaB?pTN;7C5$iLjrvk zNki7yCnybP>8VDJ=H++N)2%Q8miijV6AdT+Gd*=Mo|Ig@noDq+Yedx4wdtfSqn#Mk0d0og|$e*wFLT~ znAKy}8Ceo^oG)1a{H$ojUuP6aVz2t{|Jlr@iB8>+VCL*jMC*>y{k=@E>R7<1c&K%! zeY-}TW(|O(0v{M`+}D2RzuGHU#MIhS|C2nc_J`SZ3q8V05M~W|wVMHr);dkvH&J`t z#=r})uS4~y=*QFM-Xs(yRsBAF+be$}u^dHxa6OpO}ZD$K2C zM#j(}s!;rw=H&Gj^e+D`_F=P2&x&2eAddw#@z3J+$}Rnr@8`1K&PjBLdBA}UVJi8J&nQ!mP)5m=Ut6C3AWvC*oFCN6msd^Fa;Iq zSqTGai0~|Zrau=fj-aV3fr(*MZtS$o$X(p59rHPGq4Q*G8RY$tnS@82Hf%G@U~)us zhotp%SsgXp1EyIr_(OxOX``KlCpcY1w#a^W>{izFRwEx_EQqOmJ3{CSSy6`c+}R;N z9-0T%{hdbW74}XUdC(6kEW3@v`lGNA{}h-*@28Dj&ZFt*T#WI6wqZq5q_y5WIm~!p>iaSCnkgBAx5U zS-H6=sJTvr{@RFq@=|fflR&P)FyP+;h`iM()I;?oD8{C4bKi0Rn2zLvY^E^4qr{P; z&oAVTMOlRlQzyEAGimd6W;S; z1NI?4xEJ?y+UK~NJRR3nsVR1i3N8uD7acrKP2o@5lal}1t;jS@%nw`(gxI}s zpi~O%Br!Bhl^G&_NcD(+*h1e$lnJ_p90*mn zdZ$XTk?P!hH>p4YXDJF&5i`Hz$&N+>DN_zAGQj*Xek1Aq5+65_k9V~#noA$FDYtFM z^Mfq{3iq~B9sJ&XCQtO9o>%2}?kuhRtPcPx&E7x1mo*fFA8x==+QykD^vj+hdX)IC z!Z<>kjy>5@7}rNGO%VhU)9RowZPriJYcFXUK@Wv~ zI_b-k%9AIm5Kj6ncw5#KX0Ug^Ji!8g`{2qbj}lEtP@J^VOhlx3y z(0v9Z24uK2^-Y!|rIArY`hp>fsVr)Ey~Aqn7#9iYP65>vSPxQ?KjTt#tZ=JrKP>A% zi699G+wwt?IXNS{h*E8^>SOqDocv_*-zfaq*0*(PTp2~GwVn?%);|MPSqHepMIBUl zrkJ_MVPk@F+FFb46(F*P!%*c&J36rmDTxi65!%=F{C}WUdy-JAi;+e`(VsvX z$RgW|CEX-9+9TCNaSTDE)g$wioWzCW|Hq*ZPfN$4Us%3ipAdIGXSxuyH{g^-@ z2^xUjI{;qU4QwPyREx&ws(C6w;fIC6cF?ewGUHM%%KF<1 zfeSY_$sFiPB@CSO2no>X|2=Q8KEz2z>dh*4UAhS`SxN6qAf9n3xo06e%*N!#Mu%$$ zN#kAud@Uv3CpZviXfy8rI-kNKEuU5B{zKmJ5Sfhlt8adVWHS+a?OGfC&ZNMNSb#9Q zR@sRiJA5X!iZXUQU@(7h_#RcS3t0)*yQZ;P@=BvXWYm-e)XcFu-dgyyW=A64DmfSn z06T6~pQ8!%Cj=cP;sKTU9Jmp3{psOjrBnuxa44L39^QShRLVr8LKL7`bNe^Y;<-({Bh5X!zjFFIvY z9IU07i1vb#+etL+pU!Mrg!~f3w1P*|njSnqHJ&jQ0IHG?J}}knheY+y76w@~h9s!i z{;x20)rUMyXuR2PRB`Mi6R}3;xkYU*c?-QsPFd>!bBDt5#A5VZ@vM;6L_065b6xVj zY%E*JuzQ)J(|EH=S8Kfk-hFCWt%otk*9K(;G`)kSAlK)L=3nf&#dY7QP^Z!N?FwqJ zxyoYli;qN#JB=QMAF3seQZU!eiXotz8#@1K#pt`pmm4`w+?#?5aHY?Iz0hBEyQ z7w{5KY&UHd`fo)`#iBiG-`ZwInJf$-~Gt&*A6#q|=a*nZdBxBDWC@SIt$q^2a^ zBv4^{|4GqanV9U{u)?*u-?B^Ek&f^ou!zUw_#tY4Y=dR|_|Pt_^Cxt z4B58vj{)Y8Vyr&My^UU0QBcDzH9T)Y5rJZ_*$#6H=U zJo9IYO|x+mHnWhIF+!Xq!};Xn;4!-QjX8R{XE}PyVt9`#P6!Dh=}<1km`3#m^8x>u z>=*bfo08_p#<4^V&G>&JZz~US!U8)Hm!eJdrX5+|h9~N)VwLcqtJGqZ-Dbf@X}7_u zwYj~`TmK5S)$tvnNX;Lj-A>$2eOuC#@H~#fyk{1F4<6saWIY*K^I9MPyOhQy;Ll}Q zlc{@fa5XhG9YG&_4-)=NZVmJmJ}JLAg*%lm@rKF#M$SRb@?s*Gp^VN41mGaoojvHJ3^#d|x^pspYOB~Iz_P=8J}jBVtL~)b>7ES z!0w7@f|>yk=+d3`*p*~cJA+XrQ{#n2P5%O4^y$`)f$6r!&u1;mgMm}7Mj82STN(g5K6}^_jCAIXGW<2Iw`+>D8?NHrRKX_Z8pztTSHjA#(yK@K;gYu_OOsWO zhvz;bD8YhnRX=@%;=)U}px{YPHe6;UxN>6TbegZ>EC}Zu1OlwBU85fHr)hO@4#%CV&sYk^ zd9%CL&iMmB5T>;DUbYBD)ddlSJt)_XDP~SY=Z#UCOG2u%K6WC%_my{!=O&~;M% z{`5zZZ{qZs**%5Ve8l`X$(Ln05_=M#7OsGBwcamBJ zD7mN?Wg0Z`r4KCQ#lFuj&&0~O*6b;JA`yxXFI^_qy%(ZJ8=oZWnVrj%E(eN1dfHF%(? zAM+$OAK^u91jB~5kHZc~+wM@16xC>ou+0Ec0l;~NP~w5kC9+{gJM-~_hU?)$px-@USM zfuto5TkshGwUxIRM}dZ`Di49omGi^A|M|M;lKR{@2$aGxN&>40ctR6oGE9@7BR~qN z0;k1*0rd$!6qmS_Bn5!EyPrgiwe{ck#Oi%8^u7HNR$jKBmsw8w>*^BHTqGVF^pG-J z0gJtjr47+=jDN(InS(_sYjJ+uj546?VeS;Ni40~hHyqF5n_(t4Qy=61K^qUT%rI5m z7Cwrt=8&*Z&n{Z%%7xA#LpAY;`TnwZzq<5NQuD!$QnF&Zm!ek6Yw|Z?Ht4X~fWWt6 zFBgCn;#4msRb{p7bFiYWp#e(;TWC_k%yHi5W27vttsbH48v>M=w}~KaqUY|5g0*bI z?b1{DW8rhE{czN#b3(CCVEetj0m%r`4L1_hd|b(vP!}U^=1AJLy+9R%BlZp{yr(aD zBB|=-TPzz#JGh{h$EJ*NM@=$rch-l;O*8gEu3cAN!mL?uT|b{`6rBzBUerj^1XZP^ zAQpoq{iu7>qihbubCM%dh_-5S_YNMIeYR;J%aqj+XqO_S(Jw6*i7II97L8A&HdM9_ zuprfD$@(M!OF$iNxrxTGs?q5vvR%-n!gPG@!{b$|uU0B=g3ctCSzR;rgxF}pkgh0628A14D92l*W8~T6;+ktIkPd)rn}ud*^wnuALsTCh zv9&D$pFFHS6L4J<)CyVHIC!sTT9~N}NjQ!Y=Vu$OOK>ia(fraw(x1BpX63+9Mg~AL z$#yi&UHwCN254|JPR}mkMLA?5L;d6%>)kF@Sk&{RlFlBlmR^%nZ$N^+)G#rI$o|r? z{Rq_CKgf>4AmTV^ySLRVol>F>49sb-{k4p(pw^B(XY-HZZw`01`>Ni3Ce)lZ%v{#0v_HpO_|Pbfwurr`7wFiuGabENVpMGc&ivr1 zV?C*`m`SoNFZ=x@M$g&8bueVzj^sYjQQceIe)1)?V-A0V_p1zi|!}QKOX9 z=-TS+tEZ#UC_E#%kTLW99Kcn6b((A)jsca@(n!%;Dp@RbJ*LkU9;fJeMk&(iXU@*s zG#bVNg{d-5m>p-)oGF=>5BHKr@(gE2fT3opkAY)@gW+@qGt65b# zJ6p;0VyfVox};Dm}<~oweU1cPktY3YfRTP5psNGs3j87EgsoD z-s`c#7MqmbtaUkPKJDckE}xqopJ(y=ItWDl-qG+v_oCgk>HT~+wz2tB^_=rN99Gzy zwW+Bl*HE7~`3!XD(9v8py&5u^bFnH^yrrI-9V}fq7a|$}S_ekrrjuMX!rq1IxI&*| z^K7^BV`5d~#kb@4v+vE>)1QB0Y$Ya|gq|tZXkq1JC*+iB%tyie{wOe4nI!KNhG>&{m13 zA`tj?^+ppg=|t*+wFBOIfoeB}rvkiHDO#KrI}lT+A7hbNGmrno`A#PuqMUx_;SNK; zqR$b-ZzoIV8R`gv^s3-@RS4Uy!)OeJXkAF^$5-XNI^A~K zh>Q7j-Cd}ENQcL5Ic?6)1beva+|A~x^`8#7WX|V7%STq&T!GWjg5A?##jG!ns2Mwz zK@cKKiC%QHuLwEEHRUO1UHo_zZ$#}i9N1Ps2y8+mCH=o-I1jS$c)4;fmASVS_n(+c z_ZfHHuLZhu+l(jit|pKrwEK+C|8flDZ{`OB-|y${)WNs$_`;Jn8D=Rn7Zgn=fnAvv z5z-BF2vSbJJb#0O4g{@Hzu*2}uHnSP_HP_85aY8pUt|^z)u;7lf!UXP%yEhwxJ3Ab zmJ+AC*1g2?fkcEIWBr6!;{2S`8v3{!n_)N)JEGqX7cb9h>pPht>@w%*yJqC#mcJUf zo>oc%`2&&OXpz!quf4YEkn|ab9Uo{+8aSvpbVKxH1FbT<7$x#y4u_(2NIn67Q*YiC zq_A}iV*N>NRLy$uOcKaRLNC=VkKg0i|0EOcfS+BQok<(Y=X<}?j!bqQa~*nGzt=UZ zO`s7hjma!+M#ptbI|DA-&5IZUvu~I~rTs}y`jIO6>L{L7jn-EJP@N0@D z^)4M(QOJCbk9GGbEwfN5GOp&&6^a`;kfu@!y-<3krp@u4pU}q`C zYbb4u9&%5{F(ByiFz8qqtW%qAP#bHS1Lv@-&b+D4bgWrD)IV;O;j<#df{?t#n~EtM zS5@c^bOf8>0eiL>|AcwKXc*1$dmSC=%IpHS!RD*{l&V{?UXTQDzDtclc-Lz%?gieh z#eI*dnxSoP(akmCbjzrqC-bB(WIL>gBLsP_CZ%%h9-BPl$r<09Ot2*BB)kWlo4V;l z*m7ET47b$dH>8Qbu;lKncGDMkEw(~YQ3NjT-_|9zEHtd+-O*uG9pk6g$CQmNrjKB% zdU^3gqcsJ{O0x~U{Gm_+*=E-FfH(RJ_;s1iY{uVJeWU#BY+BD8J>emsh?@##6u?>F4sA(rHz67mF zM}rsr3=Lar101xntuzFFId%dUvMIm!i=X0`B;Se~wj_C+f$ln{ahQCSq&of}Q+Y1s z#PLA53R8RM3f50XTq~hZM%>YEn-ZL^eg*LGV1B(Ku_*ZPaFe_z|BzbEGZM%&1i= zBS=D72hvfxO4sk!@2toj85so_9TYv(AO~p;AE3pvk0P!8ECa&mNICDK<(MOIa!r2* zL5%|<(^2s5OfCNeV!Y9XyjxoF+r65Qv>vOK+TVtVQLE*aC)XVWCyL%-2FsgEY$G%F zzQ0!N((%DGZADs??-#l6X2{^Q=W%MD(@Ru={~gH=Ntj9B5u-7+vcsiRu67Gp%YI72 zicD9~G%PS|@F;;l`RTbBQDX+&L|o~$XUCwZUTR=ALKU%+t;7^7Sv_C=7SbHAqgJr( zjBlJMOe5LCX)pVbvMZOmAcTMQ^%0lT(A*D6T7?FUp>X_*%N_kua)rR~_w6nXqHrxKt9m`Xd)ma*MG^ufMc5 zVDKJm$jGYTIPL2zCZVBOPtQvP$eo1WpN~G30T@N0UJobA&V9-;YC(quH43FB!}`bi zh>xlKq_UL%8O`Zb>LG{yDw78mzBA1cgCShJi?J zX$_`WO_sR9w*Rn_Csn1PjV5YqNf%(<1_)k(kk4eHInW-ga;P(?nk?}-I!)fXO(tgQ z$nCJa#QqHY(+~;bDzAw#=Qf^f14WH|SJIoAA&lFfJEt+vRyfsP@+iY?IpJ&y(8%N< zFNAsNkDN!_X@rW4l-`@rV#)((9igKQ?w|ZtPP-yTx3R94JaX2`t=Qv}X(S0%J>rRx zqU#xUp^30|K0jr50DQH$UJYd|@a=p9eY~6ugvu=|%Gx47Pv{hL#NoHsEWvQH`el=p zBR#CjuH8ptMt}!6Ai|Y-)8(T;dqxP2i3d#@dn_mjYg=?mm}KR!1<3yF8#khv6GH}x zbxFmF8Iv#kXiWBPhg#WsgKiMhGrrXstS)4+lj}(Q5T&;iVGK8SyNSn7i7hm?GPg_v zHG;p+CQlM{%e%r2lB9wl*&^u4Md*p@V`Kg0^5`FZu$DuL@7BL-KGwp&t@a@d0Tr*8 z2c2FF@@;X_Ef7Jz1(@A1>TqOQSDGH~sQ&M0%}Ke9C*^^%FQY7P@SiUBTr5@0)gCLP zv&~boEsl%4Vfsg&z=P23OX|_0hG-cW3nQ~(0}7~>V@p_h?U_b;QJNOfPL9ekIEmfl zG{;>5bF)XVNdWckLioG1J{EUJW<LzNnDr+Iosk#dNx8SSwyd+;k7UEWgoSAKuW!~@;LI&(w>^GpJ<9D zLzyW*$n(R%D9<-xE%;mZH7?WOEq5)Ltuk4gR#?>2gWTX(e=`{txMP;AXB@|Zo&M|| zYLsGjdR6@Tcjet=1$LK+IjbZOxEBr}RQj4H(7M>gYGwnY!$Zxt_7qysz4GZYB4Yqv+6WyW*c0D zcJaAm1z+kLhaEoubQw!M(VRs26P_rYARLTkL~d@gW@q3ZEX$~xvJCDw4M+UYS<`KG z6xZ!U64hK4Z>4*w1pyV+O=uk3!|t`@4%=9rcV~WjW1j`66+|rKX8pWtl(q3W;e{2O z@~euWqPz7JwDs!sOhfk87wv*7dLWdk{db&tq02^B!a+8?=%9Rm&PeECwFs8!(^OhV!k-EW{Bgr=tPju-KdlcQM6Aq&$&#bwVDNF4! zmqT-dO3T__*Hu;Cv3_t+@G*^c%4y$i%vg-#3C|TLpv-=X3m{EHqxuYF%myfn?*}7x zA=<^&;LA6V2T;XJkFFfhwg#iqNvF%dDq?EoG#+f%o4^Mh4x8%x4+Fo{wSzn4OkIV` zHO-++jPSMD{|?ua)Jir}+*`SD#1?m?T)u1JTk_~puIksQ?4YjEOEwDv2Byt&K18bQ zcnrQHlGU#>3w`Jte76Egufc~qCj;$} zG7dK(`Q#J>T#SflmyT(dllw3DAszJ2+jUqHjf*4oz1>QRJ_H=3Iacpp+Lq7sdRQQU-Cw7=Xw$i9|7>#j z1onY4U3!GRu0VSd^f4Hsh%s+Bwa;wzn0S33q>pU3@{ZG}##GpV<)rt(y67#O(eC;6 zYaKKdy+-J3R#noDl6GEZwL;1(JQkVhW6X;9zTd(k10qmgDW;EIx>z*wY`*0WV70%E z0&X_p5p*&fNpFXT#Lv?|9ga-nRSNhO{zHTvre^CZ~iQXZuzlD9& zKE8$$RYZe*h4do=g9Imn1-lj@;|vnw{(t8}V6a~Vzk&t)SC#)236%e1!cqUxdR|`s zn*Z@dP(cmbI9TzY;#2&&H~!~xLq=Kq|1|UO$3Ur?`)_-L-C_{HP&Z;Q*n(>>MlxYf zWqc0b&(?m_!2EMFb);aBs6=64LXfZ|g0Em>b(_&(+ka8ai1t8khkrw1{-TtW90o2S zy$=lS?|-0uY;z-1p59nhwrcH2dSc7}WZ-lQpdW)wkp!#X#wc9qK7s5T@);YTxGLBF zl^>&jpzS*JXfxf#d$Q}B|JMt5#;ARC=wh!|WcTB&a?H1_lZ&_H`U8KYxq#zvm*t4o z<|~RvNtZJ2pr`BB-;rO0p3w8irXQihA48y_)7#`H->Zz_4@dk#Uk}`OG`fq0#Z{9D z#Vbzm{AL(YBnICTHm_*WFdiaOY<%K0bZsf<$owlpxxu^{WAgY60)MiMYeX8;Fe##j zt)f`_(NkZ&ObH~s;A7M8I$yt!knhndLzi;zRea%;+%3O55+CMHQQY(W4}Ivx8pP4$!c ztZ#@zuw3ZR#X|I!hi|GkHGmre8J#BAjutOcXX6*wV(!#m0tCf-K$&Pw66?DyKXNq6 zoCKx_(xmJ+Afsj5lbCZN3R>=W^iWd~RJLtA+o8_v_*L z7`N@9=n9Ym07@~^POz40-X{j7Fvi@vY~VpsLz+1&p7GmnG#_+UnMaC1b+IFYJU4P$ zZ3n$cFYxR)!(PPKS``V25=;jI?Ee7w CvqJg+ diff --git a/infra/charts/feast/charts/feast-serving/requirements.yaml b/infra/charts/feast/charts/feast-serving/requirements.yaml deleted file mode 100644 index 2cee3f81494..00000000000 --- a/infra/charts/feast/charts/feast-serving/requirements.yaml +++ /dev/null @@ -1,8 +0,0 @@ -dependencies: -- name: redis - version: 9.5.0 - repository: "@stable" - condition: redis.enabled -- name: common - version: 0.0.5 - repository: "@incubator" diff --git a/infra/charts/feast/charts/feast-serving/templates/_helpers.tpl b/infra/charts/feast/charts/feast-serving/templates/_helpers.tpl index ab670cc8cc7..49abb6b8e50 100644 --- a/infra/charts/feast/charts/feast-serving/templates/_helpers.tpl +++ b/infra/charts/feast/charts/feast-serving/templates/_helpers.tpl @@ -43,10 +43,3 @@ app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} {{- end }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end -}} - -{{/* -Helpers -*/}} -{{- define "bq_store_and_no_job_options" -}} -{{ and (eq (index .Values "store.yaml" "type") "BIGQUERY") (empty (index .Values "application.yaml" "feast" "jobs" "store-options")) }} -{{- end -}} diff --git a/infra/charts/feast/charts/feast-serving/templates/configmap.yaml b/infra/charts/feast/charts/feast-serving/templates/configmap.yaml index 934216a9d5f..7c895ce530b 100644 --- a/infra/charts/feast/charts/feast-serving/templates/configmap.yaml +++ b/infra/charts/feast/charts/feast-serving/templates/configmap.yaml @@ -10,44 +10,28 @@ metadata: release: {{ .Release.Name }} heritage: {{ .Release.Service }} data: - application.yaml: | -{{- toYaml (index .Values "application.yaml") | nindent 4 }} - -{{- if .Values.core.enabled }} - application-bundled-core.yaml: | - feast: - core-host: {{ printf "%s-feast-core" .Release.Name }} -{{- end }} - -{{- if eq (include "bq_store_and_no_job_options" .) "true" }} - application-bundled-redis.yaml: | + application-generated.yaml: | +{{- if index .Values "application-generated.yaml" "enabled" }} feast: - jobs: - store-options: - host: {{ printf "%s-redis-headless" .Release.Name }} + core-host: {{ .Release.Name }}-feast-core + + stores: + - name: online + type: REDIS + config: + host: {{ .Release.Name }}-redis-master port: 6379 + subscriptions: + - name: "*" + project: "*" + version: "*" + + job_store: + redis_host: {{ .Release.Name }}-redis-master + redis_port: 6379 {{- end }} - store.yaml: | -{{- $store := index .Values "store.yaml"}} - -{{- if and .Values.redis.enabled (eq $store.type "REDIS") }} - -{{- if eq .Values.redis.master.service.type "ClusterIP" }} -{{- $newConfig := dict "redis_config" (dict "host" (printf "%s-redis-headless" .Release.Name) "port" .Values.redis.redisPort) }} -{{- $config := mergeOverwrite $store $newConfig }} -{{- end }} - -{{- if and (eq .Values.redis.master.service.type "LoadBalancer") (not (empty .Values.redis.master.service.loadBalancerIP)) }} -{{- $newConfig := dict "redis_config" (dict "host" .Values.redis.master.service.loadBalancerIP "port" .Values.redis.redisPort) }} -{{- $config := mergeOverwrite $store $newConfig }} -{{- end }} - -{{- end }} - -{{- toYaml $store | nindent 4 }} - -{{- range $name, $content := .Values.springConfigProfiles }} - application-{{ $name }}.yaml: | -{{- toYaml $content | nindent 4 }} + application-override.yaml: | +{{- if index .Values "application-override.yaml" "enabled" }} +{{- toYaml (index .Values "application-override.yaml") | nindent 4 }} {{- end }} diff --git a/infra/charts/feast/charts/feast-serving/templates/deployment.yaml b/infra/charts/feast/charts/feast-serving/templates/deployment.yaml index 64dd3955d0c..bb8fdc55ae5 100644 --- a/infra/charts/feast/charts/feast-serving/templates/deployment.yaml +++ b/infra/charts/feast/charts/feast-serving/templates/deployment.yaml @@ -19,10 +19,11 @@ spec: template: metadata: annotations: + checksum/configmap: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }} + checksum/secret: {{ include (print $.Template.BasePath "/secret.yaml") . | sha256sum }} {{- if .Values.prometheus.enabled }} - {{ $config := index .Values "application.yaml" }} prometheus.io/path: /metrics - prometheus.io/port: "{{ $config.server.port }}" + prometheus.io/port: "{{ .Values.service.http.targetPort }}" prometheus.io/scrape: "true" {{- end }} labels: @@ -39,23 +40,29 @@ spec: - name: {{ template "feast-serving.fullname" . }}-config configMap: name: {{ template "feast-serving.fullname" . }} - {{- if .Values.gcpServiceAccount.useExistingSecret }} - - name: {{ template "feast-serving.fullname" . }}-gcpserviceaccount + - name: {{ template "feast-serving.fullname" . }}-secret + secret: + secretName: {{ template "feast-serving.fullname" . }} + {{- if .Values.gcpServiceAccount.enabled }} + - name: {{ template "feast-serving.fullname" . }}-gcp-service-account secret: secretName: {{ .Values.gcpServiceAccount.existingSecret.name }} {{- end }} containers: - name: {{ .Chart.Name }} - image: '{{ .Values.image.repository }}:{{ required "No .image.tag found. This must be provided as input." .Values.image.tag }}' + image: {{ .Values.image.repository }}:{{ .Values.image.tag }} imagePullPolicy: {{ .Values.image.pullPolicy }} volumeMounts: - name: {{ template "feast-serving.fullname" . }}-config - mountPath: "{{ .Values.springConfigMountPath }}" - {{- if .Values.gcpServiceAccount.useExistingSecret }} - - name: {{ template "feast-serving.fullname" . }}-gcpserviceaccount - mountPath: {{ .Values.gcpServiceAccount.mountPath }} + mountPath: /etc/feast + - name: {{ template "feast-serving.fullname" . }}-secret + mountPath: /etc/secrets/feast + readOnly: true + {{- if .Values.gcpServiceAccount.enabled }} + - name: {{ template "feast-serving.fullname" . }}-gcp-service-account + mountPath: /etc/secrets/google readOnly: true {{- end }} @@ -65,30 +72,43 @@ spec: - name: LOG_LEVEL value: {{ .Values.logLevel | quote }} - {{- if .Values.gcpServiceAccount.useExistingSecret }} + {{- if .Values.gcpServiceAccount.enabled }} - name: GOOGLE_APPLICATION_CREDENTIALS - value: {{ .Values.gcpServiceAccount.mountPath }}/{{ .Values.gcpServiceAccount.existingSecret.key }} + value: /etc/secrets/google/{{ .Values.gcpServiceAccount.existingSecret.key }} {{- end }} + {{- if .Values.gcpProjectId }} - name: GOOGLE_CLOUD_PROJECT value: {{ .Values.gcpProjectId | quote }} {{- end }} + {{- if .Values.javaOpts }} + - name: JAVA_TOOL_OPTIONS + value: {{ .Values.javaOpts }} + {{- end }} + + {{- range $key, $value := .Values.envOverrides }} + - name: {{ printf "%s" $key | replace "." "_" | upper | quote }} + value: {{ $value | quote }} + {{- end }} + command: - java - {{- range .Values.jvmOptions }} - - {{ . | quote }} - {{- end }} - -jar - - {{ .Values.jarPath | quote }} - - "--spring.config.location=file:{{ .Values.springConfigMountPath }}/" - {{- $profilesArray := splitList "," .Values.springConfigProfilesActive -}} - {{- $profilesArray = append $profilesArray (.Values.core.enabled | ternary "bundled-core" "") -}} - {{- $profilesArray = append $profilesArray (eq (include "bq_store_and_no_job_options" .) "true" | ternary "bundled-redis" "") -}} - {{- $profilesArray = compact $profilesArray -}} - {{- if $profilesArray }} - - "--spring.profiles.active={{ join "," $profilesArray }}" - {{- end }} + - /opt/feast/feast-serving.jar + - --spring.config.location= + {{- if index .Values "application.yaml" "enabled" -}} + classpath:/application.yml + {{- end }} + {{- if index .Values "application-generated.yaml" "enabled" -}} + ,file:/etc/feast/application-generated.yaml + {{- end }} + {{- if index .Values "application-secret.yaml" "enabled" -}} + ,file:/etc/secrets/feast/application-secret.yaml + {{- end }} + {{- if index .Values "application-override.yaml" "enabled" -}} + ,file:/etc/feast/application-override.yaml + {{- end }} ports: - name: http diff --git a/infra/charts/feast/charts/feast-serving/templates/secret.yaml b/infra/charts/feast/charts/feast-serving/templates/secret.yaml new file mode 100644 index 00000000000..2ccbccfcf7b --- /dev/null +++ b/infra/charts/feast/charts/feast-serving/templates/secret.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: Secret +metadata: + name: {{ template "feast-serving.fullname" . }} + namespace: {{ .Release.Namespace }} + labels: + app: {{ template "feast-serving.name" . }} + component: serving + chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }} + release: {{ .Release.Name }} + heritage: {{ .Release.Service }} +type: Opaque +stringData: + application-secret.yaml: | +{{- toYaml (index .Values "application-secret.yaml") | nindent 4 }} diff --git a/infra/charts/feast/charts/feast-serving/values.yaml b/infra/charts/feast/charts/feast-serving/values.yaml index 52d10cd7440..bf7b2c772a6 100644 --- a/infra/charts/feast/charts/feast-serving/values.yaml +++ b/infra/charts/feast/charts/feast-serving/values.yaml @@ -1,234 +1,148 @@ -# redis configures Redis that is installed as part of Feast Serving. -# Refer to https://github.com/helm/charts/tree/99430c4afdc88213c1ca08f40eeb03868ffcc9d7/stable/redis -# for additional configuration -redis: - # enabled specifies whether Redis should be installed as part of Feast Serving. - # - # If enabled, "redis_config" in store.yaml will be overwritten by Helm - # to the configuration in this Redis installation. - enabled: false - # usePassword specifies if password is required to access Redis. Note that - # Feast 0.3 does not support Redis with password. - usePassword: false - # cluster configuration for Redis. - cluster: - # enabled specifies if Redis should be installed in cluster mode. - enabled: false - -# core configures Feast Core in the same parent feast chart that this Feast -# Serving connects to. -core: - # enabled specifies that Feast Serving will use Feast Core installed - # in the same parent feast chart. If enabled, Helm will overwrite - # "feast.core-host" in application.yaml with the correct value. - enabled: true - -# replicaCount is the number of pods that will be created. +# replicaCount -- Number of pods that will be created replicaCount: 1 -# image configures the Docker image for Feast Serving image: + # image.repository -- Docker image repository repository: gcr.io/kf-feast/feast-serving + # image.tag -- Image tag + tag: dev + # image.pullPolicy -- Image pull policy pullPolicy: IfNotPresent -# application.yaml is the main configuration for Feast Serving application. -# -# Feast Core is a Spring Boot app which uses this yaml configuration file. -# Refer to https://github.com/gojek/feast/blob/79eb4ab5fa3d37102c1dca9968162a98690526ba/serving/src/main/resources/application.yml -# for a complete list and description of the configuration. -# -# Note that some properties defined in application.yaml may be overridden by -# Helm under certain conditions. For example, if core is enabled, then -# "feast.core-host" will be overridden. Also, if "type: BIGQUERY" is specified -# in store.yaml, "feast.jobs.store-options" will be overridden as well with -# the default option supported in Feast 0.3. application.yaml: - feast: - version: 0.3 - core-host: localhost - core-grpc-port: 6565 - tracing: - enabled: false - tracer-name: jaeger - service-name: feast-serving - store: - config-path: /etc/feast/feast-serving/store.yaml - redis-pool-max-size: 128 - redis-pool-max-idle: 64 - jobs: - staging-location: "" - store-type: "" - store-options: {} - grpc: - port: 6566 - enable-reflection: true - server: - port: 8080 - -# store.yaml is the configuration for Feast Store. -# -# Refer to this link for description: -# https://github.com/gojek/feast/blob/79eb4ab5fa3d37102c1dca9968162a98690526ba/protos/feast/core/Store.proto -# -# Use the correct store configuration depending on whether the installed -# Feast Serving is "online" or "batch", by uncommenting the correct store.yaml. -# -# Note that if "redis.enabled: true" and "type: REDIS" in store.yaml, -# Helm will override "redis_config" with configuration of Redis installed -# in this chart. -# -# Note that if "type: BIGQUERY" in store.yaml, Helm assumes Feast Online serving -# is also installed with Redis store. Helm will then override "feast.jobs.store-options" -# in application.yaml with the installed Redis store configuration. This is -# because in Feast 0.3, Redis job store is required. -# -# store.yaml: -# name: online -# type: REDIS -# redis_config: -# host: localhost -# port: 6379 -# subscriptions: -# - project: "*" -# name: "*" -# version: "*" -# -# store.yaml: -# name: bigquery -# type: BIGQUERY -# bigquery_config: -# project_id: PROJECT_ID -# dataset_id: DATASET_ID -# subscriptions: -# - project: "*" -# name: "*" -# version: "*" - -springConfigProfiles: {} -# db: | -# spring: -# datasource: -# driverClassName: org.postgresql.Driver -# url: jdbc:postgresql://${DB_HOST:127.0.0.1}:${DB_PORT:5432}/${DB_DATABASE:postgres} -springConfigProfilesActive: "" -# springConfigMountPath is the directory path where application.yaml and -# store.yaml will be mounted in the container. -springConfigMountPath: /etc/feast/feast-serving - -# gcpServiceAccount is the service account that Feast Serving will use. + # "application.yaml".enabled -- Flag to include the default [configuration](https://github.com/gojek/feast/blob/master/serving/src/main/resources/application.yml). Please set `application-override.yaml` to override this configuration. + enabled: true + +application-generated.yaml: + # "application-generated.yaml".enabled -- Flag to include Helm generated configuration for Feast Core host, Redis store and job store. This is useful for deployment that uses default configuration for Redis. Please set `application-override.yaml` to override this configuration. + enabled: true + +# "application-secret.yaml" -- Configuration to override the default [application.yaml](https://github.com/gojek/feast/blob/master/serving/src/main/resources/application.yml). Will be created as a Secret. `application-override.yaml` has a higher precedence than `application-secret.yaml`. It is recommended to either set `application-override.yaml` or `application-secret.yaml` only to simplify config management. +application-secret.yaml: + enabled: true + +# "application-override.yaml" -- Configuration to override the default [application.yaml](https://github.com/gojek/feast/blob/master/serving/src/main/resources/application.yml). Will be created as a ConfigMap. `application-override.yaml` has a higher precedence than `application-secret.yaml` +application-override.yaml: + enabled: true + gcpServiceAccount: - # useExistingSecret specifies Feast to use an existing secret containing Google + # gcpServiceAccount.enabled -- Flag to use [service account](https://cloud.google.com/iam/docs/creating-managing-service-account-keys) JSON key # Cloud service account JSON key file. - useExistingSecret: false + enabled: false existingSecret: - # name is the secret name of the existing secret for the service account. + # gcpServiceAccount.existingSecret.name -- Name of the existing secret containing the service account name: feast-gcp-service-account - # key is the secret key of the existing secret for the service account. - # key is normally derived from the file name of the JSON key file. - key: key.json - # mountPath is the directory path where the JSON key file will be mounted. - # the value of "existingSecret.key" is file name of the service account file. - mountPath: /etc/gcloud/service-accounts - -# Project ID picked up by the Cloud SDK (e.g. BigQuery run against this project) + # gcpServiceAccount.existingSecret.key -- Key in the secret data (file name of the service account) + key: credentials.json + +# gcpProjectId -- Project ID to use when using Google Cloud services such as BigQuery, Cloud Storage and Dataflow gcpProjectId: "" -# Path to Jar file in the Docker image. -# If using gcr.io/kf-feast/feast-serving this should not need to be changed. -jarPath: /opt/feast/feast-serving.jar - -# jvmOptions are options that will be passed to the Java Virtual Machine (JVM) -# running Feast Core. -# -# For example, it is good practice to set min and max heap size in JVM. -# https://stackoverflow.com/questions/6902135/side-effect-for-increasing-maxpermsize-and-max-heap-size -# -# Refer to https://docs.oracle.com/cd/E22289_01/html/821-1274/configuring-the-default-jvm-and-java-arguments.html -# to see other JVM options that can be set. -# -jvmOptions: [] -# - -Xms768m -# - -Xmx768m - -logType: JSON -logLevel: warn +# javaOpts -- [JVM options](https://docs.oracle.com/cd/E22289_01/html/821-1274/configuring-the-default-jvm-and-java-arguments.html). For better performance, it is advised to set the min and max heap:
    `-Xms2048m -Xmx2048m` +javaOpts: + +# logType -- Log format, either `JSON` or `Console` +logType: Console +# logLevel -- Default log level, use either one of `DEBUG`, `INFO`, `WARN` or `ERROR` +logLevel: WARN + +prometheus: + # prometheus.enabled -- Flag to enable scraping of Feast Core metrics + enabled: true livenessProbe: - enabled: false + # livenessProbe.enabled -- Flag to enabled the probe + enabled: true + # livenessProbe.initialDelaySeconds -- Delay before the probe is initiated initialDelaySeconds: 60 + # livenessProbe.periodSeconds -- How often to perform the probe periodSeconds: 10 + # livenessProbe.timeoutSeconds -- When the probe times out timeoutSeconds: 5 + # livenessProbe.successThreshold -- Min consecutive success for the probe to be considered successful successThreshold: 1 + # livenessProbe.failureThreshold -- Min consecutive failures for the probe to be considered failed failureThreshold: 5 readinessProbe: - enabled: false + # readinessProbe.enabled -- Flag to enabled the probe + enabled: true + # readinessProbe.initialDelaySeconds -- Delay before the probe is initiated initialDelaySeconds: 15 + # readinessProbe.periodSeconds -- How often to perform the probe periodSeconds: 10 + # readinessProbe.timeoutSeconds -- When the probe times out timeoutSeconds: 10 + # readinessProbe.successThreshold -- Min consecutive success for the probe to be considered successful successThreshold: 1 + # readinessProbe.failureThreshold -- Min consecutive failures for the probe to be considered failed failureThreshold: 5 service: + # service.type -- Kubernetes service type type: ClusterIP http: + # service.http.port -- Service port for HTTP requests port: 80 + # service.http.targetPort -- Container port serving HTTP requests targetPort: 8080 - # nodePort is the port number that each cluster node will listen to - # https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport - # - # nodePort: + # service.http.nodePort -- Port number that each cluster node will listen to + nodePort: grpc: + # service.grpc.port -- Service port for GRPC requests port: 6566 + # service.grpc.targetPort -- Container port serving GRPC requests targetPort: 6566 - # nodePort is the port number that each cluster node will listen to - # https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport - # - # nodePort: + # service.grpc.nodePort -- Port number that each cluster node will listen to + nodePort: ingress: grpc: + # ingress.grpc.enabled -- Flag to create an ingress resource for the service enabled: false + # ingress.grpc.class -- Which ingress controller to use class: nginx + # ingress.grpc.hosts -- List of hostnames to match when routing requests hosts: [] + # ingress.grpc.annotations -- Extra annotations for the ingress annotations: {} https: + # ingress.grpc.https.enabled -- Flag to enable HTTPS enabled: true + # ingress.grpc.https.secretNames -- Map of hostname to TLS secret name secretNames: {} + # ingress.grpc.whitelist -- Allowed client IP source ranges whitelist: "" auth: + # ingress.grpc.auth.enabled -- Flag to enable auth enabled: false http: + # ingress.http.enabled -- Flag to create an ingress resource for the service enabled: false + # ingress.http.class -- Which ingress controller to use class: nginx + # ingress.http.hosts -- List of hostnames to match when routing requests hosts: [] + # ingress.http.annotations -- Extra annotations for the ingress annotations: {} https: + # ingress.http.https.enabled -- Flag to enable HTTPS enabled: true + # ingress.http.https.secretNames -- Map of hostname to TLS secret name secretNames: {} + # ingress.http.whitelist -- Allowed client IP source ranges whitelist: "" auth: + # ingress.http.auth.enabled -- Flag to enable auth enabled: false + # ingress.http.auth.authUrl -- URL to an existing authentication service authUrl: http://auth-server.auth-ns.svc.cluster.local/auth -prometheus: - enabled: true - +# resources -- CPU/memory [resource requests/limit](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/#resource-requests-and-limits-of-pod-and-container) resources: {} - # We usually recommend not to specify default resources and to leave this as a conscious - # choice for the user. This also increases chances charts run on environments with little - # resources, such as Minikube. If you do want to specify resources, uncomment the following - # lines, adjust them as necessary, and remove the curly braces after 'resources:'. - # - # limits: - # cpu: 100m - # memory: 128Mi - # requests: - # cpu: 100m - # memory: 128Mi +# nodeSelector -- Node labels for pod assignment nodeSelector: {} -tolerations: [] - -affinity: {} +# envOverrides -- Extra environment variables to set +envOverrides: {} \ No newline at end of file diff --git a/infra/charts/feast/charts/grafana-5.0.5.tgz b/infra/charts/feast/charts/grafana-5.0.5.tgz new file mode 100644 index 0000000000000000000000000000000000000000..06eb83a5e58e424198c76d93dd3c929346b94609 GIT binary patch literal 19271 zcmV)cK&ZbTiwG0|00000|0w_~VMtOiV@ORlOnEsqVl!4SWK%V1T2nbTPgYhoO;>Dc zVQyr3R8em|NM&qo0POv1S0gvJI1ZoJ{uKL|o&oa1WkUm<@tMvV0(8PjH_ZcdvS#(< zfLtZp8p@@Lq%!R^JfHo)bS>4*HkZ&zhMF~LT&2s_mb7o$duznd5GAO)KSoRhGc<|+ z>(QTHuh)CEv!nj)^?K!hH+x$z?ce(S=GOL3@4w*jA*p#XE)e^#Uhkppln3_*c~DAz z#EcV~^kKT$LTT!L?gqQTRx83hWF!^p)%h4hj8R0A5eQ0S7*YnmWSGt58-co7z>f|ez$8DWRPIdBBAWmeqrc3 z7!fhf1{udJq=~?Z2tqosb=M$=z17y#zS{|U!A|Q-FUjZU2_L&krYO!Z|7{q+eER=#Yv*OJ zO#feP?mW}~r+C-^h5Z&l$S@MP4}xX51(5ReFhPSDNB*l19Pg#+9-C0+UQ3e1QxwXQ zLlkrQ4u291ozfVeW-;b{`1o@RU7^TJ;z-1TPpx4NvJ@Q5AuGi#c&#v5b-{2wpuGIaE38lN?sC* z81SbwMkIlhMiJpGQ`~cqMI$T%p5pLQ@y!WkSXO0tm@_%q1WT~MJRr0i(U5lq;#a)O zvZRAj)v0W)8`5M*Mj6AMG#YeUDUA+XSKq3x`tbQy0zM(h-V~8oPNxJ{zCxcqB*=f+ zIEDzPF`Z2$tCgS!Trec?Xm+Y#va5Em{K`Smgd{|6pH8T^k+*mzSu98z zq2s2_66fjjT8ckY;gw zLSqun`fxNnrs9NQjuX+6`*a1~s}+pmcm{I4k>O0?gE2G%It10@NfyUvIAmDxK-V(A zV3KhG0}Kz~Pos^L&61c$E*+AujJ$mxuT`|XRD=YuNwUXrSBxU z^hvK4M?=g6?}WCL8RA41KG7Qoi_nRqoX1^Lgl7X?K(;`JlCohQgiOE+tWmQL)=gpE z{yUU5btq}T`y`$jOc~%0C#jen5Y_;D%IK6x;g=+%PC_LVnDGTe@GEumN^1Y^aZW}F zAUI)!Ds0Dyb2+ylsALEn$PhRdtrbYiMf$B)M6VM`9}f^84=7?0KdRefVa8&$5_nhs zX_vsOVE5~wH+Pd&JJ73asu#)13Y^dg_Ht$?0i}`mqEPTH8sONJ(SPgDDUD7j6UPOG zMOHd+RNd#Ot(5|QwLKhuN-4)`=bARaFmH*?0R|L?m~$M#kkJXyOmGtE@`lvG+FOdE zZ&8er5HnkQU6D`9Ono3iH*ihF*iZ^r--sSZCwXN!!$K8uz|2ytR9yK|@&yT={D^^w zBB`#YGg71Nj4~48u4z+~Ve=zrd=8qHWk6pmCWF`rixZzC$u;c1-roFr6V4UZGKn!c zRzPwGe42sX=$f##pfVc(aX!1cYdyCH!d8nUBZfII&@3wuR$G(M74cx|2cg^zeMm+m z`BYm_#Jqz-#gpVtZ&zT`J61Ar$~DcE6gM0rCOY^NnxrufVj7}Y^MFG+c2$e3J<4#3 z3wH!l;&S{MG+~2oPCjl2$Qde#` z(LqVn5v~ih^Yfp2+7xpc4yBwqdt4B7BTBT9WACmY(~SZPC4zM98wr^GTjNG8<)efM zM?7+HwFW80w=*NjInhbVV?~<7^G!9uC{Yv?#WB4G6h+#)2q+U{oCu==8PX-oIqs0s zs8wpW4>zp>!~Zl>E6WH{zhr18F*HJx0geCmx9%05g)v1}9aXd+OQA%l8i}siueHg|z7Au86VV~j>(U#&-`rbWt3%o7_21v0es^)a_wMi;`%zg1I}x2AlDz)P zHALwZ!R3bJe0NY3N!t0^FqmGT{@0fz-hS}CK&c(L?--hywSBQI#bn(=8|4_ zIj2mgVI3=40jWTt5zeJ9FshAYR7xIrM}`wDAx`$J{wTY_q9^98NA#KHrra` zp07Fr;%!HN0C&Tq&jbjW09MkLdmNE}$9Z>x#MRED> z3F7>kvZxPSFq(`~^IP_9tQ6&DC0ml~N!3^DB+$((q0ki8+OApZvdj;7X1eZ}rDePF zzIeS$9b!ll3?ZFNke1X281N??W>VG9S33-5jx)G*s6+^iBOFOhOrz*>s!SU=nO@4) zCMZE8Y(~Xlt-x3ZDESqUFs50g&RGUA9d$|aZyZ`8&pRX$R5DcFQ4M$34b<|7^9|0dYxuNTMwi2G&&$2Q<9GY)E3l~VRd)p^(2pzv289! zw5T>0BZVSnU|LRW7?>(ygut4CO%l;9d@KR1UlzqI14Jj3zqEl%k|QM`#o34?9A-3w zYpI-&gbNhMdVwK`30G#GmMQ3g8<7zs;kYx%27?%D^*Kcej?H(AA>G&Iq&mk9fAyARj>Fta`ykAr7&Bq)#99gMgkzK> z$^^R7a=lnQx)(kc^HN2GX|2zGEh0^QbNC{Wd9!I>tpFxsjG5KkfU?nrEasJ<%>StS zd?)Gmdo~JkG+smnJev0$3%OIAYAB~cE1h!HpyWcqb79{7!y1S}YRothjfwS)C6}o@7Feq$&qc@BBdSb!L_l)`C z0AH|mT)EOZSTz%{>P^>?Kb1ooM^ap0T`evWBQB~ZB3!b?1BrIksn&nZR~!+cPk#M5 z)<-V>6dRl5928mAdC_bN3S^Q}rkL;X`}4y-oKwkp^D>&es(C>%r$)ETWlpH#z9P8U zWg!Lg%FKH`rI<6lOeE@$3$Zd z`m?g})`95AwnV(Q;k&s)-xZh(AJS4|odM!_cl+*bb_ToKe!BpJ-EHF%=rMp(E3_+X z{wJjy#b%wpLup|d1WJdhTL#K-peo5a_Jfjom59EV*YcK_B`{XM(*#g+Eq`t>=#O-%F@*EZboFG@JZN9V|2{)`}&LOcn0YmXjkkCLDj&XQ(F{JDw zQ!Xsr$)zzJ=^_&v;l9Kfr?LEC8oo%ZS4v^8_-?Z+EAaMU?*#UW=5nL$7?YjWKBRCv zq;X8INiwq2Mx7->;xwOCGyQr#hls@Llv5xkfDBbrTJ@BQR2-o+5Ofj)k?UZJViIY4 zX(q;j{AxFS*}*|s+9gC6oQ#r-EG>M|(|@7KV*%l=$I?<&sHm*db3937rB)4Q3Upy! z$VGcG#&IecxJcsyztXV!=g|^FZmT<1WlvY_SOYW2hb;9K*;+@ocDGh!jqXYj0T%x? z*73`to$gZ3AIdE%>bRw9`7@>f<@G(rYW7Pwmu<_1VHXie04d^nDGPVx#9anWX2KP? z)U7FS)bYHg^+1QZ0X6fNMI!(gl;wJ}Riib;ExVLl5BBFx$8#8@9ACtAG?D{*og29U z;NwTFl<=RegP}Snehux(OwE!}TH2OJQzpOezT7o0GLA1ek8QI?l#sjWLc|EXQFP)zuINa4vTtm6|7miUf!aRVVMe=3cxiI;aJ1j*L|qhXZv=gxFJhq z=;f{UL)|{`L}tV=Oc1*=Bt^?;vbM8DUbxbxalQO4jZY=wWJi2Vvp9k=jU&^k+KAX+ zv9OjUG3FdDjhy>NwRMapSP>bH#d^!vx4UAJ`dX!i;Czg&plqR$D#C`tux31ffM)|! zAWP(bC198O&86B+X(oL5^q?44)<%rB>fRQE-e{I979-tLAz_ zI%-BpVjSoQFZ#;L#@|b5u8DET%TEJHVh#lS{s;$_C5d*3()?JG8{5{ASmj#HPbt@C zeYweupUB?Zx7veFfh$FYILNumO+Ny7OmVAHzLY(>@m^&{IBKAKH7q3YvpS_N4K|J za!9nLrLIlSbYxpqzKf+{wJK&MTiUlPY2GW@kY3-q+YU|Q84U0kO$p6xQ-yjy#Zx+g z^7B!ekX$cLbt-OG<)JB|{BUGys(K&D2dhHaRtrYyKO?1Gf~b0YdKs!}i{FEe*s6an zLLvpK_fyQ6btv}Ej9mlD=x=}6(n@ap(s5nOK303S>M!p~;r}cUpTwi}u^|3?XM1b6 z6#uuqx%(Xd_ax6|0N-H&Gnz42{rnk%AGHb6c?H4k?W)p&oZ16}ncb0nsSSe-85?0? z)vrz`-_@T#=f;(`vs2m6Z3y7@w!Y9Xi(^^Pl<81qQcsZ>zixjHB;^T;-a)r*xB+cb zhE6BKA&oHn58!Z2Tdl1?$!gyf52uY)%morm3DUZ{hYaIb9jN!Ww?LDU;q2h}Op6D8 z6oJC2@to0!T4IpFGS~|EGz?73#s~ZJXCOldu=S<4w!T~Ca(^^bCoBV!L{0z_6u$0l zKq~b`tXVMj%(tFFT1lu38_iwu)yvDv7F45+Gg=n{u?SJx zHosOwlwKrQTqxUynSj^-52vfPL^tEC%LgRs=6(ulM*sR(TZE5YVe2cELw$e+A~DyN z1c`^*Gb9@FxyB0Rek<37;z#e9L+rH@#!!5XW*kC>_&89be$x=Bvf%5oVIH4}R; z2=21$oJ5bS+x3}%aoio7#DQQ49#b4;%|+>XWSC^+cj29OFKG)m5af`&H}kQ)=m#Zwgf|twGC_>llBR zGI8(ZUzMdi&Xb6U-=|&toR%p}jh$vXLv`euB-ay-k+FZJ@81ZyK`_W<4 zmT4@o6w0MoVSl_(;P+oIKdT;`4imbSBDZ(27T(?&z?6|h459rO-uVk}ms->P-esLN z{asjp-o&bec-rezPO6$*vr=>^Wrm$I!KzfdaX3D<_20(F6mEdwG?o%i`~S6}ebK)E zek$*GoAUta^hwRY7t<^qH?7lRvA~>`S!MC_XScZ4%#nDlbiLlxUZRD^;@T*+Y;)UG zl%v{4>KDK=<3Gn-JP-$J2`oZIyCY;zb8#YJc@(@98fx4ed+LZs`!3|`3hdr`6Hz=Q>k(TzhqS4lEzeRMIXgRoxw~uWRZAg=uIuoY@&vRSSt^+QwxLB$tq~d zRn(el%0QTexCr=h4bk}0;$G#ck^fT~J)Z3UUyJiU_Ij_jE9XD2UOvnJPw~|4m6S&H zT*{B^l`FfPBF;BSmCCmL&0pn={d&txod{7NP>&5+g)bLQHZ<&|!wv~J{U@5ljkRuo zBoR&oY*h^Nh2ARpz+{X;J&VFlcWudl7%{zq@9<#?`@UgAN*xrD4=T!c;VCgM17PwYQ!as2l z*A>l|_pTlC)Q{XX<8=|qI{S({A@HJht58*Th_{4H&J<_0_7e*FJEXo!y&bpIowpkb z3(@A#>y&)PY8GQdKELJ#*;HZKMcGupx-h*3*By|g%g1IFs60pT5!#(lu9eAEL>P3^ zW#RzguYU#8STQ=of|WaPa+Lt%XXBqW^$$8uK+tgasp%< z4q;%kC75rzo2lk~G z?Lp^}v7$qW*Z#aSxegCIMYro3I%oS*pOzw2g&HRn>ZC|Qas9kIJpcavL3u;hH0cf4 zbD*p)$hUjn9=@$35OW&Tj+Z%Mv2DG1|MuYU^p1@gkKWr`Idkgha8BWh=QrHmT6d3p zL6qS-Jv@8=;dKA70@;8HN0b&XoRX7d6_>|*?+(vS_G&tskAiB$K2RRXX@s&Ewow7> z?1!V1i}SZ<7e5}J9=-YJGT@b4Y=CI4v}D`5x18i{x;)QGEzw-KzN#je8db;D&QpKf zpIYPzM*bM|A|LZ(5sbXWJ5r41RMTFgkGsorSRZ?v&ujU4pSykT&@^YKFS{`E9bS|m zyu%^`R-Nv=KwG|>eq5_~lxEi^@hd4M4N_*Qv>c(8jw%!9BjRx+doK*OMx6I!=4%(S z9|>MvzbdkOAqXzkxdwnd0)EcMs*&1D3H~2Xa4+Z=R++fTkC)oZOZhdU`Tc50bIk~z znwv+Fz3|HT3U5=*l7jrAT96E3jWSppBmM)P*|P0_bDFf*E3bzn#_jd>DsuCNJ5q6W zfpzsOp@MJUL{O^V&6RYOpQL6%w<|@OmA%rRHRkKOOeaOOu3h07pI`n4IAvllPRD2= zwt{Q4^N?CEC47E6JG#RDH`=2sP>y%h;5e$%E$48RHn+v zek(ce_iB)v1DT)Q^LHt6EH{ke{X&m3kW$YkC4aTu8t?z(Qdgxs{mn6Lu13nJpdpR> zaK3-i+^KiJ&z%0;RW$W2dpdiKyE{&%!FO*i&JOoKoF1M3^J4Gd-O=&IhqJ@eDw^xd z6?H(JDH+_OLlu9(vr|0u-$Qk@$~8YIribyC)dF`c4?ps`!n2yjO_~{hvF16^+jGQV zgE^@ctu031$==!7Pw!6;*6&jI;^NKG+rt`pr@6`>3H4q+pDOxYdY{Ep;P!^e7F>s} zqvNyly|-^KPTqd_?&$bzKAPN}XLB*NDC($+0Y8P^680CB*r_-HwJYJ>`N`vMySY8r z�0Xod zrp9Koz+Q7dM7@xEv7yB}!}Njy7L8Q85qQBU6^ee=(0$LGXajQ=H@l>vZw^Y**z~7? zyX%OnR`H5OT+_$?Jga&Gr}eM1@M!tmI?kuRsg)%Ku8Lfx;+1m}-_a6)OMmLo{>UuwUrQ15m+4cn#7;M>vji`KNBZZ0Cz(5VYDs;;O$4F1I$caqi}@*g}x zE|9>(Eaq6~k*-MXa!is@t2o1T1?st?EW<0T(Io> zI4)(9UfPkM*&;R<`%785ns`u=l9Z`K{yX2aX6e3D*|=pP??nwFma=p;L3fv3SKMMt z0a9nHY64~M6#U0U$^UtuYW$DMN5p81l_mV)eJqIo-`w8mmE-?k?QB2C|2)N0BTTRX z3T+5#!APh+xJbxIG1V)QMDAAdQySxzf76Ysfif|sjQpyv&c6Cu#|&<^Y}S^3et~zc zM3?1+*o7zFKw`9??RteQzmQcEGyjIxdNpdhqITSYOXl0fazZcGW{!NYR$HvfGYi<) zT33$k(=5h18i&ch`ga3iNtWFKloFM?jrZYWyZtk)K)WrGM?y(5Oa@YhVejM!MzUZr zfA`0Bo}aY60c~^ToC-IXVm44kMp(#i%KuLO5fd)jKbKD53Nzh|5h(h0=QG`dsvorx z1T_NTW4TNH*O%U>ztuCB{tx67C!>e5fkpIxb9b{u|6jfA?LO20r+6Nn{(r02%>CHF zy~wUvrN28a!ZUT-sIkUrM4c4^T3ZVgvc~XNThAmcUM8zqOQOn{n5#_tz8nu6YnU_3 zzwZe6|LmuR{##aEV^X*uCs;)PcVCtAKW}YrZa>HWKgn|^dzMerr<<1cTMN^lMWC;k zz3xtHEfxy<##}8`z6%tJSAqKHpsVwJ?_BG~aAWUuF!zNnSy|sj;p|=RhKMDryAU?p zyS_AH$=Zdyqt#1Gmu#hYDTQmUd_BaHtre@~&AIDQi&pw4kMMg`tz-`UeLYI?m*J-;&60Vg82}Jq2Zh7gXJ#s{QMdIO-Tanjdoqh0>|*L6@{BQ zt}GIkPqw)nUXl>!t@AN=X!*H(YKvn$+0g&|h+-AZVclEYsjvJ9x3}=x{gOrN{cUVY z;`xyRHD}3-opmbF2Klfwiqv87T7!qs_PT#UTz96>se!u`HLw`SN`*3r4ba^UtfoML zWsulZ^S1?*gRWSgbwb}$4Y2ax>RyEbNxC?hf=Nfgv49TFhILL(^Fk)>kBv%wecf_e zF6EMwNH%FWbdZ+3!fQPoOKw|s+cbGpEEAwLf5GGtuSLT zbokW}{-uRSDZwT<6dge(m}UZYd+yT&eNwOUcLmpLgXR1#ZQDubduZ>4Ax=dfP@2Z( z{==?{%CfD05AA(jt#ht~z;em|QCW7ec>Ur0P3P;j2SJS!oq2pyyNM-^o*z|2;=AIkBCG>jR7IKRY`m`G2do{qouV|0K^`|4O;X_4UJ&<-H;9 zWbM8T^0Bpz8kftG=i(g{kr#JVL|&8?5oN-t*i5x+y$_Y8(M8l z2~QRSUI&G~r`#+Horc;340z7SiWfL#tgc^+sGCkJXK3ayhkMxmC&F$SyRPMr&CeFe%E}ihW@)m+mB@cT0s9__Nx9rJG-x* z>HkwaT0B)$V!>5I>&ti)ua~JjN>ko-r2fFItv`fisqAU4%0X@3K=ZBf1mZV2U`}z^ zSDrRX%_>^B?MvM_cTp}%X+|O(zxB}4fS77-_#8p?kvci?I1>yBJTl?b-25t!Ofcmi z-gsvg%-(AdU)K&IrP15Esm=lYx&o*=Z}4Z(!W52ic*V1cv8xxd%bVLA zUIgc6n8CM*V-*VCF4^JR>#Bb~Mq4|(Je&CD4cA4z=)QcH^F{xqK&bUZzcU~U&!FpI z`P>i)T^B+8u<&PpyxO6B>5=L3bH~s`F;W`s6^X{uYtvGS#!aG9zSnN=-?jAe{5 z|Jmv7Za&lhr+7*#uAO{K_uWm8lFS~Ppz?V^ym<{HKAcb{e1mY!UDX|UVfs}(mrs@dr!RONDayi7B&lWM(X6}Ufxb-$M3 z)Zj9hkYsO)NGzo*4Zr`UoMCfe_3Dx3+}h~VhXj?nFT7g}K&6QS%lQqD8J>B?Z}QaA ze|vkN-s%t51Q+Rl+q*Bz^nYu!_e}qv;wh006~meG^sqkYtBLlXnFXWo`S?Se?baOE zaU6zplDf+?0~DuYRJ!X~MF-*NgwY|1l~~OCuo}_u3Nw2P;_C9vnHpYmeW-7)VWvKC zyqBhXY(ibGHk77F{?2=r)3ww%Se6Jf!Ns1JYOj7Ewe8i|I8#QZB*r5g^|h~4%Y<*8 z*bj$1L^0A~VYURrzhs2r=z!7G*YEXouE2pwJI1kqA)^yGPnny?r284s>*N}-Xz%1GuT+{MpXQZDhL{WQ-Etq3yyCBr?~)kt*(CetdFuK9BPD(`&m#Trdpr3zOaHfDJ=_1Dm;|Meu# z965E=*YmZIoe6r&_`cW_seg!?t>IHj93uQ3Zv@m+1Z zb=B&4+`M{c%3yygl?Gnz*#BA=SG2Ijg&AHQS6_&|X1{P-5`PkyB`7{RF+=fQ%!A&q zHzztOrA(BP+1G$8OBF+jQX4x|8!ng^V^q+P#(g;7Ke2h|5gTD~($qly*$@S}@08fv zxNeVWq+*xaM(}Oee2FQxzV4+6u`eYoO%*g84bJoTi|W2J{cq&|NE-zoLkw7K|0&1+ z>~3#8pZ`C}Q%@d@McAGMEJ)R0ZOy;Y{U0Tme(UQ%9@!}VJ6B|C_J5wF<7v+SUT&7} z|JmGmwflVk&yzgAoBS`uM1g0f>pmym`#mJyTVy`2 z2m8;QKhH+yDB@^GWA1}2?+0@yzIHXPbn3fgH@PX-_+dZT$yvK*7d*A=ytSA1*uK4m zKY~>BpYv4ZKOrYp{r3@4|Ep(_{`cxt<^JED-n0GpNuHAaC#U@lLlb;W+0{L-eq7!W zK=$@UE(=qkhx*ndeI?^RBq-26m;e19qgagF@YxWzjK#0pDp60jyTqu}%`;=&M$0r2 z(g?qPv35Pe0y-+OCT1Omc}f$G<(pL94bX8{_H6d#4@NMDO?dG+WAQC)ee>5%xB+y1 z1*_fv1Ni>@{G|KYFw3pm)pgU^M`*v09k${1YiMuvdTr2`j@2oobNBxvr2<&6|Eu?Z zZ*9KZdOrVulBY)htEU0KgA4&*f?l|oxW-^~!7|(s!G6!M=$c4n8>n0#677GT8xZXk ztc|_)zI!p{9dH5V>X#Q9u72C#a`nRnmaAV{Tz?k%ez(tD`u|9wfD7fnO8ock_N(Xn zU!UZ8F#W$vl)!@n1iu6aFcETf@o$>hebLxAg^{1{Phc1B|3?S~S|tDVcFOnv_j=Fg ze^2v#sr~<;0HH5J2A(FOM-T0>!+NA%`)5AIe17ix%%%U2m|4$VEx3yi3|J&Jqw*NoL^C0{G35^~mz&&dE&h{9`lYo!A zV@j{ueb6i{Q>UW$*!kA-S0yO-qdaplZ8b%8fBq+7oi7S6b+<*g-1)84<|%9PcqS7x z!acLyX3%qR`pd(;3+dM(J`LXQVE=RNR3aF)W#Z&}S5VIL1@V{^SyX<^O>s zo@A6zh9A9;Me_g7%kuqCTf5KxpHK6wzzGrpvxI}7pcepKk8uKnjKmR1Mv$WL3XL!i zS}SlqCS08{Q6@O>F^*#x#dM&34oNcF0ES~E$P}wvXT0|)iCQa=;E^JVu$D4BB%g4k z>F9s22OzgP&_vagAVG>5#3aE%D>yj2I1`lN)(Y&?$%H2G;!69!2Qy$U?w*qt4)e1<2QCF8|^xrL&Ir{7mK`!k6KRw(#cy|~~qTis8`TKu& z>($G0{>QD}^ZEbNJS*^>UIFkujwi4`MohF?U%^qr1&U*xyzUwg!1;qplAv>A!$(v1 z=QZOVW7#WZqwadEwXy=|Z~y1u^nY5d%gf78SE~@g|TyB%}$b%zo-dIK~32 z&t8l9z38=|m`ZsC-RHSV9H>MwV0c2On1dbq4=($@ERxa$CxQdwoQ4DmOIC9v$u@L1 z+{aa*N0yElQZtJZuwJ`h1Ww5q5G42-Cg|Uk!Bp#kt=67-p*t|IBh9tnDQcybLzCg3Wb_Kf)bc=CS~;7y37^VM`Y4&8#qBy4i}igqUQz< zFb^3~ck(POt;_2>5bpZiv~D`fKX)m6pYmSK)(u?R=!eV2<3G+O1I*-surb3!A??^%NhNysFdKz_GU8LouH_kpY#uEJGLmrZb?PYSLqsfFx; zaOGt2OfV$yXx7k_T*tO3a8q_(>u~uQ+G3XCb|2cORMG4YX@mrB!|gnH6l{|NpATa7VSQKhPD)wC6K<~=mxZ?LO@GHlS^MM)S8X<=Y`5amMiZm|;Gqan$asJ0dT^Y#>SQml)v< zoZ6yGjqtt$yuDO#l`L4zfE5=+$?nwvSKIl#wfnIBYD;2CyV~jB_3J`!AV#R_!G=dP zL3RVmJVS?YVyb|uqqqs@WRyTkBksfXkPW2mZVacC6G2(i_L2P@$)b=KUC=sQTGhb( zyDW?zTmp?2)v|C2G=jCucNrHjz>+&FZhX031J`b_8*JSjV;)>-7RTDivk+ghsBvmt zFpkTk;V~5_40D{!S-6YfGWOHv_2t235=Ae7>+@WCHy$hfDA5@{d=x488p>Zv3Vsia72vWdG3MZ_W_{UO^>96euGVDwsc-AX z9&tnAnwRF*A7?*mMPn}C-p{z86RVOeZ2fWI3K#9^8z6RlnX+y+k5200dW?B2rvIsF z?nZWxqKM%FuNGA0_<@0`3u|O(?VmVWzYlngaQVrAJVhZUTu3@`bOQTF2d5m&&3z^& zRDn3Cf$QVX_oJ(1T1@lPwC?S+*`gi-Yt*cJ0OZ5vF89|b((~j(?U=~zo%)Mfhf5W9 zabJ0&dTD(VzAQXXLMMv1@c(}3goR7$1Pl1K!&eyMRNzRTC(MDX`vAqgaAp5|g|7ma ztfQO<8sK`Ab?>hHpToB`T&k!-mZA+Q!zoT8`6r%YHdEl5vr=P%3g83B!z|Y4KK2ty zM({Dzi^Evkv`umEr!&}l`>pFxYF&)RF=pBh%=_JLem7``lMxR_M2xe6x_9)NcTha& zn2~itSGzmPNQArQLpR?i-4)aSdhl5I6mvI40iWk@&x+`3DE}b1T=>9ZeQE#XL-!q? zNDa=`a`E`xm`q4PA$Z_^^&Ku2?AK!&IRogBGO6YoE~<3&O8E!D<#|^un#XfjbQ$O# zIFHtqWC_FMp7lX-29*VHJp^BhC>$k3>^ldOW$-)Sd3?}8!JLdR zmqoZL^SB6Kj|G>P=cd};O@5E8)XBAqG-*Ehk@Wr)qv$6_1b&}{cxmEag=?M;a8rN3 zZKF2uK>AgGHEgZkTMw6ipKJ-PkI*h&{hm(4PRgjjN=j=(JqoMqD;u0NssT5k90&mc z&jt_?=1gUk1tdVZ z5@spfDCfmTfoqPn{|02yPAJQE6fllYYVXjJ<^KjQ2N|CY=%;(x6YJorwe~NDOQ2Dq z{G|#*xQhjS0bI4#{>5+=E&mU-?-$^jBkfn=lHyE7(1*W*5%up)DDEof?8VB?Bd{mo zD94$G@(VFBAF_RoqD6QN7-JLQ$2U( zAEdl0xPJ7!_wNIjVaksV*S^g~HwUzNaJlG!5`_1IEr;vSB)^%5v3YRWk=Fz)Ad6Ch^y2#~Py;m95?0y_GnvKRF z(s=!Tl-(G)1R0XK=7(muT;iO7sdrN`CGm;qY=!p+fP|z(4 zI*Bh;xW@*FG2x(u67BLk#&Csax}dtn>@p~b6y7~%%TJ}k+JYkl)5$#^ZFgRD{+RdU z`OPhH6#394w&?t^7Oo}EA|GlWDdd8ldp>$Xqb1L59|x|tOQB4|wdA4gqri2t$fe2j z3tNZFD{@Unj6~h5uX)FG)b9nG z!QcKix2FnRCv)5l%Wy67H+-NS!os!iyfKH%6`kMu1J7FwTshNiU`)Axhl;Q;E=1QP z!oV?N;aKUi9`9Ds8BxT$nezVTT`ve`BGpd2M$1gA39hq!D~?^(!lg7?N#p4dOm!!M z74Rikb`K8O2$xIKpZJ^f20kR@mn`R!g>6{@3-Y$>M)@GPC>s$soi}h4S#T&Dp@eXS zwGAmJUXMwrW0$;%+Dot|5){Vcx)7IwB#8LBEzOM1y0IHc~kNOXA@@ zc=tlMBEsd0I#BENQYw+5AX6lCWYUz9NTW!8H|tFc*f=UdcPC+NAzaG;>;uu!6kGu& z-+;Xmj;)KZz@f-h5)&-O1Ncd*;6Lr1?|-k2V}_bslQ@Qm0y5vMQFue^>}PKe z4^MyyT~4_VqB!bs>tPTPYTR5iMWz>RjZf9f$=Mnp2z%0%*e3TyWs}T zuuv8`mg!^WE0ThURRA6tT5Umgca_OaoDv9Cgy~&P#Z3rT!ce7|8&Cq3Q|^Xj1QV(E zDO^?Qiw*tr>5SR>7q0YT_l^b_3$(_A@r@%kRzyn6aNt<12RfEtIn}9!GBCC$9H~Xn6_d1E()61~Rnv>p&dfJUP9+=KaEsGq z4^xMHra+dcJu9`fIrx)YJSh))Bskir z?#ntU#H_oHT6ZLo<+bMNEZqUTm%v{Wu9Z;tA?H7SPlN-@5~bJLxy!hmxFfO30mYm` zO1ZXZ%dM#3ox$GQx9X<`ktB*PxUQDoi6x9VyIj4wD(~P@5Yz)F8A|(cMBWds;zVk=7EPxg!BTT8Yei!(CIMY! zOuLzb`}t2DTw21Y7g+bT^kD;8Oq$!q)*q;`7vV}8EoojvnE^=c0j)1=Xdiuj5w)en zTnpazDBV^g!$xdq8BpISE+||YAx-LXmzRz3p4YnN%!OAmCV{zlI>1#YBJy;SSkzuY(f zsFXEHi2Vbb9Mn9=`(jY5H~hew5~B*Uu_1d-1Td<6DlnT6&J`Iz9IkP!4{_^pT!8BX z$9rkI$0kid2~y_NSzBy5(7Zvdvl?Jsa+u24*XK`s4wv!)_ zT`W=ZVOP`Vm;d;4Bl71yCAhrRQ3u{axct@iaJWp?oF*I{;2}x!+&Wt1#-DJQ$=Mss z+yc&B_B?;f`?;R!sFJ$uxAnZ^r!m&f{g!;`bh9e?9gfOW%&=$U| zIDZDE@;bFAnjiJH2&6fQ9vVRsXcW|!pKb=5!97CCYXF;*PoxI0xp_n$>YPBYlbE9D zz(3)rb8>9*9Msz&_e4`B65DFWx76g@Vm{0ba2?fq8Q;Ku#$t$QcvbKzZfqSc!;HnM zAn(dQ?__~)f~&IB>fkE+6j$JS73_X}j|Datu8Nm&9bEHH_a1n%=Ol~wQ_L8N=47a` zC5-&wj@@$`KnX6XM6F%=`G>WMqLQ7Z43|o+cK8WR(pYJ66%Rv|3~z=eajR4P=%@KL zK5hCcaW9eZ6tQkh1~y4e*Dq%R4En|z!fOi^qHp@{XUCN7s9+X-MV`w>tS{;G8)R}l&cL$jFf8lO$#Dom8VxT(d7AS_hHk$ z);Y*^5OZ~`CY>$epyQ==qC*9(vzZl+AS1$5B<#%puCx5PPo{0zO5dy7Edvoq)TqKK zqjKJG#yeb~JZJU_oZ?|0tlPKsA`7X1-+Xt=lpZBm97TQDa<_KtGr0UsZXfK!iR)8+ zjA@ACDuOZ==lwHuZNj)Sj+x{!SSn_8(v`ovIlt&mkdzgDAWNgRJLlWIv%}r(!p`v= z219$^YXZ@fleqJnRc70WVy^C;XINw`f&V>we{7~Ao~cdaX-mWwlk@#8;&^u(a1zo8 zM|#rNa4-tsJ4qbHdTURmyi8qc#xdeBz&I)To=Xxhw_m98hEl1!&S>Tgpa=%L+Xh$K zgB|cM8H%|(DMLeH&O2mNI1y}?gBRhDYNd|@O4FD~`4^jewo!ZDtpe4))~#zv|QJ=I;*{cJKVt4yu8-2p>MIL5sDS^w1`(QQ{X-~BBA zQNJzLb-t%!3o044D+(FUP&`wht#((9{fsaha-iJ(6+&k4?!(!+T$2o^u~g%wI`WsT zH)iVnP9!?r&e#A4seG?@h-%mXQJkEJu+45n*jB!CT6T%(RYdqae935NrPmntw#Hod9c{ANkUO^=!7aaO`6G`2u}ODR z8kr-?ytq12H#u$Vwt&SX$5FylZoq5Vw~tWSAbNPEypZW<}2Q-RTZ zxEv@Ot6s=n1u>>__2&>t27b+m5IAXRrlJow)ybBMNi8U{0qpA|f4lEWO-*P*Ot?v+ z4}z_uN6YtgV#XN+N9N=#3RIEnNLJmFSkpo!O$O1AD=+mI+hz;Kj>*12So z#ezsdu5jooan&leBz54h~@LrzG4)2zImRj z)|>#B@1Wv+Gf+9z??ZdD?QV#_`kI%xwYe?+V-N&CJ04ctB<=)m{w}FGGIiS|Q4xUm z-y|{NLT?e99#HO&d_`@@<(iGghHsj+fup*u(`wv0uyE^?k6-KePD}3{3#zwqV2Oc`qGv+$|@3eGb*ogItDW z*p{1VMlMTBq9JA}?!)^O{gUAp7mOsM1JgrZRNbd$h=>&L4Q`Z>Ws|DQFFx#>EmPR0 zf%E6v_YoR6;-5R($+82)wce$_9^}=nly6n+?-)wQMC4i;Wh1Fk%m+awpggO4nl~mz7H%ZHP<}v?@?5@OO>I4V$<<`ljH+>Y)`*O^Mc=;nX(hEmS#!rIrBn zw2qc&$;&hmNPhQeCM9mIion{L)>$ZnZyCMjxu@5MqxDLsx~aZyCGV3Y;u|?7K%&67 z0Ggv8LG@I@rj$XIz;8W9TBVsRjo=$8QI?gxez&Xi!7=5c|8?){UbU8;Jx4({WX~CQ z9oqMm>GbMKT2!tl-_5II+{X%a>_@NOfYSII(7d&4{Agi-I0;qsk}0TO>}6tH(_>!P z(adtK5B0EmH7yp(HBYM8sfAcETF;{a#u!DI*(eTCcj%M8 zYDX>0k-bNZANVFB)z$>1-@uUII5Gssp5BHunPTN_ET}g0cyZArQLhxC^h`^ZD4NIa zEJyCzo55x~|1M&#=EvUuReb%E9RJ=vW};BQn+pXMRd*>{!c`mtPC_7@snEUYuJS%>i<^;YpYe+j&bCci-M?@4TuIEN=ftF?Dg0YrK zEu1cO|BE$=hNNqjNk?nFk9GOy68C>??(|Cc|7^e7dcObbNuCvWi?t<*a4m^w7Y(`c z5}q*=6}x`~EK9h)6pRBJAu0bB91w~DjwAtZ9R+`gyRiNfA8-Flc9Ltn9Z5SMhZZcn z|M%5qY5%|4+}VBJ|4;E04~QnSjt!^LtZy>wple)zRI~zn)}d>VDeE@z;u4lQPrA4u zNty}X?!(8QH;P}ZM~?jFeJQq}%Gz6A%u+6#3aJA6)Tr^Zx(Bom?eCfj*KO^ACfGNi(jNndQNF9NZRg0ukex7R*6Y+RkMgw zl;F7Ly*}{xF+Z>TTk*4)#RdWFs(8jJ(5R+8bG`f|_sbxy_~|yi%9#lp1Gv$S@yYsH*^`NaLuz!`t(r{FNpOFD0DVljDPIGMOzn zIzEcRfByd{{po7iv*X5N(=Sc9AL(i4|Bt5zERz3scgp;KcdNJa%>SR_c>w>n9tWFp zycMs5&H3A>n%hX*O3hoBvq;-aYs{jcJty+Ix#e%w8^&tA1xz0+^kH?iVCj?BqBk|Y)_STuq>n}W uzt`I;@BdfN_kTal^ZE8mL-71OKhMwe^Za~?=l=@;0RR7N#sAg-t^ok+hBh?- literal 0 HcmV?d00001 diff --git a/infra/charts/feast/charts/kafka-0.20.8.tgz b/infra/charts/feast/charts/kafka-0.20.8.tgz new file mode 100644 index 0000000000000000000000000000000000000000..f61be294aeaab0dce87d53c0f19e35a2854a6196 GIT binary patch literal 31688 zcmV)uK$gEBiwG0|00000|0w_~VMtOiV@ORlOnEsqVl!4SWK%V1T2nbTPgYhoO;>Dc zVQyr3R8em|NM&qo0PMZ%cH20zFuH&9DQe5vyX}riNxmh~H$7jw6L*g%7yHES%S=z7 z7$PAFF-foq(6&0xx6W&v*E>&g7773fQj{#&&SiFoYjwvWfkL5BC=?1+g>y1GC*7wL zlFML9lK4M9z|-sXdXIK@@!wvr*ZjA?wY6ve*6;gU{q61lfDeJo%u@(S^8e`d?h2>b zxqp*~Wb7xJ3&zs{T=YGXW#!M^pcnMLh>9?0nZ#H783`v8o-3vSCLk+@F%uJC6hl>N zNFkv@kTFZgkW)F~#dsot2p|wKA#*k?B#k_PkS7U~5c6^1O?W~FFp)A7gKifYG9cJc z!1Hm}W1(t#?st}uF%{jIk9m-#WAB_!FL@q`f#(Aj_I&t*^K(iwn%j3EB$0Gf#DYpM zAuN@Ir8LJ%G$AY=z=-68{PAs0RM$z&{-8_-h427Uq7c^K&nG0U>Ler&g9{eNB;>gy z-;UMmAmm9|YtoZLmPb{UAx*y(8J829^IFLvP2aLHkE0>Y$DV$t`y+UfhZJ$OR2efl zDTb;VE(~lN>T@!Z;9DbIOORl zE@&!Q8g|3Xk9c@ab6*cldlfx`dN{oc%X7V8s~0@>KJ~2l^E_JqUy!(<;zKO}i{<~0 zdYbZor?>S*{(p>T4g7n3yfwW#47_`X?5(ZA>4d_G5#1+L0!xJ?aV%gPG9C#SP9dTr zQpD2BX%@4PhyiSuRkiF#u>?8IxL}g!Qx9JZpwkp%SiI9(TavM^c1p$r=-^t}0a$~l z7KSM};<;Hbw}M{KLtrNv4cTZ42H0^C$HzQo;Z%Lh=}513YOyP&Z@_pr;-Tmkg696X zU=g(|-&&SaKP%#xrDH2{8*m5EIUK#>@;Il0rc!rFVamouu1KMm6v*lC1r<^NNh63^ z!lYOYrI06CA*pX6imq4wIe>TXJw+l7cz}Ncgjq3wtzIuNzZ079d^&*s?)KhGhA%Bz zwJQCVD!r|pm(0U)`Wq$L5vna>03H9ARQvoEWK(Z`m zGzw}Y7VR_+c^cA8itZ)P&tpy^d?j-p$21pRN1f@e6^M;5G&YP1PrtF~DKg4wra1m6 zeMVy{sV2#ZB)P<~7m{asMzIu*NlxOJ#=vtp$|IW7=rtwL6xMl4{o&N-DGfFNUU5mm zho>ZkluOf9AR|e07_nSP$Z1T8piA{w7LwuoMlQE6q2SP4N8>GHWi7( zoDxYl9VuTULPR7{`vBp^e0)+2bzeH_O~;NgkQ7KP6^$hY5l(1S#5CUo8jJ(zDD=)d z%kd`>=b}rZgry4I+z;adRqW8pRX-_W$+DRGWjmtlt*t4>urFSz(VBxC%`aF;_rp+8 z!QoL9B$`XLxL53Gc(q*B)U;n~z^EEiLW`&g4q47Piny2v@k%q^%)0?h*m&aG_Xy&c zN46UZlPsqrnyXP0uwD%iC>GL^#Bd9-QUG`gf@VYsB}jQh#l~u)&=Oh;1R?|*r@oQ_ z3RriIj7BVFa_W;b@<}S0Z{Mg^zN=8Yq!97+D+xkT#g?&upwM4igMBkF?1IMAO^}Lv zym3q=y`(laJ#QwZ0K51i2~N5X~>IQX%bFV z$<>nl;%kd@4584 zfn@Wkwuo?ZXlKAxn}6rya&%N&`ie(X_eY?Ds|1Oje`<}r&q<7O!O+Z8KS}~xT}&eR%#Ls9980RF!HA74a8S%>$)&(0&nkO4Dq$8fA=uxZiIg4W&;b-;d#sd2J z7kz&xFt(hQO+c3Os0h_~DvFBkTFgc?oQ5&=C!C)P6vzq(mg1=UDVHpyvn@7Fef^$l zt57_K6@!UP;x*k>Jz4@vtvCv7pM$SGr?npxNsjB|u&4HAaLQw<`A~B;@*c*`XcBm?(m3S>T#{6(L5OIiv{Y&VaCloN<+n&n!L|C#z{WM@N;JD(^|u(n zufKU~;J!h9@LbPkI@Kbu5*VF>-wP7!w@p{P!6E;iW{M5CjZ|kiDEV1 zjB=r-nl~{o^%jk!DVG)r^+J+-Oyx1prTU$zf8j~*$rb`UxWbh+27*kF@ThsGiqn(~ zV;T)$L}EcTylF-uB&k{kDU%bLqfAl_s9nknPNMHfOwy3%GB=RQj{TE5zX`9ssh06H>PbOVSqJWx9Jp($J;j<0QZ2cRlpDsx1){lBFT?p8^X*8uQBlp`)U$b!WA+hkeLt$nyx9oRV;E9dhq(y{tGxfhOrW#N@-UgYJFj; znm9Q;dv$_6rn$n{_NAhmp$75~^pSBuZz*5kR)OaF({{i2M2XahCxoQ~Fe-VV)TG!0 zh@Hj{k8#G-U!I_;;x+Ek&1WUd^^jUKQGChc5l-Du#&05;#e7QhPE|(RgnGe_^KPLr z2KGyKG`by$B8n2d()v8q-zv7%3n0AE%kaW#x=d++G2|SWG1D`Ol4Jl8y`ZrY5yDgT zo-vUZXearuh{jZw5E63rV@fUvi`CQ|KwrIlfp#63@JmS4_SsQb6Eaok6;&OG3aSvj z;Bk>kl26sJo5n;?Gf-)|99_^nXAu=aWj`w$5mZW+j)j)P$gnXxr#LmVymsw13OH^} znP%w`&l4gyfe53)M_{(Kd^BPqQxlDcqD7}@u28k*sas!~lvu0D^s+~ay(;5lAc{;$ z$RMFIXQ2qR{Og$MpeX%}3H1tBJ7aCsh1J$PVW>mmICV%Nc@R?)(LBhh5m*((QUd9M zT1W(vXSH^#Qf;g2lEo5o732y@DmrMr^Hf=DhZE&CL`$Xmk^mQs)a17w2mO~{nkH`; z4QtJNqer1)ehWhlX?V%bnG3{{IHcW(i7Y15?(_ZcpYL~{A0GSt-u8cO?>{`z`#8qa zhkW$V%Ke8VjUHNQ|4`;?Hx^=hy3rHOw9p~Z1K$L__{IL= ztJ8yDP6yrYgLlVA&)ys!zaJ=RxY^kRnu;Q)I03Y|4;ffXLfzisXwLe=1QIth%k$bC zj0SM+8;h$I?9c zdP_Y02kYx?7&Y*>OUzxc#L2oknh)N+INE>q=DYnD`>&oV^p4*T?6S7D20x}DPZUpU zMNL+0h zjg=Zn(~C;R;c|tB6f|5=+#o3BfkY8Arg+7H$m{efAgv0TXKSzFN502-!ugl#+w7ctS)ZQNIb4Hc*M{W1@9z z$1++|i&5@s_CV__`-57gti(nBgm4hNu^J(rNqJ6j%*$2JStgMtl_9`37ZUOU6F^q^-$l90sLv z2+7exrf6Cnx@h=laK=(GABz$Z$JpIzn2;@I8WIjlMMf}mJTXn8iL{gyMtj(_7&!*i zzqqgdTGmigNXd4;(^jd|*)(TKkzMhlX-X0ns?oS$YMp24xP(@uR)1MX;jayx8){2r z8cK^e-h`CTGzKE0Au!VyrE<$n2q&E5zQlo6wXAjsus+VV)I8n)`9$@Aq!Cejg4DwK z;rYR)`Uw9GM;T2|)HZ6PTm)-q3_R~`a%EQa*F~)N&Lqp?soho_m(vyg{_+<%xKa`W z@xa5U%L$7WJE>jz2zNC?Z{3HKrVz%IDB9>9l#&L~rfy8?wKk_9q-HJrlW1Xt^`OX{ zikpBQJLX&?r57}ZF-k)O8-U{LsW$mqbfQ9^UQuBiCDQI;Bbu`d8UZDO#Zz6~2#=Q} z)ALeo;wB`Q8Xut$QY5kRjLA~O>Ae8j+AD(TMy;w#$ADVirro!x&KrxK>SoF%WU4Vf zvZ`fD5*ol^qQ)Q3()20jI3`g%MMW)>mb%tx9R=v@W%@Dja6g2`^q5}?!_`d&)8!MA zDMpnPh)W3pDWwrQk+q0G7r5CI7&E9M%2=&{O7bDRgk-kHsg{bD>i;D{?Lx4AN%9mA zTcp}XlRu_Fuas8GF6ai><(L;z@44ZOn>A1GU(WP_DwVdev&#vSR4vFy!?!fVQyJWB zo?(f4ZL{OTs_-_s`Z13O<^E00mONpDZnyh3jaXOz-QRi?^nzZ{AMEb#?snfMS9!up z7%{)3c}N63Dr8pZBO$A~x0b-YqrtIOPKcUi0}txeq#cw$eaO^`ou>on1YdW`0XiHt z_i2)Y+V#X!Z4cm=soDL#)Y~(h?Pg#$ZBfNDiYfJCM@j-$H3`}~$^lWmI77WUsFB(V zTO;^UnNYyrCRZglC{{PKTg@ze5QHV{lu}cMrLKt5UrK|Q8V=Kh>dw!W4M6jO*tF8X z$ZE5TGAn>X0T-D7;3$pJHjsj;?Z+1?(!THMu6sv$a2R3(LwKw`n0g;5ZzQl(br zOcLGHq8WnRFI5X9r|DM`Dm@MbRHcA{kkgoMz&^vzl129&J6U8xlt#|1UT+{MIb=&1UYhA+Wm zA$V$bQWe_%+vLip2Fq{7glz5ZwKYY?D>R2TG_tkb+uPcHvKL0XWYi~NGI{IZX;r9cFr2!@kBkLf<7xm8px+Z!r}dFeYis7X$2JRjk*?IDu4Y=OH;aPH z9gLD=W`t6;%5ekH)905)P?-U2I&Ya3qIBTWzw1t1_Fd_f^+UD#xSm|Ky8StId%c&; zo0%3`HJ_arAy2ZHp~)wox?ObVkq3H`et;Pgk(AN+ck$ND5jzg|GaD}Y!B)`wGxBtu zn&ogS!ns&jR(2#|dDyhwtE9ne>+fh>9A`P*oQhjUFAvEpJ}+OnY;ImwTwbr(Z?9;1 z9M5W2JcHD=Zg&n@v8RS|Ux$*~m!N(ysavQWPFfw&IiR%DfA)~lerX(3+Sl!el_d*( z2u~#g{3CcPnU-$!SW@L`UQ6YEXa&zDQ)i(Ml9i;3oRO--pA9ogl{1nZwVFGQeR&I) zoT_TwEysD^;+j+YI^&+x_}b>8GgHKI)0z3!aMfw~ZrycSK2<(XjgL#Z?3h8l!ENWB zIlAV$qhQav?>Iy$nbn?Qpxvou!T8^Emu<5xe#ui-LR8d3jXxtHnnd*4CAke9&<4+1 z&2lXW42aYM7(k+nXro&xd`m%*I!&R;*pxEgUsZq{ys_WU^wh@bbU7hX=ta|r^btq% z#8aNrqZ9bmA6!!ok8scZzs=`1R@c+{M!r6-wfBXuzwq^k;cI=I%@bPc z6KDwxs*mVipOC9{T*ls1)|NN;!ud@?rKnUd&~yC_BKyur_-{JNR0n7Ih5zo5e;vZ!m5^nt`w?FUdjdByj!v8PoeP1)y$gP47Ed@~*3PR^Ws@eydG!3oJx zbMoCg=)8N^f%os>fnIilY9gtR@89D`CIB7kfWr|m2@$74l~O_=zQ7|f8tYjYk`!bs=EUy=mpJDK@M{!nf42tWsR36bmOyOkTJUE1Q^d)sQpP9Ptm?#cXO~| zVhVqI{E!!^+C>x4?vnb*OrKjAFAdY_2@&SB7Q+X1sF<(#^Q($&zQR_&vMj}T`aMxg z)>rxr7pczFPLvwIS8M?fJYgNTr@qfde#)g!ub7Y`*l2BLv%L-7w%+g@yGS(~w}td< zskmNp)f`I|azZ2Lv;!k-2*pMUh#*cSh5j6JFi~&#D-0w1s=;_w4I@9$O=4{g1dC}Z zL_qodZ?36T2og=g%XoGmPO} z6iR2_dCg-i;TpO=4B(X48LEwQ(SOOD`?qp1o&r`UkmyOiE-2NG3gEJvZX? zz;G%G2u)cVux-w=JLtMlWPuhi-zYgmLb<7l9pqHf6nmD4Z}xgSk1>@(XHhe+XF6e; ztayC@kIlOois)b){izAZ@n@aW_7ZJYHDA$7tn@RJlNxQ7M0DGHsxj^WI)aXo;fGU6 z#isq0h|Y%~{eBoP+j9UC#%095jXEAELPuU@!>T6d0Di?-F8Hmr&Kr_OZ!U8tX?tzY zwTt-(sqyOPT4f8HXh*SDboS#Ie)H}>lf_yUs`(-EWc*P9x9%2-l{agqfht&3I~9?b zr(>muXf?;1H3i-&$56YwVwHlTLT6zziaJ)eNk>sT>G?kfkAw7rqcDm6(3qM4**#X(q*roQvFC$Z{m1>% z{pZ6R7k#Y5q9u-s6v5q-B1~MP#AD8g+B#`tqGBD~9QnqVh}lR8mp)2y`=b_luA|Oo zqRs8nwpz4#C9jt|>!@V;%z;iLLj6U#?{iOE%oq-w5a~~yqv=CFT6?Zcl&_9X4^9Gk zCGW9~8}dKy^>_M>{9oI9+k0Q~e|?N+O|QaAD(XF^v2x(HhoxE}D5sj|{ll1oj3lip z45w<3k6wTT1VbaRv-UiHENnEU@%YsHPdO>wdu^a!`5E1X8wn}_&RH4_l#uoS5-LeV zq;ciZTB@W%hB31Tn$8)=yLS-0HW`Sjvy=DlJ;0rp22w^G7>kfqeyiix(%VPD`$-$o zwNKCc_k-rUQ!>^qXxK?Y(x@^u`5-3F4?lE%bm$cgL8@K2KZ0&VFS=&EX{Z5YpV1=-50aZR%3vaHUX{Rr*kO<8mFo?-<(VP2AJ0?)P ztQ9Vd#u_ZN?dulZ@+E8STF6tuV~YP43C+Rxhn!0xbCUVmX;ndHX&4s~g^qSh9Tjnm z%OnJfS)cTtY|V66&3u1I#|*g#JA~Q56rb5BG7#Dc!f+b;L=51m%;SeoO%0t%O#6}< zXOP=#Q@H|E9Q7Hy(^rVRsj+8iDxa#miwk|s;?JN+5 zT{B2{M2Re{h0#XK1fKWx*FW#Ses%cjhr!ohyQ(o3vXzJq8ZlvK$XucY6_DJUS^M1R z+$_Mg2x2ofFij8R7sRU%(+de#03%AX$4*66eI&1j#e`l4V9+-qGBBc-kg&9n#s!na zu}6g9A;Udrxxdz)VIfbb+8)PpGF3Z+ckcimsL6-iH~41pyu=DVJh0h3RRzmg<-2Po zW}L!Kx(K&?$%*cFtn0MHxvo=z%Q21N1GWhd5Z@w|EOsz5%R5%LO}J?(R2>pU8ukn> zGg&O)0dqmJ(VU89tnvHzYQ|+bOXUbU|0VqYih}SB_&Mcp!RRG=h1Ye@^{j|v zYkBJGIC|t`ro-N>>@XoXWtiK>Ky$X!ylS(ITMOJ!Bi)B(Ry>`2y-aGAkjzy&Z))M) zyK<=R(%j;8vvRd=ACynM!TItme46|JHxn9XO6be%-co@t-v9S|z3t}ye`j~A_htY8 zF`jqtx?jTuO9p5k9I==x)%BZ1ktLka0es!X`P}{5JGjbBET0o~Nv)P;%K0`ag^?Co zQux(3)Vu){`$Fm4~GRw48 zSW)8Hs}ogLsZL5+-w2e}=6Xgqy?YwhjI2)YW9=ro||c30uDd4Dg-eU$~` zWAoq~{;4)}3pr`F*j%1kQ1tHHRc$~!Ee*Ef^#K_`5Bm2!PZ}puKfZo}P9fEBGhvTi z_1v~e6{>yUCI$hXKQ0q0Lc~O<)Vb-Vhg2!_f4PsTHzFFTc4h~*qDPU(9jk{u$Qs+A zs#vB~ZOKn0dY>&CeycvuQg@)JyWY?e+t@4EFvzrycv-_31{BJTMrN@p^0*Fj229Bo zxYLa2m4%`+uFaZEI;|F##sW5r?fiystSp%(hc;&Wsz%ZHReOw`OQNv%Oz{$0A|utg%XOF?S59IcURCOZ7#c z70#8FSf(>qu(%7K{@a{bz;$JJd)R@_o6hYOn)k&^QUiM7x)?VPwW+4odnK zh{zvK>qwbEuO;v@DNvW7SvdgakjU7by%wBX#vxc9m)JpLK5&PS26nS#Ti2FB!(`UJ z4M}v>dnAd$<-SPL*0p8OFrMzWqKT<{k3_N9+!s|^d$$;odJJY2&Cse`Iu<$`wiGk~ zGy}|RpV|*{8(PIG4W%=1+SluMTXeMn;KOxw9;S=B`l$^QU!IThH0^(-<=2Sp z`?wSE2LGSky+_UaU-!2AU+#bXC{II7S&xZwH+5b+p=<0g1GCiCJan{wPML$vbA5EG zP%CV=bq$Wob>&{m<@@;Z0%_ zlRrR$uf1)<2)c#H)pz`@&-`3nX2z{V(dzws@O_#}Ca1o30k9q@<0A%XyHj=QC$s4$JkahMj+MhuzJ8Lb_JO)+ z;5hRQhGzDV}g|b}U|T;F{@8-I1^QTx0BZ_r9v|T-4ppw*g+h ze;C1eU){OBu5!0FSlT+7=xVj_vO2!ETKCLLM79n0W-oO(nvH;d_YP97w<>e+TfS%E zr|J@kqWS%)W7X)u2Sn29BQ1m?VeH6uM>3O5ctFBf$>-Lh)&mA7JqhEW5H#0+RdIFo zz99hLuWnfNbxH!ssDK$W{5Nj59r7a4V}9vpIZH#9kr;jyG{>c;#6vF!rYj{r*qIp& z!LOo*k$ly!_wSuiS}-lEH+VzBG090H7OpHU+`Gu(>3%lp^Y?#mldF5W5dNn9Uw>z- zvH#oM+y3JJ@lhT}F}}fXuHEa*@v2*4A2{!rVXkl4!i{6o>^ZY@w9URvuKZ^5;$WJ@ z1Gs+JG}yH}*H!DH+FbE}kR)ENM(NPFQ#0O~xxQ**_l6UoIRJH2y)RDlzD$buMtb*I z=f<|Y+O|V;(s!%9!CTH!==`g*`L9lA8F-!FZ2OLIbdowbN#>iGy#Io}OzTzlo~IL% zrZm>qf#(~W$cW#p|6c_y=n4CSZrJM(d=`DP{;%la#$W&P*LB2k!^xRuT5^-qJm|l9 zu=VD_jw#j5DWjpjVbK6O54Jn^h4?9#B(AVu5!vk}(ZHR4g-h9burhg;ZLwyj*u~}z1GV7_f&=xnm1`2Jlxnc0Re~6&7IbeZ{i0HZw^1epq^RYpn070 zkP5*o)DEMV9@9LesZ{TD>9W|(xSAR_A#00e7V-d^be)6pYwbvFlHHeho7du)<|j1V z*t8DUHxKZV7#n!Ac(#r|j<7S>sJ7K{xjstiOAL8)1w(Vym*~w#9Z8B2|RJR#qX8R5EMnxoO(^UCB6KC?R)4hng_UI~(To1OLDN zugiyZz^d9CRSlu;AemN%eIwjipwI&K>y5v*Q!3`|ExRndS>TvDT+)VI zZ(Ac)9JI$4PL+OPHUWcL_w3Evi%@MlGi)zZ9U{)Gf%Pjfa1`@#EjOqSzVGTO-`vRG z19O?I3*nLu%}uaAiK3he@y$ZKt&4Lxj(Fo}4QTBy)FnpMfmUV2YB6if)UM)tOQS}= z*Wx=qQ_;EoqiVf&CCC!3yFkkJA2)dOCQQoPElM=1%t%Y%P*%S+0`64^6%w@z5|%-v z24*2L^OUjX`u~;>@6q!|KR2xZ{YU+#|Ico}-~Y1ye~jk_{~r|e>~$tG*pXjrCa=+4_lOq&azHG{=0Xz zUTC?0CqE>+Sk-eQJMy;70v)(TmQfjQTus~kly6v3bBgI$Y;L`y^n0D=*?iVFOba(F zktVXCp#JWxHO_sm;5;jOXSr3)sqRZ2(E;=x?d;6e1uE=r(NK#Tv4p2l)oq=<34ZOM zZx~(`-_^X;ZTgrQtF=qvm%K>pi}uZ=MkaBus%|#ix#M|YQ~qZ>Vj?dxwZIOGXiVkZ z*MS@4e{Xkpry>6z^|p4u$p4S=G!+#dJuAomI~@P!)=%dT?q2kVR1zztk7w)M;=u1F z$=ry8nJbxB-|@AF#56X1Eecr+x$3D)K6gvl%=|A^OnrmPeJFu9>3{v*?HT{??Jx8H zqdb+z&Jy;9Rf0@zQ3W}!DQ>E}L;`);5hkE6)x-J?SGSWn^%Rk1>G`2d%gCx5ja9nV z{2ksrDv4^p*DCzRIWIEA%3q!YwG;K*;iX4!Mr2mmnLahEe|cJ-ru;WdZ(>-5U#9E2 z!$NR_{NL+sZ8h!xz5dP@`TsGVhW^)D$e1jXCcQxe-+ z>NSb837JWLpv!5qLX(K^-v{dLq6CPQk)hQi_$E9LuIhN&@BTc5ZL0kHN4V+L_+bfM z^5VY%jqe55InfQRu$&oE-?`pgyzZ$w({QO$wj}y||K7LJU8Ul%;+|g1*BMnAukSX^ z*2SH3IB7LtIn7)Yo1Mi#354pLqFG(J99omaqsAm^o3u`UUj(=F!jWyA4SkfmSS~rI zxaEOt%gR=d6R1tIs@lvlxrpa))19TeB1t}`@^}f%oKUe3O57$!E0MTaxpf`Ck>WRW z&*qTs)Q<#j-D;L|$wM9w;PmOScc*bz#b;OfQpYi5u9q-7&xl?vrw5?m0IwzVWv-la zA)Mf;^lPPziCJ{qua(kZRJU8eRwCowJKWZz0J#s^{QsjoOYgZp?g>t*nOVKeQOC5lQ#w_-E3>}X zX7!l1MK*8jlttrha#eQ`Yc8>lGS(8IHG{Bok^bDp;ugnMNK3V)Ud8QjNn+eH$K1d7 z=M3+(c=jzZYzw{S8tUK@b#55Z8(mI!KFl&bPyXBMa6Ur*r+$B@CI8c-FY^CmJoi1D zE#KUAA!sr38P;qyxEQZc>KAEFNi_9o8sV)rqI7|7Sh`CFpEZ>%EOKovDfRTN|4KV; zF}`}PF2Aa^weF@jZtFlLi*Arkb86HU9&<;Z+PuBeIh^ieDf-l0{^gnX%#r_QIr=F2 z|8{TZ(Tx1x`f~s0M|l?df80V4oKhkxhfyYg*8z0DT4WL>Laq-b>&CFf~ zPgpKUJWr)n>|o2(Y;$wU>KLR`mQY@lYg6szqLx}}&m2VHpq|W9P3M~^!-p0m(Y5gO zX6Zxjl|J-raL4k~O^eTBRxBI8*E%g)c8fPn3=>G7$TsV~n5 zdS>)L=Um~w0>BpQe_K0SJG;&JubrJQ_TP{4p!*JqE@&>9ppiXT6z)wgcu)%KSzLtY z(5VUs_Bv?iYM%CMU^beIVLg8Q#4z2-|HP<)(s1!=WeZBiBb|*Xp=lYu1pT{r<;6VPvu=C~W7^CT$a5FreaDuvzQn#`=~#H@q<@9T zd$+02BJRFCEohIrpN-eqpXcf`r_3^8!W!utqY)R(tjEwuby-JR&lb1L+8R79gyf0h zded}!XH4CCKisCAPUyMaXkUaA8Wl0kE7wk~d^xY{QG(4HodrW;X8V4KIo=yh%zUe{ zL3`CjVdeS`u%|H*qTbTXr)Ktq*_w+wb(_%r+eGTG7MQu!ZCtApVQVf#&_8^76*@n!o$yqdlMFiaib?2o z;Pl}2%l`%ca=u?qSWIE?zaZkZ_ixDsslD)j`DO6%$28{Q`N362bEZ}_694q_DANoy zTUgMiKjgf~Uebi;Qw*{Gp67p~B%3Mxl3cysfBAh*LQI3zZxj#{etANrWa(H8E|U7! zw1kI95%lBHXE|p=oOjO#Pm$DT8yH04rBs};}xV|>`Fl^Tc zqP&j*W|N03nL4C-jIAHdtj;mE{jh9K5e0T`MiC3FJ%HLhl%I+9cA}>1B%S9hH$Pg+ zQO;^!p?3=?$Ct-N99zY121@m@RxxH5G^Iiu=X^-raB_N8#r*4MW@_86IuqQ!@@Y8p z&fbC2eVeJV!9T}2&A|U1PKpdoB++w+s!$0$SC^`xTxTF9z!`}#+2IKdc^ZiUZ1-Ft zqir1J$K6I-bw|y)FdgdF&f!*!+f_G+LT(!35y2`(O47`1w}PYaM)0Ahw?+lVt4 zaJ`_@!QJKiCyFqnLYz)=DkeN`pwj%h0GY*rT06QBv5`(_F$Z4r>jK0U1DcCi<7G}s z#9F7Q46_p+o>N($l8)tZ1vaiX)5^z7@?Znf0w!F*Zw(*~hFhLh8gLf#%EGa2Kw31o zH?4_{9A1uN(Y#$wdnlb-cid%Zd8tFdNH^Q{WLxP{_>YM$49SE z9YS3w7g0I;=?7>J4_{^)eZK$w^Zhq}JJ>&db98)qGCN4l7@<^B`yjh_ zY8e*|@96vQPYzB`-kcsCA3lBa`r!D*;nV%o!=qPkzTba(di1)rnTB%QcAO4vr6Azj=D}>ea#17EX4tpSj9#4$AFvVU=FQ zd>lk97mJX8arDERXNRxb2et~s?Yg(%XEYZml0zDlyPZm};p3BHh_9CKnOQ*2>2wnw zAOh~vG^c|topO-Y{lm!Pb2@czL=WtZ=tY*%ybVj22nL;%MrqMHP^bU>NOrph^;&KnMsvTxlM#cPV3!SHb~HgH4~*o|^rX2)Yz z$Z4R@>9j@ARk28J(~Uj>oKteB zPw)`>8ylUCw!X~Us?YCWOPuQrIrpFabnyE0@Z{jxn-_;Crw6YNUY~q({9^y`)#<@6 zr-N=+Q~B^%=U3LgTjwl|4h3|CR9xLze6o#qirT5YiF5nL;^x}7HY&_mp>ES1#Wa{A zV7W8mpsI!H==(rign{wl4lqyTXF{DOkI*X`YQwD3D&m5svK#U6oaWuUNS8GHDy2DM zdArruwPLqwN^^wptrqU37Hy|obETeXe)*MsE~h-O`55M|yLD={^`YCkA`7ip&RMT> zN}{7Qo;r7l-M|d(Gw5UWaXHj)t%Eeu>$X_OUG|)N9@yk{nq+eNjO7D(_uiY+^evAb zTh3EjQ~tJS*|YcGy_y-3w0Z2kY{hmlyYgLvb4ThX`Ux5qcxcX3Nw1`PtY7=kX2ogN zZLkC=FiN;(G9tyt>08W3-uZ|6I0SWVJT{&Kjb8&$8)XG3roWL zHt*VbGe}Rqr!irP%~fukeVeJbXJ?t4LYtrs`X!I>8bj#3rX>0~XObSJA?<)o_Ust) zEZNLeWVO7{Lh>9B4(F^G#BP#YD=E+shuSM2R}{!U#WN%|NjQ>0(hgZS$`AqR1^t)ljS@rLb{S~@D1@_ zTfOG}4_n(?y{#|)zaQgijsHF7(VROMX7YV0PSJN@n+y4U!zsSululbHZ)ZZ|BoLGC zg!A*x01%2f=qvqk>#kJJYSxZ~X&2!ezl`PGGCXKn3Q8-14}NI=u(p9Uc#3^R>jie+ ze9)dcpMXC2zGIH}wK*PNvyq>2>C-DFq;NNA zWmgC!R0uM*TJOt)lau`)4!-G}Mc7br)~VM!HnqxCOQuc#133n7uyRS zm%YltH~uw#%!IsArNf==s+6mDU4LwLVU+V^&d^$v4Om2(b17s_GGF_FE$`uds}A_S zLeC%4F-udHj#Yhq((We$wtBrD_Q0GAV*!mwa*qy$Vx!gD1^ z0?!;snBxTjVnX8>#xWo2a2b}4HzB7nk?ev(rgo#%dy+=p8l-fr4r z4xb$eFUWHKmWI*`SVT!zm&^Ive@F4D=l{{quMS?He5f{Vn*Y7MX8xakf9Fg7pO5m8 zEX(-?>exOEY5JCpc^nOCKK62Y!RTe{r%#Rbm*>77J^x=H>_2;X5G2tDX=CC1?{D>Y zdd>VV{hdc&;(tHNvj+Pa3ClzHztK2>rxTJ(&$G@4Bn6gaF$S4hM*>sa3EYJdFVYAm zG^Yd4gRk38?f=1JN>b;<-b7@0Yic`5;WhoeV1l8I*83Wsm*t=zY->FMlYhbf@u2`~ z=p|`XdOVDHZtao~#zY7z6r|&y$`U+a$1x=t6Ad369}y9LS<}L_ z$oiO(8Z1gx6J0F;6dK;)$c!w3h(;_Vl15b;ll7d2JW15#iZ)b#Qb@=-1ySVM4p%wP z8Y`Eg%LnbE`Wng@j4dt|CLrL3&gC+E)%=i(>Dx317J<~JRa zWqFgOz@&gPM*^V$I)mZVvPA%=B}auMg&~CyAWp{fI??gGTS_E&sXuJGxedqMTuyM{Bg(4Vb5LR$7d zwdV)=I6#ZV&+EB{v(o*ErC~SBtjt~oNJ*(vGG49b97;(tUb!nD__>C&-Jqw`ue$&# zDOH8mE(dUyQdtql<}Td3pmhJwHJlxeUU7MxQ$bUCJAxslY~0wbcuKqTid(h*k)CTf z+rAS&fu&T)lZ>Y;QrlAM{IMu=55YP8%-9K^YdBk7F3kYZJY_s&ZbK9?@5)q2sfNecYI3SNJ1?En@EyInliqTl&;3!V8Su6gP)jL&;tYTP=_O1k z$$*=nxzFgihBN<{{wOt*bZZ%)t&}?E5gfbFf~CxPwE#}MD9obdXE~(_ioWieeTfFtRXTZZl(LDv zR@6M7(Gg3jusOpN8z=>Vq>}m0yAjVJEx5dE83lrlQYnm`Z6K5_?}k?cSx6 zaQ5qO_d=<*+dOUrTv5us1H=-^f-J8lS&CrR=If_aUh@2$rDOZ9yJkW9rDgaNQmVQq zg zs;>mqS9bzZqtr~ws|ECOD22%o&}$-!XoM*b^bN>5*McGf$#Zr=6(O~vmCI@sD2b)? z)(=Ff?SAh`?;a_o@2y|fKJ`rBa?esjDn$?*S8k;HOuxm)upP{zdR7ahztuFXQ%X*= z6>JISMi@AqW*<}?y7O}lXO?eoFRL0nWv*{uZioELlt!sdQ<>Xc9wp(#9M4Cv4>=8a z{!t83b5XtbwQsfE%zl}ou60AxlIt1y8o~c4v)*IrfEju1t$?$ zKmj~M=L-Q`8}+iBU$BVgm;uFJEHKlzUw=EBi<^7bd>fV;x5Hx{8ZJOih{QBKn0QBL z>Cy4Zh)@IFn@CpTR=+l2$)UJ*6)L^M(F>vE)+35XpNDb%{y40>lVG(oEpv~-pNLA& zn8*t>GJaP?W4e@8LEH3zKvfxZ?S>=X@n=G^dz4&GX0oLLYUxm=B87wWg5^9#cgUYe z&XhamA54kys0Lf=|Mh& zUuytB=)C>E{i;m@`lI*_c$v~|#h%{vK}vN>ub#lT1_vhph@o0%0H_Eoq?tVi&Z%7S zNb4Goi4Z{2DcFo5W(7Mh#Y+GON;@)nj>@w-BGg_4-C=<`taD*lXU1jF00>D6Bo-*l z@KDe!L?&Sz3w1mg8pvIadd~}Y)^=zzke^lLw7{J+IsthE5ee;(x<=|h>V0}N1 zP5!-*2)beCpgv2eXXRaXpWtjFw}I@cs@ z0+nH~0dkV_VmtwYS7MN{q6ABYq$EP4GYA&ZkSubeygzw!b3XBwX0G9t-r4ezVg*dm zGNdD(V|RyC?S(ZY`EkyG=RenQw%z*>l&a-RpWDDS{Ec4%A4!_RI!i6J&|Z?!TuCLX zK@LChhVL5A`X7W+^`!9gT4>`bGLkg6HUmo~Oh+$(7k9LZ-2S-V!1=LR8H|~!lU#Vjj3EhV{Za6`?%(%lgsSmP9!mAZ6SH4 zH@GatK#@46G4LFYa+FH1)m+0P1%Ej8c}jx~fTtwIbK-D<`S3J*>5EBC*wcLx`Wnt$ zMDE~i*OHZP8A|;q)OPpyr>90SP%tnt-K{YBfY1G%Pc@U@EqOY8!u~+dC?t+f^>-&+ zKG>tyz8~1V_>i^xUf$F^X3*6;u+N)b*KpS5nbd)BU4`f^s5Z+}P6EOkPKQt2=wU^y zAqmfwPNXK}=k(m40be_y#1~6F|LT&G>KYEPpzFyBPc0PG68`@3m%CwiPtX0ysrF2@ zy4MT#efv5izOIRhW{fIcSvZDMG`%QBL;okVH&o>ONYDM*66WXhU3iXeC}pgAWn8bC zhZqK362$SI`XtXaoOL?)MyXZY$4v956AEvWE7$8l%{L*ZF)htbB)6t0eUUaDC^8Ct ztAkRf^mWzNu6JD8r|!{hO6`z$ckj4KE-7_6VUos7$fMy~8lo}qCW+qQsftz{K?s=g z0xn4^9TT7A=v$^KO0@M){W!p!{>z3nui8mi7nH)8763Xzd*y*ZO1U={uW)kkibvF3 zy=|SOLAyAkiT3k(?(tM=&P}wd@l+X@W@6D4@zmtB*4r{1>y9X|{(LY>EvF`2!_O0n zQ7=44fm2N}HJz-SlH8F!cSQEK&;9vkv|YNrgl|SozJ!gC?9b`BhBLuqA$ZztUQO-4 zO|E=uI`OTTkgeUlIrg}Dl)5p9{RT>XPQmNfa7MPadwW~kPxiuSmyG%(j7E?4cKcg< z+gssQkCMG1eYE%Z@oqEk9YQVcOTKmXYG_qCCx8xa!R{~LsL#W zi8ymiFZ`l53-Js~XZmnT9%V{J;ppD6Q zTxp-pBTtsP#4%^?n4e2gRMGAOKU(peBYu_!(x8-1=)BD0*S7S9HrKYG8BrU*(&P05 zs+BiCA7ts%lv-x-v!U4k28-V{oJAyMH2z(@HH)7eyA{Ik=2Pm1#qUN+eLjodHJn}a zgRS6B$1`&%wZ!7L#L}lp`RQf)2a)KSQp+rUH!ppE0zxgb_{~hd&v)^=hO;Nzy~iJ7 z>04%pywv`kg&35wJJwp4S-O2y9ek zZETa&n}BY+_AVtL6^rhb-LH_xi(UzM3S-GDSV$LdK9NGZvJG7Y5U zmH2XEXNg9ErK=_RbAc>LDdSP`sv_7moKRV%ST!M@Wwu#-S&>A$Hh-e$ zqwRxoj}5n=Qi!l@hqG>Q|g`LlRTI~ zcNuX%mr~Wc*G1gs0Au8oyDdJh)v445CE-7%2>Y-+)u7bvq3C?AY-{W16>XqK#9XrJ|JeO0^w_s^h zaW}JmkYsX-DH(3z9D>Peesgm8&w~MU`ag7T zi_hai&E3NM^-c$a$JlAlxbw57V6{YY{C0Ht1yi2LiJWKrjWr(j` zWxV|C`|4rM~10(TFjy8-|poHcwpOc|QDl5>9AB{G5)Nkookt^_J@_ ztF@x5Z-?v3oYHPW)CA~OGf~uGZGaE34`qrNFxN?I(P3nPKqs*rqw!-hr8#yAX>4ah z0Ee~WuMbX6M@6i&y^e@6n+t7<#l=wDWDw^5Mb1GZCiFnf6%f3TGZI2clrmSP&AVF5 zSm=4L@%~_J*j{cJPs_wS=m}m$kQs{^;0-0X;^wKn8@$X)XR?~Hv>qc7m?>6eDb;hR zvQg@0IOLcRQ1{Kv9V|A%W-v82l>iP?IOmx#*IHybyC9Nos#(ZmKAtLeG-ixVE$2p* z$U+q@hA|Tps&7uUsdsZ;q>-QVAxkk$q0U9<9MM)6>!l=N>Da^3p*qP3Hc*<3fOQ&-0~iva*k+@V&~8qWP#p<{ zIq?F0>*Kq35WF_B2rp51|2|Ofe$u?tNMH)@-D9Zb8%&7#o6NgdCBMqoA6PRhrqH zLXooH^*NkQmszQ&vzaS(Vvmi%gCJ{P9UHJvvqfODf5&R@m8XWM6+b^iC-ajj_0 zGn$%wxsC1);0&|do^9$s`u2~lvjFz(LO}&JIi0kGP>hL^0T~}<9!aM#;R02k3IR8q zZb(B?2v;hqDX#uODwY&RaEe=|$lIReYRT_beboG`sA@7u3dIV@%z7G_o@@l3r$oV= zWX?qDhMNpvlUQSlW>H|qFa4&>*UWG|#*EF9Uy?lX6dD8%0W^}j2{VdjOcNr*2?_!! zKrOLq@wlK)E?6zPa7HS^S9H`<2@jGyBc$;TfKV2=Mre{KnT)HulM=CoC1GDDIvcYK znpW9cGz{RWT60qBVb_tWL;}HNFPpVEQ;nx0;|0y9RilubIcEF(}SuCI!Q>09fELj#SXl0zXiYlsJGeQfrsYbZ@-21ynd)S zudV6-!-*kxr_A^q1!a!s4&EIOB-80sC5KL#(0|bFK6rP0^z6;y@%zD(-jl7(&L+?l z@BUS5mr1>XVhIJLY4C6~hxklj4;4QkA=l>MG=OVge+M}D<@Dh7tNj;)u2!alT6*Zg zgY|WkJ@617ynDX?{qy}dr7VBHv7rWMK^qU@?k(KhXLqboxZ33nmHE7&db}1P3hl)3 zAFQvpA=T&tW@*1GqR8yG!PNQl!Mhhn`_JBdxBp`Q)l-G+@%sT1*C6*}s!zOe1&FHi zUkd_Wlxl~$;@hy1MIx{oQD6%M3CU^^-~<%Gs|^c+l*Gz8_WmN?QIM^*tS^{)zFz*TT0;0tG88^>n|$T zugdinmFs!XGq?{{Aiy#B64mNH_!;!PM2vfVv#wkI&2kwDR;0t*C`T ztv|I{GiO((R-2fU2!^&``y`CmXoO0o!T|UAJThttejiRz-!a=oIbSicwy@GlWSc1y zvo+g6Mm_*7^SS0 zMQumbwktG-j(J*(>Ycs;MHblt9j3-`wY_>dAVw4UoRl1YM#*_--!zq>hU%l26q_7> zNF@pC2L@_uA|wgVeY^4Z)nxPSWE((7;oMP>qEeaKe!SW5!9(-!x8FMb#}*H*EB6wk zP`fxJN(Cyp)$*xWJTE8=Y0NJJ4Ia&G+WLhy3wu#Q`wUgP7cFfA_(x@BkrSFiLNfJ- z23vwL$y=(FfwhJ7@YtyC!)ckXzU&Q4wRyo_$3yc#4PV7vB~LW?4a#9s13Z9I@tB!- z1JG-O0*kui0BQmnftwG;N)?1WQLySa3Jl=a-_%={GHDFP#=zW(FL^xb8iVbdh-NXL z(!66->Fv#aU#Zgg?+vQ-EmX5lp+(+OrCv#kyoEwN7rryFH68VTx*GUk_7q`EC5VgA zd$s@aV9uG^=|7I$PfuU`cyfC1+CMzjzimV1i{qo$r|TzkPVM)9IPi4Kt&chHwxIgG zVEak1wbKu_`i}tMFDXyyU!C%5@Vci@k6VFlCEHJgZq4ucC4*N_jkT-Rwt}5q^?7?Y z*y{Cyt?kD_|Iv;rpqd+O_xioZvyHl~^;c-EA3W*V)^>J+ot>b+w*^jXyN_GC=(e`C zLTlSWf3H{dt-tdmc-*VCw*6#rYvr|t3&!mgPiwA1w;#7Z+MKcGaLJS4QLnYNt^VWw zYOT4wT(xhuwax9vkC9I{x6B`9Yo1!+$`#XCmj{+rUHMd%YH;J}=?mEF2^w0Kn_sKq zs%E9Fg-sdes4966Agf$@Yj}Fchp&EU`vB{8#5V@8yQgzf4ySm!)Us)+gU~X1<-KVe zkY)C!b*hvL#LV8bPV#2CYH!+x$4z_Fb^_qubbL-vsgTvaaZB$AaEE9FUs-Vlw;(T zkER^k);^AMT(`#_P*mP>DwEAqmf^y+kCNi6;qT z!V&@N#)m24LgMK??EieS;e8Eyl~V$$EW$2`4n*`M4YL5IvB!!}(gR9u9^RYI`DyG>&=B+Y8YaH}82TlB5yIqgt@8 zsQL9&VccdQ%`t=ZHZ$&-+Vc!=ek&P3r|QPY;kKxKPy1#nMccS9y7v{1id5g!7c-1E zJf}bk$rB=3h~6Qn>^IU+sS)3gX)ZzEO_DbpG{88yk(_#{RBH~yi>S*elR1e}AkzWl zf@<~V*D@$N<3f?B-8(eoIpw13HdFWL!vKsl;Q!1wE+n(?}tH3PM*-+(7CQ!(bwlctdlOo9|H{A1352M_)faGHxgS_xX=*?pT_y|J02tY1_uFD1@von1eMtjICr zuoJso-qKs%*Iv37tDz`_*<6xz3V$aT1V~nf+6M5Q@%=L1mz%((J)0GkgA^aRbisza zgiP^>Z-|vGCkAlJVM60fF=S4w!BkvVfzT%>EJp|H3zACof#xYqrGQJSMQkY$AuROO z0%gi{IBUVh1Z{@)Ik3Z|9N8Vh?s^~9*n1|>0+D5O}>pj}t#eaLfUi07H_M@HhZ+ze1 z-s|oC2iz+X^PWQKg#Xd&-4#x=bN?n!gtynzG{oZu8zNPO27#mIv_(n!TCVt|EV0q$ z!I<-LOi9MHy5Rv_*x;_Mpda+Sh>fX`1JGIG2czBIus7Tt?e<1HPqrVE-R-?6;VvaR zkDm0ldt_(y_z`{bgwSyK=+W*L8SZTFj2`Why>R=%=Ksfd zZl3>*S$m%XA>^rErmgSjOoo`vnKxf%?Y%yF{u|}6Bz70mEZmnQxMBYHdyjS-^M7|| z>&yKAC{H8inq*n|b351#cD;y-FlU*brB7)pa}u*ZXjF6KO$bA`U9~T&$9X&Uj7<~R zxaomW)K&3_37NBDp?0cbnue2{r|b_?L2cKnq8M6e+ZBN~;YwCn5#;2Js4Qq63we4& z89Z@xnE#DPNE`=aHu98ME9von_k(^f==nX*b{7rBKG1})cmOd8CoxF_dO=gc({B?N z=3MZRH12|0w5n2!KHjeC@k^SD^KZ3ytd%|Im)oiEnljmBgi=kn?6h>xs_wLxa(9M2 zRU=TdN;wXn4rRQ?TJWiv_setVXO8@L!mIDz#$x%uwY9h1Ys&wBR-@E@$d!ma0u*M8qRMjBT!P^%+F0}b9Dw`64 z=Si9;EX4)&hn$4;nC6T}C;B%30rVgDdf3-$#V)V+KWSfca=z)C zc&f6fAyvi82&X_1h>VS*)jVgPg(%5*5vZ)pQa1oMy&DFQAOK887v@aToRRsUJfmS% z1Lz#-uc5J3P~p_&DG=)uR=TXg;jzNX^?ZYO?=dbUA2OK}42VyC7SU9)(NrmTiF<%w z!MCn*=pd#9*3TzpzL45!P0fkJb-^-KOaClCN<~28Y|?bSn{Xi;>Q*r0?FMU$y!XhZ zsJrX=N(~hLA3UY*2bPZg(g(3}=vx)IrMCkE4`H=Coz4Pe(-&d5Gdf(TW^7zwR^^aD za+ZmSav7M4lH_A5wKt&o>;TYTHJmvBtS8n0(66J>xcxd?Tg~dfS>KY957yQ^>u@rd z`s@9d8#)NW(BXvJJu~_(3w@BK z3de#2OT(NJC0)Y_!EAwAGl#rLm6)Yf+FFxWEoM^26k(UBZ0bB8M8HdyG9^IDL&}Jk zH(lGFIMs^Ep|h5dML|ce>rIouM}1FJ11Sxu5G0>&ngmk`H!04-B9EsqRQiDcGLkfh zud3nuDlkBmt~0>DA{F`X+cq?ofSD$rl04Y-1r@n}DlY~MDc=~H&y?62!g%8)IdZCKyedQ1S334oHM!*|;*!prR{Pn8_ww zo|(;vuert>QGU41rwHG+pM`Duj^bFBFxC2wGSl}3_@hh`d}dIg5ReRViG=~`SVlBN z8+cSf9-)#4L%haAQ7rcVY_V%dYw;{Jno6@$v8By$MiVjxG)pEV_WxwY=h_;aASxpI zP&|xrqP-(7bQmB9I{a2Lf1ifnLX;{`K!(QWn_^OQ3kL#;W5~NpiEwe5g(^fD z!y^M241o;>SocHiT_D1x6P02&fOe({Efxt*ae<*2r-|T+F_8ofU)k1#2znL z6C60`GasO67%=)}BvA^muO`qA43FF}d`q zPeQ3Y#SQ*Y#T#BX+$=YM#Hyii76dOKB!$NK5CmWL1w`D{uE9*6#hNK@f=VT34N%%d$?_ecoU3<_3HY^HW+Ni<8u$Vo}#-(k_7)&hUW%&;N7R*lo(1wQl%9?W zlk82Xka(<)LiEO#T$+8yCV9PeDszlrug6XCGv8bP{de!%jW^%DS^xd_t8X_dR;t5= zU`SQdS+!Al=!W?@5_VfEQs*)(4V+72;iU1Y%?TDOR*Xmr3Cz ziqh>DE|{*Pl)Btv`qfr_OTWu3j(KU|U*Zr`dJ>a?mS9X$yv0e1vxEfDn?1aJS;k}F zqoA~x08wsVcw31-ksVsU*ijWm$PZ>Q?oVP&CnWHiu&bCk<66Ra<4X1+&;L*AP&ON6 zQbXX*C#5XC`+6T_4`GxS%c7OUcs;RTL) z&A9fFP>&>(q;<9KtK}#G|*Rqc#eM*p^?gwwB^IKk8j{ z^%7D|HGSziuG6@Wo*o~vg;tY0Ea>(rPP$<)zexO2nV`F{OftAz?q4ez3@W~s>GTlt z;VbOcgG-6EwR~FQyweR|9qMZd3$yiGMpi|uoKUfN;W5KdzEvF`OwQ!gVPCDuFaYkV1=X6zK<){B{!*XhOrtZQZhh*Tb#=vDg5{7n;%C>&xDR1n1?pfOu_$jTp7 zlFYJuVG8RXPCf`e)1*2QQWz1((3?#r#l&gQF^?g)*%hKn)Y!kgP$?vA&!<2KNy zSEC7IQ~Av?eMQ4q9T@vj%<5k|e7UtD2VT0-sWxG<8#{705yvlp$C|FNHQz&knm zF%FR2*vVdXTDlC&`%`7sUjsj^{dpm5Ux!ZC4mWo!VA$L5`ad+G!UuXq6zw1M+W$K2 z-@m}}calZ+h5~SB}AMdvhyT_fg&M_Or`$~pe zAIX3=b2Dg>r=>@RwRqFna@=|fa1xVojHXi@Q-1AqZ8$k&Q;LeDF#y9!Oo+0cN>m8q zap(6j@&Z4^hK3g!+st3f2Ba{yrcr?9nSxtl4~koqDxxC@HDg09%^jqRW-1G-)EeA2 zl=UJMGd&gfbeh^Yd2=zweDSF))nj~#<41F?T;Ttg>Y3wizteM)>*W4ztn&YMxAXpg zFWpA{$^Y*$p8f=d7u--}k|vvS@h9!x!(*@-nJ$z|40r1@eY`Dk*EOvyqiKzJumqGW z_rzxMK}!A_&q5X(k{@WogAqtyyMT}j9vy&9*-xMJJy>J2$QjLofS*l*12;Ez-MJl{ z_AN0IuqfXXzL0Z-TH-(8=H|ui(7JzB^N!=BIotRt){M$Sc%eR(E7YypziqQ32bWF| z$C}nV_v!anL3{u1asKp_KP%<`*#rj>)6GeAUsv!I^1rdu*vZQOM!nv6^8bF6=hLU! zbGY=UO@0VD@&laABix!YwhSk@3D0XcHx`4kI@ghkdarW+>7WIUCE;up2+oi%LbllH zd%mJ}#2qt!zNzF5gH+M`;4>ugEF8klCjZl)_GY7ze+?BoMTSF<|1Buo6B+c8Zi9O! z!@p-J5J4+Y6mi-e>t`&mbNGa{!TLdUS|>`TncPmi$0HvHo;b1Ro=aj)#C^TiW%Tm> zk0jXo9FLd2Iyy%=W7hk+9>sa?X{RCp_%_#@3v^-}Xx!7K#0 z71INnVyf~EG+Vd-t_Gc&7XnK*8=9syQ+j6tqu39V5mcU0`x&j|u!(l>GSU|Pof(0~ zr)5({UYlZ{m49A2Rs@uYBC_lgbGXXF4w?KGf6FUt@R^^4p&^FKw-u_u9S5B?m;PkJo&#;|4VnS zkJD>?;Bwb%VXN1Fx%>Z(Mq}sc{_mqaIfbFUYMhVLYjx!wZf?qRGw<=SafG6h49#W9 zEzLn>WsmMr6g3yEPtvuC*r{QIbs?}6mKs`nm6j010VX*16CQKegi!YrZf-2qFEJCG z`7}_WgL#Tb89%x4(y2Rj$2FHvIJ|V^7O!fp)FCL7=+pcFvdNN4x_iR6XB7qx#^et9 z6-@*941VabmEn%ULup{{sg<}xtZJfA1q`9~s50&8x&3on`7hjg?{^`1x%_wCdVc?} z;kr-jzsGrO+rF1`}XXx^S*N^K=se2T{$@ZzH5T+xBKn0{R4?q z{c75k8slsMlA+2%%Mw+i7Lc)7Yp;9!eOg2PV$e)vWgx}5A5+5;I4w}T)xf>Z>HE&9 z0c;dB?Z)ozt^wRR?CkXqy2l21rl@IeW-I73Isu#>+1ZVHy>0-eho}P5I8?P=6EHn& zl;oy|kM_0!_d}kLO@1$)?iyY=JQ$!YhbbJ{=X z^m5Pv8E<`bqL@TDPJB#NwI8}YHq+H@tPNSb-#uy{9JkI@IoRI0BC6Zn`~Gc9HlHcv zv&v{yzWnk1(f5b#x4l-y{$+YyHAfNHzf5VSq@rX?`{d-u_NkIWRVI>Z&p)DAk+^r* z?)6UE{U2JD>L>R0rWW{v+Vf`hlc;*r%*^ncrbg|L;KK)~vc5y>e_-8tzG1I_sN1jp z_;#ZTfBf;9XC>6fwE)L$KJ)(7?)`9f+Bs_X_kQ@`{&8bKbWiv50F4a5(XVF*`>pEw zdiQkyTX#eC0;ewhTAWwDM}8po2=R3iqhSIh_Iw`XIwn)lABFe8bW_IHegew>Z{(}~ zPA@z^20H?Cq%C|W<)^7&s+@Xw^9HJN)LQ22ia6n6D;n|rQ#o1Tm9hglQDU$mldg_V_3JImsV8MjKr|M4t~ph4gPn$)zi+&;lW<&=!PVuB*0FhTwJqA?}RHL;|Igz zEfgSkKr-?J+{zT)8O0P`Gc45q6So@MJ4gSz4~796Ud$pt^ze1dt=|a)Q!@BL7@{#{ zxpKzKZ^;{BtFR$>C`RD#)la2E^XAsbP{hNP#*Q4lE#r!kfY=j5v@HSejoHy|+*z^< zQ{eu@I{}(A0bJ-gu?eIx^_&b!UyKkn(lN}<^jTd%z2BN_05lG^!0!vRW$lbYUM{*erBzp<#=Pwu zrW51FZlO+ko!8_%=e>$bls?(`L}s~7as`ne;SutKJRF`4xof!%Rqpa^2)bo{(K$44 zGQR<6fN=!w_Ul}^I8z6)N6=b%!X|{VU~N^l{92M5qi~GXgyLuO;yO#7IJtcIrqz1G zxeYfr-xSD_5DZ2;xAC_95qJbs9vBZ_`!sP1mH)ih3w(!8z?K)a?$s@xbXU>`jW_@8 z!ke0hFKgi}2;eh}V;sRZ!{4rb@0(ovpFcko36g<|ib`cEpBU5QsbUmo4*d$avy|o{ ztp&LN@^+K)(->|Sc>V&aBGg&4C59gkFN-rmt9n)iKFe@chOeR{d=(3$mE#sQMKhn_ z-!qaQ*QI2qx2`P4W#I**q^%6#sGVZm2K;UQc#r+8wErXGrSYT(JaqK~`~THnZg1xO z|F(AAC;R^~9@9h@CT{K09pD5hL;a)}y8QpDM~JfBs1ez+5Yh6EJ-d~CStKPDV%oMV zey3ETH5NMHt(^bT)X_fL_t*qj<^S8*+0Ofa?YK|(KOf~O_5G4F;r8=9b>RPJ&GX#K zN{?}4gP~-{!C0A;)-c>`O+WwN!%8|Mw{qUGO12DDsQo6-oS#N>FIZanHHk>QEqk7~ zveFeIkqqYbolMlB6v~YEUPPgz5ot6!fP1!6c~Ne(+eMh*VbT9q@vS_7gHC$)jMpZI8!CO!egc2ud;ZbEt?H7tx@oYEq2NP zxm(Xl8@yllay>O&Ay*_RH)haM$g(|MGthk2d`RQZXB-&wHA*}r3+Wb%6D9|e&Y0c5 z+oiMM)0qa%2mSp0j-7>FcIoF??)=HHim~hz9LIzeb!%_2QU|hoox@!;!pF6CL1D`w>6oMJ^`K659z$N-h5QEAjOFu7zZo7uPdlq%bqx5d0F zjm-N+*;T|-o;v2QV-64J@(pWznhUQTpSce#z4c+1L?tLGVX>pwDtgxqeRD5fg>6tbQ zEr3YeTC!=df_1lZu}w$L0xw&muRM9{5gS{ye5f*$GDp6dPqy&-K3`ZHjEZf#g)-jV zdpuMNu@P5(DWj_ftVv#N!8c^GVUX4=g<$fm8wPWoXt&tW!WUoiVnMOufDoojzGYV0 zTGky-oU<+ELQblu#JtI(Tw&~#0pHE^DZ{YnhRfp9{yQ81l#RmM86yimXU-fcPWmh{ z8SuqoAg{<*q0PcC#=B4f_ByBU5B54d2**kdSi)b2XQa#>z%Nuo8H+YQiVK=?i8_6B zCHi2xApiXyKr@sCDPAI6NR(tVDyNZM7s*Wn!szgw{Xw!+FHB_#t0Yv4{?>9TU**2i zDvxK8%tGCUZT%!NJAsG0y0Fw zgwVvKGjrQ-6*&PJ+X24B!75sGj9-y~v;-B;Lu!-I7DpksM3~85`sMm$!I@KjOh2Yp zF(T(Yylyc9)!a|R++rzIDlWc~<t!crp4(q_3Y<*tIx z^`OuK8jaZU+z?T!FnKo03(Hz>=k2l%-+O^R*W2*WG7nu-s@^#oblG?N^Z-k|b3uIHo>L za5%)v2+l2V@ia>2`+lqw2^Jd6?=mlHMvS?=0Qu9tI`|dcwhL!DP$b6O$OoQxlTtQ} zGWy@ncwiWmr0`|Z$q+*+E|uUkH*^vb>odNRhN%B%NfS$WBoXqh^^z~Rj{nV*cdV+q zVb(QY$ld%(_TvQlekMgX63<}7T4vQ}iR}&nv#eZ|Rwl<&U|D!x8iehrf{Q_(V&~W18-A&7e z3v#s?QY*%nKE7J``tPRx>G{%U!Tdk%wD*rX&eVH|Hdgq5Zf+Oye{a3qewzP}@~nX$ z4rc?DkU0JPmc{ivvjJt2N+u+H4^u!P_|quh(QSnymuVH#l+Paw&(S~DCrJ`DYqblV zO4ad6%_BpeS!#%*gx1i2^6Usc33+J}lOVux#`l=kHe?cUKQ!~Q?lEwG03UwSX&npA z(`KzU_LIqs2N*O+5smqSS`uSin4W48G8sh==o!KB1>-1@iH zb2w2;vgAEqRx5J}NXQvA3q2eMb7ooj;V4EliD$!P7GvuLa~9P`NZ ztY#iFIXiG{bVjosSj-)GT8Vco6fd#o$_K~?)8`fZr3(=S9vN?W+Mr=N-D;L z7!EvHtw%UjBN7DU$`8j)%X$tv1+TE}_?kN<1cww)2LXl|W&KBg*;sK*Qa+T#FiN>m z(prM(h01(3i?JOM&({0>x$-nPmMx-V)i!_hd8++@r56bRatFACT?e(;|+Uc%4jUi<5Gf@&$MQO zm&*gsaFiv2=h0ZhAy(7CKp?Z!^9#tYSVOvGfD>eoFk(}b^NvO_M&1Hu0UE}HQoegl zbG_kVJ>kcrU!WuR3HeQ}RC4L!)LoPD&G)bY6cI0vc&NV4M>2q`i9ejkB5KrAae3FV zGj`y-#x=itzOe|`iN4j8yU)et5k|uas7*v%pqopO0}+6i&2+{rE>Nsr$qa@}i-?y- zB4jSS*w0M5>~k%x&H+&v;DB5?mUVu9E+g?Sse{Xu=K1-#WgU!QPG(Fth9u!TqyC7EU{?QyK+1)IaQl}p zI7jDX#&wM58}>};tN1pf7|xY+8RO)f4M}FC10z&UVqas>Zt*n3v`^1sbai7@!N9~3 z*a4e#X~GBVxP8=V0vJxhhwWY;_B$ts-CsNVP52pm(2inoU2tDE>#v$yyU?h+yS!%a zq`lW^sF*KhaV^_sf2Wu10<-FK&ZonF&wwcdb?Jy$D>I>+$I9nGVIV`#r`9~`#- zb=UzoYX1W7j*Wu$;bHgZ&OYoP^iJQM^bfknJ%IMxj+K5vIIgwL-D4MWPoaQ4jj1K+ ztf?R=23h*)|9Yaz_730m`khnz;DrC|{L=579=8wq@00Fnf4#R+)=-vO9eoX8PX&kq z9w5r7@m)wl{6-P#?z*-9-pPy2-Q8XOM??O>J7MraKL%#E?!4M{+?TH$cV~M6=H=`W zpV7QSWbkRf(>plr>;vq*J3Z|j_Y2={0XQ0t@DN?g(-yM!5B576oyZUGMbK}i9pco5#k9ag+*}&^ShY4Y(8YhSX z0%lVMI2FPW&%#jHS*fCu=_5oRE{C<^7}I^o+a4l$ea?E27RD{Ch(e3Jq12>ECMe+| zZh&=~d|?{UoQ;9PQWGQ7wM^KGLYs?~n^2#^(?YPpJlF&2WumX{E^SO#yGN@fqp;96 zH`7fMM{5bCc(|I5P>iOSVGw3FLw|1D6en&HsfQEu7|$&@dY<2j%KnQ_~n5epv-n*LZfe#mv^KE@I3` zu#Xeu2W%aNg|C*FC4#ehY35THAXZL7Xa|uWV%T(EIdy^e$M;7WJU^mK1n>PgnV|rV z&~U=w7ocUIqZ{Gy$Nir;p52-IssTTqc3@qJ4oQ7FCC#=3BF&$KWLK!L&d;3_W&U{} zvldCD-y=UDmpImAoCA+!DH9eKemZUfj657W%$CA2v$HPnyqO(1%Jkt`ank8l2it~$ zAJ%kyZfO$NMufls#jh9FZ4~Mk+eS&k zCh=fEuEDnXhi?sLbS{3&WC5*X%$Q)?Kqp`i;TtiTl6Zz+Gh>u}O`i?CI=Il_MLhcV zpBHxZlYav;J10|WRAeCjjxUqY6u;pR&Ufk?+bSX~f7xW_fQmDIL_EClT7|fO%@fy& zf4_NcS$|&G4G``nV$T4_tgK#A=E^eh6-#S!cJA}s970jD`F4f;M2nh?3f4OXY}^fE zIZUSf@CRnmf-fL)e?UZvc5&8pW)Suj`?xjqi)JiHKEH^!GfnbBa*G$oFiL3wL|=q?m$mQO<8 zgBH}UH@CNTcVF&q@8aEheRKN-gm_F6pPR1oUK_MvvtF;)E$ioYfA5FRX|pK>BKA6$ z%-Y~Kb+b1QhkL@Fm;)Um zwifn7rBbN4R?B1o-y7wacybzfMf$loj80Fw#TEjPNVxiOLuZ06F&7{>^n`_@-Ek7> jQw75S`BUmVos>O2PtViy)jj_k00960Q59|)0D=Mlq{8zo literal 0 HcmV?d00001 diff --git a/infra/charts/feast/charts/postgresql-8.6.1.tgz b/infra/charts/feast/charts/postgresql-8.6.1.tgz new file mode 100644 index 0000000000000000000000000000000000000000..c1ee74e8e82c98dbd19d6df0ae72b4dc82179515 GIT binary patch literal 31082 zcmV)PK()UgiwG0|00000|0w_~VMtOiV@ORlOnEsqVl!4SWK%V1T2nbTPgYhoO;>Dc zVQyr3R8em|NM&qo0PMYcciXnID1QIer1k7YS0NM9K0at+{r0EE12wU@$KX1_|Y2#4!IO?Cy;b6P*c) z!vD3}XKQO~>*e$3>ff!ct@6Ly+s}6F-}?Rb)1Bup{uitmjOr)l0P?BUnx3@blJ6o*)bDxn!sCSA+U`QF9Xu>)BpSPQUVxWV6V_$d- zhe(!0Ap}UE0pb`U6r&N2a4djNxWG{hVEx_c{_CT&4Y(p=EbHSa86!?O&=~M%fn!bv zBqU-2C=TRHE>IjG7I4p}Q4$gq`?%A>@rcB@2O}zyur;O;?!j1yg!j7LtE;O{PTn2L zM%@gqvd7DXnCN&F`R_NSGa0<;D;#}n6)BAVu47EnQ9G3h=e`xseSo>9f(^1 zF_Pnbt=~E2alnZGe*?^Aa}i>K#GN6_Dil7n^g9P|PE+P%1%O4=h=_4IPoBu!CdAa?x%>SS7Jb&c>ck!&P z!JCi{Pzd{KAp?of2t$GximTJ#&7yB`DLG`84b9XgRJ~0oCxXiN)6KL11R7bLzzgtzpxE2q^4L!;fVDL^OR!Id z6F@m~PH0H{iT=EghbRpNmy`WuH@rd<{shR-j8C8u5Drg5Brq3GHUWnm1II!YKO7!Y zal$aiv3R1J%dpVvc0Z*9%wjAs?-1GzsL#78$IKh0B*0zO0q?FQ40}l$h9n+&`kiV? zQa}c358SSFPwx_~&V&=-c8D<`#?S=laEcN-1{?YpL@5_AP%2QW+VB}h0tY~1MekDf z@&rjurZ_HMJurdzC*IB0j+fBD%d~^89El#BmoRdP1orG)ckO7`)Oa(xl`m$|tsRb< z3Bghu#8djrJBUe<04bRRt!qL5>?(9Q&t zJqjvWhahTZGCt<0jKeexQT1Y zus7Y9eO+i!ws{y6;pm1g8LD% zDsWE?&L7jO*wVDLxL`=bK&lBG0va1mrQ0w~beN@X-I(M{F%_zX0S@Wa^fKzqY2w#0 zjv;#|`!=+tCe&&-&_=1CZK)s?#;EvY25-32*Uf8yB)r%_8ct-Z!!%R| z|GNaoXJaxHa5X0W7{-VL_kC@_2NSt+eFX&xIIK%VZpp?ZHc=STD;#VXDfcg$LWCx8 zg*1oJ+8^PI;QEyXHnmHop$_jM1x`pD>F;Y z7>1UyMc7g-{=I4vP$)2qrL5ro7zb&H*(Tu5r~_@O0DfwBrqsCFRfht}3MKFJjEBHk zh*25}l7!gH`q6scTHdi-NxhbW?t})@>*vPH-(p6+!k>i#paNZ)WRNOE3&2Q{9&B%I zZMO=@RBv*K*(LEY?E1cxNjC>Lf6sN(9@0&BTjqu7s#2awZX*RW9qO587zGo+p9vRo z!I>7oG})1Qllq{ETP^#5aM>tV97d9`)Rp{lRn4{ujn#_LZktxi@flTumnsCxQWvRa zM+xHmin74GwLc|Ihv*WY>IQiawsS>~Phx+}XiQVCSO^gi(E#^=#+yKg!%bi~q5@x_ zBnc@F7`ciU zkI^MA;KcRZX_9z%e$bQpt&d{K?FJbBQN(WxWCv$?BRS%?bQ`C6qp+u!I!OH5caj1+ zlEqkVti~Z}%d!C%E=Z(uL6p+rw4&GnM3W2Wompy`!KqkIMmMmYavZ>w6lwi@?&M09 z5cMsC&%X6Hrq<{pzQgVyq=RmR6stQu*xf%m=tRMqUJdccL!Nb{dP?b{u{R`)3y-BS ztV=5-rUL)(jnd$wHXrP5#~~JOwEAhcyD>ET=NW1 zjIqZk6`q}o!eCp|lac#y%j3(I{r!QfpNT)+#kaS=hePx?<0Mk)Mtf z>vUSv-vDtyFeD)c5_5r3AlJFW8!+H0A&N?wx)CTFmyKJwU9jk(AfT{(_IcD`+xlE>)r-86Ba- z>f)OG3~&-CUTp0HVtnt&Sog+Mvn~)Kf&m~l;c9R+$ubS zRDhKKs^Eq4>W#5-XjRmjgOcy7F(F@ef)YZOp{O-wH+n|Ia8FQgfW45S00-WX(a2LB zdQ^-t^TZg%N(=?s{o^zwAu6Ntz#vPabea%DqP87UF~+g5T2(E1D(ffK`{K#yVmv?{ z)1~!A^&p^1Q_}vJa;dq*mlvN&c#w>K8K5GSz_q8gACy`yO>y;~U?~^s{g`r*y;g

    o(8HD6xLkJ0(++0(YsX9S0{)EsRG2;062X}l%3G5C zDk{|0iwuEXI3$+JFpUH4ct~gfF~)(yo^~Nht~6rE$7+M3k(0`&v5ynMyAoNFE%RsR{K%XpJ=Hj3{`dk3!Lshun<0^F*}n6~ zSZbx}hX}cAJQS=G?ei&)(!A|yzrYFQk`KDY#UzmuIbF&2_wNIbId zx@)pUPr@Ss&i5)UggpcO38R;iXP|f@tp{SrT=`uT`nYjUM`)tB0{$Z<3wH5pNBNHwGBgzsl89L1qA~Ds1QCuX(>(I^1dP6EHa1mT znZr5IJ0S_>oD9MVC{@+j?lC$)S0YID1pRJ+FS~pk^(9gCUpzbNYuA?AL=a=;Zun?A zBb=xB_w@?x+9(Mj88E~qnJwN`HF)>g_KOV#s%_8#xxUf7ko%`;^687`&o(w?V|oqx zDnwwQd~$$envTXmh9&u@x2?kvhf3s1TW*ZeCD9_NmF>^;@`cpTH*Jg@(3lb|#p*A$ ze+>v?d<7y(hIV5~eq%VIm&TW`-17PKb}TuScPsJ;h`f)!+Ao5aFJD4FUs6L=0(s9O zCu2N>15LCj=0M^RWjK)7=uhu?WrHFd`bu*2Eo7=wz5i4MvdZamqIdPBVWQCmCgKQm z$kEyw91g8Re6tXqq&>y|<-YodDQ`A2+g`h>O1r&jW#_oy!rDcx2j=imi6X zud6Lvy^uB}KQ-ONX&7!oOsBe1yQx69cKdKFHo+fL%Jp_j)>hstW*va_QL-avY4_K& z%`BoAv@rtk=AQ?f@)P{0-s}{}|D+;15{pSKmn-OFH{WsB;m$z?`sH%l3lWLT?zV&c zlMj2@-o4$j&srSdA!XPSvC^Ig7zQ+rgB!~=rG7RLYlhES@2eVy6%?}j1c|X0ltn1j zT9mLT&y3o>P`e4*OB0ac5VI`UsvP*nvn8pTIYDAv>0Wf;;OQWNFqu^IY z1b!DQLB`3SxCcMJAr(4=B+oPnkichxA%RDeQV713W@YFoBr^-vsmC>F%CZ&urn6jg$&UL2%)11yj?#7LwJ->5^`lbswdVHih= z**%s0pBda4ZP^ad{B!_PUVDHL4RC06>uGH}p}||XL_&kz=|TpYg5;8%{j*@`&=ugk z2iGQC$!<0T)XUrFldR^G-woH=&Fz@!_3*)o z$jc&D6_omm+VZlPY1UX(GsPBwlK}gOaqAvVXpo73(w>ZZW6PVL`M%7Arp$zg#aUDr z_{=x zM{e}UjsB9{Xa@h#i(xgq$ZhM96;-k#u)7-=7}kCl<;lWaa2;VKZPubPH#aibr%{rs z@Lc1`QL%d2FMTjX_n}o(?d0ge&hr;XM7{BoRQ}wGS}o&cGI-I0JWys)7P=<2?_;-E zHW=Q4cX2p@!&s{C90ZfgOj>iG{irxld@`n2(>A$rrDh7hX*;3;4pmS=l7y3vj>0uY zq^h#1+-j13xZXcd5yO;8TS2#GHvQC!c4EKGP>l|s(;+K_nkY|#wD$BeTw4Pg8z`ie zxapQvZOi5owKEZ8UwfB^j7G|}snMjHSuX<%0hT714fJASBSVtfFFp9Ioqg6bpjshA zz@L+l_(X+Qs}OK*d7T7ac^~4DV|su2YEWkIazfSS0TO^1&f>}434X%!`10I}QdJ@( z{IT_ZNvnquqzzCSW7Dj{wmJnyX`rWJQ<F5{j<uZ z{Aqx#H}LNEs%C_H`IKSGyE&#k%MR9bT$!C}9f+)Eu;tE-9M|HAcF^2zn{yUnrW2YY zOXrH=N-6a3lnXm{u8VF$q6dlslqd!qN%X3K3bp0k$s0OCpD=Jq>QX?^m5utF;nJ^6v{-$?c2{4 zYg5-MWCU@;&Lt%_1HY^~A$2o3ck}s{Pk;FmLAiTt@Dje5%@HUkkn9}wp?fy zYj?@Bigtuv#w4>-i(8!^i$|q*ew;e}S2}R~cm{Cl4B)4( z#$h&57j~l7WRvm8i}96ROx`6y9a>2=;PPGjW+!o%CSp?4*J^ra}_U`V+gN zx1MZ5_G4TAuxNd<(ax|C6*1+Q#m*b~q+FTa?YFzH58k#*9}8TjZ6meq(M}0(yU>9g z-Nj%cFt6+8&F=d*2eZ35lPD^6BT=B*ZQEU{ilcNZSVz-`>1LfjtdOeZQ*9SGUKO{xliN=e(85iiya*tuXp9UYFpm%%(cFJ@qi4W^Fe3Ga+FLPtOH-a zE&!9_p^F1%+;K&?q<#1GYsLIa7v~g4CXg`; zR+h{>FDONjnH{ZW!O6R`^EamlXBWHsM~BB3C%b27zrH)&fA!tEltsP>Rg>nRp6BJh zu{!M3qL^ojqaiGh!oN9<6C}p3+Fztngq&e!$3wCci(lJtt%8AHJOj@QuulWLQNVF6 z0_tG00p1#-S?GC4h?$8Gi3CG}nO&V}N-QleX1nF2M{S2D%Qajo#{2AfECSCHVgeqK zzp&)n;?ayMJWrkJX=ti@?-fSt8HFqpK#56C zA5AqhI;M6vg-cDL(3T+4-vO5_0 zs#9APh4X4!T9mm}?LyVNAgSOsbDwFUaPnko74u#uXR;GDZF@SPo#D~C##CUtgv4QZ z$M5zJE)GvhY)(ZjG+e_)=0yaK_{Xi^J6s@<@(v0DCUp??&iwe=-k1X68bwgafPq#EKacD%x_rjfec|?=S(b-w_(b%_y)k&uMT5sS92yq2=0pi&Y&MW zPod(w3>Ob*m1`PP-wTB19-f#6whpDV_ z>m&~FXRr&+-DK$yvy;EIp=P%u|If^3`>S1kjj4NC;?7ZI>AQjX@$l(B+cBHNVJ%U} z!FfZDkp7={LJ-?KFFRYEtVlXwMQP$Y{Kp(0$-brJ!5nuom7f3}C`YEQZkj5AOYk6mf;tbia#`N&Vm7BK7F=Q58hA{Iq_T zWubOzug{-_jrmkN$Aexo<2gWWJ>2$9h{$bkS-R20a*9RDw6_x_38M)kiWsu{baV)_ zb`vGqr8JvK7QIhdI|bIjade5<9*q)7MeTpQ+U`8_wl?9vwmUDqtu}P_P=eIOIRx{L zBtX+j=M51SFxg()T(U!B-o4x&V1c&V1^c7a)Bm?{2YDib&Zi&Me)Fa6q86!U1?pX= zEHjF^{s0gMm5$P^6QhC6nAOpzwJ1#RXmY2lG;DCp?%Si^y=tDQVC|$ z*E^_l0<#8S{LFH6X(l=AF6FmRtHc(XM~1(?QvQnaws+1`SHVje#c zwlg4N6F7w{tRnnWE{!M&^_5eKy^^7)=sKh$;&)oRA*VLT`-xCt-Uy_f5GvBQOguz? zPHgC<2`E$9tYZ_(kk6UJ+ROzd*?wTf$4>E*5>wO`&&jj8o}Y=|d=Gy?%5|KJ3sG)XuXgM+9GNrG(1hk1;l!P) z1@YrlRrg~oN^h#tT+EaKTvEC)F-MlTdgaVv8;`w8qt+@|>41(~_X+SyT|d_oG1EVMSu zHEUO7*1f*Xtgi3Jh55KJ?{8tgDaXv}h}79mJ2z^zk(yDw7z8SZH8m&o_3KoPiej_+ znbm4t-O7jvCk|s{0cL2f>_j$_Sl`obGY{n124_1^RT({mrDvldGAps8hS8kRiq}Ns zxytq4&CHP&^2i@m4z@bieC1}tx=6x#^}gGfyYp43xf=SgsH#qZzYxycWcl-OSD9FU zKI*EgHNc#$c$+9RN4?0O8g(`={URtVW|Xf+U3J#^`M9g8*8q94re}t8Cnz&)U_pG9 z`BF1Xn7+Q72D=EVRU8+YPE*I7NC&O<y1>yRJSRH;`Ro6WrcnyD~ROGS94SI8{jNN$WMnT43 z@<`>C0KFsx`P3G|-scA_Yq&pxH-s)tMAf~6B1*cQOxwT#{0`UGrbP0Z?mztC=M4KuZzxF6wmTMl?pw9cz~9cUtdN;+yDZ z=)X`DZdBKvR%aH8Ds}E)nBzos3O0)<{jcrL_OpcoJu|9=HcAp3O}aBH6gMl`B+yoZ zMnbHvr@TTeR%z5ptc0I2>mtl0rRl`$zAQLIWKK|M`S4V>vtkHPpkfStTfDE}>^pV(V3G@S6@@?|b6VNt z@=5GHd635Bj}*g|zHvLt)TQoM)K!is)3;&DCYxSaj8ZZ76g=vh;ySvx@6%0>&%CGd z{GTS12qm2fif(=aaQ6AXt(~o{?b7+b7u(ObAJ6~Y#p9$(p$yhLKa&vi&YO@9psjBk zF71xBH>ykJH&nMbDgv&G_c&+Yr|xntrDUnzGgEvu*JpZgx!w9i;-Hs%>01#NC_n=B zT9Esb8s%UqT*q3>n~Rd9zMZ@cRaTuksV{8~rDVmKK2p`2I;HQpSX99=W<;ojmTt*2 zbD5}9SdZRXFO6navF~POJG*8x39f;}0geUi)NS+Dk-=AeB}$>?p7FM6=izss@MO~_ z?RDZ^l)ccXd6i~BHUrndWAq98IsoyNn+&+h$CxV#^XYvC-rRBv;cRMY7m}u*sx=ZgS>~YJjOZ&fP#rcUB{L z9|hDMJeB;v7@l)e_CJ^ZKYO_~#s9ZnKJx#&cpC73bCW!d1)C(4#KO~oREpz#;p0vI zM=1a6jQ;%Uh=)S7RY@vT$E4k=S?30`eaq5dmK>NlQ+kG!bQq8rvB`cWR}33ZkQi^m zcj~SKhmU{xqOt(%@9_1j0`WWBJr?G}H4I)n!?9lmcABHl7idjL?(0H=&MQi~0s)$K z^oEU!X_(Q>UrZ`h@_&0B%8LAebNK((ix*Ez{=b(mo<92j?&PW12eCn4b}OX(zHlAR zN?EoOH*0v$H_N8&fHksg%&%HU8Z5(XX5zM*ldg3BXRd#K>91l3V6Of*b^phUXU`w? zzdLzK0WsyZU$?pq#V@ap<+pczvrD{mLtg!&wx{!*;w8^4Pp~gBps&bue7fZ(Y{FkC z+&tji<55|mm^yo-ws7&{Ov8bTmub!duBy7jx18&;^B~``5Q#$!U@q0}!B5*izBDfc zVk0c{mD#28yl@NdBIHv~54M21J6GqN`zvaG`l;mq#jJHVmH>13|IX8A<@~QNx1T$S#DTnpU|LqiFd^n-ir*ZF{~zA zzrBKUHw@_&ka$4TIB+kdH;mfh+WCumsGRWrLi}vQn^y9ALq?-wwhUs|W((2 z54;>Xq_vl(-RN8N)gVTUrb)^An~l})9L+|yqsu2DtXnWQ;5)d6*ns-uU*BE7@~4vj zb91x&N}_+~^Zyqwo=@HX@$!-X-^nxUU~@6;Pp5u+Ru}nO8gtrX_bx$yq&Rc>R|wS^ z$(fhAi`NaCoxaJoS9fx|BKdWva^rT&ahG>Cw~t0mH@QyP>D4JSAA;(P$c+YSmqYEj zpu)=miGvHSTedYPE-O{m|8vd{79Qi&?V{zO1|JL*A{okFZkN3ad$@8V+ z0%J4%yJq^UFUu_z(s#hj*)B4L3xeXC<@`$+Dtam3_mmf&S?k}JaJRkwcXoEl`rpf) z7mw@zPM#&~e?uKdhB}tB|IZ4IRI842(xX)>6E#52Hvbo@``7RjR5-V5hCjzsuwdn) zU!mmkwL2p?4ph`%<}FC7_I{fWfy4r{OLuRRX_q@t3YQ`8 z3eAoCV;rO*W^N>)RMDK*np4%)XisZnUuy2}pH92%0REy@yVNL&X@IB0P29ItQ6?32 zi>PX*;2KUd9Fmv_=OzsMroQW}&hp&~8e&$a1m|Ua=Wfkq`WKi*BvuW)VaUfPm=PMB zVV}l<3(Krx1!mZ=!MS+Be#(fL?CDH%MK8Ld-(rEB3k~@jM$@DKRJk=%W)X0z{D&?y z?)OZ(WVcPvv->w@IU(ml>11lwRDnMWh7MyQ_H+hA%y@AFU^;VgmL<$-%+0x2$Rd{Y zHjX~1mZm%_`4@j2(IE4<+Pxl77OM+jiG4i`Xp50&T1Kj4m#Um61*j7Mrnq0dK&!nR ztrU8eF;N{kSD@rsDW`Su>e#d?rI^!{`Ir|=Wv_cW`(u&gd6kfi33Eco+ zcKJAJ0@CfzcC7%eo5UFv0J84paJyMEJ&z6P){f+AD+LHI%u%c?s95RQYVI{z@1@E- z9n7q1zLzW3P-R((nWD$KQB4z>Gpb55tW-_rE522ViZQH`vZ`BEwrm#N2$+Nss%Xn* zpw7!reJZM#^Quqv?@y^c^CJi=ZE%>cb~IEnTnlhYr^t=K3hdpjGO?(k0L!tSn=l$2 zUzT+pJ+-e7&yRPH4lni(UcY}+T45@EkeNjmBQ`NdH=sR4A;;xABLX}~M|I~AY=0;3 z&d%SQ9-RHp+lvqH-o8INxY$2Dt?swofLXQYb#n6V^!(!Z{n6`#8NJLoV$+_*!WyyB z46rx*yXU*LXr7Ku+eZTJ6IO@S8ZKML!^P>r$=ky{m3aE%=-vK74JtCHZ^NeEpVeWt zLC{$B1t+smopNmB`FjrFNRxMoYKyXEbHhpX3or43PeiDbvM z+FvY6k|pfih$c&IElZT8W^O>0rIv0)mZi3qpvzf4&t;U8dD6D%DV(F6JaivsB?RB1 zcX-CIR+1roc5d_rv{XuQK4ao4$Vt|*0534158`r|>{Io%Fbz<*EXI~ue7PU~gTq&< zYE=fSOxLXU{Z5A)mEqtdz&^4Y2G!zg?uCq{rkvXtVPnQQd<~&yt`1*q&QqK5Q_U!g zd^@*`mav^F=J^|14YTn#wGez;uZY0eW@oYIqnqOIw)p5mJaj2UxF9TpkVX=_Mp~;2 z2=_@jzY`8f8uj1^N0d#9i-P=#-;#)k9&EoM4WM(+ea?~SHPSqT5xy*hjxs0Y<-+q= zCol7*%FfdyKmwl$h6El>G$N;RaTW_XsW^*Q1fOD^H=#YNV&)EDGz>{RdasJ~AWp-u z1vB}1?F48ByB8u7o!g@v_5f8mfElKzX)5&A?Pw~fTt9{QTNkr)E(Wm$_4Ob&rF`W* zytB8a-B4Ztq^yS{8kEJ12B&ghd6gn?`dn0%$Gd`T^WEEm$@)i%dD)UM*D=?trR2}b zaO)ZmO7R}~s!R1}pQ)O6{^$1dKR|PL?}ZR8@JI zBQ}=PEp)V~TKd#pC&+$MZ#V_CFub7d@UYdOTnB zc)sZIe9`0iqN!2c!TF*Yn#%M6qZx%)bi&A5XUiTjnqU9%jM3v6qkBGMwEE*ecXE)a zyc?B)b*Qe?-IZFwoWWll9v`0XzrNT#eREce^O+qicIIdf$iJd+pRlGt|GN9O7MnA` zHl5rEaD8dr!bRNkU4@iylh_|K8q<{T(I_GU+H^QwVR9=^EZmREw(prk?0t9A0=SwfQ`#6oxatN}tcQ-sg zdz)t!XHWCvalgmoehsOm2_Y{mXl`G)Jf11MwKIjwADo-Q%l_+HvRUL@+!A`4a)BJg zmzBk42Uu6eS7+@xk<}Yb!@g(X=zrc`931byetU57c7OL|zMVV4ybbM`=AacW!D+VI z3RUIMV*dC73DKB?m_B|!e0$!UMX6qfBoxg#)%(-8&00@cSR)T+`Dd+=`G6ST zJDT#7P7{WgABm_Wi@5B5pVdR9zSioYZ-c1Al3)0Elm8LQ|2m^TzdGk8b)hq#kwozPWMt*%I_<duDw@0%K6LGRjQQjEV^NYl4)8omKF z-eGPCH-u`zIKT$^y(lP|zc=^8gc?LX2@BKk)?n zWHKpyZ5Ji2N&;84{Mz)2_vx> zLOU}Nd;bx1|AYTG=V3$x{3;+^j$T1hbD+c+jvE^eGFQ4s<&NcK_w4Lmx2Vd%JYa|8 zGI@aubTez0vJyzw1#x>GsId^qn{xIWfJ95K^4q%WboDdOV6<#!g}#jHCYLoIt6RH) z6Qt;)p;apT3iV+J*GgcQ^B-<;K=AJvz-W4q+En^ zO6}Kc>RlX8dLUTpdh`$*@g96^d&W5L{E=cd@#KPPZ$jHx-G!c2(v|k_^M^ZQS@gdI?*qhFc)2I(MWAbHA^+^aF_^!gYr(6fC zPb29zJr95DwSBt-G@hf!M2e4z6roY}#Bcwp&i{DnXUX1HkN|Ra{>QEDt*6`N{EyFH zZa?OKyo<-p|9H7=vpMR-M+;RcND}*&CB&4XZxRNsGEeVR8eQ8*odaOWtsqhVi>7y*I zmZc`S{CzXt2l6=bzca>RL`E@Xc(r}Zk^fs#*cavh(`V0LKFa^Qcse5@TJL^6J~%zQ zZ%`hedw87ne|oUHe{|4^f>rl1bN%mZ?Yw;YytMvzUhF*T|9A4Nfl1hP_CIf1t&jPS z-`B@NB)r${UR_Pq@GltiLbeq184N1CEk0;)DZ@K?Uw`G9V!l6Qyj?7`Q-jfLOpipGHYYq)yjqwSJ}ye%alG zM2ULf7{f;~CL9<}Xb$&?h;cgT_%zC53YGc5yMkfd)p|fhxS=;j8?Dyb8l1oVKl`Wu zr`78B`#z014RPx`kU|nj%moTVh$f!VJ|wJz5IfDtYO0yTSTGu-zQj)*E7*Qd4C%a;3T9Ig>o9pCjXfZFpIIkTsPV4c0XmWI)rXmWfW3Q z;t_~3hL6ACFv=m2!|m|#1|-P;ghqB=S*O){ts&8a;ozef1{hL~13km4dz2(RZ|#$L zqeip=L&{vB<*YkoLx&(?fj!+mi7m9e0}d9)h{FKm7zE}*A3?dI?30G!_|4(*|AWI= zqKI#TkD2hy2sU9DrZ^UX>i2}v2#YaJ6*UT)$nF(BUjOstv^au<(V#|} zNQM)Ed8_3?j&|7YZ14Qg^7KGuG1p8(@K+%I_y1AYt^9NJAVm;Ax_= z5Rq8Hkj4RK;Y3amiH8hv!BStO3>*4V#Aqi_p{sRHGa{8QHL4U!E5?|xY|1aSwb*K1 z!}OCuQt_>-{EeP{tb_j~O|AHu3G2SW#H{+QYj3e<(en5C+!WU9);09aAwqRr^AwK= z7i`k6pN}`HIP5EF3Wu^5FjMU&Grjuf8u~E_YlkwwzJr~aM=bk!kIuTy4|CE>%oqtU zhkixnja7|z!O`6|c5JTNYV6oUel^(1D*v^xV}mDkICH<)qw>JfA=pADN?Ph?DaT6p za}a1l=y?Hyr4DWhd$&$I_9#I$?VuncafW~z3d)+ea|*g8>=?K331y<(b^$v|4|Z{4 z{mJsfeXS=K)?2}j6NgbIo@+SGKPtvkO*?KVM{-Zpvr6ogZ#AuMwJgyyS#G$G}U^AhtOqkRKQL`IT3UY zek!n|G@gg}Ttk1_ueSi29P9`*YErQRI|AMB0;+wkq5tK4_l>$P;_DK`<6tMxhqEZ{ zC=QqCR5}mpxrYAX@R*7dhB=O>l#}`Oi`bbdt!$5HCTGs!H1mI8kLAkC3M~ga>cX`7 z=*_W1hM+H5)4n3$W15Bm3@~s!(`|BR&o%Vb-EhlFErlJ)Ro@vsZf--@V6GDt6cWS) zL18!ns|G?vh*lm`q4!~coBK$wh+L5jCs;e@3SQz>D2^+U=yn*5FyvxKR$LA z|LI3*k=V3kedYh{56us=oCB)($rWXvrm1ZuE359l?n?de8V-wS(Y3t-CLV~AxDsu! z9y=BKVRP)u7#R}!zG zzZ+hm$xH=qR{d)1)Mjk#!!@YPjfeZnESF5Q#Gco$#yEx{^3@I|=aP4l7~Ul~J{yyv z0LmaG0i^M#m|n%3AX$?>60WZfU!4=^8v0wyODuz(Qm&QR?Onq!=u*oTt7)plOgKLO^-IFYe;9)*1#7_M-M5)IHdlCwB z`EJC0JuA@$?JY5lnz`0Pe}DQG26#l`SO-X`r8cB2(#?J3zxTS`G3DZ8PZHPf-2vh_ zh(G@KMRFzFKSd`mB(4)J1NM zJQ$kCsA{}#*^NZd*@K{%{8=4#MA%Gc%CWPZVdnhp8Q|DwlUs?xS^in!%2Jo*HKLu| zzhiwfR7V9$HBs>)tq@Wc*&ww2V_9fG;$Ue|mv~mVveZR?i(v;P4;ncum08){b#uI5 zyH+I25l2Y)V?ZNBVqU_L5%ap?r6Ih@vvTZcy<k4B6b!XV{M-nSg2qQGWkH zQ431N7{|hHz2D)pI_&5RE}N{ZQ%Z$hD^eeHDa2Z)ih{^}Gw2m!N8Qoc1Uvi3Qu|Zm zx6DLPt@1puXSLYL?joJJva-wH3xx6z5w#NRRAfz=WrZ3W-Kn*7_qY74#71|qlkz#n zRz-BvgMG}{F6IuZu6`LibJy0a*sq87SnM=l7}fR5*ilJP%=jx!i)%PG2Ix>{OyPEo zJ-BUjiJk2mdrk}3nZ*8>(U_)uk46!xh^mx!mQzd}A|V1I8sHA>h9SMef!U1fL7&F` zP1!|61-?K@5>A}oSBwZOe}~jZq1n%)!{Lg2Jl8jDYzjMZ8lB}MuOK3qb~2(-8=-1j ze2gw}0e5cWa7+d6!JbM80s{;qhLO-4sqritUefkoiPznQO3tqN`xsSuG_Gy_LvV^J z)1d?JIUc5=lJqFJM6r*9f@tGd2ojI(YzbaNKbl-Pa4v`{g`K7@Wz*x5vF;sXf~c}% z#EqLwAfs)6!jnP}cy`*;%bDucnD}EL_TZbNcU7G~(54L$vO++HL(I%c%dD_m)D@}i zZAWS~WI+^F0qVv24kgTMt6CB&&Qf<{e4!;32c)!1#{nQY-^NhGKx3tC-iM!UCCa6P zV)NJ3aRRr;83zmW(OgLYm83y*JXcN$w;U{f4Hpc>0gYgLOC1YDzTz-pGDdM4VMcrt zfpgIWJ5vXAs(PHl&V$u3ov|%NKxNu#WVY4P&O>`vOgmFY->Q17It=$PpB3{RiJg5i z)5Mv=4g`e3kcPp-TVB`DugLn)hjlKcxDJ#+94c?r-N+RL!O%f`*>HYe+s}9n8s1DgrIpX`rLH{7jPD&aR-X{)`)1$>B~}rV+SJW4x2k8wE9+2Q3bV^b^*p|?vh*b~1|Nux(ny`} ze<;tbVCPp9)-#_1cBCz;Ob+FBWGLpyWQL&k9b^Zq$4=pjtGdH*r)exmgzdV@GdkR0 z9JlkV7(4C-Wwo?p3(Y{&y53gQc10u&Gs>#dl-#Peyc+CGU0pX*-%+uN+B$Tu%3B(Q zTYNNjVBJKJ`^x>|0=+ROs8C|3`u4tB+R1N5%TlViSxyQU1giVkU|qY3-c=p%F3pNI zdZ(vrV;LP_CA3p{w_jbGmDsT@%t6!LXgMydt2k!rUhX!pt!vmPN-iNbk&DX+b#vp~ z(tzB`vqJ1Nu+O0^(e1V&{*3+9WKinxKSH%Y^wov;iMJJEr-6MAGqzUsrTYX2K;i`v zerwO_SJqNoxwO2dxo~muF2!P(8QsdUGe7*y7HW*ePD1P%E=a_bia1MT%xWghSuGMB%=~`KoEB?w*FyA>3Va=0XW5#4sQesGv5o z_@yrq)mPZ@$2dqs%&Iry%RL@D$51}~$RVo^>r{+w2!iof+-tI| z)zD7)(&pKF5+=~j-WqJrvP%}!4e(``k0bR%r-If-MTjo(I^Td38m#j>!`L%J8$UZ-fXWUz(L(K(Z+vE4X^3tpqz2*FrZ4 z(sXK>48SKJ_Lha?cAnK^$GsD~I?(Qf(Mu9wmBro0y!211E*S6MvuIz9=d@}su-^M6 zzbb#REUJgEIS+b(+F<2gXyf{2?C^9j3pdINO$T|_5W{W^elnar3`AW+zums+UT6V3 z^Ro0^EB10`vh(d_+4&re*`VtFpOvpH=L-2cnI*BatBQdv_h50O_t8%=3Vvlo;CHc) z7u-~zrXBL$%QA`ivicG` zTB%hIj5Bk$+RW4layjW+8x?jEDkU^0BOP$Do=XR#F5jCqu}bWq;ZW+ob25Wo!>%d! z@G-PX>(H01c6 zv9wCs(S$JHgJG~^=;nbU#=yEd>?AbUo$otsKsyNy_yfSr%CYlyKKab1od<&$1M3R0 z!w6Nm!=!ak>n4;ox-MtT2&#FGJH}*ChWUfT&8=Z)_J&p&I}Z>q7S`2ahm!#NsB$^g z_P!eg5-%28#YJW?wLEk;eQtmq?Qsv$C2nY9EoNh>mhW%FW+n1UW2c#kwYZISj~H4Z zcAA-3%h*`=hM$$N978*eOsoZMEPk(OS|RN;GqIZ6Soez<1MAAL)6B$L%Er2{rL|h@ zG%&FiwXyD1YONAGjZCb?ZLIs1Su3!yG1y^w zkv%bCS~Gc};&1{33@8;eLW20}Vl#cTR|lh(U??Op;&{nu=apo<%pr3@tL?EOdsq=q z59V1-E?IJkW6b#pql3AjxYzKjO71bCsZQ{Lpip0@CkE5X(txexSqWE;vACk_lfE=* zK6cE?I(BO4Ab0vjx z`P}n!8?^H|35idHWG_5HKIVl@ifh<)kF6?l; zJJ^urIneY0NpkyK!}FWfpT^D*kuYWWe9SN()37YOuHlHpBuXR6ex;VJ4w{=vd1P+l zNOxI~1eoDKmBRs`p}-8r=u)XmsqbSP;1%Zi%3ytQv-;E6QNVJ(7OMrFwGzHl!p>}= z^|zB*CG5-=T7Ns4Rl?3}q4l?uStabu7FvHhnH6Edh(Gggb+^YeftUa!#~)!a#wnM^ z6=mYVc?VZwqg(7WO26~BO}pbaX-UzJkVdsF4V1w#jdhH{7^5JR0^T%wuU?Fm*yv^2 znR%?Qut$tar~6Dx`AWOjT~HOA;xK>?pyU;EYA znVDdwDzOa10pS^H9!{E>+n}8z8WTZT4U;yvj*f}i95$M|pf-2tS)mPmh&JcmG{FqX z_jmK$23NM&$pZ)6aInJ4vY}rpL!6WRjnguk>@^$#;Xb`o!hJAt5qBHUYL3=aV<+=? zmXZ>^OAKzZ0zhKp53MDldwFhiWsz85c3JkkU&EPVY+yfk;BaU+<#aL-hazk$WnQc! z*^wf$+kI{mJ9TXzD zUL$c6{4w@FnHvQ4g>W(8&nQa5#qv9?=Fpl!4~ftScNiWKE?C_uq(XkvLshC1S)DecG_kUYRpm22o94v(ogVVL81epckE z`qkLcX}4>RNSvw6?GqO@`a;Ni4@oIMRvlaqDArMSCy4#qf)fW%y&Fzk2WXUeR%e6!W+_xqBFXoy?ifm;%yiN|n=5y$Xb z3qbcbebfId?T)cm_-{_%cK<4xc-c+%vMB&Aq-Nn<1t8&Qwa&*F(12b_!j2+!Pl|H% z6fZr^?lbLYCl_t+YP>~&`W?r+6Hb8Gi(5IM{DS=$ML2-I18Kk0YVC&V%$Hn<;bapK z{A>5&}mjR%>ky3YWyHYitZK`bB^RA|aPW z*Wja}eE4t%hf$O&m_VSBvo#b@pLdKGz2noUo9C41j_H*rXg7@m%(y^t;4$5xC)@Cx zk{hkoq1vqGg3)*sPJm%aKG}83a+IV10+3k8Nd~FDx!bw(y#wcC!U4xz38e{5nQ=3! z1(i7^{V|G1<#WSinD4j=G*VmWYU!Ih3=srmIK)i0q#d`d)@zLby?tt@6hVxyG)d?d zBr1vEZrTkpG{r74<8l<{k_&ZxbXIN?I2pxcNPHAqsQD&P20kMSZn#Xt7&9~=ArX?8 z)dAs-p2w3MN&eLIk0~R6%8o+dK9aK|*D&D*a1#4r8mNWV*Tm%II9GJl*NaK-qfQJ} z4vAd&G%741$_S2yp3tNYCN=ITL5%SHVt$54X^5BwLHFA!tpUO{B$A*p3M?w^!l1F7 zDbIRB8BRi?FM;pL#U&@T<+-n|<<$e_k8J%={ZX1WdUbkmoJIr8WX~!`3%50RT^Fj@ zsVxNCrA|tpcWggrSOA~I{+Q92rd+XFA|RpxuI(u=zNbp}piPIvcDcu@VsU$VfOce> ziz>n&BdP2-U^T#ToJMDPjeMwECAjL!ZD%N{kDPYf6R%ffP$C|gRZ>aAU?yJ|AuOn_)(h*7w zFH}-n4-d(?;AtXBOYL9j%_j1vo|UUH4ROU~6@}L;4Sxby90=J*L8FqG}E4geK<3Y1N3hlF|}i5~?EC6#Qo8?uH?~vORKRPlgmprAZyT*97fc z@8}A|eu?7xt=8dCAtq3{5U!9mK8(uikoU}M`4x?+lq{Aqe9Y8c(K#VF__vE9yYs)V zfibY>>qEZn1l?KdqIEu|9GjYpu8bfc@u(|NeYV8}2ZYojFgv9>2aTKzgF zyj+Hih>&7L$`30&%)g!%Enbes4I_li#7OKJ%d_Qlm*zNz`%Yg z^e9bz6{9Ts9suPgbBVz(U|CP5j5bTpDPWi{}PeAoe% zWPC}G9(wWhSh zl7geJENJO}dMyA8llOTPJdC*%uKUB7R zJ*(ZyJ_?1h2JcsekZw-qnhQHxM%IFftmoPF|72Q!Gn6%NCq`mVH+s`hMKpJIAdmcfpWw<(nx zsitbtrZ|iC;3>a_Jwv2lWouGt!9{M7+sHl5Bpe&CV&hI=F64hL3;A|u!3sM~D^wQC z_6LK(_BZ!N>AfT8BSMdvN;8v zb&8-%L_)O>LSkiynaz@H0Vq)jC;7F>w`7TPK^dOf9uAN|RyoM^Ar3}GtG5ngWo)7VM@G^M1g$MY@;3u^G0{^uA@_DfJeE1RtKkdBO@_%~T{i*Y!8{h#! zvG-G#Q7VSK8-JZu3h=1Z;6iD)uR)A-WlWu_h5Hai-06Y!qnvRV0CN%X1 zctqk2O}&z%)#Vk=l3#RMEoqTDz+P^5p2L62e_m+fQ%*1C0}?5Vr8F;&r{vAq*B;M( z98cU+u34TI?aS#oejFSRDf6+5`3n?=7y7CXV`ybf^-56~5`n(RGQqedS9!lhyXOc2EA`REJXh*6#h*cE{w)8t8rCb(Gq@ z__(jw54@8IBq*%iX1vJ6hLJ+8mZw|@r3R+E$s$M024G=}+H#`=aSkY+xF6lL#^gW~Fx0l~X77q@a+PFe}$kwaMgd;uGY&ur$u@dQs$EzPLd82&2@G(IheYT~gqLGncawxwMA@>!zi#%Yy^L1tWu0Yf1{fF=9dC$R8IB5h-lck%4y!j?czqC?uvO zGs#|TDqBX{QkbZ{DdU4xQ*4jWDX`2oT$Jg!6tg-+P^Z9Gw2)ZsArIPl4H02 zpx&!CN(F6i)|_+HtZb8yBdyzY{BcBs;#cj}+1Iw0iN{h}`i%8{sQJRrjM|NV8}jh1 zv(?$!H15NMa!v+%A3uvia!ie?)j3V)uQ4Z5A2a(iRw~B$%2v)#-7BmMG_nwAEyirs zRtIY+%`n>~RSme8h=FpKcHlMQIs|UBt^uhRO71t%l;G4h&C9w(z!=%RPRZT}&IYX# zfQ1-H%Zg1@Y4+$?)pE{(S51nmsE@7Ai|R5(!)c$eS6w;2ZZcBCtgho|-)ht+NQ_Or za*dkM&9yanpNtp^aI1AFo^Xh<6njXn6PoSJ>HS%Aw@0>VWP(l2K@Ir{#gOXO)aIiK zbkqi%IanYt=pu=ABo0GyK%;Qt2=}b74xHi$DZN#h{tP?S7jSmtJ!}N17F!N9w0RAUzNqf$B4;EP z?*k=l@;wQ`QgbtfQLhRiy)l)Vt0KT3CF^RnP8c4N&sM}qHAbnwSx1CH!lAuY#yZ-U$pCMp6K%xz45i6T^1t5H1P6Qtn5y!zx#O+pL$iF^Bay zvs?b8U{PJio}$kV93U1F%P+nIWb?a8>-SJ_Txx2Co% zN`xfNPy`NrC{_02+-xJ}qk039!<;q<-o~SrByl zM|-dP#AMvaXL62=t}0JQZ19>A@#yCM+HhipaqK?yN4Pe!Q0B|Yjr=Xt@x*dMphU4u zvVh#Q!@6-GXKxfQ7w)ghkV}lSMWlVSab1;W?_uYlbA+FbkNCBAK`|KwC_z};uIVY> zLt`Zy|Fv5-pA`z;nE2Lpjt-?C_Iqv7C$)Be4`!XNb+2>K5ofe(W57|3e%iKT z4H^yM5UUtxQ~Pgc0$0B#Dx|+=QYHNSS0u;*Ezeql?C!GbxmKoHB9h88QRK;+- zBBvIN%E*XM!d`$xgV)IIoOjG{rnhLz!zGcDVgO_utC$ZRMN-2tgXtfrSz>McKg$5s z@})zf1$$<6yp7vzOeS`ASH#ZhbKH0-9)bTrbDyKbgxVJ#hA_}AEt2A&#Jmh zbvABc-^@5rWh$jNb|=3FttkhuQ05eXTa`mLx^Ac{}w+E?}q0Ck;logmwSvRFW%$=-WSQ) zo8LS+y&Imvt9RNik{=>ro6Vgg2_>=`9sqj|r~cxIt5SP5_ZR(s=T!$KYrQW_(AOLI z3F9y>{8`NMBJ~L6A{6BUkGE1w$RYG|&1~FIxKR`6^S<81( z_ddzuScYlWoMbxlbP=@PUBA55#1o$I$J?=G%~=jS_CpS4P$;OtU3iL(ce z73S!HtfBR&`lH)op&n>^izV9BP(2XZlwj|cecF2OY)u8r9=Q^2m155FtkZK2zeW$? zsNd@xb$U*G_*$!%^&{wDQs?g(sF^>ZYC`47&ab3AaFaRLP!k_V9XC3@f@f$&uQ}3$ z;Git_0BFN);{g!GDw1)qELq}47E2GMyo0dFT<^hWBcndW`iq71 zd^eTB5`^K4>#Li);pN>x82@`*EZPCzob*k-ni?q_p!4$-yldP_t&ylu&sayrRXUgP zqpz14PKB8U*2R6BbT+^OY7VcpYF_lKTSVy+4m}K5++nz0S^viH{QT(lGK8lGjfQ zsy?k2$s<8w=IWG)ZQ)LVTc@n7Q-O@_bOY$7N9Hz|kZ5q}G|t#yMn`iuB!Po9)fU^K z*E`cJ2q;Q7@G59OjaH#<;|^}*}wDipM;v+rY#Hg(rmSuAPQS$M##aL=4V3>(mE z&G6_^uk+8&KSyF5yCFfgkNSA0urJXjgD;R9QCOAI z<5Ps#9MP%3M8R*5uFg+#7Ph0;Zmu1)Bw-3Wubh4gx}j9D=7=4uxE;Q}yt+HR?WDgm zvUs7R9ByFrPUEGBvoOR^VNq*W1rCxtOzl2Da5d6~o%f!h37FeO9omEz3`MaD5$4>=}BW_%}{VOe%$_^?JRdg9H4x*Xx!4?e!1xzwv(m;9&3QFX9=R-u%lz-uz3i_oeleKkiTR zYl(A7>`@1wlaOy4Al+0PpTTaBh;ilzULxF?3p=)09Zc(oHuO#BDO2&~_YHo<^`Fca zmh=tYFjn@4ebDt^cFLqu$H<{~ABzV>4(7QNmC8)_r^V{qa9*{Zpv=&FS#u z;W#Wi?38on(hoC1#XUbxZA zR{y8^Rn|WvvOFgn*tGtS_ABy#?{NQR{eO+$6TyML6cK1rWHa@+K%K@N;*J!de*ExP ziq zn;G%~{#Cy~*&U*YM5E#yh=zE8#x8PWslk#w;pK>PqzW7> zm1;|e23P%rRq>h2!M9vo92Zd|G|JPC<$B-w)+6w3f7bf1hn{?fHrB8I{k?;|!}9)r z)PMD|{=der)e>(htet~WIGM5|@e7`kQ9DSFH?vnWt) z5en6o@TWO^tnd@DMM-D3+Cm12Bp?PQw(OLQTU4E)LVetHmdMzdWxgl7*g(>4MX_|E zjDX7N9@Ydiz{1C}o1&Jka#SmWeH#ilj@>0}Y2}&FVQO%|j)nKbo45LNQj?F{XX7DS9?3$g z&?%S@C~eq0aswkmi?bA?#uM>oc>5DB%*)}$negN|BUm{TcYt5prh9B5EzG-;#BkcBe#rY!^z zQYcEQ#ElNx9B^1DG+(W}sy^1JbQp|P&vfe9PCgBo>x>cg0IxT&nrCM=&&h6h2^Lw7 z1Izs?D5)%rmh8smr zwIU>$D&tTR%jfG8qME4+Of;3pQ=`vuYWxeebR#$d8vSOiZ$$o*k|V`r{z%R(RI&X< zslo&o9xV3-`ftG`{2PY38DMR4V@U>LlvZxm>Wck&*S!{U"M8o^ZV=z>2RW=Zi8=dQ?=U}FdNvP)szOBaGH$o zrgpF7D$zNb9oS)RTa|!BsIV>6blT=1n0q&hf~C14^(#8fsfioZNVBWrEWYa;cH=Be{e{#x zo{w%o!4_}d9Xme{&zl6g&&IKZ3wYC&k5FXV+AQ4GyE)&SKov2#{dP*it#&OW@#Q!jB^6X z!bge2ss(w0SS;_Yd%Vzc6h+`SmS-k~p2B`+DYjvql8-ub_dM;_PGyCDI<6Sw-5OnU zi`D|_@U;7zgAx}ix#J{t;w%)~a0CfeD*w9Eurq?_m|2Y&FeraBx&UtDW|l=V#!Jgp z&!1uml*|3(kN#G+J2>fFA_9)1$D;7Djp|?wa=*#a_nB4y?YfJbQWdtrP|m<>3XpF(5TmH>d#*bNhRqQmvRT6ceT1DGo1 zxmqw^+8@Kx!RiD=n^FovWvjCa;l!*@tr2Buy;}@sKV%D6Ad_w$LtCNZrufn%1qrYu z0-^X*rd}bXQiXiRq+X%pFwp|fI766`)wB*KSQ0m&|I#>1 zQoKJ`NosCmA-q#6uvnVFgT*pI5vl*(UZ;PE*hAgDeX;NktW2aP8O*xnO_K9n)sVDe z##1C+U|&SEBXT?${5z$z0!%lerSj6O|{y z=rjwvOldzca&jT$v6M!sOhLYZhktWbln_fp_ypRnoLU8vlu1+?Ymzaa)TY=54(gIW zB#RJ9eJ-J$QvYDg5pCEF6dFOS4ha>n=wZ3*EDvM9wF+5yRxCzB>F^CWWhkXb$Wf?2 z1gv>L_O4~@E02@) z55y}{rB(62;O)SJp{dc!WN|PM%_eH<;s+V?MTB{4=uZSk(}l8WegRDFz^?UiPd@XLvlLuTuH-4@j7K!2daxuH_`aKOF4X0JpyT-u3hazN>XHPZ`s ziOZCLBs}ixod(-O6*v6l8m~faacg73h^d`))L`^cqALT3*STT4pNd5Xb3ZhM-4}Ex z$gQVk(%RIdoOF!Vq(WGL&q!TeGs;fRm^+Y@Ik38QcCCpGofUa+35SgyZB|=jB&%a7 znrhV7lR#yj!=9!woNeW!8wA}}ER!_$C#l2LOjV3s*FD#&+xjoK#h%_E{sEb{9ntS~ z=zk!pusPO^(+Xz8cCrJ^5&i+Lud+p$`GSzy5BD!dq}oCYLOMrv!!O(zVoB6`$FYzQ z_~Y1(m*y}@Wl$PF0ZaMFJ%A8RlYhr*xzYGWwZuOb`*>i4plar3x8S? z7szHE1Tw%q5YBRQ=|1?h1vLt>NxYCoxy7c6VNYne83^wf4V^BcskSOnr1L^OuqGds z^7-&~^mHn#)pPfUyp0hWi^S2QPtpOKFUwT zh!EX%&#+N4J{3Dzr}U}j$6$9OLHF3%BhyO|Dc2);%{W>Ox?+E=+0rm1lO5eK$OqT?F%<_ zXVR-7KP)#)QNpHei}=^?kB#A8+v`n@l+z(cC?}`*vC2a4i}y7H1hQdUV?L|3ulN~y zqqsvr0u5$c?4`-li<%iUUl2NQLMeV3XU9l81+TZnEJ*{-T&rHR`eDOTm!RDZXGoRl ziw__EVCx@~+zUX3Qe3|$Y#(i9(fh3FS(1&ZU(F_VqvYe)bU+BCee6Pc(&Ssbk*@c1 z?5FZ7#EmTR|04(D^;^G2hJfiwU@5n0?51*7PDuj08Dgg1VrEtkv1%C=%X(DRM?Oue z2VNhm0PV{bI%~H@opKWDl`oS#lDLDbU6Xa=kXh;l)*%&i9f?3{zTELAfI=s8Z;_{blPoU${hwEdfuA# zw>oaWpRTIt+>VMGZnrdT^X5mU0oX_0$G$Vrf}_B41h^ zLan`^6eL=?*cfHh!1$}=3fWFo;kcE;F(nWUOFmW?KLxnu1|wFzXofXO1u*R{!H&9nNXvR0N` zHqDgS!z}@-6}W@Rko&?MUYNsIU=FxI&ur;F8E_8KmYMUOa*VFj_&D?URY5rD>7)p(!kMM2+;@xXF4?Nhz;ft1Gt0h#(y?{eoyND(o1Xx{^+PKb~4m(`3 z>Wiw`E_YUV5>7$ImL)fWT7%UW^E0PG?pHswT`73p#XKwT8YfZ;8laC9ovz@yw`9Zp%y~P4d9c* zRN~p)lsx@Cg0-i^DD6jC~opcD8H zGL%WeaMG^l<5d)^v3%0e@Ts1)x*^9j;!q>qmi02;*=gv)*Kl2@Ug0R-6B)Qm+e{=x zR5Ts+;T}vum8It9K|^4+)GU^F^H?Tx6?g-2h$nZPk=EUQqhWgdY27*aV|r5RA)ab3QRl;qZnlpnc*x*Cb0{zHKC1WZ4oxPeb;||LS9OK*CJ?DVs^lyZYRnnKB^gM)5#IvwO>V`aW=HAGp5nn;$;pk$=saNWG@$ zh>sr~UWd((qAOk<_Ilz6^z7E{Oo46p7uKDiv^8LcIr$)CJl(*a?2{9{eq6IQ=`8|P z@D31wgg z>Ru023#trfh|$4yIS;<;xH=+a>Ibx|QG*xx#s}SwX7`@)fYwv;|78*;NAvdd{Pg%v zcpcBhL!cPmBH7#GczAm%e*Wq75-OOusT-(SXYMAZvU?K;ux(y%i>>3U%gfW_yR98@ zS9;m%$bI7f$9io?oSxsFik>*VJb@m-Q}N@?)y0a}1;$;(7{-Xb4i(}!Z*S=p-P>vl zbJ2$jP1dcQrlG>3NGa=7Y~6uhmf*>|c|Lq|dfqHOEO42o_M5ihp__#U;Op+k%Ty+- znt3~X_x5ysGq*sZQZqmTthcFcMm1q=&#;cFfoEazxuR^yxDtY!Y=o`(PkM zMMSSkn``9addaw6HeMwiuaJ+0sLJdAOZxV$QC*(?tKk212(ZaCW*dsb=@H6O0#dq1nU+#t zCe4BPfD5Bp4O27EtXdL;#=u(WzMls8V2II(St8^7;Z((ornR;|w#!Xv-{DVc%Dl9> zq1+p!t_6!ERJ=+Sybn8M;m+z_mU>$C>w&5Yg{{69?#D9pWQo?8Aj}7}@O+TZScnbB*SKqKJr?@e%n?jb<07cj6>uX1M z<4~EmR}5FF)9dQOXQa!o++RNd){X1tu`Lqw0y`jX=)0-~3lpKY>M_oSz25%o6$SK^ zxUi;#$R~m+J&Ce!NQ>j^ch5wK-l_tc5Q`EW8g+lKS~TV;{gh=>H@mH+lpo7aWa|1s zvYB16P_=3ajFk$c8?InkY>ToQUdgUVhApa#m((rVC`~G<7_QTh4vY zlVNrJf>7x5P>?mN_#DM3Xw=R;uL<}GgI@6s{gVCWa|Un~VafxGC_TAZyjV$@J3k8o z)uv5GE9Aa1!E&y$Qf>c)nP2>(2K@K&PjsMo`MvyJet(+Z{|^8F|No%$E_eXy0syg# Bh#3F? literal 0 HcmV?d00001 diff --git a/infra/charts/feast/charts/prometheus-11.0.2.tgz b/infra/charts/feast/charts/prometheus-11.0.2.tgz new file mode 100644 index 0000000000000000000000000000000000000000..32b3abbe4e59c96e000bbbe748d0744d1cff97f8 GIT binary patch literal 32813 zcmV)+K#0E|iwG0|00000|0w_~VMtOiV@ORlOnEsqVl!4SWK%V1T2nbTPgYhoO;>Dc zVQyr3R8em|NM&qo0PMYKcic9zFg#yx{|a2m&)CYiH_5V-p6j`fB0GuB^`ax$nRAaP zkAx=C-Hh1;1AvlyY`?$#J=jP91eabU%TDuyttNp&VXG<>YDo#6BQ`@B?H|t|VZ8;M z$A8)EGZ+j8PYw^o|AWE6{eQTBkbjf!hX?zI!@qzHqv3xt${_j6V6ZM6w{u^~1JmeN zL}-MQ5x5$5V4CJX_j|*~y+J2LG$2vR#JkgcxqyfQ2jx|Q9J_E2qq*ONBI9v`3v4|2Y>`}^z&~;@qfc)Mk6%yl-8d*@@IighvJL` zNVKXutvccK62^$IIZWUbk;cMC1C2#9I-5d|uqq_pXJbSX#1IvK zb}54lb?1nYD4-n{#W5oJI=lamr~AF{dxQI(EB$IX>qgq1O!2bMu3qFbvh{y&ry((h%HWW90dzr1SN1B zqi_T!Fs7)}iRN&MPO~^ZM*%^Mjyk)bt4Q6B_}EReIPOyMd8gCa-38CkBuWHtxQYUF z6a+X+n5aX+1QQV8d5RO1Fxul~VHPtA7%tx5r(lxB@dEsj!8n>kC=|7Oo!#Az4V7}7 zO?t_HJ00aO1j3v~;}AVpRIQJ}V~}75#t1|S zWiXCWC~BC~Bd1Wnv1)V1DY%sQZw39 zH7S^9RBXsOOftwRlicT_9(Y0AQW%I)7nsSKFC~!z?|u`Tp`e|zh3+JZ5mldnMBx;& z*$DIz3;GI;{5#FY^6Sn{@lKR^RodWPZwmy$@O2U|ECOe1FU#Qv~xpT8NDlB@RplV?yw}K(jBAagdB+R9@A+#XRN+N6C~R zN+nm4yp%6!hn1elQgUF&NN-wIN&M%^I|`l`3``QtL_eqr!gwuck=|>?O5vweg>f(7 zENle$G5JG9Q5MrK40zY?^G@Awz^Wi&F&v}V!cCB(7vi<1lhu$o8JAL;Ac9yi4v<1N z(-mJTLYNYK74fNzCVBmc>ftm|;80Bp2te1K(h&1AxjIDa62;5DE4FB^MWDb;h>j^i zDN4lRL|2F`M634N>w!13NFbR>o_#$-3Ao1OJ>QI?$HhzvMi*)kmiCIl`G)Hxj zq7gVlvRew=D;#pl6(cz&^5|iW9}Q+G%wj(M42z?ITKSxcD`^p&p-G{nviE6-1KJO8 z5}=fcKZ+UXLrSA*(oJ#L<$(K|xETmpi|9RCjKI!&v?w?`PI*a8MquYTKL~w#{vo1_ z76rsvegt;J|HW%GnV^7^_E-2^_7HtKC1`?(@!}PJ{s9FULl1_%#{ehsVsFPBI7v#% zW0v9&$iDHpX*Wa}MWj2;q7VtSQkS9_1!9!)^9c^oxp>RFo6B>jys-&qmn;|i^l!V^ zvxt&R(5oM_aEch9h_NV{s!1emCy|o_&ZB5Q#Ubs7Iar$GnwDD|`B({mHHRPGCh!VI zG5<6ILxu5MiU60fV!fx`^$^ryD?Ungak50yvIg zG&jJg!!Iw6<|$h|i-?yqYe}_C07J-p4XLBcoNs80Z)pJqQz73WzMuH+Km8S80>VWC z=TRV88;x*+G(N=&Z(QKTxOJ}tHe#3ySwW!ol4B-El#|6X1jB!lh@sa>uv|~YkhYae zmI0g(O(0rOx&$>Ae`FCs_R+BCu9u!YeZ{PY;^Y-R1gF0q=Zs17qntsF(F79&`wj>-atGUUI^SN|~TP{cTI+v)E3s_pDNR7aOEqI|k&c@sp*)PR>I3JdVS&^PSy z?W3AXvN(PS5?pEy&=dtA%6r5G9`+CfGmNQV!G`|A*n&L(9!%4HfD>@^&vV`jn1qmo zpb&Ea`1yaHKjfdmU%~4XCFcwV@Au4^*tO6Z@c~(CPJWD%kR!==u5U`@LRF=Bp%M76 z(z}1GHjpJa%Zwx^f<1dAJh&SB=R6xr$rO`?I0UyC9fsw;(J-`9Y$DZKqR@RRH(UWc zqC(9nXuq{<3@wL}3=2x8&Z02%97Q>6rUsin@*T`}!5b9EhWyH*NS4KiHJU_$m;txz z3bud{C6pl;$}(}e9GGxIWvLBZ5!)D}1s$uk6nEb zLvVhi22`F1-LW+8eT6Lk(m$;Bagr|?F>VXq7bs*leXRhb;#WwKLh{oWiZ0<_Qq)cb zST3S>}qR)_H!0U>I|$TxmHp)#aEA;#Tz1QXZ*l zT3Gh)-`G)bNevN4p0mRgae{;BEN>enAxJSlaR zu{KUQdw%rn<@4S=%vY-%+n0#WPC0(&F~?B_eLwKsm1)&3An1<_QAUMPP8`jnlIEiE z(84!j#!y$oERKfx8$+`x+bDE9L7I)ga8PLP=4dW%2oLsOM(U+TvTd!7)?Z0JBFuq4 zu-439Q-EVxPhDPS$q{`^5gCEUhldA`ikGi&a)vQ8&MM;d&je?wS$JZ<>f{qu&0LJ7 z$$zSY`N^pzb8T{|Zgip6E>kW?L{vyhC#QT;2%=nMv93;Sg`4UWr$|~U;zhcCxoWL! ze>cBijZ#IQr)FM%kLL3z zTu_|!^9!ObLA+y~-Q7<9?o7NAwhj487?l+=EWZgV;21ZzrhS5BK@~SD)=H@XXebxN zfAWH~)RPvFwAn%q$}nB9jTh>MOqSM$3?Y7%)yX|gt+jTf7afEv*4UO2*}9gc4GQxE zs0~MGw&)BJY2qes*b~IAF?nyPr3765^_Bd%D(usfXMVV+C(kUHHmEuaNULN+?bbGs zHc*p@4ZrUVJ54lft3cZ-(7sUxnzP4OvF<(2D8qB`HVq*|xhaUQnTgluhB-)X41!~R zPL}n`=Al)azLlwPB^*JVAe$MMra6pX)VE5DP=-D*KBV1H--A0GoRX_P`@k*&97{`y zyp(Qt1o~Hy^y6skAxa*GZMQ;j3a3!IfN^Z!;A1^5olb*9QgkB_#z46Q;O&Xwv*eLD zsvvl6SG`s~_N;8pd|H|Llo;~u+4xg|y;z4UAarMNzTH*qKLsdMeRrBgIs&)HH0|Ge@+TnC+ z$FsC}6I^(wh;K3gOSdC>2->VHtqtueSv*Rn`dxol1Mcclt`=HoDBG)Z9m@Je^|^xU z2@UKIL#2S-h>ACdR#Ms)j$WKN6u0H_eZ|ScHH6asAG``*T2Y7sNOD$7N1bN)tPXs7dW3ifBE|2=dUX#ay2oDdeT_5PnPXyNptz6jxv(?vDY zD7n(qP6*#W#c--vP8WzhAV^C$jwUEr1O@Fiz6qn?E=(gZiJ|a0Vg$!jFF}UoEuAtQ z<|6AA0>8X@a|(`5Pt=mlH`gRf0=dvQzol6)12EU$g(ybCWYh=@QTSXa1H%xE7kVf4 za;eyduSx!QbMw6(qv5!h;BJhA_rgKR?$(db0RNI3ErM=k&AJAzVcg{s50MvovJr_P z1?C4VP28?Y#84MUKQ!9<8&35yrfM1+ekfI%`vj2aVWj+iWTA;{?!j7MNNHzCG2 znSv}~QOqf_1gPXYu|6oknGYWShBL;cK3a_-;u{V!1V|P;8hH67!ikI8?rudK5+RKk zSL4F%jydN%``0c(bIj0%nD#;r=I?|Zw~{1pqDg)Twp%LiF|6&)u@#0nXO zPePPLEP`?LZ^@OV=}-a6uAu2y6N09JwvC$nVyGQjL-WCD4Pq^oj?`pS>2vrlHDag) z`Q=KLtFNKvsjXX9l-#WD#)s>w`kdOy__O$N^D(|n|7|(EoA=z7aBtRUyV4W5ynI%7 z2d;+o#eITnP+P?@xEkE;c?TQJvdK5tphg;pKdEPM^~8N)_h3Z?TDS-M=Ur?0M4xWR zEm)1%_NJiKO+k67ZM}lGUcpsd_tsp4%T0gN{=x0LyPSJ)Yq)KEgq37@XS{=3cWxCI z;nu+Ye!|}ye!{ImZJmX0*IBsr#Qb?ZhFcHOjk*nMzA?&o*o$Z$v36d?UJ0kswHQQj z-h*@aZ=AquD$XYa$953lmj1W?R!g3@RUW&pxG!ZSqV?g{eb=MR zYT=`J$31*OStXKt$d)N6K`@LGsi_M=99<#tLj*|BR~R5fFi)|f-V+oQKkcc{!%pQk~9Fpuct=??CMZ#MBO@4RovZF>m^ZPh>FnvcNr?DQCL;g>H} zgTV>T4ER3cr{VyM8wRN|p1}mu?jIQvig0v56Tz63HKp&i7@g4C>b8u?*N!#jJw>s8uPRHs|h<=rJ7v^rh zV_K#!5>}6oN7qN**3xw>?MA7B*t&xJhSVGuC;s{s96tBTjS43|m$ws3$voaefAiC`tl1f>Wvqw2v-jCffG?;UpeO$~Ub zuo0jhizHjJTERh5OlOo#FN7CpUr%uUMoh8t#FRrqwN#@g?nLDor5&x-5j960;ASl= z%09!_iSeCtos^6~mV&2G!NB@5!9)#y@lJ`a);m7iSBQuksZ{B=^G}YPY}T@}_TT^b z5!{o|dwKrU&!72@{T+{i@s>prT<8_P%l+dwzQM{P^h2^Jn1M^Pi61zIX$EJ$mu>`9qK;!e#`0;77@b#S1<& zoMbIgDPm1%mLwxx zg4`>=gktXI?L|uYekm7nht_ua>c4PfrRKC&2AU`nXkJlyz?(1mYb)jPpp$crBkkz! zxbS1bkHf-u(jcQa0TBF8kxW*dV|zI`q1?}?FrrT(kr#Xhd=)WNRx@&{g9lL$^+fmv zNseW#&Ambiloa;>#T_<+!AzTMh0$b!26VnD3Eti^_QT7BrWO5`jE$8Ln^ycDAKa{o$mxAtHs*9s*Ox`Q^w9d$lB9Ya=nruwNPSMYJb!gZf_ zw#P+lBG0DbL-0F4TB?(coHf3OMAUu2umM#J)n%`s6TuAe&Iq7p^I|C9yx`}IF@gjU z!0>ydB6p~TDHliR7*~kED*?-3tTwg*wO9xKO0B2LAF_MnjMe)BD zLnlJ;UgH9;(8iL4f+#T{7Q4A8!aLbb8y$xk!*dbLC$d}vNG3qiC^Y*m49FBE1h3f) zk?V+}hf*?8K;-PG#6a}FVwC36(oI zNEyijmJu66O%>@(gpU&BZj6dJKv5!1h@`DxXog+tiKtWoOe}!SU0i@H3GkeQ(1+eQ zN|YIQ&je=t9VN0a7xaCUUI>6@cvSw#eSIPJ^-60()HI34{~(mgA!#7%HXRV5u{i+} zRyY_wOz) z=7_-yV_W7o_76e%qiReP@_pBsU+_^8rGB$SHYqlYOm3@@o}v%=S@`b-$iG`;zKWTN zEBpIfUbN*!TVAx~MPDZ`YAt*71=DFq?lryDXE-0t^{@aODi(3^p!c^u&q-W<<#pAa z?!|=zI*nl<+?GbB(V&huNUws?qelnJvL=pKnW?CLIH-aWs_KM$kwP}B!?zF9sDCvS zHcGUAZ`e->{;=q)>YY}+s2jlWDx#QdnbMXiZJE-RDgC*alA<;CUKWQUk)ueAFir&Y z7%?#HJ?S0%&8U4urK74;DmoPANK#h0sU$%Rjk+q2H(#$(6^G*x5kCGRPAvbHAVuL6 zHJu@o1K8BHmNkqX4S#s#@CRe%W4Z1y5HSEfUV;H82%9$@U2W^zk0{L$?LR&^*s^-J zAGgHn`pPB77kyZ#kWj#s4;j2Vcy7JQM+`r9ZNMiSkI|C6@Gm z?`;}qzI0c|xRzr2h^`OkoI}*<4y!wW30u7bRoqG9{_g*ce*4?rga00l{uTah@4x^4 zfd5y`7w!#5_x3lyUUi=6%zh5LV}oYLsBSZ zmdLFa%UL202J~wZF)nylw{pf-&RDtwTRFo!Fk3mJLC&zKu%YlVZ1@Yi^BGPs$uTV~ z&cQOoRgLHEkyha*xExU}#a{p-A)|=9VRLEefllz4OEb9?q6M8FV;H<2;}1Y*V*eVn zFk-y4d?n-HEg@btuyymuZxUUZIA&a|;1m)#w-&=3hZ$c$?>Gm!*#Ge1w{LXywP5Kv zx&N(}D~Lh7kZ;#jqK?&CZotfJvJN@z2ylnl%q_B#dUnx<1W7M!upDhkao9*#N~YHv z)$VeRcMoB0x&6KgU5vkvSe5=MQ0U} z(gC3FI+yV2`2OUq;`Y^Gx6Ej6NBOnoHa13EZd2KpyUJ~Ll}m<^)!p{{6yYJQw9dPE z%oGSHBSKk$#WjYu0fb=)Aov8}gtHWx*;GJ1)*F6Tb0yilp)Arz!-6GQ2z7kWWLQDJ z@~H$*`-GT1s&z(2k!2bbE1WysM$Hx@sIMNx4V~JWc(SgS)*mDy;cLV~#4C^R>_Y@kVd-d761O|anPR0sFeaV-lof8|2}u2t|unW8D9oM#y;?7lgg^eSjM zmeZlv0_!^ zkvn%l>-~B9`q{;+qnFQ};Sw5^cC!_O9hjz1tH!MutcSUa)zflE1!4dEd8cwG@$-{N z48lvwy`vFsg(vTAcY?_^Bq6w`yA4Df9+#5JKRbTjR~4AjrNg_MeiTFiz(0@9UY)%9 zc~peYlM!gO1JHl|=YNVVDh$zLOp#=hBI2Hi^v}gd3u1A|oFSbsJ;fqTiOz-xkb7RHlGbY3lPORvEZA26c?kxx^R(l&0S1hQ@3oX{D)r&6PmR1yCj+|yU zF1vUq_l`<1H4EdLlVcj@<(rXaYUcEIWSSZTZcegkK~k5HZ`$qQFD~U+XnsLi=b)vm zW8wSlOFUKv?ACIR32~FXZZ}1os^`XorGzYBbdEepYVwG&GLWZ85GcbZgOuPD5w_@6 z&rwDT5u>`RLj9ZKuqzW*mFCEVXk0zlO11T~ZoYpY66dv?9i#cZ+gL}AKxhWqua=fi zGdlZOC!cE}ESpeoK~^@K+nA_q0$pBGHshPu3(CHBR#o&F({vMJvTs`NsFYm2B)&Nz zxqeo@85y~HMsG(%u5SP4B;;l~x`cq-YWIF|`Pjtr3yR0X3far#q(A9s_fayzWUlUN z6HlP-hu|ur5gT>~3KpXY`O2qzRC$*n7XK&kM}`?{fupBS%}8>le9j<4FQsu_o1rY#xs~CpZ?$?z zOZ9GJSS{5;iy^hj-^y@Sw^%)tR-M}zMpNW$F^XpKTN%X42CK)gBt_rw5ULyT7Ne(1 z-pauF+NvHi)Ba#{!&NY5n9DQ#__zRnWH6Sj8m6g`L3*8kBFXkedB&MPXK6GKCQI=g znQ`}}!Yo2@D8sOb{J&DGDx#`9P!fz%t_hp&6a{9p`+x=SnAlACK+ueW67%B?T`wXRy?qyK`>vm|{5E7xUh z;jeXZdejRyx+*g+o{O85ubzZ#AagF|Pq%gz12 zHJXEI${T7L;Pzt*(HtifH4SNAY&)V2BdUCfQ)yXNFD#n=#4ShmA5Vw9N8Q0g@L$8; zliMBQm77!jjFM^Z{rBRseS_nj;*ct9zBXg-G5%{k?gwhzhlAT4_|@B!BjRbRNn&uX zjdgHKmFGw27uSYMwkMC)>^91sQGkvlK+bP0l}M!=TGVlLD1pShD=cXM`1mpB&kq`J z2@PAKy>_i7<~^!rs!qE*>W`fjKyG+~+@t#HljK5bHBlu^!IG0zqZq20Fu#6c?MbT# zyWxp5Ppqp?nhB-VgjomWB`2#w2U0Up3XQ#r1 zvU+kn#(-#51420zqGmA_HiDs z{ii8Jbd>`H!7=JB;Cuyx(0cpN!Emte*njR1_lJY6{pW2wiVDFbtoOCoAITU7CQ&NR zIwK&y!Hmr?iT*9@ri;^=I4*SFM@cvW%8dOC$Ec%iy+<9OY#T?4_;j01@#5S<;}b}t zBt!`VkEG#yB0|fp;27H>S&ZnYq#dktz!PB()97b{vsB^V1;xosyvT!l>F@3_GeWZ9 z6(ZxjP)3neTiChlkE`_Vqv2o#z`Nga>TD3|EuH@JJWJ94ag>BnGF_GpG}8aaj~)8I zfB5L}mj2(y^M&dEk8(xbK{i0M@$V>*F+saPB7SZOy#m4d1{zgTJ)nf`+^wl>T(A9_ zI(!SBp$W%GbDNq$?*LF{x9g0^mOXrfPYe1lLZi-Mx}x;oO#g=mF8v=o9uBwk|2Ces z^#1`b@c_+Jw(u8}%j6|Y^&K@QU3)F6{6z5l90dfiy}c4kxypHl27d=xg~Uc&Ma>GYC7j5n=)+6ln7z~c?qC1vE#QC8 z<9j>0tStBW^9Xzr!4&ZS!g;J4`&1}#K7QT-pMc$K`#fO6@<8%bwNDOPb z_+do`P*4Ad2ak$xTK_XRINZ|z+jv^j{{*v&pj-?xG|A#~#B}L09j_v*6_z!EtV+WF zEH7~fDL+k7z;U1`MghY_;$wIZ>0~LbP;|VG3OpZ6JP8t^^J%r$rz&wQ?k-GDQZaJ` zGBjPtlKR)k;34(Zd<-F)0Q5(FJsF;2=M2eXq+(+P$8nTQ-=-mCXatfhj%^;}{Lpiy zhTK>h^_@;}cx1$eQpYb~=PGI#0S87t4&Q_P>cqg5Ke4qM)>yLN7cd+9DVLH}zNuTX zN?m&>3Dq09_-hwfFhgM$Bcgr8B^c}V2<%h_4|J;UbV}n|K47nV7_S{u5@8at#c>QN zwfj{1RMRgX7*F4nVJ9NSy6H9$tAaLl8=?~$sb`|OaHy}uSA6HR1=EOOBE-Y`vJ6f| zpPV*Mvp7~MF`8OS^P&zQlw6Ihv^mQO**%ew=6P~u4}mqZN(8n(1DSJ7ewQscB}(&W z!##cd?Be9q2a_LwM$QKy4WX5`$9iBLjyh4OU6m?Y) zuBZ1N$Wh)KJ$~~2LC2n(W!K0NnNnZ##R7R+u@<~5p^mPMAw_S`UiezOM&q8&bJNX8 zTsekb7zQTBTM>HAo&22IoMOU8GL~rZogf%S38M6r;4!l6&KOI7M$BEPGE-AE4MdaZ zgG9LB?Q{4G7l02D0HP#f5saUq7%t9HfRm7pz`?*yr6tqf`9B*saSaSHFIlko;l4e> zS7O1w6h;v4kjjlO_efV3bh?+d7Il}D6k0&6lpOplPwuB~7?G#dTTsy98OHAg30BelhWk%? zgWjMw9B~@en$U5)+ft+2?%oCJWNsHLC8xs!oz0QYvE2H`u6Mr_2WWR}N-*+K*snN- z)v@3Nhv;0L;tOodw~gGw0?~LTtUWg=yO3j?F!X_$#~tfK^HJRav=&wj$B4McX5(ce zHYR|zFoBZ^mjVo;tlu=^kXLPSXqTJ9bNZ^sDNuK68`y;pO_P&FRjyBF~J zvnNbjCG)6;kzWw3pdqUig^N1}{ZKI=cF*Oz{8+!jkd_cwMllucsO+L{||;ww)a1`@hofq)tG8ZUV}@f^Gdn3`+d^y zDTS;qwTw?4aA1hI*;VQsUySV>2|@6=kV`A;=iMxdnhqS67v<6EiBiqc-Y<~)T~(Fu zq;p4BS(3{%nsAvF%D$hXq@r8$`s_sFZrq*KF%R8S=-!Ylb35V#VqXVt3{HMT;+JD$~sW z?eGJuNCD0Ee+RDp$Km0lE&sobXIcJlN0=}fe;ad;>i86f#fOy?-!l4Go#1Yg&1m8+ zT!L=H?n99i?e;S{MK<%Zq76Pdzu$cG*?U&t{l1btoxDhrc{Md1k*j zIXyPUFH4T-TZ$A1AQ|T>_s{s-qhxVku!t8?l6|QDo~3Dw<|tt>HiHYOqAxSbL~%KG zLVuKVmtF7^CgUgsFbq|yP$0aMrW{Z&7vASlG6IDS;T(P#fyal32ZtR1CR93^-fE+I z8qY9h!X;`!8Jep}-9{_e@Y90-Uj;WS|L-3R-28t}w*3E=9>ukm@$NBormNi>Ro+n> zkSj52xHzu#AU*Z5Axo@S>RNrRE0S7hHGY4AQtc~_g1#4SsD5NZ=VruY#m1ve;FN>=W zuH&%cgg4@{Bcg?pQ1y1H0gC=@w^J8t_Dhq|ul8x?|LW?jJ3}yxS26@?y8n6ds1*Na zIN0j{Zsl2vk5>r?UjD0s=d`p$FhS=lZ!MnN2=Ufz6qf39oEs*r(!W^yb5?e3Wg#FZ z2)Q&6==Ag>@CqlWV|f~Blg1fisbUJLzMgO2pVB?PfI~3D0ggxD&9U}zV~|V{JGF17 zmKu1upkE?7J4J{TmtV3Qb3|t;Ld5Ed;dR!8lPGFUgsm@L@8Zy%=IZ_5n*QIM{olbs ziT)1`w)a1`^VDsNO_OxBkk+H(9w1+ya4V$R(`viIO_1kK3-^45MNYS6lLi^Dq|LsX zEl(}FuJ&p@IaSNRc+q+pm_RB=vDMUJ82$)jm;}W@*zcN0qYPsWB+kWg>P)Ct%jlS2 zZTrD{K@^fkQ`q{JRB%}dsw&o#c)u2d_n74_$J0ThAC+t&!Jz|QziGzKt-I_h{}!Z( zrcP2sIcGSs+_IdPZPq~viXHJ~k=8~FQD8)L8Am_yrMc|geOk+ZDkH)wVqk;(H+b~u zk!%0?`0>{M^H!c^&;M-`Ce_J{I)3h#iRaZm4A7rJx!7X=tHUuxs|En9-~Yo$2aop4 z`5(6X|5lzG*Z-+(h?)WvHSTmcKik=4b=BBi4h@*%@TK@TM+wuxHri<8JQcwm!4aMh!`4xoXvqo?+k4qKrPQ<+pT(|hJhy~&~3Ih>@X;2gL zFX1aNOIQ?x2hF^ZlUKG^hd_!DdAg)na8J>JdkOgbxwlGXJyO1o`a`VvUT&ix>gUcy zLEP<20^vf~hC$qhLA>)}5Y1QHD2RWeD2Oo*U@W3cY@;AZ@bi*1VqUKM@`Z?rC6->uuqv|>Ux5h1W-kRzYXJ2c2JK^4SNtIzW0H||?t zH}6=i>zB6OxvivZs9+n7z6@4n*Q22g5bGT4^=L5W!ikNzQSXL|DR=Ek926R* zek*$8);Vzb$9wku3ovV1jan_Nd>d&&*>rL-3wYBFV%_M^=pnZIP$!lG-5;geA2>DP*%=N^mftPx~`WS)c##7o!WB zm!b>0)OCZ+ZT!~_Y6bZ(`j)tfCR7I96J zt&!4}uDYa+Qmrb@g0m6TS!re18Z-auF%@RE#>@n5FDEbLa*+y6oKI;`UVb{%%*`v| z+o;zIZYKc|oWiMKOjRuxB}@dW>#cHkSUq~jT2Fg7hVLs-zFK2wgc&N3v0@M9D|$(& zrzg*Bh$3*&D$rKZepUilRu5}4-j!6<)gabFW*t>^EueK2#17S+DcWX}aTT1~D(k1e zwe^dl>yg=d8c~M=uc5zt^-Jy1tD$4{_%*5C3;nnk!!KcU-a`Mgb^yo*{m=g6gHrs5 z!zbJLPq*@T?7zJ_ApP-M2oPDPblI%(r!GLGds<&MK;#nj?|6VnO}1AJ0ckL=#u$(W zrz(|QopqJtOqN+uoycmjDNE$5Is=x3#ayhbtC0(W+hCP98?4fb5^DNLTfhuZU+V6v z1^*{FUMKoToBSW8`0q~+9&h>oZ9JRd|CoRWA)3G}W^MVC1}t{UUSXJBS#r;nAp|Ng zV>3*me+$;+=FO6#!WoXyosJU0H%?ms^H!IBJl$! z;N5*!T#t|}bcM*6js)F#cR!=Zswvv!z3Q$sF58>;uk~q3|Hm@+<7#Z6k^Vn%@BfGU zPxiO;|2Cd4bpETO-3HlKAxFPn>WyQh9LEe^v|b-HBd8g~4$a2Dqksv<;SIg8LXZT{ z?QXk4*fNJZ{IsC|YX<;q(*GUqm+ZeDZRP*lc~-IiY9Ig4du_dJ{J%yZw_?fFrf02< zb2b(Kui1R84ce{Z|25aFwt@H7T{iw-BapW7|NJwtT>QV5FxEg~WAXo1XlbqZf41FA zDFmPwoz+4BO52%s5%1dBLMqX&9BIyrbt%%^R~Rg+J^x=l09*tA_r!nO9}Ew-{Qowd zWzT__3_7YMU~X@ZAgHL+sYKc1mXzIO5MJUT8m3Qg>qhrLWP1}UOd{q zh9YaXtE3lSq>^40$|$mG=SuoIHGNIzY&+<8zil-$+h-a2zv?AGBmIBu>VNhhJsNE3 z|7|>9mj3(n`*(p26#8>cYAZD8D+roKler3g>6?T$7XbRc!Q{1TjK`Kce5+3*{V$wn zSH1vfp#M(}2d?}-cych@(*N6dTuIkM`&+wz7b~U8|2r>n_b9&C+1d@;;R~W$6GFEf zHP_>lu!KXmiT_Fwp-lkgfY3E-fm*)bw8rb(Tg&I}9ZdlGqqdqDOHb(AurL*%jL&{5KHzwPmzbGlFs)U7=-yt?DpId=Grk9RoYo zy4}h9wgYyq(YQx&#!y#=C)_DT{muQ`K7ItNgOxE4X9&h@wkZ4GdOXwfc5AAY<69fC zV(m!O-dM52`dG0;J67zHfp72lT=4!ZJrfm`Zg(DPeEV(Y-ssw2N4;yq)aj==Hoz*u zdTa4mI#zEjwCYg3cie5woby=zsPf4NLi-_xHE@pIdpB)c;#&5VbN)JrM_Yk4Oc!II!^}h7~O{+zIkx$e8|8W0_Yyb7=aO?kZ zE6H2dM(y#Psr2lC+UMu)V3;X}aPoB8(KMwY{_P@9C z)Zc(UizvxbF3JCxg;OKTvqSKubl})|otEq0S#myyAKoVL3Pv&SU*(;-Z%FK_>VkX| ztigx$<4s#*HShm*qyIJQe}-lI&x5W1@9jL!{{I%j``r%Zs1NSv5RGMH`?arrr$YOA z?*mqh>bJ_3pc_2Tio$mC%zm*Ae|K}???gbwt#`>+@HF%PtKf#^|NY_qVM+c!*yew_ zm1mV3V?$Y+V7AK5Y^8_rsgHwLynh*l*SXASm(kFdzUVGpY&0ZMtnaWdhoUbQO1+Vd zlhW{`Kbc``7<<|T*^|rAHsY$!U$_j?I~BZanQ@S^l5%+BgDR0My{ zQ(StJTh@<`&TYB9M)~TWM*hEc^sg57AEo$@gZ<&w|MOO!75snNhyU^!VywfH>w|yY zn1MlyUe!hZswrQM{FAF`PBrYZOAY06f76Qk#XAc|J9>^wTb^VaP2=1 z54ZV$Zsl3h{=*IbWtP3+=wBN-GPm|?{IeWoy!&6;gKQ%7mv|cX|M2mnLs$MEJl@8C zzm>lCs4?>M5xk83XFRra%(n#vyMnDi-w zEUQ9sd3+L_oT=vCp1m-!(xbOi1^ISl&6f24M)Q9R4odRhqruky=T@FCM*mk0{dF^J z;HCn8$=2>(tgl9<@uk=0_PL#>mi}vQyruy_1N}cdI5=?c{~tZx>i=)$aU)#Ww7+X! z*v?b7a0}+iTwh-yLLzy6xmT?NIM(Iw2;DEPE$#{1r_oc;%X?iPF-HOvR!>@Ksw3l! zz^6_T;!3l8EJ=dh*r9HGfx$f?3bNqh9e|4d@KP&}p9_7&bI7YPD94JabrojdHpzb?KGtpD6uH!FoDy8=FRh7bzi5OXH|*PregJy`VL6;Jf&`HZhTo%PC3vD{ z4z6;_^@@@)r{12uFkPcl$YvL5VdcDJb&q#9kxQpM!d)O$4;PBTf_E1eobO&3Gm{@k zelv!fpeZu3^Bz-*;xLfbo<)_fafVrP{KU;nA5YV zRtsZ>i=u}Y5$3I?|rwk!EwyP`dy@$u!|&{j}a+% z#M4|za9U|qw(Q?uTD1Nup1Skj20}lJr~dr6KRkFmaL<2_pKSd3rqUazdI=ZPCTIV;_|f^9tRZxCzChYW zjdm+E2aI-V+(oeyM4_RjBWp`VEo48*45Z2)q$q>$u=D zq_^}T>4p(`+D9ztOZcT?*JwP$_EnHvaGUXK6Pgn;ykc@_~b8phD_(P@D}ZHVU@WLiXgZVuQi;Wx%>u39C9 zL$GoIUT0vB^>>MId2|bXYWpoN2ts+gJN@H@9KUk&$s3xr%-f_mM!h&1 z6G$RNCG#*83aZ3KUZNIGu70S3Lu<*2DZl&_!%D8uh6elRe+(&dl(A+%v|ozgkL3aO z*7j}oHeAu{?3YQ@u~v2nom8QPz~H7_Fmfx{c@w<36?{9-#P!6eI=L(h4ug2bYTdY?TSv7spVk;1;(7{C^3Lx=muXu$Dg>k8pR|olV!#RC! zBB_twIV zM8fZ=N?@t2P{6Is<5R`Mb0bch^(UUJJ#}WEDo>Am)s@Y^XZlQTr+z8@#eD7k`9N=MRb^gJ(%$utF(F5S4 zkLh$hsx?oti85QO20{}XZ*Hf)#cXd*AD4~8R#W|k?c`ct zhgj41B9F)>+b(>wtlN-}?Srn>+vC-wkaN)6W{~YSRHzkhq6mOd|?(3E=>iGW# zQoo6(f&UMl?3dzyKHA^L|GAZCY5woe0CyJ>!1?L(W_Zvh(!RAcymm6aRpnb7Y$m@# z-TvP|>M!v$?f=J*T>1a;{^M=@|66$~?Z3ZZ>aW|_$TbPSYQ|urxxQ*p-FTX>%BOb! zub=u$JWc2Sa{TY%!C-s-zm@0Xx0nCxCdo&A)~|wcESvDFRm~g9^`)|L`OZal^$`mcef8L!SEdqcX-u@rolUvT@*PZ_ z{f2e$-lj3T%|@%zn8{(?h@5>_pIZLEe(EprH1Yraf$RVMcz^5veLK%h_i?fS+Uoyq1mwFtmj;Z?V6Sk3mQMp!*N4)bE@+ny zYS20zRMWMKH(2DKhCQ{Q|2J*_^=MGe|2#b0-v8dp(@g)@qv$JUeXObeW$1Q`JdggO zo2AbF^o~x;tui~-)oDDy&=vDJ)>Lns%F!L=<+3<7)?TUMjpc4!s*M{<+gM{qxh#vW zb~2N&6C65#^_x13+*&RD+GOSF+3B(C*`r5gO46X^s0c7C4b5=y!(iAJ(bWdXTh%j4 z`A<`n`dFC~#wYYwEgDsnB6cRD1-fHyl<3((ms1>e3p@X2e=<+7uV!pi3`1k%#1!T3 zLTd|?)zeB5<;?WRaxZjV)^EpB{g2fF4NLh=9cZQ0rhf{lcl+q4h5Wxx>R0hJ$p3>! z2ajC+&*A>o{^M4jmGnPm23S?2uo=0D+^?Is?x>5a`AuK();_iS-{8%+aTcEAmH_08k-0L{>7qj`xoEMMj$UBMrIqmdK7lhv3f~z4~TiS1%vDdj06t+wj#(nfYwPS8rN5>fH(3 z@YOdJzIs*ey^R>WjTqco1lUFl2IYvsHyshYr8}r~4{}wwFTc^+5pBzB-$E9>T`%l? z>Be0?oAi5@u_jHt?29RNC#@RZpgXeNu;T)&U)qka3Z<3YVV4?r*ZtY5yoM9B)dAZT z+8BQ)qW*^5qHD;4X>jE<-gw~7lM#TARn}X7GDEHvpE~=ms}$Zw{Ey+|K`H*qE zbt})R_Fq?suqc4>QN}P0U|i!g233(#yz6~s8TPA!b&A6R+!wJCEt@u7r7*XUl1?0F zXwt@P&9P-L2V_SX#!2c}wvKZtqE)vX3)}?r45cw%%u&KzYme1pre$de8B$ky0*-vc zxvKAs=UEm-1@H-kXacjCft}Zr$qx7g{>U&>{nWhJY|OZGh@&7v-gBj=%~Cz1={TbN zNUwy5Pk@nVUgHi7K7QlsSGIE!KtQ7whRdpJe3*VzScW)T%J;xJ(p&>4zj zFpcq8T6#vw^dTT9hAg^5LbYJLhe_Dk1qqspvjBJ?FSAext|K-B|GC!#LK}<|QIn$t zQba%;C8*cwJv+ZRXPBVQE;z>XIZnW@$LAo72<`Ny5$lV8CHkG-_}`>2{?#vL(?0)4 z|D;z*zW_9b!TT%)lPE^C^IeZ#r=9P5WB9)FU60Mv&UgP`XBYemNrW>BPM$rd9e!l} z9R;k@i$VnZvK+y`cRH{C`Re)E`PU?G8vFn3`O&kN&wKN5({0r6|0ho#A09aSfB*6R zHvY@4JiFjDUjd!YyW+=h4`z&|bky&ghK0Dl_Ygqv?l{I-2wusJevId7#yF|?31>+t z_IFV|NP>x|2-LQD2qFp~kU1(G^NSlrTiDU`!tZbOqy#pK3%jNq|12L^@E?haik-8p8#ZK_d~( zVnzj_0s&08U@)B`0yuAHl#2iq&k=7oj06dbD8OUE&O=T-$4HpF^*Wv1UGV0`-=CfR zz0YSP&(Y!8it#aRI;>U0znyJxLhBVHWU4IvqiBgBc_Yj4@`E5t#Dv zYcp3M%;iCV6JF_ma!*@XY1Hpq&bNCyTmZYnvOTTMLeJLMIi^18asv$Do4WTqaMCUe$i5= zUaewt^Z*C@&U*LTgI(+Q9-v6903oo-C7MD4=ZGP~e?lf!=@^+iAE;0Ad5RO1@cm9H4kF035K$}-IbE@+V5}0oMDzLE?|tZWjNe!CoWBYV{P57zdaLc}Zjf>Ve}K3f=>pF@eB5n#y?v(<}-RcsD}? zRqfjl2h_BE+wTqfIe`9Nr}LKbHGn686~y5>ier!<#djQ+Vi8g*5g&cAXABTS-oy#xCHZQe&ruS}-zfdt#P2szmZmL7oX>FcRx!finVd z9Ey<tjzQKvKPfw%caFSVHyJ@8Czw|pDm1fVNK z76mZ{V??7+QOnfbR}TS==1~lZCLR{60%H{8>rtokADoW=he;zxo&Nxu#p`(5Rsj)i zTa+^2_7?M4+)w-m7fyt3?4^|byZ{p%qUUO7dkdr)olPM_*KpzeE_*3IX=`5bE58Tn z#W;X*VY58~Pxc2hUDN3WmAC_AAd4R9RB_^mV0^=#(%aelO0O|P;4z5B-xQ%r9TIUO z<&G<Z-MQW!r#J*^xq+()0>cf{62}FJhJF!G3uvN@AyhJ~rvZX1 zNFu}*e4Yy#JlBJY*e&!SuY45=I?UO#AV_it5`qF|-pckmV)8lfRHEZG7eHx1qA>z9 ze9d=nE-Zo+hg6+gBue4mJCcG*t|53#BI-##X=A3PPgPfpbO4i1hyhgMq*gUw@@A%L z75cgwB4KwVPGudw92uWGf+mO{zU*95YmKy$w2$-KeK>`DX5CsL4lw*gja7hkiVd4DRwIT0`PQ5F-enG<h{Daq-v2(NnRdQ%CdxYTrgAr-+IPbvmcIHuwad34$aK1^kyN+Ue>C zd;;CB=U3;_5)g9f`V%;r2))ZgpkmON<-jK;Z#&hwl*a2?W%FAjsZ96;tU4|T*G;Yl zoG1zmkGfBC7W>Wc8Jso(W^l?OUJZKtgWkXor*O7w1jmc2v~C>i$>bGgrvy=yFc)_2 zkivn!5#*M?05r~~05UGm1~c<;q`9x20Uv<82*?9g=SjY)j^fzq=mhJ_wF|rp4ZHj!!$OY z1+!p)Sk>Sn5J6y)#j(_PMHF0q{Ak&@3fn||H`$E9=g*f)W6PL4C-W1j@M;qhSW3_p zKTepsuL>-MI$?}BlqpIO5qcaxd!>nznEXWWT#lPw8=_Bm0-&-@)S`emxEpc?FGo`t zAhmkboJulxI)=uocnTkdoJnYa(s!b*a_e*LTrtQN=%*$=#ol{p%AC9D!_&mQ6gCyrfpm3 z*_57?;MKs2y_!cTsR1d{cvk}BU&XTXZ|-uYc`<;hvUn>~c}pb$N>L&vTb|mj7I{mq zdigCylPs1eApMCi#CJhP8J+{E^dsU>dGyb7aP;EGy!B(8FoNS4kwP{tHdB{IQ`(zG zY?h6=`~k0N7slhR>QOhyYxjkwv#&n%i@ECWs`mHl2Zz;6)|(gS&We8Uzeg`$R6sNx zOE^gWTTe0i;+Ldkmi>_^_j&;@UW zrzuS{j0M+cs0ehE3uQ@EB1Tmf)YVi;P@`g8^7=pm05ZE&h4ffwYFz4y&`y*akX683 znI~VqJ3}!1Cy5w(odl?~FzU>zg`i7Hj%)h66(+r3<28iKT%KmsrhS|2}Of=^CUW=OkB zVZ=xyf#S1TH7*C8D!agorg~ z>N-{2E+aYvDMAFGBusIn=81n=qitG5w=a>E5?G<~`UEb&FDW2Of$vJd%AxW6Y(96_ zDMP?vp-t8fJm9Y&j^c>FHkoq_!36NGamErawu%~i3eo8qvWYYqAu zR9AuO%WFzOYQRfZ$xw-EDGy5Himsn;H|@uzaFJ4~7Fz+LZpSselT$zWrsbp_rD@VI za;476DJR5SqOUqBQE#dau@UmRf*}Zro&Uv4g0U(J9z+vuN`S(>dep1(ZHT=cQN~Hr z$tgHKd3Hv1mTE1?wP?7yb9Fndd4o}ZW&pv7)<% zm>g{Rx}ssWNX4cW$pl>Tt?_4A%iEtgXua-Agoq!UrULoGnE~k6Vl{&u-q6kn?3`g? zpd>HBcYwRh#kFj&q{*n!q1i{uPXH23YVac89y@J}zUTwUZy3yTQeQ~w+er>$V+VGBTbwjk)o-Ma< zBfGZT%uVgvY8idioy%pF;~BnAohg=Z<;rG3vACSFl>*3WeT6-=kNPATSWx8GlXNP901)N7)a%gKAXg$>~ zq3b8v7Fgc)E5xobd2cO68brLsP~e>1Lvd93P;I0;P5?uO02U z;jvh-+TGw~u-zY@;?!9))tXZal*OTK5M|yBlTdT#;!?f5Dn0sBH*%t4v|6_rC1N@& zptVwQ+PIrK&N?I0%GUh)Plu>_6`)EV=7s90K>aw^>p;D51yxl#LO)PJMzyB!hv8sQ z1!}|U)W$Y$N}cL$*ionYhpNpexr$9cq)N?ZEkSar_Fu&%T$$WOhM_z)X^`R6QU}*4 zX0TMVvKA}Nl4YUN`0CX6NtAwAm4>iQz?C#v2UW|G!`?uMl4T&aBv}WmUXHW|zCw)j zLoW-F7M3+8btb$P7Ii+b4rTR0^x4z-K;^b?J|Jb&H*Y6bvV1e!`z8$Ed_$w`-h5-* zX!d69>jp{LY@k9;Ho+|=COfeG%jsriL$jp~ifAU-X7dc+IJnel)8c&-?DeF36MD&r z&+Klq4WET*8xua`lFV$WwK1P}gc^+bD%xDxtgpUpd;Krjb_4Zpw(MK3y*eAe+IB4y zzvbGjG4`u%Qrr41*P3nhSKokR^tW6?uK8bKkXPHN*Zgm}mK{UDmJQb!0+xd^w6zAb zTJ)WLEn-@I^0g_naqx8+UJJFfMeW)hQO#CdBOD(%{~6R0AgfJqd;p!}rw^a{!>14E zh7XKQP2ALZ(bukXT=b1sepnp*P~w3VXGI^F>XV`ga9tA^GKPUl$X|WXJp1wJ*gjXP zlc5{F#JqDW?``u2+&G8Wu#&Rpns(kR9W&*JqOR|7SqG>3h?!S%fvdx1GgN)V-0uw^ zi_2>2&u7>UTKyrDx=H;wP#~@h_--r5=+JQ;gr}HtY%ZGo=^O}NS!B5C+i@C-CZ{nT z!?-6kXcsz^8gKp=d<`aoRwgW77{22=0NZ$xd&O~Sab5*fB@fduW1LJumar%WP=iz? zS>zd`B8(D;IG`nvD!^NWlLlOMBsdiY$w5%Yr4W+ZrSia&ufR2b2PO!zj39@@2`U5_ zfeB3^kxfw~IVu&VW3-nOf2$=f7dj1zfEw-FAt*r6Rb5L}%89zQSyoX{H>u(h&?X`Y zBPYH)B154dD&07mpkNWiXy-Qv{z($CA{~;{3`ES-A#*|xvm}@S$d?U`LS%XaYdTMz zjCq%9IVe*rgW}~HuE{2Wz4Frulw3sw%X9b@BoQAn>UOO8zOf9r>dvty%SoH5_TM;G zHh0L%$1+ju*xb>A683iKmA+!tJi@#W5b$es!<6_%jx8N z`pbu!>Wcn z>caXqw5ITTp((5cTdgFl2VboptZJ%BHCW$Xm3wey$8JI&sD(6@>q$(?i)bE+bEAF> z^0&&grvz68KFw%ZkXhk~XEH~o7FMN8}83MyI zxI7Ryc)6FKrJF9`st4QPSGVOJ?P;z?ncO(|8jc7(=M2BuMw1=l>X4Un>vkrMEC_{Xx0;Mha0!80eBd;5Nv%-rwF17FS-?C|6#5 zIS{93r^j9nj8l}zIDa?^ukXBBbcv4p9RrO{*TQY? zum!)qv!?MveP4Yx5zSro*{Yj8t%KU!Q4@N7KNV@H@1n!_ntP|9`Z`tttGQ1KX1P0v z^cW)ZC=mkc&jbeO6p=_A)o~J5h=->*1V%M5<@G=+D)Pn;2G$WprO4p09!SSp81I>2 zg#suqCrIz`Ff6FA5w}HP^xzpv2@0SP-h3Khk)X|u-C_#Y5~8^HZvjvm2sFdfiIpy7 z%BON{nWYibNu-(}w#Cq~@$ie4W?GQO&^PizS5%Rgh81hZxK68RU}r}~>p7FT5^BM{ z6D5I4t02_IviK(RCx|vK?9>fZo(6yEADKdf#9q0>mVJAk%g}7E%OeIO(ip&2l84yfZv zo(n}LN^(-v2_OB<_P;wZGwH~l8QJvySLQqsvzSGE zuZeU$A`z|1^%YS*3M>k;7?OCw8_Mr?i~LlQyo^zdlc^XCT!2)}BUh^`t$}BlL=ShZ z{_cu&JDpt(W!M#M$&Ut|5m`S37Bb<7E-rro(3M|D0473ptzNs}_m=~|-wM2#vrVo7 z{?jua#%rq|Dy!$Nw79f0aGYUI#$gfy{+GIRG2Q)KjT_7uoCn{@QVXY&! zN5{Gjil~(ucSdkFohkllM8T`&n_94O804O!N`i_%5NX4bg-i}bg;Gu?o+-!RyWGoH zvbf?C<pzmAhk5qS}t=ohZAoD-zn~7Au{TiJZnf zFJ&C#Yg+7RxyLzPgzsH+1!KNUg|Q2tjY$-UG{z55Z`xDAZDKUV-k3GNBSskv-uLrj zdwUN-D#@9>ip{K8iUC6g<|vrKB%*VNZWQUm-n=+}2qqDsOi?F|Hv&-Po%pMXH~9bA zd;aFOaVz^bKLt+O)=4I!C0Vi^&FxLyByBy{^?H(Adv}?g3KAiS4MlJX%848MKKl)R zgCt0Sq9S)&;g3Wnf%hIj-~o6)6Bwp?f9F!-XNWv!?B{bxRw5l@hUOW|j~i>F3pxh* z3?C{=NIt=+j=0lK6cBv(;&aPqT>l^a1Gg=wZuz{0ZH2 zPJSKFqm8mTrSMO9f{~ZpAXR6gmAXh}4E9n|Z_fBrQZ{aq>D#cJGF<5~H8vs6K_+_R z=3wZKDD0^C=13=DW79m*lKrPrC9*hL5~hRco2>eSh@zp+Zs9^z4t4%#10^%C(|fzI zq655!VN7SZ10^LhG1-J$m=YD=Lr-q8%N8e@Nn92gr11%G1J`I`pJAeAp-*-FQpbhd z<-zmVm4a?m;<*{Nq!>GtX|&jdD@JprOU{KHW(n#3b2CF+Q>XGB&hEL$7tZG>$;3TN zL17YNOTyl(C<0+T!<#rLVlCAMFhzi5lun_x;Vedkvz8%-5?l9Ps$(0J62xUPv6Bzm3uJ%S&l43#zVilu7cO?$?U~06ljLM4lA9B1BR><`GLUJ8F3u zF3vjLBhhniO)hqcF!Q}{lKc7yjC^AsI}6)$g~3Q|#O?qy)^w7{&9DUbA|P=S?gD_B>$tRPki31=C<2 zAy1mP`AdwK5rRob*nz|YGI{L;4E+m#=mp^tk!%9U3J%BTlk->OpdZZU!(etfo6i^H z^NZnNG(LZIF&>>?z8VDZyuVlshKpG+8!ZO2#k@ZpjbANBvy1TPv6>>GXQ#M;Ns#*Nm(a1WAC_bzdwnlEj{C>;e~1E;@XvBd-wiyNBbGflk*u0oUU6ZHcVQQcLq4n4PN8 zmjn|fZtg|s4n;EivA4pM#aQGPnrqnRq4x>hf5vIRPAdE%{eff3d%}3>nU^ZBXQTl2 z=>19$4E;m>|JOW`V9|8~8J z5W$O#UD{+bR%LMcs;^toV?K(0n{Lq3M;ldhO&@rbtX9Z>DXGmzsc_J${r*nt-;g`( zGRL;8|MQD+zqtO#gW+-gAEI<+25wUYSDpgCrzgsL5n7W8gjgtpN#Sx6DGh6gy4BhekD_@Gcgu(Yh`i{Zg+3vMvwrQvq2Wg$(iYa z*Hr?qygr4jU>Yzp(i9T{{#`oVN@KVG;HK+ZNjuwwQPvA%ugf3>A(><5sVThpoAbD? zpa=y$Mh)d^U8M_zG3Bc$4VlC?Qo#a5@93`-ffzHbRv3lZJy>pF3gZkR;(HXfgi61e zlI<~Nis#9GPQbwPghGVYRB!|MVHA0n@&~EaW86k9|)k zjI#TrTu>M{P5_0OHzl5c4n0>IhSU@A_0jXV7?gY0@FzwjYu$+%oKF~?E`d)xAl^y= zXI_`Y0linc>d!00%-bOoX6br>tnelZ3>oT6AnsFu62yTQa8b}^8f!YE3ShaZk1x9u z1ZiO1@X-5cD<69m1SL*my#5O+dZ)4>wVDJg-HU_HBR>ml%P3fG5dviPv%)< ziu$`i;8Bh!K8%tT^z$f$6#F*C3z5Q|Ou)&DlO`~PF4=%F0Ve^Te?qBzC*jEf)c|KG zbAp#{;w)Sv4UlrXoQF+r@(3Q4ZsIHQUj(HSa6TTNjXVGrF0D_no96+88a;>cXCrx zRizs;9;G<`8=mP00YqyY6J#{g?wk6V%%iHGtZ*9so$gmrf&-|W=5z(2cSZV14l(n> zOPLRodbZwV%rN0InyoPYl$!=bvv0LQ>6WIhC`4vm92nKKqc-;kXTJ8;uZ=f22vlf@ z84C@Od#V4>o*<=Xw5|3HG+*=6*UuZALH(G>yjq79w=-NLbi zu38!FN0=8VuN1AQ?sh%(hpf$-SKa@d?*G~@>E_1|Z{Pdbm#oPhbIbjIaCSB*-2Vp` zqtWsHe~4l^tJ{hUNBUyE^hO?U{(e6ay#)F%fe<6p2>V&Y|DN3_Ug?KZ~b-973h>mg3+Mp zO*uVj#$A;uv48yV;a2C@K9yC)TRu3<8Ud%7?_I~jb_Wvl5`6utE?%Hx*&gNftOI=T z_~>t|uGYAyPwwW@ZP<_S97dC`Ux67iESw{sIZZx3o|+xL=hhhX(*;1dZbJf+h#o=E z7v{U2q&UN~%>r;;5$&Sr^A$4^VV>(K^)z{~^lPuct4-UAUewxe~oBX7>sDeocpEzCsi5;`H&+qqV%ZUlJ)rH6>Qu z>%tTDdh~b`#?@@A1A4k5I|un(y~r*fl4sGH>a@dhc*b*Ze#U-=>-U?*BK!h6Jq--S z0sGBKeJxF@Bo4}7@4?R-7=?=v1@sclI^lbNK^!`(pHUpNLj=sw9Bv2#1g~>j9qalc zL{Y$Z33jZPcdn3tXESaS|K{dBt;e+Y4{Xq>28Bb(1;W&{=AJetFiFEW zTY%25N$=OBQ-Jne>1Y=_y0kE zbT%&R|7YXl{(qQKn7f9v7n?_>c?f$4HyIOz@mRB;E}1a%ZZJa`9C)9?I8bW7cQ``c z8f7qm8Ju`PNQb$41|~^CV$DqQ_rcY3m+Y9y>&Og3Y{=tZ~l5_CSpY`*IJL`l>fsES%8n6*%6B}Fx8jamqR zEr@3VOc}dq(-yauHCwQwFByvIlq3`^OcG)vMT2NfjIs0WOpS)omK-HucbpNbep|#< za(oLN@@*I|Z6sAeYD=L~By9ao+6Sg)Su1dD>1w)+_OXNvt%>7AFbqioxQ^jIvJB|gd+ZLf+{pLu4 zz45+D+LfC2gjh6;7FxJ@hJE$eK)H{bvtVwNL!M& zSQG5!EU*Pf>n;@S7Tztd{}^5j z3-Z75pg%au{|-^Ut^7|!v^t6bcNPQU2)#oK7647K!kQ1^0Zj|KF z*Mwc+=FIII+PVCzG3Nu6hWuX(02_D%Zqfgo4NLxiXJ_X}|G$G2lfl!o{psDHs7N-w z#-uYoU5tV&XeT?Z09J2x=ybJqA0Knu!Qq+6A-157+aA=}x|eh;n@LqGbL;BZ_pVOy zteZyuVcV@?CZgR7>Ys+A|)7}MfcEV&M;h8CA019N4=0zP15?a(pz)N07{ zSNA2)3ySabz)GmMKfd|v&D9TYjW~VGE&lf@UYo)D0Ji`XC-c(tTbQjTYN4~0S&8cX z?bYjVfRDR_lm-+4m@fI^YHnCyg6V-?Pq+HM#&m$9qBIEpvE4$T4vjVoa0Dy0|8+p6 zw!Z*!t4_ZLrlGcXBh=K+*TFRu@NR&$p1lUFR<64Nt<<5{;HcD=yP?!d@fsXj)$Im! z+tjd43kOAY4MbBr?FP7B30;G#UbpOq+)yFcU^4W^j$jMw;X2p_6|f@$rs=H%%hbDW zz$&!3HP9{z^`omh_IVGVR_-(qt8hqPS ze2xUz6=rpK?HVyhbS1@M9j20Yu=-S`t?~=-)Qj(pTSKn4P}r3~Zaei;=m`$K|L&PQ ze?Kq2GqCGg?3Di8Kr0)mz%(*?l|eCk=j+f`rFP;Lo~y3oXo1JaN$GV4mr4ReOM+7~ zu6R#hnQqOXO}s?m9f}blw<(?>^Cq#%vg8ja%U>=McB?pDAsA(=zvo|-VFM7xVHU#Z z4T|8sNCz|lW1~VAt`Xj3mUk(FLD-nGZ$eU0A*NEXm=|iTaV|?KbPFY*81(DqAu_ax zAqkPeMl1+aZC7!N89qg$$OTogpvkpzMP&pj(U{E0B^E?xn@T)SEhqnHEe-WQ^sFZ2 zM%*yZMGdq~|9^2_^#2+6`^WeX2PtLAY(AWrxrv&(A`vw!ulc_%w%&)P=U>xaVC#2* zMFIMD;du>&t$Lj#4e!DTEzPKMGG>r4yE+l+aJ+)VT~?zyZXg7z@ zyh-edb`Utpo5F&RC&jZ`hQT;$AE43zy7Eg|9dn%=KnfOaSj?;cTkXL|KrW>Db7iy z+(?$gLE*a=o9Oex5FFhs|CP#a{9mNn+7tgjKP%>c9Sz4v{r^Eq<>B9#-?+j6KxbX` zd74L-jiI|BR?4r#)8-fNlJLcwRdHUmWAV9Hvk<)$!U!I~|G1lIh|F7e|O zvZHe6Ta~u_-^Bu88~-1Tiu(Wa;n`9De~|J7{NH8=aNGu-W*gYenjmLJ(B`8Z?FpW` zIPrgZ;3bipmmGL-VPp8Om06#uhL7%@}y1kOLf zB_h7p1s_%+VaD}1%?KbX6h&Ye;TgZdhw<_yNKpi{@D4FAJ>xly1Fs8Wv}7y=JWEov z2*03!r*-_d=RWu`j_v`DSxZV1NKgu*Fh;)Tzj^=H_Zd!+*9EWfdW~c7=hyE+5T?ZQ zmtl6w{^jI*{_OARDf?GGSS?TKKk`p<7oTcCGdTaWNm%YX;=S<6=fr#A&)_HTg`cex z@5TRlUGQg^hIm83^_#cEqxXHvwmd%!5Ip7eQvA1HBBO9x|3Cia=Iy)pJLA~4{s(7; z_g2BxO~<3dSIPy(9%oMU5Gxr)qU!I z9gACWTK_6(@@_cl*Z*)lD(?S-i}U{Yxc(1Ox)$jS_#=wefZa?y?*+J~SBNMg%qQ)m zr8-=qIQ!{Y)8yRGeU{W-fR|2h0__J3LjFQ0ndK*iV)92YUElY_gXzTcrqk&Rl9l%z zyMQSuB%>Eu3mLQlBjmLii^KCCz%5MS8f7R24&?#7L1dnWEI^J+1II$)@)VjbmA!{a zuSscG{oBBO@E*Xl82w^eJ0>4ih=R){$$WI)lsh?;k60?;X_ju#l>Va4*N_G8 ziS7Xd1Ok{r0Ou(t#2l3b2gC<&|NHjsyXzm`-h8ZfnKoHM4k? zDZyn78rPAkfIGeHU-sRGJ4dx)8(i7?0hqC5dG25s1TvGi&d}VP>aUNkQ-o1%8g6eN z>({KdkXN!9k|Yu1eS+YZMwUh<*X8lo_D!|n;=i@s1Nh5|9x_vaQxIdB?V1fUg(LI4 zIpQ~|ZaA-tChGN^uRt|lz043EaQpYB1WT&<@^+@$5V|{z_4R_;7Q6&PO#d4;*6)6I z^;#?44neimmb9XM71endPe3(2Vb<`F>jj$`1ZChilJbyIn!{G@4!RqBB@M^ z(B)VsI51srKo~RjM;}Ws{RYRY2k`aLse3@r5Np(NYquZ3Th8hfWpm!j8D$mK%~zu} zx_JfFJo5SV8y1f~+Y*ku^gt4=$hVEV39H{E5dc?c>`swUMsYe7>g z_e-@%VNi2YHK~?K5!%w+l4_kK!DXzC#0yV4)OJv~)>Iq$G#pl=nWW<> zS7U8zj%ktV`V1hA`qAvR3kqcGJRdF$+sGaAqzs#-`luK5nN+jPA$ytGt*ExA5Sdm` zU6G<>mu~Ep>e9H^Qr%+Q+fi*vZZfT=+Lk(HH^SRe-GE`U>-awa!!~lWI52E8;mcDp zYYqFs#&R3C| zXh+W1g6d)#qS|vvIm1z#bI8sr>0-!*G7MoLvIfjKK_n4M71QxJdrzFpB7F>0N{ZFuON9)ZA#nLbbZ3lxN6| z$~g6Tg;K?|MN2JUB`ds%0!k|X4)qdZ%0nnuf@|Z71R$}Dc zVQyr3R8em|NM&qo0POvHciT3$I1K;()~DD%Ip@Ugn3C<>I+{t}<2q^Uw~6D&cH8Hy zv@3>4NWz#RSOB!6iSyl`g&PT8MO`eX8FOYe7K!V|#=c?i4Tb~4`$rSRL~n+s;s3dV zXK!zB@5Qrc>fgP+z2d+7d(WTPzxDh5gZ<}E{tw*6SV~XA1!DhaZ*SFhij(_`JSZl= zV#W!L25`OKL2;b^eAe6Rz32p(`;5dwy?Php2sllck9PqD*C_IF09SbCU869;zy)Qv z2gd>k2Rafs0)|JJF&u!PfH)uskU$*(E?DA=gkj*AU1J8EL_U^naqv+DK8*w-kt|1| zG2ev=;uFkwAtYSzUEo-7Kv4jkGJylxA{M;Y!O@sRcmQK6;;=KJQ#^o)5b>bjSGe9I zwBI4VoQ-_V2mSt#hzLzd&!^Kq;N#13O8~`2RSA=m11Wj{AH4&MA)r zM*RN?Fqh3u5fdcpjaXVC_o1WT=~3xLQ+?ZMkBOKhLp5vDx&7A*sDFi-r>kCDgxKHf zJ?lO1+&9WTJZpS3hAVXCcf|mj>HpIgFZT9}^#AFTg9rM*iDzpIehldlh44!4c_33X z#tR1%7!t1QkM&SA_1xNmql7UW2|(9~glHJ1C29sc&~MK0m~g>nia)3~ zr%4!|W1nFmC)Clc3_4o?`5FV5&g|bOXo|BEP7Asofaetc8PD|Bw3g(BXpE0S#JOzt z-1*%BO_m4n>2pU;)KGdjlW)zm4BeHV86IWqN)l~NpcUqq+LJ@{P!xC!^BDVrTw_yoL_+L^ zB)Zb0=z#3gh$3EqhXh65{+`F49O!|Z$Ji&M8K7*DPH9N|ng09=k5Cc{E^+MfVR(aP z{4tPGHXSLaJPwh-Ts+NVisY6l|&IxpZ61v znKw>IfcvTg-rtHD_TnTANi_EKJ3Fs99qItwu5?fDl)#0;a0c8?F#^OGngJb^FtTT$ zp?|?N;R1#jrYK5O8$QEG-~dRZwwV+Rc#M49t60v8i3!B7ct72;UQ7co6FSgX5?@UvJLT9qg`5ozY?CX46M`6N+6lk#!QgrS3`h}b}jwBTe z1|V31rOXPC;S=iEs$%MC99;20ge)$!|coqKmS?6sdldP%PV3 zk~Q(Ml9OfI8V!I(Sr@7R+z>GVxjEHNk!_`&%hs~qNyLQ|`gz9|$jxemLXI^~o#8(d z!X$C#TuR7*5;2i=d?ZmDXq4}V2#&*11wN)yyPz@gdEeIPfMgq|ruh`2CDL`S$_v>= zj;xk@Y9_3n?buLQgV`ySt)sIIl-6_RGKc6cO?J~h7N@^vhIKUf;DsGm6BS-N9bP9T zPG{IajhCgz%TwgLr^)HOOLTc2Wd=iK&asd}js79GK|~~tKAK?v>Vix$O~e46bpZZM zC`+c2x&Z|ACORAm%-)hPBpmxR3itq?OM$rqFhV4hgg33f|E>NY1-vlC;rT4``2hCi zE6OCveE#Ic-}O#D3<9Fp$m|{q8cl9Sl;!D7L<7T&o5_uC#_o3gYl`B5uKHS0KAvl zaR?!~#u4V)D8n!ci24;XI;;fr!Z@A1w_SN@H=gdMzvWa`Z+7hTY@Jj2T4Q>V8K+qM z3WrD{y++SxnnGYkXi#`5G^Q5b^2GNs=NA)(`Gkgn##oKt3B%6&XG#t?bGXxy?JqGh z#l>@L%&@j20mFYLm>UaRNTx&-mv}B=_USZE6kRg5DaBILE>bDcc4dD9Oz~9l(t~Hu z-;(sLA14FY-`ks3j{Mw6Ix(X$W`Zb`G~&;|(%iaAVixOFkbexUfOmYVVQNaB<|vX; z@bpXduE;DE*{1=PMYP775~(*71>d7k3R+s6JF(@sGa}xaSNJsYaV(_lqy<7R!s3Rq zD-w>TYlET>R&%wF2@oHBYXg=ZAa0A+L;MG+N96Ofw;@h31v zGszX1mKCjF>!V1%O}H`+*hZB+Py~((4=F83L66D4XcW%Y3?JLJ7g7{>Lo);yP|gIE zW-s4`Xoy4Sr!)HFQ=@z+j3Ah+`Td-Z&m!7m4zH-xX+W~zp42^zN}*p5!2Ilm)JI{b zV@7WT8dp=6o!m^Ym`KEe6So}j2~EO4Lf!}`mUI@xd)?l}1cRwKr2!6=eLs%FSr2}j z;K+(^s>-ff4yn3ut!l`5`jj#38-3Rg366vYmczNsL;xwWWpxvYpH@WX>O25PBr>K6 z8kqq|(z$DjsTO4;v5>qlk}|JiV~kD}mTH=#Ti7lR5dzEwiL@XgJKyPLdMye=dh`0D zMh!!-jYJ~9l)Iw}_^H6hc@O^oQaY3X)AN^PApNe@x!gAe$8Bmv(rhp!8Z*rK$@yC( z{K*>^@zPJHG$?-U7z{u3eWi%jiYykSyeZOBb^Q#BENQaXWt(lWz47T+$8^|Va7quk`OCA z)y33N*oaC*(3@KpHX4Lcge@hN6;lD+0~C`V8BJnGC0;3+J@ ze!_^D#d4n#>{PnOY*?(2a9nOxQk-JJv5$L6B=rgs;;{k^m1?H?wdOidhZuGyS9m6&qT~+22)@RFrp#x?CMK(B17E%w+rrOy zrWyBJO$LpQk2s8o-2Nou0;52ZIviaSMk9rC*N72`=iFV}A4r%H6FExEIN?Ga9P71< z^&8a!4U;K`Z9UMpYTb7_X#pP_YA=~jW@?zC2#s-Io0bjzsvDAPd=w%w&6(^tI4$;y zd(-Hxctj$pbDsWslo~gzO(X4jh)0xRJ8Z>-hZu%5i4-|a!K=m~L7LJHCORegS+0mJ zq351Irt0X8FmUC6x!vU~5gR9R5nN($loK;kyN<0Ak_cz@cC`}!5lv%F?u|J>l^sC0 z&yuLeC*6gN5oYJE#}uJEa>VW2L`ESRFFJWur+VUBTiPJ-gDheO03LKbZ;C!@{mhHC zba+ieBuI!oA;PX(lHoY?hRDAP>Dc4s-x&7y4xUbjvXIlbWGTiLF`XihW4Y)O&7|Nb z_gHSF6~QxBmHO#gV@;u|oT+6`QZMH;vPUaNwv5GMslJ9-;Bb~1afY&%Bz@6X+gRiV zjsQ<%F*78#>cpjN;nqXg(hWn{4rh8N_7)R+c` z1r4fV#D5_hgydT&0u(CkKw#kh1P4io*)HJTxCdP>D7)$|bPR`rDTyR?W}Z>|9Lv}= z2?dEm?4><%=L{}=d7=g4mL@&}_y+^u*Ja?V*jpzwkYaI&eLg78}%%5}z&_&bW`O~hFxm-wPOe-(0(4a3H2Q(VV^`7-nB#38!*%8>^brf(BTV|n3 zqa`!`I*gUkgFSM2d!z=T`}SP1!ZQ?&m44sd-|MOW^}p@@(aD{HIckG3$dpwMCF(EC zr!yVM)5=7~)m-hThO`>lTZCS zdWQB7Pt{1e-3-y>Sd}hWUJB!o2G%33q}#b3Y)WE)f4Z-}bLtptZ`bQqRqJ+l%?2~F zDu-YuD#@ZZW6{#+pcYNv2il5_j( zdU`Fi0sF1wZyHNIIKwFTjS+$0MZU&UsRIq*+aF0vw@Pc4@@8A46&2A#s@g?c4!Z@I zmh8sn-RqAr!<-ky@?YXW3AIA%W8>MbC6s1`&ZzRXFr_F18ok0&Ws-W5Spg&YHi?Xh zpp2kU`Q!_^Q#=%_BW}g%hlms3OFwZe`df+YQR*G`HOFc%IY0FxsDk+1jDL_Hak>bI z3B@r?@pQh6EP*86lBlXzg3(Y(=FWinTe0oOnrWFknsgL{Dv3Y}WicaE*;VaE&~A6d zx7Ot}v!$GCOKB!dY;C~_6?kCjOHXk#!U2KQFp(l?sp+L-$Q{F49akqbsa*z;R4j-x zvD*_1pllb2v&nZgEhfXnF-kTOyyyYxh=k^3>Nu8^O z@004`tF?`}Xma_&I5~3b--v{0ILHB*oeIIn+&FQ{cRheQj!3l@gA5}yPgFd{)f)QW) z0q8#~B9m5)*TJSgEoPwc0QP%NdwaRaL`@bS;z`=YljuEQ77r|ZH*mbLc(yy8y%Q76e#SH70As6U-Iri7N6>Kv z8;yJE{*ch%K0~qOLsBH@Rus@`X!q~?2m4(u?c{_f^~wl2D(ahzU`CV2%*r_xj$l4z znDTzsk4IxEaw<}5824Km$t_lpTdc=u&a#3WZ&)&d2;;yw9d2b`1~Anvrp#0}77mer zmHG{5gu^nyBnqXjW_&BsR+Zi08ii6;PuY#@Qg<^!;xX5{x(0}G^MoVH@Vj$(?@+1Z zhzk^k>R*;g6 z;+?mj_5X7+#Bk?>useh6jx8J4sN|ye03>DbvKLmfPq?5{XP{?Eh>cI6k-ajnWo^nM z9I*UL5m}YT#`+iU0;f50u&bqARZU6-I`M%Nu6k!C7MU{Wshmp{%2ca;wB@TQSn{O? zoiflkH?P_qmYMkqVvn<@j5wV+_i7Mp&`6DuVfdHiMP}|?7u|(ePZR|xRyM$?1n-O{ zOl@Do&@{b6S5jYKY5`QB8bic*=rZe0t-SF~GKiRRF=m+m8Kwqm+nVy64zVO?N>zD9 zZ=&oQNSew_Mw|~x{K-^*+5Pk>3nB03L%zGvgU_EmBU5xg!^@Yj=e9YL%J;$4ePs-Y z?kkC=*O(~_6K2c^o#rDdlJKljGMyqRhn(fYui?`tkb{1dzWDsP3!gu$2+Mp6x%^En zTakJby@_hU`c~OCej?*Z8Mxo&yEnkr){qa9UVa$P1m?@Z{u|ie+beX?Z|ZQ0KCaMT zq2=%3?|bm_pT#kJQ3yb|jfJ2Trini>hN!h5g>Kmv&=gL8{Vyf8N--_X)S~;X#i5_^ z?DaMA(-4Rp5?M$5ZM~RS7R12^DM@`0RG@IF<$V8a`o|7@SFwHf`IoVJ6C~M;nyku% zw)(2MEmkd@PPk^Yh7fT9&tOWTM9R{gCW&+rOIhJ0uI8le7oJ*m-sFy4Bm*#d0H8E_ z#ztBnpYGZgRA|)=;TS@Ut}zTl6kS!E*Vt2>Ifp;yl=L_v!pYR;rW=x}?JgBavR0w(PalwOu{ey3-4+fmz1vw)XR8k7NsX2_>+UjHnad7Lo+S}o~ zA8M;RmgFrIOL^6!Q8@Ed-vIiR3gVpUGc9;!UFctZ@7kevRX46R9EA*{U}mvat}2(C zv#VpE&UF}<++;^~&E$J~3>srLqHJozy|e%<^|r9G7%r7EGfeG08?Q{}f*<%W95^aPp;{zSPrpYaU*ceb@o z)p?Ij#<1d@%}Fi|fm0b5+`sv)$tJgk-xs4M3GGdbgo1jYL& z?=D`S_ryoBmh6K1Y0m$+e{k^hc_IJfljkoU@;`3m`Q{sw-BBW1jD!lqznKt|QcM|U zr074yCWm84XE=awzUg#6ee!^etYpNY_p8n#stq{3oby=MmFT40X=_JRq!rz8Q&mD1 zKv4kOZmC>y0~?;W1KZW#bK&_rve}Ne-lqe<9iE*WpZutOslHF?xCB_Y0vIpzu#gk% zI#YkC_H|d;6es9f+iIqi3kb;-4rfXXSIlvO(~wo|QiaVt!tOLWj6@Q&GMPMlgtXv^6fxP1u>MNcUv_#)8>9NiFi9Klk{x-*X0+OHLxkGonoS`sg$usR+kA zogd7X{tY7nJD+wzMTDoBKT=#zlk=Ef?CCmjZ4V@x4>31yj~rx{LV`<31$y)fYG_EW zaZfL@%1yV>TD4mmzS7!whR&?KM5Tpj(YJD+S-oqoh#kfKbyuLjSIBn?({yP>fgxT{ z7-OMFm=ZyjM8=C(n`hwc^{eCa52uIc=fAx>d-d|swv=F{>Llx$IFz34fHg*!{;vC!Vq!%{3&=|fPEU^o!p!+s7;cSo)Khh zO)bKH56+SZkdsyZ*itS~C4RW}!4{a@3gb=rt|qt^`j#dyJaR)fmmSd(9V#}pN`FPs z@4lv9W*F#teP#W_iWi zqWVq;io4{I0hXj-kJ3>WDf$AtBEys1Lgtn2`HOk-?&9?Tj)e+z66gv8f)T#4hho(+ zUHXhlVzE_0?71hvZ%8S)6P+43(R*b8k8@(#wcm1uZsx=MaSkWXU!VPYd{lv!t@~`j zguJZ4r%zInVCn#Dn?wQr2o`s@eYkWbes~7X|zO(`PUC z9_;^{cs_mVe*@QKI#7`dBNAd|qMk~y_!B&UZ~AJ}_P^=8{utZKK-^5O>WHA%G37kt zKxsF$CRbkeO}X7_=SM~JGLh7|NCj>|uq5)~`4jagnVu)35%~yRFKbLv5l@qm{OI(d zMI0tPclir!t25={%qgTq)}6Y+t1?-KF(Z)}LHED7_g|7Sxs%X;o4XThhIphC*`gE- z`mUKj6BH`vV?c4NM%wHAhBW|Gc_G0d$HHNVeUxylj<(%Q(b|kfa2V)_d*vyw573p< z_O_X|F?p#xnlbiwdY$7@9!M#-(*V+`c10R6ra-7{EpqT?v^C(%*p*kde;4o0Nz2q0 zXoVA2Pmn(zslg9kwnsZ>oGBc1D>*)$jWzdn;Q}_te|lg@5k%UU++Q z=b#u3Nk|02yeHAvw9@;Lh~genK!R?T`nZ3+Kg0snj;le#MVh7I(Gq!9g-AP$X~eKL5Yo9v za+f#BJzPf~ja@XQR?N9~P*N>Rp;Wn^{pyyt-bZyc?DwpWK;v>j`SKP7HJ6}ymf`<| z!zmo}_V>YD9i`9VvM6*Z0w5FVpmG)46vsA-0!Sp;oed)f?$a3Wc7}-n)A-+&?dqVn zzXt)uJbEl_XF$X*a0)kAg$$@vA=5a-CVjVxMDo-+hICB)Ugt36RGnG-NH(Jm1%Q)i z94a-=vd$3wduC%fB*qY~oDfiC5=8tZxMrvX9qe=9bdZlScv0BTZ z>qk}rmi)Cl73Q+;yrMN`wr(>Z=OagHqOC3ynga*tUZET~BqgeuXZ35Rp#^|eRjU$y zdB?s5j3fDLtLlJJZrH>JZ0ieD3Ir-&X~(67jV1GQtnMOf)vIW>x+S+HLs4)T#uIds zOksboJ)}&On#_^kn(RYP}#&keZj08*ALz8iE5g&pHI)gir^ z@zW>mO!BG1jgyR2PM(}u+=*kYu`X1Va4B5X_d2B~UVPS-73a=3-$wm}7H_L=>WjD3 z5=iscwn$hW*7@sSXP3;w(FFTfa@oZMGo7!>ub#r*Xy=d%N)v?iLsxnwgx~Gw#4C_mD1IopqXOeV$Y6+j19ENq zjWCLze~R%xY4*V0 z3{BS*0n{A-v-jlLi(>w-XL~Q6Kg9oR;>nz_2XMXLxgt?8FgKsQMR8|}1qzTrgAQaC znC7s=PCkJyisQ;kxeB18O)3kTAe^+Mx+4;ejWNiEOxWujO(7G~GAM;u$^v#mrhOWX zRI1DZbxn7IZBu2aWu6p2LV|;Q-oSy1dym74>OHsx5(QE$d0Mbtfzem9LCB5*!YZO$ zT^gvab1F+SPGgNDH-V$g6DT#-FpuhS6t3;7F-+E$)$1tn#ItuG8uK4?yH%zFRfl19 zK~5t?%S^=rR2EiWy(}Zq4D*uThH79gob|n8&0sDz)gH>aW-B1q7F`C$#bHz?T}j`T z)}h-M(^%(r8v1pgsM;G_@dh?4;FbYD< z`SJ;`>I1CH^Y}BNESZ+39?+Y}LnArnDG5Wu^<9UpKv60n8#|P3y>*)^b~!>Mq}P~v z=6c2D`m(KDm_~}d#E79V#G%J$k-y}8Y#ZmQi3W_Fzk$&TdNsh4*h>|uy;_P|IeQsym+Ahn|Ri*|9cvR{yJuW zm_*|b=~d((Lh7Ti$0t=b0Vn6t4Q1bERrH=$W`jpRw^x=o&H?Kcmx{0}p z*+1Pz^7L8Vh7i+;Gy`gyY|>};wJFChf~Nz1n;5y$j+?`|Mw+yPv4#V*o8m- z0b+vf#ZcIvP`99c{gDWJg*&n9Xm7iJSjI158R5VGyW-=UcZaWzPkywC61cjXDZQQ= z3B<7g5}E5fmcoWs0+vFUEM@8gs;rJG=dVvLj!#~{+4R;&jN^1Ec@N!Or8cKNiTm*B zhj+i6y!vo_`f^{-MqPWGzE=q|?H86rBuGD<{c>{r%6|L5eG=cwnzy!)84wx4cG|_u z{hjm!!13wJNBjSP^y}U~%qI(0>zKOjj3wWM&gZ%RkMxC8GHb5bkRExyC41=hdz6|s zt|EF9rJBKfi_3p;{r~PR9G~v0vF*a6Jv&3)YhjWF8*PE(TCc42Y{0jN=NGTf=I_Ho zxi4Wm7KXIIuFN;^)4TJF%I#@8cz1U3;pCUM-@iV)FZTSf=6|&{zc*f9EunuvQBc4l zSzh0e`5_?eW%to`K$r?(?LOM;?sPl0?GHL3?Mr>=(0!Cu>ena(UAf%1x4GJn{(p0; zr{im=uO|D*`A{v&-sOBtLi0k?E$``tW_#Zn@PGDGwEvhm5$DeI)h$5t?ElYSlEx_l` zsz1}tE&Q2Kk?YMs$DXg?C{ULioAVQWc47KV>&#z55(&(%vuJ4hyUh|7>oy#JOp)*> zZ*n()w_Mn{3zj0%dAUdBK%jqnwRz$1z;&^X{;%=)cRu}pUONAM@{s>~Gf&OlbL?bm zKOxSx{$GT8*3+h*o`IWLN6`O8({NseTZmlm5Gc7X&*|5>xn9@LUu8dc7;gR2SSo0^ zJ|nLuwCTkc_Tn@AXLHfQy)2$~T@4jHJRJ`D%(%d68|b2Je{nnZ!U!cFlXL4m4U z@a=7FF%5WAS53ZFZSRDvZm!aes-P|?l26?+6&74S70bTh;a5RdU|?xRr0kHQFr9CrXlT-^w$XCmp|03N?hxHwCqL;hI5m70Yk zJ5Nbe(5f{+k7y+DN6|A)x*EU~eQaN4Lb*6TJ<7%?Ul_Yz{^jZMtL#N=rx|4L7)HUn zD4d;9Dt;g#<})tv)cyWvLKqHSF&evf`&pK8_~wl)C1LRk$84QYN*i4SPK8Pwzadfb zF*htVq{Cx8S6YF2*;AUfPK&l9pJRk*;Ii)^XJjULZG^2LbNj_o0T)BYlx&0Ef zoN{5(HF-a2P^q5Ql}feNDt&NKyT+WpOT}tCH^6pWG90jC|B5G56KJgfX0IgL))x^! z>RlL!_r6CQpCU1VZgpUGcSmXNI?*)G^2FqNN~^T*Y>8g9q|m+#b2CzN`?W}NnD>8a z!j!|DtBbp04d(F0QD^>$s>`+*Qf7fETwYbacf_&dJa0I)%S(l&!@EL(412A(^&E7q=cmN$QpJIrloLeY;RAl2w@t`~DL+?1CD zN1JbQbRfIALw~F~qFj^^HIO3LFtD(mTyacl(%)qrc+ek`s6Rw};=J>G=f|xx8Yjc< z_WZF(%o01Wx064>O*m#K2%PLfW^faF6M=W8KeS8^nIBtWx;obHJ$O+u7F-2S!XNoS zJ^d^e^=_Xw(eGJl74VlI^DoHpFykZOU zJ6V3DS+>z72kJC1w~M-A9-jSp{&KtPd2&g*&`sCx-Py(G?oJt>=ftt<7XPN2H0Hcm zWr^<=F(co4R!hScC1NtSQgt@ztf`n$eVd9mr2+0jH=^bG#chyPH^NmHx^ux>Gnct7 zIEd5)Y*P$>6b$_kqA@RT=gK6ibrsAA)p2J=mFI${_GMMdS)DwzP_(`nJo+R@_5OeU zDD_xcVVk>*U1KJ3#8&J!j}=GPE;ZM@QRCOy>(e*KM~4^3?@m6vefR3M(~xqnaa8ll zS=a1_c46%0;pC`ic`Q}fq}3;DuS>5^?KDBMy3=BRKR9l5`q6#;jAv(fl>gFT4RWve zEjhITT;W+mwcL%ky*Zq_X{v{B-n{$m!|S)F7yny3;Q@4Kn0K4HC_wJr*+sdLO2^3x z87SuqRh3XsKdE?k+QKcfZd=;W>{CXExJtar6>!&9DhC9GuW*QF=Q<H?% z!ymPU(nXGfu&Oc1kyN25t?C!Xj8qf$zXBBJ`(c^})8q&_oXnXL-2zMCWTA)Ek+RU> zN?2KFYgM!?G`Bil78<-C#4P9ESTANW%xS`W+++tx%Vbu7{XMr4eR%p-3g{K&JWYaV zA0^43A~6}*xvn5Kg2#`S4%`d8h@?_YiqA6B2B5jmzJy}O}s+pXa z#$G{LYHm`B@Oi6`Ya}j)-?Rym(8I~)_JumTO#ZsZMNBh=iz5GHZ8U#*lSi7SO`&2g z?RRX(z; z@`%wX;E#lhT&0|pqd#W(C0{<;&ZUB@B(+-XRQFG3mt8oi;yjO`8}KIc&Gq0R=#~)6 zi!1xJ0q^#Oy^TjNw2I7SI))>w;(W<6Il>P>dZik(MHcJ zvDln-=)t`6VBSd~m&;+>Xy?gCv@U4tDWhbQ(WfSOww~zUo$05hixu?jniiK+v}+n$ zTGOs+Zv|Dmrp5cwwVnLeD?xF-0Vr+q4oyIHeXIjH*J1{$Yi)J(T%ReZuF0hDRk|O4E2<%O$F<0p-P6nIZwNap{m*KUWDLA0Vs_oYZg z^L&vT@!n2G;=iX$wvah=!PX*2fJZ0^1$4WWgK9?7X7Su{x7oQ=1bm!eK&XxV3FoVp7ZU%p6}eSvPs_|y^^ zm32YlMW=`-{j69mcGNgM)GWu^of}&BoDRM!5N~x6Q#E~^VSx=!Ow%9~gJ_|i>lfo| z7tHS1Dr&Z6MiRT_r!kFGn#!C9vkh`KgH)zLNmL6;Ql(e93rnYFzGHo9L7hKIz1__P z#-9o1qL5sX3(C@|H;6{$-<3HLooG~RG%;aTx?L4jS6CNbO8!_A+UN8?t;^$>izBZO zQ>|$#WBLbGAob>YTb3Gr{V`^^sKlfLb-pe#TDQ?wL(btu5(h}&bHR|nzMk!@CLGVH$7mRmX#9(+eu|h-dC>~>xAC9QJ;OTtQMdgtQvEd7%K)Mz4Ab^q zR+lel;1%0Z&4a=GxfBw67PGQ6=l@vGB_Q+ie?5Kvv~>UX{zLwcjXV_sUKLAU|0N)Q zk?bH1B&GEgAeH6k+yK(t=fc;2)K@Qzqm=luffiR>{gGWJ=3W3&Q9ghF$4)J<7QFRi ze(gF5^9}}lu7sR3mDsC1boc9^Qy;tErsY|F`|%&0yrvBAAICJX=3aHH3&GqJyT)v&iX9!-r?Nr=I=~NfeN1yaqNfhyL%qDBb_~{MnQJ z2l~H>XZ`g5dyO|=DI1_6Hl8n*UN-=y1K{w*_F&-2@L$-M(zypYRof&VYig=4O<4`i zt=|8p5u_Vgl?2S&|4(1+754v&gQpMoe{AF_`DnY?Q%4Dk79fk3t+sqEf`0Lv|7(<5 zYW9DKuJO9kKhMkmy#IW!X#aijV(;n0{@=v&Y3>!0cb6erVcD0z?Va?i4xyA|uRMa{ z=|NV7V^3AB27?{LaY%fW4&1+TU6J>Q-m#3nsbFjl9Cl zsh4+79_eDCJIqPEN;>PBfjlI&UUyRK6d&ehkghEGkkYzUmXe2*)~lzqHu$%6M(g_e zjR~zwBUmoQbVc1|@;mEhmPs&OuDN{wi>H^au2%#2(pjXgy?Pq)((Kll#kx6(b-wgN z4r}}43&>$@=)%{W!a5t{+r#sV*JrB@wW^J#5f{p@EBCh1PFBll?KIJ%n^gw;mrB2# zC6!jYgN~fiW;uU)cYg7?{93m4;pCUM-@iWlf--s+Q1EL@?Cs)c6VNrOYoFM=x%?gG z_0}EFOY2=M)tH_8i^%KkYVXC`&b}&FZ0pkK?e*H*UV262guSP_$FKcf+2-?#P7#S* zV5~DUbtN<$aQ|)|1?%3Aceb|*-EMAk zWt*fa0fbr3hvaNcsYv>{PhV6j{Z0<751I7;N|xfkj^*_slm0^{{fA8YBl3_*{XfeB zb3bN_hfL}ZnbaRLsXt^=f5@cX*yLA-pbweU*P*OGWKz%M&Q1xM#helkiPRqwsi)6F zBK3zv>iYRhCQ`3=g)G}ZAM&E-TKHP>qSt!CevN6-+qnEzNjK$s4@uD-zac2`Qn-J;Dq%EE0|Y{4Yl#yG+5o z%uK2?aDO5BL>u$JtS9~V-2C6q_V$bU-}VolJ>-Ac#M8q2YP%^pxY}%M&kp_<$qZ9Z z>e}W8uPnY;YVdp$jhVq~c}aC*@I~{2=i8c_7QCW(U6Snrm=;S4KELv(7CFJ!GTIe0 zg12w@t`maiN3w7}@O<@F>A;(-eA-kta7~%M5azvxr<(sutV>VhR+alZu z=b2KTF*WPqwjj4}Q~%Sa3N_eUoC_BFDg|A(|h8hoIXvf=ZmIPzUf`KNox zFRk}~Z-T=q8Ap`iJLqHH{@;66wErDEdH%5fH}dqxM0B>`-ESwa&(2{)LJT9y;OAtB zS%d}V9sTJq&gLGTnn&;dv)6~O-oEZlgFES?asThXc)q{?tg!$0pFMfF|7#;pXA91h zUebB5{{3TnB1Amso1AEb_IH4AKoOweTIx^%T;Z9gopycXPq305xS$O8;8@7QbR=*D z4398lH~>KbaX=s$C2+wKUnC5J2|wf{^091_gO4KXy(!C)Xv}wEg7^gUT?h#md>1%Y z|D{ae!2B2zF-0*~6rIAovkgnc=L}}XaDGQE-x>YtMwLKU?s&EPmn|> zW9>Ww$pe8zT%a(7>C7{-0&szbA=Utq&A4ryG8!blI`LbkIq)?GV#u%n3LSxK& zozC}q5PAxF-G>+wjstD)R^6jGcBXP=-l)m!C?4euS}whAUe93!ax8G9+b5A7E$@K? zy9wej#5e+hxzLDuL)n!c!^w}wC;u-TM`MOL-vu8t;h7oi!YE8|Bm&j%DWg*?COF{| zc7n#Tdj*g0e?C3S&p3;VtOgTHk=J&}Xn{{G*9GX3jG zZ_gWIfu6|kr@unX=rxfdxZF-jtgP`QQe#M>0JCr=mxx3ohSKe4uu5U4T|fn{PUnKA zXe-}pSSgsupB9a!2o9GSUSAHfeX+=1x{zHMcRszMTvH?k%?Wm4VsXw)&iYa2cAi{e z4DU}7LsKl5{EzLeIQ_8$9P90>=!t43`wC)$ghfZ&Dh?n>n1-O5kM5?^`3}DM=7PlE zd^3PIgbR>UHS^?VTU;tQ$gcD@{v{%HAW!V4et<(Pu=?!6*TMHPs7;-$cp111Pw6%0 zVAs`!tMyajfHy zpY?SD5>Z#A@xolt+$APi*0uom=o0d0@fvn|YQWm4d+K zIHW^l9Iwytm~g>nm-W#8Q5A<*iuY;PY+$C^s%yFIa|@Rd32O)4RI>;H=h}qo_KFZl z=dP`Oil%cVtVMtrLz@B0EnL3;;}W(5in;PQML}Q|Cr-l94310otVtjaAf^G|$w9z% zBdmvJ3IS6bd|6(L6Ei83_!&2q)N3=(@;IQYJ7CCLn(CUoHFP#k?R@kaGe!c;;WDkf zv;2&fo5yNfftav|Ib^{0r6F+qMG%2(RF_&X?-9Bf9C8qdDJO!?fk7SuN}T%wo?E!g z#RRv9f&&48#%=25At2DEwtMq)3zwfR4uAY`a`^W3=Y^2Kfj|~RGzSNcGddgVn4|lD zZsGEHbV9``!yHFq0T>h@(8vsJU*|FQ$!G?tM*Icv#1$!_9P>Lv(C0RC3+Kw#BKfOw zdPZS51IrH}e@c@buqf>S89NPZ-w=sG498dt?KwSh2}ZIa1nhWQLBWN9thn&(@8nq# z0tGML=5}u37mmRsuvac_AZScOI=)YOa0{1$CG#=3qNWRh?4HDimTuuBnGP|NdZ?`ni4i4Yf+InEWe&MBQE|mx zkcWU1Tv*j&27xO)vzBNHhjh|{!cYf?m&u>H1fpofG z$g?6HD8#|eql7BDIfa2@{UZ`$uFQWJ1!{*x*M!kX8Jn*WBa+14CBcv!4TXFO5GV-D zQwIxu6?H@H7!WA`zTAESj>U$Q#9KH+Q9!5R48}OZOj#(~)l|H|-u>B5FE|VW6~=`^ zWj>{mfk{LIoHsYawwMred;^37mVc?isWUGtFsuPew{VWJwSdmhG;Ckf;fDGKnR!sS zb*;m%PR_wxJfnpm3JJdtWzpg3Dk1rTx?zOH4P{qaEt)46-NI{C08ZRmYn#fF&x&*) z4*?X0^yc+Pfmu`q%UgIwqsIbFPG0asBFUX{&r$|6pGJ{BOxd8Uv?4FaLxAYhy_55| zS*-7++t;B4QEsA&<5MxgOlyWQ@;56g-ooXl#moX}%?1FR(x3tcx9|gH<=LueV;IgL zrh#$SQ)e}45&QB^o)rO5lRv4>U~Z^@t)-+w{87O6kcx@9%2Ml1+7CkMPTD$sXU~dz zpe7?x?U8{>S9JVJ?Rj&8r;|e69}uKow70j6YE%h-z=Bih5E0?(3dTGEgrKXe%MX9Bw995kzETpiC3MCo~BII<_M)J%KDNi8-H_%T%mGDoDcxvrw7ga8{Nf6mIgeU)+G z?EAx`EQ8ACF<@mJP!M1=#66aTc+P3dEu6_h_p{Wsk2yYGrly2IA&u-j2%HP$nF5>y z*z*yCICKB9nNgRW$+FLiIFQd`JLmk=S_jg(ei2?2E1sJcA<%e!S_6T5K0qy=n|gM5 z^Ur_E5SVj*S_6T5K0saZ+_Wy2a}5rBh3BSa2sE6ZewF8@rG(YZyWq7f28fN5sq&q) z;_#3}h|T^1G3A^{?a}ynzn@@+E4{O6NUky4+k4*6=h)sFg=oBx%zO)H|yzCE5@-&a_6Q~2B0ZVZkOX9vjOoRut zi3to-rx@LFz-E1bdxMH1MH5w$C48&mXhloi9h9O9KKpqqx=Qv62{IZ%{5k;Ly>Iug zDZEjc0Cwbkc^wJ_IGRYJD~Ufh_ zO-|;fxqYgWvDSw0vt_mmRd47t@wds#7Z=c}aw1=urN}wMKu{^mQ599Q;EDntN(m<9 z0>1*-(9&@)f)+Bnboqk?F+2wnB1mv)2pM7ODa$_3YD}Q^5H3-IGaU-ed-|5UMKA2oC;=hO})pXupyU>-1%!dyOQO?8NCNlED_FIZm|v z4!yeFX-c1B4YU-L#%_I?(0061y&MJD2MRJz!HKqddkL~(928>;Ke$~@3mv@^qAc0RO849G&XPP~|EcOzFLa)~s9HP;6ZlR_Nf`V8PGuZF( zpxa4`OI@fA?yeNMPFd`E_Lex2l71B$lm`bm6N^idjF;nCG=hg$vYbNm{5?wO+J!0F zWfueSLx26vYFs`9cz2D}_;dap?u>g1Ao2%)J%4WYAGPr(y50p@mQWY`o#<8~nZD`k z?Z;AEW^GHo*HEOW4h(R{94lE+KPyb62Y|T<6!`QdNo((U5GCxrilJ1p;AVH7S;6L< z_rnt?Qs)XDEo8MOQ1GQ5FG&EuB&C!DM058b>|%w4b8UOgU;KoMeOM0!%p2^yR%=dK0>Qp_J1h-zV? z{eQWPnp>U;XA@z$MGy+*Mt8*~Hq>Y}sx$xZUmAzB^2HSIO6+;D+t%z{8}_l*_#hhZJ^ zeEAvi3q;Q<87GRTTReY78JeF@CQn(@H0gP)UIvFCSdPR%WNKs<^g1^>nG z7fcP_r;QwC$q14sj{Ovgw}vTYnck#^^(dFznwHGpER*R|L;}K+6p)=|jU4_FMbI%k zs7_C>K>~7;fE=JDG?1DB4al&tyMv3Dv$^fKIB=aR zud2?jY7TGxZw{CGj?-_KJE98V)r42Wj~L)eIURe7&UMYD?jD%HFc*|IhwQwL+N&e8 z&f6{&YO;dzZ~;-Ss3(L~6Fn$$$f19kJE0$t7X%6!v4y-KY#C`nLISwxG+nS*g#Wv# zbRUflegKTs9^0wnR#iQSLd*Cyh0NXXi8k{ zYC4(?E>tyX;fP??$3(^3EcsW7?m#wL_6V1|;C;eeHgnN?(0~@3|?kU8XOCL<`gGuC%r~BDA1LTl=7Cs%)BF|Dr9-HxaF?S z*K_*7MbfXPZ(d(_A16YbNcb<&D>mHPM&@1gbsCn zBF_WR>$tc(wDcNSzm|o-*GUg7r-zd)2eabpHFD6KrHAib%?>Dq^Z77mRx)DBfPh-Y zJbI>WcuXM{$q!z+nr>Bg1#Fpe>zJ|SF@YZc%f^aOF7xkujO1iBiMKvEYaZhWwCyCheDRij);usE$dR&RxorEjSFYiKlTeP^5F znzd0*yQe%Z$67YovTU%7oiq{;&YUww8i-IHkjb&$e=32|*M)Rk z>OD7G0s+#p^Zb5Gi%MjWmHiGMQ755EA3l3lh4^d87 zV^2RaD?^6$bd>e724f%PNDJ#?^5beTRFp#bl=a2BDr2Pu^{u$GMD@jNmBkcbb`P3q zY^!ATCUPocynTjS2o_S>y7;Pz8DN-ViS?n(&aP=fY23d5I_T{w2NJ@DC{C|$FxPDq zfCHDS95bXP?NS%{9XdgJppxmB4+T*}Sg$DZb05m=ufj5_ic)b1-c!BU53>z|(h^58 z;TizLIa@X*v_@8`7ZK?LGiacDSIEi_TGlHj4L;K~>faK`Uyo}JML!ceu@iSiM<{5G$5QME6*n?%vbTf*D=)X%9m9TWGu|Ld&@e4at{GcnKUre ztEcqbr6W)ngXz)GtN%Ib$jWywjzNDq!G@$*&wy5Ak3Q+1(@uLXs2_4MS?R}o*6!#F z#a33=Z#i~5j9jN0YgM9SLkgi%Y~cJ>8PZGkPZQPDklD4|!ax>uEER-G>Hn~t-z^O7 z#o^1r?eyr`u&W5>Gl6C;tlSPW*G&C)*20Mmx;GyV=w#JgLuE(D3C}n$+Kg-r! zacHub4kS}^j|XBCTV98exdJ-SOpOB`=y{k(eNyIJNB}hQA2MW}1_-0IH=B3 zI}KN%Vg75MseaG%3(|n*nBz11?$T&hW1Cf~*@PkiSLivPBEh$kU zvQ1UG6-e0-S%AC+5||XHTFjPh{1g?(Yy%OV*q4YWxLkwEHnN>=C_QhSF`&A~lTv$N z_m1rM#{m)oHu?FTT;w-0j@-Z5)vn7v_+-GZh*HN0ckY~ti3!5}q-kP@Uwtmej5tPv zj33*8vpJgfk7-)7GC3=zkN|YWGNrBrYc^%hjC}!N4_rwK9B4YQJ5=|VT1ZqI@i?|U zR}*?=&KCP2qpWb|odLW4@H;($h>#7_gVPH`>VU^wre0P^4fO}RMH02$vdj?d$c0Xw zRhZhp-;f6f_XY06N)LWY-v^*He86{noCWvodoX*y=S5;j_g-agu5-}7R^9wAPtVC! zEc6}5H0lxwQ)wCKvUsF0lpBiE&t8Ih>qR_DN*xRQp#oA*+n^1g@~mIdYKB!_flo67)O*Gi=g>rK{np+y3a zfaDGf9lea~uzLy*4%)>>g`m`@f&mBd(

    c6w8`?kKK>~NP?u%J}6<#Iglv8X-ZWnn-J`R zje&YHAb^-L_5F@D#O_=>)&Lgl3Z3w-2!UdPOF^*?IgJe^R)*rV2N*IEf*fkIo)MFIFc2%0N^=)RD1jj zD@x5-KapwdC_3k(H*Hy*_*ZfH09uB){j1(-G=gm5PlV_|fh1*k@EIYyD^Q4#+6x&? zftcGyDAA%{x;E9%gqME}LL4YT2Mx0n8wWkc?gXi0g#kM+M3waeapSt4D-Y$s_%<0d zsmS6={go1`bPL(`wiJnsta23o+tvcF8FD3lWEI*UzrAZ9K1Z4SOU;$)0_6{x8nf%* zFvX3M|CNh+g<}5@Y)g-{CM=|WY_OoX6oTGYKc=z=Kv}X#Cbh}cRVHmg$KqtaV^~wR z9zNQBukGii?4*v3j{GQF?QT)`RbwmZH#5VzE7wRu$LbI!<_pYiq>Ts)flF{i00|O_ z%dZUu{4mdGJ0ctaENLN@>zq1pq91sq5n*s*jlu-GsW;8+3DbJs;&){?Hs%^T7-8H^w<5i!dU+020#9&~d; z#K0G1UC~5P%D|)j#fYY_QFKVCC7Y7+fh8InHG>#K5hny_q>C|ek4HWU$A^};@fP&M zdvf|OJqNJIsOfy9-<3ClT3V!MVUTshP5;|m>|l0O#Fc5A#)7ip3O(=dd`;qvBLa!#IlB>udN=gVFg(68W8(NC}OA zN!#3~_49Oa^1A$hjcItcGQ)!cJO?t$s!hJ5WdtgbG`H6=mhrll@4f+SEKc1k1%u_h zP{dvkAQ#9V8cipa_tWx1MahM8StJ|)h{aNr1c5GxeNjiH0f{)7ob?d7s>4c+&h99v z88;v(wG3v0;w2-v(Y0=bUsvFUB`HbzwPt6%noV-wPWxQ!SH{`ql%XcaI)AO~zz9I+ zm{27^P)@|mNjDc0yVq-@mqMaTnTs1n0RLXJ&Y_CPnKncj#Y!j{8PjBxbqDE z@4-J3ANt^ z-&3+xNp#})?28lTtiH}s-%4(ru5kS{t7mgwuk`rQY{8)ne543PE%DL%Bx?V4+gI#)Vr&&VN)v2h*MB@N=MDF zQXa!NsAtJo#$~giAeENlUuTq!+~ol=K~Y-EfG()n9bSp!+or#P89!(WiMq85LtMrc z`dX%bbaCEYxOQf$3p*};#H&b)|9k;r85;Rq)cP}aeoIB6}#RI+jBeB$9)w$ha;(kyaXd{l3E5Q*yB63g~& zjYny*t!4C8y6OMueidiSP+eAOYkOh~iqGTOAi zg(NEr(P)CvCqC%7Lp@+BR|;SUzMx(g8AuTohA0eBy#YQ657wK5?Vc&xS*GdRiy_-b zjF7CaU130T>8NS3(xtrWR9J{rx`TOl?|6fK(|&u_S~etBDvVM>92n>1%{Z9pl5;t2 z4~yMGi8Ej6BN5|s2UhXMce?Igs^G*>C^Py{&hks_SDEisHhQ7OV{AFprRdzwL@PvYaSm8D*s_pSG=-d?9m5 z6M4ghYwWorr_v&JMN?mr34Kp{DKfI(E15H)?I%#w9$C0Bm_*mg={4-l5$2B#or5gu zVr`(IRS{H@hOBG?Tt)0Ky{~tS%4#@*4Z4x$F{4nRlqafpft!Y*Og1W*TmKTxU{!VJ z3O^1tGh7a`jXvKXZ0~nU!hgsOc*fPie|Ylp<`yUnc#&8o#wn=Q4grqhB2fY}6kL>q z8#o=$-Wz=?&LsDjQ%ufRIKnSPWcN7?>i^2M*~vs$?7cesRm5`UssGn(#M!mI(Z_ag zT+Y!sia@G3mHbd`vn4nz0!R+hs>GCMA$!KTzd9l5f%8ZwH_owuU(`mno zI0hu=A~p$okixPr&I8v@ul+DoB~v&B%|V3|T?>qJ1QEpzi5iyPx^GY1Pk_wB1O&+> zq^}R{Gi<=C9FC7#RTa13Mpy_Y+{~$5K_(bRvokK~ryWU?l6pU}2^cJJfU;gOTt&a7 z`kH9Bd*}7%x#9VCp2i!Byn;%US)BK7K*YF3@d8Cs@dP(6B}ENCiyUXbduSSHkX$%Q zL!rWDK3>PTwe49)ni&6b*U76)$cE`M4il>Y!gmy98oonqUHn=u3qZC+zqpg(>NCi;<27#lOo8Lo<}{9PV5)%d>k*wq1~N=Q;d5G>k^8iTVcsN zU`(^mUZ190KWDuW4G_Nz)+p@v!wRvlxA%B^G+^?azoEW^0*HDMWK3lmAUMuUDU0Ff z%ldQjJX61o_=X`7gd)-_vH*ioy8L{wOjym9)Rhg^ophDSAwxOV)xM#E3j;8(Sum6)oFHLPt}!Dw}p>QTuTZ_~N9;@`?A-0s0eVtbZAOo8oCP*l&3jo>dv#bP! z;Kf-m<)b%)MFLSNWgZ=QUiW;+xfyF5Y!sO{S*uBMl0`hShlcJb^$`BN2+0 z0O7zx9beyw5w%oUiy7@avujPj$Ca+Gb~ZMcdHXYSVbPV805mwv)Al_DT|y{vjQyap zyk}&tR7=18ZS(0P0g{XPYA5wcQ|{$N zSF#Zv;~`HJs5x0bb$@{4=ZZn4xxBFtTul(`%fyg*`S}jD(e)@CW zbGSx+@B|(XbUHyTBv+(@Lrg|*0)#m(3WcC-)M;k%Q~ptq1(M6Hwd z(~{2bM^~TQ{asIaPdKViZxKGJ($$X2Imd(xERRTjD4dEI+v8UM3KPzD+b~LH#ploy z!F0P?9^+G2E`9`8gGwJ&G^~R*Z~0{5DjnGplFXq|8*ML$*!K7S_^WLcXp%V+S+}wf zPRVDGVxk-y*}TiL_i-7A6{(`wq>!d3GH{W~E?Au4j+U%UuxxHg5BYr$!{#E+m2|=*vYs`)sw++0~F6S+Rr?U zziy=~zJs2>8=h;XY_Z4IUnlh$LI1u>af#j(X8nkfR(2_}T^x8^H7=X;7b{f|G9=am z0ph_>gkQ}2^RsR_g9=6^Q;IfXS0q#F9P{yR@yhcvzuR4>$t!aH8m8&bKz-CGJ}&KG zR?uZ&C^FcUCApkFeOl?Zg)%sAz93r7i9`ZVKwKjb=>a7~x*Wou*p{;@DhY&eWSY$5 z-K9v@q(ahrwYKJJd$DBYFx}cD7|w0+Br~OYgGry5+fblVB&toFK7ibH>Iibr+Q^T?V6>U7wG#w;~AaI?C)sp?NToq3Nrg70n^CrP6~Pi-&*1jC+_3!FL9&5&xf z_sOB{I16OPT)#;@rGJK4q#b7gj}ES^7A$(x?Lu;C)j#D$bmtL7Zg8K4zEMbD6jdAR z2%MOYr+aFX8JiuRH28x7s!H_uTKBFCs~YTxL>;rr6$}5CV}I&phYnLLm%Vw}z$}5cvb6Lb0r=3tsyEJ5tZ>E7+a}41Klss8{#? z+rnit{hK^IoSfww|G?>fhf^H(h9kD@U%@&4O`gX>2!ZQ%fU{u}@3+-*(XlXH)g8#s*O>dEPSAwUbf&Lj?% zcJ?`S3;o$~W`MQCh%8D4MUu>9X6I^oqbfjusIYewI11?Mb(fuIqEFqky|v!B>wX)- z7O*0)<9^z4#kFHgPQ8rzuL~yhAIzq+{*N#4JgDceG%&xi!*uW07L8ry7patO1;G4k z@Chq~q%FFt!J*Z+@4Dlpp5s%g@w0p^ke)%oDf%OU6dvXTk`&sylBb}~uv}{#FJkWK z8VbcJ`ddQx4%~KkhrcIF8i0lOcSpPoIUGCMCfR5jAzV+ zH?0@nrt6_b1>icGqz0Qf8`XDtm1)JJS^(igPR8O(`@KCb$YF)l8ad;ieLfC$#T z_KW>+PcA-Fw{;*t+7=jcQ!0YdU{qx5oo=hg?L25{@(y|zTPMuEq7~TxW+KLF4>=Gp z#&BEwTCmi@M87Mm%$^yxVZn)AG{{4zKLV$u@+~`Yj z2H2*GcG3)4GigD26`lAkcSWM!%CKDM-x7iGz_LOWC-6aQ-l7MfRU;xT z#0yfxqC*P5A~CpHHHq7QyS+bm+|Zr9$f`q1t)FgGla$Jxme!e^-Wp_GeC&I4{7HH; zFd?UXP5SDY24XsiXL?omO}J=$=5fNveZ~-0h8MSH7~Zga?>};>A|Aj7H-gK|;hq7RY+wo`dRePjD817mgDpoI&jq8&3g&)_fH3`%dSY4uoh+;zCdQaIJx zDIRIl#9KQzc8lN{L{3a|>;{#tBxlZ)A(&;4Wc()^bd`I0_2f=;fwSQxoV?tQ$&(#Z zR&XnWEp0pRULBQrVI?z0V%W`~M7S#c(l5-Xpf^Udr)nh|e)9=hU!-Nj50N_U% zGDX}e(qjuSVo`|MmBZHRm~Mnu=NUuYF3E#nP@ zH^`8QEOf<)F>G3x&%VMID%KzpVDqXWwh>#;RN#HC*s?KcFd?u$c?E~tE^@2&z&?1_ zXzr>pZ8|jr=tyt9f2hC<#gpDAFTkcK_t6);j*fvyshAfcqG?ZZ-J+4xdbh3=oH4&u zl9OJ5uAU%QzNO&tOU2b(@=8&67PjaL2;XJIR%Pv3CcU&ErPYtaG!bMykQHZ&+*1wW zSvRW=xDC6z#NRdB13;6~P)uw!fZSQA~uaIw+f4Oj)KJ>r5Z z?t`70apC|{zV`l(DRs>s1REAUt}KaSGeg-)D$99+GEUbdNGlH)y&%TgrQxKi%9_U{ zMqX7mhCr<}CgjC8%`<_ENzN4Tun_H)2tZ#=2r_3MseN?ml9I&*GE_SY+hr{$RP}p#FP?B)_&$WtC2uF|EWE{Kgaj`n*P>x8vpm^EqD~()V9a11f{_Xm#+ge@G4c| zxWS*X!I8jaA;fr1IRXfe08N-)19{eW;HY@bj2;_`0$Nf(F|%wfmabFlJ`*R>c~7`G zozDrrUnaV0D|2c+w`fW@C+z9d_gq;=M6H~XGmLijU%)NZu-zmbdt>5vrmqrO$El5B z8ITLTv9%MMOeK|sfiMqDoggtr(qjk#jNs`oFhHj5@ErogRn9xM<5nR{r#nwnDDomx z#xVu%3nIQwwZ9n6@(R(uNc2b%R{P7&&KyDB;&znm=Acx=nQ#egA3)=|X~4S5n^r$e zr3}9GNCDd%!CCnxa%{)x2y**^W^6B0+d{wQZRP4gRy6i2 zdXt*zyvt*)^Ex0$CEs?Gd7T;MHpQtIGDHU+Gk{pXcIL3~(x~-FF-@!>CiQ1JH3sfq%-D2gg zm=4hwNEc6yw}|8X_#3ThiGpZkEv>E?IZ*~Z)}P#ncI9jxs8qob$_h1SAxJ@1m7I+Ay%kB%ApozR9raTv1!FfV_Z>YmP4(~T2O+dQPN%O}^U5!%LTawt|gTJuW zTNPQPye&0J!XZg*-&ul?;$?otxYjsI8%{OaY&D~50i_sh^fCm;DiqChcU*&&@#mrL z)wXApQH`cuVutWB^=CE>ag4`UR-T=^D@KY%=`YgSlX{-+ZLwUf($oZ0jm{C}MRCi_ z3R_YyQAq+AWiXsrt*fEx1Z}TW#fgJi6|ud!=uDWOg~g$OWesYMsFV72GqB_OD~#*N z*rRtgiuJ3OO?-X^n4#mvmu0tEey_tY*X#ry8OTz^drgs9G*r#*duX;_`K0PZ9&+`dL35fZFHLoy z?blEpSU1dSK;1`9^t80BKw#+S@5|pJ#DH@|ytl_nE1%R17T%9_g7~+EoLEBlT zpceu&=kIJbeU#8b6c$&DU}igqmOb{BCjZ@MH!V1nw1#1@4iorojL`^fz&|fgUF%*9 zo@ErJ)}j=7gq*9R?NW%GC`Ouz-ro35b5-(%Wb=ldOWnOwsm6zPae!L)OoFwyN`Un? z>_!MP4_UP6n@;CvInUDCMD!G^n6+NdwrwZQas-A+PHOOA@%h9-YJUp@5&6ul?Z z{6Gh~PgCQ!@f%hw+~3X}+hdfCLlth?x(j+RTLY(i66MPWm3C1XB35DQ|O4L&G6UR}B@;`@aCIE^X3XNioKXZ@WH z4|9FZ1)@)67C3ij0&eWZvZAbD`xG*S>^=YJ%U4l5+^JR%&38C=w*LtXf&0Bb4(kiQ z9t?cs_kkka{x`uF{QrTtBgZDn@9>GtQ~rpr;Qv5R)GQi)qUZy@$hYAC8*W?Pe)e{> zIe&!Dk*4Qogn~OZ->ZmM;oX0azd0;#{IhZ3sb&Y1|R@R zWbsEq2;{cy1pXjR;W0I=-L0Ue2!Id>!T$&7SH%^Z+93i$AO`Sw`@@I+MFNBn1yxcZ z_!kfZ7>ED}X@ejHA^;9G)A#>%`0sfB8Q6c^-);W0XmcR|BbGhBD5RpFN?#gZ#UlWU z&mIYY5QqTQe!MdVAOQahqv-f4(l7-2&!(^Zvr-dHW7B8+e|Ntz;NrySu|Due%E|`6 zZz_K08XwRueP&8I%G-wu4uXyJN#|+}`ELqB=SdRTmw-YpS+i zin8w4boV~~RKHwO<))tTpm!z#yH9-&3>*x9@bmKV6uy3Ath6?H1ngjap%eCX55>EqDWe216)bJc$1di{f!`3e1`9!xuPw!it_Sp@uj z)dQm#x+RJF)Ng|t6XxjZvTFI;FOvEa`c5jq7|J;LBC!vTb#VX+`yr5asHMp_UQ#(% zaF4|uO@hAs0Wj>XGSA*s=|IekYO0gD-(~WOfwqvqSVZv2R4E>m=_Q_Hv}ZUcj(lA3 zX&!LslNJTH++y!qZCe215Nu!Ux>iG^dCd@#A=lItv|;;i+iMq>Fb+sag*p&}3D=G~ zI$a?y-bS22Gd3Bz_-R5(^a@pYW$ug(me$)-KMfBYUTsv(q1%*R4rSl3j0zK@oY>(C z)3K^1fhgpr4R!36%c85uwV3p+h_x#}Kpwu@CemDTf^DToZ`%oTu=QLM^O8o(bgiKG z`h__8o2T?M{&&xeo9xkpFN-@46$yajmSo(3@9Z9O)4b0%B{ir{^^|?u6C%K!Kb-SCR;wzO76mr4a%jh;QOuvrvMKUXs#fH{8C^M z)}Wj!-KPSHamt&f0!o0q#z+NZR5bV{+jUgkh>pDEl+y{Paews%jm{F>eSE3>$3zTq z%?WcY!N9l?8Z1gp!AZ9o@THqUSx>>h5f<#mZBkt z(U6TWF*dC!7rm)<&fLkCrOuW}q*2_ex%il>JzOd+XY6}DvSXgYuvg=zNQ9Oe)45xL zNQbmTbW58U^$IOQ`4K;}N-y*g4?hhrsbe*hE*)c|-{?E`M7oh? zvoq4xn zQL!R>ebPP~AW`GN)y4M4$UGME1}Y7c7OidK8aKv-5XP(WxW`~farlL*V4wxG~f^+JYSQ+tfZQr zn(&*wZQC;m%V5~KV32dw()?(opV!#xw|ZvZN!#mX($`1l=>j=AH-2ctU&M>1jSUTI zl{wn?zuasO3tbddjNI98Z4UwW_7$&A#8eJ-k(KjUZGNL!(CP0Md^t`fq22KN(`#k& zGN^CzPs3NXgLd|W%`7s6s7w`13axB%qL?v9BSxl8J%A(AJ53W+N5h|dlX~oI9YKD| zRYUekqO^yf>tBCXEjVmU7&OxMlE1$&Us;vDa(%f?c0Nlq!^l|ym@Eqg#uH^v#!{a8 zuS^sS#*aah9bO*31E*_~RiqVzUu$uM>xTA6vS=Fm@cRgu2qE5AGlaS+=690uUbS_X zdH$%oe>pLIJ_h*iF!_q#Sw3u|s!zbEq*jdk`PIErwz}e&7~7O)Ph{rQTdGZ_kP1G` zhS< z*rbrs_tOhpyhH(3Rd^bT9IajqO`Tv59UmGSy-(OiXGc#B-JEYEP)J?2FeDkDWos`s zF_$_EkZ_g-iJO(YREot6NW>ZDmc~JbjC)^MGOT?TOUhr~AW=I}C-_k^QTTBVL%`(K7rCjgeQK`o3caT}BAtp&el9F~2%WT1X*ZR3 zHiXj-nyX@&1-;u!>091&dW04B__{LV`RgR4l2UIo_@Yw%2qc8TnpY)U*TW*3ANa%2{n?-@BJQz>t5LN%^ignPoA3z~uv>%GcVCoQm3U wj$EZ?ZvA|6wXj3)WI@$MUw=v#f=&y6(RKWF{nY>GH^|=`m2Y7HH^2b@3&?wbhX4Qo literal 0 HcmV?d00001 diff --git a/infra/charts/feast/files/img/dataflow-jobs.png b/infra/charts/feast/files/img/dataflow-jobs.png new file mode 100644 index 0000000000000000000000000000000000000000..2acf48f19e9611d9288654216e0194ca02f0df63 GIT binary patch literal 29728 zcmbq*1yogS*DWT9A|MFT64D{v;h|f)LFw*JgAkBX>X6di-6{e~NOvhncT3;J`~BY? z-xzoNH^yc79^f4IIeS0PT64`g*V=FJa|KBZbV76#6ch|;DKTXflOQeP?WT)kb^~>r<_6p2bqtE@omdX9hzjpc&5;hPV>3*QtXdVQ>A0!jQp{=HAw7;6(y5gd|!X&tk{x|~#MMTv9)%Ab< z?LUvN-d|nfzaB-jRVV4mvFz!0XV{^%C@X5xwS07*RVQ}hsmasQ%rfj ztEnCq{5U_K+r11F^VqLm-zw}7`RrKHPwtP~H581L8G>tnb;O_X*K2zTgQJu^uYMsd zs_kV|+ohNU>!REdep_a2i;Kj{uS}xlf~76;7ulKkv^A13oA2neE`{_=a6W1a(F4b7F8Zfh?A|*5{nb!EK<*t)(Vi=9=%3Pf6z5LvVav3qB?HRR}+Yp zwaQ5;YdNJ-UnnCkRb|2uO?aYRPmGWnjw|+|LQfR0B}SmpX_#M}#bxQG96zx{yc5ZB zR8tFVeDSoipwhOG{foxz_y>l$ir$&HlmyVs=Tj;mBKo{);ujNdTgA@ zN=`*xb!Ki&AZS@Xxh%JbFMovpuxn`YQlOg~;`|P|G&1D7JFiXrj@8dpvKKtvIy|0E zDXQ5PXp;mVwH3%J-R3aZf8V!aSg)b3X`b-PJ`K%24$Wjc2+#fCS9t9n%FNfB22FSL z_K|+4)wugMbYs3nnZNzUuTLmlD3K&NTZkY%q|&DY9T=+S%&oOhX-EbCa2vNypu-zw zvbk*ke4~^1u-o~@H7nm&X$@ftt>da2rXTJ!d+uGoSM#2GM=#;jHLY4kie#1PvH!jf zgM`R6^Y01Tjx^5wEKvocdW_*R%q+21C!V`4yEyX8>0~jw7d)uRLw%pwWPGaLH3mdR zN%%*WexF-SW>iLI|3<}yuxorBsEqVW-Is9==Y(Vsm@-+n?FfDI7QL-KUo((SD%~7;&Uy z)sitoK6&>Q>LK1Xxu(9!EtCnL23Cr;KDSG2z~F8Yae*~3eAC!j@mk1pvzl+^R=!c0 zi7Pd+j*)GcsQ$7iAI}Q4DJ?Vg_bMW(i(QMOVRo$2R*F!AVI$?P-+s9@FW!_iHx%6` zr(zX_6aAxUZSp*EktMxf-s^%MXG4x>@JtL=Jc~`<_>Ff*Vl=s07tgD-nKv(O0-Vow zi-tl-l%6x>s)l?i#cSq7v&M&}Wa5e-FhPhGi<{XC_Dt8S&$+z&O;KvD9 z&7Q$l^JJ)cv&oX|$f!QDy~jWYK5f%VU%ZGsKVe4ZI5p$0 z<{uU#9!q}*qDzt!0WNa?JG82*j=bdGI$yory=ELOd1e!i-7xL^eT7D35c(bRL90&M2^@Z&6dM3p)ow#?bkSfJ)DP6XBmawRyIV=uHfZg_pvHJNe&EkU1WMW6BsY^i<0 zMzQR#-(fH+slsMM=V*MtyA_Ts>N@PZG2rYM)e`hrFJ1Qmc}(XJNs{~EnK*mJu`!>fBY?zQl0O7WYZk>k&sOlHWyD=QZiH3_#v`>4b!x(;(6 z+_-j{vqKGU1IHDR;ZE}Jd{1<~ij$Y3( zZ^51QmusH2hz@0ID98!3CVptwIaqmZ$RHJqjZZ+BP2U$Htd0$t(xA9KoN?_5NBLiM zOL5~yZ)f*r)ojvLrb_nTeF{i83_fLlT$=xGW;x-+c#y|kSP9qJh{Oc>Cx%H2Elv|y>#!;FD))pJ@nTCw7r#O$exB7!-sLO_uJ3s zxFESvhNj0+*%058uHA0#aiC_Jnvh8FDU|Nc5g3CY4NAv+NP3#6&#*0%yS zYZWiA`wFa8?28W&knQT(ctsL-O)>njCTJfJT$W=~G>ODS=b3MPt6*R36D>SMa_?Ek z6s4cvrx3F0ayObNiH^?GbS%$pzlOz{w=i6DS+y)XCBt-SW$|59;=}5sO6#1n;w9hE zVpSNrOUrq1@*NuEOzgoC#K)xML88$DzD?6G9sMSY)uu(E`(y)L8zb<~O)iQqTlR@;n;F5HLnl2)fJuU|+s&X? zqm!g+gisPRH7mpHH&|bF`$d;tN=!dM7%DTYH`ZN{dQ&R{R+!Jr)QiQ$r#)Jg5A?i? zs45Q4E9-yL{M_^PhwumIdjWqrQ#$M-=rQ((Qx|DM-p7rOMH`xZ&6}Z$jLx&K=+FCL zeJ1!mrmcvK}TEUmN0sM72&|GFFxz%hTi44$MElFlwpX~vNI@?o3f0Shfnr`H0y#W7y9E7J2172N55m2&ZV3 zBbkeV=G{iab4{afxaRf*3kJkjahd8N^?+@_9f9xThk4Pjsbv`W6=+yZKf#E))K(uJ(w&1X<7*=IeGx&7)-CF6LpqO^3s~kVBHCle37i&D9}%LB z?N{dj9$#-*9!J_^%CS!S77dAf9%v>MpX{G3ztenT%f&}d83whIaC;o5ei`s{*xdq? zuAS`^$?jq7ba+nS9wA=fZfkAqnwjj zf#o^m4U!?t;%U0WM)zfRM0+P1`9;q$q>jeMPa^4VRVfoo!UHsNsProje@<96KVsS2 z!y=A1Owj+7U-q#$yx|TC23K}v^=MOIm-i=f<7mGfp$a)G8RN#O*|%G^8G~P>x%JXz zSe~teKpzG$Rw-X)3cBCfO88vybJ6>TMDpzCEke`bGpyoiBM!s@`2&i>yX@WQ39L{uk- zvl_?J6BGJ3Zt8kY=Eh~ocdqI!PgLlgU2Tbqk>*5{zNbd$*U=mK3)|TTl+e}s_`6M| zRJ*GRfq1dAGp-)hyaPg1S>oSb*wTVOB8hL#4|5q2K8#mpt-CH=f+P#Yi-- z5xz)AuVVIan2WS+d3at4w{vvp^~4stY}0{{sev!$&~Kp%U4;dGTD64Ryrcm`LZn%- zCc@u!mL~}3O|7<5EhLXwB0GniUxgO2(y%?V{cP5Y`SkID;8TEdwH9Gb6|d7U)$C_{ zpoxs5j|4N!C4!c0?rR@@v;Qcoe?7X}y%azfdHeZgg;kwX@Rfu+2w2qxMF-%_IU_8d zH8SL@h`T|?KZKQjjv(gx?Xc73btMY>P=z$)i?c1Ge zhb-C&oNhJBRte;cL^>|$w6u~vj?r#?NooGo>yD2g3JMupp4?jlufQYR2_+O1A!Cz= z3FYX6oU$a$)ixs3Me2>_tBzP?DT^!{I+99O?f5wv~( zj&|+pH9|v)XIED*^0tc>k%pB8KmJ<-f2ZZ+&y8wGC8@)>&Vv6?^3-;J1tuH(az2eDJdx}DhkS9)KXRY;Y4N}Y;1?! zKhh^-Gc&$_zDL24-4#!sJn4ua6ZSoGmzKUmp6Rsolilp+(CDb77A2@`lp&HMrr0za z8jfrMzY75>s!Wv}g$&6EJoEs3W?eK)%$yZ}EJF*6m8GSiS62euZq;T&DYvn)(Ppyx z2@TDlUr$dD13f*dfV;@$@Zca3oALKrhxzB<$R1R}+kf_oRLZTbt;L~Hh>nh4US7t1 z`0$5u`R=Zxfx)c9e0$pQ&8wySRn)W)uO&nMF)KSeHZG1G|EW`*E^}N#zj z=aFb=CF96!-qWiyh*X;4OBR#7`sy-fGfhXNeZt(FX*yl*rxX@$dG!hfmlgi3@z$XZ zBMuP}5h0=H&YU>m^S|EB5fPX^`7UcC+1c6tk>%y(@7}%BsxZA;=eKl)<>hR3y5TZ6 zP*D1QvB_9kTAG_L|85O>^5Bj<4QznZVz{Y-@5v;;7His-tNzk%s4y?jh4yE$e3RruTX+bCd=nuUK?-p z*!xJM81L8A*m!n!CMPGC-%Egn)mP`VbOZJNRFhZSklSc)nrM(h63;WTy<=}#*{*OB z{=?Pb(B7p{2MpNNX1`0{-W0*Lzlpt;Ri4xJF0k9Fz9)HQWvsW28~F`<4u)3-b1whR zV+sliM&Cw3Ve2HSeA0&+9v%*tq@bYCtpX1W8pT5?3%25+NeevqBD$i!3(!s?= zugtJ56rW|)t-;6Jn~>8|j4UTJlTokUdA>btZ?VS6D9qv@E20g@%h zRO1+;&{{nmf}5C?ee+JmQy+g$oF)By{GeQq@J{wk&1!pGytq8iZm9u%U4zffjU&lI z&A2)oM@#r$ZZN(%(ZWKwCga-)M5!z+6w13C`Y!2q$>fi1%N}mt^@u9a4co6XaU9Nm zmt?RdCA*Ms8g}#JiB6@{AdSDdT~QioPS!{J6MF`Jr<6m__Or-co^zJgE?>R*>I3J^ zFI~s}Sa_lF?`mh5Jyn7sufv(Z?}TQMHH2|3ObLGp}YE@zDxZ&9@!c?&)By8RXH#G6A!IlaHg;5OZ2SC9QgX8J#a|;G zok@Ha{q6+L(4SAueiDsz_K3#a_I49M>ftHL_XL zAU)`s@aKYH?#QFu9Q%(Dubmawz(7}-$@v~(_j+>3i(^3_a zcT4vADKugbPzZ(TNV#RJpN3_g?gTvJ+i2uLCAeoPs^yj;g5>@=UBy06O48M>Oszdr zwky}?Q%v#N`OIu(vWpZapElh&M^O38t($GmXg+3EwiQCMyRC&zJ0%wsrbX3lUIyy0 zUJe*>k^c#L%&*7HFIZx@wxa70<+%5r`dE44TO>`r?%_9IJpV|axZO916yl^U>r4Z! z4BIEMe`IhudiwN=o4+v7Oco3$q$>}n>k;@&5S@4sJU}FHJ)mld#6iT$TYj5Y-*mxA zj1wHCDUpqT@5qw1@1YQI$hXn$(n4q{j^>e~(!5c~Ls>%ME%8wsedC$s_+X(U;(Hdq z_zgaSmIHN&X{0*Q$=J8&OKLH^?yuV36;u01W}UKyiBHkpU_v9uqvpfNXB%8iQ?}X^ zWS(>(SJVhP*YXR6IEIABPg4v#sGEIXO6|Yi_&BBfaHXtG{@`yCOY1<2ac9Q~68Xqp zQ9sU`;AB6?j#!7M+2o%4+miLw9-4bZM2Y-vn^=hVVPOTr5P7rBYxQG!DmgxM>i(GfiH- z$^1)eYm+&Osr&o;-@kvKn$oQwZwrZcM_Bn-;p>2|WeGExL4 z{Rk5!<^0+4-j5$YAgcd3)-mZ+Pqp~9XfkqfbLadB#(IE{FUc+gxc}_gvlicTd-KBn za+9uV+v$lSof>RPiOkH*^@%D5X6DVlG|~F{`WUhll^g)(eK=qIhYv~koXz1ZfgXI? z>QCZzQb-j#|3Q<=VKMOMOsvXvl>E#!fZ68(lfLl-6ijQR42xiYfAsHxP4#$Lr&rUM+(6*;D}s$Z+e;Zw_L5GTJUV||DC&={l#K!q$nyX3i9G$e;>lq z&(LsAlFTQw-+Xhr!OYkg7lP9uR6l-WWhGeD$OW+Y?06{^6@#$Sv?rl+D;ke!@nOQ*a>S{pJPZ%!er$=NQ=6$hDIFaAC82dnoRee)G~`(CoQfa&MXEf>f{Z*;_X(YnUcE|YmkB3Z$-PiZ z3_1-}QbHFV#`?c`L=W&nz1as&Y-O~*x!kd-E#6Qq&yE?I+nk3zii?ez?Ghky<>zh+ zEmJ;uZDPEr@r^o;t72Z`SRtoGle*+=ark$6Jc02;LJTFr-n@6R%r=HSMrE3AWN{NZ zyUT8TDDTYjGYGmI6mvJv3pA&Ke}0 zGF{AWrE^Vf>OQ$Kb){kwXiE9|Bcdty6D?%j$r#v^bCo}iNj{lF-0ERCP(~)(xWi#J zsSlmT3Q6^EGZ6=H1QhZniK87a_mcF_J2mR<+fqe09-L~9Zms$I-<(LqAtBKd%s{(& z9Z0?glh@zvmvy=+k&&O5_K+MZTh1;nNTN>pM6L~%uoq#3*ogR>DphE;Q+<7Xt9wfe z3&SS&Isu3{2zx%izJa9EYj72W4TNW)T2IzG7(tLFD5VJbXlrWzU5GQV9V`6}99KSp zLs~|Ljg>X8%;q*gQc=HorOuYb`$uKA)Ac}E*$CPuClgns0ZOf>>kkgrM(>al-PE{V*oK|_E!TA!jUsze+gsS# zz*jfEhp%WZX3*8W%|c{X*c%g@e<46@dHeVgSz{8}Y3@(~at5gr0L59hlR1Mb6!hczeABt7^D(Q~a40DKyC-%Xr8?^#GQ-7g*_n)_j1?69{_`5|Kn>~Q zGA7*XJmtyf*2*$Yj>_(%{Z5F?c>G%e3zOQ4z1GQ_mi2dqx<$|PXKf^2@;R*f25$)$ zHa6-uL@h#*h-234?Cks?!;oPfBa$!A949FuF;nXx%7_;i6Vo4D)m_5G&!5<4I6gM^ zRAtP%x~OOawlKcp^4m8t#CIThHa0f=E~}LQK()0z2};`4*1&wF2|RP8w%q&Rgk3|$jyHz@Kt6vz>ntIy1-AwMd4 zOSH^fs>Nu8-O(-=r?PhJ`OPJ`4tw-RYzIYVK*shu(~qUMjCED65pf`T|1}E`9-He> zAhg~oV3Qk@qq2aENE140D(-eRv<&oadtJ!Yvu|Fb=c8t7gzYwBp19Il2>i}p@{im- z4@dWBDp8Rc&n;z)>8Z1EW9g6U3u|os3#=q|wznU#8v2V-lap5k+wr)u;MKQek9tPTTJlZ|5*mz3NB;swGJY!VXi zO#CwU^OMv^b{yr&QwP6=xY5+9X7u-=H!c|etge`2?_vnT;nUvJ=3Vm{>Gf{UUdpkR zGM|OkVD~E>6S=7$YY!0L^F_4^?whcxSYB$EG(Tv6L&8F+vulp(ktf$KT9&>yc2_GtDw?F@U4oF9;$ab9Y<@`f?y5$sMO#4p2H6vH3N^gJf}7=y}oH&o3= zZ{NDdC5FC-sL5nA?AEZPFKu+fq3sD*Hs4yfNr!NZJdvfENma>2z$)N_vBVtj%m0X)-HlTyui+fr7#Mif`tsbawW5K}1hit1Z&!rp^QoLBY>aV- zA6YJi>3$3uSVhlhI627m8>KqBWvCu(=wq6GaT{$Dh26nHEW2ZeNjP1cc>sM0#-SN7 z^Jttn(M<*21@_OzT~}pg<<*OU*w|S57<#}E3mK(?VI&fXhleL8@v=-4D2_K!1K+uhD10h(-s;b`%%a%`vq3U>E5@pkg7pV(- z>}JL;Mgr6TIF)KL=Bd)-g#sL*VuA&>{R>Q`o%s$HgJ$rQe%JErZd^Z2=$%awI?cfs zY`Rx*m;UfFVQY9{13<&g@+%{c${dcamdpex;P(=0}(1OZ&g?@&>iiGplR_K{2Bmm3FSovUyDm&?j8HfNTIFl zC6QOIR=rqH=4~><%q{9u9G0gbsBUWH^Rv~g9HVXlJ%IX&$^}P*fcvXk-K6l>Ku9uuv-)dNgU91>2%Y@k*L|)o{ z6U1hsi^^LupCT++u(BE=GRE~}bZ~Yq^nH!l#3@8Bc1m-bz`%P)9MT-gIn?my&mRNd zV^C>rJw0KuCE7*Bx$eCC$@=npj;VFJhrE%F@vzTo@{{hxr z63-5hQ;n1aF>z_C2aA>@H*VZOMGb;i&;t+FkI`v2LjG)u`heGMMY9rCy>8J5|YfgB!dHkWrXh`i&W+4sDD7lb?e8+*o(-?ocZmnA&*i+r41;Lkl45v2-TGMpgeQQzCA^*CwE`(XKG*XRDVK?P4@iB{5X;C zkC$aqiP@fmfm+B%XC?vOdafm|5^?h(LY4FjYnn0L`%zOeXqebSA-Y zCT%s=X?EVz+lK4eu73W9Lkv{ee%LX@v(Yom9-=k;#2 zjBRbTR71WDlRx&nHRYb*HLt?d@!FZiOA(XJfq~Ci*nUeMJFgTZz!dRoVdE}a4C*!C z9n71`@?z&8oHK9L;cT0kc_WE7{!vnWbQ%ZTnz@z~+8ahIpV;fM402-v=jjcGE+f#EsT z>@ztr;R$NV)HH_I4Pq0lRzRrx_wUou(FF$wqX(q>U0xWqzbE$x`}u6U6_ZK3(%#95 zNWk4*T)eGj);BRVbv8>DKS~NT6!565=d|$+@d6M-pvOU{33?r7u3=(gwt$oM`UdK9 z3%~37yZ7&{hx64yv_ljEYFXRawYcvXn))w(d5cO(LBZ#;%HguA%J&35l9}1|dn7sd z4}K9sd(eAO+w|LWBj$It1|KYph?|th{*!W^fa`j^CgXif%%h{D66@avr7ysqh6=oQ zaNx81=Lfv&^wd;1F>iW7L4hiLBBxb$Nr|1QDU|icF;`VTUpbFixAujV6}$?Nnv{^h z=eciALz4sT1IUhQt5I91(3d@bUh6nrr)NLgODB`UUzc;Upc>n6p|-0DB zaQ^lVd*bNp(e`%kPT#@y?^ejl=1@T>>OACPkFhj0HNh18Iy-B?&jm_(Wo4!9+qW}t zIG||)ZuCsO3&UXhAS7dopqGA&uOKC*(e|$|;D!JZoq_*AkD!rnOjHrEn|5EEpCa4S zMhS&|8)IUIvt;8ysF<0XkAX}D_ukR~F}JcpfQ_Bzvr=AD8gLh6_TkoSi}$f3J9`CmNXz*nlNs>+f#0SgkO(grx6dwY9Dg@te4yonEFzUmGAbteSF_}fC``KvWB2-++U|b}Q=Y>aO4cynBacDw81*CJ{j@UvU?@Fd}-5 z&-l?PIXSO(Ku+TwG$umZ=xJ%ccXh!#H9+lk3dL4Y8HBD485tSwe$cfI7!U3H#E!0( zwI-RT3U!;U7yagSKR-X{?YN!4ee=%*C|PDr3#WTbG1&%rdNRKosh}sv0L-(ZG*-17 z5Gi>J8E9yXf}Z^R`PIylD07?@2O*&G&8zO(U;dfq5*iAMIq0l)_u-I{r7kajSTpOH(x+jP8MjCz~(_`>j45XFRS0YJ&Y*m4fJAOaqq_Z z2!g}W=zH!dB7)*{dQ1=f9oNf?b8^wZO3PvNtg|m7{4ndwvH#ET>$XCf(oMLA{~U$= zZzp7bw|8`8{L@cN`yx06M|~nk7+}KptKr&zKeAm762rjA|9s;A;>G_s=6^l%KVJOL zxBA!5|ILg4@$~=i8#Y=^)o4=l^YgQ?V46T<6a|AJ1H#(F-Tl)#{Pt`5+5giLh={iN zd55cExsI49H=>aqc0sx*|E&&`w<|vWZF%q>3MnzgqPRE&!k`g&C zdX-tEOctP_r;S9vIR_(Ns8Dr91)!Q}6&&*SGkd2n9sEW|m#3I2v_Bh!Nj6jGB$@V{ z7#uF3ot9O6TwI4hL%oGs;AFxGIb~#IJWuv5#Ki-GgQY=yFDv(^3j0A<6g~ru9E{?l z&1n_-7^pRXX!DMB{*lnl1X3hU{jsjL*4@Jc8{z2WWH(i-q^oOfYrDC&RzQ*nr}F8U zS@eWtR%1pro`!~opkNckKD0w3u^vEP#l>-3kCmpUr-QJp|Z~o zv`;{}Lst9voI`@Nx3yJSjfP(_?jd*&=&Ay((bU$S7#|lG7x(b+7@wS+t#?@iIVU#- zlLxN?QBm^$Pk~FYzwZp;azBVNMn;WZ4FEJ+=1D%J&1bKVN;68bKfnTC zn<-NUcw6Xv$hRq0q|VR7GbGd^N6pU3nH6zxaM0D&RqzKh?Rk81vOL-f;DojHIu!fv zo*sVZW$?U|3a)}9;{CwYCj!X9>6yxaOkvssA0j3;cAx{AwlD|aED++ljE#0vCQCW( zx%1kHe3tSRDgE|se|;RM3R3i!Pbcj#toC4mFB-Wnvj%ecozjo18wTdZ-E~qFi-99{QhL#;{4ZuC< zd8CVet^G5Im-_mNiHQdY&;4bnT4B&sH!?Ds(u5}er{+@V%)yWX8QIqF-#gFnBpn^u zoHnj{^=F~mUV79zfv=SKj+jr~n~u;;%>{*}BOV*fFa|F5wT z5eax54!L^#k7*JV#LVUeEo2*!PhtAmvR7Yvh_=aq6G=4Yt(WP+ZVf+o-z0w4SGv0JQfEYh#1 zyL+U>z@Y04`tk1j%l+_YNIH!YeNIU3{e5qk^N^DIzB*hGfFQS2y20Ul_7}z~7H46+ zAs{FSg4FwDCFkHm8(QdlV8HM?Qn4>A_h&%01k+r|@8S%8sYcuj^RrfJAwd0JW;APOO$GcEmOOOjG9gow-f zSbul-J-Cby1nj1?xFOKXy0Y9`17Q#pPyy2`ZmB2H%)&w%?YBLTUHv+|ISB~~j3j_? z2LJxutAR*(&(4mHAR#!33mStHI2vXYMn*=iTD1e2KU&nTTUc1=R9oMp!osRU1b#&Ek<=yQ_~vTY4bo%R#qgdQQ$w<9Rau!o0zyZ zH^*kui3|R7qvCc6`_re7{g>C)a1e6x@^{E3!-<2#!=dBLz`}C5UeWvJ?OTwCFU`&4 zQ&N8Zwr`Iho2|Cd$x^NZ=O8zi_O-}U=jA?Vd9+A2Seu(m9sL83JSLsdz&G2!eM1ZQ zdU?L(2fVr&f(4%AAr8;WqYG^_#M7;^R8v+S03r^61*@Y`qT)dT3A^nMn;wzb=)u0oT{`K zq*KX`Pe@SrV}N4d1z^X(zyKN9bn)raV{M&m#Z*ucdt+l`Q1p6}cn3Q>SH~;N&;tyY z`_f>!rKP2TI2-t#bH;T+JKf-l%0H*&b6tP*=+Sr6(O&yo!D0wBi;2p-fPgk=>|5K| zs3BQ(Ni+czrKfn@_JzKubM zOH91Jw&t}pBN!#c0VM{8i{7=(fo~V_h;^>%BHcY_)lZ-kB4`T=1NeBRGTz>e$;o== z=H_r(z8A-4d}Xi&w6x|hBW3n838rQB8#S|(#U&)vRaFbMDzdFci@V}j%qqsAw6-VS z#P}Eyp$KJfZ>jfcl!hyDV_`uY4Lev2asgD;RHH|QF0-)Tg*R*q&`KJ)_)?omt}@Lt z(v4qkvp(FAnox~r8$CWM{N6hTOW?sj(^a(0%oi&;sr9ZKd|rpvRoX>Ozqn_;=NA`8 zU^!sdEfhO)nqN*dd2#z>xUG+Uk&EtHfOQ5b2j?fRpb#Dsl0GOXAaK>3Gd8YtT2erK zhXScnV@raE2U8N@Z7H7@HG*n{odO&MZKjEo%RaA%pvkDp0%|ZZ7$dU)EnMwN^D>`P zS7_JpIxSv_psNr7xdw69e74&K1qBMxbge^1v&<*)P`o1Tssu=JuqmJo1LxNPo~{wp~Yecf)N@sY1aa<=9ZVgGwNjBYMYw!?oSsdzHc^KtOp@U zDe<1iVNMJt%)HLUcT%b{G<9?^A3c4s^fOuRC^08z=`AXz@7a!Iov!}#=OQ!{V`Dr# zJj*L98)Idy9UUF*?b)zWz!CED@|5%Te}TZnBpov6xHMcN`)pNE4`U>NbKv5|#o?#+ z()h!{+s9|n5+V`u=3!^wgQy`OAON@biGQhP;rsXZoSgQKOi<0b<6pjf`3&uf>31#o zM*=)(>*(n{(sio1Xp_=xShl5 z5Z%cZTvw*@N(5L;u$GaLk=l5fK3@KjoK_<*#O`uEAoC;Uv424lI=7WU_WI~3bS~MGW|VYob@le`+koT)ezLN%57EVe_{qt2 zTgBN-)s{(7@i{L?YBCBr%zcA(3*^2-SP0t-6;U@&)zi~+fDTwYZfF~v&*0!-MOhiN zFWv!dhVeXES)C?Nml#T)8ydR0y5Jzdm&v1R!&O=2ozU2CrNJNzKL9lrPhf_9fTo=&k zUek)VZ{Grggi#Wz2h!%|x!OgWlQk?Gn4SiV+LiBwo^o&)pG**(oSb~5mY)FY1juoI zYJl6yfC&H*oN7+kSygM!#cUNI74Vjf$d18`B96GGq!sCx_L@dn`7f)gXVa=1}i@yJsp} z*xO%?@g{QFgbcj$yS>SNcRMyk*ss~JEd+3={}oKt9^AchJy80v{txrUc5tszW+oLb zG5m3(O6}nz_)bW1Wmx=|i2v_ssBnbNmKCFNtZfgUn`nZ)UV;?b4i?`&tHmkOg-o=V zG=vD=VKp~pE5fn(*I{M!Hieo+%t|0}Q``O2@bcx52cV^2)#vs$r#RiJ=@8ad%cS41 zS@)Up@m1&$?N%99kqw)hp9eO##G4v~`3C9njwXm?zZ9nAnXbU2SN`~}; z6@)TOlv=!a@j^zXbF%0eeM~XbzqxP0S0l0@Ju6cI=Rxr<0R9n976C4TG^DhxMy0hYLp37ch#Nw=ifk&Z1q^^Xb=Y*7cf1%`%TA8}!Vu5zR8=`zj;j zEtlF_T3Q06B`{#4yGaz0bDT(E0^oAR{M^VMo^d>TtEQ{l=y|XTo}Ufa7|;yMQAiF6 z4$foE7@AKUgU6W1|V|DNxEar*jh^w56yfz^epfs{w}5%QU+M zT!7sGCfz=ztMxgp23zr>$@73dvBx*VhsNLgcp(ll^ykl?Bjy~=AT>0gO;&0>UJeBX zNXye4?g`7JCFt#f0Sjyj&_T$3`-QrCtW1_RU@cH_;A=~-UcD+UeVU~VB_8lU5RR?`jaHLwPFQEx@w%fG6A9+Dc3RC{Wk@_u3^f)XJkF zji+loMYNXAc**~GDt#s2873O|R$s`(&0S+Lm|a;|qFXn&xCr%w3(h4^H6b>31p32s zbA~W*3KTa+i;9nLqP?A(l{E_M0Un+y%siQxaKilF#_$ZF3GA$YBupy-XQIOmDQUTw z=gGdH8c=Psc zSP*V1Du0XWaiEXQd_-@+4&h3i1}95TZyF4P_r$#A;Otzv&#A(*!2SU=^&8z;C5mlp z{yhr-V5?T3E}trtJQ4{E67|+Co3YZ@zB>y`ODw<`A)eukeJ;*;&+#;Yk_=?Y0+_cz zT?UsF)*uXSI)F>w58Z;H_m=ZRB68z_OljcT$EyV;AdfOKGFGUkfjz=WK;h$s?rEm< z$8FLtv(6BdAdOB>PhnO=!>_uq@V##ulU{u^DRSg0Z8s)BtLWA_1|Z?)1Dn22&vcnb zK`M53(o<5tRZs*46c%={vyNw;Zky$<#7JhR>EB#NN}lu?L8J174`A) z`Dcty&njsNY8U7Ye6*XOuc2Dh!)U&eQdCou4-nmPYgfoJxS1g|Bm_uxc0mD@J{{(` zD{~l%RFfqtj4N}4rw0}6#16O$;ASof^XdLd>)G1;JPsk@2@I@Ql)T)Ws*@v;_I8M5e1l|+ z*T6nrzy6{qd>;MrqbSg6@-mkRO2;Acp#fg&@A?BZNV+f0W5u8!fp6ZveI`F(CnX$MmYf z0)u+tU+AnJt&fYMfiBeof|IM10k?Y`<1J;Q3_qc!UR+x8+iAnUsv)4%^94=;*C7l_ zbZUwrW`of{m&nF4_6vOgQwHoDxr~0e|KX{y@GMNfD}b%s0o!CL5Zv3_>jw8hwD>jw z?Ssi#Seb)CMPXz@!hlMxw$QDEE^tP8^}*yD1^62n_;BF`w`GKcgur;CVgd)9fIw39 zH}+JwjTlhds#+UBlP9aJB*}z5_rDJhSDna?@*M zMgX;a`I4MHv8>Dih-MhcEr2bE{YU(+g{}$2T-J~FMgZMhwr2FaBsCe;V`APDuxH2q zR{$zgQh#cU>J>We{`u4P_tyY_Ebwz-EdfUZJLV5Ir%%BT<;3$55%~FFJL9bO+xj+_J>G}f&y9AwVW@Kh3V7;kpSeJ*e<-ztdej#&}^r--CP4gwug5fKrE%j;Gw*cB93~ z#l^P3?C0j@(8vL*z~n^@ZQDIO{Mp@|8OmWdGX>HO1YB!t1|T-DQ;;xMEgxyA$xqMT z-n{-Ad|Fldn68C8ckY0{1uY)jU@@3?I%ueXVm)tF?l3PgnoJlbN|jAB-}&< zmk*7$%2cS)RY2fDcO@q3*xE8X=>km5+w)8~WoKr3@BW|xHbTf@jwNUd<1rw7JUu*y z|JH;2ft~D7SQnEidJIGgNFvWphkj~-d~HHK#l*xQ@hslppa;PBL_i~OJ3zXHZEeCWQ8%w&Q!#*DhxuN|%_-h2 zWh9j45`(L2DblJ4#3~0mlD!hp9NW|+=(GOom;G`d`37&5!Pkk2gk!k}2=D-+qLgA5 z`(TWopTD6#T1`Tt9r8K8^Ba{`=hv_4aQ_k%r#eW!SJ!R;R5LL$qLFWIZp!gfoiXem z8~~j0gR2b|7%b$?TeqrUZQFV8qN9TeLWqkih6b$~(2x5~u#zs%p1nq-<<1xsoHKzy zGT_V&P$z;8gRXAjxr<5*XmPjAsbIIW7Kx1hdqiAvw77u3+%^;YkS-HTQr%jdFx0j~ z``zBv^{UwnB2J@Fv%gS_9c}@;>cAgOyOm6%z*cx;t?tt09!($Z?9{5|VW4{m}R>EiHvA2*^GP^&4Db zC^?sK&jifo0%Rl5v`SyVp!VJ6?uk}ExbGl3B}L(_v9@+XU0vN_(nQXCaBMhepF_)8 zISFCvhBPXIH3F3jmB95MLsVK`o{5(BQ!?)U=~F?$AOF`&08_nK?N@K|y|dJ=|55p&F?jLqpNw-_vpg zuY9x!pwi3_f}vUXB`Fpk7x!t!ottk~<@v8XcN-<8uo3^17>@idfI={cv@Pck>Ze<<5xH}dss^gHri;vXt1J{eis^St2POF8mL zWj9+)+T+{W;DK8dt7~dTM@Ku_u5Lx#iO~iAp{_nMF+qrp{rWO#868l%5xha@76NX5 zx(q@h$AKqBL6h?7Q~Q;HCuhFtKtI442gwDXOxyqE%S*5$-;gJaqF3GRr;$bbGX$9=1moH zW?_a9ypfWU2$9Dy5eC(8DK+98W(+(4>mb#zLC(U|jGCGnuwJmH;4k2diHQjscrC3cx4>xp5Eb>Kvy%-9g#6a}Iu#Zg zR9r~LjjgTA!ugtv3`#b(f$3>Iaq)$UU1(iE1nnOla&d6D_CEL_NGX_ktmz zM(t2U8vH-R{zqkJ9+vaI_5aLM5e;th)Fy3FD6>erk__3RkVa`!GSqHRBD*ptN|Z`M ziKLQfrc$v{iYS_tq0k+gr{{HNpX>Mh&UK#adLDmu4)@*n`x(}HulM`C)<#b}@Wc`+ zTfuLBtPAR8`;;|$dQ>g3ckXE$>2z0N8=xpbS8+`EAR!__vrexLI6u!uv)s6ju_sFtbEtjn$M~$j`6%%&*wjJa%*}Cic#*aV#psIzpBw9X;xUa9H zqoAnxq^Kw%I(kUmxLedQ6fu8n-yV6rdy~rw%gar8AlQF&pgqO&F=VluiKgy#e}N&q zy`Q0sfH!bT@mPA}9P#b=`qIhm(Ulr8Ur9+xNEAgW88~ntNd+bf%3|T}%HAkJbLM=o z32L7yH!QcLr0wtXvnVV$(868^-F9XjmHG1J3vU7TAQc+FJl6~nLA7`=`!0VNz0Yad9Z$FJ8PzN=m}IwKyjI&6_t=w1vmSj(B@}A3khw z$9ly6s*?fRh_hUb`lhDgva(y^EG0bDM~og#xhgm82(`0(AQyr|o}V7UtAL?q%$UI` z*?jMLQPGFbpWA&TP`hTx%QxSc90-sHyCm%0o40ROX3tJt+d`VA5A;qi>sunU<&~T)A>3 z)=eTu*|(>C#e}P8tEwJ7d^qCTHP@6R|H#Nl?;+z`^(0Nq3TAx&*=1f+vzy~80*t~**lG}esjg;yy*4&9(!1YD1#vT4c zl)H^Lo!1AXW6|PIX7^A$G0ARkj+mM6kb5K~Qd64a?&<#f;G(JQ3GL-aFy3j5wN|A$ z0#gf|Fsp|Q^8IXHNy+ud$fIgvEfgRH;j1wK8nMZX6|FaK#zhyv`qBNqIQu)o7E$5L z=g-1RFr1(%*iCa2lk}$aC`c0`&EOE79To>UgPb>unT8~>2@`r4KfHN+ByE1dAl_ zrmsPso)4+ul*Bo8YIYyja4iTAylj52Ke%}!rl=^|_}yM{bHV$%y6EumqQXMN!en}Q zUHVdOQ%NJrF5G?)TLC*b(g($dm`0%Cn@OG}h-+SrU+;Q7)t+>-{& zmK}1i6b&9QUQXZ!;aOYX2nZ;qdRsH`;F9!lp2tlM(1E~crp6r+$~tCd6Ft;@&YVGY zH;H>aO4a%>SI*m8>)D~{Q>QA=o_#^d)SYXE3sleb5JB_c!Gp(+4e~T)lYbXsnzaKZ zB!25NSxye2;2FKm4vO66nwpwex25)DDK1DBA@l7cahI``dQVBb6kP9Chg~iBL{y=^ zpho0v5cCC}KvEb52?+__7>ADU$EvDzMn);r8>J2OK-${cvi6WXF|R*k9`Fsma6vb8 zjnHX;mf|VRFeuIC>gwv7HtqZL{L1W+QE6#q85!nO=e>L0fBEtb%0yaPNZkR7i_tB1 z)+~^@+dOGWNoi?0A8j6kEM;|a4^qgbOB36tw?dU}+_>>p^!^-y$9)q-s+5R`h^tpU zSA~aPy&Am0;VCJtpkNkufs~ZDFJJ!2@#np1j5UnkF5_%xv32WK9)@l~7|K7tzF8iY zud`wWp8$X>i7#87P+9Qm)w$EBFWtD2406G?cp0M zwSkp_v%gH6cKm9>EZT7Mtz3$l+uL0TJU?_V03L&|gI~JTGvgNZOsH2jW*Sp~vAhPL z`)O{j;oV&o5x>ryGiQpAHl!P$ipR?n8A$^$UVvfFv+G&j{B~Ce@bNy{oC_-I#ho8A z#%)|FH%wW46k>`%kRCD&m%7>yX&K+ANpp2x-`g9GDq1(oK~wRR?ThA(1OloOw7uhI zGdHWNKY&NUnN*=hs46=^dEjY^eD}M)zC{22OO8yFeVmzj`PMCrYA*q^KYlD(^vHW; z7~rU$gh5E<;NyU1!bW+=?R)pmGo+)47zF9y@~q9q%IdCtt!&NeM-{skf4kVj*x2|- zCE${k<6h!%@jG*dbGze@KMZ5cJVf>DrvhNokAk57O0tnQ~SdgJCzP5r6UWPj!1um^Tt z6cZELKrlVZ$r1hnf(0W(bTl+JmL@qo%gpROd2z?thp|!NSFW(_S1w+>h?R{9YwPj^ z66&vhzX0p*n+W6Ioyx6Z|CB)-XI|&$A5085nQ0k^C&jb)NdMmM5fLASLeU6->(^^> z0g#jZF+GIk&zuRQ^yChmJ7)lrv9P!djN_&(R$FxFK5CMrl+^FEpUgRa-=th|rb}j4 z7JE^DfvZejB+n4M5RT2vP>Z)hIMPxZCOda#OiIcaSFMRYR<^d!%wov0dY;fu9GBFV z2Fy_|`-Tl0woW0Qhhnk`Q&2wE^xB^+@K_Zd7#J9~Bkl6#s`B#jo~C>)r~o9e8Og=Q z<_pBBUCGcNedySDbq*|*3$3sGlp@qn65$hr^UNq6v?0YQh=&@sX%hHt)2)0t8 z$;ikA1qF$?{A9;^uhfh5B=gH4GWo~;j~O?!0KmxA-FOUYkZ+=AOj9ur9(dlod(g|V zv8Oe|3^96-nlpX;`0-Pwcs;Hz1w}eK8tCiGM9I_n3EwHK>n(Pm?jQvLK-1UHPwUIX z0sD^&=x2UFW(9v>xG*pf#OJ&!5e=ibu_a}ry81*+gQ!ERR*7m!>IIiE>k)`)Ahj_! zUpDy+&Xa8>Cc;peE8(K>#u59=%Zu*j!$*&LdU-k6*`dWEiW?Xj!d+Mv!jASKlwr7s znqRkR)2K0HPMDta^wgvSo8_aTu>AX6#n-Q$p$(cE8ig-j_~ceGS%!NarWxYHE5qin zDxksQ;`!WPohzrcFY~uN#s-o$%mw9Y#Jp?pt@rLZfhm8!a}Pfq8PK+}@Mrg6!TVyA$e;T!#PiQL9`2sBBsRX=$@ec%R5o_^-zz>_G)-20Umfq4`uTdclJ&^t zS!5999C<=^%ovHb9_=LgDM=2tt_q%}dfT^rxb1%Y7kfuFqnW0CBQ-u*Gl zy+9Y@1i5{U1x!K3kA}q`xsT}Yzz!JSDhJ*{&Ua+hQl}7j4U_(~(bn!CEEqR#b=7^# zIA!q#^qQfpp+FO-b?)PVWt>r15itN(nH#UHq~zEvoS-AJ_z)q91PFHYQyw11b>=YL zj*qW@P9?z?y`CO{0ByPf9b9Quks&PU~!~ z{jkLhV1rmM{8qifcP0$Ck53GuA3zI%?4)zU2$dm$Pq6TP;~v0$?fo$Lk)g41nk@zM zh9ZJB{R|f~L6wkn&_=j@?jjUS3fy39Tvt`45niCxF(RqQH~DlYc&&1cDNk7AB~+GoK#0bj0h6aT}k?>F*fjX_{(3W%Zo*Y&o{f zlazD-WR1#gP2nx>g6e|63`#**|0$3pp8#=+a+9Tk3kyFfzX1TSbRSif=Od@u^S^o0F5w=DvRt z*x4znRS;xT+?=orgS$CjX^j+}Nekc*h_K4ayW!X8?0kcrxIRMpk>F0q^o_Us4;kH~ zxfl`Dji!G_@0>d)u(xlRv0(Yt@bDGd+L_teocKj_gTTEtYmw-`)Yi7`^mc##{5idF z?O(7l_wL<$acvR^{>v95Xt%<`%<4UQI-n@B%V4HKM7wod4-M^+mQtazJ=op7XWv8c zTSv;v%h9wRKGYoSB{gCMBUR4&`dXNqOUcM&S;oy9y^4mH?v%CgyspUms`b?pY3eTz z53P>dIHcgkk`vx=06-iF(Id{!<0T(Fm^g2q6NxA$CPv^HWt61aFoPAr#Da;c^Q4(? z@}0Hf8E3fFD4XhA8mz0ymZ3G8JWV7ngPjeJZPeD^Yl>mTy-8Zb3m~ikrE|&(0|?)6}#? z*afExJYg69DN|;gCS>?`3_2=Cr^M{pvpIvfFoXgZrW?V2%$<8KVM|d-$tG=WD>}}s zRg;pFFRThbcK9%adjCO#z~CP~d@vVamT16(MA}WAUjdn-M?2la~{pKC;*-GMj9UPwJ<|0-$2!+@^H3RYT&r(xU z14iLBbZ~IEo9XWMkqbw=16V8O?%n2`)orb`bD_i-Rn2ycn zhz4M^61kibe$D(t_Z=~7LbPj^`p~2#H)To^T@P(-l)s}FITzx8M+DHF{N2{uUHS6O zn_JedbuYTXwt>zhQntnzAWQK+w4`83dY~XX z>TKHc~Epf&8R=es0;~OWqfy6DAgQ~M40$whg)QdhdK~dMpE(|N&v(eOdGL? z3kP>Y5dv%Xjr|(pdcSOc+e+*nx|72akQ)X|NK{Kq2P@Vmz{4+~>EOtb0k#A94wsg1 z)oAayhxy!q-VdP3P->fZwxNOuX)$N6r$~sPEqr@|s10027eY!sWZsyzBxgVeg zA3+D<*|W0fK9uQmX3s`xz!IPiuEzwy0+UkF(bSI>zI=)3-3n6&we#fZ(^cBq7YjbI zdJ8x<1SjEXw8lIe*Oe-+ipI}n8MSfnamyKzGou~}D_cr@+TUl}d{ zvk0zT!3o#W@(K$x2Tbn2@*$p7#f#E0fmax$Q0grZ7m3{{=*%AcXy?3rdktxAA=h@% zBEVruNeLSdVkpG|%PLZ}hnln#N6DJAf>zai{YsBe31r5h`zA4>GD+ZxY=2^7iSNG^ z#TFLpV0H`*b1i1FcDgIO+)ixJ(8$ETQ6ClqTn!J`+OlPA$}eYfslO>5lBdEzLK#sy zz%RJeRzE&6?i7cGDP|nPFw13qPA1;Ce*ISu^@=_B$Zy+}1Q?mpi(VkS1qH!1Jha`_ zdUcIFyoH);tKHtcvyNV%IR?T;UiuM7U&P>UzlW;Jx3{*nQ8_hYWQ5{4%0SG38j&H1 z^(PeL3W+${KjW5LS6dmJ5Y}4m2&GVu9z6(5ZcK5=d3hZHhI)(R#yqwkv*?~dQJze@ z-=ME6dkC&wy5yc~vGa9oR#DM#+o~*Z5QASh9!(7m6%*9?6ylU)v;{UCBKGF#3DkSJ z7OR36j<^ysZ|_4qJcWUdlh0hcaz#c;Y8GaF(x{?gY!#=T0NgB5;dRnIM#Z*tVY_oZ zXNdU(mEmEcyd@!#m7PuU@gJ zrRC&|&){JR3KTJ@STs9XtF~dOLA{DFk>4~g`Kbe)THLB~%BOaRx8fm?Sf`{fOzLg%BHerHk^&`LJglWd! zxX|EZmdicoiyMJy+lQPJO@%PHwM0XsiY45$XAf2kmIgWx)GqL(;-H)R@RIT9i{R_Q zikOkOAQPufoo7=#o4s&yYGL3N9Sj$>5Bn2>so;sf)LN+%?N=^ zRNGa4ASy~R(Go2N)A{#O6XBS-cp6w;9E@0d@?N}<89Fq-prEZJ%)~lvv7e63_U-Bk zns7AWlHZIfmBcs2S{#Aa^wGY&_8QXtml5rbU76qbHc=RSDd#Fbgw+K;MTZO7W3{jPsSsM7b-Dc`O;nxK|iGOfnY z(ML3#(d)v)w~_BL5=Nx$+_FW~m(%G+N8Vz%WkMhld~|d|Z0!79<>_gi4JaS92-Qap zzde1Rhx?Z65fPgc?T*ZuQl~%?2KFE7YHw_8v{oHu{3pEFEeeUhgBJzrPWJNje3YAu z1#M`?Eo_7U)6FT)DmcL5_e9#1^4z+~2x@BsNG5M6j-h|OKue<3v z5mi>pu2nplDkldfxQDMab-h8Vbws+)<4Z-HN6c(c?_Tlo4ct>eoUpivt77|SUygw> zPE4vGEMlkW8N#B2{ChGWj=@Z5oLHOH)y{x?ipv>O2#=5Gm7xF5@!2=<*0`EN_s+jY z4-QXC3T(2QbtlK-8zoLfKu4M9<7&dz8STrg3d45SZP*ZIY1TXKI>P`24GK9WR;SON zZ6qTS&j~8r0CXuXf5(@;{Ra-L?DrsibRYB120F_&7#Q4zF=AyfL{ZaJCdd<9V6g5h z(OYFzRo~XMQlh$=6Ar1|rad~{UyDM!tXD=xMsx7eF>AynB=r8qlmA!UK;MuV78)xh za9u*ph25MpXU+zScO#>YxCe{5D)aOIdt{ zb!&N_UJYk|d%bx72+#wJ=0|D6_3Q7zV{xp(^PxA#PI?l&C<#xioLujY5HoZ0Pj;;| zBS8OqMh1#}5fD1BU;kllo{>}z+n8yhBvjCV)EZJPcP5w6)-6l_X>ZSu_yUg9O+K`N zCBlsd$buZdXZd90M4!V)6{k*RAxM-{<>kR=5s{IVqXIIMmj6HM(t={)_Ada7uFj@3 zt+!&l(dSpncH`+or9Gsg!gS<`J9L|nD5)UUWvw+dbmEOgt-+WSOG`VO%G~^Ye1>u@ zeSI_MgoTDK1zTWyOfWKpY%p@r63x)@->M{_<|JQ;_gzM+)76~@*FuQ}&0pA4Yx=Ru zEDlZ2gBPz~2kE^jbL$$jHRtcYVI~Um^IKHI>mYUCCOb)u>B7d3y8Wx7V#gXw9td#( zS0aBqN}LNpK>#A`=1z25*3Hr&l%@PMafXIYC{;RPYCyy4d0o*|*F9DJ zZCj_9DG6|Os=pnlw#XYeU*mIV=RV=UcRb910Y|`R6$#a>tfML`>7%~@K?Xp@vBGgm zLBKoxwry7XVoU~Y{67(!;A~2=wd?_y7d4faF1^n!p@}1To89u|N8H^Xa14ID&659G zwe?+}b}Xj^XF3>>nl;vxk%LV)sO42s*V_uTfHaO;#*#xWZhe_pjBKUtqohz zp9;+;PM+Kd;6r{V2jbX;=ZT&sTMKK?=>WJT|AMdhBkv7^MDraXXQcf)fv2g)L`F*- zW54-55SgSyVjoQ!NeIrEBFODHw$-n!<<|0ZzSw!3D1s3qLYF;&sF*+B*~&^DWANkZ znopnfuB~Hf*(}*(P@hdc5Bp$e;sEiwAgqxmEFl^yDj?PlysZ52VYsB^E+`_sL7VSb z8*;GWyTOP&px(Q;$OY(0dQq=9KlNioqB}-eTc_4k3}i@>ME!ukK)UQtI!96D61asD zRnXr!)~Ii)cg?k%-$Ct2a3SpW?imUTfvvX7<+Jz6lTY#dkA-1@Pc$}9< zxf76>IKNO*c#m>|hOL7)Hz6kgd~o2vT3BUzQTWEktCU`=+!u!bYtp1MetugmEguG! zzJ5)^JK!4gc}ZDWmtnRoC=1O2NP6_>qixmsMMcQ?AHRG-nIU9>E1`c#hOu_~@FP*FEmGSuS}LMQv&E>b&!6w1{(vie{PZa(00S6jDCg0m%G!82MMYQs z4k!8i1q(p=t`?r>m^uK@ke;5-<@qTKfMio+W7irRyP=xlQ5kpCR!N+Lv&-I|N=$lS zH99~<{**#g8-{deUD&H{pYlnA3iI+HRW>nD*w{D( zN94d7sbRxZZA*{0=FyqmOTt)zaXveDcE-N8NR1~zx!8qx%TOL zb5!P6a(qX7I|5cpa&qbo<$79=nbf(*!2vB3d*KHh^U`E>QJQag219mNa$~ zcYGy`=H~s&x>Bh24gQtO<}yjOnQ2~_pImfCjh?QeVvnRvG$~26-+}DQ^oGnMQpBF4 zBb~Wh>{)iB-R#D_6?NCtin1!kZB^QGN}981JIYy1ECd!v@Zur=A-FWNCVV6pOHiFL8rZ4YZs`S$Yr93se6KhQH>K*as;Hk9VP-D2qu zH)CS-Fs`gz`K9)4$~t|0R~NB-Z?ogIh$LmYn6_&cT?`KwwXDR9{M=8hxtI`|TGOK~ z{;}6jwWjMuZM(n4&BFvchK;$|RmL1QiC?8S+NvgkLgTtSH;yp;j_5`A% zb^YL2OLr`s?VL~wIIoUIi6x6frj-NujKiFA$<4y4(+BkK*{Nh9o+0WUP;(pHHn3YC zDj&6FaG%~l`{i<{p%B=)fUwP4THz59{l|&5QCkusUTyg4btx<67H~Hp?eeXgqz8Iv zOoUAdTg`@c&3dodrnkhcTXN#RH1JS~**m-06AFHLpDZ)x5=hI+uDw>sdj|)PjnF#S z{o2h}!Bbl;ESiTIPJ0Pi6c-zdV063k`65Z=|psvq<*2|p#gC_Qmg7uGV_K&US|Ci1F<3BmB zQaOBDL2>cG1%r7;F=Ls9ze|dKD4S$e+pWs|V-yN`-__&OL+3W%vr9IM{(kvyt2Az^ Ho4EfM?6XoL literal 0 HcmV?d00001 diff --git a/infra/charts/feast/files/img/prometheus-server.png b/infra/charts/feast/files/img/prometheus-server.png new file mode 100644 index 0000000000000000000000000000000000000000..efe31dc9e1384db6ff12eb59e63e20e197e6d3c1 GIT binary patch literal 84787 zcmce8bx_<}mu(Z05J3aMB?JxbZo!?P!6mr6TN2!pV8Pwp9fA|w-K}wV8hD5M&HOV} zQ}ey4dhb@HZqxniqx-DA*FI}^&?i|jq*u7FAP@+Wgt)K*1o8wG{PTbD9K3_apFRct z_taicLg@wg&*OzbAov@{K}6L-(b~wtSsih%A$;9y;Bhxz>J-c^I3``8~7#X;jIJg*@-+hvKN6o--XpwmU zfxLrA2!B*^N!(j-R>Js9)OCEQkR^ql^-|%f$TQ(RGAEf(T5`rR@)0E&=Il#sjrb9a zG3n3|VV@@b_V8C<+QZ)yY&RV=dZVg*{-h>Mb9iVU4y{!@Xkvc7$9%aMr~Dx}{hcqW z;EOc)Chi3YF?c!ePAc%?@rCi5e|^II5`0eVI~@1)pSQQ9pFDnn*nu4G@oiKt=G?xv z;E)itW)DsXiCB1HU%I))@oHzT5|uv^o_|@HZPR#MWMunc%Y&?m3FXMh2&B?v4qnGe18+i<7Vyx~Bi*bc zQ3V6tHImK+l>7twV(IO3Qe+&jPYh$|b!2UAk1b26NlASV78;n12k~WGAc5iGMy=nU zIUFt~{rG{p=(2_+Q=2d6etED_RI~Wyd}m^Mc9zLx_}z0PT;HN1+Jes|n~xtHE;ez+ z&Sgnqy?_7SN+`ryQ*No*bEeK7YFToreKcd7Vt?<^Pmt%{6{?oV<#-c5PU~CCJuZRL zF|pD*I2gXt9xx?RR>kCTWg{pk$U?m8bRq2P>&xJ{qXYp9(sgybW;&Ao%!W*WMyFY` zLdJ8h!I}OwCe5GDPCpdBD~N`MM$!59_O{36f-{Y3DMV3G5q5WXD9(NO_WJs|P`k-L zl0r^TS65(nvdCpC4S|uN_Qc%b9_Bh(vLv9Oz!V-Hj!3|TO+k^vMGOCqnK>je5kH>I zvFQ$h&KGr}c7rf*r%lKci9o4)dipmQtiG|avxfV7J3G6py^0~oZ&q{63ge+kUxZxj zHn1yf`*SrVr}gIPrKLZ-@2_(tU4Ap@JwYdzhWweFjCmX|78VvJjjDj2-U@-~?!t2P zXKlTeHb3@xqmlF$N_u+Z-l!>;oxI~jZs%~rzWa&7hzPmmDIKigDs!wwuUnbMrl#NQ zhwh;Nol3;7zkg%T&(BwW4?qa5|Ct3*Wn1?=Pc`Z|Nfbm{IzZR2qlo^L?F zlxY*|TU=bpk00Uao3B=7YCMlSu|7~zzGGpjBvf~DrlO`6DpJvL_SDwq%aKb-lS|=K zRaal#+w0Je;1v}XCXI`WgZTRSjWl~Q+~41;s;UmxSQCNenyE0hPKuWQKu-Q#PA+23 zfz20nb<@oA+OEbwKr$NaD3Y;~*K{}$fAj3Qv)Q0fR&FjO*rNhO4H)cnNV&SIs;YBo zDFwVs=ykQCJ`tOc(AOWwJX39nXJ}{$tT~}($qVoG>({C2Vg{3iYQFyd>6EXu8k`Kf zLUAxKFc>UnS)2~$B}7G^CnqO+T&)B=eg0g#rr2?Jk`a9L>({Rrh=>L^XWJ~MqnV=n zs&#f`{P$NRK|w(roSclNqolxcryE_FHEOIj3QC&49L(1}9s*#(E32z*KZt!kfK&Hp z_^7ueG$e$%tE-EYl=O?G<-kk@)aB$aMdj=c6x#hak^*igGgJ~f)XngvgCu_H*|TSb znsvdC)@NcOARquS9!y+0pVEP++N3$#9!n@KWf&bDb=@t>vzV_98q1N3jE#-(@qusm zM`|4yK>8ZTY&?=CgwWQG@*_H0q{8Tc=cLC5m(3zKkC}=pSh+yuXLNK}f`%fY`<{H! zxnSU}MiIQQxcCcGQ&UK0Ru-f8JuKls-`ZNo*x8syyHQoqTSm!0us*-#1m$L;9rJSn zulof--CDYWrlw|y~HB;Lx}A^aO$tz@g&dz5W^$^zz%cZ=nR-nD+MeUe_B~F>w`SN#sJd@2bIF1T0UcLGQ{OGcUJxcOSdOq|@wJ|4VP$tC*+Mc+_@^@%C+* zh1v@Me|r7DN2WZNd{HlB-8_*ylA5{KWT(7aB&DSdE)NzS?NC)sZDnVtHx!3iwZ)4U zSVU%KW~u9`5#RllzJ;Y_Syk1<8N(y;qY#DH)UX3AYP{JgASEOFnVjskaPu4y`)g7X zft8h2C?RirhFHX8E`pRUOn>5XKFDOG4f=X|-}3T4m{sA5$mhdC0aB(ZY)_ctFfwA<*qi6aJ^YA^>jWnlhs}aA zJ3CudgPoLvBe(Y$5uf9$(LlVOzJ8cw)_aaxf-pSxF0dK*celXsM6|V&jc)(^5hyGy zWMF4E?g(7btak_s4MiNdVY8go1)VlE@g{LQua0Kb?j}nSciiPn@}fq!!vpi2o|~K6 zwa0=ayQ~H!SlM$QA|IchZ>;}~6c!WHsliHDkd;LS7S}m76&tJLRW~?mLjvqF-k5J( zSW^=pysXNnW@E#2cX!8SF-cY1+}srPd5$D5BSXf?iGzoSCoLmWu;AStP5{jQcVQvT zK!7pmHEgH4rX~lBS9ITGvJl69vtJLOWaIT_0wOlUSBXgSkxJ7s+^O@n*F&b-vC+|G zCL=n)e0NZ@+@Jw)TE9IXU_7gaq(4pRh2Q=<}Eu%-o5ILu%hz{)u{o^yUMn+@d*N zu?zsV0-XlE3D;+1axamP$oTl02X|WHdO4BU>wlF*B_|ihG0WqDJs{`gjGLd=94N1> zWa;hgwQ+Fh0K0N_=5TU)TCnGSv8T4^wo3~v89s(itGC>+&uqd!z7jrFr^O3)cew=c zJ1r-t8_&8iZAN8^Fu$j#T~$M4ZEp`z$v1kfTh7GN{$za{Pu8hW1_wH1E|*$aUQS9u z5%B%{cib9YPENm;hx>pup#TQcQ7LsH^yp$fqR!6Fbtv?6oFa+u)q|sy5Vo^_58q$Pc>(mvB^1!U1S3d0sBa)Z86395OI6j`*g>msVDi zf*s7g8a_QeT}yIaBqky8>F<|b^nUOZ4aX1q`4f$hkdVc4HcF2FE~>3f@YSnV99}Sv zPsea6v9V%CM&w{97PFPx_2={b1GB(PJDWXkDe36CMn``E5Xn$b){F-60y-gKG0r9g z_<^^#7yDCD-e+5kOw7!Wt`H#WM8On;$*|AXwlWNt_ecQc;t+*`iRst(?@yc#7YDMW zVz;Ktgn+wKRTWoP$1N`YyvaX3GxK=NN3;5R1_mBCU%xwi=}2>9a}$R3ipt+9oRC+= zFaZgV?Z>ZQlD4*t06WqQ`!M7G=7LD_VML6`(}DUKHa7P1{=STb1r0D(aF||NTOV>e zdAoTo5PY$)2nEL}EiDbWzF$9nq_S>XJ>1`U-0s)J|MjdeQtme^q4)Rq2XKRii7Be2 zgyH4o1u*sGs2#=ppi*2;4jMw={Em(;+&INkC5IRh1WDj*i^$8PwYIjNZH@QN@^>h-t*w*C_`84Y~7hh0ZS zMIHU6NYOPn4+#iB=!>CUOR#Qk1xBM%q+aG~PkD25b9HkADRtasn=IB6Q=OItagWRT zPra+FE0e*Xsw!5Qcox0gNlV~D<5pL{bcGZ6fM~)fL4$U!Kd!>S*cb#WrIWWSYh#I9 zTP7+c+U+2OD+6{$PoE;2#Py`q{gTkb!-K`1&%(l@r>{@{e1~;!u?az6pXAl+*CtcN z1W33nx|avM92^{_hJ7k+ZBAdz&D%*L5W9zmOJ`5Y%gZ_3&L|feoJ)>}exR-{W-IKN z80p#CmQ5xGa$jpuzNl47 z#=L+}a8occ%G;Zp0VHm@um2jWldQBW5itr+>|2ZD`lA{3@>^pgk4mwo3^mhfIaMPZ%4r>_V^{ zw!?tN06;peI0ck~<;35yo|kV{B`5Q#sKh*deua*`98nR8-a@0Q`gvjjjez^<>A>E$ zbjo=&xj^6!tI6_TBMn4784(jqV8@Radg_VAo~M06IJCH$SO zoyB($O!@hVbWo6@;yC)w&+`m8(GALiFr>_WiZH3SS@BTM)5UGxL{>QE%fmMx<;g0f zvmDE>{9wd^>P5x!uJE_#X@!l60~*y6j*5Q65=qYjvmS~K{*HPV&{!|g1(q0l500&g zR@e=2f#`WRzt?SQxc3z=FV1=1OF0zhlamuhX6DX?q7Tr)?)exuO|2wblku(UC{JFh z=wO6;=gzSV9Erbw@p-+z|N8Z&5Ra{6t`8l>nO&twP0mSHwyAp8iIJJr@F?{gWpLY! zJtGDt<`*NQ;G7(UV!A;ZojQNm_a_h!&v^aRk`l#h=T%+|!TI{RdsA$$Bhtt!|?6^v|vcLT|tCnLG{mrrIfn!dBVvc(CQ7h=*{lc%pXaRUM-od|$#?uWCkcVQ` zX9Bm&w@~8+FdTh-eU(D(zI%8CPP;u0_e&z(iFikB9Gn&C)T%7I{0m!zaW#~VFj7+T zC%T5F&otxMkNZy}UAF6jDvvB>1;4ymqi7WH`UuB1ImyH9g}6}f&^9^gAKEW{v9~ZZ z)Iz&%ryid@PjBSmUlPGi-^}BOm69OvX%2FxV!|&-E{u#Lm)6KA8;qgMs-67)#vKPxHHgPaYXjoe6y8_orTDf6 zYZ<)`bp5Jc*Hrh<$++CFoY~m{d{3A2h7x(*__ejYFWTpNioj?zF0Ouc1=h1W&K{~( zc|}wmus=F$HFP19$mt$_jIJ}TzL=qbWhD0yRkG=kvwdzlJsgac?CGES2ZE0~6B(Hx zw!T8?rB+(nd%S2hIf+zOQL&*bOe!iWdbT%$B=!2Wot=G2Lxf~jdXV;1`yET z;NYM-n{V&|?+Xw!#{DI;+FBf`!3J}pd6xIyG&Xu5i{N)nPKs6(dM_=4ph(}y$Ujma zT{fAoebIBJww_~cDVx=)nhk{fOcmuE7`DHqY>y$8C+6Qu?+3sJYkVB_KaO#Icm zc(-C;XlOpMyOjObi`IjYiRnFvzO>vgO_6BlUjY*V!5<|J&A57UQc`89rC;95O4doZ0jot|=PZ6pn_Ry>C+0|xd z5DCxT5mSGTOioVAvpz$QK=a&TLHjAcwY`moKaLq;bnyE(%R3MR>Qkh1y6#Aah3D19 zQYS-t@yMLzd|zG7hDdb608Yos zQvF*H02>$@+f%MGl*O*@H0`fGNHv{xyVo|r|8To5pFD|Jbbsy6t#RQ;MEZf#51ck! zZjK}9_v=t>K40HV(tEA4;AtsaL;{{bzn1&l>p!^$Rz`y*{DT>pnZEQb$$hiuc-yG+ z;U&hNgD}^n;f~gChtP+%B`F~jyse>F<{Uj;8#4B)qw|#r9j{d9AwqmUyC^a#p`O2g zSF)fKOzt;1@x&pHcx+`>LY>(AT>~1w?!2fHirI; zK!#RCEKZpnmJ<71T@!A6f*L>f2p7B(WnBQ`t#?jwUM0h@E^ED zE>Hi?2Uy=qE(x&CBpvU>fTSeLqTV7}dN~vjb4Z9wzrRqx2d9aB-_HaKL)@mJS~D=r z@HEDfl#F6*_uD~j1M9?Wv1KYhmOeeO+s=FV007GvRq6wyKi+DFHVH>|Av^4j2H8XD zOBo%g<;5aacE=K{;VgI7r}y?P7+G02=C4nu7RYbLO`Y~UN9Ku)MgjmcB=c+c&;_NLD7kZnIe5Pk+M~mz!AhO9^iW|IzFCW;7jxH|NKM9lK zu~`%I5&tYs;$GQXcR5Y^HfB78=Hz4ZX=r<+S(6G?TYI{f9Y@HPC^$lp5 z;T&>!N?^7OH9wcri)Ov+B(A3w4}U)1t5*8jL-n`j8YqyFnP^C@O6C^fM_R0{>z%fR zKYk=$K|nDrqNb)cK2sc0P;%IK<+-y$>L}r52+1{R+kccr6vzZT{*9kvbva|cJ zB}~HSdE;3x9)2)S;fKp;iJ{Z=wFoAcVF8swb*ZaXq3LPW)8(&T!os3u!0$ogS&F9@ z!8R!_T?>qb=$e_eEqULIP7M;2l~t3IFh`Py5&S;sB~?(QXZBhRiC>H&+I;A}T=M$+ zdWf&RnD^GTsjxti{r~HI01~a3nfO$&{J~yGv)wkf3p8&&vcT)BZE*KQs+33UPz)o}B zpUz|qPB$s;!QmRqph1Jbd2) zJcFSieN`_qTJRshz{2`mne~m)5+1g@H|QTV-e)mW?GwH~_qeTKCZ^ll9i0?5JL5P5 zumBj$LsCjge%x%}8jyl@V4e7!POyA?Lt&dw8cI}e0iVRcd~4biz&bMB`YkF-NybG& zTs*U~vakys4Q`-$>Oy!vJQ*Qna}l6y^S0Ce_a_okQWOrm$QUBTF&x2Cl3TNNBw`T+ z6!axGf9t?eM8isfDEZcB)6rF$Zgg*A#F>1sO5u8q8o|fpOSn+1c@DBpYtxx8FD{%w z0P@*>Q+Zoqq0ulw#ig{oJX0zMIg!Ivl9~qsAZ9=a{&u9;@?E$LuKE3+CxvxJ7Z)@3 z&=<(B4EFuQ)V&3$8`=|CoR+uB)<9nY8j7DMt>+8h5<|-&C8UL_}bRXXxuQF)`yzw>;8H*xB5nx)5V085s;VN7R4k#s+_TxzRx6 z-E8XlSuyPiRk2-IDB_0?c~?Kwn~ZoHoDK^{Pwlxmx!79!`=bC_#}^b#PfK07xUa14 zDOuL!3uTA`-(pM5reglfY%hz0hc{l&$}0Q1-PJMH$?=z>!V5FJKS#tU_I#U9VprEV z?JqP57T>Gd7AsRlPmav~jWR%3Z%JMA2SdOMz9yR(xFl9_4ZZ4(ot zq8WsXtK#r*bETw-2Ec^*c-vNa2jFgIU5#;-W!+?_nHv&SNv^cCG}X@S;j4}kKl(5N z{K{%pX69#91!1PbnIFuZE7qC1k(U?ZVxs#%wd)b@ z{u%Eb$jU+fEC)PS(a6pV3L+Q4&}2sTF3McTV6&d_*XhH2`n-}j*AfD<1$#4C#c>)o z)F<~1ykj6q6IEcP!DqKuyD5vVp@H}P34G;6gVX)jpFfGUxE#klT-G-?O_73>si>%m z%L_9zpX#qsYb_aO%@U_E(I`Lf(@;LvD@kwE9N4I2q07m7ToWTBBgCz_gaa+^3IkK| zs@qvrjj=3yFYHN>-Dntw&q z_>7~Tujoo9ueLPY>QmfMwBfC_UQbC&YuI)giNn4radYAIHJP`51ndF;vWCmOwNo`6 zVAgr|b@favEFFOLX=t`oKF!@;l{l{;3I~q27WSQ|9L3e$(0{s$IN`>0GKN`U&}{U<5m@ zf)mLNvl&j#{24 z2r=pDUpyL{jxJ8tkA7yT?{a)wl-fQ)c6qzK>$l;gHE z2oM59J32ef2}&~bI|Ftmy7fUq8gPm?Zz8`2>+xye;eiOrQ>r^H1RSt zqb2>oLm)v~{0ik2Y|4vSk-2}MKQJ(K6xTlw5hc3Yb>CfeM$yE`NOHp}Y;V7E^EVG{ z8S+a}k<*6ZI|#c0L+@A|j1djn+l5{2N`X`%APUo}?CYO8qKTEk>Ex=m?gpV}zVD%= zm5dha(DcaISo=az-lnl=;M_*>D^%m5?o`TGMH*FwPES-*z0DzXnuRmc`w5)R#j8}D z41ZcAzYb4LK|$0FQkuDSH4iG?BST62av+c~#aq@P699aP+2QCNGGVa6;k^w+vQLt= ziRrWXruO+D@RjvPL**MF(O?Ih1XQ1{)Z$ull1?Ysdv~&CTw%+A<3F_k2k|E-XQb5J z4U#U9o}PYEUJ@S=F$fdzW>?Yi^QUYn=Db8U3jw+1yu8@5s!#GfC(1C z)WOZ4w7T2#Nu(l^&4cVXEW486KpyE51h-*8w?Vg=AK~=@22%px!_2%}e_I8jh4dc!3J42+M@J7Nd#<>+x*csDrK#@{;!dahfq1{q2p^xRy$9@+ zlp<%iL~e+xMk0)W^%qD&b>8N4qDB)L7;PF=Lq zsnL^OR8fmPGEJ?-ptfV~rjza%9KTlE(lvsoP&B4G{9N{)MFKZ}(tpe?%qr=U45{ zRO{zVX8twbpTPsk=~p1x@!nKd>MI?Ac5!2+(kaH&?!oD$mcS1~F&dW4wc1+|nE zo8=6+_XGD(xCU;5xf&8$ULOFn zxfS)jeSOzQ72EyScH~R_$%6;Hw}DL<0f0L0`}hQg$m!k>wU}1RNk}|fjyu-h8t3~> z-yBz<=n3Mb)i;Q7jtz9{e>0)X#xN$JMGQOY)$UR?|&}V76ar3#&L<~ zbg<|H(&I_oj>|UV06$xpjP>28Wy1*~j5ha*Gc;Bx1&*5{rC6b6yTS27&9o^}PPJRIrCBVJB&&W>^czrZS6p~LvP4_|XeD++u zZ!np=9SM2?zB!Hb{5;m`%DWpix#L|966K=Ch$2h{(cjxUviaS^18bl+AR(V z49!1j;sWv3;*rY1PV*%x2}$ISw*}@);>2bv#jjC=B>slFZgM8$u=^Klajk-+3LrH! zch1jRXf++i=B^yxx?a6lOLk4p=?){OWcZ4_Klk&OTO1k4Ivje_P40yAJH3wwIS*`> zQliJ0ac@1B6}^QV^ZuM2bb8XEPqH;qMD6BA1vuE1-5`&kO5r3AJ4AN_WFfI}A5Zw1 zTPk;6?AX=y)Y43uISR2PR zFR>B6Cuj8eRenP4x3kSy{uxi3a51jjjx|=H!ep{zGX{?u+`a@h2o<@%yTFey9G#!Q z=SvWG@Dh(+=mkv~eFd^Q-Y*?9or5+3dN_thcDig+Gq1ICMY<-<{7eTYx`q z&%c9&9-#kO9;E%0S6p~%Q`tEzkb>g9)OK?;hK;+W_(@42D>TPApp^ecA$kLOa?|40 zOwcmnxj;=Ul-c>Dz4F*Z!B8szs~LJ(L)h9~H9|XKOR=zZl?^LJCYloEIn{00J@6*} z;B#DYZ`eJBZf(r$2RhF&BX~DRo(ad0F9#XCP=Q1hOl<8cJDBxrzUgYw>t-09S}TAO zW$O23PsA{`8CjsElOs1%?D7c4PFdzDVoig_uosk{TY(f2Q6d+8an- zD>Hy%9lH#%@i3RJ_PT8xm|$9kzayqa&=-Oiy)bJ(MR6Y9Yav_3SA<%pW>$ux@v0xs zry_4{=T+`-coF;IggD#K5fj9Usx;mSqdaL!%C8iW$MDi582n`^_Bffpi}$cnu0$dJ z=8in6ywHSW`r8)oj}@L!34Vr$D!r>ZZq1tTiepph4)B8Wap_#|O)M!7b)E@csZ>W2Y0l8!kY=jI61CeobpV0+(4b3bp?Lvk{^z@ za`so6?@^jhx%mqhBa$UMD|2o~_^0tiD>hmRWPVBGdT4v^ax1U07i_eDAE|C=48>wR?%O~- z*y}r}8#*2x4sSf$HygEom0zjvd1b1y$JaWFlA^nRW;=o0qy#l`)RY`(>+VdBP6;!z zXc1VYH?{{`^EIlUM^>x8AO3zIpVs|yB!tH06fMS^D;_>)b=u)5)M>$0#Nz_W<`u`3 z*kqrQ6Ow|bL%ZPjGYxL3h3TmjvwpQFD;$ZCXY#EZSuyhD-j&gN{zfj}mTKWLGr7v_ z!rDHy!fOx2wVRqO<%%u~J_nTwtFhJ!M2n?HQK--hAj(OTeIhJmkM5Pgh0jgROlQ22 zq0kzj@6MPTNpHAemQ6eBwcrUYnim_kKdrnUcsY|%H4sQx1b1X>`S`Urqjoms6$YIY z873>60#4d!8)mFsqR+urnqRo<*&tZZ$k< zTFl0QOP0y^K!vy9a`D&Gx-+rR|CU!qOpmm$lwq$J$(6jd;NQiDIjtH$S|r0fP#=ac zIGhp2osATlPkZY}Oi*fV5^kR1Uuk|I4covkIv4f70i>zSp<7tdd1}Nw+?2n?>TA+L{%!nv_a#40w|j=WO2bm_;yure zjzF$~sgogWu}6SNQD11l+>`fhaJp`4$xt~#w`v^)?$~qxN;~UN^`g|~{F)3qtLI-c zipXLss(e1$iP9Obj%^*aUB(+4nz1GiIw{Y_{nQBb6k;AQr{M0xOOEI@nlD5~CxNGt z!84iUolfJlk3HPzBqo>vreai^&4n}|!}7q+GMuAXG^uX=#iGaB8qxSZ-lnTx)!#1& zEi%B-$Yd%;IecJn6j2D?bAU-qL+TfE$=T+#h7RY2SL#(>ZSMusF2OX*La8;Ft zjI3By`DqBHV!`!6G;vf^3>vO&IJ@_(z|^Fb`UG1UBh=1`FUxQv9`)ysn6RXc=u4T- zXaWmbnU@!~Msq8t7C{`{p)c>u7ev8kUuLVL=qd4Gn{^hWcjyd~6Bn)Oh+4tDG@a?l zGAc8&?o5Qg?3(fpRMOUC{EJhke4jVLp3MGJ@l$9}8Qdm$S5!>&>BguQMr@3tiq{5} z$4Cq%NCVfT{x;+eXQ~Xjh`s~|odYMKch-|azTj;-eKrFuLqs@9s!(C6w3t?{dJP+x z03{`r-eRw3yrYZ-&8LF5D*Hj%skKpupZBLKbz-?7p@>LZ6O%r1Qf*sflTrnr1wPMQ zi%7%1aLPm9Ffy^Erq#VfLPTOjVtuRQys!`QUk31=HH?XMcvn@FS*tLxX{btd0H`LqG82B2OJ9c5)46ebh9A$S46sIFS;1#p^7We(3%SM`b ztm{o9O)k68uuQ{^v5v`K^vT|+%;+TZxEgD7m1sUo8a?qt-6#8*A3g4;oZK33(7{S! z&-A7IoSN5mui~@hy*60E?7Bp$&td}#m0I?^D2-Z?FXro&eI&Y0Dg2cHU>qGBi;;M;R*0BAA>;lL^{AVLk5jpps`S>rU-M0^n$MOnTG z#J*Sj%Er!mf%S|=T?+ABs;BqAcMjF-@IX)^de(`t*~D`t%Ps=2oWKr5^p9c3)q{ic z#n0D|=vDlmMF;=OfKg>lOs(K8C1umK(__oE4-*^$7*T>s-+9i}!QZI=(%S^8|M(9D z&&17cV~x&_6~xERAC!yWpZ>Y!&4LH6#`nsb^yNQ=5?uewvHa&$Zd+cV2EUJ3!UQAQ z5o|&TtgLj%ZagLSAtD&4h}15qaYK*%Rugo7tb*z zxLY|_Is7K{Q#gf7(fa%)c~-yl)*UXE0R(da%kRi}SaE^qAoa^ZTQKYH zOrs&U#NS6f-SB7)CwdC5PemjN1SCLVSA9f}S7<;Fm`pf;&Ar12Q@<)_F?icTub!TF zIUzALmSgOkGz-cQ1OuseSOrLPGEe5(S^#}sPGZ;YoUJDo5$S9^n@vz>gxU}-zd1U|Eti&cg;aKFfSo1u`xs@1ynk1g+qK#m-kWlvQkTb zbQ9qO0}4$&RR~?rV}{+2@jwJsiVUywt2{2xr7a4JfS_g~Z~veDPuC7$0IBn~{&5T; zArj5RB#0_Z?nM?^mh|lwuSFr=ljZ%5zxI*Z`j?A$XIqng9hnA%I9#gc(9($yn0z%EWs?9R^4Ojgzg z4nywAHw9@x78zVQ7`0uEIG+qZuIB9|0SYQ-ook0?sNpb)oQfKBxQFm-NSSSa!o%wk z3ORkC;!*r;+~42-m3qj@#dxOx)@1Rl;bPBtH#`Ys(Z+xhfSi{dPG3LchWp(=w*CjS z2o!HD_OF-!b`#~54|a@4m6tQ|#j~MHNl7)wv4$BP$bj^*?stQvt8bVr^_~yTCm?5D92%X{y$` zJf`|r?9{mc4RDfV8F~o{;I@ z^4H|#VA{F17yQ_80=5#2)og6jb8{yPFdZCLyQe^DgdZIG{@e~XiTf$Mu93<}LYaZ- z3xYX>waM`fS;A^Sl5OS29=0+f`11eylWqJ+bl*7dq}e(s9<6IT@Mc5{X97B!t8cXVr2X=+-lLB4V!O=Z^_&Pu z261q+rD>(|AHS3UlDg;t4d+j)NpS&>nvKuKAKUb{c2${BY|c$|xEY{5BxAP~6$Gu|nu1#S7(ij8v#fU=2<9142| zK$rp_#QvLk7f!;x-=cOs@Bs}SKS;X$d!=pv{py&B7pW|5lg}%fUDF;rSIy!pS`M|} zx!gb2PBicdfW+$Y{Oic0Q30zWN*Q07S-;;ccggJ>Yc_E0rBzj>S}0Fk!925!)Ra0@m&PJyn=#y$@!ucxLh;5 z?o~!4;_}T?)b5;_$&>AE!p7@s>H9qI{CU`$_BtUm(;(?$rkXcY2weE?+qWe113uNg zEX%%Zq(HCpd^rsP;b>wn?p)g&h_Rcd)QHL}D&T&f+)%6jPFPx*eB6$y`3YWB6h5BN z7R(2vKfTK4CYuZ2?^X~Xsx5jX@PxRba(!$Lo~i6hgSW{iC)7GeSt4Y3&`NSs< z*1HQ?-`qI94ob&6|HFPU=D9Wk#X2p>4Cj|vT&>yV9Q^lmcPr#yR$yXjc zi7sGdVOVazTuO5;Nd`g2;~YD>yP5T8jXhwk?)V%Ifwn`^K=fkNPE#W76;zWg3#WS? z?CpBF&;Eg)9=iQnlMWw^PqaeCGPwRLb<4{)`=+FL^j3g^pvDFdl!x$(!TXwLL* z;e#D5XnyyZI|fI)4A?wmEXPwYC!7FiOT@SQt3WkRy=fB(5nk)*knEqH;UB|D=G${Q z?BFtu=c2=;+F~Y~vm0FVh2q{~{0zt_cE8v5a(tyxZM9bQnf!Eah@Z*nEE0xS{yoK8 zE46R?tzck%*7&c^ijk@XWs1Y*hqTp$je^GOLHFQv%3$>(6)xw)yju5bD*Kk(kMbBI zbvDQ8qv?*Rqx}tjZ&H4wQU<4cT%Y3`_8f^t0|a21dwljlySW!g8GvA0>DXjZ!S?b* z87S&?aZysl+(ocOPfJ{hj(mZ|p{_1_4( znzh;gGYY5eG-;Ouk)rdy&xHz)Ps1aifTCtfIT;sO(+#3bAX+uAzBZV3=p1S(`V}9a zuZD?k|M9VRwK_MMu(_Hq8)lsgeV>82R1rf&o-M%YpFWG`s?DoEeE1+=f4NuCcz1yC zIH0sZ@r@Nd5`eSRj6A4jAQq);)@F)>uUx5%H8@tL*dqw=Bj?Cf$l$5)Y(TQNoIh7sOI z>6@?*lIKv>7sOUM4WN#T3*eEsbm5lQ!DQ8Ipf3j^-m!8&dZ2*w=g@XHWTL*mL#WA? zQLFJH;eW8(Fe^$;g}}qZ&o9hFw%CvKKtIV+Gc~RIrq!`632wg0HQw~9lmr)+@`(Cc}f6HBowXY{yXW7t)Vm_v$f-{n)XJOn!)6gS60r9A!;aulvPxel!dnaaH%qQA|R3Va^c~mxc!4z=~e6}10J`h zb(;gAZZL#g#$ZbVNw5d&e1Z(s5@X4$$2Y)Fu-Mf;%}zT<)f_icgMOH6 zpDPJjStSo&P{^gMw)zCjvz{eFhx~d0iTatM^EHN6qj}*<`vGvg;n8yc2U4BJ!xSDF zVtNV`vMe^$w^)i9g>J~z7IWcme~!FJW<@Wi^Bmb6KxqEcpFUVkMoRigO^vv+XvdC#^evW$7`nGWs`qOeA)8 zi+)tA+iXJ8;yM*R$l;^apjuwMyJf95Hio8}j%9;7?IQUYr(0wge0J+ZC(iC0l{7^$ zR&jChKgQ|&pj3=c-NG03eR0Xb8_~JyrF5-^d}~mTNe@N5}JD~Q|qz|NAP%LaO&#nDjTI^D>v6cxR~?1j%(tkLdC855=Ci5K+wR0itydxDX6o}R(GOLrICciSXn&^(!a0!1o-tvq7XjUN9=l%fl_Vx zg+@Sq)-u<5|A{BB*4FN}pimQaTN}wyw~luZhECCb{EHt+Nj~IEOqvZhZuQeZ^7kh4 z7o1YjPNn%ACSit{-KoRcrp!7N%Vx=+jI3aDcfq=QAb}PA2FVXp<7w8}6%D$m*I4-t zCb^0?c#2UH;+UHH`qH7hK3fGbJ402eAowu?P^>;OFpp?&ga8L&O5yni)PA5$`iO&B zF*l{Ma!wTuXF#j>S5(xSTf7!fFW4KJSH^g-?LAgD^ zJ?Z!@c$@Pwxmtf_meCdJdGj#n{ggFA}_P0!U1V+113Leo2AU~A( zQECNWtl1R2tj3JXQxfjEmzFWmtx8WMeReJyaIyc>?hN};TA4*00ta!cub-~re+kCY zHs6ove0U#b*!Q{d`V%!A_#F}XD!WqKQ_M!U^H(vv_3d5t6xI9F&y4lhKBqSrX7-zgdfkE9b8L36Cr@A$X^y)p)3vVDOiY9;u_F8vT;5z< z9^fG9iWsnn)_X_H&wF$fhl+tJ9`&XM1(5zsi5rJqpF-IUx;qi#-+lV+Cow+3O=@8wJ`PTV4fQ@-d{0lRGra~4l3M@WDp`rX7|ijS z3CQh0y(xDslFi$Cie;hk1}8taDLQdUK3|!H>Sa*b`J2V|iRj(3HtTasJ}$?dK%W3) z{jtp=a5fNxo};w2Jv)6zs}((^?0tH;G~{PqllA?HMUnezTc?z2jUGBE7(;&h){sV{ z9DCNIx#!dkjKho94g2W&Z5)!R{Tj6jbi1hkm_NSYi`6VT_^)LFju{r*{g`^f_ zgr%pyw^(RK*QnY5v(WO;mHU%qIA56>{M-V1djRrd-6o)@KYvGsLhI_bmWG}^`Gz(+ zcBr>oO~7H#_7>Ff4Nr`;u64&<4O+e{Pv&i}Cg5hY*wRfCBb}IN5CoVLfcy}QgvY)D z>y+BE(hDygMy0GYAs*#OU^Vx*+B6d9btX)xvFiS|F7!!WUUYgC)nO<1Ib_Rp0v`NE z$D^FT+G;MmVS5{!(F5D)x$K%s>TH!|Nc)Em;+%Gfle|T>%BA&{)=S7(^mpIj5s+8s ztd(^~2JmH(FKQo-h}o+fV0bH(dHKt zt7~QSWXg+9IVkh?@@g}|8ucxD0O10yu7SzB6Hbt#e(-&2zRYFv`Y?eq6#Wi9)5X#-FsRX{+x6zNi=l@O5b?vC@k-{0Q%+%e8M`|fk^K4aX!5Lxefd(P+i z)b#WYv9xBG5>Gj@nkJ{0aUwJ)!1KurlZhgx<2Q+%+xJ*XmT;i?tmi%AiN|n+>>w4# zpn>A$WnY3D8b!{ZW2U{yAkD(<`uN(iRj;d z28BUVOsx}PP*||w?ni|SLB|myhq&yrvaF-joHqu#aleLeCJNctTddTAf>Ncip~zt? zzH4YLOUuwO+RFb%w}`NR+$X%#Q^^yRP{mQ5GIL%hQq@xp#ywW4I`a3eJ|Dse**;_{ zrOE-6cm9fUOnPZ4O->GTT6Xrn?osNzES{jl2FRk1@=koLvwUR(rH^(o;H)BiN9#w% zJmk9&`(4@TxasWaMlY+81d06j+QjVyvNV~l;u0(rJjH~>qzB6*kR@@r;GH|6JMy(b=UY8ROPnO{y#(zB_lnb;sf#%J<+B`FKO zfdaLT&XBRgt;>eBJ{_-Pb~bOUgSv5P28Y)`foI=}5_LiPghi@V%`YX6h$ON@kXL}v z-Ti>Q^}Lj;40dJ*3WX+YAHVK?73PnSPKF_54v*}pYHa#Oz+(Q`PcWp9$i`%Nm{Lyy z|EZHpb9!U#xiYRh$%iAA?$pU5r4FJyOovsZA@TWa9^%{08)K)TV_olV8V&e&_tP17 z$4*o_?&a%vGoq6a7vTEFPGGdZ;9cH43P!q^$Vg?hbU*RXA@<;|OH+Yz^tlzT)r~jj zrRZR1aMwSpOD3G||FakFm2w@&V%Y_h$Iu@IeIq}?l&QEBA9B=F5H33PcelLG9SB-% zZE9}(-Egj_h`VTg&PZKP?>TZlWfuIX@tYcyv+eq*Uv}~1OY|m9-vuk(x8l_iXy*oN zme0tM$;2eZH$L9;+;;NceHGr{1CZ-MAS61VgwSTyLlt3?}5@ItW1_Z-aBHwgO3 zAjm<%O_7F{=lIR=^=lHe-0bf+#Du7mC35^9YE4dm`nA5fJF|E<vBL|9#0;R zXWF+6Z1}BV`jbBjb^iQ$NUESg!3e98{pr(lrlScXO#zb(*{@Cuk=bE0PtSbB8aOfv zpY<`I>Q<|d#>IWG!F3i8wr;kL7CM(y{v@!#$WL9;bN7Y`grLp^Mww*THO! ze>diPIF5@g*=XBITMi=V1c!bRo0(-8n7>)RPAP6w_9DH)lGASdVUc4`1r`IX*^pEz>h-Gt9Z1crO2=Y0tByYfqL ztu;7NT8Kij+^zvKD+IXRQU<;Gb@qiIVdd$3oy@kY^9@TKv5=~m03fwspF#dLESY$n z(=S>>m{{j(?iCiThJV2MA@97)#q1|)M6CCgfxSxssxqHP`$9lj6bVmTKrMy5^^-8- zAtg|A|K~WT@wBJ1;kTcz%teAVt-<10OgJvCK2Cw48wv%e_yk566RfrWB zQuy0(?u&$!v&__|EnSy;WMB@&7zjI~)snlAZO|B=lTmL`jF!M~r4?~*cD8Y(py-pP zTga~V`-n`NSC%DWNnKelm98XQQhC!VX0N5I`-6wac2LaTD1B4fb5AHmA~m`8X`En7 zBK$Se!m`1z#_LDlfkTgJrj2ELinM2ot%_Q0>sQOgLhta~?5!29M(oJJfI=S*^QU(H zHTZ^`ip@`*pQXrJFbo|&>6xTR_r}S@yQc4{9JdrUc^JiroN4Yr-b^|3Etw{_MXz)R z+tII4R+dX8&!3-*JL;!P6ztldt{i~s82RXplDTKYFkmMq9r-@<@=+CUgKc*A#ungG z0nCtCr0bgQ(8uNDjEx*gTipAR^6l*$%r}QEtrj#!mhbqDuFlZYIjfyxgKC##!Ir_d zWlL`|qp>P{+v0L6qnTyvZMnhc((mG3=^hqt@B`PJ7J_;rb4xG-cIB8kyEqJ2(6OuV z;`y*4r1UnQrAEoAajQtVAJV8Mi3UfDvPKH7QnkMN;WJoeBPrp$90sjUk@e27*^G99 z;SC5Y`F1ZEg(CAd5Hy`eGu@I2(^%ygmoXK)c!^O)1X3<4p(^&+E2bWj!5luw3y*1T zj3kljXbW&OQcizHXTf#r&Yfoos;D&0tY{=WXiq|9M=BBzyVSbx9AF|4+ZLK*P2e=U z0TsPsd!aWK8j3&#R?qwbXjs85GX*fvcdt#w4-%8zy}i??e~Zof7GK8C z4%U{Qo{swva%FVmkNtKooRcBs;k2~*%0O?2G=I8x^uW%wGG|;qUFGl;o_Gq zC~AYvVpd0_z{ptFRxIf{ie@B^h1S{!2cPXKC)q&$Se>gKj^MuJ^&(Ta>eq!2lP-@X zq@L~So%!8Ruzn>A8Aoz`&U5a>J`GXV2i(r9V}g3sxPOj`>DT1M)8ws62n|7 z&|*-*;-KKNC}=7$-zFdfP8*lt1^EQQ-kR*7p`niq5NhckZFJr*LnI9rD)G%*`~r)t&k-~!lFX8EZl;1E)`^jJ_Jt*oMH^zJ#p#ib}{izGadg~`YfBsa?inbnE z^2a&E^ey6&%#V@lcBQ1(=KdrYewpwx^u>L~yyp?0G&rE!&|U<*P;9dot~mD(Qh4wdB|YU%3iW4n}#3eciW{%jIH8T&$vLOLz! z*A(N|iS+c|Ryr#0cb-VPt~LDWBW;=eo1T$JlDaWf=kcDNnhS{ z--&tk>f(AQwJawm_G5sIATF)@(aN7H*6`a8uY2?9mNO~To?*%Li7!vjqts%0+%z5p zLCRH8s~q8h>(Qdt1%M7F<|MI%EV}AP4rEQCG5g;6;|2f`D7lAaI4F3;iMmqE8hzV@ z(ZrF3wnz8!qZ$I_Ev>9FbcJ*QrK$U5h3SMJ=94q zB-1|o=EtdOZFJu^A+>jawuEzIc6%p41#R=@pVP}-YcY*(&?`~7$+z;PMvFJ#g9 zqX%nYVSz9vGQe4lI}uGRGq@_IMt+f5EG!NqH6iG6@vX2`K#()z)0xGGxLt2I>Q5LZ z_0OO8l6`VnUcOOXU7ZkGYrBsh*?6R=P97b~YiZE}P`1;5@IW!VfDLAL1~0E1Ql|0e z-DW>0MALjfCzvUQo3!K^^BkB_qH&<=^Mhzq6F(c-{OIChDi5cMx*wWZl*Ml8-KV_M z`e~(kBrv1TW5yW&W7vdt);vuA%ej((x%wYP1@# zyvSj|b2b!R$j%H|(HCWZY=zG3dPfSx%>2Tl@vlQ%0R3k;+Vcy))Zb9o(rSoKuA-!_ zqH4R^hQ97X!g+6 zbjqg_7;eYGU4K{mlroykJE(kb(To@_Sq5hE0tW|9FJZ4K{xqUI>aE@_Nlp5hSb?>Z@8m!>wIp z@{9KFZqojb7okCAOHhR{oVsob?=FuV-bOzCRUMm+^Z#n#n`<3bG3Y6Az^4pzN;=om z6$B(Nua`7YQ`5yHR_;qI!9~M6aR7Ta~Q?pv+|KC)s=Duf(st+c8^5e;z#g z=0+rxsTS<-)1mA^#5S$@UIlHMbNXlO>*cHErys79_d}!(V}a4YNd+;CSH!Z<_leHa z(q(WfvE3 zs)WkPq~o7H^xR}sm?L7d|If7d{O&3$0~AC{q9~N9c7A|=(Sfq<8T3hsA{H!r5+;A~ zp#%2$Epti?Wh66nmT?}d#|dOJFJaiBUAk^}+G|45Ybkt)YdQsZn79WC9;yg^Z5G3irnHJ=e?O)E=+W`V#tR>M zVo!Uqe$VN;h@$G6{}7p^v!zy_CGZ>h?8^K|=xeQ^I~?3QgdNJ?df%U|ec_$K-$F3! zq3UW*Zp;6CPe9Hb@>0duasge}R(_p5!xX@(`W4Whgu`4=xG;l)&Kv#Ksd2u@5P-YT z+$|XTZ`3=QSp6l|`x-KDXBLq*{sl5kP$&{t)?i3sL7`|vgX9Og74!DVvzTu9R@6iD zoR*a=Z8|v_oS)7jyTLeuPac?`P6UlWq1Z*50;Ag5wpNdW3vR1kkvp6I(0$V$mlC34 zp(jq6;*#t7j*H(in@Eceg2Yx1afy*PuG{?Ep!O@ulKONYk4-g(>=!6Ve&wfcqbCJNId#lvU*y4ZGT^PAfSHVk!*ZRHh%mN~ zL=(=BL_nXcD%R2htqCBDfK7`195L-_`wAmHa|ApbZtvcN-(&8pyUOjb`P<6P{pgu9 zr`nt76-k#HFt2IFORF1d_``1|o4@5ye$Z*X6}3Hwvh2&b1nlk2=#%w=D~SRhI{R%o62(9+Ke?mFC7+QWU){%DVkUfo~zt-RJuET&Z%NP zMAvcF#JNE)Zit+exRqZezP_iPrKl$kEV#|^s*1gPruYF(%YR7um-lOZla3!pD4E>N zn_rYzGg{D84EWv#%okUNqf%$PnmY|V3w;_nRj zgOhzHs3m%<#Xm4~;<^Jidpv(|P!Mj&mzV~~!^J|yuv(aYf9eXd1&+rX^0)+e2sT5Q zD2ju63X)1d(1472Y{4DwLeA(wm$1y zp@sU1fyiL9zKI6B-*_leSAOaZgBW#;kWi7^gj!gX9u}~3Zjp@qzty~D`j2kWI5XEO z+&;Q6@J+lWnm$T(iMq)yDc1K=zZ@h#lKy?jQ!>|%0k|MZkULSZMF*h5B@^3L=$>im znSY`&%fKJr)iu(2#rj$svQLF<=$;V}wW5~@hMrsOti&LMb!+1xawzg+9TsTsQQ z`vBPn`tIJfzZOE~cqLIu<%RGlL7&~I6!{o-6=9ToL2INlALq~0Jq`I`AKl_t?vY#O zkdxfr&>Ir9wR`^R7@&aAssA}3rUUc?`Kw0==Q1Rl|E-}g5j%^avL1-+{m`JNdbfJjN0Gw<@c!b3vHv9u4-jymuYU^EY=@8a^ ziuDWo&FAX(w*()=c)n^C%l86e=&oj~pJ0n(w1$y!%Qvr}n5K zp~tDTm2_D#mvmV*j}(l5Hhmd6!J;Xy%1w|UnA;A7;&r5uJyw?fcP+qAbgqQ#HlBGN zXrOan!skm?R_dNPZY7?*T4>L!I?v6=H`}%*>sBXRx}&T3faJOl+zEqPhrnD%PzJ)O zIj&`0`DIbaBE6ymfz;Yu*CgVRyYpmGyk#(HYPa*7nT173uxw}{0HN$%KU&>bt36L2 zBu&c(lQIgv@Z@dV_(hjk1fw&c8`ER?8U=E{XQ48`CX^em@2RId`QulX&1+lOW z1)7>&jXD?PaA=o5b$!`36U$)lZMVn*dBh1>~b5 zzrMcLR>$>6?!bqm=a-0*U4eJ%?)^=}K#a1JO5i&o;?8A>jfl!j?(-PoW78WH^{Y8y1%+11*iUy+vn+35XWAUlAiH#F>% zCi|>U+`4BcwG(xNrf=`XCc{6q7XRiP;LKN+dr>T;tF<{xzli-M2$KL9SH8>&Tyhh4vgMO@iCplO-mc zt*(6Y@`_7HAm5{G+&JlG8+V@dKbw^`y!|QX41MsQQIqfypyaP}VAs9OQrj4>S6G$s z?A)4@pMl!)dl$bP0Q^T_^N5`8ifuF(! zL5{GvtbE#0F%HSw>7K-PBj1Qx{Nn9b?E!|Vl;Ywdkn8j2a;`8skMVj!^Xj$DXjS(| zA#!AZqz{Z1t!?pHJIoS%{P#;%`7zzb_rS->7Jf0U-|y?hr;boq*~3* zzXuBrrCvjr^{QkmCCV@%escNG1K`qo?%%)?7w@uJPJQ{3qhCj_>_N7}sTNmeAF}*` zumaAPR$MHfeXmDl;{~z&Emcy(EZ^v01N)kjaQS=r;=&>#2zisf0vZ~{y5Dk64>TJ=V{=Q!O($FG zVBAEY#h6Fn(0G`$1^dItzac=AcMtpuu1Vn&{$0h+BZcjIypwQi2BTx zSyxbaIO@+IaZ7d%4&hON<~e{vMqLoNrnq=OEYHnRn<`Em<1p%WDn-Z1*&^mLi3=%E zm7AgT^e`H}mguV5a~8VZ8a{UKTs2oT^I6o9E0m?%^H!nMwaK_BOY3Zor$aOdoek@1 zu+m`-H}IQ z4XvpA#_D}`*jgy*O&IFoKBa?ZL7_Hty#KTQzQZZgx?A$NXqs$1kAAdvb2Eq$TF&mJ zG(ZU{;BAfC!)*^mG)~CYw=B^h#b6V-*7Uhl7_i!mR28G|ZEXN9m3N|1X)rfDs(ebx zC~B(&qGWmR<6r1z6BE0T%GbF-ak0E(I=WDshUyV?5D=t=xF(|i7&!tNlu6g<33_S) zud4{NzBt3pLrNS9pg$b0K>aZB73Jv3&>oCnKSub@bCp8&YYetD=LJpo1>!I}ENkH( zrXRaSZiTeY$GPnb;0SV0K`Q-uvr?M@TE|&_Ilr@u*_XMen$GS`ZcN=3t8pz*G>~aI z`u;yr?ilPX>e?cjnzkN{`hBei0P&kgQ)7LTLq@&bA>%#9iPhrosPDP=L;2F4Z!~MM zkc#S!Q_16B7VW0M^E^73`Kdh!5I@?eD@=E2+f*e!qzFK~;MHvY%RG;XeaTFuZC4Q1 zX35U%Cw3-Yv2qNH3_GBTxdbaF9!J9L-_OgJ%-K@0M}JcC!LOlITh)-126M;_fTK>>592t}bs+#Q(t z$=>>q7qA0!t6vdNH45x$u<)>74S_N%Z+gcju2nr1iW6EE%4U!4QjC{0NB@{Dq2$RC zeLg_ruRtmourqJfD(p;>_`ZaD7M#~DYVw}vg6EwjTD`O zF55wF9?yncvCdZgpsX(|bO!?VMzWco`HeaM^5_CJPty zbX{us2w;s`D!{P zptG&7`8zDDsc6KKEF0u?GEWtfnB#k`lJEKR&rs$a(PwYG7Jb^@BIueC%YU(w=08y8u;=9E9rv=us)hYaK@(eW=e^nmklRg2n^1um zCe6k&KzxhVtN#?zD+`5rn4p3`$UAva0Roda5CftvxXfHdqmz7p@2aIpL_s_5P(a;p z>ldFtp}aCxJHA%?3&CnW&2{=NsDt3>27Egrk9zgK6?7Bl9`=!`KX`DQccKRLKQth~ ziF^0^{Fp0oUlNsa-+blyH{IJ)M|XEWfRa-F))d#tN1W`glI15TzlaiQz-?tb&)_LO zUiN;$pBv-Nzg@Jp1F&0AtkP#_VnC$|?kZc9pZC$rpm=3+76}rrgwyU`LBH3w_KJ$l zX&D&-!xezlgQUXb*H0{XX#|89LV7UO2+N>7=WfRb1b##BS2{S^p1*ChfURk$tAmhb zRp#7Wrx6OpLq-I!S-8%||3o-rBm4hNIAhw3kJ2x<4Yjpb2B>OWrVZzaI(JqK^#9X> zzq2wjut9r-V1TyXSO6QwKAT2s8$yqSsA$&avEi*&eMK(p{@iYnx?AOk39PFhsg-^`Osb5Nt-=oxo@!vj6_ z)lD6g@m#+2{M7FlNme>OGD_>I|ApG6!tNvJ&D`c0-!yl$XZ366QmfnqN@je1`FV2HUR`I6wV}ab)dV!I={~?zG{PKEKOhvaRJVqxj?*{u z@+=YstgeA<2+`(%{KR2m@>0?4`x;TDn4N}wBI@AXKi`}s0mE=CKu|#-?DzK%Z?x8- zGikNLq9T-`V~k{3!h>rD;Ns)}hjR1tsMxcwO389g%h~vNcz7s3Kxv@z%XpK=Lq&cO z1M1Tce`^BCa+qV2l0w(RF7trsq{1E=6M(3u9LzZ-oFCxi=0Q4I*2K)p&%QTOW)kIV zgLr_=z+)@teiILV3;v;rgM|XQ2+~EP(ttZx*VH^OzS~%vY22Xb>$K2IuR-~9;`HQc zkg_6DHlzIhdhXRb9wRIX9_7T(aY1t$$XvrWszs%{>@KSWMV1QBcDxB5tzKRSb{&T4 zsh;=B`({1zJc*2vTXg)UrCD@sZHl$+#ejd}b{U`;ZMYScLll522F~uh4M~R1{uVz7 znxp)29#PUUV;LCO?jNeQe3U)KMYW$x3H@loJwDV$FBr$&{WpWryyHzMJXklaWnl!U z<bu`1^9marx3TWHeIkwdP zy3bo#I?zEO>SM)}T+k%4U3TL`#I~wlJYFp^jq#s{Aw--s?J6aMWG5p58B54m6H%B> z4Gm;9qi1E#0p5#`lJa>M4);ASH8m!-nV6D^CJdfm4&xybR2}pR6QXWNy|uojMl9gl z`Q%ZxNAhsi5cKqgCR66en_Kc1xmJULarZjmbR=5+-$vP#7*U1B6I}u#BK7GwN8Mw6 zcLP{)~G=K#~owSSEQ-ormPIv;dH^55(ej#%^ zqE3N_2y`-@duu}QJTB#>cd$tNw4*Uk{%oFq0v$Zh&WL+H8OV76ql-^|*$BXr4xK79 z0%alFw4haLT~j*}&$o-(4K={S1$Y($=fJLbe4GPx93-pA9)rG*$o9zW$NgEs&Lq$l zT_aXu6zv?=&h~;r5rNJ?nDIDL5c%r@{o6udCZ0L_JSt6GU|<02b&A6<$Zm`H-by?% zGJWxVW2dkj1C3%(x%=r18*6X5JjBLy=*Ve0zx3=okLCtu0}F`RYejFfLD09=1En%U%!2<{R; zF5iu2Z&sIH)nJ*b+(b}1Rc4$PG_|-Alo4B-lrkwSQ{QabjSNgpwXgZV!2|u%6lloH zl2!)4Bmuw)+8^Wu!F@1}csAaY8L-j{d-7rO7LU6D?Fwj+&%XzsX)mw9`PtEdIn~S* zDp}RatE^1qu00#a`qkZUL8&1)vhTj=<#N(On%@a*9zC2+b!z6_)Ow`MgXaBFGH5W6 zI|3jI8e<hJxL>POjjfRcZQJ+uJUzP=E3WHv122FPRM~Ip#v<`q2 zAj3sac~_#$d7DYhh5!eST1oNv$bd%n2vAdiKaH4Cqvn5q;&>}!(^7p;Rhk^*kNT`v zCJ(3tXzkN;hOm;|rNAk3^}ZKq&tu;l;F5WJ5ZkUTZHuNHwp@5#%V&hhP?tSxZ58C@ zvGuQi4lU?%v%50%rQl_^53E{p+2eo;BqrGbUe|)QPDj^eF8Tx(^b5C}&9r3W?<%tytD=i$6_13KoF+kaJ#S_cVPA6) z!o8NAiI3wgoxXMN{(UOFw{PV!Sanfzy&q*w%=&PyfzyPUD^x~)N>Gs2_($;!!f}0= z4g^fDemPCfX};TVTKoWQi$WPKt?200==S~Y?Hw-%Ga?qs@S7W(R#nh(f(Rm5+FR|N z`A4p#J*MglEQX+cy;|kQ>|r5Pu#cE;hv!URFg>`xk9W6mY+(VBDj_t|!Sh={a@22r zWCQbvrB0-M{EhSH^#}9}L8gGR+#0361P7?DZs`v~db_0iNumw5b92B~>UVed7xdVL zlC3B{=jSh2WC`v696aTD{h#}@IAh%0UnVQA2^FvntLUz(yS_Wwq?gPtr4RH_{-iu# zLInh8ON#dqtgeOw(l(w$hZ=Qk&4id9d=xav{e%6bw>9JTxPpFf|C`4vE#yC;Xo}Ik zpR*{dBx6K;CG^EpME7udLgFLqNDH7byWj#Ugp36@!fuxI4*T?5ZSN%aJ#$5r+vfHa z7Ad#y1{Pia4|=BQnQ6u6Y6q%kU1LjU?soa#F^DM(0{N3qzSkbm0^7;$WM3bZbiICX z*gl{ul{jH;dK#7eb?w9IZmwx$OjJPevkp(XuTHZQ2U{Z&70;iaGJFFG*A`rR z{y7cE(hAq{mY>)HJl0S=LpfLj3ngBF z#W*~#vensB!J>Z-bDhJIP4pO~u!9oR_{k{q)wZ^EdRa2YfzJ~)y5I==%cFI|_U3GM zIQ73!9QC~T8u*IVTi(Q7W2 zTS_J79YOKQHkikc2|ijRGJ%^%WJ^Q_JFLOf(4^?MyvzhLU`mFyl*6KpwmlIQ*&eSMQO)nv~VJWAafeb#a!$-P7mlSw0> zbJ!?hk>8ZimJU_ehr()uafcKiqksk>Itd{_z4M*BPr^3+>)?N-zM80^%=sUjV{zRW z*I>M0ka#MVm=^=o?5A0K3ixIt91twan+A|(``sukzp8$Qs%PhsX1Y(7|F=#~e$@XP z)zzv(uIQAMbga|(1*PbXAV88JD4K|>-?r{`t~RBVO~@_B%WQ1^w`*h)WgKb2jz{`) zl@?KulAXNWG>^Wt$Q*c?TL?g@jg!K380Zl*l#_EQ;Bqu0U!93%J5zB$TRMWj-TK1p z=SEH?`m4my{@4%A!7n!CnQEltDSijB_75Md!`6gEqzbTX3a%1hY4IV2>$kQY7l@%K zN5&O;95A4-{6#mjV|*Fji}AE~P*glSZ;sa$_!7f9O-FM<+}(d=Uh>21{CNY`1UUlaD0;Vh3q)mYJuTemCQXJ za3fC-f``*In(yRIX1NT)TEe;>7a!E{3xIQ}g zT1M&qhy^qn0fZO@pOG9bZJNbR<-`M?P*S?kjKE|^m&s`<&o@HptF?Jp_8pV022}`g zOqi*($B-ZbNv(+nCBwL`ZK$W$7SC(+PeJxp0>8cI&uiP^8qMq|a<~A0eIDF>)F=46 z-$JXrT35#{sQVYzPBStm^fwM6CsGV;2$6I!n{_1AWwg|E{H3=4KL*(}j2MBYd;%66 zAD*;Ez2!A`FBj z4~hVh=Tu!k?M;W&xPx>5U9&bFal#u67MZ1!FXIymp~gQZZ|+}mx&q}dP=4~+F7tkM zR+<8fp+l_UUPhSI(3Xpwd@O)lIQK6MeI(vp89jZ&ez-Y*%r?EGoFLHj;N)e!v&Mrb z21|}}sS2>#vte!i(Hj5vA0l)<4@$8Zj--S+Vd3PWpYTZw#cUW&Ad;0)}P|Kc-<^;0$8tGf1u?1t=dy@a=ox2Q9T*w_7{6GJb zLc`2#CV|X979SJoy+8pLzH3o3{@=U0Lid7hBNnve694pn*8+g_3Y4Vlx<~vl==2so z1>@OF$*u|^YMudP-*3y#vZJ8k#q_^(XT8EMPxz^Ko;oZlse*lGUJ$S|d7xfEz%FZZ z25tETxYi#(nx&Q*$;-q8RXg$HW!Ar3Xp#R*7us!j41`^xJn#_3b5g|<9Mtk~Y7n@a z5XU6y>3X+`*q7~4fRO;N=3u{y58k4Y8fa+&Ngz!!B z50+4+<1XKz<}^Kxi6{j5DFPaSTn&b^I6!g$)2yo-WCB=-kT`~7XszNbxe3Hda61qo zT0Tzgzb0hFW4yc=lH#%plK>V9Q47F$TDk42p!>!$R}4gkb#S4Ms9ze}4iEtBPeQ-? zk7pP31%;)nc-|b~Ya*|M%E7J{m^|-@?wY;x}H}q2Q~RTeuv`F~L@zD)Z>Xv^78f%2*eTM&7TJq0iop+H(ps2WwC&BY&M;JIu1>dbBAt2Y6!epyqA#Q`Y4Tk^);N4{tX zI)s5CQp-j3!Eh;lxFWKzZ_4`)?k&-&L7-8RGc;uB*J9StGK_}bjF*E@|0v*~973V) z+=&80kh=&oc6Njn$uvXm>?{JfnaMam4h0K+)JLLll;7uJSL4dATcr=Rl^^d@xsfqP zioj!F-!5h-KI{SxJP>ExO$y-Kb#Z(5GOz{*AJ!_S2JsuFfBV8aHI>HLiBVHuIqjiJ zcVf`^6q+?kv?mJ@BL+cNWxC__K&f*3QB(og*O?h(G_$%;C_cll?afSd%>ISVY}uMw zWXG+lbpY{!vNo5!#c;mb{1UHCb0xL*uhxXMnmq*c%q z80wp@cIGeUHGP?C5<|gKjs{?)sn-nJRhUxHvm)4@_jfVv*>Pbc`mli3m2FnB#G~#! zVZjR&r|(ZNSvJ0V!i!4kFSr5rxlG*ZVIhG_miV-xN}zD1EYhGX zES>ke^2^-Voyyl<{`=&mb=DPvu5?H^_vF8P-7fO4=Y2PKm(?LsIt%6R-@|J6E2JEd z)#xdx=|f;mAxwQkGh)O=fvH^^>TN_t_vjSV4v196FuyZV)>V{;SE3Fv$?UE8KffbX z)Y2W~Hx~OkIqIO^kpi$x5W~B99Hgbm6^CDK6gV?^pb$#NoU`zW>WzdgK9O*cu=^bf zO3H9W_Qk;;(YOrq&!2~LuxEhPJ~rB)=#$l=y15oDB_%IW#D$7c?<;`9@x{m~+`=Yh zo0*-_0D;vfn}rwfA7B!`z@8C#L~gnH>H;h`#td&KHKm4c%#DlP6ureWx2c*}=6fUX zNtoIt%Z(>YH@6vXu*Ze*|%JaB~s<&6&i_F6-DflK1#?5l zJPuALhIQ^~YfEIZ97|vb6=e^o9B-wu!FJKl7NC0j{Ue_d?_TmfMa6+%ziz`HX2Q8E zr63cB!#^Z;zy%U8E&x1tm7hN=GV%z%`o3xJ9m7a0f6d!t-y^}mw9v(^jcf*RnPJMG z`mOH`ZSjmMuXk#F4f=XSgfDSvyBelOCZXk4B!`~P>^iO9#I)QSFH3c(B6~FEadg;Y ztXAOa5H1)3RE+xoblEEPTk`Wg@9!32EYP#HpS23)yxNRnSlA?w>`{JEW#>nNntWuw8sx`WPmzObpD3f$Ib6L4z({+qCA|e7o9{h8zqS8Fo=-fyco=bab3b>^a z;Cqnxl{$4K&4Ly5R;>fI!Sae*E0@|^za}$>{iPlbsm>q%m@bxYPS6^H=E$)^74f~0 zF81ZN{sWw8Qd7fxk)D@F0(P)}RJFL+2ymwHZqeF0Tfc&D@2dvKO-n0l`8#)>OZKPT zKxS<$6zo=RvRDiW#TfWyZ`?KNJy>W(j9QQHK2biJ$V%0#^9*8ja&{hH5xJXDd6lTF zY_vHuoAA;{C3Ov}UBf5m+)j_hQ5WEeh1st$=l!JtK9HRhmKPsq8PZP%2}#^6bJOF;!NREX6%)ck zISOjp${8+Cmlx`ms6$u(u20|Qs}St{+kqDv&2$q60>9@E$+%imzSVfAiebLP+ew@d zWMV3J!DUU=A56)a`1omNKT|J0Pc}(B%h#(7-&9!RPyf2JdZgholeE~Zk|2obXZlcE zO^^@E2_-1&<(NLBpf5*bT2z~=BT4>Er$#|bi3dPY5HHz8wUr9Z)UY&7TU(Uu8$%tP z4gZnpK#-}KnWbkBQt*#f6^xbMruO-RhX;;IPG~7)u2#TG_f*XXr%fMTyc@}qb@Hq5 zB+2ZOZ#x|W2r|0=v)+yUU4@sU!^3QlGX={!vD;Z;=Y5vQfaP$hn(NxB~v@+W;;Nuc95d{*&AW|>>sfEn&Gl-V?i z6+(>w7pyPU8OeB(rvA#a(i~mNv^ScQat}!LvBuJ-;|k&-Yfk4-PA|xf$Q{VX%xCfK1ag-|NzYm3l^+7_XIX1@*gc8|f0LZt=Gia*F6dIB z%g&ipy-oDZ`vh@Q`m^AytJKy|WD{NRI+|cN$TG!dqp_ZZh8_SWBk>}z8oMijYF9IH(jE?-ui>}XBKbMiHg-4Pq zo~yp2k6v6YrLM2}_9XhMHtp4ib}kP*B=3Q{3wtR#ez4*Ubxo)wZ75SDGKxf2KSd04 z3rYt~92@}=0aOKGV3226B+Nz$FCtpJL4VT57>19zs;c0gZwE2ZEQmvuF)zL>G2HOG z39Zt=4uO5m&pg1{w#O-H*KTQPV>8j2>{_)LPUyNi8vH&&^6s#{{6}4({X;jCoi@eQ zq?x6G^J)qYe2Dqnj!3nc&YjyyhwvmAM>nGj?ixlpP-AyzI@=s)X9QGVIdWg{)Z7gW zGqyH8{!`p>SM}*qzpZ8I!%%RqLz|ZNsfRXwBCc|~mDRLaTa%98AjFo7(W$5+FhIV! z>GQK6RnPDO!jRcl?{u5G{E1FPyzEKPSmwQ>0bin?_IBJaU)-~~xqp^L!1qChP`E+W zUjMg8n$)Pe(O2HzrXWZ7X0^wTTZOJnb4N2hUU0Au6GJrFN=HwIyJI-!R+ZD6y~E#c zgw(D2Npf;*D>2n0j}Sk>^4SS$qE6)|0&>@ZI>&vY{2+SA>bh8bNC4l zB3v#-r^4%=2~Q{+2;~HQIIvKVIT-?1?W=^?R7*$F_x5(BTrFN?f5J9eMeR$nKJ4Ig z8J`h)!TmnFkkGcTPop433ZA!#@#W{~jo#fKLpcEdL@pt`QFy4;6#J20T=W~~{dtEk zUG!6P5keN4loVp;p}eW<4N|;VpchCf(}?We0i!(CJ;6b^Z+jI|JpG$KyiSy{5taC( z+j-IXfanwH*JWPi4X)JY-kN0XBuRRn~)3pA3Rs{mMteCO#H`FPvw+^`6v zkmmFWRGoDB7m%TRNRf;YFUfolV``!Z2O(WA-?NyhQ+*GorE><%Z17E{K&c~0S7yh% zsE0QFT1RX~=l$i*uiEA7R9+#}?bmBI!zy{siGUI5q>eb^ApaH*Ri&s=*sdO85i$S-ua{8be5*o_I*WGV&1( z4Gq?z zND-4gFUBP1MhLCUjV}(E$}bX_1|8quzMeLZ_q*9{snz-;hQgZ>vzPh!PM2p(_r$m7 zqvyla6`NZy1NqX(yO)WJZ`>_Mj*?3qT}7tJ$plX0fmLZR2(}ck)Ykcy8ogFowbiR= zX>rb_gjLI5`XoIEt+*WA8=Ds|yR{5Xh)Y-JI^MAaEtHsJHeXo*F&3hv@5~bwC2!pA z^yup`0cfWwi=PsCeEE`A?JkM(`*@R}QcT=*Vp82{b%aQ^EcQuiNF~f7MbwZpzSCEQh8e1`<%ZuV>@|Dl#jPB?b7i*(S9*s zw$ClBH8nDZgV3I2b|eM^9YyJAeor|y^{=}-G#cIY-39Q{E!`D^D~s3HKe-w zBq2J~(=^+2eMa?Z?Bst_y zBpiaG{>I3EDVaXOu1EeWvS|=E06t=Z4lg3YMYa;4!cY?;TbW#J`QQ8^`Md0K^ZKE6 zUf7f;S&Jr#dqobdzs<{hut6Z?N6DcKPy>mFl6j&ireFK>8I+O8$O`4Qcb~_bMQOKv zidr&y&K_|G6>4M4A;S#Ed+(%zp=rA1M7Cv?s6TXp{pxJQYuA}Yk!-8QacJX>^4isI z>wP&lI~fuL7~s194q1%z#5!m$8ru?t6b1#;W-pM_Iz1 zsGeUR>ENNC+v&)%qaLYBsQ-zBg5ljS7ow_{=+oavF~cd)>%6I>!V~nwt>0Dv7O;q9M=5E~laDINMkx1Sf z)ClS zT1@(v>D`ywf$;;jZ`Yb$r}*IHS0Bm4O^oJS6@!%z5;#})s{y(XA=~r!caTJmO>X=F z4UN~~+B(dK-Pc7ME5orK4Qq*n)coEzPxRINoFZ3h^^4*8^XDiq&)&4OF1;!3?!G)! zLihf-bqb~;%Rf1npuz;%wlHumQ=L)I!xiA+DK0L7HcjN##Rj-&mU$khch3+1Z_Nh**7~kmrddt363wl-}-k; z9@Sg8C}h%X^j6CF{8fUzF`>?J-Z)RU0?P-}ve5=>XVN`3dn|w%qJi?Naq1lta;rfO zuOo0BU6M&K;ibnALb9i1_33eiuJ!$0fKvsX60iFlZzx~nx|F77?;3e{!Q6pK(gz#a z!_BSPu&8sb-E6g{R`TotbG+l5cM11NVcpt9Rw)999?IB+gwU)6a;R$r(u`Q4c1+LD z;~`re4zx3tW>4rY`u+c_<_xEO8z9d+4Q9Vuhv>x_<#qqN^B z%n_xjV1+}IgG&yx;RP8^Di05dbw(jts59|VcZuHmH>Xp=jG*}B0WKge#%-naIw_?A zbSmF7)Y0<`rhCz!6NMZlkcwpbp$+Q6vdH1g09&Q!8VeEzC3Xg>r*r|Z_VVg6u@xO9 zT9`SX@7}FPv;;=1{CWDdlyM@19D<#w=ht%VkzDvi1^cuYGPk-<_@?BK?qE zGppkQz!IElMC|PBakmtiCI27@2LOYLicV+sIK6@bY%q?swYNdBdC7!wy1$D5V~S+g z45cL@0#b~8le)^wd&!yCfXb3^;WUsyQSp_-8J21al_EqHy$7W?JNx3AU>IOWdgW5L zGb))4JbL{61KnF`c^=(OG32;)kt#0uHLrWr`qW2zfN0@^625FvVPR6$O0hL@^wnMm zFMbMnu6p$fUHizTmh!aq+O^e}U*q{ezofssZ>_i6=FFpEu^a2(f*@Raj7?7eX+>%LQVStic5wyCLkn-Pw5_a}G zkK*ov#?{nl>%X!0)=^ou+urDdD4;ZmbSnsmfFdB>AtIn4C?y~WA|)XW(j9_=Ag!d7 z(%mJ}E#2LnXWs9-*52csz4v#%ea_zF8|NEm{l)Ty=ehH`=KR(4$rExtb3fYIPw?=; z+up*VUQ;*~!m!M#a6UE4O^uXXodKjNW$shCEu5d<@att|5bF$J!0IF^`O{+_IXB*G zIjO1Pa8=cEv`CK~GEM51j_6^SDe}wkLPy>v@=L2~qVo?9`UfJkwM55o1?$U$BWXhw z822kF38yi$`ipiQu2=E$c)mRwKyRI~W$#LM;CX@DFtZdKr11dou5zByd;>Pq;d0fL zy@TymRk#?&dn1PEr@~J5LdxMCfJFNE`$O38*>WVKvR4jFZKfvxO_UCSph+HPIeav z0aB^XdxfuE3Sk=X=$8CpLM3DjW9%;VO>3v7#QCKAU4~s7%3W_T*&<%S*B8**zbJgX z*&ZF?yt!qd{@ixL8Qpeekxsb+mQ)0q8Z8@6hMB&ZzY4=~0Sj!8%`Aoy+r+mtiM(Ow z8?4~H1aNQ^1sMbuyM9Qk>`+!p3sG1eNC>E)hK1JhsO{PHEKRTDecGh2{sxwo$D&W3 zcnmyJi1>(LA|Q)+-GqhOd27`GOX8QRP>h5ro;WC|A;iPiIY_l2k?(Y)DP9dRXJ%`S zHLyTr&92FZ$@%5FQM^M79H^|P8h%AWs}L)Y!gzISi`zIUaB4~o<(4lW2|XMefavoL zV-Oj-R*;^KzzUNq{gluRA|{-zWf>W-L|5~%e=}i3#X2O zW);US5!xjhXH(yPMRA$nb(@b|M8O)_HXHUSBZ^AWo4qm0lGqaPbLeCcEwZPSWDOGD z5FkN!q=5B1NYO%Q{U3LO<0vH4$Y4(l#TeVd3n3S!C3Mz-GMGpJ`7qs5HVf` zr2^#fzF1rNd@$)ldp%>@YLAxB_YnKqq_QE?)ujQS%1WB0{*PH(hmMxV32TnD5dFJ( zoyc$F#FW<6ED+@umyHLlc&?-3m9{^h!4>3uU^Br%q@|;3dh?Npth`!1l;AyzA|%&H zoVWN{MGkbz?TRiRK3Nr0$)inx#yg18b>!Ob=t|1uxz)DNT<&XfY##_Ap0smA7Gzvk z?xrewnKH*@P)?yM#B4RW^Z-44eco?yp7h!^@w6;0gE)9ML2Ho&xiCoOyWq=td-d=8 zur;bc7k#tpxDFp$8n2^W`^5{Ys^u&!?~8J1x3;l4JM~d6ivy$=?xA^_$N&12{eQo zH4~2MY0u1$Lf3aT%~adaLdX_Qgg{Y1Zy<#jbVTt&>+(BdHu_&1AcsBiqOfkMIQjZk(Jb0sHcWqAoeTDSoL z;o3+AHdLxng+-nZej0AMd*XVfQv++tkF$~h{THGqY*jny)ZOHqId_CvA8E5I!k45XIUS} z%X2wIa9qqVg(blYj(>8C)^+14-B$p_v|as#)64=@#>B0xETv(b*fk&TWC=&Be4{bV zmF3KA+=fL=badzD=H`iwF8%T!sTY4s)^1&gTsIVIulBClzE4`oOpN zwsv`&qcZ?L)||YUN~Z^xP{jI-U*oE(HFWPg&)tGzWMZ+$-_HxQd=U{8V9e)@BBMki zt=3rBEE_vTsK;@(&5zfZ2tZm#HqA?f;&g_b@Y3SsOpm+p*!i`!y+Jl9_Gn(2pGCix z&Ze0eKog)X%vf^A|4IPBG1R+DEy-XZ$7|0f;~&A)a^Pd&;J^m}$J<`u32I`ksEJ*{ zE$o>O>v~X?d0;#T>~&JmRUl=wJTXyN`DgbeF&rJ}d3o`XFrBS?Zw65VIjQ=f<`zriHwK z?spJIh=6IfUYleA6w#i;!yCJl94aRF-l2pn6Zj!y`P zDe>Wr;x=U+GR@yeGXlWT7YNrjja&lqecI~%)r?Pk(*st`FG6KK6YDBAk830bEA>fX zn2FcgXsI}z((LTV4EWeNnXHrarD*N_nhe%GS#(L%TUS1|B(4H6bMPT4kxBubNR#%Y(hOarrN)71C z>+;L(+-7u4?-8(`-L$S}EVi2G z`74yXW{-i#{rb&BcGQu-2mF|J0=<5|5DG2^-M5_lKqyi>L->tpB-cOcUXeR z1>WNX{!s?3;(1tkv;+VCln~kv_^uTKU9ksv%8z? zFaCUW>V~$kW#jtpjx`0}V7TW|V{c_L?(Z+uZM&HVFXFte za_CC00)t|?AbzGwoSPQRBn>gVs^&W?N@*RgDmx;GYDG$_BeRVWs2IoI4acW{nkEPK z&il`Co>1nJU|n`j*AY7@k}ah?txAHyc}9b|lSO#bndz(38z#rD%dXz(xRqs1eB=mH zdARpq2x9){&`@8w>qO4W1W}4~N#`m;ZReys*-`7yvf@P{`Qb%Xh*8c1{v>W7A(JN zmG9&X{IosAI62*)cCDb9JO4F6Q3YF9*kM1K%Dfy5_o{|;NVxNr4yTA+bjjupfWHq- z)iTxSsI*4%Y)3Ir)sR4#2nZcSxuBs~j`!+nT1bXVy>r(_E`x3UBf@b3yXHW~^Y1Fx}Un-qoGmbfO? z(fly!V4s!`OKo4clZ*3xkd_)eanqyQf-pB@9PgA>E|BLy?kZ8ko?@#oSVt;c)-d6PbST{shk)Jk z%EMk$cX)B^>`@bgo5U72>*(+~*723d=cw$Q+&65G!_IuP2MRv~v%H9*FwYeHZ^T?To(dcWQhBBK80w9= zdl|jb#f75*Y69A^-{FBoErGqM2$*@RgKKLsys` z$id}8KBtlKzo9_wCi}+~9(jeu&`k#RejQGa91; zlAnx!MhyT~$&K<$nsJxWs_5ASD7QW4>2C910K|4tv7d8k3X$3IkE^m^ZWMUS+84dQVCqcO(i9g**=)i|rSd7!8rEwFN;TM{X zzi$e$%4NEL*+ea?deo5|j08N9+I?h;FZFR}8-wzo;pG^EdZ!MwXm{>RvC};5^Pbw1 zfI~wxfgHS~^-YrEii((k`U1K83+2Wlb~l%g`vSnCzCr#78~E@u5Bjh0ryF{gpIyCz zE~K?GmPZ2IT@#4?L!lsb^RJMuI^n-oWqaq%(KDgeLEZb>%OBd{%?FocB4z=r^~H{bVee@#f6MkZ2wW@#)E0I$eb(<$s< zv%kp$xGpJ)RBV27Y$ZETzg~$W6%U~?AR<(r@_z|M!`?A z3l`=Gj_bGzaHoe*#r4Qu~gFhWMp5KS!@0*iA=o}G|T zEC9Nt4y&bu0~%VCPY3W14-b)Fu4POvu?D;uuvBI1H-ZK)N0LGzhNh;Q_f9f& z;S3CL8t^q|cKbUNHCpFT`^hOh@qs@1*Eb#6GDK#m_U_Hd<2zhM69tPY>>9=}mPfCs z>2E#W%owS!FHCf6ZtahQ>Rn-(_svmS5ZGH=vSZNgaVVY>`jAZv0D+I_|KK=(ZO?zq zEH5?rdr$3SBZ%J`NS0}CKchsw&^Ps}q80H5w$|Flla9gcvL&z$gOvu1Ae*g$W?NSM zZ0(N{NV`C3Pp71M31z*$cp*egBJY*?6%72HYS9q1&`;U+J>@Q^6ad>N0iw8mFe67C zT6_!7Rmj1!>Hd9-l_8JFHOP<#Bj8%$Wf*N$C}e zQnQS8E07-P_Z;0@&@=b>zVhY^n2zMw)WB!&b2DA|}GE1o>T)$aj_x;uft@X9C)AOpzn0&yPEgXoL{rpD+FshH4*AInO6R zo|7f;zkbt-SguO|TyqQgkdiNV=yCQQp=B9{B$<7tje=*2uuKB+dY~ zJiqn72;|&+uc@N$*%%~Z7GS6_9;Cc&YbdKu5%86y1#D*$fFwOHdhPYOC$+XPT%iwl z^QoAw?nhHl8SZqA8#L_RO#Yf5Q6!Vj4<_7l4u+4XI`+ubN01VQi1CU^kt9Y?<4x9 zJd=Hzo#h_NYt+L7Mns89I%ud3;4#(@#zSbQC+n&blDctFq0pIeRDAt<@zNzPqu#Wx zG7s9`@r3sm@(ToNT=Qv#dtsu?4^b1PLg1-k#jrD`HTx0q;e)%ZO1r?`)NGlYGP)NA z%8e^ox;Fdgs*KHPTGE9Cz42eY)u@NKIMq1xf$BLY9Uo;6ocrx63CbNbAf!8V2ZpYtFg2{EckseQd-f=8tl8)SP<$hA+)y0lh5RNlQ?o zg^>Y>aBTgcb*7JummY;El|ooq2V-i-4JvsJM0~N}`{LREz>-yDHi`(7Ao~#sJ&L9< zC8VJ7lXsOZY{_IYcxN<@*Ax;j32X1Wv6?y=yV>3xHT;H#oZ{4HsELt7?zQnCN;WYC z1vfyWEOy~T#GXoH&cR|+q3OHBtD`xo}fMX zF#eHEO&P`(btOB|E19AL;EiaGBhL>!ia@ES<+FeEH%Pq|lNl3m7v_89@o`g8562_} zv8|`0V}plZ*K->;kN19p=jpy)l0vYJn5z(|J z3|tZTb>O%!2H*qX064=+TE9bGtW#7%50+!>0b-WMzE6Yf*U?~jTr1S&zV;v9#d5{wa zW4|rO?VcCBzKnuVDJ*eAyA`-iiwlpe#m>Rye+Q>rfZ@Qv4_z|ToMqHc!;69*8F^R$ycKk2Xv!K4(23jx^)=E7WBB8{olPZfA)|rdw`%|wpHz? zp+Z+z_XXxhapKJDuthAk^c&wcB|evx6< z7}cgcF(PrLaMuS<`&G{7rr3__lni4|#J z$Y1u?_%kWQiI>>hYaYeBYSV)cL@v$z;nTvsd<)B)xD@gcWO z@`8VB?k>ISb?dn}f)RHMRY+S1Y>mI0Zd9PS(-s5%xR&brFTgWHJkQCKmEtqwAzrhQ zuV*wKvH(Sv?pvoIi5RvC#Du1!g&rWj3I_=if-=JVnzNf7mE-uKKQ+t|K<+2rQ*M{; zm#r=K`!6M+{ihJnB`Wxt%fwEN4q#p~xT@YRdPzwk6*7QwN`TemWgpj3@5Jz!-_ST- zLm)_=*s|`np_+UK`+E?Crv$AXA)!`T+#^I}0pN$0Y}w7H0?r3B7=&GW)Mbc=1XgW} zCfFDxC87EXQyP!*j}swXrWjY#q9jrs$A{+VDh;^;kS?-tjkDLB7kd})Y9caou;3&F z^VlMn7g}(my2jt-{5TKWSP((j;0%k{O!xAuHJh=XRBpqSi+4E9WB_b6mfCu~v^Wy5 zf3X?9zPV;72oT4YBiZOd@!X&H-*E(|D#mw#y5i2AWl>NdaF4vYE(C*FZ=be=Gb0c) z`DYtCEr}<8wFP%F9@|VGKT%YFd49@^RMZE}$_yDp2L@5Yk_bX+^q)s4%e-p-yAjG} zqyIUC@_ZZx;`GB1|1(TzLle^$>_6JWQehuL;nMN?nD1oxZ2Tx*_ceOq1^PseT{hqI zYnJbl&j+~Q3=-f{@|^VRI^8R9rgAYoH$Uy*EU0-0)>=e(S?A>VZv~fqb{D!bR5Llq zG5^wCjuRICH)t~d1NG|0>os*gNT=@C?gySaiJm>8SgWZS7Qh4~lB}XQ>PEUTAQWhv zAIDo)II&-m?rDxz2)Yj*^bQx14hAMxsAai=FA)Kf<>VICvLe&RK)499>Fz4+4&V_s z_QylccO5`lS;>57oZoFpu;uq96f%3)TDe&*=Nt@RIu(x)q&I!__3jl1{hQ4zfbV2 z{s6|6J1)Y|1rNvgGX&rT<IkWXwu-B+z?`L9q2fv>*bYJjMN4@aohlJe5r!Px;6#}J_=#{;LT5f z&w&>npqk0mdo3;Y?r`mD^DAyx<2>wqS1k&}RC6Dppg9`IT@f`U44aU*KK%&GNfXQt z(CurXfyEn8$Drzj#gTf;W=?aMlX(%gDfj3UJaC27zh;gPOXjp0eu!{euvpOd@_K^| z4-_)?yC9pO5`KI2bu@o5_-Q#@|JMpIl~u{193zIK0p{Q1UHfs-;}1`>-QjUS^ahV? zN0AFTj)uMozWeX8+PH9UIXTUIl+UJ-I?7K^KWbg>{}*8Xe_Md*(%w`0Pc6WIw-}RX zCG3F6096fMJ9wNt(tMS^Yg7>7)0}w;pJF1K46tX3*$wK!i%KLP$sU=e>3x{3ggaVl zlbMqG;N?<2IpQaUJy}u`WJqcH%mTnovam7*`}#sRCJM{R*OYXL&(EGoO6Fg`EY^)J6k%-h0f$`uUMqM}De0}e%i6pcl3rw z)uLp14hqnXuMfI=1+?^-Tm(1okK7+OatK|@cTE4hL;Sgb;1Cibj#vbFLz3<64Lf2L z9i&rsW?G2X)TJAp9}jeYgIwAGY8{1NabyItP=cgET<oxa|d>D8H=clY; zDg={_r-vu3^gVrzY&Qhq+-4i6fI>wWUyy(3>6qJ?s%weBw2>$9%U41I=engB2zU3% zBtbnF0j6O*tG{M!ECh^e9}fcc(~62NX{!gf~UH0CEktL31?EQof~r z#C;$IroMhth$-?PrRn>KKvKxvg|8p8hgGwLk4WvhYcKx4V;uIUrmS-^hmET~712BX z_<4?j8dTSqTyF{iQza&Np?rNc7-&UoaSF|cu=;pRp<5Io`c!qy1Da4Aa&mGk6=()V z9ZReAJC#<3V%%%cg%Cv;lz?O0uvbgN#4AAangoRHyDwh99Z-kJVJsivQ915Sd=t3O zFA6Pqndt=-xYd+$w1w7eUN6~4MMojo*-&O8b1e}jk%?fUZAis$>bu+!xL&=#8DNg_ zzG#ZWBkI#oQSk&}O;<&x5UA}-O5X?!iC!m~94YZ%=DmBb2DCO&WCXsTnLvWDKmcYm zw8iYVbWC;0>u50F1-EqY4?=#V!eC+`n|=lcyK^nmD0Zl-}S`)68!GM@T}Gs@bIcvF_F7e3XiOY z;y1YkgfnWFJNcOm^pxm>V~9gBoT(MmO_xw$*t-mvgSdu`jrNK{H20)-OmsAi4I*ea zt3cue;4cst{?6f$7AR?n3{;3-n}8PT>pv~TjnpQcqG(`)i!6rMTf6s2A%ZqT)8P^d z`X7-8IV0?@o81O&QTH$b?!ADDpyZv~l9g9|h{sTOO~0+!uQ)R7J>($5FeIv4hNOut zk-arRx|H1@^n;?)B+s;8Bnrl#qxYn4vdDk?z4dc$nB>`yIm7w4ek%Sie|>OHRoT;` zKYlLj$tD$`G8{KfyJ+NTSo+dY#P!~M`MgkveTTrSIQwxcCel|IkB-H}#giPkD$9Jt z<6gBA0cHVA z+@%51=c!2}vfob)D+PQ4F1Y93q)Y2Q+M;|Fc?na`c;Sj2p`7_zITe{?;Ic%efo6nXVo#?(tY4LNuH~X#e+Poq-0ZZmk>p; zuqdt}PaLM)mCMO1(fgk4WGE*`y=lLM=jyDW?1N7Y-+L#f6-6L9+i8cdRph|RYJA-D zByW$7wQPU^IZz4}GwfOE_2V?J67r{6SW*YWyI3%v5lyVd2*zSOyPJ{}tMI3Tx1RKS z@`*Zh@}8esT3Y?eh3ZalT#7FyC3YWUKTxV>XdXys%#TY=@h5p2(dm6%N)iY`;GBL1^A` zGdfDUqV?+Yv-RzHnC`^2&wgWiRz7hTyksExlm>u*jN|0R`IReMon3kF_ZPwrLfy;D z?cP64UOU=dW)xzSFdFsYeFlS467usUjh?N=?8}9@mqQp#+f9zap~B4koNwq=wo;e0&*#(l# z2Ic4HE1K=!Xctp;t{-W;3a`(?a071UQ=jC380>ECa<`bwpJXxzhde8zKQY}p;(g$B zD+0DZb`~vzL(wg5C59JO?!;0qJgr5Wu2Utha~9~ZyFA%GTL0je$cl#T>YEn>?HDB` z<`ze7y1*Z#XJ!c8|opL9lrzhDiDqwj{~>1Y!X0gncNhN&8|}~k1|8i^vdA255|+yI$3+J6R$m0| zDIh#!#1=q3;*8WstHWTxo{uv`bK52*Dz2jBPJ=uY8o2J3$6TGHqA(2HCE(h$lmmZv zw|0Vph5(g_R#Tn6jLWb~TVXyv%zDvO7ZQx_M8^~jG8;_{ZOlM}`O5MlDz&HX& zXJgkiv*%LW9n))1v+#T%o)R@7Y8}{lbgGQ0RAy_MsC(9t_r~~rQ+hfwOu1-fFIK=# zn;sxXK7xv)v&#wJPhsphm-U8ddHZjv_Oh_h3>8(G!wBt$@zY|LQ}5<=XYa$0Cl-tZ zAb%&j+l}q25F1SXbMD2j{#DtI=84M9v<#k0sGDrOptW7xpb;SiR98IY+-f4|T*yMh{VI^H6I2^UV(W|}4H#y9u$8w07{hv_%g zG`(Zx7XU}hA`Of;#Tu_-5`$FI>Biv3m*d78hbF9W$Y87tObGp!Yi#mcD?-F@u(QTD z&|msoOoGeqg)6*y3{ImMnxGTzNqLxUTju|~XdY7Z1PMIeO9*nlX{P%sZ0}!vqegr; z-_%64A7h#+$vj?L_Tj_YV68H**1&z5Amy>?lIac@Pw~P6%O(66H#(Y~j+rfhNr6Hi z)RtuPLKLj6t$Z$9SHYc+fqDE0`PoJ83@?>dD0`jvF zBck?w8pZ`?|3GfJv*UhkcPA%2A>s+C0Z|@47W->jZJ+N^3&v5D+E~`rHoR~C)!`$m zNQ#(|LBzvlT+0_Ga|te{{%nQ?l7<`36nN?SbpisiTRYk&eccyy*`9*%=(C0Of+w(o zx3FBDLQoclM#hJ9=6)1^l7LfJ{$#X_qpI%-*(^li zNd({4lGW%4ZM&6a1Z8hTTv$z3mF+6O&Ei=20RT>Y*+T3)_z7no-d9Gs;$p)ktj;%+ z3k#{`;BTI5{OK7KFG5WQ&clMMeOLJJKQ=T>Y0jhd4W`F@#oOI~kM0lN zUcRtia_3RSF=I;&z=t-tX$1xBhwRTE0`xm_R8!||^=?MZdcIwYg0J2KMw<~c*?yTYmJ5jXzu!ky&A$_oz>Cal1NiA zDyK2v_9;jGMz^KwfhU5xrD-`jb`z2*Yy1i3Hfh&k^W zypSX%zU|lh4>9Abb(bmQEf!whNR6kewDb&Dbc)`?b&S|PDv)hbS5`HZRoS%%<9JHK z#JAp*Eb7hOpq4BRMSiP2s-nu=+|X;7w-b3xSCLyn^+nSd1`>-Ob8_Y-AUE$9j6Rgk z);e#hXtz83fJEFt{ycIn{n;T2?YFhO;!r~{hVFi;HGL(h8DliM;l#bh_LI^wf=vdp zpA4MuGMEZU{8*lm<;ufMFLHEEOyv}II%*Q3^L_rTfcQC&&C$ceY5l19po_}8jczb2 zC*j+--EZ6EbC^35Vt4rY`Sq>p`PB>a5BeuhtTFJnJt4OD!YQfCvJwD6d?F)PG3riO zYX`EMV$1v>B>=~EeQ@h9*&<;r5f+!lzvM{h}wMI<;)em`PTrUi^>i&$WT1KeCaqh3JHYy(8=B+ zKmPg%omJ!`+;04h_W7>*z;M{VOw^S^pZ?*!=J}6*D16OKV2E}@l~fN-oop_(Uixq2 zHX)__|JxrVRh~aT%92R4>^o6h*HN|6JE95kfbWpmA7v&0M-AE9J&FsAcd-K7K*zp9 zG3pA~^$EH=L7uTy(AvI+y!{TAD;^bGc!D4NPVD_MZ-`7{z;E)Rb?395=})HyR?sb8 zHKFv@jneu3E?!jf&9J+KOmI?eEJ@sQZDx@&q3hbdmxPH zJ?9+vwL(PsN^BvhR09p=xGyEHB~QAEOma=q*PC2e!JrnBQ2%2qy3wr^5mA1QsRM1a zI?)fD)!{!9GA}MN-j&Cpnu4qgN6pW$)7QKk#!(;2_X)1X%E~gG6fnhrkPbAI^AT<} z3}8eSJoR->RZ@1x zd+bFnzD-z&@J>QdT2$tR=a6d$I_f&JNWGFG$KNpE4W^?5$Uu|-J_uz(waJcpRN)(3 zEihBB^x(r1fOeCzHvYKd0%Uz5=LR`+|K<}aHN4_CzUruqoU5Ji=GJ|97UA+_Y|I@tXz4w|{X z@H=BHJukJ}nqfgrz_^Bmp;X$7(uNbuln^I=ZgEo2k~f9uYx|rBE=z9D{YFX^6wD^^ z{A7sTo2`m{oQ(G(V&CpM)2PIgz7)G%R${Rlc`nB_k&!}rhzf??PSAM|=k@QH_YdSD zzM0$(hqmP-IiMv-?h0^_HNzRW{_+oyi5!8k6X+-y<@+9_H>d=cQwO8|hpID!kQVDc zaMRq#C{5qkg6?Q{)dSW#^}N?Ga~0TBvZY0@YZMe%c+UfCJTP8Bq7K=8!!JrgW&3MB zFb%@+QM^0e>gt9dFmJ~@$V}qg7h{@jaa_2hL`0kxX_)^0lJSP0C$>rBgdM#Pt_ zzR4>=GmJws=Lh*rJBxkjP-r4NvX6ceEH#)wz9y)Ri&)Mya|M#mdQFJB0?D5^?p9Om zG@GF`>T(t^Ddcb+1E0dpFe5|#DuL%EY|^@#O2$td*zdi(?n~Q;$UvvW&_qdGLb_$M z^rAW8{a~@M=2xCCBIbC{i+*^*ri<4k35oIKk+*dw3!hb5U%qg93VxbPP)~30^Qz6* z!4lzOucB}zt|+rCT|MPdyqutdutMA&aMD7Hl^zsCy~%xe^~an0goHv5<>=v|LLeOo z&4&Q9sp=cO-y9YFMO@#r)%DB}D~2mBl9Pj9T#{ngp}uFPrw%+C3=dqQ#KTh`CPRp3 zIy6tdz7A#k%yCTdPqXnz+02*+vU^jq)%}goL;>JYSFix8E_mnxF4)&xKbvM?emI4ZHj!7g_ki{CBQjn>S$(?vCED8Cl-h zv5r0tq5YO3OwW*X&txFHFpj>+1qw{$z(9>qYykGomjf+fu#j;u&nZ`zPT@J(?i z54dz0;k2fvIv}DBe(ert9swKA-%FX$g-x!ICr>t4hN@6*h5b(t_7AP#UW+M4;xkO8 z@HK`|dk6^>iDlVaQTYZpWb~Po@Pn=sM&IGaMN`16>Yr?Il;XYbjc^v`gI(5~=~$kf zrm)<+iA?0^O00hw? zB!wg-cW4rmi;Ky?W~-!Xdc}qYA9_%!;4VbyMT)7H6vDxPFNsJGpff~DHClF8dU~|i zuYc2b_RM~glb5dpvG?W6k)AYGeY49Ju3#0t1XnY6y*DhN2tK!P^-bu9Ev$~SQ?;2W zHlkEqwk7C})9x7oHQ!xVR5R~(7zh%Su#`}6usd(-CEktc>zlODnV`QlPD`uItn7i0jr+a^7^v^i8QAUGGm%XH<#P`iq0!vPu zb{AI9y7;<#db++^>W58dli|}nr+m7@&r1qZ)}*ah#Guek_?*FduuepyS49d5DU3g8 zZ@)K6+c~Rax%q-{X1SN}dB>9{b$@Eb?E=Zb>ve0#YTcu2!-FO0utUbm z+6pNhksMzAU>9Y8I0|sMIaY0WoU+|)mXK+}7)zF!lDM+9#n6-nnKn@4L@h0Qz#cYO zBtpdYH0!;0DSx@QqYA4|P8C0*)vz{tvr#eXo8f`!5j!IB*bd6Z1Kmj*IJZ zG&}c;1P-_Nm8j>@>;wc?1AnRIxV$a%|3|;Q)`D)zu5j>>3iR1V?qKJv5HOQVsO5cY z_2kLduY$F3S4s>(W9b_#bTDKW6cmJ*6;olX%O=GFc4$n}$jql*#k<=?pN7#7lj&La_j|Z+p zFQ|uTZB_(ug?c>dBezP#N6V8L7D4AD9K5|)6v~EVDJeG%3%2#-Crx-3 zRyK{%1UAL>B+TAbOHjHOpVgTF;Tv5 z2?WJ_@2=65q={>JmGW87+@xUPtvxTbAxXzw?pLjD2)I0$Ye@{LrimP#fBE_xHwdD+ z%>sYUPZ7Xi@`RN!5-9(qxgW#af@sxP+<6&;>Vc9jQoPh3AikdfdsAmm&#c*l-ku(* zo)lNQ-4|Wr`|ZQWbL(o z&B?{%?%~V~fObNT)=eWrcR^D2 zX<#)(5QVa^vYcDWN=?Jkh$$?NvVi>LH}#W@O)!ZqumO8sBC>7Ub3w=U?YJq6^^?TDaL?G*-#?S4Gb_F{9!}6#>`w-p=f?GdpC@7l2D5J zkbjWW9`&j{t}avzJ%NKjWyo8Bf>(MaQG(C<;O3j*=XH>e=O-tpUZl&oCnZfkziVgM28@2J-E z_}clKEcXbO=c+ka{?Wi;hRw__J8f|_MSuFMfg6ZtXN3P*D_%mB5UY8FSX|CakrY*= zK@`rXlp0lEL5!9J>e|0l4P?>}S7`oj=jt1Y{=0s7x_Xg8x4btp5SCr2L1?ik|B==8 z=GMRdKmb|0i$!>o0gOo$qrr!^gZK5x=H!pph<)KH$JwVq8WE`A!2x0$v{sdnfI34H zgIBXedjGiKo1lZI2qqq(V;QsvdBS?(>VFon0tqV3JO+>TQ1FKkk9QouglKMIn$w{d z(u=WdLz_q#Q#jEXmV6Ut?q^8qPLF=_3tHrH`bGizpQA(-&`TWp18U*>9&b?*ElrHeR;w zaIv~#{7B?2m_M*UK$z#!UT9G=YISpJb*wx4zFq`ii>c5lyqQfszu${L{vsRXxD#?RYL$ zY&Czo=nTMKrGaiK%@cd%T&dp^4*z-Am`BWPxu;3RUuNr-{r+<4{)z`oP_z0&0wx@r zgXh-P78p9?WYpAO@@)R#B}QG6Hq<1CXJn3J#|KUc#;3BV+~$6BqfO%()TQb(d(CaF zwZ6(@Nbi`T<@jBbx;-m~*Ofl-w?<-au98sW0{()?E+u&w6(r`jugx&wtR7vx(J8!XXp*1(N>}- zkUh6#F{zJ(bw#Wxc$LexO}2P{@zOU8Tmo_ub``_%Gg*+ni#>n7Q&R;2-QqA8?Uy@x z*H8@&w3PaFjX#c+O5fQavo1jU|Ldb7OtFSX*V@La=EUBrv-IN)#Qz4@3QA^%-Nehn zuE=eU64^JurBiyb;B{Z^gl=4OE0K`00ofe0OkYG)l=`Sb!yN!50ZuiB1WZW;6?*`P zt8Z#cD&X{fz6-h!t}IeSA<_jQ`=bb7Uj#e74dUdJLu^fQXcFzBogQ>F!=B90-Pw=XlM?&-8g1lGel>q7Z9Y~$+1mD*Hy(x$>`XOan>~BGB>`WU z3`n9o7_n@SMc z@}>TOpHpf(9Yc~vkvW1CK+M-{t0c8AeH@cc+A2VH51<)S0n>a-3%(zm$-T*CPraSZ zSR~B~{6HWRaqvzNUzL>PezhioocBOv>%PuraYoWBqX+F~qlm0TgU2ihzu-;Jr_7?C z@@f=!Z7&qLz!AQDVIvQTX`)1YKnku_{3A=@9R@Me^un0~YcbitSUcnMEQaak2c|6~q<0GIBP|X~7)oA|Vb{p^n?LE!IV_lrfXI65<7pO7`D>eYqQx0-I;i`iPXzo~`&VB)3#N7TN z^tdQoO4dZ^T;Oq%f-;{q3x-yaZIj*-UkYaq&3S@(baVk)tCRV0Y7i=TMMP|LeEZeX z>iokA1|9&}mB`4Mk-t_n?n3OK#YI28^y&D($JF-GP|4ATCXdrQ?YE#!SO$O1MhVfs zqiR@Z4w-xsL3xkJVRT`1IiHkc0~p|mxg{Kta}?EmTdE@m2ZW9_{2jH|awNGZ0n1<> zxAi4}vGW|)g#iMo21ICkt}UZC4iJ~&VccHkMRVI=5UQ>^T|6TA6nNpmrsTPM%=%{LqSmexsuzZwU>9LvmZYZ}tCHoy z8&Pgwz6|94G{r;`oA7WN3#q{$1P3}IW&1C9#`<~&Gs8%kEA6A~(HFx%?m=VB1=}@4 zKQhhl+SLdlT;(%%zU5I~O=aMS-WL{L1DDb5e5cVPtiYHF4`}8m-;t02NJ#SBm5%?K zYZUpFf3M1Ty8qYFmHg`%QLoVRrvu>9aOewEf9LYu z`F1hTSO9uhVxU!4JPFnzUm-gztr1PHR)`I|92=5xr1X3kJKxyE2U3&KQ27PilXooz z&!82J6LMs-J%NYy3p}jiK-cR_JD{Ne7C~noFk4tya0f19swMmtpqTzlL6j2DO5Q{b zIG#w}y8O&rH%?oV`u@;;BWUV|3rt|FJZ120GcxX8Vrr@{i0J5Fkdgc2vusamn%f@F zl$>%g%1mf#3OYHZP6361a04I~@CEY}wEBKXzIk((J|Y$fj_31&Es9frk7_W#U`{w+ z7*Pu#)%OuTg@4qb2T~b%+XYsC`xU8cl)&*;pkFmDpf-uY^Oi<10BwMIcmv4hZSSCa z9IV9$=@hV~2lw_ei;t@;v-ToW3i7P>X%-eY>0zE9phz%M%?43_pkv@mz+HhLKgr-( zd5eW!LWfc(9B3v$syz+Y782L-c;1Yt%=(xT89R!GG(? zydTk?3JMGHnEosau1TWO;t~}$?}aFJKm;g3yEItviUXP_e~4$q3bM0ad4Y*M%H=j? z-7ZT2uOL1t-D#$muU@GQ`KYKwbbXc6e;9v{5-DUq|D5007s1}xS-i_P_%5TL3!38X z!^sp*RKlx}wa^crY60!pq`OZJCvl)tl)dqpJI+zi%wXYh+n2kfmm_aaL;X~0V^fD< zo)Di{-w=-(pW|^avzFk8^?f{QVRyiN0S`vOGsLO;op-4`hJu{mh|X-h(&MY+K5{5f zPdabS<@*{(HCKv>=R0_j9LTXfJvLi2|=y=n>x)KuT4)4U7^*Z)? z7o&*4%vkxeyKnwbp~1TP@W_M}G+zmTAvNw!!m~vGma@>?92*IM1X%-QoqIdtG|c%H zL{)XtZj&A*1F3a%LFmnJ?zK6?lo4CT3PG-YT9LMxDgx5>*jjgcS4QY_lJxP!p$Hxv z8cz6`j6QeFE)M1CKH1s;6Kr|7CIi(Su3Q%`c!V9#;lhRwV!WE)1rD?$vvVCW9*9;1 z$oFP2LJJnm!LS}YhuxES**|~s3mUxwF87^nP*2{+LGA-Q4V%a7RnzOw?siVQj(WBN z#)PTnJ=7avz>sgacEwwl0pMi(Yw@}-~Gb1 znhb^I*VO1x9?@RTm{Yfc7ZDU-mDYkd+dbrhZN%Uu#M;q0=-b=eTh++_Vdm39%lgXTNbkwUPx$fMN=ZyPmdMYf`;xdGPx+T8fU<}vEplxYs z(S@)ADAy!9r@UyLpCbgX$%`jsdPf_t!bGG&$;ew4nwpwAz51tPe1U~SN9I7ei(*Tt z>1j{pzDqqP&J&V<66=MuYg~WXM0EeBSoi+{A^-pQKlwsYyQAT=A*H%qReY}G;-!Dm zrMgd*XZ4G7d<^(;@Dk}34;TUVK*xCp@no7PK~i0j^IST3K|v*&B_4Wz+gNUAUs}MpCb)5JmXie z1L}t#DXNhkU8u;ps=A8+MzLl4Sk4|MJ*6bk=^6cYw~#6V*p|);6H>s~eNq(84;Xe` zR>(fDcjQeohgmHC82saP3tIwcqx~a|T`;2d{ReEmib)w)a(Si2bsa?`*-qXea|EGz zG3A-36dHNX#KkF6Y2N2bKBxAdA^RgN{(SoMNe_tUd=qng06GHOZgE-oVj}W91MNzO zOXJ3OUl{bz@LyoeFD&$0&nF+j)QERMw>YbnLRABMPeAtBkIJ!uB@jf9qQJcW0}6sF z&|S4`Pn|YT*(e6cf|?$a|AV);jLNcY*F_&fLK+0=77zg`gYHrg5$R4rKtMo{M!H2( zkw%nm1f|nJq`Rb38fn&f`+eV>bI(2I+;gn4*J8Xs^c5cNxZ=Ey^N36M5kE|{pm46Gni$&$1~OtdI4xWD3n`DayX>dfGq2xrxZ{O)ys<-SYu2u>MEUYQRs}bfF1Yp@PHL(Jl?@zb6)Ctlg%2t4~7aaer=> ziyg-B%4X%#(lkCmL(qZ5Kn<}2{Agt{KnS#nU+ZHZQN9g}KY{_<%5ZSbP#xc{#1Ett(GB%Sf_nzkca_ z1yBM%e>v-M7mxG$sLSYODmiH2!*RZUvR(i%2&Adq_vi;;SQicNz`s;>le`3u8b9w% zqz6Ab`fTpoE)8!Ie9DUPhAa%XFSYSE;Kv1FoRg3|2Z=n4ZU8#Iwq}w8eTzis*+0+F z;Pbt@Z-@L#StJ2))g+_oCD>rYg#WWVTDc0K$mTxx$TMd~Ko~0|R#5Ac<d>j5sD;1Gv>p23Ce)hb5 zzJtI50K6fBSTL6ffNKtm2+OIcxGXDcZfq(Atf0^k4uJA-Bb1kY>W&@&TV)hxLuKW< zBA_!I0s^riArPCPI{FD~Zf!8K%cxrM&xS-gAiZCP$6p~8)#Egv4d_DA_fI^;{Y*3x zqSbO0>|Nm@XA_VlF9EJGH|ax+Z~-2#kO94++WEC$#oia!6@0|^udS6n8<3t?U_m|) zwk==H`UeHFfqsD3#yjdEy7z2D6>%G!VuXj!rK{I>6cSlsdw7wVd3%S5k-$z&7qA_m zG%3L=V}gMkCgi3#S1&1>4HySZKlr9}4V&iWuYK<;pg2VECtyZ8g8*H(M)<8WfwC2# z$;`~c5XT5Nu)DNn#?1T;Y8?1r#3R?&d3 zB5*6!=Y;cCSiqORqFcfjsEA8o2OpOb%ktm^fEgreZsBZJ0nk5^7>9FipW@1@s&_z+ z?(@g~il9|gl*#HI$6e0Qss02 zt4IB>jl*xBIyuRSv%z>&0;(fyV^@U^nOxUzlviZdoFDYS6^$Psz@{j>duT(v<8)X= zhQpi*#-gM!C4i6ufrQ|o{MgJ)0j5PDmmcV!h}0h_4mAb<52^-l^*^#()Ki_aADIK8 zh{To8JL#3Qtn61JbqFQ2zEtWc5FZx@*q!~KGX(GYHt@@|?X(m#a|T-*Br(f0uhs5v zK44}Zt#`U?ZME71#Pb^dTlVpGTB(x%0h%jP{DG~~m1dWN9wfntU>kmopf&T8*- zB^K+?4{Ft-JYYsr6c%0uA9%gA?IjZMmb?!$g_-m7ZRD-Olq^y{#CCY@5pD`$rEo<~ zmn&p~zl$G4(~ES)U*+Orh50xq;(|p&g20Ua!33A{7FQ~5;5&WxDdkiczeUG{aDdnV z?3~h{@0OH|+`A|2+mn9%eBctcAb*KXYd8X6%vBz$-CS2Wm$&gL0bWMvnx>$AelgPG5< z&G$TlQi+YCa0&L6@vgv0(~ zBL?8;04X;JT4f)0+{U}JfzuU5D;5NcR*7RD1=3L`Yk%HS_AXG7&#*ShcfYFYVEyA% zwZ)I*8~z!7;hDV{4So;TaR07K4;Arw3CvXzBKxNKHThl?PN9a zi5z@9WW%K5#QR5j(BPrmem293nueB++s*>PTM>~kee6w_qmU2I&XXG-4aM5ZEXYkE z0|>z_08(oheopc3Ji*3hKz^wb6g-gSiYZy~XL|m-mhnh0tqb|3YJNJAijLBj(ho)r z^dAe}1=ifBL%}t%^lruVf!GCloqs+Vipc@D9yCuc^Z=`E95^;Amg7 za^BFfK5aM`$42RY^1LAM=sa?4V}T)_T5$TC;+?oF*iC+Clmbtf=~>xXc+NzD1jgc^ zgrjfpmSkyzeHZJHl(QZeO%gu+*b8_*%7Pz=&1!GB3>oii?&sAa(J7zzjoa56{6>6QKwyJ+xpS6>F)cPFw#8s6aK>u*<=t^a*rLnvmDB!O=E}LD z{67FjiK7FZ!?s)et1xkp>l6Z@qV<5z4rg_+#TUeSHnO_RQdGxc* zq&L8kxr6Zz!{ZOxkPO2`Lgz%xb1%h_k&PmOn>)Q=Tyaj=x42J1R^#)t^*i6^=FBA? zY=2LAq;_Oqb40L$S62l2V?*q+&xtG_J;J+^%>eGL*W$GA+)1ePvCehdAafigJ=lO* z*s;}eg_t0I(kQ?mok9w_RT+DtH*VB&a|LvS#6Z5Cw8S6rrLlZ=w-wRNsk+iD@4oRo z_N*r)Y?GaVRK(jrVDCx|Z>gzvgM+?wG5ti(x3c-ms?6eYIzWuPZj@VbGM}$CKK1Wv zs`zj55^V^RM+)PtH66`|nS1>L0y{KL6GKskO*2?j#2@7z7WU7QgO&b*AS{Ou(1XZVfB_H9(x1IIoA_gP101e|^f3hfFCd7=4H?Y=Hd z8vDxF#4-rP6$jWnO>k?qo_iO8oFf+n^9U2g)0M*KwLlB#%5iGWMInWHgp)j9{ww|jT*&i=+>_?N;eb(3!BzoKm2>ITY| z$gg+V*{C?LySWEzhl!SOAJ|`swl+;Jeq5V8JyukD-dgwHXsIrscAsrT2cDs!P~_wm zAoja1JUrsqQu6B61AQ9Lh}W;LM$tM=^jY!>KVO-M z^8wBuS2+0*(u+rECM0aEfUPiSL78*b^e?U zrty0j7uT8;V|ZY*;*6+D^{SpIxXWXKsTv9dyeeoY$adYFkrS@%Pz}fp{R`nN8~7e&A6*{nJl|*L+j*(%fZ4110$I!3&SHNu!hMYft|j z4pn0b&RS;hL8vCfkbHA;x`2V&z2J^Sm79VaJuG_&^aa2m%J=g$Kn`Og6>Z?XmXVhi z7If#AU{Th;xBz$HoKEtf_d#0ZB0%rE>l1k1;pi_@QZT^jAIOcelJ{}o-m^3dbqB;X z5q%_$2z9L2`=-U>TnFb8g{M#XUq~l3=M>(6K67=M-Iy+dP&t%rV5Zx?X&L@P?6|cE zEr2BLWAb-IXuiOh1bt@9pq+^ESQiLw!(c$f|Gzb0wG#jk_bAqooDCJ`FJYhK8m)+i z_xgM<)UL#WW?Fa8U^5WdVq&&PvuZ`rLiXutMIBJjHb#O$+QRGng>kO#t3N>XYa%X~ z(eHz1*;P*$pXC<)bJ*Nv{I7vp`rclP;0k`{tCn4_!$Ekl#e3z##HUZckcAck4`Ac- zhbGY*4HF!lPqa{gNx^ZN`-a#6dNtH-^IzeJ6{G@v8juFi!90ls{{7o-) z$=5Pnc-z1?;z|u&$zMXY95-ccr3+o1Uc$I*sro^(xkJ>L-H-#3&KD7kRiB}2!c*v3 zCkLt#PR?7*>ldhRJ`ORF!7_;(azIJKEAIK|s=78_cUefN_}Ud>V0f0e%(`>w(}sXI z*6hcrIpc{O0=8r2gB5j3m=nV#&j?OVF0(N5Qr6f{N~Uzk0HYXESW&2QOXZe`5hvc~ z-<+XKD_tIlr5O;^2vQII`lsZ;XcTvk@KH$(x%K=xqKbf)h^@_YvolvkuFfCCmwo8hZAZf@Rn?B} z3qgQMYb$*~xYOd}p)H~Hv9V@gZGV?;0md@RQ*^%G?_{8ZYpFfN`?WB`wS6YOc=zq! z9T%(?jlt(syUOzdru!$La75XNs7?CADo|?C=id2er=uEI{DYnrTGW0DFZh&R@1;MN z`&$DuNe$HQZ~@a1>J1~|R3lC3g^9XtqEhIlrsC6*kFlU$=Ce6^xVzSKN+aLF+nMNa z+(mc$vr9+Ox}LGE`2pyxlFsLXY5fkU<`))BLF0j6?@my4({5`)g&sse0WV&N3cJ`E zxm-Zpjnw_M(V5uPyt{;rW738hjt~RES_d@E;Nq5qLJg5qq2(W>H2K_F(%N!@mlKSWsJ?S0AI`b`PLxo_&NuNFRTO1H|;2FL{y8_ zC~vpUUX#=C0*x}#*?=1toJ9bJ#Dkv(BG#`a|GDcAHaenW4$rdx z6B&>UL2IWrbPhfd0BO!^mDl^?!20`!7aQV!2(=dpGzW8}hZ@P!N=mu%eJhaE51Q@t!1xhbcHKpFVnE5`(_PPPZ{|Lbe{>f1wz3?X|fUh$~9+I?4IoZ^b;E{8%HXs=P-$g=w)^Ss*k7}QzdaO!?SK<@Bc#2U?c$% z2n^x=FOWxr!9vo0FDc-oeK|3&)6LQ3q?QW*8!u!M1zg03um=KG*7Ca)Nb)ta>OyM* zI{W(tLytqEiy3k*bT+?`3pW{UqJ|d%7`Q*!^6>IMCVPKQ^&x@v%9Vkty#9#I9rlbb zFK+9LQIHW5kicXq-=)fBoErJvnN@Xr@I-?a1-7}O)*S~-jRrDr4( zKTiYVGt8c;jrMqJ!O>f3IVRIhdKo0w!g zJo^gKeH!^gq`}|ci}*~wgJUglyRYu8tjkHqT(FB6Q7&b__e#sho?q~07Wgxgkn{}L z3@PbprR69mO93pF4Ir+1bZV+O<0ZA}o5pxwBhm`F4N=R5j*@=4t zl+-!6*Gc;RgmD#EO}>!4qoQIqx)m6n7_zkE^|IpOqclNrFrH$TF%!V80Z$_sp;yJ3 zVoFuD8pI;E85D?Ky;_DVF}E8E#3I;YV$O4Ud!}{p7w5J{6F^4YTy7u0$&g+tJUU7S zW1}RV1X=(bVkxkIDBV?vUe-w~I6T8Z^#jl>jy!_=$XEtylas<5NQUMXSj_$SI z?q9xCZjBco9}lAx9v5;Ik5!x_cZ)dv4tB9LH9^xNO%e#F=zrk6b_QPVNw1pY4tI*G z4du}J9-T=e0~=`2%D_}q$S%xRJtm}OWzIaCl(Aacj3NDmc_EmeAV)fJ_0|*q4vBT& z(+(9@fx9-G92~|`?JdAp8D)d}J>hG5JkW5{vrhGVrm2 zJ|^k1b){r%F)9AB{m)24$#9C#COe-&(zZ*<%EM*d^^woW7&NGV8h;TFFVx;xcFc7R z^rEw+?73NmVpPd4>0ZY~#3Vh?PW;mSC9cL%D_D@Pg2|E$R-c7W0G1*%d{)BRT#<~$ zX2_1w;B^1N^6AyI_SLD6*NRSo36nGn=cFre7fbT872rMG+ar(-XE@={UA?XLf!;A{ zHmoZ^@mCOMISEnxbn`YmPngBA)fsK6x0Rk5&@Qi!akZb5-*REjZ2hhk(lW0;Gk@*D zn4oQ|BLB@*N#JaDio+&WwlNwBj$5z2EE5Kmee?|&SP2G3*dwE}RTz~!xe#gYRkBL9 zH=|pa2%TBBGP>^^9hjo>8p!^5-7q9L*S4T=bLe{_B-gyCrw1}DzDgvG05uYrF?Q1q zQGgd1mPf_Jq@@u-`RFiObpdr-_;U;F&IX%H;u~3OpoyLV91-H2UvbQ?^b)`; zRIK&9fUx0j3CN8Ho1C`-70ioBxQ#B5Sl34%R5>NdIhmskuw580o2&FiL1txB2cuwkq57bP)-F9A%=wpw{uSMN>FY6+}8>6rI@oRB0O)@`q z=uB9kp!)Mp$NACf71T7WK}kX$Jion?d>+~jMqFjN#&VA^AsEwScNkYu<*p1=F*9VO zBxPLqzn>K0;9iWgVsvSGA4!Y?KHf&Qc^^3uWlk6Q5- z0{r~lWpa4S7PrnhMA#L^NAgbB3z>mYBlu(-2gWoCrAK2UD<8V}hm5p1Jx$Br{E>y( zd6Ntyvwm23N9&f^yFsZpln>N30xhudkSH23;JNL&O9_Yg^rA`qf4)rX?sTblUU%+< zX~Y$L{=36;SLmpRu0wb#A0LwnP3PCIF^Y=$(rRjI%%yShJ5<*q?h6Ak0XWJu`M%>RQZi1T=SO7c1<-GiOZkY4}J1z*|}z!1nUl z!BhyxI13KqzQ+E$|G<;pG@8CO&2{4!Y23H2d^_xiil>X^QaS|M*V*5}S4fTYGA#Ett)pQ=(gTHMP6g zVZs|3Fn|iR)E7G;(p0Z)Iijeqh{s^2#)&WV#ZKq6S%lUY*K|K?b&4P|H7>(xnE0a zc<&ye@4?miKG&bC)B-U$xz^E}OKQU98}L)G;s}f}lbJyw?LT&pCI}0ik{Tzy#V&Zd zd%oIx2jl}EwS2e;FJhs<c@wt3&$ zA&-+cOC1H7zrJH1&rb@(<>jS;UHI!T{4(hgT$`EB@VD!47J6+5E=pKiO>f4SkscE< z2w^dsdj9xTA!nna5}+Utt5Qz);NXi4KSVQxqeFou9*s>9`R!JnkMtW#bd&-%HZcjj z#tguQ??n;bNKm7D`m{`vTR!r!-bUJo{J(RD*kguu=QGInI#l0EV&jDqJSEXjSVcU&@%#cn!7t~9)!^e^9xvOr+ zJ~Iet4hdRIG4%&#x(Ld*Ufx@9I~B%&0OH2hVVtq54RiCk zg}E0UeRBR-d&ro4@?^rq%Gein1aM^`gS{vSu~6e>xD}>dNo|ukG}{$QQZOtefeeshzILidyh4i>>05$xfz#Zw_bBQrjFxOFWC z1^^sQquYVZZH_^QTGavXV!LmPh)poLsK*|r|^{~*r z-io=>cSLn$T7*+nU%DJ!vZ;tN7LLucU4wyPC-O{6U7ZYeh+*P#+^5@5(*_pV2?%!A zPifE-FG6p)>q>W*;&%N3dgCfjk#NxL0_&Q4Ed!K4ompvJ*= zYPh$Yg2+g6M7hb{QG{ykqm^&o4^dTY#YxM!AzUoThY4*90uz{I@aO{!*!5s_u5Rbg z%Zjcgi%ZJ8(5Z`>g@$eVaYYo}6=sgK=0y}k(zOVsp8KKtCG;ysmV2F zWN^dInK4>G?Hbzt(?Jmin$_-9uJ;GlGuS>S;~txr6rG_Vd^J~tl@{>AoIi5w|dVBo#ljuQXis;7mKheE>lYQ6WU zAO9HObuPJe;u(yA(yg-ctFIF)wfQMNw@{xSC%NP>W-^vTpj|77Im@ODE|hmleh-tx zvPYf+eh&1h;p!*c=>7@Jx6OsNCmVt(g8g5nOFFx{VxT(T@X-la{%o4dMT5z8;bY28 z&A?!jZ@W~8?a*7=&zjb=46*FfUf$vfTm(}7SOne0_V&bTm3T##%4z5@-w<>5W)@8; z0~v$nG$ZNaRC+KDlAX&Nxb3h>{7^$<>$8^8K{WMMzPX97?bHC%1-@SH*m#-rI*d5& zWyrm%n>U2t3-Uem`<#pi=nmm6twr7M2ec_7V^Zd(5%IrbU?F??$&7w^)&DsjI-k!k zQ)gGF_#@A>ES`UC4V}KZNbawue8CF{lhHFB6sV8QA$a__s(J7SO$jeA4uFr+S@N7q z3&Z4O?UA`;KAyP$Hc&pOtPX0IS)ny+$~XF~VZwr4VyS`ospeU6j96tPl-c|bzsElU z?lO2@+Dh*ZXXicWHqe=;hO=$DXz!Bbd-Bb#__OuR~EFUkN@WH~l^^ z^?G&*yD1q*+8-gi)h9QSLD}LC=sY3N77*NRr$}20zMQKa=!zdrB985~!3PiZemB34 zMKE?z{)@k2)W8qzo*vEdGyC&A_x^+po%0QykE7d#qaXedGeGppdb!2X$wH{LI-6G& z{`Ws-$-^Zl0;sKRRuAH1YWR$H!UVgu%gibb|dRE}rJhfk@?C1M*sd(+OYi+D{w+hzG(7 zd>!tW;9}za{kj_o*|dM}oFaj`g^if&A73|uR2boDB43a3lO@562Knlzy!o$>$mN5a zh33Cs56u3U%$MHPl;QH`Emim}YtDzo6g8ff~(*5X>S@i}7 zAdc`KCb-`er#1iu0G6SUk=l3id!C-BybhZs#n~F`Uw{7W$d<_Y1zpR|`hB*A5beq% zqoH@O6_T;3_(^iSh_S}b&dC`F9nZ5hKRi?PIiZA2K!4U9b|4d(PD{&Uy|#@%%DYFA z)3^asx&#U5PDq!)a!0c@RWuOTJywsaUgYJiG`M?9oIBzGcf!$8DRO0bb-770{XD)R zPYeJRGce8~Rdt}_7D=znS!gKw-geh&NLq%AW1Y&h#~ za*YKDJdL(K`3GQKSK2NP1A5c7A&Nr!w!>=+BQ)HxAtcf&;x{ zcNXU6P$;$t1`KkGdHYO^@6kU$h8a;LoC{Cw;RJ+##=dDk0$$i3&lxM6cVE7Aqe3~% zE%7-!trMntdjLw?v`zjCIY_J|BqlsdP4F}NAj4$XI*hDe_zvXl$f3)YbI06H= zXIQF?0q-j5z`CGmd>Dw#zHqcgp#b-lWW1zQc>G?S5}`OGMA3t&7%$qeu89oCWFS}I zW~=S14?B<#aBy(c)cxfWg=UpDFc6UQDyU-kEP8wCO+uot<44F^*_%e;#^!*_b+V8l z^b0P8Q)FdDin$%F_f#g+Kny1u3K8?^fBu7s+TC6Ahh%c}52BbS&~KIT^fa*@L|a$s z{Iw`SasBKlT!_BTo8_(kI3gJu7P;;Beu}_quuS4+971Pr@Pf{5ueh|6@^KY3&~hZ_L;Up48jCBE4n zaOy&V=adhEWCJW399nZlc7bkdZdsieSv>p=E|;bN1P$7!>i%b%jKpCjZ0?}8(q|$L zbJ*M@K^dD$GhWK@*I?(u*M9-i9@RL0h-YFqBqtpO@2^1EC1H4RZYD}D+&_D8-idE_ zZN$lHdFdBK6qC9j4;r9FQSa?+T`I{vKGM?1Hx{_H1CW8bykr97@WFBy8P3-42@x(G z>gI!mzJo$58`w^le)r&s`Vj-d=XkUx1jn7=^5g5>0)q0YKgB_B*!-n1e^uWAaK4ZB z=Tf(8s{P@q03;4shMLm%Tc=wT+Vcbu*yhY$~sV*4Au8`2lT5 z`ao{zOiO5-fhv_OXRK13Po9Ty>(KD?W33GWCO8dGD7F)BdwH+LAqp&4?c zH+ml9;aw;|9_wj5mOnU*y>BiBNNE5w-c#piS$ZFz^qu(!11yB#e_c^sz$I3Oe+H!1 zWF-hdJNPqTJ_ra8pNXL2?OlFX`|KMT0jBM5>j?}xICy1^ZG(We4x7BdRA#sp1NZ4z z?TMgE6XWiRg~xzrS~sR=07U#}QVhP+yr(AtfzF}Zk4+rIKTN#_+jNjhQAxc%QEl7r6-zD#6Y1|JDB zk{+1A=BAcPAo|M)Pn@ypItfHDdtewo)Yew)FjGa)Z~x$^?tHCc@Z#?tj)1t(kPwcm zhfKHuY`d%WUdv0RvlUR_m6bZBM^)`ZGNzoY>|az)Oy=qOZ>iA4;`?{*a1zjp!eNN} z16WOPp2ODE4UTLGnwlc9(fd8I7?GO0@9i@~2LtM>oMAHduK;pUQ78Anp)w?73DN@) zkW)S9NBYiX9t6$LRZvysCl2HIGx6D{LJiQ{LW7S|(nyxb2RN=VGk<~*8RUVMSL6`z zC0kjvaxyamXwC*us&2*Y5E%LjXypqpVY#6bqrc|DMNG2%@fK6PAISx*3wzMn=d2H%|6$M|7wnKQye~!0SvA_6mBS-iO>7zaalR7;?^A4`?S2cX$8;?<;QD zvzQxf`-@!d3h-B^J1t(n4-~6H-MR!AhtBIq1kA6B2*f@1-{J=_1F|0rbXWP~eFnyz zsGG`Dl~7Ih_V%!ZM^zypE7HD!b?+-6uduOZS6L5-#I%y(ST|>1jZ4@MS@w{l&=j764SDFIF2W=v@rjA*3VI$e+Ame{ z!&8UK9(e`U%S(K8tGzERkT$^;Rzu5g4_WD`UcUyjgJ$RU=w3kRIo8#LRZ@jh5V*Rm zjGvi#WqlwGU}tEL6va$xTp>=IYy=|z7fFuH?)hur@Bp!9$U6kWdJ@0SB>7OjBI81* zks^8qZk_k(o2#*K?-ee4)>D4|;35HsNCG-a?qPxHahWW>L7Aqel$oHV9X(�c5h>hd zTspUaGzM$KBWGZcgLZv+iRuBE)utFcvYz=U-SFkZ4N@1d^khjR@kW|DZv}jq;KP(y z4UwLi!DR*a$zt4&I=)gE}2kK?HJ+!e&?PBHJ>sPNvtGRw=_Mu}5<*o^4UEeD-=k4gcB2S_qFrZ6$HD^oY#8a&6?S+i2j^$O(9)w7 z_Qb?EdY+M{#Q>qjbvCFH&9j#b$utp@~r2VXLI zU>hR=mB8gk>O6|z7JQX3uwkhQi%;aj4a`^>q06_jXqN zE7eFiz_Aj@8<0n?W;<32o{swZ`m0V_=^cxF4aEkZ{Q%a!{mjL9_bNAcAe`&7-%At; zX6w^=PAZJ3ri!fz-jLinrOvgGO23#8|LVH<6VA$Y|L9viym7Uof=eX2M^3bR>w4Wd1J*DWC zs*}QBySmz;&kmYD?Oc6O0HC4jTY{x@bg00qybTy`eh~7-$=@_1xAU5C&S`FLfuF7} zfd27gj836Y-CB}Di z2`4{U73H*_6oph8*45>4C4=TZfm>Jc`sYK15h0 z`GsTa#9MlXe0OIha;?C2O~MUFHsG$~R?=6Hp6&s8Cf>Dh+$(;`$fud2`c-vvlHv8Veg+wzWTKH%M`(WXd6jQO^G)3%=n&Xn0)^hDsn?SEqFK!duwsRquT8%!q*_VoaPHB zj9;K#=xJ)M?TQWYkv5Q_1M7tk73V+3K6cu}>3q|zYLnqc*@K#}kCfnjF%8JN8_@w> zv08K2swcA{h z=y~xj&(09jQ(VMzasr8etADmRtqA?G&d7{;p|YvbDf1%J89J zaNF7EwzVtx+x9?50h$RXRxmWDeJk!op?;q9(2Ppr;y*FB$=c`PX@Y`D>qEr?>3N*q zzAtkK@;-c+UT62hKwTog%pxw1)-FG_e@tXFimk*Vv$Eg0t$~;Z=vExya_fbqVSzbH z(_|5uaztc_pt(@h#w8+)2KT)%9TM?pyBwQtpWF`aKW*Cs&*AE7kH*$$NmFyJR3Gs_aLtl6%K1m+;F+=UD6;_!j{YZK8J*lqR~^gnxJq}bYZzbhAu zi*NgC*0$*osG>14Tq1>G>3k7FBQH7}@28~=bbL|7fdMP(vn4H^2G_=gr*|mVr~jr< z_ZNVy5T>VWdJg7}lClxSLT{^}uNAuag+tJ8^wQ|4F0UXl(ynCs-rTsj#TsOm53Gs* z|1ouQ(=B^-1BQo}`KfPO`JSAhFGLEVpzm2pSZEv*#P9?Yk~%1|6pm)1ts9E(SA+(I1sAzh^zcpC2Rwk6OPQHTWrB`PJFO&$Nr2mptbq7p zgmnBkK)8|e*4k@!?Ys zGoZRZ#JF_{TJBH)#0u@=cLDUUS~BgoDxHWs@PaRQNb?J6cqOEoPMA$+T6^7=DX~W zUsf7hNPx7QRa8_yiUNK|x?*%L>z(6*D-f>Bj)F z(H2)00(F^E#PSkkFhnW&D9hYpR{;4Llc4Pdn3_!de3suvhqXa!G4Vh*#mJ2iohes&bDha7?qY#wf z&r1_Z{$ZNKjUAFpv==*y2&WxJXF6+kjMfrIAN(CVwzdT*6tbO7i8p}J3Kg924mtNr zP6Q`(PL;Mv(PC9^( z;B9rziHihO4;F_ucM3W20pS~5vRWQ35!*(LL>w_JFpfd(v#w_f@`i{|A7x~*!^ebV zGNv~=oa4 zXiD6yaoSaqAp0oJc%qbrO_ln5;5{$wtFX}PVq$U2Ied9KSLZ{-O^D4T<+t?vHx?@C zh8frbr}}?Y#{lRE?}>Omta$EJU;OR5ab0G0d>03)FdNw zXkdm0pV`r+5AgQD|AmCU^x_Kj3hCJisr%k8Vu;ABRoL$5ADj;IhpDGmKR+-x)^e!~x;`oHV!|t#nq|XoOwl6M?j}RhVDg0<)bHC`v zZB;a4=+q+)0oq&E=j8nU#ba6*$G_XLC6;pA0{2)6zbQH9k^3-ddFob*pxu4-a`LJt zlfHp|Qj}onPkrNo)UA%9B;F^=DF?H6KNJz=;ZY$?j8}4a#8v$0XJ+h;-IBgH@|nAT zRC-L%v~EX#xzDYYdWD@`MnU0y(cac#1i59s@(*;4r!+J^7>YBOCue8#|?N z*hFk@1~Q!X(EouMpUDklW(Fzl2(3h$vOFWFO;$$_uFS|o|7@lQNzT z^DC7<6P9<9*>hB0?~d?uO<}KHk|%h+zxKS+0tnPzJ?NhqQqIGoK`H{KMmlyi3EX;~ z@kFs+cb-5}y_3W_AMJ$;fz)-STlU*xtLI{?ntJsbG(QkO^qxhDgPel#&0r6!qvGa& zhs76yj|u;YoPYlq{C^!I|0jO`UxV)fabO7OV?(|Yr+Wrm_couV_VZiAHf%wAfC)GL z9|X}!$Y9@aL;v+rGwTk(oh?C0Ka98@f&Cp2it{ld>j_-uu_+RYk?4|E+sQCk`Qe0v z4?W}i8XBisKQ|U72%%>JoT6McF)uMO*bN$9X;!kIZker(mTS%6zR$A2t{Ch;8FHzC z($QmoUE!yZyK!$osnx*BqHv5}d?c5t?&*m@L7}j>&5b-(TMJew73wRgr)UR>xFjxa zPwD!txTVCPp&VS6gJ4viA019L_!#B8XPtVkYR48UxM?rl60(Q(AIiA-fDa1CGdAIq zp{12>Dg!!UADbI6vd@SPiWC?D#zb~4$R0Sw&Ulv<9MJk5=+^f zc(SCVtPRjL=z#EoMM^Y+bKoFoxG}4=2bq&{lHdhkymW5CnU+htf)c_;oBHgdizdcF z$V`Z8gw%)W(K<*Scn=mSZ$-%^=dVTtU%g5>GmB?dPKaQ#66d9ETSG>jTT6R1zW|5^ zznaF$CS1|eeTVhmez5%Io(g!ib48@^>Uesa2}IBffZ5&O)6b)p{^pB7b#>eO(|&qP z#JZ9SP4iT*BPqs92+ID)$Q!Z6lCWU`DmQdoNT13_Q$>TFCkpjs>W~a!fI~i#awj(v z|3(P#23~|9$obl*kag#$jvas?zN;!n_q&g<)TgKCF2bMugQHO>GBV)1(-jDtm`syG zS%ZdJ=rY2w`OHK56k(SPDDk5?z>jCPNj+HJ`g0W@_}$8QNrBL6K&adRvVa>J_Rq8i z<8M%aoBcis1cEq$42!yQsxVzs#&>uEU2!q(sv@rIQbSLkDC;GEHDD@GY#4^A_tWdv-nW>UIXfVpG$Z|*ieB45gq;q8Y|Ec|jr|0ARDKpF*S4VjQeoFe{m?K9;8)Y( z%^2m^e6wt(;Ijq_aB-MCnH2^2K;vvq+KNS?sHv%e(FoP3-6&?Ep3eyHd4;deCjBr& zsRB{+#1HcFMV`iWqMcp1@dK}EW7aETttZdR`JOks6-*Jgu=1id!6DiWESMo3z@+Eo zd6Jl!Fc|NJ`N;%jWXp=SA$nn=ddakYT3#y?&**y@IEyBk@aS`c+h0b

    h=Z?t@m!6C?~4aY*XFUQk^=_?x8X&01#H;@e1AA| zo}6_2(q0Rt{(8Z8x|{d*ZRhr+9u}l91?D8BUP3TW8{-DraD@epd=$m1t(jdQC;+Cx zd<50Xa#QK+SGTR1>!LnP^-XYkqXrIxv5lF5nHT49oht*7yaQEeGE1M&3TJ2Mn6wuN zb#8%Uhw)gH0VSU;P6CfkKu2_%G1Pjb{CPWl;CY0sC_pi21MVh5qk^;qv**KiP>ntQ z$LZrGOjsU7tvLiesMfD7#0*AA9rnv4W0U?s1SlZkAiv;SoTzmXL0CRK25Y6J&v717 z=Q5h>Xh(YS+BY#nZ69jnUd{4!{wWToJEf32J_w?=ny>{(F4Y2+kq3uJ59>Q2MG`eH z8T3>&SZeswRe&Gw z0Ia>xtAQYDKB6)XwR zvl88BMP3T*0TuS=XyDaS<3U74{rZ8UW9dWp0(g4Pdt(A)fcUVI3EYKuVX_ane9kjb zwD^RCUeL}e0+GJTcC3pQ)O3u%TZgtA3-j{HCHm4Zz=svzJ_5A9!ff%0<#FF_R_3RU z=YgL*+#bMCD7S|TKydt@Mz!FK3>_&!kST@kxbOX8BTT#=F99-$va%0!s*M$L%v*cQ z&=k-NPo?%@ zkpP<;OG;b^EDl3w3CH*|s^FoQ4r9VuCHdP)_=sI|@AIQy|Gg7TD?4peOI z?N*}|S3!&j;POQ%eY$5HpdX4t5x=IKg)xiN28gXBED6RZ+CJa~qV#Sy{+d?LNEabw z)R^s8!U2DsfPhtA1|Z(1z5$D>f&SKTy`3GKhJ$M$QUE(-qWq|Y2rpRCN~+4t*H(ex zYLmr`P!S=iYqtAW)^L^CWmNh>i^DG~JQP$cE5`3!OK5qVrR7;#qx{C z`Vx)J+bmlX*E3H<(g?HuEG_FS9~k-6VKUOMr2d1nEur;4A#DdhOEhJr2#&+0kap41 zxoG*nN85(xfk9!EP)&_8KVwGl8|v63|n}Z<{q{XVh{eR(YSQnEL9|q1p8EDaL7(?|)tIfT z$ZP|Fad?hOvSQq!pnUdL?CV^de62th_X(_k!U)M7X%iUA7cVKvNN|gMN>%B%R?Vl)FVC7i8aLeb zR&JQ8zLov5&5Izok*k9b^ATs5nAe9ny=R@Bop)}lIMk~UOEyeU577$;cp?min1+R49w~vA@ zTkVf;uN0f7YGWeqrn^gwxRwQavFpvv*L4JjUoQ2uz&z==*EoNG*Qv`QpHp@bwKb*; z2p8wq6D1jpd4^X54IQiR!^03#re_g&5lG9-W}N|{2Ro)(<>Bzj@)jVCDC5z?>WBOj|_1=x+cpzU(nVtV%C^TF~LrfOghD zk2qA>VWpD_%sBav5x5i(`5BKbfJfD~WjX_#X%CGb%z$UN}&Neo+Ks|Q%|A9j9 z%Kslmp`ScJSuL z!y#!UmG)M(8yeTqd763X&F)g_G4T-6+ahVsXwe(3UQAMYq_SCy#Y4~(RaEPt4ymW5 z(`vh1vDojwcYD*lUf$$=?tj1hf8XEd`~AkE4g`X=gy=hKn;5zcF)2_*1yg+R-gk8| z0?U1{)`{x7^;W3-42lkrSJqG)pnBc3$aV(r6RIth+f&PcL3L|tLaikz9gd_-9N)Q3 zP~@0H9#;J1*ykbm`GsDT9p;k#i6y2M%=n{DRi(}R8kzgut=BD_JM{i0xe?rniQv23 z3AJR&>v*BMp$HUlEymhn5s)azAC^cFzzX<>8uC!BFCVsGzpZu&0alqeQ+u`xi%e~w-?t!a$(=8})$MBU>Z zJnI85tBZ|q<$G@bSqZzEbjs7ALW zq#np{lxbXEAASkaF>!^y7Z6$EF2I%tz_+sQHfeO(gQn-V8i1)1q+%54VFs8PG2GG8 zk{aM=+xpXB0e{x!4uTLZXSCubT$4K8_E_3XzN3aFKR|9z%SeRAd z2C5#bnC^>?+L^;sDDxigma-K@iaV^aBh>BL-IosDt|X!uX*ApWxx#5 zux~QqYc?hZdS`@<(FJ~M^=92o1&%-I$sY#?iT~onOF2|m zzGlcND1iVbD&xw~^0>q~z~Y61hl0bav%@$iC=71(eIN8ZL94{7!d29(>oEW91FZC~ zB6!n4OYgI5D@{mVw&`qE56pWc+Mj26MtQCXs)U?h4cmj2Qq_~gc~QG^ODLRHKc0Vk zhuoD^L^0P)G0W0AHYouC`=9|kvw4(P1=I0(SXe3NXR&+}!v1>iKeSU|r78t`G~*|d zhF6ImXM+m%#Oj`3$!J3gAvY1#7Be<8qX7%wD0#Q~{UjW($oB*00&)+W0Xwnj&U;e? zTO67!58hqw%Na6|)%3$UGp2=Pne#~XDKO zZvz+8w^s8O|3eq=RJUb3{k0tMrJWqT1MC*6m9>Q1bsHD98)FFh;ZuUrXS@Fp?S#5W z!+x1;xGSY-PCUZvX93OuhMJqo7)_OD*FC`y?ISAHwxgqN7jFit_^mCqTCdihFNiiw zjgZC>1bgJY34xbOyp@Fo%;)oQU%rZi%3lYQ@gF%w8SY*%`x;b%Xh+|;x g_5GZ=X$%sSWJJonHoD%XHz4piaqJYcB_yu+Z?x3w2mk;8 literal 0 HcmV?d00001 diff --git a/infra/charts/feast/requirements.lock b/infra/charts/feast/requirements.lock index e441790dc76..c6ff995bacc 100644 --- a/infra/charts/feast/requirements.lock +++ b/infra/charts/feast/requirements.lock @@ -1,6 +1,30 @@ dependencies: -- name: common - repository: https://kubernetes-charts-incubator.storage.googleapis.com - version: 0.0.5 -digest: sha256:935bfb09e9ed90ff800826a7df21adaabe3225511c3ad78df44e1a5a60e93f14 -generated: 2019-12-10T14:47:49.57569Z +- name: feast-core + repository: "" + version: 0.5.0-alpha.1 +- name: feast-serving + repository: "" + version: 0.5.0-alpha.1 +- name: feast-serving + repository: "" + version: 0.5.0-alpha.1 +- name: postgresql + repository: https://kubernetes-charts.storage.googleapis.com/ + version: 8.6.1 +- name: kafka + repository: https://kubernetes-charts-incubator.storage.googleapis.com/ + version: 0.20.8 +- name: redis + repository: https://kubernetes-charts.storage.googleapis.com/ + version: 10.5.6 +- name: prometheus-statsd-exporter + repository: "" + version: 0.1.2 +- name: prometheus + repository: https://kubernetes-charts.storage.googleapis.com/ + version: 11.0.2 +- name: grafana + repository: https://kubernetes-charts.storage.googleapis.com/ + version: 5.0.5 +digest: sha256:e325439384ef9b45428fbeafe8f1e230b331d4b5482c3f26f07b71cecae06c22 +generated: "2020-05-02T15:00:45.4365217+08:00" diff --git a/infra/charts/feast/requirements.yaml b/infra/charts/feast/requirements.yaml index 1fa1826965a..7de78705752 100644 --- a/infra/charts/feast/requirements.yaml +++ b/infra/charts/feast/requirements.yaml @@ -1,12 +1,35 @@ dependencies: - name: feast-core - version: 0.4.4 + version: 0.5.0-alpha.1 condition: feast-core.enabled - name: feast-serving - alias: feast-serving-batch - version: 0.4.4 - condition: feast-serving-batch.enabled + alias: feast-online-serving + version: 0.5.0-alpha.1 + condition: feast-online-serving.enabled - name: feast-serving - alias: feast-serving-online - version: 0.4.4 - condition: feast-serving-online.enabled \ No newline at end of file + alias: feast-batch-serving + version: 0.5.0-alpha.1 + condition: feast-batch-serving.enabled +- name: postgresql + version: 8.6.1 + repository: https://kubernetes-charts.storage.googleapis.com/ + condition: postgresql.enabled +- name: kafka + version: 0.20.8 + repository: https://kubernetes-charts-incubator.storage.googleapis.com/ + condition: kafka.enabled +- name: redis + version: 10.5.6 + repository: https://kubernetes-charts.storage.googleapis.com/ + condition: redis.enabled +- name: prometheus-statsd-exporter + version: 0.1.2 + condition: prometheus-statsd-exporter.enabled +- name: prometheus + version: 11.0.2 + repository: https://kubernetes-charts.storage.googleapis.com/ + condition: prometheus.enabled +- name: grafana + version: 5.0.5 + repository: https://kubernetes-charts.storage.googleapis.com/ + condition: grafana.enabled diff --git a/infra/charts/feast/templates/tests/test-feast-batch-serving.yaml b/infra/charts/feast/templates/tests/test-feast-batch-serving.yaml new file mode 100644 index 00000000000..54173021d3b --- /dev/null +++ b/infra/charts/feast/templates/tests/test-feast-batch-serving.yaml @@ -0,0 +1,116 @@ +{{- if and (index .Values "feast-core" "enabled") (index .Values "feast-batch-serving" "enabled") }} + +apiVersion: v1 +kind: Pod +metadata: + name: "{{ .Release.Name }}-feast-batch-serving-test" + annotations: + "helm.sh/hook": test-success + namespace: {{ .Release.Namespace }} +spec: + containers: + - name: main + image: python:3.7 + command: + - bash + - -c + - | + pip install -U feast==0.4.* + + cat < featureset.yaml + kind: feature_set + spec: + name: customer_transactions + entities: + - name: customer_id + valueType: INT64 + features: + - name: daily_transactions + valueType: FLOAT + - name: total_transactions + valueType: FLOAT + maxAge: 3600s + EOF + + python < featureset.yaml + kind: feature_set + spec: + name: customer_transactions + entities: + - name: customer_id + valueType: INT64 + features: + - name: daily_transactions + valueType: FLOAT + - name: total_transactions + valueType: FLOAT + maxAge: 3600s + EOF + + python < + dataset_id: + staging_location: gs:///feast-staging-location + initial_retry_delay_seconds: 3 + total_timeout_seconds: 21600 + subscriptions: + - name: "*" + project: "*" + version: "*" + +postgresql: + existingSecret: feast-postgresql diff --git a/infra/charts/feast/values-dataflow-runner.yaml b/infra/charts/feast/values-dataflow-runner.yaml new file mode 100644 index 00000000000..0469a6349e2 --- /dev/null +++ b/infra/charts/feast/values-dataflow-runner.yaml @@ -0,0 +1,113 @@ +# values-dataflow-runner.yaml +feast-core: + gcpServiceAccount: + enabled: true + postgresql: + existingSecret: feast-postgresql + application-override.yaml: + feast: + stream: + options: + bootstrapServers: + jobs: + active_runner: dataflow + metrics: + host: + runners: + - name: dataflow + type: DataflowRunner + options: + project: + region: + zone: + tempLocation: + network: + subnetwork: + maxNumWorkers: 1 + autoscalingAlgorithm: THROUGHPUT_BASED + usePublicIps: false + workerMachineType: n1-standard-1 + deadLetterTableSpec: + +feast-online-serving: + application-override.yaml: + feast: + stores: + - name: online + type: REDIS + config: + host: + port: 6379 + subscriptions: + - name: "*" + project: "*" + version: "*" + +feast-batch-serving: + enabled: true + gcpServiceAccount: + enabled: true + application-override.yaml: + feast: + active_store: historical + stores: + - name: historical + type: BIGQUERY + config: + project_id: + dataset_id: + staging_location: gs:///feast-staging-location + initial_retry_delay_seconds: 3 + total_timeout_seconds: 21600 + subscriptions: + - name: "*" + project: "*" + version: "*" + +postgresql: + existingSecret: feast-postgresql + +kafka: + external: + enabled: true + type: LoadBalancer + annotations: + cloud.google.com/load-balancer-type: Internal + loadBalancerSourceRanges: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + firstListenerPort: 31090 + loadBalancerIP: + - + - + - + configurationOverrides: + "advertised.listeners": |- + EXTERNAL://${LOAD_BALANCER_IP}:31090 + "listener.security.protocol.map": |- + PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT + "log.retention.hours": 1 + +redis: + master: + service: + type: LoadBalancer + loadBalancerIP: + annotations: + cloud.google.com/load-balancer-type: Internal + loadBalancerSourceRanges: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + +prometheus-statsd-exporter: + service: + type: LoadBalancer + annotations: + cloud.google.com/load-balancer-type: Internal + loadBalancerSourceRanges: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + loadBalancerIP: diff --git a/infra/charts/feast/values-demo.yaml b/infra/charts/feast/values-demo.yaml deleted file mode 100644 index 2cb5ccbe741..00000000000 --- a/infra/charts/feast/values-demo.yaml +++ /dev/null @@ -1,84 +0,0 @@ -# The following are values for installing Feast for demonstration purpose: -# - Persistence is disabled since for demo purpose data is not expected -# to be durable -# - Only online serving (no batch serving) is installed to remove dependency -# on Google Cloud services. Batch serving requires BigQuery dependency. -# - Replace all occurrences of "feast.example.com" with the domain name or -# external IP pointing to your cluster -# - -feast-core: - enabled: true - - gcpServiceAccount: - useExistingSecret: false - - service: - type: NodePort - grpc: - nodePort: 32090 - - - resources: - requests: - cpu: 250m - memory: 256Mi - - postgresql: - persistence: - enabled: false - - - kafka: - enabled: true - persistence: - enabled: false - external: - enabled: true - type: NodePort - domain: feast.example.com - configurationOverrides: - "advertised.listeners": |- - EXTERNAL://feast.example.com:$((31090 + ${KAFKA_BROKER_ID})) - "listener.security.protocol.map": |- - PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT - - application.yaml: - feast: - stream: - options: - bootstrapServers: feast.example.com:31090 - -feast-serving-online: - enabled: true - redis: - enabled: true - - service: - type: NodePort - grpc: - nodePort: 32091 - - store.yaml: - name: redis - type: REDIS - subscriptions: - - name: "*" - project: "*" - version: "*" - -feast-serving-batch: -# enabled: false - enabled: true - store.yaml: - name: bigquery - type: BIGQUERY - bigquery_config: - project_id: PROJECT_ID - dataset_id: DATASET_ID - subscriptions: - - project: "*" - name: "*" - version: "*" - redis: - enabled: false \ No newline at end of file diff --git a/infra/charts/feast/values-external-store.yaml b/infra/charts/feast/values-external-store.yaml deleted file mode 100644 index d012bcec56c..00000000000 --- a/infra/charts/feast/values-external-store.yaml +++ /dev/null @@ -1,5 +0,0 @@ -# TODO @dheryanto -# -# The following are sample values for installing Feast without setting up -# Kafka and Redis stores. In other words, using Feast with external stream -# source and stores. diff --git a/infra/charts/feast/values-production.yaml b/infra/charts/feast/values-production.yaml deleted file mode 100644 index 6b53dc19ea8..00000000000 --- a/infra/charts/feast/values-production.yaml +++ /dev/null @@ -1,4 +0,0 @@ -# TODO @dheryanto -# -# The following are sample values for installing Feast for typical production -# environment. diff --git a/infra/charts/feast/values.yaml b/infra/charts/feast/values.yaml index fde03f9ad71..20ee2ab029f 100644 --- a/infra/charts/feast/values.yaml +++ b/infra/charts/feast/values.yaml @@ -1,262 +1,35 @@ -# Feast deployment installs the following components: -# - Feast Core -# - Feast Serving Online -# - Feast Serving Batch -# - Prometheus StatsD Exporter -# -# The configuration for different components can be referenced from: -# - charts/feast-core/values.yaml -# - charts/feast-serving/values.yaml -# - charts/prometheus-statsd-exporter/values.yaml -# -# Note that "feast-serving-online" and "feast-serving-batch" are -# aliases to "feast-serving" chart since in typical scenario two instances -# of Feast Serving: online and batch will be deployed. Both described -# using the same chart "feast-serving". -# -# Note that the import job by default uses DirectRunner -# https://beam.apache.org/documentation/runners/direct/ -# in this configuration since it allows Feast to run in more environments -# (unlike DataflowRunner which requires Google Cloud services). -# -# A secret containing Google Cloud service account JSON key is required -# in this configuration. -# https://cloud.google.com/iam/docs/creating-managing-service-accounts -# -# The Google Cloud service account must have the following roles: -# - bigquery.dataEditor -# - bigquery.jobUser -# -# Assuming a service account JSON key file has been downloaded to -# (please name the file key.json): -# /home/user/key.json -# -# Run the following command to create the secret in your Kubernetes cluster: -# -# kubectl create secret generic feast-gcp-service-account \ -# --from-file=/home/user/key.json -# -# Replace every instance of EXTERNAL_IP with the external IP of your GKE cluster - -# ============================================================ -# Feast Core -# ============================================================ - feast-core: - # If enabled specifies whether to install Feast Core component. - # - # Normally, this is set to "false" when Feast users need access to low latency - # Feast Serving, by deploying multiple instances of Feast Serving closest - # to the client. These instances of Feast Serving however can still use - # the same shared Feast Core. + # feast-core.enabled -- Flag to install Feast Core enabled: true - # Specify which image tag to use. Keep this consistent for all components - image: - tag: "0.4.4" +feast-online-serving: + # feast-online-serving.enabled -- Flag to install Feast Online Serving + enabled: true - # jvmOptions are options that will be passed to the Java Virtual Machine (JVM) - # running Feast Core. - # - # For example, it is good practice to set min and max heap size in JVM. - # https://stackoverflow.com/questions/6902135/side-effect-for-increasing-maxpermsize-and-max-heap-size - jvmOptions: - - -Xms1024m - - -Xmx1024m +feast-batch-serving: + # feast-batch-serving.enabled -- Flag to install Feast Batch Serving + enabled: false - # resources that should be allocated to Feast Core. - resources: - requests: - cpu: 1000m - memory: 1024Mi - limits: - memory: 2048Mi +postgresql: + # postgresql.enabled -- Flag to install Postgresql + enabled: true - # gcpServiceAccount is the Google service account that Feast Core will use. - gcpServiceAccount: - # useExistingSecret specifies Feast to use an existing secret containing - # Google Cloud service account JSON key file. - # - # This is the only supported option for now to use a service account JSON. - # Feast admin is expected to create this secret before deploying Feast. - useExistingSecret: true - existingSecret: - # name is the secret name of the existing secret for the service account. - name: feast-gcp-service-account - # key is the secret key of the existing secret for the service account. - # key is normally derived from the file name of the JSON key file. - key: key.json - # Setting service.type to NodePort exposes feast-core service at a static port - service: - type: NodePort - grpc: - # this is the port that is exposed outside of the cluster - nodePort: 32090 - # Make kafka externally accessible using NodePort - # Please set EXTERNAL_IP to your cluster's external IP - kafka: - external: - enabled: true - type: NodePort - domain: EXTERNAL_IP - configurationOverrides: - "advertised.listeners": |- - EXTERNAL://EXTERNAL_IP:$((31090 + ${KAFKA_BROKER_ID})) - "listener.security.protocol.map": |- - PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT - application.yaml: - feast: - stream: - options: - # Point to one of your Kafka brokers - # Please set EXTERNAL_IP to your cluster's external IP - bootstrapServers: EXTERNAL_IP:31090 +kafka: + # kafka.enabled -- Flag to install Kafka + enabled: true -# ============================================================ -# Feast Serving Online -# ============================================================ +redis: + # redis.enabled -- Flag to install Redis + enabled: true -feast-serving-online: - # enabled specifies whether to install Feast Serving Online component. +prometheus-statsd-exporter: + # prometheus-statsd-exporter.enabled -- Flag to install StatsD to Prometheus Exporter enabled: true - # Specify what image tag to use. Keep this consistent for all components - image: - tag: "0.4.4" - # redis.enabled specifies whether Redis should be installed as part of Feast Serving. - # - # If enabled is set to "false", Feast admin has to ensure there is an - # existing Redis running outside Feast, that Feast Serving can connect to. - # master.service.type set to NodePort exposes Redis to outside of the cluster - redis: - enabled: true - master: - service: - nodePort: 32101 - type: NodePort - # jvmOptions are options that will be passed to the Feast Serving JVM. - jvmOptions: - - -Xms1024m - - -Xmx1024m - # resources that should be allocated to Feast Serving. - resources: - requests: - cpu: 500m - memory: 1024Mi - limits: - memory: 2048Mi - # Make service accessible to outside of cluster using NodePort - service: - type: NodePort - grpc: - nodePort: 32091 - # store.yaml is the configuration for Feast Store. - # - # Refer to this link for more description: - # https://github.com/gojek/feast/blob/79eb4ab5fa3d37102c1dca9968162a98690526ba/protos/feast/core/Store.proto - store.yaml: - name: redis - type: REDIS - redis_config: - # If redis.enabled is set to false, Feast admin should uncomment and - # set the host value to an "existing" Redis instance Feast will use as - # online Store. Also use the correct port for that existing instance. - # - # Else, if redis.enabled is set to true, replace EXTERNAL_IP with your - # cluster's external IP. - # host: redis-host - host: EXTERNAL_IP - port: 32101 - subscriptions: - - name: "*" - project: "*" - version: "*" -# ============================================================ -# Feast Serving Batch -# ============================================================ +prometheus: + # prometheus.enabled -- Flag to install Prometheus + enabled: true -feast-serving-batch: - # enabled specifies whether to install Feast Serving Batch component. +grafana: + # grafana.enabled -- Flag to install Grafana enabled: true - # Specify what image tag to use. Keep this consistent for all components - image: - tag: "0.4.4" - # redis.enabled specifies whether Redis should be installed as part of Feast Serving. - # - # This is usually set to "false" for Feast Serving Batch because the default - # store is BigQuery. - redis: - enabled: false - # jvmOptions are options that will be passed to the Feast Serving JVM. - jvmOptions: - - -Xms1024m - - -Xmx1024m - # resources that should be allocated to Feast Serving. - resources: - requests: - cpu: 500m - memory: 1024Mi - limits: - memory: 2048Mi - # Make service accessible to outside of cluster using NodePort - service: - type: NodePort - grpc: - nodePort: 32092 - # gcpServiceAccount is the service account that Feast Serving will use. - gcpServiceAccount: - # useExistingSecret specifies Feast to use an existing secret containing - # Google Cloud service account JSON key file. - # - # This is the only supported option for now to use a service account JSON. - # Feast admin is expected to create this secret before deploying Feast. - useExistingSecret: true - existingSecret: - # name is the secret name of the existing secret for the service account. - name: feast-gcp-service-account - # key is the secret key of the existing secret for the service account. - # key is normally derived from the file name of the JSON key file. - key: key.json - # application.yaml is the main configuration for Feast Serving application. - # - # Feast Core is a Spring Boot app which uses this yaml configuration file. - # Refer to https://github.com/gojek/feast/blob/79eb4ab5fa3d37102c1dca9968162a98690526ba/serving/src/main/resources/application.yml - # for a complete list and description of the configuration. - application.yaml: - feast: - jobs: - # staging-location specifies the URI to store intermediate files for - # batch serving (required if using BigQuery as Store). - # - # Please set the value to an "existing" Google Cloud Storage URI that - # Feast serving has write access to. - staging-location: gs://YOUR_BUCKET_NAME/serving/batch - # Type of store to store job metadata. - # - # This default configuration assumes that Feast Serving Online is - # enabled as well. So Feast Serving Batch will share the same - # Redis instance to store job statuses. - store-type: REDIS - # Default to use the internal hostname of the redis instance deployed by Online service, - # otherwise use externally exposed by setting EXTERNAL_IP to your cluster's external IP - # store-options: - # host: EXTERNAL_IP - # port: 32101 - # store.yaml is the configuration for Feast Store. - # - # Refer to this link for more description: - # https://github.com/gojek/feast/blob/79eb4ab5fa3d37102c1dca9968162a98690526ba/protos/feast/core/Store.proto - store.yaml: - name: bigquery - type: BIGQUERY - bigquery_config: - # project_id specifies the Google Cloud Project. Please set this to the - # project id you are using BigQuery in. - project_id: PROJECT_ID - # dataset_id specifies an "existing" BigQuery dataset Feast Serving Batch - # will use. Please ensure this dataset is created beforehand. - dataset_id: DATASET_ID - subscriptions: - - name: "*" - project: "*" - version: "*" From 3d9bafd12515b872c479cfb0f7369e11eb0663d9 Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Sun, 3 May 2020 15:36:14 +0800 Subject: [PATCH 144/176] Add unique ingestion id for all batch ingestions (#656) * Add unique dataset id for all batch ingestions * Rename to ingestion_id --- protos/feast/core/Store.proto | 1 + protos/feast/types/FeatureRow.proto | 3 +++ sdk/python/feast/client.py | 17 ++++++++++++++++ sdk/python/feast/loaders/ingest.py | 20 +++++++++++++++---- .../bigquery/writer/BigQueryFeatureSink.java | 11 ++++++++-- .../bigquery/writer/FeatureRowToTableRow.java | 2 ++ 6 files changed, 48 insertions(+), 6 deletions(-) diff --git a/protos/feast/core/Store.proto b/protos/feast/core/Store.proto index 0aa4c8cd420..f35561467e1 100644 --- a/protos/feast/core/Store.proto +++ b/protos/feast/core/Store.proto @@ -69,6 +69,7 @@ message Store { // ====================|==================|================================ // - event_timestamp | TIMESTAMP | event time of the FeatureRow // - created_timestamp | TIMESTAMP | processing time of the ingestion of the FeatureRow + // - ingestion_id | STRING | unique id identifying groups of rows that have been ingested together // - job_id | STRING | identifier for the job that writes the FeatureRow to the corresponding BigQuery table // // BigQuery table created will be partitioned by the field "event_timestamp" diff --git a/protos/feast/types/FeatureRow.proto b/protos/feast/types/FeatureRow.proto index c170cd5d502..c19a393fde5 100644 --- a/protos/feast/types/FeatureRow.proto +++ b/protos/feast/types/FeatureRow.proto @@ -39,4 +39,7 @@ message FeatureRow { // /:. This value will be used by the feast ingestion job to filter // rows, and write the values to the correct tables. string feature_set = 6; + + // Identifier tying this feature row to a specific ingestion job. + string ingestion_id = 7; } diff --git a/sdk/python/feast/client.py b/sdk/python/feast/client.py index 0a38236a510..0221a79b4b1 100644 --- a/sdk/python/feast/client.py +++ b/sdk/python/feast/client.py @@ -18,6 +18,7 @@ import shutil import tempfile import time +import uuid from collections import OrderedDict from math import ceil from typing import Dict, List, Optional, Tuple, Union @@ -825,6 +826,7 @@ def ingest( # Loop optimization declarations produce = producer.produce flush = producer.flush + ingestion_id = _generate_ingestion_id(feature_set) # Transform and push data to Kafka if feature_set.source.source_type == "Kafka": @@ -832,6 +834,7 @@ def ingest( file=dest_path, row_groups=list(range(pq_file.num_row_groups)), fs=feature_set, + ingestion_id=ingestion_id, max_workers=max_workers, ): @@ -916,6 +919,20 @@ def _build_feature_references( return features +def _generate_ingestion_id(feature_set: FeatureSet) -> str: + """ + Generates a UUID from the feature set name, version, and the current time. + + Args: + feature_set: Feature set of the dataset to be ingested. + + Returns: + UUID unique to current time and the feature set provided. + """ + uuid_str = f"{feature_set.name}_{feature_set.version}_{int(time.time())}" + return str(uuid.uuid3(uuid.NAMESPACE_DNS, uuid_str)) + + def _read_table_from_source( source: Union[pd.DataFrame, str], chunk_size: int, max_workers: int ) -> Tuple[str, str]: diff --git a/sdk/python/feast/loaders/ingest.py b/sdk/python/feast/loaders/ingest.py index 4d215cc9901..34d0356ea78 100644 --- a/sdk/python/feast/loaders/ingest.py +++ b/sdk/python/feast/loaders/ingest.py @@ -26,7 +26,7 @@ def _encode_pa_tables( - file: str, feature_set: str, fields: dict, row_group_idx: int + file: str, feature_set: str, fields: dict, ingestion_id: str, row_group_idx: int ) -> List[bytes]: """ Helper function to encode a PyArrow table(s) read from parquet file(s) into @@ -49,6 +49,9 @@ def _encode_pa_tables( fields (dict[str, enum.Enum.ValueType]): A mapping of field names to their value types. + ingestion_id (str): + UUID unique to this ingestion job. + row_group_idx(int): Row group index to read and encode into byte like FeatureRow protobuf objects. @@ -81,7 +84,9 @@ def _encode_pa_tables( # Iterate through the rows for row_idx in range(table.num_rows): feature_row = FeatureRow( - event_timestamp=datetime_col[row_idx], feature_set=feature_set + event_timestamp=datetime_col[row_idx], + feature_set=feature_set, + ingestion_id=ingestion_id, ) # Loop optimization declaration ext = feature_row.fields.extend @@ -97,7 +102,11 @@ def _encode_pa_tables( def get_feature_row_chunks( - file: str, row_groups: List[int], fs: FeatureSet, max_workers: int + file: str, + row_groups: List[int], + fs: FeatureSet, + ingestion_id: str, + max_workers: int, ) -> Iterable[List[bytes]]: """ Iterator function to encode a PyArrow table read from a parquet file to @@ -115,6 +124,9 @@ def get_feature_row_chunks( fs (feast.feature_set.FeatureSet): FeatureSet describing parquet files. + ingestion_id (str): + UUID unique to this ingestion job. + max_workers (int): Maximum number of workers to spawn. @@ -128,7 +140,7 @@ def get_feature_row_chunks( field_map = {field.name: field.dtype for field in fs.fields.values()} pool = Pool(max_workers) - func = partial(_encode_pa_tables, file, feature_set, field_map) + func = partial(_encode_pa_tables, file, feature_set, field_map, ingestion_id) for chunk in pool.imap(func, row_groups): yield chunk return diff --git a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryFeatureSink.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryFeatureSink.java index d155d3f1f50..5d8f3d25cb7 100644 --- a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryFeatureSink.java +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryFeatureSink.java @@ -42,6 +42,8 @@ public abstract class BigQueryFeatureSink implements FeatureSink { "Event time for the FeatureRow"; public static final String BIGQUERY_CREATED_TIMESTAMP_FIELD_DESCRIPTION = "Processing time of the FeatureRow ingestion in Feast\""; + public static final String BIGQUERY_INGESTION_ID_FIELD_DESCRIPTION = + "Unique id identifying groups of rows that have been ingested together"; public static final String BIGQUERY_JOB_ID_FIELD_DESCRIPTION = "Feast import job ID for the FeatureRow"; @@ -108,10 +110,13 @@ public void prepareWrite(FeatureSetProto.FeatureSet featureSet) { Table table = bigquery.getTable(tableId); if (table != null) { log.info( - "Writing to existing BigQuery table '{}:{}.{}'", - getProjectId(), + "Updating and writing to existing BigQuery table '{}:{}.{}'", + datasetId.getProject(), datasetId.getDataset(), tableName); + TableDefinition tableDefinition = createBigQueryTableDefinition(featureSet.getSpec()); + TableInfo tableInfo = TableInfo.of(tableId, tableDefinition); + bigquery.update(tableInfo); return; } @@ -166,6 +171,8 @@ private TableDefinition createBigQueryTableDefinition(FeatureSetProto.FeatureSet "created_timestamp", Pair.of( StandardSQLTypeName.TIMESTAMP, BIGQUERY_CREATED_TIMESTAMP_FIELD_DESCRIPTION), + "ingestion_id", + Pair.of(StandardSQLTypeName.STRING, BIGQUERY_INGESTION_ID_FIELD_DESCRIPTION), "job_id", Pair.of(StandardSQLTypeName.STRING, BIGQUERY_JOB_ID_FIELD_DESCRIPTION)); for (Map.Entry> entry : diff --git a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/FeatureRowToTableRow.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/FeatureRowToTableRow.java index 12833b31b85..6a69b96d717 100644 --- a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/FeatureRowToTableRow.java +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/FeatureRowToTableRow.java @@ -31,6 +31,7 @@ public class FeatureRowToTableRow implements SerializableFunction { private static final String EVENT_TIMESTAMP_COLUMN = "event_timestamp"; private static final String CREATED_TIMESTAMP_COLUMN = "created_timestamp"; + private static final String INGESTION_ID_COLUMN = "ingestion_id"; private static final String JOB_ID_COLUMN = "job_id"; private final String jobId; @@ -47,6 +48,7 @@ public TableRow apply(FeatureRow featureRow) { TableRow tableRow = new TableRow(); tableRow.set(EVENT_TIMESTAMP_COLUMN, Timestamps.toString(featureRow.getEventTimestamp())); tableRow.set(CREATED_TIMESTAMP_COLUMN, Instant.now().toString()); + tableRow.set(INGESTION_ID_COLUMN, featureRow.getIngestionId()); tableRow.set(JOB_ID_COLUMN, jobId); for (Field field : featureRow.getFieldsList()) { From 6764630d236f2c9db42e13687d6190c74e744e3e Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Tue, 5 May 2020 10:41:14 +0800 Subject: [PATCH 145/176] Split Field model into distinct Feature and Entity objects (#655) * Split Field model into distinct Feature and Entity objects * Remove TFX fields for entities in testdata * Split Field model into distinct Feature and Entity objects * Index jointables * Explicitly name tables, remove redundant constructor * Integrate labels * Fix code comments * Change FeatureSetId to int * Retrieve featuresets from repository so that ids are consistent * Add uniqueness constraint to FeatureSets, fix tests * Remove feature and entity references --- .../feast/core/dao/MetricsRepository.java | 27 -- .../java/feast/core/job/JobUpdateTask.java | 8 +- .../main/java/feast/core/model/Entity.java | 73 +++++ .../main/java/feast/core/model/Feature.java | 194 +++++++++++++ .../java/feast/core/model/FeatureSet.java | 206 ++++---------- .../src/main/java/feast/core/model/Field.java | 258 ------------------ core/src/main/java/feast/core/model/Job.java | 41 +-- .../main/java/feast/core/model/Metrics.java | 64 ----- .../core/service/JobCoordinatorService.java | 12 +- .../java/feast/core/service/SpecService.java | 2 + .../service/JobCoordinatorServiceTest.java | 14 + .../feast/core/service/JobServiceTest.java | 11 +- .../feast/core/service/SpecServiceTest.java | 79 ++---- .../feast/core/service/TestObjectFactory.java | 13 +- protos/feast/core/FeatureSet.proto | 42 +-- sdk/python/feast/entity.py | 24 +- sdk/python/feast/feature_set.py | 2 + .../bikeshare_feature_set.yaml | 9 - .../tensorflow_metadata/bikeshare_schema.json | 19 -- sdk/python/tests/test_feature_set.py | 3 - 20 files changed, 390 insertions(+), 711 deletions(-) delete mode 100644 core/src/main/java/feast/core/dao/MetricsRepository.java create mode 100644 core/src/main/java/feast/core/model/Entity.java create mode 100644 core/src/main/java/feast/core/model/Feature.java delete mode 100644 core/src/main/java/feast/core/model/Field.java delete mode 100644 core/src/main/java/feast/core/model/Metrics.java diff --git a/core/src/main/java/feast/core/dao/MetricsRepository.java b/core/src/main/java/feast/core/dao/MetricsRepository.java deleted file mode 100644 index 7146e1e3ecb..00000000000 --- a/core/src/main/java/feast/core/dao/MetricsRepository.java +++ /dev/null @@ -1,27 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.core.dao; - -import feast.core.model.Metrics; -import java.util.List; -import org.springframework.data.jpa.repository.JpaRepository; -import org.springframework.stereotype.Repository; - -@Repository -public interface MetricsRepository extends JpaRepository { - List findByJob_Id(String id); -} diff --git a/core/src/main/java/feast/core/job/JobUpdateTask.java b/core/src/main/java/feast/core/job/JobUpdateTask.java index 25ce386d40c..bb876f47f22 100644 --- a/core/src/main/java/feast/core/job/JobUpdateTask.java +++ b/core/src/main/java/feast/core/job/JobUpdateTask.java @@ -16,6 +16,7 @@ */ package feast.core.job; +import com.google.common.collect.Sets; import feast.core.log.Action; import feast.core.log.AuditLogger; import feast.core.log.Resource; @@ -35,7 +36,6 @@ import java.util.concurrent.Future; import java.util.concurrent.TimeUnit; import java.util.concurrent.TimeoutException; -import java.util.stream.Collectors; import lombok.Getter; import lombok.extern.slf4j.Slf4j; @@ -102,10 +102,8 @@ public Job call() { } boolean featureSetsChangedFor(Job job) { - Set existingFeatureSetsPopulatedByJob = - job.getFeatureSets().stream().map(FeatureSet::getId).collect(Collectors.toSet()); - Set newFeatureSetsPopulatedByJob = - featureSets.stream().map(FeatureSet::getId).collect(Collectors.toSet()); + Set existingFeatureSetsPopulatedByJob = Sets.newHashSet(job.getFeatureSets()); + Set newFeatureSetsPopulatedByJob = Sets.newHashSet(featureSets); return !newFeatureSetsPopulatedByJob.equals(existingFeatureSetsPopulatedByJob); } diff --git a/core/src/main/java/feast/core/model/Entity.java b/core/src/main/java/feast/core/model/Entity.java new file mode 100644 index 00000000000..791e280d481 --- /dev/null +++ b/core/src/main/java/feast/core/model/Entity.java @@ -0,0 +1,73 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2019 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.core.model; + +import feast.core.FeatureSetProto.EntitySpec; +import feast.types.ValueProto.ValueType; +import java.util.Objects; +import javax.persistence.*; +import lombok.Getter; +import lombok.Setter; + +/** Feast entity object. Contains name and type of the entity. */ +@Getter +@Setter +@javax.persistence.Entity +@Table( + name = "entities", + uniqueConstraints = @UniqueConstraint(columnNames = {"name", "feature_set_id"})) +public class Entity { + + @Id @GeneratedValue private Long id; + + private String name; + + @ManyToOne(fetch = FetchType.LAZY) + private FeatureSet featureSet; + + /** Data type of the entity. String representation of {@link ValueType} * */ + private String type; + + public Entity() {} + + private Entity(String name, ValueType.Enum type) { + this.setName(name); + this.setType(type.toString()); + } + + public static Entity fromProto(EntitySpec entitySpec) { + Entity entity = new Entity(entitySpec.getName(), entitySpec.getValueType()); + return entity; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + Entity entity = (Entity) o; + return getName().equals(entity.getName()) && getType().equals(entity.getType()); + } + + @Override + public int hashCode() { + return Objects.hash(super.hashCode(), getName(), getType()); + } +} diff --git a/core/src/main/java/feast/core/model/Feature.java b/core/src/main/java/feast/core/model/Feature.java new file mode 100644 index 00000000000..38e2d4549ed --- /dev/null +++ b/core/src/main/java/feast/core/model/Feature.java @@ -0,0 +1,194 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2019 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.core.model; + +import feast.core.FeatureSetProto.FeatureSpec; +import feast.core.util.TypeConversion; +import feast.types.ValueProto.ValueType; +import java.util.Arrays; +import java.util.Map; +import java.util.Objects; +import javax.persistence.*; +import javax.persistence.Entity; +import lombok.Getter; +import lombok.Setter; + +/** + * Feature belonging to a featureset. Contains name, type as well as domain metadata about the + * feature. + */ +@Getter +@Setter +@Entity +@Table( + name = "features", + uniqueConstraints = @UniqueConstraint(columnNames = {"name", "feature_set_id"})) +public class Feature { + + @Id @GeneratedValue private Long id; + + private String name; + + @ManyToOne(fetch = FetchType.LAZY) + private FeatureSet featureSet; + + /** Data type of the feature. String representation of {@link ValueType} * */ + private String type; + + // Labels for this feature + @Column(name = "labels", columnDefinition = "text") + private String labels; + + // Presence constraints (refer to proto feast.core.FeatureSet.FeatureSpec) + // Only one of them can be set. + private byte[] presence; + private byte[] groupPresence; + + // Shape type (refer to proto feast.core.FeatureSet.FeatureSpec) + // Only one of them can be set. + private byte[] shape; + private byte[] valueCount; + + // Domain info for the values (refer to proto feast.core.FeatureSet.FeatureSpec) + // Only one of them can be set. + private String domain; + private byte[] intDomain; + private byte[] floatDomain; + private byte[] stringDomain; + private byte[] boolDomain; + private byte[] structDomain; + private byte[] naturalLanguageDomain; + private byte[] imageDomain; + private byte[] midDomain; + private byte[] urlDomain; + private byte[] timeDomain; + private byte[] timeOfDayDomain; + + public Feature() {} + + private Feature(String name, ValueType.Enum type) { + this.setName(name); + this.setType(type.toString()); + } + + public static Feature fromProto(FeatureSpec featureSpec) { + Feature feature = new Feature(featureSpec.getName(), featureSpec.getValueType()); + feature.labels = TypeConversion.convertMapToJsonString(featureSpec.getLabelsMap()); + + switch (featureSpec.getPresenceConstraintsCase()) { + case PRESENCE: + feature.setPresence(featureSpec.getPresence().toByteArray()); + break; + case GROUP_PRESENCE: + feature.setGroupPresence(featureSpec.getGroupPresence().toByteArray()); + break; + case PRESENCECONSTRAINTS_NOT_SET: + break; + } + + switch (featureSpec.getShapeTypeCase()) { + case SHAPE: + feature.setShape(featureSpec.getShape().toByteArray()); + break; + case VALUE_COUNT: + feature.setValueCount(featureSpec.getValueCount().toByteArray()); + break; + case SHAPETYPE_NOT_SET: + break; + } + + switch (featureSpec.getDomainInfoCase()) { + case DOMAIN: + feature.setDomain(featureSpec.getDomain()); + break; + case INT_DOMAIN: + feature.setIntDomain(featureSpec.getIntDomain().toByteArray()); + break; + case FLOAT_DOMAIN: + feature.setFloatDomain(featureSpec.getFloatDomain().toByteArray()); + break; + case STRING_DOMAIN: + feature.setStringDomain(featureSpec.getStringDomain().toByteArray()); + break; + case BOOL_DOMAIN: + feature.setBoolDomain(featureSpec.getBoolDomain().toByteArray()); + break; + case STRUCT_DOMAIN: + feature.setStructDomain(featureSpec.getStructDomain().toByteArray()); + break; + case NATURAL_LANGUAGE_DOMAIN: + feature.setNaturalLanguageDomain(featureSpec.getNaturalLanguageDomain().toByteArray()); + break; + case IMAGE_DOMAIN: + feature.setImageDomain(featureSpec.getImageDomain().toByteArray()); + break; + case MID_DOMAIN: + feature.setMidDomain(featureSpec.getMidDomain().toByteArray()); + break; + case URL_DOMAIN: + feature.setUrlDomain(featureSpec.getUrlDomain().toByteArray()); + break; + case TIME_DOMAIN: + feature.setTimeDomain(featureSpec.getTimeDomain().toByteArray()); + break; + case TIME_OF_DAY_DOMAIN: + feature.setTimeOfDayDomain(featureSpec.getTimeOfDayDomain().toByteArray()); + break; + case DOMAININFO_NOT_SET: + break; + } + return feature; + } + + public Map getLabels() { + return TypeConversion.convertJsonStringToMap(this.labels); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + Feature feature = (Feature) o; + return Objects.equals(getName(), feature.getName()) + && Objects.equals(labels, feature.labels) + && Arrays.equals(getPresence(), feature.getPresence()) + && Arrays.equals(getGroupPresence(), feature.getGroupPresence()) + && Arrays.equals(getShape(), feature.getShape()) + && Arrays.equals(getValueCount(), feature.getValueCount()) + && Objects.equals(getDomain(), feature.getDomain()) + && Arrays.equals(getIntDomain(), feature.getIntDomain()) + && Arrays.equals(getFloatDomain(), feature.getFloatDomain()) + && Arrays.equals(getStringDomain(), feature.getStringDomain()) + && Arrays.equals(getBoolDomain(), feature.getBoolDomain()) + && Arrays.equals(getStructDomain(), feature.getStructDomain()) + && Arrays.equals(getNaturalLanguageDomain(), feature.getNaturalLanguageDomain()) + && Arrays.equals(getImageDomain(), feature.getImageDomain()) + && Arrays.equals(getMidDomain(), feature.getMidDomain()) + && Arrays.equals(getUrlDomain(), feature.getUrlDomain()) + && Arrays.equals(getTimeDomain(), feature.getTimeDomain()) + && Arrays.equals(getTimeDomain(), feature.getTimeOfDayDomain()); + } + + @Override + public int hashCode() { + return Objects.hash(super.hashCode(), getName(), getType(), getLabels()); + } +} diff --git a/core/src/main/java/feast/core/model/FeatureSet.java b/core/src/main/java/feast/core/model/FeatureSet.java index ec8da77c5f9..faaee0e41f6 100644 --- a/core/src/main/java/feast/core/model/FeatureSet.java +++ b/core/src/main/java/feast/core/model/FeatureSet.java @@ -20,61 +20,26 @@ import com.google.protobuf.InvalidProtocolBufferException; import com.google.protobuf.Timestamp; import feast.core.FeatureSetProto; -import feast.core.FeatureSetProto.EntitySpec; -import feast.core.FeatureSetProto.FeatureSetMeta; -import feast.core.FeatureSetProto.FeatureSetSpec; -import feast.core.FeatureSetProto.FeatureSetStatus; -import feast.core.FeatureSetProto.FeatureSpec; +import feast.core.FeatureSetProto.*; import feast.core.util.TypeConversion; import feast.types.ValueProto.ValueType.Enum; -import java.util.ArrayList; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; -import javax.persistence.CascadeType; -import javax.persistence.CollectionTable; -import javax.persistence.Column; -import javax.persistence.ElementCollection; -import javax.persistence.Entity; -import javax.persistence.FetchType; -import javax.persistence.Id; -import javax.persistence.JoinColumn; -import javax.persistence.ManyToOne; -import javax.persistence.Table; -import javax.persistence.UniqueConstraint; +import java.util.*; +import javax.persistence.*; import lombok.Getter; import lombok.Setter; import org.apache.commons.lang3.builder.HashCodeBuilder; -import org.hibernate.annotations.Fetch; -import org.hibernate.annotations.FetchMode; -import org.tensorflow.metadata.v0.BoolDomain; -import org.tensorflow.metadata.v0.FeaturePresence; -import org.tensorflow.metadata.v0.FeaturePresenceWithinGroup; -import org.tensorflow.metadata.v0.FixedShape; -import org.tensorflow.metadata.v0.FloatDomain; -import org.tensorflow.metadata.v0.ImageDomain; -import org.tensorflow.metadata.v0.IntDomain; -import org.tensorflow.metadata.v0.MIDDomain; -import org.tensorflow.metadata.v0.NaturalLanguageDomain; -import org.tensorflow.metadata.v0.StringDomain; -import org.tensorflow.metadata.v0.StructDomain; -import org.tensorflow.metadata.v0.TimeDomain; -import org.tensorflow.metadata.v0.TimeOfDayDomain; -import org.tensorflow.metadata.v0.URLDomain; -import org.tensorflow.metadata.v0.ValueCount; +import org.tensorflow.metadata.v0.*; @Getter @Setter -@Entity -@Table(name = "feature_sets") +@javax.persistence.Entity +@Table( + name = "feature_sets", + uniqueConstraints = @UniqueConstraint(columnNames = {"name", "version", "project_name"})) public class FeatureSet extends AbstractTimestampEntity implements Comparable { // Id of the featureSet, defined as project/feature_set_name:feature_set_version - @Id - @Column(name = "id", nullable = false, unique = true) - private String id; + @Id @GeneratedValue private long id; // Name of the featureSet @Column(name = "name", nullable = false) @@ -94,19 +59,20 @@ public class FeatureSet extends AbstractTimestampEntity implements Comparable entities; + @OneToMany( + mappedBy = "featureSet", + cascade = CascadeType.ALL, + fetch = FetchType.EAGER, + orphanRemoval = true) + private Set entities; // Feature fields inside this feature set - @ElementCollection(fetch = FetchType.EAGER) - @CollectionTable( - name = "features", - joinColumns = @JoinColumn(name = "feature_set_id"), - uniqueConstraints = @UniqueConstraint(columnNames = {"name", "project", "version"})) - @Fetch(FetchMode.SUBSELECT) - private Set features; + @OneToMany( + mappedBy = "featureSet", + cascade = CascadeType.ALL, + fetch = FetchType.EAGER, + orphanRemoval = true) + private Set features; // Source on which feature rows can be found @ManyToOne(cascade = CascadeType.ALL, fetch = FetchType.EAGER) @@ -130,8 +96,8 @@ public FeatureSet( String project, int version, long maxAgeSeconds, - List entities, - List features, + List entities, + List features, Source source, Map labels, FeatureSetStatus status) { @@ -144,23 +110,16 @@ public FeatureSet( this.project = new Project(project); this.version = version; this.labels = TypeConversion.convertMapToJsonString(labels); - this.setId(project, name, version); addEntities(entities); addFeatures(features); } - private void setId(String project, String name, int version) { - this.id = project + "/" + name + ":" + version; - } - public void setVersion(int version) { this.version = version; - this.setId(getProjectName(), getName(), version); } public void setName(String name) { this.name = name; - this.setId(getProjectName(), name, getVersion()); } private String getProjectName() { @@ -173,21 +132,20 @@ private String getProjectName() { public void setProject(Project project) { this.project = project; - this.setId(project.getName(), getName(), getVersion()); } public static FeatureSet fromProto(FeatureSetProto.FeatureSet featureSetProto) { FeatureSetSpec featureSetSpec = featureSetProto.getSpec(); Source source = Source.fromProto(featureSetSpec.getSource()); - List featureSpecs = new ArrayList<>(); + List featureSpecs = new ArrayList<>(); for (FeatureSpec featureSpec : featureSetSpec.getFeaturesList()) { - featureSpecs.add(new Field(featureSpec)); + featureSpecs.add(Feature.fromProto(featureSpec)); } - List entitySpecs = new ArrayList<>(); + List entitySpecs = new ArrayList<>(); for (EntitySpec entitySpec : featureSetSpec.getEntitiesList()) { - entitySpecs.add(new Field(entitySpec)); + entitySpecs.add(Entity.fromProto(entitySpec)); } return new FeatureSet( @@ -202,40 +160,38 @@ public static FeatureSet fromProto(FeatureSetProto.FeatureSet featureSetProto) { featureSetProto.getMeta().getStatus()); } - public void addEntities(List fields) { - for (Field field : fields) { - addEntity(field); + public void addEntities(List entities) { + for (Entity entity : entities) { + addEntity(entity); } } - public void addEntity(Field field) { - field.setProject(this.project.getName()); - field.setVersion(this.getVersion()); - entities.add(field); + public void addEntity(Entity entity) { + entity.setFeatureSet(this); + entities.add(entity); } - public void addFeatures(List fields) { - for (Field field : fields) { - addFeature(field); + public void addFeatures(List features) { + for (Feature feature : features) { + addFeature(feature); } } - public void addFeature(Field field) { - field.setProject(this.project.getName()); - field.setVersion(this.getVersion()); - features.add(field); + public void addFeature(Feature feature) { + feature.setFeatureSet(this); + features.add(feature); } public FeatureSetProto.FeatureSet toProto() throws InvalidProtocolBufferException { List entitySpecs = new ArrayList<>(); - for (Field entityField : entities) { + for (Entity entityField : entities) { EntitySpec.Builder entitySpecBuilder = EntitySpec.newBuilder(); setEntitySpecFields(entitySpecBuilder, entityField); entitySpecs.add(entitySpecBuilder.build()); } List featureSpecs = new ArrayList<>(); - for (Field featureField : features) { + for (Feature featureField : features) { FeatureSpec.Builder featureSpecBuilder = FeatureSpec.newBuilder(); setFeatureSpecFields(featureSpecBuilder, featureField); featureSpecs.add(featureSpecBuilder.build()); @@ -261,61 +217,13 @@ public FeatureSetProto.FeatureSet toProto() throws InvalidProtocolBufferExceptio return FeatureSetProto.FeatureSet.newBuilder().setMeta(meta).setSpec(spec).build(); } - // setEntitySpecFields and setFeatureSpecFields methods contain duplicated code because - // Feast internally treat EntitySpec and FeatureSpec as Field class. However, the proto message - // builder for EntitySpec and FeatureSpec are of different class. - @SuppressWarnings("DuplicatedCode") - private void setEntitySpecFields(EntitySpec.Builder entitySpecBuilder, Field entityField) - throws InvalidProtocolBufferException { + private void setEntitySpecFields(EntitySpec.Builder entitySpecBuilder, Entity entityField) { entitySpecBuilder .setName(entityField.getName()) .setValueType(Enum.valueOf(entityField.getType())); - - if (entityField.getPresence() != null) { - entitySpecBuilder.setPresence(FeaturePresence.parseFrom(entityField.getPresence())); - } else if (entityField.getGroupPresence() != null) { - entitySpecBuilder.setGroupPresence( - FeaturePresenceWithinGroup.parseFrom(entityField.getGroupPresence())); - } - - if (entityField.getShape() != null) { - entitySpecBuilder.setShape(FixedShape.parseFrom(entityField.getShape())); - } else if (entityField.getValueCount() != null) { - entitySpecBuilder.setValueCount(ValueCount.parseFrom(entityField.getValueCount())); - } - - if (entityField.getDomain() != null) { - entitySpecBuilder.setDomain(entityField.getDomain()); - } else if (entityField.getIntDomain() != null) { - entitySpecBuilder.setIntDomain(IntDomain.parseFrom(entityField.getIntDomain())); - } else if (entityField.getFloatDomain() != null) { - entitySpecBuilder.setFloatDomain(FloatDomain.parseFrom(entityField.getFloatDomain())); - } else if (entityField.getStringDomain() != null) { - entitySpecBuilder.setStringDomain(StringDomain.parseFrom(entityField.getStringDomain())); - } else if (entityField.getBoolDomain() != null) { - entitySpecBuilder.setBoolDomain(BoolDomain.parseFrom(entityField.getBoolDomain())); - } else if (entityField.getStructDomain() != null) { - entitySpecBuilder.setStructDomain(StructDomain.parseFrom(entityField.getStructDomain())); - } else if (entityField.getNaturalLanguageDomain() != null) { - entitySpecBuilder.setNaturalLanguageDomain( - NaturalLanguageDomain.parseFrom(entityField.getNaturalLanguageDomain())); - } else if (entityField.getImageDomain() != null) { - entitySpecBuilder.setImageDomain(ImageDomain.parseFrom(entityField.getImageDomain())); - } else if (entityField.getMidDomain() != null) { - entitySpecBuilder.setIntDomain(IntDomain.parseFrom(entityField.getIntDomain())); - } else if (entityField.getUrlDomain() != null) { - entitySpecBuilder.setUrlDomain(URLDomain.parseFrom(entityField.getUrlDomain())); - } else if (entityField.getTimeDomain() != null) { - entitySpecBuilder.setTimeDomain(TimeDomain.parseFrom(entityField.getTimeDomain())); - } else if (entityField.getTimeOfDayDomain() != null) { - entitySpecBuilder.setTimeOfDayDomain( - TimeOfDayDomain.parseFrom(entityField.getTimeOfDayDomain())); - } } - // Refer to setEntitySpecFields method for the reason for code duplication. - @SuppressWarnings("DuplicatedCode") - private void setFeatureSpecFields(FeatureSpec.Builder featureSpecBuilder, Field featureField) + private void setFeatureSpecFields(FeatureSpec.Builder featureSpecBuilder, Feature featureField) throws InvalidProtocolBufferException { featureSpecBuilder .setName(featureField.getName()) @@ -391,36 +299,40 @@ public boolean equalTo(FeatureSet other) { } // Create a map of all fields in this feature set - Map fields = new HashMap<>(); + Map entitiesMap = new HashMap<>(); + Map featuresMap = new HashMap<>(); - for (Field e : entities) { - fields.putIfAbsent(e.getName(), e); + for (Entity e : entities) { + entitiesMap.putIfAbsent(e.getName(), e); } - for (Field f : features) { - fields.putIfAbsent(f.getName(), f); + for (Feature f : features) { + featuresMap.putIfAbsent(f.getName(), f); } // Ensure map size is consistent with existing fields - if (fields.size() != other.getFeatures().size() + other.getEntities().size()) { + if (entitiesMap.size() != other.getEntities().size()) { + return false; + } + if (featuresMap.size() != other.getFeatures().size()) { return false; } // Ensure the other entities and features exist in the field map - for (Field e : other.getEntities()) { - if (!fields.containsKey(e.getName())) { + for (Entity e : other.getEntities()) { + if (!entitiesMap.containsKey(e.getName())) { return false; } - if (!e.equals(fields.get(e.getName()))) { + if (!e.equals(entitiesMap.get(e.getName()))) { return false; } } - for (Field f : other.getFeatures()) { - if (!fields.containsKey(f.getName())) { + for (Feature f : other.getFeatures()) { + if (!featuresMap.containsKey(f.getName())) { return false; } - if (!f.equals(fields.get(f.getName()))) { + if (!f.equals(featuresMap.get(f.getName()))) { return false; } } diff --git a/core/src/main/java/feast/core/model/Field.java b/core/src/main/java/feast/core/model/Field.java deleted file mode 100644 index 213c17f954a..00000000000 --- a/core/src/main/java/feast/core/model/Field.java +++ /dev/null @@ -1,258 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.core.model; - -import feast.core.FeatureSetProto.EntitySpec; -import feast.core.FeatureSetProto.FeatureSpec; -import feast.core.util.TypeConversion; -import java.util.Arrays; -import java.util.Map; -import java.util.Objects; -import javax.persistence.Column; -import javax.persistence.Embeddable; -import lombok.Getter; -import lombok.Setter; - -@Getter -@Setter -@Embeddable -public class Field { - - // Name of the feature - @Column(name = "name", nullable = false) - private String name; - - // Type of the feature, should correspond with feast.types.ValueType - @Column(name = "type", nullable = false) - private String type; - - // Version of the field - @Column(name = "version") - private int version; - - // Project that this field belongs to - @Column(name = "project") - private String project; - - // Labels that this field belongs to - @Column(name = "labels", columnDefinition = "text") - private String labels; - - // Presence constraints (refer to proto feast.core.FeatureSet.FeatureSpec) - // Only one of them can be set. - private byte[] presence; - private byte[] groupPresence; - - // Shape type (refer to proto feast.core.FeatureSet.FeatureSpec) - // Only one of them can be set. - private byte[] shape; - private byte[] valueCount; - - // Domain info for the values (refer to proto feast.core.FeatureSet.FeatureSpec) - // Only one of them can be set. - private String domain; - private byte[] intDomain; - private byte[] floatDomain; - private byte[] stringDomain; - private byte[] boolDomain; - private byte[] structDomain; - private byte[] naturalLanguageDomain; - private byte[] imageDomain; - private byte[] midDomain; - private byte[] urlDomain; - private byte[] timeDomain; - private byte[] timeOfDayDomain; - - public Field() {} - - public Field(FeatureSpec featureSpec) { - this.name = featureSpec.getName(); - this.type = featureSpec.getValueType().toString(); - this.labels = TypeConversion.convertMapToJsonString(featureSpec.getLabelsMap()); - - switch (featureSpec.getPresenceConstraintsCase()) { - case PRESENCE: - this.presence = featureSpec.getPresence().toByteArray(); - break; - case GROUP_PRESENCE: - this.groupPresence = featureSpec.getGroupPresence().toByteArray(); - break; - case PRESENCECONSTRAINTS_NOT_SET: - break; - } - - switch (featureSpec.getShapeTypeCase()) { - case SHAPE: - this.shape = featureSpec.getShape().toByteArray(); - break; - case VALUE_COUNT: - this.valueCount = featureSpec.getValueCount().toByteArray(); - break; - case SHAPETYPE_NOT_SET: - break; - } - - switch (featureSpec.getDomainInfoCase()) { - case DOMAIN: - this.domain = featureSpec.getDomain(); - break; - case INT_DOMAIN: - this.intDomain = featureSpec.getIntDomain().toByteArray(); - break; - case FLOAT_DOMAIN: - this.floatDomain = featureSpec.getFloatDomain().toByteArray(); - break; - case STRING_DOMAIN: - this.stringDomain = featureSpec.getStringDomain().toByteArray(); - break; - case BOOL_DOMAIN: - this.boolDomain = featureSpec.getBoolDomain().toByteArray(); - break; - case STRUCT_DOMAIN: - this.structDomain = featureSpec.getStructDomain().toByteArray(); - break; - case NATURAL_LANGUAGE_DOMAIN: - this.naturalLanguageDomain = featureSpec.getNaturalLanguageDomain().toByteArray(); - break; - case IMAGE_DOMAIN: - this.imageDomain = featureSpec.getImageDomain().toByteArray(); - break; - case MID_DOMAIN: - this.midDomain = featureSpec.getMidDomain().toByteArray(); - break; - case URL_DOMAIN: - this.urlDomain = featureSpec.getUrlDomain().toByteArray(); - break; - case TIME_DOMAIN: - this.timeDomain = featureSpec.getTimeDomain().toByteArray(); - break; - case TIME_OF_DAY_DOMAIN: - this.timeOfDayDomain = featureSpec.getTimeOfDayDomain().toByteArray(); - break; - case DOMAININFO_NOT_SET: - break; - } - } - - public Field(EntitySpec entitySpec) { - this.name = entitySpec.getName(); - this.type = entitySpec.getValueType().toString(); - - switch (entitySpec.getPresenceConstraintsCase()) { - case PRESENCE: - this.presence = entitySpec.getPresence().toByteArray(); - break; - case GROUP_PRESENCE: - this.groupPresence = entitySpec.getGroupPresence().toByteArray(); - break; - case PRESENCECONSTRAINTS_NOT_SET: - break; - } - - switch (entitySpec.getShapeTypeCase()) { - case SHAPE: - this.shape = entitySpec.getShape().toByteArray(); - break; - case VALUE_COUNT: - this.valueCount = entitySpec.getValueCount().toByteArray(); - break; - case SHAPETYPE_NOT_SET: - break; - } - - switch (entitySpec.getDomainInfoCase()) { - case DOMAIN: - this.domain = entitySpec.getDomain(); - break; - case INT_DOMAIN: - this.intDomain = entitySpec.getIntDomain().toByteArray(); - break; - case FLOAT_DOMAIN: - this.floatDomain = entitySpec.getFloatDomain().toByteArray(); - break; - case STRING_DOMAIN: - this.stringDomain = entitySpec.getStringDomain().toByteArray(); - break; - case BOOL_DOMAIN: - this.boolDomain = entitySpec.getBoolDomain().toByteArray(); - break; - case STRUCT_DOMAIN: - this.structDomain = entitySpec.getStructDomain().toByteArray(); - break; - case NATURAL_LANGUAGE_DOMAIN: - this.naturalLanguageDomain = entitySpec.getNaturalLanguageDomain().toByteArray(); - break; - case IMAGE_DOMAIN: - this.imageDomain = entitySpec.getImageDomain().toByteArray(); - break; - case MID_DOMAIN: - this.midDomain = entitySpec.getMidDomain().toByteArray(); - break; - case URL_DOMAIN: - this.urlDomain = entitySpec.getUrlDomain().toByteArray(); - break; - case TIME_DOMAIN: - this.timeDomain = entitySpec.getTimeDomain().toByteArray(); - break; - case TIME_OF_DAY_DOMAIN: - this.timeOfDayDomain = entitySpec.getTimeOfDayDomain().toByteArray(); - break; - case DOMAININFO_NOT_SET: - break; - } - } - - public Map getLabels() { - return TypeConversion.convertJsonStringToMap(this.labels); - } - - @Override - public boolean equals(Object o) { - if (this == o) { - return true; - } - if (o == null || getClass() != o.getClass()) { - return false; - } - Field field = (Field) o; - return Objects.equals(name, field.name) - && Objects.equals(type, field.type) - && Objects.equals(project, field.project) - && Objects.equals(labels, field.labels) - && Arrays.equals(presence, field.presence) - && Arrays.equals(groupPresence, field.groupPresence) - && Arrays.equals(shape, field.shape) - && Arrays.equals(valueCount, field.valueCount) - && Objects.equals(domain, field.domain) - && Arrays.equals(intDomain, field.intDomain) - && Arrays.equals(floatDomain, field.floatDomain) - && Arrays.equals(stringDomain, field.stringDomain) - && Arrays.equals(boolDomain, field.boolDomain) - && Arrays.equals(structDomain, field.structDomain) - && Arrays.equals(naturalLanguageDomain, field.naturalLanguageDomain) - && Arrays.equals(imageDomain, field.imageDomain) - && Arrays.equals(midDomain, field.midDomain) - && Arrays.equals(urlDomain, field.urlDomain) - && Arrays.equals(timeDomain, field.timeDomain) - && Arrays.equals(timeOfDayDomain, field.timeOfDayDomain); - } - - @Override - public int hashCode() { - return Objects.hash(super.hashCode(), name, type, project, labels); - } -} diff --git a/core/src/main/java/feast/core/model/Job.java b/core/src/main/java/feast/core/model/Job.java index fc801f76a44..5c812f2a9ca 100644 --- a/core/src/main/java/feast/core/model/Job.java +++ b/core/src/main/java/feast/core/model/Job.java @@ -22,19 +22,8 @@ import feast.core.job.Runner; import java.util.ArrayList; import java.util.List; -import javax.persistence.CascadeType; -import javax.persistence.Column; +import javax.persistence.*; import javax.persistence.Entity; -import javax.persistence.EnumType; -import javax.persistence.Enumerated; -import javax.persistence.Id; -import javax.persistence.Index; -import javax.persistence.JoinColumn; -import javax.persistence.JoinTable; -import javax.persistence.ManyToMany; -import javax.persistence.ManyToOne; -import javax.persistence.OneToMany; -import javax.persistence.Table; import lombok.AllArgsConstructor; import lombok.Getter; import lombok.Setter; @@ -71,7 +60,7 @@ public class Job extends AbstractTimestampEntity { private Store store; // FeatureSets populated by the job - @ManyToMany + @ManyToMany(cascade = CascadeType.ALL) @JoinTable( name = "jobs_feature_sets", joinColumns = @JoinColumn(name = "job_id"), @@ -82,10 +71,6 @@ public class Job extends AbstractTimestampEntity { }) private List featureSets; - // Job Metrics - @OneToMany(mappedBy = "job", cascade = CascadeType.ALL) - private List metrics; - @Enumerated(EnumType.STRING) @Column(name = "status", length = 16) private JobStatus status; @@ -94,23 +79,6 @@ public Job() { super(); } - public Job( - String id, - String extId, - Runner runner, - Source source, - Store sink, - List featureSets, - JobStatus jobStatus) { - this.id = id; - this.extId = extId; - this.source = source; - this.runner = runner; - this.store = sink; - this.featureSets = featureSets; - this.status = jobStatus; - } - public boolean hasTerminated() { return getStatus().isTerminal(); } @@ -119,11 +87,6 @@ public boolean isRunning() { return getStatus() == JobStatus.RUNNING; } - public void updateMetrics(List newMetrics) { - metrics.clear(); - metrics.addAll(newMetrics); - } - public String getSinkName() { return store.getName(); } diff --git a/core/src/main/java/feast/core/model/Metrics.java b/core/src/main/java/feast/core/model/Metrics.java deleted file mode 100644 index 0b7514816fa..00000000000 --- a/core/src/main/java/feast/core/model/Metrics.java +++ /dev/null @@ -1,64 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.core.model; - -import javax.persistence.Entity; -import javax.persistence.FetchType; -import javax.persistence.GeneratedValue; -import javax.persistence.GenerationType; -import javax.persistence.Id; -import javax.persistence.JoinColumn; -import javax.persistence.ManyToOne; -import javax.persistence.Table; -import lombok.Getter; -import lombok.NoArgsConstructor; -import lombok.Setter; - -@NoArgsConstructor -@Getter -@Setter -@Entity -@Table(name = "metrics") -public class Metrics extends AbstractTimestampEntity { - - @Id - @GeneratedValue(strategy = GenerationType.AUTO) - private long id; - - @ManyToOne(fetch = FetchType.LAZY) - @JoinColumn(name = "job_id") - private Job job; - - /** Metrics name */ - private String name; - - /** Metrics value */ - private double value; - - /** - * Create a metrics owned by a {@code job}. - * - * @param job owner of this metrics. - * @param metricsName metrics name. - * @param value metrics value. - */ - public Metrics(Job job, String metricsName, double value) { - this.job = job; - this.name = metricsName; - this.value = value; - } -} diff --git a/core/src/main/java/feast/core/service/JobCoordinatorService.java b/core/src/main/java/feast/core/service/JobCoordinatorService.java index 6f366be5083..c0215767905 100644 --- a/core/src/main/java/feast/core/service/JobCoordinatorService.java +++ b/core/src/main/java/feast/core/service/JobCoordinatorService.java @@ -195,8 +195,14 @@ public Optional getJob(Source source, Store store) { return Optional.of(jobs.get(0)); } - // TODO: Put in a util somewhere? - private static List featureSetsFromProto(List protos) { - return protos.stream().map(FeatureSet::fromProto).collect(Collectors.toList()); + // TODO: optimize this to make less calls to the database. + private List featureSetsFromProto(List protos) { + return protos.stream() + .map(FeatureSetProto.FeatureSet::getSpec) + .map( + fs -> + featureSetRepository.findFeatureSetByNameAndProject_NameAndVersion( + fs.getName(), fs.getProject(), fs.getVersion())) + .collect(Collectors.toList()); } } diff --git a/core/src/main/java/feast/core/service/SpecService.java b/core/src/main/java/feast/core/service/SpecService.java index 8fec6ac5112..4a068cba353 100644 --- a/core/src/main/java/feast/core/service/SpecService.java +++ b/core/src/main/java/feast/core/service/SpecService.java @@ -33,6 +33,7 @@ import feast.core.CoreServiceProto.UpdateStoreRequest; import feast.core.CoreServiceProto.UpdateStoreResponse; import feast.core.FeatureSetProto; +import feast.core.FeatureSetProto.FeatureSetStatus; import feast.core.SourceProto; import feast.core.StoreProto; import feast.core.StoreProto.Store.Subscription; @@ -335,6 +336,7 @@ public ApplyFeatureSetResponse applyFeatureSet(FeatureSetProto.FeatureSet newFea // Build a new FeatureSet object which includes the new properties FeatureSet featureSet = FeatureSet.fromProto(newFeatureSet); + featureSet.setStatus(FeatureSetStatus.STATUS_PENDING.toString()); if (newFeatureSet.getSpec().getSource() == SourceProto.Source.getDefaultInstance()) { featureSet.setSource(defaultSource); } diff --git a/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java b/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java index 59fdc32b20f..26cf331c13b 100644 --- a/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java +++ b/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java @@ -25,6 +25,7 @@ import static org.mockito.Mockito.when; import static org.mockito.MockitoAnnotations.initMocks; +import com.google.common.collect.Lists; import com.google.protobuf.InvalidProtocolBufferException; import feast.core.CoreServiceProto.ListFeatureSetsRequest.Filter; import feast.core.CoreServiceProto.ListFeatureSetsResponse; @@ -194,6 +195,13 @@ public void shouldGenerateAndSubmitJobsIfAny() throws InvalidProtocolBufferExcep when(specService.listStores(any())) .thenReturn(ListStoresResponse.newBuilder().addStore(store).build()); + for (FeatureSetProto.FeatureSet fs : Lists.newArrayList(featureSet1, featureSet2)) { + FeatureSetSpec spec = fs.getSpec(); + when(featureSetRepository.findFeatureSetByNameAndProject_NameAndVersion( + spec.getName(), spec.getProject(), spec.getVersion())) + .thenReturn(FeatureSet.fromProto(fs)); + } + when(jobManager.startJob(argThat(new JobMatcher(expectedInput)))).thenReturn(expected); when(jobManager.getRunnerType()).thenReturn(Runner.DATAFLOW); @@ -318,6 +326,12 @@ public void shouldGroupJobsBySource() throws InvalidProtocolBufferException { when(jobManager.startJob(argThat(new JobMatcher(expectedInput1)))).thenReturn(expected1); when(jobManager.startJob(argThat(new JobMatcher(expectedInput2)))).thenReturn(expected2); when(jobManager.getRunnerType()).thenReturn(Runner.DATAFLOW); + for (FeatureSetProto.FeatureSet fs : Lists.newArrayList(featureSet1, featureSet2)) { + FeatureSetSpec spec = fs.getSpec(); + when(featureSetRepository.findFeatureSetByNameAndProject_NameAndVersion( + spec.getName(), spec.getProject(), spec.getVersion())) + .thenReturn(FeatureSet.fromProto(fs)); + } JobCoordinatorService jcs = new JobCoordinatorService( diff --git a/core/src/test/java/feast/core/service/JobServiceTest.java b/core/src/test/java/feast/core/service/JobServiceTest.java index b649181afbf..ba663020191 100644 --- a/core/src/test/java/feast/core/service/JobServiceTest.java +++ b/core/src/test/java/feast/core/service/JobServiceTest.java @@ -41,12 +41,7 @@ import feast.core.dao.JobRepository; import feast.core.job.JobManager; import feast.core.job.Runner; -import feast.core.model.FeatureSet; -import feast.core.model.Field; -import feast.core.model.Job; -import feast.core.model.JobStatus; -import feast.core.model.Source; -import feast.core.model.Store; +import feast.core.model.*; import feast.types.ValueProto.ValueType.Enum; import java.time.Instant; import java.util.ArrayList; @@ -148,8 +143,8 @@ public void setupJobManager() { // dummy model constructorss private FeatureSet newDummyFeatureSet(String name, int version, String project) { - Field feature = TestObjectFactory.CreateFeatureField(name + "_feature", Enum.INT64); - Field entity = TestObjectFactory.CreateEntityField(name + "_entity", Enum.STRING); + Feature feature = TestObjectFactory.CreateFeature(name + "_feature", Enum.INT64); + Entity entity = TestObjectFactory.CreateEntity(name + "_entity", Enum.STRING); FeatureSet fs = TestObjectFactory.CreateFeatureSet( diff --git a/core/src/test/java/feast/core/service/SpecServiceTest.java b/core/src/test/java/feast/core/service/SpecServiceTest.java index bb9f832bd7f..413a97e64b0 100644 --- a/core/src/test/java/feast/core/service/SpecServiceTest.java +++ b/core/src/test/java/feast/core/service/SpecServiceTest.java @@ -48,11 +48,7 @@ import feast.core.dao.ProjectRepository; import feast.core.dao.StoreRepository; import feast.core.exception.RetrievalException; -import feast.core.model.FeatureSet; -import feast.core.model.Field; -import feast.core.model.Project; -import feast.core.model.Source; -import feast.core.model.Store; +import feast.core.model.*; import feast.types.ValueProto.ValueType.Enum; import java.sql.Date; import java.time.Instant; @@ -114,9 +110,9 @@ public void setUp() { FeatureSet featureSet1v3 = newDummyFeatureSet("f1", 3, "project1"); FeatureSet featureSet2v1 = newDummyFeatureSet("f2", 1, "project1"); - Field f3f1 = TestObjectFactory.CreateFeatureField("f3f1", Enum.INT64); - Field f3f2 = TestObjectFactory.CreateFeatureField("f3f2", Enum.INT64); - Field f3e1 = TestObjectFactory.CreateEntityField("f3e1", Enum.STRING); + Feature f3f1 = TestObjectFactory.CreateFeature("f3f1", Enum.INT64); + Feature f3f2 = TestObjectFactory.CreateFeature("f3f2", Enum.INT64); + Entity f3e1 = TestObjectFactory.CreateEntity("f3e1", Enum.STRING); FeatureSet featureSet3v1 = TestObjectFactory.CreateFeatureSet( "f3", "project1", 1, Arrays.asList(f3e1), Arrays.asList(f3f2, f3f1)); @@ -472,9 +468,9 @@ public void applyFeatureSetShouldIncrementFeatureSetVersionIfAlreadyExists() public void applyFeatureSetShouldNotCreateFeatureSetIfFieldsUnordered() throws InvalidProtocolBufferException { - Field f3f1 = TestObjectFactory.CreateFeatureField("f3f1", Enum.INT64); - Field f3f2 = TestObjectFactory.CreateFeatureField("f3f2", Enum.INT64); - Field f3e1 = TestObjectFactory.CreateEntityField("f3e1", Enum.STRING); + Feature f3f1 = TestObjectFactory.CreateFeature("f3f1", Enum.INT64); + Feature f3f2 = TestObjectFactory.CreateFeature("f3f2", Enum.INT64); + Entity f3e1 = TestObjectFactory.CreateEntity("f3e1", Enum.STRING); FeatureSetProto.FeatureSet incomingFeatureSet = (TestObjectFactory.CreateFeatureSet( "f3", "project1", 5, Arrays.asList(f3e1), Arrays.asList(f3f2, f3f1))) @@ -498,46 +494,11 @@ public void applyFeatureSetShouldNotCreateFeatureSetIfFieldsUnordered() public void applyFeatureSetShouldAcceptPresenceShapeAndDomainConstraints() throws InvalidProtocolBufferException { List entitySpecs = new ArrayList<>(); - entitySpecs.add( - EntitySpec.newBuilder() - .setName("entity1") - .setValueType(Enum.INT64) - .setPresence(FeaturePresence.getDefaultInstance()) - .setShape(FixedShape.getDefaultInstance()) - .setDomain("mydomain") - .build()); - entitySpecs.add( - EntitySpec.newBuilder() - .setName("entity2") - .setValueType(Enum.INT64) - .setGroupPresence(FeaturePresenceWithinGroup.getDefaultInstance()) - .setValueCount(ValueCount.getDefaultInstance()) - .setIntDomain(IntDomain.getDefaultInstance()) - .build()); - entitySpecs.add( - EntitySpec.newBuilder() - .setName("entity3") - .setValueType(Enum.FLOAT) - .setPresence(FeaturePresence.getDefaultInstance()) - .setValueCount(ValueCount.getDefaultInstance()) - .setFloatDomain(FloatDomain.getDefaultInstance()) - .build()); - entitySpecs.add( - EntitySpec.newBuilder() - .setName("entity4") - .setValueType(Enum.STRING) - .setPresence(FeaturePresence.getDefaultInstance()) - .setValueCount(ValueCount.getDefaultInstance()) - .setStringDomain(StringDomain.getDefaultInstance()) - .build()); - entitySpecs.add( - EntitySpec.newBuilder() - .setName("entity5") - .setValueType(Enum.BOOL) - .setPresence(FeaturePresence.getDefaultInstance()) - .setValueCount(ValueCount.getDefaultInstance()) - .setBoolDomain(BoolDomain.getDefaultInstance()) - .build()); + entitySpecs.add(EntitySpec.newBuilder().setName("entity1").setValueType(Enum.INT64).build()); + entitySpecs.add(EntitySpec.newBuilder().setName("entity2").setValueType(Enum.INT64).build()); + entitySpecs.add(EntitySpec.newBuilder().setName("entity3").setValueType(Enum.FLOAT).build()); + entitySpecs.add(EntitySpec.newBuilder().setName("entity4").setValueType(Enum.STRING).build()); + entitySpecs.add(EntitySpec.newBuilder().setName("entity5").setValueType(Enum.BOOL).build()); List featureSpecs = new ArrayList<>(); featureSpecs.add( @@ -680,9 +641,9 @@ public void applyFeatureSetShouldUpdateFeatureSetWhenConstraintsAreUpdated() @Test public void applyFeatureSetShouldCreateProjectWhenNotAlreadyExists() throws InvalidProtocolBufferException { - Field f3f1 = TestObjectFactory.CreateFeatureField("f3f1", Enum.INT64); - Field f3f2 = TestObjectFactory.CreateFeatureField("f3f2", Enum.INT64); - Field f3e1 = TestObjectFactory.CreateEntityField("f3e1", Enum.STRING); + Feature f3f1 = TestObjectFactory.CreateFeature("f3f1", Enum.INT64); + Feature f3f2 = TestObjectFactory.CreateFeature("f3f2", Enum.INT64); + Entity f3e1 = TestObjectFactory.CreateEntity("f3e1", Enum.STRING); FeatureSetProto.FeatureSet incomingFeatureSet = (TestObjectFactory.CreateFeatureSet( "f3", "newproject", 5, Arrays.asList(f3e1), Arrays.asList(f3f2, f3f1))) @@ -699,9 +660,9 @@ public void applyFeatureSetShouldCreateProjectWhenNotAlreadyExists() @Test public void applyFeatureSetShouldFailWhenProjectIsArchived() throws InvalidProtocolBufferException { - Field f3f1 = TestObjectFactory.CreateFeatureField("f3f1", Enum.INT64); - Field f3f2 = TestObjectFactory.CreateFeatureField("f3f2", Enum.INT64); - Field f3e1 = TestObjectFactory.CreateEntityField("f3e1", Enum.STRING); + Feature f3f1 = TestObjectFactory.CreateFeature("f3f1", Enum.INT64); + Feature f3f2 = TestObjectFactory.CreateFeature("f3f2", Enum.INT64); + Entity f3e1 = TestObjectFactory.CreateEntity("f3e1", Enum.STRING); FeatureSetProto.FeatureSet incomingFeatureSet = (TestObjectFactory.CreateFeatureSet( "f3", "archivedproject", 5, Arrays.asList(f3e1), Arrays.asList(f3f2, f3f1))) @@ -860,8 +821,8 @@ private FeatureSet newDummyFeatureSet(String name, int version, String project) .setValueType(Enum.STRING) .putLabels("key", "value") .build(); - Field feature = new Field(f1); - Field entity = TestObjectFactory.CreateEntityField("entity", Enum.STRING); + Feature feature = Feature.fromProto(f1); + Entity entity = TestObjectFactory.CreateEntity("entity", Enum.STRING); FeatureSet fs = TestObjectFactory.CreateFeatureSet( diff --git a/core/src/test/java/feast/core/service/TestObjectFactory.java b/core/src/test/java/feast/core/service/TestObjectFactory.java index 966cb8d8163..0476dbe5c2e 100644 --- a/core/src/test/java/feast/core/service/TestObjectFactory.java +++ b/core/src/test/java/feast/core/service/TestObjectFactory.java @@ -18,8 +18,9 @@ import feast.core.FeatureSetProto; import feast.core.SourceProto; +import feast.core.model.Entity; +import feast.core.model.Feature; import feast.core.model.FeatureSet; -import feast.core.model.Field; import feast.core.model.Source; import feast.types.ValueProto; import java.util.HashMap; @@ -37,7 +38,7 @@ public class TestObjectFactory { true); public static FeatureSet CreateFeatureSet( - String name, String project, int version, List entities, List features) { + String name, String project, int version, List entities, List features) { return new FeatureSet( name, project, @@ -50,13 +51,13 @@ public static FeatureSet CreateFeatureSet( FeatureSetProto.FeatureSetStatus.STATUS_READY); } - public static Field CreateFeatureField(String name, ValueProto.ValueType.Enum valueType) { - return new Field( + public static Feature CreateFeature(String name, ValueProto.ValueType.Enum valueType) { + return Feature.fromProto( FeatureSetProto.FeatureSpec.newBuilder().setName(name).setValueType(valueType).build()); } - public static Field CreateEntityField(String name, ValueProto.ValueType.Enum valueType) { - return new Field( + public static Entity CreateEntity(String name, ValueProto.ValueType.Enum valueType) { + return Entity.fromProto( FeatureSetProto.EntitySpec.newBuilder().setName(name).setValueType(valueType).build()); } } diff --git a/protos/feast/core/FeatureSet.proto b/protos/feast/core/FeatureSet.proto index 9b60270a87a..e7e69ede562 100644 --- a/protos/feast/core/FeatureSet.proto +++ b/protos/feast/core/FeatureSet.proto @@ -69,48 +69,8 @@ message EntitySpec { // Name of the entity. string name = 1; - // Value type of the feature. + // Value type of the entity. feast.types.ValueType.Enum value_type = 2; - - // presence_constraints, shape_type and domain_info are referenced from: - // https://github.com/tensorflow/metadata/blob/36f65d1268cbc92cdbcf812ee03dcf47fb53b91e/tensorflow_metadata/proto/v0/schema.proto#L107 - - oneof presence_constraints { - // Constraints on the presence of this feature in the examples. - tensorflow.metadata.v0.FeaturePresence presence = 3; - // Only used in the context of a "group" context, e.g., inside a sequence. - tensorflow.metadata.v0.FeaturePresenceWithinGroup group_presence = 4; - } - - // The shape of the feature which governs the number of values that appear in - // each example. - oneof shape_type { - // The feature has a fixed shape corresponding to a multi-dimensional - // tensor. - tensorflow.metadata.v0.FixedShape shape = 5; - // The feature doesn't have a well defined shape. All we know are limits on - // the minimum and maximum number of values. - tensorflow.metadata.v0.ValueCount value_count = 6; - } - - // Domain for the values of the feature. - oneof domain_info { - // Reference to a domain defined at the schema level. - string domain = 7; - // Inline definitions of domains. - tensorflow.metadata.v0.IntDomain int_domain = 8; - tensorflow.metadata.v0.FloatDomain float_domain = 9; - tensorflow.metadata.v0.StringDomain string_domain = 10; - tensorflow.metadata.v0.BoolDomain bool_domain = 11; - tensorflow.metadata.v0.StructDomain struct_domain = 12; - // Supported semantic domains. - tensorflow.metadata.v0.NaturalLanguageDomain natural_language_domain = 13; - tensorflow.metadata.v0.ImageDomain image_domain = 14; - tensorflow.metadata.v0.MIDDomain mid_domain = 15; - tensorflow.metadata.v0.URLDomain url_domain = 16; - tensorflow.metadata.v0.TimeDomain time_domain = 17; - tensorflow.metadata.v0.TimeOfDayDomain time_of_day_domain = 18; - } } message FeatureSpec { diff --git a/sdk/python/feast/entity.py b/sdk/python/feast/entity.py index 9c5a027b974..012d01631af 100644 --- a/sdk/python/feast/entity.py +++ b/sdk/python/feast/entity.py @@ -29,26 +29,7 @@ def to_proto(self) -> EntityProto: Returns EntitySpec object """ value_type = ValueTypeProto.ValueType.Enum.Value(self.dtype.name) - return EntityProto( - name=self.name, - value_type=value_type, - presence=self.presence, - group_presence=self.group_presence, - shape=self.shape, - value_count=self.value_count, - domain=self.domain, - int_domain=self.int_domain, - float_domain=self.float_domain, - string_domain=self.string_domain, - bool_domain=self.bool_domain, - struct_domain=self.struct_domain, - natural_language_domain=self.natural_language_domain, - image_domain=self.image_domain, - mid_domain=self.mid_domain, - url_domain=self.url_domain, - time_domain=self.time_domain, - time_of_day_domain=self.time_of_day_domain, - ) + return EntityProto(name=self.name, value_type=value_type,) @classmethod def from_proto(cls, entity_proto: EntityProto): @@ -62,7 +43,4 @@ def from_proto(cls, entity_proto: EntityProto): Entity object """ entity = cls(name=entity_proto.name, dtype=ValueType(entity_proto.value_type)) - entity.update_presence_constraints(entity_proto) - entity.update_shape_type(entity_proto) - entity.update_domain_info(entity_proto) return entity diff --git a/sdk/python/feast/feature_set.py b/sdk/python/feast/feature_set.py index 760e947318f..ace7f165de1 100644 --- a/sdk/python/feast/feature_set.py +++ b/sdk/python/feast/feature_set.py @@ -716,6 +716,8 @@ def export_tfx_schema(self) -> schema_pb2.Schema: ] for _, field in self._fields.items(): + if isinstance(field, Entity): + continue feature = schema_pb2.Feature() for attr in attributes_to_copy_from_field_to_feature: if getattr(field, attr) is None: diff --git a/sdk/python/tests/data/tensorflow_metadata/bikeshare_feature_set.yaml b/sdk/python/tests/data/tensorflow_metadata/bikeshare_feature_set.yaml index daa0a35f0ab..48c595712cb 100644 --- a/sdk/python/tests/data/tensorflow_metadata/bikeshare_feature_set.yaml +++ b/sdk/python/tests/data/tensorflow_metadata/bikeshare_feature_set.yaml @@ -3,15 +3,6 @@ spec: entities: - name: station_id valueType: INT64 - intDomain: - min: 1 - max: 5000 - presence: - minFraction: 1.0 - minCount: 1 - shape: - dim: - - size: 1 features: - name: location valueType: STRING diff --git a/sdk/python/tests/data/tensorflow_metadata/bikeshare_schema.json b/sdk/python/tests/data/tensorflow_metadata/bikeshare_schema.json index e7a886053c1..fa9f97cca0d 100644 --- a/sdk/python/tests/data/tensorflow_metadata/bikeshare_schema.json +++ b/sdk/python/tests/data/tensorflow_metadata/bikeshare_schema.json @@ -85,25 +85,6 @@ } ] } - }, - { - "name": "station_id", - "type": "INT", - "presence": { - "minFraction": 1.0, - "minCount": "1" - }, - "int_domain": { - "min": 1, - "max": 5000 - }, - "shape": { - "dim": [ - { - "size": "1" - } - ] - } } ], "stringDomain": [ diff --git a/sdk/python/tests/test_feature_set.py b/sdk/python/tests/test_feature_set.py index 0a7d1ebabea..a2cc12fe113 100644 --- a/sdk/python/tests/test_feature_set.py +++ b/sdk/python/tests/test_feature_set.py @@ -210,9 +210,6 @@ def test_import_tfx_schema(self): feature_set.import_tfx_schema(test_input_schema) # After update - for entity in feature_set.entities: - assert entity.presence is not None - assert entity.shape is not None for feature in feature_set.features: assert feature.presence is not None assert feature.shape is not None From 4db87f3b475e2217d36f13c7cb81abba391a8c10 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Tue, 5 May 2020 12:59:14 +0800 Subject: [PATCH 146/176] Clean up Docker Compose and add test (#668) * Modularize Docker Compose files * Add docker compose test * Add GitHub Actions Docker Compose workflow * Fix typos and environment file * Get docker container address from docker * Externalize startup script for docker container * Remove unnecessary Feast SDK installation * Clean up Basic notebook parameters * Small changes (whitespace and comments) * Add wait-for-it bash script * Shut down docker compose when done * Allow docker-compose files to run independently * Allow Jupyter Notebook clone step to always succeed --- .github/workflows/docker_compose_tests.yml | 12 ++ .../{unit-tests.yml => unit_tests.yml} | 0 examples/basic/basic.ipynb | 59 +----- infra/docker-compose/.env.sample | 16 +- infra/docker-compose/docker-compose.batch.yml | 29 +++ .../docker-compose/docker-compose.online.yml | 23 +++ infra/docker-compose/docker-compose.yml | 58 +----- .../gcp-service-accounts/placeholder.json | 5 +- infra/docker-compose/jupyter/startup.sh | 15 ++ infra/scripts/test-docker-compose.sh | 35 ++++ infra/scripts/wait-for-it.sh | 183 ++++++++++++++++++ 11 files changed, 321 insertions(+), 114 deletions(-) create mode 100644 .github/workflows/docker_compose_tests.yml rename .github/workflows/{unit-tests.yml => unit_tests.yml} (100%) create mode 100644 infra/docker-compose/docker-compose.batch.yml create mode 100644 infra/docker-compose/docker-compose.online.yml create mode 100755 infra/docker-compose/jupyter/startup.sh create mode 100755 infra/scripts/test-docker-compose.sh create mode 100755 infra/scripts/wait-for-it.sh diff --git a/.github/workflows/docker_compose_tests.yml b/.github/workflows/docker_compose_tests.yml new file mode 100644 index 00000000000..28411b88802 --- /dev/null +++ b/.github/workflows/docker_compose_tests.yml @@ -0,0 +1,12 @@ +name: docker compose tests + +on: [push, pull_request] + +jobs: + basic-redis-e2e-tests-docker-compose: + runs-on: ubuntu-latest + name: basic redis e2e tests on docker compose + steps: + - uses: actions/checkout@v2 + - name: test docker compose + run: ./infra/scripts/test-docker-compose.sh \ No newline at end of file diff --git a/.github/workflows/unit-tests.yml b/.github/workflows/unit_tests.yml similarity index 100% rename from .github/workflows/unit-tests.yml rename to .github/workflows/unit_tests.yml diff --git a/examples/basic/basic.ipynb b/examples/basic/basic.ipynb index a6feb0ef13a..921577fb085 100644 --- a/examples/basic/basic.ipynb +++ b/examples/basic/basic.ipynb @@ -35,16 +35,13 @@ "import os\n", "\n", "# Feast Core acts as the central feature registry\n", - "FEAST_CORE_URL = os.getenv('FEAST_CORE_URL', 'core:6565')\n", + "FEAST_CORE_URL = os.getenv('FEAST_CORE_URL', 'localhost:6565')\n", "\n", "# Feast Online Serving allows for the retrieval of real-time feature data\n", - "FEAST_ONLINE_SERVING_URL = os.getenv('FEAST_ONLINE_SERVING_URL', 'online-serving:6566')\n", + "FEAST_ONLINE_SERVING_URL = os.getenv('FEAST_ONLINE_SERVING_URL', 'localhost:6566')\n", "\n", "# Feast Batch Serving allows for the retrieval of historical feature data\n", - "FEAST_BATCH_SERVING_URL = os.getenv('FEAST_BATCH_SERVING_URL', 'batch-serving:6567')\n", - "\n", - "# PYTHON_REPOSITORY_PATH is the path to the Python SDK inside the Feast Git Repo\n", - "PYTHON_REPOSITORY_PATH = os.getenv('PYTHON_REPOSITORY_PATH', '../../')" + "FEAST_BATCH_SERVING_URL = os.getenv('FEAST_BATCH_SERVING_URL', 'localhost:6567')" ] }, { @@ -67,44 +64,7 @@ "metadata": {}, "outputs": [], "source": [ - "!python -m pip install --ignore-installed --upgrade feast" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "(Alternative) Install from local repository" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "import sys\n", - "os.environ['PYTHON_SDK_PATH'] = os.path.join(PYTHON_REPOSITORY_PATH, 'sdk/python')\n", - "sys.path.append(os.environ['PYTHON_SDK_PATH'])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "!echo $PYTHON_SDK_PATH" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "!python -m pip install --ignore-installed --upgrade -e ${PYTHON_SDK_PATH}" + "!pip install feast" ] }, { @@ -501,17 +461,8 @@ "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.4" - }, - "pycharm": { - "stem_cell": { - "cell_type": "raw", - "metadata": { - "collapsed": false - }, - "source": [] - } } }, "nbformat": 4, "nbformat_minor": 2 -} +} \ No newline at end of file diff --git a/infra/docker-compose/.env.sample b/infra/docker-compose/.env.sample index c8652e8fe0c..8ca7ca008fb 100644 --- a/infra/docker-compose/.env.sample +++ b/infra/docker-compose/.env.sample @@ -1,21 +1,23 @@ # General COMPOSE_PROJECT_NAME=feast FEAST_VERSION=latest +FEAST_REPOSITORY_VERSION=v0.4.7 # Feast Core FEAST_CORE_IMAGE=gcr.io/kf-feast/feast-core FEAST_CORE_CONFIG=direct-runner.yml FEAST_CORE_GCP_SERVICE_ACCOUNT_KEY=placeholder.json -# Feast Serving -FEAST_SERVING_IMAGE=gcr.io/kf-feast/feast-serving -FEAST_ONLINE_SERVING_CONFIG=online-serving.yml -FEAST_ONLINE_STORE_CONFIG=redis-store.yml +# Feast Serving Batch (BigQuery) FEAST_BATCH_SERVING_CONFIG=batch-serving.yml FEAST_BATCH_STORE_CONFIG=bq-store.yml FEAST_BATCH_SERVING_GCP_SERVICE_ACCOUNT_KEY=placeholder.json -FEAST_JOB_STAGING_LOCATION=gs://your-gcs-bucket/staging +FEAST_BATCH_JOB_STAGING_LOCATION=gs://your-gcs-bucket/staging -# Jupyter -FEAST_JUPYTER_GCP_SERVICE_ACCOUNT_KEY=placeholder.json +# Feast Serving Online (Redis) +FEAST_SERVING_IMAGE=gcr.io/kf-feast/feast-serving +FEAST_ONLINE_SERVING_CONFIG=online-serving.yml +FEAST_ONLINE_STORE_CONFIG=redis-store.yml +# Jupyter +FEAST_JUPYTER_GCP_SERVICE_ACCOUNT_KEY=placeholder.json \ No newline at end of file diff --git a/infra/docker-compose/docker-compose.batch.yml b/infra/docker-compose/docker-compose.batch.yml new file mode 100644 index 00000000000..247dd0b6719 --- /dev/null +++ b/infra/docker-compose/docker-compose.batch.yml @@ -0,0 +1,29 @@ +version: "3.7" + +services: + batch-serving: + image: ${FEAST_SERVING_IMAGE}:${FEAST_VERSION} + volumes: + - ./serving/${FEAST_BATCH_SERVING_CONFIG}:/etc/feast/application.yml + - ./serving/${FEAST_BATCH_STORE_CONFIG}:/etc/feast/store.yml + - ./gcp-service-accounts/${FEAST_BATCH_SERVING_GCP_SERVICE_ACCOUNT_KEY}:/etc/gcloud/service-accounts/key.json + depends_on: + - redis + ports: + - 6567:6567 + restart: on-failure + environment: + GOOGLE_APPLICATION_CREDENTIALS: /etc/gcloud/service-accounts/key.json + FEAST_JOB_STAGING_LOCATION: ${FEAST_BATCH_JOB_STAGING_LOCATION} + command: + - "java" + - "-Xms1024m" + - "-Xmx1024m" + - "-jar" + - "/opt/feast/feast-serving.jar" + - "--spring.config.location=classpath:/application.yml,file:/etc/feast/application.yml" + + redis: + image: redis:5-alpine + ports: + - "6379:6379" \ No newline at end of file diff --git a/infra/docker-compose/docker-compose.online.yml b/infra/docker-compose/docker-compose.online.yml new file mode 100644 index 00000000000..ed96f0e0963 --- /dev/null +++ b/infra/docker-compose/docker-compose.online.yml @@ -0,0 +1,23 @@ +version: "3.7" + +services: + online-serving: + image: ${FEAST_SERVING_IMAGE}:${FEAST_VERSION} + volumes: + - ./serving/${FEAST_ONLINE_SERVING_CONFIG}:/etc/feast/application.yml + - ./serving/${FEAST_ONLINE_STORE_CONFIG}:/etc/feast/store.yml + depends_on: + - redis + ports: + - 6566:6566 + restart: on-failure + command: + - java + - -jar + - /opt/feast/feast-serving.jar + - --spring.config.location=classpath:/application.yml,file:/etc/feast/application.yml + + redis: + image: redis:5-alpine + ports: + - "6379:6379" \ No newline at end of file diff --git a/infra/docker-compose/docker-compose.yml b/infra/docker-compose/docker-compose.yml index b44212d0d32..1dda766608e 100644 --- a/infra/docker-compose/docker-compose.yml +++ b/infra/docker-compose/docker-compose.yml @@ -21,68 +21,22 @@ services: - /opt/feast/feast-core.jar - --spring.config.location=classpath:/application.yml,file:/etc/feast/application.yml - online-serving: - image: ${FEAST_SERVING_IMAGE}:${FEAST_VERSION} - volumes: - - ./serving/${FEAST_ONLINE_SERVING_CONFIG}:/etc/feast/application.yml - - ./serving/${FEAST_ONLINE_STORE_CONFIG}:/etc/feast/store.yml - depends_on: - - core - - redis - ports: - - 6566:6566 - restart: on-failure - command: - - java - - -jar - - /opt/feast/feast-serving.jar - - --spring.config.location=classpath:/application.yml,file:/etc/feast/application.yml - - batch-serving: - image: ${FEAST_SERVING_IMAGE}:${FEAST_VERSION} - volumes: - - ./serving/${FEAST_BATCH_SERVING_CONFIG}:/etc/feast/application.yml - - ./serving/${FEAST_BATCH_STORE_CONFIG}:/etc/feast/store.yml - - ./gcp-service-accounts/${FEAST_BATCH_SERVING_GCP_SERVICE_ACCOUNT_KEY}:/etc/gcloud/service-accounts/key.json - depends_on: - - core - - redis - ports: - - 6567:6567 - restart: on-failure - environment: - GOOGLE_APPLICATION_CREDENTIALS: /etc/gcloud/service-accounts/key.json - FEAST_JOB_STAGING_LOCATION: ${FEAST_JOB_STAGING_LOCATION} - command: - - "java" - - "-Xms1024m" - - "-Xmx1024m" - - "-jar" - - "/opt/feast/feast-serving.jar" - - "--spring.config.location=classpath:/application.yml,file:/etc/feast/application.yml" - jupyter: - image: jupyter/datascience-notebook:63d0df23b673 + image: jupyter/minimal-notebook:619e9cc2fc07 volumes: - - ../../:/home/jovyan/feast - ./gcp-service-accounts/${FEAST_JUPYTER_GCP_SERVICE_ACCOUNT_KEY}:/etc/gcloud/service-accounts/key.json + - ./jupyter/startup.sh:/etc/startup.sh depends_on: - core - - online-serving environment: FEAST_CORE_URL: core:6565 - FEAST_SERVING_URL: online-serving:6566 + FEAST_ONLINE_SERVING_URL: online-serving:6566 + FEAST_BATCH_SERVING_URL: batch-serving:6567 GOOGLE_APPLICATION_CREDENTIALS: /etc/gcloud/service-accounts/key.json + FEAST_REPOSITORY_VERSION: ${FEAST_REPOSITORY_VERSION} ports: - 8888:8888 - command: - - start-notebook.sh - - --NotebookApp.token='' - - redis: - image: redis:5-alpine - ports: - - "6379:6379" + command: ["/etc/startup.sh"] kafka: image: confluentinc/cp-kafka:5.2.1 diff --git a/infra/docker-compose/gcp-service-accounts/placeholder.json b/infra/docker-compose/gcp-service-accounts/placeholder.json index 9e26dfeeb6e..5609d6923cf 100644 --- a/infra/docker-compose/gcp-service-accounts/placeholder.json +++ b/infra/docker-compose/gcp-service-accounts/placeholder.json @@ -1 +1,4 @@ -{} \ No newline at end of file +{ + "type": "service_account", + "project_id": "just-some-project" +} \ No newline at end of file diff --git a/infra/docker-compose/jupyter/startup.sh b/infra/docker-compose/jupyter/startup.sh new file mode 100755 index 00000000000..edca4411796 --- /dev/null +++ b/infra/docker-compose/jupyter/startup.sh @@ -0,0 +1,15 @@ +#!/usr/bin/env bash + +set -ex + +# Clone Feast repository into Jupyter container +git clone -b ${FEAST_REPOSITORY_VERSION} --single-branch https://github.com/gojek/feast.git || true + +# Install CI requirements (only needed for running tests) +pip install -r feast/sdk/python/requirements-ci.txt + +# Install Feast SDK +pip install -e feast/sdk/python -U + +# Start Jupyter Notebook +start-notebook.sh --NotebookApp.token='' \ No newline at end of file diff --git a/infra/scripts/test-docker-compose.sh b/infra/scripts/test-docker-compose.sh new file mode 100755 index 00000000000..40a85900c2f --- /dev/null +++ b/infra/scripts/test-docker-compose.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash + +set -e + +echo " +============================================================ +Running Docker Compose tests with pytest at 'tests/e2e' +============================================================ +" + +export PROJECT_ROOT_DIR=$(git rev-parse --show-toplevel) +export COMPOSE_INTERACTIVE_NO_CLI=1 + +# Create Docker Compose configuration file +cd ${PROJECT_ROOT_DIR}/infra/docker-compose/ +cp .env.sample .env + +# Start Docker Compose containers +docker-compose -f docker-compose.yml -f docker-compose.online.yml up -d + +# Get Jupyter container IP address +export JUPYTER_DOCKER_CONTAINER_IP_ADDRESS=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' feast_jupyter_1) + +# Print Jupyter container information +docker inspect feast_jupyter_1 +docker logs feast_jupyter_1 + +# Wait for Jupyter Notebook Container to come online +${PROJECT_ROOT_DIR}/infra/scripts/wait-for-it.sh ${JUPYTER_DOCKER_CONTAINER_IP_ADDRESS}:8888 --timeout=300 + +# Run e2e tests for Redis +docker exec feast_jupyter_1 bash -c 'cd feast/tests/e2e/ && pytest -s basic-ingest-redis-serving.py --core_url core:6565 --serving_url=online-serving:6566' + +# Shut down docker-compose images +docker-compose -f docker-compose.yml -f docker-compose.online.yml down \ No newline at end of file diff --git a/infra/scripts/wait-for-it.sh b/infra/scripts/wait-for-it.sh new file mode 100755 index 00000000000..51942ce6dc4 --- /dev/null +++ b/infra/scripts/wait-for-it.sh @@ -0,0 +1,183 @@ +#!/usr/bin/env bash +# Use this script to test if a given TCP host/port are available +# Source: https://github.com/vishnubob/wait-for-it + +WAITFORIT_cmdname=${0##*/} + +echoerr() { if [[ $WAITFORIT_QUIET -ne 1 ]]; then echo "$@" 1>&2; fi } + +usage() +{ + cat << USAGE >&2 +Usage: + $WAITFORIT_cmdname host:port [-s] [-t timeout] [-- command args] + -h HOST | --host=HOST Host or IP under test + -p PORT | --port=PORT TCP port under test + Alternatively, you specify the host and port as host:port + -s | --strict Only execute subcommand if the test succeeds + -q | --quiet Don't output any status messages + -t TIMEOUT | --timeout=TIMEOUT + Timeout in seconds, zero for no timeout + -- COMMAND ARGS Execute command with args after the test finishes +USAGE + exit 1 +} + +wait_for() +{ + if [[ $WAITFORIT_TIMEOUT -gt 0 ]]; then + echoerr "$WAITFORIT_cmdname: waiting $WAITFORIT_TIMEOUT seconds for $WAITFORIT_HOST:$WAITFORIT_PORT" + else + echoerr "$WAITFORIT_cmdname: waiting for $WAITFORIT_HOST:$WAITFORIT_PORT without a timeout" + fi + WAITFORIT_start_ts=$(date +%s) + while : + do + if [[ $WAITFORIT_ISBUSY -eq 1 ]]; then + nc -z $WAITFORIT_HOST $WAITFORIT_PORT + WAITFORIT_result=$? + else + (echo > /dev/tcp/$WAITFORIT_HOST/$WAITFORIT_PORT) >/dev/null 2>&1 + WAITFORIT_result=$? + fi + if [[ $WAITFORIT_result -eq 0 ]]; then + WAITFORIT_end_ts=$(date +%s) + echoerr "$WAITFORIT_cmdname: $WAITFORIT_HOST:$WAITFORIT_PORT is available after $((WAITFORIT_end_ts - WAITFORIT_start_ts)) seconds" + break + fi + sleep 1 + done + return $WAITFORIT_result +} + +wait_for_wrapper() +{ + # In order to support SIGINT during timeout: http://unix.stackexchange.com/a/57692 + if [[ $WAITFORIT_QUIET -eq 1 ]]; then + timeout $WAITFORIT_BUSYTIMEFLAG $WAITFORIT_TIMEOUT $0 --quiet --child --host=$WAITFORIT_HOST --port=$WAITFORIT_PORT --timeout=$WAITFORIT_TIMEOUT & + else + timeout $WAITFORIT_BUSYTIMEFLAG $WAITFORIT_TIMEOUT $0 --child --host=$WAITFORIT_HOST --port=$WAITFORIT_PORT --timeout=$WAITFORIT_TIMEOUT & + fi + WAITFORIT_PID=$! + trap "kill -INT -$WAITFORIT_PID" INT + wait $WAITFORIT_PID + WAITFORIT_RESULT=$? + if [[ $WAITFORIT_RESULT -ne 0 ]]; then + echoerr "$WAITFORIT_cmdname: timeout occurred after waiting $WAITFORIT_TIMEOUT seconds for $WAITFORIT_HOST:$WAITFORIT_PORT" + fi + return $WAITFORIT_RESULT +} + +# process arguments +while [[ $# -gt 0 ]] +do + case "$1" in + *:* ) + WAITFORIT_hostport=(${1//:/ }) + WAITFORIT_HOST=${WAITFORIT_hostport[0]} + WAITFORIT_PORT=${WAITFORIT_hostport[1]} + shift 1 + ;; + --child) + WAITFORIT_CHILD=1 + shift 1 + ;; + -q | --quiet) + WAITFORIT_QUIET=1 + shift 1 + ;; + -s | --strict) + WAITFORIT_STRICT=1 + shift 1 + ;; + -h) + WAITFORIT_HOST="$2" + if [[ $WAITFORIT_HOST == "" ]]; then break; fi + shift 2 + ;; + --host=*) + WAITFORIT_HOST="${1#*=}" + shift 1 + ;; + -p) + WAITFORIT_PORT="$2" + if [[ $WAITFORIT_PORT == "" ]]; then break; fi + shift 2 + ;; + --port=*) + WAITFORIT_PORT="${1#*=}" + shift 1 + ;; + -t) + WAITFORIT_TIMEOUT="$2" + if [[ $WAITFORIT_TIMEOUT == "" ]]; then break; fi + shift 2 + ;; + --timeout=*) + WAITFORIT_TIMEOUT="${1#*=}" + shift 1 + ;; + --) + shift + WAITFORIT_CLI=("$@") + break + ;; + --help) + usage + ;; + *) + echoerr "Unknown argument: $1" + usage + ;; + esac +done + +if [[ "$WAITFORIT_HOST" == "" || "$WAITFORIT_PORT" == "" ]]; then + echoerr "Error: you need to provide a host and port to test." + usage +fi + +WAITFORIT_TIMEOUT=${WAITFORIT_TIMEOUT:-15} +WAITFORIT_STRICT=${WAITFORIT_STRICT:-0} +WAITFORIT_CHILD=${WAITFORIT_CHILD:-0} +WAITFORIT_QUIET=${WAITFORIT_QUIET:-0} + +# Check to see if timeout is from busybox? +WAITFORIT_TIMEOUT_PATH=$(type -p timeout) +WAITFORIT_TIMEOUT_PATH=$(realpath $WAITFORIT_TIMEOUT_PATH 2>/dev/null || readlink -f $WAITFORIT_TIMEOUT_PATH) + +WAITFORIT_BUSYTIMEFLAG="" +if [[ $WAITFORIT_TIMEOUT_PATH =~ "busybox" ]]; then + WAITFORIT_ISBUSY=1 + # Check if busybox timeout uses -t flag + # (recent Alpine versions don't support -t anymore) + if timeout &>/dev/stdout | grep -q -e '-t '; then + WAITFORIT_BUSYTIMEFLAG="-t" + fi +else + WAITFORIT_ISBUSY=0 +fi + +if [[ $WAITFORIT_CHILD -gt 0 ]]; then + wait_for + WAITFORIT_RESULT=$? + exit $WAITFORIT_RESULT +else + if [[ $WAITFORIT_TIMEOUT -gt 0 ]]; then + wait_for_wrapper + WAITFORIT_RESULT=$? + else + wait_for + WAITFORIT_RESULT=$? + fi +fi + +if [[ $WAITFORIT_CLI != "" ]]; then + if [[ $WAITFORIT_RESULT -ne 0 && $WAITFORIT_STRICT -eq 1 ]]; then + echoerr "$WAITFORIT_cmdname: strict mode, refusing to execute subprocess" + exit $WAITFORIT_RESULT + fi + exec "${WAITFORIT_CLI[@]}" +else + exit $WAITFORIT_RESULT +fi \ No newline at end of file From f4fbbc08ea40125c4e081dc0ea7d84f4264c04f1 Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Tue, 5 May 2020 18:29:15 +0800 Subject: [PATCH 147/176] DataflowJobManager updates existing job instance (#678) --- .../core/job/dataflow/DataflowJobManager.java | 42 ++++++++++--------- 1 file changed, 22 insertions(+), 20 deletions(-) diff --git a/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java b/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java index db9a7f90707..08b6bfbd01f 100644 --- a/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java +++ b/core/src/main/java/feast/core/job/dataflow/DataflowJobManager.java @@ -47,7 +47,6 @@ import java.util.Collections; import java.util.List; import java.util.Map; -import java.util.stream.Collectors; import lombok.extern.slf4j.Slf4j; import org.apache.beam.runners.dataflow.DataflowPipelineJob; import org.apache.beam.runners.dataflow.DataflowRunner; @@ -120,12 +119,15 @@ public Job startJob(Job job) { for (FeatureSet featureSet : job.getFeatureSets()) { featureSetProtos.add(featureSet.toProto()); } - return submitDataflowJob( - job.getId(), - featureSetProtos, - job.getSource().toProto(), - job.getStore().toProto(), - false); + String extId = + submitDataflowJob( + job.getId(), + featureSetProtos, + job.getSource().toProto(), + job.getStore().toProto(), + false); + job.setExtId(extId); + return job; } catch (InvalidProtocolBufferException e) { log.error(e.getMessage()); @@ -150,8 +152,17 @@ public Job updateJob(Job job) { for (FeatureSet featureSet : job.getFeatureSets()) { featureSetProtos.add(featureSet.toProto()); } - return submitDataflowJob( - job.getId(), featureSetProtos, job.getSource().toProto(), job.getStore().toProto(), true); + + String extId = + submitDataflowJob( + job.getId(), + featureSetProtos, + job.getSource().toProto(), + job.getStore().toProto(), + true); + + job.setExtId(extId); + return job; } catch (InvalidProtocolBufferException e) { log.error(e.getMessage()); throw new IllegalArgumentException( @@ -236,7 +247,7 @@ public JobStatus getJobStatus(Job job) { return JobStatus.UNKNOWN; } - private Job submitDataflowJob( + private String submitDataflowJob( String jobName, List featureSetProtos, SourceProto.Source source, @@ -245,17 +256,8 @@ private Job submitDataflowJob( try { ImportOptions pipelineOptions = getPipelineOptions(jobName, featureSetProtos, sink, update); DataflowPipelineJob pipelineResult = runPipeline(pipelineOptions); - List featureSets = - featureSetProtos.stream().map(FeatureSet::fromProto).collect(Collectors.toList()); String jobId = waitForJobToRun(pipelineResult); - return new Job( - jobName, - jobId, - getRunnerType(), - Source.fromProto(source), - Store.fromProto(sink), - featureSets, - JobStatus.PENDING); + return jobId; } catch (Exception e) { log.error("Error submitting job", e); throw new JobExecutionException(String.format("Error running ingestion job: %s", e), e); From a1937c374a4e39b7a75d828e7b7c3b87a64d9d6e Mon Sep 17 00:00:00 2001 From: Khor Shu Heng <32997938+khorshuheng@users.noreply.github.com> Date: Wed, 6 May 2020 16:39:15 +0800 Subject: [PATCH 148/176] Ensure that generated python code are considered as module (#679) Co-authored-by: Khor Shu Heng --- .gitignore | 10 +++++----- .prow/config.yaml | 2 +- sdk/python/feast/core/__init__.py | 0 sdk/python/feast/serving/__init__.py | 0 sdk/python/feast/storage/__init__.py | 0 sdk/python/feast/types/__init__.py | 0 sdk/python/tensorflow_metadata/__init__.py | 0 sdk/python/tensorflow_metadata/proto/__init__.py | 0 sdk/python/tensorflow_metadata/proto/v0/__init__.py | 0 9 files changed, 6 insertions(+), 6 deletions(-) create mode 100644 sdk/python/feast/core/__init__.py create mode 100644 sdk/python/feast/serving/__init__.py create mode 100644 sdk/python/feast/storage/__init__.py create mode 100644 sdk/python/feast/types/__init__.py create mode 100644 sdk/python/tensorflow_metadata/__init__.py create mode 100644 sdk/python/tensorflow_metadata/proto/__init__.py create mode 100644 sdk/python/tensorflow_metadata/proto/v0/__init__.py diff --git a/.gitignore b/.gitignore index d034c89dccf..b2c3f77f8c3 100644 --- a/.gitignore +++ b/.gitignore @@ -179,8 +179,8 @@ dmypy.json .flattened-pom.xml sdk/python/docs/html -sdk/python/feast/core/ -sdk/python/feast/serving/ -sdk/python/feast/storage/ -sdk/python/feast/types/ -sdk/python/tensorflow_metadata \ No newline at end of file + +# Generated python code +*_pb2.py +*_pb2.pyi +*_pb2_grpc.py diff --git a/.prow/config.yaml b/.prow/config.yaml index af41aab759f..9401ba8334f 100644 --- a/.prow/config.yaml +++ b/.prow/config.yaml @@ -236,7 +236,7 @@ postsubmits: - sh - -c - | - infra/scripts/publish-python-sdk.sh \ + make compile-protos-python && infra/scripts/publish-python-sdk.sh \ --directory-path sdk/python --repository pypi volumeMounts: - name: pypirc diff --git a/sdk/python/feast/core/__init__.py b/sdk/python/feast/core/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/sdk/python/feast/serving/__init__.py b/sdk/python/feast/serving/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/sdk/python/feast/storage/__init__.py b/sdk/python/feast/storage/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/sdk/python/feast/types/__init__.py b/sdk/python/feast/types/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/sdk/python/tensorflow_metadata/__init__.py b/sdk/python/tensorflow_metadata/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/sdk/python/tensorflow_metadata/proto/__init__.py b/sdk/python/tensorflow_metadata/proto/__init__.py new file mode 100644 index 00000000000..e69de29bb2d diff --git a/sdk/python/tensorflow_metadata/proto/v0/__init__.py b/sdk/python/tensorflow_metadata/proto/v0/__init__.py new file mode 100644 index 00000000000..e69de29bb2d From d1aebb923f7e79e4aa62663ce23216979d967600 Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Fri, 8 May 2020 08:58:15 +0800 Subject: [PATCH 149/176] Add grpc health probe implementation to core (#680) --- .../feast/core/grpc/HealthServiceImpl.java | 54 +++++++++++++ .../feast/core/http/HealthController.java | 71 ---------------- .../feast/core/http/HealthControllerTest.java | 80 ------------------- .../feast-core/templates/deployment.yaml | 10 +-- .../feast/charts/feast-core/values.yaml | 4 +- infra/docker/core/Dockerfile | 11 +++ 6 files changed, 72 insertions(+), 158 deletions(-) create mode 100644 core/src/main/java/feast/core/grpc/HealthServiceImpl.java delete mode 100644 core/src/main/java/feast/core/http/HealthController.java delete mode 100644 core/src/test/java/feast/core/http/HealthControllerTest.java diff --git a/core/src/main/java/feast/core/grpc/HealthServiceImpl.java b/core/src/main/java/feast/core/grpc/HealthServiceImpl.java new file mode 100644 index 00000000000..3bd2f8748fe --- /dev/null +++ b/core/src/main/java/feast/core/grpc/HealthServiceImpl.java @@ -0,0 +1,54 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.core.grpc; + +import feast.core.service.AccessManagementService; +import io.grpc.Status; +import io.grpc.health.v1.HealthGrpc.HealthImplBase; +import io.grpc.health.v1.HealthProto.HealthCheckRequest; +import io.grpc.health.v1.HealthProto.HealthCheckResponse; +import io.grpc.health.v1.HealthProto.HealthCheckResponse.ServingStatus; +import io.grpc.stub.StreamObserver; +import lombok.extern.slf4j.Slf4j; +import org.lognet.springboot.grpc.GRpcService; +import org.springframework.beans.factory.annotation.Autowired; + +@Slf4j +@GRpcService +public class HealthServiceImpl extends HealthImplBase { + private final AccessManagementService accessManagementService; + + @Autowired + public HealthServiceImpl(AccessManagementService accessManagementService) { + this.accessManagementService = accessManagementService; + } + + @Override + public void check( + HealthCheckRequest request, StreamObserver responseObserver) { + try { + accessManagementService.listProjects(); + responseObserver.onNext( + HealthCheckResponse.newBuilder().setStatus(ServingStatus.SERVING).build()); + responseObserver.onCompleted(); + } catch (Exception e) { + log.error("Health Check: unable to retrieve projects.\nError: %s", e); + responseObserver.onError( + Status.INTERNAL.withDescription(e.getMessage()).withCause(e).asRuntimeException()); + } + } +} diff --git a/core/src/main/java/feast/core/http/HealthController.java b/core/src/main/java/feast/core/http/HealthController.java deleted file mode 100644 index 2451ed793ed..00000000000 --- a/core/src/main/java/feast/core/http/HealthController.java +++ /dev/null @@ -1,71 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.core.http; - -import static org.springframework.http.HttpStatus.INTERNAL_SERVER_ERROR; - -import java.sql.Connection; -import java.sql.SQLException; -import javax.sql.DataSource; -import lombok.extern.slf4j.Slf4j; -import org.springframework.beans.factory.annotation.Autowired; -import org.springframework.http.HttpStatus; -import org.springframework.http.ResponseEntity; -import org.springframework.web.bind.annotation.RequestMapping; -import org.springframework.web.bind.annotation.RequestMethod; -import org.springframework.web.bind.annotation.RestController; - -/** Web http for pod health-check endpoints. */ -@Slf4j -@RestController -public class HealthController { - - private final DataSource db; - - @Autowired - public HealthController(DataSource datasource) { - this.db = datasource; - } - - /** - * /ping endpoint checks if the application is ready to serve traffic by checking if it is able to - * access the metadata db. - */ - @RequestMapping(value = "/ping", method = RequestMethod.GET) - public ResponseEntity ping() { - return ResponseEntity.ok("pong"); - } - - /** - * /healthz endpoint checks if the application is healthy by checking if the application still has - * access to the metadata db. - */ - @RequestMapping(value = "/healthz", method = RequestMethod.GET) - public ResponseEntity healthz() { - try (Connection conn = db.getConnection()) { - if (conn.isValid(10)) { - return ResponseEntity.ok("healthy"); - } - log.error("Unable to reach DB"); - return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR) - .body("Unable to establish connection with DB"); - } catch (SQLException e) { - log.error("Unable to reach DB: {}", e); - return ResponseEntity.status(INTERNAL_SERVER_ERROR).body(e.getMessage()); - } - } -} diff --git a/core/src/test/java/feast/core/http/HealthControllerTest.java b/core/src/test/java/feast/core/http/HealthControllerTest.java deleted file mode 100644 index 2fcd622f34a..00000000000 --- a/core/src/test/java/feast/core/http/HealthControllerTest.java +++ /dev/null @@ -1,80 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.core.http; - -import static org.junit.Assert.assertEquals; -import static org.mockito.Mockito.*; - -import java.sql.Connection; -import java.sql.SQLException; -import javax.sql.DataSource; -import org.junit.Test; -import org.springframework.http.HttpStatus; -import org.springframework.http.ResponseEntity; - -public class HealthControllerTest { - @Test - public void ping() { - HealthController healthController = new HealthController(null); - assertEquals(ResponseEntity.ok("pong"), healthController.ping()); - } - - @Test - public void healthz() { - assertEquals(ResponseEntity.ok("healthy"), mockHealthyController().healthz()); - assertEquals( - ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR) - .body("Unable to establish connection with DB"), - mockUnhealthyControllerBecauseInvalidConn().healthz()); - assertEquals( - ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body("mocked sqlexception"), - mockUnhealthyControllerBecauseSQLException().healthz()); - } - - private HealthController mockHealthyController() { - DataSource mockDataSource = mock(DataSource.class); - Connection mockConnection = mock(Connection.class); - try { - when(mockConnection.isValid(any(int.class))).thenReturn(Boolean.TRUE); - when(mockDataSource.getConnection()).thenReturn(mockConnection); - } catch (Exception e) { - e.printStackTrace(); - } - return new HealthController(mockDataSource); - } - - private HealthController mockUnhealthyControllerBecauseInvalidConn() { - DataSource mockDataSource = mock(DataSource.class); - Connection mockConnection = mock(Connection.class); - try { - when(mockConnection.isValid(any(int.class))).thenReturn(Boolean.FALSE); - when(mockDataSource.getConnection()).thenReturn(mockConnection); - } catch (Exception ignored) { - } - return new HealthController(mockDataSource); - } - - private HealthController mockUnhealthyControllerBecauseSQLException() { - DataSource mockDataSource = mock(DataSource.class); - Connection mockConnection = mock(Connection.class); - try { - when(mockDataSource.getConnection()).thenThrow(new SQLException("mocked sqlexception")); - } catch (SQLException ignored) { - } - return new HealthController(mockDataSource); - } -} diff --git a/infra/charts/feast/charts/feast-core/templates/deployment.yaml b/infra/charts/feast/charts/feast-core/templates/deployment.yaml index 1f4fd996efa..179e3a6a094 100644 --- a/infra/charts/feast/charts/feast-core/templates/deployment.yaml +++ b/infra/charts/feast/charts/feast-core/templates/deployment.yaml @@ -125,9 +125,8 @@ spec: {{- if .Values.livenessProbe.enabled }} livenessProbe: - httpGet: - path: /healthz - port: {{ .Values.service.http.targetPort }} + exec: + command: ["/usr/bin/grpc-health-probe", "-addr=:{{ .Values.service.grpc.targetPort }}"] initialDelaySeconds: {{ .Values.livenessProbe.initialDelaySeconds }} periodSeconds: {{ .Values.livenessProbe.periodSeconds }} successThreshold: {{ .Values.livenessProbe.successThreshold }} @@ -137,9 +136,8 @@ spec: {{- if .Values.readinessProbe.enabled }} readinessProbe: - httpGet: - path: /healthz - port: {{ .Values.service.http.targetPort }} + exec: + command: ["/usr/bin/grpc-health-probe", "-addr=:{{ .Values.service.grpc.targetPort }}"] initialDelaySeconds: {{ .Values.readinessProbe.initialDelaySeconds }} periodSeconds: {{ .Values.readinessProbe.periodSeconds }} successThreshold: {{ .Values.readinessProbe.successThreshold }} diff --git a/infra/charts/feast/charts/feast-core/values.yaml b/infra/charts/feast/charts/feast-core/values.yaml index 5032e8d87ae..cc7bb49f0f9 100644 --- a/infra/charts/feast/charts/feast-core/values.yaml +++ b/infra/charts/feast/charts/feast-core/values.yaml @@ -53,9 +53,11 @@ prometheus: # prometheus.enabled -- Flag to enable scraping of Feast Core metrics enabled: true +# By default we disable the liveness probe, since if the DB fails restarting core will not result +# in application healing. livenessProbe: # livenessProbe.enabled -- Flag to enabled the probe - enabled: true + enabled: false # livenessProbe.initialDelaySeconds -- Delay before the probe is initiated initialDelaySeconds: 60 # livenessProbe.periodSeconds -- How often to perform the probe diff --git a/infra/docker/core/Dockerfile b/infra/docker/core/Dockerfile index 7e469ed7f61..c7ba81a4134 100644 --- a/infra/docker/core/Dockerfile +++ b/infra/docker/core/Dockerfile @@ -25,15 +25,26 @@ RUN mvn --also-make --projects core,ingestion -Drevision=$REVISION \ RUN apt-get -qq update && apt-get -y install unar && \ unar /build/core/target/feast-core-$REVISION.jar -o /build/core/target/ +# +# Download grpc_health_probe to run health check for Feast Serving +# https://kubernetes.io/blog/2018/10/01/health-checking-grpc-servers-on-kubernetes/ +# +RUN wget -q https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.3.1/grpc_health_probe-linux-amd64 \ + -O /usr/bin/grpc-health-probe && \ + chmod +x /usr/bin/grpc-health-probe + # ============================================================ # Build stage 2: Production # ============================================================ FROM openjdk:11-jre as production ARG REVISION=dev + COPY --from=builder /build/core/target/feast-core-$REVISION.jar /opt/feast/feast-core.jar # Required for staging jar dependencies when submitting Dataflow jobs. COPY --from=builder /build/core/target/feast-core-$REVISION /opt/feast/feast-core +COPY --from=builder /usr/bin/grpc-health-probe /usr/bin/grpc-health-probe + CMD ["java",\ "-Xms2048m",\ "-Xmx2048m",\ From b00a622ae3b2528fcbb322c5bac2839a4d455966 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Fri, 8 May 2020 09:58:54 +0800 Subject: [PATCH 150/176] Update README.md docker compose steps. --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index dd44db66ebf..72aff2d5d95 100644 --- a/README.md +++ b/README.md @@ -39,10 +39,10 @@ The following commands will start Feast in online-only mode. git clone https://github.com/gojek/feast.git cd feast/infra/docker-compose cp .env.sample .env -docker-compose up -d +docker-compose -f docker-compose.yml -f docker-compose.online.yml up -d ``` -A [Jupyter Notebook](http://localhost:8888/tree/feast/examples) is now available to start using Feast. +This will start a local Feast deployment with online serving. Additionally, a [Jupyter Notebook](http://localhost:8888/tree/feast/examples) with Feast examples. Please see the links below to set up Feast for batch/historical serving with BigQuery. From 90edb71b0eea37d967dadaa0665793c31a57a4b4 Mon Sep 17 00:00:00 2001 From: Ches Martin Date: Fri, 8 May 2020 09:52:29 +0700 Subject: [PATCH 151/176] Document release steps (#476) * Document release steps Separate what concerns contributors versus what concerns maintainers. * Update release process with changelog guide * docs: Move release process updates into its new dedicated page On master we still have a monolithic `docs/contributing/contributing.md` _and_ redundant copies of some of the information in broken out pages (that have missed some useful updates to the monolith). Needs to get sorted out, but going ahead and moving release process to the dedicated page that now exists for it, in part to port easily to v0.4 docs where the monolith is gone. Co-authored-by: Willem Pienaar <6728866+woop@users.noreply.github.com> --- docs/contributing/contributing.md | 15 ------ docs/contributing/release-process.md | 68 ++++++++++++++++++++++++---- 2 files changed, 58 insertions(+), 25 deletions(-) diff --git a/docs/contributing/contributing.md b/docs/contributing/contributing.md index 0a32ae284a0..3a267dab93d 100644 --- a/docs/contributing/contributing.md +++ b/docs/contributing/contributing.md @@ -525,18 +525,3 @@ And to format: ```text $ make format-python ``` - -## 4. Release process - -Feast uses [semantic versioning](https://semver.org/). - -* Major and minor releases are cut from the `master` branch. -* Whenever a major or minor release is cut, a branch is created for that release. This is called a "release branch". For example if `0.3` is released from `master`, a branch named `v0.3-branch` is created. -* You can create a release branch via the GitHub UI. -* From this branch a git tag is created for the specific release, for example `v0.3.0`. -* Tagging a release will automatically build and push the relevant artifacts to their repositories or package managers \(docker images, Python wheels, etc\). -* A release branch should be substantially _feature complete_ with respect to the intended release. Code that is committed to `master` may be merged or cherry-picked on to a release branch, but code that is directly committed to the release branch should be solely applicable to that release \(and should not be committed back to master\). -* In general, unless you're committing code that only applies to the release stream \(for example, temporary hotfixes, backported security fixes, or image hashes\), you should commit to `master` and then merge or cherry-pick to the release branch. -* It is also important to update the [CHANGELOG.md](https://github.com/gojek/feast/blob/master/CHANGELOG.md) when submitting a new release. This can be in the same PR or a separate PR. -* Finally it is also important to create a [GitHub release](https://github.com/gojek/feast/releases) which includes a summary of important changes as well as any artifacts associated with that release. - diff --git a/docs/contributing/release-process.md b/docs/contributing/release-process.md index d00e9d43987..a4ad7804716 100644 --- a/docs/contributing/release-process.md +++ b/docs/contributing/release-process.md @@ -1,14 +1,62 @@ -# Release Process +# Releasing Feast -Feast uses [semantic versioning](https://semver.org/). +## Versioning policy and branch workflow + +Feast uses [semantic versioning](https://semver.org/). As such, while it is still pre-1.0 breaking changes will happen in minor versions. + +Contributors are encouraged to understand our branch workflow described below, for choosing where to branch when making a change (and thus the merge base for a pull request). * Major and minor releases are cut from the `master` branch. -* Whenever a major or minor release is cut, a branch is created for that release. This is called a "release branch". For example if `0.3` is released from `master`, a branch named `v0.3-branch` is created. -* You can create a release branch via the GitHub UI. -* From this branch a git tag is created for the specific release, for example `v0.3.0`. -* Tagging a release will automatically build and push the relevant artifacts to their repositories or package managers \(docker images, Python wheels, etc\). -* A release branch should be substantially _feature complete_ with respect to the intended release. Code that is committed to `master` may be merged or cherry-picked on to a release branch, but code that is directly committed to the release branch should be solely applicable to that release \(and should not be committed back to master\). -* In general, unless you're committing code that only applies to the release stream \(for example, temporary hotfixes, backported security fixes, or image hashes\), you should commit to `master` and then merge or cherry-pick to the release branch. -* It is also important to update the [CHANGELOG.md](https://github.com/gojek/feast/blob/master/CHANGELOG.md) when submitting a new release. This can be in the same PR or a separate PR. -* Finally it is also important to create a [GitHub release](https://github.com/gojek/feast/releases) which includes a summary of important changes as well as any artifacts associated with that release. +* Each major and minor release has a long-lived maintenance branch, for example `v0.3-branch`. This is called a "release branch". +* From the release branches, patch version releases are tagged, for example `v0.3.0`. + +A release branch should be substantially _feature complete_ with respect to the intended release. Code that is committed to `master` may be merged or cherry-picked on to a release branch, but code that is directly committed to a release branch should be solely applicable to that release \(and should not be committed back to master\). + +In general, unless you're committing code that only applies to a particular release stream \(for example, temporary hotfixes, backported security fixes, or image hashes\), you should base changes from `master` and then merge or cherry-pick to the release branch. + +## Release process + +For Feast maintainers, these are the concrete steps for making a new release. + +1. For a major or minor release, create and check out the release branch for the new stream, e.g. `v0.6-branch`. For a patch version, check out the stream's release branch. +1. Update the [CHANGELOG.md]. See the [Creating a change log](#creating-a-change-log) guide. +1. In the root `pom.xml`, remove `-SNAPSHOT` from the `` property, and commit. +1. Push. For a new release branch, open a PR against master. +1. When CI passes, merge. (Remember _not_ to delete the new release branch). +1. Tag the merge commit with the release version, using a `v` prefix. Push the tag. +1. Bump to the next working version and append `-SNAPSHOT` in `pom.xml`. +1. Commit the POM and open a PR. +1. Create a [GitHub release](https://github.com/gojek/feast/releases) which includes a summary of important changes as well as any artifacts associated with the release. Make sure to include the same change log as added in [CHANGELOG.md]. Use `Feast vX.Y.Z` as the title. +1. Create one final PR to the master branch and also update its [CHANGELOG.md]. + +When a tag that matches a Semantic Version string is pushed, CI will automatically build and push the relevant artifacts to their repositories or package managers \(docker images, Python wheels, etc\). JVM artifacts are promoted from Sonatype OSSRH to Maven Central, but it sometimes takes some time for them to be available. + +[CHANGELOG.md]: https://github.com/gojek/feast/blob/master/CHANGELOG.md + +### Creating a change log +We use an [open source change log generator](https://hub.docker.com/r/ferrarimarco/github-changelog-generator/) to generate change logs. The process still requires a little bit of manual effort. +1. Create a GitHub token as [per these instructions ](https://github.com/github-changelog-generator/github-changelog-generator#github-token). The token is used as an input argument (`-t`) to the changelog generator. +2. The change log generator configuration below will look for unreleased changes on a specific branch. The branch will be `master` for a major/minor release, or a release branch (`v0.4-branch`) for a patch release. You will need to set the branch using the `--release-branch` argument. +3. You should also set the `--future-release` argument. This is the version you are releasing. The version can still be changed at a later date. +4. Update the arguments below and run the command to generate the change log to the console. +``` +docker run -it --rm ferrarimarco/github-changelog-generator \ +--user gojek \ +--project feast \ +--release-branch \ +--future-release \ +--unreleased-only \ +--no-issues \ +--bug-labels kind/bug \ +--enhancement-labels kind/feature \ +--breaking-labels compat/breaking \ +-t \ +--max-issues 1 \ +-o +``` +5. Review each change log item. + - Make sure that sentences are grammatically correct and well formatted (although we will try to enforce this at the PR review stage). + - Make sure that each item is categorized correctly. You will see the following categories: `Breaking changes`, `Implemented enhancements`, `Fixed bugs`, and `Merged pull requests`. Any unlabeled PRs will be found in `Merged pull requests`. It's important to make sure that any `breaking changes`, `enhancements`, or `bug fixes` are pulled up out of `merged pull requests` into the correct category. Housekeeping, tech debt clearing, infra changes, or refactoring do not count as `enhancements`. Only enhancements a user benefits from should be listed in that category. + - Make sure that the "Full Changelog" link is actually comparing the correct tags (normally your released version against the previously version). + - Make sure that release notes and breaking changes are present. From cc118eb49a785c71484c5e3ba783833a86b41726 Mon Sep 17 00:00:00 2001 From: Khor Shu Heng <32997938+khorshuheng@users.noreply.github.com> Date: Fri, 8 May 2020 11:31:16 +0800 Subject: [PATCH 152/176] Fix typo in all types parquet yml file (e2e test) (#683) Co-authored-by: Khor Shu Heng --- tests/e2e/all_types_parquet/all_types_parquet.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/e2e/all_types_parquet/all_types_parquet.yaml b/tests/e2e/all_types_parquet/all_types_parquet.yaml index 2043b6b473d..fa95ce13be0 100644 --- a/tests/e2e/all_types_parquet/all_types_parquet.yaml +++ b/tests/e2e/all_types_parquet/all_types_parquet.yaml @@ -6,7 +6,7 @@ spec: valueType: INT64 features: - name: int32_feature_parquet - valueType: INT64 + valueType: INT32 - name: int64_feature_parquet valueType: INT64 - name: float_feature_parquet From 24531ac91ce9dab8813df6eaa45f1892af285636 Mon Sep 17 00:00:00 2001 From: Andres March Date: Wed, 6 May 2020 20:06:48 -0400 Subject: [PATCH 153/176] Create dev compose to make it easy to run current code whoops Use existing env var overrides --- infra/docker-compose/docker-compose.dev.yml | 70 +++++++++++++++++++++ infra/docker/core/Dockerfile.debug | 0 2 files changed, 70 insertions(+) create mode 100644 infra/docker-compose/docker-compose.dev.yml create mode 100644 infra/docker/core/Dockerfile.debug diff --git a/infra/docker-compose/docker-compose.dev.yml b/infra/docker-compose/docker-compose.dev.yml new file mode 100644 index 00000000000..840c6dd2673 --- /dev/null +++ b/infra/docker-compose/docker-compose.dev.yml @@ -0,0 +1,70 @@ +version: "3.7" + +services: + core: + image: maven:3.6-openjdk-11 + volumes: + - ${HOME}/.m2:/root/.m2:delegated + - ../../.:/code:cached + environment: + DB_HOST: db + FEAST_STREAM_OPTIONS_BOOTSTRAPSERVERS: kafka:9092 + GOOGLE_APPLICATION_CREDENTIALS: /etc/gcloud/service-accounts/key.json + restart: on-failure + depends_on: + - db + - kafka + ports: + - 6565:6565 + + working_dir: /code + command: + - mvn + - -pl + - core + - spring-boot:run + + jupyter: + image: jupyter/minimal-notebook:619e9cc2fc07 + volumes: + - ./gcp-service-accounts/${FEAST_JUPYTER_GCP_SERVICE_ACCOUNT_KEY}:/etc/gcloud/service-accounts/key.json + - ./jupyter/startup.sh:/etc/startup.sh + depends_on: + - core + environment: + FEAST_CORE_URL: core:6565 + FEAST_ONLINE_SERVING_URL: online-serving:6566 + FEAST_BATCH_SERVING_URL: batch-serving:6567 + GOOGLE_APPLICATION_CREDENTIALS: /etc/gcloud/service-accounts/key.json + FEAST_REPOSITORY_VERSION: ${FEAST_REPOSITORY_VERSION} + ports: + - 8888:8888 + command: ["/etc/startup.sh"] + + kafka: + image: confluentinc/cp-kafka:5.2.1 + environment: + KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 + KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 + KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9092,OUTSIDE://localhost:9094 + KAFKA_LISTENERS: INSIDE://:9092,OUTSIDE://:9094 + KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT + KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE + ports: + - "9092:9092" + - "9094:9094" + + depends_on: + - zookeeper + + zookeeper: + image: confluentinc/cp-zookeeper:5.2.1 + environment: + ZOOKEEPER_CLIENT_PORT: 2181 + + db: + image: postgres:12-alpine + environment: + POSTGRES_PASSWORD: password + ports: + - "5432:5432" diff --git a/infra/docker/core/Dockerfile.debug b/infra/docker/core/Dockerfile.debug new file mode 100644 index 00000000000..e69de29bb2d From 255b9b1bf8bac7f8c293f5cfe6b769d14704c3e5 Mon Sep 17 00:00:00 2001 From: Khor Shu Heng <32997938+khorshuheng@users.noreply.github.com> Date: Mon, 11 May 2020 10:00:17 +0800 Subject: [PATCH 154/176] Remove force update flag from e2e test (#688) Co-authored-by: Khor Shu Heng --- tests/e2e/basic-ingest-redis-serving.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/e2e/basic-ingest-redis-serving.py b/tests/e2e/basic-ingest-redis-serving.py index e80c5b7af61..50ec3854553 100644 --- a/tests/e2e/basic-ingest-redis-serving.py +++ b/tests/e2e/basic-ingest-redis-serving.py @@ -552,8 +552,7 @@ def test_all_types_infer_register_ingest_file_success(client, all_types_fs = client.get_feature_set(name="all_types_parquet") # Ingest user embedding data - client.ingest(feature_set=all_types_fs, source=all_types_parquet_file, - force_update=True) + client.ingest(feature_set=all_types_fs, source=all_types_parquet_file) # TODO: rewrite these using python SDK once the labels are implemented there From 1bc776378eb16578bbe5eaa3fbcb3aeebe3e626b Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Mon, 11 May 2020 11:20:17 +0800 Subject: [PATCH 155/176] Remove force_update from python sdk ingest() (#689) --- sdk/python/feast/client.py | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/sdk/python/feast/client.py b/sdk/python/feast/client.py index 0221a79b4b1..d50567deac1 100644 --- a/sdk/python/feast/client.py +++ b/sdk/python/feast/client.py @@ -732,7 +732,6 @@ def ingest( source: Union[pd.DataFrame, str], chunk_size: int = 10000, version: int = None, - force_update: bool = False, max_workers: int = max(CPU_COUNT - 1, 1), disable_progress_bar: bool = False, timeout: int = KAFKA_CHUNK_PRODUCTION_TIMEOUT, @@ -758,10 +757,6 @@ def ingest( version (int): Feature set version. - force_update (bool): - Automatically update feature set based on source data prior to - ingesting. This will also register changes to Feast. - max_workers (int): Number of worker processes to use to encode values. @@ -792,14 +787,6 @@ def ingest( row_count = pq_file.metadata.num_rows - # Update the feature set based on PyArrow table of first row group - if force_update: - feature_set.infer_fields_from_pa( - table=pq_file.read_row_group(0), - discard_unused_fields=True, - replace_existing_features=True, - ) - self.apply(feature_set) current_time = time.time() print("Waiting for feature set to be ready for ingestion...") From 27a37febf31612cd58fe193a428816eb29c191de Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Mon, 11 May 2020 12:49:25 +0800 Subject: [PATCH 156/176] Check for labels in FeatureSet equality (#690) --- core/src/main/java/feast/core/model/Feature.java | 2 +- core/src/main/java/feast/core/model/FeatureSet.java | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/core/src/main/java/feast/core/model/Feature.java b/core/src/main/java/feast/core/model/Feature.java index 38e2d4549ed..487de3ce378 100644 --- a/core/src/main/java/feast/core/model/Feature.java +++ b/core/src/main/java/feast/core/model/Feature.java @@ -168,7 +168,7 @@ public boolean equals(Object o) { } Feature feature = (Feature) o; return Objects.equals(getName(), feature.getName()) - && Objects.equals(labels, feature.labels) + && Objects.equals(getLabels(), feature.getLabels()) && Arrays.equals(getPresence(), feature.getPresence()) && Arrays.equals(getGroupPresence(), feature.getGroupPresence()) && Arrays.equals(getShape(), feature.getShape()) diff --git a/core/src/main/java/feast/core/model/FeatureSet.java b/core/src/main/java/feast/core/model/FeatureSet.java index faaee0e41f6..91bb2bea89f 100644 --- a/core/src/main/java/feast/core/model/FeatureSet.java +++ b/core/src/main/java/feast/core/model/FeatureSet.java @@ -286,6 +286,10 @@ public boolean equalTo(FeatureSet other) { return false; } + if (!getLabels().equals(other.getLabels())) { + return false; + } + if (!project.getName().equals(other.project.getName())) { return false; } From 8d62d9a9d31f39360a51de046c469af5fd9252fa Mon Sep 17 00:00:00 2001 From: Terence Lim Date: Mon, 11 May 2020 19:23:25 +0800 Subject: [PATCH 157/176] Add end-to-end Dataflow test (#675) * add dataflow e2e helm-chart template * add dataflow e2e test * add dataflow e2e-batch test script * update .prow config * build new GCR image for PR * use PULL_PULL_SHA for ref to new PR * set always_run flag to false * Fix kf-feast region * Run helm template for visibility * Maintain one e2e test file with pytest markers * Update command for e2e test with pytest marker * Utilize pytest fixture for infra teardown * Remove redundant command * Format prow test * Shift registering of commit image to presubmit job * Make always_run flag declarative --- .prow/config.yaml | 92 +++++++ .../scripts/test-end-to-end-batch-dataflow.sh | 246 ++++++++++++++++++ infra/scripts/test-end-to-end-batch.sh | 2 +- .../values-end-to-end-batch-dataflow.yaml | 141 ++++++++++ tests/e2e/bq-batch-retrieval.py | 38 ++- 5 files changed, 515 insertions(+), 4 deletions(-) create mode 100644 infra/scripts/test-end-to-end-batch-dataflow.sh create mode 100644 infra/scripts/test-templates/values-end-to-end-batch-dataflow.yaml diff --git a/.prow/config.yaml b/.prow/config.yaml index 9401ba8334f..fd1e2b290d6 100644 --- a/.prow/config.yaml +++ b/.prow/config.yaml @@ -225,6 +225,98 @@ presubmits: branches: - ^v0\.(3|4)-branch$ + - name: test-end-to-end-batch-dataflow + decorate: true + always_run: false + spec: + volumes: + - name: service-account-df + secret: + secretName: feast-service-account + containers: + - image: maven:3.6-jdk-11 + command: ["infra/scripts/test-end-to-end-batch-dataflow.sh"] + resources: + requests: + cpu: "6" + memory: "6144Mi" + volumeMounts: + - name: service-account-df + mountPath: "/etc/service-account" + skip_branches: + - ^v0\.(3|4)-branch$ + + - name: test-end-to-end-batch-dataflow-java-8 + decorate: true + always_run: false + spec: + volumes: + - name: service-account-df + secret: + secretName: feast-service-account + containers: + - image: maven:3.6-jdk-8 + command: ["infra/scripts/test-end-to-end-batch-dataflow.sh"] + resources: + requests: + cpu: "6" + memory: "6144Mi" + volumeMounts: + - name: service-account-df + mountPath: "/etc/service-account" + branches: + - ^v0\.(3|4)-branch$ + + - name: publish-docker-images + decorate: true + always_run: true + spec: + containers: + - image: google/cloud-sdk:273.0.0 + command: + - bash + - -c + - | + infra/scripts/download-maven-cache.sh \ + --archive-uri gs://feast-templocation-kf-feast/.m2.2019-10-24.tar \ + --output-dir $PWD/ + + infra/scripts/publish-docker-image.sh \ + --repository gcr.io/kf-feast/feast-core \ + --tag ${PULL_PULL_SHA:1} \ + --file infra/docker/core/Dockerfile \ + --google-service-account-file /etc/gcloud/service-account.json + + infra/scripts/publish-docker-image.sh \ + --repository gcr.io/kf-feast/feast-serving \ + --tag ${PULL_PULL_SHA:1} \ + --file infra/docker/serving/Dockerfile \ + --google-service-account-file /etc/gcloud/service-account.json + + docker tag gcr.io/kf-feast/feast-core:${PULL_PULL_SHA:1} + docker push gcr.io/kf-feast/feast-core:${PULL_PULL_SHA:1} + + docker tag gcr.io/kf-feast/feast-serving:${PULL_PULL_SHA:1} + docker push gcr.io/kf-feast/feast-serving:${PULL_PULL_SHA:1} + + fi + volumeMounts: + - name: docker-socket + mountPath: /var/run/docker.sock + - name: service-account + mountPath: /etc/gcloud/service-account.json + subPath: service-account.json + readOnly: true + securityContext: + privileged: true + volumes: + - name: docker-socket + hostPath: + path: /var/run/docker.sock + - name: service-account + secret: + secretName: feast-service-account + postsubmits: gojek/feast: - name: publish-python-sdk diff --git a/infra/scripts/test-end-to-end-batch-dataflow.sh b/infra/scripts/test-end-to-end-batch-dataflow.sh new file mode 100644 index 00000000000..9d10b94dae0 --- /dev/null +++ b/infra/scripts/test-end-to-end-batch-dataflow.sh @@ -0,0 +1,246 @@ +#!/usr/bin/env bash +echo "Preparing environment variables..." + +set -e +set -o pipefail + +test -z ${GOOGLE_APPLICATION_CREDENTIALS} && GOOGLE_APPLICATION_CREDENTIALS="/etc/service-account/service-account-df.json" +test -z ${GCLOUD_PROJECT} && GCLOUD_PROJECT="kf-feast" +test -z ${GCLOUD_REGION} && GCLOUD_REGION="us-central1" +test -z ${GCLOUD_NETWORK} && GCLOUD_NETWORK="default" +test -z ${GCLOUD_SUBNET} && GCLOUD_SUBNET="default" +test -z ${TEMP_BUCKET} && TEMP_BUCKET="feast-templocation-kf-feast" +test -z ${K8_CLUSTER_NAME} && K8_CLUSTER_NAME="feast-e2e-dataflow" +test -z ${HELM_RELEASE_NAME} && HELM_RELEASE_NAME="feast-e2e-release" + +echo " +This script will run end-to-end tests for Feast Core and Batch Serving using Dataflow Runner. + +1. Install gcloud SDK and required packages. +2. Create temporary BQ table for Feast Serving. +3. Generate valid names for IP addresses and k8s cluster, then store as environment variables. +4. Create GKE nodepool for Feast e2e test with DataflowRunner. +5. Setup Feast Core, Feast Serving and dependencies using Helm. + - Redis as the job store for Feast Batch Serving. + - Postgres for persisting Feast metadata. + - Kafka and Zookeeper as the Source in Feast. +6. Install Python 3.7.4, Feast Python SDK and run end-to-end tests from + tests/e2e via pytest. +7. Tear down infrastructure, including Dataflow jobs. +" + +ORIGINAL_DIR=$(pwd) +echo $ORIGINAL_DIR + +echo " +============================================================ +Installing gcloud SDK and required packages +============================================================ +" +apt-get -qq update +apt-get -y install wget netcat kafkacat build-essential gettext-base curl kubectl +curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 +chmod 700 $ORIGINAL_DIR/get_helm.sh +$ORIGINAL_DIR/get_helm.sh + +if [[ ! $(command -v gsutil) ]]; then + CURRENT_DIR=$(dirname "$BASH_SOURCE") + . "${CURRENT_DIR}"/install-google-cloud-sdk.sh +fi + +export GOOGLE_APPLICATION_CREDENTIALS +gcloud auth activate-service-account --key-file ${GOOGLE_APPLICATION_CREDENTIALS} + +gcloud config set project ${GCLOUD_PROJECT} +gcloud config set compute/region ${GCLOUD_REGION} +gcloud config list + +echo " +============================================================ +Creating temp BQ table for Feast Serving +============================================================ +" +DATASET_NAME=feast_e2e_$(date +%s) + +bq --location=US --project_id=${GCLOUD_PROJECT} mk \ + --dataset \ + --default_table_expiration 86400 \ + ${GCLOUD_PROJECT}:${DATASET_NAME} + +echo " +============================================================ +Check and generate valid k8s cluster name +============================================================ +" +count=0 +for cluster_name in $(gcloud container clusters list --format "list(name)") +do + if [[ $cluster_name == "feast-e2e-dataflow"* ]]; then + count=$((count + 1)) + fi +done +temp="$K8_CLUSTER_NAME-$count" +export K8_CLUSTER_NAME=$temp +echo "Cluster name is $K8_CLUSTER_NAME" + +echo " +============================================================ +Reserving IP addresses for Feast dependencies +============================================================ +" +feast_kafka_1_ip_name="feast-kafka-$((count*3 + 1))" +feast_kafka_2_ip_name="feast-kafka-$((count*3 + 2))" +feast_kafka_3_ip_name="feast-kafka-$((count*3 + 3))" +feast_redis_ip_name="feast-redis-$((count + 1))" +feast_statsd_ip_name="feast-statsd-$((count + 1))" +gcloud compute addresses create \ + $feast_kafka_1_ip_name $feast_kafka_2_ip_name $feast_kafka_3_ip_name $feast_redis_ip_name $feast_statsd_ip_name \ + --region ${GCLOUD_REGION} --subnet ${GCLOUD_SUBNET} + +ip_count=0 +for ip_addr_name in $feast_kafka_1_ip_name $feast_kafka_2_ip_name $feast_kafka_3_ip_name $feast_redis_ip_name $feast_statsd_ip_name +do + if [[ "$ip_count" == 0 ]]; then + export feast_kafka_1_ip=$(gcloud compute addresses describe ${ip_addr_name} --region=asia-east1 --format "value(address)") + elif [[ "$ip_count" == 1 ]]; then + export feast_kafka_2_ip=$(gcloud compute addresses describe ${ip_addr_name} --region=asia-east1 --format "value(address)") + elif [[ "$ip_count" == 2 ]]; then + export feast_kafka_3_ip=$(gcloud compute addresses describe ${ip_addr_name} --region=asia-east1 --format "value(address)") + elif [[ "$ip_count" == 3 ]]; then + export feast_redis_ip=$(gcloud compute addresses describe ${ip_addr_name} --region=asia-east1 --format "value(address)") + elif [[ "$ip_count" == 4 ]]; then + export feast_statsd_ip=$(gcloud compute addresses describe ${ip_addr_name} --region=asia-east1 --format "value(address)") + fi + ip_count=$((ip_count + 1)) + export "$(echo $ip_addr_name | tr '-' '_')=$(gcloud compute addresses describe ${ip_addr_name} --region=asia-east1 --format "value(address)")" +done + +echo " +============================================================ +Creating GKE nodepool for Feast e2e test with DataflowRunner +============================================================ +" +gcloud container clusters create ${K8_CLUSTER_NAME} --region ${GCLOUD_REGION} \ + --enable-cloud-logging \ + --enable-cloud-monitoring \ + --network ${GCLOUD_NETWORK} \ + --subnetwork ${GCLOUD_SUBNET} \ + --machine-type n1-standard-2 +sleep 120 + +echo " +============================================================ +Create feast-postgres-database Secret in GKE nodepool +============================================================ +" +kubectl create secret generic feast-postgresql --from-literal=postgresql-password=password + +echo " +============================================================ +Create feast-gcp-service-account Secret in GKE nodepool +============================================================ +" +cd $ORIGINAL_DIR/infra/scripts +kubectl create secret generic feast-gcp-service-account --from-file=${GOOGLE_APPLICATION_CREDENTIALS} + +echo " +============================================================ +Export required environment variables +============================================================ +" +export TEMP_BUCKET=$TEMP_BUCKET +export DATASET_NAME=$DATASET_NAME +export GCLOUD_PROJECT=$GCLOUD_PROJECT +export GCLOUD_NETWORK=$GCLOUD_NETWORK +export GCLOUD_SUBNET=$GCLOUD_SUBNET +export GCLOUD_REGION=$GCLOUD_REGION + +echo " +============================================================ +Helm install Feast and its dependencies +============================================================ +" +cd $ORIGINAL_DIR/infra/scripts/test-templates +envsubst $'$TEMP_BUCKET $DATASET_NAME $GCLOUD_PROJECT $GCLOUD_NETWORK \ + $GCLOUD_SUBNET $GCLOUD_REGION $feast_kafka_1_ip + $feast_kafka_2_ip $feast_kafka_3_ip $feast_redis_ip $feast_statsd_ip' < values-end-to-end-batch-dataflow.yaml > $ORIGINAL_DIR/infra/charts/feast/values-end-to-end-batch-dataflow-updated.yaml + +cd $ORIGINAL_DIR/infra/charts +helm template feast + +cd $ORIGINAL_DIR/infra/charts/feast +helm install --wait --timeout 600s --values="values-end-to-end-batch-dataflow-updated.yaml" ${HELM_RELEASE_NAME} . + +echo " +============================================================ +Installing Python 3.7 with Miniconda and Feast SDK +============================================================ +" +# Install Python 3.7 with Miniconda +wget -q https://repo.continuum.io/miniconda/Miniconda3-4.7.12-Linux-x86_64.sh \ + -O /tmp/miniconda.sh +bash /tmp/miniconda.sh -b -p /root/miniconda -f +/root/miniconda/bin/conda init +source ~/.bashrc + +# Install Feast Python SDK and test requirements +cd $ORIGINAL_DIR +make compile-protos-python +pip install -qe sdk/python +pip install -qr tests/e2e/requirements.txt + +echo " +============================================================ +Running end-to-end tests with pytest at 'tests/e2e' +============================================================ +" +# Default artifact location setting in Prow jobs +LOGS_ARTIFACT_PATH=/logs/artifacts + +cd $ORIGINAL_DIR/tests/e2e + +core_ip=$(kubectl get -o jsonpath="{.spec.clusterIP}" service ${HELM_RELEASE_NAME}-feast-core) +serving_ip=$(kubectl get -o jsonpath="{.spec.clusterIP}" service ${HELM_RELEASE_NAME}-feast-batch-serving) + +set +e +pytest bq-batch-retrieval.py -m dataflow_runner --core_url "$core_ip:6565" --serving_url "$serving_ip:6566" --gcs_path "gs://${TEMP_BUCKET}/" --junitxml=${LOGS_ARTIFACT_PATH}/python-sdk-test-report.xml +TEST_EXIT_CODE=$? + +if [[ ${TEST_EXIT_CODE} != 0 ]]; then + echo "[DEBUG] Printing logs" + ls -ltrh /var/log/feast* + cat /var/log/feast-serving-warehouse.log /var/log/feast-core.log + + echo "[DEBUG] Printing Python packages list" + pip list +fi + +cd ${ORIGINAL_DIR} +exit ${TEST_EXIT_CODE} + +echo " +============================================================ +Cleaning up +============================================================ +" +cd $ORIGINAL_DIR/tests/e2e + +# Remove BQ Dataset +bq rm -r -f ${GCLOUD_PROJECT}:${DATASET_NAME} + +# Uninstall helm release before clearing PVCs +helm uninstall ${HELM_RELEASE_NAME} +kubectl delete pvc --all + +# Release IP addresses +yes | gcloud compute addresses delete $feast_kafka_1_ip_name $feast_kafka_2_ip_name $feast_kafka_3_ip_name $feast_redis_ip_name $feast_statsd_ip_name --region=${GCLOUD_REGION} + +# Tear down GKE infrastructure +gcloud container clusters delete --region=${GCLOUD_REGION} ${K8_CLUSTER_NAME} + +# Stop Dataflow jobs from retrieved Dataflow job ids in ingesting_jobs.txt +while read line +do + echo $line + gcloud dataflow jobs cancel $line --region=asia-east1 +done < ingesting_jobs.txt \ No newline at end of file diff --git a/infra/scripts/test-end-to-end-batch.sh b/infra/scripts/test-end-to-end-batch.sh index a5d8d9556b6..4b18f6a0678 100755 --- a/infra/scripts/test-end-to-end-batch.sh +++ b/infra/scripts/test-end-to-end-batch.sh @@ -254,7 +254,7 @@ ORIGINAL_DIR=$(pwd) cd tests/e2e set +e -pytest bq-batch-retrieval.py --gcs_path "gs://${TEMP_BUCKET}/" --junitxml=${LOGS_ARTIFACT_PATH}/python-sdk-test-report.xml +pytest bq-batch-retrieval.py -m direct_runner --gcs_path "gs://${TEMP_BUCKET}/" --junitxml=${LOGS_ARTIFACT_PATH}/python-sdk-test-report.xml TEST_EXIT_CODE=$? if [[ ${TEST_EXIT_CODE} != 0 ]]; then diff --git a/infra/scripts/test-templates/values-end-to-end-batch-dataflow.yaml b/infra/scripts/test-templates/values-end-to-end-batch-dataflow.yaml new file mode 100644 index 00000000000..aff9b0d6b71 --- /dev/null +++ b/infra/scripts/test-templates/values-end-to-end-batch-dataflow.yaml @@ -0,0 +1,141 @@ +feast-core: + # feast-core.enabled -- Flag to install Feast Core + enabled: true + gcpServiceAccount: + enabled: true + postgresql: + existingSecret: feast-postgresql + image: + tag: $PULL_PULL_SHA:1 + application-override.yaml: + feast: + stream: + options: + bootstrapServers: $feast_kafka_1_ip:31090 + jobs: + active_runner: dataflow + + runners: + - name: dataflow + type: DataflowRunner + options: + project: $GCLOUD_PROJECT + region: $GCLOUD_REGION + zone: $GCLOUD_REGION-a + tempLocation: gs://$TEMP_BUCKET/tempLocation + network: $GCLOUD_NETWORK + subnetwork: regions/$GCLOUD_REGION/subnetworks/$GCLOUD_SUBNET + maxNumWorkers: 1 + autoscalingAlgorithm: THROUGHPUT_BASED + usePublicIps: false + workerMachineType: n1-standard-1 + deadLetterTableSpec: $GCLOUD_PROJECT:$DATASET_NAME.deadletter + + metrics: + enabled: true + host: $feast_statsd_ip + +feast-online-serving: + # feast-online-serving.enabled -- Flag to install Feast Online Serving + enabled: true + image: + tag: $PULL_PULL_SHA:1 + application-override.yaml: + feast: + active_store: online + + # List of store configurations + stores: + - name: online + type: REDIS + config: + host: $feast_redis_ip + port: 6379 + subscriptions: + - name: "*" + project: "*" + version: "*" + +feast-batch-serving: + # feast-batch-serving.enabled -- Flag to install Feast Batch Serving + enabled: true + image: + tag: $PULL_PULL_SHA:1 + gcpServiceAccount: + enabled: true + + application-override.yaml: + feast: + active_store: historical + + # List of store configurations + stores: + - name: historical + type: BIGQUERY + config: + project_id: $GCLOUD_PROJECT + dataset_id: $DATASET_NAME + staging_location: gs://$TEMP_BUCKET/stagingLocation + initial_retry_delay_seconds: 3 + total_timeout_seconds: 21600 + subscriptions: + - name: "*" + project: "*" + version: "*" + +postgresql: + # postgresql.enabled -- Flag to install Postgresql + enabled: true + existingSecret: feast-postgresql + +kafka: + # kafka.enabled -- Flag to install Kafka + enabled: true + external: + enabled: true + type: LoadBalancer + annotations: + cloud.google.com/load-balancer-type: Internal + loadBalancerSourceRanges: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + firstListenerPort: 31090 + loadBalancerIP: + - $feast_kafka_1_ip + - $feast_kafka_2_ip + - $feast_kafka_3_ip + configurationOverrides: + "advertised.listeners": |- + EXTERNAL://${LOAD_BALANCER_IP}:31090 + "listener.security.protocol.map": |- + PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT + "log.retention.hours": 1 + +redis: + # redis.enabled -- Flag to install Redis + enabled: true + usePassword: false + master: + service: + type: LoadBalancer + loadBalancerIP: $feast_redis_ip + annotations: + cloud.google.com/load-balancer-type: Internal + loadBalancerSourceRanges: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + +prometheus-statsd-exporter: + # prometheus-statsd-exporter.enabled -- Flag to install StatsD to Prometheus Exporter + enabled: true + service: + type: LoadBalancer + annotations: + cloud.google.com/load-balancer-type: Internal + loadBalancerSourceRanges: + - 10.0.0.0/8 + - 172.16.0.0/12 + - 192.168.0.0/16 + loadBalancerIP: $feast_statsd_ip diff --git a/tests/e2e/bq-batch-retrieval.py b/tests/e2e/bq-batch-retrieval.py index 0cf05e77e1d..c1c6ab67805 100644 --- a/tests/e2e/bq-batch-retrieval.py +++ b/tests/e2e/bq-batch-retrieval.py @@ -4,11 +4,13 @@ from datetime import timedelta from urllib.parse import urlparse +import os import uuid import numpy as np import pandas as pd import pytest import pytz +from feast.core.IngestionJob_pb2 import IngestionJobStatus from feast.client import Client from feast.entity import Entity from feast.feature import Feature @@ -59,6 +61,8 @@ def client(core_url, serving_url, allow_dirty): return client @pytest.mark.first +@pytest.mark.direct_runner +@pytest.mark.dataflow_runner def test_apply_all_featuresets(client): client.set_project(PROJECT_NAME) @@ -127,6 +131,8 @@ def test_apply_all_featuresets(client): client.apply(no_max_age_fs) +@pytest.mark.direct_runner +@pytest.mark.dataflow_runner def test_get_batch_features_with_file(client): file_fs1 = client.get_feature_set(name="file_feature_set", version=1) @@ -139,7 +145,7 @@ def test_get_batch_features_with_file(client): "feature_value1": [f"{i}" for i in range(N_ROWS)], } ) - client.ingest(file_fs1, features_1_df) + client.ingest(file_fs1, features_1_df, timeout=480) # Rename column (datetime -> event_timestamp) features_1_df = features_1_df.rename(columns={"datetime": "event_timestamp"}) @@ -157,6 +163,8 @@ def test_get_batch_features_with_file(client): assert output["entity_id"].to_list() == [int(i) for i in output["feature_value1"].to_list()] +@pytest.mark.direct_runner +@pytest.mark.dataflow_runner def test_get_batch_features_with_gs_path(client, gcs_path): gcs_fs1 = client.get_feature_set(name="gcs_feature_set", version=1) @@ -169,7 +177,7 @@ def test_get_batch_features_with_gs_path(client, gcs_path): "feature_value2": [f"{i}" for i in range(N_ROWS)], } ) - client.ingest(gcs_fs1, features_1_df) + client.ingest(gcs_fs1, features_1_df, timeout=360) # Rename column (datetime -> event_timestamp) features_1_df = features_1_df.rename(columns={"datetime": "event_timestamp"}) @@ -201,6 +209,7 @@ def test_get_batch_features_with_gs_path(client, gcs_path): assert output["entity_id"].to_list() == [int(i) for i in output["feature_value2"].to_list()] +@pytest.mark.direct_runner def test_order_by_creation_time(client): proc_time_fs = client.get_feature_set(name="processing_time", version=1) @@ -232,6 +241,7 @@ def test_order_by_creation_time(client): assert output["feature_value3"].to_list() == ["CORRECT"] * N_ROWS +@pytest.mark.direct_runner def test_additional_columns_in_entity_table(client): add_cols_fs = client.get_feature_set(name="additional_columns", version=1) @@ -263,6 +273,7 @@ def test_additional_columns_in_entity_table(client): assert output["feature_value4"].to_list() == features_df["feature_value4"].to_list() +@pytest.mark.direct_runner def test_point_in_time_correctness_join(client): historical_fs = client.get_feature_set(name="historical", version=1) @@ -294,6 +305,7 @@ def test_point_in_time_correctness_join(client): assert output["feature_value5"].to_list() == ["CORRECT"] * N_EXAMPLES +@pytest.mark.direct_runner def test_multiple_featureset_joins(client): fs1 = client.get_feature_set(name="feature_set_1", version=1) fs2 = client.get_feature_set(name="feature_set_2", version=1) @@ -337,6 +349,7 @@ def test_multiple_featureset_joins(client): assert output["other_entity_id"].to_list() == output["other_feature_value7"].to_list() +@pytest.mark.direct_runner def test_no_max_age(client): no_max_age_fs = client.get_feature_set(name="no_max_age", version=1) @@ -359,4 +372,23 @@ def test_no_max_age(client): output = feature_retrieval_job.to_dataframe() print(output.head()) - assert output["entity_id"].to_list() == output["feature_value8"].to_list() \ No newline at end of file + assert output["entity_id"].to_list() == output["feature_value8"].to_list() + + +@pytest.fixture(scope="module", autouse=True) +def infra_teardown(pytestconfig, core_url, serving_url): + client = Client(core_url=core_url, serving_url=serving_url) + client.set_project(PROJECT_NAME) + + marker = pytestconfig.getoption("-m") + yield marker + if marker == 'dataflow_runner': + ingest_jobs = client.list_ingest_jobs() + ingest_jobs = [client.list_ingest_jobs(job.id)[0].external_id for job in ingest_jobs if job.status == IngestionJobStatus.RUNNING] + + cwd = os.getcwd() + with open(f"{cwd}/ingesting_jobs.txt", "w+") as output: + for job in ingest_jobs: + output.write('%s\n' % job) + else: + print('Cleaning up not required') From 2ef1faf10b8a75bc5df521250068cc5989ecd737 Mon Sep 17 00:00:00 2001 From: Terence Date: Tue, 12 May 2020 11:04:17 +0800 Subject: [PATCH 158/176] Fix prow configs for e2e dataflow tests --- .prow/config.yaml | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/.prow/config.yaml b/.prow/config.yaml index fd1e2b290d6..a5f7cb17df3 100644 --- a/.prow/config.yaml +++ b/.prow/config.yaml @@ -232,7 +232,7 @@ presubmits: volumes: - name: service-account-df secret: - secretName: feast-service-account + secretName: feast-e2e-service-account containers: - image: maven:3.6-jdk-11 command: ["infra/scripts/test-end-to-end-batch-dataflow.sh"] @@ -242,7 +242,7 @@ presubmits: memory: "6144Mi" volumeMounts: - name: service-account-df - mountPath: "/etc/service-account" + mountPath: "/etc/service-account-df" skip_branches: - ^v0\.(3|4)-branch$ @@ -253,7 +253,7 @@ presubmits: volumes: - name: service-account-df secret: - secretName: feast-service-account + secretName: feast-e2e-service-account containers: - image: maven:3.6-jdk-8 command: ["infra/scripts/test-end-to-end-batch-dataflow.sh"] @@ -263,7 +263,7 @@ presubmits: memory: "6144Mi" volumeMounts: - name: service-account-df - mountPath: "/etc/service-account" + mountPath: "/etc/service-account-df" branches: - ^v0\.(3|4)-branch$ @@ -298,8 +298,6 @@ presubmits: docker tag gcr.io/kf-feast/feast-serving:${PULL_PULL_SHA:1} docker push gcr.io/kf-feast/feast-serving:${PULL_PULL_SHA:1} - - fi volumeMounts: - name: docker-socket mountPath: /var/run/docker.sock From de8e4e1f87338d9fffade7702c6305015b1f0aa0 Mon Sep 17 00:00:00 2001 From: Willem Pienaar Date: Wed, 13 May 2020 10:11:46 +0800 Subject: [PATCH 159/176] Remove f-strings to pass python linting --- sdk/python/feast/client.py | 4 ++-- sdk/python/feast/feature_set.py | 4 ++-- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/sdk/python/feast/client.py b/sdk/python/feast/client.py index d50567deac1..8d66f58c06c 100644 --- a/sdk/python/feast/client.py +++ b/sdk/python/feast/client.py @@ -585,7 +585,7 @@ def get_batch_features( # String based source if not entity_rows.endswith((".avro", "*")): raise Exception( - f"Only .avro and wildcard paths are accepted as entity_rows" + "Only .avro and wildcard paths are accepted as entity_rows" ) else: raise Exception( @@ -778,7 +778,7 @@ def ingest( elif isinstance(feature_set, str): name = feature_set else: - raise Exception(f"Feature set name must be provided") + raise Exception("Feature set name must be provided") # Read table and get row count dir_path, dest_path = _read_table_from_source(source, chunk_size, max_workers) diff --git a/sdk/python/feast/feature_set.py b/sdk/python/feast/feature_set.py index ace7f165de1..973c2a52a57 100644 --- a/sdk/python/feast/feature_set.py +++ b/sdk/python/feast/feature_set.py @@ -653,10 +653,10 @@ def is_valid(self): """ if not self.name: - raise ValueError(f"No name found in feature set.") + raise ValueError("No name found in feature set.") if len(self.entities) == 0: - raise ValueError(f"No entities found in feature set {self.name}") + raise ValueError("No entities found in feature set {self.name}") def import_tfx_schema(self, schema: schema_pb2.Schema): """ From 2c5130d1435ddd85fd1b75af0271eda2ec5345ed Mon Sep 17 00:00:00 2001 From: Khor Shu Heng <32997938+khorshuheng@users.noreply.github.com> Date: Wed, 13 May 2020 14:15:26 +0800 Subject: [PATCH 160/176] Include server port config on the generated application.yml (#696) * Include server port config on the generated application.yml * Add additional documentation Co-authored-by: Khor Shu Heng --- infra/charts/feast/charts/feast-core/README.md | 2 +- infra/charts/feast/charts/feast-core/templates/configmap.yaml | 3 +++ infra/charts/feast/charts/feast-core/values.yaml | 4 ++-- .../feast/charts/feast-serving/templates/configmap.yaml | 3 +++ infra/charts/feast/charts/feast-serving/values.yaml | 4 ++-- 5 files changed, 11 insertions(+), 5 deletions(-) diff --git a/infra/charts/feast/charts/feast-core/README.md b/infra/charts/feast/charts/feast-core/README.md index 4bf4578eb75..a06ce01d18b 100644 --- a/infra/charts/feast/charts/feast-core/README.md +++ b/infra/charts/feast/charts/feast-core/README.md @@ -66,5 +66,5 @@ Current chart version is `0.5.0-alpha.1` | service.grpc.targetPort | int | `6565` | Container port serving GRPC requests | | service.http.nodePort | string | `nil` | Port number that each cluster node will listen to | | service.http.port | int | `80` | Service port for HTTP requests | -| service.http.targetPort | int | `8080` | Container port serving HTTP requests | +| service.http.targetPort | int | `8080` | Container port serving HTTP requests and Prometheus metrics | | service.type | string | `"ClusterIP"` | Kubernetes service type | diff --git a/infra/charts/feast/charts/feast-core/templates/configmap.yaml b/infra/charts/feast/charts/feast-core/templates/configmap.yaml index bce32ef33a6..b48e15cc985 100644 --- a/infra/charts/feast/charts/feast-core/templates/configmap.yaml +++ b/infra/charts/feast/charts/feast-core/templates/configmap.yaml @@ -27,6 +27,9 @@ data: type: statsd host: {{ .Release.Name }}-prometheus-statsd-exporter-udp port: 9125 + + server: + port: {{ .Values.service.http.targetPort }} {{- end }} application-override.yaml: | diff --git a/infra/charts/feast/charts/feast-core/values.yaml b/infra/charts/feast/charts/feast-core/values.yaml index cc7bb49f0f9..34b9a718dff 100644 --- a/infra/charts/feast/charts/feast-core/values.yaml +++ b/infra/charts/feast/charts/feast-core/values.yaml @@ -14,7 +14,7 @@ application.yaml: enabled: true application-generated.yaml: - # "application-generated.yaml".enabled -- Flag to include Helm generated configuration for Feast database URL, Kafka bootstrap servers and jobs metrics host. This is useful for deployment that uses default configuration for Kafka, Postgres and StatsD exporter. Please set `application-override.yaml` to override this configuration. + # "application-generated.yaml".enabled -- Flag to include Helm generated configuration for http port, Feast database URL, Kafka bootstrap servers and jobs metrics host. This is useful for deployment that uses default configuration for Kafka, Postgres and StatsD exporter. Please set `application-override.yaml` to override this configuration. enabled: true # "application-secret.yaml" -- Configuration to override the default [application.yaml](https://github.com/gojek/feast/blob/master/core/src/main/resources/application.yml). Will be created as a Secret. `application-override.yaml` has a higher precedence than `application-secret.yaml`. It is recommended to either set `application-override.yaml` or `application-secret.yaml` only to simplify config management. @@ -89,7 +89,7 @@ service: http: # service.http.port -- Service port for HTTP requests port: 80 - # service.http.targetPort -- Container port serving HTTP requests + # service.http.targetPort -- Container port serving HTTP requests and Prometheus metrics targetPort: 8080 # service.http.nodePort -- Port number that each cluster node will listen to nodePort: diff --git a/infra/charts/feast/charts/feast-serving/templates/configmap.yaml b/infra/charts/feast/charts/feast-serving/templates/configmap.yaml index 7c895ce530b..011b4cbccab 100644 --- a/infra/charts/feast/charts/feast-serving/templates/configmap.yaml +++ b/infra/charts/feast/charts/feast-serving/templates/configmap.yaml @@ -29,6 +29,9 @@ data: job_store: redis_host: {{ .Release.Name }}-redis-master redis_port: 6379 + + server: + port: {{ .Values.service.http.targetPort }} {{- end }} application-override.yaml: | diff --git a/infra/charts/feast/charts/feast-serving/values.yaml b/infra/charts/feast/charts/feast-serving/values.yaml index bf7b2c772a6..396099e08d9 100644 --- a/infra/charts/feast/charts/feast-serving/values.yaml +++ b/infra/charts/feast/charts/feast-serving/values.yaml @@ -14,7 +14,7 @@ application.yaml: enabled: true application-generated.yaml: - # "application-generated.yaml".enabled -- Flag to include Helm generated configuration for Feast Core host, Redis store and job store. This is useful for deployment that uses default configuration for Redis. Please set `application-override.yaml` to override this configuration. + # "application-generated.yaml".enabled -- Flag to include Helm generated configuration for http port, Feast Core host, Redis store and job store. This is useful for deployment that uses default configuration for Redis. Please set `application-override.yaml` to override this configuration. enabled: true # "application-secret.yaml" -- Configuration to override the default [application.yaml](https://github.com/gojek/feast/blob/master/serving/src/main/resources/application.yml). Will be created as a Secret. `application-override.yaml` has a higher precedence than `application-secret.yaml`. It is recommended to either set `application-override.yaml` or `application-secret.yaml` only to simplify config management. @@ -84,7 +84,7 @@ service: http: # service.http.port -- Service port for HTTP requests port: 80 - # service.http.targetPort -- Container port serving HTTP requests + # service.http.targetPort -- Container port serving HTTP requests and Prometheus metrics targetPort: 8080 # service.http.nodePort -- Port number that each cluster node will listen to nodePort: From dfc81b9e7530b176df607defb4926dfd87df9820 Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Wed, 13 May 2020 18:40:26 +0800 Subject: [PATCH 161/176] Add support for feature set updates and remove versions (#676) * BigQuery connector migrates schema on feature set changes * Update docs, remove versions from protos * Remove versions from core, ApplyFeatureSet updates existing featuresets instead of advancing version * Remove versions from storage API and connectors * Remove versions from ingestion * Remove versions from serving * Remove versions from sdk, examples and end-to-end tests * Ignore versions if provided for backward compatibility * Regenerate golang protos, apply black * Rebase on master * Remove LAST_VERSION * Clean up job update logic * Use set comparison for entities during feature set updates, change FeatureSet.status to use enum directly * Fix tests and error messages * Move strip versions to transform before feature row validate step * Update documentation, clean up toProto method * Remove redundant status setting, correct misleading comments * Add end to end test for feature set updates, squash some bugs * Use count() instead of returning all rows --- .../feast/core/dao/FeatureSetRepository.java | 25 +- .../java/feast/core/job/JobUpdateTask.java | 20 +- .../job/direct/DirectRunnerJobManager.java | 6 +- .../main/java/feast/core/model/Entity.java | 4 + .../main/java/feast/core/model/Feature.java | 122 +++- .../java/feast/core/model/FeatureSet.java | 192 +++--- .../src/main/java/feast/core/model/Store.java | 12 +- .../core/service/JobCoordinatorService.java | 32 +- .../java/feast/core/service/JobService.java | 2 - .../java/feast/core/service/SpecService.java | 162 ++--- .../feast/core/job/JobUpdateTaskTest.java | 5 +- .../job/dataflow/DataflowJobManagerTest.java | 11 +- .../direct/DirectRunnerJobManagerTest.java | 7 +- .../FeatureSetJsonByteConverterTest.java | 8 +- .../java/feast/core/model/FeatureSetTest.java | 205 ++++++ .../service/JobCoordinatorServiceTest.java | 100 +-- .../feast/core/service/JobServiceTest.java | 11 +- .../feast/core/service/SpecServiceTest.java | 261 ++------ .../feast/core/service/TestObjectFactory.java | 3 +- examples/basic/basic.ipynb | 11 +- go.mod | 2 +- go.sum | 2 + .../main/java/feast/ingestion/ImportJob.java | 6 +- ...ava => ProcessAndValidateFeatureRows.java} | 27 +- .../transform/fn/ProcessFeatureRowDoFn.java | 37 ++ .../transform/fn/ValidateFeatureRowDoFn.java | 13 +- .../WriteDeadletterRowMetricsDoFn.java | 1 - .../metrics/WriteFeatureValueMetricsDoFn.java | 16 +- .../metrics/WriteRowMetricsDoFn.java | 12 +- .../java/feast/ingestion/utils/SpecUtil.java | 40 +- .../feast/ingestion/values/FailedElement.java | 83 --- .../java/feast/ingestion/ImportJobTest.java | 2 - ...=> ProcessAndValidateFeatureRowsTest.java} | 65 +- .../WriteFeatureValueMetricsDoFnTest.input | 6 +- .../WriteFeatureValueMetricsDoFnTest.output | 114 ++-- .../transform/WriteRowMetricsDoFnTest.input | 6 +- .../transform/WriteRowMetricsDoFnTest.output | 42 +- protos/feast/core/CoreService.proto | 29 +- protos/feast/core/FeatureSet.proto | 4 +- protos/feast/core/FeatureSetReference.proto | 4 +- protos/feast/core/Store.proto | 22 +- protos/feast/serving/ServingService.proto | 3 - protos/feast/storage/Redis.proto | 2 +- protos/feast/types/FeatureRow.proto | 2 +- sdk/go/README.md | 8 +- sdk/go/protos/feast/core/CoreService.pb.go | 416 ++++++------ sdk/go/protos/feast/core/FeatureSet.pb.go | 619 ++++-------------- .../feast/core/FeatureSetReference.pb.go | 27 +- sdk/go/protos/feast/core/Runner.pb.go | 375 +++++++++++ sdk/go/protos/feast/core/Source.pb.go | 55 +- sdk/go/protos/feast/core/Store.pb.go | 284 +++++--- .../protos/feast/serving/ServingService.pb.go | 316 +++++---- sdk/go/protos/feast/storage/Redis.pb.go | 2 +- sdk/go/protos/feast/types/FeatureRow.pb.go | 2 +- .../tensorflow_metadata/proto/v0/path.pb.go | 10 +- .../tensorflow_metadata/proto/v0/schema.pb.go | 11 +- sdk/go/request.go | 29 +- sdk/go/request_test.go | 21 +- sdk/go/response_test.go | 2 +- .../java/com/gojek/feast/FeastClient.java | 8 +- .../java/com/gojek/feast/RequestUtil.java | 46 +- .../java/com/gojek/feast/RequestUtilTest.java | 15 +- sdk/python/feast/cli.py | 20 +- sdk/python/feast/client.py | 84 +-- sdk/python/feast/feature_set.py | 45 +- sdk/python/feast/loaders/ingest.py | 4 +- sdk/python/feast/type_map.py | 8 +- sdk/python/tests/feast_core_server.py | 9 - sdk/python/tests/feast_serving_server.py | 1 - sdk/python/tests/test_client.py | 41 +- sdk/python/tests/test_feature_set.py | 4 +- serving/README.md | 3 - .../feast/serving/config/FeastProperties.java | 1 - .../serving/service/OnlineServingService.java | 10 +- .../serving/specs/CachedSpecService.java | 59 +- .../main/java/feast/serving/util/RefUtil.java | 14 - serving/src/main/resources/application.yml | 2 - .../resources/templates/join_featuresets.sql | 24 - .../templates/single_featureset_pit_join.sql | 90 --- .../ServingServiceGRpcControllerTest.java | 12 +- .../service/CachedSpecServiceTest.java | 77 +-- .../service/OnlineServingServiceTest.java | 112 +--- .../storage/api/writer/FailedElement.java | 5 - .../storage/common/testing/TestUtil.java | 2 +- .../retriever/FeatureSetQueryInfo.java | 8 - .../bigquery/retriever/QueryTemplater.java | 8 +- .../bigquery/writer/BigQueryFeatureSink.java | 39 +- .../bigquery/writer/GetTableDestination.java | 6 +- .../resources/templates/join_featuresets.sql | 4 +- .../templates/single_featureset_pit_join.sql | 18 +- .../redis/retriever/RedisOnlineRetriever.java | 5 +- .../retriever/RedisOnlineRetrieverTest.java | 37 +- .../redis/writer/RedisFeatureSinkTest.java | 36 +- .../RedisClusterOnlineRetriever.java | 3 - .../RedisClusterOnlineRetrieverTest.java | 37 +- .../writer/RedisClusterFeatureSinkTest.java | 37 +- tests/e2e/basic-ingest-redis-serving.py | 3 +- tests/e2e/bq-batch-retrieval.py | 392 ++++++++--- 98 files changed, 2511 insertions(+), 2734 deletions(-) create mode 100644 core/src/test/java/feast/core/model/FeatureSetTest.java rename ingestion/src/main/java/feast/ingestion/transform/{ValidateFeatureRows.java => ProcessAndValidateFeatureRows.java} (75%) create mode 100644 ingestion/src/main/java/feast/ingestion/transform/fn/ProcessFeatureRowDoFn.java delete mode 100644 ingestion/src/main/java/feast/ingestion/values/FailedElement.java rename ingestion/src/test/java/feast/ingestion/transform/{ValidateFeatureRowsTest.java => ProcessAndValidateFeatureRowsTest.java} (74%) create mode 100644 sdk/go/protos/feast/core/Runner.pb.go delete mode 100644 serving/src/main/resources/templates/join_featuresets.sql delete mode 100644 serving/src/main/resources/templates/single_featureset_pit_join.sql diff --git a/core/src/main/java/feast/core/dao/FeatureSetRepository.java b/core/src/main/java/feast/core/dao/FeatureSetRepository.java index 3eba2108889..b136650dfdf 100644 --- a/core/src/main/java/feast/core/dao/FeatureSetRepository.java +++ b/core/src/main/java/feast/core/dao/FeatureSetRepository.java @@ -25,25 +25,16 @@ public interface FeatureSetRepository extends JpaRepository long count(); - // Find single feature set by project, name, and version - FeatureSet findFeatureSetByNameAndProject_NameAndVersion( - String name, String project, Integer version); + // Find single feature set by project and name + FeatureSet findFeatureSetByNameAndProject_Name(String name, String project); - // Find single latest version of a feature set by project and name (LIKE) - FeatureSet findFirstFeatureSetByNameLikeAndProject_NameOrderByVersionDesc( - String name, String project); + // find all feature sets and order by name + List findAllByOrderByNameAsc(); - // find all feature sets and order by name and version - List findAllByOrderByNameAscVersionAsc(); + // find all feature sets matching the given name pattern with a specific project. + List findAllByNameLikeAndProject_NameOrderByNameAsc(String name, String project_name); - // find all feature sets within a project and order by name and version - List findAllByProject_NameOrderByNameAscVersionAsc(String project_name); - - // find all versions of feature sets matching the given name pattern with a specific project. - List findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( - String name, String project_name); - - // find all versions of feature sets matching the given name pattern and project pattern - List findAllByNameLikeAndProject_NameLikeOrderByNameAscVersionAsc( + // find all feature sets matching the given name pattern and project pattern + List findAllByNameLikeAndProject_NameLikeOrderByNameAsc( String name, String project_name); } diff --git a/core/src/main/java/feast/core/job/JobUpdateTask.java b/core/src/main/java/feast/core/job/JobUpdateTask.java index bb876f47f22..b508aa46d2f 100644 --- a/core/src/main/java/feast/core/job/JobUpdateTask.java +++ b/core/src/main/java/feast/core/job/JobUpdateTask.java @@ -17,6 +17,7 @@ package feast.core.job; import com.google.common.collect.Sets; +import feast.core.FeatureSetProto.FeatureSetStatus; import feast.core.log.Action; import feast.core.log.AuditLogger; import feast.core.log.Resource; @@ -28,7 +29,6 @@ import java.time.Instant; import java.util.List; import java.util.Optional; -import java.util.Set; import java.util.concurrent.Callable; import java.util.concurrent.ExecutionException; import java.util.concurrent.ExecutorService; @@ -84,7 +84,7 @@ public Job call() { } else { Job job = currentJob.get(); - if (featureSetsChangedFor(job)) { + if (requiresUpdate(job)) { submittedJob = executorService.submit(() -> updateJob(job)); } else { return updateStatus(job); @@ -101,11 +101,19 @@ public Job call() { } } - boolean featureSetsChangedFor(Job job) { - Set existingFeatureSetsPopulatedByJob = Sets.newHashSet(job.getFeatureSets()); - Set newFeatureSetsPopulatedByJob = Sets.newHashSet(featureSets); + boolean requiresUpdate(Job job) { + // If set of feature sets has changed + if (!Sets.newHashSet(featureSets).equals(Sets.newHashSet(job.getFeatureSets()))) { + return true; + } - return !newFeatureSetsPopulatedByJob.equals(existingFeatureSetsPopulatedByJob); + // If any existing feature set populated by the job has its status as pending + for (FeatureSet featureSet : job.getFeatureSets()) { + if (featureSet.getStatus().equals(FeatureSetStatus.STATUS_PENDING)) { + return true; + } + } + return false; } private Job createJob() { diff --git a/core/src/main/java/feast/core/job/direct/DirectRunnerJobManager.java b/core/src/main/java/feast/core/job/direct/DirectRunnerJobManager.java index 2adedbefd9f..7b160b2c3db 100644 --- a/core/src/main/java/feast/core/job/direct/DirectRunnerJobManager.java +++ b/core/src/main/java/feast/core/job/direct/DirectRunnerJobManager.java @@ -79,7 +79,7 @@ public Job startJob(Job job) { featureSetProtos.add(featureSet.toProto()); } ImportOptions pipelineOptions = - getPipelineOptions(featureSetProtos, job.getStore().toProto()); + getPipelineOptions(job.getId(), featureSetProtos, job.getStore().toProto()); PipelineResult pipelineResult = runPipeline(pipelineOptions); DirectJob directJob = new DirectJob(job.getId(), pipelineResult); jobs.add(directJob); @@ -93,7 +93,8 @@ public Job startJob(Job job) { } private ImportOptions getPipelineOptions( - List featureSets, StoreProto.Store sink) throws IOException { + String jobName, List featureSets, StoreProto.Store sink) + throws IOException { String[] args = TypeConversion.convertMapToArgs(defaultOptions); ImportOptions pipelineOptions = PipelineOptionsFactory.fromArgs(args).as(ImportOptions.class); @@ -101,6 +102,7 @@ private ImportOptions getPipelineOptions( new BZip2Compressor<>(new FeatureSetJsonByteConverter()); pipelineOptions.setFeatureSetJson(featureSetJsonCompressor.compress(featureSets)); + pipelineOptions.setJobName(jobName); pipelineOptions.setStoreJson(Collections.singletonList(JsonFormat.printer().print(sink))); pipelineOptions.setRunner(DirectRunner.class); pipelineOptions.setProject(""); // set to default value to satisfy validation diff --git a/core/src/main/java/feast/core/model/Entity.java b/core/src/main/java/feast/core/model/Entity.java index 791e280d481..190f2781d58 100644 --- a/core/src/main/java/feast/core/model/Entity.java +++ b/core/src/main/java/feast/core/model/Entity.java @@ -54,6 +54,10 @@ public static Entity fromProto(EntitySpec entitySpec) { return entity; } + public EntitySpec toProto() { + return EntitySpec.newBuilder().setName(name).setValueType(ValueType.Enum.valueOf(type)).build(); + } + @Override public boolean equals(Object o) { if (this == o) { diff --git a/core/src/main/java/feast/core/model/Feature.java b/core/src/main/java/feast/core/model/Feature.java index 487de3ce378..2ae91b1997a 100644 --- a/core/src/main/java/feast/core/model/Feature.java +++ b/core/src/main/java/feast/core/model/Feature.java @@ -16,7 +16,9 @@ */ package feast.core.model; +import com.google.protobuf.InvalidProtocolBufferException; import feast.core.FeatureSetProto.FeatureSpec; +import feast.core.FeatureSetProto.FeatureSpec.Builder; import feast.core.util.TypeConversion; import feast.types.ValueProto.ValueType; import java.util.Arrays; @@ -26,6 +28,7 @@ import javax.persistence.Entity; import lombok.Getter; import lombok.Setter; +import org.tensorflow.metadata.v0.*; /** * Feature belonging to a featureset. Contains name, type as well as domain metadata about the @@ -79,6 +82,9 @@ public class Feature { private byte[] timeOfDayDomain; public Feature() {} + // Whether this feature has been archived. A archived feature cannot be + // retrieved from or written to. + private boolean archived = false; private Feature(String name, ValueType.Enum type) { this.setName(name); @@ -88,13 +94,66 @@ private Feature(String name, ValueType.Enum type) { public static Feature fromProto(FeatureSpec featureSpec) { Feature feature = new Feature(featureSpec.getName(), featureSpec.getValueType()); feature.labels = TypeConversion.convertMapToJsonString(featureSpec.getLabelsMap()); + feature.updateSchema(featureSpec); + return feature; + } + + public FeatureSpec toProto() throws InvalidProtocolBufferException { + Builder featureSpecBuilder = + FeatureSpec.newBuilder().setName(getName()).setValueType(ValueType.Enum.valueOf(getType())); + + if (getPresence() != null) { + featureSpecBuilder.setPresence(FeaturePresence.parseFrom(getPresence())); + } else if (getGroupPresence() != null) { + featureSpecBuilder.setGroupPresence(FeaturePresenceWithinGroup.parseFrom(getGroupPresence())); + } + if (getShape() != null) { + featureSpecBuilder.setShape(FixedShape.parseFrom(getShape())); + } else if (getValueCount() != null) { + featureSpecBuilder.setValueCount(ValueCount.parseFrom(getValueCount())); + } + + if (getDomain() != null) { + featureSpecBuilder.setDomain(getDomain()); + } else if (getIntDomain() != null) { + featureSpecBuilder.setIntDomain(IntDomain.parseFrom(getIntDomain())); + } else if (getFloatDomain() != null) { + featureSpecBuilder.setFloatDomain(FloatDomain.parseFrom(getFloatDomain())); + } else if (getStringDomain() != null) { + featureSpecBuilder.setStringDomain(StringDomain.parseFrom(getStringDomain())); + } else if (getBoolDomain() != null) { + featureSpecBuilder.setBoolDomain(BoolDomain.parseFrom(getBoolDomain())); + } else if (getStructDomain() != null) { + featureSpecBuilder.setStructDomain(StructDomain.parseFrom(getStructDomain())); + } else if (getNaturalLanguageDomain() != null) { + featureSpecBuilder.setNaturalLanguageDomain( + NaturalLanguageDomain.parseFrom(getNaturalLanguageDomain())); + } else if (getImageDomain() != null) { + featureSpecBuilder.setImageDomain(ImageDomain.parseFrom(getImageDomain())); + } else if (getMidDomain() != null) { + featureSpecBuilder.setMidDomain(MIDDomain.parseFrom(getMidDomain())); + } else if (getUrlDomain() != null) { + featureSpecBuilder.setUrlDomain(URLDomain.parseFrom(getUrlDomain())); + } else if (getTimeDomain() != null) { + featureSpecBuilder.setTimeDomain(TimeDomain.parseFrom(getTimeDomain())); + } else if (getTimeOfDayDomain() != null) { + featureSpecBuilder.setTimeOfDayDomain(TimeOfDayDomain.parseFrom(getTimeOfDayDomain())); + } + + if (getLabels() != null) { + featureSpecBuilder.putAllLabels(getLabels()); + } + return featureSpecBuilder.build(); + } + + private void updateSchema(FeatureSpec featureSpec) { switch (featureSpec.getPresenceConstraintsCase()) { case PRESENCE: - feature.setPresence(featureSpec.getPresence().toByteArray()); + setPresence(featureSpec.getPresence().toByteArray()); break; case GROUP_PRESENCE: - feature.setGroupPresence(featureSpec.getGroupPresence().toByteArray()); + setGroupPresence(featureSpec.getGroupPresence().toByteArray()); break; case PRESENCECONSTRAINTS_NOT_SET: break; @@ -102,10 +161,10 @@ public static Feature fromProto(FeatureSpec featureSpec) { switch (featureSpec.getShapeTypeCase()) { case SHAPE: - feature.setShape(featureSpec.getShape().toByteArray()); + setShape(featureSpec.getShape().toByteArray()); break; case VALUE_COUNT: - feature.setValueCount(featureSpec.getValueCount().toByteArray()); + setValueCount(featureSpec.getValueCount().toByteArray()); break; case SHAPETYPE_NOT_SET: break; @@ -113,45 +172,70 @@ public static Feature fromProto(FeatureSpec featureSpec) { switch (featureSpec.getDomainInfoCase()) { case DOMAIN: - feature.setDomain(featureSpec.getDomain()); + setDomain(featureSpec.getDomain()); break; case INT_DOMAIN: - feature.setIntDomain(featureSpec.getIntDomain().toByteArray()); + setIntDomain(featureSpec.getIntDomain().toByteArray()); break; case FLOAT_DOMAIN: - feature.setFloatDomain(featureSpec.getFloatDomain().toByteArray()); + setFloatDomain(featureSpec.getFloatDomain().toByteArray()); break; case STRING_DOMAIN: - feature.setStringDomain(featureSpec.getStringDomain().toByteArray()); + setStringDomain(featureSpec.getStringDomain().toByteArray()); break; case BOOL_DOMAIN: - feature.setBoolDomain(featureSpec.getBoolDomain().toByteArray()); + setBoolDomain(featureSpec.getBoolDomain().toByteArray()); break; case STRUCT_DOMAIN: - feature.setStructDomain(featureSpec.getStructDomain().toByteArray()); + setStructDomain(featureSpec.getStructDomain().toByteArray()); break; case NATURAL_LANGUAGE_DOMAIN: - feature.setNaturalLanguageDomain(featureSpec.getNaturalLanguageDomain().toByteArray()); + setNaturalLanguageDomain(featureSpec.getNaturalLanguageDomain().toByteArray()); break; case IMAGE_DOMAIN: - feature.setImageDomain(featureSpec.getImageDomain().toByteArray()); + setImageDomain(featureSpec.getImageDomain().toByteArray()); break; case MID_DOMAIN: - feature.setMidDomain(featureSpec.getMidDomain().toByteArray()); + setMidDomain(featureSpec.getMidDomain().toByteArray()); break; case URL_DOMAIN: - feature.setUrlDomain(featureSpec.getUrlDomain().toByteArray()); + setUrlDomain(featureSpec.getUrlDomain().toByteArray()); break; case TIME_DOMAIN: - feature.setTimeDomain(featureSpec.getTimeDomain().toByteArray()); + setTimeDomain(featureSpec.getTimeDomain().toByteArray()); break; case TIME_OF_DAY_DOMAIN: - feature.setTimeOfDayDomain(featureSpec.getTimeOfDayDomain().toByteArray()); + setTimeOfDayDomain(featureSpec.getTimeOfDayDomain().toByteArray()); break; case DOMAININFO_NOT_SET: break; } - return feature; + } + + /** Archive this feature. */ + public void archive() { + this.archived = true; + } + + /** + * Update the feature object with a valid feature spec. Only schema changes are allowed. + * + * @param featureSpec {@link FeatureSpec} containing schema changes. + */ + public void updateFromProto(FeatureSpec featureSpec) { + if (isArchived()) { + throw new IllegalArgumentException( + String.format( + "You are attempting to create a feature %s that was previously archived. This isn't allowed. Please create a new feature with a different name.", + featureSpec.getName())); + } + if (ValueType.Enum.valueOf(type) != featureSpec.getValueType()) { + throw new IllegalArgumentException( + String.format( + "You are attempting to change the type of feature %s from %s to %s. This isn't allowed. Please create a new feature.", + featureSpec.getName(), type, featureSpec.getValueType())); + } + updateSchema(featureSpec); } public Map getLabels() { @@ -167,7 +251,9 @@ public boolean equals(Object o) { return false; } Feature feature = (Feature) o; - return Objects.equals(getName(), feature.getName()) + return getName().equals(feature.getName()) + && getType().equals(feature.getType()) + && isArchived() == (feature.isArchived()) && Objects.equals(getLabels(), feature.getLabels()) && Arrays.equals(getPresence(), feature.getPresence()) && Arrays.equals(getGroupPresence(), feature.getGroupPresence()) diff --git a/core/src/main/java/feast/core/model/FeatureSet.java b/core/src/main/java/feast/core/model/FeatureSet.java index 91bb2bea89f..1c8351f4790 100644 --- a/core/src/main/java/feast/core/model/FeatureSet.java +++ b/core/src/main/java/feast/core/model/FeatureSet.java @@ -16,14 +16,15 @@ */ package feast.core.model; +import com.google.common.collect.Sets; import com.google.protobuf.Duration; import com.google.protobuf.InvalidProtocolBufferException; import com.google.protobuf.Timestamp; import feast.core.FeatureSetProto; import feast.core.FeatureSetProto.*; import feast.core.util.TypeConversion; -import feast.types.ValueProto.ValueType.Enum; import java.util.*; +import java.util.stream.Collectors; import javax.persistence.*; import lombok.Getter; import lombok.Setter; @@ -35,8 +36,8 @@ @javax.persistence.Entity @Table( name = "feature_sets", - uniqueConstraints = @UniqueConstraint(columnNames = {"name", "version", "project_name"})) -public class FeatureSet extends AbstractTimestampEntity implements Comparable { + uniqueConstraints = @UniqueConstraint(columnNames = {"name", "project_name"})) +public class FeatureSet extends AbstractTimestampEntity { // Id of the featureSet, defined as project/feature_set_name:feature_set_version @Id @GeneratedValue private long id; @@ -45,10 +46,6 @@ public class FeatureSet extends AbstractTimestampEntity implements Comparable entities, List features, @@ -103,21 +100,16 @@ public FeatureSet( FeatureSetStatus status) { this.maxAgeSeconds = maxAgeSeconds; this.source = source; - this.status = status.toString(); + this.status = status; this.entities = new HashSet<>(); this.features = new HashSet<>(); this.name = name; this.project = new Project(project); - this.version = version; this.labels = TypeConversion.convertMapToJsonString(labels); addEntities(entities); addFeatures(features); } - public void setVersion(int version) { - this.version = version; - } - public void setName(String name) { this.name = name; } @@ -151,7 +143,6 @@ public static FeatureSet fromProto(FeatureSetProto.FeatureSet featureSetProto) { return new FeatureSet( featureSetProto.getSpec().getName(), featureSetProto.getSpec().getProject(), - featureSetProto.getSpec().getVersion(), featureSetSpec.getMaxAge().getSeconds(), entitySpecs, featureSpecs, @@ -160,6 +151,64 @@ public static FeatureSet fromProto(FeatureSetProto.FeatureSet featureSetProto) { featureSetProto.getMeta().getStatus()); } + // Updates the existing feature set from a proto. + public void updateFromProto(FeatureSetProto.FeatureSet featureSetProto) + throws InvalidProtocolBufferException { + FeatureSetSpec spec = featureSetProto.getSpec(); + if (this.toProto().getSpec().equals(spec)) { + return; + } + + // 1. validate + // 1a. check no change to identifiers + if (!name.equals(spec.getName())) { + throw new IllegalArgumentException( + String.format("Given feature set name %s does not match name %s.", spec.getName(), name)); + } + if (!project.getName().equals(spec.getProject())) { + throw new IllegalArgumentException( + String.format( + "You are attempting to change the project of feature set %s from %s to %s. This isn't allowed. Please create a new feature set under the desired project.", + spec.getName(), project, spec.getProject())); + } + + Set existingEntities = + entities.stream().map(Entity::toProto).collect(Collectors.toSet()); + + // 1b. check no change to entities + if (!Sets.newHashSet(spec.getEntitiesList()).equals(existingEntities)) { + throw new IllegalArgumentException( + String.format( + "You are attempting to change the entities of this feature set: Given set of entities \n{%s}\n does not match existing set of entities\n {%s}. This isn't allowed. Please create a new feature set. ", + spec.getEntitiesList(), existingEntities)); + } + + // 4. Update max age and source. + maxAgeSeconds = spec.getMaxAge().getSeconds(); + source = Source.fromProto(spec.getSource()); + + Map updatedFeatures = + spec.getFeaturesList().stream().collect(Collectors.toMap(FeatureSpec::getName, fs -> fs)); + + // 3. Tombstone features that are gone, update features that have changed + for (Feature existingFeature : features) { + String existingFeatureName = existingFeature.getName(); + FeatureSpec updatedFeatureSpec = updatedFeatures.get(existingFeatureName); + if (updatedFeatureSpec == null) { + existingFeature.archive(); + } else { + existingFeature.updateFromProto(updatedFeatureSpec); + updatedFeatures.remove(existingFeatureName); + } + } + + // 4. Add new features + for (FeatureSpec featureSpec : updatedFeatures.values()) { + Feature newFeature = Feature.fromProto(featureSpec); + addFeature(newFeature); + } + } + public void addEntities(List entities) { for (Entity entity : entities) { addEntity(entity); @@ -185,28 +234,25 @@ public void addFeature(Feature feature) { public FeatureSetProto.FeatureSet toProto() throws InvalidProtocolBufferException { List entitySpecs = new ArrayList<>(); for (Entity entityField : entities) { - EntitySpec.Builder entitySpecBuilder = EntitySpec.newBuilder(); - setEntitySpecFields(entitySpecBuilder, entityField); - entitySpecs.add(entitySpecBuilder.build()); + entitySpecs.add(entityField.toProto()); } List featureSpecs = new ArrayList<>(); for (Feature featureField : features) { - FeatureSpec.Builder featureSpecBuilder = FeatureSpec.newBuilder(); - setFeatureSpecFields(featureSpecBuilder, featureField); - featureSpecs.add(featureSpecBuilder.build()); + if (!featureField.isArchived()) { + featureSpecs.add(featureField.toProto()); + } } FeatureSetMeta.Builder meta = FeatureSetMeta.newBuilder() .setCreatedTimestamp( Timestamp.newBuilder().setSeconds(super.getCreated().getTime() / 1000L)) - .setStatus(FeatureSetStatus.valueOf(status)); + .setStatus(status); FeatureSetSpec.Builder spec = FeatureSetSpec.newBuilder() .setName(getName()) - .setVersion(getVersion()) .setProject(project.getName()) .setMaxAge(Duration.newBuilder().setSeconds(maxAgeSeconds)) .addAllEntities(entitySpecs) @@ -217,71 +263,24 @@ public FeatureSetProto.FeatureSet toProto() throws InvalidProtocolBufferExceptio return FeatureSetProto.FeatureSet.newBuilder().setMeta(meta).setSpec(spec).build(); } - private void setEntitySpecFields(EntitySpec.Builder entitySpecBuilder, Entity entityField) { - entitySpecBuilder - .setName(entityField.getName()) - .setValueType(Enum.valueOf(entityField.getType())); + @Override + public int hashCode() { + HashCodeBuilder hcb = new HashCodeBuilder(); + hcb.append(project.getName()); + hcb.append(getName()); + return hcb.toHashCode(); } - private void setFeatureSpecFields(FeatureSpec.Builder featureSpecBuilder, Feature featureField) - throws InvalidProtocolBufferException { - featureSpecBuilder - .setName(featureField.getName()) - .setValueType(Enum.valueOf(featureField.getType())); - - if (featureField.getPresence() != null) { - featureSpecBuilder.setPresence(FeaturePresence.parseFrom(featureField.getPresence())); - } else if (featureField.getGroupPresence() != null) { - featureSpecBuilder.setGroupPresence( - FeaturePresenceWithinGroup.parseFrom(featureField.getGroupPresence())); - } - - if (featureField.getShape() != null) { - featureSpecBuilder.setShape(FixedShape.parseFrom(featureField.getShape())); - } else if (featureField.getValueCount() != null) { - featureSpecBuilder.setValueCount(ValueCount.parseFrom(featureField.getValueCount())); - } - - if (featureField.getDomain() != null) { - featureSpecBuilder.setDomain(featureField.getDomain()); - } else if (featureField.getIntDomain() != null) { - featureSpecBuilder.setIntDomain(IntDomain.parseFrom(featureField.getIntDomain())); - } else if (featureField.getFloatDomain() != null) { - featureSpecBuilder.setFloatDomain(FloatDomain.parseFrom(featureField.getFloatDomain())); - } else if (featureField.getStringDomain() != null) { - featureSpecBuilder.setStringDomain(StringDomain.parseFrom(featureField.getStringDomain())); - } else if (featureField.getBoolDomain() != null) { - featureSpecBuilder.setBoolDomain(BoolDomain.parseFrom(featureField.getBoolDomain())); - } else if (featureField.getStructDomain() != null) { - featureSpecBuilder.setStructDomain(StructDomain.parseFrom(featureField.getStructDomain())); - } else if (featureField.getNaturalLanguageDomain() != null) { - featureSpecBuilder.setNaturalLanguageDomain( - NaturalLanguageDomain.parseFrom(featureField.getNaturalLanguageDomain())); - } else if (featureField.getImageDomain() != null) { - featureSpecBuilder.setImageDomain(ImageDomain.parseFrom(featureField.getImageDomain())); - } else if (featureField.getMidDomain() != null) { - featureSpecBuilder.setMidDomain(MIDDomain.parseFrom(featureField.getMidDomain())); - } else if (featureField.getUrlDomain() != null) { - featureSpecBuilder.setUrlDomain(URLDomain.parseFrom(featureField.getUrlDomain())); - } else if (featureField.getTimeDomain() != null) { - featureSpecBuilder.setTimeDomain(TimeDomain.parseFrom(featureField.getTimeDomain())); - } else if (featureField.getTimeOfDayDomain() != null) { - featureSpecBuilder.setTimeOfDayDomain( - TimeOfDayDomain.parseFrom(featureField.getTimeOfDayDomain())); + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; } - - if (featureField.getLabels() != null) { - featureSpecBuilder.putAllLabels(featureField.getLabels()); + if (!(obj instanceof FeatureSet)) { + return false; } - } - /** - * Checks if the given featureSet's schema and source has is different from this one. - * - * @param other FeatureSet to compare to - * @return boolean denoting if the source or schema have changed. - */ - public boolean equalTo(FeatureSet other) { + FeatureSet other = (FeatureSet) obj; if (!getName().equals(other.getName())) { return false; } @@ -343,29 +342,4 @@ public boolean equalTo(FeatureSet other) { return true; } - - @Override - public int hashCode() { - HashCodeBuilder hcb = new HashCodeBuilder(); - hcb.append(project.getName()); - hcb.append(getName()); - hcb.append(getVersion()); - return hcb.toHashCode(); - } - - @Override - public boolean equals(Object obj) { - if (this == obj) { - return true; - } - if (!(obj instanceof FeatureSet)) { - return false; - } - return this.equalTo(((FeatureSet) obj)); - } - - @Override - public int compareTo(FeatureSet o) { - return Integer.compare(getVersion(), o.getVersion()); - } } diff --git a/core/src/main/java/feast/core/model/Store.java b/core/src/main/java/feast/core/model/Store.java index debf211ec8d..a08c50b28a6 100644 --- a/core/src/main/java/feast/core/model/Store.java +++ b/core/src/main/java/feast/core/model/Store.java @@ -125,22 +125,18 @@ public List getSubscriptions() { } private static String convertSubscriptionToString(Subscription sub) { - if (sub.getVersion().isEmpty() || sub.getName().isEmpty() || sub.getProject().isEmpty()) { + if (sub.getName().isEmpty() || sub.getProject().isEmpty()) { throw new IllegalArgumentException( String.format("Missing arguments in subscription string: %s", sub.toString())); } - return String.format("%s:%s:%s", sub.getProject(), sub.getName(), sub.getVersion()); + return String.format("%s:%s", sub.getProject(), sub.getName()); } private Subscription convertStringToSubscription(String sub) { if (sub.equals("")) { return Subscription.newBuilder().build(); } - String[] split = sub.split(":", 3); - return Subscription.newBuilder() - .setProject(split[0]) - .setName(split[1]) - .setVersion(split[2]) - .build(); + String[] split = sub.split(":", 2); + return Subscription.newBuilder().setProject(split[0]).setName(split[1]).build(); } } diff --git a/core/src/main/java/feast/core/service/JobCoordinatorService.java b/core/src/main/java/feast/core/service/JobCoordinatorService.java index c0215767905..f61c1b829d4 100644 --- a/core/src/main/java/feast/core/service/JobCoordinatorService.java +++ b/core/src/main/java/feast/core/service/JobCoordinatorService.java @@ -17,10 +17,8 @@ package feast.core.service; import com.google.protobuf.InvalidProtocolBufferException; -import feast.core.CoreServiceProto.ListFeatureSetsRequest; import feast.core.CoreServiceProto.ListStoresRequest.Filter; import feast.core.CoreServiceProto.ListStoresResponse; -import feast.core.FeatureSetProto; import feast.core.FeatureSetProto.FeatureSetStatus; import feast.core.StoreProto; import feast.core.StoreProto.Store.Subscription; @@ -99,16 +97,11 @@ public void Poll() throws InvalidProtocolBufferException { Store store = Store.fromProto(storeSpec); for (Subscription subscription : store.getSubscriptions()) { - var featureSetSpecs = - specService - .listFeatureSets( - ListFeatureSetsRequest.Filter.newBuilder() - .setFeatureSetName(subscription.getName()) - .setFeatureSetVersion(subscription.getVersion()) - .setProject(subscription.getProject()) - .build()) - .getFeatureSetsList(); - featureSets.addAll(featureSetsFromProto(featureSetSpecs)); + List featureSetsForSub = + featureSetRepository.findAllByNameLikeAndProject_NameLikeOrderByNameAsc( + subscription.getName().replace('*', '%'), + subscription.getProject().replace('*', '%')); + featureSets.addAll(featureSetsForSub); } featureSets.stream() @@ -171,12 +164,12 @@ private void updateFeatureSetStatuses(List jobUpdateTasks) { ready.removeAll(pending); ready.forEach( fs -> { - fs.setStatus(FeatureSetStatus.STATUS_READY.toString()); + fs.setStatus(FeatureSetStatus.STATUS_READY); featureSetRepository.save(fs); }); pending.forEach( fs -> { - fs.setStatus(FeatureSetStatus.STATUS_PENDING.toString()); + fs.setStatus(FeatureSetStatus.STATUS_PENDING); featureSetRepository.save(fs); }); featureSetRepository.flush(); @@ -194,15 +187,4 @@ public Optional getJob(Source source, Store store) { // return the latest return Optional.of(jobs.get(0)); } - - // TODO: optimize this to make less calls to the database. - private List featureSetsFromProto(List protos) { - return protos.stream() - .map(FeatureSetProto.FeatureSet::getSpec) - .map( - fs -> - featureSetRepository.findFeatureSetByNameAndProject_NameAndVersion( - fs.getName(), fs.getProject(), fs.getVersion())) - .collect(Collectors.toList()); - } } diff --git a/core/src/main/java/feast/core/service/JobService.java b/core/src/main/java/feast/core/service/JobService.java index 33c118999cc..246ca91a4bc 100644 --- a/core/src/main/java/feast/core/service/JobService.java +++ b/core/src/main/java/feast/core/service/JobService.java @@ -248,7 +248,6 @@ private ListFeatureSetsRequest.Filter toListFeatureSetFilter(FeatureSetReference // match featuresets using contents of featureset reference String fsName = fsReference.getName(); String fsProject = fsReference.getProject(); - Integer fsVersion = fsReference.getVersion(); // construct list featureset request filter using feature set reference // for proto3, default value for missing values: @@ -258,7 +257,6 @@ private ListFeatureSetsRequest.Filter toListFeatureSetFilter(FeatureSetReference ListFeatureSetsRequest.Filter.newBuilder() .setFeatureSetName((fsName != "") ? fsName : "*") .setProject((fsProject != "") ? fsProject : "*") - .setFeatureSetVersion((fsVersion != 0) ? fsVersion.toString() : "*") .build(); return filter; diff --git a/core/src/main/java/feast/core/service/SpecService.java b/core/src/main/java/feast/core/service/SpecService.java index 4a068cba353..d8efde1eace 100644 --- a/core/src/main/java/feast/core/service/SpecService.java +++ b/core/src/main/java/feast/core/service/SpecService.java @@ -19,7 +19,6 @@ import static feast.core.validators.Matchers.checkValidCharacters; import static feast.core.validators.Matchers.checkValidCharactersAllowAsterisk; -import com.google.common.collect.Ordering; import com.google.protobuf.InvalidProtocolBufferException; import feast.core.CoreServiceProto.ApplyFeatureSetResponse; import feast.core.CoreServiceProto.ApplyFeatureSetResponse.Status; @@ -49,7 +48,6 @@ import java.util.ArrayList; import java.util.List; import lombok.extern.slf4j.Slf4j; -import org.apache.commons.lang3.StringUtils; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Service; import org.springframework.transaction.annotation.Transactional; @@ -99,43 +97,24 @@ public GetFeatureSetResponse getFeatureSet(GetFeatureSetRequest request) if (request.getProject().isEmpty()) { throw new IllegalArgumentException("No project provided"); } - if (request.getVersion() < 0) { - throw new IllegalArgumentException("Version number cannot be less than 0"); - } FeatureSet featureSet; - // Filter the list based on version - if (request.getVersion() == 0) { - featureSet = - featureSetRepository.findFirstFeatureSetByNameLikeAndProject_NameOrderByVersionDesc( - request.getName(), request.getProject()); + featureSet = + featureSetRepository.findFeatureSetByNameAndProject_Name( + request.getName(), request.getProject()); - if (featureSet == null) { - throw new RetrievalException( - String.format("Feature set with name \"%s\" could not be found.", request.getName())); - } - } else { - featureSet = - featureSetRepository.findFeatureSetByNameAndProject_NameAndVersion( - request.getName(), request.getProject(), request.getVersion()); - - if (featureSet == null) { - throw new RetrievalException( - String.format( - "Feature set with name \"%s\" and version \"%s\" could " + "not be found.", - request.getName(), request.getVersion())); - } + if (featureSet == null) { + throw new RetrievalException( + String.format("Feature set with name \"%s\" could not be found.", request.getName())); } - - // Only a single item in list, return successfully return GetFeatureSetResponse.newBuilder().setFeatureSet(featureSet.toProto()).build(); } /** - * Return a list of feature sets matching the feature set name, version, and project provided in - * the filter. All fields are requried. Use '*' for all three arguments in order to return all - * feature sets and versions in all projects. + * Return a list of feature sets matching the feature set name and project provided in the filter. + * All fields are requried. Use '*' for all arguments in order to return all feature sets in all + * projects. * *

    Project name can be explicitly provided, or an asterisk can be provided to match all * projects. It is not possible to provide a combination of asterisks/wildcards and text. @@ -144,25 +123,17 @@ public GetFeatureSetResponse getFeatureSet(GetFeatureSetRequest request) * sets will be returned. Regex is not supported. Explicitly defining a feature set name is not * possible if a project name is not set explicitly * - *

    The version field can be one of - '*' - This will match all versions - 'latest' - This will - * match the latest feature set version - '<number>' - This will match a specific feature - * set version. This property can only be set if both the feature set name and project name are - * explicitly set. - * - * @param filter filter containing the desired featureSet name and version filter + * @param filter filter containing the desired featureSet name * @return ListFeatureSetsResponse with list of featureSets found matching the filter */ public ListFeatureSetsResponse listFeatureSets(ListFeatureSetsRequest.Filter filter) throws InvalidProtocolBufferException { String name = filter.getFeatureSetName(); String project = filter.getProject(); - String version = filter.getFeatureSetVersion(); - if (project.isEmpty() || name.isEmpty() || version.isEmpty()) { + if (project.isEmpty() || name.isEmpty()) { throw new IllegalArgumentException( - String.format( - "Invalid listFeatureSetRequest, missing arguments. Must provide project, feature set name, and version.", - filter.toString())); + "Invalid listFeatureSetRequest, missing arguments. Must provide project and feature set name."); } checkValidCharactersAllowAsterisk(name, "featureSetName"); @@ -170,56 +141,34 @@ public ListFeatureSetsResponse listFeatureSets(ListFeatureSetsRequest.Filter fil List featureSets = new ArrayList() {}; - if (project.equals("*")) { - // Matching all projects - - if (name.equals("*") && version.equals("*")) { + if (project.contains("*")) { + // Matching a wildcard project + if (name.contains("*")) { featureSets = - featureSetRepository.findAllByNameLikeAndProject_NameLikeOrderByNameAscVersionAsc( + featureSetRepository.findAllByNameLikeAndProject_NameLikeOrderByNameAsc( name.replace('*', '%'), project.replace('*', '%')); } else { throw new IllegalArgumentException( String.format( - "Invalid listFeatureSetRequest. Version and feature set name must be set to " + "Invalid listFeatureSetRequest. Feature set name must be set to " + "\"*\" if the project name and feature set name aren't set explicitly: \n%s", filter.toString())); } } else if (!project.contains("*")) { // Matching a specific project - - if (name.contains("*") && version.equals("*")) { - // Find all feature sets matching a pattern and versions in a specific project - featureSets = - featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( - name.replace('*', '%'), project); - - } else if (!name.contains("*") && version.equals("*")) { - // Find all versions of a specific feature set in a specific project + if (name.contains("*")) { + // Find all feature sets matching a pattern in a specific project featureSets = - featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( - name, project); - - } else if (version.equals("latest")) { - // Find the latest version of a feature set matching a specific pattern in a specific - // project - FeatureSet latestFeatureSet = - featureSetRepository.findFirstFeatureSetByNameLikeAndProject_NameOrderByVersionDesc( + featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAsc( name.replace('*', '%'), project); - featureSets.add(latestFeatureSet); - } else if (!name.contains("*") && StringUtils.isNumeric(version)) { - // Find a specific version of a feature set matching a specific name in a specific project - FeatureSet specificFeatureSet = - featureSetRepository.findFeatureSetByNameAndProject_NameAndVersion( - name, project, Integer.parseInt(version)); - featureSets.add(specificFeatureSet); - - } else { - throw new IllegalArgumentException( - String.format( - "Invalid listFeatureSetRequest. Version must be set to \"*\" if the project " - + "name and feature set name aren't set explicitly: \n%s", - filter.toString())); + } else if (!name.contains("*")) { + // Find a specific feature set in a specific project + FeatureSet featureSet = + featureSetRepository.findFeatureSetByNameAndProject_Name(name, project); + if (featureSet != null) { + featureSets.add(featureSet); + } } } else { throw new IllegalArgumentException( @@ -274,8 +223,7 @@ public ListStoresResponse listStores(ListStoresRequest.Filter filter) { } /** - * Creates or updates a feature set in the repository. If there is a change in the feature set - * schema, then the feature set version will be incremented. + * Creates or updates a feature set in the repository. * *

    This function is idempotent. If no changes are detected in the incoming featureSet's schema, * this method will update the incoming featureSet spec with the latest version stored in the @@ -301,54 +249,48 @@ public ApplyFeatureSetResponse applyFeatureSet(FeatureSetProto.FeatureSet newFea throw new IllegalArgumentException(String.format("Project is archived: %s", project_name)); } - // Retrieve all existing FeatureSet objects - List existingFeatureSets = - featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( - newFeatureSet.getSpec().getName(), project_name); - - if (existingFeatureSets.size() == 0) { - // Create new feature set since it doesn't exist + // Set source to default if not set in proto + if (newFeatureSet.getSpec().getSource() == SourceProto.Source.getDefaultInstance()) { newFeatureSet = newFeatureSet .toBuilder() - .setSpec(newFeatureSet.getSpec().toBuilder().setVersion(1)) + .setSpec( + newFeatureSet.getSpec().toBuilder().setSource(defaultSource.toProto()).build()) .build(); - } else { - // Retrieve the latest feature set if the name does exist - existingFeatureSets = Ordering.natural().reverse().sortedCopy(existingFeatureSets); - FeatureSet latest = existingFeatureSets.get(0); - FeatureSet featureSet = FeatureSet.fromProto(newFeatureSet); + } + + // Retrieve existing FeatureSet + FeatureSet featureSet = + featureSetRepository.findFeatureSetByNameAndProject_Name( + newFeatureSet.getSpec().getName(), project_name); + Status status; + if (featureSet == null) { + // Create new feature set since it doesn't exist + newFeatureSet = newFeatureSet.toBuilder().setSpec(newFeatureSet.getSpec()).build(); + featureSet = FeatureSet.fromProto(newFeatureSet); + status = Status.CREATED; + } else { // If the featureSet remains unchanged, we do nothing. - if (featureSet.equalTo(latest)) { + if (featureSet.toProto().getSpec().equals(newFeatureSet.getSpec())) { return ApplyFeatureSetResponse.newBuilder() - .setFeatureSet(latest.toProto()) + .setFeatureSet(featureSet.toProto()) .setStatus(Status.NO_CHANGE) .build(); } - // TODO: There is a race condition here with incrementing the version - newFeatureSet = - newFeatureSet - .toBuilder() - .setSpec(newFeatureSet.getSpec().toBuilder().setVersion(latest.getVersion() + 1)) - .build(); - } - - // Build a new FeatureSet object which includes the new properties - FeatureSet featureSet = FeatureSet.fromProto(newFeatureSet); - featureSet.setStatus(FeatureSetStatus.STATUS_PENDING.toString()); - if (newFeatureSet.getSpec().getSource() == SourceProto.Source.getDefaultInstance()) { - featureSet.setSource(defaultSource); + featureSet.updateFromProto(newFeatureSet); + status = Status.UPDATED; } // Persist the FeatureSet object + featureSet.setStatus(FeatureSetStatus.STATUS_PENDING); project.addFeatureSet(featureSet); projectRepository.saveAndFlush(project); // Build ApplyFeatureSetResponse return ApplyFeatureSetResponse.newBuilder() .setFeatureSet(featureSet.toProto()) - .setStatus(Status.CREATED) + .setStatus(status) .build(); } @@ -366,7 +308,7 @@ public UpdateStoreResponse updateStore(UpdateStoreRequest updateStoreRequest) List subs = newStoreProto.getSubscriptionsList(); for (Subscription sub : subs) { // Ensure that all fields in a subscription contain values - if ((sub.getVersion().isEmpty() || sub.getName().isEmpty()) || sub.getProject().isEmpty()) { + if ((sub.getName().isEmpty()) || sub.getProject().isEmpty()) { throw new IllegalArgumentException( String.format("Missing parameter in subscription: %s", sub)); } diff --git a/core/src/test/java/feast/core/job/JobUpdateTaskTest.java b/core/src/test/java/feast/core/job/JobUpdateTaskTest.java index 8d179baebb1..5570c71a99a 100644 --- a/core/src/test/java/feast/core/job/JobUpdateTaskTest.java +++ b/core/src/test/java/feast/core/job/JobUpdateTaskTest.java @@ -54,7 +54,7 @@ public class JobUpdateTaskTest { private static final FeatureSetProto.FeatureSet.Builder fsBuilder = FeatureSetProto.FeatureSet.newBuilder().setMeta(FeatureSetMeta.newBuilder()); private static final FeatureSetSpec.Builder specBuilder = - FeatureSetSpec.newBuilder().setProject("project1").setVersion(1); + FeatureSetSpec.newBuilder().setProject("project1"); @Mock private JobManager jobManager; @@ -73,8 +73,7 @@ public void setUp() { .setName("test") .setType(StoreType.REDIS) .setRedisConfig(RedisConfig.newBuilder().build()) - .addSubscriptions( - Subscription.newBuilder().setProject("*").setName("*").setVersion("*").build()) + .addSubscriptions(Subscription.newBuilder().setProject("*").setName("*").build()) .build()); source = diff --git a/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java b/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java index 72b921ef694..55e8a573771 100644 --- a/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java +++ b/core/src/test/java/feast/core/job/dataflow/DataflowJobManagerTest.java @@ -103,8 +103,7 @@ public void shouldStartJobWithCorrectPipelineOptions() throws IOException { .setName("SERVING") .setType(StoreType.REDIS) .setRedisConfig(RedisConfig.newBuilder().setHost("localhost").setPort(6379).build()) - .addSubscriptions( - Subscription.newBuilder().setProject("*").setName("*").setVersion("*").build()) + .addSubscriptions(Subscription.newBuilder().setProject("*").setName("*").build()) .build(); SourceProto.Source source = @@ -124,7 +123,6 @@ public void shouldStartJobWithCorrectPipelineOptions() throws IOException { FeatureSetSpec.newBuilder() .setSource(source) .setName("featureSet") - .setVersion(1) .setMaxAge(Duration.newBuilder().build())) .build(); @@ -220,12 +218,7 @@ public void shouldThrowExceptionWhenJobStateTerminal() throws IOException { FeatureSetProto.FeatureSet featureSet = FeatureSetProto.FeatureSet.newBuilder() - .setSpec( - FeatureSetSpec.newBuilder() - .setName("featureSet") - .setVersion(1) - .setSource(source) - .build()) + .setSpec(FeatureSetSpec.newBuilder().setName("featureSet").setSource(source).build()) .build(); dfJobManager = Mockito.spy(dfJobManager); diff --git a/core/src/test/java/feast/core/job/direct/DirectRunnerJobManagerTest.java b/core/src/test/java/feast/core/job/direct/DirectRunnerJobManagerTest.java index 6980450ca4d..914177385e4 100644 --- a/core/src/test/java/feast/core/job/direct/DirectRunnerJobManagerTest.java +++ b/core/src/test/java/feast/core/job/direct/DirectRunnerJobManagerTest.java @@ -91,8 +91,7 @@ public void shouldStartDirectJobAndRegisterPipelineResult() throws IOException { .setName("SERVING") .setType(StoreType.REDIS) .setRedisConfig(RedisConfig.newBuilder().setHost("localhost").setPort(6379).build()) - .addSubscriptions( - Subscription.newBuilder().setProject("*").setName("*").setVersion("*").build()) + .addSubscriptions(Subscription.newBuilder().setProject("*").setName("*").build()) .build(); SourceProto.Source source = @@ -110,7 +109,6 @@ public void shouldStartDirectJobAndRegisterPipelineResult() throws IOException { .setSpec( FeatureSetSpec.newBuilder() .setName("featureSet") - .setVersion(1) .setMaxAge(Duration.newBuilder()) .setSource(source) .build()) @@ -118,8 +116,10 @@ public void shouldStartDirectJobAndRegisterPipelineResult() throws IOException { Printer printer = JsonFormat.printer(); + String expectedJobId = "feast-job-0"; ImportOptions expectedPipelineOptions = PipelineOptionsFactory.fromArgs("").as(ImportOptions.class); + expectedPipelineOptions.setJobName(expectedJobId); expectedPipelineOptions.setAppName("DirectRunnerJobManager"); expectedPipelineOptions.setRunner(DirectRunner.class); expectedPipelineOptions.setBlockOnRun(false); @@ -132,7 +132,6 @@ public void shouldStartDirectJobAndRegisterPipelineResult() throws IOException { expectedPipelineOptions.setFeatureSetJson( featureSetJsonCompressor.compress(Collections.singletonList(featureSet))); - String expectedJobId = "feast-job-0"; ArgumentCaptor pipelineOptionsCaptor = ArgumentCaptor.forClass(ImportOptions.class); ArgumentCaptor directJobCaptor = ArgumentCaptor.forClass(DirectJob.class); diff --git a/core/src/test/java/feast/core/job/option/FeatureSetJsonByteConverterTest.java b/core/src/test/java/feast/core/job/option/FeatureSetJsonByteConverterTest.java index 2dfeef1d969..df9044e6331 100644 --- a/core/src/test/java/feast/core/job/option/FeatureSetJsonByteConverterTest.java +++ b/core/src/test/java/feast/core/job/option/FeatureSetJsonByteConverterTest.java @@ -29,7 +29,7 @@ public class FeatureSetJsonByteConverterTest { - private FeatureSetProto.FeatureSet newFeatureSet(Integer version, Integer numberOfFeatures) { + private FeatureSetProto.FeatureSet newFeatureSet(Integer numberOfFeatures) { List features = IntStream.range(1, numberOfFeatures + 1) .mapToObj( @@ -51,7 +51,6 @@ private FeatureSetProto.FeatureSet newFeatureSet(Integer version, Integer number .setBootstrapServers("somebrokers:9092") .setTopic("sometopic"))) .addAllFeatures(features) - .setVersion(version) .addEntities( FeatureSetProto.EntitySpec.newBuilder() .setName("entity") @@ -65,12 +64,11 @@ public void shouldConvertFeatureSetsAsJsonStringBytes() throws InvalidProtocolBu int nrOfFeatures = 1; List featureSets = IntStream.range(1, nrOfFeatureSet + 1) - .mapToObj(i -> newFeatureSet(i, nrOfFeatures)) + .mapToObj(i -> newFeatureSet(nrOfFeatures)) .collect(Collectors.toList()); String expectedOutputString = - "{\"version\":1," - + "\"entities\":[{\"name\":\"entity\",\"valueType\":2}]," + "{\"entities\":[{\"name\":\"entity\",\"valueType\":2}]," + "\"features\":[{\"name\":\"feature1\",\"valueType\":6}]," + "\"source\":{" + "\"type\":1," diff --git a/core/src/test/java/feast/core/model/FeatureSetTest.java b/core/src/test/java/feast/core/model/FeatureSetTest.java new file mode 100644 index 00000000000..70d160e875c --- /dev/null +++ b/core/src/test/java/feast/core/model/FeatureSetTest.java @@ -0,0 +1,205 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.core.model; + +import static org.hamcrest.CoreMatchers.containsString; +import static org.hamcrest.CoreMatchers.equalTo; +import static org.junit.Assert.assertThat; + +import com.google.protobuf.Duration; +import com.google.protobuf.InvalidProtocolBufferException; +import feast.core.FeatureSetProto; +import feast.core.FeatureSetProto.EntitySpec; +import feast.core.FeatureSetProto.FeatureSetSpec; +import feast.core.FeatureSetProto.FeatureSetStatus; +import feast.core.FeatureSetProto.FeatureSpec; +import feast.core.SourceProto; +import feast.core.SourceProto.KafkaSourceConfig; +import feast.core.SourceProto.SourceType; +import feast.types.ValueProto.ValueType.Enum; +import org.junit.Before; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.ExpectedException; +import org.tensorflow.metadata.v0.IntDomain; + +public class FeatureSetTest { + @Rule public final ExpectedException expectedException = ExpectedException.none(); + + private FeatureSetProto.FeatureSet oldFeatureSetProto; + + @Before + public void setUp() { + SourceProto.Source oldSource = + SourceProto.Source.newBuilder() + .setType(SourceType.KAFKA) + .setKafkaSourceConfig( + KafkaSourceConfig.newBuilder() + .setBootstrapServers("kafka:9092") + .setTopic("mytopic")) + .build(); + + oldFeatureSetProto = + FeatureSetProto.FeatureSet.newBuilder() + .setSpec( + FeatureSetSpec.newBuilder() + .setName("featureSet") + .setProject("project") + .setMaxAge(Duration.newBuilder().setSeconds(100)) + .setSource(oldSource) + .addFeatures( + FeatureSpec.newBuilder().setName("feature1").setValueType(Enum.INT64)) + .addFeatures( + FeatureSpec.newBuilder().setName("feature2").setValueType(Enum.STRING)) + .addEntities( + EntitySpec.newBuilder().setName("entity").setValueType(Enum.STRING)) + .build()) + .build(); + } + + @Test + public void shouldUpdateFromProto() throws InvalidProtocolBufferException { + SourceProto.Source newSource = + SourceProto.Source.newBuilder() + .setType(SourceType.KAFKA) + .setKafkaSourceConfig( + KafkaSourceConfig.newBuilder() + .setBootstrapServers("kafka:9092") + .setTopic("mytopic-changed")) + .build(); + + FeatureSetProto.FeatureSet newFeatureSetProto = + FeatureSetProto.FeatureSet.newBuilder() + .setSpec( + FeatureSetSpec.newBuilder() + .setName("featureSet") + .setProject("project") + .setMaxAge(Duration.newBuilder().setSeconds(101)) + .setSource(newSource) + .addFeatures( + FeatureSpec.newBuilder() + .setName("feature1") + .setValueType(Enum.INT64) + .setIntDomain(IntDomain.newBuilder().setMax(10).setMin(0))) + .addFeatures( + FeatureSpec.newBuilder().setName("feature3").setValueType(Enum.STRING)) + .addEntities( + EntitySpec.newBuilder().setName("entity").setValueType(Enum.STRING)) + .build()) + .build(); + + FeatureSet actual = FeatureSet.fromProto(oldFeatureSetProto); + actual.updateFromProto(newFeatureSetProto); + + FeatureSet expected = FeatureSet.fromProto(newFeatureSetProto); + Feature archivedFeature = + Feature.fromProto( + FeatureSpec.newBuilder().setName("feature2").setValueType(Enum.STRING).build()); + archivedFeature.setArchived(true); + expected.addFeature(archivedFeature); + assertThat(actual, equalTo(expected)); + } + + @Test + public void shouldNotUpdateIfNoChange() throws InvalidProtocolBufferException { + FeatureSet actual = FeatureSet.fromProto(oldFeatureSetProto); + actual.setStatus(FeatureSetStatus.STATUS_READY); + actual.updateFromProto(oldFeatureSetProto); + + FeatureSet expected = FeatureSet.fromProto(oldFeatureSetProto); + expected.setStatus(FeatureSetStatus.STATUS_READY); + + assertThat(actual, equalTo(expected)); + } + + @Test + public void shouldThrowExceptionIfUpdateWithEntitiesChanged() + throws InvalidProtocolBufferException { + SourceProto.Source newSource = + SourceProto.Source.newBuilder() + .setType(SourceType.KAFKA) + .setKafkaSourceConfig( + KafkaSourceConfig.newBuilder() + .setBootstrapServers("kafka:9092") + .setTopic("mytopic-changed")) + .build(); + + FeatureSetProto.FeatureSet newFeatureSetProto = + FeatureSetProto.FeatureSet.newBuilder() + .setSpec( + FeatureSetSpec.newBuilder() + .setName("featureSet") + .setProject("project") + .setMaxAge(Duration.newBuilder().setSeconds(101)) + .setSource(newSource) + .addFeatures( + FeatureSpec.newBuilder() + .setName("feature1") + .setValueType(Enum.INT64) + .setIntDomain(IntDomain.newBuilder().setMax(10).setMin(0))) + .addFeatures( + FeatureSpec.newBuilder().setName("feature3").setValueType(Enum.STRING)) + .addEntities(EntitySpec.newBuilder().setName("entity").setValueType(Enum.FLOAT)) + .build()) + .build(); + + expectedException.expect(IllegalArgumentException.class); + expectedException.expectMessage(containsString("does not match existing set of entities")); + FeatureSet existingFeatureSet = FeatureSet.fromProto(oldFeatureSetProto); + existingFeatureSet.updateFromProto(newFeatureSetProto); + } + + @Test + public void shouldThrowExceptionIfUpdateWithFeatureTypesChanged() + throws InvalidProtocolBufferException { + SourceProto.Source newSource = + SourceProto.Source.newBuilder() + .setType(SourceType.KAFKA) + .setKafkaSourceConfig( + KafkaSourceConfig.newBuilder() + .setBootstrapServers("kafka:9092") + .setTopic("mytopic-changed")) + .build(); + + FeatureSetProto.FeatureSet newFeatureSetProto = + FeatureSetProto.FeatureSet.newBuilder() + .setSpec( + FeatureSetSpec.newBuilder() + .setName("featureSet") + .setProject("project") + .setMaxAge(Duration.newBuilder().setSeconds(101)) + .setSource(newSource) + .addFeatures( + FeatureSpec.newBuilder() + .setName("feature1") + .setValueType(Enum.INT64) + .setIntDomain(IntDomain.newBuilder().setMax(10).setMin(0))) + .addFeatures( + FeatureSpec.newBuilder().setName("feature2").setValueType(Enum.FLOAT)) + .addEntities( + EntitySpec.newBuilder().setName("entity").setValueType(Enum.STRING)) + .build()) + .build(); + + expectedException.expect(IllegalArgumentException.class); + expectedException.expectMessage( + containsString( + "You are attempting to change the type of feature feature2 from STRING to FLOAT.")); + FeatureSet existingFeatureSet = FeatureSet.fromProto(oldFeatureSetProto); + existingFeatureSet.updateFromProto(newFeatureSetProto); + } +} diff --git a/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java b/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java index 26cf331c13b..38683c7bd59 100644 --- a/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java +++ b/core/src/test/java/feast/core/service/JobCoordinatorServiceTest.java @@ -95,17 +95,12 @@ public void shouldDoNothingIfNoMatchingFeatureSetsFound() throws InvalidProtocol .setName("test") .setType(StoreType.REDIS) .setRedisConfig(RedisConfig.newBuilder().build()) - .addSubscriptions( - Subscription.newBuilder().setProject("*").setName("*").setVersion("*").build()) + .addSubscriptions(Subscription.newBuilder().setProject("*").setName("*").build()) .build(); when(specService.listStores(any())) .thenReturn(ListStoresResponse.newBuilder().addStore(store).build()); when(specService.listFeatureSets( - Filter.newBuilder() - .setProject("*") - .setFeatureSetName("*") - .setFeatureSetVersion("*") - .build())) + Filter.newBuilder().setProject("*").setFeatureSetName("*").build())) .thenReturn(ListFeatureSetsResponse.newBuilder().build()); JobCoordinatorService jcs = new JobCoordinatorService( @@ -121,12 +116,7 @@ public void shouldGenerateAndSubmitJobsIfAny() throws InvalidProtocolBufferExcep .setName("test") .setType(StoreType.REDIS) .setRedisConfig(RedisConfig.newBuilder().build()) - .addSubscriptions( - Subscription.newBuilder() - .setProject("project1") - .setName("features") - .setVersion("*") - .build()) + .addSubscriptions(Subscription.newBuilder().setProject("project1").setName("*").build()) .build(); Source source = Source.newBuilder() @@ -138,26 +128,26 @@ public void shouldGenerateAndSubmitJobsIfAny() throws InvalidProtocolBufferExcep .build()) .build(); - FeatureSetProto.FeatureSet featureSet1 = + FeatureSetProto.FeatureSet featureSetProto1 = FeatureSetProto.FeatureSet.newBuilder() .setSpec( FeatureSetSpec.newBuilder() .setSource(source) .setProject("project1") - .setName("features") - .setVersion(1)) + .setName("features1")) .setMeta(FeatureSetMeta.newBuilder()) .build(); - FeatureSetProto.FeatureSet featureSet2 = + FeatureSet featureSet1 = FeatureSet.fromProto(featureSetProto1); + FeatureSetProto.FeatureSet featureSetProto2 = FeatureSetProto.FeatureSet.newBuilder() .setSpec( FeatureSetSpec.newBuilder() .setSource(source) .setProject("project1") - .setName("features") - .setVersion(2)) + .setName("features2")) .setMeta(FeatureSetMeta.newBuilder()) .build(); + FeatureSet featureSet2 = FeatureSet.fromProto(featureSetProto2); String extId = "ext"; ArgumentCaptor jobArgCaptor = ArgumentCaptor.forClass(Job.class); @@ -168,7 +158,7 @@ public void shouldGenerateAndSubmitJobsIfAny() throws InvalidProtocolBufferExcep Runner.DATAFLOW, feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), - Arrays.asList(FeatureSet.fromProto(featureSet1), FeatureSet.fromProto(featureSet2)), + Arrays.asList(featureSet1, featureSet2), JobStatus.PENDING); Job expected = @@ -178,30 +168,14 @@ public void shouldGenerateAndSubmitJobsIfAny() throws InvalidProtocolBufferExcep Runner.DATAFLOW, feast.core.model.Source.fromProto(source), feast.core.model.Store.fromProto(store), - Arrays.asList(FeatureSet.fromProto(featureSet1), FeatureSet.fromProto(featureSet2)), + Arrays.asList(featureSet1, featureSet2), JobStatus.RUNNING); - when(specService.listFeatureSets( - Filter.newBuilder() - .setProject("project1") - .setFeatureSetName("features") - .setFeatureSetVersion("*") - .build())) - .thenReturn( - ListFeatureSetsResponse.newBuilder() - .addFeatureSets(featureSet1) - .addFeatureSets(featureSet2) - .build()); + when(featureSetRepository.findAllByNameLikeAndProject_NameLikeOrderByNameAsc("%", "project1")) + .thenReturn(Lists.newArrayList(featureSet1, featureSet2)); when(specService.listStores(any())) .thenReturn(ListStoresResponse.newBuilder().addStore(store).build()); - for (FeatureSetProto.FeatureSet fs : Lists.newArrayList(featureSet1, featureSet2)) { - FeatureSetSpec spec = fs.getSpec(); - when(featureSetRepository.findFeatureSetByNameAndProject_NameAndVersion( - spec.getName(), spec.getProject(), spec.getVersion())) - .thenReturn(FeatureSet.fromProto(fs)); - } - when(jobManager.startJob(argThat(new JobMatcher(expectedInput)))).thenReturn(expected); when(jobManager.getRunnerType()).thenReturn(Runner.DATAFLOW); @@ -221,12 +195,7 @@ public void shouldGroupJobsBySource() throws InvalidProtocolBufferException { .setName("test") .setType(StoreType.REDIS) .setRedisConfig(RedisConfig.newBuilder().build()) - .addSubscriptions( - Subscription.newBuilder() - .setProject("project1") - .setName("features") - .setVersion("*") - .build()) + .addSubscriptions(Subscription.newBuilder().setProject("project1").setName("*").build()) .build(); Source source1 = Source.newBuilder() @@ -247,26 +216,27 @@ public void shouldGroupJobsBySource() throws InvalidProtocolBufferException { .build()) .build(); - FeatureSetProto.FeatureSet featureSet1 = + FeatureSetProto.FeatureSet featureSetProto1 = FeatureSetProto.FeatureSet.newBuilder() .setSpec( FeatureSetSpec.newBuilder() .setSource(source1) .setProject("project1") - .setName("features") - .setVersion(1)) + .setName("features1")) .setMeta(FeatureSetMeta.newBuilder()) .build(); - FeatureSetProto.FeatureSet featureSet2 = + FeatureSet featureSet1 = FeatureSet.fromProto(featureSetProto1); + + FeatureSetProto.FeatureSet featureSetProto2 = FeatureSetProto.FeatureSet.newBuilder() .setSpec( FeatureSetSpec.newBuilder() .setSource(source2) .setProject("project1") - .setName("features") - .setVersion(2)) + .setName("features2")) .setMeta(FeatureSetMeta.newBuilder()) .build(); + FeatureSet featureSet2 = FeatureSet.fromProto(featureSetProto2); Job expectedInput1 = new Job( @@ -275,7 +245,7 @@ public void shouldGroupJobsBySource() throws InvalidProtocolBufferException { Runner.DATAFLOW, feast.core.model.Source.fromProto(source1), feast.core.model.Store.fromProto(store), - Arrays.asList(FeatureSet.fromProto(featureSet1)), + Arrays.asList(featureSet1), JobStatus.PENDING); Job expected1 = @@ -285,7 +255,7 @@ public void shouldGroupJobsBySource() throws InvalidProtocolBufferException { Runner.DATAFLOW, feast.core.model.Source.fromProto(source1), feast.core.model.Store.fromProto(store), - Arrays.asList(FeatureSet.fromProto(featureSet1)), + Arrays.asList(featureSet1), JobStatus.RUNNING); Job expectedInput2 = @@ -295,7 +265,7 @@ public void shouldGroupJobsBySource() throws InvalidProtocolBufferException { Runner.DATAFLOW, feast.core.model.Source.fromProto(source2), feast.core.model.Store.fromProto(store), - Arrays.asList(FeatureSet.fromProto(featureSet2)), + Arrays.asList(featureSet2), JobStatus.PENDING); Job expected2 = @@ -305,33 +275,19 @@ public void shouldGroupJobsBySource() throws InvalidProtocolBufferException { Runner.DATAFLOW, feast.core.model.Source.fromProto(source2), feast.core.model.Store.fromProto(store), - Arrays.asList(FeatureSet.fromProto(featureSet2)), + Arrays.asList(featureSet2), JobStatus.RUNNING); ArgumentCaptor jobArgCaptor = ArgumentCaptor.forClass(Job.class); - when(specService.listFeatureSets( - Filter.newBuilder() - .setProject("project1") - .setFeatureSetName("features") - .setFeatureSetVersion("*") - .build())) - .thenReturn( - ListFeatureSetsResponse.newBuilder() - .addFeatureSets(featureSet1) - .addFeatureSets(featureSet2) - .build()); + when(featureSetRepository.findAllByNameLikeAndProject_NameLikeOrderByNameAsc("%", "project1")) + .thenReturn(Lists.newArrayList(featureSet1, featureSet2)); + when(specService.listStores(any())) .thenReturn(ListStoresResponse.newBuilder().addStore(store).build()); when(jobManager.startJob(argThat(new JobMatcher(expectedInput1)))).thenReturn(expected1); when(jobManager.startJob(argThat(new JobMatcher(expectedInput2)))).thenReturn(expected2); when(jobManager.getRunnerType()).thenReturn(Runner.DATAFLOW); - for (FeatureSetProto.FeatureSet fs : Lists.newArrayList(featureSet1, featureSet2)) { - FeatureSetSpec spec = fs.getSpec(); - when(featureSetRepository.findFeatureSetByNameAndProject_NameAndVersion( - spec.getName(), spec.getProject(), spec.getVersion())) - .thenReturn(FeatureSet.fromProto(fs)); - } JobCoordinatorService jcs = new JobCoordinatorService( diff --git a/core/src/test/java/feast/core/service/JobServiceTest.java b/core/src/test/java/feast/core/service/JobServiceTest.java index ba663020191..0fe8e22f0a9 100644 --- a/core/src/test/java/feast/core/service/JobServiceTest.java +++ b/core/src/test/java/feast/core/service/JobServiceTest.java @@ -148,7 +148,7 @@ private FeatureSet newDummyFeatureSet(String name, int version, String project) FeatureSet fs = TestObjectFactory.CreateFeatureSet( - name, project, version, Arrays.asList(entity), Arrays.asList(feature)); + name, project, Arrays.asList(entity), Arrays.asList(feature)); fs.setCreated(Date.from(Instant.ofEpochSecond(10L))); return fs; } @@ -168,7 +168,6 @@ private List newDummyFeatureSetReferences() { return Arrays.asList( // all provided: name, version and project FeatureSetReference.newBuilder() - .setVersion(this.featureSet.getVersion()) .setName(this.featureSet.getName()) .setProject(this.featureSet.getProject().toString()) .build(), @@ -180,10 +179,7 @@ private List newDummyFeatureSetReferences() { .build(), // name and version - FeatureSetReference.newBuilder() - .setName(this.featureSet.getName()) - .setVersion(this.featureSet.getVersion()) - .build()); + FeatureSetReference.newBuilder().setName(this.featureSet.getName()).build()); } private List newDummyListRequestFilters() { @@ -192,21 +188,18 @@ private List newDummyListRequestFilters() { ListFeatureSetsRequest.Filter.newBuilder() .setFeatureSetName(this.featureSet.getName()) .setProject(this.featureSet.getProject().toString()) - .setFeatureSetVersion(String.valueOf(this.featureSet.getVersion())) .build(), // name and project ListFeatureSetsRequest.Filter.newBuilder() .setFeatureSetName(this.featureSet.getName()) .setProject(this.featureSet.getProject().toString()) - .setFeatureSetVersion("*") .build(), // name and project ListFeatureSetsRequest.Filter.newBuilder() .setFeatureSetName(this.featureSet.getName()) .setProject("*") - .setFeatureSetVersion(String.valueOf(this.featureSet.getVersion())) .build()); } diff --git a/core/src/test/java/feast/core/service/SpecServiceTest.java b/core/src/test/java/feast/core/service/SpecServiceTest.java index 413a97e64b0..576bdeb8926 100644 --- a/core/src/test/java/feast/core/service/SpecServiceTest.java +++ b/core/src/test/java/feast/core/service/SpecServiceTest.java @@ -29,7 +29,6 @@ import feast.core.CoreServiceProto.ApplyFeatureSetResponse; import feast.core.CoreServiceProto.ApplyFeatureSetResponse.Status; import feast.core.CoreServiceProto.GetFeatureSetRequest; -import feast.core.CoreServiceProto.GetFeatureSetResponse; import feast.core.CoreServiceProto.ListFeatureSetsRequest.Filter; import feast.core.CoreServiceProto.ListFeatureSetsResponse; import feast.core.CoreServiceProto.ListStoresRequest; @@ -52,15 +51,8 @@ import feast.types.ValueProto.ValueType.Enum; import java.sql.Date; import java.time.Instant; -import java.util.ArrayList; -import java.util.Arrays; -import java.util.Collections; -import java.util.Comparator; -import java.util.HashMap; -import java.util.List; -import java.util.Map; +import java.util.*; import java.util.Map.Entry; -import java.util.Optional; import java.util.stream.Collectors; import org.junit.Before; import org.junit.Rule; @@ -100,49 +92,37 @@ public class SpecServiceTest { private List stores; private Source defaultSource; + // TODO: Updates update features in place, so if tests follow the wrong order they might break. + // Refactor this maybe? @Before public void setUp() { initMocks(this); defaultSource = TestObjectFactory.defaultSource; - FeatureSet featureSet1v1 = newDummyFeatureSet("f1", 1, "project1"); - FeatureSet featureSet1v2 = newDummyFeatureSet("f1", 2, "project1"); - FeatureSet featureSet1v3 = newDummyFeatureSet("f1", 3, "project1"); - FeatureSet featureSet2v1 = newDummyFeatureSet("f2", 1, "project1"); + FeatureSet featureSet1 = newDummyFeatureSet("f1", "project1"); + FeatureSet featureSet2 = newDummyFeatureSet("f2", "project1"); Feature f3f1 = TestObjectFactory.CreateFeature("f3f1", Enum.INT64); Feature f3f2 = TestObjectFactory.CreateFeature("f3f2", Enum.INT64); Entity f3e1 = TestObjectFactory.CreateEntity("f3e1", Enum.STRING); FeatureSet featureSet3v1 = TestObjectFactory.CreateFeatureSet( - "f3", "project1", 1, Arrays.asList(f3e1), Arrays.asList(f3f2, f3f1)); + "f3", "project1", Arrays.asList(f3e1), Arrays.asList(f3f2, f3f1)); - featureSets = - Arrays.asList(featureSet1v1, featureSet1v2, featureSet1v3, featureSet2v1, featureSet3v1); + featureSets = Arrays.asList(featureSet1, featureSet2); when(featureSetRepository.findAll()).thenReturn(featureSets); - when(featureSetRepository.findAllByOrderByNameAscVersionAsc()).thenReturn(featureSets); - when(featureSetRepository.findFeatureSetByNameAndProject_NameAndVersion("f1", "project1", 1)) + when(featureSetRepository.findAllByOrderByNameAsc()).thenReturn(featureSets); + when(featureSetRepository.findFeatureSetByNameAndProject_Name("f1", "project1")) .thenReturn(featureSets.get(0)); - when(featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( - "f1", "project1")) - .thenReturn(featureSets.subList(0, 3)); - when(featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( - "f3", "project1")) - .thenReturn(featureSets.subList(4, 5)); - when(featureSetRepository.findFirstFeatureSetByNameLikeAndProject_NameOrderByVersionDesc( - "f1", "project1")) - .thenReturn(featureSet1v3); - when(featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( - "f1", "project1")) - .thenReturn(featureSets.subList(0, 3)); - when(featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( - "asd", "project1")) + when(featureSetRepository.findFeatureSetByNameAndProject_Name("f2", "project1")) + .thenReturn(featureSets.get(1)); + when(featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAsc("f1", "project1")) + .thenReturn(featureSets.subList(0, 1)); + when(featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAsc("asd", "project1")) .thenReturn(Lists.newArrayList()); - when(featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( - "f%", "project1")) + when(featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAsc("f%", "project1")) .thenReturn(featureSets); - when(featureSetRepository.findAllByNameLikeAndProject_NameLikeOrderByNameAscVersionAsc( - "%", "%")) + when(featureSetRepository.findAllByNameLikeAndProject_NameLikeOrderByNameAsc("%", "%")) .thenReturn(featureSets); when(projectRepository.findAllByArchivedIsFalse()) @@ -169,11 +149,7 @@ public void shouldGetAllFeatureSetsIfOnlyWildcardsProvided() throws InvalidProtocolBufferException { ListFeatureSetsResponse actual = specService.listFeatureSets( - Filter.newBuilder() - .setFeatureSetName("*") - .setProject("*") - .setFeatureSetVersion("*") - .build()); + Filter.newBuilder().setFeatureSetName("*").setProject("*").build()); List list = new ArrayList<>(); for (FeatureSet featureSet : featureSets) { FeatureSetProto.FeatureSet toProto = featureSet.toProto(); @@ -189,31 +165,8 @@ public void listFeatureSetShouldFailIfFeatureSetProvidedWithoutProject() throws InvalidProtocolBufferException { expectedException.expect(IllegalArgumentException.class); expectedException.expectMessage( - "Invalid listFeatureSetRequest, missing arguments. Must provide project, feature set name, and version."); - specService.listFeatureSets( - Filter.newBuilder().setFeatureSetName("f1").setFeatureSetVersion("1").build()); - } - - @Test - public void shouldGetAllFeatureSetsMatchingNameIfWildcardVersionProvided() - throws InvalidProtocolBufferException { - ListFeatureSetsResponse actual = - specService.listFeatureSets( - Filter.newBuilder() - .setProject("project1") - .setFeatureSetName("f1") - .setFeatureSetVersion("*") - .build()); - List expectedFeatureSets = - featureSets.stream().filter(fs -> fs.getName().equals("f1")).collect(Collectors.toList()); - List list = new ArrayList<>(); - for (FeatureSet expectedFeatureSet : expectedFeatureSets) { - FeatureSetProto.FeatureSet toProto = expectedFeatureSet.toProto(); - list.add(toProto); - } - ListFeatureSetsResponse expected = - ListFeatureSetsResponse.newBuilder().addAllFeatureSets(list).build(); - assertThat(actual, equalTo(expected)); + "Invalid listFeatureSetRequest, missing arguments. Must provide project and feature set name."); + specService.listFeatureSets(Filter.newBuilder().setFeatureSetName("f1").build()); } @Test @@ -221,11 +174,7 @@ public void shouldGetAllFeatureSetsMatchingNameWithWildcardSearch() throws InvalidProtocolBufferException { ListFeatureSetsResponse actual = specService.listFeatureSets( - Filter.newBuilder() - .setProject("project1") - .setFeatureSetName("f*") - .setFeatureSetVersion("*") - .build()); + Filter.newBuilder().setProject("project1").setFeatureSetName("f*").build()); List expectedFeatureSets = featureSets.stream() .filter(fs -> fs.getName().startsWith("f")) @@ -241,20 +190,12 @@ public void shouldGetAllFeatureSetsMatchingNameWithWildcardSearch() } @Test - public void shouldGetAllFeatureSetsMatchingVersionIfNoComparator() - throws InvalidProtocolBufferException { + public void shouldGetFeatureSetsByNameAndProject() throws InvalidProtocolBufferException { ListFeatureSetsResponse actual = specService.listFeatureSets( - Filter.newBuilder() - .setProject("project1") - .setFeatureSetName("f1") - .setFeatureSetVersion("1") - .build()); + Filter.newBuilder().setProject("project1").setFeatureSetName("f1").build()); List expectedFeatureSets = - featureSets.stream() - .filter(fs -> fs.getName().equals("f1")) - .filter(fs -> fs.getVersion() == 1) - .collect(Collectors.toList()); + featureSets.stream().filter(fs -> fs.getName().equals("f1")).collect(Collectors.toList()); List list = new ArrayList<>(); for (FeatureSet expectedFeatureSet : expectedFeatureSets) { FeatureSetProto.FeatureSet toProto = expectedFeatureSet.toProto(); @@ -265,80 +206,20 @@ public void shouldGetAllFeatureSetsMatchingVersionIfNoComparator() assertThat(actual, equalTo(expected)); } - @Test - public void shouldThrowExceptionIfGetAllFeatureSetsGivenVersionWithComparator() - throws InvalidProtocolBufferException { - expectedException.expect(IllegalArgumentException.class); - specService.listFeatureSets( - Filter.newBuilder() - .setProject("project1") - .setFeatureSetName("f1") - .setFeatureSetVersion(">1") - .build()); - } - - @Test - public void shouldGetLatestFeatureSetGivenMissingVersionFilter() - throws InvalidProtocolBufferException { - GetFeatureSetResponse actual = - specService.getFeatureSet( - GetFeatureSetRequest.newBuilder().setName("f1").setProject("project1").build()); - FeatureSet expected = featureSets.get(2); - assertThat(actual.getFeatureSet(), equalTo(expected.toProto())); - } - - @Test - public void shouldGetSpecificFeatureSetGivenSpecificVersionFilter() - throws InvalidProtocolBufferException { - when(featureSetRepository.findFeatureSetByNameAndProject_NameAndVersion("f1", "project1", 2)) - .thenReturn(featureSets.get(1)); - GetFeatureSetResponse actual = - specService.getFeatureSet( - GetFeatureSetRequest.newBuilder() - .setProject("project1") - .setName("f1") - .setVersion(2) - .build()); - FeatureSet expected = featureSets.get(1); - assertThat(actual.getFeatureSet(), equalTo(expected.toProto())); - } - @Test public void shouldThrowExceptionGivenMissingFeatureSetName() throws InvalidProtocolBufferException { expectedException.expect(IllegalArgumentException.class); expectedException.expectMessage("No feature set name provided"); - specService.getFeatureSet(GetFeatureSetRequest.newBuilder().setVersion(2).build()); + specService.getFeatureSet(GetFeatureSetRequest.newBuilder().build()); } @Test public void shouldThrowExceptionGivenMissingFeatureSet() throws InvalidProtocolBufferException { expectedException.expect(RetrievalException.class); - expectedException.expectMessage( - "Feature set with name \"f1000\" and version \"2\" could not be found."); + expectedException.expectMessage("Feature set with name \"f1000\" could not be found."); specService.getFeatureSet( - GetFeatureSetRequest.newBuilder() - .setName("f1000") - .setProject("project1") - .setVersion(2) - .build()); - } - - @Test - public void shouldThrowRetrievalExceptionGivenInvalidFeatureSetVersionComparator() - throws InvalidProtocolBufferException { - expectedException.expect(IllegalArgumentException.class); - expectedException.expectMessage( - "Invalid listFeatureSetRequest. Version must be set to \"*\" if the project name and feature set name aren't set explicitly: \n" - + "feature_set_name: \"f1\"\n" - + "feature_set_version: \">1\"\n" - + "project: \"project1\""); - specService.listFeatureSets( - Filter.newBuilder() - .setProject("project1") - .setFeatureSetName("f1") - .setFeatureSetVersion(">1") - .build()); + GetFeatureSetRequest.newBuilder().setName("f1000").setProject("project1").build()); } @Test @@ -373,10 +254,10 @@ public void shouldThrowRetrievalExceptionIfNoStoresFoundWithName() { } @Test - public void applyFeatureSetShouldReturnFeatureSetWithLatestVersionIfFeatureSetHasNotChanged() + public void applyFeatureSetShouldReturnFeatureSetIfFeatureSetHasNotChanged() throws InvalidProtocolBufferException { FeatureSetSpec incomingFeatureSetSpec = - featureSets.get(2).toProto().getSpec().toBuilder().clearVersion().build(); + featureSets.get(0).toProto().getSpec().toBuilder().build(); ApplyFeatureSetResponse applyFeatureSetResponse = specService.applyFeatureSet( @@ -384,21 +265,19 @@ public void applyFeatureSetShouldReturnFeatureSetWithLatestVersionIfFeatureSetHa verify(featureSetRepository, times(0)).save(ArgumentMatchers.any(FeatureSet.class)); assertThat(applyFeatureSetResponse.getStatus(), equalTo(Status.NO_CHANGE)); - assertThat(applyFeatureSetResponse.getFeatureSet(), equalTo(featureSets.get(2).toProto())); + assertThat(applyFeatureSetResponse.getFeatureSet(), equalTo(featureSets.get(0).toProto())); } @Test - public void applyFeatureSetShouldApplyFeatureSetWithInitVersionIfNotExists() + public void applyFeatureSetShouldApplyFeatureSetIfNotExists() throws InvalidProtocolBufferException { - when(featureSetRepository.findAllByNameLikeAndProject_NameOrderByNameAscVersionAsc( - "f2", "project1")) - .thenReturn(Lists.newArrayList()); + when(featureSetRepository.findFeatureSetByNameAndProject_Name("f2", "project1")) + .thenReturn(null); - FeatureSetProto.FeatureSet incomingFeatureSet = - newDummyFeatureSet("f2", 1, "project1").toProto(); + FeatureSetProto.FeatureSet incomingFeatureSet = newDummyFeatureSet("f2", "project1").toProto(); FeatureSetProto.FeatureSetSpec incomingFeatureSetSpec = - incomingFeatureSet.getSpec().toBuilder().clearVersion().build(); + incomingFeatureSet.getSpec().toBuilder().build(); ApplyFeatureSetResponse applyFeatureSetResponse = specService.applyFeatureSet( @@ -407,24 +286,16 @@ public void applyFeatureSetShouldApplyFeatureSetWithInitVersionIfNotExists() FeatureSetProto.FeatureSet expected = FeatureSetProto.FeatureSet.newBuilder() - .setSpec( - incomingFeatureSetSpec - .toBuilder() - .setVersion(1) - .setSource(defaultSource.toProto()) - .build()) + .setSpec(incomingFeatureSetSpec.toBuilder().setSource(defaultSource.toProto()).build()) .build(); assertThat(applyFeatureSetResponse.getStatus(), equalTo(Status.CREATED)); assertThat(applyFeatureSetResponse.getFeatureSet().getSpec(), equalTo(expected.getSpec())); - assertThat( - applyFeatureSetResponse.getFeatureSet().getSpec().getVersion(), - equalTo(expected.getSpec().getVersion())); } @Test - public void applyFeatureSetShouldIncrementFeatureSetVersionIfAlreadyExists() + public void applyFeatureSetShouldUpdateAndSaveFeatureSetIfAlreadyExists() throws InvalidProtocolBufferException { - FeatureSetProto.FeatureSet incomingFeatureSet = featureSets.get(2).toProto(); + FeatureSetProto.FeatureSet incomingFeatureSet = featureSets.get(0).toProto(); incomingFeatureSet = incomingFeatureSet .toBuilder() @@ -433,7 +304,6 @@ public void applyFeatureSetShouldIncrementFeatureSetVersionIfAlreadyExists() incomingFeatureSet .getSpec() .toBuilder() - .clearVersion() .addFeatures( FeatureSpec.newBuilder().setName("feature2").setValueType(Enum.STRING)) .build()) @@ -444,37 +314,27 @@ public void applyFeatureSetShouldIncrementFeatureSetVersionIfAlreadyExists() .toBuilder() .setMeta(incomingFeatureSet.getMeta().toBuilder().build()) .setSpec( - incomingFeatureSet - .getSpec() - .toBuilder() - .setVersion(4) - .setSource(defaultSource.toProto()) - .build()) + incomingFeatureSet.getSpec().toBuilder().setSource(defaultSource.toProto()).build()) .build(); ApplyFeatureSetResponse applyFeatureSetResponse = specService.applyFeatureSet(incomingFeatureSet); verify(projectRepository).saveAndFlush(ArgumentMatchers.any(Project.class)); - assertThat(applyFeatureSetResponse.getStatus(), equalTo(Status.CREATED)); + assertThat(applyFeatureSetResponse.getStatus(), equalTo(Status.UPDATED)); assertEquals( FeatureSet.fromProto(applyFeatureSetResponse.getFeatureSet()), FeatureSet.fromProto(expected)); - assertThat( - applyFeatureSetResponse.getFeatureSet().getSpec().getVersion(), - equalTo(expected.getSpec().getVersion())); } @Test public void applyFeatureSetShouldNotCreateFeatureSetIfFieldsUnordered() throws InvalidProtocolBufferException { - Feature f3f1 = TestObjectFactory.CreateFeature("f3f1", Enum.INT64); - Feature f3f2 = TestObjectFactory.CreateFeature("f3f2", Enum.INT64); - Entity f3e1 = TestObjectFactory.CreateEntity("f3e1", Enum.STRING); - FeatureSetProto.FeatureSet incomingFeatureSet = - (TestObjectFactory.CreateFeatureSet( - "f3", "project1", 5, Arrays.asList(f3e1), Arrays.asList(f3f2, f3f1))) - .toProto(); + FeatureSet featureSet = featureSets.get(1); + List features = Lists.newArrayList(featureSet.getFeatures()); + Collections.shuffle(features); + featureSet.setFeatures(Set.copyOf(features)); + FeatureSetProto.FeatureSet incomingFeatureSet = featureSet.toProto(); ApplyFeatureSetResponse applyFeatureSetResponse = specService.applyFeatureSet(incomingFeatureSet); @@ -573,12 +433,6 @@ public void applyFeatureSetShouldAcceptPresenceShapeAndDomainConstraints() @Test public void applyFeatureSetShouldUpdateFeatureSetWhenConstraintsAreUpdated() throws InvalidProtocolBufferException { - FeatureSetProto.FeatureSet existingFeatureSet = featureSets.get(2).toProto(); - assertThat( - "Existing feature set has version 3", existingFeatureSet.getSpec().getVersion() == 3); - assertThat( - "Existing feature set has at least 1 feature", - existingFeatureSet.getSpec().getFeaturesList().size() > 0); // Map of constraint field name -> value, e.g. "shape" -> FixedShape object. // If any of these fields are updated, SpecService should update the FeatureSet. @@ -603,6 +457,10 @@ public void applyFeatureSetShouldUpdateFeatureSetWhenConstraintsAreUpdated() contraintUpdates.put("time_of_day_domain", TimeOfDayDomain.getDefaultInstance()); for (Entry constraint : contraintUpdates.entrySet()) { + FeatureSet featureSet = newDummyFeatureSet("constraints", "project1"); + FeatureSetProto.FeatureSet existingFeatureSet = featureSet.toProto(); + when(featureSetRepository.findFeatureSetByNameAndProject_Name("constraints", "project1")) + .thenReturn(featureSet); String name = constraint.getKey(); Object value = constraint.getValue(); FeatureSpec newFeatureSpec = @@ -621,12 +479,8 @@ public void applyFeatureSetShouldUpdateFeatureSetWhenConstraintsAreUpdated() assertEquals( "Response should have CREATED status when field '" + name + "' is updated", - Status.CREATED, + Status.UPDATED, response.getStatus()); - assertEquals( - "FeatureSet should have new version when field '" + name + "' is updated", - existingFeatureSet.getSpec().getVersion() + 1, - response.getFeatureSet().getSpec().getVersion()); assertEquals( "Feature should have field '" + name + "' set correctly", constraint.getValue(), @@ -645,8 +499,8 @@ public void applyFeatureSetShouldCreateProjectWhenNotAlreadyExists() Feature f3f2 = TestObjectFactory.CreateFeature("f3f2", Enum.INT64); Entity f3e1 = TestObjectFactory.CreateEntity("f3e1", Enum.STRING); FeatureSetProto.FeatureSet incomingFeatureSet = - (TestObjectFactory.CreateFeatureSet( - "f3", "newproject", 5, Arrays.asList(f3e1), Arrays.asList(f3f2, f3f1))) + TestObjectFactory.CreateFeatureSet( + "f3", "project", Arrays.asList(f3e1), Arrays.asList(f3f2, f3f1)) .toProto(); ApplyFeatureSetResponse applyFeatureSetResponse = @@ -664,8 +518,8 @@ public void applyFeatureSetShouldFailWhenProjectIsArchived() Feature f3f2 = TestObjectFactory.CreateFeature("f3f2", Enum.INT64); Entity f3e1 = TestObjectFactory.CreateEntity("f3e1", Enum.STRING); FeatureSetProto.FeatureSet incomingFeatureSet = - (TestObjectFactory.CreateFeatureSet( - "f3", "archivedproject", 5, Arrays.asList(f3e1), Arrays.asList(f3f2, f3f1))) + TestObjectFactory.CreateFeatureSet( + "f3", "archivedproject", Arrays.asList(f3e1), Arrays.asList(f3f2, f3f1)) .toProto(); expectedException.expect(IllegalArgumentException.class); @@ -776,8 +630,7 @@ public void shouldUpdateStoreIfConfigChanges() throws InvalidProtocolBufferExcep .setName("SERVING") .setType(StoreType.REDIS) .setRedisConfig(RedisConfig.newBuilder()) - .addSubscriptions( - Subscription.newBuilder().setProject("project1").setName("a").setVersion(">1")) + .addSubscriptions(Subscription.newBuilder().setProject("project1").setName("a")) .build(); UpdateStoreResponse actual = specService.updateStore(UpdateStoreRequest.newBuilder().setStore(newStore).build()); @@ -814,7 +667,7 @@ public void shouldFailIfGetFeatureSetWithoutProject() throws InvalidProtocolBuff specService.getFeatureSet(GetFeatureSetRequest.newBuilder().setName("f1").build()); } - private FeatureSet newDummyFeatureSet(String name, int version, String project) { + private FeatureSet newDummyFeatureSet(String name, String project) { FeatureSpec f1 = FeatureSpec.newBuilder() .setName("feature") @@ -826,7 +679,7 @@ private FeatureSet newDummyFeatureSet(String name, int version, String project) FeatureSet fs = TestObjectFactory.CreateFeatureSet( - name, project, version, Arrays.asList(entity), Arrays.asList(feature)); + name, project, Arrays.asList(entity), Arrays.asList(feature)); fs.setCreated(Date.from(Instant.ofEpochSecond(10L))); return fs; } @@ -836,7 +689,7 @@ private Store newDummyStore(String name) { Store store = new Store(); store.setName(name); store.setType(StoreType.REDIS.toString()); - store.setSubscriptions("*:*:*"); + store.setSubscriptions("*:*"); store.setConfig(RedisConfig.newBuilder().setPort(6379).build().toByteArray()); return store; } diff --git a/core/src/test/java/feast/core/service/TestObjectFactory.java b/core/src/test/java/feast/core/service/TestObjectFactory.java index 0476dbe5c2e..2723db1d2ed 100644 --- a/core/src/test/java/feast/core/service/TestObjectFactory.java +++ b/core/src/test/java/feast/core/service/TestObjectFactory.java @@ -38,11 +38,10 @@ public class TestObjectFactory { true); public static FeatureSet CreateFeatureSet( - String name, String project, int version, List entities, List features) { + String name, String project, List entities, List features) { return new FeatureSet( name, project, - version, 100L, entities, features, diff --git a/examples/basic/basic.ipynb b/examples/basic/basic.ipynb index 921577fb085..b9e0ba9e1a4 100644 --- a/examples/basic/basic.ipynb +++ b/examples/basic/basic.ipynb @@ -250,14 +250,7 @@ "metadata": {}, "outputs": [], "source": [ - "client.apply(customer_fs)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We test the retrieval of this feature set object (not its data), to ensure that we have the latest version" + "client.apply(customer_fs)\n" ] }, { @@ -465,4 +458,4 @@ }, "nbformat": 4, "nbformat_minor": 2 -} \ No newline at end of file +} diff --git a/go.mod b/go.mod index 15160e57786..ec7cd20f804 100644 --- a/go.mod +++ b/go.mod @@ -23,7 +23,7 @@ require ( golang.org/x/lint v0.0.0-20200302205851-738671d3881b // indirect golang.org/x/net v0.0.0-20200320220750-118fecf932d8 golang.org/x/sys v0.0.0-20200321134203-328b4cd54aae // indirect - golang.org/x/tools v0.0.0-20200414032229-332987a829c3 // indirect + golang.org/x/tools v0.0.0-20200504022951-6b6965ac5dd1 // indirect google.golang.org/genproto v0.0.0-20200319113533-08878b785e9c // indirect google.golang.org/grpc v1.28.0 gopkg.in/russross/blackfriday.v2 v2.0.0 // indirect diff --git a/go.sum b/go.sum index 1b53b39cf40..49996e36f4f 100644 --- a/go.sum +++ b/go.sum @@ -446,6 +446,8 @@ golang.org/x/tools v0.0.0-20200321224714-0d839f3cf2ed h1:OCZDlBlLYiUK6T33/8+3Bno golang.org/x/tools v0.0.0-20200321224714-0d839f3cf2ed/go.mod h1:Sl4aGygMT6LrqrWclx+PTx3U+LnKx/seiNR+3G19Ar8= golang.org/x/tools v0.0.0-20200414032229-332987a829c3 h1:Z68UA+HA9shnGhQbAFXKqL1Rk/tfiTHJ57bNm/MUL/A= golang.org/x/tools v0.0.0-20200414032229-332987a829c3/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= +golang.org/x/tools v0.0.0-20200504022951-6b6965ac5dd1 h1:C8rdnd6KieI73Z2Av0sS0t4kW+geIH/M8kNX8Hmvn9E= +golang.org/x/tools v0.0.0-20200504022951-6b6965ac5dd1/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE= golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= diff --git a/ingestion/src/main/java/feast/ingestion/ImportJob.java b/ingestion/src/main/java/feast/ingestion/ImportJob.java index ef6039e5536..5afa66b5e14 100644 --- a/ingestion/src/main/java/feast/ingestion/ImportJob.java +++ b/ingestion/src/main/java/feast/ingestion/ImportJob.java @@ -27,8 +27,8 @@ import feast.ingestion.options.BZip2Decompressor; import feast.ingestion.options.ImportOptions; import feast.ingestion.options.StringListStreamConverter; +import feast.ingestion.transform.ProcessAndValidateFeatureRows; import feast.ingestion.transform.ReadFromSource; -import feast.ingestion.transform.ValidateFeatureRows; import feast.ingestion.transform.metrics.WriteFailureMetricsTransform; import feast.ingestion.transform.metrics.WriteSuccessMetricsTransform; import feast.ingestion.utils.SpecUtil; @@ -124,12 +124,12 @@ public static PipelineResult runPipeline(ImportOptions options) throws IOExcepti .setFailureTag(DEADLETTER_OUT) .build()); - // Step 2. Validate incoming FeatureRows + // Step 2. Process and validate incoming FeatureRows PCollectionTuple validatedRows = convertedFeatureRows .get(FEATURE_ROW_OUT) .apply( - ValidateFeatureRows.newBuilder() + ProcessAndValidateFeatureRows.newBuilder() .setFeatureSetSpecs(featureSetSpecsByKey) .setSuccessTag(FEATURE_ROW_OUT) .setFailureTag(DEADLETTER_OUT) diff --git a/ingestion/src/main/java/feast/ingestion/transform/ValidateFeatureRows.java b/ingestion/src/main/java/feast/ingestion/transform/ProcessAndValidateFeatureRows.java similarity index 75% rename from ingestion/src/main/java/feast/ingestion/transform/ValidateFeatureRows.java rename to ingestion/src/main/java/feast/ingestion/transform/ProcessAndValidateFeatureRows.java index 06df06c074c..53d56c667d6 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/ValidateFeatureRows.java +++ b/ingestion/src/main/java/feast/ingestion/transform/ProcessAndValidateFeatureRows.java @@ -18,6 +18,7 @@ import com.google.auto.value.AutoValue; import feast.core.FeatureSetProto; +import feast.ingestion.transform.fn.ProcessFeatureRowDoFn; import feast.ingestion.transform.fn.ValidateFeatureRowDoFn; import feast.ingestion.values.FeatureSet; import feast.storage.api.writer.FailedElement; @@ -33,7 +34,7 @@ import org.apache.commons.lang3.tuple.Pair; @AutoValue -public abstract class ValidateFeatureRows +public abstract class ProcessAndValidateFeatureRows extends PTransform, PCollectionTuple> { public abstract Map getFeatureSetSpecs(); @@ -43,7 +44,7 @@ public abstract class ValidateFeatureRows public abstract TupleTag getFailureTag(); public static Builder newBuilder() { - return new AutoValue_ValidateFeatureRows.Builder(); + return new AutoValue_ProcessAndValidateFeatureRows.Builder(); } @AutoValue.Builder @@ -56,7 +57,7 @@ public abstract Builder setFeatureSetSpecs( public abstract Builder setFailureTag(TupleTag failureTag); - public abstract ValidateFeatureRows build(); + public abstract ProcessAndValidateFeatureRows build(); } @Override @@ -67,14 +68,16 @@ public PCollectionTuple expand(PCollection input) { .map(e -> Pair.of(e.getKey(), new FeatureSet(e.getValue()))) .collect(Collectors.toMap(Pair::getLeft, Pair::getRight)); - return input.apply( - "ValidateFeatureRows", - ParDo.of( - ValidateFeatureRowDoFn.newBuilder() - .setFeatureSets(featureSets) - .setSuccessTag(getSuccessTag()) - .setFailureTag(getFailureTag()) - .build()) - .withOutputTags(getSuccessTag(), TupleTagList.of(getFailureTag()))); + return input + .apply("ProcessFeatureRows", ParDo.of(new ProcessFeatureRowDoFn())) + .apply( + "ValidateFeatureRows", + ParDo.of( + ValidateFeatureRowDoFn.newBuilder() + .setFeatureSets(featureSets) + .setSuccessTag(getSuccessTag()) + .setFailureTag(getFailureTag()) + .build()) + .withOutputTags(getSuccessTag(), TupleTagList.of(getFailureTag()))); } } diff --git a/ingestion/src/main/java/feast/ingestion/transform/fn/ProcessFeatureRowDoFn.java b/ingestion/src/main/java/feast/ingestion/transform/fn/ProcessFeatureRowDoFn.java new file mode 100644 index 00000000000..70e173d5db1 --- /dev/null +++ b/ingestion/src/main/java/feast/ingestion/transform/fn/ProcessFeatureRowDoFn.java @@ -0,0 +1,37 @@ +/* + * SPDX-License-Identifier: Apache-2.0 + * Copyright 2018-2020 The Feast Authors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * https://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package feast.ingestion.transform.fn; + +import feast.types.FeatureRowProto.FeatureRow; +import org.apache.beam.sdk.transforms.DoFn; + +public class ProcessFeatureRowDoFn extends DoFn { + + @ProcessElement + public void processElement(ProcessContext context) { + FeatureRow featureRow = context.element(); + featureRow = + featureRow.toBuilder().setFeatureSet(stripVersion(featureRow.getFeatureSet())).build(); + context.output(featureRow); + } + + // For backward compatibility. Will be deprecated eventually. + private String stripVersion(String featureSetId) { + String[] split = featureSetId.split(":"); + return split[0]; + } +} diff --git a/ingestion/src/main/java/feast/ingestion/transform/fn/ValidateFeatureRowDoFn.java b/ingestion/src/main/java/feast/ingestion/transform/fn/ValidateFeatureRowDoFn.java index 85ac3c86faa..d3e0475abf3 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/fn/ValidateFeatureRowDoFn.java +++ b/ingestion/src/main/java/feast/ingestion/transform/fn/ValidateFeatureRowDoFn.java @@ -58,14 +58,14 @@ public abstract static class Builder { public void processElement(ProcessContext context) { String error = null; FeatureRow featureRow = context.element(); - FeatureSet featureSet = getFeatureSets().getOrDefault(featureRow.getFeatureSet(), null); + FeatureSet featureSet = getFeatureSets().get(featureRow.getFeatureSet()); List fields = new ArrayList<>(); if (featureSet != null) { for (FieldProto.Field field : featureRow.getFieldsList()) { Field fieldSpec = featureSet.getField(field.getName()); if (fieldSpec == null) { // skip - break; + continue; } // If value is set in the FeatureRow, make sure the value type matches // that defined in FeatureSetSpec @@ -99,13 +99,8 @@ public void processElement(ProcessContext context) { .setPayload(featureRow.toString()) .setErrorMessage(error); if (featureSet != null) { - String[] split = featureSet.getReference().split(":"); - String[] nameSplit = split[0].split("/"); - failedElement = - failedElement - .setProjectName(nameSplit[0]) - .setFeatureSetName(nameSplit[1]) - .setFeatureSetVersion(split[1]); + String[] split = featureSet.getReference().split("/"); + failedElement = failedElement.setProjectName(split[0]).setFeatureSetName(split[1]); } context.output(getFailureTag(), failedElement.build()); } else { diff --git a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteDeadletterRowMetricsDoFn.java b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteDeadletterRowMetricsDoFn.java index b4338cda09b..828fed6ceda 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteDeadletterRowMetricsDoFn.java +++ b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteDeadletterRowMetricsDoFn.java @@ -76,7 +76,6 @@ public void processElement(ProcessContext c) { STORE_TAG_KEY + ":" + getStoreName(), PROJECT_TAG_KEY + ":" + ignored.getProjectName(), FEATURE_SET_NAME_TAG_KEY + ":" + ignored.getFeatureSetName(), - FEATURE_SET_VERSION_TAG_KEY + ":" + ignored.getFeatureSetVersion(), INGESTION_JOB_NAME_KEY + ":" + c.getPipelineOptions().getJobName()); } catch (StatsDClientException e) { log.warn("Unable to push metrics to server", e); diff --git a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java index cfecb858dcf..aa8c7a7e89a 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java +++ b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteFeatureValueMetricsDoFn.java @@ -18,7 +18,6 @@ import static feast.ingestion.transform.metrics.WriteRowMetricsDoFn.FEATURE_SET_NAME_TAG_KEY; import static feast.ingestion.transform.metrics.WriteRowMetricsDoFn.FEATURE_SET_PROJECT_TAG_KEY; -import static feast.ingestion.transform.metrics.WriteRowMetricsDoFn.FEATURE_SET_VERSION_TAG_KEY; import static feast.ingestion.transform.metrics.WriteRowMetricsDoFn.FEATURE_TAG_KEY; import static feast.ingestion.transform.metrics.WriteRowMetricsDoFn.INGESTION_JOB_NAME_KEY; import static feast.ingestion.transform.metrics.WriteRowMetricsDoFn.METRIC_PREFIX; @@ -131,25 +130,17 @@ public void processElement( "Feature set reference in the feature row is null. Please check the input feature rows from previous steps"); return; } - String[] colonSplits = featureSetRef.split(":"); - if (colonSplits.length != 2) { - log.error( - "Skip writing feature value metrics because the feature set reference '{}' does not" - + "follow the required format /:", - featureSetRef); - return; - } - String[] slashSplits = colonSplits[0].split("/"); + + String[] slashSplits = featureSetRef.split("/"); if (slashSplits.length != 2) { log.error( "Skip writing feature value metrics because the feature set reference '{}' does not" - + "follow the required format /:", + + "follow the required format /", featureSetRef); return; } String projectName = slashSplits[0]; String featureSetName = slashSplits[1]; - String version = colonSplits[1]; Map featureNameToStats = new HashMap<>(); Map> featureNameToValues = new HashMap<>(); @@ -166,7 +157,6 @@ public void processElement( STORE_TAG_KEY + ":" + getStoreName(), FEATURE_SET_PROJECT_TAG_KEY + ":" + projectName, FEATURE_SET_NAME_TAG_KEY + ":" + featureSetName, - FEATURE_SET_VERSION_TAG_KEY + ":" + version, FEATURE_TAG_KEY + ":" + featureName, INGESTION_JOB_NAME_KEY + ":" + context.getPipelineOptions().getJobName() }; diff --git a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteRowMetricsDoFn.java b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteRowMetricsDoFn.java index 2fe1f2e7f01..d8cc7635730 100644 --- a/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteRowMetricsDoFn.java +++ b/ingestion/src/main/java/feast/ingestion/transform/metrics/WriteRowMetricsDoFn.java @@ -142,15 +142,7 @@ public void processElement( "Feature set reference in the feature row is null. Please check the input feature rows from previous steps"); return; } - String[] colonSplits = featureSetRef.split(":"); - if (colonSplits.length != 2) { - log.error( - "Skip writing feature row metrics because the feature set reference '{}' does not" - + "follow the required format /:", - featureSetRef); - return; - } - String[] slashSplits = colonSplits[0].split("/"); + String[] slashSplits = featureSetRef.split("/"); if (slashSplits.length != 2) { log.error( "Skip writing feature row metrics because the feature set reference '{}' does not" @@ -161,7 +153,6 @@ public void processElement( String featureSetProject = slashSplits[0]; String featureSetName = slashSplits[1]; - String featureSetVersion = colonSplits[1]; // featureRowLagStats is stats for feature row lag for feature set "featureSetName" DescriptiveStatistics featureRowLagStats = new DescriptiveStatistics(); @@ -201,7 +192,6 @@ public void processElement( STORE_TAG_KEY + ":" + getStoreName(), FEATURE_SET_PROJECT_TAG_KEY + ":" + featureSetProject, FEATURE_SET_NAME_TAG_KEY + ":" + featureSetName, - FEATURE_SET_VERSION_TAG_KEY + ":" + featureSetVersion, INGESTION_JOB_NAME_KEY + ":" + c.getPipelineOptions().getJobName(), }; diff --git a/ingestion/src/main/java/feast/ingestion/utils/SpecUtil.java b/ingestion/src/main/java/feast/ingestion/utils/SpecUtil.java index f28dfc9ee39..9aac352fe45 100644 --- a/ingestion/src/main/java/feast/ingestion/utils/SpecUtil.java +++ b/ingestion/src/main/java/feast/ingestion/utils/SpecUtil.java @@ -34,9 +34,7 @@ public class SpecUtil { public static String getFeatureSetReference(FeatureSetSpec featureSetSpec) { - return String.format( - "%s/%s:%d", - featureSetSpec.getProject(), featureSetSpec.getName(), featureSetSpec.getVersion()); + return String.format("%s/%s", featureSetSpec.getProject(), featureSetSpec.getName()); } /** Get only feature set specs that matches the subscription */ @@ -46,28 +44,17 @@ public static List getSubscribedFeatureSets( for (FeatureSet featureSet : featureSets) { for (Subscription sub : subscriptions) { // If configuration missing, fail - if (sub.getProject().isEmpty() || sub.getName().isEmpty() || sub.getVersion().isEmpty()) { + if (sub.getProject().isEmpty() || sub.getName().isEmpty()) { throw new IllegalArgumentException( String.format("Subscription is missing arguments: %s", sub.toString())); } // If all wildcards, subscribe to everything - if (sub.getProject().equals("*") - || sub.getName().equals("*") - || sub.getVersion().equals("*")) { + if (sub.getProject().equals("*") || sub.getName().equals("*")) { subscribed.add(featureSet); break; } - // If all wildcards, subscribe to everything - if (sub.getProject().equals("*") - && (!sub.getName().equals("*") || !sub.getVersion().equals("*"))) { - throw new IllegalArgumentException( - String.format( - "Subscription cannot have feature set name and/or version set if project is not defined: %s", - sub.toString())); - } - // Match project name if (!featureSet.getSpec().getProject().equals(sub.getProject())) { continue; @@ -84,26 +71,7 @@ public static List getSubscribedFeatureSets( if (!pattern.matcher(featureSet.getSpec().getName()).matches()) { continue; } - - // If version is '*', match all - if (sub.getVersion().equals("*")) { - subscribed.add(featureSet); - break; - } else if (sub.getVersion().equals("latest")) { - // if version is "latest" - throw new RuntimeException( - String.format( - "Support for latest feature set subscription has not been implemented yet: %s", - sub.toString())); - - } else { - // If a specific version, match that version alone - int version = Integer.parseInt(sub.getVersion()); - if (featureSet.getSpec().getVersion() == version) { - subscribed.add(featureSet); - break; - } - } + subscribed.add(featureSet); } } return subscribed; diff --git a/ingestion/src/main/java/feast/ingestion/values/FailedElement.java b/ingestion/src/main/java/feast/ingestion/values/FailedElement.java deleted file mode 100644 index 9606c27d190..00000000000 --- a/ingestion/src/main/java/feast/ingestion/values/FailedElement.java +++ /dev/null @@ -1,83 +0,0 @@ -/* - * SPDX-License-Identifier: Apache-2.0 - * Copyright 2018-2019 The Feast Authors - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * https://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package feast.ingestion.values; - -import com.google.auto.value.AutoValue; -import javax.annotation.Nullable; -import org.apache.beam.sdk.schemas.AutoValueSchema; -import org.apache.beam.sdk.schemas.annotations.DefaultSchema; -import org.joda.time.Instant; - -@AutoValue -// Use DefaultSchema annotation so this AutoValue class can be serialized by Beam -// https://issues.apache.org/jira/browse/BEAM-1891 -// https://github.com/apache/beam/pull/7334 -@DefaultSchema(AutoValueSchema.class) -public abstract class FailedElement { - public abstract Instant getTimestamp(); - - @Nullable - public abstract String getJobName(); - - @Nullable - public abstract String getProjectName(); - - @Nullable - public abstract String getFeatureSetName(); - - @Nullable - public abstract String getFeatureSetVersion(); - - @Nullable - public abstract String getTransformName(); - - @Nullable - public abstract String getPayload(); - - @Nullable - public abstract String getErrorMessage(); - - @Nullable - public abstract String getStackTrace(); - - public static Builder newBuilder() { - return new AutoValue_FailedElement.Builder().setTimestamp(Instant.now()); - } - - @AutoValue.Builder - public abstract static class Builder { - public abstract Builder setTimestamp(Instant timestamp); - - public abstract Builder setProjectName(String projectName); - - public abstract Builder setFeatureSetName(String featureSetName); - - public abstract Builder setFeatureSetVersion(String featureSetVersion); - - public abstract Builder setJobName(String jobName); - - public abstract Builder setTransformName(String transformName); - - public abstract Builder setPayload(String payload); - - public abstract Builder setErrorMessage(String errorMessage); - - public abstract Builder setStackTrace(String stackTrace); - - public abstract FailedElement build(); - } -} diff --git a/ingestion/src/test/java/feast/ingestion/ImportJobTest.java b/ingestion/src/test/java/feast/ingestion/ImportJobTest.java index 13df73e96a4..cd25cdf380a 100644 --- a/ingestion/src/test/java/feast/ingestion/ImportJobTest.java +++ b/ingestion/src/test/java/feast/ingestion/ImportJobTest.java @@ -120,7 +120,6 @@ public void runPipeline_ShouldWriteToRedisCorrectlyGivenValidSpecAndFeatureRow() FeatureSetSpec spec = FeatureSetSpec.newBuilder() .setName("feature_set") - .setVersion(3) .setProject("myproject") .addEntities( EntitySpec.newBuilder() @@ -164,7 +163,6 @@ public void runPipeline_ShouldWriteToRedisCorrectlyGivenValidSpecAndFeatureRow() Subscription.newBuilder() .setProject(spec.getProject()) .setName(spec.getName()) - .setVersion(String.valueOf(spec.getVersion())) .build()) .build(); diff --git a/ingestion/src/test/java/feast/ingestion/transform/ValidateFeatureRowsTest.java b/ingestion/src/test/java/feast/ingestion/transform/ProcessAndValidateFeatureRowsTest.java similarity index 74% rename from ingestion/src/test/java/feast/ingestion/transform/ValidateFeatureRowsTest.java rename to ingestion/src/test/java/feast/ingestion/transform/ProcessAndValidateFeatureRowsTest.java index 3737a736168..be082988716 100644 --- a/ingestion/src/test/java/feast/ingestion/transform/ValidateFeatureRowsTest.java +++ b/ingestion/src/test/java/feast/ingestion/transform/ProcessAndValidateFeatureRowsTest.java @@ -39,7 +39,7 @@ import org.junit.Rule; import org.junit.Test; -public class ValidateFeatureRowsTest { +public class ProcessAndValidateFeatureRowsTest { @Rule public transient TestPipeline p = TestPipeline.create(); @@ -52,7 +52,6 @@ public void shouldWriteSuccessAndFailureTagsCorrectly() { FeatureSetSpec fs1 = FeatureSetSpec.newBuilder() .setName("feature_set") - .setVersion(1) .setProject("myproject") .addEntities( EntitySpec.newBuilder() @@ -72,8 +71,7 @@ public void shouldWriteSuccessAndFailureTagsCorrectly() { FeatureSetSpec fs2 = FeatureSetSpec.newBuilder() - .setName("feature_set") - .setVersion(2) + .setName("feature_set_2") .setProject("myproject") .addEntities( EntitySpec.newBuilder() @@ -92,8 +90,8 @@ public void shouldWriteSuccessAndFailureTagsCorrectly() { .build(); Map featureSetSpecs = new HashMap<>(); - featureSetSpecs.put("myproject/feature_set:1", fs1); - featureSetSpecs.put("myproject/feature_set:2", fs2); + featureSetSpecs.put("myproject/feature_set", fs1); + featureSetSpecs.put("myproject/feature_set_2", fs2); List input = new ArrayList<>(); List expected = new ArrayList<>(); @@ -110,7 +108,7 @@ public void shouldWriteSuccessAndFailureTagsCorrectly() { p.apply(Create.of(input)) .setCoder(ProtoCoder.of(FeatureRow.class)) .apply( - ValidateFeatureRows.newBuilder() + ProcessAndValidateFeatureRows.newBuilder() .setFailureTag(FAILURE_TAG) .setSuccessTag(SUCCESS_TAG) .setFeatureSetSpecs(featureSetSpecs) @@ -122,12 +120,59 @@ public void shouldWriteSuccessAndFailureTagsCorrectly() { p.run(); } + @Test + public void shouldStripVersions() { + FeatureSetSpec fs1 = + FeatureSetSpec.newBuilder() + .setName("feature_set") + .setProject("myproject") + .addEntities( + EntitySpec.newBuilder() + .setName("entity_id_primary") + .setValueType(Enum.INT32) + .build()) + .addEntities( + EntitySpec.newBuilder() + .setName("entity_id_secondary") + .setValueType(Enum.STRING) + .build()) + .addFeatures( + FeatureSpec.newBuilder().setName("feature_1").setValueType(Enum.STRING).build()) + .addFeatures( + FeatureSpec.newBuilder().setName("feature_2").setValueType(Enum.INT64).build()) + .build(); + + Map featureSetSpecs = new HashMap<>(); + featureSetSpecs.put("myproject/feature_set", fs1); + + List input = new ArrayList<>(); + List expected = new ArrayList<>(); + + FeatureRow randomRow = TestUtil.createRandomFeatureRow(fs1); + expected.add(randomRow); + randomRow = randomRow.toBuilder().setFeatureSet("myproject/feature_set:1").build(); + input.add(randomRow); + + PCollectionTuple output = + p.apply(Create.of(input)) + .setCoder(ProtoCoder.of(FeatureRow.class)) + .apply( + ProcessAndValidateFeatureRows.newBuilder() + .setFailureTag(FAILURE_TAG) + .setSuccessTag(SUCCESS_TAG) + .setFeatureSetSpecs(featureSetSpecs) + .build()); + + PAssert.that(output.get(SUCCESS_TAG)).containsInAnyOrder(expected); + + p.run(); + } + @Test public void shouldExcludeUnregisteredFields() { FeatureSetSpec fs1 = FeatureSetSpec.newBuilder() .setName("feature_set") - .setVersion(1) .setProject("myproject") .addEntities( EntitySpec.newBuilder() @@ -146,7 +191,7 @@ public void shouldExcludeUnregisteredFields() { .build(); Map featureSets = new HashMap<>(); - featureSets.put("myproject/feature_set:1", fs1); + featureSets.put("myproject/feature_set", fs1); List input = new ArrayList<>(); List expected = new ArrayList<>(); @@ -166,7 +211,7 @@ public void shouldExcludeUnregisteredFields() { p.apply(Create.of(input)) .setCoder(ProtoCoder.of(FeatureRow.class)) .apply( - ValidateFeatureRows.newBuilder() + ProcessAndValidateFeatureRows.newBuilder() .setFailureTag(FAILURE_TAG) .setSuccessTag(SUCCESS_TAG) .setFeatureSetSpecs(featureSets) diff --git a/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.input b/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.input index d2985711cee..42731b9fe1c 100644 --- a/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.input +++ b/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.input @@ -1,4 +1,4 @@ featuresetref,int32,int64,double,float,bool,int32list,int64list,doublelist,floatlist,boollist,bytes,byteslist,string,stringlist -project/featureset:1,1,5,8,5,true,1|4|3,5|1|12,5|7|3,-2.0,true|false,,,, -project/featureset:1,5,-10,8,10.0,true,1|12|5,,,-1.0|-3.0,false|true,,,, -project/featureset:1,6,-4,8,0.0,true,2,2|5,,,true|false,,,, \ No newline at end of file +project/featureset,1,5,8,5,true,1|4|3,5|1|12,5|7|3,-2.0,true|false,,,, +project/featureset,5,-10,8,10.0,true,1|12|5,,,-1.0|-3.0,false|true,,,, +project/featureset,6,-4,8,0.0,true,2,2|5,,,true|false,,,, \ No newline at end of file diff --git a/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.output b/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.output index 63bc7bbfa4e..12ed4b7e1f2 100644 --- a/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.output +++ b/ingestion/src/test/resources/feast/ingestion/transform/WriteFeatureValueMetricsDoFnTest.output @@ -1,66 +1,66 @@ -feast_ingestion.feature_value_min:1|g|#ingestion_job_name:job,feast_feature_name:int32,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_max:6|g|#ingestion_job_name:job,feast_feature_name:int32,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_mean:4|g|#ingestion_job_name:job,feast_feature_name:int32,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_50:5|g|#ingestion_job_name:job,feast_feature_name:int32,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_90:6|g|#ingestion_job_name:job,feast_feature_name:int32,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_min:1|g|#ingestion_job_name:job,feast_feature_name:int32,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:6|g|#ingestion_job_name:job,feast_feature_name:int32,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:4|g|#ingestion_job_name:job,feast_feature_name:int32,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:5|g|#ingestion_job_name:job,feast_feature_name:int32,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:6|g|#ingestion_job_name:job,feast_feature_name:int32,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_min:0|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_min:-10|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_max:5|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_mean:0|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_mean:-3|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_50:-4|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_90:5|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_min:0|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_min:-10|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:5|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:0|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:-3|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:-4|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:5|g|#ingestion_job_name:job,feast_feature_name:int64,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_min:8|g|#ingestion_job_name:job,feast_feature_name:double,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_max:8|g|#ingestion_job_name:job,feast_feature_name:double,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_mean:8|g|#ingestion_job_name:job,feast_feature_name:double,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_50:8|g|#ingestion_job_name:job,feast_feature_name:double,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_90:8|g|#ingestion_job_name:job,feast_feature_name:double,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_min:8|g|#ingestion_job_name:job,feast_feature_name:double,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:8|g|#ingestion_job_name:job,feast_feature_name:double,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:8|g|#ingestion_job_name:job,feast_feature_name:double,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:8|g|#ingestion_job_name:job,feast_feature_name:double,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:8|g|#ingestion_job_name:job,feast_feature_name:double,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_min:0|g|#ingestion_job_name:job,feast_feature_name:float,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_max:10|g|#ingestion_job_name:job,feast_feature_name:float,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_mean:5|g|#ingestion_job_name:job,feast_feature_name:float,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_50:5|g|#ingestion_job_name:job,feast_feature_name:float,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_90:10|g|#ingestion_job_name:job,feast_feature_name:float,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_min:0|g|#ingestion_job_name:job,feast_feature_name:float,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:10|g|#ingestion_job_name:job,feast_feature_name:float,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:5|g|#ingestion_job_name:job,feast_feature_name:float,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:5|g|#ingestion_job_name:job,feast_feature_name:float,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:10|g|#ingestion_job_name:job,feast_feature_name:float,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_min:1|g|#ingestion_job_name:job,feast_feature_name:bool,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_max:1|g|#ingestion_job_name:job,feast_feature_name:bool,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_mean:1|g|#ingestion_job_name:job,feast_feature_name:bool,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_50:1|g|#ingestion_job_name:job,feast_feature_name:bool,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_90:1|g|#ingestion_job_name:job,feast_feature_name:bool,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_min:1|g|#ingestion_job_name:job,feast_feature_name:bool,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:1|g|#ingestion_job_name:job,feast_feature_name:bool,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:1|g|#ingestion_job_name:job,feast_feature_name:bool,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:1|g|#ingestion_job_name:job,feast_feature_name:bool,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:1|g|#ingestion_job_name:job,feast_feature_name:bool,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_min:1|g|#ingestion_job_name:job,feast_feature_name:int32list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_max:12|g|#ingestion_job_name:job,feast_feature_name:int32list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_mean:4|g|#ingestion_job_name:job,feast_feature_name:int32list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_50:3|g|#ingestion_job_name:job,feast_feature_name:int32list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_90:12|g|#ingestion_job_name:job,feast_feature_name:int32list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_min:1|g|#ingestion_job_name:job,feast_feature_name:int32list,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:12|g|#ingestion_job_name:job,feast_feature_name:int32list,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:4|g|#ingestion_job_name:job,feast_feature_name:int32list,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:3|g|#ingestion_job_name:job,feast_feature_name:int32list,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:12|g|#ingestion_job_name:job,feast_feature_name:int32list,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_min:1|g|#ingestion_job_name:job,feast_feature_name:int64list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_max:12|g|#ingestion_job_name:job,feast_feature_name:int64list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_mean:5|g|#ingestion_job_name:job,feast_feature_name:int64list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_50:5|g|#ingestion_job_name:job,feast_feature_name:int64list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_90:12|g|#ingestion_job_name:job,feast_feature_name:int64list,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_min:1|g|#ingestion_job_name:job,feast_feature_name:int64list,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:12|g|#ingestion_job_name:job,feast_feature_name:int64list,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:5|g|#ingestion_job_name:job,feast_feature_name:int64list,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:5|g|#ingestion_job_name:job,feast_feature_name:int64list,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:12|g|#ingestion_job_name:job,feast_feature_name:int64list,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_min:3|g|#ingestion_job_name:job,feast_feature_name:doublelist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_max:7|g|#ingestion_job_name:job,feast_feature_name:doublelist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_mean:5|g|#ingestion_job_name:job,feast_feature_name:doublelist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_50:5|g|#ingestion_job_name:job,feast_feature_name:doublelist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_90:7|g|#ingestion_job_name:job,feast_feature_name:doublelist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_min:3|g|#ingestion_job_name:job,feast_feature_name:doublelist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:7|g|#ingestion_job_name:job,feast_feature_name:doublelist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:5|g|#ingestion_job_name:job,feast_feature_name:doublelist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:5|g|#ingestion_job_name:job,feast_feature_name:doublelist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:7|g|#ingestion_job_name:job,feast_feature_name:doublelist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_min:0|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_min:-3|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_max:0|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_max:-1|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_mean:0|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_mean:-2|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_50:0|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_50:-2|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_90:0|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_90:-1|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_min:0|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_min:-3|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:0|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:-1|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:0|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:-2|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:0|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:-2|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:0|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:-1|g|#ingestion_job_name:job,feast_feature_name:floatlist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_min:0|g|#ingestion_job_name:job,feast_feature_name:boollist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_max:1|g|#ingestion_job_name:job,feast_feature_name:boollist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_mean:0.5|g|#ingestion_job_name:job,feast_feature_name:boollist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_50:0.5|g|#ingestion_job_name:job,feast_feature_name:boollist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_percentile_90:1|g|#ingestion_job_name:job,feast_feature_name:boollist,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store \ No newline at end of file +feast_ingestion.feature_value_min:0|g|#ingestion_job_name:job,feast_feature_name:boollist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_max:1|g|#ingestion_job_name:job,feast_feature_name:boollist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_mean:0.5|g|#ingestion_job_name:job,feast_feature_name:boollist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_50:0.5|g|#ingestion_job_name:job,feast_feature_name:boollist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_percentile_90:1|g|#ingestion_job_name:job,feast_feature_name:boollist,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store \ No newline at end of file diff --git a/ingestion/src/test/resources/feast/ingestion/transform/WriteRowMetricsDoFnTest.input b/ingestion/src/test/resources/feast/ingestion/transform/WriteRowMetricsDoFnTest.input index 4d42f5bc4c4..c5543d2889d 100644 --- a/ingestion/src/test/resources/feast/ingestion/transform/WriteRowMetricsDoFnTest.input +++ b/ingestion/src/test/resources/feast/ingestion/transform/WriteRowMetricsDoFnTest.input @@ -1,4 +1,4 @@ featuresetref,int32,int64,timestamp -project/featureset:1,1,5,2020-03-30T06:10:38Z -project/featureset:1,5,8,2020-03-30T06:10:43Z -project/featureset:1,6,,2020-03-30T06:10:42Z \ No newline at end of file +project/featureset,1,5,2020-03-30T06:10:38Z +project/featureset,5,8,2020-03-30T06:10:43Z +project/featureset,6,,2020-03-30T06:10:42Z \ No newline at end of file diff --git a/ingestion/src/test/resources/feast/ingestion/transform/WriteRowMetricsDoFnTest.output b/ingestion/src/test/resources/feast/ingestion/transform/WriteRowMetricsDoFnTest.output index 318ce8eb08b..954215764f3 100644 --- a/ingestion/src/test/resources/feast/ingestion/transform/WriteRowMetricsDoFnTest.output +++ b/ingestion/src/test/resources/feast/ingestion/transform/WriteRowMetricsDoFnTest.output @@ -1,23 +1,23 @@ -feast_ingestion.feature_row_ingested_count:3|c|#ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_row_lag_ms_min:2000|g|#ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_row_lag_ms_max:7000|g|#ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_row_lag_ms_mean:4000|g|#ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_row_lag_ms_percentile_90:7000|g|#ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_row_lag_ms_percentile_95:7000|g|#ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_row_lag_ms_percentile_99:7000|g|#ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_row_ingested_count:3|c|#ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_row_lag_ms_min:2000|g|#ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_row_lag_ms_max:7000|g|#ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_row_lag_ms_mean:4000|g|#ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_row_lag_ms_percentile_90:7000|g|#ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_row_lag_ms_percentile_95:7000|g|#ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_row_lag_ms_percentile_99:7000|g|#ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_lag_ms_min:2000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_lag_ms_max:7000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_lag_ms_mean:4000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_lag_ms_percentile_90:7000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_lag_ms_percentile_95:7000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_lag_ms_percentile_99:7000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_missing_count:0|c|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_min:2000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_max:7000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_mean:4000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_percentile_90:7000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_percentile_95:7000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_percentile_99:7000|g|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_missing_count:0|c|#feast_feature_name:int32,ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_lag_ms_min:2000|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_lag_ms_max:7000|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_lag_ms_mean:4500|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_lag_ms_percentile_90:7000|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_lag_ms_percentile_95:7000|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_lag_ms_percentile_99:7000|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store -feast_ingestion.feature_value_missing_count:1|c|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_version:1,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store \ No newline at end of file +feast_ingestion.feature_value_lag_ms_min:2000|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_max:7000|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_mean:4500|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_percentile_90:7000|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_percentile_95:7000|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_lag_ms_percentile_99:7000|g|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store +feast_ingestion.feature_value_missing_count:1|c|#feast_feature_name:int64,ingestion_job_name:job,feast_featureSet_name:featureset,feast_project_name:project,feast_store:store \ No newline at end of file diff --git a/protos/feast/core/CoreService.proto b/protos/feast/core/CoreService.proto index b7760d0b9aa..9cdcbf5c657 100644 --- a/protos/feast/core/CoreService.proto +++ b/protos/feast/core/CoreService.proto @@ -52,8 +52,11 @@ service CoreService { // Create or update and existing feature set. // // This function is idempotent - it will not create a new feature set if schema does not change. - // If an existing feature set is updated, core will advance the version number, which will be - // returned in response. + // Schema changes will update the feature set if the changes are valid. + // All changes except the following are valid: + // - Changes to feature set id (name, project) + // - Changes to entities + // - Changes to feature name and type rpc ApplyFeatureSet (ApplyFeatureSetRequest) returns (ApplyFeatureSetResponse); // Updates core with the configuration of the store. @@ -102,9 +105,6 @@ message GetFeatureSetRequest { // Name of feature set (required). string name = 1; - - // Version of feature set (optional). If omitted then latest feature set will be returned. - int32 version = 2; } // Response containing a single feature set @@ -133,15 +133,6 @@ message ListFeatureSetsRequest { // - my-feature-set* can be used to match all features prefixed by "my-feature-set" // - my-feature-set-6 can be used to select a single feature set string feature_set_name = 1; - - - // Versions of the given feature sets that will be returned. - // Valid options for version: - // "latest": only the latest version is returned. - // "*": Subscribe to all versions - // [version number]: pin to a specific version. Project and feature set name must be - // explicitly defined if a specific version is pinned. - string feature_set_version = 2; } } @@ -163,23 +154,25 @@ message ListStoresResponse { } message ApplyFeatureSetRequest { - // Feature set version and source will be ignored feast.core.FeatureSet feature_set = 1; } message ApplyFeatureSetResponse { + // TODO: 0 should correspond to invalid rather than NO_CHANGE enum Status { - // Latest feature set version is consistent with provided feature set + // Latest feature set is consistent with provided feature set NO_CHANGE = 0; - // New feature set or feature set version created + // New feature set created CREATED = 1; // Error occurred while trying to apply changes ERROR = 2; + + // Changes detected and updated successfully + UPDATED = 3; } - // Feature set response has been enriched with version and source information feast.core.FeatureSet feature_set = 1; Status status = 2; } diff --git a/protos/feast/core/FeatureSet.proto b/protos/feast/core/FeatureSet.proto index e7e69ede562..f4315256a67 100644 --- a/protos/feast/core/FeatureSet.proto +++ b/protos/feast/core/FeatureSet.proto @@ -40,8 +40,8 @@ message FeatureSetSpec { // Name of the feature set. Must be unique. string name = 1; - // Feature set version. - int32 version = 2; + // Feature set version was removed in v0.5.0. + reserved 2; // List of entities contained within this featureSet. // This allows the feature to be used during joins between feature sets. diff --git a/protos/feast/core/FeatureSetReference.proto b/protos/feast/core/FeatureSetReference.proto index 2501ec0931c..fb2f56425c3 100644 --- a/protos/feast/core/FeatureSetReference.proto +++ b/protos/feast/core/FeatureSetReference.proto @@ -28,6 +28,6 @@ message FeatureSetReference { string project = 1; // Name of the FeatureSet string name = 2; - // Version no. of the FeatureSet - int32 version = 3; + // Feature set version was removed in v0.5.0. + reserved 3; } diff --git a/protos/feast/core/Store.proto b/protos/feast/core/Store.proto index f35561467e1..3e5855753c7 100644 --- a/protos/feast/core/Store.proto +++ b/protos/feast/core/Store.proto @@ -48,12 +48,7 @@ message Store { // BigQuery stores a FeatureRow element as a row in a BigQuery table. // - // Table name is derived from the feature set name and version as: - // [feature_set_name]_v[feature_set_version] - // - // For example: - // A feature row for feature set "driver" and version "1" will be written - // to table "driver_v1". + // Table name is derived is the same as the feature set name. // // The entities and features in a FeatureSetSpec corresponds to the // fields in the BigQuery table (these make up the BigQuery schema). @@ -74,11 +69,6 @@ message Store { // // BigQuery table created will be partitioned by the field "event_timestamp" // of the FeatureRow (https://cloud.google.com/bigquery/docs/partitioned-tables). - // - // Since newer version of feature set can introduce breaking, non backward- - // compatible BigQuery schema updates, incrementing the version of a - // feature set will result in the creation of a new empty BigQuery table - // with the new schema. // // The following table shows how ValueType in Feast is mapped to // BigQuery Standard SQL data types @@ -149,7 +139,6 @@ message Store { // pattern matching. string project = 3; - // Name of the desired feature set. Asterisks can be used as wildcards in the name. // Matching on names is only permitted if a specific project is defined. It is disallowed // If the project name is set to "*" @@ -159,13 +148,8 @@ message Store { // - my-feature-set-6 can be used to select a single feature set string name = 1; - // Versions of the given feature sets that will be returned. - // Valid options for version: - // "latest": only the latest version is returned. - // "*": Subscribe to all versions - // [version number]: pin to a specific version. Project and feature set name must be - // explicitly defined if a specific version is pinned. - string version = 2; + // Feature set version was removed in v0.5.0. + reserved 2; } // Name of the store. diff --git a/protos/feast/serving/ServingService.proto b/protos/feast/serving/ServingService.proto index 5145670ec9a..bd302a83b46 100644 --- a/protos/feast/serving/ServingService.proto +++ b/protos/feast/serving/ServingService.proto @@ -69,9 +69,6 @@ message FeatureReference { // Feature name string name = 2; - // Feature version - int32 version = 3; - // The features will be retrieved if: // entity_timestamp - max_age <= event_timestamp <= entity_timestamp // diff --git a/protos/feast/storage/Redis.proto b/protos/feast/storage/Redis.proto index f58b137e9c1..c373f1b0524 100644 --- a/protos/feast/storage/Redis.proto +++ b/protos/feast/storage/Redis.proto @@ -28,7 +28,7 @@ message RedisKey { // Field number 1 is reserved for a future distributing hash if needed // (for when redis is clustered). - // FeatureSet this row belongs to, this is defined as featureSetName:version. + // FeatureSet this row belongs to, this is defined as featureSetName. string feature_set = 2; // List of fields containing entity names and their respective values diff --git a/protos/feast/types/FeatureRow.proto b/protos/feast/types/FeatureRow.proto index c19a393fde5..3e9056d12c2 100644 --- a/protos/feast/types/FeatureRow.proto +++ b/protos/feast/types/FeatureRow.proto @@ -36,7 +36,7 @@ message FeatureRow { google.protobuf.Timestamp event_timestamp = 3; // Complete reference to the featureSet this featureRow belongs to, in the form of - // /:. This value will be used by the feast ingestion job to filter + // /. This value will be used by the feast ingestion job to filter // rows, and write the values to the correct tables. string feature_set = 6; diff --git a/sdk/go/README.md b/sdk/go/README.md index 6084f909931..464eccf93d6 100644 --- a/sdk/go/README.md +++ b/sdk/go/README.md @@ -17,7 +17,7 @@ func main() { ctx := context.Background() req := feast.OnlineFeaturesRequest{ - Features: []string{"my_project_1/feature1:1", "my_project_2/feature1:1", "my_project_4/feature3", "feature2:2", "feature2"}, + Features: []string{"my_project_1/feature1", "my_project_2/feature1", "my_project_4/feature3", "feature2", "feature2"}, Entities: []feast.Row{ {"entity1": feast.Int64Val(1), "entity2": feast.StrVal("bob")}, {"entity1": feast.Int64Val(1), "entity2": feast.StrVal("annie")}, @@ -40,10 +40,10 @@ func main() { If all features retrieved are of a single type, Feast provides convenience functions to retrieve your features as a vector of feature values: ```{go} arr, err := resp.Int64Arrays( - []string{"my_project_1/feature1:1", - "my_project_2/feature1:1", + []string{"my_project_1/feature1", + "my_project_2/feature1", "my_project_4/feature3", - "feature2:2", + "feature2", "feature2"}, // order of features []int64{1,2,3,4,5}) // fillNa values ``` diff --git a/sdk/go/protos/feast/core/CoreService.pb.go b/sdk/go/protos/feast/core/CoreService.pb.go index 90e5f7d2408..1af820f50bb 100644 --- a/sdk/go/protos/feast/core/CoreService.pb.go +++ b/sdk/go/protos/feast/core/CoreService.pb.go @@ -48,12 +48,14 @@ const _ = proto.ProtoPackageIsVersion4 type ApplyFeatureSetResponse_Status int32 const ( - // Latest feature set version is consistent with provided feature set + // Latest feature set is consistent with provided feature set ApplyFeatureSetResponse_NO_CHANGE ApplyFeatureSetResponse_Status = 0 - // New feature set or feature set version created + // New feature set created ApplyFeatureSetResponse_CREATED ApplyFeatureSetResponse_Status = 1 // Error occurred while trying to apply changes ApplyFeatureSetResponse_ERROR ApplyFeatureSetResponse_Status = 2 + // Changes detected and updated successfully + ApplyFeatureSetResponse_UPDATED ApplyFeatureSetResponse_Status = 3 ) // Enum value maps for ApplyFeatureSetResponse_Status. @@ -62,11 +64,13 @@ var ( 0: "NO_CHANGE", 1: "CREATED", 2: "ERROR", + 3: "UPDATED", } ApplyFeatureSetResponse_Status_value = map[string]int32{ "NO_CHANGE": 0, "CREATED": 1, "ERROR": 2, + "UPDATED": 3, } ) @@ -155,8 +159,6 @@ type GetFeatureSetRequest struct { Project string `protobuf:"bytes,3,opt,name=project,proto3" json:"project,omitempty"` // Name of feature set (required). Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - // Version of feature set (optional). If omitted then latest feature set will be returned. - Version int32 `protobuf:"varint,2,opt,name=version,proto3" json:"version,omitempty"` } func (x *GetFeatureSetRequest) Reset() { @@ -205,13 +207,6 @@ func (x *GetFeatureSetRequest) GetName() string { return "" } -func (x *GetFeatureSetRequest) GetVersion() int32 { - if x != nil { - return x.Version - } - return 0 -} - // Response containing a single feature set type GetFeatureSetResponse struct { state protoimpl.MessageState @@ -454,7 +449,6 @@ type ApplyFeatureSetRequest struct { sizeCache protoimpl.SizeCache unknownFields protoimpl.UnknownFields - // Feature set version and source will be ignored FeatureSet *FeatureSet `protobuf:"bytes,1,opt,name=feature_set,json=featureSet,proto3" json:"feature_set,omitempty"` } @@ -502,7 +496,6 @@ type ApplyFeatureSetResponse struct { sizeCache protoimpl.SizeCache unknownFields protoimpl.UnknownFields - // Feature set response has been enriched with version and source information FeatureSet *FeatureSet `protobuf:"bytes,1,opt,name=feature_set,json=featureSet,proto3" json:"feature_set,omitempty"` Status ApplyFeatureSetResponse_Status `protobuf:"varint,2,opt,name=status,proto3,enum=feast.core.ApplyFeatureSetResponse_Status" json:"status,omitempty"` } @@ -1296,13 +1289,6 @@ type ListFeatureSetsRequest_Filter struct { // - my-feature-set* can be used to match all features prefixed by "my-feature-set" // - my-feature-set-6 can be used to select a single feature set FeatureSetName string `protobuf:"bytes,1,opt,name=feature_set_name,json=featureSetName,proto3" json:"feature_set_name,omitempty"` - // Versions of the given feature sets that will be returned. - // Valid options for version: - // "latest": only the latest version is returned. - // "*": Subscribe to all versions - // [version number]: pin to a specific version. Project and feature set name must be - // explicitly defined if a specific version is pinned. - FeatureSetVersion string `protobuf:"bytes,2,opt,name=feature_set_version,json=featureSetVersion,proto3" json:"feature_set_version,omitempty"` } func (x *ListFeatureSetsRequest_Filter) Reset() { @@ -1351,13 +1337,6 @@ func (x *ListFeatureSetsRequest_Filter) GetFeatureSetName() string { return "" } -func (x *ListFeatureSetsRequest_Filter) GetFeatureSetVersion() string { - if x != nil { - return x.FeatureSetVersion - } - return "" -} - type ListStoresRequest_Filter struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache @@ -1485,202 +1464,199 @@ var file_feast_core_CoreService_proto_rawDesc = []byte{ 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x1a, 0x1d, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x2e, 0x70, - 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x5e, 0x0a, 0x14, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, + 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x44, 0x0a, 0x14, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x18, 0x0a, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, - 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, - 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, 0x07, 0x76, 0x65, 0x72, - 0x73, 0x69, 0x6f, 0x6e, 0x22, 0x50, 0x0a, 0x15, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, + 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x50, 0x0a, 0x15, 0x47, 0x65, + 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, + 0x6e, 0x73, 0x65, 0x12, 0x37, 0x0a, 0x0b, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x5f, 0x73, + 0x65, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, + 0x52, 0x0a, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x22, 0xa9, 0x01, 0x0a, + 0x16, 0x4c, 0x69, 0x73, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, + 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x41, 0x0a, 0x06, 0x66, 0x69, 0x6c, 0x74, 0x65, + 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x29, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, + 0x53, 0x65, 0x74, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x2e, 0x46, 0x69, 0x6c, 0x74, + 0x65, 0x72, 0x52, 0x06, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x1a, 0x4c, 0x0a, 0x06, 0x46, 0x69, + 0x6c, 0x74, 0x65, 0x72, 0x12, 0x18, 0x0a, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x18, + 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x12, 0x28, + 0x0a, 0x10, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x5f, 0x73, 0x65, 0x74, 0x5f, 0x6e, 0x61, + 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0e, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, + 0x65, 0x53, 0x65, 0x74, 0x4e, 0x61, 0x6d, 0x65, 0x22, 0x54, 0x0a, 0x17, 0x4c, 0x69, 0x73, 0x74, + 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, + 0x6e, 0x73, 0x65, 0x12, 0x39, 0x0a, 0x0c, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x5f, 0x73, + 0x65, 0x74, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, + 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, + 0x74, 0x52, 0x0b, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, 0x22, 0x6f, + 0x0a, 0x11, 0x4c, 0x69, 0x73, 0x74, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, + 0x65, 0x73, 0x74, 0x12, 0x3c, 0x0a, 0x06, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x18, 0x01, 0x20, + 0x01, 0x28, 0x0b, 0x32, 0x24, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, + 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, + 0x73, 0x74, 0x2e, 0x46, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x52, 0x06, 0x66, 0x69, 0x6c, 0x74, 0x65, + 0x72, 0x1a, 0x1c, 0x0a, 0x06, 0x46, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x12, 0x12, 0x0a, 0x04, 0x6e, + 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x22, + 0x3d, 0x0a, 0x12, 0x4c, 0x69, 0x73, 0x74, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, + 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x27, 0x0a, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x18, 0x01, + 0x20, 0x03, 0x28, 0x0b, 0x32, 0x11, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, + 0x65, 0x2e, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x22, 0x51, + 0x0a, 0x16, 0x41, 0x70, 0x70, 0x6c, 0x79, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, + 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x37, 0x0a, 0x0b, 0x66, 0x65, 0x61, 0x74, + 0x75, 0x72, 0x65, 0x5f, 0x73, 0x65, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, + 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, + 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x0a, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, + 0x74, 0x22, 0xd4, 0x01, 0x0a, 0x17, 0x41, 0x70, 0x70, 0x6c, 0x79, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x37, 0x0a, 0x0b, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x5f, 0x73, 0x65, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x0a, 0x66, 0x65, 0x61, 0x74, - 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x22, 0xd9, 0x01, 0x0a, 0x16, 0x4c, 0x69, 0x73, 0x74, 0x46, + 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x12, 0x42, 0x0a, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, + 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x2a, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, + 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x70, 0x70, 0x6c, 0x79, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, + 0x53, 0x65, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x53, 0x74, 0x61, 0x74, + 0x75, 0x73, 0x52, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x22, 0x3c, 0x0a, 0x06, 0x53, 0x74, + 0x61, 0x74, 0x75, 0x73, 0x12, 0x0d, 0x0a, 0x09, 0x4e, 0x4f, 0x5f, 0x43, 0x48, 0x41, 0x4e, 0x47, + 0x45, 0x10, 0x00, 0x12, 0x0b, 0x0a, 0x07, 0x43, 0x52, 0x45, 0x41, 0x54, 0x45, 0x44, 0x10, 0x01, + 0x12, 0x09, 0x0a, 0x05, 0x45, 0x52, 0x52, 0x4f, 0x52, 0x10, 0x02, 0x12, 0x0b, 0x0a, 0x07, 0x55, + 0x50, 0x44, 0x41, 0x54, 0x45, 0x44, 0x10, 0x03, 0x22, 0x1c, 0x0a, 0x1a, 0x47, 0x65, 0x74, 0x46, + 0x65, 0x61, 0x73, 0x74, 0x43, 0x6f, 0x72, 0x65, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x52, + 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x22, 0x37, 0x0a, 0x1b, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, + 0x73, 0x74, 0x43, 0x6f, 0x72, 0x65, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x52, 0x65, 0x73, + 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, + 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x22, + 0x3d, 0x0a, 0x12, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x65, + 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x27, 0x0a, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x18, 0x01, + 0x20, 0x01, 0x28, 0x0b, 0x32, 0x11, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, + 0x65, 0x2e, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x22, 0xa4, + 0x01, 0x0a, 0x13, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x65, + 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x27, 0x0a, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x18, + 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x11, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, + 0x72, 0x65, 0x2e, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x12, + 0x3e, 0x0a, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, + 0x26, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x55, 0x70, 0x64, + 0x61, 0x74, 0x65, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, + 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x22, + 0x24, 0x0a, 0x06, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x0d, 0x0a, 0x09, 0x4e, 0x4f, 0x5f, + 0x43, 0x48, 0x41, 0x4e, 0x47, 0x45, 0x10, 0x00, 0x12, 0x0b, 0x0a, 0x07, 0x55, 0x50, 0x44, 0x41, + 0x54, 0x45, 0x44, 0x10, 0x01, 0x22, 0x2a, 0x0a, 0x14, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x50, + 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x12, 0x0a, + 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, + 0x65, 0x22, 0x17, 0x0a, 0x15, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, + 0x63, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x2b, 0x0a, 0x15, 0x41, 0x72, + 0x63, 0x68, 0x69, 0x76, 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x52, 0x65, 0x71, 0x75, + 0x65, 0x73, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, + 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x18, 0x0a, 0x16, 0x41, 0x72, 0x63, 0x68, 0x69, + 0x76, 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, + 0x65, 0x22, 0x15, 0x0a, 0x13, 0x4c, 0x69, 0x73, 0x74, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, + 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x22, 0x32, 0x0a, 0x14, 0x4c, 0x69, 0x73, 0x74, + 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, + 0x12, 0x1a, 0x0a, 0x08, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x18, 0x01, 0x20, 0x03, + 0x28, 0x09, 0x52, 0x08, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x22, 0xee, 0x01, 0x0a, + 0x18, 0x4c, 0x69, 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, + 0x62, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x43, 0x0a, 0x06, 0x66, 0x69, 0x6c, + 0x74, 0x65, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2b, 0x2e, 0x66, 0x65, 0x61, 0x73, + 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, + 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x2e, + 0x46, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x52, 0x06, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x1a, 0x8c, + 0x01, 0x0a, 0x06, 0x46, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, + 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x12, 0x53, 0x0a, 0x15, 0x66, 0x65, 0x61, + 0x74, 0x75, 0x72, 0x65, 0x5f, 0x73, 0x65, 0x74, 0x5f, 0x72, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, + 0x63, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1f, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, + 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x52, 0x13, 0x66, 0x65, 0x61, 0x74, 0x75, + 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x12, 0x1d, + 0x0a, 0x0a, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x03, 0x20, 0x01, + 0x28, 0x09, 0x52, 0x09, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x4e, 0x61, 0x6d, 0x65, 0x22, 0x49, 0x0a, + 0x19, 0x4c, 0x69, 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, + 0x62, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x2c, 0x0a, 0x04, 0x6a, 0x6f, + 0x62, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x18, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, + 0x6f, 0x62, 0x52, 0x04, 0x6a, 0x6f, 0x62, 0x73, 0x22, 0x2c, 0x0a, 0x1a, 0x52, 0x65, 0x73, 0x74, + 0x61, 0x72, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, + 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, + 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x22, 0x1d, 0x0a, 0x1b, 0x52, 0x65, 0x73, 0x74, 0x61, 0x72, + 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x73, + 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x29, 0x0a, 0x17, 0x53, 0x74, 0x6f, 0x70, 0x49, 0x6e, 0x67, + 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, + 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, + 0x22, 0x1a, 0x0a, 0x18, 0x53, 0x74, 0x6f, 0x70, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, + 0x6e, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x32, 0xcb, 0x08, 0x0a, + 0x0b, 0x43, 0x6f, 0x72, 0x65, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x66, 0x0a, 0x13, + 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, 0x43, 0x6f, 0x72, 0x65, 0x56, 0x65, 0x72, 0x73, + 0x69, 0x6f, 0x6e, 0x12, 0x26, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, + 0x2e, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, 0x43, 0x6f, 0x72, 0x65, 0x56, 0x65, 0x72, + 0x73, 0x69, 0x6f, 0x6e, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x27, 0x2e, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, + 0x74, 0x43, 0x6f, 0x72, 0x65, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x52, 0x65, 0x73, 0x70, + 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x54, 0x0a, 0x0d, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, + 0x72, 0x65, 0x53, 0x65, 0x74, 0x12, 0x20, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, + 0x72, 0x65, 0x2e, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, + 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x21, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, + 0x65, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x5a, 0x0a, 0x0f, 0x4c, 0x69, + 0x73, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, 0x12, 0x22, 0x2e, + 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, - 0x74, 0x12, 0x41, 0x0a, 0x06, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, - 0x0b, 0x32, 0x29, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, + 0x74, 0x1a, 0x23, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, 0x52, 0x65, - 0x71, 0x75, 0x65, 0x73, 0x74, 0x2e, 0x46, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x52, 0x06, 0x66, 0x69, - 0x6c, 0x74, 0x65, 0x72, 0x1a, 0x7c, 0x0a, 0x06, 0x46, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x12, 0x18, - 0x0a, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, - 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x12, 0x28, 0x0a, 0x10, 0x66, 0x65, 0x61, 0x74, - 0x75, 0x72, 0x65, 0x5f, 0x73, 0x65, 0x74, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, - 0x28, 0x09, 0x52, 0x0e, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x4e, 0x61, - 0x6d, 0x65, 0x12, 0x2e, 0x0a, 0x13, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x5f, 0x73, 0x65, - 0x74, 0x5f, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, - 0x11, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x56, 0x65, 0x72, 0x73, 0x69, - 0x6f, 0x6e, 0x22, 0x54, 0x0a, 0x17, 0x4c, 0x69, 0x73, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, - 0x65, 0x53, 0x65, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x39, 0x0a, - 0x0c, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x5f, 0x73, 0x65, 0x74, 0x73, 0x18, 0x01, 0x20, - 0x03, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, - 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x0b, 0x66, 0x65, 0x61, - 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, 0x22, 0x6f, 0x0a, 0x11, 0x4c, 0x69, 0x73, 0x74, - 0x53, 0x74, 0x6f, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x3c, 0x0a, - 0x06, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x24, 0x2e, - 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x53, - 0x74, 0x6f, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x2e, 0x46, 0x69, 0x6c, - 0x74, 0x65, 0x72, 0x52, 0x06, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x1a, 0x1c, 0x0a, 0x06, 0x46, - 0x69, 0x6c, 0x74, 0x65, 0x72, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, - 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x3d, 0x0a, 0x12, 0x4c, 0x69, 0x73, - 0x74, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, - 0x27, 0x0a, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x11, - 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x74, 0x6f, 0x72, - 0x65, 0x52, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x22, 0x51, 0x0a, 0x16, 0x41, 0x70, 0x70, 0x6c, - 0x79, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, - 0x73, 0x74, 0x12, 0x37, 0x0a, 0x0b, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x5f, 0x73, 0x65, - 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, - 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, - 0x0a, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x22, 0xc7, 0x01, 0x0a, 0x17, - 0x41, 0x70, 0x70, 0x6c, 0x79, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, - 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x37, 0x0a, 0x0b, 0x66, 0x65, 0x61, 0x74, 0x75, - 0x72, 0x65, 0x5f, 0x73, 0x65, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, - 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, - 0x65, 0x53, 0x65, 0x74, 0x52, 0x0a, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, - 0x12, 0x42, 0x0a, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, - 0x32, 0x2a, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x70, - 0x70, 0x6c, 0x79, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x73, - 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x06, 0x73, 0x74, - 0x61, 0x74, 0x75, 0x73, 0x22, 0x2f, 0x0a, 0x06, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x0d, - 0x0a, 0x09, 0x4e, 0x4f, 0x5f, 0x43, 0x48, 0x41, 0x4e, 0x47, 0x45, 0x10, 0x00, 0x12, 0x0b, 0x0a, - 0x07, 0x43, 0x52, 0x45, 0x41, 0x54, 0x45, 0x44, 0x10, 0x01, 0x12, 0x09, 0x0a, 0x05, 0x45, 0x52, - 0x52, 0x4f, 0x52, 0x10, 0x02, 0x22, 0x1c, 0x0a, 0x1a, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, - 0x74, 0x43, 0x6f, 0x72, 0x65, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x52, 0x65, 0x71, 0x75, - 0x65, 0x73, 0x74, 0x22, 0x37, 0x0a, 0x1b, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, 0x43, - 0x6f, 0x72, 0x65, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, - 0x73, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x01, 0x20, - 0x01, 0x28, 0x09, 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x22, 0x3d, 0x0a, 0x12, - 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, - 0x73, 0x74, 0x12, 0x27, 0x0a, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, - 0x0b, 0x32, 0x11, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, - 0x74, 0x6f, 0x72, 0x65, 0x52, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x22, 0xa4, 0x01, 0x0a, 0x13, - 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, - 0x6e, 0x73, 0x65, 0x12, 0x27, 0x0a, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x18, 0x01, 0x20, 0x01, - 0x28, 0x0b, 0x32, 0x11, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, - 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x05, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x12, 0x3e, 0x0a, 0x06, - 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x26, 0x2e, 0x66, - 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, - 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x53, 0x74, - 0x61, 0x74, 0x75, 0x73, 0x52, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x22, 0x24, 0x0a, 0x06, - 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x0d, 0x0a, 0x09, 0x4e, 0x4f, 0x5f, 0x43, 0x48, 0x41, - 0x4e, 0x47, 0x45, 0x10, 0x00, 0x12, 0x0b, 0x0a, 0x07, 0x55, 0x50, 0x44, 0x41, 0x54, 0x45, 0x44, - 0x10, 0x01, 0x22, 0x2a, 0x0a, 0x14, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x50, 0x72, 0x6f, 0x6a, - 0x65, 0x63, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, - 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x17, - 0x0a, 0x15, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x52, - 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x2b, 0x0a, 0x15, 0x41, 0x72, 0x63, 0x68, 0x69, - 0x76, 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, - 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, - 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x18, 0x0a, 0x16, 0x41, 0x72, 0x63, 0x68, 0x69, 0x76, 0x65, 0x50, - 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x22, 0x15, - 0x0a, 0x13, 0x4c, 0x69, 0x73, 0x74, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x52, 0x65, - 0x71, 0x75, 0x65, 0x73, 0x74, 0x22, 0x32, 0x0a, 0x14, 0x4c, 0x69, 0x73, 0x74, 0x50, 0x72, 0x6f, - 0x6a, 0x65, 0x63, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x1a, 0x0a, - 0x08, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x09, 0x52, - 0x08, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x22, 0xee, 0x01, 0x0a, 0x18, 0x4c, 0x69, - 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x73, 0x52, - 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x43, 0x0a, 0x06, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, - 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2b, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, - 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, - 0x6e, 0x4a, 0x6f, 0x62, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x2e, 0x46, 0x69, 0x6c, - 0x74, 0x65, 0x72, 0x52, 0x06, 0x66, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x1a, 0x8c, 0x01, 0x0a, 0x06, - 0x46, 0x69, 0x6c, 0x74, 0x65, 0x72, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, - 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x12, 0x53, 0x0a, 0x15, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, - 0x65, 0x5f, 0x73, 0x65, 0x74, 0x5f, 0x72, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x18, - 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1f, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, - 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x66, - 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x52, 0x13, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, - 0x65, 0x74, 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x12, 0x1d, 0x0a, 0x0a, 0x73, - 0x74, 0x6f, 0x72, 0x65, 0x5f, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, - 0x09, 0x73, 0x74, 0x6f, 0x72, 0x65, 0x4e, 0x61, 0x6d, 0x65, 0x22, 0x49, 0x0a, 0x19, 0x4c, 0x69, - 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x73, 0x52, - 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x2c, 0x0a, 0x04, 0x6a, 0x6f, 0x62, 0x73, 0x18, - 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x18, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, - 0x72, 0x65, 0x2e, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, - 0x04, 0x6a, 0x6f, 0x62, 0x73, 0x22, 0x2c, 0x0a, 0x1a, 0x52, 0x65, 0x73, 0x74, 0x61, 0x72, 0x74, - 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x71, 0x75, - 0x65, 0x73, 0x74, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, - 0x02, 0x69, 0x64, 0x22, 0x1d, 0x0a, 0x1b, 0x52, 0x65, 0x73, 0x74, 0x61, 0x72, 0x74, 0x49, 0x6e, - 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, - 0x73, 0x65, 0x22, 0x29, 0x0a, 0x17, 0x53, 0x74, 0x6f, 0x70, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, - 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x0e, 0x0a, - 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x22, 0x1a, 0x0a, - 0x18, 0x53, 0x74, 0x6f, 0x70, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, - 0x62, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x32, 0xcb, 0x08, 0x0a, 0x0b, 0x43, 0x6f, - 0x72, 0x65, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x66, 0x0a, 0x13, 0x47, 0x65, 0x74, - 0x46, 0x65, 0x61, 0x73, 0x74, 0x43, 0x6f, 0x72, 0x65, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, - 0x12, 0x26, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x47, 0x65, - 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, 0x43, 0x6f, 0x72, 0x65, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, - 0x6e, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x27, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, - 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, 0x43, 0x6f, - 0x72, 0x65, 0x56, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, - 0x65, 0x12, 0x54, 0x0a, 0x0d, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, - 0x65, 0x74, 0x12, 0x20, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, - 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x71, - 0x75, 0x65, 0x73, 0x74, 0x1a, 0x21, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, - 0x65, 0x2e, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, - 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x5a, 0x0a, 0x0f, 0x4c, 0x69, 0x73, 0x74, 0x46, - 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, 0x12, 0x22, 0x2e, 0x66, 0x65, 0x61, - 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x46, 0x65, 0x61, 0x74, - 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x23, - 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, - 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, - 0x6e, 0x73, 0x65, 0x12, 0x4b, 0x0a, 0x0a, 0x4c, 0x69, 0x73, 0x74, 0x53, 0x74, 0x6f, 0x72, 0x65, - 0x73, 0x12, 0x1d, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, - 0x69, 0x73, 0x74, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, - 0x1a, 0x1e, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, - 0x73, 0x74, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, - 0x12, 0x5a, 0x0a, 0x0f, 0x41, 0x70, 0x70, 0x6c, 0x79, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, - 0x53, 0x65, 0x74, 0x12, 0x22, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, - 0x2e, 0x41, 0x70, 0x70, 0x6c, 0x79, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, - 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x23, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, - 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x70, 0x70, 0x6c, 0x79, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, - 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x4e, 0x0a, 0x0b, - 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x12, 0x1e, 0x2e, 0x66, 0x65, - 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x53, - 0x74, 0x6f, 0x72, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x1f, 0x2e, 0x66, 0x65, - 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x53, - 0x74, 0x6f, 0x72, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x54, 0x0a, 0x0d, - 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x12, 0x20, 0x2e, - 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x43, 0x72, 0x65, 0x61, 0x74, - 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, - 0x21, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x43, 0x72, 0x65, - 0x61, 0x74, 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, - 0x73, 0x65, 0x12, 0x57, 0x0a, 0x0e, 0x41, 0x72, 0x63, 0x68, 0x69, 0x76, 0x65, 0x50, 0x72, 0x6f, - 0x6a, 0x65, 0x63, 0x74, 0x12, 0x21, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, - 0x65, 0x2e, 0x41, 0x72, 0x63, 0x68, 0x69, 0x76, 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, - 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x22, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x4b, 0x0a, 0x0a, 0x4c, 0x69, 0x73, 0x74, 0x53, 0x74, + 0x6f, 0x72, 0x65, 0x73, 0x12, 0x1d, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, + 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, + 0x65, 0x73, 0x74, 0x1a, 0x1e, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, + 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, + 0x6e, 0x73, 0x65, 0x12, 0x5a, 0x0a, 0x0f, 0x41, 0x70, 0x70, 0x6c, 0x79, 0x46, 0x65, 0x61, 0x74, + 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x12, 0x22, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, + 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x70, 0x70, 0x6c, 0x79, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, + 0x53, 0x65, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x23, 0x2e, 0x66, 0x65, 0x61, + 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x70, 0x70, 0x6c, 0x79, 0x46, 0x65, 0x61, + 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, + 0x4e, 0x0a, 0x0b, 0x55, 0x70, 0x64, 0x61, 0x74, 0x65, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x12, 0x1e, + 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x55, 0x70, 0x64, 0x61, + 0x74, 0x65, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x1f, + 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x55, 0x70, 0x64, 0x61, + 0x74, 0x65, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, + 0x54, 0x0a, 0x0d, 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, + 0x12, 0x20, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x43, 0x72, + 0x65, 0x61, 0x74, 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, + 0x73, 0x74, 0x1a, 0x21, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, + 0x43, 0x72, 0x65, 0x61, 0x74, 0x65, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x52, 0x65, 0x73, + 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x57, 0x0a, 0x0e, 0x41, 0x72, 0x63, 0x68, 0x69, 0x76, 0x65, + 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x12, 0x21, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x72, 0x63, 0x68, 0x69, 0x76, 0x65, 0x50, 0x72, 0x6f, 0x6a, - 0x65, 0x63, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x51, 0x0a, 0x0c, 0x4c, - 0x69, 0x73, 0x74, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x12, 0x1f, 0x2e, 0x66, 0x65, - 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x50, 0x72, 0x6f, - 0x6a, 0x65, 0x63, 0x74, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x20, 0x2e, 0x66, - 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x50, 0x72, - 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x60, - 0x0a, 0x11, 0x4c, 0x69, 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, - 0x6f, 0x62, 0x73, 0x12, 0x24, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, - 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, - 0x62, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x25, 0x2e, 0x66, 0x65, 0x61, 0x73, - 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, - 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, - 0x12, 0x66, 0x0a, 0x13, 0x52, 0x65, 0x73, 0x74, 0x61, 0x72, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, - 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x12, 0x26, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, - 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x52, 0x65, 0x73, 0x74, 0x61, 0x72, 0x74, 0x49, 0x6e, 0x67, 0x65, - 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, - 0x27, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x52, 0x65, 0x73, - 0x74, 0x61, 0x72, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, - 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x5d, 0x0a, 0x10, 0x53, 0x74, 0x6f, 0x70, - 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x12, 0x23, 0x2e, 0x66, - 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x74, 0x6f, 0x70, 0x49, 0x6e, - 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, - 0x74, 0x1a, 0x24, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, - 0x74, 0x6f, 0x70, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, - 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x42, 0x4f, 0x0a, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, - 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x42, 0x10, 0x43, 0x6f, 0x72, 0x65, 0x53, 0x65, 0x72, 0x76, 0x69, - 0x63, 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x2f, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, - 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, - 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, - 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, + 0x65, 0x63, 0x74, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x22, 0x2e, 0x66, 0x65, 0x61, + 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x41, 0x72, 0x63, 0x68, 0x69, 0x76, 0x65, 0x50, + 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x51, + 0x0a, 0x0c, 0x4c, 0x69, 0x73, 0x74, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x12, 0x1f, + 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, + 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, + 0x20, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, + 0x74, 0x50, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, + 0x65, 0x12, 0x60, 0x0a, 0x11, 0x4c, 0x69, 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, + 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x73, 0x12, 0x24, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, + 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, + 0x6e, 0x4a, 0x6f, 0x62, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x25, 0x2e, 0x66, + 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4c, 0x69, 0x73, 0x74, 0x49, 0x6e, + 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, + 0x6e, 0x73, 0x65, 0x12, 0x66, 0x0a, 0x13, 0x52, 0x65, 0x73, 0x74, 0x61, 0x72, 0x74, 0x49, 0x6e, + 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x12, 0x26, 0x2e, 0x66, 0x65, 0x61, + 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x52, 0x65, 0x73, 0x74, 0x61, 0x72, 0x74, 0x49, + 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x71, 0x75, 0x65, + 0x73, 0x74, 0x1a, 0x27, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, + 0x52, 0x65, 0x73, 0x74, 0x61, 0x72, 0x74, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, + 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x5d, 0x0a, 0x10, 0x53, + 0x74, 0x6f, 0x70, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x12, + 0x23, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x74, 0x6f, + 0x70, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x71, + 0x75, 0x65, 0x73, 0x74, 0x1a, 0x24, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, + 0x65, 0x2e, 0x53, 0x74, 0x6f, 0x70, 0x49, 0x6e, 0x67, 0x65, 0x73, 0x74, 0x69, 0x6f, 0x6e, 0x4a, + 0x6f, 0x62, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x42, 0x4f, 0x0a, 0x0a, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x42, 0x10, 0x43, 0x6f, 0x72, 0x65, 0x53, 0x65, + 0x72, 0x76, 0x69, 0x63, 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x2f, 0x67, 0x69, 0x74, 0x68, + 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, + 0x73, 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, + 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, + 0x74, 0x6f, 0x33, } var ( @@ -2166,8 +2142,11 @@ type CoreServiceClient interface { // Create or update and existing feature set. // // This function is idempotent - it will not create a new feature set if schema does not change. - // If an existing feature set is updated, core will advance the version number, which will be - // returned in response. + // Schema changes will update the feature set if the changes are valid. + // All changes except the following are valid: + // - Changes to feature set id (name, project) + // - Changes to entities + // - Changes to feature type ApplyFeatureSet(ctx context.Context, in *ApplyFeatureSetRequest, opts ...grpc.CallOption) (*ApplyFeatureSetResponse, error) // Updates core with the configuration of the store. // @@ -2339,8 +2318,11 @@ type CoreServiceServer interface { // Create or update and existing feature set. // // This function is idempotent - it will not create a new feature set if schema does not change. - // If an existing feature set is updated, core will advance the version number, which will be - // returned in response. + // Schema changes will update the feature set if the changes are valid. + // All changes except the following are valid: + // - Changes to feature set id (name, project) + // - Changes to entities + // - Changes to feature type ApplyFeatureSet(context.Context, *ApplyFeatureSetRequest) (*ApplyFeatureSetResponse, error) // Updates core with the configuration of the store. // diff --git a/sdk/go/protos/feast/core/FeatureSet.pb.go b/sdk/go/protos/feast/core/FeatureSet.pb.go index bbf79e7d2a4..091dd1ca9fb 100644 --- a/sdk/go/protos/feast/core/FeatureSet.pb.go +++ b/sdk/go/protos/feast/core/FeatureSet.pb.go @@ -17,7 +17,7 @@ // Code generated by protoc-gen-go. DO NOT EDIT. // versions: // protoc-gen-go v1.21.0 -// protoc v3.10.1 +// protoc v3.10.0 // source: feast/core/FeatureSet.proto package core @@ -160,8 +160,6 @@ type FeatureSetSpec struct { Project string `protobuf:"bytes,7,opt,name=project,proto3" json:"project,omitempty"` // Name of the feature set. Must be unique. Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - // Feature set version. - Version int32 `protobuf:"varint,2,opt,name=version,proto3" json:"version,omitempty"` // List of entities contained within this featureSet. // This allows the feature to be used during joins between feature sets. // If the featureSet is ingested into a store that supports keys, this value @@ -176,6 +174,8 @@ type FeatureSetSpec struct { // Optional. Source on which feature rows can be found. // If not set, source will be set to the default value configured in Feast Core. Source *Source `protobuf:"bytes,6,opt,name=source,proto3" json:"source,omitempty"` + // User defined metadata + Labels map[string]string `protobuf:"bytes,8,rep,name=labels,proto3" json:"labels,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` } func (x *FeatureSetSpec) Reset() { @@ -224,13 +224,6 @@ func (x *FeatureSetSpec) GetName() string { return "" } -func (x *FeatureSetSpec) GetVersion() int32 { - if x != nil { - return x.Version - } - return 0 -} - func (x *FeatureSetSpec) GetEntities() []*EntitySpec { if x != nil { return x.Entities @@ -259,6 +252,13 @@ func (x *FeatureSetSpec) GetSource() *Source { return nil } +func (x *FeatureSetSpec) GetLabels() map[string]string { + if x != nil { + return x.Labels + } + return nil +} + type EntitySpec struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache @@ -266,35 +266,8 @@ type EntitySpec struct { // Name of the entity. Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - // Value type of the feature. + // Value type of the entity. ValueType types.ValueType_Enum `protobuf:"varint,2,opt,name=value_type,json=valueType,proto3,enum=feast.types.ValueType_Enum" json:"value_type,omitempty"` - // Types that are assignable to PresenceConstraints: - // *EntitySpec_Presence - // *EntitySpec_GroupPresence - PresenceConstraints isEntitySpec_PresenceConstraints `protobuf_oneof:"presence_constraints"` - // The shape of the feature which governs the number of values that appear in - // each example. - // - // Types that are assignable to ShapeType: - // *EntitySpec_Shape - // *EntitySpec_ValueCount - ShapeType isEntitySpec_ShapeType `protobuf_oneof:"shape_type"` - // Domain for the values of the feature. - // - // Types that are assignable to DomainInfo: - // *EntitySpec_Domain - // *EntitySpec_IntDomain - // *EntitySpec_FloatDomain - // *EntitySpec_StringDomain - // *EntitySpec_BoolDomain - // *EntitySpec_StructDomain - // *EntitySpec_NaturalLanguageDomain - // *EntitySpec_ImageDomain - // *EntitySpec_MidDomain - // *EntitySpec_UrlDomain - // *EntitySpec_TimeDomain - // *EntitySpec_TimeOfDayDomain - DomainInfo isEntitySpec_DomainInfo `protobuf_oneof:"domain_info"` } func (x *EntitySpec) Reset() { @@ -343,256 +316,6 @@ func (x *EntitySpec) GetValueType() types.ValueType_Enum { return types.ValueType_INVALID } -func (m *EntitySpec) GetPresenceConstraints() isEntitySpec_PresenceConstraints { - if m != nil { - return m.PresenceConstraints - } - return nil -} - -func (x *EntitySpec) GetPresence() *v0.FeaturePresence { - if x, ok := x.GetPresenceConstraints().(*EntitySpec_Presence); ok { - return x.Presence - } - return nil -} - -func (x *EntitySpec) GetGroupPresence() *v0.FeaturePresenceWithinGroup { - if x, ok := x.GetPresenceConstraints().(*EntitySpec_GroupPresence); ok { - return x.GroupPresence - } - return nil -} - -func (m *EntitySpec) GetShapeType() isEntitySpec_ShapeType { - if m != nil { - return m.ShapeType - } - return nil -} - -func (x *EntitySpec) GetShape() *v0.FixedShape { - if x, ok := x.GetShapeType().(*EntitySpec_Shape); ok { - return x.Shape - } - return nil -} - -func (x *EntitySpec) GetValueCount() *v0.ValueCount { - if x, ok := x.GetShapeType().(*EntitySpec_ValueCount); ok { - return x.ValueCount - } - return nil -} - -func (m *EntitySpec) GetDomainInfo() isEntitySpec_DomainInfo { - if m != nil { - return m.DomainInfo - } - return nil -} - -func (x *EntitySpec) GetDomain() string { - if x, ok := x.GetDomainInfo().(*EntitySpec_Domain); ok { - return x.Domain - } - return "" -} - -func (x *EntitySpec) GetIntDomain() *v0.IntDomain { - if x, ok := x.GetDomainInfo().(*EntitySpec_IntDomain); ok { - return x.IntDomain - } - return nil -} - -func (x *EntitySpec) GetFloatDomain() *v0.FloatDomain { - if x, ok := x.GetDomainInfo().(*EntitySpec_FloatDomain); ok { - return x.FloatDomain - } - return nil -} - -func (x *EntitySpec) GetStringDomain() *v0.StringDomain { - if x, ok := x.GetDomainInfo().(*EntitySpec_StringDomain); ok { - return x.StringDomain - } - return nil -} - -func (x *EntitySpec) GetBoolDomain() *v0.BoolDomain { - if x, ok := x.GetDomainInfo().(*EntitySpec_BoolDomain); ok { - return x.BoolDomain - } - return nil -} - -func (x *EntitySpec) GetStructDomain() *v0.StructDomain { - if x, ok := x.GetDomainInfo().(*EntitySpec_StructDomain); ok { - return x.StructDomain - } - return nil -} - -func (x *EntitySpec) GetNaturalLanguageDomain() *v0.NaturalLanguageDomain { - if x, ok := x.GetDomainInfo().(*EntitySpec_NaturalLanguageDomain); ok { - return x.NaturalLanguageDomain - } - return nil -} - -func (x *EntitySpec) GetImageDomain() *v0.ImageDomain { - if x, ok := x.GetDomainInfo().(*EntitySpec_ImageDomain); ok { - return x.ImageDomain - } - return nil -} - -func (x *EntitySpec) GetMidDomain() *v0.MIDDomain { - if x, ok := x.GetDomainInfo().(*EntitySpec_MidDomain); ok { - return x.MidDomain - } - return nil -} - -func (x *EntitySpec) GetUrlDomain() *v0.URLDomain { - if x, ok := x.GetDomainInfo().(*EntitySpec_UrlDomain); ok { - return x.UrlDomain - } - return nil -} - -func (x *EntitySpec) GetTimeDomain() *v0.TimeDomain { - if x, ok := x.GetDomainInfo().(*EntitySpec_TimeDomain); ok { - return x.TimeDomain - } - return nil -} - -func (x *EntitySpec) GetTimeOfDayDomain() *v0.TimeOfDayDomain { - if x, ok := x.GetDomainInfo().(*EntitySpec_TimeOfDayDomain); ok { - return x.TimeOfDayDomain - } - return nil -} - -type isEntitySpec_PresenceConstraints interface { - isEntitySpec_PresenceConstraints() -} - -type EntitySpec_Presence struct { - // Constraints on the presence of this feature in the examples. - Presence *v0.FeaturePresence `protobuf:"bytes,3,opt,name=presence,proto3,oneof"` -} - -type EntitySpec_GroupPresence struct { - // Only used in the context of a "group" context, e.g., inside a sequence. - GroupPresence *v0.FeaturePresenceWithinGroup `protobuf:"bytes,4,opt,name=group_presence,json=groupPresence,proto3,oneof"` -} - -func (*EntitySpec_Presence) isEntitySpec_PresenceConstraints() {} - -func (*EntitySpec_GroupPresence) isEntitySpec_PresenceConstraints() {} - -type isEntitySpec_ShapeType interface { - isEntitySpec_ShapeType() -} - -type EntitySpec_Shape struct { - // The feature has a fixed shape corresponding to a multi-dimensional - // tensor. - Shape *v0.FixedShape `protobuf:"bytes,5,opt,name=shape,proto3,oneof"` -} - -type EntitySpec_ValueCount struct { - // The feature doesn't have a well defined shape. All we know are limits on - // the minimum and maximum number of values. - ValueCount *v0.ValueCount `protobuf:"bytes,6,opt,name=value_count,json=valueCount,proto3,oneof"` -} - -func (*EntitySpec_Shape) isEntitySpec_ShapeType() {} - -func (*EntitySpec_ValueCount) isEntitySpec_ShapeType() {} - -type isEntitySpec_DomainInfo interface { - isEntitySpec_DomainInfo() -} - -type EntitySpec_Domain struct { - // Reference to a domain defined at the schema level. - Domain string `protobuf:"bytes,7,opt,name=domain,proto3,oneof"` -} - -type EntitySpec_IntDomain struct { - // Inline definitions of domains. - IntDomain *v0.IntDomain `protobuf:"bytes,8,opt,name=int_domain,json=intDomain,proto3,oneof"` -} - -type EntitySpec_FloatDomain struct { - FloatDomain *v0.FloatDomain `protobuf:"bytes,9,opt,name=float_domain,json=floatDomain,proto3,oneof"` -} - -type EntitySpec_StringDomain struct { - StringDomain *v0.StringDomain `protobuf:"bytes,10,opt,name=string_domain,json=stringDomain,proto3,oneof"` -} - -type EntitySpec_BoolDomain struct { - BoolDomain *v0.BoolDomain `protobuf:"bytes,11,opt,name=bool_domain,json=boolDomain,proto3,oneof"` -} - -type EntitySpec_StructDomain struct { - StructDomain *v0.StructDomain `protobuf:"bytes,12,opt,name=struct_domain,json=structDomain,proto3,oneof"` -} - -type EntitySpec_NaturalLanguageDomain struct { - // Supported semantic domains. - NaturalLanguageDomain *v0.NaturalLanguageDomain `protobuf:"bytes,13,opt,name=natural_language_domain,json=naturalLanguageDomain,proto3,oneof"` -} - -type EntitySpec_ImageDomain struct { - ImageDomain *v0.ImageDomain `protobuf:"bytes,14,opt,name=image_domain,json=imageDomain,proto3,oneof"` -} - -type EntitySpec_MidDomain struct { - MidDomain *v0.MIDDomain `protobuf:"bytes,15,opt,name=mid_domain,json=midDomain,proto3,oneof"` -} - -type EntitySpec_UrlDomain struct { - UrlDomain *v0.URLDomain `protobuf:"bytes,16,opt,name=url_domain,json=urlDomain,proto3,oneof"` -} - -type EntitySpec_TimeDomain struct { - TimeDomain *v0.TimeDomain `protobuf:"bytes,17,opt,name=time_domain,json=timeDomain,proto3,oneof"` -} - -type EntitySpec_TimeOfDayDomain struct { - TimeOfDayDomain *v0.TimeOfDayDomain `protobuf:"bytes,18,opt,name=time_of_day_domain,json=timeOfDayDomain,proto3,oneof"` -} - -func (*EntitySpec_Domain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_IntDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_FloatDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_StringDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_BoolDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_StructDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_NaturalLanguageDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_ImageDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_MidDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_UrlDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_TimeDomain) isEntitySpec_DomainInfo() {} - -func (*EntitySpec_TimeOfDayDomain) isEntitySpec_DomainInfo() {} - type FeatureSpec struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache @@ -629,6 +352,8 @@ type FeatureSpec struct { // *FeatureSpec_TimeDomain // *FeatureSpec_TimeOfDayDomain DomainInfo isFeatureSpec_DomainInfo `protobuf_oneof:"domain_info"` + // Labels for user defined metadata on a feature + Labels map[string]string `protobuf:"bytes,19,rep,name=labels,proto3" json:"labels,omitempty" protobuf_key:"bytes,1,opt,name=key,proto3" protobuf_val:"bytes,2,opt,name=value,proto3"` } func (x *FeatureSpec) Reset() { @@ -810,6 +535,13 @@ func (x *FeatureSpec) GetTimeOfDayDomain() *v0.TimeOfDayDomain { return nil } +func (x *FeatureSpec) GetLabels() map[string]string { + if x != nil { + return x.Labels + } + return nil +} + type isFeatureSpec_PresenceConstraints interface { isFeatureSpec_PresenceConstraints() } @@ -1011,30 +743,42 @@ var file_feast_core_FeatureSet_proto_rawDesc = []byte{ 0x04, 0x73, 0x70, 0x65, 0x63, 0x12, 0x2e, 0x0a, 0x04, 0x6d, 0x65, 0x74, 0x61, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x4d, 0x65, 0x74, 0x61, 0x52, - 0x04, 0x6d, 0x65, 0x74, 0x61, 0x22, 0xa1, 0x02, 0x0a, 0x0e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, + 0x04, 0x6d, 0x65, 0x74, 0x61, 0x22, 0x82, 0x03, 0x0a, 0x0e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x53, 0x70, 0x65, 0x63, 0x12, 0x18, 0x0a, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x18, 0x07, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, - 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, - 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, - 0x12, 0x32, 0x0a, 0x08, 0x65, 0x6e, 0x74, 0x69, 0x74, 0x69, 0x65, 0x73, 0x18, 0x03, 0x20, 0x03, - 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, - 0x45, 0x6e, 0x74, 0x69, 0x74, 0x79, 0x53, 0x70, 0x65, 0x63, 0x52, 0x08, 0x65, 0x6e, 0x74, 0x69, - 0x74, 0x69, 0x65, 0x73, 0x12, 0x33, 0x0a, 0x08, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, - 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x17, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, - 0x6f, 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x70, 0x65, 0x63, 0x52, - 0x08, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x12, 0x32, 0x0a, 0x07, 0x6d, 0x61, 0x78, - 0x5f, 0x61, 0x67, 0x65, 0x18, 0x05, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x67, 0x6f, 0x6f, - 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x44, 0x75, 0x72, - 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x06, 0x6d, 0x61, 0x78, 0x41, 0x67, 0x65, 0x12, 0x2a, 0x0a, - 0x06, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x06, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x12, 0x2e, - 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x6f, 0x75, 0x72, 0x63, - 0x65, 0x52, 0x06, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x22, 0x9b, 0x0a, 0x0a, 0x0a, 0x45, 0x6e, + 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x32, 0x0a, 0x08, 0x65, 0x6e, 0x74, 0x69, 0x74, 0x69, + 0x65, 0x73, 0x18, 0x03, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x45, 0x6e, 0x74, 0x69, 0x74, 0x79, 0x53, 0x70, 0x65, 0x63, + 0x52, 0x08, 0x65, 0x6e, 0x74, 0x69, 0x74, 0x69, 0x65, 0x73, 0x12, 0x33, 0x0a, 0x08, 0x66, 0x65, + 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x17, 0x2e, 0x66, + 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, + 0x65, 0x53, 0x70, 0x65, 0x63, 0x52, 0x08, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x12, + 0x32, 0x0a, 0x07, 0x6d, 0x61, 0x78, 0x5f, 0x61, 0x67, 0x65, 0x18, 0x05, 0x20, 0x01, 0x28, 0x0b, + 0x32, 0x19, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, + 0x75, 0x66, 0x2e, 0x44, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x06, 0x6d, 0x61, 0x78, + 0x41, 0x67, 0x65, 0x12, 0x2a, 0x0a, 0x06, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x06, 0x20, + 0x01, 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, + 0x2e, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x52, 0x06, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x12, + 0x3e, 0x0a, 0x06, 0x6c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x18, 0x08, 0x20, 0x03, 0x28, 0x0b, 0x32, + 0x26, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, + 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x53, 0x70, 0x65, 0x63, 0x2e, 0x4c, 0x61, 0x62, 0x65, + 0x6c, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x06, 0x6c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x1a, + 0x39, 0x0a, 0x0b, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, + 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, + 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x22, 0x5c, 0x0a, 0x0a, 0x45, 0x6e, 0x74, 0x69, 0x74, 0x79, 0x53, 0x70, 0x65, 0x63, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x3a, 0x0a, 0x0a, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x1b, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x54, 0x79, 0x70, 0x65, 0x2e, 0x45, 0x6e, 0x75, 0x6d, 0x52, 0x09, 0x76, + 0x61, 0x6c, 0x75, 0x65, 0x54, 0x79, 0x70, 0x65, 0x22, 0x94, 0x0b, 0x0a, 0x0b, 0x46, 0x65, 0x61, + 0x74, 0x75, 0x72, 0x65, 0x53, 0x70, 0x65, 0x63, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, + 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x3a, 0x0a, 0x0a, + 0x76, 0x61, 0x6c, 0x75, 0x65, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, + 0x32, 0x1b, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x56, + 0x61, 0x6c, 0x75, 0x65, 0x54, 0x79, 0x70, 0x65, 0x2e, 0x45, 0x6e, 0x75, 0x6d, 0x52, 0x09, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x54, 0x79, 0x70, 0x65, 0x12, 0x45, 0x0a, 0x08, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x27, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, @@ -1108,111 +852,37 @@ var file_feast_core_FeatureSet_proto_rawDesc = []byte{ 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x4f, 0x66, 0x44, 0x61, 0x79, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0f, 0x74, 0x69, 0x6d, 0x65, 0x4f, 0x66, 0x44, 0x61, 0x79, 0x44, 0x6f, 0x6d, 0x61, - 0x69, 0x6e, 0x42, 0x16, 0x0a, 0x14, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x5f, 0x63, - 0x6f, 0x6e, 0x73, 0x74, 0x72, 0x61, 0x69, 0x6e, 0x74, 0x73, 0x42, 0x0c, 0x0a, 0x0a, 0x73, 0x68, - 0x61, 0x70, 0x65, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x42, 0x0d, 0x0a, 0x0b, 0x64, 0x6f, 0x6d, 0x61, - 0x69, 0x6e, 0x5f, 0x69, 0x6e, 0x66, 0x6f, 0x22, 0x9c, 0x0a, 0x0a, 0x0b, 0x46, 0x65, 0x61, 0x74, - 0x75, 0x72, 0x65, 0x53, 0x70, 0x65, 0x63, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, - 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x3a, 0x0a, 0x0a, 0x76, - 0x61, 0x6c, 0x75, 0x65, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, - 0x1b, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x56, 0x61, - 0x6c, 0x75, 0x65, 0x54, 0x79, 0x70, 0x65, 0x2e, 0x45, 0x6e, 0x75, 0x6d, 0x52, 0x09, 0x76, 0x61, - 0x6c, 0x75, 0x65, 0x54, 0x79, 0x70, 0x65, 0x12, 0x45, 0x0a, 0x08, 0x70, 0x72, 0x65, 0x73, 0x65, - 0x6e, 0x63, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x27, 0x2e, 0x74, 0x65, 0x6e, 0x73, - 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, - 0x76, 0x30, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x50, 0x72, 0x65, 0x73, 0x65, 0x6e, - 0x63, 0x65, 0x48, 0x00, 0x52, 0x08, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x12, 0x5b, - 0x0a, 0x0e, 0x67, 0x72, 0x6f, 0x75, 0x70, 0x5f, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, - 0x18, 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x32, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, - 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, - 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x50, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x57, - 0x69, 0x74, 0x68, 0x69, 0x6e, 0x47, 0x72, 0x6f, 0x75, 0x70, 0x48, 0x00, 0x52, 0x0d, 0x67, 0x72, - 0x6f, 0x75, 0x70, 0x50, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x12, 0x3a, 0x0a, 0x05, 0x73, - 0x68, 0x61, 0x70, 0x65, 0x18, 0x05, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, 0x65, 0x6e, - 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, - 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x69, 0x78, 0x65, 0x64, 0x53, 0x68, 0x61, 0x70, 0x65, 0x48, 0x01, - 0x52, 0x05, 0x73, 0x68, 0x61, 0x70, 0x65, 0x12, 0x45, 0x0a, 0x0b, 0x76, 0x61, 0x6c, 0x75, 0x65, - 0x5f, 0x63, 0x6f, 0x75, 0x6e, 0x74, 0x18, 0x06, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, - 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, - 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, - 0x48, 0x01, 0x52, 0x0a, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x12, 0x18, - 0x0a, 0x06, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x07, 0x20, 0x01, 0x28, 0x09, 0x48, 0x02, - 0x52, 0x06, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x42, 0x0a, 0x0a, 0x69, 0x6e, 0x74, 0x5f, - 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x08, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x21, 0x2e, 0x74, - 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, - 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x49, 0x6e, 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, - 0x02, 0x52, 0x09, 0x69, 0x6e, 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x48, 0x0a, 0x0c, - 0x66, 0x6c, 0x6f, 0x61, 0x74, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x09, 0x20, 0x01, - 0x28, 0x0b, 0x32, 0x23, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, - 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x46, 0x6c, 0x6f, 0x61, - 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0b, 0x66, 0x6c, 0x6f, 0x61, 0x74, - 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x4b, 0x0a, 0x0d, 0x73, 0x74, 0x72, 0x69, 0x6e, 0x67, - 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x0a, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x24, 0x2e, - 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, - 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x53, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x44, 0x6f, 0x6d, - 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0c, 0x73, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x44, 0x6f, 0x6d, - 0x61, 0x69, 0x6e, 0x12, 0x45, 0x0a, 0x0b, 0x62, 0x6f, 0x6f, 0x6c, 0x5f, 0x64, 0x6f, 0x6d, 0x61, - 0x69, 0x6e, 0x18, 0x0b, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, - 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, - 0x30, 0x2e, 0x42, 0x6f, 0x6f, 0x6c, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0a, - 0x62, 0x6f, 0x6f, 0x6c, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x4b, 0x0a, 0x0d, 0x73, 0x74, - 0x72, 0x75, 0x63, 0x74, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x0c, 0x20, 0x01, 0x28, - 0x0b, 0x32, 0x24, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, - 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x53, 0x74, 0x72, 0x75, 0x63, - 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0c, 0x73, 0x74, 0x72, 0x75, 0x63, - 0x74, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x67, 0x0a, 0x17, 0x6e, 0x61, 0x74, 0x75, 0x72, - 0x61, 0x6c, 0x5f, 0x6c, 0x61, 0x6e, 0x67, 0x75, 0x61, 0x67, 0x65, 0x5f, 0x64, 0x6f, 0x6d, 0x61, - 0x69, 0x6e, 0x18, 0x0d, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x2d, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, - 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, - 0x30, 0x2e, 0x4e, 0x61, 0x74, 0x75, 0x72, 0x61, 0x6c, 0x4c, 0x61, 0x6e, 0x67, 0x75, 0x61, 0x67, - 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x15, 0x6e, 0x61, 0x74, 0x75, 0x72, - 0x61, 0x6c, 0x4c, 0x61, 0x6e, 0x67, 0x75, 0x61, 0x67, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, - 0x12, 0x48, 0x0a, 0x0c, 0x69, 0x6d, 0x61, 0x67, 0x65, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, - 0x18, 0x0e, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x23, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, - 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, - 0x49, 0x6d, 0x61, 0x67, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0b, 0x69, - 0x6d, 0x61, 0x67, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x42, 0x0a, 0x0a, 0x6d, 0x69, - 0x64, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x0f, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x21, - 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, - 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x4d, 0x49, 0x44, 0x44, 0x6f, 0x6d, 0x61, 0x69, - 0x6e, 0x48, 0x02, 0x52, 0x09, 0x6d, 0x69, 0x64, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x42, - 0x0a, 0x0a, 0x75, 0x72, 0x6c, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, 0x10, 0x20, 0x01, - 0x28, 0x0b, 0x32, 0x21, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, - 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x55, 0x52, 0x4c, 0x44, - 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x09, 0x75, 0x72, 0x6c, 0x44, 0x6f, 0x6d, 0x61, - 0x69, 0x6e, 0x12, 0x45, 0x0a, 0x0b, 0x74, 0x69, 0x6d, 0x65, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, - 0x6e, 0x18, 0x11, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x22, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, - 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, - 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, 0x52, 0x0a, 0x74, - 0x69, 0x6d, 0x65, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x12, 0x56, 0x0a, 0x12, 0x74, 0x69, 0x6d, - 0x65, 0x5f, 0x6f, 0x66, 0x5f, 0x64, 0x61, 0x79, 0x5f, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x18, - 0x12, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x27, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, - 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x2e, 0x54, - 0x69, 0x6d, 0x65, 0x4f, 0x66, 0x44, 0x61, 0x79, 0x44, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x48, 0x02, - 0x52, 0x0f, 0x74, 0x69, 0x6d, 0x65, 0x4f, 0x66, 0x44, 0x61, 0x79, 0x44, 0x6f, 0x6d, 0x61, 0x69, - 0x6e, 0x42, 0x16, 0x0a, 0x14, 0x70, 0x72, 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x5f, 0x63, 0x6f, - 0x6e, 0x73, 0x74, 0x72, 0x61, 0x69, 0x6e, 0x74, 0x73, 0x42, 0x0c, 0x0a, 0x0a, 0x73, 0x68, 0x61, - 0x70, 0x65, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x42, 0x0d, 0x0a, 0x0b, 0x64, 0x6f, 0x6d, 0x61, 0x69, - 0x6e, 0x5f, 0x69, 0x6e, 0x66, 0x6f, 0x22, 0x8f, 0x01, 0x0a, 0x0e, 0x46, 0x65, 0x61, 0x74, 0x75, - 0x72, 0x65, 0x53, 0x65, 0x74, 0x4d, 0x65, 0x74, 0x61, 0x12, 0x47, 0x0a, 0x11, 0x63, 0x72, 0x65, - 0x61, 0x74, 0x65, 0x64, 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x18, 0x01, - 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, - 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, - 0x52, 0x10, 0x63, 0x72, 0x65, 0x61, 0x74, 0x65, 0x64, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, - 0x6d, 0x70, 0x12, 0x34, 0x0a, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x18, 0x02, 0x20, 0x01, - 0x28, 0x0e, 0x32, 0x1c, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, - 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, - 0x52, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x2a, 0x4c, 0x0a, 0x10, 0x46, 0x65, 0x61, 0x74, - 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x12, 0x0a, 0x0e, - 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, 0x5f, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, - 0x12, 0x12, 0x0a, 0x0e, 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, 0x5f, 0x50, 0x45, 0x4e, 0x44, 0x49, - 0x4e, 0x47, 0x10, 0x01, 0x12, 0x10, 0x0a, 0x0c, 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, 0x5f, 0x52, - 0x45, 0x41, 0x44, 0x59, 0x10, 0x02, 0x42, 0x4e, 0x0a, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, - 0x63, 0x6f, 0x72, 0x65, 0x42, 0x0f, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, - 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x2f, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, - 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x64, - 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, 0x61, 0x73, - 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, + 0x69, 0x6e, 0x12, 0x3b, 0x0a, 0x06, 0x6c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x18, 0x13, 0x20, 0x03, + 0x28, 0x0b, 0x32, 0x23, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, + 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x70, 0x65, 0x63, 0x2e, 0x4c, 0x61, 0x62, 0x65, + 0x6c, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x06, 0x6c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x1a, + 0x39, 0x0a, 0x0b, 0x4c, 0x61, 0x62, 0x65, 0x6c, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, + 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, + 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x42, 0x16, 0x0a, 0x14, 0x70, 0x72, + 0x65, 0x73, 0x65, 0x6e, 0x63, 0x65, 0x5f, 0x63, 0x6f, 0x6e, 0x73, 0x74, 0x72, 0x61, 0x69, 0x6e, + 0x74, 0x73, 0x42, 0x0c, 0x0a, 0x0a, 0x73, 0x68, 0x61, 0x70, 0x65, 0x5f, 0x74, 0x79, 0x70, 0x65, + 0x42, 0x0d, 0x0a, 0x0b, 0x64, 0x6f, 0x6d, 0x61, 0x69, 0x6e, 0x5f, 0x69, 0x6e, 0x66, 0x6f, 0x22, + 0x8f, 0x01, 0x0a, 0x0e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x4d, 0x65, + 0x74, 0x61, 0x12, 0x47, 0x0a, 0x11, 0x63, 0x72, 0x65, 0x61, 0x74, 0x65, 0x64, 0x5f, 0x74, 0x69, + 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, + 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, + 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x10, 0x63, 0x72, 0x65, 0x61, 0x74, + 0x65, 0x64, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x12, 0x34, 0x0a, 0x06, 0x73, + 0x74, 0x61, 0x74, 0x75, 0x73, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x1c, 0x2e, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, + 0x53, 0x65, 0x74, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, + 0x73, 0x2a, 0x4c, 0x0a, 0x10, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x53, + 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x12, 0x0a, 0x0e, 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, 0x5f, + 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, 0x12, 0x12, 0x0a, 0x0e, 0x53, 0x54, 0x41, + 0x54, 0x55, 0x53, 0x5f, 0x50, 0x45, 0x4e, 0x44, 0x49, 0x4e, 0x47, 0x10, 0x01, 0x12, 0x10, 0x0a, + 0x0c, 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, 0x5f, 0x52, 0x45, 0x41, 0x44, 0x59, 0x10, 0x02, 0x42, + 0x4e, 0x0a, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x42, 0x0f, 0x46, + 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x2f, + 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, + 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, + 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x62, + 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, } var ( @@ -1228,7 +898,7 @@ func file_feast_core_FeatureSet_proto_rawDescGZIP() []byte { } var file_feast_core_FeatureSet_proto_enumTypes = make([]protoimpl.EnumInfo, 1) -var file_feast_core_FeatureSet_proto_msgTypes = make([]protoimpl.MessageInfo, 5) +var file_feast_core_FeatureSet_proto_msgTypes = make([]protoimpl.MessageInfo, 7) var file_feast_core_FeatureSet_proto_goTypes = []interface{}{ (FeatureSetStatus)(0), // 0: feast.core.FeatureSetStatus (*FeatureSet)(nil), // 1: feast.core.FeatureSet @@ -1236,72 +906,61 @@ var file_feast_core_FeatureSet_proto_goTypes = []interface{}{ (*EntitySpec)(nil), // 3: feast.core.EntitySpec (*FeatureSpec)(nil), // 4: feast.core.FeatureSpec (*FeatureSetMeta)(nil), // 5: feast.core.FeatureSetMeta - (*duration.Duration)(nil), // 6: google.protobuf.Duration - (*Source)(nil), // 7: feast.core.Source - (types.ValueType_Enum)(0), // 8: feast.types.ValueType.Enum - (*v0.FeaturePresence)(nil), // 9: tensorflow.metadata.v0.FeaturePresence - (*v0.FeaturePresenceWithinGroup)(nil), // 10: tensorflow.metadata.v0.FeaturePresenceWithinGroup - (*v0.FixedShape)(nil), // 11: tensorflow.metadata.v0.FixedShape - (*v0.ValueCount)(nil), // 12: tensorflow.metadata.v0.ValueCount - (*v0.IntDomain)(nil), // 13: tensorflow.metadata.v0.IntDomain - (*v0.FloatDomain)(nil), // 14: tensorflow.metadata.v0.FloatDomain - (*v0.StringDomain)(nil), // 15: tensorflow.metadata.v0.StringDomain - (*v0.BoolDomain)(nil), // 16: tensorflow.metadata.v0.BoolDomain - (*v0.StructDomain)(nil), // 17: tensorflow.metadata.v0.StructDomain - (*v0.NaturalLanguageDomain)(nil), // 18: tensorflow.metadata.v0.NaturalLanguageDomain - (*v0.ImageDomain)(nil), // 19: tensorflow.metadata.v0.ImageDomain - (*v0.MIDDomain)(nil), // 20: tensorflow.metadata.v0.MIDDomain - (*v0.URLDomain)(nil), // 21: tensorflow.metadata.v0.URLDomain - (*v0.TimeDomain)(nil), // 22: tensorflow.metadata.v0.TimeDomain - (*v0.TimeOfDayDomain)(nil), // 23: tensorflow.metadata.v0.TimeOfDayDomain - (*timestamp.Timestamp)(nil), // 24: google.protobuf.Timestamp + nil, // 6: feast.core.FeatureSetSpec.LabelsEntry + nil, // 7: feast.core.FeatureSpec.LabelsEntry + (*duration.Duration)(nil), // 8: google.protobuf.Duration + (*Source)(nil), // 9: feast.core.Source + (types.ValueType_Enum)(0), // 10: feast.types.ValueType.Enum + (*v0.FeaturePresence)(nil), // 11: tensorflow.metadata.v0.FeaturePresence + (*v0.FeaturePresenceWithinGroup)(nil), // 12: tensorflow.metadata.v0.FeaturePresenceWithinGroup + (*v0.FixedShape)(nil), // 13: tensorflow.metadata.v0.FixedShape + (*v0.ValueCount)(nil), // 14: tensorflow.metadata.v0.ValueCount + (*v0.IntDomain)(nil), // 15: tensorflow.metadata.v0.IntDomain + (*v0.FloatDomain)(nil), // 16: tensorflow.metadata.v0.FloatDomain + (*v0.StringDomain)(nil), // 17: tensorflow.metadata.v0.StringDomain + (*v0.BoolDomain)(nil), // 18: tensorflow.metadata.v0.BoolDomain + (*v0.StructDomain)(nil), // 19: tensorflow.metadata.v0.StructDomain + (*v0.NaturalLanguageDomain)(nil), // 20: tensorflow.metadata.v0.NaturalLanguageDomain + (*v0.ImageDomain)(nil), // 21: tensorflow.metadata.v0.ImageDomain + (*v0.MIDDomain)(nil), // 22: tensorflow.metadata.v0.MIDDomain + (*v0.URLDomain)(nil), // 23: tensorflow.metadata.v0.URLDomain + (*v0.TimeDomain)(nil), // 24: tensorflow.metadata.v0.TimeDomain + (*v0.TimeOfDayDomain)(nil), // 25: tensorflow.metadata.v0.TimeOfDayDomain + (*timestamp.Timestamp)(nil), // 26: google.protobuf.Timestamp } var file_feast_core_FeatureSet_proto_depIdxs = []int32{ 2, // 0: feast.core.FeatureSet.spec:type_name -> feast.core.FeatureSetSpec 5, // 1: feast.core.FeatureSet.meta:type_name -> feast.core.FeatureSetMeta 3, // 2: feast.core.FeatureSetSpec.entities:type_name -> feast.core.EntitySpec 4, // 3: feast.core.FeatureSetSpec.features:type_name -> feast.core.FeatureSpec - 6, // 4: feast.core.FeatureSetSpec.max_age:type_name -> google.protobuf.Duration - 7, // 5: feast.core.FeatureSetSpec.source:type_name -> feast.core.Source - 8, // 6: feast.core.EntitySpec.value_type:type_name -> feast.types.ValueType.Enum - 9, // 7: feast.core.EntitySpec.presence:type_name -> tensorflow.metadata.v0.FeaturePresence - 10, // 8: feast.core.EntitySpec.group_presence:type_name -> tensorflow.metadata.v0.FeaturePresenceWithinGroup - 11, // 9: feast.core.EntitySpec.shape:type_name -> tensorflow.metadata.v0.FixedShape - 12, // 10: feast.core.EntitySpec.value_count:type_name -> tensorflow.metadata.v0.ValueCount - 13, // 11: feast.core.EntitySpec.int_domain:type_name -> tensorflow.metadata.v0.IntDomain - 14, // 12: feast.core.EntitySpec.float_domain:type_name -> tensorflow.metadata.v0.FloatDomain - 15, // 13: feast.core.EntitySpec.string_domain:type_name -> tensorflow.metadata.v0.StringDomain - 16, // 14: feast.core.EntitySpec.bool_domain:type_name -> tensorflow.metadata.v0.BoolDomain - 17, // 15: feast.core.EntitySpec.struct_domain:type_name -> tensorflow.metadata.v0.StructDomain - 18, // 16: feast.core.EntitySpec.natural_language_domain:type_name -> tensorflow.metadata.v0.NaturalLanguageDomain - 19, // 17: feast.core.EntitySpec.image_domain:type_name -> tensorflow.metadata.v0.ImageDomain - 20, // 18: feast.core.EntitySpec.mid_domain:type_name -> tensorflow.metadata.v0.MIDDomain - 21, // 19: feast.core.EntitySpec.url_domain:type_name -> tensorflow.metadata.v0.URLDomain - 22, // 20: feast.core.EntitySpec.time_domain:type_name -> tensorflow.metadata.v0.TimeDomain - 23, // 21: feast.core.EntitySpec.time_of_day_domain:type_name -> tensorflow.metadata.v0.TimeOfDayDomain - 8, // 22: feast.core.FeatureSpec.value_type:type_name -> feast.types.ValueType.Enum - 9, // 23: feast.core.FeatureSpec.presence:type_name -> tensorflow.metadata.v0.FeaturePresence - 10, // 24: feast.core.FeatureSpec.group_presence:type_name -> tensorflow.metadata.v0.FeaturePresenceWithinGroup - 11, // 25: feast.core.FeatureSpec.shape:type_name -> tensorflow.metadata.v0.FixedShape - 12, // 26: feast.core.FeatureSpec.value_count:type_name -> tensorflow.metadata.v0.ValueCount - 13, // 27: feast.core.FeatureSpec.int_domain:type_name -> tensorflow.metadata.v0.IntDomain - 14, // 28: feast.core.FeatureSpec.float_domain:type_name -> tensorflow.metadata.v0.FloatDomain - 15, // 29: feast.core.FeatureSpec.string_domain:type_name -> tensorflow.metadata.v0.StringDomain - 16, // 30: feast.core.FeatureSpec.bool_domain:type_name -> tensorflow.metadata.v0.BoolDomain - 17, // 31: feast.core.FeatureSpec.struct_domain:type_name -> tensorflow.metadata.v0.StructDomain - 18, // 32: feast.core.FeatureSpec.natural_language_domain:type_name -> tensorflow.metadata.v0.NaturalLanguageDomain - 19, // 33: feast.core.FeatureSpec.image_domain:type_name -> tensorflow.metadata.v0.ImageDomain - 20, // 34: feast.core.FeatureSpec.mid_domain:type_name -> tensorflow.metadata.v0.MIDDomain - 21, // 35: feast.core.FeatureSpec.url_domain:type_name -> tensorflow.metadata.v0.URLDomain - 22, // 36: feast.core.FeatureSpec.time_domain:type_name -> tensorflow.metadata.v0.TimeDomain - 23, // 37: feast.core.FeatureSpec.time_of_day_domain:type_name -> tensorflow.metadata.v0.TimeOfDayDomain - 24, // 38: feast.core.FeatureSetMeta.created_timestamp:type_name -> google.protobuf.Timestamp - 0, // 39: feast.core.FeatureSetMeta.status:type_name -> feast.core.FeatureSetStatus - 40, // [40:40] is the sub-list for method output_type - 40, // [40:40] is the sub-list for method input_type - 40, // [40:40] is the sub-list for extension type_name - 40, // [40:40] is the sub-list for extension extendee - 0, // [0:40] is the sub-list for field type_name + 8, // 4: feast.core.FeatureSetSpec.max_age:type_name -> google.protobuf.Duration + 9, // 5: feast.core.FeatureSetSpec.source:type_name -> feast.core.Source + 6, // 6: feast.core.FeatureSetSpec.labels:type_name -> feast.core.FeatureSetSpec.LabelsEntry + 10, // 7: feast.core.EntitySpec.value_type:type_name -> feast.types.ValueType.Enum + 10, // 8: feast.core.FeatureSpec.value_type:type_name -> feast.types.ValueType.Enum + 11, // 9: feast.core.FeatureSpec.presence:type_name -> tensorflow.metadata.v0.FeaturePresence + 12, // 10: feast.core.FeatureSpec.group_presence:type_name -> tensorflow.metadata.v0.FeaturePresenceWithinGroup + 13, // 11: feast.core.FeatureSpec.shape:type_name -> tensorflow.metadata.v0.FixedShape + 14, // 12: feast.core.FeatureSpec.value_count:type_name -> tensorflow.metadata.v0.ValueCount + 15, // 13: feast.core.FeatureSpec.int_domain:type_name -> tensorflow.metadata.v0.IntDomain + 16, // 14: feast.core.FeatureSpec.float_domain:type_name -> tensorflow.metadata.v0.FloatDomain + 17, // 15: feast.core.FeatureSpec.string_domain:type_name -> tensorflow.metadata.v0.StringDomain + 18, // 16: feast.core.FeatureSpec.bool_domain:type_name -> tensorflow.metadata.v0.BoolDomain + 19, // 17: feast.core.FeatureSpec.struct_domain:type_name -> tensorflow.metadata.v0.StructDomain + 20, // 18: feast.core.FeatureSpec.natural_language_domain:type_name -> tensorflow.metadata.v0.NaturalLanguageDomain + 21, // 19: feast.core.FeatureSpec.image_domain:type_name -> tensorflow.metadata.v0.ImageDomain + 22, // 20: feast.core.FeatureSpec.mid_domain:type_name -> tensorflow.metadata.v0.MIDDomain + 23, // 21: feast.core.FeatureSpec.url_domain:type_name -> tensorflow.metadata.v0.URLDomain + 24, // 22: feast.core.FeatureSpec.time_domain:type_name -> tensorflow.metadata.v0.TimeDomain + 25, // 23: feast.core.FeatureSpec.time_of_day_domain:type_name -> tensorflow.metadata.v0.TimeOfDayDomain + 7, // 24: feast.core.FeatureSpec.labels:type_name -> feast.core.FeatureSpec.LabelsEntry + 26, // 25: feast.core.FeatureSetMeta.created_timestamp:type_name -> google.protobuf.Timestamp + 0, // 26: feast.core.FeatureSetMeta.status:type_name -> feast.core.FeatureSetStatus + 27, // [27:27] is the sub-list for method output_type + 27, // [27:27] is the sub-list for method input_type + 27, // [27:27] is the sub-list for extension type_name + 27, // [27:27] is the sub-list for extension extendee + 0, // [0:27] is the sub-list for field type_name } func init() { file_feast_core_FeatureSet_proto_init() } @@ -1372,24 +1031,6 @@ func file_feast_core_FeatureSet_proto_init() { } } } - file_feast_core_FeatureSet_proto_msgTypes[2].OneofWrappers = []interface{}{ - (*EntitySpec_Presence)(nil), - (*EntitySpec_GroupPresence)(nil), - (*EntitySpec_Shape)(nil), - (*EntitySpec_ValueCount)(nil), - (*EntitySpec_Domain)(nil), - (*EntitySpec_IntDomain)(nil), - (*EntitySpec_FloatDomain)(nil), - (*EntitySpec_StringDomain)(nil), - (*EntitySpec_BoolDomain)(nil), - (*EntitySpec_StructDomain)(nil), - (*EntitySpec_NaturalLanguageDomain)(nil), - (*EntitySpec_ImageDomain)(nil), - (*EntitySpec_MidDomain)(nil), - (*EntitySpec_UrlDomain)(nil), - (*EntitySpec_TimeDomain)(nil), - (*EntitySpec_TimeOfDayDomain)(nil), - } file_feast_core_FeatureSet_proto_msgTypes[3].OneofWrappers = []interface{}{ (*FeatureSpec_Presence)(nil), (*FeatureSpec_GroupPresence)(nil), @@ -1414,7 +1055,7 @@ func file_feast_core_FeatureSet_proto_init() { GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: file_feast_core_FeatureSet_proto_rawDesc, NumEnums: 1, - NumMessages: 5, + NumMessages: 7, NumExtensions: 0, NumServices: 0, }, diff --git a/sdk/go/protos/feast/core/FeatureSetReference.pb.go b/sdk/go/protos/feast/core/FeatureSetReference.pb.go index 52a63b6a8c1..1667565bd23 100644 --- a/sdk/go/protos/feast/core/FeatureSetReference.pb.go +++ b/sdk/go/protos/feast/core/FeatureSetReference.pb.go @@ -51,8 +51,6 @@ type FeatureSetReference struct { Project string `protobuf:"bytes,1,opt,name=project,proto3" json:"project,omitempty"` // Name of the FeatureSet Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"` - // Version no. of the FeatureSet - Version int32 `protobuf:"varint,3,opt,name=version,proto3" json:"version,omitempty"` } func (x *FeatureSetReference) Reset() { @@ -101,32 +99,23 @@ func (x *FeatureSetReference) GetName() string { return "" } -func (x *FeatureSetReference) GetVersion() int32 { - if x != nil { - return x.Version - } - return 0 -} - var File_feast_core_FeatureSetReference_proto protoreflect.FileDescriptor var file_feast_core_FeatureSetReference_proto_rawDesc = []byte{ 0x0a, 0x24, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, - 0x72, 0x65, 0x22, 0x5d, 0x0a, 0x13, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, + 0x72, 0x65, 0x22, 0x43, 0x0a, 0x13, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, - 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, - 0x6f, 0x6e, 0x18, 0x03, 0x20, 0x01, 0x28, 0x05, 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, - 0x6e, 0x42, 0x57, 0x0a, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x42, - 0x18, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, 0x74, 0x52, 0x65, 0x66, 0x65, 0x72, - 0x65, 0x6e, 0x63, 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x2f, 0x67, 0x69, 0x74, 0x68, 0x75, - 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, - 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, - 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, - 0x6f, 0x33, + 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x42, 0x57, 0x0a, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x42, 0x18, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x53, 0x65, + 0x74, 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, + 0x2f, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, + 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, + 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, + 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, } var ( diff --git a/sdk/go/protos/feast/core/Runner.pb.go b/sdk/go/protos/feast/core/Runner.pb.go new file mode 100644 index 00000000000..ae9f7c4e7d3 --- /dev/null +++ b/sdk/go/protos/feast/core/Runner.pb.go @@ -0,0 +1,375 @@ +// +// * Copyright 2020 The Feast Authors +// * +// * Licensed under the Apache License, Version 2.0 (the "License"); +// * you may not use this file except in compliance with the License. +// * You may obtain a copy of the License at +// * +// * https://www.apache.org/licenses/LICENSE-2.0 +// * +// * Unless required by applicable law or agreed to in writing, software +// * distributed under the License is distributed on an "AS IS" BASIS, +// * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// * See the License for the specific language governing permissions and +// * limitations under the License. +// + +// Code generated by protoc-gen-go. DO NOT EDIT. +// versions: +// protoc-gen-go v1.21.0 +// protoc v3.10.0 +// source: feast/core/Runner.proto + +package core + +import ( + proto "github.com/golang/protobuf/proto" + protoreflect "google.golang.org/protobuf/reflect/protoreflect" + protoimpl "google.golang.org/protobuf/runtime/protoimpl" + reflect "reflect" + sync "sync" +) + +const ( + // Verify that this generated code is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(20 - protoimpl.MinVersion) + // Verify that runtime/protoimpl is sufficiently up-to-date. + _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) +) + +// This is a compile-time assertion that a sufficiently up-to-date version +// of the legacy proto package is being used. +const _ = proto.ProtoPackageIsVersion4 + +type DirectRunnerConfigOptions struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + //* + // Controls the amount of target parallelism the DirectRunner will use. + // Defaults to the greater of the number of available processors and 3. Must be a value + // greater than zero. + TargetParallelism int32 `protobuf:"varint,1,opt,name=targetParallelism,proto3" json:"targetParallelism,omitempty"` + // BigQuery table specification, e.g. PROJECT_ID:DATASET_ID.PROJECT_ID + DeadLetterTableSpec string `protobuf:"bytes,2,opt,name=deadLetterTableSpec,proto3" json:"deadLetterTableSpec,omitempty"` +} + +func (x *DirectRunnerConfigOptions) Reset() { + *x = DirectRunnerConfigOptions{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_Runner_proto_msgTypes[0] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *DirectRunnerConfigOptions) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*DirectRunnerConfigOptions) ProtoMessage() {} + +func (x *DirectRunnerConfigOptions) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_Runner_proto_msgTypes[0] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use DirectRunnerConfigOptions.ProtoReflect.Descriptor instead. +func (*DirectRunnerConfigOptions) Descriptor() ([]byte, []int) { + return file_feast_core_Runner_proto_rawDescGZIP(), []int{0} +} + +func (x *DirectRunnerConfigOptions) GetTargetParallelism() int32 { + if x != nil { + return x.TargetParallelism + } + return 0 +} + +func (x *DirectRunnerConfigOptions) GetDeadLetterTableSpec() string { + if x != nil { + return x.DeadLetterTableSpec + } + return "" +} + +type DataflowRunnerConfigOptions struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // Project id to use when launching jobs. + Project string `protobuf:"bytes,1,opt,name=project,proto3" json:"project,omitempty"` + // The Google Compute Engine region for creating Dataflow jobs. + Region string `protobuf:"bytes,2,opt,name=region,proto3" json:"region,omitempty"` + // GCP availability zone for operations. + Zone string `protobuf:"bytes,3,opt,name=zone,proto3" json:"zone,omitempty"` + // Run the job as a specific service account, instead of the default GCE robot. + ServiceAccount string `protobuf:"bytes,4,opt,name=serviceAccount,proto3" json:"serviceAccount,omitempty"` + // GCE network for launching workers. + Network string `protobuf:"bytes,5,opt,name=network,proto3" json:"network,omitempty"` + // GCE subnetwork for launching workers. e.g. regions/asia-east1/subnetworks/mysubnetwork + Subnetwork string `protobuf:"bytes,6,opt,name=subnetwork,proto3" json:"subnetwork,omitempty"` + // Machine type to create Dataflow worker VMs as. + WorkerMachineType string `protobuf:"bytes,7,opt,name=workerMachineType,proto3" json:"workerMachineType,omitempty"` + // The autoscaling algorithm to use for the workerpool. + AutoscalingAlgorithm string `protobuf:"bytes,8,opt,name=autoscalingAlgorithm,proto3" json:"autoscalingAlgorithm,omitempty"` + // Specifies whether worker pools should be started with public IP addresses. + UsePublicIps bool `protobuf:"varint,9,opt,name=usePublicIps,proto3" json:"usePublicIps,omitempty"` + // A pipeline level default location for storing temporary files. Support Google Cloud Storage locations, + // e.g. gs://bucket/object + TempLocation string `protobuf:"bytes,10,opt,name=tempLocation,proto3" json:"tempLocation,omitempty"` + // The maximum number of workers to use for the workerpool. + MaxNumWorkers int32 `protobuf:"varint,11,opt,name=maxNumWorkers,proto3" json:"maxNumWorkers,omitempty"` + // BigQuery table specification, e.g. PROJECT_ID:DATASET_ID.PROJECT_ID + DeadLetterTableSpec string `protobuf:"bytes,12,opt,name=deadLetterTableSpec,proto3" json:"deadLetterTableSpec,omitempty"` +} + +func (x *DataflowRunnerConfigOptions) Reset() { + *x = DataflowRunnerConfigOptions{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_Runner_proto_msgTypes[1] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *DataflowRunnerConfigOptions) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*DataflowRunnerConfigOptions) ProtoMessage() {} + +func (x *DataflowRunnerConfigOptions) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_Runner_proto_msgTypes[1] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use DataflowRunnerConfigOptions.ProtoReflect.Descriptor instead. +func (*DataflowRunnerConfigOptions) Descriptor() ([]byte, []int) { + return file_feast_core_Runner_proto_rawDescGZIP(), []int{1} +} + +func (x *DataflowRunnerConfigOptions) GetProject() string { + if x != nil { + return x.Project + } + return "" +} + +func (x *DataflowRunnerConfigOptions) GetRegion() string { + if x != nil { + return x.Region + } + return "" +} + +func (x *DataflowRunnerConfigOptions) GetZone() string { + if x != nil { + return x.Zone + } + return "" +} + +func (x *DataflowRunnerConfigOptions) GetServiceAccount() string { + if x != nil { + return x.ServiceAccount + } + return "" +} + +func (x *DataflowRunnerConfigOptions) GetNetwork() string { + if x != nil { + return x.Network + } + return "" +} + +func (x *DataflowRunnerConfigOptions) GetSubnetwork() string { + if x != nil { + return x.Subnetwork + } + return "" +} + +func (x *DataflowRunnerConfigOptions) GetWorkerMachineType() string { + if x != nil { + return x.WorkerMachineType + } + return "" +} + +func (x *DataflowRunnerConfigOptions) GetAutoscalingAlgorithm() string { + if x != nil { + return x.AutoscalingAlgorithm + } + return "" +} + +func (x *DataflowRunnerConfigOptions) GetUsePublicIps() bool { + if x != nil { + return x.UsePublicIps + } + return false +} + +func (x *DataflowRunnerConfigOptions) GetTempLocation() string { + if x != nil { + return x.TempLocation + } + return "" +} + +func (x *DataflowRunnerConfigOptions) GetMaxNumWorkers() int32 { + if x != nil { + return x.MaxNumWorkers + } + return 0 +} + +func (x *DataflowRunnerConfigOptions) GetDeadLetterTableSpec() string { + if x != nil { + return x.DeadLetterTableSpec + } + return "" +} + +var File_feast_core_Runner_proto protoreflect.FileDescriptor + +var file_feast_core_Runner_proto_rawDesc = []byte{ + 0x0a, 0x17, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x52, 0x75, 0x6e, + 0x6e, 0x65, 0x72, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x22, 0x7b, 0x0a, 0x19, 0x44, 0x69, 0x72, 0x65, 0x63, 0x74, 0x52, + 0x75, 0x6e, 0x6e, 0x65, 0x72, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x4f, 0x70, 0x74, 0x69, 0x6f, + 0x6e, 0x73, 0x12, 0x2c, 0x0a, 0x11, 0x74, 0x61, 0x72, 0x67, 0x65, 0x74, 0x50, 0x61, 0x72, 0x61, + 0x6c, 0x6c, 0x65, 0x6c, 0x69, 0x73, 0x6d, 0x18, 0x01, 0x20, 0x01, 0x28, 0x05, 0x52, 0x11, 0x74, + 0x61, 0x72, 0x67, 0x65, 0x74, 0x50, 0x61, 0x72, 0x61, 0x6c, 0x6c, 0x65, 0x6c, 0x69, 0x73, 0x6d, + 0x12, 0x30, 0x0a, 0x13, 0x64, 0x65, 0x61, 0x64, 0x4c, 0x65, 0x74, 0x74, 0x65, 0x72, 0x54, 0x61, + 0x62, 0x6c, 0x65, 0x53, 0x70, 0x65, 0x63, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x13, 0x64, + 0x65, 0x61, 0x64, 0x4c, 0x65, 0x74, 0x74, 0x65, 0x72, 0x54, 0x61, 0x62, 0x6c, 0x65, 0x53, 0x70, + 0x65, 0x63, 0x22, 0xc7, 0x03, 0x0a, 0x1b, 0x44, 0x61, 0x74, 0x61, 0x66, 0x6c, 0x6f, 0x77, 0x52, + 0x75, 0x6e, 0x6e, 0x65, 0x72, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x4f, 0x70, 0x74, 0x69, 0x6f, + 0x6e, 0x73, 0x12, 0x18, 0x0a, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x18, 0x01, 0x20, + 0x01, 0x28, 0x09, 0x52, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x12, 0x16, 0x0a, 0x06, + 0x72, 0x65, 0x67, 0x69, 0x6f, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x06, 0x72, 0x65, + 0x67, 0x69, 0x6f, 0x6e, 0x12, 0x12, 0x0a, 0x04, 0x7a, 0x6f, 0x6e, 0x65, 0x18, 0x03, 0x20, 0x01, + 0x28, 0x09, 0x52, 0x04, 0x7a, 0x6f, 0x6e, 0x65, 0x12, 0x26, 0x0a, 0x0e, 0x73, 0x65, 0x72, 0x76, + 0x69, 0x63, 0x65, 0x41, 0x63, 0x63, 0x6f, 0x75, 0x6e, 0x74, 0x18, 0x04, 0x20, 0x01, 0x28, 0x09, + 0x52, 0x0e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x41, 0x63, 0x63, 0x6f, 0x75, 0x6e, 0x74, + 0x12, 0x18, 0x0a, 0x07, 0x6e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x18, 0x05, 0x20, 0x01, 0x28, + 0x09, 0x52, 0x07, 0x6e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x12, 0x1e, 0x0a, 0x0a, 0x73, 0x75, + 0x62, 0x6e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x18, 0x06, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0a, + 0x73, 0x75, 0x62, 0x6e, 0x65, 0x74, 0x77, 0x6f, 0x72, 0x6b, 0x12, 0x2c, 0x0a, 0x11, 0x77, 0x6f, + 0x72, 0x6b, 0x65, 0x72, 0x4d, 0x61, 0x63, 0x68, 0x69, 0x6e, 0x65, 0x54, 0x79, 0x70, 0x65, 0x18, + 0x07, 0x20, 0x01, 0x28, 0x09, 0x52, 0x11, 0x77, 0x6f, 0x72, 0x6b, 0x65, 0x72, 0x4d, 0x61, 0x63, + 0x68, 0x69, 0x6e, 0x65, 0x54, 0x79, 0x70, 0x65, 0x12, 0x32, 0x0a, 0x14, 0x61, 0x75, 0x74, 0x6f, + 0x73, 0x63, 0x61, 0x6c, 0x69, 0x6e, 0x67, 0x41, 0x6c, 0x67, 0x6f, 0x72, 0x69, 0x74, 0x68, 0x6d, + 0x18, 0x08, 0x20, 0x01, 0x28, 0x09, 0x52, 0x14, 0x61, 0x75, 0x74, 0x6f, 0x73, 0x63, 0x61, 0x6c, + 0x69, 0x6e, 0x67, 0x41, 0x6c, 0x67, 0x6f, 0x72, 0x69, 0x74, 0x68, 0x6d, 0x12, 0x22, 0x0a, 0x0c, + 0x75, 0x73, 0x65, 0x50, 0x75, 0x62, 0x6c, 0x69, 0x63, 0x49, 0x70, 0x73, 0x18, 0x09, 0x20, 0x01, + 0x28, 0x08, 0x52, 0x0c, 0x75, 0x73, 0x65, 0x50, 0x75, 0x62, 0x6c, 0x69, 0x63, 0x49, 0x70, 0x73, + 0x12, 0x22, 0x0a, 0x0c, 0x74, 0x65, 0x6d, 0x70, 0x4c, 0x6f, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, + 0x18, 0x0a, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0c, 0x74, 0x65, 0x6d, 0x70, 0x4c, 0x6f, 0x63, 0x61, + 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x24, 0x0a, 0x0d, 0x6d, 0x61, 0x78, 0x4e, 0x75, 0x6d, 0x57, 0x6f, + 0x72, 0x6b, 0x65, 0x72, 0x73, 0x18, 0x0b, 0x20, 0x01, 0x28, 0x05, 0x52, 0x0d, 0x6d, 0x61, 0x78, + 0x4e, 0x75, 0x6d, 0x57, 0x6f, 0x72, 0x6b, 0x65, 0x72, 0x73, 0x12, 0x30, 0x0a, 0x13, 0x64, 0x65, + 0x61, 0x64, 0x4c, 0x65, 0x74, 0x74, 0x65, 0x72, 0x54, 0x61, 0x62, 0x6c, 0x65, 0x53, 0x70, 0x65, + 0x63, 0x18, 0x0c, 0x20, 0x01, 0x28, 0x09, 0x52, 0x13, 0x64, 0x65, 0x61, 0x64, 0x4c, 0x65, 0x74, + 0x74, 0x65, 0x72, 0x54, 0x61, 0x62, 0x6c, 0x65, 0x53, 0x70, 0x65, 0x63, 0x42, 0x4a, 0x0a, 0x0a, + 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x42, 0x0b, 0x52, 0x75, 0x6e, 0x6e, + 0x65, 0x72, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x2f, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, + 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, + 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, +} + +var ( + file_feast_core_Runner_proto_rawDescOnce sync.Once + file_feast_core_Runner_proto_rawDescData = file_feast_core_Runner_proto_rawDesc +) + +func file_feast_core_Runner_proto_rawDescGZIP() []byte { + file_feast_core_Runner_proto_rawDescOnce.Do(func() { + file_feast_core_Runner_proto_rawDescData = protoimpl.X.CompressGZIP(file_feast_core_Runner_proto_rawDescData) + }) + return file_feast_core_Runner_proto_rawDescData +} + +var file_feast_core_Runner_proto_msgTypes = make([]protoimpl.MessageInfo, 2) +var file_feast_core_Runner_proto_goTypes = []interface{}{ + (*DirectRunnerConfigOptions)(nil), // 0: feast.core.DirectRunnerConfigOptions + (*DataflowRunnerConfigOptions)(nil), // 1: feast.core.DataflowRunnerConfigOptions +} +var file_feast_core_Runner_proto_depIdxs = []int32{ + 0, // [0:0] is the sub-list for method output_type + 0, // [0:0] is the sub-list for method input_type + 0, // [0:0] is the sub-list for extension type_name + 0, // [0:0] is the sub-list for extension extendee + 0, // [0:0] is the sub-list for field type_name +} + +func init() { file_feast_core_Runner_proto_init() } +func file_feast_core_Runner_proto_init() { + if File_feast_core_Runner_proto != nil { + return + } + if !protoimpl.UnsafeEnabled { + file_feast_core_Runner_proto_msgTypes[0].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*DirectRunnerConfigOptions); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_Runner_proto_msgTypes[1].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*DataflowRunnerConfigOptions); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + } + type x struct{} + out := protoimpl.TypeBuilder{ + File: protoimpl.DescBuilder{ + GoPackagePath: reflect.TypeOf(x{}).PkgPath(), + RawDescriptor: file_feast_core_Runner_proto_rawDesc, + NumEnums: 0, + NumMessages: 2, + NumExtensions: 0, + NumServices: 0, + }, + GoTypes: file_feast_core_Runner_proto_goTypes, + DependencyIndexes: file_feast_core_Runner_proto_depIdxs, + MessageInfos: file_feast_core_Runner_proto_msgTypes, + }.Build() + File_feast_core_Runner_proto = out.File + file_feast_core_Runner_proto_rawDesc = nil + file_feast_core_Runner_proto_goTypes = nil + file_feast_core_Runner_proto_depIdxs = nil +} diff --git a/sdk/go/protos/feast/core/Source.pb.go b/sdk/go/protos/feast/core/Source.pb.go index 30bd2362723..368f50c5acb 100644 --- a/sdk/go/protos/feast/core/Source.pb.go +++ b/sdk/go/protos/feast/core/Source.pb.go @@ -169,10 +169,14 @@ type KafkaSourceConfig struct { sizeCache protoimpl.SizeCache unknownFields protoimpl.UnknownFields - // - bootstrapServers: [comma delimited value of host[:port]] + // Comma separated list of Kafka bootstrap servers. Used for feature sets without a defined source host[:port]] BootstrapServers string `protobuf:"bytes,1,opt,name=bootstrap_servers,json=bootstrapServers,proto3" json:"bootstrap_servers,omitempty"` - // - topics: [Kafka topic name. This value is provisioned by core and should not be set by the user.] + // Kafka topic to use for feature sets without user defined topics Topic string `protobuf:"bytes,2,opt,name=topic,proto3" json:"topic,omitempty"` + // Number of Kafka partitions to to use for managed feature stream. + Partitions int32 `protobuf:"varint,3,opt,name=partitions,proto3" json:"partitions,omitempty"` + // Defines the number of copies of managed feature stream Kafka. + ReplicationFactor int32 `protobuf:"varint,4,opt,name=replicationFactor,proto3" json:"replicationFactor,omitempty"` } func (x *KafkaSourceConfig) Reset() { @@ -221,6 +225,20 @@ func (x *KafkaSourceConfig) GetTopic() string { return "" } +func (x *KafkaSourceConfig) GetPartitions() int32 { + if x != nil { + return x.Partitions + } + return 0 +} + +func (x *KafkaSourceConfig) GetReplicationFactor() int32 { + if x != nil { + return x.ReplicationFactor + } + return 0 +} + var File_feast_core_Source_proto protoreflect.FileDescriptor var file_feast_core_Source_proto_rawDesc = []byte{ @@ -235,20 +253,25 @@ var file_feast_core_Source_proto_rawDesc = []byte{ 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x4b, 0x61, 0x66, 0x6b, 0x61, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x48, 0x00, 0x52, 0x11, 0x6b, 0x61, 0x66, 0x6b, 0x61, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x42, 0x0f, 0x0a, - 0x0d, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x5f, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x22, 0x56, - 0x0a, 0x11, 0x4b, 0x61, 0x66, 0x6b, 0x61, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x43, 0x6f, 0x6e, - 0x66, 0x69, 0x67, 0x12, 0x2b, 0x0a, 0x11, 0x62, 0x6f, 0x6f, 0x74, 0x73, 0x74, 0x72, 0x61, 0x70, - 0x5f, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x73, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x10, - 0x62, 0x6f, 0x6f, 0x74, 0x73, 0x74, 0x72, 0x61, 0x70, 0x53, 0x65, 0x72, 0x76, 0x65, 0x72, 0x73, - 0x12, 0x14, 0x0a, 0x05, 0x74, 0x6f, 0x70, 0x69, 0x63, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, - 0x05, 0x74, 0x6f, 0x70, 0x69, 0x63, 0x2a, 0x24, 0x0a, 0x0a, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, - 0x54, 0x79, 0x70, 0x65, 0x12, 0x0b, 0x0a, 0x07, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, - 0x00, 0x12, 0x09, 0x0a, 0x05, 0x4b, 0x41, 0x46, 0x4b, 0x41, 0x10, 0x01, 0x42, 0x4a, 0x0a, 0x0a, - 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x42, 0x0b, 0x53, 0x6f, 0x75, 0x72, - 0x63, 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x2f, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, - 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, - 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, - 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, + 0x0d, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x5f, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x22, 0xa4, + 0x01, 0x0a, 0x11, 0x4b, 0x61, 0x66, 0x6b, 0x61, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x43, 0x6f, + 0x6e, 0x66, 0x69, 0x67, 0x12, 0x2b, 0x0a, 0x11, 0x62, 0x6f, 0x6f, 0x74, 0x73, 0x74, 0x72, 0x61, + 0x70, 0x5f, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x73, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, + 0x10, 0x62, 0x6f, 0x6f, 0x74, 0x73, 0x74, 0x72, 0x61, 0x70, 0x53, 0x65, 0x72, 0x76, 0x65, 0x72, + 0x73, 0x12, 0x14, 0x0a, 0x05, 0x74, 0x6f, 0x70, 0x69, 0x63, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, + 0x52, 0x05, 0x74, 0x6f, 0x70, 0x69, 0x63, 0x12, 0x1e, 0x0a, 0x0a, 0x70, 0x61, 0x72, 0x74, 0x69, + 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x18, 0x03, 0x20, 0x01, 0x28, 0x05, 0x52, 0x0a, 0x70, 0x61, 0x72, + 0x74, 0x69, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x12, 0x2c, 0x0a, 0x11, 0x72, 0x65, 0x70, 0x6c, 0x69, + 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x46, 0x61, 0x63, 0x74, 0x6f, 0x72, 0x18, 0x04, 0x20, 0x01, + 0x28, 0x05, 0x52, 0x11, 0x72, 0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x46, + 0x61, 0x63, 0x74, 0x6f, 0x72, 0x2a, 0x24, 0x0a, 0x0a, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x54, + 0x79, 0x70, 0x65, 0x12, 0x0b, 0x0a, 0x07, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, + 0x12, 0x09, 0x0a, 0x05, 0x4b, 0x41, 0x46, 0x4b, 0x41, 0x10, 0x01, 0x42, 0x4a, 0x0a, 0x0a, 0x66, + 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x42, 0x0b, 0x53, 0x6f, 0x75, 0x72, 0x63, + 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x2f, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, + 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, + 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, 0x61, + 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, } var ( diff --git a/sdk/go/protos/feast/core/Store.pb.go b/sdk/go/protos/feast/core/Store.pb.go index 55c699d788b..f3b7728ab97 100644 --- a/sdk/go/protos/feast/core/Store.pb.go +++ b/sdk/go/protos/feast/core/Store.pb.go @@ -58,12 +58,7 @@ const ( Store_REDIS Store_StoreType = 1 // BigQuery stores a FeatureRow element as a row in a BigQuery table. // - // Table name is derived from the feature set name and version as: - // [feature_set_name]_v[feature_set_version] - // - // For example: - // A feature row for feature set "driver" and version "1" will be written - // to table "driver_v1". + // Table name is derived is the same as the feature set name. // // The entities and features in a FeatureSetSpec corresponds to the // fields in the BigQuery table (these make up the BigQuery schema). @@ -84,11 +79,6 @@ const ( // BigQuery table created will be partitioned by the field "event_timestamp" // of the FeatureRow (https://cloud.google.com/bigquery/docs/partitioned-tables). // - // Since newer version of feature set can introduce breaking, non backward- - // compatible BigQuery schema updates, incrementing the version of a - // feature set will result in the creation of a new empty BigQuery table - // with the new schema. - // // The following table shows how ValueType in Feast is mapped to // BigQuery Standard SQL data types // (https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types): @@ -113,7 +103,8 @@ const ( // Store_BIGQUERY Store_StoreType = 2 // Unsupported in Feast 0.3 - Store_CASSANDRA Store_StoreType = 3 + Store_CASSANDRA Store_StoreType = 3 + Store_REDIS_CLUSTER Store_StoreType = 4 ) // Enum value maps for Store_StoreType. @@ -123,12 +114,14 @@ var ( 1: "REDIS", 2: "BIGQUERY", 3: "CASSANDRA", + 4: "REDIS_CLUSTER", } Store_StoreType_value = map[string]int32{ - "INVALID": 0, - "REDIS": 1, - "BIGQUERY": 2, - "CASSANDRA": 3, + "INVALID": 0, + "REDIS": 1, + "BIGQUERY": 2, + "CASSANDRA": 3, + "REDIS_CLUSTER": 4, } ) @@ -184,6 +177,7 @@ type Store struct { // *Store_RedisConfig_ // *Store_BigqueryConfig // *Store_CassandraConfig_ + // *Store_RedisClusterConfig_ Config isStore_Config `protobuf_oneof:"config"` } @@ -268,6 +262,13 @@ func (x *Store) GetCassandraConfig() *Store_CassandraConfig { return nil } +func (x *Store) GetRedisClusterConfig() *Store_RedisClusterConfig { + if x, ok := x.GetConfig().(*Store_RedisClusterConfig_); ok { + return x.RedisClusterConfig + } + return nil +} + type isStore_Config interface { isStore_Config() } @@ -284,12 +285,18 @@ type Store_CassandraConfig_ struct { CassandraConfig *Store_CassandraConfig `protobuf:"bytes,13,opt,name=cassandra_config,json=cassandraConfig,proto3,oneof"` } +type Store_RedisClusterConfig_ struct { + RedisClusterConfig *Store_RedisClusterConfig `protobuf:"bytes,14,opt,name=redis_cluster_config,json=redisClusterConfig,proto3,oneof"` +} + func (*Store_RedisConfig_) isStore_Config() {} func (*Store_BigqueryConfig) isStore_Config() {} func (*Store_CassandraConfig_) isStore_Config() {} +func (*Store_RedisClusterConfig_) isStore_Config() {} + type Store_RedisConfig struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache @@ -369,8 +376,11 @@ type Store_BigQueryConfig struct { sizeCache protoimpl.SizeCache unknownFields protoimpl.UnknownFields - ProjectId string `protobuf:"bytes,1,opt,name=project_id,json=projectId,proto3" json:"project_id,omitempty"` - DatasetId string `protobuf:"bytes,2,opt,name=dataset_id,json=datasetId,proto3" json:"dataset_id,omitempty"` + ProjectId string `protobuf:"bytes,1,opt,name=project_id,json=projectId,proto3" json:"project_id,omitempty"` + DatasetId string `protobuf:"bytes,2,opt,name=dataset_id,json=datasetId,proto3" json:"dataset_id,omitempty"` + StagingLocation string `protobuf:"bytes,3,opt,name=staging_location,json=stagingLocation,proto3" json:"staging_location,omitempty"` + InitialRetryDelaySeconds int32 `protobuf:"varint,4,opt,name=initial_retry_delay_seconds,json=initialRetryDelaySeconds,proto3" json:"initial_retry_delay_seconds,omitempty"` + TotalTimeoutSeconds int32 `protobuf:"varint,5,opt,name=total_timeout_seconds,json=totalTimeoutSeconds,proto3" json:"total_timeout_seconds,omitempty"` } func (x *Store_BigQueryConfig) Reset() { @@ -419,6 +429,27 @@ func (x *Store_BigQueryConfig) GetDatasetId() string { return "" } +func (x *Store_BigQueryConfig) GetStagingLocation() string { + if x != nil { + return x.StagingLocation + } + return "" +} + +func (x *Store_BigQueryConfig) GetInitialRetryDelaySeconds() int32 { + if x != nil { + return x.InitialRetryDelaySeconds + } + return 0 +} + +func (x *Store_BigQueryConfig) GetTotalTimeoutSeconds() int32 { + if x != nil { + return x.TotalTimeoutSeconds + } + return 0 +} + type Store_CassandraConfig struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache @@ -474,6 +505,70 @@ func (x *Store_CassandraConfig) GetPort() int32 { return 0 } +type Store_RedisClusterConfig struct { + state protoimpl.MessageState + sizeCache protoimpl.SizeCache + unknownFields protoimpl.UnknownFields + + // List of Redis Uri for all the nodes in Redis Cluster, comma separated. Eg. host1:6379, host2:6379 + ConnectionString string `protobuf:"bytes,1,opt,name=connection_string,json=connectionString,proto3" json:"connection_string,omitempty"` + InitialBackoffMs int32 `protobuf:"varint,2,opt,name=initial_backoff_ms,json=initialBackoffMs,proto3" json:"initial_backoff_ms,omitempty"` + MaxRetries int32 `protobuf:"varint,3,opt,name=max_retries,json=maxRetries,proto3" json:"max_retries,omitempty"` +} + +func (x *Store_RedisClusterConfig) Reset() { + *x = Store_RedisClusterConfig{} + if protoimpl.UnsafeEnabled { + mi := &file_feast_core_Store_proto_msgTypes[4] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) + } +} + +func (x *Store_RedisClusterConfig) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*Store_RedisClusterConfig) ProtoMessage() {} + +func (x *Store_RedisClusterConfig) ProtoReflect() protoreflect.Message { + mi := &file_feast_core_Store_proto_msgTypes[4] + if protoimpl.UnsafeEnabled && x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use Store_RedisClusterConfig.ProtoReflect.Descriptor instead. +func (*Store_RedisClusterConfig) Descriptor() ([]byte, []int) { + return file_feast_core_Store_proto_rawDescGZIP(), []int{0, 3} +} + +func (x *Store_RedisClusterConfig) GetConnectionString() string { + if x != nil { + return x.ConnectionString + } + return "" +} + +func (x *Store_RedisClusterConfig) GetInitialBackoffMs() int32 { + if x != nil { + return x.InitialBackoffMs + } + return 0 +} + +func (x *Store_RedisClusterConfig) GetMaxRetries() int32 { + if x != nil { + return x.MaxRetries + } + return 0 +} + type Store_Subscription struct { state protoimpl.MessageState sizeCache protoimpl.SizeCache @@ -494,19 +589,12 @@ type Store_Subscription struct { // - my-feature-set* can be used to match all features prefixed by "my-feature-set" // - my-feature-set-6 can be used to select a single feature set Name string `protobuf:"bytes,1,opt,name=name,proto3" json:"name,omitempty"` - // Versions of the given feature sets that will be returned. - // Valid options for version: - // "latest": only the latest version is returned. - // "*": Subscribe to all versions - // [version number]: pin to a specific version. Project and feature set name must be - // explicitly defined if a specific version is pinned. - Version string `protobuf:"bytes,2,opt,name=version,proto3" json:"version,omitempty"` } func (x *Store_Subscription) Reset() { *x = Store_Subscription{} if protoimpl.UnsafeEnabled { - mi := &file_feast_core_Store_proto_msgTypes[4] + mi := &file_feast_core_Store_proto_msgTypes[5] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -519,7 +607,7 @@ func (x *Store_Subscription) String() string { func (*Store_Subscription) ProtoMessage() {} func (x *Store_Subscription) ProtoReflect() protoreflect.Message { - mi := &file_feast_core_Store_proto_msgTypes[4] + mi := &file_feast_core_Store_proto_msgTypes[5] if protoimpl.UnsafeEnabled && x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -532,7 +620,7 @@ func (x *Store_Subscription) ProtoReflect() protoreflect.Message { // Deprecated: Use Store_Subscription.ProtoReflect.Descriptor instead. func (*Store_Subscription) Descriptor() ([]byte, []int) { - return file_feast_core_Store_proto_rawDescGZIP(), []int{0, 3} + return file_feast_core_Store_proto_rawDescGZIP(), []int{0, 4} } func (x *Store_Subscription) GetProject() string { @@ -549,19 +637,12 @@ func (x *Store_Subscription) GetName() string { return "" } -func (x *Store_Subscription) GetVersion() string { - if x != nil { - return x.Version - } - return "" -} - var File_feast_core_Store_proto protoreflect.FileDescriptor var file_feast_core_Store_proto_rawDesc = []byte{ 0x0a, 0x16, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x2f, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, - 0x63, 0x6f, 0x72, 0x65, 0x22, 0xa9, 0x06, 0x0a, 0x05, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x12, 0x12, + 0x63, 0x6f, 0x72, 0x65, 0x22, 0xae, 0x09, 0x0a, 0x05, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x2f, 0x0a, 0x04, 0x74, 0x79, 0x70, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x1b, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x74, @@ -584,40 +665,64 @@ var file_feast_core_Store_proto_rawDesc = []byte{ 0x20, 0x01, 0x28, 0x0b, 0x32, 0x21, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x2e, 0x43, 0x61, 0x73, 0x73, 0x61, 0x6e, 0x64, 0x72, 0x61, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x48, 0x00, 0x52, 0x0f, 0x63, 0x61, 0x73, 0x73, 0x61, - 0x6e, 0x64, 0x72, 0x61, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x1a, 0x84, 0x01, 0x0a, 0x0b, 0x52, - 0x65, 0x64, 0x69, 0x73, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x12, 0x12, 0x0a, 0x04, 0x68, 0x6f, - 0x73, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x68, 0x6f, 0x73, 0x74, 0x12, 0x12, - 0x0a, 0x04, 0x70, 0x6f, 0x72, 0x74, 0x18, 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, 0x04, 0x70, 0x6f, - 0x72, 0x74, 0x12, 0x2c, 0x0a, 0x12, 0x69, 0x6e, 0x69, 0x74, 0x69, 0x61, 0x6c, 0x5f, 0x62, 0x61, - 0x63, 0x6b, 0x6f, 0x66, 0x66, 0x5f, 0x6d, 0x73, 0x18, 0x03, 0x20, 0x01, 0x28, 0x05, 0x52, 0x10, - 0x69, 0x6e, 0x69, 0x74, 0x69, 0x61, 0x6c, 0x42, 0x61, 0x63, 0x6b, 0x6f, 0x66, 0x66, 0x4d, 0x73, - 0x12, 0x1f, 0x0a, 0x0b, 0x6d, 0x61, 0x78, 0x5f, 0x72, 0x65, 0x74, 0x72, 0x69, 0x65, 0x73, 0x18, - 0x04, 0x20, 0x01, 0x28, 0x05, 0x52, 0x0a, 0x6d, 0x61, 0x78, 0x52, 0x65, 0x74, 0x72, 0x69, 0x65, - 0x73, 0x1a, 0x4e, 0x0a, 0x0e, 0x42, 0x69, 0x67, 0x51, 0x75, 0x65, 0x72, 0x79, 0x43, 0x6f, 0x6e, - 0x66, 0x69, 0x67, 0x12, 0x1d, 0x0a, 0x0a, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x5f, 0x69, - 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, - 0x49, 0x64, 0x12, 0x1d, 0x0a, 0x0a, 0x64, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x5f, 0x69, 0x64, - 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x64, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x49, - 0x64, 0x1a, 0x39, 0x0a, 0x0f, 0x43, 0x61, 0x73, 0x73, 0x61, 0x6e, 0x64, 0x72, 0x61, 0x43, 0x6f, + 0x6e, 0x64, 0x72, 0x61, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x12, 0x58, 0x0a, 0x14, 0x72, 0x65, + 0x64, 0x69, 0x73, 0x5f, 0x63, 0x6c, 0x75, 0x73, 0x74, 0x65, 0x72, 0x5f, 0x63, 0x6f, 0x6e, 0x66, + 0x69, 0x67, 0x18, 0x0e, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x24, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x2e, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x2e, 0x52, 0x65, 0x64, 0x69, + 0x73, 0x43, 0x6c, 0x75, 0x73, 0x74, 0x65, 0x72, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x48, 0x00, + 0x52, 0x12, 0x72, 0x65, 0x64, 0x69, 0x73, 0x43, 0x6c, 0x75, 0x73, 0x74, 0x65, 0x72, 0x43, 0x6f, + 0x6e, 0x66, 0x69, 0x67, 0x1a, 0x84, 0x01, 0x0a, 0x0b, 0x52, 0x65, 0x64, 0x69, 0x73, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x12, 0x12, 0x0a, 0x04, 0x68, 0x6f, 0x73, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x68, 0x6f, 0x73, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x70, 0x6f, 0x72, 0x74, - 0x18, 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, 0x04, 0x70, 0x6f, 0x72, 0x74, 0x1a, 0x56, 0x0a, 0x0c, - 0x53, 0x75, 0x62, 0x73, 0x63, 0x72, 0x69, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x18, 0x0a, 0x07, - 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x70, - 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, - 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x76, 0x65, - 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x76, 0x65, 0x72, - 0x73, 0x69, 0x6f, 0x6e, 0x22, 0x40, 0x0a, 0x09, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x54, 0x79, 0x70, - 0x65, 0x12, 0x0b, 0x0a, 0x07, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, 0x12, 0x09, - 0x0a, 0x05, 0x52, 0x45, 0x44, 0x49, 0x53, 0x10, 0x01, 0x12, 0x0c, 0x0a, 0x08, 0x42, 0x49, 0x47, - 0x51, 0x55, 0x45, 0x52, 0x59, 0x10, 0x02, 0x12, 0x0d, 0x0a, 0x09, 0x43, 0x41, 0x53, 0x53, 0x41, - 0x4e, 0x44, 0x52, 0x41, 0x10, 0x03, 0x42, 0x08, 0x0a, 0x06, 0x63, 0x6f, 0x6e, 0x66, 0x69, 0x67, - 0x42, 0x49, 0x0a, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, 0x6f, 0x72, 0x65, 0x42, 0x0a, - 0x53, 0x74, 0x6f, 0x72, 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x2f, 0x67, 0x69, 0x74, 0x68, - 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, - 0x73, 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, - 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, 0x62, 0x06, 0x70, 0x72, 0x6f, - 0x74, 0x6f, 0x33, + 0x18, 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, 0x04, 0x70, 0x6f, 0x72, 0x74, 0x12, 0x2c, 0x0a, 0x12, + 0x69, 0x6e, 0x69, 0x74, 0x69, 0x61, 0x6c, 0x5f, 0x62, 0x61, 0x63, 0x6b, 0x6f, 0x66, 0x66, 0x5f, + 0x6d, 0x73, 0x18, 0x03, 0x20, 0x01, 0x28, 0x05, 0x52, 0x10, 0x69, 0x6e, 0x69, 0x74, 0x69, 0x61, + 0x6c, 0x42, 0x61, 0x63, 0x6b, 0x6f, 0x66, 0x66, 0x4d, 0x73, 0x12, 0x1f, 0x0a, 0x0b, 0x6d, 0x61, + 0x78, 0x5f, 0x72, 0x65, 0x74, 0x72, 0x69, 0x65, 0x73, 0x18, 0x04, 0x20, 0x01, 0x28, 0x05, 0x52, + 0x0a, 0x6d, 0x61, 0x78, 0x52, 0x65, 0x74, 0x72, 0x69, 0x65, 0x73, 0x1a, 0xec, 0x01, 0x0a, 0x0e, + 0x42, 0x69, 0x67, 0x51, 0x75, 0x65, 0x72, 0x79, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x12, 0x1d, + 0x0a, 0x0a, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, + 0x28, 0x09, 0x52, 0x09, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x49, 0x64, 0x12, 0x1d, 0x0a, + 0x0a, 0x64, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x5f, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, + 0x09, 0x52, 0x09, 0x64, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x49, 0x64, 0x12, 0x29, 0x0a, 0x10, + 0x73, 0x74, 0x61, 0x67, 0x69, 0x6e, 0x67, 0x5f, 0x6c, 0x6f, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, + 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x0f, 0x73, 0x74, 0x61, 0x67, 0x69, 0x6e, 0x67, 0x4c, + 0x6f, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x3d, 0x0a, 0x1b, 0x69, 0x6e, 0x69, 0x74, 0x69, + 0x61, 0x6c, 0x5f, 0x72, 0x65, 0x74, 0x72, 0x79, 0x5f, 0x64, 0x65, 0x6c, 0x61, 0x79, 0x5f, 0x73, + 0x65, 0x63, 0x6f, 0x6e, 0x64, 0x73, 0x18, 0x04, 0x20, 0x01, 0x28, 0x05, 0x52, 0x18, 0x69, 0x6e, + 0x69, 0x74, 0x69, 0x61, 0x6c, 0x52, 0x65, 0x74, 0x72, 0x79, 0x44, 0x65, 0x6c, 0x61, 0x79, 0x53, + 0x65, 0x63, 0x6f, 0x6e, 0x64, 0x73, 0x12, 0x32, 0x0a, 0x15, 0x74, 0x6f, 0x74, 0x61, 0x6c, 0x5f, + 0x74, 0x69, 0x6d, 0x65, 0x6f, 0x75, 0x74, 0x5f, 0x73, 0x65, 0x63, 0x6f, 0x6e, 0x64, 0x73, 0x18, + 0x05, 0x20, 0x01, 0x28, 0x05, 0x52, 0x13, 0x74, 0x6f, 0x74, 0x61, 0x6c, 0x54, 0x69, 0x6d, 0x65, + 0x6f, 0x75, 0x74, 0x53, 0x65, 0x63, 0x6f, 0x6e, 0x64, 0x73, 0x1a, 0x39, 0x0a, 0x0f, 0x43, 0x61, + 0x73, 0x73, 0x61, 0x6e, 0x64, 0x72, 0x61, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x12, 0x12, 0x0a, + 0x04, 0x68, 0x6f, 0x73, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x68, 0x6f, 0x73, + 0x74, 0x12, 0x12, 0x0a, 0x04, 0x70, 0x6f, 0x72, 0x74, 0x18, 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, + 0x04, 0x70, 0x6f, 0x72, 0x74, 0x1a, 0x90, 0x01, 0x0a, 0x12, 0x52, 0x65, 0x64, 0x69, 0x73, 0x43, + 0x6c, 0x75, 0x73, 0x74, 0x65, 0x72, 0x43, 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x12, 0x2b, 0x0a, 0x11, + 0x63, 0x6f, 0x6e, 0x6e, 0x65, 0x63, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x73, 0x74, 0x72, 0x69, 0x6e, + 0x67, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x10, 0x63, 0x6f, 0x6e, 0x6e, 0x65, 0x63, 0x74, + 0x69, 0x6f, 0x6e, 0x53, 0x74, 0x72, 0x69, 0x6e, 0x67, 0x12, 0x2c, 0x0a, 0x12, 0x69, 0x6e, 0x69, + 0x74, 0x69, 0x61, 0x6c, 0x5f, 0x62, 0x61, 0x63, 0x6b, 0x6f, 0x66, 0x66, 0x5f, 0x6d, 0x73, 0x18, + 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, 0x10, 0x69, 0x6e, 0x69, 0x74, 0x69, 0x61, 0x6c, 0x42, 0x61, + 0x63, 0x6b, 0x6f, 0x66, 0x66, 0x4d, 0x73, 0x12, 0x1f, 0x0a, 0x0b, 0x6d, 0x61, 0x78, 0x5f, 0x72, + 0x65, 0x74, 0x72, 0x69, 0x65, 0x73, 0x18, 0x03, 0x20, 0x01, 0x28, 0x05, 0x52, 0x0a, 0x6d, 0x61, + 0x78, 0x52, 0x65, 0x74, 0x72, 0x69, 0x65, 0x73, 0x1a, 0x3c, 0x0a, 0x0c, 0x53, 0x75, 0x62, 0x73, + 0x63, 0x72, 0x69, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x18, 0x0a, 0x07, 0x70, 0x72, 0x6f, 0x6a, + 0x65, 0x63, 0x74, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, + 0x63, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, + 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x22, 0x53, 0x0a, 0x09, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x54, + 0x79, 0x70, 0x65, 0x12, 0x0b, 0x0a, 0x07, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, + 0x12, 0x09, 0x0a, 0x05, 0x52, 0x45, 0x44, 0x49, 0x53, 0x10, 0x01, 0x12, 0x0c, 0x0a, 0x08, 0x42, + 0x49, 0x47, 0x51, 0x55, 0x45, 0x52, 0x59, 0x10, 0x02, 0x12, 0x0d, 0x0a, 0x09, 0x43, 0x41, 0x53, + 0x53, 0x41, 0x4e, 0x44, 0x52, 0x41, 0x10, 0x03, 0x12, 0x11, 0x0a, 0x0d, 0x52, 0x45, 0x44, 0x49, + 0x53, 0x5f, 0x43, 0x4c, 0x55, 0x53, 0x54, 0x45, 0x52, 0x10, 0x04, 0x42, 0x08, 0x0a, 0x06, 0x63, + 0x6f, 0x6e, 0x66, 0x69, 0x67, 0x42, 0x49, 0x0a, 0x0a, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x63, + 0x6f, 0x72, 0x65, 0x42, 0x0a, 0x53, 0x74, 0x6f, 0x72, 0x65, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, + 0x2f, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, + 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, + 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x63, 0x6f, 0x72, 0x65, + 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, } var ( @@ -633,26 +738,28 @@ func file_feast_core_Store_proto_rawDescGZIP() []byte { } var file_feast_core_Store_proto_enumTypes = make([]protoimpl.EnumInfo, 1) -var file_feast_core_Store_proto_msgTypes = make([]protoimpl.MessageInfo, 5) +var file_feast_core_Store_proto_msgTypes = make([]protoimpl.MessageInfo, 6) var file_feast_core_Store_proto_goTypes = []interface{}{ - (Store_StoreType)(0), // 0: feast.core.Store.StoreType - (*Store)(nil), // 1: feast.core.Store - (*Store_RedisConfig)(nil), // 2: feast.core.Store.RedisConfig - (*Store_BigQueryConfig)(nil), // 3: feast.core.Store.BigQueryConfig - (*Store_CassandraConfig)(nil), // 4: feast.core.Store.CassandraConfig - (*Store_Subscription)(nil), // 5: feast.core.Store.Subscription + (Store_StoreType)(0), // 0: feast.core.Store.StoreType + (*Store)(nil), // 1: feast.core.Store + (*Store_RedisConfig)(nil), // 2: feast.core.Store.RedisConfig + (*Store_BigQueryConfig)(nil), // 3: feast.core.Store.BigQueryConfig + (*Store_CassandraConfig)(nil), // 4: feast.core.Store.CassandraConfig + (*Store_RedisClusterConfig)(nil), // 5: feast.core.Store.RedisClusterConfig + (*Store_Subscription)(nil), // 6: feast.core.Store.Subscription } var file_feast_core_Store_proto_depIdxs = []int32{ 0, // 0: feast.core.Store.type:type_name -> feast.core.Store.StoreType - 5, // 1: feast.core.Store.subscriptions:type_name -> feast.core.Store.Subscription + 6, // 1: feast.core.Store.subscriptions:type_name -> feast.core.Store.Subscription 2, // 2: feast.core.Store.redis_config:type_name -> feast.core.Store.RedisConfig 3, // 3: feast.core.Store.bigquery_config:type_name -> feast.core.Store.BigQueryConfig 4, // 4: feast.core.Store.cassandra_config:type_name -> feast.core.Store.CassandraConfig - 5, // [5:5] is the sub-list for method output_type - 5, // [5:5] is the sub-list for method input_type - 5, // [5:5] is the sub-list for extension type_name - 5, // [5:5] is the sub-list for extension extendee - 0, // [0:5] is the sub-list for field type_name + 5, // 5: feast.core.Store.redis_cluster_config:type_name -> feast.core.Store.RedisClusterConfig + 6, // [6:6] is the sub-list for method output_type + 6, // [6:6] is the sub-list for method input_type + 6, // [6:6] is the sub-list for extension type_name + 6, // [6:6] is the sub-list for extension extendee + 0, // [0:6] is the sub-list for field type_name } func init() { file_feast_core_Store_proto_init() } @@ -710,6 +817,18 @@ func file_feast_core_Store_proto_init() { } } file_feast_core_Store_proto_msgTypes[4].Exporter = func(v interface{}, i int) interface{} { + switch v := v.(*Store_RedisClusterConfig); i { + case 0: + return &v.state + case 1: + return &v.sizeCache + case 2: + return &v.unknownFields + default: + return nil + } + } + file_feast_core_Store_proto_msgTypes[5].Exporter = func(v interface{}, i int) interface{} { switch v := v.(*Store_Subscription); i { case 0: return &v.state @@ -726,6 +845,7 @@ func file_feast_core_Store_proto_init() { (*Store_RedisConfig_)(nil), (*Store_BigqueryConfig)(nil), (*Store_CassandraConfig_)(nil), + (*Store_RedisClusterConfig_)(nil), } type x struct{} out := protoimpl.TypeBuilder{ @@ -733,7 +853,7 @@ func file_feast_core_Store_proto_init() { GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: file_feast_core_Store_proto_rawDesc, NumEnums: 1, - NumMessages: 5, + NumMessages: 6, NumExtensions: 0, NumServices: 0, }, diff --git a/sdk/go/protos/feast/serving/ServingService.pb.go b/sdk/go/protos/feast/serving/ServingService.pb.go index 2485687d81e..a8659c55814 100644 --- a/sdk/go/protos/feast/serving/ServingService.pb.go +++ b/sdk/go/protos/feast/serving/ServingService.pb.go @@ -359,8 +359,6 @@ type FeatureReference struct { Project string `protobuf:"bytes,1,opt,name=project,proto3" json:"project,omitempty"` // Feature name Name string `protobuf:"bytes,2,opt,name=name,proto3" json:"name,omitempty"` - // Feature version - Version int32 `protobuf:"varint,3,opt,name=version,proto3" json:"version,omitempty"` // The features will be retrieved if: // entity_timestamp - max_age <= event_timestamp <= entity_timestamp // @@ -415,13 +413,6 @@ func (x *FeatureReference) GetName() string { return "" } -func (x *FeatureReference) GetVersion() int32 { - if x != nil { - return x.Version - } - return 0 -} - func (x *FeatureReference) GetMaxAge() *duration.Duration { if x != nil { return x.MaxAge @@ -1095,166 +1086,165 @@ var file_feast_serving_ServingService_proto_rawDesc = []byte{ 0x54, 0x79, 0x70, 0x65, 0x52, 0x04, 0x74, 0x79, 0x70, 0x65, 0x12, 0x30, 0x0a, 0x14, 0x6a, 0x6f, 0x62, 0x5f, 0x73, 0x74, 0x61, 0x67, 0x69, 0x6e, 0x67, 0x5f, 0x6c, 0x6f, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x0a, 0x20, 0x01, 0x28, 0x09, 0x52, 0x12, 0x6a, 0x6f, 0x62, 0x53, 0x74, 0x61, - 0x67, 0x69, 0x6e, 0x67, 0x4c, 0x6f, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x22, 0x8e, 0x01, 0x0a, - 0x10, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, - 0x65, 0x12, 0x18, 0x0a, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x18, 0x01, 0x20, 0x01, - 0x28, 0x09, 0x52, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, - 0x61, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, - 0x18, 0x0a, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x18, 0x03, 0x20, 0x01, 0x28, 0x05, - 0x52, 0x07, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x12, 0x32, 0x0a, 0x07, 0x6d, 0x61, 0x78, - 0x5f, 0x61, 0x67, 0x65, 0x18, 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x19, 0x2e, 0x67, 0x6f, 0x6f, - 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x44, 0x75, 0x72, - 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x06, 0x6d, 0x61, 0x78, 0x41, 0x67, 0x65, 0x22, 0xe1, 0x03, - 0x0a, 0x18, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, - 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x3b, 0x0a, 0x08, 0x66, 0x65, - 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x18, 0x04, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x1f, 0x2e, 0x66, - 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x46, 0x65, 0x61, - 0x74, 0x75, 0x72, 0x65, 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, 0x52, 0x08, 0x66, - 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x12, 0x52, 0x0a, 0x0b, 0x65, 0x6e, 0x74, 0x69, 0x74, - 0x79, 0x5f, 0x72, 0x6f, 0x77, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x31, 0x2e, 0x66, - 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, - 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, - 0x71, 0x75, 0x65, 0x73, 0x74, 0x2e, 0x45, 0x6e, 0x74, 0x69, 0x74, 0x79, 0x52, 0x6f, 0x77, 0x52, - 0x0a, 0x65, 0x6e, 0x74, 0x69, 0x74, 0x79, 0x52, 0x6f, 0x77, 0x73, 0x12, 0x39, 0x0a, 0x19, 0x6f, - 0x6d, 0x69, 0x74, 0x5f, 0x65, 0x6e, 0x74, 0x69, 0x74, 0x69, 0x65, 0x73, 0x5f, 0x69, 0x6e, 0x5f, - 0x72, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x08, 0x52, 0x16, - 0x6f, 0x6d, 0x69, 0x74, 0x45, 0x6e, 0x74, 0x69, 0x74, 0x69, 0x65, 0x73, 0x49, 0x6e, 0x52, 0x65, - 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x1a, 0xf8, 0x01, 0x0a, 0x09, 0x45, 0x6e, 0x74, 0x69, 0x74, - 0x79, 0x52, 0x6f, 0x77, 0x12, 0x45, 0x0a, 0x10, 0x65, 0x6e, 0x74, 0x69, 0x74, 0x79, 0x5f, 0x74, - 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, - 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, - 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x0f, 0x65, 0x6e, 0x74, 0x69, - 0x74, 0x79, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x12, 0x55, 0x0a, 0x06, 0x66, - 0x69, 0x65, 0x6c, 0x64, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x3d, 0x2e, 0x66, 0x65, - 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4f, - 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, - 0x75, 0x65, 0x73, 0x74, 0x2e, 0x45, 0x6e, 0x74, 0x69, 0x74, 0x79, 0x52, 0x6f, 0x77, 0x2e, 0x46, - 0x69, 0x65, 0x6c, 0x64, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x06, 0x66, 0x69, 0x65, 0x6c, - 0x64, 0x73, 0x1a, 0x4d, 0x0a, 0x0b, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x73, 0x45, 0x6e, 0x74, 0x72, - 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, - 0x6b, 0x65, 0x79, 0x12, 0x28, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, - 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, - 0x2e, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, - 0x01, 0x22, 0x9b, 0x01, 0x0a, 0x17, 0x47, 0x65, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x46, 0x65, - 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x3b, 0x0a, - 0x08, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x18, 0x03, 0x20, 0x03, 0x28, 0x0b, 0x32, - 0x1f, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, + 0x67, 0x69, 0x6e, 0x67, 0x4c, 0x6f, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x22, 0x74, 0x0a, 0x10, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, 0x63, 0x65, - 0x52, 0x08, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x12, 0x43, 0x0a, 0x0e, 0x64, 0x61, - 0x74, 0x61, 0x73, 0x65, 0x74, 0x5f, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x02, 0x20, 0x01, - 0x28, 0x0b, 0x32, 0x1c, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, - 0x6e, 0x67, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, - 0x52, 0x0d, 0x64, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x22, - 0xad, 0x02, 0x0a, 0x19, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, - 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x57, 0x0a, - 0x0c, 0x66, 0x69, 0x65, 0x6c, 0x64, 0x5f, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x73, 0x18, 0x01, 0x20, - 0x03, 0x28, 0x0b, 0x32, 0x34, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, - 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, - 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x46, 0x69, - 0x65, 0x6c, 0x64, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x73, 0x52, 0x0b, 0x66, 0x69, 0x65, 0x6c, 0x64, - 0x56, 0x61, 0x6c, 0x75, 0x65, 0x73, 0x1a, 0xb6, 0x01, 0x0a, 0x0b, 0x46, 0x69, 0x65, 0x6c, 0x64, - 0x56, 0x61, 0x6c, 0x75, 0x65, 0x73, 0x12, 0x58, 0x0a, 0x06, 0x66, 0x69, 0x65, 0x6c, 0x64, 0x73, - 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x40, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, - 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, + 0x12, 0x18, 0x0a, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, + 0x09, 0x52, 0x07, 0x70, 0x72, 0x6f, 0x6a, 0x65, 0x63, 0x74, 0x12, 0x12, 0x0a, 0x04, 0x6e, 0x61, + 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x6e, 0x61, 0x6d, 0x65, 0x12, 0x32, + 0x0a, 0x07, 0x6d, 0x61, 0x78, 0x5f, 0x61, 0x67, 0x65, 0x18, 0x04, 0x20, 0x01, 0x28, 0x0b, 0x32, + 0x19, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, + 0x66, 0x2e, 0x44, 0x75, 0x72, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x52, 0x06, 0x6d, 0x61, 0x78, 0x41, + 0x67, 0x65, 0x22, 0xe1, 0x03, 0x0a, 0x18, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, + 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, + 0x3b, 0x0a, 0x08, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x18, 0x04, 0x20, 0x03, 0x28, + 0x0b, 0x32, 0x1f, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, + 0x67, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x65, 0x66, 0x65, 0x72, 0x65, 0x6e, + 0x63, 0x65, 0x52, 0x08, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x12, 0x52, 0x0a, 0x0b, + 0x65, 0x6e, 0x74, 0x69, 0x74, 0x79, 0x5f, 0x72, 0x6f, 0x77, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, + 0x0b, 0x32, 0x31, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, + 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, + 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x2e, 0x45, 0x6e, 0x74, 0x69, 0x74, + 0x79, 0x52, 0x6f, 0x77, 0x52, 0x0a, 0x65, 0x6e, 0x74, 0x69, 0x74, 0x79, 0x52, 0x6f, 0x77, 0x73, + 0x12, 0x39, 0x0a, 0x19, 0x6f, 0x6d, 0x69, 0x74, 0x5f, 0x65, 0x6e, 0x74, 0x69, 0x74, 0x69, 0x65, + 0x73, 0x5f, 0x69, 0x6e, 0x5f, 0x72, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x18, 0x03, 0x20, + 0x01, 0x28, 0x08, 0x52, 0x16, 0x6f, 0x6d, 0x69, 0x74, 0x45, 0x6e, 0x74, 0x69, 0x74, 0x69, 0x65, + 0x73, 0x49, 0x6e, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x1a, 0xf8, 0x01, 0x0a, 0x09, + 0x45, 0x6e, 0x74, 0x69, 0x74, 0x79, 0x52, 0x6f, 0x77, 0x12, 0x45, 0x0a, 0x10, 0x65, 0x6e, 0x74, + 0x69, 0x74, 0x79, 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x18, 0x01, 0x20, + 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, + 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, + 0x0f, 0x65, 0x6e, 0x74, 0x69, 0x74, 0x79, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, + 0x12, 0x55, 0x0a, 0x06, 0x66, 0x69, 0x65, 0x6c, 0x64, 0x73, 0x18, 0x02, 0x20, 0x03, 0x28, 0x0b, + 0x32, 0x3d, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, + 0x2e, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, + 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x2e, 0x45, 0x6e, 0x74, 0x69, 0x74, 0x79, + 0x52, 0x6f, 0x77, 0x2e, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, + 0x06, 0x66, 0x69, 0x65, 0x6c, 0x64, 0x73, 0x1a, 0x4d, 0x0a, 0x0b, 0x46, 0x69, 0x65, 0x6c, 0x64, + 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, + 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x28, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, + 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x52, 0x05, 0x76, 0x61, 0x6c, + 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x22, 0x9b, 0x01, 0x0a, 0x17, 0x47, 0x65, 0x74, 0x42, 0x61, + 0x74, 0x63, 0x68, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, + 0x73, 0x74, 0x12, 0x3b, 0x0a, 0x08, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x18, 0x03, + 0x20, 0x03, 0x28, 0x0b, 0x32, 0x1f, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, + 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x52, 0x65, 0x66, 0x65, + 0x72, 0x65, 0x6e, 0x63, 0x65, 0x52, 0x08, 0x66, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x12, + 0x43, 0x0a, 0x0e, 0x64, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x5f, 0x73, 0x6f, 0x75, 0x72, 0x63, + 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1c, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x53, + 0x6f, 0x75, 0x72, 0x63, 0x65, 0x52, 0x0d, 0x64, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x53, 0x6f, + 0x75, 0x72, 0x63, 0x65, 0x22, 0xad, 0x02, 0x0a, 0x19, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, + 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, + 0x73, 0x65, 0x12, 0x57, 0x0a, 0x0c, 0x66, 0x69, 0x65, 0x6c, 0x64, 0x5f, 0x76, 0x61, 0x6c, 0x75, + 0x65, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x34, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, + 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, + 0x73, 0x65, 0x2e, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x73, 0x52, 0x0b, + 0x66, 0x69, 0x65, 0x6c, 0x64, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x73, 0x1a, 0xb6, 0x01, 0x0a, 0x0b, + 0x46, 0x69, 0x65, 0x6c, 0x64, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x73, 0x12, 0x58, 0x0a, 0x06, 0x66, + 0x69, 0x65, 0x6c, 0x64, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x40, 0x2e, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4f, + 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, + 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x2e, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x56, 0x61, 0x6c, 0x75, 0x65, + 0x73, 0x2e, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x06, 0x66, + 0x69, 0x65, 0x6c, 0x64, 0x73, 0x1a, 0x4d, 0x0a, 0x0b, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x73, 0x45, + 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, + 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x28, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, + 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, + 0x70, 0x65, 0x73, 0x2e, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, + 0x3a, 0x02, 0x38, 0x01, 0x22, 0x40, 0x0a, 0x18, 0x47, 0x65, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, - 0x2e, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x56, 0x61, 0x6c, 0x75, 0x65, 0x73, 0x2e, 0x46, 0x69, 0x65, - 0x6c, 0x64, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x06, 0x66, 0x69, 0x65, 0x6c, 0x64, 0x73, - 0x1a, 0x4d, 0x0a, 0x0b, 0x46, 0x69, 0x65, 0x6c, 0x64, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, - 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, - 0x79, 0x12, 0x28, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, - 0x32, 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x74, 0x79, 0x70, 0x65, 0x73, 0x2e, 0x56, - 0x61, 0x6c, 0x75, 0x65, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38, 0x01, 0x22, - 0x40, 0x0a, 0x18, 0x47, 0x65, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x46, 0x65, 0x61, 0x74, 0x75, - 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x24, 0x0a, 0x03, 0x6a, - 0x6f, 0x62, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, - 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x4a, 0x6f, 0x62, 0x52, 0x03, 0x6a, 0x6f, - 0x62, 0x22, 0x35, 0x0a, 0x0d, 0x47, 0x65, 0x74, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x71, 0x75, 0x65, - 0x73, 0x74, 0x12, 0x24, 0x0a, 0x03, 0x6a, 0x6f, 0x62, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, - 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, - 0x4a, 0x6f, 0x62, 0x52, 0x03, 0x6a, 0x6f, 0x62, 0x22, 0x36, 0x0a, 0x0e, 0x47, 0x65, 0x74, 0x4a, - 0x6f, 0x62, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x24, 0x0a, 0x03, 0x6a, 0x6f, - 0x62, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, - 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x4a, 0x6f, 0x62, 0x52, 0x03, 0x6a, 0x6f, 0x62, - 0x22, 0xe2, 0x01, 0x0a, 0x03, 0x4a, 0x6f, 0x62, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01, - 0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x12, 0x2a, 0x0a, 0x04, 0x74, 0x79, 0x70, 0x65, - 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x16, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, - 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x4a, 0x6f, 0x62, 0x54, 0x79, 0x70, 0x65, 0x52, 0x04, - 0x74, 0x79, 0x70, 0x65, 0x12, 0x30, 0x0a, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x18, 0x03, - 0x20, 0x01, 0x28, 0x0e, 0x32, 0x18, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, - 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x4a, 0x6f, 0x62, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x52, 0x06, - 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x14, 0x0a, 0x05, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x18, - 0x04, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x65, 0x72, 0x72, 0x6f, 0x72, 0x12, 0x1b, 0x0a, 0x09, - 0x66, 0x69, 0x6c, 0x65, 0x5f, 0x75, 0x72, 0x69, 0x73, 0x18, 0x05, 0x20, 0x03, 0x28, 0x09, 0x52, - 0x08, 0x66, 0x69, 0x6c, 0x65, 0x55, 0x72, 0x69, 0x73, 0x12, 0x3a, 0x0a, 0x0b, 0x64, 0x61, 0x74, - 0x61, 0x5f, 0x66, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x18, 0x06, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x19, - 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x44, - 0x61, 0x74, 0x61, 0x46, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x52, 0x0a, 0x64, 0x61, 0x74, 0x61, 0x46, - 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x22, 0xd4, 0x01, 0x0a, 0x0d, 0x44, 0x61, 0x74, 0x61, 0x73, 0x65, - 0x74, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x12, 0x4a, 0x0a, 0x0b, 0x66, 0x69, 0x6c, 0x65, 0x5f, - 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x27, 0x2e, 0x66, - 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x44, 0x61, 0x74, - 0x61, 0x73, 0x65, 0x74, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x2e, 0x46, 0x69, 0x6c, 0x65, 0x53, - 0x6f, 0x75, 0x72, 0x63, 0x65, 0x48, 0x00, 0x52, 0x0a, 0x66, 0x69, 0x6c, 0x65, 0x53, 0x6f, 0x75, - 0x72, 0x63, 0x65, 0x1a, 0x65, 0x0a, 0x0a, 0x46, 0x69, 0x6c, 0x65, 0x53, 0x6f, 0x75, 0x72, 0x63, - 0x65, 0x12, 0x1b, 0x0a, 0x09, 0x66, 0x69, 0x6c, 0x65, 0x5f, 0x75, 0x72, 0x69, 0x73, 0x18, 0x01, + 0x12, 0x24, 0x0a, 0x03, 0x6a, 0x6f, 0x62, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x12, 0x2e, + 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x4a, 0x6f, + 0x62, 0x52, 0x03, 0x6a, 0x6f, 0x62, 0x22, 0x35, 0x0a, 0x0d, 0x47, 0x65, 0x74, 0x4a, 0x6f, 0x62, + 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x12, 0x24, 0x0a, 0x03, 0x6a, 0x6f, 0x62, 0x18, 0x01, + 0x20, 0x01, 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, + 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x4a, 0x6f, 0x62, 0x52, 0x03, 0x6a, 0x6f, 0x62, 0x22, 0x36, 0x0a, + 0x0e, 0x47, 0x65, 0x74, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, + 0x24, 0x0a, 0x03, 0x6a, 0x6f, 0x62, 0x18, 0x01, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x12, 0x2e, 0x66, + 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x4a, 0x6f, 0x62, + 0x52, 0x03, 0x6a, 0x6f, 0x62, 0x22, 0xe2, 0x01, 0x0a, 0x03, 0x4a, 0x6f, 0x62, 0x12, 0x0e, 0x0a, + 0x02, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x12, 0x2a, 0x0a, + 0x04, 0x74, 0x79, 0x70, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x16, 0x2e, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x4a, 0x6f, 0x62, 0x54, + 0x79, 0x70, 0x65, 0x52, 0x04, 0x74, 0x79, 0x70, 0x65, 0x12, 0x30, 0x0a, 0x06, 0x73, 0x74, 0x61, + 0x74, 0x75, 0x73, 0x18, 0x03, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x18, 0x2e, 0x66, 0x65, 0x61, 0x73, + 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x4a, 0x6f, 0x62, 0x53, 0x74, 0x61, + 0x74, 0x75, 0x73, 0x52, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x14, 0x0a, 0x05, 0x65, + 0x72, 0x72, 0x6f, 0x72, 0x18, 0x04, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x65, 0x72, 0x72, 0x6f, + 0x72, 0x12, 0x1b, 0x0a, 0x09, 0x66, 0x69, 0x6c, 0x65, 0x5f, 0x75, 0x72, 0x69, 0x73, 0x18, 0x05, 0x20, 0x03, 0x28, 0x09, 0x52, 0x08, 0x66, 0x69, 0x6c, 0x65, 0x55, 0x72, 0x69, 0x73, 0x12, 0x3a, - 0x0a, 0x0b, 0x64, 0x61, 0x74, 0x61, 0x5f, 0x66, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x18, 0x02, 0x20, + 0x0a, 0x0b, 0x64, 0x61, 0x74, 0x61, 0x5f, 0x66, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x18, 0x06, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x19, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x46, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x52, 0x0a, - 0x64, 0x61, 0x74, 0x61, 0x46, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x42, 0x10, 0x0a, 0x0e, 0x64, 0x61, - 0x74, 0x61, 0x73, 0x65, 0x74, 0x5f, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x2a, 0x6f, 0x0a, 0x10, - 0x46, 0x65, 0x61, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x54, 0x79, 0x70, 0x65, - 0x12, 0x1e, 0x0a, 0x1a, 0x46, 0x45, 0x41, 0x53, 0x54, 0x5f, 0x53, 0x45, 0x52, 0x56, 0x49, 0x4e, - 0x47, 0x5f, 0x54, 0x59, 0x50, 0x45, 0x5f, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, - 0x12, 0x1d, 0x0a, 0x19, 0x46, 0x45, 0x41, 0x53, 0x54, 0x5f, 0x53, 0x45, 0x52, 0x56, 0x49, 0x4e, - 0x47, 0x5f, 0x54, 0x59, 0x50, 0x45, 0x5f, 0x4f, 0x4e, 0x4c, 0x49, 0x4e, 0x45, 0x10, 0x01, 0x12, - 0x1c, 0x0a, 0x18, 0x46, 0x45, 0x41, 0x53, 0x54, 0x5f, 0x53, 0x45, 0x52, 0x56, 0x49, 0x4e, 0x47, - 0x5f, 0x54, 0x59, 0x50, 0x45, 0x5f, 0x42, 0x41, 0x54, 0x43, 0x48, 0x10, 0x02, 0x2a, 0x36, 0x0a, - 0x07, 0x4a, 0x6f, 0x62, 0x54, 0x79, 0x70, 0x65, 0x12, 0x14, 0x0a, 0x10, 0x4a, 0x4f, 0x42, 0x5f, - 0x54, 0x59, 0x50, 0x45, 0x5f, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, 0x12, 0x15, - 0x0a, 0x11, 0x4a, 0x4f, 0x42, 0x5f, 0x54, 0x59, 0x50, 0x45, 0x5f, 0x44, 0x4f, 0x57, 0x4e, 0x4c, - 0x4f, 0x41, 0x44, 0x10, 0x01, 0x2a, 0x68, 0x0a, 0x09, 0x4a, 0x6f, 0x62, 0x53, 0x74, 0x61, 0x74, - 0x75, 0x73, 0x12, 0x16, 0x0a, 0x12, 0x4a, 0x4f, 0x42, 0x5f, 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, - 0x5f, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, 0x12, 0x16, 0x0a, 0x12, 0x4a, 0x4f, - 0x42, 0x5f, 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, 0x5f, 0x50, 0x45, 0x4e, 0x44, 0x49, 0x4e, 0x47, - 0x10, 0x01, 0x12, 0x16, 0x0a, 0x12, 0x4a, 0x4f, 0x42, 0x5f, 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, - 0x5f, 0x52, 0x55, 0x4e, 0x4e, 0x49, 0x4e, 0x47, 0x10, 0x02, 0x12, 0x13, 0x0a, 0x0f, 0x4a, 0x4f, - 0x42, 0x5f, 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, 0x5f, 0x44, 0x4f, 0x4e, 0x45, 0x10, 0x03, 0x2a, - 0x3b, 0x0a, 0x0a, 0x44, 0x61, 0x74, 0x61, 0x46, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x12, 0x17, 0x0a, - 0x13, 0x44, 0x41, 0x54, 0x41, 0x5f, 0x46, 0x4f, 0x52, 0x4d, 0x41, 0x54, 0x5f, 0x49, 0x4e, 0x56, - 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, 0x12, 0x14, 0x0a, 0x10, 0x44, 0x41, 0x54, 0x41, 0x5f, 0x46, - 0x4f, 0x52, 0x4d, 0x41, 0x54, 0x5f, 0x41, 0x56, 0x52, 0x4f, 0x10, 0x01, 0x32, 0x92, 0x03, 0x0a, - 0x0e, 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, - 0x6c, 0x0a, 0x13, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, - 0x6e, 0x67, 0x49, 0x6e, 0x66, 0x6f, 0x12, 0x29, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, + 0x64, 0x61, 0x74, 0x61, 0x46, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x22, 0xd4, 0x01, 0x0a, 0x0d, 0x44, + 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x12, 0x4a, 0x0a, 0x0b, + 0x66, 0x69, 0x6c, 0x65, 0x5f, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x18, 0x01, 0x20, 0x01, 0x28, + 0x0b, 0x32, 0x27, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, + 0x67, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x2e, + 0x46, 0x69, 0x6c, 0x65, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x48, 0x00, 0x52, 0x0a, 0x66, 0x69, + 0x6c, 0x65, 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x1a, 0x65, 0x0a, 0x0a, 0x46, 0x69, 0x6c, 0x65, + 0x53, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x12, 0x1b, 0x0a, 0x09, 0x66, 0x69, 0x6c, 0x65, 0x5f, 0x75, + 0x72, 0x69, 0x73, 0x18, 0x01, 0x20, 0x03, 0x28, 0x09, 0x52, 0x08, 0x66, 0x69, 0x6c, 0x65, 0x55, + 0x72, 0x69, 0x73, 0x12, 0x3a, 0x0a, 0x0b, 0x64, 0x61, 0x74, 0x61, 0x5f, 0x66, 0x6f, 0x72, 0x6d, + 0x61, 0x74, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0e, 0x32, 0x19, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x44, 0x61, 0x74, 0x61, 0x46, 0x6f, 0x72, + 0x6d, 0x61, 0x74, 0x52, 0x0a, 0x64, 0x61, 0x74, 0x61, 0x46, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x42, + 0x10, 0x0a, 0x0e, 0x64, 0x61, 0x74, 0x61, 0x73, 0x65, 0x74, 0x5f, 0x73, 0x6f, 0x75, 0x72, 0x63, + 0x65, 0x2a, 0x6f, 0x0a, 0x10, 0x46, 0x65, 0x61, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, + 0x67, 0x54, 0x79, 0x70, 0x65, 0x12, 0x1e, 0x0a, 0x1a, 0x46, 0x45, 0x41, 0x53, 0x54, 0x5f, 0x53, + 0x45, 0x52, 0x56, 0x49, 0x4e, 0x47, 0x5f, 0x54, 0x59, 0x50, 0x45, 0x5f, 0x49, 0x4e, 0x56, 0x41, + 0x4c, 0x49, 0x44, 0x10, 0x00, 0x12, 0x1d, 0x0a, 0x19, 0x46, 0x45, 0x41, 0x53, 0x54, 0x5f, 0x53, + 0x45, 0x52, 0x56, 0x49, 0x4e, 0x47, 0x5f, 0x54, 0x59, 0x50, 0x45, 0x5f, 0x4f, 0x4e, 0x4c, 0x49, + 0x4e, 0x45, 0x10, 0x01, 0x12, 0x1c, 0x0a, 0x18, 0x46, 0x45, 0x41, 0x53, 0x54, 0x5f, 0x53, 0x45, + 0x52, 0x56, 0x49, 0x4e, 0x47, 0x5f, 0x54, 0x59, 0x50, 0x45, 0x5f, 0x42, 0x41, 0x54, 0x43, 0x48, + 0x10, 0x02, 0x2a, 0x36, 0x0a, 0x07, 0x4a, 0x6f, 0x62, 0x54, 0x79, 0x70, 0x65, 0x12, 0x14, 0x0a, + 0x10, 0x4a, 0x4f, 0x42, 0x5f, 0x54, 0x59, 0x50, 0x45, 0x5f, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, + 0x44, 0x10, 0x00, 0x12, 0x15, 0x0a, 0x11, 0x4a, 0x4f, 0x42, 0x5f, 0x54, 0x59, 0x50, 0x45, 0x5f, + 0x44, 0x4f, 0x57, 0x4e, 0x4c, 0x4f, 0x41, 0x44, 0x10, 0x01, 0x2a, 0x68, 0x0a, 0x09, 0x4a, 0x6f, + 0x62, 0x53, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x16, 0x0a, 0x12, 0x4a, 0x4f, 0x42, 0x5f, 0x53, + 0x54, 0x41, 0x54, 0x55, 0x53, 0x5f, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, 0x12, + 0x16, 0x0a, 0x12, 0x4a, 0x4f, 0x42, 0x5f, 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, 0x5f, 0x50, 0x45, + 0x4e, 0x44, 0x49, 0x4e, 0x47, 0x10, 0x01, 0x12, 0x16, 0x0a, 0x12, 0x4a, 0x4f, 0x42, 0x5f, 0x53, + 0x54, 0x41, 0x54, 0x55, 0x53, 0x5f, 0x52, 0x55, 0x4e, 0x4e, 0x49, 0x4e, 0x47, 0x10, 0x02, 0x12, + 0x13, 0x0a, 0x0f, 0x4a, 0x4f, 0x42, 0x5f, 0x53, 0x54, 0x41, 0x54, 0x55, 0x53, 0x5f, 0x44, 0x4f, + 0x4e, 0x45, 0x10, 0x03, 0x2a, 0x3b, 0x0a, 0x0a, 0x44, 0x61, 0x74, 0x61, 0x46, 0x6f, 0x72, 0x6d, + 0x61, 0x74, 0x12, 0x17, 0x0a, 0x13, 0x44, 0x41, 0x54, 0x41, 0x5f, 0x46, 0x4f, 0x52, 0x4d, 0x41, + 0x54, 0x5f, 0x49, 0x4e, 0x56, 0x41, 0x4c, 0x49, 0x44, 0x10, 0x00, 0x12, 0x14, 0x0a, 0x10, 0x44, + 0x41, 0x54, 0x41, 0x5f, 0x46, 0x4f, 0x52, 0x4d, 0x41, 0x54, 0x5f, 0x41, 0x56, 0x52, 0x4f, 0x10, + 0x01, 0x32, 0x92, 0x03, 0x0a, 0x0e, 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x53, 0x65, 0x72, + 0x76, 0x69, 0x63, 0x65, 0x12, 0x6c, 0x0a, 0x13, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, + 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x49, 0x6e, 0x66, 0x6f, 0x12, 0x29, 0x2e, 0x66, 0x65, + 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x46, + 0x65, 0x61, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x49, 0x6e, 0x66, 0x6f, 0x52, + 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x2a, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, 0x53, - 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x49, 0x6e, 0x66, 0x6f, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, - 0x74, 0x1a, 0x2a, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, - 0x67, 0x2e, 0x47, 0x65, 0x74, 0x46, 0x65, 0x61, 0x73, 0x74, 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, - 0x67, 0x49, 0x6e, 0x66, 0x6f, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x66, 0x0a, - 0x11, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, - 0x65, 0x73, 0x12, 0x27, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, - 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, - 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x28, 0x2e, 0x66, 0x65, - 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4f, - 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, - 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x63, 0x0a, 0x10, 0x47, 0x65, 0x74, 0x42, 0x61, 0x74, 0x63, - 0x68, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x12, 0x26, 0x2e, 0x66, 0x65, 0x61, 0x73, - 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x42, 0x61, 0x74, - 0x63, 0x68, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, - 0x74, 0x1a, 0x27, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, - 0x67, 0x2e, 0x47, 0x65, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, - 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x45, 0x0a, 0x06, 0x47, 0x65, - 0x74, 0x4a, 0x6f, 0x62, 0x12, 0x1c, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, - 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x71, 0x75, 0x65, - 0x73, 0x74, 0x1a, 0x1d, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, - 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4a, 0x6f, 0x62, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, - 0x65, 0x42, 0x54, 0x0a, 0x0d, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, - 0x6e, 0x67, 0x42, 0x0f, 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x41, 0x50, 0x49, 0x50, 0x72, - 0x6f, 0x74, 0x6f, 0x5a, 0x32, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, - 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, - 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, - 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33, + 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x49, 0x6e, 0x66, 0x6f, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, + 0x73, 0x65, 0x12, 0x66, 0x0a, 0x11, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, + 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x12, 0x27, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, + 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, + 0x1a, 0x28, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, + 0x2e, 0x47, 0x65, 0x74, 0x4f, 0x6e, 0x6c, 0x69, 0x6e, 0x65, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, + 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, 0x63, 0x0a, 0x10, 0x47, 0x65, + 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x12, 0x26, + 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, + 0x65, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x46, 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, + 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x27, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, 0x73, + 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x42, 0x61, 0x74, 0x63, 0x68, 0x46, + 0x65, 0x61, 0x74, 0x75, 0x72, 0x65, 0x73, 0x52, 0x65, 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x12, + 0x45, 0x0a, 0x06, 0x47, 0x65, 0x74, 0x4a, 0x6f, 0x62, 0x12, 0x1c, 0x2e, 0x66, 0x65, 0x61, 0x73, + 0x74, 0x2e, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4a, 0x6f, 0x62, + 0x52, 0x65, 0x71, 0x75, 0x65, 0x73, 0x74, 0x1a, 0x1d, 0x2e, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x2e, 0x47, 0x65, 0x74, 0x4a, 0x6f, 0x62, 0x52, 0x65, + 0x73, 0x70, 0x6f, 0x6e, 0x73, 0x65, 0x42, 0x54, 0x0a, 0x0d, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2e, + 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x42, 0x0f, 0x53, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, + 0x41, 0x50, 0x49, 0x50, 0x72, 0x6f, 0x74, 0x6f, 0x5a, 0x32, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, + 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, + 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x66, + 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x65, 0x72, 0x76, 0x69, 0x6e, 0x67, 0x62, 0x06, 0x70, 0x72, + 0x6f, 0x74, 0x6f, 0x33, } var ( diff --git a/sdk/go/protos/feast/storage/Redis.pb.go b/sdk/go/protos/feast/storage/Redis.pb.go index 1dca28e26af..354a0f045b6 100644 --- a/sdk/go/protos/feast/storage/Redis.pb.go +++ b/sdk/go/protos/feast/storage/Redis.pb.go @@ -46,7 +46,7 @@ type RedisKey struct { sizeCache protoimpl.SizeCache unknownFields protoimpl.UnknownFields - // FeatureSet this row belongs to, this is defined as featureSetName:version. + // FeatureSet this row belongs to, this is defined as featureSetName. FeatureSet string `protobuf:"bytes,2,opt,name=feature_set,json=featureSet,proto3" json:"feature_set,omitempty"` // List of fields containing entity names and their respective values // contained within this feature row. The entities should be sorted diff --git a/sdk/go/protos/feast/types/FeatureRow.pb.go b/sdk/go/protos/feast/types/FeatureRow.pb.go index 769f219d32e..696f138459f 100644 --- a/sdk/go/protos/feast/types/FeatureRow.pb.go +++ b/sdk/go/protos/feast/types/FeatureRow.pb.go @@ -53,7 +53,7 @@ type FeatureRow struct { // will use to perform joins, determine latest values, and coalesce rows. EventTimestamp *timestamp.Timestamp `protobuf:"bytes,3,opt,name=event_timestamp,json=eventTimestamp,proto3" json:"event_timestamp,omitempty"` // Complete reference to the featureSet this featureRow belongs to, in the form of - // /:. This value will be used by the feast ingestion job to filter + // /. This value will be used by the feast ingestion job to filter // rows, and write the values to the correct tables. FeatureSet string `protobuf:"bytes,6,opt,name=feature_set,json=featureSet,proto3" json:"feature_set,omitempty"` } diff --git a/sdk/go/protos/tensorflow_metadata/proto/v0/path.pb.go b/sdk/go/protos/tensorflow_metadata/proto/v0/path.pb.go index 5b55e5bca5a..d609c5e89c0 100644 --- a/sdk/go/protos/tensorflow_metadata/proto/v0/path.pb.go +++ b/sdk/go/protos/tensorflow_metadata/proto/v0/path.pb.go @@ -112,14 +112,14 @@ var file_tensorflow_metadata_proto_v0_path_proto_rawDesc = []byte{ 0x61, 0x74, 0x68, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x16, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x22, 0x1a, 0x0a, 0x04, 0x50, 0x61, 0x74, 0x68, 0x12, 0x12, 0x0a, 0x04, 0x73, 0x74, 0x65, - 0x70, 0x18, 0x01, 0x20, 0x03, 0x28, 0x09, 0x52, 0x04, 0x73, 0x74, 0x65, 0x70, 0x42, 0x70, 0x0a, + 0x70, 0x18, 0x01, 0x20, 0x03, 0x28, 0x09, 0x52, 0x04, 0x73, 0x74, 0x65, 0x70, 0x42, 0x64, 0x0a, 0x1a, 0x6f, 0x72, 0x67, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, - 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x50, 0x01, 0x5a, 0x4d, 0x67, + 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x50, 0x01, 0x5a, 0x41, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, 0x72, 0x6f, - 0x74, 0x6f, 0x73, 0x2f, 0x74, 0x68, 0x69, 0x72, 0x64, 0x5f, 0x70, 0x61, 0x72, 0x74, 0x79, 0x2f, - 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x64, - 0x61, 0x74, 0x61, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2f, 0x76, 0x30, 0xf8, 0x01, 0x01, + 0x74, 0x6f, 0x73, 0x2f, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x5f, 0x6d, + 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2f, 0x76, 0x30, + 0xf8, 0x01, 0x01, } var ( diff --git a/sdk/go/protos/tensorflow_metadata/proto/v0/schema.pb.go b/sdk/go/protos/tensorflow_metadata/proto/v0/schema.pb.go index 5eec2f259d0..ab7ffc7201d 100644 --- a/sdk/go/protos/tensorflow_metadata/proto/v0/schema.pb.go +++ b/sdk/go/protos/tensorflow_metadata/proto/v0/schema.pb.go @@ -3476,14 +3476,13 @@ var file_tensorflow_metadata_proto_v0_schema_proto_rawDesc = []byte{ 0x57, 0x4e, 0x10, 0x00, 0x12, 0x09, 0x0a, 0x05, 0x42, 0x59, 0x54, 0x45, 0x53, 0x10, 0x01, 0x12, 0x07, 0x0a, 0x03, 0x49, 0x4e, 0x54, 0x10, 0x02, 0x12, 0x09, 0x0a, 0x05, 0x46, 0x4c, 0x4f, 0x41, 0x54, 0x10, 0x03, 0x12, 0x0a, 0x0a, 0x06, 0x53, 0x54, 0x52, 0x55, 0x43, 0x54, 0x10, 0x04, 0x42, - 0x70, 0x0a, 0x1a, 0x6f, 0x72, 0x67, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, + 0x64, 0x0a, 0x1a, 0x6f, 0x72, 0x67, 0x2e, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x2e, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2e, 0x76, 0x30, 0x50, 0x01, 0x5a, - 0x4d, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, + 0x41, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x67, 0x6f, 0x6a, 0x65, 0x6b, 0x2f, 0x66, 0x65, 0x61, 0x73, 0x74, 0x2f, 0x73, 0x64, 0x6b, 0x2f, 0x67, 0x6f, 0x2f, 0x70, - 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x74, 0x68, 0x69, 0x72, 0x64, 0x5f, 0x70, 0x61, 0x72, 0x74, - 0x79, 0x2f, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, 0x5f, 0x6d, 0x65, 0x74, - 0x61, 0x64, 0x61, 0x74, 0x61, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2f, 0x76, 0x30, 0xf8, 0x01, - 0x01, + 0x72, 0x6f, 0x74, 0x6f, 0x73, 0x2f, 0x74, 0x65, 0x6e, 0x73, 0x6f, 0x72, 0x66, 0x6c, 0x6f, 0x77, + 0x5f, 0x6d, 0x65, 0x74, 0x61, 0x64, 0x61, 0x74, 0x61, 0x2f, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x2f, + 0x76, 0x30, 0xf8, 0x01, 0x01, } var ( diff --git a/sdk/go/request.go b/sdk/go/request.go index a3a3fe6d71a..09a00f8a239 100644 --- a/sdk/go/request.go +++ b/sdk/go/request.go @@ -3,22 +3,19 @@ package feast import ( "fmt" "github.com/gojek/feast/sdk/go/protos/feast/serving" - "strconv" "strings" ) var ( // ErrInvalidFeatureName indicates that the user has provided a feature reference with the wrong structure or contents - ErrInvalidFeatureName = "invalid feature references %s provided, feature names must be in the format /:" + ErrInvalidFeatureName = "invalid feature references %s provided, feature names must be in the format /" ) // OnlineFeaturesRequest wrapper on feast.serving.GetOnlineFeaturesRequest. type OnlineFeaturesRequest struct { // Features is the list of features to obtain from Feast. Each feature can be given as // - // : // / - // /: // The only required components are the feature name and project. Features []string @@ -50,7 +47,7 @@ func (r OnlineFeaturesRequest) buildRequest() (*serving.GetOnlineFeaturesRequest }, nil } -// buildFeatures create a slice of FeatureReferences from a slice of "/:" +// buildFeatures create a slice of FeatureReferences from a slice of "/" // It returns an error when the format is invalid func buildFeatures(featureReferences []string, defaultProject string) ([]*serving.FeatureReference, error) { var features []*serving.FeatureReference @@ -58,41 +55,25 @@ func buildFeatures(featureReferences []string, defaultProject string) ([]*servin for _, featureRef := range featureReferences { var project string var name string - var version int - var featureSplit []string projectSplit := strings.Split(featureRef, "/") if len(projectSplit) == 2 { project = projectSplit[0] - featureSplit = strings.Split(projectSplit[1], ":") + name = projectSplit[1] } else if len(projectSplit) == 1 { project = defaultProject - featureSplit = strings.Split(projectSplit[0], ":") + name = projectSplit[0] } else { return nil, fmt.Errorf(ErrInvalidFeatureName, featureRef) } - if len(featureSplit) == 2 { - name = featureSplit[0] - v, err := strconv.Atoi(featureSplit[1]) - if err != nil { - return nil, fmt.Errorf(ErrInvalidFeatureName, featureRef) - } - version = v - } else if len(featureSplit) == 1 { - name = featureSplit[0] - } else { - return nil, fmt.Errorf(ErrInvalidFeatureName, featureRef) - } - - if project == "" || name == "" || version < 0 { + if project == "" || name == "" { return nil, fmt.Errorf(ErrInvalidFeatureName, featureRef) } features = append(features, &serving.FeatureReference{ Name: name, - Version: int32(version), Project: project, }) } diff --git a/sdk/go/request_test.go b/sdk/go/request_test.go index b6866638670..5c149fe1583 100644 --- a/sdk/go/request_test.go +++ b/sdk/go/request_test.go @@ -20,7 +20,7 @@ func TestGetOnlineFeaturesRequest(t *testing.T) { { name: "valid", req: OnlineFeaturesRequest{ - Features: []string{"my_project_1/feature1:1", "my_project_2/feature1:1", "my_project_4/feature3", "feature2:2", "feature2"}, + Features: []string{"my_project_1/feature1", "my_project_2/feature1", "my_project_4/feature3", "feature2", "feature2"}, Entities: []Row{ {"entity1": Int64Val(1), "entity2": StrVal("bob")}, {"entity1": Int64Val(1), "entity2": StrVal("annie")}, @@ -33,27 +33,22 @@ func TestGetOnlineFeaturesRequest(t *testing.T) { { Project: "my_project_1", Name: "feature1", - Version: 1, }, { Project: "my_project_2", Name: "feature1", - Version: 1, }, { Project: "my_project_4", Name: "feature3", - Version: 0, }, { Project: "my_project_3", Name: "feature2", - Version: 2, }, { Project: "my_project_3", Name: "feature2", - Version: 0, }, }, EntityRows: []*serving.GetOnlineFeaturesRequest_EntityRow{ @@ -92,7 +87,6 @@ func TestGetOnlineFeaturesRequest(t *testing.T) { { Project: "project", Name: "feature1", - Version: 0, }, }, EntityRows: []*serving.GetOnlineFeaturesRequest_EntityRow{}, @@ -113,21 +107,12 @@ func TestGetOnlineFeaturesRequest(t *testing.T) { { name: "invalid_feature_name/wrong_format", req: OnlineFeaturesRequest{ - Features: []string{"fs1:3:feature1"}, + Features: []string{"/fs1:feature1"}, Entities: []Row{}, Project: "my_project", }, wantErr: true, - err: fmt.Errorf(ErrInvalidFeatureName, "fs1:3:feature1"), - }, - { - name: "invalid_feature_name/invalid_version", - req: OnlineFeaturesRequest{ - Features: []string{"project/a:feature1"}, - Entities: []Row{}, - }, - wantErr: true, - err: fmt.Errorf(ErrInvalidFeatureName, "project/a:feature1"), + err: fmt.Errorf(ErrInvalidFeatureName, "/fs1:feature1"), }, } for _, tc := range tt { diff --git a/sdk/go/response_test.go b/sdk/go/response_test.go index 5aa2c276d61..f660130475f 100644 --- a/sdk/go/response_test.go +++ b/sdk/go/response_test.go @@ -67,7 +67,7 @@ func TestOnlineFeaturesResponseToInt64Array(t *testing.T) { { name: "length mismatch", args: args{ - order: []string{"fs:1:feature2", "fs:1:feature1"}, + order: []string{"fs:feature2", "fs:feature1"}, fillNa: []int64{-1}, }, want: nil, diff --git a/sdk/java/src/main/java/com/gojek/feast/FeastClient.java b/sdk/java/src/main/java/com/gojek/feast/FeastClient.java index 8014231836e..4e18b651657 100644 --- a/sdk/java/src/main/java/com/gojek/feast/FeastClient.java +++ b/sdk/java/src/main/java/com/gojek/feast/FeastClient.java @@ -62,7 +62,7 @@ public GetFeastServingInfoResponse getFeastServingInfo() { *

    See {@link #getOnlineFeatures(List, List, String)} * * @param features list of string feature references to retrieve, feature reference follows this - * format [project]/[name]:[version] + * format [project]/[name] * @param rows list of {@link Row} to select the entities to retrieve the features for * @param defaultProject {@link String} Default project to find features in if not provided in * feature reference. @@ -76,11 +76,11 @@ public List getOnlineFeatures(List features, List rows, String * Get online features from Feast. * *

    Example of retrieving online features for the driver project, with features driver_id and - * driver_name, both version 1 + * driver_name * *

    {@code
        * FeastClient client = FeastClient.create("localhost", 6566);
    -   * List requestedFeatureIds = Arrays.asList("driver/driver_id:1", "driver/driver_name:1");
    +   * List requestedFeatureIds = Arrays.asList("driver/driver_id", "driver/driver_name");
        * List requestedRows =
        *         Arrays.asList(Row.create().set("driver_id", 123), Row.create().set("driver_id", 456));
        * List retrievedFeatures = client.getOnlineFeatures(requestedFeatureIds, requestedRows);
    @@ -88,7 +88,7 @@ public List getOnlineFeatures(List features, List rows, String
        * }
    * * @param featureRefStrings list of feature refs to retrieve, feature refs follow this format - * [project]/[name]:[version] + * [project]/[name] * @param rows list of {@link Row} to select the entities to retrieve the features for * @param defaultProject {@link String} Default project to find features in if not provided in * feature reference. diff --git a/sdk/java/src/main/java/com/gojek/feast/RequestUtil.java b/sdk/java/src/main/java/com/gojek/feast/RequestUtil.java index 874196e92bd..ebea625ff1e 100644 --- a/sdk/java/src/main/java/com/gojek/feast/RequestUtil.java +++ b/sdk/java/src/main/java/com/gojek/feast/RequestUtil.java @@ -34,58 +34,30 @@ public static List createFeatureRefs( for (String featureRefString : featureRefStrings) { String project; String name; - int version = 0; - String[] featureSplit; String[] projectSplit = featureRefString.split("/"); - if (projectSplit.length == 2) { - project = projectSplit[0]; - featureSplit = projectSplit[1].split(":"); - } else if (projectSplit.length == 1) { + if (projectSplit.length == 1) { project = defaultProject; - featureSplit = projectSplit[0].split(":"); - } else { - throw new IllegalArgumentException( - String.format( - "Feature id '%s' has invalid format. Expected format: ::.", - featureRefString)); - } - - if (featureSplit.length == 2) { - name = featureSplit[0]; - try { - version = Integer.parseInt(featureSplit[1]); - } catch (NumberFormatException e) { - throw new IllegalArgumentException( - String.format( - "Feature id '%s' contains invalid version. Expected format: /:.", - featureRefString)); - } - } else if (featureSplit.length == 1) { - name = featureSplit[0]; + name = projectSplit[0]; + } else if (projectSplit.length == 2) { + project = projectSplit[0]; + name = projectSplit[1]; } else { throw new IllegalArgumentException( String.format( - "Feature id '%s' has invalid format. Expected format: /:.", + "Feature id '%s' has invalid format. Expected format: /.", featureRefString)); } - if (project.isEmpty() || name.isEmpty() || version < 0) { + if (project.isEmpty() || name.isEmpty() || name.contains(":")) { throw new IllegalArgumentException( String.format( - "Feature id '%s' has invalid format. Expected format: /:.", + "Feature id '%s' has invalid format. Expected format: /.", featureRefString)); } - featureRefs.add( - FeatureReference.newBuilder() - .setName(name) - .setProject(project) - .setVersion(version) - .build()); + featureRefs.add(FeatureReference.newBuilder().setName(name).setProject(project).build()); } - - ; return featureRefs; } } diff --git a/sdk/java/src/test/java/com/gojek/feast/RequestUtilTest.java b/sdk/java/src/test/java/com/gojek/feast/RequestUtilTest.java index 3b9429ad8f6..a471609149b 100644 --- a/sdk/java/src/test/java/com/gojek/feast/RequestUtilTest.java +++ b/sdk/java/src/test/java/com/gojek/feast/RequestUtilTest.java @@ -36,40 +36,35 @@ class RequestUtilTest { private static Stream provideValidFeatureIds() { return Stream.of( Arguments.of( - Collections.singletonList("driver_project/driver_id:1"), + Collections.singletonList("driver_project/driver_id"), Collections.singletonList( FeatureReference.newBuilder() .setProject("driver_project") .setName("driver_id") - .setVersion(1) .build())), Arguments.of( - Arrays.asList("driver_project/driver_id:1", "driver_project/driver_name:1"), + Arrays.asList("driver_project/driver_id", "driver_project/driver_name"), Arrays.asList( FeatureReference.newBuilder() .setProject("driver_project") .setName("driver_id") - .setVersion(1) .build(), FeatureReference.newBuilder() .setProject("driver_project") .setName("driver_name") - .setVersion(1) .build())), Arguments.of( Arrays.asList( - "driver_project/driver_id:1", - "driver_project/driver_name:1", + "driver_project/driver_id", + "driver_project/driver_name", "booking_project/driver_name"), Arrays.asList( FeatureReference.newBuilder() .setProject("driver_project") - .setVersion(1) .setName("driver_id") .build(), FeatureReference.newBuilder() .setProject("driver_project") - .setVersion(1) .setName("driver_name") .build(), FeatureReference.newBuilder() @@ -96,7 +91,7 @@ void createFeatureSets_ShouldReturnFeatureSetsForValidFeatureIds( private static Stream provideInvalidFeatureRefs() { return Stream.of( - Arguments.of(Collections.singletonList("missing:bad_version")), + Arguments.of(Collections.singletonList("/noproject")), Arguments.of(Collections.singletonList(""))); } diff --git a/sdk/python/feast/cli.py b/sdk/python/feast/cli.py index 489d28fe833..2fd5a4cdf56 100644 --- a/sdk/python/feast/cli.py +++ b/sdk/python/feast/cli.py @@ -129,11 +129,11 @@ def feature_set_list(): table = [] for fs in feast_client.list_feature_sets(): - table.append([fs.name, fs.version, repr(fs)]) + table.append([fs.name, repr(fs)]) from tabulate import tabulate - print(tabulate(table, headers=["NAME", "VERSION", "REFERENCE"], tablefmt="plain")) + print(tabulate(table, headers=["NAME", "REFERENCE"], tablefmt="plain")) @feature_set.command("apply") @@ -155,17 +155,14 @@ def feature_set_create(filename): @feature_set.command("describe") @click.argument("name", type=click.STRING) -@click.argument("version", type=click.INT) -def feature_set_describe(name: str, version: int): +def feature_set_describe(name: str): """ Describe a feature set """ feast_client = Client() # type: Client - fs = feast_client.get_feature_set(name=name, version=version) + fs = feast_client.get_feature_set(name=name) if not fs: - print( - f'Feature set with name "{name}" and version "{version}" could not be found' - ) + print(f'Feature set with name "{name}" could not be found') return print(yaml.dump(yaml.safe_load(str(fs)), default_flow_style=False, sort_keys=False)) @@ -329,9 +326,6 @@ def ingest_job_restart(job_id: str): @click.option( "--name", "-n", help="Feature set name to ingest data into", required=True ) -@click.option( - "--version", "-v", help="Feature set version to ingest data into", type=int -) @click.option( "--filename", "-f", @@ -345,13 +339,13 @@ def ingest_job_restart(job_id: str): type=click.Choice(["CSV"], case_sensitive=False), help="Type of file to ingest. Defaults to CSV.", ) -def ingest(name, version, filename, file_type): +def ingest(name, filename, file_type): """ Ingest feature data into a feature set """ feast_client = Client() # type: Client - feature_set = feast_client.get_feature_set(name=name, version=version) + feature_set = feast_client.get_feature_set(name=name) feature_set.ingest_file(file_path=filename) diff --git a/sdk/python/feast/client.py b/sdk/python/feast/client.py index 8d66f58c06c..89a89fd2a8d 100644 --- a/sdk/python/feast/client.py +++ b/sdk/python/feast/client.py @@ -400,9 +400,10 @@ def _apply_feature_set(self, feature_set: FeatureSet): # If the feature set has changed, update the local copy if apply_fs_response.status == ApplyFeatureSetResponse.Status.CREATED: - print( - f'Feature set updated/created: "{applied_fs.name}:{applied_fs.version}"' - ) + print(f'Feature set created: "{applied_fs.name}"') + + if apply_fs_response.status == ApplyFeatureSetResponse.Status.UPDATED: + print(f'Feature set updated: "{applied_fs.name}"') # If no change has been applied, do nothing if apply_fs_response.status == ApplyFeatureSetResponse.Status.NO_CHANGE: @@ -412,7 +413,7 @@ def _apply_feature_set(self, feature_set: FeatureSet): feature_set._update_from_feature_set(applied_fs) def list_feature_sets( - self, project: str = None, name: str = None, version: str = None + self, project: str = None, name: str = None, ) -> List[FeatureSet]: """ Retrieve a list of feature sets from Feast Core @@ -420,7 +421,6 @@ def list_feature_sets( Args: project: Filter feature sets based on project name name: Filter feature sets based on feature set name - version: Filter feature sets based on version numbf, Returns: List of feature sets @@ -436,12 +436,7 @@ def list_feature_sets( if name is None: name = "*" - if version is None: - version = "*" - - filter = ListFeatureSetsRequest.Filter( - project=project, feature_set_name=name, feature_set_version=version - ) + filter = ListFeatureSetsRequest.Filter(project=project, feature_set_name=name) # Get latest feature sets from Feast Core feature_set_protos = self._core_service_stub.ListFeatureSets( @@ -457,16 +452,14 @@ def list_feature_sets( return feature_sets def get_feature_set( - self, name: str, version: int = None, project: str = None + self, name: str, project: str = None ) -> Union[FeatureSet, None]: """ - Retrieves a feature set. If no version is specified then the latest - version will be returned. + Retrieves a feature set. Args: project: Feast project that this feature set belongs to name: Name of feature set - version: Version of feature set Returns: Returns either the specified feature set, or raises an exception if @@ -480,14 +473,9 @@ def get_feature_set( else: raise ValueError("No project has been configured.") - if version is None: - version = 0 - try: get_feature_set_response = self._core_service_stub.GetFeatureSet( - GetFeatureSetRequest( - project=project, name=name.strip(), version=int(version) - ) + GetFeatureSetRequest(project=project, name=name.strip()) ) # type: GetFeatureSetResponse except grpc.RpcError as e: raise grpc.RpcError(e.details()) @@ -519,7 +507,7 @@ def get_batch_features( feature_refs (List[str]): List of feature references that will be returned for each entity. Each feature reference should have the following format - "project/feature:version". + "project/feature". entity_rows (Union[pd.DataFrame, str]): Pandas dataframe containing entities and a 'datetime' column. @@ -539,7 +527,7 @@ def get_batch_features( >>> from datetime import datetime >>> >>> feast_client = Client(core_url="localhost:6565", serving_url="localhost:6566") - >>> feature_refs = ["my_project/bookings_7d:1", "booking_14d"] + >>> feature_refs = ["my_project/bookings_7d", "booking_14d"] >>> entity_rows = pd.DataFrame( >>> { >>> "datetime": [pd.datetime.now() for _ in range(3)], @@ -626,11 +614,11 @@ def get_online_features( Args: feature_refs: List of feature references in the following format - [project]/[feature_name]:[version]. Only the feature name + [project]/[feature_name]. Only the feature name is a required component in the reference. example: - ["my_project/my_feature_1:3", - "my_project3/my_feature_4:1",] + ["my_project/my_feature_1", + "my_feature_4",] entity_rows: List of GetFeaturesRequest.EntityRow where each row contains entities. Timestamp should not be set for online retrieval. All entity types within a feature @@ -731,18 +719,16 @@ def ingest( feature_set: Union[str, FeatureSet], source: Union[pd.DataFrame, str], chunk_size: int = 10000, - version: int = None, max_workers: int = max(CPU_COUNT - 1, 1), disable_progress_bar: bool = False, timeout: int = KAFKA_CHUNK_PRODUCTION_TIMEOUT, - ) -> None: + ) -> str: """ Loads feature data into Feast for a specific feature set. Args: feature_set (typing.Union[str, feast.feature_set.FeatureSet]): Feature set object or the string name of the feature set - (without a version). source (typing.Union[pd.DataFrame, str]): Either a file path or Pandas Dataframe to ingest into Feast @@ -754,9 +740,6 @@ def ingest( chunk_size (int): Amount of rows to load and ingest at a time. - version (int): - Feature set version. - max_workers (int): Number of worker processes to use to encode values. @@ -767,14 +750,12 @@ def ingest( Timeout in seconds to wait for completion. Returns: - None: - None + str: + ingestion id for this dataset """ if isinstance(feature_set, FeatureSet): name = feature_set.name - if version is None: - version = feature_set.version elif isinstance(feature_set, str): name = feature_set else: @@ -793,7 +774,7 @@ def ingest( while True: if timeout is not None and time.time() - current_time >= timeout: raise TimeoutError("Timed out waiting for feature set to be ready") - feature_set = self.get_feature_set(name, version) + feature_set = self.get_feature_set(name) if ( feature_set is not None and feature_set.status == FeatureSetStatus.STATUS_READY @@ -849,7 +830,7 @@ def ingest( print("Removing temporary file(s)...") shutil.rmtree(dir_path) - return None + return ingestion_id def _build_feature_references( @@ -861,7 +842,7 @@ def _build_feature_references( Args: feature_refs: List of feature reference strings - ("project/feature:version") + ("project/feature") default_project: This project will be used if the project name is not provided in the feature reference """ @@ -870,12 +851,11 @@ def _build_feature_references( for feature_ref in feature_refs: project_split = feature_ref.split("/") - version = 0 if len(project_split) == 2: - project, feature_version = project_split + project, name = project_split elif len(project_split) == 1: - feature_version = project_split[0] + name = project_split[0] if default_project is None: raise ValueError( f"No project specified in {feature_ref} and no default project provided" @@ -883,26 +863,20 @@ def _build_feature_references( project = default_project else: raise ValueError( - f'Could not parse feature ref {feature_ref}, expecting "project/feature:version"' + f'Could not parse feature ref {feature_ref}, expecting "project/feature"' ) - feature_split = feature_version.split(":") - if len(feature_split) == 2: - name, version = feature_split - version = int(version) - elif len(feature_split) == 1: - name = feature_split[0] - else: + if len(project) == 0 or len(name) == 0: raise ValueError( - f'Could not parse feature ref {feature_ref}, expecting "project/feature:version"' + f'Could not parse feature ref {feature_ref}, expecting "project/feature"' ) - if len(project) == 0 or len(name) == 0 or version < 0: + if ":" in name: raise ValueError( - f'Could not parse feature ref {feature_ref}, expecting "project/feature:version"' + f'Could not parse feature ref {feature_ref}, expecting "project/feature". Versions were deprecated in v0.5.0.' ) - features.append(FeatureReference(project=project, name=name, version=version)) + features.append(FeatureReference(project=project, name=name)) return features @@ -916,7 +890,7 @@ def _generate_ingestion_id(feature_set: FeatureSet) -> str: Returns: UUID unique to current time and the feature set provided. """ - uuid_str = f"{feature_set.name}_{feature_set.version}_{int(time.time())}" + uuid_str = f"{feature_set.name}_{int(time.time())}" return str(uuid.uuid3(uuid.NAMESPACE_DNS, uuid_str)) diff --git a/sdk/python/feast/feature_set.py b/sdk/python/feast/feature_set.py index 973c2a52a57..3c77aa1db5e 100644 --- a/sdk/python/feast/feature_set.py +++ b/sdk/python/feast/feature_set.py @@ -68,7 +68,6 @@ def __init__( else: self._source = source self._max_age = max_age - self._version = None self._status = None self._created_timestamp = None @@ -195,20 +194,6 @@ def source(self, source: Source): """ self._source = source - @property - def version(self): - """ - Returns the version of this feature set - """ - return self._version - - @version.setter - def version(self, version): - """ - Sets the version of this feature set - """ - self._version = version - @property def max_age(self): """ @@ -621,7 +606,6 @@ def _update_from_feature_set(self, feature_set): self.name = feature_set.name self.project = feature_set.project - self.version = feature_set.version self.source = feature_set.source self.max_age = feature_set.max_age self.features = feature_set.features @@ -809,7 +793,6 @@ def from_proto(cls, feature_set_proto: FeatureSetProto): if len(feature_set_proto.spec.project) == 0 else feature_set_proto.spec.project, ) - feature_set._version = feature_set_proto.spec.version feature_set._status = feature_set_proto.meta.status feature_set._created_timestamp = feature_set_proto.meta.created_timestamp return feature_set @@ -828,7 +811,6 @@ def to_proto(self) -> FeatureSetProto: spec = FeatureSetSpecProto( name=self.name, - version=self.version, project=self.project, max_age=self.max_age, source=self.source.to_proto() if self.source is not None else None, @@ -852,10 +834,8 @@ class FeatureSetRef: Represents a reference to a featureset """ - def __init__(self, project: str = None, name: str = None, version: int = None): - self.proto = FeatureSetReferenceProto( - project=project, name=name, version=version - ) + def __init__(self, project: str = None, name: str = None): + self.proto = FeatureSetReferenceProto(project=project, name=name) @property def project(self) -> str: @@ -871,13 +851,6 @@ def name(self) -> str: """ return self.proto.name - @property - def version(self) -> int: - """ - Get the version of feature set referenced by this reference - """ - return self.proto.version - @classmethod def from_feature_set(cls, feature_set: FeatureSet): """ @@ -889,7 +862,7 @@ def from_feature_set(cls, feature_set: FeatureSet): Returns: FeatureSetRef that refers to the given feature set """ - return cls(feature_set.project, feature_set.name, feature_set.version) + return cls(feature_set.project, feature_set.name) @classmethod def from_str(cls, ref_str: str): @@ -903,15 +876,13 @@ def from_str(cls, ref_str: str): Returns: FeatureSetRef constructed from the string """ + project = "" if "/" in ref_str: project, ref_str = ref_str.split("/") - if ":" in ref_str: - ref_str, version_str = ref_str.split(":") - name = ref_str - return cls(project, name, int(version_str)) + return cls(project, ref_str) - def to_proto(self, arg1) -> FeatureSetReferenceProto: + def to_proto(self) -> FeatureSetReferenceProto: """ Convert and return this feature set reference to protobuf. @@ -926,14 +897,12 @@ def __str__(self): def __repr__(self): # return string representation of the reference - # [project/]name[:version] + # [project/]name ref_str = "" if self.proto.project: ref_str += self.proto.project + "/" if self.proto.name: ref_str += self.proto.name - if self.proto.version: - ref_str += ":" + str(self.proto.version).strip() return ref_str def __eq__(self, other): diff --git a/sdk/python/feast/loaders/ingest.py b/sdk/python/feast/loaders/ingest.py index 34d0356ea78..b439dbd3027 100644 --- a/sdk/python/feast/loaders/ingest.py +++ b/sdk/python/feast/loaders/ingest.py @@ -44,7 +44,7 @@ def _encode_pa_tables( Parquet file must have more than one row group. feature_set (str): - Feature set reference in the format f"{project}/{name}:{version}". + Feature set reference in the format f"{project}/{name}". fields (dict[str, enum.Enum.ValueType]): A mapping of field names to their value types. @@ -135,7 +135,7 @@ def get_feature_row_chunks( Iterable list of byte encoded FeatureRow(s). """ - feature_set = f"{fs.project}/{fs.name}:{fs.version}" + feature_set = f"{fs.project}/{fs.name}" field_map = {field.name: field.dtype for field in fs.fields.values()} diff --git a/sdk/python/feast/type_map.py b/sdk/python/feast/type_map.py index 8df0499239a..85def25fcb9 100644 --- a/sdk/python/feast/type_map.py +++ b/sdk/python/feast/type_map.py @@ -147,7 +147,7 @@ def convert_series_to_proto_values(row: pd.Series): event_timestamp=_pd_datetime_to_timestamp_proto( dataframe[DATETIME_COLUMN].dtype, row[DATETIME_COLUMN] ), - feature_set=feature_set.name + ":" + str(feature_set.version), + feature_set=feature_set.project + "/" + feature_set.name, ) for field_name, field in feature_set.fields.items(): @@ -185,11 +185,7 @@ def convert_dict_to_proto_values( event_timestamp=_pd_datetime_to_timestamp_proto( df_datetime_dtype, row[DATETIME_COLUMN] ), - feature_set=feature_set.project - + "/" - + feature_set.name - + ":" - + str(feature_set.version), + feature_set=f"{feature_set.project}/{feature_set.name}", ) for field_name, field in feature_set.fields.items(): diff --git a/sdk/python/tests/feast_core_server.py b/sdk/python/tests/feast_core_server.py index b6efe2cb6d1..3ac1b17d003 100644 --- a/sdk/python/tests/feast_core_server.py +++ b/sdk/python/tests/feast_core_server.py @@ -40,21 +40,12 @@ def ListFeatureSets(self, request: ListFeatureSetsRequest, context): or request.filter.feature_set_name == "*" or fs.spec.name == request.filter.feature_set_name ) - and ( - not request.filter.feature_set_version - or str(fs.spec.version) == request.filter.feature_set_version - or request.filter.feature_set_version == "*" - ) ] return ListFeatureSetsResponse(feature_sets=filtered_feature_set_response) def ApplyFeatureSet(self, request: ApplyFeatureSetRequest, context): feature_set = request.feature_set - if feature_set.spec.version is None: - feature_set.spec.version = 1 - else: - feature_set.spec.version = feature_set.spec.version + 1 if feature_set.spec.source.type == SourceTypeProto.INVALID: feature_set.spec.source.kafka_source_config.CopyFrom( diff --git a/sdk/python/tests/feast_serving_server.py b/sdk/python/tests/feast_serving_server.py index 364c1907141..983e74e8850 100644 --- a/sdk/python/tests/feast_serving_server.py +++ b/sdk/python/tests/feast_serving_server.py @@ -67,7 +67,6 @@ def GetOnlineFeatures(self, request: GetOnlineFeaturesRequest, context): feature_data_sets=[ GetOnlineFeaturesResponse.FeatureDataSet( name="feature_set_1", - version="1", feature_rows=[ FeatureRowProto.FeatureRow( feature_set="feature_set_1", diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index 3082265eccf..e87c8573353 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -198,7 +198,7 @@ def test_get_online_features(self, mocked_client, mocker): fields = dict() for feature_num in range(1, 10): - fields[f"my_project/feature_{str(feature_num)}:1"] = ValueProto.Value( + fields[f"my_project/feature_{str(feature_num)}"] = ValueProto.Value( int64_val=feature_num ) field_values = GetOnlineFeaturesResponse.FieldValues(fields=fields) @@ -222,21 +222,21 @@ def test_get_online_features(self, mocked_client, mocker): response = mocked_client.get_online_features( entity_rows=entity_rows, feature_refs=[ - "my_project/feature_1:1", - "my_project/feature_2:1", - "my_project/feature_3:1", - "my_project/feature_4:1", - "my_project/feature_5:1", - "my_project/feature_6:1", - "my_project/feature_7:1", - "my_project/feature_8:1", - "my_project/feature_9:1", + "my_project/feature_1", + "my_project/feature_2", + "my_project/feature_3", + "my_project/feature_4", + "my_project/feature_5", + "my_project/feature_6", + "my_project/feature_7", + "my_project/feature_8", + "my_project/feature_9", ], ) # type: GetOnlineFeaturesResponse assert ( - response.field_values[0].fields["my_project/feature_1:1"].int64_val == 1 - and response.field_values[0].fields["my_project/feature_9:1"].int64_val == 9 + response.field_values[0].fields["my_project/feature_1"].int64_val == 1 + and response.field_values[0].fields["my_project/feature_9"].int64_val == 9 ) @pytest.mark.parametrize( @@ -257,7 +257,6 @@ def test_get_feature_set(self, mocked_client, mocker): feature_set=FeatureSetProto( spec=FeatureSetSpecProto( name="my_feature_set", - version=2, max_age=Duration(seconds=3600), features=[ FeatureSpecProto( @@ -287,11 +286,10 @@ def test_get_feature_set(self, mocked_client, mocker): ), ) mocked_client.set_project("my_project") - feature_set = mocked_client.get_feature_set("my_feature_set", version=2) + feature_set = mocked_client.get_feature_set("my_feature_set") assert ( feature_set.name == "my_feature_set" - and feature_set.version == 2 and feature_set.fields["my_feature_1"].name == "my_feature_1" and feature_set.fields["my_feature_1"].dtype == ValueType.FLOAT and feature_set.fields["my_entity_1"].name == "my_entity_1" @@ -422,7 +420,6 @@ def test_stop_ingest_job(self, mocked_client, mocker): # feature_set=FeatureSetProto( # spec=FeatureSetSpecProto( # name="customer_fs", - # version=1, # project="my_project", # entities=[ # EntitySpecProto( @@ -454,8 +451,8 @@ def test_stop_ingest_job(self, mocked_client, mocker): # "datetime": [datetime.utcnow() for _ in range(3)], # "customer": [1001, 1002, 1003], # "transaction": [1001, 1002, 1003], - # "my_project/customer_feature_1:1": [1001, 1002, 1003], - # "my_project/customer_feature_2:1": [1001, 1002, 1003], + # "my_project/customer_feature_1": [1001, 1002, 1003], + # "my_project/customer_feature_2": [1001, 1002, 1003], # } # ) # @@ -511,8 +508,8 @@ def test_stop_ingest_job(self, mocked_client, mocker): # } # ), # feature_refs=[ - # "my_project/customer_feature_1:1", - # "my_project/customer_feature_2:1", + # "my_project/customer_feature_1", + # "my_project/customer_feature_2", # ], # ) # type: Job # @@ -521,10 +518,10 @@ def test_stop_ingest_job(self, mocked_client, mocker): # actual_dataframe = response.to_dataframe() # # assert actual_dataframe[ - # ["my_project/customer_feature_1:1", "my_project/customer_feature_2:1"] + # ["my_project/customer_feature_1", "my_project/customer_feature_2"] # ].equals( # expected_dataframe[ - # ["my_project/customer_feature_1:1", "my_project/customer_feature_2:1"] + # ["my_project/customer_feature_1", "my_project/customer_feature_2"] # ] # ) diff --git a/sdk/python/tests/test_feature_set.py b/sdk/python/tests/test_feature_set.py index a2cc12fe113..04e75c9e76d 100644 --- a/sdk/python/tests/test_feature_set.py +++ b/sdk/python/tests/test_feature_set.py @@ -268,15 +268,13 @@ def make_tfx_schema_domain_info_inline(schema): class TestFeatureSetRef: def test_from_feature_set(self): feature_set = FeatureSet("test", "test") - feature_set.version = 2 ref = FeatureSetRef.from_feature_set(feature_set) assert ref.name == "test" assert ref.project == "test" - assert ref.version == 2 def test_str_ref(self): - original_ref = FeatureSetRef(project="test", name="test", version=2) + original_ref = FeatureSetRef(project="test", name="test") ref_str = repr(original_ref) parsed_ref = FeatureSetRef.from_str(ref_str) assert original_ref == parsed_ref diff --git a/serving/README.md b/serving/README.md index f88f30923b2..39eef311033 100644 --- a/serving/README.md +++ b/serving/README.md @@ -28,7 +28,6 @@ grpc_cli call localhost:6566 GetFeastServingType '' grpc_cli call localhost:6565 ApplyFeatureSet ' feature_set { name: "driver" - version: 1 entities { name: "driver_id" value_type: STRING @@ -53,14 +52,12 @@ feature_set { grpc_cli call localhost:6565 GetFeatureSets ' filter { feature_set_name: "driver" - feature_set_version: "1" } ' grpc_cli call localhost:6566 GetBatchFeatures ' feature_sets { name: "driver" - version: 1 feature_names: "booking_completed_count" max_age { seconds: 86400 diff --git a/serving/src/main/java/feast/serving/config/FeastProperties.java b/serving/src/main/java/feast/serving/config/FeastProperties.java index b7fd0a9fed7..9eec333a148 100644 --- a/serving/src/main/java/feast/serving/config/FeastProperties.java +++ b/serving/src/main/java/feast/serving/config/FeastProperties.java @@ -386,7 +386,6 @@ public StoreProto.Store.Subscription toProto() { return StoreProto.Store.Subscription.newBuilder() .setName(getName()) .setProject(getProject()) - .setVersion(getVersion()) .build(); } } diff --git a/serving/src/main/java/feast/serving/service/OnlineServingService.java b/serving/src/main/java/feast/serving/service/OnlineServingService.java index 30addd2b9f2..28885ca4452 100644 --- a/serving/src/main/java/feast/serving/service/OnlineServingService.java +++ b/serving/src/main/java/feast/serving/service/OnlineServingService.java @@ -133,9 +133,7 @@ public GetOnlineFeaturesResponse getOnlineFeatures(GetOnlineFeaturesRequest requ } private void populateStaleKeyCountMetrics(String project, FeatureReference ref) { - Metrics.staleKeyCount - .labels(project, RefUtil.generateFeatureStringRefWithoutProject(ref)) - .inc(); + Metrics.staleKeyCount.labels(project, ref.getName()).inc(); } private void populateRequestCountMetrics(FeatureSetRequest featureSetRequest) { @@ -143,11 +141,7 @@ private void populateRequestCountMetrics(FeatureSetRequest featureSetRequest) { featureSetRequest .getFeatureReferences() .parallelStream() - .forEach( - ref -> - Metrics.requestCount - .labels(project, RefUtil.generateFeatureStringRefWithoutProject(ref)) - .inc()); + .forEach(ref -> Metrics.requestCount.labels(project, ref.getName()).inc()); } @Override diff --git a/serving/src/main/java/feast/serving/specs/CachedSpecService.java b/serving/src/main/java/feast/serving/specs/CachedSpecService.java index a117754e844..5dc57de7c2a 100644 --- a/serving/src/main/java/feast/serving/specs/CachedSpecService.java +++ b/serving/src/main/java/feast/serving/specs/CachedSpecService.java @@ -18,7 +18,6 @@ import static feast.serving.util.RefUtil.generateFeatureSetStringRef; import static feast.serving.util.RefUtil.generateFeatureStringRef; -import static java.util.Comparator.comparingInt; import static java.util.stream.Collectors.groupingBy; import com.google.common.cache.CacheBuilder; @@ -115,11 +114,8 @@ public List getFeatureSets(List featureRefe if (featureSet == null) { throw new SpecRetrievalException( String.format( - "Unable to find feature set for feature ref: " - + "(project: %s, name: %s, version: %d)", - featureReference.getProject(), - featureReference.getName(), - featureReference.getVersion())); + "Unable to find feature set for feature ref: " + "(project: %s, name: %s)", + featureReference.getProject(), featureReference.getName())); } return Pair.of(featureSet, featureReference); }) @@ -150,7 +146,11 @@ public List getFeatureSets(List featureRefe */ public void populateCache() { Map featureSetMap = getFeatureSetMap(); + + featureSetCache.invalidateAll(); featureSetCache.putAll(featureSetMap); + + featureToFeatureSetMapping.clear(); featureToFeatureSetMapping.putAll(getFeatureToFeatureSetMapping(featureSetMap)); featureSetsCount.set(featureSetCache.size()); @@ -176,8 +176,7 @@ private Map getFeatureSetMap() { .setFilter( ListFeatureSetsRequest.Filter.newBuilder() .setProject(subscription.getProject()) - .setFeatureSetName(subscription.getName()) - .setFeatureSetVersion(subscription.getVersion())) + .setFeatureSetName(subscription.getName())) .build()); for (FeatureSet featureSet : featureSetsResponse.getFeatureSetsList()) { @@ -196,39 +195,17 @@ private Map getFeatureToFeatureSetMapping( Map featureSets) { HashMap mapping = new HashMap<>(); - featureSets.values().stream() - .collect(groupingBy(featureSet -> Pair.of(featureSet.getProject(), featureSet.getName()))) - .forEach( - (group, groupedFeatureSets) -> { - groupedFeatureSets = - groupedFeatureSets.stream() - .sorted(comparingInt(FeatureSetSpec::getVersion)) - .collect(Collectors.toList()); - for (int i = 0; i < groupedFeatureSets.size(); i++) { - FeatureSetSpec featureSetSpec = groupedFeatureSets.get(i); - for (FeatureSpec featureSpec : featureSetSpec.getFeaturesList()) { - FeatureReference featureRef = - FeatureReference.newBuilder() - .setProject(featureSetSpec.getProject()) - .setName(featureSpec.getName()) - .setVersion(featureSetSpec.getVersion()) - .build(); - mapping.put( - generateFeatureStringRef(featureRef), - generateFeatureSetStringRef(featureSetSpec)); - if (i == groupedFeatureSets.size() - 1) { - featureRef = - FeatureReference.newBuilder() - .setProject(featureSetSpec.getProject()) - .setName(featureSpec.getName()) - .build(); - mapping.put( - generateFeatureStringRef(featureRef), - generateFeatureSetStringRef(featureSetSpec)); - } - } - } - }); + for (FeatureSetSpec featureSetSpec : featureSets.values()) { + for (FeatureSpec featureSpec : featureSetSpec.getFeaturesList()) { + FeatureReference featureRef = + FeatureReference.newBuilder() + .setProject(featureSetSpec.getProject()) + .setName(featureSpec.getName()) + .build(); + mapping.put( + generateFeatureStringRef(featureRef), generateFeatureSetStringRef(featureSetSpec)); + } + } return mapping; } } diff --git a/serving/src/main/java/feast/serving/util/RefUtil.java b/serving/src/main/java/feast/serving/util/RefUtil.java index c3bcb0827a2..7557bd12bf2 100644 --- a/serving/src/main/java/feast/serving/util/RefUtil.java +++ b/serving/src/main/java/feast/serving/util/RefUtil.java @@ -22,25 +22,11 @@ public class RefUtil { public static String generateFeatureStringRef(FeatureReference featureReference) { String ref = String.format("%s/%s", featureReference.getProject(), featureReference.getName()); - if (featureReference.getVersion() > 0) { - return ref + String.format(":%d", featureReference.getVersion()); - } - return ref; - } - - public static String generateFeatureStringRefWithoutProject(FeatureReference featureReference) { - String ref = String.format("%s", featureReference.getName()); - if (featureReference.getVersion() > 0) { - return ref + String.format(":%d", featureReference.getVersion()); - } return ref; } public static String generateFeatureSetStringRef(FeatureSetSpec featureSetSpec) { String ref = String.format("%s/%s", featureSetSpec.getProject(), featureSetSpec.getName()); - if (featureSetSpec.getVersion() > 0) { - return ref + String.format(":%d", featureSetSpec.getVersion()); - } return ref; } } diff --git a/serving/src/main/resources/application.yml b/serving/src/main/resources/application.yml index f6eaccf3cd4..9158ee7fa58 100644 --- a/serving/src/main/resources/application.yml +++ b/serving/src/main/resources/application.yml @@ -22,7 +22,6 @@ feast: # Wildcards match all options. No filtering is done. - name: "*" project: "*" - version: "*" - name: historical type: BIGQUERY @@ -44,7 +43,6 @@ feast: subscriptions: - name: "*" project: "*" - version: "*" tracing: # If true, Feast will provide tracing data (using OpenTracing API) for various RPC method calls diff --git a/serving/src/main/resources/templates/join_featuresets.sql b/serving/src/main/resources/templates/join_featuresets.sql deleted file mode 100644 index 60b7c7d7a12..00000000000 --- a/serving/src/main/resources/templates/join_featuresets.sql +++ /dev/null @@ -1,24 +0,0 @@ -/* - Joins the outputs of multiple point-in-time-correctness joins to a single table. - */ -WITH joined as ( -SELECT * FROM `{{ leftTableName }}` -{% for featureSet in featureSets %} -LEFT JOIN ( - SELECT - uuid, - {% for featureName in featureSet.features %} - {{ featureSet.project }}_{{ featureName }}_v{{ featureSet.version }}{% if loop.last %}{% else %}, {% endif %} - {% endfor %} - FROM `{{ featureSet.table }}` -) USING (uuid) -{% endfor %} -) SELECT - event_timestamp, - {{ entities | join(', ') }} - {% for featureSet in featureSets %} - {% for featureName in featureSet.features %} - ,{{ featureSet.project }}_{{ featureName }}_v{{ featureSet.version }} as {{ featureName }} - {% endfor %} - {% endfor %} -FROM joined \ No newline at end of file diff --git a/serving/src/main/resources/templates/single_featureset_pit_join.sql b/serving/src/main/resources/templates/single_featureset_pit_join.sql deleted file mode 100644 index f3f20828ff1..00000000000 --- a/serving/src/main/resources/templates/single_featureset_pit_join.sql +++ /dev/null @@ -1,90 +0,0 @@ -/* - This query template performs the point-in-time correctness join for a single feature set table - to the provided entity table. - - 1. Concatenate the timestamp and entities from the feature set table with the entity dataset. - Feature values are joined to this table later for improved efficiency. - featureset_timestamp is equal to null in rows from the entity dataset. - */ -WITH union_features AS ( -SELECT - -- uuid is a unique identifier for each row in the entity dataset. Generated by `QueryTemplater.createEntityTableUUIDQuery` - uuid, - -- event_timestamp contains the timestamps to join onto - event_timestamp, - -- the feature_timestamp, i.e. the latest occurrence of the requested feature relative to the entity_dataset timestamp - NULL as {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, - -- created timestamp of the feature at the corresponding feature_timestamp - NULL as created_timestamp, - -- select only entities belonging to this feature set - {{ featureSet.entities | join(', ')}}, - -- boolean for filtering the dataset later - true AS is_entity_table -FROM `{{leftTableName}}` -UNION ALL -SELECT - NULL as uuid, - event_timestamp, - event_timestamp as {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, - created_timestamp, - {{ featureSet.entities | join(', ')}}, - false AS is_entity_table -FROM `{{projectId}}.{{datasetId}}.{{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}` WHERE event_timestamp <= '{{maxTimestamp}}' -{% if featureSet.maxAge == 0 %}{% else %}AND event_timestamp >= Timestamp_sub(TIMESTAMP '{{ minTimestamp }}', interval {{ featureSet.maxAge }} second){% endif %} -), -/* - 2. Window the data in the unioned dataset, partitioning by entity and ordering by event_timestamp, as - well as is_entity_table. - Within each window, back-fill the feature_timestamp - as a result of this, the null feature_timestamps - in the rows from the entity table should now contain the latest timestamps relative to the row's - event_timestamp. - - For rows where event_timestamp(provided datetime) - feature_timestamp > max age, set the - feature_timestamp to null. - */ -joined AS ( -SELECT - uuid, - event_timestamp, - {{ featureSet.entities | join(', ')}}, - {% for featureName in featureSet.features %} - IF(event_timestamp >= {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp {% if featureSet.maxAge == 0 %}{% else %}AND Timestamp_sub(event_timestamp, interval {{ featureSet.maxAge }} second) < {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp{% endif %}, {{ featureSet.project }}_{{ featureName }}_v{{ featureSet.version }}, NULL) as {{ featureSet.project }}_{{ featureName }}_v{{ featureSet.version }}{% if loop.last %}{% else %}, {% endif %} - {% endfor %} -FROM ( -SELECT - uuid, - event_timestamp, - {{ featureSet.entities | join(', ')}}, - FIRST_VALUE(created_timestamp IGNORE NULLS) over w AS created_timestamp, - FIRST_VALUE({{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp IGNORE NULLS) over w AS {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, - is_entity_table -FROM union_features -WINDOW w AS (PARTITION BY {{ featureSet.entities | join(', ') }} ORDER BY event_timestamp DESC, is_entity_table DESC, created_timestamp DESC ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) -) -/* - 3. Select only the rows from the entity table, and join the features from the original feature set table - to the dataset using the entity values, feature_timestamp, and created_timestamps. - */ -LEFT JOIN ( -SELECT - event_timestamp as {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, - created_timestamp, - {{ featureSet.entities | join(', ')}}, - {% for featureName in featureSet.features %} - {{ featureName }} as {{ featureSet.project }}_{{ featureName }}_v{{ featureSet.version }}{% if loop.last %}{% else %}, {% endif %} - {% endfor %} -FROM `{{projectId}}.{{datasetId}}.{{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}` WHERE event_timestamp <= '{{maxTimestamp}}' -{% if featureSet.maxAge == 0 %}{% else %}AND event_timestamp >= Timestamp_sub(TIMESTAMP '{{ minTimestamp }}', interval {{ featureSet.maxAge }} second){% endif %} -) USING ({{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, created_timestamp, {{ featureSet.entities | join(', ')}}) -WHERE is_entity_table -) -/* - 4. Finally, deduplicate the rows by selecting the first occurrence of each entity table row UUID. - */ -SELECT - k.* -FROM ( - SELECT ARRAY_AGG(row LIMIT 1)[OFFSET(0)] k - FROM joined row - GROUP BY uuid -) \ No newline at end of file diff --git a/serving/src/test/java/feast/serving/controller/ServingServiceGRpcControllerTest.java b/serving/src/test/java/feast/serving/controller/ServingServiceGRpcControllerTest.java index d23f9da1d25..d292c33fba9 100644 --- a/serving/src/test/java/feast/serving/controller/ServingServiceGRpcControllerTest.java +++ b/serving/src/test/java/feast/serving/controller/ServingServiceGRpcControllerTest.java @@ -52,17 +52,9 @@ public void setUp() { validRequest = GetOnlineFeaturesRequest.newBuilder() .addFeatures( - FeatureReference.newBuilder() - .setName("feature1") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature1").setProject("project").build()) .addFeatures( - FeatureReference.newBuilder() - .setName("feature2") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature2").setProject("project").build()) .addEntityRows( EntityRow.newBuilder() .setEntityTimestamp(Timestamp.newBuilder().setSeconds(100)) diff --git a/serving/src/test/java/feast/serving/service/CachedSpecServiceTest.java b/serving/src/test/java/feast/serving/service/CachedSpecServiceTest.java index 144b967c9f5..0987b95e956 100644 --- a/serving/src/test/java/feast/serving/service/CachedSpecServiceTest.java +++ b/serving/src/test/java/feast/serving/service/CachedSpecServiceTest.java @@ -69,62 +69,50 @@ public void setUp() { .setType(StoreType.REDIS) .setRedisConfig(RedisConfig.newBuilder().setHost("localhost").setPort(6379)) .addSubscriptions( - Subscription.newBuilder() - .setProject("project") - .setName("fs1") - .setVersion("*") - .build()) + Subscription.newBuilder().setProject("project").setName("fs1").build()) .addSubscriptions( - Subscription.newBuilder() - .setProject("project") - .setName("fs2") - .setVersion("*") - .build()) + Subscription.newBuilder().setProject("project").setName("fs2").build()) .build(); when(coreService.registerStore(store)).thenReturn(store); featureSetSpecs = new LinkedHashMap<>(); featureSetSpecs.put( - "fs1:1", + "fs1", FeatureSetSpec.newBuilder() .setProject("project") .setName("fs1") - .setVersion(1) .addFeatures(FeatureSpec.newBuilder().setName("feature")) .build()); featureSetSpecs.put( - "fs1:2", + "fs1", FeatureSetSpec.newBuilder() .setProject("project") .setName("fs1") - .setVersion(2) .addFeatures(FeatureSpec.newBuilder().setName("feature")) .addFeatures(FeatureSpec.newBuilder().setName("feature2")) .build()); featureSetSpecs.put( - "fs2:1", + "fs2", FeatureSetSpec.newBuilder() .setProject("project") .setName("fs2") - .setVersion(1) .addFeatures(FeatureSpec.newBuilder().setName("feature3")) .build()); List fs1FeatureSets = Lists.newArrayList( - FeatureSetProto.FeatureSet.newBuilder().setSpec(featureSetSpecs.get("fs1:1")).build(), - FeatureSetProto.FeatureSet.newBuilder().setSpec(featureSetSpecs.get("fs1:2")).build()); + FeatureSetProto.FeatureSet.newBuilder().setSpec(featureSetSpecs.get("fs1")).build(), + FeatureSetProto.FeatureSet.newBuilder().setSpec(featureSetSpecs.get("fs1")).build()); List fs2FeatureSets = Lists.newArrayList( - FeatureSetProto.FeatureSet.newBuilder().setSpec(featureSetSpecs.get("fs2:1")).build()); + FeatureSetProto.FeatureSet.newBuilder().setSpec(featureSetSpecs.get("fs2")).build()); when(coreService.listFeatureSets( ListFeatureSetsRequest.newBuilder() .setFilter( ListFeatureSetsRequest.Filter.newBuilder() .setProject("project") .setFeatureSetName("fs1") - .setFeatureSetVersion("*") .build()) .build())) .thenReturn(ListFeatureSetsResponse.newBuilder().addAllFeatureSets(fs1FeatureSets).build()); @@ -134,7 +122,6 @@ public void setUp() { ListFeatureSetsRequest.Filter.newBuilder() .setProject("project") .setFeatureSetName("fs2") - .setFeatureSetVersion("*") .build()) .build())) .thenReturn(ListFeatureSetsResponse.newBuilder().addAllFeatureSets(fs2FeatureSets).build()); @@ -158,17 +145,9 @@ public void shouldPopulateAndReturnStore() { public void shouldPopulateAndReturnFeatureSets() { cachedSpecService.populateCache(); FeatureReference frv1 = - FeatureReference.newBuilder() - .setProject("project") - .setName("feature") - .setVersion(1) - .build(); + FeatureReference.newBuilder().setProject("project").setName("feature").build(); FeatureReference frv2 = - FeatureReference.newBuilder() - .setProject("project") - .setName("feature") - .setVersion(2) - .build(); + FeatureReference.newBuilder().setProject("project").setName("feature").build(); assertThat( cachedSpecService.getFeatureSets(Collections.singletonList(frv1)), @@ -176,7 +155,7 @@ public void shouldPopulateAndReturnFeatureSets() { Lists.newArrayList( FeatureSetRequest.newBuilder() .addFeatureReference(frv1) - .setSpec(featureSetSpecs.get("fs1:1")) + .setSpec(featureSetSpecs.get("fs1")) .build()))); assertThat( cachedSpecService.getFeatureSets(Collections.singletonList(frv2)), @@ -184,7 +163,7 @@ public void shouldPopulateAndReturnFeatureSets() { Lists.newArrayList( FeatureSetRequest.newBuilder() .addFeatureReference(frv2) - .setSpec(featureSetSpecs.get("fs1:2")) + .setSpec(featureSetSpecs.get("fs1")) .build()))); } @@ -200,7 +179,7 @@ public void shouldPopulateAndReturnLatestFeatureSetIfVersionsNotSupplied() { Lists.newArrayList( FeatureSetRequest.newBuilder() .addFeatureReference(frv1) - .setSpec(featureSetSpecs.get("fs1:2")) + .setSpec(featureSetSpecs.get("fs1")) .build()))); } @@ -208,17 +187,9 @@ public void shouldPopulateAndReturnLatestFeatureSetIfVersionsNotSupplied() { public void shouldPopulateAndReturnFeatureSetsGivenFeaturesFromDifferentFeatureSets() { cachedSpecService.populateCache(); FeatureReference frv1 = - FeatureReference.newBuilder() - .setProject("project") - .setName("feature") - .setVersion(1) - .build(); + FeatureReference.newBuilder().setProject("project").setName("feature").build(); FeatureReference fr3 = - FeatureReference.newBuilder() - .setProject("project") - .setName("feature3") - .setVersion(1) - .build(); + FeatureReference.newBuilder().setProject("project").setName("feature3").build(); assertThat( cachedSpecService.getFeatureSets(Lists.newArrayList(frv1, fr3)), @@ -226,11 +197,11 @@ public void shouldPopulateAndReturnFeatureSetsGivenFeaturesFromDifferentFeatureS Lists.newArrayList( FeatureSetRequest.newBuilder() .addFeatureReference(frv1) - .setSpec(featureSetSpecs.get("fs1:1")) + .setSpec(featureSetSpecs.get("fs1")) .build(), FeatureSetRequest.newBuilder() .addFeatureReference(fr3) - .setSpec(featureSetSpecs.get("fs2:1")) + .setSpec(featureSetSpecs.get("fs2")) .build()) .toArray())); } @@ -239,17 +210,9 @@ public void shouldPopulateAndReturnFeatureSetsGivenFeaturesFromDifferentFeatureS public void shouldPopulateAndReturnFeatureSetGivenFeaturesFromSameFeatureSet() { cachedSpecService.populateCache(); FeatureReference fr1 = - FeatureReference.newBuilder() - .setProject("project") - .setName("feature") - .setVersion(2) - .build(); + FeatureReference.newBuilder().setProject("project").setName("feature").build(); FeatureReference fr2 = - FeatureReference.newBuilder() - .setProject("project") - .setName("feature2") - .setVersion(2) - .build(); + FeatureReference.newBuilder().setProject("project").setName("feature2").build(); assertThat( cachedSpecService.getFeatureSets(Lists.newArrayList(fr1, fr2)), @@ -258,7 +221,7 @@ public void shouldPopulateAndReturnFeatureSetGivenFeaturesFromSameFeatureSet() { FeatureSetRequest.newBuilder() .addFeatureReference(fr1) .addFeatureReference(fr2) - .setSpec(featureSetSpecs.get("fs1:2")) + .setSpec(featureSetSpecs.get("fs1")) .build()))); } } diff --git a/serving/src/test/java/feast/serving/service/OnlineServingServiceTest.java b/serving/src/test/java/feast/serving/service/OnlineServingServiceTest.java index b78fcb69170..5b9421a3c13 100644 --- a/serving/src/test/java/feast/serving/service/OnlineServingServiceTest.java +++ b/serving/src/test/java/feast/serving/service/OnlineServingServiceTest.java @@ -70,17 +70,9 @@ public void shouldReturnResponseWithValuesIfKeysPresent() { GetOnlineFeaturesRequest request = GetOnlineFeaturesRequest.newBuilder() .addFeatures( - FeatureReference.newBuilder() - .setName("feature1") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature1").setProject("project").build()) .addFeatures( - FeatureReference.newBuilder() - .setName("feature2") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature2").setProject("project").build()) .addEntityRows( EntityRow.newBuilder() .setEntityTimestamp(Timestamp.newBuilder().setSeconds(100)) @@ -103,7 +95,7 @@ public void shouldReturnResponseWithValuesIfKeysPresent() { Field.newBuilder().setName("entity2").setValue(strValue("a")).build(), Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), Field.newBuilder().setName("feature2").setValue(intValue(1)).build())) - .setFeatureSet("featureSet:1") + .setFeatureSet("featureSet") .build(), FeatureRow.newBuilder() .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) @@ -113,7 +105,7 @@ public void shouldReturnResponseWithValuesIfKeysPresent() { Field.newBuilder().setName("entity2").setValue(strValue("b")).build(), Field.newBuilder().setName("feature1").setValue(intValue(2)).build(), Field.newBuilder().setName("feature2").setValue(intValue(2)).build())) - .setFeatureSet("featureSet:1") + .setFeatureSet("featureSet") .build()); FeatureSetRequest featureSetRequest = @@ -135,14 +127,14 @@ public void shouldReturnResponseWithValuesIfKeysPresent() { FieldValues.newBuilder() .putFields("entity1", intValue(1)) .putFields("entity2", strValue("a")) - .putFields("project/feature1:1", intValue(1)) - .putFields("project/feature2:1", intValue(1))) + .putFields("project/feature1", intValue(1)) + .putFields("project/feature2", intValue(1))) .addFieldValues( FieldValues.newBuilder() .putFields("entity1", intValue(2)) .putFields("entity2", strValue("b")) - .putFields("project/feature1:1", intValue(2)) - .putFields("project/feature2:1", intValue(2))) + .putFields("project/feature1", intValue(2)) + .putFields("project/feature2", intValue(2))) .build(); GetOnlineFeaturesResponse actual = onlineServingService.getOnlineFeatures(request); assertThat( @@ -154,11 +146,7 @@ public void shouldReturnKeysWithoutVersionIfNotProvided() { GetOnlineFeaturesRequest request = GetOnlineFeaturesRequest.newBuilder() .addFeatures( - FeatureReference.newBuilder() - .setName("feature1") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature1").setProject("project").build()) .addFeatures( FeatureReference.newBuilder().setName("feature2").setProject("project").build()) .addEntityRows( @@ -183,7 +171,7 @@ public void shouldReturnKeysWithoutVersionIfNotProvided() { Field.newBuilder().setName("entity2").setValue(strValue("a")).build(), Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), Field.newBuilder().setName("feature2").setValue(intValue(1)).build())) - .setFeatureSet("featureSet:1") + .setFeatureSet("featureSet") .build(), FeatureRow.newBuilder() .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) @@ -193,7 +181,7 @@ public void shouldReturnKeysWithoutVersionIfNotProvided() { Field.newBuilder().setName("entity2").setValue(strValue("b")).build(), Field.newBuilder().setName("feature1").setValue(intValue(2)).build(), Field.newBuilder().setName("feature2").setValue(intValue(2)).build())) - .setFeatureSet("featureSet:1") + .setFeatureSet("featureSet") .build()); FeatureSetRequest featureSetRequest = @@ -215,13 +203,13 @@ public void shouldReturnKeysWithoutVersionIfNotProvided() { FieldValues.newBuilder() .putFields("entity1", intValue(1)) .putFields("entity2", strValue("a")) - .putFields("project/feature1:1", intValue(1)) + .putFields("project/feature1", intValue(1)) .putFields("project/feature2", intValue(1))) .addFieldValues( FieldValues.newBuilder() .putFields("entity1", intValue(2)) .putFields("entity2", strValue("b")) - .putFields("project/feature1:1", intValue(2)) + .putFields("project/feature1", intValue(2)) .putFields("project/feature2", intValue(2))) .build(); GetOnlineFeaturesResponse actual = onlineServingService.getOnlineFeatures(request); @@ -235,17 +223,9 @@ public void shouldReturnResponseWithUnsetValuesIfKeysNotPresent() { GetOnlineFeaturesRequest request = GetOnlineFeaturesRequest.newBuilder() .addFeatures( - FeatureReference.newBuilder() - .setName("feature1") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature1").setProject("project").build()) .addFeatures( - FeatureReference.newBuilder() - .setName("feature2") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature2").setProject("project").build()) .addEntityRows( EntityRow.newBuilder() .setEntityTimestamp(Timestamp.newBuilder().setSeconds(100)) @@ -268,14 +248,14 @@ public void shouldReturnResponseWithUnsetValuesIfKeysNotPresent() { Lists.newArrayList( FeatureRow.newBuilder() .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) - .setFeatureSet("project/featureSet:1") + .setFeatureSet("project/featureSet") .addAllFields( Lists.newArrayList( Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), Field.newBuilder().setName("feature2").setValue(intValue(1)).build())) .build(), FeatureRow.newBuilder() - .setFeatureSet("project/featureSet:1") + .setFeatureSet("project/featureSet") .addAllFields( Lists.newArrayList( Field.newBuilder().setName("feature1").build(), @@ -295,14 +275,14 @@ public void shouldReturnResponseWithUnsetValuesIfKeysNotPresent() { FieldValues.newBuilder() .putFields("entity1", intValue(1)) .putFields("entity2", strValue("a")) - .putFields("project/feature1:1", intValue(1)) - .putFields("project/feature2:1", intValue(1))) + .putFields("project/feature1", intValue(1)) + .putFields("project/feature2", intValue(1))) .addFieldValues( FieldValues.newBuilder() .putFields("entity1", intValue(2)) .putFields("entity2", strValue("b")) - .putFields("project/feature1:1", Value.newBuilder().build()) - .putFields("project/feature2:1", Value.newBuilder().build())) + .putFields("project/feature1", Value.newBuilder().build()) + .putFields("project/feature2", Value.newBuilder().build())) .build(); GetOnlineFeaturesResponse actual = onlineServingService.getOnlineFeatures(request); assertThat( @@ -315,17 +295,9 @@ public void shouldReturnResponseWithUnsetValuesIfMaxAgeIsExceeded() { GetOnlineFeaturesRequest request = GetOnlineFeaturesRequest.newBuilder() .addFeatures( - FeatureReference.newBuilder() - .setName("feature1") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature1").setProject("project").build()) .addFeatures( - FeatureReference.newBuilder() - .setName("feature2") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature2").setProject("project").build()) .addEntityRows( EntityRow.newBuilder() .setEntityTimestamp(Timestamp.newBuilder().setSeconds(100)) @@ -348,7 +320,7 @@ public void shouldReturnResponseWithUnsetValuesIfMaxAgeIsExceeded() { Field.newBuilder().setName("entity2").setValue(strValue("a")).build(), Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), Field.newBuilder().setName("feature2").setValue(intValue(1)).build())) - .setFeatureSet("featureSet:1") + .setFeatureSet("featureSet") .build(), FeatureRow.newBuilder() .setEventTimestamp( @@ -359,7 +331,7 @@ public void shouldReturnResponseWithUnsetValuesIfMaxAgeIsExceeded() { Field.newBuilder().setName("entity2").setValue(strValue("b")).build(), Field.newBuilder().setName("feature1").setValue(intValue(2)).build(), Field.newBuilder().setName("feature2").setValue(intValue(2)).build())) - .setFeatureSet("featureSet:1") + .setFeatureSet("featureSet") .build()); FeatureSetSpec spec = @@ -383,14 +355,14 @@ public void shouldReturnResponseWithUnsetValuesIfMaxAgeIsExceeded() { FieldValues.newBuilder() .putFields("entity1", intValue(1)) .putFields("entity2", strValue("a")) - .putFields("project/feature1:1", intValue(1)) - .putFields("project/feature2:1", intValue(1))) + .putFields("project/feature1", intValue(1)) + .putFields("project/feature2", intValue(1))) .addFieldValues( FieldValues.newBuilder() .putFields("entity1", intValue(2)) .putFields("entity2", strValue("b")) - .putFields("project/feature1:1", Value.newBuilder().build()) - .putFields("project/feature2:1", Value.newBuilder().build())) + .putFields("project/feature1", Value.newBuilder().build()) + .putFields("project/feature2", Value.newBuilder().build())) .build(); GetOnlineFeaturesResponse actual = onlineServingService.getOnlineFeatures(request); assertThat( @@ -403,11 +375,7 @@ public void shouldFilterOutUndesiredRows() { GetOnlineFeaturesRequest request = GetOnlineFeaturesRequest.newBuilder() .addFeatures( - FeatureReference.newBuilder() - .setName("feature1") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature1").setProject("project").build()) .addEntityRows( EntityRow.newBuilder() .setEntityTimestamp(Timestamp.newBuilder().setSeconds(100)) @@ -430,7 +398,7 @@ public void shouldFilterOutUndesiredRows() { Field.newBuilder().setName("entity2").setValue(strValue("a")).build(), Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), Field.newBuilder().setName("feature2").setValue(intValue(1)).build())) - .setFeatureSet("featureSet:1") + .setFeatureSet("featureSet") .build(), FeatureRow.newBuilder() .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) @@ -440,7 +408,7 @@ public void shouldFilterOutUndesiredRows() { Field.newBuilder().setName("entity2").setValue(strValue("b")).build(), Field.newBuilder().setName("feature1").setValue(intValue(2)).build(), Field.newBuilder().setName("feature2").setValue(intValue(2)).build())) - .setFeatureSet("featureSet:1") + .setFeatureSet("featureSet") .build()); FeatureSetRequest featureSetRequest = @@ -462,12 +430,12 @@ public void shouldFilterOutUndesiredRows() { FieldValues.newBuilder() .putFields("entity1", intValue(1)) .putFields("entity2", strValue("a")) - .putFields("project/feature1:1", intValue(1))) + .putFields("project/feature1", intValue(1))) .addFieldValues( FieldValues.newBuilder() .putFields("entity1", intValue(2)) .putFields("entity2", strValue("b")) - .putFields("project/feature1:1", intValue(2))) + .putFields("project/feature1", intValue(2))) .build(); GetOnlineFeaturesResponse actual = onlineServingService.getOnlineFeatures(request); assertThat( @@ -492,21 +460,9 @@ private FeatureSetSpec getFeatureSetSpec() { return FeatureSetSpec.newBuilder() .setProject("project") .setName("featureSet") - .setVersion(1) .addEntities(EntitySpec.newBuilder().setName("entity1")) .addEntities(EntitySpec.newBuilder().setName("entity2")) .setMaxAge(Duration.newBuilder().setSeconds(30)) // default .build(); } - - private FeatureSetSpec getFeatureSetSpecWithNoMaxAge() { - return FeatureSetSpec.newBuilder() - .setProject("project") - .setName("featureSet") - .setVersion(1) - .addEntities(EntitySpec.newBuilder().setName("entity1")) - .addEntities(EntitySpec.newBuilder().setName("entity2")) - .setMaxAge(Duration.newBuilder().setSeconds(0).setNanos(0).build()) - .build(); - } } diff --git a/storage/api/src/main/java/feast/storage/api/writer/FailedElement.java b/storage/api/src/main/java/feast/storage/api/writer/FailedElement.java index d5823414772..c6db877216f 100644 --- a/storage/api/src/main/java/feast/storage/api/writer/FailedElement.java +++ b/storage/api/src/main/java/feast/storage/api/writer/FailedElement.java @@ -39,9 +39,6 @@ public abstract class FailedElement { @Nullable public abstract String getFeatureSetName(); - @Nullable - public abstract String getFeatureSetVersion(); - @Nullable public abstract String getTransformName(); @@ -66,8 +63,6 @@ public abstract static class Builder { public abstract Builder setFeatureSetName(String featureSetName); - public abstract Builder setFeatureSetVersion(String featureSetVersion); - public abstract Builder setJobName(String jobName); public abstract Builder setTransformName(String transformName); diff --git a/storage/api/src/main/java/feast/storage/common/testing/TestUtil.java b/storage/api/src/main/java/feast/storage/common/testing/TestUtil.java index 6047a93dc17..e1f5c116752 100644 --- a/storage/api/src/main/java/feast/storage/common/testing/TestUtil.java +++ b/storage/api/src/main/java/feast/storage/common/testing/TestUtil.java @@ -87,7 +87,7 @@ public static FeatureRow createRandomFeatureRow(FeatureSet featureSet, int rando private static String getFeatureSetReference(FeatureSet featureSet) { FeatureSetSpec spec = featureSet.getSpec(); - return String.format("%s/%s:%d", spec.getProject(), spec.getName(), spec.getVersion()); + return String.format("%s/%s:%d", spec.getProject(), spec.getName()); } /** diff --git a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/FeatureSetQueryInfo.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/FeatureSetQueryInfo.java index 5a7d56e9844..a99411f87b9 100644 --- a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/FeatureSetQueryInfo.java +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/FeatureSetQueryInfo.java @@ -22,7 +22,6 @@ public class FeatureSetQueryInfo { private final String project; private final String name; - private final int version; private final long maxAge; private final List entities; private final List features; @@ -31,14 +30,12 @@ public class FeatureSetQueryInfo { public FeatureSetQueryInfo( String project, String name, - int version, long maxAge, List entities, List features, String table) { this.project = project; this.name = name; - this.version = version; this.maxAge = maxAge; this.entities = entities; this.features = features; @@ -49,7 +46,6 @@ public FeatureSetQueryInfo(FeatureSetQueryInfo featureSetInfo, String table) { this.project = featureSetInfo.getProject(); this.name = featureSetInfo.getName(); - this.version = featureSetInfo.getVersion(); this.maxAge = featureSetInfo.getMaxAge(); this.entities = featureSetInfo.getEntities(); this.features = featureSetInfo.getFeatures(); @@ -64,10 +60,6 @@ public String getName() { return name; } - public int getVersion() { - return version; - } - public long getMaxAge() { return maxAge; } diff --git a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/QueryTemplater.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/QueryTemplater.java index cba997b6ab0..1fb0a617afb 100644 --- a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/QueryTemplater.java +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/retriever/QueryTemplater.java @@ -85,13 +85,7 @@ public static List getFeatureSetInfos( .collect(Collectors.toList()); featureSetInfos.add( new FeatureSetQueryInfo( - spec.getProject(), - spec.getName(), - spec.getVersion(), - maxAge.getSeconds(), - fsEntities, - features, - "")); + spec.getProject(), spec.getName(), maxAge.getSeconds(), fsEntities, features, "")); } return featureSetInfos; } diff --git a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryFeatureSink.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryFeatureSink.java index 5d8f3d25cb7..20281783cf2 100644 --- a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryFeatureSink.java +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/BigQueryFeatureSink.java @@ -100,22 +100,19 @@ public void prepareWrite(FeatureSetProto.FeatureSet featureSet) { bigquery.create(DatasetInfo.of(datasetId)); } String tableName = - String.format( - "%s_%s_v%d", - featureSetSpec.getProject(), featureSetSpec.getName(), featureSetSpec.getVersion()) + String.format("%s_%s", featureSetSpec.getProject(), featureSetSpec.getName()) .replaceAll("-", "_"); TableId tableId = TableId.of(datasetId.getProject(), datasetId.getDataset(), tableName); - // Return if there is an existing table Table table = bigquery.getTable(tableId); + TableDefinition tableDefinition = createBigQueryTableDefinition(table, featureSet.getSpec()); + TableInfo tableInfo = TableInfo.of(tableId, tableDefinition); if (table != null) { log.info( "Updating and writing to existing BigQuery table '{}:{}.{}'", datasetId.getProject(), datasetId.getDataset(), tableName); - TableDefinition tableDefinition = createBigQueryTableDefinition(featureSet.getSpec()); - TableInfo tableInfo = TableInfo.of(tableId, tableDefinition); bigquery.update(tableInfo); return; } @@ -125,8 +122,6 @@ public void prepareWrite(FeatureSetProto.FeatureSet featureSet) { tableId.getTable(), datasetId.getDataset(), datasetId.getProject()); - TableDefinition tableDefinition = createBigQueryTableDefinition(featureSet.getSpec()); - TableInfo tableInfo = TableInfo.of(tableId, tableDefinition); bigquery.create(tableInfo); } @@ -134,8 +129,18 @@ public void prepareWrite(FeatureSetProto.FeatureSet featureSet) { public PTransform, WriteResult> writer() { return new BigQueryWrite(DatasetId.of(getProjectId(), getDatasetId())); } - - private TableDefinition createBigQueryTableDefinition(FeatureSetProto.FeatureSetSpec spec) { + /** + * Creates a BigQuery {@link TableDefinition} based on the provided FeatureSetSpec and the + * existing table, if any. If a table already exists, existing fields will be retained, and new + * fields present in the feature set will be appended to the existing FieldsList. + * + * @param existingTable existing {@link Table} retrieved using bigquery.GetTable(). If the table + * does not exist, will be null. + * @param spec FeatureSet spec that this table is for + * @return {@link TableDefinition} containing all tombstoned and active fields. + */ + private TableDefinition createBigQueryTableDefinition( + Table existingTable, FeatureSetProto.FeatureSetSpec spec) { List fields = new ArrayList<>(); log.info("Table will have the following fields:"); @@ -189,9 +194,21 @@ private TableDefinition createBigQueryTableDefinition(FeatureSetProto.FeatureSet TimePartitioning.newBuilder(TimePartitioning.Type.DAY).setField("event_timestamp").build(); log.info("Table partitioning: " + timePartitioning.toString()); + List fieldsList = new ArrayList<>(); + if (existingTable != null) { + Schema existingSchema = existingTable.getDefinition().getSchema(); + fieldsList.addAll(existingSchema.getFields()); + } + + for (Field field : fields) { + if (!fieldsList.contains(field)) { + fieldsList.add(field); + } + } + return StandardTableDefinition.newBuilder() .setTimePartitioning(timePartitioning) - .setSchema(Schema.of(fields)) + .setSchema(Schema.of(FieldList.of(fieldsList))) .build(); } } diff --git a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/GetTableDestination.java b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/GetTableDestination.java index 5903d36b858..2007f8573fb 100644 --- a/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/GetTableDestination.java +++ b/storage/connectors/bigquery/src/main/java/feast/storage/connectors/bigquery/writer/GetTableDestination.java @@ -35,8 +35,7 @@ public GetTableDestination(String projectId, String datasetId) { @Override public TableDestination apply(ValueInSingleWindow input) { - String[] split = input.getValue().getFeatureSet().split(":"); - String[] splitName = split[0].split("/"); + String[] splitName = input.getValue().getFeatureSet().split("/"); TimePartitioning timePartitioning = new TimePartitioning() @@ -44,8 +43,7 @@ public TableDestination apply(ValueInSingleWindow input) { .setField(FeatureRowToTableRow.getEventTimestampColumn()); return new TableDestination( - String.format( - "%s:%s.%s_%s_v%s", projectId, datasetId, splitName[0], splitName[1], split[1]), + String.format("%s:%s.%s_%s", projectId, datasetId, splitName[0], splitName[1]), String.format("Feast table for %s", input.getValue().getFeatureSet()), timePartitioning); } diff --git a/storage/connectors/bigquery/src/main/resources/templates/join_featuresets.sql b/storage/connectors/bigquery/src/main/resources/templates/join_featuresets.sql index 60b7c7d7a12..10aafb05092 100644 --- a/storage/connectors/bigquery/src/main/resources/templates/join_featuresets.sql +++ b/storage/connectors/bigquery/src/main/resources/templates/join_featuresets.sql @@ -8,7 +8,7 @@ LEFT JOIN ( SELECT uuid, {% for featureName in featureSet.features %} - {{ featureSet.project }}_{{ featureName }}_v{{ featureSet.version }}{% if loop.last %}{% else %}, {% endif %} + {{ featureSet.project }}_{{ featureName }}{% if loop.last %}{% else %}, {% endif %} {% endfor %} FROM `{{ featureSet.table }}` ) USING (uuid) @@ -18,7 +18,7 @@ LEFT JOIN ( {{ entities | join(', ') }} {% for featureSet in featureSets %} {% for featureName in featureSet.features %} - ,{{ featureSet.project }}_{{ featureName }}_v{{ featureSet.version }} as {{ featureName }} + ,{{ featureSet.project }}_{{ featureName }} as {{ featureName }} {% endfor %} {% endfor %} FROM joined \ No newline at end of file diff --git a/storage/connectors/bigquery/src/main/resources/templates/single_featureset_pit_join.sql b/storage/connectors/bigquery/src/main/resources/templates/single_featureset_pit_join.sql index fb4c555b529..c02bf6b46c1 100644 --- a/storage/connectors/bigquery/src/main/resources/templates/single_featureset_pit_join.sql +++ b/storage/connectors/bigquery/src/main/resources/templates/single_featureset_pit_join.sql @@ -13,7 +13,7 @@ SELECT -- event_timestamp contains the timestamps to join onto event_timestamp, -- the feature_timestamp, i.e. the latest occurrence of the requested feature relative to the entity_dataset timestamp - NULL as {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, + NULL as {{ featureSet.project }}_{{ featureSet.name }}_feature_timestamp, -- created timestamp of the feature at the corresponding feature_timestamp NULL as created_timestamp, -- select only entities belonging to this feature set @@ -25,11 +25,11 @@ UNION ALL SELECT NULL as uuid, event_timestamp, - event_timestamp as {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, + event_timestamp as {{ featureSet.project }}_{{ featureSet.name }}_feature_timestamp, created_timestamp, {{ featureSet.entities | join(', ')}}, false AS is_entity_table -FROM `{{projectId}}.{{datasetId}}.{{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}` WHERE event_timestamp <= '{{maxTimestamp}}' +FROM `{{projectId}}.{{datasetId}}.{{ featureSet.project }}_{{ featureSet.name }}` WHERE event_timestamp <= '{{maxTimestamp}}' {% if featureSet.maxAge == 0 %}{% else %}AND event_timestamp >= Timestamp_sub(TIMESTAMP '{{ minTimestamp }}', interval {{ featureSet.maxAge }} second){% endif %} ), /* @@ -48,7 +48,7 @@ SELECT event_timestamp, {{ featureSet.entities | join(', ')}}, {% for featureName in featureSet.features %} - IF(event_timestamp >= {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp {% if featureSet.maxAge == 0 %}{% else %}AND Timestamp_sub(event_timestamp, interval {{ featureSet.maxAge }} second) < {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp{% endif %}, {{ featureSet.project }}_{{ featureName }}_v{{ featureSet.version }}, NULL) as {{ featureSet.project }}_{{ featureName }}_v{{ featureSet.version }}{% if loop.last %}{% else %}, {% endif %} + IF(event_timestamp >= {{ featureSet.project }}_{{ featureSet.name }}_feature_timestamp {% if featureSet.maxAge == 0 %}{% else %}AND Timestamp_sub(event_timestamp, interval {{ featureSet.maxAge }} second) < {{ featureSet.project }}_{{ featureSet.name }}_feature_timestamp{% endif %}, {{ featureSet.project }}_{{ featureName }}, NULL) as {{ featureSet.project }}_{{ featureName }}{% if loop.last %}{% else %}, {% endif %} {% endfor %} FROM ( SELECT @@ -56,7 +56,7 @@ SELECT event_timestamp, {{ featureSet.entities | join(', ')}}, FIRST_VALUE(created_timestamp IGNORE NULLS) over w AS created_timestamp, - FIRST_VALUE({{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp IGNORE NULLS) over w AS {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, + FIRST_VALUE({{ featureSet.project }}_{{ featureSet.name }}_feature_timestamp IGNORE NULLS) over w AS {{ featureSet.project }}_{{ featureSet.name }}_feature_timestamp, is_entity_table FROM union_features WINDOW w AS (PARTITION BY {{ featureSet.entities | join(', ') }} ORDER BY event_timestamp DESC, is_entity_table DESC, created_timestamp DESC ROWS BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING) @@ -67,15 +67,15 @@ WINDOW w AS (PARTITION BY {{ featureSet.entities | join(', ') }} ORDER BY event_ */ LEFT JOIN ( SELECT - event_timestamp as {{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, + event_timestamp as {{ featureSet.project }}_{{ featureSet.name }}_feature_timestamp, created_timestamp, {{ featureSet.entities | join(', ')}}, {% for featureName in featureSet.features %} - {{ featureName }} as {{ featureSet.project }}_{{ featureName }}_v{{ featureSet.version }}{% if loop.last %}{% else %}, {% endif %} + {{ featureName }} as {{ featureSet.project }}_{{ featureName }}{% if loop.last %}{% else %}, {% endif %} {% endfor %} -FROM `{{ projectId }}.{{ datasetId }}.{{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}` WHERE event_timestamp <= '{{maxTimestamp}}' +FROM `{{ projectId }}.{{ datasetId }}.{{ featureSet.project }}_{{ featureSet.name }}` WHERE event_timestamp <= '{{maxTimestamp}}' {% if featureSet.maxAge == 0 %}{% else %}AND event_timestamp >= Timestamp_sub(TIMESTAMP '{{ minTimestamp }}', interval {{ featureSet.maxAge }} second){% endif %} -) USING ({{ featureSet.project }}_{{ featureSet.name }}_v{{ featureSet.version }}_feature_timestamp, created_timestamp, {{ featureSet.entities | join(', ')}}) +) USING ({{ featureSet.project }}_{{ featureSet.name }}_feature_timestamp, created_timestamp, {{ featureSet.entities | join(', ')}}) WHERE is_entity_table ) /* diff --git a/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever/RedisOnlineRetriever.java b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever/RedisOnlineRetriever.java index 0963731988c..f25bf558f22 100644 --- a/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever/RedisOnlineRetriever.java +++ b/storage/connectors/redis/src/main/java/feast/storage/connectors/redis/retriever/RedisOnlineRetriever.java @@ -112,7 +112,7 @@ private List buildRedisKeys(List entityRows, FeatureSetSpec /** * Create {@link RedisKey} * - * @param featureSet featureSet reference of the feature. E.g. feature_set_1:1 + * @param featureSet featureSet reference of the feature. E.g. feature_set_1 * @param featureSetEntityNames entity names that belong to the featureSet * @param entityRow entityRow to build the key from * @return {@link RedisKey} @@ -213,9 +213,6 @@ private List sendMultiGet(List keys) { // TODO: Refactor this out to common package? private static String generateFeatureSetStringRef(FeatureSetSpec featureSetSpec) { String ref = String.format("%s/%s", featureSetSpec.getProject(), featureSetSpec.getName()); - if (featureSetSpec.getVersion() > 0) { - return ref + String.format(":%d", featureSetSpec.getVersion()); - } return ref; } } diff --git a/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/retriever/RedisOnlineRetrieverTest.java b/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/retriever/RedisOnlineRetrieverTest.java index 41bbfaa74c4..9e9ccbe53f6 100644 --- a/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/retriever/RedisOnlineRetrieverTest.java +++ b/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/retriever/RedisOnlineRetrieverTest.java @@ -64,14 +64,14 @@ public void setUp() { redisKeyList = Lists.newArrayList( RedisKey.newBuilder() - .setFeatureSet("project/featureSet:1") + .setFeatureSet("project/featureSet") .addAllEntities( Lists.newArrayList( Field.newBuilder().setName("entity1").setValue(intValue(1)).build(), Field.newBuilder().setName("entity2").setValue(strValue("a")).build())) .build(), RedisKey.newBuilder() - .setFeatureSet("project/featureSet:1") + .setFeatureSet("project/featureSet") .addAllEntities( Lists.newArrayList( Field.newBuilder().setName("entity1").setValue(intValue(2)).build(), @@ -89,17 +89,9 @@ public void shouldReturnResponseWithValuesIfKeysPresent() { FeatureSetRequest.newBuilder() .setSpec(getFeatureSetSpec()) .addFeatureReference( - FeatureReference.newBuilder() - .setName("feature1") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature1").setProject("project").build()) .addFeatureReference( - FeatureReference.newBuilder() - .setName("feature2") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature2").setProject("project").build()) .build(); List entityRows = ImmutableList.of( @@ -145,7 +137,7 @@ public void shouldReturnResponseWithValuesIfKeysPresent() { Lists.newArrayList( FeatureRow.newBuilder() .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) - .setFeatureSet("project/featureSet:1") + .setFeatureSet("project/featureSet") .addAllFields( Lists.newArrayList( Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), @@ -153,7 +145,7 @@ public void shouldReturnResponseWithValuesIfKeysPresent() { .build(), FeatureRow.newBuilder() .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) - .setFeatureSet("project/featureSet:1") + .setFeatureSet("project/featureSet") .addAllFields( Lists.newArrayList( Field.newBuilder().setName("feature1").setValue(intValue(2)).build(), @@ -171,17 +163,9 @@ public void shouldReturnResponseWithUnsetValuesIfKeysNotPresent() { FeatureSetRequest.newBuilder() .setSpec(getFeatureSetSpec()) .addFeatureReference( - FeatureReference.newBuilder() - .setName("feature1") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature1").setProject("project").build()) .addFeatureReference( - FeatureReference.newBuilder() - .setName("feature2") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature2").setProject("project").build()) .build(); List entityRows = ImmutableList.of( @@ -221,14 +205,14 @@ public void shouldReturnResponseWithUnsetValuesIfKeysNotPresent() { Lists.newArrayList( FeatureRow.newBuilder() .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) - .setFeatureSet("project/featureSet:1") + .setFeatureSet("project/featureSet") .addAllFields( Lists.newArrayList( Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), Field.newBuilder().setName("feature2").setValue(intValue(1)).build())) .build(), FeatureRow.newBuilder() - .setFeatureSet("project/featureSet:1") + .setFeatureSet("project/featureSet") .addAllFields( Lists.newArrayList( Field.newBuilder().setName("feature1").build(), @@ -252,7 +236,6 @@ private FeatureSetSpec getFeatureSetSpec() { return FeatureSetSpec.newBuilder() .setProject("project") .setName("featureSet") - .setVersion(1) .addEntities(EntitySpec.newBuilder().setName("entity1")) .addEntities(EntitySpec.newBuilder().setName("entity2")) .addFeatures(FeatureSpec.newBuilder().setName("feature1")) diff --git a/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/writer/RedisFeatureSinkTest.java b/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/writer/RedisFeatureSinkTest.java index beeabc2c884..c9d8cf4cfc5 100644 --- a/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/writer/RedisFeatureSinkTest.java +++ b/storage/connectors/redis/src/test/java/feast/storage/connectors/redis/writer/RedisFeatureSinkTest.java @@ -78,7 +78,6 @@ public void setUp() throws IOException { FeatureSetSpec spec1 = FeatureSetSpec.newBuilder() .setName("fs") - .setVersion(1) .setProject("myproject") .addEntities(EntitySpec.newBuilder().setName("entity").setValueType(Enum.INT64).build()) .addFeatures( @@ -89,7 +88,6 @@ public void setUp() throws IOException { FeatureSetSpec.newBuilder() .setName("feature_set") .setProject("myproject") - .setVersion(1) .addEntities( EntitySpec.newBuilder() .setName("entity_id_primary") @@ -107,7 +105,7 @@ public void setUp() throws IOException { .build(); Map specMap = - ImmutableMap.of("myproject/fs:1", spec1, "myproject/feature_set:1", spec2); + ImmutableMap.of("myproject/fs", spec1, "myproject/feature_set", spec2); StoreProto.Store.RedisConfig redisConfig = StoreProto.Store.RedisConfig.newBuilder().setHost(REDIS_HOST).setPort(REDIS_PORT).build(); @@ -127,7 +125,7 @@ public void shouldWriteToRedis() { HashMap kvs = new LinkedHashMap<>(); kvs.put( RedisKey.newBuilder() - .setFeatureSet("myproject/fs:1") + .setFeatureSet("myproject/fs") .addEntities(field("entity", 1, Enum.INT64)) .build(), FeatureRow.newBuilder() @@ -136,7 +134,7 @@ public void shouldWriteToRedis() { .build()); kvs.put( RedisKey.newBuilder() - .setFeatureSet("myproject/fs:1") + .setFeatureSet("myproject/fs") .addEntities(field("entity", 2, Enum.INT64)) .build(), FeatureRow.newBuilder() @@ -147,12 +145,12 @@ public void shouldWriteToRedis() { List featureRows = ImmutableList.of( FeatureRow.newBuilder() - .setFeatureSet("myproject/fs:1") + .setFeatureSet("myproject/fs") .addFields(field("entity", 1, Enum.INT64)) .addFields(field("feature", "one", Enum.STRING)) .build(), FeatureRow.newBuilder() - .setFeatureSet("myproject/fs:1") + .setFeatureSet("myproject/fs") .addFields(field("entity", 2, Enum.INT64)) .addFields(field("feature", "two", Enum.STRING)) .build()); @@ -181,7 +179,7 @@ public void shouldRetryFailConnection() throws InterruptedException { HashMap kvs = new LinkedHashMap<>(); kvs.put( RedisKey.newBuilder() - .setFeatureSet("myproject/fs:1") + .setFeatureSet("myproject/fs") .addEntities(field("entity", 1, Enum.INT64)) .build(), FeatureRow.newBuilder() @@ -192,7 +190,7 @@ public void shouldRetryFailConnection() throws InterruptedException { List featureRows = ImmutableList.of( FeatureRow.newBuilder() - .setFeatureSet("myproject/fs:1") + .setFeatureSet("myproject/fs") .addFields(field("entity", 1, Enum.INT64)) .addFields(field("feature", "one", Enum.STRING)) .build()); @@ -234,7 +232,7 @@ public void shouldProduceFailedElementIfRetryExceeded() { HashMap kvs = new LinkedHashMap<>(); kvs.put( RedisKey.newBuilder() - .setFeatureSet("myproject/fs:1") + .setFeatureSet("myproject/fs") .addEntities(field("entity", 1, Enum.INT64)) .build(), FeatureRow.newBuilder() @@ -245,7 +243,7 @@ public void shouldProduceFailedElementIfRetryExceeded() { List featureRows = ImmutableList.of( FeatureRow.newBuilder() - .setFeatureSet("myproject/fs:1") + .setFeatureSet("myproject/fs") .addFields(field("entity", 1, Enum.INT64)) .addFields(field("feature", "one", Enum.STRING)) .build()); @@ -266,7 +264,7 @@ public void shouldConvertRowWithDuplicateEntitiesToValidKey() { FeatureRow offendingRow = FeatureRow.newBuilder() - .setFeatureSet("myproject/feature_set:1") + .setFeatureSet("myproject/feature_set") .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) .addFields( Field.newBuilder() @@ -292,7 +290,7 @@ public void shouldConvertRowWithDuplicateEntitiesToValidKey() { RedisKey expectedKey = RedisKey.newBuilder() - .setFeatureSet("myproject/feature_set:1") + .setFeatureSet("myproject/feature_set") .addEntities( Field.newBuilder() .setName("entity_id_primary") @@ -322,7 +320,7 @@ public void shouldConvertRowWithDuplicateEntitiesToValidKey() { public void shouldConvertRowWithOutOfOrderFieldsToValidKey() { FeatureRow offendingRow = FeatureRow.newBuilder() - .setFeatureSet("myproject/feature_set:1") + .setFeatureSet("myproject/feature_set") .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) .addFields( Field.newBuilder() @@ -344,7 +342,7 @@ public void shouldConvertRowWithOutOfOrderFieldsToValidKey() { RedisKey expectedKey = RedisKey.newBuilder() - .setFeatureSet("myproject/feature_set:1") + .setFeatureSet("myproject/feature_set") .addEntities( Field.newBuilder() .setName("entity_id_primary") @@ -377,7 +375,7 @@ public void shouldConvertRowWithOutOfOrderFieldsToValidKey() { public void shouldMergeDuplicateFeatureFields() { FeatureRow featureRowWithDuplicatedFeatureFields = FeatureRow.newBuilder() - .setFeatureSet("myproject/feature_set:1") + .setFeatureSet("myproject/feature_set") .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) .addFields( Field.newBuilder() @@ -403,7 +401,7 @@ public void shouldMergeDuplicateFeatureFields() { RedisKey expectedKey = RedisKey.newBuilder() - .setFeatureSet("myproject/feature_set:1") + .setFeatureSet("myproject/feature_set") .addEntities( Field.newBuilder() .setName("entity_id_primary") @@ -433,7 +431,7 @@ public void shouldMergeDuplicateFeatureFields() { public void shouldPopulateMissingFeatureValuesWithDefaultInstance() { FeatureRow featureRowWithDuplicatedFeatureFields = FeatureRow.newBuilder() - .setFeatureSet("myproject/feature_set:1") + .setFeatureSet("myproject/feature_set") .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) .addFields( Field.newBuilder() @@ -451,7 +449,7 @@ public void shouldPopulateMissingFeatureValuesWithDefaultInstance() { RedisKey expectedKey = RedisKey.newBuilder() - .setFeatureSet("myproject/feature_set:1") + .setFeatureSet("myproject/feature_set") .addEntities( Field.newBuilder() .setName("entity_id_primary") diff --git a/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetriever.java b/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetriever.java index 713b6897b2d..1141e4eeed5 100644 --- a/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetriever.java +++ b/storage/connectors/rediscluster/src/main/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetriever.java @@ -218,9 +218,6 @@ private List sendMultiGet(List keys) { // TODO: Refactor this out to common package? private static String generateFeatureSetStringRef(FeatureSetSpec featureSetSpec) { String ref = String.format("%s/%s", featureSetSpec.getProject(), featureSetSpec.getName()); - if (featureSetSpec.getVersion() > 0) { - return ref + String.format(":%d", featureSetSpec.getVersion()); - } return ref; } } diff --git a/storage/connectors/rediscluster/src/test/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetrieverTest.java b/storage/connectors/rediscluster/src/test/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetrieverTest.java index 567e92a3d41..b89ed8b2286 100644 --- a/storage/connectors/rediscluster/src/test/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetrieverTest.java +++ b/storage/connectors/rediscluster/src/test/java/feast/storage/connectors/rediscluster/retriever/RedisClusterOnlineRetrieverTest.java @@ -64,14 +64,14 @@ public void setUp() { redisKeyList = Lists.newArrayList( RedisKey.newBuilder() - .setFeatureSet("project/featureSet:1") + .setFeatureSet("project/featureSet") .addAllEntities( Lists.newArrayList( Field.newBuilder().setName("entity1").setValue(intValue(1)).build(), Field.newBuilder().setName("entity2").setValue(strValue("a")).build())) .build(), RedisKey.newBuilder() - .setFeatureSet("project/featureSet:1") + .setFeatureSet("project/featureSet") .addAllEntities( Lists.newArrayList( Field.newBuilder().setName("entity1").setValue(intValue(2)).build(), @@ -89,17 +89,9 @@ public void shouldReturnResponseWithValuesIfKeysPresent() { FeatureSetRequest.newBuilder() .setSpec(getFeatureSetSpec()) .addFeatureReference( - FeatureReference.newBuilder() - .setName("feature1") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature1").setProject("project").build()) .addFeatureReference( - FeatureReference.newBuilder() - .setName("feature2") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature2").setProject("project").build()) .build(); List entityRows = ImmutableList.of( @@ -145,7 +137,7 @@ public void shouldReturnResponseWithValuesIfKeysPresent() { Lists.newArrayList( FeatureRow.newBuilder() .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) - .setFeatureSet("project/featureSet:1") + .setFeatureSet("project/featureSet") .addAllFields( Lists.newArrayList( Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), @@ -153,7 +145,7 @@ public void shouldReturnResponseWithValuesIfKeysPresent() { .build(), FeatureRow.newBuilder() .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) - .setFeatureSet("project/featureSet:1") + .setFeatureSet("project/featureSet") .addAllFields( Lists.newArrayList( Field.newBuilder().setName("feature1").setValue(intValue(2)).build(), @@ -171,17 +163,9 @@ public void shouldReturnResponseWithUnsetValuesIfKeysNotPresent() { FeatureSetRequest.newBuilder() .setSpec(getFeatureSetSpec()) .addFeatureReference( - FeatureReference.newBuilder() - .setName("feature1") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature1").setProject("project").build()) .addFeatureReference( - FeatureReference.newBuilder() - .setName("feature2") - .setVersion(1) - .setProject("project") - .build()) + FeatureReference.newBuilder().setName("feature2").setProject("project").build()) .build(); List entityRows = ImmutableList.of( @@ -221,14 +205,14 @@ public void shouldReturnResponseWithUnsetValuesIfKeysNotPresent() { Lists.newArrayList( FeatureRow.newBuilder() .setEventTimestamp(Timestamp.newBuilder().setSeconds(100)) - .setFeatureSet("project/featureSet:1") + .setFeatureSet("project/featureSet") .addAllFields( Lists.newArrayList( Field.newBuilder().setName("feature1").setValue(intValue(1)).build(), Field.newBuilder().setName("feature2").setValue(intValue(1)).build())) .build(), FeatureRow.newBuilder() - .setFeatureSet("project/featureSet:1") + .setFeatureSet("project/featureSet") .addAllFields( Lists.newArrayList( Field.newBuilder().setName("feature1").build(), @@ -252,7 +236,6 @@ private FeatureSetSpec getFeatureSetSpec() { return FeatureSetSpec.newBuilder() .setProject("project") .setName("featureSet") - .setVersion(1) .addEntities(EntitySpec.newBuilder().setName("entity1")) .addEntities(EntitySpec.newBuilder().setName("entity2")) .addFeatures(FeatureSpec.newBuilder().setName("feature1")) diff --git a/storage/connectors/rediscluster/src/test/java/feast/storage/connectors/rediscluster/writer/RedisClusterFeatureSinkTest.java b/storage/connectors/rediscluster/src/test/java/feast/storage/connectors/rediscluster/writer/RedisClusterFeatureSinkTest.java index cc1993636ee..c766d9b26c6 100644 --- a/storage/connectors/rediscluster/src/test/java/feast/storage/connectors/rediscluster/writer/RedisClusterFeatureSinkTest.java +++ b/storage/connectors/rediscluster/src/test/java/feast/storage/connectors/rediscluster/writer/RedisClusterFeatureSinkTest.java @@ -87,7 +87,6 @@ public void setUp() throws IOException { FeatureSetSpec spec1 = FeatureSetSpec.newBuilder() .setName("fs") - .setVersion(1) .setProject("myproject") .addEntities(EntitySpec.newBuilder().setName("entity").setValueType(Enum.INT64).build()) .addFeatures( @@ -98,7 +97,6 @@ public void setUp() throws IOException { FeatureSetSpec.newBuilder() .setName("feature_set") .setProject("myproject") - .setVersion(1) .addEntities( EntitySpec.newBuilder() .setName("entity_id_primary") @@ -116,7 +114,7 @@ public void setUp() throws IOException { .build(); Map specMap = - ImmutableMap.of("myproject/fs:1", spec1, "myproject/feature_set:1", spec2); + ImmutableMap.of("myproject/fs", spec1, "myproject/feature_set", spec2); RedisClusterConfig redisClusterConfig = RedisClusterConfig.newBuilder() .setConnectionString(CONNECTION_STRING) @@ -154,7 +152,7 @@ public void shouldWriteToRedis() { HashMap kvs = new LinkedHashMap<>(); kvs.put( RedisKey.newBuilder() - .setFeatureSet("myproject/fs:1") + .setFeatureSet("myproject/fs") .addEntities(field("entity", 1, Enum.INT64)) .build(), FeatureRow.newBuilder() @@ -163,7 +161,7 @@ public void shouldWriteToRedis() { .build()); kvs.put( RedisKey.newBuilder() - .setFeatureSet("myproject/fs:1") + .setFeatureSet("myproject/fs") .addEntities(field("entity", 2, Enum.INT64)) .build(), FeatureRow.newBuilder() @@ -174,12 +172,12 @@ public void shouldWriteToRedis() { List featureRows = ImmutableList.of( FeatureRow.newBuilder() - .setFeatureSet("myproject/fs:1") + .setFeatureSet("myproject/fs") .addFields(field("entity", 1, Enum.INT64)) .addFields(field("feature", "one", Enum.STRING)) .build(), FeatureRow.newBuilder() - .setFeatureSet("myproject/fs:1") + .setFeatureSet("myproject/fs") .addFields(field("entity", 2, Enum.INT64)) .addFields(field("feature", "two", Enum.STRING)) .build()); @@ -199,7 +197,7 @@ public void shouldRetryFailConnection() throws InterruptedException { HashMap kvs = new LinkedHashMap<>(); kvs.put( RedisKey.newBuilder() - .setFeatureSet("myproject/fs:1") + .setFeatureSet("myproject/fs") .addEntities(field("entity", 1, Enum.INT64)) .build(), FeatureRow.newBuilder() @@ -210,7 +208,7 @@ public void shouldRetryFailConnection() throws InterruptedException { List featureRows = ImmutableList.of( FeatureRow.newBuilder() - .setFeatureSet("myproject/fs:1") + .setFeatureSet("myproject/fs") .addFields(field("entity", 1, Enum.INT64)) .addFields(field("feature", "one", Enum.STRING)) .build()); @@ -254,13 +252,12 @@ public void shouldProduceFailedElementIfRetryExceeded() { FeatureSetSpec spec1 = FeatureSetSpec.newBuilder() .setName("fs") - .setVersion(1) .setProject("myproject") .addEntities(EntitySpec.newBuilder().setName("entity").setValueType(Enum.INT64).build()) .addFeatures( FeatureSpec.newBuilder().setName("feature").setValueType(Enum.STRING).build()) .build(); - Map specMap = ImmutableMap.of("myproject/fs:1", spec1); + Map specMap = ImmutableMap.of("myproject/fs", spec1); redisClusterFeatureSink = RedisClusterFeatureSink.builder() .setFeatureSetSpecs(specMap) @@ -271,7 +268,7 @@ public void shouldProduceFailedElementIfRetryExceeded() { List featureRows = ImmutableList.of( FeatureRow.newBuilder() - .setFeatureSet("myproject/fs:1") + .setFeatureSet("myproject/fs") .addFields(field("entity", 1, Enum.INT64)) .addFields(field("feature", "one", Enum.STRING)) .build()); @@ -291,7 +288,7 @@ public void shouldConvertRowWithDuplicateEntitiesToValidKey() { FeatureRow offendingRow = FeatureRow.newBuilder() - .setFeatureSet("myproject/feature_set:1") + .setFeatureSet("myproject/feature_set") .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) .addFields( Field.newBuilder() @@ -317,7 +314,7 @@ public void shouldConvertRowWithDuplicateEntitiesToValidKey() { RedisKey expectedKey = RedisKey.newBuilder() - .setFeatureSet("myproject/feature_set:1") + .setFeatureSet("myproject/feature_set") .addEntities( Field.newBuilder() .setName("entity_id_primary") @@ -347,7 +344,7 @@ public void shouldConvertRowWithDuplicateEntitiesToValidKey() { public void shouldConvertRowWithOutOfOrderFieldsToValidKey() { FeatureRow offendingRow = FeatureRow.newBuilder() - .setFeatureSet("myproject/feature_set:1") + .setFeatureSet("myproject/feature_set") .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) .addFields( Field.newBuilder() @@ -369,7 +366,7 @@ public void shouldConvertRowWithOutOfOrderFieldsToValidKey() { RedisKey expectedKey = RedisKey.newBuilder() - .setFeatureSet("myproject/feature_set:1") + .setFeatureSet("myproject/feature_set") .addEntities( Field.newBuilder() .setName("entity_id_primary") @@ -402,7 +399,7 @@ public void shouldConvertRowWithOutOfOrderFieldsToValidKey() { public void shouldMergeDuplicateFeatureFields() { FeatureRow featureRowWithDuplicatedFeatureFields = FeatureRow.newBuilder() - .setFeatureSet("myproject/feature_set:1") + .setFeatureSet("myproject/feature_set") .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) .addFields( Field.newBuilder() @@ -428,7 +425,7 @@ public void shouldMergeDuplicateFeatureFields() { RedisKey expectedKey = RedisKey.newBuilder() - .setFeatureSet("myproject/feature_set:1") + .setFeatureSet("myproject/feature_set") .addEntities( Field.newBuilder() .setName("entity_id_primary") @@ -459,7 +456,7 @@ public void shouldMergeDuplicateFeatureFields() { public void shouldPopulateMissingFeatureValuesWithDefaultInstance() { FeatureRow featureRowWithDuplicatedFeatureFields = FeatureRow.newBuilder() - .setFeatureSet("myproject/feature_set:1") + .setFeatureSet("myproject/feature_set") .setEventTimestamp(Timestamp.newBuilder().setSeconds(10)) .addFields( Field.newBuilder() @@ -477,7 +474,7 @@ public void shouldPopulateMissingFeatureValuesWithDefaultInstance() { RedisKey expectedKey = RedisKey.newBuilder() - .setFeatureSet("myproject/feature_set:1") + .setFeatureSet("myproject/feature_set") .addEntities( Field.newBuilder() .setName("entity_id_primary") diff --git a/tests/e2e/basic-ingest-redis-serving.py b/tests/e2e/basic-ingest-redis-serving.py index 50ec3854553..19c934e1067 100644 --- a/tests/e2e/basic-ingest-redis-serving.py +++ b/tests/e2e/basic-ingest-redis-serving.py @@ -557,7 +557,6 @@ def test_all_types_infer_register_ingest_file_success(client, # TODO: rewrite these using python SDK once the labels are implemented there class TestsBasedOnGrpc: - LAST_VERSION = 0 GRPC_CONNECTION_TIMEOUT = 3 LABEL_KEY = "my" LABEL_VALUE = "label" @@ -595,7 +594,7 @@ def get_feature_set(self, core_service_stub, name, project): try: get_feature_set_response = core_service_stub.GetFeatureSet( CoreService_pb2.GetFeatureSetRequest( - project=project, name=name.strip(), version=self.LAST_VERSION + project=project, name=name.strip(), ) ) # type: GetFeatureSetResponse except grpc.RpcError as e: diff --git a/tests/e2e/bq-batch-retrieval.py b/tests/e2e/bq-batch-retrieval.py index c1c6ab67805..a9a51894c77 100644 --- a/tests/e2e/bq-batch-retrieval.py +++ b/tests/e2e/bq-batch-retrieval.py @@ -1,28 +1,30 @@ +import math +import os import random import time +import uuid from datetime import datetime from datetime import timedelta from urllib.parse import urlparse -import os -import uuid import numpy as np import pandas as pd import pytest import pytz -from feast.core.IngestionJob_pb2 import IngestionJobStatus from feast.client import Client +from feast.core.CoreService_pb2 import ListStoresRequest +from feast.core.IngestionJob_pb2 import IngestionJobStatus from feast.entity import Entity from feast.feature import Feature from feast.feature_set import FeatureSet from feast.type_map import ValueType -from google.cloud import storage +from google.cloud import storage, bigquery from google.protobuf.duration_pb2 import Duration from pandavro import to_avro -pd.set_option('display.max_columns', None) +pd.set_option("display.max_columns", None) -PROJECT_NAME = 'batch_' + uuid.uuid4().hex.upper()[0:6] +PROJECT_NAME = "batch_" + uuid.uuid4().hex.upper()[0:6] @pytest.fixture(scope="module") @@ -56,62 +58,66 @@ def client(core_url, serving_url, allow_dirty): if not allow_dirty: feature_sets = client.list_feature_sets() if len(feature_sets) > 0: - raise Exception("Feast cannot have existing feature sets registered. Exiting tests.") + raise Exception( + "Feast cannot have existing feature sets registered. Exiting tests." + ) return client + @pytest.mark.first @pytest.mark.direct_runner @pytest.mark.dataflow_runner -def test_apply_all_featuresets(client): +@pytest.mark.run(order=1) +def test_batch_apply_all_featuresets(client): client.set_project(PROJECT_NAME) file_fs1 = FeatureSet( - "file_feature_set", - features=[Feature("feature_value1", ValueType.STRING)], - entities=[Entity("entity_id", ValueType.INT64)], - max_age=Duration(seconds=100), - ) + "file_feature_set", + features=[Feature("feature_value1", ValueType.STRING)], + entities=[Entity("entity_id", ValueType.INT64)], + max_age=Duration(seconds=100), + ) client.apply(file_fs1) gcs_fs1 = FeatureSet( - "gcs_feature_set", - features=[Feature("feature_value2", ValueType.STRING)], - entities=[Entity("entity_id", ValueType.INT64)], - max_age=Duration(seconds=100), - ) + "gcs_feature_set", + features=[Feature("feature_value2", ValueType.STRING)], + entities=[Entity("entity_id", ValueType.INT64)], + max_age=Duration(seconds=100), + ) client.apply(gcs_fs1) proc_time_fs = FeatureSet( - "processing_time", - features=[Feature("feature_value3", ValueType.STRING)], - entities=[Entity("entity_id", ValueType.INT64)], - max_age=Duration(seconds=100), - ) + "processing_time", + features=[Feature("feature_value3", ValueType.STRING)], + entities=[Entity("entity_id", ValueType.INT64)], + max_age=Duration(seconds=100), + ) client.apply(proc_time_fs) add_cols_fs = FeatureSet( - "additional_columns", - features=[Feature("feature_value4", ValueType.STRING)], - entities=[Entity("entity_id", ValueType.INT64)], - max_age=Duration(seconds=100), - ) + "additional_columns", + features=[Feature("feature_value4", ValueType.STRING)], + entities=[Entity("entity_id", ValueType.INT64)], + max_age=Duration(seconds=100), + ) client.apply(add_cols_fs) historical_fs = FeatureSet( - "historical", - features=[Feature("feature_value5", ValueType.STRING)], - entities=[Entity("entity_id", ValueType.INT64)], - max_age=Duration(seconds=100), - ) + "historical", + features=[Feature("feature_value5", ValueType.STRING)], + entities=[Entity("entity_id", ValueType.INT64)], + max_age=Duration(seconds=100), + ) client.apply(historical_fs) fs1 = FeatureSet( - "feature_set_1", - features=[Feature("feature_value6", ValueType.STRING)], - entities=[Entity("entity_id", ValueType.INT64)], - max_age=Duration(seconds=100), - ) + "feature_set_1", + features=[Feature("feature_value6", ValueType.STRING)], + entities=[Entity("entity_id", ValueType.INT64)], + max_age=Duration(seconds=100), + ) fs2 = FeatureSet( "feature_set_2", @@ -133,8 +139,9 @@ def test_apply_all_featuresets(client): @pytest.mark.direct_runner @pytest.mark.dataflow_runner -def test_get_batch_features_with_file(client): - file_fs1 = client.get_feature_set(name="file_feature_set", version=1) +@pytest.mark.run(order=10) +def test_batch_get_batch_features_with_file(client): + file_fs1 = client.get_feature_set(name="file_feature_set") N_ROWS = 10 time_offset = datetime.utcnow().replace(tzinfo=pytz.utc) @@ -150,23 +157,30 @@ def test_get_batch_features_with_file(client): # Rename column (datetime -> event_timestamp) features_1_df = features_1_df.rename(columns={"datetime": "event_timestamp"}) - to_avro(df=features_1_df[["event_timestamp", "entity_id"]], file_path_or_buffer="file_feature_set.avro") + to_avro( + df=features_1_df[["event_timestamp", "entity_id"]], + file_path_or_buffer="file_feature_set.avro", + ) time.sleep(15) feature_retrieval_job = client.get_batch_features( - entity_rows="file://file_feature_set.avro", feature_refs=[f"{PROJECT_NAME}/feature_value1:1"] + entity_rows="file://file_feature_set.avro", + feature_refs=[f"{PROJECT_NAME}/feature_value1"], ) output = feature_retrieval_job.to_dataframe() print(output.head()) - assert output["entity_id"].to_list() == [int(i) for i in output["feature_value1"].to_list()] + assert output["entity_id"].to_list() == [ + int(i) for i in output["feature_value1"].to_list() + ] @pytest.mark.direct_runner @pytest.mark.dataflow_runner -def test_get_batch_features_with_gs_path(client, gcs_path): - gcs_fs1 = client.get_feature_set(name="gcs_feature_set", version=1) +@pytest.mark.run(order=11) +def test_batch_get_batch_features_with_gs_path(client, gcs_path): + gcs_fs1 = client.get_feature_set(name="gcs_feature_set") N_ROWS = 10 time_offset = datetime.utcnow().replace(tzinfo=pytz.utc) @@ -184,7 +198,10 @@ def test_get_batch_features_with_gs_path(client, gcs_path): # Output file to local file_name = "gcs_feature_set.avro" - to_avro(df=features_1_df[["event_timestamp", "entity_id"]], file_path_or_buffer=file_name) + to_avro( + df=features_1_df[["event_timestamp", "entity_id"]], + file_path_or_buffer=file_name, + ) uri = urlparse(gcs_path) bucket = uri.hostname @@ -199,19 +216,21 @@ def test_get_batch_features_with_gs_path(client, gcs_path): time.sleep(15) feature_retrieval_job = client.get_batch_features( - entity_rows=f"{gcs_path}{ts}/*", - feature_refs=[f"{PROJECT_NAME}/feature_value2:1"] + entity_rows=f"{gcs_path}{ts}/*", feature_refs=[f"{PROJECT_NAME}/feature_value2"] ) output = feature_retrieval_job.to_dataframe() print(output.head()) - assert output["entity_id"].to_list() == [int(i) for i in output["feature_value2"].to_list()] + assert output["entity_id"].to_list() == [ + int(i) for i in output["feature_value2"].to_list() + ] @pytest.mark.direct_runner -def test_order_by_creation_time(client): - proc_time_fs = client.get_feature_set(name="processing_time", version=1) +@pytest.mark.run(order=12) +def test_batch_order_by_creation_time(client): + proc_time_fs = client.get_feature_set(name="processing_time") time_offset = datetime.utcnow().replace(tzinfo=pytz.utc) N_ROWS = 10 @@ -233,7 +252,8 @@ def test_order_by_creation_time(client): time.sleep(15) client.ingest(proc_time_fs, correct_df) feature_retrieval_job = client.get_batch_features( - entity_rows=incorrect_df[["datetime", "entity_id"]], feature_refs=[f"{PROJECT_NAME}/feature_value3:1"] + entity_rows=incorrect_df[["datetime", "entity_id"]], + feature_refs=[f"{PROJECT_NAME}/feature_value3"], ) output = feature_retrieval_job.to_dataframe() print(output.head()) @@ -242,13 +262,18 @@ def test_order_by_creation_time(client): @pytest.mark.direct_runner -def test_additional_columns_in_entity_table(client): - add_cols_fs = client.get_feature_set(name="additional_columns", version=1) +@pytest.mark.run(order=13) +def test_batch_additional_columns_in_entity_table(client): + add_cols_fs = client.get_feature_set(name="additional_columns") N_ROWS = 10 time_offset = datetime.utcnow().replace(tzinfo=pytz.utc) features_df = pd.DataFrame( - {"datetime": [time_offset] * N_ROWS, "entity_id": [i for i in range(N_ROWS)], "feature_value4": ["abc"] * N_ROWS} + { + "datetime": [time_offset] * N_ROWS, + "entity_id": [i for i in range(N_ROWS)], + "feature_value4": ["abc"] * N_ROWS, + } ) client.ingest(add_cols_fs, features_df) @@ -263,19 +288,25 @@ def test_additional_columns_in_entity_table(client): time.sleep(15) feature_retrieval_job = client.get_batch_features( - entity_rows=entity_df, feature_refs=[f"{PROJECT_NAME}/feature_value4:1"] + entity_rows=entity_df, feature_refs=[f"{PROJECT_NAME}/feature_value4"] ) output = feature_retrieval_job.to_dataframe().sort_values(by=["entity_id"]) print(output.head(10)) - assert np.allclose(output["additional_float_col"], entity_df["additional_float_col"]) - assert output["additional_string_col"].to_list() == entity_df["additional_string_col"].to_list() + assert np.allclose( + output["additional_float_col"], entity_df["additional_float_col"] + ) + assert ( + output["additional_string_col"].to_list() + == entity_df["additional_string_col"].to_list() + ) assert output["feature_value4"].to_list() == features_df["feature_value4"].to_list() @pytest.mark.direct_runner -def test_point_in_time_correctness_join(client): - historical_fs = client.get_feature_set(name="historical", version=1) +@pytest.mark.run(order=14) +def test_batch_point_in_time_correctness_join(client): + historical_fs = client.get_feature_set(name="historical") time_offset = datetime.utcnow().replace(tzinfo=pytz.utc) N_EXAMPLES = 10 @@ -292,13 +323,18 @@ def test_point_in_time_correctness_join(client): } ) entity_df = pd.DataFrame( - {"datetime": [time_offset - timedelta(seconds=10)] * N_EXAMPLES, "entity_id": [i for i in range(N_EXAMPLES)]} + { + "datetime": [time_offset - timedelta(seconds=10)] * N_EXAMPLES, + "entity_id": [i for i in range(N_EXAMPLES)], + } ) client.ingest(historical_fs, historical_df) time.sleep(15) - feature_retrieval_job = client.get_batch_features(entity_rows=entity_df, feature_refs=[f"{PROJECT_NAME}/feature_value5"]) + feature_retrieval_job = client.get_batch_features( + entity_rows=entity_df, feature_refs=[f"{PROJECT_NAME}/feature_value5"] + ) output = feature_retrieval_job.to_dataframe() print(output.head()) @@ -306,9 +342,10 @@ def test_point_in_time_correctness_join(client): @pytest.mark.direct_runner -def test_multiple_featureset_joins(client): - fs1 = client.get_feature_set(name="feature_set_1", version=1) - fs2 = client.get_feature_set(name="feature_set_2", version=1) +@pytest.mark.run(order=15) +def test_batch_multiple_featureset_joins(client): + fs1 = client.get_feature_set(name="feature_set_1") + fs2 = client.get_feature_set(name="feature_set_2") N_ROWS = 10 time_offset = datetime.utcnow().replace(tzinfo=pytz.utc) @@ -340,18 +377,27 @@ def test_multiple_featureset_joins(client): time.sleep(15) feature_retrieval_job = client.get_batch_features( - entity_rows=entity_df, feature_refs=[f"{PROJECT_NAME}/feature_value6:1", f"{PROJECT_NAME}/other_feature_value7:1"] + entity_rows=entity_df, + feature_refs=[ + f"{PROJECT_NAME}/feature_value6", + f"{PROJECT_NAME}/other_feature_value7", + ], ) output = feature_retrieval_job.to_dataframe() print(output.head()) - assert output["entity_id"].to_list() == [int(i) for i in output["feature_value6"].to_list()] - assert output["other_entity_id"].to_list() == output["other_feature_value7"].to_list() + assert output["entity_id"].to_list() == [ + int(i) for i in output["feature_value6"].to_list() + ] + assert ( + output["other_entity_id"].to_list() == output["other_feature_value7"].to_list() + ) @pytest.mark.direct_runner -def test_no_max_age(client): - no_max_age_fs = client.get_feature_set(name="no_max_age", version=1) +@pytest.mark.run(order=16) +def test_batch_no_max_age(client): + no_max_age_fs = client.get_feature_set(name="no_max_age") time_offset = datetime.utcnow().replace(tzinfo=pytz.utc) N_ROWS = 10 @@ -366,7 +412,8 @@ def test_no_max_age(client): time.sleep(15) feature_retrieval_job = client.get_batch_features( - entity_rows=features_8_df[["datetime", "entity_id"]], feature_refs=[f"{PROJECT_NAME}/feature_value8:1"] + entity_rows=features_8_df[["datetime", "entity_id"]], + feature_refs=[f"{PROJECT_NAME}/feature_value8"], ) output = feature_retrieval_job.to_dataframe() @@ -377,18 +424,191 @@ def test_no_max_age(client): @pytest.fixture(scope="module", autouse=True) def infra_teardown(pytestconfig, core_url, serving_url): - client = Client(core_url=core_url, serving_url=serving_url) - client.set_project(PROJECT_NAME) - - marker = pytestconfig.getoption("-m") - yield marker - if marker == 'dataflow_runner': - ingest_jobs = client.list_ingest_jobs() - ingest_jobs = [client.list_ingest_jobs(job.id)[0].external_id for job in ingest_jobs if job.status == IngestionJobStatus.RUNNING] - - cwd = os.getcwd() - with open(f"{cwd}/ingesting_jobs.txt", "w+") as output: - for job in ingest_jobs: - output.write('%s\n' % job) - else: - print('Cleaning up not required') + client = Client(core_url=core_url, serving_url=serving_url) + client.set_project(PROJECT_NAME) + + marker = pytestconfig.getoption("-m") + yield marker + if marker == "dataflow_runner": + ingest_jobs = client.list_ingest_jobs() + ingest_jobs = [ + client.list_ingest_jobs(job.id)[0].external_id + for job in ingest_jobs + if job.status == IngestionJobStatus.RUNNING + ] + + cwd = os.getcwd() + with open(f"{cwd}/ingesting_jobs.txt", "w+") as output: + for job in ingest_jobs: + output.write("%s\n" % job) + else: + print("Cleaning up not required") + + +@pytest.fixture(scope="module") +def update_featureset_dataframe(): + n_rows = 10 + time_offset = datetime.utcnow().replace(tzinfo=pytz.utc) + return pd.DataFrame( + { + "datetime": [time_offset] * n_rows, + "entity_id": [i for i in range(n_rows)], + "update_feature1": ["a" for i in range(n_rows)], + "update_feature2": [i + 2 for i in range(n_rows)], + "update_feature3": [i for i in range(n_rows)], + "update_feature4": ["b" for i in range(n_rows)], + } + ) + + +@pytest.mark.direct_runner +@pytest.mark.run(order=20) +def test_update_featureset_apply_featureset_and_ingest_first_subset( + client, update_featureset_dataframe +): + subset_columns = ["datetime", "entity_id", "update_feature1", "update_feature2"] + subset_df = update_featureset_dataframe.iloc[:5][subset_columns] + update_fs = FeatureSet( + "update_fs", + entities=[Entity(name="entity_id", dtype=ValueType.INT64)], + max_age=Duration(seconds=432000), + ) + update_fs.infer_fields_from_df(subset_df) + client.apply(update_fs) + + client.ingest(feature_set=update_fs, source=subset_df) + + time.sleep(15) + feature_retrieval_job = client.get_batch_features( + entity_rows=update_featureset_dataframe[["datetime", "entity_id"]].iloc[:5], + feature_refs=[ + f"{PROJECT_NAME}/update_feature1", + f"{PROJECT_NAME}/update_feature2", + ], + ) + + output = feature_retrieval_job.to_dataframe().sort_values(by=["entity_id"]) + print(output.head()) + + assert output["update_feature1"].to_list() == subset_df["update_feature1"].to_list() + assert output["update_feature2"].to_list() == subset_df["update_feature2"].to_list() + + +@pytest.mark.direct_runner +@pytest.mark.timeout(600) +@pytest.mark.run(order=21) +def test_update_featureset_update_featureset_and_ingest_second_subset( + client, update_featureset_dataframe +): + subset_columns = [ + "datetime", + "entity_id", + "update_feature1", + "update_feature3", + "update_feature4", + ] + subset_df = update_featureset_dataframe.iloc[5:][subset_columns] + update_fs = FeatureSet( + "update_fs", + entities=[Entity(name="entity_id", dtype=ValueType.INT64)], + max_age=Duration(seconds=432000), + ) + update_fs.infer_fields_from_df(subset_df) + client.apply(update_fs) + + # We keep retrying this ingestion until all values make it into the buffer. + # This is a necessary step because bigquery streaming caches table schemas + # and as a result, rows may be lost. + while True: + ingestion_id = client.ingest(feature_set=update_fs, source=subset_df) + time.sleep(15) # wait for rows to get written to bq + rows_ingested = get_rows_ingested(client, update_fs, ingestion_id) + if rows_ingested == len(subset_df): + print(f"Number of rows successfully ingested: {rows_ingested}. Continuing.") + break + print( + f"Number of rows successfully ingested: {rows_ingested}. Retrying ingestion." + ) + time.sleep(30) + + feature_retrieval_job = client.get_batch_features( + entity_rows=update_featureset_dataframe[["datetime", "entity_id"]].iloc[5:], + feature_refs=[ + f"{PROJECT_NAME}/update_feature1", + f"{PROJECT_NAME}/update_feature3", + f"{PROJECT_NAME}/update_feature4", + ], + ) + + output = feature_retrieval_job.to_dataframe().sort_values(by=["entity_id"]) + print(output.head()) + + assert output["update_feature1"].to_list() == subset_df["update_feature1"].to_list() + assert output["update_feature3"].to_list() == subset_df["update_feature3"].to_list() + assert output["update_feature4"].to_list() == subset_df["update_feature4"].to_list() + + +@pytest.mark.direct_runner +@pytest.mark.run(order=22) +def test_update_featureset_retrieve_all_fields(client, update_featureset_dataframe): + with pytest.raises(Exception): + feature_retrieval_job = client.get_batch_features( + entity_rows=update_featureset_dataframe[["datetime", "entity_id"]], + feature_refs=[ + f"{PROJECT_NAME}/update_feature1", + f"{PROJECT_NAME}/update_feature2", + f"{PROJECT_NAME}/update_feature3", + f"{PROJECT_NAME}/update_feature4", + ], + ) + feature_retrieval_job.result() + + +@pytest.mark.direct_runner +@pytest.mark.run(order=23) +def test_update_featureset_retrieve_valid_fields(client, update_featureset_dataframe): + feature_retrieval_job = client.get_batch_features( + entity_rows=update_featureset_dataframe[["datetime", "entity_id"]], + feature_refs=[ + f"{PROJECT_NAME}/update_feature1", + f"{PROJECT_NAME}/update_feature3", + f"{PROJECT_NAME}/update_feature4", + ], + ) + output = feature_retrieval_job.to_dataframe().sort_values(by=["entity_id"]) + print(output.head(10)) + assert ( + output["update_feature1"].to_list() + == update_featureset_dataframe["update_feature1"].to_list() + ) + # we have to convert to float because the column contains np.NaN + assert [math.isnan(i) for i in output["update_feature3"].to_list()[:5]] == [ + True + ] * 5 + assert output["update_feature3"].to_list()[5:] == [ + float(i) for i in update_featureset_dataframe["update_feature3"].to_list()[5:] + ] + assert ( + output["update_feature4"].to_list() + == [None] * 5 + update_featureset_dataframe["update_feature4"].to_list()[5:] + ) + + +def get_rows_ingested( + client: Client, feature_set: FeatureSet, ingestion_id: str +) -> int: + response = client._core_service_stub.ListStores( + ListStoresRequest(filter=ListStoresRequest.Filter(name="historical")) + ) + bq_config = response.store[0].bigquery_config + project = bq_config.project_id + dataset = bq_config.dataset_id + table = f"{PROJECT_NAME}_{feature_set.name}" + + bq_client = bigquery.Client(project=project) + rows = bq_client.query( + f'SELECT COUNT(*) as count FROM `{project}.{dataset}.{table}` WHERE ingestion_id = "{ingestion_id}"' + ).result() + + for row in rows: + return row["count"] From 561b62152a7869dc505576d4e063e820cc71b718 Mon Sep 17 00:00:00 2001 From: junhui096 <35248886+junhui096@users.noreply.github.com> Date: Thu, 14 May 2020 09:56:26 +0800 Subject: [PATCH 162/176] Kafka producer should raise an exception when it fails to connect to broker (#636) * Added exception to flush when produce fails with unit test or with messages in queue. * Fix: Throw exception in callback * Removed error_count property in abstract_producer Co-authored-by: Willem Pienaar <6728866+woop@users.noreply.github.com> --- sdk/python/feast/loaders/abstract_producer.py | 34 ++++++------------- sdk/python/tests/test_client.py | 32 +++++++++++++++++ 2 files changed, 42 insertions(+), 24 deletions(-) diff --git a/sdk/python/feast/loaders/abstract_producer.py b/sdk/python/feast/loaders/abstract_producer.py index 6030d14ecc0..14d9bc42b7d 100644 --- a/sdk/python/feast/loaders/abstract_producer.py +++ b/sdk/python/feast/loaders/abstract_producer.py @@ -25,8 +25,6 @@ class AbstractProducer: def __init__(self, brokers: str, row_count: int, disable_progress_bar: bool): self.brokers = brokers self.row_count = row_count - self.error_count = 0 - self.last_exception = "" # Progress bar will always display average rate self.pbar = tqdm( @@ -45,8 +43,7 @@ def _inc_pbar(self, meta): self.pbar.update(1) def _set_error(self, exception: str): - self.error_count += 1 - self.last_exception = exception + raise Exception(exception) def print_results(self) -> None: """ @@ -62,24 +59,7 @@ def print_results(self) -> None: print("Ingestion complete!") - failed_message = ( - "" - if self.error_count == 0 - else f"\nFail: {self.error_count / self.row_count}" - ) - - last_exception_message = ( - "" - if self.last_exception == "" - else f"\nLast exception:\n{self.last_exception}" - ) - - print( - f"\nIngestion statistics:" - f"\nSuccess: {self.pbar.n}/{self.row_count}" - f"{failed_message}" - f"{last_exception_message}" - ) + print(f"\nIngestion statistics:" f"\nSuccess: {self.pbar.n}/{self.row_count}") return None @@ -129,7 +109,10 @@ def flush(self, timeout: Optional[int]): Returns: int: Number of messages still in queue. """ - return self.producer.flush(timeout=timeout) + messages = self.producer.flush(timeout=timeout) + if messages: + raise Exception("Not all Kafka messages are successfully delivered.") + return messages def _delivery_callback(self, err: str, msg) -> None: """ @@ -200,7 +183,10 @@ def flush(self, timeout: Optional[int]): KafkaTimeoutError: failure to flush buffered records within the provided timeout """ - return self.producer.flush(timeout=timeout) + messages = self.producer.flush(timeout=timeout) + if messages: + raise Exception("Not all Kafka messages are successfully delivered.") + return messages def get_producer( diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index e87c8573353..a39c3a33816 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -601,6 +601,38 @@ def test_feature_set_ingest_success(self, dataframe, test_client, mocker): # Ingest data into Feast test_client.ingest("driver-feature-set", dataframe) + @pytest.mark.parametrize( + "dataframe,test_client,exception", + [(dataframes.GOOD, pytest.lazy_fixture("client"), Exception)], + ) + def test_feature_set_ingest_throws_exception_if_kafka_down( + self, dataframe, test_client, exception, mocker + ): + + test_client.set_project("project1") + driver_fs = FeatureSet( + "driver-feature-set", + source=KafkaSource(brokers="localhost:4412", topic="test"), + ) + driver_fs.add(Feature(name="feature_1", dtype=ValueType.FLOAT)) + driver_fs.add(Feature(name="feature_2", dtype=ValueType.STRING)) + driver_fs.add(Feature(name="feature_3", dtype=ValueType.INT64)) + driver_fs.add(Entity(name="entity_id", dtype=ValueType.INT64)) + + # Register with Feast core + test_client.apply(driver_fs) + driver_fs = driver_fs.to_proto() + driver_fs.meta.status = FeatureSetStatusProto.STATUS_READY + + mocker.patch.object( + test_client._core_service_stub, + "GetFeatureSet", + return_value=GetFeatureSetResponse(feature_set=driver_fs), + ) + + with pytest.raises(exception): + test_client.ingest("driver-feature-set", dataframe) + @pytest.mark.parametrize( "dataframe,exception,test_client", [ From 963a23f1ed466b7a613ce3587085d942b8999a5e Mon Sep 17 00:00:00 2001 From: Khor Shu Heng Date: Thu, 14 May 2020 17:58:38 +0800 Subject: [PATCH 163/176] Fix Feast Core docker image --- .prow/config.yaml | 5 ----- infra/docker/core/Dockerfile | 2 +- 2 files changed, 1 insertion(+), 6 deletions(-) diff --git a/.prow/config.yaml b/.prow/config.yaml index a5f7cb17df3..62124578f0e 100644 --- a/.prow/config.yaml +++ b/.prow/config.yaml @@ -293,11 +293,6 @@ presubmits: --file infra/docker/serving/Dockerfile \ --google-service-account-file /etc/gcloud/service-account.json - docker tag gcr.io/kf-feast/feast-core:${PULL_PULL_SHA:1} - docker push gcr.io/kf-feast/feast-core:${PULL_PULL_SHA:1} - - docker tag gcr.io/kf-feast/feast-serving:${PULL_PULL_SHA:1} - docker push gcr.io/kf-feast/feast-serving:${PULL_PULL_SHA:1} volumeMounts: - name: docker-socket mountPath: /var/run/docker.sock diff --git a/infra/docker/core/Dockerfile b/infra/docker/core/Dockerfile index c7ba81a4134..3fae3c19e80 100644 --- a/infra/docker/core/Dockerfile +++ b/infra/docker/core/Dockerfile @@ -2,7 +2,7 @@ # Build stage 1: Builder # ============================================================ -FROM maven:3.6-jdk-11-slim as builder +FROM maven:3.6-jdk-11 as builder ARG REVISION=dev COPY . /build WORKDIR /build From 1df2e70d6bf9bb1ca37191a6983aaa04beb32cf7 Mon Sep 17 00:00:00 2001 From: Terence Lim Date: Fri, 15 May 2020 10:54:37 +0800 Subject: [PATCH 164/176] Update Dataflow tests to correct compute region (#699) --- infra/scripts/test-end-to-end-batch-dataflow.sh | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/infra/scripts/test-end-to-end-batch-dataflow.sh b/infra/scripts/test-end-to-end-batch-dataflow.sh index 9d10b94dae0..1d75463b0ec 100644 --- a/infra/scripts/test-end-to-end-batch-dataflow.sh +++ b/infra/scripts/test-end-to-end-batch-dataflow.sh @@ -101,18 +101,18 @@ ip_count=0 for ip_addr_name in $feast_kafka_1_ip_name $feast_kafka_2_ip_name $feast_kafka_3_ip_name $feast_redis_ip_name $feast_statsd_ip_name do if [[ "$ip_count" == 0 ]]; then - export feast_kafka_1_ip=$(gcloud compute addresses describe ${ip_addr_name} --region=asia-east1 --format "value(address)") + export feast_kafka_1_ip=$(gcloud compute addresses describe ${ip_addr_name} --region=${GCLOUD_REGION} --format "value(address)") elif [[ "$ip_count" == 1 ]]; then - export feast_kafka_2_ip=$(gcloud compute addresses describe ${ip_addr_name} --region=asia-east1 --format "value(address)") + export feast_kafka_2_ip=$(gcloud compute addresses describe ${ip_addr_name} --region=${GCLOUD_REGION} --format "value(address)") elif [[ "$ip_count" == 2 ]]; then - export feast_kafka_3_ip=$(gcloud compute addresses describe ${ip_addr_name} --region=asia-east1 --format "value(address)") + export feast_kafka_3_ip=$(gcloud compute addresses describe ${ip_addr_name} --region=${GCLOUD_REGION} --format "value(address)") elif [[ "$ip_count" == 3 ]]; then - export feast_redis_ip=$(gcloud compute addresses describe ${ip_addr_name} --region=asia-east1 --format "value(address)") + export feast_redis_ip=$(gcloud compute addresses describe ${ip_addr_name} --region=${GCLOUD_REGION} --format "value(address)") elif [[ "$ip_count" == 4 ]]; then - export feast_statsd_ip=$(gcloud compute addresses describe ${ip_addr_name} --region=asia-east1 --format "value(address)") + export feast_statsd_ip=$(gcloud compute addresses describe ${ip_addr_name} --region=${GCLOUD_REGION} --format "value(address)") fi ip_count=$((ip_count + 1)) - export "$(echo $ip_addr_name | tr '-' '_')=$(gcloud compute addresses describe ${ip_addr_name} --region=asia-east1 --format "value(address)")" + export "$(echo $ip_addr_name | tr '-' '_')=$(gcloud compute addresses describe ${ip_addr_name} --region=${GCLOUD_REGION} --format "value(address)")" done echo " @@ -242,5 +242,5 @@ gcloud container clusters delete --region=${GCLOUD_REGION} ${K8_CLUSTER_NAME} while read line do echo $line - gcloud dataflow jobs cancel $line --region=asia-east1 + gcloud dataflow jobs cancel $line --region=${GCLOUD_REGION} done < ingesting_jobs.txt \ No newline at end of file From 727ec0a9b39444e97fd1e24bf2223bf1d03d122e Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Fri, 15 May 2020 13:15:27 +0800 Subject: [PATCH 165/176] Extract fs update tests so ci doesn't run it (#709) * Tag fs update tests separate from direct runner tests * Add prow config --- .prow/config.yaml | 21 +++++++++++++++++++++ infra/scripts/test-end-to-end-batch.sh | 16 +++++++++++++++- tests/e2e/bq-batch-retrieval.py | 19 +++++++++++++++---- 3 files changed, 51 insertions(+), 5 deletions(-) diff --git a/.prow/config.yaml b/.prow/config.yaml index 62124578f0e..2cdb4918312 100644 --- a/.prow/config.yaml +++ b/.prow/config.yaml @@ -204,6 +204,27 @@ presubmits: skip_branches: - ^v0\.(3|4)-branch$ + - name: test-end-to-end-batch-fs-update + decorate: true + always_run: false + spec: + volumes: + - name: service-account + secret: + secretName: feast-service-account + containers: + - image: maven:3.6-jdk-11 + command: ["infra/scripts/test-end-to-end-batch.sh", "-m", "fs_update"] + resources: + requests: + cpu: "6" + memory: "6144Mi" + volumeMounts: + - name: service-account + mountPath: "/etc/service-account" + skip_branches: + - ^v0\.(3|4)-branch$ + - name: test-end-to-end-batch-java-8 decorate: true always_run: true diff --git a/infra/scripts/test-end-to-end-batch.sh b/infra/scripts/test-end-to-end-batch.sh index 4b18f6a0678..fe24c0df33c 100755 --- a/infra/scripts/test-end-to-end-batch.sh +++ b/infra/scripts/test-end-to-end-batch.sh @@ -3,6 +3,20 @@ set -e set -o pipefail +PYTEST_MARK='direct_runner' #default + +print_usage() { + printf "Usage: ./test-end-to-end-batch -m pytest_mark" +} + +while getopts 'm:' flag; do + case "${flag}" in + m) PYTEST_MARK="${OPTARG}" ;; + *) print_usage + exit 1 ;; + esac +done + test -z ${GOOGLE_APPLICATION_CREDENTIALS} && GOOGLE_APPLICATION_CREDENTIALS="/etc/service-account/service-account.json" test -z ${SKIP_BUILD_JARS} && SKIP_BUILD_JARS="false" test -z ${GOOGLE_CLOUD_PROJECT} && GOOGLE_CLOUD_PROJECT="kf-feast" @@ -254,7 +268,7 @@ ORIGINAL_DIR=$(pwd) cd tests/e2e set +e -pytest bq-batch-retrieval.py -m direct_runner --gcs_path "gs://${TEMP_BUCKET}/" --junitxml=${LOGS_ARTIFACT_PATH}/python-sdk-test-report.xml +pytest bq-batch-retrieval.py -m ${PYTEST_MARK} --gcs_path "gs://${TEMP_BUCKET}/" --junitxml=${LOGS_ARTIFACT_PATH}/python-sdk-test-report.xml TEST_EXIT_CODE=$? if [[ ${TEST_EXIT_CODE} != 0 ]]; then diff --git a/tests/e2e/bq-batch-retrieval.py b/tests/e2e/bq-batch-retrieval.py index a9a51894c77..35075c92bbb 100644 --- a/tests/e2e/bq-batch-retrieval.py +++ b/tests/e2e/bq-batch-retrieval.py @@ -445,6 +445,17 @@ def infra_teardown(pytestconfig, core_url, serving_url): print("Cleaning up not required") + +''' +This suite of tests tests the apply feature set - update feature set - retrieve +event sequence. It ensures that when a feature set is updated, tombstoned features +are no longer retrieved, and added features are null for previously ingested +rows. + +It is marked separately because of the length of time required +to perform this test, due to bigquery schema caching for streaming writes. +''' + @pytest.fixture(scope="module") def update_featureset_dataframe(): n_rows = 10 @@ -461,7 +472,7 @@ def update_featureset_dataframe(): ) -@pytest.mark.direct_runner +@pytest.mark.fs_update @pytest.mark.run(order=20) def test_update_featureset_apply_featureset_and_ingest_first_subset( client, update_featureset_dataframe @@ -494,7 +505,7 @@ def test_update_featureset_apply_featureset_and_ingest_first_subset( assert output["update_feature2"].to_list() == subset_df["update_feature2"].to_list() -@pytest.mark.direct_runner +@pytest.mark.fs_update @pytest.mark.timeout(600) @pytest.mark.run(order=21) def test_update_featureset_update_featureset_and_ingest_second_subset( @@ -548,7 +559,7 @@ def test_update_featureset_update_featureset_and_ingest_second_subset( assert output["update_feature4"].to_list() == subset_df["update_feature4"].to_list() -@pytest.mark.direct_runner +@pytest.mark.fs_update @pytest.mark.run(order=22) def test_update_featureset_retrieve_all_fields(client, update_featureset_dataframe): with pytest.raises(Exception): @@ -564,7 +575,7 @@ def test_update_featureset_retrieve_all_fields(client, update_featureset_datafra feature_retrieval_job.result() -@pytest.mark.direct_runner +@pytest.mark.fs_update @pytest.mark.run(order=23) def test_update_featureset_retrieve_valid_fields(client, update_featureset_dataframe): feature_retrieval_job = client.get_batch_features( From 8ab00e0c5479bf2e1994814a263989f02aed723a Mon Sep 17 00:00:00 2001 From: Chen Zhiling Date: Fri, 15 May 2020 14:12:27 +0800 Subject: [PATCH 166/176] Ensure that batch retrieval tests clean up after themselves (#704) * Ensure that batch retrieval tests clean up after themselves, reduce flakiness of file tests * Supply client * Move test exit code --- .../scripts/test-end-to-end-batch-dataflow.sh | 5 ++-- infra/scripts/test-end-to-end-batch.sh | 3 ++- tests/e2e/bq-batch-retrieval.py | 23 +++++++++++++++++++ 3 files changed, 28 insertions(+), 3 deletions(-) diff --git a/infra/scripts/test-end-to-end-batch-dataflow.sh b/infra/scripts/test-end-to-end-batch-dataflow.sh index 1d75463b0ec..9b3c8b5a83b 100644 --- a/infra/scripts/test-end-to-end-batch-dataflow.sh +++ b/infra/scripts/test-end-to-end-batch-dataflow.sh @@ -216,7 +216,6 @@ if [[ ${TEST_EXIT_CODE} != 0 ]]; then fi cd ${ORIGINAL_DIR} -exit ${TEST_EXIT_CODE} echo " ============================================================ @@ -243,4 +242,6 @@ while read line do echo $line gcloud dataflow jobs cancel $line --region=${GCLOUD_REGION} -done < ingesting_jobs.txt \ No newline at end of file +done < ingesting_jobs.txt + +exit ${TEST_EXIT_CODE} diff --git a/infra/scripts/test-end-to-end-batch.sh b/infra/scripts/test-end-to-end-batch.sh index fe24c0df33c..ffaba183a9d 100755 --- a/infra/scripts/test-end-to-end-batch.sh +++ b/infra/scripts/test-end-to-end-batch.sh @@ -281,7 +281,6 @@ if [[ ${TEST_EXIT_CODE} != 0 ]]; then fi cd ${ORIGINAL_DIR} -exit ${TEST_EXIT_CODE} echo " ============================================================ @@ -290,3 +289,5 @@ Cleaning up " bq rm -r -f ${GOOGLE_CLOUD_PROJECT}:${DATASET_NAME} + +exit ${TEST_EXIT_CODE} \ No newline at end of file diff --git a/tests/e2e/bq-batch-retrieval.py b/tests/e2e/bq-batch-retrieval.py index 35075c92bbb..a4d8a729ef7 100644 --- a/tests/e2e/bq-batch-retrieval.py +++ b/tests/e2e/bq-batch-retrieval.py @@ -19,6 +19,7 @@ from feast.feature_set import FeatureSet from feast.type_map import ValueType from google.cloud import storage, bigquery +from google.cloud.storage import Blob from google.protobuf.duration_pb2 import Duration from pandavro import to_avro @@ -155,6 +156,7 @@ def test_batch_get_batch_features_with_file(client): client.ingest(file_fs1, features_1_df, timeout=480) # Rename column (datetime -> event_timestamp) + features_1_df['datetime'] + pd.Timedelta(seconds=1) # adds buffer to avoid rounding errors features_1_df = features_1_df.rename(columns={"datetime": "event_timestamp"}) to_avro( @@ -169,6 +171,7 @@ def test_batch_get_batch_features_with_file(client): ) output = feature_retrieval_job.to_dataframe() + clean_up_remote_files(feature_retrieval_job.get_avro_files()) print(output.head()) assert output["entity_id"].to_list() == [ @@ -194,6 +197,7 @@ def test_batch_get_batch_features_with_gs_path(client, gcs_path): client.ingest(gcs_fs1, features_1_df, timeout=360) # Rename column (datetime -> event_timestamp) + features_1_df['datetime'] + pd.Timedelta(seconds=1) # adds buffer to avoid rounding errors features_1_df = features_1_df.rename(columns={"datetime": "event_timestamp"}) # Output file to local @@ -220,6 +224,8 @@ def test_batch_get_batch_features_with_gs_path(client, gcs_path): ) output = feature_retrieval_job.to_dataframe() + clean_up_remote_files(feature_retrieval_job.get_avro_files()) + blob.delete() print(output.head()) assert output["entity_id"].to_list() == [ @@ -256,6 +262,7 @@ def test_batch_order_by_creation_time(client): feature_refs=[f"{PROJECT_NAME}/feature_value3"], ) output = feature_retrieval_job.to_dataframe() + clean_up_remote_files(feature_retrieval_job.get_avro_files()) print(output.head()) assert output["feature_value3"].to_list() == ["CORRECT"] * N_ROWS @@ -291,6 +298,7 @@ def test_batch_additional_columns_in_entity_table(client): entity_rows=entity_df, feature_refs=[f"{PROJECT_NAME}/feature_value4"] ) output = feature_retrieval_job.to_dataframe().sort_values(by=["entity_id"]) + clean_up_remote_files(feature_retrieval_job.get_avro_files()) print(output.head(10)) assert np.allclose( @@ -336,6 +344,7 @@ def test_batch_point_in_time_correctness_join(client): entity_rows=entity_df, feature_refs=[f"{PROJECT_NAME}/feature_value5"] ) output = feature_retrieval_job.to_dataframe() + clean_up_remote_files(feature_retrieval_job.get_avro_files()) print(output.head()) assert output["feature_value5"].to_list() == ["CORRECT"] * N_EXAMPLES @@ -384,6 +393,7 @@ def test_batch_multiple_featureset_joins(client): ], ) output = feature_retrieval_job.to_dataframe() + clean_up_remote_files(feature_retrieval_job.get_avro_files()) print(output.head()) assert output["entity_id"].to_list() == [ @@ -417,6 +427,7 @@ def test_batch_no_max_age(client): ) output = feature_retrieval_job.to_dataframe() + clean_up_remote_files(feature_retrieval_job.get_avro_files()) print(output.head()) assert output["entity_id"].to_list() == output["feature_value8"].to_list() @@ -499,6 +510,7 @@ def test_update_featureset_apply_featureset_and_ingest_first_subset( ) output = feature_retrieval_job.to_dataframe().sort_values(by=["entity_id"]) + clean_up_remote_files(feature_retrieval_job.get_avro_files()) print(output.head()) assert output["update_feature1"].to_list() == subset_df["update_feature1"].to_list() @@ -552,6 +564,7 @@ def test_update_featureset_update_featureset_and_ingest_second_subset( ) output = feature_retrieval_job.to_dataframe().sort_values(by=["entity_id"]) + clean_up_remote_files(feature_retrieval_job.get_avro_files()) print(output.head()) assert output["update_feature1"].to_list() == subset_df["update_feature1"].to_list() @@ -587,6 +600,7 @@ def test_update_featureset_retrieve_valid_fields(client, update_featureset_dataf ], ) output = feature_retrieval_job.to_dataframe().sort_values(by=["entity_id"]) + clean_up_remote_files(feature_retrieval_job.get_avro_files()) print(output.head(10)) assert ( output["update_feature1"].to_list() @@ -623,3 +637,12 @@ def get_rows_ingested( for row in rows: return row["count"] + + +def clean_up_remote_files(files): + storage_client = storage.Client() + for file_uri in files: + if file_uri.scheme == "gs": + blob = Blob.from_string(file_uri.geturl(), client=storage_client) + blob.delete() + From 1f40995b44dcd21cee18db61feb349980df31679 Mon Sep 17 00:00:00 2001 From: zhilingc Date: Fri, 15 May 2020 14:19:40 +0800 Subject: [PATCH 167/176] Make executable --- infra/scripts/test-end-to-end-batch-dataflow.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 infra/scripts/test-end-to-end-batch-dataflow.sh diff --git a/infra/scripts/test-end-to-end-batch-dataflow.sh b/infra/scripts/test-end-to-end-batch-dataflow.sh old mode 100644 new mode 100755 From abb53e2d1eeb3650c8047bf5af7d4ae23471fa1e Mon Sep 17 00:00:00 2001 From: Khor Shu Heng Date: Fri, 15 May 2020 11:00:19 +0800 Subject: [PATCH 168/176] Remove feature set status check for job update requirement --- .../src/main/java/feast/core/job/JobUpdateTask.java | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/core/src/main/java/feast/core/job/JobUpdateTask.java b/core/src/main/java/feast/core/job/JobUpdateTask.java index b508aa46d2f..e205b81b3cb 100644 --- a/core/src/main/java/feast/core/job/JobUpdateTask.java +++ b/core/src/main/java/feast/core/job/JobUpdateTask.java @@ -17,7 +17,6 @@ package feast.core.job; import com.google.common.collect.Sets; -import feast.core.FeatureSetProto.FeatureSetStatus; import feast.core.log.Action; import feast.core.log.AuditLogger; import feast.core.log.Resource; @@ -103,17 +102,7 @@ public Job call() { boolean requiresUpdate(Job job) { // If set of feature sets has changed - if (!Sets.newHashSet(featureSets).equals(Sets.newHashSet(job.getFeatureSets()))) { - return true; - } - - // If any existing feature set populated by the job has its status as pending - for (FeatureSet featureSet : job.getFeatureSets()) { - if (featureSet.getStatus().equals(FeatureSetStatus.STATUS_PENDING)) { - return true; - } - } - return false; + return !Sets.newHashSet(featureSets).equals(Sets.newHashSet(job.getFeatureSets())); } private Job createJob() { From 09984b3fd118ebeb0595f895516eb76c56ca22be Mon Sep 17 00:00:00 2001 From: Ches Martin Date: Fri, 15 May 2020 15:21:27 +0700 Subject: [PATCH 169/176] Add Java code coverage reporting with JaCoCo (#686) --- .dockerignore | 1 + .github/workflows/unit_tests.yml | 6 +- Makefile | 5 +- core/pom.xml | 6 +- docs/coverage/java/pom.xml | 111 +++++++++++++++++++++++++++++++ ingestion/pom.xml | 5 ++ pom.xml | 17 ++++- sdk/java/pom.xml | 8 +-- serving/pom.xml | 7 +- storage/connectors/pom.xml | 5 ++ 10 files changed, 158 insertions(+), 13 deletions(-) create mode 100644 docs/coverage/java/pom.xml diff --git a/.dockerignore b/.dockerignore index e9401f0cc9e..0e3b22687d0 100644 --- a/.dockerignore +++ b/.dockerignore @@ -1,2 +1,3 @@ docs +!docs/coverage charts diff --git a/.github/workflows/unit_tests.yml b/.github/workflows/unit_tests.yml index 2044396a84c..f71788a90bb 100644 --- a/.github/workflows/unit_tests.yml +++ b/.github/workflows/unit_tests.yml @@ -16,7 +16,11 @@ jobs: restore-keys: | ${{ runner.os }}-maven- - name: test java - run: make test-java + run: make test-java-with-coverage + - uses: actions/upload-artifact@v2 + with: + name: java-coverage-report + path: ${{ github.workspace }}/docs/coverage/java/target/site/jacoco-aggregate/ unit-test-python: runs-on: ubuntu-latest diff --git a/Makefile b/Makefile index 5cf5ae78ced..1f41ad09049 100644 --- a/Makefile +++ b/Makefile @@ -46,6 +46,9 @@ lint-java: test-java: mvn test +test-java-with-coverage: + mvn test jacoco:report-aggregate + build-java: mvn clean verify @@ -158,4 +161,4 @@ build-html: clean-html # Build Python SDK documentation $(MAKE) compile-protos-python cd $(ROOT_DIR)/sdk/python/docs && $(MAKE) html - cp -r $(ROOT_DIR)/sdk/python/docs/html/* $(ROOT_DIR)/dist/python \ No newline at end of file + cp -r $(ROOT_DIR)/sdk/python/docs/html/* $(ROOT_DIR)/dist/python diff --git a/core/pom.xml b/core/pom.xml index f4fb6c659c0..2f616d5ed41 100644 --- a/core/pom.xml +++ b/core/pom.xml @@ -32,6 +32,11 @@ + + org.jacoco + jacoco-maven-plugin + + org.springframework.boot spring-boot-maven-plugin @@ -195,7 +200,6 @@ org.mockito mockito-core - 2.23.0 test diff --git a/docs/coverage/java/pom.xml b/docs/coverage/java/pom.xml new file mode 100644 index 00000000000..05ffaf374e0 --- /dev/null +++ b/docs/coverage/java/pom.xml @@ -0,0 +1,111 @@ + + + + 4.0.0 + + + + + dev.feast + feast-parent + ${revision} + ../../.. + + + Feast Coverage Java + feast-coverage + + + true + + + + + dev.feast + feast-storage-api + ${project.version} + + + + dev.feast + feast-storage-connector-bigquery + ${project.version} + + + + dev.feast + feast-storage-connector-redis + ${project.version} + + + + dev.feast + feast-storage-connector-redis-cluster + ${project.version} + + + + dev.feast + feast-ingestion + ${project.version} + + + + dev.feast + feast-core + ${project.version} + + + + dev.feast + feast-serving + ${project.version} + + + + dev.feast + feast-sdk + ${project.version} + + + + + + + org.jacoco + jacoco-maven-plugin + + + report-aggregate + prepare-package + + report-aggregate + + + + + + + + diff --git a/ingestion/pom.xml b/ingestion/pom.xml index 64d5a41f86f..5a2fba8d658 100644 --- a/ingestion/pom.xml +++ b/ingestion/pom.xml @@ -91,6 +91,11 @@ + + + org.jacoco + jacoco-maven-plugin + diff --git a/pom.xml b/pom.xml index 7b7cd1d0fed..b62c9137921 100644 --- a/pom.xml +++ b/pom.xml @@ -35,6 +35,7 @@ core serving sdk/java + docs/coverage/java @@ -471,9 +472,9 @@ org.apache.maven.plugins maven-surefire-plugin - 2.22.1 + 3.0.0-M4 - -Xms2048m -Xmx2048m -Djdk.net.URLClassPath.disableClassPathURLCheck=true + @{argLine} -Xms2048m -Xmx2048m -Djdk.net.URLClassPath.disableClassPathURLCheck=true IntegrationTest @@ -614,6 +615,18 @@ false + + org.jacoco + jacoco-maven-plugin + 0.8.5 + + + + prepare-agent + + + + org.springframework.boot spring-boot-maven-plugin diff --git a/sdk/java/pom.xml b/sdk/java/pom.xml index e8a82a485fc..75d82edcc01 100644 --- a/sdk/java/pom.xml +++ b/sdk/java/pom.xml @@ -94,12 +94,8 @@ - maven-surefire-plugin - 2.22.2 - - - maven-failsafe-plugin - 2.22.2 + org.jacoco + jacoco-maven-plugin diff --git a/serving/pom.xml b/serving/pom.xml index d3d7ae212fd..3754777da07 100644 --- a/serving/pom.xml +++ b/serving/pom.xml @@ -40,6 +40,11 @@ + + org.jacoco + jacoco-maven-plugin + + org.springframework.boot spring-boot-maven-plugin @@ -248,11 +253,9 @@ test
    - org.mockito mockito-core - 2.23.0 test diff --git a/storage/connectors/pom.xml b/storage/connectors/pom.xml index b57fe98cd25..280b0d40bf1 100644 --- a/storage/connectors/pom.xml +++ b/storage/connectors/pom.xml @@ -32,6 +32,11 @@ + + + org.jacoco + jacoco-maven-plugin + From 09ff3dda724cb30ccc7c66042466a5f108742b99 Mon Sep 17 00:00:00 2001 From: Willem Pienaar <6728866+woop@users.noreply.github.com> Date: Sat, 16 May 2020 13:24:24 +0800 Subject: [PATCH 170/176] Change organization from gojek to feast-dev (#712) * Change organization from gojek to feast-dev * Fix indentation on prow config.yaml --- .github/pull_request_template.md | 6 +- .prow/config.yaml | 46 +- .prow/plugins.yaml | 6 +- CHANGELOG.md | 648 +++++++++--------- README.md | 12 +- .../job/dataflow/DataflowJobManagerTest.java | 2 +- datatypes/java/README.md | 2 +- docs/SUMMARY.md | 8 +- docs/administration/troubleshooting.md | 2 +- docs/contributing/contributing.md | 6 +- docs/contributing/development-guide.md | 4 +- docs/contributing/release-process.md | 6 +- docs/installation/docker-compose.md | 4 +- docs/installation/gke.md | 6 +- docs/introduction/getting-help.md | 4 +- docs/introduction/roadmap.md | 28 +- go.mod | 4 +- go.sum | 4 +- .../charts/feast/charts/feast-core/README.md | 6 +- .../feast/charts/feast-core/values.yaml | 6 +- .../feast/charts/feast-serving/README.md | 6 +- .../feast/charts/feast-serving/values.yaml | 6 +- infra/docker-compose/jupyter/startup.sh | 2 +- infra/docker/core/Dockerfile | 2 +- .../ingestion/transform/ReadFromSource.java | 2 +- pom.xml | 4 +- protos/feast/core/CoreService.proto | 2 +- protos/feast/core/FeatureSet.proto | 2 +- protos/feast/core/FeatureSetReference.proto | 2 +- protos/feast/core/IngestionJob.proto | 2 +- protos/feast/core/Runner.proto | 2 +- protos/feast/core/Source.proto | 2 +- protos/feast/core/Store.proto | 2 +- protos/feast/serving/ServingService.proto | 2 +- protos/feast/storage/Redis.proto | 2 +- protos/feast/types/FeatureRow.proto | 2 +- protos/feast/types/FeatureRowExtended.proto | 2 +- protos/feast/types/Field.proto | 2 +- protos/feast/types/Value.proto | 2 +- .../tensorflow_metadata/proto/v0/path.proto | 2 +- .../tensorflow_metadata/proto/v0/schema.proto | 2 +- sdk/go/README.md | 2 +- sdk/go/client.go | 2 +- sdk/go/go.mod | 2 +- sdk/go/protos/feast/core/FeatureSet.pb.go | 4 +- .../protos/feast/serving/ServingService.pb.go | 2 +- sdk/go/protos/feast/storage/Redis.pb.go | 2 +- sdk/go/request.go | 2 +- sdk/go/request_test.go | 4 +- sdk/go/response.go | 4 +- sdk/go/response_test.go | 4 +- sdk/go/types.go | 2 +- sdk/python/setup.py | 2 +- sdk/python/tests/test_client.py | 2 +- tests/e2e/basic-ingest-redis-serving.py | 4 +- 55 files changed, 450 insertions(+), 450 deletions(-) diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index b9c8cd6dff8..7a78437b5d4 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -1,8 +1,8 @@