|
-| apigateway:*
| arn:aws:apigateway:*::/apis/*/routes/*/routeresponses
arn:aws:apigateway:*::/apis/*/routes/*/routeresponses/*
arn:aws:apigateway:*::/apis/*/routes/*
arn:aws:apigateway:*::/apis/*/routes
arn:aws:apigateway:*::/apis/*/integrations
arn:aws:apigateway:*::/apis/*/stages/*/routesettings/*
arn:aws:apigateway:*::/apis/*
arn:aws:apigateway:*::/apis
|
-
-The following inline policy can be used to grant Feast the necessary permissions:
-
-```javascript
-{
- "Statement": [
- {
- Action = [
- "lambda:CreateFunction",
- "lambda:GetFunction",
- "lambda:DeleteFunction",
- "lambda:AddPermission",
- "lambda:UpdateFunctionConfiguration",
- ]
- Effect = "Allow"
- Resource = "arn:aws:lambda:::function:feast-*"
- },
- {
- Action = [
- "ecr:CreateRepository",
- "ecr:DescribeRepositories",
- "ecr:DeleteRepository",
- "ecr:PutImage",
- "ecr:DescribeImages",
- "ecr:BatchDeleteImage",
- "ecr:CompleteLayerUpload",
- "ecr:UploadLayerPart",
- "ecr:InitiateLayerUpload",
- "ecr:BatchCheckLayerAvailability",
- "ecr:GetDownloadUrlForLayer",
- "ecr:GetRepositoryPolicy",
- "ecr:SetRepositoryPolicy",
- "ecr:GetAuthorizationToken"
- ]
- Effect = "Allow"
- Resource = "*"
- },
- {
- Action = "iam:PassRole"
- Effect = "Allow"
- Resource = "arn:aws:iam:::role/"
- },
- {
- Effect = "Allow"
- Action = "apigateway:*"
- Resource = [
- "arn:aws:apigateway:*::/apis/*/routes/*/routeresponses",
- "arn:aws:apigateway:*::/apis/*/routes/*/routeresponses/*",
- "arn:aws:apigateway:*::/apis/*/routes/*",
- "arn:aws:apigateway:*::/apis/*/routes",
- "arn:aws:apigateway:*::/apis/*/integrations",
- "arn:aws:apigateway:*::/apis/*/stages/*/routesettings/*",
- "arn:aws:apigateway:*::/apis/*",
- "arn:aws:apigateway:*::/apis",
- ]
- },
- ],
- "Version": "2012-10-17"
-}
-```
-
-## Example
-
-After `feature_store.yaml` has been modified as described in the previous section, it can be deployed as follows:
-
-```bash
-$ feast apply
-10/07/2021 03:57:26 PM INFO:Pulling remote image feastdev/feature-server-python-aws:aws:
-10/07/2021 03:57:28 PM INFO:Creating remote ECR repository feast-python-server-key_shark-0_13_1_dev23_gb3c08320:
-10/07/2021 03:57:29 PM INFO:Pushing local image to remote 402087665549.dkr.ecr.us-west-2.amazonaws.com/feast-python-server-key_shark-0_13_1_dev23_gb3c08320:0_13_1_dev23_gb3c08320:
-10/07/2021 03:58:44 PM INFO:Deploying feature server...
-10/07/2021 03:58:45 PM INFO: Creating AWS Lambda...
-10/07/2021 03:58:46 PM INFO: Creating AWS API Gateway...
-Registered entity driver_id
-Registered feature view driver_hourly_stats
-Deploying infrastructure for driver_hourly_stats
-
-$ feast endpoint
-10/07/2021 03:59:01 PM INFO:Feature server endpoint: https://hkosgmz4m2.execute-api.us-west-2.amazonaws.com
-
-$ feast materialize-incremental $(date +%Y-%m-%d)
-Materializing 1 feature views to 2021-10-06 17:00:00-07:00 into the dynamodb online store.
-
-driver_hourly_stats from 2020-10-08 23:01:34-07:00 to 2021-10-06 17:00:00-07:00:
-100%|█████████████████████████████████████████████████████████████████| 5/5 [00:00<00:00, 16.89it/s]
-```
-
-After the feature server starts, we can execute cURL commands against it:
-
-```bash
-$ curl -X POST \
- "https://hkosgmz4m2.execute-api.us-west-2.amazonaws.com/get-online-features" \
- -H "Content-type: application/json" \
- -H "Accept: application/json" \
- -d '{
- "features": [
- "driver_hourly_stats:conv_rate",
- "driver_hourly_stats:acc_rate",
- "driver_hourly_stats:avg_daily_trips"
- ],
- "entities": {
- "driver_id": [1001, 1002, 1003]
- },
- "full_feature_names": true
- }' | jq
- % Total % Received % Xferd Average Speed Time Time Time Current
- Dload Upload Total Spent Left Speed
-100 1346 100 1055 100 291 3436 947 --:--:-- --:--:-- --:--:-- 4370
-{
- "field_values": [
- {
- "fields": {
- "driver_id": 1001,
- "driver_hourly_stats__conv_rate": 0.025330161675810814,
- "driver_hourly_stats__avg_daily_trips": 785,
- "driver_hourly_stats__acc_rate": 0.835975170135498
- },
- "statuses": {
- "driver_hourly_stats__avg_daily_trips": "PRESENT",
- "driver_id": "PRESENT",
- "driver_hourly_stats__conv_rate": "PRESENT",
- "driver_hourly_stats__acc_rate": "PRESENT"
- }
- },
- {
- "fields": {
- "driver_hourly_stats__conv_rate": 0.7595187425613403,
- "driver_hourly_stats__acc_rate": 0.1740121990442276,
- "driver_id": 1002,
- "driver_hourly_stats__avg_daily_trips": 875
- },
- "statuses": {
- "driver_hourly_stats__acc_rate": "PRESENT",
- "driver_id": "PRESENT",
- "driver_hourly_stats__avg_daily_trips": "PRESENT",
- "driver_hourly_stats__conv_rate": "PRESENT"
- }
- },
- {
- "fields": {
- "driver_hourly_stats__acc_rate": 0.7785481214523315,
- "driver_hourly_stats__conv_rate": 0.33832859992980957,
- "driver_hourly_stats__avg_daily_trips": 846,
- "driver_id": 1003
- },
- "statuses": {
- "driver_id": "PRESENT",
- "driver_hourly_stats__conv_rate": "PRESENT",
- "driver_hourly_stats__acc_rate": "PRESENT",
- "driver_hourly_stats__avg_daily_trips": "PRESENT"
- }
- }
- ]
-}
-```
diff --git a/docs/reference/feature-servers/python-feature-server.md b/docs/reference/feature-servers/python-feature-server.md
index c189f97ae03..0d8a0aef756 100644
--- a/docs/reference/feature-servers/python-feature-server.md
+++ b/docs/reference/feature-servers/python-feature-server.md
@@ -12,8 +12,6 @@ There is a CLI command that starts the server: `feast serve`. By default, Feast
One can deploy a feature server by building a docker image that bundles in the project's `feature_store.yaml`. See this [helm chart](https://github.com/feast-dev/feast/blob/master/infra/charts/feast-feature-server) for an example on how to run Feast on Kubernetes.
-A [remote feature server](alpha-aws-lambda-feature-server.md) on AWS Lambda is also available.
-
## Example
### Initializing a feature server
diff --git a/docs/reference/offline-stores/file.md b/docs/reference/offline-stores/dask.md
similarity index 87%
rename from docs/reference/offline-stores/file.md
rename to docs/reference/offline-stores/dask.md
index 4b76d9af904..d8698ba544b 100644
--- a/docs/reference/offline-stores/file.md
+++ b/docs/reference/offline-stores/dask.md
@@ -1,9 +1,8 @@
-# File offline store
+# Dask offline store
## Description
-The file offline store provides support for reading [FileSources](../data-sources/file.md).
-It uses Dask as the compute engine.
+The Dask offline store provides support for reading [FileSources](../data-sources/file.md).
{% hint style="warning" %}
All data is downloaded and joined using Python and therefore may not scale to production workloads.
@@ -17,18 +16,18 @@ project: my_feature_repo
registry: data/registry.db
provider: local
offline_store:
- type: file
+ type: dask
```
{% endcode %}
-The full set of configuration options is available in [FileOfflineStoreConfig](https://rtd.feast.dev/en/latest/#feast.infra.offline_stores.file.FileOfflineStoreConfig).
+The full set of configuration options is available in [DaskOfflineStoreConfig](https://rtd.feast.dev/en/latest/#feast.infra.offline_stores.dask.DaskOfflineStoreConfig).
## Functionality Matrix
The set of functionality supported by offline stores is described in detail [here](overview.md#functionality).
-Below is a matrix indicating which functionality is supported by the file offline store.
+Below is a matrix indicating which functionality is supported by the dask offline store.
-| | File |
+| | Dask |
| :-------------------------------- | :-- |
| `get_historical_features` (point-in-time correct join) | yes |
| `pull_latest_from_table_or_query` (retrieve latest feature values) | yes |
@@ -36,9 +35,9 @@ Below is a matrix indicating which functionality is supported by the file offlin
| `offline_write_batch` (persist dataframes to offline store) | yes |
| `write_logged_features` (persist logged features to offline store) | yes |
-Below is a matrix indicating which functionality is supported by `FileRetrievalJob`.
+Below is a matrix indicating which functionality is supported by `DaskRetrievalJob`.
-| | File |
+| | Dask |
| --------------------------------- | --- |
| export to dataframe | yes |
| export to arrow table | yes |
diff --git a/docs/reference/offline-stores/overview.md b/docs/reference/offline-stores/overview.md
index 4d7681e38c8..182eac65864 100644
--- a/docs/reference/offline-stores/overview.md
+++ b/docs/reference/offline-stores/overview.md
@@ -25,13 +25,13 @@ The first three of these methods all return a `RetrievalJob` specific to an offl
## Functionality Matrix
-There are currently four core offline store implementations: `FileOfflineStore`, `BigQueryOfflineStore`, `SnowflakeOfflineStore`, and `RedshiftOfflineStore`.
+There are currently four core offline store implementations: `DaskOfflineStore`, `BigQueryOfflineStore`, `SnowflakeOfflineStore`, and `RedshiftOfflineStore`.
There are several additional implementations contributed by the Feast community (`PostgreSQLOfflineStore`, `SparkOfflineStore`, and `TrinoOfflineStore`), which are not guaranteed to be stable or to match the functionality of the core implementations.
Details for each specific offline store, such as how to configure it in a `feature_store.yaml`, can be found [here](README.md).
Below is a matrix indicating which offline stores support which methods.
-| | File | BigQuery | Snowflake | Redshift | Postgres | Spark | Trino |
+| | Dask | BigQuery | Snowflake | Redshift | Postgres | Spark | Trino |
| :-------------------------------- | :-- | :-- | :-- | :-- | :-- | :-- | :-- |
| `get_historical_features` | yes | yes | yes | yes | yes | yes | yes |
| `pull_latest_from_table_or_query` | yes | yes | yes | yes | yes | yes | yes |
@@ -42,7 +42,7 @@ Below is a matrix indicating which offline stores support which methods.
Below is a matrix indicating which `RetrievalJob`s support what functionality.
-| | File | BigQuery | Snowflake | Redshift | Postgres | Spark | Trino | DuckDB |
+| | Dask | BigQuery | Snowflake | Redshift | Postgres | Spark | Trino | DuckDB |
| --------------------------------- | --- | --- | --- | --- | --- | --- | --- | --- |
| export to dataframe | yes | yes | yes | yes | yes | yes | yes | yes |
| export to arrow table | yes | yes | yes | yes | yes | yes | yes | yes |
diff --git a/docs/reference/online-stores/README.md b/docs/reference/online-stores/README.md
index b5f4eb8de89..0acf6701f92 100644
--- a/docs/reference/online-stores/README.md
+++ b/docs/reference/online-stores/README.md
@@ -64,4 +64,7 @@ Please see [Online Store](../../getting-started/architecture-and-components/onli
{% content-ref url="remote.md" %}
[remote.md](remote.md)
+
+{% content-ref url="singlestore.md" %}
+[singlestore.md](singlestore.md)
{% endcontent-ref %}
diff --git a/docs/reference/online-stores/singlestore.md b/docs/reference/online-stores/singlestore.md
new file mode 100644
index 00000000000..1777787f227
--- /dev/null
+++ b/docs/reference/online-stores/singlestore.md
@@ -0,0 +1,51 @@
+# SingleStore online store (contrib)
+
+## Description
+
+The SingleStore online store provides support for materializing feature values into a SingleStore database for serving online features.
+
+## Getting started
+In order to use this online store, you'll need to run `pip install 'feast[singlestore]'`. You can get started by then running `feast init` and then setting the `feature_store.yaml` as described below.
+
+## Example
+
+{% code title="feature_store.yaml" %}
+```yaml
+project: my_feature_repo
+registry: data/registry.db
+provider: local
+online_store:
+ type: singlestore
+ host: DB_HOST
+ port: DB_PORT
+ database: DB_NAME
+ user: DB_USERNAME
+ password: DB_PASSWORD
+```
+{% endcode %}
+
+## Functionality Matrix
+
+The set of functionality supported by online stores is described in detail [here](overview.md#functionality).
+Below is a matrix indicating which functionality is supported by the SingleStore online store.
+
+| | SingleStore |
+| :-------------------------------------------------------- | :----------- |
+| write feature values to the online store | yes |
+| read feature values from the online store | yes |
+| update infrastructure (e.g. tables) in the online store | yes |
+| teardown infrastructure (e.g. tables) in the online store | yes |
+| generate a plan of infrastructure changes | no |
+| support for on-demand transforms | yes |
+| readable by Python SDK | yes |
+| readable by Java | no |
+| readable by Go | no |
+| support for entityless feature views | yes |
+| support for concurrent writing to the same key | no |
+| support for ttl (time to live) at retrieval | no |
+| support for deleting expired data | no |
+| collocated by feature view | yes |
+| collocated by feature service | no |
+| collocated by entity key | no |
+
+To compare this set of functionality against other online stores, please see the full [functionality matrix](overview.md#functionality-matrix).
diff --git a/docs/tutorials/using-scalable-registry.md b/docs/tutorials/using-scalable-registry.md
index 30b8e01ed51..25746f60e23 100644
--- a/docs/tutorials/using-scalable-registry.md
+++ b/docs/tutorials/using-scalable-registry.md
@@ -49,7 +49,7 @@ When this happens, your database is likely using what is referred to as an
in `SQLAlchemy` terminology. See your database's documentation for examples on
how to set its scheme in the Database URL.
-`Psycopg2`, which is the database library leveraged by the online and offline
+`Psycopg`, which is the database library leveraged by the online and offline
stores, is not impacted by the need to speak a particular dialect, and so the
following only applies to the registry.
diff --git a/go/internal/feast/server/http_server.go b/go/internal/feast/server/http_server.go
index 75cdbe9929a..7ebab429e7e 100644
--- a/go/internal/feast/server/http_server.go
+++ b/go/internal/feast/server/http_server.go
@@ -4,14 +4,16 @@ import (
"context"
"encoding/json"
"fmt"
+ "net/http"
+ "time"
+
"github.com/feast-dev/feast/go/internal/feast"
"github.com/feast-dev/feast/go/internal/feast/model"
+ "github.com/feast-dev/feast/go/internal/feast/onlineserving"
"github.com/feast-dev/feast/go/internal/feast/server/logging"
"github.com/feast-dev/feast/go/protos/feast/serving"
prototypes "github.com/feast-dev/feast/go/protos/feast/types"
"github.com/feast-dev/feast/go/types"
- "net/http"
- "time"
)
type httpServer struct {
@@ -210,6 +212,8 @@ func (s *httpServer) getOnlineFeatures(w http.ResponseWriter, r *http.Request) {
"results": results,
}
+ w.Header().Set("Content-Type", "application/json")
+
err = json.NewEncoder(w).Encode(response)
if err != nil {
@@ -217,8 +221,6 @@ func (s *httpServer) getOnlineFeatures(w http.ResponseWriter, r *http.Request) {
return
}
- w.Header().Set("Content-Type", "application/json")
-
if featureService != nil && featureService.LoggingConfig != nil && s.loggingService != nil {
logger, err := s.loggingService.GetOrCreateLogger(featureService)
if err != nil {
@@ -250,11 +252,19 @@ func (s *httpServer) getOnlineFeatures(w http.ResponseWriter, r *http.Request) {
return
}
}
+ go releaseCGOMemory(featureVectors)
+}
+
+func releaseCGOMemory(featureVectors []*onlineserving.FeatureVector) {
+ for _, vector := range featureVectors {
+ vector.Values.Release()
+ }
}
func (s *httpServer) Serve(host string, port int) error {
s.server = &http.Server{Addr: fmt.Sprintf("%s:%d", host, port), Handler: nil}
http.HandleFunc("/get-online-features", s.getOnlineFeatures)
+ http.HandleFunc("/health", healthCheckHandler)
err := s.server.ListenAndServe()
// Don't return the error if it's caused by graceful shutdown using Stop()
if err == http.ErrServerClosed {
@@ -262,6 +272,12 @@ func (s *httpServer) Serve(host string, port int) error {
}
return err
}
+
+func healthCheckHandler(w http.ResponseWriter, r *http.Request) {
+ w.WriteHeader(http.StatusOK)
+ fmt.Fprintf(w, "Healthy")
+}
+
func (s *httpServer) Stop() error {
if s.server != nil {
return s.server.Shutdown(context.Background())
diff --git a/go/internal/feast/transformation/transformation.go b/go/internal/feast/transformation/transformation.go
index 1cf1dd3311b..7e63aec2243 100644
--- a/go/internal/feast/transformation/transformation.go
+++ b/go/internal/feast/transformation/transformation.go
@@ -46,6 +46,7 @@ func AugmentResponseWithOnDemandTransforms(
for name, values := range requestData {
requestContextArrow[name], err = types.ProtoValuesToArrowArray(values.Val, arrowMemory, numRows)
if err != nil {
+ ReleaseArrowContext(requestContextArrow)
return nil, err
}
}
@@ -53,6 +54,7 @@ func AugmentResponseWithOnDemandTransforms(
for name, values := range entityRows {
requestContextArrow[name], err = types.ProtoValuesToArrowArray(values.Val, arrowMemory, numRows)
if err != nil {
+ ReleaseArrowContext(requestContextArrow)
return nil, err
}
}
@@ -71,14 +73,24 @@ func AugmentResponseWithOnDemandTransforms(
fullFeatureNames,
)
if err != nil {
+ ReleaseArrowContext(requestContextArrow)
return nil, err
}
result = append(result, onDemandFeatures...)
+
+ ReleaseArrowContext(requestContextArrow)
}
return result, nil
}
+func ReleaseArrowContext(requestContextArrow map[string]arrow.Array) {
+ // Release memory used by requestContextArrow
+ for _, arrowArray := range requestContextArrow {
+ arrowArray.Release()
+ }
+}
+
func CallTransformations(
featureView *model.OnDemandFeatureView,
retrievedFeatures map[string]arrow.Array,
diff --git a/infra/charts/feast-feature-server/Chart.yaml b/infra/charts/feast-feature-server/Chart.yaml
index aa39b158dd7..af47692f16e 100644
--- a/infra/charts/feast-feature-server/Chart.yaml
+++ b/infra/charts/feast-feature-server/Chart.yaml
@@ -2,7 +2,7 @@ apiVersion: v2
name: feast-feature-server
description: Feast Feature Server in Go or Python
type: application
-version: 0.39.0
+version: 0.40.0
keywords:
- machine learning
- big data
diff --git a/infra/charts/feast-feature-server/README.md b/infra/charts/feast-feature-server/README.md
index 121b0cc0cd9..63ff7cf61b2 100644
--- a/infra/charts/feast-feature-server/README.md
+++ b/infra/charts/feast-feature-server/README.md
@@ -1,6 +1,6 @@
# Feast Python / Go Feature Server Helm Charts
-Current chart version is `0.39.0`
+Current chart version is `0.40.0`
## Installation
@@ -40,7 +40,7 @@ See [here](https://github.com/feast-dev/feast/tree/master/examples/python-helm-d
| fullnameOverride | string | `""` | |
| image.pullPolicy | string | `"IfNotPresent"` | |
| image.repository | string | `"feastdev/feature-server"` | Docker image for Feature Server repository |
-| image.tag | string | `"0.39.0"` | The Docker image tag (can be overwritten if custom feature server deps are needed for on demand transforms) |
+| image.tag | string | `"0.40.0"` | The Docker image tag (can be overwritten if custom feature server deps are needed for on demand transforms) |
| imagePullSecrets | list | `[]` | |
| livenessProbe.initialDelaySeconds | int | `30` | |
| livenessProbe.periodSeconds | int | `30` | |
diff --git a/infra/charts/feast-feature-server/values.yaml b/infra/charts/feast-feature-server/values.yaml
index 33430749d8d..0c46bfff854 100644
--- a/infra/charts/feast-feature-server/values.yaml
+++ b/infra/charts/feast-feature-server/values.yaml
@@ -9,7 +9,7 @@ image:
repository: feastdev/feature-server
pullPolicy: IfNotPresent
# image.tag -- The Docker image tag (can be overwritten if custom feature server deps are needed for on demand transforms)
- tag: 0.39.0
+ tag: 0.40.0
imagePullSecrets: []
nameOverride: ""
diff --git a/infra/charts/feast/Chart.yaml b/infra/charts/feast/Chart.yaml
index c724a748a6b..1f030ca6172 100644
--- a/infra/charts/feast/Chart.yaml
+++ b/infra/charts/feast/Chart.yaml
@@ -1,7 +1,7 @@
apiVersion: v1
description: Feature store for machine learning
name: feast
-version: 0.39.0
+version: 0.40.0
keywords:
- machine learning
- big data
diff --git a/infra/charts/feast/README.md b/infra/charts/feast/README.md
index d611b69d847..bf8e4d7b8d2 100644
--- a/infra/charts/feast/README.md
+++ b/infra/charts/feast/README.md
@@ -8,7 +8,7 @@ This repo contains Helm charts for Feast Java components that are being installe
## Chart: Feast
-Feature store for machine learning Current chart version is `0.39.0`
+Feature store for machine learning Current chart version is `0.40.0`
## Installation
@@ -65,8 +65,8 @@ See [here](https://github.com/feast-dev/feast/tree/master/examples/java-demo) fo
| Repository | Name | Version |
|------------|------|---------|
| https://charts.helm.sh/stable | redis | 10.5.6 |
-| https://feast-helm-charts.storage.googleapis.com | feature-server(feature-server) | 0.39.0 |
-| https://feast-helm-charts.storage.googleapis.com | transformation-service(transformation-service) | 0.39.0 |
+| https://feast-helm-charts.storage.googleapis.com | feature-server(feature-server) | 0.40.0 |
+| https://feast-helm-charts.storage.googleapis.com | transformation-service(transformation-service) | 0.40.0 |
## Values
diff --git a/infra/charts/feast/charts/feature-server/Chart.yaml b/infra/charts/feast/charts/feature-server/Chart.yaml
index 5616d463015..f91185be845 100644
--- a/infra/charts/feast/charts/feature-server/Chart.yaml
+++ b/infra/charts/feast/charts/feature-server/Chart.yaml
@@ -1,8 +1,8 @@
apiVersion: v1
description: "Feast Feature Server: Online feature serving service for Feast"
name: feature-server
-version: 0.39.0
-appVersion: v0.39.0
+version: 0.40.0
+appVersion: v0.40.0
keywords:
- machine learning
- big data
diff --git a/infra/charts/feast/charts/feature-server/README.md b/infra/charts/feast/charts/feature-server/README.md
index f4a8ea8cda8..c75fc421c62 100644
--- a/infra/charts/feast/charts/feature-server/README.md
+++ b/infra/charts/feast/charts/feature-server/README.md
@@ -1,6 +1,6 @@
# feature-server
- 
+ 
Feast Feature Server: Online feature serving service for Feast
@@ -17,7 +17,7 @@ Feast Feature Server: Online feature serving service for Feast
| envOverrides | object | `{}` | Extra environment variables to set |
| image.pullPolicy | string | `"IfNotPresent"` | Image pull policy |
| image.repository | string | `"feastdev/feature-server-java"` | Docker image for Feature Server repository |
-| image.tag | string | `"0.39.0"` | Image tag |
+| image.tag | string | `"0.40.0"` | Image tag |
| ingress.grpc.annotations | object | `{}` | Extra annotations for the ingress |
| ingress.grpc.auth.enabled | bool | `false` | Flag to enable auth |
| ingress.grpc.class | string | `"nginx"` | Which ingress controller to use |
@@ -64,4 +64,4 @@ Feast Feature Server: Online feature serving service for Feast
| transformationService.port | int | `6566` | |
----------------------------------------------
-Autogenerated from chart metadata using [helm-docs v1.13.1](https://github.com/norwoodj/helm-docs/releases/v1.13.1)
+Autogenerated from chart metadata using [helm-docs v1.14.2](https://github.com/norwoodj/helm-docs/releases/v1.14.2)
diff --git a/infra/charts/feast/charts/feature-server/values.yaml b/infra/charts/feast/charts/feature-server/values.yaml
index cd60eaf93f8..d9c964bbca6 100644
--- a/infra/charts/feast/charts/feature-server/values.yaml
+++ b/infra/charts/feast/charts/feature-server/values.yaml
@@ -5,7 +5,7 @@ image:
# image.repository -- Docker image for Feature Server repository
repository: feastdev/feature-server-java
# image.tag -- Image tag
- tag: 0.39.0
+ tag: 0.40.0
# image.pullPolicy -- Image pull policy
pullPolicy: IfNotPresent
diff --git a/infra/charts/feast/charts/transformation-service/Chart.yaml b/infra/charts/feast/charts/transformation-service/Chart.yaml
index 2e3211697f3..7e336e7a3e8 100644
--- a/infra/charts/feast/charts/transformation-service/Chart.yaml
+++ b/infra/charts/feast/charts/transformation-service/Chart.yaml
@@ -1,8 +1,8 @@
apiVersion: v1
description: "Transformation service: to compute on-demand features"
name: transformation-service
-version: 0.39.0
-appVersion: v0.39.0
+version: 0.40.0
+appVersion: v0.40.0
keywords:
- machine learning
- big data
diff --git a/infra/charts/feast/charts/transformation-service/README.md b/infra/charts/feast/charts/transformation-service/README.md
index dec106617e5..f90d5bda185 100644
--- a/infra/charts/feast/charts/transformation-service/README.md
+++ b/infra/charts/feast/charts/transformation-service/README.md
@@ -1,6 +1,6 @@
# transformation-service
- 
+ 
Transformation service: to compute on-demand features
@@ -13,7 +13,7 @@ Transformation service: to compute on-demand features
| envOverrides | object | `{}` | Extra environment variables to set |
| image.pullPolicy | string | `"IfNotPresent"` | Image pull policy |
| image.repository | string | `"feastdev/feature-transformation-server"` | Docker image for Transformation Server repository |
-| image.tag | string | `"0.39.0"` | Image tag |
+| image.tag | string | `"0.40.0"` | Image tag |
| nodeSelector | object | `{}` | Node labels for pod assignment |
| podLabels | object | `{}` | Labels to be added to Feast Serving pods |
| replicaCount | int | `1` | Number of pods that will be created |
@@ -25,4 +25,4 @@ Transformation service: to compute on-demand features
| service.type | string | `"ClusterIP"` | Kubernetes service type |
----------------------------------------------
-Autogenerated from chart metadata using [helm-docs v1.13.1](https://github.com/norwoodj/helm-docs/releases/v1.13.1)
+Autogenerated from chart metadata using [helm-docs v1.14.2](https://github.com/norwoodj/helm-docs/releases/v1.14.2)
diff --git a/infra/charts/feast/charts/transformation-service/values.yaml b/infra/charts/feast/charts/transformation-service/values.yaml
index a6935a9993c..aee47048e83 100644
--- a/infra/charts/feast/charts/transformation-service/values.yaml
+++ b/infra/charts/feast/charts/transformation-service/values.yaml
@@ -5,7 +5,7 @@ image:
# image.repository -- Docker image for Transformation Server repository
repository: feastdev/feature-transformation-server
# image.tag -- Image tag
- tag: 0.39.0
+ tag: 0.40.0
# image.pullPolicy -- Image pull policy
pullPolicy: IfNotPresent
diff --git a/infra/charts/feast/requirements.yaml b/infra/charts/feast/requirements.yaml
index 1f579a5f3c3..7b1277fc69a 100644
--- a/infra/charts/feast/requirements.yaml
+++ b/infra/charts/feast/requirements.yaml
@@ -1,12 +1,12 @@
dependencies:
- name: feature-server
alias: feature-server
- version: 0.39.0
+ version: 0.40.0
condition: feature-server.enabled
repository: https://feast-helm-charts.storage.googleapis.com
- name: transformation-service
alias: transformation-service
- version: 0.39.0
+ version: 0.40.0
condition: transformation-service.enabled
repository: https://feast-helm-charts.storage.googleapis.com
- name: redis
diff --git a/infra/feast-operator/Makefile b/infra/feast-operator/Makefile
index 1388778f9fe..f4464377513 100644
--- a/infra/feast-operator/Makefile
+++ b/infra/feast-operator/Makefile
@@ -3,7 +3,7 @@
# To re-generate a bundle for another specific version without changing the standard setup, you can:
# - use the VERSION as arg of the bundle target (e.g make bundle VERSION=0.0.2)
# - use environment variables to overwrite this value (e.g export VERSION=0.0.2)
-VERSION ?= 0.37.0
+VERSION ?= 0.39.0
# CHANNELS define the bundle channels used in the bundle.
# Add a new line here if you would like to change its default config. (E.g CHANNELS = "candidate,fast,stable")
diff --git a/infra/feast-operator/config/manager/kustomization.yaml b/infra/feast-operator/config/manager/kustomization.yaml
index 226b87118d2..1c111ac7ada 100644
--- a/infra/feast-operator/config/manager/kustomization.yaml
+++ b/infra/feast-operator/config/manager/kustomization.yaml
@@ -5,4 +5,4 @@ kind: Kustomization
images:
- name: controller
newName: feastdev/feast-operator
- newTag: 0.37.0
+ newTag: 0.39.0
diff --git a/infra/scripts/pixi/pixi.lock b/infra/scripts/pixi/pixi.lock
index f1ce2d26585..1ca8742026c 100644
--- a/infra/scripts/pixi/pixi.lock
+++ b/infra/scripts/pixi/pixi.lock
@@ -11,6 +11,9 @@ environments:
- conda: https://conda.anaconda.org/conda-forge/linux-64/libgomp-13.2.0-h807b86a_5.conda
- conda: https://conda.anaconda.org/conda-forge/linux-64/libstdcxx-ng-13.2.0-h95c4c6d_6.conda
- conda: https://conda.anaconda.org/conda-forge/linux-64/uv-0.1.39-h0ea3d13_0.conda
+ osx-64:
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/libcxx-17.0.6-heb59cac_3.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/uv-0.1.45-h4e38c46_0.conda
osx-arm64:
- conda: https://conda.anaconda.org/conda-forge/osx-arm64/libcxx-17.0.6-h5f092b4_0.conda
- conda: https://conda.anaconda.org/conda-forge/osx-arm64/uv-0.1.45-hc069d6b_0.conda
@@ -41,6 +44,21 @@ environments:
- conda: https://conda.anaconda.org/conda-forge/noarch/tzdata-2024a-h0c530f3_0.conda
- conda: https://conda.anaconda.org/conda-forge/linux-64/uv-0.1.39-h0ea3d13_0.conda
- conda: https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.6-h166bdaf_0.tar.bz2
+ osx-64:
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/bzip2-1.0.8-h10d778d_5.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/ca-certificates-2024.7.4-h8857fd0_0.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/libcxx-17.0.6-heb59cac_3.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/libffi-3.4.2-h0d85af4_5.tar.bz2
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/libsqlite-3.46.0-h1b8f9f3_0.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/libzlib-1.3.1-h87427d6_1.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/ncurses-6.5-h5846eda_0.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/openssl-3.3.1-h87427d6_1.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/python-3.10.14-h00d2728_0_cpython.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/readline-8.2-h9e318b2_1.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/tk-8.6.13-h1abcd95_1.conda
+ - conda: https://conda.anaconda.org/conda-forge/noarch/tzdata-2024a-h0c530f3_0.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/uv-0.1.45-h4e38c46_0.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/xz-5.2.6-h775f41a_0.tar.bz2
osx-arm64:
- conda: https://conda.anaconda.org/conda-forge/osx-arm64/bzip2-1.0.8-h93a5062_5.conda
- conda: https://conda.anaconda.org/conda-forge/osx-arm64/ca-certificates-2024.2.2-hf0a4a13_0.conda
@@ -84,6 +102,22 @@ environments:
- conda: https://conda.anaconda.org/conda-forge/noarch/tzdata-2024a-h0c530f3_0.conda
- conda: https://conda.anaconda.org/conda-forge/linux-64/uv-0.1.39-h0ea3d13_0.conda
- conda: https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.6-h166bdaf_0.tar.bz2
+ osx-64:
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/bzip2-1.0.8-h10d778d_5.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/ca-certificates-2024.7.4-h8857fd0_0.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/libcxx-17.0.6-heb59cac_3.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/libexpat-2.6.2-h73e2aa4_0.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/libffi-3.4.2-h0d85af4_5.tar.bz2
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/libsqlite-3.46.0-h1b8f9f3_0.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/libzlib-1.3.1-h87427d6_1.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/ncurses-6.5-h5846eda_0.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/openssl-3.3.1-h87427d6_1.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/python-3.11.9-h657bba9_0_cpython.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/readline-8.2-h9e318b2_1.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/tk-8.6.13-h1abcd95_1.conda
+ - conda: https://conda.anaconda.org/conda-forge/noarch/tzdata-2024a-h0c530f3_0.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/uv-0.1.45-h4e38c46_0.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/xz-5.2.6-h775f41a_0.tar.bz2
osx-arm64:
- conda: https://conda.anaconda.org/conda-forge/osx-arm64/bzip2-1.0.8-h93a5062_5.conda
- conda: https://conda.anaconda.org/conda-forge/osx-arm64/ca-certificates-2024.2.2-hf0a4a13_0.conda
@@ -127,6 +161,21 @@ environments:
- conda: https://conda.anaconda.org/conda-forge/noarch/tzdata-2024a-h0c530f3_0.conda
- conda: https://conda.anaconda.org/conda-forge/linux-64/uv-0.1.39-h0ea3d13_0.conda
- conda: https://conda.anaconda.org/conda-forge/linux-64/xz-5.2.6-h166bdaf_0.tar.bz2
+ osx-64:
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/bzip2-1.0.8-h10d778d_5.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/ca-certificates-2024.7.4-h8857fd0_0.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/libcxx-17.0.6-heb59cac_3.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/libffi-3.4.2-h0d85af4_5.tar.bz2
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/libsqlite-3.46.0-h1b8f9f3_0.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/libzlib-1.3.1-h87427d6_1.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/ncurses-6.5-h5846eda_0.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/openssl-3.3.1-h87427d6_1.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/python-3.9.19-h7a9c478_0_cpython.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/readline-8.2-h9e318b2_1.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/tk-8.6.13-h1abcd95_1.conda
+ - conda: https://conda.anaconda.org/conda-forge/noarch/tzdata-2024a-h0c530f3_0.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/uv-0.1.45-h4e38c46_0.conda
+ - conda: https://conda.anaconda.org/conda-forge/osx-64/xz-5.2.6-h775f41a_0.tar.bz2
osx-arm64:
- conda: https://conda.anaconda.org/conda-forge/osx-arm64/bzip2-1.0.8-h93a5062_5.conda
- conda: https://conda.anaconda.org/conda-forge/osx-arm64/ca-certificates-2024.2.2-hf0a4a13_0.conda
@@ -172,6 +221,19 @@ packages:
license_family: BSD
size: 23621
timestamp: 1650670423406
+- kind: conda
+ name: bzip2
+ version: 1.0.8
+ build: h10d778d_5
+ build_number: 5
+ subdir: osx-64
+ url: https://conda.anaconda.org/conda-forge/osx-64/bzip2-1.0.8-h10d778d_5.conda
+ sha256: 61fb2b488928a54d9472113e1280b468a309561caa54f33825a3593da390b242
+ md5: 6097a6ca9ada32699b5fc4312dd6ef18
+ license: bzip2-1.0.6
+ license_family: BSD
+ size: 127885
+ timestamp: 1699280178474
- kind: conda
name: bzip2
version: 1.0.8
@@ -222,6 +284,17 @@ packages:
license: ISC
size: 155725
timestamp: 1706844034242
+- kind: conda
+ name: ca-certificates
+ version: 2024.7.4
+ build: h8857fd0_0
+ subdir: osx-64
+ url: https://conda.anaconda.org/conda-forge/osx-64/ca-certificates-2024.7.4-h8857fd0_0.conda
+ sha256: d16f46c489cb3192305c7d25b795333c5fc17bb0986de20598ed519f8c9cc9e4
+ md5: 7df874a4b05b2d2b82826190170eaa0f
+ license: ISC
+ size: 154473
+ timestamp: 1720077510541
- kind: conda
name: ld_impl_linux-64
version: '2.40'
@@ -264,6 +337,21 @@ packages:
license_family: Apache
size: 1248885
timestamp: 1715020154867
+- kind: conda
+ name: libcxx
+ version: 17.0.6
+ build: heb59cac_3
+ build_number: 3
+ subdir: osx-64
+ url: https://conda.anaconda.org/conda-forge/osx-64/libcxx-17.0.6-heb59cac_3.conda
+ sha256: 9df841c64b19a3843869467ff8ff2eb3f6c5491ebaac8fd94fb8029a5b00dcbf
+ md5: ef15f182e353155497e13726b915bfc4
+ depends:
+ - __osx >=10.13
+ license: Apache-2.0 WITH LLVM-exception
+ license_family: Apache
+ size: 1250659
+ timestamp: 1720040263499
- kind: conda
name: libexpat
version: 2.6.2
@@ -280,6 +368,20 @@ packages:
license_family: MIT
size: 73730
timestamp: 1710362120304
+- kind: conda
+ name: libexpat
+ version: 2.6.2
+ build: h73e2aa4_0
+ subdir: osx-64
+ url: https://conda.anaconda.org/conda-forge/osx-64/libexpat-2.6.2-h73e2aa4_0.conda
+ sha256: a188a77b275d61159a32ab547f7d17892226e7dac4518d2c6ac3ac8fc8dfde92
+ md5: 3d1d51c8f716d97c864d12f7af329526
+ constrains:
+ - expat 2.6.2.*
+ license: MIT
+ license_family: MIT
+ size: 69246
+ timestamp: 1710362566073
- kind: conda
name: libexpat
version: 2.6.2
@@ -294,6 +396,19 @@ packages:
license_family: MIT
size: 63655
timestamp: 1710362424980
+- kind: conda
+ name: libffi
+ version: 3.4.2
+ build: h0d85af4_5
+ build_number: 5
+ subdir: osx-64
+ url: https://conda.anaconda.org/conda-forge/osx-64/libffi-3.4.2-h0d85af4_5.tar.bz2
+ sha256: 7a2d27a936ceee6942ea4d397f9c7d136f12549d86f7617e8b6bad51e01a941f
+ md5: ccb34fb14960ad8b125962d3d79b31a9
+ license: MIT
+ license_family: MIT
+ size: 51348
+ timestamp: 1636488394370
- kind: conda
name: libffi
version: 3.4.2
@@ -429,6 +544,20 @@ packages:
license: Unlicense
size: 859858
timestamp: 1713367435849
+- kind: conda
+ name: libsqlite
+ version: 3.46.0
+ build: h1b8f9f3_0
+ subdir: osx-64
+ url: https://conda.anaconda.org/conda-forge/osx-64/libsqlite-3.46.0-h1b8f9f3_0.conda
+ sha256: 63af1a9e3284c7e4952364bafe7267e41e2d9d8bcc0e85a4ea4b0ec02d3693f6
+ md5: 5dadfbc1a567fe6e475df4ce3148be09
+ depends:
+ - __osx >=10.13
+ - libzlib >=1.2.13,<2.0a0
+ license: Unlicense
+ size: 908643
+ timestamp: 1718050720117
- kind: conda
name: libstdcxx-ng
version: 13.2.0
@@ -504,6 +633,23 @@ packages:
license_family: Other
size: 46768
timestamp: 1716874151980
+- kind: conda
+ name: libzlib
+ version: 1.3.1
+ build: h87427d6_1
+ build_number: 1
+ subdir: osx-64
+ url: https://conda.anaconda.org/conda-forge/osx-64/libzlib-1.3.1-h87427d6_1.conda
+ sha256: 80a62db652b1da0ccc100812a1d86e94f75028968991bfb17f9536f3aa72d91d
+ md5: b7575b5aa92108dcc9aaab0f05f2dbce
+ depends:
+ - __osx >=10.13
+ constrains:
+ - zlib 1.3.1 *_1
+ license: Zlib
+ license_family: Other
+ size: 57372
+ timestamp: 1716874211519
- kind: conda
name: ncurses
version: 6.4.20240210
@@ -517,6 +663,17 @@ packages:
license: X11 AND BSD-3-Clause
size: 895669
timestamp: 1710866638986
+- kind: conda
+ name: ncurses
+ version: '6.5'
+ build: h5846eda_0
+ subdir: osx-64
+ url: https://conda.anaconda.org/conda-forge/osx-64/ncurses-6.5-h5846eda_0.conda
+ sha256: 6ecc73db0e49143092c0934355ac41583a5d5a48c6914c5f6ca48e562d3a4b79
+ md5: 02a888433d165c99bf09784a7b14d900
+ license: X11 AND BSD-3-Clause
+ size: 823601
+ timestamp: 1715195267791
- kind: conda
name: ncurses
version: '6.5'
@@ -581,6 +738,24 @@ packages:
license_family: Apache
size: 2893954
timestamp: 1716468329572
+- kind: conda
+ name: openssl
+ version: 3.3.1
+ build: h87427d6_1
+ build_number: 1
+ subdir: osx-64
+ url: https://conda.anaconda.org/conda-forge/osx-64/openssl-3.3.1-h87427d6_1.conda
+ sha256: 60eed5d771207bcef05e0547c8f93a61d0ad1dcf75e19f8f8d9ded8094d78477
+ md5: d838ffe9ec3c6d971f110e04487466ff
+ depends:
+ - __osx >=10.13
+ - ca-certificates
+ constrains:
+ - pyopenssl >=22.1
+ license: Apache-2.0
+ license_family: Apache
+ size: 2551950
+ timestamp: 1719364820943
- kind: conda
name: python
version: 3.9.19
@@ -610,6 +785,30 @@ packages:
license: Python-2.0
size: 23800555
timestamp: 1710940120866
+- kind: conda
+ name: python
+ version: 3.9.19
+ build: h7a9c478_0_cpython
+ subdir: osx-64
+ url: https://conda.anaconda.org/conda-forge/osx-64/python-3.9.19-h7a9c478_0_cpython.conda
+ sha256: 58b76be84683bc03112b3ed7e377e99af24844ebf7d7568f6466a2dae7a887fe
+ md5: 7d53d366acd9dbfb498c69326ccb520a
+ depends:
+ - bzip2 >=1.0.8,<2.0a0
+ - libffi >=3.4,<4.0a0
+ - libsqlite >=3.45.2,<4.0a0
+ - libzlib >=1.2.13,<2.0.0a0
+ - ncurses >=6.4.20240210,<7.0a0
+ - openssl >=3.2.1,<4.0a0
+ - readline >=8.2,<9.0a0
+ - tk >=8.6.13,<8.7.0a0
+ - tzdata
+ - xz >=5.2.6,<6.0a0
+ constrains:
+ - python_abi 3.9.* *_cp39
+ license: Python-2.0
+ size: 12372436
+ timestamp: 1710940037648
- kind: conda
name: python
version: 3.9.19
@@ -634,6 +833,30 @@ packages:
license: Python-2.0
size: 11847835
timestamp: 1710939779164
+- kind: conda
+ name: python
+ version: 3.10.14
+ build: h00d2728_0_cpython
+ subdir: osx-64
+ url: https://conda.anaconda.org/conda-forge/osx-64/python-3.10.14-h00d2728_0_cpython.conda
+ sha256: 00c1de2d46ede26609ef4e84a44b83be7876ba6a0215b7c83bff41a0656bf694
+ md5: 0a1cddc4382c5c171e791c70740546dd
+ depends:
+ - bzip2 >=1.0.8,<2.0a0
+ - libffi >=3.4,<4.0a0
+ - libsqlite >=3.45.2,<4.0a0
+ - libzlib >=1.2.13,<2.0.0a0
+ - ncurses >=6.4.20240210,<7.0a0
+ - openssl >=3.2.1,<4.0a0
+ - readline >=8.2,<9.0a0
+ - tk >=8.6.13,<8.7.0a0
+ - tzdata
+ - xz >=5.2.6,<6.0a0
+ constrains:
+ - python_abi 3.10.* *_cp310
+ license: Python-2.0
+ size: 11890228
+ timestamp: 1710940046031
- kind: conda
name: python
version: 3.10.14
@@ -687,6 +910,32 @@ packages:
license: Python-2.0
size: 25517742
timestamp: 1710939725109
+- kind: conda
+ name: python
+ version: 3.11.9
+ build: h657bba9_0_cpython
+ subdir: osx-64
+ url: https://conda.anaconda.org/conda-forge/osx-64/python-3.11.9-h657bba9_0_cpython.conda
+ sha256: 3b50a5abb3b812875beaa9ab792dbd1bf44f335c64e9f9fedcf92d953995651c
+ md5: 612763bc5ede9552e4233ec518b9c9fb
+ depends:
+ - __osx >=10.9
+ - bzip2 >=1.0.8,<2.0a0
+ - libexpat >=2.6.2,<3.0a0
+ - libffi >=3.4,<4.0a0
+ - libsqlite >=3.45.3,<4.0a0
+ - libzlib >=1.2.13,<2.0.0a0
+ - ncurses >=6.4.20240210,<7.0a0
+ - openssl >=3.2.1,<4.0a0
+ - readline >=8.2,<9.0a0
+ - tk >=8.6.13,<8.7.0a0
+ - tzdata
+ - xz >=5.2.6,<6.0a0
+ constrains:
+ - python_abi 3.11.* *_cp311
+ license: Python-2.0
+ size: 15503226
+ timestamp: 1713553747073
- kind: conda
name: python
version: 3.11.9
@@ -774,6 +1023,36 @@ packages:
license_family: GPL
size: 250351
timestamp: 1679532511311
+- kind: conda
+ name: readline
+ version: '8.2'
+ build: h9e318b2_1
+ build_number: 1
+ subdir: osx-64
+ url: https://conda.anaconda.org/conda-forge/osx-64/readline-8.2-h9e318b2_1.conda
+ sha256: 41e7d30a097d9b060037f0c6a2b1d4c4ae7e942c06c943d23f9d481548478568
+ md5: f17f77f2acf4d344734bda76829ce14e
+ depends:
+ - ncurses >=6.3,<7.0a0
+ license: GPL-3.0-only
+ license_family: GPL
+ size: 255870
+ timestamp: 1679532707590
+- kind: conda
+ name: tk
+ version: 8.6.13
+ build: h1abcd95_1
+ build_number: 1
+ subdir: osx-64
+ url: https://conda.anaconda.org/conda-forge/osx-64/tk-8.6.13-h1abcd95_1.conda
+ sha256: 30412b2e9de4ff82d8c2a7e5d06a15f4f4fef1809a72138b6ccb53a33b26faf5
+ md5: bf830ba5afc507c6232d4ef0fb1a882d
+ depends:
+ - libzlib >=1.2.13,<2.0.0a0
+ license: TCL
+ license_family: BSD
+ size: 3270220
+ timestamp: 1699202389792
- kind: conda
name: tk
version: 8.6.13
@@ -831,6 +1110,22 @@ packages:
license: Apache-2.0 OR MIT
size: 11891252
timestamp: 1714233659570
+- kind: conda
+ name: uv
+ version: 0.1.45
+ build: h4e38c46_0
+ subdir: osx-64
+ url: https://conda.anaconda.org/conda-forge/osx-64/uv-0.1.45-h4e38c46_0.conda
+ sha256: 8c11774ca1940dcd90187ce240afea26b76e2942f9b18d65f6d4b483534193fd
+ md5: 754ce8a22c94a30c7bbd42274c7fae31
+ depends:
+ - __osx >=10.13
+ - libcxx >=16
+ constrains:
+ - __osx >=10.12
+ license: Apache-2.0 OR MIT
+ size: 8937335
+ timestamp: 1716265195083
- kind: conda
name: uv
version: 0.1.45
@@ -871,3 +1166,14 @@ packages:
license: LGPL-2.1 and GPL-2.0
size: 235693
timestamp: 1660346961024
+- kind: conda
+ name: xz
+ version: 5.2.6
+ build: h775f41a_0
+ subdir: osx-64
+ url: https://conda.anaconda.org/conda-forge/osx-64/xz-5.2.6-h775f41a_0.tar.bz2
+ sha256: eb09823f34cc2dd663c0ec4ab13f246f45dcd52e5b8c47b9864361de5204a1c8
+ md5: a72f9d4ea13d55d745ff1ed594747f10
+ license: LGPL-2.1 and GPL-2.0
+ size: 238119
+ timestamp: 1660346964847
diff --git a/infra/scripts/pixi/pixi.toml b/infra/scripts/pixi/pixi.toml
index 10179339f70..487c6f7def1 100644
--- a/infra/scripts/pixi/pixi.toml
+++ b/infra/scripts/pixi/pixi.toml
@@ -1,7 +1,7 @@
[project]
name = "pixi-feast"
channels = ["conda-forge"]
-platforms = ["linux-64", "osx-arm64"]
+platforms = ["linux-64", "osx-arm64", "osx-64"]
[tasks]
diff --git a/java/pom.xml b/java/pom.xml
index 492e756ba57..2c1c32792c8 100644
--- a/java/pom.xml
+++ b/java/pom.xml
@@ -35,7 +35,7 @@
- 0.39.0
+ 0.40.0
https://github.com/feast-dev/feast
UTF-8
diff --git a/sdk/python/docs/source/feast.infra.contrib.rst b/sdk/python/docs/source/feast.infra.contrib.rst
index 7b2fa3cc9c5..1f46ff0abf8 100644
--- a/sdk/python/docs/source/feast.infra.contrib.rst
+++ b/sdk/python/docs/source/feast.infra.contrib.rst
@@ -4,14 +4,6 @@ feast.infra.contrib package
Submodules
----------
-feast.infra.contrib.azure\_provider module
-------------------------------------------
-
-.. automodule:: feast.infra.contrib.azure_provider
- :members:
- :undoc-members:
- :show-inheritance:
-
feast.infra.contrib.grpc\_server module
---------------------------------------
diff --git a/sdk/python/docs/source/feast.infra.feature_servers.rst b/sdk/python/docs/source/feast.infra.feature_servers.rst
index 334b5859053..ca5203504df 100644
--- a/sdk/python/docs/source/feast.infra.feature_servers.rst
+++ b/sdk/python/docs/source/feast.infra.feature_servers.rst
@@ -7,8 +7,6 @@ Subpackages
.. toctree::
:maxdepth: 4
- feast.infra.feature_servers.aws_lambda
- feast.infra.feature_servers.gcp_cloudrun
feast.infra.feature_servers.local_process
feast.infra.feature_servers.multicloud
diff --git a/sdk/python/docs/source/feast.infra.offline_stores.rst b/sdk/python/docs/source/feast.infra.offline_stores.rst
index 052a114cfb3..c770e5c13b0 100644
--- a/sdk/python/docs/source/feast.infra.offline_stores.rst
+++ b/sdk/python/docs/source/feast.infra.offline_stores.rst
@@ -28,18 +28,18 @@ feast.infra.offline\_stores.bigquery\_source module
:undoc-members:
:show-inheritance:
-feast.infra.offline\_stores.duckdb module
------------------------------------------
+feast.infra.offline\_stores.dask module
+---------------------------------------
-.. automodule:: feast.infra.offline_stores.duckdb
+.. automodule:: feast.infra.offline_stores.dask
:members:
:undoc-members:
:show-inheritance:
-feast.infra.offline\_stores.file module
----------------------------------------
+feast.infra.offline\_stores.duckdb module
+-----------------------------------------
-.. automodule:: feast.infra.offline_stores.file
+.. automodule:: feast.infra.offline_stores.duckdb
:members:
:undoc-members:
:show-inheritance:
diff --git a/sdk/python/docs/source/feast.infra.online_stores.contrib.rst b/sdk/python/docs/source/feast.infra.online_stores.contrib.rst
index d614438e3d5..9d301fcd0de 100644
--- a/sdk/python/docs/source/feast.infra.online_stores.contrib.rst
+++ b/sdk/python/docs/source/feast.infra.online_stores.contrib.rst
@@ -89,6 +89,14 @@ feast.infra.online\_stores.contrib.postgres\_repo\_configuration module
:undoc-members:
:show-inheritance:
+feast.infra.online\_stores.contrib.singlestore\_repo\_configuration module
+--------------------------------------------------------------------------
+
+.. automodule:: feast.infra.online_stores.contrib.singlestore_repo_configuration
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
Module contents
---------------
diff --git a/sdk/python/docs/source/feast.infra.registry.contrib.postgres.rst b/sdk/python/docs/source/feast.infra.registry.contrib.postgres.rst
new file mode 100644
index 00000000000..3f319908057
--- /dev/null
+++ b/sdk/python/docs/source/feast.infra.registry.contrib.postgres.rst
@@ -0,0 +1,21 @@
+feast.infra.registry.contrib.postgres package
+=============================================
+
+Submodules
+----------
+
+feast.infra.registry.contrib.postgres.postgres\_registry\_store module
+----------------------------------------------------------------------
+
+.. automodule:: feast.infra.registry.contrib.postgres.postgres_registry_store
+ :members:
+ :undoc-members:
+ :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: feast.infra.registry.contrib.postgres
+ :members:
+ :undoc-members:
+ :show-inheritance:
diff --git a/sdk/python/docs/source/feast.infra.rst b/sdk/python/docs/source/feast.infra.rst
index a1dfc864926..b0046a2719e 100644
--- a/sdk/python/docs/source/feast.infra.rst
+++ b/sdk/python/docs/source/feast.infra.rst
@@ -19,22 +19,6 @@ Subpackages
Submodules
----------
-feast.infra.aws module
-----------------------
-
-.. automodule:: feast.infra.aws
- :members:
- :undoc-members:
- :show-inheritance:
-
-feast.infra.gcp module
-----------------------
-
-.. automodule:: feast.infra.gcp
- :members:
- :undoc-members:
- :show-inheritance:
-
feast.infra.infra\_object module
--------------------------------
@@ -51,14 +35,6 @@ feast.infra.key\_encoding\_utils module
:undoc-members:
:show-inheritance:
-feast.infra.local module
-------------------------
-
-.. automodule:: feast.infra.local
- :members:
- :undoc-members:
- :show-inheritance:
-
feast.infra.passthrough\_provider module
----------------------------------------
diff --git a/sdk/python/feast/constants.py b/sdk/python/feast/constants.py
index fa8674d91d2..7dd39458211 100644
--- a/sdk/python/feast/constants.py
+++ b/sdk/python/feast/constants.py
@@ -17,9 +17,6 @@
# Maximum interval(secs) to wait between retries for retry function
MAX_WAIT_INTERVAL: str = "60"
-AWS_LAMBDA_FEATURE_SERVER_IMAGE = "feastdev/feature-server-python-aws"
-AWS_LAMBDA_FEATURE_SERVER_REPOSITORY = "feast-python-server"
-
# feature_store.yaml environment variable name for remote feature server
FEATURE_STORE_YAML_ENV_NAME: str = "FEATURE_STORE_YAML_BASE64"
@@ -44,8 +41,5 @@
# Default offline server port
DEFAULT_OFFLINE_SERVER_PORT = 8815
-# Environment variable for feature server docker image tag
-DOCKER_IMAGE_TAG_ENV_NAME: str = "FEAST_SERVER_DOCKER_IMAGE_TAG"
-
# Default feature server registry ttl (seconds)
DEFAULT_FEATURE_SERVER_REGISTRY_TTL = 5
diff --git a/sdk/python/feast/diff/registry_diff.py b/sdk/python/feast/diff/registry_diff.py
index b608757496f..9236b087d4b 100644
--- a/sdk/python/feast/diff/registry_diff.py
+++ b/sdk/python/feast/diff/registry_diff.py
@@ -20,6 +20,7 @@
OnDemandFeatureView as OnDemandFeatureViewProto,
)
from feast.protos.feast.core.OnDemandFeatureView_pb2 import OnDemandFeatureViewSpec
+from feast.protos.feast.core.SavedDataset_pb2 import SavedDataset as SavedDatasetProto
from feast.protos.feast.core.StreamFeatureView_pb2 import (
StreamFeatureView as StreamFeatureViewProto,
)
@@ -109,6 +110,7 @@ def tag_objects_for_keep_delete_update_add(
OnDemandFeatureViewProto,
StreamFeatureViewProto,
ValidationReferenceProto,
+ SavedDatasetProto,
)
diff --git a/sdk/python/feast/driver_test_data.py b/sdk/python/feast/driver_test_data.py
index 7959046e6eb..defeb404a3a 100644
--- a/sdk/python/feast/driver_test_data.py
+++ b/sdk/python/feast/driver_test_data.py
@@ -61,11 +61,11 @@ def create_orders_df(
df["order_is_success"] = np.random.randint(0, 2, size=order_count).astype(np.int32)
df[DEFAULT_ENTITY_DF_EVENT_TIMESTAMP_COL] = [
_convert_event_timestamp(
- pd.Timestamp(dt, unit="ms", tz="UTC").round("ms"),
+ pd.Timestamp(dt, unit="ms").round("ms"),
EventTimestampType(idx % 4),
)
for idx, dt in enumerate(
- pd.date_range(start=start_date, end=end_date, periods=order_count)
+ pd.date_range(start=start_date, end=end_date, periods=order_count, tz="UTC")
)
]
df.sort_values(
@@ -101,9 +101,13 @@ def create_driver_hourly_stats_df(drivers, start_date, end_date) -> pd.DataFrame
df_hourly = pd.DataFrame(
{
"event_timestamp": [
- pd.Timestamp(dt, unit="ms", tz="UTC").round("ms")
+ pd.Timestamp(dt, unit="ms").round("ms")
for dt in pd.date_range(
- start=start_date, end=end_date, freq="1h", inclusive="left"
+ start=start_date,
+ end=end_date,
+ freq="1h",
+ inclusive="left",
+ tz="UTC",
)
]
# include a fixed timestamp for get_historical_features in the quickstart
@@ -162,9 +166,13 @@ def create_customer_daily_profile_df(customers, start_date, end_date) -> pd.Data
df_daily = pd.DataFrame(
{
"event_timestamp": [
- pd.Timestamp(dt, unit="ms", tz="UTC").round("ms")
+ pd.Timestamp(dt, unit="ms").round("ms")
for dt in pd.date_range(
- start=start_date, end=end_date, freq="1D", inclusive="left"
+ start=start_date,
+ end=end_date,
+ freq="1D",
+ inclusive="left",
+ tz="UTC",
)
]
}
@@ -207,9 +215,13 @@ def create_location_stats_df(locations, start_date, end_date) -> pd.DataFrame:
df_hourly = pd.DataFrame(
{
"event_timestamp": [
- pd.Timestamp(dt, unit="ms", tz="UTC").round("ms")
+ pd.Timestamp(dt, unit="ms").round("ms")
for dt in pd.date_range(
- start=start_date, end=end_date, freq="1h", inclusive="left"
+ start=start_date,
+ end=end_date,
+ freq="1h",
+ inclusive="left",
+ tz="UTC",
)
]
}
@@ -254,9 +266,16 @@ def create_global_daily_stats_df(start_date, end_date) -> pd.DataFrame:
df_daily = pd.DataFrame(
{
"event_timestamp": [
- pd.Timestamp(dt, unit="ms", tz="UTC").round("ms")
+ pd.Timestamp(
+ dt,
+ unit="ms",
+ ).round("ms")
for dt in pd.date_range(
- start=start_date, end=end_date, freq="1D", inclusive="left"
+ start=start_date,
+ end=end_date,
+ freq="1D",
+ inclusive="left",
+ tz="UTC",
)
]
}
@@ -286,11 +305,11 @@ def create_field_mapping_df(start_date, end_date) -> pd.DataFrame:
df["column_name"] = np.random.randint(1, 100, size=size).astype(np.int32)
df[DEFAULT_ENTITY_DF_EVENT_TIMESTAMP_COL] = [
_convert_event_timestamp(
- pd.Timestamp(dt, unit="ms", tz="UTC").round("ms"),
+ pd.Timestamp(dt, unit="ms").round("ms"),
EventTimestampType(idx % 4),
)
for idx, dt in enumerate(
- pd.date_range(start=start_date, end=end_date, periods=size)
+ pd.date_range(start=start_date, end=end_date, periods=size, tz="UTC")
)
]
df["created"] = pd.to_datetime(pd.Timestamp.now(tz=None).round("ms"))
diff --git a/sdk/python/feast/errors.py b/sdk/python/feast/errors.py
index 52fefce9d90..c4c11576269 100644
--- a/sdk/python/feast/errors.py
+++ b/sdk/python/feast/errors.py
@@ -119,23 +119,11 @@ def __init__(self, provider_name):
super().__init__(f"Provider '{provider_name}' is not implemented")
-class FeastProviderNotSetError(Exception):
- def __init__(self):
- super().__init__("Provider is not set, but is required")
-
-
class FeastRegistryNotSetError(Exception):
def __init__(self):
super().__init__("Registry is not set, but is required")
-class FeastFeatureServerTypeSetError(Exception):
- def __init__(self, feature_server_type: str):
- super().__init__(
- f"Feature server type was set to {feature_server_type}, but the type should be determined by the provider"
- )
-
-
class FeastFeatureServerTypeInvalidError(Exception):
def __init__(self, feature_server_type: str):
super().__init__(
@@ -350,32 +338,6 @@ def __init__(self, feature_view_name: str):
)
-class RepoConfigPathDoesNotExist(Exception):
- def __init__(self):
- super().__init__("The repo_path attribute does not exist for the repo_config.")
-
-
-class AwsLambdaDoesNotExist(Exception):
- def __init__(self, resource_name: str):
- super().__init__(
- f"The AWS Lambda function {resource_name} should have been created properly, but does not exist."
- )
-
-
-class AwsAPIGatewayDoesNotExist(Exception):
- def __init__(self, resource_name: str):
- super().__init__(
- f"The AWS API Gateway {resource_name} should have been created properly, but does not exist."
- )
-
-
-class IncompatibleRegistryStoreClass(Exception):
- def __init__(self, actual_class: str, expected_class: str):
- super().__init__(
- f"The registry store class was expected to be {expected_class}, but was instead {actual_class}."
- )
-
-
class FeastInvalidInfraObjectType(Exception):
def __init__(self):
super().__init__("Could not identify the type of the InfraObject.")
@@ -427,3 +389,13 @@ def __init__(self, input_dict: dict):
super().__init__(
f"Failed to serialize the provided dictionary into a pandas DataFrame: {input_dict.keys()}"
)
+
+
+class ZeroRowsQueryResult(Exception):
+ def __init__(self, query: str):
+ super().__init__(f"This query returned zero rows:\n{query}")
+
+
+class ZeroColumnQueryResult(Exception):
+ def __init__(self, query: str):
+ super().__init__(f"This query returned zero columns:\n{query}")
diff --git a/sdk/python/feast/feast_object.py b/sdk/python/feast/feast_object.py
index 2d06d8d669d..d9505dcb9f0 100644
--- a/sdk/python/feast/feast_object.py
+++ b/sdk/python/feast/feast_object.py
@@ -11,11 +11,12 @@
from .protos.feast.core.FeatureService_pb2 import FeatureServiceSpec
from .protos.feast.core.FeatureView_pb2 import FeatureViewSpec
from .protos.feast.core.OnDemandFeatureView_pb2 import OnDemandFeatureViewSpec
+from .protos.feast.core.SavedDataset_pb2 import SavedDatasetSpec
from .protos.feast.core.StreamFeatureView_pb2 import StreamFeatureViewSpec
from .protos.feast.core.ValidationProfile_pb2 import (
ValidationReference as ValidationReferenceProto,
)
-from .saved_dataset import ValidationReference
+from .saved_dataset import SavedDataset, ValidationReference
from .stream_feature_view import StreamFeatureView
# Convenience type representing all Feast objects
@@ -28,6 +29,7 @@
FeatureService,
DataSource,
ValidationReference,
+ SavedDataset,
]
FeastObjectSpecProto = Union[
@@ -38,4 +40,5 @@
FeatureServiceSpec,
DataSourceProto,
ValidationReferenceProto,
+ SavedDatasetSpec,
]
diff --git a/sdk/python/feast/feature_store.py b/sdk/python/feast/feature_store.py
index b7e4ef619f0..9600732e176 100644
--- a/sdk/python/feast/feature_store.py
+++ b/sdk/python/feast/feature_store.py
@@ -86,6 +86,7 @@
from feast.repo_contents import RepoContents
from feast.saved_dataset import SavedDataset, SavedDatasetStorage, ValidationReference
from feast.stream_feature_view import StreamFeatureView
+from feast.utils import _utc_now
from feast.version import get_version
warnings.simplefilter("once", DeprecationWarning)
@@ -1246,7 +1247,7 @@ def materialize_incremental(
>>> from feast import FeatureStore, RepoConfig
>>> from datetime import datetime, timedelta
>>> fs = FeatureStore(repo_path="project/feature_repo")
- >>> fs.materialize_incremental(end_date=datetime.utcnow() - timedelta(minutes=5))
+ >>> fs.materialize_incremental(end_date=_utc_now() - timedelta(minutes=5))
Materializing...
...
@@ -1270,7 +1271,7 @@ def materialize_incremental(
f" either a ttl to be set or for materialize() to have been run at least once."
)
elif feature_view.ttl.total_seconds() > 0:
- start_date = datetime.utcnow() - feature_view.ttl
+ start_date = _utc_now() - feature_view.ttl
else:
# TODO(felixwang9817): Find the earliest timestamp for this specific feature
# view from the offline store, and set the start date to that timestamp.
@@ -1278,7 +1279,7 @@ def materialize_incremental(
f"Since the ttl is 0 for feature view {Style.BRIGHT + Fore.GREEN}{feature_view.name}{Style.RESET_ALL}, "
"the start date will be set to 1 year before the current time."
)
- start_date = datetime.utcnow() - timedelta(weeks=52)
+ start_date = _utc_now() - timedelta(weeks=52)
provider = self._get_provider()
print(
f"{Style.BRIGHT + Fore.GREEN}{feature_view.name}{Style.RESET_ALL}"
@@ -1335,7 +1336,7 @@ def materialize(
>>> from datetime import datetime, timedelta
>>> fs = FeatureStore(repo_path="project/feature_repo")
>>> fs.materialize(
- ... start_date=datetime.utcnow() - timedelta(hours=3), end_date=datetime.utcnow() - timedelta(minutes=10)
+ ... start_date=_utc_now() - timedelta(hours=3), end_date=_utc_now() - timedelta(minutes=10)
... )
Materializing...
@@ -1559,76 +1560,17 @@ def get_online_features(
... )
>>> online_response_dict = online_response.to_dict()
"""
- if isinstance(entity_rows, list):
- columnar: Dict[str, List[Any]] = {k: [] for k in entity_rows[0].keys()}
- for entity_row in entity_rows:
- for key, value in entity_row.items():
- try:
- columnar[key].append(value)
- except KeyError as e:
- raise ValueError(
- "All entity_rows must have the same keys."
- ) from e
-
- entity_rows = columnar
+ provider = self._get_provider()
- (
- join_key_values,
- grouped_refs,
- entity_name_to_join_key_map,
- requested_on_demand_feature_views,
- feature_refs,
- requested_result_row_names,
- online_features_response,
- ) = utils._prepare_entities_to_read_from_online_store(
+ return provider.get_online_features(
+ config=self.config,
+ features=features,
+ entity_rows=entity_rows,
registry=self._registry,
project=self.project,
- features=features,
- entity_values=entity_rows,
full_feature_names=full_feature_names,
- native_entity_values=True,
)
- provider = self._get_provider()
- for table, requested_features in grouped_refs:
- # Get the correct set of entity values with the correct join keys.
- table_entity_values, idxs = utils._get_unique_entities(
- table,
- join_key_values,
- entity_name_to_join_key_map,
- )
-
- # Fetch feature data for the minimum set of Entities.
- feature_data = self._read_from_online_store(
- table_entity_values,
- provider,
- requested_features,
- table,
- )
-
- # Populate the result_rows with the Features from the OnlineStore inplace.
- utils._populate_response_from_feature_data(
- feature_data,
- idxs,
- online_features_response,
- full_feature_names,
- requested_features,
- table,
- )
-
- if requested_on_demand_feature_views:
- utils._augment_response_with_on_demand_transforms(
- online_features_response,
- feature_refs,
- requested_on_demand_feature_views,
- full_feature_names,
- )
-
- utils._drop_unneeded_columns(
- online_features_response, requested_result_row_names
- )
- return OnlineResponse(online_features_response)
-
async def get_online_features_async(
self,
features: Union[List[str], FeatureService],
@@ -1664,76 +1606,17 @@ async def get_online_features_async(
Raises:
Exception: No entity with the specified name exists.
"""
- if isinstance(entity_rows, list):
- columnar: Dict[str, List[Any]] = {k: [] for k in entity_rows[0].keys()}
- for entity_row in entity_rows:
- for key, value in entity_row.items():
- try:
- columnar[key].append(value)
- except KeyError as e:
- raise ValueError(
- "All entity_rows must have the same keys."
- ) from e
-
- entity_rows = columnar
+ provider = self._get_provider()
- (
- join_key_values,
- grouped_refs,
- entity_name_to_join_key_map,
- requested_on_demand_feature_views,
- feature_refs,
- requested_result_row_names,
- online_features_response,
- ) = utils._prepare_entities_to_read_from_online_store(
+ return await provider.get_online_features_async(
+ config=self.config,
+ features=features,
+ entity_rows=entity_rows,
registry=self._registry,
project=self.project,
- features=features,
- entity_values=entity_rows,
full_feature_names=full_feature_names,
- native_entity_values=True,
)
- provider = self._get_provider()
- for table, requested_features in grouped_refs:
- # Get the correct set of entity values with the correct join keys.
- table_entity_values, idxs = utils._get_unique_entities(
- table,
- join_key_values,
- entity_name_to_join_key_map,
- )
-
- # Fetch feature data for the minimum set of Entities.
- feature_data = await self._read_from_online_store_async(
- table_entity_values,
- provider,
- requested_features,
- table,
- )
-
- # Populate the result_rows with the Features from the OnlineStore inplace.
- utils._populate_response_from_feature_data(
- feature_data,
- idxs,
- online_features_response,
- full_feature_names,
- requested_features,
- table,
- )
-
- if requested_on_demand_feature_views:
- utils._augment_response_with_on_demand_transforms(
- online_features_response,
- feature_refs,
- requested_on_demand_feature_views,
- full_feature_names,
- )
-
- utils._drop_unneeded_columns(
- online_features_response, requested_result_row_names
- )
- return OnlineResponse(online_features_response)
-
def retrieve_online_documents(
self,
feature: str,
@@ -1806,53 +1689,6 @@ def retrieve_online_documents(
)
return OnlineResponse(online_features_response)
- def _read_from_online_store(
- self,
- entity_rows: Iterable[Mapping[str, Value]],
- provider: Provider,
- requested_features: List[str],
- table: FeatureView,
- ) -> List[Tuple[List[Timestamp], List["FieldStatus.ValueType"], List[Value]]]:
- """Read and process data from the OnlineStore for a given FeatureView.
-
- This method guarantees that the order of the data in each element of the
- List returned is the same as the order of `requested_features`.
-
- This method assumes that `provider.online_read` returns data for each
- combination of Entities in `entity_rows` in the same order as they
- are provided.
- """
- entity_key_protos = utils._get_entity_key_protos(entity_rows)
-
- # Fetch data for Entities.
- read_rows = provider.online_read(
- config=self.config,
- table=table,
- entity_keys=entity_key_protos,
- requested_features=requested_features,
- )
-
- return utils._convert_rows_to_protobuf(requested_features, read_rows)
-
- async def _read_from_online_store_async(
- self,
- entity_rows: Iterable[Mapping[str, Value]],
- provider: Provider,
- requested_features: List[str],
- table: FeatureView,
- ) -> List[Tuple[List[Timestamp], List["FieldStatus.ValueType"], List[Value]]]:
- entity_key_protos = utils._get_entity_key_protos(entity_rows)
-
- # Fetch data for Entities.
- read_rows = await provider.online_read_async(
- config=self.config,
- table=table,
- entity_keys=entity_key_protos,
- requested_features=requested_features,
- )
-
- return utils._convert_rows_to_protobuf(requested_features, read_rows)
-
def _retrieve_from_online_store(
self,
provider: Provider,
diff --git a/sdk/python/feast/feature_view.py b/sdk/python/feast/feature_view.py
index ff41400eace..1a85a4b90c0 100644
--- a/sdk/python/feast/feature_view.py
+++ b/sdk/python/feast/feature_view.py
@@ -311,6 +311,16 @@ def with_join_key_map(self, join_key_map: Dict[str, str]):
return cp
+ def update_materialization_intervals(
+ self, existing_materialization_intervals: List[Tuple[datetime, datetime]]
+ ):
+ if (
+ len(existing_materialization_intervals) > 0
+ and len(self.materialization_intervals) == 0
+ ):
+ for interval in existing_materialization_intervals:
+ self.materialization_intervals.append((interval[0], interval[1]))
+
def to_proto(self) -> FeatureViewProto:
"""
Converts a feature view object to its protobuf representation.
diff --git a/sdk/python/feast/infra/aws.py b/sdk/python/feast/infra/aws.py
deleted file mode 100644
index bb896fa961f..00000000000
--- a/sdk/python/feast/infra/aws.py
+++ /dev/null
@@ -1,350 +0,0 @@
-import base64
-import hashlib
-import logging
-import os
-import uuid
-import warnings
-from typing import Optional, Sequence
-
-from colorama import Fore, Style
-
-from feast import utils
-from feast.constants import (
- AWS_LAMBDA_FEATURE_SERVER_IMAGE,
- AWS_LAMBDA_FEATURE_SERVER_REPOSITORY,
- DOCKER_IMAGE_TAG_ENV_NAME,
- FEATURE_STORE_YAML_ENV_NAME,
-)
-from feast.entity import Entity
-from feast.errors import (
- AwsAPIGatewayDoesNotExist,
- AwsLambdaDoesNotExist,
- IncompatibleRegistryStoreClass,
- RepoConfigPathDoesNotExist,
-)
-from feast.feature_view import FeatureView
-from feast.infra.feature_servers.aws_lambda.config import AwsLambdaFeatureServerConfig
-from feast.infra.passthrough_provider import PassthroughProvider
-from feast.infra.registry.registry import get_registry_store_class_from_scheme
-from feast.infra.registry.s3 import S3RegistryStore
-from feast.infra.utils import aws_utils
-from feast.version import get_version
-
-try:
- import boto3
-except ImportError as e:
- from feast.errors import FeastExtrasDependencyImportError
-
- raise FeastExtrasDependencyImportError("aws", str(e))
-
-_logger = logging.getLogger(__name__)
-
-
-class AwsProvider(PassthroughProvider):
- def update_infra(
- self,
- project: str,
- tables_to_delete: Sequence[FeatureView],
- tables_to_keep: Sequence[FeatureView],
- entities_to_delete: Sequence[Entity],
- entities_to_keep: Sequence[Entity],
- partial: bool,
- ):
- # Call update only if there is an online store
- if self.online_store:
- self.online_store.update(
- config=self.repo_config,
- tables_to_delete=tables_to_delete,
- tables_to_keep=tables_to_keep,
- entities_to_keep=entities_to_keep,
- entities_to_delete=entities_to_delete,
- partial=partial,
- )
-
- if self.repo_config.feature_server and self.repo_config.feature_server.enabled:
- warnings.warn(
- "AWS Lambda based feature serving is an experimental feature. "
- "We do not guarantee that future changes will maintain backward compatibility.",
- RuntimeWarning,
- )
-
- # Since the AWS Lambda feature server will attempt to load the registry, we
- # only allow the registry to be in S3.
- registry_path = (
- self.repo_config.registry
- if isinstance(self.repo_config.registry, str)
- else self.repo_config.registry.path
- )
- registry_store_class = get_registry_store_class_from_scheme(registry_path)
- if registry_store_class != S3RegistryStore:
- raise IncompatibleRegistryStoreClass(
- registry_store_class.__name__, S3RegistryStore.__name__
- )
-
- ecr_client = boto3.client("ecr")
- docker_image_version = _get_docker_image_version()
- repository_uri = self._create_or_get_repository_uri(ecr_client)
- # Only download & upload the docker image if it doesn't already exist in ECR
- if not ecr_client.batch_get_image(
- repositoryName=AWS_LAMBDA_FEATURE_SERVER_REPOSITORY,
- imageIds=[{"imageTag": docker_image_version}],
- ).get("images"):
- image_uri = self._upload_docker_image(
- ecr_client, repository_uri, docker_image_version
- )
- else:
- image_uri = f"{repository_uri}:{docker_image_version}"
-
- self._deploy_feature_server(project, image_uri)
-
- if self.batch_engine:
- self.batch_engine.update(
- project,
- tables_to_delete,
- tables_to_keep,
- entities_to_delete,
- entities_to_keep,
- )
-
- def _deploy_feature_server(self, project: str, image_uri: str):
- _logger.info("Deploying feature server...")
-
- if not self.repo_config.repo_path:
- raise RepoConfigPathDoesNotExist()
-
- with open(
- utils.get_default_yaml_file_path(self.repo_config.repo_path), "rb"
- ) as f:
- config_bytes = f.read()
- config_base64 = base64.b64encode(config_bytes).decode()
-
- resource_name = _get_lambda_name(project)
- lambda_client = boto3.client("lambda")
- api_gateway_client = boto3.client("apigatewayv2")
- function = aws_utils.get_lambda_function(lambda_client, resource_name)
- _logger.debug("Using function name: %s", resource_name)
- _logger.debug("Found function: %s", function)
-
- if function is None:
- # If the Lambda function does not exist, create it.
- _logger.info(" Creating AWS Lambda...")
- assert isinstance(
- self.repo_config.feature_server, AwsLambdaFeatureServerConfig
- )
- lambda_client.create_function(
- FunctionName=resource_name,
- Role=self.repo_config.feature_server.execution_role_name,
- Code={"ImageUri": image_uri},
- PackageType="Image",
- MemorySize=1769,
- Environment={"Variables": {FEATURE_STORE_YAML_ENV_NAME: config_base64}},
- Tags={
- "feast-owned": "True",
- "project": project,
- "feast-sdk-version": get_version(),
- },
- )
- function = aws_utils.get_lambda_function(lambda_client, resource_name)
- if not function:
- raise AwsLambdaDoesNotExist(resource_name)
- else:
- # If the feature_store.yaml has changed, need to update the environment variable.
- env = function.get("Environment", {}).get("Variables", {})
- if env.get(FEATURE_STORE_YAML_ENV_NAME) != config_base64:
- # Note, that this does not update Lambda gracefully (e.g. no rolling deployment).
- # It's expected that feature_store.yaml is not regularly updated while the lambda
- # is serving production traffic. However, the update in registry (e.g. modifying
- # feature views, feature services, and other definitions does not update lambda).
- _logger.info(" Updating AWS Lambda...")
-
- aws_utils.update_lambda_function_environment(
- lambda_client,
- resource_name,
- {"Variables": {FEATURE_STORE_YAML_ENV_NAME: config_base64}},
- )
-
- api = aws_utils.get_first_api_gateway(api_gateway_client, resource_name)
- if not api:
- # If the API Gateway doesn't exist, create it
- _logger.info(" Creating AWS API Gateway...")
- api = api_gateway_client.create_api(
- Name=resource_name,
- ProtocolType="HTTP",
- Target=function["FunctionArn"],
- RouteKey="POST /get-online-features",
- Tags={
- "feast-owned": "True",
- "project": project,
- "feast-sdk-version": get_version(),
- },
- )
- if not api:
- raise AwsAPIGatewayDoesNotExist(resource_name)
- # Make sure to give AWS Lambda a permission to be invoked by the newly created API Gateway
- api_id = api["ApiId"]
- region = lambda_client.meta.region_name
- account_id = aws_utils.get_account_id()
- lambda_client.add_permission(
- FunctionName=function["FunctionArn"],
- StatementId=str(uuid.uuid4()),
- Action="lambda:InvokeFunction",
- Principal="apigateway.amazonaws.com",
- SourceArn=f"arn:aws:execute-api:{region}:{account_id}:{api_id}/*/*/get-online-features",
- )
-
- def teardown_infra(
- self,
- project: str,
- tables: Sequence[FeatureView],
- entities: Sequence[Entity],
- ) -> None:
- super(AwsProvider, self).teardown_infra(project, tables, entities)
-
- if (
- self.repo_config.feature_server is not None
- and self.repo_config.feature_server.enabled
- ):
- _logger.info("Tearing down feature server...")
- resource_name = _get_lambda_name(project)
- lambda_client = boto3.client("lambda")
- api_gateway_client = boto3.client("apigatewayv2")
-
- function = aws_utils.get_lambda_function(lambda_client, resource_name)
-
- if function is not None:
- _logger.info(" Tearing down AWS Lambda...")
- aws_utils.delete_lambda_function(lambda_client, resource_name)
-
- api = aws_utils.get_first_api_gateway(api_gateway_client, resource_name)
- if api is not None:
- _logger.info(" Tearing down AWS API Gateway...")
- aws_utils.delete_api_gateway(api_gateway_client, api["ApiId"])
-
- def get_feature_server_endpoint(self) -> Optional[str]:
- project = self.repo_config.project
- resource_name = _get_lambda_name(project)
- api_gateway_client = boto3.client("apigatewayv2")
- api = aws_utils.get_first_api_gateway(api_gateway_client, resource_name)
-
- if not api:
- return None
-
- api_id = api["ApiId"]
- lambda_client = boto3.client("lambda")
- region = lambda_client.meta.region_name
- return f"https://{api_id}.execute-api.{region}.amazonaws.com"
-
- def _upload_docker_image(
- self, ecr_client, repository_uri: str, docker_image_version: str
- ) -> str:
- """
- Pulls the AWS Lambda docker image from Dockerhub and uploads it to AWS ECR.
-
- Returns:
- The URI of the uploaded docker image.
- """
- try:
- import docker
- from docker.errors import APIError
- except ImportError as e:
- from feast.errors import FeastExtrasDependencyImportError
-
- raise FeastExtrasDependencyImportError("docker", str(e))
-
- try:
- docker_client = docker.from_env()
- except APIError:
- from feast.errors import DockerDaemonNotRunning
-
- raise DockerDaemonNotRunning()
-
- dockerhub_image = f"{AWS_LAMBDA_FEATURE_SERVER_IMAGE}:{docker_image_version}"
- _logger.info(
- f"Pulling remote image {Style.BRIGHT + Fore.GREEN}{dockerhub_image}{Style.RESET_ALL}"
- )
- for line in docker_client.api.pull(dockerhub_image, stream=True, decode=True):
- _logger.debug(f" {line}")
-
- auth_token = ecr_client.get_authorization_token()["authorizationData"][0][
- "authorizationToken"
- ]
- username, password = base64.b64decode(auth_token).decode("utf-8").split(":")
-
- ecr_address = repository_uri.split("/")[0]
- _logger.info(
- f"Logging in Docker client to {Style.BRIGHT + Fore.GREEN}{ecr_address}{Style.RESET_ALL}"
- )
- login_status = docker_client.login(
- username=username, password=password, registry=ecr_address
- )
- _logger.debug(f" {login_status}")
-
- image = docker_client.images.get(dockerhub_image)
- image_remote_name = f"{repository_uri}:{docker_image_version}"
- _logger.info(
- f"Pushing local image to remote {Style.BRIGHT + Fore.GREEN}{image_remote_name}{Style.RESET_ALL}"
- )
- image.tag(image_remote_name)
- for line in docker_client.api.push(
- repository_uri, tag=docker_image_version, stream=True, decode=True
- ):
- _logger.debug(f" {line}")
-
- return image_remote_name
-
- def _create_or_get_repository_uri(self, ecr_client):
- try:
- return ecr_client.describe_repositories(
- repositoryNames=[AWS_LAMBDA_FEATURE_SERVER_REPOSITORY]
- )["repositories"][0]["repositoryUri"]
- except ecr_client.exceptions.RepositoryNotFoundException:
- _logger.info(
- f"Creating remote ECR repository {Style.BRIGHT + Fore.GREEN}{AWS_LAMBDA_FEATURE_SERVER_REPOSITORY}{Style.RESET_ALL}"
- )
- response = ecr_client.create_repository(
- repositoryName=AWS_LAMBDA_FEATURE_SERVER_REPOSITORY
- )
- return response["repository"]["repositoryUri"]
-
-
-def _get_lambda_name(project: str):
- lambda_prefix = AWS_LAMBDA_FEATURE_SERVER_REPOSITORY
- lambda_suffix = f"{project}-{_get_docker_image_version().replace('.', '_')}"
- # AWS Lambda name can't have the length greater than 64 bytes.
- # This usually occurs during integration tests where feast version is long
- if len(lambda_prefix) + len(lambda_suffix) >= 63:
- lambda_suffix = hashlib.md5(lambda_suffix.encode()).hexdigest()
- return f"{lambda_prefix}-{lambda_suffix}"
-
-
-def _get_docker_image_version() -> str:
- """Returns a version for the feature server Docker image.
-
- If the feast.constants.DOCKER_IMAGE_TAG_ENV_NAME environment variable is set,
- we return that (mostly used for integration tests, but can be used for local testing too).
-
- For public Feast releases this equals to the Feast SDK version modified by replacing "." with "_".
- For example, Feast SDK version "0.14.1" would correspond to Docker image version "0_14_1".
-
- During development (when Feast is installed in editable mode) this equals to the Feast SDK version
- modified by removing the "dev..." suffix and replacing "." with "_". For example, Feast SDK version
- "0.14.1.dev41+g1cbfa225.d20211103" would correspond to Docker image version "0_14_1". This way,
- Feast SDK will use an already existing Docker image built during the previous public release.
-
- """
- tag = os.environ.get(DOCKER_IMAGE_TAG_ENV_NAME)
- if tag is not None:
- return tag
- else:
- version = get_version()
- if "dev" in version:
- version = version[: version.find("dev") - 1]
- _logger.warning(
- "You are trying to use AWS Lambda feature server while Feast is in a development mode. "
- f"Feast will use a docker image version {version} derived from Feast SDK "
- f"version {get_version()}. If you want to update the Feast SDK version, make "
- "sure to first fetch all new release tags from Github and then reinstall the library:\n"
- "> git fetch --all --tags\n"
- "> pip install -e '.'"
- )
- return version
diff --git a/sdk/python/feast/infra/contrib/azure_provider.py b/sdk/python/feast/infra/contrib/azure_provider.py
deleted file mode 100644
index ac56a2b33e2..00000000000
--- a/sdk/python/feast/infra/contrib/azure_provider.py
+++ /dev/null
@@ -1,72 +0,0 @@
-# Copyright (c) Microsoft Corporation.
-# Licensed under the MIT license.
-from datetime import datetime
-from typing import Callable
-
-from tqdm import tqdm
-
-from feast.feature_view import FeatureView
-from feast.infra.passthrough_provider import PassthroughProvider
-from feast.infra.registry.base_registry import BaseRegistry
-from feast.repo_config import RepoConfig
-from feast.utils import (
- _convert_arrow_to_proto,
- _get_column_names,
- _run_pyarrow_field_mapping,
-)
-
-DEFAULT_BATCH_SIZE = 10_000
-
-
-class AzureProvider(PassthroughProvider):
- def materialize_single_feature_view(
- self,
- config: RepoConfig,
- feature_view: FeatureView,
- start_date: datetime,
- end_date: datetime,
- registry: BaseRegistry,
- project: str,
- tqdm_builder: Callable[[int], tqdm],
- ) -> None:
- # TODO(kevjumba): untested
- entities = []
- for entity_name in feature_view.entities:
- entities.append(registry.get_entity(entity_name, project))
-
- (
- join_key_columns,
- feature_name_columns,
- event_timestamp_column,
- created_timestamp_column,
- ) = _get_column_names(feature_view, entities)
-
- offline_job = self.offline_store.pull_latest_from_table_or_query(
- config=config,
- data_source=feature_view.batch_source,
- join_key_columns=join_key_columns,
- feature_name_columns=feature_name_columns,
- timestamp_field=event_timestamp_column,
- created_timestamp_column=created_timestamp_column,
- start_date=start_date,
- end_date=end_date,
- )
-
- table = offline_job.to_arrow()
-
- if feature_view.batch_source.field_mapping is not None:
- table = _run_pyarrow_field_mapping(
- table, feature_view.batch_source.field_mapping
- )
-
- join_keys = {entity.join_key: entity.value_type for entity in entities}
-
- with tqdm_builder(table.num_rows) as pbar:
- for batch in table.to_batches(DEFAULT_BATCH_SIZE):
- rows_to_write = _convert_arrow_to_proto(batch, feature_view, join_keys)
- self.online_write_batch(
- self.repo_config,
- feature_view,
- rows_to_write,
- lambda x: pbar.update(x),
- )
diff --git a/sdk/python/feast/infra/feature_servers/aws_lambda/Dockerfile b/sdk/python/feast/infra/feature_servers/aws_lambda/Dockerfile
deleted file mode 100644
index 929227a8106..00000000000
--- a/sdk/python/feast/infra/feature_servers/aws_lambda/Dockerfile
+++ /dev/null
@@ -1,26 +0,0 @@
-FROM public.ecr.aws/lambda/python:3.9
-
-RUN yum install -y git
-
-
-# Copy app handler code
-COPY sdk/python/feast/infra/feature_servers/aws_lambda/app.py ${LAMBDA_TASK_ROOT}
-
-# Copy necessary parts of the Feast codebase
-COPY sdk/python sdk/python
-COPY protos protos
-COPY go go
-COPY setup.py setup.py
-COPY pyproject.toml pyproject.toml
-COPY README.md README.md
-
-# Install Feast for AWS with Lambda dependencies
-# We need this mount thingy because setuptools_scm needs access to the
-# git dir to infer the version of feast we're installing.
-# https://github.com/pypa/setuptools_scm#usage-from-docker
-# I think it also assumes that this dockerfile is being built from the root of the directory.
-RUN --mount=source=.git,target=.git,type=bind pip3 install --no-cache-dir -e '.[aws,redis]'
-RUN pip3 install -r sdk/python/feast/infra/feature_servers/aws_lambda/requirements.txt --target "${LAMBDA_TASK_ROOT}"
-
-# Set the CMD to your handler (could also be done as a parameter override outside of the Dockerfile)
-CMD [ "app.handler" ]
diff --git a/sdk/python/feast/infra/feature_servers/aws_lambda/__init__.py b/sdk/python/feast/infra/feature_servers/aws_lambda/__init__.py
deleted file mode 100644
index e69de29bb2d..00000000000
diff --git a/sdk/python/feast/infra/feature_servers/aws_lambda/app.py b/sdk/python/feast/infra/feature_servers/aws_lambda/app.py
deleted file mode 100644
index e90364ed680..00000000000
--- a/sdk/python/feast/infra/feature_servers/aws_lambda/app.py
+++ /dev/null
@@ -1,27 +0,0 @@
-import base64
-import os
-import tempfile
-from pathlib import Path
-
-from mangum import Mangum
-
-from feast import FeatureStore
-from feast.constants import FEATURE_STORE_YAML_ENV_NAME
-from feast.feature_server import get_app
-
-# Load RepoConfig
-config_base64 = os.environ[FEATURE_STORE_YAML_ENV_NAME]
-config_bytes = base64.b64decode(config_base64)
-
-# Create a new unique directory for writing feature_store.yaml
-repo_path = Path(tempfile.mkdtemp())
-
-with open(repo_path / "feature_store.yaml", "wb") as f:
- f.write(config_bytes)
-
-# Initialize the feature store
-store = FeatureStore(repo_path=str(repo_path.resolve()))
-
-# Create the FastAPI app and AWS Lambda handler
-app = get_app(store)
-handler = Mangum(app)
diff --git a/sdk/python/feast/infra/feature_servers/aws_lambda/config.py b/sdk/python/feast/infra/feature_servers/aws_lambda/config.py
deleted file mode 100644
index 946831a18fb..00000000000
--- a/sdk/python/feast/infra/feature_servers/aws_lambda/config.py
+++ /dev/null
@@ -1,21 +0,0 @@
-from typing import Literal
-
-from pydantic import StrictBool, StrictStr
-
-from feast.infra.feature_servers.base_config import BaseFeatureServerConfig
-
-
-class AwsLambdaFeatureServerConfig(BaseFeatureServerConfig):
- """Feature server config for AWS Lambda."""
-
- type: Literal["aws_lambda"] = "aws_lambda"
- """Feature server type selector."""
-
- public: StrictBool = True
- """Whether the endpoint should be publicly accessible."""
-
- auth: Literal["none", "api-key"] = "none"
- """Authentication method for the endpoint."""
-
- execution_role_name: StrictStr
- """The execution role for the AWS Lambda function."""
diff --git a/sdk/python/feast/infra/feature_servers/aws_lambda/requirements.txt b/sdk/python/feast/infra/feature_servers/aws_lambda/requirements.txt
deleted file mode 100644
index 845aa14802d..00000000000
--- a/sdk/python/feast/infra/feature_servers/aws_lambda/requirements.txt
+++ /dev/null
@@ -1,2 +0,0 @@
-fastapi
-mangum
diff --git a/sdk/python/feast/infra/feature_servers/gcp_cloudrun/Dockerfile b/sdk/python/feast/infra/feature_servers/gcp_cloudrun/Dockerfile
deleted file mode 100644
index 6b89d4f73c1..00000000000
--- a/sdk/python/feast/infra/feature_servers/gcp_cloudrun/Dockerfile
+++ /dev/null
@@ -1,32 +0,0 @@
-FROM python:3.11-slim
-
-RUN apt-get update && apt-get install -y git
-
-# Allow statements and log messages to immediately appear in the Knative logs
-ENV PYTHONUNBUFFERED True
-
-# Copy local code to the container image.
-ENV APP_HOME /app
-WORKDIR $APP_HOME
-
-# Copy app handler code
-COPY sdk/python/feast/infra/feature_servers/gcp_cloudrun/app.py ./app.py
-
-# Copy necessary parts of the Feast codebase
-COPY sdk/python ./sdk/python
-COPY protos ./protos
-COPY setup.py setup.py
-COPY pyproject.toml pyproject.toml
-COPY README.md ./README.md
-
-# Install production dependencies.
-RUN --mount=source=.git,target=.git,type=bind pip install --no-cache-dir \
- -e '.[gcp,redis]' \
- -r ./sdk/python/feast/infra/feature_servers/gcp_cloudrun/requirements.txt
-
-# Run the web service on container startup. Here we use the gunicorn
-# webserver, with one worker process and 8 threads.
-# For environments with multiple CPU cores, increase the number of workers
-# to be equal to the cores available.
-# Timeout is set to 0 to disable the timeouts of the workers to allow Cloud Run to handle instance scaling.
-CMD exec gunicorn -k uvicorn.workers.UvicornWorker --bind :$PORT --workers 1 --threads 8 --timeout 0 app:app
diff --git a/sdk/python/feast/infra/feature_servers/gcp_cloudrun/__init__.py b/sdk/python/feast/infra/feature_servers/gcp_cloudrun/__init__.py
deleted file mode 100644
index e69de29bb2d..00000000000
diff --git a/sdk/python/feast/infra/feature_servers/gcp_cloudrun/app.py b/sdk/python/feast/infra/feature_servers/gcp_cloudrun/app.py
deleted file mode 100644
index 06749b0cd39..00000000000
--- a/sdk/python/feast/infra/feature_servers/gcp_cloudrun/app.py
+++ /dev/null
@@ -1,24 +0,0 @@
-import base64
-import os
-import tempfile
-from pathlib import Path
-
-from feast import FeatureStore
-from feast.constants import FEATURE_STORE_YAML_ENV_NAME
-from feast.feature_server import get_app
-
-# Load RepoConfig
-config_base64 = os.environ[FEATURE_STORE_YAML_ENV_NAME]
-config_bytes = base64.b64decode(config_base64)
-
-# Create a new unique directory for writing feature_store.yaml
-repo_path = Path(tempfile.mkdtemp())
-
-with open(repo_path / "feature_store.yaml", "wb") as f:
- f.write(config_bytes)
-
-# Initialize the feature store
-store = FeatureStore(repo_path=str(repo_path.resolve()))
-
-# Create the FastAPI app
-app = get_app(store)
diff --git a/sdk/python/feast/infra/feature_servers/gcp_cloudrun/config.py b/sdk/python/feast/infra/feature_servers/gcp_cloudrun/config.py
deleted file mode 100644
index ddcbde7924a..00000000000
--- a/sdk/python/feast/infra/feature_servers/gcp_cloudrun/config.py
+++ /dev/null
@@ -1,18 +0,0 @@
-from typing import Literal
-
-from pydantic import StrictBool
-
-from feast.infra.feature_servers.base_config import BaseFeatureServerConfig
-
-
-class GcpCloudRunFeatureServerConfig(BaseFeatureServerConfig):
- """Feature server config for GCP CloudRun."""
-
- type: Literal["gcp_cloudrun"] = "gcp_cloudrun"
- """Feature server type selector."""
-
- public: StrictBool = True
- """Whether the endpoint should be publicly accessible."""
-
- auth: Literal["none", "api-key"] = "none"
- """Authentication method for the endpoint."""
diff --git a/sdk/python/feast/infra/feature_servers/gcp_cloudrun/requirements.txt b/sdk/python/feast/infra/feature_servers/gcp_cloudrun/requirements.txt
deleted file mode 100644
index 8f22dccf99a..00000000000
--- a/sdk/python/feast/infra/feature_servers/gcp_cloudrun/requirements.txt
+++ /dev/null
@@ -1 +0,0 @@
-gunicorn
diff --git a/sdk/python/feast/infra/gcp.py b/sdk/python/feast/infra/gcp.py
deleted file mode 100644
index 512378237a6..00000000000
--- a/sdk/python/feast/infra/gcp.py
+++ /dev/null
@@ -1,9 +0,0 @@
-from feast.infra.passthrough_provider import PassthroughProvider
-
-
-class GcpProvider(PassthroughProvider):
- """
- This class only exists for backwards compatibility.
- """
-
- pass
diff --git a/sdk/python/feast/infra/key_encoding_utils.py b/sdk/python/feast/infra/key_encoding_utils.py
index ca834f19176..1f9ffeef140 100644
--- a/sdk/python/feast/infra/key_encoding_utils.py
+++ b/sdk/python/feast/infra/key_encoding_utils.py
@@ -20,7 +20,23 @@ def _serialize_val(
return struct.pack(" ValueProto:
+ if value_type == ValueType.INT64:
+ value = struct.unpack(" bytes:
@@ -50,6 +66,15 @@ def serialize_entity_key(
serialize to the same byte string[1].
[1] https://developers.google.com/protocol-buffers/docs/encoding
+
+ Args:
+ entity_key_serialization_version: version of the entity key serialization
+ version 1: int64 values are serialized as 4 bytes
+ version 2: int64 values are serialized as 8 bytes
+ version 3: entity_key size is added to the serialization for deserialization purposes
+ entity_key: EntityKeyProto
+
+ Returns: bytes of the serialized entity key
"""
sorted_keys, sorted_values = zip(
*sorted(zip(entity_key.join_keys, entity_key.entity_values))
@@ -58,6 +83,8 @@ def serialize_entity_key(
output: List[bytes] = []
for k in sorted_keys:
output.append(struct.pack(" 2:
+ output.append(struct.pack(" EntityKeyProto:
+ """
+ Deserialize entity key from a bytestring. This function can only be used with entity_key_serialization_version > 2.
+ Args:
+ entity_key_serialization_version: version of the entity key serialization
+ serialized_entity_key: serialized entity key bytes
+
+ Returns: EntityKeyProto
+
+ """
+ if entity_key_serialization_version <= 2:
+ raise ValueError(
+ "Deserialization of entity key with version <= 2 is not supported. Please use version > 2 by setting entity_key_serialization_version=3"
+ )
+ offset = 0
+ keys = []
+ values = []
+ while offset < len(serialized_entity_key):
+ key_type = struct.unpack_from(" Infra:
- infra = Infra()
- if self.online_store:
- infra_objects: List[InfraObject] = self.online_store.plan(
- config, desired_registry_proto
- )
- infra.infra_objects += infra_objects
- return infra
diff --git a/sdk/python/feast/infra/materialization/kubernetes/k8s_materialization_engine.py b/sdk/python/feast/infra/materialization/kubernetes/k8s_materialization_engine.py
index 2e7129b0376..510b6b4e4c7 100644
--- a/sdk/python/feast/infra/materialization/kubernetes/k8s_materialization_engine.py
+++ b/sdk/python/feast/infra/materialization/kubernetes/k8s_materialization_engine.py
@@ -306,7 +306,9 @@ def _create_kubernetes_job(self, job_id, paths, feature_view):
def _create_configuration_map(self, job_id, paths, feature_view, namespace):
"""Create a Kubernetes configmap for this job"""
- feature_store_configuration = yaml.dump(self.repo_config.dict(by_alias=True))
+ feature_store_configuration = yaml.dump(
+ self.repo_config.model_dump(by_alias=True)
+ )
materialization_config = yaml.dump(
{"paths": paths, "feature_view": feature_view.name}
diff --git a/sdk/python/feast/infra/offline_stores/bigquery.py b/sdk/python/feast/infra/offline_stores/bigquery.py
index 36334b606d4..ef12eba442b 100644
--- a/sdk/python/feast/infra/offline_stores/bigquery.py
+++ b/sdk/python/feast/infra/offline_stores/bigquery.py
@@ -45,7 +45,7 @@
from feast.on_demand_feature_view import OnDemandFeatureView
from feast.repo_config import FeastConfigBaseModel, RepoConfig
from feast.saved_dataset import SavedDatasetStorage
-from feast.utils import get_user_agent
+from feast.utils import _utc_now, get_user_agent
from .bigquery_source import (
BigQueryLoggingDestination,
@@ -59,7 +59,6 @@
from google.auth.exceptions import DefaultCredentialsError
from google.cloud import bigquery
from google.cloud.bigquery import Client, SchemaField, Table
- from google.cloud.bigquery._pandas_helpers import ARROW_SCALAR_IDS_TO_BQ
from google.cloud.storage import Client as StorageClient
except ImportError as e:
@@ -67,6 +66,16 @@
raise FeastExtrasDependencyImportError("gcp", str(e))
+try:
+ from google.cloud.bigquery._pyarrow_helpers import _ARROW_SCALAR_IDS_TO_BQ
+except ImportError:
+ try:
+ from google.cloud.bigquery._pandas_helpers import ( # type: ignore
+ ARROW_SCALAR_IDS_TO_BQ as _ARROW_SCALAR_IDS_TO_BQ,
+ )
+ except ImportError as e:
+ raise FeastExtrasDependencyImportError("gcp", str(e))
+
def get_http_client_info():
return http_client_info.ClientInfo(user_agent=get_user_agent())
@@ -701,7 +710,7 @@ def _upload_entity_df(
# Ensure that the table expires after some time
table = client.get_table(table=table_name)
- table.expires = datetime.utcnow() + timedelta(minutes=30)
+ table.expires = _utc_now() + timedelta(minutes=30)
client.update_table(table, ["expires"])
return table
@@ -794,10 +803,10 @@ def arrow_schema_to_bq_schema(arrow_schema: pyarrow.Schema) -> List[SchemaField]
for field in arrow_schema:
if pyarrow.types.is_list(field.type):
detected_mode = "REPEATED"
- detected_type = ARROW_SCALAR_IDS_TO_BQ[field.type.value_type.id]
+ detected_type = _ARROW_SCALAR_IDS_TO_BQ[field.type.value_type.id]
else:
detected_mode = "NULLABLE"
- detected_type = ARROW_SCALAR_IDS_TO_BQ[field.type.id]
+ detected_type = _ARROW_SCALAR_IDS_TO_BQ[field.type.id]
bq_schema.append(
SchemaField(name=field.name, field_type=detected_type, mode=detected_mode)
diff --git a/sdk/python/feast/infra/offline_stores/contrib/mssql_offline_store/mssql.py b/sdk/python/feast/infra/offline_stores/contrib/mssql_offline_store/mssql.py
index 5fe58571466..875d584568b 100644
--- a/sdk/python/feast/infra/offline_stores/contrib/mssql_offline_store/mssql.py
+++ b/sdk/python/feast/infra/offline_stores/contrib/mssql_offline_store/mssql.py
@@ -1,44 +1,109 @@
-# Copyright (c) Microsoft Corporation.
-# Licensed under the MIT license.
-import warnings
-from datetime import datetime
+from datetime import datetime, timedelta
from pathlib import Path
-from typing import Any, Callable, Dict, List, Literal, Optional, Set, Tuple, Union
+from typing import Any, Callable, Iterable, List, Literal, Optional, Tuple, Union
+from urllib import parse
-import numpy as np
-import pandas
+import ibis
+import pandas as pd
import pyarrow
-import pyarrow as pa
-import sqlalchemy
-from pydantic.types import StrictStr
-from sqlalchemy import create_engine
-from sqlalchemy.engine import Engine
-from sqlalchemy.orm import sessionmaker
+from ibis.expr.types import Table
+from pydantic import StrictStr
-from feast import FileSource, errors
from feast.data_source import DataSource
-from feast.errors import InvalidEntityType
from feast.feature_logging import LoggingConfig, LoggingSource
from feast.feature_view import FeatureView
-from feast.infra.offline_stores import offline_utils
-from feast.infra.offline_stores.file_source import SavedDatasetFileStorage
-from feast.infra.offline_stores.offline_store import OfflineStore, RetrievalMetadata
-from feast.infra.offline_stores.offline_utils import (
- DEFAULT_ENTITY_DF_EVENT_TIMESTAMP_COL,
- build_point_in_time_query,
- get_feature_view_query_context,
+from feast.infra.offline_stores.contrib.mssql_offline_store.mssqlserver_source import (
+ MsSqlServerSource,
)
-from feast.infra.provider import RetrievalJob
+from feast.infra.offline_stores.ibis import (
+ get_historical_features_ibis,
+ offline_write_batch_ibis,
+ pull_all_from_table_or_query_ibis,
+ pull_latest_from_table_or_query_ibis,
+ write_logged_features_ibis,
+)
+from feast.infra.offline_stores.offline_store import OfflineStore, RetrievalJob
from feast.infra.registry.base_registry import BaseRegistry
-from feast.on_demand_feature_view import OnDemandFeatureView
from feast.repo_config import FeastConfigBaseModel, RepoConfig
-from feast.saved_dataset import SavedDatasetStorage
-from feast.type_map import pa_to_mssql_type
-# Make sure warning doesn't raise more than once.
-warnings.simplefilter("once", RuntimeWarning)
-EntitySchema = Dict[str, np.dtype]
+def get_ibis_connection(config: RepoConfig):
+ connection_params = parse.urlparse(config.offline_store.connection_string)
+ additional_kwargs = dict(parse.parse_qsl(connection_params.query))
+ return ibis.mssql.connect(
+ user=connection_params.username,
+ password=connection_params.password,
+ host=connection_params.hostname,
+ port=connection_params.port,
+ database=connection_params.path.strip("/"),
+ **additional_kwargs,
+ )
+
+
+def get_table_column_names_and_types(
+ config: RepoConfig, data_source: MsSqlServerSource
+) -> Iterable[Tuple[str, str]]:
+ con = get_ibis_connection(config)
+
+ # assert isinstance(config.offline_store, MsSqlServerOfflineStoreConfig)
+ # conn = create_engine(config.offline_store.connection_string)
+ # self._mssqlserver_options.connection_str = (
+ # config.offline_store.connection_string
+ # )
+ name_type_pairs = []
+ if len(data_source.table_ref.split(".")) == 2:
+ database, table_name = data_source.table_ref.split(".")
+ columns_query = f"""
+ SELECT COLUMN_NAME, DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS
+ WHERE TABLE_NAME = '{table_name}' and table_schema = '{database}'
+ """
+ else:
+ columns_query = f"""
+ SELECT COLUMN_NAME, DATA_TYPE FROM INFORMATION_SCHEMA.COLUMNS
+ WHERE TABLE_NAME = '{data_source.table_ref}'
+ """
+
+ table_schema = con.sql(columns_query).execute()
+
+ name_type_pairs.extend(
+ list(
+ zip(
+ table_schema["COLUMN_NAME"].to_list(),
+ table_schema["DATA_TYPE"].to_list(),
+ )
+ )
+ )
+ return name_type_pairs
+
+
+def _build_data_source_reader(config: RepoConfig):
+ con = get_ibis_connection(config)
+
+ def _read_data_source(data_source: DataSource) -> Table:
+ assert isinstance(data_source, MsSqlServerSource)
+ return con.table(data_source.table_ref)
+
+ return _read_data_source
+
+
+def _build_data_source_writer(config: RepoConfig):
+ con = get_ibis_connection(config)
+
+ def _write_data_source(
+ table: Table,
+ data_source: DataSource,
+ mode: str = "append",
+ allow_overwrite: bool = False,
+ ):
+ assert isinstance(data_source, MsSqlServerSource)
+ con.insert(table_name=data_source.table_ref, obj=table.to_pandas())
+
+ return _write_data_source
+
+
+def mssql_event_expire_timestamp_fn(timestamp_field: str, ttl: timedelta) -> str:
+ ttl_seconds = int(ttl.total_seconds())
+ return f"DATEADD(ss, {ttl_seconds}, {timestamp_field})"
class MsSqlServerOfflineStoreConfig(FeastConfigBaseModel):
@@ -52,18 +117,7 @@ class MsSqlServerOfflineStoreConfig(FeastConfigBaseModel):
format: SQLAlchemy connection string, e.g. mssql+pyodbc://sa:yourStrong(!)Password@localhost:1433/feast_test?driver=ODBC+Driver+17+for+SQL+Server"""
-def make_engine(config: MsSqlServerOfflineStoreConfig) -> Engine:
- return create_engine(config.connection_string)
-
-
class MsSqlServerOfflineStore(OfflineStore):
- """
- Microsoft SQL Server based offline store, supporting Azure Synapse or Azure SQL.
-
- Note: to use this, you'll need to have Microsoft ODBC 17 installed.
- See https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/install-microsoft-odbc-driver-sql-server-macos?view=sql-server-ver15#17
- """
-
@staticmethod
def pull_latest_from_table_or_query(
config: RepoConfig,
@@ -75,43 +129,45 @@ def pull_latest_from_table_or_query(
start_date: datetime,
end_date: datetime,
) -> RetrievalJob:
- warnings.warn(
- "The Azure Synapse + Azure SQL offline store is an experimental feature in alpha development. "
- "Some functionality may still be unstable so functionality can change in the future.",
- RuntimeWarning,
+ return pull_latest_from_table_or_query_ibis(
+ config=config,
+ data_source=data_source,
+ join_key_columns=join_key_columns,
+ feature_name_columns=feature_name_columns,
+ timestamp_field=timestamp_field,
+ created_timestamp_column=created_timestamp_column,
+ start_date=start_date,
+ end_date=end_date,
+ data_source_reader=_build_data_source_reader(config),
+ data_source_writer=_build_data_source_writer(config),
)
- assert type(data_source).__name__ == "MsSqlServerSource"
- from_expression = data_source.get_table_query_string().replace("`", "")
-
- partition_by_join_key_string = ", ".join(join_key_columns)
- if partition_by_join_key_string != "":
- partition_by_join_key_string = (
- "PARTITION BY " + partition_by_join_key_string
- )
- timestamps = [timestamp_field]
- if created_timestamp_column:
- timestamps.append(created_timestamp_column)
- timestamp_desc_string = " DESC, ".join(timestamps) + " DESC"
- field_string = ", ".join(join_key_columns + feature_name_columns + timestamps)
-
- query = f"""
- SELECT {field_string}
- FROM (
- SELECT {field_string},
- ROW_NUMBER() OVER({partition_by_join_key_string} ORDER BY {timestamp_desc_string}) AS _feast_row
- FROM {from_expression} inner_t
- WHERE {timestamp_field} BETWEEN CONVERT(DATETIMEOFFSET, '{start_date}', 120) AND CONVERT(DATETIMEOFFSET, '{end_date}', 120)
- ) outer_t
- WHERE outer_t._feast_row = 1
- """
- engine = make_engine(config.offline_store)
- return MsSqlServerRetrievalJob(
- query=query,
- engine=engine,
- config=config.offline_store,
- full_feature_names=False,
- on_demand_feature_views=None,
+ @staticmethod
+ def get_historical_features(
+ config: RepoConfig,
+ feature_views: List[FeatureView],
+ feature_refs: List[str],
+ entity_df: Union[pd.DataFrame, str],
+ registry: BaseRegistry,
+ project: str,
+ full_feature_names: bool = False,
+ ) -> RetrievalJob:
+ # TODO avoid this conversion
+ if type(entity_df) == str:
+ con = get_ibis_connection(config)
+ entity_df = con.sql(entity_df).execute()
+
+ return get_historical_features_ibis(
+ config=config,
+ feature_views=feature_views,
+ feature_refs=feature_refs,
+ entity_df=entity_df,
+ registry=registry,
+ project=project,
+ full_feature_names=full_feature_names,
+ data_source_reader=_build_data_source_reader(config),
+ data_source_writer=_build_data_source_writer(config),
+ event_expire_timestamp_fn=mssql_event_expire_timestamp_fn,
)
@staticmethod
@@ -124,114 +180,32 @@ def pull_all_from_table_or_query(
start_date: datetime,
end_date: datetime,
) -> RetrievalJob:
- assert type(data_source).__name__ == "MsSqlServerSource"
- warnings.warn(
- "The Azure Synapse + Azure SQL offline store is an experimental feature in alpha development. "
- "Some functionality may still be unstable so functionality can change in the future.",
- RuntimeWarning,
- )
- from_expression = data_source.get_table_query_string().replace("`", "")
- timestamps = [timestamp_field]
- field_string = ", ".join(join_key_columns + feature_name_columns + timestamps)
-
- query = f"""
- SELECT {field_string}
- FROM (
- SELECT {field_string}
- FROM {from_expression}
- WHERE {timestamp_field} BETWEEN TIMESTAMP '{start_date}' AND TIMESTAMP '{end_date}'
- )
- """
- engine = make_engine(config.offline_store)
-
- return MsSqlServerRetrievalJob(
- query=query,
- engine=engine,
- config=config.offline_store,
- full_feature_names=False,
- on_demand_feature_views=None,
+ return pull_all_from_table_or_query_ibis(
+ config=config,
+ data_source=data_source,
+ join_key_columns=join_key_columns,
+ feature_name_columns=feature_name_columns,
+ timestamp_field=timestamp_field,
+ start_date=start_date,
+ end_date=end_date,
+ data_source_reader=_build_data_source_reader(config),
+ data_source_writer=_build_data_source_writer(config),
)
@staticmethod
- def get_historical_features(
+ def offline_write_batch(
config: RepoConfig,
- feature_views: List[FeatureView],
- feature_refs: List[str],
- entity_df: Union[pandas.DataFrame, str],
- registry: BaseRegistry,
- project: str,
- full_feature_names: bool = False,
- ) -> RetrievalJob:
- warnings.warn(
- "The Azure Synapse + Azure SQL offline store is an experimental feature in alpha development. "
- "Some functionality may still be unstable so functionality can change in the future.",
- RuntimeWarning,
- )
-
- expected_join_keys = _get_join_keys(project, feature_views, registry)
- assert isinstance(config.offline_store, MsSqlServerOfflineStoreConfig)
- engine = make_engine(config.offline_store)
- if isinstance(entity_df, pandas.DataFrame):
- entity_df_event_timestamp_col = (
- offline_utils.infer_event_timestamp_from_entity_df(
- dict(zip(list(entity_df.columns), list(entity_df.dtypes)))
- )
- )
- entity_df[entity_df_event_timestamp_col] = pandas.to_datetime(
- entity_df[entity_df_event_timestamp_col], utc=True
- ).fillna(pandas.Timestamp.now())
-
- elif isinstance(entity_df, str):
- raise ValueError(
- "string entities are currently not supported in the MsSQL offline store."
- )
- (
- table_schema,
- table_name,
- ) = _upload_entity_df_into_sqlserver_and_get_entity_schema(
- engine, config, entity_df, full_feature_names=full_feature_names
- )
-
- _assert_expected_columns_in_sqlserver(
- expected_join_keys,
- entity_df_event_timestamp_col,
- table_schema,
- )
-
- entity_df_event_timestamp_range = _get_entity_df_event_timestamp_range(
- entity_df,
- entity_df_event_timestamp_col,
- engine,
- )
-
- # Build a query context containing all information required to template the SQL query
- query_context = get_feature_view_query_context(
- feature_refs,
- feature_views,
- registry,
- project,
- entity_df_timestamp_range=entity_df_event_timestamp_range,
- )
-
- # Generate the SQL query from the query context
- query = build_point_in_time_query(
- query_context,
- left_table_query_string=table_name,
- entity_df_event_timestamp_col=entity_df_event_timestamp_col,
- entity_df_columns=table_schema.keys(),
- full_feature_names=full_feature_names,
- query_template=MULTIPLE_FEATURE_VIEW_POINT_IN_TIME_JOIN,
- )
- query = query.replace("`", "")
-
- job = MsSqlServerRetrievalJob(
- query=query,
- engine=engine,
- config=config.offline_store,
- full_feature_names=full_feature_names,
- on_demand_feature_views=registry.list_on_demand_feature_views(project),
+ feature_view: FeatureView,
+ table: pyarrow.Table,
+ progress: Optional[Callable[[int], Any]],
+ ):
+ offline_write_batch_ibis(
+ config=config,
+ feature_view=feature_view,
+ table=table,
+ progress=progress,
+ data_source_writer=_build_data_source_writer(config),
)
- return job
@staticmethod
def write_logged_features(
@@ -241,410 +215,10 @@ def write_logged_features(
logging_config: LoggingConfig,
registry: BaseRegistry,
):
- raise NotImplementedError()
-
- @staticmethod
- def offline_write_batch(
- config: RepoConfig,
- feature_view: FeatureView,
- table: pyarrow.Table,
- progress: Optional[Callable[[int], Any]],
- ):
- raise NotImplementedError()
-
-
-def _assert_expected_columns_in_dataframe(
- join_keys: Set[str], entity_df_event_timestamp_col: str, entity_df: pandas.DataFrame
-):
- entity_df_columns = set(entity_df.columns.values)
- expected_columns = join_keys.copy()
- expected_columns.add(entity_df_event_timestamp_col)
-
- missing_keys = expected_columns - entity_df_columns
-
- if len(missing_keys) != 0:
- raise errors.FeastEntityDFMissingColumnsError(expected_columns, missing_keys)
-
-
-def _assert_expected_columns_in_sqlserver(
- join_keys: Set[str], entity_df_event_timestamp_col: str, table_schema: EntitySchema
-):
- entity_columns = set(table_schema.keys())
- expected_columns = join_keys.copy()
- expected_columns.add(entity_df_event_timestamp_col)
-
- missing_keys = expected_columns - entity_columns
-
- if len(missing_keys) != 0:
- raise errors.FeastEntityDFMissingColumnsError(expected_columns, missing_keys)
-
-
-def _get_join_keys(
- project: str, feature_views: List[FeatureView], registry: BaseRegistry
-) -> Set[str]:
- join_keys = set()
- for feature_view in feature_views:
- entities = feature_view.entities
- for entity_name in entities:
- entity = registry.get_entity(entity_name, project)
- join_keys.add(entity.join_key)
- return join_keys
-
-
-def _infer_event_timestamp_from_sqlserver_schema(table_schema) -> str:
- if any(
- schema_field["COLUMN_NAME"] == DEFAULT_ENTITY_DF_EVENT_TIMESTAMP_COL
- for schema_field in table_schema
- ):
- return DEFAULT_ENTITY_DF_EVENT_TIMESTAMP_COL
- else:
- datetime_columns = list(
- filter(
- lambda schema_field: schema_field["DATA_TYPE"] == "DATETIMEOFFSET",
- table_schema,
- )
- )
- if len(datetime_columns) == 1:
- print(
- f"Using {datetime_columns[0]['COLUMN_NAME']} as the event timestamp. To specify a column explicitly, please name it {DEFAULT_ENTITY_DF_EVENT_TIMESTAMP_COL}."
- )
- return datetime_columns[0].name
- else:
- raise ValueError(
- f"Please provide an entity_df with a column named {DEFAULT_ENTITY_DF_EVENT_TIMESTAMP_COL} representing the time of events."
- )
-
-
-class MsSqlServerRetrievalJob(RetrievalJob):
- def __init__(
- self,
- query: str,
- engine: Engine,
- config: MsSqlServerOfflineStoreConfig,
- full_feature_names: bool,
- on_demand_feature_views: Optional[List[OnDemandFeatureView]] = None,
- metadata: Optional[RetrievalMetadata] = None,
- drop_columns: Optional[List[str]] = None,
- ):
- self.query = query
- self.engine = engine
- self._config = config
- self._full_feature_names = full_feature_names
- self._on_demand_feature_views = on_demand_feature_views or []
- self._drop_columns = drop_columns
- self._metadata = metadata
-
- @property
- def full_feature_names(self) -> bool:
- return self._full_feature_names
-
- @property
- def on_demand_feature_views(self) -> List[OnDemandFeatureView]:
- return self._on_demand_feature_views
-
- def _to_df_internal(self, timeout: Optional[int] = None) -> pandas.DataFrame:
- return pandas.read_sql(self.query, con=self.engine).fillna(value=np.nan)
-
- def _to_arrow_internal(self, timeout: Optional[int] = None) -> pyarrow.Table:
- result = pandas.read_sql(self.query, con=self.engine).fillna(value=np.nan)
- return pyarrow.Table.from_pandas(result)
-
- ## Implements persist in Feast 0.18 - This persists to filestorage
- ## ToDo: Persist to Azure Storage
- def persist(
- self,
- storage: SavedDatasetStorage,
- allow_overwrite: Optional[bool] = False,
- timeout: Optional[int] = None,
- ):
- assert isinstance(storage, SavedDatasetFileStorage)
-
- filesystem, path = FileSource.create_filesystem_and_path(
- storage.file_options.uri,
- storage.file_options.s3_endpoint_override,
+ write_logged_features_ibis(
+ config=config,
+ data=data,
+ source=source,
+ logging_config=logging_config,
+ registry=registry,
)
-
- if path.endswith(".parquet"):
- pyarrow.parquet.write_table(
- self.to_arrow(), where=path, filesystem=filesystem
- )
- else:
- # otherwise assume destination is directory
- pyarrow.parquet.write_to_dataset(
- self.to_arrow(), root_path=path, filesystem=filesystem
- )
-
- def supports_remote_storage_export(self) -> bool:
- return False
-
- def to_remote_storage(self) -> List[str]:
- raise NotImplementedError()
-
- @property
- def metadata(self) -> Optional[RetrievalMetadata]:
- return self._metadata
-
-
-def _upload_entity_df_into_sqlserver_and_get_entity_schema(
- engine: sqlalchemy.engine.Engine,
- config: RepoConfig,
- entity_df: Union[pandas.DataFrame, str],
- full_feature_names: bool,
-) -> Tuple[Dict[Any, Any], str]:
- """
- Uploads a Pandas entity dataframe into a SQL Server table and constructs the
- schema from the original entity_df dataframe.
- """
- table_id = offline_utils.get_temp_entity_table_name()
- session = sessionmaker(bind=engine)()
-
- if type(entity_df) is str:
- # TODO: This should be a temporary table, right?
- session.execute(f"SELECT * INTO {table_id} FROM ({entity_df}) t") # type: ignore
-
- session.commit()
-
- limited_entity_df = MsSqlServerRetrievalJob(
- f"SELECT TOP 1 * FROM {table_id}",
- engine,
- config.offline_store,
- full_feature_names=full_feature_names,
- on_demand_feature_views=None,
- ).to_df()
-
- entity_schema = (
- dict(zip(limited_entity_df.columns, limited_entity_df.dtypes)),
- table_id,
- )
-
- elif isinstance(entity_df, pandas.DataFrame):
- # Drop the index so that we don't have unnecessary columns
- engine.execute(_df_to_create_table_sql(entity_df, table_id)) # type: ignore
- entity_df.to_sql(name=table_id, con=engine, index=False, if_exists="append")
- entity_schema = dict(zip(entity_df.columns, entity_df.dtypes)), table_id
-
- else:
- raise ValueError(
- f"The entity dataframe you have provided must be a SQL Server SQL query,"
- f" or a Pandas dataframe. But we found: {type(entity_df)} "
- )
-
- return entity_schema
-
-
-def _df_to_create_table_sql(df: pandas.DataFrame, table_name: str) -> str:
- pa_table = pa.Table.from_pandas(df)
-
- columns = [f""""{f.name}" {pa_to_mssql_type(f.type)}""" for f in pa_table.schema]
-
- return f"""
- CREATE TABLE "{table_name}" (
- {", ".join(columns)}
- );
- """
-
-
-def _get_entity_df_event_timestamp_range(
- entity_df: Union[pandas.DataFrame, str],
- entity_df_event_timestamp_col: str,
- engine: Engine,
-) -> Tuple[datetime, datetime]:
- if isinstance(entity_df, pandas.DataFrame):
- entity_df_event_timestamp = entity_df.loc[
- :, entity_df_event_timestamp_col
- ].infer_objects()
- if pandas.api.types.is_string_dtype(entity_df_event_timestamp):
- entity_df_event_timestamp = pandas.to_datetime(
- entity_df_event_timestamp, utc=True
- )
- entity_df_event_timestamp_range = (
- entity_df_event_timestamp.min().to_pydatetime(),
- entity_df_event_timestamp.max().to_pydatetime(),
- )
- elif isinstance(entity_df, str):
- # If the entity_df is a string (SQL query), determine range
- # from table
- df = pandas.read_sql(entity_df, con=engine).fillna(value=np.nan)
- entity_df_event_timestamp = df.loc[
- :, entity_df_event_timestamp_col
- ].infer_objects()
- if pandas.api.types.is_string_dtype(entity_df_event_timestamp):
- entity_df_event_timestamp = pandas.to_datetime(
- entity_df_event_timestamp, utc=True
- )
- entity_df_event_timestamp_range = (
- entity_df_event_timestamp.min().to_pydatetime(),
- entity_df_event_timestamp.max().to_pydatetime(),
- )
- else:
- raise InvalidEntityType(type(entity_df))
-
- return entity_df_event_timestamp_range
-
-
-# TODO: Optimizations
-# * Use NEWID() instead of ROW_NUMBER(), or join on entity columns directly
-# * Precompute ROW_NUMBER() so that it doesn't have to be recomputed for every query on entity_dataframe
-# * Create temporary tables instead of keeping all tables in memory
-
-MULTIPLE_FEATURE_VIEW_POINT_IN_TIME_JOIN = """
-/*
- Compute a deterministic hash for the `left_table_query_string` that will be used throughout
- all the logic as the field to GROUP BY the data
-*/
-WITH entity_dataframe AS (
- SELECT *,
- {{entity_df_event_timestamp_col}} AS entity_timestamp
- {% for featureview in featureviews %}
- ,CONCAT(
- {% for entity_key in unique_entity_keys %}
- {{entity_key}},
- {% endfor %}
- {{entity_df_event_timestamp_col}}
- ) AS {{featureview.name}}__entity_row_unique_id
- {% endfor %}
- FROM {{ left_table_query_string }}
-),
-
-{% for featureview in featureviews %}
-
-{{ featureview.name }}__entity_dataframe AS (
- SELECT
- {{ featureview.entities | join(', ')}}{% if featureview.entities %},{% else %}{% endif %}
- entity_timestamp,
- {{featureview.name}}__entity_row_unique_id
- FROM entity_dataframe
- GROUP BY
- {{ featureview.entities | join(', ')}}{% if featureview.entities %},{% else %}{% endif %}
- entity_timestamp,
- {{featureview.name}}__entity_row_unique_id
-),
-
-/*
- This query template performs the point-in-time correctness join for a single feature set table
- to the provided entity table.
-
- 1. We first join the current feature_view to the entity dataframe that has been passed.
- This JOIN has the following logic:
- - For each row of the entity dataframe, only keep the rows where the timestamp_field`
- is less than the one provided in the entity dataframe
- - If there a TTL for the current feature_view, also keep the rows where the `timestamp_field`
- is higher the the one provided minus the TTL
- - For each row, Join on the entity key and retrieve the `entity_row_unique_id` that has been
- computed previously
-
- The output of this CTE will contain all the necessary information and already filtered out most
- of the data that is not relevant.
-*/
-
-{{ featureview.name }}__subquery AS (
- SELECT
- {{ featureview.timestamp_field }} as event_timestamp,
- {{ featureview.created_timestamp_column ~ ' as created_timestamp,' if featureview.created_timestamp_column else '' }}
- {{ featureview.entity_selections | join(', ')}}{% if featureview.entity_selections %},{% else %}{% endif %}
- {% for feature in featureview.features %}
- {{ feature }} as {% if full_feature_names %}{{ featureview.name }}__{{featureview.field_mapping.get(feature, feature)}}{% else %}{{ featureview.field_mapping.get(feature, feature) }}{% endif %}{% if loop.last %}{% else %}, {% endif %}
- {% endfor %}
- FROM {{ featureview.table_subquery }}
- WHERE {{ featureview.timestamp_field }} <= '{{ featureview.max_event_timestamp }}'
- {% if featureview.ttl == 0 %}{% else %}
- AND {{ featureview.timestamp_field }} >= '{{ featureview.min_event_timestamp }}'
- {% endif %}
-),
-
-{{ featureview.name }}__base AS (
- SELECT
- subquery.*,
- entity_dataframe.{{entity_df_event_timestamp_col}} AS entity_timestamp,
- entity_dataframe.{{featureview.name}}__entity_row_unique_id
- FROM {{ featureview.name }}__subquery AS subquery
- INNER JOIN entity_dataframe
- ON 1=1
- AND subquery.event_timestamp <= entity_dataframe.{{entity_df_event_timestamp_col}}
-
- {% if featureview.ttl == 0 %}{% else %}
- AND {{ featureview.ttl }} > = DATEDIFF(SECOND, subquery.event_timestamp, entity_dataframe.{{entity_df_event_timestamp_col}})
- {% endif %}
-
- {% for entity in featureview.entities %}
- AND subquery.{{ entity }} = entity_dataframe.{{ entity }}
- {% endfor %}
-),
-
-/*
- 2. If the `created_timestamp_column` has been set, we need to
- deduplicate the data first. This is done by calculating the
- `MAX(created_at_timestamp)` for each event_timestamp.
- We then join the data on the next CTE
-*/
-{% if featureview.created_timestamp_column %}
-{{ featureview.name }}__dedup AS (
- SELECT
- {{featureview.name}}__entity_row_unique_id,
- event_timestamp,
- MAX(created_timestamp) as created_timestamp
- FROM {{ featureview.name }}__base
- GROUP BY {{featureview.name}}__entity_row_unique_id, event_timestamp
-),
-{% endif %}
-
-/*
- 3. The data has been filtered during the first CTE "*__base"
- Thus we only need to compute the latest timestamp of each feature.
-*/
-{{ featureview.name }}__latest AS (
- SELECT
- {{ featureview.name }}__base.{{ featureview.name }}__entity_row_unique_id,
- MAX({{ featureview.name }}__base.event_timestamp) AS event_timestamp
- {% if featureview.created_timestamp_column %}
- ,MAX({{ featureview.name }}__base.created_timestamp) AS created_timestamp
- {% endif %}
-
- FROM {{ featureview.name }}__base
- {% if featureview.created_timestamp_column %}
- INNER JOIN {{ featureview.name }}__dedup
- ON {{ featureview.name }}__dedup.{{ featureview.name }}__entity_row_unique_id = {{ featureview.name }}__base.{{ featureview.name }}__entity_row_unique_id
- AND {{ featureview.name }}__dedup.event_timestamp = {{ featureview.name }}__base.event_timestamp
- AND {{ featureview.name }}__dedup.created_timestamp = {{ featureview.name }}__base.created_timestamp
- {% endif %}
-
- GROUP BY {{ featureview.name }}__base.{{ featureview.name }}__entity_row_unique_id
-),
-
-/*
- 4. Once we know the latest value of each feature for a given timestamp,
- we can join again the data back to the original "base" dataset
-*/
-{{ featureview.name }}__cleaned AS (
- SELECT base.*
- FROM {{ featureview.name }}__base as base
- INNER JOIN {{ featureview.name }}__latest
- ON base.{{ featureview.name }}__entity_row_unique_id = {{ featureview.name }}__latest.{{ featureview.name }}__entity_row_unique_id
- AND base.event_timestamp = {{ featureview.name }}__latest.event_timestamp
- {% if featureview.created_timestamp_column %}
- AND base.created_timestamp = {{ featureview.name }}__latest.created_timestamp
- {% endif %}
-){% if loop.last %}{% else %}, {% endif %}
-
-{% endfor %}
-
-/*
- Joins the outputs of multiple time travel joins to a single table.
- The entity_dataframe dataset being our source of truth here.
- */
-
-SELECT {{ final_output_feature_names | join(', ')}}
-FROM entity_dataframe
-{% for featureview in featureviews %}
-LEFT JOIN (
- SELECT
- {{featureview.name}}__entity_row_unique_id
- {% for feature in featureview.features %}
- ,{% if full_feature_names %}{{ featureview.name }}__{{featureview.field_mapping.get(feature, feature)}}{% else %}{{ featureview.field_mapping.get(feature, feature) }}{% endif %}
- {% endfor %}
- FROM "{{ featureview.name }}__cleaned"
-) {{ featureview.name }}__cleaned
-ON
-{{ featureview.name }}__cleaned.{{ featureview.name }}__entity_row_unique_id = entity_dataframe.{{ featureview.name }}__entity_row_unique_id
-{% endfor %}
-"""
diff --git a/sdk/python/feast/infra/offline_stores/contrib/mssql_offline_store/mssqlserver_source.py b/sdk/python/feast/infra/offline_stores/contrib/mssql_offline_store/mssqlserver_source.py
index 6b126fa40c0..39abd1c9e74 100644
--- a/sdk/python/feast/infra/offline_stores/contrib/mssql_offline_store/mssqlserver_source.py
+++ b/sdk/python/feast/infra/offline_stores/contrib/mssql_offline_store/mssqlserver_source.py
@@ -3,15 +3,10 @@
import json
import warnings
from typing import Callable, Dict, Iterable, Optional, Tuple
-
-import pandas
-from sqlalchemy import create_engine
+from urllib import parse
from feast import type_map
from feast.data_source import DataSource
-from feast.infra.offline_stores.contrib.mssql_offline_store.mssql import (
- MsSqlServerOfflineStoreConfig,
-)
from feast.protos.feast.core.DataSource_pb2 import DataSource as DataSourceProto
from feast.repo_config import RepoConfig
from feast.value_type import ValueType
@@ -20,6 +15,21 @@
warnings.simplefilter("once", RuntimeWarning)
+def get_ibis_connection(config: RepoConfig):
+ import ibis
+
+ connection_params = parse.urlparse(config.offline_store.connection_string)
+ additional_kwargs = dict(parse.parse_qsl(connection_params.query))
+ return ibis.mssql.connect(
+ user=connection_params.username,
+ password=connection_params.password,
+ host=connection_params.hostname,
+ port=connection_params.port,
+ database=connection_params.path.strip("/"),
+ **additional_kwargs,
+ )
+
+
class MsSqlServerOptions:
"""
DataSource MsSQLServer options used to source features from MsSQLServer query
@@ -114,11 +124,11 @@ def __init__(
tags: Optional[Dict[str, str]] = None,
owner: Optional[str] = None,
):
- warnings.warn(
- "The Azure Synapse + Azure SQL data source is an experimental feature in alpha development. "
- "Some functionality may still be unstable so functionality can change in the future.",
- RuntimeWarning,
- )
+ # warnings.warn(
+ # "The Azure Synapse + Azure SQL data source is an experimental feature in alpha development. "
+ # "Some functionality may still be unstable so functionality can change in the future.",
+ # RuntimeWarning,
+ # )
self._mssqlserver_options = MsSqlServerOptions(
connection_str=connection_str, table_ref=table_ref
)
@@ -222,11 +232,8 @@ def source_datatype_to_feast_value_type() -> Callable[[str], ValueType]:
def get_table_column_names_and_types(
self, config: RepoConfig
) -> Iterable[Tuple[str, str]]:
- assert isinstance(config.offline_store, MsSqlServerOfflineStoreConfig)
- conn = create_engine(config.offline_store.connection_string)
- self._mssqlserver_options.connection_str = (
- config.offline_store.connection_string
- )
+ con = get_ibis_connection(config)
+
name_type_pairs = []
if len(self.table_ref.split(".")) == 2:
database, table_name = self.table_ref.split(".")
@@ -240,7 +247,8 @@ def get_table_column_names_and_types(
WHERE TABLE_NAME = '{self.table_ref}'
"""
- table_schema = pandas.read_sql(columns_query, conn)
+ table_schema = con.sql(columns_query).execute()
+
name_type_pairs.extend(
list(
zip(
diff --git a/sdk/python/feast/infra/offline_stores/contrib/mssql_offline_store/tests/data_source.py b/sdk/python/feast/infra/offline_stores/contrib/mssql_offline_store/tests/data_source.py
index ccf826c068f..9c87b8d7520 100644
--- a/sdk/python/feast/infra/offline_stores/contrib/mssql_offline_store/tests/data_source.py
+++ b/sdk/python/feast/infra/offline_stores/contrib/mssql_offline_store/tests/data_source.py
@@ -1,15 +1,15 @@
from typing import Dict, List, Optional
+import ibis
import pandas as pd
import pytest
-from sqlalchemy import create_engine
from testcontainers.core.waiting_utils import wait_for_logs
from testcontainers.mssql import SqlServerContainer
from feast.data_source import DataSource
+from feast.feature_logging import LoggingDestination
from feast.infra.offline_stores.contrib.mssql_offline_store.mssql import (
MsSqlServerOfflineStoreConfig,
- _df_to_create_table_sql,
)
from feast.infra.offline_stores.contrib.mssql_offline_store.mssqlserver_source import (
MsSqlServerSource,
@@ -26,7 +26,7 @@
@pytest.fixture(scope="session")
def mssql_container():
container = SqlServerContainer(
- user=MSSQL_USER,
+ username=MSSQL_USER,
password=MSSQL_PASSWORD,
image="mcr.microsoft.com/azure-sql-edge:1.0.6",
)
@@ -56,8 +56,10 @@ def __init__(
)
def create_offline_store_config(self) -> MsSqlServerOfflineStoreConfig:
+ connection_string = self.container.get_connection_url()
+ connection_string += "?driver=FreeTDS"
return MsSqlServerOfflineStoreConfig(
- connection_string=self.container.get_connection_url(),
+ connection_string=connection_string,
)
def create_data_source(
@@ -66,7 +68,7 @@ def create_data_source(
destination_name: str,
created_timestamp_column="created_ts",
field_mapping: Optional[Dict[str, str]] = None,
- timestamp_field: Optional[str] = "ts",
+ timestamp_field: Optional[str] = None,
) -> DataSource:
# Make sure the field mapping is correct and convert the datetime datasources.
if timestamp_field in df:
@@ -79,13 +81,23 @@ def create_data_source(
).fillna(pd.Timestamp.now())
connection_string = self.create_offline_store_config().connection_string
- engine = create_engine(connection_string)
+
+ con = ibis.mssql.connect(
+ user=self.container.username,
+ password=self.container.password,
+ host=self.container.get_container_host_ip(),
+ port=self.container.get_exposed_port(self.container.port),
+ database=self.container.dbname,
+ driver="FreeTDS",
+ )
+
destination_name = self.get_prefixed_table_name(destination_name)
- # Create table
- engine.execute(_df_to_create_table_sql(df, destination_name)) # type: ignore
- # Upload dataframe to azure table
- df.to_sql(destination_name, engine, index=False, if_exists="append")
+ con.create_table(
+ name=destination_name,
+ schema=ibis.Schema.from_pandas(df.dtypes.to_dict().items()),
+ )
+ con.insert(table_name=destination_name, obj=df)
self.tables.append(destination_name)
return MsSqlServerSource(
@@ -100,8 +112,12 @@ def create_data_source(
def create_saved_dataset_destination(self) -> SavedDatasetStorage:
raise NotImplementedError
+ def create_logged_features_destination(self) -> LoggingDestination:
+ raise NotImplementedError
+
def get_prefixed_table_name(self, destination_name: str) -> str:
return f"{self.project_name}_{destination_name}"
def teardown(self):
- raise NotImplementedError
+ pass
+ # raise NotImplementedError
diff --git a/sdk/python/feast/infra/offline_stores/contrib/postgres_offline_store/postgres.py b/sdk/python/feast/infra/offline_stores/contrib/postgres_offline_store/postgres.py
index cb08b5f0168..c4740a960ef 100644
--- a/sdk/python/feast/infra/offline_stores/contrib/postgres_offline_store/postgres.py
+++ b/sdk/python/feast/infra/offline_stores/contrib/postgres_offline_store/postgres.py
@@ -19,11 +19,11 @@
import pandas as pd
import pyarrow as pa
from jinja2 import BaseLoader, Environment
-from psycopg2 import sql
+from psycopg import sql
from pytz import utc
from feast.data_source import DataSource
-from feast.errors import InvalidEntityType
+from feast.errors import InvalidEntityType, ZeroColumnQueryResult, ZeroRowsQueryResult
from feast.feature_view import DUMMY_ENTITY_ID, DUMMY_ENTITY_VAL, FeatureView
from feast.infra.offline_stores import offline_utils
from feast.infra.offline_stores.contrib.postgres_offline_store.postgres_source import (
@@ -274,8 +274,10 @@ def to_sql(self) -> str:
def _to_arrow_internal(self, timeout: Optional[int] = None) -> pa.Table:
with self._query_generator() as query:
with _get_conn(self.config.offline_store) as conn, conn.cursor() as cur:
- conn.set_session(readonly=True)
+ conn.read_only = True
cur.execute(query)
+ if not cur.description:
+ raise ZeroColumnQueryResult(query)
fields = [
(c.name, pg_type_code_to_arrow(c.type_code))
for c in cur.description
@@ -331,16 +333,19 @@ def _get_entity_df_event_timestamp_range(
entity_df_event_timestamp.max().to_pydatetime(),
)
elif isinstance(entity_df, str):
- # If the entity_df is a string (SQL query), determine range
- # from table
+ # If the entity_df is a string (SQL query), determine range from table
with _get_conn(config.offline_store) as conn, conn.cursor() as cur:
- (
- cur.execute(
- f"SELECT MIN({entity_df_event_timestamp_col}) AS min, MAX({entity_df_event_timestamp_col}) AS max FROM ({entity_df}) as tmp_alias"
- ),
- )
+ query = f"""
+ SELECT
+ MIN({entity_df_event_timestamp_col}) AS min,
+ MAX({entity_df_event_timestamp_col}) AS max
+ FROM ({entity_df}) AS tmp_alias
+ """
+ cur.execute(query)
res = cur.fetchone()
- entity_df_event_timestamp_range = (res[0], res[1])
+ if not res:
+ raise ZeroRowsQueryResult(query)
+ entity_df_event_timestamp_range = (res[0], res[1])
else:
raise InvalidEntityType(type(entity_df))
diff --git a/sdk/python/feast/infra/offline_stores/contrib/postgres_offline_store/postgres_source.py b/sdk/python/feast/infra/offline_stores/contrib/postgres_offline_store/postgres_source.py
index bbb3f768fda..c216328b8d0 100644
--- a/sdk/python/feast/infra/offline_stores/contrib/postgres_offline_store/postgres_source.py
+++ b/sdk/python/feast/infra/offline_stores/contrib/postgres_offline_store/postgres_source.py
@@ -4,7 +4,7 @@
from typeguard import typechecked
from feast.data_source import DataSource
-from feast.errors import DataSourceNoNameException
+from feast.errors import DataSourceNoNameException, ZeroColumnQueryResult
from feast.infra.utils.postgres.connection_utils import _get_conn
from feast.protos.feast.core.DataSource_pb2 import DataSource as DataSourceProto
from feast.protos.feast.core.SavedDataset_pb2 import (
@@ -111,7 +111,11 @@ def get_table_column_names_and_types(
self, config: RepoConfig
) -> Iterable[Tuple[str, str]]:
with _get_conn(config.offline_store) as conn, conn.cursor() as cur:
- cur.execute(f"SELECT * FROM {self.get_table_query_string()} AS sub LIMIT 0")
+ query = f"SELECT * FROM {self.get_table_query_string()} AS sub LIMIT 0"
+ cur.execute(query)
+ if not cur.description:
+ raise ZeroColumnQueryResult(query)
+
return (
(c.name, pg_type_code_to_pg_type(c.type_code)) for c in cur.description
)
diff --git a/sdk/python/feast/infra/offline_stores/contrib/trino_offline_store/trino_queries.py b/sdk/python/feast/infra/offline_stores/contrib/trino_offline_store/trino_queries.py
index 50472407bc6..3a26583af24 100644
--- a/sdk/python/feast/infra/offline_stores/contrib/trino_offline_store/trino_queries.py
+++ b/sdk/python/feast/infra/offline_stores/contrib/trino_offline_store/trino_queries.py
@@ -1,6 +1,5 @@
from __future__ import annotations
-import datetime
import signal
from dataclasses import dataclass
from enum import Enum
@@ -16,6 +15,7 @@
from feast.infra.offline_stores.contrib.trino_offline_store.trino_type_map import (
trino_to_pa_value_type,
)
+from feast.utils import _utc_now
class QueryStatus(Enum):
@@ -97,12 +97,12 @@ def __init__(self, query_text: str, cursor: Cursor):
def execute(self) -> Results:
try:
self.status = QueryStatus.RUNNING
- start_time = datetime.datetime.utcnow()
+ start_time = _utc_now()
self._cursor.execute(operation=self.query_text)
rows = self._cursor.fetchall()
- end_time = datetime.datetime.utcnow()
+ end_time = _utc_now()
self.execution_time = end_time - start_time
self.status = QueryStatus.COMPLETED
diff --git a/sdk/python/feast/infra/offline_stores/file.py b/sdk/python/feast/infra/offline_stores/dask.py
similarity index 97%
rename from sdk/python/feast/infra/offline_stores/file.py
rename to sdk/python/feast/infra/offline_stores/dask.py
index af2570ebc08..4a63baf6467 100644
--- a/sdk/python/feast/infra/offline_stores/file.py
+++ b/sdk/python/feast/infra/offline_stores/dask.py
@@ -39,20 +39,20 @@
from feast.saved_dataset import SavedDatasetStorage
from feast.utils import _get_requested_feature_views_to_features_dict
-# FileRetrievalJob will cast string objects to string[pyarrow] from dask version 2023.7.1
+# DaskRetrievalJob will cast string objects to string[pyarrow] from dask version 2023.7.1
# This is not the desired behavior for our use case, so we set the convert-string option to False
# See (https://github.com/dask/dask/issues/10881#issuecomment-1923327936)
dask.config.set({"dataframe.convert-string": False})
-class FileOfflineStoreConfig(FeastConfigBaseModel):
- """Offline store config for local (file-based) store"""
+class DaskOfflineStoreConfig(FeastConfigBaseModel):
+ """Offline store config for dask store"""
- type: Literal["file"] = "file"
+ type: Union[Literal["dask"], Literal["file"]] = "dask"
""" Offline store type selector"""
-class FileRetrievalJob(RetrievalJob):
+class DaskRetrievalJob(RetrievalJob):
def __init__(
self,
evaluation_function: Callable,
@@ -122,7 +122,7 @@ def supports_remote_storage_export(self) -> bool:
return False
-class FileOfflineStore(OfflineStore):
+class DaskOfflineStore(OfflineStore):
@staticmethod
def get_historical_features(
config: RepoConfig,
@@ -133,7 +133,7 @@ def get_historical_features(
project: str,
full_feature_names: bool = False,
) -> RetrievalJob:
- assert isinstance(config.offline_store, FileOfflineStoreConfig)
+ assert isinstance(config.offline_store, DaskOfflineStoreConfig)
for fv in feature_views:
assert isinstance(fv.batch_source, FileSource)
@@ -283,7 +283,7 @@ def evaluate_historical_retrieval():
return entity_df_with_features.persist()
- job = FileRetrievalJob(
+ job = DaskRetrievalJob(
evaluation_function=evaluate_historical_retrieval,
full_feature_names=full_feature_names,
on_demand_feature_views=OnDemandFeatureView.get_requested_odfvs(
@@ -309,7 +309,7 @@ def pull_latest_from_table_or_query(
start_date: datetime,
end_date: datetime,
) -> RetrievalJob:
- assert isinstance(config.offline_store, FileOfflineStoreConfig)
+ assert isinstance(config.offline_store, DaskOfflineStoreConfig)
assert isinstance(data_source, FileSource)
# Create lazy function that is only called from the RetrievalJob object
@@ -372,7 +372,7 @@ def evaluate_offline_job():
return source_df[list(columns_to_extract)].persist()
# When materializing a single feature view, we don't need full feature names. On demand transforms aren't materialized
- return FileRetrievalJob(
+ return DaskRetrievalJob(
evaluation_function=evaluate_offline_job,
full_feature_names=False,
)
@@ -387,10 +387,10 @@ def pull_all_from_table_or_query(
start_date: datetime,
end_date: datetime,
) -> RetrievalJob:
- assert isinstance(config.offline_store, FileOfflineStoreConfig)
+ assert isinstance(config.offline_store, DaskOfflineStoreConfig)
assert isinstance(data_source, FileSource)
- return FileOfflineStore.pull_latest_from_table_or_query(
+ return DaskOfflineStore.pull_latest_from_table_or_query(
config=config,
data_source=data_source,
join_key_columns=join_key_columns
@@ -410,7 +410,7 @@ def write_logged_features(
logging_config: LoggingConfig,
registry: BaseRegistry,
):
- assert isinstance(config.offline_store, FileOfflineStoreConfig)
+ assert isinstance(config.offline_store, DaskOfflineStoreConfig)
destination = logging_config.destination
assert isinstance(destination, FileLoggingDestination)
@@ -441,7 +441,7 @@ def offline_write_batch(
table: pyarrow.Table,
progress: Optional[Callable[[int], Any]],
):
- assert isinstance(config.offline_store, FileOfflineStoreConfig)
+ assert isinstance(config.offline_store, DaskOfflineStoreConfig)
assert isinstance(feature_view.batch_source, FileSource)
pa_schema, column_names = get_pyarrow_schema_from_batch_source(
diff --git a/sdk/python/feast/infra/offline_stores/ibis.py b/sdk/python/feast/infra/offline_stores/ibis.py
index 6cc1606a458..4de16cbda3c 100644
--- a/sdk/python/feast/infra/offline_stores/ibis.py
+++ b/sdk/python/feast/infra/offline_stores/ibis.py
@@ -1,3 +1,5 @@
+import random
+import string
import uuid
from datetime import datetime, timedelta
from pathlib import Path
@@ -95,7 +97,9 @@ def _get_entity_df_event_timestamp_range(
entity_df_event_timestamp = entity_df.loc[
:, entity_df_event_timestamp_col
].infer_objects()
- if pd.api.types.is_string_dtype(entity_df_event_timestamp):
+ if pd.api.types.is_string_dtype(
+ entity_df_event_timestamp
+ ) or pd.api.types.is_object_dtype(entity_df_event_timestamp):
entity_df_event_timestamp = pd.to_datetime(entity_df_event_timestamp, utc=True)
entity_df_event_timestamp_range = (
entity_df_event_timestamp.min().to_pydatetime(),
@@ -107,7 +111,10 @@ def _get_entity_df_event_timestamp_range(
def _to_utc(entity_df: pd.DataFrame, event_timestamp_col):
entity_df_event_timestamp = entity_df.loc[:, event_timestamp_col].infer_objects()
- if pd.api.types.is_string_dtype(entity_df_event_timestamp):
+
+ if pd.api.types.is_string_dtype(
+ entity_df_event_timestamp
+ ) or pd.api.types.is_object_dtype(entity_df_event_timestamp):
entity_df_event_timestamp = pd.to_datetime(entity_df_event_timestamp, utc=True)
entity_df[event_timestamp_col] = entity_df_event_timestamp
@@ -146,6 +153,7 @@ def get_historical_features_ibis(
full_feature_names: bool = False,
staging_location: Optional[str] = None,
staging_location_endpoint_override: Optional[str] = None,
+ event_expire_timestamp_fn=None,
) -> RetrievalJob:
entity_schema = _get_entity_schema(
entity_df=entity_df,
@@ -218,6 +226,7 @@ def read_fv(
for feature_view in feature_views
],
event_timestamp_col=event_timestamp_col,
+ event_expire_timestamp_fn=event_expire_timestamp_fn,
)
odfvs = OnDemandFeatureView.get_requested_odfvs(feature_refs, project, registry)
@@ -335,11 +344,8 @@ def deduplicate(
if created_timestamp_col:
order_by_fields.append(ibis.desc(table[created_timestamp_col]))
- table = (
- table.group_by(by=group_by_cols)
- .order_by(order_by_fields)
- .mutate(rn=ibis.row_number())
- )
+ window = ibis.window(group_by=group_by_cols, order_by=order_by_fields, following=0)
+ table = table.mutate(rn=ibis.row_number().over(window))
return table.filter(table["rn"] == ibis.literal(0)).drop("rn")
@@ -348,6 +354,7 @@ def point_in_time_join(
entity_table: Table,
feature_tables: List[Tuple[Table, str, str, Dict[str, str], List[str], timedelta]],
event_timestamp_col="event_timestamp",
+ event_expire_timestamp_fn=None,
):
# TODO handle ttl
all_entities = [event_timestamp_col]
@@ -378,6 +385,19 @@ def point_in_time_join(
feature_refs,
ttl,
) in feature_tables:
+ if ttl:
+ if not event_expire_timestamp_fn:
+ feature_table = feature_table.mutate(
+ event_expire_timestamp=feature_table[timestamp_field]
+ + ibis.literal(ttl)
+ )
+ else:
+ alias = "".join(random.choices(string.ascii_uppercase, k=10))
+
+ feature_table = feature_table.alias(alias=alias).sql(
+ f"SELECT *, {event_expire_timestamp_fn(timestamp_field, ttl)} AS event_expire_timestamp FROM {alias}"
+ )
+
predicates = [
feature_table[k] == entity_table[v] for k, v in join_key_map.items()
]
@@ -388,8 +408,8 @@ def point_in_time_join(
if ttl:
predicates.append(
- feature_table[timestamp_field]
- >= entity_table[event_timestamp_col] - ibis.literal(ttl)
+ feature_table["event_expire_timestamp"]
+ >= entity_table[event_timestamp_col]
)
feature_table = feature_table.inner_join(
diff --git a/sdk/python/feast/infra/online_stores/contrib/postgres.py b/sdk/python/feast/infra/online_stores/contrib/postgres.py
index 3eddd8ba203..ff73a4a3473 100644
--- a/sdk/python/feast/infra/online_stores/contrib/postgres.py
+++ b/sdk/python/feast/infra/online_stores/contrib/postgres.py
@@ -2,19 +2,35 @@
import logging
from collections import defaultdict
from datetime import datetime
-from typing import Any, Callable, Dict, List, Literal, Optional, Sequence, Tuple
+from typing import (
+ Any,
+ AsyncGenerator,
+ Callable,
+ Dict,
+ Generator,
+ List,
+ Literal,
+ Optional,
+ Sequence,
+ Tuple,
+ Union,
+)
-import psycopg2
import pytz
-from psycopg2 import sql
-from psycopg2.extras import execute_values
-from psycopg2.pool import SimpleConnectionPool
+from psycopg import AsyncConnection, sql
+from psycopg.connection import Connection
+from psycopg_pool import AsyncConnectionPool, ConnectionPool
from feast import Entity
from feast.feature_view import FeatureView
from feast.infra.key_encoding_utils import get_list_val_str, serialize_entity_key
from feast.infra.online_stores.online_store import OnlineStore
-from feast.infra.utils.postgres.connection_utils import _get_conn, _get_connection_pool
+from feast.infra.utils.postgres.connection_utils import (
+ _get_conn,
+ _get_conn_async,
+ _get_connection_pool,
+ _get_connection_pool_async,
+)
from feast.infra.utils.postgres.postgres_config import ConnectionType, PostgreSQLConfig
from feast.protos.feast.types.EntityKey_pb2 import EntityKey as EntityKeyProto
from feast.protos.feast.types.Value_pb2 import Value as ValueProto
@@ -39,15 +55,20 @@ class PostgreSQLOnlineStoreConfig(PostgreSQLConfig):
class PostgreSQLOnlineStore(OnlineStore):
- _conn: Optional[psycopg2._psycopg.connection] = None
- _conn_pool: Optional[SimpleConnectionPool] = None
+ _conn: Optional[Connection] = None
+ _conn_pool: Optional[ConnectionPool] = None
+
+ _conn_async: Optional[AsyncConnection] = None
+ _conn_pool_async: Optional[AsyncConnectionPool] = None
@contextlib.contextmanager
- def _get_conn(self, config: RepoConfig):
+ def _get_conn(self, config: RepoConfig) -> Generator[Connection, Any, Any]:
assert config.online_store.type == "postgres"
+
if config.online_store.conn_type == ConnectionType.pool:
if not self._conn_pool:
self._conn_pool = _get_connection_pool(config.online_store)
+ self._conn_pool.open()
connection = self._conn_pool.getconn()
yield connection
self._conn_pool.putconn(connection)
@@ -56,6 +77,24 @@ def _get_conn(self, config: RepoConfig):
self._conn = _get_conn(config.online_store)
yield self._conn
+ @contextlib.asynccontextmanager
+ async def _get_conn_async(
+ self, config: RepoConfig
+ ) -> AsyncGenerator[AsyncConnection, Any]:
+ if config.online_store.conn_type == ConnectionType.pool:
+ if not self._conn_pool_async:
+ self._conn_pool_async = await _get_connection_pool_async(
+ config.online_store
+ )
+ await self._conn_pool_async.open()
+ connection = await self._conn_pool_async.getconn()
+ yield connection
+ await self._conn_pool_async.putconn(connection)
+ else:
+ if not self._conn_async:
+ self._conn_async = await _get_conn_async(config.online_store)
+ yield self._conn_async
+
def online_write_batch(
self,
config: RepoConfig,
@@ -65,58 +104,54 @@ def online_write_batch(
],
progress: Optional[Callable[[int], Any]],
) -> None:
- project = config.project
+ # Format insert values
+ insert_values = []
+ for entity_key, values, timestamp, created_ts in data:
+ entity_key_bin = serialize_entity_key(
+ entity_key,
+ entity_key_serialization_version=config.entity_key_serialization_version,
+ )
+ timestamp = _to_naive_utc(timestamp)
+ if created_ts is not None:
+ created_ts = _to_naive_utc(created_ts)
- with self._get_conn(config) as conn, conn.cursor() as cur:
- insert_values = []
- for entity_key, values, timestamp, created_ts in data:
- entity_key_bin = serialize_entity_key(
- entity_key,
- entity_key_serialization_version=config.entity_key_serialization_version,
- )
- timestamp = _to_naive_utc(timestamp)
- if created_ts is not None:
- created_ts = _to_naive_utc(created_ts)
-
- for feature_name, val in values.items():
- vector_val = None
- if config.online_store.pgvector_enabled:
- vector_val = get_list_val_str(val)
- insert_values.append(
- (
- entity_key_bin,
- feature_name,
- val.SerializeToString(),
- vector_val,
- timestamp,
- created_ts,
- )
+ for feature_name, val in values.items():
+ vector_val = None
+ if config.online_store.pgvector_enabled:
+ vector_val = get_list_val_str(val)
+ insert_values.append(
+ (
+ entity_key_bin,
+ feature_name,
+ val.SerializeToString(),
+ vector_val,
+ timestamp,
+ created_ts,
)
- # Control the batch so that we can update the progress
- batch_size = 5000
- for i in range(0, len(insert_values), batch_size):
- cur_batch = insert_values[i : i + batch_size]
- execute_values(
- cur,
- sql.SQL(
- """
- INSERT INTO {}
- (entity_key, feature_name, value, vector_value, event_ts, created_ts)
- VALUES %s
- ON CONFLICT (entity_key, feature_name) DO
- UPDATE SET
- value = EXCLUDED.value,
- vector_value = EXCLUDED.vector_value,
- event_ts = EXCLUDED.event_ts,
- created_ts = EXCLUDED.created_ts;
- """,
- ).format(sql.Identifier(_table_id(project, table))),
- cur_batch,
- page_size=batch_size,
)
- conn.commit()
- if progress:
- progress(len(cur_batch))
+
+ # Create insert query
+ sql_query = sql.SQL(
+ """
+ INSERT INTO {}
+ (entity_key, feature_name, value, vector_value, event_ts, created_ts)
+ VALUES (%s, %s, %s, %s, %s, %s)
+ ON CONFLICT (entity_key, feature_name) DO
+ UPDATE SET
+ value = EXCLUDED.value,
+ vector_value = EXCLUDED.vector_value,
+ event_ts = EXCLUDED.event_ts,
+ created_ts = EXCLUDED.created_ts;
+ """
+ ).format(sql.Identifier(_table_id(config.project, table)))
+
+ # Push data into the online store
+ with self._get_conn(config) as conn, conn.cursor() as cur:
+ cur.executemany(sql_query, insert_values)
+ conn.commit()
+
+ if progress:
+ progress(len(data))
def online_read(
self,
@@ -125,67 +160,107 @@ def online_read(
entity_keys: List[EntityKeyProto],
requested_features: Optional[List[str]] = None,
) -> List[Tuple[Optional[datetime], Optional[Dict[str, ValueProto]]]]:
- result: List[Tuple[Optional[datetime], Optional[Dict[str, ValueProto]]]] = []
+ keys = self._prepare_keys(entity_keys, config.entity_key_serialization_version)
+ query, params = self._construct_query_and_params(
+ config, table, keys, requested_features
+ )
- project = config.project
with self._get_conn(config) as conn, conn.cursor() as cur:
- # Collecting all the keys to a list allows us to make fewer round trips
- # to PostgreSQL
- keys = []
- for entity_key in entity_keys:
- keys.append(
- serialize_entity_key(
- entity_key,
- entity_key_serialization_version=config.entity_key_serialization_version,
- )
- )
+ cur.execute(query, params)
+ rows = cur.fetchall()
- if not requested_features:
- cur.execute(
- sql.SQL(
- """
- SELECT entity_key, feature_name, value, event_ts
- FROM {} WHERE entity_key = ANY(%s);
- """
- ).format(
- sql.Identifier(_table_id(project, table)),
- ),
- (keys,),
- )
- else:
- cur.execute(
- sql.SQL(
- """
- SELECT entity_key, feature_name, value, event_ts
- FROM {} WHERE entity_key = ANY(%s) and feature_name = ANY(%s);
- """
- ).format(
- sql.Identifier(_table_id(project, table)),
- ),
- (keys, requested_features),
- )
+ return self._process_rows(keys, rows)
- rows = cur.fetchall()
+ async def online_read_async(
+ self,
+ config: RepoConfig,
+ table: FeatureView,
+ entity_keys: List[EntityKeyProto],
+ requested_features: Optional[List[str]] = None,
+ ) -> List[Tuple[Optional[datetime], Optional[Dict[str, ValueProto]]]]:
+ keys = self._prepare_keys(entity_keys, config.entity_key_serialization_version)
+ query, params = self._construct_query_and_params(
+ config, table, keys, requested_features
+ )
- # Since we don't know the order returned from PostgreSQL we'll need
- # to construct a dict to be able to quickly look up the correct row
- # when we iterate through the keys since they are in the correct order
- values_dict = defaultdict(list)
- for row in rows if rows is not None else []:
- values_dict[row[0].tobytes()].append(row[1:])
-
- for key in keys:
- if key in values_dict:
- value = values_dict[key]
- res = {}
- for feature_name, value_bin, event_ts in value:
- val = ValueProto()
- val.ParseFromString(bytes(value_bin))
- res[feature_name] = val
- result.append((event_ts, res))
- else:
- result.append((None, None))
+ async with self._get_conn_async(config) as conn:
+ async with conn.cursor() as cur:
+ await cur.execute(query, params)
+ rows = await cur.fetchall()
+
+ return self._process_rows(keys, rows)
+ @staticmethod
+ def _construct_query_and_params(
+ config: RepoConfig,
+ table: FeatureView,
+ keys: List[bytes],
+ requested_features: Optional[List[str]] = None,
+ ) -> Tuple[sql.Composed, Union[Tuple[List[bytes], List[str]], Tuple[List[bytes]]]]:
+ """Construct the SQL query based on the given parameters."""
+ if requested_features:
+ query = sql.SQL(
+ """
+ SELECT entity_key, feature_name, value, event_ts
+ FROM {} WHERE entity_key = ANY(%s) AND feature_name = ANY(%s);
+ """
+ ).format(
+ sql.Identifier(_table_id(config.project, table)),
+ )
+ params = (keys, requested_features)
+ else:
+ query = sql.SQL(
+ """
+ SELECT entity_key, feature_name, value, event_ts
+ FROM {} WHERE entity_key = ANY(%s);
+ """
+ ).format(
+ sql.Identifier(_table_id(config.project, table)),
+ )
+ params = (keys, [])
+ return query, params
+
+ @staticmethod
+ def _prepare_keys(
+ entity_keys: List[EntityKeyProto], entity_key_serialization_version: int
+ ) -> List[bytes]:
+ """Prepare all keys in a list to make fewer round trips to the database."""
+ return [
+ serialize_entity_key(
+ entity_key,
+ entity_key_serialization_version=entity_key_serialization_version,
+ )
+ for entity_key in entity_keys
+ ]
+
+ @staticmethod
+ def _process_rows(
+ keys: List[bytes], rows: List[Tuple]
+ ) -> List[Tuple[Optional[datetime], Optional[Dict[str, ValueProto]]]]:
+ """Transform the retrieved rows in the desired output.
+
+ PostgreSQL may return rows in an unpredictable order. Therefore, `values_dict`
+ is created to quickly look up the correct row using the keys, since these are
+ actually in the correct order.
+ """
+ values_dict = defaultdict(list)
+ for row in rows if rows is not None else []:
+ values_dict[
+ row[0] if isinstance(row[0], bytes) else row[0].tobytes()
+ ].append(row[1:])
+
+ result: List[Tuple[Optional[datetime], Optional[Dict[str, ValueProto]]]] = []
+ for key in keys:
+ if key in values_dict:
+ value = values_dict[key]
+ res = {}
+ for feature_name, value_bin, event_ts in value:
+ val = ValueProto()
+ val.ParseFromString(bytes(value_bin))
+ res[feature_name] = val
+ result.append((event_ts, res))
+ else:
+ result.append((None, None))
return result
def update(
diff --git a/sdk/python/feast/infra/online_stores/contrib/singlestore_online_store/singlestore.py b/sdk/python/feast/infra/online_stores/contrib/singlestore_online_store/singlestore.py
new file mode 100644
index 00000000000..e17a059c1a8
--- /dev/null
+++ b/sdk/python/feast/infra/online_stores/contrib/singlestore_online_store/singlestore.py
@@ -0,0 +1,235 @@
+from __future__ import absolute_import
+
+from collections import defaultdict
+from datetime import datetime
+from typing import Any, Callable, Dict, List, Literal, Optional, Sequence, Tuple
+
+import pytz
+import singlestoredb
+from pydantic import StrictStr
+from singlestoredb.connection import Connection, Cursor
+from singlestoredb.exceptions import InterfaceError
+
+from feast import Entity, FeatureView, RepoConfig
+from feast.infra.key_encoding_utils import serialize_entity_key
+from feast.infra.online_stores.online_store import OnlineStore
+from feast.protos.feast.types.EntityKey_pb2 import EntityKey as EntityKeyProto
+from feast.protos.feast.types.Value_pb2 import Value as ValueProto
+from feast.repo_config import FeastConfigBaseModel
+
+
+class SingleStoreOnlineStoreConfig(FeastConfigBaseModel):
+ """
+ Configuration for the SingleStore online store.
+ NOTE: The class *must* end with the `OnlineStoreConfig` suffix.
+ """
+
+ type: Literal["singlestore"] = "singlestore"
+
+ host: Optional[StrictStr] = None
+ user: Optional[StrictStr] = None
+ password: Optional[StrictStr] = None
+ database: Optional[StrictStr] = None
+ port: Optional[int] = None
+
+
+class SingleStoreOnlineStore(OnlineStore):
+ """
+ An online store implementation that uses SingleStore.
+ NOTE: The class *must* end with the `OnlineStore` suffix.
+ """
+
+ _conn: Optional[Connection] = None
+
+ def _init_conn(self, config: RepoConfig) -> Connection:
+ online_store_config = config.online_store
+ assert isinstance(online_store_config, SingleStoreOnlineStoreConfig)
+ return singlestoredb.connect(
+ host=online_store_config.host or "127.0.0.1",
+ user=online_store_config.user or "test",
+ password=online_store_config.password or "test",
+ database=online_store_config.database or "feast",
+ port=online_store_config.port or 3306,
+ autocommit=True,
+ )
+
+ def _get_cursor(self, config: RepoConfig) -> Any:
+ # This will try to reconnect also.
+ # In case it fails, we will have to create a new connection.
+ if not self._conn:
+ self._conn = self._init_conn(config)
+ try:
+ self._conn.ping(reconnect=True)
+ except InterfaceError:
+ self._conn = self._init_conn(config)
+ return self._conn.cursor()
+
+ def online_write_batch(
+ self,
+ config: RepoConfig,
+ table: FeatureView,
+ data: List[
+ Tuple[EntityKeyProto, Dict[str, ValueProto], datetime, Optional[datetime]]
+ ],
+ progress: Optional[Callable[[int], Any]],
+ ) -> None:
+ project = config.project
+ with self._get_cursor(config) as cur:
+ insert_values = []
+ for entity_key, values, timestamp, created_ts in data:
+ entity_key_bin = serialize_entity_key(
+ entity_key,
+ entity_key_serialization_version=2,
+ ).hex()
+ timestamp = _to_naive_utc(timestamp)
+ if created_ts is not None:
+ created_ts = _to_naive_utc(created_ts)
+
+ for feature_name, val in values.items():
+ insert_values.append(
+ (
+ entity_key_bin,
+ feature_name,
+ val.SerializeToString(),
+ timestamp,
+ created_ts,
+ )
+ )
+ # Control the batch so that we can update the progress
+ batch_size = 50000
+ for i in range(0, len(insert_values), batch_size):
+ current_batch = insert_values[i : i + batch_size]
+ cur.executemany(
+ f"""
+ INSERT INTO {_table_id(project, table)}
+ (entity_key, feature_name, value, event_ts, created_ts)
+ values (%s, %s, %s, %s, %s)
+ ON DUPLICATE KEY UPDATE
+ value = VALUES(value),
+ event_ts = VALUES(event_ts),
+ created_ts = VALUES(created_ts);
+ """,
+ current_batch,
+ )
+ if progress:
+ progress(len(current_batch))
+
+ def online_read(
+ self,
+ config: RepoConfig,
+ table: FeatureView,
+ entity_keys: List[EntityKeyProto],
+ requested_features: Optional[List[str]] = None,
+ ) -> List[Tuple[Optional[datetime], Optional[Dict[str, ValueProto]]]]:
+ project = config.project
+ result: List[Tuple[Optional[datetime], Optional[Dict[str, ValueProto]]]] = []
+ with self._get_cursor(config) as cur:
+ keys = []
+ for entity_key in entity_keys:
+ keys.append(
+ serialize_entity_key(
+ entity_key,
+ entity_key_serialization_version=2,
+ ).hex()
+ )
+
+ if not requested_features:
+ entity_key_placeholders = ",".join(["%s" for _ in keys])
+ cur.execute(
+ f"""
+ SELECT entity_key, feature_name, value, event_ts FROM {_table_id(project, table)}
+ WHERE entity_key IN ({entity_key_placeholders})
+ ORDER BY event_ts;
+ """,
+ tuple(keys),
+ )
+ else:
+ entity_key_placeholders = ",".join(["%s" for _ in keys])
+ requested_features_placeholders = ",".join(
+ ["%s" for _ in requested_features]
+ )
+ cur.execute(
+ f"""
+ SELECT entity_key, feature_name, value, event_ts FROM {_table_id(project, table)}
+ WHERE entity_key IN ({entity_key_placeholders}) and feature_name IN ({requested_features_placeholders})
+ ORDER BY event_ts;
+ """,
+ tuple(keys + requested_features),
+ )
+ rows = cur.fetchall() or []
+
+ # Since we don't know the order returned from MySQL we'll need
+ # to construct a dict to be able to quickly look up the correct row
+ # when we iterate through the keys since they are in the correct order
+ values_dict = defaultdict(list)
+ for row in rows:
+ values_dict[row[0]].append(row[1:])
+
+ for key in keys:
+ if key in values_dict:
+ key_values = values_dict[key]
+ res = {}
+ res_ts: Optional[datetime] = None
+ for feature_name, value_bin, event_ts in key_values:
+ val = ValueProto()
+ val.ParseFromString(bytes(value_bin))
+ res[feature_name] = val
+ res_ts = event_ts
+ result.append((res_ts, res))
+ else:
+ result.append((None, None))
+ return result
+
+ def update(
+ self,
+ config: RepoConfig,
+ tables_to_delete: Sequence[FeatureView],
+ tables_to_keep: Sequence[FeatureView],
+ entities_to_delete: Sequence[Entity],
+ entities_to_keep: Sequence[Entity],
+ partial: bool,
+ ) -> None:
+ project = config.project
+ with self._get_cursor(config) as cur:
+ # We don't create any special state for the entities in this implementation.
+ for table in tables_to_keep:
+ cur.execute(
+ f"""CREATE TABLE IF NOT EXISTS {_table_id(project, table)} (entity_key VARCHAR(512),
+ feature_name VARCHAR(256),
+ value BLOB,
+ event_ts timestamp NULL DEFAULT NULL,
+ created_ts timestamp NULL DEFAULT NULL,
+ PRIMARY KEY(entity_key, feature_name),
+ INDEX {_table_id(project, table)}_ek (entity_key))"""
+ )
+
+ for table in tables_to_delete:
+ _drop_table_and_index(cur, project, table)
+
+ def teardown(
+ self,
+ config: RepoConfig,
+ tables: Sequence[FeatureView],
+ entities: Sequence[Entity],
+ ) -> None:
+ project = config.project
+ with self._get_cursor(config) as cur:
+ for table in tables:
+ _drop_table_and_index(cur, project, table)
+
+
+def _drop_table_and_index(cur: Cursor, project: str, table: FeatureView) -> None:
+ table_name = _table_id(project, table)
+ cur.execute(f"DROP INDEX {table_name}_ek ON {table_name};")
+ cur.execute(f"DROP TABLE IF EXISTS {table_name}")
+
+
+def _table_id(project: str, table: FeatureView) -> str:
+ return f"{project}_{table.name}"
+
+
+def _to_naive_utc(ts: datetime) -> datetime:
+ if ts.tzinfo is None:
+ return ts
+ else:
+ return ts.astimezone(pytz.utc).replace(tzinfo=None)
diff --git a/sdk/python/feast/infra/online_stores/contrib/singlestore_repo_configuration.py b/sdk/python/feast/infra/online_stores/contrib/singlestore_repo_configuration.py
new file mode 100644
index 00000000000..2debe0f0ee1
--- /dev/null
+++ b/sdk/python/feast/infra/online_stores/contrib/singlestore_repo_configuration.py
@@ -0,0 +1,10 @@
+from tests.integration.feature_repos.integration_test_repo_config import (
+ IntegrationTestRepoConfig,
+)
+from tests.integration.feature_repos.universal.online_store.singlestore import (
+ SingleStoreOnlineStoreCreator,
+)
+
+FULL_REPO_CONFIGS = [
+ IntegrationTestRepoConfig(online_store_creator=SingleStoreOnlineStoreCreator),
+]
diff --git a/sdk/python/feast/infra/online_stores/datastore.py b/sdk/python/feast/infra/online_stores/datastore.py
index b33767cea56..9ae10792f5a 100644
--- a/sdk/python/feast/infra/online_stores/datastore.py
+++ b/sdk/python/feast/infra/online_stores/datastore.py
@@ -44,7 +44,7 @@
from feast.protos.feast.types.EntityKey_pb2 import EntityKey as EntityKeyProto
from feast.protos.feast.types.Value_pb2 import Value as ValueProto
from feast.repo_config import FeastConfigBaseModel, RepoConfig
-from feast.utils import get_user_agent
+from feast.utils import _utc_now, get_user_agent
LOGGER = logging.getLogger(__name__)
@@ -122,7 +122,7 @@ def update(
entity = datastore.Entity(
key=key, exclude_from_indexes=("created_ts", "event_ts", "values")
)
- entity.update({"created_ts": datetime.utcnow()})
+ entity.update({"created_ts": _utc_now()})
client.put(entity)
for table in tables_to_delete:
@@ -457,7 +457,7 @@ def update(self):
entity = datastore.Entity(
key=key, exclude_from_indexes=("created_ts", "event_ts", "values")
)
- entity.update({"created_ts": datetime.utcnow()})
+ entity.update({"created_ts": _utc_now()})
client.put(entity)
def teardown(self):
diff --git a/sdk/python/feast/infra/online_stores/online_store.py b/sdk/python/feast/infra/online_stores/online_store.py
index 05983a494c0..9cf2ef95f68 100644
--- a/sdk/python/feast/infra/online_stores/online_store.py
+++ b/sdk/python/feast/infra/online_stores/online_store.py
@@ -14,13 +14,17 @@
from abc import ABC, abstractmethod
from datetime import datetime
-from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple
+from typing import Any, Callable, Dict, List, Mapping, Optional, Sequence, Tuple, Union
-from feast import Entity
+from feast import Entity, utils
+from feast.feature_service import FeatureService
from feast.feature_view import FeatureView
from feast.infra.infra_object import InfraObject
+from feast.infra.registry.base_registry import BaseRegistry
+from feast.online_response import OnlineResponse
from feast.protos.feast.core.Registry_pb2 import Registry as RegistryProto
from feast.protos.feast.types.EntityKey_pb2 import EntityKey as EntityKeyProto
+from feast.protos.feast.types.Value_pb2 import RepeatedValue
from feast.protos.feast.types.Value_pb2 import Value as ValueProto
from feast.repo_config import RepoConfig
@@ -105,6 +109,180 @@ async def online_read_async(
f"Online store {self.__class__.__name__} does not support online read async"
)
+ def get_online_features(
+ self,
+ config: RepoConfig,
+ features: Union[List[str], FeatureService],
+ entity_rows: Union[
+ List[Dict[str, Any]],
+ Mapping[str, Union[Sequence[Any], Sequence[ValueProto], RepeatedValue]],
+ ],
+ registry: BaseRegistry,
+ project: str,
+ full_feature_names: bool = False,
+ ) -> OnlineResponse:
+ if isinstance(entity_rows, list):
+ columnar: Dict[str, List[Any]] = {k: [] for k in entity_rows[0].keys()}
+ for entity_row in entity_rows:
+ for key, value in entity_row.items():
+ try:
+ columnar[key].append(value)
+ except KeyError as e:
+ raise ValueError(
+ "All entity_rows must have the same keys."
+ ) from e
+
+ entity_rows = columnar
+
+ (
+ join_key_values,
+ grouped_refs,
+ entity_name_to_join_key_map,
+ requested_on_demand_feature_views,
+ feature_refs,
+ requested_result_row_names,
+ online_features_response,
+ ) = utils._prepare_entities_to_read_from_online_store(
+ registry=registry,
+ project=project,
+ features=features,
+ entity_values=entity_rows,
+ full_feature_names=full_feature_names,
+ native_entity_values=True,
+ )
+
+ for table, requested_features in grouped_refs:
+ # Get the correct set of entity values with the correct join keys.
+ table_entity_values, idxs = utils._get_unique_entities(
+ table,
+ join_key_values,
+ entity_name_to_join_key_map,
+ )
+
+ entity_key_protos = utils._get_entity_key_protos(table_entity_values)
+
+ # Fetch data for Entities.
+ read_rows = self.online_read(
+ config=config,
+ table=table,
+ entity_keys=entity_key_protos,
+ requested_features=requested_features,
+ )
+
+ feature_data = utils._convert_rows_to_protobuf(
+ requested_features, read_rows
+ )
+
+ # Populate the result_rows with the Features from the OnlineStore inplace.
+ utils._populate_response_from_feature_data(
+ feature_data,
+ idxs,
+ online_features_response,
+ full_feature_names,
+ requested_features,
+ table,
+ )
+
+ if requested_on_demand_feature_views:
+ utils._augment_response_with_on_demand_transforms(
+ online_features_response,
+ feature_refs,
+ requested_on_demand_feature_views,
+ full_feature_names,
+ )
+
+ utils._drop_unneeded_columns(
+ online_features_response, requested_result_row_names
+ )
+ return OnlineResponse(online_features_response)
+
+ async def get_online_features_async(
+ self,
+ config: RepoConfig,
+ features: Union[List[str], FeatureService],
+ entity_rows: Union[
+ List[Dict[str, Any]],
+ Mapping[str, Union[Sequence[Any], Sequence[ValueProto], RepeatedValue]],
+ ],
+ registry: BaseRegistry,
+ project: str,
+ full_feature_names: bool = False,
+ ) -> OnlineResponse:
+ if isinstance(entity_rows, list):
+ columnar: Dict[str, List[Any]] = {k: [] for k in entity_rows[0].keys()}
+ for entity_row in entity_rows:
+ for key, value in entity_row.items():
+ try:
+ columnar[key].append(value)
+ except KeyError as e:
+ raise ValueError(
+ "All entity_rows must have the same keys."
+ ) from e
+
+ entity_rows = columnar
+
+ (
+ join_key_values,
+ grouped_refs,
+ entity_name_to_join_key_map,
+ requested_on_demand_feature_views,
+ feature_refs,
+ requested_result_row_names,
+ online_features_response,
+ ) = utils._prepare_entities_to_read_from_online_store(
+ registry=registry,
+ project=project,
+ features=features,
+ entity_values=entity_rows,
+ full_feature_names=full_feature_names,
+ native_entity_values=True,
+ )
+
+ for table, requested_features in grouped_refs:
+ # Get the correct set of entity values with the correct join keys.
+ table_entity_values, idxs = utils._get_unique_entities(
+ table,
+ join_key_values,
+ entity_name_to_join_key_map,
+ )
+
+ entity_key_protos = utils._get_entity_key_protos(table_entity_values)
+
+ # Fetch data for Entities.
+ read_rows = await self.online_read_async(
+ config=config,
+ table=table,
+ entity_keys=entity_key_protos,
+ requested_features=requested_features,
+ )
+
+ feature_data = utils._convert_rows_to_protobuf(
+ requested_features, read_rows
+ )
+
+ # Populate the result_rows with the Features from the OnlineStore inplace.
+ utils._populate_response_from_feature_data(
+ feature_data,
+ idxs,
+ online_features_response,
+ full_feature_names,
+ requested_features,
+ table,
+ )
+
+ if requested_on_demand_feature_views:
+ utils._augment_response_with_on_demand_transforms(
+ online_features_response,
+ feature_refs,
+ requested_on_demand_feature_views,
+ full_feature_names,
+ )
+
+ utils._drop_unneeded_columns(
+ online_features_response, requested_result_row_names
+ )
+ return OnlineResponse(online_features_response)
+
@abstractmethod
def update(
self,
diff --git a/sdk/python/feast/infra/online_stores/sqlite.py b/sdk/python/feast/infra/online_stores/sqlite.py
index 41af14aaf16..9896b766d47 100644
--- a/sdk/python/feast/infra/online_stores/sqlite.py
+++ b/sdk/python/feast/infra/online_stores/sqlite.py
@@ -12,6 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
import itertools
+import logging
import os
import sqlite3
import struct
@@ -20,7 +21,6 @@
from pathlib import Path
from typing import Any, Callable, Dict, List, Literal, Optional, Sequence, Tuple, Union
-import sqlite_vec
from google.protobuf.internal.containers import RepeatedScalarFieldContainer
from pydantic import StrictStr
@@ -84,7 +84,9 @@ def _get_conn(self, config: RepoConfig):
if not self._conn:
db_path = self._get_db_path(config)
self._conn = _initialize_conn(db_path)
- if sys.version_info[0:2] == (3, 10):
+ if sys.version_info[0:2] == (3, 10) and config.online_store.vec_enabled:
+ import sqlite_vec # noqa: F401
+
self._conn.enable_load_extension(True) # type: ignore
sqlite_vec.load(self._conn)
@@ -410,6 +412,10 @@ def retrieve_online_documents(
def _initialize_conn(db_path: str):
+ try:
+ import sqlite_vec # noqa: F401
+ except ModuleNotFoundError:
+ logging.warning("Cannot use sqlite_vec for vector search")
Path(db_path).parent.mkdir(exist_ok=True)
return sqlite3.connect(
db_path,
@@ -482,8 +488,13 @@ def from_proto(sqlite_table_proto: SqliteTableProto) -> Any:
def update(self):
if sys.version_info[0:2] == (3, 10):
- self.conn.enable_load_extension(True)
- sqlite_vec.load(self.conn)
+ try:
+ import sqlite_vec # noqa: F401
+
+ self.conn.enable_load_extension(True)
+ sqlite_vec.load(self.conn)
+ except ModuleNotFoundError:
+ logging.warning("Cannot use sqlite_vec for vector search")
self.conn.execute(
f"CREATE TABLE IF NOT EXISTS {self.name} (entity_key BLOB, feature_name TEXT, value BLOB, vector_value BLOB, event_ts timestamp, created_ts timestamp, PRIMARY KEY(entity_key, feature_name))"
)
diff --git a/sdk/python/feast/infra/passthrough_provider.py b/sdk/python/feast/infra/passthrough_provider.py
index e707f9495db..c3c3048a896 100644
--- a/sdk/python/feast/infra/passthrough_provider.py
+++ b/sdk/python/feast/infra/passthrough_provider.py
@@ -1,5 +1,5 @@
from datetime import datetime, timedelta
-from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple, Union
+from typing import Any, Callable, Dict, List, Mapping, Optional, Sequence, Tuple, Union
import pandas as pd
import pyarrow as pa
@@ -12,6 +12,7 @@
from feast.feature_logging import FeatureServiceLoggingSource
from feast.feature_service import FeatureService
from feast.feature_view import FeatureView
+from feast.infra.infra_object import Infra, InfraObject
from feast.infra.materialization.batch_materialization_engine import (
BatchMaterializationEngine,
MaterializationJobStatus,
@@ -22,7 +23,10 @@
from feast.infra.online_stores.helpers import get_online_store_from_config
from feast.infra.provider import Provider
from feast.infra.registry.base_registry import BaseRegistry
+from feast.online_response import OnlineResponse
+from feast.protos.feast.core.Registry_pb2 import Registry as RegistryProto
from feast.protos.feast.types.EntityKey_pb2 import EntityKey as EntityKeyProto
+from feast.protos.feast.types.Value_pb2 import RepeatedValue
from feast.protos.feast.types.Value_pb2 import Value as ValueProto
from feast.repo_config import BATCH_ENGINE_CLASS_FOR_TYPE, RepoConfig
from feast.saved_dataset import SavedDataset
@@ -103,6 +107,17 @@ def batch_engine(self) -> BatchMaterializationEngine:
self._batch_engine = _batch_engine
return _batch_engine
+ def plan_infra(
+ self, config: RepoConfig, desired_registry_proto: RegistryProto
+ ) -> Infra:
+ infra = Infra()
+ if self.online_store:
+ infra_objects: List[InfraObject] = self.online_store.plan(
+ config, desired_registry_proto
+ )
+ infra.infra_objects += infra_objects
+ return infra
+
def update_infra(
self,
project: str,
@@ -180,6 +195,48 @@ def online_read(
)
return result
+ def get_online_features(
+ self,
+ config: RepoConfig,
+ features: Union[List[str], FeatureService],
+ entity_rows: Union[
+ List[Dict[str, Any]],
+ Mapping[str, Union[Sequence[Any], Sequence[ValueProto], RepeatedValue]],
+ ],
+ registry: BaseRegistry,
+ project: str,
+ full_feature_names: bool = False,
+ ) -> OnlineResponse:
+ return self.online_store.get_online_features(
+ config=config,
+ features=features,
+ entity_rows=entity_rows,
+ registry=registry,
+ project=project,
+ full_feature_names=full_feature_names,
+ )
+
+ async def get_online_features_async(
+ self,
+ config: RepoConfig,
+ features: Union[List[str], FeatureService],
+ entity_rows: Union[
+ List[Dict[str, Any]],
+ Mapping[str, Union[Sequence[Any], Sequence[ValueProto], RepeatedValue]],
+ ],
+ registry: BaseRegistry,
+ project: str,
+ full_feature_names: bool = False,
+ ) -> OnlineResponse:
+ return await self.online_store.get_online_features_async(
+ config=config,
+ features=features,
+ entity_rows=entity_rows,
+ registry=registry,
+ project=project,
+ full_feature_names=full_feature_names,
+ )
+
async def online_read_async(
self,
config: RepoConfig,
diff --git a/sdk/python/feast/infra/provider.py b/sdk/python/feast/infra/provider.py
index 93077f40b97..9940af1d028 100644
--- a/sdk/python/feast/infra/provider.py
+++ b/sdk/python/feast/infra/provider.py
@@ -1,7 +1,7 @@
from abc import ABC, abstractmethod
from datetime import datetime
from pathlib import Path
-from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple, Union
+from typing import Any, Callable, Dict, List, Mapping, Optional, Sequence, Tuple, Union
import pandas as pd
import pyarrow
@@ -15,17 +15,19 @@
from feast.infra.infra_object import Infra
from feast.infra.offline_stores.offline_store import RetrievalJob
from feast.infra.registry.base_registry import BaseRegistry
+from feast.online_response import OnlineResponse
from feast.protos.feast.core.Registry_pb2 import Registry as RegistryProto
from feast.protos.feast.types.EntityKey_pb2 import EntityKey as EntityKeyProto
+from feast.protos.feast.types.Value_pb2 import RepeatedValue
from feast.protos.feast.types.Value_pb2 import Value as ValueProto
from feast.repo_config import RepoConfig
from feast.saved_dataset import SavedDataset
PROVIDERS_CLASS_FOR_TYPE = {
- "gcp": "feast.infra.gcp.GcpProvider",
- "aws": "feast.infra.aws.AwsProvider",
- "local": "feast.infra.local.LocalProvider",
- "azure": "feast.infra.contrib.azure_provider.AzureProvider",
+ "gcp": "feast.infra.passthrough_provider.PassthroughProvider",
+ "aws": "feast.infra.passthrough_provider.PassthroughProvider",
+ "local": "feast.infra.passthrough_provider.PassthroughProvider",
+ "azure": "feast.infra.passthrough_provider.PassthroughProvider",
}
@@ -230,6 +232,36 @@ def online_read(
"""
pass
+ @abstractmethod
+ def get_online_features(
+ self,
+ config: RepoConfig,
+ features: Union[List[str], FeatureService],
+ entity_rows: Union[
+ List[Dict[str, Any]],
+ Mapping[str, Union[Sequence[Any], Sequence[ValueProto], RepeatedValue]],
+ ],
+ registry: BaseRegistry,
+ project: str,
+ full_feature_names: bool = False,
+ ) -> OnlineResponse:
+ pass
+
+ @abstractmethod
+ async def get_online_features_async(
+ self,
+ config: RepoConfig,
+ features: Union[List[str], FeatureService],
+ entity_rows: Union[
+ List[Dict[str, Any]],
+ Mapping[str, Union[Sequence[Any], Sequence[ValueProto], RepeatedValue]],
+ ],
+ registry: BaseRegistry,
+ project: str,
+ full_feature_names: bool = False,
+ ) -> OnlineResponse:
+ pass
+
@abstractmethod
async def online_read_async(
self,
diff --git a/sdk/python/feast/infra/registry/base_registry.py b/sdk/python/feast/infra/registry/base_registry.py
index bc08796e39d..03bec648306 100644
--- a/sdk/python/feast/infra/registry/base_registry.py
+++ b/sdk/python/feast/infra/registry/base_registry.py
@@ -29,7 +29,19 @@
from feast.infra.infra_object import Infra
from feast.on_demand_feature_view import OnDemandFeatureView
from feast.project_metadata import ProjectMetadata
+from feast.protos.feast.core.Entity_pb2 import Entity as EntityProto
+from feast.protos.feast.core.FeatureService_pb2 import (
+ FeatureService as FeatureServiceProto,
+)
+from feast.protos.feast.core.FeatureView_pb2 import FeatureView as FeatureViewProto
+from feast.protos.feast.core.OnDemandFeatureView_pb2 import (
+ OnDemandFeatureView as OnDemandFeatureViewProto,
+)
from feast.protos.feast.core.Registry_pb2 import Registry as RegistryProto
+from feast.protos.feast.core.SavedDataset_pb2 import SavedDataset as SavedDatasetProto
+from feast.protos.feast.core.StreamFeatureView_pb2 import (
+ StreamFeatureView as StreamFeatureViewProto,
+)
from feast.saved_dataset import SavedDataset, ValidationReference
from feast.stream_feature_view import StreamFeatureView
from feast.transformation.pandas_transformation import PandasTransformation
@@ -705,3 +717,19 @@ def to_dict(self, project: str) -> Dict[str, List[Any]]:
self._message_to_sorted_dict(infra_object.to_proto())
)
return registry_dict
+
+ @staticmethod
+ def deserialize_registry_values(serialized_proto, feast_obj_type) -> Any:
+ if feast_obj_type == Entity:
+ return EntityProto.FromString(serialized_proto)
+ if feast_obj_type == SavedDataset:
+ return SavedDatasetProto.FromString(serialized_proto)
+ if feast_obj_type == FeatureView:
+ return FeatureViewProto.FromString(serialized_proto)
+ if feast_obj_type == StreamFeatureView:
+ return StreamFeatureViewProto.FromString(serialized_proto)
+ if feast_obj_type == OnDemandFeatureView:
+ return OnDemandFeatureViewProto.FromString(serialized_proto)
+ if feast_obj_type == FeatureService:
+ return FeatureServiceProto.FromString(serialized_proto)
+ return None
diff --git a/sdk/python/feast/infra/registry/caching_registry.py b/sdk/python/feast/infra/registry/caching_registry.py
index 6336dd7fee5..298639028d5 100644
--- a/sdk/python/feast/infra/registry/caching_registry.py
+++ b/sdk/python/feast/infra/registry/caching_registry.py
@@ -1,6 +1,8 @@
+import atexit
import logging
+import threading
from abc import abstractmethod
-from datetime import datetime, timedelta
+from datetime import timedelta
from threading import Lock
from typing import List, Optional
@@ -15,23 +17,24 @@
from feast.project_metadata import ProjectMetadata
from feast.saved_dataset import SavedDataset, ValidationReference
from feast.stream_feature_view import StreamFeatureView
+from feast.utils import _utc_now
logger = logging.getLogger(__name__)
class CachingRegistry(BaseRegistry):
- def __init__(
- self,
- project: str,
- cache_ttl_seconds: int,
- ):
+ def __init__(self, project: str, cache_ttl_seconds: int, cache_mode: str):
self.cached_registry_proto = self.proto()
proto_registry_utils.init_project_metadata(self.cached_registry_proto, project)
- self.cached_registry_proto_created = datetime.utcnow()
+ self.cached_registry_proto_created = _utc_now()
self._refresh_lock = Lock()
self.cached_registry_proto_ttl = timedelta(
seconds=cache_ttl_seconds if cache_ttl_seconds is not None else 0
)
+ self.cache_mode = cache_mode
+ if cache_mode == "thread":
+ self._start_thread_async_refresh(cache_ttl_seconds)
+ atexit.register(self._exit_handler)
@abstractmethod
def _get_data_source(self, name: str, project: str) -> DataSource:
@@ -318,25 +321,38 @@ def refresh(self, project: Optional[str] = None):
self.cached_registry_proto, project
)
self.cached_registry_proto = self.proto()
- self.cached_registry_proto_created = datetime.utcnow()
+ self.cached_registry_proto_created = _utc_now()
def _refresh_cached_registry_if_necessary(self):
- with self._refresh_lock:
- expired = (
- self.cached_registry_proto is None
- or self.cached_registry_proto_created is None
- ) or (
- self.cached_registry_proto_ttl.total_seconds()
- > 0 # 0 ttl means infinity
- and (
- datetime.utcnow()
- > (
- self.cached_registry_proto_created
- + self.cached_registry_proto_ttl
+ if self.cache_mode == "sync":
+ with self._refresh_lock:
+ expired = (
+ self.cached_registry_proto is None
+ or self.cached_registry_proto_created is None
+ ) or (
+ self.cached_registry_proto_ttl.total_seconds()
+ > 0 # 0 ttl means infinity
+ and (
+ _utc_now()
+ > (
+ self.cached_registry_proto_created
+ + self.cached_registry_proto_ttl
+ )
)
)
- )
+ if expired:
+ logger.info("Registry cache expired, so refreshing")
+ self.refresh()
+
+ def _start_thread_async_refresh(self, cache_ttl_seconds):
+ self.refresh()
+ if cache_ttl_seconds <= 0:
+ return
+ self.registry_refresh_thread = threading.Timer(
+ cache_ttl_seconds, self._start_thread_async_refresh, [cache_ttl_seconds]
+ )
+ self.registry_refresh_thread.setDaemon(True)
+ self.registry_refresh_thread.start()
- if expired:
- logger.info("Registry cache expired, so refreshing")
- self.refresh()
+ def _exit_handler(self):
+ self.registry_refresh_thread.cancel()
diff --git a/sdk/python/feast/infra/registry/contrib/azure/azure_registry_store.py b/sdk/python/feast/infra/registry/contrib/azure/azure_registry_store.py
index 9c00170b0f6..f9317bf7a45 100644
--- a/sdk/python/feast/infra/registry/contrib/azure/azure_registry_store.py
+++ b/sdk/python/feast/infra/registry/contrib/azure/azure_registry_store.py
@@ -3,7 +3,6 @@
import os
import uuid
-from datetime import datetime
from pathlib import Path
from tempfile import TemporaryFile
from urllib.parse import urlparse
@@ -11,6 +10,7 @@
from feast.infra.registry.registry import RegistryConfig
from feast.infra.registry.registry_store import RegistryStore
from feast.protos.feast.core.Registry_pb2 import Registry as RegistryProto
+from feast.utils import _utc_now
REGISTRY_SCHEMA_VERSION = "1"
@@ -89,7 +89,7 @@ def teardown(self):
def _write_registry(self, registry_proto: RegistryProto):
registry_proto.version_id = str(uuid.uuid4())
- registry_proto.last_updated.FromDatetime(datetime.utcnow())
+ registry_proto.last_updated.FromDatetime(_utc_now())
file_obj = TemporaryFile()
file_obj.write(registry_proto.SerializeToString())
diff --git a/sdk/python/feast/infra/registry/file.py b/sdk/python/feast/infra/registry/file.py
index 7117a0d2c6b..ae783bf82c4 100644
--- a/sdk/python/feast/infra/registry/file.py
+++ b/sdk/python/feast/infra/registry/file.py
@@ -1,10 +1,10 @@
import uuid
-from datetime import datetime
from pathlib import Path
from feast.infra.registry.registry_store import RegistryStore
from feast.protos.feast.core.Registry_pb2 import Registry as RegistryProto
from feast.repo_config import RegistryConfig
+from feast.utils import _utc_now
class FileRegistryStore(RegistryStore):
@@ -37,7 +37,7 @@ def teardown(self):
def _write_registry(self, registry_proto: RegistryProto):
registry_proto.version_id = str(uuid.uuid4())
- registry_proto.last_updated.FromDatetime(datetime.utcnow())
+ registry_proto.last_updated.FromDatetime(_utc_now())
file_dir = self._filepath.parent
file_dir.mkdir(exist_ok=True)
with open(self._filepath, mode="wb", buffering=0) as f:
diff --git a/sdk/python/feast/infra/registry/gcs.py b/sdk/python/feast/infra/registry/gcs.py
index 7e4b7104cf1..72498ad054d 100644
--- a/sdk/python/feast/infra/registry/gcs.py
+++ b/sdk/python/feast/infra/registry/gcs.py
@@ -1,5 +1,4 @@
import uuid
-from datetime import datetime
from pathlib import Path
from tempfile import TemporaryFile
from urllib.parse import urlparse
@@ -7,6 +6,7 @@
from feast.infra.registry.registry_store import RegistryStore
from feast.protos.feast.core.Registry_pb2 import Registry as RegistryProto
from feast.repo_config import RegistryConfig
+from feast.utils import _utc_now
class GCSRegistryStore(RegistryStore):
@@ -62,7 +62,7 @@ def teardown(self):
def _write_registry(self, registry_proto: RegistryProto):
registry_proto.version_id = str(uuid.uuid4())
- registry_proto.last_updated.FromDatetime(datetime.utcnow())
+ registry_proto.last_updated.FromDatetime(_utc_now())
# we have already checked the bucket exists so no need to do it again
gs_bucket = self.gcs_client.get_bucket(self._bucket)
blob = gs_bucket.blob(self._blob)
diff --git a/sdk/python/feast/infra/registry/registry.py b/sdk/python/feast/infra/registry/registry.py
index 39cdedb4906..fe44e6253a8 100644
--- a/sdk/python/feast/infra/registry/registry.py
+++ b/sdk/python/feast/infra/registry/registry.py
@@ -47,6 +47,7 @@
from feast.repo_contents import RepoContents
from feast.saved_dataset import SavedDataset, ValidationReference
from feast.stream_feature_view import StreamFeatureView
+from feast.utils import _utc_now
REGISTRY_SCHEMA_VERSION = "1"
@@ -217,7 +218,7 @@ def clone(self) -> "Registry":
if self.cached_registry_proto
else RegistryProto()
)
- new_registry.cached_registry_proto_created = datetime.utcnow()
+ new_registry.cached_registry_proto_created = _utc_now()
new_registry._registry_store = NoopRegistryStore()
return new_registry
@@ -248,7 +249,7 @@ def get_infra(self, project: str, allow_cache: bool = False) -> Infra:
def apply_entity(self, entity: Entity, project: str, commit: bool = True):
entity.is_valid()
- now = datetime.utcnow()
+ now = _utc_now()
if not entity.created_timestamp:
entity.created_timestamp = now
entity.last_updated_timestamp = now
@@ -265,9 +266,13 @@ def apply_entity(self, entity: Entity, project: str, commit: bool = True):
existing_entity_proto.spec.name == entity_proto.spec.name
and existing_entity_proto.spec.project == project
):
+ entity.created_timestamp = (
+ existing_entity_proto.meta.created_timestamp.ToDatetime()
+ )
+ entity_proto = entity.to_proto()
+ entity_proto.spec.project = project
del self.cached_registry_proto.entities[idx]
break
-
self.cached_registry_proto.entities.append(entity_proto)
if commit:
self.commit()
@@ -330,7 +335,7 @@ def delete_data_source(self, name: str, project: str, commit: bool = True):
def apply_feature_service(
self, feature_service: FeatureService, project: str, commit: bool = True
):
- now = datetime.utcnow()
+ now = _utc_now()
if not feature_service.created_timestamp:
feature_service.created_timestamp = now
feature_service.last_updated_timestamp = now
@@ -346,6 +351,11 @@ def apply_feature_service(
== feature_service_proto.spec.name
and existing_feature_service_proto.spec.project == project
):
+ feature_service.created_timestamp = (
+ existing_feature_service_proto.meta.created_timestamp.ToDatetime()
+ )
+ feature_service_proto = feature_service.to_proto()
+ feature_service_proto.spec.project = project
del registry.feature_services[idx]
registry.feature_services.append(feature_service_proto)
if commit:
@@ -381,7 +391,7 @@ def apply_feature_view(
):
feature_view.ensure_valid()
- now = datetime.utcnow()
+ now = _utc_now()
if not feature_view.created_timestamp:
feature_view.created_timestamp = now
feature_view.last_updated_timestamp = now
@@ -421,6 +431,18 @@ def apply_feature_view(
):
return
else:
+ existing_feature_view = type(feature_view).from_proto(
+ existing_feature_view_proto
+ )
+ feature_view.created_timestamp = (
+ existing_feature_view.created_timestamp
+ )
+ if isinstance(feature_view, (FeatureView, StreamFeatureView)):
+ feature_view.update_materialization_intervals(
+ existing_feature_view.materialization_intervals
+ )
+ feature_view_proto = feature_view.to_proto()
+ feature_view_proto.spec.project = project
del existing_feature_views_of_same_type[idx]
break
@@ -496,7 +518,7 @@ def apply_materialization(
existing_feature_view.materialization_intervals.append(
(start_date, end_date)
)
- existing_feature_view.last_updated_timestamp = datetime.utcnow()
+ existing_feature_view.last_updated_timestamp = _utc_now()
feature_view_proto = existing_feature_view.to_proto()
feature_view_proto.spec.project = project
del self.cached_registry_proto.feature_views[idx]
@@ -518,7 +540,7 @@ def apply_materialization(
existing_stream_feature_view.materialization_intervals.append(
(start_date, end_date)
)
- existing_stream_feature_view.last_updated_timestamp = datetime.utcnow()
+ existing_stream_feature_view.last_updated_timestamp = _utc_now()
stream_feature_view_proto = existing_stream_feature_view.to_proto()
stream_feature_view_proto.spec.project = project
del self.cached_registry_proto.stream_feature_views[idx]
@@ -643,7 +665,7 @@ def apply_saved_dataset(
project: str,
commit: bool = True,
):
- now = datetime.utcnow()
+ now = _utc_now()
if not saved_dataset.created_timestamp:
saved_dataset.created_timestamp = now
saved_dataset.last_updated_timestamp = now
@@ -660,6 +682,17 @@ def apply_saved_dataset(
existing_saved_dataset_proto.spec.name == saved_dataset_proto.spec.name
and existing_saved_dataset_proto.spec.project == project
):
+ saved_dataset.created_timestamp = (
+ existing_saved_dataset_proto.meta.created_timestamp.ToDatetime()
+ )
+ saved_dataset.min_event_timestamp = (
+ existing_saved_dataset_proto.meta.min_event_timestamp.ToDatetime()
+ )
+ saved_dataset.max_event_timestamp = (
+ existing_saved_dataset_proto.meta.max_event_timestamp.ToDatetime()
+ )
+ saved_dataset_proto = saved_dataset.to_proto()
+ saved_dataset_proto.spec.project = project
del self.cached_registry_proto.saved_datasets[idx]
break
@@ -780,7 +813,7 @@ def _prepare_registry_for_changes(self, project: str):
registry_proto = RegistryProto()
registry_proto.registry_schema_version = REGISTRY_SCHEMA_VERSION
self.cached_registry_proto = registry_proto
- self.cached_registry_proto_created = datetime.utcnow()
+ self.cached_registry_proto_created = _utc_now()
# Initialize project metadata if needed
assert self.cached_registry_proto
@@ -816,7 +849,7 @@ def _get_registry_proto(
self.cached_registry_proto_ttl.total_seconds()
> 0 # 0 ttl means infinity
and (
- datetime.utcnow()
+ _utc_now()
> (
self.cached_registry_proto_created
+ self.cached_registry_proto_ttl
@@ -839,7 +872,7 @@ def _get_registry_proto(
logger.info("Registry cache expired, so refreshing")
registry_proto = self._registry_store.get_registry_proto()
self.cached_registry_proto = registry_proto
- self.cached_registry_proto_created = datetime.utcnow()
+ self.cached_registry_proto_created = _utc_now()
if not project:
return registry_proto
diff --git a/sdk/python/feast/infra/registry/remote.py b/sdk/python/feast/infra/registry/remote.py
index 0eddf03cf64..9fa6d8ebee5 100644
--- a/sdk/python/feast/infra/registry/remote.py
+++ b/sdk/python/feast/infra/registry/remote.py
@@ -296,6 +296,7 @@ def apply_materialization(
start_date_timestamp.FromDatetime(start_date)
end_date_timestamp.FromDatetime(end_date)
+ # TODO: for this to work for stream feature views, ApplyMaterializationRequest needs to be updated
request = RegistryServer_pb2.ApplyMaterializationRequest(
feature_view=feature_view.to_proto(),
project=project,
diff --git a/sdk/python/feast/infra/registry/s3.py b/sdk/python/feast/infra/registry/s3.py
index cbae3af11cc..8aac4d52ee3 100644
--- a/sdk/python/feast/infra/registry/s3.py
+++ b/sdk/python/feast/infra/registry/s3.py
@@ -1,6 +1,5 @@
import os
import uuid
-from datetime import datetime
from pathlib import Path
from tempfile import TemporaryFile
from urllib.parse import urlparse
@@ -9,6 +8,7 @@
from feast.infra.registry.registry_store import RegistryStore
from feast.protos.feast.core.Registry_pb2 import Registry as RegistryProto
from feast.repo_config import RegistryConfig
+from feast.utils import _utc_now
try:
import boto3
@@ -70,7 +70,7 @@ def teardown(self):
def _write_registry(self, registry_proto: RegistryProto):
registry_proto.version_id = str(uuid.uuid4())
- registry_proto.last_updated.FromDatetime(datetime.utcnow())
+ registry_proto.last_updated.FromDatetime(_utc_now())
# we have already checked the bucket exists so no need to do it again
file_obj = TemporaryFile()
file_obj.write(registry_proto.SerializeToString())
diff --git a/sdk/python/feast/infra/registry/snowflake.py b/sdk/python/feast/infra/registry/snowflake.py
index aaf6c4c48dc..f2bc09e7e42 100644
--- a/sdk/python/feast/infra/registry/snowflake.py
+++ b/sdk/python/feast/infra/registry/snowflake.py
@@ -2,7 +2,7 @@
import os
import uuid
from binascii import hexlify
-from datetime import datetime, timedelta
+from datetime import datetime, timedelta, timezone
from enum import Enum
from threading import Lock
from typing import Any, Callable, List, Literal, Optional, Set, Union
@@ -10,7 +10,6 @@
from pydantic import ConfigDict, Field, StrictStr
import feast
-from feast import utils
from feast.base_feature_view import BaseFeatureView
from feast.data_source import DataSource
from feast.entity import Entity
@@ -54,6 +53,7 @@
from feast.repo_config import RegistryConfig
from feast.saved_dataset import SavedDataset, ValidationReference
from feast.stream_feature_view import StreamFeatureView
+from feast.utils import _utc_now, has_all_tags
logger = logging.getLogger(__name__)
@@ -126,16 +126,15 @@ def __init__(
with GetSnowflakeConnection(self.registry_config) as conn:
sql_function_file = f"{os.path.dirname(feast.__file__)}/infra/utils/snowflake/registry/snowflake_table_creation.sql"
with open(sql_function_file, "r") as file:
- sqlFile = file.read()
-
- sqlCommands = sqlFile.split(";")
- for command in sqlCommands:
+ sql_file = file.read()
+ sql_cmds = sql_file.split(";")
+ for command in sql_cmds:
query = command.replace("REGISTRY_PATH", f"{self.registry_path}")
execute_snowflake_statement(conn, query)
self.cached_registry_proto = self.proto()
proto_registry_utils.init_project_metadata(self.cached_registry_proto, project)
- self.cached_registry_proto_created = datetime.utcnow()
+ self.cached_registry_proto_created = _utc_now()
self._refresh_lock = Lock()
self.cached_registry_proto_ttl = timedelta(
seconds=registry_config.cache_ttl_seconds
@@ -154,7 +153,7 @@ def refresh(self, project: Optional[str] = None):
self.cached_registry_proto, project
)
self.cached_registry_proto = self.proto()
- self.cached_registry_proto_created = datetime.utcnow()
+ self.cached_registry_proto_created = _utc_now()
def _refresh_cached_registry_if_necessary(self):
with self._refresh_lock:
@@ -165,7 +164,7 @@ def _refresh_cached_registry_if_necessary(self):
self.cached_registry_proto_ttl.total_seconds()
> 0 # 0 ttl means infinity
and (
- datetime.utcnow()
+ _utc_now()
> (
self.cached_registry_proto_created
+ self.cached_registry_proto_ttl
@@ -182,7 +181,6 @@ def teardown(self):
sql_function_file = f"{os.path.dirname(feast.__file__)}/infra/utils/snowflake/registry/snowflake_table_deletion.sql"
with open(sql_function_file, "r") as file:
sqlFile = file.read()
-
sqlCommands = sqlFile.split(";")
for command in sqlCommands:
query = command.replace("REGISTRY_PATH", f"{self.registry_path}")
@@ -281,7 +279,7 @@ def _apply_object(
name = name or (obj.name if hasattr(obj, "name") else None)
assert name, f"name needs to be provided for {obj}"
- update_datetime = datetime.utcnow()
+ update_datetime = _utc_now()
if hasattr(obj, "last_updated_timestamp"):
obj.last_updated_timestamp = update_datetime
@@ -416,7 +414,7 @@ def _delete_object(
if cursor.rowcount < 1 and not_found_exception: # type: ignore
raise not_found_exception(name, project)
- self._set_last_updated_metadata(datetime.utcnow(), project)
+ self._set_last_updated_metadata(_utc_now(), project)
return cursor.rowcount
@@ -787,7 +785,7 @@ def _list_objects(
obj = python_class.from_proto(
proto_class.FromString(row[1][proto_field_name])
)
- if utils.has_all_tags(obj.tags, tags):
+ if has_all_tags(obj.tags, tags):
objects.append(obj)
return objects
return []
@@ -994,7 +992,7 @@ def _get_last_updated_metadata(self, project: str):
if df.empty:
return None
- return datetime.utcfromtimestamp(int(df.squeeze()))
+ return datetime.fromtimestamp(int(df.squeeze()), tz=timezone.utc)
def _infer_fv_classes(self, feature_view):
if isinstance(feature_view, StreamFeatureView):
diff --git a/sdk/python/feast/infra/registry/sql.py b/sdk/python/feast/infra/registry/sql.py
index d0af6872c1c..a2b16a3a091 100644
--- a/sdk/python/feast/infra/registry/sql.py
+++ b/sdk/python/feast/infra/registry/sql.py
@@ -1,6 +1,6 @@
import logging
import uuid
-from datetime import datetime
+from datetime import datetime, timezone
from enum import Enum
from pathlib import Path
from typing import Any, Callable, Dict, List, Optional, Set, Union
@@ -60,6 +60,7 @@
from feast.repo_config import RegistryConfig
from feast.saved_dataset import SavedDataset, ValidationReference
from feast.stream_feature_view import StreamFeatureView
+from feast.utils import _utc_now
metadata = MetaData()
@@ -192,7 +193,9 @@ def __init__(
)
metadata.create_all(self.engine)
super().__init__(
- project=project, cache_ttl_seconds=registry_config.cache_ttl_seconds
+ project=project,
+ cache_ttl_seconds=registry_config.cache_ttl_seconds,
+ cache_mode=registry_config.cache_mode,
)
def teardown(self):
@@ -591,7 +594,7 @@ def apply_user_metadata(
table.c.project_id == project,
)
row = conn.execute(stmt).first()
- update_datetime = datetime.utcnow()
+ update_datetime = _utc_now()
update_time = int(update_datetime.timestamp())
if row:
values = {
@@ -703,7 +706,7 @@ def _apply_object(
assert name, f"name needs to be provided for {obj}"
with self.engine.begin() as conn:
- update_datetime = datetime.utcnow()
+ update_datetime = _utc_now()
update_time = int(update_datetime.timestamp())
stmt = select(table).where(
getattr(table.c, id_field_name) == name, table.c.project_id == project
@@ -713,6 +716,24 @@ def _apply_object(
obj.last_updated_timestamp = update_datetime
if row:
+ if proto_field_name in [
+ "entity_proto",
+ "saved_dataset_proto",
+ "feature_view_proto",
+ "feature_service_proto",
+ ]:
+ deserialized_proto = self.deserialize_registry_values(
+ row._mapping[proto_field_name], type(obj)
+ )
+ obj.created_timestamp = (
+ deserialized_proto.meta.created_timestamp.ToDatetime()
+ )
+ if isinstance(obj, (FeatureView, StreamFeatureView)):
+ obj.update_materialization_intervals(
+ type(obj)
+ .from_proto(deserialized_proto)
+ .materialization_intervals
+ )
values = {
proto_field_name: obj.to_proto().SerializeToString(),
"last_updated_timestamp": update_time,
@@ -752,7 +773,7 @@ def _apply_object(
def _maybe_init_project_metadata(self, project):
# Initialize project metadata if needed
with self.engine.begin() as conn:
- update_datetime = datetime.utcnow()
+ update_datetime = _utc_now()
update_time = int(update_datetime.timestamp())
stmt = select(feast_metadata).where(
feast_metadata.c.metadata_key == FeastMetadataKeys.PROJECT_UUID.value,
@@ -785,7 +806,7 @@ def _delete_object(
rows = conn.execute(stmt)
if rows.rowcount < 1 and not_found_exception:
raise not_found_exception(name, project)
- self._set_last_updated_metadata(datetime.utcnow(), project)
+ self._set_last_updated_metadata(_utc_now(), project)
return rows.rowcount
@@ -885,7 +906,7 @@ def _get_last_updated_metadata(self, project: str):
return None
update_time = int(row._mapping["last_updated_timestamp"])
- return datetime.utcfromtimestamp(update_time)
+ return datetime.fromtimestamp(update_time, tz=timezone.utc)
def _get_all_projects(self) -> Set[str]:
projects = set()
diff --git a/sdk/python/feast/infra/utils/postgres/connection_utils.py b/sdk/python/feast/infra/utils/postgres/connection_utils.py
index 0d99c8ab993..7b37ea981f4 100644
--- a/sdk/python/feast/infra/utils/postgres/connection_utils.py
+++ b/sdk/python/feast/infra/utils/postgres/connection_utils.py
@@ -1,50 +1,80 @@
-from typing import Dict
+from typing import Any, Dict
import numpy as np
import pandas as pd
-import psycopg2
-import psycopg2.extras
+import psycopg
import pyarrow as pa
-from psycopg2.pool import SimpleConnectionPool
+from psycopg import AsyncConnection, Connection
+from psycopg_pool import AsyncConnectionPool, ConnectionPool
from feast.infra.utils.postgres.postgres_config import PostgreSQLConfig
from feast.type_map import arrow_to_pg_type
-def _get_conn(config: PostgreSQLConfig):
- conn = psycopg2.connect(
- dbname=config.database,
- host=config.host,
- port=int(config.port),
- user=config.user,
- password=config.password,
- sslmode=config.sslmode,
- sslkey=config.sslkey_path,
- sslcert=config.sslcert_path,
- sslrootcert=config.sslrootcert_path,
- options="-c search_path={}".format(config.db_schema or config.user),
+def _get_conn(config: PostgreSQLConfig) -> Connection:
+ """Get a psycopg `Connection`."""
+ conn = psycopg.connect(
+ conninfo=_get_conninfo(config),
keepalives_idle=config.keepalives_idle,
+ **_get_conn_kwargs(config),
)
return conn
-def _get_connection_pool(config: PostgreSQLConfig):
- return SimpleConnectionPool(
- config.min_conn,
- config.max_conn,
- dbname=config.database,
- host=config.host,
- port=int(config.port),
- user=config.user,
- password=config.password,
- sslmode=config.sslmode,
- sslkey=config.sslkey_path,
- sslcert=config.sslcert_path,
- sslrootcert=config.sslrootcert_path,
- options="-c search_path={}".format(config.db_schema or config.user),
+async def _get_conn_async(config: PostgreSQLConfig) -> AsyncConnection:
+ """Get a psycopg `AsyncConnection`."""
+ conn = await psycopg.AsyncConnection.connect(
+ conninfo=_get_conninfo(config),
+ keepalives_idle=config.keepalives_idle,
+ **_get_conn_kwargs(config),
+ )
+ return conn
+
+
+def _get_connection_pool(config: PostgreSQLConfig) -> ConnectionPool:
+ """Get a psycopg `ConnectionPool`."""
+ return ConnectionPool(
+ conninfo=_get_conninfo(config),
+ min_size=config.min_conn,
+ max_size=config.max_conn,
+ open=False,
+ kwargs=_get_conn_kwargs(config),
+ )
+
+
+async def _get_connection_pool_async(config: PostgreSQLConfig) -> AsyncConnectionPool:
+ """Get a psycopg `AsyncConnectionPool`."""
+ return AsyncConnectionPool(
+ conninfo=_get_conninfo(config),
+ min_size=config.min_conn,
+ max_size=config.max_conn,
+ open=False,
+ kwargs=_get_conn_kwargs(config),
)
+def _get_conninfo(config: PostgreSQLConfig) -> str:
+ """Get the `conninfo` argument required for connection objects."""
+ return (
+ f"postgresql://{config.user}"
+ f":{config.password}"
+ f"@{config.host}"
+ f":{int(config.port)}"
+ f"/{config.database}"
+ )
+
+
+def _get_conn_kwargs(config: PostgreSQLConfig) -> Dict[str, Any]:
+ """Get the additional `kwargs` required for connection objects."""
+ return {
+ "sslmode": config.sslmode,
+ "sslkey": config.sslkey_path,
+ "sslcert": config.sslcert_path,
+ "sslrootcert": config.sslrootcert_path,
+ "options": "-c search_path={}".format(config.db_schema or config.user),
+ }
+
+
def _df_to_create_table_sql(entity_df, table_name) -> str:
pa_table = pa.Table.from_pandas(entity_df)
columns = [
@@ -63,16 +93,14 @@ def df_to_postgres_table(
"""
Create a table for the data frame, insert all the values, and return the table schema
"""
+ nr_columns = df.shape[1]
+ placeholders = ", ".join(["%s"] * nr_columns)
+ query = f"INSERT INTO {table_name} VALUES ({placeholders})"
+ values = df.replace({np.NaN: None}).to_numpy().tolist()
+
with _get_conn(config) as conn, conn.cursor() as cur:
cur.execute(_df_to_create_table_sql(df, table_name))
- psycopg2.extras.execute_values(
- cur,
- f"""
- INSERT INTO {table_name}
- VALUES %s
- """,
- df.replace({np.NaN: None}).to_numpy(),
- )
+ cur.executemany(query, values)
return dict(zip(df.columns, df.dtypes))
@@ -82,7 +110,7 @@ def get_query_schema(config: PostgreSQLConfig, sql_query: str) -> Dict[str, np.d
new table
"""
with _get_conn(config) as conn:
- conn.set_session(readonly=True)
+ conn.read_only = True
df = pd.read_sql(
f"SELECT * FROM {sql_query} LIMIT 0",
conn,
diff --git a/sdk/python/feast/offline_server.py b/sdk/python/feast/offline_server.py
index 718da1b109f..be92620d682 100644
--- a/sdk/python/feast/offline_server.py
+++ b/sdk/python/feast/offline_server.py
@@ -74,14 +74,15 @@ def do_put(
logger.debug(f"do_put: command is{command}, data is {data}")
self.flights[key] = data
- self._call_api(command, key)
+ self._call_api(command["api"], command, key)
else:
logger.warning(f"No 'api' field in command: {command}")
- def _call_api(self, command: dict, key: str):
+ def _call_api(self, api: str, command: dict, key: str):
+ assert api is not None, "api can not be empty"
+
remove_data = False
try:
- api = command["api"]
if api == OfflineServer.offline_write_batch.__name__:
self.offline_write_batch(command, key)
remove_data = True
@@ -89,7 +90,7 @@ def _call_api(self, command: dict, key: str):
self.write_logged_features(command, key)
remove_data = True
elif api == OfflineServer.persist.__name__:
- self.persist(command["retrieve_func"], command, key)
+ self.persist(command, key)
remove_data = True
except Exception as e:
remove_data = True
@@ -150,6 +151,9 @@ def list_feature_views_by_name(
for index, fv_name in enumerate(feature_view_names)
]
+ def _validate_do_get_parameters(self, command: dict):
+ assert "api" in command, "api parameter is mandatory"
+
# Extracts the API parameters from the flights dictionary, delegates the execution to the FeatureStore instance
# and returns the stream of data
def do_get(self, context: fl.ServerCallContext, ticket: fl.Ticket):
@@ -159,6 +163,9 @@ def do_get(self, context: fl.ServerCallContext, ticket: fl.Ticket):
return None
command = json.loads(key[1])
+
+ self._validate_do_get_parameters(command)
+
api = command["api"]
logger.debug(f"get command is {command}")
logger.debug(f"requested api is {api}")
@@ -180,13 +187,26 @@ def do_get(self, context: fl.ServerCallContext, ticket: fl.Ticket):
del self.flights[key]
return fl.RecordBatchStream(table)
- def offline_write_batch(self, command: dict, key: str):
+ def _validate_offline_write_batch_parameters(self, command: dict):
+ assert (
+ "feature_view_names" in command
+ ), "feature_view_names is a mandatory parameter"
+ assert "name_aliases" in command, "name_aliases is a mandatory parameter"
+
feature_view_names = command["feature_view_names"]
assert (
len(feature_view_names) == 1
), "feature_view_names list should only have one item"
+
name_aliases = command["name_aliases"]
assert len(name_aliases) == 1, "name_aliases list should only have one item"
+
+ def offline_write_batch(self, command: dict, key: str):
+ self._validate_offline_write_batch_parameters(command)
+
+ feature_view_names = command["feature_view_names"]
+ name_aliases = command["name_aliases"]
+
project = self.store.config.project
feature_views = self.list_feature_views_by_name(
feature_view_names=feature_view_names,
@@ -194,19 +214,25 @@ def offline_write_batch(self, command: dict, key: str):
project=project,
)
- assert len(feature_views) == 1
+ assert len(feature_views) == 1, "incorrect feature view"
table = self.flights[key]
self.offline_store.offline_write_batch(
self.store.config, feature_views[0], table, command["progress"]
)
+ def _validate_write_logged_features_parameters(self, command: dict):
+ assert "feature_service_name" in command
+
def write_logged_features(self, command: dict, key: str):
+ self._validate_write_logged_features_parameters(command)
table = self.flights[key]
feature_service = self.store.get_feature_service(
command["feature_service_name"]
)
- assert feature_service.logging_config is not None
+ assert (
+ feature_service.logging_config is not None
+ ), "feature service must have logging_config set"
self.offline_store.write_logged_features(
config=self.store.config,
@@ -218,7 +244,23 @@ def write_logged_features(self, command: dict, key: str):
registry=self.store.registry,
)
+ def _validate_pull_all_from_table_or_query_parameters(self, command: dict):
+ assert (
+ "data_source_name" in command
+ ), "data_source_name is a mandatory parameter"
+ assert (
+ "join_key_columns" in command
+ ), "join_key_columns is a mandatory parameter"
+ assert (
+ "feature_name_columns" in command
+ ), "feature_name_columns is a mandatory parameter"
+ assert "timestamp_field" in command, "timestamp_field is a mandatory parameter"
+ assert "start_date" in command, "start_date is a mandatory parameter"
+ assert "end_date" in command, "end_date is a mandatory parameter"
+
def pull_all_from_table_or_query(self, command: dict):
+ self._validate_pull_all_from_table_or_query_parameters(command)
+
return self.offline_store.pull_all_from_table_or_query(
self.store.config,
self.store.get_data_source(command["data_source_name"]),
@@ -229,7 +271,23 @@ def pull_all_from_table_or_query(self, command: dict):
utils.make_tzaware(datetime.fromisoformat(command["end_date"])),
)
+ def _validate_pull_latest_from_table_or_query_parameters(self, command: dict):
+ assert (
+ "data_source_name" in command
+ ), "data_source_name is a mandatory parameter"
+ assert (
+ "join_key_columns" in command
+ ), "join_key_columns is a mandatory parameter"
+ assert (
+ "feature_name_columns" in command
+ ), "feature_name_columns is a mandatory parameter"
+ assert "timestamp_field" in command, "timestamp_field is a mandatory parameter"
+ assert "start_date" in command, "start_date is a mandatory parameter"
+ assert "end_date" in command, "end_date is a mandatory parameter"
+
def pull_latest_from_table_or_query(self, command: dict):
+ self._validate_pull_latest_from_table_or_query_parameters(command)
+
return self.offline_store.pull_latest_from_table_or_query(
self.store.config,
self.store.get_data_source(command["data_source_name"]),
@@ -258,20 +316,33 @@ def list_actions(self, context):
),
]
+ def _validate_get_historical_features_parameters(self, command: dict, key: str):
+ assert key in self.flights, f"missing key={key}"
+ assert "feature_view_names" in command, "feature_view_names is mandatory"
+ assert "name_aliases" in command, "name_aliases is mandatory"
+ assert "feature_refs" in command, "feature_refs is mandatory"
+ assert "project" in command, "project is mandatory"
+ assert "full_feature_names" in command, "full_feature_names is mandatory"
+
def get_historical_features(self, command: dict, key: str):
+ self._validate_get_historical_features_parameters(command, key)
+
# Extract parameters from the internal flights dictionary
entity_df_value = self.flights[key]
entity_df = pa.Table.to_pandas(entity_df_value)
+
feature_view_names = command["feature_view_names"]
name_aliases = command["name_aliases"]
feature_refs = command["feature_refs"]
project = command["project"]
full_feature_names = command["full_feature_names"]
+
feature_views = self.list_feature_views_by_name(
feature_view_names=feature_view_names,
name_aliases=name_aliases,
project=project,
)
+
retJob = self.offline_store.get_historical_features(
config=self.store.config,
feature_views=feature_views,
@@ -281,10 +352,19 @@ def get_historical_features(self, command: dict, key: str):
project=project,
full_feature_names=full_feature_names,
)
+
return retJob
- def persist(self, retrieve_func: str, command: dict, key: str):
+ def _validate_persist_parameters(self, command: dict):
+ assert "retrieve_func" in command, "retrieve_func is mandatory"
+ assert "data_source_name" in command, "data_source_name is mandatory"
+ assert "allow_overwrite" in command, "allow_overwrite is mandatory"
+
+ def persist(self, command: dict, key: str):
+ self._validate_persist_parameters(command)
+
try:
+ retrieve_func = command["retrieve_func"]
if retrieve_func == OfflineServer.get_historical_features.__name__:
ret_job = self.get_historical_features(command, key)
elif (
diff --git a/sdk/python/feast/on_demand_feature_view.py b/sdk/python/feast/on_demand_feature_view.py
index 839ce4d64ca..586f5d1bac9 100644
--- a/sdk/python/feast/on_demand_feature_view.py
+++ b/sdk/python/feast/on_demand_feature_view.py
@@ -2,7 +2,6 @@
import functools
import inspect
import warnings
-from datetime import datetime
from types import FunctionType
from typing import Any, Optional, Union
@@ -34,6 +33,7 @@
from feast.transformation.pandas_transformation import PandasTransformation
from feast.transformation.python_transformation import PythonTransformation
from feast.transformation.substrait_transformation import SubstraitTransformation
+from feast.utils import _utc_now
from feast.value_type import ValueType
warnings.simplefilter("once", DeprecationWarning)
@@ -549,7 +549,7 @@ def _construct_random_input(self) -> dict[str, list[Any]]:
ValueType.DOUBLE: [1.0],
ValueType.FLOAT: [1.0],
ValueType.BOOL: [True],
- ValueType.UNIX_TIMESTAMP: [datetime.utcnow()],
+ ValueType.UNIX_TIMESTAMP: [_utc_now()],
ValueType.BYTES_LIST: [[str.encode("hello world")]],
ValueType.STRING_LIST: [["hello world"]],
ValueType.INT32_LIST: [[1]],
@@ -557,7 +557,7 @@ def _construct_random_input(self) -> dict[str, list[Any]]:
ValueType.DOUBLE_LIST: [[1.0]],
ValueType.FLOAT_LIST: [[1.0]],
ValueType.BOOL_LIST: [[True]],
- ValueType.UNIX_TIMESTAMP_LIST: [[datetime.utcnow()]],
+ ValueType.UNIX_TIMESTAMP_LIST: [[_utc_now()]],
}
feature_dict = {}
diff --git a/sdk/python/feast/registry_server.py b/sdk/python/feast/registry_server.py
index 1b6798b022c..4a96ba76a8e 100644
--- a/sdk/python/feast/registry_server.py
+++ b/sdk/python/feast/registry_server.py
@@ -3,6 +3,7 @@
import grpc
from google.protobuf.empty_pb2 import Empty
+from pytz import utc
from feast import FeatureStore
from feast.data_source import DataSource
@@ -313,10 +314,10 @@ def ApplyMaterialization(
feature_view=FeatureView.from_proto(request.feature_view),
project=request.project,
start_date=datetime.fromtimestamp(
- request.start_date.seconds + request.start_date.nanos / 1e9
+ request.start_date.seconds + request.start_date.nanos / 1e9, tz=utc
),
end_date=datetime.fromtimestamp(
- request.end_date.seconds + request.end_date.nanos / 1e9
+ request.end_date.seconds + request.end_date.nanos / 1e9, tz=utc
),
commit=request.commit,
)
diff --git a/sdk/python/feast/repo_config.py b/sdk/python/feast/repo_config.py
index d5b3160b566..fc2792e3237 100644
--- a/sdk/python/feast/repo_config.py
+++ b/sdk/python/feast/repo_config.py
@@ -12,16 +12,15 @@
StrictInt,
StrictStr,
ValidationError,
+ ValidationInfo,
field_validator,
model_validator,
)
from feast.errors import (
FeastFeatureServerTypeInvalidError,
- FeastFeatureServerTypeSetError,
FeastOfflineStoreInvalidName,
FeastOnlineStoreInvalidName,
- FeastProviderNotSetError,
FeastRegistryNotSetError,
FeastRegistryTypeInvalidError,
)
@@ -65,10 +64,12 @@
"ikv": "feast.infra.online_stores.contrib.ikv_online_store.ikv.IKVOnlineStore",
"elasticsearch": "feast.infra.online_stores.contrib.elasticsearch.ElasticSearchOnlineStore",
"remote": "feast.infra.online_stores.remote.RemoteOnlineStore",
+ "singlestore": "feast.infra.online_stores.contrib.singlestore_online_store.singlestore.SingleStoreOnlineStore",
}
OFFLINE_STORE_CLASS_FOR_TYPE = {
- "file": "feast.infra.offline_stores.file.FileOfflineStore",
+ "file": "feast.infra.offline_stores.dask.DaskOfflineStore",
+ "dask": "feast.infra.offline_stores.dask.DaskOfflineStore",
"bigquery": "feast.infra.offline_stores.bigquery.BigQueryOfflineStore",
"redshift": "feast.infra.offline_stores.redshift.RedshiftOfflineStore",
"snowflake.offline": "feast.infra.offline_stores.snowflake.SnowflakeOfflineStore",
@@ -82,17 +83,9 @@
}
FEATURE_SERVER_CONFIG_CLASS_FOR_TYPE = {
- "aws_lambda": "feast.infra.feature_servers.aws_lambda.config.AwsLambdaFeatureServerConfig",
- "gcp_cloudrun": "feast.infra.feature_servers.gcp_cloudrun.config.GcpCloudRunFeatureServerConfig",
"local": "feast.infra.feature_servers.local_process.config.LocalFeatureServerConfig",
}
-FEATURE_SERVER_TYPE_FOR_PROVIDER = {
- "aws": "aws_lambda",
- "gcp": "gcp_cloudrun",
- "local": "local",
-}
-
class FeastBaseModel(BaseModel):
"""Feast Pydantic Configuration Class"""
@@ -132,6 +125,24 @@ class RegistryConfig(FeastBaseModel):
sqlalchemy_config_kwargs: Dict[str, Any] = {}
""" Dict[str, Any]: Extra arguments to pass to SQLAlchemy.create_engine. """
+ cache_mode: StrictStr = "sync"
+ """ str: Cache mode type, Possible options are sync and thread(asynchronous caching using threading library)"""
+
+ @field_validator("path")
+ def validate_path(cls, path: str, values: ValidationInfo) -> str:
+ if values.data.get("registry_type") == "sql":
+ if path.startswith("postgresql://"):
+ _logger.warning(
+ "The `path` of the `RegistryConfig` starts with a plain "
+ "`postgresql` string. We are updating this to `postgresql+psycopg` "
+ "to ensure that the `psycopg3` driver is used by `sqlalchemy`. If "
+ "you want to use `psycopg2` pass `postgresql+psycopg2` explicitely "
+ "to `path`. To silence this warning, pass `postgresql+psycopg` "
+ "explicitely to `path`."
+ )
+ return path.replace("postgresql://", "postgresql+psycopg://")
+ return path
+
class RepoConfig(FeastBaseModel):
"""Repo config. Typically loaded from `feature_store.yaml`"""
@@ -142,7 +153,7 @@ class RepoConfig(FeastBaseModel):
provider account, as long as they have different project ids.
"""
- provider: StrictStr
+ provider: StrictStr = "local"
""" str: local or gcp or aws """
registry_config: Any = Field(alias="registry", default="data/registry.db")
@@ -175,10 +186,12 @@ class RepoConfig(FeastBaseModel):
used when writing data to the online store.
A value <= 1 uses the serialization scheme used by feast up to Feast 0.22.
A value of 2 uses a newer serialization scheme, supported as of Feast 0.23.
- The main difference between the two scheme is that the serialization scheme v1 stored `long` values as `int`s,
- which would result in errors trying to serialize a range of values.
- v2 fixes this error, but v1 is kept around to ensure backwards compatibility - specifically the ability to read
+ A value of 3 uses the latest serialization scheme, supported as of Feast 0.38.
+ The main difference between the three schema is that
+ v1: the serialization scheme v1 stored `long` values as `int`s, which would result in errors trying to serialize a range of values.
+ v2: fixes this error, but v1 is kept around to ensure backwards compatibility - specifically the ability to read
feature values for entities that have already been written into the online store.
+ v3: add entity_key value length to serialized bytes to enable deserialization, which can be used in retrieval of entity_key in document retrieval.
"""
coerce_tz_aware: Optional[bool] = True
@@ -193,30 +206,10 @@ def __init__(self, **data: Any):
self.registry_config = data["registry"]
self._offline_store = None
- if "offline_store" in data:
- self.offline_config = data["offline_store"]
- else:
- if data["provider"] == "local":
- self.offline_config = "file"
- elif data["provider"] == "gcp":
- self.offline_config = "bigquery"
- elif data["provider"] == "aws":
- self.offline_config = "redshift"
- elif data["provider"] == "azure":
- self.offline_config = "mssql"
+ self.offline_config = data.get("offline_store", "dask")
self._online_store = None
- if "online_store" in data:
- self.online_config = data["online_store"]
- else:
- if data["provider"] == "local":
- self.online_config = "sqlite"
- elif data["provider"] == "gcp":
- self.online_config = "datastore"
- elif data["provider"] == "aws":
- self.online_config = "dynamodb"
- elif data["provider"] == "rockset":
- self.online_config = "rockset"
+ self.online_config = data.get("online_store", "sqlite")
self._batch_engine = None
if "batch_engine" in data:
@@ -327,20 +320,11 @@ def _validate_online_store_config(cls, values: Any) -> Any:
values["online_store"] = None
return values
- # Make sure that the provider configuration is set. We need it to set the defaults
- if "provider" not in values:
- raise FeastProviderNotSetError()
-
# Set the default type
# This is only direct reference to a provider or online store that we should have
# for backwards compatibility.
if "type" not in values["online_store"]:
- if values["provider"] == "local":
- values["online_store"]["type"] = "sqlite"
- elif values["provider"] == "gcp":
- values["online_store"]["type"] = "datastore"
- elif values["provider"] == "aws":
- values["online_store"]["type"] = "dynamodb"
+ values["online_store"]["type"] = "sqlite"
online_store_type = values["online_store"]["type"]
@@ -363,20 +347,9 @@ def _validate_offline_store_config(cls, values: Any) -> Any:
if not isinstance(values["offline_store"], Dict):
return values
- # Make sure that the provider configuration is set. We need it to set the defaults
- if "provider" not in values:
- raise FeastProviderNotSetError()
-
# Set the default type
if "type" not in values["offline_store"]:
- if values["provider"] == "local":
- values["offline_store"]["type"] = "file"
- elif values["provider"] == "gcp":
- values["offline_store"]["type"] = "bigquery"
- elif values["provider"] == "aws":
- values["offline_store"]["type"] = "redshift"
- if values["provider"] == "azure":
- values["offline_store"]["type"] = "mssql"
+ values["offline_store"]["type"] = "dask"
offline_store_type = values["offline_store"]["type"]
@@ -400,15 +373,7 @@ def _validate_feature_server_config(cls, values: Any) -> Any:
if not isinstance(values["feature_server"], Dict):
return values
- # Make sure that the provider configuration is set. We need it to set the defaults
- if "provider" not in values:
- raise FeastProviderNotSetError()
-
- default_type = FEATURE_SERVER_TYPE_FOR_PROVIDER.get(values["provider"])
- defined_type = values["feature_server"].get("type", default_type)
- # Make sure that the type is either not set, or set correctly, since it's defined by the provider
- if defined_type not in (default_type, "local"):
- raise FeastFeatureServerTypeSetError(defined_type)
+ defined_type = values["feature_server"].get("type", "local")
values["feature_server"]["type"] = defined_type
# Validate the dict to ensure one of the union types match
diff --git a/sdk/python/feast/repo_operations.py b/sdk/python/feast/repo_operations.py
index 274a0af02b0..a3100ca9d7e 100644
--- a/sdk/python/feast/repo_operations.py
+++ b/sdk/python/feast/repo_operations.py
@@ -83,6 +83,14 @@ def get_repo_files(repo_root: Path) -> List[Path]:
# Read ignore paths from .feastignore and create a set of all files that match any of these paths
ignore_paths = read_feastignore(repo_root)
ignore_files = get_ignore_files(repo_root, ignore_paths)
+ ignore_paths += [
+ ".git",
+ ".feastignore",
+ ".venv",
+ ".pytest_cache",
+ "__pycache__",
+ ".ipynb_checkpoints",
+ ]
# List all Python files in the root directory (recursively)
repo_files = {
@@ -376,8 +384,8 @@ def cli_check_repo(repo_path: Path, fs_yaml_file: Path):
def init_repo(repo_name: str, template: str):
import os
- from distutils.dir_util import copy_tree
from pathlib import Path
+ from shutil import copytree
from colorama import Fore, Style
@@ -404,7 +412,7 @@ def init_repo(repo_name: str, template: str):
template_path = str(Path(Path(__file__).parent / "templates" / template).absolute())
if not os.path.exists(template_path):
raise IOError(f"Could not find template {template}")
- copy_tree(template_path, str(repo_path))
+ copytree(template_path, str(repo_path), dirs_exist_ok=True)
# Seed the repository
bootstrap_path = repo_path / "bootstrap.py"
diff --git a/sdk/python/feast/templates/postgres/bootstrap.py b/sdk/python/feast/templates/postgres/bootstrap.py
index 9f6e8a988d6..6ed13e4e39a 100644
--- a/sdk/python/feast/templates/postgres/bootstrap.py
+++ b/sdk/python/feast/templates/postgres/bootstrap.py
@@ -1,5 +1,5 @@
import click
-import psycopg2
+import psycopg
from feast.file_utils import replace_str_in_file
from feast.infra.utils.postgres.connection_utils import df_to_postgres_table
@@ -34,12 +34,14 @@ def bootstrap():
'Should I upload example data to Postgres (overwriting "feast_driver_hourly_stats" table)?',
default=True,
):
- db_connection = psycopg2.connect(
- dbname=postgres_database,
- host=postgres_host,
- port=int(postgres_port),
- user=postgres_user,
- password=postgres_password,
+ db_connection = psycopg.connect(
+ conninfo=(
+ f"postgresql://{postgres_user}"
+ f":{postgres_password}"
+ f"@{postgres_host}"
+ f":{int(postgres_port)}"
+ f"/{postgres_database}"
+ ),
options=f"-c search_path={postgres_schema}",
)
diff --git a/sdk/python/feast/transformation/pandas_transformation.py b/sdk/python/feast/transformation/pandas_transformation.py
index e9dab721608..41e437fb6b7 100644
--- a/sdk/python/feast/transformation/pandas_transformation.py
+++ b/sdk/python/feast/transformation/pandas_transformation.py
@@ -40,15 +40,27 @@ def infer_features(self, random_input: dict[str, list[Any]]) -> list[Field]:
df = pd.DataFrame.from_dict(random_input)
output_df: pd.DataFrame = self.transform(df)
- return [
- Field(
- name=f,
- dtype=from_value_type(
- python_type_to_feast_value_type(f, type_name=str(dt))
- ),
+ fields = []
+ for feature_name, feature_type in zip(output_df.columns, output_df.dtypes):
+ feature_value = output_df[feature_name].tolist()
+ if len(feature_value) <= 0:
+ raise TypeError(
+ f"Failed to infer type for feature '{feature_name}' with value "
+ + f"'{feature_value}' since no items were returned by the UDF."
+ )
+ fields.append(
+ Field(
+ name=feature_name,
+ dtype=from_value_type(
+ python_type_to_feast_value_type(
+ feature_name,
+ value=feature_value[0],
+ type_name=str(feature_type),
+ )
+ ),
+ )
)
- for f, dt in zip(output_df.columns, output_df.dtypes)
- ]
+ return fields
def __eq__(self, other):
if not isinstance(other, PandasTransformation):
diff --git a/sdk/python/feast/transformation/python_transformation.py b/sdk/python/feast/transformation/python_transformation.py
index 2a9c7db8763..d828890b1e1 100644
--- a/sdk/python/feast/transformation/python_transformation.py
+++ b/sdk/python/feast/transformation/python_transformation.py
@@ -40,15 +40,26 @@ def transform(self, input_dict: dict) -> dict:
def infer_features(self, random_input: dict[str, list[Any]]) -> list[Field]:
output_dict: dict[str, list[Any]] = self.transform(random_input)
- return [
- Field(
- name=f,
- dtype=from_value_type(
- python_type_to_feast_value_type(f, type_name=type(dt[0]).__name__)
- ),
+ fields = []
+ for feature_name, feature_value in output_dict.items():
+ if len(feature_value) <= 0:
+ raise TypeError(
+ f"Failed to infer type for feature '{feature_name}' with value "
+ + f"'{feature_value}' since no items were returned by the UDF."
+ )
+ fields.append(
+ Field(
+ name=feature_name,
+ dtype=from_value_type(
+ python_type_to_feast_value_type(
+ feature_name,
+ value=feature_value[0],
+ type_name=type(feature_value[0]).__name__,
+ )
+ ),
+ )
)
- for f, dt in output_dict.items()
- ]
+ return fields
def __eq__(self, other):
if not isinstance(other, PythonTransformation):
diff --git a/sdk/python/feast/transformation/substrait_transformation.py b/sdk/python/feast/transformation/substrait_transformation.py
index 17c40cf0a16..1de60aed00a 100644
--- a/sdk/python/feast/transformation/substrait_transformation.py
+++ b/sdk/python/feast/transformation/substrait_transformation.py
@@ -60,16 +60,28 @@ def infer_features(self, random_input: dict[str, list[Any]]) -> list[Field]:
df = pd.DataFrame.from_dict(random_input)
output_df: pd.DataFrame = self.transform(df)
- return [
- Field(
- name=f,
- dtype=from_value_type(
- python_type_to_feast_value_type(f, type_name=str(dt))
- ),
- )
- for f, dt in zip(output_df.columns, output_df.dtypes)
- if f not in random_input
- ]
+ fields = []
+ for feature_name, feature_type in zip(output_df.columns, output_df.dtypes):
+ feature_value = output_df[feature_name].tolist()
+ if len(feature_value) <= 0:
+ raise TypeError(
+ f"Failed to infer type for feature '{feature_name}' with value "
+ + f"'{feature_value}' since no items were returned by the UDF."
+ )
+ if feature_name not in random_input:
+ fields.append(
+ Field(
+ name=feature_name,
+ dtype=from_value_type(
+ python_type_to_feast_value_type(
+ feature_name,
+ value=feature_value[0],
+ type_name=str(feature_type),
+ )
+ ),
+ )
+ )
+ return fields
def __eq__(self, other):
if not isinstance(other, SubstraitTransformation):
diff --git a/sdk/python/feast/type_map.py b/sdk/python/feast/type_map.py
index a0859f2f7ad..4e9b54c6316 100644
--- a/sdk/python/feast/type_map.py
+++ b/sdk/python/feast/type_map.py
@@ -155,13 +155,15 @@ def python_type_to_feast_value_type(
"uint16": ValueType.INT32,
"uint8": ValueType.INT32,
"int8": ValueType.INT32,
+ "bool_": ValueType.BOOL, # np.bool_
"bool": ValueType.BOOL,
"boolean": ValueType.BOOL,
"timedelta": ValueType.UNIX_TIMESTAMP,
"timestamp": ValueType.UNIX_TIMESTAMP,
"datetime": ValueType.UNIX_TIMESTAMP,
"datetime64[ns]": ValueType.UNIX_TIMESTAMP,
- "datetime64[ns, tz]": ValueType.UNIX_TIMESTAMP,
+ "datetime64[ns, tz]": ValueType.UNIX_TIMESTAMP, # special dtype of pandas
+ "datetime64[ns, utc]": ValueType.UNIX_TIMESTAMP,
"category": ValueType.STRING,
}
@@ -598,7 +600,9 @@ def mssql_to_feast_value_type(mssql_type_as_str: str) -> ValueType:
"char": ValueType.STRING,
"date": ValueType.UNIX_TIMESTAMP,
"datetime": ValueType.UNIX_TIMESTAMP,
+ "datetimeoffset": ValueType.UNIX_TIMESTAMP,
"float": ValueType.FLOAT,
+ "int": ValueType.INT32,
"nchar": ValueType.STRING,
"nvarchar": ValueType.STRING,
"nvarchar(max)": ValueType.STRING,
diff --git a/sdk/python/feast/ui/package.json b/sdk/python/feast/ui/package.json
index 66daf7b993e..36777ca0be3 100644
--- a/sdk/python/feast/ui/package.json
+++ b/sdk/python/feast/ui/package.json
@@ -6,7 +6,7 @@
"@elastic/datemath": "^5.0.3",
"@elastic/eui": "^55.0.1",
"@emotion/react": "^11.9.0",
- "@feast-dev/feast-ui": "0.39.0",
+ "@feast-dev/feast-ui": "0.40.0",
"@testing-library/jest-dom": "^5.16.4",
"@testing-library/react": "^13.2.0",
"@testing-library/user-event": "^13.5.0",
diff --git a/sdk/python/feast/ui/yarn.lock b/sdk/python/feast/ui/yarn.lock
index 005035db2d1..cd1913bbb1f 100644
--- a/sdk/python/feast/ui/yarn.lock
+++ b/sdk/python/feast/ui/yarn.lock
@@ -1451,10 +1451,10 @@
minimatch "^3.1.2"
strip-json-comments "^3.1.1"
-"@feast-dev/feast-ui@0.39.0":
- version "0.39.0"
- resolved "https://registry.yarnpkg.com/@feast-dev/feast-ui/-/feast-ui-0.39.0.tgz#9ab9bdcfd866399383b489f192e3d907590ac841"
- integrity sha512-ggTyiv+D/i6sF5WZRxEFmVKMVgWmrdP3bnUzbDYnMpJ6A1UKFOdj29Ukh4F8DXDvrAskV1LjF+DZVkaD5lF4TQ==
+"@feast-dev/feast-ui@0.40.0":
+ version "0.40.0"
+ resolved "https://registry.yarnpkg.com/@feast-dev/feast-ui/-/feast-ui-0.40.0.tgz#0dc60cbbd4f63d161927321c0bbf57bbfe6b7d09"
+ integrity sha512-jiCtMYCBvNSfHCjemFRa0NFIIAR5y6spWBnUZyc4GXY2YxGcznw+PZSzOoi7JrOwpNzNPB0PTBUqJgBAxus20w==
dependencies:
"@elastic/datemath" "^5.0.3"
"@elastic/eui" "^55.0.1"
@@ -3560,11 +3560,11 @@ brace-expansion@^2.0.1:
balanced-match "^1.0.0"
braces@^3.0.2, braces@~3.0.2:
- version "3.0.2"
- resolved "https://registry.yarnpkg.com/braces/-/braces-3.0.2.tgz#3454e1a462ee8d599e236df336cd9ea4f8afe107"
- integrity sha512-b8um+L1RzM3WDSzvhm6gIz1yfTbBt6YTlcEKAvsmqCZZFw46z626lVj9j1yEPW33H5H+lBQpZMP1k8l+78Ha0A==
+ version "3.0.3"
+ resolved "https://registry.yarnpkg.com/braces/-/braces-3.0.3.tgz#490332f40919452272d55a8480adc0c441358789"
+ integrity sha512-yQbXgO/OSZVD2IsiLlro+7Hf6Q18EJrKSEsdoMzKePKXct3gvD8oLcOQdIzGupr5Fj+EDe8gO/lxc1BzfMpxvA==
dependencies:
- fill-range "^7.0.1"
+ fill-range "^7.1.1"
broadcast-channel@^3.4.1:
version "3.7.0"
@@ -5400,10 +5400,10 @@ filesize@^8.0.6:
resolved "https://registry.yarnpkg.com/filesize/-/filesize-8.0.7.tgz#695e70d80f4e47012c132d57a059e80c6b580bd8"
integrity sha512-pjmC+bkIF8XI7fWaH8KxHcZL3DPybs1roSKP4rKDvy20tAWwIObE4+JIseG2byfGKhud5ZnM4YSGKBz7Sh0ndQ==
-fill-range@^7.0.1:
- version "7.0.1"
- resolved "https://registry.yarnpkg.com/fill-range/-/fill-range-7.0.1.tgz#1919a6a7c75fe38b2c7c77e5198535da9acdda40"
- integrity sha512-qOo9F+dMUmC2Lcb4BbVvnKJxTPjCm+RRpe4gDuGrzkL7mEVl/djYSu2OdQ2Pa302N4oqkSg9ir6jaLWJ2USVpQ==
+fill-range@^7.1.1:
+ version "7.1.1"
+ resolved "https://registry.yarnpkg.com/fill-range/-/fill-range-7.1.1.tgz#44265d3cac07e3ea7dc247516380643754a05292"
+ integrity sha512-YsGpe3WHLK8ZYi4tWDg2Jy3ebRz2rXowDxnld4bkQB00cc/1Zw9AWnC0i9ztDJitivtQvaI9KaLyKrc+hBW0yg==
dependencies:
to-regex-range "^5.0.1"
@@ -10977,14 +10977,14 @@ write-file-atomic@^3.0.0:
typedarray-to-buffer "^3.1.5"
ws@^7.4.6:
- version "7.5.7"
- resolved "https://registry.yarnpkg.com/ws/-/ws-7.5.7.tgz#9e0ac77ee50af70d58326ecff7e85eb3fa375e67"
- integrity sha512-KMvVuFzpKBuiIXW3E4u3mySRO2/mCHSyZDJQM5NQ9Q9KHWHWh0NHgfbRMLLrceUK5qAL4ytALJbpRMjixFZh8A==
+ version "7.5.10"
+ resolved "https://registry.yarnpkg.com/ws/-/ws-7.5.10.tgz#58b5c20dc281633f6c19113f39b349bd8bd558d9"
+ integrity sha512-+dbF1tHwZpXcbOJdVOkzLDxZP1ailvSxM6ZweXTegylPny803bFhA+vqBYw4s31NSAk4S2Qz+AKXK9a4wkdjcQ==
ws@^8.4.2:
- version "8.6.0"
- resolved "https://registry.yarnpkg.com/ws/-/ws-8.6.0.tgz#e5e9f1d9e7ff88083d0c0dd8281ea662a42c9c23"
- integrity sha512-AzmM3aH3gk0aX7/rZLYvjdvZooofDu3fFOzGqcSnQ1tOcTWwhM/o+q++E8mAyVVIyUdajrkzWUGftaVSDLn1bw==
+ version "8.18.0"
+ resolved "https://registry.yarnpkg.com/ws/-/ws-8.18.0.tgz#0d7505a6eafe2b0e712d232b42279f53bc289bbc"
+ integrity sha512-8VbfWfHLbbwu3+N6OKsOMpBdT4kXPDDB9cJk2bJ6mh9ucxdlnNvH1e+roYkKmN9Nxw2yjz7VzeO9oOz2zJ04Pw==
xml-name-validator@^3.0.0:
version "3.0.0"
diff --git a/sdk/python/feast/utils.py b/sdk/python/feast/utils.py
index a6c893c954c..0467393aa2e 100644
--- a/sdk/python/feast/utils.py
+++ b/sdk/python/feast/utils.py
@@ -5,7 +5,7 @@
import typing
import warnings
from collections import Counter, defaultdict
-from datetime import datetime
+from datetime import datetime, timezone
from pathlib import Path
from typing import (
Any,
@@ -1052,3 +1052,7 @@ def tags_str_to_dict(tags: str = "") -> dict[str, str]:
cast(tuple[str, str], tag.split(":", 1)) for tag in tags_list if ":" in tag
).items()
}
+
+
+def _utc_now() -> datetime:
+ return datetime.now(tz=timezone.utc)
diff --git a/sdk/python/requirements/py3.10-ci-requirements.txt b/sdk/python/requirements/py3.10-ci-requirements.txt
index 97bdfc159ba..a9f1d625a84 100644
--- a/sdk/python/requirements/py3.10-ci-requirements.txt
+++ b/sdk/python/requirements/py3.10-ci-requirements.txt
@@ -1,6 +1,6 @@
# This file was autogenerated by uv via the following command:
# uv pip compile --system --no-strip-extras setup.py --extra ci --output-file sdk/python/requirements/py3.10-ci-requirements.txt
-aiobotocore==2.13.0
+aiobotocore==2.13.1
aiohttp==3.9.5
# via aiobotocore
aioitertools==0.11.0
@@ -11,9 +11,9 @@ alabaster==0.7.16
# via sphinx
altair==4.2.2
# via great-expectations
-annotated-types==0.6.0
+annotated-types==0.7.0
# via pydantic
-anyio==4.3.0
+anyio==4.4.0
# via
# httpx
# jupyter-server
@@ -43,11 +43,11 @@ attrs==23.2.0
# aiohttp
# jsonschema
# referencing
-azure-core==1.30.1
+azure-core==1.30.2
# via
# azure-identity
# azure-storage-blob
-azure-identity==1.16.0
+azure-identity==1.17.1
azure-storage-blob==12.20.0
babel==2.15.0
# via
@@ -59,22 +59,22 @@ bidict==0.23.1
# via ibis-framework
bleach==6.1.0
# via nbconvert
-boto3==1.34.99
+boto3==1.34.131
# via moto
-botocore==1.34.99
+botocore==1.34.131
# via
# aiobotocore
# boto3
# moto
# s3transfer
build==1.2.1
- # via pip-tools
-cachecontrol==0.14.0
- # via firebase-admin
+ # via
+ # pip-tools
+ # singlestoredb
cachetools==5.3.3
# via google-auth
cassandra-driver==3.29.1
-certifi==2024.2.2
+certifi==2024.7.4
# via
# elastic-transport
# httpcore
@@ -110,9 +110,9 @@ comm==0.2.2
# via
# ipykernel
# ipywidgets
-coverage[toml]==7.5.3
+coverage[toml]==7.5.4
# via pytest-cov
-cryptography==42.0.7
+cryptography==42.0.8
# via
# azure-identity
# azure-storage-blob
@@ -124,19 +124,19 @@ cryptography==42.0.7
# snowflake-connector-python
# types-pyopenssl
# types-redis
-dask[dataframe]==2024.5.0
+dask[dataframe]==2024.6.2
# via dask-expr
-dask-expr==1.1.0
+dask-expr==1.1.6
# via dask
db-dtypes==1.2.0
# via google-cloud-bigquery
-debugpy==1.8.1
+debugpy==1.8.2
# via ipykernel
decorator==5.1.1
# via ipython
defusedxml==0.7.1
# via nbconvert
-deltalake==0.17.3
+deltalake==0.18.1
dill==0.3.8
distlib==0.3.8
# via virtualenv
@@ -147,15 +147,11 @@ docker==7.1.0
docutils==0.19
# via sphinx
duckdb==0.10.3
- # via
- # duckdb-engine
- # ibis-framework
-duckdb-engine==0.12.1
# via ibis-framework
elastic-transport==8.13.1
# via elasticsearch
-elasticsearch==8.13.2
-email-validator==2.1.1
+elasticsearch==8.14.0
+email-validator==2.2.0
# via fastapi
entrypoints==0.4
# via altair
@@ -169,16 +165,14 @@ execnet==2.1.1
executing==2.0.1
# via stack-data
fastapi==0.111.0
- # via fastapi-cli
-fastapi-cli==0.0.2
+fastapi-cli==0.0.4
# via fastapi
-fastjsonschema==2.19.1
+fastjsonschema==2.20.0
# via nbformat
-filelock==3.14.0
+filelock==3.15.4
# via
# snowflake-connector-python
# virtualenv
-firebase-admin==5.4.0
fqdn==1.5.1
# via jsonschema
frozenlist==1.4.1
@@ -191,65 +185,51 @@ geojson==2.5.0
# via rockset
geomet==0.2.1.post1
# via cassandra-driver
-google-api-core[grpc]==2.19.0
+google-api-core[grpc]==2.19.1
# via
- # firebase-admin
- # google-api-python-client
# google-cloud-bigquery
# google-cloud-bigquery-storage
# google-cloud-bigtable
# google-cloud-core
# google-cloud-datastore
- # google-cloud-firestore
# google-cloud-storage
-google-api-python-client==2.131.0
- # via firebase-admin
-google-auth==2.29.0
+google-auth==2.30.0
# via
# google-api-core
- # google-api-python-client
- # google-auth-httplib2
# google-cloud-bigquery-storage
# google-cloud-core
- # google-cloud-firestore
# google-cloud-storage
# kubernetes
-google-auth-httplib2==0.2.0
- # via google-api-python-client
-google-cloud-bigquery[pandas]==3.12.0
+google-cloud-bigquery[pandas]==3.13.0
google-cloud-bigquery-storage==2.25.0
-google-cloud-bigtable==2.23.1
+google-cloud-bigtable==2.24.0
google-cloud-core==2.4.1
# via
# google-cloud-bigquery
# google-cloud-bigtable
# google-cloud-datastore
- # google-cloud-firestore
# google-cloud-storage
google-cloud-datastore==2.19.0
-google-cloud-firestore==2.16.0
- # via firebase-admin
-google-cloud-storage==2.16.0
- # via firebase-admin
+google-cloud-storage==2.17.0
google-crc32c==1.5.0
# via
# google-cloud-storage
# google-resumable-media
-google-resumable-media==2.7.0
+google-resumable-media==2.7.1
# via
# google-cloud-bigquery
# google-cloud-storage
-googleapis-common-protos[grpc]==1.63.0
+googleapis-common-protos[grpc]==1.63.2
# via
# google-api-core
# grpc-google-iam-v1
# grpcio-status
-great-expectations==0.18.15
+great-expectations==0.18.16
greenlet==3.0.3
# via sqlalchemy
-grpc-google-iam-v1==0.13.0
+grpc-google-iam-v1==0.13.1
# via google-cloud-bigtable
-grpcio==1.64.0
+grpcio==1.64.1
# via
# google-api-core
# google-cloud-bigquery
@@ -276,19 +256,15 @@ hazelcast-python-client==5.4.0
hiredis==2.3.2
httpcore==1.0.5
# via httpx
-httplib2==0.22.0
- # via
- # google-api-python-client
- # google-auth-httplib2
httptools==0.6.1
# via uvicorn
httpx==0.27.0
# via
# fastapi
# jupyterlab
-ibis-framework[duckdb]==8.0.0
+ibis-framework[duckdb]==9.1.0
# via ibis-substrait
-ibis-substrait==3.2.0
+ibis-substrait==4.0.0
identify==2.5.36
# via pre-commit
idna==3.7
@@ -302,7 +278,7 @@ idna==3.7
# yarl
imagesize==1.4.1
# via sphinx
-importlib-metadata==7.1.0
+importlib-metadata==8.0.0
# via dask
iniconfig==2.0.0
# via pytest
@@ -340,7 +316,7 @@ json5==0.9.25
# via jupyterlab-server
jsonpatch==1.33
# via great-expectations
-jsonpointer==2.4
+jsonpointer==3.0.0
# via
# jsonpatch
# jsonschema
@@ -380,7 +356,7 @@ jupyter-server==2.14.1
# notebook-shim
jupyter-server-terminals==0.5.3
# via jupyter-server
-jupyterlab==4.2.1
+jupyterlab==4.2.3
# via notebook
jupyterlab-pygments==0.3.0
# via nbconvert
@@ -402,7 +378,7 @@ markupsafe==2.1.5
# jinja2
# nbconvert
# werkzeug
-marshmallow==3.21.2
+marshmallow==3.21.3
# via great-expectations
matplotlib-inline==0.1.7
# via
@@ -418,21 +394,17 @@ mistune==3.0.2
mmh3==4.1.0
mock==2.0.0
moto==4.2.14
-msal==1.28.0
+msal==1.29.0
# via
# azure-identity
# msal-extensions
-msal-extensions==1.1.0
+msal-extensions==1.2.0
# via azure-identity
-msgpack==1.0.8
- # via cachecontrol
multidict==6.0.5
# via
# aiohttp
# yarl
-multipledispatch==1.0.0
- # via ibis-framework
-mypy==1.10.0
+mypy==1.10.1
# via sqlalchemy
mypy-extensions==1.0.0
# via mypy
@@ -449,9 +421,9 @@ nbformat==5.10.4
# nbconvert
nest-asyncio==1.6.0
# via ipykernel
-nodeenv==1.9.0
+nodeenv==1.9.1
# via pre-commit
-notebook==7.2.0
+notebook==7.2.1
# via great-expectations
notebook-shim==0.2.4
# via
@@ -469,16 +441,15 @@ numpy==1.26.4
# scipy
oauthlib==3.2.2
# via requests-oauthlib
-orjson==3.10.3
+orjson==3.10.5
# via fastapi
overrides==7.7.0
# via jupyter-server
-packaging==24.0
+packaging==24.1
# via
# build
# dask
# db-dtypes
- # duckdb-engine
# google-cloud-bigquery
# great-expectations
# gunicorn
@@ -488,7 +459,6 @@ packaging==24.0
# jupyterlab
# jupyterlab-server
# marshmallow
- # msal-extensions
# nbconvert
# pytest
# snowflake-connector-python
@@ -505,6 +475,8 @@ pandas==2.2.2
# snowflake-connector-python
pandocfilters==1.5.1
# via nbconvert
+parsimonious==0.10.0
+ # via singlestoredb
parso==0.8.4
# via jedi
parsy==2.1
@@ -515,7 +487,7 @@ pbr==6.0.0
# via mock
pexpect==4.9.0
# via ipython
-pip==24.0
+pip==24.1.1
# via pip-tools
pip-tools==7.4.1
platformdirs==3.11.0
@@ -527,21 +499,20 @@ pluggy==1.5.0
# via pytest
ply==3.11
# via thriftpy2
-portalocker==2.8.2
+portalocker==2.10.0
# via msal-extensions
pre-commit==3.3.1
prometheus-client==0.20.0
# via jupyter-server
-prompt-toolkit==3.0.45
+prompt-toolkit==3.0.47
# via ipython
-proto-plus==1.23.0
+proto-plus==1.24.0
# via
# google-api-core
# google-cloud-bigquery
# google-cloud-bigquery-storage
# google-cloud-bigtable
# google-cloud-datastore
- # google-cloud-firestore
protobuf==4.25.3
# via
# google-api-core
@@ -549,7 +520,6 @@ protobuf==4.25.3
# google-cloud-bigquery-storage
# google-cloud-bigtable
# google-cloud-datastore
- # google-cloud-firestore
# googleapis-common-protos
# grpc-google-iam-v1
# grpcio-health-checking
@@ -562,7 +532,11 @@ protobuf==4.25.3
# substrait
psutil==5.9.0
# via ipykernel
-psycopg2-binary==2.9.9
+psycopg[binary, pool]==3.1.19
+psycopg-binary==3.1.19
+ # via psycopg
+psycopg-pool==3.2.2
+ # via psycopg
ptyprocess==0.7.0
# via
# pexpect
@@ -595,11 +569,11 @@ pyasn1-modules==0.4.0
pybindgen==0.22.1
pycparser==2.22
# via cffi
-pydantic==2.7.1
+pydantic==2.7.4
# via
# fastapi
# great-expectations
-pydantic-core==2.18.2
+pydantic-core==2.18.4
# via pydantic
pygments==2.18.0
# via
@@ -610,6 +584,7 @@ pygments==2.18.0
pyjwt[crypto]==2.8.0
# via
# msal
+ # singlestoredb
# snowflake-connector-python
pymssql==2.3.0
pymysql==1.1.1
@@ -617,9 +592,7 @@ pyodbc==5.1.0
pyopenssl==24.1.0
# via snowflake-connector-python
pyparsing==3.1.2
- # via
- # great-expectations
- # httplib2
+ # via great-expectations
pyproject-hooks==1.1.0
# via
# build
@@ -689,11 +662,11 @@ referencing==0.35.1
# jsonschema
# jsonschema-specifications
# jupyter-events
-regex==2024.4.28
-requests==2.31.0
+regex==2024.5.15
+ # via parsimonious
+requests==2.32.3
# via
# azure-core
- # cachecontrol
# docker
# google-api-core
# google-cloud-bigquery
@@ -705,12 +678,13 @@ requests==2.31.0
# msal
# requests-oauthlib
# responses
+ # singlestoredb
# snowflake-connector-python
# sphinx
# trino
requests-oauthlib==2.0.0
# via kubernetes
-responses==0.25.0
+responses==0.25.3
# via moto
rfc3339-validator==0.1.4
# via
@@ -733,20 +707,23 @@ rsa==4.9
# via google-auth
ruamel-yaml==0.17.17
# via great-expectations
-ruff==0.4.6
-s3transfer==0.10.1
+ruff==0.4.10
+s3transfer==0.10.2
# via boto3
-scipy==1.13.1
+scipy==1.14.0
# via great-expectations
send2trash==1.8.3
# via jupyter-server
-setuptools==70.0.0
+setuptools==70.1.1
# via
# grpcio-tools
+ # jupyterlab
# kubernetes
# pip-tools
+ # singlestoredb
shellingham==1.5.4
# via typer
+singlestoredb==1.4.0
six==1.16.0
# via
# asttokens
@@ -766,7 +743,7 @@ sniffio==1.3.1
# httpx
snowballstemmer==2.2.0
# via sphinx
-snowflake-connector-python[pandas]==3.10.1
+snowflake-connector-python[pandas]==3.11.0
sortedcontainers==2.4.0
# via snowflake-connector-python
soupsieve==2.5
@@ -784,16 +761,12 @@ sphinxcontrib-qthelp==1.0.7
# via sphinx
sphinxcontrib-serializinghtml==1.1.10
# via sphinx
-sqlalchemy[mypy]==2.0.30
- # via
- # duckdb-engine
- # ibis-framework
- # sqlalchemy-views
-sqlalchemy-views==0.3.2
- # via ibis-framework
-sqlglot==20.11.0
+sqlalchemy[mypy]==2.0.31
+sqlglot==25.1.0
# via ibis-framework
sqlite-vec==0.0.1a10
+sqlparams==6.0.1
+ # via singlestoredb
stack-data==0.6.3
# via ipython
starlette==0.37.2
@@ -801,13 +774,13 @@ starlette==0.37.2
substrait==0.19.0
# via ibis-substrait
tabulate==0.9.0
-tenacity==8.3.0
+tenacity==8.4.2
terminado==0.18.1
# via
# jupyter-server
# jupyter-server-terminals
testcontainers==4.4.0
-thriftpy2==0.5.0
+thriftpy2==0.5.1
# via happybase
tinycss2==1.3.0
# via nbconvert
@@ -821,6 +794,7 @@ tomli==2.0.1
# pip-tools
# pytest
# pytest-env
+ # singlestoredb
tomlkit==0.12.5
# via snowflake-connector-python
toolz==0.12.1
@@ -855,14 +829,14 @@ traitlets==5.14.3
# nbconvert
# nbformat
trino==0.328.0
-typeguard==4.2.1
+typeguard==4.3.0
typer==0.12.3
# via fastapi-cli
types-cffi==1.16.0.20240331
# via types-pyopenssl
types-protobuf==3.19.22
# via mypy-protobuf
-types-pymysql==1.1.0.20240425
+types-pymysql==1.1.0.20240524
types-pyopenssl==24.1.0.20240425
# via types-redis
types-python-dateutil==2.9.0.20240316
@@ -871,22 +845,25 @@ types-pytz==2024.1.0.20240417
types-pyyaml==6.0.12.20240311
types-redis==4.6.0.20240425
types-requests==2.30.0.0
-types-setuptools==70.0.0.20240524
+types-setuptools==70.1.0.20240627
# via types-cffi
types-tabulate==0.9.0.20240106
types-urllib3==1.26.25.14
# via types-requests
-typing-extensions==4.11.0
+typing-extensions==4.12.2
# via
# anyio
# async-lru
# azure-core
+ # azure-identity
# azure-storage-blob
# fastapi
# great-expectations
# ibis-framework
# ipython
# mypy
+ # psycopg
+ # psycopg-pool
# pydantic
# pydantic-core
# snowflake-connector-python
@@ -901,13 +878,11 @@ tzlocal==5.2
# via
# great-expectations
# trino
-ujson==5.9.0
+ujson==5.10.0
# via fastapi
uri-template==1.3.0
# via jsonschema
-uritemplate==4.1.1
- # via google-api-python-client
-urllib3==1.26.18
+urllib3==1.26.19
# via
# botocore
# docker
@@ -919,19 +894,17 @@ urllib3==1.26.18
# responses
# rockset
# testcontainers
-uvicorn[standard]==0.29.0
- # via
- # fastapi
- # fastapi-cli
+uvicorn[standard]==0.30.1
+ # via fastapi
uvloop==0.19.0
# via uvicorn
virtualenv==20.23.0
# via pre-commit
-watchfiles==0.21.0
+watchfiles==0.22.0
# via uvicorn
wcwidth==0.2.13
# via prompt-toolkit
-webcolors==1.13
+webcolors==24.6.0
# via jsonschema
webencodings==0.5.1
# via
@@ -946,7 +919,9 @@ websockets==12.0
werkzeug==3.0.3
# via moto
wheel==0.43.0
- # via pip-tools
+ # via
+ # pip-tools
+ # singlestoredb
widgetsnbextension==4.0.11
# via ipywidgets
wrapt==1.16.0
@@ -957,5 +932,5 @@ xmltodict==0.13.0
# via moto
yarl==1.9.4
# via aiohttp
-zipp==3.18.1
+zipp==3.19.1
# via importlib-metadata
diff --git a/sdk/python/requirements/py3.10-requirements.txt b/sdk/python/requirements/py3.10-requirements.txt
index 99c9bfc3fee..0cca1068634 100644
--- a/sdk/python/requirements/py3.10-requirements.txt
+++ b/sdk/python/requirements/py3.10-requirements.txt
@@ -11,7 +11,7 @@ attrs==23.2.0
# via
# jsonschema
# referencing
-certifi==2024.2.2
+certifi==2024.7.4
# via
# httpcore
# httpx
@@ -186,5 +186,5 @@ watchfiles==0.21.0
# via uvicorn
websockets==12.0
# via uvicorn
-zipp==3.18.1
+zipp==3.19.1
# via importlib-metadata
diff --git a/sdk/python/requirements/py3.11-ci-requirements.txt b/sdk/python/requirements/py3.11-ci-requirements.txt
index f6db0af6bc0..ce2e50b8a7e 100644
--- a/sdk/python/requirements/py3.11-ci-requirements.txt
+++ b/sdk/python/requirements/py3.11-ci-requirements.txt
@@ -1,6 +1,6 @@
# This file was autogenerated by uv via the following command:
# uv pip compile --system --no-strip-extras setup.py --extra ci --output-file sdk/python/requirements/py3.11-ci-requirements.txt
-aiobotocore==2.13.0
+aiobotocore==2.13.1
aiohttp==3.9.5
# via aiobotocore
aioitertools==0.11.0
@@ -11,9 +11,9 @@ alabaster==0.7.16
# via sphinx
altair==4.2.2
# via great-expectations
-annotated-types==0.6.0
+annotated-types==0.7.0
# via pydantic
-anyio==4.3.0
+anyio==4.4.0
# via
# httpx
# jupyter-server
@@ -39,11 +39,11 @@ attrs==23.2.0
# aiohttp
# jsonschema
# referencing
-azure-core==1.30.1
+azure-core==1.30.2
# via
# azure-identity
# azure-storage-blob
-azure-identity==1.16.0
+azure-identity==1.17.1
azure-storage-blob==12.20.0
babel==2.15.0
# via
@@ -55,22 +55,22 @@ bidict==0.23.1
# via ibis-framework
bleach==6.1.0
# via nbconvert
-boto3==1.34.99
+boto3==1.34.131
# via moto
-botocore==1.34.99
+botocore==1.34.131
# via
# aiobotocore
# boto3
# moto
# s3transfer
build==1.2.1
- # via pip-tools
-cachecontrol==0.14.0
- # via firebase-admin
+ # via
+ # pip-tools
+ # singlestoredb
cachetools==5.3.3
# via google-auth
cassandra-driver==3.29.1
-certifi==2024.2.2
+certifi==2024.7.4
# via
# elastic-transport
# httpcore
@@ -106,9 +106,9 @@ comm==0.2.2
# via
# ipykernel
# ipywidgets
-coverage[toml]==7.5.3
+coverage[toml]==7.5.4
# via pytest-cov
-cryptography==42.0.7
+cryptography==42.0.8
# via
# azure-identity
# azure-storage-blob
@@ -120,19 +120,19 @@ cryptography==42.0.7
# snowflake-connector-python
# types-pyopenssl
# types-redis
-dask[dataframe]==2024.5.0
+dask[dataframe]==2024.6.2
# via dask-expr
-dask-expr==1.1.0
+dask-expr==1.1.6
# via dask
db-dtypes==1.2.0
# via google-cloud-bigquery
-debugpy==1.8.1
+debugpy==1.8.2
# via ipykernel
decorator==5.1.1
# via ipython
defusedxml==0.7.1
# via nbconvert
-deltalake==0.17.4
+deltalake==0.18.1
dill==0.3.8
distlib==0.3.8
# via virtualenv
@@ -143,15 +143,11 @@ docker==7.1.0
docutils==0.19
# via sphinx
duckdb==0.10.3
- # via
- # duckdb-engine
- # ibis-framework
-duckdb-engine==0.12.1
# via ibis-framework
elastic-transport==8.13.1
# via elasticsearch
-elasticsearch==8.13.2
-email-validator==2.1.1
+elasticsearch==8.14.0
+email-validator==2.2.0
# via fastapi
entrypoints==0.4
# via altair
@@ -160,16 +156,14 @@ execnet==2.1.1
executing==2.0.1
# via stack-data
fastapi==0.111.0
- # via fastapi-cli
-fastapi-cli==0.0.2
+fastapi-cli==0.0.4
# via fastapi
-fastjsonschema==2.19.1
+fastjsonschema==2.20.0
# via nbformat
-filelock==3.14.0
+filelock==3.15.4
# via
# snowflake-connector-python
# virtualenv
-firebase-admin==5.4.0
fqdn==1.5.1
# via jsonschema
frozenlist==1.4.1
@@ -182,65 +176,51 @@ geojson==2.5.0
# via rockset
geomet==0.2.1.post1
# via cassandra-driver
-google-api-core[grpc]==2.19.0
+google-api-core[grpc]==2.19.1
# via
- # firebase-admin
- # google-api-python-client
# google-cloud-bigquery
# google-cloud-bigquery-storage
# google-cloud-bigtable
# google-cloud-core
# google-cloud-datastore
- # google-cloud-firestore
# google-cloud-storage
-google-api-python-client==2.131.0
- # via firebase-admin
-google-auth==2.29.0
+google-auth==2.30.0
# via
# google-api-core
- # google-api-python-client
- # google-auth-httplib2
# google-cloud-bigquery-storage
# google-cloud-core
- # google-cloud-firestore
# google-cloud-storage
# kubernetes
-google-auth-httplib2==0.2.0
- # via google-api-python-client
-google-cloud-bigquery[pandas]==3.12.0
+google-cloud-bigquery[pandas]==3.13.0
google-cloud-bigquery-storage==2.25.0
-google-cloud-bigtable==2.23.1
+google-cloud-bigtable==2.24.0
google-cloud-core==2.4.1
# via
# google-cloud-bigquery
# google-cloud-bigtable
# google-cloud-datastore
- # google-cloud-firestore
# google-cloud-storage
google-cloud-datastore==2.19.0
-google-cloud-firestore==2.16.0
- # via firebase-admin
-google-cloud-storage==2.16.0
- # via firebase-admin
+google-cloud-storage==2.17.0
google-crc32c==1.5.0
# via
# google-cloud-storage
# google-resumable-media
-google-resumable-media==2.7.0
+google-resumable-media==2.7.1
# via
# google-cloud-bigquery
# google-cloud-storage
-googleapis-common-protos[grpc]==1.63.0
+googleapis-common-protos[grpc]==1.63.2
# via
# google-api-core
# grpc-google-iam-v1
# grpcio-status
-great-expectations==0.18.15
+great-expectations==0.18.16
greenlet==3.0.3
# via sqlalchemy
-grpc-google-iam-v1==0.13.0
+grpc-google-iam-v1==0.13.1
# via google-cloud-bigtable
-grpcio==1.64.0
+grpcio==1.64.1
# via
# google-api-core
# google-cloud-bigquery
@@ -267,19 +247,15 @@ hazelcast-python-client==5.4.0
hiredis==2.3.2
httpcore==1.0.5
# via httpx
-httplib2==0.22.0
- # via
- # google-api-python-client
- # google-auth-httplib2
httptools==0.6.1
# via uvicorn
httpx==0.27.0
# via
# fastapi
# jupyterlab
-ibis-framework[duckdb]==8.0.0
+ibis-framework[duckdb]==9.1.0
# via ibis-substrait
-ibis-substrait==3.2.0
+ibis-substrait==4.0.0
identify==2.5.36
# via pre-commit
idna==3.7
@@ -293,7 +269,7 @@ idna==3.7
# yarl
imagesize==1.4.1
# via sphinx
-importlib-metadata==7.1.0
+importlib-metadata==8.0.0
# via dask
iniconfig==2.0.0
# via pytest
@@ -331,7 +307,7 @@ json5==0.9.25
# via jupyterlab-server
jsonpatch==1.33
# via great-expectations
-jsonpointer==2.4
+jsonpointer==3.0.0
# via
# jsonpatch
# jsonschema
@@ -371,7 +347,7 @@ jupyter-server==2.14.1
# notebook-shim
jupyter-server-terminals==0.5.3
# via jupyter-server
-jupyterlab==4.2.1
+jupyterlab==4.2.3
# via notebook
jupyterlab-pygments==0.3.0
# via nbconvert
@@ -393,7 +369,7 @@ markupsafe==2.1.5
# jinja2
# nbconvert
# werkzeug
-marshmallow==3.21.2
+marshmallow==3.21.3
# via great-expectations
matplotlib-inline==0.1.7
# via
@@ -409,21 +385,17 @@ mistune==3.0.2
mmh3==4.1.0
mock==2.0.0
moto==4.2.14
-msal==1.28.0
+msal==1.29.0
# via
# azure-identity
# msal-extensions
-msal-extensions==1.1.0
+msal-extensions==1.2.0
# via azure-identity
-msgpack==1.0.8
- # via cachecontrol
multidict==6.0.5
# via
# aiohttp
# yarl
-multipledispatch==1.0.0
- # via ibis-framework
-mypy==1.10.0
+mypy==1.10.1
# via sqlalchemy
mypy-extensions==1.0.0
# via mypy
@@ -440,9 +412,9 @@ nbformat==5.10.4
# nbconvert
nest-asyncio==1.6.0
# via ipykernel
-nodeenv==1.9.0
+nodeenv==1.9.1
# via pre-commit
-notebook==7.2.0
+notebook==7.2.1
# via great-expectations
notebook-shim==0.2.4
# via
@@ -460,16 +432,15 @@ numpy==1.26.4
# scipy
oauthlib==3.2.2
# via requests-oauthlib
-orjson==3.10.3
+orjson==3.10.5
# via fastapi
overrides==7.7.0
# via jupyter-server
-packaging==24.0
+packaging==24.1
# via
# build
# dask
# db-dtypes
- # duckdb-engine
# google-cloud-bigquery
# great-expectations
# gunicorn
@@ -479,7 +450,6 @@ packaging==24.0
# jupyterlab
# jupyterlab-server
# marshmallow
- # msal-extensions
# nbconvert
# pytest
# snowflake-connector-python
@@ -496,6 +466,8 @@ pandas==2.2.2
# snowflake-connector-python
pandocfilters==1.5.1
# via nbconvert
+parsimonious==0.10.0
+ # via singlestoredb
parso==0.8.4
# via jedi
parsy==2.1
@@ -506,7 +478,7 @@ pbr==6.0.0
# via mock
pexpect==4.9.0
# via ipython
-pip==24.0
+pip==24.1.1
# via pip-tools
pip-tools==7.4.1
platformdirs==3.11.0
@@ -518,21 +490,20 @@ pluggy==1.5.0
# via pytest
ply==3.11
# via thriftpy2
-portalocker==2.8.2
+portalocker==2.10.0
# via msal-extensions
pre-commit==3.3.1
prometheus-client==0.20.0
# via jupyter-server
-prompt-toolkit==3.0.45
+prompt-toolkit==3.0.47
# via ipython
-proto-plus==1.23.0
+proto-plus==1.24.0
# via
# google-api-core
# google-cloud-bigquery
# google-cloud-bigquery-storage
# google-cloud-bigtable
# google-cloud-datastore
- # google-cloud-firestore
protobuf==4.25.3
# via
# google-api-core
@@ -540,7 +511,6 @@ protobuf==4.25.3
# google-cloud-bigquery-storage
# google-cloud-bigtable
# google-cloud-datastore
- # google-cloud-firestore
# googleapis-common-protos
# grpc-google-iam-v1
# grpcio-health-checking
@@ -553,7 +523,11 @@ protobuf==4.25.3
# substrait
psutil==5.9.0
# via ipykernel
-psycopg2-binary==2.9.9
+psycopg[binary, pool]==3.1.19
+psycopg-binary==3.1.19
+ # via psycopg
+psycopg-pool==3.2.2
+ # via psycopg
ptyprocess==0.7.0
# via
# pexpect
@@ -586,11 +560,11 @@ pyasn1-modules==0.4.0
pybindgen==0.22.1
pycparser==2.22
# via cffi
-pydantic==2.7.1
+pydantic==2.7.4
# via
# fastapi
# great-expectations
-pydantic-core==2.18.2
+pydantic-core==2.18.4
# via pydantic
pygments==2.18.0
# via
@@ -601,6 +575,7 @@ pygments==2.18.0
pyjwt[crypto]==2.8.0
# via
# msal
+ # singlestoredb
# snowflake-connector-python
pymssql==2.3.0
pymysql==1.1.1
@@ -608,9 +583,7 @@ pyodbc==5.1.0
pyopenssl==24.1.0
# via snowflake-connector-python
pyparsing==3.1.2
- # via
- # great-expectations
- # httplib2
+ # via great-expectations
pyproject-hooks==1.1.0
# via
# build
@@ -681,10 +654,10 @@ referencing==0.35.1
# jsonschema-specifications
# jupyter-events
regex==2024.5.15
-requests==2.31.0
+ # via parsimonious
+requests==2.32.3
# via
# azure-core
- # cachecontrol
# docker
# google-api-core
# google-cloud-bigquery
@@ -696,12 +669,13 @@ requests==2.31.0
# msal
# requests-oauthlib
# responses
+ # singlestoredb
# snowflake-connector-python
# sphinx
# trino
requests-oauthlib==2.0.0
# via kubernetes
-responses==0.25.0
+responses==0.25.3
# via moto
rfc3339-validator==0.1.4
# via
@@ -724,20 +698,23 @@ rsa==4.9
# via google-auth
ruamel-yaml==0.17.17
# via great-expectations
-ruff==0.4.6
-s3transfer==0.10.1
+ruff==0.4.10
+s3transfer==0.10.2
# via boto3
-scipy==1.13.1
+scipy==1.14.0
# via great-expectations
send2trash==1.8.3
# via jupyter-server
-setuptools==70.0.0
+setuptools==70.1.1
# via
# grpcio-tools
+ # jupyterlab
# kubernetes
# pip-tools
+ # singlestoredb
shellingham==1.5.4
# via typer
+singlestoredb==1.4.0
six==1.16.0
# via
# asttokens
@@ -757,7 +734,7 @@ sniffio==1.3.1
# httpx
snowballstemmer==2.2.0
# via sphinx
-snowflake-connector-python[pandas]==3.10.1
+snowflake-connector-python[pandas]==3.11.0
sortedcontainers==2.4.0
# via snowflake-connector-python
soupsieve==2.5
@@ -775,16 +752,12 @@ sphinxcontrib-qthelp==1.0.7
# via sphinx
sphinxcontrib-serializinghtml==1.1.10
# via sphinx
-sqlalchemy[mypy]==2.0.30
- # via
- # duckdb-engine
- # ibis-framework
- # sqlalchemy-views
-sqlalchemy-views==0.3.2
- # via ibis-framework
-sqlglot==20.11.0
+sqlalchemy[mypy]==2.0.31
+sqlglot==25.1.0
# via ibis-framework
sqlite-vec==0.0.1a10
+sqlparams==6.0.1
+ # via singlestoredb
stack-data==0.6.3
# via ipython
starlette==0.37.2
@@ -792,13 +765,13 @@ starlette==0.37.2
substrait==0.19.0
# via ibis-substrait
tabulate==0.9.0
-tenacity==8.3.0
+tenacity==8.4.2
terminado==0.18.1
# via
# jupyter-server
# jupyter-server-terminals
testcontainers==4.4.0
-thriftpy2==0.5.0
+thriftpy2==0.5.1
# via happybase
tinycss2==1.3.0
# via nbconvert
@@ -837,14 +810,14 @@ traitlets==5.14.3
# nbconvert
# nbformat
trino==0.328.0
-typeguard==4.2.1
+typeguard==4.3.0
typer==0.12.3
# via fastapi-cli
types-cffi==1.16.0.20240331
# via types-pyopenssl
types-protobuf==3.19.22
# via mypy-protobuf
-types-pymysql==1.1.0.20240425
+types-pymysql==1.1.0.20240524
types-pyopenssl==24.1.0.20240425
# via types-redis
types-python-dateutil==2.9.0.20240316
@@ -853,20 +826,23 @@ types-pytz==2024.1.0.20240417
types-pyyaml==6.0.12.20240311
types-redis==4.6.0.20240425
types-requests==2.30.0.0
-types-setuptools==70.0.0.20240524
+types-setuptools==70.1.0.20240627
# via types-cffi
types-tabulate==0.9.0.20240106
types-urllib3==1.26.25.14
# via types-requests
-typing-extensions==4.11.0
+typing-extensions==4.12.2
# via
# azure-core
+ # azure-identity
# azure-storage-blob
# fastapi
# great-expectations
# ibis-framework
# ipython
# mypy
+ # psycopg
+ # psycopg-pool
# pydantic
# pydantic-core
# snowflake-connector-python
@@ -880,13 +856,11 @@ tzlocal==5.2
# via
# great-expectations
# trino
-ujson==5.9.0
+ujson==5.10.0
# via fastapi
uri-template==1.3.0
# via jsonschema
-uritemplate==4.1.1
- # via google-api-python-client
-urllib3==1.26.18
+urllib3==1.26.19
# via
# botocore
# docker
@@ -898,19 +872,17 @@ urllib3==1.26.18
# responses
# rockset
# testcontainers
-uvicorn[standard]==0.29.0
- # via
- # fastapi
- # fastapi-cli
+uvicorn[standard]==0.30.1
+ # via fastapi
uvloop==0.19.0
# via uvicorn
virtualenv==20.23.0
# via pre-commit
-watchfiles==0.21.0
+watchfiles==0.22.0
# via uvicorn
wcwidth==0.2.13
# via prompt-toolkit
-webcolors==1.13
+webcolors==24.6.0
# via jsonschema
webencodings==0.5.1
# via
@@ -925,7 +897,9 @@ websockets==12.0
werkzeug==3.0.3
# via moto
wheel==0.43.0
- # via pip-tools
+ # via
+ # pip-tools
+ # singlestoredb
widgetsnbextension==4.0.11
# via ipywidgets
wrapt==1.16.0
@@ -936,5 +910,5 @@ xmltodict==0.13.0
# via moto
yarl==1.9.4
# via aiohttp
-zipp==3.18.1
+zipp==3.19.1
# via importlib-metadata
diff --git a/sdk/python/requirements/py3.11-requirements.txt b/sdk/python/requirements/py3.11-requirements.txt
index c34b610d14c..687e4bfe52e 100644
--- a/sdk/python/requirements/py3.11-requirements.txt
+++ b/sdk/python/requirements/py3.11-requirements.txt
@@ -11,7 +11,7 @@ attrs==23.2.0
# via
# jsonschema
# referencing
-certifi==2024.2.2
+certifi==2024.7.4
# via
# httpcore
# httpx
@@ -180,5 +180,5 @@ watchfiles==0.21.0
# via uvicorn
websockets==12.0
# via uvicorn
-zipp==3.18.1
+zipp==3.19.1
# via importlib-metadata
diff --git a/sdk/python/requirements/py3.9-ci-requirements.txt b/sdk/python/requirements/py3.9-ci-requirements.txt
index 135b65a0ccc..017c1c8920a 100644
--- a/sdk/python/requirements/py3.9-ci-requirements.txt
+++ b/sdk/python/requirements/py3.9-ci-requirements.txt
@@ -1,6 +1,6 @@
# This file was autogenerated by uv via the following command:
# uv pip compile --system --no-strip-extras setup.py --extra ci --output-file sdk/python/requirements/py3.9-ci-requirements.txt
-aiobotocore==2.13.0
+aiobotocore==2.13.1
aiohttp==3.9.5
# via aiobotocore
aioitertools==0.11.0
@@ -11,9 +11,9 @@ alabaster==0.7.16
# via sphinx
altair==4.2.2
# via great-expectations
-annotated-types==0.6.0
+annotated-types==0.7.0
# via pydantic
-anyio==4.3.0
+anyio==4.4.0
# via
# httpx
# jupyter-server
@@ -43,11 +43,11 @@ attrs==23.2.0
# aiohttp
# jsonschema
# referencing
-azure-core==1.30.1
+azure-core==1.30.2
# via
# azure-identity
# azure-storage-blob
-azure-identity==1.16.0
+azure-identity==1.17.1
azure-storage-blob==12.20.0
babel==2.15.0
# via
@@ -59,22 +59,22 @@ bidict==0.23.1
# via ibis-framework
bleach==6.1.0
# via nbconvert
-boto3==1.34.99
+boto3==1.34.131
# via moto
-botocore==1.34.99
+botocore==1.34.131
# via
# aiobotocore
# boto3
# moto
# s3transfer
build==1.2.1
- # via pip-tools
-cachecontrol==0.14.0
- # via firebase-admin
+ # via
+ # pip-tools
+ # singlestoredb
cachetools==5.3.3
# via google-auth
cassandra-driver==3.29.1
-certifi==2024.2.2
+certifi==2024.7.4
# via
# elastic-transport
# httpcore
@@ -110,9 +110,9 @@ comm==0.2.2
# via
# ipykernel
# ipywidgets
-coverage[toml]==7.5.3
+coverage[toml]==7.5.4
# via pytest-cov
-cryptography==42.0.7
+cryptography==42.0.8
# via
# azure-identity
# azure-storage-blob
@@ -124,19 +124,19 @@ cryptography==42.0.7
# snowflake-connector-python
# types-pyopenssl
# types-redis
-dask[dataframe]==2024.5.0
+dask[dataframe]==2024.6.2
# via dask-expr
-dask-expr==1.1.0
+dask-expr==1.1.6
# via dask
db-dtypes==1.2.0
# via google-cloud-bigquery
-debugpy==1.8.1
+debugpy==1.8.2
# via ipykernel
decorator==5.1.1
# via ipython
defusedxml==0.7.1
# via nbconvert
-deltalake==0.17.4
+deltalake==0.18.1
dill==0.3.8
distlib==0.3.8
# via virtualenv
@@ -147,15 +147,11 @@ docker==7.1.0
docutils==0.19
# via sphinx
duckdb==0.10.3
- # via
- # duckdb-engine
- # ibis-framework
-duckdb-engine==0.12.1
# via ibis-framework
elastic-transport==8.13.1
# via elasticsearch
-elasticsearch==8.13.2
-email-validator==2.1.1
+elasticsearch==8.14.0
+email-validator==2.2.0
# via fastapi
entrypoints==0.4
# via altair
@@ -169,16 +165,14 @@ execnet==2.1.1
executing==2.0.1
# via stack-data
fastapi==0.111.0
- # via fastapi-cli
-fastapi-cli==0.0.2
+fastapi-cli==0.0.4
# via fastapi
-fastjsonschema==2.19.1
+fastjsonschema==2.20.0
# via nbformat
-filelock==3.14.0
+filelock==3.15.4
# via
# snowflake-connector-python
# virtualenv
-firebase-admin==5.4.0
fqdn==1.5.1
# via jsonschema
frozenlist==1.4.1
@@ -191,65 +185,51 @@ geojson==2.5.0
# via rockset
geomet==0.2.1.post1
# via cassandra-driver
-google-api-core[grpc]==2.19.0
+google-api-core[grpc]==2.19.1
# via
- # firebase-admin
- # google-api-python-client
# google-cloud-bigquery
# google-cloud-bigquery-storage
# google-cloud-bigtable
# google-cloud-core
# google-cloud-datastore
- # google-cloud-firestore
# google-cloud-storage
-google-api-python-client==2.131.0
- # via firebase-admin
-google-auth==2.29.0
+google-auth==2.30.0
# via
# google-api-core
- # google-api-python-client
- # google-auth-httplib2
# google-cloud-bigquery-storage
# google-cloud-core
- # google-cloud-firestore
# google-cloud-storage
# kubernetes
-google-auth-httplib2==0.2.0
- # via google-api-python-client
-google-cloud-bigquery[pandas]==3.12.0
+google-cloud-bigquery[pandas]==3.13.0
google-cloud-bigquery-storage==2.25.0
-google-cloud-bigtable==2.23.1
+google-cloud-bigtable==2.24.0
google-cloud-core==2.4.1
# via
# google-cloud-bigquery
# google-cloud-bigtable
# google-cloud-datastore
- # google-cloud-firestore
# google-cloud-storage
google-cloud-datastore==2.19.0
-google-cloud-firestore==2.16.0
- # via firebase-admin
-google-cloud-storage==2.16.0
- # via firebase-admin
+google-cloud-storage==2.17.0
google-crc32c==1.5.0
# via
# google-cloud-storage
# google-resumable-media
-google-resumable-media==2.7.0
+google-resumable-media==2.7.1
# via
# google-cloud-bigquery
# google-cloud-storage
-googleapis-common-protos[grpc]==1.63.0
+googleapis-common-protos[grpc]==1.63.2
# via
# google-api-core
# grpc-google-iam-v1
# grpcio-status
-great-expectations==0.18.15
+great-expectations==0.18.16
greenlet==3.0.3
# via sqlalchemy
-grpc-google-iam-v1==0.13.0
+grpc-google-iam-v1==0.13.1
# via google-cloud-bigtable
-grpcio==1.64.0
+grpcio==1.64.1
# via
# google-api-core
# google-cloud-bigquery
@@ -276,19 +256,15 @@ hazelcast-python-client==5.4.0
hiredis==2.3.2
httpcore==1.0.5
# via httpx
-httplib2==0.22.0
- # via
- # google-api-python-client
- # google-auth-httplib2
httptools==0.6.1
# via uvicorn
httpx==0.27.0
# via
# fastapi
# jupyterlab
-ibis-framework[duckdb]==8.0.0
+ibis-framework[duckdb]==9.0.0
# via ibis-substrait
-ibis-substrait==3.2.0
+ibis-substrait==4.0.0
identify==2.5.36
# via pre-commit
idna==3.7
@@ -302,7 +278,7 @@ idna==3.7
# yarl
imagesize==1.4.1
# via sphinx
-importlib-metadata==7.1.0
+importlib-metadata==8.0.0
# via
# build
# dask
@@ -349,7 +325,7 @@ json5==0.9.25
# via jupyterlab-server
jsonpatch==1.33
# via great-expectations
-jsonpointer==2.4
+jsonpointer==3.0.0
# via
# jsonpatch
# jsonschema
@@ -389,7 +365,7 @@ jupyter-server==2.14.1
# notebook-shim
jupyter-server-terminals==0.5.3
# via jupyter-server
-jupyterlab==4.2.1
+jupyterlab==4.2.3
# via notebook
jupyterlab-pygments==0.3.0
# via nbconvert
@@ -411,7 +387,7 @@ markupsafe==2.1.5
# jinja2
# nbconvert
# werkzeug
-marshmallow==3.21.2
+marshmallow==3.21.3
# via great-expectations
matplotlib-inline==0.1.7
# via
@@ -427,21 +403,17 @@ mistune==3.0.2
mmh3==4.1.0
mock==2.0.0
moto==4.2.14
-msal==1.28.0
+msal==1.29.0
# via
# azure-identity
# msal-extensions
-msal-extensions==1.1.0
+msal-extensions==1.2.0
# via azure-identity
-msgpack==1.0.8
- # via cachecontrol
multidict==6.0.5
# via
# aiohttp
# yarl
-multipledispatch==1.0.0
- # via ibis-framework
-mypy==1.10.0
+mypy==1.10.1
# via sqlalchemy
mypy-extensions==1.0.0
# via mypy
@@ -458,9 +430,9 @@ nbformat==5.10.4
# nbconvert
nest-asyncio==1.6.0
# via ipykernel
-nodeenv==1.9.0
+nodeenv==1.9.1
# via pre-commit
-notebook==7.2.0
+notebook==7.2.1
# via great-expectations
notebook-shim==0.2.4
# via
@@ -478,16 +450,15 @@ numpy==1.26.4
# scipy
oauthlib==3.2.2
# via requests-oauthlib
-orjson==3.10.3
+orjson==3.10.5
# via fastapi
overrides==7.7.0
# via jupyter-server
-packaging==24.0
+packaging==24.1
# via
# build
# dask
# db-dtypes
- # duckdb-engine
# google-cloud-bigquery
# great-expectations
# gunicorn
@@ -497,7 +468,6 @@ packaging==24.0
# jupyterlab
# jupyterlab-server
# marshmallow
- # msal-extensions
# nbconvert
# pytest
# snowflake-connector-python
@@ -514,6 +484,8 @@ pandas==2.2.2
# snowflake-connector-python
pandocfilters==1.5.1
# via nbconvert
+parsimonious==0.10.0
+ # via singlestoredb
parso==0.8.4
# via jedi
parsy==2.1
@@ -524,7 +496,7 @@ pbr==6.0.0
# via mock
pexpect==4.9.0
# via ipython
-pip==24.0
+pip==24.1.1
# via pip-tools
pip-tools==7.4.1
platformdirs==3.11.0
@@ -536,21 +508,20 @@ pluggy==1.5.0
# via pytest
ply==3.11
# via thriftpy2
-portalocker==2.8.2
+portalocker==2.10.0
# via msal-extensions
pre-commit==3.3.1
prometheus-client==0.20.0
# via jupyter-server
-prompt-toolkit==3.0.45
+prompt-toolkit==3.0.47
# via ipython
-proto-plus==1.23.0
+proto-plus==1.24.0
# via
# google-api-core
# google-cloud-bigquery
# google-cloud-bigquery-storage
# google-cloud-bigtable
# google-cloud-datastore
- # google-cloud-firestore
protobuf==4.25.3
# via
# google-api-core
@@ -558,7 +529,6 @@ protobuf==4.25.3
# google-cloud-bigquery-storage
# google-cloud-bigtable
# google-cloud-datastore
- # google-cloud-firestore
# googleapis-common-protos
# grpc-google-iam-v1
# grpcio-health-checking
@@ -571,7 +541,11 @@ protobuf==4.25.3
# substrait
psutil==5.9.0
# via ipykernel
-psycopg2-binary==2.9.9
+psycopg[binary, pool]==3.1.18
+psycopg-binary==3.1.18
+ # via psycopg
+psycopg-pool==3.2.2
+ # via psycopg
ptyprocess==0.7.0
# via
# pexpect
@@ -604,11 +578,11 @@ pyasn1-modules==0.4.0
pybindgen==0.22.1
pycparser==2.22
# via cffi
-pydantic==2.7.1
+pydantic==2.7.4
# via
# fastapi
# great-expectations
-pydantic-core==2.18.2
+pydantic-core==2.18.4
# via pydantic
pygments==2.18.0
# via
@@ -619,6 +593,7 @@ pygments==2.18.0
pyjwt[crypto]==2.8.0
# via
# msal
+ # singlestoredb
# snowflake-connector-python
pymssql==2.3.0
pymysql==1.1.1
@@ -626,9 +601,7 @@ pyodbc==5.1.0
pyopenssl==24.1.0
# via snowflake-connector-python
pyparsing==3.1.2
- # via
- # great-expectations
- # httplib2
+ # via great-expectations
pyproject-hooks==1.1.0
# via
# build
@@ -699,10 +672,10 @@ referencing==0.35.1
# jsonschema-specifications
# jupyter-events
regex==2024.5.15
-requests==2.31.0
+ # via parsimonious
+requests==2.32.3
# via
# azure-core
- # cachecontrol
# docker
# google-api-core
# google-cloud-bigquery
@@ -714,12 +687,13 @@ requests==2.31.0
# msal
# requests-oauthlib
# responses
+ # singlestoredb
# snowflake-connector-python
# sphinx
# trino
requests-oauthlib==2.0.0
# via kubernetes
-responses==0.25.0
+responses==0.25.3
# via moto
rfc3339-validator==0.1.4
# via
@@ -744,20 +718,23 @@ ruamel-yaml==0.17.17
# via great-expectations
ruamel-yaml-clib==0.2.8
# via ruamel-yaml
-ruff==0.4.6
-s3transfer==0.10.1
+ruff==0.4.10
+s3transfer==0.10.2
# via boto3
scipy==1.13.1
# via great-expectations
send2trash==1.8.3
# via jupyter-server
-setuptools==70.0.0
+setuptools==70.1.1
# via
# grpcio-tools
+ # jupyterlab
# kubernetes
# pip-tools
+ # singlestoredb
shellingham==1.5.4
# via typer
+singlestoredb==1.4.0
six==1.16.0
# via
# asttokens
@@ -777,7 +754,7 @@ sniffio==1.3.1
# httpx
snowballstemmer==2.2.0
# via sphinx
-snowflake-connector-python[pandas]==3.10.1
+snowflake-connector-python[pandas]==3.11.0
sortedcontainers==2.4.0
# via snowflake-connector-python
soupsieve==2.5
@@ -795,16 +772,12 @@ sphinxcontrib-qthelp==1.0.7
# via sphinx
sphinxcontrib-serializinghtml==1.1.10
# via sphinx
-sqlalchemy[mypy]==2.0.30
- # via
- # duckdb-engine
- # ibis-framework
- # sqlalchemy-views
-sqlalchemy-views==0.3.2
- # via ibis-framework
-sqlglot==20.11.0
+sqlalchemy[mypy]==2.0.31
+sqlglot==23.12.2
# via ibis-framework
sqlite-vec==0.0.1a10
+sqlparams==6.0.1
+ # via singlestoredb
stack-data==0.6.3
# via ipython
starlette==0.37.2
@@ -812,13 +785,13 @@ starlette==0.37.2
substrait==0.19.0
# via ibis-substrait
tabulate==0.9.0
-tenacity==8.3.0
+tenacity==8.4.2
terminado==0.18.1
# via
# jupyter-server
# jupyter-server-terminals
testcontainers==4.4.0
-thriftpy2==0.5.0
+thriftpy2==0.5.1
# via happybase
tinycss2==1.3.0
# via nbconvert
@@ -832,6 +805,7 @@ tomli==2.0.1
# pip-tools
# pytest
# pytest-env
+ # singlestoredb
tomlkit==0.12.5
# via snowflake-connector-python
toolz==0.12.1
@@ -866,7 +840,7 @@ traitlets==5.14.3
# nbconvert
# nbformat
trino==0.328.0
-typeguard==4.2.1
+typeguard==4.3.0
typer==0.12.3
# via fastapi-cli
types-cffi==1.16.0.20240331
@@ -882,23 +856,26 @@ types-pytz==2024.1.0.20240417
types-pyyaml==6.0.12.20240311
types-redis==4.6.0.20240425
types-requests==2.30.0.0
-types-setuptools==70.0.0.20240524
+types-setuptools==70.1.0.20240627
# via types-cffi
types-tabulate==0.9.0.20240106
types-urllib3==1.26.25.14
# via types-requests
-typing-extensions==4.11.0
+typing-extensions==4.12.2
# via
# aioitertools
# anyio
# async-lru
# azure-core
+ # azure-identity
# azure-storage-blob
# fastapi
# great-expectations
# ibis-framework
# ipython
# mypy
+ # psycopg
+ # psycopg-pool
# pydantic
# pydantic-core
# snowflake-connector-python
@@ -914,13 +891,11 @@ tzlocal==5.2
# via
# great-expectations
# trino
-ujson==5.9.0
+ujson==5.10.0
# via fastapi
uri-template==1.3.0
# via jsonschema
-uritemplate==4.1.1
- # via google-api-python-client
-urllib3==1.26.18
+urllib3==1.26.19
# via
# botocore
# docker
@@ -933,19 +908,17 @@ urllib3==1.26.18
# rockset
# snowflake-connector-python
# testcontainers
-uvicorn[standard]==0.29.0
- # via
- # fastapi
- # fastapi-cli
+uvicorn[standard]==0.30.1
+ # via fastapi
uvloop==0.19.0
# via uvicorn
virtualenv==20.23.0
# via pre-commit
-watchfiles==0.21.0
+watchfiles==0.22.0
# via uvicorn
wcwidth==0.2.13
# via prompt-toolkit
-webcolors==1.13
+webcolors==24.6.0
# via jsonschema
webencodings==0.5.1
# via
@@ -960,7 +933,9 @@ websockets==12.0
werkzeug==3.0.3
# via moto
wheel==0.43.0
- # via pip-tools
+ # via
+ # pip-tools
+ # singlestoredb
widgetsnbextension==4.0.11
# via ipywidgets
wrapt==1.16.0
@@ -971,5 +946,5 @@ xmltodict==0.13.0
# via moto
yarl==1.9.4
# via aiohttp
-zipp==3.18.1
+zipp==3.19.1
# via importlib-metadata
diff --git a/sdk/python/requirements/py3.9-requirements.txt b/sdk/python/requirements/py3.9-requirements.txt
index 149a96626ef..096f54ab1fa 100644
--- a/sdk/python/requirements/py3.9-requirements.txt
+++ b/sdk/python/requirements/py3.9-requirements.txt
@@ -11,7 +11,7 @@ attrs==23.2.0
# via
# jsonschema
# referencing
-certifi==2024.2.2
+certifi==2024.7.4
# via
# httpcore
# httpx
@@ -189,5 +189,5 @@ watchfiles==0.21.0
# via uvicorn
websockets==12.0
# via uvicorn
-zipp==3.18.1
+zipp==3.19.1
# via importlib-metadata
diff --git a/sdk/python/tests/conftest.py b/sdk/python/tests/conftest.py
index 48f482f5428..1fd510d1048 100644
--- a/sdk/python/tests/conftest.py
+++ b/sdk/python/tests/conftest.py
@@ -15,7 +15,7 @@
import multiprocessing
import os
import random
-from datetime import datetime, timedelta
+from datetime import timedelta
from multiprocessing import Process
from sys import platform
from typing import Any, Dict, List, Tuple, no_type_check
@@ -27,6 +27,7 @@
from feast.data_source import DataSource
from feast.feature_store import FeatureStore # noqa: E402
+from feast.utils import _utc_now
from feast.wait import wait_retry_backoff # noqa: E402
from tests.data.data_creator import ( # noqa: E402
create_basic_driver_dataset,
@@ -133,7 +134,7 @@ def pytest_collection_modifyitems(config, items: List[Item]):
@pytest.fixture
def simple_dataset_1() -> pd.DataFrame:
- now = datetime.utcnow()
+ now = _utc_now()
ts = pd.Timestamp(now).round("ms")
data = {
"id_join_key": [1, 2, 1, 3, 3],
@@ -153,7 +154,7 @@ def simple_dataset_1() -> pd.DataFrame:
@pytest.fixture
def simple_dataset_2() -> pd.DataFrame:
- now = datetime.utcnow()
+ now = _utc_now()
ts = pd.Timestamp(now).round("ms")
data = {
"id_join_key": ["a", "b", "c", "d", "e"],
@@ -257,12 +258,7 @@ def pytest_generate_tests(metafunc: pytest.Metafunc):
extra_dimensions: List[Dict[str, Any]] = [{}]
if "python_server" in metafunc.fixturenames:
- extra_dimensions.extend(
- [
- {"python_feature_server": True},
- {"python_feature_server": True, "provider": "aws"},
- ]
- )
+ extra_dimensions.extend([{"python_feature_server": True}])
configs = []
if offline_stores:
@@ -277,17 +273,6 @@ def pytest_generate_tests(metafunc: pytest.Metafunc):
**dim,
}
- # aws lambda works only with dynamo
- if (
- config.get("python_feature_server")
- and config.get("provider") == "aws"
- and (
- not isinstance(online_store, dict)
- or online_store["type"] != "dynamodb"
- )
- ):
- continue
-
c = IntegrationTestRepoConfig(**config)
if c not in _config_cache:
@@ -407,8 +392,8 @@ def fake_ingest_data():
"conv_rate": [0.5],
"acc_rate": [0.6],
"avg_daily_trips": [4],
- "event_timestamp": [pd.Timestamp(datetime.utcnow()).round("ms")],
- "created": [pd.Timestamp(datetime.utcnow()).round("ms")],
+ "event_timestamp": [pd.Timestamp(_utc_now()).round("ms")],
+ "created": [pd.Timestamp(_utc_now()).round("ms")],
}
return pd.DataFrame(data)
diff --git a/sdk/python/tests/data/data_creator.py b/sdk/python/tests/data/data_creator.py
index 1be96f753a7..15d09c5a40a 100644
--- a/sdk/python/tests/data/data_creator.py
+++ b/sdk/python/tests/data/data_creator.py
@@ -5,6 +5,7 @@
from pytz import timezone, utc
from feast.types import FeastType, Float32, Int32, Int64, String
+from feast.utils import _utc_now
def create_basic_driver_dataset(
@@ -13,7 +14,7 @@ def create_basic_driver_dataset(
feature_is_list: bool = False,
list_has_empty_list: bool = False,
) -> pd.DataFrame:
- now = datetime.utcnow().replace(microsecond=0, second=0, minute=0)
+ now = _utc_now().replace(microsecond=0, second=0, minute=0)
ts = pd.Timestamp(now).round("ms")
data = {
"driver_id": get_entities_for_feast_type(entity_type),
@@ -86,14 +87,14 @@ def create_document_dataset() -> pd.DataFrame:
"embedding_float": [[4.0, 5.0], [1.0, 2.0], [3.0, 4.0]],
"embedding_double": [[4.0, 5.0], [1.0, 2.0], [3.0, 4.0]],
"ts": [
- pd.Timestamp(datetime.utcnow()).round("ms"),
- pd.Timestamp(datetime.utcnow()).round("ms"),
- pd.Timestamp(datetime.utcnow()).round("ms"),
+ pd.Timestamp(_utc_now()).round("ms"),
+ pd.Timestamp(_utc_now()).round("ms"),
+ pd.Timestamp(_utc_now()).round("ms"),
],
"created_ts": [
- pd.Timestamp(datetime.utcnow()).round("ms"),
- pd.Timestamp(datetime.utcnow()).round("ms"),
- pd.Timestamp(datetime.utcnow()).round("ms"),
+ pd.Timestamp(_utc_now()).round("ms"),
+ pd.Timestamp(_utc_now()).round("ms"),
+ pd.Timestamp(_utc_now()).round("ms"),
],
}
return pd.DataFrame(data)
diff --git a/sdk/python/tests/doctest/test_all.py b/sdk/python/tests/doctest/test_all.py
index 814a7ca7985..52348e7da4e 100644
--- a/sdk/python/tests/doctest/test_all.py
+++ b/sdk/python/tests/doctest/test_all.py
@@ -6,13 +6,14 @@
import unittest
import feast
+from feast.utils import _utc_now
FILES_TO_IGNORE = {"app"}
def setup_feature_store():
"""Prepares the local environment for a FeatureStore docstring test."""
- from datetime import datetime, timedelta
+ from datetime import timedelta
from feast import Entity, FeatureStore, FeatureView, Field, FileSource
from feast.repo_operations import init_repo
@@ -42,8 +43,8 @@ def setup_feature_store():
)
fs.apply([driver_hourly_stats_view, driver])
fs.materialize(
- start_date=datetime.utcnow() - timedelta(hours=3),
- end_date=datetime.utcnow() - timedelta(minutes=10),
+ start_date=_utc_now() - timedelta(hours=3),
+ end_date=_utc_now() - timedelta(minutes=10),
)
diff --git a/sdk/python/tests/foo_provider.py b/sdk/python/tests/foo_provider.py
index bd1e247a7b9..8e8f54db242 100644
--- a/sdk/python/tests/foo_provider.py
+++ b/sdk/python/tests/foo_provider.py
@@ -1,6 +1,6 @@
from datetime import datetime
from pathlib import Path
-from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple, Union
+from typing import Any, Callable, Dict, List, Mapping, Optional, Sequence, Tuple, Union
import pandas
import pyarrow
@@ -11,7 +11,9 @@
from feast.infra.offline_stores.offline_store import RetrievalJob
from feast.infra.provider import Provider
from feast.infra.registry.base_registry import BaseRegistry
+from feast.online_response import OnlineResponse
from feast.protos.feast.types.EntityKey_pb2 import EntityKey as EntityKeyProto
+from feast.protos.feast.types.Value_pb2 import RepeatedValue
from feast.protos.feast.types.Value_pb2 import Value as ValueProto
from feast.saved_dataset import SavedDataset
@@ -138,3 +140,31 @@ def validate_data_source(
data_source: DataSource,
):
pass
+
+ def get_online_features(
+ self,
+ config: RepoConfig,
+ features: Union[List[str], FeatureService],
+ entity_rows: Union[
+ List[Dict[str, Any]],
+ Mapping[str, Union[Sequence[Any], Sequence[ValueProto], RepeatedValue]],
+ ],
+ registry: BaseRegistry,
+ project: str,
+ full_feature_names: bool = False,
+ ) -> OnlineResponse:
+ pass
+
+ async def get_online_features_async(
+ self,
+ config: RepoConfig,
+ features: Union[List[str], FeatureService],
+ entity_rows: Union[
+ List[Dict[str, Any]],
+ Mapping[str, Union[Sequence[Any], Sequence[ValueProto], RepeatedValue]],
+ ],
+ registry: BaseRegistry,
+ project: str,
+ full_feature_names: bool = False,
+ ) -> OnlineResponse:
+ pass
diff --git a/sdk/python/tests/integration/feature_repos/repo_configuration.py b/sdk/python/tests/integration/feature_repos/repo_configuration.py
index 7123bd0fc15..48f5070f1e2 100644
--- a/sdk/python/tests/integration/feature_repos/repo_configuration.py
+++ b/sdk/python/tests/integration/feature_repos/repo_configuration.py
@@ -21,6 +21,7 @@
)
from feast.infra.feature_servers.local_process.config import LocalFeatureServerConfig
from feast.repo_config import RegistryConfig, RepoConfig
+from feast.utils import _utc_now
from tests.integration.feature_repos.integration_test_repo_config import (
IntegrationTestRepoConfig,
RegistryLocation,
@@ -412,7 +413,7 @@ class Environment:
fixture_request: Optional[pytest.FixtureRequest] = None
def __post_init__(self):
- self.end_date = datetime.utcnow().replace(microsecond=0, second=0, minute=0)
+ self.end_date = _utc_now().replace(microsecond=0, second=0, minute=0)
self.start_date: datetime = self.end_date - timedelta(days=3)
def setup(self):
@@ -474,27 +475,12 @@ def construct_test_environment(
else:
online_creator = None
- if test_repo_config.python_feature_server and test_repo_config.provider == "aws":
- from feast.infra.feature_servers.aws_lambda.config import (
- AwsLambdaFeatureServerConfig,
- )
-
- feature_server: Any = AwsLambdaFeatureServerConfig(
- enabled=True,
- execution_role_name=os.getenv(
- "AWS_LAMBDA_ROLE",
- "arn:aws:iam::402087665549:role/lambda_execution_role",
- ),
- )
- else:
- feature_server = LocalFeatureServerConfig(
- feature_logging=FeatureLoggingConfig(enabled=True)
- )
+ feature_server = LocalFeatureServerConfig(
+ feature_logging=FeatureLoggingConfig(enabled=True)
+ )
repo_dir_name = tempfile.mkdtemp()
- if (
- test_repo_config.python_feature_server and test_repo_config.provider == "aws"
- ) or test_repo_config.registry_location == RegistryLocation.S3:
+ if test_repo_config.registry_location == RegistryLocation.S3:
aws_registry_path = os.getenv(
"AWS_REGISTRY_PATH", "s3://feast-int-bucket/registries"
)
diff --git a/sdk/python/tests/integration/feature_repos/universal/data_sources/file.py b/sdk/python/tests/integration/feature_repos/universal/data_sources/file.py
index f7ab55d868a..5174e160465 100644
--- a/sdk/python/tests/integration/feature_repos/universal/data_sources/file.py
+++ b/sdk/python/tests/integration/feature_repos/universal/data_sources/file.py
@@ -20,8 +20,8 @@
from feast.data_format import DeltaFormat, ParquetFormat
from feast.data_source import DataSource
from feast.feature_logging import LoggingDestination
+from feast.infra.offline_stores.dask import DaskOfflineStoreConfig
from feast.infra.offline_stores.duckdb import DuckDBOfflineStoreConfig
-from feast.infra.offline_stores.file import FileOfflineStoreConfig
from feast.infra.offline_stores.file_source import (
FileLoggingDestination,
SavedDatasetFileStorage,
@@ -84,7 +84,7 @@ def get_prefixed_table_name(self, suffix: str) -> str:
return f"{self.project_name}.{suffix}"
def create_offline_store_config(self) -> FeastConfigBaseModel:
- return FileOfflineStoreConfig()
+ return DaskOfflineStoreConfig()
def create_logged_features_destination(self) -> LoggingDestination:
d = tempfile.mkdtemp(prefix=self.project_name)
@@ -334,7 +334,7 @@ def get_prefixed_table_name(self, suffix: str) -> str:
return f"{suffix}"
def create_offline_store_config(self) -> FeastConfigBaseModel:
- return FileOfflineStoreConfig()
+ return DaskOfflineStoreConfig()
def teardown(self):
self.minio.stop()
@@ -381,7 +381,7 @@ def setup(self, registry: RegistryConfig):
repo_path = Path(tempfile.mkdtemp())
with open(repo_path / "feature_store.yaml", "w") as outfile:
- yaml.dump(config.dict(by_alias=True), outfile)
+ yaml.dump(config.model_dump(by_alias=True), outfile)
repo_path = str(repo_path.resolve())
self.server_port = free_port()
diff --git a/sdk/python/tests/integration/feature_repos/universal/online_store/singlestore.py b/sdk/python/tests/integration/feature_repos/universal/online_store/singlestore.py
new file mode 100644
index 00000000000..d3a02421d0a
--- /dev/null
+++ b/sdk/python/tests/integration/feature_repos/universal/online_store/singlestore.py
@@ -0,0 +1,43 @@
+import subprocess
+import time
+from typing import Dict
+
+from testcontainers.core.container import DockerContainer
+
+from tests.integration.feature_repos.universal.online_store_creator import (
+ OnlineStoreCreator,
+)
+
+
+class SingleStoreOnlineStoreCreator(OnlineStoreCreator):
+ def __init__(self, project_name: str, **kwargs):
+ super().__init__(project_name)
+ self.container = (
+ DockerContainer("ghcr.io/singlestore-labs/singlestoredb-dev:latest")
+ .with_exposed_ports(3306)
+ .with_env("USER", "root")
+ .with_env("ROOT_PASSWORD", "test")
+ # this license key is authorized solely for use in SingleStore Feast tests and is subject to strict usage restrictions
+ # if you want a free SingleStore license for your own use please visit https://www.singlestore.com/cloud-trial/
+ .with_env(
+ "LICENSE_KEY",
+ "BGIxODZiYTg1YWUxYjRlODRhYzRjMGFmYTA1OTkxYzgyAAAAAAAAAAABAAAAAAAAACgwNQIZANx4NIXJ7CWvKYYb3wIyRXxBY7fdAnLeSwIYLy2Q0jA124GAkl04yuGrD59Zpv85DVYXAA==",
+ )
+ )
+
+ def create_online_store(self) -> Dict[str, str]:
+ self.container.start()
+ time.sleep(30)
+ exposed_port = self.container.get_exposed_port("3306")
+ command = f"mysql -uroot -ptest -P {exposed_port} -e 'CREATE DATABASE feast;'"
+ subprocess.run(command, shell=True, check=True)
+ return {
+ "type": "singlestore",
+ "user": "root",
+ "password": "test",
+ "database": "feast",
+ "port": exposed_port,
+ }
+
+ def teardown(self):
+ self.container.stop()
diff --git a/sdk/python/tests/integration/materialization/test_snowflake.py b/sdk/python/tests/integration/materialization/test_snowflake.py
index adb2bd7e7df..f12191363b4 100644
--- a/sdk/python/tests/integration/materialization/test_snowflake.py
+++ b/sdk/python/tests/integration/materialization/test_snowflake.py
@@ -8,6 +8,7 @@
from feast.entity import Entity
from feast.feature_view import FeatureView
from feast.types import Array, Bool, Bytes, Float64, Int32, Int64, String, UnixTimestamp
+from feast.utils import _utc_now
from tests.data.data_creator import create_basic_driver_dataset
from tests.integration.feature_repos.integration_test_repo_config import (
IntegrationTestRepoConfig,
@@ -146,7 +147,7 @@ def test_snowflake_materialization_consistency_internal_with_lists(
split_dt = df["ts_1"][4].to_pydatetime() - timedelta(seconds=1)
print(f"Split datetime: {split_dt}")
- now = datetime.utcnow()
+ now = _utc_now()
full_feature_names = True
start_date = (now - timedelta(hours=5)).replace(tzinfo=utc)
@@ -231,7 +232,7 @@ def test_snowflake_materialization_entityless_fv():
print(f"Split datetime: {split_dt}")
- now = datetime.utcnow()
+ now = _utc_now()
start_date = (now - timedelta(hours=5)).replace(tzinfo=utc)
end_date = split_dt
diff --git a/sdk/python/tests/integration/offline_store/test_offline_write.py b/sdk/python/tests/integration/offline_store/test_offline_write.py
index b8c465946df..63bdc4755ac 100644
--- a/sdk/python/tests/integration/offline_store/test_offline_write.py
+++ b/sdk/python/tests/integration/offline_store/test_offline_write.py
@@ -1,5 +1,5 @@
import random
-from datetime import datetime, timedelta
+from datetime import timedelta
import numpy as np
import pandas as pd
@@ -7,6 +7,7 @@
from feast import FeatureView, Field
from feast.types import Float32, Int32
+from feast.utils import _utc_now
from tests.integration.feature_repos.repo_configuration import (
construct_universal_feature_views,
)
@@ -23,7 +24,7 @@ def test_reorder_columns(environment, universal_data_sources):
driver_fv = feature_views.driver
store.apply([driver(), driver_fv])
- now = datetime.utcnow()
+ now = _utc_now()
ts = pd.Timestamp(now).round("ms")
# This dataframe has columns in the wrong order.
@@ -53,7 +54,7 @@ def test_writing_incorrect_schema_fails(environment, universal_data_sources):
driver_fv = feature_views.driver
store.apply([driver(), driver_fv])
- now = datetime.utcnow()
+ now = _utc_now()
ts = pd.Timestamp(now).round("ms")
expected_df = pd.DataFrame.from_dict(
@@ -91,7 +92,7 @@ def test_writing_consecutively_to_offline_store(environment, universal_data_sour
), # This is to make sure all offline store data is out of date since get_historical_features() only searches backwards for a ttl window.
)
- now = datetime.utcnow()
+ now = _utc_now()
ts = pd.Timestamp(now, unit="ns")
entity_df = pd.DataFrame.from_dict(
diff --git a/sdk/python/tests/integration/offline_store/test_push_features_to_offline_store.py b/sdk/python/tests/integration/offline_store/test_push_features_to_offline_store.py
index 0b1db9011a5..5e3d72e671b 100644
--- a/sdk/python/tests/integration/offline_store/test_push_features_to_offline_store.py
+++ b/sdk/python/tests/integration/offline_store/test_push_features_to_offline_store.py
@@ -1,10 +1,9 @@
-import datetime
-
import numpy as np
import pandas as pd
import pytest
from feast.data_source import PushMode
+from feast.utils import _utc_now
from tests.integration.feature_repos.repo_configuration import (
construct_universal_feature_views,
)
@@ -20,7 +19,7 @@ def test_push_features_and_read(environment, universal_data_sources):
location_fv = feature_views.pushed_locations
store.apply([location(), location_fv])
- now = pd.Timestamp(datetime.datetime.utcnow()).round("ms")
+ now = pd.Timestamp(_utc_now()).round("ms")
entity_df = pd.DataFrame.from_dict({"location_id": [1], "event_timestamp": [now]})
before_df = store.get_historical_features(
diff --git a/sdk/python/tests/integration/offline_store/test_universal_historical_retrieval.py b/sdk/python/tests/integration/offline_store/test_universal_historical_retrieval.py
index bfb8a56200a..ecaa5f40db6 100644
--- a/sdk/python/tests/integration/offline_store/test_universal_historical_retrieval.py
+++ b/sdk/python/tests/integration/offline_store/test_universal_historical_retrieval.py
@@ -15,6 +15,7 @@
DEFAULT_ENTITY_DF_EVENT_TIMESTAMP_COL,
)
from feast.types import Float32, Int32
+from feast.utils import _utc_now
from tests.integration.feature_repos.repo_configuration import (
construct_universal_feature_views,
table_name_from_data_source,
@@ -144,11 +145,11 @@ def test_historical_features_main(
files = job_from_df.to_remote_storage()
assert len(files) # 0 # This test should be way more detailed
- start_time = datetime.utcnow()
+ start_time = _utc_now()
actual_df_from_df_entities = job_from_df.to_df()
print(f"actual_df_from_df_entities shape: {actual_df_from_df_entities.shape}")
- end_time = datetime.utcnow()
+ end_time = _utc_now()
print(str(f"Time to execute job_from_df.to_df() = '{(end_time - start_time)}'\n"))
assert sorted(expected_df.columns) == sorted(actual_df_from_df_entities.columns)
@@ -303,9 +304,9 @@ def test_historical_features_with_entities_from_query(
full_feature_names=full_feature_names,
)
- start_time = datetime.utcnow()
+ start_time = _utc_now()
actual_df_from_sql_entities = job_from_sql.to_df()
- end_time = datetime.utcnow()
+ end_time = _utc_now()
print(str(f"\nTime to execute job_from_sql.to_df() = '{(end_time - start_time)}'"))
event_timestamp = (
@@ -618,11 +619,11 @@ def test_historical_features_containing_backfills(environment):
full_feature_names=False,
)
- start_time = datetime.utcnow()
+ start_time = _utc_now()
actual_df = offline_job.to_df()
print(f"actual_df shape: {actual_df.shape}")
- end_time = datetime.utcnow()
+ end_time = _utc_now()
print(str(f"Time to execute job_from_df.to_df() = '{(end_time - start_time)}'\n"))
assert sorted(expected_df.columns) == sorted(actual_df.columns)
diff --git a/sdk/python/tests/integration/offline_store/test_validation.py b/sdk/python/tests/integration/offline_store/test_validation.py
index 1731f823c89..6f0496e8c8d 100644
--- a/sdk/python/tests/integration/offline_store/test_validation.py
+++ b/sdk/python/tests/integration/offline_store/test_validation.py
@@ -16,7 +16,7 @@
LoggingConfig,
)
from feast.protos.feast.serving.ServingService_pb2 import FieldStatus
-from feast.utils import make_tzaware
+from feast.utils import _utc_now, make_tzaware
from feast.wait import wait_retry_backoff
from tests.integration.feature_repos.repo_configuration import (
construct_universal_feature_views,
@@ -316,8 +316,7 @@ def test_e2e_validation_via_cli(environment, universal_data_sources):
"avg_passenger_count": [0],
"lifetime_trip_count": [0],
"event_timestamp": [
- make_tzaware(datetime.datetime.utcnow())
- - datetime.timedelta(hours=1)
+ make_tzaware(_utc_now()) - datetime.timedelta(hours=1)
],
}
)
diff --git a/sdk/python/tests/integration/online_store/test_push_features_to_online_store.py b/sdk/python/tests/integration/online_store/test_push_features_to_online_store.py
index 42561563f99..98fe3ab1ec0 100644
--- a/sdk/python/tests/integration/online_store/test_push_features_to_online_store.py
+++ b/sdk/python/tests/integration/online_store/test_push_features_to_online_store.py
@@ -1,8 +1,7 @@
-import datetime
-
import pandas as pd
import pytest
+from feast.utils import _utc_now
from tests.integration.feature_repos.repo_configuration import (
construct_universal_feature_views,
)
@@ -21,8 +20,8 @@ def test_push_features_and_read(environment, universal_data_sources):
data = {
"location_id": [1],
"temperature": [4],
- "event_timestamp": [pd.Timestamp(datetime.datetime.utcnow()).round("ms")],
- "created": [pd.Timestamp(datetime.datetime.utcnow()).round("ms")],
+ "event_timestamp": [pd.Timestamp(_utc_now()).round("ms")],
+ "created": [pd.Timestamp(_utc_now()).round("ms")],
}
df_ingest = pd.DataFrame(data)
diff --git a/sdk/python/tests/integration/online_store/test_python_feature_server.py b/sdk/python/tests/integration/online_store/test_python_feature_server.py
index 089efd7a562..1010e731788 100644
--- a/sdk/python/tests/integration/online_store/test_python_feature_server.py
+++ b/sdk/python/tests/integration/online_store/test_python_feature_server.py
@@ -1,5 +1,4 @@
import json
-from datetime import datetime
from typing import List
import pytest
@@ -7,6 +6,7 @@
from feast.feast_object import FeastObject
from feast.feature_server import get_app
+from feast.utils import _utc_now
from tests.integration.feature_repos.repo_configuration import (
construct_universal_feature_views,
)
@@ -67,8 +67,8 @@ def test_push(python_fs_client):
"df": {
"location_id": [1],
"temperature": [initial_temp * 100],
- "event_timestamp": [str(datetime.utcnow())],
- "created": [str(datetime.utcnow())],
+ "event_timestamp": [str(_utc_now())],
+ "created": [str(_utc_now())],
},
}
)
@@ -98,8 +98,8 @@ def test_push_source_does_not_exist(python_fs_client):
"df": {
"location_id": [1],
"temperature": [initial_temp * 100],
- "event_timestamp": [str(datetime.utcnow())],
- "created": [str(datetime.utcnow())],
+ "event_timestamp": [str(_utc_now())],
+ "created": [str(_utc_now())],
},
}
),
diff --git a/sdk/python/tests/integration/online_store/test_remote_online_store.py b/sdk/python/tests/integration/online_store/test_remote_online_store.py
index 759a9c7a87b..1d5dd0fca0a 100644
--- a/sdk/python/tests/integration/online_store/test_remote_online_store.py
+++ b/sdk/python/tests/integration/online_store/test_remote_online_store.py
@@ -1,12 +1,12 @@
import os
import subprocess
import tempfile
-from datetime import datetime
from textwrap import dedent
import pytest
from feast.feature_store import FeatureStore
+from feast.utils import _utc_now
from feast.wait import wait_retry_backoff
from tests.utils.cli_repo_creator import CliRunner
from tests.utils.http_server import check_port_open, free_port
@@ -150,7 +150,7 @@ def _default_store(temp_dir, project_name) -> FeatureStore:
fs = FeatureStore(repo_path=repo_path)
fs.materialize_incremental(
- end_date=datetime.utcnow(), feature_views=["driver_hourly_stats"]
+ end_date=_utc_now(), feature_views=["driver_hourly_stats"]
)
return fs
diff --git a/sdk/python/tests/integration/online_store/test_universal_online.py b/sdk/python/tests/integration/online_store/test_universal_online.py
index e78c1053bf8..2ffe869ef50 100644
--- a/sdk/python/tests/integration/online_store/test_universal_online.py
+++ b/sdk/python/tests/integration/online_store/test_universal_online.py
@@ -22,6 +22,7 @@
from feast.infra.utils.postgres.postgres_config import ConnectionType
from feast.online_response import TIMESTAMP_POSTFIX
from feast.types import Float32, Int32, String
+from feast.utils import _utc_now
from feast.wait import wait_retry_backoff
from tests.integration.feature_repos.repo_configuration import (
Environment,
@@ -39,13 +40,18 @@
@pytest.mark.integration
@pytest.mark.universal_online_stores(only=["postgres"])
+@pytest.mark.parametrize(
+ "conn_type",
+ [ConnectionType.singleton, ConnectionType.pool],
+ ids=lambda v: f"conn_type:{v}",
+)
def test_connection_pool_online_stores(
- environment, universal_data_sources, fake_ingest_data
+ environment, universal_data_sources, fake_ingest_data, conn_type
):
if os.getenv("FEAST_IS_LOCAL_TEST", "False") == "True":
return
fs = environment.feature_store
- fs.config.online_store.conn_type = ConnectionType.pool
+ fs.config.online_store.conn_type = conn_type
fs.config.online_store.min_conn = 1
fs.config.online_store.max_conn = 10
@@ -131,9 +137,9 @@ def test_write_to_online_store_event_check(environment):
fs = environment.feature_store
# write same data points 3 with different timestamps
- now = pd.Timestamp(datetime.datetime.utcnow()).round("ms")
- hour_ago = pd.Timestamp(datetime.datetime.utcnow() - timedelta(hours=1)).round("ms")
- latest = pd.Timestamp(datetime.datetime.utcnow() + timedelta(seconds=1)).round("ms")
+ now = pd.Timestamp(_utc_now()).round("ms")
+ hour_ago = pd.Timestamp(_utc_now() - timedelta(hours=1)).round("ms")
+ latest = pd.Timestamp(_utc_now() + timedelta(seconds=1)).round("ms")
data = {
"id": [123, 567, 890],
@@ -216,7 +222,7 @@ def test_write_to_online_store_event_check(environment):
# writes to online store via datasource (dataframe_source) materialization
fs.materialize(
start_date=datetime.datetime.now() - timedelta(hours=12),
- end_date=datetime.datetime.utcnow(),
+ end_date=_utc_now(),
)
df = fs.get_online_features(
@@ -245,8 +251,8 @@ def test_write_to_online_store(environment, universal_data_sources):
"conv_rate": [0.85],
"acc_rate": [0.91],
"avg_daily_trips": [14],
- "event_timestamp": [pd.Timestamp(datetime.datetime.utcnow()).round("ms")],
- "created": [pd.Timestamp(datetime.datetime.utcnow()).round("ms")],
+ "event_timestamp": [pd.Timestamp(_utc_now()).round("ms")],
+ "created": [pd.Timestamp(_utc_now()).round("ms")],
}
df_data = pd.DataFrame(data)
@@ -482,7 +488,7 @@ def test_online_retrieval_with_event_timestamps(environment, universal_data_sour
@pytest.mark.integration
-@pytest.mark.universal_online_stores(only=["redis", "dynamodb"])
+@pytest.mark.universal_online_stores(only=["redis", "dynamodb", "postgres"])
def test_async_online_retrieval_with_event_timestamps(
environment, universal_data_sources
):
diff --git a/sdk/python/tests/integration/registration/test_universal_odfv_feature_inference.py b/sdk/python/tests/integration/registration/test_universal_odfv_feature_inference.py
index ce960b9c358..151f629289f 100644
--- a/sdk/python/tests/integration/registration/test_universal_odfv_feature_inference.py
+++ b/sdk/python/tests/integration/registration/test_universal_odfv_feature_inference.py
@@ -1,5 +1,3 @@
-from datetime import datetime
-
import pandas as pd
import pytest
@@ -7,6 +5,7 @@
from feast.errors import SpecifiedFeaturesNotPresentError
from feast.infra.offline_stores.file_source import FileSource
from feast.types import Float64
+from feast.utils import _utc_now
from tests.integration.feature_repos.universal.entities import customer, driver, item
from tests.integration.feature_repos.universal.feature_views import (
conv_rate_plus_100_feature_view,
@@ -50,8 +49,8 @@ def test_infer_odfv_list_features(environment, infer_features, tmp_path):
"item_id": [0],
"embedding_float": [fake_embedding],
"embedding_double": [fake_embedding],
- "event_timestamp": [pd.Timestamp(datetime.utcnow())],
- "created": [pd.Timestamp(datetime.utcnow())],
+ "event_timestamp": [pd.Timestamp(_utc_now())],
+ "created": [pd.Timestamp(_utc_now())],
}
)
output_path = f"{tmp_path}/items.parquet"
diff --git a/sdk/python/tests/integration/registration/test_universal_registry.py b/sdk/python/tests/integration/registration/test_universal_registry.py
index cd741853cc5..b0738c84190 100644
--- a/sdk/python/tests/integration/registration/test_universal_registry.py
+++ b/sdk/python/tests/integration/registration/test_universal_registry.py
@@ -22,12 +22,13 @@
import pandas as pd
import pytest
from pytest_lazyfixture import lazy_fixture
+from pytz import utc
from testcontainers.core.container import DockerContainer
from testcontainers.core.waiting_utils import wait_for_logs
from testcontainers.minio import MinioContainer
from testcontainers.mysql import MySqlContainer
-from feast import FileSource, RequestSource
+from feast import FeatureService, FileSource, RequestSource
from feast.data_format import AvroFormat, ParquetFormat
from feast.data_source import KafkaSource
from feast.entity import Entity
@@ -45,6 +46,7 @@
from feast.repo_config import RegistryConfig
from feast.stream_feature_view import Aggregation, StreamFeatureView
from feast.types import Array, Bytes, Float32, Int32, Int64, String
+from feast.utils import _utc_now
from feast.value_type import ValueType
from tests.integration.feature_repos.universal.entities import driver
@@ -123,7 +125,7 @@ def minio_registry() -> Registry:
logger = logging.getLogger(__name__)
-@pytest.fixture(scope="session")
+@pytest.fixture(scope="function")
def pg_registry():
container = (
DockerContainer("postgres:latest")
@@ -135,6 +137,35 @@ def pg_registry():
container.start()
+ registry_config = _given_registry_config_for_pg_sql(container)
+
+ yield SqlRegistry(registry_config, "project", None)
+
+ container.stop()
+
+
+@pytest.fixture(scope="function")
+def pg_registry_async():
+ container = (
+ DockerContainer("postgres:latest")
+ .with_exposed_ports(5432)
+ .with_env("POSTGRES_USER", POSTGRES_USER)
+ .with_env("POSTGRES_PASSWORD", POSTGRES_PASSWORD)
+ .with_env("POSTGRES_DB", POSTGRES_DB)
+ )
+
+ container.start()
+
+ registry_config = _given_registry_config_for_pg_sql(container, 2, "thread")
+
+ yield SqlRegistry(registry_config, "project", None)
+
+ container.stop()
+
+
+def _given_registry_config_for_pg_sql(
+ container, cache_ttl_seconds=2, cache_mode="sync"
+):
log_string_to_wait_for = "database system is ready to accept connections"
waited = wait_for_logs(
container=container,
@@ -146,23 +177,42 @@ def pg_registry():
container_port = container.get_exposed_port(5432)
container_host = container.get_container_host_ip()
- registry_config = RegistryConfig(
+ return RegistryConfig(
registry_type="sql",
- path=f"postgresql://{POSTGRES_USER}:{POSTGRES_PASSWORD}@{container_host}:{container_port}/{POSTGRES_DB}",
+ cache_ttl_seconds=cache_ttl_seconds,
+ cache_mode=cache_mode,
+ # The `path` must include `+psycopg` in order for `sqlalchemy.create_engine()`
+ # to understand that we are using psycopg3.
+ path=f"postgresql+psycopg://{POSTGRES_USER}:{POSTGRES_PASSWORD}@{container_host}:{container_port}/{POSTGRES_DB}",
sqlalchemy_config_kwargs={"echo": False, "pool_pre_ping": True},
)
+
+@pytest.fixture(scope="function")
+def mysql_registry():
+ container = MySqlContainer("mysql:latest")
+ container.start()
+
+ registry_config = _given_registry_config_for_mysql(container)
+
yield SqlRegistry(registry_config, "project", None)
container.stop()
-@pytest.fixture(scope="session")
-def mysql_registry():
+@pytest.fixture(scope="function")
+def mysql_registry_async():
container = MySqlContainer("mysql:latest")
container.start()
- # testing for the database to exist and ready to connect and start testing.
+ registry_config = _given_registry_config_for_mysql(container, 2, "thread")
+
+ yield SqlRegistry(registry_config, "project", None)
+
+ container.stop()
+
+
+def _given_registry_config_for_mysql(container, cache_ttl_seconds=2, cache_mode="sync"):
import sqlalchemy
engine = sqlalchemy.create_engine(
@@ -170,16 +220,14 @@ def mysql_registry():
)
engine.connect()
- registry_config = RegistryConfig(
+ return RegistryConfig(
registry_type="sql",
path=container.get_connection_url(),
+ cache_ttl_seconds=cache_ttl_seconds,
+ cache_mode=cache_mode,
sqlalchemy_config_kwargs={"echo": False, "pool_pre_ping": True},
)
- yield SqlRegistry(registry_config, "project", None)
-
- container.stop()
-
@pytest.fixture(scope="session")
def sqlite_registry():
@@ -265,6 +313,17 @@ def mock_remote_registry():
lazy_fixture("sqlite_registry"),
]
+async_sql_fixtures = [
+ pytest.param(
+ lazy_fixture("pg_registry_async"),
+ marks=pytest.mark.xdist_group(name="pg_registry_async"),
+ ),
+ pytest.param(
+ lazy_fixture("mysql_registry_async"),
+ marks=pytest.mark.xdist_group(name="mysql_registry_async"),
+ ),
+]
+
@pytest.mark.integration
@pytest.mark.parametrize("test_registry", all_fixtures)
@@ -308,6 +367,22 @@ def test_apply_entity_success(test_registry):
# After the first apply, the created_timestamp should be the same as the last_update_timestamp.
assert entity.created_timestamp == entity.last_updated_timestamp
+ # Update entity
+ updated_entity = Entity(
+ name="driver_car_id",
+ description="Car driver Id",
+ tags={"team": "matchmaking"},
+ )
+ test_registry.apply_entity(updated_entity, project)
+
+ updated_entity = test_registry.get_entity("driver_car_id", project)
+
+ # The created_timestamp for the entity should be set to the created_timestamp value stored from the previous apply
+ assert (
+ updated_entity.created_timestamp is not None
+ and updated_entity.created_timestamp == entity.created_timestamp
+ )
+
test_registry.delete_entity("driver_car_id", project)
assert_project_uuid(project, project_uuid, test_registry)
entities = test_registry.list_entities(project)
@@ -601,11 +676,54 @@ def odfv1(feature_df: pd.DataFrame) -> pd.DataFrame:
data["odfv1_my_feature_2"] = feature_df["my_input_1"].astype("int32")
return data
+ def simple_udf(x: int):
+ return x + 3
+
+ entity_sfv = Entity(name="sfv_my_entity_1", join_keys=["test_key"])
+
+ stream_source = KafkaSource(
+ name="kafka",
+ timestamp_field="event_timestamp",
+ kafka_bootstrap_servers="",
+ message_format=AvroFormat(""),
+ topic="topic",
+ batch_source=FileSource(path="some path"),
+ watermark_delay_threshold=timedelta(days=1),
+ )
+
+ sfv = StreamFeatureView(
+ name="test kafka stream feature view",
+ entities=[entity_sfv],
+ ttl=timedelta(days=30),
+ owner="test@example.com",
+ online=True,
+ schema=[Field(name="dummy_field", dtype=Float32)],
+ description="desc",
+ aggregations=[
+ Aggregation(
+ column="dummy_field",
+ function="max",
+ time_window=timedelta(days=1),
+ ),
+ Aggregation(
+ column="dummy_field2",
+ function="count",
+ time_window=timedelta(days=24),
+ ),
+ ],
+ timestamp_field="event_timestamp",
+ mode="spark",
+ source=stream_source,
+ udf=simple_udf,
+ tags={},
+ )
+
project = "project"
# Register Feature Views
test_registry.apply_feature_view(odfv1, project)
test_registry.apply_feature_view(fv1, project)
+ test_registry.apply_feature_view(sfv, project)
# Modify odfv by changing a single feature dtype
@on_demand_feature_view(
@@ -621,6 +739,8 @@ def odfv1(feature_df: pd.DataFrame) -> pd.DataFrame:
data["odfv1_my_feature_2"] = feature_df["my_input_1"].astype("int32")
return data
+ existing_odfv = test_registry.get_on_demand_feature_view("odfv1", project)
+
# Apply the modified odfv
test_registry.apply_feature_view(odfv1, project)
@@ -655,6 +775,11 @@ def odfv1(feature_df: pd.DataFrame) -> pd.DataFrame:
and list(request_schema.values())[0] == ValueType.INT32
)
+ assert (
+ feature_view.created_timestamp is not None
+ and feature_view.created_timestamp == existing_odfv.created_timestamp
+ )
+
# Make sure fv1 is untouched
feature_views = test_registry.list_feature_views(project, tags=fv1.tags)
@@ -675,7 +800,162 @@ def odfv1(feature_df: pd.DataFrame) -> pd.DataFrame:
and feature_view.entities[0] == "fs1_my_entity_1"
)
- test_registry.teardown()
+ # Simulate materialization
+ current_date = _utc_now()
+ end_date = current_date.replace(tzinfo=utc)
+ start_date = (current_date - timedelta(days=1)).replace(tzinfo=utc)
+ test_registry.apply_materialization(feature_view, project, start_date, end_date)
+ materialized_feature_view = test_registry.get_feature_view(
+ "my_feature_view_1", project
+ )
+
+ # Check if created_timestamp, along with materialized_intervals are updated
+ assert (
+ materialized_feature_view.created_timestamp is not None
+ and materialized_feature_view.created_timestamp
+ == feature_view.created_timestamp
+ and len(materialized_feature_view.materialization_intervals) > 0
+ and materialized_feature_view.materialization_intervals[0][0] == start_date
+ and materialized_feature_view.materialization_intervals[0][1] == end_date
+ )
+
+ # Modify fv1 by changing a single dtype
+ updated_fv1 = FeatureView(
+ name="my_feature_view_1",
+ schema=[
+ Field(name="test", dtype=Int64),
+ Field(name="fs1_my_feature_1", dtype=String),
+ ],
+ entities=[entity],
+ tags={"team": "matchmaking"},
+ source=batch_source,
+ ttl=timedelta(minutes=5),
+ )
+
+ # Check that these fields are empty before apply
+ assert updated_fv1.created_timestamp is None
+ assert len(updated_fv1.materialization_intervals) == 0
+
+ # Apply the modified fv1
+ test_registry.apply_feature_view(updated_fv1, project)
+
+ # Verify feature view after modification
+ updated_feature_views = test_registry.list_feature_views(project)
+
+ # List Feature Views
+ assert (
+ len(updated_feature_views) == 1
+ and updated_feature_views[0].name == "my_feature_view_1"
+ and updated_feature_views[0].features[0].name == "fs1_my_feature_1"
+ and updated_feature_views[0].features[0].dtype == String
+ and updated_feature_views[0].entities[0] == "fs1_my_entity_1"
+ )
+
+ updated_feature_view = test_registry.get_feature_view("my_feature_view_1", project)
+ assert (
+ updated_feature_view.name == "my_feature_view_1"
+ and updated_feature_view.features[0].name == "fs1_my_feature_1"
+ and updated_feature_view.features[0].dtype == String
+ and updated_feature_view.entities[0] == "fs1_my_entity_1"
+ )
+
+ # Check if materialization_intervals and created_timestamp values propagates on each apply
+ # materialization_intervals will populate only when it's empty
+ assert (
+ updated_feature_view.created_timestamp is not None
+ and updated_feature_view.created_timestamp == feature_view.created_timestamp
+ and len(updated_feature_view.materialization_intervals) == 1
+ and updated_feature_view.materialization_intervals[0][0] == start_date
+ and updated_feature_view.materialization_intervals[0][1] == end_date
+ )
+
+ # Simulate materialization a second time
+ current_date = _utc_now()
+ end_date_1 = current_date.replace(tzinfo=utc)
+ start_date_1 = (current_date - timedelta(days=1)).replace(tzinfo=utc)
+ test_registry.apply_materialization(
+ updated_feature_view, project, start_date_1, end_date_1
+ )
+ materialized_feature_view_1 = test_registry.get_feature_view(
+ "my_feature_view_1", project
+ )
+
+ assert (
+ materialized_feature_view_1.created_timestamp is not None
+ and materialized_feature_view_1.created_timestamp
+ == feature_view.created_timestamp
+ and len(materialized_feature_view_1.materialization_intervals) == 2
+ and materialized_feature_view_1.materialization_intervals[0][0] == start_date
+ and materialized_feature_view_1.materialization_intervals[0][1] == end_date
+ and materialized_feature_view_1.materialization_intervals[1][0] == start_date_1
+ and materialized_feature_view_1.materialization_intervals[1][1] == end_date_1
+ )
+
+ # Modify sfv by changing the dtype
+
+ sfv = StreamFeatureView(
+ name="test kafka stream feature view",
+ entities=[entity_sfv],
+ ttl=timedelta(days=30),
+ owner="test@example.com",
+ online=True,
+ schema=[Field(name="dummy_field", dtype=String)],
+ description="desc",
+ aggregations=[
+ Aggregation(
+ column="dummy_field",
+ function="max",
+ time_window=timedelta(days=1),
+ ),
+ Aggregation(
+ column="dummy_field2",
+ function="count",
+ time_window=timedelta(days=24),
+ ),
+ ],
+ timestamp_field="event_timestamp",
+ mode="spark",
+ source=stream_source,
+ udf=simple_udf,
+ tags={},
+ )
+
+ existing_sfv = test_registry.get_stream_feature_view(
+ "test kafka stream feature view", project
+ )
+ # Apply the modified sfv
+ test_registry.apply_feature_view(sfv, project)
+
+ # Verify feature view after modification
+ updated_stream_feature_views = test_registry.list_stream_feature_views(project)
+
+ # List Feature Views
+ assert (
+ len(updated_stream_feature_views) == 1
+ and updated_stream_feature_views[0].name == "test kafka stream feature view"
+ and updated_stream_feature_views[0].features[0].name == "dummy_field"
+ and updated_stream_feature_views[0].features[0].dtype == String
+ and updated_stream_feature_views[0].entities[0] == "sfv_my_entity_1"
+ )
+
+ updated_sfv = test_registry.get_stream_feature_view(
+ "test kafka stream feature view", project
+ )
+ assert (
+ updated_sfv.name == "test kafka stream feature view"
+ and updated_sfv.features[0].name == "dummy_field"
+ and updated_sfv.features[0].dtype == String
+ and updated_sfv.entities[0] == "sfv_my_entity_1"
+ )
+
+ # The created_timestamp for the stream feature view should be set to the created_timestamp value stored from the
+ # previous apply
+ # Materialization_intervals is not set
+ assert (
+ updated_sfv.created_timestamp is not None
+ and updated_sfv.created_timestamp == existing_sfv.created_timestamp
+ and len(updated_sfv.materialization_intervals) == 0
+ )
@pytest.mark.integration
@@ -774,6 +1054,44 @@ def test_registry_cache(test_registry):
test_registry.teardown()
+@pytest.mark.integration
+@pytest.mark.parametrize(
+ "test_registry",
+ async_sql_fixtures,
+)
+def test_registry_cache_thread_async(test_registry):
+ # Create Feature View
+ batch_source = FileSource(
+ name="test_source",
+ file_format=ParquetFormat(),
+ path="file://feast/*",
+ timestamp_field="ts_col",
+ created_timestamp_column="timestamp",
+ )
+
+ project = "project"
+
+ # Register data source
+ test_registry.apply_data_source(batch_source, project)
+ registry_data_sources_cached = test_registry.list_data_sources(
+ project, allow_cache=True
+ )
+ # async ttl yet to expire, so there will be a cache miss
+ assert len(registry_data_sources_cached) == 0
+
+ # Wait for cache to be refreshed
+ time.sleep(4)
+ # Now objects exist
+ registry_data_sources_cached = test_registry.list_data_sources(
+ project, allow_cache=True
+ )
+ assert len(registry_data_sources_cached) == 1
+ registry_data_source = registry_data_sources_cached[0]
+ assert registry_data_source == batch_source
+
+ test_registry.teardown()
+
+
@pytest.mark.integration
@pytest.mark.parametrize(
"test_registry",
@@ -825,7 +1143,7 @@ def simple_udf(x: int):
project = "project"
- # Register Feature View
+ # Register Stream Feature View
test_registry.apply_feature_view(sfv, project)
stream_feature_views = test_registry.list_stream_feature_views(
@@ -843,6 +1161,100 @@ def simple_udf(x: int):
test_registry.teardown()
+@pytest.mark.integration
+@pytest.mark.parametrize(
+ "test_registry",
+ all_fixtures,
+)
+def test_apply_feature_service_success(test_registry):
+ # Create Feature Service
+ file_source = FileSource(name="my_file_source", path="test.parquet")
+ feature_view = FeatureView(
+ name="my_feature_view",
+ entities=[],
+ schema=[
+ Field(name="feature1", dtype=Float32),
+ Field(name="feature2", dtype=Float32),
+ ],
+ source=file_source,
+ )
+ fs = FeatureService(
+ name="my_feature_service_1", features=[feature_view[["feature1", "feature2"]]]
+ )
+ project = "project"
+
+ # Register Feature Service
+ test_registry.apply_feature_service(fs, project)
+
+ feature_services = test_registry.list_feature_services(project)
+
+ # List Feature Services
+ assert len(feature_services) == 1
+ assert feature_services[0] == fs
+
+ # Delete Feature Service
+ test_registry.delete_feature_service("my_feature_service_1", project)
+ feature_services = test_registry.list_feature_services(project)
+ assert len(feature_services) == 0
+
+ test_registry.teardown()
+
+
+@pytest.mark.integration
+@pytest.mark.parametrize(
+ "test_registry",
+ all_fixtures,
+)
+def test_modify_feature_service_success(test_registry):
+ # Create Feature Service
+ file_source = FileSource(name="my_file_source", path="test.parquet")
+ feature_view = FeatureView(
+ name="my_feature_view",
+ entities=[],
+ schema=[
+ Field(name="feature1", dtype=Float32),
+ Field(name="feature2", dtype=Float32),
+ ],
+ source=file_source,
+ )
+ fs = FeatureService(
+ name="my_feature_service_1", features=[feature_view[["feature1", "feature2"]]]
+ )
+ project = "project"
+
+ # Register Feature service
+ test_registry.apply_feature_service(fs, project)
+
+ feature_services = test_registry.list_feature_services(project)
+
+ # List Feature Services
+ assert len(feature_services) == 1
+ assert feature_services[0] == fs
+
+ # Modify Feature Service by removing a feature
+ fs = FeatureService(
+ name="my_feature_service_1", features=[feature_view[["feature1"]]]
+ )
+
+ # Apply modified Feature Service
+ test_registry.apply_feature_service(fs, project)
+
+ updated_feature_services = test_registry.list_feature_services(project)
+
+ # Verify Feature Services
+ assert len(updated_feature_services) == 1
+ assert updated_feature_services[0] == fs
+ # The created_timestamp for the feature service should be set to the created_timestamp value stored from the
+ # previous apply
+ assert (
+ updated_feature_services[0].created_timestamp is not None
+ and updated_feature_services[0].created_timestamp
+ == feature_services[0].created_timestamp
+ )
+
+ test_registry.teardown()
+
+
@pytest.mark.integration
def test_commit():
fd, registry_path = mkstemp()
diff --git a/sdk/python/tests/integration/registration/test_universal_types.py b/sdk/python/tests/integration/registration/test_universal_types.py
index ca15681c9b2..928d05ad31e 100644
--- a/sdk/python/tests/integration/registration/test_universal_types.py
+++ b/sdk/python/tests/integration/registration/test_universal_types.py
@@ -20,6 +20,7 @@
String,
UnixTimestamp,
)
+from feast.utils import _utc_now
from tests.data.data_creator import create_basic_driver_dataset
from tests.integration.feature_repos.universal.entities import driver
from tests.integration.feature_repos.universal.feature_views import driver_feature_view
@@ -93,7 +94,7 @@ def test_feature_get_historical_features_types_match(
entity_df = pd.DataFrame()
entity_df["driver_id"] = [1, 3]
- ts = pd.Timestamp(datetime.utcnow()).round("ms")
+ ts = pd.Timestamp(_utc_now()).round("ms")
entity_df["ts"] = [
ts - timedelta(hours=4),
ts - timedelta(hours=2),
diff --git a/sdk/python/tests/unit/cli/test_cli_chdir.py b/sdk/python/tests/unit/cli/test_cli_chdir.py
index 12ca8f6b084..dd592db0743 100644
--- a/sdk/python/tests/unit/cli/test_cli_chdir.py
+++ b/sdk/python/tests/unit/cli/test_cli_chdir.py
@@ -1,7 +1,8 @@
import tempfile
-from datetime import datetime, timedelta
+from datetime import timedelta
from pathlib import Path
+from feast.utils import _utc_now
from tests.utils.cli_repo_creator import CliRunner
@@ -29,7 +30,7 @@ def test_cli_chdir() -> None:
)
assert result.returncode == 0
- end_date = datetime.utcnow()
+ end_date = _utc_now()
start_date = end_date - timedelta(days=100)
result = runner.run(
[
diff --git a/sdk/python/tests/unit/infra/offline_stores/test_offline_store.py b/sdk/python/tests/unit/infra/offline_stores/test_offline_store.py
index fd50d376322..6d5eeb90c71 100644
--- a/sdk/python/tests/unit/infra/offline_stores/test_offline_store.py
+++ b/sdk/python/tests/unit/infra/offline_stores/test_offline_store.py
@@ -9,9 +9,6 @@
AthenaOfflineStoreConfig,
AthenaRetrievalJob,
)
-from feast.infra.offline_stores.contrib.mssql_offline_store.mssql import (
- MsSqlServerRetrievalJob,
-)
from feast.infra.offline_stores.contrib.postgres_offline_store.postgres import (
PostgreSQLOfflineStoreConfig,
PostgreSQLRetrievalJob,
@@ -23,7 +20,7 @@
from feast.infra.offline_stores.contrib.trino_offline_store.trino import (
TrinoRetrievalJob,
)
-from feast.infra.offline_stores.file import FileRetrievalJob
+from feast.infra.offline_stores.dask import DaskRetrievalJob
from feast.infra.offline_stores.offline_store import RetrievalJob, RetrievalMetadata
from feast.infra.offline_stores.redshift import (
RedshiftOfflineStoreConfig,
@@ -100,11 +97,10 @@ def metadata(self) -> Optional[RetrievalMetadata]:
@pytest.fixture(
params=[
MockRetrievalJob,
- FileRetrievalJob,
+ DaskRetrievalJob,
RedshiftRetrievalJob,
SnowflakeRetrievalJob,
AthenaRetrievalJob,
- MsSqlServerRetrievalJob,
PostgreSQLRetrievalJob,
SparkRetrievalJob,
TrinoRetrievalJob,
@@ -112,8 +108,8 @@ def metadata(self) -> Optional[RetrievalMetadata]:
]
)
def retrieval_job(request, environment):
- if request.param is FileRetrievalJob:
- return FileRetrievalJob(lambda: 1, full_feature_names=False)
+ if request.param is DaskRetrievalJob:
+ return DaskRetrievalJob(lambda: 1, full_feature_names=False)
elif request.param is RedshiftRetrievalJob:
offline_store_config = RedshiftOfflineStoreConfig(
cluster_id="feast-int-bucket",
@@ -124,7 +120,7 @@ def retrieval_job(request, environment):
iam_role="arn:aws:iam::585132637328:role/service-role/AmazonRedshift-CommandsAccessRole-20240403T092631",
workgroup="",
)
- config = environment.config.copy(
+ config = environment.config.model_copy(
update={"offline_config": offline_store_config}
)
return RedshiftRetrievalJob(
@@ -147,7 +143,7 @@ def retrieval_job(request, environment):
storage_integration_name="FEAST_S3",
blob_export_location="s3://feast-snowflake-offload/export",
)
- config = environment.config.copy(
+ config = environment.config.model_copy(
update={"offline_config": offline_store_config}
)
environment.project = "project"
@@ -173,13 +169,6 @@ def retrieval_job(request, environment):
config=environment.config,
full_feature_names=False,
)
- elif request.param is MsSqlServerRetrievalJob:
- return MsSqlServerRetrievalJob(
- query="query",
- engine=MagicMock(),
- config=environment.config,
- full_feature_names=False,
- )
elif request.param is PostgreSQLRetrievalJob:
offline_store_config = PostgreSQLOfflineStoreConfig(
host="str",
diff --git a/sdk/python/tests/unit/infra/online_store/test_dynamodb_online_store.py b/sdk/python/tests/unit/infra/online_store/test_dynamodb_online_store.py
index 6045dbc6ce0..6ff7b3c3605 100644
--- a/sdk/python/tests/unit/infra/online_store/test_dynamodb_online_store.py
+++ b/sdk/python/tests/unit/infra/online_store/test_dynamodb_online_store.py
@@ -5,7 +5,7 @@
import pytest
from moto import mock_dynamodb
-from feast.infra.offline_stores.file import FileOfflineStoreConfig
+from feast.infra.offline_stores.dask import DaskOfflineStoreConfig
from feast.infra.online_stores.dynamodb import (
DynamoDBOnlineStore,
DynamoDBOnlineStoreConfig,
@@ -40,7 +40,7 @@ def repo_config():
provider=PROVIDER,
online_store=DynamoDBOnlineStoreConfig(region=REGION),
# online_store={"type": "dynamodb", "region": REGION},
- offline_store=FileOfflineStoreConfig(),
+ offline_store=DaskOfflineStoreConfig(),
entity_key_serialization_version=2,
)
diff --git a/sdk/python/tests/unit/infra/scaffolding/test_repo_config.py b/sdk/python/tests/unit/infra/scaffolding/test_repo_config.py
index e1839fbd8b4..98d82ce3575 100644
--- a/sdk/python/tests/unit/infra/scaffolding/test_repo_config.py
+++ b/sdk/python/tests/unit/infra/scaffolding/test_repo_config.py
@@ -33,36 +33,6 @@ def _test_config(config_text, expect_error: Optional[str]):
return rc
-def test_nullable_online_store_aws():
- _test_config(
- dedent(
- """
- project: foo
- registry: "registry.db"
- provider: aws
- online_store: null
- entity_key_serialization_version: 2
- """
- ),
- expect_error="4 validation errors for RepoConfig\nregion\n Field required",
- )
-
-
-def test_nullable_online_store_gcp():
- _test_config(
- dedent(
- """
- project: foo
- registry: "registry.db"
- provider: gcp
- online_store: null
- entity_key_serialization_version: 2
- """
- ),
- expect_error=None,
- )
-
-
def test_nullable_online_store_local():
_test_config(
dedent(
@@ -125,20 +95,6 @@ def test_local_config_with_full_online_class_directly():
assert isinstance(c.online_store, SqliteOnlineStoreConfig)
-def test_gcp_config():
- _test_config(
- dedent(
- """
- project: foo
- registry: gs://registry.db
- provider: gcp
- entity_key_serialization_version: 2
- """
- ),
- expect_error=None,
- )
-
-
def test_extra_field():
_test_config(
dedent(
@@ -224,3 +180,18 @@ def test_invalid_project_name():
),
expect_error="alphanumerical values ",
)
+
+
+def test_no_provider():
+ _test_config(
+ dedent(
+ """
+ project: foo
+ registry: "registry.db"
+ online_store:
+ path: "blah"
+ entity_key_serialization_version: 2
+ """
+ ),
+ expect_error=None,
+ )
diff --git a/sdk/python/tests/unit/infra/scaffolding/test_repo_operations.py b/sdk/python/tests/unit/infra/scaffolding/test_repo_operations.py
index 70c8b05c2ed..aa4ff1c40f7 100644
--- a/sdk/python/tests/unit/infra/scaffolding/test_repo_operations.py
+++ b/sdk/python/tests/unit/infra/scaffolding/test_repo_operations.py
@@ -15,12 +15,14 @@ def feature_repo(feastignore_contents: Optional[str]):
repo_root = Path(tmp_dir)
(repo_root / "foo").mkdir()
(repo_root / "foo1").mkdir()
+ (repo_root / ".ipynb_checkpoints/").mkdir()
(repo_root / "foo1/bar").mkdir()
(repo_root / "bar").mkdir()
(repo_root / "bar/subdir1").mkdir()
(repo_root / "bar/subdir1/subdir2").mkdir()
(repo_root / "a.py").touch()
+ (repo_root / ".ipynb_checkpoints/test-checkpoint.ipynb").touch()
(repo_root / "foo/b.py").touch()
(repo_root / "foo1/c.py").touch()
(repo_root / "foo1/bar/d.py").touch()
diff --git a/sdk/python/tests/unit/infra/test_key_encoding_utils.py b/sdk/python/tests/unit/infra/test_key_encoding_utils.py
index df691ea21e0..658c04c358b 100644
--- a/sdk/python/tests/unit/infra/test_key_encoding_utils.py
+++ b/sdk/python/tests/unit/infra/test_key_encoding_utils.py
@@ -1,8 +1,14 @@
import pytest
-from feast.infra.key_encoding_utils import serialize_entity_key
+from feast.infra.key_encoding_utils import (
+ _deserialize_value,
+ _serialize_val,
+ deserialize_entity_key,
+ serialize_entity_key,
+)
from feast.protos.feast.types.EntityKey_pb2 import EntityKey as EntityKeyProto
from feast.protos.feast.types.Value_pb2 import Value as ValueProto
+from feast.protos.feast.types.Value_pb2 import ValueType
def test_serialize_entity_key():
@@ -28,3 +34,66 @@ def test_serialize_entity_key():
join_keys=["user"], entity_values=[ValueProto(int64_val=int(2**31))]
),
)
+
+
+def test_deserialize_entity_key():
+ serialized_entity_key = serialize_entity_key(
+ EntityKeyProto(
+ join_keys=["user"], entity_values=[ValueProto(int64_val=int(2**15))]
+ ),
+ entity_key_serialization_version=3,
+ )
+
+ deserialized_entity_key = deserialize_entity_key(
+ serialized_entity_key, entity_key_serialization_version=3
+ )
+ assert deserialized_entity_key == EntityKeyProto(
+ join_keys=["user"], entity_values=[ValueProto(int64_val=int(2**15))]
+ )
+
+
+def test_serialize_value():
+ v, t = _serialize_val("string_val", ValueProto(string_val="test"))
+ assert t == ValueType.STRING
+ assert v == b"test"
+
+ v, t = _serialize_val("bytes_val", ValueProto(bytes_val=b"test"))
+ assert t == ValueType.BYTES
+ assert v == b"test"
+
+ v, t = _serialize_val("int32_val", ValueProto(int32_val=1))
+ assert t == ValueType.INT32
+ assert v == b"\x01\x00\x00\x00"
+
+ # default entity_key_serialization_version is 1, so the result should be 4 bytes
+ v, t = _serialize_val("int64_val", ValueProto(int64_val=1))
+ assert t == ValueType.INT64
+ assert v == b"\x01\x00\x00\x00"
+
+ # current entity_key_serialization_version is 2, so the result should be 8 bytes
+ v, t = _serialize_val(
+ "int64_val", ValueProto(int64_val=1), entity_key_serialization_version=2
+ )
+ assert t == ValueType.INT64
+ assert v == b"\x01\x00\x00\x00\x00\x00\x00\x00"
+
+ # new entity_key_serialization_version is 3, the result should be same as version 2
+ v, t = _serialize_val(
+ "int64_val", ValueProto(int64_val=1), entity_key_serialization_version=3
+ )
+ assert t == ValueType.INT64
+ assert v == b"\x01\x00\x00\x00\x00\x00\x00\x00"
+
+
+def test_deserialize_value():
+ v = _deserialize_value(ValueType.STRING, b"test")
+ assert v.string_val == "test"
+
+ v = _deserialize_value(ValueType.BYTES, b"test")
+ assert v.bytes_val == b"test"
+
+ v = _deserialize_value(ValueType.INT32, b"\x01\x00\x00\x00")
+ assert v.int32_val == 1
+
+ v = _deserialize_value(ValueType.INT64, b"\x01\x00\x00\x00\x00\x00\x00\x00")
+ assert v.int64_val == 1
diff --git a/sdk/python/tests/unit/local_feast_tests/test_init.py b/sdk/python/tests/unit/local_feast_tests/test_init.py
index c5d3cbe57d4..4543a239796 100644
--- a/sdk/python/tests/unit/local_feast_tests/test_init.py
+++ b/sdk/python/tests/unit/local_feast_tests/test_init.py
@@ -1,8 +1,9 @@
import tempfile
-from datetime import datetime, timedelta
+from datetime import timedelta
from pathlib import Path
from textwrap import dedent
+from feast.utils import _utc_now
from tests.utils.cli_repo_creator import CliRunner
@@ -20,7 +21,7 @@ def test_repo_init() -> None:
result = runner.run(["apply"], cwd=repo_path)
assert result.returncode == 0
- end_date = datetime.utcnow()
+ end_date = _utc_now()
start_date = end_date - timedelta(days=100)
result = runner.run(
["materialize", start_date.isoformat(), end_date.isoformat()], cwd=repo_path
diff --git a/sdk/python/tests/unit/local_feast_tests/test_local_feature_store.py b/sdk/python/tests/unit/local_feast_tests/test_local_feature_store.py
index 6b7856f347c..0e834e314b1 100644
--- a/sdk/python/tests/unit/local_feast_tests/test_local_feature_store.py
+++ b/sdk/python/tests/unit/local_feast_tests/test_local_feature_store.py
@@ -134,7 +134,7 @@ def test_apply_feature_view(test_feature_store):
tags_filter = utils.tags_str_to_dict("('team:matchmaking',)")
assert tags_filter == tags_dict
tags_filter = utils.tags_list_to_dict(("team:matchmaking", "test"))
- assert tags_dict == tags_dict
+ assert tags_filter == tags_dict
# List Feature Views
feature_views = test_feature_store.list_batch_feature_views(tags=tags_filter)
@@ -389,6 +389,20 @@ def test_reapply_feature_view(test_feature_store, dataframe_source):
# Check Feature View
fv_stored = test_feature_store.get_feature_view(fv1.name)
+ assert len(fv_stored.materialization_intervals) == 1
+
+ # Change and apply Feature View, this time, only the name
+ fv2 = FeatureView(
+ name="my_feature_view_2",
+ schema=[Field(name="int64_col", dtype=Int64)],
+ entities=[e],
+ source=file_source,
+ ttl=timedelta(minutes=5),
+ )
+ test_feature_store.apply([fv2])
+
+ # Check Feature View
+ fv_stored = test_feature_store.get_feature_view(fv2.name)
assert len(fv_stored.materialization_intervals) == 0
test_feature_store.teardown()
diff --git a/sdk/python/tests/unit/online_store/test_online_retrieval.py b/sdk/python/tests/unit/online_store/test_online_retrieval.py
index 1e8cf45dcc6..0b552c04531 100644
--- a/sdk/python/tests/unit/online_store/test_online_retrieval.py
+++ b/sdk/python/tests/unit/online_store/test_online_retrieval.py
@@ -3,7 +3,6 @@
import sqlite3
import sys
import time
-from datetime import datetime
import numpy as np
import pandas as pd
@@ -17,6 +16,7 @@
from feast.protos.feast.types.Value_pb2 import FloatList as FloatListProto
from feast.protos.feast.types.Value_pb2 import Value as ValueProto
from feast.repo_config import RegistryConfig
+from feast.utils import _utc_now
from tests.integration.feature_repos.universal.feature_views import TAGS
from tests.utils.cli_repo_creator import CliRunner, get_example_repo
@@ -51,8 +51,8 @@ def test_get_online_features() -> None:
"lat": ValueProto(double_val=0.1),
"lon": ValueProto(string_val="1.0"),
},
- datetime.utcnow(),
- datetime.utcnow(),
+ _utc_now(),
+ _utc_now(),
)
],
progress=None,
@@ -72,8 +72,8 @@ def test_get_online_features() -> None:
"name": ValueProto(string_val="John"),
"age": ValueProto(int64_val=3),
},
- datetime.utcnow(),
- datetime.utcnow(),
+ _utc_now(),
+ _utc_now(),
)
],
progress=None,
@@ -90,8 +90,8 @@ def test_get_online_features() -> None:
(
customer_key,
{"trips": ValueProto(int64_val=7)},
- datetime.utcnow(),
- datetime.utcnow(),
+ _utc_now(),
+ _utc_now(),
)
],
progress=None,
@@ -318,8 +318,8 @@ def test_online_to_df():
"lat": ValueProto(double_val=d * lat_multiply),
"lon": ValueProto(string_val=str(d * lon_multiply)),
},
- datetime.utcnow(),
- datetime.utcnow(),
+ _utc_now(),
+ _utc_now(),
)
],
progress=None,
@@ -348,8 +348,8 @@ def test_online_to_df():
"name": ValueProto(string_val=name + str(c)),
"age": ValueProto(int64_val=c * age_multiply),
},
- datetime.utcnow(),
- datetime.utcnow(),
+ _utc_now(),
+ _utc_now(),
)
],
progress=None,
@@ -372,8 +372,8 @@ def test_online_to_df():
(
combo_keys,
{"trips": ValueProto(int64_val=c * d)},
- datetime.utcnow(),
- datetime.utcnow(),
+ _utc_now(),
+ _utc_now(),
)
],
progress=None,
@@ -468,8 +468,8 @@ def test_sqlite_get_online_documents() -> None:
)
)
},
- datetime.utcnow(),
- datetime.utcnow(),
+ _utc_now(),
+ _utc_now(),
)
)
@@ -488,7 +488,7 @@ def test_sqlite_get_online_documents() -> None:
)
for i in range(n)
],
- "event_timestamp": [datetime.utcnow() for _ in range(n)],
+ "event_timestamp": [_utc_now() for _ in range(n)],
}
)
diff --git a/sdk/python/tests/unit/test_feature_views.py b/sdk/python/tests/unit/test_feature_views.py
index 0220d1a8a95..981968df0de 100644
--- a/sdk/python/tests/unit/test_feature_views.py
+++ b/sdk/python/tests/unit/test_feature_views.py
@@ -12,6 +12,7 @@
from feast.infra.offline_stores.file_source import FileSource
from feast.protos.feast.types.Value_pb2 import ValueType
from feast.types import Float32
+from feast.utils import _utc_now, make_tzaware
def test_create_feature_view_with_conflicting_entities():
@@ -117,3 +118,53 @@ def test_hash():
def test_field_types():
with pytest.raises(TypeCheckError):
Field(name="name", dtype=ValueType.INT32)
+
+
+def test_update_materialization_intervals():
+ batch_source = FileSource(path="some path")
+ entity = Entity(name="entity_1", description="Some entity")
+ # Create a feature view that is already present in the SQL registry
+ stored_feature_view = FeatureView(
+ name="my-feature-view",
+ entities=[entity],
+ ttl=timedelta(days=1),
+ source=batch_source,
+ )
+
+ # Update the Feature View without modifying anything
+ updated_feature_view = FeatureView(
+ name="my-feature-view",
+ entities=[entity],
+ ttl=timedelta(days=1),
+ source=batch_source,
+ )
+ updated_feature_view.update_materialization_intervals(
+ stored_feature_view.materialization_intervals
+ )
+ assert len(updated_feature_view.materialization_intervals) == 0
+
+ current_time = _utc_now()
+ start_date = make_tzaware(current_time - timedelta(days=1))
+ end_date = make_tzaware(current_time)
+ updated_feature_view.materialization_intervals.append((start_date, end_date))
+
+ # Update the Feature View, i.e. simply update the name
+ second_updated_feature_view = FeatureView(
+ name="my-feature-view-1",
+ entities=[entity],
+ ttl=timedelta(days=1),
+ source=batch_source,
+ )
+
+ second_updated_feature_view.update_materialization_intervals(
+ updated_feature_view.materialization_intervals
+ )
+ assert len(second_updated_feature_view.materialization_intervals) == 1
+ assert (
+ second_updated_feature_view.materialization_intervals[0][0]
+ == updated_feature_view.materialization_intervals[0][0]
+ )
+ assert (
+ second_updated_feature_view.materialization_intervals[0][1]
+ == updated_feature_view.materialization_intervals[0][1]
+ )
diff --git a/sdk/python/tests/unit/test_on_demand_pandas_transformation.py b/sdk/python/tests/unit/test_on_demand_pandas_transformation.py
index c5f066dd83d..1a04a466fbc 100644
--- a/sdk/python/tests/unit/test_on_demand_pandas_transformation.py
+++ b/sdk/python/tests/unit/test_on_demand_pandas_transformation.py
@@ -1,15 +1,31 @@
import os
+import re
import tempfile
from datetime import datetime, timedelta
import pandas as pd
+import pytest
-from feast import Entity, FeatureStore, FeatureView, FileSource, RepoConfig
+from feast import (
+ Entity,
+ FeatureStore,
+ FeatureView,
+ FileSource,
+ RepoConfig,
+ RequestSource,
+)
from feast.driver_test_data import create_driver_hourly_stats_df
from feast.field import Field
from feast.infra.online_stores.sqlite import SqliteOnlineStoreConfig
from feast.on_demand_feature_view import on_demand_feature_view
-from feast.types import Float32, Float64, Int64
+from feast.types import (
+ Array,
+ Bool,
+ Float32,
+ Float64,
+ Int64,
+ String,
+)
def test_pandas_transformation():
@@ -91,3 +107,237 @@ def pandas_view(inputs: pd.DataFrame) -> pd.DataFrame:
assert online_response["conv_rate_plus_acc"].equals(
online_response["conv_rate"] + online_response["acc_rate"]
)
+
+
+def test_pandas_transformation_returning_all_data_types():
+ with tempfile.TemporaryDirectory() as data_dir:
+ store = FeatureStore(
+ config=RepoConfig(
+ project="test_on_demand_python_transformation",
+ registry=os.path.join(data_dir, "registry.db"),
+ provider="local",
+ entity_key_serialization_version=2,
+ online_store=SqliteOnlineStoreConfig(
+ path=os.path.join(data_dir, "online.db")
+ ),
+ )
+ )
+
+ # Generate test data.
+ end_date = datetime.now().replace(microsecond=0, second=0, minute=0)
+ start_date = end_date - timedelta(days=15)
+
+ driver_entities = [1001, 1002, 1003, 1004, 1005]
+ driver_df = create_driver_hourly_stats_df(driver_entities, start_date, end_date)
+ driver_stats_path = os.path.join(data_dir, "driver_stats.parquet")
+ driver_df.to_parquet(path=driver_stats_path, allow_truncated_timestamps=True)
+
+ driver = Entity(name="driver", join_keys=["driver_id"])
+
+ driver_stats_source = FileSource(
+ name="driver_hourly_stats_source",
+ path=driver_stats_path,
+ timestamp_field="event_timestamp",
+ created_timestamp_column="created",
+ )
+
+ driver_stats_fv = FeatureView(
+ name="driver_hourly_stats",
+ entities=[driver],
+ ttl=timedelta(days=0),
+ schema=[
+ Field(name="conv_rate", dtype=Float32),
+ Field(name="acc_rate", dtype=Float32),
+ Field(name="avg_daily_trips", dtype=Int64),
+ ],
+ online=True,
+ source=driver_stats_source,
+ )
+
+ request_source = RequestSource(
+ name="request_source",
+ schema=[
+ Field(name="avg_daily_trip_rank_thresholds", dtype=Array(Int64)),
+ Field(name="avg_daily_trip_rank_names", dtype=Array(String)),
+ ],
+ )
+
+ @on_demand_feature_view(
+ sources=[request_source, driver_stats_fv],
+ schema=[
+ Field(name="highest_achieved_rank", dtype=String),
+ Field(name="avg_daily_trips_plus_one", dtype=Int64),
+ Field(name="conv_rate_plus_acc", dtype=Float64),
+ Field(name="is_highest_rank", dtype=Bool),
+ Field(name="achieved_ranks", dtype=Array(String)),
+ Field(name="trips_until_next_rank_int", dtype=Array(Int64)),
+ Field(name="trips_until_next_rank_float", dtype=Array(Float64)),
+ Field(name="achieved_ranks_mask", dtype=Array(Bool)),
+ ],
+ mode="pandas",
+ )
+ def pandas_view(inputs: pd.DataFrame) -> pd.DataFrame:
+ df = pd.DataFrame()
+ df["conv_rate_plus_acc"] = inputs["conv_rate"] + inputs["acc_rate"]
+ df["avg_daily_trips_plus_one"] = inputs["avg_daily_trips"] + 1
+
+ df["trips_until_next_rank_int"] = inputs[
+ ["avg_daily_trips", "avg_daily_trip_rank_thresholds"]
+ ].apply(
+ lambda x: [max(threshold - x.iloc[0], 0) for threshold in x.iloc[1]],
+ axis=1,
+ )
+ df["trips_until_next_rank_float"] = df["trips_until_next_rank_int"].map(
+ lambda values: [float(value) for value in values]
+ )
+ df["achieved_ranks_mask"] = df["trips_until_next_rank_int"].map(
+ lambda values: [value <= 0 for value in values]
+ )
+
+ temp = pd.concat(
+ [df[["achieved_ranks_mask"]], inputs[["avg_daily_trip_rank_names"]]],
+ axis=1,
+ )
+ df["achieved_ranks"] = temp.apply(
+ lambda x: [
+ rank if achieved else "Locked"
+ for achieved, rank in zip(x.iloc[0], x.iloc[1])
+ ],
+ axis=1,
+ )
+ df["highest_achieved_rank"] = (
+ df["achieved_ranks"]
+ .map(
+ lambda ranks: str(
+ ([rank for rank in ranks if rank != "Locked"][-1:] or ["None"])[
+ 0
+ ]
+ )
+ )
+ .astype("string")
+ )
+ df["is_highest_rank"] = df["achieved_ranks"].map(
+ lambda ranks: ranks[-1] != "Locked"
+ )
+ return df
+
+ store.apply([driver, driver_stats_source, driver_stats_fv, pandas_view])
+
+ entity_rows = [
+ {
+ "driver_id": 1001,
+ "avg_daily_trip_rank_thresholds": [100, 250, 500, 1000],
+ "avg_daily_trip_rank_names": ["Bronze", "Silver", "Gold", "Platinum"],
+ }
+ ]
+ store.write_to_online_store(
+ feature_view_name="driver_hourly_stats", df=driver_df
+ )
+
+ online_response = store.get_online_features(
+ entity_rows=entity_rows,
+ features=[
+ "driver_hourly_stats:conv_rate",
+ "driver_hourly_stats:acc_rate",
+ "driver_hourly_stats:avg_daily_trips",
+ "pandas_view:avg_daily_trips_plus_one",
+ "pandas_view:conv_rate_plus_acc",
+ "pandas_view:trips_until_next_rank_int",
+ "pandas_view:trips_until_next_rank_float",
+ "pandas_view:achieved_ranks_mask",
+ "pandas_view:achieved_ranks",
+ "pandas_view:highest_achieved_rank",
+ "pandas_view:is_highest_rank",
+ ],
+ ).to_df()
+ # We use to_df here to ensure we use the pandas backend, but convert to a dict for comparisons
+ result = online_response.to_dict(orient="records")[0]
+
+ # Type assertions
+ # Materialized view
+ assert type(result["conv_rate"]) == float
+ assert type(result["acc_rate"]) == float
+ assert type(result["avg_daily_trips"]) == int
+ # On-demand view
+ assert type(result["avg_daily_trips_plus_one"]) == int
+ assert type(result["conv_rate_plus_acc"]) == float
+ assert type(result["highest_achieved_rank"]) == str
+ assert type(result["is_highest_rank"]) == bool
+
+ assert type(result["trips_until_next_rank_int"]) == list
+ assert all([type(e) == int for e in result["trips_until_next_rank_int"]])
+
+ assert type(result["trips_until_next_rank_float"]) == list
+ assert all([type(e) == float for e in result["trips_until_next_rank_float"]])
+
+ assert type(result["achieved_ranks"]) == list
+ assert all([type(e) == str for e in result["achieved_ranks"]])
+
+ assert type(result["achieved_ranks_mask"]) == list
+ assert all([type(e) == bool for e in result["achieved_ranks_mask"]])
+
+ # Value assertions
+ expected_trips_until_next_rank = [
+ max(threshold - result["avg_daily_trips"], 0)
+ for threshold in entity_rows[0]["avg_daily_trip_rank_thresholds"]
+ ]
+ expected_mask = [value <= 0 for value in expected_trips_until_next_rank]
+ expected_ranks = [
+ rank if achieved else "Locked"
+ for achieved, rank in zip(
+ expected_mask, entity_rows[0]["avg_daily_trip_rank_names"]
+ )
+ ]
+ highest_rank = (
+ [rank for rank in expected_ranks if rank != "Locked"][-1:] or ["None"]
+ )[0]
+
+ assert result["conv_rate_plus_acc"] == result["conv_rate"] + result["acc_rate"]
+ assert result["avg_daily_trips_plus_one"] == result["avg_daily_trips"] + 1
+ assert result["highest_achieved_rank"] == highest_rank
+ assert result["is_highest_rank"] == (expected_ranks[-1] != "Locked")
+
+ assert result["trips_until_next_rank_int"] == expected_trips_until_next_rank
+ assert result["trips_until_next_rank_float"] == [
+ float(value) for value in expected_trips_until_next_rank
+ ]
+ assert result["achieved_ranks_mask"] == expected_mask
+ assert result["achieved_ranks"] == expected_ranks
+
+
+def test_invalid_pandas_transformation_raises_type_error_on_apply():
+ with tempfile.TemporaryDirectory() as data_dir:
+ store = FeatureStore(
+ config=RepoConfig(
+ project="test_on_demand_python_transformation",
+ registry=os.path.join(data_dir, "registry.db"),
+ provider="local",
+ entity_key_serialization_version=2,
+ online_store=SqliteOnlineStoreConfig(
+ path=os.path.join(data_dir, "online.db")
+ ),
+ )
+ )
+
+ request_source = RequestSource(
+ name="request_source",
+ schema=[
+ Field(name="driver_name", dtype=String),
+ ],
+ )
+
+ @on_demand_feature_view(
+ sources=[request_source],
+ schema=[Field(name="driver_name_lower", dtype=String)],
+ mode="pandas",
+ )
+ def pandas_view(inputs: pd.DataFrame) -> pd.DataFrame:
+ return pd.DataFrame({"driver_name_lower": []})
+
+ with pytest.raises(
+ TypeError,
+ match=re.escape(
+ "Failed to infer type for feature 'driver_name_lower' with value '[]' since no items were returned by the UDF."
+ ),
+ ):
+ store.apply([request_source, pandas_view])
diff --git a/sdk/python/tests/unit/test_on_demand_python_transformation.py b/sdk/python/tests/unit/test_on_demand_python_transformation.py
index 72e9b53a101..c5bd68d6a8f 100644
--- a/sdk/python/tests/unit/test_on_demand_python_transformation.py
+++ b/sdk/python/tests/unit/test_on_demand_python_transformation.py
@@ -1,4 +1,5 @@
import os
+import re
import tempfile
import unittest
from datetime import datetime, timedelta
@@ -7,12 +8,19 @@
import pandas as pd
import pytest
-from feast import Entity, FeatureStore, FeatureView, FileSource, RepoConfig
+from feast import (
+ Entity,
+ FeatureStore,
+ FeatureView,
+ FileSource,
+ RepoConfig,
+ RequestSource,
+)
from feast.driver_test_data import create_driver_hourly_stats_df
from feast.field import Field
from feast.infra.online_stores.sqlite import SqliteOnlineStoreConfig
from feast.on_demand_feature_view import on_demand_feature_view
-from feast.types import Float32, Float64, Int64
+from feast.types import Array, Bool, Float32, Float64, Int64, String
class TestOnDemandPythonTransformation(unittest.TestCase):
@@ -248,3 +256,234 @@ def test_python_docs_demo(self):
+ online_python_response["acc_rate"][0]
== online_python_response["conv_rate_plus_val2_python"][0]
)
+
+
+class TestOnDemandPythonTransformationAllDataTypes(unittest.TestCase):
+ def setUp(self):
+ with tempfile.TemporaryDirectory() as data_dir:
+ self.store = FeatureStore(
+ config=RepoConfig(
+ project="test_on_demand_python_transformation",
+ registry=os.path.join(data_dir, "registry.db"),
+ provider="local",
+ entity_key_serialization_version=2,
+ online_store=SqliteOnlineStoreConfig(
+ path=os.path.join(data_dir, "online.db")
+ ),
+ )
+ )
+
+ # Generate test data.
+ end_date = datetime.now().replace(microsecond=0, second=0, minute=0)
+ start_date = end_date - timedelta(days=15)
+
+ driver_entities = [1001, 1002, 1003, 1004, 1005]
+ driver_df = create_driver_hourly_stats_df(
+ driver_entities, start_date, end_date
+ )
+ driver_stats_path = os.path.join(data_dir, "driver_stats.parquet")
+ driver_df.to_parquet(
+ path=driver_stats_path, allow_truncated_timestamps=True
+ )
+
+ driver = Entity(name="driver", join_keys=["driver_id"])
+
+ driver_stats_source = FileSource(
+ name="driver_hourly_stats_source",
+ path=driver_stats_path,
+ timestamp_field="event_timestamp",
+ created_timestamp_column="created",
+ )
+
+ driver_stats_fv = FeatureView(
+ name="driver_hourly_stats",
+ entities=[driver],
+ ttl=timedelta(days=0),
+ schema=[
+ Field(name="conv_rate", dtype=Float32),
+ Field(name="acc_rate", dtype=Float32),
+ Field(name="avg_daily_trips", dtype=Int64),
+ ],
+ online=True,
+ source=driver_stats_source,
+ )
+
+ request_source = RequestSource(
+ name="request_source",
+ schema=[
+ Field(name="avg_daily_trip_rank_thresholds", dtype=Array(Int64)),
+ Field(name="avg_daily_trip_rank_names", dtype=Array(String)),
+ ],
+ )
+
+ @on_demand_feature_view(
+ sources=[request_source, driver_stats_fv],
+ schema=[
+ Field(name="highest_achieved_rank", dtype=String),
+ Field(name="avg_daily_trips_plus_one", dtype=Int64),
+ Field(name="conv_rate_plus_acc", dtype=Float64),
+ Field(name="is_highest_rank", dtype=Bool),
+ Field(name="achieved_ranks", dtype=Array(String)),
+ Field(name="trips_until_next_rank_int", dtype=Array(Int64)),
+ Field(name="trips_until_next_rank_float", dtype=Array(Float64)),
+ Field(name="achieved_ranks_mask", dtype=Array(Bool)),
+ ],
+ mode="python",
+ )
+ def python_view(inputs: dict[str, Any]) -> dict[str, Any]:
+ output = {}
+ trips_until_next_rank = [
+ [max(threshold - row[1], 0) for threshold in row[0]]
+ for row in zip(
+ inputs["avg_daily_trip_rank_thresholds"],
+ inputs["avg_daily_trips"],
+ )
+ ]
+ mask = [[value <= 0 for value in row] for row in trips_until_next_rank]
+ ranks = [
+ [rank if mask else "Locked" for mask, rank in zip(*row)]
+ for row in zip(mask, inputs["avg_daily_trip_rank_names"])
+ ]
+ highest_rank = [
+ ([rank for rank in row if rank != "Locked"][-1:] or ["None"])[0]
+ for row in ranks
+ ]
+
+ output["conv_rate_plus_acc"] = [
+ sum(row) for row in zip(inputs["conv_rate"], inputs["acc_rate"])
+ ]
+ output["avg_daily_trips_plus_one"] = [
+ row + 1 for row in inputs["avg_daily_trips"]
+ ]
+ output["highest_achieved_rank"] = highest_rank
+ output["is_highest_rank"] = [row[-1] != "Locked" for row in ranks]
+
+ output["trips_until_next_rank_int"] = trips_until_next_rank
+ output["trips_until_next_rank_float"] = [
+ [float(value) for value in row] for row in trips_until_next_rank
+ ]
+ output["achieved_ranks_mask"] = mask
+ output["achieved_ranks"] = ranks
+ return output
+
+ self.store.apply(
+ [driver, driver_stats_source, driver_stats_fv, python_view]
+ )
+ self.store.write_to_online_store(
+ feature_view_name="driver_hourly_stats", df=driver_df
+ )
+
+ def test_python_transformation_returning_all_data_types(self):
+ entity_rows = [
+ {
+ "driver_id": 1001,
+ "avg_daily_trip_rank_thresholds": [100, 250, 500, 1000],
+ "avg_daily_trip_rank_names": ["Bronze", "Silver", "Gold", "Platinum"],
+ }
+ ]
+ online_response = self.store.get_online_features(
+ entity_rows=entity_rows,
+ features=[
+ "driver_hourly_stats:conv_rate",
+ "driver_hourly_stats:acc_rate",
+ "driver_hourly_stats:avg_daily_trips",
+ "python_view:avg_daily_trips_plus_one",
+ "python_view:conv_rate_plus_acc",
+ "python_view:trips_until_next_rank_int",
+ "python_view:trips_until_next_rank_float",
+ "python_view:achieved_ranks_mask",
+ "python_view:achieved_ranks",
+ "python_view:highest_achieved_rank",
+ "python_view:is_highest_rank",
+ ],
+ ).to_dict()
+ result = {name: value[0] for name, value in online_response.items()}
+
+ # Type assertions
+ # Materialized view
+ assert type(result["conv_rate"]) == float
+ assert type(result["acc_rate"]) == float
+ assert type(result["avg_daily_trips"]) == int
+ # On-demand view
+ assert type(result["avg_daily_trips_plus_one"]) == int
+ assert type(result["conv_rate_plus_acc"]) == float
+ assert type(result["highest_achieved_rank"]) == str
+ assert type(result["is_highest_rank"]) == bool
+
+ assert type(result["trips_until_next_rank_int"]) == list
+ assert all([type(e) == int for e in result["trips_until_next_rank_int"]])
+
+ assert type(result["trips_until_next_rank_float"]) == list
+ assert all([type(e) == float for e in result["trips_until_next_rank_float"]])
+
+ assert type(result["achieved_ranks"]) == list
+ assert all([type(e) == str for e in result["achieved_ranks"]])
+
+ assert type(result["achieved_ranks_mask"]) == list
+ assert all([type(e) == bool for e in result["achieved_ranks_mask"]])
+
+ # Value assertions
+ expected_trips_until_next_rank = [
+ max(threshold - result["avg_daily_trips"], 0)
+ for threshold in entity_rows[0]["avg_daily_trip_rank_thresholds"]
+ ]
+ expected_mask = [value <= 0 for value in expected_trips_until_next_rank]
+ expected_ranks = [
+ rank if achieved else "Locked"
+ for achieved, rank in zip(
+ expected_mask, entity_rows[0]["avg_daily_trip_rank_names"]
+ )
+ ]
+ highest_rank = (
+ [rank for rank in expected_ranks if rank != "Locked"][-1:] or ["None"]
+ )[0]
+
+ assert result["conv_rate_plus_acc"] == result["conv_rate"] + result["acc_rate"]
+ assert result["avg_daily_trips_plus_one"] == result["avg_daily_trips"] + 1
+ assert result["highest_achieved_rank"] == highest_rank
+ assert result["is_highest_rank"] == (expected_ranks[-1] != "Locked")
+
+ assert result["trips_until_next_rank_int"] == expected_trips_until_next_rank
+ assert result["trips_until_next_rank_float"] == [
+ float(value) for value in expected_trips_until_next_rank
+ ]
+ assert result["achieved_ranks_mask"] == expected_mask
+ assert result["achieved_ranks"] == expected_ranks
+
+
+def test_invalid_python_transformation_raises_type_error_on_apply():
+ with tempfile.TemporaryDirectory() as data_dir:
+ store = FeatureStore(
+ config=RepoConfig(
+ project="test_on_demand_python_transformation",
+ registry=os.path.join(data_dir, "registry.db"),
+ provider="local",
+ entity_key_serialization_version=2,
+ online_store=SqliteOnlineStoreConfig(
+ path=os.path.join(data_dir, "online.db")
+ ),
+ )
+ )
+
+ request_source = RequestSource(
+ name="request_source",
+ schema=[
+ Field(name="driver_name", dtype=String),
+ ],
+ )
+
+ @on_demand_feature_view(
+ sources=[request_source],
+ schema=[Field(name="driver_name_lower", dtype=String)],
+ mode="python",
+ )
+ def python_view(inputs: dict[str, Any]) -> dict[str, Any]:
+ return {"driver_name_lower": []}
+
+ with pytest.raises(
+ TypeError,
+ match=re.escape(
+ "Failed to infer type for feature 'driver_name_lower' with value '[]' since no items were returned by the UDF."
+ ),
+ ):
+ store.apply([request_source, python_view])
diff --git a/sdk/python/tests/unit/test_stream_feature_view.py b/sdk/python/tests/unit/test_stream_feature_view.py
index b53f9a593ae..4f93691028e 100644
--- a/sdk/python/tests/unit/test_stream_feature_view.py
+++ b/sdk/python/tests/unit/test_stream_feature_view.py
@@ -15,6 +15,7 @@
)
from feast.stream_feature_view import StreamFeatureView, stream_feature_view
from feast.types import Float32
+from feast.utils import _utc_now, make_tzaware
def test_create_batch_feature_view():
@@ -250,3 +251,82 @@ def test_stream_feature_view_copy():
aggregations=[],
)
assert sfv == copy.copy(sfv)
+
+
+def test_update_materialization_intervals():
+ entity = Entity(name="driver_entity", join_keys=["test_key"])
+ stream_source = KafkaSource(
+ name="kafka",
+ timestamp_field="event_timestamp",
+ kafka_bootstrap_servers="",
+ message_format=AvroFormat(""),
+ topic="topic",
+ batch_source=FileSource(path="some path"),
+ )
+
+ # Create a stream feature view that is already present in the SQL registry
+ stored_stream_feature_view = StreamFeatureView(
+ name="test kafka stream feature view",
+ entities=[entity],
+ ttl=timedelta(days=30),
+ owner="test@example.com",
+ online=True,
+ schema=[Field(name="dummy_field", dtype=Float32)],
+ description="desc",
+ aggregations=[
+ Aggregation(
+ column="dummy_field",
+ function="max",
+ time_window=timedelta(days=1),
+ )
+ ],
+ timestamp_field="event_timestamp",
+ mode="spark",
+ source=stream_source,
+ udf=simple_udf,
+ tags={},
+ )
+ current_time = _utc_now()
+ start_date = make_tzaware(current_time - timedelta(days=1))
+ end_date = make_tzaware(current_time)
+ stored_stream_feature_view.materialization_intervals.append((start_date, end_date))
+
+ # Update the stream feature view i.e. here it's simply the name
+ updated_stream_feature_view = StreamFeatureView(
+ name="test kafka stream feature view updated",
+ entities=[entity],
+ ttl=timedelta(days=30),
+ owner="test@example.com",
+ online=True,
+ schema=[Field(name="dummy_field", dtype=Float32)],
+ description="desc",
+ aggregations=[
+ Aggregation(
+ column="dummy_field",
+ function="max",
+ time_window=timedelta(days=1),
+ )
+ ],
+ timestamp_field="event_timestamp",
+ mode="spark",
+ source=stream_source,
+ udf=simple_udf,
+ tags={},
+ )
+
+ updated_stream_feature_view.update_materialization_intervals(
+ stored_stream_feature_view.materialization_intervals
+ )
+
+ assert (
+ updated_stream_feature_view.materialization_intervals is not None
+ and len(stored_stream_feature_view.materialization_intervals) == 1
+ )
+ assert (
+ updated_stream_feature_view.materialization_intervals[0][0]
+ == stored_stream_feature_view.materialization_intervals[0][0]
+ )
+ assert (
+ updated_stream_feature_view.materialization_intervals[0][1]
+ == stored_stream_feature_view.materialization_intervals[0][1]
+ )
diff --git a/sdk/python/tests/utils/basic_read_write_test.py b/sdk/python/tests/utils/basic_read_write_test.py
index 5a93a05a1f5..c09a94083f3 100644
--- a/sdk/python/tests/utils/basic_read_write_test.py
+++ b/sdk/python/tests/utils/basic_read_write_test.py
@@ -1,9 +1,10 @@
-from datetime import datetime, timedelta
+from datetime import timedelta
from typing import Optional
from feast.feature_store import FeatureStore
from feast.protos.feast.types.EntityKey_pb2 import EntityKey as EntityKeyProto
from feast.protos.feast.types.Value_pb2 import Value as ValueProto
+from feast.utils import _utc_now
def basic_rw_test(
@@ -65,13 +66,13 @@ def _driver_rw_test(event_ts, created_ts, write, expect_read):
""" 1. Basic test: write value, read it back """
- time_1 = datetime.utcnow()
+ time_1 = _utc_now()
_driver_rw_test(
event_ts=time_1, created_ts=time_1, write=(1.1, "3.1"), expect_read=(1.1, "3.1")
)
""" Values with an new event_ts should overwrite older ones """
- time_3 = datetime.utcnow()
+ time_3 = _utc_now()
_driver_rw_test(
event_ts=time_1 + timedelta(hours=1),
created_ts=time_3,
diff --git a/sdk/python/tests/utils/dynamo_table_creator.py b/sdk/python/tests/utils/dynamo_table_creator.py
index 20bac122b37..0ebc939dc11 100644
--- a/sdk/python/tests/utils/dynamo_table_creator.py
+++ b/sdk/python/tests/utils/dynamo_table_creator.py
@@ -1,11 +1,10 @@
-from datetime import datetime
-
import boto3
from feast import utils
from feast.infra.online_stores.helpers import compute_entity_id
from feast.protos.feast.types.EntityKey_pb2 import EntityKey as EntityKeyProto
from feast.protos.feast.types.Value_pb2 import Value as ValueProto
+from feast.utils import _utc_now
def create_n_customer_test_samples(n=10):
@@ -19,7 +18,7 @@ def create_n_customer_test_samples(n=10):
"name": ValueProto(string_val="John"),
"age": ValueProto(int64_val=3),
},
- datetime.utcnow(),
+ _utc_now(),
None,
)
for i in range(n)
diff --git a/sdk/python/tests/utils/e2e_test_validation.py b/sdk/python/tests/utils/e2e_test_validation.py
index 885798db109..1a8bedc7968 100644
--- a/sdk/python/tests/utils/e2e_test_validation.py
+++ b/sdk/python/tests/utils/e2e_test_validation.py
@@ -10,6 +10,7 @@
from pytz import utc
from feast import FeatureStore, FeatureView, RepoConfig
+from feast.utils import _utc_now
from tests.integration.feature_repos.integration_test_repo_config import (
IntegrationTestRepoConfig,
)
@@ -31,7 +32,7 @@
def validate_offline_online_store_consistency(
fs: FeatureStore, fv: FeatureView, split_dt: datetime
) -> None:
- now = datetime.utcnow()
+ now = _utc_now()
full_feature_names = True
check_offline_store: bool = True
@@ -78,6 +79,16 @@ def validate_offline_online_store_consistency(
# run materialize_incremental()
fs.materialize_incremental(feature_views=[fv.name], end_date=now)
+ updated_fv = fs.registry.get_feature_view(fv.name, fs.project)
+
+ # Check if materialization_intervals was updated by the registry
+ assert (
+ len(updated_fv.materialization_intervals) == 2
+ and updated_fv.materialization_intervals[0][0] == start_date
+ and updated_fv.materialization_intervals[0][1] == end_date
+ and updated_fv.materialization_intervals[1][0] == end_date
+ and updated_fv.materialization_intervals[1][1] == now.replace(tzinfo=utc)
+ )
# check result of materialize_incremental()
_check_offline_and_online_features(
diff --git a/sdk/python/tests/utils/feature_records.py b/sdk/python/tests/utils/feature_records.py
index 2c26f3c0000..bd3567c9eeb 100644
--- a/sdk/python/tests/utils/feature_records.py
+++ b/sdk/python/tests/utils/feature_records.py
@@ -1,4 +1,4 @@
-from datetime import datetime, timedelta
+from datetime import datetime, timedelta, timezone
from typing import Any, Dict, List, Optional
import numpy as np
@@ -520,7 +520,7 @@ def get_last_feature_row(df: pd.DataFrame, driver_id, max_date: datetime):
"""Manually extract last feature value from a dataframe for a given driver_id with up to `max_date` date"""
filtered = df[
(df["driver_id"] == driver_id)
- & (df["event_timestamp"] < max_date.replace(tzinfo=utc))
+ & (df["event_timestamp"] < max_date.replace(tzinfo=timezone.utc))
]
max_ts = filtered.loc[filtered["event_timestamp"].idxmax()]["event_timestamp"]
filtered_by_ts = filtered[filtered["event_timestamp"] == max_ts]
diff --git a/sdk/python/tests/utils/online_write_benchmark.py b/sdk/python/tests/utils/online_write_benchmark.py
index 8a138f41dbc..9b1a4eb0b27 100644
--- a/sdk/python/tests/utils/online_write_benchmark.py
+++ b/sdk/python/tests/utils/online_write_benchmark.py
@@ -2,7 +2,7 @@
import random
import string
import tempfile
-from datetime import datetime, timedelta
+from datetime import timedelta
import click
import pyarrow as pa
@@ -16,7 +16,7 @@
from feast.field import Field
from feast.repo_config import RepoConfig
from feast.types import Float32, Int32
-from feast.utils import _convert_arrow_to_proto
+from feast.utils import _convert_arrow_to_proto, _utc_now
def create_driver_hourly_stats_feature_view(source):
@@ -69,7 +69,7 @@ def benchmark_writes():
provider = store._get_provider()
- end_date = datetime.utcnow()
+ end_date = _utc_now()
start_date = end_date - timedelta(days=14)
customers = list(range(100))
data = create_driver_hourly_stats_df(customers, start_date, end_date)
diff --git a/sdk/python/tests/utils/test_log_creator.py b/sdk/python/tests/utils/test_log_creator.py
index ec0d92814cb..987c8d77ef7 100644
--- a/sdk/python/tests/utils/test_log_creator.py
+++ b/sdk/python/tests/utils/test_log_creator.py
@@ -8,12 +8,12 @@
import numpy as np
import pandas as pd
import pyarrow
-import pytz
from feast import FeatureService, FeatureStore, FeatureView
from feast.errors import FeatureViewNotFoundException
from feast.feature_logging import LOG_DATE_FIELD, LOG_TIMESTAMP_FIELD, REQUEST_ID_FIELD
from feast.protos.feast.serving.ServingService_pb2 import FieldStatus
+from feast.utils import _utc_now
def get_latest_rows(
@@ -64,9 +64,7 @@ def generate_expected_logs(
logs[f"{col}__status"] = FieldStatus.PRESENT
if feature_view.ttl:
logs[f"{col}__status"] = logs[f"{col}__status"].mask(
- df[timestamp_column]
- < datetime.datetime.utcnow().replace(tzinfo=pytz.UTC)
- - feature_view.ttl,
+ df[timestamp_column] < _utc_now() - feature_view.ttl,
FieldStatus.OUTSIDE_MAX_AGE,
)
@@ -119,7 +117,7 @@ def prepare_logs(
f"{destination_field}__status"
].mask(
logs_df[f"{destination_field}__timestamp"]
- < (datetime.datetime.utcnow() - view.ttl),
+ < (_utc_now() - view.ttl).replace(tzinfo=None),
FieldStatus.OUTSIDE_MAX_AGE,
)
diff --git a/setup.py b/setup.py
index 9b3d0e55e62..b983617712c 100644
--- a/setup.py
+++ b/setup.py
@@ -18,21 +18,14 @@
import shutil
import subprocess
import sys
-from distutils.cmd import Command
-from pathlib import Path
-
-from setuptools import find_packages
-try:
- from setuptools import setup
- from setuptools.command.build_ext import build_ext as _build_ext
- from setuptools.command.build_py import build_py
- from setuptools.command.develop import develop
- from setuptools.command.install import install
+from pathlib import Path
-except ImportError:
- from distutils.command.build_py import build_py
- from distutils.core import setup
+from setuptools import find_packages, setup, Command
+from setuptools.command.build_ext import build_ext as _build_ext
+from setuptools.command.build_py import build_py
+from setuptools.command.develop import develop
+from setuptools.command.install import install
NAME = "feast"
DESCRIPTION = "Python SDK for Feast"
@@ -65,15 +58,15 @@
"fastapi>=0.68.0",
"uvicorn[standard]>=0.14.0,<1",
"gunicorn; platform_system != 'Windows'",
- "dask[dataframe]>=2024.4.2",
+ "dask[dataframe]>=2024.2.1",
]
GCP_REQUIRED = [
"google-api-core>=1.23.0,<3",
"googleapis-common-protos>=1.52.0,<2",
- "google-cloud-bigquery[pandas]>=2,<3.13.0",
+ "google-cloud-bigquery[pandas]>=2,<4",
"google-cloud-bigquery-storage >= 2.0.0,<3",
- "google-cloud-datastore>=2.1.0,<3",
+ "google-cloud-datastore>=2.16.0,<3",
"google-cloud-storage>=1.34.0,<3",
"google-cloud-bigtable>=2.11.0,<3",
"fsspec<=2024.1.0",
@@ -84,7 +77,7 @@
"hiredis>=2.0.0,<3",
]
-AWS_REQUIRED = ["boto3>=1.17.0,<2", "docker>=5.0.2", "fsspec<=2024.1.0", "aiobotocore>2,<3"]
+AWS_REQUIRED = ["boto3>=1.17.0,<2", "fsspec<=2024.1.0", "aiobotocore>2,<3"]
KUBERNETES_REQUIRED = ["kubernetes<=20.13.0"]
@@ -102,7 +95,7 @@
TRINO_REQUIRED = ["trino>=0.305.0,<0.400.0", "regex"]
POSTGRES_REQUIRED = [
- "psycopg2-binary>=2.8.3,<3",
+ "psycopg[binary,pool]>=3.0.0,<4",
]
MYSQL_REQUIRED = ["pymysql", "types-PyMySQL"]
@@ -138,8 +131,8 @@
]
IBIS_REQUIRED = [
- "ibis-framework>=8.0.0,<9",
- "ibis-substrait<=3.2.0",
+ "ibis-framework>=9.0.0,<10",
+ "ibis-substrait>=4.0.0",
]
GRPCIO_REQUIRED = [
@@ -149,12 +142,16 @@
"grpcio-health-checking>=1.56.2,<2",
]
-DUCKDB_REQUIRED = ["ibis-framework[duckdb]>=8.0.0,<9"]
+DUCKDB_REQUIRED = ["ibis-framework[duckdb]>=9.0.0,<10"]
DELTA_REQUIRED = ["deltalake"]
ELASTICSEARCH_REQUIRED = ["elasticsearch>=8.13.0"]
+SINGLESTORE_REQUIRED = ["singlestoredb"]
+
+MSSQL_REQUIRED = ["ibis-framework[mssql]>=9.0.0,<10"]
+
CI_REQUIRED = (
[
"build",
@@ -182,7 +179,6 @@
"pytest-env",
"Sphinx>4.0.0,<7",
"testcontainers==4.4.0",
- "firebase-admin>=5.2.0,<6",
"pre-commit<3.3.2",
"assertpy==1.1",
"pip-tools",
@@ -218,6 +214,7 @@
+ DELTA_REQUIRED
+ ELASTICSEARCH_REQUIRED
+ SQLITE_VEC_REQUIRED
+ + SINGLESTORE_REQUIRED
)
DOCS_REQUIRED = CI_REQUIRED
@@ -373,6 +370,7 @@ def run(self):
"postgres": POSTGRES_REQUIRED,
"azure": AZURE_REQUIRED,
"mysql": MYSQL_REQUIRED,
+ "mssql": MSSQL_REQUIRED,
"ge": GE_REQUIRED,
"hbase": HBASE_REQUIRED,
"docs": DOCS_REQUIRED,
@@ -386,6 +384,7 @@ def run(self):
"delta": DELTA_REQUIRED,
"elasticsearch": ELASTICSEARCH_REQUIRED,
"sqlite_vec": SQLITE_VEC_REQUIRED,
+ "singlestore": SINGLESTORE_REQUIRED,
},
include_package_data=True,
license="Apache",
diff --git a/ui/feature_repo/features.py b/ui/feature_repo/features.py
index e02bb3de5d0..40a42a9e99e 100644
--- a/ui/feature_repo/features.py
+++ b/ui/feature_repo/features.py
@@ -11,7 +11,7 @@
name="zipcode",
description="A zipcode",
tags={
- "owner": "danny@tecton.ai",
+ "owner": "danny@feast.ai",
"team": "hack week",
},
)
@@ -40,7 +40,7 @@
tags={
"date_added": "2022-02-7",
"experiments": "experiment-A,experiment-B,experiment-C",
- "access_group": "feast-team@tecton.ai",
+ "access_group": "feast-team@feast.ai",
},
online=True,
)
@@ -62,7 +62,7 @@
tags={
"date_added": "2022-02-7",
"experiments": "experiment-A,experiment-B,experiment-C",
- "access_group": "feast-team@tecton.ai",
+ "access_group": "feast-team@feast.ai",
},
online=True,
)
@@ -80,7 +80,7 @@
tags={
"date_added": "2022-02-7",
"experiments": "experiment-A,experiment-B,experiment-C",
- "access_group": "feast-team@tecton.ai",
+ "access_group": "feast-team@feast.ai",
},
online=True,
)
@@ -89,7 +89,7 @@
name="dob_ssn",
description="Date of birth and last four digits of social security number",
tags={
- "owner": "tony@tecton.ai",
+ "owner": "tony@feast.ai",
"team": "hack week",
},
)
@@ -121,7 +121,7 @@
tags={
"date_added": "2022-02-6",
"experiments": "experiment-A",
- "access_group": "feast-team@tecton.ai",
+ "access_group": "feast-team@feast.ai",
},
online=True,
)
@@ -157,7 +157,7 @@ def transaction_gt_last_credit_card_due(inputs: pd.DataFrame) -> pd.DataFrame:
credit_history[["credit_card_due", "missed_payments_1y"]],
zipcode_features,
],
- tags={"owner": "tony@tecton.ai", "stage": "staging"},
+ tags={"owner": "tony@feast.ai", "stage": "staging"},
description="Credit scoring model",
)
@@ -167,7 +167,7 @@ def transaction_gt_last_credit_card_due(inputs: pd.DataFrame) -> pd.DataFrame:
credit_history[["mortgage_due", "credit_card_due", "missed_payments_1y"]],
zipcode_features,
],
- tags={"owner": "tony@tecton.ai", "stage": "prod"},
+ tags={"owner": "tony@feast.ai", "stage": "prod"},
description="Credit scoring model",
)
@@ -178,7 +178,7 @@ def transaction_gt_last_credit_card_due(inputs: pd.DataFrame) -> pd.DataFrame:
zipcode_features,
transaction_gt_last_credit_card_due,
],
- tags={"owner": "tony@tecton.ai", "stage": "dev"},
+ tags={"owner": "tony@feast.ai", "stage": "dev"},
description="Credit scoring model",
)
@@ -187,7 +187,7 @@ def transaction_gt_last_credit_card_due(inputs: pd.DataFrame) -> pd.DataFrame:
features=[
zipcode_features,
],
- tags={"owner": "amanda@tecton.ai", "stage": "dev"},
+ tags={"owner": "amanda@feast.ai", "stage": "dev"},
description="Location model",
)
@@ -196,6 +196,6 @@ def transaction_gt_last_credit_card_due(inputs: pd.DataFrame) -> pd.DataFrame:
features=[
zipcode_money_features,
],
- tags={"owner": "amanda@tecton.ai", "stage": "dev"},
+ tags={"owner": "amanda@feast.ai", "stage": "dev"},
description="Location model",
)
diff --git a/ui/package.json b/ui/package.json
index de37f4394a0..cd80859aa15 100644
--- a/ui/package.json
+++ b/ui/package.json
@@ -1,6 +1,6 @@
{
"name": "@feast-dev/feast-ui",
- "version": "0.39.0",
+ "version": "0.40.0",
"private": false,
"files": [
"dist"
@@ -118,7 +118,7 @@
"Feature",
"Store"
],
- "author": "tony@tecton.ai",
+ "author": "tony@feast.ai",
"license": "Apache-2.0",
"bugs": {
"url": "https://github.com/feast-dev/feast/issues"
diff --git a/ui/yarn.lock b/ui/yarn.lock
index 89107de0b89..26c833fa11f 100644
--- a/ui/yarn.lock
+++ b/ui/yarn.lock
@@ -11640,14 +11640,14 @@ write-file-atomic@^3.0.0:
typedarray-to-buffer "^3.1.5"
ws@^7.4.6:
- version "7.5.6"
- resolved "https://registry.yarnpkg.com/ws/-/ws-7.5.6.tgz#e59fc509fb15ddfb65487ee9765c5a51dec5fe7b"
- integrity sha512-6GLgCqo2cy2A2rjCNFlxQS6ZljG/coZfZXclldI8FB/1G3CCI36Zd8xy2HrFVACi8tfk5XrgLQEk+P0Tnz9UcA==
+ version "7.5.10"
+ resolved "https://registry.yarnpkg.com/ws/-/ws-7.5.10.tgz#58b5c20dc281633f6c19113f39b349bd8bd558d9"
+ integrity sha512-+dbF1tHwZpXcbOJdVOkzLDxZP1ailvSxM6ZweXTegylPny803bFhA+vqBYw4s31NSAk4S2Qz+AKXK9a4wkdjcQ==
ws@^8.1.0:
- version "8.4.2"
- resolved "https://registry.yarnpkg.com/ws/-/ws-8.4.2.tgz#18e749868d8439f2268368829042894b6907aa0b"
- integrity sha512-Kbk4Nxyq7/ZWqr/tarI9yIt/+iNNFOjBXEWgTb4ydaNHBNGgvf2QHbS9fdfsndfjFlFwEd4Al+mw83YkaD10ZA==
+ version "8.17.1"
+ resolved "https://registry.yarnpkg.com/ws/-/ws-8.17.1.tgz#9293da530bb548febc95371d90f9c878727d919b"
+ integrity sha512-6XQFvXTkbfUOZOKKILFG1PDK2NDQs4azKQl26T0YS5CxqWLgXajbPZ+h4gZekJyRqFU8pvnbAbbs/3TgRPy+GQ==
xml-name-validator@^3.0.0:
version "3.0.0"