You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/reference/data-sources/overview.md
+73Lines changed: 73 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -10,6 +10,79 @@ However, not every batch data source supports all of these types.
10
10
11
11
For more details on the Feast type system, see [here](../type-system.md).
12
12
13
+
## Per-DataSource Credentials (ConnectionRef)
14
+
15
+
By default, every data source inherits connection credentials from the global `feature_store.yaml` offline store configuration. The `ConnectionRef` feature allows each data source to declare its own external credential reference, enabling:
16
+
17
+
-**Multi-tenant deployments** where different feature views access different accounts or databases.
18
+
-**Credential isolation** where secrets are resolved at runtime from external providers (Kubernetes Secrets, HashiCorp Vault, environment variables) rather than stored in configuration files.
19
+
-**Hybrid offline store** routing where a single Feast deployment connects to multiple backends, each with independent credentials.
20
+
21
+
### ConnectionRef structure
22
+
23
+
A `ConnectionRef` is attached to any data source via the `connection_ref` parameter:
24
+
25
+
```python
26
+
from feast.credentials import ConnectionRef
27
+
from feast.infra.offline_stores.snowflake_source import SnowflakeSource
|`kubernetes`| Kubernetes Secrets | Secret name | K8s namespace |
59
+
|`vault`| HashiCorp Vault | Secret path | Vault mount |
60
+
61
+
Custom providers can be registered via:
62
+
63
+
```python
64
+
from feast.credentials import register_credential_provider, CredentialProvider, ConnectionRef
65
+
66
+
classMyProvider(CredentialProvider):
67
+
defprovider_type(self) -> str:
68
+
return"my-provider"
69
+
70
+
defresolve(self, ref: ConnectionRef) -> dict:
71
+
# Return key-value credential pairs
72
+
return {"username": "...", "password": "..."}
73
+
74
+
register_credential_provider(MyProvider())
75
+
```
76
+
77
+
### How it works
78
+
79
+
1. When an offline store needs to connect, it checks whether the data source has a `connection_ref`.
80
+
2. If present, credentials are resolved from the external provider at runtime.
81
+
3. Resolved credentials (and any `params` from the `ConnectionRef`) are merged and used to override the global offline store configuration for that specific operation.
82
+
4. If no `connection_ref` is set, the data source uses the global `feature_store.yaml` configuration as before.
83
+
84
+
For usage with the Hybrid Offline Store, see [Hybrid Offline Store](../offline-stores/hybrid.md).
85
+
13
86
## Functionality Matrix
14
87
15
88
There are currently four core batch data source implementations: `FileSource`, `BigQuerySource`, `SnowflakeSource`, and `RedshiftSource`.
Copy file name to clipboardExpand all lines: docs/reference/offline-stores/hybrid.md
+78Lines changed: 78 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -82,6 +82,84 @@ store.materialize(
82
82
)
83
83
```
84
84
85
+
## Using ConnectionRef with Hybrid Offline Store
86
+
87
+
When using the HybridOfflineStore, each data source can carry its own credentials via `ConnectionRef`. This is particularly useful when different feature views connect to different accounts or clusters — you no longer need to embed all credentials in `feature_store.yaml`.
88
+
89
+
### Example: Per-DataSource Credentials
90
+
91
+
{% code title="feature_store.yaml" %}
92
+
```yaml
93
+
project: my_feature_repo
94
+
registry: data/registry.db
95
+
provider: local
96
+
offline_store:
97
+
type: hybrid_offline_store.HybridOfflineStore
98
+
offline_stores:
99
+
- type: snowflake.offline
100
+
- type: bigquery
101
+
```
102
+
{% endcode %}
103
+
104
+
```python
105
+
from feast import FeatureView, Entity, ValueType
106
+
from feast.credentials import ConnectionRef
107
+
from feast.infra.offline_stores.snowflake_source import SnowflakeSource
108
+
from feast.infra.offline_stores.bigquery_source import BigQuerySource
# BigQuery source with credentials from a Kubernetes Secret
130
+
feature_view2 = FeatureView(
131
+
name="user_activity",
132
+
entities=["user_id"],
133
+
ttl=None,
134
+
source=BigQuerySource(
135
+
table="my_project.dataset.user_activity",
136
+
connection_ref=ConnectionRef(
137
+
provider="kubernetes",
138
+
name="bigquery-team-b-creds",
139
+
namespace="ml-team",
140
+
connection_type="bigquery",
141
+
),
142
+
),
143
+
)
144
+
```
145
+
146
+
In this setup:
147
+
- No sensitive credentials are stored in `feature_store.yaml`.
148
+
- Each data source resolves its credentials independently at runtime from the referenced Kubernetes Secret.
149
+
- The HybridOfflineStore routes operations to the correct backend based on the source type.
150
+
151
+
### How credential resolution works
152
+
153
+
1. The HybridOfflineStore determines which backend to use based on the data source class (e.g., `SnowflakeSource` → Snowflake offline store).
154
+
2. Before connecting, the offline store checks if the data source has a `connection_ref`.
155
+
3. If present, credentials are fetched from the external provider (e.g., reading a Kubernetes Secret).
156
+
4. Resolved credentials override the global offline store config for that operation.
157
+
5. If no `connection_ref` is set, the global `feature_store.yaml` configuration is used as a fallback.
158
+
159
+
This pattern is especially valuable in multi-tenant environments where a shared Feast deployment serves multiple teams, each with isolated credentials and backend accounts.
160
+
161
+
For details on the `ConnectionRef` structure and supported providers, see [Data Sources Overview](../data-sources/overview.md#per-datasource-credentials-connectionref).
0 commit comments