Skip to content

Commit 8f6e8df

Browse files
author
Tsotne Tabidze
authored
Document permissions for AWS (DynamoDB & Redshift) (#1753)
* Document permissions for AWS (DynamoDB & Redshift) Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai> * Add dynamodb & redshift pages in parent pages Signed-off-by: Tsotne Tabidze <tsotne@tecton.ai>
1 parent 082bbff commit 8f6e8df

File tree

8 files changed

+236
-97
lines changed

8 files changed

+236
-97
lines changed

‎docs/getting-started/create-a-feature-repository.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -24,13 +24,13 @@ Creating a new Feast repository in /<...>/tiny_pika.
2424
{% tab title="AWS template" %}
2525
```text
2626
feast init -t aws
27-
[?] AWS Region (e.g. us-west-2): ...
28-
[?] Redshift Cluster ID: ...
29-
[?] Redshift Database Name: ...
30-
[?] Redshift User Name: ...
31-
[?] Redshift S3 Staging Location (s3://*): ...
32-
[?] Redshift IAM Role for S3 (arn:aws:iam::*:role/*): ...
33-
[?] Should I upload example data to Redshift (overwriting 'feast_driver_hourly_stats' table)? (Y/n):
27+
AWS Region (e.g. us-west-2): ...
28+
Redshift Cluster ID: ...
29+
Redshift Database Name: ...
30+
Redshift User Name: ...
31+
Redshift S3 Staging Location (s3://*): ...
32+
Redshift IAM Role for S3 (arn:aws:iam::*:role/*): ...
33+
Should I upload example data to Redshift (overwriting 'feast_driver_hourly_stats' table)? (Y/n):
3434
3535
Creating a new Feast repository in /<...>/tiny_pika.
3636
```

‎docs/reference/data-sources/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,4 @@ Please see [Data Source](../../concepts/feature-view.md#data-source) for an expl
66

77
{% page-ref page="bigquery.md" %}
88

9+
{% page-ref page="redshift.md" %}

‎docs/reference/offline-stores/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,4 @@ Please see [Offline Store](../../concepts/offline-store.md) for an explanation o
66

77
{% page-ref page="bigquery.md" %}
88

9+
{% page-ref page="redshift.md" %}

‎docs/reference/offline-stores/redshift.md‎

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,3 +29,166 @@ offline_store:
2929
3030
Configuration options are available [here](https://github.com/feast-dev/feast/blob/bf557bcb72c7878a16dccb48443bbbe9dc3efa49/sdk/python/feast/infra/offline_stores/redshift.py#L22).
3131
32+
### Permissions
33+
34+
Feast requires the following permissions in order to execute commands for Redshift offline store:
35+
36+
<table>
37+
<thead>
38+
<tr>
39+
<th style="text-align:left"><b>Command</b></th>
40+
<th style="text-align:left">Permissions</th>
41+
<th style="text-align:left">Resources</th>
42+
</tr>
43+
</thead>
44+
<tbody>
45+
<tr>
46+
<td style="text-align:left"><b>Apply</b></td>
47+
<td style="text-align:left">
48+
<p>redshift-data:DescribeTable</p>
49+
<p>redshift:GetClusterCredentials</p>
50+
</td>
51+
<td style="text-align:left">
52+
<p>arn:aws:redshift:&lt;region&gt;:&lt;account_id&gt;:dbuser:&lt;redshift_cluster_id&gt;/&lt;redshift_username&gt;</p>
53+
<p>arn:aws:redshift:&lt;region&gt;:&lt;account_id&gt;:dbname:&lt;redshift_cluster_id&gt;/&lt;redshift_database_name&gt;</p>
54+
<p>arn:aws:redshift:&lt;region&gt;:&lt;account_id&gt;:cluster:&lt;redshift_cluster_id&gt;</p>
55+
</td>
56+
</tr>
57+
<tr>
58+
<td style="text-align:left" rowspan="3"><b>Materialize</b></td>
59+
<td style="text-align:left">
60+
<p>redshift-data:ExecuteStatement</p>
61+
</td>
62+
<td style="text-align:left">
63+
<p>arn:aws:redshift:&lt;region&gt;:&lt;account_id&gt;:cluster:&lt;redshift_cluster_id&gt;</p>
64+
</td>
65+
</tr>
66+
<tr>
67+
<td style="text-align:left">
68+
<p>redshift-data:DescribeStatement</p>
69+
</td>
70+
<td style="text-align:left">*</td>
71+
</tr>
72+
<tr>
73+
<td style="text-align:left">
74+
<p>s3:ListBucket</p>
75+
<p>s3:GetObject</p>
76+
<p>s3:DeleteObject</p>
77+
</td>
78+
<td style="text-align:left">
79+
<p>arn:aws:s3:::&lt;bucket_name&gt;</p>
80+
<p>arn:aws:s3:::&lt;bucket_name&gt;/*</p>
81+
</td>
82+
</tr>
83+
<tr>
84+
<td style="text-align:left" rowspan="3"><b>Get Historical Features</b></td>
85+
<td style="text-align:left">
86+
<p>redshift-data:ExecuteStatement</p>
87+
<p>redshift:GetClusterCredentials</p>
88+
</td>
89+
<td style="text-align:left">
90+
<p>arn:aws:redshift:&lt;region&gt;:&lt;account_id&gt;:dbuser:&lt;redshift_cluster_id&gt;/&lt;redshift_username&gt;</p>
91+
<p>arn:aws:redshift:&lt;region&gt;:&lt;account_id&gt;:dbname:&lt;redshift_cluster_id&gt;/&lt;redshift_database_name&gt;</p>
92+
<p>arn:aws:redshift:&lt;region&gt;:&lt;account_id&gt;:cluster:&lt;redshift_cluster_id&gt;</p>
93+
</td>
94+
</tr>
95+
<tr>
96+
<td style="text-align:left">
97+
<p>redshift-data:DescribeStatement</p>
98+
</td>
99+
<td style="text-align:left">*</td>
100+
</tr>
101+
<tr>
102+
<td style="text-align:left">
103+
<p>s3:ListBucket</p>
104+
<p>s3:GetObject</p>
105+
<p>s3:PutObject</p>
106+
<p>s3:DeleteObject</p>
107+
</td>
108+
<td style="text-align:left">
109+
<p>arn:aws:s3:::&lt;bucket_name&gt;</p>
110+
<p>arn:aws:s3:::&lt;bucket_name&gt;/*</p>
111+
</td>
112+
</tr>
113+
</tbody>
114+
</table>
115+
116+
The following inline policy can be used to grant Feast the necessary permissions:
117+
118+
```json
119+
{
120+
"Statement": [
121+
{
122+
"Action": [
123+
"s3:ListBucket",
124+
"s3:PutObject",
125+
"s3:GetObject",
126+
"s3:DeleteObject"
127+
],
128+
"Effect": "Allow",
129+
"Resource": [
130+
"arn:aws:s3:::<bucket_name>/*",
131+
"arn:aws:s3:::<bucket_name>"
132+
]
133+
},
134+
{
135+
"Action": [
136+
"redshift-data:DescribeTable",
137+
"redshift:GetClusterCredentials",
138+
"redshift-data:ExecuteStatement"
139+
],
140+
"Effect": "Allow",
141+
"Resource": [
142+
"arn:aws:redshift:<region>:<account_id>:dbuser:<redshift_cluster_id>/<redshift_username>",
143+
"arn:aws:redshift:<region>:<account_id>:dbname:<redshift_cluster_id>/<redshift_database_name>",
144+
"arn:aws:redshift:<region>:<account_id>:cluster:<redshift_cluster_id>"
145+
]
146+
},
147+
{
148+
"Action": [
149+
"redshift-data:DescribeStatement"
150+
],
151+
"Effect": "Allow",
152+
"Resource": "*"
153+
}
154+
],
155+
"Version": "2012-10-17"
156+
}
157+
```
158+
159+
In addition to this, Redshift offline store requires an IAM role that will be used by Redshift itself to interact with S3. More concretely, Redshift has to use this IAM role to run <a href="https://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html">UNLOAD</a> and <a href="https://docs.aws.amazon.com/redshift/latest/dg/r_COPY.html">COPY</a> commands. Once created, this IAM role needs to be configured in `feature_store.yaml` file as `offline_store: iam_role`.
160+
161+
The following inline policy can be used to grant Redshift necessary permissions to access S3:
162+
163+
```json
164+
{
165+
"Statement": [
166+
{
167+
"Action": "s3:*",
168+
"Effect": "Allow",
169+
"Resource": [
170+
"arn:aws:s3:::feast-integration-tests",
171+
"arn:aws:s3:::feast-integration-tests/*"
172+
]
173+
}
174+
],
175+
"Version": "2012-10-17"
176+
}
177+
```
178+
179+
While the following trust relationship is necessary to make sure that Redshift, and only Redshift can assume this role:
180+
181+
```json
182+
{
183+
"Version": "2012-10-17",
184+
"Statement": [
185+
{
186+
"Effect": "Allow",
187+
"Principal": {
188+
"Service": "redshift.amazonaws.com"
189+
},
190+
"Action": "sts:AssumeRole"
191+
}
192+
]
193+
}
194+
```

‎docs/reference/online-stores/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,3 +8,4 @@ Please see [Online Store](../../concepts/online-store.md) for an explanation of
88

99
{% page-ref page="datastore.md" %}
1010

11+
{% page-ref page="dynamodb.md" %}

‎docs/reference/online-stores/dynamodb.md‎

Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,65 @@ online_store:
2525
{% endcode %}
2626
2727
Configuration options are available [here](https://github.com/feast-dev/feast/blob/17bfa6118d6658d2bff53d7de8e2ccef5681714d/sdk/python/feast/infra/online_stores/dynamodb.py#L36).
28+
29+
### Permissions
30+
31+
Feast requires the following permissions in order to execute commands for DynamoDB online store:
32+
33+
<table>
34+
<thead>
35+
<tr>
36+
<th style="text-align:left"><b>Command</b></th>
37+
<th style="text-align:left">Permissions</th>
38+
<th style="text-align:left">Resources</th>
39+
</tr>
40+
</thead>
41+
<tbody>
42+
<tr>
43+
<td style="text-align:left"><b>Apply</b></td>
44+
<td style="text-align:left">
45+
<p>dynamodb:CreateTable</p>
46+
<p>dynamodb:DescribeTable</p>
47+
<p>dynamodb:DeleteTable</p>
48+
</td>
49+
<td style="text-align:left">arn:aws:dynamodb:&lt;region&gt;:&lt;account_id&gt;:table/*</td>
50+
</tr>
51+
<tr>
52+
<td style="text-align:left"><b>Materialize</b></td>
53+
<td style="text-align:left">
54+
<p>dynamodb.BatchWriteItem</p>
55+
</td>
56+
<td style="text-align:left">arn:aws:dynamodb:&lt;region&gt;:&lt;account_id&gt;:table/*</td>
57+
</tr>
58+
<tr>
59+
<td style="text-align:left"><b>Get Online Features</b></td>
60+
<td style="text-align:left">dynamodb.GetItem</td>
61+
<td style="text-align:left">arn:aws:dynamodb:&lt;region&gt;:&lt;account_id&gt;:table/*</td>
62+
</tr>
63+
</tbody>
64+
</table>
65+
66+
The following inline policy can be used to grant Feast the necessary permissions:
67+
68+
```json
69+
{
70+
"Statement": [
71+
{
72+
"Action": [
73+
"dynamodb:CreateTable",
74+
"dynamodb:DescribeTable",
75+
"dynamodb:DeleteTable",
76+
"dynamodb:BatchWriteItem",
77+
"dynamodb:GetItem"
78+
],
79+
"Effect": "Allow",
80+
"Resource": [
81+
"arn:aws:dynamodb:<region>:<account_id>:table/*"
82+
]
83+
}
84+
],
85+
"Version": "2012-10-17"
86+
}
87+
```
88+
89+
Lastly, this IAM role needs to be associated with the desired Redshift cluster. Please follow the official AWS guide for the necessary steps [here](https://docs.aws.amazon.com/redshift/latest/dg/c-getting-started-using-spectrum-add-role.html).

‎docs/reference/providers/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,4 @@ Please see [Provider](../../concepts/provider.md) for an explanation of provider
66

77
{% page-ref page="google-cloud-platform.md" %}
88

9+
{% page-ref page="amazon-web-services.md" %}

‎docs/reference/providers/amazon-web-services.md‎

Lines changed: 0 additions & 90 deletions
Original file line numberDiff line numberDiff line change
@@ -25,93 +25,3 @@ offline_store:
2525
iam_role: arn:aws:iam::123456789012:role/redshift_s3_access_role
2626
```
2727
{% endcode %}
28-
29-
<!--
30-
TODO: figure out the permissions
31-
32-
### **Permissions**
33-
34-
<table>
35-
<thead>
36-
<tr>
37-
<th style="text-align:left"><b>Command</b>
38-
</th>
39-
<th style="text-align:left">Component</th>
40-
<th style="text-align:left">Permissions</th>
41-
<th style="text-align:left">Recommended Role</th>
42-
</tr>
43-
</thead>
44-
<tbody>
45-
<tr>
46-
<td style="text-align:left"><b>Apply</b>
47-
</td>
48-
<td style="text-align:left">BigQuery (source)</td>
49-
<td style="text-align:left">
50-
<p>bigquery.jobs.create</p>
51-
<p>bigquery.readsessions.create</p>
52-
<p>bigquery.readsessions.getData</p>
53-
</td>
54-
<td style="text-align:left">roles/bigquery.user</td>
55-
</tr>
56-
<tr>
57-
<td style="text-align:left"><b>Apply</b>
58-
</td>
59-
<td style="text-align:left">Datastore (destination)</td>
60-
<td style="text-align:left">
61-
<p>datastore.entities.allocateIds</p>
62-
<p>datastore.entities.create</p>
63-
<p>datastore.entities.delete</p>
64-
<p>datastore.entities.get</p>
65-
<p>datastore.entities.list</p>
66-
<p>datastore.entities.update</p>
67-
</td>
68-
<td style="text-align:left">roles/datastore.owner</td>
69-
</tr>
70-
<tr>
71-
<td style="text-align:left"><b>Materialize</b>
72-
</td>
73-
<td style="text-align:left">BigQuery (source)</td>
74-
<td style="text-align:left">bigquery.jobs.create</td>
75-
<td style="text-align:left">roles/bigquery.user</td>
76-
</tr>
77-
<tr>
78-
<td style="text-align:left"><b>Materialize</b>
79-
</td>
80-
<td style="text-align:left">Datastore (destination)</td>
81-
<td style="text-align:left">
82-
<p>datastore.entities.allocateIds</p>
83-
<p>datastore.entities.create</p>
84-
<p>datastore.entities.delete</p>
85-
<p>datastore.entities.get</p>
86-
<p>datastore.entities.list</p>
87-
<p>datastore.entities.update</p>
88-
<p>datastore.databases.get</p>
89-
</td>
90-
<td style="text-align:left">roles/datastore.owner</td>
91-
</tr>
92-
<tr>
93-
<td style="text-align:left"><b>Get Online Features</b>
94-
</td>
95-
<td style="text-align:left">Datastore</td>
96-
<td style="text-align:left">datastore.entities.get</td>
97-
<td style="text-align:left">roles/datastore.user</td>
98-
</tr>
99-
<tr>
100-
<td style="text-align:left"><b>Get Historical Features</b>
101-
</td>
102-
<td style="text-align:left">BigQuery (source)</td>
103-
<td style="text-align:left">
104-
<p>bigquery.datasets.get</p>
105-
<p>bigquery.tables.get</p>
106-
<p>bigquery.tables.create</p>
107-
<p>bigquery.tables.updateData</p>
108-
<p>bigquery.tables.update</p>
109-
<p>bigquery.tables.delete</p>
110-
<p>bigquery.tables.getData</p>
111-
</td>
112-
<td style="text-align:left">roles/bigquery.dataEditor</td>
113-
</tr>
114-
</tbody>
115-
</table>
116-
117-
-->

0 commit comments

Comments
 (0)