Repository navigation
feast-operator: support setting the ServiceAccount annotations for cloud platform IAM use case #5302
Description
Is your feature request related to a problem? Please describe.
Kubernetes platforms on AWS and GKE allows pods to assume the IAM roles/service account by setting certain annotations on the ServiceAccount resource. In the background, this annotation will be used to inject the credentials required for the Pod to access internal services of the cloud platform.
In AWS EKS, it uses annotation: eks.amazonaws.com/role-arn
In GCP GKE, it uses annotation: iam.gke.io/gcp-service-account
Describe the solution you'd like
Add section inside the FeatureStore CRD to set configure the annotations for the ServiceAccount that will be created. Maybe can add it to featurestore.spec.services.offlineStore.serviceAccount.annotations.
edit: add specific to just offlineStore instead of directly under services
Describe alternatives you've considered
Manually adding the annotation to the created ServiceAccount resource after the FeatureStore has been created and the feast-operator has created the ServiceAccount.
Additional context
I dug around the feast-operator code and found this function that sets the metadata of the ServiceAccount. From this I think can be done to also add the annotations when specified in the FeatureStore CR spec.
Activity
tchughesiv commentedon Apr 28, 2025
i wonder if this should be moved up a level instead and the annotations get added to all objects created by the Operator (Deployment, ServiceAccount, etc.)?
featurestore.spec.services.annotationsThis would allow support the following as well -
https://kubernetes-on-aws.readthedocs.io/en/latest/user-guide/iam-roles.html
Or do we think the setting of annotations needs to be object/Kind specific?
pokgak commentedon Apr 29, 2025
i am for the granular option because you might not want to grant the same role to different services eg in my use case the OnlineStore deployment is using redis which doesn't need any cloud specific IAM while my OfflineStore deployment uses BigQuery which requires the GCP IAM. Granting the role with blanket policy will open abuse of that credentials from other unrelated service imo.
edit: just noticed your question is more about not doing it per kind which is what i'm suggesting in my OP above. i still think the same, having the flexibility to do it on per service per kind level grants more flexibility on the use cases possible.
tchughesiv commentedon Apr 29, 2025
@pokgak yes, per
Kindwould be the only way we could be granular since the different feast components (online/offline/registry/ui) all live within the same Deployment/Pod and all use the same ServiceAccount. The more i think about it I'm not sure there's any real benefit toKindgranularity either.My vote is
featurestore.spec.services.annotationsand the annotations are set across the board in the FeatureStore's ServiceAccount/Deployment/Service/etc. This would allow users to leverage annotations for any purpose they choose.pokgak commentedon Apr 29, 2025
i'm open to any as long as i can use the annotation :)
tchughesiv commentedon May 7, 2025
@pokgak after trying out different solutions, the syncing/overriding of annotations from the CR spec can become a complex situation and most of my attempts resulted in either infinite loops updating the objects (competing with k8s controllers) or stale annotations being left behind.
as it turns out, the best/simplest solution already exists... you can annotate the SA directly after installing your FeatureStore CR -
pokgak commentedon May 7, 2025
understandable. thats the way i'm doing it also right now tho i'd say its more of a workaround than a fix since it still requires manual steps still after the installation. that works for me currently tho so i guess thats the best we have right now. i might take a stab a it if i manage to find time to play around with the code too.
thanks for looking into this anywyas @tchughesiv! feel free to close the issue.
tchughesiv commentedon May 7, 2025
Of course! while it's easy to add new annotations, it gets complex ensuring those same annotations get cleaned up when later removed from the same CR spec.
That said, i did submit a small PR to ensure ALL operator created objects can be annotated going forward. There were a couple the operator was overriding.