Feast apply can leak resources and create inconsistent state #2116

Consider feature store with AWS as a configured provider.
When we call feast apply under the hood AwsProvider.update_infra is being called:

def apply(...):
    ... preparation ...
    provider.update_infra(...)
    registry.commit()

Now, update_infra could be a multi-staged operation. For example, current AWS implementation consists of updating online store and then creating Lambda feature server:

def update_infra(...):
    online_store.update(...)
    deploy_lambda_server(...)

When an exception occurs in deploy_lambda_server online store is already created, however since registry.commit() is called only when all infra was successfully updated - these changes will be lost and resources from online store won't be cleaned up properly on teardown.

When number of different components that are managed by the provider will grow even further - the risk to leak resources and create inconsistent state will also increase. Thus, we need to add rollback mechanism, that will be triggered in case of exception.

Expected Behavior

Either all infrastructure is updated or none.

Current Behavior

Infrastructure can be updated partially and the registry will not have records about resources that were actually created.

Steps to reproduce

Specifications

• Version: 0.15.1
• Platform:
• Subsystem:

Possible Solution

Create rollback API.