Summary: This issue tracks the addition of a job/dataset Id to track the ingestion of batch data through the Python SDK so that a dataset can be removed from a warehouse database by the user.
Currently there is no easy way for users to undo a failed ingestion. Once data is ingested into Feast through the Python SDK it is written to both feature warehouses and online stores. The current workaround is to reingest the same data (the same entities and timestaps). Feast will allow the user to retrieve the last inserted value and deduplicate results by itself.
In the case of historical data, however, this could lead to large amounts of failed ingestions remaining in the historical store. In an ideal case the user would be able to remove this data after ingestion.
The easiest way to do this is to add an identifier to an ingestion job that tracks all feature rows that are inserted. The user can then go directly to the database and drop these rows.
Summary: This issue tracks the addition of a job/dataset Id to track the ingestion of batch data through the Python SDK so that a dataset can be removed from a warehouse database by the user.
Currently there is no easy way for users to undo a failed ingestion. Once data is ingested into Feast through the Python SDK it is written to both feature warehouses and online stores. The current workaround is to reingest the same data (the same entities and timestaps). Feast will allow the user to retrieve the last inserted value and deduplicate results by itself.
In the case of historical data, however, this could lead to large amounts of failed ingestions remaining in the historical store. In an ideal case the user would be able to remove this data after ingestion.
The easiest way to do this is to add an identifier to an ingestion job that tracks all feature rows that are inserted. The user can then go directly to the database and drop these rows.