We initialize view snapshots used by ad hoc queries by performing a "warm-up" step on startup. This step is normally performed in the INITIALIZING state, which guarantees that the snapshots are initialized by the time the pipeline is RUNNING. However, when bootstrapping is involved, we still need to perform the warm-up step post-boostrapping to initialize any views that did not participate in bootstrapping. This step is currently performed in the RUNNING state, introducing a race where the pipeline is running but ad hoc queries don't return correct results. The pipeline can be in this state for a long time if the first post-bootstrapping step takes a long time to complete, e.g., because the pipeline is stalled waiting for out
The solution involves either returning some type of not-yet-ready-try-again error from the adhoc endpoint or, ideally, not changing pipeline state to RUNNING until the warm up step has been performed.
We should also consider a design that doesn't rely on the circuit to perform a step to refresh the snapshots.
We initialize view snapshots used by ad hoc queries by performing a "warm-up" step on startup. This step is normally performed in the INITIALIZING state, which guarantees that the snapshots are initialized by the time the pipeline is RUNNING. However, when bootstrapping is involved, we still need to perform the warm-up step post-boostrapping to initialize any views that did not participate in bootstrapping. This step is currently performed in the RUNNING state, introducing a race where the pipeline is running but ad hoc queries don't return correct results. The pipeline can be in this state for a long time if the first post-bootstrapping step takes a long time to complete, e.g., because the pipeline is stalled waiting for out
The solution involves either returning some type of not-yet-ready-try-again error from the adhoc endpoint or, ideally, not changing pipeline state to RUNNING until the warm up step has been performed.
We should also consider a design that doesn't rely on the circuit to perform a step to refresh the snapshots.