Signed-off-by: Mihai Budiu <mbudiu@feldera.com>

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>

When a table row is retrieved for update purposes, the row itself
is now locked by adding `FOR UPDATE` to the query. Although the current
global lock mechanism on the database connection of the API server
prevents concurrent access, the runner can separately access/change the
rows (although, only certain fields when they are in its domain).
One such concurrent one is setting of the desired resources status, and
the transition of current resources status based on it.

A lock timeout of 10 seconds is added along with two custom error types
`LockTookTooLong` and `DeadlockDetected`. Both correspond to HTTP error
code 503, indicating to the client it should retry.

One example of concurrent access:
1. Assume a pipeline is fully compiled
2. User: calls `pipeline.start(wait=False)`
3. API server: `/start` is called -> sets desired status to
  `Provisioned` -> returns
4. User: call returns as `wait=False`
5. Runner: tasked with transitioning the current status towards
   `Provisioned`
6. Runner: starts transaction
7. Runner: retrieves the pipeline to check that the current status is
   `Stopped` and desired status is not `Stopped`, as otherwise it will
   not still transition toward `Provisioned`. This is not the case.
8. User calls `pipeline.stop(wait=True)`
9. API server: `/stop` is called -> sets desired status to `Stopped` ->
   returns
10. API server: `GET pipeline` is called -> returns
11. User call returns: sees current and desired status is `Stopped`
12. Runner: transitions the status to Provisioning (as check at step (7)
    passed)
13. Runner: commits transaction
14. Runner: tasked with transitioning the current status towards
    `Stopped`, and performs that operation. Current status: `Stopping`.
15. User: calls `pipeline.start()`
16. API server: `/start` is called -> desired status is right now
    `Stopped`, and current status is `Stopping`, so setting desired
    status to `Provisioned` is not allowed -> returns error
    `IllegalPipelineAction`
17. User: `FelderaAPIError` exception is raised

This fixes it by having the runner take an FOR UPDATE lock at step (7),
and as such the API server at step (9) will be forced to wait until
the runner is finished before being able to acquire the lock itself.
After which, at step (11) the user will get back that the current status
is not yet `Stopped` and will continue to poll until it is. As such,
the user starting the pipeline again (step (15)) only happens once the
pipeline is truly stopped, and will succeed.

Signed-off-by: Simon Kassing <simon.kassing@feldera.com>

The number of refresh changes depends on the number of runtime, storage,
and resources status updates, which can be more frequent when under
heavy load.

Signed-off-by: Simon Kassing <simon.kassing@feldera.com>

Signed-off-by: Mihai Budiu <mbudiu@feldera.com>