UN-3753 [GATED-FEAT] Route webhook notifications through PG-queue transport (flag-gated)#2198
Conversation
…nsport (flag-gated) PG-queue analogue for the buffered-webhook dispatch, gated by the pg_queue_enabled Flipt flag. Flag OFF (prod default) keeps the existing Celery send_task byte-unchanged; flag ON enqueues send_webhook_notification onto the PG `notifications` queue, drained by the PG notification consumer. - Dispatch seam: notification_dispatch.py routes send_webhook_notification through resolve_transport — PG via enqueue_task when enabled for the org, else the prior celery_app.send_task (byte-identical, zero regression). _send_clubbed (the buffer-flush path) uses the seam; _org_identifier resolves the org pk -> string for the Flipt decision, keeping the pk in kwargs (the buffer/worker mark contract). + test_notification_dispatch.py (6 cases). - Sites 2 & 3 (WebhookSend / WebhookBatch internal endpoints) stay on Celery: they use countdown stagger, which the PG queue has no delayed-visibility for. Follow-up. Dev-tested on a live stack: flag-off took the Celery branch (0 PG rows); an enqueued send_webhook_notification was drained by the pg_queue_consumer (WORKER_TYPE=notification), delivered to a sink (200), and the row acked. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
muhammad-ali-e
left a comment
There was a problem hiding this comment.
Automated PR review (PR Review Toolkit: code-reviewer, silent-failure-hunter, type-design-analyzer, pr-test-analyzer, comment-analyzer, code-simplifier).
No blocking correctness bugs found — the flag-gated transport seam is correct, fail-closed, and byte-identical on the Celery path. The comments below are ranked robustness / test-coverage / clarity improvements. Highest-value items: (1) the broad except in _send_clubbed mislabels permanent dispatch errors as transient broker failures and retries them forever, and (2) the PR's actual behavioral change in internal_api_views.py has no test coverage.
| # org, else Celery (byte-identical to the prior send_task). resolve_transport | ||
| # keys on the org STRING id, but the buffer/worker contract below uses the | ||
| # org pk — hence _org_identifier(org_id) for routing, org_id in kwargs. | ||
| dispatch_webhook_notification( |
There was a problem hiding this comment.
[P1 · Robustness] Broad except Exception mislabels permanent dispatch errors as broker_failure and retries them forever.
The except Exception wrapping this call (below, ~L591) assumes a transient broker outage: it logs result=broker_failure and reverts the rows SENDING → PENDING to retry next tick. But it catches every exception type from dispatch_webhook_notification identically, including permanent ones:
ValueErrorfromenqueue_task(priority range / reply_key+callback mutual-exclusion validation)TypeError/serialization errors from_json_safe(...)on a non-encodable payload- programming errors (
AttributeError/KeyError) in the seam, or a DB error from the_org_identifier(org_id)lookup (evaluated inside thetryas a call arg)
All of these are permanent — they will fail identically every flush tick, re-render + re-dispatch the group, and emit a full logger.exception traceback to Sentry mislabeled as a transport failure, until the dispatch-attempt cap eventually dead-letters it.
Fix: branch on error class — mark permanent errors (ValueError, serialization TypeError) DEAD_LETTER immediately with a distinct metric (e.g. result=dispatch_error), and keep the revert-to-PENDING path only for genuine transport/broker exceptions.
Minor related: a PG enqueue failure is logged with logger.exception both inside enqueue_task and again here → two Sentry issues per PG failure vs one on the Celery path. Pick one owner for the traceback.
| "organization_id": org_id, | ||
| }, | ||
| queue="notifications", | ||
| organization_id=_org_identifier(org_id), |
There was a problem hiding this comment.
[P1 · Tests] The PR's actual behavioral change here is untested.
The six new tests exercise dispatch_webhook_notification in isolation, but nothing covers this call site — where the two highest-risk regressions live:
- The two-org-values contract. Routing uses the string id (
organization_id=_org_identifier(org_id)) while the worker's buffer-mark contract uses the pk (kwargs["organization_id"]=org_id). Swapping them would pass every existing test yet silently (a) fail routing closed to Celery for every org (a pk never matches a Flipt string-id rule) and (b) strand rows inSENDING(the mark endpoint filters on the pk). Add a mock-based test assertingcall.kwargs["organization_id"] == "<string id>"andcall.kwargs["kwargs"]["organization_id"] == <pk>. - The revert-on-failure recovery.
test_pg_enqueue_failure_propagates...only asserts the dispatch raises; the safety-critical half —_send_clubbedreverting onlySENDINGrows toPENDING,dispatched_at=None,dispatch_attemptsF(-1), leaving already-DISPATCHEDrows untouched — is untested. Best as a real DB test (seed oneSENDING+ oneDISPATCHEDrow, patch dispatch to raise, assert the conditional update). _org_identifieritself has no test: org found → string id (not"None"), org missing →None.
| ``resolve_transport`` keys its Flipt decision on the org's string identifier, | ||
| but the buffer stores/uses the Organization pk. One indexed pk lookup per | ||
| dispatch group (post-commit) — negligible before the webhook HTTP call it | ||
| precedes. ``None`` (org deleted) fails closed to Celery in resolve_transport. |
There was a problem hiding this comment.
[P2 · Observability + comment accuracy] Silent None return, and the "org deleted" framing is misleading given the CASCADE FK.
NotificationBuffer.organization is on_delete=models.CASCADE, so deleting an Organization cascade-deletes its buffer rows — the "live buffer row + missing org" state this docstring describes is essentially unreachable in normal operation. If _org_identifier does return None, it signals a real data-integrity anomaly (dangling FK), yet it's swallowed with no log here; the only trace is resolve_transport's warning keyed on the random dispatch UUID, untraceable back to the org/rows.
Fix: log a warning (including org_pk) when the lookup returns None, and reword the comment to "data-anomaly guard" rather than an expected-deletion path.
Nits: "negligible before the webhook HTTP call it precedes" — the HTTP call happens later in the worker, not here; prefer "negligible relative to the downstream webhook dispatch." Also org_pk: Any could be narrowed to int (it's a pk throughout).
| args: list[Any], | ||
| kwargs: dict[str, Any], | ||
| queue: str, | ||
| organization_id: str | None, |
There was a problem hiding this comment.
[P2 · Type design] The organization_id name is overloaded across the boundary — the core latent trap in this seam.
This param is the org string identifier (for Flipt routing), while kwargs["organization_id"] (passed by the caller) is the org pk (int, the worker buffer-mark contract). Same name, two incompatible meanings, both effectively untyped (str | None and a value inside dict[str, Any]). A future maintainer who "unifies" or swaps them gets silent mis-routing, not a type error: resolve_transport would str() the pk to "5", match no org segment, and quietly force Celery.
Cheapest high-value fix: rename the routing param to org_string_id / routing_org_identifier (one call-site edit) so the two identities can't be confused. Optionally encode them as distinct NewTypes (OrgStringId, OrgPk) to make a swap a type error at zero runtime cost.
|
|
||
| Returns: | ||
| A task id string — the Celery ``AsyncResult`` id on the Celery path, or | ||
| the minted PG task id on the PG path (usable by callers that surface it |
There was a problem hiding this comment.
[P2 · Comment accuracy] The return value's "usable by callers that surface it in an API response" is aspirational — no caller uses it.
The sole caller, _send_clubbed, is a fire-and-forget transaction.on_commit path and discards the return entirely; there is no API response in sight. The claim invites a future maintainer to believe a real consumer depends on it.
The rest of the Returns: block is accurate and worth keeping: I confirmed the function returns the minted dispatch_id (uuid str) on the PG path — deliberately not enqueue_task's int msg_id — giving Celery-parity str semantics, and three tests pin the dispatch_id == task_id == Flipt bucket invariant.
Fix: drop the parenthetical or mark it explicitly hypothetical ("returned for symmetry / any future caller").
| dispatch_id = str(uuid.uuid4()) | ||
| transport = resolve_transport( | ||
| execution_id=dispatch_id, | ||
| organization_id=organization_id or None, |
There was a problem hiding this comment.
[P3 · Simplify] organization_id or None is redundant.
resolve_transport already normalizes falsy input on its first line (if not organization_id: return celery), so "" and None resolve identically — the or None (converting "" → None) can't change the outcome. It's also unreachable in practice: the sole caller passes _org_identifier(...), whose .values_list(...).first() yields str | None, never "". Passing organization_id directly is behavior-preserving (the test_none_org_fails_closed_to_celery identity assertion still holds).
Note: the sibling org_id=organization_id or "" on the PG path is not redundant — it satisfies enqueue_task's org_id: str contract, so leave that one.
…ing, call-site tests - Robustness (P1): _send_clubbed's broad `except` mislabeled permanent dispatch errors as broker_failure and reverted to PENDING → retry-forever + mislabeled Sentry tracebacks until the attempt cap. Branch on class: ValueError/TypeError (enqueue_task validation / payload serialization) → dead-letter now with a distinct `result=dispatch_error` metric; keep revert-to-PENDING only for genuine transport/broker exceptions. - Type design (P2): renamed the seam's routing param organization_id → org_string_id so it can't be conflated with the org pk in `kwargs["organization_id"]` (the worker buffer-mark contract) — swapping them was a silent mis-route to Celery. - Observability (P2): _org_identifier now logs a warning (with org_pk) when the lookup returns None — a dangling FK is a data anomaly (CASCADE makes it otherwise unreachable), not an expected "org deleted" path; reworded the docstring and narrowed org_pk: int. Fixed the "before the webhook HTTP call" wording. - Comment accuracy (P2): dropped the aspirational "usable by callers that surface it in an API response" from the Returns block (the sole caller discards it). - Simplify (P3): dropped the redundant `or None` on the routing arg (resolve_transport already normalizes falsy); kept the load-bearing `or ""` on enqueue_task's org_id. - Tests (P1): new test_send_clubbed.py locks the two-org-identifier contract (string routes, pk in kwargs), the transient→PENDING vs permanent→DEAD_LETTER recovery split, and _org_identifier (string id / None+warn). 11 tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Addressed all 6 threads in
|
|



What
notification_v2/notification_dispatch.py— a flag-gated transport seam for the buffered-webhook dispatch, mirroringpipeline_dispatch.py._send_clubbed(the notification buffer-flush path) through the seam: PG queue whenpg_queue_enabledis on for the org, else the priorcelery_app.send_task(byte-identical)._org_identifier()resolves the buffer's org pk → the stringorganization_idthatresolve_transportneeds for its Flipt decision (the org pk stays in the task kwargs — the worker's buffer-mark contract).test_notification_dispatch.py) pinning both routes + fail-closed behavior.Why
How
resolve_transport(execution_id, organization_id)decides the transport;is_pg_transport()is the single "counts as PG" check.enqueue_task(task_name="send_webhook_notification", queue="notifications", …)— the exact args/kwargs the Celery call used, JSON-normalized by the producer.send_taskwith identical args/kwargs/queue._send_clubbed, the production flush path) is migrated. TheWebhookSend/WebhookBatchinternal endpoints stay on Celery — they use acountdownstagger the PG queue has no delayed-visibility for (a follow-up).Can this PR break any existing features?
pg_queue_enabledOFF (the default)resolve_transportfails closed to Celery and the seam calls the samesend_taskas before — byte-identical. The PG branch only runs when the flag is explicitly enabled for an org. Dev-tested on a live stack: flag-off produced 0 PG rows (Celery taken); an enqueued task was drained by the PG notification consumer and delivered end-to-end.Database Migrations
Env Config
pg_queue_enabledFlipt flag).Notes on Testing
pytest notification_v2/tests/test_notification_dispatch.py— 6 tests pass;pipeline_dispatchregression suite green.pg_queue_consumer(WORKER_TYPE=notification) → webhook delivered (200) → row acked.Related Issues or PRs
workerPgNotificationchart (batched into the aux-workers cloud PR). Depends on the epic branch merging first, then rebased to main.