Skip to content

UN-3753 [GATED-FEAT] Route webhook notifications through PG-queue transport (flag-gated)#2198

Draft
muhammad-ali-e wants to merge 2 commits into
feat/UN-3445-pg-celery-decommissionfrom
feat/UN-3753-pg-notification-seam
Draft

UN-3753 [GATED-FEAT] Route webhook notifications through PG-queue transport (flag-gated)#2198
muhammad-ali-e wants to merge 2 commits into
feat/UN-3445-pg-celery-decommissionfrom
feat/UN-3753-pg-notification-seam

Conversation

@muhammad-ali-e

Copy link
Copy Markdown
Contributor

What

  • Adds notification_v2/notification_dispatch.py — a flag-gated transport seam for the buffered-webhook dispatch, mirroring pipeline_dispatch.py.
  • Routes _send_clubbed (the notification buffer-flush path) through the seam: PG queue when pg_queue_enabled is on for the org, else the prior celery_app.send_task (byte-identical).
  • _org_identifier() resolves the buffer's org pk → the string organization_id that resolve_transport needs for its Flipt decision (the org pk stays in the task kwargs — the worker's buffer-mark contract).
  • 6 unit tests (test_notification_dispatch.py) pinning both routes + fail-closed behavior.

Why

  • Part of migrating the notification worker off Celery onto the PG queue (UN-3753, under the PG-queue epic UN-3445), so Celery can eventually be decommissioned.
  • Universal pattern: flag-gate the enqueue point; flag OFF keeps Celery byte-unchanged, flag ON enqueues onto PG (drained by the PG notification consumer).

How

  • resolve_transport(execution_id, organization_id) decides the transport; is_pg_transport() is the single "counts as PG" check.
  • PG path: enqueue_task(task_name="send_webhook_notification", queue="notifications", …) — the exact args/kwargs the Celery call used, JSON-normalized by the producer.
  • Celery path: unchanged send_task with identical args/kwargs/queue.
  • Scope: only site 1 (_send_clubbed, the production flush path) is migrated. The WebhookSend / WebhookBatch internal endpoints stay on Celery — they use a countdown stagger the PG queue has no delayed-visibility for (a follow-up).

Can this PR break any existing features?

  • No. With pg_queue_enabled OFF (the default) resolve_transport fails closed to Celery and the seam calls the same send_task as before — byte-identical. The PG branch only runs when the flag is explicitly enabled for an org. Dev-tested on a live stack: flag-off produced 0 PG rows (Celery taken); an enqueued task was drained by the PG notification consumer and delivered end-to-end.

Database Migrations

  • None.

Env Config

  • None (uses the existing pg_queue_enabled Flipt flag).

Notes on Testing

  • pytest notification_v2/tests/test_notification_dispatch.py — 6 tests pass; pipeline_dispatch regression suite green.
  • Live dev-test: flag-off Celery parity (0 PG rows); PG round-trip enqueue → pg_queue_consumer (WORKER_TYPE=notification) → webhook delivered (200) → row acked.

Related Issues or PRs

  • UN-3753 (sub-task of UN-3445). Cloud counterpart: workerPgNotification chart (batched into the aux-workers cloud PR). Depends on the epic branch merging first, then rebased to main.

…nsport (flag-gated)

PG-queue analogue for the buffered-webhook dispatch, gated by the pg_queue_enabled
Flipt flag. Flag OFF (prod default) keeps the existing Celery send_task
byte-unchanged; flag ON enqueues send_webhook_notification onto the PG
`notifications` queue, drained by the PG notification consumer.

- Dispatch seam: notification_dispatch.py routes send_webhook_notification through
  resolve_transport — PG via enqueue_task when enabled for the org, else the prior
  celery_app.send_task (byte-identical, zero regression). _send_clubbed (the
  buffer-flush path) uses the seam; _org_identifier resolves the org pk -> string
  for the Flipt decision, keeping the pk in kwargs (the buffer/worker mark
  contract). + test_notification_dispatch.py (6 cases).
- Sites 2 & 3 (WebhookSend / WebhookBatch internal endpoints) stay on Celery: they
  use countdown stagger, which the PG queue has no delayed-visibility for. Follow-up.

Dev-tested on a live stack: flag-off took the Celery branch (0 PG rows); an enqueued
send_webhook_notification was drained by the pg_queue_consumer
(WORKER_TYPE=notification), delivered to a sink (200), and the row acked.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 90982e10-45c9-42c3-86ea-2318395c4bd4

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/UN-3753-pg-notification-seam

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@muhammad-ali-e muhammad-ali-e left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated PR review (PR Review Toolkit: code-reviewer, silent-failure-hunter, type-design-analyzer, pr-test-analyzer, comment-analyzer, code-simplifier).

No blocking correctness bugs found — the flag-gated transport seam is correct, fail-closed, and byte-identical on the Celery path. The comments below are ranked robustness / test-coverage / clarity improvements. Highest-value items: (1) the broad except in _send_clubbed mislabels permanent dispatch errors as transient broker failures and retries them forever, and (2) the PR's actual behavioral change in internal_api_views.py has no test coverage.

# org, else Celery (byte-identical to the prior send_task). resolve_transport
# keys on the org STRING id, but the buffer/worker contract below uses the
# org pk — hence _org_identifier(org_id) for routing, org_id in kwargs.
dispatch_webhook_notification(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1 · Robustness] Broad except Exception mislabels permanent dispatch errors as broker_failure and retries them forever.

The except Exception wrapping this call (below, ~L591) assumes a transient broker outage: it logs result=broker_failure and reverts the rows SENDING → PENDING to retry next tick. But it catches every exception type from dispatch_webhook_notification identically, including permanent ones:

  • ValueError from enqueue_task (priority range / reply_key+callback mutual-exclusion validation)
  • TypeError/serialization errors from _json_safe(...) on a non-encodable payload
  • programming errors (AttributeError/KeyError) in the seam, or a DB error from the _org_identifier(org_id) lookup (evaluated inside the try as a call arg)

All of these are permanent — they will fail identically every flush tick, re-render + re-dispatch the group, and emit a full logger.exception traceback to Sentry mislabeled as a transport failure, until the dispatch-attempt cap eventually dead-letters it.

Fix: branch on error class — mark permanent errors (ValueError, serialization TypeError) DEAD_LETTER immediately with a distinct metric (e.g. result=dispatch_error), and keep the revert-to-PENDING path only for genuine transport/broker exceptions.

Minor related: a PG enqueue failure is logged with logger.exception both inside enqueue_task and again here → two Sentry issues per PG failure vs one on the Celery path. Pick one owner for the traceback.

"organization_id": org_id,
},
queue="notifications",
organization_id=_org_identifier(org_id),

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1 · Tests] The PR's actual behavioral change here is untested.

The six new tests exercise dispatch_webhook_notification in isolation, but nothing covers this call site — where the two highest-risk regressions live:

  1. The two-org-values contract. Routing uses the string id (organization_id=_org_identifier(org_id)) while the worker's buffer-mark contract uses the pk (kwargs["organization_id"]=org_id). Swapping them would pass every existing test yet silently (a) fail routing closed to Celery for every org (a pk never matches a Flipt string-id rule) and (b) strand rows in SENDING (the mark endpoint filters on the pk). Add a mock-based test asserting call.kwargs["organization_id"] == "<string id>" and call.kwargs["kwargs"]["organization_id"] == <pk>.
  2. The revert-on-failure recovery. test_pg_enqueue_failure_propagates... only asserts the dispatch raises; the safety-critical half — _send_clubbed reverting only SENDING rows to PENDING, dispatched_at=None, dispatch_attempts F(-1), leaving already-DISPATCHED rows untouched — is untested. Best as a real DB test (seed one SENDING + one DISPATCHED row, patch dispatch to raise, assert the conditional update).
  3. _org_identifier itself has no test: org found → string id (not "None"), org missing → None.

``resolve_transport`` keys its Flipt decision on the org's string identifier,
but the buffer stores/uses the Organization pk. One indexed pk lookup per
dispatch group (post-commit) — negligible before the webhook HTTP call it
precedes. ``None`` (org deleted) fails closed to Celery in resolve_transport.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2 · Observability + comment accuracy] Silent None return, and the "org deleted" framing is misleading given the CASCADE FK.

NotificationBuffer.organization is on_delete=models.CASCADE, so deleting an Organization cascade-deletes its buffer rows — the "live buffer row + missing org" state this docstring describes is essentially unreachable in normal operation. If _org_identifier does return None, it signals a real data-integrity anomaly (dangling FK), yet it's swallowed with no log here; the only trace is resolve_transport's warning keyed on the random dispatch UUID, untraceable back to the org/rows.

Fix: log a warning (including org_pk) when the lookup returns None, and reword the comment to "data-anomaly guard" rather than an expected-deletion path.

Nits: "negligible before the webhook HTTP call it precedes" — the HTTP call happens later in the worker, not here; prefer "negligible relative to the downstream webhook dispatch." Also org_pk: Any could be narrowed to int (it's a pk throughout).

args: list[Any],
kwargs: dict[str, Any],
queue: str,
organization_id: str | None,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2 · Type design] The organization_id name is overloaded across the boundary — the core latent trap in this seam.

This param is the org string identifier (for Flipt routing), while kwargs["organization_id"] (passed by the caller) is the org pk (int, the worker buffer-mark contract). Same name, two incompatible meanings, both effectively untyped (str | None and a value inside dict[str, Any]). A future maintainer who "unifies" or swaps them gets silent mis-routing, not a type error: resolve_transport would str() the pk to "5", match no org segment, and quietly force Celery.

Cheapest high-value fix: rename the routing param to org_string_id / routing_org_identifier (one call-site edit) so the two identities can't be confused. Optionally encode them as distinct NewTypes (OrgStringId, OrgPk) to make a swap a type error at zero runtime cost.


Returns:
A task id string — the Celery ``AsyncResult`` id on the Celery path, or
the minted PG task id on the PG path (usable by callers that surface it

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2 · Comment accuracy] The return value's "usable by callers that surface it in an API response" is aspirational — no caller uses it.

The sole caller, _send_clubbed, is a fire-and-forget transaction.on_commit path and discards the return entirely; there is no API response in sight. The claim invites a future maintainer to believe a real consumer depends on it.

The rest of the Returns: block is accurate and worth keeping: I confirmed the function returns the minted dispatch_id (uuid str) on the PG path — deliberately not enqueue_task's int msg_id — giving Celery-parity str semantics, and three tests pin the dispatch_id == task_id == Flipt bucket invariant.

Fix: drop the parenthetical or mark it explicitly hypothetical ("returned for symmetry / any future caller").

dispatch_id = str(uuid.uuid4())
transport = resolve_transport(
execution_id=dispatch_id,
organization_id=organization_id or None,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3 · Simplify] organization_id or None is redundant.

resolve_transport already normalizes falsy input on its first line (if not organization_id: return celery), so "" and None resolve identically — the or None (converting ""None) can't change the outcome. It's also unreachable in practice: the sole caller passes _org_identifier(...), whose .values_list(...).first() yields str | None, never "". Passing organization_id directly is behavior-preserving (the test_none_org_fails_closed_to_celery identity assertion still holds).

Note: the sibling org_id=organization_id or "" on the PG path is not redundant — it satisfies enqueue_task's org_id: str contract, so leave that one.

…ing, call-site tests

- Robustness (P1): _send_clubbed's broad `except` mislabeled permanent dispatch
  errors as broker_failure and reverted to PENDING → retry-forever + mislabeled
  Sentry tracebacks until the attempt cap. Branch on class: ValueError/TypeError
  (enqueue_task validation / payload serialization) → dead-letter now with a
  distinct `result=dispatch_error` metric; keep revert-to-PENDING only for genuine
  transport/broker exceptions.
- Type design (P2): renamed the seam's routing param organization_id → org_string_id
  so it can't be conflated with the org pk in `kwargs["organization_id"]` (the
  worker buffer-mark contract) — swapping them was a silent mis-route to Celery.
- Observability (P2): _org_identifier now logs a warning (with org_pk) when the
  lookup returns None — a dangling FK is a data anomaly (CASCADE makes it otherwise
  unreachable), not an expected "org deleted" path; reworded the docstring and
  narrowed org_pk: int. Fixed the "before the webhook HTTP call" wording.
- Comment accuracy (P2): dropped the aspirational "usable by callers that surface
  it in an API response" from the Returns block (the sole caller discards it).
- Simplify (P3): dropped the redundant `or None` on the routing arg (resolve_transport
  already normalizes falsy); kept the load-bearing `or ""` on enqueue_task's org_id.
- Tests (P1): new test_send_clubbed.py locks the two-org-identifier contract
  (string routes, pk in kwargs), the transient→PENDING vs permanent→DEAD_LETTER
  recovery split, and _org_identifier (string id / None+warn). 11 tests pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@muhammad-ali-e

Copy link
Copy Markdown
Contributor Author

Addressed all 6 threads in 9f13db367.

  • P1 · broad except mislabels permanent errors_send_clubbed now branches on class: except (ValueError, TypeError) (enqueue_task validation / payload serialization) → dead-letter now with a distinct result=dispatch_error metric; except Exception keeps the transient revert-to-PENDING + attempt refund. So a permanent error no longer re-renders/re-dispatches and emits a mislabeled broker_failure traceback every tick until the cap. (On the double-traceback: _send_clubbed owns the domain metric+traceback; enqueue_task's log is a producer-level breadcrumb shared by all callers — leaving that as-is rather than touching the shared producer here.)
  • P1 · call site untested — new test_send_clubbed.py: (1) the two-org-identifier contract — asserts org_string_id == "<string>" for routing while kwargs["organization_id"] == <pk>; (2) recovery — transient → SENDING-guarded revert to PENDING, permanent → DEAD_LETTER; (3) _org_identifier — string id, and None→warn. 11 tests pass.
  • P2 · overloaded organization_id — renamed the seam's routing param to org_string_id (one call-site + test edit) so it can't be confused with the pk in kwargs["organization_id"].
  • P2 · silent None + "org deleted" framing_org_identifier now logs a warning with org_pk on None, reworded to a "data-anomaly guard" (CASCADE FK makes the missing-org state otherwise unreachable), narrowed org_pk: int, and fixed the "before the webhook HTTP call" wording.
  • P2 · aspirational Returns comment — dropped "usable by callers that surface it in an API response"; now "returned for symmetry / any future caller; the sole current caller discards it."
  • P3 · redundant or None — routing arg now passes org_string_id straight through (resolve_transport normalizes falsy); kept the load-bearing or "" on enqueue_task's org_id.

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant