UN-3798 [FIX] Skip broken file-path task load for PG pluggable workers#2197
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The top-level worker.py loaded a pluggable worker's tasks.py via
spec_from_file_location("tasks", ...) — a bare module name with no parent
package — so the plugin's relative imports (from .clients import ...) failed
with "attempted relative import with no known parent package", crash-looping
every cloud PG pluggable worker (agentic_callback/UN-3754, agentic_studio/
UN-3779, bulk_download/UN-3752) on startup under `python -m pg_queue_consumer`.
The tasks are already registered by that point: WorkerBuilder.build_celery_app()
(called just above) verifies a pluggable type by importing
pluggable_worker.{type}.worker as a proper package (_verify_pluggable_worker_exists
→ import_module), which runs the plugin's `from . import tasks` and registers the
tasks on this same app. So the file-path load is both redundant and broken.
Fix: skip the file-path load for pluggable workers; non-pluggable (top-level)
workers keep it unchanged. The Celery path is unaffected — it runs via
`celery -A pluggable_worker.{type}.worker` (a dotted package import) and never
touches this loader; the is_pluggable() file-path branch never ran successfully.
Validated on a running dev stack via `python -m pg_queue_consumer`:
agentic_callback and agentic_studio now start clean ("tasks already registered
… skipping file-path task load" → "ready for Celery"); general (non-pluggable)
loads unchanged. Prerequisite for UN-3752 / UN-3754 / UN-3779.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
c407b45 to
9314a14
Compare
muhammad-ali-e
left a comment
There was a problem hiding this comment.
Automated PR review (PR Review Toolkit: Code Reviewer, Silent-Failure Hunter, Type-Design Analyzer, Test Analyzer, Comment Analyzer, Code Simplifier).
Verdict: the fix itself is correct and behavior-preserving — the file-path load is cleanly narrowed to the non-pluggable branch, and the pluggable skip is safe (a missing/broken plugin already raises ImportError in build_celery_app before this code runs). No blocking bugs. The comments below are mostly comment-accuracy fixes plus optional hardening/refactor suggestions.
| # `importlib.import_module("pluggable_worker.{type}.worker")` — a proper | ||
| # PACKAGE import that runs the plugin's own `from . import tasks`, resolving | ||
| # its relative imports (`from .clients import ...`) and registering the tasks | ||
| # on this same app. The generic file-path load below loads tasks.py under a |
There was a problem hiding this comment.
[Comment accuracy — MEDIUM] "registering the tasks on this same app" is contradicted by call ordering. _verify_pluggable_worker_exists() performs the import_module at builder.py:60, before app = Celery(app_name) is created at builder.py:74. The app object this file uses does not exist yet when the plugin is imported, so registration must go through Celery's deferred binding (@shared_task / connect_on_app_finalize), not a direct attach to "this same app".
Suggested wording: "…runs the plugin's task-registration code. Celery binds those tasks to the worker's app when it finalizes, so the tasks are already registered by this point."
| # on this same app. The generic file-path load below loads tasks.py under a | ||
| # bare "tasks" spec (no parent package), which BREAKS those relative imports | ||
| # ("attempted relative import with no known parent package"), so it MUST be | ||
| # skipped for pluggable workers — otherwise every cloud PG pluggable worker |
There was a problem hiding this comment.
[Comment accuracy — LOW] "otherwise every cloud PG pluggable worker crashes on startup" overstates the guarantee. The attempted relative import with no known parent package crash only fires for plugins whose tasks.py actually uses relative imports; a plugin written with absolute imports would not crash. Suggest softening to the real precondition, e.g. "…which breaks any relative imports in the plugin's tasks.py, so the file-path load must be skipped for pluggable workers."
| # build_celery_app() above verifies the plugin via | ||
| # WorkerBuilder._verify_pluggable_worker_exists, which does | ||
| # `importlib.import_module("pluggable_worker.{type}.worker")` — a proper | ||
| # PACKAGE import that runs the plugin's own `from . import tasks`, resolving |
There was a problem hiding this comment.
[Maintainability / rot risk — LOW] This comment hard-codes the internal structure of cloud-only plugin code that is not in this repo (pluggable_worker/ is gitignored in OSS): from . import tasks, from .clients import .... Nobody editing the OSS side can validate these specifics, and they silently rot if a plugin registers tasks differently. Prefer stating the contract ("pluggable workers are proper packages; verification imports pluggable_worker.<type>.worker as a package, running its own task registration incl. relative imports") and marking the concrete from . import ... examples as illustrative ("e.g.").
| # Pluggable workers live inside workers/pluggable_worker/{worker_name} | ||
| worker_directory = os.path.join("pluggable_worker", worker_type.value) | ||
| worker_path = os.path.join(base_dir, worker_directory) | ||
| # Pluggable workers' tasks are ALREADY registered by this point: |
There was a problem hiding this comment.
[Structure / testability — suggestion] Three agents (Simplifier, Test Analyzer, Type-Design) converged here: extract lines 445–490 into a load_worker_tasks(worker_type) helper. Benefits: (1) the giant explanatory comment becomes a proper docstring; (2) nesting collapses from depth 3 to depth 1 via guard-clause returns (impossible at module scope today); (3) it creates the seam for the single highest-value regression test — assert spec_from_file_location is never called when is_pluggable() is True — which permanently pins this fix against a future re-merge of the branches. All locals (base_dir, worker_directory, worker_path, tasks_file, spec, tasks_module) are used only in this block, so extraction leaks nothing. Reasonable to defer as a fast-follow given this is a hotfix.
| logger.warning(f"⚠️ No tasks.py found at: {tasks_file}") | ||
| else: | ||
| logger.error(f"❌ Worker directory not found: {worker_path}") | ||
| logger.error(f"❌ Worker directory not found: {worker_path}") |
There was a problem hiding this comment.
[Silent failure — MEDIUM] Two zero-task boot paths survive here: (a) this else logs an error but execution continues to the ✅ Successfully loaded line at 492 — a misleading success after a fatal misconfig (pre-existing, just re-indented); (b) after removing the file-path load for pluggable workers, plugin-side import is now the sole task-registration mechanism with no post-check. Celery does not error on an empty task registry — the worker starts and silently processes nothing. Consider one guard after the if/else that covers both paths:
registered = [n for n in app.tasks if not n.startswith("celery.")]
if not registered:
raise RuntimeError(
f"No tasks registered for worker '{worker_type.value}' "
f"(pluggable={worker_type.is_pluggable()}) — worker would start but process nothing."
)| # Non-pluggable (top-level) workers register tasks via a file-path load of | ||
| # their tasks.py — they use absolute imports, with their dir on sys.path. | ||
| # Enum values use underscores (Python module names); a few on-disk dirs still | ||
| # use hyphens (e.g. api-deployment). Derive the directory from the |
There was a problem hiding this comment.
[Type design — LOW] Deriving the on-disk directory as worker_type.to_import_path().rsplit(".", 1)[0] (line 468) reuses an import-path string as a filesystem path — it only works because every path is "{dir}.tasks". Consider an explicit WorkerType.to_directory() -> str that owns the hyphen/underscore mapping, so intent is named and the mapping lives in one place. (Keep the task-loading logic itself out of the enum — the other WorkerType methods are pure derivations; a load_tasks() would drag sys.path/importlib/the Celery app into a lightweight str, Enum.)
| if os.path.exists(tasks_file): | ||
| logger.info(f"📋 Loading tasks from: {tasks_file}") | ||
| # Import the tasks module to register tasks with the app | ||
| import importlib.util |
There was a problem hiding this comment.
[Nit] import importlib.util is done inside the innermost if. Since os/sys are already imported at module top, moving this up with them is more consistent. Style-only.
…tory(), zero-task guard - Extract the task-load block into load_worker_tasks(worker_type) with guard-clause returns (depth 3 -> 1) and an accurate docstring: pluggable tasks register via WorkerBuilder's package import and Celery binds them on app finalize; the file-path load is skipped for them because it breaks any relative imports in the plugin's tasks.py. Softened the overstated "every worker crashes" and marked the cloud-plugin `from . import tasks` example as illustrative (contract, not internals). - Add WorkerType.to_directory() as the single source for the underscore->hyphen dir mapping; to_import_path() and the file-path loader both read it (no more slicing the import path). + tests. - Add a post-load zero-task check as a WARNING (not a hard raise): pluggable tasks bind on app finalize which can be after this point, so a raise would false-positive on a correctly-configured pluggable worker (the exact regression this PR fixes). - Move `import importlib.util` to the module-top imports. Deferred (per reviewer's hotfix note): the "spec_from_file_location never called for pluggable" regression test — worker.py runs infra init + builds the Celery app at import, so there is no clean seam to exercise the loader in isolation without a larger main()-guard refactor. The extraction creates that seam for a fast-follow. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Addressed all 7 threads in
Deferred (per your hotfix note): the " |
|
74f72df
into
feat/UN-3445-pg-queue-aux-workers
|
| Filename | Overview |
|---|---|
| workers/worker.py | Separates task loading by worker type, skipping the broken bare-module import for pluggable workers while preserving non-pluggable loading. |
| workers/shared/enums/worker_enums_base.py | Extracts filesystem-directory resolution into to_directory() and keeps existing import-path behavior derived from it. |
| workers/tests/test_worker_enums_directory.py | Verifies the hyphenated API-deployment mapping, normal passthrough behavior, and consistency between directory and import paths. |
Flowchart
%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[Build Celery app] --> B{Pluggable worker?}
B -->|Yes| C[WorkerBuilder imports plugin worker package]
C --> D[Plugin tasks register through package imports]
D --> E[Skip bare file-path task load]
B -->|No| F[Resolve WorkerType.to_directory]
F --> G[Load tasks.py from worker directory]
E --> H[Inspect application task registry]
G --> H
Reviews (1): Last reviewed commit: "UN-3798 [FIX] Address #2197 review: accu..." | Re-trigger Greptile



What
WORKER_TYPEs, the top-levelworkers/worker.pyno longer loads the plugin'stasks.pyby file path under a bare"tasks"module name. Their tasks are already registered byWorkerBuilder.build_celery_app(); the file-path load is skipped.Why
spec_from_file_location("tasks", ...)— a bare name with no parent package — so every plugintasks.pyrelative import (from .clients import ...) failed withImportError: attempted relative import with no known parent package, crash-looping the worker on startup underpython -m pg_queue_consumer.celery -A pluggable_worker.{type}.worker(a proper dotted package import), which resolves relative imports and never touches this loader. Theis_pluggable()file-path branch never actually ran successfully.How
build_celery_app()(called just above the loader) verifies a pluggable type via_verify_pluggable_worker_exists→importlib.import_module("pluggable_worker.{type}.worker")— a package import that runs the plugin'sfrom . import tasksand registers the tasks on this same app. So the file-path load is redundant + broken; skip it for pluggable workers.Can this PR break any existing features?
elsebranch). Removes a code path that never ran successfully. PG and Celery register identical tasks (both viabuild_celery_app).Database Migrations
Env Config
Notes on Testing
python -m pg_queue_consumer:agentic_callbackandagentic_studionow start clean (tasks already registered … skipping file-path task load→ready for Celery);general(non-pluggable) loads unchanged via the file-path path.worker.pyis module-level startup code and this repo has no pluggable-worker fixture; the runtime worker-start is the validation.Related Issues or PRs
🤖 Generated with Claude Code