Skip to content

fix(canvas): fire task lifecycle hooks for map/starmap items - #10880

Open
Sanjays2402 wants to merge 1 commit into
celery:mainfrom
Sanjays2402:fix-map-starmap-lifecycle-hooks-7585
Open

Sanjays2402 wants to merge 1 commit into
celery:mainfrom
Sanjays2402:fix-map-starmap-lifecycle-hooks-7585

Conversation

@Sanjays2402

@Sanjays2402 Sanjays2402 commented Oct 18, 2026 •

Copy link
Copy Markdown

Description

Fixes #7585
Fixes #7586

The celery.map and celery.starmap builtin tasks ran each item by calling the task instance directly (task(item) / task(*item)), which skips the task trace completely. That means before_start, on_success, on_failure and after_return never fired for mapped items, and task signals were never sent, even though all of them fire for the same task executed on its own. chunks is affected too since it fans out through xstarmap.

The fix routes every item through Task.apply() so each one gets a full trace. .get(disable_sync_subtasks=False) is safe here: the EagerResult is already computed, so it never blocks, and passing False keeps this working inside real workers where the join guard is on. Exceptions still propagate out of the map task exactly as before, and now on_failure fires for them.

Repro (before the fix, map apply hooks was []):

from celery import Celery, Task

app = Celery('x', broker='memory://', backend='cache+memory://')
app.conf.task_always_eager = True

calls = []

class MyTask(Task):
    def before_start(self, task_id, args, kwargs):
        calls.append(('before_start', args))
    def after_return(self, status, retval, task_id, args, kwargs, einfo):
        calls.append(('after_return', status))

@app.task(base=MyTask)
def add1(x):
    return x + 1

add1.map([1, 2, 3]).apply()
print(calls)  # now: before_start/after_return per item

Tests: added regression tests in t/unit/app/test_builtins.py covering hook order per mapped item for map and starmap, plus on_failure firing when an item raises. Ran t/unit/app/test_builtins.py, t/unit/tasks/test_canvas.py, t/unit/app/test_trace.py, t/unit/tasks/test_result.py, t/unit/tasks/test_tasks.py: 638 passed, 1 skipped, 1 xfailed, all green.

Also fixes #7586: autoretry_for was ignored for mapped items because the retry logic lives in the trace path. Verified: a task with autoretry_for=(ValueError,) and max_retries=3 raised after a single attempt through map before this change; now each item retries independently.

The celery.map and celery.starmap builtin tasks executed each item by
calling the task directly, bypassing the task trace entirely. Task
lifecycle hooks (before_start/on_success/on_failure/after_return) and
task signals never fired for mapped items, unlike single tasks.

Route every item through Task.apply() so the trace runs per item.
get(disable_sync_subtasks=False) is safe here because the EagerResult
is already computed and never blocks.

Fixes celery#7585
@auvipy
auvipy self-requested a review October 18, 2026 03:57
@codecov

codecov Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 90.57%. Comparing base (defd4d5) to head (faaec5c).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10880      +/-   ##
==========================================
- Coverage   90.58%   90.57%   -0.01%     
==========================================
  Files         153      153              
  Lines       21447    21447              
  Branches     2633     2633              
==========================================
- Hits        19427    19426       -1     
  Misses       1707     1707              
- Partials      313      314       +1     
Flag Coverage Δ
unittests 90.55% <100.00%> (-0.01%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

autoretry ignored on map/chunk calls map/chunks ignore task inheritance class before_start/after_return/on_success/on_failure

1 participant