asyncio.gather with a single task raises an unexpected CancelledError exception when SystemExit occurs in this task. #108549

Checklist

• I am confident this is a bug in CPython, not a bug in a third-party project
• I have searched the CPython issue tracker,
  and am confident this bug has not been reported before

CPython versions tested on:

3.8, 3.9, 3.10, 3.11, 3.12

Operating systems tested on:

Linux, macOS

Description:

Occasionaly since last summer, I am working on this issue #93122 to figure out why behaviors are different in asyncio.gather when there is at least one task passed to this function, and when one of them, raises an SystemExit or KeybordInterrupt.

And I found odd behavior when there is only one task that raises SystemExit (or KeybordInterrupt), as below:

import asyncio

async def sub_task():
    raise SystemExit

async def asyncio_gather():
    try:
        await asyncio.gather(
            sub_task(),
        )

    except BaseException as ee:
        print(f'{ee = }, id={hex(id(ee))}')
        raise ee

if __name__ == '__main__':
    try:
        asyncio.run(asyncio_gather())
    except BaseException as ee:
        print(f'END {ee = }, id={hex(id(ee))}')

Result is:

ee = CancelledError(), id=0x102c01e40
END ee = SystemExit(), id=0x102c01d80

Raising CancelledError exception in asyncio_gather coroutine is really unexpected. IMO it should be SystemExit. Just to check, I created a new example with a TaskGroup where the same sub_task()coroutine is created in.

import asyncio

async def sub_task():
    raise SystemExit

async def asyncio_taskgroup():
    try:
        async with asyncio.TaskGroup() as tg:
            tg.create_task(sub_task())

    except BaseException as ee:
        print(f'{ee = }, id={hex(id(ee))}')
        raise ee

if __name__ == '__main__':
    try:
        asyncio.run(asyncio_taskgroup())
    except BaseException as ee:
        print(f'END {ee = }, id={hex(id(ee))}') 

Result is:

ee = SystemExit(), id=0x103174520
END ee = SystemExit(), id=0x103174520
Task exception was never retrieved
future: <Task finished name='Task-1' coro=<asyncio_taskgroup() done, defined at /Users/yves/Desktop/Bugs/test_ghxxxxx.py:31> exception=SystemExit()>
Traceback (most recent call last):
...
...

And to complete the problem analysis, from the first example, when asyncio.gather is called with 2 tasks as await asyncio.gather(sub_task(), asyncio.sleep(0.5)), SystemExit is raised, not CancelledError. It seems really inconsistent to raise a CancelledError exception.

Analysis:

I tried to understand why CancelledError is raised in the first example.

When the sub_task coroutine raises systemExit, this exception is propagated up to the RunnerContext Manager created in asyncio.run and finally treated in the __exit__ method of this CM.

Through the Runner.close method, there is a call to cancel_all_tasks. In this function, only one task is pending and candidate to cancel: asyncio_gather. The sub_task coroutine which raised SystemExit is finsihed. This part is processing in the call to asyncio.run_until_complete(asyncio.gather(*to_cancel, return_exceptions=True)).

At this point, asyncio_gather task waits for its GatheringFuture future ending. Later in the processing of event loop, two callbacks are executed:

• gather.<locals>._done_callback on the pending GatheringFuture future, setting its attribut _exception to SystemExit as result of sub_task coroutine.
• Task.__wakeup on asyncio_gather pending task, where a call to future.result() cause an SystemExit exception that is going to pass as argument to self.__step.

In this _step method, exc is changed to CancelledError, and passed to self.__step_run_and_handle_result(exc). That triggers the CancelledError exception.

Question:

Is this really an issue ? Could a cancelling task be raised CancelledError when its _exception attribute is already set to SystemExit or KeybordInterrupt ?
Maybe we need to discuss this before validating the problem and finding a fix?

Possible fix (if this is really an issue):

In case of issue/feature, a fix is to change the behavior of Task.__step as follow:

when a task is 'to cancel', the exc variable could be overwrite to CancelledError, only if exc is not already an instance of CancelledError, SystemExit or KeybordInterrupt, as below:

index 8d5bde09ea..07f916dc6d 100644
--- a/Lib/asyncio/tasks.py
+++ b/Lib/asyncio/tasks.py
@@ -283,7 +283,12 @@ def __step(self, exc=None):
             raise exceptions.InvalidStateError(
                 f'_step(): already done: {self!r}, {exc!r}')
         if self._must_cancel:
-            if not isinstance(exc, exceptions.CancelledError):
+            if not isinstance(exc, (exceptions.CancelledError,
+                                   SystemExit, KeyboardInterrupt
+                                   )):
                 exc = self._make_cancelled_error()
             self._must_cancel = False
         self._fut_waiter = None

if we validate this feature, the fix should also be made to the asyncio module: _asynciomodule.c to validate test.test_asyncio.

Linked PRs

• gh-108549: In asyncio, do not cancel the current task in Task.__step when a SystemExit exception was just raised  #156309