asyncio.gather behaves inconsistently when handling KeyboardInterruption/SystemExit

Bug report

Here is a minimal example:

import asyncio

e = KeyboardInterrupt  # or SystemExit


async def main_task():
    await asyncio.gather(
        sub_task(),
    )


async def sub_task():
    raise e


if __name__ == '__main__':
    try:
        asyncio.run(main_task())
    except e:
        print(f'Handle {e}')

This code handles the Interrupt normally as I expected.

Handle <class 'KeyboardInterrupt'>

Process finished with exit code 0

But when I add the asyncio.sleep(0) (can be replaced by other task, not important) into main_task's asyncio.gather

import asyncio

e = KeyboardInterrupt  # or SystemExit


async def main_task():
    await asyncio.gather(
        sub_task(),
        asyncio.sleep(0)
    )


async def sub_task():
    raise e


if __name__ == '__main__':
    try:
        asyncio.run(main_task())
    except e:
        print(f'Handle {e}')

There is an unexpected traceback print out which is really confusing 💦, this traceback indicates that there is
another KeyboardInterrupt raised.

Full traceback

Traceback (most recent call last):
  File "/Users/huanghuiling/PycharmProjects/Lighting-bilibili-download/tests/log_test.py", line 45, in <module>
    asyncio.run(main_task())
  File "/opt/homebrew/Caskroom/miniforge/base/envs/test9/lib/python3.9/asyncio/runners.py", line 47, in run
    _cancel_all_tasks(loop)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/test9/lib/python3.9/asyncio/runners.py", line 63, in _cancel_all_tasks
    loop.run_until_complete(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/test9/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete
    self.run_forever()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/test9/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
    self._run_once()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/test9/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once
    handle._run()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/test9/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/huanghuiling/PycharmProjects/Lighting-bilibili-download/tests/log_test.py", line 7, in main_task
    await asyncio.gather(
  File "/opt/homebrew/Caskroom/miniforge/base/envs/test9/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/opt/homebrew/Caskroom/miniforge/base/envs/test9/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete
    self.run_forever()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/test9/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
    self._run_once()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/test9/lib/python3.9/asyncio/base_events.py", line 1890, in _run_once
    handle._run()
  File "/opt/homebrew/Caskroom/miniforge/base/envs/test9/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/huanghuiling/PycharmProjects/Lighting-bilibili-download/tests/log_test.py", line 14, in sub_task
    raise e
KeyboardInterrupt

Process finished with exit code 0

So I go deeply into the asyncio.run, and write code (a simplified asyncio.run) below to figure out what happen.

import asyncio

e = KeyboardInterrupt  # or SystemExit


async def main_task():
    await asyncio.gather(
        sub_task(),
        asyncio.sleep(0)
    )


async def sub_task():
    raise e


if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    try:
        loop.run_until_complete(main_task())
    except e:
        print(f'Expected {e}')
    finally:
        try:
            tasks = asyncio.all_tasks(loop)
            for t in tasks:
                t.cancel()
            # ⬇️ this line will raise another KeyboardInterrupt which is unexpected ⬇️ 
            loop.run_until_complete(asyncio.gather(*tasks, return_exceptions=True))
        except e:
            print(f'Unexpected {e} !!!!')

This line will raise another KeyboardInterrupt which is unexpected.

loop.run_until_complete(asyncio.gather(*tasks, return_exceptions=True))

Expected <class 'KeyboardInterrupt'>
Unexpected <class 'KeyboardInterrupt'> !!!!

Process finished with exit code 0

Note that this line is used to cancel all tasks during gracefully shutdown (also in asyncio.run), and when I change
e to other error like IndexError(any BaseException), this code works fine without unexpected another exception.
I believe this is related to the asyncio treats SystemExit and KeyboardInterrupt in different way. For example
in events.py

def _run(self):
    try:
        self._context.run(self._callback, *self._args)
    except (SystemExit, KeyboardInterrupt):
        raise
    except BaseException as exc:
        cb = format_helpers._format_callback_source(
            self._callback, self._args)
        msg = f'Exception in callback {cb}'
        context = {
            'message': msg,
            'exception': exc,
            'handle': self,
        }
        if self._source_traceback:
            context['source_traceback'] = self._source_traceback
        self._loop.call_exception_handler(context)
    self = None  # Needed to break cycles when an exception occurs.

My question is:

Why asyncio.gather behaves inconsistently.
Is there any reason to treat KeyboardInterrupt differently, since the simplest way
to solve this bug is to handle it same as BaseException.

I think user would like to handle all error consistently during the running of a task whether
it's KeyboardInterrupt or BaseException.

Even asyncio treats them in different way (incase really necessary)

asyncio.gather(sub_task())

and

asyncio.gather(sub_task(), asyncio.sleep(0))

should behave consistently, so I think this is a bug in asyncio.

Your environment

CPython versions tested on: 3.8, 3.9, 3.10
Operating system and architecture: both macOS and windows

Can you test 3.11 and main branch as the signal handling was changed recently ?

@kumaraditya303

Can you test 3.11 and main branch as the signal handling was changed recently ?

@kumaraditya303 I have tested on docker 3.11.0b1-bullseye, but still got the same problem. May test main branch later.

update: tested on main branch still have the same problem

The traceback message appears at the end of the script after the last line of __name__ == __main__ block.
This is bout the main_taskcoroutine: Task exception was never retrieved

So I tried to cancel this current task when exception raises, updating your main_task coroutine as below:

async def main_task():
    try:
        r = await asyncio.gather(sub_task(), asyncio.sleep(1))

    except (SystemExit, KeyboardInterrupt): 
        # cancel current task
       asyncio.current_task().cancel()

        # give time to other tasks to run (including me)
        await asyncio.sleep(0)
        raise

    except:
        raise

I don't really figure out in detail why but now the traceback message does not appear.
I suppose that other tasks run updating their internal state. This is including the current task main_wait.
And may be some loop flags are updated too.

I wonder whether something is not missing in except (SystemExit, KeyboardInterrupt): part of the def _run(self)method ?

I'd like to understand too.

@YvesDup

@YvesDup In your case

import asyncio

e = KeyboardInterrupt  # or SystemExit


async def main_task():
    try:
        r = await asyncio.gather(sub_task(), asyncio.sleep(1))

    except (SystemExit, KeyboardInterrupt):
        # cancel current task
        asyncio.current_task().cancel()
        # give time to other tasks to run (including me)
        await asyncio.sleep(0)
        print('never reach here')
        raise
    except:
        print('')
        raise


async def sub_task():
    raise e


if __name__ == '__main__':
    try:
        asyncio.run(main_task())
    except e:
        print(f'Handle {e}')

There are two independent KeyboardInterrupt raised. One in sub_task and one in await asyncio.sleep(0).

print('never reach here')
raise

is nerver reached (test by debug). So two try except catch these two interrupt, no traceback anymore, but still very confusing why await asyncio.sleep(0) raised another interrupt🥲, or not in sleep but in event loop somewhere, I dont know.

asyncio.sleep(0) raised CancelErrorcause by asyncio.current_task().cancel() the line above
Sorry, bad solution and no more explanation.

asyncio.sleep(0) raised CancelErrorcause by asyncio.current_task().cancel() the line above Sorry, bad solution and no more explanation.

I'm pretty sure it's not a CancelError but a KeyboradInterrupt.

@HFrost0

Hi @HFrost0
I checked the documentation on asyncio.gather and there is nothing about the keyboardInterrupt exception. Before opening a PR, I think it would be a good idea to send a message to python-dev asking what the behavior of asyncio.gather is when a keyboardInterrupt is raised.

IMO there is also another gap in the documentation about what happens when two or more exceptions raise in tasks.

@YvesDup

Hi @YvesDup

Agree, but I'm a little bit busy right now and not familiar with python-dev. Could you send this message please?

I notice you already asked this question link, thank you

async def main_task():
    try:
        await asyncio.gather(
            sub_task(),
            asyncio.sleep(0)
        )
    except Exception as e:
        raise
   
    except BaseException as e:
        print(repr(e))

if you catch the exception in the main_task() coroutine, there is no more error with one or two tasks.

Any update for this?

I'd like to chip in, hope it's fine. With wait one gets rid of the unexpected exception (run the excerpt below, change use_gather to True for the old behavior), although to make the code run without the additional "Task exception was never retrieved" it gets a bit messy. One could also change the ALL_COMPLETED, which simulates the behavior of gather to FIRST_EXCEPTION and then the task creations and retrieving their exceptions all goes away, however, one would need to iterate and make sure pending tasks are finished, but that's another topic.

import asyncio

e = KeyboardInterrupt  # or SystemExit


async def main_task(use_gather=True):
    loop = asyncio.get_running_loop()

    tasks = [loop.create_task(sub_task()), loop.create_task(asyncio.sleep(0))]

    try:
        if use_gather:
            await asyncio.gather(*tasks)
        else:
            await asyncio.wait(tasks, return_when=asyncio.ALL_COMPLETED)
    except:
        for t in tasks:
            try:
                # Prevent "Task exception was never retrieved"
                t.exception()
            except:
                pass
        raise


async def sub_task():
    raise e


if __name__ == '__main__':
    try:
        asyncio.run(main_task(use_gather=False))
    except e:
        print(f'Handle {e}')

I also bumped into the same problem as the never reach here above, an example and the wait-based solution can be found in my gist.

I think except and finally blocks not running through is a serious issue which should be addressed and before it's solved the documentation should warn about this.

This would be a good issue for triagers to test the current behavior and document here.

I've also stumbled upon it the same or similar bug. In my case after SystemExit the IOLoop became broken and didn't allow other tasks to finish their work gracefully.

I was debugging that using relatively fresh version and found this block that already looks incorrect — the code "saves" the exception to "handle it later" (set_exception) and same time re-raises it right now. That exactly doubles the same exception in cases mentioned above.

That causes that issue with this part of code — the IOLoop and its Handles don't catch these two errors and just reraise them transparently all the way up to run_until_complete or run_forever or so.
These exceptions stops the IOLoop abruptly here and now.

Then in case of asyncio.run use, it tries to gracefully cancel all tasks. It cancels all the tasks and reruns the IOLoop to let them handle cancellation and exit. But here the copy of the original exception (the set_exception() branch) is re-reraised and immediately breaks the finalization loop as well.

I've tried to remove one super().set_exception(exc) or replace it with self.cancel() — these changes look logical to me. But it didn't work nicely, the task was stuck in cancelling state and the IOLoop was infinitely waiting for it.

That looks like another bug that was masked with set_exception() + raise hack.

I didn't use asyncio.gather nor asyncio.wait anywhere in my tests, these look irrelevant to the problem

There are my test snippets. If we have only one task in the IOLoop, it gracefully stops before crashing the loop, there is nothing to stop after it, everything is nice and clean:

import asyncio.runners
import asyncio.tasks

async def do_raise():
    try:
        print("Task raises")
        raise SystemExit

    finally:
        print("task is stopping")
        await asyncio.sleep(1)
        print("task has stopped")

async def main():
    try:
        await do_raise()
        await asyncio.Future()

    finally:
        print("main is stopping")
        await asyncio.sleep(1)
        print("main has stopped")

with asyncio.runners.Runner() as runner:
    try:
        runner.run(main())
    except SystemExit as e:
        print("`runner.run` was gracefully stopped by", repr(e))
        raise

When I've replaced:

- await do_raise()
+ task = asyncio.create_task(do_raise())

then the exception has stopped the do_raise normally, then main was cancelled gracefully. But the copy of the exception inside the do_raise: "Task exception was never retrieved".

When I've added a dummy handler:

  task = asyncio.create_task(do_raise())
+ task.add_done_callback(lambda f: f.result())

the do_raise task was stopped fine (as it happened before the exception reached the IOLoop) but then everything was messed up as described above:

main is stopping
Exception ignored while closing generator <coroutine object main at 0x7f3261ecb5c0>:
Traceback (most recent call last):
  File "async-base-exc.py", line 18, in main
    await asyncio.Future()
GeneratorExit

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "async-base-exc.py", line 22, in main
    await asyncio.sleep(1)
  File "Lib/asyncio/tasks.py", line 696, in sleep
    loop = events.get_running_loop()
RuntimeError: no running event loop
Task was destroyed but it is pending!
task: <Task cancelling name='Task-1' coro=<main() done, defined at async-base-exc.py:14> wait_for=<Future cancelled> cb=[gather.<locals>._done_callback() at Lib/asyncio/tasks.py:810]>

From your example, when you suggest:

When I've replaced:

- await do_raise()
+ task = asyncio.create_task(do_raise())

You have to wait for the task

task = asyncio.create_task(do_raise())
+ await task

@YvesDup

@YvesDup , sure, normally yes.

With this example I've showed that unawaited task doesn't crash the IOLoop:

then the exception has stopped the do_raise normally, then main was cancelled gracefully. But the copy of the exception inside the do_raise: "Task exception was never retrieved".

But of course this code is incorrect. But all correct cases cause the issue.

For some reason I haven't mentioned the most obvious case, but it also fails:

task = asyncio.create_task(do_raise())
await task

But of course this code is incorrect. But all correct cases cause the issue.

Please, would you provide a simple example that causes the issue as I am a bit lost.

Sure

Run "free-float" background task detached (not awaited) from main:

import asyncio.runners
import asyncio.tasks

async def do_raise():
    try:
        raise SystemExit

    finally:
        await asyncio.sleep(1)
        print("Task has gracefully stopped")

async def main():
    try:
        global task
        task = asyncio.create_task(do_raise())
        await asyncio.Future()

    finally:
        await asyncio.sleep(1)
        print("Main has gracefully stopped")

with asyncio.runners.Runner() as runner:
    try:
        runner.run(main())
    except SystemExit as e:
        print("`runner.run` was stopped by", repr(e))
        raise

logs:

Task has gracefully stopped
`runner.run` was stopped by SystemExit()
Main has gracefully stopped
Task exception was never retrieved
future: <Task finished name='Task-2' coro=<do_raise() done, defined at test.py> exception=SystemExit()>

✔️ Task is stopped gracefully
✔️ Main too
😬 Task is not awaited, bad practice
❓ IOLoop was stopped indirectly even by a free-floating task

Second case, detach task from main and add dummy callback:

import asyncio.runners
import asyncio.tasks

async def do_raise():
    try:
        raise SystemExit

    finally:
        await asyncio.sleep(1)
        print("Task has gracefully stopped")

async def main():
    try:
        global task
        task = asyncio.create_task(do_raise())
        task.add_done_callback(lambda f: f.result())
        await asyncio.Future()

    finally:
        await asyncio.sleep(1)
        print("Main has gracefully stopped")

with asyncio.runners.Runner() as runner:
    try:
        runner.run(main())
    except SystemExit as e:
        print("`runner.run` was stopped by", repr(e))
        raise

Logs:

Task has gracefully stopped
`runner.run` was stopped by SystemExit()
Task was destroyed but it is pending!
task: <Task cancelling name='Task-1' coro=<main() running at test.py> wait_for=<Future cancelled> cb=[gather.<locals>._done_callback() at Lib/asyncio/tasks.py:810]>
Exception ignored while closing generator <coroutine object main at 0x7ff063b0b5c0>:
RuntimeError: no running event loop

✔️ Task is stopped gracefully
❌ Main had no chances to shut down gracefully
❌ Main task was cancelled but the IOLoop was closed before that

These examples are maybe a bit esoteric — I've tried to reproduce that with the minimal amount of isolated tasks — 2 to eliminate gather from the equation.

In real code main gather-awaits task1 and task2 — no free floating, non-awaited tasks. And then task1 receives SystemExit and task2 has no chances to complete its job

Navigation Menu

Repository navigation

asyncio.gather behaves inconsistently when handling KeyboardInterruption/SystemExit #93122

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

asyncio.gather behaves inconsistently when handling KeyboardInterruption/SystemExit #93122

Description

Activity

kumaraditya303 commented on May 24, 2022

HFrost0 commented on May 24, 2022

YvesDup commented on May 24, 2022

HFrost0 commented on May 24, 2022

YvesDup commented on May 24, 2022

HFrost0 commented on May 24, 2022

YvesDup commented on Jun 3, 2022

HFrost0 commented on Jun 9, 2022

YvesDup commented on Jun 9, 2022

HFrost0 commented on Aug 19, 2022

tfarago commented on Sep 20, 2022

willingc commented on Jun 21, 2024

anton-ryzhov commented on Feb 26, 2026

anton-ryzhov commented on Feb 26, 2026

YvesDup commented on Feb 28, 2026

anton-ryzhov commented on Mar 2, 2026

YvesDup commented on Mar 10, 2026

anton-ryzhov commented on Mar 11, 2026

anton-ryzhov commented on Mar 11, 2026

anton-ryzhov commented on Mar 11, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions