contextlib.ExitStack tries to make exception handling work like it would with a bunch of nested with statements. This is difficult because the interpreter doesn't provide a mechanism for setting the active exception from Python code (relevant: #89524) so the __context__ of exceptions raised by context manager __exit__ methods winds up incorrect at first and must be fixed up after-the-fact.
Two years ago, the __context__ fixup was too liberal: it would set things as __context__ that the equivalent nested with blocks wouldn't. Then #88760 was filed, and #27089 was committed to fix it. Now the __context__ fixup is too conservative: it sometimes doesn't set things as context that the equivalent nested with blocks would.
The fundamental problem here is that ExitStack can't tell the difference between a __context__ that is None because the interpreter wanted to chain with the active exception but there wasn't one, and a __context__ that is None because no chaining should have occurred (e.g. because the context manager was reraising an exception from some other place in the program). When ExitStack.__exit__ runs with an active exception, it can look for that exception in the __context__ chain. When there isn't an active exception, matching the behavior of nested with blocks is a lot more complicated.
Solutions I could imagine:
try:
raise Exception("internal exception used for ExitStack __context__ fixup")
except Exception as frame_exc:
# now there's an active exception and the existing logic will work
- Do nothing, and accept the divergence between ExitStack and nested
with blocks.
(We definitely should not just revert #27089; the problems associated with too much __context__ are in practice far worse than the problems associated with too little. Most recently I ran into python-trio/trio#2577 on 3.8.)
Reproducer (chainer.py):
from contextlib import contextmanager, ExitStack
import traceback
class raise_on_exit:
def __init__(self, ty):
self._ty = ty
def __enter__(self):
pass
def __exit__(self, *exc):
raise self._ty()
def example_regular():
with raise_on_exit(KeyError):
with raise_on_exit(ValueError):
pass
def example_stacked():
with ExitStack() as stack:
stack.enter_context(raise_on_exit(KeyError))
stack.enter_context(raise_on_exit(ValueError))
try:
example_regular()
except Exception:
traceback.print_exc()
print("--------")
try:
example_stacked()
except Exception:
traceback.print_exc()
Output on 3.8.7 (does not have #27089): the ValueError is correctly left in __context__ in both cases
Traceback (most recent call last):
File "chainer.py", line 17, in example_regular
pass
File "chainer.py", line 12, in __exit__
raise self._ty()
ValueError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "chainer.py", line 25, in <module>
example_regular()
File "chainer.py", line 17, in example_regular
pass
File "chainer.py", line 12, in __exit__
raise self._ty()
KeyError
--------
Traceback (most recent call last):
File "/opt/hrt/hrtpy38/root/usr/lib/python3.8/contextlib.py", line 510, in __exit__
if cb(*exc_details):
File "chainer.py", line 12, in __exit__
raise self._ty()
ValueError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "chainer.py", line 32, in <module>
example_stacked()
File "chainer.py", line 22, in example_stacked
stack.enter_context(raise_on_exit(ValueError))
File "/usr/lib/python3.8/contextlib.py", line 525, in __exit__
raise exc_details[1]
File "/usr/lib/python3.8/contextlib.py", line 510, in __exit__
if cb(*exc_details):
File "chainer.py", line 12, in __exit__
raise self._ty()
KeyError
Output on 3.10.10 (has #27089): the ValueError context is lost when using ExitStack
Traceback (most recent call last):
File "[...]/chainer.py", line 16, in example_regular
with raise_on_exit(ValueError):
File "[...]/chainer.py", line 12, in __exit__
raise self._ty()
ValueError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "[...]/chainer.py", line 25, in <module>
example_regular()
File "[...]/chainer.py", line 15, in example_regular
with raise_on_exit(KeyError):
File "[...]/chainer.py", line 12, in __exit__
raise self._ty()
KeyError
--------
Traceback (most recent call last):
File "[...]/chainer.py", line 32, in <module>
example_stacked()
File "[...]/chainer.py", line 20, in example_stacked
with ExitStack() as stack:
File "/usr/lib/python3.10/contextlib.py", line 576, in __exit__
raise exc_details[1]
File "/usr/lib/python3.10/contextlib.py", line 561, in __exit__
if cb(*exc_details):
File "[...]/chainer.py", line 12, in __exit__
raise self._ty()
KeyError
contextlib.ExitStacktries to make exception handling work like it would with a bunch of nestedwithstatements. This is difficult because the interpreter doesn't provide a mechanism for setting the active exception from Python code (relevant: #89524) so the__context__of exceptions raised by context manager__exit__methods winds up incorrect at first and must be fixed up after-the-fact.Two years ago, the
__context__fixup was too liberal: it would set things as__context__that the equivalent nestedwithblocks wouldn't. Then #88760 was filed, and #27089 was committed to fix it. Now the__context__fixup is too conservative: it sometimes doesn't set things as context that the equivalent nestedwithblocks would.The fundamental problem here is that ExitStack can't tell the difference between a
__context__that is None because the interpreter wanted to chain with the active exception but there wasn't one, and a__context__that is None because no chaining should have occurred (e.g. because the context manager was reraising an exception from some other place in the program). WhenExitStack.__exit__runs with an active exception, it can look for that exception in the__context__chain. When there isn't an active exception, matching the behavior of nestedwithblocks is a lot more complicated.Solutions I could imagine:
ExitStack.__exit__within a construct like:withblocks.(We definitely should not just revert #27089; the problems associated with too much
__context__are in practice far worse than the problems associated with too little. Most recently I ran into python-trio/trio#2577 on 3.8.)Reproducer (
chainer.py):Output on 3.8.7 (does not have #27089): the ValueError is correctly left in
__context__in both casesOutput on 3.10.10 (has #27089): the ValueError context is lost when using ExitStack