How to handle MultiErrors containing pytest magic exceptions?

[oremanj]

If you say pytest.skip() inside the body of a test, and also one of your async fixtures has spawned a task that fails, the result of your test will be a MultiError containing the skip exception and the fixture task failure. pytest doesn't interpret this as a skip. The same thing applies to xfail() and so on. How should we handle these? One option would be to let the pytest magic exception propagate with the rest of the MultiError as its __context__.

[njsmith]

idk, pytest doesn't interpret this as a skip either:

@pytest.fixture
def broken():
    raise RuntimeError

def test_foo(broken):
    pytest.skip()

I feel like for a testing tool, when weird things happen, it's probably better to err on the side of dumping a traceback?

[Zac-HD]

I wrote a bit about this in pytest-dev/pytest#9680, and concluded that just raising the whole ExceptionGroup and letting the test fail was the safest solution, since we don't want to hide actual errors from the user. There's one case where that doesn't work though: the new strict_exception_groups option means that carefully raising a Skipped or XFailed exception will nonetheless end up wrapped in a group¹.

I therefore propose that exception groups with a single leaf exception which is either Skipped or XFailed should be unwrapped and the original group set as __cause__, and will send a PR for that and plus the prerequisite #128 soon.

Footnotes

1. if you try out trio.run = partial(trio.run, strict_exception_groups=True); otherwise I admit it's a more distant concern 😉 ↩