Improve reporting when multiple teardowns raise an exception

[bluetech]

What's the problem this feature will solve?

This is relatively minor but good to have an issue for it.

pytest nodes (meaning tests, and collectors like class, module, session) can have multiple finalizers attached to them. These can come from the builtin teardown function of the node, and from finalizers added through the addfinalizer API. When the relevant node needs to be torn down (or a set of nodes -- for example when moving from module1/test1 to module2/test2, both module1 and test1 are torn down), pytest runs all of these finalizers.

Now a situation can occur where more than one of these finalizers raises. What pytest does currently is only report the first exception:

pytest/src/_pytest/runner.py

Lines 446 to 459 in bbd22e1

	def _teardown_towards(self, needed_collectors) -> None:
	exc = None
	while self.stack:
	if self.stack == needed_collectors[: len(self.stack)]:
	break
	try:
	self._pop_and_teardown()
	except TEST_OUTCOME as e:
	# XXX Only first exception will be seen by user,
	# ideally all should be reported.
	if exc is None:
	exc = e
	if exc:
	raise exc

Therefore the user doesn't get a full picture of everything that failed.

Describe the solution you'd like

It would be good to display information on all of the exceptions.

This situation comes up in various other places. For example it happens in the trio concurrency library with its "nursery" concept, also in other languages like C# async/await support. The solution there is to have an "aggregate exception", which is an exception which holds a list of other exceptions.

Trio MultiError, MultiError v2 discussion

C# AggregateException

JS AggregateError

Looking at the Trio discussion, it looks like something like this might also be added to Python proper, at least it's actively being worked on: https://github.com/python/exceptiongroups

The idea is to add some aggregate exception implementation, keep it internal, make the teardown code propagate it, and then pytest will handle it specially by displaying all of the errors in some nice formatted way.

Alternative Solutions

We can keep reporting just the first exception. We haven't really had any reports about this issue that I know of. Also for pytest it is easy enough to just run the test again and fix the exceptions one by one, unlike a trio situation where transient potentially important information would get lost.

We can wait until Python itself figures it out and switch to it then. If Python itself adds it, then we'll probably need to handle it for user code any way.

[RonnyPfannschmidt]

Can't we just have a a subtest style reporting for both setups and teardoens?

[bluetech]

I am not familiar with subtest style reporting, can you describe it?

[RonnyPfannschmidt]

basically https://pypi.org/project/pytest-subtests/ , sorry for not providing the reference