Failing a test from a finalizer within a plugin

[toddsifleet]

Hi,

I am working on an issue from the mocking library doubles.

The doubles library sets and validates expectations (e.g. expect(requests).post(...)), in order to verify all expectations have been met for a particular test we run finalizer code to check (and teardown) each expectation (code).

With our current code, the doubles.verify method may raise an AssertionError (previously it called pytest.fail (change)). Unfortunately our implementation using a finalizer causes the test to be reported as an error not a failure.

I think this could be one of a few issues:

1. A bug/mistake in how the doubles plugin is implemented.
2. A bug/mistake in how pytest handles finalizer's failures/AssertionErrors.
3. Neither -- pytest intentionally doesn't allow failing tests from finalizers.

If this is a bug/mistake in doubles code do you have any ideas about how we could update our pytest plugin to report these AssertionErrors as failures instead of errors?

Thanks,

Todd

[pytestbot]

GitMate.io thinks possibly related issues are #2305 (Sandboxed test execution fails when any plugin is installed), #2040 (isfile check fails within pytest), #747 (flakes test fail), #1777 (sigalrm fails test), and #662 (fail on zero tests found).

[RonnyPfannschmidt]

pytest itself considers a faiure in setup/teardown an actual error

this can be elevated by a own pytest_report_teststatus hook on order to override what

pytest/_pytest/runner.py

Lines 144 to 152 in 015626c

	def pytest_report_teststatus(report):
	if report.when in ("setup", "teardown"):
	if report.failed:
	# category, shortletter, verbose-word
	return "error", "E", "ERROR"
	elif report.skipped:
	return "skipped", "s", "SKIPPED"
	else:
	return "", "", ""

and

pytest/_pytest/terminal.py

Lines 123 to 132 in ed118d7

	def pytest_report_teststatus(report):
	if report.passed:
	letter = "."
	elif report.skipped:
	letter = "s"
	elif report.failed:
	letter = "F"
	if report.when != "call":
	letter = "f"
	return report.outcome, letter, report.outcome.upper()

do - i believe a tryfirst hook that integrates with your framework should suffice

another alternative is to have the assertion happen within the test call hook

[toddsifleet]

@RonnyPfannschmidt Thanks for the help!

FWIW here is my proposed fix: https://github.com/uber/doubles/pull/131/files

[RonnyPfannschmidt]

@toddsifleet thanks for the link, i believe your question is resolved, please close the issue if that's the case