Test that unexpectedly passes doesn't produce exit code

[pytestbot]

Originally reported by: Jack Riches (BitBucket: jriches, GitHub: jriches)

---

If test is marked xfail, but the test passes, there should be a non-zero exit code (e.g. for CI to flag a warning).

I believe this once existed, as documented by the release notes for 2.0.0:

“xpass” (unexpected pass) tests don’t cause exitcode!=0

---

• Bitbucket: https://bitbucket.org/pytest-dev/pytest/issue/259

[pytestbot]

Original comment by Ronny Pfannschmidt (BitBucket: RonnyPfannschmidt, GitHub: RonnyPfannschmidt):

---

seems i introduced that between 1.3 and 2.0 - i didnt document it extensively enough,
so i'll check again in detail how it came up

[dariuspbom]

I also raised this in #716

See #814 for the discussion about flaky tests.

[nicoddemus]

I'm closing this as a duplicate of the ongoing discussion at #814, thanks @dariuspbom for bringing this to attention.