Add pytest as a testing framework for Python

[suic86]

Feature

pytest test framework support for Python 2 and 3.

Rationale

Issues with the current framework(s)

The currently available test frameworks involve a lot of boilerplate. This is true first and foremost about the most widely used test.assert_equals.

The code using this test framework has very high noise to signal ratio and it's hard to parse visually as test.assert_equals isn't syntax-highlighted and there's no significantly visible separation (it's a single comma) between actual and expected value.* Also, this framework is custom and can't be used outside of CW.

I'll use the following example (taken from this discussion) in which 37.5% is boilerplate, to support my point

# Example:

test.assert_equals(changed((0, 0, 1, 1)), 2)
test.assert_equals(changed((10, 10, 10, 10, 99, 99)), 4)
test.assert_equals(changed(([], [], [], [], (), ())), 4)
test.assert_equals(changed(("this", "that")), 1)
test.assert_equals(changed(()), None)
test.assert_equals(changed((1,)), None)
test.assert_equals(changed((1, 1, 1, 1, 1)), None)

# Analysis

# test.assert_equals(changed((1, 1, 1, 1, 1)), None)
# \----------------/
#         noise

code_length = len(of_your_code_including_line_breaks) == 336
test_clutter = len(all_of_test_assert_equals_parens_excluded) == 126
test_clutter/code_length == 0.375 # (i.e. 37.5% is noise) 

A comparison with the same tests written in Groovy using junit demonstrate how cluttered this Python code is. The two main differences I want to point out:

1. The code isn't that much shorter but it is easier to parse visually as assert is highlighted and there's a highlighted separation (==) between actual and expected values (the highlighting on CW is even better in this respect).
2. The same code can be used outside CW.

@Test
void "Example Tests"() {
    assert changed([0, 0, 1, 1]) == 2
    assert changed([10, 10, 10, 10, 99, 99]) == 4
    assert changed([[], [], [], [], (), ()]) == 4
    assert changed(["this", "that"]) == 1
    assert changed([]) == null
    assert changed([1,]) == null
    assert changed([1, 1, 1, 1, 1]) == null
}

Proposed Solution: pytest

pytest test framework offers user experience similar to Groovy with little clutter and detailed info about the failing tests. As a demonstration: The Groovy code above can be used in pytest with minimal changes and the code (as it is) can be used outside CW too.

def test_examples():
    assert changed([0, 0, 1, 1]) == 2
    assert changed([10, 10, 10, 10, 99, 99]) == 4
    assert changed([[], [], [], [], (), ()]) == 4
    assert changed(["this", "that"]) == 1
    assert changed([]) == None
    assert changed([1,]) == None
    assert changed([1, 1, 1, 1, 1]) == None

Notes

* IMHO the use of this framework can be by no means to be considered as best practice.

[kazk]

Codewars can only support one test framework per language at the moment because of how the database is set up. So we won't be able to add another test framework until we do the rewrite which I'm planning to start working on after my vacation.

I personally don't like having to maintain custom frameworks when there're existing ones we can use (e.g., JS/Python/Ruby, etc.), so I'd like to deprecate them in the future for new contents. We still need to keep them for a while for existing contents though.

For Python, I like pytest too and if I remember correctly, it supports custom reporter so it should be possible to add.

[suic86]

@kazk: Thanks for quick reply. I'll wait patiently. :) Also, I completely agree with your stance on custom frameworks. Enjoy your vacation and good luck with the rewrite.

[suic86]

@kazk: I came back to this after some time. I guess there's been no progress on this and the single test framework per language limitation is still there. One possible option would be to emulate the existing framework with pytest (i.e. the current framework would be an alternative syntax for pytest. Is this a sensible idea according to you?

[kazk]

No progress on Codewars part.

Have you looked into if pytest supports custom reporters?

One possible option would be to emulate the existing framework with pytest (i.e. the current framework would be an alternative syntax for pytest. Is this a sensible idea according to you?

I'll consider it if it's fully backwards compatible and also depending on the amount of effort required.

[Blind4Basics]

@suic86 : but what is the feeback for the user in case a test fails? Seems to me you're comparing assert_equals with an equivalent of expect. Not a good idea, imo.

[kazk]

@Blind4Basics pytest actually produces nice failure message from simple assertions.

Similar to power asserts from Groovy. So

x = 1
assert x == 2

Fails with something like

Assertion failed:
  assert x == 2
         |  |
         1  false

It can produce more useful output depending on the test case.

[kazk]

For reporting pytest results, we should be able to make custom reporter similar to https://github.com/pchomik/pytest-spec

I haven't looked into if we can simulate Codewars test framework with pytest so please let me know if you have some proposals. I'd be surprised if it turns out possible because codewars_test has lots of quirks.