Run all case *without* a tag

[saroad2]

Hello!

First and foremost: this library is becoming more and more beneficial to my testing suites in all of my repositories. It makes the code much more granular and makes me focus my effort on cases instead of behaviors. Thank you for that!

One thing that I'm currently missing is to use parametrize_with_cases without a tag. For example, Assuming that I have a lot of cases, and I want to mark a few of them as "failure" tests. I would like to have the ability to run a test only on cases without the "failure" tag.

Currently, parametrize_with_cases supports the has_tag parameter. I think it should be clever to change that name of the parameter to tag and allow adding a prefix that indicates that we do not want to use a tag. Here are some code examples:

@parametrize_with_cases(tag="tag1")
def test_one()
   """Run with all cases that contain 'tag1'."""

@parametrize_with_cases(tag="!tag1")
def test_two()
   """Run with all cases that does not contain 'tag1'."""

@parametrize_with_cases(tag=["tag1", "!tag2"])
def test_two()
   """Run with all cases that contain 'tag1' and not containing 'tag2'."""

What do you think?

[smarie]

Thanks so much for the positive feedback @saroad2 ! I am very glad to read that this library reaches its goal: spending more time on the test cases.

Concerning your need, I am more and more convinced that putting the information directly in the case id (case function name) is easier in term of API complexity and code readibility, this is why I introduced glob and filter in addition to tag. Note that filter can work both on the id, or on everything stored in case_fun._pytestcase (for example the tags).

This is how you would do with filter, if the tag to not look for should be in the id (not a tag)

@parametrize_with_cases('a', filter=lambda cf: "tag1" not in cf._pytestcase.id)
def test_foo(a):
    pass

of course you can write it once and for all by saving the filter somewhere:

def not_tag_1(cf):
    return "tag1" not in cf._pytestcase.id

@parametrize_with_cases('a', filter=not_tag_1)
def test_foo(a):
    pass

Details in the documentation : https://smarie.github.io/python-pytest-cases/#filters-and-tags

An alternative could be to extend the glob-like syntax, but it only works on the case id, not the tags.

[saroad2]

Maybe pytest-case can provide some sort of "default filters" such as contains_tag and not_conatains_tag. In my example:

@parametrize_with_cases(filter=contains_tag("tag1"))
def test_one()
   """Run with all cases that contain 'tag1'."""

@parametrize_with_cases(filter=not_contains_tag("tag1"))
def test_two()
   """Run with all cases that does not contain 'tag1'."""

@parametrize_with_cases(filter=lambda cf: contains_tag("tag1") and not_contains_tag("tag2"))
def test_two()
   """Run with all cases that contain 'tag1' and not containing 'tag2'."""

How about that?

[smarie]

This is a good idea. we can even make it more pythonic by providing a function creating a function. so filter=contains_tag("tag1") directly. I would need to think about it a bit longer to confirm that this would actually be better than letting users define this shortcut themselves.

May I ask, do you actually prefer tags (legacy, set with @case(tags=...) over keywords directly in the case ids (new since v2) ? If so, is this for a particular use case where keyword in ids would be cumbersome ? I would like to understand better then.

[saroad2]

I prefer tags because it allows me to keep the case name specific and compact. I like that I don't have to add the @case decorator in order for pytest-case to know it's a case. It makes everything much more intuitive.

But, if I can suggest another API for the tags, how about that:

@case_tag("tag1")
def case_with_tag1():
   """This case now tagged with 'tag1'."""

@case_tag("tag2")
def case_with_tag2():
   """This case now tagged with 'tag2'."""

@case_tag("tag1", "tag2")
def case_with_multiple_tags():
   """This case now tagged with both 'tag1' and 'tag2'."""

def case_without_tag():
   """This case doesn't have tags."""

In that way, we get the cases by searching for functions starting with case_, but we can tag some of them if we want to.

[smarie]

So basically you were preferring the api from v1 ? https://github.com/smarie/python-pytest-cases/blob/1.17.0/docs/api_reference.md#case_tags

I made it disappear because I thought that it was overkill to have a dedicated decorator "just for tags" :)
I'm open to bring it back of course

[saroad2]

I don't have a strong opinion either way. Just a suggestion :)