Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Print the SUBCASE path when an assert fails in the TEST_CASE body #125

Closed
obfuscated opened this issue Apr 12, 2018 · 3 comments
Closed

Print the SUBCASE path when an assert fails in the TEST_CASE body #125

obfuscated opened this issue Apr 12, 2018 · 3 comments

Comments

@obfuscated
Copy link

obfuscated commented Apr 12, 2018 •
edited by onqtam
Loading

Simple test:

TEST_CASE("better message needed")
{
    CHECK(false);
    SUBCASE("subcase 1")
    {
        CHECK(false);
    }
    CHECK(false);
}

The result is

===============================================================================
test.cpp(28)
TEST CASE:  better message needed
test.cpp:30: error:!
  CHECK( false )
with expansion:
  CHECK( false )
===============================================================================
test.cpp(28)
TEST CASE:  better message needed
  subcase 1
test.cpp:33: error:!
  CHECK( false )
with expansion:
  CHECK( false )
===============================================================================
test.cpp(28)
TEST CASE:  better message needed
test.cpp:35: error:!
  CHECK( false )
with expansion:
  CHECK( false )
===============================================================================

The problem is that in the current version it is impossible to find out (without using a debugger or printing) which test failed if an assert in the body fails. Not user friendly.

  • doctest version: fec6e4a
  • Operating System: Linux
  • Compiler+version: GCC 5.4
@onqtam
Copy link
Member

onqtam commented May 23, 2018 •
edited
Loading

actually currently you do get a "stack of currently entered subcases" - see that in the second assert failure you have "subcase 1" printed after the test case name

But I do think this can be improved a lot... And it would be nice to be able to see not just which subcases are currently entered, but also which ones were already entered and exited in the current test case entrance:

TEST_CASE("data-driven") {
    int a;
    SUBCASE("subcase 1")
        a = 5;
    SUBCASE("subcase 2")
        a = 42;
    CHECK(a == 6);
}

here it would be cool to see which subcase was entered (and already exited) for when the asserts gets triggered. This is in the roadmap.

@blastrock blastrock mentioned this issue May 28, 2018
Closed
@onqtam
Copy link
Member

onqtam commented May 29, 2018 •
edited
Loading

also this seems to be related: catchorg/Catch2#734
Also this by the same user: #215

@onqtam onqtam mentioned this issue Mar 27, 2019
Closed
jktjkt added a commit to jktjkt/doctest that referenced this issue Apr 30, 2020
@jktjkt
Print subcase stack upon a subcase entry
f0d2eda 
This is largely a hack to give us at least *something* for data driven
testing (doctest#215). In that pattern, it's common to structure code in a way
that first "generates the data", and only then executes the actual test.
It can look like this:

  TEST_CASE("parsing stuff") {
    std::string input;
    Something expected;

    DOCTEST_SUBCASE("single-line") {
      DOCTEST_SUBCASE("empty") {}
      DOCTEST_SUBCASE("trivial") {
        input = "create foo";
	expected = {OP_CREATE, {"foo",}};
      }
      DOCTEST_SUBCASE("two args") {
        input = "create blah bar";
	expected = {OP_CREATE, {"blah", "bar",}};
      }
    }
    DOCTEST_SUBCASE("multi-line") {
      DOCTEST_SUBCASE("trailing whitespace") {
        input = "create foo\n\n";
	expected = {OP_CREATE, {"foo,}};
      }
      DOCTEST_SUBCASE("one word per line, two words") {
        input = "create\nfoo";
	expected = {OP_CREATE, {"foo",}};
      }
    }

    REQUIRE(parse(input) == expected);
  }

The important part is that once we're executing actual tests, the
subsections will have been exited already, so the usual ways of showing
the subcase stack can be no longer applied.

In Catch2, there was a patch which tried to track the deepest subcase
stack so far. Let's try to implement something very simple, just
printing the current state of the stack once it is entered. It does not
tie into any error handling, so the output will be always there, and it
also probably does not support any fancy filtering when skipping
subcases. However, it's something which at least shows what is going on
when an error gets reported.

Cc: doctest#215
Cc: doctest#125
jktjkt added a commit to jktjkt/doctest that referenced this issue Apr 30, 2020
@jktjkt
Print subcase stack upon a subcase entry
300ed25 
This is largely a hack to give us at least *something* for data driven
testing (doctest#215). In that pattern, it's common to structure code in a way
that first "generates the data", and only then executes the actual test.
It can look like this:

  TEST_CASE("parsing stuff") {
    std::string input;
    Something expected;

    DOCTEST_SUBCASE("single-line") {
      DOCTEST_SUBCASE("empty") {}
      DOCTEST_SUBCASE("trivial") {
        input = "create foo";
	expected = {OP_CREATE, {"foo",}};
      }
      DOCTEST_SUBCASE("two args") {
        input = "create blah bar";
	expected = {OP_CREATE, {"blah", "bar",}};
      }
    }
    DOCTEST_SUBCASE("multi-line") {
      DOCTEST_SUBCASE("trailing whitespace") {
        input = "create foo\n\n";
	expected = {OP_CREATE, {"foo,}};
      }
      DOCTEST_SUBCASE("one word per line, two words") {
        input = "create\nfoo";
	expected = {OP_CREATE, {"foo",}};
      }
    }

    REQUIRE(parse(input) == expected);
  }

The important part is that once we're executing actual tests, the
subsections will have been exited already, so the usual ways of showing
the subcase stack can be no longer applied.

In Catch2, there was a patch which tried to track the deepest subcase
stack so far. Let's try to implement something very simple, just
printing the current state of the stack once it is entered. It does not
tie into any error handling, so the output will be always there, and it
also probably does not support any fancy filtering when skipping
subcases. However, it's something which at least shows what is going on
when an error gets reported.

Cc: doctest#215
Cc: doctest#125
@jktjkt jktjkt mentioned this issue Apr 30, 2020
Closed
onqtam added a commit that referenced this issue May 17, 2020
@onqtam
fixed #215 fixed #125 relates #364 - printing the deepest subcase sta…
1a65a92 
…ck reached - even if at the point of failure of an assertion the stack has become smaller - that way users can determine the exact path of data initialization in previous subcases before the failing assert.
@onqtam
Copy link
Member

onqtam commented May 17, 2020

done - for a program like this:

TEST_CASE("data-driven") {
    int a;
    SUBCASE("subcase 1")
        a = 5;
    SUBCASE("subcase 2")
        a = 42;
    CHECK(a == 6);
}

the output will be something like this:

===============================================================================
E:\doctest\scripts\playground\test.cpp:61:
TEST CASE:  data-driven

DEEPEST SUBCASE STACK REACHED (DIFFERENT FROM THE CURRENT ONE):
  subcase 1

E:\doctest\scripts\playground\test.cpp:67: ERROR: CHECK( a == 6 ) is NOT correct
!
  values: CHECK( 5 == 6 )

===============================================================================
E:\doctest\scripts\playground\test.cpp:61:
TEST CASE:  data-driven

DEEPEST SUBCASE STACK REACHED (DIFFERENT FROM THE CURRENT ONE):
  subcase 2

E:\doctest\scripts\playground\test.cpp:67: ERROR: CHECK( a == 6 ) is NOT correct
!
  values: CHECK( 42 == 6 )

that message will be printed only if the deepest subcase stack is different from the current one. Also the current subcase stack will be printed below the TEST CASE: as before.

@onqtam onqtam closed this as completed May 17, 2020
onqtam added a commit that referenced this issue May 17, 2020
@onqtam
fixed #215 fixed #125 relates #364 - printing the deepest subcase sta…
72bf24e 
…ck reached - even if at the point of failure of an assertion the stack has become smaller - that way users can determine the exact path of data initialization in previous subcases before the failing assert.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@obfuscated @onqtam