gtest sharding breaks lit's xfail inputs #102264
Labels
Comments
|
Related to #56492 |
This was referenced Aug 7, 2024
JDevlieghere
pushed a commit
that referenced
this issue
Aug 16, 2024
…02986) When a test case inside of a gtest suite fails, we report a failure which causes the entire `ninja check-lldb` invocation to fail, even if the outer test case is marked as XFAIL - each test case result is reported as its own lit test run. This PR updates lit so it checks whether each test case's parent test suite is XFAIL before setting the status to FAIL. This is especially problematic because the failing tests can't manually be marked as XFAIL, due to #102264. Fixes #102265 ### Repro instructions 1. Modify any gtest test case to generate a failure. 2. Mark the outer lit test with XFAIL using either `--xfail-tests` flag or `LIT_XFAIL` env var. 3. Run the tests 4. Observe the lit test is XFAIL as expected, but the failed child test cases show up as separate failures. Co-authored-by: kendal <kendal@thebrowser.company>
kendalharland
added a commit
to kendalharland/llvm-project
that referenced
this issue
Sep 12, 2024
…vm#102986) When a test case inside of a gtest suite fails, we report a failure which causes the entire `ninja check-lldb` invocation to fail, even if the outer test case is marked as XFAIL - each test case result is reported as its own lit test run. This PR updates lit so it checks whether each test case's parent test suite is XFAIL before setting the status to FAIL. This is especially problematic because the failing tests can't manually be marked as XFAIL, due to llvm#102264. Fixes llvm#102265 ### Repro instructions 1. Modify any gtest test case to generate a failure. 2. Mark the outer lit test with XFAIL using either `--xfail-tests` flag or `LIT_XFAIL` env var. 3. Run the tests 4. Observe the lit test is XFAIL as expected, but the failed child test cases show up as separate failures. Co-authored-by: kendal <kendal@thebrowser.company> (cherry picked from commit 9f89d31)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Lit marks tests as XFAIL by generating a list of
discovered_testsand marking each of them as XFAIL if their relative filenames or fullnames appear in itsopts.xfailwhich is populated from either the--xfail-testsflag orLIT_XFAILenv var before the tests actually run. When gtest sharding is enabled, every test's name indiscovered_testsinitially contains a numeric suffix likeFooTest\1\2before the tests runs, so the test names don't match the names the User passes to the arguments above. Thegoogletestmodule will replace these numbers with the actual test suite and case names by reading them from gtest json files only after the tests are finished running. This means that it's impossible to mark a test as XFAIL using the above mechanisms when gtest sharding is enabled. One can only use the Python test decorators, or the shell testXFAIL:headers, or any of the other source-code based mechanisms.The text was updated successfully, but these errors were encountered: