Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Lit's XFAIL inputs can't handle nested test cases in unit tests #102265

Closed
kendalharland opened this issue Aug 7, 2024 · 1 comment · Fixed by #102986
Closed

Lit's XFAIL inputs can't handle nested test cases in unit tests #102265

kendalharland opened this issue Aug 7, 2024 · 1 comment · Fixed by #102986
Labels
llvm-lit

Comments

@kendalharland
Copy link
Contributor

Unit tests such as this one from the downstream swiftlang/llvm-project repo can't be marked as XFAIL using lit's --xfail flag and LIT_XFAIL env var.

If these tests are run, the final lit summary output says that these tests failed:

lldb-unit :: Symbol/./SymbolTests.exe
lldb-unit :: Symbol/ClangArgs/UniquingCollisionWithAddedFlags
lldb-unit :: Symbol/ClangArgs/UniquingCollisionWithExistingFlags
lldb-unit :: Symbol/TestSwiftASTContext/ApplyWorkingDir
lldb-unit :: Symbol/TestSwiftASTContext/PluginPath

These are all the same test: The actual test is lldb-unit :: Symbol/./SymbolTests.exe and it can be marked as XFAIL using the CLI flag or env var, because it exists by the time lit attempts to mark the list of discovered tests as XFAIL. But the last four are test cases inside SymbolTests.exe and lit doesn't know about them until after SymbolTests.exe runs and fails, at which point the googletest module reads them from the gtest json file and registers them as failures.
@kendalharland
Copy link
Contributor Author

kendalharland commented Aug 7, 2024 •
edited
Loading

Possibly related to #56492 given that this test is using gtest.

This was referenced Aug 7, 2024
Merged
Merged
kendalharland added a commit to kendalharland/llvm-project that referenced this issue Sep 12, 2024
@kendalharland
[lldb][test] Mark gtest cases as XFAIL if the test suite is XFAIL (ll…
4abee04 
…vm#102986)

When a test case inside of a gtest suite fails, we report a failure
which causes the entire `ninja check-lldb` invocation to fail, even if
the outer test case is marked as XFAIL - each test case result is
reported as its own lit test run. This PR updates lit so it checks
whether each test case's parent test suite is XFAIL before setting the
status to FAIL.

This is especially problematic because the failing tests can't manually
be marked as XFAIL, due to
llvm#102264.

Fixes llvm#102265

### Repro instructions

1. Modify any gtest test case to generate a failure.
2. Mark the outer lit test with XFAIL using either `--xfail-tests` flag
or `LIT_XFAIL` env var.
3. Run the tests
4. Observe the lit test is XFAIL as expected, but the failed child test
cases show up as separate failures.

Co-authored-by: kendal <kendal@thebrowser.company>
(cherry picked from commit 9f89d31)
@kendalharland kendalharland mentioned this issue Sep 12, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
llvm-lit
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[lldb][test] Mark gtest cases as XFAIL if the test suite is XFAIL kendalharland/llvm-project
2 participants
@kendalharland @EugeneZelenko