upload core dumps from failed tests in CI #1850
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spoonincode
changed the title
heifner
approved these changes
Nov 1, 2023
linh2931
approved these changes
Nov 1, 2023
spoonincode
commented
Nov 1, 2023
| @@ -181,12 +188,17 @@ jobs: | |||
| log-tarball-prefix: ${{matrix.cfg.name}} | |||
| tests-label: nonparallelizable_tests | |||
| test-timeout: 420 | |||
| - name: Export core dumps | |||
| run: docker run --mount type=bind,source=/var/lib/systemd/coredump,target=/cores alpine sh -c 'tar -C /cores/ -c .' | tar x | |||
There was a problem hiding this comment.
I know this looks weird, and to add more confusion it's different than the parallel tests job because NP/LR tests don't run the job in a container until parallel-ctest-containers step.
On ENF runners the user is not root but rather a user named enf that does not have sudo privileges (this is unlike GitHub runners). But /var/lib/systemd/coredump files are only readable by root. Since docker is not creating a userns in this case, we can use docker as a cheat to get access to the files without sudo and without addtional modificatons on the ENF runners.
This was referenced Nov 2, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Core dumps will now be uploaded from crashing tests in CI. Resolves #640. Example of failure and uploaded files can be found here,
https://github.com/AntelopeIO/leap/actions/runs/6724910321
(I trimmed the run down to just Tests & NP Tests on a single platform)
While I didn't change anything on the ENF runners to accomplish this, it's worth noting that this probably only works with ENF runners because of their behavior to put core dumps in to
/var/lib/systemd/coredump. Something like Github's runners don't do that. While I like to keep the workflows runner-agnostic so that forks could potentially work in the future, afaik this won't break the workflow on other runners; you'll just miss out on the core dumps.Coming along for the ride in some separate commits were a couple modifications to use node20 instead of node16. In some of our repos starting to notice warnings about using node16, for example: https://github.com/AntelopeIO/bls12-381/actions/runs/6633721660 (I'm not sure why we haven't seen warnings in leap; possibly a slow rollout by Github?). So,