Reduce CI flakiness by using cargo-nextest

[soareschen]

Description

The CI is getting very flaky. I'm trying to use cargo nextest with retries to retry individual tests, rather than retrying the whole integration test when one fails.

---

PR author checklist:

• Added changelog entry, using unclog.
• Added tests: integration (for Hermes) or unit/mock tests (for modules).
• Linked to GitHub issue.
• Updated code comments and documentation (e.g., docs/).
• Tagged one reviewer who will be the one responsible for shepherding this PR.

Reviewer checklist:

• Reviewed Files changed in the GitHub PR explorer.
• Manually tested (in case integration/unit/mock tests are absent).

[ljoss17]

Looks great!

[romac]

Great idea!

[romac]

Let's use this as well for all tests, including the multi chain tests

[romac]

Not sure why the multi-chains test got slower, and therefore killed after 60min, whereas they used to run in ~40min. I increased the timeout to see how long they take now.

[romac]

Oh but everything is slower now that we don't pass --test-threads=2 anymore.

[soareschen]

Yeah it looks like cargo nextest do not allow multiple threads when --nocapture is on. Makes sense considering that the output from multiple tests would interleave with each other. Or perhaps we just don't capture the output and rely on local tests for any log message?

[soareschen]

Just figured there is a --failure-output option which we can use to only output logs for a test only when it failed. This should significantly reduce the CI logs and also allow us to parallelize the tests again.

[romac]

Wow, great find, that's amazing!

[romac]

@soareschen It looks your commits aren't signed anymore. I can still merge the branch, but that's maybe something to look into.

[soareschen]

I think I just didn't configure about git commit signing. Will try signing with GPG in future commits.

[romac]

You can also sign commits with your SSH key if it's easier to setup