Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: support retries in e2e tests in CI #2145

Merged
merged 8 commits into from
Nov 29, 2024

Conversation

jpraynaud
Copy link
Member

@jpraynaud jpraynaud commented Nov 27, 2024

Content

This PR includes support for retrying e2e test runs in case of transient error on the Cardano network of the underlying devnet (which happen because of the custom configuration used for it). This will overall remove the main source of flakiness in the CI runs.

Error handling improvements:

Main execution flow restructuring:

Enhancements to GitHub workflows:

Examples

Here is an example of retry in the CI:

Pre-submit checklist

  • Branch
    • Tests are provided (if possible)
    • Crates versions are updated (if relevant)
    • Commit sequence broadly makes sense
    • Key commits have useful messages
  • PR
    • No clippy warnings in the CI
    • Self-reviewed the diff
    • Useful pull request description
    • Reviewer requested

Issue(s)

Closes #2123

@jpraynaud jpraynaud self-assigned this Nov 27, 2024
Copy link

github-actions bot commented Nov 27, 2024

Test Results

    4 files  ±0     52 suites  +1   11m 35s ⏱️ -8s
1 417 tests +2  1 417 ✅ +2  0 💤 ±0  0 ❌ ±0 
1 628 runs  +2  1 628 ✅ +2  0 💤 ±0  0 ❌ ±0 

Results for commit fcb728e. ± Comparison against base commit 44004bf.

♻️ This comment has been updated with latest results.

@jpraynaud jpraynaud force-pushed the jpraynaud/2123-retries-e2e-test branch from edbd7d3 to 24062de Compare November 27, 2024 15:39
@jpraynaud jpraynaud temporarily deployed to testing-sanchonet November 27, 2024 15:49 — with GitHub Actions Inactive
@jpraynaud jpraynaud temporarily deployed to testing-sanchonet November 27, 2024 16:03 — with GitHub Actions Inactive
@jpraynaud jpraynaud force-pushed the jpraynaud/2123-retries-e2e-test branch from e752dd4 to 2724af6 Compare November 27, 2024 16:15
@jpraynaud jpraynaud force-pushed the jpraynaud/2123-retries-e2e-test branch from 2724af6 to 3520b73 Compare November 27, 2024 16:22
@jpraynaud jpraynaud temporarily deployed to testing-sanchonet November 27, 2024 16:30 — with GitHub Actions Inactive
@jpraynaud jpraynaud force-pushed the jpraynaud/2123-retries-e2e-test branch 3 times, most recently from df45fe3 to 7676e66 Compare November 27, 2024 17:39
@jpraynaud jpraynaud temporarily deployed to testing-sanchonet November 27, 2024 17:48 — with GitHub Actions Inactive
@jpraynaud jpraynaud force-pushed the jpraynaud/2123-retries-e2e-test branch 2 times, most recently from 2301030 to 01bd176 Compare November 27, 2024 18:20
@jpraynaud jpraynaud force-pushed the jpraynaud/2123-retries-e2e-test branch 2 times, most recently from 4154bff to d78c7a3 Compare November 27, 2024 18:41
@jpraynaud jpraynaud temporarily deployed to testing-sanchonet November 27, 2024 18:51 — with GitHub Actions Inactive
@jpraynaud jpraynaud force-pushed the jpraynaud/2123-retries-e2e-test branch from d78c7a3 to 8807497 Compare November 28, 2024 10:27
@jpraynaud jpraynaud temporarily deployed to testing-sanchonet November 28, 2024 10:36 — with GitHub Actions Inactive
@jpraynaud jpraynaud force-pushed the jpraynaud/2123-retries-e2e-test branch from 8807497 to e59d250 Compare November 28, 2024 10:57
@jpraynaud jpraynaud temporarily deployed to testing-sanchonet November 28, 2024 11:06 — with GitHub Actions Inactive
Copy link
Collaborator

@Alenar Alenar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚀, just some minor details and a question relating user cancellation.

mithril-test-lab/mithril-end-to-end/src/main.rs Outdated Show resolved Hide resolved
mithril-test-lab/mithril-end-to-end/src/main.rs Outdated Show resolved Hide resolved
Copy link
Collaborator

@sfauvel sfauvel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@dlachaume dlachaume left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 👍

@jpraynaud jpraynaud force-pushed the jpraynaud/2123-retries-e2e-test branch 2 times, most recently from a528655 to 1e7719c Compare November 28, 2024 17:24
@jpraynaud jpraynaud temporarily deployed to testing-sanchonet November 28, 2024 17:33 — with GitHub Actions Inactive
@jpraynaud jpraynaud force-pushed the jpraynaud/2123-retries-e2e-test branch from 1e7719c to fcb728e Compare November 29, 2024 10:07
@jpraynaud jpraynaud temporarily deployed to testing-sanchonet November 29, 2024 10:16 — with GitHub Actions Inactive
@jpraynaud jpraynaud merged commit 60db5ce into main Nov 29, 2024
47 of 49 checks passed
@jpraynaud jpraynaud deleted the jpraynaud/2123-retries-e2e-test branch November 29, 2024 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement retries in e2e tests in CI
4 participants