Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build: Improve E2E tests with ESLint rules #29041

Merged
merged 4 commits into from
Sep 4, 2024
Merged

Conversation

JReinhold
Copy link
Contributor

@JReinhold JReinhold commented Sep 3, 2024

What I did

Initially I fixed some flaky tests by converting to auto-retrying assertions.

Then I realised this this the third time I do this, so I installed and configured eslint-plugin-playwright to hopefully ensure it won't happen again. This required a lot of fixing of all the E2E tests, so I did that too.

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

  • stories
  • unit tests
  • integration tests
  • end-to-end tests

Manual testing

This section is mandatory for all contributions. If you believe no manual test is necessary, please state so explicitly. Thanks!

Documentation

  • Add or update documentation reflecting your changes
  • If you are deprecating/removing a feature, make sure to update
    MIGRATION.MD

Checklist for Maintainers

  • When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli/src/sandbox-templates.ts

  • Make sure this PR contains one of the labels below:

    Available labels
    • bug: Internal changes that fixes incorrect behavior.
    • maintenance: User-facing maintenance tasks.
    • dependencies: Upgrading (sometimes downgrading) dependencies.
    • build: Internal-facing build tooling & test updates. Will not show up in release changelog.
    • cleanup: Minor cleanup style change. Will not show up in release changelog.
    • documentation: Documentation only changes. Will not show up in release changelog.
    • feature request: Introducing a new feature.
    • BREAKING CHANGE: Changes that break compatibility in some way with current major version.
    • other: Changes that don't fit in the above categories.

🦋 Canary release

This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the @storybookjs/core team here.

core team members can create a canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=<PR_NUMBER>

name before after diff z %
createSize 0 B 0 B 0 B - -
generateSize 76.5 MB 76.5 MB 0 B 2.43 0%
initSize 161 MB 161 MB 8.44 kB -0.26 0%
diffSize 84.7 MB 84.7 MB 8.44 kB -0.67 0%
buildSize 7.48 MB 7.48 MB 0 B -1.39 0%
buildSbAddonsSize 1.62 MB 1.62 MB 0 B 2.64 0%
buildSbCommonSize 195 kB 195 kB 0 B - 0%
buildSbManagerSize 2.31 MB 2.31 MB 0 B -3 0%
buildSbPreviewSize 352 kB 352 kB 0 B - 0%
buildStaticSize 0 B 0 B 0 B - -
buildPrebuildSize 4.47 MB 4.47 MB 0 B 2.44 0%
buildPreviewSize 3 MB 3 MB 0 B -1.46 0%
testBuildSize 0 B 0 B 0 B - -
testBuildSbAddonsSize 0 B 0 B 0 B - -
testBuildSbCommonSize 0 B 0 B 0 B - -
testBuildSbManagerSize 0 B 0 B 0 B - -
testBuildSbPreviewSize 0 B 0 B 0 B - -
testBuildStaticSize 0 B 0 B 0 B - -
testBuildPrebuildSize 0 B 0 B 0 B - -
testBuildPreviewSize 0 B 0 B 0 B - -
name before after diff z %
createTime 6.3s 18.3s 12s 0.58 65.7%
generateTime 18.3s 20.3s 1.9s -0.49 9.8%
initTime 14.6s 17s 2.4s -0.5 14.1%
buildTime 11.9s 11.2s -690ms -0.8 -6.1%
testBuildTime 0ms 0ms 0ms - -
devPreviewResponsive 9.4s 6.8s -2s -623ms -0.21 -38.2%
devManagerResponsive 6.3s 4.4s -1s -848ms -0.07 -41.3%
devManagerHeaderVisible 1.2s 689ms -525ms -0.85 -76.2%
devManagerIndexVisible 1.2s 722ms -535ms -0.9 -74.1%
devStoryVisibleUncached 1.7s 896ms -885ms -1.9 🔰-98.8%
devStoryVisible 1.2s 734ms -520ms -0.78 -70.8%
devAutodocsVisible 1.1s 667ms -506ms -0.44 -75.9%
devMDXVisible 900ms 694ms -206ms -0.09 -29.7%
buildManagerHeaderVisible 779ms 748ms -31ms 0.1 -4.1%
buildManagerIndexVisible 815ms 753ms -62ms -0.06 -8.2%
buildStoryVisible 816ms 1s 222ms 2.23 🔺21.4%
buildAutodocsVisible 788ms 702ms -86ms 0.04 -12.3%
buildMDXVisible 709ms 623ms -86ms -1.1 -13.8%

Greptile Summary

This PR improves the E2E tests for Storybook by implementing ESLint rules for Playwright tests and updating test files to use more robust practices.

  • Added eslint-plugin-playwright and configured rules in code/.eslintrc.js
  • Updated E2E test files to use auto-retrying assertions, reducing flaky tests
  • Removed unnecessary 'await' keywords and improved selector specificity
  • Refactored test structure in framework-specific tests (Next.js, Svelte) for better organization
  • Updated JSON file tests to use stricter assertions with toStrictEqual()

@JReinhold JReinhold self-assigned this Sep 3, 2024
@JReinhold JReinhold added build Internal-facing build tooling & test updates ci:daily Run the CI jobs that normally run in the daily job. labels Sep 3, 2024
Copy link

nx-cloud bot commented Sep 3, 2024

☁️ Nx Cloud Report

CI is running/has finished running commands for commit bb3301c. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution


✅ Successfully ran 2 targets

Sent with 💌 from NxCloud.

@@ -8,14 +8,14 @@
"vitest",
"testing"
],
"homepage": "https://github.com/storybookjs/storybook/tree/next/code/addons/vitest",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks <3

@JReinhold JReinhold added ci:daily Run the CI jobs that normally run in the daily job. and removed ci:daily Run the CI jobs that normally run in the daily job. labels Sep 4, 2024
@JReinhold JReinhold marked this pull request as ready for review September 4, 2024 09:13
@JReinhold JReinhold merged commit ab66906 into next Sep 4, 2024
115 of 117 checks passed
@JReinhold JReinhold deleted the jeppe/fix-e2e-flake branch September 4, 2024 09:15
@github-actions github-actions bot mentioned this pull request Sep 4, 2024
9 tasks
Copy link
Contributor

@greptile-apps greptile-apps bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

19 file(s) reviewed, 3 comment(s)
Edit PR Review Bot Settings

Comment on lines +1 to +3
/* eslint-disable playwright/no-conditional-expect */

/* eslint-disable playwright/no-conditional-in-test */
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

style: These ESLint disables might be masking potential issues in the test code. Consider addressing the underlying problems instead of disabling the rules.

).toBeVisible();

await page.locator('[id="storybook\\@7\\.6\\.18_components-badge"]').click();
await expect(
await page.locator('[id="storybook\\@7\\.6\\.18_components-badge--default"]')
page.locator('[id="storybook\\@7\\.6\\.18_components-badge--default"]')
).toBeVisible();

// Expect composed stories `to be available in the search
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

syntax: There's a backtick in the comment that should be removed


const noAutodocsAnchor = await root.locator('#anchor--core-tags-remove--no-autodocs').all();
expect(noAutodocsAnchor.length).toBe(0);
// Autodocs should include autodocs and exclude dev, test
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

logic: This comment is incorrect. It should be 'Autodocs should include no-dev, no-test and exclude no-autodocs'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build Internal-facing build tooling & test updates ci:daily Run the CI jobs that normally run in the daily job.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants