Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test(runner): preserve browser tracing if test fails #59469

Merged
merged 1 commit into from
Dec 11, 2023
Merged

test(runner): preserve browser tracing if test fails #59469

merged 1 commit into from
Dec 11, 2023

Conversation

kwonoj
Copy link
Contributor

@kwonoj kwonoj commented Dec 10, 2023 •
edited
Loading

Description

I was tinkering an idea for debugging flaky tests, if we could attach brwoser side tracing with playwright?

then I found

  1. We already have it configured
  2. But it actually doesn't do anything, since 1. trace never stops to teardown 2. run-tests wipes out after run

PR adjusts the logics for the tracing

  1. Calls the context.tracing.stop when page closes or context is being closed
  2. Preserve files under test/traces under certain conditions - if it's in the CI, and it's retry, and the test is failed after retry

So we can download it and audit to get some more context to debug.

Closes PACK-2129

@ijjk ijjk added area: tests created-by: Turbopack team PRs by the Turbopack team. labels Dec 10, 2023
@ijjk
Copy link
Member

ijjk commented Dec 10, 2023 •
edited
Loading

Tests Passed

@ijjk
Copy link
Member

ijjk commented Dec 10, 2023 •
edited
Loading

Stats from current PR

Default Build
General
vercel/next.js canary vercel/next.js test-playwright-traces Change
buildDuration 10.6s 10.7s N/A
buildDurationCached 6.1s 6s N/A
nodeModulesSize 200 MB 200 MB ✓
nextStartRea..uration (ms) 422ms 419ms N/A
Client Bundles (main, webpack)
vercel/next.js canary vercel/next.js test-playwright-traces Change
170-HASH.js gzip 26.9 kB 26.9 kB N/A
199.HASH.js gzip 181 B 185 B N/A
3f784ff6-HASH.js gzip 53.3 kB 53.3 kB ✓
framework-HASH.js gzip 45.2 kB 45.2 kB ✓
main-app-HASH.js gzip 240 B 241 B N/A
main-HASH.js gzip 31.8 kB 31.7 kB N/A
webpack-HASH.js gzip 1.7 kB 1.7 kB N/A
Overall change 98.5 kB 98.5 kB ✓
Legacy Client Bundles (polyfills)
vercel/next.js canary vercel/next.js test-playwright-traces Change
polyfills-HASH.js gzip 31 kB 31 kB ✓
Overall change 31 kB 31 kB ✓
Client Pages
vercel/next.js canary vercel/next.js test-playwright-traces Change
_app-HASH.js gzip 195 B 194 B N/A
_error-HASH.js gzip 183 B 182 B N/A
amp-HASH.js gzip 501 B 501 B ✓
css-HASH.js gzip 321 B 321 B ✓
dynamic-HASH.js gzip 2.5 kB 2.5 kB N/A
edge-ssr-HASH.js gzip 255 B 255 B ✓
head-HASH.js gzip 349 B 350 B N/A
hooks-HASH.js gzip 368 B 369 B N/A
image-HASH.js gzip 4.27 kB 4.27 kB N/A
index-HASH.js gzip 255 B 256 B N/A
link-HASH.js gzip 2.61 kB 2.6 kB N/A
routerDirect..HASH.js gzip 311 B 309 B N/A
script-HASH.js gzip 384 B 384 B ✓
withRouter-HASH.js gzip 307 B 306 B N/A
1afbb74e6ecf..834.css gzip 106 B 106 B ✓
Overall change 1.57 kB 1.57 kB ✓
Client Build Manifests
vercel/next.js canary vercel/next.js test-playwright-traces Change
_buildManifest.js gzip 484 B 482 B N/A
Overall change 0 B 0 B ✓
Rendered Page Sizes
vercel/next.js canary vercel/next.js test-playwright-traces Change
index.html gzip 529 B 527 B N/A
link.html gzip 541 B 541 B ✓
withRouter.html gzip 524 B 521 B N/A
Overall change 541 B 541 B ✓
Edge SSR bundle Size
vercel/next.js canary vercel/next.js test-playwright-traces Change
edge-ssr.js gzip 93.8 kB 93.8 kB N/A
page.js gzip 146 kB 146 kB N/A
Overall change 0 B 0 B ✓
Middleware size
vercel/next.js canary vercel/next.js test-playwright-traces Change
middleware-b..fest.js gzip 628 B 622 B N/A
middleware-r..fest.js gzip 151 B 151 B ✓
middleware.js gzip 37.4 kB 37.4 kB N/A
edge-runtime..pack.js gzip 1.92 kB 1.92 kB ✓
Overall change 2.07 kB 2.07 kB ✓
Next Runtimes
vercel/next.js canary vercel/next.js test-playwright-traces Change
app-page-exp...dev.js gzip 168 kB 168 kB ✓
app-page-exp..prod.js gzip 94 kB 94 kB ✓
app-page-tur..prod.js gzip 94.7 kB 94.7 kB ✓
app-page-tur..prod.js gzip 89.3 kB 89.3 kB ✓
app-page.run...dev.js gzip 138 kB 138 kB ✓
app-page.run..prod.js gzip 88.6 kB 88.6 kB ✓
app-route-ex...dev.js gzip 24 kB 24 kB ✓
app-route-ex..prod.js gzip 16.7 kB 16.7 kB ✓
app-route-tu..prod.js gzip 16.7 kB 16.7 kB ✓
app-route-tu..prod.js gzip 16.3 kB 16.3 kB ✓
app-route.ru...dev.js gzip 23.4 kB 23.4 kB ✓
app-route.ru..prod.js gzip 16.3 kB 16.3 kB ✓
pages-api-tu..prod.js gzip 9.37 kB 9.37 kB ✓
pages-api.ru...dev.js gzip 9.64 kB 9.64 kB ✓
pages-api.ru..prod.js gzip 9.37 kB 9.37 kB ✓
pages-turbo...prod.js gzip 22 kB 22 kB ✓
pages.runtim...dev.js gzip 22.6 kB 22.6 kB ✓
pages.runtim..prod.js gzip 21.9 kB 21.9 kB ✓
server.runti..prod.js gzip 49.4 kB 49.4 kB ✓
Overall change 930 kB 930 kB ✓
Commit: 231cb0c

@kwonoj kwonoj force-pushed the test-playwright-traces branch 2 times, most recently from 9ac98c8 to b4c854f Compare December 10, 2023 23:18
@kwonoj kwonoj changed the title [DONOTMERGE] [TEST] test(runner): preserve browser tracing if test fails Dec 10, 2023
@kwonoj kwonoj force-pushed the test-playwright-traces branch 2 times, most recently from a1e6ef1 to 927eb40 Compare December 10, 2023 23:37
@kwonoj kwonoj marked this pull request as ready for review December 11, 2023 00:09
@kwonoj kwonoj requested review from S3Prototype and leerob and removed request for a team December 11, 2023 00:09
@kwonoj kwonoj merged commit 25d58d4 into canary Dec 11, 2023
69 checks passed
@kwonoj kwonoj deleted the test-playwright-traces branch December 11, 2023 16:21
ForsakenHarmony
ForsakenHarmony reviewed Dec 11, 2023
View reviewed changes
.github/workflows/build_reusable.yml
@@ -178,6 +178,7 @@ jobs:
with:
name: test-reports
path: |
test/traces
Copy link
Contributor

@ForsakenHarmony ForsakenHarmony Dec 11, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should probably have been put in a different artifact as this artifact is being downloaded in CI for datadog and it will probably make the download a lot slower

but I guess it's only for failed tests which shouldn't happen on canary

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point, will make changes

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#59496

@kwonoj kwonoj mentioned this pull request Dec 11, 2023
Merged
ForsakenHarmony added a commit that referenced this pull request Dec 11, 2023
@kwonoj @ForsakenHarmony
ci(test): upload playwright artifacts seperately (#59496)
618c728 
###

Updating snapshot upload location per suggestion at
#59469 (comment)

Closes PACK-2132

---------

Co-authored-by: Leah <8845940+ForsakenHarmony@users.noreply.github.com>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 26, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@ForsakenHarmony ForsakenHarmony ForsakenHarmony left review comments

@sokra sokra sokra approved these changes

@timneutkens timneutkens Awaiting requested review from timneutkens

@ijjk ijjk Awaiting requested review from ijjk

@shuding shuding Awaiting requested review from shuding

@huozhi huozhi Awaiting requested review from huozhi

@feedthejim feedthejim Awaiting requested review from feedthejim

@ztanner ztanner Awaiting requested review from ztanner

@wyattjoh wyattjoh Awaiting requested review from wyattjoh

@S3Prototype S3Prototype Awaiting requested review from S3Prototype S3Prototype was automatically assigned from vercel/devex

@leerob leerob Awaiting requested review from leerob leerob was automatically assigned from vercel/devex

Assignees
No one assigned
Labels
created-by: Turbopack team PRs by the Turbopack team. locked
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kwonoj @ijjk @sokra @ForsakenHarmony