[BUG] screenshots broken on Firefox in version 1.30.0

[jnizet]

Context:

• Playwright Version: 1.30.0
• Operating System: Linux; Mac
• Node.js version: 16.15.0
• Browser: Firefox

Code Snippet

TypeError: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined

  18 |     await page.click('text=Carte');
  19 |     await page.waitForTimeout(3000);
> 20 |     await test.expect(page.locator('dn-map')).toHaveScreenshot('map.png', { maxDiffPixelRatio: 0.05 });
     |     ^
  21 |   });
  22 | });
  23 |

    at /Users/jb/projects/decouvertes-nature/e2e/activities-map.spec.ts:20:5

Describe the bug

A screenshot test that passed with the previous version of Playwright now fails on Firefox since version 1.30.0. Regenerating the screenshot doesn't even work (see the error message above).
Opening the trace file shows this error:

{
  "type": "action",
  "metadata": {
    "id": "call@17",
    "stack": [],
    "apiName": "page.screenshot",
    "internal": false,
    "objectId": "page@4644ee2e3d87a8fe60f397da9ac044f3",
    "pageId": "page@4644ee2e3d87a8fe60f397da9ac044f3",
    "wallTime": 1674810094035,
    "startTime": 350952.257,
    "endTime": 350959.482,
    "type": "Page",
    "method": "screenshot",
    "params": {
      "timeout": 5000,
      "type": "png",
      "caret": "initial"
    },
    "log": [
      "taking page screenshot"
    ],
    "snapshots": [
      {
        "title": "before",
        "snapshotName": "before@call@17"
      },
      {
        "title": "after",
        "snapshotName": "after@call@17"
      }
    ],
    "afterSnapshot": "after@call@17",
    "error": {
      "error": {
        "message": "Unexpected token u in JSON at position 0",
        "stack": "SyntaxError: Unexpected token u in JSON at position 0\n    at JSON.parse (<anonymous>)\n    at Frame.waitForFunctionValueInUtility (/home/runner/work/decouvertes-nature/decouvertes-nature/node_modules/playwright-core/lib/server/frames.js:1381:17)\n    at FFPage.takeScreenshot (/home/runner/work/decouvertes-nature/decouvertes-nature/node_modules/playwright-core/lib/server/firefox/ffPage.js:402:28)\n    at Screenshotter._screenshot (/home/runner/work/decouvertes-nature/decouvertes-nature/node_modules/playwright-core/lib/server/screenshotter.js:262:20)\n    at /home/runner/work/decouvertes-nature/decouvertes-nature/node_modules/playwright-core/lib/server/screenshotter.js:168:22",
        "name": "SyntaxError"
      }
    }
  }
}

[pavelfeldman]

Things work for me locally, is there anything I can reproduce? (publicly available page?).

The error argument must be of type string or an instance of Buffer, TypedArray, or DataView suggests that we are writing undefined into a file, which means that the screenshot was not obtained. You can probably troubleshoot it if you call await page.locator('dn-map').screenshot().

• It would be best of us to get a repro scenario to try locally, but if it is not possible,
• At the very least you could save a trace for the test execution and share it with us

With the trace, we'll be able to render the page and attempt taking the screenshot in Firefox.

[jnizet]

Hi @pavelfeldman.

Thanks for investigating.

If you go to https://github.com/jnizet/decouvertes-nature/tree/bug/playwright and checkout that bug/playwright branch, I've set up the project so that it runs only the test that does a screenshot. I've removed the expected snapshot files for both chromium and firefox.

When running the test, the chromium test fails as expected (since the expected snapshot isn't there), but the chromium screenshot is properly generated under e2e/activities-map.spec.ts-snapshots.
The same test however, with firefox, does not re-generate the snapshot as it should.

Under e2e/test-results/activities-map-Activities-map-should-display-activities-on-a-map-chromium, I see three files being generated: map-actual.png, test-failed-1.png and trace.zip.

On the other hand, under e2e/activities-map-Activities-map-should-display-activities-on-a-map-firefox, the only file being generated is trace.zip.

Here are the two trace files:

• chromium: trace.zip
• firefox: trace.zip

To reproduce locally, from the root directory:

• yarn
• yarn e2e:ci

This will build the app and serve the app, start firebase emulators, and then run the test on chromium and firefox.

[cexbrayat]

As additional information, I'm seeing the same issue with Firefox and WebKit (but not Chromium) with v1.30

[djahandarie]

I seem to be running into this same error on 1.31.2, Chromium. But it only occurs when I do multiple screenshots in a single test. The following reproduces it consistently for me:

        await page.goto(`file://` + path.join(root, `test/data/html/test-document1.html`));
        await expect(page).toHaveScreenshot();
        await expect(page).toHaveScreenshot();
        await expect(page).toHaveScreenshot();
        await expect(page).toHaveScreenshot();
        await expect(page).toHaveScreenshot();

    TypeError: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined

      56 |         await expect(page).toHaveScreenshot();
      57 |         await expect(page).toHaveScreenshot();
    > 58 |         await expect(page).toHaveScreenshot();
         |         ^
      59 |
    ```

[aslushnikov]

@djahandarie could you please file separately with a repro steps?

[djahandarie]

I was actually attempting to do exactly that, but discovered that it seems to only occur when I have my extension loaded so it's hard to build a minimal test case. I will try to debug it myself, and failing that I'll just share the full repo I guess... Thanks for the response!

[djahandarie]

I figured it out! This error was occurring when the screenshot was hanging. Apparently when you are screenshotting a background page, chromium is very very slow, and sometimes hangs indefinitely. The extension I was loading was opening a "welcome to this extension" tab on install, which caused the test page to end up in the background. Using page.bringToFront() resolved the issue.

I think two things would have helped:

1. A better error which made it clear this was a timeout due to the screenshot taking too long, as opposed to the TypeError: The "data" argument must be of type string or an instance of Buffer, TypedArray, or DataView. Received undefined error.
2. Some note either in the error or in the screenshot docs that you should definitely page.bringToFront() otherwise you may run into this extremely confusing behavior.

[aslushnikov]

@djahandarie this does look like a bug on our side. Any chance you can file it separately with a repro?