tests(devtools): ensure Lighthouse starts in smoke tests #15459
Conversation
adamraine
requested review from
brendankenny
and removed request for
a team
core/scripts/pptr-run-devtools.js
Outdated
| @@ -328,6 +341,16 @@ async function testUrlFromDevtools(url, options = {}) { | |||
| const result = await evaluateInSession(inspectorSession, runLighthouse, [addSniffer]); | |||
|
|
|||
| return {...result, logs}; | |||
| } catch (err) { | |||
| if (options.detailedError && inspectorSession) { | |||
There was a problem hiding this comment.
why not always do this? debug mode is more about reducing output noise, but giving context on failure is always useful
There was a problem hiding this comment.
The screenshot URL is very long compared to other output. This isn't really "noise" for the reason you mentioned but it is extra debug information. Since this is for errors and not expectation differences it makes more sense to always show the screenshot.
There was a problem hiding this comment.
Could it save to disk in .tmp and just print the path? And then debug mode would print the contents too (for purposes of CI..) ... or we could upload a folder called .tmp/debug-info.
There was a problem hiding this comment.
Ehh I like having the screenshot easily accessible in CI logs. I don't want to have to download CI failure artifacts if I don't have to.
Plus, this file is organized to be called by a smoke runner but isn't aware of any smoke artifact folders itself.
A lot of our flaky DevTools smoke failures are timeouts on the protocol command that waits for Lighthouse to finish (example). After some investigation, I discovered that the Lighthouse run never actually starts in these cases. For whatever reason, clicking the start button just once isn't 100% reliable.
This PR adds a loop that clicks the start button every 100ms until we detect the run has started via JS sniffer.
I also added some code that lets us view a screenshot of the DT frontend when a timeout occurs. This screenshot diagnostic is already used in DT e2e tests, and was super helpful for debugging this problem.