Navigation API test results inaccurate?

[domenic]

https://wpt.fyi/results/navigation-api?label=master&label=experimental&aligned shows lots of failures in Chromium. The results are inconsistent between Chrome and Edge.

Those failures do not appear in Chromium CI. They do not appear in the stability checker when CLs are exported. They do not appear when running the tests using wpt.live.

One guess as to why this is happening is due to the unique nature of these tests as testing navigation and history traversal, which makes them sensitive to things like browsing context reuse. Perhaps when collecting results for wpt.fyi, the tests are run in a window that is then navigated to a second test, instead of being run in a fresh window? That would definitely break most of these tests.

[domenic]

As a simple example, https://wpt.fyi/results/navigation-api/navigation-methods/back-forward-multiple-frames.html?label=experimental&label=master&aligned&view=subtest expects the browsing context to have a single history entry (the test itself), but fails immediately because apparently in the WPT runner, there are two history entries at the start of the test. Maybe some driver or harness is corrupting things?

[foolip]

How much of this remains after #36069? Are there still differences between Chromium CI and the results on wpt.fyi?

[domenic]

No tests in the navigation API test suite were .window.js tests, so this problem remains. In particular, every failing-on-wpt.fyi test in the test suite (66 failures out of 400) passes on Chromium CI / when run on wpt.live.

[foolip]

I see, so this is about how wptrunner with the WebDriver executor opens the tests:

wpt/tools/wptrunner/wptrunner/executors/executorwebdriver.py

Lines 493 to 507 in 0136915

	# Now start the test harness
	protocol.base.execute_script("window.open('about:blank', '%s', 'noopener')" % self.window_id)
	test_window = protocol.testharness.get_test_window(self.window_id,
	parent_window,
	timeout=5*self.timeout_multiplier)
	self.protocol.base.set_window(test_window)
	# Wait until about:blank has been loaded
	protocol.base.execute_script(self.window_loaded_script, asynchronous=True)
	handler = WebDriverCallbackHandler(self.logger, protocol, test_window)
	protocol.webdriver.url = url
	if not self.supports_eager_pageload:
	self.wait_for_load(protocol)

There's a tab open that doesn't run any tests, and we use that to window.open() the test tab as about:blank, and then navigate that to the test URL.

So tests can't assume that they get loaded without any history.

@jgraham any idea how we could provide guarantees here?

[jgraham]

The Firefox (marionette) implementation already uses a tab per test:

wpt/tools/wptrunner/wptrunner/executors/executormarionette.py

Lines 926 to 935 in 0136915

	parent_window = protocol.testharness.close_old_windows(self.last_environment["protocol"])
	if self.protocol.coverage.is_enabled:
	self.protocol.coverage.reset()
	format_map = {"url": strip_server(url)}
	protocol.base.execute_script("window.open('about:blank', '%s', 'noopener')" % self.window_id)
	test_window = protocol.testharness.get_test_window(self.window_id, parent_window,
	timeout=10 * self.timeout_multiplier)

Historically Firefox has been more concerned about stability and others more concerned about performance, which might explain this difference (or it might be unintentional, I don't know).

[jgraham]

Oh, sorry my comment is nonsense; it is the same.

But yes, I agree that we explicitly load about:blank when creating the test window. That does have substantive advantages e.g. in the Firefox case we allow enabling devtools before navigating to the test.

I think if tests really want detailed control over the state of session history, they probably ought to open a new window that they own directly.

[foolip]

@domenic would it be practical to run these tests in a new window?

[domenic]

It would require rewriting the entire test suite. (And probably many tests in html/browsers/ as well.) I don't think that's a fair thing to require of test authors, especially given that it works fine on wpt.live / Chromium CI / etc.

[OrKoN]

@jgraham would user contexts in WebDriver BiDi help with this issue? I imagine we can configure the test runner to run certain tests in a fresh user context so that the test code itself does not need to be updated.