-
Notifications
You must be signed in to change notification settings - Fork 495
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Test failures because of Saucelab tests on macOS, Safari 11 #105
Comments
This is quite critical, as it prevents any PRs from being merged. |
Some context: Example for test failure:
|
cordova-plugin-media-capture/.travis.yml Lines 19 to 22 in 5815be4
Full setup and test command: cordova-plugin-media-capture/.travis.yml Lines 69 to 86 in 5815be4
|
For comparison, this is a successful run (with Firefox): https://travis-ci.org/apache/cordova-plugin-media-capture/jobs/421191978 |
So something is going wrong on Saucelab where it can't connect to Safari or other way around. The tests report a test URL on saucelab: |
Looking at those, we see that on FF it is a short session of 4 steps:
On Safari, it is an endless:
The screenshots/video of Safari also show that it doesn't really load the page as it does in FF. |
Looking at the
After that the test goes into an endless loop. |
As I know nothing about Saucelabs and Appium etc, I went through the builds and commits to find the last one that did not fail: But I also noticed that those tests were failing before sometimes: |
Comparing the logs of the successful run vs. the current failing one only show minimal differences, mostly in |
The error message that is spammed in the log:
comes from |
The code seems to read something from |
The orange url bar seems to be normal for Safari in automation Somehow it just can't load the site fully.
|
I can find some forum threads for the Saucelabs and our tests seems to be running Safari 11:
From: https://saucelabs.com/beta/tests/f8cc93a41cd242a18835e5345c203df0/metadata#2 |
The last successful test ran on the same versions (of everything) as the current failing one: |
Ok, recap time:
|
There are two "known issues" with Safari with that exception:
|
I have access to the Saucelabs account being used now. Tests for Safari 11 started failing between 2018/07/20 and 2018/07/19: |
There is now a support community question over at Saucelabs: https://support.saucelabs.com/hc/en-us/community/posts/360016821133-Tests-on-Safari-11-started-failing-between-2018-07-19-and-2018-07-20 |
Are our node packages up to date? Should we try to update some of them perhaps? |
There is no indication at all that this is connected to any outdated node packages. The plugin and all the tests work as they should - it is the testrun on Saucelab for macOS Safari that is failing. |
The Saucelabs community support question was turned into an email conversation that is currently ongoing. |
Good stuff so far! Keep it up! |
No feedback from Saucelabs yet. With the error message form above, this is also google-able: Unfortunately no other solution than disabling the extension. |
Aha, on their High Sierra machines there is no extension installed. |
Great find! Is that the cause then? And I guess that's something they need to fix? Has it been reported to them? |
They pointed me in the direction that it might be necessary to remove the extension to make it work, and that they are working on it. Nobody mentioned that the extension is not installed on High Sierry though :/ Paramedic, the tool running the tests on Saucelabs, was hardcoded to use macOS Sierra for Safari tests. I changed it to use High Sierra (still hardcoded, should be changed) but this doesn't succeed currently. Investigating. |
Turns out Safari on High Sierra is/has to be 11.1 instead of 10.1 on Sierra. Tests are now running again. |
And success. |
Great! So we can close this now? |
No, not until I confirmed that this is not just a fluke but a systemic fix that works all the time. Doing that now. |
At least in this repo the tests that failed because of that are not failing any more. Yay. |
The build for all PRs is failing for Safari Selenium tests on
LinuxmacOS.The text was updated successfully, but these errors were encountered: