Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test failures because of Saucelab tests on macOS, Safari 11 #105

Closed
ffMathy opened this issue Sep 21, 2018 · 32 comments
Closed

Test failures because of Saucelab tests on macOS, Safari 11 #105

ffMathy opened this issue Sep 21, 2018 · 32 comments

Comments

@ffMathy
Copy link

ffMathy commented Sep 21, 2018

The build for all PRs is failing for Safari Selenium tests on Linux macOS.

@ffMathy
Copy link
Author

ffMathy commented Sep 21, 2018

This is quite critical, as it prevents any PRs from being merged.

@janpio
Copy link
Member

janpio commented Sep 21, 2018

Some context:

Example for test failure:
https://travis-ci.org/apache/cordova-plugin-media-capture/builds/418337045
Node 4.2, platform=browser-safari
https://travis-ci.org/apache/cordova-plugin-media-capture/jobs/418337048

...
cordova-paramedic: connecting webdriver
................
cordova-paramedic: connecting to app
cordova-paramedic: waiting for test results
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
cordova-paramedic: Tests failed to complete; ending appium session. The error is:
Error: Seems like device not connected to local server in 300 secs
    at null._onTimeout (/tmp/paramedic/lib/paramedic.js:467:24)
    at Timer.listOnTimeout (timers.js:92:15)
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
cordova-paramedic: Closing Sauce Connect process...
cordova-paramedic: Successfully closed Sauce Connect process
Unsupported platform for Appium test run: browser
Completed tests at 5:39:48 PM
Getting saucelabs jobs details...
============================================================================================
Job name: travis-plugin-media-capture-211.3
Job ID: 05bf4db92bf44e2287e737dd47306eca
Job URL: https://saucelabs.com/beta/tests/05bf4db92bf44e2287e737dd47306eca
Video: https://assets.saucelabs.com/jobs/05bf4db92bf44e2287e737dd47306eca/video.flv
Appium logs: https://assets.saucelabs.com/jobs/05bf4db92bf44e2287e737dd47306eca/selenium-server.log
============================================================================================
local-server: killing local file transfer server if it's up...
cordova-paramedic: Deleting the application: /tmp/tmp-2965IU12VGU4tPCj
/home/travis/build/apache/cordova-plugin-media-capture
Finished with exit code 1
The command "node /tmp/paramedic/main.js --config pr/$PLATFORM --plugin $(pwd) --shouldUseSauce --buildName travis-plugin-media-capture-$TRAVIS_JOB_NUMBER" exited with 1.
Done. Your build exited with 1.

@janpio
Copy link
Member

janpio commented Sep 21, 2018

- env: PLATFORM=browser-safari
os: linux
language: node_js
node_js: '4.2'

Full setup and test command:

before_install:
- rm -rf ~/.nvm && git clone https://github.com/creationix/nvm.git ~/.nvm && (cd ~/.nvm
&& git checkout `git describe --abbrev=0 --tags`) && source ~/.nvm/nvm.sh && nvm
install $TRAVIS_NODE_VERSION
- node --version
- if [[ "$PLATFORM" =~ android ]]; then gradle --version; fi
- if [[ "$PLATFORM" =~ ios ]]; then npm install -g ios-deploy; fi
- if [[ "$PLATFORM" =~ android ]]; then echo y | android update sdk -u --filter android-22,android-23,android-24,android-25,android-26;
fi
- git clone https://github.com/apache/cordova-paramedic /tmp/paramedic && pushd /tmp/paramedic
&& npm install && popd
- npm install -g cordova
install:
- npm install
script:
- npm test
- node /tmp/paramedic/main.js --config pr/$PLATFORM --plugin $(pwd) --shouldUseSauce
--buildName travis-plugin-media-capture-$TRAVIS_JOB_NUMBER

@janpio
Copy link
Member

janpio commented Sep 21, 2018

For comparison, this is a successful run (with Firefox): https://travis-ci.org/apache/cordova-plugin-media-capture/jobs/421191978
And again, here the failing one: https://travis-ci.org/apache/cordova-plugin-media-capture/jobs/421191979

@janpio
Copy link
Member

janpio commented Sep 21, 2018

So something is going wrong on Saucelab where it can't connect to Safari or other way around.

The tests report a test URL on saucelab:
Success with FF: https://saucelabs.com/beta/tests/a50cb73525164226a1e3adbae6e721f1
Failure with Safari: https://saucelabs.com/beta/tests/f8cc93a41cd242a18835e5345c203df0

@janpio
Copy link
Member

janpio commented Sep 21, 2018

Looking at those, we see that on FF it is a short session of 4 steps:

POST /session
POST url
POST execute
DELETE /session/a50cb73525164226a1e3adbae6e721f1

On Safari, it is an endless:

POST url
POST execute
POST url
POST execute
POST url
POST execute
POST execute
POST url
POST execute
POST url
POST execute
POST execute
...

The screenshots/video of Safari also show that it doesn't really load the page as it does in FF.

@janpio
Copy link
Member

janpio commented Sep 21, 2018

Looking at the selenium-server.log of the both and comparing, it seems to go all wrong for Safari here:

10:12:57.177 INFO - Executing: [set window position])
10:12:57.204 WARN - Exception thrown
org.openqa.selenium.NoSuchWindowException: A request to use a window could not be satisfied because the window could not be found. (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 13 milliseconds
Build info: version: '3.4.0', revision: 'unknown', time: 'unknown'
System info: host: 'itako290.prod.miso', ip: 'fe80:0:0:0:c3f:8934:b675:8b9c%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.12.6', java.version: '9.0.4'
Driver info: org.openqa.selenium.safari.SafariDriver
Capabilities [{applicationCacheEnabled=true, rotatable=false, databaseEnabled=true, handlesAlerts=true, version=12604.5.6.1.1, cleanSession=true, platform=MAC, nativeEvents=true, locationContextEnabled=false, webStorageEnabled=true, browserName=safari, javascriptEnabled=true, platformName=macOS, cssSelectorsEnabled=true}]
Session ID: F3C5C069-19D7-4436-B699-B705F59B57E5
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
	at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:215)
	at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:167)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:671)
	at org.openqa.selenium.remote.RemoteWebDriver$RemoteWebDriverOptions$RemoteWindow.setPosition(RemoteWebDriver.java:895)
	at org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWindow.setPosition(EventFiringWebDriver.java:629)
	at org.openqa.selenium.remote.server.handler.SetWindowPosition.call(SetWindowPosition.java:53)
	at org.openqa.selenium.remote.server.handler.SetWindowPosition.call(SetWindowPosition.java:27)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.openqa.selenium.remote.server.DefaultSession$1.run(DefaultSession.java:176)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.base/java.lang.Thread.run(Thread.java:844)
10:12:57.208 WARN - Exception: A request to use a window could not be satisfied because the window could not be found. (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 13 milliseconds
Build info: version: '3.4.0', revision: 'unknown', time: 'unknown'
System info: host: 'itako290.prod.miso', ip: 'fe80:0:0:0:c3f:8934:b675:8b9c%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.12.6', java.version: '9.0.4'
Driver info: org.openqa.selenium.safari.SafariDriver
Capabilities [{applicationCacheEnabled=true, rotatable=false, databaseEnabled=true, handlesAlerts=true, version=12604.5.6.1.1, cleanSession=true, platform=MAC, nativeEvents=true, locationContextEnabled=false, webStorageEnabled=true, browserName=safari, javascriptEnabled=true, platformName=macOS, cssSelectorsEnabled=true}]
Session ID: F3C5C069-19D7-4436-B699-B705F59B57E5
...

After that the test goes into an endless loop.

@janpio
Copy link
Member

janpio commented Sep 21, 2018

As I know nothing about Saucelabs and Appium etc, I went through the builds and commits to find the last one that did not fail:
https://travis-ci.org/apache/cordova-plugin-media-capture/builds/367501510
https://travis-ci.org/apache/cordova-plugin-media-capture/jobs/367501513

But I also noticed that those tests were failing before sometimes:
https://travis-ci.org/apache/cordova-plugin-media-capture/builds/317203713
https://travis-ci.org/apache/cordova-plugin-media-capture/builds/298245732

@janpio
Copy link
Member

janpio commented Sep 21, 2018

Comparing the logs of the successful run vs. the current failing one only show minimal differences, mostly in npm install and similar.

@janpio
Copy link
Member

janpio commented Sep 21, 2018

The error message that is spammed in the log:

appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null
appium: Error: Cannot get the event cache: it doesn't exist in the app. Got this instead: null

comes from paramedic:
https://github.com/apache/cordova-paramedic/blob/31155dfd9287e9c53b034bc00dba4473f7c9bf5c/lib/appium/helpers/wdHelper.js#L255

@janpio
Copy link
Member

janpio commented Sep 21, 2018

The code seems to read something from window and go into error handling if it doesn't find it:
https://github.com/apache/cordova-paramedic/blob/31155dfd9287e9c53b034bc00dba4473f7c9bf5c/lib/appium/helpers/wdHelper.js#L229-L255

@janpio
Copy link
Member

janpio commented Sep 21, 2018

The orange url bar seems to be normal for Safari in automation
https://saucelabs.com/beta/tests/f8cc93a41cd242a18835e5345c203df0/commands#2.

Somehow it just can't load the site fully.
selenium-server.log indicates that something goes wrong even before that, with:

10:12:57.177 INFO - Executing: [set window position])
10:12:57.204 WARN - Exception thrown
org.openqa.selenium.NoSuchWindowException: A request to use a window could not be satisfied because the window could not be found. (WARNING: The server did not provide any stacktrace information)
Command duration or timeout: 13 milliseconds
Build info: version: '3.4.0', revision: 'unknown', time: 'unknown'
System info: host: 'itako290.prod.miso', ip: 'fe80:0:0:0:c3f:8934:b675:8b9c%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.12.6', java.version: '9.0.4'
Driver info: org.openqa.selenium.safari.SafariDriver
Capabilities [{applicationCacheEnabled=true, rotatable=false, databaseEnabled=true, handlesAlerts=true, version=12604.5.6.1.1, cleanSession=true, platform=MAC, nativeEvents=true, locationContextEnabled=false, webStorageEnabled=true, browserName=safari, javascriptEnabled=true, platformName=macOS, cssSelectorsEnabled=true}]
Session ID: F3C5C069-19D7-4436-B699-B705F59B57E5
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
	at org.openqa.selenium.remote.ErrorHandler.createThrowable(ErrorHandler.java:215)
	at org.openqa.selenium.remote.ErrorHandler.throwIfResponseFailed(ErrorHandler.java:167)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:671)
	at org.openqa.selenium.remote.RemoteWebDriver$RemoteWebDriverOptions$RemoteWindow.setPosition(RemoteWebDriver.java:895)
	at org.openqa.selenium.support.events.EventFiringWebDriver$EventFiringWindow.setPosition(EventFiringWebDriver.java:629)
	at org.openqa.selenium.remote.server.handler.SetWindowPosition.call(SetWindowPosition.java:53)
	at org.openqa.selenium.remote.server.handler.SetWindowPosition.call(SetWindowPosition.java:27)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at org.openqa.selenium.remote.server.DefaultSession$1.run(DefaultSession.java:176)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.base/java.lang.Thread.run(Thread.java:844)

@janpio
Copy link
Member

janpio commented Sep 21, 2018

I can find some forum threads for the org.openqa.selenium.NoSuchWindowException: A request to use a window could not be satisfied because the window could not be found. (WARNING: The server did not provide any stacktrace information) error message, but those all are for Safari 10.x

Saucelabs and our tests seems to be running Safari 11:

Browser
safari
Browser version
11.0.
OS
Mac 10.12 

From: https://saucelabs.com/beta/tests/f8cc93a41cd242a18835e5345c203df0/metadata#2

@janpio
Copy link
Member

janpio commented Sep 21, 2018

The last successful test ran on the same versions (of everything) as the current failing one:
https://saucelabs.com/beta/tests/f8cc93a41cd242a18835e5345c203df0/metadata
https://saucelabs.com/beta/tests/7610de8c2de14706b4137e5a17da974c/metadata
Unfortunately test assets are already deleted, so no way to watch the video and look at the logs :/

@janpio
Copy link
Member

janpio commented Sep 21, 2018

Ok, recap time:

  • Some time in the past the tests stopped working although the environment did not change.
  • When Sauce executes a command (probably to maximize the browser window, but I couldn't find out if we trigger that or if this is an automatic thing) an exception is triggered:
10:12:56.965 INFO - Executing: [execute script: return screen.width, []])
10:12:57.153 INFO - Done: [execute script: return screen.width, []]
10:12:57.163 INFO - Executing: [execute script: return screen.height, []])
10:12:57.170 INFO - Done: [execute script: return screen.height, []]
10:12:57.177 INFO - Executing: [set window position])
10:12:57.204 WARN - Exception thrown
org.openqa.selenium.NoSuchWindowException: A request to use a window could not be satisfied because the window could not be found. (WARNING: The server did not provide any stacktrace information)
...

@janpio
Copy link
Member

janpio commented Sep 21, 2018

There are two "known issues" with Safari with that exception:

  1. SAFARI: NoSuchWindowError: A request to use a window could not be satisfied because the window could not be found. SeleniumHQ/selenium#3796 in Safari 10.1, which was solved in Safari 11 - we use Safari 11
  2. Another that is described as Tests on Mac OS 10.12 with Safari 10.1/11.0 Fail Due to Losing Focus to Gridlauncher/GridlauncherV3 but which has a different error message

@janpio
Copy link
Member

janpio commented Sep 24, 2018

I have access to the Saucelabs account being used now.

Tests for Safari 11 started failing between 2018/07/20 and 2018/07/19:
image
https://saucelabs.com/beta/archives?query=browser:%22Safari%2011%22&sort=Best%20Match

@janpio
Copy link
Member

janpio commented Sep 24, 2018

@janpio janpio changed the title Build is failing Test failures because of Saucelab tests on macOS, Safari 11 Sep 24, 2018
@ffMathy
Copy link
Author

ffMathy commented Sep 24, 2018

Are our node packages up to date? Should we try to update some of them perhaps?

@janpio
Copy link
Member

janpio commented Sep 24, 2018

There is no indication at all that this is connected to any outdated node packages. The plugin and all the tests work as they should - it is the testrun on Saucelab for macOS Safari that is failing.

@janpio
Copy link
Member

janpio commented Sep 26, 2018

The Saucelabs community support question was turned into an email conversation that is currently ongoing.

@janpio
Copy link
Member

janpio commented Sep 27, 2018

Reply from Saucelabs support indicated that there might be a problem with the WebDriver extension installed in Safari on their VMs. Manually taking over a test run and uninstalling the VM indeed fixed the loading problem of our tests page.

Loading the page with the extension still installed:
image
image
I am really confused by that error as it happens in a comment!?
image

Loading the page with the extension uninstalled:
image

Now waiting for feedback on our options for resolution.

@ffMathy
Copy link
Author

ffMathy commented Sep 29, 2018

Good stuff so far! Keep it up!

@janpio
Copy link
Member

janpio commented Sep 29, 2018

Aha, on their High Sierra machines there is no extension installed.

@ffMathy
Copy link
Author

ffMathy commented Sep 29, 2018

Great find! Is that the cause then? And I guess that's something they need to fix? Has it been reported to them?

@janpio
Copy link
Member

janpio commented Sep 29, 2018

They pointed me in the direction that it might be necessary to remove the extension to make it work, and that they are working on it. Nobody mentioned that the extension is not installed on High Sierry though :/

Paramedic, the tool running the tests on Saucelabs, was hardcoded to use macOS Sierra for Safari tests. I changed it to use High Sierra (still hardcoded, should be changed) but this doesn't succeed currently. Investigating.

@janpio
Copy link
Member

janpio commented Sep 29, 2018

Turns out Safari on High Sierra is/has to be 11.1 instead of 10.1 on Sierra. Tests are now running again.

@janpio
Copy link
Member

janpio commented Sep 29, 2018

And success.
I think I fixed it.
At least some of the previously failing tests succeed now.

@ffMathy
Copy link
Author

ffMathy commented Sep 29, 2018

Great! So we can close this now?

@janpio
Copy link
Member

janpio commented Sep 29, 2018

No, not until I confirmed that this is not just a fluke but a systemic fix that works all the time. Doing that now.

@janpio
Copy link
Member

janpio commented Sep 30, 2018

At least in this repo the tests that failed because of that are not failing any more. Yay.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants