Having many browsers fails #40

wnr · 2014-06-04T12:25:43Z

Hi,

I have 11 browsers that I want to run on saucelabs. When running the browsers in batches of 4-4-3, everything works perfectly. When running all 11 browsers, things are not working.

This is what's happening:

I get the INFO message in my console saying INFO [launcher]: Starting browser ... on SauceLabs 11 times, which is expected.
I see all 11 browsers appear on SauceLabs, queueing to run (I can only run 2 in parallel).
Two of the 11 queued browsers on SauceLabs starts to run, and the session message appears in the console `INFO [launcher.sauce]: safari 7 (Mac 10.9) session at https://saucelabs.com/tests/0b0c09c45f7144a3837dadd7c14888ba
INFO [launcher.sauce]: firefox 28 (Mac 10.9) session at https://saucelabs.com/tests/6474f7dee7fb4e0cb9a9e7deb3516a5f``
Nothing happens.
Selenium reports error of the two running browsers: "Test did not see a new command for 90 seconds. Timing out."
3 to 5 repeats for all remaining queued browsers.

Any thoughts?

Thanks

The text was updated successfully, but these errors were encountered:

ChrisWren · 2014-06-27T03:56:28Z

Maybe we could check the maximum concurrency for a user and only run that number of tests at any point in time: https://docs.saucelabs.com/reference/rest-api/#get-user-concurrency

wnr · 2014-06-27T10:21:46Z

That sounds great!

pkozlowski-opensource · 2014-06-27T10:59:39Z

I can see a lot of those failures as well when running tests in batches of > 3-4 browsers. Due to this I was obliged to split my build into several steps. Not a big deal but not ideal either. While I agree that we could limit the number of browser tests started by Karma (I'm still not familiar with Karma internals enough to say how much effort it would involve) maybe there are some parts that we could try to "fix" in the launcher / sauce itself?

I must say that I've spent already substantial amount of time trying to track-down those issues, fiddled with timeouts etc. but with limited luck so far.... I would be willing to devote some more time to understanding / fixing those issues if someone could give me a hand with some aspect of Saucelabs working. @ChrisWren could we somehow co-operate on this topic? I think that having good Karma + Sauce integration would be beneficial for the whole JS community.

alexgorbatchev · 2014-07-06T23:52:55Z

👍 on this, lots of errors related to timeouts...

INFO [launcher]: Trying to start safari 7 (OS X 10.9) on SauceLabs again (1/2).
INFO [launcher]: Trying to start safari 7 (OS X 10.9) on SauceLabs again (2/2).
INFO [launcher]: Trying to start chrome 35 (OS X 10.9) on SauceLabs again (2/2).
ERROR [launcher.sauce]: Can not start chrome 34 (OS X 10.9)
  [get("http://localhost:9876/?id=44559012")] Unexpected data in simpleCallback.
  ERROR Job 0430f4d2177141c2a9beaa370ab0f6f6 is not in progress. It may have recently finished, or experienced an error. You can learn more at https://saucelabs.com/jobs/0430f4d2177141c2a9beaa370ab0f6f6
ERROR [launcher]: chrome 34 (OS X 10.9) on SauceLabs failed 2 times (timeout). Giving up.
ERROR [launcher.sauce]: Can not start safari 6 (OS X 10.8)
  [get("http://localhost:9876/?id=48254307")] Unexpected data in simpleCallback.
  ERROR Job 5ebe9943b71a4b4ea001c5511a5eb448 is not in progress. It may have recently finished, or experienced an error. You can learn more at https://saucelabs.com/jobs/5ebe9943b71a4b4ea001c5511a5eb448
ERROR [launcher]: safari 6 (OS X 10.8) on SauceLabs failed 2 times (timeout). Giving up.
ERROR [launcher]: safari 6 (OS X 10.8) on SauceLabs failed 2 times (timeout). Giving up.
ERROR [launcher]: chrome 34 (OS X 10.9) on SauceLabs failed 2 times (timeout). Giving up.
WARN [launcher]: safari 7 (OS X 10.9) on SauceLabs have not captured in 60000 ms, killing.
ERROR [launcher]: safari 7 (OS X 10.9) on SauceLabs failed 2 times (timeout). Giving up.
WARN [launcher]: chrome 35 (OS X 10.9) on SauceLabs have not captured in 60000 ms, killing.
ERROR [launcher]: chrome 35 (OS X 10.9) on SauceLabs failed 2 times (timeout). Giving up.
INFO [launcher.sauce]: Shutting down Sauce Connect

and on the saucelabs side of things:

if i reduces number of browsers to about 4 they all pass

ChrisWren · 2014-07-07T05:38:32Z

The sauce launcher gets called for every browser configured in the customLaunchers object so we don't have an easy way to manage concurrency externally without modifying karma's code. I guess we could have some internal system registering requested browsers and then hit the Sauce concurrency REST API and see if there is available concurrency to run the test. Then when the concurrency is maxed out and a session completes we can start another browser.

alexgorbatchev · 2014-07-08T13:46:32Z

I switched the test to run in all browsers in series like so

#!/bin/bash

test() {
 ./node_modules/karma/bin/karma start --single-run --no-auto-watch
}

if [ -n "$SAUCE_USERNAME" ] && [ -n "$SAUCE_ACCESS_KEY" ]; then
  BROWSER='safari'              VERSION='5'    PLATFORM='OS X 10.6'   test &&
  BROWSER='safari'              VERSION='6'    PLATFORM='OS X 10.8'   test &&
  BROWSER='safari'              VERSION='7'    PLATFORM='OS X 10.9'   test &&
  BROWSER='chrome'              VERSION='35'   PLATFORM='OS X 10.9'   test &&
  BROWSER='chrome'              VERSION='34'   PLATFORM='OS X 10.9'   test &&
  BROWSER='firefox'             VERSION='29'   PLATFORM='OS X 10.9'   test &&
  BROWSER='firefox'             VERSION='30'   PLATFORM='OS X 10.9'   test &&
  BROWSER='internet explorer'   VERSION='9'    PLATFORM='Windows 7'   test &&
  BROWSER='internet explorer'   VERSION='10'   PLATFORM='Windows 7'   test &&
  BROWSER='internet explorer'   VERSION='11'   PLATFORM='Windows 7'   test
else
  test
fi

There are a couple of "issues" with this:

It's pretty slow because I believe the tunnel is being opened and closed every time. Looking at sauce-connect-launcher it doesn't seem like it's built to reuse existing tunnel. Perhaps karma-sauce-launcher could check if there's a selenium running before executing sauce-connect-launcher.
I'm still seeing a lot of WARN [launcher]: chrome 34 (OS X 10.9) on SauceLabs have not captured in 60000 ms, killing. errors. Curious if this is a karma issue? I'm not certain what is the root cause here.

thom4parisot · 2014-07-10T10:10:47Z

Very interested to know more about it because all my CI stuff is broken without me knowing where it belongs really.

Krinkle · 2014-07-18T17:13:03Z

Having no option to specify (let alone auto-detect from the user account via the REST API) what the concurrency should be basically means this always fails when using more than 2 to 4 browsers and forces users to fragment run targets (e.g. in Grunt) – which slows things down quite a bit (not in the least due to #14).

I reckon the same applies to BrowserStack and other cloud services. Perhaps that justifies a change to the core Karma software to support this natively?

Something like:

option.browserConcurrency (Number|Function) Number (or function that invokes its callback parameter with a number), indicating the number of browsers to launch at once. Defaults to null which indicates all browsers will be launched at once.

The main complication I expect will come up relates to the ability to run Karma continuously in the background. It would have to kill and restart browsers each run.

EDIT: On second thought, while it still needs support from Karma internally, the actual concurrency limit should probably be handled by the launcher so that one can have e.g. unlimited local browsers that run continuously, with 10 browserstack and/or 3 saucelabs browsers being killed after each run.

nkzawa · 2014-09-01T15:54:20Z

I had the same issue here.

markthethomas · 2014-09-10T21:25:45Z

Running into the same thing. Is there more-native (to SauceLabs) way besides multiple grunt tasks that could be implemented? Seems like @Krinkle's edit would be a great way to accomplish this.

bvarga · 2014-10-15T12:16:49Z

I would like to share how I put my tasks into batches. In the command line I call the karma tests like this:

karma start karma.saucelabs.conf.js --batch-chrome1
karma start karma.saucelabs.conf.js --batch-chrome2

In the ´karma.saucelabs.conf.js´ file reading the batch id example: chrome1 with the help of underscore

var _ = require('underscore');
var batchId = null;
var barg = _.find(process.argv, function(arg){
  return arg.indexOf('--batch-') === 0;
});
if (barg) {
  batchId = barg.substr(8);
}

than defining the launchers, with one extra parameter batch:

var customLaunchers = {
  sl_chrome_36: {
    base: 'SauceLabs',
    browserName: 'chrome',
    platform: 'Windows 7',
    version: '36',
    batch: 'chrome1',
  },
  sl_chrome_37: {
    base: 'SauceLabs',
    browserName: 'chrome',
    platform: 'Windows 7',
    version: '37',
    batch: 'chrome2',
  },
...
}

and than filtering out the specified part:

customLaunchers = _.reduce(customLaunchers, function(memo, value, key){
  if (!batchId || batchId === value.batch) {
    memo[key] = _.omit(value, 'batch');
  }
  return memo;
},{});

Also I agree that an option like ´browserConcurrency´ would be the best.

teppeis · 2014-12-02T16:44:56Z

zuul has concurrency option. The same option is needed.

JamesMGreene · 2015-01-23T17:48:39Z

Still seeing the same issues today:

mightyiam · 2015-02-09T14:06:27Z

🐹

due to karma-runner/karma-sauce-launcher#40

MoOx · 2015-04-13T05:53:52Z

Getting so many issue with too many browsers. It make this plugin so painful to use :/
Is there someone here that might handle something ?
poke #64

monolithed · 2015-04-28T23:58:41Z

+1, there're many strange errors https://travis-ci.org/monolithed/polyfiller/jobs/60458589

Krinkle · 2015-04-29T00:55:23Z

@monolithed Unrelated. Your build fails with error "Uh oh, you've run out of minutes!". So.. may wanna check your SauceLabs account and/or contact their support. 😉

Especially services like Browserstack and SauceLabs have limitations on how many browsers can be launched at the same time. The new config option `concurrency` allows to specify an upper limit of how many browsers are allowed to run at the same time. Ref: karma-runner/karma-sauce-launcher#40 Closes karma-runner#1465

dignifiedquire · 2015-10-21T19:50:04Z

Concurrency limit will be shipped tomorrow in karma :) karma-runner/karma#1646 sorry for the delay

Especially services like Browserstack and SauceLabs have limitations on how many browsers can be launched at the same time. The new config option `concurrency` allows to specify an upper limit of how many browsers are allowed to run at the same time. Ref: karma-runner/karma-sauce-launcher#40 Closes karma-runner#1465

dignifiedquire · 2015-10-22T12:40:15Z

Fixed via the new concurrency option in karma@0.13.12 :)

JamesMGreene · 2015-12-25T18:18:20Z

Updated REST API doc location for getting the user's concurrency limit:

https://wiki.saucelabs.com/display/DOCS/Account+Methods#AccountMethods-GetUserConcurrency

Or mobile version:

https://wiki.saucelabs.com/plugins/servlet/mobile#AccountMethods-GetUserConcurrency

GuillaumeLeclerc · 2016-06-25T09:42:18Z

I still face the same problem any update ?

mightyiam · 2016-06-25T10:24:33Z

@GuillaumeLeclerc are you using 0.13.12 or newer? Are you using the concurrency option?

Per karma-runner/karma-sauce-launcher#40, it seems that we can only run 1 browser at a time in sauce labs. This updates karma to a version that supports the `concurrency` option.

karma-runner/karma-sauce-launcher#40 karma-runner/karma#1646

pkozlowski-opensource mentioned this issue Jun 30, 2014

pendingHeartBeat timeout not cleared properly #42

Closed

Krinkle mentioned this issue Jul 18, 2014

captureTimeout should start after a browser is granted #37

Closed

ndhoule mentioned this issue Sep 1, 2014

concurrent Sauce Labs runs time out ndhoule/fungus#52

Open

flibbertigibbet mentioned this issue Jan 7, 2015

Reduce number of browsers to run in Sauce Labs azavea/nih-wayfinding#17

Merged

mightyiam added a commit to PolicyStat/edited that referenced this issue Feb 9, 2015

karma-sauce-launcher: test with few browsers

eeabd68

due to karma-runner/karma-sauce-launcher#40

megawac mentioned this issue Feb 21, 2015

Set up sauce test runner jashkenas/backbone#3510

Merged

2 tasks

ndhoule mentioned this issue Jun 10, 2015

Add browser testing to CI mochajs/mocha#1732

Closed

dignifiedquire mentioned this issue Oct 21, 2015

feat(launcher): Add concurrency limit karma-runner/karma#1646

Merged

dignifiedquire added the type: feature label Oct 21, 2015

dignifiedquire closed this as completed Oct 22, 2015

This was referenced Oct 29, 2015

Concurrency limit #8

Closed

Use concurrency skatejs/skatejs#426

Closed

chore(saucelabs): use concurrency option introduced in Karma 0.13.12 skatejs/skatejs#427

Merged

chore(saucelabs): use concurrency option introduced in Karma 0.13.12 skatejs/build#3

Merged

maciejsmolinski mentioned this issue Mar 16, 2016

Logging: Make sure logging is bound with a correct context. mateuszgachowski/Carpet.js#10

Open

eventualbuddha mentioned this issue Jul 13, 2016

chore: fix sauce labs tests eventualbuddha/keysim.js#33

Merged

lucasfcosta mentioned this issue Jul 20, 2016

Several open Greenkeeper-PRs chaijs/type-detect#55

Closed

Uzlopak added a commit to parallax/jsPDF that referenced this issue Feb 15, 2018

Update saucelabs.karma.conf.js

c235adc

karma-runner/karma-sauce-launcher#40 karma-runner/karma#1646

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Having many browsers fails #40

Having many browsers fails #40

wnr commented Jun 4, 2014

ChrisWren commented Jun 27, 2014

wnr commented Jun 27, 2014

pkozlowski-opensource commented Jun 27, 2014

alexgorbatchev commented Jul 6, 2014

ChrisWren commented Jul 7, 2014

alexgorbatchev commented Jul 8, 2014

thom4parisot commented Jul 10, 2014

Krinkle commented Jul 18, 2014

nkzawa commented Sep 1, 2014

markthethomas commented Sep 10, 2014

bvarga commented Oct 15, 2014

teppeis commented Dec 2, 2014

JamesMGreene commented Jan 23, 2015

mightyiam commented Feb 9, 2015

MoOx commented Apr 13, 2015

monolithed commented Apr 28, 2015

Krinkle commented Apr 29, 2015

dignifiedquire commented Oct 21, 2015

dignifiedquire commented Oct 22, 2015

JamesMGreene commented Dec 25, 2015

GuillaumeLeclerc commented Jun 25, 2016

mightyiam commented Jun 25, 2016

Having many browsers fails #40

Having many browsers fails #40

Comments

wnr commented Jun 4, 2014

ChrisWren commented Jun 27, 2014

wnr commented Jun 27, 2014

pkozlowski-opensource commented Jun 27, 2014

alexgorbatchev commented Jul 6, 2014

ChrisWren commented Jul 7, 2014

alexgorbatchev commented Jul 8, 2014

thom4parisot commented Jul 10, 2014

Krinkle commented Jul 18, 2014

nkzawa commented Sep 1, 2014

markthethomas commented Sep 10, 2014

bvarga commented Oct 15, 2014

teppeis commented Dec 2, 2014

JamesMGreene commented Jan 23, 2015

mightyiam commented Feb 9, 2015

MoOx commented Apr 13, 2015

monolithed commented Apr 28, 2015

Krinkle commented Apr 29, 2015

dignifiedquire commented Oct 21, 2015

dignifiedquire commented Oct 22, 2015

JamesMGreene commented Dec 25, 2015

GuillaumeLeclerc commented Jun 25, 2016

mightyiam commented Jun 25, 2016