Wrong exit status when testing URLs that do not work

[soulgalore]

$ bin/sitespeed.js https://failing -n 1
...
$ echo $?
0

[beenanner]

Looks like this is caused by https://github.com/sitespeedio/sitespeed.io/blob/master/bin/sitespeed.js#L39 being undefined by default. Maybe we should had a default budget that the initial URL must load with a 2xx or 3xx.

[soulgalore]

good, that is one thing: that check should only run when we configured a budget, I'll fix that first.

[soulgalore]

I think we can just check for errors in the budget, I've missed that when I implemented it.

[soulgalore]

no that will not work :) lets think about it for a while.

[beenanner]

We possible might need something upstream returned in browsertime for the status of the initial URL. The returned object here doesn't know about the failed URL.
https://github.com/sitespeedio/sitespeed.io/blob/master/lib/sitespeed.js#L126

[soulgalore]

Yes. I think there's an issue to add an error element in the JSON from Browsertime, but it will be some work so it has been put aside. One ugly solution for now could be to compare number of URLs in the budget and the ones tested or something like that.

[jchoksi-whitbread]

Do you think it would be possible to somehow check for the test URL failing to load in the --browsertime.preScript and do something in the script to stop the test? If so, what could we do to stop the sitespeed.io process from within the preScript?

I'm basically looking for a way to get notified when a Jenkins job running sitespeed.io against a URL fails because the target site is not up. Currently, as per above, the exit code returned is still 0 for v5.6.5:

[2017-11-01 18:33:28] INFO: Versions OS: linux 4.9.0-4-amd64 nodejs: v6.11.5 sitespeed.io: 5.6.5 browsertime: 1.9.4 coach: 0.36.0
[2017-11-01 18:33:28] INFO: Starting chrome for analysing https://failing 1 time(s)
[2017-11-01 18:33:28] INFO: Testing url https://failing run 1
[2017-11-01 18:33:32] ERROR: Could not load URLUrlLoadError: Failed to load https://failing
[2017-11-01 18:33:32] ERROR: https://failing generated the following error in Browsertime UrlLoadError: Failed to load https://failing
    at BrowsertimeError (/opt/npm-global/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/support/errors.js:5:5)
    at UrlLoadError (/opt/npm-global/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/support/errors.js:19:5)
    at getUrl.then.then.catch.catch.e (/opt/npm-global/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/core/seleniumRunner.js:175:15)
From previous event:
    at SeleniumRunner.loadAndWait (/opt/npm-global/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/core/seleniumRunner.js:173:13)
    at Promise.resolve.tap.tap.tap.tap.tap (/opt/npm-global/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/core/engine.js:333:27)
From previous event:
    at runIteration (/opt/npm-global/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/core/engine.js:333:10)
    at Promise.reduce (/opt/npm-global/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/core/engine.js:409:27)
From previous event:
    at Promise.resolve.tap.tap.tap.tap.result (/opt/npm-global/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/core/engine.js:406:17)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)
From previous event:
    at Engine.run (/opt/npm-global/lib/node_modules/sitespeed.io/node_modules/browsertime/lib/core/engine.js:405:8)
    at engine.start.then (/opt/npm-global/lib/node_modules/sitespeed.io/lib/plugins/browsertime/analyzer.js:95:26)
From previous event:
    at Object.analyzeUrl (/opt/npm-global/lib/node_modules/sitespeed.io/lib/plugins/browsertime/analyzer.js:95:8)
    at storageManager.createDirForUrl.then.then (/opt/npm-global/lib/node_modules/sitespeed.io/lib/plugins/browsertime/index.js:173:32)
    at /opt/npm-global/lib/node_modules/sitespeed.io/node_modules/mkdirp/index.js:30:20
    at FSReqWrap.oncomplete (fs.js:123:15)
From previous event:
    at Object.processMessage (/opt/npm-global/lib/node_modules/sitespeed.io/lib/plugins/browsertime/index.js:173:12)
    at queue.process.message (/opt/npm-global/lib/node_modules/sitespeed.io/lib/support/queueHandler.js:192:32)
    at drainItem (/opt/npm-global/lib/node_modules/sitespeed.io/node_modules/concurrent-queue/index.js:96:21)
    at Immediate.drain (/opt/npm-global/lib/node_modules/sitespeed.io/node_modules/concurrent-queue/index.js:77:84)
    at runCallback (timers.js:672:20)
    at tryOnImmediate (timers.js:645:5)
    at processImmediate [as _immediateCallback] (timers.js:617:5)
[2017-11-01 18:33:34] INFO: HTML stored in /opt/jenkins/home/d/sitespeed-result/failing/2017-11-01-18-33-28
[2017-11-01 18:33:34] INFO: Finished analysing https://failing
0

[soulgalore]

@jchoksi-whitbread I've totally missed this one, next week I'll work 100% with sitespeed.io projects, I'll fix this then.

[soulgalore]

This was pushed in 6.0, it catches if one of the URLs fail. However we still have work 2do to make sure some errors give right exit code.

[vs-odessa]

@soulgalore Can't find another similar issue, maybe there are some updates here

1. Is it possible to set a non-0 exit code on any error that occurred in the Sitespeed test without budget config?
2. If exit code can be managed only with budget config - is it possible to configure it or errors number or percentage?
   I mean errors like not found element on the page, can't click on an element etc.