Crawl doesn't work with basic/auth

[soulgalore]

Hi @soulgalore - I don't know if I should open a new issue for this, but when i use the ugly fix (http://u:p@domain.com) on a crawl, sitespeed.io doesn't crawl, just gives me the result for the first page. Is there something I am doing wrong?

Command running is sitespeed.io http://user:pass@sub.domain.com/ -d 2 and I get the output

[2017-03-10 08:57:17] Versions OS: darwin 16.4.0 nodejs: v7.0.0 sitespeed.io: 4.5.1 browsertime: 1.0.0-beta.28 coach: 0.31.0
[2017-03-10 08:57:17] Starting chrome for analysing http://user:pass@sub.domain.com/ 3 time(s)
[2017-03-10 08:57:17] Testing url http://user:pass@sub.domain.com/ run 1
[2017-03-10 08:57:23] Testing url http://user:pass@sub.domain.com/ run 2
[2017-03-10 08:57:29] Testing url http://user:pass@sub.domain.com/ run 3
[2017-03-10 08:57:37] 88 requests, 2129.09 kb, firstPaint: 821ms (±73.57ms), DOMContentLoaded: 1.13s (±126.04ms), Load: 2.09s (±360.68ms), rumSpeedIndex: 1171 (±142.75) (3 runs)
[2017-03-10 08:57:38] Render HTML for 1 page(s)
[2017-03-10 08:57:40] HTML stored in /Users/jbeaver/Desktop/sitespeed-result/sub.domain.com/2017-03-10-08-57-17
[2017-03-10 08:57:40] Finished analysing http://user:pass@sub.domain.com/

Is there any way to make a crawl work with this method?

Reported by @joebeaver89

[joebeaver89]

Thanks for opening this for me @soulgalore, appreciated

[papandreou]

A shot in the dark: Maybe --browsertime-requestheader 'Authorization: Basic '$(echo -n u:p | base64) would work?

[soulgalore]

Hi @papandreou yes you are right, from latest release we can do request headers in Chrome (we wait on next Selenium release for Firefox, @tobli has fix there that needs to be released). For the crawler I see that I missed adding support for request header, let me do that tonight so it's there for next release.

Best
Peter

[soulgalore]

Ooops I never fixed that. If someone else has time that would be great :)

[fayazmd]

Is this still in unresolved state.? Currently, I am using url as follows,

https://abcuser@xyzcompany.com:passwordtest@10.1.179.39

And failing. Is there any work around.?

Note: here abcuser@xyzcompany.com is the username and passwordtest is the password

[soulgalore]

It should be implemented in the crawler https://github.com/sitespeedio/sitespeed.io/blob/master/lib/plugins/crawler/index.js - and I think Simplecrawler, do you have time to have a go @fayazmd ? I can help out test it?

[fayazmd]

Hey, I have tried to analysis the index.js file started with question, how are we passing creds..?? and found the lines.. (line no 46 - line no 51),

        if (this.basicAuth) {
           const userAndPassword = this.basicAuth.split('@');
           crawler.needsAuth = true;
           crawler.authUser = userAndPassword[0];
           crawler.authPass = userAndPassword[1];
        }

we are basically splitting based on '@' for the input value, https://u:p@domain.com, doesn't it end up at like, https://u:p in index 0th position and domain.com at index 1st position.?

My query is,
1) Is that how we really need to consider.? If answer is 'No', we need to update the code (in line no. 47) to extract 'username' and 'password' from the input string "https://u:p@domain.com" as

const userAndPassword = this.basicAuth.substring(this.basicAuth.lastIndexOf('/') + 1, this.basicAuth.lastIndexOf('@')).split(':');

Please correct me, if I am wrong. Sorry for my limited knowledge

Note: the above code doesn't bother existence of '@' either in username or password. In other words, if my code is good to consider, then we can have '@' in username and password as well.

[soulgalore]

Hey @fayazmd you are right, there is an implementation but maybe then not working, sorry I forgot about it and didn't check the code :(

No it should work like this: You supply the basicAuth on the side so you add the URL and the --basicAuth userName@password or --browsertime.basicAuth (we got a shorthand for that) and then we take that option from browsertime and use it in the crawler: https://github.com/sitespeedio/sitespeed.io/blob/master/lib/plugins/crawler/index.js#L17

So according to the code this should work:
sitespeed.io --basicAuth user@password --crawler.depth 2 https://your.url

But it doesn't or? I don't have a site behind basic auth at the moment.

[soulgalore]

Could someone help us test if this work or not?

[jholsgrove]

Hi @soulgalore - I was looking for something else and decided to get distracted with this. When tested against The Internet - browsertime did throw an error.

I'm using windows/ docker/ latest sitespeed as of time of writing.

Cmd:

docker run -v /c/users/jholsgrove:/sitespeed.io sitespeedio/sitespeed.io --basicAuth admin@admin -d 2 -b chrome https://the-internet.herokuapp.com/basic_auth --viewPort 1920x1080 --video false

"{ "browsertime": "Error" }"

From the log:

[2018-06-22 14:08:30] ERROR: [root] Error: EISDIR: illegal operation on a directory, unlink '/sitespeed.io/sitespeed-result/the-internet.herokuapp.com/2018-06-22-14-07-32/pages/the-internet.herokuapp.com/basic_auth/data/video/images''

When run with video enabled, the Docker cmd window came up with the following...

Docker window:

[2018-06-22 13:54:00] INFO: Testing url https://the-internet.herokuapp.com/basic_auth iteration 1
[2018-06-22 13:54:34] ERROR: {"frames":[{"tags":{"lavfi.signalstats.SATAVG":"81.8692"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8712"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8712"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8712"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"0.153764"}}]}
[2018-06-22 13:54:34] ERROR: Error: Command failed: ffmpeg -nostdin -ss NaN -t NaN -i /sitespeed.io/sitespeed-result/the-internet.herokuapp.com/2018-06-22-13-53-57/pages/the-internet.herokuapp.com/basic_auth/data/video/1.mp4 -c:v libx264 -an -vf drawtext=timecode='00:00:00:00':rate=30: x=(w-280)/2: y=H-60:fontcolor=white:fontsize=48:box=1:boxcolor=0x000000AA,drawtext=enable='between(t,0,30)':x=(w-tw)/2: y=H-230:fontcolor=white:fontsize=26:box=1:boxcolor=0x000000AA:boxborderw=2:text='FirstVisualChange 0',drawtext=enable='between(t,0,30)':x=(w-tw)/2: y=H-200:fontcolor=white:fontsize=26:box=1:boxcolor=0x000000AA:boxborderw=2:text='SpeedIndex 0',drawtext=enable='between(t,0,30)':x=(w-tw)/2: y=H-170:fontcolor=white:fontsize=26:box=1:boxcolor=0x000000AA:boxborderw=2:text='VisualComplete85 0',drawtext=enable='between(t,0,30)':x=(w-tw)/2: y=H-140:fontcolor=white:fontsize=26:box=1:boxcolor=0x000000AA:boxborderw=2:text='LastVisualChange 0',drawtext=enable='between(t,2.655,30)':x=(w-tw)/2: y=H-110:fontcolor=white:fontsize=26:box=1:boxcolor=0x000000AA:boxborderw=2:text='DOMContentLoaded 2655' -y /sitespeed.io/sitespeed-result/the-internet.herokuapp.com/2018-06-22-13-53-57/pages/the-internet.herokuapp.com/basic_auth/data/video/tmp.mp4
ffmpeg version 4.0-static https://johnvansickle.com/ffmpeg/ Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-libxml2 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg
libavutil 56. 14.100 / 56. 14.100
libavcodec 58. 18.100 / 58. 18.100
libavformat 58. 12.100 / 58. 12.100
libavdevice 58. 3.100 / 58. 3.100
libavfilter 7. 16.100 / 7. 16.100
libswscale 5. 1.100 / 5. 1.100
libswresample 3. 1.100 / 3. 1.100
libpostproc 55. 1.100 / 55. 1.100
Invalid duration specification for ss: NaN

The report gives me metrics, a screen grab of a successful login and a video which shows the successful bit, I guess it has a hard time rendering/ capturing the auth part.

I can perhaps pull more info out if you're able to point me in the right direction, but hope that helps nonetheless.

[soulgalore]

Hi @jholsgrove thanks for trying it out!

First about the video: It fails when we try to find the first frame that isn't orange ... Think something goes wrong with the URL (I can see it is orange and then only one frame that changes). If you have time you can add --browsertime.videoParams.keepOriginalVideo to your run, then the original (uncut) video will be saved, and then we can see what's going on.

docker run -v /c/users/jholsgrove:/sitespeed.io sitespeedio/sitespeed.io --basicAuth admin@admin -d 2 -b chrome https://the-internet.herokuapp.com/basic_auth --viewPort 1920x1080 --video false

It's not so obvoius but we have both video and visual metrics on by default so to disable fully recording a video you need --video false --visualMetrics false. Sorry! Can you try with that?

Best
Peter

[jholsgrove]

Thanks @soulgalore - seems to have cleared that part up! The video just gave me a white screen/ no visuals with --browsertime.videoParams.keepOriginalVideo, fully disabling got rid of the errors, yay!

With regard to the original bug it doesn't seem to crawl the additional links to elemental selenium and the github fork using the -d 2 param. Verbose log doesn't seem to indicate anything going wrong. I am facing a similar issue with a site I use which is HTTPS. Is there much else I can do to help?

Creds for the basicAuth test site are admin/admin

[soulgalore]

Thanks @jholsgrove hmm ok, but that URL, does it have any URLS to crawl? It will only crawl on the same host if I remember correctly.

Best
Peter

[jholsgrove]

Ah yes, a slight oversight on my part there. I rigged up my own basic auth site and added a link and it crawled just fine. I think you can safely close this particular issue @soulgalore :)

If you would like to see yourself: www.goatpod.com/basicAuth
Creds: admin/admin

[soulgalore]

Closing, please re-open if you have any issues :)