Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crawl doesn't work with basic/auth #1506

Closed
soulgalore opened this issue Mar 10, 2017 · 15 comments
Closed

Crawl doesn't work with basic/auth #1506

soulgalore opened this issue Mar 10, 2017 · 15 comments
Labels

Comments

@soulgalore
Copy link
Member

Hi @soulgalore - I don't know if I should open a new issue for this, but when i use the ugly fix (http://u:p@domain.com) on a crawl, sitespeed.io doesn't crawl, just gives me the result for the first page. Is there something I am doing wrong?

Command running is sitespeed.io http://user:pass@sub.domain.com/ -d 2 and I get the output

[2017-03-10 08:57:17] Versions OS: darwin 16.4.0 nodejs: v7.0.0 sitespeed.io: 4.5.1 browsertime: 1.0.0-beta.28 coach: 0.31.0
[2017-03-10 08:57:17] Starting chrome for analysing http://user:pass@sub.domain.com/ 3 time(s)
[2017-03-10 08:57:17] Testing url http://user:pass@sub.domain.com/ run 1
[2017-03-10 08:57:23] Testing url http://user:pass@sub.domain.com/ run 2
[2017-03-10 08:57:29] Testing url http://user:pass@sub.domain.com/ run 3
[2017-03-10 08:57:37] 88 requests, 2129.09 kb, firstPaint: 821ms (±73.57ms), DOMContentLoaded: 1.13s (±126.04ms), Load: 2.09s (±360.68ms), rumSpeedIndex: 1171 (±142.75) (3 runs)
[2017-03-10 08:57:38] Render HTML for 1 page(s)
[2017-03-10 08:57:40] HTML stored in /Users/jbeaver/Desktop/sitespeed-result/sub.domain.com/2017-03-10-08-57-17
[2017-03-10 08:57:40] Finished analysing http://user:pass@sub.domain.com/

Is there any way to make a crawl work with this method?

Reported by @joebeaver89

@joebeaver89
Copy link

Thanks for opening this for me @soulgalore, appreciated

@papandreou
Copy link

A shot in the dark: Maybe --browsertime-requestheader 'Authorization: Basic '$(echo -n u:p | base64) would work?

@soulgalore
Copy link
Member Author

soulgalore commented May 2, 2017

Hi @papandreou yes you are right, from latest release we can do request headers in Chrome (we wait on next Selenium release for Firefox, @tobli has fix there that needs to be released). For the crawler I see that I missed adding support for request header, let me do that tonight so it's there for next release.

Best
Peter

@soulgalore
Copy link
Member Author

soulgalore commented Nov 30, 2017

Ooops I never fixed that. If someone else has time that would be great :)

@fayazmd
Copy link

fayazmd commented May 29, 2018

Is this still in unresolved state.? Currently, I am using url as follows,

https://abcuser@xyzcompany.com:passwordtest@10.1.179.39

And failing. Is there any work around.?

Note: here abcuser@xyzcompany.com is the username and passwordtest is the password

@soulgalore
Copy link
Member Author

It should be implemented in the crawler https://github.com/sitespeedio/sitespeed.io/blob/master/lib/plugins/crawler/index.js - and I think Simplecrawler, do you have time to have a go @fayazmd ? I can help out test it?

@fayazmd
Copy link

fayazmd commented Jun 1, 2018

Hey, I have tried to analysis the index.js file started with question, how are we passing creds..?? and found the lines.. (line no 46 - line no 51),

        if (this.basicAuth) {
           const userAndPassword = this.basicAuth.split('@');
           crawler.needsAuth = true;
           crawler.authUser = userAndPassword[0];
           crawler.authPass = userAndPassword[1];
        }

we are basically splitting based on '@' for the input value, https://u:p@domain.com, doesn't it end up at like, https://u:p in index 0th position and domain.com at index 1st position.?

My query is,
1) Is that how we really need to consider.? If answer is 'No', we need to update the code (in line no. 47) to extract 'username' and 'password' from the input string "https://u:p@domain.com" as

const userAndPassword = this.basicAuth.substring(this.basicAuth.lastIndexOf('/') + 1, this.basicAuth.lastIndexOf('@')).split(':');

Please correct me, if I am wrong. Sorry for my limited knowledge

Note: the above code doesn't bother existence of '@' either in username or password. In other words, if my code is good to consider, then we can have '@' in username and password as well.

@soulgalore
Copy link
Member Author

Hey @fayazmd you are right, there is an implementation but maybe then not working, sorry I forgot about it and didn't check the code :(

No it should work like this: You supply the basicAuth on the side so you add the URL and the --basicAuth userName@password or --browsertime.basicAuth (we got a shorthand for that) and then we take that option from browsertime and use it in the crawler: https://github.com/sitespeedio/sitespeed.io/blob/master/lib/plugins/crawler/index.js#L17

So according to the code this should work:
sitespeed.io --basicAuth user@password --crawler.depth 2 https://your.url

But it doesn't or? I don't have a site behind basic auth at the moment.

@soulgalore
Copy link
Member Author

Could someone help us test if this work or not?

@jholsgrove
Copy link

jholsgrove commented Jun 22, 2018

Hi @soulgalore - I was looking for something else and decided to get distracted with this. When tested against The Internet - browsertime did throw an error.

I'm using windows/ docker/ latest sitespeed as of time of writing.

Cmd:

docker run -v /c/users/jholsgrove:/sitespeed.io sitespeedio/sitespeed.io --basicAuth admin@admin -d 2 -b chrome https://the-internet.herokuapp.com/basic_auth --viewPort 1920x1080 --video false

"{ "browsertime": "Error" }"

From the log:

[2018-06-22 14:08:30] ERROR: [root] Error: EISDIR: illegal operation on a directory, unlink '/sitespeed.io/sitespeed-result/the-internet.herokuapp.com/2018-06-22-14-07-32/pages/the-internet.herokuapp.com/basic_auth/data/video/images''

When run with video enabled, the Docker cmd window came up with the following...

Docker window:

[2018-06-22 13:54:00] INFO: Testing url https://the-internet.herokuapp.com/basic_auth iteration 1
[2018-06-22 13:54:34] ERROR: {"frames":[{"tags":{"lavfi.signalstats.SATAVG":"81.8692"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8712"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8712"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8712"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"81.8731"}},{"tags":{"lavfi.signalstats.SATAVG":"0.153764"}}]}
[2018-06-22 13:54:34] ERROR: Error: Command failed: ffmpeg -nostdin -ss NaN -t NaN -i /sitespeed.io/sitespeed-result/the-internet.herokuapp.com/2018-06-22-13-53-57/pages/the-internet.herokuapp.com/basic_auth/data/video/1.mp4 -c:v libx264 -an -vf drawtext=timecode='00:00:00:00':rate=30: x=(w-280)/2: y=H-60:fontcolor=white:fontsize=48:box=1:boxcolor=0x000000AA,drawtext=enable='between(t,0,30)':x=(w-tw)/2: y=H-230:fontcolor=white:fontsize=26:box=1:boxcolor=0x000000AA:boxborderw=2:text='FirstVisualChange 0',drawtext=enable='between(t,0,30)':x=(w-tw)/2: y=H-200:fontcolor=white:fontsize=26:box=1:boxcolor=0x000000AA:boxborderw=2:text='SpeedIndex 0',drawtext=enable='between(t,0,30)':x=(w-tw)/2: y=H-170:fontcolor=white:fontsize=26:box=1:boxcolor=0x000000AA:boxborderw=2:text='VisualComplete85 0',drawtext=enable='between(t,0,30)':x=(w-tw)/2: y=H-140:fontcolor=white:fontsize=26:box=1:boxcolor=0x000000AA:boxborderw=2:text='LastVisualChange 0',drawtext=enable='between(t,2.655,30)':x=(w-tw)/2: y=H-110:fontcolor=white:fontsize=26:box=1:boxcolor=0x000000AA:boxborderw=2:text='DOMContentLoaded 2655' -y /sitespeed.io/sitespeed-result/the-internet.herokuapp.com/2018-06-22-13-53-57/pages/the-internet.herokuapp.com/basic_auth/data/video/tmp.mp4
ffmpeg version 4.0-static https://johnvansickle.com/ffmpeg/ Copyright (c) 2000-2018 the FFmpeg developers
built with gcc 6.3.0 (Debian 6.3.0-18+deb9u1) 20170516
configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --disable-ffplay --disable-indev=sndio --disable-outdev=sndio --cc=gcc-6 --enable-libxml2 --enable-fontconfig --enable-frei0r --enable-gnutls --enable-gray --enable-libaom --enable-libfribidi --enable-libass --enable-libfreetype --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librubberband --enable-libsoxr --enable-libspeex --enable-libvorbis --enable-libopus --enable-libtheora --enable-libvidstab --enable-libvo-amrwbenc --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg
libavutil 56. 14.100 / 56. 14.100
libavcodec 58. 18.100 / 58. 18.100
libavformat 58. 12.100 / 58. 12.100
libavdevice 58. 3.100 / 58. 3.100
libavfilter 7. 16.100 / 7. 16.100
libswscale 5. 1.100 / 5. 1.100
libswresample 3. 1.100 / 3. 1.100
libpostproc 55. 1.100 / 55. 1.100
Invalid duration specification for ss: NaN

The report gives me metrics, a screen grab of a successful login and a video which shows the successful bit, I guess it has a hard time rendering/ capturing the auth part.

I can perhaps pull more info out if you're able to point me in the right direction, but hope that helps nonetheless.

@soulgalore
Copy link
Member Author

Hi @jholsgrove thanks for trying it out!

First about the video: It fails when we try to find the first frame that isn't orange ... Think something goes wrong with the URL (I can see it is orange and then only one frame that changes). If you have time you can add --browsertime.videoParams.keepOriginalVideo to your run, then the original (uncut) video will be saved, and then we can see what's going on.

docker run -v /c/users/jholsgrove:/sitespeed.io sitespeedio/sitespeed.io --basicAuth admin@admin -d 2 -b chrome https://the-internet.herokuapp.com/basic_auth --viewPort 1920x1080 --video false

It's not so obvoius but we have both video and visual metrics on by default so to disable fully recording a video you need --video false --visualMetrics false. Sorry! Can you try with that?

Best
Peter

@jholsgrove
Copy link

Thanks @soulgalore - seems to have cleared that part up! The video just gave me a white screen/ no visuals with --browsertime.videoParams.keepOriginalVideo, fully disabling got rid of the errors, yay!

With regard to the original bug it doesn't seem to crawl the additional links to elemental selenium and the github fork using the -d 2 param. Verbose log doesn't seem to indicate anything going wrong. I am facing a similar issue with a site I use which is HTTPS. Is there much else I can do to help?

Creds for the basicAuth test site are admin/admin

@soulgalore
Copy link
Member Author

Thanks @jholsgrove hmm ok, but that URL, does it have any URLS to crawl? It will only crawl on the same host if I remember correctly.

Best
Peter

@jholsgrove
Copy link

jholsgrove commented Jun 26, 2018

Ah yes, a slight oversight on my part there. I rigged up my own basic auth site and added a link and it crawled just fine. I think you can safely close this particular issue @soulgalore :)

If you would like to see yourself: www.goatpod.com/basicAuth
Creds: admin/admin

@soulgalore
Copy link
Member Author

Closing, please re-open if you have any issues :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants