console.log() in JavaScript blocks should go to stdout

[simonw]

I was trying to debug a javascript: block in a shots.yml file and realized that console.log() would be useful.

https://playwright.dev/python/docs/api/class-consolemessage#console-message-args says it's as easy as doing this:

# Listen for all console logs
page.on("console", lambda msg: print(msg.text))

[simonw]

That msg object actually has properties that look like this:

{'args': [<JSHandle preview=Hello world>, <JSHandle preview=JSHandle@node>], 'location': {'url': '', 'lineNumber': 15, 'columnNumber': 8}, 'text': 'Hello world JSHandle@node', 'type': 'log'}

That was triggered by console.log("Hello world", document.body).

I think I can just use text though, which is the default string representation.

[simonw]

One catch with this: anything that a web page writes to console.log() will end up on standard output as well.

This is a problem for shot-scraper javascript since it could result in pages messing with the desired output.

As such, I think this mechanism should write to stderr, not stdout.

[simonw]

Maybe I should control this with a --console-log option, otherwise things are going to get noisy for all sorts of pages that use console.log() in unexpected ways.

[simonw]

For example, here's what it does on https://www.facebook.com/

% shot-scraper facebook.com



                                            
 .d8888b.  888                       888    
d88P  Y88b 888                       888    
Y88b.      888                       888    This is a browser feature intended for 
 "Y888b.   888888  .d88b.  88888b.   888    developers. If someone told you to copy-paste 
    "Y88b. 888    d88""88b 888 "88b  888    something here to enable a Facebook feature 
      "888 888    888  888 888  888  Y8P    or "hack" someone's account, it is a 
Y88b  d88P Y88b.  Y88..88P 888 d88P         scam and will give them access to your 
 "Y8888P"   "Y888  "Y88P"  88888P"   888    Facebook account.
                           888              
                           888              
                           888              

See https://www.facebook.com/selfxss for more information.

Screenshot of 'http://facebook.com' written to 'facebook-com.png'

[simonw]

I think --log is clear enough.

[simonw]

I'm going to call it --log-console for consistency with the existing --log-requests option.

[simonw]

% shot-scraper javascript facebook.com 'document.title'
"Facebook - log in or sign up"
% shot-scraper javascript facebook.com 'document.title' --log-console



                                            
 .d8888b.  888                       888    
d88P  Y88b 888                       888    
Y88b.      888                       888    This is a browser feature intended for 
 "Y888b.   888888  .d88b.  88888b.   888    developers. If someone told you to copy-paste 
    "Y88b. 888    d88""88b 888 "88b  888    something here to enable a Facebook feature 
      "888 888    888  888 888  888  Y8P    or "hack" someone's account, it is a 
Y88b  d88P Y88b.  Y88..88P 888 d88P         scam and will give them access to your 
 "Y8888P"   "Y888  "Y88P"  88888P"   888    Facebook account.
                           888              
                           888              
                           888              

See https://www.facebook.com/selfxss for more information.

"Facebook - log in or sign up"

[simonw]

Re-opening because I need to do some manual testing and think about how to document this.

[simonw]

Need to manually test all of these, each with and without the --log-console option:

• shot-scraper accessibility facebook.com --log-console
• shot-scraper auth https://facebook.com /tmp/auth.json --log-console
• shot-scraper html facebook.com --log-console
• shot-scraper javascript facebook.com "document.title" --log-console
• shot-scraper pdf facebook.com --log-console
• shot-scraper shot facebook.com --log-console
• echo '- url: https://facebook.com' | shot-scraper multi - --log-console