Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to bypass Content-Security-Policy when executing Javascript #116

Merged
merged 1 commit into from
Nov 1, 2023

Conversation

sesh
Copy link
Contributor

@sesh sesh commented Jul 30, 2023

Refs: #114

Adds a --bypass-csp option to the commands that allow Javascript to be executed.

The additional test case that has been added loads Github and attempts to load an external module. With the --bypass-csp flag this will work. You can execute the following on the current version of shot-scraper to see it failing:

shot-scraper javascript github.com "async () => { await import('https://cdn.jsdelivr.net/npm/left-pad/+esm'); return 'content-security-policy ignored' }"

The above will continue to fail with this change until --bypass-csp is added.

I have added the flag to the documentation by have not added a new documentation block to the Javascript page for this yet. I'm happy to write up an example if you're keen to accept this PR.

I'm also interested in feedback in how the help text should be phrased. I went with the simplest possible phrasing but it does assume that the user knows what a CSP is.


📚 Documentation preview 📚: https://shot-scraper--116.org.readthedocs.build/en/116/

@simonw simonw added the enhancement New feature or request label Nov 1, 2023
@simonw
Copy link
Owner

simonw commented Nov 1, 2023

Thanks for this - I'm going to land it as-is and then update the documentation. It's a really good implementation.

@simonw simonw merged commit 3d14b03 into simonw:main Nov 1, 2023
@simonw
Copy link
Owner

simonw commented Nov 1, 2023

Manually tested this like so. First, without the flag:

shot-scraper javascript github.com "
  async () => {
    await import('https://cdn.jsdelivr.net/npm/left-pad/+esm');
    return 'content-security-policy ignored' }
"

Error: TypeError: Failed to fetch dynamically imported module: https://cdn.jsdelivr.net/npm/left-pad/+esm

Then with the flag:

shot-scraper javascript github.com "
  async () => {
    await import('https://cdn.jsdelivr.net/npm/left-pad/+esm');
    return 'content-security-policy ignored' }
" --bypass-csp

"content-security-policy ignored"

simonw added a commit that referenced this pull request Nov 1, 2023
simonw added a commit that referenced this pull request Nov 1, 2023
@simonw
Copy link
Owner

simonw commented Nov 1, 2023

@jamesking
Copy link

@sesh @simonw thank you for this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants