Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test failure: Timed out while waiting for element to become available. #158

Closed
simonw opened this issue Sep 27, 2024 · 10 comments
Closed

Test failure: Timed out while waiting for element to become available. #158

simonw opened this issue Sep 27, 2024 · 10 comments
Labels
bug Something isn't working ci

Comments

@simonw
Copy link
Owner

simonw commented Sep 27, 2024

Got this on https://github.com/simonw/shot-scraper/actions/runs/11063153856/job/30738817452

Screenshot of '#bighead, .overband' on 'https://simonwillison.net/' written to 'examples/bighead-multi-selector.png'
Error: Timed out while waiting for element to become available.

Locator.screenshot: Timeout 29952ms exceeded.
Call log:
taking element screenshot
  - waiting for fonts to load...
  - fonts loaded
  - attempting scroll into view action
  -   waiting for element to be stable
  -   element is not visible
  ...
@simonw simonw added bug Something isn't working ci labels Sep 27, 2024
@simonw
Copy link
Owner Author

simonw commented Sep 27, 2024

Relevant snippet of tests (I think):

shot-scraper/tests/run_examples.sh

Lines 23 to 45 in a5e9707

# Multiple selectors
shot-scraper https://simonwillison.net/ \
-s '#bighead' -s .overband --padding 20 \
-o examples/bighead-multi-selector.png
# --selector-all
shot-scraper https://simonwillison.net/ \
--selector-all .day --padding 20 \
-o examples/selector-all.png
# Height and width
shot-scraper https://simonwillison.net/ -w 400 -h 800 -o examples/simon-narrow.png
# Omit background
shot-scraper https://simonwillison.net/ \
-h 800 -o examples/simonwillison-transparent.png \
--javascript "document.querySelector('body').style.backgroundColor='transparent'" \
--omit-background
# JPEG quality
shot-scraper https://simonwillison.net/ \
-h 800 -o examples/simonwillison-quality-80.jpg --quality 80
# Selector with a wait for remote page element
shot-scraper 'https://www.owlsnearme.com/?place=127871' \
--selector 'section.secondary' \
-o examples/owlsnearme-wait.jpg \
--wait-for "!!document.querySelector('section.secondary')"

@simonw
Copy link
Owner Author

simonw commented Sep 27, 2024

Here's the problem:

shot-scraper https://simonwillison.net/ \
   --selector-all .day --padding 20 \
   -o examples/selector-all.png

I redesigned my blog and removed the .day bit a while ago.

@simonw
Copy link
Owner Author

simonw commented Sep 27, 2024

I'll do this instead:

shot-scraper https://simonwillison.net/ \
   --selector-all '#secondary li:nth-child(-n+5)' \
   --padding 20 \
   -o examples/selector-all.png

selector-all

simonw added a commit that referenced this issue Sep 27, 2024
@simonw
Fixed selectors in tests, refs #158
dfe72b0
@simonw
Copy link
Owner Author

simonw commented Sep 27, 2024

Stil getting an error - this time I can recreate with bug.yaml:

- output: selectors-all-from-multi.png
  url: https://simonwillison.net/
  selectors_all:
  - #secondary li:nth-child(-n+5)
  - .entry:nth-of-type(1)
  padding: 20

And then:

shot-scraper multi bug.yaml

Error: Timed out while waiting for element to become available.

Timeout 29963ms exceeded.
=========================== logs ===========================
taking element screenshot
  waiting for element to be visible and stable
    element is not visible - waiting...
============================================================

@simonw
Copy link
Owner Author

simonw commented Sep 27, 2024

I'll do this instead:

- output: selectors-all-from-multi.png
  url: https://simonwillison.net/
  selectors_all:
  - "#secondary li:nth-child(-n+5)"
  - "#secondary li:nth-child(8)"
  padding: 20

simonw added a commit that referenced this issue Sep 27, 2024
@simonw
Different selector for test, refs #158
2b6bd70
@simonw
Copy link
Owner Author

simonw commented Sep 27, 2024

New error: https://github.com/simonw/shot-scraper/actions/runs/11063333005/job/30739290051

playwright._impl._errors.Error: Page.goto: net::ERR_CONNECTION_REFUSED at https://www.whatismybrowser.com/detect/what-is-my-user-agent/
Call log:
navigating to "https://www.whatismybrowser.com/detect/what-is-my-user-agent/", waiting until "load"

Maybe they are blocking GitHub IPs? I'm going to change that test to not depend on that site.

@simonw
Copy link
Owner Author

simonw commented Sep 27, 2024

I tried this:

# Different browsers
echo '<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>User Agent</title>
</head>
<body>
    <h1>Your User Agent:</h1>
    <p id="ua"></p>
    <script>
        document.getElementById("ua").textContent = navigator.userAgent;
    </script>
</body>
</html>' > user-agent.html
shot-scraper user-agent.html \
  -o examples/useragent-default-chromium.png -h 400 -w 800
shot-scraper user-agent.html \
  -o examples/useragent-firefox.png -h 400 -w 800 -b firefox
shot-scraper user-agent.html \
  -o examples/useragent-webkit.png -h 400 -w 800 -b webkit
rm user-agent.html

It passed for Chrome and Firefox but failed for WebKit:

playwright._impl._api_types.Error: A server with the specified hostname could not be found.
=========================== logs ===========================
navigating to "http://user-agent.html/", waiting until "load"
============================================================

Looks like there's a bug where WebKit doesn't correctly work with files loaded from disk.

@simonw simonw mentioned this issue Sep 27, 2024
Open
@simonw
Copy link
Owner Author

simonw commented Sep 27, 2024

Instead I'll add myself a user-agent.html in my https://github.com/simonw/tools repo.

simonw added a commit to simonw/tools that referenced this issue Sep 27, 2024
@simonw
/user-agent tool
d053b94 
Refs simonw/shot-scraper#158 (comment)
simonw added a commit that referenced this issue Sep 27, 2024
@simonw
Use https://tools.simonwillison.net/user-agent refs #158
b6fb35b
@simonw
Copy link
Owner Author

simonw commented Sep 27, 2024 

  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call
    raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None
playwright._impl._errors.Error: Page.goto: net::ERR_SSL_PROTOCOL_ERROR at https://localhost:9043/
Call log:
navigating to "https://localhost:9043/", waiting until "load"

simonw added a commit that referenced this issue Sep 27, 2024
@simonw
http not https, refs #158
14a06e6
@simonw
Copy link
Owner Author

simonw commented Sep 27, 2024

Fixed!

@simonw simonw closed this as completed Sep 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working ci
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@simonw