Cannot fully scrape Ahrefs traffic checker

[Asekos08]

Hello everyone,
Please help me to scrape this website https://ahrefs.com/traffic-checker/

I am using Seleniumbase version - seleniumbase==4.30.8
While running locally everything is fine, but on docker I face a lot of different error which include:

The problem is that usually, Cloudflare should start after I click the check traffic button. Other errors are:
seleniumbase.common.exceptions.NoSuchElementException: Message:
celery_1 | Element {input[placeholder="Enter domain or URL"]} was not present after 20 seconds!

Exception: Text {Checking} in {html} was still visible after 12 seconds!

seleniumbase.common.exceptions.NoSuchElementException: Message:
celery_1 | Element {span:contains("Check traffic")} was not present after 20 seconds!

The problem is that the page contains those elements, but Seleniumbase cannot find those elements.

My script is almost the same as on the Seleniumbase docs:

with SB(uc=True, test=True, incognito=True, xvfb=True, locale_code="en") as sb:
url = "https://ahrefs.com/traffic-checker"
input_field = 'input[placeholder="Enter domain or URL"]'
submit_button = 'span:contains("Check traffic")'
sb.uc_open_with_reconnect(url) # The bot-check is later
sb.uc_gui_click_captcha()
sb.wait_for_element_visible(input_field, timeout=30)
sb.type(input_field, domain)
sb.reconnect(0.5)
sb.wait_for_element_visible(submit_button, timeout=30)
sb.uc_click(submit_button, reconnect_time=4)
sb.uc_gui_click_captcha()
sb.wait_for_text_not_visible("Checking", timeout=30)
page_source = sb.get_page_source()

        Another error I am getting is packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert self.write_plain_text(text, cursor, len(text)) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 210, in write_plain_text self.wrapped.write(text[start:end]) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 47, in write self.__convertor.write(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write self.write_and_convert(text) File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 199, in write_and_convert text = self.convert_osc(text) RecursionError: maximum recursion depth exceeded at 2024-10-04 10:44:03.746886+00:00
        
   Thank you in advance for your answers!

[mdmintz]

This script is working for me:

from seleniumbase import SB

with SB(uc=True, test=True, incognito=True, locale_code="en") as sb:
    url = "https://ahrefs.com/website-authority-checker"
    input_field = 'input[placeholder="Enter domain"]'
    submit_button = 'span:contains("Check Authority")'
    sb.uc_open_with_reconnect(url)  # The bot-check is later
    sb.type(input_field, "github.com/seleniumbase/SeleniumBase")
    sb.reconnect(0.1)
    sb.uc_click(submit_button, reconnect_time=3.25)
    sb.uc_gui_click_captcha()
    sb.wait_for_text_not_visible("Checking", timeout=11.5)
    sb.highlight('p:contains("github.com/seleniumbase/SeleniumBase")')
    sb.highlight('a:contains("Top 100 backlinks")')
    sb.set_messenger_theme(location="bottom_center")
    sb.post_message("SeleniumBase wasn't detected!")

Note that UC Mode is not fully compatible with Docker because Docker adds a fingerprint that makes UC Mode detectable.

[Asekos08]

Thanks for the reply!!! I have also noticed this:

1. Element not found error appears even if the element exists in page
2. I have a related error. That is what it looks like. self.__convertor.write(text)
   celery_1 | File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write
   celery_1 | self.write_and_convert(text)
   celery_1 | File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 205, in write_and_convert
   celery_1 | self.write_plain_text(text, cursor, len(text))
   celery_1 | File "...", line -1, in [rest of traceback truncated]
   celery_1 | RecursionError: maximum recursion depth exceeded
3. Also, I have noticed that errors opened with sb do not raise errors in docker, can you explain me why it might happen?

[mdmintz]

1. If the element is not found, then your selector may be incorrect. Note that iframes must be switched into first to see in there. And note that closed Shadow DOM is not reachable by Selenium.
2. celery is an external program. This space is for SeleniumBase questions only.
3. Docker environments are different from regular environments. Save those questions for the Docker people.

[Asekos08]

That is the script that I have used:

    try:
        with SB(uc=True, test=True, incognito=True, xvfb=True, locale_code="en") as sb:
            url = "https://ahrefs.com/traffic-checker"
            input_field = 'input[placeholder="Enter domain or URL"]'
            submit_button = 'span:contains("Check traffic")'
            sb.uc_open_with_reconnect(url)  # The bot-check is later

            sb.uc_gui_click_captcha()
                # sb.switch_to_default_content()  # Return to default frame after solving
        
        
            sb.wait_for_element_visible(input_field, timeout=20)
            sb.type(input_field, domain)
            sb.reconnect(0.1)
            sb.wait_for_element_visible(submit_button, timeout=20)
            sb.uc_click(submit_button, reconnect_time=4)
            sb.uc_gui_click_captcha()
            sb.wait_for_text_not_visible("Checking", timeout=12)
            page_source = sb.get_page_source()
            
            soup = BeautifulSoup(page_source, 'html.parser')

        print('everything is finished')
    except Exception as e:
        print('exception handled')

1. From this sometimes I have got this error:
   Element {span:contains("Check traffic")} was not present after 20 seconds!
   By 'element' I meant such cases when element is present, but sometimes seleniumbase was not able to detect it.
2. Second question was related colorama library which is used in seleniumbase as I have understand. Is it possible to completely ignore or remove it from seleniumbase implementation?
   File "/usr/local/lib/python3.9/site-packages/colorama/ansitowin32.py", line 177, in write
3. Let me give you an example what I meant by this:
   This code cannot detect 'chec detect'

    try:
        with SB(uc=True, test=True, incognito=True, xvfb=True, locale_code="en") as sb:
            url = "https://ahrefs.com/traffic-checker"
            input_field = 'input[placeholder="Enter domain or URL"]'
            submit_button = 'span:contains("Chec traffic")'
            sb.uc_open_with_reconnect(url)  # The bot-check is later

            sb.uc_gui_click_captcha()
                # sb.switch_to_default_content()  # Return to default frame after solving
        
        
            sb.wait_for_element_visible(input_field, timeout=20)
            sb.type(input_field, domain)
            sb.reconnect(0.1)
            sb.wait_for_element_visible(submit_button, timeout=20)
            sb.uc_click(submit_button, reconnect_time=4)
            sb.uc_gui_click_captcha()
            sb.wait_for_text_not_visible("Checking", timeout=12)
            page_source = sb.get_page_source()
            
            soup = BeautifulSoup(page_source, 'html.parser')

        print('everything is finished')
    except Exception as e:
        print('exception handled')

Logically in Selenium in that case Exception is raised, but in Seleniumbase this situation is different:

I run this code on python file locally just to show what I meant. All those issues are directly connected Selniumbase, that is why your opinion and answers really matters for me!
Thank you for your responses, and for this package!!!!

[Asekos08]

I will be really grateful if can help me with the questions above!!