WebSurfer Updated (Selenium, Playwright, and support for many filetypes)

[afourney]

Why are these changes needed?

This PR add Selenium and Playwright variants of the Markdown Web Browser used by WebSurfer. It also adds support for many additional content-types, and support for alternate search engines.

All MarkdownBrowser variants work via the following principle:
1. Fetch a page,
2. Convert it to markdown,
3. Operate on the Markdown

Such browsers are simple, and suitable for read-only agentic use -- they cannot be used to interact with complex web applications. Nevertheless, they are a great stopgap, and super useful when browsing local files (file:///user/afourney/repos/autogen) etc. because they can handle many different file formats (Office docs, PDFs, etc.), provide a common interface for Q&A, summarization, passage extraction etc.

Instructions

When installing AutoGen, use the [websurfer] optional dependencies.

If using Selenium, you must also pip install selenium

If using Playwright you must both pip install playwright and playwright install --with-deps chromium

Related issue number

#1481, #1534, #1733, #1832

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 60.62133% with 469 lines in your changes are missing coverage. Please review.

Project coverage is 50.75%. Comparing base (c3193f8) to head (6ba05c9).

Files	Patch %	Lines
autogen/browser_utils/mdconvert.py	71.69%	133 Missing and 34 partials ⚠️
autogen/browser_utils/markdown_search.py	22.98%	120 Missing and 4 partials ⚠️
autogen/browser_utils/requests_markdown_browser.py	71.84%	51 Missing and 16 partials ⚠️
...togen/browser_utils/playwright_markdown_browser.py	27.41%	45 Missing ⚠️
autogen/agentchat/contrib/web_surfer.py	46.15%	23 Missing and 5 partials ⚠️
autogen/browser_utils/selenium_markdown_browser.py	35.71%	27 Missing ⚠️
autogen/browser_utils/abstract_markdown_browser.py	71.79%	11 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1929       +/-   ##
===========================================
+ Coverage   37.94%   50.75%   +12.80%     
===========================================
  Files          77       83        +6     
  Lines        7784     8776      +992     
  Branches     1667     2040      +373     
===========================================
+ Hits         2954     4454     +1500     
+ Misses       4580     3946      -634     
- Partials      250      376      +126     

Flag	Coverage Δ
unittest	12.75% <0.08%> (?)
unittests	49.80% <60.62%> (+11.86%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[afourney]

@signalprime @vijaykramesh @INF800

With this PR, I tried to combine your Selenium browser PRs together in one place. Even if it doesn't show in the commit history, I used and learned a lot from each of your contributions, and welcome your further comments and contributions here. Once this is ready, the final PR will credit each of you, and we can perhaps co-author a Blog post.

Further, I believe @INF800 and @vijaykramesh 's PRs used Selenium to call Bing search -- which is clever in that it simplifies requirements to get up and running (you don't need to register for an API key). However, I opted to leave this out in favor of the API because it is a better fit for our automated use. Bing actively discourages scraping, and supporting that approach long term would involve actively evading bot detection. I am open to adding further modularity and configurability to add other search engines, perhaps DuckDuckGo, ArXiv etc. that don't require an API key.