Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Issue: Can't load a public webpage #6079

Closed
jackfrost1411 opened this issue Jun 13, 2023 · 1 comment · Fixed by #6107 or #6248
Closed

Issue: Can't load a public webpage #6079

jackfrost1411 opened this issue Jun 13, 2023 · 1 comment · Fixed by #6107 or #6248

Comments

@jackfrost1411
Copy link
Contributor

jackfrost1411 commented Jun 13, 2023 •
edited
Loading

I want to load in the webpage below.

Hi,
Trying to extract some webpage using webbaseloader:
"""
loader = WebBaseLoader("https://researchadmin.asu.edu/)
data = loader.load()
"""
But gives the following error:
SSLError: HTTPSConnectionPool(host='researchadmin.asu.edu', port=443): Max retries exceeded with url: / (Caused by
SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get
local issuer certificate (_ssl.c:1002)')))

It is a public web page. Can anyone help?

Suggestion:

No response
@devstein
Copy link

Generated by a 🤖

I found related issues that might be helpful. I did my best to summarize the solutions, but I recommend looking at the related issues yourself.

Related Issues

Open Issues

[SSL: CERTIFICATE_VERIFY_FAILED] while load from SitemapLoader

This issue is referenced by one PR

This response is meant to be helpful and save users time. It is not meant to be a precise solution.

Please leave a 👍 if this is helpful and 👎 if it is irrelevant

This was referenced Jun 15, 2023
Merged
Merged
hwchase17 added a commit that referenced this issue Jun 17, 2023
@jackfrost1411 @hwchase17
update web_base.py to have verify option (#6107)
2eec687 
We propose an enhancement to the web-based loader initialize method by
introducing a "verify" option. This enhancement addresses the issue of
SSL verification errors encountered on certain web pages. By providing
users with the option to set the verify parameter to False, we offer
greater flexibility and control.
<!--
Thank you for contributing to LangChain! Your PR will appear in our
release under the title you set. Please make sure it highlights your
valuable contribution.

Replace this with a description of the change, the issue it fixes (if
applicable), and relevant context. List any dependencies required for
this change.

After you're done, someone will review your PR. They may suggest
improvements. If no one reviews your PR within a few days, feel free to
@-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd
like us to shout you out on Twitter, please also include your handle!
-->

### Fixes #6079 

#### Who can review?
@eyurtsev @hwchase17

---------

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
hwchase17 added a commit that referenced this issue Jun 19, 2023
@jackfrost1411 @hwchase17
Update web_base.ipynb for guiding purposes (#6248)
ba90e3c 
To bypass SSL verification errors during fetching, you can include the
`verify=False` parameter. This markdown proves useful, especially for
beginners in the field of web scraping.

<!--
Thank you for contributing to LangChain! Your PR will appear in our
release under the title you set. Please make sure it highlights your
valuable contribution.

Replace this with a description of the change, the issue it fixes (if
applicable), and relevant context. List any dependencies required for
this change.

After you're done, someone will review your PR. They may suggest
improvements. If no one reviews your PR within a few days, feel free to
@-mention the same people again, as notifications can get lost.

Finally, we'd love to show appreciation for your contribution - if you'd
like us to shout you out on Twitter, please also include your handle!
-->

Fixes #6079 

#### Who can review?

Tag maintainers/contributors who might be interested:
@hwchase17 
@eyurtsev

---------

Co-authored-by: Harrison Chase <hw.chase.17@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
2 participants
@devstein @jackfrost1411