New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

URL UnicodeEncodeError #79

Open

wkingnet opened this issue Apr 28, 2022 · 0 comments

wkingnet commented Apr 28, 2022 •

edited

Loading

If the URL contains UNICODE encoding, python will report an error.

debug info:

INFO:root:Crawling #1: https://gvo.wiki/html/NPC掉落書籍.html
DEBUG:root:https://gvo.wiki/html/NPC掉落書籍.html ==> 'ascii' codec can't encode characters in position 13-16: ordinal no
t in range(128)

Solution:

edit crawler.py
Add the following code at the top

import string
from urllib.parse import unquote

then search
current_url = self.urls_to_crawl.pop()
add a line below

current_url = self.urls_to_crawl.pop()
current_url = quote(current_url, safe=string.printable)

The text was updated successfully, but these errors were encountered:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment