-
Notifications
You must be signed in to change notification settings - Fork 59
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Problem with empty content redirection from BNF errored crawls #426
Labels
Comments
Another example with http/https redirections:
|
boogheta
added a commit
that referenced
this issue
Nov 3, 2021
New log with latests commits:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
archive requested: 01/01/21 12:00
archive contains:
The manual solution relies in just changing the requested date by 1 day, hence collecting a functioning archive. But that's not user-guessable at all!
Since the redirection to the following is unannounced and only handled by the wayback through a query, we cannot guess the good one to test, and since we force our desired date to always remain close to it, we end up requesting duplicates on the empty page
A possible solution might be to:
catch line 234 more cases of redirection when using BNF archives by checking whether http code is between 300 & 400 and the response is empty; in those case run a new request with an extra argument that would tell to the _request function to remove the bnf prefix for this url in the next request, benefiting then from the wayback's intelligence
Other potential ideas:
The text was updated successfully, but these errors were encountered: