You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If I want to crawl some website for new pages I would want to start from some particular page, not the root. Like https://habr.com/en/feed/ for example. But spider always loads only that page and doesn't go further by links on that page.
How can I circumvent this?
And the same behavior is with addition of trailing slash: https://habr.com is starting crawling, but https://habr.com/ is loading only that page and then stops.
The text was updated successfully, but these errors were encountered:
If I want to crawl some website for new pages I would want to start from some particular page, not the root. Like
https://habr.com/en/feed/
for example. But spider always loads only that page and doesn't go further by links on that page.How can I circumvent this?
And the same behavior is with addition of trailing slash:
https://habr.com
is starting crawling, buthttps://habr.com/
is loading only that page and then stops.The text was updated successfully, but these errors were encountered: