Skip to content

v1.95.27
Compare
Choose a tag to compare
@j-mendez j-mendez released this 28 May 15:33
· 310 commits to main since this release
v1.95.27
43531f5

Whats Changed

This release provides a major fix for crawls being delayed by respect robots or crawl delays. If you set a limit or budget for the crawl and a robots.txt contains a delay of 10s this would be a bottleneck for the entire crawl when limits applied since the we would have to wait for each link to process prior to exiting. The robots delay is now maxed at 60s for efficiency.

  • chore(cli): fix limit respecting
  • chore(robots): fix respect robots [#184]
  • bump chromiumoxide@0.6.0
  • bump tiktoken-rs@0.5.9
  • bump hashbrown@0.14.5
  • add zstd support reqwest
  • unpin smallvec
  • chore(website): fix crawl limit immediate exit
  • chore(robots): add max delay respect

Full Changelog: v1.95.6...v1.95.27

Assets 7
Loading