robots.txt files are not being respected correctly

[div72]

I have a robots.txt not being respected by spider-rs currently.

Upon debugging there seems to be two issues:

1. URLs are not getting passed to the RobotFileParser correctly. It seems that RobotFileParser is expecting just paths while Website::is_allowed_robots is passing whole URLs:

   spider/spider/src/website.rs

   Lines 401 to 407 in 4b8a604

   	pub fn is_allowed_robots(&self, link: &str) -> bool {
   	if self.configuration.respect_robots_txt {
   	unsafe {
   	self.robot_file_parser
   	.as_ref()
   	.unwrap_unchecked()
   	.can_fetch("*", link)

Here's GDB output from the can_fetch call inside the function:

(gdb) p url_str
$16 = "https://div72.xyz"
(gdb) p default_entry.allowance(url_str)
$17 = true
(gdb) p default_entry.allowance("/")
$18 = false
(gdb) p default_entry.allowance("")
$19 = true

2. Website::is_allowed_robots is not passing the user-agent to the RobotFileParser. The '*' parameter in the code above should be self.configuration.user_agent rather than '*'.

[j-mendez]

fixed in 1.95.21 thanks!

[ubedan]

I think it's still not quite right...

I had a robots.txt file that had:

Disallow: /careers/login

And I got a scrape:

...url.com%2Fcareers%2Flogin%3FreturnTo%3D%2Fcareers%2Ffield%2Dsales.html

Which decodes to:

...url.com/careers/login?returnTo=/careers/field-sales.html

Does everyone want me to take a stab at fixing it?

[j-mendez]

I think it's still not quite right...

I had a robots.txt file that had:

Disallow: /careers/login

And I got a scrape:

...url.com%2Fcareers%2Flogin%3FreturnTo%3D%2Fcareers%2Ffield%2Dsales.html

Which decodes to:

...url.com/careers/login?returnTo=/careers/field-sales.html

Does everyone want me to take a stab at fixing it?

Hello, what website did you use and what user-agent? This will help us test etc.

[ubedan]

Oxide.Computer, and I was using a slightly modified (set respect_robots_txt = true and subdomains=true) version of the download.rs example which uses SpiderBot.

let website_name = "https://oxide.computer";
let mut website: Website = Website::new(website_name);
website.configuration.respect_robots_txt = true;
website.configuration.subdomains = true;
website.configuration.delay = 3340; // Defaults to 250 ms
website.configuration.user_agent = Some(Box::new("SpiderBot".into()));

[j-mendez]

@ubedan f32820c fixed in 1.98.3 ty

[ubedan]

Wow! Awesome! So fast!!! Thanks!