Aborting requests at prerequest

[ishan-marikar]

I've been trying to write my own logic to skip duplicates and was wondering if there was a possible way to abort requests at the preRequest stage.

var c = new Crawler({
    preRequest: function(options, done) {
        if (crawledURLs.includes(options.uri)) {
            // Abort the request?
        } else {
            // Continue with the request.
            return done();
        }
    },
    callback: function(err, res, done) {
        if(err) {
	    console.log(err)
	} else {
	    console.log(res.statusCode)
	}
    }
});

[mike442144]

Please do not do it in preRequest stage, because you'll be confused with requests rate. reRequest is the last stage before actual requesting. I think you should remove it when queuing new tasks, and suggest you to use seenreq module to simplify the code.

[mike442144]

close due to inactive, feel free to reopen