Not Found (HTTP 404): spp_cites_legislation bulk analysis

[fleurhierink]

spp_cites_legislation() generates HTTP 404 errors when trying to fetch multiple taxon id's. The error appears at different times when trying to reproduce it. Sometimes the error appears after 6 taxon id's whereas the next time it might be after 16 taxon id's. I have been able to successfully run the code below until mid November 2019. A reproducible example:

# load library
library(rcites)

# object with multiple taxon id's
taxonid <- c("11007", "3576",  "7074",  "4824",  "5865",  
"12290", "8816",  "8876",  "10073", "5276",  "12244", "9253",  
"30606", "10652", "10505", "7211",  "6373",  "65782", "11053", "6235" )

# unsuccessful data fetching due to HTTP 404 error
spp_legislation <- spp_cites_legislation(taxon_id = taxonid, scope = "all", raw = TRUE) 

[KevCaz]

Thank you for reporting this, I also encounter an 4040 error when running the example, specifically I get:

R> spp_legislation <- spp_cites_legislation(taxon_id = taxonid, scope = "all"
   , raw = TRUE)                                                             
→ Now processing taxon_id '11007'.....................✔ 
→ Now processing taxon_id '3576'......................✔ 
→ Now processing taxon_id '7074'......................✔ 
→ Now processing taxon_id '4824'......................✔ 
→ Now processing taxon_id '5865'......................✔ 
→ Now processing taxon_id '12290'.....................✔ 
→ Now processing taxon_id '8816'......................✔ 
→ Now processing taxon_id '8876'......................✔ 
→ Now processing taxon_id '10073'.....................✔ 
→ Now processing taxon_id '5276'......................✔ 
→ Now processing taxon_id '12244'.....................✔ 
→ Now processing taxon_id '9253'......................✔ 
→ Now processing taxon_id '30606'.....................✔ 
→ Now processing taxon_id '10652'.....................✔ 
→ Now processing taxon_id '10505'.....................✔ 
→ Now processing taxon_id '7211'......................✔ 
→ Now processing taxon_id '6373'......................✔ 
→ Now processing taxon_id '65782'.....................✔ 
→ Now processing taxon_id '11053'.....................Error in rcites_res(q_url, token, ...) : Not Found (HTTP 404).

(I've tried it 3 times and got the same results every time) I will try to investigate this issue latter today. In the meanwhile, looks like it's working when running one at a time.

[KevCaz]

It does not seem to be related with the code. Looks like the number of requests per IP has been lowered, my IP has just been banned, it never happened during the dev period were we did way more requests.

[KevCaz]

An easy fix (tested on the example you provided) is to add a pause between two requests as follows:

taxonid <- c("11007", "3576",  "7074",  "4824",  "5865",  
"12290", "8816",  "8876",  "10073", "5276",  "12244", "9253",  
"30606", "10652", "10505", "7211",  "6373",  "65782", "11053", "6235" )

res <- list()
for (i in seq_along(taxonid)) {
    res[[i]] <- spp_cites_legislation(taxon_id = taxonid[i], scope = "all", raw = TRUE)
    Sys.sleep(1)
}
# using an unexported function 
spp_legislation <- rcites:::rcites_combine_lists(res, taxonid, raw = TRUE)

I'm considering adding this feature in the package.

[fleurhierink]

Thank you for this quick workaround! This indeed works.

[KevCaz]

I am closing this and opening a separate issue about the pause feature.
Thank you for the report.