Loop for crossref

[jshleap]

Session Info

R version 3.5.1 (2018-07-02)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Pop!_OS 18.04 LTS

Matrix products: default
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_CA.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_CA.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_CA.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] fulltext_1.1.0.9130

loaded via a namespace (and not attached):
 [1] storr_1.2.1       tidyselect_0.2.5  purrr_0.2.5       reshape2_1.4.3   
 [5] colorspace_1.3-2  miniUI_0.1.1.1    htmltools_0.3.6   aRxiv_0.5.16     
 [9] XML_3.98-1.16     rlang_0.3.0.1     later_0.7.5       pillar_1.3.0     
[13] httpcode_0.2.0    glue_1.3.0        rappdirs_0.3.1    solrium_1.0.0    
[17] rentrez_1.2.1     bindrcpp_0.2.2    bindr_0.1.1       plyr_1.8.4       
[21] stringr_1.3.1     munsell_0.5.0     microdemic_0.4.0  gtable_0.2.0     
[25] htmlwidgets_1.3   httpuv_1.4.5      curl_3.2          hoardr_0.5.0     
[29] triebeard_0.3.0   urltools_1.7.1    rplos_0.8.4       Rcpp_1.0.0       
[33] xtable_1.8-3      promises_1.0.1    scales_1.0.0      DT_0.5           
[37] jsonlite_1.5      rcrossref_0.8.4   mime_0.6          ggplot2_3.1.0    
[41] digest_0.6.18     stringi_1.2.4     dplyr_0.7.8       shiny_1.2.0      
[45] grid_3.5.1        bibtex_0.4.2      tools_3.5.1       magrittr_1.5     
[49] lazyeval_0.2.1    tibble_1.4.2      crul_0.6.0        whisker_0.3-2    
[53] crayon_1.3.4      pkgconfig_2.0.2   data.table_1.11.8 xml2_1.2.0       
[57] lubridate_1.7.4   assertthat_0.2.0  httr_1.3.1        R6_2.3.0         
[61] compiler_3.5.1 

Hi, my query has over 1000 entries for crossref, how can I loop over it?

[sckott]

thanks for the report and the session info.

please share the fulltext function(s) you tried and point to the exact spot you're having trouble

[jshleap]

Hi! I have used ft_search(query = 'GWAS', from = 'crossref'), and then tried to use the offset option of rcrossref::cr_works, however, ft_search has a hard limit of 1000, so cannot even paginate over that.

[jshleap]

Hi! I have successfully worked around the crossref issue, by using:
crossref <- rcrossref::cr_works(query="'GWAS'", cursor='*', cursor_max=68232, filter=c(type="journal-article", has_full_text = TRUE)), and it seems to work as expected. However, when I tried to loop over the europmc, pagination does not work, neither in the ft_search ft_search(query = 'GWAS', euroopts=c(page=2), from='europmc'), nor in the eupmc_search eupmc_search(query = 'GWAS', per_page=1000, page=1). In both cases when changing the page, the result is the same.
Best

[sckott]

thanks @jshleap will have a look

[sckott]

they've changed the way paging is done - see https://europepmc.org/RestfulWebService#!/Europe32PMC32Articles32RESTful32API/search - they used to have page and page size parameters, now there is page size, but page has been replacedw ith a cursor - i'll have to change the eupmc_search function to account for this.

[sckott]

@jshleap reinstall remotes::install_github("ropensci/fulltext"), reload R, and try again, should work now. though you have to do paging differently than before.

# do a search
res <- ft_search(query='ecology', from='europmc')
res$europmc
# get the next batch of results, using the cursorMark string
# the same number of results per chunk will be returned as the first request, default: 25
# see ?eupmc_search
ft_search(query='ecology', from='europmc', 
  euroopts = list(cursorMark = res$europmc$cursorMark))