Support httr config argument for OGC service clients #64

maelle · 2022-03-11T08:12:01Z

👋 here!

We're dealing with a rather old web service that requires special handling on Ubuntu, it only works with a special curl handle as shown below.

Would it be possible to somehow pass a handle/handle parameters to the WFSclient? (We're also in contact with the web service maintainers)

url <- 'https://drive.emodnet-geology.eu'
req1 <- curl::curl_fetch_memory(url)
#> Error in curl::curl_fetch_memory(url): error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
h <- curl::new_handle(ssl_cipher_list='DEFAULT@SECLEVEL=1')
req2 <- curl::curl_fetch_memory(url, handle = h)

^{Created on 2022-03-11 by the reprex package (v2.0.1)}

maelle · 2022-03-11T08:12:10Z

Cc @annakrystalli

maelle · 2022-03-11T08:17:03Z

I've looked at the docs of the curl package, it does not seem there's a way to customize the handle via options / environment variables.

maelle · 2022-03-11T08:42:17Z

Oh but i should look into httr::handle_pool() 🤔

eblondel · 2022-03-11T08:43:43Z

@maelle I will have a look, not sure an easy solution can be done through ows4R, considering this is plateform dependent. On Windows it works well.

maelle · 2022-03-11T08:45:49Z

Oh we'd do the platform handling, we via EMODnetWFS would just pass a different handle / a different set of handle options to ows4r::WFSClient$new(). (that ows4r would pass to httr::GET() handle argument)

I'm also trying to understand if we can influence the "handle pool" directly somehow.

Thank you!

eblondel · 2022-03-11T08:50:53Z

I'm not sure to understand what is the exact problem you deal with, can you share the R code that fails for you? i've just tested the WFS Client from an R Studio server deployed on Ubuntu, and it works:

wfs = WFSClient$new(
  url = "https://drive.emodnet-geology.eu/geoserver/gtk/wfs", 
  serviceVersion = "2.0.0", 
  logger = "DEBUG"
)

session info

R version 4.1.2 (2021-11-01)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.4 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.9.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8        LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8   
 [6] LC_MESSAGES=C.UTF-8    LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C           LC_TELEPHONE=C        
[11] LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C   

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] geosapi_0.5-1 ows4R_0.3     atom4R_0.1-2  geonapi_0.5-3 keyring_1.2.0 geometa_0.6-6

loaded via a namespace (and not attached):
 [1] tidyselect_1.1.2   xfun_0.30          purrr_0.3.4        sf_1.0-7           lattice_0.20-45    V8_4.1.0          
 [7] vctrs_0.3.8        generics_0.1.2     utf8_1.2.2         XML_3.99-0.9       rlang_1.0.2        e1071_1.7-9       
[13] pillar_1.7.0       glue_1.6.2         DBI_1.1.2          sp_1.4-6           lifecycle_1.0.1    stringr_1.4.0     
[19] rdflib_0.2.5       zip_2.2.0          knitr_1.37         tzdb_0.2.0         curl_4.3.2         parallel_4.1.2    
[25] class_7.3-20       fansi_1.0.2        Rcpp_1.0.8         KernSmooth_2.23-20 readr_2.1.2        openssl_2.0.0     
[31] classInt_0.4-3     redland_1.0.17-16  jsonlite_1.8.0     hms_1.1.1          askpass_1.1        stringi_1.7.6     
[37] jsonld_2.2         dplyr_1.0.8        grid_4.1.2         rgdal_1.5-28       cli_3.2.0          tools_4.1.2       
[43] magrittr_2.0.2     proxy_0.4-26       tibble_3.1.6       crayon_1.5.0       tidyr_1.2.0        pkgconfig_2.0.3   
[49] ellipsis_0.3.2     xml2_1.3.3         assertthat_0.2.1   roxygen2_7.1.2     httr_1.4.2         R6_2.5.1          
[55] units_0.8-0        compiler_4.1.2

maelle · 2022-03-11T08:56:15Z

So, after digging a bit, I don't think one can modify the handle pool externally, so we'd really need ows4r to let us pass the handle to httr::GET() and friends. (I understand if it's not prioritary 🙂 )

maelle · 2022-03-11T09:01:16Z

Noting that httr2 has httr2::req_options() which would be perfect here (but maybe switching to httr2 would be even more work here 😅 )

However there are not options we could set externally, we'd still need a way to pass them. 🤔

maelle · 2022-03-11T09:05:07Z

I had missed your reprex, running it now.

wfs = ows4R::WFSClient$new(
  url = "https://drive.emodnet-geology.eu/geoserver/gtk/wfs", 
  serviceVersion = "2.0.0", 
  logger = "DEBUG"
)
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> Loading IANA mime types...
#> No encoding supplied: defaulting to UTF-8.
#> [ows4R][INFO] OWSGetCapabilities - Fetching https://drive.emodnet-geology.eu/geoserver/gtk/wfs?service=WFS&version=2.0.0&request=GetCapabilities
#> Error in curl::curl_fetch_memory(url, handle = handle): error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type

^{Created on 2022-03-11 by the reprex package (v2.0.1)}

maelle · 2022-03-11T09:09:09Z

And with system info

wfs = ows4R::WFSClient$new(
  url = "https://drive.emodnet-geology.eu/geoserver/gtk/wfs", 
  serviceVersion = "2.0.0", 
  logger = "DEBUG"
)
#> Loading ISO 19139 XML schemas...
#> Loading ISO 19115 codelists...
#> Loading IANA mime types...
#> No encoding supplied: defaulting to UTF-8.
#> [ows4R][INFO] OWSGetCapabilities - Fetching https://drive.emodnet-geology.eu/geoserver/gtk/wfs?service=WFS&version=2.0.0&request=GetCapabilities
#> Error in curl::curl_fetch_memory(url, handle = handle): error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type

^{Created on 2022-03-11 by the reprex package (v2.0.1)}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.1.0 (2021-05-18)
#>  os       Ubuntu 20.04.1 LTS
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language (EN)
#>  collate  fr_FR.UTF-8
#>  ctype    fr_FR.UTF-8
#>  tz       Europe/Paris
#>  date     2022-03-11
#>  pandoc   2.16.2 @ /usr/lib/rstudio/bin/quarto/bin/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version    date (UTC) lib source
#>  askpass       1.1        2019-01-13 [1] CRAN (R 4.1.0)
#>  assertthat    0.2.1      2019-03-21 [1] CRAN (R 4.1.0)
#>  backports     1.4.1      2021-12-13 [1] CRAN (R 4.1.0)
#>  class         7.3-19     2021-05-03 [4] CRAN (R 4.0.5)
#>  classInt      0.4-3      2020-04-07 [1] CRAN (R 4.1.0)
#>  cli           3.2.0      2022-02-14 [1] CRAN (R 4.1.0)
#>  codetools     0.2-18     2020-11-04 [4] CRAN (R 4.0.3)
#>  crayon        1.5.0      2022-02-14 [1] CRAN (R 4.1.0)
#>  curl          4.3.2      2021-06-23 [1] CRAN (R 4.1.0)
#>  DBI           1.1.2      2021-12-20 [1] CRAN (R 4.1.0)
#>  digest        0.6.29     2021-12-01 [1] CRAN (R 4.1.0)
#>  dplyr         1.0.8      2022-02-08 [1] CRAN (R 4.1.0)
#>  e1071         1.7-9      2021-09-16 [1] CRAN (R 4.1.0)
#>  ellipsis      0.3.2      2021-04-29 [1] CRAN (R 4.1.0)
#>  evaluate      0.15       2022-02-18 [1] CRAN (R 4.1.0)
#>  fansi         1.0.2      2022-01-14 [1] CRAN (R 4.1.0)
#>  fastmap       1.1.0      2021-01-25 [1] CRAN (R 4.1.0)
#>  fs            1.5.2      2021-12-08 [1] CRAN (R 4.1.0)
#>  generics      0.1.2      2022-01-31 [1] CRAN (R 4.1.0)
#>  geometa       0.6-6      2022-01-26 [1] CRAN (R 4.1.0)
#>  glue          1.6.2      2022-02-24 [1] CRAN (R 4.1.0)
#>  highr         0.9        2021-04-16 [1] CRAN (R 4.1.0)
#>  htmltools     0.5.2.9000 2022-01-03 [1] Github (rstudio/htmltools@9490b62)
#>  httr          1.4.2      2020-07-20 [1] CRAN (R 4.1.0)
#>  jsonlite      1.8.0      2022-02-22 [1] CRAN (R 4.1.0)
#>  KernSmooth    2.23-20    2021-05-03 [4] CRAN (R 4.0.5)
#>  keyring       1.3.0      2021-11-29 [1] CRAN (R 4.1.0)
#>  knitr         1.37       2021-12-16 [1] CRAN (R 4.1.0)
#>  lattice       0.20-44    2021-05-02 [4] CRAN (R 4.1.0)
#>  lifecycle     1.0.1      2021-09-24 [1] CRAN (R 4.1.0)
#>  magrittr      2.0.2      2022-01-26 [1] CRAN (R 4.1.0)
#>  openssl       2.0.0      2022-03-02 [1] CRAN (R 4.1.0)
#>  ows4R         0.3        2022-03-11 [1] Github (eblondel/ows4r@5824587)
#>  pillar        1.7.0      2022-02-01 [1] CRAN (R 4.1.0)
#>  pkgconfig     2.0.3      2019-09-22 [1] CRAN (R 4.1.0)
#>  proxy         0.4-26     2021-06-07 [1] CRAN (R 4.1.0)
#>  purrr         0.3.4      2020-04-17 [1] CRAN (R 4.1.0)
#>  R.cache       0.15.0     2021-04-30 [1] CRAN (R 4.1.0)
#>  R.methodsS3   1.8.1      2020-08-26 [1] CRAN (R 4.1.0)
#>  R.oo          1.24.0     2020-08-26 [1] CRAN (R 4.1.0)
#>  R.utils       2.11.0     2021-09-26 [1] CRAN (R 4.1.0)
#>  R6            2.5.1      2021-08-19 [1] CRAN (R 4.1.0)
#>  raster        3.5-15     2022-01-22 [1] CRAN (R 4.1.0)
#>  Rcpp          1.0.8      2022-01-13 [1] CRAN (R 4.1.0)
#>  reprex        2.0.1      2021-08-05 [1] CRAN (R 4.1.0)
#>  rgdal         1.5-28     2021-12-15 [1] CRAN (R 4.1.0)
#>  rlang         1.0.2.9000 2022-03-07 [1] Github (r-lib/rlang@6ef0256)
#>  rmarkdown     2.12.1     2022-03-07 [1] Github (rstudio/rmarkdown@b5a8002)
#>  rstudioapi    0.13       2020-11-12 [1] CRAN (R 4.1.0)
#>  sessioninfo   1.2.2      2021-12-06 [1] CRAN (R 4.1.0)
#>  sf            1.0-7      2022-03-07 [1] CRAN (R 4.1.0)
#>  sp            1.4-6      2021-11-14 [1] CRAN (R 4.1.0)
#>  stringi       1.7.6      2021-11-29 [1] CRAN (R 4.1.0)
#>  stringr       1.4.0      2019-02-10 [1] CRAN (R 4.1.0)
#>  styler        1.6.2      2021-09-23 [1] CRAN (R 4.1.0)
#>  terra         1.5-21     2022-02-17 [1] CRAN (R 4.1.0)
#>  tibble        3.1.6      2021-11-07 [1] CRAN (R 4.1.0)
#>  tidyselect    1.1.2      2022-02-21 [1] CRAN (R 4.1.0)
#>  units         0.8-0      2022-02-05 [1] CRAN (R 4.1.0)
#>  utf8          1.2.2      2021-07-24 [1] CRAN (R 4.1.0)
#>  vctrs         0.3.8      2021-04-29 [1] CRAN (R 4.1.0)
#>  withr         2.5.0      2022-03-03 [1] CRAN (R 4.1.0)
#>  xfun          0.30       2022-03-02 [1] CRAN (R 4.1.0)
#>  XML           3.99-0.9   2022-02-24 [1] CRAN (R 4.1.0)
#>  yaml          2.3.5      2022-02-21 [1] CRAN (R 4.1.0)
#> 
#>  [1] /home/maelle/R/x86_64-pc-linux-gnu-library/4.1
#>  [2] /usr/local/lib/R/site-library
#>  [3] /usr/lib/R/site-library
#>  [4] /usr/lib/R/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

maelle · 2022-03-11T09:10:09Z

We were also seeing this error on GitHub Actions with more recent R versions.

maelle · 2022-03-11T10:32:57Z

What could also work I think is a way to pass an httr config httr::config() https://msmith.de/2020/10/02/httr-curl-ubuntu-20-04.html

eblondel · 2022-03-11T12:11:15Z

I will dig a bit in the issue, and let you know when i have time to work on that. I will run the ows4R over the most recent R, and will probably include some tests over Emodnet WFS services.

maelle · 2022-03-11T14:48:36Z

Thank you! As it's quite critical for us (for the services with this problem, Linux users would need to tweak settings which is cumbersome) I'd be happy to help / test-drive this.

…apabilities for now)

eblondel · 2022-03-11T21:19:33Z

@maelle I've implemented a solution which extend present ows4R clients with a 'config' argument, to pass a httr::config() where you cn specify curl options

I've setup a test over Emodnet geology WFS and all seems ok now: See how it is configured:

ows4R/tests/testthat/test_WFSClient_v2_0_0.R

Line 33 in 1a9842d

test_that("WFS 2.0.0 - Emodnet Geology", {

maelle · 2022-03-14T11:57:59Z

Thanks so much @eblondel!! 🚀

eblondel · 2022-03-14T11:58:44Z

you're welcome!

maelle mentioned this issue Mar 11, 2022

Fix R-CMD-CHECK workflow ubuntu issues EMODnet/emodnet.wfs#39

Closed

maelle mentioned this issue Mar 11, 2022

Geology web services on Linux EMODnet/emodnet.wfs#69

Closed

eblondel self-assigned this Mar 11, 2022

eblondel added OGC - WFS user support software:geoserver labels Mar 11, 2022

eblondel added a commit that referenced this issue Mar 11, 2022

#64 test adding httr::config() arg to ows4R (test limited to WFS GetC…

abc8601

…apabilities for now)

eblondel added a commit that referenced this issue Mar 11, 2022

#64 test passing httr config to OWSHttRequest

00da6e9

eblondel added a commit that referenced this issue Mar 11, 2022

#64

62bc736

eblondel added a commit that referenced this issue Mar 11, 2022

#64 generalize config arg to other OGC services

1a9842d

eblondel added enhancement OGC - OWS OGC - WCS OGC - WMS and removed software:geoserver labels Mar 11, 2022

eblondel added OGC - WPS OGC - CSW labels Mar 11, 2022

eblondel added this to the 0.3 milestone Mar 11, 2022

eblondel changed the title ~~Could one pass a handle to WFSclient?~~ Support httr config argument for OGC service clients Mar 11, 2022

eblondel closed this as completed Mar 14, 2022

eblondel mentioned this issue Mar 14, 2022

Extent geoflow software to support request header(s) and http configs r-geoflow/geoflow#238

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support httr config argument for OGC service clients #64

Support httr config argument for OGC service clients #64

maelle commented Mar 11, 2022

maelle commented Mar 11, 2022

maelle commented Mar 11, 2022

maelle commented Mar 11, 2022

eblondel commented Mar 11, 2022

maelle commented Mar 11, 2022

eblondel commented Mar 11, 2022

maelle commented Mar 11, 2022

maelle commented Mar 11, 2022 •

edited

Loading

maelle commented Mar 11, 2022

maelle commented Mar 11, 2022

maelle commented Mar 11, 2022

maelle commented Mar 11, 2022

eblondel commented Mar 11, 2022

maelle commented Mar 11, 2022

eblondel commented Mar 11, 2022

maelle commented Mar 14, 2022

eblondel commented Mar 14, 2022

Support httr config argument for OGC service clients #64

Support httr config argument for OGC service clients #64

Comments

maelle commented Mar 11, 2022

maelle commented Mar 11, 2022

maelle commented Mar 11, 2022

maelle commented Mar 11, 2022

eblondel commented Mar 11, 2022

maelle commented Mar 11, 2022

eblondel commented Mar 11, 2022

maelle commented Mar 11, 2022

maelle commented Mar 11, 2022 • edited Loading

maelle commented Mar 11, 2022

maelle commented Mar 11, 2022

maelle commented Mar 11, 2022

maelle commented Mar 11, 2022

eblondel commented Mar 11, 2022

maelle commented Mar 11, 2022

eblondel commented Mar 11, 2022

maelle commented Mar 14, 2022

eblondel commented Mar 14, 2022

maelle commented Mar 11, 2022 •

edited

Loading