sf trying to access proj online server

[angelXmonster]

I am currently trying to run a script on Compute Canada clusters but it seems that SF is trying to access proj online files, which makes my code crash:

1: In CPL_transform(x, crs, aoi, pipeline, reverse, desired_accuracy,  :

  GDAL Error 1: PROJ: Cannot open https://cdn.proj.org/us_noaa_nadcon5_nad83_fbn_nad83_2007_conus.tif: Failed to connect to [cdn.proj.org](http://cdn.proj.org/) port 443 after 300301 ms: Timeout was reached

2: In CPL_transform(x, crs, aoi, pipeline, reverse, desired_accuracy,  :

  GDAL Error 1: PROJ: Cannot open https://cdn.proj.org/us_noaa_nadcon5_nad83_harn_nad83_fbn_conus.tif: Failed to connect to [cdn.proj.org](http://cdn.proj.org/) port 443 after 300027 ms: Timeout was reached

3: In CPL_transform(x, crs, aoi, pipeline, reverse, desired_accuracy,  :

  GDAL Error 1: PROJ: Cannot open https://cdn.proj.org/us_noaa_nadcon5_nad83_1986_nad83_harn_conus.tif: Failed to connect to [cdn.proj.org](http://cdn.proj.org/) port 443 after 300031 ms: Timeout was reached

Execution halted 

Why is the case? Is there a way to prevent this from happening?

Thanks for the help,

Angela M

[rsbivand]

@angelXmonster If the cluster nodes have no internet access, that is what to expect. Please provide a single node minimal reproducible example showing the specific pipelines (transformation definitions) that need these grids. See for example https://r-spatial.github.io/sf/reference/proj_tools.html, use sf_proj_pipelines to show the pipelines.

https://cdn.proj.org/us_noaa_nadcon5_nad83_fbn_nad83_2007_conus.tif is a 40MB file, https://cdn.proj.org/us_noaa_nadcon5_nad83_harn_nad83_fbn_conus.tif 8MB but less accurate, https://cdn.proj.org/us_noaa_nadcon5_nad83_1986_nad83_harn_conus.tif 3.4MB, so even less accurate. The first is defined as:

USA - Conterminous - EPSG:8860 (NAD83(FBN)) to EPSG:4893 (NAD83(NSRS2007)). Size: 40.7 MB. Last modified: 2022-12-14

One possibility is to turn off sf_proj_network, meaning that your positional data will be transformed either with unknown (ballpark) accuracy, or using a 3- or 7-parameter transformation applied uniformly across your study area. See https://r-spatial.org/book/02-Spaces.html#sec-projlib for some details, https://r-spatial.github.io/evolution/CRS_projections_transformations.html for more.

If your cluster cannot access the internet, you might investigate installing the grids that you need locally in PROJ_LIB, that is the directory where proj.db lives. Knowing the output of sf::sf_extSoftVersion(), and of course the output of sessionInfo() for the compute nodes - what is the platform, the R and sf versions, and the versions of the libraries sf is built against?

[angelXmonster]

@rsbivand yes, I was almost sure that was the case, the cluster not being able to get the files online. That is why I did not report it as an issue because it is hard to provide a reproducible example as the files I am using are not of public access, and I imagine each grid depends on the projection. But from what I understand from what you said, it seems that the issue I have is using st_transform() with some raster from North America. So I think I can ask Compute Canada to include those grids locally, as I do not have permission to change any of the core folders in the clusters.

I tried to put some of the things you requested, sorry it is not reproducible as you wanted, but I guess you can see the idea:

loc <-  visit %>% 
  st_as_sf(coords=c("longitude", "latitude"), crs=4326, remove=FALSE) %>% 
  st_transform(crs=5072)
rast.i <- Is a raster from North America 
 
      loc.cov<- loc %>% 
        st_transform(crs(rast.i)) %>% 
        terra::extract(x=rast.i, bind=TRUE) %>% 
        st_as_sf() %>% 
        dplyr::rename(cov = names(rast.i)[1]) %>% 
        dplyr::filter(!is.na(cov)) %>% 
        dplyr::select(-cov) %>% 
        st_transform(crs=5072) %>% 
        st_buffer(200) %>% 
        st_transform(crs(rast.i))

Here is the pipeline:

> sf_proj_pipelines(crs(rast.i), crs(loc.n.j))
Candidate coordinate operations found:  1 
Strict containment:     FALSE 
Axis order auth compl:  FALSE 
Source:  PROJCRS["NAD83(NSRS2007) / Conus Albers",
    BASEGEOGCRS["NAD83(NSRS2007)",
        DATUM["NAD83 (National Spatial Reference System 2007)",
            ELLIPSOID["GRS 1980",6378137,298.257222101,
                LENGTHUNIT["metre",1]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4759]],
    CONVERSION["Conus Albers",
        METHOD["Albers Equal Area",
            ID["EPSG",9822]],
        PARAMETER["Latitude of false origin",23,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8821]],
        PARAMETER["Longitude of false origin",-96,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8822]],
        PARAMETER["Latitude of 1st standard parallel",29.5,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8823]],
        PARAMETER["Latitude of 2nd standard parallel",45.5,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8824]],
        PARAMETER["Easting at false origin",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8826]],
        PARAMETER["Northing at false origin",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8827]]],
    CS[Cartesian,2],
        AXIS["easting (X)",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["northing (Y)",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    USAGE[
        SCOPE["Data analysis and small scale data presentation for contiguous lower 48 states."],
        AREA["United States (USA) - CONUS onshore - Alabama; Arizona; Arkansas; California; Colorado; Connecticut; Delaware; Florida; Georgia; Idaho; Illinois; Indiana; Iowa; Kansas; Kentucky; Louisiana; Maine; Maryland; Massachusetts; Michigan; Minnesota; Mississippi; Missouri; Montana; Nebraska; Nevada; New Hampshire; New Jersey; New Mexico; New York; North Carolina; North Dakota; Ohio; Oklahoma; Oregon; Pennsylvania; Rhode Island; South Carolina; South Dakota; Tennessee; Texas; Utah; Vermont; Virginia; Washington; West Virginia; Wisconsin; Wyoming."],
        BBOX[24.41,-124.79,49.38,-66.91]],
    ID["EPSG",5072]] 
Target:  PROJCRS["NAD83(NSRS2007) / Conus Albers",
    BASEGEOGCRS["NAD83(NSRS2007)",
        DATUM["NAD83 (National Spatial Reference System 2007)",
            ELLIPSOID["GRS 1980",6378137,298.257222101,
                LENGTHUNIT["metre",1]]],
        PRIMEM["Greenwich",0,
            ANGLEUNIT["degree",0.0174532925199433]],
        ID["EPSG",4759]],
    CONVERSION["Conus Albers",
        METHOD["Albers Equal Area",
            ID["EPSG",9822]],
        PARAMETER["Latitude of false origin",23,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8821]],
        PARAMETER["Longitude of false origin",-96,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8822]],
        PARAMETER["Latitude of 1st standard parallel",29.5,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8823]],
        PARAMETER["Latitude of 2nd standard parallel",45.5,
            ANGLEUNIT["degree",0.0174532925199433],
            ID["EPSG",8824]],
        PARAMETER["Easting at false origin",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8826]],
        PARAMETER["Northing at false origin",0,
            LENGTHUNIT["metre",1],
            ID["EPSG",8827]]],
    CS[Cartesian,2],
        AXIS["easting (X)",east,
            ORDER[1],
            LENGTHUNIT["metre",1]],
        AXIS["northing (Y)",north,
            ORDER[2],
            LENGTHUNIT["metre",1]],
    USAGE[
        SCOPE["Data analysis and small scale data presentation for contiguous lower 48 states."],
        AREA["United States (USA) - CONUS onshore - Alabama; Arizona; Arkansas; California; Colorado; Connecticut; Delaware; Florida; Georgia; Idaho; Illinois; Indiana; Iowa; Kansas; Kentucky; Louisiana; Maine; Maryland; Massachusetts; Michigan; Minnesota; Mississippi; Missouri; Montana; Nebraska; Nevada; New Hampshire; New Jersey; New Mexico; New York; North Carolina; North Dakota; Ohio; Oklahoma; Oregon; Pennsylvania; Rhode Island; South Carolina; South Dakota; Tennessee; Texas; Utah; Vermont; Virginia; Washington; West Virginia; Wisconsin; Wyoming."],
        BBOX[24.41,-124.79,49.38,-66.91]],
    ID["EPSG",5072]] 
Best instantiable operation has accuracy: 0 m
Description: Inverse of Conus Albers + Conus Albers 
Definition:  +proj=noop +ellps=GRS80 

Here is the sf_extSoftVersion():

      GEOS           GDAL         proj.4 GDAL_with_GEOS     USE_PROJ_H
    "3.12.0"        "3.7.2"        "9.2.0"         "true"         "true"
        PROJ
     "9.2.0"

and here the sessionInfo():

R version 4.4.0 (2024-04-24)
Platform: x86_64-pc-linux-gnu
Running under: AlmaLinux 9.3 (Shamrock Pampas Cat)

Matrix products: default
BLAS/LAPACK: FlexiBLAS IMKL;  LAPACK version 3.11.0

locale:
 [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8
 [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8
 [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C
[10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C

time zone: America/Vancouver
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] sf_1.0-19

loaded via a namespace (and not attached):
 [1] compiler_4.4.0     magrittr_2.0.3     class_7.3-22       DBI_1.2.3
 [5] tools_4.4.0        units_0.8-5        proxy_0.4-27       Rcpp_1.0.13-1
 [9] KernSmooth_2.23-22 grid_4.4.0         e1071_1.7-16       classInt_0.4-10

Thanks for the help!

[rsbivand]

@angelXmonster I don't think that the example pipeline you show is what needs the internet connection. It is a noop, as the source and target CRS are identical. I think that there are differences in the versions of the datum used in the files that raise errors. The PROJ app projinfo can also be used from the command line to query the PROJ database, see https://proj.org/en/stable/apps/projinfo.html. If you need good accuracy (sub-meter), you need to address this (say with a local grid install, I see https://rhel.pkgs.org/9/epel-x86_64/proj-data-ca-8.2.0-1.el9.noarch.rpm.html for Canadian grids, there should be a US equivalent https://rhel.pkgs.org/9/epel-x86_64/proj-data-us-8.2.0-1.el9.noarch.rpm.html ?, the version here is arbitrary, but I'd go with the latest. If you don't need surveying accuracy, turn off the PROJ network https://proj.org/en/stable/resource_files.html#where-are-proj-resource-files-looked-for. See also https://proj.org/en/stable/apps/projsync.html#projsync for how to download grid files independently of your package manager - they may be placed in another directory and its path added to PROJ_LIB.