[help] With cloud storage, download in scratch/ deleted immediately causing R objects containing file pointers to be broken

[Aariq]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

I'm not sure if this is intended behavior or not, but it appears that when using cloud storage with targets running tar_read() or tar_make() 1) downloads the target from the cloud to tempdir()/_targets/scratch/, 2) runs the read function for the target, 3) deletes the file in scratch/. This causes proxy objects like terra SpatRaster objects to be broken because they do not necessarily contain the data, just a file pointer. Would it maybe make more sense for the files in scratch/ to stick around for the lifetime of the interactive R session? This also seems to cause trouble for downstream targets though, so maybe there's a different solution.

Reprex with geotargets:

targets::tar_dir({
  targets::tar_script({
    library(targets)
    library(geotargets)
    tar_option_set(
      repository = "aws",
      resources = tar_resources(
        aws = tar_resources_aws(
          bucket = "test123456",
          prefix = "targets_test",
          endpoint = "https://js2.jetstream-cloud.org:8001"
        )
      )
    )
    list(
      tar_target(
        file,
        system.file("ex/elev.tif", package = "terra"),
        format = "file", 
        repository = "local"
      ),
      tar_terra_rast(
        rast_example,
        terra::rast(file)
      )
    )
  })
  
  library(targets)
  library(terra)
  tar_make()
  tar_load(rast_example)
  plot(rast_example) #errors
})
#> terra 1.7.83
#> ✔ skipped target file
#> Listing objects in AWS S3 bucket test123456 prefix targets_test/objects
#> ✔ skipped target rast_example
#> ✔ skipped pipeline [1.005 seconds]
#> Error: [readStart] file does not exist: /private/var/folders/wr/by_lst2d2fngf67mknmgf4340000gn/T/Rtmpb3uyBC/_targets/scratch/rast_exampleb5e64cad4f44

^{Created on 2024-11-04 with reprex v2.1.1}

Simple reprex showing that the download immediately ceases to exist:

targets::tar_dir({
  targets::tar_script({
    library(targets)
    tar_option_set(
      repository = "aws",
      resources = tar_resources(
        aws = tar_resources_aws(
          bucket = "test123456",
          prefix = "targets_test",
          endpoint = "https://js2.jetstream-cloud.org:8001"
        )
      )
    )
    list(
      tar_target(
        where_am_i,
        iris,
        format = tar_format(
          read = function(path) path,
          write = function(object, path) saveRDS(object, path)
        )
      )
    )
  })
  targets::tar_make()
  targets::tar_load(where_am_i)
  where_am_i
  fs::file_exists(where_am_i)
})
#> Listing objects in AWS S3 bucket test123456 prefix targets_test/objects
#> ✔ skipped target where_am_i
#> ✔ skipped pipeline [3.959 seconds]
#> /var/folders/wr/by_lst2d2fngf67mknmgf4340000gn/T//Rtmp2ZDikH/_targets/scratch/where_am_ib60d649da6a0 
#>                                                                                                FALSE

^{Created on 2024-11-04 with reprex v2.1.1}

(You'll have to either have your own S3 bucket to try this with, or I can privately share credentials for this one if neccessary)

[wlandau]

it appears that when using cloud storage with targets running tar_read() or tar_make() 1) downloads the target from the cloud to tempdir()/_targets/scratch/, 2) runs the read function for the target, 3) deletes the file in scratch/.

targets promptly deletes individual scratch files in because otherwise they could accumulate and overburden the local machine.

Is it possible to read the raster data completely back into memory? Presumably the object was originally in memory before it was saved to disk, so I would be surprised if terra did not have a way to completely invert its own writeRaster() function.

Here is a version of the reprex for terra.

callr::r(func = function() {
  library(terra)
  raster <- rast(nrows = 10L, ncols = 10L)
  values(raster) <- seq_len(100L)
  writeRaster(raster, "output.tif", overwrite = TRUE)
  NULL
})
#> NULL

library(terra)
#> terra 1.7.78
object <- rast("output.tif")
unlink("output.tif")
values(object)
#> Error: [readStart] file does not exist: /private/var/folders/4v/vh7xp8553lsbl49svl48g7p00000gp/T/RtmpxuqUGO/reprex-2fcf72a02fe6-surly-egg/output.tif

^{Created on 2024-11-05 with reprex v2.1.1}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.4.0 (2024-04-24)
#>  os       macOS 15.1
#>  system   aarch64, darwin20
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       America/Indiana/Indianapolis
#>  date     2024-11-05
#>  pandoc   3.2 @ /Applications/RStudio.app/Contents/Resources/app/quarto/bin/tools/aarch64/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version date (UTC) lib source
#>  callr         3.7.6   2024-03-25 [1] CRAN (R 4.4.0)
#>  cli           3.6.3   2024-06-21 [1] CRAN (R 4.4.0)
#>  codetools     0.2-20  2024-03-31 [1] CRAN (R 4.4.0)
#>  digest        0.6.37  2024-08-19 [1] CRAN (R 4.4.1)
#>  evaluate      1.0.1   2024-10-10 [1] CRAN (R 4.4.1)
#>  fastmap       1.2.0   2024-05-15 [1] CRAN (R 4.4.0)
#>  fs            1.6.4   2024-04-25 [1] CRAN (R 4.4.0)
#>  glue          1.8.0   2024-09-30 [1] CRAN (R 4.4.0)
#>  htmltools     0.5.8.1 2024-04-04 [1] CRAN (R 4.4.0)
#>  knitr         1.48    2024-07-07 [1] CRAN (R 4.4.0)
#>  lifecycle     1.0.4   2023-11-07 [1] CRAN (R 4.4.0)
#>  processx      3.8.4   2024-03-16 [1] CRAN (R 4.4.0)
#>  ps            1.8.0   2024-09-12 [1] CRAN (R 4.4.1)
#>  R6            2.5.1   2021-08-19 [1] CRAN (R 4.4.0)
#>  Rcpp          1.0.13  2024-07-17 [1] CRAN (R 4.4.0)
#>  reprex        2.1.1   2024-07-06 [1] CRAN (R 4.4.0)
#>  rlang         1.1.4   2024-06-04 [1] CRAN (R 4.4.0)
#>  rmarkdown     2.28    2024-08-17 [1] CRAN (R 4.4.0)
#>  rstudioapi    0.16.0  2024-03-24 [1] CRAN (R 4.4.0)
#>  sessioninfo   1.2.2   2021-12-06 [1] CRAN (R 4.4.0)
#>  terra       * 1.7-78  2024-05-22 [1] CRAN (R 4.4.0)
#>  withr         3.0.1   2024-07-31 [1] CRAN (R 4.4.0)
#>  xfun          0.48    2024-10-03 [1] CRAN (R 4.4.1)
#>  yaml          2.3.10  2024-07-26 [1] CRAN (R 4.4.0)
#> 
#>  [1] /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

[Aariq]

(although maybe /vsis3/ doesn't work for terra: rspatial/terra#251)

[Aariq]

I can confirm that switching the read function from function(path) terra::rast(path) to function(path) terra::rast(path) + 0 works, and I guess that's a viable workaround, although SpatRasters that point to memory appear to lose some properties such as a native file blocksize that we take advantage of for dynamic branching over tiles workflows in geotargets.

[wlandau]

Glad that worked. In stantargets, I had to do something similar with CmdStanFit objects from cmdstanr: https://github.com/ropensci/stantargets/blob/448518fae309ef002c7746481c621106c45c75aa/R/tar_stan_mcmc.R#L523-L529

[Aariq]

I'll have to make sure there aren't any unintended consequences of that workaround, and check if there is a more "official" way to do it, but I suspect adding this kind of behavior an optional force_in_memory argument to tar_terra_rast() might do the trick. I suppose if you're doing a lot of work with larger than memory rasters, maybe geotargets with cloud storage is just not the way to go.

However, out of curiosity, is there any chance that I could use the new content-addressable storage features to create a "hack" where the download function actually just returns a path in the form /vsis3/<bucket>/<key> and doesn't download anything? (assuming that rast("/vsis3/<bucket>/<key>") works as advertised at some point). I'm guessing the download function probably applies to all targets and has no way of knowing whether they're going to be read in by rast() or readRDS(), so probably no?

[wlandau]

However, out of curiosity, is there any chance that I could use the new content-addressable storage features to create a "hack" where the download function actually just returns a path in the form /vsis3// and doesn't download anything? (assuming that rast("/vsis3//") works as advertised at some point).

It's worth a try.

I'm guessing the download function probably applies to all targets and has no way of knowing whether they're going to be read in by rast() or readRDS(), so probably no?

If you know in advance which one applies to which target, then you could define different CAS specifications for each.