Using the ff package with future.callr

[imola]

I would like to pass large datasets that are stored as ff files (which map memory to storage) but I have been encountering an issue with using the future.callr package. I was wondering if there is a way to overcome this issue?
`

library(future)
plan(future.callr::callr, workers = 8L)
x <- ff::ff(1:100)
y %<-% { x[sample(100,10)]}
y
Error: callr subprocess failed: could not start R, exited with non-zero status, has crashed or was killed
`

[HenrikBengtsson]

You need to guide the future framework here. It fails to identify ff as a package dependency when doing static code inspection of x[sample(100,10)]. Another example where this can happen is when you work with data.table, cf. https://future.futureverse.org/articles/future-4-issues.html#missing-packages-false-negatives (I'll try to add an example for ff there too).

The solution is to pass f <- future({ ... }, packages = "ff"), or with future assignments, y %<-% { ... } %packages% "ff" as in:

library(future)
x <- ff::ff(1:10)

plan(sequential)
y %<-% x[1:3] %packages% "ff"
y
#> [1] 1 2 3

plan(future.callr::callr, workers = 1L)
y %<-% x[1:3] %packages% "ff"
y
#> opening ff /tmp/alice/RtmpBlBVrL/ff/ff5db453b3a726.ff
#> [1] 1 2 3

plan(cluster, workers = 1L)
y %<-% x[1:3] %packages% "ff"
y
#> opening ff /tmp/alice/RtmpBlBVrL/ff/ff5db453b3a726.ff
#> [1] 1 2 3

This is not specific to future.callr, it's true for all parallel workers. For completeness, you should also add this when running sequentially. Again, it's one of the corner cases where the framework fail to be automagic. Maybe one day it'll take care of this too ...

[HenrikBengtsson]

BTW, even without using %packages% "ff", you shouldn't get that error when running your example. That's weird, because it indicates a full-on crash of the parallel worker, which takes something like running out of memory to do. Intead, you should get all NULLs, e.g.

library(future)
plan(future.callr::callr, workers = 8L)
x <- ff::ff(1:100)
y %<-% { x[sample(100,10)] }
str(y)
#> List of 10
#>  $ : NULL
#>  $ : NULL
#>  $ : NULL
#>  $ : NULL
#>  $ : NULL
#>  $ : NULL
#>  $ : NULL
#>  $ : NULL
#>  $ : NULL
#>  $ : NULL
#> Warning message:
#>
#> UNRELIABLE VALUE: Future ('<none>') unexpectedly generated random numbers without
#> specifying argument 'seed'. There is a risk that those random numbers are not
#> statistically sound and the overall results might be invalid. To fix this,
#> specify 'seed=TRUE'. This ensures that proper, parallel-safe random numbers are
#> produced via the L'Ecuyer-CMRG method. To disable this check, use 'seed=NULL',
#> or set option 'future.rng.onMisuse' to "ignore".

If you don't get this, make sure to update all your R packages and retry.

[imola]

Thank you very much for your response- I'll be sure to pass the package argument in the future.

I am still getting the same error

library(future)
x <- ff::ff(1:10)
plan(future.callr::callr, workers = 1L)
y %<-% x[1:3] %packages% "ff"
y
Error: callr subprocess failed: could not start R, exited with non-zero status, has crashed or was killed

I will upgrade to the most recent version of R (currently 4.1.0) and I'll hope that's what is causing the issue.

Thanks for making such a great package!

[imola]

I found out that the reason for this error was that I had an error in my .Rprofile .

[HenrikBengtsson]

@imola solved this #588 (reply in thread):

I found out that the reason for this error was that I had an error in my .Rprofile .

The following example illustrates what's going on under the hood:

## Run this example in the temporary folder
opwd <- setwd(tempdir())

## Create a temporary .Rprofile that produces an error
cat("stop('boom')", file=".Rprofile")

## Use callr to evaluate a function
y <- callr::r(function() 42)
#> Error in get_result(output = out, options) : 
#>  callr subprocess failed: could not start R, exited with non-zero status, has crashed or was killed

which is why we get:

## Use future.callr
library(future.callr)
plan(callr)
f <- future(42)
value(f)
#> Error: callr subprocess failed: could not start R, exited with non-zero status, has crashed or was killed

Interestingly, it does not happen with PSOCK workers:

cl <- parallel::makePSOCKcluster(1L)
parallel::clusterEvalQ(cl, 42)
#> [[1]]
#> [1] 42

library(future)
plan(multisession)
f <- future(42)
value(f)
#> [1] 42

or batchtools;

library(future.batchtools)
plan(batchtools_local)
f <- future(42)
value(f)
#> [1] 42