Non-blocking functions

[wlandau]

For targets and https://github.com/wlandau/crew, I would like to scale out to a large number of simultaneous futures. Unfortunately, I am limited by the overhead of future(), resolved(), and value(). I have tried plans future.callr::callr and future.batchtools::batchtools_sge, and for both kinds of future, the main event loop spends most of its time waiting for futures to be created and polled.

As a user, it would be super convenient if each of these API functions could operate asynchronously:

• future(): create the object and return control to the R session while things initialize in the background.
• resolved(): always return FALSE the first time, but secretly start a non-blocking background process that takes its time check whether the future actually completed.
• value(): either through value() itself or through another function, first asynchronously download the value to a fast cache, then let the user read synchronously read from that cache.

I realize this is complicated and may not work for all backends, but it would help workflows with future scale, and would allow task queues to be developed on top of future like https://www.tidyverse.org/blog/2019/09/callr-task-q/ does with callr.

As a workaround for https://github.com/wlandau/crew, I have considered sending futures to background callr processes to complete tasks like resolved() without blocking the main process. However, I am not sure which futures are exportable. The following code seems to work using SGE futures, but not callr or multicore futures.

library(future.batchtools)
plan(batchtools_sge, template = "template.tmpl")
f <- future({
  Sys.sleep(15)
  NULL
})
p <- callr::r_bg(function(f) future::value(f), args = list(f = f))
Sys.sleep(16)
p$get_result()