Interrupting parallel tasks in R based on a global condition using future.apply

[martinju]

Hi

I am honestly not sure what title to use for this but what I would like is to parallelize a function call, and then interrupt the future when a certain (global) condition is met.

Here is a dirty, conceptual example:

library(future.apply)
plan(sequential)
set.seed(1)
y <- rnorm(1)
sd_y <- Inf

fun <- function(i) {
  
  if(sd_y < 0.15){
    return(NULL) # or break or stop()
  } else {
    # Some heavy computation
    Sys.sleep(0.1)
    x <- rnorm(1)
    y <<- c(y,x)
    sd_y <<- sd(y)/sqrt(length(y))
    return(sd_y)
  }
}

future.apply::future_lapply(1:100, fun,future.seed=TRUE)

Running this with plan(sequential) works fine, but this (obvilosly) does not work when running in parallell as the workers does not receive the update values of y and sd_y. So can I get the same functionality in a different way?
Further, if there is a way to actually stop further execution of the futures instead of returning NULL, that would be preferrable.

Moreover, I am sure using the '<<-' operator to set the global variables is not the recommended way to get access to intermediate results from the futures. Writing to disk is an alternative, but I guess that causes quite some overhead too.

Any guidance on how to approach this would be highly appreciated. I am perfectly happy with a solution that avoids starting NEW calls to fun if that is easier than also interrupting calls that are running in parallel with the call that meets the stopping condition.

[martinju]

@HenrikBengtsson I would highly appreciate any ideas, comments or other input on this problem :-)