Export only specific objects to processing nodes when running within a function

[elgabbas]

Hello,

Is it possible to strictly provide a list of objects that will be visible by cores? In the following, I export only V1 and expect the function to give an error when printing V2. However, this is not the case as it recognizes both V1 and V2.

require(snow)
require(dplyr)
V1 <- 4
V2 <- 7
V3 <- 100
cl <- snow::makeSOCKcluster(3)
future::plan(cluster, cl)

future.apply::future_lapply(
  X = 1, 
  FUN = function(x) {
    
    cat(paste0(
      "Objects in function environment: ", 
      paste0(ls(envir = environment()), collapse = " + "), "\n"))
    
    cat(paste0(
      "Objects in global environment: ", 
      paste0(ls(envir = .GlobalEnv), collapse = " + "), "\n"))
    
    cat(paste0("V1 = ", V1, " & V2 = ", V2, "\n"))
  }, 
  future.seed = TRUE, future.globals = c("V1")) %>% 
  invisible()

# Objects in function environment: x
# Objects in global environment: cl + V1 + V2 + V3
# V1 = 4 & V2 = 7

The challenge is that it seems that all objects in the global environment is copies to cores, which makes parallel processing very slow.

In my custom R package, the use of future.apply also inherits all objects from the calling function. In the following, parallelFunction function inherits all objects from the environment of the parent function Fun1 instead of having access only to X.

Fun1 <- function(x) {
   # do something; create X, Y, and Z
   # ........

   # call another function that includes parallel computation
   parallelFunction(arg1 = X)      # This makes Y and Z also available for parallel processes
}

Is it possible to select which objects should be distributed into cores in a strict way?

NB: I asked about this on StackOverflow 5 months ago without an answer.

Thanks

[saeed6944]

Hello,

Is it possible to strictly provide a list of objects that will be visible by cores? In the following, I export only V1 and expect the function to give an error when printing V2. However, this is not the case as it recognizes both V1 and V2.

require(snow)
require(dplyr)
V1 <- 4
V2 <- 7
V3 <- 100
cl <- snow::makeSOCKcluster(3)
future::plan(cluster, cl)

future.apply::future_lapply(
  X = 1, 
  FUN = function(x) {
    
    cat(paste0(
      "Objects in function environment: ", 
      paste0(ls(envir = environment()), collapse = " + "), "\n"))
    
    cat(paste0(
      "Objects in global environment: ", 
      paste0(ls(envir = .GlobalEnv), collapse = " + "), "\n"))
    
    cat(paste0("V1 = ", V1, " & V2 = ", V2, "\n"))
  }, 
  future.seed = TRUE, future.globals = c("V1")) %>% 
  invisible()

# Objects in function environment: x
# Objects in global environment: cl + V1 + V2 + V3
# V1 = 4 & V2 = 7

The challenge is that it seems that all objects in the global environment is copies to cores, which makes parallel processing very slow.

In my custom R package, the use of future.apply also inherits all objects from the calling function. In the following, parallelFunction function inherits all objects from the environment of the parent function Fun1 instead of having access only to X.

Fun1 <- function(x) {
   # do something; create X, Y, and Z
   # ........

   # call another function that includes parallel computation
   parallelFunction(arg1 = X)      # This makes Y and Z also available for parallel processes
}

Is it possible to select which objects should be distributed into cores in a strict way?

NB: I asked about this on StackOverflow 5 months ago without an answer.

Thanks