furrr::future_map_df, max.concurrent.jobs for torque cluster

[richardbeare]

Hi,
I'm not sure what I'm missing here. I have a problem that I want to pack into jobs on a pbs cluster with a size that can run within a two hour time limit. This will result in some thousands of jobs. Per user cluster limits prohibit that many jobs from being submitted simultaneously. The call generating the jobs is of this form:

result <- future_map_dfr(chunks, ~fitfunction(.x, ...))

The batchtools option of max.concurrent.jobs appears to address this issue, but I can't seem to set it via plan or config file, and haven't found any examples.

Attempts:
a batchtools.conf.R file containing

max.concurrent.jobs <- 200

Is that the right format?

adding registry to the plan call:

plan(batchtools_torque, workers=10000, template="batchtools.pbs.tmpl", 
resources=list(nodes="1:ppn=4", walltime="2:00:00", pmem="6G"),
registry=list(max.concurrent.jobs = 30L))

I can see the submitJobs code is doing:

  max.concurrent.jobs = assertCount(resources$max.concurrent.jobs, null.ok = TRUE) %??%
    assertCount(reg$max.concurrent.jobs, null.ok = TRUE) %??% NA_integer_

So the registry looks like it is the right place to put this, but what is the right way of doing this from furrr or future_map?

I appreciate any advice

[saeed6944]

Hi,
I'm not sure what I'm missing here. I have a problem that I want to pack into jobs on a pbs cluster with a size that can run within a two hour time limit. This will result in some thousands of jobs. Per user cluster limits prohibit that many jobs from being submitted simultaneously. The call generating the jobs is of this form:

result <- future_map_dfr(chunks, ~fitfunction(.x, ...))

The batchtools option of max.concurrent.jobs appears to address this issue, but I can't seem to set it via plan or config file, and haven't found any examples.

Attempts:
a batchtools.conf.R file containing

max.concurrent.jobs <- 200

Is that the right format?

adding registry to the plan call:

plan(batchtools_torque, workers=10000, template="batchtools.pbs.tmpl", 
resources=list(nodes="1:ppn=4", walltime="2:00:00", pmem="6G"),
registry=list(max.concurrent.jobs = 30L))

I can see the submitJobs code is doing:

  max.concurrent.jobs = assertCount(resources$max.concurrent.jobs, null.ok = TRUE) %??%
    assertCount(reg$max.concurrent.jobs, null.ok = TRUE) %??% NA_integer_

So the registry looks like it is the right place to put this, but what is the right way of doing this from furrr or future_map?

I appreciate any advice