Get progress updates when using batchtools_slurm backend

[frederikziebell]

I recently came across this discussion, where @HenrikBengtsson explained that no future.batchtools backends are capable of progressr updates. Is there another way of getting progress updates? I came up with the following script that kind of works. The idea is to write one line per, say 10, iterations into a file and compute progress from the number of lines in the file:

test.R

library("future")
library("batchtools")
library("future.batchtools")
library("furrr")

slurm <- tweak(batchtools_slurm, template = system.file("templates/slurm-simple.tmpl", package = "batchtools"),
  workers = 5, 
  resources = list(
    ncpus = 1, 
    memory = 1*1024, # megabytes
    walltime = 10*60 # seconds
  )
)
plan(slurm)
message("Number of parallel workers: ", nbrOfWorkers())

# function to evaluate in parallel
f <- function(i) {
  Sys.sleep(.5)
  i
}

file.create("slurm.progress")
start <- Sys.time()

n <- 1000
progress_report_interval <- 10
result <- future_map(1:n, function(i){
  f(i)

  if(i %% progress_report_interval == 0) {

    # number of lines of slurm.progress file
    num_lines <- as.integer(system2("wc", args = c("-l slurm.progress | awk '{print $1}'"), stdout = TRUE))

    # fraction of iterations already completed
    progress_fraction <- progress_report_interval * (num_lines + 1) / n

    # compute expected time of arrival
    time_elapsed <- as.numeric(Sys.time() - start)
    eta <- (1-progress_fraction) / progress_fraction * time_elapsed
    eta_parsed <- sprintf("%02dd %02dh %02dm %02ds", eta %/% 86400,  eta %% 86400 %/% 3600, eta %% 3600 %/% 60,  eta %% 60 %/% 1)

    f <- file("slurm.progress","a")
    writeLines(paste0("progress: ", round(100*progress_fraction, 1), "%, ETA: ", eta_parsed), f)
    flush(f)
  }
})
file.remove("slurm.progress")

which is called via

test.sh

#!/bin/bash

#SBATCH --time=0-00:10:00
#SBATCH --ntasks=1
#SBATCH --output=slurm.out
Rscript test.R

The only issue is that the slurm.progress file stops at around 70%, potentially because there are some issues with many workers trying to write to the same file simultaneously in combination with some file-system caching. Can this method be improved? My only idea is to create a progress folder with one file per progress report and then compute progress from the number of files, but this is not very clearn.

Edit: I also found out that the way I compute the ETA assumes that the clocks of all the workers are synchronized, which might not be the case. Is there a way around this, e.g. by letting the parallel workers ask the master process for it's time after work has been done?