callr process failure while using future_lapply with callr

[NikoloAz]

While using a script that tests all R-packages that we have such failure occurs:

This failure doesn't occur on regular bases but only once in a while and is not reproducible. The only part where futures are used is this one:

plan(callr)

ListeTestergebnis = future_lapply(as.list(params$Paket),\(x){
  r(getPaketTest,
    args = list(stage = params$stage,
                Paket = x[[1]]))
},
future.seed = TRUE)

I would be really grateful if you could provide an idea, which could solve this issue. The issue has started occuring after we installed the R Version R 4.3.3 and updated the packages callr, future.callr. future and future.apply to versions respectively 3.7.6, 0.8.2, 1.33.2 and 1.11.2.
Thanks in advance!

[HenrikBengtsson]

Thanks for the report. Before anything else, can you please replace the screenshot with a cut'n'pasted code block? That way it's searchable and accessible.

[HenrikBengtsson]

And please also add your sessionInfo().

[NikoloAz]

Thank you for your response. Here is the screenshot as cut'n'pasted code block.

processing file: DashboardReactablePakete.Rmd
  |.................................................................|
  |................................................. 50% [testdurchfuehrung]

quitting from lines at lines 35-100 [testdurchfuehrung] (DashboardReactablePakete.Rmd)
Error:
! callr subprocess failed: could not start R, exited with non-zero status, has crashed or was killed
Backtrace:
  1. future.apply::future_lapply(...)
  2. future.apply:::future_xapply(...)
  3. future::value.list(fs)
  4. future::resolve.list(...)
  5. future (local) SignalConditionsASAP(resignal = FALSE, pos = 1L)
  6. future::signalConditions(...)

For additional information, here is the function getPaketTest:

getPaketTest = function(stage,
                        Paket) {
  MathKV.Base::setStage(stage = stage)
  library(testthat)
  MVOrdner = list(prd = "I:/Programme/PRD/",
                  tst = "I:/Programme/TST/",
                  ent = "C:/MVArbeit/R/ENT")
  
  Pakete = list.files(path = file.path(MVOrdner[[stage]]))
  
  stopifnot(Paket %in% Pakete)
  
  pfad = file.path(MVOrdner[[stage]],Paket,"tests/testthat")
  test_pkg_env <- function(package) {
    list2env(
      as.list(getNamespace(package), all.names = TRUE),
      parent = parent.env(getNamespace(package))
    )
  }
  
  library(package = Paket, character.only = TRUE)
  testergebnis = test_dir(path = pfad,
                          env = test_pkg_env(Paket),
                          reporter = ListReporter,
                          stop_on_failure = FALSE)
  return(testergebnis)
}

Here is the session info.

R version 4.3.3 (2024-02-29 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19045)

Matrix products: default


locale:
[1] LC_COLLATE=German_Germany.utf8  LC_CTYPE=German_Germany.utf8    LC_MONETARY=German_Germany.utf8 LC_NUMERIC=C                    LC_TIME=German_Germany.utf8    

time zone: Europe/Berlin
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] future.apply_1.11.2 future.callr_0.8.2  future_1.33.2       callr_3.7.6         htmltools_0.5.8.1   reactablefmtr_2.0.0 reactable_0.4.4     testthat_3.2.1     
 [9] data.table_1.15.4   flexdashboard_0.6.2

loaded via a namespace (and not attached):
 [1] tidyselect_1.2.1                 dplyr_1.1.4                      blob_1.2.4                       arrow_15.0.1                    
 [5] fastmap_1.1.1                    duckdb_1.0.0                     promises_1.3.0                   digest_0.6.35                   
 [9] mime_0.12                        timechange_0.3.0                 lifecycle_1.0.4                  LaF_0.8.4                       
[13] processx_3.8.4                   RSQLite_2.3.6                    MathKV.Rechnungsgrundlagen_2.1.0 magrittr_2.0.3                  
[17] compiler_4.3.3                   rlang_1.1.3                      RODBC_1.3-23                     sass_0.4.9                      
[21] tools_4.3.3                      utf8_1.2.4                       yaml_2.3.8                       knitr_1.46                      
[25] lambda.r_1.2.4                   htmlwidgets_1.6.4                bit_4.0.5                        plyr_1.8.9                      
[29] purrr_1.0.2                      fansi_1.0.6                      git2r_0.33.0                     xtable_1.8-4                    
[33] globals_0.16.3                   cli_3.6.2                        rmarkdown_2.26                   generics_0.1.3                  
[37] rstudioapi_0.16.0                tzdb_0.4.0                       MathKV.KRhnMitRZuExcel_3.0.0     DBI_1.2.2                       
[41] cachem_1.0.8                     stringr_1.5.1                    assertthat_0.2.1                 parallel_4.3.3                  
[45] formatR_1.14                     vctrs_0.6.5                      jsonlite_1.8.8                   MathKV.Base_2.0.2               
[49] hms_1.1.3                        bit64_4.0.5                      listenv_0.9.1                    crosstalk_1.2.1                 
[53] fontawesome_0.5.2                jquerylib_0.1.4                  glue_1.7.0                       parallelly_1.37.1               
[57] reactR_0.5.0                     codetools_0.2-19                 ps_1.7.6                         lubridate_1.9.3                 
[61] stringi_1.8.3                    later_1.3.2                      tibble_3.2.1                     pillar_1.9.0                    
[65] brio_1.1.4                       MathKV.Protokoll_2.0.0           R6_2.5.1                         evaluate_0.23                   
[69] shiny_1.8.1.1                    readr_2.1.5                      openxlsx_4.2.5.2                 prompt_1.0.2                    
[73] memoise_2.0.1                    httpuv_1.6.15                    bslib_0.7.0                      Rcpp_1.0.12                     
[77] zip_2.3.1                        MathKV.Mpm7001_1.0.0             xfun_0.43                        pkgconfig_2.0.3

[HenrikBengtsson]

It's hard to say without a reproducible example. But, it looks like you're processing a bunch of tasks (params$Paket), and it fails after a while;

processing file: DashboardReactablePakete.Rmd
  |.................................................................|
  |................................................. 50% [testdurchfuehrung]

From

quitting from lines at lines 35-100 [testdurchfuehrung] (DashboardReactablePakete.Rmd)
Error:
! callr subprocess failed: could not start R, exited with non-zero status, has crashed or was killed

it looks like one of the tasks causes the background R worker to completely crash. This is most likely unrelated to Futureverse. Instead, it is something in your code/packages, or something in the input data that triggers a severe bug in the code.

It could also be that you're running out of memory and MS Windows kills the parallel worker. If you run out of memory, try with a smaller number of parallel workers, e.g. plan(future.callr::callr, workers = 2). If that works, then it suggests it's a memory issue and you can only run so many tasks in parallel before running out of memory.

My suggestion is also to try with different subsets of params$Paket to see if you can narrow in on a small, problematic set of tasks. That's the most common strategy for these type of problems.

PS. I've edited your comment to use code blocks formatting; please spend some time reading https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax on how to do that yourself in the future.

[NikoloAz]

Thanks for the quick response and the ideas.

Another follow-up question. If it is a memory issue, then plan(sequential) should definitely work, right?

A colleague of mine has encountered the same problem while changing plan(future.callr::callr) to plan(sequential). So I think there is some other bug in the code, as before the update to 4.3.3. we had no issues with this script.

I will try out subsetting of param$Paket and let's see.

Thank you for the link. I will surely spend some time on it.

[HenrikBengtsson]

If it is a memory issue, then plan(sequential) should definitely work, right?

It can still be the case that you don't have enough memory to do one task at the time. But, yes, in general, the fewer parallel workers down to running it in the main R session lowers the memory requirements.

So I think there is some other bug in the code, as before the update to 4.3.3. we had no issues with this script.

If so, and you used the exact same data, then you could always try to roll back to the version of R and the package versions that worked and update bit by bit until you've narrowed in on the minimal change that triggers the error.