Network launcher/ad-hoc clusters?

[mglev1n]

Within the future ecosystem there is the ability to create an ad-hoc cluster of workers across several networked machines, and then allocate tasks to these workers in parallel (eg. https://future.futureverse.org/articles/future-3-topologies.html#an-ad-hoc-compute-cluster). Currently, it seems like crew supports distributing tasks across many local workers, or submitting tasks to a high-performance computing cluster via crew.cluster. Is it possible within the current structure of crew and mirai to allocate tasks to workers spread across several networked machines like with future?

My specific use-case is a bit unique (I have access to a large HPC, which has begun allocating resources at the node-level, rather than simply by cpu/core, necessitating parallelization beyond crew.cluster), but I think is more broadly generalizable to cases where a user has access to several networked machines, and wants to define a combined set of resources and then distribute tasks.

[wlandau]

That's the idea behind plugins: https://wlandau.github.io/crew/articles/plugins.html.

[mglev1n]

That makes sense - I wasn't sure if this had been sketched already. It seems like within the plugin framework, I'd use system-level calls within the launcher to ensure R processes are started on the available remote machines?:

crew/R/crew_launcher_local.R

Lines 95 to 98 in 317597e

	path <- file.path(R.home("bin"), bin)
	processx::process$new(
	command = path,
	args = c("-e", call),

[wlandau]

Yes, exactly. That's how crew.cluster works.