Efficiency of tar_make() with {crew} vs tar_make_clustermq()

[wlandau-lilly]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

This crew test with lots of small targets:

targets/tests/hpc/test-crew_local.R

Lines 85 to 121 in 7463ece

	tar_test("heavily parallel workload should run fast", {
	skip_on_cran()
	skip_if_not_installed("crew")
	tar_script({
	library(targets)
	controller <- crew::crew_controller_local(workers = 4)
	tar_option_set(controller = controller)
	list(
	tar_target(
	index_batch,
	seq_len(100),
	),
	tar_target(
	data_continuous,
	index_batch,
	pattern = map(index_batch)
	),
	tar_target(
	data_discrete,
	index_batch,
	pattern = map(index_batch)
	),
	tar_target(
	fit_continuous,
	data_continuous,
	pattern = map(data_continuous)
	),
	tar_target(
	fit_discrete,
	data_discrete,
	pattern = map(data_discrete)
	)
	)
	})
	tar_make()
	expect_equal(tar_outdated(callr_function = NULL), character(0))
	})

is slower than the equivalent clustermq test:

targets/tests/hpc/test-clustermq_local.R

Lines 150 to 181 in 7463ece

	tar_script({
	library(targets)
	options(clustermq.scheduler = "multicore")
	list(
	tar_target(
	index_batch,
	seq_len(100),
	),
	tar_target(
	data_continuous,
	index_batch,
	pattern = map(index_batch)
	),
	tar_target(
	data_discrete,
	index_batch,
	pattern = map(index_batch)
	),
	tar_target(
	fit_continuous,
	data_continuous,
	pattern = map(data_continuous)
	),
	tar_target(
	fit_discrete,
	data_discrete,
	pattern = map(data_discrete)
	)
	)
	})
	# Should deploy targets in a timely manner.
	proffer::pprof(tar_make_clustermq(workers = 4, callr_function = NULL))

proffer flame graphs do not show an overt bottleneck, so it's hard to say what might be slowing things down. It might have to do with my solution to #1074. Reverting it seems to shave a couple seconds off.

On the other hand, maybe the above test is not truly representative of the work that takes place in real projects. In a proprietary internal simulation study, tar_make_clustermq() took 2.007 minutes, whereas tar_make() with the equivalent number of persistent crew workers took about 2.661 minutes. From previous profiling studies and from looking at what is in the pipeline, I believe I can explain this difference in terms of serialization. clustermq supports "common data", whereas crew and mirai do not (c.f. wlandau/crew#33).

[wlandau]

Ultimately, targets that run quickly are much better off with deployment = "main".

[wlandau]

So after further profiling studies, I think the limitation is in crew itself rather than the integration of crew in targets. I will post a separate discussion in the crew repo.