[general] General questions on multi-core processes and dispatched targets.

[QuinnAsena]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

Hey all, hope it is ok to combine my questions in one thread:

1. my computer crashed... Will dispatched targets be continued or restarted?

2. is there a way to force targets to recognise specific targets as complete/incomplete? Say I add a couple of comments, targets thinks that part of the pipeline is out-of-date but functionally it has not changed.

3. Is there a way to see print statements across cores when using tar_make_future? I figure it is not possible since it is effectively spinning up a bunch of R instances, the console on my screen can't print from sub processes and that's why I only see print statements from the last remaining process. But it would be super useful if I could see the progress of multiple processes.

4. Is nesting multi-core processes bad practice (or even possible...?)? Say I have a function that can run over multiple cores and I call that function in a targets pipeline. Take this example:

# some function that is working on 2 cores called twice in a targets pipeline:
tar_target(object_tar, func(object, ncore = 2), deployment = "worker"),
tar_target(object_tar2, func(object, ncore = 2), deployment = "worker"),

# Run the targets pipeline with two workers: 
> tar_make_future(workers = 2)

# will this run on 4 cores or will it get very very confused...  

Appreciate anyone who reads all that! Hopefully they are simple questions that I haven't been able to dig up the answers for in the documentation!
TIA!your topic and provide a reprex if troubleshooting.

[joelnitta]

I think I can answer some of your questions...

1. Any target that has not finished (is out of date) will be restarted. If the target is composed of branches, it will pick up from wherever it left off (it won't need to re-run the branches that are already done).

2. {targets} will ignore comments and whitespace in a function. So adding / removing comments whitespace will not change the status of a target. If you want to force {targets} to re-run a particular target, there are a couple of ways to do that. I usually use tar_invalidate() for one-time usage. You could also use the cue option of tar_target() if for example it is a target that should always be run, etc.

3. I think any messages from targets run in parallel will show up in the output of tar_make().

4. AFAIK, targets() is only aware of the number of workers you tell it to run. For example, if you were to do tar_option_set(controller = crew_controller_local(workers = 2)), {targets} would run 2 parallel instances when possible, but {targets} has no idea/control over the number of cores each of those may use if they are running functions that are themselves parallelized, as in your example. I have done something like this and had no problems (and more than 2 cores running at once). I suppose whether problems arise or not will depend on the specific set-up you use for parallelization.

[QuinnAsena]

Thanks @joelnitta. I'll read up about branching and the documentation of those functions that you suggested.
for point 2 I should have given a better example like including a print() statement. Do you know if I can force targets to recognise a target as complete? My current method is to comment out the target so that it doesn't run.

For point 3 I do get all the print statements when single-threaded in tar_make() but not when using tar_make_future(workers = n). I see now that tar_make_future() is superseded in favour of tar_make() with crew, so I'll do a little homework to try the new method and see if I get the print() statements from parallel processes.

I might need to delve a bit more into point 4, I think targets can be used in a much more advanced way than my current understanding.
Thanks again!

[joelnitta]

for point 2 I should have given a better example like including a print() statement

Yes, a reprex would make it easier to answer your question.

Do you know if I can force targets to recognise a target as complete?

If {targets} is not recognizing it as complete, it sounds like there is something you need to fix instead of just ignoring it by commenting-out the code. Again, a reprex would help. Alternatively, if you have a good reason to always ignore a particular target, you could do this with the cue argument of tar_target(), by setting cue = tar_cue(mode = "never")).

For point 3...

Yes, I recommend trying crew first and see how that goes, and if it doesn't work as expected, please post a reprex.