Recommend workflow for multiple targets within same project

[alexpghayes]

I'm curious what you recommend workflow-wise for large projects. I am currently using targets for a research paper. In the paper, I both run an extensive simulation, and then do a fairly involved data analysis. The simulations and data analysis are totally distinct code wise, but I want to keep them in the same repo as the paper manuscript to make paper writing easy. The issue is now that I have two huge, separate "flows" living in the same _targets.R file, and there are fair number of name collisions, etc, etc.

Ideally I'd be able to have subfolders for each flow, but targets no longer allows this if I understand correctly (i.e. 1 _targets.R <-> 1 RStudio project) is my understanding. Thoughts on how to organize this?

[wlandau]

If the paper is a tar_render() Rmd report that gets dependencies from both parts of the pipeline, then I would say it is best to continue with a single _targets.R. But it sounds like this particular paper is not set up with continuous rendering.

As for multiple sub-pipelines in general, I think we are still working that out, and the discussion is mostly at #297. The unitar package started some of this for the local filesystem. Alternatively, depending on the size of the computation and data, you could continuously deploy each part of the pipeline separately to two different repos with an approach similar to https://github.com/wlandau/targets-minimal#continuous-deployment. Having multiple pipelines in separate locations might be easier if they are on the cloud.

When I work with multiple pipelines for the same overall project, I split them into one pipeline per repo or branch, with different branches concurrently running in different parts of the local file system. Otherwise, the nested folder structure makes version control and source file management cumbersome.