converting an old project to targets with time-consuming intermediate data objects

[venpopov]

Help

• I understand and agree to https://books.ropensci.org/targets/help.html.

Description

In the process of learning how to work with targets, I decided to convert an existing ongoing project. I've learned how to do it from scratch, but I have the following "problem". There is a computationally demanding step which simulates some models, which originally took about a week to complete. The object produced by this step is used in downstream analyses. I would like to avoid recomputing this object and just reuse it. Is it somehow possible to setup the targets pipeline such that it will consider some steps completed and reuse the (externally-generated) object as if it was produced by a step in the targets pipeline? I hope my question makes sense.

[joelnitta]

You could save the object to an RDS file, then use tar_file() to track the file contents and read it in to your pipeline. If for any reason the contents of the RDS file changed it would trigger downstream steps that depend on it. But AFAIK {targets} has no way of knowing how changes upstream of the model object would affect it, other then completely re-doing the workflow in {targets}.

[venpopov]

That's a really good justification for why not to allow this behavior, even if it were possible to do :)
I guess I'll just live with the annoyances that inevitably come with any workflow transition, and hope the long-term payoff is worth it.

I like that function, it's cleaner than having commented targets.

[joelnitta]

I'm glad it was helpful 🙂

TBH in my experience, it's not that unusual to end up in a situation where you have to re-run a long-running target even if you didn't mean to. You may have inadvertently changed something upstream, or on rare occasions there are updates to {targets} that can't use the old version's cache. You might consider gittargets to version-control the cache, if that is a concern going forward (mind though that this comes with its own mental and logistical overhead).

[venpopov]

Yeah, I had exactly that experience 6 months ago when I originally tried to transition to targets. Tried gittargets and eventually gave up because of the mental overhead exactly. My existing mental models and workflows - refactoring functions, switching branches for experiments, etc, didn't seem compatible with targets. Eventually what drew me to targets - the promise to avoid recomputing expensive steps - was counteracted. I'm hoping this time it goes better as I do like the idea behind it.

[joelnitta]

I suppose the ideal situation would be if {gittargets} wasn't needed as a separate package because that functionality was totally abstracted away and embedded within {targets}: if it saw you were re-running the code in a state that matched something exactly in the past, it could just restore the cache from that previous state. But this probably is much easier said than done!

[wlandau]

I may actually supersede gittargets in favor of content-addressable storage, c.f. ropensci/gittargets#18