generate renv.lock file given a list of packages

[cboettig]

Apologies if this is out of scope, but I would love a way to give renv a list of bare package names and have it generate the corresponding renv.lock file (perhaps analogous to the Gemfile vs Gemfile.lock behavior in Ruby), e.g. something like:

snapshot(packages = c("tidyverse", "devtools"))

In which it would capture the listed packages and their dependencies (possibly configurable by dependencies = c("Depends", "Imports", "LinkingTo", "Suggests") ?).

This would also make it possible to create lockfiles for my full system (e.g. capture all the packages installed on a given Docker environment or VM, rather than just within a given project), with something like snapshot(packages = installed.packages()[,1]), which would be super useful to me.

Thoughts? Maybe there is some way to approximate this behavior with a DESCRIPTION file but I haven't gotten that to work. (e.g. if I create a blank new project and add some dependencies to DESCRIPTION, renv::dependencies() detects them but currently they do not seem to get added to my renv.lock when I do renv::snapshot() ??

[kevinushey]

This is the intended (default) behavior. From the renv vignette:

renv/vignettes/renv.Rmd

Lines 237 to 243 in 85137ba

	2. Packrat tried to maintain the distinction between so-called 'stale' packages;
	those being R packages which were installed by Packrat but are not recorded
	in the lockfile for some reason. This distinction was (1) overall not useful,
	and (2) confusing. `renv` no longer makes this distinction:
	`snapshot()` saves the state of your project library to `renv.lock`,
	`restore()` loads the state of your project library from `renv.lock`, and
	that's all.

If you want to snapshot a specific library, you can do so:

renv::snapshot(library = <...>)

It's not clear to me how snapshotting based on a package name only would work (how does renv know what version of the package you want and where you want that package to be installed from?) unless you mean packages as a filter on which packages enter the lockfile while still using an R library as a source of truth.

[cboettig]

@kevinushey That's awesome!

For some reason that does not always work for me consistently. Locally that is working fine. If I try in a docker container, e.g.

docker run --rm -ti rocker/tidyverse:3.5.3 R

(note this runs as root, and is just in working dir / by default), it errors. Running anything from / is dumb of course, so if I switch to ~/, snapshot() runs without error, but does not get any of the packages that live in libPaths()[[1]]. Running in a more normal workflow, e.g. from RStudio instance with non-root user, but using the same container, everything seems to works as expected.

It looks like the default behavior grabs only the first library on .libPaths(). I get a warning about the fact that some dependencies are not installed because in this case, base R packages like Matrix are on the second library in .libPaths(), which is maybe good enough, but it would be convenient to get everything from .libPaths() into the lock file without having to reinstall.

[kevinushey]

(note this runs as root, and is just in working dir / by default), it errors.

What error are you seeing, exactly? This reminds me that when you try to snapshot an arbitrary library, we do attempt to filter what enters the lockfile based on what's used in the 'active' project (ie: current working directory)

renv/R/snapshot.R

Lines 35 to 38 in 85137ba

	# if we're snapshotting a non-project library, then filter the
	# packages that will enter the lockfile based on what's in project
	filter <- if (!identical(library, renv_paths_library(project = project)))
	renv_snapshot_filter(project = project)

The intention here was that you probably don't want everything in an arbitrary user library to enter the lockfile, but I don't think that matches your intent here.

It looks like the default behavior grabs only the first library on .libPaths(). I get a warning about the fact that some dependencies are not installed because in this case, base R packages like Matrix are on the second library in .libPaths(), which is maybe good enough, but it would be convenient to get everything from .libPaths() into the lock file without having to reinstall.

It seems like we should do this. The main question here is: do we need to track which library a particular package dependency came from? Ie, if someone wants to call renv::restore() later, do we need to confirm whether a package should get installed into the system library vs. user library vs. private library?

[cboettig]

Running from root dir (/) it takes a while to error and has a long callback that just fills my page, but appears to be path related:

   29: renv_dependencies_discover_dir(path, root)
   28: FUN(X[[i]], ...)
   27: lapply(children, renv_dependencies_discover, root = root)
   26: renv_dependencies_discover_dir(path, root)
   25: FUN(X[[i]], ...)
   24: lapply(children, renv_dependencies_discover, root = root)
   23: renv_dependencies_discover_dir(path, root)
   22: FUN(X[[i]], ...)
   21: lapply(children, renv_dependencies_discover, root = root)
   20: renv_dependencies_discover_dir(path, root)
   19: FUN(X[[i]], ...)
   18: lapply(children, renv_dependencies_discover, root = root)
   17: renv_dependencies_discover_dir(path, root)
   16: FUN(X[[i]], ...)
   15: lapply(children, renv_dependencies_discover, root = root)
   14: renv_dependencies_discover_dir(path, root)
   13: FUN(X[[i]], ...)
   12: lapply(children, renv_dependencies_discover, root = root)
   11: renv_dependencies_discover_dir(path, root)
   10: FUN(X[[i]], ...)
    9: lapply(children, renv_dependencies_discover, root = root)
    8: renv_dependencies_discover_dir(path, root)
    7: FUN(X[[i]], ...)
    6: lapply(children, renv_dependencies_discover, root = root)
    5: renv_dependencies_discover_dir(path, root)
    4: renv_dependencies_discover(path, path)
    3: dependencies(project)
    2: renv_snapshot_filter(project = project)
    1: renv::snapshot()

though this bug may be too much of an edge case to really worry about.

Not sure why I don't get the error but also don't get my system library when running from root's home dir, /root, as root. that feels more like a bug.

Regarding which library to restore to, my take is that renv shouldn't track that. I see renv as providing a portable abstraction of the project dependencies, which should be agnostic to whether those are in the system library, user library or private library (e.g. I can give you a renv.lock of the system library in my docker image and you can deploy that environment in a private library on your laptop if you prefer). But of course my perspective may be biased by my own use cases.

[kevinushey]

I've now implemented support for snapshotting multiple libraries -- something like:

renv::snapshot(library = .libPaths())

should work as expected. Let me know if you bump into any issues.

[kevinushey]

I'm going to close this; please open a new issue if you bump into other problems. Thanks!

[michalc]

I would love a way to give renv a list of bare package names and have it generate the corresponding renv.lock file (perhaps analogous to the Gemfile vs Gemfile.lock behavior in Ruby),

In case helpful, I came up with this one liner (to be run from the shell), that clears the renv.lock file, and regenerates it based on the dependencies listed in DESCRIPTION. But note! It will delete all installed packages from your local environment.

Rscript -e 'if (file.exists("renv.lock")) {file.remove("renv.lock")}; remove.packages(installed.packages(priority="NA")[,1]); install.packages("renv", repos="https://cloud.r-project.org"); renv::settings$snapshot.type("explicit"); renv::install(library=tempdir(), lock=TRUE)'

This makes it fairly close to Python’s requirements.in / requirements.txt pattern, JavaScript’s/Node’s package.json / package-lock.json pattern, and (I assume) Ruby’s Gemfile / Gemfile.lock pattern.

(Because of the clearing of the lock file, there is no concept of “only upgrading certain packages and no others”, but I can live with that…)