Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support restore from r-universe repositories #789

Closed
kevinushey opened this issue Jun 28, 2021 · 19 comments
Closed

support restore from r-universe repositories #789

kevinushey opened this issue Jun 28, 2021 · 19 comments

Comments

@kevinushey
Copy link
Collaborator

For example, with latest rlang:

Package: rlang
Version: 0.4.11.9000
Title: Functions for Base Types and Core R and 'Tidyverse' Features
Description: A toolbox for working with base types, core R features
  like the condition system, and core 'Tidyverse' features like tidy
  evaluation.
Authors@R: c(
    person("Lionel", "Henry", ,"lionel@rstudio.com", c("aut", "cre")),
    person("Hadley", "Wickham", ,"hadley@rstudio.com", "aut"),
    person(given = "mikefc",
           email = "mikefc@coolbutuseless.com", 
           role = "cph", 
           comment = "Hash implementation based on Mike's xxhashlite"),
    person(given = "Yann",
           family = "Collet",
           role = "cph", 
           comment = "Author of the embedded xxHash library"),
    person("RStudio", role = "cph")
    )
License: MIT + file LICENSE
ByteCompile: true
Biarch: true
Depends: R (>= 3.3.0)
Imports: utils
Suggests: cli (>= 2.5.0), covr, crayon, fs, glue, knitr, magrittr,
        methods, pak, pillar, rmarkdown, stats, testthat (>= 3.0.0),
        tibble, usethis, vctrs (>= 0.2.3), withr
Enhances: winch
SystemRequirements: C++11
Encoding: UTF-8
RoxygenNote: 7.1.1
Roxygen: list(markdown = TRUE)
URL: https://rlang.r-lib.org, https://github.com/r-lib/rlang
BugReports: https://github.com/r-lib/rlang/issues
Config/testthat/edition: 3
Repository: https://r-lib.r-universe.dev
RemoteUrl: https://github.com/r-lib/rlang
RemoteRef: HEAD
RemoteSha: 869f00a27acfda52b2e428cdf7292895c4e8061c
NeedsCompilation: yes
Packaged: 2021-06-28 12:21:58 UTC; root
Author: Lionel Henry [aut, cre],
  Hadley Wickham [aut],
  mikefc [cph] (Hash implementation based on Mike's xxhashlite),
  Yann Collet [cph] (Author of the embedded xxHash library),
  RStudio [cph]
Maintainer: Lionel Henry <lionel@rstudio.com>
Built: R 4.1.0; x86_64-w64-mingw32; 2021-06-28 12:24:59 UTC; windows
Archs: i386, x64

I think we would want to detect the use of r-universe within the Repository field, and then use + other remote fields to set the repositories to use before restore.

@jeroen
Copy link

jeroen commented Sep 9, 2021

Let me know if I can help here!

R-universe currently does not archive old versions of packages, because they may be updated very often. But all r-universe packages are taken from git by design. Hence my intention was that by including the upstream RemoteUrl and RemoteSha in the DESCRIPTION for r-universe packages, renv woud be able to restore the package from Git, in the same way as when it had been installed from Git directly, rather than via R-universe.

@zkamvar
Copy link
Contributor

zkamvar commented Oct 11, 2021

+1 for this support. I've taken some notes about my experience mixing the two here: carpentries/sandpaper#163.

@florisvdh
Copy link

florisvdh commented Oct 14, 2021

A related aspect is that currently renv seems to connect a locally built package to the r-universe repository if the package has an (older) build in r-universe, without warning. (Let me know if I should move this to a separate issue)

I.e. in the following situation:

  • having built a package 'n2khab' locally, with a hash that is not the one released through r-universe (typical situation while developing)
sessioninfo::session_info("n2khab")
Session info ───────────────────────────────────────────────────────────────────────────────
 setting  value                       
 version  R version 4.1.1 (2021-08-10)
 os       Linux Mint 20               
 system   x86_64, linux-gnu           
 ui       RStudio                     
 language nl_BE:nl                    
 collate  nl_BE.UTF-8                 
 ctype    nl_BE.UTF-8                 
 tz       Europe/Brussels             
 date     2021-10-14Packages ───────────────────────────────────────────────────────────────────────────────────
 package    * version   date       lib source        
 assertthat   0.2.1     2019-03-21 [1] CRAN (R 4.1.0)
 class        7.3-19    2021-05-03 [4] CRAN (R 4.1.0)
 classInt     0.4-3     2020-04-07 [1] CRAN (R 4.1.0)
 cli          3.0.1     2021-07-17 [1] CRAN (R 4.1.1)
 cpp11        0.4.0     2021-09-22 [1] CRAN (R 4.1.1)
 crayon       1.4.1     2021-02-08 [1] CRAN (R 4.1.0)
 DBI          1.1.1     2021-01-15 [1] CRAN (R 4.1.0)
 dplyr        1.0.7     2021-06-18 [1] CRAN (R 4.1.1)
 e1071        1.7-9     2021-09-16 [1] CRAN (R 4.1.1)
 ellipsis     0.3.2     2021-04-29 [1] CRAN (R 4.1.0)
 fansi        0.5.0     2021-05-25 [1] CRAN (R 4.1.0)
 forcats      0.5.1     2021-01-27 [1] CRAN (R 4.1.0)
 generics     0.1.0     2020-10-31 [1] CRAN (R 4.1.0)
 git2r        0.28.0    2021-01-10 [1] CRAN (R 4.1.0)
 git2rdata    0.3.1     2021-01-21 [1] CRAN (R 4.1.0)
 glue         1.4.2     2020-08-27 [1] CRAN (R 4.1.0)
 KernSmooth   2.23-20   2021-05-03 [4] CRAN (R 4.1.0)
 lifecycle    1.0.1     2021-09-24 [1] CRAN (R 4.1.1)
 magrittr     2.0.1     2020-11-17 [1] CRAN (R 4.1.0)
 MASS         7.3-54    2021-05-03 [1] CRAN (R 4.1.0)
 n2khab       0.5.0.900 2021-10-14 [1] local         
 pillar       1.6.3     2021-09-26 [1] CRAN (R 4.1.1)
 pkgconfig    2.0.3     2019-09-22 [1] CRAN (R 4.1.0)
 plyr         1.8.6     2020-03-03 [1] CRAN (R 4.1.0)
 proxy        0.4-26    2021-06-07 [1] CRAN (R 4.1.0)
 purrr        0.3.4     2020-04-17 [1] CRAN (R 4.1.0)
 R6           2.5.1     2021-08-19 [1] CRAN (R 4.1.1)
 Rcpp         1.0.7     2021-07-07 [1] CRAN (R 4.1.1)
 rlang        0.4.11    2021-04-30 [1] CRAN (R 4.1.0)
 rprojroot    2.0.2     2020-11-15 [1] CRAN (R 4.1.0)
 s2           1.0.7     2021-09-28 [1] CRAN (R 4.1.1)
 sf           1.0-3     2021-10-07 [1] CRAN (R 4.1.1)
 stringi      1.7.5     2021-10-04 [1] CRAN (R 4.1.1)
 stringr      1.4.0     2019-02-10 [1] CRAN (R 4.1.0)
 tibble       3.1.5     2021-09-30 [1] CRAN (R 4.1.1)
 tidyr        1.1.4     2021-09-27 [1] CRAN (R 4.1.1)
 tidyselect   1.1.1     2021-04-30 [1] CRAN (R 4.1.0)
 units        0.7-2     2021-06-08 [1] CRAN (R 4.1.1)
 utf8         1.2.2     2021-07-24 [1] CRAN (R 4.1.1)
 vctrs        0.3.8     2021-04-29 [1] CRAN (R 4.1.0)
 withr        2.4.2     2021-04-18 [1] CRAN (R 4.1.0)
 wk           0.5.0     2021-07-13 [1] CRAN (R 4.1.1)
 yaml         2.2.1     2020-02-01 [1] CRAN (R 4.1.0)

[1] /home/floris/lib/R/library
[2] /usr/local/lib/R/site-library
[3] /usr/lib/R/site-library
[4] /usr/lib/R/library
  • with the corresponding r-universe repository set (where this package is present already):
> getOption("repos")[3]
                         inbo 
"https://inbo.r-universe.dev" 
Then the following happens (using `renv` at current master - 4642f5d) - this discussion is continued in #872
> renv::init()
* Initializing project ...
* Discovering package dependencies ... Done!
* Copying packages into the cache ... [123/123] Done!
The following package(s) will be updated in the lockfile:

# CRAN ===============================
## <long table omitted>

# GitHub =============================
- bookdown        [* -> rstudio/bookdown@fix/1268-css-toc-has-sub]
- renv            [* -> rstudio/renv@HEAD]

# https://inbo.r-universe.dev ========
- INBOtheme       [* -> 0.5.9]

# inbo ===============================
- n2khab          [* -> 0.5.0.900]

* Lockfile written to <projectdir>.

Restarting R session...

There's also another package 'INBOtheme' which was effectively installed from the same r-universe, this looks OK.

Two strange things to note here:

  1. the 'inbo' r-universe is depicted twice, in a different manner (its URL vs its name)
  2. n2khab package is misjudged as coming from the 'inbo' r-universe, while it's locally built. I assume this cannot work on restore. I'd rather expect this should give a warning that it's not recommended to record locally built packages in renv.lock; anyway it should not link to the r-universe.

Extracts from renv.lock:

{
  "R": {
    "Version": "4.1.1",
    "Repositories": [
      {
        "Name": "CRAN",
        "URL": "https://cloud.r-project.org"
      },
      {
        "Name": "INLA",
        "URL": "https://inla.r-inla-download.org/R/stable"
      },
      {
        "Name": "inbo",
        "URL": "https://inbo.r-universe.dev"
      }
    ]
  },
  "Packages": {
    "INBOtheme": {
      "Package": "INBOtheme",
      "Version": "0.5.9",
      "Source": "Repository",
      "Repository": "https://inbo.r-universe.dev",
      "Remotes": "clauswilke/colorblindr",
      "RemoteUrl": "https://github.com/inbo/inbotheme",
      "RemoteRef": "HEAD",
      "RemoteSha": "2a41272624e9092d24f0e92772bfebeef12c695c",
      "Hash": "b9a442f47af951da83cf0dedb630a083"
    },
    "n2khab": {
      "Package": "n2khab",
      "Version": "0.5.0.900",
      "Source": "Repository",
      "Repository": "inbo",
      "Hash": "9aafc0465394ce0bd93a7220c20db697"
    },
    "renv": {
      "Package": "renv",
      "Version": "0.14.0-8",
      "Source": "GitHub",
      "RemoteType": "github",
      "RemoteHost": "api.github.com",
      "RemoteRepo": "renv",
      "RemoteUsername": "rstudio",
      "RemoteRef": "HEAD",
      "RemoteSha": "4642f5dc7058e067438966ef99a28c0b84fd87c8",
      "Hash": "df4aead47901866f9915b905ee3af4a1"
    },

@kevinushey
Copy link
Collaborator Author

A related aspect is that currently renv seems to connect a locally built package to the r-universe repository if the package has an (older) build in r-universe, without warning. (Let me know if I should move this to a separate issue)

n2khab package is misjudged as coming from the 'inbo' r-universe, while it's locally built. I assume this cannot work on restore. I'd rather expect this should give a warning that it's not recommended to record locally built packages in renv.lock; anyway it should not link to the r-universe.

This is at least partly intentional; see

renv/R/snapshot.R

Lines 684 to 697 in 4642f5d

# NOTE: this is sort of a hack that allows renv to declare packages which
# appear to be installed from sources, but are actually available on the
# active R package repositories, as though they were retrieved from that
# repository. however, this is often what users intend, especially if
# they haven't configured their repository to tag the packages it makes
# available with the 'Repository:' field in the DESCRIPTION file.
#
# still, this has the awkward side-effect of a package's source potentially
# depending on what repositories happen to be active at the time of snapshot,
# so it'd be nice to tighten up the logic here if possible
entry <- local({
renv_scope_options(renv.verbose = FALSE)
catch(renv_available_packages_latest(package))
})
. The intention here is to allow packages which were installed from sources, but at least appear to be available from a package repository, to be restored from that repository.

My guess is that this package is indeed available (in some version) from that universe repository URL, and so it's recorded as such.

the 'inbo' r-universe is depicted twice, in a different manner (its URL vs its name)

This seems correct to me? The actual repository name + URL pair itself is recorded in the Repository field (since that's what was active in your session), and the package was recorded with that repository name in the lockfile. Of course, the fact that all of the remote fields are missing likely implies renv will fail to restore the package, but the solution seems clear to me -- install an updated version of the package directly from the r-universe repository.

@kevinushey
Copy link
Collaborator Author

(and sorry, I've been taking an extended leave from work so haven't had a chance to look through these issues in more detail)

@florisvdh
Copy link

florisvdh commented Oct 15, 2021

Thank you @kevinushey for explaining the logic and pointing to the remark in snapshot.R, makes sense.

If such locally built package is absent from any repository, renv::status() triggers a warning (and renv::init() explicitly says it's from an 'unknown' source):

> renv::status()
* The project is already synchronized with the lockfile.
The following package(s) were installed from an unknown source:
         _
  n2khab   [0.5.0.900]

renv may be unable to restore these packages in the future.
Consider reinstalling these packages from a known source (e.g. CRAN).

This warning would also hold for the previous case, where it is present in a repo though not in the local state, but in that case renv::status() is silent (just saying 'The project is already synchronized with the lockfile'). Going off-topic here, since that's indeed not specific to r-universe repositories 🤔. I agree with the logic that you explained, but IMHO it would be good that at least renv::init() throws a warning about what it is doing in such case. Would that make sense to you?

@florisvdh
Copy link

florisvdh commented Oct 15, 2021

Regarding how the 'inbo' r-universe is depicted in two different ways, I was referring to the difference between packages INBOtheme and n2khab in the output of renv::init():

# https://inbo.r-universe.dev ========  ## THIS LINE
- INBOtheme       [* -> 0.5.9]

versus

# inbo ===============================  ## THIS LINE
- n2khab          [* -> 0.5.0.900]

Same in the above example renv.lock:

      "Source": "Repository",
      "Repository": "https://inbo.r-universe.dev",

versus

      "Source": "Repository",
      "Repository": "inbo",

Sure it is not wrong, it's just not 100% consistent; I'd expect either the name or the URL in both cases.

BTW - I love renv!

@kevinushey
Copy link
Collaborator Author

@florisvdh, coming back to this now -- I wonder if this is ultimately due to a change in how r-universe annotated the repository in the DESCRIPTION file for a downloaded package? For example:

> options(repos = c(
+     rstudio = 'https://rstudio.r-universe.dev',
+     CRAN = 'https://cloud.r-project.org'))
> renv::install("rstudioapi")
Installing rstudioapi [0.13.0-9000] ...
	OK [copied cache]
> desc <- renv:::renv_description_read(package = "rstudioapi")
> desc[["Repository"]]
[1] "https://rstudio.r-universe.dev"

I wonder if r-universe changed how it recorded the Repository field for these different universes.

From what I can see, renv can handle r-universe repositories without any changes required, as long as those repositories are also recorded within the lockfile at the top level. However, we could probably make this a bit more robust -- in particular, if the Repository field starts with https://, then we should explicitly use that repository URL when attempting to restore the package.

@kevinushey
Copy link
Collaborator Author

R-universe currently does not archive old versions of packages, because they may be updated very often. But all r-universe packages are taken from git by design. Hence my intention was that by including the upstream RemoteUrl and RemoteSha in the DESCRIPTION for r-universe packages, renv woud be able to restore the package from Git, in the same way as when it had been installed from Git directly, rather than via R-universe.

This makes sense -- I'll see if we can use this when installing a package from the "archive" rather than looking into the regular repository archive locations.

@florisvdh
Copy link

florisvdh commented Nov 16, 2021

@kevinushey thanks for the hints.

n2khab has no Repository field in its local build:

> desc <- renv:::renv_description_read(package = "n2khab")
> desc[["Repository"]]
NULL

It is the locally built package that caused the # inbo ======== heading (same in renv.lock).

Further digging, part of which is recycled in #872

Presumably that inbo name is based on the GitHub "inbo" account where the package refers to - or could it be that it derives the mere presence of the package at r-universe from a list of packages present at inbo.r-universe.dev, and makes the connection in that way? Apart from emailadresses, the only references to inbo in the (locally built) package DESCRIPTION are:

URL: https://inbo.github.io/n2khab, https://github.com/inbo/n2khab
BugReports: https://github.com/inbo/n2khab/issues

Anyway, the absence of r-universe (or any) Repository from DESCRIPTION will indeed be explanation of not appearing under the # https://inbo.r-universe.dev ====== heading (same in renv.lock), so it doesn't come from a change in r-universe but just from using the local package source. The corresponding source package at r-universe does have Repository: https://inbo.r-universe.dev in its DESCRIPTION, so that one should work normal.

inbo is also the name of the r-universe repository in renv.lock:

Extract from renv.lock:

{
  "R": {
    "Version": "4.1.2",
    "Repositories": [
<lines omitted>
      {
        "Name": "inbo",
        "URL": "https://inbo.r-universe.dev"
      }
    ]
  },
  "Packages": {
    "n2khab": {
      "Package": "n2khab",
      "Version": "0.5.0.900",
      "Source": "Repository",
      "Repository": "inbo",
      "Hash": "9aafc0465394ce0bd93a7220c20db697"
    },

So the essential point is that the locally built package is assumed to come from a Repository (notably inbo), while it doesn't. It should be treated as just another locally built package (and ideally warned for like above).

EDIT: the fact that it is linked to the repository should be considered just fine, considering #789 (comment).
The only inconsistency is the repository name "inbo" versus "https://inbo.r-universe.dev", depending on the installation source (local versus r-universe)

@kevinushey
Copy link
Collaborator Author

Restoration of packages installed from r-universe repositories should now be handled. For example, with the lockfile:

{
  "R": {
    "Version": "4.1.2",
    "Repositories": [
      {
        "Name": "rlib",
        "URL": "https://r-lib.r-universe.dev"
      }
    ]
  },
  "Packages": {
    "rlang": {
      "Package": "rlang",
      "Version": "0.4.11.9001",
      "Source": "Repository",
      "Repository": "https://r-lib.r-universe.dev",
      "RemoteUrl": "https://github.com/r-lib/rlang",
      "RemoteRef": "HEAD",
      "RemoteSha": "26c4a878473578cf7b4c630f457a3bdfc46c8c31"
    }
  }
}

I see:

> renv::restore(packages = "rlang")
The following package(s) will be updated:

# https://r-lib.r-universe.dev =======
- rlang   [* -> 0.4.11.9001]

Do you want to proceed? [y/N]: y
* Querying repositories for available source packages ... Done!
Installing rlang [0.4.11.9001] ...
        OK [built from source]
Moving rlang [0.4.11.9001] into the cache ...
        OK [moved to cache in 0.29 milliseconds]

In this case, renv saw that no binary or source copy of this version of rlang was available in the current r-universe repository, and so fell back to installing directly using git.

@pfuehrlich-pik
Copy link

That's great news, thank you so much! 😄 Is there a CRAN release planned anytime soon?

@florisvdh
Copy link

florisvdh commented Nov 18, 2021

Amazing 🚀, thanks a lot @kevinushey. Could reproduce that! I've experimented a little further.

As you explained earlier, the behaviour of renv to tie local packages to a repository when possible, is intentional, and this already seemed to work in renv 0.14.0 for any repository, including r-universe repositories - so my latest reported output should be considered just fine, sorry about the confusion.

Below part has been copied to #872

There seems to be a change for the case where the local package cannot be linked to an active repository:

Both below cases only have the CRAN repo enabled (options(repos = c(CRAN = "https://cloud.r-project.org/"))).

with renv 0.14.0-50 (github):

This locally built package (which is not on CRAN, only in the inbo r-universe repo = disabled in this session) is now stored like:

    "n2khab": {
      "Package": "n2khab",
      "Version": "0.5.0.900",
      "Source": "Repository",
      "Repository": null,
      "Hash": "9aafc0465394ce0bd93a7220c20db697"
    },

So a local package is now always set as coming from a repository, here being the 'null' repository (before, it would have been source 'unknown', not considered as coming from a repo).

In this case no warning came anymore from snapshot() (which now reports this package under # Repository ========) or status().

==> Suggestion: should a warning not have been risen, as it was in 0.14.0? - see below

with renv 0.14.0 (CRAN):

Local package registered as:

    "n2khab": {
      "Package": "n2khab",
      "Version": "0.5.0.900",
      "Source": "unknown",
      "Hash": "9aafc0465394ce0bd93a7220c20db697"
    },

Where warnings occurred:

> renv::snapshot()
The following package(s) were installed from an unknown source:
         _
  n2khab   [0.5.0.900]

renv may be unable to restore these packages in the future.
Consider reinstalling these packages from a known source (e.g. CRAN).

Do you want to proceed? [y/N]: y
The following package(s) will be updated in the lockfile:
# CRAN ===============================
...
# (Unknown Source) ===================
- n2khab       [* -> 0.5.0.900]
Do you want to proceed? [y/N]: y
* Lockfile written to '/.../renv.lock'.

> renv::status()
* The project is already synchronized with the lockfile.
The following package(s) were installed from an unknown source:
         _
  n2khab   [0.5.0.900]

renv may be unable to restore these packages in the future.
Consider reinstalling these packages from a known source (e.g. CRAN).

@zkamvar
Copy link
Contributor

zkamvar commented Nov 18, 2021

I'm wondering if there is a way to flag universe repositories as "in development" in the lockfile and have {renv} restore from the latest version.

The reason I ask is that I use the universe for two reasons:

  1. to distribute packages that are undergoing rapid development
  2. to distribute compiled dependencies that are frustrating to install on mac and windows when their versions update on CRAN ({stringi} and {textshaping} are two examples that update often enough where this becomes a problem).

In the latter case, it would be really good to have a mechanism for people to restore to the latest binary version.

@ddsjoberg
Copy link

@zkamvar perhaps this issue about tracking the most recent release (rather than the current version on GitHub) will be helpful?
#792

@kevinushey
Copy link
Collaborator Author

I'll think about a CRAN release soon, but given the amount of code changes that have become part of the development version I'd like to give users (like yourselves!) more of a chance to kick the tires to make sure there aren't any unintended regressions.

Re: #789 (comment); would you mind distilling this into a separate issue?

@zkamvar
Copy link
Contributor

zkamvar commented Nov 18, 2021

@ddsjoberg, you are my hero today! This along with the '*release' branch tag in the r-universe packages.json is exactly what I needed ^_^

@florisvdh
Copy link

florisvdh commented Nov 19, 2021

Re: #789 (comment); would you mind distilling this into a separate issue?

👍 Done: #872 (+ shortened some related comments in current issue)

@kevinushey
Copy link
Collaborator Author

This has been implemented on the main branch now; I think this can be closed. Please feel free to file an issue if you encounter any other issues!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants