renv project fails to load when there is Remotes to private repository

[mcanouil]

I tried many options (https://rstudio.github.io/renv/reference/config.html), but none of them seems to matters at startup, such as renv.config.install.remotes or renv.config.dependency.errors.

I have a DESCRIPTION file which includes a remote entry to a private GitHub repository.

Imports:
    mypackage
Remotes:
    github::myorg/myrepo:src/packages/mypackage@mypackage-v1.0.0

Token is not always defined before R starts and not even needed.
But this means renv cannot activate without the token which I can't understand the rational (if there is one) behind that behaviour.

The following is the error tweaked a bit to hide sensitive information:

Error: failed to resolve remote 'github::myorg/myrepo:src/packages/mypackage@mypackage-v1.0.0' -- error downloading 'https://api.github.com/repos/myorg/myrepo/commits/mypackage-v1.0.0' [curl: (22) The requested URL returned error: 404]
Traceback (most recent calls last):
37: source("renv/activate.R")
36: withVisible(eval(ei, envir))
35: eval(ei, envir)
34: eval(ei, envir)
33: local(...) at activate.R#2
32: eval.parent(substitute(eval(quote(expr), envir)))
31: eval(expr, p)
30: eval(expr, p)
29: eval(quote(...), new.env())
28: eval(quote(...), new.env())
27: renv_bootstrap_exec(project, libpath, version) at activate.R#1176
26: if (!renv_bootstrap_load(project, libpath, version))
      renv_bootstrap_run(version, libpath) at activate.R#1009
25: renv::load(project) at activate.R#864
24: renv_load_bioconductor(project, lockfile$Bioconductor)
23: renv_bioconductor_init()
22: renv_bioconductor_init_biocmanager(library)
21: install("BiocManager", library = library)
20: overlay(renv_project_remotes(project), remotes)
19: modifyList(as.list(lhs), as.list(rhs))
18: stopifnot(is.list(x), is.list(val))
17: as.list(lhs)
16: renv_project_remotes(project)
15: renv_description_remotes(descpath)
14: lapply(splat, renv_remotes_resolve)
13: FUN(X[[i]], ...)
12: withCallingHandlers(renv_remotes_resolve_impl(spec, latest), 
        error = error)
11: renv_remotes_resolve_impl(spec, latest)
10: renv_remotes_resolve_github(remote)
 9: case(nzchar(pull) ~ renv_remotes_resolve_github_sha_pull(host, 
        user, repo, pull), nzchar(ref) ~ renv_remotes_resolve_github_sha_ref(host, 
        user, repo, ref))
 8: renv_remotes_resolve_github_sha_ref(host, user, repo, ref)
 7: download(url, destfile = shafile, type = "github", quiet = TRUE, 
        headers = headers)
 6: renv_download_error(url, "%s", conditionMessage(status))
 5: stopf("error downloading '%s' [%s]", url, msg, call. = FALSE)
 4: stop(sprintf(fmt, ...), call. = call.)
 3: .handleSimpleError(function (e) 
    {
        ...
    }, "error downloading 'https://api.github.com/repos/myorg/myrepo/commits/mypackage-v1.0.0' [curl: (22) The requested URL returned error: 404]", 
        base::quote(NULL))
 2: h(simpleError(msg, call))
 1: stop(simpleError(message = message, call = e$call))

Did I miss something to avoid this error?

[kevinushey]

Thanks for the bug report -- I don't think you've missed anything here; renv is trying to (perhaps over-aggressively) resolve project remotes during an attempt to install BiocManager on startup, and that is failing.

[mcanouil]

A bit of follow up.
it appears the issue is not only at startup (and kind of really annoying here I must say), i.e., the error is triggered even when trying to install completely unrelated R package:

> renv::install("here")
Error: failed to resolve remote 'github::myorg/myrepo:src/packages/mypackage@mypackage-v1.0.0' -- error downloading 'https://api.github.com/repos/myorg/myrepo/commits/mypackage-v1.0.0' [curl: (22) The requested URL returned error: 404]
Traceback (most recent calls last):
24: renv::install("here")
23: overlay(renv_project_remotes(project), remotes)
22: modifyList(as.list(lhs), as.list(rhs))
21: stopifnot(is.list(x), is.list(val))
20: as.list(lhs)
19: renv_project_remotes(project)
18: renv_description_remotes(descpath)
17: lapply(splat, renv_remotes_resolve)
16: FUN(X[[i]], ...)
15: withCallingHandlers(renv_remotes_resolve_impl(spec, latest), 
        error = error)
14: renv_remotes_resolve_impl(spec, latest)
13: renv_remotes_resolve_github(remote)
12: remote$ref %||% renv_remotes_resolve_github_ref(host, user, repo)
11: renv_remotes_resolve_github_ref(host, user, repo)
10: tryCatch(renv_remotes_resolve_github_ref_impl(host, user, repo), 
        error = function(e) {
            warning(e)
            getOption("renv.github.default_branch", default = "master")
        })
 9: tryCatchList(expr, classes, parentenv, handlers)
 8: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 7: value[[3L]](cond)
 6: warning(e)
 5: withRestarts({
        .Internal(.signalCondition(cond, message, call))
        .Internal(.dfltWarn(message, call))
    }, muffleWarning = function() NULL)
 4: withOneRestart(expr, restarts[[1L]])
 3: doWithOneRestart(return(expr), restart)
 2: <condition-handler>(...)
 1: stop(simpleError(message = message, call = e$call))

[kevinushey]

For context, is there a reason why you can't make the requisite authentication token available, so that renv could successfully query that remote?

[mcanouil]

1. The token would be visible to people different from the one that generated the token.
2. The token cannot be stored permanently.
3. The token, when set, has an expiration date.

With the current behaviour, with the above "rules", the setup will break at some point.

[kevinushey]

Should be fixed via 5ea540a; please let me know if that seems to suffice for your use case.

[mcanouil]

Thanks with this, I can start R and activate renv.
But, I still need to set the token to my package to install any not related packages, as described in #1755 (comment).
Is it something that could also be handled by renv?

> renv::install("ggtext")
Error: failed to resolve remote 'github::myorg/myrepo:src/packages/mypackage@mypackage-v1.0.0' -- error downloading 'https://api.github.com/repos/myorg/myrepo/commits/mypackage-v1.0.0' [curl: (22) The requested URL returned error: 404]
Traceback (most recent calls last):
24: renv::install("ggtext")
23: records <- overlay(renv_project_remotes(project), remotes) at install.R#188
22: modifyList(as.list(lhs), as.list(rhs)) at utils.R#566
21: stopifnot(is.list(x), is.list(val))
20: modifyList(as.list(lhs), as.list(rhs)) at utils.R#566
19: records <- overlay(renv_project_remotes(project), remotes) at install.R#188
18: remotes <- renv_description_remotes(descpath) at project.R#112
17: resolved <- lapply(splat, renv_remotes_resolve) at description.R#193
16: FUN(X[[i]], ...)
15: withCallingHandlers(
      renv_remotes_resolve_impl(spec, latest),
      error = error
    ) at remotes.R#62
14: withCallingHandlers(
      renv_remotes_resolve_impl(spec, latest),
      error = error
    ) at remotes.R#62
13: resolved <- switch(
      remote$type,
      bioc       = renv_remotes_resolve_bioc(remote),
      bitbucket  = renv_remotes_resolve_bitbucket(remote),
      gitlab     = renv_remotes_resolve_gitlab(remote),
      github     = renv_remotes_resolve_github(remote),
      repository = renv_remotes_resolve_repository(remote, latest),
      git        = renv_remotes_resolve_git(remote),
      url        = renv_remotes_resolve_url(remote$url, quiet = TRUE),
      stopf("unknown remote type '%s'", remote$type %||% "<NA>")
    ) at remotes.R#81
12: ref <- remote$ref %||% renv_remotes_resolve_github_ref(host, user, repo) at remotes.R#629
11: ref <- remote$ref %||% renv_remotes_resolve_github_ref(host, user, repo) at remotes.R#629
10: tryCatch(
      renv_remotes_resolve_github_ref_impl(host, user, repo),
      error = function(e) {
        warning(e)
        getOption("renv.github.default_branch", default = "master")
      }
    ) at remotes.R#589
 9: tryCatchList(expr, classes, parentenv, handlers)
 8: tryCatchOne(expr, names, parentenv, handlers[[1L]])
 7: value[[3L]](cond)
 6: warning(e) at remotes.R#592
 5: withRestarts({
        .Internal(.signalCondition(cond, message, call))
        .Internal(.dfltWarn(message, call))
    }, muffleWarning = function() NULL)
 4: withOneRestart(expr, restarts[[1L]])
 3: doWithOneRestart(return(expr), restart)
 2: <condition-handler>(...)
 1: stop(simpleError(message = message, call = e$call)) at remotes.R#57

[kevinushey]

I think so; let me take a look.

[kevinushey]

I've made a change in b194034 that should help support this. The main requirement here is that you'll need to explicitly name the remote, by giving a prefix of mypackage=<...>. For example:

Remotes:
    mypackage=github::myorg/myrepo:src/packages/mypackage@mypackage-v1.0.0

Can you give this a shot with the development version of renv and let me know if it seems to help?

[mcanouil]

Thank you, it does the trick!

[kevinushey]

🎉 Thanks for your patience, and thanks for testing!