Remotes fails with private package dependency

[muschellij2]

I think this is the overall issue similar to #701 and the same issue as #697.

Description: If you pass in auth_token to install_github and don't have GITHUB_PAT set (so that remotes:::github_pat() returns NULL), you will get a failure. Let's say we have pkgA (private repo, muschellij2/pkgA) and pkgB (private repo, muschellij2/pkgB), that depends on pkgA and has pkgA set in the Remotes field.

Below, I have a reprex that demonstrates the issue, but the issue is that the packages are private (which is the overall issue). I'm happy to make an r-lib maintainer to make things easier. I also created a video demonstrating the issue: https://youtu.be/qZi_50QojbM.

Issue: Expected behavior is that if auth_token is passed to install_github, then GITHUB_PAT does not need to be set. The issue is that auth_token doesn't get passed to the dependency remotes. The failure happens at remote_sha, and package2remote (as package2remote uses remotes:::github_pat() for the auth_token parameter).

Solution: If you have private dependencies, you must set GITHUB_PAT for this to work. The auth_token will not pass through.

Reprex

GITHUB_PAT is in object auth_token - done outside of reprex (deleted line)

nchar(auth_token)
#> [1] 40

Demonstration of the issue

Overall, things are fine if GITHUB_PAT or GITHUB_TOKEN are set as environment variables, because remotes::github_pat will pick them up. In our case, we see they are not set:

Sys.getenv("GITHUB_PAT")
#> [1] ""
Sys.getenv("GITHUB_TOKEN")
#> [1] ""

Try to install package - fails because pkgA is private

remotes::install_github("muschellij2/pkgA")
#> Error: Failed to install 'unknown package' from GitHub:
#>   HTTP error 404.
#>   Not Found
#> 
#>   Did you spell the repo owner (`muschellij2`) and repo name (`pkgA`) correctly?
#>   - If spelling is correct, check that you have the required permissions to access the repo.

Pass in auth_token directly - succeeds

remotes::install_github("muschellij2/pkgA", auth_token = auth_token)
#> Downloading GitHub repo muschellij2/pkgA@HEAD
#> * checking for file ‘/tmp/RtmpzJUpNR/remotes1acb477ff0ad/muschellij2-pkgA-9d94302887cda58639c76f1e06ffc3664e29138d/DESCRIPTION’ ... OK
#> * preparing ‘pkgA’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building ‘pkgA_0.0.0.9000.tar.gz’
#> Installing package into '/home/gcer/R'
#> (as 'lib' is unspecified)

Following fails because pkgA is private and auth_token passed. But note that it says Failed to install 'pkgB' from GitHub:

remotes::install_github("muschellij2/pkgB", auth_token = auth_token)
#> Downloading GitHub repo muschellij2/pkgB@HEAD
#> Error: Failed to install 'pkgB' from GitHub:
#>   HTTP error 404.
#>   Not Found
#> 
#>   Did you spell the repo owner (`muschellij2`) and repo name (`pkgA`) correctly?
#>   - If spelling is correct, check that you have the required permissions to access the repo.

Issue resolved if GITHUB_PAT is set

Here we will set GITHUB_PAT, show remotes:::github_pat picks it up, and install_github works fine.

Sys.getenv("GITHUB_PAT")
#> [1] ""
Sys.setenv(GITHUB_PAT = auth_token)
nchar(remotes:::github_pat())
#> [1] 40

The installation succeeds below because it is using GITHUB_PAT for both auth_token (initial package call) and when remotes:::github_pat() called in dependency.

remotes::install_github("muschellij2/pkgB")
#> Using github PAT from envvar GITHUB_PAT. Use `gitcreds::gitcreds_set()` and unset GITHUB_PAT in .Renviron (or elsewhere) if you want to use the more secure git credential store instead.
#> Downloading GitHub repo muschellij2/pkgB@HEAD
#> 
#> * checking for file ‘/tmp/RtmpzJUpNR/remotes1acb60a90f49/muschellij2-pkgB-c3b57cff32842cd365e1d5582f981648a28e7705/DESCRIPTION’ ... OK
#> * preparing ‘pkgB’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building ‘pkgB_0.0.0.9000.tar.gz’
#> Installing package into '/home/gcer/R'
#> (as 'lib' is unspecified)
remove.packages("pkgB")
#> Removing package from '/home/gcer/R'
#> (as 'lib' is unspecified)

This includes succeeding when auth_token is passed (auth_token used for pkgB, remotes:::github_pat used for pkgA and all dependencies:

remotes::install_github("muschellij2/pkgB", auth_token = auth_token)
#> Downloading GitHub repo muschellij2/pkgB@HEAD
#> 
#> * checking for file ‘/tmp/RtmpzJUpNR/remotes1acb2efeb5e2/muschellij2-pkgB-c3b57cff32842cd365e1d5582f981648a28e7705/DESCRIPTION’ ... OK
#> * preparing ‘pkgB’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building ‘pkgB_0.0.0.9000.tar.gz’
#> Installing package into '/home/gcer/R'
#> (as 'lib' is unspecified)
remove.packages("pkgB")
#> Removing package from '/home/gcer/R'
#> (as 'lib' is unspecified)
Sys.unsetenv("GITHUB_PAT")

Demonstrating it is GITHUB_PAT

Here can can pass in garbage to GITHUB_PAT to demonstrate the main repo (pkgB) is using auth_token by getting information. Note below says Failed to install 'pkgB' from GitHub:

Sys.setenv(GITHUB_PAT = "asdlfkjas;dfja")
remotes::install_github("muschellij2/pkgB", auth_token = auth_token)
#> Downloading GitHub repo muschellij2/pkgB@HEAD
#> Error: Failed to install 'pkgB' from GitHub:
#>   HTTP error 401.
#>   Bad credentials
#> 
#>   Rate limit remaining: 53/60
#>   Rate limit reset at: 2023-06-02 19:06:18 UTC
#> 
#> 
remove.packages("pkgB")
#> Removing package from '/home/gcer/R'
#> (as 'lib' is unspecified)
#> Error in find.package(pkgs, lib): there is no package called 'pkgB'
Sys.unsetenv("GITHUB_PAT")

Here can can pass in garbage to auth_token to show that it fails at the point of reading the main package, such that it is not using GITHUB_PAT for the main repo read. Note unknown package vs. pkgB above:

Sys.getenv("GITHUB_PAT")
#> [1] ""
Sys.setenv(GITHUB_PAT = auth_token)
remotes::install_github("muschellij2/pkgB", auth_token = "asdfdsf")
#> Error: Failed to install 'unknown package' from GitHub:
#>   HTTP error 401.
#>   Bad credentials
#> 
#>   Rate limit remaining: 52/60
#>   Rate limit reset at: 2023-06-02 19:06:18 UTC
#> 
#> 
remove.packages("pkgB")
#> Removing package from '/home/gcer/R'
#> (as 'lib' is unspecified)
#> Error in find.package(pkgs, lib): there is no package called 'pkgB'
Sys.unsetenv("GITHUB_PAT")

Where the problem occurs

The issue occurs in dev_package_deps. In order to use dev_package_deps locally, we need to clone the package, and need to set GITHUB_PAT for git2r::cred_token:

Sys.getenv("GITHUB_PAT")
#> [1] ""
Sys.setenv(GITHUB_PAT = auth_token)
pkgdir = tempfile()
git2r::clone(url = "https://github.com/muschellij2/pkgB", 
             local_path = pkgdir,
             credentials = git2r::cred_token())
#> cloning into '/tmp/RtmpzJUpNR/file1acb67656799'...
#> Receiving objects:   8% (1/12),    9 kb
#> Receiving objects:  16% (2/12),    9 kb
#> Receiving objects:  25% (3/12),    9 kb
#> Receiving objects:  33% (4/12),    9 kb
#> Receiving objects:  41% (5/12),    9 kb
#> Receiving objects:  58% (7/12),    9 kb
#> Receiving objects:  66% (8/12),   13 kb
#> Receiving objects:  75% (9/12),   13 kb
#> Receiving objects:  83% (10/12),   13 kb
#> Receiving objects:  91% (11/12),   13 kb
#> Receiving objects: 100% (12/12),   13 kb, done.
#> Local:    main /tmp/RtmpzJUpNR/file1acb67656799
#> Remote:   main @ origin (https://github.com/muschellij2/pkgB)
#> Head:     [c3b57cf] 2023-06-02: added things

We can see that it does work:

readLines(file.path(pkgdir, "DESCRIPTION"))
#>  [1] "Package: pkgB"                                                                            
#>  [2] "Title: What the Package Does (One Line, Title Case)"                                      
#>  [3] "Version: 0.0.0.9000"                                                                      
#>  [4] "Authors@R: "                                                                              
#>  [5] "    person(\"First\", \"Last\", , \"first.last@example.com\", role = c(\"aut\", \"cre\"),"
#>  [6] "           comment = c(ORCID = \"YOUR-ORCID-ID\"))"                                       
#>  [7] "Description: What the package does (one paragraph)."                                      
#>  [8] "License: GPL (>= 3)"                                                                      
#>  [9] "Encoding: UTF-8"                                                                          
#> [10] "Roxygen: list(markdown = TRUE)"                                                           
#> [11] "RoxygenNote: 7.2.3"                                                                       
#> [12] "Imports: "                                                                                
#> [13] "    pkgA"                                                                                 
#> [14] "Remotes:"                                                                                 
#> [15] "    muschellij2/pkgA"

Pulling package deps succeeds because GITHUB_PAT set

r = remotes::dev_package_deps(pkgdir = pkgdir)

But dev_package_deps fails after we unset GITHUB_PAT

Sys.unsetenv("GITHUB_PAT")
remotes::dev_package_deps(pkgdir = pkgdir)
#> Error: HTTP error 404.
#>   Not Found
#> 
#>   Did you spell the repo owner (`muschellij2`) and repo name (`pkgA`) correctly?
#>   - If spelling is correct, check that you have the required permissions to access the repo.

Digging into dev_package_deps

Code extracted from dev_package_deps:

# setting up vars like defaults in `dev_package_deps`
repos = getOption("repos")
dependencies = NA
type = getOption("pkgType")

pkg <- remotes:::load_pkg_description(pkgdir)
repos <- c(repos, remotes:::parse_additional_repositories(pkg))

deps <- remotes:::local_package_deps(pkgdir = pkgdir, dependencies = dependencies)
deps
#> [1] "pkgA"

In dev_package_deps, the code fails at package_deps (

remotes/R/deps.R

Line 143 in e199c1b

cran_deps <- package_deps(deps, repos = repos, type = type)

)

cran_deps <- remotes:::package_deps(deps, repos = repos, type = type)
#> Error: HTTP error 404.
#>   Not Found
#> 
#>   Did you spell the repo owner (`muschellij2`) and repo name (`pkgA`) correctly?
#>   - If spelling is correct, check that you have the required permissions to access the repo.

Digging into remotes:::package_deps

The variables change names a bit, but we pass deps into the packages argument for remotes:::package_deps

packages = deps

repos <- remotes:::fix_repositories(repos)
cran <- remotes:::available_packages(repos, type)

deps <- remotes:::find_deps(packages, available = cran, top_dep = dependencies)
deps
#> [1] "pkgA"

Here, we see no auth_token is passed as package2remote uses github_pat:

remotes/R/install-remote.R

Line 241 in e199c1b

auth_token = github_pat()),

remote <- structure(lapply(deps, remotes:::package2remote, repos = repos, type = type), class = "remotes")

Without GITHUB_PAT set, we do not get the auth needed:

remote[[1]]$auth_token
#> NULL

And setting GITHUB_PAT, it gets correctly passed to remote:

Sys.setenv(GITHUB_PAT = auth_token)
remote <- structure(lapply(deps, remotes:::package2remote, repos = repos, type = type), class = "remotes")
nchar(remote[[1]]$auth_token)
#> [1] 40

^{Created on 2023-06-02 with reprex v2.0.2}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value
#>  version  R version 4.2.2 (2022-10-31)
#>  os       Ubuntu 22.04.1 LTS
#>  system   x86_64, linux-gnu
#>  ui       X11
#>  language (EN)
#>  collate  en_US.UTF-8
#>  ctype    en_US.UTF-8
#>  tz       Etc/UTC
#>  date     2023-06-02
#>  pandoc   2.19.2 @ /usr/lib/rstudio-server/bin/quarto/bin/tools/ (via rmarkdown)
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package     * version    date (UTC) lib source
#>  callr         3.7.3      2022-11-02 [2] RSPM (R 4.2.0)
#>  cli           3.6.1      2023-03-23 [1] RSPM (R 4.2.0)
#>  crayon        1.5.2      2022-09-29 [1] RSPM
#>  curl          5.0.0      2023-01-12 [1] RSPM (R 4.2.0)
#>  digest        0.6.31     2022-12-11 [1] RSPM (R 4.2.0)
#>  evaluate      0.19       2022-12-13 [1] RSPM (R 4.2.0)
#>  fastmap       1.1.1      2023-02-24 [1] RSPM
#>  fs            1.6.1      2023-02-06 [1] RSPM (R 4.2.0)
#>  gcloud        0.6.0      2023-04-27 [1] local
#>  git2r         0.30.1     2022-03-16 [1] RSPM (R 4.2.0)
#>  glue          1.6.2      2022-02-24 [1] RSPM
#>  highr         0.10       2022-12-22 [1] RSPM (R 4.2.0)
#>  htmltools     0.5.4      2022-12-07 [1] RSPM (R 4.2.0)
#>  jsonlite      1.8.4      2022-12-06 [1] RSPM (R 4.2.0)
#>  knitr         1.41       2022-11-18 [1] RSPM (R 4.2.0)
#>  lifecycle     1.0.3      2022-10-07 [1] RSPM
#>  magrittr      2.0.3      2022-03-30 [1] RSPM
#>  pkgbuild      1.3.1      2021-12-20 [2] RSPM (R 4.2.0)
#>  prettyunits   1.1.1      2020-01-24 [2] RSPM (R 4.2.0)
#>  processx      3.8.0      2022-10-26 [2] RSPM (R 4.2.0)
#>  ps            1.7.2      2022-10-26 [2] RSPM (R 4.2.0)
#>  R6            2.5.1      2021-08-19 [1] RSPM
#>  remotes       2.4.2.9000 2023-06-02 [1] local
#>  reprex        2.0.2      2022-08-17 [2] RSPM (R 4.2.0)
#>  rlang         1.1.0      2023-03-14 [1] RSPM
#>  rmarkdown     2.18       2022-11-09 [1] RSPM (R 4.2.0)
#>  rprojroot     2.0.3      2022-04-02 [2] RSPM (R 4.2.0)
#>  rstudioapi    0.14       2022-08-22 [2] RSPM (R 4.2.0)
#>  sessioninfo   1.2.2.9000 2022-11-19 [1] Github (r-lib/sessioninfo@0d74fe9)
#>  stringi       1.7.12     2023-01-11 [1] RSPM (R 4.2.0)
#>  stringr       1.5.0      2022-12-02 [1] RSPM (R 4.2.0)
#>  vctrs         0.6.2      2023-04-19 [1] RSPM (R 4.2.0)
#>  withr         2.5.0      2022-03-03 [1] RSPM
#>  xfun          0.36       2022-12-21 [1] RSPM (R 4.2.0)
#>  yaml          2.3.6      2022-10-18 [2] RSPM (R 4.2.0)
#> 
#>  [1] /home/gcer/R
#>  [2] /usr/local/lib/R/site-library
#>  [3] /usr/local/lib/R/library
#> 
#> ──────────────────────────────────────────────────────────────────────────────

[gaborcsardi]

Thanks for the detailed report! I suggest you try the pak package for this, it might work out of the box.