VCS install using https credentials in env variables fails for Pipfile #6195

dennisvang · 2024-07-09T14:07:53Z

Issue description

Trying to install a package over https, from a private AWS CodeCommit repo, on an Ubuntu system.

Instead of using git credential.helper, we would like to inject credentials into our Pipfile using env variables.

So, in a bash terminal, we define USERNAME and PASSWORD (url-encoded).

Installing into an empty dir, directly from the command line, using these credentials, appears to work without issue:

pipenv install -e "git+https://${USERNAME}:${PASSWORD}@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage@main#egg=mypackage"

This implies that the credentials and user permissions are correct.

However, it turns out the username and password end up in both Pipfile and Pipfile.lock...

That does not sound like a good idea.

Luckily, the docs for Injecting credentials into Pipfile via environment variables say:

Keep in mind that environment variables are expanded in runtime, leaving the entries in Pipfile or Pipfile.lock untouched. This is to avoid the accidental leakage of credentials in the source code.

However, if we try to install from a Pipfile into an otherwise empty dir, using the exact same url with same USERNAME and PASSWORD, we get a status 403 (Forbidden):

# ...
[packages]
mypackage = {editable = true, ref = "main", git = "git+https://${USERNAME}:${PASSWORD}@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage"}
# ...

yields

...
INFO:pip.subprocessor:fatal: unable to access 'https://git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage/': 
The requested URL returned error: 403
ERROR:pip.subprocessor:git clone --filter=blob:none --quiet 'https://myusername:****@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage' /tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21
exited with 128
...

Expected result

I expect installation from Pipfile to work without issue, just like installation from the command line.

Actual result

command line output

/tmp/pipenvtest $ pipenv install --verbose
Pipfile.lock not found, creating...
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
INFO:pipenv.patched.pip._internal.operations.prepare:Collecting mypackage@ 
git+https://myusername:****@git-codecommit.eu-west-3.a
mazonaws.com/v1/repos/mypackage@main (from -r 
/tmp/pipenv-2_t8zu99-requirements/pipenv-3cdgwj18-constraints.txt (line 2))
INFO:pipenv.patched.pip._internal.vcs.git:Cloning 
https://myusername:****@git-codecommit.eu-west-3.amazo
naws.com/v1/repos/mypackage (to revision main) to 
/tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21
INFO:pip.subprocessor:Running command git clone --filter=blob:none --quiet 
'https://myusername:****@git-codecommit.eu-west-3.amaz
onaws.com/v1/repos/mypackage' 
/tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21
INFO:pip.subprocessor:fatal: unable to access 
'https://git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage/': The 
requested URL returned error: 403
ERROR:pip.subprocessor:git clone --filter=blob:none --quiet 
'https://myusername:****@git-codecommit.eu-west-3.amaz
onaws.com/v1/repos/mypackage' 
/tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21 exited with 
128
Traceback (most recent call last):
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py",
line 455, in resolve
    results = resolver.resolve(constraints, check_supported_wheels=False)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/resolver.py", line 76, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/factory.py", line 534, in collect_root_requirements
    reqs = list(
           ^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/factory.py", line 490, in _make_requirements_from_install_req
    cand = self._make_base_candidate_from_link(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
    self._link_candidate_cache = LinkCandidate(
                                       ^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/candidates.py", line 294, in __init__
    super().__init__(
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/candidates.py", line 157, in __init__
    self.dist = self._prepare()
                ^^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/candidates.py", line 223, in _prepare
    dist = self._prepare_distribution()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/reso
lution/resolvelib/candidates.py", line 305, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/oper
ations/prepare.py", line 525, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/oper
ations/prepare.py", line 596, in _prepare_linked_requirement
    local_file = unpack_url(
                 ^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/oper
ations/prepare.py", line 157, in unpack_url
    unpack_vcs_link(link, location, verbosity=verbosity)
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/oper
ations/prepare.py", line 80, in unpack_vcs_link
    vcs_backend.unpack(location, url=hide_url(link.url), verbosity=verbosity)
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/vcs/
versioncontrol.py", line 608, in unpack
    self.obtain(location, url=url, verbosity=verbosity)
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/vcs/
versioncontrol.py", line 521, in obtain
    self.fetch_new(dest, url, rev_options, verbosity=verbosity)
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/vcs/
git.py", line 276, in fetch_new
    self.run_command(
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/vcs/
versioncontrol.py", line 650, in run_command
    return call_subprocess(
           ^^^^^^^^^^^^^^^^
  File 
"/home/me/.local/lib/python3.11/site-packages/pipenv/patched/pip/_internal/util
s/subprocess.py", line 224, in call_subprocess
    raise error
pipenv.patched.pip._internal.exceptions.InstallationSubprocessError: git clone 
--filter=blob:none --quiet 
'https://myusername:****@git-codecommit.eu-west-3.amaz
onaws.com/v1/repos/mypackage' 
/tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21 exited with 
128
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line 
675, in <module>
    main()
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line 
661, in main
    _main(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line 
645, in _main
    resolve_packages(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line 
612, in resolve_packages
    results, resolver = resolve(
                        ^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/resolver.py", line 
592, in resolve
    return resolve_deps(
           ^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py",
line 932, in resolve_deps
    results, hashes, internal_resolver = actually_resolve_deps(
                                         ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py",
line 700, in actually_resolve_deps
    resolver.resolve()
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py",
line 457, in resolve
    raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: [31m[1mERROR[0m: [33mgit clone 
--filter=blob:none --quiet 
'https://myusername:****@git-codecommit.eu-west-3.amaz
onaws.com/v1/repos/mypackage' 
/tmp/pip-temp-5vfi10zn/mypackage_c61911349e2c4296b14b1168cf062f21 exited with 
128[0m
✘ Locking Failed!
⠋ Locking packages...
Traceback (most recent call last):
  File "/home/me/.local/bin/pipenv", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/cli/command.py", line 209, in install
    do_install(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/routines/install.py", line 93, in do_install
    do_init(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/routines/install.py", line 653, in do_init
    do_lock(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/routines/lock.py", line 66, in do_lock
    venv_resolve_deps(
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 873, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/me/.local/lib/python3.11/site-packages/pipenv/utils/resolver.py", line 737, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

Steps to replicate

first

create a private aws codecommit repo with a minimal pyproject.toml, as follows, and setup a user with git credentials (perhaps a private github repo would also work, but haven't tried that yet)
```
[build-system]
requires = ["setuptools >= 61.0"]
build-backend = "setuptools.build_meta"

[project]
name = "mypackage"
version = "0.0.1"
```
in a bash terminal on the local system, create env variables USERNAME and PASSWORD (e.g. using read -s) with the corresponding values (url-encoded)

create an empty dir, locally, and use pipenv to install the repo, in editable mode (not sure if that matters), as in:

pipenv install -e "git+https://${USERNAME}:${PASSWORD}@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage@main#egg=mypackage"

observe that this installation succeeds

then

in the same terminal, create another empty dir

copy the Pipfile, generated above, into the new dir, and replace the actual username and password in the file by the corresponding env variables, as in:

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
mypackage = {editable = true, ref = "main", git = "git+https://${USERNAME}:${PASSWORD}@git-codecommit.eu-west-3.amazonaws.com/v1/repos/mypackage"}

[dev-packages]

[requires]
python_version = "3.11"

run pipenv install
observe the status 403 error

$ pipenv --support

Pipenv version: '2024.0.1'

Pipenv location: '/home/me/.local/lib/python3.11/site-packages/pipenv'

Python location: '/home/me/.pyenv/versions/3.11.6/bin/python3.11'

OS Name: 'posix'

User pip version: '24.0'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.11.6',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '6.5.0-41-generic',
 'platform_system': 'Linux',
 'platform_version': '#41~22.04.2-Ubuntu SMP PREEMPT_DYNAMIC Mon Jun  3 '
                     '11:32:55 UTC 2',
 'python_full_version': '3.11.6',
 'python_version': '3.11',
 'sys_platform': 'linux'}

System environment variables:

SHELL
SESSION_MANAGER
QT_ACCESSIBILITY
PIPENV_VENV_IN_PROJECT
COLORTERM
PYENV_SHELL
XDG_CONFIG_DIRS
SSH_AGENT_LAUNCHER
XDG_MENU_PREFIX
GNOME_DESKTOP_SESSION_ID
LANGUAGE
LC_ADDRESS
GNOME_SHELL_SESSION_MODE
LC_NAME
SSH_AUTH_SOCK
GIT_PS1_SHOWDIRTYSTATE
XMODIFIERS
DESKTOP_SESSION
LC_MONETARY
GTK_MODULES
PWD
LOGNAME
XDG_SESSION_DESKTOP
XDG_SESSION_TYPE
SYSTEMD_EXEC_PID
XAUTHORITY
HOME
USERNAME
IM_CONFIG_PHASE
LC_PAPER
LANG
LS_COLORS
XDG_CURRENT_DESKTOP
VTE_VERSION
WAYLAND_DISPLAY
GNOME_TERMINAL_SCREEN
GNOME_SETUP_DISPLAY
LESSCLOSE
XDG_SESSION_CLASS
TERM
LC_IDENTIFICATION
LESSOPEN
USER
GNOME_TERMINAL_SERVICE
DISPLAY
SHLVL
LC_TELEPHONE
QT_IM_MODULE
LC_MEASUREMENT
PAPERSIZE
XDG_RUNTIME_DIR
PYENV_ROOT
LC_TIME
XDG_DATA_DIRS
PATH
GDMSESSION
DBUS_SESSION_BUS_ADDRESS
LC_NUMERIC
_
PIP_DISABLE_PIP_VERSION_CHECK
PYTHONDONTWRITEBYTECODE
PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

PIPENV_VENV_IN_PROJECT: 1

Debug–specific environment variables:

PATH: /home/me/.pyenv/versions/3.9.18/bin:/home/me/.pyenv/versions/3.8.18/bin:/home/me/.pyenv/versions/3.11.6/bin:/home/me/.pyenv/versions/3.8.13/bin:/home/me/.pyenv/shims:/home/me/.npm-global/bin:/home/me/.local/bin:/home/me/.pyenv/bin:/home/me/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/snap/bin
SHELL: /bin/bash
LANG: en_CA.UTF-8
PWD: /home/me

The text was updated successfully, but these errors were encountered:

matteius · 2024-07-29T01:50:05Z

I am wondering if anyone can setup a sample private repo for testing this case -- I'd like to help improve the code around it.

dennisvang · 2024-07-29T10:06:47Z

@matteius I've tried to reproduce this using one of my private github repos, because those are easier to set up. However, in this case pipenv already fails at replication step 3. mentioned above (non-editable).

For example, pip succeeds (with github username and fine-grained personal access token as credentials.)

pip install "git+https://${USERNAME}:${PASSWORD}@github.com/myusername/my-private-repo.git"

but pipenv, with the same command, same terminal, same credentials

pipenv install "git+https://${USERNAME}:${PASSWORD}@github.com/myusername/my-private-repo.git"

fails with

...
Added my-private-repo to Pipfile's [packages] ...
✔ Installation Succeeded
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.
Installing dependencies from Pipfile.lock (13a959)...
[pipenv.exceptions.InstallError]: Collecting my-private-repo@ git+https://myusername:****@0888...b155 (from -r /tmp/pipenv-hhg4plkm-requirements/pipenv-vmlg4kcl-reqs.txt (line 1))
[pipenv.exceptions.InstallError]:   Cloning https://myusername:****@0888...b155 to /tmp/pip-install-_1xf_5i1/my-private-repo_3cef3c3357fd4617av89cf07dea69a8e
[pipenv.exceptions.InstallError]: Running command git clone --filter=blob:none --quiet 'https://myusername:****@0888...b155' /tmp/pip-install-_1xf_5i1/my-private-repo_3cef3c3357fd4617av89cf07dea69a8e
[pipenv.exceptions.InstallError]:   fatal: unable to access 'https://0888...b155/': Could not resolve host: 0888...b155
...
ERROR: Couldn't install package: {}
Package installation failed...

It looks unrelated to the present issue. Should I open a new issue for this?

matteius · 2024-09-14T03:03:50Z

@dennisvang Could you test this report against this branch with I believe does better with the VCS env variables: #6242

dennisvang changed the title ~~Editable install from VCS using https credentials from env variables fails in Pipfile~~ Editable install from VCS using https credentials in env variables fails for Pipfile Jul 9, 2024

dennisvang changed the title ~~Editable install from VCS using https credentials in env variables fails for Pipfile~~ VCS install using https credentials in env variables fails for Pipfile Jul 9, 2024

matteius self-assigned this Jul 29, 2024

matteius added Category: Security Relates to security triage Contributor Candidate The issue has been identified/triaged and contributions are welcomed/encouraged. labels Aug 25, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VCS install using https credentials in env variables fails for Pipfile #6195

VCS install using https credentials in env variables fails for Pipfile #6195

dennisvang commented Jul 9, 2024 •

edited

Loading

matteius commented Jul 29, 2024

dennisvang commented Jul 29, 2024 •

edited

Loading

matteius commented Sep 14, 2024

VCS install using https credentials in env variables fails for Pipfile #6195

VCS install using https credentials in env variables fails for Pipfile #6195

Comments

dennisvang commented Jul 9, 2024 • edited Loading

Issue description

Expected result

Actual result

Steps to replicate

matteius commented Jul 29, 2024

dennisvang commented Jul 29, 2024 • edited Loading

matteius commented Sep 14, 2024

dennisvang commented Jul 9, 2024 •

edited

Loading

dennisvang commented Jul 29, 2024 •

edited

Loading