Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipfile {git = "file:///path/to/git_repo.git"} resolution failed #6218

Closed
evan0greenup opened this issue Aug 2, 2024 · 9 comments · Fixed by #6242
Closed

Pipfile {git = "file:///path/to/git_repo.git"} resolution failed #6218

evan0greenup opened this issue Aug 2, 2024 · 9 comments · Fixed by #6242
Labels

Comments

@evan0greenup
Copy link

Issue description

When add a local git repository as dependency, it will cause resolution failure.

Here is the Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

typer = {git = "file:///home/myuser/PycharmProjects/typer.git"}

[dev-packages]

[requires]
python_version = "3.12"

Expected result

Just like how Cargo.toml do, it should clone the repository and install the dependency.

Actual result

Resolution failure.

Here is the output

Building requirements...
Resolving dependencies...
✘ Locking Failed!
⠋ Locking packages...False
INFO:pip.subprocessor:Running command git clone --filter=blob:none --quiet file:///home/myuser/PycharmProjects/typer.git /home/myuser/.local/share/virtualenvs/hello_pipenv-kJdo0SfZ/src/typer
INFO:pip.subprocessor:warning: filtering not recognized by server, ignoring
INFO:pip.subprocessor:Running command git clone --filter=blob:none file:///home/myuser/PycharmProjects/typer.git /tmp/tmpgpkr82se
INFO:pip.subprocessor:Cloning into '/tmp/tmpgpkr82se'...
INFO:pip.subprocessor:warning: filtering not recognized by server, ignoring
Traceback (most recent call last):
  File "/usr/lib/python3.12/site-packages/pipenv/resolver.py", line 675, in <module>
    main()
  File "/usr/lib/python3.12/site-packages/pipenv/resolver.py", line 661, in main
    _main(
  File "/usr/lib/python3.12/site-packages/pipenv/resolver.py", line 645, in _main
    resolve_packages(
  File "/usr/lib/python3.12/site-packages/pipenv/resolver.py", line 622, in resolve_packages
    results = clean_results(results, resolver, project, category)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/resolver.py", line 561, in clean_results
    entry_dict = translate_markers(entry.get_cleaned_dict)
                                   ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/resolver.py", line 533, in __getattribute__
    return super().__getattribute__(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/functools.py", line 995, in __get__
    val = self.func(instance)
          ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/resolver.py", line 240, in get_cleaned_dict
    self.validate_constraints()
  File "/usr/lib/python3.12/site-packages/pipenv/resolver.py", line 477, in validate_constraints
    pinned_version = self.updated_version
                     ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/resolver.py", line 533, in __getattribute__
    return super().__getattribute__(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/resolver.py", line 383, in updated_version
    version = str(self.entry.specifier)
                  ^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/resolver.py", line 533, in __getattribute__
    return super().__getattribute__(key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/resolver.py", line 281, in entry
    self._entry = self.make_requirement(self.name, self.entry_dict)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/resolver.py", line 140, in make_requirement
    return from_pipfile(name, entry)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/utils/dependencies.py", line 1111, in from_pipfile
    install_req, markers, req_str = install_req_from_pipfile(name, pipfile)
                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/utils/dependencies.py", line 1097, in install_req_from_pipfile
    install_req, _ = expansive_install_req_from_line(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/utils/dependencies.py", line 962, in expansive_install_req_from_line
    return install_req_from_editable(pip_line, line_source), name
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/patched/pip/_internal/req/constructors.py", line 240, in install_req_from_editable
    parts = parse_req_from_editable(editable_req)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/patched/pip/_internal/req/constructors.py", line 209, in parse_req_from_editable
    name, url, extras_override = parse_editable(editable_req)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/patched/pip/_internal/req/constructors.py", line 126, in parse_editable
    raise InstallationError(
pipenv.patched.pip._internal.exceptions.InstallationError: git+  is not a valid editable requirement. It should either be a path to a local project or a VCS URL (beginning with bzr+http, bzr+https, bzr+ssh, bzr+sftp, bzr+ftp, bzr+lp, bzr+file, git+http, git+https, git+ssh, git+git, git+file, hg+file, hg+http, hg+https, hg+ssh, hg+static-http, svn+ssh, svn+http, svn+https, svn+svn, svn+file).

Traceback (most recent call last):
  File "/usr/bin/pipenv", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/cli/options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/vendor/click/decorators.py", line 92, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/vendor/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/cli/command.py", line 209, in install
    do_install(
  File "/usr/lib/python3.12/site-packages/pipenv/routines/install.py", line 93, in do_install
    do_init(
  File "/usr/lib/python3.12/site-packages/pipenv/routines/install.py", line 624, in do_init
    do_update(
  File "/usr/lib/python3.12/site-packages/pipenv/routines/update.py", line 61, in do_update
    do_lock(
  File "/usr/lib/python3.12/site-packages/pipenv/routines/lock.py", line 66, in do_lock
    venv_resolve_deps(
  File "/usr/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 873, in venv_resolve_deps
    c = resolve(cmd, st, project=project)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/site-packages/pipenv/utils/resolver.py", line 737, in resolve
    raise RuntimeError("Failed to lock Pipfile.lock!")
RuntimeError: Failed to lock Pipfile.lock!

Steps to replicate

Provide the steps to replicate (which usually at least includes the commands and the Pipfile).


Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

If you're on macOS, run the following:

$ pipenv --support | pbcopy

If you're on Windows, run the following:

> pipenv --support | clip

If you're on Linux, run the following:

$ pipenv --support | xclip
$ pipenv --support

Pipenv version: '2024.0.1'

Pipenv location: '/usr/lib/python3.12/site-packages/pipenv'

Python location: '/usr/bin/python'

OS Name: 'posix'

User pip version: '24.0'

user Python installations found:

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.12.4',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '6.10.2-arch1-1',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT_DYNAMIC Sat, 27 Jul 2024 16:49:55 +0000',
 'python_full_version': '3.12.4',
 'python_version': '3.12',
 'sys_platform': 'linux'}

System environment variables:

  • SHELL
  • SESSION_MANAGER
  • COLORTERM
  • CSF_MDTVTexturesDirectory
  • XDG_MENU_PREFIX
  • TERM_PROGRAM_VERSION
  • CSF_DrawPluginDefaults
  • MATHEMATICA_HOME
  • CSF_LANGUAGE
  • SSH_AUTH_SOCK
  • CSF_MIGRATION_TYPES
  • MEMORY_PRESSURE_WRITE
  • DESKTOP_SESSION
  • CSF_OCCTResourcePath
  • CSF_STEPDefaults
  • PWD
  • XDG_SESSION_DESKTOP
  • LOGNAME
  • XDG_SESSION_TYPE
  • DRAWHOME
  • SYSTEMD_EXEC_PID
  • XAUTHORITY
  • CSF_StandardLiteDefaults
  • GJS_DEBUG_TOPICS
  • WINDOWPATH
  • MOTD_SHOWN
  • GDM_LANG
  • HOME
  • USERNAME
  • LANG
  • XDG_CURRENT_DESKTOP
  • MEMORY_PRESSURE_WATCH
  • VTE_VERSION
  • INVOCATION_ID
  • MANAGERPID
  • CSF_ShadersDirectory
  • CSF_EXCEPTION_PROMPT
  • CSF_XmlOcafResource
  • GJS_DEBUG_OUTPUT
  • CSF_SHMessage
  • XDG_SESSION_CLASS
  • TERM
  • USER
  • CUDA_PATH
  • CSF_StandardDefaults
  • CSF_IGESDefaults
  • DISPLAY
  • CSF_XCAFDefaults
  • SHLVL
  • CSF_PluginDefaults
  • CSF_TObjMessage
  • CASROOT
  • XDG_RUNTIME_DIR
  • MKLROOT
  • NVCC_CCBIN
  • DEBUGINFOD_URLS
  • JOURNAL_STREAM
  • CSF_XSMessage
  • MMGT_CLEAR
  • PATH
  • CSF_TObjDefaults
  • GDMSESSION
  • DBUS_SESSION_BUS_ADDRESS
  • HG
  • MAIL
  • DRAWDEFAULT
  • GIO_LAUNCHED_DESKTOP_FILE_PID
  • GIO_LAUNCHED_DESKTOP_FILE
  • TERM_PROGRAM
  • _
  • PIP_DISABLE_PIP_VERSION_CHECK
  • PYTHONDONTWRITEBYTECODE
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: /home/myuser/.pyenv/versions/3.11.9/bin:/home/myuser/.pyenv/versions/3.11.8/bin:~/.local/share/JetBrains/Toolbox/scripts/:/usr/local/bin:/usr/bin:/usr/local/sbin:/opt/cuda/bin:/opt/cuda/nsight_compute:/opt/cuda/nsight_systems/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/usr/lib/rustup/bin
  • SHELL: /bin/bash
  • LANG: en_US.UTF-8
  • PWD: /home/myuser/PycharmProjects/hello_pipenv

Contents of Pipfile ('/home/myuser/PycharmProjects/hello_pipenv/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]

typer = {git = "file:///home/myuser/PycharmProjects/typer.git"}

[dev-packages]

[requires]
python_version = "3.12"

Contents of Pipfile.lock ('/home/myuser/PycharmProjects/hello_pipenv/Pipfile.lock'):

{
    "_meta": {
        "hash": {
            "sha256": "702ad05de9bc9de99a4807c8dde1686f31e0041d7b5f6f6b74861195a52110f5"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.12"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {},
    "develop": {}
}
@aidencullo
Copy link
Contributor

Steps to replicate are missing. Also, the pipenv --support instructions are not necessary in the post

@evan0greenup
Copy link
Author

@aidencullo , just copy the Pipfile above and run pipenv install

@matteius
Copy link
Member

@evan0greenup we don't have that local .git -- Its not clear to me how to test a local .git repository, wouldn't it normally be a local file install?

@dennisvang
Copy link

dennisvang commented Sep 11, 2024

@matteius I've been able to reproduce this, with an arbitrary local git repo (system specs at bottom).

workaround

Here's a possible workaround for the OP (at least it works on my system):

Before running pipenv, set an env variable, using absolute path without leading slash:

export REPO_PATH="home/myuser/PycharmProjects/typer.git"

In the pipfile, use this env var and file:/// with triple slash:

typer = {git = "file:///${REPO_PATH}"}

In addition, although I'm not sure if this is always required, I also had to remove the existing Pipfile.lock before running pipenv install or pipenv update.

More details below.

details

Using only pip 24.2, the following works without issue:

pip install git+file:///home/me/my-package

Note git+file:// is a form of VCS installation, allowing us to install e.g. a specific version from a local repo.

Afaik, "translating" this to Pipfile format yields:

...
mypackage = {git = "file:///home/me/my-package"}
...

This is what the OP has, and I run into the same errors as the OP.

Trying to understand what's going on, I switched to using an env variable for the path:

export ABSOLUTE_PATH_TO_MY_PACKAGE="/home/me/my-package"

and in the Pipfle:

mypackage = {git = "file://${ABSOLUTE_PATH_TO_MY_PACKAGE}"}

For some reason, this does work, and installation succeeds.

However, if I add a ref, as in:

mypackage = {git = "file://${ABSOLUTE_PATH_TO_MY_PACKAGE}",  ref = "2024.1"}

again, I see the same error as the OP (or very similar).

On a hunch, I tried adding a slash to the Pipfile, as in file:///, and removing the leading slash from the env variable value, as in home/me/my-package.

That works, both with the ref, and without.

If we use triple slash in the pipfile (file:///) and leading slash in the env var (/home/me/my-package), installation fails with:

...
INFO:pip.subprocessor:Running command git clone --filter=blob:none --quiet file://home/me/my-package /tmp/pipfile-envvar/.venv/src/fsproc
INFO:pip.subprocessor:fatal: '/me/my-package' does not appear to be a git repository
...

Showing that the /home part is being stripped.

Stepping back again to "pure" pip:

Using a requirements.txt file with git+file:// (double slash) and ref:

git+file://${ABSOLUTE_PATH_TO_MY_PACKAGE}@2024.1

and env variable with /home/me/my-package, i.e. with proper leading slash, installation succeeds (using pip install -r requirements.txt).

Side note;

On Windows, if we use the same requirements.txt, and specify a path with windows separators (\) , e.g. $env:ABSOLUTE_PATH_TO_MY_PACKAGE = 'c:\users\me\my-package' in powershell 5.1, then pip fails again:

...
collecting git+file://C:****@2024.1 (from -r requirements.txt (line1))
... does not appear to be a git repository
...

In this case pip install only succeeds if the expanded value looks like the following, with triple slash:

git+file:///c:\users\me\my-package@2024.1

system:

  • ubuntu 22.04.4
  • python 3.11.6
  • pipenv 2024.0.1
  • pip 24.2

@evan0greenup
Copy link
Author

evan0greenup commented Sep 18, 2024

In my point of view, the feature for checkout and install local file:/// package dependencies should not depend on pip, it should implement its own capability for checkout the local bare repository into an temporary directory and pack it into a wheel and install it.

And the scheme should not be git+file, this is too redundant, for project dependency manager for other programming languages, they all support {git = "file:///path/to/repo.git"} if they use TOML as project specification.

And git in git+file is unnecessary because there is already a git in {git = ... }.

@dennisvang
Copy link

dennisvang commented Sep 18, 2024

@evan0greenup There are different types of local installation, both with valid use-cases:

  • "direct" install from local path (pip, pipenv):
    mypackage = {path = "/path/to/mypackage"}
    
    Even if the path represents a git repo, the above will simply install the current state of the workspace.
  • VCS install from local git repository (pip, pipenv):
    mypackage = {git = "file:///path/to/mypackage", ref = "2024.37"}
    
    This allows you to install a specific reference state from the git repository, e.g. a specific tag or branch.

And the scheme should not be git+file, this is too redundant, for project dependency manager for other programming languages, they all support {git = "file:///path/to/repo.git"} if they use TOML as project specification.

Afaik, pipenv supports both {git = "file://...}" and {git = "git+file://...}".

(@matteius please correct me if I'm wrong here)

@matteius
Copy link
Member

I am not sure if #6242 will change the behavior here, since git: git+file is overly verbose I believe {git = "file://...}"is preferred over {git = "git+file://...}

@matteius
Copy link
Member

I just checked and #6242 does not break the existing assumptions -- but I think the confusion with this ticket is when installing a vcs repository from a local file path, it typically does not end with .git because its just a directory path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants