Trying to lock the Cuda version of PyTorch sends pipenv into what seems to be an infinite loop

[mrakgr]

Issue description

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[[source]]
url = "https://download.pytorch.org/whl/cu117/"
verify_ssl = true
name = "downloadpytorch"

[packages]
torch = {version = "==2.0.1", index = "downloadpytorch"}

[requires]
python_version = "3.11"

I can't get the above Pipfile to finish running when executing:

PS E:\Webdev\ML-take-home\ml-trainer> pipenv lock --clear
>>
Locking [packages] dependencies...
Building requirements...
Resolving dependencies...
[  ==] Locking...

It never goes past the locking stage no matter how long I wait. Deleting the cache manually made it work a bit better as it no longer seems to cause my entire PC to lag, but I can't get it to get to the part where it creates the lock file.

---

Please run $ pipenv --support, and paste the results here. Don't put backticks (`) around it! The output already contains Markdown formatting.

PS E:\Webdev\ML-take-home\ml-trainer> pipenv --support | clip
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Marko\AppData\Roaming\Python\Python311\Scripts\pipenv.exe\__main__.py", line 7, in <module>
  File "C:\Users\Marko\AppData\Roaming\Python\Python311\site-packages\pipenv\vendor\click\core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marko\AppData\Roaming\Python\Python311\site-packages\pipenv\cli\options.py", line 58, in main
    return super().main(*args, **kwargs, windows_expand_args=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marko\AppData\Roaming\Python\Python311\site-packages\pipenv\vendor\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\Marko\AppData\Roaming\Python\Python311\site-packages\pipenv\vendor\click\core.py", line 1635, in invoke
    rv = super().invoke(ctx)
         ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marko\AppData\Roaming\Python\Python311\site-packages\pipenv\vendor\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marko\AppData\Roaming\Python\Python311\site-packages\pipenv\vendor\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marko\AppData\Roaming\Python\Python311\site-packages\pipenv\vendor\click\decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marko\AppData\Roaming\Python\Python311\site-packages\pipenv\vendor\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marko\AppData\Roaming\Python\Python311\site-packages\pipenv\vendor\click\decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Marko\AppData\Roaming\Python\Python311\site-packages\pipenv\cli\command.py", line 142, in cli
    get_pipenv_diagnostics(state.project)
  File "C:\Users\Marko\AppData\Roaming\Python\Python311\site-packages\pipenv\help.py", line 49, in get_pipenv_diagnostics
    print("Pipenv–specific environment variables:")
UnicodeEncodeError: 'cp932' codec can't encode character '\u2013' in position 6: illegal multibyte sequence

Anyway, I am running 2023.6.18 version of pipenv.

[matteius]

What is your connection like? Locking pytorch causes several gigabytes of downloads because they haven't included package hashes in the generated index. I actually have a PR out with the pytorch builder that would add this and that would make locking much faster. It has to download the wheels for each system that match the version you are locking and that can be like 16+ GB.

[mrakgr]

Not fast, 40 megabits per second. I've looked at PMs other than pipenv and it seems PyTorch is a pain point for all of them. Because none of the PMs report progress during locking when resolving PyTorch, I've come to the conclusion that it might be better to just leave that package out and install it separately using regular pip after starting the virtual env.

[matteius]

Yeah, if pytorch/builder#1433 ever gets merged then it should be a much better landscape.

[matteius]

This is a lot better on 2023.8.25 because I replace the requests Session with the PipSession sub-class and got the file caching to work, so once you lock once and download all the matching wheels, subsequent locks re-use those already downloaded files in the cache directory to generate the hashes. This is a big improvement until that builder PR gets merged.