PyTorch Custom Source with Optional Dependencies Not Working with poetry lock and poetry remove

[sobhanAhmadian]

Description

When using Poetry with PyTorch's custom package sources and optional dependencies, and using sentence-transformers library as dependency, the poetry lock and poetry remove commands fail with "torch not found" errors, even though the package is properly installed and visible via pip list.

When using poetry install --extras "cuda" the dependencies will be installed and the lock file will be generated. However, later if we want to use poetry lock or poetry remove <some-package> we'll receive an error that "torch not found".

Current Behavior

Using the following configuration in pyproject.toml:

[project]
name = "test"
version = "0.1.0"
description = ""
authors = [
    { name = "Sobhan Ahmadian", email = "" },
]
requires-python = ">=3.12,<4.0"
dependencies = [
    "pandas~=2.2.3",
    "sentence-transformers~=3.3.1",
]

[project.optional-dependencies]
cpu = ["torch (==2.5.1+cpu)"]
cuda = ["torch (==2.5.1+cu124)"]

[tool.poetry]
requires-poetry = ">=2.0"

[tool.poetry.dependencies]
torch = [
    { markers = "extra == 'cpu' and extra != 'cuda'", source = "pytorch-cpu" },
    { markers = "extra == 'cuda' and extra != 'cpu'", source = "pytorch-cuda" },
]

[[tool.poetry.source]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
priority = "explicit"

[[tool.poetry.source]]
name = "pytorch-cuda"
url = "https://download.pytorch.org/whl/cu124"
priority = "explicit"

When running poetry install --extras "cuda" or poetry install --extras "cpu" every thing is OK, and the package is properly installed and visible via pip:

$ pip list | grep torch
torch                    2.5.1+cu124

However, when running poetry lock or poetry remove, I get the following error:

Package torch (2.5.1+cu124) not found.

Steps to Reproduce

1. Create a new Poetry project
2. Add the toml file I've provided above.
3. Run poetry install --extras "cuda".
4. Try to run poetry lock or poetry remove pandas
5. Observe the "Package not found" error

Workarounds

We should delete lock file every time and use poetry install --extras "cuda" agian

Poetry Installation Method

Poetry's official installer script

Operating System

Ubuntu 24.04

Poetry Version

2.0.1

Poetry Configuration

cache-dir = "/home/sobhan/.cache/pypoetry"
installer.max-workers = null
installer.no-binary = null
installer.only-binary = null
installer.parallel = true
installer.re-resolve = true
keyring.enabled = true
repositories.pytorch-cpu.url = "https://download.pytorch.org/whl/cpu"
repositories.pytorch-cuda.url = "https://download.pytorch.org/whl/cu124"
requests.max-retries = 0
solver.lazy-wheel = true
system-git-client = false
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/sobhan/.cache/pypoetry/virtualenvs
virtualenvs.prompt = "{project_name}-py{python_version}"
virtualenvs.use-poetry-python = false

Python Sysconfig

sysconfig.log

sysconfigoutput.txt

Example pyproject.toml

[project]
name = "test"
version = "0.1.0"
description = ""
authors = [
    { name = "Sobhan Ahmadian", email = "sobhan@cartesianlabs.io" },
]
requires-python = ">=3.12,<4.0"
dependencies = [
    "pandas~=2.2.3",
    "sentence-transformers~=3.3.1",
]

[project.optional-dependencies]
cpu = ["torch (==2.5.1+cpu)"]
cuda = ["torch (==2.5.1+cu124)"]

[tool.poetry]
requires-poetry = ">=2.0"

[tool.poetry.dependencies]
torch = [
    { markers = "extra == 'cpu' and extra != 'cuda'", source = "pytorch-cpu" },
    { markers = "extra == 'cuda' and extra != 'cpu'", source = "pytorch-cuda" },
]

[[tool.poetry.source]]
name = "pytorch-cpu"
url = "https://download.pytorch.org/whl/cpu"
priority = "explicit"

[[tool.poetry.source]]
name = "pytorch-cuda"
url = "https://download.pytorch.org/whl/cu124"
priority = "explicit"

Poetry Runtime Logs

poetry-runtime.log

poetrylock.log

[dimbleby]

Package torch (2.5.1+cu124) not found.

this does not appear in the log that you uploaded

[sobhanAhmadian]

this does not appear in the log that you uploaded

Sorry, I've updated my log file, and this is the error part:

line 2792:

Package torch (2.5.1+cu124) not found.

[dimbleby]

does not reproduce. The sequence poetry lock, poetry install --extras "cuda", poetry lock, poetry remove pandas produces no error with the pyproject.toml that you give.

Maybe put together a dockerfile if you think this can be reproduced

[sobhanAhmadian]

You're right. I thought the problem was with torch itself. Adding sentence-transformers to the dependencies should allow you to see the error:

dependencies = [
    "pandas~=2.2.3",
    "sentence-transformers~=3.3.1",  # Added
]

sentence-transformers depends on torch itself. I've edited the issue completely.
The description and the log file are new. I've tested it myself:

1. I've created an empty Python project with the name test.
2. It contains a `src/test/init.py' file and the toml file I've provided in the issue.
3. The sequence poetry lock, poetry install --extras "cuda", poetry lock raise the error.

new log file

Thank you for following up on the issue.

[dimbleby]

if you want to depend on torch from an explicit repository then you need to declare that as a dependency

edit: maybe the optional dependency should be enough. But probably it's something to do with the search first encountering torch as a transitive dependency, with no extras set - which torch do you even expect it to choose?

I expect that being explicit about what you want is probably the way to get going again anyway

[sobhanAhmadian]

We are collaborating on our project, and some team members need to install the CPU version while others require the CUDA version. Therefore, we need to specify these as extras.

[dimbleby]

most of the torch-is-difficult-to-install issues are a result of torch trying to be too clever with local versions, with multiple indexes, and suchlike. This workaround with extras is at best fragile!

Recommend that if you want a long term solution then you should go and petition torch, IMO they should probably be releasing separate packages torch-cpu and torch-cuda or similar, and they should completely abandon their attempted trickery with local versions.

See also #6409, if you have too much time on your hands