Rye workspace possibly not handling repos where package name != repo name (!= module name)?

[rachtsingh]

Steps to Reproduce

I might be doing something wrong, so bear with me. I have an organization with 5 packages, let's call them package_a to package_e. For unfortunate historical reasons, package_e has the following non-standard setup: the module name is package_e, but the package is actually called package_e-data, and the repo name is Package_e (i.e. it's at github.com/orgname/Package_e.git. These are all private.

I created a new directory, orgname-meta, and created a pyproject.toml with the following contents in it:

[project]
name = "orgname-meta"
version = "0.1.0"
description = "This is the virtual package that glues together the different installations"
authors = [
    { name = "Rachit Singh", email = "..." }
]
dependencies = []
readme = "README.md"
requires-python = ">= 3.10"

[tool.rye]
virtual = true
managed = true
dev-dependencies = [
    "deptry>=0.12.0",
]

[tool.rye.workspace]
members = ["package_a", "package_b", "package_c", "package_d", "Package_e"]

package_a has a dependency to package_e (the only one to do so) with the following line in project.dependencies:

    "package_e-data @ git+ssh://git@github.com/orgname/Package_e.git",

I cloned each of those packages into the orgname-meta folder, and ran rye sync.

Expected Result

rye sync should complete, and the dependency in package_a should be met via the completed local editable install of package_e.

Actual Result

Generating production lockfile: /Users/rachit/proj/orgname-meta/requirements.lock
   Built file:///Users/rachit/proj/orgname/Package_e
   Built file:///Users/rachit/proj/orgname/package_b
   Built file:///Users/rachit/proj/orgname/package_a
   Built file:///Users/rachit/proj/orgname/package_d
   Built file:///Users/rachit/proj/orgname/package_c
Built 5 editables in 37.05s
Updating ssh://git (github.com/orgname/Package_e.git)
error: Failed to download and build: package_e-data @ git+ssh://git@github.com/orgname/Package_e.git
  Caused by: Git operation failed
  Caused by: failed to fetch into: /Users/rachit/Library/Caches/uv/git-v0/db/f81ed0a494f1ba1c
  Caused by: failed to connect to the repository
  Caused by: failed to resolve address for git: nodename nor servname provided, or not known; class=Net (12)
error: could not write production lockfile for workspace

Caused by:
    failed to generate lockfile

Do we know why rye is trying to download Package_e.git to satisfy the requirement for package_a (I think that's what it's doing?). I would have kind of expected it to use the existing Package_e local install, but maybe it's getting confused?

Version Info

rye 0.24.0
commit: 0.24.0 (8d56aa18a 2024-02-15)
platform: macos (x86_64)
self-python: cpython@3.12
symlink support: true

Stacktrace

Stack backtrace:
   0: std::backtrace::Backtrace::create
   1: std::backtrace::Backtrace::capture
   2: anyhow::error::<impl anyhow::Error>::msg
   3: rye::lock::generate_lockfile
   4: rye::lock::update_workspace_lockfile
   5: rye::sync::sync
   6: rye::cli::sync::execute
   7: rye::cli::execute
   8: rye::main
   9: std::sys_common::backtrace::__rust_begin_short_backtrace
  10: std::rt::lang_start::{{closure}}
  11: std::rt::lang_start_internal
  12: _main
  13: <unknown>

[mitsuhiko]

Rye current does not do anything to map url references to local workspaces. You currently cannot have a git based dependency thus be shadowed by one in the workspace. This is something I will need to address. Question is if it should always shadow?

[rachtsingh]

Assuming I'm understanding the case correctly, I very much like the current shadowing default (i.e. "When a package depends on another package it's first located in the workspace locally before it's attempted to be downloaded from an index"). It makes sense to me - you check out a package locally if you mean to make local modifications; it means that every package lives in the same virtualenv, which to me is the biggest advantage of the workspace feature (very appreciated by the way).

If it didn't shadow, wouldn't that imply that two versions of the same package would have to be installed in the workspace virtualenv? I'm not sure how that would work, actually.

I think separately I wasn't able to figure out why the Git invocation failed. This StackOverflow seems to indicate it's an issue with my project.dependencies but I couldn't figure out a solution that would make it get past that line.

[mitsuhiko]

In general project.dependencies is intended for publishing to pypi which obviously does not support git dependencies. To the best of my knowledge there is no agreed upon system defined in standards today that would permit git dependencies in that section. However rye probably won't prevent these declarations today so you probably accidentally can install these (at least partially).

[rachtsingh]

Ah, I think I was assuming Poetry's specification is a standard. This spec seems to indicate that it's part of the spec to have a URL-based lookup (though the example doesn't mention SSH as a protocol, only HTTPS).

So for anyone who runs into this in the future, before it's resolved, changing the above line in project.dependencies to:

    "package_e-data @ git+https://github.com/orgname/Package_e.git#package_e-data",

(and installing Git Credentials Manager, which let me use HTTPS-based clones) seems to have fixed it.

and actually... it seems to have worked exactly as I expected (i.e. shadowed by local install)?

I see the following now:

Reusing already existing virtualenv
Generating production lockfile: /Users/rachit/proj/orgname-meta/requirements.lock
   Built file:///Users/rachit/proj/orgname-meta/Package_e
   Built file:///Users/rachit/proj/orgname-meta/package_d
   Built file:///Users/rachit/proj/orgname-meta/package_c
   Built file:///Users/rachit/proj/orgname-meta/package_b
   Built file:///Users/rachit/proj/orgname-meta/package_a
Built 5 editables in 16.26s
 Updated https://github.com/orgname/package_b.git (11ebfbc)
 Updated https://github.com/orgname/Package_e.git (8b5e180)
Resolved 156 packages in 18.81s
Generating dev lockfile: /Users/rachit/proj/orgname-meta/requirements-dev.lock
   Built file:///Users/rachit/proj/orgname-meta/Package_e
   Built file:///Users/rachit/proj/orgname-meta/package_d
   Built file:///Users/rachit/proj/orgname-meta/package_c
   Built file:///Users/rachit/proj/orgname-meta/package_b
   Built file:///Users/rachit/proj/orgname-meta/package_a
Built 5 editables in 15.37s
Updating https://github.com/orgname/package_b.git (HEAD)
Updating https://github.com/orgname/Package_e.git (HEAD)
 Updated https://github.com/orgname/package_b.git (11ebfbc)
 Updated https://github.com/orgname/Package_e.git (8b5e180)
Installing dependencies
   Built file:///Users/rachit/proj/orgname-meta/Package_e
   Built file:///Users/rachit/proj/orgname-meta/package_d
   Built file:///Users/rachit/proj/orgname-meta/package_c
   Built file:///Users/rachit/proj/orgname-meta/package_b
   Built file:///Users/rachit/proj/orgname-meta/package_a
Built 5 editables in 15.46s
Resolved 109 packages in 72ms
...
Downloaded 109 packages in 3m 31s
Installed 159 packages in 1.44s
 + aiobotocore==2.4.2
 + aiohttp==3.9.3
 + aioitertools==0.11.0
 + aiosignal==1.3.1
 + alembic==1.13.1
 + package_a==0.1.0 (from file:///Users/rachit/proj/orgname-meta/package_a)
 + package_b==0.1.0 (from file:///Users/rachit/proj/orgname-meta/package_b)
 + appnope==0.1.4
 + asttokens==2.4.1
 + attrs==23.2.0
 + banal==1.0.6
 + package_e-data==xxx (from file:///Users/rachit/proj/orgname-meta/Package_e)
...
Done!

And at least at first glance, it seems to work correctly (i.e. the modules are being imported from the local install, changes those files are immediately showing up in my virtualenv). I'm not sure it should have built those packages 3 times, but in this case it wasn't a big deal. It does seem to redo the build every time I add a new --dev dependency, which could hopefully be avoided.

[rachtsingh]

Maybe to make it more tractable to close this issue, maybe it's worth splitting up into smaller questions?

1. Should Rye workspaces support HTTPS-based project references in project.dependencies of pyproject.toml? uv seems to support this behavior, which is probably why it worked for me.
2. Should the local installation of a package in a Rye workspace always shadow any required installs by other packages (e.g. if flask is installed locally, and there's a package that depends on Flask, should it use the locally installed version)?
3. Is that currently what's happening above, or something different (I can probably figure this out)
4. Are all the "built" logs above separate builds, and do we need that? (I might be able to figure this out)

I understand this isn't the easiest decision to make because I know there's a lurking question about multi-version support (which is I guess how Cargo solves the related issue). From what I follow of Alex Crichton's response here, it sounds like Cargo chooses not to shadow; I think this is probably because every Cargo build is essentially "isolated".

For my two cents, I think that behavior would be pretty confusing for anyone who is trying to get started with debugging / editing a library. Imagine writing your app to use Flask as a dependency, cloning Flask down to hack with something in the library to see how it works, and then when you run it, you don't see a change? I think it would be a lot clearer if it shadowed.