Editable Dependency Installation?

[RobertRosca]

The scenario I'd like to deal with is having multiple local dependencies installed in editable mode, for example a monorepo/project with a namespace layout might have a directory structure like:

.
├── namespace
│  ├── proj1
│  │  ├── pyproject.toml
│  │  └── ...
│  ├── proj2
│  │  ├── pyproject.toml
│  │  └── ...
│  ├── proj3
│  │  ├── pyproject.toml
│  │  └── ...
├── pyproject.toml
└── ...

With a corresponding pyproject.toml file like:

[project]
dependencies = [
  "proj1 @ {root:uri}/namespace/proj1",
  "proj2 @ {root:uri}/namespace/proj2",
]

[tool.hatch.envs.with3]
extra-dependencies = [
  "proj3 @ {root:uri}/namespace/proj3",
]

As fas as I could tell from the docs/testing it doesn't seem possible to mark these dependencies to be installed in editable mode, but maybe I'm missing something? If it isn't possible then some approaches to this:

1. Extend dev-mode-dirs to work with environment installation as well as builds.
2. Add another dependency category, something like editable-dependencies, which would always perform a pip install -e ... on the provided dependency.
3. Additional syntax for the dependency specification, maybe -e proj @ URI, which when found installs the dependency in editable mode.

Option 2 would be the clearest, but would clutter up the tables with more fields (editable-dependencies, and probably extra-editable-dependencies); option 3 would avoid the clutter but would no longer be following PEP 508 and associated PEPs.

Right now this is possible by using pre/post install scripts which just run pip install -e ./path, but this isn't ideal as it bypasses dependency resolution and can be a bit confusing if you don't know the purpose.

For reference, here are related PEPs:

• PEP 440 – Version Identification and Dependency Specification
• PEP 508 – Dependency specification for Python Software Packages
• PEP 517 – A build-system independent format for source trees
• PEP 621 – Storing project metadata in pyproject.toml
• PEP 631 – Dependency specification in pyproject.toml based on PEP 508
• PEP 660 – Editable installs for pyproject.toml based builds (wheel based)

[RobertRosca]

Doing this as an environment plugin would also be alright, but I think this kind of functionality is useful to have in general for (albeit rare) cases of monorepo packages/developing packages which are closely linked together.

For more context, I'm interested in this because of work on https://github.com/jupyter-server/jupyverse which uses a plugin architecture, where the main jupyverse project has multiple plugins that are all published independently, so they are all in a plugins/ namespace.

During plugin development it's essential to have all plugins installed in editable mode, which is why having the ability to specify all the plugins as editable installs would be useful.

[sinus-x]

I'd like to bump this.

I have two repositories: a library and an application that is using it. The application is specifying the library in project.dependencies, but I'd need it to load local ../library/ editable version instead when developing the application and library at the same time. So Robert's issue describes my problem. Here, Pip would be required to use the local version instead of the one specified in the dependencies section, because it would (probably) complain with ResolutionImpossible.

[RobertRosca]

I have an example implementation here: #589

If I understood your usecase correctly, then with my PR you could do something like:

[project]
dependencies = [
  "library",
]

[tool.hatch.envs.dev]
dependencies = [
  "-e library @ {root:uri}/../library",
]

The entry in [project.dependencies] is there for builds or for normal installations, and it would pull the library from PyPI; the entry in envs.dev.extra-dependencies would install the library from a local path in editable mode for the dev environment.

Here is the behaviour with the above entries:

$ hatch run pip list
Package    Version Editable project location
---------- ------- ----------------------------------------------------------
library    0.0.1
package    0.0.1   /home/roscar/work/github.com/RobertRosca/hatch/tmp/package


$ hatch -e dev run pip list
Package    Version Editable project location
---------- ------- ----------------------------------------------------------
library    0.0.1   /home/roscar/work/github.com/RobertRosca/hatch/tmp/library
package    0.0.1   /home/roscar/work/github.com/RobertRosca/hatch/tmp/package

In the default environment library is a standard package, whereas in dev it is installed from a path in editable mode.

Is this roughly what you had in mind?

[OldGrumpyViking]

Maybe this works for your problem?
pyproject.toml

...

dependencies = [
    "<my-package-name>",
]

...

[tool.hatch.envs.codev]
pre-install-commands = [  # Changes to this line requires re-creation of the environment "hatch env remove codev".
  "pip install -e ../<my-package-name>"  # Used for developing on two projects in editable mode simultaniously
]

Then use the environment using hatch run codev:<CMD>

[RobertRosca]

Yes that does work, it's what is currently done for jupyverse (see https://github.com/jupyter-server/jupyverse/blob/9e5009ff86e6724a0dfa72323da9c1358e891a92/pyproject.toml#L46), however since this calls multiple pip install commands in a row separate to the original dependency list each install command does its own dependency resolution check(s) so it runs much slower than having everything in one list of dependencies.

Also, hach env show and hatch dep both lose track of the dependencies.

[sinus-x]

Is this roughly what you had in mind?

Yes, that is exactly like what I'd need. I wanted to test the PR locally, but whole pip/pipx decided to break down on me (it's a really old system which started by me not knowing how to properly do virtual environments, I don't blame it). But from what you have outlined it is what I was trying to solve.