Allow using maturin develop outside of virtualenv

[programmerjake]

This seems like something that would be useful to allow, perhaps requiring a --force-develop flag of some sort if retaining the current error message seems useful to direct beginners to using virtualenv.

[konstin]

When would this be required? I generally advise against installing anything but standalone cli tools into the global environment, and even that breaks easily. I wouldn't want to encourage using the user/system site packages with maturin, especially since it's still simple enough to use maturin build -i python && pip install <...>.

[programmerjake]

I submitted this bug report for my coworker who doesn't have a github account. Here's his response:

i did manage to use "pip3 install something.whl" however it was the
complete lack of documentation that that was even possible which
tripped me up. this gave the false impression that the only thing
possible was to use virtualenv.

So maybe adding documentation to the error message on how to install using maturin build && pip install ... is what's needed.

[g-braeunlich]

setuptools also has

./setup.py build_ext --inplace

This would build the .so and move it directly into the python package folder.
I personally would also like this.

[konstin]

I've updated the message to make the options clearer. I'm however not interested in supporting anything that could break the user's global environment.

[ntakouris]

I'm using docker for development so I don't really care about using venv or something else. We still need a --skip-venv-check option.

For now, I am doing pip install . but I am not sure if it does a release configuration.

[konstin]

Is your dev workflow to maturin develop into the global environment inside the docker container? If so, is there a reason not to have a virtualenv inside the container? It would be cached build step so it wouldn't incur any overhead

For now, I am doing pip install . but I am not sure if it does a release configuration.

It actually always does a release build, PEP 517 has no way to communicate whether you want dev or prod (https://discuss.python.org/t/pep-517-debug-vs-release-builds/1924)

[ntakouris]

There is no reason to have a virtual env with just one env. It's the dev docker container and contains everything needed for that project. You can revert to use virutalenv inside docker but

1. It's unnecessary; isolation of dev env is the whole point of using container for dev
2. I will have to migrate every other shell/ci/etl script to use that

I wouldn't force people to use virtualenv if features do not directly depend on venv specific internals. It's just a python env. Maybe remove the venv guard and print a warning without a --i-understand flag being present.

[thehappycheese]

I am trying to run maturin on repl.it but I can't do it because repl's package management system is preventing me from installing venv :(

All I want is for maturin to do its thing and put the dev binary in the python_src folder

See broken repl here: https://replit.com/@thehappycheese/Megamerge#pyproject.toml ... hopefully this link takes you to the explorer/editor view and not the stupid preview screen that repl.it sometimes shows when you are not logged in.

[konstin]

you can still python -m venv .venv and then activate that

[thehappycheese]

you can still python -m venv .venv and then activate that

Not true. Try it. repl.it prevents the installation of the venv package, and it is not in the preinstalled libraries as you might expect, so that is impossible.

To be fair, its more of a problem with repl.it than with maturin. I understand why maturin wants to be inside a venv. But still maturin seems at least 10% at fault for being so inflexible.

[udalrich]

I would like this feature for use within a CI job. We have a docker image that is used for python CI builds. It is based on something fairly old and was providing pip 18.x. So I updated the Dockerfile to update pip to the current version, which worked fine.

I then started trying to use it for a mixed rust/python project. Since I am in a single use container, there is no problem with installing into the global environment. So I tried

maturin build
python -m pip install target/wheels/my-project.whl[test]
python -m pytest my_project

which is apparently not the correct way to install the maturin wheel locally. After several more tries, I gave up on maturin build and tried maturin develop, since that does the right thing (tm). That failed, because there was not a venv.

So I created a venv, activated it and ran the commands inside the venv. Which then failed, because the venv was using pip 18, not the updated pip from the global environment.

Eventually, I got it working by creating the venv, activating the venv, updating pip, and then doing the stuff I actually wanted to do. Which is a lot of extra code to understand when looking at the CI config file, plus extra work the CI server is doing, making the job take longer. And if there is some other tool in the global workspace that was also updated, I'll need to update the in the venv as well.

So it is a lot harder to understand and maintain, and slower to run, that being able to do

maturin develop --without-venv-which-is-a-bad-idea-if-this-is-not-a-throwaway-environment

[konstin]

i don't know you're exact setup, but there's nothing that maturin develop can do and that maturin build followed by pip install doesn't.

Eventually, I got it working by creating the venv, activating the venv, updating pip, and then doing the stuff I actually wanted to do. Which is a lot of extra code to understand when looking at the CI config file, plus extra work the CI server is doing, making the job take longer. And if there is some other tool in the global workspace that was also updated, I'll need to update the in the venv as well.

if you do pip install virtualenv && virtualenv .venv, you get a venv with recent pip. i generally start by this and then install everything in the venv instead of the global environment. This one command at the beginning doesn't have a noticeable impact on build but greatly improves debuggability and avoids conflict with python libraries installed by the os. but again, just use maturin build followed by pip install if you want the global environment.

[stinodego]

I just want to add one thing to this discussion:

maturin develop does an editable install of the Python package, while building the wheel first with maturin build and then installing the wheel will always result in a non-editable install.

This is relevant, as for local development an editable install is needed, as well as for some other edge cases like running pytest-cov in your CI pipeline.

Requiring a virtual environment is just needlessly restrictive. The responsibility should lie with the developer to manage their environments correctly. Now maturin develop enforces virtual environments, which needlessly restricts some workflows, as well as getting this wrong by not supporting pyenv-virtualenv and anything that's not a standard Python venv / Conda venv.

Unless there is a technical reason maturin develop must be run in one of these specific virtual environment types, this requirement should be dropped.

[konstin]

This is relevant, as for local development an editable install is needed, as well as for some other edge cases like running pytest-dev/pytest-cov#388 in your CI pipeline.

When would you do local development in your global environment? Wouldn't this mean that all of your projects share one environment, potentially overwriting each other, and also interfering with globally pip installed tools.

Now maturin develop enforces virtual environments, which needlessly restricts some workflows, as well as getting this wrong by not supporting pyenv-virtualenv and anything that's not a standard Python venv / Conda venv.

How does this impact pyenv-virtualenv, i'd expect a tool called pyenv-virtualenv to create virtualenvs? And what are those not standard venv workflows?

i do understand that with docker using the global environment is not a problem, but for deployment cases you should use separate build stages for build and deploy anyway, and also in containers there can be os installed python tools and debugging the global environment is a nightmare.

[messense]

as well as getting this wrong by not supporting pyenv-virtualenv

I'm using pyenv-virtualenv on macOS, I haven't encountered any issue.

[g-braeunlich]

When would you do local development in your global environment? Wouldn't this mean that all of your projects share one environment, potentially overwriting each other, and also interfering with globally pip installed tools.

You probably should not, but that also should not be the concern of maturin. Also there are some packages which simply dont have any dependencies. In this case a venv is simply an overhead.

[konstin]

You probably should not, but that also should not be the concern of maturin. Also there are some packages which simply dont have any dependencies. In this case a venv is simply an overhead.

One of the key motivations for maturin is that it prevents you from shooting yourself in the foot. It will make you go the extra mile to prevent you from seemingly easy solutions that end up breaking things.

[stinodego]

as well as getting this wrong by not supporting pyenv-virtualenv

I'm using pyenv-virtualenv on macOS, I haven't encountered any issue.

For reference (this is on WSL2):

$ pyenv virtualenv maturin-check
$ pyenv shell maturin-check
$ pip install maturin
$ maturin develop
💥 maturin failed
  Caused by: You need to be inside a virtualenv or conda environment to use develop (neither VIRTUAL_ENV nor CONDA_PREFIX are set). See https://virtualenv.pypa.io/en/latest/index.html on how to use virtualenv or use `maturin build` and `pip install <path/to/wheel>` instead.
$ echo $VIRTUAL_ENV  # empty - pyenv does not set this environment variable

[g-braeunlich]

You probably should not, but that also should not be the concern of maturin. Also there are some packages which simply dont have any dependencies. In this case a venv is simply an overhead.

One of the key motivations for maturin is that it prevents you from shooting yourself in the foot. It will make you go the extra mile to prevent you from seemingly easy solutions that end up breaking things.

Just as a feedback from me as one of many users of this great tool:
IMHO, it in this case it is more than preventing to shoot myself in my foot - it is also patronizing, by enforcing to install / use 3rd party tools and thus also requiring an extra step.
In my (simple) usecases so far, I just did not use the develop mode from maturin, but used cargo directly to build the extension.

[stinodego]

Let's step away from hypotheticals for a moment and consider a very common use case: calculating test coverage in a CI workflow.

In this case:

• I need an editable install, for reasons described here.
• I do not want to create a virtual environment, since CI runners are a throwaway environment.

In fact, working with virtual environments in Github Actions is a big pain as you need to find a way to activate it and keep that state between multiple steps. You can do it by adding it to the GITHUB_PATH, but maturin does not recognize this, and I still have to call source activate explicitly.

See an example here for a workflow where you see this in action. This could be much simpler if a virtual environment was not enforced.

As a 'middle ground', I like the suggestion of adding a --skip-venv-check option. That would enable power users to work more efficiently in specific cases, while still taking new users by the hand and warning them they should use a virtual environment.

[messense]

@stinodego Have you tried pyenv activate? It does set VIRTUAL_ENV in its code https://github.com/pyenv/pyenv-virtualenv/blob/017ea60cd35c8e20a659cc09498cd51bd3925035/bin/pyenv-sh-activate#L175-L178

My pyenv shell command also set VIRTUAL_ENV and PYENV_VIRTUAL_ENV.

 ~  ❯
pyenv shell test
 ~ via 🐍 v3.9.11 (test)  ❯
env | grep VIRTUAL_ENV
PYENV_VIRTUAL_ENV=/Users/messense/.pyenv/versions/3.9.11/envs/test
VIRTUAL_ENV=/Users/messense/.pyenv/versions/3.9.11/envs/test

[ThrashAbaddon]

I'm hitting the same problem. Why does this thing enforce using venvs?

[messense]

FYI, you can fake a virtualenv by setting the VIRTUAL_ENV env var if you know what you are doing:

export VIRTUAL_ENV=$(python3 -c 'import sys; print(sys.base_prefix)')
maturin develop

or simply:

env VIRTUAL_ENV=$(python3 -c 'import sys; print(sys.base_prefix)') maturin develop

[messense]

BTW, there is a new PEP 704 pull request today that require virtual environments by default for installers (like pip).

[mitsuhiko]

I wonder if maturin could auto detect if it's run from a virtualenv. I usually have makefiles like this:

foo:
        ./.venv/bin/my-tool do-something

Unfortunately with maturin this doesn't work, it requires the venv to be activated to do something useful.

[messense]

I think it's possible, we can read ${dirname(maturin executable)}/../pyvenv.cfg to see if maturin is installed in a venv and assume we're going to use that venv when no other venv is activated.

But this can be confusing when user installs maturin using pipx since pipx creates a dedicated venv for installing maturin, that venv isn't intended for other use cases so maturin develop installing packages into it isn't appropriate.

[konstin]

@mitsuhiko Would it work if did something similar to the PEP 704 suggestion where we check if a virtualenv .venv exists in the current or any parent directory and if maturin was launched from .venv/bin/maturin (or windows equivalent)? I think by checking whether the virtualenv is in cwd (or a parent) we can avoid the pipx problem.

[oscarbenjamin]

Would it work if did something similar to the PEP 704 suggestion where we check if a virtualenv .venv exists in the current or any parent directory and if maturin was launched from .venv/bin/maturin (or windows equivalent)?

There is no necessary relationship between the current working directory and the environment that is used to run maturin. There is no reason why a virtual environment should necessarily be found in any parent of CWD and there is also no reason why it should be named .venv. You could just have a situation like this:

$ ../envs/3.8/bin/maturin develop

The way that I use pyenv-virtualenv is to create environments like:

$ cd myproject
$ pyenv virtualenv myproject-py311.git
$ pyenv local myproject-py311.git

These virtual environments are not located in the current directory but rather in ~/.pyenv/versions/myproject-311.git. The local command will create a local file .python-version which pyenv will later look at to know which virtual environment should be associated with this directory by default. Now commands like python, pip or maturin when run inside the repo dir will be run from the specified env without that env needing to be activated (because pyenv manages the redirection with its shims). This works fine apart from:

$ maturin develop
💥 maturin failed
  Caused by: Couldn't find a virtualenv or conda environment, but you need one to use this command. For maturin to find your virtualenv you need to either set VIRTUAL_ENV (through activate), set CONDA_PREFIX (through conda activate) or have a virtualenv called .venv in the current or any parent folder.  See https://virtualenv.pypa.io/en/latest/index.html on how to use virtualenv or use `maturin build` and `pip install <path/to/wheel>` instead.

The method used by maturin to check if it is being run from a virtual environment fails when the virtual environment is used by pyenv without activation because the environment variable is not set in that case.

The way to find if maturin is running in a virtual environment created by either venv or virtualenv is to look for pyvenv.cfg as described in PEP 405:
https://peps.python.org/pep-0405/

If a pyvenv.cfg file is found either adjacent to the Python executable or one directory above it (if the executable is a symlink, it is not dereferenced), this file is scanned for lines of the form key = value. If a home key is found, this signifies that the Python binary belongs to a virtual environment, and the value of the home key is the directory containing the Python executable used to create this virtual environment.

The pyvenv.cfg file is what Python interpreters (including pypy) use at runtime to know if they are part of a virtual environment. This is the definition of how the file system records the fact that a python executable is part of a virtual environment. Activation just places that binary on PATH: it is the pyvenv.cfg file that makes it a venv regardless of whether it has been activated. Checking for anything other than pyvenv.cfg will fail in many situations.

I think it's possible, we can read ${dirname(maturin executable)}/../pyvenv.cfg to see if maturin is installed in a venv and assume we're going to use that venv when no other venv is activated.

In principle pyvenv.cfg can also be in the same directory as the executable according to the PEP although I don't know when that would happen.

But this can be confusing when user installs maturin using pipx since pipx creates a dedicated venv for installing maturin, that venv isn't intended for other use cases so maturin develop installing packages into it isn't appropriate.

It might be confusing but apart from a special case check for whether or not maturin is being run from a venv created by pipx I don't think think that there is anything that maturin can do about that besides mentioning it in the docs. Presumably the same problem would exist for other things like pipx install poetry && poetry install etc (I haven't checked).

[konstin]

The way to find if maturin is running in a virtual environment created by either venv or virtualenv is to look for pyvenv.cfg as described in PEP 405

We do look for pyvenv.cfg for .venv, but we can't assume that the venv maturin is installed in is the one we want to do the editable install in.

Presumably the same problem would exist for other things like pipx install poetry && poetry install etc (I haven't checked).

No, poetry will look for - in that order - an activated venv, .venv (with virtualenvs.in-project) or one that it created or create a new venv. The main poetry installer also creates a venv to install poetry in it. While we can detect pyvenv.cfg

The local command will create a local file .python-version which pyenv will later look at to know which virtual environment should be associated with this directory by default.

Given the popularity of pyenv, it might make sense to support .python-version as a fallback for .venv.

[oscarbenjamin]

The way to find if maturin is running in a virtual environment created by either venv or virtualenv is to look for pyvenv.cfg as described in PEP 405

We do look for pyvenv.cfg for .venv, but we can't assume that the venv maturin is installed in is the one we want to do the editable install in.

Why not?

That's exactly how I tell pip which environment to install into. That is also how virtual environments always work: everything is pegged to the particular python executable which can be picked up through PATH, through pyenv, or through an explicit path or anything else e.g.:

$ mkdir envs
$ python -m venv envs/3.8
$ envs/3.8/bin/pip install maturin
$ envs/3.8/bin/maturin new demo
$ envs/3.8/bin/pip install ./demo
$ cd demo/
$ ../envs/3.8/bin/maturin develop
💥 maturin failed
  Caused by: Couldn't find a virtualenv or conda environment...

Here maturin develop needs to look in the current directory to know what we want to build and install but the current directory has no bearing on where we want to install it: maturin should install into the env from which it is run like pip does.

[MrPowers]

I got this error: Caused by: Both VIRTUAL_ENV and CONDA_PREFIX are set. Please unset one of them

Fixed it by running unset CONDA_PREFIX

My machine has both conda & Poetry installed. Just posting here to help others overcome this error.

[caine-spectre]

Hello.

[caine-spectre]

I am interested in your discussion.
Could I ask some questions?

[caine-spectre]

I want to know it's possible to use maturin build without virtual env.

[trim21]

there are still some case, for example, it make no sense to setup a virtualenv in ci.

even PEP 704 allow tools to provide a tool to disable this requirement

[messense]

Recently lots of newer operating systems with newer python/pip version are requered to use --break-system-packages for pip to install without venv, it's already causing trouble for us: PyO3/maturin-action#291

My feeling is that it's easier and consisent to just always use a venv.

[davidhewitt]

Given that the motivation of requiring virtualenvs is to avoid breaking system packages, but the ecosystem has now broadly implemented Externally Managed Environments as per the above breakages, is it time to follow suit in maturin?

i.e. we could downgrade the requirement from "you need a virtualenv" to "you cannot use develop in externally managed environments"?

[trim21]

I think "you cannot use develop in externally managed environments" make more sense.

And another workaround here is, skip maturin and call cargo directly, since pyd is just a dll and cargo doesn't care if there is a python venv.

cargo build
cp target/debug/lib.dll lib/__lib.pyd
# or
cp target/debug/lib.so lib/__lib.so

this is almost the same behavoir as ./setup.py build_ext --inplace, except you need copy final artifacts by yourself.

[konstin]

We can support using the global environment, which is e.g. used in some docker workflows.

We should not install into the user-wise site packages, as they aren't isolated from other projects.

[oscarbenjamin]

There might be good reasons for not installing into the user site-packages but I don't think maturin should try to detect this for anything other than printing out a warning. If nothing else then at least there are always going to be cases where the detection logic gets it wrong as in pyenv and other examples above.

It would be better if maturin just installs the package without trying to guess what environment to use and without trying to judge whether the target environment is reasonable. Unless there is some technical limitation that prevents maturin develop from working in particular environments then it should really be up to the user or some higher level environment management tool to ensure that the appropriate environment is targeted. Maturin is a build tool and should generally just build/install where it is asked to do so.

[trim21]

there may be someone install maturin with pipx