Create virtualenvs under common directory à la WORKON_HOME

[asmod3us]

Hello, I am converting a project from "just" direnv and its layout python statement to direnv and mise for managing tools.

Plain python venv management supports the env var WORKON_HOME to create all venvs in this directory.
With direnv, I used a shell function in direnvrc to customize the layout command:

: ${XDG_CACHE_HOME:=$HOME/.cache}

# Create venvs etc. in XDG_CACHE_HOME/directoryname-hash
direnv_layout_dir() {
  local pwd_hash
  pwd_hash=$(basename "$PWD")-$(echo -n "$PWD" | shasum | cut -d ' ' -f 1 | head -c 7)
  echo "$XDG_CACHE_HOME/direnv/layouts/$pwd_hash"
}

I would like to be able to do this in mise as well. Looks like I can automatically create and activate venvs, but only specify a relative or absolute path. Is there a way to achieve this?

[jdx]

I'm not sure off the top of my head. You'll need to look at templates and probably also the Tera docs to see what filters are available. If between them we're missing anything to make this easier I'm certainly open to adding more filters.

[asmod3us]

I attempted this and almost got it to work with exec(command=...) but then ran into #1262. env vars set from direnv do not make it to mise via use mise, I guess that's by design. I gave up on it and went back to pure direnv:

PROJECT_NAME=$(basename "$PWD")
PWD_HASH=$(echo -n "$PWD" | shasum | cut -f1 -d' ' | head -c 7)
WORKON_HOME="${XDG_CACHE_HOME}/venv/${PROJECT_NAME}-${PWD_HASH}"
export WORKON_HOME
export MISE_PYTHON_VERSION=3.10

use mise
mise install

layout python3

[jdx]

totally understand, that is a problem I've been doing some design work on recently. It's definitely going to require a lot of refactoring in the codebase though so I'm not sure when we'll get to implementing it.

When we get there, I think the solution will make this a lot easier since you could just do this (it will have previously generated env vars too):

[env]
_.script = ".build_venv.sh"

[jdx]

related to that, I wonder if it might be possible for mise to be able to read .envrc files entirely, like imagine you could do this:

[env]
_.direnv = '.envrc'

which would export all of the env vars that envrc file exports. My sense from talking to users about using direnv/mise is that it's a lot nicer to just use one tool. I want to make sure the direnv/mise integration works well, but I think long term I want to make sure that mise can support anything you can do in direnv just for people that prefer a single tool and don't need to use both.

When I first built the direnv integration about a year ago now I didn't want to reimplement direnv, but I think it might actually be a decent idea at least as a migration path to support whatever you can do with direnv.

[rsyring]

I've been working around this with:

[env]
# ...snip...

[tools]
python.version = '3.9.1'
python.virtualenv='{{env.WORKON_HOME}}/twt'

I'm not sure if the explicit use of the variable ends up being a positive or negative. It would be more flexible if you could specify a default in the main config and let the location be overriden per user:

# .mise.toml
# ...snip...

[tools]
python.version = '3.9.1'
python.virtualenv='.venv'

And then customize it if desired:

# .mise.local.toml

[tools]
python.virtualenv='{{env.WORKON_HOME}}/twt'

However, that is not supported by the current toml syntax as virtualenv item can't exist without the version item. Although changing that would also enable picking up the Python version from .python-version but also letting the user specify virtualenv location in mise.

Another option: use a mise environment variable and/or setting to set a default location for virtualenv files. However, that is problematic when you have more than one project specify '.venv' thinking the venv would be local to the project folder.

How about python.virtualenv = true or python.virtualenv = '!auto' which indicates that mise is supposed to create the virtualenv and choose the location itself. The location could default to WORKON_HOME if set and a mise cache directory if not set. Some care would need to be taken to avoid name collisions when choosing the venv root name.

[jdx]

In general I think I want to get away from adding configuration directly into [tools]. As you've pointed out, it makes it hard to override things. One could imagine a team committing a .mise.toml with a python version and an individual developer wanting to just add a venv to it, but right now they will have to duplicate the version which could go out of date.

I don't think it would make sense to attempt to try to merge these either. The way mise works is individual tools get completely overridden and I think that's behavior that should probably stay. Maybe, but my gut says that's not the way to go. I feel the logic around that might get quite complicated. I'm thinking what happens if you have a venv defined in your global config, should you be able to turn it off in a local one?

[tools] also causes problems because it doesn't work with the "system" version which mise generally sees as a "no-op" and does nothing with, including managing the venv. We could get around that by not needing to have a system tool at all and just putting it in [env]

The way I've thought of this is that the venv isn't actually related to the python "tool"—it's actually part of the environment which is why I think it should get moved there, e.g.:

[env]
_.python.virtualenv = '.venv'
[tools]
python = 'latest'

and this could have details on it as well:

[env._.python.virtualenv]
path = '.venv'
create = true

Right now the autocreate is behind a setting, but I think something like above would be clearer.

[rsyring]

I guess I thought env was mostly for environment variables. So putting what are essentially tool configuration/settings in the environment section seems weird to me. But mise is already way more useful than what I could ever build so what do I know. :)

[jdx]

so a lot of people are saying that to me, but the answer is sort of two-fold:

1. I just think it's the right place. It aligns well inside the code (which often makes me thing I have the right design) and I think venvs are just env vars with some side effects (though you might be able to argue tools are as well—so it's admittedly subjective)
2. it makes it possible to for the venv to use env vars previously defined or later vars to use $VIRTUAL_ENV:

[env]
_.python.virtualenv = '.venv'
MY_VENV_FOO = '{{env.VIRTUAL_ENV}}/foo'

that's obviously a bit contrived, but I want all env var stuff to work like this and I'm sure it will be useful to use previously defined vars.

[rsyring]

so a lot of people are saying that to me

:)

I think venvs are just env vars with some side effects

My guess is that this is the root of the disconnect in our perspectives. When I think about an environment variable, its just text passed from process to process. That's it. No side effects.

While a virtualenv "Works" through the manipulation of environment variables, I think the concept of a virtualenv is a tool. As evidenced by the fact that you have to decide where you want it to live, create it, and there are a number of different options available when you do create it. There are at least two popular tools you can use to create it (python's venv and virtualenv), it can be deleted and/or recreated (b/c you change the Python version you are using), etc.

I'm glad to have the tool and that it manages virtualenvs so probably don't care much in the end how it gets configured. But since others were telling you the same thing, I thought I'd elaborate some.

[jdx]

keep in mind I'm trying to get out of managing the virtualenv. I don't think it's wise for mise to be trying to do that. That logic often has problems and I think it's better for the user to create the venv on their own and just let mise activate it. In that scenario, all mise is doing is modifying PATH and VIRTUAL_ENV—nothing more

[rsyring]

keep in mind I'm trying to get out of managing the virtualenv.

FWIW, that was the conclusion I came to based on the issues I've been a part of regarding virtualenvs in the past couple days: mise shouldn't manage them, at least not in the core.

But if mise isn't going to manage it, why have mise activate it? IMO, the problem with the current state of virtualenv activation is that the configuration of the location is not easily customizable per developer. Dev A is fine putting all their virtualenvs in the project directory under .venv. Other devs want them all out of the way in some system temp or cache directory where they aren't going to be backed up, etc. I believe when the activation is moved away from the python tool config and into the env config, this easily becomes configurable per developer by using a .local config file and/or an env_file. So a strong +1 there.

At the same time, I also think its true that a lot of us don't even want to think about a virtualenv if we can help it. I absolutely LOVE that when I'm in a project folder that's using mise the venv is already activated. But if/when to create it and where to put that venv in the filesystem is the crux of the problem. I think, as you can see from the various issues, people want configurability. And the problem with your current approach, IMO, is that it's not going to solve the fact that on different dev machines the venv is going to be in different locations. I absolutely DO NOT want .venv folders scattered around my projects. I don't mind our entire team needing to define WORKON_HOME and using that to set the location. I don't mind having to create the venv myself, although its a much nicer dev experience if some tool/plugin would do that for me.

But, the best case scenario is that there is a plugin of some kind that lets me configure how a virtualenv should be created. I can configure the location and any settings in my user-level config file. I.e. as a developer, I want all my virtualenvs for all my projects handled "this way" by default. Another dev can use a different plugin for managing virtualenvs and/or configure it differently and it has nothing to do with the project config. Is the Python tool active at this point? Could a plugin just use whatever python is on its path when its called to interact with a virtualenv?

What's not clear to me (only because I'm not familiar, I'm sure the info is out there) is whether or not plugins in this ecosystem could connect with mise core and tell it "Here's the location of the virtualenv I want you to activate." Because, what I don't want to do, is have a plugin managing the virtualenv as described above but then still have to setup the location of that virtualenv manually in every project. Its doable and better than alternatives, but certainly not ideal. Or, is it the case that if a plugin is managing the virtualenv it can also just activate it as well? The only disadvantage there is activation is in the hot path and likely much slower than the activation mise core can do? Maybe not because of the cache?

I'm sure a lot of these things are more obvious to you and others who knows mise/asdf better than I do. I'm just thinking out loud as a user.

[jdx]

It's not easy, and there are a few gotchas, but it's absolutely possible to do what you're thinking in a plugin. That's how mise-poetry works.

None of this really is in the core anyhow, it's in a core plugin which is different. Better in this case because it means an installed plugin can do all the same stuff (though with some performance caveats running in bash and not rust)

If you or another python dev wanted to make a mise-virtualenv plugin I would be more than happy to support it, help you write it, and make whatever changes we need in the plugin interface to make it better. Also promote it in the docs.

[jdx]

I think mise-python is archived but it probably still works. That might be a good starting point, fork that, have a look at bin/exec-env where the important stuff is.

It's already creating the venv so you should be able to just extend it with WORKON_DIR or whatever other functionality you want.

I say "you" but I'm really just speaking to anyone interested in investing the time here. It can't be me, I'm just never going to be as familiar with python idiosyncrasies as a bonafide python dev. More than happy to make modifications to core in order to support it though. I've already made several to make it possible to manage venvs in the way plugins currently do.

Admittedly not all those modifications are documented so you're probably going to need to read the code and/or talk to me about them.