[RFC] The next generation of virtualenv

[gaborbernat]

Following the discussion in 1362 it became apparent that for virtualenv to be able to support the new world (Windows Store installs, venv interpreters) we need some major change of direction.

Here's my proposal for going ahead with this project. This will be a major refactor for the project, that plans to keep existing interface compatibility though.

Project goal

• create a fresh copy of the system (invoker) python that has its own custom, separately controllable package list,
• an interface and behaviour that's as much as possible consistent across all supported Python environments (e.g. backport new venv features to older interpreters),
• extensible (both creation and shell activation script support wise),
• PyPi package (ability to upgrade out of band with interpreter).

Planned features

• universal pypi wheel,

• cross-platform - Windows, all UNIX flavours, MacOS.

• support for most supported pythons, the initial pool of supported Python version: python2.7, python3.4, python3.5, python3.5, pypy3, pypy2 (hoping for IronPython and Jython at some point - any other ones we should support?)

• A two year grace period of support: our interface will keep the interpreter support for two more years past its EOL: creating virtual environments is so basic that this package is the last one that should drop compatibility. During this transitional two years period, we guarantee bug fixes and compatibility. New feature support is a best effort though.

• ability to specify the target python (we'll use PEP-514/PEP-394/explicit link to discover these versions) and create virtual environments even if that target does not have this package installed

• prefer built-in venv: if the target python has venv we'll create the environment using that (and then perform subsequent operations on that to facilitate other guarantees we offer)

• ability to provision seed packages (after the creation of the virtual environments this packages will be already installed):

  • we accept PEP-508 format,
  • ability to reach out to PyPi to get latest matching the spec, or offline only (such install packages must be offline)
  • by default pip, setuptools and wheel are the seed packages (these packages are also automatically injected as offline wheel packages)

• interfaces:

  • CLI interface (python -m virtualenv, virtualenv)
  • wheel interface (env PYTHONPATH=/t/path/to/virtualenv.whl python -m virtualenv) - universal wheel,
  • zipapp interface,
  • API interface: import virtualenv; virtualenv.create("target")

• shell support - activation/deactivation scripts and prompt environment variables - by default we generate:

  • bash / zsh,
  • csh,
  • fish,
  • powershell,
  • xonsh,
  • cmd,
  • python,
  • a well-defined API to install custom shell scripts (maybe as part of the plugin system).

• three-layered configuration system, each option is determined as follows:

  • first, ini configuration support (a global ini config present in the users home),
  • second, environment variables having the VIRTUALENV_ prefix,
  • finally, command line options (argparse powered)

• plugin system these are extension points the user can inject their own custom logic, and generate an extended version of virtualenv (current bootstrapping logic):

  • extend parser (add your own custom CLI flags),
  • adjust options (change options after CLI parse),
  • after install (perform some operation once the virtual environment creation is done)

• virtualenv support - the operation should work even if the invoking python is a virtualenv created python,

• venv support - the operation should work even if the invoking python is a virtualenv created python,

• relocatable environments: an environment should keep working as long as the root python is not removed from the OS (e.g. renaming the root environment folder should not break things).

Difference from stdlib venv

How we differ from the standard library venv? The virtualenv package plans to be:

• a PyPi package, as such will be more often upgraded, easier to keep up to date and offer feature parity across pythons,
• built-in interpreter discovery and cross interpreter support,
• more interfaces (zippapp, wheel, installed package),
• easier customization - plugin system,
• be the testing ground for new features (which down the line may make it into venv),
• longer EOL.

In general, offer features that other downstream tools (for example tox, pre-commit, pipenv, pipsi, pipx) who need virtual environments as an API would want.

PyPy members (and the public please too) share your thoughts or plus one if you agree. @pfmoore @dstufft @di @pradyunsg @cjerdonek @ncoghlan @jaraco @techalchemy @uranusjr @pganssle @benoit-pierre @dholth @lkollar @takluyver @zooba

[jaraco]

An ambitious proposal!

ini configuration support (a global ini config present in the users home)

Consider using appdirs (or similar) to discover configuration in platform-preferred locations.

[gaborbernat]

appdirs well could be an option, though then we'd need to start the business of vendoring stuff into our package. To be fair 70% of this is already done, would just need a better reorganization.

[pfmoore]

Why vendor rather than just depending on it? (Not that it's hard to vendor, it's just a single file). IMO, we really need to accept that (with the notable exception of pip) packaging tools really should be built just like any other application. If having dependencies is sufficiently hard that PyPA apps avoid them, what does that say about how well we're doing making it easy for our users to develop applications?

(Sorry, not making some sort of big point about virtualenv here, I just see so many people saying "adding a PyPI dependency isn't a big deal", then the PyPA tools I work on vendoring stuff makes me wonder...).

[gaborbernat]

Without vendoring the cross interpreter feature will be hard I think. E.g. creating a virtual environment for an interpreter not having virtualenv installed.

[pradyunsg]

PYTHONPATH=/t/path/to/virtualenv.whl python -m virtualenv

Uh, this would mean that we can't depend on any additional packages from virtualenv. Since we will provide a zipapp, what usecases does this enable?

[pfmoore]

@gaborbernat You mean the -p option? I guess so. It seems like exactly the sort of situation that zipapps were intended to help with, but no-one has ever really done much to thrash out the details of making it work well.

From @pradyunsg:

Uh, this would mean that we can't depend on any additional packages from virtualenv

Actually, as @gaborbernat said, the -p option probably implies that :-( But I do wonder why we need the "wheel on PYTHONPATH" option as well as the zipapp - especially given that wheels on PYTHONPATH are unsupported in general (we use them internally within virtualenv because of the usual pip bootstrapping reasons - pip is pretty much an exception to everything ;-))

[pradyunsg]

two year grace period of support

Hmm...

My worry here is that the network effects of this might cause slowdown of adoption of newer versions of Python and the combinational explosion of number of supported Pythons. CPython is discussing different release cadence for 3.9, so 2 years might even be longer than what some releases might have from core devs.

That said, I don't want to be blocking this but mostly just making sure I voice that I'm not keen on doing this.

(hoping for IronPython and Jython at some point - any other ones we should support?)

Definitely. @gaborbernat probably knows I'd say this -- these would be good to have but I'm pretty sure folks who are more familiar with the implementation would be best positioned to do this. At the very least, in terms of support in the long term, and ideally, also the work of getting it well supported in the first place (CI, tests, etc).

first, ini configuration support (a global ini config present in the users home),

I'm biased -- I'd really like to see this be TOML instead. :)

(I have not completely processed this yet since I have to go somewhere -- but this does look like a great overall goal!)

[gaborbernat]

we already can't really depend on additional packages because of the cross interpreter feature, so I don't consider this as a major drawback, and I would like to have a simple no install/dependency way of running virtual environments (needed when people invoke us from node packages). Given it does not cost us much offering besides zipapp seems cheap.

[pganssle]

prefer built-in venv: if the target python has venv we'll create the environment using that (and then perform subsequent operations on that to facilitate other guarantees we offer)

I'm not sure I understand the rationale for this. It seems like it will make the maintenance of virtualenv harder and it will make the behaviors harder for end-users to understand. setuptools is already going the other way by trying to absorb distutils rather than continue to monkey patch it.

[gaborbernat]

@pganssle the scope is different and the problem space too. virtualenv problem is that often systems place additional constraints on system packages, so they monkey patch the built-in venv to adhere to these. Replicating all these would be hard, see e.g. #1362.

[dstufft]

FWIW I think using the built in venv package for isolation is the right idea, it's built into the actual interpreter (rather than distutils which is just a stdlib module) so it can do things much much cleaner than virtualenv can.

Virtualenv also wouldn't need to monkeypatch anything, it would just use the public API which should be fine.

FWIW I would recommend checking my old rewrite virtualenv branch, it did a lot of these things already.

[dstufft]

I would probably suggest a plugin system isn't super useful for virtualenv though, at most there's likely to just be one hook, a post create step and even then you can probably get away just having a list of things to install.

[dstufft]

ALSO, you can still depend on stuff "naturally" and still use zipapps, you just have to vendor inside the zipapp itself. I would recommend moving away from the re-execute under target python option, it is a bad thing and can be done much cleaner.

[pradyunsg]

#697 -- @dstufft's older PR

[pradyunsg]

PYTHONPATH=/t/path/to/virtualenv.whl python -m virtualenv

OTOH, we probably shouldn't be doing this anyway -- https://www.python.org/dev/peps/pep-0427/#is-it-possible-to-import-python-code-directly-from-a-wheel-file

[gaborbernat]

@pradyunsg so any other methods for bootstrapping pip without putting the wheel on the path?

[pfmoore]

Bootstrapping pip is probably OK - it's what virtualenv uses now¹. But I'd strongly recommend only using it for pip, and not as a means of running virtualenv itself.

¹ But I'd avoid any complicated functionality - using a wheel on PATH to install pip from that wheel is pretty much guaranteed to be OK. But you might have issues if you, for example, access the internet using such a wheel, as I think certifi needs its embedded certificate bundle as an actual file (but I may be wrong, that's never caused me a problem, but I know get-pip.py extracts the certificates from the wheel explicitly to address this point.

[gaborbernat]

Any reason why we don't ship pip as zipapp? That would avoid this need as I understand, and zipapp is suppported with python 2.7 🤔

[techalchemy]

@gaborbernat have you tried it? does it work? if the approach you outlined works on windows I don't have any initial concerns but @uranusjr likely would have feedback. My only comments are in line with @dstuff which is to say -- simpler is better

[pfmoore]

Any reason why we don't ship pip as zipapp?

Because in certain cases, pip won't run direct from a zipfile (or a wheel, it's the same thing), as I said above. 99% (alert - made up number) of the time, it works, but sometimes the fact that the certificate bundle needs to be a real file matters.

It would probably work fine to bundle pip as a shiv though, as shiv unpacks the zipapp before running it, precisely to avoid the corner cases where running from a zip breaks. Nobody to my knowledge has tried it, though.

As usual, the main reasons it hasn't happened are:

1. No-one thought of it.
2. Nobody had the time/motivation to do it.
3. The current approach works fine, and there are bigger fish to fry.

[pradyunsg]

so any other methods for bootstrapping pip without putting the wheel on the path?

I don't think we need that here -- virtualenv is using the pip wheel to install from itself IIRC, which should work fine. As @pfmoore said, the only "extra" thing that get-pip.py does is unpack the certificates and provide a path to them. Since virtualenv's usecase won't hit the network, we don't need to do anything new here.

The current approach works fine, and there are bigger fish to fry.

This, much more than the others IMO. We could but there's more impactful issues to tackle. :)

[dstufft]

I had a PR at one point that built pip as a zip app, it worked but IIRC I had to make changes to pip to make it fully functional.

[pradyunsg]

That might be complicated a slight bit now since pip does invoke itself as a subprocess of itself for PEP 517. I'm not sure but that's definitely worth exploring.

[uranusjr]

At this point shiv (or another self-extracting option) would be more suitable for pip than trying to work pip into zip-runnable. But I don’t think this is a particularly important thing at this point. ensurepip already goes the wheel-on-path route, why not just do the same. Alternative pip bootstrapping can be explored after we fry all the bigger fishes, and potentially be contributed back to stdlib.

[pradyunsg]

Alternative pip bootstrapping can be explored after we fry all the bigger fishes

Sounds like a plan [one that we're following already. ;)]

[gaborbernat]

Thanks for the links, based on feedback I've started some initial work over at https://github.com/gaborbernat/virtualenv/tree/rewrite/src/virtualenv. The plan is to get it into a release shape within the next month (and have a week of open review before that). Based on the impact of the change, I plan to make it release 20.0.0.

[mattip]

+1 for using venv if it exists. That would make PyPy's use case simpler.

[gaborbernat]

So the rewrite got to a state where I feel comfortable people having a look over it and giving some feedback. Please reach out to me if you have some free time to help with this; the plan is to have it be released by end of month 🤞 Note https://github.com/pypa/virtualenv/milestone/7 still need to be implemented, however, the core idea is there.

PS. Created a feedback PR under #1481

[warnes]

Feature request: Provide a mechanism for defining custom environment variables, for instance by placing a env.txt file in the venv bin directory with the standard format:

VARNAME=value
OTHERVAR=othervalue

[gaborbernat]

I don't think I understand your use case? How would these environment variables be used, set, when?

Edit. NM. I see you expanded on #1124.

[gaborbernat]

A small update; I've managed to close most blocking issues, other than two which still need some love: testing include headers + caching python interrogation calls with some lock applied over the app data, so we can create virtual environments in parallel (to different locations). At the current pace I might be a few day late... but definitely should be out by 7th February.

[pradyunsg]

What's our plan in terms of the rollout? Would we have a beta released prior to the main release?

[gaborbernat]

Yeah, want to get something out Monday... And then leave tentatively a week for feedback, fixes... And then replace master with rewrite and release.

[pradyunsg]

I'm not 100% sure that a week would be enough and we should definitely communicate across multiple channels that we want folks to try out the beta.

[gaborbernat]

The keyword there was tentatively. I'm optimistic here, given the amount of effort I've put in I'm not expecting major issues, but 🤷‍♂

[gaborbernat]

virtualenv 20.0.0 beta1 is available.

[gaborbernat]

As the beta is out, I feel this issue served is the purpose. I'll close and we can discuss more focused on dedicated issues.