pixi global design proposal

[Hofer-Julian]

We are revamping pixi global to allow shareable global environments and inject dependencies. The details of the design proposal can be found here. We would love to hear your thoughts.

Tracking milestone: https://github.com/prefix-dev/pixi/milestone/9

[vigneshmanick]

very nice feature, some questions from me

• does the manifest also allo pip dependencies defined ? also will there be a possiblity of running activation and deactivation scripts and setting variables?

  [envs.python.pypi.dependencies]
  pkg = 'version'

• will there be a way to see which binaries will be exposed before entering them in the list? i feel this exposing will lead to issues if one of the exposed binary is dependent upon another that has been masked by the user (without realizing that's the case)

• I have a mixed feeling about allowing multiple global manifests, this goes against global since now you can have as many. Why not by default allow one inside the PIXI_HOME/global-manifest.toml (note there is no manifests folder ) and for the rest only allow via cli pixi global --manifest company-manifest.toml but don't allow other global manifests in the PIXI_HOME/ folder. This way an admin can setup a global manifest and users can set their own

• Based on the above point, why not just rename the file PIXI_HOME/global-manifest.toml , feel there is an overload of pixi term in the paths :)

• i don't like environment variable option since that can easily be overridden by changing the variable. I feel this should be a value in the pixi.toml or at the user level config file.

[Hofer-Julian]

does the manifest also allo pip dependencies defined ?

Not the first iteration. To me, it is an obvious next step.

also will there be a possiblity of running activation and deactivation scripts and setting variables?

I would like to avoid that. What is your use case for that?

will there be a way to see which binaries will be exposed before entering them in the list? i feel this exposing will lead to issues if one of the exposed binary is dependent upon another that has been masked by the user (without realizing that's the case)

Do you have an example of a binary that only works if another binary of the same package is in your PATH?

I have a mixed feeling about allowing multiple global manifests, this goes against global since now you can have as many.

To be honest, I don't see why multiple manifests would make it less global. I don't care about this functionality for my personal needs, but people are asking for it, and the use case sounds valid to me.

feel there is an overload of pixi term in the paths :)

That is true. How about default.toml? @ruben-arts @nichmor

i don't like environment variable option since that can easily be overridden by changing the variable. I feel this should be a value in the pixi.toml or at the user level config file.

I'd be fine with using the user config file instead of setting an environment variable to configure the place where pixi global searches for manifests. Any opinions here? @ruben-arts @nichmor

[vigneshmanick]

also will there be a possiblity of running activation and deactivation scripts and setting variables?

I would like to avoid that. What is your use case for that?

Some environments or packages require some modification to the environment via variables/scripts. On top of my head, tensorflow requires some modification to ld_library_path for the installation to work properly.

export LD_LIBRARY_PATH=$(find /app/venv/lib -type d -path "*/python*/site-packages/nvidia/*" -prune -exec printf ':%s/lib' {} +):${LD_LIBRARY_PATH}

There are other similar cases where some sort of modifications/customizations are required. Example, global dev environment/ global prod environment -> run activation scripts to perform housekeeping/preparation tasks for those.

This just allows lot of flexiblity for the users.

Do you have an example of a binary that only works if another binary of the same package is in your PATH?

direct example no, but a python package can define an entrypoint and these can depend on other packages in PATH

[Yura52]

Thanks for the proposal! I would be very nice to define all global packages in a single file that could be reused across machines. I would like to share some first impressions and questions.

(1) I wonder if the file format can be simpler in simple cases. For example:

[envs]
zsh = "*"
git = "2.46.0"
python = "3.12.*"

If two versions of python are required, then, the above file could be incrementally extended to the following:

[envs]
zsh = "*"
git = "2.46.0"
python = "3.12.*"

[envs.python-3-10.dependencies]
python = "3.10.*"

[envs.python-3-10.exposed]
"python3.10" = "python"

(2) Continuing the above example, personally, I have the most recent version of python installed with pixi global, and I would use pixi exec -s python=3.10 python if, for some reason, I needed to launch specifically python=3.10. I am curious what are the cases when two version of python (or of some other package) are needed to be globally available?

(3) When should I prefer defining an environment in the global manifest over defining it as a separate pixi project?

(4) Will it be possible to selectively turn on/off global manifests? For example, I would like my "personal" global manifest to be always active so that I always have access to my basic tools like zsh, git, python, etc. However, I may want to selectively turn on/off, say, a global manifest provided by my company where I work.

[Hofer-Julian]

I wonder if the file format can be simpler in simple cases

That is a good idea, but with the current proposal exposed is a required key. We considered exposing all binaries implicitly if exposed wasn't given, but that has different problems

I am curious what are the cases when two version of python (or of some other package) are needed to be globally available?

Sometimes you need one nodejs version for one project and another version for another project. With this functionality pixi can replace existing nodejs version managers. Or pyenv for that matter

However, I may want to selectively turn on/off, say, a global manifest provided by my company where I work.

You could remove the .toml extension temporarily, then your company manifest wouldn't be considered anymore

[Yura52]

Thank you for the reply!

Sometimes you need one nodejs version for one project and another version for another project.

I find this somewhat relevant to my question (3) above. I thought that granular per-project dependency management is where standard pixi projects (backed by pixi.toml or pyproject.toml) were supposed to be used. And the new functionality seems, to some extent, to overlap with already existing functionality, thought I am a bit unsure to what extent.

You could remove the .toml extension temporarily, then your company manifest wouldn't be considered anymore

I see. I have no evidence to support my intuition, but I think that this "selective activation" of global manifests may be a useful feature, and this scenario could be a first-class citizen in pixi global. Also, if there is a dedicated command for this selection (e.g. pixi global use <global manifest names>), perhaps, it can be a chance for the global environment to run activation scripts to address some of the limitations mentioned in this discussion.

[dhirschfeld]

I think I've mentioned before, but I can't help thinking most problems would be solved by treating "global" as an ordinary pixi project. Each tool (or set of tools) would then be an environment in this global manifest.

The only difference between a normal pixi project and the global project would then be some syntatic/CLI sugar and the ability to expose binaries outside of the environments.

[Hofer-Julian]

@dhirschfeld I was actually planning on implementing on supporting that.
But I am convinced now that this will do more harm than good. Looking at #1610 many people want the normal pixi workflow, just with a manifest that is accessible globally, selected via a (short) string. pixi global will never provide that, so we should keep those two things separate.

[dhirschfeld]

I think pixi global and conda envs are two very separate things.

It's understandable why people familiar with conda try to use pixi global to recreate the conda env experience as they don't want an environment tied to a particular "project"/directory.

IMO the conda people really just want pixi shell --manifest-path which is what I use myself. It's not as discoverable though and adds a lot of boilerplate to the usual mamba activate dev-312 workflow.

I think pixi global should just be an ordinary pixi environment in a "well-known" location which should have the added benefit of exposing binaries. "conda"-like environments could then just be pixi projects in a "well-known" location with maybe CLI sugar to "activate" which would open a shell for that manifest-path.

[dhirschfeld]

Do you have an example of a binary that only works if another binary of the same package is in your PATH?

I believe git-credential-manager requires git to be installed alongside it.

[dhirschfeld]

I would like to avoid that. What is your use case for that?

Some tools (e.g. dotnet) don't work without running activation scripts which set PATH entries and environment variables.

[olivier-lacroix]

@Hofer-Julian , a couple of thoughts

• I wonder if the global & project CLI commands could be made more consistent. In particular pixi install in a project is more or less the equivalent of pixi global sync.
• I am not sure we need both the pixi global install, and the pixi global add. Couldn't the add command behave like the install command in case a specific environment is not passed?

[Hofer-Julian]

I am not sure we need both the pixi global install, and the pixi global add. Couldn't the add command behave like the install command in case a specific environment is not passed?

The very first version of this proposal worked like this, using pixi global install for both use cases instead of pixi global add like you propose. The people I asked found this really confusing, but I don't mind re-evaluating here.

[Hofer-Julian]

I wonder if the global & project CLI commands could be made more consistent. In particular pixi install in a project is more or less the equivalent of pixi global sync.

It also bothers me a bit. The cleanest way would be to rename pixi install to pixi sync, but I doubt it's worth breaking for this.

[ashwinvis]

👍 on pixi global add.

1. This would harmonize with project level pixi add .
2. Similar to yarn global add for those familiar with that.

Instead of the inventing a new way of adding dependencies, maybe we can add pixi global inject package [dependencies ...] (similar to how pipx inject works)

[ashwinvis]

A minor detail. In the section Simple workflow, perhaps illustrate the use of pixi global using an actual command line tool like git or ripgrep or nvim instead of python.

Is this approach of installing python globally encouraged? I would suppose it will make the user's environment very messy.

[Hofer-Julian]

Yeah, I had the same thought yesterday.

I think I will change python&pip to ipython&numpy.

[cdeil]

It's understandable why people familiar with conda try to use pixi global to recreate the conda env experience as they don't want an environment tied to a particular "project"/directory.

I was expecting / hoping this pixi global design proposal would be that, to offer the global conda env equivalent.

@Hofer-Julian - IMO it would be great if you could make it clearer at the start of the document what this is about for normal users.

You write as first goal:

Shareable global environments.

But then the word "share" doesn't appear any more in the document and even after skimming it it's not clear to me what you mean with "share" here, if it's between shells or pixi envs on my computer, or between colleagues on multiple computers etc.

Would be great IMO if you could add a few more words in the intro on the WHY and WHAT this is if the target audience is normal / casual / new users of pixi. :-)

[dhirschfeld]

@cdeil - if you're looking for a conda env type workflow, I believe the latest design proposal for pixi is:

• Feature Idea: pixi env "conda workflow" #1610 (comment)

[jamesmyatt]

Can you be more explicit about how pixi global is supposed to work compared with pipx and condax, which I think are all trying to do the same thing? First of all, are they really trying to do the same thing? If so, then say so. How do the commands map across? etc

[Hofer-Julian]

The goal of condax is indeed very similar. pipx targets PyPI, while pixi global is for now specific to the conda ecosystem.

[Hofer-Julian]

pixi global is now released, so this design proposal discussion can be closed now.

Thanks for the input everyone 🫶

[matthewfeickert]

As the v0.34.0 pixi Global Manifest docs still mention

This document is work in progress, and community feedback is greatly appreciated. Please share your thoughts at our GitHub discussion.

I'm coming here with a question:

As I noted in GitHub Discussion #2378

The v0.34.0 docs mention mention the ability to do

pixi global update [--package PACKAGE] <ENV>...

and

pixi global update-all [--assume-yes]

but those commands don't seem to exist yet.

Are there plans to add pixi global update-all and pixi global update --package in the future? Or are those options not being considered anymore?

[Hofer-Julian]

Update fulfils both of these functions. Run pixi global update to update all environments. Run pixi global update ENV to update one specific one

[matthewfeickert]

Okay, so the docs need to be updated then to not include these commands.

[Hofer-Julian]

Kinda. It is only the proposal, the actual docs should be accurate. Maybe we should remove the proposal. What do think @ruben-arts?

[matthewfeickert]

Ah, very good point! I actually didn't realize it was still the proposal as I hadn't paid attention to the sidebar

I'll defer to the pixi team on if it should get removed or not, but making it much more prominent that it is a design proposal would probably be good (the term "proposal" doesn't show up anywhere in the document itself).