Can we reconsider env._.*?

[camsteffen]

I'm a big fan of mise and have used mise/rtx for about a year. 😀 This post might sound a bit rant-y but it is coming from a place of care and respect for this project and its maintainers.

Recently there was some restructuring of environment variable related config. env_file became env._.file and some more configs under env._.* are on the way. I think this is a poor design decision and it comes at the same time as the name migration which is more churn for users (though I understand the reasoning for the rename and think that was a good idea).

Every time someone sees the underscore for the first time, it will lead to "huh, what does that mean?" and you have to go read the docs to find out (answer: it doesn't really mean anything?). The old/flat structure is just easier to read.

env_file -> env._.file is a loss of clarity. It takes a term that people are familiar with - "Env file" - and effectively chops it into "Env > file" which is not as self-documenting. Also env_file matches docker compose syntax which many devs are familiar with.

The name env naturally makes a mental association with .env files which are a set of env var values and nothing else. But now this label is turning into "environment variables and also a category of things that dynamically set environment variables and maybe some things that can do anything whatsoever (env._.source). The website currently puts all this under the heading "Environments" which indicates that this is a cohesive category in someone's mind, but to me it is really unclear what that category actually is. There was some talk of maybe putting python venv config under env. python venvs are an entirely different thing than env vars. Let's not invent super abstract categories that don't make sense to the common folk users like me.

Fewer top-level sections in the config file is not inherently simpler. I would say TOML is actually designed to encourage flat structure for simplicity rather than deeply nested structure. Just because env and env_file are similar in purpose, doesn't mean they have to be lumped into a hierarchical structure. Hierarchies add another thing to grok.

I suspect that a large majority of mise config files will be very very small - like 1-5 lines. Think about this. The esoteric underscore syntax will look especially unnecessary in these cases. I wonder if we've been too focused on all the extreme complex edge cases because those are the uses that tend to appear in new issues and discussions.

[jdx]

there is a very long discussion about this here if you haven't seen it where this was debated pretty heavily: #1357

My mind is not fully made up, especially when it comes to the specific syntax, however the old env_file stuff certainly won't work for what I want to do so that's not on the table.

It will make it so you can do something like this:

[env]
_.source = {file="get_conda_venv.sh", cache_glob="pyproject.toml"}
MY_VENV = "{{ env.VIRTUALENV }}"

where "get_conda_venv.sh" is a bash script that exports the virtualenv env vars, making #1514 an easy thing for a user to resolve on their own without needing to build an entire plugin.

This will be a great simplification in the way mise works under the hood. A lot of the logic around caching and how tools set their env vars I've found can use the same logic which will simplify the codebase by reusing the same logic I'd be exposing to users. The more I dive into env._ the more convinced I am it's the right direction. It just solves too many problems in an elegant way for me not to.

I don't think that env._ will be too much of a leap to users. I get that it's a bit bizarre at first glance though. A lot of this is that we're limited by toml where other configuration languages could offer a more elegant syntax for sure. At some point down the road I may look at adding an alternative configuration language that would meet the needs a bit better. kdl is one that I think might work well. That said, if that ever happened I don't think toml would ever go away either.

Anyhow, the syntax for "env 2.0" is up for discussion, but whatever we come up with is going to need to be able to do what I did in that code sample above. env_file and friends are not capable so it's not going to be any use trying to convince me to go back to that.

Other tasks this will resolve:

• env._.source #1447
• Allow env variables to reference previously defined env vars in same configuration file #1262
• new venv syntax #1303
• Cache env vars with slow exec function calls in templating #1261

[booniepepper]

This does not constitute a language, not even a dialect. It's modeling hierarchical data in TOML. (TOML is a language)

Like I would expect that you can convert a toml file to json and it still makes sense.

Why does it fail to make sense, and also who benefits if it makes sense in json? I really am not understanding your goals here, so it might need elaboration

I think I would advise against using mise.toml configurations outside mise. The goal I'm seeing Jeff go for here is to have a human-readable configuration file that mise can use as well-formed data at runtime to set up execution contexts

[camsteffen]

This does not constitute a language, not even a dialect. It's modeling hierarchical data in TOML. (TOML is a language)

Absolutely.

I really am not understanding your goals here, so it might need elaboration

I guess my goal is that every item in the hierarchy has a discernible and straightforward meaning. In env._.file there are three items. I can tell you exactly what the file part means. For the other two env and _, I'm at a loss.

[jdx]

if you want to see this changed, you'll need to give me some ideas on how this could be otherwise represented in toml. Right now I think env._ is the best solution. I don't agree that users will struggle with this very much. It will look weird at first, sure, but I don't think this is going to be a major issue for folks.

I mean sure, if you want to write a parser for mise.toml this is going to make it hard, but that's not a use-case I care about whatsoever. These files are for humans, not machines.

The old syntax really doesn't work. It won't be possible to have env vars depend on other vars. I also never liked putting things at the top level, I much prefer grouping everything under [env] like we group things under [tools] and [tasks]—this fits the behavior for the other mise axes.

[camsteffen]

The old syntax really doesn't work. It won't be possible to have env vars depend on other vars.

Really? I might try implementing it if there's interest.

I also never liked putting things at the top level

I guess that is ultimately the answer. I can't say I follow your rationale but this is your project to do as you please.

[jdx]

users should be able to use env vars provided by vars/env_file/virtualenv/scripts as inputs to later vars/env_file/virtualenv/scripts. That's why I'm putting everything into [env] because you need the ability to say what order they get loaded.

If env_file sits at the top level it isn't realistic to specify the order. Maybe with crazy parsing logic but even then I can thing of a few use-cases that would not work.

[rsyring]

there is a very long discussion about this here if you haven't seen it where this was debated pretty heavily: #1357

There was also discussion regarding this at: #1438 (comment)

@jdx This is a great tool, really. Remarkable in many ways. Mise, along with your suggestions to use Fish and Spaceship have revolutionized my cli experience and I've been using Linux for over 20 years now. Thank you for all the time and effort you have put into this.

I'm relatively inactive in this project, but have hung around more than my usual and commented more than my usual because of how valuable mise has become in my workflow. But, even in just that relatively small amount of exposure, I've seen this brought up a few times. I really think you may be underestimating how jarring this is to user experience.

These files are for humans, not machines.

I agree wholeheartedly! :)

This will be a great simplification in the way mise works under the hood. A lot of the logic around caching and how tools set their env vars I've found can use the same logic which will simplify the codebase by reusing the same logic I'd be exposing to users. The more I dive into env._ the more convinced I am it's the right direction. It just solves too many problems in an elegant way for me not to.

You mentioned simplification of the implementation in #1438 as well. But it seems to me like the implementation is bleeding too far into the user experience / configuration.

if you want to see this changed, you'll need to give me some ideas on how this could be otherwise represented in toml. Right now I think env._ is the best solution....

It won't be possible to have env vars depend on other vars...

... users should be able to use env vars provided by vars/env_file/virtualenv/scripts as inputs to later vars/env_file/virtualenv/scripts. That's why I'm putting everything into [env] because you need the ability to say what order they get loaded.

FWIW, this is the first time I've groked that you have configuration scenarios in mind that this env._ solution solves and its not just to help you simplify the implementation (which is a worthy goal). Do you have a set of configuration scenarios that I/we could review that demonstrate the various scenarios that are being solved with this new syntax? If so, I'd be happy to take a look and see if I can think of something more user friendly.

[jdx]

we talk about the order problem a fair bit in this thread #1357 (reply in thread)

[rsyring]

@jdx thanks for pointing that out. I've now read #1357 and I've also read the current env._ documentation.

Critique

From you in #1357:

My goal is to also teach people how mise works under the hood so when they want to branch out to something like running a custom script to generate their own env vars it'll look familiar because it's similar to how they setup a venv or added a dotenv file.

It's a slight hit to DX up front, but it helps users along the path to building more complex config files in the direction I want them to head.

I'd like to respectfully suggest that you don't actually want to teach people how mise works under the hood. That would imply that you want us to understand how the Rust implementation works and, I think, you'd probably agree that doesn't and won't ever matter to almost everyone who uses mise.

I think, what you may really mean, is that you want to make it intuitive for people to understand that whether they are getting env vars by specifying them manually in [env] or by running a script or by using a plugin (e.g. for Python virtualenvs) those methods are all just doing the same thing: creating env variables that are available to CLI tools.

FWIW, I believe separating [tools] and [env] makes sense and I like the 3-axis idea. I think the objection from us users of the Python virtualenv support is probably due to:

• Because the current implementation puts the tool and activation in the same area, we feel they should be put there. But I can see now why they'd be separate and I think, in the long run, its better.
• I think, had I just been coming into mise with no prior knowledge, separating the install of Python from the activation of the virtualenv (which is just env manipulation) is not that hard to grok.

Surprisingly, to me anyway, after I read the current docs on env._ I was beginning to think it might not be that bad until I hit this:

[env]
_.source = "./script_1.sh"
_.source = "./script_2.sh" # invalid

For this use-case, you can optionally make [env] an array-of-tables instead by using [[env]] instead:
toml

[[env]]
_.source = "./script_1.sh"
[[env]]
_.source = "./script_2.sh"

That's just...ugh! :) You mention in #1357 that you didn't want to use "@nix".flake = "flake.nix" due to the quotes but needing to turn env into an array is worse than that IMO.

We have discussed the issue of ordering in [env] to facilitate something like:

[env]
mise.python.virtualenv = '.venv'
MY_VENV = '{{env.VIRTUAL_ENV}}'

However, the TOML specification does not ensure ordering of keys and I think this becomes more obvious this is problematic for mise when we consider:

[env]
_.python.virtualenv = '.venv'
MY_VENV = '{{env.VIRTUAL_ENV}}'
FOO = '{{env.VAR_FROM_SOME_PLUGIN}}'

[env._.some_plugin]
a = 'bunch'
of = 'settings'
that = 'this'
plugin = 'needs' 
to = 'function'
venv =  '{{env.VIRTUAL_ENV}}'

Based on the examples I see in #1357, I'd be forced to configure it like:

[env]
_.python.virtualenv = '.venv'
MY_VENV = '{{env.VIRTUAL_ENV}}'
_.some_plugin = {a='bunch', of='settings', that='this', plugin='needs', to='function', venv='{{env.VIRTUALENV}}'}
FOO = '{{env.VAR_FROM_SOME_PLUGIN}}'

In TOML, the above two examples are supposed to be the exact same, and for good reason. One of the main points of TOML is to be able to break things out in sections when it increases readability. The env._ would prevent that when ordering is important.

IMO, given the two issues noted above the current proposal and implementation for env._ is both a violation of how TOML is intended to be used and 2) a bad DX sometimes and a terrible one at other times and should therefore be rejected (please :)).

Proposal: env key structure

How about:

[env]
COMBINED_VAR = '{{env.FAVORITE_VEGGIE}} + {{env.API_TOKEN}}'
MY_VENV = '{{env.VIRTUAL_ENV}}'
API_TOKEN = '{{env.CLOUDFLARE_API_TOKEN}}'
FAVORITE_VEGGIE = '{{env.FAVORITE}}'

# Option if you don't like scripts as a map.  Commented to keep from syntax error.
# mise.scripts =  ['./get_my_tokens.sh', './rubarb.sh']

[env.mise.scripts]
# The keys don't matter, just nicer syntax IMO
tokens = './get_my_tokens.sh'
vegetables = './rubarb.sh'

# Might be problematic, see below
remote = './get_secret_from_AWS_using_API_TOKEN.sh'

# virtualenv is a "plugin" (or whatever) that provides environment variables
[env.virtualenv]
path = ".venv"
default-packages = []
create = true

The only downfall I currently see with that is the possibility that someone will want an environment variable like "mise.scripts." But, if they do, they have to quote it anyway due to the dot:

[env]
"mise.scripts" = 'a variable value'
mise.scripts = []

While theoretically the above could be confusing I doubt it will ever be much of problem practically.

Proposal: Variable Interpolation

Get rid of the idea of ordering for variable dependencies and create the environment variables in two phases.

1. Gather all the variables from the various sources (direct, scripts, plugins, etc.)
2. Defer the resolution of any {{env....}} value until its dependency has been resolved. Do that in a loop and in short order everything gets resolved.

A potential problem is if you want to be able to support something like get_secret_from_AWS_using_API_TOKEN.sh where that script wont' run without API_TOKEN. I have some ideas for how to handle that but don't want to detail them here because I'm not sure if supporting that scenario is necessary.

HTH.

[jdx]

I'd like to respectfully suggest that you don't actually want to teach people how mise works under the hood.

I want people to be comfortable with the mechanics, not the code necessarily. I think if people have a rough sense of what probably is happening under the hood they'll experience more success with the tool and things like this ordering will just "make sense".

Also, being a great simplification of the codebase does help all users because it means us contributors have less to maintain and it's easier for new contributors to pitch in. That will help those that do not touch the code directly. I think everyone here is really discounting how valuable it is to have a "simple" tool. By that I don't mean a tool that doesn't do much, but one that follows relatively few rules. Fewer bugs. Streamlined features. Faster performance.

"@nix".flake = "flake.nix" due to the quotes but needing to turn env into an array is worse than that IMO.

This reads like you think using quotes is somehow a substitute for the [[env]] workaround but it isn't. Even if we used quoted keys we'd still have to use [[env]] or come up with some other workaround.

TOML specification does not ensure ordering of keys

true, but I am relying on a specific implementation of toml which does ensure ordering of keys. This means that potentially some toml parsers that are written to spec may not be capable of correctly parsing the file. That is a tradeoff I believe is acceptable. I also think that like with json it will be relatively rare for parsers to not ensure ordering or at least have that as an option so I see this as just an academic problem—not a practical one. This is what I was talking about when I said the format is for humans, not machines.

Proposal: env key structure

I don't see how this is different from what we're doing with the exception that you used mise instead of _ in the first example and nothing for the second. If it's just that you put multiple scripts into a single env._.scripts that is already supported. That doesn't handle every use-case though because conceivably a user may run a script, set some env vars, then need to run a different script.

environment variables in two phases

Users expect (and kind of need) the ability to set environment variable to be used in a script or plugin and vice-versa. For example, this should "just work":

[env]
_.script = "script1.sh"
_.virtualenv = ".env" # uses something from script1.sh
MY_PYTHON_VENV = "{{ env.VIRTUAL_ENV }}"

If your main complaint here is that you just don't like nesting things under [env] then just don't write it that way, write it this way:

env.script = "script1.sh"
env.virtualenv = ".env" # uses something from script1.sh

[env]
MY_PYTHON_VENV = "{{ env.VIRTUAL_ENV }}"

[rsyring]

This reads like you think using quotes is somehow a substitute for the [[env]] workaround

That wasn't what I intended. I was just noting that you objected to the DX aesthetics of needing a quoted key. I was pointing out that I think the DX aesthetics of repeated [[env]] sections I quoted were worse.

I want people to be comfortable with the mechanics, not the code necessarily. I think if people have a rough sense of what probably is happening under the hood they'll experience more success with the tool and things like this ordering will just "make sense".

I think we are on the same page. The config needs to be intuitive to the average developer who uses mise.

Also, being a great simplification of the codebase does help all users because it means us contributors have less to maintain and it's easier for new contributors to pitch in. That will help those that do not touch the code directly. I think everyone here is really discounting how valuable it is to have a "simple" tool.

Sorry if it came across that way. In my post above, I was trying to demonstrate that I see the value of this both to the code base and to the user's mental model when I mentioned that I supported separating tools and env and using the 3-axis approach.

true, but I am relying on a specific implementation of toml which does ensure ordering of keys.

Granted. But this is not intuitive to a user who understands what TOML is supposed to be. And the example config that gets very DX unfriendly has not been addressed. Ordering the keys conflicts with a fundamental design decision of TOML which ensures a DX friendly config format regardless of implementation details.

I just really doubt that this is the config that we want someone to have to write:

[env]
_.python.virtualenv = '.venv'
MY_VENV = '{{env.VIRTUAL_ENV}}'
_.some_plugin = {a='bunch', of='settings', that='this', plugin='needs', to='function', venv='{{env.VIRTUALENV}}'}
FOO = '{{env.VAR_FROM_SOME_PLUGIN}}'

I don't see how this is different from what we're doing with the exception that you used mise instead of _ in the first example and nothing for the second. If it's just that you put multiple scripts into a single env._.scripts that is already supported.

Actually, what I was intending to demonstrate was a syntax like:

• env.mise: defers implementation and config building to the tool registering itself as 'mise'
• env.mise.scripts: mise isn't special here other than mise (the application) registers its own plugin under the key 'mise'.
• env.virtualenv: 'mise' (or _) isn't missing here. Its saying that the plugin/tool that provides environment variables is registered as 'virtualenv'.

You could go one step further with this by not having any kind of registration system at all and doing it all in the config:

OMBINED_VAR = '{{env.FAVORITE_VEGGIE}} + {{env.API_TOKEN}}'
MY_VENV = '{{env.VIRTUAL_ENV}}'

[env.virtualenv]
_mise_plugin = 'rsyrings.venv-plugin'  # which would need to be installed already with `mise plugins install ...`
path = ".venv"
default-packages = []
create = true

# That could also have been written as
[env.foo_bar_baz]
_mise_plugin = 'rsyrings.venv-plugin'  # which would need to be installed already with `mise plugins install ...`
path = ".venv"
default-packages = []
create = true

If your main complaint here is that you just don't like nesting things under [env] then just don't write it that way, write it this way...

I think we are missing each other. Its not at all an issue of nesting things under [env]. My proposal still did that. :)

Its that your current plans are very DX awkward and violate how toml is supposed to be used, which is unintuitive to the user. So you are doing all this work to give the user a mental model that is clean and intuitive but have chosen a syntax that is unintuitive at best and terrible (DX speaking) at worst for anything but the most simple of cases.

env.mise... is much better than env._... but that is not my main complaint. My main complaint is what I stated just above, that you are making decisions that are going to give you and unworkable DX for all but the most trivial of cases.

And, even if you head in the proposed direction, you may be fighting this battle forever. Every developer who ever has to write something like:

[env]
_.python.virtualenv = '.venv'
MY_VENV = '{{env.VIRTUAL_ENV}}'
_.some_plugin = {a='bunch', of='settings', that='this', plugin='needs', to='function', venv='{{env.VIRTUALENV}}'}
FOO = '{{env.VAR_FROM_SOME_PLUGIN}}'

is going to at least consider filing an issue requesting better syntax. :)

Users expect (and kind of need) the ability to set environment variable to be used in a script or plugin and vice-versa. For example, this should "just work":

[env]
_.script = "script1.sh"
_.virtualenv = ".env" # uses something from script1.sh
MY_PYTHON_VENV = "{{ env.VIRTUAL_ENV }}"

Fair enough and I can see how that would be useful. Just off the top of my head:

[env]
MY_PYTHON_VENV = "{{ env.VIRTUAL_ENV }}"

# note that order doesn't matter
FOO = '{{env.BAR}}'
BAR = 'foo'

[mise]
env_file = '.env'
scripts.secrets = './get_secrets.sh'

[mise.scripts.script2]
_depends = ['env.mise.scripts.secrets']
path = './script2.sh'

[env.virtualenv]
_depends = ['env.mise.scripts.script2']
path = "{{env.PATH_FROM_SCRIPT2}}"

But that's only for the more complex cases. The simple cases are still simple:

[env]
MY_PYTHON_VENV = "{{ env.VIRTUAL_ENV }}"
FOO = '{{env.BAR}}'
BAR = 'foo'

mise.env_file = '.env'
mise.scripts = ['./get_secrets.sh', './script2.sh']

virtualenv = '.env'

Benefits:

1. Dependencies are not that hard to implement. Presumably mise has something similar already for task dependencies.
2. Doing deferred-and-looping variable resolution would be relatively simple as far as I can tell.
3. The config syntax is intuitive 1) its easy to tell whats going on; and 2) it conforms to how TOML is supposed to work and therefore doesn't violate an established mental model & standard.
4. The verbosity is progressive. The simple cases are still simple and the config only gets more verbose if you are using more advanced features. But, even the advanced cases remain intuitive.

I'd be happy to iterate on this more if helpful. But, only if you are coming around to dropping the current plans. :) I want to make the best use of our collective time and, if my examples and critiques above really haven't swayed you that the current plan is bad, then continuing to iterate on a new proposal is probably irrelevant.

One final suggestion: I don't know how much critique you get on various things. Given the increasing popularity of the project, my guess is there are plenty and you have to have a thick skin and confidence in your own judgement to get anything done otherwise analysis paralysis sets in. But, I think you may be too close to the implementation to see this issue clearly from the user's perspective.

Unless I've misunderstood how the more complex config cases will be handled, as evidenced by my example configs above which I note you didn't address yet, the proposal to use env._ and treat the keys as ordered is terrible DX. Please don't do it. :)

And give just the comments I've seen so far, I think there are a lot of people saying the same thing. That doesn't mean they are right, but I'd strongly encourage you to reconsider. :)

Thanks.

[jdx]

keys as ordered is terrible DX

I could not disagree more strongly. Ordering object keys is a useful way to make ordering implied and developers should not be afraid of this quality. Forcing things into an array adds verbosity which is sometimes useful but sometimes it adds too much noise—and this is a perfect example of this. I used it in oclif and it worked well there. Clap uses it everywhere for ordering the commands and args. indexmap is one of the most popular rust crates.

There is a reason nobody has suggested the obvious solution:

env = [
  {key = "FOO", value="1"},
  {virtualenv=".env"},
]

it's ugly and loses the most important quality about the current syntax: that a declarative [env] table makes simple use-cases great.

Dependencies are not that hard to implement.

Task dependencies might be the second most complex part of the codebase IMO. It will make the DX far worse than ordered object keys as well. Think about the simple case where you just want to reuse a previously defined var:

[env]
MY_HOME="{{env.HOME}}/.myproj"
MY_HOME_CACHE="{{.env.MY_HOME}}/cache"

Env dependencies would be incredibly painful. This is not a terribly advanced use-case either, what I've put above is something that should just work. Ordered keys get us there.

I know people dislike this, but so far any alternative I've seen has worse problems. I'm also really not seeing any issues with ordered keys with the exception of [[env]] which is a disappointing workaround but not a dealbreaker in my book.

Even the one you showed works fine with my syntax

[env]
_.python.virtualenv = '.venv'
MY_VENV = '{{env.VIRTUAL_ENV}}'

[env._.some_plugin]
a='bunch'
of='settings'
that='this'
plugin='needs'
to='function'
venv='{{env.VIRTUALENV}}'}

[env]
FOO = '{{env.VAR_FROM_SOME_PLUGIN}}'

is that the cleanest possible way to represent that? no, but I also think this will be rare in practice. Technically [tools] has the same problem as well as a ton of other toml files. I'm not arguing this and [[env]] are not downsides. I'm saying the downsides aren't enough to warrant losing the declarative [env] syntax, losing the ability to have env vars depend on others, and certainly not worth bringing a dependency resolver into the mix.

I'm not saying env._ is elegant. I'm saying I think it will meet everyone's needs the best when you way all the pros and cons. Keep bringing ideas, I'm happy to keep brainstorming. That said, keep in mind I've been working on this problem for over a year so if there was an obvious answer I likely would've found it by now.

If anyone wants to complain, go right ahead, it won't offend me. I don't think this is lovely syntax either. However if your goal is to actually do something different here, you'll need something that solves our use-cases better.

I don't take the criticism personally. It's a reflection of this just being a hard problem that probably lacks any good solution. Nobody has really complained about anything else in the file because those all had obvious answers—this doesn't. If you want to point out that this sucks—I commiserate. However I also don't think there is much we can do about it.

[rsyring]

@jdx I'm going to break out my replies to your last post because I think some things are getting missed due to the large wall of text:

It will make the DX far worse than ordered object keys as well. Think about the simple case where you just want to reuse a previously defined var:

[env]
MY_HOME="{{env.HOME}}/.myproj"
MY_HOME_CACHE="{{env.MY_HOME}}/cache"

I don't understand your concerns with DX here regarding my proposal. In my proposal, the above works just fine and is the same syntax.

But, in my proposal, this also works:

[env]
MY_HOME_CACHE="{{env.MY_HOME}}/cache"
MY_HOME="{{env.HOME}}/.myproj"

So, I think the simple use case is the same DX.

[rsyring]

I think your last post, at the same time, both identifies why you shouldn't use this syntax and also doubles-down on it. I'm not sure how to address that so I'm just going to move on. :)

Also, if there was a better design that I just hadn't thought of: I think someone else would've suggested it by now.

I don't know who defines "better" here but, by your own admission, there are plenty of people who are pushing back on this. Not just that its not better, but that its subjectively bad. The fact that you are abusing ordering in TOML is proof, IMO, that there is also an objective quality to this solution not being better. You are in the drivers seat so, in the end, you choose the implementation. But this design is not better.

You've built a great tool here, I'm grateful for it. I'd love to see the syntax changed. But, I think we've probably reached an impasse on this topic and its better for us to use our collective time elsewhere. You've probably come to the same conclusion. :)

If you ever want to revisit this, I propose a friendly challenge: setup the various cases you think the config file needs to handle. Or, if they are already all present in mise tests, I'll go there. I'll propose a syntax that accommodates all of them in a way that doesn't put an unreasonable burden on the implementation. Then we'll put them both out there and ask users what they think is "better" DX.

Thanks again, really, for your work on mise and this discussion.

[jdx]

My goal is not to minimize complaints or to make something with nice aesthetics—it's to make people productive. I'm certain there are syntaxes fewer people would complain about. These things are nearly always aligned but this is a rare case where I don't think they are.

[jdx]

I agree with you too, I think this topic has been discussed to a point there isn't much value in anymore. Unless someone thinks they've really cracked it, I would encourage folks to discuss the many other topics we could talk about that would benefit from discussion.

Keep in mind I've been thinking about this problem for almost an entire year now, so for me this is not a new thing.

[booniepepper]

One thing I'm seeing consistently in feedback is that no one is a fan of the churn that comes from changing the syntax... Naturally this kind of issue will only attract people dissatisfied with the syntactic choice, and have zero representation from anyone who likes it or doesn't care.

I suggest we listen to that consistent feedback and avoid needles churn first and foremost. Changing the syntax leads to churn and it should be avoided, simple enough.

How the human condition can start opposed to change itself and then conclude the solution may be more change with the same effect should boggle our minds with wonder.

[camsteffen]

"boggle our minds with wonder" LOL

[rsyring]

@jdx regarding complexity of implementation:

Task dependencies might be the second most complex part of the codebase IMO.

Fair enough. However, for the sake of discussion, I want to be clear that in my proposal there are two types of "dependencies":

1. Dependency: an environment plugin being dependent on a separate environment plugin or variable
2. Interpolation: A variable being dependent on another variable

Now, regarding Dependencies:

1. Tasks got dependencies because they make sense, even though the implementation was difficult
2. The dependency implementation exists. Presumably, it could be shared with, in code or in spirit, the dependencies of the various methods of getting environment variables (manual, scripts, plugins).

Regarding interpolation, I have a hard time seeing how its difficult to just do the resolution in a loop. If you are not wholly opposed to this idea, I'll put together a Python implementation as an example (and maybe prove myself wrong in the process).

[jdx]

The dependency implementation exists. Presumably, it could be shared with, in code or in spirit, the dependencies of the various methods of getting environment variables (manual, scripts, plugins).

This is not the case at all. The logic would help us get started. It is not something that could be shared.

[rsyring]

@jdx regarding

Even the one you showed works fine with my syntax

[env]
_.python.virtualenv = '.venv'
MY_VENV = '{{env.VIRTUAL_ENV}}'

[env._.some_plugin]
a='bunch'
of='settings'
that='this'
plugin='needs'
to='function'
venv='{{env.VIRTUALENV}}'}

[env]
FOO = '{{env.VAR_FROM_SOME_PLUGIN}}'

:(

The TOML specification requires that each key, including the keys for tables, be unique within its scope. I'm not being pedantic here. TOML was created over the course of years to address problems with existing config formats. It works for really sound foundational reasons. Your proposal violates two of them.

This is not intuitive.

[jdx]

it makes schema parsing worse and it prevents the ability to have values be strings (which in most cases is the only thing it would be).

e.g.: you can't do this:

[env]
source = "myscript.sh"

I also see _ as a helpful indicator to the user that these are directives. It makes it clear.

[rsyring]

I have some questions about this but, if all else failed, since my proposal is not dependent on ordering, create a separate but complementary key that still suggests the mechanics keeps the parser implementation simpler:

[env]
source = 'myscript.sh'

[env_from]
virtualenv = './venv'
mise.scripts.foo = 'foo.sh'
mise.scripts.bar = 'bar.sh'

The name for that section is debatable. Some other options off the top of my head. env_src, env_provider, env_plugins

[jdx]

Let's just move on from the resolving idea. You're correct it would work here, but I don't think it will work for us for other reasons.

I have considered this though, but I don't like it as much as what I already have:

[[env.kv]]
FOO = "bar"

[[env.source]]
run = "foo.sh"

[[env.source]]
run = "bar.sh"

that's more predictable and has fewer quirks to learn about, but I don't think it's worth it to lose [env] for simple use-cases. I know I'm mixing declarative and imperative behavior here but I've thought about the downsides and I'm not seeing any dealbreakers.

In general I don't like toml's table-of-tables and want to really try to avoid it except in cases I can't get around it. I think for a new developer, [[env.kv]] is going to be nonsense to them. I'd rather make [env] work even if it breaks down in more advanced use-cases and an alternative needs to be swapped in.

[rsyring]

You could avoid the table syntax by using keys when needed?

[env.kv]
FOO = "bar"

# Could also be `env.script.1`
# If only one exists, then you don't need the postifx?
[env.script.foo]
source = "foo.sh"

[env.plugin.virtualenv]
path = '.venv'

[env.script.bar] # or env.script.2
source = "bar.sh"

[env.kv.2]
FOO = "bar"

[jdx]

we discussed this in another thread, I hate the idea of forcing users to label scripts with names.

I also think the rust parser is not capable of handling this since it wouldn't be able to ensure the proper order.

[rsyring]

@jdx regarding your final assessment of my proposal

I'm saying the downsides aren't enough to warrant losing the declarative [env] syntax, losing the ability to have env vars depend on others, and certainly not worth bringing a dependency resolver into the mix.

1. Am I missing a reason why my proposed syntax makes [env] not declarative?
2. My proposal keeps the ability to have environment variable dependencies
3. Dependency resolver: IMO, this is debatable depending on the actual complexity needed compared to the value it provides. You already have dependencies implemented for tasks, so it can't be total out of the question, can it?

[jdx]

I think to summarize: environment variables are an iterative concept at their core. It's not something you can treat in a functional way because they are fundamentally unpure. The config should be iterative because what it is modeling is also iterative.

[rsyring]

I agree. And, FWIW, my proposal is iterative in this way. It lets you build up the environment iteratively from various sources, some of which may be dependent on others.

[rsyring]

For those that don't want to read through all these comments and threads but do care about the conclusion, I think this is it: #1523 (reply in thread)

[andrewthauer]

I'm coming in late to this discussion and just recently started using env. I will say that my first look at the _.directive syntax confused me a bit. I haven't actually used it for anything, but it make sense at a glance. In the interest of discussion and debate I came up with what I think is possibly a different proposal/idea for handling this. I can't say it jives with the code base, etc. I'm also not a toml expert so, but here it is:

Rename [env] to [environment] to avoid confusion with .env files. It's purpose is to "prepare" the environment. More analogue and inspired by direnv's .envrc files but adapted to toml (maybe). Within the environment section every top level key is a directive to do something specific (e.g. set an env var, load a dotenv file, source a file, etc.).

For example:

[environment]
var.FOO = "FOO"
var.BAR = "BAR"
var.FOOBAR = "{{env.FOO}}{{env.bar}}"

# load .env files 
dotenv = [".env"]

# source files (make an array to avoid `[[env.source]]`
source = ["./file-source"]

# etc.

The main things I think this helps address are:

• confusion and overuse of the term env in relation to .env (dotenv)
• standardizes everything as a directive
• creates a consistent flat structure for all environment things
• directives can be arrays to avoid the nested [[env.source]] problem

Couple downsides I see are:

• More breaking changes
• Each directive might have different values
• Referencing output of some directives may not be clear. {{env.FOO}} might be less obvious, but this could also be be {{var.FOO}} which might be more clear
• No idea if this helps keep the code to handle this consistent under the covers.

[rsyring]

The rename to "environment" is novel but that could actually be helpful for backwards compatibility purposes. Otherwise, I believe this is basically the same as: #1523 (reply in thread)

Well, maybe not due to the arrays.

[jdx]

directives are already arrays, that is not a solution to the problem that [[env]] addresses—you'd still need it with this proposal.

I don't think flipping directives so that var.FOO is better. It makes directives easier to understand—sure, but we'd lose the ability to have simple k/v which I don't want to give up. If we were going to go this route I would prefer env = [] over this.