env_prefix falls back to env vars without prefix

[hofrob]

Using my example from #437: https://github.com/hofrob/pydantic-settings-env-prefix

If you uv run hello the result is:

hello='ABC'

This value is coming from the first line in .env: HELLO=ABC

I don't think the intention of env_prefix is to just fall back to whatever env var it can find that matches the variable without the prefix. If you have nested or complex settings with multiple API keys managed via different env_prefix values, you'll not want to use an API key with the wrong service. I'd consider this API key compromised.

See also this comment: #437 (comment)

[ericchansen]

I'm confused about env_prefix. When I use env_prefix, the original env var gets ignored. It will only load an env var if it it matches the name with the env_prefix.

[hofrob]

Yes, it works when there an env var exists with the prefix. But I think it should stay empty if it does not find the env var + prefix as opposed to fall back to an env var without the prefix.

• SOME_PREFIX_HELLO=FOO uv run hello: hello='FOO' ✔️
• HELLO=FOO uv run hello: hello='ABC' ❌
• uv run hello: hello='ABC' ❌

[ericchansen]

Thanks! I thought that setting env_prefix acted like an alias, such that in your example, either HELLO or SOME_PREFIX_HELLO should work. I guess the documentation is confusing me. It would be nice to get a clear answer from the devs as to what the intended behavior is.

I'm currently struggling with pydantic as well, but it seems like we're observing different behavior. In my use case, I was expecting the env var to get loaded from the raw variable or the variable name with the env_prefix (in other words, like I described above, where it would work like an alias). However, it ONLY works for the variable name with the env_prefix (the raw variable name doesn't work, unlike alias behavior). @hofrob, does this sounds like the opposite problem that you're having?

EDIT: I'm learning more. It seems the behavior is different if there's a default value. If there is a default value, then it will not check the variable name without the env_prefix. If there isn't a default value, it will check the variable name without the env_prefix.

I think the order of priority is:

1. env_prefix + var
2. default
3. var

[apowers313]

It seems like the intuition people have about .env is that gets treated the exact same way as the OS environment variables. Maybe it would make sense to merge both os.environ and python-dotenv into a single dict and then pass that into the config, rather than treating them differently?

[hofrob]

I'm not sure I follow. Are the prefixes treated differently depending on the source?

The main point about this issue is that pydantic-settings should not fall back in case there is no env var with the defined prefix.

[apowers313]

I thought they were being treated differently, but apparently there was a bug in my repro code. My bad, sorry for the confusion.

[vcc-ehemdal]

Not sure if this should be interpreted as a new bug, but here's my reproduction of this bug: https://github.com/vcc-ehemdal/pydantic_settings_env_prefix_bug

I'm using a list in this case since the issue is more severe since it crashes the application with a ValidationError

EDIT: I realize that this is the same test/issue as @apowers313's above.

[emilhem]

Here's a fork with a test capturing this fault (no fix though): https://github.com/emilhem/pydantic-settings/tree/dotenv_prefix_fix_list

I believe a check is needed here to check if the variable env_value is written to a env_name which has to be prefixed.

pydantic-settings/pydantic_settings/sources.py

Line 1060 in 65929cd

data[env_name] = env_value