environment variables are not parsed as JSON when missing specified env_prefix

[apowers313]

Reproduction code is here

My model is specified as:

class Config(BaseSettings):
    model_config = SettingsConfigDict(env_prefix='my_prefix_', env_file=".env")
    foo: list[str]

If I set an environment variable WITH a prefix it gets parsed as a list and works fine:

export my_prefix_foo='["testy"]'

If I set an environment variable WITHOUT a prefix it picks up the environment variable but interprets it as a string rather than a list and fails validation:

export foo='["testy"]'

Error is:

1 validation error for Config
foo
  Input should be a valid list [type=list_type, input_value='["testy"]', input_type=str]
    For further information visit https://errors.pydantic.dev/2.10/v/list_type

[hramezani]

pydantic-settings collects all the values in .env file and passes them to Pydantic at the end.

In your example, pydantic-settings removes the prefix from field name and parses my_prefix_foo field value to a list and passes it like {'foo': ['testy']} to pydantic. that's why you have a valid list.
this is not happening for foo and pydanti-settings consider foo as an extra value and passes it to pydantic like {'foo': '["testy"]'}.

probably a duplicate of #441

Here is the doc:
Pydantic settings loads all the values from dotenv file and passes it to the model, regardless of the model's env_prefix. So if you provide extra values in a dotenv file, whether they start with env_prefix or not, a ValidationError will be raised.

[apowers313]

Note that the problem exists for both OS environment variables and .env variables, I don't think this is just a .env parsing problem?

Also, I updated my model_config to use extra="forbid" to try and cause a validation error when I was about to trigger this footgun, but it still passes validation. Shouldn't that cause foo in my .env to be considered an extra variable and trigger an error?

[hramezani]

Note that the problem exists for both OS environment variables and .env variables, I don't think this is just a .env parsing problem?

I think it is only for dotenv. by the following code:

from pydantic_settings import BaseSettings, SettingsConfigDict


class Config(BaseSettings):
    model_config = SettingsConfigDict(env_prefix='my_prefix_')  # I remove the env_file config
    foo: list[str]

print(Config())

Here is the output:

> my_prefix_foo='["test", "test1"]' python /tmp/test8.py      
foo=['test', 'test1']
 > foo='["test", "test1"]' python /tmp/test8.py                        
Traceback (most recent call last):
  File "/tmp/test8.py", line 8, in <module>
    print(Config())
          ^^^^^^^^
  File "/Users/hasan/work/pydantic-settings/pydantic_settings/main.py", line 171, in __init__
    super().__init__(
  File "/Users/hasan/work/pydantic/pydantic/main.py", line 230, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 1 validation error for Config
foo
  Field required [type=missing, input_value={}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.10/v/missing

You can see I got a validation error in the second case that I removed the prefix.

Also, I updated my model_config to use extra="forbid" to try and cause a validation error when I was about to trigger this footgun, but it still passes validation. Shouldn't that cause foo in my .env to be considered an extra variable and trigger an error?

in case of dotenv as I mentioned in my previous comment, {'foo': '["testy"]'} will be passed to pydantic for validation. pydantic doesn't know about the prefix and only considers foo. So, that's the reason you won't get a validation error.

[apowers313]

Gotcha. You're right, this boils down to being a duplicate of #441