pydantic.Json[dict[str, str]] | None raise error (BUG)

[aleksey925]

Description

The following code raise an error

Traceback (most recent call last):
  File "/Users/alekseipetrunnik/PycharmProjects/temp/temp6.py", line 14, in <module>
    s = Settings()
  File "/Users/alekseipetrunnik/PycharmProjects/temp/venv/lib/python3.10/site-packages/pydantic_settings/main.py", line 71, in __init__
    super().__init__(
  File "/Users/alekseipetrunnik/PycharmProjects/temp/venv/lib/python3.10/site-packages/pydantic/main.py", line 159, in __init__
    __pydantic_self__.__pydantic_validator__.validate_python(data, self_instance=__pydantic_self__)
pydantic_core._pydantic_core.ValidationError: 1 validation error for Settings
data
  JSON input should be string, bytes or bytearray [type=json_type, input_value={'key': 'value'}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.1/v/json_type

.env file

DATA='{"key":"value"}'

Example Code

from pydantic import Json, Field
from pydantic_settings import BaseSettings, SettingsConfigDict


class Settings(BaseSettings):
    data: Json[dict[str, str]] | None = Field(default=None)

    model_config = SettingsConfigDict(
        env_file='.env',
        env_file_encoding='utf-8',
    )


s = Settings()
print(s.data)
print(type(s.data))

pydantic version: 2.1.1\n        pydantic-core version: 2.4.0\n          pydantic-core build: profile=release pgo=false mimalloc=true\n                 install path: /Users/alekseipetrunnik/PycharmProjects/temp/venv/lib/python3.10/site-packages/pydantic\n               python version: 3.10.8 (main, Oct 29 2022, 22:54:57) [Clang 14.0.0 (clang-1400.0.29.102)]\n                     platform: macOS-13.4.1-arm64-arm-64bit\n     optional deps. installed: ['typing-extensions']

pydantic-settings 2.0.2

Selected Assignee: @dmontagu

[hramezani]

Thanks @aleksey925 for reporting this issue 🙏

pydantic-settings considers the union field as a complex field and it json parses it. That's why you get the error. because the value is already parsed inpydantic-settings and converted to a dict, then pydantic tries to json parse it again(as it is a Json) field. That's why you see this error message.

You can solve the error by changing the data: Json[dict[str, str]] | None = Field(default=None) to data: Json[dict[str, str]] = {}

[aleksey925]

Thanks for the help. Yes, I use almost this approach now. Maybe describe this behavior in the documentation? I think this will make life much easier.

Another solution:

from pydantic import Json, Field
from pydantic_settings import BaseSettings, SettingsConfigDict


class Settings(BaseSettings):
    data: Json[dict[str, str] | None] = Field(default='null')

    model_config = SettingsConfigDict(
        env_file='.env',
        env_file_encoding='utf-8',
    )


s = Settings()
print(s.data)
print(type(s.data))

[hramezani]

Yeah, I think we can document this behavior. I would be happy to review a PR if you want to work on it

[dmontagu]

@hramezani it might be reasonable to make Json[<some complex type>] be treated by pydantic-settings as equivalent to <some complex type>

[aleksey925]

@hramezani I rechecked the code with hint like this Json[dict[str, str] | None] and that doesn't work either.

As a result, the current implementation does not allow you to define a field with different possible schemas. I think this is a really important thing. In development, it is quite common to declare that a field can contain either a dictionary or None.
Maybe you could think about tweaking the validation/deserialization capabilities for this type?

[dmontagu]

@aleksey925 does the following not work for you?

from pydantic import Json, Field
from pydantic_settings import BaseSettings, SettingsConfigDict


class Settings(BaseSettings):
    data: dict[str, str] | None = Field(default=None)

    model_config = SettingsConfigDict(
        env_file='.env',
        env_file_encoding='utf-8',
    )


s = Settings()
print(s.data)
#> {'key': 'value'}
print(type(s.data))
#> <class 'dict'>

As @hramezani said, pydantic-settings is already automatically treating the type as JSON-encoded, so applying the Json[...] to one of the union arguments means that it will assume that item is doubly JSON-encoded. If you just drop the Json[...] from your annotation it works correctly for me, but I'm not sure if it's important for you to have that for other reasons. If this doesn't work for you, it would probably be helpful for us to understand why this doesn't work when proposing other solutions.

---

I'll also note that you can put the following in the .env file, which accounts for the double-decoding:

DATA='"{\\"key\\":\\"value\\"}"'

and loads as I think you expected. Obviously that's not great as a general solution though.

[hramezani]

@aleksey925 I've created a PR to fix the problem.
It would be great to test it