Clean up serialization warnings for invalid data in unions

[sydney-runkle]

Fix pydantic/pydantic#10363
Fix pydantic/pydantic#10344

This makes it such that we report all warnings associated with union serialization. This is similar to union validation, where the errors can be quite verbose. One example that clearly demonstrates the change is the following. We used to raise one error saying expected Union[A, B], got C (or something similar). Now we report on all warnings.

from pydantic import BaseModel, TypeAdapter

class A(BaseModel):
    a: int

class B(BaseModel):
    b: int

class C(BaseModel):
    c: int

ta = TypeAdapter(A | B)
print(ta.dump_python(C(c=1)))

"""
PydanticSerializationUnexpectedValue: Expected `A` but got `C` with value `C(c=1)` - serialized value may not be as expected
PydanticSerializationUnexpectedValue: Expected `B` but got `C` with value `C(c=1)` - serialized value may not be as expected
"""

I think there's still more to improve here - we should probably group the errors more intuitively, and use shared logic across much of the PydanticSerializationUnexpectedValue initialization steps. Also, in the long term, we should remove the fallback to the union serializer for tagged unions and jump straight to inference instead. Will probably retain this for backwards compatibility for a few minor versions as we iron out small issues like this one.

[codspeed-hq]

CodSpeed Performance Report

Merging #1449 will not alter performance

_{Comparing warning-fix (d24aad5) with main (16331d7)}

Summary

✅ 155 untouched benchmarks

[sydney-runkle]

Currently, we don't warn on tagged unions. I'd like to move us to a place where we eventually don't fall back to untagged union serialization for tagged unions, at which point we can add in warnings for tagged unions.

[sydney-runkle]

A few test cases here:

from typing import Tuple

import pydantic


class Model(pydantic.BaseModel):
    values: Tuple[float, ...]


model = Model(values=(0, 1))
print(pydantic.__version__, repr(model), model.model_dump())
model = model.model_copy(deep=True, update={"values": [1, 2]})
print(pydantic.__version__, repr(model), model.model_dump())
model = model.model_validate(model)
print(pydantic.__version__, repr(model), model.model_dump())

"""
Model(values=(0.0, 1.0)) {'values': (0.0, 1.0)}
/Users/programming/pydantic_work/pydantic/pydantic/main.py:387: UserWarning: Pydantic serializer warnings:
  Expected `tuple[float, ...]` but got `list` with value `[1, 2]` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_python(
Model(values=[1, 2]) {'values': [1, 2]}
Model(values=[1, 2]) {'values': [1, 2]}
"""

from pydantic import BaseModel, TypeAdapter

class A(BaseModel):
    a: int

class B(BaseModel):
    b: int

class C(BaseModel):
    c: int

ta = TypeAdapter(A | B)
print(ta.dump_python(C(c=1)))

"""
PydanticSerializationUnexpectedValue: Expected `A` but got `C` with value `C(c=1)` - serialized value may not be as expected
PydanticSerializationUnexpectedValue: Expected `B` but got `C` with value `C(c=1)` - serialized value may not be as expected
"""

from typing import Annotated, Literal, Union

from pydantic import BaseModel, Field


class Bar(BaseModel):
    discriminator_key: Literal["Bar"] = "Bar"
    name: str


class Foo(BaseModel):
    discriminator_key: Literal["Foo"] = "Foo"


class Container(BaseModel):

    prop: Annotated[Union[Bar, Foo], Field(..., discriminator="discriminator_key")]


bar = Bar.model_construct()
foo_child = Foo()
container = Container.model_construct(prop=bar)
print(bar.model_dump())
print(container.model_dump())

"""
PydanticSerializationUnexpectedValue: Expected 2 fields but got 1 for type `Bar` with value `Bar(discriminator_key='Bar')` - serialized value may not be as expected.
PydanticSerializationUnexpectedValue: Expected `Foo` but got `Bar` with value `Bar(discriminator_key='Bar')` - serialized value may not be as expected
"""

[sydney-runkle]

Admittedly is a bit redundant with existing code - would like to standardize how we raise serialization errors in pydantic-core for the v2.10 release :)

[sydney-runkle]

This case shows why we might want to do that - could probably minimize the duplicates here:

from typing import Annotated, FrozenSet, Union, Literal, List

from pydantic import BaseModel, PlainSerializer, IPvAnyAddress, Field
from ipaddress import IPv4Network

SortedFrozenNetworkSet = Annotated[
    FrozenSet[IPv4Network],
    PlainSerializer(
        lambda x: sorted(str(network) for network in x),
        return_type=FrozenSet[str],
    ),
]


class ModelA(BaseModel):
    type: Literal["A"]
    addresses: SortedFrozenNetworkSet


class ModelB(BaseModel):
    type: Literal["B"]
    hostname: str
    address: IPvAnyAddress


AnnotatedUnionType = Annotated[
    Union[ModelA, ModelB],
    Field(discriminator="type")
]


class CompositeModel(BaseModel):
    composite_field: AnnotatedUnionType


data = {"composite_field": {"type": "A", "addresses": ["192.168.1.0/24", "127.0.0.1"]}}
model = CompositeModel(**data)
print("Deserialized:", model.model_dump_json())

"""
PydanticSerializationUnexpectedValue: Expected `frozenset[str]` but got `list` with value `['127.0.0.1/32', '192.168.1.0/24']` - serialized value may not be as expected
PydanticSerializationUnexpectedValue: Expected `ModelB` but got `ModelA` with value `ModelA(type='A', addresse...twork('127.0.0.1/32')}))` - serialized value may not be as expected
Expected `frozenset[str]` but got `list` with value `['127.0.0.1/32', '192.168.1.0/24']` - serialized value may not be as expected
"""

[sydney-runkle]

I've opened #1454 and #1455 to hold us accountable to the suggestions made above.

[lukas503]

Unexpected Warning in Pydantic 2.9.2 with Discriminated Union and Overlapping Literal Names

I'm encountering a warning with the following code when using discriminated unions in Pydantic 2.9.2. While the code still behaves as expected, I am curious if the warning is intentional. Specifically, could the warning be due to multiple classes using the same Literal value for the name field? Is that correct usage? 🤔

from typing import Union, Literal, Annotated, Optional
from pydantic import BaseModel, Field, ConfigDict
import pydantic
print(pydantic.__version__)

class _BaseConfig(BaseModel):
    model_config = ConfigDict(
        extra="forbid"  # Don't allow extra values
    )

class A(_BaseConfig):
    c: int
    name: Literal["A"] = "A"
    
    def __repr__(self):
        return "ClassA"

class B(_BaseConfig):
    b: int
    name: Literal["B"] = "B"
    
    def __repr__(self):
        return "ClassB"

class C(_BaseConfig):
    c: int
    d: int
    name: Literal["A"] = "A"
        
    def __repr__(self):
        return "ClassC"

class D(_BaseConfig):
    c: int
    d: int
    name: Literal["B"] = "B"
    
    def __repr__(self):
        return "ClassD"

Var = Union[
    Annotated[A, Field(discriminator='name')], 
    Annotated[B, Field(discriminator='name')], 
    Annotated[C, Field(discriminator='name')], 
    Annotated[D, Field(discriminator='name')]
]

class T(BaseModel):
    a: Optional[Union[dict[str, Var], Var]] = None

model = T(a={"test": {"name": "A", "c": 1, "d": 2}})
print(model, model.model_dump(), T(**model.model_dump()))

model = T(a={"test": {"name": "A", "c": 1,}})
print(model, model.model_dump(), T(**model.model_dump()))

model = T(a={"test": {"name": "B", "b": 1,}})
print(model, model.model_dump(), T(**model.model_dump()))

model = T(a={"test": {"name": "B", "c": 1, "d": 5}})
print(model, model.model_dump(), T(**model.model_dump()))

model = T(a={"name": "B", "c": 1, "d": 2})
print(model, model.model_dump(), T(**model.model_dump()))

Output

2.9.2
a={'test': ClassC} {'a': {'test': {'c': 1, 'd': 2, 'name': 'A'}}} a={'test': ClassC}
a={'test': ClassA} {'a': {'test': {'c': 1, 'name': 'A'}}} a={'test': ClassA}
a={'test': ClassB} {'a': {'test': {'b': 1, 'name': 'B'}}} a={'test': ClassB}
a={'test': ClassD} {'a': {'test': {'c': 1, 'd': 5, 'name': 'B'}}} a={'test': ClassD}
a=ClassD {'a': {'c': 1, 'd': 2, 'name': 'B'}} a=ClassD
.../envs/dev_l1/lib/python3.9/site-packages/pydantic/main.py:390: UserWarning: Pydantic serializer warnings:
  PydanticSerializationUnexpectedValue: Expected `A` but got `C` with value `ClassC` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_python(
.../envs/dev_l1/lib/python3.9/site-packages/pydantic/main.py:390: UserWarning: Pydantic serializer warnings:
  PydanticSerializationUnexpectedValue: Expected `A` but got `B` with value `ClassB` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_python(
.../envs/dev_l1/lib/python3.9/site-packages/pydantic/main.py:390: UserWarning: Pydantic serializer warnings:
  PydanticSerializationUnexpectedValue: Expected `A` but got `D` with value `ClassD` - serialized value may not be as expected
  return self.__pydantic_serializer__.to_python(

My environment:

             pydantic version: 2.9.2
        pydantic-core version: 2.23.4
          pydantic-core build: profile=release pgo=false
                 install path: .../envs/dev_l1/lib/python3.9/site-packages/pydantic
               python version: 3.9.19 (main, May  6 2024, 19:43:03)  [GCC 11.2.0]
                     platform: Linux-6.5.0-1023-gcp-x86_64-with-glibc2.35
             related packages: typing_extensions-4.12.2 pyright-1.1.358

---

Question: Is the warning related to using the same literal value for multiple classes, or is this a bug in the Pydantic serialization process? Any insights into the cause of this behavior would be appreciated.

[davidhewitt]

@lukas503 your use of discriminators is wrong, see pydantic/pydantic#10482. Indeed they should not be overlapping. It looks like we can help protect against this case better.

[lukas503]

Thanks. Better protection sounds helpful 👍