Bug: APIGatewayRestResolver(enable_validation=True) incompatible with from __future__ import annotations

[jmahlik]

Expected Behaviour

Using the data validation feature of the event resolvers doesn't appear compatible with from __future__ import annotations. There's undefined refs when trying to rebuild the model in the TypeAdapter.

It would be ideal to import annotations so type defs are ignored at runtime. Additionally, when using boto stubs they really shouldn't be shipped at runtime since they can be pretty large.

Current Behaviour

See details for stack trace.

Traceback (most recent call last):
  File "\venv\Lib\site-packages\pydantic\type_adapter.py", line 277, in _init_core_attrs
    self._core_schema = _getattr_no_parents(self._type, '__pydantic_core_schema__')
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\pydantic\type_adapter.py", line 119, in _getattr_no_parents
    raise AttributeError(attribute)
AttributeError: __pydantic_core_schema__

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "\venv\Lib\site-packages\pydantic\_internal\_generate_schema.py", line 724, in _resolve_forward_ref
    obj = _typing_extra.eval_type_backport(obj, globalns=self._types_namespace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\pydantic\_internal\_typing_extra.py", line 264, in eval_type_backport
    return typing._eval_type(  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Python\Python312\Lib\typing.py", line 415, in _eval_type
    return t._evaluate(globalns, localns, type_params, recursive_guard=recursive_guard)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "Python\Python312\Lib\typing.py", line 947, in _evaluate
    eval(self.__forward_code__, globalns, localns),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 1, in <module>
NameError: name 'Input' is not defined. Did you mean: 'input'?

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "\f.py", line 44, in <module>
    print(app.resolve(event, {}))
          ^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\aws_lambda_powertools\event_handler\api_gateway.py", line 2100, in resolve
    response = self._resolve().build(self.current_event, self._cors)
               ^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\aws_lambda_powertools\event_handler\api_gateway.py", line 2209, in _resolve
    return self._call_route(route, route_keys)  # pass fn args
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\aws_lambda_powertools\event_handler\api_gateway.py", line 2316, in _call_route
    route(router_middlewares=self._router_middlewares, app=self, route_arguments=route_arguments),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\aws_lambda_powertools\event_handler\api_gateway.py", line 450, in __call__
    return self._middleware_stack(app)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\aws_lambda_powertools\event_handler\api_gateway.py", line 1436, in __call__
    return self.current_middleware(app, self.next_middleware)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\aws_lambda_powertools\event_handler\middlewares\base.py", line 121, in __call__
    return self.handler(app, next_middleware)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\aws_lambda_powertools\event_handler\middlewares\openapi_validation.py", line 80, in handler
    route.dependant.path_params,
    ^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\aws_lambda_powertools\event_handler\api_gateway.py", line 499, in dependant
    self._dependant = get_dependant(path=self.openapi_path, call=self.func, responses=self.responses)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\aws_lambda_powertools\event_handler\openapi\dependant.py", line 192, in get_dependant
    param_field = analyze_param(
                  ^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\aws_lambda_powertools\event_handler\openapi\params.py", line 965, in analyze_param
    field = _create_model_field(field_info, type_annotation, param_name, is_path_param)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\aws_lambda_powertools\event_handler\openapi\params.py", line 1098, in _create_model_field
    return create_response_field(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\aws_lambda_powertools\event_handler\openapi\params.py", line 1067, in create_response_field
    return ModelField(**kwargs)  # type: ignore[arg-type]
           ^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 6, in __init__
  File "\venv\Lib\site-packages\aws_lambda_powertools\event_handler\openapi\compat.py", line 86, in __post_init__
    self._type_adapter: TypeAdapter[Any] = TypeAdapter(
                                           ^^^^^^^^^^^^
  File "\venv\Lib\site-packages\pydantic\type_adapter.py", line 264, in __init__
    self._init_core_attrs(rebuild_mocks=False)
  File "\venv\Lib\site-packages\pydantic\type_adapter.py", line 142, in wrapped
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\pydantic\type_adapter.py", line 284, in _init_core_attrs
    self._core_schema = _get_schema(self._type, config_wrapper, parent_depth=self._parent_depth)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\pydantic\type_adapter.py", line 102, in _get_schema
    schema = gen.generate_schema(type_)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\pydantic\_internal\_generate_schema.py", line 512, in generate_schema
    schema = self._generate_schema_inner(obj)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\pydantic\_internal\_generate_schema.py", line 768, in _generate_schema_inner
    return self._annotated_schema(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\pydantic\_internal\_generate_schema.py", line 1818, in _annotated_schema
    source_type, *annotations = self._get_args_resolving_forward_refs(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\pydantic\_internal\_generate_schema.py", line 746, in _get_args_resolving_forward_refs
    args = tuple([self._resolve_forward_ref(a) if isinstance(a, ForwardRef) else a for a in args])
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "\venv\Lib\site-packages\pydantic\_internal\_generate_schema.py", line 726, in _resolve_forward_ref
    raise PydanticUndefinedAnnotation.from_name_error(e) from e
pydantic.errors.PydanticUndefinedAnnotation: name 'Input' is not defined

For further information visit https://errors.pydantic.dev/2.8/u/undefined-annotation

Code snippet

# Works if removed
from __future__ import annotations

import json
from typing import TYPE_CHECKING

from aws_lambda_powertools.event_handler import APIGatewayRestResolver
from pydantic import BaseModel

if TYPE_CHECKING:
    # Actual use case imports boto stubs
    # These are giant pyi files that shouldn't be shipped at runtime
    from typing import Iterable


class Input(BaseModel):
    email: str


# Tried this as well
# Input = ForwardRef("Input")


class Output(BaseModel):
    response: str


app = APIGatewayRestResolver(enable_validation=True)


@app.post("/hello")
def hello(event: Input) -> Output:
    return Output(response=f"Hello {event.email}")


def func(a: Iterable[int]): ...


event = {
    "path": "/hello",
    "httpMethod": "POST",
    "requestContext": {
        "requestId": "227b78aa-779d-47d4-a48e-ce62120393b8",
    },
    "body": json.dumps(
        {
            "email": "hello@test.com",
        }
    ),
}

print(app.resolve(event, {}))

Possible Solution

The closest thing I could find was this fastapi issue fastapi/fastapi#10007. I've tried Input = ForwardRef("Input") from the pydantic docs but that doesn't appear to resolve the issue.

One workaround is to stringify the annotations not required at runtime i.e.

def func(a: "Iterable[int]"): 
    ...

This isn't ideal but it does work.

Steps to Reproduce

Run the file.

Environment:

aws-lambda-powertools    2.43.1
pydantic                             2.8.2
pydantic_core                    2.20.1

Powertools for AWS Lambda (Python) version

latest

AWS Lambda function runtime

3.12

Packaging format used

PyPi

Debugging logs

No response

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

[leandrodamascena]

Looking at this now.

[leandrodamascena]

Hey @jmahlik! I can confirm that we have something weird here when we try to use TypeAdapter to convert the model. I know that Pydantic has some limitations when working with __future__ annotations but I'll need a time to see what is going on with our integration.

It would be ideal to import annotations so type defs are ignored at runtime. Additionally, when using boto stubs they really shouldn't be shipped at runtime since they can be pretty large.

I totally agree that you shouldn't ship boto stubs to production, they are more suitable for development environments. In the meantime, you can use quotes ("Iterable[int]") for forward references and avoid issues.

Thanks.

[sthulb]

@jmahlik Thanks, we're pausing new additions for a couple of weeks whilst we work on v3, once this is released we'll get back to working on features and bugs :D

[yumatsuchiya]

I've been struggling for a long time to find this out...

The sample code below is from the doc, and it does not work with from __future__ import annotations added at the beginning of the file.

I use Ruff as a linter/formatter, and Ruff wants me to add from __future__ import annotations when I write something like dict[str, Any]. So I did so, and enable_validation=True started to act up.

from typing import Optional

import requests
from pydantic import BaseModel, Field
from typing_extensions import Annotated

from aws_lambda_powertools import Logger, Tracer
from aws_lambda_powertools.event_handler import APIGatewayRestResolver
from aws_lambda_powertools.event_handler.openapi.params import Path
from aws_lambda_powertools.logging import correlation_paths
from aws_lambda_powertools.utilities.typing import LambdaContext

tracer = Tracer()
logger = Logger()
app = APIGatewayRestResolver(enable_validation=True)


class Todo(BaseModel):
    userId: int
    id_: Optional[int] = Field(alias="id", default=None)
    title: str
    completed: bool


@app.get("/todos/<todo_id>")
@tracer.capture_method
def get_todo_by_id(todo_id: Annotated[int, Path(lt=999)]) -> Todo:  
    todo = requests.get(f"https://jsonplaceholder.typicode.com/todos/{todo_id}")
    todo.raise_for_status()

    return todo.json()


@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_HTTP)
@tracer.capture_lambda_handler
def lambda_handler(event: dict, context: LambdaContext) -> dict:
    return app.resolve(event, context)

[leandrodamascena]

Just a quick update: I’ll be focusing on this issue this week and will provide some updates soon.

[rafrafek]

I encountered this issue yesterday, and it was very difficult for me to identify the root cause.

Would it be possible for powertools-lambda-python to detect the use of from __future__ import annotations and raise an exception as a temporary workaround until a proper fix is implemented?