Add support for serializing Enum values automatically in API Gateway Event Handler #519

marcioemiranda · 2021-07-10T23:10:13Z

If a route function returns a dict containing an Enum, the json encoder will fail.

Expected Behavior

Since the Response was not returned, API Gateway event handler will create one. It calls json.dumps to serialize the dict using its own encoder.
The expected json output would transform the Enum in its value.

Current Behavior

The current behavior is that the encoder throws a TypeError saying Enum is not serializable.

Possible Solution

My suggestion is to check if the object is an Enum and output its value.
The example bellow handle Enum and also a set

class CustomEncoder(JSONEncoder):
    def default(self, o):
        if isinstance(o, Enum):
            return o.value
        try:
            iterable = iter(o)
        except TypeError:
            pass
        else:
            return list(iterable)    
        # Let the base class default method raise the TypeError    
        return JSONEncoder.default(self, o)

For now the workaround is to always create a Response and call json.dumps with my own custom encoder

Steps to Reproduce (for bugs)

Try the following code

from enum import Enum

class Color(Enum): 
   RED =1
   BLUE = 2

@app.get("/colors")
def get_color() -> Dict:
   return {
      "color": Color.RED,
      "variations": {"light", "dark"} 
   }

Environment

Lambda Power Tools version 1.17.1
Running with PyTest

../../../../Library/Caches/pypoetry/virtualenvs/te-admin-api-y-cvF00h-py3.8/lib/python3.8/site-packages/aws_lambda_powertools/tracing/tracer.py:313: in decorate
    response = lambda_handler(event, context, **kwargs)
_admin_controller.py:273: in handler
    return app.resolve(event, context)
../../../../Library/Caches/pypoetry/virtualenvs/te-admin-api-y-cvF00h-py3.8/lib/python3.8/site-packages/aws_lambda_powertools/event_handler/api_gateway.py:418: in resolve
    return self._resolve().build(self.current_event, self._cors)
../../../../Library/Caches/pypoetry/virtualenvs/te-admin-api-y-cvF00h-py3.8/lib/python3.8/site-packages/aws_lambda_powertools/event_handler/api_gateway.py:450: in _resolve
    return self._call_route(route, match.groupdict())
../../../../Library/Caches/pypoetry/virtualenvs/te-admin-api-y-cvF00h-py3.8/lib/python3.8/site-packages/aws_lambda_powertools/event_handler/api_gateway.py:478: in _call_route
    return ResponseBuilder(self._to_response(route.func(**args)), route)
../../../../Library/Caches/pypoetry/virtualenvs/te-admin-api-y-cvF00h-py3.8/lib/python3.8/site-packages/aws_lambda_powertools/event_handler/api_gateway.py:497: in _to_response
    body=json.dumps(result, separators=(",", ":"), cls=Encoder),
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/__init__.py:234: in dumps
    return cls(
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py:199: in encode
    chunks = self.iterencode(o, _one_shot=True)
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py:257: in iterencode
    return _iterencode(o, 0)
../../../../Library/Caches/pypoetry/virtualenvs/te-admin-api-y-cvF00h-py3.8/lib/python3.8/site-packages/aws_lambda_powertools/shared/json_encoder.py:16: in default
    return super().default(obj)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <aws_lambda_powertools.shared.json_encoder.Encoder object at 0x7f915fbaea90>
o = <Region.CENTRAL: 6>

    def default(self, o):
        """Implement this method in a subclass such that it returns
        a serializable object for ``o``, or calls the base implementation
        (to raise a ``TypeError``).
    
        For example, to support arbitrary iterators, you could
        implement default like this::
    
            def default(self, o):
                try:
                    iterable = iter(o)
                except TypeError:
                    pass
                else:
                    return list(iterable)
                # Let the base class default method raise the TypeError
                return JSONEncoder.default(self, o)
    
        """
>       raise TypeError(f'Object of type {o.__class__.__name__} '
                        f'is not JSON serializable')
E       TypeError: Object of type Region is not JSON serializable

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/json/encoder.py:179: TypeError

The text was updated successfully, but these errors were encountered:

heitorlessa · 2021-07-16T19:28:45Z

Hey @marcioemiranda, thanks for raising the issue.

This is not a bug though but expected behaviour as Enum are not serialized - I'm changing it to a feature request instead and wait for other customers to chime in with 👍.

You'd have the same behaviour when using other web frameworks like Flask for example:

from flask import jsonify, Flask
from enum import Enum


class Color(Enum):
    RED = 1
    BLUE = 2


app = Flask(__name__)


@app.route("/colors")
def get_color():
    return jsonify({"color": Color.RED, "variations": {"light", "dark"}})

[2021-07-16 21:22:13,054] ERROR in app: Exception on /colors [GET]
Traceback (most recent call last):
  File "/Users/lessa/.pyenv/versions/3.8.2/envs/work3.8/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/lessa/.pyenv/versions/3.8.2/envs/work3.8/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/lessa/.pyenv/versions/3.8.2/envs/work3.8/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/Users/lessa/.pyenv/versions/3.8.2/envs/work3.8/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/Users/lessa/.pyenv/versions/3.8.2/envs/work3.8/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/Users/lessa/.pyenv/versions/3.8.2/envs/work3.8/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/Users/lessa/DEV/pt-issue-520/app.py", line 15, in get_color
    return jsonify({"color": Color.RED, "variations": {"light", "dark"}})
  File "/Users/lessa/.pyenv/versions/3.8.2/envs/work3.8/lib/python3.8/site-packages/flask/json/__init__.py", line 370, in jsonify
    dumps(data, indent=indent, separators=separators) + "\n",
  File "/Users/lessa/.pyenv/versions/3.8.2/envs/work3.8/lib/python3.8/site-packages/flask/json/__init__.py", line 211, in dumps
    rv = _json.dumps(obj, **kwargs)
  File "/Users/lessa/.pyenv/versions/3.8.2/lib/python3.8/json/__init__.py", line 234, in dumps
    return cls(
  File "/Users/lessa/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/Users/lessa/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/Users/lessa/.pyenv/versions/3.8.2/envs/work3.8/lib/python3.8/site-packages/flask/json/__init__.py", line 100, in default
    return _json.JSONEncoder.default(self, o)
  File "/Users/lessa/.pyenv/versions/3.8.2/lib/python3.8/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Color is not JSON serializable

marcioemiranda · 2021-07-16T19:52:55Z

@heitorlessa correct, it's a feature request. Meanwhile I am using a custom encoder. It would be nice though to have types such as Enum and set handled by LPT json encoder.

ran-isenberg · 2021-07-18T18:08:47Z

Enums are great but when you return a value for the handler to return back to the user, it must be a json-able value. Instead of Color.RED, write Color.RED.value or str(Color.red). Not sure what kind of feature is required here.

marcioemiranda · 2021-07-21T02:40:20Z

@risenberg-cyberark, thanks for the comment.

I was thinking something similar to this JSON encoder
https://pypi.org/project/orjson/

It handles types such as Dataclass, Enum, datetime, Decimal, UUID and others. Other cool features in the roadmap, such as filtering attributes of a Dataclass in the encoding process.

The ApiGatewayResolver can wrap the value returned by a handler in a Response object if it can handle the serialization, otherwise I have to use a custom encoder and create the Response myself. This is totally fine, but creates some boilerplate code. It would be nicer if ApiGatewayResolver json encoder could handle more types natively.

Other alternatives could be:

Let one specify an alternative json encoder in ApiGatewayResolver configuration
Let one specify a default function to handle the encode of non-supported types in ApiGatewayResolver configuration

michaelbrewer · 2021-07-24T00:02:21Z

@marcioemiranda should be easy to implement, but i would agree with @heitorlessa about see what the consensus should be on this. I can put up a draft PR to see what the impact would be

Let one specify an alternative json encoder in ApiGatewayResolver configuration

Otherwise alternative option 1, but be a good solution to allow for more flexibility.

michaelbrewer · 2021-07-24T04:26:18Z

@marcioemiranda you can also use IntEnum:

>>> from enum import IntEnum
>>> import json
>>>
>>>
>>> class Shape(IntEnum):
...     Circle = 1
...     Square = 2
...
...
>>> x = {"square": Shape.Square}
>>>
>>> print(json.dumps(x))
{"square": 2}

ran-isenberg · 2021-07-24T04:32:41Z

What we usually do it to define a schema for input validation and output validation in a lambda. We define it using Pydantic.
We return the main handler an output object, so all it needs to is to call the object's json() or dict() function which does the serialisation for it. This way I know that i'm returning a valid output back the api gw.

see https://pydantic-docs.helpmanual.io/usage/exporting_models/

michaelbrewer · 2021-07-24T04:34:37Z

@risenberg-cyberark haha, but this is not for Pydantic.

This works from for me:

from enum import IntEnum

class Color(IntEnum): 
   RED =1
   BLUE = 2

@app.get("/colors")
def get_color() -> Dict:
   return {
      "color": Color.RED,
      "variations": {"light", "dark"} 
   }

michaelbrewer · 2021-07-24T04:41:19Z

A full example using IntEnum (and a List):

from enum import IntEnum
from aws_lambda_powertools.event_handler import ApiGatewayResolver

class Color(IntEnum):
    RED = 1
    BLUE = 2

app = ApiGatewayResolver()

@app.get("/colors")
def get_color():
    return {
        "color": Color.RED,
        "variations": ["light", "dark"],
    }

print(app({"httpMethod": "GET", "path": "/colors"}, None))

# Output
{'statusCode': 200, 'headers': {'Content-Type': 'application/json'}, 'body': '{"color":1,"variations":["light","dark"]}', 'isBase64Encoded': False}

michaelbrewer · 2021-07-24T04:45:28Z

@marcioemiranda

Alternatively, if you want a custom one now you can always do:

import json
from enum import Enum
from json import JSONEncoder
from aws_lambda_powertools.event_handler import ApiGatewayResolver
from aws_lambda_powertools.event_handler.api_gateway import Response


class CustomEncoder(JSONEncoder):
    def default(self, o):
        if isinstance(o, Enum):
            return o.value
        try:
            iterable = iter(o)
        except TypeError:
            pass
        else:
            return list(iterable)
        # Let the base class default method raise the TypeError
        return JSONEncoder.default(self, o)

class Color(Enum):
    RED = 1
    BLUE = 2

app = ApiGatewayResolver()

@app.get("/colors")
def get_color() -> Response:
    return Response(
        200,
        "application/json",
        json.dumps(
            {
                "color": Color.RED,
                "variations": {"light", "dark"},
            },
            cls=CustomEncoder,
        ),
    )

print(app({"httpMethod": "GET", "path": "/colors"}, None))

# Output
{'statusCode': 200, 'headers': {'Content-Type': 'application/json'}, 'body': '{"color": 1, "variations": ["light", "dark"]}', 'isBase64Encoded': False}

marcioemiranda · 2021-07-26T13:08:01Z

@michaelbrewer this is exactly what I am doing now since Enum is not the only type that is not supported.
The feature request was about not having to create the Response object and let ApiGatewayResolver create it for me. This is currently supported, but then I get the problem with JSON encoder.
Creating a Response for each handler generates some boilerplate code. I could define my custom JSON encoder in just one place instead.

michaelbrewer · 2021-07-26T16:03:45Z

sure @marcioemiranda i am just putting out short term solutions that can work:

Using IntEnum
Using Response and use your own JSON encoder

I will defer the decision on how we implement a solution for the handler to @heitorlessa

heitorlessa · 2021-07-26T19:00:48Z

Happy with accepting a custom JSON Encoder in the constructor. we do that in Logger today in a simpler fashion to support orjson etc. https://awslabs.github.io/aws-lambda-powertools-python/latest/core/logger/#bring-your-own-json-serializer That way, if you end up using something else afterwards you can bring the encoder with you too and it’ll just work (tm)

…

On Mon, 26 Jul 2021 at 18:03, Michael Brewer ***@***.***> wrote: sure @marcioemiranda <https://github.com/marcioemiranda> i am just putting out short term solutions that can work: 1. Using IntEnum 2. Using Response and use your own JSON encoder I will defer the decision on how we implement a solution for the handler to @heitorlessa <https://github.com/heitorlessa> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#519 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBAXOAN3XM3DGHJXRRLTZWBO3ANCNFSM5AEVL2CA> .

michaelbrewer · 2021-07-26T19:02:52Z

@heitorlessa so no point for adding a deserialization support for Enum? Considering IntEnum is already natively supported and would make the original example, i am not sure if this is a big enough, but i am open either way.

heitorlessa · 2021-07-26T19:08:38Z

Yep I’d say not. Best to support a custom encoder so they can not only bring support for what they’d like to deserialize/serialize - Enums, dataclasses, Classes, binary data - but also faster implementations like orjson if they must. That would be a conscious choice and something they can reuse elsewhere in the future too

…

On Mon, 26 Jul 2021 at 21:03, Michael Brewer ***@***.***> wrote: @heitorlessa <https://github.com/heitorlessa> so no point for adding a deserialization support for Enum? Considering IntEnum is already natively supported and would make the original example, i am not sure if this is a big enough, but i am open either way. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#519 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAZPQBGAYMJ5HMVVSNQ7BLTTZWWOPANCNFSM5AEVL2CA> .

michaelbrewer · 2021-07-27T04:00:53Z

Ok @heitorlessa @marcioemiranda i will have a look at this next.

@marcioemiranda would you be able to have a review of this when it is up as a PR.

marcioemiranda · 2021-07-27T12:53:07Z

@michaelbrewer , sure.

heitorlessa · 2021-08-10T19:20:16Z

This is coming up tomorrow morning (EMEA timezone) in 1.19.0

heitorlessa · 2021-08-11T08:48:31Z

Now released as part of 1.19.0: https://github.com/awslabs/aws-lambda-powertools-python/releases/tag/v1.19.0

marcioemiranda added bug Something isn't working triage Pending triage from maintainers labels Jul 10, 2021

heitorlessa added area/event_handlers feature-request feature request and removed bug Something isn't working triage Pending triage from maintainers labels Jul 16, 2021

heitorlessa changed the title ~~JSON encoder to support types such as Enum inside a Dict in API Gateway Event Handler~~ Add support for serializing Enum values automatically in API Gateway Event Handler Jul 16, 2021

heitorlessa added the pending-release Fix or implementation already in dev waiting to be released label Aug 10, 2021

heitorlessa closed this as completed Aug 11, 2021

heitorlessa removed the pending-release Fix or implementation already in dev waiting to be released label Aug 11, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for serializing Enum values automatically in API Gateway Event Handler #519

Add support for serializing Enum values automatically in API Gateway Event Handler #519

marcioemiranda commented Jul 10, 2021

heitorlessa commented Jul 16, 2021

marcioemiranda commented Jul 16, 2021

ran-isenberg commented Jul 18, 2021

marcioemiranda commented Jul 21, 2021

michaelbrewer commented Jul 24, 2021

michaelbrewer commented Jul 24, 2021

ran-isenberg commented Jul 24, 2021

michaelbrewer commented Jul 24, 2021

michaelbrewer commented Jul 24, 2021 •

edited

Loading

michaelbrewer commented Jul 24, 2021 •

edited

Loading

marcioemiranda commented Jul 26, 2021

michaelbrewer commented Jul 26, 2021

heitorlessa commented Jul 26, 2021 via email

michaelbrewer commented Jul 26, 2021

heitorlessa commented Jul 26, 2021 via email

michaelbrewer commented Jul 27, 2021

marcioemiranda commented Jul 27, 2021

heitorlessa commented Aug 10, 2021

heitorlessa commented Aug 11, 2021

Add support for serializing Enum values automatically in API Gateway Event Handler #519

Add support for serializing Enum values automatically in API Gateway Event Handler #519

Comments

marcioemiranda commented Jul 10, 2021

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Environment

heitorlessa commented Jul 16, 2021

marcioemiranda commented Jul 16, 2021

ran-isenberg commented Jul 18, 2021

marcioemiranda commented Jul 21, 2021

michaelbrewer commented Jul 24, 2021

michaelbrewer commented Jul 24, 2021

ran-isenberg commented Jul 24, 2021

michaelbrewer commented Jul 24, 2021

michaelbrewer commented Jul 24, 2021 • edited Loading

michaelbrewer commented Jul 24, 2021 • edited Loading

marcioemiranda commented Jul 26, 2021

michaelbrewer commented Jul 26, 2021

heitorlessa commented Jul 26, 2021 via email

michaelbrewer commented Jul 26, 2021

heitorlessa commented Jul 26, 2021 via email

michaelbrewer commented Jul 27, 2021

marcioemiranda commented Jul 27, 2021

heitorlessa commented Aug 10, 2021

heitorlessa commented Aug 11, 2021

michaelbrewer commented Jul 24, 2021 •

edited

Loading

michaelbrewer commented Jul 24, 2021 •

edited

Loading