[Feature] Dead letter structure for unhandled exceptions of KytosEvents indexed by their names

[viniarck]

Fixes #197

This dead letter in-memory structure is meant to provide a mechanism of last resort to allow re-injection of failed KytosEvent when unhandled exception happens on a listen_to handler (for instance, maximum retries reached, full outage of async events or even unexpected ones). This is meant to ensure production-grade error handling for errors that are transient. So far we don't have a full blown message broker (yet), so this structure fills that gap, as long as the kytosd is up an running it should be possible to list, reinject or delete these events. The underlying structure is an OrderedDict instead of a Queue to provide more flexibility when re-injecting and deleting arbitrary events.

This PR is on top of PR #203 (to facilitate development)

Description of the change

• Added an in-memory dead letter structure for unhandled exceptions of KytosEvents indexed by their names
• Added core endpoints for the dead letter structure:

   GET /api/kytos/core/dead_letter/?event_name=<name>
   PATCH /api/kytos/core/dead_letter/ (requires request body)
   DELETE /api/kytos/core/dead_letter/ (requires request body)

Practical example

• I've forced an intentional unhandled exception (in this case the error isn't transient though) on kytos/core.switch.new and kytos/core.switch.reconnected, if you list the endpoint, you'll see the events:

❯ http localhost:8181/api/kytos/core/dead_letter/
HTTP/1.0 200 OK
Access-Control-Allow-Origin: *
Content-Length: 753
Content-Type: application/json
Date: Tue, 12 Apr 2022 21:47:58 GMT
Server: Werkzeug/1.0.1 Python/3.9.12

{
    "kytos/core.switch.new": {
        "99e7d63c-1673-49e2-be63-8b7b1206663c": {
            "content": "{'switch': Switch('00:00:00:00:00:00:00:01')}",
            "id": "99e7d63c-1673-49e2-be63-8b7b1206663c",
            "name": "kytos/core.switch.new",
            "reinjections": 0,
            "timestamp": "2022-04-12T21:46:44"
        }
    },
    "kytos/core.switch.reconnected": {
        "10e70a1b-ac19-460b-ad16-51e53decde55": {
            "content": "{'switch': Switch('00:00:00:00:00:00:00:01')}",
            "id": "10e70a1b-ac19-460b-ad16-51e53decde55",
            "name": "kytos/core.switch.reconnected",
            "reinjections": 0,
            "timestamp": "2022-04-12T21:46:44"
        },
        "860b2ac1-baf4-4650-a365-2f7ca7db9d64": {
            "content": "{'switch': Switch('00:00:00:00:00:00:00:01')}",
            "id": "860b2ac1-baf4-4650-a365-2f7ca7db9d64",
            "name": "kytos/core.switch.reconnected",
            "reinjections": 0,
            "timestamp": "2022-04-12T21:46:44"
        }
    }
}

• You can try re-inject all kytos/core.switch.new events on the app kytos queue buffer to all subscribers again (or you could also selectively specify the KytosEvent ids):

echo '{"event_name": "kytos/core.switch.new", "kytos_queue_buffer": "app"}' | http PATCH localhost:8181/api/kytos/core/dead_letter/

HTTP/1.0 200 OK
Access-Control-Allow-Origin: *
Content-Length: 3
Content-Type: application/json
Date: Tue, 12 Apr 2022 21:48:38 GMT
Server: Werkzeug/1.0.1 Python/3.9.12

{}

• If you list again, notice that reinjections was incremented and it was reinjected on the app queue:

❯ http localhost:8181/api/kytos/core/dead_letter/
HTTP/1.0 200 OK
Access-Control-Allow-Origin: *
Content-Length: 753
Content-Type: application/json
Date: Tue, 12 Apr 2022 21:48:41 GMT
Server: Werkzeug/1.0.1 Python/3.9.12

{
    "kytos/core.switch.new": {
        "99e7d63c-1673-49e2-be63-8b7b1206663c": {
            "content": "{'switch': Switch('00:00:00:00:00:00:00:01')}",
            "id": "99e7d63c-1673-49e2-be63-8b7b1206663c",
            "name": "kytos/core.switch.new",
            "reinjections": 1,
            "timestamp": "2022-04-12T21:46:44"
        }
    },
    "kytos/core.switch.reconnected": {
        "10e70a1b-ac19-460b-ad16-51e53decde55": {
            "content": "{'switch': Switch('00:00:00:00:00:00:00:01')}",
            "id": "10e70a1b-ac19-460b-ad16-51e53decde55",
            "name": "kytos/core.switch.reconnected",
            "reinjections": 0,
            "timestamp": "2022-04-12T21:46:44"
        },
        "860b2ac1-baf4-4650-a365-2f7ca7db9d64": {
            "content": "{'switch': Switch('00:00:00:00:00:00:00:01')}",
            "id": "860b2ac1-baf4-4650-a365-2f7ca7db9d64",
            "name": "kytos/core.switch.reconnected",
            "reinjections": 0,
            "timestamp": "2022-04-12T21:46:44"
        }
    }
}

kytos $> 2022-04-12 18:48:38,159 - ERROR [kytos.core.helpers] (ThreadPoolExecutor-0_6) listen_to handler: <function Main.on_new_switch2 at 0x7fc9086183a0>, args: (<Main(topology, stopp
ed 140500912821824)>, KytosEvent('kytos/core.switch.new', {'switch': Switch('00:00:00:00:00:00:00:01')})), exception: <class 'ValueError'>: crashed2022-04-12 18:48:38,165 - INFO [kytos.napps.kytos/flow_manager] (Thread-41) Send FlowMod from request dpid: 00:00:00:00:00:00:00:01, command: add, force: False, flows_dict: {'flows': [
{'priority': 1000, 'table_id': 0, 'cookie': 12321848580485677057, 'match': {'dl_type': 35020, 'dl_vlan': 3799}, 'actions': [{'action_type': 'output', 'port': 4294967293}]}]}
2022-04-12 18:48:38,177 - INFO [kytos.napps.kytos/flow_manager] (ThreadPoolExecutor-0_7) Flow saved in kytos.flow.persistence.51953dfb51854c5490dcfcbc83ee8390
2022-04-12 18:48:38,179 - INFO [kytos.napps.kytos/flow_manager] (ThreadPoolExecutor-0_5) Flow saved in kytos.flow.persistence.51953dfb51854c5490dcfcbc83ee8390
kytos $>

[viniarck]

Thanks for reviewing, Antonio