[Feature] Elasticsearch APM

[viniarck]

Fixes #207

This PR is on top of PR #206.

Description of the change

Elasticsearch APM (application performance monitoring) integration

• New --apm configuration option that supports elasticsearch APM (Application Performance Monitoring)
• kytosd Elastic APM integration provides instrumentation for MongoDB, Flask, requests and KytosEvent
• @begin_span decorator for on-demand APM custom functions/methods instrumentation
• Augmented docker-compose.yml to also spin up Elastsearch, Kibana and APM server with authentication
• KytosEvent now optionally supports a trace_parent argument for APM distributed tracing to also instrument and correlate KytosEvent

Practical Examples

1. KytosEvent distributed tracing (passing trace_parent to subsequent events) involving multiple NApps (flow_manager and topology), notice that you can correlate all the events including instrumented IO operations.

2. mef_eline leveraging the on-demand @begin_span decorator to further instrument underlying methods/functions:

Trying out locally

• docker-compese up
• kytosd -f -E --database mongodb --apm es
• Open Kibana's web page http://localhost:5601/app/apm/traces on your browser and authenticate with the credentials (check out on docker-compose or overwrite the values)

[italovalcy]

Looks good to me! Thank you for bringing this powerful tool to Kytos-NG. It will certainly be very useful. Just left a comment regarding the kytos.conf file and usage of APM parameters there (it looks like Mongo DB has a similar issue, as described in #217)

[italovalcy]

I didn't see how those configuration variables are being loaded. I believe they are just being loaded from host environment variables, correct?

I've tested defining those variables in the config (kytos.conf) - but not in the host environment variables - and they seems to be ignored:

root@2b16de904ed4:/# grep -A 6 'APM backend.' /etc/kytos/kytos.conf
# APM backend. Supported values: es (elasticsearch, don't use quotes in the string)
apm = es
# The following environment variables should be accordingly if you're using elasticsearch:
ELASTIC_APM_URL=http://apm-server:8200
# Optionally, these environment variables can also be set:
# ELASTIC_APM_SERVICE_NAME=kytos
# ELASTIC_APM_SECRET_TOKEN=

root@2b16de904ed4:/# kytosd -f -E
Web update - Web UI was not updated
2022-05-19 17:36:12,981 - INFO [kytos.core.logs] (MainThread) Logging config file "//etc/kytos/logging.ini" loaded successfully.
2022-05-19 17:36:12,981 - INFO [kytos.core.db] (MainThread) Starting DB connection
2022-05-19 17:36:12,984 - INFO [kytos.core.db] (MainThread) Trying to run 'hello' command on MongoDB...
2022-05-19 17:36:13,010 - INFO [kytos.core.db] (MainThread) Ran 'hello' command on MongoDB successfully. It's ready!
2022-05-19 17:36:13,026 - WARNING [urllib3.connectionpool] (eapm event processor thread) Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9a9d39190>: Failed to establish a new connection: [Errno 111] Connection refused')': /
2022-05-19 17:36:13,027 - WARNING [urllib3.connectionpool] (eapm event processor thread) Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9a9d39310>: Failed to establish a new connection: [Errno 111] Connection refused')': /
2022-05-19 17:36:13,030 - WARNING [urllib3.connectionpool] (eapm event processor thread) Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9a84ac430>: Failed to establish a new connection: [Errno 111] Connection refused')': /
2022-05-19 17:36:13,032 - WARNING [elasticapm.transport.http] (eapm event processor thread) HTTP error while fetching server information: HTTPConnectionPool(host='localhost', port=8200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9a84acc70>: Failed to establish a new connection: [Errno 111] Connection refused'))
...
      _          _
     | |        | |
     | | ___   _| |_ ___  ___
     | |/ / | | | __/ _ \/ __|
     |   <| |_| | || (_) \__ \
     |_|\_\__,  |\__\___/|___/
            __/ |
           |___/

    Welcome to Kytos SDN Platform!

    We are making a huge effort to make sure that this console will work fine
    but for now it's still experimental.

    Kytos website.: https://kytos.io/
    Documentation.: https://docs.kytos.io/
    OF Address....: tcp://0.0.0.0:6653
    WEB UI........: http://0.0.0.0:8181/
    Kytos Version.: 2022.1.1

kytos $> 2022-05-19 17:36:22,831 - WARNING [urllib3.connectionpool] (eapm event processor thread) Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9884f3730>: Failed to establish a new connection: [Errno 111] Connection refused')': /intake/v2/events
2022-05-19 17:36:22,832 - WARNING [urllib3.connectionpool] (eapm event processor thread) Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9883b3a60>: Failed to establish a new connection: [Errno 111] Connection refused')': /intake/v2/events
2022-05-19 17:36:22,833 - WARNING [urllib3.connectionpool] (eapm event processor thread) Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb98833e1c0>: Failed to establish a new connection: [Errno 111] Connection refused')': /intake/v2/events
2022-05-19 17:36:22,834 - ERROR [elasticapm.transport] (eapm event processor thread) Failed to submit message: 'Connection to APM Server timed out (url: http://localhost:8200/intake/v2/events, timeout: 5 seconds)'
2022-05-19 17:36:43,419 - WARNING [urllib3.connectionpool] (eapm event processor thread) Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb9883b3c40>: Failed to establish a new connection: [Errno 111] Connection refused')': /intake/v2/events

[viniarck]

@italovalcy, yes. The rationale was that since APM and MongoDB have sensitive information that need to be defined, all of that are only supported via environment variables to avoid having these in the conf file, and in the conf file it's only specified which backend is being used. That way, as long as they are injected safely it should make it easier to deploy and to avoid leaking or storing credentials, like some cloud platforms typically do (with docker secrets loaded on env vars, or kubernetes secrets loaded on env vars). Let me know if you have other suggestions or preference on how to manage this, especially also considering what AmLight will likely use in prod.

[viniarck]

@italovalcy I'll go ahead and merge this, let's continue this thread discussion on issue #217, really appreciated your feedback and review.