Exporter fails on sql datatypes with mysql based airflow DB

[arihantsurana]

When deploying the airflow exporter on my Airflow 1.10 installaion, I face following error -

webserver_1  | [2018-11-13 00:14:01,448] ERROR - unsupported operand type(s) for -: 'decimal.Decimal' and 'datetime.datetime'
webserver_1  | Traceback (most recent call last):
webserver_1  |   File "/usr/local/lib/python3.6/site-packages/airflow/plugins_manager.py", line 86, in <module>
webserver_1  |     m = imp.load_source(namespace, filepath)
webserver_1  |   File "/usr/local/lib/python3.6/imp.py", line 172, in load_source
webserver_1  |     module = _load(spec)
webserver_1  |   File "<frozen importlib._bootstrap>", line 684, in _load
webserver_1  |   File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
webserver_1  |   File "<frozen importlib._bootstrap_external>", line 678, in exec_module
webserver_1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
webserver_1  |   File "/usr/local/airflow/plugins/airflow-exporter/prometheus_exporter.py", line 119, in <module>
webserver_1  |     REGISTRY.register(MetricsCollector())
webserver_1  |   File "/usr/local/lib/python3.6/site-packages/prometheus_client/core.py", line 96, in register
webserver_1  |     names = self._get_names(collector)
webserver_1  |   File "/usr/local/lib/python3.6/site-packages/prometheus_client/core.py", line 136, in _get_names
webserver_1  |     for metric in desc_func():
webserver_1  |   File "/usr/local/airflow/plugins/airflow-exporter/prometheus_exporter.py", line 114, in collect
webserver_1  |     for dag in get_dag_duration_info():
webserver_1  |   File "/usr/local/airflow/plugins/airflow-exporter/prometheus_exporter.py", line 77, in get_dag_duration_info
webserver_1  |     DagRun.state == State.RUNNING
webserver_1  |   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/query.py", line 2703, in all
webserver_1  |     return list(self)
webserver_1  |   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 90, in instances
webserver_1  |     util.raise_from_cause(err)
webserver_1  |   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
webserver_1  |     reraise(type(exception), exception, tb=exc_tb, cause=cause)
webserver_1  |   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 187, in reraise
webserver_1  |     raise value
webserver_1  |   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 78, in instances
webserver_1  |     for row in fetch]
webserver_1  |   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 78, in <listcomp>
webserver_1  |     for row in fetch]
webserver_1  |   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py", line 77, in <listcomp>
webserver_1  |     rows = [keyed_tuple([proc(row) for proc in process])
webserver_1  |   File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/sqltypes.py", line 1717, in process
webserver_1  |     return value - epoch
webserver_1  | TypeError: unsupported operand type(s) for -: 'decimal.Decimal' and 'datetime.datetime'
webserver_1  | [2018-11-13 00:14:01,567] ERROR - Failed to import plugin /usr/local/airflow/plugins/airflow-exporter/prometheus_exporter.py

[elephantum]

@cleverCat take a look plz

[elephantum]

@arihantsurana can you please check that v0.4.4 fixes this issue?

[ebartels]

We have a fix for this and will make a PR in the next week

[elephantum]

I believe this one is fixed

[alfccc]

did this get deployed in the latest release? I recently had the exact same issue

[ebartels]

@alfccc This was fixed:

#29
#38

[alfccc]

my connection string starts with mysql+pymysql, could that be a problem?
AIRFLOW__CORE__SQL_ALCHEMY_CONN='mysql+pymysql://...'

[elephantum]

@alfccc do you mind opening new issue with specific details: stack trace that you get, relevant details about DB and connection string?