You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I get a hang within 2-12 hours processing my rules. Using pdb to debug is impossible because the hang appears to be in an infinite loop inside the blist module which is called with the GIL held
gdb (py-bt) show this:
Thread 8 (Thread 0x7fb85c229700 (LWP 3141)):
Traceback (most recent call first):
File "/.venv/lib/python3.7/site-packages/blist/_sortedlist.py", line 183, in __getitem__
return self._i2u(self._blist[index])
File "/.venv/lib/python3.7/site-packages/elastalert/ruletypes.py", line 336, in duration
return self.get_ts(self.data[-1]) - self.get_ts(self.data[0])
File "/.venv/lib/python3.7/site-packages/elastalert/ruletypes.py", line 324, in append
while self.duration() >= self.timeframe:
File "/.venv/lib/python3.7/site-packages/elastalert/ruletypes.py", line 620, in garbage_collect
({self.ts_field: ts}, 0)
File "/.venv/lib/python3.7/site-packages/elastalert/elastalert.py", line 908, in run_rule
rule['type'].garbage_collect(endtime)
File "/.venv/lib/python3.7/site-packages/elastalert/elastalert.py", line 1269, in handle_rule_execution
num_matches = self.run_rule(rule, endtime, rule.get('initial_starttime'))
File "/.venv/lib/python3.7/site-packages/apscheduler/executors/base.py", line 125, in run_job
retval = job.func(*job.args, **job.kwargs)
File "/opt/python-3.7.0/lib/python3.7/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/opt/python-3.7.0/lib/python3.7/concurrent/futures/thread.py", line 80, in _worker
work_item.run()
File "/opt/python-3.7.0/lib/python3.7/threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "/opt/python-3.7.0/lib/python3.7/threading.py", line 917, in _bootstrap_inner
self.run()
File "/opt/python-3.7.0/lib/python3.7/threading.py", line 885, in _bootstrap
self._bootstrap_inner()
The actual hang seems to be in _blist.c in ext_make_clean() in the loop at line 2075
I get a hang within 2-12 hours processing my rules. Using pdb to debug is impossible because the hang appears to be in an infinite loop inside the blist module which is called with the GIL held
gdb (py-bt) show this:
The actual hang seems to be in _blist.c in ext_make_clean() in the loop at line 2075
blist: 1.3.6
python: 3.7.0
elastalert: 1334b61
I have replaced blist with the drop in replacement from sortedcontainers:
And the hangs have gone away (and since this is pure-python, debug should be a lot easier).
The text was updated successfully, but these errors were encountered: