Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elastalert hangs during rule processing in blist module #2693

Open
PhracturedBlue opened this issue Feb 21, 2020 · 0 comments
Open

Elastalert hangs during rule processing in blist module #2693

PhracturedBlue opened this issue Feb 21, 2020 · 0 comments

Comments

@PhracturedBlue
Copy link

PhracturedBlue commented Feb 21, 2020

I get a hang within 2-12 hours processing my rules. Using pdb to debug is impossible because the hang appears to be in an infinite loop inside the blist module which is called with the GIL held

gdb (py-bt) show this:

Thread 8 (Thread 0x7fb85c229700 (LWP 3141)):
Traceback (most recent call first):         
  File "/.venv/lib/python3.7/site-packages/blist/_sortedlist.py", line 183, in __getitem__
    return self._i2u(self._blist[index])                                                                                              
  File "/.venv/lib/python3.7/site-packages/elastalert/ruletypes.py", line 336, in duration
    return self.get_ts(self.data[-1]) - self.get_ts(self.data[0])                                                                     
  File "/.venv/lib/python3.7/site-packages/elastalert/ruletypes.py", line 324, in append  
    while self.duration() >= self.timeframe:                                                                                          
  File "/.venv/lib/python3.7/site-packages/elastalert/ruletypes.py", line 620, in garbage_collect
    ({self.ts_field: ts}, 0)                                                                                                                 
  File "/.venv/lib/python3.7/site-packages/elastalert/elastalert.py", line 908, in run_rule      
    rule['type'].garbage_collect(endtime)                                                                                                    
  File "/.venv/lib/python3.7/site-packages/elastalert/elastalert.py", line 1269, in handle_rule_execution
    num_matches = self.run_rule(rule, endtime, rule.get('initial_starttime'))                                                                        
  File "/.venv/lib/python3.7/site-packages/apscheduler/executors/base.py", line 125, in run_job          
    retval = job.func(*job.args, **job.kwargs)                                                                                                       
  File "/opt/python-3.7.0/lib/python3.7/concurrent/futures/thread.py", line 57, in run                                                               
    result = self.fn(*self.args, **self.kwargs)                                                                                                      
  File "/opt/python-3.7.0/lib/python3.7/concurrent/futures/thread.py", line 80, in _worker                                                           
    work_item.run()                                                                                                                                  
  File "/opt/python-3.7.0/lib/python3.7/threading.py", line 865, in run                                                                              
    self._target(*self._args, **self._kwargs)                                                                                                        
  File "/opt/python-3.7.0/lib/python3.7/threading.py", line 917, in _bootstrap_inner                                                                 
    self.run()                                                                                                                                       
  File "/opt/python-3.7.0/lib/python3.7/threading.py", line 885, in _bootstrap                                                                       
    self._bootstrap_inner()                                                                                                                          

The actual hang seems to be in _blist.c in ext_make_clean() in the loop at line 2075

blist: 1.3.6
python: 3.7.0
elastalert: 1334b61

I have replaced blist with the drop in replacement from sortedcontainers:

ruletypes.c:
< from blist import sortedlist
> from sortedcontainers import SortedList as sortedlist

And the hangs have gone away (and since this is pure-python, debug should be a lot easier).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant