Cardinality Rule not working

[himanshigpta]

Hi All, I've recently installed Elastalert2 on RHEL7 server, tested few rules & its working as expected! We've been using custom scripts for each alert but this seemed way better solution! But I'm unable to implement the Cardinality rule, my use case is pretty basic - I need an alert if any value in a field exceeds the threshold in last 10mins.
This is my config :

index: hive*
name:  Hive Alerts

type: cardinality

cardinality_field: "loglevel"
max_cardinality: 10
query_key: "loglevel"

timeframe:
  hours: 4

alert:
- "email"

email:
- "redacted"
smtp_host: "redacted"
smtp_port: redacted
#testing subject
alert_subject: "ElastAlert: Loglevel {0} encountered at {1}"

alert_subject_args:
- loglevel
- "@timestamp"

Loglevel could be one of INFO, ERROR or WARN, and if any of these occur more than 10 times in last 4hours, an alert should be triggered.
I tried es_debug_trace which helped in some way, but I haven't achieved the expected result. I can see hits in the log but now matches, not sure why:

INFO:elastalert:Queried rule Hive Alerts from 2021-06-03 21:30 +0530 to 2021-06-03 21:45 +0530: 920 / 920 hits
INFO:elastalert:Ran Hive Alerts from 2021-06-03 21:30 +0530 to 2021-06-03 21:45 +0530: 920 query hits (370 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Hive Alerts range 900
INFO:elastalert:Background configuration change check run at 2021-06-03 21:46 +0530
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-06-03 21:46 +0530
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999692 seconds
INFO:elastalert:Queried rule Hive Alerts from 2021-06-03 21:31 +0530 to 2021-06-03 21:46 +0530: 1013 / 1013 hits
INFO:elastalert:Ran Hive Alerts from 2021-06-03 21:31 +0530 to 2021-06-03 21:46 +0530: 1013 query hits (919 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Hive Alerts range 900
INFO:elastalert:Background configuration change check run at 2021-06-03 21:47 +0530
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-06-03 21:47 +0530
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999796 seconds
INFO:elastalert:Queried rule Hive Alerts from 2021-06-03 21:32 +0530 to 2021-06-03 21:47 +0530: 1091 / 1091 hits
INFO:elastalert:Ran Hive Alerts from 2021-06-03 21:32 +0530 to 2021-06-03 21:47 +0530: 1091 query hits (1012 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Hive Alerts range 900

I've gone through lots of open/closed issues, tried multiple suggested solutions, but somehow cant make it work(trying for more than 2 days now)! The use case seems pretty straightforward for Elastalert2 I believe but I'm missing something, can anyone please help me debug or share a sample that I can refer maybe?

Thanks!

[jertel]

Cardinality is the measure of unique values for a given field. So the only way it would trigger is if you had 10 different log levels show up in a 4 hour timeframe. Ex: INFO, DEBUG, ERROR, WARN, TRACE, FATAL, VERBOSE, XXX, YYY, ZZZ.

I think what would make more sense for your need is a frequency rule.

[jertel]

I removed the pic since it had your email address in it.

I'm not entirely clear on what the remaining issue is -- are you saying you want just one email showing aggregation of all threshold breaches?

[himanshigpta]

Yes exactly, instead of getting email for each breach(loglevel in this case). Thanks I didn't realise it had my email address! :/

[jertel]

I think you're on the right track. I didn't focus closely on the email timestamps but I wonder if your short 5m aggregation window was resulting in multiple aggregation windows falling into the buffer time and/or timeframe, which would be longer than your 5m window.

[himanshigpta]

I waited for around 10mins but didn't get any alert, & in the logs I could see below error which is strange because without aggregation minutes I was getting proper value for num_hits:

INFO:elastalert:Adding alert for new Hive Frequency Alerts to aggregation(id: BCeG03kB3av5nWvDQOV6, aggregation_key: None), next alert at 2021-06-03 20:22:41.495392+00:00
INFO:elastalert:Adding alert for new Hive Frequency Alerts to aggregation(id: BCeG03kB3av5nWvDQOV6, aggregation_key: None), next alert at 2021-06-03 20:22:41.495392+00:00
INFO:elastalert:Adding alert for new Hive Frequency Alerts to aggregation(id: BCeG03kB3av5nWvDQOV6, aggregation_key: None), next alert at 2021-06-03 20:22:41.495392+00:00
INFO:elastalert:Adding alert for new Hive Frequency Alerts to aggregation(id: BCeG03kB3av5nWvDQOV6, aggregation_key: None), next alert at 2021-06-03 20:22:41.495392+00:00
INFO:elastalert:Adding alert for new Hive Frequency Alerts to aggregation(id: BCeG03kB3av5nWvDQOV6, aggregation_key: None), next alert at 2021-06-03 20:22:41.495392+00:00
INFO:elastalert:Ran new Hive Frequency Alerts from 2021-06-04 01:33 +0530 to 2021-06-04 01:48 +0530: 6165 query hits (5767 already seen), 398 matches, 0 alerts sent
INFO:elastalert:new Hive Frequency Alerts range 900
WARNING:apscheduler.scheduler:Execution of job "ElastAlerter.handle_pending_alerts (trigger: interval[0:01:00], next run at: 2021-06-04 01:49:12 IST)" skipped: maximum number  running instances reached (1)
INFO:elastalert:Background configuration change check run at 2021-06-04 01:49 +0530
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999893 seconds
/test/elastalert2/lib64/python3.6/site-packages/urllib3-1.26.4-py3.6.egg/urllib3/connection.py:462: SubjectAltNameWarning: Certificate for redacted has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://githubom/urllib3/urllib3/issues/497 for details.)
  SubjectAltNameWarning,
ERROR:elastalert:Traceback (most recent call last):
  File "/home/abinitio/test/elastalert2-master/elastalert/elastalert.py", line 1505, in alert
    return self.send_alert(matches, rule, alert_time=alert_time, retried=retried)
  File "/home/abinitio/test/elastalert2-master/elastalert/elastalert.py", line 1540, in send_alert
    counts = self.get_top_counts(rule, start, end, keys, qk=qk)
  File "/home/abinitio/test/elastalert2-master/elastalert/elastalert.py", line 2051, in get_top_counts
    hits_terms = self.get_hits_terms(rule, starttime, endtime, index, key, qk, number)
  File "/home/abinitio/test/elastalert2-master/elastalert/elastalert.py", line 554, in get_hits_terms
    self.thread_data.num_hits += len(buckets)
**AttributeError: '_thread._local' object has no attribute 'num_hits'**

ERROR:elastalert:Uncaught exception running rule new Hive Frequency Alerts: '_thread._local' object has no attribute 'num_hits'
INFO:elastalert:Rule new Hive Frequency Alerts disabled
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-06-04 01:49 +0530
INFO:elastalert:Background configuration change check run at 2021-06-04 01:50 +0530
INFO:elastalert:Disabled rules are: ['new Hive Frequency Alerts']
INFO:elastalert:Sleeping for 59.999778 seconds
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-06-04 01:50 +0530
INFO:elastalert:Background configuration change check run at 2021-06-04 01:51 +0530
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-06-04 01:51 +0530
INFO:elastalert:Disabled rules are: ['new Hive Frequency Alerts']
INFO:elastalert:Sleeping for 59.999807 seconds
INFO:elastalert:Background configuration change check run at 2021-06-04 01:52 +0530
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-06-04 01:52 +0530
INFO:elastalert:Disabled rules are: ['new Hive Frequency Alerts']
INFO:elastalert:Sleeping for 59.999775 seconds

[jertel]

I suspect that the use_terms_query/query_key and aggregation/aggregation_key settings are incompatible when used in the same rule.