Having issues with summary_table_fields

[dakotacody]

I'm trying to get one rule to create one alert that includes the summary of all the matches during that aggregation period. I've read through the documentation a lot. I originally started with frequency type rule but have now switched to any. No matter what I've tried I can only get the summary_fields table to show one value. However, top_count_keys does display the individual values for the keys I've specified, it's just not what I really want/need.

Aggregation resulted in the following data for summary_table_fields ==> ['syslog_hostname', 'syslog_program', 'count']:

+---------------------+----------------+-------+
|   syslog_hostname   | syslog_program | count |
+=====================+================+=======+
| alertmanager-1 | sshd           | 1     |
+---------------------+----------------+-------+

alert-errors

syslog_hostname:
alertmanager-1: 40
alertmanager-2: 40
alertmanager-3: 40

syslog_program:
sshd: 120

You can see I tried to force it to aggregate by the syslog_program so all alerts would bucket together thinking that what the issue with the summary table. You can also see below the that the top_count items all add up and are counted individually.

Basically, this is a rudimentary POC I'm trying to get this simple case working so I can understand how to actually do it with other interesting data.

What I'm expecting the table to look like in this case is this, but I only ever get one line and the count is always 1.

+---------------------+----------------+-------+
|   syslog_hostname   | syslog_program | count |
+=====================+================+=======+
| alertmanager-1 | sshd           | 40     |
| alertmanager-2 | sshd           | 40     |
| alertmanager-3 | sshd           | 40     |
+---------------------+----------------+-------+

I've tried using the attach_related etc.. but I only ever get the one column and the "related" matches are always only for the one one host.

Here's the current iteration I'm working with which has been changed and modified so many times that I'm finally reaching out. I'll leave the commented lines in to give some indication of what I've tried.

#type: frequency
type: any 
index: "syslog-*"
num_events: 1 
timeframe:
    minutes: 10
realert:
    minutes: 0
#use_terms_query: true
#aggregation_key:
#- syslog_program

#query_key:
#- syslog_hostname
#- syslog_program

aggregation: 
  minutes: 2

#scan_entire_timeframe: true
filter:
- term: 
    syslog_program: "sshd"
- query:
    wildcard:
      syslog_hostname: "alertmanager-*"

#attach_related: true
#include:
 #- syslog_hostname
 #- syslog_program

summary_table_fields:
  - syslog_hostname
  - syslog_program

top_count_number: 5
raw_count_keys: false
top_count_keys: 
  - syslog_hostname
  - syslog_program

[jertel]

Is there a summary_table_max_rows defined in your ElastAlert2 config file? (Let's rule out the simple explanation first)

I would recommend enabling debug so you can look at the true number of matches being returned by Elasticsearch. If you see multiple records being returned by your rule query but only one row shows up in the summary table then that will tell you the problem is not with your query or with Elasticsearch, but with the summary table rendering logic in ElastAlert 2. If you only see one record returned then you will know to focus on your query, aggregation window, or Elasticsearch server settings.

[dakotacody]

Yes. I set summary_table_max_rows last night with same intent of eliminating the simplest possibility. I also set the aggregation to be very long so it would find lots of matches overnight.

I looked at the code to see how it gets the matches and it appears to try to get the matches using the aggregate_id associated with them in the writeback index. Looking at the contents of that index it appears that there are matches there. (If I'm understanding this correctly)

In the alerts.py code it seems this is where the summary counts would be calculated.

263             # Maintain an aggregate count for each unique key encountered in the aggregation period`
264             for match in matches:
265                 key_tuple = tuple([str(lookup_es_key(match, key)) for key in summary_table_fields])

However on line 264 the len of matches is always 1 so I'm trying to chase back to where the matches are calculated and passed in to the get_aggregation_summary_text method.

I'll set debug as you suggested and see if that gets me any further along.

[dakotacody]

I found the issue and it is in the alerter we are using. This is happening because the alerta alerter is only sending one match. The following changes to alerta.py seem to fix the issue.

51c51
<         alerta_payload = self.get_json_payload(matches[0])
---
>         alerta_payload = self.get_json_payload(matches)
73c73
<     def get_json_payload(self, match):
---
>     def get_json_payload(self, matches):
81a82,83
>         # use the first match in the list for setting attributes
>         match = matches[0]
111c113
<             'rawData': self.create_alert_body([match]),
---
>             'rawData': self.create_alert_body(matches),
118a121
> 

and the results are now as expected.

Aggregation resulted in the following data for summary_table_fields ==> ['syslog_hostname', 'syslog_program', 'count']:

+---------------------+----------------+-------+
|   syslog_hostname   | syslog_program | count |
+=====================+================+=======+
| alertmanager-2 | sshd           | 10    |
+---------------------+----------------+-------+
| alertmanager-1 | sshd           | 9     |
+---------------------+----------------+-------+
| alertmanager-3 | sshd           | 7     |
+---------------------+----------------+-------+

[jertel]

Nice job tracking that down. I looked at the original PR for Alerta support and it was always hard-coded to use the first match. If you're up for it, and happy with the new behavior, feel free to submit a PR with this change so that it will be in your next ElastAlert 2 upgrade.

[dakotacody]

Created PR - #1068