libbeat 7.12.1 default logging logs excessive amount of data when failing to create index template

[gpapakyriakopoulos]

As the title mentions, auditbeat on the default logging level (INFO based on official documentation) logs an excessive amount of data on syslog when failing to create an index template (ex. Due to insufficient access privileges).

After logging the initial exception error, it proceeds to also log the full index template as part of the exception error log, which usually amounts to several thousand lines of JSON. The log output is repeated continuously while the error is present, resulting in quickly filling up massive amounts of disk space.

An example (truncated) error log that fits the above description is as follows :

Connection marked as failed because the onConnect callback failed: error loading template: could not load template. Elasticsearch returned: couldn't load template: 403 Forbidden: {"error":{"roo
t_cause":[{"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [beats_writer], this action is granted by the cluster privileges [manage_index_templates,manag
e,all]"}],"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [beats_writer], this action is granted by the cluster privileges [manage_index_templates,manage
,all]"},"status":403}. Response body: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [beats_writer], this action is granted by
the cluster privileges [manage_index_templates,manage,all]"}],"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [beats_writer], this action is granted by t
he cluster privileges [manage_index_templates,manage,all]"},"status":403}. Template is: {
Apr 30 00:00:04 siem packetbeat[18558]:   "index_patterns": [
Apr 30 00:00:04 siem packetbeat[18558]:     "packetbeat-7.12.1-*"
Apr 30 00:00:04 siem packetbeat[18558]:   ],
Apr 30 00:00:04 siem packetbeat[18558]:   "mappings": {
Apr 30 00:00:04 siem packetbeat[18558]:     "_meta": {
Apr 30 00:00:04 siem packetbeat[18558]:       "beat": "packetbeat",
Apr 30 00:00:04 siem packetbeat[18558]:       "version": "7.12.1"
Apr 30 00:00:04 siem packetbeat[18558]:     },
Apr 30 00:00:04 siem packetbeat[18558]:     "date_detection": false,
Apr 30 00:00:04 siem packetbeat[18558]:     "dynamic_templates": [
Apr 30 00:00:04 siem packetbeat[18558]:       {
Apr 30 00:00:04 siem packetbeat[18558]:         "labels": {
Apr 30 00:00:04 siem packetbeat[18558]:           "mapping": {
Apr 30 00:00:04 siem packetbeat[18558]:             "type": "keyword"
Apr 30 00:00:04 siem packetbeat[18558]:           },
Apr 30 00:00:04 siem packetbeat[18558]:           "match_mapping_type": "string",
Apr 30 00:00:04 siem packetbeat[18558]:           "path_match": "labels.*"
Apr 30 00:00:04 siem packetbeat[18558]:         }
Apr 30 00:00:04 siem packetbeat[18558]:       },
Apr 30 00:00:04 siem packetbeat[18558]:       {
Apr 30 00:00:04 siem packetbeat[18558]:         "container.labels": {
Apr 30 00:00:04 siem packetbeat[18558]:           "mapping": {
Apr 30 00:00:04 siem packetbeat[18558]:             "type": "keyword"
Apr 30 00:00:04 siem packetbeat[18558]:           },
Apr 30 00:00:04 siem packetbeat[18558]:           "match_mapping_type": "string",
Apr 30 00:00:04 siem packetbeat[18558]:           "path_match": "container.labels.*"
Apr 30 00:00:04 siem packetbeat[18558]:         }
Apr 30 00:00:04 siem packetbeat[18558]:       },
Apr 30 00:00:04 siem packetbeat[18558]:       {
Apr 30 00:00:04 siem packetbeat[18558]:         "fields": {
Apr 30 00:00:04 siem packetbeat[18558]:           "mapping": {

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[ChrsMark]

Thanks for reporting this @gpapakyriakopoulos! Could you please add more information regarding version you are using and how you run auditbeat (configuration, cli flags, binary/service etc)?

cc: @andrewkroh @adriansr

[tsigouris007]

Our ES cluster runs on version 7.12.1.
Our beats + elastic-agent run on version 7.12.1.
Our auditbeat configuration is the following:

# =========================== Modules configuration ============================
auditbeat.config.modules:
  # Set to true to enable config reloading
  reload.enabled: true
  reload.period: 10s
  # Glob pattern for configuration reloading
  path: [ '${path.config}/audit.rules.d/*.conf' ]

auditbeat.modules:

- module: auditd
  # Load audit rules from separate files. Same format as audit.rules(7).
  audit_rule_files: [ '${path.config}/audit.rules.d/*.conf' ]
  audit_rules: |

- module: file_integrity
  paths:
    - /bin
    - /usr/bin
    - /sbin
    - /usr/sbin
    - /etc/sudoers
    - /etc/sudoers.d
    - /etc/shadow
    - /etc/shadow-
    - /etc/passwd
    - /etc/passwd-
    - /etc/gshadow
    - /etc/gshadow-
    - /etc/group
    - /etc/group-
    - /etc/pam.d
    - /etc/pam.conf
    - /etc/ntp.conf
    - /etc/ferm
    - /etc/cron.d
    - /etc/cron.daily
    - /etc/cron.hourly
    - /etc/cron.monthly
    - /etc/cron.weekly
    - /etc/crontab
    - /etc/ca-certificates
    - /etc/ca-certificates/conf
    - /etc/subgid
    - /etc/subgid-
    - /etc/subuid
    - /etc/subuid-
    - /etc/ssh
    - /etc/hosts
    - /etc/hosts.allow
    - /etc/hosts.deny
    - /etc/hostname
    - /etc/resolv.conf
    - /etc/selinux
    - /etc/security
    - /etc/ssl
    - /etc/fstab
    - /etc/services
    - /root
    - /boot/grub/grub.conf
  exclude_files:
    - /root/.bash_history
    - /root/.bashrc
  recursive: true

- module: system
  datasets:
    - package # Installed, updated, and removed packages
  period: 5m

- module: system
  datasets:
    - login # User logins, logouts, and system boots.
  period: 1m

- module: system
  datasets:
    - user # User information
  period: 10m

- module: system
  datasets:
    - host # General host information, e.g. uptime, IPs
  period: 10m

- module: system
  datasets:
    - socket # Opened and closed sockets
  period: 1m

  # How often datasets send state updates with the
  # current state of the system (e.g. all currently
  # running processes, all open sockets).
  state.period: 12h

  # Enabled by default. Auditbeat will read password fields in
  # /etc/passwd and /etc/shadow and store a hash locally to
  # detect any changes.
  user.detect_password_changes: true

  # File patterns of the login record files.
  login.wtmp_file_pattern: /var/log/wtmp*
  login.btmp_file_pattern: /var/log/btmp*

# =================================== Elastic ===================================
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: 
      - "https://...:9200"
  
  protocol: "https"
   
  ssl.certificate_authorities: [...]
  ssl.certificate: ...
  ssl.key: ...
  ssl.verification_mode: certificate
  
  username: ...
  password: ...
  
# ======================= Elasticsearch template setting =======================
setup.template.settings:
  index.number_of_shards: 3

Our environment is:

Environment="GODEBUG='madvdontneed=1'"
Environment="BEAT_LOG_OPTS="
Environment="BEAT_CONFIG_OPTS=-c /etc/auditbeat/auditbeat.yml"
Environment="BEAT_PATH_OPTS=--path.home /usr/share/auditbeat --path.config /etc/auditbeat --path.data /var/lib/auditbeat --path.logs /var/log/auditbeat"
ExecStart=/usr/share/auditbeat/bin/auditbeat --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS

The service is running as systemd default service with no other customization.

[andrewkroh]

This is a problem affecting all Beats since this logging is part of libbeat. I'd probably stop the template logging entirely. The template itself can be dumped using beatname export template --es.version=7.11.0 if needed for debugging purposes.

[elasticmachine]

Pinging @elastic/agent (Team:Agent)