CHANGELOG.asciidoc

Beats version 8.1.0

View commits

Breaking changes

Filebeat

• Remove Recorded Future fileset integration from threatintel module. 30564

Bugfixes

Auditbeat

• auditd: Add error.message to events when processing fails. 30009

• Fix handling of execve call events which have no argument. 30585 30586

Filebeat

• Fix ECS version string in threatintel to be consistent with other modules and add event.timezone. 30499 30570

• Add default paths value to MySQL Enterprise module to prevent issues with pipeline installations 30598

Winlogbeat

• Add provider names to Security pipeline conditional check in routing pipeline. 27288 29781

Functionbeat

• Pass AWS region configuration correctly. 28520 30238

Added

Affecting all Beats

• Name all k8s workqueue. 28085

• Discover changes in Kubernetes nodes metadata as soon as they happen. 23139

• Update k8s library 29394

• Add support for latest k8s versions v1.23 and v1.22 29575

• Add script processor to all beats 29269 29752

• Only connect to Elasticsearch instances with the same version or newer. 29683

• Move umask from code to service files. 29708

• Add metadata change support for some processors 30183

Auditbeat

• system/socket: Add process.entity_id capture for socket events. 30230 30231

Filebeat

• Add support for filtering in journald input with unit, kernel, identifiers and include_matches. 29294

• Add new userAgent and beatInfo template functions for httpjson input 29528

• Add pipeline in FB’s supported hints. 30212

Metricbeat

• Add add_resource_metadata configuration to Kubernetes module. 29133

• Add containerd module with cpu, memory, blkio metricsets. 29247

• Add container.id and container.runtime ECS fields in container metricset. 29560

• Add memory.workingset.limit.pct field in Kubernetes container/pod metricset. 29547

• Add k8s metadata in state_cronjob metricset. 29572

• Add xpack.enabled support for Enterprise Search module. 29871

• Add gcp firestore metricset. 29918

• Remove strict parsing on RabbitMQ module 30090

Packetbeat

• Add automated OEM Npcap installation handling. 29112 30438 30493

• Add support for capturing TLS random number and OCSP status request details. 29962 30102

Beats version 8.0.1

View commits

Bugfixes

Filebeat

• tcp/unix input: Stop accepting connections after socket is closed. 29712

• Fix using log_group_name_prefix in aws-cloudwatch input. 29695

• Fix multiple instances of the same module configured within filebeat.modules in filebeat.yml. 29649 29952

• aws-s3: fix race condition in states used by s3-poller. 30123 30131

Filebeat - Fix broken Kafka input 29746 30277 - cisco module: Fix change the broke ASA and FTD configs that used var.input: syslog. 30072 - aws-s3: fix race condition in states used by s3-poller. 30123 30131

Heartbeat - Fix missing mapping for service.name. 30324

Winlogbeat

• Fix run loop when reading from evtx file 30006

Beats version 8.0.0

View commits

Breaking changes

Affecting all Beats

• Remove the deprecated xpack.monitoring. settings. Going forward only monitoring. settings may be used. 9424 18608

• Remove deprecated/undocumented IncludeCreatorMetadata setting from kubernetes metadata config options. 28006

• Remove deprecated fields from kubernetes module. 28046

• Remove deprecated config option aws_partition. 28120

• Improve stats API by adding host metadata. 27963

• Libbeat: logp package forces ECS compliant logs. Logs are JSON formatted. Options to enable ECS/JSON have been removed. 15544 28573

• Remove auto from the available options of setup.ilm.enabled and set the default value to true. 28671

• add_process_metadata processor: Replace usage of deprecated process.ppid field with process.parent.pid. 28620

• add_docker_metadata processor: Replace usage of deprecated process.ppid field with process.parent.pid. 28620

• Use data streams instead of indices for storing events from Beats. 28450

• Remove option setup.template.type and always load composable template with data streams. 28450

• Remove several ILM options (rollover_alias and pattern) as data streams do not require index aliases. 28450

• Populate index template’s default_fields setting with ECS fields only. 28596 28215

• Remove deprecated --template and --ilm-policy flags. Use --index-management instead. 28870

• Remove logging.files.suffix option, and default to datetime endings in log file names. The format of the new name is {beatname}-{date}(-n)?.ndjson. Example log file names from oldest to newest: filebeat-20200101.ndjson, filebeat-20200101-1.ndjson, filebeat-20200101-2.ndjson. 28927

• Align kubernetes configuration settings. 29908

• Change log file extension for Beats and Elastic Agent to .ndjson. If you are collecting the logs, you must change the path configuration to /path/to/logs/{beatname}*.ndjson to avoid any issues. 28927

• Remove legacy support for SSLv3. 30071

Filebeat

• Add while_pattern type to multiline reader. 19662

• auditd dataset: Use process.args to store program arguments instead of auditd.log.aNNN fields. 29601

• Remove deprecated old awscloudwatch input name. 29844

Metricbeat

• Remove network and diskio metrics from ec2 metricset. 28316

• Rename read/write_io.ops_per_sec to read/write.iops in rds metricset. 28350

• system/process metricset: Replace usage of deprecated process.ppid field with process.parent.pid. 28620

Packetbeat

• event.category no longer contains the value network_traffic because this is not a valid ECS event category value. 20556

• Remove deprecated TLS fields in favor of tls.server.x509 and tls.client.x509 ECS fields. 28487

• HTTP: The field http.request.method will maintain its original case. 28620

Winlogbeat

• Remove top level hash property from sysmon events. 20653

• Move module processing from local Javascript processor to ingest node. 29184 29435

Bugfixes

Auditbeat

• libbeat/processors/add_process_metadata: Fix memory leak in process cache. 24890 29717

Filebeat

• Fix using log_group_name_prefix in aws-cloudwatch input. 29695

Heartbeat

• Add fonts to support more types of characters for multiple languages. 29861

Metricbeat

• Extract correct index property in kibana.stats metricset. 29622

• Fixed bug with elasticsearch/cluster_stats metricset not recording license expiration date correctly. 29711

Packetbeat

• Prevent incorrect use of AMQP protocol parsing from causing silent failure. 29017

• Fix error handling in MongoDB protocol parsing. 29017

• Redis: fix incorrectly handle with two-words redis command. 14872 14873

• Unify gopacket dependencies. 29167

Added

Affecting all Beats

• Add config option rotate_on_startup to file output. 19150 19347

• Update to ECS 8.0 fields. 28620

• Support custom analyzers in fields.yml. 28540 28926

• Support self-signed certificates on outputs. 29229

• Add FIPS configuration option for all AWS API calls. 28899

• Warn users when connecting to older versions of Elasticsearch instances. 29723

• add_fields processor is now able to set metadata in events. 30092

Auditbeat

• system/process: Prevent hashing files in other mnt namespaces. 25777 29678 29786

Metricbeat

• Add preliminary AIX support. 27954

• Add option to skip older k8s events. 29396

• Add elasticsearch.cluster.id field to Beat and Kibana modules. 29577

• Add elasticsearch.cluster.id field to Logstash module. 29625

Winlogbeat

• Add support for sysmon event ID 26; FileDeleteDetected. 26280 29957

Elastic Log Driver

• Fixed docs for hosts. 23644

Beats version 7.17.0

View commits

Breaking changes

Affecting all Beats

• Change Docker base image from CentOS 7 to Ubuntu 20.04 29681

Bugfixes

Affecting all Beats

• Enrich kubernetes metadata with node annotations. 29605

Auditbeat

• system/socket: Fix startup errors on newer 5.x kernels due to missing _do_fork function. 29607 29744

• system/package: Fix parsing of Installed-Size field of DEB packages. 16661 17188

• system module: Fix panic during initialisation when /proc/stat can’t be read. 17569

• system/package: Fix an error that can occur while trying to persist package metadata. 18536 18887

• system/socket: Fix bugs leading to wrong process being attributed to flows. 29166 17165

• system/socket: Fix process name and arg truncation for long names, paths and args lists. 24667 29410

Filebeat

• aws-s3: Stop trying to increase SQS message visibility after ReceiptHandleIsInvalid errors. 29480

• Fix handling of IPv6 addresses in netflow flow events. 19210 29383

• Fix sophos KV splitting and syslog header handling 24237 29331

• Undo deletion of endpoint config from cloudtrail fileset in 29415. 29450

• Make Cisco ASA and FTD modules conform to the ECS definition for event.outcome and event.type. 29581 29698

• ibmmq: Fixed @timestamp not being populated with correct values. 29773

• aws-s3: Improve gzip detection to avoid false negatives. 29968

• decode_cef: Fix panic when recovering from invalid CEF extensions that contain escape characters. 30010

Heartbeat

• Fix race condition in http monitors using mode:all that can cause crashes. pull

• Fix broken ICMP availability check that prevented heartbeat from starting in rare cases. pull

• Fix broken macOS ICMP python e2e test. 29900

• Only add monitor.status to browser events when summary. 29460

• Also add summary to journeys for which the synthetics runner crashes. 29606

• Update size of ICMP packets to adhere to standard min size. 29948

Metricbeat

• Use xpack.enabled on SM modules to write into .monitoring indices when using Metricbeat standalone 28365

• Fix in rename processor to ingest metrics for write.iops to proper field instead of write_iops in rds metricset. 28960

• Enhance filter check in kubernetes event metricset. 29470

• Fix gcp metrics metricset apply aligner to all metric_types 29513

• Fixed GCP GKE Overview dashboard 29913

• Remove overriding of index pattern on the Kubernetes overview dashboard. 29676

Added

Affecting all Beats

• SASL/SCRAM in the Kafka output is no longer beta. 29126

• Add job.name in pods controlled by Jobs 28954

Heartbeat

• More errors are now visible in ES with new logic failing monitors later to ease debugging. pull

Winlogbeat

• Add support for custom XML queries 1054 29330

Deprecated

Known Issue

Beats version 7.16.3

View commits

Bugfixes

Affecting all Beats

• Fields of type match_only_text (i.e. message) and wildcard were missing from the template’s default_field list. 29633 29634

Beats version 7.16.2

View commits

Bugfixes

Filebeat

• Resolve issue with @timestamp for defender_atp. 28272

• Fix handling of escaped newlines in the decode_cef processor. 16995 29268

Added

Filebeat

• Update Cisco module to enable TCP input. 26118 28821 26159

Winlogbeat

• Add configuration option for registry file flush timeout 29001 29053

Beats version 7.16.1

View commits

Bugfixes

Affecting all Beats

• Overwrite index name in index template correctly. 28571 29299

Added

Filebeat

• Add elapsed time information to aws-s3 input errors and log messages. 29328

Beats version 7.16.0

View commits

Breaking changes

Affecting all Beats

• Load index templates v2 (composable index templates) by default when talking to ES 7.16 or ES 8.x. Please note that you cannot load templates into Elasticsearch 7.8 or older with this default. To load templates to these ES version, set setup.template.type back to legacy. 28538

• Previously, RE2 and thus Golang had a bug where (|a)* matched more characters than (|a)+. To stay consistent with PCRE, the bug was fixed. Configurations that rely on the old, buggy behaviour has to be adjusted. See more about Golang bug: golang/go#46123 27543

• Remove Journalbeat. Use journald input of Filebeat instead. 29131

Heartbeat

• Change behavior in case of duplicate monitor IDs in configs to be last monitor wins. 29041

Metricbeat

• Align fields to Beats naming conventions in GCP module. 27231 27974

Functionbeat

• Support for Google Cloud Functions have been removed, as it has been in Beta for a long time and been broken for a few releases. Please use other tools provided by Elastic to fetch data from GCP (e.g. Filebeat).

Bugfixes

Affecting all Beats

• Fix discovery of Nomad allocations with multiple events during startup. 28700

• Fix the wrong beat name on monitoring and state endpoint 27755

• Skip configuration checks in autodiscover for configurations that are already running 29048

• Fix decode_json_processor to always respect add_error_key 29107

• Fix add_labels flattening of array values. 29211

• Skip add_kubernetes_metadata processor when Kubernetes metadata are already present 27689

Auditbeat

• Fix handling of root and relative paths 24430 28354

• Fix handling of long file names on Windows. 25334 28517

• System/socket dataset: Fix uninstallation of return kprobes. 28608 28609

• Fix auditbeat tracing struct decoding. 28580

Filebeat

• Update indentation for azure filebeat configuration. 26604

• Tolerate faults when Windows Event Log session is interrupted 27947 28191

• Add support for username in Cisco ASA security negotiation logs 26975

• Relax time parsing and capture group and session type in Cisco ASA module 24710 28325

• Correctly track bytes read when max_bytes is exceeded. 28317 28352

• Fix parsing of apache log levels including numbers. 28717

• Upgrade azure-eventhub SDK reference, contains potential checkpoint fixes. 28919

• Revert usageDetails api version to 2019-01-01. 28995

• Fix in aws-s3 input regarding provider discovery through endpoint 28963

• Fix threatintel.misp filters configuration. 27970

• Fix opening files on Windows in filestream so open files can be deleted. 29113 29180

Heartbeat

• Fix broken seccomp filtering and improve security via setcap and setuid when running as root on linux in containers. 27878

• Log browser zip_url download failures as warn instead of as info. 28440

• Properly locate base stream in fleet configs. 28455

• Stop logging params values. 28774

• Remove accidentally included cups library in Docker images. pull

• Fix broken monitors with newer versions of image relying on dup3. elastic#28938[pull

Metricbeat

• beat module respects basepath config option. 28162

• Fix list_docker.go 28374

• Fix RDS metadata in Cloudwatch metricset. 29106

• Errors should be thrown as errors. Metricsets inside metricbeat will now throw errors as the error log level. 27804

Winlogbeat

• Tolerate faults when Windows Event Log session is interrupted 27947 28191

• Add ECS 1.9 new users fields 26509

• Don’t split hyphenated tokens 28483

• Correctly handle AccessMask if it is an integer or list of masks. 29016

Added

Affecting all Beats

• Allow non-padded base64 data to be decoded by decode_base64_field 27311, 27021

• The Kafka support library Sarama has been updated to 1.29.1. 27717

• Kafka is now supported up to version 2.8.0. 27720

• Add Huawei Cloud provider to add_cloud_metadata. 27607

• Add default seccomp policy for linux arm64. 27955

• Add cluster level add_kubernetes_metadata support for centralized enrichment 24621

• Update cloud.google.com/go library. 28229

• Add additional metadata to the root HTTP endpoint. 28265

• Upgrade k8s.io/client-go library. 28228

• Update ECS to 1.12.0. 27770

• Fields mapped as match_only_text will automatically fallback to a text mapping when using Elasticsearch versions that do not support match_only_text. 27770

• Do not load ML jobs to Elasticsearch 8.x from new Beats 7.x releases. 27771

• Update kubernetes scheduler and controllermanager endpoints in elastic-agent-standalone-kubernetes.yaml with secure ports 28675

• Add default seccomp policy for Linux arm64. 27955

• Add http.pprof.enabled option to libbeat to allow http/pprof endpoints on the socket that libbeat creates for metrics. 21965

• Enable IMDSv2 support for add_cloud_metadata processor on AWS. 22101 28285

Filebeat

• Add timezone config option to the decode_cef processor. 27232 27727

• Add timezone config option to the syslog input. 27727

• Added support for parsing syslog dates containing a leading 0 (e.g. Sep 01) rather than a space. 27775

• Add base64 Encode functionality to httpjson input. 27681

• Add join and sprintf functions to httpjson input. 27735

• Improve memory usage of line reader of log and filestream input. 27782

• Add ignore_empty_value flag to httpjson split processor. 27880

• Add support for passing a prefix on S3 bucket list mode for AWS-S3 input 28252 27965

• Update Cisco ASA/FTD ingest pipeline grok/dissect patterns for multiple message IDs. 26869 26879

• Add write access to url.value from request.transforms in httpjson input. 27937

• Add Base64 encoded HMAC and UUID template functions to httpjson input 27873

• Release checkpoint module as GA. 27814

• Make aws-cloudwatch input GA. 28161

• Move processing to ingest node for AWS vpcflow fileset. 28168

• Release zoom module as GA. 28106

• Add support for secondary object attribute handling in ThreatIntel MISP module 28124

• Azure signinlogs - Add support for ManagedIdentitySignInLogs, NonInteractiveUserSignInLogs, and ServicePrincipalSignInLogs. 23653

• Add base64Decode and base64DecodeNoPad functions to httpsjon templates. 28385

• Add 'early_limit' config option for Rate-Limiting httpjson. Default rate-limiting for Okta will start when remaining is 1. 28513

• Add latency config option for aws-cloudwatch input. 28509

• Added proxy support to threatintel/malwarebazaar. 28533

• Sophos UTM: Support logs containing hostname in Syslog header. 28638

• Moving Oracle Filebeat module to GA. 28754

• Add support in aws-s3 input for S3 notification from SNS to SQS. 28800

• Add support in aws-s3 input for custom script parsing of S3 notifications. 28946

• Improve error handling in aws-s3 input for malformed S3 notifications. 28828 28946

• filestream and log inputs accept null (\u0000) as line terminator. 28998

Heartbeat

• Support JSON expressions / validation of JSON arrays. 28073

• Experimental run once mode. 25972

• Add keyword multi-field mapping for synthetics.step.name. 28452

Metricbeat

• Enable journald input type in Filebeat. 7955 27351

• Added a new beta enterprisesearch module for Elastic Enterprise Search 27549

• Register additional name for storage metricset in the azure module. 28447

• Update reference to gosigar pacakge for filesystem windows fix. 28909

• Override Host() on statsd MetricSet 29103

• Add Linux pressure metricset 27355

• Add User-Agent header to HTTP requests. 18160 27509

Functionbeat

• Add support for AWS Kinesis record deaggregation 28241

Winlogbeat

• Add support for event language selection from config file 19818

Deprecated

Affecting all Beats

• Deprecate setup.template.type. In the future Beats will load data streams instead of regular indices.

Filebeat

• Deprecate log input in favour of filestream input. 28623

Beats version 7.15.2

View commits

Bugfixes

Affecting all Beats

• Beats dashboards use custom index when setup.dashboards.index is set. 21232 27901

• Fix handling of float data types within processors. 28279 28280

• Allow clone3 syscall in seccomp filters. 28117

• Remove unnecessary escaping step in dashboard loading, so they can be displayed in Kibana. 28395

• Fix AWS proxy_url config from url to string type. 28725

• Fix fingerprint processor to give it access to the @timestamp field. 28683

Filebeat

• Fix initialization of http client in Cloudfoundry input. 28271 28277

• Fix aws-s3 input by checking if GetObject API call response content type exists. 28457

• Set url as a pointer in the httpjson template context to ensure access to all methods. 28695

• Fix google_workspace documentation links. 28657

Metricbeat

• Divide RDS metric cpu.total.pct by 100. 28456

Packetbeat

• Handle truncated DNS records more gracefully. 21495 28297

• Fix data stream name for network flows when running under Elastic Agent and Fleet. 28408

Beats version 7.15.1

View commits

Bugfixes

Filebeat

• Update Sophos xg module pipeline to deal with missing date and time fields. 27834

• sophos/xg fileset: Add missing pipeline for System Health logs. 27827 27826

Metricbeat

• Add support for kube-state-metrics v2.0.0. 27552

Beats version 7.15.0

View commits

Breaking changes

Affecting all Beats

• Loading Kibana assets (dashboards, index templates) rely on Saved Object API. So to provide a reliable service, Beats can only import and export dashboards using at least Kibana 7.15. 20672 27220

Filebeat

• Remove all alias fields pointing to ECS fields from modules. This affects the Suricata and Traefik modules. 10535 26627

• Fix Crowdstrike ingest pipeline that was creating flattened process fields. 27622 27623

• Rename log.path to log.file.path in filestream to be consistent with log input and ECS. 27761

Heartbeat - Remove long deprecated watch_poll functionality. 27166 - Fix inconsistency in event.dataset values between heartbeat and fleet by always setting this value to the monitor type / fleet dataset. 27535

Metricbeat

• Fix Elasticsearch jvm.gc.collectors.old being exposed as young 19636 26616

Bugfixes

Affecting all Beats

• Improve perfmon metricset performance. 26886

• Preserve annotations in a kubernetes namespace metadata 27045

• Fix build constraint that caused issues with doc builds. 27381

• Do not try to load ILM policy if check_exists is false. 27508 26322

• Fix bug with cgroups hierarchy override path in cgroups 27620

• Beat setup kibana command may use the elasticsearch API key defined in output.elasticsearch.api_key. 24015 27540

• Fix decode_xml handling of array merging when using to_lower: true. 27922

• Separate namespaces for V1 and V2 controller paths 27676

• Do not try to load ILM policy if check_exists is false. 27508 26322

• Kubernetes autodiscover fails in node scope if node name cannot be discovered 26947

Auditbeat

• File Integrity Module: Honor include_files when doing initial scan. 27273 27722

Filebeat

• Update Filebeat compatibility function to remove processor description field on ES < 7.9.0 27774

• Make filestream events ECS compliant. 27776

Metricbeat

• Allow metric prefix override per service in gcp module. 26960

• Update metrics configuration and dashboards after changes in the Azure Monitor 27520

Winlogbeat

• Fix an issue with message template caching in the wineventlog-experimental API implementation. 26826

Added

Affecting all Beats

• Add proxy support for AWS functions. 26832

• Added policies to the Elasticsearch output for non indexible events 26952

• Add logging.metrics.namespaces config option to control what metric groups are reported in logs. 25727

• Add sha256 digests to RPM packages. 23670

• Add new 'offline' docker image for Elastic Agent. 27052

• Add cgroups V2 support 27242

• Update ECS field definitions to ECS 1.11.0. 27107

• The disk queue is now GA. 27515

• Add daemonset.name in pods controlled by DaemonSets 26808, 25816

Filebeat

• Add new template functions and value_type parameter to httpjson transforms. 26847

• Add support to merge registry updates in the filestream input across multiple ACKed batches in case of backpressure in the registry or disk. 25976

• Add support to decode_cef for MAC addresses that do not contain separator characters. 27050 27109

• Add new hmac template function for httpjson input 27168

• Update tags and threatintel.indicator.provider fields in threatintel.anomali ingest pipeline 24746 27141

• Move AWS module and filesets to GA. 27428

• Update ecs.version to ECS 1.11.0. 27107

• Add option for S3 input to work without SQS notification 18205 27332

Metricbeat

• Move openmetrics module to oss. 26561

• Fix release state of kubernetes metricsets. 26864

• Add gke metricset collection to gcp module 26824

• Added statsd.mappings configuration for Statsd module 26220

• Added Airflow lightweight module 26220

• Add state_job metricset to Kubernetes modulehttps://github.com/elastic/pull/26479[26479]

• Bump AWS SDK version to v0.24.0 for WebIdentity authentication flow 19393 27126

Beats version 7.14.2

View commits

Bugfixes

Filebeat

• Auditd module: Fix the top exec commands dashboard visualization. 27638

• Store offset in log.offset field of events from the filestream input. 27688

• Fix httpjson input rate limit processing and documentation. 27739

Beats version 7.14.1

View commits

Bugfixes

Affecting all Beats

• Allow conditional processing in decode_xml and decode_xml_wineventlog. 27159

Filebeat

• Convert the o365 module’s client.port and source.port to numbers (from strings) in events. 22939

• Fix the Snyk module to work with the new API changes. 27358

• Fix a bug in http_endpoint that caused numbers encoded as strings. 27382 27480

Metricbeat

• Change server_status_path default setting to nginx_status for the nginx module. 26642

• Change startTime and endTime of GetMetricData API in cloudwatch metricset to be only one collection period apart. 27327

• Fix cloudwatch metricset collecting duplicate data points. 27248

• Add percent formatters to system/process. 27374

• Fix instance machineType reporting in compute metricset of GCP module. 27363

Added

Filebeat

• Update Elasticsearch module’s ingest pipeline for parsing new deprecation logs. 26857 26880

Beats version 7.14.0

View commits

Breaking changes

Affecting all Beats

• Removed beats central management 25696, 23908

• MacOSX minimum supported version set to 10.14 24193

Filebeat

• Change logging in logs input to structure logging. Some log message formats have changed. 25299

• All url.* fields apart from url.original in the Apache, Nginx, IIS, Traefik, S3Access, Cisco, F5, Fortinet, Google Workspace, Imperva, Microsoft, Netscout, O365, Sophos, Squid, Suricata, Zeek, Zia, Zoom, and ZScaler modules are now url unescaped due to using the Elasticsearch uri_parts processor. 24699

• Change source field for event.action in fortinet.firewall module to fortinet.firewall.action instead of fortinet.firewall.eventtype. 24816

• threatintel module: Changed the type of threatintel.indicator.first_seen from keyword to date. 26765

Heartbeat

• Add support for screenshot blocks and use newer synthetics flags that only works in newer synthetics betas. 25808

Metricbeat

• Adjust host fields to adopt new names from 1.9.0 ECS. 24312

Bugfixes

Affecting all Beats

• Omit full index template from errors that occur while loading the template. 25743

• In the script processor, the decode_xml and decode_xml_wineventlog processors are now available as DecodeXML and DecodeXMLWineventlog respectively.

• Fix encoding errors when using the disk queue on nested data with multi-byte characters 26484

Auditbeat

• file_integrity: Create fsnotify watcher only when starting file_integrity module 19505

• system/socket: Fix kprobe grouping to allow running more than one instance. 20325

• system/socket: Fixed a crash due to concurrent map read and write. 21192 21690

• auditd: Fix an error condition causing a lot of audit_send_reply kernel threads being created. 22673

• system/socket: Fixed start failure when run under config reloader. 20851 21693

• system/socket: Having some CPUs unavailable to Auditbeat could cause startup errors or event loss. 22827

Filebeat

• Fix mapping of fortinet.firewall.mem as integer. 19335

• Add shared_credential_file to cloudtrail config 15652 15656

• Fix integer overflow in S3 offsets when collecting very large files. 22523

• Fix issue with m365_defender, when parsing incidents that has no alerts attached: 25421

• Fix default config template values for paths on oracle module: 26276

• Fix Elasticsearch compatibility for modules that use copy_from in set processors. 26629

• Change type of max_bytes in all configs to be cfgtype.ByteSize 26699

• Change checkpoint.source_object from Long to Keyword. 25124 25145

• Fix Nginx module pipelines. 19088 24699

• Fix incorrect field name appending to related.hash in threatintel.abusechmalware ingest pipeline. 25151 25674

• Add improvements to the azure activitylogs and platformlogs ingest pipelines. 26148

• Fix kibana.log pipeline when event.duration calculation becomes a Long. 24556 25675

• Removed incorrect http.request.referrer field from aws.elb module. 26435 26441

• Fix threatintel.indicator.url.full not being populated. 26351 26508

• Fix Suricata metadata fields breaking visualizations, moved out of flattened datatype. 26710

• Fix httpjson template data key for url.params. 26848

• Cisco asa/ftd: Fix reversed usage of observer ingress and egress interfaces. 26265

• Fix aws.s3access pipeline when remote IP is a -. 26913 26940

• Fix service name in aws-cloudwatch input from cloudwatchlogs to logs. 27007

Heartbeat

• Add Context to otherwise ambiguous HTTP body read errors. 25499

Metricbeat

• Major refactor of system/cpu and system/core metrics. 25771

• Fix GCP Project ID being ingested as cloud.account.id in gcp.billing module 26357 26412

• Fix memory leak in SQL module when database is not available. 25840 26607

• Fix aws metric tags with resourcegroupstaggingapi paginator. 26385 26443

• Fix quoting in GCP billing table name 26855 26870

• Recover service.address field in vsphere module 26902 26904

Winlogbeat

• Fix related.ip field in renameCommonAuthFields 24892

Functionbeat

• Expose region in AWS configuration so Functionbeat can deploy the Lambda in the correct place. 26523

Added

Affecting all Beats

• Add support for defining explicitly named dynamic templates without path/type match criteria 25422

• Improve ES output error insights. 25825

• Add orchestrator.cluster.name/url fields as k8s metadata 26056

• Libbeat: report beat version to monitoring. 26214

• Ensure common proxy settings support in HTTP clients: proxy_disabled, proxy_url, proxy_headers and typical environment variables HTTP_PROXY, HTTPS_PROXY, NOPROXY. 25219

Filebeat

• Update PanOS module to parse Global Protect & User ID logs. 24722 24724 24927

• Add HMAC signature validation support for http_endpoint input. 24918

• Add new grok pattern for iptables module for Ubiquiti UDM 25615 25616

• Add multiline support to aws-s3 input. 25249 25710 25873

• Add monitoring metrics to the aws-s3 input. 25711

• Added network.direction fields to Zeek and Suricata modules using the add_network_direction processor 24620

• Add Content-Type override to aws-s3 input. 25697 25772

• In Cisco Umbrella fileset add users from cisco.umbrella.identities to related.user. 25776

• Add fingerprint processor to generate fixed ids for google_workspace events. 25841

• Update PanOS module to parse HIP Match logs. 24350 25686

• Support MongoDB 4.4 in filebeat’s MongoDB module. 20501 24774

• Enhance GCP module to populate orchestrator.* fields for GKE / K8S logs 25368

• Add log_group_name_prefix config into aws-cloudwatch input. 26187

• Move Filebeat azure module to GA. 26114 26168

• Make filestream input GA. 26127

• http_endpoint: Support multiple documents in a single request by POSTing an array or NDJSON format. 25764

• Add new parser to filestream input: container. 26115

• Add support for ISO8601 timestamps in Zeek fileset 25564

• Add possibility to include headers in resulting docs and preserve the original event in http_endpoint input 26279

• Add preserve_original_event option to o365audit input. 26273

• Add log.flags to events created by the aws-s3 input. 26267

• Add include_s3_metadata config option to the aws-s3 input for including object metadata in events. 26267

• RFC 5424 and UNIX socket support in the Syslog input are now GA 26293

• Update grok patterns for HA Proxy module 25827 25835

• Update PanOS module’s date processor formats to parse strict_date_optional_time_nanos. 26033 26158

• Update Okta module to parse additional fields to okta.debug_context.debug_data. 25689 25818

• Added dataset anomalithreatstream to the threatintel module to ingest indicators from Anomali ThreatStream 26350

• Add support for copytruncate method when rotating input logs with an external tool in filestream input. 23457

• Add uri_parts and user_agent ingest processors to aws.elb module. 26435 26441

• Added dataset recordedfuture to the threatintel module to ingest indicators from Recorded Future Connect API 26481

• Update fortinet ingest pipelines. 22136 25254 24816

• Release Filebeat Stack Monitoring modules as GA 26226

• Use default add_locale for fortinet.firewall 20300 26524

Heartbeat

• Add support for copytruncate method when rotating input logs with an external tool in filestream input. 23457

• Add proxy_headers to HTTP monitor. 25219

• Suppress too many bad message error logs when reading from corrupted journal for 5 seconds. 26224

• Add replicas.ready field to state_statefulset in Kubernetes module 26088

Metricbeat

• Refactor state_* metricsets to share response from endpoint. 25640

• Add server id to zookeeper events. 25550

• Add additional network metrics to docker/network 25354

• Migrate ec2 metricsets to use cloudwatch input. 25924

• Reduce number of requests done by kubernetes metricsets to kubelet. 25782

• Migrate rds metricsets to use cloudwatch input. 26077

• Migrate sqs metricsets to use cloudwatch input. 26117

• Collect linked account information in AWS billing. 26285

• Add total CPU to vSphere virtual machine metrics. 26167

• Add AWS Kinesis metricset. 25989

• Add Cluster filter on ECS Kubernetes overview dashboard and corresponding section on Kubernetes module documentation page. 26919

Packetbeat

• Add url.extension to HTTP events 25990 25999

Winlogbeat

• Changed the log level of the "Successfully published events" message from info to debug to reduce verbosity of the info logging level. To track event log reader activity use the published_events metric. 25617

Deprecated

Filebeat

• Deprecate the MISP module. The Threat Intel module should be used instead. 25240

Beats version 7.13.4

View commits

Bugfixes

Affecting all Beats

• Fix add_process_metadata processor complaining about valid pid fields not being valid integers. 26829 26830

Auditbeat

• Do not close filestream harvester if an unexpected error is returned when close.on_state_change.* is enabled. 26411

Filebeat

• Fix Elasticsearch compatibility for modules that use type: ip with convert processors. 26629 26676

• Fix Elasticsearch compatibility for modules that use the network_direction processor. 26629 26676

• Fix Elasticsearch compatibility for modules that use the registered_domain processor. 26629 26676

Beats version 7.13.3

View commits

Bugfixes

Filebeat

• Fix bug in aws-s3 input where the end of gzipped log files might have been discarded. 26260

• Clone value when copy fields in processors to avoid crash. 19206 20500

• Fix bug in httpjson that prevented first_event getting updated. 26407

• Fix bug in the Syslog input that misparsed rfc5424 days starting with 0. 26419

Beats version 7.13.2

View commits

Bugfixes

Affecting all Beats

• Fix ILM alias creation when write alias exists and initial index does not exist. 26146

• Fix ILM setup log reporting that a policy or an alias was created, even though the creation of any resource was disabled. 24046 24480

• Fix ILM alias not being created if setup.ilm.check_exists: false and setup.ilm.overwrite: true has been configured. 24480

• Allow cgroup self-monitoring to see alternate hostfs paths. 24334

• Fix make setup instructions for a new Beat. 24944

• Fix out-of-date FreeBSD vagrantbox. 25652

• Fix handling of file_selectors in aws-s3 input. 25792

• Include date separator in the filename prefix of dateRotator to make sure nothing gets purged accidentally. 26176

Auditbeat

• auditd: Fix kernel deadlock when netlink congestion causes "no buffer space available" errors. 26031 26032

Filebeat

• o365: Avoid mapping exception for Parameters and ExtendedProperties fields of string type. 26164

Beats version 7.13.1

View commits

Bugfixes

Auditbeat

• Mitigate deadlock is aws-s3 input when SQS visibility timeout is exceeded. 25750

• Fix httpjson cursor override with empty values by adding ignore_empty_value option. 25802

Filebeat

• Improve inode reuse handling by removing state for removed files more eagerly from the internal state table in the logs inputs. 25756

Beats version 7.13.0

View commits

Breaking changes

Affecting all Beats

• Use alias to report container image in k8s metadata. 24380

• Set cleanup_timeout to zero by default in docker and kubernetes autodiscover in all beats except Filebeat where it is kept to 60 seconds. 24681

• Update to ECS 1.9.0. 24909

Filebeat

• Changes filebeat httpjson input’s append transform to create a list even with only a single valuehttps://github.com/elastic/pull/25074[25074]

• Deprecated the cyberark module (replaced by cyberarkpas). 25261 25505

Metricbeat

• Store cloudfoundry.container.cpu.pct in decimal form and as scaled_float. 24219

• Remove index_stats.created field from Elasticsearch/index Metricset 25113

Bugfixes

Affecting all Beats

• Fix events being dropped if they contain a floating point value of NaN or Inf. 25051

• Fix templates being overwritten if there was an error when check for the template existance. 24332

• Add expand_keys to the list of permitted config fields for decode_json_fields 24862

• Fix discovery of short-living and failing pods in Kubernetes autodiscover 22718 24742

• Fix panic when overwriting metadata 24741

• Fix role_arn to work with access keys for AWS. 25446

• Fix community_id processor so that ports greater than 65535 aren’t valid. 25409

Auditbeat

• Fix o365 module config when client_secret contains special characters. 25058

Filebeat

• Fix date parsing in GSuite/login fileset. 24694

• Improve Cisco ASA/FTD parsing of messages 23766

• Better support for identity FW messages.

• Change network.bytes, source.bytes, and destination.bytes to long from integer since value can exceed integer capacity.

• Add descriptions for various processors for easier pipeline editing in Kibana UI.

• Fix usage of unallowed ECS event.outcome values in Cisco ASA/FTD pipeline. 24744.

• Fix IPtables Pipeline and Ubiquiti dashboard. 24878 24928

• Strip Azure Eventhub connection string in debug logs. {pulll}25066[25066]

• Updating Oauth2 flow for m365_defender fileset. 24829

• Fix o365 module config when client_secret contains special characters. 25058

• Fix s3 input when there is a blank line in the log file. 25357

• Remove space from field sophos.xg.trans_src_ ip. 25154 25250

• Fix checkpoint.action_reason when its a string, not a Long. 25575 25609

• Fix fortinet.firewall.addr when its a string, not an IP address. 25585 25608

Metricbeat

• Sort correctly the keys when accessing JMX through the Jolokia module 25631

• Change lookup_fields from metricset.host to service.address 15883

• Fix incorrect types of fields GetHits and Ops in NodeInterestingStats for Couchbase module in Metricbeat 21021 23287

• Fix GCP not able to request Cloudfunctions metrics if a region filter was set 24218

• Fix type of uwsgi.status.worker.rss type. 24468

• Accept text/plain type by default for prometheus client scraping. 24622

• Use working set bytes to calculate the pod memory limit pct when memory usage is not reported (ie. Windows pods). 25428

• Fix copy-paste error in libbeat docs. 25448

• Fix azure billing dashboard. 25554

Winlogbeat

• Change event.code and winlog.event_id from int to keyword. 25176

Added

Affecting all Beats

• Add wineventlog schema to decode_xml processor. 23910 24726

• Add new ECS 1.9 field cloud.service.name to add_cloud_metadata processor. 24993

• Libbeat: report queue capacity, output batch size, and output client count to monitoring. 24700

• Add kubernetes.pod.ip field in kubernetes metadata. 25037

• Discover changes in Kubernetes namespace metadata as soon as they happen. 25117

• Add decode_xml_wineventlog processor. 23910 25115

• Add new setting gc_percent for tuning the garbage collector limits via configuration file. 25394

• Add unit and metric_type properties to fields.yml for populating field metadata in Elasticsearch templates 25419

• Add new option suffix to logging.files to control how log files are rotated. 25464

• Validate that required functionality in Elasticsearch is available upon initial connection. 25351

Filebeat

• Support X-Forwarder-For in IIS logs. 192142

• Add support for logs generated by servers configured with log_statement and log_duration in PostgreSQL module. 24607

• Added fifteen new message IDs to Cisco ASA/FTD pipeline. 24744

• Added NTP fileset to Zeek module 24224

• Add proxy_url config for httpjson v2 input. 24615 24662

• Change okta.target to flattened field type. 24354 24636

• Added http.request.id to nginx/ingress_controller and elasticsearch/audit. 24994

• Add awsfargate module to collect container logs from Amazon ECS on Fargate. 25041

• New module cyberarkpas for CyberArk Privileged Access Security audit logs. 24803

• Add uri_parts processor to Apache, Nginx, IIS, Traefik, S3Access, Cisco, F5, Fortinet, Google Workspace, Imperva, Microsoft, Netscout, O365, Sophos, Squid, Suricata, Zeek, Zia, Zoom, and ZScaler modules ingest pipelines. 19088 24699

• New module zookeeper for Zookeeper service and audit logs 25061 25128

• Add parsing for haproxy.http.request.raw_request_line field 25480 25482

• Mark filestream input beta. 25560

• Add User Agent Parser for Azure Sign In Logs Ingest Pipeline 23201

Heartbeat

• Handle datastreams for fleet. 24223

• Add --sandbox option for browser monitor. 24172

• Support additional 'root' fields from synthetics. 24770

• Browser zip_url source type. 24714

Metricbeat

• Add support for Consul 1.9. 24123

• Add support for defining metrics_filters for prometheus module in hints. 24264

• Add support for PostgreSQL 10, 11, 12 and 13. 24402

• Add support for SASL/SCRAM authentication to the Kafka module. 24810

Winlogbeat

• Add support for sysmon v13 events 24 and 25. 24217 24945

Beats version 7.12.1

View commits

Breaking changes

Filebeat

• Possible values for Netflow’s locality fields (source.locality, destination.locality and flow.locality) are now internal and external, instead of private and public. 24272 24295

Bugfixes

Affecting all Beats

• Fix templates being overwritten if there was an error when check for the template existance. 24332

• Fix Kubernetes autodiscovery provider to correctly handle pod states and avoid missing event data 17223

• Fix inode removal tracking code when files are replaced by files with the same name 25002

• Fix mage GenerateCustomBeat instructions for a new beat 17679

• Fix bug with annotations dedot config on k8s not used 25111

• Fix negative Kafka partition bug 25048

Filebeat

• Properly update offset in case of unparasable line. 22685

• Fix Cisco ASA parser for message 722051. 24410

• Fix google_workspace pagination. 24668

• Fix netflow module ignoring detect_sequence_reset flag. 24268 24270

• Fix Cisco ASA parser for message 302022. 24405 24697

• Fix Cisco AMP @metadata._id calculation 24717 24718

• Fix date parsing in GSuite/login and Google Workspace/login filesets. 24694

• Fix gcp/vpcflow module error where input type was defaulting to file. 24719

• Improve PanOS parsing and ingest pipeline. 22413 22748 24799

• Fix S3 input validation for non amazonaws.com domains. 24420 24861

• Fix google_workspace and okta modules pagination when next page template is empty. 24967

• Fix gcp module field names to use gcp instead of googlecloud. 25038

Heartbeat

• Fix panic when initialization of ICMP monitors fail twice. 25073

Metricbeat

• Ignore unsupported derive types for filesystem metricset. 22501 24502

Added

Filebeat

• Updating field mappings for Cisco AMP module, fixing certain fields. 24661

• Add support for upper case field names in Sophos XG module 24693

• Add fail_on_template_error option for httpjson input. 24784

Beats version 7.12.0

View commits

Breaking changes

Filebeat

• Rename s3 input to aws-s3 input. 23469

Heartbeat

• Refactor synthetics configuration to new syntax. 23467

Bugfixes

Affecting all Beats

• Fix nested subfield handling in generated Elasticsearch templates. 23178 23183

• Fix CPU usage metrics on VMs with dynamic CPU config 23154

• Allow configuring credential_profile_name and shared_credential_file when using role_arn. 24174

• Fix panic with inline SSL when the certificate or key was smaller than 256 bytes. 23820 23858

Auditbeat

• system/login: Fixed offset reset on inode reuse. 24414

• system/login: Add additional offset check for utmp files. 24515

Filebeat

• CheckPoint Firewall module: Change event.severity JSON data type to a number because the field mapping is a long. 23424

• Cisco IOS: Change icmp.type/code and igmp.type JSON data types to strings because the fields mappings are keyword. 23424

• CrowdStrike Falcon: Change JSON field types to match the field mappings. 23424

• Fortinet Firewall: Drop fortinet.firewall.assignip when the value is "N/A". 23424

• Juniper SRX: Change JSON field types to match the field mappings. 23424

• Suricata EVE: Convert suricata.eve.flow_id to string because the field is a keyword in the mapping. 23424

• Zeek DNS: Ignore failures in data type conversions. And change dns.id JSON field to a string to match its keyword mapping. 23424

• Update filestream reader offset when a line is skipped. 23417

• Add check for empty values in azure module. 24156

• Change the event.created in Netflow events to be the time the event was created by Filebeat

• Fix Zoom module parameters for basic auth and url path. 23779

• Use rfc6587 framing for fortinet firewall and clientendpoint filesets when transferring over tcp. 23837

• Fix httpjson input logging so it doesn’t conflict with ECS. 23972

• Fix Logstash module handling of logstash.log.log_event.action field. 20709

• aws/s3access dataset was populating event.duration using the wrong unit. 23920

• Zoom module pipeline failed to ingest some chat_channel events. 23904

• Fix Netlow module issue with missing internal_networks config parameter. 24094 24110

• in httpjson input using encode_as "application/x-www-form-urlencoded" now sets Content-Type correctly 24331 24336

• Fix default scope in add_nomad_metadata. 24559

Metricbeat

• Add stack monitoring section to elasticsearch module documentation 23286

• Fix ec2 metricset fields.yml and the integration test 23726

• Unskip s3_request integration test. 23887

• Add system.hostfs configuration option for system module. 23831

Added

Affecting all Beats

• Honor kube event resysncs to handle missed watch events 22668

• Add autodiscover provider and metadata processor for Nomad. 14954 23324

• Add processors.rate_limit.n.dropped monitoring counter metric for the rate_limit processor. 23330

• Deprecate aws_partition config parameter for AWS, use endpoint instead. 23539

• Update the baseline version of Sarama (Kafka support library) to 1.27.2. 23595

• Add kubernetes.volume.fs.used.pct field. 23564

• Add the enable_krb5_fast flag to the Kafka output to explicitly opt-in to FAST authentication. 23629

• Added new decode_xml processor to libbeat that is available to all beat types. 23678

• Add deployment name in pod’s meta. 23610

• Added ECS 1.8 host.os.type field to add_host_metadata processor. 23513

• Add selector information in Kubernetes services' metadata. 23730

Auditbeat

• Improve file_integrity monitoring when a file is created/deleted in quick succession. 17347 22170

• system/host: Add new ECS 1.8 field os.type in host.os.type. 23513

• Update Auditbeat auditd module to ECS 1.8 23594 23118

Filebeat

• Add parsing of tcp flags to AWS vpcflow fileset 22820 23157

• Added support for first_event context in Filebeat httpjson input 23437

• Adding Threat Intel module 21795

• Added username parsing from Cisco ASA message 302013. 21196

• Added encode_as and decode_as options to httpjson along with pluggable encoders/decoders 23478

• Added feature to modules to adapt Ingest Node pipelines for compatibility with older Elasticsearch versions by removing unsupported processors. 23763

• Added support for Cisco AMP API as a new fileset. 22768

• Added RFC6587 framing option for tcp and unix inputs 23663 23724

• Added application/x-ndjson as decode option for httpjson input 23521

• Added application/x-www-form-urlencoded as encode option for httpjson input 23521

• Move aws-s3 input to GA. 23631

• Populate source.mac and destination.mac for Suricata EVE events. 23706 23721

• Added string splitting for httpjson input 24022

• Added Signatures fileset to Zeek module 23772

• Upgrade Cisco ASA/FTD/Umbrella to ECS 1.8.0. 23819

• Add new ECS user and categories features to google_workspace/gsuite 23118 23709

• Move crowdstrike JS processor to ingest pipelines and upgrade to ECS 1.8.0 23118 23875

• Update Filebeat auditd dataset to ECS 1.8.0. 23723 23118

• Updated microsoft defender_atp and m365_defender to ECS 1.8. 23897 23118

• Updated o365 module to ECS 1.8. 23118 23896

• Upgrade CEF module to ECS 1.8.0. 23832

• Upgrade fortinet/firewall to ECS 1.8 23118 23902

• Upgrade Zeek to ECS 1.8.0. 23118 23847

• Updated azure module to ECS 1.8. 23118 23927

• Update aws/s3access to ECS 1.8. 23118 23920

• Upgrade panw module to ECS 1.8 23118 23931

• Updated aws/cloudtrail fileset to ECS 1.8. 23118 23911

• Upgrade juniper/srx to ECS 1.8.0. 23118 23936

• Update mysqlenterprise module to ECS 1.8. 23118 23978

• Upgrade sophos/xg fileset to ECS 1.8.0. 23118 23967

• Upgrade system/auth to ECS 1.8 23118 23961

• Upgrade elasticsearch/audit to ECS 1.8 23118 24000

• Upgrade okta to ECS 1.8.0 and move js processor to ingest pipeline 23118 23929

• Update zoom module to ECS 1.8. 23904 23118

• Add fileset to ingest PostgreSQL CSV logs. 23334

• Add beta support for RFC 5424 to the Syslog input. 23954

Heartbeat

• Bundle synthetics dependencies with Heartbeat docker image. 23274

Heartbeat

• Update Journalbeat to ECS 1.8. 23737

Metricbeat

• Enrich events of state_service metricset with Kubernetes services' metadata. 23730

• Add support for Darwin/arm M1. 24019

• Check fields are documented in AWS metricsets. 23887

• Add container.image.name and containe.name ECS fields for state_container. 23802

• Add support for the MemoryPressure, DiskPressure, OutOfDisk and PIDPressure status conditions in state_node. 23905

Packetbeat

• Upgrade to ECS 1.8.0. 23783

• Add event.type: [connection] to flow events and include end for final flows. 24564

Functionbeat

• Provide more ways to set AWS credentials. 12464 23344

• Add support for multiple regions 21065

Heartbeat

• Add support for script processor. 23229

Winlogbeat

• Add Audit and Authentication Policy Change Events and related.ip information 20684

• Add new ECS 1.8 improvements. 23563

• Remove deprecated eventlogging API that was used for Windows XP/2003 and associated unused code. 24463

Deprecated

Affecting all Beats

• Selecting full in ssl.verification_mode option will not treat CommonName field in x509 certificates as a hostname when Subject Alternative Name is not present from v8.0. Please update your certificates so it contains at least one DNSName instead of relying on CommonName in the new major version of Beats.

Beats version 7.11.2

View commits

Bugfixes

Affecting all Beats

• Fix issue discovering docker containers and metadata after reconnections 24318

Filebeat

• Fix Okta default date formatting. 24018 24025

• Fix aws/vpcflow generating errors for empty logs or unidentified formats. 24167

• Add nodes to filebeat-kubernetes.yaml ClusterRole. 24051 24052

Metricbeat

• Add check for iis/application_pool metricset for nil worker process id values. 23605 23647

Beats version 7.11.1

View commits

Bugfixes

Filebeat

• Fix goroutines leak with some inputs in autodiscover. 23722

• Fix various processing errors in the Suricata module. 23236

Elastic Logging Plugin

• Fix out of date CLI flags on docs. 23628

Beats version 7.11.0

View commits

Breaking changes

Affecting all Beats

• Allow embedding of CAs, Certificate of private keys for anything that support TLS in ouputs and inputs. 21179

• Update to ECS 1.7.0. 22571

• Add support for SCRAM-SHA-512 and SCRAM-SHA-256 in Kafka output. 12867

Auditbeat

• Use ECS 1.7 ingress/egress network directions instead of inbound/outbound for system/socket. 22991

• Use ingress/egress instead of inbound/outbound for ECS 1.7 in auditd module. 23000

Filebeat

• Add fileset to ingest Kibana’s ECS audit logs. 22696

• Remove suricata.eve.timestamp alias field. 10535 22095

• Rename bad ECS field name tracing.trace.id to trace.id in aws elb fileset. 22571

• Fix parsing issues with nested JSON payloads in Elasticsearch audit log fileset. 22975

• Rename network.direction values in crowdstrike/falcon to ingress/egress. 23041

Heartbeat - Adds negative body match. 20728

Metricbeat

• Change cloud.provider from googlecloud to gcp. 21775

• Rename googlecloud module to gcp module. 22246

• Use ingress/egress instead of inbound/outbound for system/socket metricset. 22992

• Change types of numeric metrics from Kubelet summary api to double so as to cover big numbers. 23335

Packetbeat

• Update how Packetbeat classifies network directionality to bring it in line with ECS 1.7 22996

Winlogbeat

• Use ECS 1.7 ingress/egress instead of inbound/outbound network.direction in sysmon. 22997

Bugfixes

Affecting all Beats

• Fix memory leak and events duplication in docker autodiscover and add_docker_metadata. 21851

• Fix duplicated pod events in kubernetes autodiscover for pods with init or ephemeral containers. 22438

• Fix FileVersion contained in Windows exe files. 22581

• Log debug message if the Kibana dashboard can not be imported from the archive because of the invalid archive directory structure 12211, 13387

• Periodic metrics in logs will now report libbeat.output.events.active and beat.memstats.rss as gauges (rather than counters). 22877

• Use PROGRAMDATA environment variable instead of C:\ProgramData for windows install service 22874

• Fix reporting of cgroup metrics when running under Docker 22879

• Fix typo in config docs 23185

• Fix panic due to unhandled DeletedFinalStateUnknown in k8s OnDelete 23419

• Fix error loop with runaway CPU use when the Kafka output encounters some connection errors 23484

Auditbeat

• file_integrity: stop monitoring excluded paths 21278 21282

• Note incompatibility of system/socket on ARM. 23381

Filebeat

• Fix Zeek dashboard reference to zeek.ssl.server.name field. 21696

• Fix network.direction logic in zeek connection fileset. 22967

• Fix aws s3 overview dashboard. 23045

• Fix bad network.direction values in Fortinet/firewall fileset. 23072

• Fix Cisco ASA/FTD module’s parsing of WebVPN log message 716002. 22966

• Add support for organization and custom prefix in AWS/CloudTrail fileset. 23109 23126

• Simplify regex for organization custom prefix in AWS/CloudTrail fileset. 23203 23204

• Fix syslog header parsing in infoblox module. 23272 23273

• Fix concurrent modification exception in Suricata ingest node pipeline. 23534

• Fix handling of ModifiedProperties field in Office 365. 23777

Heartbeat

• Fixed missing tls fields when connecting to https via proxy. 15797 22190

Metricbeat

• Change Session ID type from int to string 22359

• Fix filesystem types on Windows in filesystem metricset. 22531

• Fix failiures caused by custom beat names with more than 15 characters 22550

• Update NATS dashboards to leverage connection and route metricsets 22646

• Fix rate metrics in Kafka broker metricset by using last minute rate instead of mean rate. 22733

• Update config in windows.yml file. 23027https://github.com/elastic/beats/pull/23327[23327]

• Fix metric grouping for windows/perfmon module 23489 23505

Packetbeat

• Fix SIP parser logic related to line length check. 23411

Winlogbeat

• Protect against accessing an undefined variable in Security module. 22937

• Add source.ip validation for event ID 4778 in the Security module. 19627

Added

Affecting all Beats

• Add istiod metricset. 21519

• Add support for OpenStack SSL metadata APIs in add_cloud_metadata. 21590

• Add cloud.account.id for GCP into add_cloud_metadata processor. 21776

• Add proxy metricset for istio module. 21751

• Add kubernetes.node.hostname metadata of Kubernetes node. 22189

• Enable always add_resource_metadata for Pods and Services of kubernetes autodiscovery. 22189

• Add add_resource_metadata option setting (always enabled) for add_kubernetes_metadata setting. 22189

• Add support for ephemeral containers in kubernetes autodiscover and add_kubernetes_metadata. 22389 22439

• Added support for wildcard fields and keyword fallback in beats setup commands. 22521

• Fix polling node when it is not ready and monitor by hostname 22666

• Add expand_keys option to decode_json_fields processor and json input, to recusively de-dot and expand json keys into hierarchical object structures 22849

• Update k8s client and release k8s leader lock gracefully 22919

• Improve event normalization performance 22974

• Add tini as init system in docker images 22137

• Added "detect_mime_type" processor for detecting mime types 22940

• Added "add_network_direction" processor for determining perimeter-based network direction. 23076

• Added new rate_limit processor for enforcing rate limits on event throughput. 22883

• Allow node/namespace metadata to be disabled on kubernetes metagen and ensure add_kubernetes_metadata honors host 23012

• Improve equals check. 22778

Auditbeat

• Add several improvements for auditd module for improved ECS field mapping 22647

• Add ECS 1.7 configuration categorization in certain events in auditd module. 23000

Filebeat

• Adding support for Oracle Database Audit Logs 21991

• Add max_number_of_messages config into s3 input. 21993

• Add SSL option to checkpoint module 19560

• Added support for MySQL Enterprise audit logs. 22273

• Rename googlecloud module to gcp module. 22214

• Rename awscloudwatch input to aws-cloudwatch. 22228

• Rename google-pubsub input to gcp-pubsub. 22213

• Copy tag names from MISP data into events. 21664

• Added TLS JA3 fingerprint, certificate not_before/not_after, certificate SHA1 hash, and certificate subject fields to Zeek SSL dataset. 21696

• Add platform logs in the azure filebeat module. 22371

• Added event.ingested field to data from the Netflow module. 22412

• Improve panw ECS url fields mapping. 22481

• Improve Nats filebeat dashboard. 22726

• Add support for UNIX datagram sockets in unix input. {issues}18632[18632] 22699

• Add http.request.mime_type for Elasticsearch audit log fileset. 22975

• Add new httpjson input features and mark old config ones for deprecation 22320

• Add configuration option to set external and internal networks for panw panos fileset 22998

• Add subbdomain fields for rsa2elk modules. 23035

• Add subdomain enrichment for suricata/eve fileset. 23011

• Add subdomain enrichment for zeek/dns fileset. 23011

• Add event.category "configuration" to auditd module events. 23010

• Add event.category "configuration" to gsuite module events. 23010

• Add event.category "configuration" to o365 module events. 23010

• Add event.category "configuration" to zoom module events. 23010

• Add network.direction to auditd/log fileset. 23041

• Add logic for external network.direction in sophos xg fileset 22973

• Preserve AWS CloudTrail eventCategory in aws.cloudtrail.event_category. 22776 22805

• Add top_level_domain enrichment for suricata/eve fileset. 23046

• Add top_level_domain enrichment for zeek/dns fileset. 23046

• Add observer.egress.zone and observer.ingress.zone for cisco/asa and cisco/ftd filesets. 23068

• Allow cisco/asa and cisco/ftd filesets to override network directionality based off of zones. 23068

• Allow cef and checkpoint modules to override network directionality based off of zones 23066

• Add network.direction to netflow/log fileset. 23052

• Add the ability to override network.direction based on interfaces in Fortinet/firewall fileset. 23072

• Add network.direction override by specifying internal_networks in gcp module. 23081

• Migrate microsoft/defender_atp to httpjson v2 config 23017

• Migrate microsoft/m365_defender to httpjson v2 config 23018

• Migrate okta to httpjson v2 config 23059

• Add support for Snyk Vulnerability and Audit API. 22677

• Misp improvements: Migration to httpjson v2 config, pagination and deduplication ID 23070

• Add Google Workspace module and mark Gsuite module as deprecated 22950

• Mark m365 defender, defender atp, okta and google workspace modules as GA 23113

• Added alternative_host option to google pubsub input 23215

Heartbeat

• Add mime type detection for http responses. 22976

Metricbeat

• Move s3_daily_storage and s3_request metricsets to use cloudwatch input. 21703

• Duplicate system.process.cmdline field with process.command_line ECS field name. 22325

• Add awsfargate module task_stats metricset to monitor AWS ECS Fargate. 22034

• Add connection and route metricsets for nats metricbeat module to collect metrics per connection/route. 22445

• Add unit file states to system/service 22557

• kibana module: stats metricset no-longer collects usage-related data. 22732

• Add more TCP states to Metricbeat system socket_summary. 14347

• Add io.ops in fields exported by system.diskio. 22066

• Adjust the Apache status fields in the fleet mode. 22821

• Add AWS Fargate overview dashboard. 22941

• Add process.state, process.cpu.pct, process.cpu.start_time and process.memory.pct. 22845

• Move IIS module to GA and map fields. 22609 23024

• Apache: convert status.total_kbytes to status.total_bytes in fleet mode. 23022

• Release MSSQL as GA 23146

Packetbeat

• Add support for overriding the published index on a per-protocol/flow basis. 22134

• Change build process for x-pack distribution 21979

• Tuned the internal queue size to reduce the chances of events being dropped. 22650

• Add support for "http.request.mime_type" and "http.response.mime_type". 22940

Winlogbeat

• Add file.pe and process.pe fields to ProcessCreate & LoadImage events in Sysmon module. 17335 22217

• Add dns.question.subdomain fields for sysmon DNS events. 22999

• Add additional event categorization for security and sysmon modules. 22988

• Add dns.question.top_level_domain fields for sysmon DNS events. 23046

Elastic Log Driver

• Add new winlogbeat security dashboard 18775

Deprecated

Filebeat

• The experimental modules for Citrix Netscaler and Symantec Endpoint Protection have been removed. As we continue to expand our coverage of common security data sources, we may consider supporting Citrix Netscaler and Symantec Endpoint Protection in a future release. 23129 23130

Known Issue

Beats version 7.10.2

View commits

Bugfixes

Filebeat

• Add JSON body check for SQS message. 21727

• Fix cisco umbrella module config by adding input variable. 22892

• Fix CredentialsJSON unpacking for gcp-pubsub and httpjson inputs. 23277

Metricbeat

• Change vsphere.datastore.capacity.used.pct value to betweeen 0 and 1. 23148

Beats version 7.10.1

View commits

Bugfixes

Affecting all Beats

• Fix denial of service flaw where a remote attacker could cause the {beats} process to crash by presenting a specially malformed TLS public key. For more information, see Beats 7.10.1 Security Update.

• Fix index template loading when the new index format is selected. 22482 22682

Auditbeat

• auditd: Fix error condition that caused a lot of audit_send_reply kernel threads to be created. 22673

• system/socket: Fix start failure when run under config reloader. 20851 21693

• system/socket: Fix startup error with some 5.x kernels. 18755 22787

• system/socket: Fix startup errors and event loss caused by some CPUs being unavailable to Auditbeat. 22827

Filebeat

• Fix missing variable when loading aws pipelines. 22645

• Fix parsing error by dropping aws.vpcflow.pkt_srcaddr and aws.vpcflow.pkt_dstaddr when equal to "-". 22721 22716

Heartbeat

• Replace the service_name monitor option with service.name, which is more correct. We will support the old option until 8.0. 20330

• Fix problem where the enabled: false setting on monitors prevented Heartbeat from starting. 22829

Metricbeat

• Stop generating NaN values from Cloud Foundry module to avoid errors in outputs. 22634

• Fix Logstash module to no longer emit redundant events when xpack.enabled: true is set. 22808

Added

Filebeat

• Add DNS response IP addresses to related.ip in Suricata module. 22291

Functionbeat

• Add support for parallelization factor for kinesis. 20727

Beats version 7.10.0

View commits

Breaking changes

Affecting all Beats

• Added certificate TLS verification mode to ignore server name mismatch. 12283 20293

• Remove redundant cloudfoundry.*.timestamp fields. This value is set in @timestamp. 21175

• Allow embedding of CAs, Certificate of private keys for anything that supports TLS in outputs and inputs 21179

• API address is a required setting in add_cloudfoundry_metadata. 21759

Auditbeat

• Change network.direction values to ECS recommended values (inbound, outbound). 12445 20695

• Docker container needs to be explicitly run as user root for auditing. 21202

• File integrity dataset no longer includes the leading dot in file.extension values (e.g. it will report "png" instead of ".png") to comply with ECS. 21644

Filebeat

• Cisco 18753

• CrowdStrike 19132

• Fortinet 19133

• iptables 18756

• Checkpoint 18754

• Netflow 19087

• Zeek 19113 (forwarded tag is not included by default)

• Suricata 19107 (forwarded tag is not included by default)

• CoreDNS 19134 (forwarded tag is not included by default)

• Envoy Proxy 19134 (forwarded tag is not included by default)

  • Move file metrics to dataset endpoint 19977

  • Fix PANW field spelling "veredict" to "verdict" on event.action 18808

  • Tracking session end reason in panw module. 18705

  • API address and shard ID are required settings in the Cloud Foundry input. 21759

Heartbeat

Journalbeat

Metricbeat

• Remove "invalid zero" metrics on Windows and Darwin, don’t report linux-only memory and disk I/O metrics when running under agent. 21457

• API address and shard ID are required settings in the Cloud Foundry module. 21759

Packetbeat

Winlogbeat

Functionbeat

Bugfixes

Affecting all Beats

• Remove unnecessary restarts of metricsets while using Node autodiscover 19974

• [Metricbeat][Kubernetes] Change cluster_ip field from ip to keyword. 20571

• [Autodiscover] Handle input-not-finished errors in config reload. 20915

• Orderly close processors when processing pipelines are not needed anymore to release their resources. 16349

• Fix parsing of expired licences. 21112 22180

Auditbeat

• auditd: Fix spelling of anomaly in event.category.

• auditd: Fix typo in event.action of removed-user-role-from. 19300

• auditd: Fix typo in event.action of used-suspicious-link. 19300

Filebeat

• Fix mapping of fortinet.firewall.mem as integer. 19335

• Fix auditd module syscall table for ppc64 and ppc64le. 20052

• Fix Filebeat OOMs on very long lines 19500, 19552

• Ignore missing in Zeek module when dropping unecessary fields. 19984

• Fix event.outcome logic for azure/siginlogs fileset 20254

• Improve validation checks for Azure configuration 20369 20389

• Fix event.kind for system/syslog pipeline 20365 20390

• Fix event.type for zeek/ssl and duplicate event.category for zeek/connection 20696

• Remove wrongly mapped tls.client.server_name from fortinet/firewall fileset. 20983

• Handle multiple upstreams in ingress-controller. 21215

• Provide backwards compatibility for the append processor when Elasticsearch is less than 7.10.0. 21159

• Fix checkpoint module when logs contain time field. 20567

• Fix syslog RFC 5424 parsing in the CheckPoint module. 21854

• Fix incorrect connection state mapping in zeek connection pipeline. 22151 22149

• Fix for field [source] not present as part of path [source.ip] error in azure pipelines. 22377

• Fix handing missing eventtime and assignip field being set to N/A for fortinet module. 22361

Heartbeat

• Add support for new service_name option to all monitors. 19932.

Journalbeat

Metricbeat

• Add support for azure light metricset app_stats. 20639

• Fix ec2 disk and network metrics to use Sum statistic method. 20680

• Fix ec2 disk and network metrics to use Sum statistic method. 20680

• Update fields.yml in the azure module, missing metrics field. 20918

• Disable Kafka metricsets based on Jolokia by default. They require a different configuration. 20989

• Fix timestamp handling in remote_write. 21166

• Visualization title fixes in aws, azure and googlecloud compute dashboards. 21098

• Fix retrieving resources by ID for the azure module. 21711 21707

• Use timestamp from CloudWatch API when creating events. 21498

• Report the correct windows events for system/filesystem 21758

• Fix regular expression in windows/permfon. 22146 21125

• Fix azure storage event format. 21845

• Fix panic in kubernetes autodiscover related to keystores 21843 21880

• [Kubernetes] Remove redundant dockersock volume mount 22009

• Revert change to report process.memory.rss as process.memory.wss on Windows. 22055

• Add interval information to monitor metricset in azure. 22152

• Remove io.time from windows 22237

• Fix instance name in perfmon metricset. 22218 22261

Packetbeat

• Add "network" to event.category 20364 20392

Winlogbeat

• Fix invalid IP addresses in DNS query results from Sysmon data. 18432 18436

• Fix event.outcome in the security module for non-English languages. 20079 20564

• Fields from Winlogbeat modules were not being included in index templates and patterns. 18983

• Protect against accessing undefined variables in Sysmon module. 22219 22236

Functionbeat

• Fix catchall bucket config errors by adding more validation. 17572 20887

• Fix Google Cloud Function configuration issue. 20864 22156

Added

Affecting all Beats

• Add minimum cache TTL for successful DNS responses. 18986

• Add support for DNS over TLS for the dns processor. 19321

• Add leader election for Kubernetes autodiscover. 20281

• Add capability of enriching process metadata with container id also for non-privileged containers in add_process_metadata processor. 19767

• Add replace_fields config option in add_host_metadata for replacing host fields. 20490 20464

• Add ingress controller dashboards. 21052

• Added experimental citrix module. 20820

• Added experimental cyberark module. 20820

• Added experimental proofpoint module. 20820

• Added experimental snort module. 20820

• Added experimental symantec module. 20820

• Added experimental dataset barracuda/spamfirewall. 20820

• Added experimental dataset cisco/meraki. 20820

• Added experimental dataset f5/bigipafm. 20820

• Added experimental dataset fortinet/fortimail. 20820

• Added experimental dataset fortinet/fortimanager. 20820

• Added experimental dataset juniper/netscreen. 20820

• Added experimental dataset sophos/utm. 20820

• Add Cloud Foundry tags in related events. 21177

• Cloud Foundry metadata is cached to disk. 20775

• Add option to select the type of index template to load: legacy, component, index. 21212

• Release add_cloudfoundry_metadata as GA. 21525

• Added Kafka version 2.2 to the list of supported versions. 22328

Auditbeat

• Add enrichment of auditd seccomp events with name of the architecture, syscall, and signal. 14055 19300

Filebeat

• Add support for reading auditd logs that are prefixed with node=. 19659

• Add event.ingested to all Filebeat modules. 20386

• Add event.ingested for Suricata module 20220

• Add support for custom header and headersecret for filebeat http_endpoint input 20435

• Convert httpjson to v2 input 20226

• Add event.ingested to all Filebeat modules. 20386

• Return error when log harvester tries to open a named pipe. 18682 20450

• Avoid goroutine leaks in Filebeat readers. 19193 20455

• Improve Zeek x509 module with x509 ECS mappings 20867

• Improve Zeek SSL module with x509 ECS mappings 20927

• Added new properties field support for event.outcome in azure module 20998

• Improve Zeek Kerberos module with x509 ECS mappings 20958

• Improve Fortinet firewall module with x509 ECS mappings 20983

• Improve Santa module with x509 ECS mappings 20976

• Improve Suricata Eve module with x509 ECS mappings 20973

• Added new module for Zoom webhooks 20414

• Add type and sub_type to panw panos fileset 20912

• Always attempt community_id processor on zeek module 21155

• Add related.hosts ecs field to all modules 21160

• Keep cursor state between httpjson input restarts 20751

• Convert aws s3 to v2 input 20005

• Add support for additional fields from V2 ALB logs. 21540

• Release Cloud Foundry input as GA. 21525

• New Cisco Umbrella dataset 21504

• New juniper.srx dataset for Juniper SRX logs. 20017

• Adding support for Microsoft 365 Defender (Microsoft Threat Protection) 21446

• Adding support for FIPS in s3 input 21446

• Update Okta documentation for new stateful restarts. 22091

• Use workers in aws-s3 input to process SQS messages. 27199

Heartbeat

• Add index and pipeline settings to monitor configurations. 20610

Journalbeat

Metricbeat

• Add state_statefulset metricset to Metricbeat recommended configuration for k8s. 17627

• Infer types in Prometheus remote_write. 19944

• Add cloud.instance.name into aws ec2 metricset. 20077

• Add host inventory metrics into aws ec2 metricset. 20171

• Add scope setting for Elasticsearch module, allowing it to monitor an Elasticsearch cluster behind a load-balancing proxy. 18539 18547

• Add state_daemonset metricset for Kubernetes Metricbeat module 20649

• Add host inventory metrics to googlecloud compute metricset. 20391

• Add host inventory metrics to azure compute_vm metricset. 20641

• Add host inventory metrics to system module. 20415

• Add billing data collection from Cost Explorer into aws billing metricset. 20527 20103

• Migrate compute_vm metricset to a light one, map cloud.instance.id field. 20889

• Request prometheus endpoints to be gzipped by default 20766

• Add latency config parameter into aws module. 20875

• Add billing metricset into googlecloud module. 20812 20738

• Release all kubernetes state metricsets as GA 20901

• Move compute_vm_scaleset to light metricset. 21038 20985

• Sanitize event.host. 21022

• Add support for different Azure Cloud environments in the metricbeat azure module. 21044 20988

• Add overview and platform health dashboards to Cloud Foundry module. 21124

• Release lambda metricset in aws module as GA. 21251 21255

• Add dashboard for pubsub metricset in googlecloud module. 21326 17137

• Move Prometheus query & remote_write to GA. 21507

• Map cloud data filed cloud.account.id to azure subscription. 21483 21381

• Expand unsupported option from namespace to metrics in the azure module. 21486

Packetbeat

• Add an example to packetbeat.yml of using the forwarded tag to disable

• Add 100-continue support 15830 19349

• Add initial SIP protocol support 21221

Functionbeat

Winlogbeat

Elastic Log Driver - Add support to change beat name, and support for Kibana Logs. 20522

Deprecated

• N/A

Beats version 7.9.3

View commits

Bugfixes

Affecting all Beats

• The o365audit input and o365 module now recover from an authentication problem or other fatal errors, instead of terminating. 21258

Auditbeat

• system/socket: Fix a crash due to concurrent map read and write. 21192 21690

Filebeat

• Add field limit check for AWS Cloudtrail flattened fields. 21388 21382

Metricbeat

• Fix remote_write flaky test. 21173

• Fix panic in Kubernetes autodiscovery caused by storing stateless keystores. 21843 21880

• Remove redundant dockersock volume mount to avoid problems on Kubernetes deployments that do not use docker as the container runtime. 22009

Beats version 7.9.2

View commits

Breaking changes

Affecting all Beats

• Autodiscover doesn’t generate any configuration when a variable is missing. Previously it generated an incomplete configuration. 20898

Bugfixes

Affecting all Beats

• Explicitly detect missing variables in autodiscover configuration, log them at the debug level. 20568 20898

• Fix libbeat.output.write.bytes and libbeat.output.read.bytes metrics of the Elasticsearch output. 20752 21197

Filebeat

• Provide backwards compatibility for the set processor when Elasticsearch is less than 7.9.0. 20908

• Fix an error updating file size being logged when EOF is reached. 21048

• Fix error when processing AWS Cloudtrail Digest logs. 21086 20943

Metricbeat

• The Kibana collector applies backoff when errored at getting usage stats 20772

• The elasticsearch/index metricset only requests wildcard expansion for hidden indices if the monitored Elasticsearch cluster supports it. 20938

• Fix panic index out of range error when getting AWS account name. 21101 21095

• Handle missing counters in the application_pool metricset. 21071

Functionbeat

• Do not need Google credentials if not required for the operation. 17329 21072

• Fix dependency issues of GCP functions. 20830 21070

Added

Affecting all Beats

• Add container ECS fields in kubernetes metadata. 20984

Beats version 7.9.1

View commits

Breaking changes

Affecting all Beats

• Removed experimental modules citrix, kaspersky, rapid7 and tenable. 20706

Bugfixes

Affecting all Beats

• Update replicaset group to apps/v1 15854

• Rename cloud.provider az value to azure inside the add_cloud_metadata processor. 20689

• Add missing country_name geo field in add_host_metadata and add_observer_metadata processors. 20796 20811

Filebeat

• Fix long registry migration times. 20717 20705

• Fix event types and categories in auditd module to comply with ECS 20652

• Update documentation in the azure module filebeat. 20815

Heartbeat

• Stop rescheduling tasks of stopped monitors. 20570

Metricbeat

• Updates vm_compute metricset with more info on guest metrics. 20448

• Add fallback for PdhExpandWildCardPathW failing in perfmon metricset. 20139 20630

• Fix resource tags in aws cloudwatch metricset 20326 20385

• Fill cloud.account.name with accountID if account alias doesn’t exist. 20736

Winlogbeat

• Fix duplicated field error when exporting index-pattern with migration.6_to_7.enabled. 20521 20540

• Fix event.outcome in the security module for non-English languages. 20079 20564

Added

Affecting all Beats

• Added support for more message types for Cisco ASA and FTD. 20565

Beats version 7.9.0

View commits

Breaking changes

Affecting all Beats

• Ensure dynamic template names are unique for the same field. 18849

Filebeat

• With the default configuration the cloud modules (AWS, Azure, Googlecloud, o365, Okta) will no longer send the host field that contains information about the host Filebeat is running on. This is because the host field specifies the host on which the event happened. 13920 18223

• With the default configuration the following modules will no longer send the host field. You can revert this change by configuring tags for the module and omitting forwarded from the list.

  • Cisco 18753

  • CrowdStrike 19132

  • Fortinet 19133

  • Iptables 18756

  • Checkpoint 18754

  • Netflow 19087

  • Zeek 19113 (forwarded tag is not included by default)

  • Suricata 19107 (forwarded tag is not included by default)

  • CoreDNS 19134 (forwarded tag is not included by default)

  • Envoy Proxy 19134 (forwarded tag is not included by default)

  • CEF module 13920 18223

  • Palo Alto Networks module 13920 18223

• Okta module now requires objects instead of JSON strings for the http_headers, http_request_body, pagination, rate_limit, and ssl variables. 18953

• Add oauth support for httpjson input. 18415 18892

• Add split_events_by option to httpjson input. 19246

• Add date_cursor option to httpjson input. 19483

• Add Gsuite module with SAML support. 19329

• Add Gsuite User Accounts support. 19329

• Add Gsuite Login audit support. 19702

• Add Gsuite Admin support. 19769

• Add Gsuite Drive support. 19704

• Add Gsuite Groups support. 19725

Metricbeat

• Move service config under metrics and simplify metric types. 18691

• Fix ECS compliance of user.id field in system/users metricset. 19019

• Rename googlecloud stackdriver metricset to metrics. 19718

Winlogbeat

• Add PowerShell module. Support for event ID’s: 400, 403, 600, 800, 4103, 4014, 4105, 4106. 16262 18526

• Fix PowerShell processing of downgraded engine events. 18966

• Fix unprefixed fields in fields.yml for PowerShell module. 18984

Bugfixes

Affecting all Beats

• Fix potential race condition in fingerprint processor. 18738

• Add better handling for Kubernetes Update and Delete watcher events. 18882

• Fix config reload metrics (libbeat.config.module.start/stops/running). 19168

• Fix metrics hints builder to avoid wrong container metadata usage when port is not exposed. 18979

• Server-side TLS config now validates that certificate and key settings are both specified. 19584

Auditbeat

• system/socket: Fix issue with dataset using 100% CPU and becoming unresponsive in some scenarios. 19033 19764

Filebeat

• Fix Kubernetes Watcher goroutine leaks when input config is invalid and input.reload is enabled. 18629 18630

• Okta module now sets the Elasticsearch _id field to the Okta UUID value contained in each system log to minimize the possibility of duplicating events. 18953

• Fix netflow module to support 7 bytepad for IPFIX template. 18098

• Fix improper nesting of session_issuer object in AWS cloudtrail fileset. 18894 18915

• Fix Cisco ASA 3020** and 106023 messages. 17964

• Add missing default_field: false to AWS filesets fields.yml. 19568

• Fix memory leak in tcp and unix input sources. 19459

• Fix Cisco ASA dissect pattern for 313008 & 313009 messages. 19149

• Fix bug with empty filter values in system/service. 19812

Metricbeat

• Fix incorrect usage of hints builder when exposed port is a substring of the hint. 19052

• Stop counterCache only when already started. 19103

• Remove dedot for tag values in AWS module. 19112 19221

• Fix empty field name errors in the application pool metricset. 19537

• Fix mapping of service start type in the service metricset of the Windows module. 19551

• Fix config example in the perfmon configuration files. 19539

• Fix k8s scheduler compatibility issue. 19699

• Fix SQL module mapping NULL values as string. 18955 18898

Packetbeat

• Fix process monitoring when ipv6 is disabled under Linux. 19941 19945

Added

Affecting all Beats

• Add initial instrument of Beats with APM GO Agent. 17938

• Add optional regex based cid extractor to add_kubernetes_metadata processor. 17360

• Add k8s keystore backend. 18096

• Change ownership of files in docker images so they can be used in secured environments. 12905

• Upgrade k8s.io/client-go and k8s keystore tests. 18817

• Add support for multiple sets of hints on autodiscover. 18883

• Add a configurable delay between retries when app metadata cannot be retrieved by add_cloudfoundry_metadata. 19181

• Add data type conversion in dissect processor for converting string values to other basic data types. 18683

• Add the ignore_failure configuration option to the dissect processor. 19464

• Add the overwrite_keys configuration option to the dissect processor. 19464

• Add support to trim captured values in the dissect processor. 19464

• Add the max_cached_sessions option to the script processor. 19562

Auditbeat

• Add ECS categorization info for Auditd module. 18596

Filebeat

• Add http_endpoint input. 18298

• Add observer.vendor, observer.product, and observer.type to Palo Alto Networks module events. 18223

• The logstash module can now automatically detect the log file format (JSON or plaintext) and process it accordingly. 9964 18095

• Improve ECS categorization field mappings in CoreDNS module. 16159 18424

• Improve ECS categorization field mappings in Envoyproxy module. 16161 18395

• Improve ECS categorization field mappings in Cisco module. 16028 18537

• The s3 input can now automatically detect gzipped objects. 18283 18764

• Add geoip AS lookup and improve ECS categorization in AWS cloudtrail fileset. 18644 18958

• Add support for v1 consumer API in Cloud Foundry input and use it by default. 19125

• Add new mode to multiline reader to aggregate constant number of lines. 18352

• Explicitly set ECS version in all Filebeat modules. 19198

• Add awscloudwatch input. 19025

• Add automatic retries and exponential backoff to httpjson input. 18956

• Change the Palo Alto Networks module to pass through (rather than drop) message types other than threat and traffic. 16815 19375

• Improve ECS categorization field mappings in Traefik module. 16183 19379

• Improve ECS categorization field mappings in Azure module. 16155 19376

• Add automatic retries and exponential backoff to httpjson input. 18956

• Add text and flattened versions of fields with unknown subfields in AWS cloudtrail fileset. 18866 19121

• Add Microsoft Defender ATP Module. 17997 19197

• Add initial support for configurable file identity tracking. 18748

• Add experimental dataset tomcat/log for Apache Tomcat logs. 19713

• Add experimental dataset netscout/sightline for Netscout Arbor Sightline logs. 19713

• Add experimental dataset barracuda/waf for Barracuda Web Application Firewall logs. 19713

• Add experimental dataset f5/bigipapm for F5 Big-IP Access Policy Manager logs. 19713

• Add experimental dataset bluecoat/director for Bluecoat Director logs. 19713

• Add experimental dataset cisco/nexus for Cisco Nexus logs. 19713

• Add experimental dataset citrix/virtualapps for Citrix Virtual Apps logs. 19713

• Add experimental dataset cylance/protect for Cylance Protect logs. 19713

• Add experimental dataset fortinet/clientendpoint for Fortinet FortiClient Endpoint Protection logs. 19713

• Add experimental dataset imperva/securesphere for Imperva Secure Sphere logs. 19713

• Add experimental dataset infoblox/nios for Infoblox Network Identity Operating System logs. 19713

• Add experimental dataset juniper/junos for Juniper Junos OS logs. 19713

• Add experimental dataset kaspersky/av for Kaspersky Anti-Virus logs. 19713

• Add experimental dataset microsoft/dhcp for Microsoft DHCP Server logs. 19713

• Add experimental dataset tenable/nessus_security for Tenable Nessus Security Scanner logs. 19713

• Add experimental dataset rapid7/nexpose for Rapid7 Nexpose logs. 19713

• Add experimental dataset radware/defensepro for Radware DefensePro logs. 19713

• Add experimental dataset sonicwall/firewall for Sonicwall Firewalls logs. 19713

• Add experimental dataset squid/log for Squid Proxy Server logs. 19713

• Add experimental dataset zscaler/zia for Zscaler Internet Access logs. 19713

Heartbeat

• Record HTTP response headers. 18327

Journalbeat

• Added an id config option to inputs to allow running multiple inputs on the same journal. 18467

• Add basic ECS categorization and log.syslog fields. 19176

Metricbeat

• Add client address to events from http server module. 18336

• Add new fields to HAProxy module. 18523

• Add Tomcat overview dashboard. 14026

• Accept prefix as metric_types config parameter in googlecloud stackdriver metricset. 19345

• Add dashboards for googlecloud load balancing metricset. 18369

• Add support for v1 consumer API in Cloud Foundry module and use it by default. 19268

• Add support for named ports in autodiscover. 19398

• Add param aws_partition to support aws-cn, aws-us-gov regions. 18850 19423

• Add support for wildcard * in dimension value of AWS CloudWatch metrics config. 18050 19660

• The elasticsearch/index metricset now collects metrics for hidden indices. 18639 18703

• Added performance and query metricsets to mysql module. 18955

• The elasticsearch-xpack/index metricset now reports hidden indices as such. 18639 18706

• Adds support for app insights metrics in the Azure module. 18570 18940

• Added cache and connection_errors metrics to status metricset of MySQL module. 16955 19844

• Update MySQL dashboard with connection errors and cache metrics. 19913 16955

Packetbeat

• Add an example to packetbeat.yml of using the forwarded tag to disable host metadata fields when processing network data from network tap or mirror port. 19209

• Add ECS fields for x509 certs, event categorization, and related IP info. 19167

Functionbeat

• Add basic ECS categorization and cloud fields. 19174

Elastic Log Driver

• Add support for docker logs command. 19531

Deprecated

Metricbeat

• Deprecate tags config parameter in cloudwatch metricset. 16733

• Deprecate tags.resource_type_filter config parameter and replace with resource_type. 19688

Beats version 7.8.1

View commits

Breaking changes

Filebeat

• Adds check on <no value> config option value for the azure input resource_manager_endpoint. 18890

Bugfixes

Affecting all Beats

• The monitoring.elasticsearch.api_key value is correctly base64-encoded before being sent to the monitoring Elasticsearch cluster. 18939 18945

• Fix kafka topic setting not allowing upper case characters. 18854 18640

• Fix redis key setting not allowing upper case characters. 18854 18640

Auditbeat

• system/package: Fix librpm loading on Fedora 31/32. NNNN

Filebeat

• Fix date and timestamp formats for fortigate module 19316

• Fix googlecloud.audit pipeline to only take in fields that are explicitly defined by the dataset. 18465 18472

• Fix a rate limit related issue in httpjson input for Okta module. 18530 18534

• Fix tls mapping in suricata module 19492 19494

Metricbeat

• Set tags correctly if the dimension value is ARN 19111 19433

• Fix bug incorrect parsing of float numbers as integers in Couchbase module 18949 19055

• Add missing info about the rest of the azure metricsets in the documentation. 19601

Added

Filebeat

• Add support for timezone offsets and Z to decode_cef timestamp parser. 19346

Metricbeat

• Update Couchbase to version 6.5 18595 19055

Beats version 7.8.0

View commits

Breaking changes

Affecting all Beats

• Introduce APM instrumentation, which is active when running the beat with ELASTIC_APM_ACTIVE=true. 17938

Filebeat

• Improve ECS field mappings in panw module. event.outcome now only contains success or failure, as recommended by the {ecs-ref}/ecs-event.html[ECS specification]. 16025 17910

• Improve ECS categorization field mappings for nginx module. http.request.referrer is now lowercase, and it is only populated when nginx sets a value. 16174 17844

• Improve ECS field mappings in santa module. hash.sha256 is moved to process.hash.sha256, and certificate fields are now under santa.certificate. 16180 17982

Bugfixes

Affecting all Beats

• Fix a bug in config reloading that could result in memory leaks or lost events when an output was rapidly reloaded multiple times. 10491 17381

• Fix panic when assigning a key to a nil value in an event. 18143

Heartbeat

• Fix TCP TLS checks to properly validate hostnames. In previous 7.x versions, this only worked for IP SANs. 17549

Metricbeat

• No longer send NaNs for memory metrics that don’t exist on the platform being monitored. 17400

• Add a switch to the driver definition on SQL module to use pretty names. 17378

Added

Affecting all Beats

• Update supported versions of redis output. 17198

• Add replace processor for replacing string values of fields. 17342

• Add urldecode processor for decoding URL-encoded fields. 17505

• Add support for AWS IAM role_arn in credentials config. 17658 12464

• Add Kerberos support to Elasticsearch output. 17927

• Set agent.name to the hostname by default. 16377 18000

• Add keystore support for autodiscover static configurations. 16306

• Add support for basic ECS logging. 17974

• Add config example of how to skip the add_host_metadata processor when forwarding logs. 13920 18153

• Add backoff configuration options for the Kafka output. 16777 17808

• Add keystore support for autodiscover static configurations. 16306

• Add Kerberos support to Elasticsearch output. 17927

• Add support for fixed length extraction in dissect processor. 17191

Auditbeat

• Add system module process dataset ECS categorization fields. 18032

• Add system module user dataset ECS categorization fields. 18035

• Add system module login dataset ECS categorization fields. 18034

• Add system module package dataset ECS categorization fields. 18033

• Add ECS categories for system module host dataset. 18031

• Add system module socket dataset ECS categorization fields. 18036

• Add file integrity module ECS categorization fields. 18012

• Add file.mime_type, file.extension, and file.drive_letter for file integrity module. 18012

Filebeat

• Add source field in k8s events. 17209

• Add new crowdstrike module for ingesting Crowdstrike Falcon streaming API endpoint event data. 16988

• Improve ECS categorization field mappings in mongodb module. 16170 17371

• Improve ECS categorization field mappings for mssql module. 16171 17376

• Improve ECS categorization field mappings for mysql module. 16172 17491

• Add new Checkpoint Syslog filebeat module. 17682

• Add config option to select a different azure cloud env in the azure-eventhub input and azure module. 17649 17659

• Enhance elasticsearch/server fileset to handle ECS-compatible logs emitted by Elasticsearch. 17715 17714

• Add Unix stream socket support as an input source and a syslog input source. 17492

• Improve ECS categorization field mappings in misp module. 16026 17344

• Enhance elasticsearch/deprecation fileset to handle ECS-compatible logs emitted by Elasticsearch. 17715 17728

• Make decode_cef processor GA. 17944

• Add new Fortigate Syslog filebeat module. 17890

• Improve ECS categorization field mappings in redis module. 16179 17918

• Improve ECS categorization field mappings in rabbitmq module. 16178 17916

• Improve ECS categorization field mappings in postgresql module. 16177 17914

• Improve ECS categorization field mappings for nginx module. 16174 17844

• Add support for Google Application Default Credentials to the Google Pub/Sub input and Google Cloud modules. 15668

• Improve ECS categorization field mappings for zeek module. 16029 17738

• Improve ECS categorization field mappings for netflow module. 16135 18108

• Add an input option publisher_pipeline.disable_host to disable host.name from being added to events by default. 18159

• Improve ECS categorization field mappings in system module. 16031 18065

• Improve ECS categorization field mappings in osquery module. 16176 17881

• Add support for v10, v11 and v12 logs on Postgres 13810 17732

• Add dashboard for Google Cloud Audit and AWS CloudTrail. 17379

Heartbeat

• Add additional ECS compatible fields for TLS information. 17687

Metricbeat

• Refactor windows/perfmon metricset configuration options and event output. 17596

• Add more detailed error messages, system tests and small refactoring to the service metricset in windows. 17725

• Stack Monitoring modules now auto-configure required metricsets when xpack.enabled: true is set. 16471 17609

• Add Metricbeat IIS module dashboards. 17966

• Add dashboard for the azure database account metricset. 17901

• Allow partial region and zone name in googlecloud module config. 17913

• Add aggregation aligner as a config parameter for googlecloud stackdriver metricset. 17141 17719

• Move the perfmon metricset to GA. 16608 17879

• Stack Monitoring modules now auto-configure required metricsets when xpack.enabled: true is set. 16471 17609

• Add static mapping for metricsets under aws module. 17614 17650

• Add dashboard for googlecloud storage metricset. 18172

• Collect new bulk indexing metrics from Elasticsearch when xpack.enabled:true is set. 17977 17992

• Remove requirement to connect as sysdba in Oracle module. 15846 18182

• Update MSSQL module to fix some SSPI authentication and add brackets to USE statements. 17862

Winlogbeat

• Set process.command_line and process.parent.command_line from Sysmon Event ID 1. 17327

• Add support for event IDs 4673,4674,4697,4698,4699,4700,4701,4702,4768,4769,4770,4771,4776,4778,4779,4964 to the Security module. 17517

• Add registry and code signature information and ECS categorization fields for sysmon module. 18058

Beats version 7.7.1

View commits

Bugfixes

Affecting all Beats

• Fix keystore add command hanging on Windows. 18649 18654

Filebeat

• Unescape filenames in SQS messages to resolve file paths correctly. 18370

• Improve failure handler for Cisco ASA and FTD pipelines to avoid mapping temporary fields. 18391 18392

• Fix source.address field not being set for the Nginx ingress_controller fileset. 18511

• Fix Google Cloud audit fileset to only take in fields that are explicitly defined by the fileset. 18465 18472

• Fix rate limit related issue in the httpjson input for the Okta module. 18530 18534

• Fix Cisco ASA and FTD parsing errors caused by NAT fields that contain a hostname instead of an IP. 14034 18376

• Fix PANW module to use correct mappings for bytes and packets counters. 18522 18525

• Fix Office 365 ingest failures caused by IP addresses surrounded by square brackets. 18587 18591

Metricbeat

• Fix tags_filter setting to work correctly for the AWS cloudwatch metricset. 18524

Added

Filebeat

• Add support for Google Application Default Credentials to the Google Pub/Sub input and Google Cloud modules. 15668

• Make decode_cef processor GA. 17944

Beats version 7.7.0

View commits

Breaking changes

Affecting all Beats

• Environment variables can no longer reference other environment variables or objects. 15937

• Change aws_elb autodiscovery provider field name from elb_listener. to aws.elb.. 16219 16402

• Remove support for using add_docker_metadata and add_kubernetes_metadata processors from the script processor. They can still be used as normal processors in the configuration. 16349 16514

Bugfixes

Affecting all Beats

• Fix Kubernetes autodiscovery provider to correctly handle pod states and avoid missing event data. 17223

• Fix add_cloud_metadata processor to better support modifying sub-fields with other processors. 13808

• Fix panic in the Logstash output when trying to send events to closed connection. 15568

• Fix logging target settings being ignored when Beats are started via systemd or docker. 12024 15442

• Fix issue where default go logger is not discarded when either * or stdout is selected. 10251 15708

• Remove superfluous use of number_of_routing_shards setting from the default template. 16038

• Automatically convert index names to lowercase. 16081

• Fix loading processor annotation hints, allowing the value to be a full configuration section. 16348

• Add ssl.ca_sha256 to the list of supported TLS options. This option allows you to check that a specific certificate is used as part of the verified chain. 15717

• Fix NewContainerMetadataEnricher to use default config for kubernetes module. No longer requires the user to have labels.dedot: true in the configuration as it is now properly the default. 16857

• Improve logging messages for the add_kubernetes_metadata processor. 16866

• Fail to start if httpprof is used and it cannot be initialized. 17028

• Fix concurrency issues in convert processor when used in the global context. 17032

• Fix bug with monitoring.cluster_uuid setting not always being exposed via GET /state Beats API. 16732 17420

• Fix building on FreeBSD by removing build flags from add_cloudfoundry_metadata processor. 17486

Filebeat

• Fix mapping error when zeek weird logs do not contain IP addresses. 15906

• Fix merging of fileset inputs to replace paths and append processors. 16450

• Fix Elasticsearch _id field set by S3 and Google Pub/Sub inputs. 17026

• Fix various Cisco FTD parsing issues. 16863 16889

• Fix default index pattern in IBM MQ Filebeat dashboard. 17146

• Fix a mapping exception when ingesting Logstash plain logs (7.4+) with pipeline ids containing non alphanumeric chars. 17242 17243

• Fix MySQL slowlog module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. 17086 17156

• Fix elasticsearch.audit data ingest pipeline to be more forgiving with date formats found in Elasticsearch audit logs. 17406

• Fix decoding errors caused by trailing spaces in CEF messages. 17253

• Fix activemq module causing "regular expression has redundant nested repeat operator" warning in Elasticsearch. 17428

Metricbeat

• Change lookup_fields setting from metricset.host to service.address. 15883

• Make logstash-xpack module once again have parity with internally-collected Logstash monitoring data. 16198

• Improve metrics collection in the system/service metricset on older linux distributions. 16902

• Use max in k8s apiserver dashboard aggregations. 17018

• Check if CCR feature is available on Elasticsearch cluster before attempting to call CCR APIs from elasticsearch/ccr metricset. 16511 17073

• Use max in k8s overview dashboard aggregations. 17015

• Fix Disk Used and Disk Usage visualizations in the Metricbeat System dashboards. 12435 17272

• Fix missing Accept header for Prometheus and OpenMetrics module. 16870 17291

• Combine cloudwatch aggregated metrics into single event. 17345

• Fix how we filter services by name in system/service. 17400

• Fix problem where cloudwatch metricset was not collecting tags correctly. 17419 17424

• Check if cpuOptions field is nil in DescribeInstances output in ec2 metricset. 17418

• Fix aws.s3.bucket.name terms_field in s3 overview dashboard. 17542

• Fix Unix socket path in memcached module. 17512

• Fix vsphere VM dashboard host aggregation visualizations. 17555

Added

Affecting all Beats

• Include network information by default when using the add_host_metadata or add_observer_metadata processor. 15347 16077

• Add aws_ec2 provider for autodiscovery. 12518 14823

• Add support for multiple passwords in redis output. 16058 16206

• Add support for Histogram type in fields.yml. 16570

• Windows .exe files now have embedded file version info. 15232

• Remove experimental flag from setup.template.append_fields. 16576

• Add add_cloudfoundry_metadata processor to annotate events with Cloud Foundry application data. 16621

• Add translate_sid processor on Windows for converting Windows security identifier (SID) values to names. 7451 16013

• Add support for Kubernetes provider to recognize namespace level defaults. 16321

• Add ability to enrich the container.id with the process id by using the add_process_metadata processor. 15947

• Update RPM packages contained in Beat Docker images. 17035

• Add Kerberos support to Kafka input and output. 16781

Auditbeat

• Add examples to the kubernetes manifests to show how to configure the auditd module and use processors to enrich events with metadata.

• In the kubernetes manifests, mount the data directory from the host, so data persist between executions in the same node. 17429

• Log to stderr when using kubernetes manifests. 174443

• Fix memory leak on when we miss socket close kprobe events. 17500

Filebeat

• Add ECS tls fields to the smtp, rdp, and ssl filesets in the zeek module, and the s3access and elb filesets in the aws module. 15757 15936

• Add Nginx ingress_controller fileset. 16197

• Add ECS tls and categorization fields to apache module. 16032 16121

• Add MQTT input. 15602 16204

• Improve ECS categorization, container, and process field mappings in auditd module. 16153 16280

• Add ECS categorization fields to activemq module. 16151 16201

• Improve ECS field mappings in aws module. 16154 16307

• Improve ECS categorization field mappings in googlecloud module. 16030 16500

• Add cloudwatch and ec2 filesets to aws module. 13716 16579

• Improve ECS categorization field mappings in kibana module. 16168 16652

• Add cloudfoundry input to send events from Cloud Foundry. 16586

• Improve ECS field mappings in haproxy module. 16162 16529

• Allow users to override pipeline ID in fileset input config. 9531 16561

• Improve ECS categorization field mappings in logstash module. 16169 16668

• Improve ECS categorization field mappings in iis module. 16165 16618

• Improve the decode_cef processor by reducing the number of memory allocations. 16587

• Improve ECS categorization field mapping in kafka module. 16167 16645

• Improve ECS categorization field mapping in icinga module. 16164 16533

• Improve ECS categorization field mappings in ibmmq module. 16163 16532

• Add custom string mapping to CEF module to support Forcepoint NGFW. 14663 15910

• Add ECS fields to CEF module. 16157 16338

• Improve ECS categorization and host field mappings in elasticsearch module. 16160 16469

• Improve ECS categorization field mappings in suricata module. 16181 16843

• Release ActiveMQ module as GA. 17047 17049

• Improve ECS categorization field mappings in iptables module. 16166 16637

• Add pattern for Cisco ASA / FTD Message 734001. 16212 16612

• Add o365audit input type for consuming events from Office 365 Management Activity API. 16196 16244

• Add custom string mapping to CEF module to support Check Point devices. 16041 16907

• Add o365 module for ingesting Office 365 management activity API events. 16196 16386

• Add Okta module. 16362

• Improve AWS cloudtrail field mappings. 16086 16110 17155

• Make the azure-eventhub input GA. 15671 17313

• Add access_key_id, secret_access_key, and session_token to the aws module config. 17456

Heartbeat

• Allow a list of status codes for HTTP checks. 15587

Journalbeat

• Improve parsing of syslog.pid in Journalbeat to strip the username when present. 16116

Metricbeat

• Add lambda metricset in aws module. 15260

• Add DynamoDB AWS light module. 15097

• Add IBM MQ light-weight module. 15301

• Add mixer metricset for Istio Metricbeat module. 15696

• Add mesh metricset for Istio Metricbeat module. 15535

• Add pilot metricset for Istio Metricbeat module. 15761

• Add galley metricset for Istio Metricbeat module. 15857

• Add key/value mode for SQL module. 15770 15845

• Add support for Unix socket in Memcached module. 13685 15822

• Make the system/cpu metricset collect normalized CPU metrics by default. 15618 15729

• Add kubernetes storage class support via kube-state-metrics. 16145

• Add up metric to prometheus metrics collected from host. 15948

• Add citadel metricset for Istio Metricbeat module. 15990

• Add support for processors in light modules. 14740 15923

• Add ability to collect AuroraDB metrics in rds metricset. 14142 16004

• Reuse connections in SQL module. 16001

• Improve the logstash module (when xpack.enabled is set to true) to use the override cluster_uuid returned by Logstash APIs. 15772 15795

• Add region parameter in googlecloud module. 15780 16203

• Add database_account azure metricset. 15758

• Add support for Dropwizard metrics 4.1. 16332

• Add support for NATS 2.1. 16317

• Add azure container metricset in order to monitor containers. 15751 16421

• Improve the haproxy module to support metrics exposed via HTTPS. 14579 16333

• Add filtering option for prometheus collector. 16420

• Add metricsets based on Ceph Manager Daemon to the ceph module. 7723 16254

• Add Load Balancing metricset to GCP. 15559

• Release statsd module as GA. 16447 14280

• Add collecting tags and tags_filter for rds metricset in aws module. 16605 16358

• Add OpenMetrics module. 16596

• Add redisenterprise module. 16482 15269

• Add cloudfoundry module to send events from Cloud Foundry. 16671

• Add system/users metricset as beta. 16569

• Align fields to ECS and add more tests for the azure module. 16024 16754

• Add additional cgroup fields to docker/diskio. 16638

• Add overview dashboard for googlecloud compute metricset. 16534 16819

• Add Prometheus remote write endpoint. 16609

• Release STAN module as GA. 16980

• Add query metricset for prometheus module. 17104

• Release ActiveMQ module as GA. 17047 17049

• Add support for CouchDB v2. 16352 16455

• Add dashboards for the azure container metricsets. 17194

• Separate the vpc metricset into three smaller metricsets: vpn, transitgateway, and natgateway. 16892

• Use Elasticsearch histogram type to store Prometheus histograms. 17061

• Allow to rate Prometheus counters when scraping them. 17061

• Release the Oracle module as GA. 14279 16833

• Add Storage metricsets to GCP module. 15598

• Release the vsphere module as GA. 15798 17119

• Add PubSub metricset to Google Cloud Platform module. 15536

• Add dashboard for redisenterprise module. 16752

• Add dashboard for VSphere host cluster and virtual machine. 14135

• Add test for documented fields check for metricsets without a http input. 17315 17334

• Release the azure module as GA. 17319

• In the kubernetes manifests, mount the data directory from the host, so data persist between executions in the same node. 17429

Packetbeat

• Add dns.question.subdomain and dns.question.top_level_domain fields. 14578

• Add redact_headers configuration option to allow HTTP request headers to be redacted whilst keeping the header field included in the Beat. 15353

• Enable setting promiscuous mode automatically. 11366

Winlogbeat

• Add Audit and Log Management, Computer Object Management, and Distribution Group related events to the Security module. 15217

• Add experimental event log reader implementation that should be faster in most cases. 6585 16849

Beats version 7.6.2

View commits

Breaking changes

Affecting all Beats

• Fix an issue that could cause redundant configuration reloads. 16440

• Fix metadata enrichers to use default config for kubernetes module. 17020

Metricbeat

• Make use of secure port when accessing Kubelet API 16063

Bugfixes

Affecting all Beats

• Fix k8s metadata issue regarding node labels not shown up on root level of metadata. 16834

Filebeat

• Ensure all zeek timestamps include millisecond precision. 14599 16766

• Fix issue where autodiscover hints default configuration was not being copied. 16987

Metricbeat

• Convert increments of 100 nanoseconds/ticks to milliseconds for WriteTime and ReadTime in diskio metricset (Windows) for consistency. 14233

• Fix diskio issue for windows 32 bit on disk_performance struct alignment. 16680

Added

Affecting all Beats

• Add monitoring variable libbeat.config.scans to distinguish scans of the configuration directory from actual reloads of its contents. 16440

Winlogbeat

• Add more DNS error codes to the Sysmon module. 15685

Beats version 7.6.1

View commits

Bugfixes

Affecting all Beats

• Fix k8s pods labels broken schema. 16480

• Fix k8s pods annotations broken schema. 16554

Filebeat

• Fix a connection error in httpjson input. 16123

• Fix mapping error for cloudtrail additionalEventData field 16088

• Rewrite azure filebeat dashboards, due to changes in kibana. 16466

• Adding the var definitions in azure manifest files, fix for errors when executing command setup. 16270 16468

Heartbeat

• Fix scheduler shutdown issues which would in rare situations cause a panic due to semaphore misuse. 16397

Metricbeat

• Avoid parsing errors returned from prometheus endpoints. 15712

• Change sqs metricset to use average as statistic method. 16438

Functionbeat

• Fix timeout option of GCP functions. 16282 16287

Added

Winlogbeat

• Made the event parser more lenient w.r.t. invalid event log definition version numbers. 15838

Beats version 7.6.0

View commits

Breaking changes

Affecting all Beats

• Remove version information from default ILM policy for improved upgrade experience on custom policies. 14745

• Running setup cmd respects setup.ilm.overwrite setting for improved support of custom policies. 14741

• Cleanup the x-pack licenser code to use the new license endpoint and the new format. Replaces the url /_xpack/license with /_license. 15091

• The document id fields has been renamed from @metadata.id to @metadata._id 15859

• Two Beat instances with the same data path cannot be run concurrently. 14069

Filebeat

• CEF extensions are now mapped to the data types defined in the CEF guide. 14342

Journalbeat

• Remove broken dashboard. 15288

Metricbeat

• Update cloudwatch metricset mapping for both metrics and dimensions. 15245

Packetbeat

• TLS: Fields have been changed to adapt to ECS. 15497

• TLS: The behavior of send_certificates and include_raw_certificates options has changed. 15497

Bugfixes

Affecting all Beats

• Fix spooling to disk blocking infinitely if the lock file can not be acquired. 15338

• Fix metricbeat test output with an ipv6 ES host in the output.hosts. 15368

• Fix convert processor conversion of string to integer with leading zeros. 15513 15557

• Fix existing agent.*, ecs.version, and host.name fields getting overwritten by Beats if they are already present in the original event. 14407

• Fix issue where TLS settings would be ignored when a forward proxy was in use. $15516

• Beats no longer attempts to load dashboards if they are unavailable. 15802

Auditbeat

• system/socket: Fix compatibility issue with kernel 5.x. 15771

Filebeat

• Fix a problem in Filebeat input httpjson where interval is not used as time.Duration. 14728

• Fix SSL config in input.yml for Filebeat httpjson input in the MISP module. 14767

• Check content-type when creating new reader in s3 input. 15252 15225

• Fix session reset detection and a crash in Netflow input. 14904

• Handle errors in handleS3Objects function and add more debug messages for s3 input. 15545

• netflow: Allow for options templates without scope fields. 15449

• netflow: Fix bytes/packets counters on some devices (NSEL and Netstream). 15449

• netflow: Fix compatibility with some Cisco devices by changing the field class_id from short to long. 15449

• Fix dashboard for Cisco ASA Firewall. 15420 15553

• Fix s3 input hanging with GetObjectRequest API call by adding context_timeout config. 15502 15590

• Add shared_credential_file to cloudtrail config. 15652 15656

• Fix typos in zeek notice fileset config file. 15764 15765

• Prevent Elasticsearch from spewing log warnings about redundant wildcards when setting up ingest pipelines for the elasticsearch module. 15840 15900

• Improve elasticsearch/audit fileset to handle timestamps correctly. 15942

Heartbeat

• Fix excessive memory usage introduced in 7.5 due to over-allocating memory for HTTP checks. 15639

Metricbeat

• Fix regular expression to detect instance name in perfmon metricset. 14273 14666

• Fix docker.container.size fields values 14979 15224

• Make kibana module more resilient to Kibana unavailability. 15258 15270

• Fix panic exception with some unicode strings in perfmon metricset. 15264

• Make logstash module more resilient to Logstash unavailability. 15276 15306

• Add username/password in Metricbeat autodiscover hints 15349

• Add dedot for tags in ec2 metricset and cloudwatch metricset. 15843 15844

• Use RFC3339 format for timestamps collected using the SQL module. 15847

• Add dedot for cloudwatch metric name. 15916 15917

• Fixed issue logstash-xpack module suddenly ceasing to monitor Logstash. 15974 16044

Added

Affecting all Beats

• Add a friendly log message when a request to docker has exceeded the deadline. 15336

• GA the script processor. 14325

• Add fingerprint processor. 11173 14205

• Add support for API keys in Elasticsearch outputs. 14324

• Add consumer_lag in Kafka consumergroup metricset 14822

• Make use of consumer_lag in Kafka dashboard 14863

• Refactor kubernetes autodiscover to enable different resource based discovery 14738

• Add add_id processor. 14524

• Enable TLS 1.3 in all beats. 12973

• Spooling to disk creates a lockfile on each platform. 15338

• Enable DEP (Data Execution Protection) for Windows packages. 15149

• Users can now specify monitoring.cloud. to override monitoring.elasticsearch. settings. 14399 15254

• Add support to kubernetes autodiscovery to add additional metadata from other source to events. 14875

• Update to ECS 1.4.0. 14844

• Add document_id setting to decode_json_fields processor. 15859

Filebeat

• Add new fileset googlecloud/audit for ingesting Google Cloud Audit logs. 15200

• Add dashboards to the CEF module (ported from the Logstash ArcSight module). 14342

• Add expand_event_list_from_field support in s3 input for reading json format AWS logs. 15357 15370

• Add azure-eventhub input which will use the azure eventhub go sdk. 14092 14882

• Expose more metrics of harvesters (e.g. read_offset, start_time). 13395

• Include log.source.address for unparseable syslog messages. 13268 15453

• Release aws elb fileset as GA. 15426 15380

• Integrate the azure-eventhub with filebeat azure module (replace the kafka input). 15480

• Release aws s3access fileset to GA. 15431 15430

• Add cloudtrail fileset to AWS module. 14657 15227

• New fileset googlecloud/firewall for ingesting Google Cloud Firewall logs. 14553

• google-pubsub input: ACK pub/sub message when acknowledged by publisher. 13346 14715

• Remove Beta label from google-pubsub input. 13346 14715

• Add dashboard for AWS ELB fileset. 15804

• Set event.outcome field based on googlecloud audit log output. 15731

• Add dashboard for AWS vpcflow fileset. 16007

Heartbeat

Metricbeat

• Expand data for the system/memory metricset 15492

• Add azure storage metricset in order to retrieve metric values for storage accounts. 14548 15342

• Add cost warnings for the azure module. 15356

• Release elb module as GA. 15485

• Add a system/network_summary metricset 15196

• Allow Metricbeat’s beat module to read monitoring information over a named pipe or unix domain socket. 14558

• Enable script processor. 14711

• Add STAN dashboard 15654

Functionbeat

• Add monitoring info about triggered functions. 14876

• Add Google Cloud Platform support. 13598

Beats version 7.5.2

View commits

Breaking changes

Journalbeat

• Remove broken dashboard. 15288

Bugfixes

Affecting all Beats

• Fix convert processor conversion of string to integer with leading zeros. 15513 15557

Filebeat

• Check content-type when creating new reader in s3 input. 15252 15225

• Fix session reset detection and a crash in Netflow input. 14904

• netflow: Allow for options templates without scope fields. 15449

• netflow: Fix bytes/packets counters on some devices (NSEL and Netstream). 15449

• netflow: Fix compatibility with some Cisco devices by changing the field class_id from short to long. 15449

• Fix dashboard for Cisco ASA Firewall. 15420 15553

Metricbeat

• Fix regular expression to detect instance name in perfmon metricset. 14273 14666

• Fix docker.container.size fields values 14979 15224

• Make kibana module more resilient to Kibana unavailability. 15258 15270

• Fix panic exception with some unicode strings in perfmon metricset. 15264

• Make logstash module more resilient to Logstash unavailability. 15276 15306

Added

Affecting all Beats

• Add a friendly log message when a request to docker has exceeded the deadline. 15336

Filebeat

• Include log.source.address for unparseable syslog messages. 13268 15453

Beats version 7.5.1

View commits

Bugfixes

Affecting all Beats

• Fix proxy_url option in Elasticsearch output. 14950

• Fix bug with potential concurrent reads and writes from event.Meta map by Kafka output. 14542 14568

• Fix license detection, when a beats successfully connect to Elasticsearch the detected license will be show in the log at info level. 15834

• Fix the parameters option configured in the Elasticsearch output so the values are added to the query string on bulk request. 18325

Filebeat

• Change iis url path grok pattern from URIPATH to NOTSPACE. 12710 13225 7951 13378 14754

• Fix azure filesets test files. 14185 14235

• Update Logstash module’s Grok patterns to support Logstash 7.4 logs. 14743

• Remove references to non-existent Zeek signatures fileset. 18878

Metricbeat

• Fix perfmon expanding counter path/adding counter to query when OS language is not english. 14684 14800

• Add extra check on ignore_non_existent_counters flag if the PdhExpandWildCardPathW returns no errors but does not expand the counter path successfully in windows/perfmon metricset. 14797

• Fix rds metricset from reporting same values for different instances. 14702

• Closing handler after verifying the registry key in diskio metricset. 14683 14759

• Fix docker network stats when multiple interfaces are configured. 14586 14825

• Fix ListMetrics pagination in aws module. 14926 14942

• Fix CPU count in docker/cpu in cases where no online_cpus are reported 15070

• Add domain state to kvm module 17673

• Fix Kubernetes Overview Dashboard to correctly display non 10s intervals for node usage 19675

Beats version 7.5.0

View commits

Breaking changes

Affecting all Beats

• By default, all Beats-created files and folders will have a umask of 0027 (on POSIX systems). 14119

Filebeat

Heartbeat

• JSON/Regex checks against HTTP bodies will only consider the first 100MiB of the HTTP body to prevent excessive memory usage. 14223

Metricbeat

Bugfixes

Affecting all Beats

• Disable add_kubernetes_metadata if no matchers found. 13709

• Better wording for xpack beats when the _xpack endpoint is not reachable. 13771

• Kubernetes watcher at add_kubernetes_metadata fails with StatefulSets 13905

• Fix panics that could result from invalid TLS certificates. This can affect Beats that connect over TLS or Beats that accept connections over TLS and validate client certificates. 14146

• Fix memory leak in kubernetes autodiscover provider and add_kubernetes_metadata processor happening when pods are terminated without sending a delete event. 14259

• Fix kubernetes metaGenerator.ResourceMetadata when parent reference controller is nil 14320 14329

Auditbeat

• Socket dataset: Fix start errors when IPv6 is disabled on the kernel. 13953 13966

Filebeat

• Fix a denial of service flaw when parsing malformed DSA public keys in Go. If {filebeat} is configured to accept incoming TLS connections with client authentication enabled, a remote attacker could cause the Beat to stop processing events. (CVE-2019-17596) See https://www.elastic.co/community/security/

• Fix timezone parsing of rabbitmq module ingest pipelines. 13879

• Fix conditions and error checking of date processors in ingest pipelines that use event.timezone to parse dates. 13883

• Fix timezone parsing of Cisco module ingest pipelines. 13893

• Fix timezone parsing of logstash module ingest pipelines. 13890

• Fix timezone parsing of iptables, mssql and panw module ingest pipelines. 13926

• Fixed increased memory usage with large files when multiline pattern does not match. 14068

• Fix azure fields names. 14098 14132

• Fix calculation of network.bytes and network.packets for bi-directional netflow events. 14111

• Accept '-' as http.response.body.bytes in apache module. 14137

• Fix timezone parsing of MySQL module ingest pipelines. 14130

• Improve error message in s3 input when handleSQSMessage failed. 14113

• Fix race condition in S3 input plugin. 14359

Heartbeat

• Fix storage of HTTP bodies to work when JSON/Regex body checks are enabled. 14223

Metricbeat

• Fix a denial of service flaw when parsing malformed DSA public keys in Go. If {metricbeat} is configured to accept incoming TLS connections with client authentication enabled, a remote attacker could cause the Beat to stop processing events. (CVE-2019-17596) See https://www.elastic.co/community/security/

• PdhExpandWildCardPathW will not expand counter paths in 32 bit windows systems, workaround will use a different function. 12590 12622

• Fix docker.cpu.system.pct calculation by using the reported number online cpus instead of the number of metrics per cpu. 13691

• Change kubernetes.event.message to text 13964

• Fix performance counter values for windows/perfmon metricset.https://github.com/elastic/beats/issues/14036[14036] 14039 14108

• Add FailOnRequired when applying schema and fix metric names in mongodb metrics metricset. 14143

• Convert indexed ms-since-epoch timestamp fields in elasticsearch/ml_job metricset to ints from float64s. 14220 14222

• Fix ARN parsing function to work for ELB ARNs. 14316

• Update azure configuration example. 14224

• Limit some of the error messages to the logs only 14317 14327

• Fix cloudwatch metricset with names and dimensions in config. 14376 14391

• Fix marshaling of ms-since-epoch values in elasticsearch/cluster_stats metricset. 14378

Packetbeat

• Fix parsing of the HTTP host header when it contains a port or an IPv6 address. 14215

Added

Affecting all Beats

• Fail with error when autodiscover providers have no defined configs. 13078

• Add autodetection mode for add_docker_metadata and enable it by default in included configuration fileshttps://github.com/elastic/pull/13374[13374]

• Add autodetection mode for add_kubernetes_metadata and enable it by default in included configuration files. 13473

• Use less restrictive API to check if template exists. 13847

• Do not check for alias when setup.ilm.check_exists is false. 13848

• Add support for numeric time zone offsets in timestamp processor. 13902

• Add condition to the config file template for add_kubernetes_metadata 14056

• Marking Central Management deprecated. 14018

• Add keep_null setting to allow Beats to publish null values in events. 5522 13928

• Add shared_credential_file option in aws related config for specifying credential file directory. 14157 14178

• Ensure that init containers are no longer tailed after they stop. 14394

• Libbeat HTTP’s Server can listen to a unix socket using the unix:///tmp/hello.sock syntax. 13655

• Libbeat HTTP’s Server can listen to a Windows named pipe using the npipe:///hello syntax. 13655

• Adding new Enterprise license type to the licenser. 14246

• Add endpoint config in AWS config to support using custom endpoint accessing AWS APIs. 16245 16263

Auditbeat

• Socket: Add DNS enrichment. 14004

Filebeat

• Add support for virtual host in Apache access logs 12778

• Update CoreDNS module to populate ECS DNS fields. 13320 13505

• Parse query steps in PostgreSQL slowlogs. 13496 13701

• Add filebeat azure module with activitylogs, auditlogs, signinlogs filesets. 13776

• Add support to set the document id in the json reader. 5844

• Add input httpjson. 13545 13546

• Filebeat Netflow input: Remove beta label. 13858

• Remove event.timezone from events that don’t need it in some modules that support log formats with and without timezones. 13918

• Add ExpandEventListFromField config option in the kafka input. 13965

• Add ELB fileset to AWS module. 14020

• Add module for MISP (Malware Information Sharing Platform). 13805

• Add filebeat azure module with activitylogs, auditlogs, signinlogs filesets. 13776 14033 14107

• Add support for all the ObjectCreated events in S3 input. 14077

• Add source.bytes and source.packets for uni-directional netflow events. 14111

• Add Kibana Dashboard for MISP module. 14147

• Add support for gzipped files in S3 input 13980

• Add Filebeat Azure Dashboards 14127

• Add support for space or time sync character before timestamp in syslog input. 13278 13269

• Add support for thread ID in Filebeat Kafka module. 19463

Heartbeat

• Add non-privileged icmp on linux and darwin(mac). 13795 11498

• Allow hosts to be used to configure http monitors 13703

Metricbeat

• Add refresh list of perf counters at every fetch 13091

• Add proc/vmstat data to the system/memory metricset on linux 13322

• Add support for NATS version 2. 13601

• Add docker.cpu.*.norm.pct metrics for cpu metricset of Docker Metricbeat module. 13695

• Add instance label by default when using Prometheus collector. 13737

• Add azure module. 13196 13859 13988

• Add Apache Tomcat module 13491

• Add ECS container.id and container.runtime to kubernetes state_container metricset. 13884

• Add job label by default when using Prometheus collector. 13878

• Add state_resourcequota metricset for Kubernetes module. 13693

• Add tags filter in ec2 metricset. 13872 13145

• Add cloud.account.id and cloud.account.name into events from aws module. 13551 13558

• Add metrics_path as known hint for autodiscovery 13996

• Leverage KUBECONFIG when creating k8s client. 13916

• Add ability to filter by tags for cloudwatch metricset. 13758 13145

• Release cloudwatch, s3_daily_storage, s3_request, sqs and rds metricset as GA. 14114 14059

• Add elasticsearch/enrich metricset. 14243 14221

• Add new dashboards for Azure vms, vm guest metrics, vm scale sets 14000

• Add vpc metricset for aws module. 16111 14854

Functionbeat

• Make bulk_max_size configurable in outputs. 13493

Winlogbeat

• Fill event.provider. 13937

• Add support for user management events to the Security module. 13530

Deprecated

Metricbeat

• kubernetes.container.id field for state_container is deprecated in favour of ECS container.id and container.runtime. 13884

Beats version 7.4.2

View commits

Bugfixes

Filebeat

• panw module: Use geo.name instead of geo.country_iso_code for free-form location. 13272

Beats version 7.4.1

View commits

Bugfixes

Affecting all Beats

• Recover from panics in the javascript process and log details about the failure to aid in future debugging. 13690

• Make the script processor concurrency-safe. 13690 13857

Filebeat

• Fixed early expiration of templates (Netflow v9 and IPFIX). 13821

• Fixed bad handling of sequence numbers when multiple observation domains were exported by a single device (Netflow V9 and IPFIX). 13821

• cisco asa and ftd filesets: Fix parsing of message 106001. 13891 13903

• Fix merging of fields specified in global scope with fields specified under an input’s scope. 3628 13909

• Fix delay in enforcing close_renamed and close_removed options. 13488 13907

• Fix missing netflow fields in index template. 13768 13914

• Fix cisco module’s asa and ftd filesets parsing of domain names where an IP address is expected. 14034

Metricbeat

• Mark Kibana usage stats as collected only if API call succeeds. 13881

Beats version 7.4.0

View commits

Breaking changes

Affecting all Beats

• Update to Golang 1.12.7. 12931

• Remove in_cluster configuration parameter for Kuberentes, now in-cluster configuration is used only if no other kubeconfig is specified 13051

Auditbeat

• Socket dataset: New implementation using Kprobes for finer-grained monitoring and UDP support. 13058

Filebeat

• Fix a race condition in the TCP input when close the client socket. 13038

• cisco/asa fileset: Renamed log.original to event.original and cisco.asa.list_id to cisco.asa.rule_name. 13286

• cisco/asa fileset: Fix parsing of 302021 message code. 13476

Metricbeat

• Add new Dashboard for PostgreSQL database stats 13187

• Add new dashboard for CouchDB database 13198

• Add new dashboard for Ceph cluster stats 13216

• Add new dashboard for Aerospike database stats 13217

• Add new dashboard for Couchbase cluster stats 13212

• Add new dashboard for Prometheus server stats 13126

• Add statistic option into cloudwatch metricset. If there is no statistic method specified, default is to collect Average, Sum, Maximum, Minimum and SampleCount. 12370 12840

• Fix rds metricset dashboard. 13721

Functionbeat

• Separate management and functions in Functionbeat. 12939

Bugfixes

Affecting all Beats

• ILM: Use GET instead of HEAD when checking for alias to expose detailed error message. 12886

• Fix unexpected stops on docker autodiscover when a container is restarted before cleanup_timeout. 12962 13127

• Fix some incorrect types and formats in field.yml files. 13188

• Load DLLs only from Windows system directory. 13234 13384

• Fix mapping for kubernetes.labels and kubernetes.annotations in add_kubernetes_metadata. 12638 13226

• Fix case insensitive regular expressions not working correctly. 13250

Auditbeat

• Host dataset: Export Host fields to gob encoder. 12940

Filebeat

• Fix filebeat autodiscover fileset hint for container input. 13296

• Fix incorrect references to index patterns in AWS and CoreDNS dashboards. 13303

• Fix timezone parsing of system module ingest pipelines. 13308

• Fix timezone parsing of elasticsearch module ingest pipelines. 13367

• Change iis url path grok pattern from URIPATH to NOTSPACE. 12710 13225 7951 13378

• Add timezone information to apache error fileset. 12772 13304

• Fix timezone parsing of nginx module ingest pipelines. 13369

• Allow path variables to be used in files loaded from modules.d. 13184

• Fix incorrect field references in envoyproxy dashboard 13420 13421

Heartbeat

• Fix integer comparison on JSON responses. 13348

Metricbeat

• Ramdisk is not filtered out when collecting disk performance counters in diskio metricset 12814 12829

• Fix redis key metricset dashboard references to index pattern. 13303

• Check if fields in DBInstance is nil in rds metricset. 13294 13037

• Fix silent failures in kafka and prometheus module. 13353 13252

• Fix module-level fields in Kubernetes metricsets. 13433 13544

• Fix panic in Redis Key metricset when collecting information from a removed key. 13426

• In the elasticsearch/node_stats metricset, if xpack is enabled, make parsing of ES node load average optional as ES on Windows doesn’t report load average. 12866

• Print errors that were being omitted in vSphere metricsets. 12816

• Fix issue with aws cloudwatch module where dimensions and/or namespaces that contain space are not being parsed correctly 13389

• Fix reporting empty events in cloudwatch metricset. 13458

• Fix data race affecting config validation at startup. 13005

Packetbeat

• Fix parsing the extended RCODE in the DNS parser. 12805

Functionbeat

• Fix Cloudwatch logs timestamp to use timestamp of the log record instead of when the record was processed 13291

• Look for the keystore under the correct path. 13332

Added

Affecting all Beats

• Add support for reading the network.iana_number field by default to the community_id processor. 12701

• Add a check so alias creation explicitely fails if there is an index with the same name. 13070

• Update kubernetes watcher to use official client-go libraries. 13051

• Add support for unix epoch time values in the timestamp processor. 13319

• add_host_metadata is now GA. 13148

• Add an ignore_missing configuration option the drop_fields processor. 13318

• Add registered_domain processor for deriving the registered domain from a given FQDN. 13326

• Add support for RFC3339 time zone offsets in JSON output. 13227

• Added monitoring.cluster_uuid setting to associate Beat data with specified ES cluster in Stack Monitoring UI. 13182

Filebeat

• Add netflow dashboards based on Logstash netflow. 12857

• Parse more fields from Elasticsearch slowlogs. 11939

• Update module pipelines to enrich events with autonomous system fields. 13036

• Add module for ingesting IBM MQ logs. 8782

• Add S3 input to retrieve logs from AWS S3 buckets. 12640 12582

• Add aws module s3access metricset. 13170 12880

• Update Suricata module to populate ECS DNS fields and handle EVE DNS version 2. 13320 13329

• Update PAN-OS fileset to use the ECS NAT fields. 13320 13330

• Add fields to the Zeek DNS fileset for ECS DNS. 13320 13324

• Add container image in Kubernetes metadata 13356 12688

• Add module for ingesting Cisco FTD logs over syslog. 13286

Heartbeat

• Record HTTP body metadata and optionally contents in http.response.body.* fields. 13022

Metricbeat

• Add Kubernetes proxy dashboard to Kubernetes module 12734

• Add Kubernetes controller manager dashboard to Kubernetes module 12744

• Add metrics to kubernetes apiserver metricset. 12922

• Add Kubernetes scheduler dashboard to Kubernetes module 12749

• Collect client provided name for rabbitmq connection. 12851 12852

• Add support to load default aws config file to get credentials. 12727 12708

• Add statistic option into cloudwatch metricset. 12370 12840

• Add support for kubernetes cronjobs 13001

• Add cgroup memory stats to docker/memory metricset 12916

• Add AWS elb metricset. 12952 11701

• Add AWS ebs metricset. 13167 11699

• Add metricset.period field with the configured fetching period. 13242 12616

• Add rate metrics for ec2 metricset. 13203

• Add Performance metricset to Oracle module 12547

• Use DefaultMetaGeneratorConfig in MetadataEnrichers to initialize configurations 13414

• Add module for statsd. 13109

Packetbeat

• Update DNS protocol plugin to produce events with ECS fields for DNS. 13320 13354

Functionbeat

• Add timeout option to reference configuration. 13351

• Configurable tags for Lambda functions. 13352

• Add input for Cloudwatch logs through Kinesis. 13317

• Enable Logstash output. 13345

Winlogbeat

• Add support for event ID 4634 and 4647 to the Security module. 12906

• Add network.community_id to Sysmon network events (event ID 3). 13034

• Add event.module to Winlogbeat modules. 13047

• Add event.category: process and event.type: process_start/process_end to Sysmon process events (event ID 1 and 5). 13047

• Add support for event ID 4672 to the Security module. 12975

• Add support for event ID 22 (DNS query) to the Sysmon module. 12960

• Add support for event ID 4634 and 4647 to the Security module. 12906

• Add network.community_id to Sysmon network events (event ID 3). 13034

• Add event.module to Winlogbeat modules. 13047

• Add event.category: process and event.type: process_start/process_end to Sysmon process events (event ID 1 and 5). 13047

• Add support for event ID 4672 to the Security module. 12975

• Add support for event ID 22 (DNS query) to the Sysmon module. 12960

• Add certain winlog.event_data.* fields to the index template. 13700 13704

Beats version 7.3.2

View commits

Bugfixes

Filebeat

• Fix filebeat autodiscover fileset hint for container input. 13296

• Fix timezone parsing of system module ingest pipelines. 13308

• Fix timezone parsing of elasticsearch module ingest pipelines. 13367

• Fix timezone parsing of nginx module ingest pipelines. 13369

Metricbeat

• Fix module-level fields in Kubernetes metricsets. 13433 13544

• Fix panic in Redis Key metricset when collecting information from a removed key. 13426

Beats version 7.3.1

View commits

Bugfixes

Affecting all Beats

• Fix install-service.ps1’s ability to set Windows service’s delay start configuration. 13173

• Fix decode_base64_field processor. 13092, 13144

Filebeat

• Fix multiline pattern in Postgres which was too permissive. 12078 13069

Metricbeat

• Fix logstash/node_stats metricset to also collect logstash_stats.events.duration_in_millis field when xpack.enabled: true is set. 13082

• Fix logstash/node metricset to also collect logstash_state.pipeline.representation.{type,version,hash} fields when xpack.enabled: true is set. 13133

Added

Metricbeat

• Make the beat module defensive about determining ES cluster UUID when xpack.enabled: true is set. 13020

Beats version 7.3.0

View commits

Breaking changes

Affecting all Beats

• Update to ECS 1.0.1. 12284 12317

• Default of output.kafka.metadata.full is set to false by now. This reduced the amount of metadata to be queried from a kafka cluster. 12738

Filebeat

• convert_timezone option is removed and locale is always added to the event so timezone is used when parsing the timestamp, this behaviour can be overriden with processors. 12410

Bugfixes

Affecting all Beats

• Fix typo in TLS renegotiation configuration and setting the option correctly 10871, 12354

• Add configurable bulk_flush_frequency in kafka output. 12254

• Fixed setting bulk max size in kafka output. 12254

• Add additional nil pointer checks to Docker client code to deal with vSphere Integrated Containers 12628

• Fix seccomp policy preventing some features to function properly on 32bit Linux systems. 12990 13008

Auditbeat

• Package dataset: Close librpm handle. 12215

• Package dataset: Improve dpkg parsing. 12325

• Host dataset: Fix reboot detection logic. 12591

• Add syscalls used by librpm for the system/package dataset to the default Auditbeat seccomp policy. 12578 12617

• Host dataset: Export Host fields to gob encoder. 12940

Filebeat

• Parse timezone in PostgreSQL logs as part of the timestamp 12338

• When TLS is configured for the TCP input and a certificate_authorities is configured we now default to required for the client_authentication. 12584

• Syslog input will now omit the process object from events if it is empty. 12700

• Apply max_message_size to incoming message buffer. 11966

Heartbeat

Journalbeat

• Iterate over journal correctly, so no duplicate entries are sent. 12716

• Preserve host name when reading from remote journal. 12714

Metricbeat

• Refactored Windows perfmon metricset: replaced method to retrieve counter paths with PdhExpandWildCardPathW, separated code by responsibility, removed unused functions 12212

• Validate that kibana/status metricset cannot be used when xpack is enabled. 12264

• In the kibana/stats metricset, only log error (don’t also index it) if xpack is enabled. 12265

• Fix an issue listing all processes when run under Windows as a non-privileged user. 12301 12475

• When TLS is configured for the http metricset and a certificate_authorities is configured we now default to required for the client_authentication. 12584

• Reuse connections in PostgreSQL metricsets. 12504 12603

• PdhExpandWildCardPathW will not expand counter paths in 32 bit windows systems, workaround will use a different function.https://github.com/elastic/beats/issues/12590[12590]12622

• Print errors that were being omitted in vSphere metricsets 12816

• In the elasticsearch/node_stats metricset, if xpack is enabled, make parsing of ES node load average optional as ES on Windows doesn’t report load average. 12866

• Fix incoherent behaviour in redis key metricset when keyspace is specified both in host URL and key pattern 12913

• Fix connections leak in redis module 12914 12950

Packetbeat

Added

Affecting all Beats

• Add proxy_disable output flag to explicitly ignore proxy environment variables. 11713 12243

• Processor add_cloud_metadata adds fields cloud.account.id and cloud.image.id for AWS EC2. 12307

• Add decode_base64_field processor for decoding base64 field. 11914

• Add aws overview dashboard. 11007 12175

• Add decompress_gzip_field processor. 12733

• Add timestamp processor for parsing time fields. 12699

• Add Oracle Tablespaces Dashboard 12736

• Add proxy_disable output flag to explicitly ignore proxy environment variables. 11713 12243

Auditbeat

Filebeat

• Add timeouts on communication with docker daemon. 12310

• Add specific date processor to convert timezones so same pipeline can be used when convert_timezone is enabled or disabled. 12253

• Add MSSQL module 12079

• Add ISO8601 date parsing support for system module. 12568 12579

• Update Kubernetes deployment manifest to use container input. 12632

• Add google-pubsub input type for consuming messages from a Google Cloud Pub/Sub topic subscription. 12746

• Add module for ingesting Cisco IOS logs over syslog. 12748

• Add module for ingesting Google Cloud VPC flow logs. 12747

• Report host metadata for Filebeat logs in Kubernetes. 12790

Metricbeat

• Add overview dashboard to Consul module 10665

• New fields were added in the mysql/status metricset. 12227

• Add Kubernetes metricset proxy. 12312

• Always report Pod UID in the pod metricset. 12345

• Add Vsphere Virtual Machine operating system to os field in Vsphere virtualmachine module. 12391

• Add CockroachDB module. 12467

• Add support for metricbeat modules based on existing modules (a.k.a. light modules) 12270 12465

• Add a system/entropy metricset 12450

• Add kubernetes metricset controllermanager 12409

• Allow redis URL format in redis hosts config. 12408

• Add tags into ec2 metricset. 12263 12372

• Add kubernetes metricset scheduler 12521

• Add Kubernetes scheduler dashboard to Kubernetes module 12749

• Add beat module. 12181 12615

• Collect tags for cloudwatch metricset in aws module. 12263 12480

• Add AWS RDS metricset. 11620 10054

• Add Oracle Module 11890

• Add Kubernetes proxy dashboard to Kubernetes module 12734

• Add Kubernetes controller manager dashboard to Kubernetes module 12744

Functionbeat

• Export automation templates used to create functions. 11923

• Configurable Amazon endpoint. 12369

Deprecated

Filebeat

• postgresql.log.timestamp field is deprecated in favour of @timestamp. 12338

Beats version 7.2.1

View commits

Bugfixes

Affecting all Beats

• Fix Central Management enroll under Windows 12797 12799

• Fixed a crash under Windows when fetching processes information. 12833

Filebeat

• Add support for client addresses with port in Apache error logs 12695

• Load correct pipelines when system module is configured in modules.d. 12340

Metricbeat

• Fix wrong uptime reporting by system/uptime metricset under Windows. 12915

Packetbeat

• Limit memory usage of Redis replication sessions. 12657

Beats version 7.2.0

View commits

Breaking changes

Affecting all Beats

• Update to Golang 1.12.4. 11782

Auditbeat

• Auditd module: Normalized value of event.category field from user-login to authentication. 11432

• Auditd module: Unset auditd.session and user.audit.id fields are removed from audit events. 11431 11815

• Socket dataset: Exclude localhost by default 11993

Filebeat

• Add read_buffer configuration option. 11739

Heartbeat

• Removed the add_host_metadata and add_cloud_metadata processors from the default config. These don’t fit well with ECS for Heartbeat and were rarely used.

Journalbeat

Metricbeat

• Add new option OpMultiplyBuckets to scale histogram buckets to avoid decimal points in final events 10994

• system/raid metricset now uses /sys/block instead of /proc/mdstat for data. 11613

Packetbeat

• Add support for mongodb opcode 2013 (OP_MSG). 6191 8594

• NFSv4: Always use opname ILLEGAL when failed to match request to a valid nfs operation. 11503

Winlogbeat

Functionbeat

Bugfixes

Affecting all Beats

• Ensure all beat commands respect configured settings. 10721

• Add missing fields and test cases for libbeat add_kubernetes_metadata processor. 11133, 11134

• decode_json_field: process objects and arrays only 11312

• decode_json_field: do not process arrays when flag not set. 11318

• Report faulting file when config reload fails. 11304

• Fix a typo in libbeat/outputs/transport/client.go by updating c.conn.LocalAddr() to c.conn.RemoteAddr(). 11242

• Management configuration backup file will now have a timestamps in their name. 11034

• [CM] Parse enrollment_token response correctly 11648

• Not hiding error in case of http failure using elastic fetcher 11604

• Escape BOM on JsonReader before trying to decode line 11661

• Fix matching of string arrays in contains condition. 11691

• Replace wmi queries with win32 api calls as they were consuming CPU resources 3249 and 11840

• Fix queue.spool.write.flush.events config type. 12080

• Fixed a memory leak when using the add_process_metadata processor under Windows. 12100

• Fix of docker json parser for missing "log" jsonkey in docker container’s log 11464

• Fixed Beat ID being reported by GET / API. 12180

• Add host.os.codename to fields.yml. 12261

• Fix @timestamp being duplicated in events if @timestamp is set in a processor (or by any code utilizing PutValue() on a beat.Event).

• Fix leak in script processor when using Javascript functions in a processor chain. 12600

Auditbeat

• Process dataset: Fixed a memory leak under Windows. 12100

• Login dataset: Fix re-read of utmp files. 12028

• Package dataset: Fixed a crash inside librpm after Auditbeat has been running for a while. 12147 12168

• Fix formatting of config files on macOS and Windows. 12148

• Fix direction of incoming IPv6 sockets. 12248

• Package dataset: Auto-detect package directories. 12289

• System module: Start system module without host ID. 12373

Filebeat

• Add support for Cisco syslog format used by their switch. 10760

• Cover empty request data, url and version in Apache2 modulehttps://github.com/elastic/pull/10730[10730]

• Fix registry entries not being cleaned due to race conditions. 10747

• Improve detection of file deletion on Windows. 10747

• Add missing Kubernetes metadata fields to Filebeat CoreDNS module, and fix a documentation error. 11591

• Reduce memory usage if long lines are truncated to fit max_bytes limit. The line buffer is copied into a smaller buffer now. This allows the runtime to release unused memory earlier. 11524

• Fix memory leak in Filebeat pipeline acker. 12063

• Fix goroutine leak caused on initialization failures of log input. 12125

• Fix goroutine leak on non-explicit finalization of log input. 12164

• Require client_auth by default when ssl is enabled for tcp input 12333

• Fix timezone offset parsing in system/syslog. 12529

Heartbeat

• Fix NPEs / resource leaks when executing config checks. 11165

• Fix duplicated IPs on mode: all monitors. 12458

Journalbeat

• Use backoff when no new events are found. 11861

Metricbeat

• Change diskio metrics retrieval method (only for Windows) from wmi query to DeviceIOControl function using the IOCTL_DISK_PERFORMANCE control code 11635

• Call GetMetricData api per region instead of per instance. 11820 11882

• Update documentation with cloudwatch:ListMetrics permission. 11987

• Check permissions in system socket metricset based on capabilities. 12039

• Get process information from sockets owned by current user when system socket metricset is run without privileges. 12039

• Avoid generating hints-based configuration with empty hosts when no exposed port is suitable for the hosts hint. 8264 12086

• Fixed a socket leak in the postgresql module under Windows when SSL is disabled on the server. 11393

• Change some field type from scaled_float to long in aws module. 11982

• Fixed RabbitMQ queue metricset gathering when consumer_utilisation is set empty at the metrics source 12089

• Fix direction of incoming IPv6 sockets. 12248

• Ignore prometheus metrics when their values are NaN or Inf. 12084 10849

• Require client_auth by default when ssl is enabled for module http metricset serverhttps://github.com/elastic/pull/12333[12333]

• The elasticsearch/index_summary metricset gracefully handles an empty Elasticsearch cluster when xpack.enabled: true is set. 12489 12487

Packetbeat

• Prevent duplicate packet loss error messages in HTTP events. 10709

• Fixed a memory leak when using process monitoring under Windows. 12100

• Improved debug logging efficiency in PGQSL module. 12150

Winlogbeat

Functionbeat

• Fix function name reference for Kinesis streams in CloudFormation templates 11646

Added

Affecting all Beats

• Add an option to append to existing logs rather than always rotate on start. 11953

• Add network condition to processors for matching IP addresses against CIDRs. 10743

• Add if/then/else support to processors. 10744

• Add community_id processor for computing network flow hashes. 10745

• Add output test to kafka output 10834

• Gracefully shut down on SIGHUP 10704

• New processor: copy_fields. 11303

• Add error.message to events when fail_on_error is set in rename and copy_fields processors. 11303

• New processor: truncate_fields. 11297

• Allow a beat to ship monitoring data directly to an Elasticsearch monitoring clsuter. 9260

• Updated go-seccomp-bpf library to v1.1.0 which updates syscall lists for Linux v5.0. NNNN

• Add add_observer_metadata processor. 11394

• Add decode_csv_fields processor. 11753

• Add convert processor for converting data types of fields. 8124 11686

• New extract_array processor. 11761

• Add number of goroutines to reported metrics. 12135

Auditbeat

• Auditd module: Add event.outcome and event.type for ECS. 11432

• Process: Add file hash of process executable. 11722

• Socket: Add network.transport and network.community_id. 12231

• Host: Fill top-level host fields. 12259

Filebeat

• Add more info to message logged when a duplicated symlink file is found 10845

• Add option to configure docker input with paths 10687

• Add Netflow module to enrich flow events with geoip data. 10877

• Set event.category: network_traffic for Suricata. 10882

• Allow custom default settings with autodiscover (for example, use of CRI paths for logs). 12193

• Allow to disable hints based autodiscover default behavior (fetching all logs). 12193

• Change Suricata module pipeline to handle destination.domain being set if a reverse DNS processor is used. 10510

• Add the network.community_id flow identifier to field to the IPTables, Suricata, and Zeek modules. 11005

• New Filebeat coredns module to ingest coredns logs. It supports both native coredns deployment and coredns deployment in kubernetes. 11200

• New module for Cisco ASA logs. 9200 11171

• Added support for Cisco ASA fields to the netflow input. 11201

• Configurable line terminator. 11015

• Add Filebeat envoyproxy module. 11700

• Add apache2(httpd) log path (/var/log/httpd) to make apache2 module work out of the box on Redhat-family OSes. 11887 11888

• Add support to new MongoDB additional diagnostic information 11952

• New module panw for Palo Alto Networks PAN-OS logs. 11999

• Add RabbitMQ module. 12032

• Add new container input. 12162

Heartbeat

• Enable add_observer_metadata processor in default config. 11394

Metricbeat

• Add AWS SQS metricset. 10684 10053

• Add AWS s3_request metricset. 10949 10055

• Add s3_daily_storage metricset. 10940 10055

• Add coredns metricbeat module. 10585

• Add SSL support for Metricbeat HTTP server. 11482 11457

• The elasticsearch.index metricset (with xpack.enabled: true) now collects refresh.external_total_time_in_millis fields from Elasticsearch. 11616

• Allow module configurations to have variants 9118

• Add timeseries.instance field calculation. 10293

• Added new disk states and raid level to the system/raid metricset. 11613

• Added path_name and start_name to service metricset on windows module 8364 11877

• Add check on object name in the counter path if the instance name is missing 6528 11878

• Add AWS cloudwatch metricset. 11798 11734

• Add regions in aws module config to specify target regions for querying cloudwatch metrics. 11932 11956

• Keep etcd followers members from reporting leader metricset events 12004

• Add validation for elasticsearch and kibana modules' metricsets when xpack.enabled is set to true. 12386

Functionbeat

• Add new options to configure roles and VPC. 11779

Winlogbeat

• Add support for reading from .evtx files. 4450

Deprecated

Filebeat

• Deprecate docker input in favor of container. 12162

Functionbeat

Known Issue

Journalbeat

Beats version 7.1.1

View commits

No changes in this release.

Beats version 7.1.0

View commits

• Updates to support changes to licensing of security features.

  Some Elastic Stack security features, such as encrypted communications, file and native authentication, and role-based access control, are now available in more subscription levels. For details, see https://www.elastic.co/subscriptions.

Beats version 7.0.1

View commits

Breaking changes

Metricbeat

• Change cloud.provider from ec2 to aws and from gce to gcp in add_cloud_metadata to align with ECS. 10775 11687

Bugfixes

Affecting all Beats

• Fix formatting for event.duration, "human readable" was not working well for this. 11675

• Fix initialization of the TCP input logger. 11605

Auditbeat

• Package dataset: Log error when Homebrew is not installed. 11667

Heartbeat

• Fix NPE on some monitor configuration errors. 11910

Metricbeat

• Change add_cloud_metadata processor to not overwrite cloud field when it already exist in the event. 11612 11305

Added

Auditbeat

• Add support to the system package dataset for the SUSE OS family. 11634

Deprecated

Metricbeat

• Prevent the docker/memory metricset from processing invalid events before container start 11676

libbeat/docs/release-notes/7.0.0.asciidoc

Beats version 7.0.0-GA

View commits

The list below covers the changes between 7.0.0-rc2 and 7.0.0 GA only.

Bugfixes

Affecting all Beats

• Relax validation of the X-Pack license UID value. 11640

• Fix a parsing error with the X-Pack license check on 32-bit system. 11650

• Fix ILM policy always being overwritten. 11671

• Fix template always being overwritten. 11671

Auditbeat

• Package dataset: Nullify Librpm’s rpmsqEnable. 11628

Filebeat

• Fix add_docker_metadata source matching, using log.file.path field now. 11577

Beats version 7.0.0-rc2

Check the HEAD diff

Breaking changes

Auditbeat

• Process dataset: Only report processes with executable. 11232

• Shorten entity IDs. 11405

Metricbeat

• Add connection and request timeouts for HTTP helper. 11032

Bugfixes

Affecting all Beats

• Fixed OS family classification in add_host_metadata for Amazon Linux, Raspbian, and RedHat Linux. 9134 11494

• Allow 'ilm.rollover_alias' to expand global fields like agent.version. 12233

Auditbeat

• Package dataset: dlopen versioned librpm shared objects. 11565

Filebeat

• Don’t apply multiline rules in Logstash json logs. 11346

• Fix panic in add_kubernetes_metadata processor when key log does not exist. 11543 11549

• Fix goroutine leak happening when harvesters are dynamically stopped. 11263

Metricbeat

• Add _bucket to histogram metrics in Prometheus Collector 11578

Added

Auditbeat

• Login dataset: Add event category and type. 11339

Filebeat

• Add support for MySQL 8.0, Percona 8.0 and MariaDB 10.3. 11417

Beats version 7.0.0-rc1

Check the HEAD diff

Breaking changes

Affecting all Beats

• On Google Cloud Engine (GCE) the add_cloud_metadata will now trim the project info from the cloud.machine.type and cloud.availability_zone. 10968

• Add cleanup_timeout option to docker autodiscover, to wait some time before removing configurations after a container is stopped. 10374 10905

• Empty meta.json file will be treated as a missing meta file. 8558

• Rename migration.enabled config to migration.6_to_7.enabled. 11284

• Initialize the Paths before the keystore and save the keystore into data/{beatname}.keystore. 10706

• Beats Xpack now checks for Basic license on connect. 11296

Auditbeat

• Process dataset: Only report processes with executable. 11232

Filebeat

• Set ecs: true in user_agent processors when loading pipelines with Filebeat 7.0.x into Elasticsearch 6.7.x. 10655 10875

Metricbeat

• Migrate docker module to ECS. 10927

Functionbeat

• Correctly extract Kinesis Data field from the Kinesis Record. 11141

Bugfixes

Affecting all Beats

• Reconnections of Kubernetes watchers are now logged at debug level when they are harmless. 10988

• Add missing host.* fields to fields.yml. 11016

• Include ip and boolean type when generating index pattern. 10995

• Using an environment variable for the password when enrolling a beat will now raise an error if the variable doesn’t exist. 10936

• Cancelling enrollment of a beat will not enroll the beat. 10150

• Allow to configure Kafka fetching strategy for the topic metadata. 10682

Auditbeat

• Package: Disable librpm signal handlers. 10694

• Login: Handle different bad login UTMP types. 10865

• System module: Fix and unify bucket closing logic. 10897

• User dataset: Numerous fixes to error handling. 10942

Filebeat

• Fix errors in filebeat Zeek dashboard and README files. Add notice.log support. 10916

• Fix a bug when converting NetFlow fields to snake_case. 10950

• Add on_failure handler for Zeek ingest pipelines. Fix one field name error for notice and add an additional test case. 11004 11105

• Fix issue preventing docker container events to be stored if the container has a network interface without ip address. 11225 11247

• Change URLPATH grok pattern to support brackets. 11135 11252

• Add support for iis log with different address format. 11255 11256

Heartbeat

• Fix checks for TCP send/receive data 11118

Metricbeat

• Migrate docker autodiscover to ECS. 10757 10862

• Fix issue in kubernetes module preventing usage percentages to be properly calculated. 10946

• Fix for not reusable http client leading to connection leaks in Jolokia module 11014

• Fix parsing error using GET in Jolokia module. 11075 11071

• Collect metrics when EC2 instances are not in running state. 11008 11023

• Change ECS field cloud.provider to aws. 11023

• Add documentation about jolokia autodiscover fields. 10925 10979

• Add missing aws.ec2.instance.state.name into fields.yml. 11219 11221

• Fix ec2 metricset to collect metrics from Cloudwatch with the same timestamp. 11142

• Fix potential memory leak in stopped docker metricsets 11294

Packetbeat

• Avoid reporting unknown MongoDB opcodes more than once. 10878

Winlogbeat

• Prevent Winlogbeat from dropping events with invalid XML. 11006

• Fix Winlogbeat escaping CR, LF and TAB characters. 11328 11357

Functionbeat

Added

Affecting all Beats

• Add ip fields to default_field in Elasticsearch template. 11035

Auditbeat

• Move System module to beta. 10800

Filebeat

• Add ISO8601 timestamp support in syslog metricset. 8716 10736

• Add support for loading custom NetFlow and IPFIX field definitions to netflow input. 10945 11223

• Added categorization fields for SSH login events in the system/auth fileset. 11334

Metricbeat

• Add filters and pie chart for AWS EC2 dashboard. 10596

Winlogbeat

• Add an index option to all event logs to specify the output index for events from that source. 15062

Known Issue

Journalbeat

• Journalbeat requires at least systemd v233 in order to follow entries after journal changes (rotation, vacuum).

Beats version 7.0.0-beta1

Check the HEAD diff

Breaking changes

Affecting all Beats

• Embedded html is not escaped anymore by default. 9914

• Remove port settings from Logstash and Redis output. 9934

• Rename process.exe to process.executable in add_process_metadata to align with ECS. 9949

• Import ECS change ecs#308: leaf field user.group is now the group field set. 10275

• Update the code of Central Management to align with the new returned format. 10019

• Docker and Kubernetes labels/annotations will be "dedoted" by default. 10338

• Remove --setup command line flag. 10138

• Remove --version command line flag. 10138

• Remove --configtest command line flag. 10138

• Move output.elasticsearch.ilm settings to setup.ilm. 10347

• ILM will be available by default if Elasticsearch > 7.0 is used. 10347

Auditbeat

• Rename process.exe to process.executable in auditd module to align with ECS. 9949

• Rename process.cwd to process.working_directory in auditd module to align with ECS. 10195

• Change data type of process.pid and process.ppid to number in JSON output of the auditd module. 10195

• Change data type of file.uid and file.gid to string in JSON output of the FIM module. 10195

• Field file.origin changed type from text to keyword. 10544

• Rename user fields to ECS in auditd module. 10456

• Rename event.type to auditd.message_type in auditd module because event.type is reserved for future use by ECS. 10536

• Rename auditd.messages to event.original and auditd.warnings to error.message. 10577

Filebeat

• Rename many kibana.log.* fields to map to ECS. 9301

• Modify apache/error dataset to follow ECS. 8963

• Rename many traefik.access.* fields to map to ECS. 9005

• Fix parsing of GC entries in elasticsearch server log. 9513 9810

• Rename read_timestamp to event.created for Redis input. 9924

• Rename a few elasticsearch.audit.* fields to map to ECS. 9293

• Rename read_timestamp to event.created for all Filebeat modules using it. 10139

• Rename many iis.error.* fields to map to ECS. 9955

• Adjust fileset haproxy.log to map to ECS. 10143

• Rename a few logstash.* fields to map to ECS, remove logstash.slowlog.message. 9935

• Rename a few mongodb.* fields to map to ECS. 10009

• Rename a few mysql.* fields to map to ECS. 10008

• Rename a few nginx.error.* fields to map to ECS. 10007

• Rename many auditd.log.* fields to map to ECS. 10192

• Filesets with multiple ingest pipelines added in 8914 only work with Elasticsearch >= 6.5.0 10001

• Remove service.name from Elastcsearch module. Replace by service.type. 10042

• Remove numeric coercions for user.id and group.id. IDs should be keyword. 10233

• Add grok pattern to support redis 5.0.3 log timestamp. 9819 10033

• Now save the 'first seen' timestamp in event.created (previously read_timestamp), instead of saving the parsed date. Now aligned with event.created semantics elsewhere. 10139

• Rename mysql.error.thread_id and mysql.slowlog.id to mysql.thread_id. 10161

• Remove mysql.error.timestamp and mysql.slowlog.timestamp. 10161

• Migrate multiple fields to event.duration, from modules "apache", "elasticsearch", "haproxy", "iis", "kibana", "mysql", "nginx", "postgresql" and "traefik", including http.response.elapsed_time (ECS). 10188, 10274

• Rename multiple fields to http.response.body.bytes, from modules "apache", "iis", "kibana", "nginx" and "traefik", including http.response.content_length (ECS). 10188

• Change type from haproxy.log fileset fields from text to keyword: response.captured_headers, request.captured_headers, raw_request_line, mode. 10397

• Change type of field backend_url and frontend_name in traefik.access metricset to type keyword. 10401

• Ingesting Elasticsearch audit logs is only supported with Elasticsearch 6.5.0 and above 10352

• Migrate Elasticsearch audit logs fields to ECS 10352

• Several text fields in the Logstash module are now indexed as keyword fields with text multi-fields (ECS). 10417

• Several text fields in the Elasticsearch module are now indexed as keyword fields with text multi-fields (ECS). 10414

• Move dissect pattern for traefik.access fileset from Filbeat to Elasticsearch. 10442

• The elasticsearch/deprecation fileset now indexes the component field under elasticsearch instead of elasticsearch.server. 10445

• Remove field kafka.log.trace.full from kafka.log fielset. 10398

• Change field kafka.log.class for kafka.log fileset from text to keyword. 10398

• Address add_kubernetes_metadata processor issue where old source field is still used for matcher. 10505 10506

• Change type of haproxy.source from text to keyword. 10506

• Rename event.type to suricata.eve.event_type in Suricata module because event.type is reserved for future use by ECS. 10575

• Populate more ECS fields in the Suricata module. 10006

• Rename setting filebeat.registry_flush to filebeat.registry.flush. 10504

• Rename setting filebeat.registry_file_permission to filebeat.registry.file_permission. 10504

• Remove setting filebeat.registry_file in favor of filebeat.registry.path. The registry file will be stored in a sub-directory by now. 10504

Heartbeat

• Remove monitor generator script that was rarely used. 9648

• monitor IDs are now configurable. Auto generated monitor IDs now use a different formula based on a hash of their config values. If you wish to have continuity with the old format of monitor IDs you’ll need to set the id property explicitly. 9697

• A number of fields have been aliased to their relevant counterparts in the url.* field. Existing visualizations should mostly work. The fields that have been moved are monitor.scheme → url.scheme, monitor.host → url.domain, resolve.host → url.domain, http.url → url.full, tcp.port → url.port. In addition to these moves the new fields url.username, url.password, url.path, and url.query are now present. It should be noted that the url.password field does not contain actual password values, but rather the text <hidden> 9570.

• The included Kibana HTTP dashboard is now removed in favor of the Uptime app in Kibana. 10294

Journalbeat

• Rename read_timestamp to event.created to align with ECS. 10043, 10139

• Rename host.name to host.hostname to align with ECS. 10043

• Fix typo in the field name container.id_truncated. 10525

• Rename container.image.tag to container.log.tag. 10561

• Change type of text fields to keyword. 10542

Metricbeat

• Migrate system process metricset fields to ECS. 10332

• Refactor Prometheus metric mappings 9948

• Removed Prometheus stats metricset in favor of just using Prometheus collector 9948

• Migrate system socket metricset fields to ECS. 10339

• Renamed direction values in sockets to ECS recommendations, from incoming/outcoming to inbound/outbound. 10339

• Adjust Redis.info metricset fields to ECS. 10319

• Change type of field docker.container.ip_addresses to ip instead of keyword. 10364

• Rename http.request.body field to http.request.body.content. 10315

• Adjust php_fpm.process metricset fields to ECS. 10366

• Adjust mongodb.status metricset to to ECS. 10368

• Refactor munin module to collect an event per plugin and to have more strict field mappings. namespace option has been removed, and will be replaced by service.name. 10322

• Change the following fields from type text to keyword: 10318

• ceph.osd_df.name

• ceph.osd_tree.name

• ceph.osd_tree.children

• kafka.consumergroup.meta

• kibana.stats.name

• mongodb.metrics.replication.executor.network_interface

• php_fpm.process.request_uri

• php_fpm.process.script

• Add service.name option to all modules to explicitly set service.name if it is unset. 10427

• Update a few elasticsearch.* fields to map to ECS. 10350

• Update a few logstash.* fields to map to ECS. 10350

• Update a few kibana.* fields to map to ECS. 10350

• Update rabbitmq.* fields to map to ECS. 10563

• Update haproxy.* fields to map to ECS. 10558 10568

• Collect all EC2 meta data from all instances in all states. 10628

• Fix MongoDB dashboard that had some incorrect field names from status Metricset 9795 9715

Packetbeat

• Adjust Packetbeat http fields to ECS Beta 2 9645

• http.request.body moves to http.request.body.content

• http.response.body moves to http.response.body.content

• Changed Packetbeat fields to align with ECS. 7968

• Removed trailing dot from domain names reported by the DNS protocol. 9941

Winlogbeat

• Adjust Winlogbeat fields to map to ECS. 10333

Functionbeat

• Correctly normalize Cloudformation resource name. 10087

• Functionbeat can now deploy a function for Kinesis. {10116}10116[10116]

• Allow functionbeat to use the keystore. 9009

Bugfixes

Affecting all Beats

• Fix config appender registration. 9873

• Gracefully handle TLS options when enrolling a Beat. 9129

• The backing off now implements jitter to better distribute the load. 10172

• Fix TLS certificate DoS vulnerability. 10302

• Fix panic and file unlock in spool on atomic operation (arm, x86-32). File lock was not released when panic occurs, leading to the beat deadlocking on startup. 10289

• Fix encoding of timestamps when using disk spool. 10099

• Fix stopping of modules started by kubernetes autodiscover. 10476

• Fix a issue when remote and local configuration didn’t match when fetching configuration from Central Management. 10587

• Fix unauthorized error when loading dashboards by adding username and password into kibana config. 10513 10675

• Fix exclude_labels when there are dotted keys 10154

• Fix registry handle leak on Windows (elastic/go-sysinfo#33). 9920

Auditbeat

• Enable System module config on Windows. 10237

Filebeat

• Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869, access log: 9955.

• Support haproxy log lines without captured headers. 9463 9958

• Make elasticsearch/audit fileset be more lenient in parsing node name. 10035 10135

• Fix bad bytes count in docker input when filtering by stream. 10211

• Fixed data types for roles and indices fields in elasticsearch/audit fileset 10307

• Ensure source.address is always populated by the nginx module (ECS). 10418

• Support mysql 5.7.22 slowlog starting with time information. 7892 9647

Heartbeat

• Made monitors.d configuration part of the default config. 9004

• Fixed rare issue where TLS connections to endpoints with x509 certificates missing either notBefore or notAfter would cause the check to fail with a stacktrace. 9566

Journalbeat

• Do not stop collecting events when journal entries change. 9994

Metricbeat

• Fix panics in vsphere module when certain values where not returned by the API. 9784

• Fix pod UID metadata enrichment in Kubernetes module. 10081

• Fix issue that would prevent collection of processes without command line on Windows. 10196

• Fixed data type for tags field in docker/container metricset 10307

• Fixed data type for tags field in docker/image metricset 10307

• Fixed data type for isr field in kafka/partition metricset 10307

• Fixed data types for various hosts fields in mongodb/replstatus metricset 10307

• Added function to close sql database connection. 10355

• Fix issue with elasticsearch/node_stats metricset (x-pack) not indexing source_node field. 10639

Packetbeat

• Fix DHCPv4 dashboard that wouldn’t load in Kibana. 9850

• Fixed a crash when using af_packet capture 10477

Winlogbeat

• Close handle on signalEvent. 9838

Functionbeat

• Ensure that functionbeat is logging at info level not debug. 10262

• Add the required permissions to the role when deployment SQS functions. 9152

Added

Affecting all Beats

• Update field definitions for http to ECS Beta 2 9645

• Add agent.id and agent.ephemeral_id fields to all beats. 9404

• Add name config option to add_host_metadata processor. 9943

• Add add_labels and add_tags processors. 9973

• Add missing file encoding to readers. 10080

• Introduce migration.enabled configuration. 9805

• Add alias field support in Kibana index pattern. 10075

• Add add_fields processor. 10119

• Add Kibana field formatter to bytes fields. 10184

• Document a few more auditd.log.* fields. 10192

• Support Kafka 2.1.0. 10440

• Add ILM mode auto to setup.ilm.enabled setting. This new default value detects if ILM is available 10347

• Add support to read ILM policy from external JSON file. 10347

• Add overwrite and check_exists settings to ILM support. 10347

• Generate Kibana index pattern on demand instead of using a local file. 10478

• Calls to Elasticsearch X-Pack APIs made by Beats won’t cause deprecation logs in Elasticsearch logs. {9656}9656[9656]

• Allow to unenroll a Beat from the UI. 9452

• Release Jolokia autodiscover as GA. 9706

• Allow Central Management to send events back to kibana. 9382

Auditbeat

• Add system module. 9546

• Add user.id (UID) and user.name for ECS. 10195

• Add group.id (GID) and group.name for ECS. 10195

• System module process dataset: Add user information to processes. 9963

• Add system package dataset. 10225

• Add system module login dataset. 9327

• Add entity_id fields. 10500

• Add seven dashboards for the system module. 10511

Filebeat

• Add convert_timezone option to Elasticsearch module to convert dates to UTC. 9756 9761

• Added module for parsing Google Santa logs. 9540

• Added netflow input type that supports NetFlow v1, v5, v6, v7, v8, v9 and IPFIX. 9399

• Add option to modules.yml file to indicate that a module has been moved 9432.

• Add support for ssl_request_log in apache2 module. 8088 9833

• Add support for iis 7.5 log format. 9753 9967

• Add service.type field to all Modules. By default the field is set with the module name. It can be overwritten with service.type config. 10042

• Add support for MariaDB in the slowlog fileset of mysql module. 9731

• Apache module’s error fileset now performs GeoIP lookup, like the access fileset. 10273

• Elasticsearch module’s slowlog now populates event.duration (ECS). 9293

• HAProxy module now populates event.duration and http.response.bytes (ECS). 10143

• Teach elasticsearch/audit fileset to parse out some more fields. 10134 10137

• Add convert_timezone to nginx module. 9839 10148

• Add support for Percona in the slowlog fileset of mysql module. 6665 10227

• Added support for ingesting structured Elasticsearch audit logs 10352

• Added support for ingesting structured Elasticsearch slow logs 10445

• Added support for ingesting structured Elasticsearch deprecation logs 10445

• New iptables module that receives iptables/ip6tables logs over syslog or file. Supports Ubiquiti Firewall extensions. 8781 10176

• Added support for ingesting structured Elasticsearch server logs 10428

• Populate more ECS fields in the Suricata module. 10006

• Add module zeek. 9931 10034

Heartbeat

• Autodiscover metadata is now included in events by default. So, if you are using the docker provider for instance, you’ll see the correct fields under the docker key. 10258

Journalbeat

• Migrate registry from previously incorrect path. 10486

Metricbeat

• Add key metricset to the Redis module. 9582 9657 9746

• Add socket_summary metricset to system defaults, removing experimental tag and supporting Windows 9709

• Add docker event metricset. 9856

• Add 'performance' metricset to x-pack mssql module 9826

• Add DeDot for kubernetes labels and annotations. 9860 9939

• Add more meaningful metrics to 'performance' Metricset on 'MSSQL' module 10011

• Rename some fields in performance Metricset on MSSQL module to match the updated documentation from Microsoft 10074

• Add AWS EC2 module. 9257 9300

• Release windows Metricbeat module as GA. 10163

• Release traefik Metricbeat module as GA. 10166

• Release Elastic stack modules (Elasticsearch, Logstash, and Kibana) as GA. 10094

• List filesystems on Windows that have an access path but not an assigned letter 8916 10196

• Add nats module. 10071

• Release uswgi Metricbeat module GA. 10164

• Release php_fpm module as GA. 10198

• Release Memcached module as GA. 10199

• Release etcd module as GA. 10200

• Release Ceph module as GA. 10202

• Release aerospike module as GA. 10203

• Release kubernetes apiserver and event metricsets as GA 10212

• Release Couchbase module as GA. 10201

• Release RabbitMQ module GA. 10165

• Release envoyproxy module GA. 10223

• Release mongodb.metrics and mongodb.replstatus as GA. 10242

• Release mysql.galera_status as GA. 10242

• Release postgresql.statement as GA. 10242

• Release RabbitMQ Metricbeat module GA. 10165

• Release Dropwizard module as GA. 10240

• Release Graphite module as GA. 10240

• Release kvm module as beta. 10279

• Release http.server metricset as GA. 10240

• Release Nats module as GA. 10281

• Release munin module as GA. 10311

• Release Golang module as GA. 10312

• Release use of xpack.enabled: true flag in Elasticsearch and Kibana modules as GA. 10222

• Add support for MySQL 8.0 and tests also for Percona and MariaDB. 10261

• Rename 'db' Metricset to 'transaction_log' in MSSQL Metricbeat module 10109

• Add process arguments and the path to its executable file in the system process metricset 10332

• Added 'server' Metricset to Zookeeper Metricbeat module 8938 10341

• Release AWS module as GA. 10345

• Add overview dashboard to Zookeeper Metricbeat module 10379

Packetbeat

• Add network.community_id to Packetbeat flow events. 10061

• Add aliases for flow fields that were renamed. 7968 10063

• Add support to decode mysql prepare statement command. 8084

Functionbeat

• Mark Functionbeat as GA. 10564

Beats version 7.0.0-alpha2

Check the HEAD diff

Breaking changes

Affecting all Beats

• Update add_cloud_metadata fields to adjust to ECS. 9265

• Automaticall cap signed integers to 63bits. 8991

• Rename beat.timezone to event.timezone. 9458

• Use _doc as document type. 9056

• Removed dashboards and index patterns generation for Kibana 5. 8927

• On systems with systemd, the Beats log is now written to journald by default rather than file. To revert this behaviour override BEAT_LOG_OPTS with an empty value. 8942.

Auditbeat

• Remove warning for deprecated option: "filters". 9002

Filebeat

• Allow beats to blacklist certain part of the configuration while using Central Management. 9099

• Remove warnings for deprecated options: "spool_size", "publish_async", "idle_timeout". 9002

• Rename many haproxy.* fields to map to ECS. 9117

• Rename many iis.access.* fields to map to ECS. 9084

• IIS module’s user agent string is no longer encoded (+ replaced with spaces). 9084

• Rename many system.syslog.* fields to map to ECS. 9135

• Rename many nginx.access.* fields to map to ECS. 9081

• Rename many system.auth.* fields to map to ECS. 9138

• Rename many apache2.access.* fields to map to ECS. 9245

• Rename apache2 module to apache. 9402

Metricbeat

• Allow beats to blacklist certain part of the configuration while using Central Management. 9099

• Remove warning for deprecated option: "filters". 9002

Packetbeat

• Renamed the flow event fields to follow Elastic Common Schema. 9121

• Renamed several client and server fields. IP, port, and process metadata are now contained under the client and server namespaces. 9303

Functionbeat

• The CLI will now log CloudFormation Stack events. 8912

• Function concurrency is now set to 5 instead of unreserved. 8992

Bugfixes

Affecting all Beats

• Propagate Sync error when running SafeFileRotate. 9069

• Fix autodiscover configurations stopping when metadata is missing. 8851

• Log events at the debug level when dropped by encoding problems. 9251

• Refresh host metadata in add_host_metadata. 9359

• When collecting swap metrics for beats telemetry or system metricbeat module handle cases of free swap being bigger than total swap by assuming no swap is being used. 6271 9383

• Adding logging traces at debug level when the pipeline client receives the following events: onFilteredOut, onDroppedOnPublish. 9016

• Ignore non index fields in default_field for Elasticsearch. 9549

• Update Kibana index pattern attributes for objects that are disabled. 9644

• Enforce validation for the Central Management access token. 9621

• Update to Golang 1.11.4. 9627

Auditbeat

Filebeat

• Correctly parse December or Dec in the Syslog input. 9349

• Fix installation of haproxy dashboard. 9307 9313

• Don’t generate incomplete configurations when logs collection is disabled by hints. 9305

• Stop runners disabled by hints after previously being started. 9305

• Fix saved objects in filebeat haproxy dashboard. 9417

• Use log.source.address instead of log.source.ip for network input sources. 9487

• Rename many redis.log.* fields to map to ECS. 9315

• Rename many icinga.* fields to map to ECS. 9294

• Rename many postgresql.log.* fields to map to ECS. 9308

• Rename many kafka.log.* fields to map to ECS. 9297

• Add convert_timezone option to Logstash module to convert dates to UTC. 9756 9797

Metricbeat

• Fix issue preventing diskio metrics collection for idle disks. 9124 9125

• Fix panic on docker healthcheck collection on dockers without healthchecks. 9171

• Fix issue with not collecting Elasticsearch cross-cluster replication stats correctly. 9179

• The node.name field in the elasticsearch/node metricset now correctly reports the Elasticsarch node name. Previously this field was incorrectly reporting the node ID instead. 9209

Packetbeat

• Fix issue with process monitor associating traffic to the wrong process. 9151 9443

Added

Affecting all Beats - Unify dashboard exporter tools. 9097

• Unify dashboard exporter tools. 9097

• Add cache.ttl to add_host_metadata. 9359

• Add support for index lifecycle management (beta). 7963

• Always include Pod UID as part of Pod metadata. 9517

• Autodiscovery no longer requires that the condition field be set. If left unset all configs will be matched. 9029

• Add geo fields to add_host_metadata processor. 9392

Filebeat - Added detect_null_bytes selector to detect null bytes from a io.reader. 9210

• Added the redirect_stderr option that allows panics to be logged to log files. 8430

• Added detect_null_bytes selector to detect null bytes from a io.reader. 9210

• Added syslog_host variable to HAProxy module to allow syslog listener to bind to configured host. 9366

• Added support on Traefik for Common Log Format and Combined Log Format mixed which is the default Traefik format 8015 6111 8768.

• Add support for multi-core thread_id in postgresql module 9156 9482

Heartbeat

• Add last monitor status to dashboard table. Further break out monitors in dashboard table by monitor.ip. 9022

• Add central management support. 9254

Journalbeat

• Add cursor_seek_fallback option. 9234

Metricbeat

• Add settings to disable docker and cgroup cpu metrics per core. 9187 9194 9589

• The elasticsearch/node metricset now reports the Elasticsearch cluster UUID. 8771

• Add service.type field to Metricbeat. 8965

• Support GET requests in Jolokia module. 8566 9226

• Add freebsd support for the uptime metricset. 9413

• Add host.os.name field to add_host_metadata processor. 8948 9405

• Add more TCP statuses to socket_summary metricset. 9430

• Remove experimental tag from ceph metricsets. 9708

• Add MS SQL module to X-Pack 9414

Deprecated

Metricbeat

• event.duration is now in nano and not microseconds anymore. 8941

Beats version 7.0.0-alpha1

View commits

Breaking changes

Affecting all Beats

• Dissect syntax change, use * instead of ? when working with field reference. 8054

Auditbeat

• Use initial_scan action for new paths. 7954

• Rename beat.name to agent.type, beat.hostname to agent.hostname, beat.version to agent.version.

• Rename source.hostname to source.domain in the auditd module. 9027

Filebeat

• Rename fileset.name to event.name. 8879

• Rename fileset.module to event.module. 8879

• Rename source to log.file.path and log.source.ip 8902

• Remove the deprecated prospector(s) option in the configuration use input(s) instead. 8909

• Rename offset to log.offset. 8923

• Rename source_ecs to source in the Filebeat Suricata module. 8983

Bugfixes

Affecting all Beats

• Fixed -d CLI flag by trimming spaces from selectors. 7864

• Fixed Support add_docker_metadata in Windows by identifying systems' path separator. 7797

• Do not panic when no tokenizer string is configured for a dissect processor. 8895

• Start autodiscover consumers before producers. 7926

Filebeat

• Fixed a memory leak when harvesters are closed. 7820

• Fix improperly set config for CRI Flag in Docker Input 8899

• Just enabling the elasticsearch fileset and starting Filebeat no longer causes an error. 8891

• Fix macOS default log path for elasticsearch module based on homebrew paths. {pul}8939[8939]

Heartbeat

• Heartbeat now always downloads the entire body of HTTP endpoints, even if no checks against the body content are declared. This fixes an issue where timing metrics would be incorrect in scenarios where the body wasn’t used since the connection would be closed soon after the headers were sent, but before the entire body was. 8894

• Host header can now be overridden for HTTP requests sent by Heartbeat monitors. 9516

Metricbeat

• Fix golang.heap.gc.cpu_fraction type from long to float in Golang module. 7789

• Add missing namespace field in http server metricset 7890

• Fix race condition when enriching events with kubernetes metadata. 9055 9067

Packetbeat

• Fixed the mysql missing transactions if monitoring a connection from the start. 8173

Added

Affecting all Beats

• Add field host.os.kernel to the add_host_metadata processor and to the internal monitoring data. 7807

• Add debug check to logp.Logger 7965

• Count HTTP 429 responses in the elasticsearch output 8056

• Allow Bus to buffer events in case listeners are not configured. 8527

• Dissect will now flag event on parsing error. 8751

• add_cloud_metadata initialization is performed asynchronously to avoid delays on startup. 8845

• Add DeDot method in add_docker_metadata processor in libbeat. 9350 9505

Filebeat

• Make inputsource generic taking bufio.SplitFunc as input 7746

• Add custom unpack to log hints config to avoid env resolution 7710

• Make docker input check if container strings are empty 7960

• Keep unparsed user agent information in user_agent.original. 8537

• Allow to force CRI format parsing for better performance 8424

Heartbeat

• Add automatic config file reloading. 8023

Journalbeat

• Add the ability to check against JSON HTTP bodies with conditions. 8667

Metricbeat

• Add metrics about cache size to memcached module 7740

• Add experimental socket summary metricset to system module 6782

• Collect custom cluster display_name in elasticsearch/cluster_stats metricset. 8445

• Test etcd module with etcd 3.3. 9068

• All elasticsearch metricsets now have module-level cluster.id and cluster.name fields. 8770 8771 9164 9165 9166 9168

• All elasticsearch node-level metricsets now have node.id and node.name fields. 9168 9209

Packetbeat

• Add support to decode HTTP bodies compressed with gzip and deflate. 7915

• Added support to calculate certificates' fingerprints (MD5, SHA-1, SHA-256). 8180

• Support new TLS version negotiation introduced in TLS 1.3. 8647.

Beats version 6.8.13

View commits

Added

Filebeat

• Add container image in Kubernetes metadata. 13356 12688

Beats version 6.8.12

View commits

Bugfixes

Filebeat

• Fix Filebeat OOMs on very long lines 19500, 19552

Beats version 6.8.11

View commits

Bugfixes

Metricbeat

• Fix bug incorrect parsing of float numbers as integers in Couchbase module 18949 19055

Beats version 6.8.10

View commits

Bugfixes

Affecting all Beats

• Fix add_cloud_metadata to better support modifying sub-fields with other processors. 13808

Beats version 6.8.9

View commits

Bugfixes

Heartbeat

• Fix crashes when multiple TCP ports are specified. 17262

Beats version 6.8.8

View commits

Bugfixes

Filebeat

• Add support for Cisco syslog format used by their switch. 10760

Beats version 6.8.7

View commits

Bugfixes

Metricbeat

• Fix bug with elasticsearch/cluster_stats metricset not recording license expiration date correctly. 14541 14591

• Make kibana module more resilient to Kibana unavailability. 15258 15270

Beats version 6.8.6

View commits

Bugfixes

Heartbeat

• Fix recording of SSL cert metadata for Expired/Unvalidated x509 certs. 13687

Metricbeat

• Fix marshaling of ms-since-epoch values in elasticsearch/cluster_stats metricset. 14378

• Fix bug with elasticsearch/cluster_stats metricset not recording license ID in the correct field. 14592

Beats version 6.8.5

View commits

Bugfixes

Metricbeat

• Convert indexed ms-since-epoch timestamp fields in elasticsearch/ml_job metricset to ints from float64s. 14220 14222

Beats version 6.8.4

View commits

Breaking changes

Filebeat

• Fix delay in enforcing close_renamed and close_removed options. 13488 13907

Bugfixes

Filebeat

• Fix merging of fields specified in global scope with fields specified under an input’s scope. 3628 13909

• Fix early expiration of templates (Netflow v9 and IPFIX). 13821

• Fix bad handling of sequence numbers when multiple observation domains were exported by a single device (Netflow V9 and IPFIX). 13821

• Fix increased memory usage with large files when multiline pattern does not match. 14068

Metricbeat

• Mark Kibana usage stats as collected only if API call succeeds. 13881

Beats version 6.8.3

https://github.com/elastic/beats/compare/v6.8.2...v6.8.3[View commits

Bugfixes

Journalbeat

• Iterate over journal correctly, so no duplicate entries are sent. 12716

Metricbeat

• Fix panic in Redis Key metricset when collecting information from a removed key. 13426

Added

Metricbeat

• Remove _nodes field from under cluster_stats as it’s not being used. 13010

• Collect license expiry date fields as well. 11652

Beats version 6.8.2

View commits

Bugfixes

Auditbeat

• Process dataset: Do not show non-root warning on Windows. 12740

Filebeat

• Skipping unparsable log entries from docker json reader 12268

Packetbeat

• Limit memory usage of Redis replication sessions. 12657

Beats version 6.8.1

View commits

Bugfixes

Affecting all Beats

• Fixed a memory leak when using the add_process_metadata processor under Windows. 12100

Auditbeat

• Package dataset: Log error when Homebrew is not installed. 11667

• Process dataset: Fixed a memory leak under Windows. 12100

• Login dataset: Fix re-read of utmp files. 12028

• Package dataset: Fixed a crash inside librpm after Auditbeat has been running for a while. 12147 12168

• Fix direction of incoming IPv6 sockets. 12248

• Package dataset: Auto-detect package directories. 12289

• System module: Start system module without host ID. 12373

• Host dataset: Fix reboot detection logic. 12591

Filebeat

• Fix goroutine leak happening when harvesters are dynamically stopped. 11263

• Fix initialization of the TCP input logger. 11605

• Fix goroutine leak caused on initialization failures of log input. 12125

• Fix memory leak in Filebeat pipeline acker. 12063

• Fix goroutine leak on non-explicit finalization of log input. 12164

• When TLS is configured for the TCP input and a certificate_authorities is configured we now default to required for the client_authentication. 12584

Metricbeat

• Avoid generating hints-based configuration with empty hosts when no exposed port is suitable for the hosts hint. 8264 12086

• Fix direction of incoming IPv6 sockets. 12248

• Validate that kibana/status metricset cannot be used when xpack is enabled. 12264

• In the kibana/stats metricset, only log error (don’t also index it) if xpack is enabled. 12353

• The elasticsearch/index_summary metricset gracefully handles an empty Elasticsearch cluster when xpack.enabled: true is set. 12489 12487

• When TLS is configured for the http metricset and a certificate_authorities is configured we now default to required for the client_authentication. 12584

Packetbeat

• Fixed a memory leak when using process monitoring under Windows. 12100

• Improved debug logging efficiency in PGQSL module. 12150

Added

Auditbeat

• Add support to the system package dataset for the SUSE OS family. 11634

Metricbeat

• Add validation for elasticsearch and kibana modules' metricsets when xpack.enabled is set to true. 12386

Beats version 6.8.0

• Updates to support changes to licensing of security features.

  Some Elastic Stack security features, such as encrypted communications, file and native authentication, and role-based access control, are now available in more subscription levels. For details, see https://www.elastic.co/subscriptions.

Beats version 6.7.2

View commits

Bugfixes

Affecting all Beats

• Relax validation of the X-Pack license UID value. 11640

• Fix a parsing error with the X-Pack license check on 32-bit system. 11650

• Fix OS family classification in add_host_metadata for Amazon Linux, Raspbian, and RedHat Linux. 9134 11494

• Fix false positives reported in the host.containerized field added by add_host_metadata. 11494

• Fix the add_host_metadata’s host.id field on older Linux versions. 11494

Auditbeat

• Package dataset: dlopen versioned librpm shared objects. 11565

• Package dataset: Nullify Librpm’s rpmsqEnable. 11628

Filebeat

• Don’t apply multiline rules in Logstash json logs. 11346

• Fix goroutine leak happening when harvesters are dynamically stopped. 11263

• Fix initialization of the TCP input logger. 11605

Metricbeat

• Prevent the docker/memory metricset from processing invalid events before container start 11676

Added

Auditbeat

• Add support to the system package dataset for the SUSE OS family. 11634

Beats version 6.7.1

View commits

Breaking changes

Affecting all Beats

• Initialize the Paths before the keystore and save the keystore into data/{beatname}.keystore. 10706

Bugfixes

Affecting all Beats

• Remove IP fields from default_field in Elasticsearch template. 11399

Beats version 6.7.0

View commits

Breaking changes

Affecting all Beats

• Port settings have been deprecated in redis/logstash output and will be removed in 7.0. 9915

• Update the code of Central Management to align with the new returned format. 10019

• Allow Central Management to send events back to kibana. 9382

• Fix panic if fields settting is used to configure hosts.x fields. 10824 10935

• Introduce query.default_field as part of the template. 11205

• Beats Xpack now checks for Basic license on connect. 11296

Filebeat

• Filesets with multiple ingest pipelines added in 8914 only work with Elasticsearch >= 6.5.0 10001

• Add grok pattern to support redis 5.0.3 log timestamp. 9819 10033

• Ingesting Elasticsearch audit logs is only supported with Elasticsearch 6.5.0 and above 8852

• Remove ecs option from user_agent processors when loading pipelines with Filebeat 6.7.x into Elasticsearch < 6.7.0. 10655 11362

Heartbeat

• Remove monitor generator script that was rarely used. 9648

Bugfixes

Affecting all Beats

• Fix TLS certificate DoS vulnerability. 10303

• Fix panic and file unlock in spool on atomic operation (arm, x86-32). File lock was not released when panic occurs, leading to the beat deadlocking on startup. 10289

• Adding logging traces at debug level when the pipeline client receives the following events: onFilteredOut, onDroppedOnPublish. 9016

• Do not panic when no tokenizer string is configured for a dissect processor. 8895

• Fix a issue when remote and local configuration didn’t match when fetching configuration from Central Management. 10587

• Add ECS-like selectors and dedotting to docker autodiscover. 10757 10862

• Fix encoding of timestamps when using disk spool. 10099

• Include ip and boolean type when generating index pattern. 10995

• Using an environment variable for the password when enrolling a beat will now raise an error if the variable doesn’t exist. 10936

• Cancelling enrollment of a beat will not enroll the beat. 10150

• Remove IP fields from default_field in Elasticsearch template. 11399

Auditbeat

• Package: Disable librpm signal handlers. 10694

• Login: Handle different bad login UTMP types. 10865

• Fix hostname references in System module dashbords. 11064

• User dataset: Numerous fixes to error handling. 10942

Filebeat

• Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869 access log: 10029

• Fix bad bytes count in docker input when filtering by stream. 10211

• Fixed data types for roles and indices fields in elasticsearch/audit fileset 10307

• Cover empty request data, url and version in Apache2 modulehttps://github.com/elastic/pull/10846[10846]

• Fix a bug with the convert_timezone option using the incorrect timezone field. 11055 11164

• Change URLPATH grok pattern to support brackets. 11135 11252

• Add support for iis log with different address format. 11255 11256

• Add fix to parse syslog message with priority value 0. 11010

Heartbeat

• Host header can now be overridden for HTTP requests sent by Heartbeat monitors. 9516

• Fix checks for TCP send/receive data 10777

Journalbeat

• Do not stop collecting events when journal entries change. 9994

Metricbeat

• Fix MongoDB dashboard that had some incorrect field names from status Metricset 9795 9715

• Fix issue that would prevent collection of processes without command line on Windows. 10196

• Fixed data type for tags field in docker/container metricset 10307

• Fixed data type for tags field in docker/image metricset 10307

• Fixed data type for isr field in kafka/partition metricset 10307

• Fixed data types for various hosts fields in mongodb/replstatus metricset 10307

• Added function to close sql database connection. 10355

• Fix parsing error using GET in Jolokia module. 11075 11071

Winlogbeat

• Fix Winlogbeat escaping CR, LF and TAB characters. 11328 11357

Functionbeat

• Correctly extract Kinesis Data field from the Kinesis Record. 11141

• Add the required permissions to the role when deployment SQS functions. 9152

Added

Affecting all Beats

• Add ip fields to default_field in Elasticsearch template. 11035

• Add cleanup_timeout option to docker autodiscover, to wait some time before removing configurations after a container is stopped. 10374 10905

Auditbeat

• System module process dataset: Add user information to processes. 9963

• Add system package dataset. 10225

• Add system module login dataset. 9327

• Add entity_id fields. 10500

• Add seven dashboards for the system module. 10511

Filebeat

• Add field log.source.address and log.file.path to replace source. 9435

• Support mysql 5.7.22 slowlog starting with time information. 7892 9647

• Add support for ssl_request_log in apache2 module. 8088 9833

• Add support for iis 7.5 log format. 9753 9967

• Add support for MariaDB in the slowlog fileset of mysql module. 9731

• Add convert_timezone to nginx module. 9839 10148

• Add support for Percona in the slowlog fileset of mysql module. 6665 10227

• Added support for ingesting structured Elasticsearch audit logs 8852

• New iptables module that receives iptables/ip6tables logs over syslog or file. Supports Ubiquiti Firewall extensions. 8781 10176

• Populate more ECS fields in the Suricata module. 10006

Heartbeat

• Made monitors.d configuration part of the default config. 9004

• Autodiscover metadata is now included in events by default. So, if you are using the docker provider for instance, you’ll see the correct fields under the docker key. 10258

Metricbeat

• Add field event.dataset which is {module}.{metricset}.

• Add more TCP statuses to socket_summary metricset. 9430

• Remove experimental tag from ceph metricsets. 9708

• Add key metricset to the Redis module. 9582 9657

• Add DeDot for kubernetes labels and annotations. 9860 9939

• Add docker event metricset. 9856

• Release Ceph module as GA. 10202

• Release windows Metricbeat module as GA. 10163

• Release traefik Metricbeat module as GA. 10166

• List filesystems on Windows that have an access path but not an assigned letter 8916 10196

• Release uswgi Metricbeat module GA. 10164

• Release php_fpm module as GA. 10198

• Release Memcached module as GA. 10199

• Release etcd module as GA. 10200

• Release kubernetes apiserver and event metricsets as GA 10212

• Release Couchbase module as GA. 10201

• Release aerospike module as GA. 10203

• Release envoyproxy module GA. 10223

• Release mongodb.metrics and mongodb.replstatus as GA. 10242

• Release mysql.galera_status as Beta. 10242

• Release postgresql.statement as GA. 10242

• Release RabbitMQ Metricbeat module GA. 10165

• Release Dropwizard module as GA. 10240

• Release Graphite module as GA. 10240

• Release http.server metricset as GA. 10240

• Add support for MySQL 8.0 and tests also for Percona and MariaDB. 10261

• Release use of xpack.enabled: true flag in Elasticsearch and Kibana modules as GA. 10222

• Release Elastic stack modules (Elasticsearch, Logstash, and Kibana) as GA. 10094

• Add remaining memory metrics of pods in Kubernetes metricbeat module 10157

• Added 'server' Metricset to Zookeeper Metricbeat module 8938 10341

• Add overview dashboard to Zookeeper Metricbeat module 10379

Functionbeat

• Mark Functionbeat as GA. 10564

• Functionbeat can now deploy a function for Kinesis. 10116

• Allow functionbeat to use the keystore. 9009

Deprecated

Filebeat

• Deprecate field source. Will be replaced by log.source.address and log.file.path in 7.0. 9435

Metricbeat

• Deprecate field metricset.rtt. Replaced by event.duration which is in nano instead of micro seconds.

Packetbeat

• Support new TLS version negotiation introduced in TLS 1.3. 8647.

Known Issue

Journalbeat

• Journalbeat requires at least systemd v233 in order to follow entries after journal changes (rotation, vacuum).

Beats version 6.6.2

View commits

Bugfixes

Auditbeat

• System module: Fix and unify bucket closing logic. 10897

Filebeat

• Fix a bug when converting NetFlow fields to snake_case. 10950

Metricbeat

• Fix issue in kubernetes module preventing usage percentages to be properly calculated. 10946

Packetbeat

• Avoid reporting unknown MongoDB opcodes more than once. 10878

Winlogbeat

• Prevent Winlogbeat from dropping events with invalid XML. 11006

Beats version 6.6.1

View commits

Breaking changes

Affecting all Beats

• Fix stopping of modules started by kubernetes autodiscover. 10476

Auditbeat

• Enable System module config on Windows. 10237

Filebeat

• Fix bad bytes count in docker input when filtering by stream. 10211

• Add convert_timezone option to Logstash module to convert dates to UTC. 9756 9797

• Add convert_timezone option to Elasticsearch module to convert dates to UTC. 9756 9761

• Make elasticsearch/audit fileset be more lenient in parsing node name. 10035 10135

Journalbeat

• Fix fields.yml indentation of audit group which had the effect of creating an incomplete Elasticsearch index template. 10556

Metricbeat

• Fix issue with elasticsearch/node_stats metricset (x-pack) not indexing source_node field. 10639

Packetbeat

• Fixed a crash when using af_packet capture 10477

Functionbeat

• Ensure that functionbeat is logging at info level not debug. 10262

Added

Filebeat

• Teach elasticsearch/audit fileset to parse out some more fields. 10134 10137

Journalbeat

• Migrate registry from previously incorrect path. 10486

Beats version 6.6.0

View commits

Breaking changes

Affecting all Beats

• Dissect syntax change, use * instead of ? when working with field reference. 8054

Filebeat

• Allow beats to blacklist certain part of the configuration while using Central Management. 9099

Metricbeat

• Allow beats to blacklist certain part of the configuration while using Central Management. 9099

Functionbeat

• The CLI will now log CloudFormation Stack events. 8912

• Correctly normalize Cloudformation resource name. 10087

Bugfixes

Affecting all Beats

• Fix autodiscover configurations stopping when metadata is missing. 8851

• Refresh host metadata in add_host_metadata. 9359

• When collecting swap metrics for beats telemetry or system metricbeat module handle cases of free swap being bigger than total swap by assuming no swap is being used. 6271 9383

• Ignore non index fields in default_field for Elasticsearch. 9549

• Update Golang to 1.10.6. 9563

• Update Kibana index pattern attributes for objects that are disabled. 9644

• Enforce validation for the Central Management access token. 9621

• Fix registry handle leak on Windows (elastic/go-sysinfo#33). 9920

• Gracefully handle TLS options when enrolling a Beat. 9129

• Allow to unenroll a Beat from the UI. 9452

• The backing off now implements jitter to better distribute the load. 10172

• Fix config appender registration. 9873

• Fix TLS certificate DoS vulnerability. 10304

Filebeat

• Fix improperly set config for CRI Flag in Docker Input 8899

• Just enabling the elasticsearch fileset and starting Filebeat no longer causes an error. 8891

• Fix macOS default log path for elasticsearch module based on homebrew paths. {pul}8939[8939]

• Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869 access log: 10030

• Support haproxy log lines without captured headers. 9463 9958

Heartbeat

• Heartbeat now always downloads the entire body of HTTP endpoints, even if no checks against the body content are declared. This fixes an issue where timing metrics would be incorrect in scenarios where the body wasn’t used since the connection would be closed soon after the headers were sent, but before the entire body was. 8894

Metricbeat

• Add missing namespace field in http server metricset 7890

• Fix issue with not collecting Elasticsearch cross-cluster replication stats correctly. 9179

• The node.name field in the elasticsearch/node metricset now correctly reports the Elasticsarch node name. Previously this field was incorrectly reporting the node ID instead. 9209

• Fix panics in vsphere module when certain values where not returned by the API. 9784

• Fix pod UID metadata enrichment in Kubernetes module. 10081

Packetbeat

• Fix issue with process monitor associating traffic to the wrong process. 9151 9443

• Fix DHCPv4 dashboard that wouldn’t load in Kibana. 9850

Added

Affecting all Beats

• Unify dashboard exporter tools. 9097

• Dissect will now flag event on parsing error. 8751

• Added the redirect_stderr option that allows panics to be logged to log files. 8430

• Add cache.ttl to add_host_metadata. 9359

• Add support for index lifecycle management (beta). 7963

• Always include Pod UID as part of Pod metadata. 9517

• Release Jolokia autodiscover as GA. 9706

Auditbeat

• Add system module. 9546

Filebeat - Added detect_null_bytes selector to detect null bytes from a io.reader. 9210 - Added syslog_host variable to HAProxy module to allow syslog listener to bind to configured host. 9366 - Allow to force CRI format parsing for better performance 8424 - Add event.dataset to module events. 9457 - Add field log.source.address and log.file.path to replace source. 9435 - Add support for multi-core thread_id in postgresql module 9156 9482 - Added netflow input type that supports NetFlow v1, v5, v6, v7, v8, v9 and IPFIX. 9399

Journalbeat

• Add the ability to check against JSON HTTP bodies with conditions. 8667

• Add cursor_seek_fallback option. 9234

Metricbeat

• Collect custom cluster display_name in elasticsearch/cluster_stats metricset. 8445

• Test etcd module with etcd 3.3. 9068

• All elasticsearch metricsets now have module-level cluster.id and cluster.name fields. 8770 8771 9164 9165 9166 9168

• All elasticsearch node-level metricsets now have node.id and node.name fields. 9168 9209

• Add settings to disable docker and cgroup cpu metrics per core. 9187 9194 9589

• The elasticsearch/node metricset now reports the Elasticsearch cluster UUID. 8771

• Support GET requests in Jolokia module. 8566 9226

• Add freebsd support for the uptime metricset. 9413

• Add host.os.name field to add_host_metadata processor. 8948 9405

• Add field event.dataset which is {module}.{metricset). 9393

Deprecated

Filebeat - Deprecate field source. Will be replaced by log.source.address and log.file.path in 7.0. 9435

Metricbeat

• Deprecate field metricset.rtt. Replaced by event.duration which is in nano instead of micro seconds. 9393

Packetbeat

• Support new TLS version negotiation introduced in TLS 1.3. 8647.

Beats version 6.5.4

View commits

Bugfixes

Affecting all Beats

• Update Golang to 1.10.6. This fixes an issue in remote certificate validation CVE-2018-16875. 9563

Filebeat

• Fix saved objects in filebeat haproxy dashboard. 9417

• Fixed a memory leak when harvesters are closed. 7820

Added

Filebeat

• Added support on Traefik for Common Log Format and Combined Log Format mixed which is the default Traefik format 8015 6111 8768.

Beats version 6.5.3

View commits

Bugfixes

Affecting all Beats

• Log events at the debug level when dropped by encoding problems. 9251

Filebeat

• Correctly parse December or Dec in the Syslog input. 9349

• Don’t generate incomplete configurations when logs collection is disabled by hints. 9305

• Stop runners disabled by hints after previously being started. 9305

• Fix installation of haproxy dashboard. 9307 9313

Beats version 6.5.2

View commits

Bugfixes

Affecting all Beats

• Propagate Sync error when running SafeFileRotate. 9069

Metricbeat

• Fix panic on docker healthcheck collection on dockers without healthchecks. 9171

• Fix issue preventing diskio metrics collection for idle disks. 9124 9125

Beats version 6.5.1

View commits

Bugfixes

Affecting all Beats - Fix windows binaries not having an enroll command. 9096 8836

Journalbeat - Fix journalbeat sometimes hanging if output is unavailable. 9106

Metricbeat - Fix race condition when enriching events with kubernetes metadata. 9055 9067

Added

Journalbeat - Add minimal kibana dashboard. 9106

Beats version 6.5.0

View commits

Bugfixes

Affecting all Beats

• Fixed add_host_metadata not initializing correctly on Windows. 7715

• Fixed missing file unlock in spool file on Windows, so file can be reopened and locked. 7859

• Fix spool file opening/creation failing due to file locking on Windows. 7859

• Fix size of maximum mmaped read area in spool file on Windows. 7859

• Fix potential data loss on OS X in spool file by using fcntl with F_FULLFSYNC. 7859

• Improve fsync on linux, by assuming the kernel resets error flags of failed writes. 7859

• Remove unix-like permission checks on Windows, so files can be opened. 7849

• Replace index patterns in TSVB visualizations. 7929

• Deregister pipeline loader callback when inputsRunner is stopped. 7893[7893]

• Add backoff support to x-pack monitoring outputs. 7966

• Removed execute permissions systemd unit file. 7873

• Fix a race condition with the add_host_metadata and the event serialization. 8223 8653

• Enforce that data used by k8s or docker doesn’t use any reference. 8240

• Switch to different UUID lib due to to non-random generated UUIDs. 8485

• Fix race condition when publishing monitoring data. 8646

• Fix bug in loading dashboards from zip file. 8051

• Fix in-cluster kubernetes configuration on IPv6. 8754

• The export config subcommand should not display real value for field reference. 8769

• The setup command will not fail if no dashboard is available to import. 8977

• Fix central management configurations reload when a configuration is removed in Kibana. 9010

Auditbeat

• Fixed a crash in the file_integrity module under Linux. 7753

• Fixed the RPM by designating the config file as configuration data in the RPM spec. 8075

• Fixed a concurrent map write panic in the auditd module. 8158

• Fixed a data race in the file_integrity module. 8009

• Fixed a deadlock in the file_integrity module. 8027

Filebeat

• Fix date format in Mongodb Ingest pipeline. 7974

• Fixed a docker input error due to the offset update bug in partial log join.https://github.com/elastic/beats/pull/8177[8177]

• Update CRI format to support partial/full tags. 8265

• Fix some errors happening when stopping syslog input. 8347

• Fix RFC3339 timezone and nanoseconds parsing with the syslog input. 8346

• Mark the TCP and UDP input as GA. 8125

• Support multiline logs in logstash/log fileset of Filebeat. 8562

• Support different timestamp format in postgresql module. 9494 9650

Heartbeat

• Fixed bug where HTTP responses with larger bodies would incorrectly report connection errors. 8660

Metricbeat

• Fix golang.heap.gc.cpu_fraction type from long to float in Golang module. 7789

• Fixed the RPM by designating the modules.d config files as configuration data in the RPM spec. 8075

• Fixed the location of the modules.d dir in Deb and RPM packages. 8104

• Add docker diskio stats on Windows. 6815 8126

• Fix incorrect type conversion of average response time in Haproxy dashboards 8404

• Added io disk read and write times to system module 8473 8508

• Avoid mapping issues in kubernetes module. 8487

• Recover metrics for old apache versions removed by mistake on #6450. 7871

• Fix dropwizard module parsing of metric names. 8365 8385

• Fix issue that would prevent kafka module to find a proper broker when port is not set 8613

• Fix range colors in multiple visualizations. 8633 8634

• Fix incorrect header parsing on http metricbeat module 8564 8585

• Fixed a panic when the kvm module cannot establish a connection to libvirtd. 7792.

Packetbeat

• Fixed a seccomp related error where the fcntl64 syscall was not permitted on 32-bit Linux and the sniffer failed to start. 7839

• Added missing cmdline and client_cmdline fields to index template. 8258

Added

Affecting all Beats

• Added time-based log rotation. 8349

• Add backoff on error support to redis output. 7781

• Allow for cloud-id to specify a custom port. This makes cloud-id work in ECE contexts. 7887

• Add support to grow or shrink an existing spool file between restarts. 7859

• Make kubernetes autodiscover ignore events with empty container IDs 7971

• Implement CheckConfig in RunnerFactory to make autodiscover check configs 7961

• Add DNS processor with support for performing reverse lookups on IP addresses. 7770

• Support for Kafka 2.0.0 in kafka output 8399

• Add setting setup.kibana.space.id to support Kibana Spaces 7942

• Better tracking of number of open file descriptors. 7986

• Report number of open file handles on Windows. 8329

• Added the add_process_metadata processor to enrich events with process information. 6789

• Add Beats Central Management 8559

• Report configured queue type. 8091

• Enable host and cloud metadata processors by default. 8596

Filebeat

• Add tag "truncated" to "log.flags" if incoming line is longer than configured limit. 7991

• Add haproxy module. 8014

• Add tag "multiline" to "log.flags" if event consists of multiple lines. 7997

• Release docker input as GA. 8328

• Keep unparsed user agent information in user_agent.original. 7832

• Added default and TCP parsing formats to HAproxy module 8311 8637

• Add Suricata IDS/IDP/NSM module. 8153 8693

• Support for Kafka 2.0.0 8853

Heartbeat

• Heartbeat is marked as GA.

• Add automatic config file reloading. 8023

• Added autodiscovery support 8415

• Added support for extra TLS/x509 metadata. 7944

• Added stats and state metrics for number of monitors and endpoints started. 8621

• Add last monitor status to dashboard table. Further break out monitors in dashboard table by monitor.ip. 9022

Journalbeat

• Add journalbeat. 8703

Metricbeat

• Add replstatus metricset to MongoDB module 7604

• Add experimental socket summary metricset to system module 6782

• Move common kafka fields (broker, topic and partition.id) to the module level to facilitate events correlation 7767

• Add fields for memory fragmentation, memory allocator stats, copy on write, master-slave status, and active defragmentation to info metricset of Redis module. 7695

• Increase ignore_above for system.process.cmdline to 2048. 8100

• Add support to renamed fields planned for redis 5.0. 8167

• Allow TCP helper to support delimiters and graphite module to accept multiple metrics in a single payload. 8278

• Added 'died' PID state to process_system metricset on system module 8275

• Add metrics metricset to MongoDB module. 7611

• Added ccr metricset to Elasticsearch module. 8335

• Support for Kafka 2.0.0 8399

• Added support for query params in configuration 8286 8292

• Add container image for docker metricsets. 8214 8438

• Precalculate composed id fields for kafka dashboards. 8504

• Add support for full status page output for php-fpm module as a separate metricset called process. 8394

• Add Kafka dashboard. 8457

• Release Kafka module as GA. 8854

Packetbeat

• Added DHCP protocol support. 7647

Functionbeat

• Initial version of Functionbeat. 8678

Deprecated

Heartbeat

• watch.poll_file is now deprecated and superceded by automatic config file reloading.

Metricbeat

• Redis info replication.master_offset has been deprecated in favor of replication.master.offset.https://github.com/elastic/beats/pull/7695[7695]

• Redis info clients fields longest_output_list and biggest_input_buf have been renamed to max_output_buffer and max_input_buffer based on the names they will have in Redis 5.0, both fields will coexist during a time with the same value 8167.

• Move common kafka fields (broker, topic and partition.id) to the module level 7767.

Beats version 6.4.3

View commits

Bugfixes

Affecting all Beats

• Fix a race condition with the add_host_metadata and the event serialization. 8223 8653

• Fix race condition when publishing monitoring data. 8646

• Fix bug in loading dashboards from zip file. 8051

• The export config subcommand should not display real value for field reference. 8769

Filebeat

• Fix typo in Filebeat IIS Kibana visualization. 8604

Metricbeat

• Recover metrics for old Apache versions removed by mistake on #6450. 7871

• Avoid mapping issues in Kubernetes module. 8487

• Fixed a panic when the KVM module cannot establish a connection to libvirtd. 7792

Beats version 6.4.2

View commits

Bugfixes

Filebeat

• Fix some errors happening when stopping syslog input. 8347

• Fix RFC3339 timezone and nanoseconds parsing with the syslog input. 8346

Metricbeat

• Fix incorrect type conversion of average response time in Haproxy dashboards 8404

• Fix dropwizard module parsing of metric names. 8365 8385

Beats version 6.4.1

View commits

Bugfixes

Affecting all Beats

• Add backoff support to x-pack monitoring outputs. 7966

• Removed execute permissions systemd unit file. 7873

• Fix a race condition with the add_host_metadata and the event serialization. 8223

• Enforce that data used by k8s or docker doesn’t use any reference. 8240

• Implement CheckConfig in RunnerFactory to make autodiscover check configs 7961

• Make kubernetes autodiscover ignore events with empty container IDs 7971

Auditbeat

• Fixed a concurrent map write panic in the auditd module. 8158

• Fixed the RPM by designating the config file as configuration data in the RPM spec. 8075

Filebeat

• Fixed a docker input error due to the offset update bug in partial log join.https://github.com/elastic/beats/pull/8177[8177]

• Update CRI format to support partial/full tags. 8265

Metricbeat

• Fixed the location of the modules.d dir in Deb and RPM packages. 8104

• Fixed the RPM by designating the modules.d config files as configuration data in the RPM spec. 8075

• Fix golang.heap.gc.cpu_fraction type from long to float in Golang module. 7789

Packetbeat

• Added missing cmdline and client_cmdline fields to index template. 8258

Beats version 6.4.0

View commits

Known issue

Due to a packaging mistake, the modules.d configuration directory is installed in the wrong path in the Metricbeat DEB and RPM packages. This issue results in an empty list when you run metricbeat modules list and failures when you try to enable or disable modules. To work around this issue, run the following command:

sudo cp -r /usr/share/metricbeat/modules.d /etc/metricbeat/

This issue affects all new installations on DEB and RPM. Upgrades will run, but use old configurations defined in the modules.d directory from the previous installation.

The issue will be fixed in the 6.4.1 release.

Breaking changes

Affecting all Beats

• Set default kafka version to 1.0.0 in kafka output. Older versions are still supported by configuring the version setting. Minimally supported version is 0.11 (older versions might work, but are untested). 7025

Heartbeat

• Rename http.response.status to http.response.status_code to align with ECS. 7274

• Remove type field as not needed. 7307

Metricbeat

• Fixed typo in values for state_container status.phase, from terminate to terminated. 6916

• RabbitMQ management plugin path is now configured at the module level instead of having to do it in each of the metricsets. New management_path_prefix option should be used now 7074

• RabbitMQ node metricset only collects metrics of the instance it connects to, node.collect: cluster can be used to collect all nodes as before. 6556 6971

• Change http/server metricset to put events by default under http.server and prefix config options with server.. 7100

• Disable dedotting in docker module configuration. This will change the out-of-the-box behaviour, but not the one of already configured instances. 7485

• Fix typo in etcd/self metricset fields from *.bandwithrate to *.bandwidthrate. 7456

• Changed the definition of the system.cpu.total.pct and system.cpu.total.norm.cou fields to exclude the IOWait time. 7691

Bugfixes

Affecting all Beats

• Error out on invalid Autodiscover template conditions settings. 7200

• Allow to override the ignore_above option when defining new field with the type keyword. 7238

• Fix a panic on the Dissect processor when we have data remaining after the last delimiter. 7449

• When we fail to build a Kubernetes' indexer or matcher we produce a warning but we don’t add them to the execution. 7466

• Fix default value for logging.files.keepfiles. It was being set to 0 and now it’s set to the documented value of 7. 7494

• Retain compatibility with older Docker server versions. 7542

• Fix errors unpacking configs modified via CLI by ignoring -E key=value pairs with missing value. 7599

Auditbeat

• Allow auditbeat setup to run without requiring elevated privileges for the audit client. 7111

• Fix goroutine leak that occurred when the auditd module was stopped. 7163

Filebeat

• Fix a data race between stopping and starting of the harvesters. 6879

• Fix an issue when parsing ISO8601 dates with timezone definition 7367

• Fix Grok pattern of MongoDB module. 7568

• Fix registry duplicates and log resending on upgrade. 7634

Metricbeat

• Fix Windows service metricset when using a 32-bit binary on a 64-bit OS. 7294

• Do not report Metricbeat container host as hostname in Kubernetes deployment. 7199

• Ensure metadata updates don’t replace existing pod metrics. 7573

• Fix kubernetes pct fields reporting. 7677

• Add support for new kube_node_status_condition in Kubernetes state_node. 7699

Added

Affecting all Beats

• Add dissect processor. 6925

• Add IP-addresses and MAC-addresses to add_host_metadata. 6878

• Added a seccomp (secure computing) filter on Linux that whitelists the necessary system calls used by each Beat. 5213

• Ship fields.yml as part of the binary 4834

• Added options to dev-tools/cmd/dashboards/export_dashboard.go: -indexPattern to include index-pattern in output, -quiet to be quiet. 7101

• Add Indexer indexing by pod uid. Enable pod uid metadata gathering in add_kubernetes_metadata. Extended Matcher log_path matching to support volume mounts 7072

• Add default_fields to Elasticsearch template when connecting to Elasticsearch >= 7.0. 7015

• Add support for loading a template.json file directly instead of using fields.yml. 7039

• Add support for keyword multifields in field.yml. 7131

• Add experimental Jolokia Discovery autodiscover provider. 7141

• Add owner object info to Kubernetes metadata. 7231

• Add Beat export dashboard command. 7239

• Add support for docker autodiscover to monitor containers on host network 6708

• Add ability to define input configuration as stringified JSON for autodiscover. 7372

• Add processor definition support for hints builder 7386

• Add support to disable html escaping in outputs. 7445

• Refactor error handing in schema.Apply(). 7335

• Add additional types to Kubernetes metadata 7457

• Add module state reporting for Beats Monitoring. 7075

• Release the rename processor as GA. 7656

• Add support for Openstack Nova in add_cloud_metadata processor. 7663

• Add support to set Beats services to automatic-delayed start on Windows. 8711

Auditbeat

• Added XXH64 hash option for file integrity checks. 7311

• Added the show auditd-rules and show auditd-status commands to show kernel rules and status. 7114

• Add Kubernetes specs for auditbeat file integrity monitoring 7642

Filebeat

• Add Kibana module with log fileset. 7052

• Support MySQL 5.7.19 by mysql/slowlog 6969

• Correctly join partial log lines when using docker input. 6967

• Add support for TLS with client authentication to the TCP input 7056

• Converted part of pipeline from treafik/access metricSet to dissect to improve efficiency. 7209

• Add GC fileset to the Elasticsearch module. 7305

• Add Audit log fileset to the Elasticsearch module. 7365

• Add Slow log fileset to the Elasticsearch module. 7473

• Add deprecation fileset to the Elasticsearch module. 7474

• Add convert_timezone option to Kafka module to convert dates to UTC. 7546 7578

• Add patterns for kafka 1.1 logs. 7608

• Move debug messages in tcp input source 7712

Metricbeat

• Add experimental Elasticsearch index metricset. 6881

• Add dashboards and visualizations for haproxy metrics. 6934

• Add Jolokia agent in proxy mode. 6475

• Add message rates to the RabbitMQ queue metricset 6442 6606

• Add exchanges metricset to the RabbitMQ module 6442 6607

• Add Elasticsearch index_summary metricset. 6918

• Add shard metricset to Elasticsearch module. 7006

• Add apiserver metricset to Kubernetes module. 7059

• Add maxmemory to redis info metricset. 7127

• Set guest as default user in RabbitMQ module. 7107

• Add postgresql statement metricset. 7048 7060

• Update state_container metricset to support latest kube-state-metrics version. 7216

• Add TLS support to MongoDB module. 7401

• Added Traefik module with health metricset. 7413

• Add Elasticsearch ml_job metricsets. 7196

• Add support for bearer token files to HTTP helper. 7527

• Add Elasticsearch index recovery metricset. 7225

• Add locks, global_locks, oplatencies and process fields to status metricset of MongoDB module. 7613

• Run Kafka integration tests on version 1.1.0 7616

• Release raid and socket metricset from system module as GA. 7658

• Release elasticsearch module and all its metricsets as beta. 7662

• Release munin and traefik module as beta. 7660

• Add envoyproxy module. 7569

• Release prometheus collector metricset as GA. 7660

• Add Elasticsearch cluster_stats metricset. 7638

• Added basepath setting for HTTP-based metricsets 7700

• Add couchdb module. 9406

Packetbeat

• The process monitor now reports the command-line for all processes, under Linux and Windows. 7135

• Updated the TLS protocol parser with new cipher suites added to TLS 1.3. 7455

• Flows are enriched with process information using the process monitor. 7507

• Added UDP support to process monitor. 7571

Deprecated

Metricbeat

• Kubernetes state_container cpu.limit.nanocores and cpu.request.nanocores have been deprecated in favor of cpu.*.cores. 6916

Beats version 6.3.1

View commits

Bugfixes

Affecting all Beats

• Allow index-pattern only setup when setup.dashboards.only_index=true. 7285

• Preserve the event when source matching fails in add_docker_metadata. 7133

• Negotiate Docker API version from our client instead of using a hardcoded one. 7165

• Fix duplicating dynamic_fields in template when overwriting the template. 7352

Auditbeat

• Fixed parsing of AppArmor audit messages. 6978

Filebeat

• Comply with PostgreSQL database name format 7198

• Optimize PostgreSQL ingest pipeline to use anchored regexp and merge multiple regexp into a single expression. 7269

• Keep different registry entry per container stream to avoid wrong offsets. 7281

• Fix offset field pointing at end of a line. 6514

• Commit registry writes to stable storage to avoid corrupt registry files. 6792

Metricbeat

• Fix field mapping for the system process CPU ticks fields. 7230

• Ensure canonical naming for JMX beans is disabled in Jolokia module. 7047

• Fix Jolokia attribute mapping when using wildcards and MBean names with multiple properties. 7321

Packetbeat

• Fix an out of bounds access in HTTP parser caused by malformed request. 6997

• Fix missing type for http.response.body field. 7169

Added

Auditbeat

• Added caching of UID and GID values to auditd module. 6978

• Updated syscall tables for Linux 4.16. 6978

• Added better error messages for when the auditd module fails due to the Linux kernel not supporting auditing (CONFIG_AUDIT=n). 7012

Metricbeat

• Collect accumulated docker network metrics and mark old ones as deprecated. 7253

Beats version 6.3.0

View commits

Breaking changes

Affecting all Beats

• De dot keys of labels and annotations in kubernetes meta processors to prevent collisions. 6203

• Rename beat.cpu..time metrics to beat.cpu..time.ms. 6449

• Add host.name field to all events, to avoid mapping conflicts. This could be breaking Logstash configs if you rely on the host field being a string. 7051

Filebeat

• Add validation for Stdin, when Filebeat is configured with Stdin and any other inputs, Filebeat will now refuse to start. 6463

• Mark system.syslog.message and system.auth.message as text instead of keyword. 6589

Metricbeat

• De dot keys in kubernetes/event metricset to prevent collisions. 6203

• Add config option for windows/perfmon metricset to ignore non existent counters. 6432

• Refactor docker CPU calculations to be more consistent with docker stats. 6608

• Update logstash.node_stats metricset to write data under logstash.node.stats.*. 6714

Bugfixes

Affecting all Beats

• Fix panic when Events containing a float32 value are normalized. 6129

• Fix setup.dashboards.always_kibana when using Kibana 5.6. 6090

• Fix for Kafka logger. 6430

• Remove double slashes in Windows service script. 6491

• Ensure Kubernetes labels/annotations don’t break mapping 6490

• Ensure that the dashboard zip files can’t contain files outside of the kibana directory. 6921

• Fix map overwrite panics by cloning shared structs before doing the update. 6947

• Fix delays on autodiscovery events handling caused by blocking runner stops. 7170

• Do not emit Kubernetes autodiscover events for Pods without IP address. 7235

• Fix self metrics when containerized 6641

Auditbeat

• Add hex decoding for the name field in audit path records. 6687

• Fixed a deadlock in the file_integrity module under Windows. 6864

• Fixed parsing of AppArmor audit messages. 6978

• Allow auditbeat setup to run without requiring elevated privileges for the audit client. 7111

• Fix goroutine leak that occurred when the auditd module was stopped. 7163

Filebeat

• Fix panic when log prospector configuration fails to load. 6800

• Fix memory leak in log prospector when files cannot be read. 6797

• Add raw JSON to message field when JSON parsing fails. 6516

• Commit registry writes to stable storage to avoid corrupt registry files. 6877

• Fix a parsing issue in the syslog input for RFC3339 timestamp and time with nanoseconds. 7046

• Fix an issue with an overflowing wait group when using the TCP input. 7202

Heartbeat

• Fix race due to updates of shared a map, that was not supposed to be shared between multiple go-routines. 6616

Metricbeat

• Fix the default configuration for Logstash to include the default port. 6279

• Fix dealing with new process status codes in Linux kernel 4.14+. 6306

• Add filtering option by exact device names in system.diskio. diskio.include_devices. 6085

• Add connections metricset to RabbitMQ module 6548

• Fix panic in http dependent modules when invalid config was used. 6205

• Fix system.filesystem.used.pct value to match what df reports. 5494

• Fix namespace disambiguation in Kubernetes state_* metricsets. 6281

• Fix Windows perfmon metricset so that it sends metrics when an error occurs. 6542

• Fix Kubernetes calculated fields store. 6564

• Exclude bind mounts in fsstat and filesystem metricsets. 6819

• Don’t stop Metricbeat if aerospike server is down. 6874

• disk reads and write count metrics in RabbitMQ queue metricset made optional. 6876

• Add mapping for docker metrics per cpu. 6843

Winlogbeat

• Fixed a crash under Windows 2003 and XP when an event had less insert strings than required by its format string. 6247

• Fix config validation to allow event_logs.processors. 6217

Added

Affecting all Beats

• Update Golang 1.9.4 6326

• Add the ability to log to the Windows Event Log. 5913

• The node name can be discovered automatically by machine-id matching when beat deployed outside Kubernetes cluster. 6146

• Panics will be written to the logger before exiting. 6199

• Add builder support for autodiscover and annotations builder 6408

• Add plugin support for autodiscover builders, providers 6457

• Preserve runtime from container statuses in Kubernetes autodiscover 6456

• Experimental feature setup.template.append_fields added. 6024

• Add appender support to autodiscover 6469

• Add add_host_metadata processor 5968

• Retry configuration to load dashboards if Kibana is not reachable when the beat starts. 6560

• Add has_fields conditional to filter events based on the existence of all the given fields. 6285 6653

• Add support for spooling to disk to the beats event publishing pipeline. 6581

• Added logging of system info at Beat startup. 5946

• Do not log errors if X-Pack Monitoring is enabled but Elastisearch X-Pack is not. 6627

• Add rename processor. 6292

• Allow override of dynamic template match_mapping_type for fields with object_type. 6691

Filebeat

• Add IIS module to parse access log and error log. 6127

• Renaming of the prospector type to the input type and all prospectors are now moved to the input folder, to maintain backward compatibility type aliasing was used to map the old type to the new one. This change also affect YAML configuration. 6078

• Addition of the TCP input 6700

• Add option to convert the timestamps to UTC in the system module. 5647

• Add Logstash module support for main log and the slow log, support the plain text or structured JSON format 5481

• Add stream filtering when using docker prospector. 6057

• Add support for CRI logs format. 5630

• Add json.ignore_decoding_error config to not log json decoding erors. 6547

• Make registry file permission configurable. 6455

• Add MongoDB module. 6238

• Add Ingest pipeline loading to setup. 6814

• Add support of log_format combined to NGINX access logs. 6858

• Release config reloading feature as GA.

• Add support human friendly size for the UDP input. 6886

• Add Syslog input to ingest RFC3164 Events via TCP and UDP 6842

• Remove the undefined username option from the Redis input and clarify the documentation. 6662

Heartbeat

• Made the URL field of Heartbeat aggregateable. 6263

• Use match.Matcher for checking Heartbeat response bodies with regular expressions. 6539

Metricbeat

• Support apache status pages for versions older than 2.4.16. 6450

• Add support for huge pages on Linux. 6436

• Support to optionally 'de dot' keys in http/json metricset to prevent collisions. 5970

• Add graphite protocol metricbeat module. 4734

• Add http server metricset to support push metrics via http. 4770

• Make config object public for graphite and http server 4820

• Add system uptime metricset. 4848

• Add experimental queue metricset to RabbitMQ module. 4788

• Add additional php-fpm pool status kpis for Metricbeat module 5287

• Add etcd module. 4970

• Add ip address of docker containers to event. 5379

• Add ceph osd tree information to metricbeat 5498

• Add ceph osd_df to metricbeat 5606

• Add basic Logstash module. 5540

• Add dashboard for Windows service metricset. 5603

• Add pct calculated fields for Pod and container CPU and memory usages. 6158

• Add statefulset support to Kubernetes module. 6236

• Refactor prometheus endpoint parsing to look similar to upstream prometheus 6332

• Making the http/json metricset GA. 6471

• Add support for array in http/json metricset. 6480

• Making the jolokia/jmx module GA. 6143

• Making the MongoDB module GA. 6554

• Allow to disable labels dedot in Docker module, in favor of a safe way to keep dots. 6490

• Add experimental module to collect metrics from munin nodes. 6517

• Add support for wildcards and explicit metrics grouping in jolokia/jmx. 6462

• Set collector as default metricset in Prometheus module. 6636 6747

• Set mntr as default metricset in Zookeeper module. 6674

• Set default metricsets in vSphere module. 6676

• Set status as default metricset in Apache module. 6673

• Set namespace as default metricset in Aerospike module. 6669

• Set service as default metricset in Windows module. 6675

• Set all metricsets as default metricsets in uwsgi module. 6688

• Allow autodiscover to monitor unexposed ports 6727

• Mark kubernetes.event metricset as beta. 6715

• Set all metricsets as default metricsets in couchbase module. 6683

• Mark uwsgi module and metricset as beta. 6717

• Mark Golang module and metricsets as beta. 6711

• Mark system.raid metricset as beta. 6710

• Mark http.server metricset as beta. 6712

• Mark metricbeat logstash module and metricsets as beta. 6713

• Set all metricsets as default metricsets in Ceph module. 6676

• Set container, cpu, diskio, healthcheck, info, memory and network in docker module as default. 6718

• Set cpu, load, memory, network, process and process_summary as default metricsets in system module. 6689

• Set collector as default metricset in Dropwizard module. 6669

• Set info and keyspace as default metricsets in redis module. 6742

• Set connection as default metricset in rabbitmq module. 6743

• Set all metricsets as default metricsets in Elasticsearch module. 6755

• Set all metricsets as default metricsets in Etcd module. 6756

• Set server metricsets as default in Graphite module. 6757

• Set all metricsets as default metricsets in HAProxy module. 6758

• Set all metricsets as default metricsets in Kafka module. 6759

• Set all metricsets as default metricsets in postgresql module. 6761

• Set status metricsets as default in Kibana module. 6762

• Set all metricsets as default metricsets in Logstash module. 6763

• Set container, node, pod, system, volume as default in Kubernetes module. https://github.com/elastic/beats/pull/ 6764[6764]

• Set stats as default in memcached module. 6765

• Set all metricsets as default metricsets in Mongodb module. 6766

• Set pool as default metricset for php_fpm module. 6768

• Set status as default metricset for mysql module. https://github.com/elastic/beats/pull/ 6769[6769]

• Set stubstatus as default metricset for nginx module. 6770

• Added support for haproxy 1.7 and 1.8. 6793

• Add accumulated I/O stats to diskio in the line of docker stats. 6701

• Ignore virtual filesystem types by default in system module. 6819

• Release config reloading feature as GA. 6891

• Kubernetes deployment: Add ServiceAccount config to system metricbeat. 6824

• Kubernetes deployment: Add DNS Policy to system metricbeat. 6656

Packetbeat

• Add support for condition on bool type 5659 5954

• Fix high memory usage on HTTP body if body is not published. 6680

• Allow to capture the HTTP request or response bodies independently. 6784

• HTTP publishes an Error event for unmatched requests or responses. 6794

Winlogbeat

• Use bookmarks to persist the last published event. 6150

Beats version 6.2.3

View commits

Breaking changes

Affecting all Beats

• Fix conditions checking on autodiscover Docker labels. 6412

Bugfixes

Affecting all Beats

• Avoid panic errors when processing nil Pod events in add_kubernetes_metadata. 6372

• Fix infinite failure on Kubernetes watch 6504

Metricbeat

• Fix Kubernetes overview dashboard views for non default time ranges. 6395

Beats version 6.2.2

View commits

Bugfixes

Affecting all Beats

• Add logging when monitoring cannot connect to Elasticsearch. 6365

• Fix infinite loop when event unmarshal fails in Kubernetes pod watcher. 6353

Filebeat

• Fix a conversion issue for time related fields in the Logstash module for the slowlog fileset. 6317

Beats version 6.2.1

View commits

No changes in this release.

Beats version 6.2.0

View commits

Breaking changes

Affecting all Beats

• The log format may differ due to logging library changes. 5901

• The default value for pipelining is reduced to 2 to avoid high memory in the Logstash beats input. 6250

Auditbeat

• Split the audit.kernel and audit.file metricsets into their own modules named auditd and file_integrity, respectively. This change requires existing users to update their config. 5422

• Renamed file_integrity module fields. 5423 5995

• Renamed auditd module fields. 5423 6080

Metricbeat

• Rename golang.heap.system.optained field to golang.heap.system.obtained. 5703

• De dot keys in jolokia/jmx metricset to prevent collisions. 5957

Bugfixes

Auditbeat

• Fixed an issue where the proctitle value was being truncated. 6080

• Fixed an issue where values were incorrectly interpreted as hex data. 6080

• Fixed parsing of the key value when multiple keys are present. 6080

• Fix possible resource leak if file_integrity module is used with config reloading on Windows or Linux. 6198

Filebeat

• Fix variable name for convert_timezone in the system module. 5936

Metricbeat

• Fix error datastore '*' not found in Vsphere module. 4879

• Fix error NotAuthenticated in Vsphere module. 4673

• Fix mongodb session consistency mode to allow command execution on secondary nodes. 4689

• Fix kubernetes state_pod status.phase so that the active phase is returned instead of unknown. 5980

• Fix error collecting network_names in Vsphere module. 5962

• Fix process cgroup memory metrics for memsw, kmem, and kmem_tcp. 6033

• Fix kafka OffsetFetch request missing topic and partition parameters. 5880

Packetbeat

• Fix mysql SQL parser to trim \r from Windows Server SELECT\r\n\t1. 5572

Added

Affecting all Beats

• Adding a local keystore to allow user to obfuscate password 5687

• Add autodiscover for kubernetes. 6055

• Add Beats metrics reporting to Xpack. 3422

• Update the command line library cobra and add support for zsh completion 5761

• Update to Golang 1.9.2

• Moved ip_port indexer for add_kubernetes_metadata to all beats. 5707

• ip_port indexer now index both IP and IP:port pairs. 5721

• Add the ability to write structured logs. 5901

• Use structured logging for the metrics that are periodically logged via the logging.metrics feature. 5915

• Improve Elasticsearch output metrics to count number of dropped and duplicate (if event ID is given) events. 5811

• Add the ability for the add_docker_metadata process to enrich based on process ID. 6100

• The add_docker_metadata and add_kubernetes_metadata processors are now GA, instead of Beta. 6105

• Update go-ucfg library to support top level key reference and cyclic key reference for the keystore 6098

Auditbeat

• Auditbeat is marked as GA, no longer Beta. 5432

• Add support for BLAKE2b hash algorithms to the file integrity module. 5926

• Add support for recursive file watches. 5575 5833

Filebeat

• Add Osquery module. 5971

• Add stream filtering when using docker prospector. 6057

Metricbeat

• Add ceph osd_df to metricbeat 5606

• Add field network_names of hosts and virtual machines. 5646

• Add experimental system/raid metricset. 5642

• Add a dashboard for the Nginx module. 5991

• Add experimental mongodb/collstats metricset. 5852

• Update the MySQL dashboard to use the Time Series Visual Builder. 5996

• Add experimental uwsgi module. 6006

• Docker and Kubernetes modules are now GA, instead of Beta. 6105

• Support haproxy stats gathering using http (additionally to tcp socket). 5819

• Support to optionally 'de dot' keys in http/json metricset to prevent collisions. 5957

Packetbeat

• Configure good defaults for add_kubernetes_metadata. 5707

Beats version 6.1.3

View commits

No changes in this release.

Beats version 6.1.2

View commits

Bugfixes

Auditbeat

• Add an error check to the file integrity scanner to prevent a panic when there is an error reading file info via lstat. 6005

Added

Filebeat

• Switch to docker prospector in sample manifests for Kubernetes deployment 5963

Beats version 6.1.1

View commits

No changes in this release.

Beats version 6.1.0

View commits

Breaking changes

Auditbeat

• Changed audit.file.path to be a multi-field so that path is searchable. 5625

Metricbeat

• Rename heap_init field to heap.init in the Elasticsearch module. 5320

• Rename http.response.status_code field to http.response.code in the HTTP module. 5521

Bugfixes

Affecting all Beats

• Remove ID() from Runner interface 5153

• Correctly send configured Host header to the remote server. 4842

• Change add_kubernetes_metadata to attempt detection of namespace. 5482

• Avoid double slash when join url and path 5517

• Fix console color output for Windows. 5611

• Fix logstash output debug message. https://github.com/elastic/beats/pull/5799{5799]

• Fix isolation of modules when merging local and global field settings. 5795

• Report ephemeral ID and uptime in monitoring events on all platforms 6501

Filebeat

• Add support for adding string tags 5395

• Fix race condition when limiting the number of harvesters running in parallel 5458

• Fix relative paths in the prospector definitions. 5443

• Fix recursive_globe.enabled option. 5443

Metricbeat

• Change field type of http header from nested to object 5258

• Fix the fetching of process information when some data is missing under MacOS X. 5337

• Change MySQL active connections visualization title to MySQL total connections. 4812

• Fix ProcState on Linux and FreeBSD when process names contain parentheses. 5775

• Fix incorrect Mem.Used calculation under linux. 5775

• Fix open_file_descriptor_count and max_file_descriptor_count lost in zookeeper module 5902

• Fix system process metricset for kernel processes. 5700

• Change kubernetes.node.cpu.allocatable.cores to float. 6130

Packetbeat

• Fix http status phrase parsing not allow spaces. 5312

• Fix http parse to allow to parse get request with space in the URI. 5495

• Fix mysql SQL parser to trim \r from Windows Server SELECT\r\n\t1. 5572

• Fix corruption when parsing repeated headers in an HTTP request or response. 6325

• Fix panic when parsing partial AMQP messages. 6384

• Fix out of bounds access to slice in MongoDB parser. 6256

• Fix sniffer hanging on exit under Linux. 6535

• Fix bounds check error in http parser causing a panic. 6750

Winlogbeat

• Fix the registry file. It was not correctly storing event log names, and upon restart it would begin reading at the start of each event log. 5813

Added

Affecting all Beats

• Support dashboard loading without Elasticsearch 5653

• Changed the hashbang used in the beat helper script from /bin/bash to /usr/bin/env bash. 5051

• Changed beat helper script to use exec when running the beat. 5051

• Fix reloader error message to only print on actual error 5066

• Add support for enabling TLS renegotiation. 4386

• Add Azure VM support for add_cloud_metadata processor 5355

• Add output.file.permission config option. 4638

• Refactor add_kubernetes_metadata to support autodiscovery 5434

• Improve custom flag handling and CLI flags usage message. 5543

• Add number_of_routing_shards config set to 30 5570

• Set log level for kafka output. 5397

• Move TCP UDP start up into server.Start() 4903

• Update to Golang 1.9.2

Auditbeat

• Add support for SHA3 hash algorithms to the file integrity module. 5345

• Add dashboards for Linux audit framework events (overview, executions, sockets). 5516

Filebeat

• Add PostgreSQL module with slowlog support. 4763

• Add Kafka log module. 4885

• Add support for /var/log/containers/ log path in add_kubernetes_metadata processor. 4981

• Remove error log from runnerfactory as error is returned by API. 5085

• Add experimental Docker json-file prospector . 5402

• Add experimental Docker autodiscover functionality. 5245

• Add option to convert the timestamps to UTC in the system module. 5647

• Add Logstash module support for main log and the slow log, support the plain text or structured JSON format 5481

Metricbeat

• Add graphite protocol metricbeat module. 4734

• Add http server metricset to support push metrics via http. 4770

• Make config object public for graphite and http server 4820

• Add system uptime metricset. 4848

• Add experimental queue metricset to RabbitMQ module. 4788

• Add additional php-fpm pool status kpis for Metricbeat module 5287

• Add etcd module. 4970

• Add ip address of docker containers to event. 5379

• Add ceph osd tree information to Metricbeat 5498

• Add basic Logstash module. 5540

• Add dashboard for Windows service metricset. 5603

• Add experimental Docker autodiscover functionality. 5245

• Add Windows service metricset in the windows module. 5332

• Update gosigar to v0.6.0. 5775

Packetbeat

• Add support for decoding the TLS envelopes. 5476

• HTTP parses successfully on empty status phrase. 6176

• HTTP parser supports broken status line. 6631

Beats version 6.0.1

View commits

Bugfixes

Affecting all Beats

• Fix documentation links in README.md files. 5710

• Fix add_docker_metadata dropping some containers. 5788

Heartbeat

• Fix the "HTTP up status" visualization. 5564

Metricbeat

• Fix map overwrite in docker diskio module. 5582

• Fix connection leak in mongodb module. 5688

• Fix the include top N processes feature for cases where there are fewer processes than N. 5729

libbeat/docs/release-notes/6.0.0.asciidoc

Beats version 6.0.0-GA

View commits

The list below covers the changes between 6.0.0-rc2 and 6.0.0 GA only.

Bugfixes

Filebeat

• Fix machine learning jobs setup for dynamic modules. 5509

Packetbeat

• Fix missing length check in the PostgreSQL module. 5457

• Fix panic in ACK handler if event is dropped on blocked queue 5524

Added

Filebeat

• Add Kubernetes manifests to deploy Filebeat. 5349

• Add container short ID matching to add_docker_metadata. 6172

Metricbeat

• Add Kubernetes manifests to deploy Metricbeat. 5349

Beats version 6.0.0-rc2

View commits

Breaking changes

Packetbeat

• Remove not-working runoptions.uid and runoptions.gid options in Packetbeat. 5261

Bugfixes

Affecting all Beats

• Fix data race accessing watched containers. 5147

• Do not require template if index change and template disabled 5319

• Fix missing ACK in redis output. 5404

Filebeat

• Fix default paths for redis 4.0.1 logs on macOS 5173

• Fix Filebeat not starting if command line and modules configs are used together. 5376

• Fix double @timestamp field when JSON decoding was used. 5436

Metricbeat

• Use beat.name instead of beat.hostname in the Host Overview dashboard. 5340

• Fix the loading of 5.x dashboards. 5277

Added

Metricbeat

• Auto-select a hostname (based on the host on which the Beat is running) in the Host Overview dashboard. 5340

Deprecated

Filebeat

• The filebeat.config_dir option is deprecated. Use filebeat.config.prospector options instead. 5321

Beats version 6.0.0-rc1

View commits

Bugfixes

Affecting all Beats

• Fix the /usr/bin/beatname script to accept -d "*" as a parameter. 5040

• Combine fields.yml properties when they are defined in different sources. 5075

• Keep Docker & Kubernetes pod metadata after container dies while they are needed by processors. 5084

• Fix fields.yml lookup when using export template with a custom path.config param. 5089

• Remove runner creation from every reload check 5141

• Fix add_kubernetes_metadata matcher registry lookup. 5159

Metricbeat

• Fix a memory allocation issue where more memory was allocated than needed in the windows-perfmon metricset. 5035

• Don’t start metricbeat if external modules config is wrong and reload is disabled 5053

• The MongoDB module now connects on each fetch, to avoid stopping the whole Metricbeat instance if MongoDB is not up when starting. 5120

• Fix kubernetes events module to be able to index time fields properly. 5093

• Fixed cmd_set and cmd_get being mixed in the Memcache module. 5189

Added

Affecting all Beats

• Enable flush timeout by default. 5150

• Add @metadata.version to events send to Logstash. 5166

Auditbeat

• Changed the number of shards in the default configuration to 3. 5095

• Add support for receiving audit events using a multicast socket. 4850

Filebeat

• Changed the number of shards in the default configuration to 3. 5095

• Don’t start filebeat if external modules/prospectors config is wrong and reload is disabled 5053

• Add filebeat.registry_flush setting, to delay the registry updates. 5146

Heartbeat

• Changed the number of shards in the default configuration to 1. 5095

Packetbeat

• Changed the number of shards in the default configuration to 3. 5095

Winlogbeat

• Changed the number of shards in the default configuration to 3. 5095

Beats version 6.0.0-beta2

View commits

Breaking changes

Affecting all Beats

• The log directory (path.log) for Windows services is now set to C:\ProgramData\[beatname]\logs. 4764

• The _all field is disabled in Elasticsearch 6.0. This means that searching by individual words only work on text fields. 4901

• Fail if removed setting output.X.flush_interval is explicitly configured.

• Rename the /usr/bin/beatname.sh script (e.g. metricbeat.sh) to /usr/bin/beatname. 4933

• Beat does not start if elasticsearch index pattern was modified but not the template name and pattern. 4769

• Fail if removed setting output.X.flush_interval is explicitly configured. 4880

Bugfixes

Affecting all Beats

• Register kubernetes field_format matcher and remove logger in Encode API 4888

• Fix go plugins not loaded when beat starts 4799

• Add support for initContainers in add_kubernetes_metadata processor. 4825

• Eliminate deprecated default mapping in 6.x 4864

• Fix pod name indexer to use both namespace, pod name to frame index key 4775

Filebeat

• Fix issue where the fileset.module could have the wrong value. 4761

Heartbeat

• Fix monitor.name being empty by default. 4852

• Fix wrong event timestamps. 4851

Metricbeat

• Added missing mongodb configuration file to the modules.d folder. 4870

• Fix wrong MySQL CRUD queries timelion visualization 4857

• Add new metrics to CPU metricset 4969

Packetbeat

• Update flow timestamp on each packet being received. 4895

Added

Affecting all Beats

• Add setting to enable/disable the slow start in logstash output. 4972

• Update init scripts to use the test config subcommand instead of the deprecated -configtest flag. 4600

• Get by default the credentials for connecting to Kibana from the Elasticsearch output configuration. 4867

• Added cloud.id and cloud.auth settings, for simplifying using Beats with the Elastic Cloud. 4959

• Add lz4 compression support to kafka output. 4977

• Add newer kafka versions to kafka output. 4977

• Configure the index name when loading the dashboards and the index pattern. 4949

Metricbeat

• Add filesystem.ignore_types to system module for ignoring filesystem types. 4685

• Add support to exclude labels from kubernetes pod metadata. 4757

Beats version 6.0.0-beta1

View commits

Breaking changes

Affecting all Beats

• Rename kubernetes processor to add_kubernetes_metadata. 4473

• Rename .full.yml config files to .reference.yml. 4563

• The scripts/import_dashboards is removed from packages. Use the setup command instead. 4586

• Change format of the saved kibana dashboards to have a single JSON file for each dashboard 4413

• Rename configtest command to test config. 4590

• Remove setting queue_size and bulk_queue_size. 4650

• Remove setting dashboard.snapshot and dashboard.snapshot_url. They are no longer needed because the dashboards are included in the packages by default. 4675

• Beats can no longer be launched from Windows Explorer (GUI), command line is required. 4420

Auditbeat

• Changed file metricset config to make file.paths a list instead of a dictionary. 4796

Heartbeat

• Renamed the heartbeat RPM/DEB name to heartbeat-elastic. 4601

Metricbeat

• Change all system.cpu.*.pct metrics to be scaled by the number of CPU cores. This will make the CPU usage percentages from the system cpu metricset consistent with the system process metricset. The documentation for these metrics already stated that on multi-core systems the percentages could be greater than 100%. 4544

• Remove filters setting from metricbeat modules. 4699

• Added type field to filesystem metrics. 4717

Packetbeat

• Remove the already unsupported pf_ring sniffer option. 4608

Bugfixes

Affecting all Beats

• Don’t stop with error loading the ES template if the ES output is not enabled. 4436

• Fix race condition in internal logging rotator. 4519

• Normalize all times to UTC to ensure proper index naming. 4569

• Fix issue with loading dashboards to ES 6.0 when .kibana index did not already exist. 4659

Auditbeat

• Fix file.max_file_size config option for the audit file metricset. 4796

Filebeat

• Fix issue where the fileset.module could have the wrong value. 4761

Metricbeat

• Fix issue affecting Windows services timing out at startup. 4491

• Fix incorrect docker.diskio.total metric calculation. 4507

• Vsphere module: used memory field corrected. 4461

Packetbeat

• Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing. 4442

Winlogbeat

• Removed validation of top-level config keys. This behavior was inconsistent with other Beats and caused maintainability issues. 4657

Added

Affecting all Beats

• New cli subcommands interface. 4420

• Allow source path matching in add_docker_metadata processor. 4495

• Add support for analyzers and multifields in fields.yml. 4574

• Add support for JSON logging. 4523

• Add test output command, to test Elasticsearch and Logstash output settings. 4590

• Introduce configurable event queue settings: queue.mem.events, queue.mem.flush.min_events and queue.mem.flush.timeout. 4650

• Enable pipelining in Logstash output by default. 4650

• Added 'result' field to Elasticsearch QueryResult struct for compatibility with 6.x Index and Delete API responses. 4661

• The sample dashboards are now included in the Beats packages. 4675

• Add pattern option to be used in the fields.yml to specify the pattern for a number field. 4731

Auditbeat

• Added file.hash_types config option for controlling the hash types. 4796

• Added the ability to specify byte unit suffixes to file.max_file_size. 4796

Filebeat

• Add experimental Redis module. 4441

• Nginx module: use the first not-private IP address as the remote_ip. 4417

• Load Ingest Node pipelines when the Elasticsearch connection is established, instead of only once at startup. 4479

• Add support for loading Xpack Machine Learning configurations from the modules, and added sample configurations for the Nginx module. 4506 4609

• Add udp prospector type. 4452

• Enabled Cgo which means libc is dynamically compiled. 4546

• Add Beta module config reloading mechanism 4566

• Remove spooler and publisher components and settings. 4644

Heartbeat

• Enabled Cgo which means libc is dynamically compiled. 4546

Metricbeat

• Add random startup delay to each metricset to avoid the thundering herd problem. 4010

• Add the ability to configure audit rules to the kernel module. 4482

• Add the ability to configure kernel’s audit failure mode. 4516

• Add experimental Aerospike module. 4560

• Vsphere module: collect custom fields from virtual machines. 4464

• Add test modules command, to test modules expected output. 4656

• Add processors setting to metricbeat modules. 4699

• Support npipe protocol (Windows) in Docker module. 4751

Winlogbeat

• Add the ability to use LevelRaw if Level isn’t populated in the event XML. 4257

Auditbeat

• Add file integrity metricset to the audit module. 4486

Beats version 6.0.0-alpha2

View commits

Breaking changes

Filebeat

• Rename input_type field to prospector.type 4294

• The @metadata.type field, added by the Logstash output, is now hardcoded to doc and will be removed in future versions. 4331.

Bugfixes

Affecting all Beats

• Fix importing the dashboards when the limit for max open files is too low. 4244

• Fix configuration documentation for kubernetes processor 4313

• Fix misspelling in add_locale configuration option for abbreviation.

Filebeat

• Fix race condition on harvester stopping with reloading enabled. 3779

• Fix recursive glob config parsing and resolution across restarts. 4269

• Allow string characters in user agent patch version (NGINX and Apache) 4415

• Fix grok pattern in filebeat module system/auth without hostname. 4224

Metricbeat

• Set correct format for percent fields in memory module. 4619

• Fix a debug statement that said a module wrapper had stopped when it hadn’t. 4264

• Use MemAvailable value from /proc/meminfo on Linux 3.14. 4316

• Fix panic when events were dropped by filters. 4327

• Add filtering to system filesystem metricset to remove relative mountpoints like those from Linux network namespaces. 4370

• Remove unnecessary print statement in schema apis. 4355

• Fix type of field haproxy.stat.check.health.last. 4407

Packetbeat - Enable memcache filtering only if a port is specified in the config file. 4335 - Enable memcache filtering only if a port is specified in the config file. 4335

Added

Affecting all Beats

• Upgraded to Golang 1.8.3. 4401

• Added the possibility to set Elasticsearch mapping template settings from the Beat configuration file. 4284 4317

• Add a variable to the SysV init scripts to make it easier to change the user. 4340

• Add the option to write the generated Elasticsearch mapping template into a file. 4323

• Add instance_name in GCE add_cloud_metadata processor. 4414

• Add add_docker_metadata processor. 4352

• Add logging.files permissions option. 4295

Filebeat - Added ability to sort harvested files. 4374 - Add experimental Redis slow log prospector type. 4180

Metricbeat

• Add macOS implementation of the system diskio metricset. 4144

• Add process_summary metricset that records high level metrics about processes. 4231

• Add kube-state-metrics based metrics to kubernetes module 4253

• Add debug logging to Jolokia JMX metricset. 4341

• Add events metricset for kubernetes metricbeat module 4315

• Change Metricbeat default configuration file to be better optimized for most users. 4329

• Add experimental RabbitMQ module. 4394

• Add Kibana dashboard for the Kubernetes modules. 4138

Packetbeat

Winlogbeat

Deprecated

Affecting all Beats

• The @metadata.type field, added by the Logstash output, is deprecated, hardcoded to doc and will be removed in future versions. 4331.

Filebeat

• Deprecate input_type prospector config. Use type config option instead. 4294

Known Issue

• If the Elasticsearch output is not enabled, but setup.template options are present (like it’s the case in the default Metricbeat configuration), the Beat stops with an error: "Template loading requested but the Elasticsearch output is not configured/enabled". To avoid this error, disable the template loading explicitly setup.template.enabled: false.

Beats version 6.0.0-alpha1

View commits

Breaking changes

Affecting all Beats

• Introduce beat version in the Elasticsearch index and mapping template 3527

• Usage of field _type is now ignored and hardcoded to doc. 3757

• Change vendor manager from glide to govendor. 3851

• Rename error field to error.message. 3987

• Change dashboards. config options to setup.dashboards.. 3921

• Change outputs.elasticsearch.template.* to `setup.template.* 4080

Filebeat

• Remove code to convert states from 1.x. 3767

• Remove deprecated config options force_close_files and close_older. 3768

• Change clean_removed behaviour to also remove states for files which cannot be found anymore under the same name. 3827

• Remove document_type config option. Use fields instead. 4204

• Move json_error under error.message and error.key. 4167

Packetbeat

• Remove deprecated geoip. 3766

• Replace waitstop command line argument by shutdown_timeout in configuration file. 3588

Winlogbeat

• Remove metrics endpoint. Replaced by http endpoint in libbeat (see #3717). 3901

Bugfixes

Affecting all Beats

• Add _id, _type, _index and _score fields in the generated index pattern. 3282

Filebeat

• Fix the Mysql slowlog parsing of IP addresses. 4183

• Fix issue that new prospector was not reloaded on conflict 4128

Heartbeat

• Use IP type of elasticsearch for ip field. 3926

Metricbeat

• Support common.Time in mapstriface.toTime() 3812

• Fix MongoDB dbstats fields mapping. 4025

• Fixing prometheus collector to aggregate metrics based on metric family. 4075

• Fixing multiEventFetch error reporting when no events are returned 4153

Added

Affecting all Beats

• Initialize a beats UUID from file on startup. 3615

• Add new add_locale processor to export the local timezone with an event. 3902

• Add http endpoint. 3717

• Updated to Go 1.8.1. 4033

• Add kubernetes processor 3888

• Add support for include_labels and include_annotations in kubernetes processor 4043

• Support new index_patterns field when loading templates for Elasticsearch >= 6.0 4056

• Adding goimports support to make check and fmt 4114

• Make kubernetes indexers/matchers pluggable 4151

• Abstracting pod interface in kubernetes plugin to enable easier vendoring 4152

Filebeat

• Restructure input.Event to be inline with outputs.Data 3823

• Add base for supporting prospector level processors 3853

• Add filebeat.config.path as replacement for config_dir. 4051

• Add a recursive_glob.enabled setting to expand ** in patterns. 3980

• Add Icinga module. 3904

• Add ability to parse nginx logs exposing the X-Forwarded-For header instead of the remote address.

Heartbeat

• Event format and field naming changes in Heartbeat and sample Dashboard. 4091

Metricbeat

• Add experimental metricset perfmon to Windows module. 3758

• Add memcached module with stats metricset. 3693

• Add the process.cmdline.cache.enabled config option to the System Process Metricset. 3891

• Add new MetricSet interfaces for developers (Closer, ReportingFetcher, and PushMetricSet). 3908

• Add kubelet module 3916

• Add dropwizard module 4022

• Adding query APIs for metricsets and modules from metricbeat registry 4102

• Fixing nil pointer on prometheus collector when http response is nil 4119

• Add http module with json metricset. 4092

• Add the option to the system module to include only the first top N processes by CPU and memory. 4127.

• Add experimental Vsphere module. 4028

• Add experimental Elasticsearch module. 3903

• Add experimental Kibana module. 3895

• Move elasticsearch metricset node_stats under node.stats namespace. 4142

• Make IP port indexer constructor public 4434

Packetbeat

• Add fields and fields_under_root to Packetbeat protocols configurations. 3518

• Add list style Packetbeat protocols configurations. This change supports specifying multiple configurations of the same protocol analyzer. 3518

Winlogbeat

Deprecated

Affecting all Beats

• Usage of field _type is deprecated. It should not be used in queries or dashboards. 3409

Packetbeat

• Deprecate dictionary style protocols configuration. 3518

Winlogbeat

Known Issue

Filebeat

• Prospector reloading only works properly with new files. 3546

Beats version 5.6.14

View commits

No changes in this version.

Beats version 5.6.13

View commits

No changes in this version.

Beats version 5.6.12

View commits

No changes in this version.

Beats version 5.6.11

View commits

No changes in this version.

Beats version 5.6.10

View commits

Bugfixes

Packetbeat

• Fix an out of bounds access in HTTP parser caused by malformed request. 6997

Beats version 5.6.9

View commits

Bugfixes

Affecting all Beats

• Fix a type issue when specifying certicate authority when using the import_dashboards command. 6678

Packetbeat

• Fix http status phrase parsing not allow spaces. 5312

• Fix http parse to allow to parse get request with space in the URI. 5495

• Fix mysql SQL parser to trim \r from Windows Server SELECT\r\n\t1. 5572

• Fix corruption when parsing repeated headers in an HTTP request or response. 6325

• Fix panic when parsing partial AMQP messages. 6384

• Fix out of bounds access to slice in MongoDB parser. 6256

• Fix sniffer hanging on exit under Linux. 6535

• Fix bounds check error in http parser causing a panic. 6750

• HTTP parses successfully on empty status phrase. 6176

• HTTP parser supports broken status line. 6631

Beats version 5.6.8

View commits

Bugfixes

Winlogbeat

• Fixed a crash under Windows 2003 and XP when an event had less insert strings than required by its format string. 6247

Beats version 5.6.7

View commits

No changes in this release.

Beats version 5.6.6

View commits

No changes in this release.

Beats version 5.6.5

View commits

Bugfixes

Affecting all Beats

• Fix duplicate batches of events in retry queue. 5520

Metricbeat

• Clarify meaning of percentages reported by system core metricset. 5565

• Fix map overwrite in docker diskio module. 5582

Beats version 5.6.4

View commits

Bugfixes

Affecting all Beats

• Fix race condition in internal logging rotator. 4519

Packetbeat

• Fix missing length check in the PostgreSQL module. 5457

Added

Affecting all Beats

• Add support for enabling TLS renegotiation. 4386

• Add setting to enable/disable the slow start in logstash output. 5400

Beats version 5.6.3

View commits

No changes in this release.

Beats version 5.6.2

View commits

No changes in this release.

Beats version 5.6.1

View commits

No changes in this release.

Beats version 5.6.0

View commits

Breaking changes

Affecting all Beats

• The _all.norms setting in the Elasticsearch template is no longer disabled. This increases the storage size with one byte per document, but allows for a better upgrade experience to 6.0. 4901

Bugfixes

Filebeat

• Fix issue where the fileset.module could have the wrong value. 4761

Packetbeat

• Update flow timestamp on each packet being received. 4895

Metricbeat

• Fix a debug statement that said a module wrapper had stopped when it hadn’t. 4264

• Use MemAvailable value from /proc/meminfo on Linux 3.14. 4316

• Fix panic when events were dropped by filters. 4327

Added

Affecting all Beats

• Add option to the import_dashboards script to load the dashboards via Kibana API. 4682

Filebeat

• Add support for loading Xpack Machine Learning configurations from the modules, and added sample configurations for the Nginx module. 4506 4609

• Add ability to parse nginx logs exposing the X-Forwarded-For header instead of the remote address. 4351

Metricbeat

• Add filesystem.ignore_types to system module for ignoring filesystem types. 4685

Deprecated

Affecting all Beats

• Loading more than one output is deprecated and will be removed in 6.0. 4907

Beats version 5.5.3

View commits

No changes in this release.

Beats version 5.5.2

View commits

No changes in this release.

Beats version 5.5.1

View commits

Bugfixes

Affecting all Beats

• Normalize all times to UTC to ensure proper index naming. 4569

Beats version 5.5.0

View commits

Breaking changes

Affecting all Beats

• Usage of field _type is now ignored and hardcoded to doc. 3757

Metricbeat - Change all system.cpu.*.pct metrics to be scaled by the number of CPU cores. This will make the CPU usage percentages from the system cpu metricset consistent with the system process metricset. The documentation for these metrics already stated that on multi-core systems the percentages could be greater than 100%. 4544

Bugfixes

Affecting all Beats

• Fix console output. 4045

Filebeat

• Allow string characters in user agent patch version (NGINX and Apache) 4415

Metricbeat

• Fix type of field haproxy.stat.check.health.last. 4407

Packetbeat

• Fix packetbeat.interface options that contain underscores (e.g. with_vlans or bpf_filter). 4378

• Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing. 4442

Deprecated

Filebeat

• Deprecate document_type prospector config option as _type is removed in elasticsearch 6.0. Use fields instead. 4225

Winlogbeat

• Deprecated metrics endpoint. It is superseded by a libbeat feature that can serve metrics on an HTTP endpoint. 4145

Beats version 5.4.2

View commits

Bugfixes

Affecting all Beats

• Removed empty sections from the template files, causing indexing errors for array objects. 4488

Metricbeat

• Fix issue affecting Windows services timing out at startup. 4491

• Add filtering to system filesystem metricset to remove relative mountpoints like those from Linux network namespaces. 4370

Packetbeat

• Clean configured geoip.paths before attempting to open the database. 4306

Beats version 5.4.1

View commits

Bugfixes

Affecting all Beats

• Fix importing the dashboards when the limit for max open files is too low. 4244

• Fix console output. 4045

Filebeat

• Fix issue that new prospector was not reloaded on conflict. 4128

• Fix grok pattern in filebeat module system/auth without hostname. 4224

• Fix the Mysql slowlog parsing of IP addresses. 4183

Added

Affecting all Beats

• Binaries upgraded to Go 1.7.6 which contains security fixes. 4400

Winlogbeat

• Add the ability to use LevelRaw if Level isn’t populated in the event XML. 4257

Beats version 5.4.0

View commits

Bugfixes

Affecting all Beats

• Improve error message when downloading the dashboards fails. 3805

• Fix potential Elasticsearch output URL parsing error if protocol scheme is missing. 3671

• Downgrade Elasticsearch per batch item failure log to debug level. 3953

• Make @timestamp accessible from format strings. 3721

Filebeat

• Allow log lines without a program name in the Syslog fileset. 3944

• Don’t stop Filebeat when modules are used with the Logstash output. 3929

Metricbeat

• Fixing panic on the Prometheus collector when label has a comma. 3947

• Make system process metricset honor the cpu_ticks config option. 3590

Winlogbeat

• Fix null terminators include in raw XML string when include_xml is enabled. 3943

Added

Affecting all Beats

• Update index mappings to support future Elasticsearch 6.X. 3778

Filebeat

• Add auditd module for reading audit logs on Linux. 3750 3941

• Add fileset for the Linux authorization logs. 3669

Heartbeat

• Add default ports in HTTP monitor. 3924

Metricbeat

• Add beta Jolokia module. 3844

• Add dashboard for the MySQL module. 3716

• Module configuration reloading is now beta instead of experimental. 3841

• Marked http fields from the HAProxy module optional to improve compatibility with 1.5. 3788

• Add support for custom HTTP headers and TLS for the Metricbeat modules. 3945

Packetbeat

• Add DNS dashboard for an overview the DNS traffic. 3883

• Add DNS Tunneling dashboard to highlight domains with large numbers of subdomains or high data volume. 3884

Beats version 5.3.2

View commits

Bugfixes

Filebeat

• Properly shut down crawler in case one prospector is misconfigured. 4037

• Fix panic in JSON decoding code if the input line is "null". 4042

Beats version 5.3.1

View commits

Bugfixes

Affecting all Beats

• Fix panic when testing regex-AST to match against date patterns. 3889

• Fix panic due to race condition in kafka output. 4098

Filebeat

• Fix modules default file permissions. 3879

• Allow - in Apache access log byte count. 3863

Metricbeat

• Avoid errors when some Apache status fields are missing. 3074

Beats version 5.3.0

View commits

Breaking changes

Affecting all Beats

• Configuration files must be owned by the user running the Beat or by root, and they must not be writable by others. 3544 3689

• Change Beat generator. Use $GOPATH/src/github.com/elastic/beats/script/generate.py to generate a beat. 3452

Filebeat

• Always use absolute path for event and registry. This can lead to issues when relative paths were used before. 3328

Metricbeat

• Linux cgroup metrics are now enabled by default for the system process metricset. The configuration option for the feature was renamed from cgroups to process.cgroups.enabled. 3519

• Change field names couchbase.node.couch..actual_disk_size. to couchbase.node.couch..disk_size. 3545

Bugfixes

Affecting all Beats

• Add _id, _type, _index and _score fields in the generated index pattern. 3282

Filebeat - Always use absolute path for event and registry. 3328 - Raise an exception in case there is a syntax error in one of the configuration files available under filebeat.config_dir. 3573 - Fix empty registry file on machine crash. 3537

Metricbeat

• Add error handling to system process metricset for when Linux cgroups are missing from the kernel. 3692

• Add labels to the Docker healthcheck metricset output. 3707

Winlogbeat

• Fix handling of empty strings in event_data. 3705

Added

Affecting all Beats

• Files created by Beats (logs, registry, file output) will have 0600 permissions. 3387.

• RPM/deb packages will now install the config file with 0600 permissions. 3382

• Add the option to pass custom HTTP headers to the Elasticsearch output. 3400

• Unify regexp and contains conditionals, for both to support array of strings and convert numbers to strings if required. 3469

• Add the option to load the sample dashboards during the Beat startup phase. 3506

• Disabled date detection in Elasticsearch index templates. Date fields must be explicitly defined in index templates. 3528

• Using environment variables in the configuration file is now GA, instead of experimental. 3525

Filebeat

• Add Filebeat modules for system, apache2, mysql, and nginx. 3159

• Add the pipeline config option at the prospector level, for configuring the Ingest Node pipeline ID. 3433

• Update regular expressions used for matching file names or lines (multiline, include/exclude functionality) to new matchers improving performance of simple string matches. 3469

• The symlinks and harvester_limit settings are now GA, instead of experimental. 3525

• close_timeout is also applied when the output is blocking. 3511

• Improve handling of different path variants on Windows. 3781

• Add multiline.flush_pattern option, for specifying the 'end' of a multiline pattern 4019

Heartbeat

• Add tags, fields and fields_under_root in monitors configuration. 3623

Metricbeat

• Add experimental dbstats metricset to MongoDB module. 3228

• Use persistent, direct connections to the configured nodes for MongoDB module. 3228

• Add dynamic configuration reloading for modules. 3281

• Add docker health metricset 3357

• Add docker image metricset 3467

• System module uses new matchers for white-listing processes. 3469

• Add Beta CEPH module with health metricset. 3311

• Add Beta php_fpm module with pool metricset. 3415

• The Docker, Kafka, and Prometheus modules are now Beta, instead of experimental. 3525

• The HAProxy module is now GA, instead of experimental. 3525

• Add the ability to collect the environment variables from system processes. 3337

Deprecated

Affecting all Beats

• Usage of field _type is deprecated. It should not be used in queries or dashboards. 3409

Filebeat

• The experimental publish_async option is now deprecated and is planned to be removed in 6.0. 3525

Beats version 5.2.2

View commits

Metricbeat

• Fix bug docker module hanging when docker container killed. 3610

• Set timeout to period instead of 1s by default as documented. 3612

Beats version 5.2.1

View commits

Bugfixes

Metricbeat

• Fix go routine leak in docker module. 3492

Packetbeat

• Fix error in the NFS sample dashboard. 3548

Winlogbeat

• Fix error in the Winlogbeat sample dashboard. 3548

Beats version 5.2.0

View commits

Bugfixes

Affecting all Beats

• Fix overwriting explicit empty config sections. 2918

Filebeat

• Fix alignment issue were Filebeat compiled with Go 1.7.4 was crashing on 32 bits system. 3273

Metricbeat

• Fix service times-out at startup. 3056

• Kafka module case sensitive host name matching. 3193

• Fix interface conversion panic in couchbase module 3272

Packetbeat

• Fix issue where some Cassandra visualizations were showing data from all protocols. 3314

Added

Affecting all Beats

• Add support for passing list and dictionary settings via -E flag.

• Support for parsing list and dictionary setting from environment variables.

• Added new flags to import_dashboards (-cacert, -cert, -key, -insecure). 3139 3163

• The limit for the number of fields is increased via the mapping template. 3275

• Updated to Go 1.7.4. 3277

• Added a NOTICE file containing the notices and licenses of the dependencies. 3334.

Heartbeat

• First release, containing monitors for ICMP, TCP, and HTTP.

Filebeat

• Add enabled config option to prospectors. 3157

• Add target option for decoded_json_field. 3169

Metricbeat

• Kafka module broker matching enhancements. 3129

• Add a couchbase module with metricsets for node, cluster and bucket. 3081

• Export number of cores for CPU module. 3192

• Experimental Prometheus module. 3202

• Add system socket module that reports all TCP sockets. 3246

• Kafka consumer groups metricset. 3240

• Add jolokia module with dynamic jmx metricset. 3570

Winlogbeat

• Reduced amount of memory allocated while reading event log records. 3113 3118

Beats version 5.1.2

View commits

Bugfixes

Filebeat

• Fix registry migration issue from old states where files were only harvested after second restart. 3322

Packetbeat

• Fix error on importing dashboards due to colons in the Cassandra dashboard. 3140

• Fix error on importing dashboards due to the wrong type for the geo_point fields. 3147

Winlogbeat

• Fix for "The array bounds are invalid" error when reading large events. 3076

Beats version 5.1.1

View commits

Breaking changes

Metricbeat

• Change data structure of experimental haproxy module. 3003

Filebeat

• If a file is falling under ignore_older during startup, offset is now set to end of file instead of 0. With the previous logic the whole file was sent in case a line was added and it was inconsistent with files which were harvested previously. 2907

• tail_files is now only applied on the first scan and not for all new files. 2932

Bugfixes

Affecting all Beats

• Fix empty benign errors logged by processor actions. 3046

Metricbeat

• Calculate the fsstat values per mounting point, and not filesystem. 2777

Added

Affecting all Beats

• Add add_cloud_metadata processor for collecting cloud provider metadata. 2728

• Added decode_json_fields processor for decoding fields containing JSON strings. 2605

• Add Tencent Cloud provider for add_cloud_metadata processor. 4023

• Add Alibaba Cloud provider for add_cloud_metadata processor. 4111

Metricbeat

• Add experimental Docker module. Provided by Ingensi and @douaejeouit based on dockbeat.

• Add a sample Redis Kibana dashboard. 2916

• Add support for MongoDB 3.4 and WiredTiger metrics. 2999

• Add experimental kafka module with partition metricset. 2969

• Add raw config option for mysql/status metricset. 3001

• Add command fields for mysql/status metricset. 3251

Filebeat

• Add command line option -once to run Filebeat only once and then close. 2456

• Only load matching states into prospector to improve state handling 2840

• Reset all states ttl on startup to make sure it is overwritten by new config 2840

• Persist all states for files which fall under ignore_older to have consistent behaviour 2859

• Improve shutdown behaviour with large number of files. 3035

Winlogbeat

• Add event_logs.batch_read_size configuration option. 2641

Beats version 5.1.0 (skipped)

Version 5.1.0 doesn’t exist because, for a short period of time, the Elastic Yum and Apt repositories included unreleased binaries labeled 5.1.0. To avoid confusion and upgrade issues for the people that have installed these without realizing, we decided to skip the 5.1.0 version and release 5.1.1 instead.

Beats version 5.0.2

View commits

Bugfixes

Metricbeat

• Fix the password option in the MongoDB module. 2995

Beats version 5.0.1

View commits

Bugfixes

Metricbeat

• Fix system.process.start_time on Windows. 2848

• Fix system.process.ppid on Windows. 2860

• Fix system process metricset for Windows XP and 2003. cmdline will be unavailable. 1704

• Fix access denied issues in system process metricset by enabling SeDebugPrivilege on Windows. 1897

• Fix system diskio metricset for Windows XP and 2003. 2885

Packetbeat

• Fix 'index out of bounds' bug in Packetbeat DNS protocol plugin. 2872

Filebeat

• Fix registry cleanup issue when files falling under ignore_older after restart. 2818

Added

Metricbeat

• Add username and password config options to the PostgreSQL module. 2890

• Add username and password config options to the MongoDB module. 2889

• Add system core metricset for Windows. 2883

Packetbeat

• Define client_geoip.location as geo_point in the mappings to be used by the GeoIP processor in the Ingest Node pipeline. 2795

Filebeat

• Stop Filebeat on registrar loading error. 2868

libbeat/docs/release-notes/5.0.0.asciidoc

Beats version 5.0.0-GA

View commits

The list below covers the changes between 5.0.0-rc1 and 5.0.0 GA only.

Bugfixes

Affecting all Beats

• Fix kafka output re-trying batches with too large events. 2735

• Fix kafka output protocol error if version: 0.10 is configured. 2651

• Fix kafka output connection closed by broker on SASL/PLAIN. 2717

Metricbeat

• Fix high CPU usage on macOS when encountering processes with long command lines. 2747

• Fix high value of system.memory.actual.free and system.memory.actual.used. 2653

• Change several OpenProcess calls on Windows to request the lowest possible access privilege. 1897

• Fix system.memory.actual.free high value on Windows. 2653

Filebeat

• Fix issue when clean_removed and clean_inactive were used together that states were not directly removed from the registry.

• Fix issue where upgrading a 1.x registry file resulted in duplicate state entries. 2792

Added

Affecting all Beats

• Add beat.version fields to all events.

Beats version 5.0.0-rc1

View commits

Breaking changes

Affecting all Beats

• A dynamic mapping rule is added to the default Elasticsearch template to treat strings as keywords by default. 2688

Bugfixes

Affecting all Beats

• Make sure Beats sent always float values when they are defined as float by sending 5.00000 instead of 5. 2627

• Fix ignoring all fields from drop_fields in case the first field is unknown. 2685

• Fix dynamic configuration int/uint to float type conversion. 2698

• Fix primitive types conversion if values are read from environment variables. 2698

Metricbeat

• Fix default configuration file on Windows to not enabled the load metricset. 2632

Packetbeat

• Fix the bpf_filter setting. 2660

Filebeat

• Fix input buffer on encoding problem. 2416

Deprecated

Affecting all Beats

• Setting port has been deprecated in Redis and Logstash outputs. 2620

Beats version 5.0.0-beta1

View commits

Breaking changes

Affecting all Beats

• Change Elasticsearch output index configuration to be based on format strings. If index has been configured, no date will be appended anymore to the index name. 2119

• Replace output.kafka.use_type by output.kafka.topic accepting a format string. 2188

• If the path specified by the -c flag is not absolute and -path.config is not specified, it is considered relative to the current working directory. 2245

• rename tls configurations section to ssl. 2330

• rename certificate_key configuration to key. 2330

• replace tls.insecure with ssl.verification_mode setting. 2330

• replace tls.min/max_version with ssl.supported_protocols setting requiring full protocol name. 2330

Metricbeat

• Change field type system.process.cpu.start_time from keyword to date. 1565

• redis/info metricset fields were renamed up according to the naming conventions.

Packetbeat

• Group HTTP fields under http.request and http.response 2167

• Export http.request.body and http.response.body when configured under include_body_for 2167

• Move ignore_outgoing config to packetbeat.ignore_outgoing 2393

Filebeat

• Set close_inactive default to 5 minutes (was 1 hour before)

• Set clean_removed and close_removed to true by default

Bugfixes

Affecting all Beats

• Fix logstash output handles error twice when asynchronous sending fails. 2441

• Fix Elasticsearch structured error response parsing error. 2229

• Fixed the run script to allow the overriding of the configuration file. 2171

• Fix logstash output crash if no hosts are configured. 2325

• Fix array value support in -E CLI flag. 2521

• Fix merging array values if -c CLI flag is used multiple times. 2521

• Fix beats failing to start due to invalid duplicate key error in configuration file. 2521

• Fix panic on non writable logging directory. 2571

Metricbeat

• Fix module filters to work properly with drop_event filter. 2249

Packetbeat

• Fix mapping for some Packetbeat flow metrics that were not marked as being longs. 2177

• Fix handling of messages larger than the maximum message size (10MB). 2470

Filebeat

• Fix processor failure in Filebeat when using regex, contain, or equals with the message field. 2178

• Fix async publisher sending empty events 2455

• Fix potential issue with multiple harvester per file on large file numbers or slow output 2541

Winlogbeat

• Fix corrupt registry file that occurs on power loss by disabling file write caching. 2313

Added

Affecting all Beats

• Add script to generate the Kibana index-pattern from fields.yml. 2122

• Enhance Redis output key selection based on format string. 2169

• Configurable Redis keys using filters and format strings. 2169

• Add format string support to output.kafka.topic. 2188

• Add output.kafka.topics for more advanced kafka topic selection per event. 2188

• Add support for Kafka 0.10. 2190

• Add SASL/PLAIN authentication support to kafka output. 2190

• Make Kafka metadata update configurable. 2190

• Add Kafka version setting (optional) enabling kafka broker version support. 2190

• Add Kafka message timestamp if at least version 0.10 is configured. 2190

• Add configurable Kafka event key setting. 2284

• Add settings for configuring the kafka partitioning strategy. 2284

• Add partitioner settings reachable_only to ignore partitions not reachable by network. 2284

• Enhance contains condition to work on fields that are arrays of strings. 2237

• Lookup the configuration file relative to the -path.config CLI flag. 2245

• Re-write import_dashboards.sh in Golang. 2155

• Update to Go 1.7. 2306

• Log total non-zero internal metrics on shutdown. 2349

• Add support for encrypted private key files by introducing ssl.key_passphrase setting. 2330

• Add experimental symlink support with symlinks config 2478

• Improve validation of registry file on startup.

Metricbeat

• Use the new scaled_float Elasticsearch type for the percentage values. 2156

• Add experimental cgroup metrics to the system/process MetricSet. 2184

• Added a PostgreSQL module. 2253

• Improve mapping by converting half_float to scaled_float and integers to long. 2430

• Add experimental haproxy module. 2384

• Add Kibana dashboard for cgroups data 2555

Packetbeat

• Add Cassandra protocol analyzer to Packetbeat. 1959

• Match connections with IPv6 addresses to processes 2254

• Add IP address to -devices command output 2327

• Add configuration option for the maximum message size. Used to be hard-coded to 10 MB. 2470

Filebeat

• Introduce close_timeout harvester options 1926

• Strip BOM from first message in case of BOM files 2351

• Add harvester_limit option 2417

Deprecated

Affecting all Beats

• Topology map is deprecated. This applies to the settings: refresh_topology_freq, topology_expire, save_topology, host_topology, password_topology, db_topology.

Beats version 5.0.0-alpha5

View commits

Breaking changes

Affecting all Beats

• Rename the filters section to processors. 1944

• Introduce the condition with when in the processor configuration. 1949

• The Elasticsearch template is now loaded by default. 1993

• The Redis output index setting is renamed to key. index still works but it’s deprecated. 2077

• The undocumented file output index setting was removed. Use filename instead. 2077

Metricbeat

• Create a separate metricSet for load under the system module and remove load information from CPU stats. 2101

• Add system.load.norm.1, system.load.norm.5 and system.load.norm.15. 2101

• Add threads fields to mysql module. 2484

Packetbeat

• Set enabled ` in packetbeat.protocols.icmp configuration to true by default. 1988

Bugfixes

Affecting all Beats

• Fix sync publisher PublishEvents return value if client is closed concurrently. 2046

Metricbeat

• Do not send zero values when no value was present in the source. 1972

Filebeat

• Fix potential data loss between Filebeat restarts, reporting unpublished lines as published. 2041

• Fix open file handler issue. 2028 2020

• Fix filtering of JSON events when using integers in conditions. 2038

Winlogbeat

• Fix potential data loss between Winlogbeat restarts, reporting unpublished lines as published. 2041

Added

Affecting all Beats

• Periodically log internal metrics. 1955

• Add enabled setting to all output modules. 1987

• Command line flag -c can be used multiple times. 1985

• Add OR/AND/NOT to the condition associated with the processors. 1983

• Add -E CLI flag for overwriting single config options via command line. 1986

• Choose the mapping template file based on the Elasticsearch version. 1993

• Check stdout being available when console output is configured. 2035

Metricbeat

• Add pgid field to process information. https://github.com/elastic/beats/pull/ 2021[2021]

Packetbeat

• Add enabled setting to Packetbeat protocols. 1988

• Add enabled setting to Packetbeat network flows configuration. 1988

Filebeat

• Introduce close_removed and close_renamed harvester options. 1600

• Introduce close_eof harvester option. 1600

• Add clean_removed and clean_inactive config option. 1600

Deprecated

Filebeat

• Deprecate close_older option and replace it with close_inactive. 2051

• Deprecate force_close_files option and replace it with close_removed and close_renamed. 1600

Beats version 5.0.0-alpha4

View commits

Breaking changes

Affecting all Beats

• The topology_expire option of the Elasticsearch output was removed. 1907

Filebeat

• Stop following symlink. Symlinks are now ignored: 1686

Bugfixes

Affecting all Beats

• Reset backoff factor on partial ACK. 1803

• Fix beats load balancer deadlock if max_retries: -1 or publish_async is enabled in filebeat. 1829

• Fix logstash output with pipelining mode enabled not reconnecting. 1876

• Empty configuration sections become merge-able with variables containing full path. 1900

• Fix error message about required fields missing not printing the missing field name. 1900

Metricbeat

• Fix the CPU values returned for each core. 1863

Packetbeat

• Add missing nil-check to memcached GapInStream handler. 1162

• Fix NFSv4 Operation returning the first found first-class operation available in compound requests. 1821

• Fix TCP overlapping segments not being handled correctly. 1898

Winlogbeat

• Fix issue with rendering forwarded event log records. 1891

Added

Affecting all Beats

• Improve error message if compiling regular expression from config files fails. 1900

• Compression support in the Elasticsearch output. 1835

Metricbeat

• Add MongoDB module. 1837

Beats version 5.0.0-alpha3

View commits

Breaking changes

Affecting all Beats

• All configuration settings under shipper: are moved to be top level configuration settings. I.e. shipper.name: becomes name: in the configuration file. 1570

Topbeat

• Topbeat is replaced by Metricbeat.

Filebeat

• The state for files which fall under ignore_older is not stored anymore. This has the consequence, that if a file which fell under ignore_older is updated, the whole file will be crawled.

Bugfixes

Winlogbeat

• Adding missing argument to the "Stop processing" log message. 1590

Added

Affecting all Beats

• Add conditions to generic filtering. 1623

Metricbeat

• First public release, containing the following modules: apache, mysql, nginx, redis, system, and zookeeper.

Filebeat

• The registry format was changed to an array instead of dict. The migration to the new format will happen automatically at the first startup. 1703

Deprecated

Affecting all Beats

• The support for doing GeoIP lookups is deprecated and will be removed in version 6.0. 1601

Beats version 5.0.0-alpha2

View commits

Breaking changes

Affecting all Beats

• On DEB/RPM installations, the binary files are now found under /usr/share/{{beat_name}}/bin, not in /usr/bin. 1385

• The logs are written by default to self rotating files, instead of syslog. 1371

• Remove deprecated host option from elasticsearch, logstash and redis outputs. 1474

Packetbeat

• Configuration of redis topology support changed. 1353

• Move all Packetbeat configuration options under the packetbeat namespace 1417

Filebeat

• Default location for the registry file was changed to be data/registry from the binary directory, rather than .filebeat in the current working directory. This affects installations for zip/tar.gz/source, the location for DEB and RPM packages stays the same. 1373

Bugfixes

Affecting all Beats

• Drain response buffers when pipelining is used by Redis output. 1353

• Unterminated environment variable expressions in config files will now cause an error 1389

• Fix issue with the automatic template loading when Elasticsearch is not available on Beat start. 1321

• Fix bug affecting -cpuprofile, -memprofile, and -httpprof CLI flags 1415

• Fix race when multiple outputs access the same event with logstash output manipulating event 1410 1428

• Seed random number generator using crypto.rand package. https://github.com/elastic/beats/pull/1503{1503]

• Fix beats hanging in -configtest 1213

• Fix kafka log message output 1516

Filebeat

• Improvements in registrar dealing with file rotation. 1281

• Fix issue with JSON decoding where @timestamp or type keys with the wrong type could cause Filebeat to crash. 1378

• Fix issue with JSON decoding where values having null as values could crash Filebeat. 1466

• Multiline reader normalizing newline to use \n. 1552

Winlogbeat

• Fix panic when reading messages larger than 32K characters on Windows XP and 2003. 1498

• Fix panic that occurs when reading a large events on Windows Vista and newer. 1499

Added

Affecting all Beats

• Add support for TLS to Redis output. 1353

• Add SOCKS5 proxy support to Redis output. 1353

• Failover and load balancing support in redis output. 1353

• Multiple-worker per host support for redis output. 1353

• Added ability to escape ${x} in config files to avoid environment variable expansion 1389

• Configuration options and CLI flags for setting the home, data and config paths. 1373

• Configuration options and CLI flags for setting the default logs path. 1437

• Update to Go 1.6.2 1447

• Add Elasticsearch template files compatible with Elasticsearch 2.x. 1501

• Add scripts for managing the dashboards of a single Beat 1359

Packetbeat

• Fix compile issues for OpenBSD. 1347

Topbeat

• Updated elastic/gosigar version so Topbeat can compile on OpenBSD. 1403

Beats version 5.0.0-alpha1

View commits

Breaking changes

libbeat

• Run function to start a Beat now returns an error instead of directly exiting. 771

• The method signature of HandleFlags() was changed to allow returning an error 1249

• Require braces for environment variable expansion in config files 1304

Packetbeat

• Rename output fields in the dns package. Former flag recursion_allowed becomes recursion_available. 803 Former SOA field ttl becomes minimum. 803

• The fully qualified domain names which are part of output fields values of the dns package now terminate with a dot. 803

• Remove the count field from the exported event 1210

Topbeat

• Rename proc.cpu.user_p with proc.cpu.total_p as it includes CPU time spent in kernel space 631

• Remove count field from the exported fields 1207

• Rename input top level config option to topbeat

Filebeat

• Scalar values in used in the fields configuration setting are no longer automatically converted to strings. 1092

• Count field was removed from event as not used in filebeat 778

Winlogbeat

• The message_inserts field was replaced with the event_data field 1053

• The category field was renamed to task to better align with the Windows Event Log API naming 1053

• Remove the count field from the exported event 1218

Bugfixes

Affecting all Beats

• Logstash output will not retry events that are not JSON-encodable 927

Packetbeat

• Create a proper BPF filter when ICMP is the only enabled protocol 757

• Check column length in pgsql parser. 565

• Harden pgsql parser. 565

Topbeat

• Fix issue with cpu.system_p being greater than 1 on Windows 1128

Filebeat

• Stop filebeat if started without any prospectors defined or empty prospectors 644 647

• Improve shutdown of crawler and prospector to wait for clean completion 720

• Omit fields from Filebeat events when null 899

Winlogbeat

Added

Affecting all Beats

• Update builds to Golang version 1.6

• Add option to Elasticsearch output to pass http parameters in index operations 805

• Improve Logstash and Elasticsearch backoff behavior. 927

• Add experimental Kafka output. 942

• Add config file option to configure GOMAXPROCS. 969

• Improve shutdown handling in libbeat. 1075

• Add fields and fields_under_root options under the shipper configuration 1092

• Add the ability to use a SOCKS5 proxy with the Logstash output 823

• The -configtest flag will now print "Config OK" to stdout on success 1249

Packetbeat

• Change the DNS library used throughout the dns package to github.com/miekg/dns. 803

• Add support for NFS v3 and v4. 1231

• Add support for EDNS and DNSSEC. 1292

Topbeat

• Add username to processes 845

Filebeat

• Add the ability to set a list of tags for each prospector 1092

• Add JSON decoding support 1143

Winlogbeat

• Add caching of event metadata handles and the system render context for the wineventlog API 888

• Improve config validation by checking for unknown top-level YAML keys. 1100

• Add the ability to set tags, fields, and fields_under_root as options for each event log 1092

• Add additional data to the events published by Winlogbeat. The new fields are activity_id, event_data, keywords, opcode, process_id, provider_guid, related_activity_id, task, thread_id, user_data, and version. 1053

• Add event_id, level, and provider configuration options for filtering events 1218

• Add include_xml configuration option for including the raw XML with the event 1218

Known issues

• All Beats can hang or panic on shutdown if the next server in the pipeline (e.g. Elasticsearch or Logstash) is not reachable. 1319

• When running the Beats as a service on Windows, you need to manually load the Elasticsearch mapping template. 1315

• The ES template automatic load doesn’t work if Elasticsearch is not available when the Beat is starting. 1321

Beats version 1.3.1

View commits

Bugfixes

Filebeat

• Fix a concurrent bug on filebeat startup with a large number of prospectors defined. 2509

Packetbeat

• Fix description for the -I CLI flag. 2480

Winlogbeat

• Fix corrupt registry file that occurs on power loss by disabling file write caching. 2313

Beats version 1.3.0

View commits

Deprecated

Filebeat

• Undocumented support for following symlinks is deprecated. Filebeat will not follow symlinks in version 5.0. 1767

Bugfixes

Affecting all Beats

• Fix beats load balancer deadlock if max_retries: -1 or publish_async is enabled in filebeat. 1829

• Fix output modes backoff counter reset. 1803 1814 1818

• Set logstash output default bulk_max_size to 2048. 1662

• Seed random number generator using crypto.rand package. 1503

• Check stdout being available when console output is configured. 2063

Packetbeat

• Add missing nil-check to memcached GapInStream handler. 1162

• Fix NFSv4 Operation returning the first found first-class operation available in compound requests. 1821

• Fix TCP overlapping segments not being handled correctly. 1917

Added

Affecting all Beats

• Updated to Go 1.7

Beats version 1.2.3

View commits

Bugfixes

Topbeat

• Fix high CPU usage when using filtering under Windows. 1598

Filebeat

• Fix rotation issue with ignore_older. 1528

Winlogbeat

• Fix panic when reading messages larger than 32K characters on Windows XP and 2003. 1498

Added

Filebeat

• Prevent file opening for files which reached ignore_older. 1649

Beats version 1.2.2

View commits

Bugfixes

Affecting all Beats

• Fix race when multiple outputs access the same event with Logstash output manipulating event. 1410

• Fix go-daemon (supervisor used in init scripts) hanging when executed over SSH. 1394

Filebeat

• Improvements in registrar dealing with file rotation. 1281

Beats version 1.2.1

View commits

Breaking changes

Affecting all Beats

• Require braces for environment variable expansion in config files 1304

• Removed deprecation warning for the Redis output. 1282

Topbeat

• Fixed name of the setting stats.proc to stats.process in the default configuration file. 1343

• Fix issue with cpu.system_p being greater than 1 on Windows 1128

Added

Topbeat

• Add username to processes 845

Beats version 1.2.0

View commits

Breaking changes

Filebeat

• Default config for ignore_older is now infinite instead of 24h, means ignore_older is disabled by default. Use close_older to only close file handlers.

Bugfixes

Packetbeat

• Split real_ip_header value when it contains multiple IPs 1241

Winlogbeat

• Fix invalid event_id on Windows XP and Windows 2003 1227

Added

Affecting all Beats

• Add ability to override configuration settings using environment variables 114

• Libbeat now always exits through a single exit method for proper cleanup and control 736

• Add ability to create Elasticsearch mapping on startup 639

Topbeat

• Add the command line used to start processes 533

Filebeat

• Add close_older configuration option to complete ignore_older 181

Beats version 1.1.2

View commits

Bugfixes

Filebeat

• Fix registrar bug for rotated files 1010

Beats version 1.1.1

View commits

Bugfixes

Affecting all Beats

• Fix logstash output loop hanging in infinite loop on too many output errors. 944

• Fix critical bug in filebeat and winlogbeat potentially dropping events. 953

Beats version 1.1.0

View commits

Bugfixes

Affecting all Beats

• Fix logging issue with file based output where newlines could be misplaced during concurrent logging 650

• Reduce memory usage by separate queue sizes for single events and bulk events. 649 516

• Set default default bulk_max_size value to 2048 628

Packetbeat

• Fix setting direction to out and use its value to decide when dropping events if ignore_outgoing is enabled 557

• Fix logging issue with file-based output where newlines could be misplaced during concurrent logging 650

• Reduce memory usage by having separate queue sizes for single events and bulk events. 649 516

• Set default bulk_max_size value to 2048 628

• Fix logstash window size of 1 not increasing. 598

Packetbeat

• Fix the condition that determines whether the direction of the transaction is set to "outgoing". Packetbeat uses the direction field to determine which transactions to drop when dropping outgoing transactions. 557

• Allow PF_RING sniffer type to be configured using pf_ring or pfring 671

Filebeat

• Set spool_size default value to 2048 628

Added

Affecting all Beats

• Add include_fields and drop_fields as part of generic filtering 1120

• Make logstash output compression level configurable. 630

• Some publisher options refactoring in libbeat 684

• Move event preprocessor applying GeoIP to packetbeat 772

Packetbeat

• Add support for capturing DNS over TCP network traffic. 486 554

Topbeat

• Group all CPU usage per core statistics and export them optionally if cpu_per_core is configured 496

Filebeat

• Add multiline support for combining multiple related lines into one event. 461

• Add exclude_lines and include_lines options for regexp based line filtering. 430

• Add exclude_files configuration option. 563

• Add experimental option to enable filebeat publisher pipeline to operate asynchronously 782

Winlogbeat

• First public release of Winlogbeat

Beats version 1.0.1

Check 1.0.1 diff

Bugfixes

Filebeat

• Fix force_close_files in case renamed file appeared very fast. 302

Packetbeat

• Improve MongoDB message correlation. 377

• Improve redis parser performance. 422

• Fix panic on nil in redis protocol parser. 384

• Fix errors redis parser when messages are split in multiple TCP segments. 402

• Fix errors in redis parser when length prefixed strings contain sequences of CRLF. 402

• Fix errors in redis parser when dealing with nested arrays. 402

Beats version 1.0.0

Check 1.0.0 diff

Breaking changes

Topbeat

• Change proc type to process #138

Bugfixes

Affecting all Beats

• Fix random panic on shutdown by calling shutdown handler only once. elastic/filebeat#204

• Fix credentials are not send when pinging an elasticsearch host. elastic/filebeat#287

Filebeat

• Fix problem that harvesters stopped reading after some time and filebeat stopped processing events #257

• Fix line truncating by internal buffers being reused by accident #258

• Set default ignore_older to 24 hours #282

Beats version 1.0.0-rc2

Check 1.0.0-rc2 diff

Breaking changes

Affecting all Beats

• The shipper output field is renamed to beat.name. #285

• Use of enabled as a configuration option for outputs (elasticsearch, logstash, etc.) has been removed. #264

• Use of disabled as a configuration option for tls has been removed. #264

• The -test command line flag was renamed to -configtest. #264

• Disable geoip by default. To enable it uncomment in config file. #305

Filebeat

• Removed utf-16be-bom encoding support. Support will be added with fix for #205

• Rename force_close_windows_files to force_close_files and make it available for all platforms.

Bugfixes

Affecting all Beats

• Disable logging to stderr after configuration phase. #276

• Set the default file logging path when not set in config. #275

• Fix bug silently dropping records based on current window size. elastic/filebeat#226

• Fix direction field in published events. #300

• Fix elasticsearch structured errors breaking error handling. #309

Packetbeat

• Packetbeat will now exit if a configuration error is detected. #357

• Fixed an issue handling DNS requests containing no questions. #369

Topbeat

• Fix leak of Windows handles. #98

• Fix memory leak of process information. #104

Filebeat

• Filebeat will now exit if a configuration error is detected. #198

• Fix to enable prospector to harvest existing files that are modified. #199

• Improve line reading and encoding to better keep track of file offsets based on encoding. #224

• Set input_type by default to "log"

Added

Affecting all Beats

• Added beat.hostname to contain the hostname where the Beat is running on as returned by the operating system. #285

• Added timestamp for file logging. #291

Filebeat

• Handling end of line under windows was improved #233

Beats version 1.0.0-rc1

Check 1.0.0-rc1 diff

Breaking changes

Affecting all Beats

• Rename timestamp field with @timestamp. #237

Packetbeat

• Rename timestamp field with @timestamp. #343

Topbeat

• Rename timestamp field with @timestamp for a better integration with Logstash. #80

Filebeat

• Rename the timestamp field with @timestamp #168

• Rename tail_on_rotate prospector config to tail_files

• Removal of line field in event. Line number was not correct and does not add value. #217

Bugfixes

Affecting all Beats

• Use stderr for console log output. #219

• Handle empty event array in publisher. #207

• Respect '*' debug selector in IsDebug. #226 (elastic#339)

• Limit number of workers for Elasticsearch output. elastic#226

• On Windows, remove service related error message when running in the console. #242

• Fix waitRetry no configured in single output mode configuration. elastic/filebeat#144

• Use http as the default scheme in the elasticsearch hosts #253

• Respect max bulk size if bulk publisher (collector) is disabled or sync flag is set.

• Always evaluate status code from Elasticsearch responses when indexing events. #192

• Use bulk_max_size configuration option instead of bulk_size. #256

• Fix max_retries=0 (no retries) configuration option. #266

• Filename used for file based logging now defaults to beat name. #267

Packetbeat

• Close file descriptors used to monitor processes. #337

• Remove old RPM spec file. It moved to elastic/beats-packer. #334

Topbeat

• Don’t wait for one period until shutdown #75

Filebeat

• Omit 'fields' from event JSON when null. #126

• Make offset and line value of type long in elasticsearch template to prevent overflow. #140

• Fix locking files for writing behaviour. #156

• Introduce 'document_type' config option per prospector to define document type for event stored in elasticsearch. #133

• Add 'input_type' field to published events reporting the prospector type being used. #133

• Fix high CPU usage when not connected to Elasticsearch or Logstash. #144

• Fix issue that files were not crawled anymore when encoding was set to something other then plain. #182

Added

Affecting all Beats

• Add Console output plugin. #218

• Add timestamp to log messages #245

• Send @metadata.beat to Logstash instead of @metadata.index to prevent possible name clashes and give user full control over index name used for Elasticsearch

• Add logging messages for bulk publishing in case of error #229

• Add option to configure number of parallel workers publishing to Elasticsearch or Logstash.

• Set default bulk size for Elasticsearch output to 50.

• Set default http timeout for Elasticsearch to 90s.

• Improve publish retry if sync flag is set by retrying only up to max bulk size events instead of all events to be published.

Filebeat

• Introduction of backoff, backoff_factor, max_backoff, partial_line_waiting, force_close_windows_files config variables to make crawling more configurable.

• All Godeps dependencies were updated to master on 2015-10-21 [#122]

• Set default value for ignore_older config to 10 minutes. #164

• Added the fields_under_root setting to optionally store the custom fields top level in the output dictionary. #188

• Add more encodings by using x/text/encodings/htmlindex package to select encoding by name.

Beats version 1.0.0-beta4

Check 1.0.0-beta4 diff

Breaking changes

Affecting all Beats

• Update tls config options naming from dash to underline #162

• Feature/output modes: Introduction of PublishEvent(s) to be used by beats #118 #115

Packetbeat

• Renamed http module config file option 'strip_authorization' to 'redact_authorization'

• Save_topology is set to false by default

• Rename elasticsearch index to [packetbeat-]YYYY.MM.DD

Topbeat

• Percentage fields (e.g user_p) are exported as a float between 0 and 1 #34

Bugfixes

Affecting all Beats

• Determine Elasticsearch index for an event based on UTC time #81

• Fixing ES output’s defaultDeadTimeout so that it is 60 seconds #103

• ES outputer: fix timestamp conversion #91

• Fix TLS insecure config option #239

• ES outputer: check bulk API per item status code for retransmit on failure.

Packetbeat

• Support for lower-case header names when redacting http authorization headers

• Redact proxy-authorization if redact-authorization is set

• Fix some multithreading issues #203

• Fix negative response time #216

• Fix memcache TCP connection being nil after dropping stream data. #299

• Add missing DNS protocol configuration to documentation #269

Topbeat

• Don’t divide the reported memory by an extra 1024 #60

Added

Affecting all Beats

• Add logstash output plugin #151

• Integration tests for Beat → Logstash → Elasticsearch added #195 #188 #168 #137 #128 #112

• Large updates and improvements to the documentation

• Add direction field to publisher output to indicate inbound/outbound transactions #150

• Add tls configuration support to elasticsearch and logstash outputers #139

• All external dependencies were updated to the latest version. Update to Golang 1.5.1 #162

• Guarantee ES index is based in UTC time zone #164

• Cache: optional per element timeout #144

• Make it possible to set hosts in different ways. #135

• Expose more TLS config options #124

• Use the Beat name in the default configuration file path #99

Packetbeat

• add [.editorconfig file](http://editorconfig.org/)

• add (experimental/unsupported?) saltstack files

• Sample config file cleanup

• Moved common documentation to [libbeat repository](https://github.com/elastic/libbeat)

• Update build to go 1.5.1

• Adding device descriptions to the -device output.

• Generate coverage for system tests

• Move go-daemon dependency to beats-packer

• Rename integration tests to system tests

• Made the -devices option more user friendly in case sudo is not used. Issue #296.

• Publish expired DNS transactions #301

• Update protocol guide to libbeat changes

• Add protocol registration to new protocol guide

• Make transaction timeouts configurable #300

• Add direction field to the exported fields #317

Topbeat

• Document fields in a standardized format (etc/fields.yml) #34

• Updated to use new libbeat Publisher #37 #41

• Update to go 1.5.1 #43

• Updated configuration files with comments for all options #65

• Documentation improvements

Deprecated

Affecting all Beats

• Redis output was deprecated #169 #145

• Host and port configuration options are deprecated. They are replaced by the hosts configuration option. #141