Skip to content

Latest commit

 

History

History
628 lines (559 loc) · 46.3 KB

CHANGELOG.next.asciidoc

File metadata and controls

628 lines (559 loc) · 46.3 KB
Raw

Beats version HEAD

Breaking changes

Affecting all Beats

  • Update to Golang 1.12.1. 11330

  • Update to Golang 1.12.4. 11782

  • Update to ECS 1.0.1. 12284 12317

  • Default of output.kafka.metadata.full is set to false by now. This reduced the amount of metadata to be queried from a kafka cluster. 12738

  • Fixed a crash under Windows when fetching processes information. 12833

  • Update to Golang 1.12.7. 12931

  • Remove in_cluster configuration parameter for Kuberentes, now in-cluster configuration is used only if no other kubeconfig is specified 13051

  • Disable Alibaba Cloud and Tencent Cloud metadata providers by default. 12812

  • Libbeat HTTP’s Server can listen to a unix socket using the unix:///tmp/hello.sock syntax. 13655

  • Libbeat HTTP’s Server can listen to a Windows named pipe using the npipe:///hello syntax. 13655

  • By default, all Beats-created files and folders will have a umask of 0027 (on POSIX systems). 14119

  • Adding new Enterprise license type to the licenser. 14246

  • Change wording when we fail to load a CA file to the cert pool. 14309

  • Allow Metricbeat’s beat module to read monitoring information over a named pipe or unix domain socket. 14558

  • Remove version information from default ILM policy for improved upgrade experience on custom policies. 14745

  • Running setup cmd respects setup.ilm.overwrite setting for improved support of custom policies. 14741

  • Libbeat: Do not overwrite agent.*, ecs.version, and host.name. 14407

  • Libbeat: Cleanup the x-pack licenser code to use the new license endpoint and the new format. 15091

Auditbeat

  • Auditd module: Normalized value of event.category field from user-login to authentication. 11432

  • Auditd module: Unset auditd.session and user.audit.id fields are removed from audit events. 11431 11815

  • Socket dataset: Exclude localhost by default 11993

  • Socket dataset: New implementation using Kprobes for finer-grained monitoring and UDP support. 13058

Filebeat

  • Add Filebeat Azure Dashboards 14127

  • Add read_buffer configuration option. 11739

  • convert_timezone option is removed and locale is always added to the event so timezone is used when parsing the timestamp, this behaviour can be overriden with processors. 12410

  • Fix a race condition in the TCP input when close the client socket. 13038

  • cisco/asa fileset: Renamed log.original to event.original and cisco.asa.list_id to cisco.asa.rule_name. 13286

  • cisco/asa fileset: Fix parsing of 302021 message code. 13476

  • google pubsub & httpjson inputs: HTTP User agent is now Elastic-Heartbeat/Version instead of Elastic Heartbeat/Version to stay RFC compliant. 14748

  • CEF extensions are now mapped to the data types defined in the CEF guide. 14342

  • Remove --machine-learning from setup subcommand. 14705

Heartbeat

  • Removed the add_host_metadata and add_cloud_metadata processors from the default config. These don’t fit well with ECS for Heartbeat and were rarely used.

  • Fixed/altered redirect behavior. max_redirects now defaults to 0 (no redirects). Following redirects now works across hosts, but some timing fields will not be reported. 14125

  • Removed host.name field that should never have been included. Heartbeat uses observer.* fields instead. 14140

  • Changed default user-agent to be Elastic-Heartbeat/VERSION (PLATFORM_INFO) as the current default Go-http-client/1.1 is often blacklisted. 14291

  • JSON/Regex checks against HTTP bodies will only consider the first 100MiB of the HTTP body to prevent excessive memory usage. pull

  • Heartbeat now starts monitors scheduled with the '@every X' syntax instantaneously on startup, rather than waiting for the given interval to pass before running them. 14890

Journalbeat

  • Remove broken dashboard. 15288

Metricbeat

  • Add new dashboards for Azure vms, vm guest metrics, vm scale sets 14000

  • Add new Dashboard for PostgreSQL database stats 13187

  • Add new dashboard for CouchDB database 13198

  • Add new dashboard for Ceph cluster stats 13216

  • Add new dashboard for Aerospike database stats 13217

  • Add new dashboard for Couchbase cluster stats 13212

  • Add new dashboard for Prometheus server stats 13126

  • Add new dashboard for VSphere host cluster and virtual machine 14135

  • Add new option OpMultiplyBuckets to scale histogram buckets to avoid decimal points in final events 10994

  • system/raid metricset now uses /sys/block instead of /proc/mdstat for data. 11613

  • kubernetes.container.cpu.limit.cores and kubernetes.container.cpu.requests.cores are now floats. 11975

  • Add statistic option into cloudwatch metricset. If there is no statistic method specified, default is to collect Average, Sum, Maximum, Minimum and SampleCount. 12370 12840

  • Add sql module that fetches metrics from a SQL database 13257

Packetbeat

  • Add dns.question.subdomain and dns.question.top_level_domain fields. 14578

  • Add support for mongodb opcode 2013 (OP_MSG). 6191 8594

  • NFSv4: Always use opname ILLEGAL when failed to match request to a valid nfs operation. 11503

Winlogbeat

Functionbeat

  • Separate management and functions in Functionbeat. 12939

Bugfixes

Affecting all Beats

  • Make the behavior of clientWorker and netClientWorker consistent when error is returned from publisher pipeline

  • Fix a bug, publisher pipeline exits if output returns an error, irrespective of pipeline is closed or not

  • Fix typo in TLS renegotiation configuration and setting the option correctly 10871, 12354

  • Ensure all beat commands respect configured settings. 10721

  • Add missing fields and test cases for libbeat add_kubernetes_metadata processor. 11133, 11134

  • decode_json_field: process objects and arrays only 11312

  • decode_json_field: do not process arrays when flag not set. 11318

  • Report faulting file when config reload fails. 11304

  • Fix a typo in libbeat/outputs/transport/client.go by updating c.conn.LocalAddr() to c.conn.RemoteAddr(). 11242

  • Management configuration backup file will now have a timestamps in their name. 11034

  • [CM] Parse enrollment_token response correctly 11648

  • Not hiding error in case of http failure using elastic fetcher 11604

  • Escape BOM on JsonReader before trying to decode line 11661

  • Fix matching of string arrays in contains condition. 11691

  • Replace wmi queries with win32 api calls as they were consuming CPU resources 3249 and 11840

  • Fix a race condition with the Kafka pipeline client, it is possible that Close() get called before Connect() . 11945

  • Fix queue.spool.write.flush.events config type. 12080

  • Fixed a memory leak when using the add_process_metadata processor under Windows. 12100

  • Fix of docker json parser for missing "log" jsonkey in docker container’s log 11464

  • Fixed Beat ID being reported by GET / API. 12180

  • Fixed setting bulk max size in kafka output. 12254

  • Add host.os.codename to fields.yml. 12261

  • Fix @timestamp being duplicated in events if @timestamp is set in a processor (or by any code utilizing PutValue() on a beat.Event).

  • Fix leak in script processor when using Javascript functions in a processor chain. 12600

  • Add additional nil pointer checks to Docker client code to deal with vSphere Integrated Containers 12628

  • Fixed json.add_error_key property setting for delivering error messages from beat events 11298

  • Fix Central Management enroll under Windows 12797 12799

  • ILM: Use GET instead of HEAD when checking for alias to expose detailed error message. 12886

  • Fix seccomp policy preventing some features to function properly on 32bit Linux systems. 12990 13008

  • Fix unexpected stops on docker autodiscover when a container is restarted before cleanup_timeout. 12962 13127

  • Fix install-service.ps1’s ability to set Windows service’s delay start configuration. 13173

  • Fix some incorrect types and formats in field.yml files. 13188

  • Load DLLs only from Windows system directory. 13234 13384

  • Fix mapping for kubernetes.labels and kubernetes.annotations in add_kubernetes_metadata. 12638 13226

  • Fix case insensitive regular expressions not working correctly. 13250

  • Disable add_kubernetes_metadata if no matchers found. 13709

  • Better wording for xpack beats when the _xpack endpoint is not reachable. 13771

  • Recover from panics in the javascript process and log details about the failure to aid in future debugging. 13690

  • Make the script processor concurrency-safe. 13690 13857

  • Kubernetes watcher at add_kubernetes_metadata fails with StatefulSets 13905

  • Fix panics that could result from invalid TLS certificates. This can affect Beats that connect over TLS or Beats that accept connections over TLS and validate client certificates. 14146

  • Support usage of custom builders without hints and mappers 13839

  • Fix memory leak in kubernetes autodiscover provider and add_kubernetes_metadata processor happening when pods are terminated without sending a delete event. 14259

  • Fix kubernetes metaGenerator.ResourceMetadata when parent reference controller is nil 14320 14329

  • Allow users to configure only cluster_uuid setting under monitoring namespace. 14338

  • Fix proxy_url option in Elasticsearch output. 14950

  • Fix bug with potential concurrent reads and writes from event.Meta map by Kafka output. 14542 14568

Auditbeat

  • Process dataset: Fixed a memory leak under Windows. 12100

  • Login dataset: Fix re-read of utmp files. 12028

  • Package dataset: Fixed a crash inside librpm after Auditbeat has been running for a while. 12147 12168

  • Fix formatting of config files on macOS and Windows. 12148

  • Fix direction of incoming IPv6 sockets. 12248

  • Package dataset: Close librpm handle. 12215

  • Package dataset: Auto-detect package directories. 12289

  • Package dataset: Improve dpkg parsing. 12325

  • System module: Start system module without host ID. 12373

  • Host dataset: Fix reboot detection logic. 12591

  • Add syscalls used by librpm for the system/package dataset to the default Auditbeat seccomp policy. 12578 12617

  • Process dataset: Do not show non-root warning on Windows. 12740

  • Host dataset: Export Host fields to gob encoder. 12940

  • Socket dataset: Fix start errors when IPv6 is disabled on the kernel. 13953 13966

Filebeat

  • Add support for Cisco syslog format used by their switch. 10760

  • Cover empty request data, url and version in Apache2 modulehttps://github.com/elastic/pull/10730[10730]

  • Fix registry entries not being cleaned due to race conditions. 10747

  • Improve detection of file deletion on Windows. 10747

  • Add missing Kubernetes metadata fields to Filebeat CoreDNS module, and fix a documentation error. 11591

  • Reduce memory usage if long lines are truncated to fit max_bytes limit. The line buffer is copied into a smaller buffer now. This allows the runtime to release unused memory earlier. 11524

  • Fix memory leak in Filebeat pipeline acker. 12063

  • Fix goroutine leak caused on initialization failures of log input. 12125

  • Fix goroutine leak on non-explicit finalization of log input. 12164

  • Skipping unparsable log entries from docker json reader 12268

  • Parse timezone in PostgreSQL logs as part of the timestamp 12338

  • Load correct pipelines when system module is configured in modules.d. 12340

  • Fix timezone offset parsing in system/syslog. 12529

  • When TLS is configured for the TCP input and a certificate_authorities is configured we now default to required for the client_authentication. 12584

  • Apply max_message_size to incoming message buffer. 11966

  • Syslog input will now omit the process object from events if it is empty. 12700

  • Fix multiline pattern in Postgres which was too permissive 12078 13069

  • Allow path variables to be used in files loaded from modules.d. 13184

  • Fix filebeat autodiscover fileset hint for container input. 13296

  • Fix incorrect references to index patterns in AWS and CoreDNS dashboards. 13303

  • Fix timezone parsing of system module ingest pipelines. 13308

  • Fix timezone parsing of elasticsearch module ingest pipelines. 13367

  • Change iis url path grok pattern from URIPATH to NOTSPACE. 12710 13225 7951 13378 14754

  • Fix timezone parsing of nginx module ingest pipelines. 13369

  • Fix incorrect field references in envoyproxy dashboard 13420 13421

  • Fixed early expiration of templates (Netflow v9 and IPFIX). 13821

  • Fixed bad handling of sequence numbers when multiple observation domains were exported by a single device (Netflow V9 and IPFIX). 13821

  • Fix timezone parsing of rabbitmq module ingest pipelines. 13879

  • Fix conditions and error checking of date processors in ingest pipelines that use event.timezone to parse dates. 13883

  • Fix timezone parsing of Cisco module ingest pipelines. 13893

  • Fix timezone parsing of logstash module ingest pipelines. 13890

  • cisco asa and ftd filesets: Fix parsing of message 106001. 13891 13903

  • Fix timezone parsing of iptables, mssql and panw module ingest pipelines. 13926

  • Fix merging of fields specified in global scope with fields specified under an input’s scope. 3628 13909

  • Fix delay in enforcing close_renamed and close_removed options. 13488 13907

  • Fix missing netflow fields in index template. 13768 13914

  • Fix cisco module’s asa and ftd filesets parsing of domain names where an IP address is expected. 14034

  • Fixed increased memory usage with large files when multiline pattern does not match. 14068

  • panw module: Use geo.name instead of geo.country_iso_code for free-form location. 13272

  • Fix azure fields names. 14098

  • Fix calculation of network.bytes and network.packets for bi-directional netflow events. 14111

  • Accept '-' as http.response.body.bytes in apache module. 14137

  • Fix timezone parsing of MySQL module ingest pipelines. 14130

  • Fix azure filesets test files. 14185 14235

  • Improve error message in s3 input when handleSQSMessage failed. 14113

  • Close chan of Closer first before calling callback 14231

  • Fix race condition in S3 input plugin. 14359

  • Decode hex values in auditd module. 14471

  • Fix parse of remote addresses that are not IPs in nginx logs. 14505

  • Fix handling multiline log entries in nginx module. 14349 14499

  • Fix parsing of Elasticsearch node name by elasticsearch/slowlog fileset. 14547

  • cisco/asa fileset: Fix parsing of 302021 message code. 14519

  • Fix filebeat azure dashboards, event category should be Alert. 14668

  • Update Logstash module’s Grok patterns to support Logstash 7.4 logs. 14743

  • Fix a problem in Filebeat input httpjson where interval is not used as time.Duration. 14752 14753

  • Fix SSL config in input.yml for Filebeat httpjson input in the MISP module. 14767

Heartbeat

  • Fix NPEs / resource leaks when executing config checks. 11165

  • Fix duplicated IPs on mode: all monitors. 12458

  • Fix integer comparison on JSON responses. 13348

  • Fix storage of HTTP bodies to work when JSON/Regex body checks are enabled. 14223

  • Fix recording of SSL cert metadata for Expired/Unvalidated x509 certs. 13687

  • The heartbeat scheduler no longer drops scheduled items when under very high load causing missed deadlines. 14890

Journalbeat

  • Use backoff when no new events are found. 11861

  • Iterate over journal correctly, so no duplicate entries are sent. 12716

  • Preserve host name when reading from remote journal. 12714

Metricbeat

  • Change diskio metrics retrieval method (only for Windows) from wmi query to DeviceIOControl function using the IOCTL_DISK_PERFORMANCE control code 11635

  • Call GetMetricData api per region instead of per instance. 11820 11882

  • Update documentation with cloudwatch:ListMetrics permission. 11987

  • Check permissions in system socket metricset based on capabilities. 12039

  • Get process information from sockets owned by current user when system socket metricset is run without privileges. 12039

  • Avoid generating hints-based configuration with empty hosts when no exposed port is suitable for the hosts hint. 8264 12086

  • Fixed a socket leak in the postgresql module under Windows when SSL is disabled on the server. 11393

  • Change some field type from scaled_float to long in aws module. 11982

  • Fixed RabbitMQ queue metricset gathering when consumer_utilisation is set empty at the metrics source 12089

  • Fix direction of incoming IPv6 sockets. 12248

  • Refactored Windows perfmon metricset: replaced method to retrieve counter paths with PdhExpandWildCardPathW, separated code by responsibility, removed unused functions 12212

  • Validate that kibana/status metricset cannot be used when xpack is enabled. 12264

  • Ignore prometheus metrics when their values are NaN or Inf. 12084 10849

  • In the kibana/stats metricset, only log error (don’t also index it) if xpack is enabled. 12265

  • Fix an issue listing all processes when run under Windows as a non-privileged user. 12301 12475

  • The elasticsearch/index_summary metricset gracefully handles an empty Elasticsearch cluster when xpack.enabled: true is set. 12489 12487

  • When TLS is configured for the http metricset and a certificate_authorities is configured we now default to required for the client_authentication. 12584

  • Reuse connections in PostgreSQL metricsets. 12504 12603

  • PdhExpandWildCardPathW will not expand counter paths in 32 bit windows systems, workaround will use a different function. 12590 12622

  • In the elasticsearch/node_stats metricset, if xpack is enabled, make parsing of ES node load average optional as ES on Windows doesn’t report load average. 12866

  • Ramdisk is not filtered out when collecting disk performance counters in diskio metricset 12814 12829

  • Fix incoherent behaviour in redis key metricset when keyspace is specified both in host URL and key pattern 12913

  • Fix connections leak in redis module 12914 12950

  • Fix wrong uptime reporting by system/uptime metricset under Windows. 12915

  • Print errors that were being omitted in vSphere metricsets. 12816

  • Fix redis key metricset dashboard references to index pattern. 13303

  • Check if fields in DBInstance is nil in rds metricset. 13294 13037

  • Fix silent failures in kafka and prometheus module. 13353 13252

  • Fix issue with aws cloudwatch module where dimensions and/or namespaces that contain space are not being parsed correctly 13389

  • Fix panic in Redis Key metricset when collecting information from a removed key. 13426

  • Fix module-level fields in Kubernetes metricsets. 13433 13544

  • Fix reporting empty events in cloudwatch metricset. 13458

  • Fix docker.cpu.system.pct calculation by using the reported number online cpus instead of the number of metrics per cpu. 13691

  • Fix rds metricset dashboard. 13721

  • Ignore prometheus untyped metrics with NaN value. 13750 13790

  • Change kubernetes.event.message to text. 13964

  • Fix performance counter values for windows/perfmon metricset. 14036 14039

  • Add FailOnRequired when applying schema and fix metric names in mongodb metrics metricset. 14143

  • Change server_status_path default setting for nginx module 13806 14099

  • Convert increments of 100 nanoseconds/ticks to milliseconds for WriteTime and ReadTime in diskio metricset (Windows) for consistency. 14233

  • Limit some of the error messages to the logs only 14317 14327

  • Convert indexed ms-since-epoch timestamp fields in elasticsearch/ml_job metricset to ints from float64s. 14220 14222

  • Fix ARN parsing function to work for ELB ARNs. 14316

  • Update azure configuration example. 14224

  • Fix cloudwatch metricset with names and dimensions in config. 14376 14391

  • Fix marshaling of ms-since-epoch values in elasticsearch/cluster_stats metricset. 14378

  • Fix checking tagsFilter using length in cloudwatch metricset. 14525

  • Log bulk failures from bulk API requests to monitoring cluster. 14303 14356

  • Fixed bug with elasticsearch/cluster_stats metricset not recording license expiration date correctly. 14541 14591

  • Fix regular expression to detect instance name in perfmon metricset. 14273 14666

  • Vshpere module splits virtualmachine.host into virtualmachine.host.id and virtualmachine.host.hostname. 7187 7213

  • Fixed bug with elasticsearch/cluster_stats metricset not recording license ID in the correct field. 14592

  • Fix perfmon expanding counter path/adding counter to query when OS language is not english. 14684 14800

  • Add extra check on ignore_non_existent_counters flag if the PdhExpandWildCardPathW returns no errors but does not expand the counter path successfully in windows/perfmon metricset. 14797

  • Fix rds metricset from reporting same values for different instances. 14702

  • Closing handler after verifying the registry key in diskio metricset. 14683 14759

  • Fix docker network stats when multiple interfaces are configured. 14586 14825

  • Fix ListMetrics pagination in aws module. 14926 14942

  • Fix CPU count in docker/cpu in cases where no online_cpus are reported 15070

  • Fix mixed modules loading standard and light metricsets 15011

  • Fix docker.container.size fields values 14979 15224

  • Make kibana module more resilient to Kibana unavailability. 15258 15270

  • Make logstash module more resilient to Logstash unavailability. 15276 15306

Packetbeat

  • Prevent duplicate packet loss error messages in HTTP events. 10709

  • Fixed a memory leak when using process monitoring under Windows. 12100

  • Improved debug logging efficiency in PGQSL module. 12150

  • Limit memory usage of Redis replication sessions. 12657

  • Fix parsing the extended RCODE in the DNS parser. 12805

  • Fix parsing of the HTTP host header when it contains a port or an IPv6 address. 14215

Winlogbeat

  • Fix data race affecting config validation at startup. 13005

  • Set host.name to computername in Windows event logs & sysmon. Requires 14407 in libbeat to work 13706

Functionbeat

  • Fix function name reference for Kinesis streams in CloudFormation templates 11646

  • Fix Cloudwatch logs timestamp to use timestamp of the log record instead of when the record was processed 13291

  • Look for the keystore under the correct path. 13332

Added

Affecting all Beats

  • Decouple Debug logging from fail_on_error logic for rename, copy, truncate processors 12451

  • Add an option to append to existing logs rather than always rotate on start. 11953

  • Add network condition to processors for matching IP addresses against CIDRs. 10743

  • Add if/then/else support to processors. 10744

  • Add community_id processor for computing network flow hashes. 10745

  • Add output test to kafka output 10834

  • Gracefully shut down on SIGHUP 10704

  • New processor: copy_fields. 11303

  • Add error.message to events when fail_on_error is set in rename and copy_fields processors. 11303

  • New processor: truncate_fields. 11297

  • Allow a beat to ship monitoring data directly to an Elasticsearch monitoring cluster. 9260

  • Updated go-seccomp-bpf library to v1.1.0 which updates syscall lists for Linux v5.0. 11394

  • Add add_observer_metadata processor. 11394

  • Add decode_csv_fields processor. 11753

  • Add convert processor for converting data types of fields. 8124 11686

  • New extract_array processor. 11761

  • Add number of goroutines to reported metrics. 12135

  • Add proxy_disable output flag to explicitly ignore proxy environment variables. 11713 12243

  • Processor add_cloud_metadata adds fields cloud.account.id and cloud.image.id for AWS EC2. 12307

  • Add configurable bulk_flush_frequency in kafka output. 12254

  • Add decode_base64_field processor for decoding base64 field. 11914

  • Add support for reading the network.iana_number field by default to the community_id processor. 12701

  • Add aws overview dashboard. 11007 12175

  • Add decompress_gzip_field processor. 12733

  • Add timestamp processor for parsing time fields. 12699

  • Fail with error when autodiscover providers have no defined configs. 13078

  • Add a check so alias creation explicitely fails if there is an index with the same name. 13070

  • Update kubernetes watcher to use official client-go libraries. 13051

  • Add support for unix epoch time values in the timestamp processor. 13319

  • add_host_metadata is now GA. 13148

  • Add an ignore_missing configuration option the drop_fields processor. 13318

  • add_host_metadata is no GA. 13148

  • Add registered_domain processor for deriving the registered domain from a given FQDN. 13326

  • Add support for RFC3339 time zone offsets in JSON output. 13227

  • Add autodetection mode for add_docker_metadata and enable it by default in included configuration fileshttps://github.com/elastic/pull/13374[13374]

  • Added monitoring.cluster_uuid setting to associate Beat data with specified ES cluster in Stack Monitoring UI. 13182

  • Add autodetection mode for add_kubernetes_metadata and enable it by default in included configuration files. 13473

  • Add providers setting to add_cloud_metadata processor. 13812

  • Use less restrictive API to check if template exists. 13847

  • Do not check for alias when setup.ilm.check_exists is false. 13848

  • Add support for numeric time zone offsets in timestamp processor. 13902

  • Add condition to the config file template for add_kubernetes_metadata 14056

  • Marking Central Management deprecated. 14018

  • Add keep_null setting to allow Beats to publish null values in events. 5522 13928

  • Add shared_credential_file option in aws related config for specifying credential file directory. 14157 14178

  • GA the script processor. 14325

  • Add fingerprint processor. 11173 14205

  • Add support for API keys in Elasticsearch outputs. 14324

  • Ensure that init containers are no longer tailed after they stop 14394

  • Add consumer_lag in Kafka consumergroup metricset 14822

  • Make use of consumer_lag in Kafka dashboard 14863

  • Refactor kubernetes autodiscover to enable different resource based discovery 14738

  • Add add_id processor. 14524

  • Enable TLS 1.3 in all beats. 12973

  • Enable DEP (Data Execution Protection) for Windows packages. 15149

Auditbeat

  • Auditd module: Add event.outcome and event.type for ECS. 11432

  • Process: Add file hash of process executable. 11722

  • Socket: Add network.transport and network.community_id. 12231

  • Host: Fill top-level host fields. 12259

  • Socket: Add DNS enrichment. 14004

Filebeat

  • Add more info to message logged when a duplicated symlink file is found 10845

  • Add option to configure docker input with paths 10687

  • Add Netflow module to enrich flow events with geoip data. 10877

  • Set event.category: network_traffic for Suricata. 10882

  • Allow custom default settings with autodiscover (for example, use of CRI paths for logs). 12193

  • Allow to disable hints based autodiscover default behavior (fetching all logs). 12193

  • Change Suricata module pipeline to handle destination.domain being set if a reverse DNS processor is used. 10510

  • Add the network.community_id flow identifier to field to the IPTables, Suricata, and Zeek modules. 11005

  • New Filebeat coredns module to ingest coredns logs. It supports both native coredns deployment and coredns deployment in kubernetes. 11200

  • New module for Cisco ASA logs. 9200 11171

  • Added support for Cisco ASA fields to the netflow input. 11201

  • Configurable line terminator. 11015

  • Add Filebeat envoyproxy module. 11700

  • Add apache2(httpd) log path (/var/log/httpd) to make apache2 module work out of the box on Redhat-family OSes. 11887 11888

  • Add support to new MongoDB additional diagnostic information 11952

  • New module panw for Palo Alto Networks PAN-OS logs. 11999

  • Add RabbitMQ module. 12032

  • Add new container input. 12162

  • Add timeouts on communication with docker daemon. 12310

  • container and docker inputs now support reading of labels and env vars written by docker JSON file logging driver. 8358

  • Add specific date processor to convert timezones so same pipeline can be used when convert_timezone is enabled or disabled. 12253

  • Add MSSQL module 12079

  • Add ISO8601 date parsing support for system module. 12568 12579

  • Update Kubernetes deployment manifest to use container input. 12632

  • Use correct OS path separator in add_kubernetes_metadata to support Windows nodes. 9205

  • Add support for virtual host in Apache access logs 12778

  • Add support for client addresses with port in Apache error logs 12695

  • Add google-pubsub input type for consuming messages from a Google Cloud Pub/Sub topic subscription. 12746

  • Add module for ingesting Cisco IOS logs over syslog. 12748

  • Add module for ingesting Google Cloud VPC flow logs. 12747

  • Report host metadata for Filebeat logs in Kubernetes. 12790

  • Add netflow dashboards based on Logstash netflow. 12857

  • Parse more fields from Elasticsearch slowlogs. 11939

  • Update module pipelines to enrich events with autonomous system fields. 13036

  • Add module for ingesting IBM MQ logs. 8782

  • Add S3 input to retrieve logs from AWS S3 buckets. 12640 12582

  • Add aws module s3access metricset. 13170 12880

  • Update Suricata module to populate ECS DNS fields and handle EVE DNS version 2. 13320 13329

  • Update PAN-OS fileset to use the ECS NAT fields. 13320 13330

  • Add fields to the Zeek DNS fileset for ECS DNS. 13320 13324

  • Add container image in Kubernetes metadata 13356 12688

  • Add timezone information to apache error fileset. 12772 13304

  • Add module for ingesting Cisco FTD logs over syslog. 13286

  • Update CoreDNS module to populate ECS DNS fields. 13320 13505

  • Parse query steps in PostgreSQL slowlogs. 13496 13701

  • Add filebeat azure module with activitylogs, auditlogs, signinlogs filesets. 13776 14033

  • Add support to set the document id in the json reader. 5844

  • Add input httpjson. 13545 13546

  • Filebeat Netflow input: Remove beta label. 13858

  • Remove event.timezone from events that don’t need it in some modules that support log formats with and without timezones. 13918

  • Add ExpandEventListFromField config option in the kafka input. 13965

  • Add ELB fileset to AWS module. 14020

  • Add module for MISP (Malware Information Sharing Platform). 13805

  • Add source.bytes and source.packets for uni-directional netflow events. 14111

  • Add support for gzipped files in S3 input. 13980

  • Add support for all the ObjectCreated events in S3 input. 14077

  • Add Kibana Dashboard for MISP module. 14147

  • Add JSON options to autodiscover hints 14208

  • Add more filesets to Zeek module. 14150

  • Add index option to all inputs to directly set a per-input index value. 14010

  • Remove beta flag for some filebeat modules. 14374

  • Add support for http hostname in nginx filebeat module. 14505

  • Add attack_pattern_kql field to MISP threat indicators. 14470

  • Add fileset to the Zeek module for the intel.log. 14404

  • Add vpc flow log fileset to AWS module. 13880 14345

  • New fileset googlecloud/firewall for ingesting Google Cloud Firewall logs. 14553

  • Add document for Filebeat input httpjson. 14602

  • Add more configuration options to the Netflow module. https://github.com/elastic/beats/pull/14628{14628}

  • Add dashboards to the CEF module (ported from the Logstash ArcSight module).

  • Add dashboards to the CEF module (ported from the Logstash ArcSight module). 14342

  • Fix timezone parsing in haproxy pipeline. 14755

  • Add module for ActiveMQ. 14840

  • Add dashboards for the ActiveMQ Filebeat module. 14880

  • Add STAN Metricbeat module. 14839

  • Add new fileset googlecloud/audit for ingesting Google Cloud Audit logs. 15200

Heartbeat - Add non-privileged icmp on linux and darwin(mac). 13795 11498

  • Enable add_observer_metadata processor in default config. 11394

  • Record HTTP body metadata and optionally contents in http.response.body.* fields. 13022

  • Add monitor.timespan field for optimized queries in kibana. 13672

  • Allow hosts to be used to configure http monitors 13703

  • google-pubsub input: ACK pub/sub message when acknowledged by publisher. 13346 14715

  • Remove Beta label from google-pubsub input. 13346 14715

Journalbeat - Add index option to all inputs to directly set a per-input index value. 15063 15071

Metricbeat

  • Add AWS SQS metricset. 10684 10053

  • Add AWS s3_request metricset. 10949 10055

  • Add s3_daily_storage metricset. 10940 10055

  • Add coredns metricbeat module. 10585

  • Add SSL support for Metricbeat HTTP server. 11482 11457

  • The elasticsearch.index metricset (with xpack.enabled: true) now collects refresh.external_total_time_in_millis fields from Elasticsearch. 11616

  • Allow module configurations to have variants 9118

  • Add timeseries.instance field calculation. 10293

  • Added new disk states and raid level to the system/raid metricset. 11613

  • Added path_name and start_name to service metricset on windows module 8364 11877

  • Add check on object name in the counter path if the instance name is missing 6528 11878

  • Add AWS cloudwatch metricset. 11798 11734

  • Add regions in aws module config to specify target regions for querying cloudwatch metrics. 11932 11956

  • Keep etcd followers members from reporting leader metricset events 12004

  • Add overview dashboard to Consul module 10665

  • New fields were added in the mysql/status metricset. 12227

  • Add Kubernetes metricset proxy. 12312

  • Add Kubernetes proxy dashboard to Kubernetes module 12734

  • Always report Pod UID in the pod metricset. 12345

  • Add Vsphere Virtual Machine operating system to os field in Vsphere virtualmachine module. 12391

  • Add validation for elasticsearch and kibana modules' metricsets when xpack.enabled is set to true. 12386

  • Add CockroachDB module. 12467

  • Add support for metricbeat modules based on existing modules (a.k.a. light modules) 12270 12465

  • Add a system/entropy metricset 12450

  • Add kubernetes metricset controllermanager 12409

  • Add Kubernetes controller manager dashboard to Kubernetes module 12744

  • Allow redis URL format in redis hosts config. 12408

  • Add tags into ec2 metricset. 1226312263 12372

  • Add metrics to kubernetes apiserver metricset. 12922

  • Add kubernetes metricset scheduler 12521

  • Add Kubernetes scheduler dashboard to Kubernetes module 12749

  • Add beat module. 12181 12615

  • Collect tags for cloudwatch metricset in aws module. 1226312263 12480

  • Add AWS RDS metricset. 11620 10054

  • Add Oracle Module 11890

  • Add Oracle Tablespaces Dashboard 12736

  • Collect client provided name for rabbitmq connection. 12851 12852

  • Add support to load default aws config file to get credentials. 12727 12708

  • Add statistic option into cloudwatch metricset. 12370 12840

  • Add support for kubernetes cronjobs 13001

  • Add cgroup memory stats to docker/memory metricset 12916

  • Add AWS elb metricset. 12952 11701

  • Add AWS ebs metricset. 13167 11699

  • Add metricset.period field with the configured fetching period. 13242 12616

  • Add rate metrics for ec2 metricset. 13203

  • Add refresh list of perf counters at every fetch 13091

  • Add Performance metricset to Oracle module 12547

  • Add proc/vmstat data to the system/memory metricset on linux 13322

  • Use DefaultMetaGeneratorConfig in MetadataEnrichers to initialize configurations 13414

  • Add module for statsd. 13109

  • Add support for NATS version 2. 13601

  • Add docker.cpu.*.norm.pct metrics for cpu metricset of Docker Metricbeat module. 13695

  • Add instance label by default when using Prometheus collector. 13737

  • Add azure module. 13196 13859 13988

  • Add Apache Tomcat module 13491

  • Add ECS container.id and container.runtime to kubernetes state_container metricset. 13884

  • Add job label by default when using Prometheus collector. 13878

  • Add state_resourcequota metricset for Kubernetes module. 13693

  • Add tags filter in ec2 metricset. 13872 13145

  • Add cloud.account.id and cloud.account.name into events from aws module. 13551 13558

  • Add metrics_path as known hint for autodiscovery 13996

  • Leverage KUBECONFIG when creating k8s client. 13916

  • Add ability to filter by tags for cloudwatch metricset. 13758 13145

  • Release cloudwatch, s3_daily_storage, s3_request, sqs and rds metricset as GA. 14114 14059

  • Add Oracle overview dashboard 14021

  • Release CoreDNS module as GA. 14308

  • Release CouchDB module as GA. 14300

  • Add elasticsearch/enrich metricset. 14243 14221

  • Add support for Application ELB and Network ELB. 14123 13538 13539

  • Release aws ebs metricset as GA. 14312 14060

  • Add connection.state field for RabbitMQ module. 13981

  • Add more TCP states to Metricbeat system socket_summary. 14347

  • Add Kafka JMX metricsets. 14330

  • Add metrics to envoyproxy server metricset and support for envoy proxy 1.12. 14416 13642

  • Release kubernetes modules controllermanager, scheduler, proxy, state_cronjob and state_resourcequota as GA. 14584

  • Add module for ActiveMQ. 14580

  • Enable script processor. 14711

  • Enable wildcard for cloudwatch metricset namespace. 14971 14965

  • Add kube-state-metrics state_service metrics for kubernetes module. 14794

  • Add kube-state-metrics state_persistentvolume metrics for kubernetes module. 14859

  • Add kube-state-metrics state_persistentvolumeclaim metrics for kubernetes module. 15066

  • Add usage metricset in aws modules. 14925 14935

  • Add billing metricset in aws modules. 14801 14934

  • Add AWS SNS metricset. 14946

  • Add overview dashboard for AWS SNS module 14977

  • Add index option to all modules to specify a module-specific output index. 15100

  • Add a system/service metricset for systemd data. 14206

Packetbeat

  • Update DNS protocol plugin to produce events with ECS fields for DNS. 13320 13354

Functionbeat

  • New options to configure roles and VPC. 11779

  • Export automation templates used to create functions. 11923

  • Configurable Amazon endpoint. 12369

  • Add timeout option to reference configuration. 13351

  • Configurable tags for Lambda functions. 13352

  • Add input for Cloudwatch logs through Kinesis. 13317

  • Enable Logstash output. 13345

  • Make bulk_max_size configurable in outputs. 13493

  • Add index option to all functions to directly set a per-function index value. 15064 15101

Winlogbeat

  • Add support for reading from .evtx files. 4450

  • Add support for event ID 4634 and 4647 to the Security module. 12906

  • Add network.community_id to Sysmon network events (event ID 3). 13034

  • Add event.module to Winlogbeat modules. 13047

  • Add event.category: process and event.type: process_start/process_end to Sysmon process events (event ID 1 and 5). 13047

  • Add support for event ID 4672 to the Security module. 12975

  • Add support for event ID 22 (DNS query) to the Sysmon module. 12960

  • Add certain winlog.event_data.* fields to the index template. 13700 13704

  • Fill event.provider. 13937

  • Add support for user management events to the Security module. 13530

  • GA the Winlogbeat sysmon module. 14326

  • Add support for event ID 4688 & 4689 (Process create & exit) to the Security module. 14038

Deprecated

Affecting all Beats

Filebeat

  • docker input is deprecated in favour container. 12162

  • postgresql.log.timestamp field is deprecated in favour of @timestamp. 12338

Heartbeat

Journalbeat

Metricbeat

  • kubernetes.container.id field for state_container is deprecated in favour of ECS container.id and container.runtime. 13884

Packetbeat

Winlogbeat

Functionbeat

Known Issue

Journalbeat