diff --git a/CHANGELOG.asciidoc b/CHANGELOG.asciidoc index 1bfd81eb38b..375b86ac6e8 100644 --- a/CHANGELOG.asciidoc +++ b/CHANGELOG.asciidoc @@ -3,6 +3,175 @@ :issue: https://github.com/elastic/beats/issues/ :pull: https://github.com/elastic/beats/pull/ +[[release-notes-7.14.0]] +=== Beats version 7.14.0 +https://github.com/elastic/beats/compare/v7.13.4...v7.14.0[View commits] + +==== Breaking changes + +*Affecting all Beats* + +- Removed beats central management {pull}25696[25696], {issue}23908[23908] +- MacOSX minimum supported version set to 10.14 {issue}24193[24193] + +*Filebeat* + +- Change logging in logs input to structure logging. Some log message formats have changed. {pull}25299[25299] +- All url.* fields apart from url.original in the Apache, Nginx, IIS, Traefik, S3Access, Cisco, F5, Fortinet, Google Workspace, Imperva, Microsoft, Netscout, O365, Sophos, Squid, Suricata, Zeek, Zia, Zoom, and ZScaler modules are now url unescaped due to using the Elasticsearch uri_parts processor. {pull}24699[24699] +- Change source field for `event.action` in `fortinet.firewall` module to `fortinet.firewall.action` instead of `fortinet.firewall.eventtype`. {pull}24816[24816] +- threatintel module: Changed the type of `threatintel.indicator.first_seen` from `keyword` to `date`. {pull}26765[26765] + +*Heartbeat* + +- Add support for screenshot blocks and use newer synthetics flags that only works in newer synthetics betas. {pull}25808[25808] + +*Metricbeat* + +- Adjust host fields to adopt new names from 1.9.0 ECS. {pull}24312[24312] + +==== Bugfixes + +*Affecting all Beats* + +- Omit full index template from errors that occur while loading the template. {pull}25743[25743] +- In the script processor, the `decode_xml` and `decode_xml_wineventlog` processors are now available as `DecodeXML` and `DecodeXMLWineventlog` respectively. +- Fix encoding errors when using the disk queue on nested data with multi-byte characters {pull}26484[26484] + +*Auditbeat* + +- file_integrity: Create fsnotify watcher only when starting file_integrity module {pull}19505[19505] +- system/socket: Fix kprobe grouping to allow running more than one instance. {pull}20325[20325] +- system/socket: Fixed a crash due to concurrent map read and write. {issue}21192[21192] {pull}21690[21690] +- auditd: Fix an error condition causing a lot of `audit_send_reply` kernel threads being created. {pull}22673[22673] +- system/socket: Fixed start failure when run under config reloader. {issue}20851[20851] {pull}21693[21693] +- system/socket: Having some CPUs unavailable to Auditbeat could cause startup errors or event loss. {pull}22827[22827] + +*Filebeat* + +- Fix mapping of `fortinet.firewall.mem` as integer. {pull}19335[19335] +- Add `shared_credential_file` to cloudtrail config {issue}15652[15652] {pull}15656[15656] +- Fix integer overflow in S3 offsets when collecting very large files. {pull}22523[22523] +- Fix issue with m365_defender, when parsing incidents that has no alerts attached: {pull}25421[25421] +- Fix default config template values for paths on oracle module: {pull}26276[26276] +- Fix Elasticsearch compatibility for modules that use `copy_from` in `set` processors. {issue}26629[26629] +- Change type of max_bytes in all configs to be cfgtype.ByteSize {pull}26699[26699] +- Change `checkpoint.source_object` from Long to Keyword. {issue}25124[25124] {pull}25145[25145] +- Fix Nginx module pipelines. {issue}19088[19088] {pull}24699[24699] +- Fix incorrect field name appending to `related.hash` in `threatintel.abusechmalware` ingest pipeline. {issue}25151[25151] {pull}25674[25674] +- Add improvements to the azure activitylogs and platformlogs ingest pipelines. {pull}26148[26148] +- Fix `kibana.log` pipeline when `event.duration` calculation becomes a Long. {issue}24556[24556] {pull}25675[25675] +- Removed incorrect `http.request.referrer` field from `aws.elb` module. {issue}26435[26435] {pull}26441[26441] +- Fix `threatintel.indicator.url.full` not being populated. {issue}26351[26351] {pull}26508[26508] +- Fix Suricata metadata fields breaking visualizations, moved out of flattened datatype. {pull}26710[26710] +- Fix `httpjson` template data key for `url.params`. {pull}26848[26848] +- Cisco asa/ftd: Fix reversed usage of observer ingress and egress interfaces. {pull}26265[26265] +- Fix `aws.s3access` pipeline when remote IP is a `-`. {issue}26913[26913] {pull}26940[26940] +- Fix service name in aws-cloudwatch input from cloudwatchlogs to logs. {pull}27007[27007] + +*Heartbeat* + +- Add Context to otherwise ambiguous HTTP body read errors. {pull}25499[25499] + +*Metricbeat* + +- Major refactor of system/cpu and system/core metrics. {pull}25771[25771] +- Fix GCP Project ID being ingested as `cloud.account.id` in `gcp.billing` module {issue}26357[26357] {pull}26412[26412] +- Fix memory leak in SQL module when database is not available. {issue}25840[25840] {pull}26607[26607] +- Fix aws metric tags with resourcegroupstaggingapi paginator. {issue}26385[26385] {pull}26443[26443] +- Fix quoting in GCP billing table name {issue}26855[26855] {pull}26870[26870] +- Recover `service.address` field in vsphere module {issue}26902[26902] {pull}26904[26904] + +*Winlogbeat* + +- Fix `related.ip` field in renameCommonAuthFields {pull}24892[24892] + +*Functionbeat* + +- Expose region in AWS configuration so Functionbeat can deploy the Lambda in the correct place. {pull}26523[26523] + +==== Added + +*Affecting all Beats* + +- Add support for defining explicitly named dynamic templates without path/type match criteria {pull}25422[25422] +- Improve ES output error insights. {pull}25825[25825] +- Add orchestrator.cluster.name/url fields as k8s metadata {pull}26056[26056] +- Libbeat: report beat version to monitoring. {pull}26214[26214] +- Ensure common proxy settings support in HTTP clients: `proxy_disabled`, `proxy_url`, `proxy_headers` and typical environment variables `HTTP_PROXY`, `HTTPS_PROXY`, `NOPROXY`. {pull}25219[25219] + +*Filebeat* + +- Update PanOS module to parse Global Protect & User ID logs. {issue}24722[24722] {issue}24724[24724] {pull}24927[24927] +- Add HMAC signature validation support for http_endpoint input. {pull}24918[24918] +- Add new grok pattern for iptables module for Ubiquiti UDM {issue}25615[25615] {pull}25616[25616] +- Add multiline support to aws-s3 input. {issue}25249[25249] {pull}25710[25710] {pull}25873[25873] +- Add monitoring metrics to the `aws-s3` input. {pull}25711[25711] +- Added `network.direction` fields to Zeek and Suricata modules using the `add_network_direction` processor {pull}24620[24620] +- Add Content-Type override to aws-s3 input. {issue}25697[25697] {pull}25772[25772] +- In Cisco Umbrella fileset add users from cisco.umbrella.identities to related.user. {pull}25776[25776] +- Add fingerprint processor to generate fixed ids for `google_workspace` events. {pull}25841[25841] +- Update PanOS module to parse HIP Match logs. {issue}24350[24350] {pull}25686[25686] +- Support MongoDB 4.4 in filebeat's MongoDB module. {issue}20501[20501] {pull}24774[24774] +- Enhance GCP module to populate orchestrator.* fields for GKE / K8S logs {pull}25368[25368] +- Add log_group_name_prefix config into aws-cloudwatch input. {pull}26187[26187] +- Move Filebeat azure module to GA. {pull}26114[26114] {pull}26168[26168] +- Make `filestream` input GA. {pull}26127[26127] +- http_endpoint: Support multiple documents in a single request by POSTing an array or NDJSON format. {pull}25764[25764] +- Add new `parser` to `filestream` input: `container`. {pull}26115[26115] +- Add support for ISO8601 timestamps in Zeek fileset {pull}25564[25564] +- Add possibility to include headers in resulting docs and preserve the original event in http_endpoint input {pull}26279[26279] +- Add `preserve_original_event` option to `o365audit` input. {pull}26273[26273] +- Add `log.flags` to events created by the `aws-s3` input. {pull}26267[26267] +- Add `include_s3_metadata` config option to the `aws-s3` input for including object metadata in events. {pull}26267[26267] +- RFC 5424 and UNIX socket support in the Syslog input are now GA {pull}26293[26293] +- Update grok patterns for HA Proxy module {issue}25827[25827] {pull}25835[25835] +- Update PanOS module's date processor formats to parse `strict_date_optional_time_nanos`. {issue}26033[26033] {pull}26158[26158] +- Update Okta module to parse additional fields to `okta.debug_context.debug_data`. {issue}25689[25689] {pull}25818[25818] +- Added dataset `anomalithreatstream` to the `threatintel` module to ingest indicators from Anomali ThreatStream {pull}26350[26350] + +- Add support for `copytruncate` method when rotating input logs with an external tool in `filestream` input. {pull}23457[23457] +- Add `uri_parts` and `user_agent` ingest processors to `aws.elb` module. {issue}26435[26435] {pull}26441[26441] +- Added dataset `recordedfuture` to the `threatintel` module to ingest indicators from Recorded Future Connect API {pull}26481[26481] +- Update `fortinet` ingest pipelines. {issue}22136[22136] {issue}25254[25254] {pull}24816[24816] +- Release Filebeat Stack Monitoring modules as GA {pull}26226[26226] +- Use default add_locale for fortinet.firewall {issue}20300[20300] {pull}26524[26524] + +*Heartbeat* + +- Add support for `copytruncate` method when rotating input logs with an external tool in `filestream` input. {pull}23457[23457] +- Add `proxy_headers` to HTTP monitor. {pull}25219[25219] +- Suppress too many bad message error logs when reading from corrupted journal for 5 seconds. {pull}26224[26224] +- Add `replicas.ready` field to state_statefulset in Kubernetes module {pull}26088[26088] + +*Metricbeat* + +- Refactor `state_*` metricsets to share response from endpoint. {pull}25640[25640] +- Add server id to zookeeper events. {pull}25550[25550] +- Add additional network metrics to docker/network {pull}25354[25354] +- Migrate ec2 metricsets to use cloudwatch input. {pull}25924[25924] +- Reduce number of requests done by kubernetes metricsets to kubelet. {pull}25782[25782] +- Migrate rds metricsets to use cloudwatch input. {pull}26077[26077] +- Migrate sqs metricsets to use cloudwatch input. {pull}26117[26117] +- Collect linked account information in AWS billing. {pull}26285[26285] +- Add total CPU to vSphere virtual machine metrics. {pull}26167[26167] +- Add AWS Kinesis metricset. {pull}25989[25989] +- Add Cluster filter on ECS Kubernetes overview dashboard and corresponding section on Kubernetes module documentation page. {pull}26919[26919] + +*Packetbeat* + +- Add `url.extension` to HTTP events {issue}25990[25990] {pull}25999[25999] + +*Winlogbeat* + +- Changed the log level of the "Successfully published events" message from `info` to `debug` to reduce verbosity of the `info` logging level. To track event log reader activity use the `published_events` metric. {pull}25617[25617] + +==== Deprecated + +*Filebeat* + +- Deprecate the MISP module. The Threat Intel module should be used instead. {issue}25240[25240] + + [[release-notes-7.13.4]] === Beats version 7.13.4 https://github.com/elastic/beats/compare/v7.13.3...v7.13.4[View commits] diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc index 9c54c476c19..3b51000d2ab 100644 --- a/CHANGELOG.next.asciidoc +++ b/CHANGELOG.next.asciidoc @@ -10,8 +10,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d *Affecting all Beats* -- Update to Golang 1.12.1. {pull}11330[11330] -- Disable Alibaba Cloud and Tencent Cloud metadata providers by default. {pull}13812[12812] - Libbeat: Do not overwrite agent.*, ecs.version, and host.name. {pull}14407[14407] - Libbeat: Cleanup the x-pack licenser code to use the new license endpoint and the new format. {pull}15091[15091] - Refactor metadata generator to support adding metadata across resources {pull}14875[14875] @@ -19,20 +17,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Introduce APM libbeat instrumentation, active when running the beat with ELASTIC_APM_ACTIVE=true. {pull}17938[17938] - Make error message about locked data path actionable. {pull}18667[18667] - Fix panic with inline SSL when the certificate or key were small than 256 bytes. {pull}23820[23820] -- Ensure dynamic template names are unique for the same field. {pull}18849[18849] - Remove the deprecated `xpack.monitoring.*` settings. Going forward only `monitoring.*` settings may be used. {issue}9424[9424] {pull}18608[18608] -- Added `certificate` TLS verification mode to ignore server name mismatch. {issue}12283[12283] {pull}20293[20293] -- Autodiscover doesn't generate any configuration when a variable is missing. Previously it generated an incomplete configuration. {pull}20898[20898] -- Remove redundant `cloudfoundry.*.timestamp` fields. This value is set in `@timestamp`. {pull}21175[21175] -- Allow embedding of CAs, Certificate of private keys for anything that support TLS in ouputs and inputs. {pull}21179[21179] -- Update to Golang 1.12.1. {pull}11330[11330] -- Disable Alibaba Cloud and Tencent Cloud metadata providers by default. {pull}13812[12812] -- API address is a required setting in `add_cloudfoundry_metadata`. {pull}21759[21759] -- Update to ECS 1.7.0. {pull}22571[22571] -- Add support for SCRAM-SHA-512 and SCRAM-SHA-256 in Kafka output. {pull}12867[12867] -- Remove id_field_data {pull}25239[25239] -- Removed beats central management {pull}25696[25696], {issue}23908[23908] -- MacOSX minimum supported version set to 10.14 {issue}24193{24193} - Add daemonset.name in pods controlled by DaemonSets {pull}26808[26808], {issue}25816[25816] - Kubernetes autodiscover fails in node scope if node name cannot be discovered {pull}26947[26947] @@ -40,13 +25,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - File integrity dataset (macOS): Replace unnecessary `file.origin.raw` (type keyword) with `file.origin.text` (type `text`). {issue}12423[12423] {pull}15630[15630] - Change event.kind=error to event.kind=event to comply with ECS. {issue}18870[18870] {pull}20685[20685] -- Change network.direction values to ECS recommended values (inbound, outbound). {issue}12445[12445] {pull}20695[20695] -- Docker container needs to be explicitly run as user root for auditing. {pull}21202[21202] -- File integrity dataset no longer includes the leading dot in `file.extension` values (e.g. it will report "png" instead of ".png") to comply with ECS. {pull}21644[21644] -- Use ECS 1.7 ingress/egress network directions instead of inbound/outbound. {pull}22991[22991] -- Use ingress/egress instead of inbound/outbound for ECS 1.7 in auditd module. {pull}23000[23000] - -*Auditbeat* *Filebeat* @@ -67,24 +45,10 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Disable the option of running --machine-learning on its own. {pull}20241[20241] - Fix PANW field spelling "veredict" to "verdict" on event.action {pull}18808[18808] - Add support for GMT timezone offsets in `decode_cef`. {pull}20993[20993] -- API address and shard ID are required settings in the Cloud Foundry input. {pull}21759[21759] -- Remove `suricata.eve.timestamp` alias field. {issue}10535[10535] {pull}22095[22095] -- Rename bad ECS field name tracing.trace.id to trace.id in aws elb fileset. {pull}22571[22571] -- Fix parsing issues with nested JSON payloads in Elasticsearch audit log fileset. {pull}22975[22975] -- Rename `network.direction` values in crowdstrike/falcon to `ingress`/`egress`. {pull}23041[23041] -- Rename `s3` input to `aws-s3` input. {pull}23469[23469] -- Add User Agent Parser for Azure Sign In Logs Ingest Pipeline {pull}23201[23201] -- Changes filebeat httpjson input's append transform to create a list even with only a single value{pull}25074[25074] -- Change logging in logs input to structure logging. Some log message formats have changed. {pull}25299[25299] -- All url.* fields apart from url.original in the Apache, Nginx, IIS, Traefik, S3Access, Cisco, F5, Fortinet, Google Workspace, Imperva, Microsoft, Netscout, O365, Sophos, Squid, Suricata, Zeek, Zia, Zoom, and ZScaler modules are now url unescaped due to using the Elasticsearch uri_parts processor. {pull}24699[24699] -- Deprecated the cyberark module (replaced by cyberarkpas). {issue}25261[25261] {pull}25505[25505] -- Change source field for `event.action` in `fortinet.firewall` module to `fortinet.firewall.action` instead of `fortinet.firewall.eventtype`. {pull}24816[24816] - Release Filebeat Stack Monitoring modules as GA {pull}26226[26226] -- threatintel module: Changed the type of `threatintel.indicator.first_seen` from `keyword` to `date`. {pull}26765[26765] - Remove all alias fields pointing to ECS fields from modules. This affects the Suricata and Traefik modules. {issue}10535[10535] {pull}26627[26627] *Heartbeat* -- Add support for screenshot blocks and use newer synthetics flags that only works in newer synthetics betas. {pull}25808[25808] *Journalbeat* @@ -97,23 +61,10 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - kubernetes.container.cpu.limit.cores and kubernetes.container.cpu.requests.cores are now floats. {issue}11975[11975] - Fix ECS compliance of user.id field in system/users metricset {pull}19019[19019] - Remove "invalid zero" metrics on Windows and Darwin, don't report linux-only memory and diskio metrics when running under agent. {pull}21457[21457] -- Change cloud.provider from googlecloud to gcp. {pull}21775[21775] -- API address and shard ID are required settings in the Cloud Foundry module. {pull}21759[21759] -- Rename googlecloud module to gcp module. {pull}22246[22246] -- Use ingress/egress instead of inbound/outbound for system/socket metricset. {pull}22992[22992] -- Change types of numeric metrics from Kubelet summary api to double so as to cover big numbers. {pull}23335[23335] -- Add container.image.name and containe.name ECS fields for state_container. {pull}23802[23802] -- Add support for Consul 1.9. {pull}24123[24123] -- Add support for the MemoryPressure, DiskPressure, OutOfDisk and PIDPressure status conditions in state_node. {pull}23905[23905] -- Store `cloudfoundry.container.cpu.pct` in decimal form and as `scaled_float`. {pull}24219[24219] -- Remove `index_stats.created` field from Elasticsearch/index Metricset {pull}25113[25113] -- Adjust host fields to adopt new names from 1.9.0 ECS. {pull}24312[24312] -- Add replicas.ready field to state_statefulset in Kubernetes module{pull}26088[26088] - Fix Elasticsearch jvm.gc.collectors.old being exposed as young {issue}19636[19636] {pull}26616[26616] - Added `statsd.mappings` configuration for Statsd module {pull}26220[26220] - Added Airflow lightweight module {pull}26220[26220] - Add state_job metricset to Kubernetes module{pull}26479[26479] -- Recover service.address field in vsphere module {issue}26902[26902] {pull}26904[26904] *Packetbeat* @@ -138,7 +89,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d *Affecting all Beats* -- Fix events being dropped if they contain a floating point value of NaN or Inf. {pull}25051[25051] - Fix a race condition with the Kafka pipeline client, it is possible that `Close()` get called before `Connect()` . {issue}11945[11945] - Allow users to configure only `cluster_uuid` setting under `monitoring` namespace. {pull}14338[14338] - Update replicaset group to apps/v1 {pull}15854[15802] @@ -170,23 +120,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Add service resource in k8s cluster role. {pull}20546[20546] - [Metricbeat][Kubernetes] Change cluster_ip field from ip to keyword. {pull}20571[20571] - The `o365input` and `o365` module now recover from an authentication problem or other fatal errors, instead of terminating. {pull}21258[21258] -- Use PROGRAMDATA environment variable instead of C:\ProgramData for windows install service {pull}22874[22874] -- Fix reporting of cgroup metrics when running under Docker {pull}22879[22879] -- Fix typo in config docs {pull}23185[23185] -- Fix `nested` subfield handling in generated Elasticsearch templates. {issue}23178[23178] {pull}23183[23183] -- Fix CPU usage metrics on VMs with dynamic CPU config {pull}23154[23154] -- Fix panic due to unhandled DeletedFinalStateUnknown in k8s OnDelete {pull}23419[23419] -- Fix error loop with runaway CPU use when the Kafka output encounters some connection errors {pull}23484[23484] -- Allow configuring credential_profile_name and shared_credential_file when using role_arn. {pull}24174[24174] -- Add `expand_keys` to the list of permitted config fields for `decode_json_fields` {24862}[24862] -- Fix discovery of short-living and failing pods in Kubernetes autodiscover {issue}22718[22718] {pull}24742[24742] -- Fix panic when overwriting metadata {pull}24741[24741] -- Fix role_arn to work with access keys for AWS. {pull}25446[25446] -- Fix `community_id` processor so that ports greater than 65535 aren't valid. {pull}25409[25409] -- Fix ILM alias creation when write alias exists and initial index does not exist {pull}26143[26143] -- Omit full index template from errors that occur while loading the template. {pull}25743[25743] -- In the script processor, the `decode_xml` and `decode_xml_wineventlog` processors are now available as `DecodeXML` and `DecodeXMLWineventlog` respectively. -- Fix encoding errors when using the disk queue on nested data with multi-byte characters {pull}26484[26484] - Preserve annotations in a kubernetes namespace metadata {pull}27045[27045] - Allow conditional processing in `decode_xml` and `decode_xml_wineventlog`. {pull}27159[27159] @@ -198,47 +131,12 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - system/package: Fix an error that can occur while trying to persist package metadata. {issue}18536[18536] {pull}18887[18887] - system/socket: Fix dataset using 100% CPU and becoming unresponsive in some scenarios. {pull}19033[19033] {pull}19764[19764] - system/socket: Fixed tracking of long-running connections. {pull}19033[19033] -- system/package: Fix librpm loading on Fedora 31/32. {pull}NNNN[NNNN] -- file_integrity: Create fsnotify watcher only when starting file_integrity module {pull}19505[19505] -- auditd: Fix spelling of anomaly in `event.category`. -- auditd: Fix typo in `event.action` of `removed-user-role-from`. {pull}19300[19300] -- auditd: Fix typo in `event.action` of `used-suspicious-link`. {pull}19300[19300] -- system/socket: Fix kprobe grouping to allow running more than one instance. {pull}20325[20325] -- system/socket: Fixed a crash due to concurrent map read and write. {issue}21192[21192] {pull}21690[21690] -- file_integrity: stop monitoring excluded paths {issue}21278[21278] {pull}21282[21282] -- auditd: Fix an error condition causing a lot of `audit_send_reply` kernel threads being created. {pull}22673[22673] -- system/socket: Fixed start failure when run under config reloader. {issue}20851[20851] {pull}21693[21693] -- system/socket: Having some CPUs unavailable to Auditbeat could cause startup errors or event loss. {pull}22827[22827] -- Note incompatibility of system/socket on ARM. {pull}23381[23381] - -*Filebeat* - -- Fix mapping of fortinet.firewall.mem as integer. {pull}19335[19335] -- Ensure all zeek timestamps include millisecond precision. {issue}14599[14599] {pull}16766[16766] -- Fix s3 input hanging with GetObjectRequest API call by adding context_timeout config. {issue}15502[15502] {pull}15590[15590] -- Add shared_credential_file to cloudtrail config {issue}15652[15652] {pull}15656[15656] -- Fix typos in zeek notice fileset config file. {issue}15764[15764] {pull}15765[15765] -- Fix mapping error when zeek weird logs do not contain IP addresses. {pull}15906[15906] -- Improve `elasticsearch/audit` fileset to handle timestamps correctly. {pull}15942[15942] -- Prevent Elasticsearch from spewing log warnings about redundant wildcards when setting up ingest pipelines for the `elasticsearch` module. {issue}15840[15840] {pull}15900[15900] -- Fix mapping error for cloudtrail additionalEventData field {pull}16088[16088] -- Fix a connection error in httpjson input. {pull}16123[16123] -- Fix integer overflow in S3 offsets when collecting very large files. {pull}22523[22523] -- Fix CredentialsJSON unpacking for `gcp-pubsub` and `httpjson` inputs. {pull}23277[23277] -- Strip Azure Eventhub connection string in debug logs. {pulll}25066[25066] -- Fix o365 module config when client_secret contains special characters. {issue}25058[25058] -- Fix issue with m365_defender, when parsing incidents that has no alerts attached: {pull}25421[25421] -- Improve inode reuse handling by removing state for removed files more eagerly from the internal state table in the logs inputs. {pull}25756[25756] -- Fix default config template values for paths on oracle module: {pull}26276[26276] -- Change type of max_bytes in all configs to be cfgtype.ByteSize {pull}26699[26699] -- Fix Elasticsearch compatibility for modules that use `copy_from` in `set` processors. {issue}26629[26629] *Filebeat* - cisco/asa fileset: Fix parsing of 302021 message code. {pull}14519[14519] - Fix filebeat azure dashboards, event category should be `Alert`. {pull}14668[14668] - Fixed dashboard for Cisco ASA Firewall. {issue}15420[15420] {pull}15553[15553] -- Add shared_credential_file to cloudtrail config {issue}15652[15652] {pull}15656[15656] - Fix s3 input with cloudtrail fileset reading json file. {issue}16374[16374] {pull}16441[16441] - Add queue_url definition in manifest file for aws module. {pull}16640{16640} - Fixed various Cisco FTD parsing issues. {issue}16863[16863] {pull}16889[16889] @@ -279,57 +177,13 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Fix event.type for zeek/ssl and duplicate event.category for zeek/connection {pull}20696[20696] - Add json body check for sqs message. {pull}21727[21727] - Drop aws.vpcflow.pkt_srcaddr and aws.vpcflow.pkt_dstaddr when equal to "-". {pull}22721[22721] {issue}22716[22716] -- Fix cisco umbrella module config by adding input variable. {pull}22892[22892] -- Fix network.direction logic in zeek connection fileset. {pull}22967[22967] -- Fix aws s3 overview dashboard. {pull}23045[23045] -- Fix bad `network.direction` values in Fortinet/firewall fileset. {pull}23072[23072] -- Fix Cisco ASA/FTD module's parsing of WebVPN log message 716002. {pull}22966[22966] -- Add support for organization and custom prefix in AWS/CloudTrail fileset. {issue}23109[23109] {pull}23126[23126] -- Simplify regex for organization custom prefix in AWS/CloudTrail fileset. {issue}23203[23203] {pull}23204[23204] -- Fix syslog header parsing in infoblox module. {issue}23272[23272] {pull}23273[23273] -- Fix CredentialsJSON unpacking for `gcp-pubsub` and `httpjson` inputs. {pull}23277[23277] -- Fix concurrent modification exception in Suricata ingest node pipeline. {pull}23534[23534] -- Change the `event.created` in Netflow events to be the time the event was created by Filebeat - to be consistent with ECS. {pull}23094[23094] -- Fix Zoom module parameters for basic auth and url path. {pull}23779[23779] -- Fix handling of ModifiedProperties field in Office 365. {pull}23777[23777] -- Use rfc6587 framing for fortinet firewall and clientendpoint filesets when transferring over tcp. {pull}23837[23837] -- Fix httpjson input logging so it doesn't conflict with ECS. {pull}23972[23972] -- Fix Logstash module handling of logstash.log.log_event.action field. {issue}20709[20709] -- aws/s3access dataset was populating event.duration using the wrong unit. {pull}23920[23920] -- Zoom module pipeline failed to ingest some chat_channel events. {pull}23904[23904] -- Fix Netlow module issue with missing `internal_networks` config parameter. {issue}24094[24094] {pull}24110[24110] -- in httpjson input using encode_as "application/x-www-form-urlencoded" now sets Content-Type correctly {issue}24331[24331] {pull}24336[24336] -- Fix default `scope` in `add_nomad_metadata`. {issue}24559[24559] - Improve Cisco ASA/FTD parsing of messages - better support for identity FW messages. Change network.bytes, source.bytes, and destination.bytes to long from integer since value can exceed integer capacity. Add descriptions for various processors for easier pipeline editing in Kibana UI. {pull}23766[23766] -- Updating Oauth2 flow for m365_defender fileset. {pull}24829[24829] -- Fix usage of unallowed ECS event.outcome values in Cisco ASA/FTD pipeline. {pull}24744[24744]. -- Updating Oauth2 flow for m365_defender fileset. {pull}24829[24829] -- Fix IPtables Pipeline and Ubiquiti dashboard. {issue}24878[24878] {pull}24928[24928] -- Strip Azure Eventhub connection string in debug logs. {pulll}25066[25066] -- Change `checkpoint.source_object` from Long to Keyword. {issue}25124[25124] {pull}25145[25145] -- Fix s3 input when there is a blank line in the log file. {pull}25357[25357] -- Fix Nginx module pipelines. {issue}19088[19088] {pull}24699[24699] -- Remove space from field `sophos.xg.trans_src_ ip`. {issue}25154[25154] {pull}25250[25250] -- Fix `checkpoint.action_reason` when its a string, not a Long. {issue}25575[25575] {pull}25609[25609] -- Fix `fortinet.firewall.addr` when its a string, not an IP address. {issue}25585[25585] {pull}25608[25608] -- Fix incorrect field name appending to `related.hash` in `threatintel.abusechmalware` ingest pipeline. {issue}25151[25151] {pull}25674[25674] -- Add improvements to the azure activitylogs and platformlogs ingest pipelines. {pull}26148[26148] -- Fix `kibana.log` pipeline when `event.duration` calculation becomes a Long. {issue}24556[24556] {pull}25675[25675] -- Removed incorrect `http.request.referrer` field from `aws.elb` module. {issue}26435[26435] {pull}26441[26441] -- Fix `threatintel.indicator.url.full` not being populated. {issue}26351[26351] {pull}26508[26508] -- Fix Suricata metadata fields breaking visualizations, moved out of flattened datatype. {pull}26710[26710] -- Fix `httpjson` template data key for `url.params`. {pull}26848[26848] -- Cisco asa/ftd: Fix reversed usage of observer ingress and egress interfaces. {pull}26265[26265] -- Fix `aws.s3access` pipeline when remote IP is a `-`. {issue}26913[26913] {pull}26940[26940] -- Fix service name in aws-cloudwatch input from cloudwatchlogs to logs. {pull}27007[27007] *Heartbeat* - Fixed excessive memory usage introduced in 7.5 due to over-allocating memory for HTTP checks. {pull}15639[15639] - Fixed scheduler shutdown issues which would in rare situations cause a panic due to semaphore misuse. {pull}16397[16397] - Fixed TCP TLS checks to properly validate hostnames, this broke in 7.x and only worked for IP SANs. {pull}17549[17549] -- Add Context to otherwise ambiguous HTTP body read errors. {pull}25499[25499] *Journalbeat* @@ -384,39 +238,17 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Remove io.time from windows {pull}22237[22237] - Fix `logstash` module when `xpack.enabled: true` is set from emitting redundant events. {pull}22808[22808] - Change vsphere.datastore.capacity.used.pct value to betweeen 0 and 1. {pull}23148[23148] -- Fix incorrect types of fields GetHits and Ops in NodeInterestingStats for Couchbase module in Metricbeat {issue}21021[21021] {pull}23287[23287] -- Update config in `windows.yml` file. {issue}23027[23027]{pull}23327[23327] -- Add stack monitoring section to elasticsearch module documentation {pull}#23286[23286] -- Fix metric grouping for windows/perfmon module {issue}23489[23489] {pull}23505[23505] -- Fix ec2 metricset fields.yml and the integration test {pull}23726[23726] -- Unskip s3_request integration test. {pull}23887[23887] -- Add system.hostfs configuration option for system module. {pull}23831[23831] -- Fix GCP not able to request Cloudfunctions metrics if a region filter was set {pull}24218[24218] -- Fix type of `uwsgi.status.worker.rss` type. {pull}24468[24468] -- Accept text/plain type by default for prometheus client scraping. {pull}24622[24622] -- Use working set bytes to calculate the pod memory limit pct when memory usage is not reported (ie. Windows pods). {pull}25428[25428] -- Fix copy-paste error in libbeat docs. {pull}25448[25448] -- Fix azure billing dashboard. {pull}25554[25554] -- Major refactor of system/cpu and system/core metrics. {pull}25771[25771] -- Fix GCP Project ID being ingested as `cloud.account.id` in `gcp.billing` module {issue}26357[26357] {pull}26412[26412] -- Fix memory leak in SQL module when database is not available. {issue}25840[25840] {pull}26607[26607] -- Fix aws metric tags with resourcegroupstaggingapi paginator. {issue}26385[26385] {pull}26443[26443] -- Fix quoting in GCP billing table name {issue}26855[26855] {pull}26870[26870] - Allow metric prefix override per service in gcp module. {pull}26960[26960] *Packetbeat* - *Winlogbeat* -- Change `event.code` and `winlog.event_id` from int to keyword. {pull}25176[25176] -- Fix related.ip field in renameCommonAuthFields {pull}24892[24892] - Fix an issue with message template caching in the `wineventlog-experimental` API implementation. {pull}26826[26826] *Functionbeat* -- Expose region in AWS configuration so Functionbeat can deploy the Lamba in the correct place. {pull}26523[26523] *Elastic Logging Plugin* @@ -429,7 +261,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Allow a beat to ship monitoring data directly to an Elasticsearch monitoring cluster. {pull}9260[9260] - Updated go-seccomp-bpf library to v1.1.0 which updates syscall lists for Linux v5.0. {pull}11394[11394] - add_host_metadata is no GA. {pull}13148[13148] -- Add `providers` setting to `add_cloud_metadata` processor. {pull}13812[13812] - Ensure that init containers are no longer tailed after they stop {pull}14394[14394] - Fingerprint processor adds a new xxhash hashing algorithm {pull}15418[15418] - Add configuration for APM instrumentation and expose the tracer trough the Beat object. {pull}17938[17938] @@ -453,21 +284,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Add capability of enriching process metadata with contianer id also for non-privileged containers in `add_process_metadata` processor. {pull}19767[19767] - Add replace_fields config option in add_host_metadata for replacing host fields. {pull}20490[20490] {issue}20464[20464] - Add option to select the type of index template to load: legacy, component, index. {pull}21212[21212] -- Add `wineventlog` schema to `decode_xml` processor. {issue}23910[23910] {pull}24726[24726] -- Add new ECS 1.9 field `cloud.service.name` to `add_cloud_metadata` processor. {pull}24993[24993] -- Libbeat: report queue capacity, output batch size, and output client count to monitoring. {pull}24700[24700] -- Add kubernetes.pod.ip field in kubernetes metadata. {pull}25037[25037] -- Discover changes in Kubernetes namespace metadata as soon as they happen. {pull}25117[25117] -- Add `decode_xml_wineventlog` processor. {issue}23910[23910] {pull}25115[25115] -- Add support for defining explicitly named dynamic templates without path/type match criteria {pull}25422[25422] -- Add new setting `gc_percent` for tuning the garbage collector limits via configuration file. {pull}25394[25394] -- Add `unit` and `metric_type` properties to fields.yml for populating field metadata in Elasticsearch templates {pull}25419[25419] -- Add new option `suffix` to `logging.files` to control how log files are rotated. {pull}25464[25464] -- Validate that required functionality in Elasticsearch is available upon initial connection. {pull}25351[25351] -- Improve ES output error insights. {pull}25825[25825] -- Add orchestrator.cluster.name/url fields as k8s metadata {pull}26056[26056] -- Libbeat: report beat version to monitoring. {pull}26214[26214] -- Ensure common proxy settings support in HTTP clients: proxy_disabled, proxy_url, proxy_headers and typical environment variables HTTP_PROXY, HTTPS_PROXY, NOPROXY. {pull}25219[25219] - Add proxy support for AWS functions. {pull}26832[26832] *Auditbeat* @@ -480,7 +296,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d *Filebeat* - - `container` and `docker` inputs now support reading of labels and env vars written by docker JSON file logging driver. {issue}8358[8358] - Add `index` option to all inputs to directly set a per-input index value. {pull}14010[14010] - Add ECS tls fields to zeek:smtp,rdp,ssl and aws:s3access,elb {issue}15757[15757] {pull}15935[15936] @@ -560,26 +375,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Added DNS response IP addresses to `related.ip` in Suricata module. {pull}22291[22291] - Added support for first_event context in filebeat httpjson input {pull}23437[23437] - Added feature to modules to adapt Ingest Node pipelines for compatibility with older Elasticsearch versions by - removing unsupported processors. {pull}23763[23763] -- Added support for Cisco AMP API as a new fileset. {pull}22768[22768] -- Added RFC6587 framing option for tcp and unix inputs {issue}23663[23663] {pull}23724[23724] -- Added `application/x-ndjson` as decode option for httpjson input {pull}23521[23521] -- Added `application/x-www-form-urlencoded` as encode option for httpjson input {pull}23521[23521] -- Move aws-s3 input to GA. {pull}23631[23631] -- Populate `source.mac` and `destination.mac` for Suricata EVE events. {issue}23706[23706] {pull}23721[23721] -- Added string splitting for httpjson input {pull}24022[24022] -- Added Signatures fileset to Zeek module {pull}23772[23772] -- Upgrade Cisco ASA/FTD/Umbrella to ECS 1.8.0. {pull}23819[23819] -- Add new ECS user and categories features to google_workspace/gsuite {issue}23118[23118] {pull}23709[23709] -- Move crowdstrike JS processor to ingest pipelines and upgrade to ECS 1.8.0 {issue}23118[23118] {pull}23875[23875] -- Update Filebeat auditd dataset to ECS 1.8.0. {pull}23723[23723] {issue}23118[23118] -- Updated microsoft defender_atp and m365_defender to ECS 1.8. {pull}23897[23897] {issue}23118[23118] -- Updated o365 module to ECS 1.8. {issue}23118[23118] {pull}23896[23896] -- Upgrade CEF module to ECS 1.8.0. {pull}23832[23832] -- Upgrade fortinet/firewall to ECS 1.8 {issue}23118[23118] {pull}23902[23902] -- Upgrade Zeek to ECS 1.8.0. {issue}23118[23118] {pull}23847[23847] -- Updated azure module to ECS 1.8. {issue}23118[23118] {pull}23927[23927] -- Update aws/s3access to ECS 1.8. {issue}23118[23118] {pull}23920[23920] - Upgrade panw module to ecs 1.8 {issue}23118[23118] {pull}23931[23931] - Upgrade juniper/srx to ecs 1.8.0. {issue}23118[23118] {pull}23936[23936] - Upgrade okta to ecs 1.8.0 and move js processor to ingest pipeline {issue}23118[23118] {pull}23929[23929] @@ -638,14 +433,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d *Heartbeat* - Bundle synthetics deps with heartbeat docker image. {pull}23274[23274] -- Add support for `copytruncate` method when rotating input logs with an external tool in `filestream` input. {pull}23457[23457] -- Add `proxy_headers` to HTTP monitor. {pull}25219[25219] - -*Heartbeat* - -- Suppress too many bad message error logs when reading from corrupted journal for 5 seconds. {pull}26224[26224] - -*Heartbeat* *Journalbeat* @@ -704,53 +491,20 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d - Add dashboard for pubsub metricset in googlecloud module. {pull}21326[21326] {issue}17137[17137] - Enrich events of `state_service` metricset with kubernetes services' metadata. {pull}23730[23730] - Check fields are documented in aws metricsets. {pull}23887[23887] -- Add support for defining metrics_filters for prometheus module in hints. {pull}24264[24264] -- Add support for PostgreSQL 10, 11, 12 and 13. {pull}24402[24402] -- Add support for SASL/SCRAM authentication to the Kafka module. {pull}24810[24810] -- Refactor state_* metricsets to share response from endpoint. {pull}25640[25640] -- Add server id to zookeeper events. {pull}25550[25550] -- Add additional network metrics to docker/network {pull}25354[25354] -- Migrate ec2 metricsets to use cloudwatch input. {pull}25924[25924] -- Reduce number of requests done by kubernetes metricsets to kubelet. {pull}25782[25782] -- Migrate rds metricsets to use cloudwatch input. {pull}26077[26077] -- Migrate sqs metricsets to use cloudwatch input. {pull}26117[26117] -- Collect linked account information in AWS billing. {pull}26285[26285] -- Add total CPU to vSphere virtual machine metrics. {pull}26167[26167] -- Add AWS Kinesis metricset. {pull}25989[25989] - Move openmetrics module to oss. {pull}26561[26561] - Fix release state of kubernetes metricsets. {pull}26864[26864] -- Add Cluster filter on ECS Kubernetes overview dashboard and corresponding section on Kubernetes module documentation page. {pull}26919[26919] *Packetbeat* -- Add an example to packetbeat.yml of using the `forwarded` tag to disable - `host` metadata fields when processing network data from network tap or mirror - port. {pull}19209[19209] -- Add ECS fields for x509 certs, event categorization, and related IP info. {pull}19167[19167] -- Add 100-continue support {issue}15830[15830] {pull}19349[19349] -- Add initial SIP protocol support {pull}21221[21221] -- Add support for overriding the published index on a per-protocol/flow basis. {pull}22134[22134] -- Change build process for x-pack distribution {pull}21979[21979] -- Tuned the internal queue size to reduce the chances of events being dropped. {pull}22650[22650] -- Add support for "http.request.mime_type" and "http.response.mime_type". {pull}22940[22940] -- Upgrade to ECS 1.8.0. {pull}23783[23783] -- Add `event.type: [connection]` to flow events and include `end` for final flows. {pull}24564[24564] -- Add `url.extension` to HTTP events {issue}25990[25990] {pull}25999[25999] - *Functionbeat* -*Heartbeat* - - *Winlogbeat* - Set process.command_line and process.parent.command_line from Sysmon Event ID 1. {pull}17327[17327] - Add support for event IDs 4673,4674,4697,4698,4699,4700,4701,4702,4768,4769,4770,4771,4776,4778,4779,4964 to the Security module {pull}17517[17517] - Add registry and code signature information and ECS categorization fields for sysmon module {pull}18058[18058] -- Add support for sysmon v13 events 24 and 25. {issue}24217[24217] {pull}24945[24945] -- Changed the log level of the "Successfully published events" message from `info` to `debug` to reduce verbosity of the `info` logging level. To track event log reader activity use the `published_events` metric. {pull}25617[25617] *Elastic Log Driver* @@ -763,7 +517,6 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d *Filebeat* -- Deprecate the MISP module. The Threat Intel module should be used instead. {issue}25240[25240] *Heartbeat* diff --git a/libbeat/docs/release.asciidoc b/libbeat/docs/release.asciidoc index 027d737a888..26c9265f8da 100644 --- a/libbeat/docs/release.asciidoc +++ b/libbeat/docs/release.asciidoc @@ -8,6 +8,7 @@ This section summarizes the changes in each release. Also read <> for more detail about changes that affect upgrade. +* <> * <> * <> * <>