Merge branch 'master' into archive-system-tests

* master: (25 commits) fix: Force PLATFORMS environment variable when we build Elastic Agent dependencies on arm64 (elastic#26415) macos for metricbeat to run in the extended meta-stage (elastic#26573) Packaging: add arm7 platform in the main pipeline (elastic#26575) [Heartbeat] Skip flakey timer queue test (elastic#26592) Update to "read_pipeline" permission (elastic#26465) (elastic#26580) API keys do not reflect the need for read_pipeline (elastic#26466) (elastic#26582) Add Fleet agent.id to Agent monitoring data (elastic#26548) Add kinesis metricset (elastic#25989) Refactor of system/memory metricset (elastic#26334) Introduce httpcommon package in libbeat (add support for Proxy) (elastic#25219) [Filebeat] change multiline configuration in awss3 input to parsers (elastic#25873) docs: Hint for the error "Error extracting container id" (elastic#25824) [Docs] Fixed metricbeat redis exported field CPU descriptions (elastic#25846) (elastic#26496) Update urllib to 1.26.5. (elastic#26380) Update golang.org/x/crypto (elastic#26448) [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816) Move parsers outside of filestream input so others can use them as well (elastic#26541) [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508) [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620) Logging code cleanup related to Nomad auto-discovery (elastic#26498) ...
mdelapenya · Jun 30, 2021 · 218ee9a · 218ee9a
2 parents d4a87cb + 5488dcf
commit 218ee9a
Show file tree

Hide file tree

Showing 365 changed files with 17,245 additions and 4,303 deletions.
diff --git a/CHANGELOG.next.asciidoc b/CHANGELOG.next.asciidoc
@@ -104,6 +104,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Fix parsing issues with nested JSON payloads in Elasticsearch audit log fileset. {pull}22975[22975]
 - Rename `network.direction` values in crowdstrike/falcon to `ingress`/`egress`. {pull}23041[23041]
 - Change logging in logs input to structure logging. Some log message formats have changed. {pull}25299[25299]
+- Change source field for `event.action` in `fortinet.firewall` module to `fortinet.firewall.action` instead of `fortinet.firewall.eventtype`. {pull}24816[24816]
 
 *Heartbeat*
 - Add support for screenshot blocks and use newer synthetics flags that only works in newer synthetics betas. {pull}25808[25808]
@@ -388,6 +389,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add improvements to the azure activitylogs and platformlogs ingest pipelines. {pull}26148[26148]
 - Fix `kibana.log` pipeline when `event.duration` calculation becomes a Long. {issue}24556[24556] {pull}25675[25675]
 - Removed incorrect `http.request.referrer` field from `aws.elb` module. {issue}26435[26435] {pull}26441[26441]
+- Fix `threatintel.indicator.url.full` not being populated. {issue}26351[26351] {pull}26508[26508]
 
 *Heartbeat*
 
@@ -596,6 +598,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Improve ES output error insights. {pull}25825[25825]
 - Add orchestrator.cluster.name/url fields as k8s metadata {pull}26056[26056]
 - Libbeat: report beat version to monitoring. {pull}26214[26214]
+- Ensure common proxy settings support in HTTP clients: proxy_disabled, proxy_url, proxy_headers and typical environment variables HTTP_PROXY, HTTPS_PROXY, NOPROXY. {pull}25219[25219]
 
 *Auditbeat*
 
@@ -811,8 +814,9 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Update PanOS module to parse Global Protect & User ID logs. {issue}24722[24722] {issue}24724[24724] {pull}24927[24927]
 - Add HMAC signature validation support for http_endpoint input. {pull}24918[24918]
 - Add new grok pattern for iptables module for Ubiquiti UDM {issue}25615[25615] {pull}25616[25616]
-- Add multiline support to aws-s3 input. {issue}25249[25249] {pull}25710[25710]
+- Add multiline support to aws-s3 input. {issue}25249[25249] {pull}25710[25710] {pull}25873[25873]
 - Add monitoring metrics to the `aws-s3` input. {pull}25711[25711]
+- Added `network.direction` fields to Zeek and Suricata modules using the `add_network_direction` processor {pull}24620[24620]
 - Add Content-Type override to aws-s3 input. {issue}25697[25697] {pull}25772[25772]
 - In Cisco Umbrella fileset add users from cisco.umbrella.identities to related.user. {pull}25776[25776]
 - Add fingerprint processor to generate fixed ids for `google_workspace` events. {pull}25841[25841]
@@ -837,10 +841,12 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add support for `copytruncate` method when rotating input logs with an external tool in `filestream` input. {pull}23457[23457]
 - Add `uri_parts` and `user_agent` ingest processors to `aws.elb` module. {issue}26435[26435] {pull}26441[26441]
 - Added dataset `recordedfuture` to the `threatintel` module to ingest indicators from Recorded Future Connect API {pull}26481[26481]
+- Update `fortinet` ingest pipelines. {issue}22136[22136] {issue}25254[25254] {pull}24816[24816]
 
 *Heartbeat*
 
 - Add mime type detection for http responses. {pull}22976[22976]
+- Add `proxy_headers` to HTTP monitor. {pull}25219[25219]
 
 *Journalbeat*
 
@@ -974,6 +980,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Migrate sqs metricsets to use cloudwatch input. {pull}26117[26117]
 - Collect linked account information in AWS billing. {pull}26285[26285]
 - Add total CPU to vSphere virtual machine metrics. {pull}26167[26167]
+- Add AWS Kinesis metricset. {pull}25989[25989]
 
 *Packetbeat*
 

diff --git a/Jenkinsfile b/Jenkinsfile
@@ -353,9 +353,9 @@ def packagingLinux(Map args = [:]) {
                 'linux/amd64',
                 'linux/386',
                 'linux/arm64',
+                'linux/armv7',
                 // The platforms above are disabled temporarly as crossbuild images are
                 // not available. See: https://github.com/elastic/golang-crossbuild/issues/71
-                //'linux/armv7',
                 //'linux/ppc64le',
                 //'linux/mips64',
                 //'linux/s390x',

diff --git a/NOTICE.txt b/NOTICE.txt
diff --git a/dev-tools/cmd/dashboards/export_dashboards.go b/dev-tools/cmd/dashboards/export_dashboards.go
@@ -29,6 +29,7 @@ import (
 
 	"github.com/pkg/errors"
 
+	"github.com/elastic/beats/v7/libbeat/common/transport/httpcommon"
 	"github.com/elastic/beats/v7/libbeat/dashboards"
 	"github.com/elastic/beats/v7/libbeat/kibana"
 )
@@ -64,14 +65,18 @@ func main() {
 		user = u.User.Username()
 		pass, _ = u.User.Password()
 	}
+
+	transport := httpcommon.DefaultHTTPTransportSettings()
+	transport.Timeout = kibanaTimeout
+
 	client, err := kibana.NewClientWithConfig(&kibana.ClientConfig{
-		Protocol: u.Scheme,
-		Host:     u.Host,
-		Username: user,
-		Password: pass,
-		Path:     u.Path,
-		SpaceID:  *spaceID,
-		Timeout:  kibanaTimeout,
+		Protocol:  u.Scheme,
+		Host:      u.Host,
+		Username:  user,
+		Password:  pass,
+		Path:      u.Path,
+		SpaceID:   *spaceID,
+		Transport: transport,
 	})
 	if err != nil {
 		log.Fatalf("Error while connecting to Kibana: %v", err)

diff --git a/dev-tools/packaging/templates/docker/Dockerfile.tmpl b/dev-tools/packaging/templates/docker/Dockerfile.tmpl
@@ -47,7 +47,7 @@ ENV NODE_PATH={{ $beatHome }}/.node
 RUN echo \
     $NODE_PATH \
     {{ $beatHome }}/.config \
-    {{ $beatHome }}/suites \
+    {{ $beatHome }}/.synthetics \
     {{ $beatHome }}/.npm \
     {{ $beatHome }}/.cache \
     | xargs -IDIR sh -c 'mkdir -p DIR && chmod 0770 DIR'

diff --git a/filebeat/autodiscover/builder/hints/logs.go b/filebeat/autodiscover/builder/hints/logs.go
@@ -52,55 +52,48 @@ var validModuleNames = regexp.MustCompile("[^a-zA-Z0-9\\_\\-]+")
 type logHints struct {
 	config   *config
 	registry *fileset.ModuleRegistry
+	log      *logp.Logger
 }
 
 // NewLogHints builds a log hints builder
 func NewLogHints(cfg *common.Config) (autodiscover.Builder, error) {
 	config := defaultConfig()
-	err := cfg.Unpack(&config)
-
-	if err != nil {
-		return nil, fmt.Errorf("unable to unpack hints config due to error: %v", err)
+	if err := cfg.Unpack(&config); err != nil {
+		return nil, fmt.Errorf("unable to unpack hints config due to error: %w", err)
 	}
 
 	moduleRegistry, err := fileset.NewModuleRegistry(nil, beat.Info{}, false)
 	if err != nil {
 		return nil, err
 	}
 
-	return &logHints{&config, moduleRegistry}, nil
+	return &logHints{&config, moduleRegistry, logp.NewLogger("hints.builder")}, nil
 }
 
 // Create config based on input hints in the bus event
 func (l *logHints) CreateConfig(event bus.Event, options ...ucfg.Option) []*common.Config {
 	var hints common.MapStr
-	hIface, ok := event["hints"]
-	if ok {
-		hints, _ = hIface.(common.MapStr)
-	}
-
-	inputConfig := l.getInputsConfigs(hints)
-
-	// If default config is disabled return nothing unless it's explicty enabled
-	if !l.config.DefaultConfig.Enabled() && !builder.IsEnabled(hints, l.config.Key) {
-		logp.Debug("hints.builder", "default config is disabled: %+v", event)
-		return []*common.Config{}
+	if hintsIfc, found := event["hints"]; found {
+		hints, _ = hintsIfc.(common.MapStr)
 	}
 
-	// If explictly disabled, return nothing
-	if builder.IsDisabled(hints, l.config.Key) {
-		logp.Debug("hints.builder", "logs disabled by hint: %+v", event)
-		return []*common.Config{}
+	// Hint must be explicitly enabled when default_config sets enabled=false.
+	if !l.config.DefaultConfig.Enabled() && !builder.IsEnabled(hints, l.config.Key) ||
+		builder.IsDisabled(hints, l.config.Key) {
+		l.log.Debugw("Hints config is not enabled.", "autodiscover.event", event)
+		return nil
 	}
 
-	if inputConfig != nil {
-		configs := []*common.Config{}
+	if inputConfig := l.getInputsConfigs(hints); inputConfig != nil {
+		var configs []*common.Config
 		for _, cfg := range inputConfig {
 			if config, err := common.NewConfigFrom(cfg); err == nil {
 				configs = append(configs, config)
+			} else {
+				l.log.Warnw("Failed to create config from input.", "error", err)
 			}
 		}
-		logp.Debug("hints.builder", "generated config %+v", configs)
+		l.log.Debugf("Generated %d input configs from hint.", len(configs))
 		// Apply information in event to the template to generate the final config
 		return template.ApplyConfigTemplate(event, configs)
 	}

diff --git a/filebeat/autodiscover/builder/hints/logs_test.go b/filebeat/autodiscover/builder/hints/logs_test.go
@@ -22,6 +22,7 @@ import (
 	"testing"
 
 	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
 
 	"github.com/elastic/beats/v7/libbeat/common"
 	"github.com/elastic/beats/v7/libbeat/common/bus"
@@ -692,9 +693,9 @@ func TestGenerateHints(t *testing.T) {
 	for _, test := range tests {
 		// Configure path for modules access
 		abs, _ := filepath.Abs("../../..")
-		err := paths.InitPaths(&paths.Path{
+		require.NoError(t, paths.InitPaths(&paths.Path{
 			Home: abs,
-		})
+		}))
 
 		l, err := NewLogHints(test.config)
 		if err != nil {
@@ -927,17 +928,17 @@ func TestGenerateHintsWithPaths(t *testing.T) {
 
 		// Configure path for modules access
 		abs, _ := filepath.Abs("../../..")
-		err := paths.InitPaths(&paths.Path{
+		require.NoError(t, paths.InitPaths(&paths.Path{
 			Home: abs,
-		})
+		}))
 
 		l, err := NewLogHints(cfg)
 		if err != nil {
 			t.Fatal(err)
 		}
 
 		cfgs := l.CreateConfig(test.event)
-		assert.Equal(t, test.len, len(cfgs), test.msg)
+		require.Equal(t, test.len, len(cfgs), test.msg)
 		if test.len != 0 {
 			config := common.MapStr{}
 			err := cfgs[0].Unpack(&config)

diff --git a/filebeat/docs/autodiscover-hints.asciidoc b/filebeat/docs/autodiscover-hints.asciidoc
@@ -259,7 +259,9 @@ You can label Docker containers with useful info to decode logs structured as JS
 [float]
 ==== Nomad
 
-Nomad autodiscover provider supports hints using the https://www.nomadproject.io/docs/job-specification/meta.html[`meta` stanza]. To enable it just set `hints.enabled`:
+Nomad autodiscover provider supports hints using the
+https://www.nomadproject.io/docs/job-specification/meta.html[`meta` stanza]. To
+enable it just set `hints.enabled`:
 
 [source,yaml]
 -----
@@ -269,7 +271,8 @@ filebeat.autodiscover:
       hints.enabled: true
 -----
 
-You can configure the default config that will be launched when a new job is seen, like this:
+You can configure the default config that will be launched when a new job is
+seen, like this:
 
 [source,yaml]
 -----
@@ -278,31 +281,81 @@ filebeat.autodiscover:
     - type: nomad
       hints.enabled: true
       hints.default_config:
-        type: nomad
+        type: log
         paths:
-          - /var/lib/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.*
+          - /opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.*
 -----
 
-You can also disable default settings entirely, so only Jobs annotated like `co.elastic.logs/enabled: true`
-will be retrieved:
+You can also disable the default config such that only logs from jobs explicitly
+annotated with `"co.elastic.logs/enabled" = "true"` will be collected:
 
 [source,yaml]
 -----
 filebeat.autodiscover:
   providers:
     - type: nomad
       hints.enabled: true
-      hints.default_config.enabled: false
+      hints.default_config:
+        enabled: false
+        type: log
+        paths:
+          - /opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.*
 -----
 
-You can annotate Nomad Jobs using the `meta` stanza with useful info to spin up {beatname_uc} inputs
-or modules:
+You can annotate Nomad Jobs using the `meta` stanza with useful info to spin up
+{beatname_uc} inputs or modules:
 
 [source,hcl]
 -----
 meta {
-  "co.elastic.logs/multiline.pattern" = "^\["
-  "co.elastic.logs/multiline.negate"  = true
-  "co.elastic.logs/multiline.match"   = after
+  "co.elastic.logs/enabled"           = "true"
+  "co.elastic.logs/multiline.pattern" = "^\\["
+  "co.elastic.logs/multiline.negate"  = "true"
+  "co.elastic.logs/multiline.match"   = "after"
 }
 -----
+
+If you are using autodiscover then in most cases you will want to use the
+<<add-nomad-metadata,`add_nomad_metadata`>> processor to enrich events with
+Nomad metadata. This example configures {{beatname_uc}} to connect to the local
+Nomad agent over HTTPS and adds the Nomad allocation ID to all events from the
+input. Later in the pipeline the `add_nomad_metadata` processor will use that ID
+to enrich the event.
+
+[source,yaml]
+-----
+filebeat.autodiscover:
+  providers:
+    - type: nomad
+      address: https://localhost:4646
+      hints.enabled: true
+      hints.default_config:
+        enabled: false <1>
+        type: log
+        paths:
+          - /opt/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.*
+        processors:
+          - add_fields: <2>
+              target: nomad
+              fields:
+                allocation.id: ${data.nomad.allocation.id}
+
+processors:
+  - add_nomad_metadata: <3>
+      when.has_fields.fields: [nomad.allocation.id]
+      address: https://localhost:4646
+      default_indexers.enabled: false
+      default_matchers.enabled: false
+      indexers:
+        - allocation_uuid:
+      matchers:
+        - fields:
+            lookup_fields:
+              - 'nomad.allocation.id'
+-----
+<1> The default config is disabled meaning any task without the
+`"co.elastic.logs/enabled" = "true"` metadata will be ignored.
+<2> The `add_fields` processor populates the `nomad.allocation.id` field with
+the Nomad allocation UUID.
+<3> The `add_nomad_metadata` processor is configured at the global level so
+that it is only instantiated one time which saves resources.
diff --git a/filebeat/docs/autodiscover-nomad-config.asciidoc b/filebeat/docs/autodiscover-nomad-config.asciidoc
@@ -43,6 +43,6 @@ filebeat.autodiscover:
                     - /var/lib/nomad/alloc/${data.nomad.allocation.id}/alloc/logs/${data.nomad.task.name}.*
 -------------------------------------------------------------------------------------
 
-WARNING: The `docker` input is currently not supported. Nomad doesn't expose the container id
-associated with the allocation. Without the container id, there is no way of generating the proper
+WARNING: The `docker` input is currently not supported. Nomad doesn't expose the container ID
+associated with the allocation. Without the container ID, there is no way of generating the proper
 path for reading the container's logs.
diff --git a/filebeat/docs/faq.asciidoc b/filebeat/docs/faq.asciidoc
@@ -5,6 +5,13 @@ This section describes common problems you might encounter with
 {beatname_uc}. Also check out the
 https://discuss.elastic.co/c/beats/{beatname_lc}[{beatname_uc} discussion forum].
 
+[[filebeat-kubernetes-metadata-error-extracting-container-id]]
+=== Error extracting container id while using Kubernetes metadata
+
+The `add_kubernetes_metadata` processor might throw the error `Error extracting container id - source value does not contain matcher's logs_path`.
+There might be some issues with the matchers definitions or the location of `logs_path`.
+Please verify the Kubernetes pod is healthy.
+
 [[filebeat-network-volumes]]
 === Can't read log files from network volumes
 

diff --git a/filebeat/docs/fields.asciidoc b/filebeat/docs/fields.asciidoc
@@ -62883,6 +62883,16 @@ type: keyword
 ESP Transform
 
 
+type: keyword
+
+--
+
+*`fortinet.firewall.eventtype`*::
++
+--
+UTM Event Type
+
+
 type: keyword
 
 --
@@ -65363,6 +65373,16 @@ type: integer
 Security action performed by UTM
 
 
+type: keyword
+
+--
+
+*`fortinet.firewall.utmref`*::
++
+--
+Reference to UTM
+
+
 type: keyword
 
 --