Merge remote-tracking branch 'upstream/master' into feature/windows-2016

* upstream/master: (26 commits)
  [Ingest Manager] Send updating state (elastic#21461)
  [Filebeat][New Fileset] Cisco Umbrella support (elastic#21504)
  [Ingest Manager] Download asc from artifact store specified in spec (elastic#21488)
  Implementation of fileProspector (elastic#21479)
  [Metricbeat] Add latency config option into aws module (elastic#20875)
  Skip filestream flaky tests (elastic#21490)
  Ignore unsupported metrics in the azure module (elastic#21486)
  Do not run symlink tests on Windows (elastic#21472)
  Map `cloud.account.id` to azure sub id (elastic#21483)
  Add support for app_state metricset (elastic#20639)
  Include original error when metricbeat fails to connect with Kafka (elastic#21484)
  Prompt only when agent is already enrolled (elastic#21473)
  Fix leftover delpoyment example (elastic#21474)
  Bump version to ECS 1.6 in modules without ECS updates (elastic#21455)
  Clarify input type configuration options (elastic#19284)
  Increase index pattern size check to 10MiB (elastic#21487)
  Migrate S3 Input to Filebeat Input V2 (elastic#20005)
  [libbeat] Add configurable exponential backoff for disk queue write errors (elastic#21493)
  Revert "Revert "[JJBB] Set shallow cloning to 10 (elastic#21409)" (elastic#21447)" (elastic#21467)
  Fix format of debug messages in tlscommon (elastic#21482)
  ...

.ci/jobs/apm-beats-update.yml

@@ -48,7 +48,7 @@
             before: true
         prune: true
         shallow-clone: true
-        depth: 3
+        depth: 10
         do-not-fetch-tags: true
         submodule:
             disable: false

.ci/jobs/beats-tester.yml

@@ -44,7 +44,7 @@
               before: true
           prune: true
           shallow-clone: true
-          depth: 3
+          depth: 10
           do-not-fetch-tags: true
           submodule:
               disable: false

.ci/jobs/beats.yml

@@ -46,7 +46,7 @@
             before: true
         prune: true
         shallow-clone: true
-        depth: 3
+        depth: 10
         do-not-fetch-tags: true
         submodule:
             disable: false

.ci/jobs/golang-crossbuild-mbp.yml

@@ -31,7 +31,7 @@
             before: true
           prune: true
           shallow-clone: true
-          depth: 4
+          depth: 10
           do-not-fetch-tags: true
           submodule:
             disable: false

.ci/jobs/packaging.yml

@@ -44,7 +44,7 @@
               before: true
           prune: true
           shallow-clone: true
-          depth: 3
+          depth: 10
           do-not-fetch-tags: true
           submodule:
               disable: false

CHANGELOG.next.asciidoc

@@ -349,6 +349,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add missing info about the rest of the azure metricsets in the documentation. {pull}19601[19601]
 - Fix k8s scheduler compatibility issue. {pull}19699[19699]
 - Fix SQL module mapping NULL values as string {pull}18955[18955] {issue}18898[18898
+- Add support for azure light metricset app_stats. {pull}20639[20639]
 - Fix ec2 disk and network metrics to use Sum statistic method. {pull}20680[20680]
 - Fill cloud.account.name with accountID if account alias doesn't exist. {pull}20736[20736]
 - The Kibana collector applies backoff when errored at getting usage stats {pull}20772[20772]
@@ -603,6 +604,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Always attempt community_id processor on zeek module {pull}21155[21155]
 - Add related.hosts ecs field to all modules {pull}21160[21160]
 - Keep cursor state between httpjson input restarts {pull}20751[20751]
+- Convert aws s3 to v2 input {pull}20005[20005]
 
 *Heartbeat*
 
@@ -722,13 +724,16 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
 - Add billing data collection from Cost Explorer into aws billing metricset. {pull}20527[20527] {issue}20103[20103]
 - Migrate `compute_vm` metricset to a light one, map `cloud.instance.id` field. {pull}20889[20889]
 - Request prometheus endpoints to be gzipped by default {pull}20766[20766]
+- Add latency config parameter into aws module. {pull}20875[20875]
 - Release all kubernetes `state` metricsets as GA {pull}20901[20901]
 - Add billing metricset into googlecloud module. {pull}20812[20812] {issue}20738[20738]
 - Move `compute_vm_scaleset` to light metricset. {pull}21038[21038] {issue}20985[20985]
 - Sanitize `event.host`. {pull}21022[21022]
 - Add overview and platform health dashboards to Cloud Foundry module. {pull}21124[21124]
 - Release lambda metricset in aws module as GA. {issue}21251[21251] {pull}21255[21255]
 - Add dashboard for pubsub metricset in googlecloud module. {pull}21326[21326] {issue}17137[17137]
+- Expand unsupported option from namespace to metrics in the azure module. {pull}21486[21486]
+- Map cloud data filed `cloud.account.id` to azure subscription.  {pull}21483[21483] {issue}21381[21381]
 
 *Packetbeat*
 

Jenkinsfile

@@ -24,6 +24,7 @@ pipeline {
     PIPELINE_LOG_LEVEL = 'INFO'
     PYTEST_ADDOPTS = "${params.PYTEST_ADDOPTS}"
     RUNBLD_DISABLE_NOTIFICATIONS = 'true'
+    SLACK_CHANNEL = "#beats-ci-builds"
     TERRAFORM_VERSION = "0.12.24"
     XPACK_MODULE_PATTERN = '^x-pack\\/[a-z0-9]+beat\\/module\\/([^\\/]+)\\/.*'
   }
@@ -121,7 +122,7 @@ pipeline {
       runbld(stashedTestReports: stashedTestReports, project: env.REPO)
     }
     cleanup {
-      notifyBuildResult(prComment: true)
+      notifyBuildResult(prComment: true, slackComment: true)
     }
   }
 }

Jenkinsfile.yml

@@ -28,11 +28,13 @@ changeset:
         - "^\\.ci/scripts/.*"
     oss:
         - "^go.mod"
+        - "^pytest.ini"
         - "^dev-tools/.*"
         - "^libbeat/.*"
         - "^testing/.*"
     xpack:
         - "^go.mod"
+        - "^pytest.ini"
         - "^dev-tools/.*"
         - "^libbeat/.*"
         - "^testing/.*"

auditbeat/cmd/root.go

@@ -35,7 +35,7 @@ const (
 	Name = "auditbeat"
 
 	// ecsVersion specifies the version of ECS that Auditbeat is implementing.
-	ecsVersion = "1.5.0"
+	ecsVersion = "1.6.0"
 )
 
 // RootCmd for running auditbeat.

dev-tools/dependencies-report

@@ -48,7 +48,7 @@ go list -m -json all $@ | go run go.elastic.co/go-licence-detector \
 # name,url,version,revision,license
 ubi8url='https://catalog.redhat.com/software/containers/ubi8/ubi-minimal/5c359a62bed8bd75a2c3fba8'
 ubi8source='https://oss-dependencies.elastic.co/redhat/ubi/ubi-minimal-8-source.tar.gz'
-ubilicense='Custom;https://www.redhat.com/licenses/EULA_Red_Hat_Universal_Base_Image_English_20190422.pdf,https://oss-dependencies.elastic.co/redhat/ubi/ubi-minimal-8-source.tar.gz'
+ubilicense='Custom;https://www.redhat.com/licenses/EULA_Red_Hat_Universal_Base_Image_English_20190422.pdf'
 cat <<EOF >> $outfile
 Red Hat Universal Base Image,$ubi8url,8,,$ubilicense,$ubi8source
 EOF

filebeat/docs/fields.asciidoc

@@ -26581,6 +26581,153 @@ type: keyword
 --
 This key captures values or decorators used within a registry entry
 
+type: keyword
+
+--
+
+[float]
+=== cisco.umbrella
+
+Fields for Cisco Umbrella.
+
+
+
+*`cisco.umbrella.identities`*::
++
+--
+An array of the different identities related to the event.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.categories`*::
++
+--
+The security or content categories that the destination matches.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.policy_identity_type`*::
++
+--
+The first identity type matched with this request. Available in version 3 and above.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.identity_types`*::
++
+--
+The type of identity that made the request. For example, Roaming Computer or Network.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.blocked_categories`*::
++
+--
+The categories that resulted in the destination being blocked. Available in version 4 and above.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.content_type`*::
++
+--
+The type of web content, typically text/html.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.sha_sha256`*::
++
+--
+Hex digest of the response content.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.av_detections`*::
++
+--
+The detection name according to the antivirus engine used in file inspection.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.puas`*::
++
+--
+A list of all potentially unwanted application (PUA) results for the proxied file as returned by the antivirus scanner.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.amp_disposition`*::
++
+--
+The status of the files proxied and scanned by Cisco Advanced Malware Protection (AMP) as part of the Umbrella File Inspection feature; can be Clean, Malicious or Unknown.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.amp_malware_name`*::
++
+--
+If Malicious, the name of the malware according to AMP.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.amp_score`*::
++
+--
+The score of the malware from AMP. This field is not currently used and will be blank.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.datacenter`*::
++
+--
+The name of the Umbrella Data Center that processed the user-generated traffic.
+
+
+type: keyword
+
+--
+
+*`cisco.umbrella.origin_id`*::
++
+--
+The unique identity of the network tunnel.
+
+
 type: keyword
 
 --

filebeat/docs/modules/cisco.asciidoc

@@ -10,13 +10,15 @@ This file is generated! See scripts/docs_collector.py
 
 == Cisco module
 
-This is a module for Cisco network device's logs. It includes the following
+This is a module for Cisco network device's logs and Cisco Umbrella. It includes the following
 filesets for receiving logs over syslog or read from a file:
 
 - `asa` fileset: supports Cisco ASA firewall logs.
 - `ftd` fileset: supports Cisco Firepower Threat Defense logs.
 - `ios` fileset: supports Cisco IOS router and switch logs.
 - `nexus` fileset: supports Cisco Nexus switch logs.
+- `meraki` fileset: supports Cisco Meraki logs.
+- `umbrella` fileset: supports Cisco Umbrella logs.
 
 Cisco ASA devices also support exporting flow records using NetFlow, which is
 supported by the {filebeat-ref}/filebeat-module-netflow.html[netflow module] in
@@ -32,6 +34,8 @@ The module is by default configured to run via syslog on port 9001 for ASA and
 port 9002 for IOS. However it can also be configured to read from a file path.
 See the following example.
 
+Cisco Umbrella publishes its logs in a compressed CSV format to a S3 bucket.
+
 ["source","yaml",subs="attributes"]
 -----
 - module: cisco
@@ -379,6 +383,59 @@ will be found under `rsa.raw`. The default is false.
 
 :fileset_ex!:
 
+[float]
+==== `umbrella` fileset settings
+
+The Cisco Umbrella fileset primarily focuses on reading CSV files from an S3 bucket using the filebeat S3 input.
+
+To configure Cisco Umbrella to log to either your own S3 bucket or one that is managed by Cisco please follow the https://docs.umbrella.com/deployment-umbrella/docs/log-management[Cisco Umbrella User Guide.]
+
+This fileset supports all 4 log types:
+- Proxy
+- Cloud Firewall
+- IP Logs
+- DNS logs
+
+The Cisco Umbrella fileset depends on the original file path structure being followed. This structure is documented https://docs.umbrella.com/deployment-umbrella/docs/log-formats-and-versioning[Umbrella Log Formats and Versioning]:
+
+<subfolder>/<YYYY>-<MM>-<DD>/<YYYY>-<MM>-<DD>-<hh>-<mm>-<xxxx>.csv.gz
+dnslogs/<year>-<month>-<day>/<year>-<month>-<day>-<hour>-<minute>.csv.gz
+
+When configuring the fileset, please ensure that the Queue URL is set to the root folder that includes each of these subfolders above.
+
+Example config:
+
+[source,yaml]
+----
+- module: cisco
+  umbrella:
+    enabled: true
+    var.input: s3
+    var.queue_url: https://sqs.us-east-1.amazonaws.com/ID/CiscoQueue
+    var.access_key_id: 123456
+    var.secret_access_key: PASSWORD
+----
+
+*`var.input`*::
+
+The input from which messages are read. Can be S3 or file.
+
+*`var.queue_url`*::
+
+The URL to the SQS queue if the input type is S3.
+
+*`var.access_key_id`*::
+
+The ID for the access key used to read from the SQS queue.
+
+*`var.secret_access_key`*::
+
+The secret token used for authenticating to the SQS queue.
+
+:has-dashboards!:
+
+:fileset_ex!:
+
 [float]
 === Example dashboard
 

filebeat/docs/reload-configuration.asciidoc

@@ -33,9 +33,11 @@ definitions.
 
 TIP: The first line of each external configuration file must be an input
 definition that starts with `- type`. Make sure you omit the line
-+{beatname_lc}.config.inputs+ from this file.
-
-For example:
++{beatname_lc}.config.inputs+ from this file. All <<filebeat-input-types,`input type configuration options`>> 
+must be specified within each external configuration file.  Specifying these
+configuration options at the global `filebeat.config.inputs` level is not supported.
+
+Example external configuration file:
 
 [source,yaml]
 ------------------------------------------------------------------------------