Add support for platform logs in Filebeat Azure module

[narph]

What does this PR do?

Add support for platform logs in Filebeat Azure module

Why is it important?

Platform logs provide detailed diagnostic and auditing information for Azure resources and the Azure platform they depend on.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Example platform logs:

{
   "ActivityId":"30ed877c-a36b-491a-bd4d-ddd847fe55b8",
   "Caller":"Portal",
   "Environment":"PROD",
   "EventName":"Retreive ConsumerGroup",
   "EventProperties":"{\"SubscriptionId\":\"...\",\"Namespace\":\"obstesteventhubs\",\"Via\":\"sb://obstesteventhubs.servicebus.windows.net/insights-logs-operationallogs/consumergroups?api-version=2017-04\u0026$skip=0\u0026$top=100\",\"TrackingId\":\"30ed877c-a36b-491a-bd4d-ddd847fe55b8_M2CH3_M2CH3_G3S2\"}",
   "EventTimeString":"11/3/2020 9:06:42 AM +00:00",
   "Region":"West Europe",
   "ScaleUnit":"PROD-AM3-AZ501",
   "Status":"Succeeded",
   "SubscriptionId":"....",
   "category":"OperationalLogs",
   "resourceId":"/SUBSCRIPTIONS/.../RESOURCEGROUPS/OBS-TEST/PROVIDERS/MICROSOFT.EVENTHUB/NAMESPACES/OBSTESTEVENTHUBS"
}

{
   "Cloud":"AzureCloud",
   "Environment":"prod",
   "UnderlayClass":"hcp-underlay",
   "UnderlayName":"hcp-underlay-westeurope-cx-316",
   "attrs":"{\"annotation.io.kubernetes.container.hash\"=\u003e\"b74d7ef3\", \"annotation.io.kubernetes.container.ports\"=\u003e\"[{\\\"name\\\":\\\"https\\\",\\\"containerPort\\\":4444,\\\"protocol\\\":\\\"TCP\\\"}]\", \"annotation.io.kubernetes.container.preStopHandler\"=\u003e\"{\\\"exec\\\":{\\\"command\\\":[\\\"/bin/bash\\\",\\\"-c\\\",\\\"sleep 20\\\"]}}\", \"annotation.io.kubernetes.container.restartCount\"=\u003e\"0\", \"annotation.io.kubernetes.container.terminationMessagePath\"=\u003e\"/dev/termination-log\", \"annotation.io.kubernetes.container.terminationMessagePolicy\"=\u003e\"File\", \"annotation.io.kubernetes.pod.terminationGracePeriod\"=\u003e\"30\", \"io.kubernetes.container.logpath\"=\u003e\"/var/log/pods/5e4bf4baee195b00017cdbfa_kube-apiserver-666bd4b459-vgc5h_53331907-669d-458e-ab0e-6744e56164f9/kube-apiserver/0.log\", \"io.kubernetes.container.name\"=\u003e\"kube-apiserver\", \"io.kubernetes.docker.type\"=\u003e\"container\", \"io.kubernetes.pod.name\"=\u003e\"kube-apiserver-666bd4b459-vgc5h\", \"io.kubernetes.pod.namespace\"=\u003e\"5e4bf4baee195b00017cdbfa\", \"io.kubernetes.pod.uid\"=\u003e\"53331907-669d-458e-ab0e-6744e56164f9\", \"io.kubernetes.sandbox.id\"=\u003e\"09097fc97b3c0130bde19eec329b588a328430f3c221c1f8e1520933b16dde9f\"}",
   "category":"kube-apiserver",
   "ccpNamespace":"5e4bf4baee195b00017cdbfa",
   "operationName":"Microsoft.ContainerService/managedClusters/diagnosticLogs/Read",
   "properties":{
      "containerID":"ca7ca3b15f428368fabab4dff0c14879a838f8653f84312833d5024547a008f4",
      "log":"I1105 09:40:47.768168       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io\n",
      "pod":"kube-apiserver-666bd4b459-vgc5h",
      "stream":"stderr"
   },
   "resourceId":"/SUBSCRIPTIONS/.../RESOURCEGROUPS/OBS-INFRASTRUCTURE/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/OBSKUBE",
   "time":"2020-11-05T09:40:47.0000000Z"
}

Ex outputs:

{
        "_index" : "filebeat-8.0.0-2020.11.05-000001",
        "_type" : "_doc",
        "_id" : "YQVzmHUBvB2moownK3Qu",
        "_score" : null,
        "_source" : {
          "agent" : {
            "name" : "DESKTOP-RFOOE09",
            "id" : "17cead09-57ad-4668-8a0e-b9025f8b0cb0",
            "type" : "filebeat",
            "ephemeral_id" : "a9a122d4-4b37-4c34-ac03-4a2f765a9ea1",
            "version" : "8.0.0"
          },
          "fileset" : {
            "name" : "platformlogs"
          },
          "azure-eventhub" : {
            "sequence_number" : 14,
            "consumer_group" : "$Default",
            "offset" : 4294975208,
            "eventhub" : "insights-logs-operationallogs",
            "enqueued_time" : "2020-11-05T10:27:43.439Z"
          },
          "tags" : [
            "forwarded"
          ],
          "cloud" : {
            "provider" : "azure",
            "region" : "West Europe"
          },
          "input" : {
            "type" : "azure-eventhub"
          },
          "@timestamp" : "2020-11-05T10:26:33.000Z",
          "ecs" : {
            "version" : "1.6.0"
          },
          "service" : {
            "type" : "azure"
          },
          "event" : {
            "ingested" : "2020-11-05T12:48:05.927630600Z",
            "kind" : "event",
            "module" : "azure",
            "action" : "Retreive Namespace",
            "dataset" : "azure.platformlogs",
            "outcome" : "succeeded"
          },
          "azure" : {
            "subscription_id" : "....",
            "platformlogs" : {
              "Status" : "Succeeded",
              "SubscriptionId" : "....",
              "Caller" : "Portal",
              "ActivityId" : "724eb3a4-6c1d-4a2f-8037-8d4db250154c",
              "EventTimeString" : "11/5/2020 10:26:33 AM +00:00",
              "Environment" : "PROD",
              "category" : "OperationalLogs",
              "event_category" : "Administrative",
              "ScaleUnit" : "PROD-AM3-AZ501",
              "properties" : {
                "SubscriptionId" : "...",
                "TrackingId" : "724eb3a4-6c1d-4a2f-8037-8d4db250154c_M6SN1_M6SN1_G10S1",
                "Namespace" : "obstesteventhubs",
                "Via" : "https://obstesteventhubs.servicebus.windows.net/$Resources/eventhubs?api-version=2017-04&$skip=0&$top=100"
              }
            },
            "resource" : {
              "provider" : "MICROSOFT.EVENTHUB/NAMESPACES",
              "name" : "OBSTESTEVENTHUBS",
              "id" : "/SUBSCRIPTIONS/.../RESOURCEGROUPS/OBS-TEST/PROVIDERS/MICROSOFT.EVENTHUB/NAMESPACES/OBSTESTEVENTHUBS",
              "group" : "OBS-TEST"
            }
          }
        }

{
        "_index" : "filebeat-8.0.0-2020.11.05-000001",
        "_type" : "_doc",
        "_id" : "oQRumHUBvB2moownKezJ",
        "_score" : null,
        "_source" : {
          "agent" : {
            "name" : "DESKTOP-RFOOE09",
            "id" : "17cead09-57ad-4668-8a0e-b9025f8b0cb0",
            "ephemeral_id" : "a6339387-a2fe-4db0-9d13-ca3123f92366",
            "type" : "filebeat",
            "version" : "8.0.0"
          },
          "fileset" : {
            "name" : "platformlogs"
          },
          "message" : """I1105 12:41:49.339404       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
""",
          "azure-eventhub" : {
            "sequence_number" : 45,
            "consumer_group" : "$Default",
            "offset" : 100168,
            "eventhub" : "insights-logs-kube-apiserver",
            "enqueued_time" : "2020-11-05T12:42:14.453Z"
          },
          "tags" : [
            "forwarded"
          ],
          "cloud" : {
            "provider" : "azure"
          },
          "input" : {
            "type" : "azure-eventhub"
          },
          "@timestamp" : "2020-11-05T12:41:49.000Z",
          "ecs" : {
            "version" : "1.6.0"
          },
          "service" : {
            "type" : "azure"
          },
          "event" : {
            "ingested" : "2020-11-05T12:42:37.895235200Z",
            "kind" : "event",
            "module" : "azure",
            "action" : "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read",
            "dataset" : "azure.platformlogs"
          },
          "azure" : {
            "subscription_id" : "...",
            "platformlogs" : {
              "ccpNamespace" : "5e4bf4baee195b00017cdbfa",
              "operation_name" : "Microsoft.ContainerService/managedClusters/diagnosticLogs/Read",
              "Cloud" : "AzureCloud",
              "Environment" : "prod",
              "UnderlayClass" : "hcp-underlay",
              "UnderlayName" : "hcp-underlay-westeurope-cx-316",
              "category" : "kube-apiserver",
              "event_category" : "Administrative",
              "properties" : {
                "pod" : "kube-apiserver-666bd4b459-vgc5h",
                "stream" : "stderr",
                "containerID" : "ca7ca3b15f428368fabab4dff0c14879a838f8653f84312833d5024547a008f4"
              },
              "attrs" : """{"annotation.io.kubernetes.container.hash"=>"b74d7ef3", "annotation.io.kubernetes.container.ports"=>"[{\"name\":\"https\",\"containerPort\":4444,\"protocol\":\"TCP\"}]", "annotation.io.kubernetes.container.preStopHandler"=>"{\"exec\":{\"command\":[\"/bin/bash\",\"-c\",\"sleep 20\"]}}", "annotation.io.kubernetes.container.restartCount"=>"0", "annotation.io.kubernetes.container.terminationMessagePath"=>"/dev/termination-log", "annotation.io.kubernetes.container.terminationMessagePolicy"=>"File", "annotation.io.kubernetes.pod.terminationGracePeriod"=>"30", "io.kubernetes.container.logpath"=>"/var/log/pods/5e4bf4baee195b00017cdbfa_kube-apiserver-666bd4b459-vgc5h_53331907-669d-458e-ab0e-6744e56164f9/kube-apiserver/0.log", "io.kubernetes.container.name"=>"kube-apiserver", "io.kubernetes.docker.type"=>"container", "io.kubernetes.pod.name"=>"kube-apiserver-666bd4b459-vgc5h", "io.kubernetes.pod.namespace"=>"5e4bf4baee195b00017cdbfa", "io.kubernetes.pod.uid"=>"53331907-669d-458e-ab0e-6744e56164f9", "io.kubernetes.sandbox.id"=>"09097fc97b3c0130bde19eec329b588a328430f3c221c1f8e1520933b16dde9f"}"""
            },
            "resource" : {
              "provider" : "MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS",
              "name" : "OBSKUBE",
              "id" : "/SUBSCRIPTIONS/.../RESOURCEGROUPS/OBS-INFRASTRUCTURE/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/OBSKUBE",
              "group" : "OBS-INFRASTRUCTURE"
            }
          }
        },

[narph]

ECS defines the set of values that are allowed to be used in event.category. The azure.platformlogs.category could be mapped to one or more of the allowed values. Like maybe have a table of expected or common azure categories that map to ECS categories (and possibly event.types).

• https://www.elastic.co/guide/en/ecs/current/ecs-allowed-values-event-category.html
• https://www.elastic.co/guide/en/ecs/current/ecs-allowed-values-event-type.html

thanks @andrewkroh , looking at the expected types I don't think we could match azure.platform.category to any of the two ECS fields. Here are the possible values we expect https://docs.microsoft.com/en-us/azure/azure-monitor/platform/resource-logs-categories.

[andrewkroh]

I think there are categories that could be applied. For example there are some database logs in there. And some web activity like for HTTP requests to the data lake (had to look at the data lake schema to check).

[exekias]

This field doesn't have a mapping, right? I see you use them to fill other fields, should it be dropped or mapped?

[narph]

added mapping for this field

[exekias]

It would be nice to have several lines here with different types, to give a better idea of what we can expect

[narph]

additional example in

[exekias]

This is looking great! I left a few comments