Skip to content

Commit

Permalink
Add User Agent Parser for Azure Sign In Logs (#23201)
Browse files Browse the repository at this point in the history
* Add User Agent Parser for Azure Sign In Logs

This will be a nice addition for parsing the user agent in the Azure sign in logs. This would allow for some great detections on unusual user agents for sign in activity.

* Update CHANGELOG.next.asciidoc

* Add example log with UA

* Update signinlogs.log

Update example log using event.original from filebeat initial message.

* Convert more fields to snake_case, update fields.yml

The new log sample exposed fields that were missing from the mapping. It also
exposed some new fields listed at https://docs.microsoft.com/en-us/azure/azure-monitor/reference/tables/signinlogs
that were not yet converted to snake_case. So I added rename processors to convert them
to snake_case and added descriptions in fields.yml.

Since user_agent is part of ECS I renamed the Azure userAgent field to user_agent.original.

* Use better mappings

* Update field docs

* Update generated

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
  • Loading branch information
nicpenning and andrewkroh authored Mar 18, 2021
1 parent e40b421 commit a2e8969
Show file tree
Hide file tree
Showing 7 changed files with 412 additions and 2 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Rename `network.direction` values in crowdstrike/falcon to `ingress`/`egress`. {pull}23041[23041]
- Rename `s3` input to `aws-s3` input. {pull}23469[23469]
- Possible values for Netflow's locality fields (source.locality, destination.locality and flow.locality) are now `internal` and `external`, instead of `private` and `public`. {issue}24272[24272] {pull}24295[24295]
- Add User Agent Parser for Azure Sign In Logs Ingest Pipeline {pull}23201[23201]

*Heartbeat*
- Adds negative body match. {pull}20728[20728]
Expand Down
118 changes: 118 additions & 0 deletions filebeat/docs/fields.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -3569,6 +3569,124 @@ type: keyword
Status


type: keyword

--

*`azure.signinlogs.properties.authentication_requirement_policies`*::
+
--
Set of CA policies that apply to this sign-in, each as CA: policy name, and/or MFA: Per-user.


type: keyword

--

*`azure.signinlogs.properties.applied_conditional_access_policies`*::
+
--
Details of the conditional access policies being applied for the sign-in.


type: nested

--

*`azure.signinlogs.properties.resource_tenant_id`*::
+
--
The resource tenantId for B2B(business-to-business) scenarios.


type: keyword

--

*`azure.signinlogs.properties.authentication_details`*::
+
--
A record of each step of authentication undertaken in the sign-in.


type: nested

--

*`azure.signinlogs.properties.authentication_processing_details`*::
+
--
Provides the details associated with authentication processor.


type: flattened

--

*`azure.signinlogs.properties.flagged_for_review`*::
+
--
Event was flagged for review.

type: boolean

--

*`azure.signinlogs.properties.network_location_details`*::
+
--
Provides the details associated with authentication processor.


type: keyword

--

*`azure.signinlogs.properties.risk_event_types`*::
+
--
The list of risk event types associated with the sign-in.


type: keyword

--

*`azure.signinlogs.properties.risk_event_types_v2`*::
+
--
The list of risk event types associated with the sign-in.


type: keyword

--

*`azure.signinlogs.properties.authentication_requirement`*::
+
--
Type of authentication required for the sign-in. If set to multiFactorAuthentication, an MFA step was required. If set to singleFactorAuthentication, no MFA was required


type: keyword

--

*`azure.signinlogs.properties.resource_id`*::
+
--
ID of the resource that the user signed into.


type: keyword

--

*`azure.signinlogs.properties.user_type`*::
+
--
User type.

type: keyword

--
Expand Down
2 changes: 1 addition & 1 deletion x-pack/filebeat/module/azure/fields.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

49 changes: 48 additions & 1 deletion x-pack/filebeat/module/azure/signinlogs/_meta/fields.yml
Original file line number Diff line number Diff line change
Expand Up @@ -163,4 +163,51 @@
type: keyword
description: >
Status
- name: authentication_requirement_policies
type: keyword
description: >
Set of CA policies that apply to this sign-in, each as CA: policy name, and/or MFA: Per-user.
- name: applied_conditional_access_policies
type: nested
description: >
Details of the conditional access policies being applied for the sign-in.
- name: resource_tenant_id
type: keyword
description: >
The resource tenantId for B2B(business-to-business) scenarios.
- name: authentication_details
type: nested
description: >
A record of each step of authentication undertaken in the sign-in.
- name: authentication_processing_details
type: flattened
description: >
Provides the details associated with authentication processor.
- name: flagged_for_review
type: boolean
description: Event was flagged for review.
- name: network_location_details
type: keyword
description: >
Provides the details associated with authentication processor.
- name: risk_event_types
type: keyword
description: >
The list of risk event types associated with the sign-in.
- name: risk_event_types_v2
type: keyword
description: >
The list of risk event types associated with the sign-in.
- name: authentication_requirement
type: keyword
description: >
Type of authentication required for the sign-in. If set to
multiFactorAuthentication, an MFA step was required. If set to
singleFactorAuthentication, no MFA was required
- name: resource_id
type: keyword
description: >
ID of the resource that the user signed into.
- name: user_type
type: keyword
description: User type.
31 changes: 31 additions & 0 deletions x-pack/filebeat/module/azure/signinlogs/ingest/pipeline.yml
Original file line number Diff line number Diff line change
Expand Up @@ -155,6 +155,10 @@ processors:
field: azure.signinlogs.properties.userId
target_field: azure.signinlogs.properties.user_id
ignore_missing: true
- rename:
field: azure.signinlogs.properties.userType
target_field: azure.signinlogs.properties.user_type
ignore_missing: true
- rename:
field: azure.signinlogs.properties.appId
target_field: azure.signinlogs.properties.app_id
Expand Down Expand Up @@ -247,6 +251,30 @@ processors:
field: azure.signinlogs.properties.servicePrincipalId
target_field: azure.signinlogs.properties.service_principal_id
ignore_missing: true
- rename:
field: azure.signinlogs.properties.resourceTenantId
target_field: azure.signinlogs.properties.resource_tenant_id
ignore_missing: true
- rename:
field: azure.signinlogs.properties.flaggedForReview
target_field: azure.signinlogs.properties.flagged_for_review
ignore_missing: true
- rename:
field: azure.signinlogs.properties.riskEventTypes
target_field: azure.signinlogs.properties.risk_event_types
ignore_missing: true
- rename:
field: azure.signinlogs.properties.riskEventTypes_v2
target_field: azure.signinlogs.properties.risk_event_types_v2
ignore_missing: true
- rename:
field: azure.signinlogs.properties.authenticationRequirement
target_field: azure.signinlogs.properties.authentication_requirement
ignore_missing: true
- rename:
field: azure.signinlogs.properties.userAgent
target_field: user_agent.original
ignore_missing: true
- remove:
field:
- azure.signinlogs.properties.location
Expand Down Expand Up @@ -307,6 +335,9 @@ processors:
field: source.as.organization_name
target_field: source.as.organization.name
ignore_missing: true
- user_agent:
field: user_agent.original
ignore_missing: true
- pipeline:
name: '{< IngestPipeline "azure-shared-pipeline" >}'
on_failure:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
{"Level":4,"callerIpAddress":"81.171.241.231","category":"SignInLogs","correlationId":"8a4de8b5-095c-47d0-a96f-a75130c61d53","durationMs":0,"identity":"Test LTest","location":"FR","operationName":"Sign-in activity","operationVersion":"1.0","properties":{"appDisplayName":"Office 365","appId":"8a4de8b5-095c-47d0-a96f-a75130c61d53","clientAppUsed":"Browser","conditionalAccessStatus":"notApplied","correlationId":"8a4de8b5-095c-47d0-a96f-a75130c61d53","createdDateTime":"2019-10-18T04:45:48.0729893-05:00","deviceDetail":{"browser":"Chrome 77.0.3865","deviceId":"","operatingSystem":"MacOs"},"id":"8a4de8b5-095c-47d0-a96f-a75130c61d53","ipAddress":"81.171.241.231","isInteractive":false,"location":{"city":"Champs-Sur-Marne","countryOrRegion":"FR","geoCoordinates":{"latitude":48.12341234,"longitude":2.12341234},"state":"Seine-Et-Marne"},"originalRequestId":"8a4de8b5-095c-47d0-a96f-a75130c61d53","processingTimeInMilliseconds":239,"riskDetail":"none","riskLevelAggregated":"none","riskLevelDuringSignIn":"none","riskState":"none","servicePrincipalId":"","status":{"errorCode":50140,"failureReason":"This error occurred due to 'Keep me signed in' interrupt when the user was signing-in."},"tokenIssuerName":"","tokenIssuerType":"AzureAD","userDisplayName":"Test LTest","userId":"8a4de8b5-095c-47d0-a96f-a75130c61d53","userPrincipalName":"test@elastic.co"},"resourceId":"/tenants/8a4de8b5-095c-47d0-a96f-a75130c61d53/providers/Microsoft.aadiam","resultDescription":"This error occurred due to 'Keep me signed in' interrupt when the user was signing-in.","resultSignature":"None","resultType":"50140","tenantId":"8a4de8b5-095c-47d0-a96f-a75130c61d53","time":"2019-10-18T09:45:48.0729893Z"}
{"Level":4,"callerIpAddress":"8.8.8.8","category":"SignInLogs","correlationId":"a8d4eb85-90c5-740d-9af6-7a15036cd135","durationMs":0,"identity":"Test LTest","location":"FR","operationName":"Sign-in activity","operationVersion":"1.0","properties":{"appDisplayName":"Office 365","appId":"8a4de8b5-095c-47d0-a96f-a75130c61d53","clientAppUsed":"Browser","conditionalAccessStatus":"notApplied","correlationId":"8a4de8b5-095c-47d0-a96f-a75130c61d53","createdDateTime":"2019-10-18T04:45:48.0729893-05:00","deviceDetail":{"browser":"Chrome 77.0.3865","deviceId":"","operatingSystem":"MacOs"},"id":"8a4de8b5-095c-47d0-a96f-a75130c61d53","ipAddress":"81.171.241.231","isInteractive":false,"location":{"city":"Champs-Sur-Marne","countryOrRegion":"FR","geoCoordinates":{"latitude":48.12341234,"longitude":2.12341234},"state":"Seine-Et-Marne"},"originalRequestId":"8a4de8b5-095c-47d0-a96f-a75130c61d53","processingTimeInMilliseconds":239,"riskDetail":"none","riskLevelAggregated":"none","riskLevelDuringSignIn":"none","riskState":"none","servicePrincipalId":"","status":{"errorCode":50140,"failureReason":"This error occurred due to 'Keep me signed in' interrupt when the user was signing-in."},"tokenIssuerName":"","tokenIssuerType":"AzureAD","userDisplayName":"Test LTest","userId":"8a4de8b5-095c-47d0-a96f-a75130c61d53","userPrincipalName":"c3813493-bf92-5123-2717-8a8b2979c38b"},"resourceId":"/tenants/8a4de8b5-095c-47d0-a96f-a75130c61d53/providers/Microsoft.aadiam","resultDescription":"This error occurred due to 'Keep me signed in' interrupt when the user was signing-in.","resultSignature":"None","resultType":"50140","tenantId":"8a4de8b5-095c-47d0-a96f-a75130c61d53","time":"2019-10-18T09:45:48.0729893Z"}
{"Level":4,"callerIpAddress":"8.8.8.8","category":"SignInLogs","correlationId":"1ba108d9-9609-48be-baee-afc0885baa06","durationMs":0,"identity":"Doe, John","location":"US","operationName":"Sign-in activity","operationVersion":"1.0","properties":{"appDisplayName":"Office365 Shell WCSS-Client","appId":"89bee1f7-5e6e-4d8a-9f3d-ecd601259da7","appliedConditionalAccessPolicies":[{"conditionsNotSatisfied":2,"conditionsSatisfied":1,"displayName":"On-Prem Access Only","enforcedGrantControls":["Block"],"enforcedSessionControls":[],"id":"123ebbf1-e868-4a77-bfd9-b59bd6c2412e","result":"notApplied"},{"conditionsNotSatisfied":0,"conditionsSatisfied":0,"displayName":"ForceMFAfor B2C","enforcedGrantControls":[],"enforcedSessionControls":[],"id":"0dff3d49-001e-413f-86eb-2800e789674c","result":"notEnabled"},{"conditionsNotSatisfied":2,"conditionsSatisfied":1,"displayName":"Baseline policy: Require MFA for admins","enforcedGrantControls":["Mfa"],"enforcedSessionControls":[],"id":"a5527e71-9da1-41d0-859b-7ca84dae03a7","result":"notApplied"},{"conditionsNotSatisfied":2,"conditionsSatisfied":1,"displayName":"Baseline Policy: Blocks legacy authentication","enforcedGrantControls":["Block"],"enforcedSessionControls":[],"id":"c1311105-97ac-4ebd-a866-5b215d066765","result":"notApplied"},{"conditionsNotSatisfied":1,"conditionsSatisfied":0,"displayName":"Netscaler MFA","enforcedGrantControls":["Mfa"],"enforcedSessionControls":["SignInFrequency"],"id":"ee756a5f-8c3b-41eb-8ace-0839597f718a","result":"notApplied"},{"conditionsNotSatisfied":8,"conditionsSatisfied":19,"displayName":"Enforce Verification on External Access","enforcedGrantControls":["Mfa"],"enforcedSessionControls":["SignInFrequency"],"id":"913f5adc-cd20-4b35-93b8-fbe145f68444","result":"notApplied"},{"conditionsNotSatisfied":2,"conditionsSatisfied":1,"displayName":"Test Policy","enforcedGrantControls":["Mfa"],"enforcedSessionControls":[],"id":"cf0d2cec-b974-4fd3-a1d3-da4ae1e896fa","result":"notApplied"}],"authenticationDetails":[{"RequestSequence":0,"StatusSequence":0,"authenticationMethod":"Previously satisfied","authenticationStepDateTime":"2021-01-26T13:39:55.7863053+00:00","authenticationStepRequirement":"Primary authentication","authenticationStepResultDetail":"First factor requirement satisfied by claim in the token","succeeded":true}],"authenticationProcessingDetails":[{"key":"Domain Hint Present","value":"True"},{"key":"Login Hint Present","value":"True"},{"key":"Private Link Id","value":"0"},{"key":"Azure AD App Authentication Library","value":"Family: ADAL Library: ADAL.Js 1.0.15 Platform: JS"},{"key":"IsCAEToken","value":"False"}],"authenticationRequirement":"singleFactorAuthentication","authenticationRequirementPolicies":[],"clientAppUsed":"Browser","conditionalAccessStatus":"success","correlationId":"1ba108d9-9609-48be-baee-afc0885baa06","createdDateTime":"2021-01-26T13:39:55.7863053+00:00","deviceDetail":{"browser":"Chrome 87.0.4280","deviceId":"","operatingSystem":"Windows 10"},"flaggedForReview":false,"id":"a9222177-db03-40ef-9b86-5b207ed72000","ipAddress":"192.168.108.29","isInteractive":true,"location":{"city":"Pierre","countryOrRegion":"US","geoCoordinates":{"latitude":44.567081451416016,"longitude":-100.26722717285156},"state":"South Dakota"},"networkLocationDetails":[],"originalRequestId":"a9222177-db03-40ef-9b86-5b207ed72000","processingTimeInMilliseconds":162,"resourceDisplayName":"Microsoft Graph","resourceId":"00000003-0000-0000-c000-000000000000","resourceTenantId":"19aa547c-22ab-606d-a4b6-541c5ce52b71","riskDetail":"none","riskEventTypes":[],"riskEventTypes_v2":[],"riskLevelAggregated":"none","riskLevelDuringSignIn":"none","riskState":"none","servicePrincipalId":"","status":{"errorCode":0},"tokenIssuerName":"","tokenIssuerType":"AzureAD","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Safari/537.36","userDisplayName":"Doe, John","userId":"762a6171-29d0-456b-b88b-ca7f7d99728d","userPrincipalName":"john.doe@example.com","userType":"Member"},"resourceId":"/tenants/19aa547c-22ab-606d-a4b6-541c5ce52b71/providers/Microsoft.aadiam","resultSignature":"None","resultType":"0","tenantId":"19aa547c-22ab-606d-a4b6-541c5ce52b71","time":"2021-01-26T13:39:55.7863053Z"}
Loading

0 comments on commit a2e8969

Please sign in to comment.