Cherry-pick #24829 to 7.x: [Filebeat] Update Oauth2 flow for m365 defender fileset #24847

P1llus · 2021-03-30T13:45:58Z

Cherry-pick of PR #24829 to 7.x branch. Original message:

What does this PR do?

Microsoft changed the Oauth2 flow from resource to scope based, so the default configuration and documentation needs to reflect that.

Why is it important?

Module won't work without it.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

…ith capital letters, and update pytest (elastic#24762) * updating sophos KV processor to support keys starting with capital letters, adding some missing fields and updating pytest * Updating changelog * updating tests to fix errors * updating tests to fix errors * removing new tests for now, waiting for kibana module fixes to be applied * removing new tests for now, waiting for kibana module fixes to be applied (cherry picked from commit db0d43e)

* updating m365 config and documentation to match new Oauth2 authentication flow * Update changelog (cherry picked from commit 55be313)

elasticmachine · 2021-03-30T13:46:01Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2021-03-30T13:55:26Z

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #24847 opened
Reason: Aborted from #2
Start Time: 2021-03-30T13:50:43.469+0000
Duration: 5 min 32 sec
Commit: 4f8eeb0

Trends 🧪

Log output

Expand to view the last 100 lines of log output

[2021-03-30T13:51:34.820Z]  > git config --get remote.origin.url # timeout=10
[2021-03-30T13:51:34.824Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-03-30T13:51:34.830Z]  > git merge eb87a3138008aff500b862589c1f0eef39a9d948 # timeout=10
[2021-03-30T13:51:34.846Z]  > git rev-parse HEAD^{commit} # timeout=10
[2021-03-30T13:51:34.855Z]  > git config core.sparsecheckout # timeout=10
[2021-03-30T13:51:34.860Z]  > git checkout -f 4f8eeb03af79cfc31f00e3c1c759f23f08b16e4a # timeout=15
[2021-03-30T13:51:40.006Z] Commit message: "Update CHANGELOG.next.asciidoc"
[2021-03-30T13:51:40.007Z] First time build. Skipping changelog.
[2021-03-30T13:51:40.007Z] Cleaning workspace
[2021-03-30T13:51:40.202Z]  > git --version # timeout=10
[2021-03-30T13:51:40.205Z]  > git --version # 'git version 2.17.1'
[2021-03-30T13:51:40.212Z] fatal: bad object de5f52a9091c857929a1b04bd53726cce85ab70b
[2021-03-30T13:51:41.156Z] Timeout set to expire in 3 hr 0 min
[2021-03-30T13:51:41.195Z] The timestamps step is unnecessary when timestamps are enabled for all Pipeline builds.
[2021-03-30T13:51:41.503Z] [INFO] Number of builds to be searched 10
[2021-03-30T13:51:40.009Z]  > git rev-parse --verify HEAD # timeout=10
[2021-03-30T13:51:40.014Z] Resetting working tree
[2021-03-30T13:51:40.014Z]  > git reset --hard # timeout=10
[2021-03-30T13:51:40.109Z]  > git clean -fdx # timeout=10
[2021-03-30T13:51:42.150Z] [INFO] 'shallow' is forced to be disabled when running on PullRequests
[2021-03-30T13:51:42.186Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-24847/src/github.com/elastic/beats
[2021-03-30T13:51:42.245Z] [INFO] gitCheckout: Checkout SCM PR-24847 with default customisation from the Item.
[2021-03-30T13:51:42.302Z] [INFO] Override default checkout
[2021-03-30T13:51:42.409Z] Sleeping for 10 sec
[2021-03-30T13:51:52.450Z] The recommended git tool is: git
[2021-03-30T13:51:52.463Z] using credential f6c7695a-671e-4f4f-a331-acdce44ff9ba
[2021-03-30T13:51:52.504Z] Wiping out workspace first.
[2021-03-30T13:51:52.517Z] Cloning the remote Git repository
[2021-03-30T13:51:52.517Z] Using shallow clone with depth 10
[2021-03-30T13:51:52.517Z] Avoid fetching tags
[2021-03-30T13:51:52.525Z] Cloning repository git@github.com:elastic/beats.git
[2021-03-30T13:51:52.526Z]  > git init /var/lib/jenkins/workspace/Beats_beats_PR-24847/src/github.com/elastic/beats # timeout=10
[2021-03-30T13:51:52.533Z] Fetching upstream changes from git@github.com:elastic/beats.git
[2021-03-30T13:51:52.534Z]  > git --version # timeout=10
[2021-03-30T13:51:52.537Z]  > git --version # 'git version 2.17.1'
[2021-03-30T13:51:52.537Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-03-30T13:51:52.541Z]  > git fetch --no-tags --progress -- git@github.com:elastic/beats.git +refs/heads/*:refs/remotes/origin/* # timeout=15
[2021-03-30T13:52:15.283Z] Cleaning workspace
[2021-03-30T13:52:15.297Z] Using shallow fetch with depth 10
[2021-03-30T13:52:15.297Z] Pruning obsolete local branches
[2021-03-30T13:52:15.265Z]  > git config remote.origin.url git@github.com:elastic/beats.git # timeout=10
[2021-03-30T13:52:15.269Z]  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2021-03-30T13:52:15.277Z]  > git config remote.origin.url git@github.com:elastic/beats.git # timeout=10
[2021-03-30T13:52:15.285Z]  > git rev-parse --verify HEAD # timeout=10
[2021-03-30T13:52:15.289Z] No valid HEAD. Skipping the resetting
[2021-03-30T13:52:15.289Z]  > git clean -fdx # timeout=10
[2021-03-30T13:52:15.299Z] Fetching upstream changes from git@github.com:elastic/beats.git
[2021-03-30T13:52:15.300Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-03-30T13:52:15.303Z]  > git fetch --no-tags --progress --prune -- git@github.com:elastic/beats.git +refs/pull/24847/head:refs/remotes/origin/PR-24847 +refs/heads/7.x:refs/remotes/origin/7.x # timeout=15
[2021-03-30T13:52:16.505Z] Merging remotes/origin/7.x commit eb87a3138008aff500b862589c1f0eef39a9d948 into PR head commit 4f8eeb03af79cfc31f00e3c1c759f23f08b16e4a
[2021-03-30T13:52:18.193Z] Merge succeeded, producing 4f8eeb03af79cfc31f00e3c1c759f23f08b16e4a
[2021-03-30T13:52:18.193Z] Checking out Revision 4f8eeb03af79cfc31f00e3c1c759f23f08b16e4a (PR-24847)
[2021-03-30T13:52:16.507Z]  > git config core.sparsecheckout # timeout=10
[2021-03-30T13:52:16.511Z]  > git checkout -f 4f8eeb03af79cfc31f00e3c1c759f23f08b16e4a # timeout=15
[2021-03-30T13:52:18.162Z]  > git remote # timeout=10
[2021-03-30T13:52:18.165Z]  > git config --get remote.origin.url # timeout=10
[2021-03-30T13:52:18.169Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2021-03-30T13:52:18.172Z]  > git merge eb87a3138008aff500b862589c1f0eef39a9d948 # timeout=10
[2021-03-30T13:52:18.187Z]  > git rev-parse HEAD^{commit} # timeout=10
[2021-03-30T13:52:18.196Z]  > git config core.sparsecheckout # timeout=10
[2021-03-30T13:52:18.200Z]  > git checkout -f 4f8eeb03af79cfc31f00e3c1c759f23f08b16e4a # timeout=15
[2021-03-30T13:52:18.462Z] Commit message: "Update CHANGELOG.next.asciidoc"
[2021-03-30T13:52:18.462Z] Cleaning workspace
[2021-03-30T13:52:18.796Z]  > git --version # timeout=10
[2021-03-30T13:52:18.800Z]  > git --version # 'git version 2.17.1'
[2021-03-30T13:52:18.804Z] fatal: bad object de5f52a9091c857929a1b04bd53726cce85ab70b
[2021-03-30T13:52:19.710Z] Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD
[2021-03-30T13:52:20.097Z] + git fetch https://****:****@github.com/elastic/beats.git +refs/pull/*/head:refs/remotes/origin/pr/*
[2021-03-30T13:52:18.464Z]  > git rev-parse --verify HEAD # timeout=10
[2021-03-30T13:52:18.468Z] Resetting working tree
[2021-03-30T13:52:18.468Z]  > git reset --hard # timeout=10
[2021-03-30T13:52:18.712Z]  > git clean -fdx # timeout=10
[2021-03-30T13:53:28.106Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-24847/src/github.com/elastic/beats/.git
[2021-03-30T13:53:28.590Z] Archiving artifacts
[2021-03-30T13:53:29.616Z] + git rev-parse HEAD
[2021-03-30T13:53:30.046Z] + git rev-parse HEAD
[2021-03-30T13:53:30.427Z] + git rev-parse origin/pr/24847
[2021-03-30T13:53:30.568Z] [INFO] githubEnv: Found Git Build Cause: pr
[2021-03-30T13:53:31.829Z] Masking supported pattern matches of $GITHUB_TOKEN
[2021-03-30T13:53:33.372Z] [WARN] githubApiCall: The REST API call https://api.github.com/repos/elastic/beats/pulls/24847/reviews return 0 elements
[2021-03-30T13:53:33.524Z] [INFO] githubPrCheckApproved: Title: Cherry-pick #24829 to 7.x: [Filebeat] Update Oauth2 flow for m365 defender fileset - User: P1llus - Author Association: MEMBER
[2021-03-30T13:53:35.503Z] [INFO] stashV2: JOB_GCS_BUCKET is set. bucket param got precedency instead.
[2021-03-30T13:53:35.573Z] [INFO] stashV2: JOB_GCS_CREDENTIALS is set. credentialsId param got precedency instead.
[2021-03-30T13:53:36.362Z] + tar --version
[2021-03-30T13:53:36.870Z] + tar --exclude=source.tgz -czf source.tgz .
[2021-03-30T13:54:33.273Z] [INFO] Override default googleStorageUpload with some sleep
[2021-03-30T13:54:33.336Z] Sleeping for 1 min 13 sec
[2021-03-30T13:55:08.486Z] Stage "Lint" skipped due to earlier failure(s)
[2021-03-30T13:55:08.588Z] Stage "Build&Test" skipped due to earlier failure(s)
[2021-03-30T13:55:08.689Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-03-30T13:55:08.818Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-24847/src/github.com/elastic/beats
[2021-03-30T13:55:15.020Z] Running on beats-ci-immutable-ubuntu-2004-1617110332978113884 in /var/lib/jenkins/workspace/Beats_beats_PR-24847
[2021-03-30T13:55:15.159Z] [INFO] getVaultSecret: Getting secrets
[2021-03-30T13:55:15.264Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-03-30T13:55:16.218Z] + chmod 755 generate-build-data.sh
[2021-03-30T13:55:16.218Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24847/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24847/runs/1 ABORTED 272436
[2021-03-30T13:55:16.218Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24847/runs/1/steps/?limit=10000 -o steps-info.json
[2021-03-30T13:55:16.218Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24847/runs/1/tests/?status=FAILED -o tests-errors.json
[2021-03-30T13:55:16.218Z] Retry 1/3 exited 22, retrying in 1 seconds...
[2021-03-30T13:55:17.608Z] Retry 2/3 exited 22, retrying in 2 seconds...

elasticmachine · 2021-03-30T15:00:12Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #24847 updated
Start Time: 2021-03-30T13:51:53.079+0000
Duration: 68 min 0 sec
Commit: 4f8eeb0

Test stats 🧪

Test	Results
Failed	0
Passed	13139
Skipped	2285
Total	15424

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	13139
Skipped	2285
Total	15424

P1llus added 2 commits March 30, 2021 15:45

[Filebeat] Update Oauth2 flow for m365 defender fileset (elastic#24829)

2081ce9

* updating m365 config and documentation to match new Oauth2 authentication flow * Update changelog (cherry picked from commit 55be313)

P1llus added [zube]: In Review backport Team:Security-External Integrations labels Mar 30, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 30, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 30, 2021

P1llus added 2 commits March 30, 2021 15:47

Merge branch '7.x' into backport_24829_7.x

ae56b2f

Update CHANGELOG.next.asciidoc

4f8eeb0

P1llus closed this Mar 30, 2021

P1llus deleted the backport_24829_7.x branch March 30, 2021 14:10

zube bot added [zube]: Done and removed [zube]: In Review labels Mar 30, 2021

zube bot removed the [zube]: Done label Jun 29, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cherry-pick #24829 to 7.x: [Filebeat] Update Oauth2 flow for m365 defender fileset #24847

Cherry-pick #24829 to 7.x: [Filebeat] Update Oauth2 flow for m365 defender fileset #24847

P1llus commented Mar 30, 2021 •

edited by zube bot

Loading

elasticmachine commented Mar 30, 2021

elasticmachine commented Mar 30, 2021

Build stats

Trends 🧪

elasticmachine commented Mar 30, 2021

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

Cherry-pick #24829 to 7.x: [Filebeat] Update Oauth2 flow for m365 defender fileset #24847

Cherry-pick #24829 to 7.x: [Filebeat] Update Oauth2 flow for m365 defender fileset #24847

Conversation

P1llus commented Mar 30, 2021 • edited by zube bot Loading

What does this PR do?

Why is it important?

Checklist

elasticmachine commented Mar 30, 2021

elasticmachine commented Mar 30, 2021

❕ Build Aborted

Build stats

Trends 🧪

Log output

elasticmachine commented Mar 30, 2021

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

P1llus commented Mar 30, 2021 •

edited by zube bot

Loading