stop monitoring excluded paths #21282

newly12 · 2020-09-24T07:40:39Z

What does this PR do?

If the files match the excluded pattern, recursiveWatcher does not emit events for them, as well as errors if any.

Why is it important?

This eliminates unnecessary warning logs for file paths that user intends to exclude.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~I have made corresponding changes to the documentation~~
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

run test case TestRecursiveExcludedPaths

Related issues

Closes should not emit fsnotify watcher error for files match excluded_files pattern #21278

elasticmachine · 2020-09-24T07:40:41Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2020-09-24T07:40:41Z

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

elasticmachine · 2020-09-24T07:48:46Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: [marc-gr commented: jenkins run tests]
Start Time: 2020-11-09T08:20:43.238+0000
Duration: 26 min 52 sec

Test stats 🧪

Test	Results
Failed	0
Passed	456
Skipped	60
Total	516

elasticmachine · 2020-09-24T16:39:11Z

Pinging @elastic/siem (Team:SIEM)

newly12 · 2020-10-09T01:34:03Z

Hi,

Could someone please have a look at this PR? Feel free to let me know your thought on this. Thanks in advance.

marc-gr · 2020-10-16T08:17:00Z

jenkins run tests

newly12 · 2020-10-31T01:01:11Z

Hi,

the test is completed. is there any concern regarding this PR? Please let me know.

adriansr · 2020-11-02T10:09:54Z

auditbeat/module/file_integrity/monitor/recursive.go

@@ -154,7 +161,7 @@ func (watcher *recursiveWatcher) forwardEvents() error {
 			switch event.Op {
 			case fsnotify.Create:
 				err := watcher.addRecursive(event.Name)
-				if err != nil {
+				if err != nil && !watcher.isExcludedPath(event.Name) {


So the changes to this function are ignoring errors related to ignored paths. Does it still make sense to act on events for ignored paths?

For example, adding a recursive watch on an ignored directory. I wonder if it will make more sense to just skip those events.

@adriansr Thanks for the review, it makes more sense to stop watching them. I've updated this PR regarding your suggestion, please have a look.

adriansr · 2020-11-05T09:47:03Z

jenkins, test this

adriansr · 2020-11-05T19:00:50Z

jenkins, test this please

vjsamuel · 2020-11-06T00:21:47Z

@adriansr do we have anything left to address as part of this PR? would love to get this in so that we can reenable log shipping on our end.

adriansr · 2020-11-06T08:38:12Z

Jenkins run tests

elasticmachine · 2020-11-06T09:09:05Z

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	456
Skipped	60
Total	516

adriansr

@newly12, all is looking good. Only needs a changelog entry in CHANGELOG.next.asciidoc.

newly12 · 2020-11-08T11:58:32Z

Thanks @adriansr ! I've CHANGELOG.next.asciidoc and title of this PR as stop monitoring excluded paths to reflect what it really does. Please let me know if anything.

marc-gr · 2020-11-09T08:20:16Z

jenkins run tests

elasticmachine · 2020-11-09T11:34:38Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

If the files match the excluded pattern, recursiveWatcher does not emit events for them, as well as errors if any. (cherry picked from commit af8eedd)

If the files match the excluded pattern, recursiveWatcher does not emit events for them, as well as errors if any. (cherry picked from commit af8eedd) Co-authored-by: Peter Deng <yundeng@ebay.com>

newly12 requested a review from a team as a code owner September 24, 2020 07:40

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 24, 2020

newly12 marked this pull request as draft September 24, 2020 12:27

newly12 marked this pull request as ready for review September 24, 2020 12:49

jsoriano added the Team:SIEM label Sep 24, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 24, 2020

marc-gr self-assigned this Oct 19, 2020

adriansr suggested changes Nov 2, 2020

View reviewed changes

newly12 force-pushed the ignore_excluded_paths branch from 3e68cf9 to 05ac7c6 Compare November 5, 2020 05:56

adriansr reviewed Nov 6, 2020

View reviewed changes

newly12 added 5 commits November 8, 2020 19:51

ignore fsnotify events for excluded files

baba49e

do not emit error and event for excluded path

f7d02c3

update comments

d7a4889

stop monitoring excluded path

ce32e55

update CHANGELOG.next.asciidoc

e0f4aa8

newly12 force-pushed the ignore_excluded_paths branch from 05ac7c6 to e0f4aa8 Compare November 8, 2020 11:56

newly12 changed the title ~~ignore fsnotify events for excluded files~~ stop monitoring excluded paths Nov 8, 2020

adriansr approved these changes Nov 9, 2020

View reviewed changes

adriansr merged commit af8eedd into elastic:master Nov 9, 2020

adriansr added bug needs_backport PR is waiting to be backported to other branches. Auditbeat Team:Security-External Integrations and removed Team:SIEM labels Nov 9, 2020

marc-gr mentioned this pull request Nov 9, 2020

Cherry-pick #21282 to 7.x: stop monitoring excluded paths #22486

Merged

6 tasks

marc-gr added v7.11.0 and removed needs_backport PR is waiting to be backported to other branches. labels Nov 9, 2020

newly12 deleted the ignore_excluded_paths branch December 7, 2020 04:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

stop monitoring excluded paths #21282

stop monitoring excluded paths #21282

newly12 commented Sep 24, 2020 •

edited by andresrc

Loading

elasticmachine commented Sep 24, 2020

elasticmachine commented Sep 24, 2020

elasticmachine commented Sep 24, 2020 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

elasticmachine commented Sep 24, 2020

newly12 commented Oct 9, 2020

marc-gr commented Oct 16, 2020

newly12 commented Oct 31, 2020

adriansr Nov 2, 2020 •

edited

Loading

newly12 Nov 5, 2020

adriansr commented Nov 5, 2020

adriansr commented Nov 5, 2020

vjsamuel commented Nov 6, 2020

adriansr commented Nov 6, 2020 •

edited

Loading

elasticmachine commented Nov 6, 2020

Test stats 🧪

adriansr left a comment

newly12 commented Nov 8, 2020

marc-gr commented Nov 9, 2020

elasticmachine commented Nov 9, 2020

stop monitoring excluded paths #21282

stop monitoring excluded paths #21282

Conversation

newly12 commented Sep 24, 2020 • edited by andresrc Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

elasticmachine commented Sep 24, 2020

elasticmachine commented Sep 24, 2020

elasticmachine commented Sep 24, 2020 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

elasticmachine commented Sep 24, 2020

newly12 commented Oct 9, 2020

marc-gr commented Oct 16, 2020

newly12 commented Oct 31, 2020

adriansr Nov 2, 2020 • edited Loading

Choose a reason for hiding this comment

newly12 Nov 5, 2020

Choose a reason for hiding this comment

adriansr commented Nov 5, 2020

adriansr commented Nov 5, 2020

vjsamuel commented Nov 6, 2020

adriansr commented Nov 6, 2020 • edited Loading

elasticmachine commented Nov 6, 2020

💚 Flaky test report

Test stats 🧪

adriansr left a comment

Choose a reason for hiding this comment

newly12 commented Nov 8, 2020

marc-gr commented Nov 9, 2020

elasticmachine commented Nov 9, 2020

newly12 commented Sep 24, 2020 •

edited by andresrc

Loading

elasticmachine commented Sep 24, 2020 •

edited by jenkins-beats-ci bot

Loading

adriansr Nov 2, 2020 •

edited

Loading

adriansr commented Nov 6, 2020 •

edited

Loading