[Auditbeat] User metricset: Fetch groups by user #9732
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cwurm
added
review
needs_backport
|
Pinging @elastic/secops |
webmat
approved these changes
Dec 21, 2018
There was a problem hiding this comment.
The issue's paste of the error truncates the interesting bits, alas 😂
Looking at the failure in TEST-go-unit.out from the zip file (pasted below, non-truncated for posterity), it looks to me like there's actually two errors:
error while reading group file, which you're addressing herefailed to get users, from user:291
I'm not actually sure how to interpret the full stack trace. Is this two goroutines failing at roughly the same time? Or is one failure causing the other?
The tests are now passing, so this looks promising. But I think someone else should have a look as well, to get another opinion.
=== RUN TestData
--- FAIL: TestData (0.05s)
user_test.go:20: received error: numerical result out of range
error while reading group file
github.com/elastic/beats/x-pack/auditbeat/module/system/user.readGroupFile
/var/lib/jenkins/workspace/elastic+beats+pull-request+multijob-linux/beat/x-pack/auditbeat/label/ubuntu/src/github.com/elastic/beats/x-pack/auditbeat/module/system/user/users_linux.go:153
github.com/elastic/beats/x-pack/auditbeat/module/system/user.enrichWithGroups
/var/lib/jenkins/workspace/elastic+beats+pull-request+multijob-linux/beat/x-pack/auditbeat/label/ubuntu/src/github.com/elastic/beats/x-pack/auditbeat/module/system/user/users_linux.go:95
github.com/elastic/beats/x-pack/auditbeat/module/system/user.GetUsers
/var/lib/jenkins/workspace/elastic+beats+pull-request+multijob-linux/beat/x-pack/auditbeat/label/ubuntu/src/github.com/elastic/beats/x-pack/auditbeat/module/system/user/users_linux.go:36
github.com/elastic/beats/x-pack/auditbeat/module/system/user.(*MetricSet).reportState
/var/lib/jenkins/workspace/elastic+beats+pull-request+multijob-linux/beat/x-pack/auditbeat/label/ubuntu/src/github.com/elastic/beats/x-pack/auditbeat/module/system/user/user.go:289
github.com/elastic/beats/x-pack/auditbeat/module/system/user.(*MetricSet).Fetch
/var/lib/jenkins/workspace/elastic+beats+pull-request+multijob-linux/beat/x-pack/auditbeat/label/ubuntu/src/github.com/elastic/beats/x-pack/auditbeat/module/system/user/user.go:270
github.com/elastic/beats/metricbeat/mb/testing.ReportingFetchV2
/var/lib/jenkins/workspace/elastic+beats+pull-request+multijob-linux/beat/x-pack/auditbeat/label/ubuntu/src/github.com/elastic/beats/metricbeat/mb/testing/modules.go:203
github.com/elastic/beats/x-pack/auditbeat/module/system/user.TestData
/var/lib/jenkins/workspace/elastic+beats+pull-request+multijob-linux/beat/x-pack/auditbeat/label/ubuntu/src/github.com/elastic/beats/x-pack/auditbeat/module/system/user/user_test.go:18
testing.tRunner
/var/lib/jenkins/.gvm/versions/go1.11.3.linux.amd64/src/testing/testing.go:827
runtime.goexit
/var/lib/jenkins/.gvm/versions/go1.11.3.linux.amd64/src/runtime/asm_amd64.s:1333
failed to get users
github.com/elastic/beats/x-pack/auditbeat/module/system/user.(*MetricSet).reportState
/var/lib/jenkins/workspace/elastic+beats+pull-request+multijob-linux/beat/x-pack/auditbeat/label/ubuntu/src/github.com/elastic/beats/x-pack/auditbeat/module/system/user/user.go:291
github.com/elastic/beats/x-pack/auditbeat/module/system/user.(*MetricSet).Fetch
/var/lib/jenkins/workspace/elastic+beats+pull-request+multijob-linux/beat/x-pack/auditbeat/label/ubuntu/src/github.com/elastic/beats/x-pack/auditbeat/module/system/user/user.go:270
github.com/elastic/beats/metricbeat/mb/testing.ReportingFetchV2
/var/lib/jenkins/workspace/elastic+beats+pull-request+multijob-linux/beat/x-pack/auditbeat/label/ubuntu/src/github.com/elastic/beats/metricbeat/mb/testing/modules.go:203
github.com/elastic/beats/x-pack/auditbeat/module/system/user.TestData
/var/lib/jenkins/workspace/elastic+beats+pull-request+multijob-linux/beat/x-pack/auditbeat/label/ubuntu/src/github.com/elastic/beats/x-pack/auditbeat/module/system/user/user_test.go:18
testing.tRunner
/var/lib/jenkins/.gvm/versions/go1.11.3.linux.amd64/src/testing/testing.go:827
runtime.goexit
/var/lib/jenkins/.gvm/versions/go1.11.3.linux.amd64/src/runtime/asm_amd64.s:1333
FAIL
FAIL github.com/elastic/beats/x-pack/auditbeat/module/system/user 0.064s
|
@cw Be careful about that Just delete that file and you're good. I just spent 1h+ figuring this out ;-) |
cwurm
force-pushed
the
groups_by_user
branch
from
bf6b6da
to
e0ba204
Compare
|
Thanks @webmat - I've just rebased, hopefully successfully. |
cwurm
pushed a commit
to cwurm/beats
that referenced
this pull request
Jan 3, 2019
Changes the user metricset to looking up groups by user instead of users by groups. Also changes the types of the system.audit.user.uid and system.audit.user.gid fields from integer to keyword to accommodate Windows in the future. Fixes elastic#9679. (cherry picked from commit 42421e9)
cwurm
added
v6.7.0
and removed
needs_backport
cwurm
pushed a commit
to cwurm/beats
that referenced
this pull request
Jan 3, 2019
Changes the user metricset to looking up groups by user instead of users by groups. Also changes the types of the system.audit.user.uid and system.audit.user.gid fields from integer to keyword to accommodate Windows in the future. Fixes elastic#9679. (cherry picked from commit 42421e9)
cwurm
pushed a commit
that referenced
this pull request
Jan 4, 2019
… user (#9870) Cherry-pick of PR #9732 to 6.x branch. Original message: Changes the user metricset to looking up groups by user instead of users by groups. Also changes the types of the system.audit.user.uid and system.audit.user.gid fields from integer to keyword to accommodate Windows in the future. Fixes #9679.
cwurm
pushed a commit
that referenced
this pull request
Jan 4, 2019
… user (#9872) Cherry-pick of PR #9732 to 6.6 branch. Original message: Changes the user metricset to looking up groups by user instead of users by groups. Also changes the types of the system.audit.user.uid and system.audit.user.gid fields from integer to keyword to accommodate Windows in the future. Fixes #9679.
leweafan
pushed a commit
to leweafan/beats
that referenced
this pull request
Apr 28, 2023
…oups by user (elastic#9872) Cherry-pick of PR elastic#9732 to 6.6 branch. Original message: Changes the user metricset to looking up groups by user instead of users by groups. Also changes the types of the system.audit.user.uid and system.audit.user.gid fields from integer to keyword to accommodate Windows in the future. Fixes elastic#9679.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, the
usermetricset reads all users, then reads all groups and their members and matches one to the other. This can be a problem when groups have a lot of members (see #9679).This changes to looking up groups of individual users.
It also changes the types of the
system.audit.user.uidandsystem.audit.user.gidfields fromintegertokeywordto accommodate Windows in the future (Go'sUserandGroupstructs usestrings, so does ECSuser.id/group.id).Because the internal structure of the
Userstruct changes, this invalidates previousbeat.dbfiles. I have not added any conversion logic this time since this metricset is not released yet - but we will have to do it in the future.Fixes #9679.