[Auditbeat] Add user information to processes #9963

cwurm · 2019-01-09T14:55:08Z

Since go-sysinfo can now report the UIDs and GIDs of a process, this adds this information to the process metricset.

The added fields are:

user.id (UID or SID)
user.name
user.group.id (GID or SID of primary group)
user.group.name
user.effective.id (EUID)
user.effective.group.id (EGID)
user.saved.id (SUID)
user.saved.group.id (SGID)

Also adds some unit tests and tightens the system test.

elasticmachine · 2019-01-09T14:55:10Z

Pinging @elastic/secops

webmat

One minor nit that I'm ok if we fix subsequently (you decide).

Otherwise LGTM.

Thanks for the ping, btw, I created elastic/ecs#299

webmat · 2019-01-09T19:24:21Z

x-pack/auditbeat/module/system/process/_meta/data.json

+    },
+    "user": {
+        "id": "1002",
+        "name": "elastic"


Note that ECS now defines the group field set as well. So you should also copy system.audit.process.gid to group.id.

Other than that, this event looks good

Great, I've added group.id and group.name. I'm less comfortable having two duplicated fields (probably shouldn't even had one in the first place since we don't have those in other places either), so I've removed system.audit.process.uid and system.audit.process.gid.

webmat

Yes, this works for me. LGTM

andrewkroh · 2019-01-14T17:56:53Z

Auditbeat's auditd module already defines fields like user.sgid. It would be better to reuse the existing fields this way users can query and get all data that has the given user ID value. You'll need move fields from auditd up to the field.common.yml.

beats/auditbeat/module/auditd/_meta/fields.yml

Line 30 in c956b80

- name: sgid

cwurm · 2019-01-16T11:17:45Z

@andrewkroh You mean extend the top-level user field, right? I don't know about that - to me it feels that euid/suid/egid/sgid are more properties of the process entity, not the user (technically, they're also different users/accounts).

I took a look at the auditd module's sample data - it does not look ECS compliant to me? Fields like user.name_map and user.uid (instead of user.id) are not in ECS, and are not correctly documented in the fields.yml either (they're documented under the auditd.user namespace, but are actually in the top-level user object). That should not be I think? In Metricbeat and the Auditbeat system module, the system tests call Libbeat's assert_fields_are_documented(evt) to make sure that does not happen. Maybe this is something we should do for auditd as well?

cwurm · 2019-01-16T11:31:16Z

I don't know about that - to me it feels that euid/suid/egid/sgid are more properties of the process entity, not the user (technically, they're also different users/accounts).

I could see how they would be part of the user object if that one is treated more like "any user information" rather than "the user entity". I wonder what it should look like then? The challenge is that at the moment ECS assumes the user fields to represent a single user, with one id, one name, one email, etc. And now we have multiple values.

If we follow Auditd, it would probably be:

"user": {
  "id": "1000",
  "name": "elastic",
  "euid": "1000",
  "suid": "1000",
  "name_map": {
    "euid": "elastic",
    "suid": "elastic"
  }

But I wonder, what if we ever wanted add more information than the name? Would we need more maps?

Maybe it could allow nesting, e.g.:

"user": {
  "id": "1000",
  "name": "elastic",
  "email": "elastic@localhost",    // example additional field
  "created": "2019-01-01 19:37:56" // example additional field
  "effective": { // Nested user object
    "id": "1000",
    "name": "elastic",
    "email": "elastic@localhost",    // example additional field
    "created": "2019-01-01 19:37:56" // example additional field
  },
  "saved": { // Nested user object
    "id": "1000",
    "name": "elastic",
    "email": "elastic@localhost",    // example additional field
    "created": "2019-01-01 19:37:56" // example additional field
  }
}

The group object is also affected, but would probably just follow whatever we do for user.

Anyway, if this is a change to the ECS user object, maybe this discussion has to happen in the ECS repo?

webmat · 2019-01-16T21:53:15Z

Yeah I addressed these fields under user in my review of Auditbeat/auditd here: #10111.

Waiting on input from Nic on it before bugging you guys. But my conclusion over in that issue is that despite not being in ECS (yet?), I don't think it's a problem. Read up over there for my thinking on this.

cwurm · 2019-01-17T10:02:40Z

@webmat I see that in #10111 we have the group IDs under user as well: user.gid, user.egid, etc.

Whereas here we concluded to have them under group.

I'm confused now - I think we should have one or the other, and ECS should specify which.

Personally, I like the idea of having all user information under user and not have a top-level group object at all.

webmat · 2019-01-17T16:04:06Z

To follow up on the discussion we've had elsewhere, I would also recommend populating group.id with user.egid. This will help correlation with other places that are using group.id this way.

I agree that ECS will need to:

define how to represent user privilege escalation
clarify or review whether a user's group ID, when seen on an event belongs at group.id or nested under the user. This work will be related to the privilege escalation work. Just like with user IDs, multiple group IDs may have to be considered before determining the effective group ID

@cwurm I also think you mentioned that the sets of IDs represented here for privilege escalation is different than what has been happening in the new system module. I agree we would benefit from harmonizing this. However since this isn't in ECS yet and we can't tackle this before FF, I would say it's a decision that is up to you and Andrew on whether to move the fields around on one of the modules to align with the other.

Although you made the point that these details may make more sense under process. than under user., which also makes sense. Perhaps we touch neither of them for now?

webmat · 2019-01-17T16:16:30Z

Oops that first paragraph was meant as a recommendation for #10111. You're already doing this here :-) I have too many PRs to review LOL

webmat · 2019-01-17T16:17:46Z

Actually this whole comment was written for #10111 :-)

cwurm · 2019-01-21T15:23:08Z

I've updated to the nested user format proposed in #10111 (comment) and rebased on master.

It would be nice to have resolved user and group names for effective and saved UIDs and GIDs (user.effective.name, user.effective.group.name, user.saved.name, user.saved.group.name), but I didn't want to do it here without a cache in place for those lookups. Something for a follow-up though.

webmat

Loving this representation of all the considered users, in determining the effective rights.

Let's wait for people's input tomorrow, when the US isn't on holiday. But I think this is good to go :-)

LGTM

webmat · 2019-01-21T18:28:28Z

x-pack/auditbeat/module/system/process/_meta/data.json

+                "id": "1000"
+            },
+            "id": "1000"
+        }


andrewkroh

LGTM. Can you please update the PR description to reflect the final format used in the PR.

webmat · 2019-01-23T20:58:20Z

Agree with @andrewkroh.

However this PR depends on #10275

webmat

LGTM, but a few points:

A rebase will get you green again
Please adjust the PR description with the final outcome
I think we're fine like this, wrt field definitions. ECS change was only adding user.group.id and user.group.name. You're defining everything you need here already. I'm defining them locally to Filebeat for the Fb auditd module.

This reverts commit eb8f7bfbc38d5d4e7a6cca9e867105f51991457c.

cwurm · 2019-01-28T12:32:23Z

jenkins, test this

cwurm · 2019-01-28T14:14:10Z

jenkins, test this

Adds real, effective, and saved UID and GID information to the process dataset. (cherry picked from commit fa40a54)

…es (#10395) Adds real, effective, and saved UID and GID information to the process dataset. (cherry picked from commit fa40a54, then adjusted)

cwurm added review needs_backport PR is waiting to be backported to other branches. Auditbeat SecOps labels Jan 9, 2019

cwurm requested review from a team as code owners January 9, 2019 14:55

urso removed the request for review from a team January 9, 2019 16:17

webmat mentioned this pull request Jan 9, 2019

Consider adding support for effective&saved ID fields to process elastic/ecs#299

Closed

webmat approved these changes Jan 9, 2019

View reviewed changes

webmat approved these changes Jan 10, 2019

View reviewed changes

cwurm requested a review from a team January 11, 2019 12:30

cwurm mentioned this pull request Jan 14, 2019

[Auditbeat] Fix skipping processes on permission error #10048

Merged

This was referenced Jan 16, 2019

[Auditbeat] System module 6.6 #8725

Closed

[Auditbeat] System module 6.7 / 7.0 #10103

Closed

cwurm mentioned this pull request Jan 18, 2019

[Auditbeat] Review Auditbeat's auditd and FIM modules with regards to conforming to ECS #10111

Closed

cwurm force-pushed the process_user_info branch from 9804719 to b5319d7 Compare January 21, 2019 15:18

cwurm mentioned this pull request Jan 21, 2019

Convert the Filebeat auditd module to ECS #10192

Merged

8 tasks

webmat approved these changes Jan 21, 2019

View reviewed changes

x-pack/auditbeat/module/system/process/_meta/data.json

"id": "1000"

},

"id": "1000"

}

Copy link

Contributor

webmat Jan 21, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❤️

webmat mentioned this pull request Jan 21, 2019

user.group should not be a keyword field, but a place where we can nest the group field set elastic/ecs#304

Closed

cwurm requested a review from a team January 22, 2019 09:58

andrewkroh approved these changes Jan 23, 2019

View reviewed changes

webmat mentioned this pull request Jan 24, 2019

Import user.group changes from ECS #10275

Merged

cwurm force-pushed the process_user_info branch from b5319d7 to f92d87e Compare January 25, 2019 10:21

webmat approved these changes Jan 25, 2019

View reviewed changes

Christoph Wurm added 12 commits January 28, 2019 12:11

Add user info.

0bc232d

Tests

6bb790a

Move fields to MetricSetFields, add unit test.

3116ae7

Update field docs

abfd4d2

Allow error field in socket metricset.

94c10af

Add group.id and group.name

ca51586

Host Overview dashboard and visualizations.

0114e84

Adjust fields.

baef1b4

Field docs

9b096a4

Remove superfluous user.group.id and user.group.name

9cf0979

Revert "Host Overview dashboard and visualizations."

22a1b79

This reverts commit eb8f7bfbc38d5d4e7a6cca9e867105f51991457c.

Use User.Name if User.Username is empty.

1c0d8f1

cwurm force-pushed the process_user_info branch from 0397c47 to 1c0d8f1 Compare January 28, 2019 12:13

Changelog

b8487d3

Fix system test on Windows.

6232e2e

cwurm merged commit fa40a54 into elastic:master Jan 29, 2019

cwurm deleted the process_user_info branch January 29, 2019 12:33

cwurm pushed a commit to cwurm/beats that referenced this pull request Jan 29, 2019

[Auditbeat] Add user information to processes (elastic#9963)

b2e4866

Adds real, effective, and saved UID and GID information to the process dataset. (cherry picked from commit fa40a54)

cwurm mentioned this pull request Jan 29, 2019

[Auditbeat] Cherry-pick #9963 to 6.x: Add user information to processes #10395

Merged

cwurm added v6.7.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jan 29, 2019

webmat mentioned this pull request Mar 27, 2020

[ECS] Multiple users in an event proposal elastic/ecs#809

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Auditbeat] Add user information to processes #9963

[Auditbeat] Add user information to processes #9963

cwurm commented Jan 9, 2019 •

edited

Loading

elasticmachine commented Jan 9, 2019

webmat left a comment

webmat Jan 9, 2019

cwurm Jan 10, 2019

webmat left a comment

andrewkroh commented Jan 14, 2019

cwurm commented Jan 16, 2019

cwurm commented Jan 16, 2019 •

edited

Loading

webmat commented Jan 16, 2019

cwurm commented Jan 17, 2019

webmat commented Jan 17, 2019 •

edited

Loading

webmat commented Jan 17, 2019

webmat commented Jan 17, 2019 •

edited

Loading

cwurm commented Jan 21, 2019

webmat left a comment

webmat Jan 21, 2019

andrewkroh left a comment

webmat commented Jan 23, 2019 •

edited

Loading

webmat left a comment

cwurm commented Jan 28, 2019

cwurm commented Jan 28, 2019

[Auditbeat] Add user information to processes #9963

[Auditbeat] Add user information to processes #9963

Conversation

cwurm commented Jan 9, 2019 • edited Loading

elasticmachine commented Jan 9, 2019

webmat left a comment

Choose a reason for hiding this comment

webmat Jan 9, 2019

Choose a reason for hiding this comment

cwurm Jan 10, 2019

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

andrewkroh commented Jan 14, 2019

cwurm commented Jan 16, 2019

cwurm commented Jan 16, 2019 • edited Loading

webmat commented Jan 16, 2019

cwurm commented Jan 17, 2019

webmat commented Jan 17, 2019 • edited Loading

webmat commented Jan 17, 2019

webmat commented Jan 17, 2019 • edited Loading

cwurm commented Jan 21, 2019

webmat left a comment

Choose a reason for hiding this comment

webmat Jan 21, 2019

Choose a reason for hiding this comment

andrewkroh left a comment

Choose a reason for hiding this comment

webmat commented Jan 23, 2019 • edited Loading

webmat left a comment

Choose a reason for hiding this comment

cwurm commented Jan 28, 2019

cwurm commented Jan 28, 2019

cwurm commented Jan 9, 2019 •

edited

Loading

cwurm commented Jan 16, 2019 •

edited

Loading

webmat commented Jan 17, 2019 •

edited

Loading

webmat commented Jan 17, 2019 •

edited

Loading

webmat commented Jan 23, 2019 •

edited

Loading