Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Convert the Filebeat auditd module to ECS #10192

Merged
merged 22 commits into from
Jan 30, 2019
Merged

Convert the Filebeat auditd module to ECS #10192

Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
1cd825c
First draft at migrating Filebeat's auditd module to ECS
Jan 19, 2019
269dc62
Add missing coercions for process ids
Jan 19, 2019
8f4d3fb
Add log file with more exciting content
Jan 21, 2019
a1c57a1
Update the user structure to represent what was discussed with @cwurm:
Jan 21, 2019
8941175
Turn fields that were defined into aliases...
Jan 21, 2019
395b076
Add a bunch of fields that weren't defined, but are being migrated.
Jan 21, 2019
ce10c80
Add missing `migration: true` to the fields.yml
Jan 21, 2019
1f981f9
Get rid of the `user.group_` workaround.
Jan 23, 2019
6770379
Arch goes to host.architecture, not host.os.architecture.
Jan 23, 2019
d638458
Document the field migrations in ecs-migration
Jan 23, 2019
c33fd94
Changelog
Jan 23, 2019
c233180
Define the fields representing the various permissions considered for…
Jan 25, 2019
4917ea2
tty and terminal are mapped to user.terminal instead of process.terminal
Jan 29, 2019
6482222
Add a few more interesting logs to the main test log
Jan 29, 2019
40963c2
Dig up a few more fields to transition.
Jan 29, 2019
b45465a
Fix big mistake: module's main fields def must end with opening of th…
Jan 29, 2019
b5753d5
Update test files with all of tonight's changes
Jan 29, 2019
3c762dc
Update reference documentation vs the auditd/_meta/fields.yml fix
Jan 29, 2019
adca8c8
Try getting the fields.yml right
Jan 29, 2019
3aa9189
Revert the tty => terminal mapping
Jan 29, 2019
b8e7a76
Undo another dubious translation: the local/remote address details
Jan 29, 2019
3b9eaf6
2nd changelog for the improvements
Jan 29, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.next.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Rename a few `mongodb.*` fields to map to ECS. {pull}10009[10009]
- Rename a few `mysql.*` fields to map to ECS. {pull}10008[10008]
- Rename a few `nginx.error.*` fields to map to ECS. {pull}10007[10007]
- Rename many `auditd.log.*` fields to map to ECS. {pull}10192[10192]
- Filesets with multiple ingest pipelines added in {pull}8914[8914] only work with Elasticsearch >= 6.5.0 {pull}10001[10001]
- Remove service.name from Elastcsearch module. Replace by service.type. {pull}10042[10042]
- Remove numeric coercions for `user.id` and `group.id`. IDs should be `keyword`. {pull}10233[10233]
Expand Down
178 changes: 178 additions & 0 deletions dev-tools/ecs-migration.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,184 @@

# Filebeat modules

# Auditd module

- from: auditd.log.acct
to: user.name
alias: true
beat: filebeat

- from: auditd.log.pid
to: process.pid
alias: true
beat: filebeat

- from: auditd.log.ppid
to: process.ppid
alias: true
beat: filebeat

- from: auditd.log.res
to: event.outcome
alias: true
beat: filebeat

- from: auditd.log.record_type
to: event.action
alias: true
beat: filebeat

- from: auditd.log.arch
to: host.architecture
alias: true
beat: filebeat

- from: auditd.log.gid
to: user.group.id
alias: true
beat: filebeat

- from: auditd.log.uid
to: user.id
alias: true
beat: filebeat

- from: auditd.log.agid
to: user.audit.group.id
alias: true
beat: filebeat

- from: auditd.log.auid
to: user.audit.id
alias: true
beat: filebeat

- from: auditd.log.fsgid
to: user.filesystem.group.id
alias: true
beat: filebeat

- from: auditd.log.egid
to: user.effective.group.id
alias: true
beat: filebeat

- from: auditd.log.euid
to: user.effective.id
alias: true
beat: filebeat

- from: auditd.log.sgid
to: user.saved.group.id
alias: true
beat: filebeat

- from: auditd.log.suid
to: user.saved.id
alias: true
beat: filebeat

- from: auditd.log.ogid
to: user.owner.group.id
alias: true
beat: filebeat

- from: auditd.log.ouid
to: user.owner.id
alias: true
beat: filebeat

- from: auditd.log.terminal
to: user.terminal
alias: true
beat: filebeat

- from: auditd.log.tty
to: user.terminal
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure terminal and tty will never be both filled? What would happen if they were?

Copy link
Contributor Author

@webmat webmat Jan 29, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I agree it's not perfect. Here's what I considered:

  • Based on the logs samples we have, only one of them is used at a time.
  • Based on the docs I've seen, there isn't a big difference between the two (one's the "terminal", the other's the "terminal device"). I get the sense different auditd event sources use one of the two interchangeably. Even if you look for terminal= in the rhel7 log file, you'll see "/dev/pts/0" and "pts/0" being used interchangeably.

Another thing we could do is append it instead. So if both are set at the same time, you'll get `user.terminal: ["pts0", "/dev/pts/0"] instead of one of the two being overwritten.

Or perhaps we can leave tty where it is for now, only map terminal to user.terminal for now and revisit this later.

The last option seems the most straightforward in that it's making sure we don't introduce a bug (the overwrite), while still taking a step towards normalization (at least mapping terminal).

Finally, the same potential overwrite is happening with the IP addresses. It's hard to tell from the PR body because the fields are sorted alphabetically. But in the pipeline it's more visible, as they're grouped by concern. Address manipulation is starting here

alias: true
beat: filebeat

- from: auditd.log.comm
to: process.name
alias: true
beat: filebeat

- from: auditd.log.cmd
to: process.args
alias: false
beat: filebeat
comment: Was a cmdline string, whereas args is an array of keywords.

- from: auditd.log.exe
to: process.executable
alias: true
beat: filebeat

- from: auditd.log.msg
to: message
alias: true
beat: filebeat

- from: auditd.log.src
to: source.address
alias: true
beat: filebeat

- from: auditd.log.addr
to: source.address
alias: true
beat: filebeat

- from: auditd.log.rport
to: source.port
alias: true
beat: filebeat

- from: auditd.log.dst
to: destination.address
alias: true
beat: filebeat

- from: auditd.log.laddr
to: destination.address
alias: true
beat: filebeat

- from: auditd.log.lport
to: destination.port
alias: true
beat: filebeat

- from: auditd.log.geoip.continent_name
to: source.geo.continent_name
alias: true
beat: filebeat

- from: auditd.log.geoip.country_iso_code
to: source.geo.country_iso_code
alias: true
beat: filebeat

- from: auditd.log.geoip.location
to: source.geo.location
alias: true
beat: filebeat

- from: auditd.log.geoip.region_name
to: source.geo.region_name
alias: true
beat: filebeat

- from: auditd.log.geoip.city_name
to: source.geo.city_name
alias: true
beat: filebeat

- from: auditd.log.geoip.region_iso_code
to: source.geo.region_iso_code
alias: true
beat: filebeat

# Suricata module

- from: source_ecs.ip
Expand Down
Loading