[Auditbeat] Process: Add hash of executable

[cwurm]

Adds the hash(es) of the process executable to process.hash.*. The default is to add SHA-1 only as process.hash.sha1.

Design

Re-using the hashing code from file_integrity (see next section for some of the copied places) introduces a FileHasher type in a new package auditbeat/helper/hasher. Configured using its own Config and created using NewFileHasher(config, done), it has only one other useful method, HashFile(path) that does the hashing incl. throttling.

In the future, we could re-use this hasher in other places (e.g. auditd.paths, Winlogbeat's process.[parent.]executable, the add_process_metadata processor).

Executable hashing can be quite expensive. When hashing all execs of all about 450 running processes on my macOS it takes about 10 seconds. That's why I moved the hashing logic to an enrichProcess() function that is only called when a new process is discovered. When the dataset is run for the first time, it will take a bit, but afterwards, it should be much better.

Code

The code is mostly copied from the file_integrity module:

• Hashing

  beats/auditbeat/module/file_integrity/event.go

  Line 357 in bb3f116

  func hashFile(name string, hashType ...HashType) (map[HashType]Digest, error) {

• Configuration

  beats/auditbeat/module/file_integrity/config.go

  Lines 32 to 33 in bb3f116

  	// HashType identifies a cryptographic algorithm.
  	type HashType string

• Throttling

  beats/auditbeat/module/file_integrity/scanner.go

  Line 178 in bb3f116

  func (s *scanner) throttle(fileSize uint64) {

Configuration

3 new parameters, same as for file_integrity just with a process.hash prefix:

• process.hash.hash_types (default [sha1])
• process.hash.max_file_size (default 100 MiB)
• process.hash.scan_rate_per_sec (default 50 MiB)

Follow up

Two immediate ones:

1. Refactor the file_integrity module to use the new FileHasher. I tried doing this in this PR, but it became quite complex surprisingly quickly. The hashing logic is strewn across various files and types.
2. Introduce a hash_max_delay (naming tbd) parameter to configure how long to wait for a hashing to happen (based on scan_rate_per_sec which we honor here even for regular process events, something we might want to do in file_integrity as well). At the moment, it seems the hashing time is essentially unbounded - there is max_file_size, but given enough files hashing could take over CPU and I/O.

@andrewkroh @adriansr - adding you both as reviewers since you're both familiar with the hashing code from file_integrity.

[elasticmachine]

Pinging @elastic/secops

[adriansr]

LGTM except for the CHANGELOG entry. Feel free to ignore the other two comments if you don't have the time to add new behavior or code cleanup. It's a refactor after all.

[adriansr]

I think this is too cryptic. Can you make it more descriptive? "Support hashing the process executable" or similar.

[cwurm]

I've changed to Process: Add file hash of process executable..

[adriansr]

jenkins, test this.

[codecov]

Codecov Report

Merging #11722 into master will increase coverage by <.01%.
The diff coverage is 38.59%.

@@            Coverage Diff             @@
##           master   #11722      +/-   ##
==========================================
+ Coverage   57.25%   57.26%   +<.01%     
==========================================
  Files         901      904       +3     
  Lines       56056    56176     +120     
==========================================
+ Hits        32095    32167      +72     
- Misses      21036    21089      +53     
+ Partials     2925     2920       -5

Impacted Files	Coverage Δ
auditbeat/helper/hasher/hasher.go	38.59% <38.59%> (ø)
auditbeat/module/file_integrity/action.go	68.42% <0%> (-12.29%)	⬇️
metricbeat/module/docker/memory/helper.go	89.47% <0%> (-10.53%)	⬇️
libbeat/publisher/queue/memqueue/ackloop.go	93.1% <0%> (-3.45%)	⬇️
metricbeat/module/zookeeper/server/data.go	55.75% <0%> (ø)	⬆️
metricbeat/module/docker/memory/memory.go	70.37% <0%> (ø)	⬆️
metricbeat/module/system/system_other.go	0% <0%> (ø)
metricbeat/module/http/json/data.go	17.14% <0%> (ø)
libbeat/publisher/queue/spool/inbroker.go	68.35% <0%> (+1.17%)	⬆️
... and 14 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bb3f116...99cd81d. Read the comment docs.

[cwurm]

1. Rebased on master to resolve conflicts.
2. Simplified hash construction - thanks @adriansr!
3. Moved detailed hash config to auditbeat.reference.yml - same as for file_integrity.

[andrewkroh]

LGTM. I recommend opening an issue to track the follow up items. It would be nice to be able to share this code with the file_integrity module to reduce duplication and simplify maintenance.

[andrewkroh]

q: What's does the ,replace option do? I haven't see that used before.

[cwurm]

replace allows process.hash.hash_types to be empty and disable hashing altogether that way (e.g. if there is a performance problem, or simply to restore current behavior). Only when the config option is not present at all does the default (sha1) take effect.

Without replace, the default value will be used even when the option is present but empty, so there would be no way of disabling it.

I would prefer something called allow_empty or similar, but replace is what I found works as well.

@urso maybe you can take a quick look at whether this is the right use.

[willemdh]

@cwurm

What kind of hashes will the functionality allow to create? Any chance and imphash is on the roadmap?

"The imphash value can be used to identify malware samples that are related to one another. It can also be utilized to identify similar samples created by a certain threat group."

Maybe not ideal for high granluarity process auditing, but combine this with the file_integrity module and you could kind of create your own antivirus with VirusTotal imphash lookups in a Logstash pipeline

https://www.virustotal.com/intelligence/help/