Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modified auditd ingest pipeline to handle node=hostname format #19659

Merged
merged 7 commits into from
Aug 10, 2020

Conversation

ipnerds
Copy link
Contributor

@ipnerds ipnerds commented Jul 5, 2020

What does this PR do?

This change permits auditd logs to be ingested whether or not they begin with a node=. Previous to this change a type= was expected.

It handles the case where name_format (man 5 auditd.conf) is a value other than none.

Also adding 10 lines of resulting audit log with name_format = hostname parameter on CentOS 7.

Why is it important?

When running auditd with name_format other than none, the log messages will begin with node= parameter rather than type=. This behavior caused such audit logs to fail ingestion. The use case for setting name_format would be with an audit server collecting audit logs from multiple hosts.

Checklist

  • [ X] My code follows the style guidelines of this project
  • [ ] I have commented my code, particularly in hard-to-understand areas
    -~~ [ ] I have made corresponding changes to the documentation~~
    -~~ [ ] I have made corresponding change to the default configuration files~~
    -~~ [ ] I have added tests that prove my fix is effective or that my feature works~~
    -~~ [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

…(man 5 auditd.conf) is a value other than none.
@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

1 similar comment
@elasticmachine
Copy link
Collaborator

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 5, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Jul 5, 2020

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19659 updated]

  • Start Time: 2020-08-10T20:51:33.494+0000

  • Duration: 52 min 54 sec

Test stats 🧪

Test Results
Failed 1
Passed 5524
Skipped 846
Total 6371

Test errors

Expand to view the tests failures

  • Name: Build and Test / Filebeat oss / test_default_settings – filebeat.tests.system.test_autodiscover.TestAutodiscover

    • Age: 1
    • Duration: 90.003
    • Error Details: Failed: Timeout >90.0s

Steps errors

Expand to view the steps failures

  • Name: Mage build test
    • Description: mage build test

    • Duration: 27 min 6 sec

    • Start Time: 2020-08-10T21:15:41.061+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-08-10T21:42:34.116Z] ======= 1 failed, 314 passed, 5 skipped, 7 warnings in 632.96s (0:10:32) =======
[2020-08-10T21:42:34.116Z] >> python test: Integration Testing Complete
[2020-08-10T21:42:34.718Z] Error: running "/go/src/github.com/elastic/beats/build/ve/docker/bin/pytest --timeout=90 --durations=20 --junit-xml=build/TEST-python-integration.xml tests/system/test_autodiscover.py tests/system/test_base.py tests/system/test_cmd.py tests/system/test_container.py tests/system/test_crawler.py tests/system/test_deprecated.py tests/system/test_fields.py tests/system/test_generate.py tests/system/test_harvester.py tests/system/test_index_pattern.py tests/system/test_input.py tests/system/test_json.py tests/system/test_keystore.py tests/system/test_load.py tests/system/test_modules.py tests/system/test_multiline.py tests/system/test_pipeline.py tests/system/test_processors.py tests/system/test_publisher.py tests/system/test_redis.py tests/system/test_registrar.py tests/system/test_registrar_upgrade.py tests/system/test_reload_inputs.py tests/system/test_reload_modules.py tests/system/test_setup.py tests/system/test_shutdown.py tests/system/test_stdin.py tests/system/test_syslog.py tests/system/test_tcp.py tests/system/test_tcp_tls.py tests/system/test_udp.py tests/system/test_unix.py" failed with exit code 1
[2020-08-10T21:42:46.982Z] Error: running "docker-compose -p filebeat_8_0_0_88611ac3a9-snapshot run -e DOCKER_COMPOSE_PROJECT_NAME=filebeat_8_0_0_88611ac3a9-snapshot -e BEAT_STRICT_PERMS=false -e STACK_ENVIRONMENT=snapshot -e TESTING_ENVIRONMENT=snapshot -e GOCACHE=/go/src/github.com/elastic/beats/build/docker-gocache -v /var/lib/jenkins/workspace/Beats_beats_PR-19659/pkg/mod/cache/download:/gocache:ro -e GOPROXY=file:///gocache,direct -e EXEC_UID=1154 -e EXEC_GID=1155 -e BEATS_INSIDE_INTEGRATION_TEST_ENV=true -e GOFLAGS=-mod=readonly -e TEST_COVERAGE=true -e RACE_DETECTOR=true -e TEST_TAGS=null,oracle -e MODULE=auditd beat /go/src/github.com/elastic/beats/filebeat/build/mage-linux-amd64 pythonIntegTest" failed with exit code 1
[2020-08-10T21:42:47.371Z] Client: Docker Engine - Community
[2020-08-10T21:42:47.371Z]  Version:           19.03.12
[2020-08-10T21:42:47.371Z]  API version:       1.40
[2020-08-10T21:42:47.371Z]  Go version:        go1.13.10
[2020-08-10T21:42:47.371Z]  Git commit:        48a66213fe
[2020-08-10T21:42:47.371Z]  Built:             Mon Jun 22 15:45:36 2020
[2020-08-10T21:42:47.371Z]  OS/Arch:           linux/amd64
[2020-08-10T21:42:47.371Z]  Experimental:      false
[2020-08-10T21:42:47.371Z] 
[2020-08-10T21:42:47.371Z] Server: Docker Engine - Community
[2020-08-10T21:42:47.371Z]  Engine:
[2020-08-10T21:42:47.371Z]   Version:          19.03.12
[2020-08-10T21:42:47.371Z]   API version:      1.40 (minimum version 1.12)
[2020-08-10T21:42:47.371Z]   Go version:       go1.13.10
[2020-08-10T21:42:47.371Z]   Git commit:       48a66213fe
[2020-08-10T21:42:47.371Z]   Built:            Mon Jun 22 15:44:07 2020
[2020-08-10T21:42:47.371Z]   OS/Arch:          linux/amd64
[2020-08-10T21:42:47.371Z]   Experimental:     false
[2020-08-10T21:42:47.371Z]  containerd:
[2020-08-10T21:42:47.371Z]   Version:          1.2.13
[2020-08-10T21:42:47.371Z]   GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
[2020-08-10T21:42:47.371Z]  runc:
[2020-08-10T21:42:47.371Z]   Version:          1.0.0-rc10
[2020-08-10T21:42:47.371Z]   GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
[2020-08-10T21:42:47.371Z]  docker-init:
[2020-08-10T21:42:47.371Z]   Version:          0.18.0
[2020-08-10T21:42:47.371Z]   GitCommit:        fec3683
[2020-08-10T21:42:47.371Z] Unable to find image 'alpine:3.4' locally
[2020-08-10T21:42:47.942Z] 3.4: Pulling from library/alpine
[2020-08-10T21:42:48.203Z] c1e54eec4b57: Pulling fs layer
[2020-08-10T21:42:48.464Z] c1e54eec4b57: Download complete
[2020-08-10T21:42:48.464Z] c1e54eec4b57: Pull complete
[2020-08-10T21:42:48.464Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2020-08-10T21:42:48.464Z] Status: Downloaded newer image for alpine:3.4
[2020-08-10T21:42:50.155Z] + python .ci/scripts/pre_archive_test.py
[2020-08-10T21:42:51.101Z] Copy ./filebeat/build into build/filebeat/build
[2020-08-10T21:42:51.112Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19659/src/github.com/elastic/beats/build
[2020-08-10T21:42:51.138Z] Recording test results
[2020-08-10T21:42:53.442Z] Stashed 4 file(s)
[2020-08-10T21:42:53.453Z] Archiving artifacts
[2020-08-10T21:42:54.032Z] + python .ci/scripts/search_system_tests.py
[2020-08-10T21:42:54.049Z] [INFO] system-tests='build/filebeat/build/system-tests'. If no empty then let's create a tarball
[2020-08-10T21:42:54.385Z] + tar --version
[2020-08-10T21:42:54.705Z] + tar --exclude=filebeat--system-tests-linux.tgz -czf filebeat--system-tests-linux.tgz build/filebeat/build/system-tests
[2020-08-10T21:43:00.011Z] Archiving artifacts
[2020-08-10T21:43:04.463Z] Failed in branch Filebeat oss
[2020-08-10T21:43:04.604Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19659/src/github.com/elastic/beats
[2020-08-10T21:43:04.915Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-08-10T21:43:04.928Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19659/src/github.com/elastic/beats/Lint
[2020-08-10T21:43:05.015Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19659/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-08-10T21:43:05.100Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19659/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-08-10T21:43:05.182Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19659/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-08-10T21:43:05.275Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19659/src/github.com/elastic/beats/Filebeat-Windows
[2020-08-10T21:43:05.368Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19659/src/github.com/elastic/beats/Filebeat-x-pack
[2020-08-10T21:43:05.449Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-19659/src/github.com/elastic/beats/Filebeat-oss
[2020-08-10T21:43:05.819Z] + cat
[2020-08-10T21:43:05.819Z] + /usr/local/bin/runbld ./runbld-script
[2020-08-10T21:43:05.819Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-08-10T21:43:12.418Z] runbld>>> runbld started
[2020-08-10T21:43:12.418Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-08-10T21:43:13.797Z] runbld>>> The following profiles matched the job 'Beats/beats/PR-19659' in order of occurrence in the config (last value wins).
[2020-08-10T21:43:14.735Z] runbld>>> Debug logging enabled.
[2020-08-10T21:43:14.996Z] runbld>>> Storing result
[2020-08-10T21:43:14.996Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-08-10T21:43:14.996Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200810214314-34680958
[2020-08-10T21:43:14.996Z] runbld>>> Adding system facts.
[2020-08-10T21:43:16.382Z] runbld>>> Adding vcs info for the latest commit:  88611ac3a9c9e8589e7d7374f019bca00fafc18d
[2020-08-10T21:43:16.382Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-08-10T21:43:16.382Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-08-10T21:43:16.382Z] Processing JUnit reports with runbld...
[2020-08-10T21:43:16.382Z] + echo 'Processing JUnit reports with runbld...'
[2020-08-10T21:43:16.646Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-08-10T21:43:16.646Z] runbld>>> DURATION: 21ms
[2020-08-10T21:43:16.646Z] runbld>>> STDOUT: 40 bytes
[2020-08-10T21:43:16.646Z] runbld>>> STDERR: 49 bytes
[2020-08-10T21:43:16.646Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-08-10T21:43:16.646Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-19659/src/github.com/elastic/beats
[2020-08-10T21:43:17.587Z] runbld>>> Storing build metadata: 
[2020-08-10T21:43:17.587Z] runbld>>> Adding test report.
[2020-08-10T21:43:17.587Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-19659/src/github.com/elastic/beats
[2020-08-10T21:43:18.533Z] runbld>>> Found 16 test output files
[2020-08-10T21:43:19.915Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 6371 Skipped: 820
[2020-08-10T21:43:20.174Z] runbld>>> Storing result
[2020-08-10T21:43:20.174Z] runbld>>> FAILURES: 1
[2020-08-10T21:43:20.433Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-08-10T21:43:20.433Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200810214314-34680958
[2020-08-10T21:43:20.693Z] runbld>>> Email notification disabled by environment variable.
[2020-08-10T21:43:20.693Z] runbld>>> Slack notification disabled by environment variable.
[2020-08-10T21:43:26.653Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-19659
[2020-08-10T21:43:26.880Z] [INFO] getVaultSecret: Getting secrets
[2020-08-10T21:43:26.958Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-08-10T21:43:27.732Z] + chmod 755 generate-build-data.sh
[2020-08-10T21:43:27.733Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19659/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19659/runs/7 FAILURE 3113973
[2020-08-10T21:43:27.733Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19659/runs/7/steps/?limit=10000 -o steps-info.json
[2020-08-10T21:43:28.283Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19659/runs/7/tests/?status=FAILED -o tests-errors.json
[2020-08-10T21:43:28.834Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-19659/runs/7/log/ -o pipeline-log.txt

@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 6, 2020
Copy link
Member

@andrewkroh andrewkroh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please add a sample log file to https://github.com/elastic/beats/tree/117a97e23de9ec3617a9c6a05eff975726776bea/filebeat/module/auditd/log/test so that the change can be tested.

We'll update the PR afterwards to add the generated file (it's a matter of running GENERATE=true NOSE_TESTMATCH=auditd mage -d filebeat -v pythonIntegTest).

@ipnerds
Copy link
Contributor Author

ipnerds commented Jul 6, 2020 via email

@marc-gr
Copy link
Contributor

marc-gr commented Jul 15, 2020

Thanks for the test samples @ipnerds ! I generated the test files and waiting for CI now.

@marc-gr
Copy link
Contributor

marc-gr commented Jul 15, 2020

jenkins run tests

@marc-gr marc-gr requested a review from andrewkroh July 15, 2020 09:12
@marc-gr
Copy link
Contributor

marc-gr commented Jul 22, 2020

jenkins run tests

@andrewkroh andrewkroh merged commit b48be52 into elastic:master Aug 10, 2020
@andrewkroh andrewkroh changed the title Modified auditd ingest pipeline to handle the case where name_format … Modified auditd ingest pipeline to handle node=hostname format Aug 10, 2020
@andrewkroh andrewkroh added the needs_backport PR is waiting to be backported to other branches. label Aug 10, 2020
andrewkroh pushed a commit to andrewkroh/beats that referenced this pull request Aug 10, 2020
This change permits auditd logs to be ingested whether or not they begin with a node=. Previous to this change a type= was expected.

It handles the case where name_format (man 5 auditd.conf) is a value other than none.

Also adding 10 lines of resulting audit log with name_format = hostname parameter on CentOS 7.

Co-authored-by: Marc Guasch <marc.guasch@elastic.co>
(cherry picked from commit b48be52)
@andrewkroh andrewkroh added v7.10.0 and removed needs_backport PR is waiting to be backported to other branches. labels Aug 10, 2020
andrewkroh added a commit that referenced this pull request Aug 11, 2020
…node=hostname format (#20526)

* Modified auditd ingest pipeline to handle node=hostname (#19659)

This change permits auditd logs to be ingested whether or not they begin with a node=. Previous to this change a type= was expected.

It handles the case where name_format (man 5 auditd.conf) is a value other than none.

Also adding 10 lines of resulting audit log with name_format = hostname parameter on CentOS 7.

Co-authored-by: Marc Guasch <marc.guasch@elastic.co>
(cherry picked from commit b48be52)

* Update CHANGELOG.next.asciidoc

Co-authored-by: ipnerds <ipnerds@users.noreply.github.com>
v1v added a commit to v1v/beats that referenced this pull request Aug 12, 2020
…ne-2.0

* upstream/master: (39 commits)
  [ITs] Revert healthcheck for elasticsearchssl service to the previous behaviour (elastic#20558)
  [Heartbeat] Fix packaging (elastic#20566)
  [Heartbeat] Add Magefile to X-Pack (elastic#20549)
  [Packetbeat] Add "network" to event.category (elastic#20392)
  fix typo in docs (elastic#20541)
  Add service resource in k8s cluster role (elastic#20546)
  Update Golang version to 1.14.7 (elastic#20508)
  Add missing inputs to filebeat spec (elastic#20388)
  add warning log in aws and googlecloud module for API cost (elastic#20523)
  Fix fortinet.firewall.mem value to be interpreted as integer (elastic#19335)
  [CI] add more resilience (elastic#20505)
  [JJBB] fix credentials with a service account for golang-crossbuild (elastic#20537)
  [ITs] change healthcheck for elasticsearch (elastic#20514)
  [JJBB] fix credentials with a service account (elastic#20535)
  chore(ci): use build step for checking if is PR (elastic#20536)
  [CI] runbld project name (elastic#20466)
  Add panw.panos.endreason field (elastic#18705)
  [Filebeat] Fix PANW field spelling "veredict" to "verdict" (elastic#18808)
  Fix typo in netflow module docs (elastic#18992)
  Modified auditd ingest pipeline to handle node=hostname (elastic#19659)
  ...
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
This change permits auditd logs to be ingested whether or not they begin with a node=. Previous to this change a type= was expected.

It handles the case where name_format (man 5 auditd.conf) is a value other than none.

Also adding 10 lines of resulting audit log with name_format = hostname parameter on CentOS 7.

Co-authored-by: Marc Guasch <marc.guasch@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants