First pass at auditbeat support

[fearful-symmetry]

What does this PR do?

This PR adds auditbeat to the V2 setup/init system, as part of this: #32786

I've never touched auditbeat before, so I had bit of trouble setting this up and testing it, but I can say that in it's current state, it doesn't not work. I'm hoping that someone with more auditbeat knowledge can better prod at this and see where the missing pieces are, if any.

What this does

For anyone coming at this with auditbeat knowledge that's new to V2, the core component here is the auditbeatCfg function performs any necessary transformation of the config that comes from fleet.

Here's an example auditbeat config I managed to generate. This is a yaml representation of what's received by the rawIn argument to to auditbeatCfg :

  - id: audit/auditd-auditd-5aac139d-166a-4d8f-af34-a751364f7635
    name: auditd_manager-1
    revision: 1
    type: audit/auditd
    use_output: default
    meta:
      package:
        name: auditd_manager
        version: 1.3.0
    data_stream:
      namespace: default
    streams:
      - id: >-
          audit/auditd-auditd_manager.auditd-5aac139d-166a-4d8f-af34-a751364f7635
        type: audit/auditd
        data_stream:
          dataset: auditd_manager.auditd
          type: logs
        condition: '${host.platform} == ''linux'''
        include_raw_message: true
        socket_type: unicast
        immutable: false
        resolve_ids: true
        failure_mode: silent
        audit_rules: '-a always,exit -F arch=b32 -S all -F key=32bit-abi'
        backlog_limit: 8192
        rate_limit: 0
        include_warnings: false
        backpressure_strategy: auto
        tags:
          - auditd_manager-auditd

The auditbeatCfg should take this config and process it in a way to make it readable to auditbeat. Specifically, each item in the streams YAML array (which becomes that modules array in auditbeatCfg) will be sent to an auditbeat beater instance. Right now, the only change is to add the module: key based on the type value in the YAML.

To test this, just build this PR into auditbeat/agent, run against an 8.5 cluster, and configure auditbeat via fleet. Make sure whatever integrations and config you're adding are actually being applied.

How to test this

This code must be tested against the V2-enabled code in the feature-arch-v2 code. To test this PR:

• Checkout the feature-arch-v2 branch of elastic-agent
• Checkout this PR in the beats repo
• in the main elastic-agent magefile.go, add an "auditbeat" entry to the packedBeats var, like this:

packedBeats := []string{"filebeat", "heartbeat", "metricbeat", "osquerybeat", "auditbeat"}

(note: there are other methods of building elastic-agent with a test/development beat, I just find this to be the most convenient. Feel free to use your own method if you prefer)

• build elastic-agent with PACKAGES=TarGz mage dev:package (note:you may want to further constrain the build targets with PLATFORMS=linux/amd64 or something similar)
• Configure elastic-agent. In the hard-coded elastic-agent.yml file, replace any existing data under the inputs: field with the example input config included above in the What this does section. Alternative, write your own input config, or modify the example as needed.
• Either re-configure the included elasticsearch output in elastic-agent.yml, or use a file output:

outputs:
  default:
     type: file
     enabled: true
     path: "/tmp/testbeat_out"
     filename: "testbeat_log"

• alternatively, instead of configuring elastic-agent.yml, connect elasticsearch to an existing 8.5.0-SNAPSHOT cluster, enroll the test elastic-agent binary and configure via fleet as usual.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-09-15T23:45:37.598+0000

• Duration: 40 min 28 sec

Test stats 🧪

Test	Results
Failed	0
Passed	276
Skipped	49
Total	325

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[efd6]

Nits only.

[fearful-symmetry]

@efd6 did you get a chance to test this? I've never actually touched auditbeat before so it was a bit of a struggle, so I was hoping someone with more of a background might have better luck with it.

[efd6]

I have now had a chance to test this and it looks like it is behaving correctly.