Cherry-pick #24810 to 7.x: feat: add support for SCRAM authentication for kafka metricbeat module, fixes #19648

[jsoriano]

Cherry-pick of PR #24810 to 7.x branch. Original message:

What does this PR do?

Adds SCRAM authentication for kafka to metricbeat's kafka module. Moves shared code from kafka output plugin to libbeat/common/kafka/

Why is it important?

Currently metricbeat can't pull metrics from Kafka clusters that require authentication, which is a problem in enterprise environments where authentication is a requirement.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

I think most of this isn't applicable, as this PR virtually just moves files around and makes couple things public in the shared code. This virtually no new code introduced.

Reg. "I have made corresponding changes to the documentation", it seems that none of the kafka module's fields in metricbeat are separately documented. Instead the config.yml is pulled into the asciidoc file, which I updated.

How to test this PR locally

As this PR mostly just moves existing files around and introduces only minimal code changes, the test procedures remain the same as those for #12867 which initially introduced SCRAM code. Though it seems those were not documented at the time. Generally, though, the procedure documented in beats/blob/master/metricbeat/module/kafka/README.md remains the same, with the exception that the test kafka cluster needs to use one of the SCAM mechanisms, eg. in server.properties have listener.name.external.sasl.enabled.mechanisms=SCRAM-SHA-256 configured. (if you external listener is called EXTERNAL)

Related issues

• Closes Add SCRAM authentication to kafka metrics source #19648

Use cases

This PR introduces sasl.mechanism in metricbeat's kafka input module. It works the same way as the similarly named configuration directive in libbeat's kafka output plugin.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #24889 opened

• Start Time: 2021-04-01T09:10:45.663+0000

• Duration: 140 min 22 sec

• Commit: 327171d

Test stats 🧪

Test	Results
Failed	0
Passed	45844
Skipped	5051
Total	50895

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	45844
Skipped	5051
Total	50895