Put data stream so there is no need for additional permissions

[kvch]

What does this PR do?

This PR adds a new step to loading templates. Now not only the template is loaded, but the data stream is created as well. Given that users might load templates from JSON file that are not data streams, I added a new option called setup.template.json.data_stream. It has to be set, if the JSON template is a data stream.

Why is it important?

Without this change users needed more permissions to publish events. Now create_doc priviledge is enough to publish events to the data stream.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Closes #30647
Closes #30567

[mergify]

This pull request does not have a backport label. Could you fix it @kvch? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-04-05T14:31:17.554+0000

• Duration: 94 min 24 sec

Test stats 🧪

Test	Results
Failed	0
Passed	22289
Skipped	1937
Total	24226

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[cmacknz]

Without this change users needed more permissions to publish events. Now create_doc priviledge is enough to publish events to the data stream.

With this change we would still attempt to create the datastream with the user used for publishing events, does that safely do nothing if the datastream already exists and the user used for publishing lacks the permissions that would be required?

How did we test this? Is there an automated test we can write for this use case where the setup user has more permissions that the user used for publishing to ES?

[kvch]

Without this change users needed more permissions to publish events. Now create_doc priviledge is enough to publish events to the data stream.

With this change we would still attempt to create the datastream with the user used for publishing events, does that safely do nothing if the datastream already exists and the user used for publishing lacks the permissions that would be required?

No, we only attempt to create the data stream, if the template is loaded. Given that during setup both the index template and the data stream is loaded, Beats will not try to load the data stream during publishing.

How did we test this? Is there an automated test we can write for this use case where the setup user has more permissions that the user used for publishing to ES?

In libbeat we test everything without enabling security in Elasticsearch. In the past we relied on manual testing for these use cases. I am fine with adding more tests, but given that it requires a bit more work to create the environment for such tests, I would prefer to do it in a follow up PR.

[cmacknz]

In libbeat we test everything without enabling security in Elasticsearch. In the past we relied on manual testing for these use cases. I am fine with adding more tests, but given that it requires a bit more work to create the environment for such tests, I would prefer to do it in a follow up PR.

Fine with me, we should add some tests for this but it can be done as a follow up as long we do a manual check that this works as well.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b fix-libbeat-put-data-stream-on-setup upstream/fix-libbeat-put-data-stream-on-setup
git merge upstream/main
git push upstream fix-libbeat-put-data-stream-on-setup

[cmacknz]

Let's make sure we have a test case where we ensure we don't delete a data stream that already exists.

[cmacknz]

Added labels to backport into 8.2 since this is a bug fix (and was created just before the 8.2 feature freeze).

Skipping 8.1 as there currently aren't any more planned 8.1 releases

[cmacknz]

Let's backport to 8.1 in case we end up doing another 8.1.x release.