Load inputs from external configuration files in Elastic Agent in standalone mode

[kvch]

What does this PR do?

This PR adds support for loading external configuration files to load inputs when running Agent in standalone mode.
Users have to put their configuration files into {path.config}/inputs.d with the extension yml.

When it is configured, Agent takes the inputs configuration from the last standalone configuration and appends the inputs from the external configuration files to it. When the standalone configurations are reloaded, external input configurations are reloaded as well.

The external configuration file must include inputs, otherwise the file is invalid. Example for correct file:

inputs:
- data_stream:
    dataset: system.auth
    type: logs
  id: logfile-system.auth-my-id
  paths:
  - /var/log/auth.log*
  use_output: default

Why is it important?

Users can configure external configuration files, similar to Filebeat. So managing configuration can be done by Git or Chef instead of Fleet.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[botelastic]

This pull request doesn't have a Team:<team> label.

[mergify]

This pull request does not have a backport label. Could you fix it @kvch? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Reason: null

• Start Time: 2022-02-01T12:26:40.666+0000

• Duration: 106 min 1 sec

• Commit: e2fef84

Test stats 🧪

Test	Results
Failed	0
Passed	7248
Skipped	16
Total	7264

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[kvch]

@ph are the errors in the E2E tests related to my change?

[ph]

@kvch I do not think they are

[2022-01-28T13:55:52.855Z] WARN[2022-01-28T13:55:52Z] The Elastic artifacts API is not available yet  artifact=elastic-agent artifactName=elastic-agent-8.1.0-SNAPSHOT-linux-x86_64.tar.gz elapsedTime=52.96205476s error="GET request failed with 404" retry=15 statusEndpoint="https://artifacts-api.elastic.co/v1/search/8.1.0-dbc834fd/elastic-agent?x-elastic-no-kpi=true" version=8.1.0-dbc834fd

I see the unified build failed on master. https://internal-ci.elastic.co/job/elastic+unified-release+master+distributed-snapshot-multijob-master/1059/

[ph]

/test

[ruflin]

Lets make sure we test this also as part of the e2e test framework to have some integration tests for it.

@kvch In the Beats days, I remember there were always discussions around using inputs (in this case) in the inputs.d/*.yml files vs not. No strong preference on either but it would be nice if users use the wrong config, a clear message shows how it should be or link to the docs.

[kvch]

Lets make sure we test this also as part of the e2e test framework to have some integration tests for it.

👍

@kvch In the Beats days, I remember there were always discussions around using inputs (in this case) in the inputs.d/*.yml files vs not. No strong preference on either but it would be nice if users use the wrong config, a clear message shows how it should be or link to the docs.

I have somewhat strong preference because otherwise the file cannot be parsed. I can probably workaround the problem but I rather keep it this way. If the users miss the inputs line, Agent will report the error.

[ruflin]

I have somewhat strong preference because otherwise the file cannot be parsed. I can probably workaround the problem but I rather keep it this way. If the users miss the inputs line, Agent will report the error.

My strong preference is to make sure only 1 way works :-) Lets make sure the error that is shown to the user is very descriptive.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature-agent-load-input-config-from-dir upstream/feature-agent-load-input-config-from-dir
git merge upstream/master
git push upstream feature-agent-load-input-config-from-dir

[cmacknz]

Looks good to me, but I'll wait to see if someone with more knowledge of the agent configuration files can review this before approving.

[kvch]

e2e tests require elastic/e2e-testing#2064

[blakerouse]

Overall the code looks good.

I do want to confirm the behavior that I am reading in the code. It seems this is only used when in standalone mode (which I am glad to see). But in the description of the PR it describes that it allows usage of infrastructure as code instead of Fleet. How does this help solve that, when a single elastic-agent.yml also did the same thing. The single elastic-agent.yml could be committed in Git as well. Just trying to understand the reason for adding this.

[kvch]

But in the description of the PR it describes that it allows usage of infrastructure as code instead of Fleet. How does this help solve that, when a single elastic-agent.yml also did the same thing. The single elastic-agent.yml could be committed in Git as well. Just trying to understand the reason for adding this.

When a separate, dedicated team manages Elastic Agent, they might want to limit the access to a single elastic-agent.yml file. If we provide a folder from where Elastic Agent can load input configurations, the Agent manager team can allow other teams in their org to put their own input configuration there without giving access to the whole infrastructure.

[kvch]

Ooops, I merged it without a changelog entry. I will open a follow-up PR.

[kvch]

Changelog: #30144

[ph]

LGTM, after merge.

[kvch]

Documentation: elastic/observability-docs#1515