Debounce input reload on autodiscover

[belimawr]

What does this PR do?

The Kubernetes autodiscover feature now incorporates a debounce logic when reloading inputs. By default, it waits for at least 1 second before invoking the Reload method. In case of an error, it introduces a 10-second delay before retrying.

The channel used for test synchronisation has been removed and tests now use (assert/require).Eventually.

When Autodiscover calls cfgfile.NewRunnerList to instantiate a RunnerList, it now specifies a different logger name, enabling more granular log filtering.

Debug logs now provide information about the reasons for invoking Reload.

Certain tests that perform sequential actions now utilise require instead of assert to maintain a consistent state avoid cascading failures.

Tests that required updates now leverage require.Eventually instead of wait, providing additional information on failure causes.

Documentation for cfgfile.RunnerList has been improved to enhance clarity.

Why is it important?

Fixes #34388

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

## Author's Checklist

How to test this PR locally

Start a Kubernetes cluster (I used Minikube with none driver on a VM)
Deploy some pods to generate logs constantly

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "2"
  labels:
    app: flog-01
  name: flog-01
spec:
  progressDeadlineSeconds: 600
  replicas: 4
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: flog-01
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: flog-01
    spec:
      containers:
      - args:
        - -l
        - -s
        - "1"
        - -d
        - "1"
        image: mingrammer/flog
        imagePullPolicy: Always
        name: flog
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30

Start Filebeat with the following configuration

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
      hints.default_config:
        type: container
        id: "runner-${data.kubernetes.container.id}"
        paths:
          - /var/log/containers/*-${data.kubernetes.container.id}.log
        fields:
          role: kubernetes

logging:
  level: debug
  selectors:
    - autodiscover
    - autodiscover.cfgfile
output.elasticsearch:
  hosts:
    - https://foo.bar.cloud.elastic.co:443
  protocol: https
  username: foo
  password: bar

There should be no data loss or constant log messages like:

Error creating runner from config: Can only start an input when all related states are finished

Related issues

• Fixes Filebeat: Error when using autodiscover + hints + templates with filebeat 8.6 #34388
• Relates Make Autodiscover more selective when watching K8s events #34717

## Use cases
## Screenshots
## Logs

[mergify]

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @belimawr? 🙏.
For such, you'll need to label your PR with:

• The upcoming major version of the Elastic Stack
• The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-06-20T14:45:54.810+0000

• Duration: 84 min 19 sec

Test stats 🧪

Test	Results
Failed	0
Passed	27449
Skipped	2013
Total	29462

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[ycombinator]

Change LGTM. Just left a couple of minor suggestions.

[belimawr]

@ycombinator all changes you requested are on b82eda4.

[ycombinator]

LGTM.

[belimawr]

/test

[belimawr]

Rebasing onto main is the easiest way to get the tests to run again 🤷‍♂️

[belimawr]

This PR is finally ready for a final review. The tests were failing because of a bug I introduced, it has been fixed on 1f38a4c and
65208cb adds tests for that case.

@ycombinator could you review it again?

I changed the implementation

[belimawr]

Tests were failing due to CI issues (pip was failing). I re-based onto main and force-pushed.

[ycombinator]

LGTM.

[belimawr]

This also seems to relate to #34717, I'll edit the description of the PR adding this information.