Cherry-pick #24681 to 7.x: Disable cleanup_timeout by default in docker and kubernetes autodiscover

[jsoriano]

Cherry-pick of PR #24681 to 7.x branch. Original message:

What does this PR do?

Disable cleanup_timeout by default in docker and kubernetes autodiscover for all beats except Filebeat.

It is kept to 60 seconds in Filebeat, to give a time to collect logs.

Why is it important?

Keeping configurations running for some time after containers have stopped is needed in some cases to complete the collection of logs. But in the rest of cases it is not usually needed, and leads to errors when querying endpoints known to be down.
It can also lead to query IPs that are being reused in newer containers, what can be misleading if the newer pod answers because these events will still have the metadata of the old container.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

• Start metricbeat and filebeat in Kubernetes with autodiscover and some configuration.
• Create a pod in Kubernetes that matches the existing configurations.
• Check that metricbeat and filebat start collecting metrics and logs.
• Delete the pod.
• Check that metricbeat stops collecting metrics.
• Check that filebeat stops collecting metrics about 60 seconds later.

Related issues

• Fixes Set a different autodiscover cleanup_timeout per beat #20543.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

❕ Build Aborted

Either there was a build timeout or someone aborted the build.'}

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: Pull request #24730 opened

• Start Time: 2021-03-24T14:27:56.925+0000

• Duration: 183 min 58 sec

• Commit: a6732f8

Test stats 🧪

Test	Results
Failed	0
Passed	45406
Skipped	4961
Total	50367

Trends 🧪

Log output

Expand to view the last 100 lines of log output

[2021-03-24T16:50:21.808Z] >> Building packetbeat.reference.yml for linux/amd64
[2021-03-24T16:50:21.808Z] >> Building packetbeat.docker.yml for linux/amd64
[2021-03-24T16:50:21.808Z] >> check: Checking source code for common problems
[2021-03-24T16:51:30.060Z] + mage update
[2021-03-24T16:51:30.060Z] Generated fields.yml for packetbeat to /var/lib/jenkins/workspace/PR-24730-1-c5f412e2-3fda-4aab-968e-c5a15d4fe48c/src/github.com/elastic/beats/x-pack/packetbeat/fields.yml
[2021-03-24T16:51:30.060Z] >> Building packetbeat.yml for linux/amd64
[2021-03-24T16:51:30.060Z] >> Building packetbeat.reference.yml for linux/amd64
[2021-03-24T16:51:30.060Z] >> Building packetbeat.docker.yml for linux/amd64
[2021-03-24T16:51:30.832Z] Cleaning up /var/lib/jenkins/workspace/PR-24730-1-c5f412e2-3fda-4aab-968e-c5a15d4fe48c
[2021-03-24T16:51:30.832Z] Client: Docker Engine - Community
[2021-03-24T16:51:30.832Z]  Version:           20.10.3
[2021-03-24T16:51:30.832Z]  API version:       1.41
[2021-03-24T16:51:30.832Z]  Go version:        go1.13.15
[2021-03-24T16:51:30.832Z]  Git commit:        48d30b5
[2021-03-24T16:51:30.832Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-03-24T16:51:30.832Z]  OS/Arch:           linux/amd64
[2021-03-24T16:51:30.832Z]  Context:           default
[2021-03-24T16:51:30.832Z]  Experimental:      true
[2021-03-24T16:51:30.832Z] 
[2021-03-24T16:51:30.832Z] Server: Docker Engine - Community
[2021-03-24T16:51:30.832Z]  Engine:
[2021-03-24T16:51:30.832Z]   Version:          20.10.3
[2021-03-24T16:51:30.832Z]   API version:      1.41 (minimum version 1.12)
[2021-03-24T16:51:30.832Z]   Go version:       go1.13.15
[2021-03-24T16:51:30.832Z]   Git commit:       46229ca
[2021-03-24T16:51:30.832Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-03-24T16:51:30.832Z]   OS/Arch:          linux/amd64
[2021-03-24T16:51:30.832Z]   Experimental:     false
[2021-03-24T16:51:30.832Z]  containerd:
[2021-03-24T16:51:30.832Z]   Version:          1.4.4
[2021-03-24T16:51:30.832Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-03-24T16:51:30.832Z]  runc:
[2021-03-24T16:51:30.832Z]   Version:          1.0.0-rc93
[2021-03-24T16:51:30.832Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-03-24T16:51:30.832Z]  docker-init:
[2021-03-24T16:51:30.832Z]   Version:          0.19.0
[2021-03-24T16:51:30.832Z]   GitCommit:        de40ad0
[2021-03-24T16:51:30.832Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-03-24T16:51:30.832Z] Unable to find image 'alpine:3.4' locally
[2021-03-24T16:51:31.404Z] 3.4: Pulling from library/alpine
[2021-03-24T16:51:31.664Z] c1e54eec4b57: Pulling fs layer
[2021-03-24T16:51:31.925Z] c1e54eec4b57: Verifying Checksum
[2021-03-24T16:51:31.925Z] c1e54eec4b57: Download complete
[2021-03-24T16:51:32.188Z] c1e54eec4b57: Pull complete
[2021-03-24T16:51:32.188Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-03-24T16:51:32.188Z] Status: Downloaded newer image for alpine:3.4
[2021-03-24T16:51:34.103Z] Change permissions with write access of all files inside the specific folder
[2021-03-24T16:51:35.231Z] Running in /var/lib/jenkins/workspace/PR-24730-1-c5f412e2-3fda-4aab-968e-c5a15d4fe48c/src/github.com/elastic/beats/build
[2021-03-24T16:51:40.057Z] + rm -rf ve
[2021-03-24T16:51:40.061Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-03-24T16:51:40.790Z] + python .ci/scripts/pre_archive_test.py
[2021-03-24T16:51:40.790Z] Copy ./x-pack/packetbeat/build into build/x-pack/packetbeat/build
[2021-03-24T16:51:40.881Z] Running in /var/lib/jenkins/workspace/PR-24730-1-c5f412e2-3fda-4aab-968e-c5a15d4fe48c/src/github.com/elastic/beats/build
[2021-03-24T16:51:40.976Z] Recording test results
[2021-03-24T16:51:44.359Z] None of the test reports contained any result
[2021-03-24T16:51:44.377Z] [Checks API] No suitable checks publisher found.
[2021-03-24T16:51:45.088Z] + go clean -modcache
[2021-03-24T16:51:53.878Z] Cleaning up /var/lib/jenkins/workspace/PR-24730-1-c5f412e2-3fda-4aab-968e-c5a15d4fe48c
[2021-03-24T16:51:53.878Z] Client: Docker Engine - Community
[2021-03-24T16:51:53.878Z]  Version:           20.10.3
[2021-03-24T16:51:53.878Z]  API version:       1.41
[2021-03-24T16:51:53.878Z]  Go version:        go1.13.15
[2021-03-24T16:51:53.878Z]  Git commit:        48d30b5
[2021-03-24T16:51:53.878Z]  Built:             Fri Jan 29 14:33:13 2021
[2021-03-24T16:51:53.878Z]  OS/Arch:           linux/amd64
[2021-03-24T16:51:53.878Z]  Context:           default
[2021-03-24T16:51:53.878Z]  Experimental:      true
[2021-03-24T16:51:53.878Z] 
[2021-03-24T16:51:53.878Z] Server: Docker Engine - Community
[2021-03-24T16:51:53.878Z]  Engine:
[2021-03-24T16:51:53.878Z]   Version:          20.10.3
[2021-03-24T16:51:53.878Z]   API version:      1.41 (minimum version 1.12)
[2021-03-24T16:51:53.878Z]   Go version:       go1.13.15
[2021-03-24T16:51:53.878Z]   Git commit:       46229ca
[2021-03-24T16:51:53.878Z]   Built:            Fri Jan 29 14:31:25 2021
[2021-03-24T16:51:53.878Z]   OS/Arch:          linux/amd64
[2021-03-24T16:51:53.878Z]   Experimental:     false
[2021-03-24T16:51:53.878Z]  containerd:
[2021-03-24T16:51:53.878Z]   Version:          1.4.4
[2021-03-24T16:51:53.878Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-03-24T16:51:53.878Z]  runc:
[2021-03-24T16:51:53.878Z]   Version:          1.0.0-rc93
[2021-03-24T16:51:53.878Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-03-24T16:51:53.878Z]  docker-init:
[2021-03-24T16:51:53.878Z]   Version:          0.19.0
[2021-03-24T16:51:53.878Z]   GitCommit:        de40ad0
[2021-03-24T16:51:53.879Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-03-24T16:51:54.841Z] Change permissions with write access of all files inside the specific folder
[2021-03-24T16:51:55.477Z] Running in /var/lib/jenkins/workspace/PR-24730-1-c5f412e2-3fda-4aab-968e-c5a15d4fe48c
[2021-03-24T17:30:45.100Z] Cancelling nested steps due to timeout
[2021-03-24T17:30:45.246Z] Failed in branch filebeat-windows-windows-2019
[2021-03-24T17:30:50.839Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-03-24T17:30:51.008Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-24730/src/github.com/elastic/beats
[2021-03-24T17:30:54.319Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-24730
[2021-03-24T17:30:54.430Z] [INFO] getVaultSecret: Getting secrets
[2021-03-24T17:30:54.540Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-03-24T17:30:55.645Z] + chmod 755 generate-build-data.sh
[2021-03-24T17:30:55.645Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24730/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24730/runs/1 ABORTED 10978432
[2021-03-24T17:30:55.896Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24730/runs/1/steps/?limit=10000 -o steps-info.json
[2021-03-24T17:31:02.377Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24730/runs/1/tests/?status=FAILED -o tests-errors.json