Cherry-pick #20305 to 7.x: [Autodiscovery] Ignore ErrInputNotFinished errors in autodiscover config checks #20337

ChrsMark · 2020-07-30T07:11:35Z

Cherry-pick of PR #20305 to 7.x branch. Original message:

What does this PR do?

This PR ignores ErrInputNotFinished error occur in autodiscover stop/start process. This is required in order to avoid having stoped configs that never come back since the start event fails due to this error on first attempt if the previous state is not cleaned yet.

Currently configs that fail due to ErrInputNotFinished are skipped at

beats/libbeat/autodiscover/autodiscover.go

Line 212 in fb52d26

continue

. However, this is not a config error but a state error and in that case we need to add them in the list of configs at

beats/libbeat/autodiscover/autodiscover.go

Line 223 in fb52d26

a.configs[eventID][hash] = &reload.ConfigWithMeta{

so as to be handled properly by the retry mechanism of autodiscover at

beats/libbeat/autodiscover/autodiscover.go

Line 164 in fb52d26

retry = err != nil

Why is it important?

In order to resolve a permanent issue with updated Pods, which makes Filebeat stop collecting logs after a Pod is updated.

How to test this PR locally

Deploy Filebeat on k8s using the following config for autodiscover (set a valid output too so as to ship logs to ES):

filebeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      templates:
        - condition:
            equals:
              kubernetes.pod.name: "mytarget3"
          config:
            - type: container
              paths:
                - /var/log/containers/*${data.kubernetes.container.id}.log

While Filebeat is up and running deploy a target pod to be autodiscovered and make Filebeat collects its logs:

---
apiVersion: v1
kind: Pod
metadata:
  name: mytarget3
  labels:
    app: test
spec:
  containers:
    - name: test
      image: ubuntu:latest
      command:
        - bash
        - -c
        - |
          #!/bin/bash
          echo "$(date): started the process"

          while :
          do
                 echo "$(date): sleeping 5 seconds"
                 sleep 5
          done

Update the target Pod's manifest by adding an extra label like team: qa
apply the Pod's update with kubectl apply -f <manifest_filename>.yml
Make sure that after a while, Filebeat continues collecting logs after the update of the Pod.

…fig checks (elastic#20305) (cherry picked from commit 7a34167)

elasticmachine · 2020-07-30T07:11:39Z

Pinging @elastic/integrations-platforms (Team:Platforms)

elasticmachine · 2020-07-30T07:25:26Z

❕ Build Aborted

There is a new build on-going so the previous on-going builds have been aborted.

Expand to view the summary

Build stats

Build Cause: [Pull request #20337 opened]
Reason: Aborted from #2
Start Time: 2020-07-30T07:12:54.929+0000
Duration: 12 min 27 sec
Commit: 23db800

Log output

Expand to view the last 100 lines of log output

[2020-07-30T07:24:51.234Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-07-30T07:24:51.248Z] Stage "Packetbeat OSS" skipped due to earlier failure(s)
[2020-07-30T07:24:51.269Z] Stage "dockerlogbeat" skipped due to earlier failure(s)
[2020-07-30T07:24:51.292Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-07-30T07:24:51.317Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-07-30T07:24:51.336Z] Stage "Journalbeat" skipped due to earlier failure(s)
[2020-07-30T07:24:51.352Z] Stage "Generators" skipped due to earlier failure(s)
[2020-07-30T07:24:52.903Z] Click here to forcibly terminate running steps
[2020-07-30T07:24:53.463Z] Failed in branch Elastic Agent x-pack
[2020-07-30T07:24:53.473Z] Failed in branch Elastic Agent x-pack Windows
[2020-07-30T07:24:53.483Z] Failed in branch Elastic Agent Mac OS X
[2020-07-30T07:24:53.495Z] Failed in branch Filebeat oss
[2020-07-30T07:24:53.508Z] Failed in branch Filebeat x-pack
[2020-07-30T07:24:53.521Z] Failed in branch Filebeat Mac OS X
[2020-07-30T07:24:53.530Z] Failed in branch Filebeat x-pack Mac OS X
[2020-07-30T07:24:53.539Z] Failed in branch Filebeat Windows
[2020-07-30T07:24:53.550Z] Failed in branch Filebeat x-pack Windows
[2020-07-30T07:24:53.561Z] Failed in branch Auditbeat oss Linux
[2020-07-30T07:24:53.570Z] Failed in branch Auditbeat crosscompile
[2020-07-30T07:24:53.580Z] Failed in branch Auditbeat oss Mac OS X
[2020-07-30T07:24:53.589Z] Failed in branch Auditbeat oss Windows
[2020-07-30T07:24:53.599Z] Failed in branch Auditbeat x-pack
[2020-07-30T07:24:53.610Z] Failed in branch Auditbeat x-pack Mac OS X
[2020-07-30T07:24:53.619Z] Failed in branch Auditbeat x-pack Windows
[2020-07-30T07:24:53.628Z] Failed in branch Libbeat x-pack
[2020-07-30T07:24:53.638Z] Failed in branch Metricbeat OSS Unit tests
[2020-07-30T07:24:53.651Z] Failed in branch Metricbeat OSS Go Integration tests
[2020-07-30T07:24:53.661Z] Failed in branch Metricbeat OSS Python Integration tests
[2020-07-30T07:24:53.670Z] Failed in branch Metricbeat crosscompile
[2020-07-30T07:24:53.704Z] Failed in branch Metricbeat Mac OS X
[2020-07-30T07:24:53.716Z] Failed in branch Metricbeat x-pack Mac OS X
[2020-07-30T07:24:53.732Z] Failed in branch Metricbeat Windows
[2020-07-30T07:24:53.753Z] Failed in branch Metricbeat x-pack Windows
[2020-07-30T07:24:53.763Z] Failed in branch Winlogbeat Windows x-pack
[2020-07-30T07:24:53.772Z] Failed in branch Kubernetes
[2020-07-30T07:24:54.687Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-07-30T07:24:54.717Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-07-30T07:24:54.734Z] Stage "Metricbeat x-pack" skipped due to earlier failure(s)
[2020-07-30T07:24:54.748Z] Stage "Packetbeat OSS" skipped due to earlier failure(s)
[2020-07-30T07:24:54.759Z] Stage "Winlogbeat" skipped due to earlier failure(s)
[2020-07-30T07:24:54.770Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-07-30T07:24:54.786Z] Stage "Generators" skipped due to earlier failure(s)
[2020-07-30T07:24:55.583Z] Failed in branch dockerlogbeat
[2020-07-30T07:24:55.593Z] Failed in branch Journalbeat
[2020-07-30T07:24:56.778Z] Stage "Heartbeat" skipped due to earlier failure(s)
[2020-07-30T07:24:56.799Z] Stage "Libbeat" skipped due to earlier failure(s)
[2020-07-30T07:24:56.814Z] Stage "Packetbeat OSS" skipped due to earlier failure(s)
[2020-07-30T07:24:56.831Z] Stage "Functionbeat" skipped due to earlier failure(s)
[2020-07-30T07:24:56.844Z] Stage "Generators" skipped due to earlier failure(s)
[2020-07-30T07:24:57.000Z] Failed in branch Metricbeat x-pack
[2020-07-30T07:24:57.014Z] Failed in branch Winlogbeat
[2020-07-30T07:24:57.806Z] Failed in branch Heartbeat
[2020-07-30T07:24:57.816Z] Failed in branch Libbeat
[2020-07-30T07:24:57.825Z] Failed in branch Packetbeat OSS
[2020-07-30T07:24:57.835Z] Failed in branch Functionbeat
[2020-07-30T07:24:57.836Z] Stage "Generators" skipped due to earlier failure(s)
[2020-07-30T07:24:58.152Z] Failed in branch Generators
[2020-07-30T07:25:00.763Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20337/src/github.com/elastic/beats
[2020-07-30T07:25:01.117Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-30T07:25:01.144Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-20337/src/github.com/elastic/beats/Lint
[2020-07-30T07:25:01.629Z] + cat
[2020-07-30T07:25:01.629Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-30T07:25:01.629Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-30T07:25:08.238Z] runbld>>> runbld started
[2020-07-30T07:25:08.238Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-30T07:25:09.625Z] runbld>>> The following profiles matched the job 'Beats/beats/PR-20337' in order of occurrence in the config (last value wins).
[2020-07-30T07:25:11.075Z] runbld>>> Debug logging enabled.
[2020-07-30T07:25:11.075Z] runbld>>> Storing result
[2020-07-30T07:25:11.075Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-30T07:25:11.075Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200730072510-F624D56B
[2020-07-30T07:25:11.075Z] runbld>>> Adding system facts.
[2020-07-30T07:25:12.463Z] runbld>>> Adding vcs info for the latest commit:  23db800ee1ea26031c724e1395ede42109d2a051
[2020-07-30T07:25:12.463Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-30T07:25:12.463Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-30T07:25:12.463Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-30T07:25:12.463Z] Processing JUnit reports with runbld...
[2020-07-30T07:25:12.733Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-30T07:25:12.733Z] runbld>>> DURATION: 19ms
[2020-07-30T07:25:12.733Z] runbld>>> STDOUT: 40 bytes
[2020-07-30T07:25:12.733Z] runbld>>> STDERR: 49 bytes
[2020-07-30T07:25:12.733Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-30T07:25:12.733Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-20337/src/github.com/elastic/beats
[2020-07-30T07:25:13.324Z] runbld>>> Storing build metadata: 
[2020-07-30T07:25:13.325Z] runbld>>> Adding test report.
[2020-07-30T07:25:13.325Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-20337/src/github.com/elastic/beats
[2020-07-30T07:25:14.275Z] runbld>>> Found 0 test output files
[2020-07-30T07:25:14.275Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 0 Skipped: 0
[2020-07-30T07:25:14.275Z] runbld>>> Storing result
[2020-07-30T07:25:14.539Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-30T07:25:14.539Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200730072510-F624D56B
[2020-07-30T07:25:14.799Z] runbld>>> Email notification disabled by environment variable.
[2020-07-30T07:25:14.799Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-30T07:25:20.614Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-20337
[2020-07-30T07:25:20.855Z] [INFO] getVaultSecret: Getting secrets
[2020-07-30T07:25:21.011Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-30T07:25:22.326Z] + chmod 755 generate-build-data.sh
[2020-07-30T07:25:22.326Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20337/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20337/runs/1 ABORTED 747121
[2020-07-30T07:25:22.326Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20337/runs/1/steps/?limit=10000 -o steps-info.json
[2020-07-30T07:25:22.877Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20337/runs/1/tests/?status=FAILED -o tests-errors.json
[2020-07-30T07:25:22.877Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-20337/runs/1/log/ -o pipeline-log.txt

[Autodiscovery] Ignore ErrInputNotFinished errors in autodiscover con…

23db800

…fig checks (elastic#20305) (cherry picked from commit 7a34167)

ChrsMark added [zube]: In Review backport Team:Platforms Label for the Integrations - Platforms team labels Jul 30, 2020

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 30, 2020

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 30, 2020

Update CHANGELOG.next.asciidoc

ad20af3

zube bot added [zube]: Inbox and removed [zube]: In Review labels Jul 30, 2020

ChrsMark requested a review from a team July 30, 2020 07:22

ChrsMark requested a review from jsoriano July 30, 2020 07:34

jsoriano approved these changes Jul 30, 2020

View reviewed changes

ChrsMark merged commit 0c6b26f into elastic:7.x Jul 30, 2020

zube bot added [zube]: Done and removed [zube]: Inbox labels Jul 30, 2020

zube bot removed the [zube]: Done label Oct 28, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cherry-pick #20305 to 7.x: [Autodiscovery] Ignore ErrInputNotFinished errors in autodiscover config checks #20337

Cherry-pick #20305 to 7.x: [Autodiscovery] Ignore ErrInputNotFinished errors in autodiscover config checks #20337

ChrsMark commented Jul 30, 2020 •

edited by zube bot

Loading

elasticmachine commented Jul 30, 2020

elasticmachine commented Jul 30, 2020

Build stats

Cherry-pick #20305 to 7.x: [Autodiscovery] Ignore ErrInputNotFinished errors in autodiscover config checks #20337

Cherry-pick #20305 to 7.x: [Autodiscovery] Ignore ErrInputNotFinished errors in autodiscover config checks #20337

Conversation

ChrsMark commented Jul 30, 2020 • edited by zube bot Loading

What does this PR do?

Why is it important?

How to test this PR locally

elasticmachine commented Jul 30, 2020

elasticmachine commented Jul 30, 2020

❕ Build Aborted

Build stats

Log output

ChrsMark commented Jul 30, 2020 •

edited by zube bot

Loading