Cherry-pick #25782 to 7.x: Refactor kubelet metricsets to share response from endpoint #25982

ChrsMark · 2021-05-28T07:48:29Z

Cherry-pick of PR #25782 to 7.x branch. Original message:

What does this PR do?

Follow up of #25640. This PR changes how kubernetes module handle metricsets which collect metrics from kubelet's API and which share same target endpoint.
Metricsets affected:

system
pod
node
volume
container

Why is it important?

To improve the performance of the module by avoid fetching same content multiple times.

How to test this PR locally

Deploy Metricbeat on k8s with https://github.com/elastic/beats/blob/master/deploy/kubernetes/metricbeat-kubernetes.yaml using the proper docker image (ie a BC, snapshot etc). Modify the configmaps accordingly using the configs below.
state_* config using leaderelection:

metricbeat.autodiscover:
  providers:
    - type: kubernetes
      scope: cluster
      node: ${NODE_NAME}
      unique: true
      templates:
        - config:
            - module: kubernetes
              hosts: ["kube-state-metrics:8080"]
              period: 10s
              add_metadata: true
              metricsets:
                - state_node
                - state_deployment
                - state_daemonset
                - state_replicaset
                - state_pod
            - module: kubernetes
              hosts: ["kube-state-metrics:8080"]
              period: 10s
              add_metadata: true
              metricsets:
                - state_container
                - state_cronjob
                - state_service
                - state_resourcequota
                - state_statefulset

Kubelet's metricsets' config:

- module: kubernetes
  metricsets:
    - container
    - volume
  period: 10s
  host: ${NODE_NAME}
  hosts: ["https://${NODE_NAME}:10250"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  ssl.verification_mode: "none" 
- module: kubernetes
  metricsets:
    - node
    - system
    - pod
  period: 10s
  host: ${NODE_NAME}
  hosts: ["https://${NODE_NAME}:10250"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  ssl.verification_mode: "none"

Verify that all metricsets are being populated (like in the screenshot below) and that k8s metadata are properly attached on the events.

Related issues

Closes #24869

Screenshots

…25782) (cherry picked from commit a39dd00)

elasticmachine · 2021-05-28T07:48:32Z

Pinging @elastic/integrations (Team:Integrations)

botelastic · 2021-05-28T07:48:36Z

This pull request doesn't have a Team:<team> label.

elasticmachine · 2021-05-28T09:17:51Z

💔 Build Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #25982 opened
Start Time: 2021-05-28T07:48:48.877+0000
Duration: 89 min 47 sec
Commit: fc42acd

Test stats 🧪

Test	Results
Failed	0
Passed	8305
Skipped	2264
Total	10569

Trends 🧪

Steps errors

Expand to view the steps failures

`metricbeat-packaging-linux - mage package`

Took 6 min 25 sec . View more details on here
Description: mage package

`x-pack/metricbeat-packaging-linux - mage package`

Took 9 min 35 sec . View more details on here
Description: mage package

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-05-28T09:16:56.407Z] + docker image inspect docker.elastic.co/beats/metricbeat-ubi8:7.14.0-SNAPSHOT
[2021-05-28T09:16:56.407Z] + echo 'docker image docker.elastic.co/beats/metricbeat-ubi8:7.14.0-SNAPSHOT does not exist'
[2021-05-28T09:16:56.407Z] docker image docker.elastic.co/beats/metricbeat-ubi8:7.14.0-SNAPSHOT does not exist
[2021-05-28T09:16:58.548Z] Cleaning up /var/lib/jenkins/workspace/PR-25982-1-484f6530-c331-4e82-8c20-04dc709b94b7
[2021-05-28T09:16:58.549Z] Client: Docker Engine - Community
[2021-05-28T09:16:58.549Z]  Version:           20.10.3
[2021-05-28T09:16:58.549Z]  API version:       1.41
[2021-05-28T09:16:58.549Z]  Go version:        go1.13.15
[2021-05-28T09:16:58.549Z]  Git commit:        48d30b5
[2021-05-28T09:16:58.549Z]  Built:             Fri Jan 29 14:33:34 2021
[2021-05-28T09:16:58.549Z]  OS/Arch:           linux/arm64
[2021-05-28T09:16:58.549Z]  Context:           default
[2021-05-28T09:16:58.549Z]  Experimental:      true
[2021-05-28T09:16:58.549Z] 
[2021-05-28T09:16:58.549Z] Server: Docker Engine - Community
[2021-05-28T09:16:58.549Z]  Engine:
[2021-05-28T09:16:58.549Z]   Version:          20.10.3
[2021-05-28T09:16:58.549Z]   API version:      1.41 (minimum version 1.12)
[2021-05-28T09:16:58.549Z]   Go version:       go1.13.15
[2021-05-28T09:16:58.549Z]   Git commit:       46229ca
[2021-05-28T09:16:58.549Z]   Built:            Fri Jan 29 14:31:49 2021
[2021-05-28T09:16:58.549Z]   OS/Arch:          linux/arm64
[2021-05-28T09:16:58.549Z]   Experimental:     false
[2021-05-28T09:16:58.549Z]  containerd:
[2021-05-28T09:16:58.549Z]   Version:          1.4.4
[2021-05-28T09:16:58.549Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-05-28T09:16:58.549Z]  runc:
[2021-05-28T09:16:58.549Z]   Version:          1.0.0-rc93
[2021-05-28T09:16:58.549Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-05-28T09:16:58.549Z]  docker-init:
[2021-05-28T09:16:58.549Z]   Version:          0.19.0
[2021-05-28T09:16:58.549Z]   GitCommit:        de40ad0
[2021-05-28T09:16:58.549Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-05-28T09:16:58.549Z] Unable to find image 'arm64v8/alpine:3' locally
[2021-05-28T09:16:59.947Z] 3: Pulling from arm64v8/alpine
[2021-05-28T09:16:59.947Z] 595b0fe564bb: Pulling fs layer
[2021-05-28T09:16:59.947Z] 595b0fe564bb: Verifying Checksum
[2021-05-28T09:16:59.947Z] 595b0fe564bb: Download complete
[2021-05-28T09:17:00.694Z] 595b0fe564bb: Pull complete
[2021-05-28T09:17:00.694Z] Digest: sha256:8f18fae117ec6e5777cc62ba78cbb3be10a8a38639ccfb949521abd95c8301a4
[2021-05-28T09:17:00.694Z] Status: Downloaded newer image for arm64v8/alpine:3
[2021-05-28T09:17:02.238Z] Change permissions with write access of all files inside the specific folder
[2021-05-28T09:17:03.447Z] Running in /var/lib/jenkins/workspace/PR-25982-1-484f6530-c331-4e82-8c20-04dc709b94b7/src/github.com/elastic/beats/build
[2021-05-28T09:17:04.796Z] + rm -rf ve
[2021-05-28T09:17:04.796Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-05-28T09:17:06.552Z] + python .ci/scripts/pre_archive_test.py
[2021-05-28T09:17:07.296Z] Copy ./metricbeat/build into build/metricbeat/build
[2021-05-28T09:17:07.296Z] Copy ./metricbeat/build/golang-crossbuild into build/metricbeat/build/golang-crossbuild
[2021-05-28T09:17:07.296Z] Copy ./metricbeat/build/package/metricbeat-oss/metricbeat-oss-linux-arm64.docker/docker-build into build/metricbeat/build/package/metricbeat-oss/metricbeat-oss-linux-arm64.docker/docker-build
[2021-05-28T09:17:07.754Z] Running in /var/lib/jenkins/workspace/PR-25982-1-484f6530-c331-4e82-8c20-04dc709b94b7/src/github.com/elastic/beats/build
[2021-05-28T09:17:07.913Z] Recording test results
[2021-05-28T09:17:15.439Z] None of the test reports contained any result
[2021-05-28T09:17:15.450Z] [Checks API] No suitable checks publisher found.
[2021-05-28T09:17:16.947Z] + go clean -modcache
[2021-05-28T09:17:21.390Z] Cleaning up /var/lib/jenkins/workspace/PR-25982-1-484f6530-c331-4e82-8c20-04dc709b94b7
[2021-05-28T09:17:21.390Z] Client: Docker Engine - Community
[2021-05-28T09:17:21.390Z]  Version:           20.10.3
[2021-05-28T09:17:21.390Z]  API version:       1.41
[2021-05-28T09:17:21.390Z]  Go version:        go1.13.15
[2021-05-28T09:17:21.390Z]  Git commit:        48d30b5
[2021-05-28T09:17:21.390Z]  Built:             Fri Jan 29 14:33:34 2021
[2021-05-28T09:17:21.390Z]  OS/Arch:           linux/arm64
[2021-05-28T09:17:21.390Z]  Context:           default
[2021-05-28T09:17:21.390Z]  Experimental:      true
[2021-05-28T09:17:21.390Z] 
[2021-05-28T09:17:21.390Z] Server: Docker Engine - Community
[2021-05-28T09:17:21.390Z]  Engine:
[2021-05-28T09:17:21.390Z]   Version:          20.10.3
[2021-05-28T09:17:21.390Z]   API version:      1.41 (minimum version 1.12)
[2021-05-28T09:17:21.391Z]   Go version:       go1.13.15
[2021-05-28T09:17:21.391Z]   Git commit:       46229ca
[2021-05-28T09:17:21.391Z]   Built:            Fri Jan 29 14:31:49 2021
[2021-05-28T09:17:21.391Z]   OS/Arch:          linux/arm64
[2021-05-28T09:17:21.391Z]   Experimental:     false
[2021-05-28T09:17:21.391Z]  containerd:
[2021-05-28T09:17:21.391Z]   Version:          1.4.4
[2021-05-28T09:17:21.391Z]   GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
[2021-05-28T09:17:21.391Z]  runc:
[2021-05-28T09:17:21.391Z]   Version:          1.0.0-rc93
[2021-05-28T09:17:21.391Z]   GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
[2021-05-28T09:17:21.391Z]  docker-init:
[2021-05-28T09:17:21.391Z]   Version:          0.19.0
[2021-05-28T09:17:21.391Z]   GitCommit:        de40ad0
[2021-05-28T09:17:21.391Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-05-28T09:17:22.144Z] Change permissions with write access of all files inside the specific folder
[2021-05-28T09:17:23.359Z] Running in /var/lib/jenkins/workspace/PR-25982-1-484f6530-c331-4e82-8c20-04dc709b94b7
[2021-05-28T09:17:29.682Z] + gsutil --version
[2021-05-28T09:17:31.114Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-05-28T09:17:31.469Z] + gcloud auth activate-service-account --key-file ****
[2021-05-28T09:17:32.046Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-05-28T09:17:32.682Z] + gsutil -m -q cp -a public-read bWV0cmljYmVhdC1wYWNrYWdpbmctYXJtLWFybWZjNDJhY2Q3NGY4M2VjYzk0ZGUyMzhjZTFhNmE5NjMwMTc0ZDkxMmE gs://beats-ci-temp/ci/cache/
[2021-05-28T09:17:34.956Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-05-28T09:17:35.083Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-25982/src/github.com/elastic/beats
[2021-05-28T09:17:35.471Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-25982
[2021-05-28T09:17:35.561Z] [INFO] getVaultSecret: Getting secrets
[2021-05-28T09:17:35.611Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-05-28T09:17:36.597Z] + chmod 755 generate-build-data.sh
[2021-05-28T09:17:36.597Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25982/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25982/runs/1 FAILURE 5327445
[2021-05-28T09:17:36.597Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25982/runs/1/steps/?limit=10000 -o steps-info.json
[2021-05-28T09:17:39.079Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-25982/runs/1/tests/?status=FAILED -o tests-errors.json

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	8305
Skipped	2264
Total	10569

Refactor kubelet metricsets to share response from endpoint (elastic#…

fc42acd

…25782) (cherry picked from commit a39dd00)

ChrsMark added [zube]: In Review backport Team:Integrations Label for the Integrations team labels May 28, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 28, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 28, 2021

ChrsMark merged commit 0ec351b into elastic:7.x May 28, 2021

zube bot added [zube]: Done and removed [zube]: In Review labels May 28, 2021

zube bot removed the [zube]: Done label Aug 26, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cherry-pick #25782 to 7.x: Refactor kubelet metricsets to share response from endpoint #25982

Cherry-pick #25782 to 7.x: Refactor kubelet metricsets to share response from endpoint #25982

ChrsMark commented May 28, 2021 •

edited by zube bot

Loading

elasticmachine commented May 28, 2021

botelastic bot commented May 28, 2021

elasticmachine commented May 28, 2021

Build stats

Test stats 🧪

Trends 🧪

`metricbeat-packaging-linux - mage package`

`x-pack/metricbeat-packaging-linux - mage package`

`Error signal`

Test stats 🧪

Cherry-pick #25782 to 7.x: Refactor kubelet metricsets to share response from endpoint #25982

Cherry-pick #25782 to 7.x: Refactor kubelet metricsets to share response from endpoint #25982

Conversation

ChrsMark commented May 28, 2021 • edited by zube bot Loading

What does this PR do?

Why is it important?

How to test this PR locally

Related issues

Screenshots

elasticmachine commented May 28, 2021

botelastic bot commented May 28, 2021

elasticmachine commented May 28, 2021

💔 Build Failed

Build stats

Test stats 🧪

Trends 🧪

Steps errors

metricbeat-packaging-linux - mage package

x-pack/metricbeat-packaging-linux - mage package

Error signal

Log output

💚 Flaky test report

Test stats 🧪

ChrsMark commented May 28, 2021 •

edited by zube bot

Loading

`metricbeat-packaging-linux - mage package`

`x-pack/metricbeat-packaging-linux - mage package`

`Error signal`