Add k8s metadata in state_cronjob metricset #29572

ChrsMark · 2021-12-22T08:27:03Z

What does this PR do?

This PR adds metadata enrichment for state_cronjob metricset.

Why is it important?

So as to be aligned with other metricsets.

Related issues

Closes state_cronjob lacks enricher #29189

Logs

Sample event

{
  "@timestamp": "2021-12-22T08:12:32.510Z",
  "@metadata": {
    "beat": "metricbeat",
    "type": "_doc",
    "version": "8.1.0"
  },
  "orchestrator": {
    "cluster": {
      "url": "kind-control-plane:6443",
      "name": "kind"
    }
  },
  "ecs": {
    "version": "8.0.0"
  },
  "host": {
    "name": "kind-control-plane"
  },
  "agent": {
    "name": "kind-control-plane",
    "type": "metricbeat",
    "version": "8.1.0",
    "ephemeral_id": "5b73917e-a756-4a78-9a26-db5fd108c1c5",
    "id": "8d46ce8b-bcd6-41f4-baa4-c2af42952625"
  },
  "service": {
    "address": "http://kube-state-metrics:8080/metrics",
    "type": "kubernetes"
  },
  "event": {
    "dataset": "kubernetes.cronjob",
    "module": "kubernetes",
    "duration": 315608278
  },
  "kubernetes": {
    "cronjob": {
      "is_suspended": false,
      "next_schedule": {
        "sec": 1.64016078e+09
      },
      "last_schedule": {
        "sec": 1.64016072e+09
      },
      "created": {
        "sec": 1.640159561e+09
      },
      "active": {
        "count": 0
      },
      "name": "hello"
    },
    "labels": {
      "foo": "bar"
    },
    "namespace": "default"
  },
  "metricset": {
    "period": 10000,
    "name": "state_cronjob"
  }
}

Signed-off-by: chrismark <chrismarkou92@gmail.com>

mergify · 2021-12-22T08:27:29Z

This pull request does not have a backport label. Could you fix it @ChrsMark? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v./d./d./d is the label to automatically backport to the 7./d branch. /d is the digit

NOTE: backport-skip has been added to this pull request.

elasticmachine · 2021-12-22T08:27:39Z

Pinging @elastic/integrations (Team:Integrations)

Signed-off-by: chrismark <chrismarkou92@gmail.com>

elasticmachine · 2021-12-22T08:35:29Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-01-10T14:35:59.148+0000
Duration: 131 min 0 sec
Commit: 9e7177c

Test stats 🧪

Test	Results
Failed	0
Passed	47267
Skipped	4155
Total	51422

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

Signed-off-by: chrismark <chrismarkou92@gmail.com>

tetianakravchenko

LGTM! I've left few questions

tetianakravchenko · 2021-12-22T09:52:35Z

libbeat/common/kubernetes/metadata/pod.go

@@ -88,6 +88,7 @@ func (p *pod) GenerateK8s(obj kubernetes.Resource, opts ...FieldOptions) common.
 	out := p.resource.GenerateK8s("pod", obj, opts...)

 	// check if Pod is handled by a ReplicaSet which is controlled by a Deployment
+	// TODO: same happens with CronJob vs Job. The hierarcy there is CronJob->Job->Pod


do we have a ticket for that?

tetianakravchenko · 2021-12-22T13:38:45Z

metricbeat/module/kubernetes/state_cronjob/state_cronjob_integration_test.go

@@ -30,7 +30,7 @@ import (
 )

 func TestFetchMetricset(t *testing.T) {
-	config := test.GetKubeStateMetricsConfig(t, "state_cronjob")
+	config := test.GetKubeStateMetricsConfigWithMetaDisabled(t, "state_cronjob")


why for cronjob only add_metadata must be disabled? as I see for other k8s resources used default value - true

Cause in version before v1.21 the the current stable api is not available and hence we cannot support metadata in tests for v1.20

tetianakravchenko · 2021-12-23T14:07:59Z

libbeat/common/kubernetes/metadata/resource.go

@@ -118,7 +118,9 @@ func (r *Resource) GenerateK8s(kind string, obj kubernetes.Resource, options ...
 			case "Deployment",
 				"ReplicaSet",
 				"StatefulSet",
-				"DaemonSet":
+				"DaemonSet",
+				"Job",


with this change we also can close this PR: #28954 ?

@tetianakravchenko my PR is a little bit tricky cause has some tricky parts around k8s api versions. I think we can move on with the community PR and I can rebase mine on top of it. wdyt?

mergify · 2021-12-26T23:44:28Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b k8s_cron_meta_enricher upstream/k8s_cron_meta_enricher
git merge upstream/master
git push upstream k8s_cron_meta_enricher

…icher

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark · 2022-01-10T13:26:05Z

/test

ChrsMark · 2022-01-11T09:29:12Z

@MichaelKatsoulis @tetianakravchenko I will need a final review on this. I changed a little bit the behaviour and the add_metadata setting is disabled by default in state_cronjob metricset until we only support stable versions of the corresponding API.

tetianakravchenko

LGTM!

…b-for-macos * upstream/master: (172 commits) [Elastic Agent] Fix issue with ensureServiceToken. (elastic#29800) [Winlogbeat] Add provider name to Security routing pipeline check (elastic#29781) Add summary to journeys which don't emit journey:end (early node subprocess exits) (elastic#29606) Prepare 8.0.0-rc1 changelog (elastic#29795) (elastic#29806) Change docker image from CentOS 7 to Ubuntu 20.04 (elastic#29681) libbeat/processors/add_process_metadata: implement a process cache eviction policy (elastic#29717) [Automation] Update elastic stack version to 8.1.0-7004acda for testing (elastic#29783) Missing changelog entry for elastic#29773 (elastic#29791) Add a readme for k8s autodiscover provider (elastic#28213) Remove overriding of index pattern on the Kubernetes overview dashboard (elastic#29676) jjbb: remove obsoleted branches (<7.16) (elastic#29707) Add k8s metadata in state_cronjob metricset (elastic#29572) ibmmq: Fix timestamp parsing (elastic#29773) Do not add date to index if `@meta.index` is set (elastic#29775) ci: uses aliases for the branches (elastic#29706) Filebeat tests: Restore `@timestamp` field validation (elastic#29772) Forward port 7.16.3 changelog to master (elastic#29777) auditd: Store program arguments in process.args array (elastic#29601) System/socket: Support kernel_clone() replacement for _do_fork() (elastic#29744) Do not mention removal if version is not specified in `cfgwarn` messages (elastic#29727) ...

Add k8s metadata in cronjob metricset

68fdec9

Signed-off-by: chrismark <chrismarkou92@gmail.com>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 22, 2021

ChrsMark requested review from tetianakravchenko and MichaelKatsoulis December 22, 2021 08:27

mergify bot added the backport-skip Skip notification from the automated backport with mergify label Dec 22, 2021

ChrsMark self-assigned this Dec 22, 2021

ChrsMark added the Team:Integrations Label for the Integrations team label Dec 22, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 22, 2021

ChrsMark added the backport-v8.1.0 Automated backport with mergify label Dec 22, 2021

mergify bot removed the backport-skip Skip notification from the automated backport with mergify label Dec 22, 2021

Add changelog

8605023

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark added 2 commits December 22, 2021 10:10

Disable metadata in cronjob tests

c3fd58e

Signed-off-by: chrismark <chrismarkou92@gmail.com>

fmt

d1722e3

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark mentioned this pull request Dec 22, 2021

Add support for latest k8s versions #29575

Merged

tetianakravchenko approved these changes Dec 22, 2021

View reviewed changes

tetianakravchenko reviewed Dec 23, 2021

View reviewed changes

ChrsMark mentioned this pull request Dec 24, 2021

Drop support for k8s versions that kubernetes has also dropped support #29604

Closed

ChrsMark added 2 commits January 10, 2022 10:56

Merge remote-tracking branch 'upstream/master' into k8s_cron_meta_enr…

0767797

…icher

Add metadata only on demand

9e7177c

Signed-off-by: chrismark <chrismarkou92@gmail.com>

ChrsMark mentioned this pull request Jan 10, 2022

Revisit how metadata are added for jobs/cronjobs #29771

Closed

ChrsMark requested a review from tetianakravchenko January 10, 2022 16:03

tetianakravchenko approved these changes Jan 11, 2022

View reviewed changes

ChrsMark merged commit 2f79c77 into elastic:master Jan 11, 2022

ChrsMark mentioned this pull request Jan 27, 2022

Add missing job.name and cronjob.name meta fields elastic/integrations#2608

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add k8s metadata in state_cronjob metricset #29572

Add k8s metadata in state_cronjob metricset #29572

ChrsMark commented Dec 22, 2021

mergify bot commented Dec 22, 2021

elasticmachine commented Dec 22, 2021

elasticmachine commented Dec 22, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

tetianakravchenko left a comment

tetianakravchenko Dec 22, 2021

ChrsMark Jan 10, 2022

tetianakravchenko Dec 22, 2021

ChrsMark Jan 10, 2022

tetianakravchenko Dec 23, 2021 •

edited

Loading

ChrsMark Dec 24, 2021

mergify bot commented Dec 26, 2021

ChrsMark commented Jan 10, 2022

ChrsMark commented Jan 11, 2022

tetianakravchenko left a comment

Add k8s metadata in state_cronjob metricset #29572

Add k8s metadata in state_cronjob metricset #29572

Conversation

ChrsMark commented Dec 22, 2021

What does this PR do?

Why is it important?

Related issues

Logs

mergify bot commented Dec 22, 2021

elasticmachine commented Dec 22, 2021

elasticmachine commented Dec 22, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

tetianakravchenko left a comment

Choose a reason for hiding this comment

tetianakravchenko Dec 22, 2021

Choose a reason for hiding this comment

ChrsMark Jan 10, 2022

Choose a reason for hiding this comment

tetianakravchenko Dec 22, 2021

Choose a reason for hiding this comment

ChrsMark Jan 10, 2022

Choose a reason for hiding this comment

tetianakravchenko Dec 23, 2021 • edited Loading

Choose a reason for hiding this comment

ChrsMark Dec 24, 2021

Choose a reason for hiding this comment

mergify bot commented Dec 26, 2021

ChrsMark commented Jan 10, 2022

ChrsMark commented Jan 11, 2022

tetianakravchenko left a comment

Choose a reason for hiding this comment

elasticmachine commented Dec 22, 2021 •

edited by jenkins-beats-ci bot

Loading

tetianakravchenko Dec 23, 2021 •

edited

Loading