Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Agent example input autodiscover config for standalone k8s #24157

Merged
merged 1 commit into from
Feb 23, 2021

Conversation

ChrsMark
Copy link
Member

@ChrsMark ChrsMark commented Feb 22, 2021

What does this PR do?

This PR is a follow up of #23938 to add a commented out section within the k8s manifest as an example for enabling autodiscover for specific target based on dynamic inputs and k8s provider.

It comes along with elastic/observability-docs#401.

Part of #23613

Why is it important?

So as to provide an easy way for the users to define their own autodiscover condition based inputs.

How to test this PR locally

  1. Uncoment the added section
  2. Deploy Agent on k8s: kubectl apply -f elastic-agent-standalone-kubernetes.yml
  3. Deploy a target Redis Pod:
apiVersion: v1
kind: Service
metadata:
  name: redis
  labels:
    app: redis
spec:
  clusterIP: None
  ports:
  - name: web
    port: 6379
    protocol: TCP
  selector:
    app: redis
  type: ClusterIP
---
apiVersion: v1
kind: Pod
metadata:
  name: redis
  labels:
    role: main
    app: redis
spec:
  containers:
    - name: redis
      image: redis
  1. Verify that redis integration start collecting redis metrics from the Pod. In Kibana discover filter on redis.info for event.dataset and expect to events arriving looking like the one below inScreenshots part.

Screenshots

Sample event:

{
  "_index": ".ds-metrics-redis.info-default-2021.02.22-000001",
  "_type": "_doc",
  "_id": "m_cZyncB1P74KB3B7J-v",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2021-02-22T14:16:55.832Z",
    "service": {
      "version": "6.0.10",
      "address": "redis://10.244.1.8:6379",
      "type": "redis"
    },
    "os": {
      "full": "Linux 4.9.184-linuxkit x86_64"
    },
    "redis": {
      "info": {
        "stats": {
          "latest_fork_usec": 0,
          "keyspace": {
            "hits": 0,
            "misses": 0
          },
          "active_defrag": {
            "key_hits": 0,
            "key_misses": 0,
            "hits": 0,
            "misses": 0
          },
          "pubsub": {
            "patterns": 0,
            "channels": 0
          },
          "connections": {
            "received": 1,
            "rejected": 0
          },
          "keys": {
            "expired": 0,
            "evicted": 0
          },
          "commands_processed": 18,
          "sync": {
            "full": 0,
            "partial": {
              "err": 0,
              "ok": 0
            }
          },
          "instantaneous": {
            "input_kbps": 0,
            "output_kbps": 0,
            "ops_per_sec": 0
          },
          "migrate_cached_sockets": 0,
          "slave_expires_tracked_keys": 0,
          "net": {
            "input": {
              "bytes": 504
            },
            "output": {
              "bytes": 32651
            }
          }
        },
        "cluster": {
          "enabled": false
        },
        "clients": {
          "blocked": 0,
          "connected": 1,
          "longest_output_list": 0,
          "biggest_input_buf": 8,
          "max_output_buffer": 0,
          "max_input_buffer": 8
        },
        "persistence": {
          "loading": false,
          "rdb": {
            "bgsave": {
              "in_progress": false,
              "last_time": {
                "sec": -1
              },
              "current_time": {
                "sec": -1
              },
              "last_status": "ok"
            },
            "copy_on_write": {
              "last_size": 0
            },
            "last_save": {
              "changes_since": 0,
              "time": 1614003324
            }
          },
          "aof": {
            "fsync": {},
            "enabled": false,
            "rewrite": {
              "last_time": {
                "sec": -1
              },
              "current_time": {
                "sec": -1
              },
              "buffer": {},
              "in_progress": false,
              "scheduled": false
            },
            "bgrewrite": {
              "last_status": "ok"
            },
            "write": {
              "last_status": "ok"
            },
            "copy_on_write": {
              "last_size": 0
            },
            "buffer": {},
            "size": {}
          }
        },
        "replication": {
          "slave": {},
          "role": "master",
          "connected_slaves": 0,
          "master_offset": 0,
          "backlog": {
            "size": 1048576,
            "first_byte_offset": 0,
            "histlen": 0,
            "active": 0
          },
          "master": {
            "sync": {},
            "offset": 0,
            "second_offset": -1
          }
        },
        "memory": {
          "max": {
            "value": 0,
            "policy": "noeviction"
          },
          "fragmentation": {
            "ratio": 9.53,
            "bytes": 7031992
          },
          "active_defrag": {
            "is_running": false
          },
          "allocator": "jemalloc-5.1.0",
          "allocator_stats": {
            "allocated": 1135544,
            "active": 1355776,
            "resident": 4169728,
            "fragmentation": {
              "ratio": 1.19,
              "bytes": 220232
            },
            "rss": {
              "bytes": 2813952,
              "ratio": 3.08
            }
          },
          "used": {
            "rss": 7856128,
            "peak": 866680,
            "lua": 37888,
            "dataset": 43008,
            "value": 866680
          }
        },
        "cpu": {
          "used": {
            "sys_children": 0,
            "user_children": 0,
            "sys": 0.07,
            "user": 0.07
          }
        },
        "server": {
          "mode": "standalone",
          "run_id": "fa112bef9093d2ac5ccff855be0ae270375ad0f6",
          "lru_clock": 3390679,
          "hz": 10,
          "gcc_version": "8.3.0",
          "build_id": "1c3aed2f9fd88f03",
          "config_file": "",
          "git_dirty": "0",
          "uptime": 91,
          "tcp_port": 6379,
          "arch_bits": "64",
          "multiplexing_api": "epoll",
          "git_sha1": "00000000"
        },
        "slowlog": {
          "count": 0
        }
      }
    },
    "data_stream": {
      "type": "metrics",
      "dataset": "redis.info",
      "namespace": "default"
    },
    "elastic_agent": {
      "id": "9392caf0-cdfc-4abf-981b-f98cb6b9edad",
      "snapshot": true,
      "version": "8.0.0"
    },
    "host": {
      "hostname": "kind-worker",
      "architecture": "x86_64",
      "os": {
        "family": "redhat",
        "name": "CentOS Linux",
        "kernel": "4.9.184-linuxkit",
        "codename": "Core",
        "platform": "centos",
        "version": "7 (Core)"
      },
      "id": "17af4efd6a5716fb64a196f3cd53c4f8",
      "containerized": true,
      "name": "kind-worker",
      "ip": [
        "10.244.1.1",
        "10.244.1.1",
        "10.244.1.1",
        "172.18.0.2",
        "fc00:f853:ccd:e793::2",
        "fe80::42:acff:fe12:2"
      ],
      "mac": [
        "ba:4a:a1:a2:9a:81",
        "9a:80:f1:73:34:7a",
        "2e:0a:1c:40:46:f8",
        "02:42:ac:12:00:02"
      ]
    },
    "process": {
      "pid": 1
    },
    "event": {
      "module": "redis",
      "duration": 666897,
      "dataset": "redis.info"
    },
    "metricset": {
      "name": "info",
      "period": 10000
    },
    "kubernetes": {
      "namespace": "default",
      "pod": {
        "labels": {
          "role": "main",
          "app": "redis"
        },
        "name": "redis",
        "uid": "150148b5-b0a0-4237-b1ea-b2c2634fba59",
        "ip": "10.244.1.8"
      }
    },
    "agent": {
      "ephemeral_id": "7b49f72c-73c2-49be-99c9-b09d55fcd7b9",
      "id": "22e06d3f-a7cb-464c-b813-dee4302ca78a",
      "name": "kind-worker",
      "type": "metricbeat",
      "version": "8.0.0"
    },
    "ecs": {
      "version": "1.7.0"
    }
  },
  "fields": {
    "@timestamp": [
      "2021-02-22T14:16:55.832Z"
    ]
  },
  "highlight": {
    "event.dataset": [
      "@kibana-highlighted-field@redis.info@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1614003415832
  ]
}

Signed-off-by: chrismark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark added Team:Integrations Label for the Integrations team kubernetes Enable builds in the CI for kubernetes Team:Elastic-Agent Label for the Agent team labels Feb 22, 2021
@ChrsMark ChrsMark self-assigned this Feb 22, 2021
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations (Team:Integrations)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/agent (Team:Agent)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Feb 22, 2021
@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #24157 opened

  • Start Time: 2021-02-22T14:32:11.177+0000

  • Duration: 53 min 54 sec

  • Commit: b828ce8

Trends 🧪

Image of Build Times

❕ Flaky test report

No test was executed to be analysed.

@ChrsMark ChrsMark merged commit 05b523a into elastic:master Feb 23, 2021
v1v added a commit to v1v/beats that referenced this pull request Feb 23, 2021
…-arm

* upstream/master: (24 commits)
  Add example input autodsicover config (elastic#24157)
  Empty configuration options generate `<no value>` string for azure-eventhub input (elastic#24156)
  Remove OSS reference for kibana and elasticsearch (elastic#24164)
  Skip flaky TestActions on MacOSx (elastic#23966)
  [Filebeat][AWS] Fix vpcflow pipeline exception: Cannot invoke "Object.getClass()" because "receiver" is null (elastic#24167)
  [Elastic Agent] Fix docker entrypoint for elastic-agent. (elastic#24155)
  [PACKAGING] Push docker images with the architecture in the version (elastic#24121)
  [Agent] Add agent standalone manifests for system module & Pod's log collection (elastic#23938)
  indicator type url is in upper case (elastic#24152)
  [Filebeat] Document netflow internal_networks and set default (elastic#24110)
  [Filebeat] Adding fixes to the TI module (elastic#24133)
  [Enhancement] Add RotateOnStartup feature flag for file output (elastic#19347)
  [Ingest Manager] Fix: Successfully installed and enrolled agent running standalone (elastic#24128)
  Set Elastic licence type for APM server Beats update job (elastic#24122)
  Add logrotation section on Running Filebeat on k8s (elastic#24120)
  [CI] Run if manual UI (elastic#24116)
  [CI] enable x-pack/heartbeat in the CI (elastic#23873)
  chore: comment out the E2E (elastic#24109)
  chore: add-backport-next (elastic#24098)
  Adjust the position of the architecture name in Dockerlogbeat tarball (elastic#24095)
  ...
v1v added a commit to v1v/beats that referenced this pull request Feb 23, 2021
…dows-7

* upstream/master:
  [CI] Add ARM packaging (elastic#24041)
  Add example input autodsicover config (elastic#24157)
  Empty configuration options generate `<no value>` string for azure-eventhub input (elastic#24156)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kubernetes Enable builds in the CI for kubernetes Team:Elastic-Agent Label for the Agent team Team:Integrations Label for the Integrations team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants