Skip to content
This repository has been archived by the owner on Feb 22, 2022. It is now read-only.

[incubator/kafka] Add JMX Exporter Agent #10799

Closed
wants to merge 20 commits into from
Closed

[incubator/kafka] Add JMX Exporter Agent #10799

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
0c742fc
[incubator/kafka] JMX exporter with agent
niqdev Jan 7, 2019
3cfa5ad
[incubator/kafka] Disable requirements
niqdev Jan 14, 2019
2d32bc1
[incubator/kafka] Disable requirements
niqdev Jan 14, 2019
6ff3382
[incubator/kafka] Update jmx configMap
niqdev Jan 14, 2019
f982bcb
[incubator/kafka] Update mountPath
niqdev Jan 14, 2019
44785b3
[incubator/kafka] Use curl instead of wget
niqdev Jan 14, 2019
514dc20
[incubator/kafka] Use docker image with curl
niqdev Jan 14, 2019
59fb4bb
[incubator/kafka] Add default JMX options
niqdev Jan 16, 2019
5d824fd
[incubator/kafka] Fix typo
niqdev Jan 16, 2019
0c0e58e
[incubator/kafka] Add service and service monitor
niqdev Jan 16, 2019
f46e09b
[incubator/kafka] Update service label
niqdev Jan 16, 2019
2f45f10
[incubator/kafka] Add servicemonitor labels
niqdev Jan 17, 2019
2a1c18e
[incubator/kafka] Update servicemonitor labels
niqdev Jan 17, 2019
e293651
[incubator/kafka] Update servicemonitor labels
niqdev Jan 17, 2019
68c6d59
[incubator/kafka] Remove servicemonitor labels
niqdev Jan 17, 2019
54f69ca
[incubator/kafka] Replace toYaml with loop for argocd
niqdev Jan 17, 2019
f854d1b
[incubator/kafka] Replace toYaml with loop for argocd
niqdev Jan 17, 2019
7e3282b
[incubator/kafka] Restore requirements
niqdev Jan 17, 2019
fc40792
[incubator/kafka] Update README
niqdev Jan 17, 2019
b179186
[incubator/kafka] Bump chart version
niqdev Jan 18, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion incubator/kafka/Chart.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ apiVersion: v1
description: Apache Kafka is publish-subscribe messaging rethought as a distributed
commit log.
name: kafka
version: 0.13.5
version: 0.14.0
appVersion: 5.0.1
keywords:
- kafka
Expand Down
11 changes: 8 additions & 3 deletions incubator/kafka/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,19 +105,24 @@ following configurable parameters:
| `jmx.port` | The jmx port which JMX style metrics are exposed (note: these are not scrapeable by Prometheus) | `5555` |
| `jmx.whitelistObjectNames` | Allows setting which JMX objects you want to expose to via JMX stats to JMX Exporter | (see `values.yaml`) |
| `prometheus.jmx.resources` | Allows setting resource limits for jmx sidecar container | `{}` |
| `prometheus.jmx.enabled` | Whether or not to expose JMX metrics to Prometheus | `false` |
| `prometheus.jmx.enabled` | Whether or not to expose JMX metrics to Prometheus as an independent HTTP server (alternative to `prometheus.agent`) | `false` |
| `prometheus.jmx.image` | JMX Exporter container image | `solsson/kafka-prometheus-jmx-exporter@sha256` |
| `prometheus.jmx.imageTag` | JMX Exporter container image tag | `a23062396cd5af1acdf76512632c20ea6be76885dfc20cd9ff40fb23846557e8` |
| `prometheus.jmx.interval` | Interval that Prometheus scrapes JMX metrics when using Prometheus Operator | `10s` |
| `prometheus.jmx.scrapeTimeout` | Timeout that Prometheus scrapes JMX metrics when using Prometheus Operator | `10s` |
| `prometheus.jmx.scrapeTimeout` | Timeout that Prometheus scrapes JMX metrics when using Prometheus Operator | `10s` |
| `prometheus.jmx.port` | JMX Exporter Port which exposes metrics in Prometheus format for scraping | `5556` |
| `prometheus.kafka.enabled` | Whether or not to create a separate Kafka exporter | `false` |
| `prometheus.kafka.image` | Kafka Exporter container image | `danielqsj/kafka-exporter` |
| `prometheus.kafka.imageTag` | Kafka Exporter container image tag | `v1.2.0` |
| `prometheus.kafka.interval` | Interval that Prometheus scrapes Kafka metrics when using Prometheus Operator | `10s` |
| `prometheus.kafka.scrapeTimeout` | Timeout that Prometheus scrapes Kafka metrics when using Prometheus Operator | `10s` |
| `prometheus.kafka.scrapeTimeout` | Timeout that Prometheus scrapes Kafka metrics when using Prometheus Operator | `10s` |
| `prometheus.kafka.port` | Kafka Exporter Port which exposes metrics in Prometheus format for scraping | `9308` |
| `prometheus.kafka.resources` | Allows setting resource limits for kafka-exporter pod | `{}` |
| `prometheus.agent.enabled` | Whether or not to expose JMX metrics to Prometheus as a Java Agent (alternative to `prometheus.jmx`) | `false` |
| `prometheus.agent.version` | JMX Agent Exporter version | `0.3.1` |
| `prometheus.agent.interval` | Interval that Prometheus scrapes JMX metrics when using Prometheus Operator | `10s` |
| `prometheus.agent.scrapeTimeout` | Timeout that Prometheus scrapes JMX metrics when using Prometheus Operator | `10s` |
| `prometheus.agent.port` | JMX Exporter Port which exposes metrics in Prometheus format for scraping | `8080` |
| `prometheus.operator.enabled` | True if using the Prometheus Operator, False if not | `false` |
| `prometheus.operator.serviceMonitor.namespace` | Namespace which Prometheus is running in. Default to kube-prometheus install. | `monitoring` |
| `prometheus.operator.serviceMonitor.selector` | Default to kube-prometheus install (CoreOS recommended), but should be set according to Prometheus install | `{ prometheus: kube-prometheus }` |
Expand Down
2 changes: 1 addition & 1 deletion incubator/kafka/templates/configmap-jmx.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{{- if and .Values.prometheus.jmx.enabled .Values.jmx.configMap.enabled }}
{{- if .Values.jmx.configMap.enabled }}
apiVersion: v1
kind: ConfigMap
metadata:
Expand Down
21 changes: 21 additions & 0 deletions incubator/kafka/templates/service-brokers.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,3 +42,24 @@ spec:
app: {{ include "kafka.name" . }}-exporter
release: {{ .Release.Name }}
{{- end }}
---
{{- if and .Values.prometheus.agent.enabled .Values.prometheus.operator.enabled }}
apiVersion: v1
kind: Service
metadata:
name: {{ include "kafka.fullname" . }}-agent-exporter
labels:
app: {{ include "kafka.name" . }}-agent-exporter
chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:
ports:
- name: agent-exporter
protocol: TCP
port: {{ .Values.prometheus.agent.port }}
targetPort: {{ .Values.prometheus.agent.port }}
selector:
app: {{ include "kafka.name" . }}
release: {{ .Release.Name }}
{{- end }}
29 changes: 28 additions & 1 deletion incubator/kafka/templates/servicemonitors.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,9 @@ metadata:
name: {{ include "kafka.fullname" . }}-exporter
namespace: {{ .Values.prometheus.operator.serviceMonitor.namespace }}
labels:
{{ toYaml .Values.prometheus.operator.serviceMonitor.selector | indent 4 }}
{{- range $key, $value := .Values.prometheus.operator.serviceMonitor.selector }}
{{ $key }}: {{ $value | quote }}
{{- end }}
spec:
selector:
matchLabels:
Expand All @@ -43,3 +45,28 @@ spec:
namespaceSelector:
any: true
{{ end }}
---
{{ if and .Values.prometheus.agent.enabled .Values.prometheus.operator.enabled }}
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: {{ include "kafka.fullname" . }}-agent-exporter
namespace: {{ .Values.prometheus.operator.serviceMonitor.namespace }}
labels:
{{- range $key, $value := .Values.prometheus.operator.serviceMonitor.selector }}
{{ $key }}: {{ $value | quote }}
{{- end }}
spec:
selector:
matchLabels:
app: {{ include "kafka.name" . }}-agent-exporter
release: {{ .Release.Name }}
endpoints:
- port: agent-exporter
interval: {{ .Values.prometheus.agent.interval }}
{{- if .Values.prometheus.agent.scrapeTimeout }}
scrapeTimeout: {{ .Values.prometheus.agent.scrapeTimeout }}
{{- end }}
namespaceSelector:
any: true
{{ end }}
36 changes: 34 additions & 2 deletions incubator/kafka/templates/statefulset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,9 +34,11 @@ spec:
{{- if .Values.rbac.enabled }}
serviceAccountName: {{ .Release.Name }}
{{- end }}
{{- if or .Values.prometheus.agent.enabled .Values.external.enabled }}
initContainers:
{{- end }}
{{- if .Values.external.enabled }}
## ref: https://github.com/Yolean/kubernetes-kafka/blob/master/kafka/50kafka.yml
initContainers:
- name: init-ext
image: "{{ .Values.external.init.image }}:{{ .Values.external.init.imageTag }}"
imagePullPolicy: "{{ .Values.external.init.imagePullPolicy }}"
Expand All @@ -54,6 +56,19 @@ spec:
fieldRef:
fieldPath: metadata.namespace
{{- end }}
{{- if .Values.prometheus.agent.enabled }}
- name: curl-jmx-jar
image: governmentpaas/curl-ssl
imagePullPolicy: IfNotPresent
command:
- curl
- "-o"
- "/tmp/jmx-agent/jmx_prometheus_javaagent-{{ .Values.prometheus.agent.version }}.jar"
- https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/{{ .Values.prometheus.agent.version }}/jmx_prometheus_javaagent-{{ .Values.prometheus.agent.version }}.jar
volumeMounts:
- mountPath: /tmp/jmx-agent
name: jmx-agent
{{- end }}
{{- if .Values.tolerations }}
tolerations:
{{ toYaml .Values.tolerations | indent 8 }}
Expand Down Expand Up @@ -164,6 +179,12 @@ spec:
fieldPath: status.podIP
- name: KAFKA_HEAP_OPTS
value: {{ .Values.kafkaHeapOptions }}
{{- if .Values.prometheus.agent.enabled }}
- name: KAFKA_JMX_OPTS_DEFAULT
value: "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
- name: KAFKA_JMX_OPTS
value: "$(KAFKA_JMX_OPTS_DEFAULT) -javaagent:/var/lib/prometheus/jmx_prometheus_javaagent-{{ .Values.prometheus.agent.version }}.jar={{ .Values.prometheus.agent.port }}:/etc/prometheus/jmx-kafka-prometheus.yml"
{{- end }}
{{- if not (hasKey .Values.configurationOverrides "zookeeper.connect") }}
- name: KAFKA_ZOOKEEPER_CONNECT
value: {{ include "zookeeper.url" . | quote }}
Expand Down Expand Up @@ -226,12 +247,19 @@ spec:
{{- end }}
{{- end }}
{{- end }}
{{- if .Values.prometheus.agent.enabled }}
- name: jmx-config
mountPath: /etc/prometheus
- name: jmx-agent
mountPath: /var/lib/prometheus/jmx_prometheus_javaagent-{{ .Values.prometheus.agent.version }}.jar
subPath: jmx_prometheus_javaagent-{{ .Values.prometheus.agent.version }}.jar
{{- end }}
volumes:
{{- if not .Values.persistence.enabled }}
- name: datadir
emptyDir: {}
{{- end }}
{{- if .Values.prometheus.jmx.enabled }}
{{- if .Values.jmx.configMap.enabled }}
- name: jmx-config
configMap:
{{- if .Values.jmx.configMap.overrideName }}
Expand All @@ -240,6 +268,10 @@ spec:
name: {{ include "kafka.fullname" . }}-metrics
{{- end }}
{{- end }}
{{- if .Values.prometheus.agent.enabled }}
- name: jmx-agent
emptyDir: {}
{{- end }}
{{- range .Values.secrets }}
- name: {{ include "kafka.fullname" $ }}-{{ .name }}
secret:
Expand Down
16 changes: 16 additions & 0 deletions incubator/kafka/values.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -306,6 +306,22 @@ prometheus:
# cpu: 100m
# memory: 100Mi

## Prometheus Java Agent JMX Exporter
agent:
enabled: false

## https://github.com/prometheus/jmx_exporter/releases
version: 0.3.1

## Interval at which Prometheus scrapes metrics, note: only used by Prometheus Operator
interval: 10s

## Timeout at which Prometheus timeouts scrape run, note: only used by Prometheus Operator
scrapeTimeout: 10s

## Port agent exposes for Prometheus to scrape metrics
port: 8080

operator:
## Are you using Prometheus Operator?
enabled: false
Expand Down