Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Clarifying metric yam configuration file for ksql #90

Open
okouyad opened this issue Oct 7, 2022 · 1 comment
Open

Clarifying metric yam configuration file for ksql #90

okouyad opened this issue Oct 7, 2022 · 1 comment

Comments

@okouyad
Copy link

okouyad commented Oct 7, 2022

I would like to understand the meaning of the comments on top of the metrics extraction as in this example

  #"kafka.consumer:type=consumer-node-metrics,client-id=*, node-id=*"
  # "kafka.consumer:type=consumer-fetch-manager-metrics,client-id=*, topic=*"
  # "kafka.producer:type=producer-node-metrics,client-id=*, node-id=*"
  # "kafka.producer:type=producer-topic-metrics,client-id=*, topic=*"
  - pattern: "kafka.(.+)<type=(.+), (.+)=(.+), (.+)=(.+)><>(.+):"
    name: kafka_$1_$2_$7
    type: GAUGE
    labels:
      client_type: $1
      $3: "$4"
      $5: "$6"

Do they stand for example of metrics extracted by the rule ?

Where i get confused is that, if that is the case why is it that at the top of the file we have in the blacklist the following:
blacklistObjectNames:

  - "io.confluent.ksql.metrics:name=*"
  - kafka.streams:type=kafka-metrics-count
  # This will ignore the admin client metrics from KSQL server and will blacklist certain metrics
  # that do not make sense for ingestion.
  - "kafka.admin.client:*"
  - "kafka.consumer:type=*,id=*"
  - "kafka.consumer:type=*,client-id=*"
  - "kafka.consumer:type=*,client-id=*,node-id=*"
  - "kafka.producer:type=*,id=*"
  - "kafka.producer:type=*,client-id=*"
  - "kafka.producer:type=*,client-id=*,node-id=*"
  - "kafka.streams:type=stream-processor-node-metrics,thread-id=*,task-id=*,processor-node-id=*"
  - "kafka.*:type=kafka-metrics-count,*"
  - "io.confluent.ksql.metrics:type=_confluent-ksql-rest-app-command-runner,*"

It feels like a contradiction to me.

I would like to inspire myself from the file, but i find it confusing at time. It seems we are blacklisting everything about consumer and producer, yet the rule below show how some of them might extract the very pattern that is blacklisted.

Can someone help me clear my confusion please

https://github.com/confluentinc/jmx-monitoring-stacks/blob/7.1-post/shared-assets/jmx-exporter/confluent_ksql.yml

@dberardo-com
Copy link

i have noticed that the exported metrics do not match up with the dashboard quantities in grafana ... is there a way to map it ??

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants