Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Promtail] Duplicate prometheus register call when using Kafka #4865

Closed
taisho6339 opened this issue Dec 3, 2021 · 1 comment · Fixed by #4866
Closed

[Promtail] Duplicate prometheus register call when using Kafka #4865

taisho6339 opened this issue Dec 3, 2021 · 1 comment · Fixed by #4866

Comments

@taisho6339
Copy link
Contributor

taisho6339 commented Dec 3, 2021

Describe the bug
When we use Kafka target in Promtail, we can't use metrics stage due to the /metrics call failure with the following error.

14 error(s) occurred:
* collected metric "my_http_request_duration" { label:<name:"service" value:"nova-api" > label:<name:"env" value:"dev" > histogram:<sample_count:3151 sample_sum:81.74437090000009 bucket:<cumulative_count:3051 upper_bound:0.1 > bucket:<cumulative_count:3130 upper_bound:0.5 > bucket:<cumulative_count:3130 upper_bound:1 > bucket:<cumulative_count:3139 upper_bound:2.5 > bucket:<cumulative_count:3151 upper_bound:5 > > } was collected before with the same name and label values

In kafka target, this ConsumeClaim function is called in every kafka partition.
https://github.com/grafana/loki/blob/main/clients/pkg/promtail/targets/kafka/consumer.go#L86

It causes the duplicate Register call as the following.

(c *consumer) func ConsumeClaim -> (t *targetSyncer) NewTarget -> stages.NewPipeline -> stages.New -> newMetrics

https://github.com/grafana/loki/blob/main/clients/pkg/logentry/stages/metrics.go#L127

To Reproduce
Steps to reproduce the behavior:

  1. Started Promtail 2.4.1 with the config like this.
scrape_configs:
  - job_name: dev-kafka
    kafka:
      use_incoming_timestamp: true
      version: 2.4.2
      brokers:
        - xxxx
      group_id: promtail-dev
      topics:
        - grafana-loki-dev
      labels:
        env: dev
    pipeline_stages:
      - match:
          selector: '{env="dev"} |= "access_log"'
          stages:
            - json:
                expressions:
                  request_time:
            - metrics:
                http_request_duration:
                  type: Histogram
                  prefix: my_
                  source: request_time
                  description: "http request duration in API"
                  max_idle_duration: 24h
                  config:
                    action: inc
                    buckets:
                      - 0.1
                      - 0.5
                      - 1.0
                      - 2.5
                      - 5.0
  1. call /metrics endpoint

Expected behavior
A clear and concise description of what you expected to happen.

  • /metrics call success
@taisho6339
Copy link
Contributor Author

Let me fix this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant