Support for changing the cluster Label on Dashboards

[Daniel-Vaz]

Feature Request: Customizable Cluster Label Name in Exported Metrics

Is your feature request related to a problem? Please describe.

Currently, Harvest uses the cluster label on most of its metrics to reference an ONTAP cluster. In Grafana dashboards, this cluster label is utilized by the $Cluster variable to control various aspects of the panels.

However, the cluster label is commonly used by other exporters and Prometheus stack implementations. This overlap leads to conflicts, resulting in broken Grafana dashboards.

For example, when using a Prometheus stack hosted in Kubernetes to scrape and work with Harvest-exposed metrics, the cluster label is typically reserved as a Kubernetes cluster Prometheus external label to correctly identify the origin Kubernetes cluster. This practice is essential for maintaining large-scale, multi-tenant Prometheus setups.

Describe the solution you'd like

It would be beneficial to have the ability to customize the label name for the existing cluster label in the exported metrics.

Describe alternatives you've considered

For metrics ingestion, we can control the relabeling of the cluster label. However, for dashboards, the only workaround I've found so far is to manually edit all panels across all dashboards to change the $Cluster variable to something else, like $netapp_cluster.

[cgrinds]

hi @Daniel-Vaz what if we added a command line argument to bin/harvest grafana import that rewrote the dashboard panel expressions to use a custom cluster label?

You would run something like bin/grafana import --cluster-label netapp_cluster this would rewrite panel expressions. For example, this expression

sum(svm_vol_read_data+svm_vol_write_data{datacenter=~"$Datacenter",cluster=~"$Cluster",svm=~"$SVM"})

would be rewritten to the following, if you specified the argument --cluster-label netapp_cluster

sum(svm_vol_read_data+svm_vol_write_data{datacenter=~"$Datacenter",netapp_cluster=~"$Cluster",svm=~"$SVM"})

Would that work for you?

[Daniel-Vaz]

@cgrinds

Yes that was exactly what I was envisioning 😄
Would it be something that could be easily implemented ?

[cgrinds]

@Daniel-Vaz Yes, we can add this. Can you share your Prometheus relabel config so we can test the customized dashboards?

[Daniel-Vaz]

In our setup we scrape Harvest using the "Prometheus Operator"-managed ScrapeConfig Custom Resource.
For the relabelling we use the following metric relabelling logic:

  metricRelabelings:

# Transform the 'cluster' label into 'netapp_cluster' on all scraped metrics.
  - sourceLabels: [cluster]
    action: replace
    targetLabel: netapp_cluster

# Drop the 'cluster' label all together from the scraped metrics.
  - regex: cluster
    action: labeldrop

The cluster label for our Kubernetes cluster is then by default added to all our metrics on the RemoteWrite to a external Mimir stack.

    remoteWrite:
    - url: "{{ $mimirurl }}"
      name: mimir
      writeRelabelConfigs:
      - action: replace
        targetLabel: "cloud"
        replacement: "{{ $cloud }}"
      - action: replace
        targetLabel: "cluster"
        replacement: "{{ $k8sCluster }}"
      - action: replace
        targetLabel: "environment"
        replacement: "{{ $env }}"

If using the regular Prometheus config file make sure to change the configuration keys to the appropriate names (sourceLabels to source_labels, etc)

[cgrinds]

hi @Daniel-Vaz these changes are in the nightly build. Please let use know how they work for you.

bin/harvest grafana import now includes a --cluster-label option which rewrites all panel expressions to use the specified cluster label instead of the default 'cluster'

[cgrinds]

This is the config I used for local Prometheus testing

- job_name: 'harvest-no-cluster'
    metric_relabel_configs:   
    - source_labels: [cluster]
      target_label: 'netapp_cluster'
    - regex: cluster
      action: labeldrop

    static_configs:
    - targets:
        - 'localhost:17001'

[cgrinds]

Verified in 24.11 ddb97c5

bin/harvest grafana customize --output-dir /tmp/dash2 --cluster-label foople --directory grafana/dashboards/cmode
rg -vF 'source_cluster=~' /tmp/dash2 | rg -F 'cluster=~'
(no hits)