Metricbeat GCP module: Dashboards do not fully work

[felix-lessoer]

I'm working on ingesting GCP related data into an Elastic Cloud cluster.
I'm using the GCP Metricbeat module to ingest all relevant metric data as their is no Elastic Agent equivalent atm.

The GKE overview dashboard has some issues. Main one is that the field gcp.labels.resource.node_name gets not populated in my case. Also the namespace visualization has a query inside that is filtering to an explicit cluster name

Please include configurations and logs if available.
Metricbeat configuration:

# Module: gcp
# Docs: https://www.elastic.co/guide/en/beats/metricbeat/7.16/metricbeat-module-gcp.html

- module: gcp
  metricsets:
    - compute
    - pubsub
    - loadbalancing
    - storage
  #region: "us-"
  project_id: "***"
  credentials_file_path: "***"
  exclude_labels: false
  period: 1m

- module: gcp
  metricsets:
    - metrics
  project_id: "***"
  credentials_file_path: "***"
  exclude_labels: false
  period: 1m
  metrics:
    - aligner: ALIGN_NONE
      service: compute
      metric_types:
        - "instance/cpu/reserved_cores"
        - "instance/cpu/usage_time"
        - "instance/cpu/utilization"
        - "instance/uptime"

- module: gcp
  metricsets:
    - metrics
  project_id: "***"
  credentials_file_path: "***"
  exclude_labels: false
  period: 1m
  metrics:
    - aligner: ALIGN_NONE
      service: gke
      service_metric_prefix: kubernetes.io/
      metric_types:
        - "container/cpu/core_usage_time"

- module: gcp
  metricsets:
    - billing
  period: 24h
  project_id: "***"
  credentials_file_path: "***"
  dataset_id: "dataset id"
  table_pattern: "table pattern"
  cost_type: "regular"

For confirmed bugs, please report:

• Version: 7.16.2
• Operating System: Elastic Cloud and GCP
• Discuss Forum URL: https://elastic.slack.com/archives/CEFAY0599/p1641984148008700
• Steps to Reproduce:
  1.) Create a new cluster
  2.) Configure metricbeat to load as much data as possible from GCP module into ES. Use GCP project that has GKE enabled.
  3.) Setup Metricbeat
  4.) Open the GKE Overview dashboard

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[ChrsMark]

cc: @endorama

[endorama]

A fix for the unnecessary cluster name in the query is on the way.

[endorama]

I'm investigating why the gcp.labels.resource.node_name label is not included.

@felix-lessoer I see you are not using the gke metricset, but collecting metrics manually through metrics metricset and service gke. Is there any particular reason for this? (I don't think this is the cause though).

[felix-lessoer]

@endorama

The configuration above is what we have in the Docs. So thats the reason why it is like this.
I think the kubernetes cluster does not have kube state metrics installed if this is necessary.

Also I started to collect metrics from dataflow and logging metric types. It turned out that it is a bit hard to identify the service these metrics belong to if the user does not know the documentation from google. If you are working on that it would be a good idea to fix that too by publishing the service of the metrics too.

[endorama]

A fix for the wrong query values in ongoing in the linked PR.

@felix-lessoer I suspect that configuration is the issue; may you please try with:

  - module: gcp
    metricsets:
      - gke
    project_id: "gcpprojectid"
    credentials_file_path: "....json"
    exclude_labels: false
    period: 1m

GKE has a dedicated metricset. I tested it out today and gcp.labels.resource.node_name was collected as expected, and the dashboard was properly filled (see screenshot in linked PR). The documentation requires an update, I'd just like to understand if the configuration is the issue.

[endorama]

The fix to the dashboard will be release as part of 7.17.

[endorama]

@felix-lessoer do you have any update on this?

[endorama]

I published an update to the documentation in #30771 so that default configuration uses the gke metricset instead of the generic metrics one, target 8.2 and 8.1.