[Metricbeat] Add Load Balancing metricset to Google Cloud Platform module #15559

sayden · 2020-01-14T22:17:39Z

Adds Load Balancing metricset to Google Cloud Platform module.

We must take a look at the format of the distribution data that is returned by GCP in case we want to omit them as the "nature" of the lightweight module doesn't allow special modifications in the incoming data from the "engine"

c.f. #15811

mtojek

Nice! Only a couple of nitpicks.

mtojek · 2020-01-15T10:07:44Z

x-pack/metricbeat/module/googlecloud/loadbalancing/_meta/docs.asciidoc

+- `loadbalancing.https.backend_request_count`: The number of requests served by backends of HTTP/S load balancer.
+- `loadbalancing.https.backend_response_bytes_count`: The number of bytes sent as responses from backends (or cache) to HTTP/S load balancer.
+- `loadbalancing.https.frontend_tcp_rtt`: A distribution of the RTT measured for each connection between client and proxy.
+- `loadbalancing.https.backend_latencies`: A distribution of the latency calculated from when the request was sent by the proxy to the backend until the proxy received from the backend the last byte of response.


nit: you can sort all fields for searching purposes

Good idea, indeed. Sorted.

mtojek · 2020-01-15T10:08:19Z

x-pack/metricbeat/module/googlecloud/loadbalancing/_meta/docs.asciidoc

@@ -0,0 +1,32 @@
+Load Balancing Metricset to fetch metrics from https://cloud.google.com/load-balancing/[Load Balancing] in Google Cloud Platform.


nit:s/Metricset/metricset/g

mtojek · 2020-01-15T10:09:00Z

x-pack/metricbeat/module/googlecloud/loadbalancing/manifest.yml

+        - "loadbalancing.googleapis.com/https/backend_request_bytes_count"
+        - "loadbalancing.googleapis.com/https/backend_request_count"
+        - "loadbalancing.googleapis.com/https/backend_response_bytes_count"
+        - "loadbalancing.googleapis.com/https/frontend_tcp_rtt"


nit: please sort

Aaand sorted

mtojek · 2020-01-15T10:16:33Z

x-pack/metricbeat/module/googlecloud/stackdriver/metadata_services.go

+	case googlecloud.ServicePubsub:
+		return nil, nil
+	case googlecloud.ServiceLoadBalancing:
+		return nil, nil


nit: case googlecloud.ServicePubsub, googlecloud.ServiceLoadBalancing:

mtojek · 2020-01-15T10:17:01Z

x-pack/metricbeat/module/googlecloud/stackdriver/metrics_requester.go

+	case googlecloud.ServicePubsub:
+		return
+	case googlecloud.ServiceLoadBalancing:
+		return


nit: case googlecloud.ServicePubsub, googlecloud.ServiceLoadBalancing:

exekias · 2020-01-15T10:22:16Z

x-pack/metricbeat/module/googlecloud/stackdriver/metadata_services.go

+	case googlecloud.ServicePubsub:
+		return nil, nil
+	case googlecloud.ServiceLoadBalancing:
+		return nil, nil
 	default:
 		return nil, errors.Errorf("service '%s' not supported", c.ServiceName)


what do you think about returning nil, nil by default when the service is not supported? This way you don't need to keep adding services when adding a new metricset

... or the other way round - nil, nil for supported services except customized ones. The code will need to be adjusted if only you need a custom call to NewMetadataService. Unsupported service will still report an error.

problem with that is that the list of services is forever growing (see they are just constants we defined in our side). Every time a new service is added to stackdriver we would need to add it here, with little pay off, as this is a function to optionally enrich events when we know how to retrieve their metadata. It should do nothing for the rest.

What do you think about such construct? The order is important, that's the only requirement, but we can cover it with unit tests:

func NewMetadataServiceForConfig(c config) (googlecloud.MetadataService, error) { switch c.ServiceName { case googlecloud.ServiceCompute: return compute.NewMetadataService(c.ProjectID, c.Zone, c.opt...) case supportedServicesAsEnum(): return nil, nil default: return nil, errors.Errorf("service '%s' not supported", c.ServiceName) } }

supportedServicesAsEnum() would read all defined constant values.

I think I understand what you propose. My concern comes from this scenario:

I'm happily using the stackdriver metricset with Metricbeat 7.6, I use it to monitor all my GCP services. After some time GCP adds a new service to their offering, I try to monitor it with stackdriver metricset and get this error: service 'foo' not supported. Why is it not supported if the way to collect metrics is always the same?

Then I will need to wait for the Beats team to include the service name in this enum and release a new version, then upgrade to it. All this because of how we handle metadata, which is, at the end, optional

Agree here, but this implicates getting rid of constants at all.

EDIT:

To be clear - I'm not blocking this PR. Feel free to go!

I see your points. The thing was to support 3 cases here:

A correct / known GCP Service which is supported, having or not a specific implementation for labels (potentially we should aim to reach the highest quality and to have implementations for each metricset).

A correct GCP Service which is not supported: Service recognized but labelling is not supported yet. Action point -> Open enhancement request.

An incorrect / unknown GCP service: Service not recognized -> Action point: check service name.

The problem is that in the first PR we removed all "future metricsets" as a request and this function stay, looking naive.

One thing that is worth mention: When adding a new metricset, you don't "just" add the metrics you want to fetch. You must add the fields.yml and the docs.asciidoc and, potentially, you might add an implementation to fetch labels if the current metrics looks incomplete. I like this explicit approach where my future self will face a "metricset not recognized" while developing a new metricset if I don't pass by here to see that a labelling implementation might be needed.

I don't want to discourage comments here but this was discussed in the main module PR 😅 We can change it here but I think it must go in a different PR (because we have 3 live PR's which are using this code right now).

…dule (elastic#15559) (cherry picked from commit 978d676)

…dule (elastic#15559)

…to Google Cloud Platform module (#16422) * [Metricbeat] Add Load Balancing metricset to Google Cloud Platform module (#15559) (cherry picked from commit 978d676) * Update changelog entry

sayden added enhancement Metricbeat Metricbeat labels Jan 14, 2020

sayden requested a review from a team as a code owner January 14, 2020 22:17

sayden self-assigned this Jan 14, 2020

mtojek approved these changes Jan 15, 2020

View reviewed changes

exekias reviewed Jan 15, 2020

View reviewed changes

sayden force-pushed the feature/xp/mb/gcp/loadbalancing branch from b5b9e00 to d8ca690 Compare January 16, 2020 14:48

andresrc added [zube]: Inbox Team:Integrations Label for the Integrations team [zube]: In Review and removed [zube]: Inbox labels Jan 27, 2020

andresrc requested a review from a team January 27, 2020 10:33

sayden added 11 commits February 13, 2020 20:34

Loadbalancing atomic commit

463274f

Add docs

8987ad0

Update yamls

e8f0d00

Add code to Stackdriver metricset

f925f2b

add data.json file

c067874

add fields.yml

5ae141c

Update fields.go

25b1f50

Remove region from config

c3fdec6

Add Changelog entry

280e709

Address comments

f2f3b5e

Fix changelog

a445896

sayden force-pushed the feature/xp/mb/gcp/loadbalancing branch from d8ca690 to a445896 Compare February 13, 2020 19:36

sayden merged commit 978d676 into elastic:master Feb 14, 2020

zube bot added [zube]: Done and removed [zube]: In Review labels Feb 14, 2020

sayden added the needs_backport PR is waiting to be backported to other branches. label Feb 14, 2020

andresrc removed the [zube]: Done label Feb 19, 2020

sayden mentioned this pull request Feb 19, 2020

Cherry-pick #15559 to 7.x: [Metricbeat] Add Load Balancing metricset to Google Cloud Platform module #16422

Merged

sayden added a commit to sayden/beats that referenced this pull request Feb 20, 2020

[Metricbeat] Add Load Balancing metricset to Google Cloud Platform mo…

98879a3

…dule (elastic#15559) (cherry picked from commit 978d676)

kvch pushed a commit to kvch/beats that referenced this pull request Feb 20, 2020

[Metricbeat] Add Load Balancing metricset to Google Cloud Platform mo…

ac6af39

…dule (elastic#15559)

kaiyan-sheng mentioned this pull request Mar 19, 2020

[Metricbeat] Add dashboard for loadbalancing metricset in googlecloud module #16535

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Metricbeat] Add Load Balancing metricset to Google Cloud Platform module #15559

[Metricbeat] Add Load Balancing metricset to Google Cloud Platform module #15559

sayden commented Jan 14, 2020 •

edited by andresrc

Loading

mtojek left a comment

mtojek Jan 15, 2020

sayden Jan 16, 2020

mtojek Jan 15, 2020

sayden Jan 16, 2020

mtojek Jan 15, 2020

sayden Jan 16, 2020

mtojek Jan 15, 2020

sayden Jan 16, 2020

mtojek Jan 15, 2020

sayden Jan 16, 2020

exekias Jan 15, 2020

mtojek Jan 15, 2020

exekias Jan 15, 2020

mtojek Jan 15, 2020 •

edited

Loading

exekias Jan 15, 2020

mtojek Jan 15, 2020 •

edited

Loading

sayden Jan 16, 2020

		@@ -0,0 +1,32 @@
		Load Balancing Metricset to fetch metrics from https://cloud.google.com/load-balancing/[Load Balancing] in Google Cloud Platform.

[Metricbeat] Add Load Balancing metricset to Google Cloud Platform module #15559

[Metricbeat] Add Load Balancing metricset to Google Cloud Platform module #15559

Conversation

sayden commented Jan 14, 2020 • edited by andresrc Loading

mtojek left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtojek Jan 15, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtojek Jan 15, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sayden commented Jan 14, 2020 •

edited by andresrc

Loading

mtojek Jan 15, 2020 •

edited

Loading

mtojek Jan 15, 2020 •

edited

Loading