Update to OpenCensus version 0.22.3 #1446

aLekSer · 2020-04-02T16:25:58Z

This should fix Prometheus error messages on exporting with empty tags
list . Allocator Metrics.

Takeover this PR #893
Part of #1330
Closes #892

Fixing this error in logs (happened periodically):

textPayload: "2020/02/07 15:14:14 Failed to export to Prometheus: inconsistent label cardinality: expected 1 label values but got 0 in []string(nil)

agones-bot · 2020-04-02T16:36:43Z

Build Failed 😱

Build Id: 756bedfa-a170-43fe-81ba-cc0b82d3eccd

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2020-04-03T09:26:05Z

Build Failed 😱

Build Id: 886a4d3f-701d-439a-b665-90959e7607a7

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2020-04-03T09:28:40Z

Build Failed 😱

Build Id: 2a354681-7c89-4083-954a-3733796c5ea1

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2020-04-03T09:45:33Z

Build Succeeded 👏

Build Id: cb45e65e-87e9-4f40-a0f7-8e1ede5fe740

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.5.0-82fd75e
image: gcr.io/agones-images/agones-sdk:1.5.0-82fd75e
image: gcr.io/agones-images/agones-ping:1.5.0-82fd75e
Linux C++ SDK (build): agonessdk-1.5.0-82fd75e-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.5.0-82fd75e.zip

A preview of the website (the last 30 builds are retained):

https://82fd75e-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1446/head:pr_1446 && git checkout pr_1446
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.5.0-82fd75e

agones-bot · 2020-04-03T10:19:01Z

Build Succeeded 👏

Build Id: 05af2bc5-7f24-48e6-b9d8-27cf385c4e7f

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.5.0-88fddd5
image: gcr.io/agones-images/agones-sdk:1.5.0-88fddd5
image: gcr.io/agones-images/agones-ping:1.5.0-88fddd5
Linux C++ SDK (build): agonessdk-1.5.0-88fddd5-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.5.0-88fddd5.zip

A preview of the website (the last 30 builds are retained):

https://88fddd5-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1446/head:pr_1446 && git checkout pr_1446
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.5.0-88fddd5

aLekSer · 2020-04-03T16:08:50Z

Tested this change with Grafana and Prometheus.
Error with inconsistent label cardinality now does not happen in the logs.
#1330 (comment)

markmandel · 2020-04-03T16:11:13Z

@roberthbailey @pooneh-m should we aim to merge this in this release, or hold it until next release to give it time to bake? (Since RC is next Tuesday)

I'm inclined to push this to the next release, just to give it time to be tested over time. WDYT?

aLekSer · 2020-04-03T16:14:06Z

Setting it to on-hold. Test on Stackdriver failed:

2020/04/03 16:11:56 Failed to export to Stackdriver: rpc error: code = InvalidArgument desc = Field timeSeries[17].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry.

aLekSer · 2020-04-03T16:18:15Z

Strange enough, but I was fixing similar thing a while ago:
#554
Testing an update of stackdriver exporter:

-       contrib.go.opencensus.io/exporter/stackdriver v0.8.0
+       contrib.go.opencensus.io/exporter/stackdriver v0.13.1

roberthbailey · 2020-04-03T16:24:28Z

@aLekSer - can you split the vendor changes into a separate commit from the changes to pkg/ to make it easier to see how much impact there is on our code?

@markmandel - at first glance this seems to be making non-trivial code changes to our metric exporting which makes me tend to agree to push it to the next release to give it time to bake. On the other hand, how much extra testing does that actually get us? how much testing of this code path happens during our e2es? and would anyone test a pre-release build to verify the functionality on top of what the author and reviewer do? If not and it isn't destabilizing, then that would argue to get it in now since we would only get feedback on how well it works once it's released.

aLekSer · 2020-04-03T16:28:15Z

Similar error is mentioned here also:
census-ecosystem/opencensus-go-exporter-stackdriver#234
@roberthbailey I am going to split these two changes properly (./pkg/ and go get), will try to do it inside this branch.

markmandel · 2020-04-09T18:30:46Z

Just taking a look - at first glance LGTM, @cyriltovena do you have time to double check this, since it was originally your code?

cyriltovena · 2020-04-09T18:32:07Z

I'll run through it quickly now

cyriltovena · 2020-04-09T18:33:06Z

pkg/gameserverallocations/metrics.go

-	tag.Insert(keyFleetName, "none"),
-	tag.Insert(keyNodeName, "none"),
-	tag.Insert(keyStatus, "none"),
+	tag.Insert(keyMultiCluster, ""),


I'm guessing now this is allowed to be empty ?

Insert() should add this tag https://github.com/census-instrumentation/opencensus-go/blob/master/tag/map.go#L107:6
I will check tomorrow on how this applies to this metrics. Probably add a test for this.

@cyriltovena This change break Stackdriver exporter, so I readded "none" for now. I think this change could be done separately.

cyriltovena

LGTM. I guess you tested this with our dashboard to make sure they still work ?

Really glad we don't have to do all those hacky moves anymore and that most of the issues are fixed.

I guess this buy us some time before we have to move to OpenTelemetry. 😁

cyriltovena · 2020-04-11T12:23:51Z

cyriltovena · 2020-04-11T12:27:30Z

The only one that does not work is the go-client cache. But I assume I did not need cache for this try out ? Code looks unchanged and the metrics comes from k8s client.

aLekSer · 2020-04-12T09:07:06Z

Thanks @cyriltovena so much for providing screenshots, I need to fix stackdriver error message which is left for this PR.

2020/04/03 16:11:56 Failed to export to Stackdriver: rpc error: code = InvalidArgument desc = Field timeSeries[17].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry.

This should fix Prometheus error messages on exporting with empty tags list in Allocator Metrics.

agones-bot · 2020-04-12T14:21:39Z

Build Succeeded 👏

Build Id: 8101da27-e52d-4d2d-9fc4-7482c1ff117a

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.5.0-e4c126a
image: gcr.io/agones-images/agones-sdk:1.5.0-e4c126a
image: gcr.io/agones-images/agones-ping:1.5.0-e4c126a
Linux C++ SDK (build): agonessdk-1.5.0-e4c126a-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.5.0-e4c126a.zip

A preview of the website (the last 30 builds are retained):

https://e4c126a-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1446/head:pr_1446 && git checkout pr_1446
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.5.0-e4c126a

Fix tests.

aLekSer · 2020-04-12T14:57:09Z

With "none" in tag values export to Stackdriver still is not working.
Example of Stackdriver output for version 1.4.0 as a reference point:

agones-bot · 2020-04-12T15:09:48Z

Build Succeeded 👏

Build Id: ae78ffcf-f440-4cc4-82d9-f9d130542c0f

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.5.0-5147a06
image: gcr.io/agones-images/agones-sdk:1.5.0-5147a06
image: gcr.io/agones-images/agones-ping:1.5.0-5147a06
Linux C++ SDK (build): agonessdk-1.5.0-5147a06-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.5.0-5147a06.zip

A preview of the website (the last 30 builds are retained):

https://5147a06-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1446/head:pr_1446 && git checkout pr_1446
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.5.0-5147a06

aLekSer · 2020-04-13T08:04:37Z

Debugged Opencensus Stackdriver exporter code a bit, added more logging:

E 2020-04-13T07:28:37.571709353Z 2020/04/13 07:28:37 Failed to export this to Stackdriver 0: gameservers_count
E 2020-04-13T07:28:37.571744133Z 2020/04/13 07:28:37 Failed to export this to Stackdriver 1: k8s_client_workqueue_unfinished_work_seconds
E 2020-04-13T07:28:37.571749105Z 2020/04/13 07:28:37 Failed to export this to Stackdriver 2: fleets_replicas_count
E 2020-04-13T07:28:37.571752888Z 2020/04/13 07:28:37 Failed to export this to Stackdriver 3: k8s_client_workqueue_work_duration_seconds
E 2020-04-13T07:28:37.571796974Z 2020/04/13 07:28:37 Failed to export to Stackdriver: rpc error: code = InvalidArgument desc = Field timeSeries[8].points[0].distributionValue had an invalid value: Distribution |explicit_buckets.bounds| does not have at least one entry.

Stackdriver export is not working on the most recent stackdriver exporter also.

However the error with Stackdriver metrics is gone only when I comment out whole file: https://github.com/googleforgames/agones/blob/master/pkg/metrics/kubernetes_client.go

Need to understand which metric or group of them lead to this situation, because error on exporter like timeSeries[8] does not help to find erroneous metric.

aLekSer · 2020-04-13T08:48:20Z

Narrowing down - k8s.io/client-go/util/workqueue metrics are failing with Stackdriver exporter workqueue.SetProvider(c).

aLekSer · 2020-04-13T10:18:09Z

Was able to fix Stackdriver exporter uploading if parameter is changed from view.Distribution(0) or view.Distribution() (both cause an error like above) to something like:

	runtime.Must(view.Register(&view.View{
		Name:        "k8s_client_workqueue_latency_seconds",
		Measure:     workQueueLatencyStats,
		Description: "How long an item stays in the work queue.",
		Aggregation: view.Distribution(0.0001),
		TagKeys:     []tag.Key{keyQueueName},
	}))

Going to put this in a separate PR.

Here is a good example how metric above got registered, it uses aggregation with number of buckets and not view.Distribution(0) (with 2 buckets):

return measureView(wp.Latency, view.Distribution(BucketsNBy10(1e-08, 10)...))

https://github.com/knative/pkg/blob/4e57475bc87c1aba3e7fcc0207b593dc7186eb83/metrics/workqueue.go#L78

markmandel · 2020-04-15T20:50:09Z

How is this PR looking? I think it's good to merge, but wanted to check first.

aLekSer · 2020-04-16T08:47:27Z

This PR will fix errors in Prometheus exporter, and I will produce a separate PR for fixing Stackdriver exporter view.Distribution(..) - with multiple parameters. (Even adding 0.0001 instead of 0 is fixing Stackdriver)

markmandel

This PR will fix errors in Prometheus exporter, and I will produce a separate PR for fixing Stackdriver exporter view.Distribution(..) - with multiple parameters. (Even adding 0.0001 instead of 0 is fixing Stackdriver)

SG! Approving now!

google-oss-robot · 2020-04-16T19:55:25Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aLekSer, cyriltovena, markmandel

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [cyriltovena,markmandel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

google-oss-robot · 2020-04-16T19:55:44Z

New changes are detected. LGTM label has been removed.

agones-bot · 2020-04-16T20:13:05Z

Build Succeeded 👏

Build Id: b5bdda16-c908-423d-bfff-08ad5dbda2be

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.6.0-1a98bf6
image: gcr.io/agones-images/agones-sdk:1.6.0-1a98bf6
image: gcr.io/agones-images/agones-ping:1.6.0-1a98bf6
Linux C++ SDK (build): agonessdk-1.6.0-1a98bf6-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.6.0-1a98bf6.zip

A preview of the website (the last 30 builds are retained):

https://1a98bf6-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1446/head:pr_1446 && git checkout pr_1446
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-1a98bf6

agones-bot · 2020-04-16T20:33:47Z

Build Succeeded 👏

Build Id: d2b54fcf-3a46-4f08-b89d-7963dbdfb256

The following development artifacts have been built, and will exist for the next 30 days:

image: gcr.io/agones-images/agones-controller:1.6.0-b06d6d2
image: gcr.io/agones-images/agones-sdk:1.6.0-b06d6d2
image: gcr.io/agones-images/agones-ping:1.6.0-b06d6d2
Linux C++ SDK (build): agonessdk-1.6.0-b06d6d2-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.6.0-b06d6d2.zip

A preview of the website (the last 30 builds are retained):

https://b06d6d2-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/GoogleCloudPlatform/agones.git pull/1446/head:pr_1446 && git checkout pr_1446
helm install ./install/helm/agones --namespace agones-system --name agones --set agones.image.tag=1.6.0-b06d6d2

* Update to OpenCensus version 0.22.3 This should fix Prometheus error messages on exporting with empty tags list in Allocator Metrics. * Updates to the Metrics Fix tests. Co-authored-by: Mark Mandel <markmandel@google.com>

google-oss-robot added the do-not-merge/work-in-progress label Apr 2, 2020

google-oss-robot requested review from dzlier-gcp and roberthbailey April 2, 2020 16:26

google-oss-robot added the size/XXL label Apr 2, 2020

roberthbailey removed request for roberthbailey and dzlier-gcp April 2, 2020 17:42

aLekSer force-pushed the metrics/opencensus-update branch from c59a19a to cf1e86b Compare April 3, 2020 09:17

aLekSer force-pushed the metrics/opencensus-update branch from 7ad5556 to 82fd75e Compare April 3, 2020 09:28

aLekSer force-pushed the metrics/opencensus-update branch from 82fd75e to 88fddd5 Compare April 3, 2020 10:02

aLekSer marked this pull request as ready for review April 3, 2020 10:02

google-oss-robot removed the do-not-merge/work-in-progress label Apr 3, 2020

aLekSer added the do-not-merge/hold label Apr 3, 2020

markmandel added feature-freeze-do-not-merge Only eligible to be merged once we are out of feature freeze (next full release) area/operations Installation, updating, metrics etc kind/cleanup Refactoring code, fixing up documentation, etc labels Apr 7, 2020

cyriltovena reviewed Apr 9, 2020

View reviewed changes

cyriltovena approved these changes Apr 9, 2020

View reviewed changes

google-oss-robot removed the lgtm label Apr 9, 2020

Update to OpenCensus version 0.22.3

f604b3f

This should fix Prometheus error messages on exporting with empty tags list in Allocator Metrics.

aLekSer force-pushed the metrics/opencensus-update branch from 88fddd5 to e4c126a Compare April 12, 2020 14:04

Updates to the Metrics

5147a06

Fix tests.

aLekSer force-pushed the metrics/opencensus-update branch from e4c126a to 5147a06 Compare April 12, 2020 14:52

aLekSer removed the do-not-merge/hold label Apr 12, 2020

markmandel removed the feature-freeze-do-not-merge Only eligible to be merged once we are out of feature freeze (next full release) label Apr 14, 2020

markmandel approved these changes Apr 16, 2020

View reviewed changes

google-oss-robot added the lgtm label Apr 16, 2020

Merge branch 'master' into metrics/opencensus-update

1a98bf6

google-oss-robot removed the lgtm label Apr 16, 2020

Merge branch 'master' into metrics/opencensus-update

b06d6d2

markmandel merged commit 3d22c68 into googleforgames:master Apr 16, 2020

markmandel added this to the 1.6.0 milestone Apr 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to OpenCensus version 0.22.3 #1446

Update to OpenCensus version 0.22.3 #1446

aLekSer commented Apr 2, 2020 •

edited

Loading

agones-bot commented Apr 2, 2020

agones-bot commented Apr 3, 2020

agones-bot commented Apr 3, 2020

agones-bot commented Apr 3, 2020

agones-bot commented Apr 3, 2020

aLekSer commented Apr 3, 2020

markmandel commented Apr 3, 2020

aLekSer commented Apr 3, 2020

aLekSer commented Apr 3, 2020 •

edited

Loading

roberthbailey commented Apr 3, 2020

aLekSer commented Apr 3, 2020 •

edited

Loading

markmandel commented Apr 9, 2020

cyriltovena commented Apr 9, 2020

cyriltovena Apr 9, 2020

aLekSer Apr 9, 2020

aLekSer Apr 12, 2020

cyriltovena left a comment

cyriltovena commented Apr 11, 2020

cyriltovena commented Apr 11, 2020 •

edited

Loading

aLekSer commented Apr 12, 2020

agones-bot commented Apr 12, 2020

aLekSer commented Apr 12, 2020 •

edited

Loading

agones-bot commented Apr 12, 2020

aLekSer commented Apr 13, 2020 •

edited

Loading

aLekSer commented Apr 13, 2020

aLekSer commented Apr 13, 2020 •

edited

Loading

markmandel commented Apr 15, 2020

aLekSer commented Apr 16, 2020

markmandel left a comment

google-oss-robot commented Apr 16, 2020

google-oss-robot commented Apr 16, 2020

agones-bot commented Apr 16, 2020

agones-bot commented Apr 16, 2020

Update to OpenCensus version 0.22.3 #1446

Update to OpenCensus version 0.22.3 #1446

Conversation

aLekSer commented Apr 2, 2020 • edited Loading

agones-bot commented Apr 2, 2020

agones-bot commented Apr 3, 2020

agones-bot commented Apr 3, 2020

agones-bot commented Apr 3, 2020

agones-bot commented Apr 3, 2020

aLekSer commented Apr 3, 2020

markmandel commented Apr 3, 2020

aLekSer commented Apr 3, 2020

aLekSer commented Apr 3, 2020 • edited Loading

roberthbailey commented Apr 3, 2020

aLekSer commented Apr 3, 2020 • edited Loading

markmandel commented Apr 9, 2020

cyriltovena commented Apr 9, 2020

cyriltovena Apr 9, 2020

Choose a reason for hiding this comment

aLekSer Apr 9, 2020

Choose a reason for hiding this comment

aLekSer Apr 12, 2020

Choose a reason for hiding this comment

cyriltovena left a comment

Choose a reason for hiding this comment

cyriltovena commented Apr 11, 2020

cyriltovena commented Apr 11, 2020 • edited Loading

aLekSer commented Apr 12, 2020

agones-bot commented Apr 12, 2020

aLekSer commented Apr 12, 2020 • edited Loading

agones-bot commented Apr 12, 2020

aLekSer commented Apr 13, 2020 • edited Loading

aLekSer commented Apr 13, 2020

aLekSer commented Apr 13, 2020 • edited Loading

markmandel commented Apr 15, 2020

aLekSer commented Apr 16, 2020

markmandel left a comment

Choose a reason for hiding this comment

google-oss-robot commented Apr 16, 2020

google-oss-robot commented Apr 16, 2020

agones-bot commented Apr 16, 2020

agones-bot commented Apr 16, 2020

aLekSer commented Apr 2, 2020 •

edited

Loading

aLekSer commented Apr 3, 2020 •

edited

Loading

aLekSer commented Apr 3, 2020 •

edited

Loading

cyriltovena commented Apr 11, 2020 •

edited

Loading

aLekSer commented Apr 12, 2020 •

edited

Loading

aLekSer commented Apr 13, 2020 •

edited

Loading

aLekSer commented Apr 13, 2020 •

edited

Loading