Grafana Agent Crow #773

rfratto · 2021-07-22T22:53:34Z

PR Description

Croooow!

This PR creates a metrics validation tool called Grafana Agent Crow.

This is similar to Cortex's text-exporter, which wasn't working well for me. text-exporter assumes that the process is always being scraped from process boot time, which can cause false positive errors when rolling out new agents.

Crow instead will queue samples for validation at scrape time, querying against Prometheus looking for those samples a configurable number of times before marking them as failed. Crow is not a load tester. It isn't designed to generate a significant amount of series. It is only used to make sure that:

Scraped samples get sent.
Scraped samples get sent with the proper value.

This will be used alongside loki-canary and tempo-vulture in an automated smoke testing environment.

Which issue(s) this PR fixes

Notes to the Reviewer

I'm opening this in draft to get some early feedback on the approach.

PR Checklist

CHANGELOG updated
Documentation added
Tests updated

pkg/crow/samples.go

mapno

I like the overall approach. Since generating testing metrics depends on crow being scraped, I think it would be good to alert on not being scraped.

pkg/crow/crow.go

mapno · 2021-07-23T08:36:16Z

pkg/crow/crow.go

+
+	c := &Crow{
+		cfg:        cfg,
+		m:          newMetrics(),


I may have missed it, but are these metrics registered at any point?

Nope, I'm trying a different pattern here. Instead of registering metrics directly, Crow exposes its two collectors (internal state & test sample generator) via a method. This means the caller has to register Crow's metrics, but it allows the caller to arbitrarily register and unregister the metrics at runtime.

rfratto · 2021-08-12T17:46:10Z

Makefile

-	CGO_ENABLED=1 go test $(CGO_FLAGS) -race -cover -coverprofile=cover.out -p=4 ./... -- -online
-	CGO_ENABLED=1 go test $(CGO_FLAGS) -cover -coverprofile=cover-norace.out -p=4 ./pkg/integrations/node_exporter ./pkg/logs -- -online


Turns out flags after -- aren't read and you can't omit the -- otherwise -online will be passed to every package, even packages that don't have them defined. Build tags are the best choice here.

rfratto · 2021-08-12T17:47:32Z

example/docker-compose/agent/config/agent.yaml

-      labels:
-        - svc
-        - status


The labels field was removed at some point.

rfratto · 2021-08-12T17:53:23Z

This is ready for review. I let it run for 10 minutes in docker-compose and there's a 100% success rate:

pkg/crow/samples.go

mattdurham

LGTM once docker hub is ready.

flokli · 2021-08-19T15:47:57Z

pkg/operator/selector_eventhandler_test.go

@@ -1,12 +1,11 @@
 // These tests depend on test assets from controller-runtime which don't work on Windows.

-// +build !windows
+// +build !windows,has_network


This seems to execute despite me not setting has_network during a build…

This can be reproduced by running nix-build -A grafana-agent from the root of a checkout of https://github.com/flokli/nixpkgs, grafana-agent-remove-patch branch.

🤔 That's strange, I don't see the same behavior:

$ go test -v ./pkg/operator ? github.com/grafana/agent/pkg/operator [no test files] $ go test -tags=has_network -v ./pkg/operator === RUN TestEnqueueRequestForSelector # ...

What version of Go are you using?

(We had to change this because of this comment: turns out I was wrong about how the flags passing worked and the build tags are the only way to do this sanely, otherwise we'd need to declare the network flag in every package that has tests defined)

That's go 1.16.7. If you grok at https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/go-modules/generic/default.nix#L168, that's how we invoke the build.

Ah, thanks, I was struggling to find that.

I see what the problem is now: v0.18.2 is a cherry-picked release containing neither your flag change or this PR: https://github.com/grafana/agent/blob/v0.18.2/pkg/operator/selector_eventhandler_test.go.

We've been having to do a few cherry-pick releases since main contains unfinished work that we can't release yet. We thinks it's confusing (leads to situations like this) and also it's a pain to manage. We're currently working on unblocking main and coming up with a better defined branching and release strategy to avoid getting in this situation again. The current plan is to have a full release of main prior to the 10th of September.

Hm, I can reproduce the commands above do the same. It might be something specific to our test phase - we might not honor tags there properly. I'll check back on this, sorry for the noise.

Well, apparently this landed in main after the v0.18.2 tag, my checkout was a recent main, but the nixpkgs was pointing at v0.18.2, so I can debug this all day 🤦

Okay, I can confirm this works just fine on current main. Sorry for the noise.

No worries, sorry for the confusion with v0.18.2!

* basic working example * increase default sample backoff to 1.25s * increase check frequency * tests for ready * review feedback * finish initial implementation * sign drone * fix lint issues * Update pkg/crow/samples.go

rfratto force-pushed the crow branch from 6f26a11 to 5bba888 Compare July 22, 2021 22:55

rfratto commented Jul 22, 2021

View reviewed changes

pkg/crow/samples.go Outdated Show resolved Hide resolved

mapno reviewed Jul 23, 2021

View reviewed changes

rfratto added 5 commits August 12, 2021 11:36

basic working example

a56c980

increase default sample backoff to 1.25s

0332dd2

increase check frequency

e1d03cf

tests for ready

45253d8

review feedback

468873e

rfratto force-pushed the crow branch from f011abd to 468873e Compare August 12, 2021 15:37

finish initial implementation

3590cfa

rfratto force-pushed the crow branch from d8d896f to 3590cfa Compare August 12, 2021 17:46

rfratto commented Aug 12, 2021

View reviewed changes

rfratto marked this pull request as ready for review August 12, 2021 17:48

rfratto requested a review from mattdurham August 12, 2021 17:48

rfratto changed the title ~~Draft: Grafana Agent Crow~~ Grafana Agent Crow Aug 12, 2021

rfratto added 2 commits August 12, 2021 14:40

sign drone

cd6142e

fix lint issues

309231b

rfratto commented Aug 12, 2021

View reviewed changes

pkg/crow/samples.go Outdated Show resolved Hide resolved

Update pkg/crow/samples.go

7f593f2

mattdurham approved these changes Aug 12, 2021

View reviewed changes

rfratto merged commit cbde019 into grafana:main Aug 12, 2021

rfratto deleted the crow branch August 12, 2021 23:22

This was referenced Aug 12, 2021

Drone: merges to main not tagging images properly #823

Closed

Automated smoke tests for metrics #825

Merged

flokli reviewed Aug 19, 2021

View reviewed changes

This was referenced Aug 19, 2021

TestEnqueueRequestForSelector: only run test if -online flag is set #776

Merged

Integration test fails inside Nix sandbox #333

Closed

mattdurham mentioned this pull request Sep 7, 2021

crow doc rfratto/agent#8

Closed

3 tasks

github-actions bot added the frozen-due-to-age Locked due to a period of inactivity. Please open new issues or PRs if more discussion is needed. label Apr 9, 2024

github-actions bot locked as resolved and limited conversation to collaborators Apr 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Grafana Agent Crow #773

Grafana Agent Crow #773

rfratto commented Jul 22, 2021

mapno left a comment

mapno Jul 23, 2021

rfratto Jul 23, 2021

rfratto Aug 12, 2021

rfratto Aug 12, 2021

rfratto commented Aug 12, 2021

mattdurham left a comment

flokli Aug 19, 2021

flokli Aug 19, 2021 •

edited

Loading

rfratto Aug 19, 2021

rfratto Aug 19, 2021 •

edited

Loading

flokli Aug 19, 2021

rfratto Aug 19, 2021

flokli Aug 19, 2021

flokli Aug 19, 2021 •

edited

Loading

flokli Aug 19, 2021

rfratto Aug 19, 2021

		CGO_ENABLED=1 go test $(CGO_FLAGS) -race -cover -coverprofile=cover.out -p=4 ./... -- -online
		CGO_ENABLED=1 go test $(CGO_FLAGS) -cover -coverprofile=cover-norace.out -p=4 ./pkg/integrations/node_exporter ./pkg/logs -- -online

Grafana Agent Crow #773

Grafana Agent Crow #773

Conversation

rfratto commented Jul 22, 2021

PR Description

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

mapno left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rfratto commented Aug 12, 2021

mattdurham left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flokli Aug 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rfratto Aug 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flokli Aug 19, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

flokli Aug 19, 2021 •

edited

Loading

rfratto Aug 19, 2021 •

edited

Loading

flokli Aug 19, 2021 •

edited

Loading