Refactor group and check to not be tightly coupled #3750

mstoykov · 2024-05-21T15:47:22Z

What?

Group and check were tightly coupled between them and also to the end of test summary and the REST API.

This change the tight coupling is removed. Group and check no longer interact with each other while being run.

In order to keep the same end of test summary and REST API behavior the code makes the same aggregation it used to be but only based on the metrics emitted by check and group.

There are several breaking changes still in it as a bunch of things are no longer useful, and technically not implementable if we want to remove the tight coupling.

The only one that is relevant though is that
lib.State no longer has Group.

There are 2 extensions using that and both of them use it to use the "magic" that tight group and check together.

Why?

The previous code apart from having problems code smell wise, meant that you can't implement this functionality in JS.

Which also means that they act very strangely especially as if you do not look closely it just seems like they set a few tags and emit metrics.

While working on #2728 it became apparent that it is unlikely we will get a perfect solution and one of the big problems is that you can't even experiment without having to write a fairly complicated async go code.

So this also makes that issue a bit more actionable. And definitely a thing that now people can have not perfect or even 80%, but still workable for them solutions.

Checklist

I have performed a self-review of my code.
I have added tests for my changes.
I have run linter locally (make lint) and all checks pass.
I have run tests locally (make tests) and all tests pass.
I have commented on my code, particularly in hard-to-understand areas.

Related PR(s)/Issue(s)

#2869

Group and check were tightly coupled between them and also to the end of test summary and the REST API. This change the tight coupling is removed. Group and check no longer interact with each other while being run. In order to keep the same end of test summary and REST API behavior the code makes the same aggregation it used to be but *only* based on the metrics emitted by `check` and `group`. There are several breaking changes still in it as a bunch of things are no longer useful, and technically not implementable if we want to remove the tight coupling. The only one that is relevant though is that `lib.State` no longer has `Group`. There are 2 extensions using that and both of them use it to use the "magic" that tight `group` and `check` together. Fixes #2869

mstoykov · 2024-05-21T15:48:46Z

lib/test_state.go

+
+// GroupSummaryDescription is the description of the GroupSummary used to identify and ignore it
+// for the purposes of the cli descriptions.
+const GroupSummaryDescription = "Internal Group Summary output"
+
+// GroupSummary is an internal output implementation that facilitates the aggregation of
+// group and check metrics for the purposes of the end of test summary and the REST API
+type GroupSummary struct {
+	group   *Group // TODO(@mstoykov): move the whole type outside of lib later
+	ch      chan []metrics.SampleContainer
+	stopped chan struct{}
+	logger  logrus.FieldLogger
+}
+
+// NewGroupSummary returns new GroupSummary ready to be started.


I kind of wanted to move the whole Group, Check and GroupSummary somewhere else, but then I decided that is quite a big change and will make the review even harder.

It also turned out to not be easy.

So if we have any good places to move them I would prefer to do them in a separate PR.

Ideally, I would use this opportunity to move them to startup an internal package and move GroupSummary there. This is my unique request, mostly to reflect what the comment is mentioning.

I am not against this - but that will create a much bigger change that will make following what happens ever harder. So I will prefer to keep it in a separate PR.

Not certain internal is needed as well so 🤷 on that particular part. But it is probably a good usage as well.

Yeah, especially with the v1.0 on the road, would be great to start considering hiding internal details behind an internal package, to reduce the public surface ( 👋🏻 say hello to Hyrum's Law).

That said, I fully agree on doing so on a separate PR, to avoid making this even harder to read, follow and review.

mstoykov · 2024-05-21T15:52:04Z

lib/vu_state.go

-	// Current group; all emitted metrics are tagged with this.
-	Group *Group
-


This will break 2 extensions:

xk6-fasthttp/checks.go 41: check, err := state.Group.Check("check status is " + strconv.FormatInt(int64(wantStatus), 10)) xk6-g0/g0/addon/assert.go 67: check, err := state.Group.Check(name)

I don't really think that keeping this compatible is possible, and even more importantly warranted.

As you can see both extensions basically hit exactly the same issue with this being the way to create a new check instead of just emitting the metrics.

codecov-commenter · 2024-05-21T15:58:39Z

Codecov Report

Attention: Patch coverage is 83.72093% with 14 lines in your changes are missing coverage. Please review.

Project coverage is 70.83%. Comparing base (153e0e5) to head (f408082).
Report is 3 commits behind head on master.

❗ Current head f408082 differs from pull request most recent head ba596ca

Please upload reports for the commit ba596ca to get more accurate results.

Files	Patch %	Lines
lib/test_state.go	73.91%	6 Missing and 6 partials ⚠️
lib/models.go	80.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3750      +/-   ##
==========================================
- Coverage   70.85%   70.83%   -0.02%     
==========================================
  Files         289      290       +1     
  Lines       21159    21191      +32     
==========================================
+ Hits        14992    15011      +19     
- Misses       5210     5215       +5     
- Partials      957      965       +8

Flag	Coverage Δ
ubuntu	`70.75% <83.72%> (-0.05%)`	⬇️
windows	`70.68% <83.72%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

lib/test_state.go

mstoykov · 2024-05-23T08:58:03Z

lib/test_state.go

+
+		if sample.Metric.Name != "checks" {


Suggested change

if sample.Metric.Name != "checks" {

// In case of checks we need to also generate the check and add it values. Otherwise we are done.

if sample.Metric.Name != "checks" {

api/v1/group_routes_test.go

joanlopez · 2024-05-23T12:33:12Z

js/modules/k6/k6_test.go

 		groupTag, ok := state.Tags.GetCurrentValues().Tags.Get("group")
 		require.True(t, ok)
-		assert.Equal(t, groupTag, root.Name)
+		assert.Equal(t, groupTag, "")


Suggested change

assert.Equal(t, groupTag, "")

assert.Equal(t, groupTag, lib.RootGroupPath)

I actually purposefully left it here not using the constant to catch it if ever changes - I think there is one more place where this is done in the tests as well.

I am not particularly attached to this, but as changing this constant will be a breaking change I kind of prefer if changing it breaks a few tests for sure.

Yes, you're right, sometimes you used "", sometimes the constant.

but as changing this constant will be a breaking change I kind of prefer if changing it breaks a few tests for sure

I fully agree with end goal, it was me just probably confused about where and for which tests it makes more sense to use the constant and for which not. Let's keep it as-is.

That said, as I had previous negative experiences with this kind of tradeoff: "de-coupling" the test from the constant, to notice any change, would make sense to also leave a comment together with the constant, stating that explicitly, for the ourselves of the future, who might forget/omit it by just updating the tests anyway?

joanlopez

Looks good despite of the minor comments/suggestions!

Co-authored-by: Joan López de la Franca Beltran <5459617+joanlopez@users.noreply.github.com>

joanlopez

Approved! ✅ Although I guess we need a to reach a consensus around the breaking changes to move forward. Nevertheless, I don't see many alternatives at this point.

oleiade

LGTM 🚀 👏🏻

I concur with what has been said about starting an internal package too, as it's inline with our ongoing plans for v1.

lib/models.go

codebien · 2024-05-28T14:43:06Z

lib/test_state.go

+// GroupSummary is an internal output implementation that facilitates the aggregation of
+// group and check metrics for the purposes of the end of test summary and the REST API
+type GroupSummary struct {
+	group   *Group // TODO(@mstoykov): move the whole type outside of lib later


Suggested change

group *Group // TODO(@mstoykov): move the whole type outside of lib later

group *Group // TODO(@mstoykov): move the whole type outside of lib later

Where do you plan to move it? Under output? It would like to have a suggestion directly in the code so we have shared knowledge on ideas/plans.

codebien · 2024-05-28T14:57:01Z

lib/test_state.go

+
+// AddMetricSamples is part of the output.Output interface
+func (gs *GroupSummary) AddMetricSamples(samples []metrics.SampleContainer) {
+	gs.ch <- samples


Are we sure we can serve the rate expected and we don't block here? Is the buffer size of the channel enough?

See grafana/k6#3750 for more info.

mstoykov added refactor breaking change for PRs that need to be mentioned in the breaking changes section of the release notes labels May 21, 2024

mstoykov added this to the v0.52.0 milestone May 21, 2024

mstoykov requested a review from a team as a code owner May 21, 2024 15:47

mstoykov requested review from oleiade and joanlopez and removed request for a team May 21, 2024 15:47

mstoykov commented May 21, 2024

View reviewed changes

pablochacin reviewed May 22, 2024

View reviewed changes

lib/test_state.go Show resolved Hide resolved

mstoykov commented May 23, 2024

View reviewed changes

joanlopez reviewed May 23, 2024

View reviewed changes

api/v1/group_routes_test.go Outdated Show resolved Hide resolved

joanlopez reviewed May 23, 2024

View reviewed changes

api/v1/group_routes_test.go Outdated Show resolved Hide resolved

joanlopez reviewed May 23, 2024

View reviewed changes

joanlopez previously approved these changes May 23, 2024

View reviewed changes

Apply suggestions from code review

8321b07

Co-authored-by: Joan López de la Franca Beltran <5459617+joanlopez@users.noreply.github.com>

mstoykov dismissed joanlopez’s stale review via 8321b07 May 23, 2024 13:30

joanlopez previously approved these changes May 27, 2024

View reviewed changes

fixup! Apply suggestions from code review

e82c53e

mstoykov dismissed joanlopez’s stale review via e82c53e May 27, 2024 05:49

oleiade previously approved these changes May 27, 2024

View reviewed changes

mstoykov linked an issue May 28, 2024 that may be closed by this pull request

Refactor group() and check() tight coupling with the VU.State #2869

Closed

mstoykov commented May 28, 2024

View reviewed changes

lib/models.go Show resolved Hide resolved

Update lib/models.go

ba596ca

mstoykov dismissed oleiade’s stale review via ba596ca May 28, 2024 14:31

mstoykov requested review from oleiade, joanlopez and codebien May 28, 2024 14:32

joanlopez approved these changes May 28, 2024

View reviewed changes

codebien approved these changes May 28, 2024

View reviewed changes

mstoykov merged commit 39951ab into master May 29, 2024
23 checks passed

mstoykov deleted the lessGroupCheckMagic branch May 29, 2024 08:48

inancgumus mentioned this pull request May 29, 2024

k6-core compatibility: Fix Group and Tag usage grafana/xk6-browser#1357

Merged

3 tasks

This was referenced Jun 7, 2024

Stop setting Group in tests - that has been dropped phymbert/xk6-sse#13

Merged

Drop direct usage of Group.Check domsolutions/xk6-fasthttp#18

Merged

Don't use Group.Check directly szkiba/xk6-g0#9

Merged

Drop direct usage of Group mostafa/xk6-kafka#296

Merged

mostafa pushed a commit to mostafa/xk6-kafka that referenced this pull request Jun 7, 2024

Drop direct usage of Group (#296)

9fa0159

See grafana/k6#3750 for more info.

BrewTestBot mentioned this pull request Jun 25, 2024

k6 0.52.0 Homebrew/homebrew-core#175582

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor group and check to not be tightly coupled #3750

Refactor group and check to not be tightly coupled #3750

mstoykov commented May 21, 2024 •

edited

Loading

mstoykov May 21, 2024

codebien May 21, 2024 •

edited

Loading

mstoykov May 23, 2024

joanlopez May 23, 2024

mstoykov May 21, 2024

codecov-commenter commented May 21, 2024 •

edited

Loading

mstoykov May 23, 2024

joanlopez May 23, 2024

mstoykov May 23, 2024

joanlopez May 24, 2024

joanlopez left a comment •

edited

Loading

joanlopez left a comment

oleiade left a comment

codebien May 28, 2024

codebien May 28, 2024

		// Current group; all emitted metrics are tagged with this.
		Group *Group

	assert.Equal(t, groupTag, "")
	assert.Equal(t, groupTag, lib.RootGroupPath)

	group *Group // TODO(@mstoykov): move the whole type outside of lib later
	group *Group // TODO(@mstoykov): move the whole type outside of lib later

Refactor group and check to not be tightly coupled #3750

Refactor group and check to not be tightly coupled #3750

Conversation

mstoykov commented May 21, 2024 • edited Loading

What?

Why?

Checklist

Related PR(s)/Issue(s)

Choose a reason for hiding this comment

codebien May 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented May 21, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joanlopez left a comment • edited Loading

Choose a reason for hiding this comment

joanlopez left a comment

Choose a reason for hiding this comment

oleiade left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mstoykov commented May 21, 2024 •

edited

Loading

codebien May 21, 2024 •

edited

Loading

codecov-commenter commented May 21, 2024 •

edited

Loading

joanlopez left a comment •

edited

Loading