Groups in the end-of-test summary are unordered

[na--]

Environment

• k6 version: k6 version above v0.21.1

Expected Behavior

The hierarchical group information at the end of a test should be ordered in some way.

Actual Behavior

As this user in the community forum complained, and I've confirmed, the information for the hierarchical groups k6 emits at the end of the test run doesn't seem sorted in any way.

Testing old k6 version, this seems to have worked in k6 up to version v0.21.1, which ordered groups alphabetically, and way broken by k6 v0.22.0...

Steps to Reproduce the Problem

Run the following script multiple times:

import { group, check } from "k6";
function test(i) { group(`test ${i}`, () => { subTest(i); }); }
function subTest(i) { group(`subtest ${i}`, () => { check(1, { "check": () => true == true }) }); }
export default function () { for (let i = 1; i <= 10; i++) { test(i); } }

[sniku]

I have also noticed that custom metrics have a seemingly random order in the CLI output.
I believe the custom metrics used to be displayed in the order they have been created/executed.

[na--]

@sniku, I'm not sure we ever showed metrics in the order they occurred in the script, but we probably should... I see that it seems like we're sorting them alphabetically now, which is not ideal, but as far as I can see has been the case since at least k6 v0.19.0 and maybe earlier. For example, running the following script with k6 v0.26.0:

import { Counter } from "k6/metrics";
import { group, check, sleep } from "k6";

let counter1 = Counter("bbb");
let counter2 = Counter("ccc");
let counter3 = Counter("aaa");

export let options = {
    vus: 10,
    duration: "2s",
}

const checks = {
    "1) is below 0.5": (v) => v < 0.5,
    "2) is equal to 0.5": (v) => v == 0.5,
    "3) is above 0.5": (v) => v > 0.5,
};

export default function () {
    group("group 1", function () {
        counter1.add(__ITER);
        check(Math.random(), checks);
    });

    group("group 2", function () {
        counter2.add(__ITER);
        check(Math.random(), checks);
    });

    group("group 3", function () {
        counter3.add(__ITER);
        check(Math.random(), checks);
    });
    sleep(0.5);
}

may result in something like this:

    █ group 2

      ✗ 1) is below 0.5
       ↳  55% — ✓ 22 / ✗ 18
      ✗ 2) is equal to 0.5
       ↳  0% — ✓ 0 / ✗ 40
      ✗ 3) is above 0.5
       ↳  45% — ✓ 18 / ✗ 22

    █ group 3

      ✗ 2) is equal to 0.5
       ↳  0% — ✓ 0 / ✗ 40
      ✗ 3) is above 0.5
       ↳  52% — ✓ 21 / ✗ 19
      ✗ 1) is below 0.5
       ↳  47% — ✓ 19 / ✗ 21

    █ group 1

      ✗ 1) is below 0.5
       ↳  40% — ✓ 16 / ✗ 24
      ✗ 2) is equal to 0.5
       ↳  0% — ✓ 0 / ✗ 40
      ✗ 3) is above 0.5
       ↳  60% — ✓ 24 / ✗ 16

    aaa..................: 60     29.99754/s
    bbb..................: 60     29.99754/s
    ccc..................: 60     29.99754/s
    checks...............: 33.33% ✓ 120  ✗ 240 
    data_received........: 0 B    0 B/s
    data_sent............: 0 B    0 B/s
    group_duration.......: avg=52.93µs  min=20.2µs   med=36.83µs  max=375.24µs p(90)=109.49µs p(95)=131.02µs
    iteration_duration...: avg=500.59ms min=500.11ms med=500.47ms max=501.81ms p(90)=501.26ms p(95)=501.36ms
    iterations...........: 30     14.99877/s
    vus..................: 10     min=10 max=10
    vus_max..............: 10     min=10 max=10

🤦‍♂️ ...

I guess the most immediate and biggest reason for the disorderly listing of groups and checks in the summary is that iteration order of maps is not guaranteed in any way in Go, and we save the groups and the checks as string maps: https://github.com/loadimpact/k6/blob/74f11a6b6703282c282c2baf42ccaa8fa3a23d31/lib/models.go#L110-L112

[safebear]

Would it be possible to have the groups ordered in the order they are executed in the script, with the first at the top? I'm not sure alphabetical would be intuitive...

[na--]

@safebear, probably yes. At the moment, to me, displaying the groups in chronological order seems to be the most logical and to offer the best UX. That said, if we sort the group names alphabetically, you can always turn that in a chronological order by just naming your first group 01 First group name, the second 02 whatever and so on...

Ideally, though, I'd like for us to have both options (and maybe more) through the planned templating functionality for the end-of-test summary (#1319 (comment)). It's just a matter of carefully and efficiently storing the groups and checks information in a way that doesn't loose the chronological order, and then providing some simple helper functions in the template that allows for different sort orders... 🤞 😅

[ryl]

I would like to throw in some support for chronological order. This is what I was expecting to see, kind of like the result tree in JMeter. I was surprised when they weren't in order, and had to add some console.log statements to verify things were not occurring in an unexpected order. I think chronological order (order of execution) would make the best default.

[imiric]

@na-- Can we consider using an ordered map for storing the groups and checks? wk8/go-ordered-map seems sensible to import. This would allow us to print the groups in the order they were defined in, as well as alphabetical if needed.

[mstoykov]

I think we can make our own version:

1. Skip on the interface{}, arguably this will make the overall code less if we use a custom one :)
2. This one uses container/list supposedly for faster Delete operations ... which might be true, but we don't need that or half of it's other methods

[na--]

As I mentioned in #1316 (comment), we should first have template support for the end-of-test summary and then tackle this. That way we can order the groups however we want (by time, by name, etc.), as well as make any number of other adjustments that someone might want. It would still likely require that we change the data structure from a map to some sort of an ordered data structure (though a slice seems better than an ordered map), but

[sniku]

Hurray for this feature 🎉