Per-VU init lifecycle function

[na--]

As mentioned in the comments of #784, there isn't a very good way for users to implement per-VU initialization, since we don't support making HTTP and websocket calls in the init script phase (for good reasons, also mentioned in that issue).

The two current workarounds are:

1. Using the __ITER execution context variable and initializing things in the first iteration of each VU. The problem with this is that the longer first iteration could then potentially skew the iteration_duration metric by quite a lot and it would also eat up the script execution duration for initialization purposes.
2. Using setup() to initialize things for all VUs, return the data in an array, and then use the __VU execution context variable so each VU uses its own data. That's a bit awkward, but it won't skew the iteration_duration metric. Another potential drawback is that in the distributed execution context, all requests would be made from a single IP address, which isn't ideal if we want the init code to do login or sign-up or something like that.

One potential solution to this issue is to add a new per-VU lifecycle function called init(setupData) / initVU(setupData) / setupVU(setupData) or something like that. It will be called once in each VU, after the global setup() is done, so it will receive the data that setup() returned. It will also be called before the actual load test (looping execution of the default function) starts, so it won't be counted as part of the script execution duration or stages.

That per-VU function will be executed in the actual VU runtimes, unlike setup() and teardown(), so it can save any state as global variables in the VU for later access by the default function. And having it as a separate function won't interfere with the actual init phase when we get the script options, won't skew the iteration_duration metric for the first iteration and won't take up some of the configured script execution duration. We can even tag any resulting metrics appropriately, like we do with the ones from setup() and teardown() now.

Details to be considered:

• Errors (uncaught exceptions) in one of the per-VU functions should probably abort the whole script execution.
• Since all those init functions have to finish before we can start the test, we should probably have a separate timeout for them, like we have with setupTimeout and teardownTimeout.
• Since we'd execute this new function when initializing the VUs, for hundreds of VUs we probably don't want to have a fully sequential or fully-parallel initialization, so some sort of configurable rate-limiting would be useful. We can probably move and reuse the SlotLimiter we use in http.batch() requests or use something like it.

[na--]

(this is a copy of the long explanation I wrote in https://community.k6.io/t/setup-and-teardown-running-by-virtual-user/720/3, since the workaround possible in the new k6 v0.27.0 will probably be useful to people who are waiting on this issue)

In a lot of cases, you can easily log in and log out (or create and destroy) with multiple user accounts in setup() and teardown(). This can be done by just having a higher setupTimeout / teardownTimeout values and/or making the HTTP requests with http.batch() and returning the resulting credentials in an array. Then each VUs can just pick its own credentials from that array by using the __VU execution context variable and some modulo arithmetic, somewhat like this:

export default function(setupData) {
  let myCredentials = setupData[__VU % options.vus];
  // ...
}

That said, the recently released k6 v0.27.0 also adds another option - the per-vu-iterations executor. It, and fact that you can have multiple sequential scenarios, combined with the property that k6 will reuse VUs between non-overlapping scenarios, means that you can make something like a per-VU initialization function! 🎉 This can be done by just having a scenario that does 1 iteration per VU and persisting any data or credentials in the global scope. Here's a very complicated example that demonstrates this workaround, and it even includes thresholds to make sure that the per-VU setup and teardown methods are executed correctly:

import http from 'k6/http';
import { sleep, check } from 'k6';
import { Counter } from 'k6/metrics';
import { uuidv4 } from 'https://jslib.k6.io/k6-utils/1.0.0/index.js';

let VUsCount = __ENV.VUS ? __ENV.VUS : 5;
const vuInitTimeoutSecs = 5; // adjust this if you have a longer init!
const loadTestDurationSecs = 30;
const loadTestGracefulStopSecs = 5;

let vuSetupsDone = new Counter('vu_setups_done');

export let options = {
    scenarios: {
        // This is the per-VU setup/init equivalent:
        vu_setup: {
            executor: 'per-vu-iterations',
            vus: VUsCount,
            iterations: 1,
            maxDuration: `${vuInitTimeoutSecs}s`,
            gracefulStop: '0s',

            exec: 'vuSetup',
        },

        // You can have any type of executor here, or multiple ones, as long as
        // the number of VUs is the pre-initialized VUsCount above.
        my_api_test: {
            executor: 'constant-arrival-rate',
            startTime: `${vuInitTimeoutSecs}s`, // start only after the init is done
            preAllocatedVUs: VUsCount,

            rate: 5,
            timeUnit: '1s',
            duration: `${loadTestDurationSecs}s`,
            gracefulStop: `${loadTestGracefulStopSecs}s`,

            // Add extra tags to emitted metrics from this scenario. This way
            // our thresholds below can only be for them. We can also filter by
            // the `scenario:my_api_test` tag for that, but setting a custom tag
            // here allows us to set common thresholds for multi-scenario tests.
            tags: { type: 'loadtest' },
            exec: 'apiTest',
        },

        // This is the per-VU teardown/cleanup equivalent:
        vu_teardown: {
            executor: 'per-vu-iterations',
            startTime: `${vuInitTimeoutSecs + loadTestDurationSecs + loadTestGracefulStopSecs}s`,
            vus: VUsCount,
            iterations: 1,
            maxDuration: `${vuInitTimeoutSecs}s`,
            exec: 'vuTeardown',
        },
    },
    thresholds: {
        // Make sure all of the VUs finished their setup successfully, so we can
        // ensure that the load test won't continue with broken VU "setup" data
        'vu_setups_done': [{
            threshold: `count==${VUsCount}`,
            abortOnFail: true,
            delayAbortEval: `${vuInitTimeoutSecs}s`,
        }],
        // Also make sure all of the VU teardown calls finished uninterrupted:
        'iterations{scenario:vu_teardown}': [`count==${VUsCount}`],

        // Ignore HTTP requests from the VU setup or teardown here
        'http_req_duration{type:loadtest}': ['p(99)<300', 'p(99.9)<500', 'max<1000'],
    },
    summaryTrendStats: ['min', 'med', 'avg', 'p(90)', 'p(95)', 'p(99)', 'p(99.9)', 'max'],
};

let vuCrocName = uuidv4();
let httpReqParams = { headers: {} }; // token is set in init()

export function vuSetup() {
    vuSetupsDone.add(0); // workaround for https://github.com/loadimpact/k6/issues/1346

    let user = `croco${vuCrocName}`
    let pass = `pass${__VU}`

    let res = http.post('https://test-api.k6.io/user/register/', {
        first_name: 'Crocodile',
        last_name: vuCrocName,
        username: user,
        password: pass,
    });
    check(res, { 'Created user': (r) => r.status === 201 });

    // Add some bogus wait time to see how VU setup "timeouts" are handled, and
    // how these requests are not included in the http_req_duration threshold.
    let randDelay = Math.floor(Math.random() * 4)
    http.get(`https://httpbin.test.k6.io/delay/${randDelay}`);

    let loginRes = http.post(`https://test-api.k6.io/auth/token/login/`, {
        username: user,
        password: pass
    });

    let vuAuthToken = loginRes.json('access');
    if (check(vuAuthToken, { 'Logged in user': (t) => t !== '' })) {
        console.log(`VU ${__VU} was logged in with username ${user} and token ${vuAuthToken}`);

        // Set the data back in the global VU context:
        httpReqParams.headers['Authorization'] = `Bearer ${vuAuthToken}`;
        vuSetupsDone.add(1);
    }
}


export function apiTest() {
    const url = 'https://test-api.k6.io/my/crocodiles/';
    const payload = {
        name: `Name ${uuidv4()}`,
        sex: 'M',
        date_of_birth: '2001-01-01',
    };

    let newCrocResp = http.post(url, payload, httpReqParams);
    if (check(newCrocResp, { 'Croc created correctly': (r) => r.status === 201 })) {
        console.log(`[${__VU}] Created a new croc with id ${newCrocResp.json('id')}`);
    }

    let resp = http.get(url, httpReqParams);
    if (resp.status == 200 && resp.json().length > 3) {
        let data = resp.json();
        if (data.length > 3) {
            let id = data[0].id;
            console.log(`[${__VU}] We have ${data.length} crocs, so deleting the oldest one ${id}`);
            let r = http.del(`${url}/${id}/`, null, httpReqParams);
            check(newCrocResp, { 'Croc deleted correctly': (r) => r.status === 201 })
        }
    }
}


export function vuTeardown() {
    console.log(`VU ${__VU} (${vuCrocName}) is tearing itself down...`);

    // In the real world, that will be actual clean up code and fancy error
    // catching like in vuSetup() above. For the demo, you can increase the
    // bogus wait time below to see how VU teardown "timeouts" are handled.
    sleep(Math.random() * 5);

    console.log(`VU ${__VU} (${vuCrocName}) was torn down!`);
}

[dbarrett84]

I've come here from #1638, and found this a nice workaround for one scenario but if I need to setup different data per scenario I'm in a bind.

So I then went back to using one setup function for my scenarios, and creating arrays of what I needed as mentioned above.
Except my setup function times out.

Is there a way to configure setupTimeout when using a setup with scenarios??

[na--]

Is there a way to configure setupTimeout when using a setup with scenarios??

Yes, you just configure options.setupTimeout, as usual. It's a global option, it doesn't matter if you use scenarios or not.

[na--]

Added the evaluation needed tag because of a few reasons:

• given how VUs can be reused between scenarios, it might make more sense to have a per-scenario function that initializes VUs, not a global one... some people have requested that in the forums and in the issue above (Per-VU init lifecycle function #785 (comment))
• the problem with implementing something like per-scenario VU init is that this initialization function will also take some time to run, so we need to take that time into account, i.e. it will cause some of the same issues we have with startAfter (Test suites / execute multiple scripts with k6 #1342 (comment))
• even worse, since we can initialize VUs mid-test with the arrival-rate executors, having a per-VU init() function (either a global one or a per-scenario one) can be very tricky to manage
• we can, of course, ignore all of the complexity and just silently use up the scenarios' configured duration for the VU init, but that should be an explicit decision to make the tradeoff... and we probably need to improve the progressbars so there's more visibility that VUs are initializing

In any case, #1320 is probably more important, since it will partially solve this issue by giving us the following workaround:

if (getVuIterationInCurrentScenario() == 1) {
  // ... initialize VU for this scenario ...
}

Of course, that has the cost of likely making the iteration_duration metric unusable, but it's still better than nothing when you have multiple scenarios and the workaround above doesn't work.

[badeball]

From #785 (comment):

Then each VUs can just pick its own credentials from that array by using the __VU execution context variable and some modulo arithmetic, somewhat like this:

I like this approah / workaround. Is there a way to know, in the setup() method, how many VU's are going to be invoked?

[mstoykov]

Hi @badeball,

I would recommend using k6/execution instead of __VU as that will (more correctly) work in distributed/cloud run as well. For example using execution.vu.idInTest.

Is( there a way to know, in the setup() method, how many VU's are going to be invoked?

This is what is configured in the options. Since v0.38.0 as you also can have the final options from k6/execution and calculate from there, although that will likely be a bit involved.

But again in most cases you will have configured it so it shouldn't be a thing you can't have inside your script easily.

p.s. Fairly hackish way that works for not distributed/cloud test will be to use execution.instance.vusInitialized in the setup

[rgordill]

Hi.

I get to this point because I want to make a big number of concurrent persistent connections, but not all at the same time to avoid saturating the server and the network stacks.

I was thinking about using the per-vu-iterations , but I guess the best choice will be to use the VU Id to define an sleep interval previous to the logic in the setup.

Anyway, a per-VU setup with ramp-up function will be more than desired for this kind of use cases.

[wuhkuh]

(this is a copy of the long explanation I wrote in https://community.k6.io/t/setup-and-teardown-running-by-virtual-user/720/3, since the workaround possible in the new k6 v0.27.0 will probably be useful to people who are waiting on this issue)

(...)

This workaround does not seem to work at the moment of writing.

[joanlopez]

This workaround does not seem to work at the moment of writing.

I literally copy-pasted the script and it worked.

Sometimes it fails because the default vuInitTimeoutSecs in the example (5s) isn't enough, probably because there are multiple HTTP requests involved, including the randDelay, plus the thresholds might be crossed, but the strategy and the workaround as such remains valid.

Could you bring more details of what did fail for you, please?

Thanks! 🙇🏻