[Feature Request]: Add stepped-arrival-rate

[LaserPhaser]

Feature Description

In addition to the ramping-arrival-rate
one of the most common features in loadtesting is stepping.
For example when you know that your system must hold up to 1000RPS.
You want to test it step by step in the following manner:
100 rps for 5 minutes
up to 300 rps for 5 minutes
up to 600 rps for 10 minutes
up to 1000 rps for 30 minutes.

Right now it's possible to configure such steps with the following scenarios

cenarios: {
    contacts: {
      executor: 'ramping-arrival-rate',
      startRate: 10,
      timeUnit: '1s',
      preAllocatedVUs: 50,
      maxVUs: 1000,
      stages: [
         {target: 100, duration: '1s' },
        {target: 100, duration: '5m' },
         {target: 300, duration: '1s' },
        {target: 300, duration: '5m' },
         {target: 600, duration: '1s' },
        {target: 600, duration: '10m' },
         {target: 1000, duration: '1s' },
        {target: 1000, duration: '30m' },

      ],
    },

For me right now it's looks like woraround.

Suggested Solution (optional)

My suggestion is to add "stepped-arrival-rate":
Where target will mean not the final target of this stage but the RPS which we should hold during the 'duration'.

With such configuration, the scenarios will looks like

scenarios: {
    contacts: {
      executor: 'stepped-arrival-rate',
      startRate: 10,
      timeUnit: '1s',
      preAllocatedVUs: 50,
      maxVUs: 1000,
      stages: [
        {target: 100, duration: '5m' },
        {target: 300, duration: '5m' },
        {target: 600, duration: '10m' },
        {target: 1000, duration: '30m' },
      ],
    },

Which is more clear to read and understand in my opinion

[LaserPhaser]

Basically I've added "bad" PR just to explain the idea.
The code is almost the same as for constant-arrival-rate but just one block of code (that calculate arrival rate has bee removed).

If the idea will be approved by the dev team, I will prepare clean PR/docs and tests.

[na--]

Hmm, I think I understand what you want to achieve, but I am not sure a completely new executor is the right solution here 🤔 It doesn't significantly change the actual script execution (i.e. how iterations are executed by VUs), it is just a change in how the iteration rate is "ramped" between the stages. Besides, I can see how someone could want a stepped-vus equivalent of this for ramping-vus, that seems like a valid use case.

Right now, the ramping-arrival-rate and ramping-vus executors work like this: the iteration rate or the number of active VUs respectively get linearly increased or decreased from what they were at the start of the stage (i.e. either the startRate/startVUs option of the scenario, for the first stage, or the previous stage's target value) to become equal to the target at the end of the stage's duration. You could have duration: 0 stages btw, for instant ramp-ups, no need for the 1s in your example. But yeah, I can see how these 0-duration stages are a bit annoying...

We already want to change this, since linear progression is not always what people want. In fact, @oleiade is about to open a new issue for supporting custom non-linear interpolation/progression functions in stages, e.g. exponential, ease-in, etc. And I think your use case might be more easily covered by also having an instant progression function that will immediately jump to the target value at the start of the stage. So, something like this:

scenarios: {
    contacts: {
      executor: 'stepped-arrival-rate',
      startRate: 10,
      timeUnit: '1s',
      preAllocatedVUs: 50,
      maxVUs: 1000,
      stages: [
        {target: 100, duration: '5m', progression: 'instant' },
        {target: 300, duration: '5m', progression: 'instant' },
        {target: 600, duration: '10m', progression: 'instant' },
        {target: 1000, duration: '30m', progression: 'instant' },
      ],
    },

There is some repetition here, but that can be easily taken care of by a small helper function to generate the whole stages or even a single stage in the options 🤷‍♂️ . One benefit is that won't need to have a whole separate executor with an almost matching implementation to the ramping-arrival-rate one, and we will be able to make use of the same config for the ramping-vus executor as well. Another benefit will be that you'd be able to mix and match different progression functions, which might be useful in some use cases, e.g.:

stages: [
    {target: 100, duration: '5m', progression: 'logarithmic' }, // gradually ramp up to 100 iters/s
    {target: 300, duration: '5m', progression: 'instant' }, // immediately spike to 300 iters/s
    {target: 600, duration: '10m', progression: 'linear' }, // linearly ramp up to 600
    {target: 1000, duration: '30m', progression: 'instant' }, // another spike to 1000
],

@LaserPhaser, @oleiade, what do you think about this?

[LaserPhaser]

For me it's even better and much more clear than a new executor.
Progression it's somehting that could be extended in future.
If you do not mind, I will try to implement it on WeekEnd )

[na--]

@LaserPhaser, sure, a PR with this will be very welcome! We weren't planning to work on this ourselves for now, we just wanted to open a new issue about it, to see if there was interest in the funcitonality, so I'm happy to see that, indeed, there is 😅 We'll close this issue when we write up the one for a stages progression function. We'll also try to help out if you have any issues with the implementation, there is going to be some complexity here.

Also, thinking a bit more about this, we should probably also open another new issue for gauging interest in adding xk6 extension support for custom executors. Even if this specific case would be better handled another way, there might be other very valid use cases for custom executors that we might not want to (immediately) merge into the k6 core 🤷‍♂️ Since the ExecutorConfig and Executor interfaces are fairly well defined, and since executors already have to register themselves very much like how xk6 extensions must do, it should be fairly simple to implement, requiring at most some minor refactoring and module path cleanups 🤞

[LaserPhaser]

@na-- could you please comment my PR (of cause I will add tests little bit later)
I can do it as extension or as part of K6 itself, please guide me where do the K6 dev team prefer to see it )

[oleiade]

Hi @LaserPhaser,

As @na-- mentioned it, I have now created a dedicated issue referencing this one: #2256. I also took the liberty to reference you #2255 PR. I'm on board with @na-- on this one, and assume we wouldn't need a dedicated executor. Not being an expert of the codebase nor of how scheduling works yet, I'm assuming, there should indeed be a way to control the interpolation between stages by using a dedicated function taking some starting point, end point, and delta duration (from the beginning of the ramping stage).

[LaserPhaser]

@oleiade Thank you!
Sounds good,
So what do you suggest,
To close my current issue and PR, and prepare a new PR for #2256 ?
Or k6 team will handle with it?

[na--]

Your PR should stay open, we'll review it in the next few days and discuss any changes that might be required to merge it in there. We didn't plan to work on this now ourselves, but we'll try and support your effort. However, I'll close this issue in favor of #2256, no point in having two different issues for the same thing.