Fix externally-controlled maxVUs scaling

[imiric]

Closes #1511

I'm not sure this is the right fix, and the test is a bit hacky, but let me know.

[imiric]

@mstoykov We need that errorcheck linter ;)

Also note the change to a 400 response code here, which is what was expected by the test, and will probably make sense in most cases.

[mstoykov]

Was this the actual issue of not scaling down ?

[imiric]

Yes :D initializedVUs wasn't decremented anywhere. Though not sure if this is the right approach or place for it :-/

[na--]

This should only be done when we're not returning the VU back to the buffer pool though, like this:

			if i < rs.startMaxVUs {
				// return the initial planned VUs to the common buffer
				executionState.ReturnVU(rs.vuHandles[i].initVU, false)
			} else {
				executionState.ModInitializedVUsCount(-1)
			}

[na--]

To demonstrate, use the following options:

export let options = {
    scenarios: {
        extc1: {
            executor: "externally-controlled",
            vus: 5,
            maxVUs: 10,
            duration: "10s",
        },
        const_loop_vus: {
            executor: "constant-vus",
            vus: 10,
            duration: "20s",
            startTime: "10s",
            gracefulStop: "0s",
        },
    },
};

see the 00/00 VUs in the description here:

running (14.1s), 00/00 VUs, 139 complete and 15 interrupted iterations
extc1          ✓ [======================================] 05/10 VUs  10s   
const_loop_vus ✗ [======>-------------------------------] 10 VUs     04.1s/20s

[codecov-commenter]

Codecov Report

Merging #1517 into new-schedulers will increase coverage by 0.58%.
The diff coverage is 85.71%.

@@                Coverage Diff                 @@
##           new-schedulers    #1517      +/-   ##
==================================================
+ Coverage           76.64%   77.23%   +0.58%     
==================================================
  Files                 163      163              
  Lines               13044    13046       +2     
==================================================
+ Hits                 9998    10076      +78     
+ Misses               2532     2454      -78     
- Partials              514      516       +2     

Impacted Files	Coverage Δ
api/v1/status_routes.go	55.10% <66.66%> (+44.89%)	⬆️
lib/executor/externally_controlled.go	76.12% <100.00%> (+14.80%)	⬆️
lib/executors.go	94.11% <0.00%> (+1.96%)	⬆️
lib/executor/helpers.go	96.70% <0.00%> (+2.19%)	⬆️
core/local/local.go	76.21% <0.00%> (+2.20%)	⬆️
lib/execution.go	93.89% <0.00%> (+2.29%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a9903e0...d84fe0e. Read the comment docs.

[imiric]

Phew, I think TestExternallyControlledRun should finally be flake-free 🤞 See fc48c46.

I'll see if I can apply this same pattern to other executor tests to stabilize them. We should preferably have something like a MiniExecutionScheduler 😄 that would standardize a stable test structure and maybe allow inspection and expose some test helpers.

[na--]

This should only be done when we're not returning the VU back to the buffer pool though, like this:

			if i < rs.startMaxVUs {
				// return the initial planned VUs to the common buffer
				executionState.ReturnVU(rs.vuHandles[i].initVU, false)
			} else {
				executionState.ModInitializedVUsCount(-1)
			}

[imiric]

@na-- See 64f30fc with that change.

Though there are still some inconsistencies:

1. If two executors are started, for example:

import http from 'k6/http';
import { sleep } from 'k6';

export let options = {
  scenarios: {
    adjustable_at_will: {
      executor: 'externally-controlled',
      vus: 0,
      maxVUs: 10,
      duration: '10m',
    },
    constant_vus: {
      executor: 'constant-vus',
      vus: 20,
      duration: '2m',
    },
  }
};

export default function() {
  sleep(0.2);
}

2. k6 status returns vus: "20", vus-max: "30", taking into account the global counts.

3. Yet trying k6 scale -u 30 will of course error out because of the 10 maxVUs configured for the externally-controlled exec. Similarly with scale -m, etc.

It's somewhat confusing that the API returns the global status, but can only modify externally-controlled parameters. Not sure how big of a problem this would be in practice, though it would be best to avoid any such inconsistencies. Maybe transparently convert any scale values to the externally-controlled config (i.e. -u 30 actually becomes -u 10)? Or do you think just documenting this would be enough?

[mstoykov]

LGTM, but I haven't tried it locally

[imiric]

#1357 strikes again 😞

[na--]

hmm I think we should just document this, and fix it in the v2 REST API, whenever that is implemented...

[imiric]

OK, I'll mention it in the release notes and the new docs somewhere.