Support for custom xk6 executor extensions

[na--]

As mentioned in #2252 (comment), it should be relatively easy to add xk6 extension support for custom executor types. There might be valid use cases that requires custom executors which will be too specific or problematic to merge into the core of k6. Or executors we might want to merge in the future, but would like to test and iterate on as extensions for a while before we do. These are some issues I remember that might benefit from this, but there might have also been others: #645, #2107

This should also be fairly simple to implement. The ExecutorConfig and Executor interfaces are reasonably well defined, we have additional single-method types for extra executor capabilities, and executors already have to register themselves very much like how xk6 extensions must do.

It will probably be best if we clean up the module paths (#1302) before we support executor extensions, and maybe clean up the interfaces a bit. For example, #1499 will probably require some changes to them, so it's probably best done before that. Still, the actual support for xk6 executor extensions should be fairly easy, these things are just prerequisites to minimize the breaking changes afterwards by changing the things we already know have to be changed beforehand.

[imiric]

A probably unpopular opinion: I'm concerned we're too eager to make everything externally extensible. I know we've discussed making logging extensible as well, which makes sense as it's similar to outputs, but executors are something deeply internal to k6, and the implementation of them is much more complex than other types of extensions.

Just because we have well defined interfaces for things and it's relatively easy to do doesn't mean it should qualify to become an extension. There's a lot of direct and indirect overhead of doing so: internal interfaces become public contracts and are difficult to change, we need to add more documentation for how to create them, support authors when they run into issues, make sure xk6 supports all these different use cases, etc.

So I'm slightly against this feature 🙂

[na--]

There's a lot of direct and indirect overhead of doing so: internal interfaces become public contracts and are difficult to change, we need to add more documentation for how to create them, support authors when they run into issues, make sure xk6 supports all these different use cases, etc.

Yeah, somewhat agree on the overhead when it comes to docs and support, but as a counterpoint, we also support the same people when there isn't an extension interface, with probably more overall effort required... 😅 See #2252 from today and all of the output PRs before we supported output extensions. In aggregate, I'd wager that the benefits we get from extensions here far outweigh the drawbacks.

I also disagree about the fact that "internal interfaces become public contracts and are difficult to change". We very explicitly don't offer any stability guarantees on any Go APIs yet. Not to mention that we have a very recent counter-example, we made a big change one major such interface in k6 v0.35.0 just this week...

Finally, I don't expect we'll have executor extensions any time soon, unless there's a substantial and previously unseen interest in the topic from our users. I imagine they would be orders of magnitude less useful to users than JS and output extensions, or even extensions for supporting alternative non-JS runtimes... I've mostly created this issue as a place to collect 👍 and point people to, if necessary. And also as a place to link to the refactoring tasks that are in the way of executor extensions, since those are usually code smells and technical debt we want to fix anyway, even if we don't have extensions.

[imiric]

We very explicitly don't offer any stability guarantees on any Go APIs yet.

Right, but that doesn't avoid incurring additional work for everyone involved, which in a lot of cases is still ourselves for our own extensions. And the hesitation of deciding to change the interfaces unless absolutely necessary.

Not to mention that we have a very recent counter-example, we made a big change one major such interface in k6 v0.35.0 just this week...

Yeah, and I don't think that was a good experience. 😄 We postponed it until it became clear we needed to do it, instead of being free to iterate on it internally. And it still adds support overhead for every extension.

But sure, let's leave this open and see what interest there is.

[mstoykov]

I kind of feel like we can close this issue https://github.com/mstoykov/xk6-executor-example ;)

[mstoykov]

While the workaround still works, there has been very little interest in this over the 3 years this has been opened.

I am going to close it as "it works, but probably not a thing we want to say we support."