proposal: Prombench Custom Benchmarks

[bwplotka]

Proposes implementation for prometheus/test-infra#659

[krajorama]

Thank you for this. I agree in general, this seems like a good idea. Made some comments.

[ArthurSens]

One thing I liked very much about this proposal is its easy and fast implementation! The lack of custom benchmarks blocks a lot of work, so having something going quickly is nice.

When we discussed this in the past, I had something different in mind inspired by how Prow parses comments to add extra functionality to plugins.

The idea is to improve comment-monitor with new commands:

/prombench main
/extra-args --enable-features=native-histograms,wal-records --web.enable-otlp-receiver
/with avalanche
/with agent-mode

• extra-args is a special command that adds extra command line flags to Prometheus; it can be used to add feature flags or "regular" commands
• with is used for more complex modifications, like replacing/updating the load generator or features that require changing several parts of Prometheus. For example, agent mode requires an extra flag and removing parts of the config file.

This proposal is a lot harder though, not sure how much time that would take 😬

---

The main downside of the current proposal is that maintaining several branches for each kind of benchmarks seems like a lot of work. If we make a change in the main branch that should be added to all others, the number of PRs would get out of control 😬

[bwplotka]

Nice @ArthurSens - one of my alternatives is similar to that, so essentially "prombench options for every variation" - where you have some kind of custom config passed via /prombench or comment in general. I will add your syntax too to this proposal doc.

This proposal is a lot harder though, not sure how much time that would take 😬

It's a good question, changing comment-monitor is easy, applying those options further down (and validating) it's bit harder but I like branching a bit more, because it unlocks almost all customizations vs carefully curating prombench options and debating which ones to support, which one you have (you have to document those) etc.

Also, my proposal and yours are not conflicting. We could implement branching so we can experiment and do more complex variations, but since it's harder to maintain those branches over time, we could implement common "variation" options in the prombench long term (so your more proposal). WDYT?

cc @bboreham @krajorama @jesusvazquez @codesome any preferences on "branching" vs "prombech options" so far?

[bwplotka]

There is also a bit more "maintainable" path where instead of branches we do custom directories on main and you can specify different directory than default one. Then we can even use kustomize to create variations or just copy - bit easier to discover what's possible and maintain 🤔 Will add as a 3rd option.

[krajorama]

Approved with nit comment.

[ArthurSens]

LGTM, we can proceed with the alternative in another proposal if we ever feel like it :)