Performance Load Test Suite

[wynan]

Looking to add some performance test suites to make performance testing easier and to hopefully give users tools to identify the outbox load that could be supported by their database. Was thinking to implement this in JMeter but I would be open to other testing tools as well if there is a preference.

[badgerwithagun]

Interesting! JMeter sounds good.

What we found doing our own performance testing is that the following were key:

1. The work the jobs are doing (yes, I know, seems obvious, but CPU-bound vs network bound, long-lived vs short-lived can have a huge impact and dramatically change the impact on the database)
2. Yes, the database
3. The number of application instances competing to process work (and to what degree you are clustering the workload)
4. The size of the thread pool in the Executor used for the Submitter on the submitting app instance (assuming you're using DefaultSubmitter; it all gets more complicated when you consider clustering)
5. Under load that thread pool will always get maxed out, no matter how large, so the Executor's BlockingQueue length starts to become relevant. A LinkedBlockingQueue can expand boundlessly and we saw OOMs in production after a while, but an ArrayBlockingQueue will start to reject work, forcing work to be processed by calls to flush().

We have never performance tested ordered messages. We have always assumed that this will be slow. It can't not be!