Performance tests part 1 #3434

kciesielski · 2024-01-02T16:48:20Z

This PR introduces a new set of performance tests with a customizable runner. Documentation will be added once the runner passes an initial review.

Short description:
Tests can be run using perfTests/Test/runMain sttp.tapir.perf.PerfTestSuiteRunner -s servers -m simulations -d duration -users -g where some parameters are optional. The runner uses scopt library to handle params, here's its generated usage help:

[error] Usage: perf [options]
[error]   -s, --server <value>    Comma-separated list of short server names, or '*' for all. Available servers: http4s.TapirMulti, http4s.Tapir, http4s.VanillaMulti, http4s.Vanilla, netty.cats.TapirMulti, netty.cats.Tapir, netty.future.TapirMulti, netty.future.Tapir, pekko.TapirMulti, pekko.Tapir, pekko.VanillaMulti, pekko.Vanilla, play.TapirMulti, play.Tapir, play.VanillaMulti, play.Vanilla, vertx.TapirMulti, vertx.Tapir, vertx.VanillaMulti, vertx.Vanilla, vertx.cats.TapirMulti, vertx.cats.Tapir
[error]   -m, --sim <value>       Comma-separated list of short simulation names, or '*' for all. Available simulations: PostBytes, PostFile, PostLongBytes, PostLongString, PostString, SimpleGetMultiRoute, SimpleGet
[error]   -u, --users <value>     Number of concurrent users, default is 1
[error]   -d, --duration <value>  Single simulation duration in seconds, default is 10
[error]   -g, --gatling-reports   Generate Gatling reports for individuals sims, may significantly affect total time (disabled by default)

Number of concurrent users is 1 by default.
Each simulation starts with 5 seconds of warmup sending GETs and POSTs.

Example HTML report fragment:

Some Netty fixes are still in this PR, but they have been also extracted to a separate PR: #3469

TODO:

Documentation

kciesielski · 2024-01-02T17:13:37Z

@adamw Can I ask for a pre-review of the test running mechanism? After a deeper investigation I noticed that we cannot handle this with sbt tasks, because a dynamic expression like

  (Compile / runMain).toTask(s" sttp.tapir.perf.${servName}Server").value

works only if servName is known at compilation time. It is a limitation imposed by sbt macros, which fail with quite a cryptic error.

My proposition is a command, but one that runs just a single task - the gatling perf test. This is because running a server from a command cannot be nicely coordinated with running a test (and then closing the server) using multiple tasks. I'm passing server name to the gatling simulation using system properties (ouch) and then resolving the server runner using reflection (OUCH), but ultimately this solution allows us to gracefully start server before tests, and close it afterwards.
We can then run tests with

perf http4s.Tapir OneRoute
perf netty.TapirMulti PostString

etc.

I'm working now on:

Defining the rest of servers and scenarios
Adding more params to perf, like user count (possibly with additional commands like perfSingleUser or sth like that)
Docs, etc

adamw · 2024-01-03T10:29:50Z

build.sbt

  .jvmPlatform(scalaVersions = List(scala2_13))
-  .dependsOn(core, akkaHttpServer, http4sServer)
+  .dependsOn(core, akkaHttpServer, http4sServer, nettyServer)


let's just use pekko & scala 3? :)

Gatling is based on Akka and requires Scala 2.13 😞 But we could test Pekko instead of Akka anyway, that won't hurt.

adamw · 2024-01-03T10:30:43Z

build.sbt

+  args match {
+    case Seq(servName, simName) =>
+      System.setProperty("tapir.perf.serv-name", servName)
+      Command.process(s"perfTests/Gatling/testOnly sttp.tapir.perf.${simName}Simulation", state)


using the run task (or is it a command?) w/ arguments doesn't work?

then maybe we could have (Compile / runMain).toTask(s"sttp.tapir.perf.Server").value with dynamically-determined arguments? Then Server could pattern-match on the required server impl and run it

We still need dynamic, which doesn't work with tasks. Additionally, running and stopping server in the simulation using before and after seems to give a better control over the server.

ok, sounds reasonable - put it in a comment for posterity ;)

adamw · 2024-01-03T10:56:51Z

perf-tests/src/main/scala/sttp/tapir/perf/Common.scala

-    .errorOut(stringBody)
-    .out(stringBody)
+  def replyWithStr[F[_]](endpoint: PublicEndpoint[_, _, String, Any])(implicit monad: MonadError[F]): ServerEndpoint[Any, F] =
+    endpoint.serverLogicSuccess[F](_ => monad.eval("ok"))


just to stay close to what a user would write, maybe we should avoid using our monad here, and just accept a function Unit => F[String]? so that specific impls would have to provide it explicitly

adamw · 2024-01-22T14:33:28Z

One note about the help screen: great idea, I'd add defaults next to the options (what are the servers, number of concurrent users etc.)

adamw · 2024-01-22T14:39:00Z

perf-tests/src/main/scala/sttp/tapir/perf/Common.scala

+  val LargeInputSize = 5 * 1024 * 1024
+  val Port = 8080
+  val TmpDir: File = new java.io.File(System.getProperty("java.io.tmpdir")).getAbsoluteFile
+  def tempFilePath(): Path = TmpDir.toPath.resolve(s"tapir-${new Date().getTime}-${Random.nextLong()}")


that's a path to a new temp file?

Yes, I'll rename this function to make it clearer.

adamw · 2024-01-22T14:44:26Z

perf-tests/src/test/scala/sttp/tapir/perf/Simulations.scala

+  private val httpProtocol = http.baseUrl(baseUrl)
+
+  // Scenarios
+  val warmUpScenario = scenario("Warm-Up Scenario")


warmup info would be great to include in the help as well

What should be shown there? It is not configurable right now.

well you have the "help" screen, it could just mention that there's a 5-second 3-user warmup before the tests are run, configured as below: (list of options)

@adamw You mean this screen?

[info] =========================================================================================== [info] Running a suite of 1 tests, each for 1 users and 10 seconds [info] Servers: List(pekko.Vanilla) [info] Simulations: List(SimpleGet) [info] Expected total duration: at least 1 minutes [info] Generated suite report paths will be printed to stdout after all tests are finished. [info] ===========================================================================================

Let's call it "info screen" to tell it apart from the message displayed when input params are missing/incorrect :)
So the help screen would be:

[error] Usage: perf [options] [error] -s, --server <value> Comma-separated list of short server names, or '*' for all. Available servers: http4s.TapirMulti, http4s.Tapir, http4s.VanillaMulti, http4s.Vanilla, netty.cats.TapirMulti, netty.cats.Tapir, netty.future.TapirMulti, netty.future.Tapir, pekko.TapirMulti, pekko.Tapir, pekko.VanillaMulti, pekko.Vanilla, play.TapirMulti, play.Tapir, play.VanillaMulti, play.Vanilla, vertx.TapirMulti, vertx.Tapir, vertx.VanillaMulti, vertx.Vanilla, vertx.cats.TapirMulti, vertx.cats.Tapir [error] -m, --sim <value> Comma-separated list of short simulation names, or '*' for all. Available simulations: PostBytes, PostFile, PostLongBytes, PostLongString, PostString, SimpleGetMultiRoute, SimpleGet [error] -u, --users <value> Number of concurrent users, default is 1 [error] -d, --duration <value> Single simulation duration in seconds, default is 10 [error] -g, --gatling-reports Generate Gatling reports for individuals sims, may significantly affect total time (disabled by default)

Hm well taking inspiration from JMH, it always prints the full config in the info screen - including warmup. So maybe that's an even better place (I did mean the help screen originally).

adamw · 2024-01-22T14:45:36Z

Looks good!

As a follow-up, I think the with-interceptor test is missing, from the initial design? Maybe others as well?

kciesielski · 2024-01-22T14:55:09Z

@adamw Regarding interceptors: I'm not sure whether they should be separate servers, or configured with runner parameters? Something like -i log,metrics for example? I'll add them in another PR once we decide.
BTW logging interceptor is enabled by default, so I'll probably need to disable it in this PR for all servers, right?

adamw · 2024-01-22T14:58:30Z

@kciesielski I don't think the logging interceptor is there by default, but maybe this differs between interpreters. If that's the case, we should definitely run all with the same set of interceptors.

I think -i log,metrics (with only these 2 options available) would be great

adamw · 2024-01-22T14:58:54Z

(run all with the same set of interceptors = disable the logging interceptor by default, only enable on-demand)

kciesielski · 2024-01-22T15:05:27Z

@adamw I checked all server options for Vert.x,Netty,http4s,Pekko, and Play - they have some variant of .serverLog(defaultServerLog) as a default:

object VertxCatsServerOptions {

  /** Allows customising the interceptors used by the server interpreter. */
  def customiseInterceptors[F[_]: Async](
      dispatcher: Dispatcher[F]
  ): CustomiseInterceptors[F, VertxCatsServerOptions[F]] =
    CustomiseInterceptors(
      createOptions = (ci: CustomiseInterceptors[F, VertxCatsServerOptions[F]]) =>
        VertxCatsServerOptions(
          dispatcher,
          VertxServerOptions.uploadDirectory(),
          file => Sync[F].delay(Defaults.deleteFile()(file)),
          maxQueueSizeForReadStream = 16,
          ci.interceptors
        )
    ).serverLog(defaultServerLog(LoggerFactory.getLogger("tapir-vertx")))

  def default[F[_]: Async](dispatcher: Dispatcher[F]): VertxCatsServerOptions[F] = customiseInterceptors(dispatcher).options

adamw · 2024-01-22T15:16:18Z

@kciesielski ah ... well the log impls are different, so it might be best to turn it off by default. And turn on on-demand.

…-2023

kciesielski · 2024-01-22T15:52:07Z

@adamw I disabled all the logs and updated README.md. Also applied updates after your initial review. Should I also add the configurable interceptors to this PR?

adamw · 2024-01-22T15:54:55Z

Great :) Let's merge this, smaller PRs are better PRs. Just let's not forget about the interceptors test, we might as well add it after running & profiling these tests.

kciesielski added 2 commits December 15, 2023 12:01

wip

767d9c2

Dynamic perf test runner

1998a34

kciesielski added this to the Performance milestone Jan 2, 2024

kciesielski added 2 commits January 2, 2024 17:49

Merge branch 'master' into perf-tests-2023

32ba29e

Restore netty server dependency

f2c5d79

kciesielski added 5 commits January 2, 2024 18:18

Remove unneeded files

7347583

Remove unneded build utils

67973b1

Apply base simulation to all sims

917dfeb

Remove old way of running perf tests

4695510

Fix file size to 5MB

7f8110b

adamw reviewed Jan 3, 2024

View reviewed changes

kciesielski added 12 commits January 8, 2024 08:12

Explain why a command is used

4342d97

Parameterize active user count

6a9bc15

Switch from Akka to Pekko

5bcc177

Add Netty Cats server

fc414b2

Add Play server

881f2c3

Merge branch 'master' into perf-tests-2023

01536c0

Vanilla Play server skeleton

5ec7484

Finish Play

987ccd1

Refator vanilla Play

dc71db9

Use new ActorSystem for each Pekko/Play server run

01f91bc

Initial suite runner and log processor

2aa228a

Save HTML report

eda178b

kciesielski force-pushed the perf-tests-2023 branch from 97255e9 to eda178b Compare January 11, 2024 14:30

kciesielski added 3 commits January 11, 2024 20:40

CSV reported and inmprovements to HTML reporter

13291a0

Ensure same filename for all reports

50a191d

Allow running all servers or all simulations with '*'

1449769

kciesielski added 2 commits January 22, 2024 10:17

Merge branch 'master' into perf-tests-2023

ce012a2

Include warmup in total duration estimations

818c953

kciesielski mentioned this pull request Jan 22, 2024

[bugfix] Release Netty buffers #3469

Merged

Restore

a4ceeb9

adamw reviewed Jan 22, 2024

View reviewed changes

kciesielski added 2 commits January 22, 2024 16:08

Display default values of optional parameters

cb814cb

Disable logging

f8dbbc1

kciesielski added 7 commits January 22, 2024 16:41

List available servers and sims in the help view

1bf6442

Documentation

76a5e4b

Merge branch 'master' into perf-tests-2023

c40e455

Merge branch 'master' into perf-tests-2023

6aca7f8

Merge remote-tracking branch 'origin/perf-tests-2023' into perf-tests…

bc9bad5

…-2023

Remove accidental double release after merge

6afe52f

Less personal example ;)

f0f9f85

kciesielski marked this pull request as ready for review January 22, 2024 15:51

kciesielski changed the title ~~[wip] Performance tests~~ Performance tests part 1 Jan 23, 2024

kciesielski added 2 commits January 23, 2024 09:56

Restore ideSkipProject

18c11df

Add warmup to the info screen

8904ef2

kciesielski merged commit 334f8a6 into master Jan 23, 2024
28 checks passed

kciesielski deleted the perf-tests-2023 branch January 23, 2024 09:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Performance tests part 1 #3434

Performance tests part 1 #3434

kciesielski commented Jan 2, 2024 •

edited

Loading

kciesielski commented Jan 2, 2024

adamw Jan 3, 2024

kciesielski Jan 3, 2024

adamw Jan 3, 2024

adamw Jan 3, 2024

kciesielski Jan 3, 2024

adamw Jan 3, 2024

adamw Jan 3, 2024

adamw commented Jan 22, 2024

adamw Jan 22, 2024

kciesielski Jan 22, 2024

adamw Jan 22, 2024

kciesielski Jan 22, 2024

adamw Jan 22, 2024

kciesielski Jan 22, 2024

adamw Jan 22, 2024

adamw commented Jan 22, 2024

kciesielski commented Jan 22, 2024

adamw commented Jan 22, 2024

adamw commented Jan 22, 2024

kciesielski commented Jan 22, 2024

adamw commented Jan 22, 2024

kciesielski commented Jan 22, 2024

adamw commented Jan 22, 2024

Performance tests part 1 #3434

Performance tests part 1 #3434

Conversation

kciesielski commented Jan 2, 2024 • edited Loading

kciesielski commented Jan 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamw commented Jan 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adamw commented Jan 22, 2024

kciesielski commented Jan 22, 2024

adamw commented Jan 22, 2024

adamw commented Jan 22, 2024

kciesielski commented Jan 22, 2024

adamw commented Jan 22, 2024

kciesielski commented Jan 22, 2024

adamw commented Jan 22, 2024

kciesielski commented Jan 2, 2024 •

edited

Loading