fix(ktor): scalability for server #559
Conversation
* noticed slow processing in ktor, when there is a high amount of load. The thread pool sizes (connectionGroupSize, workerGroupSize, and callGroupSize) dynamically calculated based on the available processors, ensuring optimal performance.
This setup will be 16, 32, 32 instead of 8, 8, 16 -> This configuration assumes that application processing is more CPU-intensive than connection handling. Not true looking at metrics.
checking the docs:
If /token processing is delayed, we can increase callGroupSize gradually to handle more concurrent requests.
consider:
install(IdleTimeout) {
requestTimeoutMillis = 15000
idleTimeoutMillis = 60000
}
to handle connection not consuming use.
*
|
Nice work! How does Is the slow processing present when scaling with multiple replicas as CPU load goes up? |
In Kubernetes, the value adapts to the CPU limits set on the container. CPU Requests: CPU Limits: like, If no limits are set, availableProcessors() will reflect the total CPUs on the node. If we increase the replicas we can observe how availableProcessors() scales. So we want to set a cpu limit, I tested in dev-gcp and it gave me ->16, thats from the calculation in my PR. |
…Size never exceeds the database maxConnectionPool, that could start starvation or exceed the database `max_connection` limit. * set max_connections flag = 200 * increase the production pool size * 10 pods * 20 connections = 200 connections (matches a database with max_connections = 200)
|
heres an example of the server taking over 1 sec to start to handle the request: |
charts/templates/tokendings.yaml
Outdated
| flags: | ||
| - name: max_connections | ||
| value: "200" |
There was a problem hiding this comment.
This might cause a restart of the sql instance. Otherwise looks good to me!
There was a problem hiding this comment.
Yes, it seems so. I suggest merging this during maintenance hours when traffic is low.
…nt connections for the database matches the replicas * pol_max
Noticed slow processing in ktor, when there is a high amount of load. The thread pool sizes (connectionGroupSize, workerGroupSize, and callGroupSize) dynamically calculated based on the available processors, ensuring optimal performance.
This setup will be 16, 32, 32 instead of 8, 8, 16 -> This configuration assumes that application processing is more CPU-intensive than connection handling. Not true looking at metrics.
checking the docs:
If /token processing is delayed, we can increase callGroupSize gradually to handle more concurrent requests.
consider:
install(IdleTimeout) {
requestTimeoutMillis = 15000
idleTimeoutMillis = 60000
}
to handle connection not consuming resources use