Skip to content

Commit

Permalink
[Client] Support passing existing executor providers to the client (#…
Browse files Browse the repository at this point in the history
…12037)

### Motivation

In load and performance testing, there's a need to simulate production use cases and production workloads.
For this purpose, it would be useful to be able to share the thread pools used by Pulsar client instances in order to be able to run a large amount of Pulsar clients in a single JVM without the overhead of a lot of threads. 
In the current solution, it's already possible to share the EventLoopGroup and HashedWheelTimer instances.
The solution for sharing the thread pools for the external / internal executors was missing. This PR adds support for that.

Example usage:

```java
// shared thread pool related resources
ExecutorProvider internalExecutorProvider = new ExecutorProvider(8, "shared-internal-executor");
ExecutorProvider externalExecutorProvider = new ExecutorProvider(8, "shared-external-executor");
Timer sharedTimer = new HashedWheelTimer(getThreadFactory("shared-pulsar-timer"), 1, TimeUnit.MILLISECONDS);
EventLoopGroup sharedEventLoopGroup = new EpollEventLoopGroup();

// example of creating a client which uses the shared thread pools
PulsarClientImpl client = PulsarClientImpl.builder().conf(conf)
                .internalExecutorProvider(internalExecutorProvider)
                .externalExecutorProvider(externalExecutorProvider)
                .timer(sharedTimer)
                .eventLoopGroup(sharedEventLoopGroup)
                .build();
```

It seems that this would also improve the performance of the Pulsar Proxy since new thread pools for every client connection.

That happens in the Pulsar Proxy currently: 
https://github.com/apache/pulsar/blob/af63e96d4aaa0ae4c4086583aa4f9b1edd72279b/pulsar-proxy/src/main/java/org/apache/pulsar/proxy/server/ProxyConnection.java#L445-L451

An optimization was added in #9802 for sharing the timer, but it would be useful to also share the internal / external executors.
  • Loading branch information
lhotari authored Jan 19, 2022
1 parent c2e2d57 commit 4591a21
Show file tree
Hide file tree
Showing 2 changed files with 130 additions and 72 deletions.
Loading

0 comments on commit 4591a21

Please sign in to comment.