Java subscriber throughput limited to 130,000 qps

[dpcollins-google]

Environment details

• OS: Ubuntu
• Java version: 8
• google-cloud-java version(s): 1.62.0

Steps to reproduce

1. Run a publisher publishing with > 130,000 qps of 1kb sized messages
2. Run a subscriber doing minimal work in the callback other than incrementing an atomic counter

Code snippet

Integer BYTES_PER_WORKER = 100000000;
Integer workerCount = 5* Runtime.getRuntime().availableProcessors();
Subscriber.newBuilder(subscription, MinimalMessageReceiver)
                            .setParallelPullCount(workerCount)
                            .setFlowControlSettings(FlowControlSettings.newBuilder()
                                    .setMaxOutstandingElementCount(Long.MAX_VALUE)
                                    .setMaxOutstandingRequestBytes(BYTES_PER_WORKER * workerCount)
                                    .build())
                            .build();

Any additional information below

I've been working to improve the Cloud Pub/Sub loadtest framework. I've been unable to achieve the same throughput in java as I have in golang under the same conditions. I'm unable to increase subscriber throughput on a 16 core machine beyond 130MB/s with 1KB messages, but have been seeing 300+MB/s rates with golang under the same conditions. I've also tried setting the channel provider and executors to different values to no avail. Since the cpu load is stuck at about 35%, perhaps this is a contention or artificial limitation somewhere else in the client library?

In progress branch for updates here https://github.com/dpcollins-google/pubsub/tree/loadtest_updates

[sduskis]

@jadekler, @kolea2 FYI.

Let's discuss this offline. Please include Kir in the conversation.

[jeanbza]

👍

[dpcollins-google]

sure. dpcollins@. Put something on my calendar.

[sduskis]

@dpcollins-google, can we close this?

[dpcollins-google]

Yes, I thought it would be automatically closed by putting resolves in the last pr description. You can close or I can when I get home.

…

On Thu, Mar 14, 2019, 1:53 PM Solomon Duskis ***@***.***> wrote: @dpcollins-google <https://github.com/dpcollins-google>, can we close this? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4485 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Amn1sv9I-pcXrR307T1PHbNG7R1g5m0Lks5vWox-gaJpZM4a4MyK> .