Remove unnecessary locks #8207
Merged
Remove unnecessary locks #8207
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/pulsarbot run-failure-checks |
|
@315157973 Can you provide the exact setup on reproducing the perf test results? |
|
eolivelli
reviewed
Oct 7, 2020
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
Outdated
Show resolved
Hide resolved
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
Outdated
Show resolved
Hide resolved
pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
Outdated
Show resolved
Hide resolved
eolivelli
reviewed
Oct 7, 2020
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerBase.java
Outdated
Show resolved
Hide resolved
|
/pulsarbot run-failure-checks |
Added this into the description |
|
/pulsarbot run-failure-checks |
1 similar comment
|
/pulsarbot run-failure-checks |
wolfstudy
pushed a commit
that referenced
this pull request
Oct 30, 2020
### Motivation There are many unnecessary locks in MultiTopicsConsumerImpl, which affect performance. BlockingQueue is inherently thread-safe, and there is no need to lock in many places. ### Modifications Remove unnecessary locks ### Verifying this change Use the perf tool, 3 * 8-core 16G nodes,recording time is about 2 minutes 1. Prepare a 3-node pulsar cluster and produce some data（topic with 4 partitions） 2. Use pulsar-perf on another machine, 3. `bin/pulsar-perf consume -u 'http://x.x.x.x:8080' -s my-sub-6 -sp Earliest -q 100000 persistent://public/default/p-topic` Pressure test twice, the first time with the original one, and the second time to replace the pulsar-client-original.jar in the lib folder before removing： Aggregated throughput stats --- 11715556 records received --- 68813.420 msg/s --- 537.605 Mbit/s after removing： Aggregated throughput stats --- 25062077 records received --- 161656.814 msg/s --- 1262.944 Mbit/s (cherry picked from commit 99627c9)
huangdx0726
pushed a commit
to huangdx0726/pulsar
that referenced
this pull request
Nov 13, 2020
### Motivation There are many unnecessary locks in MultiTopicsConsumerImpl, which affect performance. BlockingQueue is inherently thread-safe, and there is no need to lock in many places. ### Modifications Remove unnecessary locks ### Verifying this change Use the perf tool, 3 * 8-core 16G nodes,recording time is about 2 minutes 1. Prepare a 3-node pulsar cluster and produce some data（topic with 4 partitions） 2. Use pulsar-perf on another machine, 3. `bin/pulsar-perf consume -u 'http://x.x.x.x:8080' -s my-sub-6 -sp Earliest -q 100000 persistent://public/default/p-topic` Pressure test twice, the first time with the original one, and the second time to replace the pulsar-client-original.jar in the lib folder before removing： Aggregated throughput stats --- 11715556 records received --- 68813.420 msg/s --- 537.605 Mbit/s after removing： Aggregated throughput stats --- 25062077 records received --- 161656.814 msg/s --- 1262.944 Mbit/s
merlimat
pushed a commit
to merlimat/pulsar
that referenced
this pull request
Dec 19, 2020
### Motivation There are many unnecessary locks in MultiTopicsConsumerImpl, which affect performance. BlockingQueue is inherently thread-safe, and there is no need to lock in many places. ### Modifications Remove unnecessary locks ### Verifying this change Use the perf tool, 3 * 8-core 16G nodes,recording time is about 2 minutes 1. Prepare a 3-node pulsar cluster and produce some data（topic with 4 partitions） 2. Use pulsar-perf on another machine, 3. `bin/pulsar-perf consume -u 'http://x.x.x.x:8080' -s my-sub-6 -sp Earliest -q 100000 persistent://public/default/p-topic` Pressure test twice, the first time with the original one, and the second time to replace the pulsar-client-original.jar in the lib folder before removing： Aggregated throughput stats --- 11715556 records received --- 68813.420 msg/s --- 537.605 Mbit/s after removing： Aggregated throughput stats --- 25062077 records received --- 161656.814 msg/s --- 1262.944 Mbit/s
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
There are many unnecessary locks in MultiTopicsConsumerImpl, which affect performance.
BlockingQueue is inherently thread-safe, and there is no need to lock in many places.
Modifications
Remove unnecessary locks
Verifying this change
Use the perf tool, 3 * 8-core 16G nodes,recording time is about 2 minutes
bin/pulsar-perf consume -u 'http://x.x.x.x:8080' -s my-sub-6 -sp Earliest -q 100000 persistent://public/default/p-topicPressure test twice, the first time with the original one, and the second time to replace the pulsar-client-original.jar in the lib folder
before removing：
Aggregated throughput stats --- 11715556 records received --- 68813.420 msg/s --- 537.605 Mbit/s
after removing：
Aggregated throughput stats --- 25062077 records received --- 161656.814 msg/s --- 1262.944 Mbit/s