Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve performance for 4.9.2 #3585

Closed
areyouok opened this issue Dec 6, 2021 · 2 comments · Fixed by #3586, #3588, #3589, #3590 or #3591
Closed

Improve performance for 4.9.2 #3585

areyouok opened this issue Dec 6, 2021 · 2 comments · Fixed by #3586, #3588, #3589, #3590 or #3591

Comments

@areyouok
Copy link
Contributor

areyouok commented Dec 6, 2021

We have some performance improvements based on 4.9.2

  1. [Part A] eliminate reverse DNS lookup in MessageExt
  2. [Part B] Improve header encode/decode performance
  3. [Part B] Improve RocketMQSerializable performance with zero-copy
  4. [Part C] cache result for parseChannelRemoteAddr()
  5. [Part D] improve performance of createUniqID()
  6. [Part E] eliminate duplicated getNamespace() call when where is no namespace
  7. [Part F] eliminate regex match in topic/group name check
  8. [Part G] [Work in progress] support send batch message with different topic/queue
  9. [Part H] eliminate StringBuilder auto resize in PullRequestHoldService.buildKey() when topic length is greater than 14, this method called twice for each message.
  10. [Part I] Avoid unnecessary StringBuffer resizing and String Formatting
  11. [Part J] Use mmap buffer instead of FileChannel when writing consume queue and slave commit log, which greatly speed up consume tps.
  12. Part K move execution of notifyMessageArriving() from ReputMessageService thread to PullRequestHoldService thread.

These commits almost eliminate bad performance methods in the cpu flame graph in producer side.

@areyouok
Copy link
Contributor Author

areyouok commented Dec 8, 2021

Part A to F is complete. Unit test fixed.

Zanglei06 pushed a commit to Zanglei06/rocketmq that referenced this issue Dec 9, 2021
Zanglei06 pushed a commit to Zanglei06/rocketmq that referenced this issue Dec 9, 2021
areyouok added a commit to areyouok/rocketmq that referenced this issue Dec 16, 2021
…putMessageService thread to PullRequestHoldService thread.

This commit speed up consume qps greatly, in our test up to 200,000 qps.
areyouok added a commit to areyouok/rocketmq that referenced this issue Dec 24, 2021
… (about 45ms) under performance test, this commit fix it.
areyouok added a commit to areyouok/rocketmq that referenced this issue Dec 29, 2021
areyouok added a commit to areyouok/rocketmq that referenced this issue Feb 10, 2022
…) from ReputMessageService thread to PullRequestHoldService thread
areyouok added a commit to areyouok/rocketmq that referenced this issue Feb 10, 2022
@kimmking
Copy link
Member

kimmking commented Mar 5, 2023

About Part J:
Actually, mmap is better than fileChannel in small size message case. In large scale size message, fileChannel is more better.
There could be an option to switch mmap/fileChannel.
See https://www.jianshu.com/p/d0b4ac90dbcb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment