[RIP-28] light message queue(LMQ) #3694

tianliuliu · 2022-01-04T07:47:10Z

Brief changelog

#3666 Support light message queue (LMQ) multi-consumeQueue atomic dispatch, at the same time support LMQ metaData and offset management

Some messaging scenarios require light message queue, Such as MQTT multi-level topic or AMQP lightweight queue can be set at will by users when sending and subscription message. Therefore, the implementation of LMQ's queue model is required

Implement multi-dimensional dispatch of a message based on ConsumeQueue, and a message can support multiple lightweight queue consumption.
Implement multi-level topic management, so topic metadata also needs adaptation management, such as topic verification, data cleaning and so on.
Implement lightweight queue consumption offset management

Follow this checklist to help us incorporate your contribution quickly and easily. Notice, it would be helpful if you could finish the following 5 checklist(the last one is not necessary)before request the community to review your PR.

Make sure there is a Github issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a Github issue. Your pull request should address just this issue, without pulling in other changes - one PR resolves one issue.
Format the pull request title like [ISSUE #123] Fix UnknownException when host config not exist. Each commit in the pull request should have a meaningful subject line and body.
Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
Write necessary unit-test(over 80% coverage) to verify your logic correction, more mock a little better when cross module dependency exist. If the new feature or significant change is committed, please remember to add integration-test in test module.
Run mvn -B clean apache-rat:check findbugs:findbugs checkstyle:checkstyle to make sure basic checks pass. Run mvn clean install -DskipITs to make sure unit-test pass. Run mvn clean test-compile failsafe:integration-test to make sure integration-test pass.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

coveralls · 2022-01-04T09:13:31Z

Coverage decreased (-2.1%) to 51.032% when pulling f4dc125 on tianliuliu:support_mqtt into a7b5903 on apache:develop.

codecov-commenter · 2022-01-04T09:13:48Z

Codecov Report

Attention: Patch coverage is 34.07080% with 596 lines in your changes missing coverage. Please review.

Project coverage is 46.98%. Comparing base (a7b5903) to head (f4dc125).
Report is 6717 commits behind head on develop.

Files with missing lines	Patch %	Lines
...n/java/org/apache/rocketmq/test/util/StatUtil.java	16.19%	231 Missing and 7 partials ⚠️
...che/rocketmq/test/lmq/benchmark/BenchLmqStore.java	51.98%	90 Missing and 7 partials ⚠️
...etmq/broker/plugin/AbstractPluginMessageStore.java	0.00%	58 Missing ⚠️
...he/rocketmq/store/stats/LmqBrokerStatsManager.java	0.00%	57 Missing ⚠️
...org/apache/rocketmq/store/DefaultMessageStore.java	47.88%	24 Missing and 13 partials ⚠️
.../java/org/apache/rocketmq/store/MultiDispatch.java	66.25%	17 Missing and 10 partials ⚠️
.../broker/longpolling/LmqPullRequestHoldService.java	0.00%	22 Missing ⚠️
...cketmq/broker/offset/LmqConsumerOffsetManager.java	44.44%	16 Missing and 4 partials ⚠️
...n/java/org/apache/rocketmq/store/ConsumeQueue.java	64.51%	7 Missing and 4 partials ⚠️
...a/org/apache/rocketmq/broker/BrokerController.java	0.00%	0 Missing and 5 partials ⚠️
... and 7 more

Additional details and impacted files

@@              Coverage Diff              @@
##             develop    #3694      +/-   ##
=============================================
- Coverage      47.19%   46.98%   -0.21%     
+ Complexity      5032     4857     -175     
=============================================
  Files            628      636       +8     
  Lines          41400    42244     +844     
  Branches        5377     5521     +144     
=============================================
+ Hits           19540    19850     +310     
- Misses         19437    19909     +472     
- Partials        2423     2485      +62

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

yuz10 · 2022-01-06T01:23:34Z

Is there documentation about how to use the light message queue?

tianliuliu · 2022-01-06T02:45:57Z

Is there documentation about how to use the light message queue?

There is no document yet, you can refer to the demo in PR: test/src/test/java/org/apache/rocketmq/test/lmq/TestBenchLmqStore.java

tianliuliu · 2022-01-06T12:11:18Z

Is there documentation about how to use the light message queue?

https://github.com/tianliuliu/rocketmq/blob/support_mqtt/docs/cn/Example_LMQ.md

duhenglucky · 2022-01-07T03:56:05Z

broker/src/test/java/org/apache/rocketmq/broker/BrokerPathConfigHelperTest.java

+
+import org.junit.Test;
+
+import static org.junit.Assert.*;


Don't use * to replace all imports.

duhenglucky · 2022-01-07T03:58:27Z

store/src/main/java/org/apache/rocketmq/store/ConsumeQueue.java

+                    log.warn("[BUG]put commit log position info to " + queueName + ":" + queueId + " " + request.getCommitLogOffset()
+                        + " failed, retry " + j + " times");
+
+                    try {


Would you like to reduce the nesting level？

duhenglucky · 2022-01-07T03:59:58Z

test/src/main/java/org/apache/rocketmq/test/lmq/benchmark/BenchLmqStore.java

+                            logger.error(" ", e);
+                        }
+                    }
+                } catch (Exception e) {


Same with the last comment，nesting level should be reduced and error log may be can more clear

some details should be polish.

ShannonDing · 2022-01-10T03:41:43Z

LMQ的消息通过客户端暴露出来的userProperty参数区分，在服务端动态获取LMQ并创建逻辑队列。这里建议对客户端传递的参数和服务端的逻辑队列数做下校验，防止客户端传递大量非法值，引起服务端逻辑队列创建过多引发资源耗尽。
建议在服务端broker.conf里增加max_lmq_size保护门限，可根据实际情况调整LMQ最大服务的数量。

yuz10 · 2022-01-11T16:11:05Z

docs/cn/Example_LMQ.md

+
+Long offset = defaultMQPullConsumer.maxOffset(mq);
+
+defaultMQPullConsumer.pullBlockIfNotFound(


Hi, I tested the doc and found some problems with the example:

producer: queueId is illegal

caused by: org.apache.rocketmq.client.exception.MQBrokerException: CODE: 1 DESC: request queueId[3] is illegal, TopicConfig [topicName=TopicTest, readQueueNums=1, writeQueueNums=1, perm=RW-, topicFilterType=SINGLE_TAG, topicSysFlag=0, order=false] Producer: 172.18.40.26:50188 BROKER: 172.18.40.26:10911 For more information, please visit the url, http://rocketmq.apache.org/docs/faq/ at org.apache.rocketmq.client.impl.MQClientAPIImpl.processSendResponse(MQClientAPIImpl.java:662) at org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessageSync(MQClientAPIImpl.java:508) at org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:490) at org.apache.rocketmq.client.impl.MQClientAPIImpl.sendMessage(MQClientAPIImpl.java:434) at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendKernelImpl(DefaultMQProducerImpl.java:838) at org.apache.rocketmq.client.impl.producer.DefaultMQProducerImpl.sendDefaultImpl(DefaultMQProducerImpl.java:581) ... 4 more

consumer: The broker[broker-a] not exist

Exception in thread "main" org.apache.rocketmq.client.exception.MQClientException: The broker[broker-a] not exist For more information, please visit the url, http://rocketmq.apache.org/docs/faq/ at org.apache.rocketmq.client.impl.MQAdminImpl.maxOffset(MQAdminImpl.java:219) at org.apache.rocketmq.client.impl.consumer.DefaultMQPullConsumerImpl.maxOffset(DefaultMQPullConsumerImpl.java:170) at org.apache.rocketmq.client.consumer.DefaultMQPullConsumer.maxOffset(DefaultMQPullConsumer.java:155) at org.apache.rocketmq.example.simple.PullConsumer.main(PullConsumer.java:49)

Add Thead.sleep(30000) will solve it, because the lmq topic is not in nameserver, should wait until the timer fetch broker names.

and I enhanced the consumer, to make it consume continously:

DefaultMQPullConsumer defaultMQPullConsumer = new DefaultMQPullConsumer(); defaultMQPullConsumer.setNamesrvAddr("localhost:9876"); defaultMQPullConsumer.setVipChannelEnabled(false); defaultMQPullConsumer.setConsumerGroup("CID_RMQ_SYS_LMQ_TEST"); defaultMQPullConsumer.setInstanceName("CID_RMQ_SYS_LMQ_TEST"); defaultMQPullConsumer.setRegisterTopics(new HashSet<>(Collections.singletonList("TopicTest"))); defaultMQPullConsumer.start(); Thread.sleep(30000); String brokerName = "broker-a"; MessageQueue mq = new MessageQueue("%LMQ%123", brokerName, 0); long offset = defaultMQPullConsumer.fetchConsumeOffset(mq, false); while (true) { PullResult result = defaultMQPullConsumer.pullBlockIfNotFound(mq, "*", offset, 32); if (result.getMsgFoundList() == null || result.getMsgFoundList().isEmpty()) { continue; } for (MessageExt messageExt : result.getMsgFoundList()) { System.out.printf("%s%n", messageExt); } offset = result.getNextBeginOffset(); defaultMQPullConsumer.updateConsumeOffset(mq, offset); }

request queueId[3] is illegal, TopicConfig [topicName=TopicTest, readQueueNums=1, writeQueueNums=1, perm=RW-, topicFilterType=SINGLE_TAG, topicSysFlag=0, order=false] Producer: 172.18.40.26:50188 BROKER: 172.18.40.26:10911
For more information, please visit the url, http://rocketmq.apache.org/docs/faq/

TopicTest 多创建几个队列就可以了这个只有一个队列所以队列queueId 3拿不到
2. broker-a 不存在问题就是你部署的mq集群中，没有这个broker

TopicTest有3个队列，因为light队列中将队列数改为1导致判断有问题。

broker-a 是存在的，defaultMQPullConsumer.maxOffset中代码

String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName()); if (null == brokerAddr) { this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic()); brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName()); }

updateTopicRouteInfoFromNameServer是通过topic去nameserver更新broker列表，nameserver中不存在这个topic（%LMQ%123），所以无法获取broker。sleep 30秒等待定时器同步broker列表之后就能在缓存中拿到。

放出来的示例我试验的结果是这样的，不知道你有没有试过

TopicTest有3个队列，因为light队列中将队列数改为1导致判断有问题。

broker-a 是存在的，defaultMQPullConsumer.maxOffset中代码

String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName()); if (null == brokerAddr) { this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic()); brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName()); }

updateTopicRouteInfoFromNameServer是通过topic去nameserver更新broker列表，nameserver中不存在这个topic（%LMQ%123），所以无法获取broker。

%LMQ%这种是不会上报nameserver的，根据那个需要分发的普通topic来获取，TopicTest是普通的，不能当作lmq, lmq只能内部分发，lmq topic只支持命名格式%LMQ%开头的name

TopicTest有3个队列，因为light队列中将队列数改为1导致判断有问题。

broker-a 是存在的，defaultMQPullConsumer.maxOffset中代码

String brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName()); if (null == brokerAddr) { this.mQClientFactory.updateTopicRouteInfoFromNameServer(mq.getTopic()); brokerAddr = this.mQClientFactory.findBrokerAddressInPublish(mq.getBrokerName()); }

updateTopicRouteInfoFromNameServer是通过topic去nameserver更新broker列表，nameserver中不存在这个topic（%LMQ%123），所以无法获取broker。sleep 30秒等待定时器同步broker列表之后就能在缓存中拿到。

放出来的示例我试验的结果是这样的，不知道你有没有试过

这个地方brokerName 我不是从nameserver动态获取的，我知道lmq会分发在我的demo broker-a上，直接写死的，你的lmq能分发到哪个broker 就从哪个broker上拉取消息这个lmq不会上报nameServer,只能根据普通topic来获取brokername列表，还是需要根据普通topic更新到本地缓存的，之前写的demo只是伪逻辑哈

zhouxinyu · 2022-01-12T07:43:46Z

broker/src/main/java/org/apache/rocketmq/broker/topic/LmqTopicConfigManager.java

+import org.apache.rocketmq.common.TopicConfig;
+import org.apache.rocketmq.common.constant.PermName;
+
+public class LmqTopicConfigManager extends TopicConfigManager {


Does LMQ need a special TopicConfigManager? The broker should have unified topic management ability.

LMQ topic will not register nameServer. As a lightweight resource , TopicConfigManager is only used for broker verification and logical processing. LMQ topic resource will be managed at the proxy layer.

odbozhou added the type/rip label Jan 4, 2022

duhenglucky added this to the 4.9.3 milestone Jan 4, 2022

tianliuliu added 9 commits January 6, 2022 20:46

for support mqtt protocol

9da826e

for fix bug and benchmark test

876a820

for add lmq subscription group manager

c8b4a58

for Thread local decoder and remove wait prop string

891bf33

for lmq Code style

1f608e4

just for lmq Code style

97ee08b

for apache#3666 for add unit test

b40d06e

for apache#3666 add unit test more

9e7af1c

for apache#3666 add unit test more and docs

3e07fd2

tianliuliu force-pushed the support_mqtt branch from 7e96e21 to 3e07fd2 Compare January 6, 2022 12:46

RongtongJin approved these changes Jan 6, 2022

View reviewed changes

odbozhou approved these changes Jan 7, 2022

View reviewed changes

tianliuliu requested a review from odbozhou January 7, 2022 03:30

duhenglucky previously approved these changes Jan 7, 2022

View reviewed changes

duhenglucky reviewed Jan 7, 2022

View reviewed changes

tianliuliu added 2 commits January 7, 2022 14:15

for apache#3666 format code and reduce nesting level

54b9942

for apache#3666 just for add more unit tests

1204bb0

for apache#3666 for add lmq clear and num limit

44fb087

yuz10 reviewed Jan 11, 2022

View reviewed changes

for apache#3666 for fix unit test

97512fd

duhenglucky added the module/broker label Jan 12, 2022

duhenglucky added the module/store label Jan 12, 2022

tianliuliu added 3 commits January 12, 2022 14:48

for apache#3666 for fix example demo

4b80706

for apache#3666 for delete lmq topic comment

3513c4d

for apache#3666 for example docs

f4dc125

duhenglucky approved these changes Jan 13, 2022

View reviewed changes

RongtongJin merged commit 14b63ee into apache:develop Jan 13, 2022

zhouxinyu reviewed Jan 14, 2022

View reviewed changes

This was referenced Jan 14, 2022

[ISSUE #3666] fix build error in RIP-28 at getmQclientFactory #3753

Merged

[RIP-28]Support light message queue (LMQ) multi-consumeQueue atomic dispatch #3666

Closed

devfat pushed a commit to devfat/rocketmq that referenced this pull request Mar 3, 2022

[RIP-28] light message queue(LMQ) (apache#3694)

fcb7da5

GenerousMan pushed a commit to GenerousMan/rocketmq that referenced this pull request Aug 12, 2022

[RIP-28] light message queue(LMQ) (apache#3694)

6f1805c

carlvine500 pushed a commit to carlvine500/rocketmq-apache that referenced this pull request Sep 10, 2024

[RIP-28] light message queue(LMQ) (apache#3694)

62cfc64

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[RIP-28] light message queue(LMQ) #3694

[RIP-28] light message queue(LMQ) #3694

tianliuliu commented Jan 4, 2022 •

edited

Loading

coveralls commented Jan 4, 2022 •

edited

Loading

codecov-commenter commented Jan 4, 2022 •

edited

Loading

yuz10 commented Jan 6, 2022

tianliuliu commented Jan 6, 2022

tianliuliu commented Jan 6, 2022

duhenglucky Jan 7, 2022

duhenglucky Jan 7, 2022

duhenglucky Jan 7, 2022

ShannonDing commented Jan 10, 2022

yuz10 Jan 11, 2022 •

edited

Loading

tianliuliu Jan 12, 2022 •

edited

Loading

yuz10 Jan 12, 2022 •

edited

Loading

tianliuliu Jan 12, 2022

tianliuliu Jan 12, 2022 •

edited

Loading

zhouxinyu Jan 12, 2022

tianliuliu Jan 14, 2022 •

edited

Loading


		Long offset = defaultMQPullConsumer.maxOffset(mq);

		defaultMQPullConsumer.pullBlockIfNotFound(

[RIP-28] light message queue(LMQ) #3694

[RIP-28] light message queue(LMQ) #3694

Conversation

tianliuliu commented Jan 4, 2022 • edited Loading

Brief changelog

coveralls commented Jan 4, 2022 • edited Loading

codecov-commenter commented Jan 4, 2022 • edited Loading

Codecov Report

yuz10 commented Jan 6, 2022

tianliuliu commented Jan 6, 2022

tianliuliu commented Jan 6, 2022

duhenglucky Jan 7, 2022

Choose a reason for hiding this comment

duhenglucky Jan 7, 2022

Choose a reason for hiding this comment

duhenglucky Jan 7, 2022

Choose a reason for hiding this comment

ShannonDing commented Jan 10, 2022

yuz10 Jan 11, 2022 • edited Loading

Choose a reason for hiding this comment

tianliuliu Jan 12, 2022 • edited Loading

Choose a reason for hiding this comment

yuz10 Jan 12, 2022 • edited Loading

Choose a reason for hiding this comment

tianliuliu Jan 12, 2022

Choose a reason for hiding this comment

tianliuliu Jan 12, 2022 • edited Loading

Choose a reason for hiding this comment

zhouxinyu Jan 12, 2022

Choose a reason for hiding this comment

tianliuliu Jan 14, 2022 • edited Loading

Choose a reason for hiding this comment

tianliuliu commented Jan 4, 2022 •

edited

Loading

coveralls commented Jan 4, 2022 •

edited

Loading

codecov-commenter commented Jan 4, 2022 •

edited

Loading

yuz10 Jan 11, 2022 •

edited

Loading

tianliuliu Jan 12, 2022 •

edited

Loading

yuz10 Jan 12, 2022 •

edited

Loading

tianliuliu Jan 12, 2022 •

edited

Loading

tianliuliu Jan 14, 2022 •

edited

Loading