[rocketmq-replicator] Replicator optimization list

[Git-Yang]

Based on the existing foundation of rocketmq-replicator, the optimization of data synchronization and offset synchronization mainly includes the following points:

1. Supports direct data synchronization, avoiding intermediate storage
   1.1 Add WokerDirectTask, responsible for managing RmqSourceTask and RmqSinkTask at the same time. (Read the message through RmqSourceTask, and then write the message directly to the target cluster through RmqSinkTask, omitting intermediate storage.) (done)
   1.2 Add synchronous and asynchronous data transmission. (doing)
   1.3 Add a retry mechanism for sending failures. (doing)

2. Offset synchronization: directly synchronize the consumption offset to the target cluster
   2.1 Based on WokerDirectTask, add MetaSinkTask. Implementation logic is the same as 1.1 (done)
   2.2 Increase the check mechanism to ensure that the synchronized offset matches the actual location as much as possible (done)

3. Other
   3.1 Support ACL (done)
   3.2 Support automatic creation of subscription group (done)
   3.3 Instance creation optimization: (done)

• DefaultMQPullConsumer uses a fixed group, avoids instance conflicts in the same process by creating a unique instance name, and avoids the flood of subscription groups
• DefaultMQPullConsumer uses prefix+wokerId to ensure the reuse of the same instance subscription group. There are two main benefits: one is to avoid the flooding of subscription group data, and the other is to avoid the problem of slow startup caused by consumption from scratch when the service restarts (slow startup) It is due to the addition of configuration verification logic).

基于rocketmq-replicator现有基础上，对数据同步及offset同步进行优化，主要包含一下几点：

1. 支持数据直接同步，避免中间存储
   1.1 增加WokerDirectTask，负责同时管理RmqSourceTask和RmqSinkTask。（通过RmqSourceTask读取消息，再通过RmqSinkTask将消息直接写入目标集群，省略中间存储。） (done)
   1.2 增加同步和异步数据发送 （doing）
   1.3 增加发送失败重试机制 （doing）

2. offset同步：直接将消费offset同步到目标集群
   2.1 基于WokerDirectTask，增加MetaSinkTask。实现逻辑同 1.1 （done）
   2.2 增加check机制，尽可能保证同步的offset与实际位置匹配 (done)

3. 其他
   3.1 支持ACL（done）
   3.2 支持自动创建订阅组（done）
   3.3 实例创建优化：（done）

• DefaultMQPullConsumer 使用固定group，通过创建唯一实例名避免同进程下实例冲突，避免订阅组数量泛滥
• DefaultMQPullConsumer 使用前缀+wokerId的方式，保证同一个实例订阅组复用，主要有两点好处：一是避免订阅组数据泛滥，二是避免服务重启时从头开始消费造成的启动慢的问题（启动慢是由于增加了配置检验逻辑）。
  3.4 部分bug修复：
• topic构建路由逻辑bug（done）

3. Indicate the importance of this issue to you (blocker, must-have, should-have, nice-to-have). Are you currently using any workarounds to address this issue?

• blocker

4. If there are some sub-tasks using -[] for each subtask and create a corresponding issue to map to the sub task:

• 1: 1.1 Support WorkerDirectTask
• 2: 1.1 Add RmqSinTask
• 3: 3.2 Auto create subGroup
• 4: 3.4 Topic build routing logic error
• 5 Not yet submitted
• ...

[Git-Yang]

@duhenglucky @odbozhou
For Replicator, I disassembled what I am currently doing and what I have completed, and the list of solutions is as above. In the last meetup, some implementation logic was also simply synchronized. I hope you can review it if you have time. If there is any discrepancy with the future plan of the community, I hope you can give some suggestions. Thanks!
对于Replicator，我对当前我个人正在做的和已经完成的内容进行了拆解，方案清单如上。 在上次的meetup中，也简单的同步了一些实现逻辑。 希望各位大佬有时间可以review一下，如果与社区未来的计划不符的地方，希望能给够给出一些建议。谢谢！

[duhenglucky]

@duhenglucky @odbozhou For Replicator, I disassembled what I am currently doing and what I have completed, and the list of solutions is as above. In the last meetup, some implementation logic was also simply synchronized. I hope you can review it if you have time. If there is any discrepancy with the future plan of the community, I hope you can give some suggestions. Thanks! 对于Replicator，我对当前我个人正在做的和已经完成的内容进行了拆解，方案清单如上。 在上次的meetup中，也简单的同步了一些实现逻辑。 希望各位大佬有时间可以review一下，如果与社区未来的计划不符的地方，希望能给够给出一些建议。谢谢！

赞，很详细的计划，此外，WokerDirectTask 这个我们可以一起再细化一下

[vongosling]

Up, do we have a dashboard plan? Do we support the bidirectional replicator now?

[Git-Yang]

@duhenglucky @odbozhou For Replicator, I disassembled what I am currently doing and what I have completed, and the list of solutions is as above. In the last meetup, some implementation logic was also simply synchronized. I hope you can review it if you have time. If there is any discrepancy with the future plan of the community, I hope you can give some suggestions. Thanks! 对于Replicator，我对当前我个人正在做的和已经完成的内容进行了拆解，方案清单如上。 在上次的meetup中，也简单的同步了一些实现逻辑。 希望各位大佬有时间可以review一下，如果与社区未来的计划不符的地方，希望能给够给出一些建议。谢谢！

赞，很详细的计划，此外，WokerDirectTask 这个我们可以一起再细化一下

ok，十一之后再找个时间沟通一下

[Git-Yang]

Up, do we have a dashboard plan? Do we support the bidirectional replicator now?

There is currently no dashboard plan. We have also discussed bidirectional replicator internally, but the plan is not yet mature.