-
Notifications
You must be signed in to change notification settings - Fork 7.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
v1.1.3-alpha-3版kafka主题为多分区时,消费到的数据不完整 #1636
Comments
wangsaner
changed the title
v1.1.3-alpha-3版kafka主题为多分区时,消费端丢数据
v1.1.3-alpha-3版kafka主题为多分区时,消费到的数据不完整
Mar 27, 2019
测试和验证的方式? 我之前测试过非flat模式,数据会被拆分到多个parition,总数是对的上 |
我是直接update一张表的所有记录,当主题为单分区的时候,消费端可以消费到所有变更记录, |
找到问题,数据拆分到多个parition时,只取了for循环里面的最后一份拆分数据(com.alibaba.otter.canal.kafka.CanalKafkaProducer) : 之前:
修改后:
|
wangsaner
added a commit
to wangsaner/canal
that referenced
this issue
Mar 27, 2019
非flat模式,当MQ主题为多分区时,数据拆分只取了for循环里面的最后一份数据, 导致消费端只能消费到一个分区的数据
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
environment
Issue Description
kafka主题为多分区时,消费端消费到的数据不完整
canal.properties主要配置如下:
######### binlog filter config
canal.instance.filter.druid.ddl = true
canal.instance.filter.query.dcl = true
canal.instance.filter.query.dml = false
canal.instance.filter.query.ddl = true
canal.instance.filter.table.error = true
canal.instance.filter.rows = false
canal.instance.filter.transaction.entry = false
##################################################
######### MQ #############
##################################################
canal.mq.servers = 192.168.1.15:9092,192.169.1.16:9092,192.168.1.15:9092
canal.mq.retries = 3
canal.mq.batchSize = 16384
canal.mq.maxRequestSize = 1048576
canal.mq.lingerMs = 1
canal.mq.bufferMemory = 33554432
canal.mq.canalBatchSize = 50
canal.mq.canalGetTimeout = 100
canal.mq.flatMessage = false
canal.mq.compressionType = none
canal.mq.acks = all
######### use transaction for kafka flatMessage batch produce
canal.mq.transaction = false
instance.properties主要配置如下:
######### table regex
canal.instance.filter.regex=schema.table
######### table black regex
canal.instance.filter.black.regex=
######### mq config
canal.mq.topic=topic
canal.mq.partition=0
######### hash partition config
canal.mq.partitionsNum=8
canal.mq.partitionHash=.*\\..*
######### kafka消费端 ########
@KafkaListener(topics = "${spring.kafka.consumer.topic}",
containerFactory = "kafkaListenerContainerFactory")
public void consumerListener(KafkaMessage message, Acknowledgment ack) {
try {
boolean success = true;
Message canalMessage = message.getMessage();
if(canalMessage != null) {
if (canalMessage.getId() != -1 &&
canalMessage.getEntries().size() > 0) {
success = printEntry(canalMessage.getEntries());
}
}
if(success)
ack.acknowledge();
}catch (Exception e) {
logger.error(e.getMessage(), e);
}
}
Steps to reproduce
Expected behaviour
Actual behaviour
The text was updated successfully, but these errors were encountered: