Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kafka生产端增加按pk hash到对应partition功能 #958

Merged
merged 7 commits into from
Sep 21, 2018
Merged

kafka生产端增加按pk hash到对应partition功能 #958

merged 7 commits into from
Sep 21, 2018

Conversation

rewerma
Copy link
Collaborator

@rewerma rewerma commented Sep 20, 2018

kafka生产端增加按pk hash到对应partition功能,未设置分区hash的或则默认状态全部发送到分区0
server端kafka增加如下配置:

canalDestinations:
  - canalDestination: example
    topic: expample
    #对应topic分区数量
    partitionsNum: 3
    partitionHash:
      #库名.表名: 唯一主键  指定库表中的唯一主键用其值作为hash分发partition
      mytest.person: id

此功能只适用于flatMessage模式

@rewerma rewerma changed the title 默认将数据发到kafka的partition 0 kafka生产端增加按pk hash到对应partition功能 Sep 20, 2018
@agapple
Copy link
Member

agapple commented Sep 21, 2018

建议可以优化一下表的hash字段定义,方便在大量表的配置问题:

  1. 增加通配符匹配表,同时支持pk这样的默认关键字
  2. 未命中通配符的,按照 schema.table : columngA 进行匹配

@shuaicloud
Copy link

迫不及待

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants