fix: v1.1.5-snapshot rdb-adapter 由关键字引起的异常 (#3026 #2783) #3020

XuDaojie · 2020-09-08T08:20:53Z

问题描述

~~当库名或表名包含关键字或 - 时抛异常 #3019 问题已关闭，第一个commit原本是解决这个问题的，后面发现可以通过调整配置解决~~
postgres 驱动不兼容 postgres:12.4 升级驱动为42.2.16
canal 1.1.5-alpha2 同步不兼容postgresql #3026 因有个表用了explain字段，源表新增数据后adapter同步时报SQLSyntaxErrorException异常 #2783
此pull request最初是想解决当库名或表名包含关键字或 - 时抛异常 #3019，后经@yansheng105 提醒这种实现不能兼容postgres，查看文档后发现rdb-adapter原本设计的是支持多种rdb数据库，#2358中兼容了mysql关键字后导致rdb-adapter只能兼容MySQL

解决过程

关键字处理主要涉及表名、字段名，对于表名可以通过在配置yml文件时直接加上标识符解决，而字段名存在关键字并且mapAll=true的话则目前代码无法处理，目前代码主要是处理的字段名关键字标识符的问题
不同数据库处理保留字的标识符不同，第一种方案是通过判断数据库连接自动确定标识符，第二种是通过增加配置项，
目前选的第二种方案，主要考虑不同数据库标识符不同，出现了未知的数据库也可以自行配置

mysql: select `name`,`date` from user;
postgresql: select "name","date" from user;
mssql: select [name],[date] from user;

新增配置

# postgres
dbMapping:
  database: retl
  table: xdual
  targetTable: '"a-b"'       # 处理表名包含关键字或`-`，直接引号通过声明表名即可解决
  targetPk:
    id: id
  targetKeywordsIdentifier:  # 新增配置项 关键字、保留字标识符(对表名无效)
    prefix: '"'              # 默认值为 `
    suffix: '"'              # 默认值为 `
  mapAll: true
  commitBatch: 3000

CLAassistant · 2020-09-08T08:20:58Z

All committers have signed the CLA.

codecov-commenter · 2020-09-08T08:46:38Z

Codecov Report

Merging #3020 into master will increase coverage by 0.04%.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##             master   #3020      +/-   ##
===========================================
+ Coverage      4.76%   4.81%   +0.04%     
  Complexity      239     239              
===========================================
  Files           392     396       +4     
  Lines         35709   35717       +8     
  Branches       5562    5555       -7     
===========================================
+ Hits           1702    1720      +18     
+ Misses        33758   33746      -12     
- Partials        249     251       +2

Impacted Files	Coverage Δ	Complexity Δ
...canal/client/adapter/rdb/config/MappingConfig.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
.../canal/client/adapter/rdb/config/RDBConstants.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (?)`
...nal/client/adapter/rdb/service/RdbSyncService.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...anal/client/adapter/rdb/support/BatchExecutor.java	`0.00% <0.00%> (ø)`	`0.00 <0.00> (ø)`
...ter/canal/client/adapter/rdb/support/SyncUtil.java	`0.00% <ø> (ø)`	`0.00 <0.00> (ø)`
...a/com/alibaba/otter/canal/filter/PatternUtils.java	`44.44% <0.00%> (-5.56%)`	`0.00% <0.00%> (ø%)`
...r/canal/sink/entry/HeartBeatEntryEventHandler.java	`33.33% <0.00%> (-2.39%)`	`3.00% <0.00%> (ø%)`
...otter/canal/filter/aviater/AviaterRegexFilter.java	`86.11% <0.00%> (-1.39%)`	`0.00% <0.00%> (ø%)`
...com/taobao/tddl/dbsync/binlog/event/LogHeader.java	`32.20% <0.00%> (-0.56%)`	`14.00% <0.00%> (ø%)`
...java/com/taobao/tddl/dbsync/binlog/LogDecoder.java	`0.00% <0.00%> (ø)`	`0.00% <0.00%> (ø%)`
... and 64 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b81e9df...925a940. Read the comment docs.

yansheng105 · 2020-09-10T01:18:28Z

这种写法不兼容postgresql

yansheng105 · 2020-09-15T01:22:27Z

还有一处忘记修改了哦。
com.alibaba.otter.canal.client.adapter.rdb.support.SyncUtil

    public static String getDbTableName(MappingConfig.DbMapping dbMapping) {
        Map<String, String> keywordsIdentifier = dbMapping.getKeywordsIdentifier();
        String result = "";
        if (StringUtils.isNotEmpty(dbMapping.getTargetDb())) {
            result += keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_PREFIX)
                    + dbMapping.getTargetDb()
                    + keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_SUFFIX)
                    + ".";
        }
        result += keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_PREFIX)
                + dbMapping.getTargetTable()
                + keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_SUFFIX);
        return result;
    }

XuDaojie · 2020-09-15T03:31:47Z

还有一处忘记修改了哦。
com.alibaba.otter.canal.client.adapter.rdb.support.SyncUtil

    public static String getDbTableName(MappingConfig.DbMapping dbMapping) {
        Map<String, String> keywordsIdentifier = dbMapping.getKeywordsIdentifier();
        String result = "";
        if (StringUtils.isNotEmpty(dbMapping.getTargetDb())) {
            result += keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_PREFIX)
                    + dbMapping.getTargetDb()
                    + keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_SUFFIX)
                    + ".";
        }
        result += keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_PREFIX)
                + dbMapping.getTargetTable()
                + keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_SUFFIX);
        return result;
    }

这里不是忘了，就是特意没改，通过调整配置处理的，以postgres为例

targetTable: date -> targetTable: '"date"'
targetTable: db.date -> targetTable: '"db"."date"'
上面那段代码里其实还少了段判断第2种情况的代码，但是有一个情况无法处理就是表名里就带了 .号的

yansheng105 · 2020-09-15T06:52:33Z

还有一处忘记修改了哦。
com.alibaba.otter.canal.client.adapter.rdb.support.SyncUtil
    public static String getDbTableName(MappingConfig.DbMapping dbMapping) {
        Map<String, String> keywordsIdentifier = dbMapping.getKeywordsIdentifier();
        String result = "";
        if (StringUtils.isNotEmpty(dbMapping.getTargetDb())) {
            result += keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_PREFIX)
                    + dbMapping.getTargetDb()
                    + keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_SUFFIX)
                    + ".";
        }
        result += keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_PREFIX)
                + dbMapping.getTargetTable()
                + keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_SUFFIX);
        return result;
    }
这里不是忘了，就是特意没改，通过调整配置处理的，以postgres为例

targetTable: date -> targetTable: '"date"'

targetTable: db.date -> targetTable: '"db"."date"'

上面那段代码里其实还少了段判断第2种情况的代码，但是有一个情况无法处理就是表名里就带了 .号的

第二种情况我认为应该使用targetDb和targetTable两个配置项来配置，这样表名带点号的也可以进行处理了。

题外话：
最近我在使用adapter远程配置的时候遇到修改application.yml会导致无限重启的问题，不知道您有没有遇到过？

XuDaojie · 2020-09-15T07:29:07Z

还有一处忘记修改了哦。
com.alibaba.otter.canal.client.adapter.rdb.support.SyncUtil
    public static String getDbTableName(MappingConfig.DbMapping dbMapping) {
        Map<String, String> keywordsIdentifier = dbMapping.getKeywordsIdentifier();
        String result = "";
        if (StringUtils.isNotEmpty(dbMapping.getTargetDb())) {
            result += keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_PREFIX)
                    + dbMapping.getTargetDb()
                    + keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_SUFFIX)
                    + ".";
        }
        result += keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_PREFIX)
                + dbMapping.getTargetTable()
                + keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_SUFFIX);
        return result;
    }
这里不是忘了，就是特意没改，通过调整配置处理的，以postgres为例

targetTable: date -> targetTable: '"date"'

targetTable: db.date -> targetTable: '"db"."date"'

上面那段代码里其实还少了段判断第2种情况的代码，但是有一个情况无法处理就是表名里就带了 .号的
第二种情况我认为应该使用targetDb和targetTable两个配置项来配置，这样表名带点号的也可以进行处理了。

题外话：
最近我在使用adapter远程配置的时候遇到修改application.yml会导致无限重启的问题，不知道您有没有遇到过？

你说的那个是可以那样处理，我也是习惯两个分开配置，这本来是可选项，但是代码里这样处理的话就变成必选项了

目前没遇到

yansheng105 · 2020-09-15T07:36:58Z

还有一处忘记修改了哦。
com.alibaba.otter.canal.client.adapter.rdb.support.SyncUtil
    public static String getDbTableName(MappingConfig.DbMapping dbMapping) {
        Map<String, String> keywordsIdentifier = dbMapping.getKeywordsIdentifier();
        String result = "";
        if (StringUtils.isNotEmpty(dbMapping.getTargetDb())) {
            result += keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_PREFIX)
                    + dbMapping.getTargetDb()
                    + keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_SUFFIX)
                    + ".";
        }
        result += keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_PREFIX)
                + dbMapping.getTargetTable()
                + keywordsIdentifier.get(RDBConstants.KEYWORDS_IDENTIFIER_SUFFIX);
        return result;
    }
这里不是忘了，就是特意没改，通过调整配置处理的，以postgres为例

targetTable: date -> targetTable: '"date"'

targetTable: db.date -> targetTable: '"db"."date"'

上面那段代码里其实还少了段判断第2种情况的代码，但是有一个情况无法处理就是表名里就带了 .号的
第二种情况我认为应该使用targetDb和targetTable两个配置项来配置，这样表名带点号的也可以进行处理了。
题外话：
最近我在使用adapter远程配置的时候遇到修改application.yml会导致无限重启的问题，不知道您有没有遇到过？
你说的那个是可以那样处理，我也是习惯两个分开配置，这本来是可选项，但是代码里这样处理的话就变成必选项了

目前没遇到

adapter的配置重载都有问题，application时改了就无限重启，rdb映射文件改了要重启才能生效，后面那个找到原因了，第一个还没找到

agapple · 2021-04-16T13:28:18Z

麻烦重新提交一个PR，代码有冲突

# Conflicts: # client-adapter/rdb/src/main/java/com/alibaba/otter/canal/client/adapter/rdb/support/SyncUtil.java

XuDaojie · 2021-04-19T15:10:15Z

麻烦重新提交一个PR，代码有冲突

已重新提交PR。

liminglin1995 · 2021-05-11T09:58:29Z

你好，我按你的分支改了源码重新编译后，adapter起不来，报这个错，能帮忙看下吗？
2021-05-11 17:56:00.841 [main] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## something goes wrong when starting up thecanal client adapters:
java.lang.NullPointerException: null
at com.alibaba.otter.canal.adapter.launcher.loader.AdapterProcessor.(AdapterProcessor.java:69) ~[client-adapter.launcher-1.1.6-SNAPSHOT.jar:na]
at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.lambda$init$0(CanalAdapterLoader.java:65) ~[client-adapter.launcher-1.1.6-SNAPSHOT.jar:na]
at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[na:1.8.0_251]
at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.init(CanalAdapterLoader.java:60) ~[client-adapter.launcher-1.1.6-SNAPSHOT.jar:na]
at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterService.init(CanalAdapterService.java:60) ~[client-adapter.launcher-1.1.6-SNAPSHOT.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_251]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_251]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_251]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_251]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:365) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:308) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:135) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:579) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$1(AbstractBeanFactory.java:353) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:390) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE]
at org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:184) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:350) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1089) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.cloud.context.scope.refresh.RefreshScope.eagerlyInitialize(RefreshScope.java:126) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE]
at org.springframework.cloud.context.scope.refresh.RefreshScope.start(RefreshScope.java:117) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_251]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_251]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_251]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_251]
at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:264) [spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:182) [spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:144)[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:400) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:354) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:888) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:161) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:327) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
at com.alibaba.otter.canal.adapter.launcher.CanalAdapterApplication.main(CanalAdapterApplication.java:19) ~[client-adapter.launcher-1.1.6-SNAPSHOT.jar:na]
2021-05-11 17:56:00.848 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"]
2021-05-11 17:56:00.853 [main] INFO org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2021-05-11 17:56:00.935 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path ''
2021-05-11 17:56:00.939 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 5.759 seconds (JVM running for 6.371)

XuDaojie · 2021-05-11T11:02:58Z

你好，我按你的分支改了源码重新编译后，adapter起不来，报这个错，能帮忙看下吗？
2021-05-11 17:56:00.841 [main] ERROR c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## something goes wrong when starting up thecanal client adapters:
java.lang.NullPointerException: null
at com.alibaba.otter.canal.adapter.launcher.loader.AdapterProcessor.(AdapterProcessor.java:69) ~[client-adapter.launcher-1.1.6-SNAPSHOT.jar:na]
at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.lambda$init$0(CanalAdapterLoader.java:65) ~[client-adapter.launcher-1.1.6-SNAPSHOT.jar:na]
at java.util.HashMap.computeIfAbsent(HashMap.java:1127) ~[na:1.8.0_251]
at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterLoader.init(CanalAdapterLoader.java:60) ~[client-adapter.launcher-1.1.6-SNAPSHOT.jar:na]
at com.alibaba.otter.canal.adapter.launcher.loader.CanalAdapterService.init(CanalAdapterService.java:60) ~[client-adapter.launcher-1.1.6-SNAPSHOT.jar:na]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_251]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_251]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_251]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_251]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:365) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:308) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:135) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:422) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:579) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:501) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$1(AbstractBeanFactory.java:353) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.cloud.context.scope.GenericScope$BeanLifecycleWrapper.getBean(GenericScope.java:390) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE]
at org.springframework.cloud.context.scope.GenericScope.get(GenericScope.java:184) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:350) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) [spring-beans-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1089) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.cloud.context.scope.refresh.RefreshScope.eagerlyInitialize(RefreshScope.java:126) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE]
at org.springframework.cloud.context.scope.refresh.RefreshScope.start(RefreshScope.java:117) ~[spring-cloud-context-2.0.0.RELEASE.jar:2.0.0.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_251]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_251]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_251]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_251]
at org.springframework.context.event.ApplicationListenerMethodAdapter.doInvoke(ApplicationListenerMethodAdapter.java:264) [spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.event.ApplicationListenerMethodAdapter.processEvent(ApplicationListenerMethodAdapter.java:182) ~~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.event.ApplicationListenerMethodAdapter.onApplicationEvent(ApplicationListenerMethodAdapter.java:144)~~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE] at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:400) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:354) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:888) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.finishRefresh(ServletWebServerApplicationContext.java:161) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:553) ~[spring-context-5.0.5.RELEASE.jar:5.0.5.RELEASE]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:140) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:395) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:327) ~[spring-boot-2.0.1.RELEASE.jar:2.0.1.RELEASE]
at com.alibaba.otter.canal.adapter.launcher.CanalAdapterApplication.main(CanalAdapterApplication.java:19) ~[client-adapter.launcher-1.1.6-SNAPSHOT.jar:na]
2021-05-11 17:56:00.848 [main] INFO org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"]
2021-05-11 17:56:00.853 [main] INFO org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2021-05-11 17:56:00.935 [main] INFO o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path ''
2021-05-11 17:56:00.939 [main] INFO c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 5.759 seconds (JVM running for 6.371)

应该是canal.conf.consumerProperties 你没配置，感觉你配置文件可能是拷贝的1.1.4的，这个参数应该是1.1.5新增的。

liminglin1995 · 2021-05-12T03:18:59Z

谢谢，就像你所说的，然后我按最新的1.1.5的模板来配置就可以了👍

liminglin1995 · 2021-05-25T03:16:36Z

hello，我升级到最新的版本用了你的分支，运行一段时间没什么问题，就是在canal重启的时候会丢数据，这个版本不能够断点重连吗？

XuDaojie · 2021-05-26T03:35:32Z

hello，我升级到最新的版本用了你的分支，运行一段时间没什么问题，就是在canal重启的时候会丢数据，这个版本不能够断点重连吗？

我这是没问题的，用zk存储的进度。

liminglin1995 · 2021-06-30T11:54:44Z

我引用的是canal.instance.global.spring.xml = classpath:spring/file-instance.xml这个配置，meta.dat在运行时记录消费位点信息，每次停止adapter，meta.dat的内容就变成了{"clientDatas":[],"destination":"example"}了，你知道为什么吗？

agapple · 2022-05-23T11:17:18Z

其他PR提交已经支持，#3984

fix: rdb 目标库名或表名带 - 或名字和关键字相同时抛异常

62543dd

fix: rdb-adapter 数据库目前不兼容postgres(alibaba#3026)

c08455c

XuDaojie changed the title ~~fix: rdb 目标库名或表名带 - 或名字和关键字相同时抛异常(#3019)~~ fix: v1.1.5-snapshot rdb-adapter 不兼容postgres(#3026) Sep 11, 2020

XuDaojie changed the title ~~fix: v1.1.5-snapshot rdb-adapter 不兼容postgres(#3026)~~ fix: v1.1.5-snapshot rdb-adapter 不兼容postgres Sep 11, 2020

update client-adapter README.md

c30ee85

XuDaojie changed the title ~~fix: v1.1.5-snapshot rdb-adapter 不兼容postgres~~ fix: v1.1.5-snapshot rdb-adapter 由关键字引起的异常 (#3026 #2783) Oct 6, 2020

update: keywordsIdentifier 修改为 targetKeywordsIdentifier

925a940

XuDaojie added 2 commits April 19, 2021 16:11

Merge branch 'master' into fix-issues-alibaba#3026

e1696f7

# Conflicts: # client-adapter/rdb/src/main/java/com/alibaba/otter/canal/client/adapter/rdb/support/SyncUtil.java

删除无用代码

ef8a6ed

agapple closed this May 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: v1.1.5-snapshot rdb-adapter 由关键字引起的异常 (#3026 #2783) #3020

fix: v1.1.5-snapshot rdb-adapter 由关键字引起的异常 (#3026 #2783) #3020

XuDaojie commented Sep 8, 2020 •

edited

Loading

CLAassistant commented Sep 8, 2020 •

edited

Loading

codecov-commenter commented Sep 8, 2020 •

edited

Loading

yansheng105 commented Sep 10, 2020

yansheng105 commented Sep 15, 2020

XuDaojie commented Sep 15, 2020

yansheng105 commented Sep 15, 2020

XuDaojie commented Sep 15, 2020

yansheng105 commented Sep 15, 2020

agapple commented Apr 16, 2021

XuDaojie commented Apr 19, 2021

liminglin1995 commented May 11, 2021

XuDaojie commented May 11, 2021 •

edited

Loading

liminglin1995 commented May 12, 2021

liminglin1995 commented May 25, 2021

XuDaojie commented May 26, 2021

liminglin1995 commented Jun 30, 2021

agapple commented May 23, 2022

fix: v1.1.5-snapshot rdb-adapter 由关键字引起的异常 (#3026 #2783) #3020

fix: v1.1.5-snapshot rdb-adapter 由关键字引起的异常 (#3026 #2783) #3020

Conversation

XuDaojie commented Sep 8, 2020 • edited Loading

问题描述

解决过程

新增配置

CLAassistant commented Sep 8, 2020 • edited Loading

codecov-commenter commented Sep 8, 2020 • edited Loading

Codecov Report

yansheng105 commented Sep 10, 2020

yansheng105 commented Sep 15, 2020

XuDaojie commented Sep 15, 2020

yansheng105 commented Sep 15, 2020

XuDaojie commented Sep 15, 2020

yansheng105 commented Sep 15, 2020

agapple commented Apr 16, 2021

XuDaojie commented Apr 19, 2021

liminglin1995 commented May 11, 2021

XuDaojie commented May 11, 2021 • edited Loading

liminglin1995 commented May 12, 2021

liminglin1995 commented May 25, 2021

XuDaojie commented May 26, 2021

liminglin1995 commented Jun 30, 2021

agapple commented May 23, 2022

XuDaojie commented Sep 8, 2020 •

edited

Loading

CLAassistant commented Sep 8, 2020 •

edited

Loading

codecov-commenter commented Sep 8, 2020 •

edited

Loading

XuDaojie commented May 11, 2021 •

edited

Loading