Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

增量同步(根据时间戳进行同步),变量替换{${lastTime}) ,(${currentTime})时,sql执行出现问题。 #198

Closed
kanbuq opened this issue Jul 9, 2020 · 11 comments
Labels
question Further information is requested

Comments

@kanbuq
Copy link

kanbuq commented Jul 9, 2020

具体问题描述为,[读取数据库数据失败. 请检查您的配置的 column/table/where/querySql或者向 DBA 寻求帮助.]. - 执行的SQL为: select * from emp where FROM_UNIXTIME(operationDate) >= FROM_UNIXTIME(${lastTime}) and operationDate < FROM_UNIXTIME(${currentTime}) 具体错误信息为:com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '{lastTime}) and operationDate < FROM_UNIXTIME(${currentTime})' ,
增量时间字段为:-D${lastTime}=%s -D${currentTime}=%s
json字段为:
"querySql": [
"select * from emp where FROM_UNIXTIME(operationDate) >= FROM_UNIXTIME(${lastTime}) and operationDate < FROM_UNIXTIME(${currentTime})"
]

@WeiYe-Jing
Copy link
Owner

WeiYe-Jing commented Jul 9, 2020

@WeiYe-Jing WeiYe-Jing added the question Further information is requested label Jul 9, 2020
Repository owner deleted a comment from kanbuq Jul 9, 2020
@WeiYe-Jing
Copy link
Owner

配置可以截个图看看

Repository owner deleted a comment from kanbuq Jul 9, 2020
@kanbuq
Copy link
Author

kanbuq commented Jul 9, 2020

本机的具体配置如下所示

24049333-5c41001a5a5e1c85

@WeiYe-Jing
Copy link
Owner

WeiYe-Jing commented Jul 9, 2020

image
上面发你的文档看样子你并没有去看,https://github.com/WeiYe-Jing/datax-web/blob/master/doc/datax-web/time-increment-desc.md
文档截图中的参数不是这样配置的,一点要参考文档

image

@kanbuq
Copy link
Author

kanbuq commented Jul 9, 2020

问题得到了解决,最后增量时间字段为:-DlastTime='%s' -DcurrentTime='%s',需要记得加冒号,下面的sql语句为
"select * from emp where operationDate >= FROM_UNIXTIME(${lastTime}) and operationDate < FROM_UNIXTIME(${currentTime})",字段名一致,感谢。

@fu-xin
Copy link

fu-xin commented Sep 6, 2020

oracle转mysql,oracle日期字段类型为date,mysql为datetime类型,增量迁移 报类型转换错误

2020-09-06 12:43:36.665 [0-0-0-reader] INFO CommonRdbmsReader$Task - Finished read record by Sql: [select * from agent."TB_TESTX" where updated_at >= '1599365051' / (60 * 60 * 24) + TO_DATE('1970-01-01 08:00:00', 'YYYY-MM-DD HH:MI:SS') and updated_at < '1599367409' / (60 * 60 * 24) + TO_DATE('1970-01-01 08:00:00', 'YYYY-MM-DD HH:MI:SS')
2020-09-06 12:43:36 [AnalysisStatistics.analysisStatisticsLog-53] ] jdbcUrl:[jdbc:oracle:thin:@//192.168.2.90:1521/ORCL].
2020-09-06 12:43:37 [AnalysisStatistics.analysisStatisticsLog-53] 2020-09-06 12:43:37.567 [0-0-0-writer] WARN CommonRdbmsWriter$Task - 回滚此次写入, 采用每次写入一行方式提交. 因为:TIMESTAMP 类型转换错误:[{"byteSize":7,"rawData":"testXY1","type":"STRING"}]
2020-09-06 12:43:37 [AnalysisStatistics.analysisStatisticsLog-53] 2020-09-06 12:43:37.668 [0-0-0-writer] ERROR StdoutPluginCollector -
2020-09-06 12:43:37 [AnalysisStatistics.analysisStatisticsLog-53] java.sql.SQLException: TIMESTAMP 类型转换错误:[{"byteSize":7,"rawData":"testXY1","type":"STRING"}]
2020-09-06 12:43:37 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.fillPreparedStatementColumnType(CommonRdbmsWriter.java:497) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
2020-09-06 12:43:37 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.fillPreparedStatement(CommonRdbmsWriter.java:405) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
2020-09-06 12:43:37 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.doOneInsert(CommonRdbmsWriter.java:380) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
2020-09-06 12:43:37 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.doBatchInsert(CommonRdbmsWriter.java:362) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
2020-09-06 12:43:37 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWriteWithConnection(CommonRdbmsWriter.java:297) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
2020-09-06 12:43:37 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.rdbms.writer.CommonRdbmsWriter$Task.startWrite(CommonRdbmsWriter.java:319) [plugin-rdbms-util-0.0.1-SNAPSHOT.jar:na]
2020-09-06 12:43:37 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.plugin.writer.mysqlwriter.MysqlWriter$Task.startWrite(MysqlWriter.java:78) [mysqlwriter-0.0.1-SNAPSHOT.jar:na]
2020-09-06 12:43:37 [AnalysisStatistics.analysisStatisticsLog-53] at com.alibaba.datax.core.taskgroup.runner.WriterRunner.run(WriterRunner.java:56) [datax-core-0.0.1-SNAPSHOT.jar:na]
2020-09-06 12:43:37 [AnalysisStatistics.analysisStatisticsLog-53] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_231]
2020-09-06 12:43:37 [AnalysisStatistics.analysisStatisticsLog-53] 2020-09-06 12:43:37.671 [0-0-0-writer] ERROR StdoutPluginCollector - 脏数据:
2020-09-06 12:43:37 [AnalysisStatistics.analysisStatisticsLog-53] {"exception":"TIMESTAMP 类型转换错误:[{"byteSize":7,"rawData":"testXY1","type":"STRING"}]","record":[{"byteSize":1,"index":0,"rawData":"1","type":"DOUBLE"},{"byteSize":7,"index":1,"rawData":"testXY1","type":"STRING"},{"byteSize":8,"index":2,"rawData":1599358896000,"type":"DATE"},{"byteSize":8,"index":3,"rawData":1599365529000,"type":"DATE"}],"type":"writer"}
2020-09-06 12:43:37 [AnalysisStatistics.analysisStatisticsLog-53] 2020-09-06 12:43:37.723 [taskGroup-0] INFO TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[1214]ms

@CLinPhone
Copy link

Snipaste_2020-09-15_11-13-42
像数据表分表的配置,如何在json配置中实现

@WeiYe-Jing
Copy link
Owner

Snipaste_2020-09-15_11-13-42
像数据表分表的配置,如何在json配置中实现

麻烦重新新建一个issue哈,方便让更多开发者参与讨论,这个主题是增量,别人很难看到下面在讨论这么有吸引力的功能。

@meihuaA0010
Copy link

hbase的增量任务‘增量时间段’这里需要怎么配置

@renjiutong
Copy link

com.wugui.datax.rpc.util.XxlRpcException: xxl-rpc response status invalid.

@JavaLearner
Copy link

JavaLearner commented Dec 29, 2021

你好,我这边创建任务,在配置where条件的时候加上了条件,但在生成JSON文件的时候,并没有加上where条件,请问是什么原因呢
image

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

7 participants