[Bug] [Connector-V2] Fix InfluxDB SQL invalide when SQL contains tz function #4283

zhengyuan-cn · 2023-03-06T05:32:50Z

[Bug] connector-influxdb dependency influxdb-java's version shuld be 2.15 not 2.21 #4230

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

zhengyuan-cn · 2023-03-06T05:46:52Z

[Bug] connector-influxdb module InfluxDBSource class's initColumnsIndex method, sql invalide when sql contains tz function cause by direct append QUERY_LIMIT #4231

commit id: [1d1afc5] (1d1afc5)

Hisoka-X · 2023-03-06T07:48:21Z

Hi, why downgraded version？IMO, we should fix a bug by upgrade version or code, not downgraded version.

zhengyuan-cn · 2023-03-06T08:03:27Z

hi, version 2.21 ,It's Happened NoSuchMethod when the dependency of influxdb-java's version 2.11 .
downgraded version to 2.15 it works correct.

exception detail:
23/02/27 10:50:06 INFO AbstractPluginDiscovery: Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='InfluxDB'} from classpath
Exception in thread "main" java.lang.NoSuchMethodError: kotlin.collections.ArraysKt.copyInto([B[BIII)[B
at shaded.okio.Segment.writeTo(Segment.kt:169)
at shaded.okio.Segment.compact(Segment.kt:152)
at shaded.okio.Buffer.write(Buffer.kt:1842)
at shaded.okio.Buffer.read(Buffer.kt:1854)
at shaded.okio.Buffer.writeAll(Buffer.kt:1642)
at shaded.okio.Options$Companion.buildTrieRecursive(Options.kt:187)
at shaded.okio.Options$Companion.buildTrieRecursive(Options.kt:174)
at shaded.okio.Options$Companion.buildTrieRecursive$default(Options.kt:113)
at shaded.okio.Options$Companion.of(Options.kt:72)
at shaded.okhttp3.internal.Util.(Util.kt:71)
at shaded.okhttp3.internal.concurrent.TaskRunner.(TaskRunner.kt:309)
at shaded.okhttp3.ConnectionPool.(ConnectionPool.kt:41)
at shaded.okhttp3.ConnectionPool.(ConnectionPool.kt:47)
at shaded.okhttp3.OkHttpClient$Builder.(OkHttpClient.kt:471)
at org.apache.seatunnel.connectors.seatunnel.influxdb.client.InfluxDBClient.getInfluxDB(InfluxDBClient.java:42)
at org.apache.seatunnel.connectors.seatunnel.influxdb.source.InfluxDBSource.prepare(InfluxDBSource.java:84)
at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.initializePlugins(SourceExecuteProcessor.java:88)
at org.apache.seatunnel.core.starter.spark.execution.AbstractPluginExecuteProcessor.(AbstractPluginExecuteProcessor.java:49)
at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.(SourceExecuteProcessor.java:51)
at org.apache.seatunnel.core.starter.spark.execution.SparkExecution.(SparkExecution.java:50)
at org.apache.seatunnel.core.starter.spark.command.SparkApiTaskExecuteCommand.execute(SparkApiTaskExecuteCommand.java:54)
at org.apache.seatunnel.core.starter.Seatunnel.run(Seatunnel.java:39)
at org.apache.seatunnel.core.starter.spark.SeatunnelSpark.main(SeatunnelSpark.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:855)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:930)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:939)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Hisoka-X · 2023-03-06T08:11:33Z

hi, version 2.21 ,It's Happened NoSuchMethod when the dependency of influxdb-java's version 2.11 . downgraded version to 2.15 it works correct.

exception detail: 23/02/27 10:50:06 INFO AbstractPluginDiscovery: Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='InfluxDB'} from classpath Exception in thread "main" java.lang.NoSuchMethodError: kotlin.collections.ArraysKt.copyInto([B[BIII)[B at shaded.okio.Segment.writeTo(Segment.kt:169) at shaded.okio.Segment.compact(Segment.kt:152) at shaded.okio.Buffer.write(Buffer.kt:1842) at shaded.okio.Buffer.read(Buffer.kt:1854) at shaded.okio.Buffer.writeAll(Buffer.kt:1642) at shaded.okio.Options$Companion.buildTrieRecursive(Options.kt:187) at shaded.okio.Options$Companion.buildTrieRecursive(Options.kt:174) at shaded.okio.Options$Companion.buildTrieRecursive$default(Options.kt:113) at shaded.okio.Options$Companion.of(Options.kt:72) at shaded.okhttp3.internal.Util.(Util.kt:71) at shaded.okhttp3.internal.concurrent.TaskRunner.(TaskRunner.kt:309) at shaded.okhttp3.ConnectionPool.(ConnectionPool.kt:41) at shaded.okhttp3.ConnectionPool.(ConnectionPool.kt:47) at shaded.okhttp3.OkHttpClient$Builder.(OkHttpClient.kt:471) at org.apache.seatunnel.connectors.seatunnel.influxdb.client.InfluxDBClient.getInfluxDB(InfluxDBClient.java:42) at org.apache.seatunnel.connectors.seatunnel.influxdb.source.InfluxDBSource.prepare(InfluxDBSource.java:84) at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.initializePlugins(SourceExecuteProcessor.java:88) at org.apache.seatunnel.core.starter.spark.execution.AbstractPluginExecuteProcessor.(AbstractPluginExecuteProcessor.java:49) at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.(SourceExecuteProcessor.java:51) at org.apache.seatunnel.core.starter.spark.execution.SparkExecution.(SparkExecution.java:50) at org.apache.seatunnel.core.starter.spark.command.SparkApiTaskExecuteCommand.execute(SparkApiTaskExecuteCommand.java:54) at org.apache.seatunnel.core.starter.Seatunnel.run(Seatunnel.java:39) at org.apache.seatunnel.core.starter.spark.SeatunnelSpark.main(SeatunnelSpark.java:34) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:855) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:930) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:939) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Are we use old api to access influxDB? Can we upgrade version and modify the way access influxDB?

zhengyuan-cn · 2023-03-06T09:31:10Z

hi, version 2.21 ,It's Happened NoSuchMethod when the dependency of influxdb-java's version 2.11 . downgraded version to 2.15 it works correct.
exception detail: 23/02/27 10:50:06 INFO AbstractPluginDiscovery: Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='InfluxDB'} from classpath Exception in thread "main" java.lang.NoSuchMethodError: kotlin.collections.ArraysKt.copyInto([B[BIII)[B at shaded.okio.Segment.writeTo(Segment.kt:169) at shaded.okio.Segment.compact(Segment.kt:152) at shaded.okio.Buffer.write(Buffer.kt:1842) at shaded.okio.Buffer.read(Buffer.kt:1854) at shaded.okio.Buffer.writeAll(Buffer.kt:1642) at shaded.okio.Options$Companion.buildTrieRecursive(Options.kt:187) at shaded.okio.Options$Companion.buildTrieRecursive(Options.kt:174) at shaded.okio.Options$Companion.buildTrieRecursive$default(Options.kt:113) at shaded.okio.Options$Companion.of(Options.kt:72) at shaded.okhttp3.internal.Util.(Util.kt:71) at shaded.okhttp3.internal.concurrent.TaskRunner.(TaskRunner.kt:309) at shaded.okhttp3.ConnectionPool.(ConnectionPool.kt:41) at shaded.okhttp3.ConnectionPool.(ConnectionPool.kt:47) at shaded.okhttp3.OkHttpClient$Builder.(OkHttpClient.kt:471) at org.apache.seatunnel.connectors.seatunnel.influxdb.client.InfluxDBClient.getInfluxDB(InfluxDBClient.java:42) at org.apache.seatunnel.connectors.seatunnel.influxdb.source.InfluxDBSource.prepare(InfluxDBSource.java:84) at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.initializePlugins(SourceExecuteProcessor.java:88) at org.apache.seatunnel.core.starter.spark.execution.AbstractPluginExecuteProcessor.(AbstractPluginExecuteProcessor.java:49) at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.(SourceExecuteProcessor.java:51) at org.apache.seatunnel.core.starter.spark.execution.SparkExecution.(SparkExecution.java:50) at org.apache.seatunnel.core.starter.spark.command.SparkApiTaskExecuteCommand.execute(SparkApiTaskExecuteCommand.java:54) at org.apache.seatunnel.core.starter.Seatunnel.run(Seatunnel.java:39) at org.apache.seatunnel.core.starter.spark.SeatunnelSpark.main(SeatunnelSpark.java:34) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:855) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:930) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:939) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Are we use old api to access influxDB? Can we upgrade version and modify the way access influxDB?

I use influxDB version 1.8.10, seatunnel-influxDB use simple config. as below
`InfluxDB {
url="http://10.x.x.22:8086"
database="data_storage"
password="xxxxxx"
sql="select temp3,time from lunzi where time >= '2023-01-11 16:18:00' and time < '2023-01-11 16:19:00' tz('Asia/Shanghai')"
fields {
temp3 = float
time = bigint
}

result_table_name="source1"

}`

zhengyuan-cn · 2023-03-06T09:35:35Z

hi, version 2.21 ,It's Happened NoSuchMethod when the dependency of influxdb-java's version 2.11 . downgraded version to 2.15 it works correct.
exception detail: 23/02/27 10:50:06 INFO AbstractPluginDiscovery: Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='InfluxDB'} from classpath Exception in thread "main" java.lang.NoSuchMethodError: kotlin.collections.ArraysKt.copyInto([B[BIII)[B at shaded.okio.Segment.writeTo(Segment.kt:169) at shaded.okio.Segment.compact(Segment.kt:152) at shaded.okio.Buffer.write(Buffer.kt:1842) at shaded.okio.Buffer.read(Buffer.kt:1854) at shaded.okio.Buffer.writeAll(Buffer.kt:1642) at shaded.okio.Options$Companion.buildTrieRecursive(Options.kt:187) at shaded.okio.Options$Companion.buildTrieRecursive(Options.kt:174) at shaded.okio.Options$Companion.buildTrieRecursive$default(Options.kt:113) at shaded.okio.Options$Companion.of(Options.kt:72) at shaded.okhttp3.internal.Util.(Util.kt:71) at shaded.okhttp3.internal.concurrent.TaskRunner.(TaskRunner.kt:309) at shaded.okhttp3.ConnectionPool.(ConnectionPool.kt:41) at shaded.okhttp3.ConnectionPool.(ConnectionPool.kt:47) at shaded.okhttp3.OkHttpClient$Builder.(OkHttpClient.kt:471) at org.apache.seatunnel.connectors.seatunnel.influxdb.client.InfluxDBClient.getInfluxDB(InfluxDBClient.java:42) at org.apache.seatunnel.connectors.seatunnel.influxdb.source.InfluxDBSource.prepare(InfluxDBSource.java:84) at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.initializePlugins(SourceExecuteProcessor.java:88) at org.apache.seatunnel.core.starter.spark.execution.AbstractPluginExecuteProcessor.(AbstractPluginExecuteProcessor.java:49) at org.apache.seatunnel.core.starter.spark.execution.SourceExecuteProcessor.(SourceExecuteProcessor.java:51) at org.apache.seatunnel.core.starter.spark.execution.SparkExecution.(SparkExecution.java:50) at org.apache.seatunnel.core.starter.spark.command.SparkApiTaskExecuteCommand.execute(SparkApiTaskExecuteCommand.java:54) at org.apache.seatunnel.core.starter.Seatunnel.run(Seatunnel.java:39) at org.apache.seatunnel.core.starter.spark.SeatunnelSpark.main(SeatunnelSpark.java:34) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52) at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:855) at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161) at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184) at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86) at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:930) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:939) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Are we use old api to access influxDB? Can we upgrade version and modify the way access influxDB?

I tested influxdb-java version 2.21-2.23 them not support, the Exception still .

Hisoka-X · 2023-03-06T09:40:49Z

I tested influxdb-java version 2.21-2.23 them not support, the Exception still .

Maybe we should update the code which SeaTunnel access InfluxDB.

zhengyuan-cn · 2023-03-09T01:47:24Z

I tested influxdb-java version 2.21-2.23 them not support, the Exception still .

Maybe we should update the code which SeaTunnel access InfluxDB.
hi, code in this commit, ,for bug #4231
[Bug] connector-influxdb module InfluxDBSource class's initColumnsIndex method, sql invalide when sql contains tz function cause by direct append QUERY_LIMIT #4231

zhengyuan-cn · 2023-03-15T02:42:03Z

The failure check (Backend / unit-test (11, ubuntu-latest) need resolve? how can I resolve it?

Hisoka-X · 2023-03-15T02:56:04Z

Try merge from dev.

seatunnel-connectors-v2/connector-influxdb/pom.xml

Hisoka-X · 2023-03-17T02:38:42Z

Hi, can you help add this test case (SQL contains tz function) to InfluxDB E2E test?

zhengyuan-cn · 2023-03-17T02:42:00Z

Ok

hailin0 · 2023-05-19T09:36:26Z

...ector-influxdb-e2e/src/test/java/org/apache/seatunnel/e2e/connector/influxdb/InfluxdbIT.java

+
+    @TestTemplate
+    public void testInfluxdbSqlTest(TestContainer container) throws IOException, InterruptedException {
+        Container.ExecResult execResult = container.executeJob("/influxdb-to-influxdb-sqltest.conf");


un commit this file？

influxdb-to-influxdb-sqltest.conf

@zhengyuan-cn

sorry I forgot commit it,

Now, commited

hailin0 · 2023-05-19T10:25:58Z

check ci error

zhengyuan-cn · 2023-06-25T01:31:36Z

recommited RP see: #4829

Hisoka-X added First-time contributor First-time contributor connectors-v2 labels Mar 6, 2023

Hisoka-X linked an issue Mar 6, 2023 that may be closed by this pull request

[Bug] connector-influxdb module InfluxDBSource class's initColumnsIndex method, sql invalide when sql contains tz function cause by direct append QUERY_LIMIT #4231

Open

3 tasks

Hisoka-X added the bug label Mar 6, 2023

Hisoka-X reviewed Mar 16, 2023

View reviewed changes

seatunnel-connectors-v2/connector-influxdb/pom.xml Outdated Show resolved Hide resolved

Hisoka-X changed the title ~~[Bug] connector-influxdb dependency influxdb-java's version shuld be …~~ [Bug] [Connector-V2] Fix InfluxDB SQL invalide when SQL contains tz function Mar 16, 2023

hailin0 reviewed May 19, 2023

View reviewed changes

hailin0 closed this May 19, 2023

hailin0 reopened this May 19, 2023

zhengyuan-cn closed this May 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug] [Connector-V2] Fix InfluxDB SQL invalide when SQL contains tz function #4283

[Bug] [Connector-V2] Fix InfluxDB SQL invalide when SQL contains tz function #4283

zhengyuan-cn commented Mar 6, 2023 •

edited

Loading

zhengyuan-cn commented Mar 6, 2023

Hisoka-X commented Mar 6, 2023

zhengyuan-cn commented Mar 6, 2023

Hisoka-X commented Mar 6, 2023

zhengyuan-cn commented Mar 6, 2023

zhengyuan-cn commented Mar 6, 2023

Hisoka-X commented Mar 6, 2023

zhengyuan-cn commented Mar 9, 2023

zhengyuan-cn commented Mar 15, 2023

Hisoka-X commented Mar 15, 2023

Hisoka-X commented Mar 17, 2023

zhengyuan-cn commented Mar 17, 2023

hailin0 May 19, 2023

hailin0 May 19, 2023

zhengyuan-cn May 23, 2023

hailin0 commented May 19, 2023

zhengyuan-cn commented Jun 25, 2023

[Bug] [Connector-V2] Fix InfluxDB SQL invalide when SQL contains tz function #4283

[Bug] [Connector-V2] Fix InfluxDB SQL invalide when SQL contains tz function #4283

Conversation

zhengyuan-cn commented Mar 6, 2023 • edited Loading

Purpose of this pull request

Check list

zhengyuan-cn commented Mar 6, 2023

Hisoka-X commented Mar 6, 2023

zhengyuan-cn commented Mar 6, 2023

Hisoka-X commented Mar 6, 2023

zhengyuan-cn commented Mar 6, 2023

zhengyuan-cn commented Mar 6, 2023

Hisoka-X commented Mar 6, 2023

zhengyuan-cn commented Mar 9, 2023

zhengyuan-cn commented Mar 15, 2023

Hisoka-X commented Mar 15, 2023

Hisoka-X commented Mar 17, 2023

zhengyuan-cn commented Mar 17, 2023

hailin0 May 19, 2023

Choose a reason for hiding this comment

hailin0 May 19, 2023

Choose a reason for hiding this comment

zhengyuan-cn May 23, 2023

Choose a reason for hiding this comment

hailin0 commented May 19, 2023

zhengyuan-cn commented Jun 25, 2023

zhengyuan-cn commented Mar 6, 2023 •

edited

Loading