Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][plugin] Add snowflake datasource in datasource plugin #13729

Merged
merged 23 commits into from
May 6, 2023
Merged
Show file tree
Hide file tree
Changes from 11 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
6bcafee
[Feature][plugin] Add snowflake datasource in datasource plugin
zhaohehuhu Mar 13, 2023
cd4af90
[Feature][ui] add snowflake datasource plugin
tongwl Nov 21, 2022
7cc88ca
apply spotless
zhaohehuhu Mar 14, 2023
6c4251d
[Feature][plugin] Add snowflake datasource in datasource plugin
zhaohehuhu Mar 13, 2023
8a0d208
[Feature][ui] add snowflake datasource plugin
tongwl Nov 21, 2022
19b6b4b
apply spotless
zhaohehuhu Mar 14, 2023
6355fb5
Merge remote-tracking branch 'origin/Feature-13727' into Feature-13727
zhaohehuhu Mar 16, 2023
6bc1896
fix snowflake dependency
zhaohehuhu Mar 17, 2023
29dffda
Add snowflake license
zhaohehuhu Mar 17, 2023
5585abc
delete extra snowflake dependency
zhaohehuhu Mar 17, 2023
ed97a09
add more UT
zhaohehuhu Mar 21, 2023
9896fc6
delete duplicated version
zhaohehuhu Mar 27, 2023
47b9843
Merge branch 'dev' of github.com:apache/dolphinscheduler into Feature…
zhaohehuhu Mar 31, 2023
2cc2824
add doc link
zhaohehuhu Mar 31, 2023
fbf89c1
reformat code
zhaohehuhu Apr 10, 2023
b4b6904
Merge remote-tracking branch 'ds/dev' into Feature-13727
zhaohehuhu Apr 11, 2023
4bbe2d9
add more UTs
zhaohehuhu Apr 11, 2023
58e4962
Merge remote-tracking branch 'ds/dev' into Feature-13727
zhaohehuhu Apr 23, 2023
6245e2f
add UTs
zhaohehuhu Apr 23, 2023
a4b04ca
add a blank after SNOWFLAKE
zhaohehuhu Apr 24, 2023
97af859
Merge remote-tracking branch 'ds/dev' into Feature-13727
zhaohehuhu Apr 26, 2023
4f1d478
fix UT
zhaohehuhu Apr 27, 2023
2da6fd5
Merge remote-tracking branch 'ds/dev' into Feature-13727
zhaohehuhu May 6, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions docs/docs/en/guide/datasource/snowflake.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# SNOWFLAKE

## Use Snowflake

![snowflake](../../../../img/new_ui/dev/datasource/snowflake.png)

## Datasource Parameters

| **Datasource** | **Description** |
|----------------------------|--------------------------------------------------------------|
| Datasource | Select SNOWFLAKE. |
| Datasource name | Enter the name of the DataSource. |
| Description | Enter a description of the DataSource. |
| IP/Host Name | Enter the SNOWFLAKE service IP. |
| Port | Enter the SNOWFLAKE service port. |
| Username | Set the username for SNOWFLAKE connection. |
| Password | Set the password for SNOWFLAKE connection. |
| Database name | Enter the database name of the SNOWFLAKE connection. |
| Datawarehouse name | Enter the datawarehouse name of the SNOWFLAKE connection. |
| Jdbc connection parameters | Parameter settings for SNOWFLAKE connection, in JSON format. |

17 changes: 17 additions & 0 deletions docs/docs/zh/guide/datasource/snowflake.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# SNOWFLAKE数据源

![snowflake](../../../../img/new_ui/dev/datasource/snowflake.png)

## 数据源参数

- 数据源:选择 SNOWFLAKE数据源
- 数据源名称:输入数据源的名称
- 描述:输入数据源的描述
- IP 主机名:输入连接 SNOWFLAKE数据源 的 IP
- 端口:输入连接 SNOWFLAKE数据源 的端口
- 用户名:设置连接 SNOWFLAKE数据源 的用户名
- 密码:设置连接 SNOWFLAKE数据源 的密码
- 数据库名:输入连接 SNOWFLAKE数据源 的数据库名称
- 数据仓库: 输入数据 SNOWFLAKE的数据仓库名称
- Jdbc 连接参数:用于 SNOWFLAKE数据源 连接的参数设置,以 JSON 形式填写

Binary file added docs/img/new_ui/dev/datasource/snowflake.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
9 changes: 9 additions & 0 deletions dolphinscheduler-bom/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,10 @@
<aws-java-sdk-redshift.version>1.12.300</aws-java-sdk-redshift.version>
<azure-resourcemanager.version>2.21.0</azure-resourcemanager.version>
<azure-resourcemanager-datafactory.version>1.0.0-beta.19</azure-resourcemanager-datafactory.version>
<snowflake-jdbc.version>3.13.10</snowflake-jdbc.version>
<google-cloud-storage.version>2.18.0</google-cloud-storage.version>
<snowflake-jdbc.version>3.13.10</snowflake-jdbc.version>

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Duplicated.

</properties>

<dependencyManagement>
Expand Down Expand Up @@ -829,6 +832,12 @@
<version>${aws-java-sdk-redshift.version}</version>
<scope>test</scope>
</dependency>

<dependency>
<groupId>net.snowflake</groupId>
<artifactId>snowflake-jdbc</artifactId>
<version>${snowflake-jdbc.version}</version>
</dependency>
</dependencies>
</dependencyManagement>
<profiles>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@ public class DataSourceConstants {
public static final String COM_TRINO_JDBC_DRIVER = "io.trino.jdbc.TrinoDriver";
public static final String COM_DAMENG_JDBC_DRIVER = "dm.jdbc.driver.DmDriver";
public static final String COM_OCEANBASE_JDBC_DRIVER = "com.oceanbase.jdbc.Driver";

public static final String NET_SNOWFLAKE_JDBC_DRIVER = "net.snowflake.client.jdbc.SnowflakeDriver";
/**
* validation Query
*/
Expand All @@ -54,6 +54,7 @@ public class DataSourceConstants {
public static final String ATHENA_VALIDATION_QUERY = "select 1";
public static final String TRINO_VALIDATION_QUERY = "select 1";
public static final String DAMENG_VALIDATION_QUERY = "select 1";
public static final String SNOWFLAKE_VALIDATION_QUERY = "select 1";

/**
* jdbc url
Expand All @@ -73,6 +74,7 @@ public class DataSourceConstants {
public static final String JDBC_TRINO = "jdbc:trino://";
public static final String JDBC_DAMENG = "jdbc:dm://";
public static final String JDBC_OCEANBASE = "jdbc:oceanbase://";
public static final String JDBC_SNOWFLAKE = "jdbc:snowflake://";

/**
* database type
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -107,5 +107,10 @@
<artifactId>dolphinscheduler-datasource-dameng</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-snowflake</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ Licensed to the Apache Software Foundation (ASF) under one or more
~ contributor license agreements. See the NOTICE file distributed with
~ this work for additional information regarding copyright ownership.
~ The ASF licenses this file to You under the Apache License, Version 2.0
~ (the "License"); you may not use this file except in compliance with
~ the License. You may obtain a copy of the License at
~
~ http://www.apache.org/licenses/LICENSE-2.0
~
~ Unless required by applicable law or agreed to in writing, software
~ distributed under the License is distributed on an "AS IS" BASIS,
~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
~ See the License for the specific language governing permissions and
~ limitations under the License.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-plugin</artifactId>
<version>dev-SNAPSHOT</version>
</parent>

<artifactId>dolphinscheduler-datasource-snowflake</artifactId>
<packaging>jar</packaging>
<name>${project.artifactId}</name>

<dependencies>

<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-spi</artifactId>
<scope>provided</scope>
</dependency>

<dependency>
<groupId>org.apache.dolphinscheduler</groupId>
<artifactId>dolphinscheduler-datasource-api</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>net.snowflake</groupId>
<artifactId>snowflake-jdbc</artifactId>
</dependency>
</dependencies>
</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.dolphinscheduler.plugin.datasource.snowflake;

import org.apache.dolphinscheduler.spi.datasource.BaseConnectionParam;
import org.apache.dolphinscheduler.spi.datasource.DataSourceChannel;
import org.apache.dolphinscheduler.spi.datasource.DataSourceClient;
import org.apache.dolphinscheduler.spi.enums.DbType;

public class SnowflakeDataSourceChannel implements DataSourceChannel {

@Override
public DataSourceClient createDataSourceClient(BaseConnectionParam baseConnectionParam, DbType dbType) {
return new SnowflakeDataSourceClient(baseConnectionParam, dbType);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.dolphinscheduler.plugin.datasource.snowflake;

import org.apache.dolphinscheduler.spi.datasource.DataSourceChannel;
import org.apache.dolphinscheduler.spi.datasource.DataSourceChannelFactory;

import com.google.auto.service.AutoService;

@AutoService(DataSourceChannelFactory.class)
public class SnowflakeDataSourceChannelFactory implements DataSourceChannelFactory {

@Override
public String getName() {
return "snowflake";
}

@Override
public DataSourceChannel create() {
return new SnowflakeDataSourceChannel();
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.dolphinscheduler.plugin.datasource.snowflake;

import org.apache.dolphinscheduler.plugin.datasource.api.client.CommonDataSourceClient;
import org.apache.dolphinscheduler.spi.datasource.BaseConnectionParam;
import org.apache.dolphinscheduler.spi.enums.DbType;

public class SnowflakeDataSourceClient extends CommonDataSourceClient {

public SnowflakeDataSourceClient(BaseConnectionParam baseConnectionParam, DbType dbType) {
super(baseConnectionParam, dbType);
}

}
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.dolphinscheduler.plugin.datasource.snowflake.param;

import org.apache.dolphinscheduler.spi.datasource.BaseConnectionParam;

public class SnowflakeConnectionParam extends BaseConnectionParam {

@Override
public String toString() {
return "SnowFlakeConnectionParam{"
+ "user='" + user + '\''
+ ", password='" + password + '\''
+ ", address='" + address + '\''
+ ", database='" + database + '\''
+ ", jdbcUrl='" + jdbcUrl + '\''
+ ", driverLocation='" + driverLocation + '\''
+ ", driverClassName='" + driverClassName + '\''
+ ", validationQuery='" + validationQuery + '\''
+ ", other='" + other + '\''
+ '}';
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.dolphinscheduler.plugin.datasource.snowflake.param;

import org.apache.dolphinscheduler.plugin.datasource.api.datasource.BaseDataSourceParamDTO;
import org.apache.dolphinscheduler.spi.enums.DbType;

public class SnowflakeDatasourceParamDTO extends BaseDataSourceParamDTO {

@Override
public String toString() {
return "SnowFlakeDatasourceParamDTO{"
+ "name='" + name + '\''
+ ", note='" + note + '\''
+ ", host='" + host + '\''
+ ", port=" + port
+ ", database='" + database + '\''
+ ", userName='" + userName + '\''
+ ", password='" + password + '\''
+ ", other='" + other + '\''
+ '}';
}

@Override
public DbType getType() {
return DbType.SNOWFLAKE;
}
}
Loading