Skip to content

Commit

Permalink
Fixed merge conflict
Browse files Browse the repository at this point in the history
  • Loading branch information
etsybaev committed Mar 23, 2022
1 parent 83e03a5 commit 9eddce9
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 4 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -21,10 +21,8 @@
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.time.LocalTime;
import java.util.Base64;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Properties;
import java.util.concurrent.TimeUnit;
Expand All @@ -38,7 +36,7 @@
public class SnowflakeDatabase {

private static final Logger LOGGER = LoggerFactory.getLogger(SnowflakeDatabase.class);
private static final int PAUSE_BETWEEN_TOKEN_REFRESH_MIN = 7;
private static final int PAUSE_BETWEEN_TOKEN_REFRESH_MIN = 7; // snowflake access token's TTL is 10min and can't be modified

private static final Duration NETWORK_TIMEOUT = Duration.ofMinutes(1);
private static final Duration QUERY_TIMEOUT = Duration.ofHours(3);
Expand Down Expand Up @@ -86,7 +84,7 @@ private static HikariDataSource createDataSource(final JsonNode config) {

// thread to keep the refresh token up to date
SnowflakeDestination.SCHEDULED_EXECUTOR_SERVICE.schedule(getRefreshTokenTask(dataSource),
PAUSE_BETWEEN_TOKEN_REFRESH_MIN, TimeUnit.SECONDS);
PAUSE_BETWEEN_TOKEN_REFRESH_MIN, TimeUnit.MINUTES);

} else if (credentials != null && credentials.has("password")) {
LOGGER.info("User/password login mode is used");
Expand Down
17 changes: 17 additions & 0 deletions docs/integrations/destinations/snowflake.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,9 @@ To use Azure Blob Storage, you will need to [create a storage account](https://d

Navigate to the Airbyte UI to set up Snowflake as a destination. You'll need the following information to configure the Snowflake destination:

#### There are 2 way ways of oauth supported: login\pass and oauth2.

### Login and Password
| Field | Description |
|---|---|
| [Host](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html) | The host domain of the snowflake instance (must include the account, region, cloud environment, and end with snowflakecomputing.com). Example: `accountname.us-east-2.aws.snowflakecomputing.com` |
Expand All @@ -153,6 +156,20 @@ Navigate to the Airbyte UI to set up Snowflake as a destination. You'll need the
| Password | The password associated with the username. |
| [JDBC URL Params](https://docs.snowflake.com/en/user-guide/jdbc-parameters.html) (Optional) | Additional properties to pass to the JDBC URL string when connecting to the database formatted as `key=value` pairs separated by the symbol `&`. Example: `key1=value1&key2=value2&key3=value3` |


### OAuth 2.0
Field | Description |
|---|---|
| [Host](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html) | The host domain of the snowflake instance (must include the account, region, cloud environment, and end with snowflakecomputing.com). Example: `accountname.us-east-2.aws.snowflakecomputing.com` |
| [Role](https://docs.snowflake.com/en/user-guide/security-access-control-overview.html#roles) | The role you created in Step 1 for Airbyte to access Snowflake. Example: `AIRBYTE_ROLE` |
| [Warehouse](https://docs.snowflake.com/en/user-guide/warehouses-overview.html#overview-of-warehouses) | The warehouse you created in Step 1 for Airbyte to sync data into. Example: `AIRBYTE_WAREHOUSE` |
| [Database](https://docs.snowflake.com/en/sql-reference/ddl-database.html#database-schema-share-ddl) | The database you created in Step 1 for Airbyte to sync data into. Example: `AIRBYTE_DATABASE` |
| [Schema](https://docs.snowflake.com/en/sql-reference/ddl-database.html#database-schema-share-ddl) | The default schema used as the target schema for all statements issued from the connection that do not explicitly specify a schema name. |
| Username | The username you created in Step 1 to allow Airbyte to access the database. Example: `AIRBYTE_USER` |
| OAuth2 | The Login name and password to obtain auth token. |
| [JDBC URL Params](https://docs.snowflake.com/en/user-guide/jdbc-parameters.html) (Optional) | Additional properties to pass to the JDBC URL string when connecting to the database formatted as `key=value` pairs separated by the symbol `&`. Example: `key1=value1&key2=value2&key3=value3` |


To use AWS S3 as the cloud storage, enter the information for the S3 bucket you created in Step 2:

| Field | Description |
Expand Down

1 comment on commit 9eddce9

@github-actions
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SonarQube Report

SonarQube report for Airbyte Connectors Destination Snowflake(#11093)

Measures

Name Value Name Value Name Value
Quality Gate Status ERROR Lines to Cover 186 Security Rating A
Lines of Code 1265 Coverage 0.0 Reliability Rating A
Vulnerabilities 0 Bugs 1 Duplicated Lines (%) 4.3
Duplicated Blocks 2 Code Smells 12 Blocker Issues 0
Critical Issues 10 Major Issues 20 Minor Issues 4

Detected Issues

Rule File Description Message
common-java:DuplicatedBlocks (MAJOR) snowflake/SnowflakeInternalStagingDestination.java Source files should not have any duplicated blocks 1 duplicated blocks of code must be removed.
java:S1192 (CRITICAL) snowflake/SnowflakeInternalStagingSqlOperations.java:91 String literals should not be duplicated Define a constant instead of duplicating this literal "stage" 5 times.
java:S112 (MAJOR) snowflake/SnowflakeInternalStagingSqlOperations.java:96 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S1192 (CRITICAL) snowflake/SnowflakeInternalStagingSqlOperations.java:98 String literals should not be duplicated Define a constant instead of duplicating this literal "Executing query: {}" 6 times.
java:S112 (MAJOR) snowflake/SnowflakeInternalStagingSqlOperations.java:102 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S3457 (MAJOR) snowflake/SnowflakeInternalStagingSqlOperations.java:163 Printf-style format strings should be used correctly Format specifiers should be used instead of string concatenation.
java:S1155 (MINOR) snowflake/SnowflakeSqlOperations.java:66 Collection.isEmpty() should be used to test for emptiness Use isEmpty() to check whether the collection is empty or not.
common-java:DuplicatedBlocks (MAJOR) snowflake/SnowflakeS3StagingDestination.java Source files should not have any duplicated blocks 1 duplicated blocks of code must be removed.
java:S117 (MINOR) snowflake/SnowflakeS3StagingDestination.java:45 Local variable and method parameter names should comply with a naming convention Rename this local variable to match the regular expression '^[a-z][a-zA-Z0-9]*$'.
java:S112 (MAJOR) snowflake/SnowflakeS3StagingDestination.java:66 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S1192 (CRITICAL) snowflake/SnowflakeS3StagingSqlOperations.java:70 String literals should not be duplicated Define a constant instead of duplicating this literal "stage" 4 times.
java:S1172 (MAJOR) snowflake/SnowflakeS3StagingSqlOperations.java:109 Unused method parameters should be removed Remove this unused method parameter "stageName".
java:S1192 (CRITICAL) snowflake/SnowflakeDatabase.java:89 String literals should not be duplicated Define a constant instead of duplicating this literal "password" 3 times.
java:S112 (MAJOR) snowflake/SnowflakeAzureBlobStorageStreamCopier.java:61 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S1192 (CRITICAL) snowflake/SnowflakeDestinationResolver.java:28 String literals should not be duplicated Define a constant instead of duplicating this literal "loading_method" 9 times.
java:S1118 (MAJOR) snowflake/SnowflakeDestinationResolver.java:13 Utility classes should not have public constructors Add a private constructor to hide the implicit public one.
java:S1192 (CRITICAL) snowflake/SnowflakeDatabase.java:71 String literals should not be duplicated Define a constant instead of duplicating this literal "client_id" 4 times.
java:S1192 (CRITICAL) snowflake/SnowflakeDatabase.java:72 String literals should not be duplicated Define a constant instead of duplicating this literal "refresh_token" 6 times.
java:S1192 (CRITICAL) snowflake/SnowflakeDatabase.java:72 String literals should not be duplicated Define a constant instead of duplicating this literal "client_secret" 4 times.
java:S112 (MAJOR) snowflake/SnowflakeDatabase.java:74 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) snowflake/SnowflakeDatabase.java:164 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S2142 (MAJOR) snowflake/SnowflakeDatabase.java:167 "InterruptedException" should not be ignored Either re-interrupt this method or rethrow the "InterruptedException" that can be caught here.
java:S1104 (MINOR) snowflake/SnowflakeDestination.java Class variable fields should not have public accessibility Make isAlive a static final constant or non-public and provide accessors if needed.
java:S1444 (MINOR) snowflake/SnowflakeDestination.java "public static" fields should be constant Make this "public static isAlive" field final
java:S112 (MAJOR) snowflake/SnowflakeGcsStreamCopier.java:79 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) snowflake/SnowflakeParallelCopyStreamCopier.java:40 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) snowflake/SnowflakeS3StreamCopier.java:119 Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S112 (MAJOR) snowflake/SnowflakeInternalStagingSqlOperations.java Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S1192 (CRITICAL) snowflake/SnowflakeInternalStagingSqlOperations.java String literals should not be duplicated Define a constant instead of duplicating this literal "stage" 3 times.
java:S112 (MAJOR) snowflake/SnowflakeInternalStagingDestination.java Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S5361 (CRITICAL) snowflake/SnowflakeInternalStagingDestination.java:66 "String#replace" should be preferred to "String#replaceAll" Replace this call to "replaceAll()" by a call to the "replace()" method.
java:S112 (MAJOR) snowflake/SnowflakeInternalStagingSqlOperations.java Generic exceptions should never be thrown Define and throw a dedicated exception instead of using a generic one.
java:S1068 (MAJOR) snowflake/SnowflakeInsertDestination.java:19 Unused "private" fields should be removed Remove this unused "LOGGER" private field.
java:S1118 (MAJOR) snowflake/SnowflakeDatabase.java:36 Utility classes should not have public constructors Add a private constructor to hide the implicit public one.

Coverage (0.0%)

File Coverage File Coverage
src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeAzureBlobStorageStreamCopier.java 0.0 src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeAzureBlobStorageStreamCopierFactory.java 0.0
src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeCopyAzureBlobStorageDestination.java 0.0 src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeCopyGcsDestination.java 0.0
src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeCopyS3Destination.java 0.0 src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeDatabase.java 0.0
src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeDestination.java 0.0 src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeDestinationResolver.java 0.0
src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeGcsStreamCopier.java 0.0 src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeGcsStreamCopierFactory.java 0.0
src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeInsertDestination.java 0.0 src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeInternalStagingDestination.java 0.0
src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeInternalStagingSqlOperations.java 0.0 src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeParallelCopyStreamCopier.java 0.0
src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeS3StagingDestination.java 0.0 src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeS3StagingSqlOperations.java 0.0
src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeS3StreamCopier.java 0.0 src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeS3StreamCopierFactory.java 0.0
src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeSQLNameTransformer.java 0.0 src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeSqlOperations.java 0.0

Please sign in to comment.