🐞 Destination databricks: update jdbc driver to patch log4j #7622

tuliren · 2021-11-04T07:00:52Z

What

Update the Spark JDBC driver to patch Log4j vulnerability.
Download the driver file automatically in test and publish commands.
The new version will be published in 🎉 Destination databricks: update fields specifications #9153.

tuliren · 2021-11-04T07:01:32Z

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/1420186884
❌ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/1420186884
🐛

tuliren · 2021-11-04T07:59:07Z

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/1420333982
❌ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/1420333982
🐛

tuliren · 2021-11-04T08:06:32Z

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/1420358298
❌ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/1420358298
🐛

tuliren · 2021-11-04T08:15:56Z

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/1420386204
✅ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/1420386204
No Python unittests run

tuliren · 2021-11-04T08:37:57Z

@davinchia, I manually created an io-airbyte-build-dependencies bucket in GCS. Should I do it through some Terraform script instead?

davinchia · 2021-11-04T09:09:12Z

@tuliren yes we should. You can either do it on your own, or create a ticket for the infrastructure team to help you with it.

Have you seen https://docs.google.com/document/d/1Xqr9fC1toPlMoj74jb9e5FaTMjmsBIFd9BY3c3Zv5qc/edit#bookmark=id.ik4m6mwqpw5v on how to work with the infrastructure team?

davinchia · 2021-11-04T09:14:40Z

Question from me:

Did you do anything special to grant the spec service account permissions to this bucket?
Maybe this is a dumb question - can we put the JDBC driver into our CloudRepo Maven repository instead? This way all our public java artifacts are in one place and we don't have an additional bucket to manage.

tuliren · 2021-11-04T20:02:02Z

Did you do anything special to grant the spec service account permissions to this bucket?

I granted this service account the read permission to this bucket on the permission page on GCS.

Can we put the JDBC driver into our CloudRepo Maven repository instead? This way all our public java artifacts are in one place and we don't have an additional bucket to manage.

Where is this CloudRepo Maven repository?
What do you mean by "public"? This jar cannot be publicly downloadable by anyone from the internet. When downloading it, people need to accept to a certain terms & conditions.
The jar is publicly available here. I just don't know if we can download directly from that link programmatically. So if we can put it somewhere private, we won't have the potential legal issues.

davinchia · 2021-11-05T06:16:06Z

CloudRepo is the Java hosting service we are using: cloudrepo.io. Login credentials are in Lastpass under cloudrepo.io in the shared-engineering folder. This is where java artifacts from OSS are pushed to.

Got it. I didn't realise we want to keep this private. If you aren't rushing, I think I would prefer creating a private repository in CloudRepo and set things up so we publish and pull from there. Main thing here is to keep all the java artifacts in one place. I can take a stab at this on Monday/Tuesday if you aren't free.

What do you think?

tuliren · 2021-11-05T06:19:19Z

Got it. Thanks. I think I will just change it to download from Databricks each time. That does not seem to be breaking the terms & conditions. It's not worth the time to be too careful. But thank you all the same.

davinchia · 2021-11-05T06:21:29Z

Okay! Happy to relook when you make those changes. Thanks Liren.

tuliren · 2022-01-04T23:38:11Z

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/1656038661
❌ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/1656038661
🐛

tuliren · 2022-01-04T23:49:45Z

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/1656069372
❌ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/1656069372
🐛

tuliren · 2022-01-05T04:31:55Z

airbyte-workers/src/main/java/io/airbyte/workers/DefaultCheckConnectionWorker.java

@@ -77,7 +77,7 @@ public StandardCheckConnectionOutput run(final StandardCheckConnectionInput inpu
        LOGGER.debug("Check connection job received output: {}", output);
        return output;
      } else {
-        throw new WorkerException("Error while getting checking connection.");
+        throw new WorkerException(String.format("Error checking connection, status: %s, exit code: %d", status, exitCode));


@jrhizor, a check operation failed in the acceptance test. I added more information to the exception message and saw this:

Error checking connection, status: Optional[io.airbyte.protocol.models.AirbyteConnectionStatus@605d010e[status=SUCCEEDED,message=<null>,additionalProperties=***]], exit code: 143

Do you know what might be the root cause behind this exit code? Could it be related to the recent change in WorkerUtils#gentleClose?

I don't think the gentle close changes look related. Are you sure this docker container isn't actually outputting an error code?

Here is what happens:

The database connection somehow requires an explicit close method call, otherwise, the connector will not shutdown.

The worker will wait for 1 minute for the connector to shutdown, otherwise it will close it by force (code here) by sending a SIGTERM.

The SIGTERM results in a 143 exit code.

So the root cause is that we must always close the database at the end for Databricks, which was not the case in the default check command implementation in CopyDestination. The issue has been resolved.

tuliren · 2022-01-07T03:57:43Z

/test connector=connectors/destination-databricks

🕑 connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/1666057934
✅ connectors/destination-databricks https://github.com/airbytehq/airbyte/actions/runs/1666057934
No Python unittests run

jrhizor temporarily deployed to more-secrets November 4, 2021 07:04 Inactive

tuliren temporarily deployed to more-secrets November 4, 2021 07:55 Inactive

tuliren temporarily deployed to more-secrets November 4, 2021 08:00 Inactive

jrhizor temporarily deployed to more-secrets November 4, 2021 08:01 Inactive

tuliren temporarily deployed to more-secrets November 4, 2021 08:07 Inactive

jrhizor temporarily deployed to more-secrets November 4, 2021 08:08 Inactive

jrhizor temporarily deployed to more-secrets November 4, 2021 08:18 Inactive

tuliren requested review from sherifnada and davinchia November 4, 2021 08:32

tuliren temporarily deployed to more-secrets November 4, 2021 08:33 Inactive

tuliren mentioned this pull request Jan 4, 2022

🎉 Destination databricks: update fields specifications #9153

Merged

16 tasks

tuliren force-pushed the liren/auto-download-spark-jdbc branch from 736c0d6 to 76315db Compare January 4, 2022 19:00

tuliren temporarily deployed to more-secrets January 4, 2022 19:02 Inactive

github-actions bot added the area/connectors Connector related issues label Jan 4, 2022

tuliren temporarily deployed to more-secrets January 4, 2022 23:39 Inactive

jrhizor temporarily deployed to more-secrets January 4, 2022 23:40 Inactive

tuliren temporarily deployed to more-secrets January 4, 2022 23:50 Inactive

tuliren commented Jan 5, 2022

View reviewed changes

tuliren mentioned this pull request Jan 5, 2022

Destination databricks: integration test hanging after the destination completes #9298

Closed

tuliren added 13 commits January 6, 2022 15:12

Download spark jdbc driver in build cmd

45d3c79

Download jdbc driver in ci integration test

8bc3e9c

Update comments

eb4d03a

Set up cloud sdk

e83feb8

Add comments

4970475

Download jdbc driver from databricks directly

d863236

Update readme

dea916c

Use unzip command

9bcacee

Install unzip for databricks

a200f2e

Add databricks build status

e92e582

Close database

3b8b2bf

Log more error information

7b71fef

Close database when checking connection

6b5d420

tuliren force-pushed the liren/auto-download-spark-jdbc branch from 32aa7bb to 6b5d420 Compare January 7, 2022 03:56

tuliren temporarily deployed to more-secrets January 7, 2022 03:58 Inactive

jrhizor temporarily deployed to more-secrets January 7, 2022 04:00 Inactive

Merge branch 'master' into liren/auto-download-spark-jdbc

e2c2b3e

tuliren temporarily deployed to more-secrets January 7, 2022 04:03 Inactive

tuliren changed the title ~~Download jdbc driver for databricks connector~~ 🐞 Destination databricks: update jdbc driver to patch log4j Jan 7, 2022

Update spec

a37a2a6

tuliren temporarily deployed to more-secrets January 7, 2022 04:14 Inactive

tuliren linked an issue Jan 7, 2022 that may be closed by this pull request

Destination databricks: integration test hanging after the destination completes #9298

Closed

tuliren merged commit dfb7b8c into master Jan 7, 2022

tuliren deleted the liren/auto-download-spark-jdbc branch January 7, 2022 04:22

jrhizor mentioned this pull request Jan 7, 2022

Bump Airbyte version from 0.35.3-alpha to 0.35.4-alpha #9353

Merged

karinakuz added connectors/destinations-warehouse connectors/destination/databricks labels Jan 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐞 Destination databricks: update jdbc driver to patch log4j #7622

🐞 Destination databricks: update jdbc driver to patch log4j #7622

tuliren commented Nov 4, 2021 •

edited

Loading

tuliren commented Nov 4, 2021 •

edited by github-actions bot

Loading

tuliren commented Nov 4, 2021 •

edited by github-actions bot

Loading

tuliren commented Nov 4, 2021 •

edited by github-actions bot

Loading

tuliren commented Nov 4, 2021 •

edited by github-actions bot

Loading

tuliren commented Nov 4, 2021 •

edited

Loading

davinchia commented Nov 4, 2021

davinchia commented Nov 4, 2021

tuliren commented Nov 4, 2021

davinchia commented Nov 5, 2021

tuliren commented Nov 5, 2021

davinchia commented Nov 5, 2021

tuliren commented Jan 4, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 4, 2022 •

edited by github-actions bot

Loading

tuliren Jan 5, 2022

jrhizor Jan 7, 2022

tuliren Jan 7, 2022 •

edited

Loading

tuliren commented Jan 7, 2022 •

edited by github-actions bot

Loading

🐞 Destination databricks: update jdbc driver to patch log4j #7622

🐞 Destination databricks: update jdbc driver to patch log4j #7622

Conversation

tuliren commented Nov 4, 2021 • edited Loading

What

tuliren commented Nov 4, 2021 • edited by github-actions bot Loading

tuliren commented Nov 4, 2021 • edited by github-actions bot Loading

tuliren commented Nov 4, 2021 • edited by github-actions bot Loading

tuliren commented Nov 4, 2021 • edited by github-actions bot Loading

tuliren commented Nov 4, 2021 • edited Loading

davinchia commented Nov 4, 2021

davinchia commented Nov 4, 2021

tuliren commented Nov 4, 2021

davinchia commented Nov 5, 2021

tuliren commented Nov 5, 2021

davinchia commented Nov 5, 2021

tuliren commented Jan 4, 2022 • edited by github-actions bot Loading

tuliren commented Jan 4, 2022 • edited by github-actions bot Loading

tuliren Jan 5, 2022

Choose a reason for hiding this comment

jrhizor Jan 7, 2022

Choose a reason for hiding this comment

tuliren Jan 7, 2022 • edited Loading

Choose a reason for hiding this comment

tuliren commented Jan 7, 2022 • edited by github-actions bot Loading

tuliren commented Nov 4, 2021 •

edited

Loading

tuliren commented Nov 4, 2021 •

edited by github-actions bot

Loading

tuliren commented Nov 4, 2021 •

edited by github-actions bot

Loading

tuliren commented Nov 4, 2021 •

edited by github-actions bot

Loading

tuliren commented Nov 4, 2021 •

edited by github-actions bot

Loading

tuliren commented Nov 4, 2021 •

edited

Loading

tuliren commented Jan 4, 2022 •

edited by github-actions bot

Loading

tuliren commented Jan 4, 2022 •

edited by github-actions bot

Loading

tuliren Jan 7, 2022 •

edited

Loading

tuliren commented Jan 7, 2022 •

edited by github-actions bot

Loading