Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade Hive JDBC version to 3.1.3 #5

Closed
wants to merge 3 commits into from
Closed

Conversation

ebyhr
Copy link
Member

@ebyhr ebyhr commented May 29, 2023

The previous version 0.13.0 throws an exception
when executing a query on Spark 3.4.0.

Relates to trinodb/trino#17667

@cla-bot cla-bot bot added the cla-signed label May 29, 2023
ebyhr added 3 commits May 29, 2023 12:49
Otherwise, it throws an IllegalArgumentException.
The previous version 0.13.0 throws an exception
when executing an query on Spark 3.4.0.
Copy link
Member

@electrum electrum left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for updating this. We need to either add more exclusions or adjust the shading rules. If you look at the current release JAR, everything is under one of these:

io/trino/hive/jdbc/$internal
org/apache/hive/jdbc

The version from this PR has lots of extra stuff, which you can see by running the following:

jar tf target/hive-apache-jdbc-*-SNAPSHOT.jar | sort

I suspect that we can exclude most of these things. It might help to look at the exclusion rules from our trino-hive-apache project, as those will match the current Hive version.

<dep.guava.version>11.0.2</dep.guava.version>
<dep.slf4j.version>1.7.10</dep.slf4j.version>
<dep.hadoop.version>1.2.1</dep.hadoop.version>
<dep.hive.version>3.1.3</dep.hive.version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update the project version to 3.1.3-1-SNAPSHOT so that it matches the Hive version.

Update the README to use the new version, or simply remove the Hive version from the README.

@ebyhr
Copy link
Member Author

ebyhr commented Jun 5, 2023

Hmm, it seems the new Hive JDBC driver can't connect to the old Hive server (e.g. hdp2.6-hive in product tests). Delta Lake 2.4.0 requires at least Hive 3.0.0 JDBC driver, but the version isn't compatible with the above HDP environment.

@ebyhr
Copy link
Member Author

ebyhr commented Jun 8, 2023

Installing standalone Hive JDBC driver at runtime like trinodb/trino#17667 looks better because we need to use two JDBC versions for connecting to old Hive server and new Spark server.

@ebyhr ebyhr closed this Jun 8, 2023
@ebyhr ebyhr deleted the ebi/3.1.3 branch June 8, 2023 02:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

3 participants