Adds support to connections with Thrift over HTTP transport. #325

joaopedroantonio · 2020-04-04T15:37:17Z

Supports HTTP transport for Thrift protocol.
Three types of authentication supported: NONE, NOSASL, BASIC and KERBEROS.
BASIC authentication is useful when the Thrift HTTP interface is behind a proxy (e.g. in Azure HDInsight clusters).

joaopedroantonio · 2020-04-04T15:39:51Z

This PR solves issue #69. Let me know what you think and if it looks good for you I'll invest time in the tests that are missing in this PR. :)

- Supports HTTP transport for Thrift protocol. - Three types of authentication supported: NONE, BASIC and KERBEROS.

codecov · 2020-04-04T18:00:56Z

Codecov Report

Attention: Patch coverage is 91.71975% with 13 lines in your changes missing coverage. Please review.

Project coverage is 93.67%. Comparing base (c448ed9) to head (97fe26e).
Report is 25 commits behind head on master.

Files	Patch %	Lines
pyhive/hive.py	86.76%	4 Missing and 5 partials ⚠️
pyhive/tests/test_hive.py	95.50%	0 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #325      +/-   ##
==========================================
+ Coverage   93.23%   93.67%   +0.44%     
==========================================
  Files          14       14              
  Lines        1523     1677     +154     
  Branches      165      185      +20     
==========================================
+ Hits         1420     1571     +151     
+ Misses         75       74       -1     
- Partials       28       32       +4

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bkyryliuk · 2020-04-16T23:00:07Z

will take a pass on it next week, unfortunately can't get to this earlier

joaopedroantonio · 2020-04-22T09:55:17Z

pyhive/hive.py

+# TODO
+# Setting the Cookie in the headers should be implemented in the thrift library.
+# We'll keep this here until that change is available in there.
+class TCookieHttpClient(thrift.transport.THttpClient.THttpClient):


In the meantime, this commit was merged into Thrift master branch, we just have to wait for a new Thrift release to get rid of this TCookieHttpClient:

apache/thrift@69642f3

It looks like this fix has been included in the 0.13.0 Release

bkyryliuk · 2020-04-28T16:47:11Z

@joaopedroantonio I'm aligned with the approach here, let's add the unit tests & add the test plan in the PR description if you don't mind.

joaopedroantonio · 2020-04-30T16:45:39Z

Thank you for the feedback @bkyryliuk, will start working on the tests between this weekend and start of next week.

MinuMichaelOlassayil · 2020-06-16T00:27:47Z

Any update on this?

joaopedroantonio · 2020-06-16T08:48:30Z

Any update on this?

Oh well, tbh this got lost on my backlog, thanks for the bump. :) Will try to wrap it up this week.

joaopedroantonio · 2020-06-26T16:38:09Z

Added unit tests for different HTTP scenarios and fixed a few issues in the process.

@bkyryliuk the codecov checks failed but I guess that's mostly because I moved the binary transport implementation and there are a few line/branch misses. Do you think it's relevant to cover those misses or can we move forward as it is?

joaopedroantonio · 2020-07-13T08:48:45Z

@bkyryliuk, any idea when you might be able to pick this up? :)

joaopedroantonio · 2020-08-21T13:46:08Z

@bkyryliuk bump. :)

joaopedroantonio · 2020-10-15T18:06:06Z

Hello again everyone! Can you give some feedback on this PR? Thanks! (cc @bkyryliuk )

gthomas-slack · 2020-11-13T21:56:25Z

pyhive/hive.py

+        if http_path is None:
+            http_path = '/'
+
+        socket = TCookieHttpClient('http://{}:{}{}'.format(host, port, http_path))


It would be great if we could also pass through a http_protocol, either http or https and have this set.

gthomas-slack · 2020-11-13T21:59:58Z

I've tested this PR and this is working nicely for our HiveServer2 service running over http. Awesome work @joaopedroantonio - would love to see this get merged

gthomas-slack · 2020-11-13T22:08:04Z

For our particular use-case we need a few more features:

Support for connecting to a https endpoint
Support for passing through custom http headers
Support for passing through custom cookie values

I don't think these should block this PR though, I'd be happy to attempt to add these features once this initial version is in, unless of course you want to add them :)

gthomas-slack · 2020-11-14T01:03:04Z

pyhive/hive.py

+        if auth is None:
+            auth = 'NONE'
+        if http_path is None:
+            http_path = '/'


It could be better to default this to /cliservice, as that seems to be the default path HiveServer2 provides

joaopedroantonio · 2020-11-18T23:37:16Z

@gthomas-slack sorry for the late reply! I'm glad this was helpful! I'll take a better look at your comments over the weekend and I'm happy to include your suggestions if they aren't too complex. :)

lvazquez · 2020-11-19T21:11:19Z

For thrift=0.13.0 (current version in conda/pip) it doesn't work with the included TCookieHttpClient wrapper class, and I get the following error in the connect constructor when executing the first "use default" query:

  File "/root/superset/pyhive/pyhive/hive.py", line 189, in __init__
    cursor.execute('USE `{}`'.format(database))
  File "/root/superset/pyhive/pyhive/hive.py", line 459, in execute
    response = self._connection.client.ExecuteStatement(req)
  File "/root/superset/pyhive/TCLIService/TCLIService.py", line 280, in ExecuteStatement
    return self.recv_ExecuteStatement()
  File "/root/superset/pyhive/TCLIService/TCLIService.py", line 292, in recv_ExecuteStatement
    (fname, mtype, rseqid) = iprot.readMessageBegin()
  File "/root/superset/pyhive/env/lib/python3.7/site-packages/thrift/protocol/TBinaryProtocol.py", line 148, in readMessageBegin
    name = self.trans.readAll(sz)
  File "/root/superset/pyhive/env/lib/python3.7/site-packages/thrift/transport/TTransport.py", line 68, in readAll
    raise EOFError()

After replacing TCookieHttpClient with the original THttpClient it seems to work ok for http connections (no SSL support included).

lvazquez · 2020-11-19T21:39:18Z

I believe the parameters name for the HTTP Thrift should be renamed to match the (simpler) names used in the original JDBC Thrift URI. Also 'binary', 'http', 'sasl' are not named "protocols" but transport "modes" according to JDBC/Hive documentation
Connection URL When HiveServer2 Is Running in HTTP Mode

Example Thrift JDBC over HTTPS URI:
jdbc:hive2://{host}:{port}/{database};transportMode=http;httpPath=/hive;ssl=true;sslTrustStore=truststore.jks;trustStorePassword=changeit

Based on this it seems the names should be:

thrift_transport_protocol -> transport_mode (transportMode in JDBC/Thrift)
http_path -> http_path (httpPath in JDBC/Thrift)

The remaining parameters needed to support HTTPS (HTTP over SSL) should be:

ssl = True/False (could also be handled with transport_mode=https but this seems cleaner).
cacert, ca_cert or cafile (Python uses plain PEM cert files instead of Java keystores/truststores
OPTIONAL: cert_file, key_file and key_pass - to support SSL client authentication (as intended in Support for thrift over HTTPS for Hive connections #306)

kent-pawar · 2020-12-10T21:38:20Z

Hi @joaopedroantonio . Bumping this. Thank you for working on this! Could you pls look at the new feedback. Thank you

mhconradt · 2020-12-14T04:59:37Z

I've tested this with my Spark Thrift HiveServer2 deployment behind a load balancer on Kubernetes and it works great.
I'm looking forward to using this to connect Superset to my Spark Thrift server and would like to see this accepted soon.
Thanks @joaopedroantonio and everyone in the thread.

gthomas-slack · 2020-12-14T17:07:24Z

For anyone who needs to use http/https HiveServer2 in production right now, It's actually possible with the current release of PyHive, as the HiveServer2 Connection class exposes a thrift_transport which allows you to pass in a custom configured transport to use: https://github.com/dropbox/PyHive/blob/master/pyhive/hive.py#L112-L113

Here is a basic example using HTTPS and adding a custom authentication header:

from pyhive import hive
import base64
import thrift.transport.THttpClient


def thrift_http_transport():
    transport = thrift.transport.THttpClient.THttpClient(uri_or_host='https://my-hiveserver2.com:443/cliservice')

    auth_credentials = '{}:{}'.format('test', 'test').encode('UTF-8')
    auth_credentials_base64 = base64.standard_b64encode(auth_credentials).decode('UTF-8')
    transport.setCustomHeaders(
        {
            'Authorization': 'Basic {}'.format(auth_credentials_base64),  # HiveServer2 BASIC auth
            'X-Auth-Token': 'xxx' # Custom header to auth with some kind of middleware
        }
    )
    return transport


conn = hive.connect(thrift_transport=thrift_http_transport())
cursor = conn.cursor()
cursor.execute("""SELECT SUM(1) from model.dim_date""")
data = cursor.fetchall()
cursor.close()
conn.close()
print(data)

mithunjmistry · 2021-06-21T21:12:31Z

Can we use http protocol with SQLAlchemy?

ajprabhu09 · 2021-10-13T10:09:36Z

Does anyone have an example for http connection with kerberos auth ?

gthomas-slack · 2021-11-04T17:25:30Z

FYI there is currently a bug using HTTP transport connections when using thrift 0.15.0, some details here:
#417 (comment)

CLAassistant · 2022-04-16T21:44:52Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
0 out of 2 committers have signed the CLA.

❌ Joao Antonio
❌ joaopedroantonio

Joao Antonio seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

danjampro · 2022-11-28T20:13:44Z

Hi @joaopedroantonio are there any updates on this?

joaopedroantonio mentioned this pull request Apr 4, 2020

PyHive and Transport mode - HTTP #69

Open

Adds support to connections with Thrift over HTTP transport.

84d2768

- Supports HTTP transport for Thrift protocol. - Three types of authentication supported: NONE, BASIC and KERBEROS.

joaopedroantonio force-pushed the master branch from 2c461de to 84d2768 Compare April 4, 2020 17:54

joaopedroantonio commented Apr 22, 2020

View reviewed changes

Implement unit tests.

95f69ac

Add unit tests to increase code coverage.

97fe26e

gthomas-slack reviewed Nov 13, 2020

View reviewed changes

gthomas-slack reviewed Nov 14, 2020

View reviewed changes

vakarisbk mentioned this pull request Oct 3, 2023

Add support for Spark Connect (SQL models) dbt-labs/dbt-spark#899

Open

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds support to connections with Thrift over HTTP transport. #325

Adds support to connections with Thrift over HTTP transport. #325

joaopedroantonio commented Apr 4, 2020 •

edited

Loading

joaopedroantonio commented Apr 4, 2020

codecov bot commented Apr 4, 2020 •

edited

Loading

bkyryliuk commented Apr 16, 2020

joaopedroantonio Apr 22, 2020

gthomas-slack Nov 28, 2020

bkyryliuk commented Apr 28, 2020

joaopedroantonio commented Apr 30, 2020

MinuMichaelOlassayil commented Jun 16, 2020

joaopedroantonio commented Jun 16, 2020

joaopedroantonio commented Jun 26, 2020

joaopedroantonio commented Jul 13, 2020

joaopedroantonio commented Aug 21, 2020

joaopedroantonio commented Oct 15, 2020

gthomas-slack Nov 13, 2020

gthomas-slack commented Nov 13, 2020

gthomas-slack commented Nov 13, 2020

gthomas-slack Nov 14, 2020

joaopedroantonio commented Nov 18, 2020

lvazquez commented Nov 19, 2020

lvazquez commented Nov 19, 2020 •

edited

Loading

kent-pawar commented Dec 10, 2020

mhconradt commented Dec 14, 2020

gthomas-slack commented Dec 14, 2020 •

edited

Loading

mithunjmistry commented Jun 21, 2021

ajprabhu09 commented Oct 13, 2021

gthomas-slack commented Nov 4, 2021

CLAassistant commented Apr 16, 2022 •

edited

Loading

danjampro commented Nov 28, 2022

Adds support to connections with Thrift over HTTP transport. #325

Are you sure you want to change the base?

Adds support to connections with Thrift over HTTP transport. #325

Conversation

joaopedroantonio commented Apr 4, 2020 • edited Loading

joaopedroantonio commented Apr 4, 2020

codecov bot commented Apr 4, 2020 • edited Loading

Codecov Report

bkyryliuk commented Apr 16, 2020

joaopedroantonio Apr 22, 2020

Choose a reason for hiding this comment

gthomas-slack Nov 28, 2020

Choose a reason for hiding this comment

bkyryliuk commented Apr 28, 2020

joaopedroantonio commented Apr 30, 2020

MinuMichaelOlassayil commented Jun 16, 2020

joaopedroantonio commented Jun 16, 2020

joaopedroantonio commented Jun 26, 2020

joaopedroantonio commented Jul 13, 2020

joaopedroantonio commented Aug 21, 2020

joaopedroantonio commented Oct 15, 2020

gthomas-slack Nov 13, 2020

Choose a reason for hiding this comment

gthomas-slack commented Nov 13, 2020

gthomas-slack commented Nov 13, 2020

gthomas-slack Nov 14, 2020

Choose a reason for hiding this comment

joaopedroantonio commented Nov 18, 2020

lvazquez commented Nov 19, 2020

lvazquez commented Nov 19, 2020 • edited Loading

kent-pawar commented Dec 10, 2020

mhconradt commented Dec 14, 2020

gthomas-slack commented Dec 14, 2020 • edited Loading

mithunjmistry commented Jun 21, 2021

ajprabhu09 commented Oct 13, 2021

gthomas-slack commented Nov 4, 2021

CLAassistant commented Apr 16, 2022 • edited Loading

danjampro commented Nov 28, 2022

joaopedroantonio commented Apr 4, 2020 •

edited

Loading

codecov bot commented Apr 4, 2020 •

edited

Loading

lvazquez commented Nov 19, 2020 •

edited

Loading

gthomas-slack commented Dec 14, 2020 •

edited

Loading

CLAassistant commented Apr 16, 2022 •

edited

Loading