Ensure we are using pyopenssl (See ome/omero-py#240)

[chris-allan]

As of (psf/requests#5443) released in requests 2.24.0 [1] the urllib3
monkeypatching in of pyopenssl now only happens if the SNI is
unavailable (mostly just Python 2). The prevailing intent within
the requests project seems to be to avoid pyopenssl use where
possible. Obviously, this is not what we need so we're going to
have to follow the requests and urllib3 advice to do it explictly.

I have chosen to only perform the explicit injection for the upgrade
check. Other uses of requests which want to avoid the sort of weird
OpenSSL errors like those outlined in (#240) will need to do
the same thing.

1. https://github.com/psf/requests/blob/main/HISTORY.md#2240-2020-06-17

[chris-allan]

Can be tested by upgrading to requests 2.24.0 or later and following the pattern outlined on #240. You will get the same or similar exception below (depending on your major Python and OpenSSL version) without injecting explicitly:

Python 3.6.8 (default, Nov 16 2020, 16:55:22)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: client.closeSession()

In [2]: import requests

In [3]: r = requests.get('https://www.glencoesoftware.com/')
---------------------------------------------------------------------------
SSLError                                  Traceback (most recent call last)
~/OMERO.venv.py36/lib64/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    676                 headers=headers,
--> 677                 chunked=chunked,
    678             )

~/OMERO.venv.py36/lib64/python3.6/site-packages/urllib3/connectionpool.py in _make_request(self, conn, method, url, timeout, chunked, **httplib_request_kw)
    380         try:
--> 381             self._validate_conn(conn)
    382         except (SocketTimeout, BaseSSLError) as e:

~/OMERO.venv.py36/lib64/python3.6/site-packages/urllib3/connectionpool.py in _validate_conn(self, conn)
    977         if not getattr(conn, "sock", None):  # AppEngine might not have  `.sock`
--> 978             conn.connect()
    979

~/OMERO.venv.py36/lib64/python3.6/site-packages/urllib3/connection.py in connect(self)
    344                 ssl_version=resolve_ssl_version(self.ssl_version),
--> 345                 cert_reqs=resolve_cert_reqs(self.cert_reqs),
    346             )

~/OMERO.venv.py36/lib64/python3.6/site-packages/urllib3/util/ssl_.py in create_urllib3_context(ssl_version, cert_reqs, options, ciphers)
    259     """
--> 260     context = SSLContext(ssl_version or PROTOCOL_TLS)
    261

/usr/lib64/python3.6/ssl.py in __new__(cls, protocol, *args, **kwargs)
    350     def __new__(cls, protocol=PROTOCOL_TLS, *args, **kwargs):
--> 351         self = _SSLContext.__new__(cls, protocol)
    352         return self

SSLError: [SSL: UNABLE_TO_LOAD_SSL2_MD5_ROUTINES] unknown error (_ssl.c:2830)

During handling of the above exception, another exception occurred:

MaxRetryError                             Traceback (most recent call last)
~/OMERO.venv.py36/lib64/python3.6/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    448                     retries=self.max_retries,
--> 449                     timeout=timeout
    450                 )

~/OMERO.venv.py36/lib64/python3.6/site-packages/urllib3/connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
    726             retries = retries.increment(
--> 727                 method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
    728             )

~/OMERO.venv.py36/lib64/python3.6/site-packages/urllib3/util/retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
    445         if new_retry.is_exhausted():
--> 446             raise MaxRetryError(_pool, url, error or ResponseError(cause))
    447

MaxRetryError: HTTPSConnectionPool(host='www.glencoesoftware.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(336236785, '[SSL: UNABLE_TO_LOAD_SSL2_MD5_ROUTINES] unknown error (_ssl.c:2830)'),))

During handling of the above exception, another exception occurred:

SSLError                                  Traceback (most recent call last)
<ipython-input-3-6e3c8ba4cf42> in <module>
----> 1 r = requests.get('https://www.glencoesoftware.com/')

~/OMERO.venv.py36/lib64/python3.6/site-packages/requests/api.py in get(url, params, **kwargs)
     74
     75     kwargs.setdefault('allow_redirects', True)
---> 76     return request('get', url, params=params, **kwargs)
     77
     78

~/OMERO.venv.py36/lib64/python3.6/site-packages/requests/api.py in request(method, url, **kwargs)
     59     # cases, and look like a memory leak in others.
     60     with sessions.Session() as session:
---> 61         return session.request(method=method, url=url, **kwargs)
     62
     63

~/OMERO.venv.py36/lib64/python3.6/site-packages/requests/sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
    528         }
    529         send_kwargs.update(settings)
--> 530         resp = self.send(prep, **send_kwargs)
    531
    532         return resp

~/OMERO.venv.py36/lib64/python3.6/site-packages/requests/sessions.py in send(self, request, **kwargs)
    641
    642         # Send the request
--> 643         r = adapter.send(request, **kwargs)
    644
    645         # Total elapsed time of the request (approximately)

~/OMERO.venv.py36/lib64/python3.6/site-packages/requests/adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
    512             if isinstance(e.reason, _SSLError):
    513                 # This branch is for urllib3 v1.22 and later.
--> 514                 raise SSLError(e, request=request)
    515
    516             raise ConnectionError(e, request=request)

SSLError: HTTPSConnectionPool(host='www.glencoesoftware.com', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLError(336236785, '[SSL: UNABLE_TO_LOAD_SSL2_MD5_ROUTINES] unknown error (_ssl.c:2830)'),))

With injecting explicitly:

Python 3.6.8 (default, Nov 16 2020, 16:55:22)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: client.closeSession()

In [2]: import requests

In [3]: import urllib3.contrib.pyopenssl

In [4]: urllib3.contrib.pyopenssl.inject_into_urllib3()

In [5]: r = requests.get('https://www.glencoesoftware.com/')

In [6]: r.status_code
Out[6]: 200

[sbesson]

Tested the scenario described in the PR on our CI systems after installing pyopenssl in the daily virtual environment. In addition to the minimal example, I tested the module patched by this change with the following example snippet:

>>> import omero
>>> client=omero.client('localhost')
>>> client.closeSession()
>>> from omero.util.upgrade_check import UpgradeCheck
>>> uc=UpgradeCheck('test', url='https://www.openmicroscopy.org')
>>> uc.run()

Without this PR, the last call throws SSLError(SSLError(336236785, '[SSL: UNABLE_TO_LOAD_SSL2_MD5_ROUTINES] unknown error (_ssl.c:2830)'. With this PR included, the final call successfully returns the HTML content.

No objection from side to having this change in an imminent patch release. I'll leave the other requested reviewers to express their views.

[chris-allan]

As discussed with @sbesson out of band, the fact that this doesn't blow up spectacularly with every use of UpgradeCheck right now is due to us not using HTTPS for the open source upgrade checks. Effectively, with the conditions outlined in #240 after use of Ice in a given Python process the ssl Python module is substantially broken due to the calling of EVP_Cleanup(). This is not something localized to our use of requests and also happens with use of the Python standard library. For example:

Python 3.6.8 (default, Nov 16 2020, 16:55:22)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: client.closeSession()

In [2]: import urllib.request

In [3]: try:
   ...:     r = urllib.request.urlopen('https://www.openmicroscopy.org/')
   ...: except Exception as e:
   ...:     print(e)
   ...:
[SSL: UNABLE_TO_LOAD_SSL2_MD5_ROUTINES] unknown error (_ssl.c:2830)

[joshmoore]

All 👍 (The only other usage I see in omero-py is the download of jars of omero import if not using the server zip)