[BUG] - SSL Certificate Verification Failure in Dask Gateway Unit Tests

[viniciusdc]

Describe the bug

Our unit tests have begun failing due to SSL certificate verification errors when connecting to the Dask Gateway. Specifically, we encounter an SSLCertVerificationError caused by a self-signed certificate.

=================================== FAILURES ===================================
______________________________ test_dask_gateway _______________________________
....
  File "/home/runnerx/miniconda3/envs/test/lib/python3.11/site-packages/dask_gateway/client.py", line 397, in _request
    resp = await session.request(method, url, json=json, **self._request_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runnerx/miniconda3/envs/test/lib/python3.11/site-packages/aiohttp/client.py", line [659](https://github.com/nebari-dev/nebari/actions/runs/11022572877/job/30612182623?pr=2739#step:16:660), in _request
    conn = await self._connector.connect(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runnerx/miniconda3/envs/test/lib/python3.11/site-packages/aiohttp/connector.py", line 557, in connect
    proto = await self._create_connection(req, traces, timeout)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runnerx/miniconda3/envs/test/lib/python3.11/site-packages/aiohttp/connector.py", line 1002, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runnerx/miniconda3/envs/test/lib/python3.11/site-packages/aiohttp/connector.py", line 1336, in _create_direct_connection
    raise last_exc
  File "/home/runnerx/miniconda3/envs/test/lib/python3.11/site-packages/aiohttp/connector.py", line 1305, in _create_direct_connection
    transp, proto = await self._wrap_create_connection(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/runnerx/miniconda3/envs/test/lib/python3.11/site-packages/aiohttp/connector.py", line 1070, in _wrap_create_connection
    raise ClientConnectorCertificateError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host github-actions.nebari.dev:443 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate (_ssl.c:1006)')]

We've been using a monkey patch to disable SSL verification as a temporary workaround, which has allowed tests to pass so far; however, this does not work anymore.

nebari/tests/tests_deployment/utils.py

Lines 115 to 132 in b2afe1a

	def monkeypatch_ssl_context():
	"""
	This is a workaround monkeypatch to disable ssl checking to avoid SSL
	failures.
	TODO: A better way to do this would be adding the Traefik's default certificate's
	CA public key to the trusted certificate authorities.
	"""
	def create_default_context(context):
	def _inner(*args, **kwargs):
	context.check_hostname = False
	context.verify_mode = ssl.CERT_NONE
	return context
	return _inner
	sslcontext = ssl.create_default_context()
	ssl.create_default_context = create_default_context(sslcontext)

My guess, based on a similar encounter, is that aiohttp the base lib used by dask to handle the connections has changed something internally regarding the TCPConnector default behavior (this happens internally within the requests, and is decided by a combination of parameters that dask handles based on the provided proxy schema, which in our case we were passing for jupyterhub authentication)

    async def _request(self, method, url, json=None):
        if self._session is None:
            # "unsafe" allows cookies to be set for ip addresses, which can
            # commonly serve dask-gateway deployments. Since this client is
            # only ever used with a single endpoint, there is no danger of
            # leaking cookies to a different server that happens to have the
            # same ip.
            self._session = aiohttp.ClientSession(
                cookie_jar=aiohttp.CookieJar(unsafe=True)
            )
        session = self._session

        resp = await session.request(method, url, json=json, **self._request_kwargs)

To address this adequately, I propose overriding the dask_gateway.Gateway class using self._request_kwargs object above to allow the TLS parameter to be disabled within our test context.

Expected behavior

• unit tests should run correctly

OS and architecture in which you are running Nebari

Linux (CI/GHA)

How to Reproduce the problem?

https://github.com/nebari-dev/nebari/actions/runs/11022572877/job/30612182623?pr=2739

Command output

No response

Versions and dependencies used.

No response

Compute environment

kind

Integrations

No response

Anything else?

No response