[BUG] - Python tests (CLI, deployment) are failing locally and on forks

[krassowski]

Describe the bug

The developer experience for adding tests is not great.

1. Running tests locally does not work - it fails to collect them saying that KEYCLOAK_USERNAME is undefined. It is not mentioned in the documentation
2. Running tests on forks does not work. This means that even on CI some magic things need to be configured.
   
3. The documentation presents two commands, one for CLI:

pytest tests/

and one for deployment tests

pytest tests_deployment/

Both are incorrect. The first command will invoke not only CLI tests but also the deployment tests because these are in tests/tests_deployment. The CLI tests are in fact in tests/tests_unit. The second command will just error out with ERROR: file or directory not found: tests_deployment/

Expected behavior

Tests work not only on main branch but also on forks and can be run locally.

OS and architecture in which you are running Nebari

Yes

How to Reproduce the problem?

As per https://www.nebari.dev/docs/community/nebari-tests/#test-cli-with-pytest run:

pytest tests/

See

platform linux -- Python 3.10.12, pytest-8.0.0, pluggy-1.3.0
rootdir: ~/nebari
configfile: pytest.ini
plugins: cov-4.1.0, playwright-0.4.4, base-url-2.1.0, timeout-2.2.0
collected 484 items / 3 errors                                                                                                                                                                                                                                                                                                                                                                                                           

======================================= ERRORS =======================================
_________________________________________________ ERROR collecting tests/tests_deployment/test_dask_gateway.py ___________________________________________________
Traceback (most recent call last):
  File "/site-packages/_pytest/runner.py", line 345, in from_call
    result: Optional[TResult] = func()
  File "/site-packages/_pytest/runner.py", line 390, in collect
    return list(collector.collect())
  File "/site-packages/_pytest/python.py", line 588, in collect
    self._inject_setup_module_fixture()
  File "/site-packages/_pytest/python.py", line 602, in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
  File "/site-packages/_pytest/python.py", line 316, in obj
    self._obj = obj = self._getobj()
  File "/site-packages/_pytest/python.py", line 585, in _getobj
    return importtestmodule(self.path, self.config)
  File "/site-packages/_pytest/python.py", line 537, in importtestmodule
    mod = import_path(path, mode=importmode, root=config.rootpath)
  File "/site-packages/_pytest/pathlib.py", line 567, in import_path
    importlib.import_module(module_name)
  File "/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "/site-packages/_pytest/assertion/rewrite.py", line 175, in exec_module
    exec(co, module.__dict__)
  File "nebari/tests/tests_deployment/test_dask_gateway.py", line 6, in <module>
    from tests.tests_deployment import constants
  File "nebari/tests/tests_deployment/constants.py", line 6, in <module>
    KEYCLOAK_USERNAME = os.environ["KEYCLOAK_USERNAME"]
  File "/lib/python3.10/os.py", line 680, in __getitem__
    raise KeyError(key) from None
KeyError: 'KEYCLOAK_USERNAME'

ERROR tests/tests_deployment/test_dask_gateway.py - KeyError: 'KEYCLOAK_USERNAME'
ERROR tests/tests_deployment/test_grafana_api.py - KeyError: 'KEYCLOAK_USERNAME'
ERROR tests/tests_deployment/test_jupyterhub_ssh.py - KeyError: 'KEYCLOAK_USERNAME'

Command output

No response

Versions and dependencies used.

No response

Compute environment

None

Integrations

No response

Anything else?

No response

[krassowski]

Even after defining the test user and passing KEYCLOAK_USERNAME/KEYCLOAK_PASSWORD I get errors because some tests are using hard-coded github-actions.nebari.dev namespace rather than my local deployment:

FAILED tests/tests_deployment/test_grafana_api.py::test_grafana_api_not_accessible_with_default_credentials - requests.exceptions.ConnectionError: HTTPSConnectionPool(host='github-actions.nebari.dev', port=443): Max retries exceeded with url: /monitoring/api/datasources (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x741df1cd1f90>: Failed to establish a new connection: [Errno 113] No route to host'))
ERROR tests/tests_deployment/test_dask_gateway.py::test_dask_gateway - requests.exceptions.ConnectionError: HTTPSConnectionPool(host='github-actions.nebari.dev', port=443): Max retries exceeded with url: /hub/oauth_login (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x741df1cd2140>: Failed to establish a new connection: [Errno 113] No route to host'))
ERROR tests/tests_deployment/test_dask_gateway.py::test_dask_gateway_cluster_options - requests.exceptions.ConnectionError: HTTPSConnectionPool(host='github-actions.nebari.dev', port=443): Max retries exceeded with url: /hub/oauth_login (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x741df1cd3ac0>: Failed to establish a new connection: [Errno 113] No route to host'))
ERROR tests/tests_deployment/test_jupyterhub_ssh.py::test_simple_jupyterhub_ssh - requests.exceptions.ConnectionError: HTTPSConnectionPool(host='github-actions.nebari.dev', port=443): Max retries exceeded with url: /hub/oauth_login (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x741df1cd21a0>: Failed to establish a new connection: [Errno 113] No route to host'))
ERROR tests/tests_deployment/test_jupyterhub_ssh.py::test_print_jupyterhub_ssh - requests.exceptions.ConnectionError: HTTPSConnectionPool(host='github-actions.nebari.dev', port=443): Max retries exceeded with url: /hub/oauth_login (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x741df1cd21a0>: Failed to establish a new connection: [Errno 113] No route to host'))
ERROR tests/tests_deployment/test_jupyterhub_ssh.py::test_exact_jupyterhub_ssh - requests.exceptions.ConnectionError: HTTPSConnectionPool(host='github-actions.nebari.dev', port=443): Max retries exceeded with url: /hub/oauth_login (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x741df1cd21a0>: Failed to establish a new connection: [Errno 113] No route to host'))
ERROR tests/tests_deployment/test_jupyterhub_ssh.py::test_contains_jupyterhub_ssh - requests.exceptions.ConnectionError: HTTPSConnectionPool(host='github-actions.nebari.dev', port=443): Max retries exceeded with url: /hub/oauth_login (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x741df1cd21a0>: Failed to establish a new connection: [Errno 113] No route to host'))

[krassowski]

It looks like that can be changed by undocumented NEBARI_HOSTNAME variable

[krassowski]

Still the tests are failing for me on setup when getting the JupyterHub token ( ERROR at setup of test_contains_jupyterhub_ssh):

tests/tests_deployment/test_dask_gateway.py::test_dask_gateway ERROR
tests/tests_deployment/test_dask_gateway.py::test_dask_gateway_cluster_options ERROR
tests/tests_deployment/test_jupyterhub_ssh.py::test_simple_jupyterhub_ssh ERROR
tests/tests_deployment/test_jupyterhub_ssh.py::test_print_jupyterhub_ssh ERROR
tests/tests_deployment/test_jupyterhub_ssh.py::test_exact_jupyterhub_ssh ERROR
tests/tests_deployment/test_jupyterhub_ssh.py::test_contains_jupyterhub_ssh ERROR

  File "/home/krassowski/nebari/tests/tests_deployment/test_jupyterhub_ssh.py", line 19, in api_token
    return get_jupyterhub_token("jupyterhub-ssh")
  File "/home/krassowski/nebari/tests/tests_deployment/utils.py", line 44, in get_jupyterhub_token
    return r.json()["token"]
KeyError: 'token'

The response itself is:

{'status': 403, 'message': 'User test-user not found or no permissions to generate tokens'}

[krassowski]

Ok, the token issue was a genuine problem on my branch. But the, running the ssh tests results in a prompt to unlock ssh agent, which I think should not be the case:

Declining leads to:

    raise SSHException("No existing session")
paramiko.ssh_exception.SSHException: No existing session
--------------- Captured log setup ------------------
2024-05-06 11:56:00      INFO 1909 transport: Connected (version 2.0, client AsyncSSH_2.13.2)
2024-05-06 11:56:32     ERROR 1909 transport: Exception (client): key cannot be used for signing
2024-05-06 11:56:32     ERROR 1907 transport: Traceback (most recent call last):
2024-05-06 11:56:32     ERROR 1907 transport:   File "envs/nebari/lib/python3.10/site-packages/paramiko/transport.py", line 2220, in run
2024-05-06 11:56:32     ERROR 1907 transport:     handler(m)
2024-05-06 11:56:32     ERROR 1907 transport:   File "site-packages/paramiko/auth_handler.py", line 404, in _parse_service_accept
2024-05-06 11:56:32     ERROR 1907 transport:     sig = self.private_key.sign_ssh_data(blob, algorithm)
2024-05-06 11:56:32     ERROR 1907 transport:   File "envs/nebari/lib/python3.10/site-packages/paramiko/agent.py", line 496, in sign_ssh_data
2024-05-06 11:56:32     ERROR 1907 transport:     raise SSHException("key cannot be used for signing")
2024-05-06 11:56:32     ERROR 1907 transport: paramiko.ssh_exception.SSHException: key cannot be used for signing
2024-05-06 11:56:32     ERROR 1907 transport:

[krassowski]

Just for visibility, I opened nebari-dev/nebari-docs#449 to update documentation on running tests which solves some of the issues mentioned above, but many remain.

[viniciusdc]

Based on what was discussed last week, this is not expected behavior, and as you just suggested, the docs for those tests env variables are lacking, I would recommend to not only addressing those in our docs, but also having an extra check for the current available credentials to at least give the user more visibility on whats happening

[marcelovilla]

@krassowski do you think we can close this or is there anything that we need to address?

[krassowski]

I believe some workflows are still failing on forks. My fork may not be the best example as I still had develop as default so it did not trigger all workflows when syncing

[krassowski]

Also I get spam like this:

I could just disable disable these workflows or they could have a condition to only apply in this repo.

[viniciusdc]

I could just disable disable these workflows or they could have a condition to only apply in this repo.

A similar control variable has been introduced: #2632, though I agree that's still not the best scenario. Those tests, in particular, should not run from a fork.