Devops: Add tox environment that runs tests with sqlite backend

[agoscinski]

The problem with the current tox test environments is that they do require psql database which require a service running in the background. With the merge of #6425, we can use the sqlite backend to run tests more encapsulated with less dependencies on global configurations.

I still need to figure out how to set up a ssh config in a virtual-environment-friendly way. We could just run the .github/workflow/setup_ssh.sh file, but then it messes too much with existing ssh configuration. One could add an include in ${HOME}/.ssh/config to include another config file that is stored locally in the tox environment.

Include ${AIIDA_DEV_TOX_ENVIRONMENT}/custom_ssh_config

This just adds one line to the user config, that only works inside the tox environment. So does not have any effect outside of the environment to the functionality of ssh.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 77.85%. Comparing base (ef60b66) to head (8f66720).
Report is 181 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6450      +/-   ##
==========================================
+ Coverage   77.51%   77.85%   +0.35%     
==========================================
  Files         560      566       +6     
  Lines       41444    42088     +644     
==========================================
+ Hits        32120    32765     +645     
+ Misses       9324     9323       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[danielhollas]

@agoscinski FYI One another improvement that I wanted to do but did not get to is to make graphviz dependency optional i.e. to automatically skip tests that need it if it is not available. We already do this for other external optional dependencies (e.x. xcrysden et al).

[sphuber]

@agoscinski what is the status for this PR?

[agoscinski]

I wanted to allow that the developer can set an ssh key that is not ~/.ssh/id_rsa. Some tests however rely on ssh_auto which determines the ssh key automatically which basically just checks for ~/.ssh/id_rsa and other encryption protocols. Changing the default .ssh folder is I think not possible with tox.ini (at least not in a convenient way). For these ssh_auto tests I would monkey patch the SshAutoTransport instance to set the ssh key only in case AIIDA_PYTEST_SSH_KEY is set. In the CI we still can use the default key, so this monkey patching only effects tests that run locally and set this environment (so I would remove my current changes in the .github/* files).

There are two ssh_key fixtures (tools/pytest_fixtures/orm.py and manage/tests/pytest_fixtures). I feel like this should be cleaned up (removing one), but I did not want to touch it for now in this PR. I am generally not sure about the usefulness of creating the ssh key, since we need to add it to authorized keys so we are able to ssh to localhost. At least my understanding of the tests that uses the ssh_key fixture is that the generated ssh key is rejected and then it fall backs to the default one. Keeping the logic that generates a temporary key gives the wrong impression that the generated ssh key is really used. In my opinion we should do something like this

@pytest.fixture(scope='session')
def ssh_key(tmp_path_factory) -> t.Generator[pathlib.Path, None, None]:
    import os
    if (ssh_key_path := os.environ.get('AIIDA_PYTEST_SSH_KEY')) is not None:
        yield pathlib.Path(ssh_key_path)
    else:
        yield None # fallback to default key

We could move the logic that generates a new key to a new fixture, but I have the impression that it is not really used anywhere, so why bother. Since running these tests requires some global configuration, I would also mark these tests (in the pytest sense) with something like ssh_localhost.

[sphuber]

Thanks @agoscinski left a few comments

[sphuber]

I take it this is necessary because you do ${{ github.workspace }} in the GHA workflow. But wouldn't it be better to use the temporary file system or ${HOME} as well there? Or do those options not work?

[agoscinski]

One can also put the key to some tmp folder, but then one has to add each generated key to authorized_keys which fills up the file over time (as the tmp folder will be deleted). Since the authorized_keys is a user file that one cannot just be deleted I rather don't do this. So I did the intermediate solution and put it in the project folder. ${HOME} would also work, maybe this is even better since a project folder might be also deleted frequently.

[agoscinski]

probably not needed

[agoscinski]

Will check if this work and remove the ssh part into another PR, because it is orthogonal to this one

[agoscinski]

needed to specify for pre-commit

[agoscinski]

I removed now the ssh part and will reintroduce this in another PR

[agoscinski]

I would still keep the commit structure and merge them as Rebase and merge but the applied review commit has to be splitted up. I just did not want to do it, to simplify review. So if this is approved I can do it.

[sphuber]

Thanks @agoscinski . I approve the changes, please rebase the last three commits in the first four and we can rebase merge

[agoscinski]

tests/engine/test_process_function.py::test_submit_launchers has failed, keeping log in case this becomes important to debug
test_submit_launchers.log

[sphuber]

Thanks @agoscinski