System test framework

[ikerexxe]

As discussed at length, this is the implementation of the new system
tests framework for shadow. This is a proof of concept that contains the
key elements to be able to run basic user (i.e. useradd, usermod) and
group (i.e. usermod) tests. If you like the framework the rest of the
functionality will be added in the future.

Some useful facts:

• It is implemented in python
• It is based on pytest and pytest-mh
• It works on all the distributions that are part of our CI
• It can be run in the cloud (VM or container) as well as on-premises
• After the execution of each test the environment is cleaned up
• Logs and other artifacts for failed tests are collected
• It has a rich API that can be extended and extended to cover new
  functionalities

In addition, the PR contains a set of tests that make use of the framework
to test the functionality. These tests are located under tests/system/tests.
They offer a good example of the high-level API and how to model the test
cases.

Finally, the system test are run in the CI. This way we make sure that the
code can run successfully and avoid any regressions.

Closes: #835

[ikerexxe]

@hallyn I'd appreciate your feedback on this PR, particularly the tests in the tests/system/tests folder. This PR is quite extensive, so I'd like to focus our initial review on these tests. They showcase the new testing framework and how it interacts with the shadow utilities and other tools. Your input on this core part will be valuable as we move forward with the review.

[hallyn]

Thanks for the ping, I'm putting this on my next_actions list.

[ikerexxe]

For the moment, let's just ignore the CI errors. I'll fix them when I figure out how to do it

[alejandro-colomar]

How should I run the tests in my computer?

Also, are those tests (potentially) destructive? That is, do I need to run them inside a container?

Or does the test suite spin up containers as necessary?

[hallyn]

Hm, but the script does set -e, so result.stdout_lines could in theory be empty?

I guess it'll just give us an indexerror so no big deal? However, catching a return value != 0 early might be less confusing

[ikerexxe]

How about the following?

if not result.stdout_lines:
    return PurePosixPath(result.stdout_lines[-1].strip())
else:
    return -1

[pbrezina]

It shouldn't be empty, if the script fails it raises ProcessError from run since raise_on_error=True by default.

[hallyn]

I'm only partly through this (at patch 5), but it looks good so far, thanks.

[ikerexxe]

How should I run the tests in my computer?

Also, are those tests (potentially) destructive? That is, do I need to run them inside a container?

Or does the test suite spin up containers as necessary?

The instructions from doc/contributions/build_install.md still apply. Running them will create the container, build and install shadow and run the tests inside the container.

You can of course run the test on you own system, but I'd recommend against it since you may break your system with a shadow change. I plan to extend the documentation to explain how the framework works, the API, how to integrate it, etc. But that will come in another PR, once we all agree that the PoC is in a good shape and it can be merged.

I'm only partly through this (at patch 5), but it looks good so far, thanks.

So you've already gone through the first test cases. What do you think about them? Is the API clear to manage shadow and other utilities clear enough?

[alejandro-colomar]

How should I run the tests in my computer?
Also, are those tests (potentially) destructive? That is, do I need to run them inside a container?
Or does the test suite spin up containers as necessary?

The instructions from doc/contributions/build_install.md still apply. Running them will create the container, build and install shadow and run the tests inside the container.

You can of course run the test on you own system, but I'd recommend against it since you may break your system with a shadow change. I plan to extend the documentation to explain how the framework works, the API, how to integrate it, etc. But that will come in another PR, once we all agree that the PoC is in a good shape and it can be merged.

Just to check if I've understood that:

So, if I run share/container-build.sh, this new test suite will be run?

[ikerexxe]

So, if I run share/container-build.sh, this new test suite will be run?

Yes, the test suite will run within share/container-build.sh

[ikerexxe]

Ok, now the CI is in the state where I wanted it to be last week. Alpine build is failing due to https://gitlab.alpinelinux.org/alpine/aports/-/issues/16633. This is already fixed but it takes some time for those changes to arrive to users.

[hallyn]

So you've already gone through the first test cases. What do you think about them? Is the API clear to manage shadow and other utilities clear enough?

I think so, yes.

[pbrezina]

Hi, I went over the code briefly.

I think this needs clean up. It looks like you copy&paste lost of stuff from sssd-test-framework that you actually don't use. I suggest you to go over it and remove what is unused (like SSHKillableProcess, BaseTopologyController, ShadowTopologyController) to simplify the code.

Some things, maybe, you do not need to copy&paste but import it instead (like stuff in tools.py). But that is of course up to your preference.

[ikerexxe]

Some things, maybe, you do not need to copy&paste but import it instead (like stuff in tools.py). But that is of course up to your preference.

I'd prefer to avoid any dependency towards a project with no direct relation to this one (i.e. sssd-test-framework). Shadow's test framework should be smaller and way simpler, even simpler than I expected from your last comments.

[ikerexxe]

@alejandro-colomar @hallyn it'd be nice to have a final review from your side

[alejandro-colomar]

@alejandro-colomar @hallyn it'd be nice to have a final review from your side

Mine will be more an opinion than an actual review:

I don't understand Python. I can maybe read 5 or 10 lines for a small script, but this is huge, and inscrutable to me.

I wish this was done in bash(1), which is a language I can read much better (and I guess many Linux/Unix programmers too, but I don't know). However, the old framework was written as shell scripts, and I only used it maybe once or twice, so I can't complain.

So if you and Serge understand this framework and make it easy for others to use, it's a net improvement. I'll use it, but won't be able to contribute to it.

Thank you for your work! I appreciate it, even if I don't understand it. :-)

Anyway, if I ever have the time to write a testing framework in bash(1), I'll let you know. For the time being, this is better than no tests (nobody was adding tests with the old test suite). I have other food in my plate for the time being. :-)