Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tests: Add pytest and nushell based tests #625

Merged
merged 3 commits into from
Jun 24, 2024
Merged

tests: Add pytest and nushell based tests #625

merged 3 commits into from
Jun 24, 2024

Commits on Jun 24, 2024

  1. tests: Prune testcloud cache on each run 

    Because it's not a cache, it's a copy (there's no cache invalidation,
GC, etc.)

Signed-off-by: Colin Walters <walters@verbum.org>
    @cgwalters
    cgwalters committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    cc42685 View commit details
    Browse the repository at this point in the history

  2. dockerignore: Exclude more things 

    This avoids doing a container rebuild when touching the tmt
bits for example.

Signed-off-by: Colin Walters <walters@verbum.org>
    @cgwalters
    cgwalters committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    4874db3 View commit details
    Browse the repository at this point in the history

  3. tests: Add pytest and nushell based tests 

    I've been trying to keep this project in "one" programming
language by writing even tests in Rust...but specifically
for our integration tests it's pretty painful not just to
compile them but have to deal with baking them into the base image.

The tmt framework is very GHA like in that it scrapes the
git source tree and copies it into the target environment, which
works really well with scripts.

Now, if you know me you know I am not a fan of dynamic programming
languages like bash and Python. I'm one of those folks that actually
tries to use Rust for things that feel like "scripts" i.e. they're
*mostly* about forking external processes (see the xtask/
crate which uses "xshell").

Some of our testing code is in Rust too. However...there's a giant
tension here because:

- Iteration speed is very important for tests and scripts
- The artifact being an architecture-dependent binary pushes us
  to inject it into container images; having the binary part
  of the bootc image under test conceptually forces us to reprovision
  for each test change, which is super expensive

Most other people when faced with the testing challenge would
just write shell scripts (or Python); that's definitely what tmt
expects people to do.

The podman project has a mix of a "bats" suite which is all
bash based, and a Go-based framework.

The thing is: bash is easy to mess up and has very little ability
to do static analysis. Go (and Python) are very verbose for forking external
processes.

I've been using https://www.nushell.sh/ for my interactive shell
for quite a while; I know just enough to get by day to day
(but honestly sometimes I still type "bash" and run a few things there
 that I know how to express in bash but not nu)

Anyways though, nushell has a lot of desirable properties for
tests (which are basically scripts):

- Architecture independent
- Running an external process requires zero ceremony; it's the
  default!
- But it *is* easy to e.g. scrape JSON from an external binary
  into a rich data structure
- A decently rich standard library

The downside is, it's a new language. And in the end, I'm
not going to say it's the only way to write tests...maybe we
do end up with some more bash. It wouldn't be the end of the world.
But...after playing with this, I definitely like the result.

OK, and after some debate we decided to add Python too, so this
demos a pytest test.

Signed-off-by: Colin Walters <walters@verbum.org>
    @cgwalters
    cgwalters committed Jun 24, 2024
    Configuration menu
    Copy the full SHA
    9a758e3 View commit details
    Browse the repository at this point in the history