Quicktest

Quickly and automatically test systems inside Quickemu virtual machines

Quicktest runs test cases automatically in Qemu virtual machines using Quickemu by injecting keypresses and reading the screen with Tesseract OCR.

Introduction

Quicktest is a companion for the excellent Quickemu that attempts to automate some testing usually performed by volunteers in their own time, before release.

However, it could be expanded beyond that initial remit.

Video Demo

Here's a ~6 minute video explaining Quicktest, with a (speeded up) demo.

Here's an asciinema of running a test case in quicktest.

This is the resulting video which is a timelapse of all the screenshots taken during the above session.

Goals

At a high level, this is what Quicktest is all about:

• Providing a utility that enthusiasts can use to automate some OS tests in the Linux community
• Creating a set of tests that are understandable and maintainable
• Enabling technically savvy enthusiasts to develop and contribute new tests

Quick Start

Quicktest requires quickemu (and quickget), qemu, ffmpeg, socat, convert (from imagemagick) and tesseract : e.g.

sudo apt install socat ffmpeg imagemagick tesseract-ocr

1. Follow the installation instructions for quickemu 4.9.3 or newer

2. Install tesseract-ocr along with appropriate language packages, e.g. tesseract-ocr-fra for French.

   sudo apt install tesseract-ocr

3. Clone this repo

   git clone https://github.com/quickemu-project/quicktest

4. Run a test

   This will download an ISO image for Alpine Linux using quickget, start a VM using quickemu then run automated tests.

Note

Do not interact with the Qemu window as it may invalidate the test results.

cd quicktest
QUICKEMU_WIDTH=800 QUICKEMU_HEIGHT=600 ./quicktest test_boot_to_login alpine v3.19

There are many options, which can be overridden at launch.

DEBUG=true \
QT_KEEP_SCREENSHOTS=false \
QT_KEEP_TESSERACT_TEXT=false \
QT_CREATE_TIMELAPSE=true \
QUICKEMU_WIDTH=800 \
QUICKEMU_HEIGHT=600 \
./quicktest test_boot_to_login alpine v3.19

Results will be found in ./results/.

tree results/
results/
└── alpine
    └── v3.19
        └── test_boot_to_login
            └── 20240513-133341
                ├── quicktest.log
                └── quicktest.mp4

Further tests

There are other tests that ship with Quicktest. List them all with quicktest -ls. We are keen to add to the selection of tests. Please consider writing and maintaining new tests. See WRITING_TESTS.md for some tips.

Ubuntu is the default OS when launching quicktest and 24.04 is the default release.

./quicktest test_install_entire_disk_with_defaults

To do a similar test with Ubuntu MATE 24.04:

For example, to do an install test of Ubuntu MATE 24.04:

./quicktest test_install_entire_disk_with_defaults ubuntu-mate 24.04

Philosophy

Quicktest aims to treat the machine being tested as a user would. That is, it injects keypresses and mouse clicks as a human tester would, and inspects the screen as a human subject would.

The aim is (where possible) to do everything necessary from 'outside' the virtual machine. That is, not to install software to operate the test machine inside the guest, but start the machine and use the keyboard and mouse only.

This is to ensure the guest OS is not tainted which could change the outcome of the test.

Enthusiastic users should be able to both run and contribute to tests. New tests are welcome as pull requests.

Important Notes

Quicktest main goal is to automate some of the testing required by Linux distro maintainers and release managers. It is not designed, nor should it be perceived to replace human testing. This is a convenience script.

Quicktest is very new and not widely tested yet. As such the features, names of options, file locations and behaviours may change before the first stable release.

MVP

The Minimum Viable Product (MVP) should be able to successfully run a test or two. The results should be human-understandable. The test case should be usable as a guide for further test cases, so will likely contain many of the common attributes of a typical test case.

As such things may be a bit messy, and unfinished. This is to be expected, as this was only started recently, and has had less modest development work put into it. Feedback is welcome. but please be kind.

• Interpret command line options
• Display command line help
• Display version string
• Display available tests
• Prepare and launch the required VM using quickget/quickemu
• Send all valid keypresses into the VM
• Interpret test script, to perform an installation of Ubuntu
• Generate log file
• Take screenshots
• Grep for text in screenshots
• Add option to wait, and take periodic screenshots - basically already there, just make numbers bigger
• Fix send_string upper case (likely related to next issue) (it's not)
• Debug and fix why some characters don't get sent to qemu (! and .)
• Eject medium and reboot at end of installation in install_entire_disk_with_defaults
• Add post-install reboot, and start the machine up - after reboot, login, first run wizard
• Test the GTK backend for Qemu rather than Spice - sadly same annoying dialog
• Complete one full test case - default install process
• Keep all screenshots and texts
• Make all log output prefixed with emoji
• Truncate all informational text to 80 cols
• Try scaling screenshots 300% to improve results
• Optimize getting words from screenshots to reduce the number of shots taken. e.g. title and body
• Add a variant test that disables network
• Add a test which does an install in a different language
• Debug fr_FR switch in QT_TEST_LANG=fr_FR ./quicktest test_install_entire_disk_with_defaults - relative path on load
• Work out why the i18n file won't load
• Add a very small distro as a ci test (alpine, test boot to grub only and power off)
• Figure out how to deal with tests written in other languages than en_GB
• Get QT_SCALE_IMAGE working
• • If using spice display, have an option to poke the annoying spice dialog, off by default
• Write a test for Ubuntu daily live
• Stitch together the screenshot_*.ppm files with 5 seconds wait on each frame as a video
• Maybe move VMs and ISOs elsewhere in a subdirectory like vm/
• Give screenshots meaningful names
• Break up test into named steps (usable in screenshot names)
• Consider shipping one big shell script, only keepin the tests and texts separate
• Figure out where to put lang texts and what to call them
• Allow configuration overrides in the environment
• Create quickcapture, which is a cut-down quicktest that only takes periodic screenshots and tesseracts them
• Figure out optimal tesseract settings
• Tidy up -ls output
• Force resolution of qemu window
• Thoughtfully consider config option naming
• Perhaps check the disk to see if it has already been written to
• Enable use of --status-quo to throw away disk commits (works via QUICKEMU_OPTS=--status-quo)
• Enable daily testing of a flagship distro a possibility

TO-DO

High

• Have default retry and interval settings for qt_wait_for_text - figure out how to pass function name?
• Fix French keymap - if it can be - as an example for non-en_US testing
• Add a non-Linux test as an example - e.g. CalibreOS
• Add a test case that uses an encrypted disk - reboot and test it encrypted it
• Figure out where buttons are on screen so maybe we can mouse click them
• Test mouse click top right, down arrow, right right, fire (maybe use --width --height to get mouse top right corner)

Medium

• Detect language rather than hard wire it?
• Add a function to hold a key down for a period rather than tap it
• If using spice display, have an option to poke the annoying spice dialog, off by default

Low

• Use escalating delay times rather than screenshotting every n seconds to test a condition
• Allow send_key_combo to take a number so we can do tabx2
• Offer a snapshot once complete
• Enable starting from a snapshot
• Optional notifications at each step
• Ensure quicktest works in WSL on Windows, so people can run tests outside of Ubuntu

Unprioritised

• Send mouse events into the VM
• Create quicktestrecorder to generate test files
• Write test for Haiku (haiku r1beta3 x86_64)

Open Questions

• How do we insert characters such as CJK characters into Qemu?
• How should this be packaged? (currently, clone the repo and run from that folder)
• Do we dynamically adjust delay timings based on detected disk/cpu speed of machine?
• Should we snapshot of the installation prior to starting at end?
• Should we allow skipping tests with a --start-on to make it skip tests you already did?
• Should we train Tesseract on the Ubuntu font?

Other uses of quicktest

• Automated application browser/website testing
• Automate complex installation processes (e.g. macos in quickemu)
• Add automation to OS installs that don't have automation as a convenience

Limitations

• Tesseract has trouble identifying text above 2560x1600.
• Qemu doesn't present a screen size below 30" so at high resolutions, hidpi is not enabled in desktops

EDID notes

• 2560x1600, xrandr reports a 30.2 inch monitor with edid unspecified, or edid=on specified.