Configuration design

Introduction

This document describes the pieces within Kyua that need to be customizable by users, what the requirements of such configuration are, proposes different alternatives to manage these configuration files and concludes with the best option for our purposes.

As a general comment, the main point of this document is to have a clear idea of the technical requirements for the configuration files and to make an informed decision from the following options:

• Parse the configuration files with a hand-written lexer and parser.
• Parse the configuration files with a lex/yacc parser.
• Treat the configuration files as full-blown scripts and use Lua to interpret them.

Requirements

There are two main configuration domains in the context of Kyua: the configuration of the Kyua framework itself and the configuration of every test suite. The former is hardcoded into the source of Kyua, while the latter is defined by every test suite (which are third-party entities).

Configuration for every test suite

Test suites are collections of test programs referenced by a particular test suite configuration file. These configuration files are known as Atffiles in ATF and we will call them Kyuafiles in Kyua. In their most basic form, these files are a simple list of test programs to run. (You can think of them as Makefiles listing the targets to test and the directories to recurse into.)

In general, the layout of the system-wide test suite (/usr/tests) on disk looks like the following:

• Kyuafile: This references *, */Kyuafile or similar to recurse into the subdirectories automatically.
• libfoo/
  • Kyuafile: References *_test to run the test programs in this directory and includes detail/Kyuafile to run the tests in detail.
  • strings_test
  • files_test
  • detail/
    • Kyuafile: References *_test to run the test programs in this directory.
    • internal_test
• libbar/
  • Kyuafile: References *_test to run the test programs in this directory.
  • button_test
  • window_test
• baz/
  • Kyuafile: References *_test to run the test programs in this directory.
  • integration_test

There is a very important detail to consider in the scheme above: the top-level Kyuafile exists to mark the root of the test suite and its only purpose is to recurse into all subdirectories. These subdirectories are not known a priori: consider libfoo, libbar and baz being third-party tools shipped as binary packages, all of which place tests into subdirectories of /usr/tests. It is important to prevent these packages from having to register and unregister themselves from the top-level Kyuafile file because this becomes a maintenance nightmare in binary packages.

The test programs themselves (the executables matching *_test) are not very special; any binary can be a test program as long as it follows the expected test interface. According to the interface, test programs must be configurable by end users through arbitrary key/value pairs representing configuration variables. As an example, tests cases that try to reproduce a race condition may define an iterations configuration variable that specifies how many times to repeat a racy action before considering the test a success; such iterations property could be used in the different test programs of a test suite. The test suite may want to provide a default, reasonable value for such variables but the end user needs to be able to tune them.

With the above requisites in mind, files representing test suites must support the following features:

• List of the test programs to run (and optionally subsets of their test cases). It must be possible to specify these test programs by their name or by a glob. (We probably want to use globs rather than regular expressions when referring to files to avoid confusion.)
• Definition of what properties are valid within the realm of a test suite. This is not enforced by ATF but it is something to consider, specially if we make it possible to define the types of the properties. That said, this may have nasty interactions with the current require.config property of test cases.
• Definition of default values for the test suite configuration properties (see previous point). Given the example above, we must be able to say iterations=100 by default, and get this value propagated to the test programs.
• Recursion into subdirectories (potentially by glob). A test suite can span over multiple subdirectories and we want each subdirectory to have its own Kyuafile file. This can be implemented either by explicit recursion or by file inclusion; see the corresponding section below for details.
• Comments.

Ideally, the contents of the test suite configuration files would be purely declarative to simplify reasoning about the behavior of Kyua on a particular test suite. Note that the above list does not define test-suite hooks on purpose.

Configuration for the Kyua framework

The runtime engine of Kyua is a command-line tool and, as is common in tools of this nature, provides features that need to be customizable by the end users. The areas that must be covered by a configuration file are as follows:

• Settings that control the behavior of the Kyua runtime engine per se. These settings are explicitly referenced by the source code of Kyua so they must either have reasonable default values or be optional. Some of these settings include: architecture, platform and unprivileged-user. Note that these settings must be propagated to test programs like any other test suite configuration variable; in occasions, the test programs will want to query the architecture Kyua is running on and they must be able to do so.
• User-defined values that override the default test suite configuration variables set in the Kyuafile files. Even if these files define, say, iterations=100, the end users must be able to permanently override such settings.
• Programmable hooks for the framework so that users can extend its functionality without rebuilding the sources (this is very important; being binary-friendly is a major design goal of Kyua and ATF). ATF currently provides this feature to allow users to extend the generated reports with custom information. We can leverage the same idea here and extend it to other areas of the framework. For example, hook into the execution of test cases to be able to conditionally save .core files or to format the way test results are printed on the screen. (The problem is where to draw the line: what do we put in hooks and what we don't? But this is out of the scope of this design document.)

A configuration file able to express the above settings must support the following features:

• Definition of the Kyua-specific properties (name/value pairs).
• Definition of overrides for the test suite configuration variables (those defined in Kyuafile files and/or required by the test cases to run).
• Definition of hooks as functions.
• Comments.

Considerations on test suite configuration variables

If you are an avid reader, you may have noticed that the test suite configuration variables have a default value defined in the test suite configuration file and also have an override in the user configuration file. You may also have noticed that the test suite configuration variables are just that, specific to a test suite; therefore, there may be different test suites on the same system, each exposing their own variables, all of which may interfere with each other.

There is a clear need to scope the test suite configuration variables by the test suite name. By scoping the names, different test suites can define their own variables without overlapping names.

As already mentioned, the variables specific to Kyua (e.g. unprivileged-user) must also be passed down to the test programs. This means that these variables can also conflict with the per-test-suite variable names, so they must also be split on their own namespace.

Design decisions

One format vs. multiple formats

One thing the criticisms of ATF is that it has a different file format for every possible external file: there is the format for configuration files (application/X-atf-config); there is the format for the Atffiles (application/X-atf-atffile); and there is the format for the hooks (plain shell scripts). As it turns out, this is bad for usability: the user must learn the syntax of every file type when editing one of these files. And it is also bad from a programming point of view because there is a need for three parsers/interpreters.

The alternative is to go with one single format for all possible files. The disadvantage is that such format must cover the requirements of all file types and, among these possible files, there is the need for hooks. Hooks, by definition, are imperative scripts. Therefore, having a common language for every configuration file means that such language must be imperative, which effectively prevents the configuration files from being purely declarative. On the other hand, having a single format makes things much easier for the end user and easier for implementation purposes (one single parser to write and validate).

That said, the fact that the configuration files could ideally be purely declarative is not a strong enough reason to warrant their own parsers.

Decision: Kyua will have one single format for all external configuration files.

Parser implementation

There are several alternatives for the implementation choices of the configuration file parser.

The first option is to implement a hand-written parser. The clear advantages of this approach are that the language can have have any syntax we like, and that we do not have to depend on any external tools nor libraries. The disadvantages: it is too much work to implement a correct and efficient custom parser, and yet-another-configuration-language is a bad idea for usability reasons. ATF chose this route to avoid external dependencies on tools and, today, its code contains an ugly attempt of generalizing the parsing of the different file types, all of which are confusing to end users because they are non-standard.

The second option is to implement the parser using the traditional lex/yacc combination. This is similar to the previous point in that it allows us free will in the design of the language but the implementation of the parser becomes easier. That said, this does not prevent us from having to implement extensive tests for all possible input cases, and also means that the code depends on external tools. This is all fine, but the common denominators of lex/yacc do not play well with C++, and depending on the particular GNU implementations (Flex and Bison) is not something Kyua should do.

The third option is to avoid writing the parser ourselves altogether by leveraging some other existing configuration language. Given our requirements for the configuration, we need to be able to provide hooks for run-time extensibility, which means that our files need to support scripting. We also must consider that the parser needs to be small because it has to be imported into the NetBSD tree, which rules out things like Python. The only reasonable choices are either shell or Lua.

Shell is not really an option: ATF chose to implement hooks this way and, while it was a good idea at the time, the results have proven it was not. Shell scripts have too much external dependencies (binaries, environment variables, ...), they are noticeably slow, the syntax is too fragile and there is no way to provide a clean interface between the native code and the interpreted code.

The only other realistic alternative is Lua, a very small language interpreter that is fully customizable from C. Lua already ships with the NetBSD base system, which is a plus, but we must not forget that we want other BSDs to be able to import Kyua into their system. That said, importing Lua alongside Kyua in, e.g. the FreeBSD base system, does not seem like a big barrier: the interpreter is small and the license is BSD-compatible. The advantages of Lua are: we do not have to implement nor test a language parser. The disadvantages: we rely on an external tool and the fact that we use Lua prevents us from having purely declarative configuration files.

Decision: use Lua as the parser and format for the configuration files. I don't personally fancy the idea of hacking another custom parser, I don't see how lex/yacc play nicely with C++ (been there, done that) and I don't want to have to redesign a new language that will be incomplete and completely unknown to users. Lua lets us avoid all this for a couple small tradeoffs.

Recursion vs. inclusion

In the list of requirements above, we mentioned that test suite configuration files must have a way to specify that some subdirectories must be recursed into to execute their test programs. We can do this by explicitly recursing into directories or by including their Kyuafile files.

ATF implements this by specifying which directories (read: not files) to recurse into. This is problematic: if we do this, we lose knowledge of the file name that contains the test suite definition. Some users have asked before for a feature that allows them to specify different test suite configuration names, and with this approach it is hard to provide. On the other hand, this makes the root-level Atffile trivial: it only needs to do recurse * and it would enter all the subdirectories. Additionally, this makes it possible to parse Atffiles along the way: at startup, you only need to load the Atffile in the current directory to know what to do next, because such file cannot include other files.

The alternative is to let the Kyuafile files explicitly include other files. In this way, we can recurse into subdirectories by doing something like include detail/Kyuafile from within libfoo/Kyuafile. One advantage is that a Kyuafile file has exact control on what file in each subdirectory will be processed (e.g. foo.suite can trivially include dir/bar.suite without relying on the same basename). Another advantage is that parsing a given Kyuafile file and its dependencies yields a complete view of what needs to be run; this is very important for the implementation of parallel execution of tests, as a global view of the tree gives more degrees of parallelization. (I find Recursive Make Considered Harmful very insightful in this area.) The drawback comes in when implementing the top-level Kyuafile: does it have to assume that all the subdirectories have a Kyuafile in them? Must we do this scriptable so that distributions of Kyua can customize how the top-level Kyuafile behaves?

Decision: Kyua will implement explicit inclusion of files in subdirectories. The fact that this approach gives a complete view of the tests to run means that parallelization efforts later on will yield more impressive time savings. We will implement inclusion by globs so that the top-level Kyuafile file can just perform a include */Kyuafile or just perform a walk of the entries in the current directory and chose those that have a Kyuafile file inside.

(As a side effect, note that this addresses a concern raised by FreeBSD developers: what if /usr/tests is shared among different testing frameworks? How can each of them know what subdirectories belong to themselves? If we go this route, the fact that we only recognize directories with Kyuafile files means Kyua can coexist with other testing frameworks in the same directory.)

Variable scoping

As mentioned above, the configuration variables passed to test programs are generally defined on a test suite level, although there is a set of variables that affect the runtime engine and thus are global. To prevent name clashes, we must scope these variables with their test suite names. This will allow users to override the default values with the correct granularity.

Decision: Kyua will have the following scopes (specific naming to be decided later):

• kyua.<variable-name>: For example: kyua.unprivileged_user.
• suite.<suite-name>.<variable-name>: For example: suite.netbsd.file_systems.

Note that for compatibility reasons we will have to continue to export the old-named variables to test programs. In particular, this means that, for example, the globals.kyua.unprivileged_user variable will be passed also as unprivileged-user to test programs.

File layout

Users must be able to override any configuration settings of Kyua either through the command line, through a configuration file on their home directory or both.

Additionally, test suites must be able to install sample configuration files for their custom set of variables. An option is to have a centralized file that holds the definition of all possible variables; this does not play well with binary packages because they would need to manually mess around with a central file. Another option is to have a per-test-suite file: this allows test suites to blindly install their own configuration file into <sysconfdir>, ensuring that they do not override any settings for the engine nor for other test suites.

Decision: As follows depending on what configuration files we are talking about. (Specific naming of the files to be decided later.)

The resolution order for the Kyua runtime configuration is:

1 Load one of: * The user-specific configuration file from <home>/.kyua/kyua.conf if it exists. * Otherwise the system-wide configuration file from <sysconfdir>/kyua/kyua.conf if it exists. 1 Apply any overrides specified through the command line. E.g. kyua test -v kyua.architecture=amd64.

Things are a bit different when specifying the configuration of the test suite variables:

1 Load the test suite configuration file (Kyuafile) to get the test suite name and the default values for the variables (if any). 1 Apply any overrides defined in one of: * The user-specific configuration file from <home>/.kyua/suites/<suite>.conf if it exists. * Otherwise the system-wide configuration file from <sysconfdir>/kyua/suites/<suite>.conf if it exists. 1 Apply any overrides specified through the command line. E.g. kyua test -v suite.netbsd.file_systems=ffs.

This scheme allows a user to completely override any settings defined at a system level or at the test suite level without having to modify any of the existing files. He can do so in two manners: overrides on the command-line always take preference over settings stored in any configuration files; overrides provided in the "overlay" configuration file (i.e. the file in the home directory) always override system-wide settings and test-suite-specific settings.

Conclusion

This document has explained what the requirements for Kyua configuration files are and what the different alternatives to implement such files are. The main summary of the decisions taken is that Kyua will use Lua to process both its configuration and the configuration of the test suites, and will also use Lua to offer extensibility to users by means of runtime hooks.

Lua is a realistic possibility because it has been imported into the NetBSD source tree and will ship with NetBSD 6.0. Due to the nature of Lua and its license, it seems like this choice will not be a big barrier for other BSDs to consider importing Kyua into their base system.