End to end testing

End-to-end testing framework

End-to-end tests and support code is found in the e2e directory.

Good Practices for creating tests

Use our helper libraries for executing commands instead of the standard library

Package: github.com/sylabs/singularity/internal/pkg/test/exec

Verbose struct declarations for test cases

Example

        {
                desc:            "signed image known signature require authenticated",
                srcURI:          "library://alpine:3.8",
                unauthenticated: false,
                expectSuccess:   true,
        }

Check all cases

Consider a test that runs a command and you want to verify that things work when they should work and that they fail when they should fail. You can include a field in a struct that indicates the expected result:

    type testcase struct {
        expectSuccess bool
        // ...
    }

and then in the test code, you could have something like:

    result := cmd.Run(t)

    if tc.expectSuccess && result.ExitCode != 0 {
        // expect success, but command failed
    }

    if !tc.expectSuccess && result.ExitCode == 0 {
        // expect failure, but command succeeded
    }

this looks fine, but it's missing two cases:

    if tc.expectSuccess && result.ExitCode == 0 {
        // expect success, command succeeded
    }

    if !tc.expectSuccess && result.ExitCode != 0 {
        // expect failure, command failed
    }

The reason why you want to consider the other two cases is because it makes the test easier to read, and it makes it explicit that there are additional checks that are performed only in specific cases, for example, if the command succeeded, you might want to check the output or if the command failed, you might want to check for temporary files.

Note that this is simpler to write in this form:

    switch {
    case tc.expectSuccess && result.ExitCode == 0:
        // expect success, command succeeded

    case !tc.expectSuccess && result.ExitCode != 0:
        // expect failure, command failed

    case tc.expectSuccess && result.ExitCode != 0:
        // expect success, command failed

    case !tc.expectSuccess && result.ExitCode == 0:
        // expect failure, command succeeded
    }

Running end-to-end tests

Common patterns that could be refactored into libraries

• Temporary file/directory creation and clean up.

Environment set up

The main test harness sets up some environment variables so that tests can access common values:

• E2E_CMD_PATH: path to singularity's binary
• E2E_RUN_DISABLED: run disabled tests
• E2E_IMAGE_PATH: path to test image
• E2E_TEST_DIR: path to temporary test directory

In order to make it easy to access these values, include a call like the following near the top of the test:

	e2e.LoadEnv(t, &testenv)

where testenv has been declared like this:

var testenv struct {
	CmdPath     string `split_words:"true"`
	ImagePath   string `split_words:"true"`
	RunDisabled bool   `split_words:"true",default:"false"`
	TestDir     string `split_words:"true"`
}

Libraries

You can use any package from the standard library, including testing and iotest. Singularity already imports gotest.tools, too, so subpackages like golden, env and fs are also available.