kola: Add isolation=readonly|dynamicuser

Part of my war against duplicative comments in our kola tests.
We have a few tests that write a comment like this:

```
 # - exclusive: false
 #   - This test doesn't make meaningful changes to the system and
 #     should be able to be combined with other tests.
```
Of course, this comment is already redundant because the
meaning of the `exclusive` tag is defined canonically in coreos-assembler
(here) and copy-pasting that into every test that uses it would
be pointlessly verbose.

But - we can do one better.  Instead of having a test flag which
is mainly an "I promise not to mutate the system in a way which could
interfere with other tests", let's add a field that *enforces* this.
Then it doesn't need to be commented; we have a variety of
tests which are just "system inspection" (e.g. query rpmdb) and
run just fine with `DynamicUser=yes` and hence *cannot* affect
the system, and hence are inherently isolated from other concurrent
tests.

docs/kola/external-tests.md

@@ -264,6 +264,12 @@ is simple and is not expected to conflict with other tests, it should be marked
 `exclusive: false`. When the `exclusive` key is not provided, tests are marked
 `exclusive: true` by default.
 
+The `isolation` key can take two values: `readonly` and `dynamicuser`.  These
+are thin wrappers for the equivalent systemd options; `readonly` equals `ProtectSystem=strict`,
+and `dynamicuser` means `DynamicUser=yes`.  Setting either of these options
+also implies `exclusive: false`.  Use these for tests that are mostly about
+read-only system inspection.
+
 The `conflicts` key takes a list of test names that conflict with this test.
 This key can only be specified if `exclusive` is marked `false` since
 `exclusive: true` tests are run exclusively in their own VM.  At runtime,

mantle/kola/harness.go

@@ -826,6 +826,7 @@ type externalTestMeta struct {
 	AppendKernelArgs          string   `json:"appendKernelArgs,omitempty"`
 	AppendFirstbootKernelArgs string   `json:"appendFirstbootKernelArgs,omitempty"`
 	Exclusive                 bool     `json:"exclusive"`
+	Isolation                 string   `json:"isolation"`
 	TimeoutMin                int      `json:"timeoutMin"`
 	Conflicts                 []string `json:"conflicts"`
 	AllowConfigWarnings       bool     `json:"allowConfigWarnings"`
@@ -952,6 +953,16 @@ func registerExternalTest(testname, executable, dependencydir string, userdata *
 		return errors.Wrapf(err, "Parsing config.ign")
 	}
 
+	switch targetMeta.Isolation {
+	case "readonly":
+	case "dynamicuser":
+		targetMeta.Exclusive = false
+	case "":
+		break
+	default:
+		return fmt.Errorf("test %v specifies unknown isolation=%v", testname, targetMeta.Isolation)
+	}
+
 	// Services that are exclusive will be marked by a 0 at the end of the name
 	num := 0
 	unitName := fmt.Sprintf("%s.service", KoletExtTestUnit)
@@ -980,6 +991,16 @@ Environment=KOLA_TEST_EXE=%s
 Environment=%s=%s
 ExecStart=%s
 `, unitName, testname, base, kolaExtBinDataEnv, destDataDir, remotepath)
+	switch targetMeta.Isolation {
+	case "readonly":
+		unit += "ProtectSystem=strict\nPrivateTmp=yes\n"
+	case "dynamicuser":
+		unit += "DynamicUser=yes\n"
+	case "":
+		break
+	default:
+		return fmt.Errorf("test %v specifies unknown isolation=%v", testname, targetMeta.Isolation)
+	}
 	if targetMeta.InjectContainer {
 		if CosaBuild == nil {
 			return fmt.Errorf("test %v uses injectContainer, but no cosa build found", testname)

tests/kola-ci-self/tests/kola/exclusive-readonly

@@ -0,0 +1,7 @@
+#!/bin/bash
+# kola: { "isolation": "dynamicuser" }
+set -xeuo pipefail
+
+id=$(id -u)
+test "$id" '!=' 0
+echo "ok dynamicuser"