ci: Test full-emulation cosa run

[cgwalters]

ci: Test full-emulation cosa run

We should have some CI that verifies this works.

---

devshell: Don't output status if not on a tty

This is just better behavior for the case of e.g.
cosa run -x "somecmd" > out.txt to execute a command and
capture the output - we don't want to intermix status stuff in
there.

Further, this makes it easier to test the code when not on
a tty (in e.g. CI).

---

[jlebon]

We have a "CLI Tests" stage below where we already run at least one other test. It'd be nice to keep it all centralized there.

[cgwalters]

Yeah, but I didn't want to wait for all the kola tests to run to see the result of this 😄 Hmm...will it work to create a separate pod like this?

diff --git a/.cci.jenkinsfile b/.cci.jenkinsfile
index 895f6e359..fabaf2f01 100644
--- a/.cci.jenkinsfile
+++ b/.cci.jenkinsfile
@@ -52,9 +52,6 @@ pod(image: imageName + ":latest", kvm: true, cpu: "${cpuCount}", memory: "${memo
         utils.cosaCmd(cosaDir: "/srv", args: "buildupload --dry-run s3 --acl=public-read my-nonexistent-bucket/my/prefix")
     }
 
-    stage("cosa run tests") {
-        shwrap("./tests/test-cosa-run.sh")
-    }
 
     // Random other tests that aren't about building. XXX: These should be part of `make
     // check` or something and use dummy cosa builds.
@@ -65,3 +62,11 @@ pod(image: imageName + ":latest", kvm: true, cpu: "${cpuCount}", memory: "${memo
         """)
     }
 }
+
+pod(image: imageName + ":latest", kvm: true, cpu: "${cpuCount}", memory: "4096Mi") {
+    checkout scm
+
+    stage("cosa run tests") {
+        shwrap("./tests/test-cosa-run.sh")
+    }
+}
\ No newline at end of file

[jlebon]

Yeah, but I didn't want to wait for all the kola tests to run to see the result of this 😄

Ahh gotcha. :) Feel free to iterate that way and then once it passes, we move it down? (Either here or in a follow-up).

Hmm...will it work to create a separate pod like this?

It's syntactically valid, but it won't run it in parallel if that's what you want. You'd need a parallel for that, but I'm not sure it's worth the increased complexity.

[cgwalters]

It's syntactically valid, but it won't run it in parallel if that's what you want. You'd need a parallel for that, but I'm not sure it's worth the increased complexity.

Hmm, it seems useful in general to start breaking up some of the tests into parallel pods, no?

[jlebon]

Indeed, it's useful but makes the Jenkinsfile harder to follow and the BlueOcean view is severely affected. (Obviously there are bugs there that need to be addressed.) Eventually though I'd like to be able to better split tests into separate jobs and separate commit status updates with dependencies between each other which would also solve this.

Anyway LGTM, but yuck that BlueOcean view is a mess.

[cgwalters]

Anyway LGTM, but yuck that BlueOcean view is a mess.

Yeah it is. There is another option of course here...we could use Github Actions for this test. We don't actually need /dev/kvm for this test notice, and further GHA has a nicer/more flexible visualization than BlueOcean around parallel and dependent tests.

[cgwalters]

Ahh right, needed to fix cosa run to handle the "not on a tty" case.

[jlebon]

One thing that could help the BlueOcean thing is to make it parallel with e.g. the "Unit tests" stage or cosaBuild. The latter is longer anyway than this test.