kola testiso tests should check for badness in console/journal output

[jlebon]

Based on the report in coreos/fedora-coreos-tracker#1722, this is not currently happening, but should.

(This is another thing we should get for free if we fold kola testiso tests into kola run.)

[c4rt0]

If it's not pressing, I would love to look into this when I'm done with the issues I am currently working on.

[jbtrystram]

@c4rt0 folding testiso into regular tests is on my radar let's collab on this if you want !

[c4rt0]

@c4rt0 folding testiso into regular tests is on my radar let's collab on this if you want !

Great idea, I'd appreciate that.

[c4rt0]

I have a question.
In kola/testiso.go we have the result
which simply checks if there was an error while writing test results to output. Going further, the reporter sends FAIL if atLeastOneFailed. 
When comparing this to kola.go I see that this functionality is much more expanded. Once tests fail we implement runRerun, with even more checks within that function. At first I started implementing something like cmdRerun (https://github.com/coreos/coreos-assembler/blob/main/mantle/cmd/kola/kola.go#L157) but that is obviously outside of spec of this particular issue. My silly question is: what exactly do you mean by checking badness in console/journal output @jlebon?
Or maybe I should attempt implementing something like the mentioned above cmdRerun, simillar to the one in kola.go?

[jlebon]

Specifically what we want is two bits from kola:

1. calling CheckConsole() on the console output.
2. calling CheckMachine() on the system. That one is trickier because it takes a Machine, which we don't have here. That's one thing that'd get transparently fixed once we make the testiso tests regular tests. Kinda yucky, but one thing we could do for now is to have CheckMachine() take a closure instead for running ssh commands to the node.