Make it possible to execute tests using tty as stdout/stderr?

[nsf]

As far as I understand when executing tests, the sub-process stdout and stderr are always buffered: https://github.com/constabulary/gb/blob/master/test/test.go#L181-L189

Perhaps there is a reason for that, for example if you do that in parallel. But some software can be confused by that decision and switches to different logging style. I understand maybe we should fix the subject software and/or use options it provides to force particular style of output. But also buffering output can be undesired when executing very long running tests and you want to be able to terminate them in the middle of the way if you notice somehing goes terribly wrong.

Example setup:

[nsf @ testgo]$ tree src
src
└── projectx
    └── main_test.go

gb vendor fetch github.com/Sirupsen/logrus

Contents of main_test.go:

package main

import (
    "github.com/Sirupsen/logrus"
    "testing"
)

func TestX(t *testing.T) {
    logrus.WithField("a", 42).WithField("b", 42).Info("Testing projectx")
}

Running with gb test -v:

[nsf @ testgo]$ gb test -v
projectx
=== RUN   TestX
time="2015-12-06T23:41:39+05:00" level=info msg="Testing projectx" a=42 b=42
--- PASS: TestX (0.00s)
PASS

Running with GOPATH=$PWD:$PWD/vendor go test -v projectx:

[nsf @ testgo]$ GOPATH=$PWD:$PWD/vendor go test -v projectx
=== RUN   TestX
INFO[0000] Testing projectx                              a=42 b=42
--- PASS: TestX (0.00s)
PASS
ok      projectx        0.002s

Here the output style used by logrus is different, because logrus runs a syscall to check if the stdout is a terminal or not. It does so by using TCGETS: https://github.com/Sirupsen/logrus/blob/master/terminal_notwindows.go#L15-L21

[davecheney]

@nsf thanks for raising this issue. To be honest i'm not sure how to approach this one. I see the need -- the test runner should have as little impact on the thing it is testing as possible -- or at least document the constraints it does place, like the explicit setting of PWD to the source directory of the test that cmd/go and gb both do.

There are a few things that I've changed in the way gb test works that I think make it more consistent compared to go test. The big one here is that output is always buffered, no matter how many packages are tested, which is a departure from the more nuanced way that go test works.

While I could set a rule that if there is only one package being tested then connect it to stdout and then figure out some way to manage the buffering of output, but this feels arbitrary. Also, I'm concerned that if tests are written to demand a tty connected to stdout, then they'll break unless run in this very specific scenario -- and that is a trap I want to avoid setting for myself.

So, for the moment, with no simple answer I'm going to sit on my hands on this one til a better idea comes along.

[nsf]

I understand your point. Well, it's not a big deal, I guess something we got used to using go test.

[davecheney]

@nsf just so I'm clear, am I right that you have to test one package at a time to achieve this behaviour ?

One idea to make this work, and get buffered output is to use a pty, and read the results of the test from the other end of the pty -- it's not much different to the pipe(2) approach that is in use now.

It won't be completely cross platform, but we can do fallback for windows -- and I suspect your logurus example also has caveats where windows is concerned.

[nsf]

Yes, I'm talking about testing one package at a time. go test doesn't run them in parallel as far as I understand. Believe it or not, never tried running tests on multiple packages at once.

[davecheney]

I reckon the pty trick could work, and probably be applicable for a wide
range of uses ... i'm not sure how appropriating ptys for test processes is
going to work out in general, but tmux and ssh get away with it just fine,
so I think we'll be ok.

This will need some plumbing in the x/term and x/sys/unix repos before this
can happen, so I cannot promise to turn around this feature in a short
time. If you want to work on it, feel free, otherwise I'll put it on the
backlog.

On Thu, Dec 10, 2015 at 1:06 PM, nsf notifications@github.com wrote:

Yes, I'm talking about testing one package at a time. go test doesn't run
them in parallel as far as I understand. Believe it or not, never tried
running tests on multiple packages at once.

—
Reply to this email directly or view it on GitHub
#472 (comment).