Ginkgo should error if a package does not have a suite file

[benmoss]

Just wasted ~20 minutes of my life figuring out why Ginkgo couldn't find my test file before remembering that a suite file is required. Maybe there could be a warning?

[onsi]

hmm. sorry to sound dense but can you tell me exactly what you were doing and what you saw? Ginkgo doesn't need a *_suite_test.go file to realize this is a Ginkgo suite (it looks for an import of ginkgo). It does need there to be some sort of *_test.go file otherwise it should say "Found no test suites". If it sees *_test.go files but none of the import "github.com/onsi/ginkgo" it will think you are pointing it at a set of go tests and will simply run go test for you.

[benmoss]

I made a repo that shows the problem: https://github.com/benmoss/ginkgo-repro

If you try to run ginkgo . you get:

testing: warning: no tests to run
PASS

Ginkgo ran 1 suite in 505.647784ms
Test Suite Passed

Apologies if I was a little terse in my description, I'm gonna see if I can put together a PR later today :)

[onsi]

welp. that's a totes legit edge case.

We could do something ugly like bail if we can't find a line that matches RunSpecs. I wouldn't want to require there be a *_suite_test.go file as that is a convention not a rule. Alternatively we could parse the output of go test and look for testing: warning: no tests to run and say something like "did you remember to ginkgo boostrap?"

Either way, a PR would be awesome @benmoss

[andrehp]

I don't feel like this is a positive addition to ginkgo, showing a warning is ok, but breaking the build because a package doesn't have a suite test file is too extreme.

[onsi]

@andrehp we pulled in a change that prints a warning. Are you seeing builds break?

[andrehp]

Yes, I'm seeing breaking builds, I'd guess the problem is at line 467 https://github.com/onsi/ginkgo/blob/master/ginkgo/testrunner/test_runner.go#L467

It sets passed to false when it detects no tests were run.

[andrehp]

@onsi so, any chance of reverting this change? Or at least removing line 467?

[onsi]

Yes. Fastest way would be a PR that removes that line if you're up for submitting one.

…

On Apr 25, 2017, at 5:59 AM, Andre Hahn Pereira ***@***.***> wrote: @onsi so, any chance of reverting this change? Or at least removing line 467? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[andrehp]

Created it
#347

[benmoss]

FWIW this shouldn't be breaking your test suite if you don't have any tests in a package. It should only break in the case where you have Ginkgo tests but no suite file, in which case none of your tests are running anyway, which seems like a thing you'd want to know about.

Can you give an example of the problem you're encountering before we merge this? I feel like something else may be the problem here.

I've tested this matrix of possibilities:

package has go testing package test file: passes with ginkgo, passes with go test

$ ginkgo
=== RUN   TestBar
--- PASS: TestBar (0.00s)
PASS

Ginkgo ran 1 suite in 289.590825ms
Test Suite Passed

$ go test
PASS
ok      repl-1/bar      0.007s

package has ginkgo test file but no suite_test OR go testing tests: fails/exits 1 with ginkgo, passes/exits 0 with go test (with "testing: warning: no tests to run")

$ ginkgo
PASS
testing: warning: no tests to run
Found no test suites, did you forget to run "ginkgo bootstrap"?
Ginkgo ran 1 suite in 1.192450978s
Test Suite Failed

go test
testing: warning: no tests to run
PASS
ok      repl-1/bar      0.013s

• package has ginkgo test file with suite file OR go testing test that calls ginkgo.RunSpecs(): both ginkgo and go test pass, exit 0

$ ginkgo
Running Suite: Bar Suite
========================
Random Seed: 1493130826
Will run 1 of 1 specs

•
Ran 1 of 1 Specs in 0.000 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped PASS

Ginkgo ran 1 suite in 1.334456566s
Test Suite Passed

$ go test
Running Suite: Bar Suite
========================
Random Seed: 1493130831
Will run 1 of 1 specs

•
Ran 1 of 1 Specs in 0.000 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped PASS
ok      repl-1/bar      0.013s

Package has no ginkgo or testing tests: both fail, ginkgo exits 1, go test exits 0

$ ginkgo
Found no test suites
For usage instructions:
        ginkgo help

$ echo $?
1

$ go test
?       repl-1/bar      [no test files]

$ echo $?
0

[benmoss]

And this change didn't change the behavior of what ginkgo does when there are no test suites, so I feel like this actually makes it more consistent. The wrinkle there is that ginkgo will fail if run against 1 package with no suite, but will pass if run against multiple packages where only some have a suite. It will now fail if they both lack a suite and have a _test.go file.

$ ginkgo -r
PASS
testing: warning: no tests to run
Found no test suites, did you forget to run "ginkgo bootstrap"?
Ginkgo ran 1 suite in 298.518477ms
Test Suite Failed

$ go test ./...
ok      repl-1/bar      0.008s [no tests to run]
ok      repl-1/foo      0.013s

[andrehp]

So, the case this change is breaking my build, a project runs ginkgo with:

ginkgo -cover -r -randomizeAllSpecs -randomizeSuites -skipMeasurements ${TEST_PACKAGES}

Where TEST_PACKAGES is generated by

glide novendor | egrep -v features | egrep -v '^[.]$$' | sed 's@\/[.][.][.]@@'

In the case of the project it is

./api
./bench
./cmd
./errors
./metadata
./migrations
./models
./perf
./testing
./util

But the testing folder does not actually contain tests, it just contains some mocks and helpers, and ginkgo outputs:

testing: warning: no tests to run
coverage: 0.0% of statements
Found no test suites, did you forget to run "ginkgo bootstrap"?

The actual command run is:

ginkgo -cover -r -randomizeAllSpecs -randomizeSuites -skipMeasurements ./api ./bench ./cmd ./errors ./metadata ./migrations ./models ./perf ./testing ./util

I reverted to the last commit before your PR and everything works fine here. The testing package does not contain either suite or _test.go files.

[onsi]

wdyt @benmoss ?

@andrehp I'm a bit surprised that ginkgo is picking up the testing package. I tried this locally by creating a packaged named testing that only includes non-_test.go files and it worked.

[benmoss]

@andrehp can you tell me more about what files are in your testing package? Are any of the files in a *_test package?

[andrehp]

@benmoss The project in question is: https://github.com/topfreegames/offers
The first build that broke due to this change was a simple change in the docs: https://travis-ci.org/topfreegames/offers/builds/225319179

[benmoss]

It looks like it's due to the bench package, not the testing package. I think it's due to there being _test files in there that have no tests in them, they just have benchmarks. This used to just print "testing: warning: no tests to run", but now would cause ginkgo to think you have an empty suite. It looks like there's an ambiguity here. Alas. I guess printing the message is still an ok idea, so we can just remove the os.Exit(1) like the PR has.

[onsi]

ok thanks. merging it in.