-
-
Notifications
You must be signed in to change notification settings - Fork 650
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[feature request] Test execution and reporting controller #1422
Comments
thanks @ivelichkovich the simplest way to do this would be to orchestrate (i.e. shell out to) the ginkgo cli to have it run suites for you on demand. this gives you clean isolation, will support parallelization, and allows you to write ginkgo suites as you're used to. of course the downside is the need to distribute your binary with additional test binaries (you could include precompiled test binaries, it wouldn't have to be source code, but I get that this is ugly). If you want a way to invoke ginkgo specs in-process as code this is possible but we'd need to add some additional tooling to make it a decent experience. The only thing that would be very challenging to do would be to run the specs in parallel. This is because Ginkgo's shared memory model (the different closures in a container hierarchy communicate via shared variables) doesn't translate directly to an in-process parallelization model. But if you're ok running specs in serial then read on. Today, when call This gets tricky, however, if you want to use Ginkgo as a library and (potentially) have multiple suites - or be able to run a single suite multiple times. Ginkgo's own internal_integration suite solves this by (swapping out the global suite)[https://github.com/onsi/ginkgo/blob/master/internal/internal_integration/internal_integration_suite_test.go#L75-L82] object on-demand (look here for an example). Along these lines I could imagine a new package under
There are some important caveats here - since We could implement workarounds for both of these limitations but they would require you to pass the suite object and a particular gomega instance around. I think this could get ugly quickly (but it's the sort of pass-things-around boilerplate that Go users are used to):
As you can see the caveats start to pile up and I worry that a user new to all this will struggle to make sense of it. So i wonder if we should explore some alternatives. Perhaps a better approach would be tooling that can handle calling |
It would be nice to have a supported way to package ginkgo tests and run them with various configurations using a "canary controller" (simple pod wrapper of ginkgo executions) that reruns tests on some configurable interval and then exports results in various configurable ways initially maybe as just prometheus metrics. I'm not sure if this belongs in ginkgo but it seems like a pattern that could be useful for others so looking into open source options for building this rather than just doing it internally.
The text was updated successfully, but these errors were encountered: