Error from gingko tests: flag provided but not defined

[gee4vee]

Bug Report

We are in the process of writing E2E tests using gingko/gomega and hit this error when trying to run them locally.
WIP code is here: IBM/starter-kit-operator#6

What did you do?
Execute operator-sdk test local ./test/e2e --up-local --operator-namespace=e2e-testing --debug

What did you expect to see?
Tests are executed

What did you see instead? Under which circumstances?

operator-sdk test local ./test/e2e --up-local --operator-namespace=e2e-testing --debug
[Deprecation Notice] Operator SDK has a new CLI and project layout that is aligned with Kubebuilder.
See `operator-sdk init -h` and the following doc on how to scaffold a new project:
https://sdk.operatorframework.io/docs/golang/quickstart/
To migrate existing projects to the new layout see:
https://sdk.operatorframework.io/docs/golang/migration/project_migration_guide/

INFO[0000] Testing operator locally.
go: downloading golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
go: downloading github.com/google/go-cmp v0.5.0
go: extracting github.com/google/go-cmp v0.5.0
go: extracting golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9
flag provided but not defined: -namespacedMan
Usage of /var/folders/6b/d9669vfn5232887q421rxv7r0000gn/T/go-build511609038/b001/e2e.test:
  -ginkgo.debug
    	If set, ginkgo will emit node output to files when running in parallel.
  -ginkgo.dryRun
    	If set, ginkgo will walk the test hierarchy without actually running anything.  Best paired with -v.
  -ginkgo.failFast
    	If set, ginkgo will stop running a test suite after a failure occurs.
  -ginkgo.failOnPending
    	If set, ginkgo will mark the test suite as failed if any specs are pending.
  -ginkgo.flakeAttempts int
    	Make up to this many attempts to run each spec. Please note that if any of the attempts succeed, the suite will not be failed. But any failures will still be recorded. (default 1)
  -ginkgo.focus string
    	If set, ginkgo will only run specs that match this regular expression.
  -ginkgo.noColor
    	If set, suppress color output in default reporter.
  -ginkgo.noisyPendings
    	If set, default reporter will shout about pending tests. (default true)
  -ginkgo.noisySkippings
    	If set, default reporter will shout about skipping tests. (default true)
  -ginkgo.parallel.node int
    	This worker node's (one-indexed) node number.  For running specs in parallel. (default 1)
  -ginkgo.parallel.streamhost string
    	The address for the server that the running nodes should stream data to.
  -ginkgo.parallel.synchost string
    	The address for the server that will synchronize the running nodes.
  -ginkgo.parallel.total int
    	The total number of worker nodes.  For running specs in parallel. (default 1)
  -ginkgo.progress
    	If set, ginkgo will emit progress information as each spec runs to the GinkgoWriter.
  -ginkgo.randomizeAllSpecs
    	If set, ginkgo will randomize all specs together.  By default, ginkgo only randomizes the top level Describe, Context and When groups.
  -ginkgo.regexScansFilePath
    	If set, ginkgo regex matching also will look at the file path (code location).
  -ginkgo.reportFile string
    	Override the default reporter output file path.
  -ginkgo.reportPassed
    	If set, default reporter prints out captured output of passed tests.
  -ginkgo.seed int
    	The seed used to randomize the spec suite. (default 1597856956)
  -ginkgo.skip string
    	If set, ginkgo will only run specs that do not match this regular expression.
  -ginkgo.skipMeasurements
    	If set, ginkgo will skip any measurement specs.
  -ginkgo.slowSpecThreshold float
    	(in seconds) Specs that take longer to run than this threshold are flagged as slow by the default reporter. (default 5)
  -ginkgo.succinct
    	If set, default reporter prints out a very succinct report
  -ginkgo.trace
    	If set, default reporter prints out the full stack trace when a failure occurs
  -ginkgo.v
    	If set, default reporter print out all specs as they begin.
  -kubeconfig string
    	Paths to a kubeconfig. Only required if out-of-cluster.
  -master --kubeconfig
    	(Deprecated: switch to --kubeconfig) The address of the Kubernetes API server. Overrides any value in kubeconfig. Only required if out-of-cluster.
  -test.bench regexp
    	run only benchmarks matching regexp
  -test.benchmem
    	print memory allocations for benchmarks
  -test.benchtime d
    	run each benchmark for duration d (default 1s)
  -test.blockprofile file
    	write a goroutine blocking profile to file
  -test.blockprofilerate rate
    	set blocking profile rate (see runtime.SetBlockProfileRate) (default 1)
  -test.count n
    	run tests and benchmarks n times (default 1)
  -test.coverprofile file
    	write a coverage profile to file
  -test.cpu list
    	comma-separated list of cpu counts to run each test with
  -test.cpuprofile file
    	write a cpu profile to file
  -test.failfast
    	do not start new tests after the first test failure
  -test.list regexp
    	list tests, examples, and benchmarks matching regexp then exit
  -test.memprofile file
    	write an allocation profile to file
  -test.memprofilerate rate
    	set memory allocation profiling rate (see runtime.MemProfileRate)
  -test.mutexprofile string
    	write a mutex contention profile to the named file after execution
  -test.mutexprofilefraction int
    	if >= 0, calls runtime.SetMutexProfileFraction() (default 1)
  -test.outputdir dir
    	write profiles to dir
  -test.parallel n
    	run at most n tests in parallel (default 8)
  -test.run regexp
    	run only tests and examples matching regexp
  -test.short
    	run smaller test suite to save time
  -test.testlogfile file
    	write test action log to file (for use only by cmd/go)
  -test.timeout d
    	panic test binary after duration d (default 0, timeout disabled)
  -test.trace file
    	write an execution trace to file
  -test.v
    	verbose: print additional output
FAIL	github.com/ibm/starter-kit-operator/test/e2e	1.132s
FAIL
FATA[0017] Failed to build test binary: failed to exec []string{"go", "test", "./test/e2e/...", "-namespacedMan", "/var/folders/6b/d9669vfn5232887q421rxv7r0000gn/T/empty.yaml848509990", "-globalMan", "/var/folders/6b/d9669vfn5232887q421rxv7r0000gn/T/global-manifest.yaml677992269", "-root", "/Users/gvalenc/go/src/github.com/IBM/starter-kit-operator", "-parallel=1", "-localOperator", "-skipCleanupOnError=false"}: exit status 1
make: [test-e2e] Error 1 (ignored)

Environment

• operator-sdk version: v0.19.2

• go version: go1.14.5 darwin/amd64

• Kubernetes version information: v1.18.2

• Kubernetes cluster kind: OpenShift 4.3

• Are you writing your operator in ansible, helm, or go? go

Possible Solution

Additional context

[gee4vee]

I added this to my e2e_suite_test.go and I get a different error.

func TestMain(m *testing.M) {
	test.MainEntry(m)
}

operator-sdk test local ./test/e2e --up-local --operator-namespace=e2e-testing --debug
[Deprecation Notice] Operator SDK has a new CLI and project layout that is aligned with Kubebuilder.
See `operator-sdk init -h` and the following doc on how to scaffold a new project:
https://sdk.operatorframework.io/docs/golang/quickstart/
To migrate existing projects to the new layout see:
https://sdk.operatorframework.io/docs/golang/migration/project_migration_guide/

INFO[0000] Testing operator locally.
/var/folders/6b/d9669vfn5232887q421rxv7r0000gn/T/go-build611494574/b001/e2e.test flag redefined: kubeconfig
panic: /var/folders/6b/d9669vfn5232887q421rxv7r0000gn/T/go-build611494574/b001/e2e.test flag redefined: kubeconfig

goroutine 1 [running]:
flag.(*FlagSet).Var(0xc00009c180, 0x2982dc0, 0xc00051a610, 0x26fa734, 0xa, 0x27153fb, 0x12)
	/usr/local/Cellar/go@1.13/1.13.14/libexec/src/flag/flag.go:848 +0x4ae
flag.(*FlagSet).StringVar(...)
	/usr/local/Cellar/go@1.13/1.13.14/libexec/src/flag/flag.go:751
github.com/operator-framework/operator-sdk/pkg/test.(*frameworkOpts).addToFlagSet(0xc00051a600, 0xc00009c180)
	/Users/gvalenc/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.16.0/pkg/test/framework.go:106 +0x1bf
github.com/operator-framework/operator-sdk/pkg/test.MainEntry(0xc000694200)
	/Users/gvalenc/go/pkg/mod/github.com/operator-framework/operator-sdk@v0.16.0/pkg/test/main_entry.go:27 +0x50
github.com/ibm/starter-kit-operator/test/e2e_test.TestMain(...)
	/Users/gvalenc/go/src/github.com/IBM/starter-kit-operator/test/e2e/e2e_suite_test.go:34
main.main()
	_testmain.go:42 +0x136
FAIL	github.com/ibm/starter-kit-operator/test/e2e	1.177s
FAIL
FATA[0006] Failed to build test binary: failed to exec []string{"go", "test", "./test/e2e/...", "-namespacedMan", "/var/folders/6b/d9669vfn5232887q421rxv7r0000gn/T/empty.yaml756110626", "-globalMan", "/var/folders/6b/d9669vfn5232887q421rxv7r0000gn/T/global-manifest.yaml196965401", "-root", "/Users/gvalenc/go/src/github.com/IBM/starter-kit-operator", "-parallel=1", "-localOperator", "-skipCleanupOnError=false"}: exit status 1
make: [test-e2e] Error 1 (ignored)

This seems related to #2731 but should already be fixed in v0.19.x if I read it correctly.

[camilamacedo86]

HI @gee4vee,

The test-framework as the command operator-sdk test is deprecated and removed from the latest release (1.0). In this way, the recommendation is to: migrate the projects to the new layout (1.0) and use envtest with the adopted libs ( gingko/gomega )

However, you are also able to use envtest with the adopted libs ( gingko/gomega ) in projects scaffold with the previous versions. But it means, call and use it as described below and not with operator-sdk test local which was not designed to work with envtest. The deprecated and removed test-framework works with the fake-client instead of it.

• To run the test you would use the makefile target that is scaffolded by default with 1.0.

make tests

• Also, you could call the tests using go tools such as:

go test ./test/mytests -ginkgo.v

See that in the new layout (1.0) projects are scaffolded with the makefile target test, and the suite_test.go as they are using envtest already. For further information, check the Migrate your tests section of the Migration guide.

NOTE: In the above documentation you will see links which will bring more info and examples. Feel free to check the SDK tests to ensure the Helm projects scaffold with the tool and the controller-runtime tests as just examples of the tests with the stack adopted too.

I hope that it helps you with.

[camilamacedo86]

Flag as doc since we need to provide better guidance on how to do the tests. @dmesser also suggested we have a section to cover testing in it’s own chapter, most likely per operator-type (golang, ansible, helm) etc.

[gee4vee]

Thanks for the guidance! We will likely migrate our project to get it working.

[openshift-bot]

Issues go stale after 90d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[openshift-bot]

Stale issues rot after 30d of inactivity.

Mark the issue as fresh by commenting /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
Exclude this issue from closing by commenting /lifecycle frozen.

If this issue is safe to close now please do so with /close.

/lifecycle rotten
/remove-lifecycle stale

[openshift-bot]

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

[openshift-ci-robot]

@openshift-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.

Reopen the issue by commenting /reopen.
Mark the issue as fresh by commenting /remove-lifecycle rotten.
Exclude this issue from closing again by commenting /lifecycle frozen.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.