Makefile should not need to be modified to get tests to run #3692

jmrodri · 2020-08-07T20:14:40Z

Bug Report

A user should not have to update their Makefile after scaffolding an operator to get make test to work.

What did you do?

mkdir zeus-operator
cd zeus-operator
operator-sdk init --repo=github.com/jmrodri/zeus-operator
operator-sdk create api --group test --version v1 --kind Zeus --controller --resource
make test

What did you expect to see?
I expected make test to run the tests and if it needs other binaries to install them.

What did you see instead? Under which circumstances?
After running make test it spewed tons of errors:

go fmt ./...                                                                                                                                                                                  
go vet ./...                                                                                                                                                                                  
/home/devel/jesusr/dev/bin/controller-gen "crd:trivialVersions=true" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
go test ./... -coverprofile cover.out                                                          
?       github.com/jmrodri/zeus-operator        [no test files]                                                                                                                               
?       github.com/jmrodri/zeus-operator/api/v1 [no test files]            
Running Suite: Controller Suite                                                                                                                                                               
===============================                                                                
Random Seed: 1596830400                                                                                                                                                                       
Will run 0 of 0 specs                                                                          
                                                                                               
STEP: bootstrapping test environment
2020-08-07T16:00:00.616-0400    DEBUG   controller-runtime.test-env     starting control plane  {"api server flags": []}
2020-08-07T16:00:00.619-0400    ERROR   controller-runtime.test-env     unable to start the controlplane        {"tries": 0, "error": "fork/exec /usr/local/kubebuilder/bin/etcd: no such file
 or directory"}                                                                                
github.com/go-logr/zapr.(*zapLogger).Error                                                     
        /home/devel/jesusr/dev/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128                                                                                                             
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane
        /home/devel/jesusr/dev/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/envtest/server.go:279
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start                                                                                                                               
        /home/devel/jesusr/dev/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/envtest/server.go:241                                                                                        
github.com/jmrodri/zeus-operator/controllers.glob..func1
        /home/devel/jesusr/dev/zeus-operator/controllers/suite_test.go:61        
reflect.Value.call                                                                             
        /usr/lib/golang/src/reflect/value.go:460                                                                                                                                              
reflect.Value.Call                                                                             
        /usr/lib/golang/src/reflect/value.go:321                                                                                                                                              
github.com/onsi/ginkgo/internal/leafnodes.newRunner.func1
        /home/devel/jesusr/dev/pkg/mod/github.com/onsi/ginkgo@v1.12.1/internal/leafnodes/runner.go:49
github.com/onsi/ginkgo/internal/leafnodes.(*runner).runAsync.func1
        /home/devel/jesusr/dev/pkg/mod/github.com/onsi/ginkgo@v1.12.1/internal/leafnodes/runner.go:86
2020-08-07T16:00:00.620-0400    ERROR   controller-runtime.test-env     unable to start the controlplane        {"tries": 1, "error": "fork/exec /usr/local/kubebuilder/bin/etcd: no such file
 or directory"}                                                                                
github.com/go-logr/zapr.(*zapLogger).Error                                                     
        /home/devel/jesusr/dev/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128                                            
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane
        /home/devel/jesusr/dev/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/envtest/server.go:279
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start
        /home/devel/jesusr/dev/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/envtest/server.go:241
github.com/jmrodri/zeus-operator/controllers.glob..func1
        /home/devel/jesusr/dev/zeus-operator/controllers/suite_test.go:61
reflect.Value.call
        /usr/lib/golang/src/reflect/value.go:460
reflect.Value.Call
        /usr/lib/golang/src/reflect/value.go:321
github.com/onsi/ginkgo/internal/leafnodes.newRunner.func1
        /home/devel/jesusr/dev/pkg/mod/github.com/onsi/ginkgo@v1.12.1/internal/leafnodes/runner.go:49
github.com/onsi/ginkgo/internal/leafnodes.(*runner).runAsync.func1
        /home/devel/jesusr/dev/pkg/mod/github.com/onsi/ginkgo@v1.12.1/internal/leafnodes/runner.go:86
2020-08-07T16:00:00.620-0400    ERROR   controller-runtime.test-env     unable to start the controlplane        {"tries": 2, "error": "fork/exec /usr/local/kubebuilder/bin/etcd: no such file
 or directory"}
github.com/go-logr/zapr.(*zapLogger).Error
        /home/devel/jesusr/dev/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane
        /home/devel/jesusr/dev/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/envtest/server.go:279
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start
        /home/devel/jesusr/dev/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/envtest/server.go:241
github.com/jmrodri/zeus-operator/controllers.glob..func1
        /home/devel/jesusr/dev/zeus-operator/controllers/suite_test.go:61
reflect.Value.call
        /usr/lib/golang/src/reflect/value.go:460
reflect.Value.Call
        /usr/lib/golang/src/reflect/value.go:321
github.com/onsi/ginkgo/internal/leafnodes.newRunner.func1
        /home/devel/jesusr/dev/pkg/mod/github.com/onsi/ginkgo@v1.12.1/internal/leafnodes/runner.go:49
github.com/onsi/ginkgo/internal/leafnodes.(*runner).runAsync.func1
        /home/devel/jesusr/dev/pkg/mod/github.com/onsi/ginkgo@v1.12.1/internal/leafnodes/runner.go:86
2020-08-07T16:00:00.621-0400    ERROR   controller-runtime.test-env     unable to start the controlplane        {"tries": 3, "error": "fork/exec /usr/local/kubebuilder/bin/etcd: no such file
 or directory"}
github.com/go-logr/zapr.(*zapLogger).Error
        /home/devel/jesusr/dev/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane
        /home/devel/jesusr/dev/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/envtest/server.go:279
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start
        /home/devel/jesusr/dev/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/envtest/server.go:241
github.com/jmrodri/zeus-operator/controllers.glob..func1
        /home/devel/jesusr/dev/zeus-operator/controllers/suite_test.go:61
reflect.Value.call
        /usr/lib/golang/src/reflect/value.go:460
reflect.Value.Call
        /usr/lib/golang/src/reflect/value.go:321
github.com/onsi/ginkgo/internal/leafnodes.newRunner.func1
        /home/devel/jesusr/dev/pkg/mod/github.com/onsi/ginkgo@v1.12.1/internal/leafnodes/runner.go:49
github.com/onsi/ginkgo/internal/leafnodes.(*runner).runAsync.func1
        /home/devel/jesusr/dev/pkg/mod/github.com/onsi/ginkgo@v1.12.1/internal/leafnodes/runner.go:86
2020-08-07T16:00:00.621-0400    ERROR   controller-runtime.test-env     unable to start the controlplane        {"tries": 4, "error": "fork/exec /usr/local/kubebuilder/bin/etcd: no such file
 or directory"}
github.com/go-logr/zapr.(*zapLogger).Error
        /home/devel/jesusr/dev/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane
        /home/devel/jesusr/dev/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/envtest/server.go:279
sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start
        /home/devel/jesusr/dev/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.2/pkg/envtest/server.go:241
github.com/jmrodri/zeus-operator/controllers.glob..func1
        /home/devel/jesusr/dev/zeus-operator/controllers/suite_test.go:61
reflect.Value.call
        /usr/lib/golang/src/reflect/value.go:460
reflect.Value.Call
        /usr/lib/golang/src/reflect/value.go:321
github.com/onsi/ginkgo/internal/leafnodes.newRunner.func1
        /home/devel/jesusr/dev/pkg/mod/github.com/onsi/ginkgo@v1.12.1/internal/leafnodes/runner.go:49
github.com/onsi/ginkgo/internal/leafnodes.(*runner).runAsync.func1
        /home/devel/jesusr/dev/pkg/mod/github.com/onsi/ginkgo@v1.12.1/internal/leafnodes/runner.go:86
Failure [0.005 seconds]
[BeforeSuite] BeforeSuite 
/home/devel/jesusr/dev/zeus-operator/controllers/suite_test.go:52

  Unexpected error:
      <*fmt.wrapError | 0xc00000f4a0>: {
          msg: "failed to start the controlplane. retried 5 times: fork/exec /usr/local/kubebuilder/bin/etcd: no such file or directory",
          err: {
              Op: "fork/exec",
              Path: "/usr/local/kubebuilder/bin/etcd",
              Err: 0x2,
          },
      }
      failed to start the controlplane. retried 5 times: fork/exec /usr/local/kubebuilder/bin/etcd: no such file or directory
  occurred

  /home/devel/jesusr/dev/zeus-operator/controllers/suite_test.go:62
------------------------------


Ran 0 of 0 Specs in 0.006 seconds
FAIL! -- 0 Passed | 0 Failed | 0 Pending | 0 Skipped
--- FAIL: TestAPIs (0.01s)
FAIL
coverage: 0.0% of statements
FAIL    github.com/jmrodri/zeus-operator/controllers    0.027s
FAIL
make: *** [Makefile:32: test] Error 1

Environment

operator-sdk version:

$ operator-sdk version
operator-sdk version: "v1.0.0-alpha.2-16-g8bca0209", commit: "8bca020931136d196ae5f4135c1d70738a501135", kubernetes version: "v1.18.2", go version: "go1.13.11 linux/amd64", GOOS: "linux", GOARCH: "amd64"

go version:

go version go1.13.11 linux/amd64

Kubernetes version information:

Client Version: version.Info{Major:"1", Minor:"11", GitVersion:"v1.11.0", GitCommit:"91e7b4fd31fcd3d5f436da26c980becec37ceefe", GitTreeState:"clean", BuildDate:"2018-06-27T20:17:28Z", GoVersion:"go1.10.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-30T20:19:45Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Kubernetes cluster kind:

$ kind version
kind v0.8.1 go1.13.11 linux/amd64

Are you writing your operator in ansible, helm, or go?
Go

Possible Solution
If the test target requires the user to modify it to work, why can't the scaffolder generate a Makefile with the proper test target as outlined below.

Additional context
This issue apparently was brought up in Issue #3461 because kubebuilder executables are required for the tests. They are still not in the Prerequisites: Golang Based Operator SDK Installation. The solution suggested in the documentation, is for the user to modify the newly scaffolded Makefile with the following:

# Run tests
ENVTEST_ASSETS_DIR=$(shell pwd)/testbin
test: generate fmt vet manifests
        mkdir -p ${ENVTEST_ASSETS_DIR}
        test -f ${ENVTEST_ASSETS_DIR}/setup-envtest.sh || curl -sSLo ${ENVTEST_ASSETS_DIR}/setup-envtest.sh https://raw.githubusercontent.com/kubernetes-sigs/controller-runtime/master/hack/setup-envtest.sh
        source ${ENVTEST_ASSETS_DIR}/setup-envtest.sh; fetch_envtest_tools $(ENVTEST_ASSETS_DIR); setup_envtest_env $(ENVTEST_ASSETS_DIR); go test ./... -coverprofile cover.out

The text was updated successfully, but these errors were encountered:

jmccormick2001 · 2020-08-07T20:25:19Z

haha, I just hit this same error! The docs are misleading to me, the part about 'setting up your test environment' is something I skipped because I really just wanted to build my operator not test it, however, that test step actually provides the missing dependencies required to do a build.

camilamacedo86 · 2020-08-07T20:27:04Z

It will be solved when we are able to merge; kubernetes-sigs/kubebuilder#1626 and support the go/v3-alpha plugin from kubebuilder. Needs to be post 1.0.

tomqwpl · 2020-09-07T14:29:08Z

Yes, I don't understand the situation here.
I have just created a new project with operator-sdk v1.0.0, then all I did was copy the bit of Makefile from the documentation into my makefile and the result worked. I did nothing else. I didn't understand the linked documentation section "Setup the envtest binaries and environment". The Makefile section it tells you to add downloads the envtest stuff if it's not present.

jmrodri · 2020-09-30T18:32:06Z

There are 3 possible solutions:

Update the quickstart documentation to ensure folks always add it by hand: https://sdk.operatorframework.io/docs/building-operators/golang/quickstart/
Update our scaffolding code to add it automatically to the generated Makefile
Update the Makefile so that the docker-build target does not depend on the test target.

jmrodri · 2020-09-30T18:32:53Z

Consensus on the grooming call was to do option 2.

camilamacedo86 · 2020-10-01T14:03:36Z

It is great news @jmrodri. Also, I think the best option is 2.

jberkhahn · 2020-10-01T17:02:57Z

@camilamacedo86 Doesn't your comment indicate we get this for free because it's been fixed upstream? We just need to bump the version?

jberkhahn · 2020-10-06T22:38:48Z

waiting for 1711 in kubebuilder to be merged before we bump our dependency on it

jberkhahn · 2020-10-12T18:40:24Z

We're going to merge #3983 and wait on the upstream changes to fix it for good.

… run the commands (#3983) **Description** Customize the makefile target tests to download the binaries. Closes: #3692

jberkhahn · 2020-10-14T23:23:14Z

I'd like to keep this issue open to track the issue upstream. Will move it into the next release and take off the blocker tag.

camilamacedo86 · 2020-10-14T23:24:05Z

It was

camilamacedo86 · 2020-10-15T00:58:58Z

In SDK, this problem is solved with the PR #3983 and it was cherry-picked to v1.0.x with #4038.

Also, the same solution is merged already for the v3+ plugins in upstream and a better approach has been discussed and might be pushed to v2+ as well. More info: kubernetes-sigs/kubebuilder#1711.

In this way, we can consider it solved.

jmrodri changed the title ~~Users should not have to update their Makefile to get make test working after scaffolding a new project~~ Makefile should not need to be modified to get tests to run Aug 7, 2020

camilamacedo86 mentioned this issue Aug 10, 2020

doc: make implicity the users must set envtest #3709

Closed

camilamacedo86 added the v3-plugin label Aug 10, 2020

camilamacedo86 self-assigned this Aug 10, 2020

jberkhahn added this to the v1.1.0 milestone Aug 10, 2020

jberkhahn added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Aug 10, 2020

estroz added language/go Issue is related to a Go operator project and removed go/v3-alpha labels Aug 25, 2020

jberkhahn added the release-blocker This issue blocks the parent release milestone label Sep 30, 2020

jberkhahn self-assigned this Sep 30, 2020

camilamacedo86 mentioned this issue Oct 6, 2020

change makefile test target for no longer be required manual steps to run the commands #3983

Merged

2 tasks

camilamacedo86 closed this as completed in #3983 Oct 14, 2020

camilamacedo86 added a commit that referenced this issue Oct 14, 2020

change makefile test target for no longer be required manual steps to…

c36f1b6

… run the commands (#3983) **Description** Customize the makefile target tests to download the binaries. Closes: #3692

jberkhahn reopened this Oct 14, 2020

jberkhahn removed the release-blocker This issue blocks the parent release milestone label Oct 14, 2020

jberkhahn modified the milestones: v1.1.0, v1.2.0 Oct 14, 2020

camilamacedo86 closed this as completed Oct 14, 2020

camilamacedo86 reopened this Oct 14, 2020

camilamacedo86 closed this as completed Oct 15, 2020

jberkhahn mentioned this issue Oct 15, 2020

Remove our changes to kubebuilder generated makefiles #4044

Closed

adams0619 mentioned this issue Nov 11, 2020

build is failing redhat-marketplace/marketplace-games-operator#6

Closed

vashirov mentioned this issue Nov 18, 2020

Tests are failing in OpenShift CI freeipa/freeipa-operator#19

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Makefile should not need to be modified to get tests to run #3692

Makefile should not need to be modified to get tests to run #3692

jmrodri commented Aug 7, 2020

jmccormick2001 commented Aug 7, 2020

camilamacedo86 commented Aug 7, 2020

tomqwpl commented Sep 7, 2020

jmrodri commented Sep 30, 2020

jmrodri commented Sep 30, 2020

camilamacedo86 commented Oct 1, 2020

jberkhahn commented Oct 1, 2020

jberkhahn commented Oct 6, 2020

jberkhahn commented Oct 12, 2020

jberkhahn commented Oct 14, 2020

camilamacedo86 commented Oct 14, 2020

camilamacedo86 commented Oct 15, 2020

Makefile should not need to be modified to get tests to run #3692

Makefile should not need to be modified to get tests to run #3692

Comments

jmrodri commented Aug 7, 2020

Bug Report

jmccormick2001 commented Aug 7, 2020

camilamacedo86 commented Aug 7, 2020

tomqwpl commented Sep 7, 2020

jmrodri commented Sep 30, 2020

jmrodri commented Sep 30, 2020

camilamacedo86 commented Oct 1, 2020

jberkhahn commented Oct 1, 2020

jberkhahn commented Oct 6, 2020

jberkhahn commented Oct 12, 2020

jberkhahn commented Oct 14, 2020

camilamacedo86 commented Oct 14, 2020

camilamacedo86 commented Oct 15, 2020