Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failed to start envtest on Mac with macOS 12 Monterey #1858

Closed
tenzen-y opened this issue May 3, 2022 · 2 comments · Fixed by #1861
Closed

Failed to start envtest on Mac with macOS 12 Monterey #1858

tenzen-y opened this issue May 3, 2022 · 2 comments · Fixed by #1861
Assignees
Labels

Comments

@tenzen-y
Copy link
Member

tenzen-y commented May 3, 2022

/kind bug

What steps did you take and what happened:
[A clear and concise description of what the bug is.]
When I run make test, etcd does not start and the test crashes.

{"level":"error","ts":1651553200.312333,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":4,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngit.luolix.top/kubeflow/katib/pkg/controller.v1beta1/experiment.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/experiment_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
2022/05/03 13:46:40 failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start
stacktrace
$ make test
go test ./pkg/... ./cmd/... -coverprofile coverage.out
# github.com/shirou/gopsutil/cpu
../../../../1.17.8/pkg/mod/github.com/shirou/gopsutil@v2.20.7+incompatible/cpu/cpu_darwin_cgo.go:13:5: warning: 'TARGET_OS_MAC' is not defined, evaluates to 0 [-Wundef-prefix=TARGET_OS_]
?       github.com/kubeflow/katib/pkg/apis/controller   [no test files]
?       github.com/kubeflow/katib/pkg/apis/controller/common    [no test files]
?       github.com/kubeflow/katib/pkg/apis/controller/common/v1beta1    [no test files]
?       github.com/kubeflow/katib/pkg/apis/controller/experiments       [no test files]
?       github.com/kubeflow/katib/pkg/apis/controller/experiments/v1beta1       [no test files]
?       github.com/kubeflow/katib/pkg/apis/controller/suggestions       [no test files]
?       github.com/kubeflow/katib/pkg/apis/controller/suggestions/v1beta1       [no test files]
?       github.com/kubeflow/katib/pkg/apis/controller/trials    [no test files]
?       github.com/kubeflow/katib/pkg/apis/controller/trials/v1beta1    [no test files]
?       github.com/kubeflow/katib/pkg/apis/manager/health       [no test files]
?       github.com/kubeflow/katib/pkg/apis/manager/v1beta1      [no test files]
?       github.com/kubeflow/katib/pkg/apis/v1beta1      [no test files]
?       github.com/kubeflow/katib/pkg/cert-generator/v1beta1    [no test files]
?       github.com/kubeflow/katib/pkg/cert-generator/v1beta1/consts     [no test files]
ok      github.com/kubeflow/katib/pkg/cert-generator/v1beta1/generate   2.975s  coverage: 81.6% of statements
?       github.com/kubeflow/katib/pkg/client/controller/clientset/versioned     [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/fake        [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/scheme      [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/typed/common/v1beta1        [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/typed/common/v1beta1/fake   [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/typed/experiments/v1beta1   [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/typed/experiments/v1beta1/fake      [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/typed/suggestions/v1beta1   [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/typed/suggestions/v1beta1/fake      [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/typed/trials/v1beta1        [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/clientset/versioned/typed/trials/v1beta1/fake   [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/informers/externalversions      [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/informers/externalversions/experiments  [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/informers/externalversions/experiments/v1beta1  [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/informers/externalversions/internalinterfaces   [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/informers/externalversions/suggestions  [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/informers/externalversions/suggestions/v1beta1  [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/informers/externalversions/trials       [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/informers/externalversions/trials/v1beta1       [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/listers/experiments/v1beta1     [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/listers/suggestions/v1beta1     [no test files]
?       github.com/kubeflow/katib/pkg/client/controller/listers/trials/v1beta1  [no test files]
?       github.com/kubeflow/katib/pkg/common/v1beta1    [no test files]
?       github.com/kubeflow/katib/pkg/controller.v1beta1        [no test files]
?       github.com/kubeflow/katib/pkg/controller.v1beta1/consts [no test files]
{"level":"error","ts":1651552960.309113,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":0,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngit.luolix.top/kubeflow/katib/pkg/controller.v1beta1/experiment.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/experiment_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
{"level":"error","ts":1651553020.309948,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":1,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngit.luolix.top/kubeflow/katib/pkg/controller.v1beta1/experiment.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/experiment_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
{"level":"error","ts":1651553080.3106759,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":2,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngit.luolix.top/kubeflow/katib/pkg/controller.v1beta1/experiment.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/experiment_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
{"level":"error","ts":1651553140.311522,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":3,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngit.luolix.top/kubeflow/katib/pkg/controller.v1beta1/experiment.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/experiment_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
{"level":"error","ts":1651553200.312333,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":4,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngit.luolix.top/kubeflow/katib/pkg/controller.v1beta1/experiment.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/experiment_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
2022/05/03 13:46:40 failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start
FAIL    github.com/kubeflow/katib/pkg/controller.v1beta1/experiment     301.173s
ok      github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/manifest    1.034s  coverage: 75.9% of statements
?       github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/suggestion  [no test files]
?       github.com/kubeflow/katib/pkg/controller.v1beta1/experiment/util        [no test files]
{"level":"error","ts":1651552921.0477371,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":0,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngit.luolix.top/kubeflow/katib/pkg/controller.v1beta1/suggestion.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/suggestion/suggestion_controller_suite_test.go:47\nmain.main\n\t_testmain.go:99\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
{"level":"error","ts":1651552941.048397,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":1,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngit.luolix.top/kubeflow/katib/pkg/controller.v1beta1/suggestion.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/suggestion/suggestion_controller_suite_test.go:47\nmain.main\n\t_testmain.go:99\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
{"level":"error","ts":1651552961.048811,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":2,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngit.luolix.top/kubeflow/katib/pkg/controller.v1beta1/suggestion.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/suggestion/suggestion_controller_suite_test.go:47\nmain.main\n\t_testmain.go:99\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
{"level":"error","ts":1651552981.049254,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":3,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngit.luolix.top/kubeflow/katib/pkg/controller.v1beta1/suggestion.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/suggestion/suggestion_controller_suite_test.go:47\nmain.main\n\t_testmain.go:99\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
{"level":"error","ts":1651553001.049622,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":4,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngit.luolix.top/kubeflow/katib/pkg/controller.v1beta1/suggestion.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/suggestion/suggestion_controller_suite_test.go:47\nmain.main\n\t_testmain.go:99\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
2022/05/03 13:43:21 failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start
FAIL    github.com/kubeflow/katib/pkg/controller.v1beta1/suggestion     101.779s
2022/05/03 13:43:20 failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start
FAIL    github.com/kubeflow/katib/pkg/controller.v1beta1/suggestion/composer    101.519s
ok      github.com/kubeflow/katib/pkg/controller.v1beta1/suggestion/suggestionclient    0.552s  coverage: 98.1% of statements
{"level":"error","ts":1651552961.4102862,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":0,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngit.luolix.top/kubeflow/katib/pkg/controller.v1beta1/trial.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/trial/trial_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
{"level":"error","ts":1651553021.411232,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":1,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngit.luolix.top/kubeflow/katib/pkg/controller.v1beta1/trial.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/trial/trial_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
{"level":"error","ts":1651553081.411948,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":2,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngit.luolix.top/kubeflow/katib/pkg/controller.v1beta1/trial.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/trial/trial_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
{"level":"error","ts":1651553141.412628,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":3,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngit.luolix.top/kubeflow/katib/pkg/controller.v1beta1/trial.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/trial/trial_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
{"level":"error","ts":1651553201.413376,"logger":"controller-runtime.test-env","msg":"unable to start the controlplane","tries":4,"error":"timeout waiting for process etcd to start","stacktrace":"sigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).startControlPlane\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:291\nsigs.k8s.io/controller-runtime/pkg/envtest.(*Environment).Start\n\t/Users/tenzen-y/go/1.17.8/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.2/pkg/envtest/server.go:253\ngit.luolix.top/kubeflow/katib/pkg/controller.v1beta1/trial.TestMain\n\t/Users/tenzen-y/go/src/github.com/kubeflow/katib/pkg/controller.v1beta1/trial/trial_controller_suite_test.go:54\nmain.main\n\t_testmain.go:101\nruntime.main\n\t/Users/tenzen-y/.goenv/versions/1.17.8/src/runtime/proc.go:255"}
2022/05/03 13:46:41 failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start
FAIL    github.com/kubeflow/katib/pkg/controller.v1beta1/trial  302.123s
?       github.com/kubeflow/katib/pkg/controller.v1beta1/trial/managerclient    [no test files]
ok      github.com/kubeflow/katib/pkg/controller.v1beta1/trial/util     1.440s  coverage: 32.9% of statements
?       github.com/kubeflow/katib/pkg/controller.v1beta1/util   [no test files]
?       github.com/kubeflow/katib/pkg/db/v1beta1        [no test files]
?       github.com/kubeflow/katib/pkg/db/v1beta1/common [no test files]
ok      github.com/kubeflow/katib/pkg/db/v1beta1/mysql  0.504s  coverage: 70.4% of statements
?       github.com/kubeflow/katib/pkg/metricscollector/v1beta1/common   [no test files]
ok      github.com/kubeflow/katib/pkg/metricscollector/v1beta1/file-metricscollector    2.214s  coverage: 54.4% of statements
?       github.com/kubeflow/katib/pkg/mock/v1beta1/api  [no test files]
?       github.com/kubeflow/katib/pkg/mock/v1beta1/db   [no test files]
?       github.com/kubeflow/katib/pkg/mock/v1beta1/experiment/manifest  [no test files]
?       github.com/kubeflow/katib/pkg/mock/v1beta1/experiment/suggestion        [no test files]
?       github.com/kubeflow/katib/pkg/mock/v1beta1/suggestion/suggestionclient  [no test files]
?       github.com/kubeflow/katib/pkg/mock/v1beta1/trial/managerclient  [no test files]
?       github.com/kubeflow/katib/pkg/mock/v1beta1/util/katibclient     [no test files]
?       github.com/kubeflow/katib/pkg/new-ui/v1beta1    [no test files]
ok      github.com/kubeflow/katib/pkg/suggestion/v1beta1/goptuna        1.019s  coverage: 37.8% of statements
?       github.com/kubeflow/katib/pkg/ui/v1beta1        [no test files]
ok      github.com/kubeflow/katib/pkg/util/v1beta1/env  0.217s  coverage: 100.0% of statements
?       github.com/kubeflow/katib/pkg/util/v1beta1/katibclient  [no test files]
ok      github.com/kubeflow/katib/pkg/util/v1beta1/katibconfig  0.785s  coverage: 97.4% of statements
?       github.com/kubeflow/katib/pkg/webhook/v1beta1   [no test files]
?       github.com/kubeflow/katib/pkg/webhook/v1beta1/common    [no test files]
?       github.com/kubeflow/katib/pkg/webhook/v1beta1/experiment        [no test files]
ok      github.com/kubeflow/katib/pkg/webhook/v1beta1/experiment/validator      1.474s  coverage: 98.6% of statements
--- FAIL: TestGetMetricsCollectorArgs (100.00s)
    inject_webhook_test.go:249: failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start
    inject_webhook_test.go:255: 
        Unexpected error:
            <*errors.errorString | 0xc0004d6140>: {
                s: "must specify Config",
            }
            must specify Config
        occurred
--- FAIL: TestGetKatibJob (100.00s)
    inject_webhook_test.go:696: failed to start the controlplane. retried 5 times: timeout waiting for process etcd to start
    inject_webhook_test.go:702: 
        Unexpected error:
            <*errors.errorString | 0xc00024af60>: {
                s: "must specify Config",
            }
            must specify Config
        occurred
FAIL
coverage: 32.0% of statements
FAIL    github.com/kubeflow/katib/pkg/webhook/v1beta1/pod       200.934s
?       github.com/kubeflow/katib/cmd/cert-generator/v1beta1    [no test files]
ok      github.com/kubeflow/katib/cmd/db-manager/v1beta1        1.027s  coverage: 36.1% of statements
?       github.com/kubeflow/katib/cmd/katib-controller/v1beta1  [no test files]
?       github.com/kubeflow/katib/cmd/metricscollector/v1beta1/file-metricscollector    [no test files]
?       github.com/kubeflow/katib/cmd/new-ui/v1beta1    [no test files]
?       github.com/kubeflow/katib/cmd/suggestion/goptuna/v1beta1        [no test files]
?       github.com/kubeflow/katib/cmd/ui/v1beta1        [no test files]
FAIL
make: *** [test] Error 1

According to the Go upstream wiki, macOS Monterey has discontinued support for programs built with versions earlier than Go 1.10.x.

If you reached this page because you saw an error message like the following printed by a Go program running on macOS 12 Monterey:

fatal error: runtime: bsdthread_register error

runtime stack:
runtime.throw(0x20594e0, 0x21)
	/usr/local/go/src/runtime/panic.go:619 +0x81 fp=0x7ff7bfeff878 sp=0x7ff7bfeff858 pc=0x1029751
runtime.goenvs()
	/usr/local/go/src/runtime/os_darwin.go:129 +0x83 fp=0x7ff7bfeff8a8 sp=0x7ff7bfeff878 pc=0x10272d3
runtime.schedinit()
	/usr/local/go/src/runtime/proc.go:496 +0xa4 fp=0x7ff7bfeff900 sp=0x7ff7bfeff8a8 pc=0x102c014
runtime.rt0_go(0x7ff7bfeff930, 0x3, 0x7ff7bfeff930, 0x1000000, 0x3, 0x7ff7bfeffab0, 0x7ff7bfeffabf, 0x7ff7bfeffac3, 0x0, > 0x7ff7bfeffacc, ...)
	/usr/local/go/src/runtime/asm_amd64.s:252 +0x1f4 fp=0x7ff7bfeff908 sp=0x7ff7bfeff900 pc=0x1052c64

then you are running a program built with an old version of Go (Go 1.10 or before). You will need to update your program or rebuild it with a newer version of Go.

I checked the etcd version on Ubuntu machine and found that the kubebuilder v2.3.0 used by Katib comes bundled with etcd built with go 1.10.7, so etcd could not be started.

  • On Mac with macOS 12 Monterey
$ /usr/local/kubebuilder/bin/etcd --version
fatal error: runtime: bsdthread_register error

runtime stack:
runtime.throw(0x1bed3fa, 0x21)
        /usr/local/go/src/runtime/panic.go:616 +0x81 fp=0x7ff7bfeff2d8 sp=0x7ff7bfeff2b8 pc=0x102a871
runtime.goenvs()
        /usr/local/go/src/runtime/os_darwin.go:129 +0x83 fp=0x7ff7bfeff308 sp=0x7ff7bfeff2d8 pc=0x10283f3
runtime.schedinit()
        /usr/local/go/src/runtime/proc.go:501 +0xd6 fp=0x7ff7bfeff370 sp=0x7ff7bfeff308 pc=0x102d166
runtime.rt0_go(0x7ff7bfeff3a0, 0x2, 0x7ff7bfeff3a0, 0x1000000, 0x2, 0x7ff7bfeff5a8, 0x7ff7bfeff5c8, 0x0, 0x7ff7bfeff5d2, 0x7ff7bfeff5e2, ...)
        /usr/local/go/src/runtime/asm_amd64.s:252 +0x1f4 fp=0x7ff7bfeff378 sp=0x7ff7bfeff370 pc=0x1056474
  • On Ubuntu machine
$ ./etcd --version
etcd Version: 3.3.11
Git SHA: 2cf9e51d2
Go Version: go1.10.7
Go OS/Arch: linux/amd64

So I downloaded kubebuilder-tools for K8s v1.20.2 from GCS bucket maintained by controller-runtime project, extracted only etcd, and installed it in /usr/local/kubebuilder/bin/etcd, I was able to run make test successfully.

What did you expect to happen:
Envtest succeeds.

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

  • Katib version (check the Katib controller image version): latest version(commit hash: 3993404)
  • Kubernetes version: (kubectl version): none
  • OS (uname -a): macOS 12 Monterey

Impacted by this bug? Give it a 👍 We prioritize the issues with the most 👍

@tenzen-y
Copy link
Member Author

tenzen-y commented May 3, 2022

We may have to consider upgrading the kubebuilder version from v2.3.0 to v3.x.y.

/cc @kubeflow/wg-automl-leads

@tenzen-y tenzen-y changed the title Failed to start envtest on Mac with macOS 12 Moterey Failed to start envtest on Mac with macOS 12 Monterey May 3, 2022
@tenzen-y
Copy link
Member Author

tenzen-y commented May 3, 2022

Since kubebuilder supports K8s v1.22 in v3.2.0, I will upgrade kubebuilder version to v3.2.0.

/assign

⚠️ Breaking Changes
(go/v3) update deps to use k8s 1.22 (kubernetes-sigs/kubebuilder#2340)
This is only a breaking change if using v1beta1 CRDs or webhook configs

https://github.com/kubernetes-sigs/kubebuilder/releases/tag/v3.2.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant