Reconciler error happening on my pod while running with ansible-operator.version v1.19.1 and v1.20.0

[becalledbyJoyce]

Bug Report

What did you do?

We are trying with ansible-operator.version v1.19.1 and v1.20.0 on our pod and its get "reconciler error"
{"level":"error","ts":1651176144.8050282,"logger":"controller.masterdatamanagement-controller","msg":"Reconciler error","name":"mdm-cr","namespace":"mdmcpd","error":"Get "https://x.x.x.x/apis/mdm.cpd.ibm.com/v1/namespaces/mdmcpd/masterdatamanagements/mdm-cr\": dial tcp x.x.x.x: connect: connection refused","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1651176144.8061016,"logger":"reconciler","msg":"Unable to update the status to mark cr as running","job":"3368134264259628468","name":"mdm-cr","namespace":"mdmcpd","error":"Get "https://x.x.x.x/apis/mdm.cpd.ibm.com/v1/namespaces/mdmcpd/masterdatamanagements/mdm-cr\": dial tcp x.x.x.x: connect: connection refused","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1651176144.8062005,"logger":"controller.masterdatamanagement-controller","msg":"Reconciler error","name":"mdm-cr","namespace":"mdmcpd","error":"Get "https://x.x.x.x/apis/mdm.cpd.ibm.com/v1/namespaces/mdmcpd/masterdatamanagements/mdm-cr\": dial tcp x.x.x.x: connect: connection refused","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
error: http2: server sent GOAWAY and closed the connection; LastStreamID=3, ErrCode=NO_ERROR, debug=""

What did you expect to see?

We don't expect any reconciler error(expect to be working like 1.17.0)

What did you see instead? Under which circumstances?

Environment

Operator type:

Ansible

Kubernetes cluster type:

Openshift

$ operator-sdk version

<ansible-operator.version>v1.19.1</ansible-operator.version>
and
<ansible-operator.version>v1.20.0</ansible-operator.version>

$ go version (if language is Go)

$ kubectl version

Server Version: version.Info{Major:"1", Minor:"21", BuildDate:"2022-03-10T05:52:32Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}

Possible Solution

Additional context

Please find the full error
{"level":"error","ts":1651176144.8050282,"logger":"controller.masterdatamanagement-controller","msg":"Reconciler error","name":"mdm-cr","namespace":"mdmcpd","error":"Get "https://x.x.x.x/apis/mdm.cpd.ibm.com/v1/namespaces/mdmcpd/masterdatamanagements/mdm-cr\": dial tcp x.x.x.x: connect: connection refused","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1651176144.8061016,"logger":"reconciler","msg":"Unable to update the status to mark cr as running","job":"3368134264259628468","name":"mdm-cr","namespace":"mdmcpd","error":"Get "https://x.x.x.x/apis/mdm.cpd.ibm.com/v1/namespaces/mdmcpd/masterdatamanagements/mdm-cr\": dial tcp x.x.x.x: connect: connection refused","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:114\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:311\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
{"level":"error","ts":1651176144.8062005,"logger":"controller.masterdatamanagement-controller","msg":"Reconciler error","name":"mdm-cr","namespace":"mdmcpd","error":"Get "https://x.x.x.x/apis/mdm.cpd.ibm.com/v1/namespaces/mdmcpd/masterdatamanagements/mdm-cr\": dial tcp x.x.x.x: connect: connection refused","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.11.0/pkg/internal/controller/controller.go:227"}
error: http2: server sent GOAWAY and closed the connection; LastStreamID=3, ErrCode=NO_ERROR, debug=""

[VenkatRamaraju]

Hi @becalledbyJoyce,

Thanks for submitting the issue. Can you provide us with context around this.

1. What command did you run? With which flags?
2. Did you follow the tutorial to create your operator?
3. Since you're running Ansible, what are the playbook tasks doing.

Any other information will be helpful in debugging this. If I understand correctly, this issue shows up only after 1.17.0, correct? Please let me know the answer to the above questions and I will take a look at it.

[becalledbyJoyce]

Hi Venkat,
We are not facing any functionality issue of ansible-operator. We just need the go version in ansible-operator to get updated to below version(1.18.1 ,1.17.9 )

Please find in the attached screenshot

[becalledbyJoyce]

Hi @VenkatRamaraju / @asmacdo ,

Is there update on my request.

Thanks and regards,
Vasanth

[camilamacedo86]

@becalledbyJoyce,

The go version used is not the go version supported. Could you please use go 1.17 and let us know if the error persists?

[becalledbyJoyce]

HI @camilamacedo86 , We are already using the version and we are getting the above vulnerabilities.

[becalledbyJoyce]

hi @camilamacedo86 , cant you do anything regarding this request. without the go version update, we will face vulnerability in our product.

[camilamacedo86]

Hi @becalledbyJoyce,

see the error:

Reconciler error","name":"mdm-cr","namespace":"mdmcpd","error":"Get "[https://x.x.x.x/apis/mdm.cpd.ibm.com/v1/namespaces/mdmcpd/masterdatamanagements/mdm-cr\](https://x.x.x.x/apis/mdm.cpd.ibm.com/v1/namespaces/mdmcpd/masterdatamanagements/mdm-cr%5C)": dial tcp x.x.x.x: connect: connection refused","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.

That shows an issue to connect to these apis. I mean, It does not seem to be related to what is scaffold by default and by SDK tool. It is related to what is implemented by you in your specific project and/or your specific environment.

Regards the go version it is not supported with the latest SDK release and we are working to support it in the next releases. More info: kubernetes-sigs/kubebuilder#2559. However, you are using Ansible so you do not need to have to go installed either.

So, maybe if you provide additional info we might find something to try to help you out:
See the questions raised: #5767 (comment)

Do we need to know the steps to reproduce this issue?
Could you share that?
Are you able to reproduce this issue with default and new project scaffold with SDK or its samples?

[becalledbyJoyce]

Hi @camilamacedo86 ,
We are sure that the issue happened only after upgrading ansible-operator.version to v1.19.1 and v1.20.0. With older version , it worked fine and never breaks it. So any newer release of ansible-operator would be great (along with go updated which wants for resolving vulnerabilities). We can reproduce this issue by upgrading the ansible-operator to v1.19.1 and v1.20.0.

Thanks and regards,
vasanth

[asmacdo]

@becalledbyJoyce could you try upgrading to operator-sdk 1.22? This uses go 1.18. If this is still an issue please reopen.