Update rook to latest master build. #1141

obnoxxx · 2021-03-30T09:45:46Z

For 4.8 preparation, we need a newer rook image.

obnoxxx · 2021-03-30T10:06:12Z

rook update pulls in an update of the controller-runtime dependency from 0.6.3 to 0.8.0.
Seems we need to adapt our code to that.

obnoxxx · 2021-03-30T11:09:53Z

Weeding out the build errors one by one...
I am left with two:

$ make
Running deps-update
go mod tidy && go mod vendor
go: creating new go.mod: module tmp
go get: added sigs.k8s.io/controller-tools v0.4.1
Updating generated code
/Users/madam/go/bin/controller-gen object:headerFile="hack/boilerplate.go.txt" paths="./..."
Building the ocs-operator binary
hack/go-build.sh
# github.com/openshift/ocs-operator/controllers/storagecluster
controllers/storagecluster/external_resources.go:182:4: cannot use "60s" (type string) as type *"k8s.io/apimachinery/pkg/apis/meta/v1".Duration in field value
controllers/storagecluster/kms_resources.go:79:38: cannot use runtimeObj (type "k8s.io/apimachinery/pkg/runtime".Object) as type client.Object in argument to r.Client.Delete:
	"k8s.io/apimachinery/pkg/runtime".Object does not implement client.Object (missing GetAnnotations method)
make: *** [build-go] Error 2

obnoxxx · 2021-03-30T11:26:21Z

Remaining error:

# github.com/openshift/ocs-operator/controllers/storagecluster
controllers/storagecluster/kms_resources.go:79:38: cannot use runtimeObj (type "k8s.io/apimachinery/pkg/runtime".Object) as type client.Object in argument to r.Client.Delete:
	"k8s.io/apimachinery/pkg/runtime".Object does not implement client.Object (missing GetAnnotations method)

obnoxxx · 2021-03-30T11:40:42Z

Locally, I am getting this error now:

"error loading manifests from directory: error checking provided apis in bundle ocs-operator.v4.8.0: couldn't find ceph.rook.io/v1/CephNFS (cephnfses) in bundle. found: map[noobaa.io/v1alpha1/BackingStore (backingstores):{} noobaa.io/v1alpha1/BucketClass (bucketclasses):{} noobaa.io/v1alpha1/NamespaceStore (namespacestores):{} noobaa.io/v1alpha1/NooBaa (noobaas):{} objectbucket.io/v1alpha1/ObjectBucket (objectbuckets):{} objectbucket.io/v1alpha1/ObjectBucketClaim (objectbucketclaims):{} ocs.openshift.io/v1/OCSInitialization (ocsinitializations):{} ocs.openshift.io/v1/StorageCluster (storageclusters):{}]"

leseb · 2021-03-30T13:06:56Z

controllers/storagecluster/external_resources.go

@@ -176,10 +176,11 @@ func (r *StorageClusterReconciler) newExternalCephObjectStoreInstances(
 		return nil, err
 	}
 	// enable bucket healthcheck
+	time60s, _ := time.ParseDuration("60s")


You don't need to parse time anymore, just use 1 * time.Minute

Thanks, fixed (yet to push). Actually time.Minute is sufficient.

leseb

Why did all the CRDs get deleted in c428b8b?

obnoxxx · 2021-03-30T15:53:32Z

Why did all the CRDs get deleted in c428b8b?

Right that is weird... Investigating.

obnoxxx · 2021-03-30T15:59:36Z

updated with the current state
still some lint/test compile errors left
and then to clarify why the crds got removed after updating the rook version

obnoxxx · 2021-03-30T23:21:02Z

We are making progress. Now rook's image is fixed upstream via rook/rook#7512 and the image target starts to pass. Still some things to fix from the controller-runtime update...

obnoxxx · 2021-04-01T09:43:32Z

@jarrpa - I incorporated your patches on top of my PR and rebased. Looks a lot better! 😄 Only waiting for the e2e tests to complete now - everything else passed!

obnoxxx · 2021-04-01T10:42:09Z

From the test output:

...
time="2021-04-01T09:38:40Z" level=info msg="Pod ocs-operator-bundle-e2e-aws-ipi-install-install succeeded after 34m10s"
time="2021-04-01T09:38:40Z" level=info msg="Executing pod \"ocs-operator-bundle-e2e-aws-optional-operators-ci-subscribe\" running image \"stable:cli\""
time="2021-04-01T09:38:50Z" level=info msg="Container place-entrypoint in pod ocs-operator-bundle-e2e-aws-optional-operators-ci-subscribe completed successfully"
time="2021-04-01T09:38:50Z" level=info msg="Container cp-entrypoint-wrapper in pod ocs-operator-bundle-e2e-aws-optional-operators-ci-subscribe completed successfully"
time="2021-04-01T09:49:40Z" level=info msg="Container sidecar in pod ocs-operator-bundle-e2e-aws-optional-operators-ci-subscribe completed successfully"
Error from server (NotFound): namespaces "openshift-storage" not found
OO_INSTALL_NAMESPACE is 'openshift-storage' which does not exist: creating
Installing "ocs-operator" in namespace "openshift-storage"
OO_TARGET_NAMESPACES is '!install': targeting operator installation namespace (openshift-storage)
OperatorGroup does not exist: creating it
OperatorGroup name is "oo-z6rqx"
Creating CatalogSource
CatalogSource name is "oo-dzq4l"
Creating Subscription
Subscription name is "oo-qjvv5"
Waiting for installPlan to be created
installplan.operators.coreos.com/install-vz8f9 patched
Install Plan approved
Waiting for ClusterServiceVersion to become ready...
Timed out waiting for csv to become ready
Dumping Namespace openshift-storage as /logs/artifacts/ns-openshift-storage.yaml
Dumping OperatorGroup oo-z6rqx as /logs/artifacts/og-oo-z6rqx.yaml
Dumping CatalogSource oo-dzq4l as /logs/artifacts/cs-oo-dzq4l.yaml
Dumping Subscription oo-qjvv5 as /logs/artifacts/sub-oo-qjvv5.yaml
  Subscription oo-qjvv5 status state: UpgradePending
ClusterServiceVersion was never created
Dumping all ClusterServiceVersions in namespace openshift-storage to /logs/artifacts/openshift-storage-all-csvs.yaml
Dumping all installPlans in namespace openshift-storage as /logs/artifacts/installPlans.yaml
{"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"prow/entrypoint/run.go:80","func":"k8s.io/test-infra/prow/entrypoint.Options.Run","level":"error","msg":"Error executing test process","severity":"error","time":"2021-04-01T09:49:38Z"}
error: failed to execute wrapped command: exit status 1
time="2021-04-01T09:49:40Z" level=info msg="Container test in pod ocs-operator-bundle-e2e-aws-optional-operators-ci-subscribe failed, exit code 1, reason Error"
time="2021-04-01T09:49:40Z" level=info msg="Executing pod \"ocs-operator-bundle-e2e-aws-gather-aws-console\" running image \"pipeline:ocp-4.5-upi-installer\""
time="2021-04-01T09:49:51Z" level=info msg="Container place-entrypoint in pod ocs-operator-bundle-e2e-aws-gather-aws-console completed successfully"
time="2021-04-01T09:49:51Z" level=info msg="Container cp-entrypoint-wrapper in pod ocs-operator-bundle-e2e-aws-gather-aws-console completed successfully"
...

Not sure if this is a flake or real.

FWIW, I see pipeline:ocp-4.5-upi-installer - I think we shouldn't be running OCP 4.5 for testing OCS 4.8 / master, right?

umangachapagain · 2021-04-01T11:38:44Z

From the test output:

...
time="2021-04-01T09:38:40Z" level=info msg="Pod ocs-operator-bundle-e2e-aws-ipi-install-install succeeded after 34m10s"
time="2021-04-01T09:38:40Z" level=info msg="Executing pod \"ocs-operator-bundle-e2e-aws-optional-operators-ci-subscribe\" running image \"stable:cli\""
time="2021-04-01T09:38:50Z" level=info msg="Container place-entrypoint in pod ocs-operator-bundle-e2e-aws-optional-operators-ci-subscribe completed successfully"
time="2021-04-01T09:38:50Z" level=info msg="Container cp-entrypoint-wrapper in pod ocs-operator-bundle-e2e-aws-optional-operators-ci-subscribe completed successfully"
time="2021-04-01T09:49:40Z" level=info msg="Container sidecar in pod ocs-operator-bundle-e2e-aws-optional-operators-ci-subscribe completed successfully"
Error from server (NotFound): namespaces "openshift-storage" not found
OO_INSTALL_NAMESPACE is 'openshift-storage' which does not exist: creating
Installing "ocs-operator" in namespace "openshift-storage"
OO_TARGET_NAMESPACES is '!install': targeting operator installation namespace (openshift-storage)
OperatorGroup does not exist: creating it
OperatorGroup name is "oo-z6rqx"
Creating CatalogSource
CatalogSource name is "oo-dzq4l"
Creating Subscription
Subscription name is "oo-qjvv5"
Waiting for installPlan to be created
installplan.operators.coreos.com/install-vz8f9 patched
Install Plan approved
Waiting for ClusterServiceVersion to become ready...
Timed out waiting for csv to become ready
Dumping Namespace openshift-storage as /logs/artifacts/ns-openshift-storage.yaml
Dumping OperatorGroup oo-z6rqx as /logs/artifacts/og-oo-z6rqx.yaml
Dumping CatalogSource oo-dzq4l as /logs/artifacts/cs-oo-dzq4l.yaml
Dumping Subscription oo-qjvv5 as /logs/artifacts/sub-oo-qjvv5.yaml
  Subscription oo-qjvv5 status state: UpgradePending
ClusterServiceVersion was never created
Dumping all ClusterServiceVersions in namespace openshift-storage to /logs/artifacts/openshift-storage-all-csvs.yaml
Dumping all installPlans in namespace openshift-storage as /logs/artifacts/installPlans.yaml
{"component":"entrypoint","error":"wrapped process failed: exit status 1","file":"prow/entrypoint/run.go:80","func":"k8s.io/test-infra/prow/entrypoint.Options.Run","level":"error","msg":"Error executing test process","severity":"error","time":"2021-04-01T09:49:38Z"}
error: failed to execute wrapped command: exit status 1
time="2021-04-01T09:49:40Z" level=info msg="Container test in pod ocs-operator-bundle-e2e-aws-optional-operators-ci-subscribe failed, exit code 1, reason Error"
time="2021-04-01T09:49:40Z" level=info msg="Executing pod \"ocs-operator-bundle-e2e-aws-gather-aws-console\" running image \"pipeline:ocp-4.5-upi-installer\""
time="2021-04-01T09:49:51Z" level=info msg="Container place-entrypoint in pod ocs-operator-bundle-e2e-aws-gather-aws-console completed successfully"
time="2021-04-01T09:49:51Z" level=info msg="Container cp-entrypoint-wrapper in pod ocs-operator-bundle-e2e-aws-gather-aws-console completed successfully"
...

Not sure if this is a flake or real.

FWIW, I see pipeline:ocp-4.5-upi-installer - I think we shouldn't be running OCP 4.5 for testing OCS 4.8 / master, right?

We're using OCP 4.7.
The CSV was never created, so it doesn't look like a flake to me.

umangachapagain · 2021-04-01T11:39:03Z

/retest

jarrpa · 2021-04-01T16:25:51Z

Okay, the e2e failures may indeed be a flake. It seems it failed to even install the OCS CSV. Looking around, I found that the CatalogSource they generate to hold the CSV was in TRANSIENT_FAILURE:

time="2021-04-01T12:41:49Z" level=info msg="state.Key.Namespace=openshift-storage state.Key.Name=oo-bfs6t state.State=TRANSIENT_FAILURE"
time="2021-04-01T12:41:49Z" level=error msg="failed to list bundles: rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial tcp 172.30.74.197:50051: connect: no route to host\"" catalog="{oo-bfs6t openshift-storage}"

So, as is often the case, we are probably dealing with network issues and/or API limits.

/test ocs-operator-bundle-e2e-aws

jarrpa · 2021-04-01T20:01:53Z

/retest

obnoxxx · 2021-04-02T10:55:02Z

Created PR #1144 for only the CSV part of the rook upgrade.
Interestingly, it also fails e2e tests, similarly to this one.

obnoxxx · 2021-04-19T09:22:58Z

@raghavendra-talur yay, your patch fixes it!
I'm going to polish this PR and then we can move forward.

obnoxxx · 2021-04-19T11:43:31Z

When working on polishing / squashing the patches, I came across the test code that @jarrpa commented out. Removing the comments lets the unit tests fail this way:

=== RUN   TestCreateSCCs                                                                                                                              
    ocsinitialization_controller_test.go:234:                                                                                                         
                Error Trace:    ocsinitialization_controller_test.go:234                                                                              
                Error:          Received unexpected error:                                                                                            
                                Operation cannot be fulfilled on ocsinitializations.ocs.openshift.io "ocsinit": object was modified                   
                Test:           TestCreateSCCs                                                                                                        
                Messages:       [Case 1]: failed to update ocsInit status                                                                             
--- FAIL: TestCreateSCCs (0.00s)

Not sure really how to fix this. But ideally, we should be commenting out such lines...

obnoxxx · 2021-04-19T12:00:56Z

@jarrpa , @raghavendra-talur , @umangachapagain - any idea?

jarrpa · 2021-04-19T13:56:06Z

When working on polishing / squashing the patches, I came across the test code that @jarrpa commented out. Removing the comments lets the unit tests fail this way:

=== RUN   TestCreateSCCs                                                                                                                              
    ocsinitialization_controller_test.go:234:                                                                                                         
                Error Trace:    ocsinitialization_controller_test.go:234                                                                              
                Error:          Received unexpected error:                                                                                            
                                Operation cannot be fulfilled on ocsinitializations.ocs.openshift.io "ocsinit": object was modified                   
                Test:           TestCreateSCCs                                                                                                        
                Messages:       [Case 1]: failed to update ocsInit status                                                                             
--- FAIL: TestCreateSCCs (0.00s)

Not sure really how to fix this. But ideally, we should be commenting out such lines...

No idea. I wasn't able to figure anything further out, hence why I had to leave it like that just to make progress. It has something to do with how the fake reconciler is implemented, but that is still fairly opaque to me.

obnoxxx · 2021-04-19T15:34:42Z

@jarrpa - do you suggest to move forward with the commented-out lines?

(It just feels odd, since the lines seemed to have been there for a purpose... ;-)

jarrpa · 2021-04-19T15:39:59Z

For the sake of expediency, yes. There should at least me a TODO to figure out why they were suddenly a problem.

Latest rook master image. Signed-off-by: Michael Adam <obnox@redhat.com>

Signed-off-by: Michael Adam <obnox@redhat.com>

Latest master version with crds (re)addition to image. Picking up (among other things) crd updates, including volumereplication support. This is committing the result of: `go get github.com/rook/rook@1b77b9e` Signed-off-by: Michael Adam <obnox@redhat.com>

Signed-off-by: Michael Adam <obnox@redhat.com>

Why is there a diff now?... Signed-off-by: Michael Adam <obnox@redhat.com>

Quite some signatures have changed. Most importantly: - reconciler.Reconcile()now requries a context argument - fix use of interval to new Duration type - UpdateEvent has changed - NewFakeClientWithScheme is deprecated, using NewClientBuilder instead. Signed-off-by: Michael Adam <obnox@redhat.com>

Signed-off-by: Jose A. Rivera <jarrpa@redhat.com>

Prevent us from creating a second operatorgroup in the namsepace. Co-authored-by: Raghavendra Talur <raghavendra.talur@gmail.com> Signed-off-by: Michael Adam <obnox@redhat.com>

The controller runtime now uses leases for its leader election mechanism by default. It looks like the RBAC configuration in kubebuilder does not add this permission by default though. We add it manually here. Signed-off-by: Raghavendra Talur <raghavendra.talur@gmail.com>

obnoxxx · 2021-04-19T17:09:27Z

updated to the latest and greatest rook image rook/ceph:v1.6.0.95.gf4cfc7a
and polished the patches

still the only weird thing is the commented out test code lines... I added a TODO comment

jarrpa

/lgtm

openshift-ci-robot · 2021-04-19T17:15:12Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jarrpa

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jarrpa]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jarrpa · 2021-04-19T17:15:25Z

/override ci/prow/ci-index-dev-master-dependencies

openshift-ci-robot · 2021-04-19T17:15:39Z

@jarrpa: Overrode contexts on behalf of jarrpa: ci/prow/ci-index-dev-master-dependencies

In response to this:

/override ci/prow/ci-index-dev-master-dependencies

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

obnoxxx force-pushed the update-rook branch from 55ba7b3 to c428b8b Compare March 30, 2021 09:59

obnoxxx force-pushed the update-rook branch 2 times, most recently from 73dd135 to 60fe1aa Compare March 30, 2021 10:58

leseb reviewed Mar 30, 2021

View reviewed changes

obnoxxx force-pushed the update-rook branch from 52f4631 to a96362e Compare March 30, 2021 15:58

obnoxxx force-pushed the update-rook branch from a96362e to b0a4951 Compare March 30, 2021 23:09

obnoxxx force-pushed the update-rook branch 2 times, most recently from e209b68 to 2576c25 Compare March 31, 2021 00:36

umangachapagain mentioned this pull request Mar 31, 2021

[WIP] Upgrade to controller-runtime v0.8.1 and Kubernetes v0.20.2 #1017

Closed

obnoxxx force-pushed the update-rook branch from 2576c25 to d39bdf6 Compare April 1, 2021 08:47

obnoxxx mentioned this pull request Apr 1, 2021

Update rook csv rook/ceph:v1.5.0.737.g5478956 #1144

Closed

obnoxxx force-pushed the update-rook branch 2 times, most recently from 26e6275 to d8ef033 Compare April 1, 2021 23:27

obnoxxx force-pushed the update-rook branch 2 times, most recently from e3d0638 to 52a1e92 Compare April 9, 2021 08:02

obnoxxx force-pushed the update-rook branch from 8207ba6 to 64575ae Compare April 19, 2021 07:00

obnoxxx and others added 9 commits April 19, 2021 18:22

csv: update rook dependency to rook/ceph:v1.6.0.95.gf4cfc7a

1b544ef

Latest rook master image. Signed-off-by: Michael Adam <obnox@redhat.com>

csv: commit the result of make gen-latest-csv after rook update

69aacbf

Signed-off-by: Michael Adam <obnox@redhat.com>

deps: commit the result of make deps-update after rook update

72e4eea

Signed-off-by: Michael Adam <obnox@redhat.com>

csv: commit result of make gen-latest-csv again

b75ef25

Why is there a diff now?... Signed-off-by: Michael Adam <obnox@redhat.com>

ocsinit tests: always check errors

8561881

Signed-off-by: Jose A. Rivera <jarrpa@redhat.com>

deploy-manager: prevent creation of more than one operatorGroup

ecae19f

Prevent us from creating a second operatorgroup in the namsepace. Co-authored-by: Raghavendra Talur <raghavendra.talur@gmail.com> Signed-off-by: Michael Adam <obnox@redhat.com>

obnoxxx force-pushed the update-rook branch from 64575ae to 93a2928 Compare April 19, 2021 17:04

jarrpa approved these changes Apr 19, 2021

View reviewed changes

openshift-ci-robot assigned jarrpa Apr 19, 2021

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Apr 19, 2021

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 19, 2021

obnoxxx removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 19, 2021

raghavendra-talur mentioned this pull request Apr 19, 2021

Fix tests after fake client update #1157

Merged

openshift-merge-robot merged commit dee571f into red-hat-storage:master Apr 19, 2021

raghavendra-talur mentioned this pull request Apr 19, 2021

WIP: testing #1152

Closed

thotz mentioned this pull request Oct 13, 2021

add permissions for ocs-metrics-exporter #1368

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update rook to latest master build. #1141

Update rook to latest master build. #1141

obnoxxx commented Mar 30, 2021

obnoxxx commented Mar 30, 2021

obnoxxx commented Mar 30, 2021 •

edited

Loading

obnoxxx commented Mar 30, 2021

obnoxxx commented Mar 30, 2021

leseb Mar 30, 2021

obnoxxx Mar 30, 2021

leseb left a comment

obnoxxx commented Mar 30, 2021

obnoxxx commented Mar 30, 2021

obnoxxx commented Mar 30, 2021

obnoxxx commented Apr 1, 2021

obnoxxx commented Apr 1, 2021

umangachapagain commented Apr 1, 2021

umangachapagain commented Apr 1, 2021

jarrpa commented Apr 1, 2021

jarrpa commented Apr 1, 2021

obnoxxx commented Apr 2, 2021

obnoxxx commented Apr 19, 2021

obnoxxx commented Apr 19, 2021

obnoxxx commented Apr 19, 2021

jarrpa commented Apr 19, 2021

obnoxxx commented Apr 19, 2021

jarrpa commented Apr 19, 2021

obnoxxx commented Apr 19, 2021

jarrpa left a comment

openshift-ci-robot commented Apr 19, 2021

jarrpa commented Apr 19, 2021

openshift-ci-robot commented Apr 19, 2021

Update rook to latest master build. #1141

Update rook to latest master build. #1141

Conversation

obnoxxx commented Mar 30, 2021

obnoxxx commented Mar 30, 2021

obnoxxx commented Mar 30, 2021 • edited Loading

obnoxxx commented Mar 30, 2021

obnoxxx commented Mar 30, 2021

leseb Mar 30, 2021

Choose a reason for hiding this comment

obnoxxx Mar 30, 2021

Choose a reason for hiding this comment

leseb left a comment

Choose a reason for hiding this comment

obnoxxx commented Mar 30, 2021

obnoxxx commented Mar 30, 2021

obnoxxx commented Mar 30, 2021

obnoxxx commented Apr 1, 2021

obnoxxx commented Apr 1, 2021

umangachapagain commented Apr 1, 2021

umangachapagain commented Apr 1, 2021

jarrpa commented Apr 1, 2021

jarrpa commented Apr 1, 2021

obnoxxx commented Apr 2, 2021

obnoxxx commented Apr 19, 2021

obnoxxx commented Apr 19, 2021

obnoxxx commented Apr 19, 2021

jarrpa commented Apr 19, 2021

obnoxxx commented Apr 19, 2021

jarrpa commented Apr 19, 2021

obnoxxx commented Apr 19, 2021

jarrpa left a comment

Choose a reason for hiding this comment

openshift-ci-robot commented Apr 19, 2021

jarrpa commented Apr 19, 2021

openshift-ci-robot commented Apr 19, 2021

obnoxxx commented Mar 30, 2021 •

edited

Loading