Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

check for galera service list before proceeding #399

Closed

Conversation

zzzeek
Copy link
Contributor

@zzzeek zzzeek commented Apr 8, 2024

This updates galera operator to use a version which will wait for a Service to exist before proceeding to attempt setting the galera hostname

Copy link
Contributor

openshift-ci bot commented Apr 8, 2024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: zzzeek
Once this PR has been reviewed and has the lgtm label, please assign frenzyfriday for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@zzzeek
Copy link
Contributor Author

zzzeek commented Apr 8, 2024

recheck

@abays
Copy link
Contributor

abays commented Apr 8, 2024

@zzzeek: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:
Test name Commit Details Required Rerun command
ci/prow/keystone-operator-build-deploy-kuttl aea008a link true /test keystone-operator-build-deploy-kuttl

Full PR test history. Your PR dashboard.

KUTTL test failure could be legitimate:

    case.go:366: resource KeystoneAPI:keystone-kuttl-tests/keystone: .metadata.finalizers: slice length mismatch: 2 != 3
    logger.go:42: 18:35:28 | keystone_resources | skipping kubernetes event logging
=== CONT  kuttl
    harness.go:405: run tests finished
    harness.go:513: cleaning up
    harness.go:570: removing temp folder: ""
--- FAIL: kuttl (447.23s)
    --- FAIL: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/change_keystone_config (92.86s)
        --- PASS: kuttl/harness/keystone_scale (69.31s)
        --- PASS: kuttl/harness/keystone_tls (50.64s)
        --- FAIL: kuttl/harness/keystone_resources (234.19s)

@zzzeek
Copy link
Contributor Author

zzzeek commented Apr 8, 2024

@zzzeek: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:
Test name Commit Details Required Rerun command
ci/prow/keystone-operator-build-deploy-kuttl aea008a link true /test keystone-operator-build-deploy-kuttl
Full PR test history. Your PR dashboard.

KUTTL test failure could be legitimate:

    case.go:366: resource KeystoneAPI:keystone-kuttl-tests/keystone: .metadata.finalizers: slice length mismatch: 2 != 3
    logger.go:42: 18:35:28 | keystone_resources | skipping kubernetes event logging
=== CONT  kuttl
    harness.go:405: run tests finished
    harness.go:513: cleaning up
    harness.go:570: removing temp folder: ""
--- FAIL: kuttl (447.23s)
    --- FAIL: kuttl/harness (0.00s)
        --- PASS: kuttl/harness/change_keystone_config (92.86s)
        --- PASS: kuttl/harness/keystone_scale (69.31s)
        --- PASS: kuttl/harness/keystone_tls (50.64s)
        --- FAIL: kuttl/harness/keystone_resources (234.19s)

the 03-cleanup-keystoneendpoint is supposed to remove the placement endpoint. 03-assert sees the finalizer still there. my code change here has not affected deletes or finalizers at all, so looks like it's taking too long to remove the finalizer.

@zzzeek
Copy link
Contributor Author

zzzeek commented Apr 8, 2024

also I think we just saw this same error with a previous PR I was getting merged and it auto-cleared

@zzzeek
Copy link
Contributor Author

zzzeek commented Apr 8, 2024

also I think we just saw this same error with a previous PR I was getting merged and it auto-cleared

identical failure in this job on April 2: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openstack-k8s-operators_keystone-operator/393/pull-ci-openstack-k8s-operators-keystone-operator-main-keystone-operator-build-deploy-kuttl/1775150559943200768

@zzzeek
Copy link
Contributor Author

zzzeek commented Apr 8, 2024

/retest

@zzzeek
Copy link
Contributor Author

zzzeek commented Apr 8, 2024

the new message is in the logs, right here:

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openstack-k8s-operators_keystone-operator/399/pull-ci-openstack-k8s-operators-keystone-operator-main-keystone-operator-build-deploy-kuttl/1777418823331221504/artifacts/keystone-operator-build-deploy-kuttl/openstack-k8s-operators-gather/artifacts/must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-0081c32a29b7405a10ce5ef5c41fbcb65534e1d308ed12060db023744a0c7acb/namespaces/openstack-operators/pods/keystone-operator-controller-manager-754c57dbd4-v6cxd/logs/manager.log

2024-04-08T20:04:48Z	INFO	Controllers.KeystoneAPI	Found zero services for Galera instance openstack, requeueing ....	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "5658d5aa-67a1-4869-a7de-8e427b86bdd0"}

it hit the error three times, then applied the hostname:

2024-04-08T20:04:48Z	INFO	Controllers.KeystoneAPI	Found zero services for Galera instance openstack, requeueing ....	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "5658d5aa-67a1-4869-a7de-8e427b86bdd0"}
2024-04-08T20:04:48Z	INFO	Controllers.KeystoneAPI	Reconciling Service	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "a137b2d0-5c22-43ce-a596-2b9d97d66811"}
2024-04-08T20:04:48Z	INFO	Controllers.KeystoneAPI	Successfully ensured MariaDBAccount keystone exists; database username is keystone_625b	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "a137b2d0-5c22-43ce-a596-2b9d97d66811", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-08T20:04:48Z	INFO	Controllers.KeystoneAPI	Found zero services for Galera instance openstack, requeueing ....	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "a137b2d0-5c22-43ce-a596-2b9d97d66811"}
2024-04-08T20:04:48Z	INFO	Controllers.KeystoneAPI	Reconciling Service	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "94241196-79de-45a5-8f22-4c7de94f6988"}
2024-04-08T20:04:48Z	INFO	Controllers.KeystoneAPI	Successfully ensured MariaDBAccount keystone exists; database username is keystone_625b	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "94241196-79de-45a5-8f22-4c7de94f6988", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-08T20:04:48Z	INFO	Controllers.KeystoneAPI	Found zero services for Galera instance openstack, requeueing ....	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "94241196-79de-45a5-8f22-4c7de94f6988"}
2024-04-08T20:04:49Z	INFO	Memcached memcached is used by KeystoneAPI CR keystone
2024-04-08T20:04:49Z	INFO	Memcached memcached is used by KeystoneAPI CR keystone
2024-04-08T20:04:49Z	INFO	Controllers.KeystoneAPI	Reconciling Service	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "91113c93-bb87-48ef-b822-e1e5cf0ae6d1"}
2024-04-08T20:04:49Z	INFO	Controllers.KeystoneAPI	Successfully ensured MariaDBAccount keystone exists; database username is keystone_625b	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "91113c93-bb87-48ef-b822-e1e5cf0ae6d1", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-08T20:04:49Z	INFO	Controllers.KeystoneAPI	Applied new databasehostname openstack.openstack.svc to MariaDBDatabase keystone	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "91113c93-bb87-48ef-b822-e1e5cf0ae6d1"}

so it would seem this condition is common. Which might mean, we hit this error all the time and it just causes the reconcile to happen again anyway, and this was not actually a "problem" even though not that smooth? I'll look in an older run of this job and see what's there for this.

@zzzeek
Copy link
Contributor Author

zzzeek commented Apr 8, 2024

yup, this is common and non-fatal

here's an older run with the older version of the error:

https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/pr-logs/pull/openstack-k8s-operators_keystone-operator/385/pull-ci-openstack-k8s-operators-keystone-operator-main-keystone-operator-build-deploy-kuttl/1776177879902588928/artifacts/keystone-operator-build-deploy-kuttl/openstack-k8s-operators-gather/artifacts/must-gather/quay-io-openstack-k8s-operators-openstack-must-gather-sha256-0081c32a29b7405a10ce5ef5c41fbcb65534e1d308ed12060db023744a0c7acb/namespaces/openstack-operators/pods/keystone-operator-controller-manager-d4479f4fd-mxq8j/logs/manager.log

2024-04-05T10:00:00Z	ERROR	Reconciler error	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "572795c2-2ca5-4bb8-98fd-81ca62ccd0cf", "error": "error getting the DB service using label map[app:mariadb cr:mariadb-openstack]: %!w(<nil>)"}

it's messier, but gets to the same place

"KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "572795c2-2ca5-4bb8-98fd-81ca62ccd0cf"}
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Successfully ensured MariaDBAccount keystone exists; database username is keystone_7628	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "572795c2-2ca5-4bb8-98fd-81ca62ccd0cf", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-05T10:00:00Z	ERROR	Reconciler error	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "572795c2-2ca5-4bb8-98fd-81ca62ccd0cf", "error": "error getting the DB service using label map[app:mariadb cr:mariadb-openstack]: %!w(<nil>)"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:227
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Reconciling Service	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "fc69c0be-12ed-4bec-9ce5-7ec0c921c153"}
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Successfully ensured MariaDBAccount keystone exists; database username is keystone_7628	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "fc69c0be-12ed-4bec-9ce5-7ec0c921c153", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-05T10:00:00Z	ERROR	Reconciler error	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "fc69c0be-12ed-4bec-9ce5-7ec0c921c153", "error": "error getting the DB service using label map[app:mariadb cr:mariadb-openstack]: %!w(<nil>)"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:227
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Reconciling Service	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "328b8889-0086-4836-8ac3-bd4968696713"}
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Successfully ensured MariaDBAccount keystone exists; database username is keystone_7628	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "328b8889-0086-4836-8ac3-bd4968696713", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-05T10:00:00Z	ERROR	Reconciler error	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "328b8889-0086-4836-8ac3-bd4968696713", "error": "error getting the DB service using label map[app:mariadb cr:mariadb-openstack]: %!w(<nil>)"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:227
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Reconciling Service	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "e4321b62-1c10-4dfb-969d-7ad88763f28c"}
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Successfully ensured MariaDBAccount keystone exists; database username is keystone_7628	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "e4321b62-1c10-4dfb-969d-7ad88763f28c", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-05T10:00:00Z	ERROR	Reconciler error	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "e4321b62-1c10-4dfb-969d-7ad88763f28c", "error": "error getting the DB service using label map[app:mariadb cr:mariadb-openstack]: %!w(<nil>)"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:227
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Reconciling Service	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "73f412dd-fe92-412f-ac1d-84ca928ddfa4"}
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Successfully ensured MariaDBAccount keystone exists; database username is keystone_7628	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "73f412dd-fe92-412f-ac1d-84ca928ddfa4", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-05T10:00:00Z	ERROR	Reconciler error	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "73f412dd-fe92-412f-ac1d-84ca928ddfa4", "error": "error getting the DB service using label map[app:mariadb cr:mariadb-openstack]: %!w(<nil>)"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:227
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Reconciling Service	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "6186e78a-e757-4175-8692-a74a4b57d042"}
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Successfully ensured MariaDBAccount keystone exists; database username is keystone_7628	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "6186e78a-e757-4175-8692-a74a4b57d042", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-05T10:00:00Z	ERROR	Reconciler error	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "6186e78a-e757-4175-8692-a74a4b57d042", "error": "error getting the DB service using label map[app:mariadb cr:mariadb-openstack]: %!w(<nil>)"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:227
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Reconciling Service	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "18b2ac43-8e4c-4755-b6c1-e7c7a5873172"}
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Successfully ensured MariaDBAccount keystone exists; database username is keystone_7628	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "18b2ac43-8e4c-4755-b6c1-e7c7a5873172", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-05T10:00:00Z	ERROR	Reconciler error	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "18b2ac43-8e4c-4755-b6c1-e7c7a5873172", "error": "error getting the DB service using label map[app:mariadb cr:mariadb-openstack]: %!w(<nil>)"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:227
2024-04-05T10:00:00Z	INFO	Memcached memcached is used by KeystoneAPI CR keystone
2024-04-05T10:00:00Z	INFO	Memcached memcached is used by KeystoneAPI CR keystone
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Reconciling Service	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "ba888efd-bb76-4a32-931f-2535e6e69a78"}
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Successfully ensured MariaDBAccount keystone exists; database username is keystone_7628	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "ba888efd-bb76-4a32-931f-2535e6e69a78", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-05T10:00:00Z	ERROR	Reconciler error	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "ba888efd-bb76-4a32-931f-2535e6e69a78", "error": "error getting the DB service using label map[app:mariadb cr:mariadb-openstack]: %!w(<nil>)"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:227
2024-04-05T10:00:00Z	INFO	Memcached memcached is used by KeystoneAPI CR keystone
2024-04-05T10:00:00Z	INFO	Memcached memcached is used by KeystoneAPI CR keystone
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Reconciling Service	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "59269a92-3312-4fbf-a2e0-27f4f1b0f9c5"}
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Successfully ensured MariaDBAccount keystone exists; database username is keystone_7628	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "59269a92-3312-4fbf-a2e0-27f4f1b0f9c5", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-05T10:00:00Z	ERROR	Reconciler error	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "59269a92-3312-4fbf-a2e0-27f4f1b0f9c5", "error": "error getting the DB service using label map[app:mariadb cr:mariadb-openstack]: %!w(<nil>)"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:227
2024-04-05T10:00:00Z	INFO	keystoneapi-resource	default	{"name": "keystone"}
2024-04-05T10:00:00Z	INFO	keystoneapi-resource	validate update	{"name": "keystone"}
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Reconciling Service	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "441593b8-39d7-41a9-a553-cc3f58941831"}
2024-04-05T10:00:00Z	INFO	Controllers.KeystoneAPI	Successfully ensured MariaDBAccount keystone exists; database username is keystone_7628	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "441593b8-39d7-41a9-a553-cc3f58941831", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-05T10:00:00Z	ERROR	Reconciler error	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "441593b8-39d7-41a9-a553-cc3f58941831", "error": "error getting the DB service using label map[app:mariadb cr:mariadb-openstack]: %!w(<nil>)"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:227
2024-04-05T10:00:01Z	INFO	keystoneapi-resource	default	{"name": "keystone"}
2024-04-05T10:00:01Z	INFO	keystoneapi-resource	validate update	{"name": "keystone"}
2024-04-05T10:00:01Z	INFO	Controllers.KeystoneAPI	Reconciling Service	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "d39e0cfa-87ad-4010-8685-e5a6db00a4a4"}
2024-04-05T10:00:01Z	INFO	Controllers.KeystoneAPI	Successfully ensured MariaDBAccount keystone exists; database username is keystone_7628	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "d39e0cfa-87ad-4010-8685-e5a6db00a4a4", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-05T10:00:01Z	ERROR	Reconciler error	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "d39e0cfa-87ad-4010-8685-e5a6db00a4a4", "error": "error getting the DB service using label map[app:mariadb cr:mariadb-openstack]: %!w(<nil>)"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.5/pkg/internal/controller/controller.go:227
2024-04-05T10:00:01Z	INFO	keystoneapi-resource	default	{"name": "keystone"}
2024-04-05T10:00:01Z	INFO	keystoneapi-resource	validate update	{"name": "keystone"}
2024-04-05T10:00:01Z	INFO	keystoneapi-resource	default	{"name": "keystone"}
2024-04-05T10:00:01Z	INFO	keystoneapi-resource	validate update	{"name": "keystone"}
2024-04-05T10:00:02Z	INFO	keystoneapi-resource	default	{"name": "keystone"}
2024-04-05T10:00:02Z	INFO	keystoneapi-resource	validate update	{"name": "keystone"}
2024-04-05T10:00:02Z	INFO	Controllers.KeystoneAPI	Reconciling Service	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "cd3fa4b0-b968-4cb8-a445-566021463162"}
2024-04-05T10:00:02Z	INFO	Controllers.KeystoneAPI	Successfully ensured MariaDBAccount keystone exists; database username is keystone_7628	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "cd3fa4b0-b968-4cb8-a445-566021463162", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-05T10:00:02Z	INFO	Controllers.KeystoneAPI	Applied new databasehostname openstack.openstack.svc to MariaDBDatabase keystone	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "cd3fa4b0-b968-4cb8-a445-566021463162"}
2024-04-05T10:00:02Z	INFO	Controllers.KeystoneAPI	MariaDBDatabase object keystone created or patched	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "cd3fa4b0-b968-4cb8-a445-566021463162", "ObjectType": "*v1beta1.MariaDBDatabase", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-05T10:00:02Z	INFO	Controllers.KeystoneAPI	Reconciling Service	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "f09a1e2c-93b9-45e7-abd8-497d25f1c4ae"}
2024-04-05T10:00:02Z	INFO	Controllers.KeystoneAPI	Successfully ensured MariaDBAccount keystone exists; database username is keystone_7628	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "f09a1e2c-93b9-45e7-abd8-497d25f1c4ae", "ObjectType": "*v1beta1.MariaDBAccount", "ObjectNamespace": "openstack", "ObjectName": "keystone"}
2024-04-05T10:00:02Z	INFO	Controllers.KeystoneAPI	Applied new databasehostname openstack.openstack.svc to MariaDBDatabase keystone	{"controller": "keystoneapi", "controllerGroup": "keystone.openstack.org", "controllerKind": "KeystoneAPI", "KeystoneAPI": {"name":"keystone","namespace":"openstack"}, "namespace": "openstack", "name": "keystone", "reconcileID": "f09a1e2c-93b9-45e7-abd8-497d25f1c4ae"}

@zzzeek
Copy link
Contributor Author

zzzeek commented Apr 8, 2024

so that said, I would definitely merge the change in Galera and apply to all the operators, starting with the early init ones like Keystone here. just not sure if this is what leads to your dbsync condition.

@abays
Copy link
Contributor

abays commented Apr 9, 2024

Okay, good to know that it's transient. But yeah, if we merge the change across all operators, we'll have to wait and observe to see if it resolves the DBSync problem.

@zzzeek zzzeek force-pushed the check_for_galera_services branch 2 times, most recently from d8c3ab6 to 32003ca Compare April 9, 2024 13:54
This updates galera operator to use a version which will
wait for a Service to exist before proceeding to attempt setting the
galera hostname
@zzzeek zzzeek force-pushed the check_for_galera_services branch from 32003ca to b31afb1 Compare April 9, 2024 15:54
@zzzeek
Copy link
Contributor Author

zzzeek commented Apr 9, 2024

/retest

@zzzeek
Copy link
Contributor Author

zzzeek commented Apr 9, 2024

/restest

@zzzeek
Copy link
Contributor Author

zzzeek commented Apr 9, 2024

/retest

@abays
Copy link
Contributor

abays commented Apr 16, 2024

@zzzeek Was this waiting for some MariaDB PRs to merge so that you could remove the replace in the go.mod?

@zzzeek
Copy link
Contributor Author

zzzeek commented Apr 16, 2024

this PR was mostly to make sure the mariadbdatabase API part of the change didnt break, the fix to mariadb is on the operator side so I think in this PR it doesnt actually take effect unless it's in openstack-operator, so bumping at this level is not critical right now.

@zzzeek zzzeek closed this Apr 16, 2024
@zzzeek zzzeek deleted the check_for_galera_services branch April 16, 2024 12:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants