🌱 Improve CCT error logging

[sbueringer]

Signed-off-by: Stefan Büringer buringerst@vmware.com

What this PR does / why we need it:

Got some errors like this reported:

E0605 23:20:08.816431 1 machineset_controller.go:885] "Unable to retrieve Node status" err="failed to create cluster accessor: failed to get lock for cluster: cluster is locked already" ...

which is relatively confusing for folks not familiar with CCT

After this PR:

E0703 20:13:50.850650 176 machineset_controller.go:935] "Unable to retrieve Node status" err="failed to get client: failed to create cluster accessor: failed to get lock for cluster (probably because another worker is trying to create the client at the moment): cluster is locked already" ...

And the log message of the worker that tries & fails to create a client

E0703 20:13:13.387351 96 machineset_controller.go:935] "Unable to retrieve Node status" err="failed to get client: failed to create cluster accessor: error creating http client and mapper for remote cluster "default/development-32537": error creating client for remote cluster "default/development-32537": cluster is not reachable: client rate limiter Wait returned an error: context deadline exceeded - error from a previous attempt: EOF" ...

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):

[sbueringer]

/assign @fabriziopandini @chrischdi @vincepri

[sbueringer]

/test ?

[k8s-ci-robot]

@sbueringer: The following commands are available to trigger required jobs:

• /test pull-cluster-api-build-main
• /test pull-cluster-api-e2e-blocking-main
• /test pull-cluster-api-e2e-conformance-ci-latest-main
• /test pull-cluster-api-e2e-conformance-main
• /test pull-cluster-api-e2e-main
• /test pull-cluster-api-e2e-mink8s-main
• /test pull-cluster-api-e2e-upgrade-1-30-1-31-main
• /test pull-cluster-api-test-main
• /test pull-cluster-api-test-mink8s-main
• /test pull-cluster-api-verify-main

The following commands are available to trigger optional jobs:

• /test pull-cluster-api-apidiff-main

Use /test all to run the following jobs that were automatically triggered:

• pull-cluster-api-apidiff-main
• pull-cluster-api-build-main
• pull-cluster-api-e2e-blocking-main
• pull-cluster-api-test-main
• pull-cluster-api-verify-main

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[sbueringer]

/test pull-cluster-api-e2e-conformance-ci-latest-main
/test pull-cluster-api-e2e-conformance-main
/test pull-cluster-api-e2e-main
/test pull-cluster-api-e2e-mink8s-main
/test pull-cluster-api-e2e-upgrade-1-30-1-31-main

[vincepri]

/approve
/lgtm
/cherry-pick release-1.7

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 7eff86a8db0168bcf44c2dc29a52fb403f7c6b53

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[vincepri]

/cherry-pick release-1.7

[k8s-infra-cherrypick-robot]

@vincepri: #10826 failed to apply on top of branch "release-1.7":

Applying: Improve CCT error logging
Using index info to reconstruct a base tree...
M	controllers/remote/cluster_cache_tracker.go
Falling back to patching base and 3-way merge...
Auto-merging controllers/remote/cluster_cache_tracker.go
CONFLICT (content): Merge conflict in controllers/remote/cluster_cache_tracker.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 Improve CCT error logging
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.