Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[release-1.8] Periodically refresh APIServer CR in memory #2131

Merged
merged 1 commit into from
Nov 7, 2022
Merged

[release-1.8] Periodically refresh APIServer CR in memory #2131

merged 1 commit into from
Nov 7, 2022

Conversation

kubevirt-bot
Copy link
Contributor

This is an automated cherry-pick of #2128

/assign tiraboschi

Periodically refresh APIServer CR in memory

@tiraboschi
Periodically refresh APIServer CR in memory
6fd8359 
If not explicitly set on its CR,
HCO webhook is consuming TLS configuration
from Openshift cluster-wide APIServer CR.
For performance reason it's not reading it on each request
to the HCO CR but it's consuming a cached representation.
The cache was only refreshed by a controller
based on an informer.
We got reports that due to the nature
of changes in the APIServer CR, the connection
to the APIserver itself could become stuck:
```
W1025 13:50:16.898592       1 reflector.go:424] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: failed to list *v1.APIServer: Get "https://172.30.0.1:443/apis/config.openshift.io/v1/apiservers?resourceVersion=1572273": dial tcp 172.30.0.1:443: connect: connection refused
E1025 13:50:16.898683       1 reflector.go:140] sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262: Failed to watch *v1.APIServer: failed to list *v1.APIServer: Get "https://172.30.0.1:443/apis/config.openshift.io/v1/apiservers?resourceVersion=1572273": dial tcp 172.30.0.1:443: connect: connection refused
I1025 13:50:43.182360       1 trace.go:205] Trace[621733159]: "Reflector ListAndWatch" name:sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262 (25-Oct-2022 13:50:19.338) (total time: 23843ms):
Trace[621733159]: ---"Objects listed" error:<nil> 23843ms (13:50:43.182)
Trace[621733159]: [23.843677488s] [23.843677488s] END
I1025 13:50:43.716723       1 trace.go:205] Trace[255710357]: "Reflector ListAndWatch" name:sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262 (25-Oct-2022 13:50:12.260) (total time: 31456ms):
Trace[255710357]: ---"Objects listed" error:<nil> 31456ms (13:50:43.716)
Trace[255710357]: [31.45666834s] [31.45666834s] END
I1025 13:50:43.968506       1 trace.go:205] Trace[2001360213]: "Reflector ListAndWatch" name:sigs.k8s.io/controller-runtime/pkg/cache/internal/informers_map.go:262 (25-Oct-2022 13:50:11.520) (total time: 32447ms):
Trace[2001360213]: ---"Objects listed" error:<nil> 32447ms (13:50:43.968)
Trace[2001360213]: [32.44785055s] [32.44785055s] END
```

On controller runtime the default SyncPeriod when all the
watched resources are refreshed is 10 hourse (
see kubernetes-sigs/controller-runtime#521
for its reasons) but it appears
too long for this specific use case.

Let's ensure we read the APIServer CR at least once every minute.

Make the logs less verbose.

Fixes: https://bugzilla.redhat.com/show_bug.cgi?id=2137896

Remove this once kubernetes-sigs/controller-runtime#2032
is properly addressed

Signed-off-by: Simone Tiraboschi <stirabos@redhat.com>
@kubevirt-bot kubevirt-bot mentioned this pull request Nov 2, 2022
Merged
11 tasks
@kubevirt-bot kubevirt-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Nov 2, 2022
@sonarcloud
Copy link

sonarcloud bot commented Nov 2, 2022

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
No Duplication information No Duplication information

@hco-bot
Copy link
Collaborator

hco-bot commented Nov 2, 2022

hco-e2e-upgrade-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-index-sno-azure

@kubevirt-bot
Copy link
Contributor Author

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-upgrade-index-sno-azure

In response to this:

hco-e2e-upgrade-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-index-sno-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@hco-bot
Copy link
Collaborator

hco-bot commented Nov 2, 2022

hco-e2e-upgrade-prev-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-sno-azure
hco-e2e-image-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-image-index-sno-azure

@kubevirt-bot
Copy link
Contributor Author

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-image-index-sno-azure, ci/prow/hco-e2e-upgrade-prev-index-sno-azure

In response to this:

hco-e2e-upgrade-prev-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-sno-azure
hco-e2e-image-index-sno-aws lane succeeded.
/override ci/prow/hco-e2e-image-index-sno-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@hco-bot
Copy link
Collaborator

hco-bot commented Nov 2, 2022

hco-e2e-upgrade-prev-index-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-azure
hco-e2e-upgrade-index-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-index-azure

@kubevirt-bot
Copy link
Contributor Author

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-upgrade-index-azure, ci/prow/hco-e2e-upgrade-prev-index-azure

In response to this:

hco-e2e-upgrade-prev-index-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-prev-index-azure
hco-e2e-upgrade-index-aws lane succeeded.
/override ci/prow/hco-e2e-upgrade-index-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@tiraboschi
Copy link
Member

/test hco-e2e-image-index-azure

@kubevirt-bot
Copy link
Contributor Author

@tiraboschi: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

  • /test build-hco-test-utils-image
  • /test pull-hyperconverged-cluster-operator-e2e-k8s-1.23
  • /test pull-hyperconverged-cluster-operator-e2e-k8s-1.24

Use /test all to run the following jobs that were automatically triggered:

  • pull-hyperconverged-cluster-operator-e2e-k8s-1.23
  • pull-hyperconverged-cluster-operator-e2e-k8s-1.24

In response to this:

/test hco-e2e-image-index-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@hco-bot
Copy link
Collaborator

hco-bot commented Nov 2, 2022

hco-e2e-kv-smoke-gcp lane succeeded.
/override ci/prow/hco-e2e-kv-smoke-azure

@kubevirt-bot
Copy link
Contributor Author

@hco-bot: Overrode contexts on behalf of hco-bot: ci/prow/hco-e2e-kv-smoke-azure

In response to this:

hco-e2e-kv-smoke-gcp lane succeeded.
/override ci/prow/hco-e2e-kv-smoke-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@kubevirt-bot kubevirt-bot added the lgtm Indicates that a PR is ready to be merged. label Nov 2, 2022
@tiraboschi
Copy link
Member

/approve

@kubevirt-bot
Copy link
Contributor Author

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tiraboschi

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubevirt-bot kubevirt-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 2, 2022
@tiraboschi
Copy link
Member

/override-bot

@tiraboschi
Copy link
Member

/retest

@tiraboschi
Copy link
Member

/test hco-e2e-image-index-azure

@kubevirt-bot
Copy link
Contributor Author

@tiraboschi: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

  • /test build-hco-test-utils-image
  • /test pull-hyperconverged-cluster-operator-e2e-k8s-1.23
  • /test pull-hyperconverged-cluster-operator-e2e-k8s-1.24

Use /test all to run the following jobs that were automatically triggered:

  • pull-hyperconverged-cluster-operator-e2e-k8s-1.23
  • pull-hyperconverged-cluster-operator-e2e-k8s-1.24

In response to this:

/test hco-e2e-image-index-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@tiraboschi
Copy link
Member

/retest

1 similar comment
@tiraboschi
Copy link
Member

/retest

@tiraboschi
Copy link
Member

/retest

3 similar comments
@tiraboschi
Copy link
Member

/retest

@tiraboschi
Copy link
Member

/retest

@tiraboschi
Copy link
Member

/retest

@openshift-ci
Copy link

openshift-ci bot commented Nov 5, 2022

@kubevirt-bot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/hco-e2e-upgrade-index-sno-azure 6fd8359 link false /test hco-e2e-upgrade-index-sno-azure
ci/prow/hco-e2e-image-index-sno-azure 6fd8359 link false /test hco-e2e-image-index-sno-azure
ci/prow/hco-e2e-upgrade-prev-index-sno-azure 6fd8359 link false /test hco-e2e-upgrade-prev-index-sno-azure
ci/prow/hco-e2e-upgrade-index-azure 6fd8359 link true /test hco-e2e-upgrade-index-azure
ci/prow/hco-e2e-upgrade-prev-index-azure 6fd8359 link true /test hco-e2e-upgrade-prev-index-azure
ci/prow/hco-e2e-kv-smoke-azure 6fd8359 link true /test hco-e2e-kv-smoke-azure
ci/prow/hco-e2e-image-index-azure 6fd8359 link true /test hco-e2e-image-index-azure

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@tiraboschi
Copy link
Member

hco-e2e-image-index-aws and hco-e2e-image-index-gcp passed, ignoring hco-e2e-image-index-azure
/override ci/prow/hco-e2e-image-index-azure

@kubevirt-bot
Copy link
Contributor Author

@tiraboschi: Overrode contexts on behalf of tiraboschi: ci/prow/hco-e2e-image-index-azure

In response to this:

hco-e2e-image-index-aws and hco-e2e-image-index-gcp passed, ignoring hco-e2e-image-index-azure
/override ci/prow/hco-e2e-image-index-azure

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@kubevirt-bot kubevirt-bot merged commit 3dfd5b3 into kubevirt:release-1.8 Nov 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tiraboschi tiraboschi tiraboschi approved these changes

@sradco sradco Awaiting requested review from sradco

Assignees

@tiraboschi tiraboschi

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. lgtm Indicates that a PR is ready to be merged. release-note Denotes a PR that will be considered when it comes time to generate release notes. size/S
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@kubevirt-bot @hco-bot @tiraboschi