Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KEDA frequent reconciling causing ApiServer LIST verb Saturation #3678

Closed
djsly opened this issue Sep 19, 2022 · 5 comments
Closed

KEDA frequent reconciling causing ApiServer LIST verb Saturation #3678

djsly opened this issue Sep 19, 2022 · 5 comments
Labels
bug Something isn't working stale All issues that are marked as stale due to inactivity

Comments

@djsly
Copy link

djsly commented Sep 19, 2022

Report

Keda operator goes into a "restart" loop causing a constant reconciliation loop. Due to the non cache LIST operation the apiserver was not able to answer LIST request for other services as well.

Expected Behavior

Keda Operator should use cached List operation and it should also not get into reconciliation loop

Actual Behavior

Graph showing the apiserver saturation time

apiserver-list-saturation

Steps to Reproduce the Problem

  1. Hard to tell, it started after Keda Operator lost it's leader election due to a slow ApiServer response : Helm operator leader election lost operator-framework/operator-sdk#5186

Logs from KEDA operator

keda-pods-apiserver-list-saturation.csv

KEDA Version

2.6.1

Kubernetes Version

1.23

Platform

Microsoft Azure

Scaler Details

Prometheus

Anything else?

Azure Support: 2209160030000259 for the impact and severity of such a bug

@djsly djsly added the bug Something isn't working label Sep 19, 2022
@tomkerkhove tomkerkhove moved this to Proposed in Roadmap - KEDA Core Sep 19, 2022
@zroubalik
Copy link
Member

zroubalik commented Sep 19, 2022

Hi, this is probably a duplicate of #2914

This issue has been solved most likely. By chance, could you please update KEDA version to the latest one?

@zroubalik zroubalik changed the title Keda frequent reconciling causing ApiServer LIST verb Saturation KEDA frequent reconciling causing ApiServer LIST verb Saturation Sep 19, 2022
@djsly
Copy link
Author

djsly commented Sep 19, 2022

we certainly can, what do you mean "by chance" ?

@zroubalik
Copy link
Member

zroubalik commented Sep 19, 2022

we certainly can, what do you mean "by chance" ?

the original sentece was missing a . - I hope that it makes sense now ;)

@stale
Copy link

stale bot commented Nov 18, 2022

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale All issues that are marked as stale due to inactivity label Nov 18, 2022
@stale
Copy link

stale bot commented Nov 25, 2022

This issue has been automatically closed due to inactivity.

@stale stale bot closed this as completed Nov 25, 2022
Repository owner moved this from Proposed to Ready To Ship in Roadmap - KEDA Core Nov 25, 2022
@JorTurFer JorTurFer moved this from Ready To Ship to Done in Roadmap - KEDA Core Mar 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working stale All issues that are marked as stale due to inactivity
Projects
Archived in project
Development

No branches or pull requests

2 participants