-
Notifications
You must be signed in to change notification settings - Fork 788
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Operator does not see updated services #635
Comments
Have you installed the latest operator version 3.1.1? Is the first version supporting Cluster Scope |
The docker image that was pulled was xabarilcoding/healthchecksui-k8s-operator:latest I did the manual installation of this: First the CRD, then the rest. |
Could you paste the output for the following command to see the actual deployment image version that is being used? kubectl describe pod healthchecks-ui-k8s-operator-675949bc7-xrgtp -n healthchecks |
|
Thank you. And could you paste the output of this? kubectl describe crd healthchecks.aspnetcore.ui If you can, do a kubectl logs podname -n healthchecks as well and paste all the operator log. |
Operator logs:
|
I'm trying to reproduce the issue without any luck, this is the output for my tests: Test: Unlabel and label a service kubectl label svc hcnamespaces HealthChecks- -n namespaceddemo logs: [PushService] Namespace demo - Sending Type: Deleted - Service hcnamespaces with uri : http://10.96.67.66:80/health to ui endpoint: http://10.108.108.106:80 kubectl label svc hcnamespaces HealthChecks= -n namespaceddemo logs: [PushService] Namespace demo - Sending Type: Added - Service hcnamespaces with uri : http://10.96.67.66:80/health to ui endpoint: http://10.108.108.106:80 Test: Create a new deployment, create a service and label after a while kubectl create ns newns [15:44:31 INF] [PushService] Namespace demo - Sending Type: Added - Service newns-deploy with uri : http://10.97.252.242:80/health to ui endpoint: http://10.108.108.106:80 kubectl label svc newns-deploy HealthChecks- -n newns logs: [PushService] Namespace demo - Sending Type: Deleted - Service newns-deploy with uri : http://10.97.252.242:80/health to ui endpoint: http://10.108.108.106:80 Labelled service modification: kubectl label svc newns-deploy HealthChecksPath=health -n newns logs: [PushService] Namespace demo - Sending Type: Added - Service newns-deploy with uri : http://10.97.252.242:80/health to ui endpoint: http://10.108.108.106:80 [PushService] Namespace demo - Sending Type: Modified - Service newns-deploy with uri : http://10.97.252.242:80/health to ui endpoint: http://10.108.108.106:80 |
Could you delete and reinstall all operator required definitions to discard problems? We have an install / uninstall tool you can download and execute in the terminal: wget https://github.com/Xabaril/AspNetCore.Diagnostics.HealthChecks/blob/master/deploy/operator/installer/releases/operator-installer-win.exe && operator-installer-win.exe --delete and then operator-installer-win.exe If you are using linux, you also have the linux installer. This will uninstall and reinstall all the operator definitions |
OK, will try |
Hello @brechtvhb, any updates? |
Tomorrow or Friday I should have the time to have a look at it. |
I still have the issue even when using the installer. |
Very weird. I've just updated some labels and annotations after days of not using the operator and is responsive. If you are inside the cluster it uses inside cluster configuration: Here is the source implementation: To be honest, I do not know what's going on as the operator logs dropped watch connections. Could you show me the commands you use to add/update your services please? |
I use kubectl apply -f, through azure dev ops. Here's an example yaml:
Could you add a log statement that tells if InClusterConfig or BuildConfig was used? Just to be sure InClusterConfig is being used. |
I had the same problem but I could resolve it by redeploying the operator (deleting the pod and wait until a new pod has been started). |
@Kampfmoehre Yes indeed that's my workaround too till there's a permanent fix. |
This started happening when I updated the Kubernetes Client version. I do not know if the watch open connection drops. These days we are going to start migrating everything to dotnet 5.0 and I'll try all this stuff |
Ok, I think I found the problem. This is a regression from the previous version. [02:10:10 ERR] The request was canceled due to the configured HttpClient.Timeout of 100 seconds elapsing. The previous operator version used to reconnect on watch errors and I am gonna bring it back. This will be available in the next version |
Nice, thank you! |
We've suffered same behaviour in a different project. Will try to fix this ASAP with a workaround |
Shouldn't this issue be opened again in that case? |
What happened:
When updating a service, the operator does not see that the HealthChecks label got added to the service so it does not get pushed to the healthchecks ui.
What you expected to happen:
I expect to see the updated service in the health checks ui.
How to reproduce it (as minimally and precisely as possible):
Add the "HealthChecks" label to the service and apply the changes.
Note: The namespace of the service (default) is in a diffirent namespace than the healthchecks operator / dashboard (healthchecks).
The logs don't show anything that can give me a clue. The logs end after the last successfull PushService call.
[07:06:37 INF] [PushService] Notification result for infrachecker-api-service - status code: OK
Is it somehow possible to enable more verbose logs that might give a clue?
Environment:
The text was updated successfully, but these errors were encountered: