Pods fails to terminate on Windows nodes on AKS

[andbess]

What happened:
Sporadically some of the pods fails to de terminated on Windows nodes

What you expected to happen:
All the pods are terminated automatically

How to reproduce it:
Since the issue happens sporadically, it's hard to reproduce

Additional information:
We experience the same issue in our AKS cluster as was reported in microsoft/Windows-Containers microsoft/Windows-Containers#106, except we're not using fluentbit or fluentd logging agents (we only use Azure Monitor with combination of Grafana). Sporadically some the pods are not automatically terminated and we have to manually run "kubectl delete --force" to force terminate the pods.
Could it be that Microsoft Monitoring Agent also sometime holds the container log and that's the reason that it cannot be removed?

The error message is always the same:
Message: unable to remove filesystem for 5bbd9142e090352ac34b5c118a69f0b76efd1f79c35e50bb264aad0c3731664f: CreateFile C:\ProgramData\docker\containers\5bbd9142e090352ac34b5c118a69f0b76efd1f79c35e50bb264aad0c3731664f\5bbd9142e090352ac34b5c118a69f0b76efd1f79c35e50bb264aad0c3731664f-json.log: Access is denied.

Environment:

• Kernel version: 10.0.17763.2114
• OS Image: Windows Server 2019 Datacenter
• Container runtime version: docker://20.10.6
• Kubernetes version: 1.21.2
• Size of cluster: 8 working Windows nodes
• The version of Microsoft Monitoring Agent we're using on our Windows Node: 10.20.18053.0

[ghost]

Hi andbess, AKS bot here 👋
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such:

1. If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster.
2. Please abide by the AKS repo Guidelines and Code of Conduct.
3. If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics?
4. Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS.
5. Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.
6. If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

[ghost]

Triage required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

This issue has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs within 15 days of this comment.

[ghost]

This issue will now be closed because it hasn't had any activity for 7 days after stale. andbess feel free to comment again on the next 7 days to reopen or open a new issue after that time if you still have a question/issue or suggestion.