Customize node configuration: add pod-max-pids to avoid PID exhaustion

[tdihp]

What happened:

Applications can allocate too many threads, triggering EAGAIN when kubelet/containerd tries to create new thread with pthread_create. We observe PLEG failures and nodes not ready due to some offending application.

What you expected to happen:

Add pod-pid-limits as an option for custom node configuration. Configure a smaller value for pods should provide safety for node readiness.

How to reproduce it (as minimally and precisely as possible):

Simply add a testing Python pod in 2 steps and wait for around 6 mins to wait for node not ready:

kubectl run -it --rm --restart=Never --image=python:3-slim python -- python

import time

def thread_burst(n=1000, t=600):
    import threading
    
    threads = []
    for i in range(n):
        thread = threading.Thread(target=time.sleep, args=(t,))
        try:
            thread.start()
        except Exception as e:
            print('got exception when starting thread: %s' % e)
            break
        threads.append(thread)
    return threads

def main():
    threads = []
    while True:
        print('bursting threads')
        threads.extend(thread_burst(n=1000))
        time.sleep(10)

main()

Environment:

• Kubernetes version (use kubectl version): 1.19.7

[ghost]

Hi tdihp, AKS bot here 👋
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such:

1. If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster.
2. Please abide by the AKS repo Guidelines and Code of Conduct.
3. If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics?
4. Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS.
5. Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.
6. If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

[tdihp]

related: #323

[ghost]

Triage required from @Azure/aks-pm

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[ghost]

Issue needing attention of @Azure/aks-leads

[justindavies]

A PR has been raised to enable this in the azure cli for the next release, and the documentation will also be updated. I'll keep this open until this has been completed

[zacharias33]

Hi I am experiencing a similar issue on Kubernetes version 1.19.11. The PID space of a random node gets exhausted and my only solution for now is to restart that node. Do we have any updates on this feature?

[thunter1000]

Hi this is also impacting us with Kubernetes version 1.191.11. When the PID space is exhausted this impacts our calico-node pod which impacts everything on the node. Restarting the node does seem to resolve the issue. Has this change been released yet?

[ghost]

Action required from @Azure/aks-pm

[ghost]

Issue needing attention of @Azure/aks-leads

[tdihp]

For users affected by this, I'd suggest to identify the culprit application causing the exhaustion.

[tdihp]

Closing as custom node configuration is now GA with podMaxPids