Network inaccessible when pod starting, when using Azure CNI + Calico #2750

alpeb · 2022-01-25T15:18:34Z

What happened:

Linkerd's control-plane pods have a sidecar proxy that starts before the main container. The proxy manifest has a postStart hook that blocks the creation of the main container until the proxy is ready. Only when using the combo Azure CNI + Calico, the proxy container appears to have no network, which avoids it to become ready, forbidding the pod startup to complete (it remains in status ContainerCreating).

What you expected to happen:

The network should be available as soon as the pod starts.

How to reproduce it:

Create a single-node AKS instance with Azure CNI (using its default settings) and Calico for Network Policy (although we deploy no NetworkPolicy resources for this repro).
Deploy a pod that attempts to hit the network with a postStart lifecycle hook that blocks the pod creation indefinitely:

apiVersion: v1
kind: Pod
metadata:
  name: curl
spec:
  containers:
  - image: curlimages/curl
    name: curl
    command: [ "sh", "-c", "--" ]
    args: [ "while true; do curl -k https://10.0.0.1; done;" ]
    lifecycle:
      postStart:
        exec:
          command: [ "sh", "-c", "--", "while true; do sleep 30; done;" ]

The pod remains in the ContainerCreating status as expected, but the curl command times out. This status forbids us from checking the pod's logs through kubectl logs, but we can get them by getting into the node through kubectl debug:

# cat default_curl_e03159c5-8660-474c-81f3-fa7b139fbbd7/curl/0.log
2022-01-25T14:55:03.076566072Z stderr F   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2022-01-25T14:55:03.076603573Z stderr F                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:02:10 --:--:--     0
2022-01-25T14:57:13.30494582Z stderr F curl: (28) Failed to connect to 10.0.0.1 port 443 after 130228 ms: Operation timed out
2022-01-25T14:57:13.30920836Z stderr F   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2022-01-25T14:57:13.309407062Z stderr F                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:02:11 --:--:--     0
2022-01-25T14:59:24.376433615Z stderr F curl: (28) Failed to connect to 10.0.0.1 port 443 after 131067 ms: Operation timed out
2022-01-25T14:59:24.380944273Z stderr F   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
2022-01-25T14:59:24.380961573Z stderr F                                  Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:02:11 --:--:--     0
2022-01-25T15:01:35.448517926Z stderr F curl: (28) Failed to connect to 10.0.0.1 port 443 after 131067 ms: Operation timed out
...

If we remove the lifecycle snippet, then curl works as expected:

$ k logs curl                                                                                                                                                                                                        
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                                                                                                                                      
                                 Dload  Upload   Total   Spent    Left  Speed                                                                                                                                        
100   165  100   165    0     0    150      0  0:00:01  0:00:01 --:--:--   150                                                                                                                                       
{                                                                                                                                                                                                                    
  "kind": "Status",                                                                                                                                                                                                  
  "apiVersion": "v1",                                                                                                                                                                                                
  "metadata": {                                                                                           
                                                                                                                                                                                                                     
  },                                                                                                                                                                                                                 
  "status": "Failure",                                                                                    
  "message": "Unauthorized",                                                                                                                                                                                         
  "reason": "Unauthorized",                                                                                                                                                                                          
  "code": 401                                                                                             
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current                           
                                 Dload  Upload   Total   Spent    Left  Speed                                                                                                                                        
}
...

Anything else we need to know?:

If we try with Kubenet+Calico, or Azure CNI without Calico, there is no issue.

Environment:

Kubernetes version: 1.21.7
Size of cluster: 1 node

The text was updated successfully, but these errors were encountered:

ghost · 2022-01-25T15:18:38Z

Hi alpeb, AKS bot here 👋
Thank you for posting on the AKS Repo, I'll do my best to get a kind human from the AKS team to assist you.

I might be just a bot, but I'm told my suggestions are normally quite good, as such:

If this case is urgent, please open a Support Request so that our 24/7 support team may help you faster.
Please abide by the AKS repo Guidelines and Code of Conduct.
If you're having an issue, could it be described on the AKS Troubleshooting guides or AKS Diagnostics?
Make sure your subscribed to the AKS Release Notes to keep up to date with all that's new on AKS.
Make sure there isn't a duplicate of this issue already reported. If there is, feel free to close this one and '+1' the existing issue.
If you have a question, do take a look at our AKS FAQ. We place the most common ones there!

ghost · 2022-01-27T18:00:44Z

Triage required from @Azure/aks-pm

ghost · 2022-01-27T18:23:33Z

@aanandr, @phealy would you be able to assist?

Issue Details