k8s: Cross-host pod connections not properly rendered or missing

[2opremio]

Pod connections (no services nor kube-proxy involved) across hosts in kubernetes are not properly reflected in the UI.

Following a modified example from https://github.com/kelseyhightower/craft-kubernetes-workshop , I am using an nginx frontend pod (IP 10.152.1.6) which uses persistent connections to loadbalance requests to some hello pod backends with IPs (10.152.1.5, 10.152.0.3 and 10.152.4.5).

nginx is using this configuration:

upstream hello {                                                                                                                                            
    server 10.152.1.5;                                                                                                                                      
    server 10.152.0.3;                                                                                                                                      
    server 10.152.4.5;                                                                                                                                      
    keepalive 100;                                                                                                                                            
}                                                                                                                                                           

upstream auth {                                                                                                                                             
    server auth.default.svc.cluster.local;                                                                                                                  
    keepalive 2;                                                                                                                                            
}                                                                                                                                                           

server {                                                                                                                                                    
    listen 443;                                                                                                                                             
    ssl    on;                                                                                                                                              

    ssl_certificate     /etc/tls/cert.pem;                                                                                                                  
    ssl_certificate_key /etc/tls/key.pem;                                                                                                                   

    proxy_http_version 1.1;                                                                                                                                 
    proxy_set_header Connection "";                                                                                                                         

    location / {                                                                                                                                            
        proxy_pass http://hello;                                                                                                                            
    }                                                                                                                                                       

    location /login {                                                                                                                                       
        proxy_pass http://auth;                                                                                                                             
    }                                                                                                                                                       
}

The frontend shows connections to all the pods:

$ netstat -tpn | grep ESTABLISHED  | grep 10.152.0.3                                                                                                         
tcp        0      0 10.152.1.6:60463        10.152.0.3:80           ESTABLISHED -                                                                           
$ netstat -tpn | grep ESTABLISHED  | grep 10.152.1.5                                                                                                         
tcp        0      0 10.152.1.6:42595        10.152.1.5:80           ESTABLISHED -                                                                           
$ netstat -tpn | grep ESTABLISHED  | grep 10.152.4.5                                                                                                         
tcp        0      0 10.152.1.6:42416        10.152.4.5:80           ESTABLISHED -                                                                           

Similarly, the backends show connections from the frontend. For instance:

/ # netstat -pnt                                                                                                                                            
Active Internet connections (w/o servers)                                                                                                                   
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name                                                            
tcp        0      0 ::ffff:10.152.0.3:80    ::ffff:10.152.1.6:60463 ESTABLISHED 1/hello                                                                     
/ #                                                                                                                                                         
Connected

/ # netstat -npt                                                                                                                                            
Active Internet connections (w/o servers)                                                                                                                   
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name                                                            
tcp        0      0 ::ffff:10.152.1.5:80    ::ffff:10.152.1.6:42595 ESTABLISHED 1/hello                                                                     
/ #                                                                                                                                                         

Pods 10.152.1.6 and 10.152.1.5 are co-located in the same host, and their container connections are properly rendered:

However that's not the case for cross-host pod connections. There are three problems:

1. One end is attributed to the /pause container, which is tagged as a system container and consequently is not displayed in the normal view. Scope has the PID of the destination process since the connection is persistent so it should be able to attribute it to right container (Related Containers using other containers net namespace don't have IP addresses #1394 )

2. The other end is attributed to The Internet. Although each kubernetes node uses a different network for its pod IPs the app should know how to join them together.

   

3. Some connections aren't rendered, for instance: 10.152.1.6:60463 <-> 10.152.0.3:80.
   

   But the report seems to have enough information to render it (although I wonder why the connection is conntracked instead of obtained from /proc).

   

It seems like connections figured out from /proc aren't properly merged with the ones figured out from conntrack (the first should have higher precedence since they provide PIDs).

Report: report_crosshost_pod_connections.json.gz

[kelseyhightower]

@2opremio This issue captures everything perfectly.

[2opremio]

This issue captures everything perfectly.

Great, we will try to get this fixed soon, hopefully next week.

[tomwilkie]

Fixed by #1410