nginx-ingress: occasional 503 Service Temporarily Unavailable #821

troian · 2017-06-07T11:24:22Z

I'm experiencing often 503 response from nginx-ingress-controller which returns as well
Kubernetes Ingress Controller Fake Certificate (2) instead of provided wildcard certificate.
Image is gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.7

Looks like at some point nginx cannot resolve proper server_name and returns fake. But then why it ignores --default-ssl-certificate argument.
Anyway I'm out of thoughts thus any help appreciated

Cluster is running at GKE

Good request

$ curl -I -v -L https://environment.trysimply.com/cluster/dashboard -k
*   Trying 104.197.245.109...
* TCP_NODELAY set
* Connected to environment.trysimply.com (104.197.245.109) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: *.trysimply.com
* Server certificate: Go Daddy Secure Certificate Authority - G2
* Server certificate: Go Daddy Root Certificate Authority - G2
> HEAD /cluster/dashboard HTTP/1.1
> Host: environment.trysimply.com
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 302 Moved Temporarily
HTTP/1.1 302 Moved Temporarily
< Server: nginx/1.13.0
Server: nginx/1.13.0
< Date: Wed, 07 Jun 2017 11:12:29 GMT
Date: Wed, 07 Jun 2017 11:12:29 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 161
Content-Length: 161
< Connection: keep-alive
Connection: keep-alive
< Location: https://environment.trysimply.com/oauth2/sign_in
Location: https://environment.trysimply.com/oauth2/sign_in
< Strict-Transport-Security: max-age=15724800; includeSubDomains;
Strict-Transport-Security: max-age=15724800; includeSubDomains;

<
* Curl_http_done: called premature == 0
* Connection #0 to host environment.trysimply.com left intact
* Issue another request to this URL: 'https://environment.trysimply.com/oauth2/sign_in'
* Found bundle for host environment.trysimply.com: 0x7ffc31c0c130 [can pipeline]
* Re-using existing connection! (#0) with host environment.trysimply.com
* Connected to environment.trysimply.com (104.197.245.109) port 443 (#0)
> HEAD /oauth2/sign_in HTTP/1.1
> Host: environment.trysimply.com
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Server: nginx/1.13.0
Server: nginx/1.13.0
< Date: Wed, 07 Jun 2017 11:12:29 GMT
Date: Wed, 07 Jun 2017 11:12:29 GMT
< Content-Type: text/html; charset=utf-8
Content-Type: text/html; charset=utf-8
< Connection: keep-alive
Connection: keep-alive
< Set-Cookie: environment_oauth2_proxy=; Path=/; Domain=environment.trysimply.com; Expires=Wed, 07 Jun 2017 10:12:29 GMT; HttpOnly; Secure
Set-Cookie: environment_oauth2_proxy=; Path=/; Domain=environment.trysimply.com; Expires=Wed, 07 Jun 2017 10:12:29 GMT; HttpOnly; Secure
< Strict-Transport-Security: max-age=15724800; includeSubDomains;
Strict-Transport-Security: max-age=15724800; includeSubDomains;

<
* Curl_http_done: called premature == 0
* Connection #0 to host environment.trysimply.com left intact

Bad request

$ curl -I -v -L https://environment.trysimply.com/cluster/dashboard -k
*   Trying 104.197.245.109...
* TCP_NODELAY set
* Connected to environment.trysimply.com (104.197.245.109) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate: Kubernetes Ingress Controller Fake Certificate
> HEAD /cluster/dashboard HTTP/1.1
> Host: environment.trysimply.com
> User-Agent: curl/7.51.0
> Accept: */*
>
< HTTP/1.1 503 Service Temporarily Unavailable
HTTP/1.1 503 Service Temporarily Unavailable
< Server: nginx/1.13.0
Server: nginx/1.13.0
< Date: Wed, 07 Jun 2017 11:12:33 GMT
Date: Wed, 07 Jun 2017 11:12:33 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 213
Content-Length: 213
< Connection: keep-alive
Connection: keep-alive
< Strict-Transport-Security: max-age=15724800; includeSubDomains;
Strict-Transport-Security: max-age=15724800; includeSubDomains;

<
* Curl_http_done: called premature == 0
* Connection #0 to host environment.trysimply.com left intact

Configs used

apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-ingress
  namespace: kube-system
data:
  force-ssl-redirect: "true"
  ssl-redirect: "true"
  use-proxy-protocol: "false"

---

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
  namespace: kube-system
  labels:
    app: nginx-ingress
spec:
  type: LoadBalancer
  ports:
  - port: 80
    name: http
  - port: 443
    name: https
  - port: 18443
    name: vpn
  selector:
    k8s-app: nginx-ingress

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: redirected-environment.trysimply.com
  namespace: kube-system
  annotations:
    ingress.kubernetes.io/auth-signin: "https://environment.trysimply.com/oauth2/sign_in"
    ingress.kubernetes.io/auth-url: "https://environment.trysimply.com/oauth2/auth"
    kubernetes.io/ingress.class: "nginx"
    ingress.kubernetes.io/rewrite-target: /
spec:
  tls:
  - hosts:
    - environment.trysimply.com
    secretName: star-trysimply-com
  rules:
  - host: environment.trysimply.com
    http:
      paths:
      - path: /cluster/dashboard
        backend:
          serviceName: kubernetes-dashboard
          servicePort: 80

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: oauth2-proxy
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
  - hosts:
    - environment.trysimply.com
    secretName: star-trysimply-com
  rules:
  - host: environment.trysimply.com
    http:
      paths:
      - path: /oauth2
        backend:
          serviceName: oauth2-proxy
          servicePort: 4180

---

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress
  namespace: kube-system
spec:
  replicas: 1
  strategy:
    type: Recreate
  revisionHistoryLimit: 1
  template:
    metadata:
      labels:
        k8s-app: nginx-ingress
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - image: gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.7
        name: nginx-ingress
        imagePullPolicy: Always
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        - containerPort: 18443
          hostPort: 18443
        args:
        - /nginx-ingress-controller
        - --default-backend-service=kube-system/default-http-backend
        - --tcp-services-configmap=kube-system/tcp
        - --configmap=kube-system/nginx-ingress
#        - --watch-namespace=kube-system
#        - --ingress-class=nginx
#        - --force-namespace-isolation=true
#        - --healthz-port=10254
#        - --logtostderr
        - --default-ssl-certificate=kube-system/star-trysimply-com
        - --v=2

Pod nginx.conf

$ kubectl exec -it -n kube-system nginx-ingress-455914881-16zs1 -- cat /etc/nginx/nginx.conf

daemon off;

worker_processes 1;
pid /run/nginx.pid;

worker_rlimit_nofile 1047552;
events {
    multi_accept        on;
    worker_connections  16384;
    use                 epoll;
}

http {
    set_real_ip_from    0.0.0.0/0;
    real_ip_header      X-Forwarded-For;

    real_ip_recursive   on;

    geoip_country       /etc/nginx/GeoIP.dat;
    geoip_city          /etc/nginx/GeoLiteCity.dat;
    geoip_proxy_recursive on;
    # lua section to return proper error codes when custom pages are used
    lua_package_path '.?.lua;/etc/nginx/lua/?.lua;/etc/nginx/lua/vendor/lua-resty-http/lib/?.lua;';
    init_by_lua_block {
        require("error_page")
    }

    sendfile            on;
    aio                 threads;
    tcp_nopush          on;
    tcp_nodelay         on;

    log_subrequest      on;

    reset_timedout_connection on;

    keepalive_timeout  75s;
    keepalive_requests 100;

    client_header_buffer_size       1k;
    large_client_header_buffers     4 8k;
    client_body_buffer_size         8k;

    http2_max_field_size            4k;
    http2_max_header_size           16k;

    types_hash_max_size             2048;
    server_names_hash_max_size      1024;
    server_names_hash_bucket_size   64;
    map_hash_bucket_size            64;

    underscores_in_headers          off;
    ignore_invalid_headers          on;

    include /etc/nginx/mime.types;
    default_type text/html;
    gzip on;
    gzip_comp_level 5;
    gzip_http_version 1.1;
    gzip_min_length 256;
    gzip_types application/atom+xml application/javascript application/x-javascript application/json application/rss+xml application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/svg+xml image/x-icon text/css text/plain text/x-component;
    gzip_proxied any;

    server_tokens on;

    # disable warnings
    uninitialized_variable_warn off;

    log_format upstreaminfo '$the_x_forwarded_for - [$the_x_forwarded_for] - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" $request_length $request_time [$proxy_upstream_name] $upstream_addr $upstream_response_length $upstream_response_time $upstream_status';

    map $request_uri $loggable {
        default 1;
    }

    access_log /var/log/nginx/access.log upstreaminfo if=$loggable;
    error_log  /var/log/nginx/error.log notice;

    resolver 10.59.240.10 valid=30s;

    # Retain the default nginx handling of requests without a "Connection" header
    map $http_upgrade $connection_upgrade {
        default          upgrade;
        ''               close;
    }

    # trust http_x_forwarded_proto headers correctly indicate ssl offloading
    map $http_x_forwarded_proto $pass_access_scheme {
        default          $http_x_forwarded_proto;
        ''               $scheme;
    }

    map $http_x_forwarded_port $pass_server_port {
       default           $http_x_forwarded_port;
       ''                $server_port;
    }

    map $pass_access_scheme $the_x_forwarded_for {
       default           $remote_addr;
       https             $proxy_protocol_addr;
    }

    map $pass_access_scheme $the_real_ip {
       default           $remote_addr;
       https             $proxy_protocol_addr;
    }

    # map port 442 to 443 for header X-Forwarded-Port
    map $pass_server_port $pass_port {
        442              443;
        default          $pass_server_port;
    }

    # Map a response error watching the header Content-Type
    map $http_accept $httpAccept {
        default          html;
        application/json json;
        application/xml  xml;
        text/plain       text;
    }

    map $httpAccept $httpReturnType {
        default          text/html;
        json             application/json;
        xml              application/xml;
        text             text/plain;
    }

    # Obtain best http host
    map $http_host $best_http_host {
        default          $http_host;
        ''               $host;
    }

    server_name_in_redirect off;
    port_in_redirect        off;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;

    # turn on session caching to drastically improve performance
    ssl_session_cache builtin:1000 shared:SSL:10m;
    ssl_session_timeout 10m;

    # allow configuring ssl session tickets
    ssl_session_tickets on;

    # slightly reduce the time-to-first-byte
    ssl_buffer_size 4k;

    # allow configuring custom ssl ciphers
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
    ssl_prefer_server_ciphers on;

    ssl_ecdh_curve secp384r1;

    # In case of errors try the next upstream server before returning an error
    proxy_next_upstream                     error timeout invalid_header http_502 http_503 http_504;

    proxy_ssl_session_reuse on;

    upstream kube-system-kubernetes-dashboard-80 {
        # Load balance algorithm; empty for round robin, which is the default
        least_conn;
        server 10.56.3.4:9090 max_fails=0 fail_timeout=0;
    }
    upstream kube-system-oauth2-proxy-4180 {
        # Load balance algorithm; empty for round robin, which is the default
        least_conn;
        server 10.56.1.122:4180 max_fails=0 fail_timeout=0;
    }
    upstream upstream-default-backend {
        # Load balance algorithm; empty for round robin, which is the default
        least_conn;
        server 10.56.3.48:8080 max_fails=0 fail_timeout=0;
    }

    server {
        server_name _;
        listen 80 default_server reuseport backlog=511;
        listen [::]:80 default_server reuseport backlog=511;
        set $proxy_upstream_name "-";

        listen 442 proxy_protocol default_server reuseport backlog=511 ssl http2;
        listen [::]:442 proxy_protocol  default_server reuseport backlog=511 ssl http2;
        # PEM sha: 21cdeddc99bd37f37685a924de5fe00d1ea91465
        ssl_certificate                         /ingress-controller/ssl/kube-system-star-trysimply-com.pem;
        ssl_certificate_key                     /ingress-controller/ssl/kube-system-star-trysimply-com.pem;

        more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains;";
        location / {
            set $proxy_upstream_name "upstream-default-backend";
            port_in_redirect off;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            # Pass Real IP
            proxy_set_header X-Real-IP              $the_real_ip;

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Real-IP              $the_real_ip;
            proxy_set_header X-Forwarded-For        $the_x_forwarded_for;
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            proxy_set_header X-Original-URI         $request_uri;
            proxy_set_header X-Scheme               $pass_access_scheme;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            proxy_pass http://upstream-default-backend;
        }

        # health checks in cloud providers require the use of port 80
        location /healthz {
            access_log off;
            return 200;
        }

        # this is required to avoid error if nginx is being monitored
        # with an external software (like sysdig)
        location /nginx_status {
            allow 127.0.0.1;
            allow ::1;
            deny all;

            access_log off;
            stub_status on;
        }
    }

    server {
        server_name environment.trysimply.com;
        listen 80;
        listen [::]:80;
        set $proxy_upstream_name "-";

        listen 442 proxy_protocol ssl http2;
        listen [::]:442 proxy_protocol  ssl http2;
        # PEM sha: 21cdeddc99bd37f37685a924de5fe00d1ea91465
        ssl_certificate                         /ingress-controller/ssl/kube-system-star-trysimply-com.pem;
        ssl_certificate_key                     /ingress-controller/ssl/kube-system-star-trysimply-com.pem;

        more_set_headers                        "Strict-Transport-Security: max-age=15724800; includeSubDomains;";
        # enforce ssl on server side
        if ($pass_access_scheme = http) {
            return 301 https://$best_http_host$request_uri;
        }
        location /oauth2 {
            set $proxy_upstream_name "kube-system-oauth2-proxy-4180";
            port_in_redirect off;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            # Pass Real IP
            proxy_set_header X-Real-IP              $the_real_ip;

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Real-IP              $the_real_ip;
            proxy_set_header X-Forwarded-For        $the_x_forwarded_for;
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            proxy_set_header X-Original-URI         $request_uri;
            proxy_set_header X-Scheme               $pass_access_scheme;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            proxy_pass http://kube-system-oauth2-proxy-4180;
        }
        # enforce ssl on server side
        if ($pass_access_scheme = http) {
            return 301 https://$best_http_host$request_uri;
        }

        location = /_external-auth-L2NsdXN0ZXIvZGFzaGJvYXJk {
            internal;
            set $proxy_upstream_name "internal";

            proxy_pass_request_body     off;
            proxy_set_header            Content-Length "";
            proxy_pass_request_headers  on;
            proxy_set_header            Host environment.trysimply.com;
            proxy_ssl_server_name       on;

            set $target https://environment.trysimply.com/oauth2/auth;
            proxy_pass $target;
        }

        location ~* ^/cluster/dashboard\/?(?<baseuri>.*) {
            set $proxy_upstream_name "kube-system-kubernetes-dashboard-80";
            port_in_redirect off;

            # this location requires authentication
            auth_request /_external-auth-L2NsdXN0ZXIvZGFzaGJvYXJk;

            error_page 401 = https://environment.trysimply.com/oauth2/sign_in;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            # Pass Real IP
            proxy_set_header X-Real-IP              $the_real_ip;

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Real-IP              $the_real_ip;
            proxy_set_header X-Forwarded-For        $the_x_forwarded_for;
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            proxy_set_header X-Original-URI         $request_uri;
            proxy_set_header X-Scheme               $pass_access_scheme;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

	rewrite /cluster/dashboard/(.*) /$1 break;
	rewrite /cluster/dashboard / break;
	proxy_pass http://kube-system-kubernetes-dashboard-80;

        }
        location / {
            set $proxy_upstream_name "upstream-default-backend";
            port_in_redirect off;

            client_max_body_size                    "1m";

            proxy_set_header Host                   $best_http_host;

            # Pass the extracted client certificate to the backend

            # Pass Real IP
            proxy_set_header X-Real-IP              $the_real_ip;

            # Allow websocket connections
            proxy_set_header                        Upgrade           $http_upgrade;
            proxy_set_header                        Connection        $connection_upgrade;

            proxy_set_header X-Real-IP              $the_real_ip;
            proxy_set_header X-Forwarded-For        $the_x_forwarded_for;
            proxy_set_header X-Forwarded-Host       $best_http_host;
            proxy_set_header X-Forwarded-Port       $pass_port;
            proxy_set_header X-Forwarded-Proto      $pass_access_scheme;
            proxy_set_header X-Original-URI         $request_uri;
            proxy_set_header X-Scheme               $pass_access_scheme;

            # mitigate HTTPoxy Vulnerability
            # https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
            proxy_set_header Proxy                  "";

            # Custom headers

            proxy_connect_timeout                   5s;
            proxy_send_timeout                      60s;
            proxy_read_timeout                      60s;

            proxy_redirect                          off;
            proxy_buffering                         off;
            proxy_buffer_size                       "4k";
            proxy_buffers                           4 "4k";

            proxy_http_version                      1.1;

            proxy_cookie_domain                     off;
            proxy_cookie_path                       off;

            proxy_pass http://upstream-default-backend;
        }

    }
    # default server, used for NGINX healthcheck and access to nginx stats
    server {
        # Use the port 18080 (random value just to avoid known ports) as default port for nginx.
        # Changing this value requires a change in:
        # https://github.com/kubernetes/contrib/blob/master/ingress/controllers/nginx/nginx/command.go#L104
        listen 18080 default_server reuseport backlog=511;
        listen [::]:18080 default_server reuseport backlog=511;
        set $proxy_upstream_name "-";

        location /healthz {
            access_log off;
            return 200;
        }

        location /nginx_status {
            set $proxy_upstream_name "internal";

            access_log off;
            stub_status on;
        }

        # this location is used to extract nginx metrics
        # using prometheus.
        # TODO: enable extraction for vts module.
        location /internal_nginx_status {
            set $proxy_upstream_name "internal";

            allow 127.0.0.1;
            allow ::1;
            deny all;

            access_log off;
            stub_status on;
        }

        location / {
            set $proxy_upstream_name "upstream-default-backend";
            proxy_pass             http://upstream-default-backend;
        }

    }

    # default server for services without endpoints
    server {
        listen 8181;
        set $proxy_upstream_name "-";

        location / {
            return 503;
        }
    }
}

stream {
    log_format log_stream [$time_local] $protocol $status $bytes_sent $bytes_received $session_time;

    access_log /var/log/nginx/access.log log_stream;

    error_log  /var/log/nginx/error.log;

    # TCP services
    upstream tcp-kube-system-openvpn-18443 {
        server                  10.56.1.112:443;
    }

    server {
        listen                  18443;
        proxy_pass              tcp-kube-system-openvpn-18443;
    }

    # UDP services
}

The text was updated successfully, but these errors were encountered:

weitzj · 2017-06-07T14:33:47Z

I sometimes see the same problem and my guess is the controller does not pick up updated/new ingress rules populated in the meantime. (I think this started happening for me when going from nginx-ingress-controller:0.9.0-beta.5 to nginx-ingress-controller:0.9.0-beta.7)

The only thing working for me was to gradually restart the old nginx-ingress instances. The fresh ones work as expected.

Here is a bash-script, which does these restarts:

#!/bin/bash -
set -o nounset
BASE=$(cd "$(dirname "$0")" && pwd)
pushd "${BASE}"
for i in $(kubectl get pods -n kube-system | grep nginx-ingress-lb | awk '{print $1}')
do
  echo "will kill ${i}"
  kubectl delete "pod/${i}" -n kube-system

  echo "Waiting 30 seconds for new pod to come up before killing next old pod..."
  sleep 30
done

caseylucas · 2017-06-07T14:51:41Z

@weitzj I wonder if this may be related to #768 - especially if a restart fixes the problem.

aledbf · 2017-06-07T15:31:59Z

@weitzj please update the image to quay.io/aledbf/nginx-ingress-controller:0.132 (current master)

troian · 2017-06-07T15:54:18Z

@weitzj restart does not work for my case.
@aledbf does your ingress 0.132 contain something specific to that issue? Anyway I'll try it soon

aledbf · 2017-06-07T15:55:58Z

@troian the fix for 768 and PRs 822, 823 and 824

weitzj · 2017-06-07T15:57:35Z

@aledbf Your image quay.io/aledbf/nginx-ingress-controller:0.132 works for me.

The steps I took:

Use your image in my_nginx_controller.yaml
kubectl apply -f my_nginx_controller.yaml
restart the nginx pods (with my bash-script from above)
Using kubectl describe pod/nginx-ingress-... to see, whether your image is in use (it is by showing git-1ea89a61

Btw.:

The nginx controller runs using the cluster-admin Role for now, since I thought RBAC might be an issue.

troian · 2017-06-07T15:58:10Z

@aledbf thanks

The issue I wonder is why it produces Fake certificate even if --default-ssl-certificate specified in argument and ingress contains only one domain with same certificate chain

weitzj · 2017-06-07T16:00:37Z

@troian I also see these 503 timeouts with the current quay.io/aledbf/nginx-ingress-controller:0.132 - but only if liveness/readiness probes did not succeed.
But I guess this is the intended behaviour, which makes sense to me.

aledbf · 2017-06-07T16:01:51Z

but only if liveness/readiness probes did not succeed.

There is nothing we can do to avoid 503 in that situation

troian · 2017-06-07T16:04:13Z

@weitzj, @aledbf ok, make sense. I'm not familiar with that yet. Any particular reason they might not succeed? Even in 5 minutes after pod start
One of root-cause (presumably) that chrome shows such error if ingress returns Fake Certificate

troian · 2017-06-07T16:51:45Z

Seems image quay.io/aledbf/nginx-ingress-controller:0.132 helps.
Thanks everyone
Resolving

We had intermittent 503 errors which might be connected to kubernetes/ingress-nginx#821 [#148743365] Signed-off-by: Brendan Nolan <bnolan@pivotal.io>

DerSalvador · 2019-06-02T16:05:33Z

I sometimes see the same problem and my guess is the controller does not pick up updated/new ingress rules populated in the meantime. (I think this started happening for me when going from nginx-ingress-controller:0.9.0-beta.5 to nginx-ingress-controller:0.9.0-beta.7)

The only thing working for me was to gradually restart the old nginx-ingress instances. The fresh ones work as expected.

Here is a bash-script, which does these restarts:
#!/bin/bash -
set -o nounset
BASE=$(cd "$(dirname "$0")" && pwd)
pushd "${BASE}"
for i in $(kubectl get pods -n kube-system | grep nginx-ingress-lb | awk '{print $1}')
do
  echo "will kill ${i}"
  kubectl delete "pod/${i}" -n kube-system

  echo "Waiting 30 seconds for new pod to come up before killing next old pod..."
  sleep 30
done

works then for minikube as well with

kubectl get pods -n kube-system --selector="app.kubernetes.io/name=nginx-ingress-controller" -oname

troian changed the title ~~Occasional 503 Service Temporarily Unavailable~~ nginx-ingress: occasional 503 Service Temporarily Unavailable Jun 7, 2017

troian closed this as completed Jun 7, 2017

Jaesang mentioned this issue Aug 18, 2017

Nginx Ingress Controller frequently giving HTTP 503 kubernetes-retired/contrib#1718

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nginx-ingress: occasional 503 Service Temporarily Unavailable #821

nginx-ingress: occasional 503 Service Temporarily Unavailable #821

troian commented Jun 7, 2017 •

edited

Loading

weitzj commented Jun 7, 2017 •

edited

Loading

caseylucas commented Jun 7, 2017

aledbf commented Jun 7, 2017

troian commented Jun 7, 2017 •

edited

Loading

aledbf commented Jun 7, 2017

weitzj commented Jun 7, 2017 •

edited

Loading

troian commented Jun 7, 2017

weitzj commented Jun 7, 2017

aledbf commented Jun 7, 2017

troian commented Jun 7, 2017 •

edited

Loading

troian commented Jun 7, 2017 •

edited

Loading

DerSalvador commented Jun 2, 2019 •

edited

Loading

nginx-ingress: occasional 503 Service Temporarily Unavailable #821

nginx-ingress: occasional 503 Service Temporarily Unavailable #821

Comments

troian commented Jun 7, 2017 • edited Loading

weitzj commented Jun 7, 2017 • edited Loading

caseylucas commented Jun 7, 2017

aledbf commented Jun 7, 2017

troian commented Jun 7, 2017 • edited Loading

aledbf commented Jun 7, 2017

weitzj commented Jun 7, 2017 • edited Loading

troian commented Jun 7, 2017

weitzj commented Jun 7, 2017

aledbf commented Jun 7, 2017

troian commented Jun 7, 2017 • edited Loading

troian commented Jun 7, 2017 • edited Loading

DerSalvador commented Jun 2, 2019 • edited Loading

troian commented Jun 7, 2017 •

edited

Loading

weitzj commented Jun 7, 2017 •

edited

Loading

troian commented Jun 7, 2017 •

edited

Loading

weitzj commented Jun 7, 2017 •

edited

Loading

troian commented Jun 7, 2017 •

edited

Loading

troian commented Jun 7, 2017 •

edited

Loading

DerSalvador commented Jun 2, 2019 •

edited

Loading