Httpproxy configured with fallback certificate fails when SNI not included

[moondev]

According to https://projectcontour.io/docs/v1.6.1/httpproxy/ the new fallback certificate feature is designed to present the specified certificate in scenarios where the client does not send SNI.

Fallback Certificate: Contour provides virtual host based routing, so that any TLS request is routed to the appropriate service based on both the server name requested by the TLS client and the HOST header in the HTTP request. As the HOST Header is encrypted during TLS handshake, it can’t be used for virtual host based routing unless the client sends HTTPS requests specifying hostname using the TLS server name, or the request is first decrypted using a default TLS certificate. Some legacy TLS clients do not send the server name, so Envoy does not know how to select the right certificate. A fallback certificate is needed for these clients.

The use-case for not sending SNI is required for some load balancers such as f5.

What steps did you take and what happened:

Deploy tls secrets + contour with the following configmap to enable fallback certificate support

apiVersion: v1
kind: ConfigMap
metadata:
  name: contour
  namespace: projectcontour
data:
  contour.yaml: |
    disablePermitInsecure: true
    tls:
      fallback-certificate:
        name: tls-fallback-instakube-io
        namespace: default
    accesslog-format: envoy

Create tlscertifcatedelegations, services and httpproxy

kubectl apply -f - << EOF
apiVersion: v1
kind: List
items:
  - apiVersion: projectcontour.io/v1
    kind: TLSCertificateDelegation
    metadata:
      name: tls-wildcard-instakube-io
    spec:
      delegations:
        - secretName: tls-wildcard-instakube-io
          targetNamespaces:
            - default
  - apiVersion: projectcontour.io/v1
    kind: TLSCertificateDelegation
    metadata:
      name: tls-fallback-instakube-io
    spec:
      delegations:
        - secretName: tls-fallback-instakube-io
          targetNamespaces:
            - default
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      labels:
        app: kuard
      name: kuard
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: kuard
      template:
        metadata:
          labels:
            app: kuard
        spec:
          containers:
            - image: gcr.io/kuar-demo/kuard-amd64:1
              name: kuard
  - apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: kuard
      name: kuard
    spec:
      ports:
        - port: 80
          protocol: TCP
          targetPort: 8080
      selector:
        app: kuard
      sessionAffinity: None
      type: ClusterIP
  - apiVersion: projectcontour.io/v1
    kind: HTTPProxy
    metadata:
      name: fallback
      namespace: default
    spec:
      virtualhost:
        fqdn: fallback.instakube.io
        tls:
          secretName: tls-wildcard-instakube-io
          enableFallbackCertificate: true
      routes:
        - services:
            - name: kuard
              port: 80
EOF

Create an ngnix proxy deployment and nodeport service to use as a control

kubectl apply -f - << EOF
apiVersion: v1
kind: List
items:
  - apiVersion: v1
    kind: Secret
    type: kubernetes.io/tls
    metadata:
      name: instakube-io-wildcard
    data:
      tls.crt: <redacted>
      tls.key: <redacted>
  - apiVersion: v1
    kind: ConfigMap
    metadata:
      name: nginx
      namespace: default
    data:
      nginx.conf: |
        server {
            listen 443 ssl;
            server_name  nginx-proxy.instakube.io;
            ssl_certificate /etc/nginx/ssl/tls.crt;
            ssl_certificate_key /etc/nginx/ssl/tls.key;
            location / {
                proxy_pass http://kuard.default/;
                error_log /var/log/front_end_errors.log;
            }
        }
      index.html: |
        <h1>welcome to nginx</h1>
  - apiVersion: apps/v1
    kind: Deployment
    metadata:
      creationTimestamp: null
      labels:
        app: nginx
      name: nginx
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: nginx
      template:
        metadata:
          labels:
            app: nginx
        spec:
          volumes:
            - name: certs
              secret:
                secretName: tls-wildcard-instakube-io
            - name: nginx
              configMap:
                name: nginx
            - name: index
              configMap:
                name: nginx
          containers:
            - image: nginx:latest
              volumeMounts:
                - mountPath: /etc/nginx/ssl
                  name: certs
                  readOnly: true
                - mountPath: /etc/nginx/conf.d/nginx.conf
                  name: nginx
                  subPath: nginx.conf
                - mountPath: /html/index.html
                  name: index
                  subPath: index.html
              name: nginx
              resources: {}
  - apiVersion: v1
    kind: Service
    metadata:
      labels:
        app: nginx
      name: nginx-lb
      namespace: default
    spec:
      ports:
        - port: 443
          nodePort: 30555
          protocol: TCP
          targetPort: 443
      selector:
        app: nginx
      sessionAffinity: None
      type: NodePort
EOF
kubectl rollout status nginx

What did you expect to happen:

Once everything is rolled out, example curl tests:

WORKING: curl httpproxy with SNI

> curl -v https://fallback.instakube.io:8843/healthy

*   Trying 127.0.0.1:8843...
* Connected to fallback.instakube.io (127.0.0.1) port 8843 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.instakube.io
*  start date: Jul  6 06:36:35 2020 GMT
*  expire date: Oct  4 06:36:35 2020 GMT
*  subjectAltName: host "fallback.instakube.io" matched cert's "*.instakube.io"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55bca4c1e9f0)
> GET /healthy HTTP/2
> Host: fallback.instakube.io:8843
> user-agent: curl/7.71.1
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 200 
< content-type: text/plain
< date: Wed, 22 Jul 2020 15:37:33 GMT
< content-length: 2
< x-envoy-upstream-service-time: 0
< server: envoy
< 
* Connection #0 to host fallback.instakube.io left intact
ok%                    

WORKING: curl nginx+nodeport with SNI

> curl -v https://nginx-proxy.instakube.io:8855/healthy

*   Trying 127.0.0.1:8855...
* Connected to nginx-proxy.instakube.io (127.0.0.1) port 8855 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.instakube.io
*  start date: Jul  6 06:36:35 2020 GMT
*  expire date: Oct  4 06:36:35 2020 GMT
*  subjectAltName: host "nginx-proxy.instakube.io" matched cert's "*.instakube.io"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> GET /healthy HTTP/1.1
> Host: nginx-proxy.instakube.io:8855
> User-Agent: curl/7.71.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.19.1
< Date: Wed, 22 Jul 2020 15:38:08 GMT
< Content-Type: text/plain
< Content-Length: 2
< Connection: keep-alive
< 
* Connection #0 to host nginx-proxy.instakube.io left intact
ok%    

WORKING: curl nginx+nodeport without sni

> curl -v -k -H "Host:nginx-proxy.instakube.io" https://127.0.0.1:8855/healthy

*   Trying 127.0.0.1:8855...
* Connected to 127.0.0.1 (127.0.0.1) port 8855 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.instakube.io
*  start date: Jul  6 06:36:35 2020 GMT
*  expire date: Oct  4 06:36:35 2020 GMT
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> GET /healthy HTTP/1.1
> Host:nginx-proxy.instakube.io
> User-Agent: curl/7.71.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: nginx/1.19.1
< Date: Wed, 22 Jul 2020 15:38:55 GMT
< Content-Type: text/plain
< Content-Length: 2
< Connection: keep-alive
< 
* Connection #0 to host 127.0.0.1 left intact
ok%  

FAILING: curl httpproxy without SNI

> curl -v -k -H "Host:fallback.instakube.io" https://127.0.0.1:8843/healthy
*   Trying 127.0.0.1:8843...
* Connected to 127.0.0.1 (127.0.0.1) port 8843 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:8843 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:8843

Anything else you would like to add:

Expected behavior is success when not sending SNI and curled via

curl -v -k -H "Host:<VIRTUALHOST>" https://<IP>:<PORT>

Environment:

• Contour version: v1.6.1
• Kubernetes version: v1.18.6
• Kubernetes installer & version: kind

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node:${K8S_V}
- role: worker
  image: kindest/node:${K8S_V}
- role: worker
  image: kindest/node:${K8S_V}
  extraPortMappings:
  - containerPort: 443
    hostPort: 8843
  - containerPort: 30444
    hostPort: 8844
  - containerPort: 30555
    hostPort: 8855

• Cloud provider or hardware configuration: none - local
• OS (e.g. from /etc/os-release):

cat /etc/os-release 
NAME="Ubuntu"
VERSION="20.04 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

[stevesloka]

Can you please provide the envoy config that is in use for this configuration? It's in the Admin page under config_dump. Also please redact any tls secrets that you would not want to be shared publicly.

[youngnick]

Seems like @moondev is saying that the feature doesn't work as intended, so I've asked @jpeach to check this out urgently. That way, once we find out what's going on, we can get a fix in for 1.7.