Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

proxy to externalName service returns 503 #2873

Closed
stevesloka opened this issue Sep 2, 2020 · 0 comments · Fixed by #2894
Closed

proxy to externalName service returns 503 #2873

stevesloka opened this issue Sep 2, 2020 · 0 comments · Fixed by #2894
Assignees
@stevesloka
Labels
lifecycle/accepted Denotes an issue that has been triaged and determined to be valid.

Comments

@stevesloka
Copy link
Member

When configuring a route to an externalName service, requests don't get fulfilled properly, but return 503 errors. My first thought is that Envoy is not proxying when its DNS lookup returns an ipv6 address for the external resource.

Below are some examples which recreate the issue. One thing to note is if you use my domain (e.g. stevesloka.com), it results in an ipv4 address and the proxy works just fine. My cluster is running in DigitalOcean proxying to a Node directly fronted with Contour.

HTTPProxy:

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  name: my-ingress
spec:
  virtualhost:
    fqdn: baremetal.slokalabs.io
    # tls:
    #   secretName: steveslokadev-tls
  routes:
    - conditions:
        - prefix: /
      services:
        - name: external-svc
          port: 443
          protocol: tls
      requestHeadersPolicy:
        set:
        - name: Host
          value: google.com
    - conditions:
        - prefix: /potato
      services:
        - name: rootapp
          port: 80

Service:

apiVersion: v1
kind: Service
metadata:
  name: external-svc
  annotations:
    projectcontour.io/upstream-protocol.tls: "external-svc,443"
spec:
  type: ExternalName
  externalName: google.com
  ports:
    - name: external-svc
      port: 443
      targetPort: 443

Logs from Envoy:

[2020-09-02 20:19:12.544][1][debug][upstream] [source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address [2607:f8b0:4004:807::200e]:443
[2020-09-02 20:19:12.545][1][debug][upstream] [source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for google.com, refresh rate 5000 ms
[2020-09-02 20:19:12.593][1][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:100] starting async DNS resolution for 127.0.0.1
[2020-09-02 20:19:12.593][1][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:108] async DNS resolution complete for 127.0.0.1
[2020-09-02 20:19:12.593][1][debug][upstream] [source/common/upstream/logical_dns_cluster.cc:153] DNS refresh rate reset for 127.0.0.1, refresh rate 5000 ms
[2020-09-02 20:19:12.653][1][debug][main] [source/server/server.cc:189] flushing stats
[2020-09-02 20:19:13.225][1][debug][upstream] [source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address 10.100.207.214:8001
[2020-09-02 20:19:13.225][1][debug][upstream] [source/common/upstream/strict_dns_cluster.cc:167] DNS refresh rate reset for contour, refresh rate 5000 ms
[2020-09-02 20:19:16.055][17][debug][conn_handler] [source/server/connection_handler_impl.cc:422] [C19] new connection
[2020-09-02 20:19:16.055][12][debug][conn_handler] [source/server/connection_handler_impl.cc:422] [C20] new connection
[2020-09-02 20:19:16.055][17][debug][http] [source/common/http/conn_manager_impl.cc:261] [C19] new stream
[2020-09-02 20:19:16.056][17][debug][http] [source/common/http/conn_manager_impl.cc:808] [C19][S8889813871901120571] request headers complete (end_stream=true):
':authority', 'baremetal.slokalabs.io'
':path', '/'
':method', 'GET'
'connection', 'keep-alive'
'cache-control', 'max-age=0'
'upgrade-insecure-requests', '1'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36'
'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
'accept-encoding', 'gzip, deflate'
'accept-language', 'en-US,en;q=0.9'
'cookie', '_ga=GA1.2.702597985.1599075532; _gid=GA1.2.763115516.1599075532'

[2020-09-02 20:19:16.056][17][debug][http] [source/common/http/conn_manager_impl.cc:1377] [C19][S8889813871901120571] request end stream
[2020-09-02 20:19:16.056][17][debug][router] [source/common/router/router.cc:426] [C19][S8889813871901120571] cluster 'projectcontour-roots/external-svc/443/da39a3ee5e' match for URL '/'
[2020-09-02 20:19:16.056][17][debug][router] [source/common/router/router.cc:583] [C19][S8889813871901120571] router decoding headers:
':authority', 'google.com'
':path', '/'
':method', 'GET'
':scheme', 'https'
'cache-control', 'max-age=0'
'upgrade-insecure-requests', '1'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36'
'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
'accept-encoding', 'gzip, deflate'
'accept-language', 'en-US,en;q=0.9'
'cookie', '_ga=GA1.2.702597985.1599075532; _gid=GA1.2.763115516.1599075532'
'x-forwarded-for', '10.52.131.186'
'x-forwarded-proto', 'http'
'x-envoy-internal', 'true'
'x-request-id', '87cf37cf-e332-4c21-8c4a-e42967919e7f'
'x-envoy-expected-rq-timeout-ms', '15000'
'x-request-start', 't=1599077956.056'

[2020-09-02 20:19:16.056][17][debug][pool] [source/common/http/conn_pool_base.cc:71] queueing request due to no available connections
[2020-09-02 20:19:16.056][17][debug][pool] [source/common/conn_pool/conn_pool_base.cc:53] creating a new connection
[2020-09-02 20:19:16.056][17][debug][client] [source/common/http/codec_client.cc:35] [C21] connecting
[2020-09-02 20:19:16.056][17][debug][connection] [source/common/network/connection_impl.cc:753] [C21] connecting to [2607:f8b0:4004:807::200e]:443
[2020-09-02 20:19:16.056][17][debug][connection] [source/common/network/connection_impl.cc:773] [C21] immediate connection error: 101
[2020-09-02 20:19:16.056][17][debug][connection] [source/common/network/connection_impl.cc:517] [C21] raising immediate error
[2020-09-02 20:19:16.056][17][debug][connection] [source/common/network/connection_impl.cc:208] [C21] closing socket: 0
[2020-09-02 20:19:16.056][17][debug][client] [source/common/http/codec_client.cc:92] [C21] disconnect. resetting 0 pending requests
[2020-09-02 20:19:16.056][17][debug][pool] [source/common/conn_pool/conn_pool_base.cc:255] [C21] client disconnected, failure reason: 
[2020-09-02 20:19:16.056][17][debug][router] [source/common/router/router.cc:1022] [C19][S8889813871901120571] upstream reset: reset reason connection failure
[2020-09-02 20:19:16.056][17][debug][http] [source/common/http/conn_manager_impl.cc:1520] [C19][S8889813871901120571] Sending local reply with details upstream_reset_before_response_started{connection failure}
[2020-09-02 20:19:16.056][17][debug][http] [source/common/http/conn_manager_impl.cc:1777] [C19][S8889813871901120571] encoding headers via codec (end_stream=false):
':status', '503'
'content-type', 'text/plain'
'vary', 'Accept-Encoding'
'content-encoding', 'gzip'
'date', 'Wed, 02 Sep 2020 20:19:16 GMT'
'server', 'envoy'

[2020-09-02 20:19:16.218][17][debug][http] [source/common/http/conn_manager_impl.cc:261] [C19] new stream
[2020-09-02 20:19:16.218][17][debug][http] [source/common/http/conn_manager_impl.cc:808] [C19][S10594451152967010302] request headers complete (end_stream=true):
':authority', 'baremetal.slokalabs.io'
':path', '/favicon.ico'
':method', 'GET'
'connection', 'keep-alive'
'pragma', 'no-cache'
'cache-control', 'no-cache'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36'
'accept', 'image/webp,image/apng,image/*,*/*;q=0.8'
'referer', 'http://baremetal.slokalabs.io/'
'accept-encoding', 'gzip, deflate'
'accept-language', 'en-US,en;q=0.9'
'cookie', '_ga=GA1.2.702597985.1599075532; _gid=GA1.2.763115516.1599075532'

[2020-09-02 20:19:16.218][17][debug][http] [source/common/http/conn_manager_impl.cc:1377] [C19][S10594451152967010302] request end stream
[2020-09-02 20:19:16.218][17][debug][router] [source/common/router/router.cc:426] [C19][S10594451152967010302] cluster 'projectcontour-roots/external-svc/443/da39a3ee5e' match for URL '/favicon.ico'
[2020-09-02 20:19:16.218][17][debug][router] [source/common/router/router.cc:583] [C19][S10594451152967010302] router decoding headers:
':authority', 'google.com'
':path', '/favicon.ico'
':method', 'GET'
':scheme', 'https'
'pragma', 'no-cache'
'cache-control', 'no-cache'
'user-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135 Safari/537.36'
'accept', 'image/webp,image/apng,image/*,*/*;q=0.8'
'referer', 'http://baremetal.slokalabs.io/'
'accept-encoding', 'gzip, deflate'
'accept-language', 'en-US,en;q=0.9'
'cookie', '_ga=GA1.2.702597985.1599075532; _gid=GA1.2.763115516.1599075532'
'x-forwarded-for', '10.52.131.186'
'x-forwarded-proto', 'http'
'x-envoy-internal', 'true'
'x-request-id', '77a6dd34-cd0b-43e3-9174-70d9842e53b2'
'x-envoy-expected-rq-timeout-ms', '15000'
'x-request-start', 't=1599077956.218'

[2020-09-02 20:19:16.218][17][debug][pool] [source/common/http/conn_pool_base.cc:71] queueing request due to no available connections
[2020-09-02 20:19:16.218][17][debug][pool] [source/common/conn_pool/conn_pool_base.cc:53] creating a new connection
[2020-09-02 20:19:16.218][17][debug][client] [source/common/http/codec_client.cc:35] [C22] connecting
[2020-09-02 20:19:16.218][17][debug][connection] [source/common/network/connection_impl.cc:753] [C22] connecting to [2607:f8b0:4004:807::200e]:443
[2020-09-02 20:19:16.218][17][debug][connection] [source/common/network/connection_impl.cc:773] [C22] immediate connection error: 101
[2020-09-02 20:19:16.219][17][debug][connection] [source/common/network/connection_impl.cc:517] [C22] raising immediate error
[2020-09-02 20:19:16.219][17][debug][connection] [source/common/network/connection_impl.cc:208] [C22] closing socket: 0
[2020-09-02 20:19:16.219][17][debug][client] [source/common/http/codec_client.cc:92] [C22] disconnect. resetting 0 pending requests
[2020-09-02 20:19:16.219][17][debug][pool] [source/common/conn_pool/conn_pool_base.cc:255] [C22] client disconnected, failure reason: 
[2020-09-02 20:19:16.219][17][debug][router] [source/common/router/router.cc:1022] [C19][S10594451152967010302] upstream reset: reset reason connection failure
[2020-09-02 20:19:16.219][17][debug][http] [source/common/http/conn_manager_impl.cc:1520] [C19][S10594451152967010302] Sending local reply with details upstream_reset_before_response_started{connection failure}
[2020-09-02 20:19:16.219][17][debug][http] [source/common/http/conn_manager_impl.cc:1777] [C19][S10594451152967010302] encoding headers via codec (end_stream=false):
':status', '503'
'content-type', 'text/plain'
'vary', 'Accept-Encoding'
'content-encoding', 'gzip'
'date', 'Wed, 02 Sep 2020 20:19:16 GMT'
'server', 'envoy'

[2020-09-02 20:19:17.545][1][debug][upstream] [source/common/upstream/upstream_impl.cc:278] transport socket match, socket default selected for host with address [2607:f8b0:4004:807::200e]:443

//cc @brunopadz
@youngnick youngnick added the lifecycle/investigating Denotes an issue that is assigned and is being actively investigated by the issue owner. label Sep 3, 2020
stevesloka added a commit to stevesloka/contour that referenced this issue Sep 9, 2020
@stevesloka
internal/envoy: Set the dns lookup family on externalName type clusters
4377594 
Adds the DnsLookupFamily: V4_ONLY to any cluster that is referenced via an
externalName type cluster. This ensures that lookups to external resources
are resolved correctly.

Fixes projectcontour#2873

Signed-off-by: Steve Sloka <slokas@vmware.com>
@stevesloka stevesloka mentioned this issue Sep 9, 2020
Merged
stevesloka added a commit to stevesloka/contour that referenced this issue Sep 16, 2020
@stevesloka
internal/envoy: Set the dns lookup family on externalName type clusters
8c95549 
Adds a config option for DnsLookupFamily allowing users to define what
dns lookup family is used on any cluster that is referenced via an
externalName type cluster. This ensures that lookups to external resources
are resolved correctly.

Fixes projectcontour#2873

Signed-off-by: Steve Sloka <slokas@vmware.com>
stevesloka added a commit to stevesloka/contour that referenced this issue Sep 16, 2020
@stevesloka
internal/envoy: Set the dns lookup family on externalName type clusters
c4aa289 
Adds a config option for DnsLookupFamily allowing users to define what
dns lookup family is used on any cluster that is referenced via an
externalName type cluster. This ensures that lookups to external resources
are resolved correctly.

Fixes projectcontour#2873

Signed-off-by: Steve Sloka <slokas@vmware.com>
stevesloka added a commit to stevesloka/contour that referenced this issue Sep 17, 2020
@stevesloka
internal/envoy: Set the dns lookup family on externalName type clusters
a9b8e70 
Adds a config option for DnsLookupFamily allowing users to define what
dns lookup family is used on any cluster that is referenced via an
externalName type cluster. This ensures that lookups to external resources
are resolved correctly.

Fixes projectcontour#2873

Signed-off-by: Steve Sloka <slokas@vmware.com>
stevesloka added a commit to stevesloka/contour that referenced this issue Sep 17, 2020
@stevesloka
internal/envoy: Set the dns lookup family on externalName type clusters
6cadac7 
Adds a config option for DnsLookupFamily allowing users to define what
dns lookup family is used on any cluster that is referenced via an
externalName type cluster. This ensures that lookups to external resources
are resolved correctly.

Fixes projectcontour#2873

Signed-off-by: Steve Sloka <slokas@vmware.com>
stevesloka added a commit to stevesloka/contour that referenced this issue Sep 18, 2020
@stevesloka
internal/envoy: Set the dns lookup family on externalName type clusters
e2730d0 
Adds a config option for DnsLookupFamily allowing users to define what
dns lookup family is used on any cluster that is referenced via an
externalName type cluster. This ensures that lookups to external resources
are resolved correctly.

Fixes projectcontour#2873

Signed-off-by: Steve Sloka <slokas@vmware.com>
@stevesloka stevesloka added lifecycle/accepted Denotes an issue that has been triaged and determined to be valid. and removed lifecycle/investigating Denotes an issue that is assigned and is being actively investigated by the issue owner. labels Sep 18, 2020
stevesloka added a commit to stevesloka/contour that referenced this issue Sep 18, 2020
@stevesloka
internal/envoy: Set the dns lookup family on externalName type clusters
efa1a86 
Adds a config option for DnsLookupFamily allowing users to define what
dns lookup family is used on any cluster that is referenced via an
externalName type cluster. This ensures that lookups to external resources
are resolved correctly.

Fixes projectcontour#2873

Signed-off-by: Steve Sloka <slokas@vmware.com>
stevesloka added a commit to stevesloka/contour that referenced this issue Sep 23, 2020
@stevesloka
internal/envoy: Set the dns lookup family on externalName type clusters
35e524b 
Adds a config option for DnsLookupFamily allowing users to define what
dns lookup family is used on any cluster that is referenced via an
externalName type cluster. This ensures that lookups to external resources
are resolved correctly.

Fixes projectcontour#2873

Signed-off-by: Steve Sloka <slokas@vmware.com>
stevesloka added a commit that referenced this issue Sep 23, 2020
@stevesloka
internal/envoy: Set the dns lookup family on externalName type cluste…
12ace66 
…rs (#2894)

Adds a config option for DnsLookupFamily allowing users to define what
dns lookup family is used on any cluster that is referenced via an
externalName type cluster. This ensures that lookups to external resources
are resolved correctly.

Fixes #2873

Signed-off-by: Steve Sloka <slokas@vmware.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stevesloka stevesloka

Labels
lifecycle/accepted Denotes an issue that has been triaged and determined to be valid.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

internal/envoy: Set the dns lookup family on externalName type clusters stevesloka/contour
2 participants
@stevesloka @youngnick