Using kube-dns doesn't resolve correctly.

[jimmiebtlr]

Setup

With the env variables

        env:
          - name: KONG_DNS_RESOLVER
            value: 10.3.240.10
          - name: KONG_DNSMASQ
            value: "off"
         ...

Note 10.3.240.10 is the result of

kubectl get svc kube-dns --namespace=kube-system | grep kube-dns | awk '{print $2}'

Expected

    {
         "preserve_host" : false,
         "upstream_url" : "http://console-graphql-service/graphiql",
         "created_at" : 1481226046000,
         "strip_request_path" : true,
         "name" : "graphiql",
         "id" : "b3369902-e18b-4eb6-99d7-d3faef64a1c9",
         "request_path" : "/graphiql"
      }

Should resolve to the console-graphql-service.

Actual

Kong Error

An invalid response was received from the upstream server.

Note, substituting console-graphql-service in the api config with the service ip does return the expected result.

[shashiranjan84]

strange, I tried testing locally and it just worked fine for me

I am using https://github.com/TheNewNormal/kube-cluster-osx to create local kubernetes managed cluster.

Can you try testing with another api

curl -X POST localhost:8001/apis -d "name=mockbin" -d "request_host=mockbin.com" -d "upstream_url=http://www.mockbin.com"

[shashiranjan84]

@jimmiebtlr you might have to use fqdn for service.

[pamiel]

Hi all,

Like @jimmiebtlr and most probably many other people (looking to the number of issues opened around this topic in various GitHub repos), I’m facing the same issue… but at 2 different locations:

• For the access from Kong to Cassandra: usage of host name was generating random connectivity issues, but usage of FQDN “looks” to solve the issue
• For the access to the upstream servers… but this time, using FQDN has not solved anything :(

I‘ve also experimented the on-boarding of go-dnsmasq on the same Pod as recommended by @gavinzhou on another thread… but without success.

Indeed there are dozens of issues related to the connectivity or the DNS topic, either here or in the Kong GitHub, referring also to potential issues in nginx or in Kubenetes (and while I’m typing this comment, another one just pop up on the Kong Github…):

• On Kong: search directive in resolv.conf is ignored. kong#2201, Cannot access Kong admin port 8001 after kong started  kong#2207, Different DNS resolution between startup and actual runtime kong#1104, An invalid response was received from the upstream server kong#1601, and the one just coming: Help needed: Cassandra hosts not found, probably just a misconfiguration, Kong 0.10.0 kong#2239
• On Kubernetes: Using nginx 1.9 DNS resolver output error periodically kubernetes/kubernetes#26309
• On Kong distrib for Kubernetes: Crash on kong >v0.9.1  #3 and Using kube-dns doesn't resolve correctly. #6
• On other: Support search and ndots configuration options lua-resty-dns-client#6

At the end, frankly, I don’t know if anyone really understood what the root cause is, and it is really difficult to claim that Kong is working on Kubernetes for production purpose!
I fear that solutions with FQDN are temporary solution that will not work with a coming update or change in the system.

At the end, I’m then wondering whether someone has a concrete analysis of the problem, identification of the exact issue, identification of the component that is the root cause of it, and a potential workaround or solution to the issue?