An invalid response was received from the upstream server

[PierrickLozach]

Summary

Testing kong with docker in kubernetes.
Here is the .yaml file I used to create the replication controller:

    apiVersion: v1
    kind: ReplicationController
    metadata:
      labels:
        name: kong
      name: kong
    spec:
      replicas: 1
      selector:
        name: kong
      template:
        metadata:
          labels:
            name: kong
        spec:
          containers:
            - image: mashape/kong
              name: kong
              env:
                - name: KONG_DATABASE
                  value: cassandra
                - name: KONG_CASSANDRA_CONTACT_POINTS
                  value: cassandra
                - name: KONG_PG_HOST
                  value: cassandra
                - name: KONG_LOG_LEVEL
                  value: info
              ports:
                - containerPort: 8000
                  name: proxy
                - containerPort: 8443
                  name: proxyssl
                - containerPort: 8001
                  name: adminapi
                - containerPort: 7946
                  name: cluster
                - containerPort: 7946
                  name: clusterudp
                  protocol: UDP

I logged into my kong instance and tested my own api, hosted in a different pod:

curl -X GET http://ping:3000/api/v1/ping

which returned the appropriate response:

{"message":"No host specified","error":{"status":400}}

This is an expected response since I did not pass the required parameters.

So, I used the following in Kong to redirect to my API:

curl -d 'name=ping&request_path=/api/v1/ping&upstream_url=http://ping:3000/' http://localhost:8001/apis/

which returned:

{"upstream_url":"http:\/\/ping:3000\/","strip_request_path":false,"request_path":"\/api\/v1\/ping","id":"4eeb968e-683c-4d9e-9c66-26ab4cba124e","created_at":1473174056000,"preserve_host":false,"name":"ping"}

but a GET request (curl -i -X GET --url http://localhost:8000/api/v1/ping) returns this:

HTTP/1.1 502 Bad Gateway
Date: Tue, 06 Sep 2016 15:11:07 GMT
Content-Type: text/plain; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Server: kong/0.9.0

An invalid response was received from the upstream server

/logs/error.log shows this:

2016/09/06 15:11:07 [error] 125#0: *766 ping could not be resolved (2: Server failure), client: 127.0.0.1, server: kong, request: "GET /api/v1/ping HTTP/1.1", host: "localhost:8000"

ping ping returns this:

PING ping.default.svc.cluster.local (10.0.108.177) 56(84) bytes of data.
--- ping.default.svc.cluster.local ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3024ms

How is Kong resolving my host (ping)? As you saw above, curl manages to call my API without any problem using curl -X GET http://ping:3000/api/v1/ping. but Kong can't.

Additional Details & Logs

• Kong version: 0.9.0
• Operating System: Debian 3.16.36-1+deb8u1 (2016-09-03)
• Kong Health:

serf........running
dnsmasq.....running
nginx.......running

Kong is healthy at /usr/local/kong

[Tieske]

Kong uses dnsmasq to resolve the host names. You can configure it in the Kong configuration, or disable it and use your own.

[PierrickLozach]

Thanks @Tieske. Still not working though.

I have disabled dnsmaq in my .conf file (/etc/kong.conf):

dnsmasq = off                    # Toggles if Kong should start/stop dnsmasq,
                                 # which can be used as the Nginx DNS resolver.
                                 # Using dnsmasq allows Nginx to resolve
                                 # domains defined in /etc/hosts.
                                 # dnsmasq must be installed and available in
                                 # your $PATH.

#dnsmasq_port = 8053             # The port on which dnsmasq should listen to
                                 # for queries.

dns_resolver = 10.0.0.10         # Configure a name server to be used by Nginx.
                                 # Only valid when `dnsmasq` is disabled. 

and used the one configured in my machine:

[root@kong-jh19q etc]# cat /etc/resolv.conf
search default.svc.cluster.local svc.cluster.local cluster.local eu-central-1.compute.internal
nameserver 10.0.0.10
options ndots:5

I then reloaded kong and ran kong check

[root@kong-jh19q etc]# kong reload
Kong reloaded
[root@kong-jh19q etc]# kong check
configuration at /etc/kong.conf is valid

Finally, I re-ran curl to check if it worked but I got the same error:

[root@kong-jh19q etc]# curl http://localhost:8000/api/v1/ping
An invalid response was received from the upstream server

Extract from error.log:

2016/09/06 18:17:16 [error] 10308#0: *1148 ping could not be resolved (2: Server failure), client: 127.0.0.1, server: kong, request: "GET /api/v1/ping HTTP/1.1", host: "localhost:8000"

[gavinzhou]

@PierrickI3
maybe nginx resolver is bug
kubernetes/kubernetes#26309

I run go-dnsmasq in the kong pod ,working

containers:
      ### https://github.com/kubernetes/kubernetes/issues/26309
      ### use go-dnsmasq in pod
      - name: go-dnsmasq
        image: "janeczku/go-dnsmasq:release-1.0.5"
        args:
          - --listen
          - "127.0.0.1:53"
          - --default-resolver
          - --append-search-domains
          - --hostsfile=/etc/hosts

[PierrickLozach]

@gavinzhou That worked. Thanks!

[zliang-min]

Although the issue has closed, I'd like to thank @gavinzhou, your solution solved the same issue I just encountered.

[rikatz]

Actually, we've had the same issue here. The solution was to call the 'FQDN' of the service, instead of only 'ping'. Kong/NGINX doesn't respect the search path of the resolver.

So, calling ping.namespace.svc.cluster.local (assuming ping is the name of the service, namespace is your namespace, svc.cluster.local is the configured DNS / domain name of your kubernetes cluster) it did worked fine.