Different DNS resolution between startup and actual runtime #1104
Comments
|
Without investigation, the reason that seems obvious at first sight would be dnsmasq. Kong manages dnsmasq along Nginx. In the CLI, dnsmasq is not started, but once Kong started, dnsmasq is used as the resolver. From there, I suggest different approaches to try to fix this:
|
|
To better investigate this, you can use the following command to check how the address is being resolved by a DNS server: $ nslookup -port={dns_server_port} {address_to_resolve} {dns_server_address}like $ nslookup -port=53 google.com 8.8.8.8Now, you can start kong/dnsmasq and try to see what's the output of: $ nslookup -port=8053 cassandra-01 127.0.0.1Port $ nslookup -port=8053 cassandra-01 127.0.0.1 # Using dnsmasq
$ nslookup cassandra-01 # Using system settingsIt seems like in your environment this is not the case, so we can see the output of these commands and take it from there. |
|
Ok, so I'm running following commands from within the Kong Docker container, and surprisingly they both show the correct address: But Kong still doesn't resolve it correctly. |
|
So two notes:
Can you see if in the |
|
Sure. After making a request to See this gist for the full |
|
What happens if you make a request to |
|
I receive an error message, and the log shows: |
|
I am seeing the exact same behavior, also running mashape/kong:0.7.0 From inside the container, I see Kong starts up fine, correctly finding cassandra, but then the error log is full of similar errors to @noamelf : |
I should note that I am trying to do the second thing, with the following config (full kong.yml at https://gist.github.com/rileylark/a22e2146b85dec3faa9a7c82162b3b29 ) With these settings, kong can find |
|
@rileylark ark in your case what's the output of: $ nslookup -port=8053 cassandra 127.0.0.1and is it the same as: ? |
|
@thefosk Ah, of course I should have thought of that before. Sorry! So my case is different after all: So it feels like the kong.yml file is being respected at some levels of kong, but then deep down somewhere kong is still trying to resolve names at 127.0.0.1:8053 . So, maybe to progress at this point, I should focus on configuring the local dnsmasq instead of trying to disable it and provide my own nameserver? |
|
Ok, I tried NOT providing a custom nameserver, and instead letting dnsmasq do the trick, and now I get results exactly like @noamelf. kong.yml is exactly copy/pasted from exact copy/paste from https://github.com/Mashape/docker-kong/blob/master/config.docker/kong.yml : Docker container output: |
|
This is weird and I will need more time to replicate the issue and trying to figure it out. |
|
@thefosk Yesterday I got a vanilla nginx setup working in the exact same kubernetes cluster, which I mention just to confirm that nginx can work in this environment w/o any extra config. If you have a Kubernetes cluster running already (easy to set up on Google Container Engine), you can get an exact repro of my situation from my yaml files at https://github.com/peardeck/kubernetes-boilerplate/tree/more-kong/kong First, I'm happy to help however I can - let me know if you can think of ways for me to provide more info or debugging context! |
|
I got the same problem when running Kong-with-postgres-backend on Kubernetes. but i still got error: Version:Kong: v0.8.0 |
Set the database host if passed in the environment. This is useful for deploying this container out to Kubernetes where there is some issue between dnsmasq and the Kubernetes resolver when trying to access a service. Kong/kong#1104
|
I ran into this problem as well trying to deploy Kong's official docker image onto a Kubernetes cluster. I work around this issue by using the fully-qualified Kubernetes service name in And I have modified The Automated Build image with this change is at: https://hub.docker.com/r/zymbit/kong/ Please let me know if this is useful and I will open a PR. Thanks! |
|
By the way, there is an issue specific to Ruby that is exposed when Kubernetes sets Perhaps this is going on in the bowels of dnsmasq / lua? |
|
Thx, rca. I can confirm that your changes fix the issue we were having on kubernetes |
|
Encountered this issue in my Kubernetes clusters today as well. After running in circles chasing dnsmasq, I finally gave up on trying to use the shorter DNS entry for my service Nothing else I tried worked including pointing the It's worth mentioning that Kong was running in the same namespace as the service I was trying to proxy to as well (default). |
|
@thenayr you can try it |
|
I'm running another Nginx proxy (with open-resty / Lua modules etc) and have 0 issues with Nginx DNS resolution inside Kubernets. All of my |
thibaultcha
added
the
task/needs-investigation
|
Just came across this while looking into Kong as a possible solution for something. This sounds oddly familiar to a problem we had with running an open-resty solution for an api-proxy in a Kubernetes cluster. We ended solving it with having the nginx.conf be a jinja2 template that was parsed at start time (ugly, but solved other issues as well) that has : in the http block and the python start script that starts things up did a : to set the nameserver in the config (along with a bunch of other environment specific vars) then used the output as the live nginx.conf for runtime. This worked fine for doing the short name looking http://appname work fine where the open-resty api-proxy would run in different namespaces and talk to the various microservice pods in the same namespace via the short name. Not sure it that will help at all, but might be a start path to head down. |
|
In 0.10 we'll have our internal dns resolution, see #1587, used both from the cli and in nginx. So the inconsistencies should be resolved. That branch is just up for review now and all tests pass. So if some of you are willing to give it a try, please do. NOTE: see this code, the config read from Really looking for feedback on this. PS. to run the branch you'd need to manually install |
|
@Tieske, this sounds great. Any chance you can roll this up into a test image that I can |
|
@rca I just pushed it at |
|
Anyone else having an issue running the kong:dns branch? |
|
It's We use |
|
Am I missing something, this is the command I run: |
|
I am experiencing that same issue running it with docker-compose locally. |
|
Same issue, running mashape/kong:dns in kubernetes: |
|
Pulling and running mashape/kong seems to have solved this problem. |
|
... almost. Now I've got no problem connecting to the postgres database, but now upstream forwarding doesn't work. My client: In the Kong logs: But dnsmasq can see it -- I suspect this one's harder to get around -- does nginx, itself, need to be told to use dnsmaq? |
|
We have updated how we resolve DNS hostnames in > 0.10.x (and removed the dnsmasq dependency) - Can you try to replicate with 0.10.x and see if the problem has been fixed? |
|
Any updates here, following the changes in 0.10.x? |
|
We resolve DNS externally and only ever pass IPs to Kong, because of the
various DNS issues we encountered previously. I'm not in a great position
to test the fix right now, but we are not blocked by the issue that I
reported back then.
…On Sun, Jun 4, 2017 at 10:31 PM, Robert ***@***.***> wrote:
Any updates here, following the changes in 0.10.x?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#1104 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/ABLW3I-F8utSGMgpIL9siCeynlXQWMp2ks5sA3amgaJpZM4H51Ue>
.
|
|
@derrley understood - if you want to give it another try please let us know the outcome. Closing this issue for now, but happy to re-open it if Kong >= 0.10 still doesn't work. |
Problem
It seems that kong startup/migration is using different DNS resolution the actual run time (or Lua DNS resolution, I'm not sure what is doing the resolution). Example: I have an AWS internal hosted zone DNS by the name of
qa2. If Kong config doesn't have the hosted zone suffix, e.g.:Kong will start without any errors but when asked to answer a simple request, will fail:
The error log will show that it can't find the DB instance:
Nevertheless, if I use a "proper" DNS name, everything works:
Like I mentioned before, it seems that the different part of kong are using different DNS resolution methods, that produce different behaviour. This is fairly confusing since you expect kong to fail from the beginning if the Cassandra DNS is bad.
Versions
Kong on docker: v0.7.0
Cassandra: v2.2.4
The text was updated successfully, but these errors were encountered: