Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cassandra custom port is replaced by default port at some point #47

Closed
allisthere2love opened this issue Apr 8, 2016 · 2 comments
Closed

Comments

@allisthere2love
Copy link

When I'm using a cassandra custom port (not 9042) and it's a cassandra cluster, the module gets "lost" at some point and tries to use the default port.

I was able to replicate the problem in the following scenario:
1 - 2 hosts (let's call it A and B), each one running 2 Cassandras (let's call it A1, B1, A2 and B2).
2 - One cluster is composed by A1 and B1 using default port, 9042
3 - One cluster is composed by A2 and B2 using custom port, 39044.

In this scenario, when I run a simple lua module to create a keyspace, table, insert some data and then select it, it only performs the first operation in the correct cluster (the one using custom ports). After that, seems it falls back to the default port and then get a "bad credentials" or a "missing column_family" error.
If I'm running a single node cassandra with custom ports there's no error.

To validate the problem, I moved the A2-B2 cluster to hosts that doesn't have Cassandra running in the default port. Then it all worked well.

@thibaultcha
Copy link
Owner

Thanks for investigating this! So, as we discussed privately, the proposed patch for now is:

diff --git a/src/cassandra.lua b/src/cassandra.lua
index de83543..461a4ad 100644
--- a/src/cassandra.lua
+++ b/src/cassandra.lua
@@ -1136,7 +1136,7 @@ function Cassandra.refresh_hosts(options)
     end

     for _, row in ipairs(rows) do
-      local address = options.policies.address_resolution(row["rpc_address"])
+      local address = options.policies.address_resolution(row["rpc_address"], options.protocol_options.default_port)
       log.info("Adding host "..address)
       hosts[address] = {
         unhealthy_at = 0,

used in accordance with a defined default_port in the protocol options. I tested this locally on a 2 nodes cluster listening on 9043, with only one contact_point defined. Without this patch, the driver assumes the other node listens on 9042.

This patch mimics the nodejs Datastax driver's behavior.

Let me know how this patch goes on your end :)

@allisthere2love
Copy link
Author

Hi,
I was able to sucessfully connect to the cluster after I've applied the suggested patch and defined the variable in my test module.

Now I'm checking how to pass this config in Kong.

thibaultcha added a commit to Kong/kong that referenced this issue Apr 11, 2016
Clusters with nodes not running on the default `9042` port couldn't be
properly reached by Kong. This bumps the driver (including the fix), and
adds a `port` options to the configuration. Thanks @allisthere2love for
the investigation.

This issue was never encountered before and my guess is because such
clusters added their contact_points including the port (`x.x.x.x:9043`),
and the driver could connect to it, but not to the other nodes, since it
was trying `9042` on those. So it simply considered the other nodes as
DOWN, and thus still works (if one doesn't check the logs and those lags
are not displaying warnings, it seems like nothing is going wrong).
However @allisthere2love had nodes from another C* cluster listening on
`9042`, leading to inconsistencies and thus, actual errors that allowed
us to track it down.

As per other datastax drivers, all nodes of a cluster must listen on the
same port.

Fix #1139
See thibaultcha/lua-cassandra#47
thibaultcha added a commit to Kong/kong that referenced this issue Apr 11, 2016
Clusters with nodes not running on the default `9042` port couldn't be
properly reached by Kong. This bumps the driver (including the fix), and
adds a `port` options to the configuration. Thanks @allisthere2love for
the investigation.

This issue was never encountered before and my guess is because such
clusters added their contact_points including the port (`x.x.x.x:9043`),
and the driver could connect to it, but not to the other nodes, since it
was trying `9042` on those. So it simply considered the other nodes as
DOWN, and thus still works (if one doesn't check the logs and those lags
are not displaying warnings, it seems like nothing is going wrong).
However @allisthere2love had nodes from another C* cluster listening on
`9042`, leading to inconsistencies and thus, actual errors that allowed
us to track it down.

As per other datastax drivers, all nodes of a cluster must listen on the
same port.

Fix #1139
See thibaultcha/lua-cassandra#47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants