api configurations not getting updated when changes are made (local cache not getting invalidated)

[chrisst]

Right now we have 2 kong app instance running, lets call them kong-a and kong-b. When I make a change to an api, the change appears successful in the admin api but doesn't actually apply. More details below:

I am currently using the key-auth plugin configuration to test configuration changes by enabling/disabling the key auth and seeing if I can curl the associated api endpoint. I have gotten the key-auth enabled endpoint to successfully block requests without an apikey and successfully allow requests with the key so I'm confident that it's not a plugin thing. Also I reproduced the issue by changing the raw api configurations (chaning the paths) to rule out if the sticky configuration problems were isolated to plugins.

Repro Steps:

• Api is set up at api_path with key-auth plugin enabled
• curl -X GET http://kong-a/api_path is correctly blocked from accessing the api since no apikey is present
• Patch kong-a:8001/api_path_id/plugins/<key_auth_plugin_id> with enabled = false. Patch succeeds with no errors.
• Watch the message about the update show up in /tmp/payload
  -- {"type":"ENTITY_UPDATED","collection":"plugins","entity":{"api_id":"blah","id":"stuff","name":"key-auth","enabled":true,"created_at":1457565476000,"config":{}}}
• GET kong-a:8001/api_path_id/plugins/<key_auth_plugin_id> to verify that the enabled is actually set to false.
• curl -X GET http://kong-a/api_path is still blocked (incorrectly) from accessing due to lack of apikey.
• Trying other server, just in case: curl -X GET http://kong-b/api_path also blocked (incorrectly)
• sudo restart kong on server: kong-a
• See the member exit, member join messages in /tmp/payload
• curl -X GET http://kong-a/api_path is now successful without the key
• curl -X GET http://kong-b/api_path is still blocked because no key is provided.

During this whole time I was tailing the access, error and serf logs. I saw nothing other then the restarts registered in the error or the serf logs. Normal stuff in the access logs.

[chrisst]

More info:

kong version: 0.7.0 running from deb install
nginx: version that is bundled in with kong
server: Ubuntu 14.04.4
cassandra: 2.2.0 running in 3 machine cluster

[subnetmarco]

@ThorArakis in your setup:

1. What is the output of kong cluster members?
2. The error logs you checked also include /usr/local/kong/logs/error.log?

Also, can you try to manually set +x permissions to the serf_event.sh file inside the working directory?

[chrisst]

sudo kong cluster members
[INFO] Using configuration: /etc/kong/kong.yml
kong-573_0.0.0.0:7946  <ip-redacted>:7946    alive
kong-434_0.0.0.0:7946  <ip-redacted>:7946    alive

1. Yes that is one of the logs I'm checking. Looks like today we've gotten some errors in it, but I think they are unrelated:

2016/03/11 00:23:35 [error] 5818#0: [lua] statsd_logger.lua:62: gauge(): failed to send data to localhost:8125: connection refused, context: ngx.timer, client: 69.164.175.238, server: 0.0.0.0:443
2016/03/11 00:23:35 [error] 5818#0: send() failed (111: Connection refused), context: ngx.timer, client: 69.164.175.238, server: 0.0.0.0:443

Permissions are currently 777 on serf_event.sh but I'll try and update and re-test.

[subnetmarco]

Thanks. This is very weird, because there is a test that specifically checks the invalidations and it works. Also, are the two nodes reachable on port 7946 on both TCP and UDP protocols?

[chrisst]

We ran tcpdump and looks like both are reachable.

[chrisst]

The lua yaml library has a bug in its is_number validation. lubyk/yaml#1

This causes 0.0.0.0:8001 to be misparsed as the number 0

our configuration management system (chef) composes the YAML for kong.yml from chef attributes. IIRC, the YAML spec prefers unquoted strings unless there is something (like trailing spaces) which forces quoting. (I think you see where this is going)

we were unable to find a way to change the YAML dialect, and force this behavior. So, our workaround was to intentionally leave a trailing space on the chef attributes, which would result in the admin API listen address being "0.0.0.0:8001 "

This workaround did great, except for this, which resulted in this from the serf_event .post("http://0.0.0.0:8001 /cluster/events/"

[subnetmarco]

To force the YAML library to load the value properly, try to enclose the value in quotes, like:

cluster_listen: "0.0.0.0:7946"

[chrisst]

We managed to throw some custom wrapper code to make sure that we add the double quotes during provisioning, so we're good to go for now. Thanks!