Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kong v0.11.0 Error during migrations on an already migrated cassandra database #2975

Closed
brockmiller opened this issue Oct 19, 2017 · 6 comments
Closed

Kong v0.11.0 Error during migrations on an already migrated cassandra database #2975

brockmiller opened this issue Oct 19, 2017 · 6 comments
Labels
pending author feedback Waiting for the issue author to get back to a maintainer with findings, more details, etc...

Comments

@brockmiller
Copy link

brockmiller commented Oct 19, 2017 •
edited
Loading

Summary

Executing kong migrations up on an already migrated and upgraded kong cluster is causing a migration error: Error: /usr/local/share/lua/5.1/kong/cmd/migrations.lua:34: [cassandra error] Error during migration 2016-09-05-212515_retries_step_1: [Invalid] Invalid column name retries because it conflicts with an existing column.

We'd expect this command to perform a no-op in this case as migrations are already done and tables / columns created. This appears to be a pretty bad bug as this will prevent us from upgrading kong beyond v0.11.0.

We successfully upgraded our v0.9.8 kong cluster to v0.11.0 on 10/6/17 following the upgrade guide. The v0.11.0 migrations were successfully executed and then the new nodes brought online. Below is the log output for the migrations, which successfully ran:

00:04:49 2017/10/06 00:04:18 [verbose] Kong: 0.11.0
00:04:49 2017/10/06 00:04:18 [verbose] no config file found at /etc/kong/kong.conf
00:04:49 2017/10/06 00:04:18 [verbose] no config file found at /etc/kong.conf
00:04:49 2017/10/06 00:04:18 [verbose] no config file, skipping loading
00:04:49 2017/10/06 00:04:18 [verbose] prefix in use: /usr/local/kong
00:04:49 2017/10/06 00:04:18 [verbose] running datastore migrations
00:04:49 2017/10/06 00:04:21 [info] migrating core for keyspace kong
00:04:49 2017/10/06 00:04:31 [info] core migrated up to: 2016-09-05-212515_retries_step_1
00:04:49 2017/10/06 00:04:37 [info] core migrated up to: 2016-09-05-212515_retries_step_2
00:04:49 2017/10/06 00:04:39 [info] core migrated up to: 2016-09-16-141423_upstreams
00:04:49 2017/10/06 00:04:39 [info] core migrated up to: 2016-12-14-172100_move_ssl_certs_to_core
00:04:49 2017/10/06 00:04:40 [info] core migrated up to: 2016-11-11-151900_new_apis_router_1
00:04:49 2017/10/06 00:04:41 [info] core migrated up to: 2016-11-11-151900_new_apis_router_2
00:04:49 2017/10/06 00:04:42 [info] core migrated up to: 2016-11-11-151900_new_apis_router_3
00:04:49 2017/10/06 00:04:42 [info] core migrated up to: 2017-01-24-132600_upstream_timeouts
00:04:49 2017/10/06 00:04:42 [info] core migrated up to: 2017-01-24-132600_upstream_timeouts_2
00:04:49 2017/10/06 00:04:43 [info] core migrated up to: 2017-03-27-132300_anonymous
00:04:49 2017/10/06 00:04:43 [info] core migrated up to: 2017-04-04-145100_cluster_events
00:04:49 2017/10/06 00:04:43 [info] core migrated up to: 2017-05-19-173100_remove_nodes_table
00:04:49 2017/10/06 00:04:43 [info] migrating statsd for keyspace kong
00:04:49 2017/10/06 00:04:45 [info] statsd migrated up to: 2017-06-09-160000_statsd_schema_changes
00:04:49 2017/10/06 00:04:45 [info] migrating cors for keyspace kong
00:04:49 2017/10/06 00:04:45 [info] cors migrated up to: 2017-03-14_multiple_orgins
00:04:49 2017/10/06 00:04:45 [info] migrating datadog for keyspace kong
00:04:49 2017/10/06 00:04:45 [info] datadog migrated up to: 2017-06-09-160000_datadog_schema_changes
00:04:49 2017/10/06 00:04:45 [info] migrating hmac-auth for keyspace kong
00:04:49 2017/10/06 00:04:45 [info] hmac-auth migrated up to: 2017-06-21-132400_init_hmacauth
00:04:49 2017/10/06 00:04:45 [info] migrating oauth2 for keyspace kong
00:04:49 2017/10/06 00:04:46 [info] oauth2 migrated up to: 2016-09-19-oauth2_code_index
00:04:49 2017/10/06 00:04:47 [info] oauth2 migrated up to: 2016-09-19-oauth2_api_id
00:04:49 2017/10/06 00:04:49 [info] oauth2 migrated up to: 2016-12-15-set_global_credentials
00:04:49 2017/10/06 00:04:49 [info] 19 migrations ran
00:04:49 2017/10/06 00:04:49 [info] waiting for Cassandra schema consensus (10000ms timeout)...
00:04:49 2017/10/06 00:04:49 [info] Cassandra schema consensus: reached
00:04:49 2017/10/06 00:04:49 [verbose] migrations up to date

However, if I now try to execute the command kong migrations up on the same cluster, I am now getting a migration error and it appears that kong is trying to run the same migrations that have already executed. I can confirm that the migrations have already ran (see log output below) and that the tables and columns already exist in the cassandra keyspace. Why is kong trying to execute these migrations again? This will be a big problem for us the next time we need to upgrade versions and actually run new migrations.

kong migrations list

bash-4.2$ kong migrations list -v
2017/10/19 18:02:10 [verbose] Kong: 0.11.0
2017/10/19 18:02:10 [verbose] no config file found at /etc/kong/kong.conf
2017/10/19 18:02:10 [verbose] no config file found at /etc/kong.conf
2017/10/19 18:02:10 [verbose] no config file, skipping loading
2017/10/19 18:02:10 [verbose] prefix in use: /usr/local/kong
2017/10/19 18:02:10 [info] Executed migrations for keyspace 'kong':
2017/10/19 18:02:10 [info] response-transformer: 2016-03-10-160000_resp_trans_schema_changes, 2016-03-10-160000_resp_trans_schema_changes
2017/10/19 18:02:10 [info] datadog: 2017-06-09-160000_datadog_schema_changes
2017/10/19 18:02:10 [info] hmac-auth: 2015-09-16-132400_init_hmacauth, 2015-09-16-132400_init_hmacauth, 2017-06-21-132400_init_hmacauth
2017/10/19 18:02:10 [info] core: 2015-01-12-175310_skeleton, 2015-01-12-175310_init_schema, 2015-11-23-817313_nodes, 2016-02-25-160900_remove_null_consumer_id, 2016-02-29-121813_remove_ttls, 2015-01-12-175310_skeleton, 2015-01-12-175310_init_schema, 2015-11-23-817313_nodes, 2016-02-25-160900_remove_null_consumer_id, 2016-02-29-121813_remove_ttls, 2016-09-05-212515_retries_step_1, 2016-09-05-212515_retries_step_2, 2016-09-16-141423_upstreams, 2016-12-14-172100_move_ssl_certs_to_core, 2016-11-11-151900_new_apis_router_1, 2016-11-11-151900_new_apis_router_2, 2016-11-11-151900_new_apis_router_3, 2017-01-24-132600_upstream_timeouts, 2017-01-24-132600_upstream_timeouts_2, 2017-03-27-132300_anonymous, 2017-04-04-145100_cluster_events, 2017-05-19-173100_remove_nodes_table
2017/10/19 18:02:10 [info] ip-restriction: 2016-05-24-remove-cache
2017/10/19 18:02:10 [info] statsd: 2017-06-09-160000_statsd_schema_changes
2017/10/19 18:02:10 [info] acl: 2015-08-25-841841_init_acl
2017/10/19 18:02:10 [info] cors: 2017-03-14_multiple_orgins
2017/10/19 18:02:10 [info] galileo: 2016-04-15_galileo-import-mashape-analytics, 2016-04-15_galileo-import-mashape-analytics
2017/10/19 18:02:10 [info] basic-auth: 2015-08-03-132400_init_basicauth, 2015-08-03-132400_init_basicauth
2017/10/19 18:02:10 [info] key-auth: 2015-07-31-172400_init_keyauth, 2015-07-31-172400_init_keyauth
2017/10/19 18:02:10 [info] rate-limiting: 2015-08-03-132400_init_ratelimiting, 2016-07-25-471385_ratelimiting_policies
2017/10/19 18:02:10 [info] request-transformer: 2016-03-10-160000_req_trans_schema_changes
2017/10/19 18:02:10 [info] jwt: 2015-06-09-jwt-auth, 2016-03-07-jwt-alg, 2015-06-09-jwt-auth
2017/10/19 18:02:10 [info] response-ratelimiting: 2015-08-21_init_response-rate-limiting, 2016-08-04-321512_response-rate-limiting_policies, 2015-08-21_init_response-rate-limiting, 2016-08-04-321512_response-rate-limiting_policies
2017/10/19 18:02:10 [info] oauth2: 2015-08-03-132400_init_oauth2, 2015-08-24-215800_cascade_delete_index, 2016-02-29-435612_remove_ttl, 2016-04-14-283949_serialize_redirect_uri, 2016-07-15-oauth2_code_credential_id, 2016-09-19-oauth2_code_index, 2016-09-19-oauth2_api_id, 2016-12-15-set_global_credentials

kong migrations up

bash-4.2$ kong migrations up -v
2017/10/19 18:02:16 [verbose] Kong: 0.11.0
2017/10/19 18:02:16 [verbose] no config file found at /etc/kong/kong.conf
2017/10/19 18:02:16 [verbose] no config file found at /etc/kong.conf
2017/10/19 18:02:16 [verbose] no config file, skipping loading
2017/10/19 18:02:16 [verbose] prefix in use: /usr/local/kong
2017/10/19 18:02:16 [verbose] running datastore migrations
2017/10/19 18:02:16 [info] migrating core for keyspace kong
Error:
/usr/local/share/lua/5.1/kong/cmd/migrations.lua:34: [cassandra error] Error during migration 2016-09-05-212515_retries_step_1: [Invalid] Invalid column name retries because it conflicts with an existing column
stack traceback:
	[C]: in function 'assert'
	/usr/local/share/lua/5.1/kong/cmd/migrations.lua:34: in function 'cmd_exec'
	/usr/local/share/lua/5.1/kong/cmd/init.lua:88: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:88>
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.1/kong/cmd/init.lua:88: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:45>
	/usr/local/bin/kong:7: in function 'file_gen'
	init_worker_by_lua:39: in function <init_worker_by_lua:37>
	[C]: in function 'xpcall'
	init_worker_by_lua:46: in function <init_worker_by_lua:44>

Steps To Reproduce

  1. Run v0.11.0 migrations as part of upgrade procedure (using cassandra db) from a single node
  2. Start up new v0.11.0 nodes
  3. If we try to run this command again, from a single node, we see the referenced error.

Additional Details & Logs

  • Kong version ($ kong version)
    v0.11.0

  • Kong debug-level startup logs ($ kong start --vv)

bash-4.2$ kong migrations list --vv
2017/10/19 18:11:14 [verbose] Kong: 0.11.0
2017/10/19 18:11:14 [debug] ngx_lua: 10008
2017/10/19 18:11:14 [debug] nginx: 1011002
2017/10/19 18:11:14 [debug] Lua: LuaJIT 2.1.0-beta2
2017/10/19 18:11:14 [verbose] no config file found at /etc/kong/kong.conf
2017/10/19 18:11:14 [verbose] no config file found at /etc/kong.conf
2017/10/19 18:11:14 [verbose] no config file, skipping loading
2017/10/19 18:11:14 [debug] KONG_ADMIN_LISTEN ENV found with "0.0.0.0:8001"
2017/10/19 18:11:14 [debug] KONG_CUSTOM_PLUGINS ENV found with "brigade-auth,brigade-cors,request-filter,api-client-enforcement"
2017/10/19 18:11:14 [debug] KONG_PROXY_LISTEN ENV found with "0.0.0.0:8000"
2017/10/19 18:11:14 [debug] KONG_DB_UPDATE_PROPAGATION ENV found with "1"
2017/10/19 18:11:14 [debug] KONG_DATABASE ENV found with "cassandra"
2017/10/19 18:11:14 [debug] KONG_NGINX_WORKER_PROCESSES ENV found with "8"
2017/10/19 18:11:14 [debug] KONG_PROXY_LISTEN_SSL ENV found with "0.0.0.0:8443"
2017/10/19 18:11:14 [debug] KONG_CASSANDRA_REPL_FACTOR ENV found with "3"
2017/10/19 18:11:14 [debug] KONG_NGINX_DAEMON ENV found with "off"
2017/10/19 18:11:14 [debug] KONG_ANONYMOUS_REPORTS ENV found with "no"
2017/10/19 18:11:14 [debug] KONG_CASSANDRA_CONTACT_POINTS ENV found with "production-cassandra-api-transport.service.consul"
2017/10/19 18:11:14 [debug] KONG_CASSANDRA_CONSISTENCY ENV found with "QUORUM"
2017/10/19 18:11:14 [debug] admin_access_log = "logs/admin_access.log"
2017/10/19 18:11:14 [debug] admin_error_log = "logs/error.log"
2017/10/19 18:11:14 [debug] admin_http2 = false
2017/10/19 18:11:14 [debug] admin_listen = "0.0.0.0:8001"
2017/10/19 18:11:14 [debug] admin_listen_ssl = "0.0.0.0:8444"
2017/10/19 18:11:14 [debug] admin_ssl = true
2017/10/19 18:11:14 [debug] anonymous_reports = false
2017/10/19 18:11:14 [debug] cassandra_consistency = "QUORUM"
2017/10/19 18:11:14 [debug] cassandra_contact_points = {"production-cassandra-api-transport.service.consul"}
2017/10/19 18:11:14 [debug] cassandra_data_centers = {"dc1:2","dc2:3"}
2017/10/19 18:11:14 [debug] cassandra_keyspace = "kong"
2017/10/19 18:11:14 [debug] cassandra_lb_policy = "RoundRobin"
2017/10/19 18:11:14 [debug] cassandra_port = 9042
2017/10/19 18:11:14 [debug] cassandra_repl_factor = 3
2017/10/19 18:11:14 [debug] cassandra_repl_strategy = "SimpleStrategy"
2017/10/19 18:11:14 [debug] cassandra_schema_consensus_timeout = 10000
2017/10/19 18:11:14 [debug] cassandra_ssl = false
2017/10/19 18:11:14 [debug] cassandra_ssl_verify = false
2017/10/19 18:11:14 [debug] cassandra_timeout = 5000
2017/10/19 18:11:14 [debug] cassandra_username = "kong"
2017/10/19 18:11:14 [debug] client_body_buffer_size = "8k"
2017/10/19 18:11:14 [debug] client_max_body_size = "0"
2017/10/19 18:11:14 [debug] client_ssl = false
2017/10/19 18:11:14 [debug] custom_plugins = {"brigade-auth","brigade-cors","request-filter","api-client-enforcement"}
2017/10/19 18:11:14 [debug] database = "cassandra"
2017/10/19 18:11:14 [debug] db_cache_ttl = 3600
2017/10/19 18:11:14 [debug] db_update_frequency = 5
2017/10/19 18:11:14 [debug] db_update_propagation = 1
2017/10/19 18:11:14 [debug] dns_error_ttl = 1
2017/10/19 18:11:14 [debug] dns_hostsfile = "/etc/hosts"
2017/10/19 18:11:14 [debug] dns_no_sync = false
2017/10/19 18:11:14 [debug] dns_not_found_ttl = 30
2017/10/19 18:11:14 [debug] dns_order = {"LAST","SRV","A","CNAME"}
2017/10/19 18:11:14 [debug] dns_resolver = {}
2017/10/19 18:11:14 [debug] dns_stale_ttl = 4
2017/10/19 18:11:14 [debug] error_default_type = "text/plain"
2017/10/19 18:11:14 [debug] http2 = false
2017/10/19 18:11:14 [debug] latency_tokens = true
2017/10/19 18:11:14 [debug] log_level = "notice"
2017/10/19 18:11:14 [debug] lua_code_cache = "on"
2017/10/19 18:11:14 [debug] lua_package_cpath = ""
2017/10/19 18:11:14 [debug] lua_package_path = "?/init.lua;./kong/?.lua"
2017/10/19 18:11:14 [debug] lua_socket_pool_size = 30
2017/10/19 18:11:14 [debug] lua_ssl_verify_depth = 1
2017/10/19 18:11:14 [debug] mem_cache_size = "128m"
2017/10/19 18:11:14 [debug] nginx_daemon = "off"
2017/10/19 18:11:14 [debug] nginx_optimizations = true
2017/10/19 18:11:14 [debug] nginx_user = "nobody nobody"
2017/10/19 18:11:14 [debug] nginx_worker_processes = "8"
2017/10/19 18:11:14 [debug] pg_database = "kong"
2017/10/19 18:11:14 [debug] pg_host = "127.0.0.1"
2017/10/19 18:11:14 [debug] pg_port = 5432
2017/10/19 18:11:14 [debug] pg_ssl = false
2017/10/19 18:11:14 [debug] pg_ssl_verify = false
2017/10/19 18:11:14 [debug] pg_user = "kong"
2017/10/19 18:11:14 [debug] prefix = "/usr/local/kong/"
2017/10/19 18:11:14 [debug] proxy_access_log = "logs/access.log"
2017/10/19 18:11:14 [debug] proxy_error_log = "logs/error.log"
2017/10/19 18:11:14 [debug] proxy_listen = "0.0.0.0:8000"
2017/10/19 18:11:14 [debug] proxy_listen_ssl = "0.0.0.0:8443"
2017/10/19 18:11:14 [debug] real_ip_header = "X-Real-IP"
2017/10/19 18:11:14 [debug] real_ip_recursive = "off"
2017/10/19 18:11:14 [debug] server_tokens = true
2017/10/19 18:11:14 [debug] ssl = true
2017/10/19 18:11:14 [debug] ssl_cipher_suite = "modern"
2017/10/19 18:11:14 [debug] ssl_ciphers = "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256"
2017/10/19 18:11:14 [debug] trusted_ips = {}
2017/10/19 18:11:14 [debug] upstream_keepalive = 60
2017/10/19 18:11:14 [verbose] prefix in use: /usr/local/kong
2017/10/19 18:11:14 [info] Executed migrations for keyspace 'kong':
2017/10/19 18:11:14 [info] response-transformer: 2016-03-10-160000_resp_trans_schema_changes, 2016-03-10-160000_resp_trans_schema_changes
2017/10/19 18:11:14 [info] datadog: 2017-06-09-160000_datadog_schema_changes
2017/10/19 18:11:14 [info] hmac-auth: 2015-09-16-132400_init_hmacauth, 2015-09-16-132400_init_hmacauth, 2017-06-21-132400_init_hmacauth
2017/10/19 18:11:14 [info] core: 2015-01-12-175310_skeleton, 2015-01-12-175310_init_schema, 2015-11-23-817313_nodes, 2016-02-25-160900_remove_null_consumer_id, 2016-02-29-121813_remove_ttls, 2015-01-12-175310_skeleton, 2015-01-12-175310_init_schema, 2015-11-23-817313_nodes, 2016-02-25-160900_remove_null_consumer_id, 2016-02-29-121813_remove_ttls, 2016-09-05-212515_retries_step_1, 2016-09-05-212515_retries_step_2, 2016-09-16-141423_upstreams, 2016-12-14-172100_move_ssl_certs_to_core, 2016-11-11-151900_new_apis_router_1, 2016-11-11-151900_new_apis_router_2, 2016-11-11-151900_new_apis_router_3, 2017-01-24-132600_upstream_timeouts, 2017-01-24-132600_upstream_timeouts_2, 2017-03-27-132300_anonymous, 2017-04-04-145100_cluster_events, 2017-05-19-173100_remove_nodes_table
2017/10/19 18:11:14 [info] ip-restriction: 2016-05-24-remove-cache
2017/10/19 18:11:14 [info] statsd: 2017-06-09-160000_statsd_schema_changes
2017/10/19 18:11:14 [info] acl: 2015-08-25-841841_init_acl
2017/10/19 18:11:14 [info] cors: 2017-03-14_multiple_orgins
2017/10/19 18:11:14 [info] galileo: 2016-04-15_galileo-import-mashape-analytics, 2016-04-15_galileo-import-mashape-analytics
2017/10/19 18:11:14 [info] basic-auth: 2015-08-03-132400_init_basicauth, 2015-08-03-132400_init_basicauth
2017/10/19 18:11:14 [info] key-auth: 2015-07-31-172400_init_keyauth, 2015-07-31-172400_init_keyauth
2017/10/19 18:11:14 [info] rate-limiting: 2015-08-03-132400_init_ratelimiting, 2016-07-25-471385_ratelimiting_policies
2017/10/19 18:11:14 [info] request-transformer: 2016-03-10-160000_req_trans_schema_changes
2017/10/19 18:11:14 [info] jwt: 2015-06-09-jwt-auth, 2016-03-07-jwt-alg, 2015-06-09-jwt-auth
2017/10/19 18:11:14 [info] response-ratelimiting: 2015-08-21_init_response-rate-limiting, 2016-08-04-321512_response-rate-limiting_policies, 2015-08-21_init_response-rate-limiting, 2016-08-04-321512_response-rate-limiting_policies
2017/10/19 18:11:14 [info] oauth2: 2015-08-03-132400_init_oauth2, 2015-08-24-215800_cascade_delete_index, 2016-02-29-435612_remove_ttl, 2016-04-14-283949_serialize_redirect_uri, 2016-07-15-oauth2_code_credential_id, 2016-09-19-oauth2_code_index, 2016-09-19-oauth2_api_id, 2016-12-15-set_global_credentials
  • A peek inside cassandra
cqlsh> use kong;
cqlsh:kong> describe tables;

schema_migrations      oauth2_tokens               cluster_events
ratelimiting_metrics   upstreams                   hmacauth_credentials
jwt_secrets            oauth2_authorization_codes  response_ratelimiting_metrics
consumers              keyauth_credentials         ssl_servers_names
apis                   oauth2_credentials          targets
ssl_certificates       acls
basicauth_credentials  plugins

cqlsh:kong> select * from schema_migrations;

 id                    | migrations
-----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
         rate-limiting |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  ['2015-08-03-132400_init_ratelimiting', '2016-07-25-471385_ratelimiting_policies']
               datadog |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        ['2017-06-09-160000_datadog_schema_changes']
                   acl |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      ['2015-08-25-841841_init_acl']
                statsd |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ['2017-06-09-160000_statsd_schema_changes']
            basic-auth |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                            ['2015-08-03-132400_init_basicauth', '2015-08-03-132400_init_basicauth']
  response-transformer |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      ['2016-03-10-160000_resp_trans_schema_changes', '2016-03-10-160000_resp_trans_schema_changes']
        ip-restriction |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ['2016-05-24-remove-cache']
                  cors |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      ['2017-03-14_multiple_orgins']
 response-ratelimiting |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      ['2015-08-21_init_response-rate-limiting', '2016-08-04-321512_response-rate-limiting_policies', '2015-08-21_init_response-rate-limiting', '2016-08-04-321512_response-rate-limiting_policies']
                oauth2 |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    ['2015-08-03-132400_init_oauth2', '2015-08-24-215800_cascade_delete_index', '2016-02-29-435612_remove_ttl', '2016-04-14-283949_serialize_redirect_uri', '2016-07-15-oauth2_code_credential_id', '2016-09-19-oauth2_code_index', '2016-09-19-oauth2_api_id', '2016-12-15-set_global_credentials']
                   jwt |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ['2015-06-09-jwt-auth', '2016-03-07-jwt-alg', '2015-06-09-jwt-auth']
                  core | ['2015-01-12-175310_skeleton', '2015-01-12-175310_init_schema', '2015-11-23-817313_nodes', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2015-01-12-175310_skeleton', '2015-01-12-175310_init_schema', '2015-11-23-817313_nodes', '2016-02-25-160900_remove_null_consumer_id', '2016-02-29-121813_remove_ttls', '2016-09-05-212515_retries_step_1', '2016-09-05-212515_retries_step_2', '2016-09-16-141423_upstreams', '2016-12-14-172100_move_ssl_certs_to_core', '2016-11-11-151900_new_apis_router_1', '2016-11-11-151900_new_apis_router_2', '2016-11-11-151900_new_apis_router_3', '2017-01-24-132600_upstream_timeouts', '2017-01-24-132600_upstream_timeouts_2', '2017-03-27-132300_anonymous', '2017-04-04-145100_cluster_events', '2017-05-19-173100_remove_nodes_table']
   request-transformer |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      ['2016-03-10-160000_req_trans_schema_changes']
             hmac-auth |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           ['2015-09-16-132400_init_hmacauth', '2015-09-16-132400_init_hmacauth', '2017-06-21-132400_init_hmacauth']
              key-auth |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                ['2015-07-31-172400_init_keyauth', '2015-07-31-172400_init_keyauth']
               galileo |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      ['2016-04-15_galileo-import-mashape-analytics', '2016-04-15_galileo-import-mashape-analytics']

(16 rows)
@hishamhm
Copy link
Contributor

Hi! Thank you for the detailed report!

Run v0.11.0 migrations as part of upgrade procedure (using cassandra db)
Start up new v0.11.0 nodes
Execute kong migrations up on the cluster (expecting a no-op)

This is the source of the issue: this is not the documented procedure for upgrading.

You should never run kong migrations up on the entire cluster. Since the removal of the Serf dependency in 0.11, executing the migrations is now a manual procedure that must be executed on only one Kong node. Then you can kong start the entire cluster normally.

I know that this sounds like the old joke that goes "Doctor, it hurts when I do this!" and then the doctor says "Well, so stop doing that!", but really, that's all there is to it: kong migrations up is a single-shot, single-node operation. Making it no-op in subsequent/concurrent executions would require cluster-wide synchronization, which in turn would require additional dependencies. We went for the more straightforward approach in 0.11: just run the migrations once in a single Kong node, and then you're good to go. :)

@brockmiller
Copy link
Author

Hey @hishamhm - Can you re-open this? My steps were actually in the wrong order: We did run migrations on a single node before starting up the new nodes.

The procedure went:

  • We left the existing v0.9.8 nodes running
  • We executed v0.11.0 migrations from a SINGLE node (we never run this command on the entire cluster)
  • Once the migrations ran successfully, we started up our v0.11.0 nodes.

@hishamhm hishamhm reopened this Oct 19, 2017
@hishamhm hishamhm added the task/needs-investigation Requires investigation and reproduction before classifying it as a bug or not. label Oct 19, 2017
@Tieske
Copy link
Member

Tieske commented Oct 23, 2017

As mentioned on Gitter, have you tried #2869 ?

@brockmiller
Copy link
Author

Ah I suspect that that is the root cause, however, it doesn't appear to be in an available release yet. Is that the case? I'm not sure if I'll be able to test this out in our production cluster without a properly released version.

Let me get back to you and see what I find out.

@Tieske
Copy link
Member

Tieske commented Oct 26, 2017

Actually it has just been released in 0.11.1

@hishamhm hishamhm added pending author feedback Waiting for the issue author to get back to a maintainer with findings, more details, etc... and removed task/needs-investigation Requires investigation and reproduction before classifying it as a bug or not. labels Oct 26, 2017
@brockmiller
Copy link
Author

Hey all, the update to v0.11.1 seems to have addressed the problem. Thanks!

2017/11/06 13:44:11 [verbose] Kong: 0.11.1
2017/11/06 13:44:11 [verbose] no config file found at /etc/kong/kong.conf
2017/11/06 13:44:11 [verbose] no config file found at /etc/kong.conf
2017/11/06 13:44:11 [verbose] no config file, skipping loading
2017/11/06 13:44:11 [warn] You are using Cassandra but your 'db_update_propagation' setting is set to '0' (default). Due to the distributed nature of Cassandra, you should increase this value.
2017/11/06 13:44:11 [verbose] prefix in use: /usr/local/opt/kong
2017/11/06 13:44:11 [verbose] running datastore migrations
2017/11/06 13:44:12 [verbose] migrations up to date

@kikito kikito closed this as completed Jan 18, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
pending author feedback Waiting for the issue author to get back to a maintainer with findings, more details, etc...
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@kikito @hishamhm @brockmiller @Tieske