Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Patching an upstream results in an ever increasing number of healthcheck events #7406

Closed
onematchfox opened this issue May 31, 2021 · 1 comment · Fixed by #7408
Closed

Patching an upstream results in an ever increasing number of healthcheck events #7406

onematchfox opened this issue May 31, 2021 · 1 comment · Fixed by #7408
Labels
core/balancer task/bug

Comments

@onematchfox
Copy link
Contributor

onematchfox commented May 31, 2021 •
edited
Loading

Summary

Kong version: 2.4.1 (reproduced in 1.5.0 as well)

While investigating some odd healthcheck behaviour during an upgrade from Kong 1.5.0 to Kong 2.4.1 I noticed that every time I patched an upstream (e.g. changing an attribute of the active healthchecks) I got an increasing number of event logs from the healthcheck module. I'm not sure why this is occurring nor whether it had any negative side effects. But I thought I would report it nonetheless as I suspect that this could be seen as a memory leak as it would seem like something is not being correctly disposed of.

Steps To Reproduce

  1. Start Kong - create a single upstream with associated target and active healthchecks enabled
  2. Observe the logs - notice the single target added event log
  3. Patch the upstream - don't actually even need to change any value - just a simple curl -X PATCH http://localhost:8001//upstreams/$guid is sufficient
  4. Observe the logs - notice that there are now 2 (or n depending on iteration) log events like:
...
[healthcheck] (mock) event: local cache cleared
[healthcheck] (mock) event: local cache cleared
...
[healthcheck] (mock) event: target added 'mock(172.26.128.4:3000)'
[healthcheck] (mock) event: target status 'mock(172.26.128.4:3000)' from 'false' to 'true'
[healthcheck] (mock) event: target added 'mock(172.26.128.4:3000)'
[healthcheck] (mock) event: target status 'mock(172.26.128.4:3000)' from 'false' to 'true'
  1. Repeat steps 3 and 4 to your heart's content noticing the ever increasing number of logs

Additional Details & Logs

  • Kong debug-level startup logs ($ kong start --vv)
  • Kong error logs (<KONG_PREFIX>/logs/error.log)
kong_1            | Waiting for PostgreSQL on 'kong-database:5432'.  up!
kong_1            | Database already bootstrapped
kong_1            | Database is already up-to-date
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] globalpatches.lua:10: installing the globalpatches
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:455: init(): [dns-client] (re)configuring dns client
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:460: init(): [dns-client] staleTtl = 4
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:463: init(): [dns-client] validTtl = nil
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:467: init(): [dns-client] noSynchronisation = false
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:486: init(): [dns-client] query order = LAST, SRV, A, CNAME
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:526: init(): [dns-client] adding A-record from 'hosts' file: localhost = 127.0.0.1
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: localhost = [::1]
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-mcastprefix = [ff00::0]
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-allnodes = [ff02::1]
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-allrouters = [ff02::2]
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-localhost = [::1]
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-loopback = [::1]
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:526: init(): [dns-client] adding A-record from 'hosts' file: 2605b5c63a4e = 172.26.128.5
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:526: init(): [dns-client] adding A-record from 'hosts' file: vagranthost = 10.0.2.2
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-localnet = [fe00::0]
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:585: init(): [dns-client] nameserver 127.0.0.11
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:590: init(): [dns-client] attempts = 5
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:597: init(): [dns-client] no_random = true
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:606: init(): [dns-client] timeout = 2000 ms
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:610: init(): [dns-client] ndots = 1
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:612: init(): [dns-client] search =
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:618: init(): [dns-client] badTtl = 1 s
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:620: init(): [dns-client] emptyTtl = 30 s
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] globalpatches.lua:263: randomseed(): seeding PRNG from OpenSSL RAND_bytes()
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:455: init(): [dns-client] (re)configuring dns client
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:460: init(): [dns-client] staleTtl = 4
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:463: init(): [dns-client] validTtl = nil
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:467: init(): [dns-client] noSynchronisation = false
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:486: init(): [dns-client] query order = LAST, SRV, A, CNAME
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:526: init(): [dns-client] adding A-record from 'hosts' file: localhost = 127.0.0.1
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: localhost = [::1]
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-mcastprefix = [ff00::0]
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-allnodes = [ff02::1]
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-allrouters = [ff02::2]
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-localhost = [::1]
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-loopback = [::1]
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:526: init(): [dns-client] adding A-record from 'hosts' file: 2605b5c63a4e = 172.26.128.5
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:526: init(): [dns-client] adding A-record from 'hosts' file: vagranthost = 10.0.2.2
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-localnet = [fe00::0]
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:585: init(): [dns-client] nameserver 127.0.0.11
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:590: init(): [dns-client] attempts = 5
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:597: init(): [dns-client] no_random = true
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:606: init(): [dns-client] timeout = 2000 ms
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:610: init(): [dns-client] ndots = 1
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:612: init(): [dns-client] search =
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:618: init(): [dns-client] badTtl = 1 s
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] client.lua:620: init(): [dns-client] emptyTtl = 30 s
kong_1            | 2021/05/31 13:04:17 [debug] 1#0: [lua] plugins.lua:245: load_plugin(): Loading plugin: request-transformer
kong_1            | 2021/05/31 13:04:17 [notice] 1#0: using the "epoll" event method
kong_1            | 2021/05/31 13:04:17 [notice] 1#0: openresty/1.19.3.1
kong_1            | 2021/05/31 13:04:17 [notice] 1#0: built by gcc 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)
kong_1            | 2021/05/31 13:04:17 [notice] 1#0: OS: Linux 4.15.0-143-generic
kong_1            | 2021/05/31 13:04:17 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576
kong_1            | 2021/05/31 13:04:17 [notice] 1#0: start worker processes
kong_1            | 2021/05/31 13:04:17 [notice] 1#0: start worker process 45
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *1 [lua] globalpatches.lua:263: randomseed(): seeding PRNG from OpenSSL RAND_bytes()
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *1 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=resty-worker-events, event=started, pid=45, data=nil
kong_1            | 2021/05/31 13:04:17 [notice] 45#0: *1 [lua] warmup.lua:92: single_dao(): Preloading 'services' into the core_cache..., context: init_worker_by_lua*
kong_1            | 2021/05/31 13:04:17 [notice] 45#0: *1 [lua] warmup.lua:129: single_dao(): finished preloading 'services' into the core_cache (in 50ms), context: init_worker_by_lua*
kong_1            | 2021/05/31 13:04:17 [notice] 45#0: *2 [lua] warmup.lua:34: warming up DNS entries ..., context: ngx.timer
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *3 [lua] base.lua:1503: new(): [upstream:mock 1] balancer_base created
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *3 [lua] round_robin.lua:165: new(): [upstream:mock 1] round_robin balancer created
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *3 [lua] base.lua:917: newHost(): [upstream:mock 1] created a new host for: mock
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *3 [lua] base.lua:655: queryDns(): [upstream:mock 1] querying dns for mock
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *3 [lua] base.lua:570: f(): [upstream:mock 1] dns record type changed for mock, nil -> 1
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *3 [lua] base.lua:370: newAddress(): [upstream:mock 1] new address for host 'mock' created: 172.26.128.2:3000 (weight 10)
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *3 [lua] base.lua:634: f(): [upstream:mock 1] updating balancer based on dns changes for mock
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *3 [lua] base.lua:644: f(): [upstream:mock 1] querying dns and updating for mock completed
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *3 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Got initial target list (0 targets)
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *3 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) active check flagged as active
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *3 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Healthchecker started!
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *3 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=healthy, pid=45, data=table: 0x7f62b5874988
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *3 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *3 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:17 [debug] 45#0: *3 [lua] balancer.lua:844: create_balancers(): initialized 1 balancer(s), 0 error(s)
kong_1            | 2021/05/31 13:04:18 [notice] 45#0: *2 [lua] warmup.lua:58: finished warming up DNS entries' into the cache (in 571ms), context: ngx.timer
kong_1            | 2021/05/31 13:04:22 [debug] 45#0: *39 [lua] init.lua:288: [cluster_events] polling events from: 1622466257.532
kong_1            | 2021/05/31 13:04:27 [debug] 45#0: *47 [lua] init.lua:288: [cluster_events] polling events from: 1622466257.532
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] init.lua:27: Loading Admin API endpoints
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] init.lua:118: No API endpoints loaded for plugin: request-transformer
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] events.lua:211: do_event(): worker-events: handling event; source=dao:crud, event=update, pid=nil, data=table: 0x7f62b15c2f98
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] init.lua:250: invalidate(): [DB cache] broadcasting (cluster) invalidation for key: 'upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=upstreams, pid=nil, data=table: 0x7f62b15c2f98
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=balancer, event=upstreams, pid=45, data=table: 0x7f62b15c5580
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:upstreams'
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca'
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca'
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) healthchecker stopped
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] base.lua:1503: new(): [upstream:mock 2] balancer_base created
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] round_robin.lua:165: new(): [upstream:mock 2] round_robin balancer created
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] base.lua:917: newHost(): [upstream:mock 2] created a new host for: mock
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] base.lua:655: queryDns(): [upstream:mock 2] querying dns for mock
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] base.lua:570: f(): [upstream:mock 2] dns record type changed for mock, nil -> 1
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] base.lua:370: newAddress(): [upstream:mock 2] new address for host 'mock' created: 172.26.128.2:3000 (weight 10)
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] base.lua:634: f(): [upstream:mock 2] updating balancer based on dns changes for mock
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] base.lua:644: f(): [upstream:mock 2] querying dns and updating for mock completed
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Got initial target list (0 targets)
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) active check flagged as active
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Healthchecker started!
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=balancer:upstreams
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=balancer:upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=clear, pid=45, data=table: 0x7f62b159dd08
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=healthy, pid=45, data=table: 0x7f62b159e248
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:30 [debug] 45#0: *54 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=upstreams:update, pid=nil, data=table: 0x7f62b15c2f98
kong_1            | 172.26.128.6 - - [31/May/2021:13:04:30 +0000] "PATCH /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca HTTP/1.1" 200 949 "-" "-"
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *58 [lua] init.lua:288: [cluster_events] polling events from: 1622466257.532
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] events.lua:211: do_event(): worker-events: handling event; source=dao:crud, event=update, pid=nil, data=table: 0x7f62b15882d8
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] init.lua:250: invalidate(): [DB cache] broadcasting (cluster) invalidation for key: 'upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=upstreams, pid=nil, data=table: 0x7f62b15882d8
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=balancer, event=upstreams, pid=45, data=table: 0x7f62b1579aa8
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:upstreams'
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca'
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca'
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) healthchecker stopped
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] base.lua:1503: new(): [upstream:mock 3] balancer_base created
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] round_robin.lua:165: new(): [upstream:mock 3] round_robin balancer created
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] base.lua:917: newHost(): [upstream:mock 3] created a new host for: mock
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] base.lua:655: queryDns(): [upstream:mock 3] querying dns for mock
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] base.lua:570: f(): [upstream:mock 3] dns record type changed for mock, nil -> 1
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] base.lua:370: newAddress(): [upstream:mock 3] new address for host 'mock' created: 172.26.128.2:3000 (weight 10)
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] base.lua:634: f(): [upstream:mock 3] updating balancer based on dns changes for mock
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] base.lua:644: f(): [upstream:mock 3] querying dns and updating for mock completed
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Got initial target list (0 targets)
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) active check flagged as active
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Healthchecker started!
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=balancer:upstreams
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=balancer:upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=clear, pid=45, data=table: 0x7f62b155ed30
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=healthy, pid=45, data=table: 0x7f62b155f110
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:32 [debug] 45#0: *62 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=upstreams:update, pid=nil, data=table: 0x7f62b15882d8
kong_1            | 172.26.128.6 - - [31/May/2021:13:04:32 +0000] "PATCH /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca HTTP/1.1" 200 949 "-" "-"
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] events.lua:211: do_event(): worker-events: handling event; source=dao:crud, event=update, pid=nil, data=table: 0x7f62b1541bf0
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] init.lua:250: invalidate(): [DB cache] broadcasting (cluster) invalidation for key: 'upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=upstreams, pid=nil, data=table: 0x7f62b1541bf0
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=balancer, event=upstreams, pid=45, data=table: 0x7f62b1546240
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:upstreams'
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca'
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca'
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) healthchecker stopped
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] base.lua:1503: new(): [upstream:mock 4] balancer_base created
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] round_robin.lua:165: new(): [upstream:mock 4] round_robin balancer created
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] base.lua:917: newHost(): [upstream:mock 4] created a new host for: mock
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] base.lua:655: queryDns(): [upstream:mock 4] querying dns for mock
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] base.lua:570: f(): [upstream:mock 4] dns record type changed for mock, nil -> 1
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] base.lua:370: newAddress(): [upstream:mock 4] new address for host 'mock' created: 172.26.128.2:3000 (weight 10)
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] base.lua:634: f(): [upstream:mock 4] updating balancer based on dns changes for mock
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] base.lua:644: f(): [upstream:mock 4] querying dns and updating for mock completed
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Got initial target list (0 targets)
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) active check flagged as active
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Healthchecker started!
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=balancer:upstreams
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=balancer:upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=clear, pid=45, data=table: 0x7f62b1516fb8
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=healthy, pid=45, data=table: 0x7f62b1517ba8
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:33 [debug] 45#0: *66 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=upstreams:update, pid=nil, data=table: 0x7f62b1541bf0
kong_1            | 172.26.128.6 - - [31/May/2021:13:04:33 +0000] "PATCH /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca HTTP/1.1" 200 949 "-" "-"
kong_1            | 2021/05/31 13:04:34 [debug] 45#0: *70 [lua] events.lua:211: do_event(): worker-events: handling event; source=dao:crud, event=update, pid=nil, data=table: 0x7f62b14f8b20
kong_1            | 2021/05/31 13:04:34 [debug] 45#0: *70 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 13:04:34 [debug] 45#0: *70 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85
kong_1            | 2021/05/31 13:04:34 [debug] 45#0: *70 [lua] init.lua:250: invalidate(): [DB cache] broadcasting (cluster) invalidation for key: 'upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=upstreams, pid=nil, data=table: 0x7f62b14f8b20
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=balancer, event=upstreams, pid=45, data=table: 0x7f62b14fbaa8
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:upstreams'
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca'
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca'
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) healthchecker stopped
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] base.lua:1503: new(): [upstream:mock 5] balancer_base created
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] round_robin.lua:165: new(): [upstream:mock 5] round_robin balancer created
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] base.lua:917: newHost(): [upstream:mock 5] created a new host for: mock
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] base.lua:655: queryDns(): [upstream:mock 5] querying dns for mock
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] base.lua:570: f(): [upstream:mock 5] dns record type changed for mock, nil -> 1
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] base.lua:370: newAddress(): [upstream:mock 5] new address for host 'mock' created: 172.26.128.2:3000 (weight 10)
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] base.lua:634: f(): [upstream:mock 5] updating balancer based on dns changes for mock
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] base.lua:644: f(): [upstream:mock 5] querying dns and updating for mock completed
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Got initial target list (0 targets)
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) active check flagged as active
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Healthchecker started!
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=balancer:upstreams
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=balancer:upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=clear, pid=45, data=table: 0x7f62b14ce9e8
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=healthy, pid=45, data=table: 0x7f62b14cf150
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:35 [debug] 45#0: *70 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=upstreams:update, pid=nil, data=table: 0x7f62b14f8b20
kong_1            | 172.26.128.6 - - [31/May/2021:13:04:35 +0000] "PATCH /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca HTTP/1.1" 200 949 "-" "-"
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] events.lua:211: do_event(): worker-events: handling event; source=dao:crud, event=update, pid=nil, data=table: 0x7f62b14b2930
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] init.lua:250: invalidate(): [DB cache] broadcasting (cluster) invalidation for key: 'upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=upstreams, pid=nil, data=table: 0x7f62b14b2930
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=balancer, event=upstreams, pid=45, data=table: 0x7f62b14b4298
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:upstreams'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) healthchecker stopped
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] base.lua:1503: new(): [upstream:mock 6] balancer_base created
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] round_robin.lua:165: new(): [upstream:mock 6] round_robin balancer created
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] base.lua:917: newHost(): [upstream:mock 6] created a new host for: mock
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] base.lua:655: queryDns(): [upstream:mock 6] querying dns for mock
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] base.lua:570: f(): [upstream:mock 6] dns record type changed for mock, nil -> 1
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] base.lua:370: newAddress(): [upstream:mock 6] new address for host 'mock' created: 172.26.128.2:3000 (weight 10)
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] base.lua:634: f(): [upstream:mock 6] updating balancer based on dns changes for mock
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] base.lua:644: f(): [upstream:mock 6] querying dns and updating for mock completed
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Got initial target list (0 targets)
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) active check flagged as active
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Healthchecker started!
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=balancer:upstreams
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=balancer:upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=45, data=balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=clear, pid=45, data=table: 0x7f62b14c5218
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=healthy, pid=45, data=table: 0x7f62b14c5c50
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.2:3000)'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] healthcheck.lua:1123: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.2:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 13:04:36 [debug] 45#0: *74 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=upstreams:update, pid=nil, data=table: 0x7f62b14b2930
kong_1            | 172.26.128.6 - - [31/May/2021:13:04:36 +0000] "PATCH /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca HTTP/1.1" 200 949 "-" "-"
kong_1            | 2021/05/31 13:04:37 [debug] 45#0: *78 [lua] init.lua:288: [cluster_events] polling events from: 1622466272.523

{
  "node_id": "41ee5847-31d2-4678-a2f0-2bac937f89b2",
  "tagline": "Welcome to kong",
  "timers": {
    "running": 1,
    "pending": 5
  },
  "pids": {
    "workers": [
      45
    ],
    "master": 1
  },
  "configuration": {
    "pg_port": 5432,
    "pg_password": "******",
    "pg_ssl": false,
    "pg_ssl_verify": false,
    "pg_max_concurrent_queries": 0,
    "pg_semaphore_timeout": 60000,
    "lua_package_path": "./?.lua;./?/init.lua;",
    "pg_ro_ssl": false,
    "pg_ro_ssl_verify": false,
    "database": "postgres",
    "cassandra_contact_points": [
      "127.0.0.1"
    ],
    "cassandra_port": 9042,
    "status_ssl_enabled": false,
    "cassandra_ssl": false,
    "cassandra_ssl_verify": false,
    "cassandra_write_consistency": "ONE",
    "status_ssl_cert_default": "/usr/local/kong/ssl/status-kong-default.crt",
    "cluster_listeners": [
      {
        "port": 8005,
        "listener": "0.0.0.0:8005",
        "proxy_protocol": false,
        "deferred": false,
        "reuseport": false,
        "backlog=%d+": false,
        "bind": false,
        "ip": "0.0.0.0",
        "ssl": false,
        "http2": false
      }
    ],
    "lua_ssl_trusted_certificate_combined": "/usr/local/kong/.ca_combined",
    "status_ssl_cert_default_ecdsa": "/usr/local/kong/ssl/status-kong-default-ecdsa.crt",
    "status_ssl_cert_key_default_ecdsa": "/usr/local/kong/ssl/status-kong-default-ecdsa.key",
    "cluster_control_plane": "127.0.0.1:8005",
    "cassandra_lb_policy": "RequestRoundRobin",
    "cluster_mtls": "shared",
    "kic": false,
    "cassandra_repl_strategy": "SimpleStrategy",
    "proxy_listen": [
      "0.0.0.0:8443\nssl"
    ],
    "admin_listen": [
      "0.0.0.0:8444 ssl"
    ],
    "cassandra_repl_factor": 1,
    "stream_listen": [
      "off"
    ],
    "nginx_http_keepalive_timeout": "0",
    "admin_ssl_cert": [
      "/etc/kong/certs/kong-cert.pem"
    ],
    "admin_ssl_cert_key": [
      "/etc/kong/certs/kong-key.pem"
    ],
    "status_ssl_cert": {},
    "status_ssl_cert_key": {},
    "headers": [
      "server_tokens",
      "latency_tokens"
    ],
    "nginx_user": "kong\nkong",
    "nginx_main_user": "kong kong",
    "nginx_daemon": "off",
    "nginx_http_ssl_prefer_server_ciphers": "off",
    "nginx_stream_ssl_prefer_server_ciphers": "off",
    "ssl_dhparam": "ffdhe2048",
    "nginx_http_gzip_disable": "msie6",
    "role": "traditional",
    "trusted_ips": {},
    "db_update_frequency": 5,
    "nginx_proxy_real_ip_header": "X-Real-IP",
    "db_update_propagation": 0,
    "nginx_http_ssl_session_timeout": "1d",
    "client_max_body_size": "0",
    "proxy_access_log": "/dev/stdout",
    "proxy_error_log": "/dev/stderr",
    "proxy_stream_access_log": "logs/access.log\nbasic",
    "proxy_stream_error_log": "logs/error.log",
    "nginx_proxy_set": "$template_root /usr/local/share/templates",
    "admin_error_log": "/dev/stderr",
    "status_access_log": "off",
    "nginx_http_large_client_header_buffers": "4 16k",
    "log_level": "debug",
    "nginx_optimizations": true,
    "lua_ssl_trusted_certificate": [
      "/etc/ssl/certs/ca-certificates.crt"
    ],
    "lua_ssl_verify_depth": 1,
    "lua_ssl_protocols": "TLSv1.1\nTLSv1.2 TLSv1.3",
    "nginx_http_lua_ssl_protocols": "TLSv1.1 TLSv1.2 TLSv1.3",
    "nginx_stream_lua_ssl_protocols": "TLSv1.1 TLSv1.2 TLSv1.3",
    "lua_socket_pool_size": 30,
    "untrusted_lua": "sandbox",
    "untrusted_lua_sandbox_requires": {},
    "untrusted_lua_sandbox_environment": {},
    "proxy_ssl_enabled": true,
    "plugins": [
      "request-transformer"
    ],
    "loaded_plugins": {
      "request-transformer": true
    },
    "cluster_ocsp": "off",
    "prefix": "/usr/local/kong",
    "ssl_cipher_suite": "intermediate",
    "upstream_keepalive_pool_size": 60,
    "nginx_http_client_body_buffer_size": "8k",
    "client_body_buffer_size": "8k",
    "db_cache_ttl": 0,
    "nginx_stream_ssl_session_tickets": "on",
    "nginx_worker_processes": "1",
    "nginx_main_worker_rlimit_nofile": "auto",
    "stream_listeners": {},
    "pluginserver_names": {},
    "lua_package_cpath": "",
    "nginx_events_multi_accept": "on",
    "nginx_proxy_error_page": "401\n402 403 405 406 407 409 410 415 416 417 418 421 422 423 424 425 426 428 429 431 451 501 505 506 507 508 510 511 /kong_error_handler",
    "mem_cache_size": "128m",
    "nginx_proxy_real_ip_recursive": "off",
    "status_listeners": {},
    "ssl_prefer_server_ciphers": "on",
    "ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
    "upstream_keepalive_idle_timeout": 60,
    "upstream_keepalive_max_requests": 100,
    "cassandra_refresh_frequency": 60,
    "go_plugins_dir": "off",
    "ssl_session_timeout": "1d",
    "ssl_cert_key_default_ecdsa": "/usr/local/kong/ssl/kong-default-ecdsa.key",
    "cassandra_schema_consensus_timeout": 10000,
    "cassandra_data_centers": [
      "dc1:2",
      "dc2:3"
    ],
    "go_pluginserver_exe": "/usr/local/bin/go-pluginserver",
    "nginx_http_types_hash_max_size": "2048",
    "port_maps": {},
    "nginx_admin_client_max_body_size": "10m",
    "cluster_data_plane_purge_delay": 1209600,
    "dns_order": [
      "LAST",
      "SRV",
      "A",
      "CNAME"
    ],
    "nginx_admin_client_body_buffer_size": "10m",
    "dns_stale_ttl": 4,
    "admin_ssl_cert_default_ecdsa": "/usr/local/kong/ssl/admin-kong-default-ecdsa.crt",
    "client_ssl": false,
    "db_resurrect_ttl": 30,
    "dns_hostsfile": "/etc/hosts",
    "ssl_cert_key": [
      "/etc/kong/certs/kong-key.pem"
    ],
    "error_default_type": "text/plain",
    "status_ssl_cert_key_default": "/usr/local/kong/ssl/status-kong-default.key",
    "host_ports": {},
    "anonymous_reports": false,
    "db_cache_warmup_entities": [
      "services"
    ],
    "cluster_listen": [
      "0.0.0.0:8005"
    ],
    "pg_host": "kong-database",
    "admin_listeners": [
      {
        "port": 8444,
        "listener": "0.0.0.0:8444\nssl",
        "proxy_protocol": false,
        "deferred": false,
        "reuseport": false,
        "backlog=%d+": false,
        "bind": false,
        "ip": "0.0.0.0",
        "ssl": true,
        "http2": false
      }
    ],
    "nginx_http_ssl_protocols": "TLSv1.2 TLSv1.3",
    "admin_ssl_cert_key_default_ecdsa": "/usr/local/kong/ssl/admin-kong-default-ecdsa.key",
    "pg_database": "kong",
    "pg_timeout": 5000,
    "admin_access_log": "/dev/stdout",
    "worker_state_update_frequency": 5,
    "admin_ssl_cert_default": "/usr/local/kong/ssl/admin-kong-default.crt",
    "ssl_cert": [
      "/etc/kong/certs/kong-cert.pem"
    ],
    "pg_user": "kong",
    "cassandra_keyspace": "kong",
    "nginx_http_sendfile": "on",
    "nginx_http_client_max_body_size": "0",
    "cassandra_read_consistency": "ONE",
    "ssl_ciphers": "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384",
    "status_error_log": "logs/status_error.log",
    "cassandra_timeout": 5000,
    "nginx_events_worker_connections": "auto",
    "nginx_http_ssl_session_tickets": "on",
    "worker_consistency": "strict",
    "ssl_protocols": "TLSv1.1\nTLSv1.2 TLSv1.3",
    "nginx_stream_ssl_dhparam": "ffdhe2048",
    "nginx_http_ssl_dhparam": "ffdhe2048",
    "nginx_main_directives": [
      {
        "name": "daemon",
        "value": "off"
      },
      {
        "name": "user",
        "value": "kong kong"
      },
      {
        "name": "worker_processes",
        "value": "1"
      },
      {
        "name": "worker_rlimit_nofile",
        "value": "auto"
      }
    ],
    "nginx_http_gzip": "on",
    "nginx_main_daemon": "off",
    "nginx_events_directives": [
      {
        "name": "multi_accept",
        "value": "on"
      },
      {
        "name": "worker_connections",
        "value": "auto"
      }
    ],
    "ssl_session_tickets": "on",
    "nginx_http_directives": [
      {
        "name": "client_body_buffer_size",
        "value": "8k"
      },
      {
        "name": "client_max_body_size",
        "value": "0"
      },
      {
        "name": "default_type",
        "value": "application/octet-stream"
      },
      {
        "name": "gzip",
        "value": "on"
      },
      {
        "name": "gzip_disable",
        "value": "msie6"
      },
      {
        "name": "keepalive_timeout",
        "value": "0"
      },
      {
        "name": "large_client_header_buffers",
        "value": "4\n16k"
      },
      {
        "name": "lua_ssl_protocols",
        "value": "TLSv1.1 TLSv1.2 TLSv1.3"
      },
      {
        "name": "sendfile",
        "value": "on"
      },
      {
        "name": "ssl_dhparam",
        "value": "/usr/local/kong/ssl/ffdhe2048.pem"
      },
      {
        "name": "ssl_prefer_server_ciphers",
        "value": "off"
      },
      {
        "name": "ssl_protocols",
        "value": "TLSv1.2\nTLSv1.3"
      },
      {
        "name": "ssl_session_tickets",
        "value": "on"
      },
      {
        "name": "ssl_session_timeout",
        "value": "1d"
      },
      {
        "name": "tcp_nodelay",
        "value": "on"
      },
      {
        "name": "tcp_nopush",
        "value": "on"
      },
      {
        "name": "types_hash_max_size",
        "value": "2048"
      }
    ],
    "enabled_headers": {
      "X-Kong-Upstream-Status": false,
      "server_tokens": true,
      "X-Kong-Proxy-Latency": true,
      "Via": true,
      "X-Kong-Response-Latency": true,
      "X-Kong-Admin-Latency": true,
      "Server": true,
      "X-Kong-Upstream-Latency": true,
      "latency_tokens": true
    },
    "nginx_http_tcp_nodelay": "on",
    "nginx_main_worker_processes": "1",
    "real_ip_header": "X-Real-IP",
    "nginx_upstream_directives": {},
    "nginx_stream_ssl_session_timeout": "1d",
    "nginx_proxy_directives": [
      {
        "name": "error_page",
        "value": "401\n402 403 405 406 407 409 410 415 416 417 418 421 422 423 424 425 426 428 429 431 451 501 505 506 507 508 510 511 /kong_error_handler"
      },
      {
        "name": "real_ip_header",
        "value": "X-Real-IP"
      },
      {
        "name": "real_ip_recursive",
        "value": "off"
      },
      {
        "name": "set",
        "value": "$template_root\n/usr/local/share/templates"
      }
    ],
    "status_listen": [
      "off"
    ],
    "nginx_status_directives": {},
    "admin_ssl_cert_key_default": "/usr/local/kong/ssl/admin-kong-default.key",
    "nginx_admin_directives": [
      {
        "name": "client_body_buffer_size",
        "value": "10m"
      },
      {
        "name": "client_max_body_size",
        "value": "10m"
      }
    ],
    "cassandra_username": "kong",
    "nginx_stream_directives": [
      {
        "name": "lua_ssl_protocols",
        "value": "TLSv1.1\nTLSv1.2 TLSv1.3"
      },
      {
        "name": "ssl_dhparam",
        "value": "/usr/local/kong/ssl/ffdhe2048.pem"
      },
      {
        "name": "ssl_prefer_server_ciphers",
        "value": "off"
      },
      {
        "name": "ssl_protocols",
        "value": "TLSv1.2 TLSv1.3"
      },
      {
        "name": "ssl_session_tickets",
        "value": "on"
      },
      {
        "name": "ssl_session_timeout",
        "value": "1d"
      }
    ],
    "nginx_http_tcp_nopush": "on",
    "nginx_supstream_directives": {},
    "dns_resolver": {},
    "nginx_sproxy_directives": {},
    "nginx_stream_ssl_protocols": "TLSv1.2\nTLSv1.3",
    "dns_error_ttl": 1,
    "dns_not_found_ttl": 30,
    "nginx_http_upstream_directives": {},
    "nginx_http_status_directives": {},
    "dns_no_sync": false,
    "nginx_pid": "/usr/local/kong/pids/nginx.pid",
    "client_ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
    "client_ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
    "nginx_err_logs": "/usr/local/kong/logs/error.log",
    "ssl_cert_default_ecdsa": "/usr/local/kong/ssl/kong-default-ecdsa.crt",
    "ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
    "nginx_acc_logs": "/usr/local/kong/logs/access.log",
    "real_ip_recursive": "off",
    "admin_acc_logs": "/usr/local/kong/logs/admin_access.log",
    "admin_ssl_enabled": true,
    "nginx_conf": "/usr/local/kong/nginx.conf",
    "stream_proxy_ssl_enabled": false,
    "nginx_kong_conf": "/usr/local/kong/nginx-kong.conf",
    "proxy_listeners": [
      {
        "port": 8443,
        "listener": "0.0.0.0:8443\nssl",
        "proxy_protocol": false,
        "deferred": false,
        "reuseport": false,
        "backlog=%d+": false,
        "bind": false,
        "ip": "0.0.0.0",
        "ssl": true,
        "http2": false
      }
    ],
    "nginx_kong_stream_conf": "/usr/local/kong/nginx-kong-stream.conf",
    "nginx_http_default_type": "application/octet-stream",
    "kong_env": "/usr/local/kong/.kong_env",
    "ssl_cert_csr_default": "/usr/local/kong/ssl/kong-default.csr"
  },
  "plugins": {
    "available_on_server": {
      "request-transformer": true
    },
    "enabled_in_cluster": [
      "request-transformer"
    ]
  },
  "version": "2.4.1",
  "hostname": "2605b5c63a4e",
  "lua_version": "LuaJIT\n2.1.0-beta3"
}
  • Operating system - Ubuntu Bionic
@onematchfox onematchfox mentioned this issue May 31, 2021
Closed
@onematchfox onematchfox changed the title Patching an upstream results in an ever increasing number of healthcheck _events_ Patching an upstream results in an ever increasing number of healthcheck events May 31, 2021
@onematchfox onematchfox mentioned this issue May 31, 2021
Closed
@locao locao added the task/bug label Jun 23, 2021
This was referenced Jun 23, 2021
Merged
Merged
@locao
Copy link
Contributor

locao commented Feb 8, 2022

Fixed months ago by #7408. Thanks @onematchfox!

@locao locao closed this as completed Feb 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
core/balancer task/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix(balancer) ensure existing healthchecker is stopped when processing target event onematchfox/kong
3 participants
@kikito @onematchfox @locao