Healthchecks duplicated when updating target #7407

onematchfox · 2021-05-31T18:04:13Z

Summary

Kong version: 2.4.1

Whilst investigating #7406 I noticed that when deleting and adding targets to an upstream, the healthchecks associated with the upstream were duplicated.

Steps To Reproduce

Start Kong - create a single upstream with associated target and active healthchecks enabled
Delete the target
Add a new target
Observe the logs - notice that there are now 2 sets of healthchecks happening
Patch the upstream healthchecks - e.g change the path.
Observe the logs - notice that only the latest version of the healthcheck is updated so there are 2 different sets of healthchecks occurring

In the logs below, mock endpoint is using httpbin. Healthcheck path is initially set to / - which redirects (status 301) - and healthy interval at 30 (remains the same throughout). Firstly, the target is firstly deleted and readed to illutstrate the duplicate checks. The healthcheck path is subsequently patched to /status/500 and finally /status/200 to further illustrate the point and perhaps remove any potential suspicion regarding this being the same underlying healthcheck.

Additional Details & Logs

Kong debug-level startup logs ($ kong start --vv)
Kong error logs (<KONG_PREFIX>/logs/error.log)

kong_1            | Waiting for PostgreSQL on 'kong-database:5432'.  up!
kong_1            | Database already bootstrapped
kong_1            | Database is already up-to-date
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] globalpatches.lua:10: installing the globalpatches
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:455: init(): [dns-client] (re)configuring dns client
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:460: init(): [dns-client] staleTtl = 4
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:463: init(): [dns-client] validTtl = nil
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:467: init(): [dns-client] noSynchronisation = false
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:486: init(): [dns-client] query order = LAST, SRV, A, CNAME
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-mcastprefix = [ff00::0]
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-allnodes = [ff02::1]
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-allrouters = [ff02::2]
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-localhost = [::1]
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-loopback = [::1]
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:526: init(): [dns-client] adding A-record from 'hosts' file: vagranthost = 10.0.2.2
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:526: init(): [dns-client] adding A-record from 'hosts' file: localhost = 127.0.0.1
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: localhost = [::1]
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:526: init(): [dns-client] adding A-record from 'hosts' file: a104f7f3b383 = 172.26.128.7
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-localnet = [fe00::0]
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:585: init(): [dns-client] nameserver 127.0.0.11
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:590: init(): [dns-client] attempts = 5
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:597: init(): [dns-client] no_random = true
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:606: init(): [dns-client] timeout = 2000 ms
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:610: init(): [dns-client] ndots = 1
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:612: init(): [dns-client] search = 
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:618: init(): [dns-client] badTtl = 1 s
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:620: init(): [dns-client] emptyTtl = 30 s
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] globalpatches.lua:263: randomseed(): seeding PRNG from OpenSSL RAND_bytes()
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:455: init(): [dns-client] (re)configuring dns client
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:460: init(): [dns-client] staleTtl = 4
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:463: init(): [dns-client] validTtl = nil
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:467: init(): [dns-client] noSynchronisation = false
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:486: init(): [dns-client] query order = LAST, SRV, A, CNAME
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-mcastprefix = [ff00::0]
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-allnodes = [ff02::1]
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-allrouters = [ff02::2]
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-localhost = [::1]
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-loopback = [::1]
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:526: init(): [dns-client] adding A-record from 'hosts' file: vagranthost = 10.0.2.2
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:526: init(): [dns-client] adding A-record from 'hosts' file: localhost = 127.0.0.1
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: localhost = [::1]
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:526: init(): [dns-client] adding A-record from 'hosts' file: a104f7f3b383 = 172.26.128.7
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:541: init(): [dns-client] adding AAAA-record from 'hosts' file: ip6-localnet = [fe00::0]
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:585: init(): [dns-client] nameserver 127.0.0.11
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:590: init(): [dns-client] attempts = 5
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:597: init(): [dns-client] no_random = true
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:606: init(): [dns-client] timeout = 2000 ms
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:610: init(): [dns-client] ndots = 1
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:612: init(): [dns-client] search = 
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:618: init(): [dns-client] badTtl = 1 s
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] client.lua:620: init(): [dns-client] emptyTtl = 30 s
kong_1            | 2021/05/31 17:48:07 [debug] 1#0: [lua] plugins.lua:245: load_plugin(): Loading plugin: request-transformer
kong_1            | 2021/05/31 17:48:07 [notice] 1#0: using the "epoll" event method
kong_1            | 2021/05/31 17:48:07 [notice] 1#0: openresty/1.19.3.1
kong_1            | 2021/05/31 17:48:07 [notice] 1#0: built by gcc 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04) 
kong_1            | 2021/05/31 17:48:07 [notice] 1#0: OS: Linux 4.15.0-143-generic
kong_1            | 2021/05/31 17:48:07 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 1048576:1048576
kong_1            | 2021/05/31 17:48:07 [notice] 1#0: start worker processes
kong_1            | 2021/05/31 17:48:07 [notice] 1#0: start worker process 44
kong_1            | 2021/05/31 17:48:07 [debug] 44#0: *1 [lua] globalpatches.lua:263: randomseed(): seeding PRNG from OpenSSL RAND_bytes()
kong_1            | 2021/05/31 17:48:07 [debug] 44#0: *1 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=resty-worker-events, event=started, pid=44, data=nil
kong_1            | 2021/05/31 17:48:07 [notice] 44#0: *1 [lua] warmup.lua:92: single_dao(): Preloading 'services' into the core_cache..., context: init_worker_by_lua*
kong_1            | 2021/05/31 17:48:07 [notice] 44#0: *1 [lua] warmup.lua:129: single_dao(): finished preloading 'services' into the core_cache (in 78ms), context: init_worker_by_lua*
kong_1            | 2021/05/31 17:48:07 [notice] 44#0: *2 [lua] warmup.lua:34: warming up DNS entries ..., context: ngx.timer
kong_1            | 2021/05/31 17:48:08 [debug] 44#0: *3 [lua] base.lua:1503: new(): [upstream:mock 1] balancer_base created
kong_1            | 2021/05/31 17:48:08 [debug] 44#0: *3 [lua] round_robin.lua:165: new(): [upstream:mock 1] round_robin balancer created
kong_1            | 2021/05/31 17:48:08 [debug] 44#0: *3 [lua] base.lua:917: newHost(): [upstream:mock 1] created a new host for: mock
kong_1            | 2021/05/31 17:48:08 [debug] 44#0: *3 [lua] base.lua:655: queryDns(): [upstream:mock 1] querying dns for mock
kong_1            | 2021/05/31 17:48:08 [debug] 44#0: *3 [lua] base.lua:570: f(): [upstream:mock 1] dns record type changed for mock, nil -> 1
kong_1            | 2021/05/31 17:48:08 [debug] 44#0: *3 [lua] base.lua:370: newAddress(): [upstream:mock 1] new address for host 'mock' created: 172.26.128.4:3000 (weight 100)
kong_1            | 2021/05/31 17:48:08 [debug] 44#0: *3 [lua] base.lua:634: f(): [upstream:mock 1] updating balancer based on dns changes for mock
kong_1            | 2021/05/31 17:48:08 [debug] 44#0: *3 [lua] base.lua:644: f(): [upstream:mock 1] querying dns and updating for mock completed
kong_1            | 2021/05/31 17:48:08 [debug] 44#0: *3 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Got initial target list (0 targets)
kong_1            | 2021/05/31 17:48:08 [debug] 44#0: *3 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) active check flagged as active
kong_1            | 2021/05/31 17:48:08 [debug] 44#0: *3 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) starting timer to check active checks
kong_1            | 2021/05/31 17:48:08 [debug] 44#0: *3 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Healthchecker started!
kong_1            | 2021/05/31 17:48:08 [debug] 44#0: *3 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=healthy, pid=44, data=table: 0x7ff08fbd0b20
kong_1            | 2021/05/31 17:48:08 [debug] 44#0: *3 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.4:3000)'
kong_1            | 2021/05/31 17:48:08 [debug] 44#0: *3 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.4:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 17:48:08 [debug] 44#0: *3 [lua] balancer.lua:847: create_balancers(): initialized 1 balancer(s), 0 error(s)
kong_1            | 2021/05/31 17:48:09 [debug] 44#0: *35 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) worker 0 (pid: 44) starting active check timer
kong_1            | 2021/05/31 17:48:09 [notice] 44#0: *2 [lua] warmup.lua:58: finished warming up DNS entries' into the cache (in 2097ms), context: ngx.timer
kong_1            | 2021/05/31 17:48:12 [debug] 44#0: *41 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:48:17 [debug] 44#0: *49 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:48:22 [debug] 44#0: *57 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:48:23 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:48:27 [debug] 44#0: *65 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:48:32 [debug] 44#0: *73 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:48:37 [debug] 44#0: *81 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:48:38 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:48:38 [debug] 44#0: *86 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:48:38 [debug] 44#0: *86 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently healthy)
kong_1            | 2021/05/31 17:48:38 [debug] 44#0: *86 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 301)
kong_1            | 2021/05/31 17:48:42 [debug] 44#0: *91 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:48:47 [debug] 44#0: *99 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:48:52 [debug] 44#0: *107 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:48:53 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:48:57 [debug] 44#0: *115 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:49:02 [debug] 44#0: *123 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:49:07 [debug] 44#0: *132 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:49:08 [debug] 44#0: *136 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:49:08 [debug] 44#0: *136 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently healthy)
kong_1            | 2021/05/31 17:49:08 [debug] 44#0: *136 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 301)
kong_1            | 2021/05/31 17:49:08 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:49:17 [debug] 44#0: *150 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:49:22 [debug] 44#0: *158 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:49:23 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:49:27 [debug] 44#0: *166 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:49:32 [debug] 44#0: *174 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:49:37 [debug] 44#0: *182 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:49:38 [debug] 44#0: *186 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:49:38 [debug] 44#0: *186 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently healthy)
kong_1            | 2021/05/31 17:49:38 [debug] 44#0: *186 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 301)
kong_1            | 2021/05/31 17:49:38 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:49:47 [debug] 44#0: *200 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:49:52 [debug] 44#0: *208 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:49:53 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:49:57 [debug] 44#0: *216 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:50:02 [debug] 44#0: *224 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:50:07 [debug] 44#0: *233 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:50:08 [debug] 44#0: *237 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:50:08 [debug] 44#0: *237 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently healthy)
kong_1            | 2021/05/31 17:50:08 [debug] 44#0: *237 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 301)
kong_1            | 2021/05/31 17:50:08 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:50:12 [debug] 44#0: *243 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:50:17 [debug] 44#0: *251 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:50:22 [debug] 44#0: *259 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:50:23 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:50:27 [debug] 44#0: *267 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:50:32 [debug] 44#0: *275 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:50:37 [debug] 44#0: *283 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:50:38 [debug] 44#0: *287 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:50:38 [debug] 44#0: *287 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently healthy)
kong_1            | 2021/05/31 17:50:38 [debug] 44#0: *287 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 301)
kong_1            | 2021/05/31 17:50:39 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:50:42 [debug] 44#0: *293 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:50:47 [debug] 44#0: *301 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:50:52 [debug] 44#0: *309 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:50:54 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:50:57 [debug] 44#0: *317 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:51:02 [debug] 44#0: *325 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:51:07 [debug] 44#0: *334 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:51:08 [debug] 44#0: *338 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:51:08 [debug] 44#0: *338 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently healthy)
kong_1            | 2021/05/31 17:51:08 [debug] 44#0: *338 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 301)
kong_1            | 2021/05/31 17:51:09 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:51:12 [debug] 44#0: *344 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:51:17 [debug] 44#0: *354 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *362 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] init.lua:27: Loading Admin API endpoints
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] init.lua:118: No API endpoints loaded for plugin: request-transformer
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] events.lua:211: do_event(): worker-events: handling event; source=dao:crud, event=delete, pid=nil, data=table: 0x7ff08a1b9438
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'targets:35ebb1a3-a464-4c9f-913a-025b7ea52ed1:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=44, data=targets:35ebb1a3-a464-4c9f-913a-025b7ea52ed1:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] init.lua:250: invalidate(): [DB cache] broadcasting (cluster) invalidation for key: 'targets:35ebb1a3-a464-4c9f-913a-025b7ea52ed1:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=targets, pid=nil, data=table: 0x7ff08a1b9438
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=balancer, event=targets, pid=44, data=table: 0x7ff08a1bc550
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] balancer.lua:649: on_target_event(): target delete for upstream 26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca'
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] base.lua:1503: new(): [upstream:mock 2] balancer_base created
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] round_robin.lua:165: new(): [upstream:mock 2] round_robin balancer created
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Got initial target list (1 targets)
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Got initial status healthy mock 172.26.128.4:3000
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) active check flagged as active
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Healthchecker started!
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=44, data=balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 17:51:22 [debug] 44#0: *363 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=targets:delete, pid=nil, data=table: 0x7ff08a1b9438
kong_1            | 172.26.128.8 - - [31/May/2021:17:51:22 +0000] "DELETE /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca/targets/35ebb1a3-a464-4c9f-913a-025b7ea52ed1 HTTP/1.1" 204 0 "-" "-"
kong_1            | 172.26.128.8 - - [31/May/2021:17:51:22 +0000] "GET /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca/targets?size=1000 HTTP/1.1" 200 23 "-" "-"
kong_1            | 172.26.128.8 - - [31/May/2021:17:51:22 +0000] "GET /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca/health?size=1000 HTTP/1.1" 200 72 "-" "-"
kong_1            | 2021/05/31 17:51:24 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] events.lua:211: do_event(): worker-events: handling event; source=dao:crud, event=create, pid=nil, data=table: 0x7ff08a1778e0
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'targets:72092901-e8fd-4e4f-bd27-8277d9ca3590:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=44, data=targets:72092901-e8fd-4e4f-bd27-8277d9ca3590:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] init.lua:250: invalidate(): [DB cache] broadcasting (cluster) invalidation for key: 'targets:72092901-e8fd-4e4f-bd27-8277d9ca3590:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=targets, pid=nil, data=table: 0x7ff08a1778e0
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=balancer, event=targets, pid=44, data=table: 0x7ff08a1797a8
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] balancer.lua:649: on_target_event(): target create for upstream 26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca'
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] base.lua:1503: new(): [upstream:mock 3] balancer_base created
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] round_robin.lua:165: new(): [upstream:mock 3] round_robin balancer created
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] base.lua:917: newHost(): [upstream:mock 3] created a new host for: mock
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] base.lua:655: queryDns(): [upstream:mock 3] querying dns for mock
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] base.lua:570: f(): [upstream:mock 3] dns record type changed for mock, nil -> 1
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] base.lua:370: newAddress(): [upstream:mock 3] new address for host 'mock' created: 172.26.128.4:3000 (weight 100)
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] base.lua:634: f(): [upstream:mock 3] updating balancer based on dns changes for mock
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] base.lua:644: f(): [upstream:mock 3] querying dns and updating for mock completed
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Got initial target list (1 targets)
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Got initial status healthy mock 172.26.128.4:3000
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) active check flagged as active
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Healthchecker started!
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) adding an existing target: mock 172.26.128.4:3000 (ignoring)
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=44, data=balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 17:51:30 [debug] 44#0: *376 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=targets:create, pid=nil, data=table: 0x7ff08a1778e0
kong_1            | 172.26.128.8 - - [31/May/2021:17:51:30 +0000] "POST /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca/targets HTTP/1.1" 201 176 "-" "-"
kong_1            | 172.26.128.8 - - [31/May/2021:17:51:30 +0000] "GET /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca/targets?size=1000 HTTP/1.1" 200 199 "-" "-"
kong_1            | 172.26.128.8 - - [31/May/2021:17:51:30 +0000] "GET /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca/health?size=1000 HTTP/1.1" 200 464 "-" "-"
kong_1            | 2021/05/31 17:51:32 [debug] 44#0: *386 [lua] init.lua:288: [cluster_events] polling events from: 1622483287.368
kong_1            | 2021/05/31 17:51:37 [debug] 44#0: *394 [lua] init.lua:288: [cluster_events] polling events from: 1622483492.591
kong_1            | 2021/05/31 17:51:37 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:51:38 [debug] 44#0: *396 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:51:38 [debug] 44#0: *396 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently healthy)
kong_1            | 2021/05/31 17:51:38 [debug] 44#0: *396 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 301)
kong_1            | 2021/05/31 17:51:39 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:51:42 [debug] 44#0: *404 [lua] init.lua:288: [cluster_events] polling events from: 1622483492.591
kong_1            | 2021/05/31 17:51:45 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:51:47 [debug] 44#0: *412 [lua] init.lua:288: [cluster_events] polling events from: 1622483492.591
kong_1            | 2021/05/31 17:51:52 [debug] 44#0: *420 [lua] init.lua:288: [cluster_events] polling events from: 1622483492.591
kong_1            | 2021/05/31 17:51:52 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:51:52 [debug] 44#0: *421 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:51:52 [debug] 44#0: *421 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently healthy)
kong_1            | 2021/05/31 17:51:52 [debug] 44#0: *421 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 301)
kong_1            | 2021/05/31 17:51:54 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:51:57 [debug] 44#0: *430 [lua] init.lua:288: [cluster_events] polling events from: 1622483492.591
kong_1            | 2021/05/31 17:52:00 [debug] 44#0: *433 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:52:00 [debug] 44#0: *433 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently healthy)
kong_1            | 2021/05/31 17:52:00 [debug] 44#0: *433 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 301)
kong_1            | 2021/05/31 17:52:00 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:52:02 [debug] 44#0: *440 [lua] init.lua:288: [cluster_events] polling events from: 1622483492.591
kong_1            | 172.26.128.8 - - [31/May/2021:17:52:03 +0000] "GET /upstreams?size=1000 HTTP/1.1" 200 974 "-" "-"
kong_1            | 172.26.128.8 - - [31/May/2021:17:52:06 +0000] "GET /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca HTTP/1.1" 200 951 "-" "-"
kong_1            | 172.26.128.8 - - [31/May/2021:17:52:06 +0000] "GET /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca/targets?size=1000 HTTP/1.1" 200 199 "-" "-"
kong_1            | 172.26.128.8 - - [31/May/2021:17:52:06 +0000] "GET /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca/health?size=1000 HTTP/1.1" 200 464 "-" "-"
kong_1            | 2021/05/31 17:52:07 [debug] 44#0: *453 [lua] init.lua:288: [cluster_events] polling events from: 1622483492.591
kong_1            | 2021/05/31 17:52:07 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:52:08 [debug] 44#0: *455 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:52:08 [debug] 44#0: *455 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently healthy)
kong_1            | 2021/05/31 17:52:08 [debug] 44#0: *455 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 301)
kong_1            | 2021/05/31 17:52:09 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:52:12 [debug] 44#0: *462 [lua] init.lua:288: [cluster_events] polling events from: 1622483492.591
kong_1            | 2021/05/31 17:52:15 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:52:17 [debug] 44#0: *470 [lua] init.lua:288: [cluster_events] polling events from: 1622483492.591
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] events.lua:211: do_event(): worker-events: handling event; source=dao:crud, event=update, pid=nil, data=table: 0x7ff08a0ff660
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=44, data=upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] init.lua:250: invalidate(): [DB cache] broadcasting (cluster) invalidation for key: 'upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=upstreams, pid=nil, data=table: 0x7ff08a0ff660
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=balancer, event=upstreams, pid=44, data=table: 0x7ff08a102188
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:upstreams'
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca'
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca'
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) healthchecker stopped
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] base.lua:1503: new(): [upstream:mock 4] balancer_base created
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] round_robin.lua:165: new(): [upstream:mock 4] round_robin balancer created
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] base.lua:917: newHost(): [upstream:mock 4] created a new host for: mock
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] base.lua:655: queryDns(): [upstream:mock 4] querying dns for mock
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] base.lua:570: f(): [upstream:mock 4] dns record type changed for mock, nil -> 1
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] base.lua:370: newAddress(): [upstream:mock 4] new address for host 'mock' created: 172.26.128.4:3000 (weight 100)
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] base.lua:634: f(): [upstream:mock 4] updating balancer based on dns changes for mock
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] base.lua:644: f(): [upstream:mock 4] querying dns and updating for mock completed
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Got initial target list (0 targets)
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) active check flagged as active
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Healthchecker started!
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=44, data=balancer:upstreams
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=44, data=balancer:upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=44, data=balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=clear, pid=44, data=table: 0x7ff08a0c5158
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=healthy, pid=44, data=table: 0x7ff08a0c5918
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.4:3000)'
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.4:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.4:3000)'
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.4:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 17:52:22 [warn] 44#0: *476 [lua] balancer.lua:319: [healthchecks] failed setting peer status (upstream: bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock): no peer found by name 'mock' and address 172.26.128.4:3000, client: 172.26.128.8, server: kong_admin, request: "PATCH /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca HTTP/1.1", host: "kong:8444"
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.4:3000)'
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.4:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *476 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=upstreams:update, pid=nil, data=table: 0x7ff08a0ff660
kong_1            | 172.26.128.8 - - [31/May/2021:17:52:22 +0000] "PATCH /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca HTTP/1.1" 200 964 "-" "-"
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *481 [lua] init.lua:288: [cluster_events] polling events from: 1622483492.591
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *483 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *483 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently healthy)
kong_1            | 2021/05/31 17:52:22 [debug] 44#0: *483 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 301)
kong_1            | 2021/05/31 17:52:24 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:52:27 [debug] 44#0: *491 [lua] init.lua:288: [cluster_events] polling events from: 1622483542.687
kong_1            | 2021/05/31 17:52:32 [debug] 44#0: *499 [lua] init.lua:288: [cluster_events] polling events from: 1622483542.687
kong_1            | 2021/05/31 17:52:37 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:52:37 [debug] 44#0: *507 [lua] init.lua:288: [cluster_events] polling events from: 1622483542.687
kong_1            | 2021/05/31 17:52:37 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:52:39 [debug] 44#0: *510 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:52:39 [debug] 44#0: *510 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently healthy)
kong_1            | 2021/05/31 17:52:39 [debug] 44#0: *510 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 301)
kong_1            | 2021/05/31 17:52:39 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:52:42 [debug] 44#0: *517 [lua] init.lua:288: [cluster_events] polling events from: 1622483542.687
kong_1            | 2021/05/31 17:52:52 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:52:52 [debug] 44#0: *531 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:52:52 [debug] 44#0: *531 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently healthy)
kong_1            | 2021/05/31 17:52:52 [debug] 44#0: *531 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 500)
kong_1            | 2021/05/31 17:52:52 [warn] 44#0: *531 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) unhealthy HTTP increment (1/5) for 'mock(172.26.128.4:3000)', context: ngx.timer
kong_1            | 2021/05/31 17:52:52 [debug] 44#0: *531 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=mostly_healthy, pid=44, data=table: 0x7ff08a0d81f0
kong_1            | 2021/05/31 17:52:52 [debug] 44#0: *531 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.4:3000)' from 'true' to 'true'
kong_1            | 2021/05/31 17:52:52 [debug] 44#0: *531 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.4:3000)' from 'true' to 'true'
kong_1            | 2021/05/31 17:52:52 [debug] 44#0: *531 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.4:3000)' from 'true' to 'true'
kong_1            | 2021/05/31 17:52:52 [debug] 44#0: *535 [lua] init.lua:288: [cluster_events] polling events from: 1622483542.687
kong_1            | 2021/05/31 17:52:52 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:52:52 [debug] 44#0: *537 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:52:52 [debug] 44#0: *537 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently mostly_healthy)
kong_1            | 2021/05/31 17:52:52 [debug] 44#0: *537 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 301)
kong_1            | 2021/05/31 17:52:52 [warn] 44#0: *537 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) healthy SUCCESS increment (1/2) for 'mock(172.26.128.4:3000)', context: ngx.timer
kong_1            | 2021/05/31 17:52:54 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:52:57 [debug] 44#0: *545 [lua] init.lua:288: [cluster_events] polling events from: 1622483542.687
kong_1            | 2021/05/31 17:53:02 [debug] 44#0: *553 [lua] init.lua:288: [cluster_events] polling events from: 1622483542.687
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] events.lua:211: do_event(): worker-events: handling event; source=dao:crud, event=update, pid=nil, data=table: 0x7ff08a081238
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=44, data=upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] init.lua:250: invalidate(): [DB cache] broadcasting (cluster) invalidation for key: 'upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca:::::bdd6a8cd-50a2-4650-aba2-2e2cf172dc85'
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=upstreams, pid=nil, data=table: 0x7ff08a081238
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=balancer, event=upstreams, pid=44, data=table: 0x7ff08a082f78
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:upstreams'
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca'
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] init.lua:229: invalidate_local(): [DB cache] invalidating (local): 'balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca'
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) healthchecker stopped
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] base.lua:1503: new(): [upstream:mock 5] balancer_base created
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] round_robin.lua:165: new(): [upstream:mock 5] round_robin balancer created
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] base.lua:917: newHost(): [upstream:mock 5] created a new host for: mock
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] base.lua:655: queryDns(): [upstream:mock 5] querying dns for mock
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] base.lua:570: f(): [upstream:mock 5] dns record type changed for mock, nil -> 1
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] base.lua:370: newAddress(): [upstream:mock 5] new address for host 'mock' created: 172.26.128.4:3000 (weight 100)
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] base.lua:634: f(): [upstream:mock 5] updating balancer based on dns changes for mock
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] base.lua:644: f(): [upstream:mock 5] querying dns and updating for mock completed
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Got initial target list (0 targets)
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) active check flagged as active
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Healthchecker started!
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=44, data=balancer:upstreams
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=44, data=balancer:upstreams:26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=mlcache, event=mlcache:invalidations:kong_core_db_cache, pid=44, data=balancer:targets:26c5e536-8e5f-4f5d-a291-702883dedbca
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=clear, pid=44, data=table: 0x7ff08a0985a8
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: local cache cleared
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock], event=healthy, pid=44, data=table: 0x7ff08a0989f0
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.4:3000)'
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.4:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.4:3000)'
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.4:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 17:53:04 [warn] 44#0: *556 [lua] balancer.lua:319: [healthchecks] failed setting peer status (upstream: bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock): no peer found by name 'mock' and address 172.26.128.4:3000, client: 172.26.128.8, server: kong_admin, request: "PATCH /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca HTTP/1.1", host: "kong:8444"
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target added 'mock(172.26.128.4:3000)'
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) event: target status 'mock(172.26.128.4:3000)' from 'false' to 'true'
kong_1            | 2021/05/31 17:53:04 [debug] 44#0: *556 [lua] events.lua:211: do_event(): worker-events: handling event; source=crud, event=upstreams:update, pid=nil, data=table: 0x7ff08a081238
kong_1            | 172.26.128.8 - - [31/May/2021:17:53:04 +0000] "PATCH /upstreams/26c5e536-8e5f-4f5d-a291-702883dedbca HTTP/1.1" 200 964 "-" "-"
kong_1            | 2021/05/31 17:53:07 [debug] 44#0: *565 [lua] init.lua:288: [cluster_events] polling events from: 1622483542.687
kong_1            | 2021/05/31 17:53:08 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:53:09 [debug] 44#0: *568 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:53:09 [debug] 44#0: *568 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently healthy)
kong_1            | 2021/05/31 17:53:09 [debug] 44#0: *568 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 301)
kong_1            | 2021/05/31 17:53:09 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:53:12 [debug] 44#0: *575 [lua] init.lua:288: [cluster_events] polling events from: 1622483587.779
kong_1            | 2021/05/31 17:53:17 [debug] 44#0: *583 [lua] init.lua:288: [cluster_events] polling events from: 1622483587.779
kong_1            | 2021/05/31 17:53:19 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:53:22 [debug] 44#0: *591 [lua] init.lua:288: [cluster_events] polling events from: 1622483587.779
kong_1            | 2021/05/31 17:53:23 [debug] 44#0: *593 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:53:23 [debug] 44#0: *593 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently healthy)
kong_1            | 2021/05/31 17:53:23 [debug] 44#0: *593 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 301)
kong_1            | 2021/05/31 17:53:23 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:53:24 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:53:27 [debug] 44#0: *601 [lua] init.lua:288: [cluster_events] polling events from: 1622483587.779
kong_1            | 2021/05/31 17:53:32 [debug] 44#0: *609 [lua] init.lua:288: [cluster_events] polling events from: 1622483587.779
kong_1            | 2021/05/31 17:53:34 [debug] 44#0: *612 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking healthy targets: #1
kong_1            | 2021/05/31 17:53:34 [debug] 44#0: *612 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Checking mock 172.26.128.4:3000 (currently healthy)
kong_1            | 2021/05/31 17:53:34 [debug] 44#0: *612 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) Reporting 'mock (172.26.128.4:3000)' (got HTTP 200)
kong_1            | 2021/05/31 17:53:34 [debug] 44#0: *38 [lua] healthcheck.lua:1127: log(): [healthcheck] (bdd6a8cd-50a2-4650-aba2-2e2cf172dc85:mock) checking unhealthy targets: nothing to do
kong_1            | 2021/05/31 17:53:37 [debug] 44#0: *619 [lua] init.lua:288: [cluster_events] polling events from: 1622483587.779

Kong configuration (the output of a GET request to Kong's Admin port - see
https://docs.konghq.com/latest/admin-api/#retrieve-node-information)

{
  "node_id": "41ee5847-31d2-4678-a2f0-2bac937f89b2",
  "tagline": "Welcome to kong",
  "timers": {
    "running": 1,
    "pending": 5
  },
  "pids": {
    "workers": [
      45
    ],
    "master": 1
  },
  "configuration": {
    "pg_port": 5432,
    "pg_password": "******",
    "pg_ssl": false,
    "pg_ssl_verify": false,
    "pg_max_concurrent_queries": 0,
    "pg_semaphore_timeout": 60000,
    "lua_package_path": "./?.lua;./?/init.lua;",
    "pg_ro_ssl": false,
    "pg_ro_ssl_verify": false,
    "database": "postgres",
    "cassandra_contact_points": [
      "127.0.0.1"
    ],
    "cassandra_port": 9042,
    "status_ssl_enabled": false,
    "cassandra_ssl": false,
    "cassandra_ssl_verify": false,
    "cassandra_write_consistency": "ONE",
    "status_ssl_cert_default": "/usr/local/kong/ssl/status-kong-default.crt",
    "cluster_listeners": [
      {
        "port": 8005,
        "listener": "0.0.0.0:8005",
        "proxy_protocol": false,
        "deferred": false,
        "reuseport": false,
        "backlog=%d+": false,
        "bind": false,
        "ip": "0.0.0.0",
        "ssl": false,
        "http2": false
      }
    ],
    "lua_ssl_trusted_certificate_combined": "/usr/local/kong/.ca_combined",
    "status_ssl_cert_default_ecdsa": "/usr/local/kong/ssl/status-kong-default-ecdsa.crt",
    "status_ssl_cert_key_default_ecdsa": "/usr/local/kong/ssl/status-kong-default-ecdsa.key",
    "cluster_control_plane": "127.0.0.1:8005",
    "cassandra_lb_policy": "RequestRoundRobin",
    "cluster_mtls": "shared",
    "kic": false,
    "cassandra_repl_strategy": "SimpleStrategy",
    "proxy_listen": [
      "0.0.0.0:8443\nssl"
    ],
    "admin_listen": [
      "0.0.0.0:8444 ssl"
    ],
    "cassandra_repl_factor": 1,
    "stream_listen": [
      "off"
    ],
    "nginx_http_keepalive_timeout": "0",
    "admin_ssl_cert": [
      "/etc/kong/certs/kong-cert.pem"
    ],
    "admin_ssl_cert_key": [
      "/etc/kong/certs/kong-key.pem"
    ],
    "status_ssl_cert": {},
    "status_ssl_cert_key": {},
    "headers": [
      "server_tokens",
      "latency_tokens"
    ],
    "nginx_user": "kong\nkong",
    "nginx_main_user": "kong kong",
    "nginx_daemon": "off",
    "nginx_http_ssl_prefer_server_ciphers": "off",
    "nginx_stream_ssl_prefer_server_ciphers": "off",
    "ssl_dhparam": "ffdhe2048",
    "nginx_http_gzip_disable": "msie6",
    "role": "traditional",
    "trusted_ips": {},
    "db_update_frequency": 5,
    "nginx_proxy_real_ip_header": "X-Real-IP",
    "db_update_propagation": 0,
    "nginx_http_ssl_session_timeout": "1d",
    "client_max_body_size": "0",
    "proxy_access_log": "/dev/stdout",
    "proxy_error_log": "/dev/stderr",
    "proxy_stream_access_log": "logs/access.log\nbasic",
    "proxy_stream_error_log": "logs/error.log",
    "nginx_proxy_set": "$template_root /usr/local/share/templates",
    "admin_error_log": "/dev/stderr",
    "status_access_log": "off",
    "nginx_http_large_client_header_buffers": "4 16k",
    "log_level": "debug",
    "nginx_optimizations": true,
    "lua_ssl_trusted_certificate": [
      "/etc/ssl/certs/ca-certificates.crt"
    ],
    "lua_ssl_verify_depth": 1,
    "lua_ssl_protocols": "TLSv1.1\nTLSv1.2 TLSv1.3",
    "nginx_http_lua_ssl_protocols": "TLSv1.1 TLSv1.2 TLSv1.3",
    "nginx_stream_lua_ssl_protocols": "TLSv1.1 TLSv1.2 TLSv1.3",
    "lua_socket_pool_size": 30,
    "untrusted_lua": "sandbox",
    "untrusted_lua_sandbox_requires": {},
    "untrusted_lua_sandbox_environment": {},
    "proxy_ssl_enabled": true,
    "plugins": [
      "request-transformer"
    ],
    "loaded_plugins": {
      "request-transformer": true
    },
    "cluster_ocsp": "off",
    "prefix": "/usr/local/kong",
    "ssl_cipher_suite": "intermediate",
    "upstream_keepalive_pool_size": 60,
    "nginx_http_client_body_buffer_size": "8k",
    "client_body_buffer_size": "8k",
    "db_cache_ttl": 0,
    "nginx_stream_ssl_session_tickets": "on",
    "nginx_worker_processes": "1",
    "nginx_main_worker_rlimit_nofile": "auto",
    "stream_listeners": {},
    "pluginserver_names": {},
    "lua_package_cpath": "",
    "nginx_events_multi_accept": "on",
    "nginx_proxy_error_page": "401\n402 403 405 406 407 409 410 415 416 417 418 421 422 423 424 425 426 428 429 431 451 501 505 506 507 508 510 511 /kong_error_handler",
    "mem_cache_size": "128m",
    "nginx_proxy_real_ip_recursive": "off",
    "status_listeners": {},
    "ssl_prefer_server_ciphers": "on",
    "ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
    "upstream_keepalive_idle_timeout": 60,
    "upstream_keepalive_max_requests": 100,
    "cassandra_refresh_frequency": 60,
    "go_plugins_dir": "off",
    "ssl_session_timeout": "1d",
    "ssl_cert_key_default_ecdsa": "/usr/local/kong/ssl/kong-default-ecdsa.key",
    "cassandra_schema_consensus_timeout": 10000,
    "cassandra_data_centers": [
      "dc1:2",
      "dc2:3"
    ],
    "go_pluginserver_exe": "/usr/local/bin/go-pluginserver",
    "nginx_http_types_hash_max_size": "2048",
    "port_maps": {},
    "nginx_admin_client_max_body_size": "10m",
    "cluster_data_plane_purge_delay": 1209600,
    "dns_order": [
      "LAST",
      "SRV",
      "A",
      "CNAME"
    ],
    "nginx_admin_client_body_buffer_size": "10m",
    "dns_stale_ttl": 4,
    "admin_ssl_cert_default_ecdsa": "/usr/local/kong/ssl/admin-kong-default-ecdsa.crt",
    "client_ssl": false,
    "db_resurrect_ttl": 30,
    "dns_hostsfile": "/etc/hosts",
    "ssl_cert_key": [
      "/etc/kong/certs/kong-key.pem"
    ],
    "error_default_type": "text/plain",
    "status_ssl_cert_key_default": "/usr/local/kong/ssl/status-kong-default.key",
    "host_ports": {},
    "anonymous_reports": false,
    "db_cache_warmup_entities": [
      "services"
    ],
    "cluster_listen": [
      "0.0.0.0:8005"
    ],
    "pg_host": "kong-database",
    "admin_listeners": [
      {
        "port": 8444,
        "listener": "0.0.0.0:8444\nssl",
        "proxy_protocol": false,
        "deferred": false,
        "reuseport": false,
        "backlog=%d+": false,
        "bind": false,
        "ip": "0.0.0.0",
        "ssl": true,
        "http2": false
      }
    ],
    "nginx_http_ssl_protocols": "TLSv1.2 TLSv1.3",
    "admin_ssl_cert_key_default_ecdsa": "/usr/local/kong/ssl/admin-kong-default-ecdsa.key",
    "pg_database": "kong",
    "pg_timeout": 5000,
    "admin_access_log": "/dev/stdout",
    "worker_state_update_frequency": 5,
    "admin_ssl_cert_default": "/usr/local/kong/ssl/admin-kong-default.crt",
    "ssl_cert": [
      "/etc/kong/certs/kong-cert.pem"
    ],
    "pg_user": "kong",
    "cassandra_keyspace": "kong",
    "nginx_http_sendfile": "on",
    "nginx_http_client_max_body_size": "0",
    "cassandra_read_consistency": "ONE",
    "ssl_ciphers": "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384",
    "status_error_log": "logs/status_error.log",
    "cassandra_timeout": 5000,
    "nginx_events_worker_connections": "auto",
    "nginx_http_ssl_session_tickets": "on",
    "worker_consistency": "strict",
    "ssl_protocols": "TLSv1.1\nTLSv1.2 TLSv1.3",
    "nginx_stream_ssl_dhparam": "ffdhe2048",
    "nginx_http_ssl_dhparam": "ffdhe2048",
    "nginx_main_directives": [
      {
        "name": "daemon",
        "value": "off"
      },
      {
        "name": "user",
        "value": "kong kong"
      },
      {
        "name": "worker_processes",
        "value": "1"
      },
      {
        "name": "worker_rlimit_nofile",
        "value": "auto"
      }
    ],
    "nginx_http_gzip": "on",
    "nginx_main_daemon": "off",
    "nginx_events_directives": [
      {
        "name": "multi_accept",
        "value": "on"
      },
      {
        "name": "worker_connections",
        "value": "auto"
      }
    ],
    "ssl_session_tickets": "on",
    "nginx_http_directives": [
      {
        "name": "client_body_buffer_size",
        "value": "8k"
      },
      {
        "name": "client_max_body_size",
        "value": "0"
      },
      {
        "name": "default_type",
        "value": "application/octet-stream"
      },
      {
        "name": "gzip",
        "value": "on"
      },
      {
        "name": "gzip_disable",
        "value": "msie6"
      },
      {
        "name": "keepalive_timeout",
        "value": "0"
      },
      {
        "name": "large_client_header_buffers",
        "value": "4\n16k"
      },
      {
        "name": "lua_ssl_protocols",
        "value": "TLSv1.1 TLSv1.2 TLSv1.3"
      },
      {
        "name": "sendfile",
        "value": "on"
      },
      {
        "name": "ssl_dhparam",
        "value": "/usr/local/kong/ssl/ffdhe2048.pem"
      },
      {
        "name": "ssl_prefer_server_ciphers",
        "value": "off"
      },
      {
        "name": "ssl_protocols",
        "value": "TLSv1.2\nTLSv1.3"
      },
      {
        "name": "ssl_session_tickets",
        "value": "on"
      },
      {
        "name": "ssl_session_timeout",
        "value": "1d"
      },
      {
        "name": "tcp_nodelay",
        "value": "on"
      },
      {
        "name": "tcp_nopush",
        "value": "on"
      },
      {
        "name": "types_hash_max_size",
        "value": "2048"
      }
    ],
    "enabled_headers": {
      "X-Kong-Upstream-Status": false,
      "server_tokens": true,
      "X-Kong-Proxy-Latency": true,
      "Via": true,
      "X-Kong-Response-Latency": true,
      "X-Kong-Admin-Latency": true,
      "Server": true,
      "X-Kong-Upstream-Latency": true,
      "latency_tokens": true
    },
    "nginx_http_tcp_nodelay": "on",
    "nginx_main_worker_processes": "1",
    "real_ip_header": "X-Real-IP",
    "nginx_upstream_directives": {},
    "nginx_stream_ssl_session_timeout": "1d",
    "nginx_proxy_directives": [
      {
        "name": "error_page",
        "value": "401\n402 403 405 406 407 409 410 415 416 417 418 421 422 423 424 425 426 428 429 431 451 501 505 506 507 508 510 511 /kong_error_handler"
      },
      {
        "name": "real_ip_header",
        "value": "X-Real-IP"
      },
      {
        "name": "real_ip_recursive",
        "value": "off"
      },
      {
        "name": "set",
        "value": "$template_root\n/usr/local/share/templates"
      }
    ],
    "status_listen": [
      "off"
    ],
    "nginx_status_directives": {},
    "admin_ssl_cert_key_default": "/usr/local/kong/ssl/admin-kong-default.key",
    "nginx_admin_directives": [
      {
        "name": "client_body_buffer_size",
        "value": "10m"
      },
      {
        "name": "client_max_body_size",
        "value": "10m"
      }
    ],
    "cassandra_username": "kong",
    "nginx_stream_directives": [
      {
        "name": "lua_ssl_protocols",
        "value": "TLSv1.1\nTLSv1.2 TLSv1.3"
      },
      {
        "name": "ssl_dhparam",
        "value": "/usr/local/kong/ssl/ffdhe2048.pem"
      },
      {
        "name": "ssl_prefer_server_ciphers",
        "value": "off"
      },
      {
        "name": "ssl_protocols",
        "value": "TLSv1.2 TLSv1.3"
      },
      {
        "name": "ssl_session_tickets",
        "value": "on"
      },
      {
        "name": "ssl_session_timeout",
        "value": "1d"
      }
    ],
    "nginx_http_tcp_nopush": "on",
    "nginx_supstream_directives": {},
    "dns_resolver": {},
    "nginx_sproxy_directives": {},
    "nginx_stream_ssl_protocols": "TLSv1.2\nTLSv1.3",
    "dns_error_ttl": 1,
    "dns_not_found_ttl": 30,
    "nginx_http_upstream_directives": {},
    "nginx_http_status_directives": {},
    "dns_no_sync": false,
    "nginx_pid": "/usr/local/kong/pids/nginx.pid",
    "client_ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
    "client_ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
    "nginx_err_logs": "/usr/local/kong/logs/error.log",
    "ssl_cert_default_ecdsa": "/usr/local/kong/ssl/kong-default-ecdsa.crt",
    "ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
    "nginx_acc_logs": "/usr/local/kong/logs/access.log",
    "real_ip_recursive": "off",
    "admin_acc_logs": "/usr/local/kong/logs/admin_access.log",
    "admin_ssl_enabled": true,
    "nginx_conf": "/usr/local/kong/nginx.conf",
    "stream_proxy_ssl_enabled": false,
    "nginx_kong_conf": "/usr/local/kong/nginx-kong.conf",
    "proxy_listeners": [
      {
        "port": 8443,
        "listener": "0.0.0.0:8443\nssl",
        "proxy_protocol": false,
        "deferred": false,
        "reuseport": false,
        "backlog=%d+": false,
        "bind": false,
        "ip": "0.0.0.0",
        "ssl": true,
        "http2": false
      }
    ],
    "nginx_kong_stream_conf": "/usr/local/kong/nginx-kong-stream.conf",
    "nginx_http_default_type": "application/octet-stream",
    "kong_env": "/usr/local/kong/.kong_env",
    "ssl_cert_csr_default": "/usr/local/kong/ssl/kong-default.csr"
  },
  "plugins": {
    "available_on_server": {
      "request-transformer": true
    },
    "enabled_in_cluster": [
      "request-transformer"
    ]
  },
  "version": "2.4.1",
  "hostname": "2605b5c63a4e",
  "lua_version": "LuaJIT\n2.1.0-beta3"
}

Operating system - Ubuntu Bionic

The text was updated successfully, but these errors were encountered:

GoneGo1ng · 2021-06-21T08:19:01Z

I had the same problem. #7045

akira28 · 2021-09-30T14:58:47Z

is there are workaround for this issue? We are facing something similar and it causes several issues with some ratelimited internal services

onematchfox · 2021-09-30T17:56:06Z

is there are workaround for this issue? We are facing something similar and it causes several issues with some ratelimited internal services

@akira28 what version are you running? I believe this should be resolved in a newer version of Kong than 2.4.1 although I can't confirm for certain as I'm still running an older version with my monkey-patched version of the affected code. I see the backport of the fix I put through has made it into the release notes for 2.6.0:

Healthchecks are now stopped while the Balancer is being recreated #7549.

akira28 · 2021-10-01T10:30:05Z

@onematchfox I'm trying release 2.6.0, but it seems that it still has the issue. I'll investigate further

gbulanov · 2022-03-22T23:48:25Z

Any news on this one ? I am facing the issue on Kong 2.5.1. When i update the path in the upstream it continues to send healthchecks to the old and new paths

StarlightIbuki · 2023-10-11T06:35:15Z

Dear contributor,
We're closing this issue as there hasn't been any update to it for a long time. If the issue is still relevant in the latest version, please feel free to reopen it. We're more than happy to revisit it again. Your contribution is greatly appreciated!
Please have a look at our pledge to the community for more information.
Sincerely,
Kong Gateway Team

onematchfox mentioned this issue May 31, 2021

fix(balancer) ensure existing healthchecker is stopped when processing target event #7408

Merged

javierguerragiraldez added core/balancer task/bug task/needs-investigation Requires investigation and reproduction before classifying it as a bug or not. labels Jun 21, 2021

StarlightIbuki closed this as completed Oct 11, 2023

StarlightIbuki added the stale/pending revisit Too old ticket. Closed, but we may revisit later. label Oct 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Healthchecks duplicated when updating target #7407

Healthchecks duplicated when updating target #7407

onematchfox commented May 31, 2021 •

edited

Loading

GoneGo1ng commented Jun 21, 2021

akira28 commented Sep 30, 2021

onematchfox commented Sep 30, 2021

akira28 commented Oct 1, 2021

gbulanov commented Mar 22, 2022

StarlightIbuki commented Oct 11, 2023

Healthchecks duplicated when updating target #7407

Healthchecks duplicated when updating target #7407

Comments

onematchfox commented May 31, 2021 • edited Loading

Summary

Steps To Reproduce

Additional Details & Logs

GoneGo1ng commented Jun 21, 2021

akira28 commented Sep 30, 2021

onematchfox commented Sep 30, 2021

akira28 commented Oct 1, 2021

gbulanov commented Mar 22, 2022

StarlightIbuki commented Oct 11, 2023

onematchfox commented May 31, 2021 •

edited

Loading