deadlock on table cluster_events

[luo985]

hello, when i change route or targets frequently by admin api, then found deadlock on the admin_log:

2019/08/12 14:43:29 [error] 21332#0: *795492 [lua] cache.lua:315: invalidate(): [DB cache] failed to broadcast cached entity invalidation: could not insert invalidation row: ERROR: deadlock detected Process 26420 waits for ShareLock on transaction 53508147; blocked by process 23771. Process 23771 waits for ShareLock on transaction 53507968; blocked by process 20058. Process 20058 waits for AccessExclusiveLock on tuple (1878,2) of relation 41571 of database 16682; blocked by process 26420., client: 127.0.0.1, server: kong_admin, request: "DELETE /upstreams/upstream.cl.dev.1.modify-configuration-test71.system/targets/172.24.140.49:8103 HTTP/1.1", host: "apigw.kong.mng.ucm9tioz0htbll1u" 2019/08/12 14:43:30 [error] 21333#0: *793042 [lua] handler.lua:272: failed broadcasting target create to cluster: could not insert invalidation row: ERROR: deadlock detected Process 15074 waits for ShareLock on transaction 53508147; blocked by process 23771. Process 23771 waits for ShareLock on transaction 53507968; blocked by process 20058. Process 20058 waits for AccessExclusiveLock on tuple (1878,2) of relation 41571 of database 16682; blocked by process 15074., client: 127.0.0.1, server: kong_admin, request: "POST /upstreams/upstream.cl.dev.1.modify-configuration-test53.system/targets HTTP/1.1", host: "apigw.kong.mng.ucm9tioz0htbll1u" 2019/08/12 14:43:30 [error] 21327#0: *793591 [lua] cache.lua:315: invalidate(): [DB cache] failed to broadcast cached entity invalidation: could not insert invalidation row: receive_message: failed to get type: timeout, client: 127.0.0.1, server: kong_admin, request: "POST /upstreams/upstream.cl.dev.1.modify-configuration-test12.system/targets HTTP/1.1", host: "apigw.kong.mng.ucm9tioz0htbll1u" 2019/08/12 14:43:31 [error] 21330#0: *794451 [lua] cache.lua:315: invalidate(): [DB cache] failed to broadcast cached entity invalidation: could not insert invalidation row: ERROR: deadlock detected

i remove the TRIGGER（delete_expired_cluster_events） on table cluster_events, then issues did not occur。

Additional Details & Logs

• Kong version (1.2.0)

[bungle]

@luo985 thanka for the report!

[bungle]

@luo985 mind you try: #5118

As you already removed the trigger all you need to do is this:
https://github.com/Kong/kong/pull/5118/files#diff-126f4d3eab7dd01ac7ab193d1eb7f847

And reload your kong. Can you please report us if this solves the issue?

[luo985]

@bungle
yes， it solves the issue

[bungle]

@luo985, the pr got merged, so closing this. thank you for trying it out!