fix(healthcheck) Use pair instead ipair for hcs weak table

[kayano]

This PR fixes an issue with active healthchecks being disabled after sometime when targets are being removed/added to an upstream in kong.

This PR is based on #78 but for lua-resty-healthcheck version 1.5.0 used in kong 2.8.0.

Steps to reproduce the issue.

1. In a new shell run docker-compose up with following docker-compose.yml

version: '3.7'

volumes:
  kong_data: {}

networks:
  kong-net:
    external: false

services:
  kong:
    image: kong:2.8.0
    depends_on:
      - db
    environment:
      KONG_LOG_LEVEL: debug
      KONG_NGINX_DAEMON: "off"
      KONG_ADMIN_LISTEN: 0.0.0.0:8001
      KONG_ADMIN_ACCESS_LOG: /dev/stdout
      KONG_ADMIN_ERROR_LOG: /dev/stderr
      KONG_PROXY_LISTEN: 0.0.0.0:8000
      KONG_PROXY_ACCESS_LOG: /dev/stdout
      KONG_PROXY_ERROR_LOG: /dev/stderr
      KONG_NGINX_WORKER_PROCESSES: "2"
      KONG_DATABASE: postgres
      KONG_PG_DATABASE: kong
      KONG_PG_HOST: db
      KONG_PG_USER: kong
      KONG_PG_PASSWORD: kong
    ports:
      - 8000:8000
      - 8001:8001
    healthcheck:
      test: ["CMD", "kong", "health"]
      interval: 10s
      timeout: 10s
      retries: 10
    restart: on-failure
    networks:
      - kong-net

  kong-migrations:
    image: kong:2.8.0
    command: kong migrations bootstrap
    environment:
      KONG_DATABASE: postgres
      KONG_PG_DATABASE: kong
      KONG_PG_HOST: db
      KONG_PG_USER: kong
      KONG_PG_PASSWORD: kong
    depends_on:
      - db
    restart: on-failure
    networks:
      - kong-net

  kong-migrations-up:
    image: kong:2.8.0
    command: kong migrations up && kong migrations finish
    environment:
      KONG_DATABASE: postgres
      KONG_PG_DATABASE: kong
      KONG_PG_HOST: db
      KONG_PG_USER: kong
      KONG_PG_PASSWORD: kong
    depends_on:
      - db
    restart: on-failure
    networks:
      - kong-net

  db:
    image: postgres:12
    environment:
      POSTGRES_DB: kong
      POSTGRES_USER: kong
      POSTGRES_PASSWORD: kong
    healthcheck:
      interval: 5s
      retries: 10
      test:
      - CMD
      - pg_isready
      - --dbname=kong
      - --username=kong
      timeout: 10s
    stop_signal: SIGKILL
    restart: on-failure
    networks:
      - kong-net
    volumes:
      - kong_data:/var/lib/postgresql/data

It starts a single kong node and a postgres db.

2. In a new shell setup a kong service, a route and an upstream with 2 targets running (in the directory with docker-compose.yml from step 1):

#!/usr/bin/env bash

curl -s -X POST http://localhost:8001/services \
     --data name=example_service \
     --data host='example_upstream'

curl -s -X POST http://localhost:8001/services/example_service/routes \
     --data 'paths[]=/mock' \
     --data name=mocking

curl -s -X POST http://localhost:8001/upstreams \
     --data name=example_upstream

curl -s -X POST http://localhost:8001/upstreams/example_upstream/targets \
     --data target='mockbin.org:80'

curl -s -X POST http://localhost:8001/upstreams/example_upstream/targets \
     --data target='httpbin.org:80'


curl -X PATCH -H 'content-type: application/json' http://localhost:8001/upstreams/example_upstream \
     -d @- << EOF
{
  "healthchecks": {
    "active": {
      "healthy": {
        "interval": 5,
        "successes": 2
      },
      "unhealthy": {
        "http_failures": 3,
        "interval": 2,
        "timeouts": 5
      }
    },
    "passive": {
      "unhealthy": {
        "http_failures": 3,
        "timeouts": 5
      },
      "healthy": {
        "successes": 2
      }
    }
  }
}
EOF

At this point active healthchecks should be running for both targets and in the shell from step 1 should periodically appear logs like

kong_1                | 2022/03/09 21:17:13 [debug] 1111#0: *439 [lua] healthcheck.lua:1235: log(): [healthcheck] (0fdb1f6d-5629-472b-ac71-8680db1ae231:example_upstream) Reporting 'mockbin.org (172.67.201.247:80)' (got HTTP 200)
kong_1                | 2022/03/09 21:17:13 [debug] 1111#0: *439 [lua] healthcheck.lua:1235: log(): [healthcheck] (0fdb1f6d-5629-472b-ac71-8680db1ae231:example_upstream) Reporting 'mockbin.org (104.21.60.227:80)' (got HTTP 200)
kong_1                | 2022/03/09 21:17:13 [debug] 1111#0: *439 [lua] healthcheck.lua:1235: log(): [healthcheck] (0fdb1f6d-5629-472b-ac71-8680db1ae231:example_upstream) Reporting 'httpbin.org (54.221.78.73:80)' (got HTTP 200)
kong_1                | 2022/03/09 21:17:13 [debug] 1111#0: *439 [lua] healthcheck.lua:1235: log(): [healthcheck] (0fdb1f6d-5629-472b-ac71-8680db1ae231:example_upstream) Reporting 'httpbin.org (52.55.211.119:80)' (got HTTP 200)
kong_1                | 2022/03/09 21:17:13 [debug] 1111#0: *439 [lua] healthcheck.lua:1235: log(): [healthcheck] (0fdb1f6d-5629-472b-ac71-8680db1ae231:example_upstream) Reporting 'httpbin.org (52.86.136.68:80)' (got HTTP 200)
kong_1                | 2022/03/09 21:17:13 [debug] 1111#0: *439 [lua] healthcheck.lua:1235: log(): [healthcheck] (0fdb1f6d-5629-472b-ac71-8680db1ae231:example_upstream) Reporting 'httpbin.org (54.85.66.171:80)' (got HTTP 200)
kong_1                | 2022/03/09 21:17:13 [debug] 1111#0: *439 [lua] healthcheck.lua:1235: log(): [healthcheck] (0fdb1f6d-5629-472b-ac71-8680db1ae231:example_upstream) Reporting 'httpbin.org (3.209.99.235:80)' (got HTTP 200)
kong_1                | 2022/03/09 21:17:13 [debug] 1111#0: *439 [lua] healthcheck.lua:1235: log(): [healthcheck] (0fdb1f6d-5629-472b-ac71-8680db1ae231:example_upstream) Reporting 'httpbin.org (34.227.133.25:80)' (got HTTP 200)
kong_1                | 2022/03/09 21:17:14 [debug] 1111#0: *119 [lua] healthcheck.lua:1235: log(): [healthcheck] (0fdb1f6d-5629-472b-ac71-8680db1ae231:example_upstream) checking unhealthy targets: nothing to do

indicating that active healthchecks are working.

3. To quicker reproduce the issue run:

docker-compose exec -u 0 kong bash -c "sed -i '1590 i collectgarbage("collect")' /usr/local/share/lua/5.1/resty/healthcheck.lua"

It adds a line to run one complete cycle of garbage collection.

4. Reload kong after library change running:

docker-compose exec kong bash -c "kong reload"

5. When kong is reloaded and working again delete one target running:

curl -X DELETE "localhost:8001/upstreams/example_upstream/targets/httpbin.org:80"

6. Looking at logs active healthchecks are disabled now for all targets in example_upstream but they shouldn't.
   When passive healthcheck fails for all targets in the example_upstream kong returns eventually

{"message":"failure to get a peer from the ring-balancer"}

even after targets are healthy again.

Changing ipairs to pairs while iterating over hcs table fixes the issue.

[CLAassistant]

All committers have signed the CLA.

[kayano]

@locao as you released lua-resty-healthcheck 1.5.0 can you help to have my change released? What needs to be done to have the library bump in kong also if my change makes sense?

[locao]

Hi @kayano! Thanks for your contribution!

I will release a new version of this module with your fix. For the Kong side, we need to bump the dependency and also release a new version.