THREESCALE-9003 in boot mode on worker init check configuration expiration

[eguzki]

What

Fixes https://issues.redhat.com/browse/THREESCALE-9003

Note; no test provided. boot.init and boot.init_worker are hard to test and they are not tested. Manual test required.

Verification steps

Test 1 === Re-spawn nginx worker before TTL expired

• Run development environment

make development
make dependencies

• Start APIcast in development mode boot with a non-zero TTL (i.e. 30s) and 1 worker

APICAST_LOG_LEVEL=info APICAST_WORKERS=1 THREESCALE_PORTAL_ENDPOINT=https://ACCESS_TOKEN@3scale-admin.3scale-domain BACKEND_ENDPOINT_OVERRIDE=http://localhost:8081 APICAST_CONFIGURATION_LOADER=boot APICAST_CONFIGURATION_CACHE=30 THREESCALE_DEPLOYMENT_ENV=staging ./bin/apicast

• Check logs for the initial configuration being loaded

2023/03/24 14:44:41 [info] 48229#48229: [lua] configuration_store.lua:132: store(): added service 1163 configuration with hosts: b.com ttl: 30

• Before the TTL expires, kill the worker process. In other terminal:

docker exec -ti -u root apicast_build_0-development-1 /bin/bash
bash-4.4# kill $(ls -l /proc/*/exe | grep nginx | sed -n '2p' | awk -F/ '{print $3}')

• Check APIcast logs. Configuration should not be immediately be re-loaded. It will wait the TTL time to re-load the configuration

Test 2 === Re-spawn nginx worker after TTL expired

• Get any example product (echo API for example) working and add a HTTP Status Code Overwrite policy to change the return code to 200 (used only for debug purposes)
• Run development environment

make development
make dependencies

• Start APIcast in development mode boot with a non-zero TTL (i.e. 30s) and 1 worker

APICAST_LOG_LEVEL=info APICAST_WORKERS=1 THREESCALE_PORTAL_ENDPOINT=https://ACCESS_TOKEN@3scale-admin.3scale-domain BACKEND_ENDPOINT_OVERRIDE=http://localhost:8081 APICAST_CONFIGURATION_LOADER=boot APICAST_CONFIGURATION_CACHE=30 THREESCALE_DEPLOYMENT_ENV=staging ./bin/apicast

• Check if the rule is working and returning the configured status code

curl -v -H "Host: b.com" http://${APICAST_IP}:8080/?user_key=123456
< HTTP/1.1 200 OK
< Server: openresty
....

The user_key value is irrelevant because the backend endpoint is overridden. The host must match the 3scale product host.

• Changed the policy HTTP Status Code Overwrite of Response Code to return 201
• Manually promote to production and staging
• Wait for the loop response code to present the publication result 201

curl -v -H "Host: b.com" http://${APICAST_IP}:8080/?user_key=123456
< HTTP/1.1 201 OK
< Server: openresty

• Kill the worker process. Make sure TTL secs have passed since apicast boot time. In other terminal:

docker exec -ti -u root apicast_build_0-development-1 /bin/bash
bash-4.4# kill $(ls -l /proc/*/exe | grep nginx | sed -n '2p' | awk -F/ '{print $3}')

• After killing the worker process, before TTL expires, run request to verify the new response code 201 is returned indicating that the NGiNX process spawned a child and the child immediately loaded new configuration and did not use the one loaded at the apicast boot time.

curl -v -H "Host: b.com" http://${APICAST_IP}:8080/?user_key=123456
< HTTP/1.1 201 OK
< Server: openresty

• APIcast logs should also show the killing of the worker process and immediate configuration load

2023/03/24 15:07:22 [notice] 49339#49339: signal 15 (SIGTERM) received from 48450, exiting
2023/03/24 15:07:22 [info] 49339#49339: epoll_wait() failed (4: Interrupted system call)
2023/03/24 15:07:22 [notice] 49339#49339: exiting
2023/03/24 15:07:22 [notice] 49339#49339: exit
2023/03/24 15:07:22 [notice] 49326#49326: signal 17 (SIGCHLD) received from 49339
2023/03/24 15:07:22 [notice] 49326#49326: worker process 49339 exited with code 0
2023/03/24 15:07:22 [notice] 49326#49326: start worker process 49875
2023/03/24 15:07:22 [notice] 49326#49326: signal 29 (SIGIO) received
2023/03/24 15:07:22 [info] 49875#49875: *72 [lua] configuration_loader.lua:215: init_worker(): boot time configuration has expired, context: init_worker_by_lua*
2023/03/24 15:07:22 [info] 49875#49875: *73 [lua] configuration_loader.lua:194: auto updating configuration, context: ngx.timer
2023/03/24 15:07:22 [info] 49875#49875: *73 [lua] configuration_store.lua:132: store(): added service 1163 configuration with hosts: b.com ttl: 30, context: ngx.timer

The log lines show that within one second, the worker process is re-spawned and configuration is loaded.

[ernaniaz]

Tested on a quicklab and solved the issue.