THREESCALE-10675 apicast crashloop when setting large number of APICAST_SERVICES_LIST

[MStokluska]

Jira: https://issues.redhat.com/browse/THREESCALE-10675

Verification:

• checkout this branch
• Create apicast-config.json

cat <<EOF >apicast-config.json
{
    "services": [
        {
            "id": "1",
            "backend_version": "1",
            "proxy": {
                "hosts": [
                    "one"
                ],
                "api_backend": "https://echo-api.3scale.net:443",
                "backend": {
                    "endpoint": "http://127.0.0.1:8081",
                    "host": "backend"
                },
                "policy_chain": [
                    {
                        "name": "apicast.policy.apicast"
                    }
                ],
                "proxy_rules": [
                    {
                        "http_method": "GET",
                        "pattern": "/",
                        "metric_system_name": "hits",
                        "delta": 1,
                        "parameters": [],
                        "querystring_parameters": {}
                    }
                ]
            }
        }
    ]
} 
EOF

• Start development env

make development
make dependencies

• Run apicast locally

THREESCALE_DEPLOYMENT_ENV=staging APICAST_LOG_LEVEL=debug APICAST_WORKER=1 APICAST_CONFIGURATION_LOADER=lazy APICAST_CONFIGURATION_CACHE=0 APICAST_SERVICES_LIST=1,001,002,003,004,005,006,007,008,009,010,011,012,013,014,015,016,017,018,019,020,021,022,023,024,025,026,027,028,029,030,031,032,033,034,035,036,037,038,039,040,041,042,043,044,045,046,047,048,049,050,051,052,053,054,055,056,057,058,059,060,061,062,063,064,065,066,067,068,069,070,071,072,073,074,075,076,077,078,079,080,081,082,083,084,085,086,087,088,089,090,091,092,093,094,095,096,097,098,099,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,352,353,354,355,356,357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,384,385,386,387,388,389,390,391,392,393,394,395,396,397,398,399,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,419,420,421,422,423,424,425,426,427,428,429,430,431,432,433,434,435,436,437,438,439,440,441,442,443,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,461,462,463,464,465,466,467,468,469,470,471,472,473,474,475,476,477,478,479,480,481,482,483,484,485,486,487,488,489,490,491,492,493,494,495,496,497,498,499,500,501,502,503,504,505,506,507,508,509,510,511,512,513,514,515,516,517,518,519,520,521,522,523,524,525,526,527,528,529,530,531,532,533,534,535,536,537,538,539,540,541,542,543,544,545,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,565,566,567,568,569,570,571,572,573,574,575,576,577,578,579,580,581,582,583,584,585,586,587,588,589,590,591,592,593,594,595,596,597,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,615,616,617,618,619,620,621,622,623,624,625,626,627,628,629,630,631,632,633,634,635,636,637,638,639,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,662,663,664,665,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,733,734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752,753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,771,772,773,774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792,793,794,795,796,797,798,799,800,801,802,803,804,805,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,826,827,828,829,830,831,832,833,834,835,836,837,838,839,840,841,842,843,844,845,846,847,848,849,850,851,852,853,854,855,856,857,858,859,860,861,862,863,864,865,866,867,868,869,870,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937,938,939,940,941,942,943,944,945,946,947,948,949,950,951,952,953,954,955,956,957,958,959,960,961,962,963,964,965,966,967,968,969,970,971,972,973,974,975,976,977,978,979,980,981,982,983,984,985,986,987,988,989,990,991,992,993,994,995,996,997,998,999 THREESCALE_CONFIG_FILE=apicast-config.json ./bin/apicast

• Capture APIcast IP

APICAST_IP=$(docker inspect apicast_build_0-development-1 | yq e -P '.[0].NetworkSettings.Networks.apicast_build_0_default.IPAddress' -)

• Send a request

curl -i -k -H "Host: one" -H "Accept: application/json" "http://${APICAST_IP}:8080/?user_key="

The expectation is that:

1. APIcast won't crash
2. 200 response

TODO:
Integration test

[austincunningham]

Verification

• checked out this branch
• Ran the Apicast locally in a container with a long service list

THREESCALE_DEPLOYMENT_ENV=staging APICAST_LOG_LEVEL=debug APICAST_WORKER=1 APICAST_CONFIGURATION_LOADER=lazy APICAST_CONFIGURATION_CACHE=0 APICAST_SERVICES_LIST=1,001,002,003,004,005,006,007,008,009,010,011,012,013,014,015,016,017,018,019,020,021,022,023,024,025,026,027,028,029,030,031,032,033,034,035,036,037,038,039,040,041,042,043,044,045,046,047,048,049,050,051,052,053,054,055,056,057,058,059,060,061,062,063,064,065,066,067,068,069,070,071,072,073,074,075,076,077,078,079,080,081,082,083,084,085,086,087,088,089,090,091,092,093,094,095,096,097,098,099,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,352,353,354,355,356,357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,384,385,386,387,388,389,390,391,392,393,394,395,396,397,398,399,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,419,420,421,422,423,424,425,426,427,428,429,430,431,432,433,434,435,436,437,438,439,440,441,442,443,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,461,462,463,464,465,466,467,468,469,470,471,472,473,474,475,476,477,478,479,480,481,482,483,484,485,486,487,488,489,490,491,492,493,494,495,496,497,498,499,500,501,502,503,504,505,506,507,508,509,510,511,512,513,514,515,516,517,518,519,520,521,522,523,524,525,526,527,528,529,530,531,532,533,534,535,536,537,538,539,540,541,542,543,544,545,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,565,566,567,568,569,570,571,572,573,574,575,576,577,578,579,580,581,582,583,584,585,586,587,588,589,590,591,592,593,594,595,596,597,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,615,616,617,618,619,620,621,622,623,624,625,626,627,628,629,630,631,632,633,634,635,636,637,638,639,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,662,663,664,665,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,733,734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752,753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,771,772,773,774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792,793,794,795,796,797,798,799,800,801,802,803,804,805,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,826,827,828,829,830,831,832,833,834,835,836,837,838,839,840,841,842,843,844,845,846,847,848,849,850,851,852,853,854,855,856,857,858,859,860,861,862,863,864,865,866,867,868,869,870,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937,938,939,940,941,942,943,944,945,946,947,948,949,950,951,952,953,954,955,956,957,958,959,960,961,962,963,964,965,966,967,968,969,970,971,972,973,974,975,976,977,978,979,980,981,982,983,984,985,986,987,988,989,990,991,992,993,994,995,996,997,998,999 THREESCALE_CONFIG_FILE=apicast-config.json ./bin/apicast
2024/09/30 07:23:45 [notice] 1914217#1914217: using the "epoll" event method
2024/09/30 07:23:45 [notice] 1914217#1914217: openresty/1.21.4.3
2024/09/30 07:23:45 [notice] 1914217#1914217: OS: Linux 6.7.6-200.fc39.x86_64
2024/09/30 07:23:45 [notice] 1914217#1914217: getrlimit(RLIMIT_NOFILE): 1073741816:1073741816
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_VALIDATE_POLICY_CONFIGS = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: TEST_NGINX_BINARY = openresty
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_POLICY_LOAD_PATH = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_DIR = /opt/app-root/src/gateway
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_BUILTIN_POLICY_LOAD_PATH = /opt/app-root/src/gateway/src/apicast/policy
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] policy_loader.lua:150: loading policy: apicast version: builtin
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] policy_loader.lua:150: loading policy: load_configuration version: builtin
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_PATH_ROUTING = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_PATH_ROUTING_ENABLED = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_PATH_ROUTING_ONLY = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_SERVICE_CACHE_SIZE = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] evp.lua:142: openssl11=true err=nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: PWD = /opt/app-root/src
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: all_proxy = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: ALL_PROXY = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: http_proxy = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: HTTP_PROXY = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: https_proxy = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: HTTPS_PROXY = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: no_proxy = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: NO_PROXY = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_OIDC_LOG_LEVEL = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: OPENSSL_VERIFY = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_CONFIGURATION_LOADER = lazy
2024/09/30 07:23:45 [info] 1914217#1914217: *2 [lua] configuration_loader.lua:273: new(): using lazy configuration loader, context: ngx.timer
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] policy_loader.lua:150: loading policy: find_service version: builtin
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] find_service.lua:23: find_service_func(): Using host-based routing
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] policy_loader.lua:150: loading policy: local_chain version: builtin
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_CUSTOM_CONFIG = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: BACKEND_ENDPOINT_OVERRIDE = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_RESPONSE_CODES = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_REPORTING_THREADS = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_MODULE = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] policy_loader.lua:150: loading policy: apicast version: builtin
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] policy_loader.lua:150: loading policy: nginx_metrics version: builtin
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_EXTENDED_METRICS = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: NGINX_METRICS_MAX_LOGS = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] policy_loader.lua:150: loading policy: clear_context version: builtin
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: OPENTRACING_TRACER = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] policy_loader.lua:150: loading policy: load_configuration version: builtin
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] policy_loader.lua:150: loading policy: find_service version: builtin
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] find_service.lua:23: find_service_func(): Using host-based routing
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] policy_loader.lua:150: loading policy: local_chain version: builtin
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] policy_loader.lua:150: loading policy: nginx_metrics version: builtin
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] policy_loader.lua:150: loading policy: clear_context version: builtin
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] resolver.lua:115: parse_nameservers(): /etc/resolv.conf:
nameserver 127.0.0.11
options edns0 trust-ad ndots:0

2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: RESOLVER = nil
2024/09/30 07:23:45 [info] 1914217#1914217: *2 [lua] resolver.lua:161: init_nameservers(): adding 127.0.0.11:53 as default nameserver, context: ngx.timer
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] environment.lua:55: cpu_shares(): detecting cpus in Cgroups v2
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] environment.lua:84: cpus(): cpu_shares = 3
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: THREESCALE_DEPLOYMENT_ENV = staging
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_LOADED_ENVIRONMENTS = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_ENVIRONMENT = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_CONFIGURATION = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_WORKERS = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_CONFIGURATION_CACHE = 0
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_LOG_LEVEL = debug
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_LOG_FILE = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_ACCESS_LOG_FILE = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_OPENRESTY_BINARY = nil
loading staging environment configuration: /opt/app-root/src/gateway/config/staging.lua
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: SSL_CERT_FILE = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: OPENTELEMETRY = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: HTTP_KEEPALIVE_TIMEOUT = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: OPENTRACING_FORWARD_HEADER = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_PROXY_HTTPS_CERTIFICATE = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: *2 [lua] env.lua:86: fetch(): env: APICAST_UPSTREAM_RETRY_CASES = nil
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: LD_LIBRARY_PATH = :/opt/app-root/lib
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: LINES = 16
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: LUA_CPATH = /opt/app-root/src/gateway/lib/?.so;./lua_modules/lib/lua/5.1/?.so;/opt/app-root/lua_modules/lib64/lua/5.1/?.so;/opt/app-root/lua_modules/lib64/lua/5.1/?/?.so;;
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: PWD = /opt/app-root/src
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: LUA_PATH = /opt/app-root/src/gateway/src/?.lua;./lua_modules/share/lua/5.1/?.lua;./lua_modules/share/lua/5.1/?/init.lua;/usr/lib64/lua/5.1/?.lua;/usr/share/lua/5.1/?.lua;/opt/app-root/lua_modules/share/lua/5.1/?.lua;/opt/app-root/lua_modules/share/lua/5.1/?/?.lua/usr/local/openresty/lualib/?.lua;
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: PATH = ./lua_modules/bin:/usr/local/openresty/luajit/bin/:/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/app-root/src/lua_modules/bin
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: _ = ./bin/apicast
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: PROJECT_PATH = /opt/app-root/src
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: ROVER = /usr/local/openresty/luajit/bin/rover
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: SHLVL = 1
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: TERM = xterm
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: APICAST_BUILTIN_POLICY_LOAD_PATH = /opt/app-root/src/gateway/src/apicast/policy
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: TEST_NGINX_BINARY = openresty
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: APICAST_CONFIGURATION_CACHE = 0
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: APICAST_CONFIGURATION_LOADER = lazy
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: THREESCALE_CONFIG_FILE = apicast-config.json
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: APP_ROOT = /opt/app-root
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: THREESCALE_DEPLOYMENT_ENV = staging
2024/09/30 07:23:45 [debug] 1914217#1914217: env.lua:86: fetch(): env: APICAST_SERVICES_LIST = 1,001,002,003,004,005,006,007,008,009,010,011,012,013,014,015,016,017,018,019,020,021,022,023,024,025,026,027,028,029,030,031,032,033,034,035,036,037,038,039,040,041,042,043,044,045,046,047,048,049,050,051,052,053,054,055,056,057,058,059,060,061,062,063,064,065,066,067,068,069,070,071,072,073,074,075,076,077,078,079,080,081,082,083,084,085,086,087,088,089,090,091,092,093,094,095,096,097,098,099,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,206,207,208,209,210,211,212,213,214,215,216,217,218,219,220,221,222,223,224,225,226,227,228,229,230,231,232,233,234,235,236,237,238,239,240,241,242,243,244,245,246,247,248,249,250,251,252,253,254,255,256,257,258,259,260,261,262,263,264,265,266,267,268,269,270,271,272,273,274,275,276,277,278,279,280,281,282,283,284,285,286,287,288,289,290,291,292,293,294,295,296,297,298,299,300,301,302,303,304,305,306,307,308,309,310,311,312,313,314,315,316,317,318,319,320,321,322,323,324,325,326,327,328,329,330,331,332,333,334,335,336,337,338,339,340,341,342,343,344,345,346,347,348,349,350,351,352,353,354,355,356,357,358,359,360,361,362,363,364,365,366,367,368,369,370,371,372,373,374,375,376,377,378,379,380,381,382,383,384,385,386,387,388,389,390,391,392,393,394,395,396,397,398,399,400,401,402,403,404,405,406,407,408,409,410,411,412,413,414,415,416,417,418,419,420,421,422,423,424,425,426,427,428,429,430,431,432,433,434,435,436,437,438,439,440,441,442,443,444,445,446,447,448,449,450,451,452,453,454,455,456,457,458,459,460,461,462,463,464,465,466,467,468,469,470,471,472,473,474,475,476,477,478,479,480,481,482,483,484,485,486,487,488,489,490,491,492,493,494,495,496,497,498,499,500,501,502,503,504,505,506,507,508,509,510,511,512,513,514,515,516,517,518,519,520,521,522,523,524,525,526,527,528,529,530,531,532,533,534,535,536,537,538,539,540,541,542,543,544,545,546,547,548,549,550,551,552,553,554,555,556,557,558,559,560,561,562,563,564,565,566,567,568,569,570,571,572,573,574,575,576,577,578,579,580,581,582,583,584,585,586,587,588,589,590,591,592,593,594,595,596,597,598,599,600,601,602,603,604,605,606,607,608,609,610,611,612,613,614,615,616,617,618,619,620,621,622,623,624,625,626,627,628,629,630,631,632,633,634,635,636,637,638,639,640,641,642,643,644,645,646,647,648,649,650,651,652,653,654,655,656,657,658,659,660,661,662,663,664,665,666,667,668,669,670,671,672,673,674,675,676,677,678,679,680,681,682,683,684,685,686,687,688,689,690,691,692,693,694,695,696,697,698,699,700,701,702,703,704,705,706,707,708,709,710,711,712,713,714,715,716,717,718,719,720,721,722,723,724,725,726,727,728,729,730,731,732,733,734,735,736,737,738,739,740,741,742,743,744,745,746,747,748,749,750,751,752,753,754,755,756,757,758,759,760,761,762,763,764,765,766,767,768,769,770,771,772,773,774,775,776,777,778,779,780,781,782,783,784,785,786,787,788,789,790,791,792,793,794,795,796,797,798,799,800,801,802,803,804,805,806,807,808,809,810,811,812,813,814,815,816,817,818,819,820,821,822,823,824,825,826,827,828,829,830,831,832,833,834,835,836,837,838,839,840,841,842,843,844,845,846,847,848,849,850,851,852,853,854,855,856,857,858,859,860,861,862,863,864,865,866,867,868,869,870,871,872,873,874,875,876,877,878,879,880,881,882,883,884,885,886,887,888,889,890,891,892,893,894,895,896,897,898,899,900,901,902,903,904,905,906,907,908,909,910,911,912,913,914,915,916,917,918,919,920,921,922,923,924,925,926,927,928,929,930,931,932,933,934,935,936,937,938,939,940,941,942,943,944,945,946,947,948,949,950,951,952,953,954,955,956,957,958,959,960,961,962,963,964,965,966,967,968,969,970,971,972,973,974,975,976,977,978,979,980,981,982,983,984,985,986,987,988,989,990,991,992,993,994,995,996,997,998,999

• hit the endpoint and got a 200

APICAST_IP=$(docker inspect apicast_build_0-development-1 | yq e -P '.[0].NetworkSettings.Networks.apicast_build_0_default.IPAddress' -)
➜  APIcast git:(THREESCALE-10675) ✗ curl -i -k -H "Host: one" -H "Accept: application/json" "http://${APICAST_IP}:8080/?user_key="
HTTP/1.1 200 OK
Server: openresty
Date: Mon, 30 Sep 2024 07:24:38 GMT
Content-Type: application/json
Content-Length: 581
Connection: keep-alive
x-3scale-echo-api: echo-api/1.0.3
vary: Origin
x-content-type-options: nosniff
x-envoy-upstream-service-time: 1

{
  "method": "GET",
  "path": "/",
  "args": "user_key=",
  "body": "",
  "headers": {
    "HTTP_VERSION": "HTTP/1.1",
    "HTTP_HOST": "echo-api.3scale.net",
    "HTTP_X_REAL_IP": "172.22.0.1",
    "HTTP_USER_AGENT": "curl/8.2.1",
    "HTTP_ACCEPT": "application/json",
    "HTTP_X_FORWARDED_FOR": "185.94.47.128",
    "HTTP_X_FORWARDED_PROTO": "https",
    "HTTP_X_ENVOY_EXTERNAL_ADDRESS": "185.94.47.128",
    "HTTP_X_REQUEST_ID": "d7b777fd-c6ad-4683-882f-9c4eb3f1416f",
    "HTTP_X_ENVOY_EXPECTED_RQ_TIMEOUT_MS": "15000"
  },
  "uuid": "bb9c3d6c-cb10-4707-860f-7326b356c696"
}%                                      

• logs confirm the 200

754da3799ae while connecting to upstream, client: 172.22.0.1, server: _, request: "GET /?user_key= HTTP/1.1", host: "echo-api.3scale.net"
2024/09/30 07:24:37 [debug] 1914223#1914223: *4 policy_chain.lua:214: balancer(): policy chain execute phase: balancer, policy: Metrics, i: 4, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:37 [debug] 1914223#1914223: *4 policy_chain.lua:214: balancer(): policy chain execute phase: balancer, policy: Clear Context, i: 5, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 Connection Handler started for request:'0000555B82B56590' and ctx:'0000555B82CC7670'
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 SetProxyCert: no certificate was set
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 SetProxyCertKey: certificate key was not found
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 executor.lua:26: header_filter(): executor phase: header_filter, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: header_filter(): policy chain execute phase: header_filter, policy: Load Configuration, i: 1, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: header_filter(): policy chain execute phase: header_filter, policy: Find Service Policy, i: 2, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: header_filter(): policy chain execute phase: header_filter, policy: Local Policy Chain, i: 3, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: policy chain execute phase: header_filter, policy: APIcast, i: 1, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: header_filter(): policy chain execute phase: header_filter, policy: Metrics, i: 4, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: header_filter(): policy chain execute phase: header_filter, policy: Clear Context, i: 5, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 executor.lua:26: body_filter(): executor phase: body_filter, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: body_filter(): policy chain execute phase: body_filter, policy: Load Configuration, i: 1, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: body_filter(): policy chain execute phase: body_filter, policy: Find Service Policy, i: 2, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: body_filter(): policy chain execute phase: body_filter, policy: Local Policy Chain, i: 3, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: policy chain execute phase: body_filter, policy: APIcast, i: 1, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: body_filter(): policy chain execute phase: body_filter, policy: Metrics, i: 4, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: body_filter(): policy chain execute phase: body_filter, policy: Clear Context, i: 5, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 executor.lua:26: body_filter(): executor phase: body_filter, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: body_filter(): policy chain execute phase: body_filter, policy: Load Configuration, i: 1, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: body_filter(): policy chain execute phase: body_filter, policy: Find Service Policy, i: 2, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: body_filter(): policy chain execute phase: body_filter, policy: Local Policy Chain, i: 3, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: policy chain execute phase: body_filter, policy: APIcast, i: 1, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: body_filter(): policy chain execute phase: body_filter, policy: Metrics, i: 4, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: body_filter(): policy chain execute phase: body_filter, policy: Clear Context, i: 5, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 executor.lua:26: post_action(): executor phase: post_action, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: post_action(): policy chain execute phase: post_action, policy: Load Configuration, i: 1, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: post_action(): policy chain execute phase: post_action, policy: Find Service Policy, i: 2, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: post_action(): policy chain execute phase: post_action, policy: Local Policy Chain, i: 3, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: policy chain execute phase: post_action, policy: APIcast, i: 1, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [info] 1914223#1914223: *4 proxy.lua:337: [async] skipping after action, no cached key, requestID=8436eadc759f4606a887a754da3799ae while sending to client, client: 172.22.0.1, server: _, request: "GET /?user_key= HTTP/1.1", upstream: "https://3.213.166.152:443/?user_key=", host: "echo-api.3scale.net"
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: post_action(): policy chain execute phase: post_action, policy: Metrics, i: 4, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: post_action(): policy chain execute phase: post_action, policy: Clear Context, i: 5, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 executor.lua:26: log(): executor phase: log, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: log(): policy chain execute phase: log, policy: Load Configuration, i: 1, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: log(): policy chain execute phase: log, policy: Find Service Policy, i: 2, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: log(): policy chain execute phase: log, policy: Local Policy Chain, i: 3, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: policy chain execute phase: log, policy: APIcast, i: 1, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: log(): policy chain execute phase: log, policy: Metrics, i: 4, requestID=8436eadc759f4606a887a754da3799ae
2024/09/30 07:24:38 [debug] 1914223#1914223: *4 policy_chain.lua:214: log(): policy chain execute phase: log, policy: Clear Context, i: 5, requestID=8436eadc759f4606a887a754da3799ae
[30/Sep/2024:07:24:38 +0000] one:8080 172.22.0.1:34766 "GET /?user_key= HTTP/1.1" 200 581 (0.348) 0
2024/09/30 07:24:38 [info] 1914223#1914223: *4 client 172.22.0.1 closed keepalive connection
2024/09/30 07:24:45 [notice] 1914227#1914227: http file cache: /tmp/cache 0.000M, bsize: 4096
2024/09/30 07:24:45 [notice] 1914217#1914217: signal 17 (SIGCHLD) received from 1914227
2024/09/30 07:24:45 [notice] 1914217#1914217: cache loader process 1914227 exited with code 0
2024/09/30 07:24:45 [notice] 1914217#1914217: signal 29 (SIGIO) received
2024/09/30 07:25:37 [info] 1914224#1914224: *6 client 127.0.0.1 closed keepalive connection

[tkan145]

Prove tests failed.

For example:

diff --git a/t/configuration-loading-lazy.t b/t/configuration-loading-lazy.t
index 80abf384..56901d80 100644
--- a/t/configuration-loading-lazy.t
+++ b/t/configuration-loading-lazy.t
@@ -163,6 +163,7 @@ OIDC url is not valid, uri:
 === TEST 6: Load paginated proxy config list
 This test is configured to provide 3 pages of proxy configs.
 The test will try requests with hostnames existing a different pages
+--- ONLY
 --- env eval
 (
   'APICAST_CONFIGURATION_LOADER' => 'lazy',

And run

make prove PROVE_FILES=./t/configuration-loading-lazy.t

Result:

/usr/local/openresty/luajit/bin/rover exec script/prove --verbose -j16 --harness=TAP::Harness ./t/configuration-loading-lazy.t
# I found ONLY: maybe you're debugging?                                                                                       
nginx: [warn] conflicting server name "_" on 0.0.0.0:8282, ignored                                                            
#   Failed test 'TEST 6: Load paginated proxy config list - status code ok'                                                   
#   at /opt/app-root/src/local/lib/perl5/Test/Nginx/Socket.pm line 922.                                                       
#          got: '404'                                                                                                         
#     expected: '200'                                                                                                         
#   Failed test 'TEST 6: Load paginated proxy config list - response_body - response is expected (repeated req 0, req 0)'     
#   at /opt/app-root/src/local/lib/perl5/Test/Nginx/Socket.pm line 1387.                                                      
#          got: ""                                                                                                            
#       length: 0                                                                                                             
#     expected: "yay, api backend\x{0a}"                                                                                      
#       length: 17                                                                                                            
#     strings begin to differ at char 1 (line 1 column 1)                                                                     
#   Failed test 'TEST 6: Load paginated proxy config list - status code ok'                                                   
#   at /opt/app-root/src/local/lib/perl5/Test/Nginx/Socket.pm line 922.                                                       
#          got: '404'                                                                                                         
#     expected: '200'                                                                                                         
#   Failed test 'TEST 6: Load paginated proxy config list - response_body - response is expected (repeated req 0, req 1)'     
#   at /opt/app-root/src/local/lib/perl5/Test/Nginx/Socket.pm line 1387.                                                      
#          got: ""                                                                                                            
#       length: 0                                                                                                             
#     expected: "yay, api backend\x{0a}"                                                                                      
#       length: 17                                                                                                            
#     strings begin to differ at char 1 (line 1 column 1)                                                                     
#   Failed test 'TEST 6: Load paginated proxy config list - status code ok'                                                   
#   at /opt/app-root/src/local/lib/perl5/Test/Nginx/Socket.pm line 922.                                                       
#          got: '404'                                                                                                         
#     expected: '200'                                                                                                         
#   Failed test 'TEST 6: Load paginated proxy config list - response_body - response is expected (repeated req 0, req 2)'     
#   at /opt/app-root/src/local/lib/perl5/Test/Nginx/Socket.pm line 1387.                                                      
#          got: ""                                                                                                            
#       length: 0                                                                                                             
#     expected: "yay, api backend\x{0a}"                                                                                      
#       length: 17                                                                                                            
#     strings begin to differ at char 1 (line 1 column 1)                                                                     
# Looks like you failed 6 tests of 6.                                                                                         
./t/configuration-loading-lazy.t ..                                                                                           
not ok 1 - TEST 6: Load paginated proxy config list - status code ok                                                          
not ok 2 - TEST 6: Load paginated proxy config list - response_body - response is expected (repeated req 0, req 0)            
not ok 3 - TEST 6: Load paginated proxy config list - status code ok                                                          
not ok 4 - TEST 6: Load paginated proxy config list - response_body - response is expected (repeated req 0, req 1)            
not ok 5 - TEST 6: Load paginated proxy config list - status code ok                                                          
not ok 6 - TEST 6: Load paginated proxy config list - response_body - response is expected (repeated req 0, req 2)            

---

Env var available with your patch

"THREESCALE_PORTAL_ENDPOINT" "http://test:13462"
"APICAST_SERVICE_CACHE_SIZE" "2000"
"APICAST_LOADED_ENVIRONMENTS" "production|/tmp/GRbo5hpiDG"
"APICAST_DIR" "/opt/app-root/src/gateway"
"APICAST_CONFIGURATION_LOADER" "lazy"
"APICAST_OPENRESTY_BINARY" "openresty"

Without your patch, in addition to the env variables returned by resty_env:list, it would set these additional variables.

"THREESCALE_DEPLOYMENT_ENV" "production"
"APICAST_BUILTIN_POLICY_LOAD_PATH" "/opt/app-root/src/gateway/src/apicast/policy"
"THREESCALE_CONFIG_FILE" ""
"APICAST_CONFIGURATION_CACHE" "300"
"APICAST_POLICY_LOAD_PATH" "/opt/app-root/src/gateway/policies"

The most important one is APICAST_BUILTIN_POLICY_LOAD_PATH, if you rerun the step above with APICAST_BUILTIN_POLICY_LOAD_PATH the test will pass

APICAST_BUILTIN_POLICY_LOAD_PATH=/opt/app-root/src/gateway/src/apicast/policy
 make prove PROVE_FILES=./t/configuration-loading-lazy.t

The env var is set here https://github.com/3scale/APIcast/blob/master/gateway/bin/apicast#L58. So oneway to fix is read this from ENV and save it to resty_env.

Why?

When you run

   APICAST_LOG_LEVEL=debug \                   
   APICAST_WORKERS=1 \                         
   THREESCALE_DEPLOYMENT_ENV=staging \         
   APICAST_SERVICES_LIST="41" \                
   THREESCALE_CONFIG_FILE=apicast-config.json \
   ./bin/apicast                               

It translates to the following

/usr/bin/perl /usr/bin/resty --nginx openresty --errlog-level debug -I /opt/app-root/src/gateway/src /opt/app-root/src/gateway/bin/apicast_cli.lua

If you follow that /bin/apicast file, you can see that its actually set some env var and then passed it to apicast_cli.lua file. So resty_env.list() will contain everything

 ENV: {                                                                                                                                                                                                        
  APICAST_BUILTIN_POLICY_LOAD_PATH = "/opt/app-root/src/gateway/src/apicast/policy",                                                                                                                           
  APICAST_CONFIGURATION_LOADER = "lazy",                                                                                                                                                                       
  APICAST_DIR = "/opt/app-root/src/gateway",                                                                                                                                                                   
  APICAST_LOADED_ENVIRONMENTS = "/opt/app-root/src/gateway/config/staging.lua",                                                                                                                                
  APICAST_LOG_LEVEL = "debug",                                                                                                                                                                                 
  APICAST_POLICY_LOAD_PATH = "/opt/app-root/src/gateway/policies",                                                                                                                                             
  APICAST_SERVICES_LIST = "41",                                                                                                                                                                                
  APICAST_WORKERS = "1",                                                                                                                                                                                       
  APP_ROOT = "/opt/app-root",                                                                                                                                                                                  
  ARGV0 = "./bin/apicast",                                                                                                                                                                                     
  COLUMNS = "86",                                                                                                                                                                                              
  EDITOR = "vi",                                                                                                                                                                                               
  GIT_COMMITTER_EMAIL = '""',                                                                                                                                                                                  
  GIT_COMMITTER_NAME = "4211872",                                                                                                                                                                              
  HOME = "/opt/app-root/src/",                                                                                                                                                                                 
  HOSTNAME = "53e67f6dafd3",                                                                                                                                                                                   
  LD_LIBRARY_PATH = ":/opt/app-root/lib",                                                                                                                                                                      
  LINES = "64",                                                                                                                                                                                                
  LUA_CPATH = "/opt/app-root/src/gateway/lib/?.so;./lua_modules/lib/lua/5.1/?.so;/opt/app-root/lua_modules/lib64/lua/5.1/?.so;/opt/app-root/lua_modules/lib64/lua/5.1/?/?.so;;",                               
  LUA_PATH = "/opt/app-root/src/gateway/src/?.lua;./lua_modules/share/lua/5.1/?.lua;./lua_modules/share/lua/5.1/?/init.lua;/usr/lib64/lua/5.1/?.lua;/usr/share/lua/5.1/?.lua;/opt/app-root/lua_modules/share/lu
a/5.1/?.lua;/opt/app-root/lua_modules/share/lua/5.1/?/?.lua/usr/local/openresty/lualib/?.lua;",                                                                                                                
  OLDPWD = "/tmp",                                                                                                                                                                                             
  PATH = "./lua_modules/bin:/usr/local/openresty/luajit/bin/:/opt/app-root/src/bin:/opt/app-root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/app-root/src/lua_modules/bin",          
  PLATFORM = "el8",                                                                                                                                                                                            
  PROJECT_PATH = "/opt/app-root/src",                                                                                                                                                                          
  PWD = "/opt/app-root/src",                                                                                                                                                                                   
  ROVER = "/usr/local/openresty/luajit/bin/rover",                                                                                                                                                             
  SHLVL = "2",                                                                                                                                                                                                 
  TERM = "xterm",                                                                                                                                                                                              
  TEST_NGINX_APICAST_PATH = "/opt/app-root/src/gateway",                                                                                                                                                       
  TEST_NGINX_BINARY = "openresty",                                                                                                                                                                             
  TEST_NGINX_REDIS_HOST = "redis",                                                                                                                                                                             
  THREESCALE_CONFIG_FILE = "apicast-config.json",                                                                                                                                                              
  THREESCALE_DEPLOYMENT_ENV = "staging",                                                                                                                                                                       
  _ = "./bin/apicast",                                                                                                                                                                                         
  container = "oci"                                                                                                                                                                                            
}                                                                                                                                                                                                              

However it's a bit different when you run make prove PROVE_FILES=./t/configuration-loading-lazy.t.

First it will render the config file (https://github.com/3scale/Test-APIcast/blob/master/lib/Test/APIcast/Blackbox.pm#L208) with the following command:

APICAST_CONFIGURATION_LOADER='test' bin/apicast start --test --environment production --environment /tmp/3N_x3CAksE

And then it will start the test (https://github.com/openresty/test-nginx/blob/master/lib/Test/Nginx/Util.pm#L2032) with the following:

openresty -p /opt/app-root/src/t/servroot_12630/ -c /opt/app-root/src/t/servroot_12630/conf/nginx.conf > /dev/null

At this point all env var set by apicast start is not available anymore, so resty_env.list() will only returned the global env var. And I think this is why we wrote those env variables into the config file in the first place.