bug: Vault Secret Engine Issue LUA Trust Certificates

[GrayHatLabs]

Current Behavior

I am running Api Six in stand-alone mode and want to use Vault for secret management.

I am using the Docker images, and I keep getting this error. I don't know how to add certificates to the trust.

global_rules:
-
id: 1
plugins:
key-auth:
header: "Authorization"

routes:
- id: "test_route"
uri: "/test"
plugins:
key-auth: {}
upstream:
type: roundrobin
scheme: "https"
nodes:
"postb.in:443": 1

consumers:
- username: nemus_dupper
plugins:
key-auth:
key: $secret://vault/1/nemus_dupper/auth-key

secrets:
- id: vault/1
ssl_verify: false
prefix: apisix
token: hvs.asdfasdfasdfasdf
uri: https://vault.mydomain.com:8200/

api-gateway-1 | 2024/11/07 06:41:12 [error] 37#37: *1755 [lua] secret.lua:180: fetch(): failed to fetch secret value: failed to retrtive data from vault kv engine: 20: unable to get local issuer certificate, client: 172.18.0.1, server: _, request: "GET / HTTP/1.1", host: "127.0.0.1:8080"
api-gateway-1 | 2024/11/07 06:41:12 [warn] 37#37: *1755 [lua] plugin.lua:1174: run_plugin(): key-auth exits with http status code 401, client: 172.18.0.1, server: _, request: "GET / HTTP/1.1", host: "127.0.0.1:8080"

Expected Behavior

I would like it to call the vault server I've specified in the config.

Error Logs

api-gateway-1  | 2024/11/07 06:41:12 [error] 37#37: *1755 [lua] secret.lua:180: fetch(): failed to fetch secret value: failed to retrtive data from vault kv engine: 20: unable to get local issuer certificate, client: 172.18.0.1, server: _, request: "GET / HTTP/1.1", host: "127.0.0.1:8080"
api-gateway-1  | 2024/11/07 06:41:12 [warn] 37#37: *1755 [lua] plugin.lua:1174: run_plugin(): key-auth exits with http status code 401, client: 172.18.0.1, server: _, request: "GET / HTTP/1.1", host: "127.0.0.1:8080"
api-gateway-1  | 172.18.0.1 - - [07/Nov/2024:06:41:12 +0000] 127.0.0.1:8080 "GET / HTTP/1.1" 401 52 0.006 "-" "curl/8.2.1" - - - "http://127.0.0.1:8080"

Steps to Reproduce

services:
    api-gateway:
        image: apache/apisix:latest
        environment:
            - APISIX_STAND_ALONE=true
            - LUA_SSL_TRUSTED_CERTIFICATE=/usr/local/share/ca-certificates/vault-ca.crt
        volumes:
            - ${CONFIGS:-./configs}/apisix/apisix.yaml:/usr/local/apisix/conf/apisix.yaml:ro
            - ./vault_ca.crt:/usr/local/share/ca-certificates/vault-ca.crt  # Mount the CA cert into the container
        extra_hosts:
            - "vault.mydomain.com:192.168.10.60"
        ports:
            - '${LISTEN_ADDRESS:-127.0.0.1}:8080:9080'
            - '${LISTEN_ADDRESS:-127.0.0.1}:8443:9443'
        networks:
            - public
networks:
   public:
    external: true

curl -H 'Authorization:asdfasdfasdfas' -H "Content-Type: application/json" -i http://127.0.0.1:8080   

Environment

• APISIX version (run apisix version):
• Operating system (run uname -a):
• OpenResty / Nginx version (run openresty -V or nginx -V):
• etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
• APISIX Dashboard version, if relevant:
• Plugin runner version, for issues related to plugin runners:
• LuaRocks version, for installation issues (run luarocks --version):

 docker exec -it  apisix-api-gateway-1 bash                                                                                                                                                                                        ❌1 00:01
apisix@b7e90f1785f2:/usr/local/apisix$ apisix version
/usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua version
3.11.0
apisix@b7e90f1785f2:/usr/local/apisix$ uname -a
Linux b7e90f1785f2 5.15.153.1-microsoft-standard-WSL2 #1 SMP Fri Mar 29 23:14:13 UTC 2024 x86_64 GNU/Linux
apisix@b7e90f1785f2:/usr/local/apisix$ openresty -V` or `nginx -V`
> ^C
apisix@b7e90f1785f2:/usr/local/apisix$ 'penresty -V` or `nginx -V`
> ^C
apisix@b7e90f1785f2:/usr/local/apisix$ 'Openresty -V` or `nginx -V`
> ^C
apisix@b7e90f1785f2:/usr/local/apisix$ `Openresty -V` or `nginx -V`
bash: Openresty: command not found
nginx version: openresty/1.25.3.2
built by gcc 10.2.1 20210110 (Debian 10.2.1-6)
built with OpenSSL 3.2.0 23 Nov 2023
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2 -DAPISIX_RUNTIME_VER=1.2.1 -DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/zlib/include -I/usr/local/openresty/pcre/include -I/usr/local/openresty/openssl3/include' --add-module=../ngx_devel_kit-0.3.3 --add-module=../echo-nginx-module-0.63 --add-module=../xss-nginx-module-0.06 --add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.33 --add-module=../form-input-nginx-module-0.12 --add-module=../encrypted-session-nginx-module-0.09 --add-module=../srcache-nginx-module-0.33 --add-module=../ngx_lua-0.10.26 --add-module=../ngx_lua_upstream-0.07 --add-module=../headers-more-nginx-module-0.37 --add-module=../array-var-nginx-module-0.06 --add-module=../memc-nginx-module-0.20 --add-module=../redis2-nginx-module-0.15 --add-module=../redis-nginx-module-0.3.9 --add-module=../ngx_stream_lua-0.0.14 --with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib -Wl,-rpath,/usr/local/openresty/wasmtime-c-api/lib -L/usr/local/openresty/zlib/lib -L/usr/local/openresty/pcre/lib -L/usr/local/openresty/openssl3/lib -Wl,-rpath,/usr/local/openresty/zlib/lib:/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl3/lib' --add-module=/tmp/tmp.0vt0zLPiwq/openresty-1.25.3.2/../mod_dubbo-1.0.2 --add-module=/tmp/tmp.0vt0zLPiwq/openresty-1.25.3.2/../ngx_multi_upstream_module-1.2.0 --add-module=/tmp/tmp.0vt0zLPiwq/openresty-1.25.3.2/../apisix-nginx-module-1.16.1 --add-module=/tmp/tmp.0vt0zLPiwq/openresty-1.25.3.2/../apisix-nginx-module-1.16.1/src/stream --add-module=/tmp/tmp.0vt0zLPiwq/openresty-1.25.3.2/../apisix-nginx-module-1.16.1/src/meta --add-module=/tmp/tmp.0vt0zLPiwq/openresty-1.25.3.2/../wasm-nginx-module-0.7.0 --add-module=/tmp/tmp.0vt0zLPiwq/openresty-1.25.3.2/../lua-var-nginx-module-v0.5.3 --add-module=/tmp/tmp.0vt0zLPiwq/openresty-1.25.3.2/../lua-resty-events-0.2.0 --with-poll_module --with-pcre-jit --with-stream --with-stream_ssl_module --with-stream_ssl_preread_module --with-http_v2_module --with-http_v3_module --without-mail_pop3_module --without-mail_imap_module --without-mail_smtp_module --with-http_stub_status_module --with-http_realip_module --with-http_addition_module --with-http_auth_request_module --with-http_secure_link_module --with-http_random_index_module --with-http_gzip_static_module --with-http_sub_module --with-http_dav_module --with-http_flv_module --with-http_mp4_module --with-http_gunzip_module --with-threads --with-compat --with-stream --without-pcre2 --with-http_ssl_module
bash: or: command not found

apisix@b7e90f1785f2:/usr/local/apisix$ luarocks --version
bash: luarocks: command not found

[HuanXin-Chen]

This issue may will help you：#11657

Adding this config block to config.yaml：

apisix:
  ssl:
    ssl_trusted_certificate: /etc/ssl/certs/ca-certificates.crt

[GrayHatLabs]

This issue may will help you：#11657

Adding this config block to config.yaml：

apisix:
  ssl:
    ssl_trusted_certificate: /etc/ssl/certs/ca-certificates.crt

Thank you I will try this.

[GrayHatLabs]

I added this to the config.yml and confirmed that both files exist on the container. I am still seeing the same error.

Also, please note that the vault server certificate is a valid certificate signed by Godaddy, which has a CA chain that might be part of the issue.

Is there any way for me to tell APISIX to trust a certificate?

apisix:
  ssl:
    ssl_trusted_certificate: /etc/ssl/certs/ca-certificates.crt
    ssl_trusted_certificate: /usr/local/share/ca-certificates/vault-ca.crt

global_rules:
    -
        id: 1
        plugins:
            Key-auth:
                header: "Authorization"

routes:
  - id: "test_route"
    uri: "/test"
    plugins:
      key-auth: {}
    upstream:
      type: roundrobin
      scheme: "https"
      nodes:
        "postb.in:443": 1

consumers:
  - username: nemus_dupper
    plugins:
      key-auth:
        key: $secret://vault/1/nemus_dupper/auth-key

                                                                     
secrets:
  - id: vault/1
    ssl_verify: false
    prefix: apisix
    token: hvs.asdfasdfasdfasdfasdf
    uri: https://vault.mydomain.com:8200

api-gateway-1  | 2024/11/22 22:57:04 [error] 39#39: *97758 [lua] secret.lua:180: fetch(): failed to fetch secret value: failed to retrtive data from vault kv engine: 20: unable to get local issuer certificate, client: 172.18.0.1, server: _, request: "GET / HTTP/1.1", host: "127.0.0.1:8080"
api-gateway-1  | 2024/11/22 22:57:04 [warn] 39#39: *97758 [lua] plugin.lua:1174: run_plugin(): key-auth exits with http status code 401, client: 172.18.0.1, server: _, request: "GET / HTTP/1.1", host: "127.0.0.1:8080"
api-gateway-1  | 172.18.0.1 - - [22/Nov/2024:22:57:04 +0000] 127.0.0.1:8080 "GET / HTTP/1.1" 401 52 0.006 "-" "curl/8.2.1" - - - "http://127.0.0.1:8080"