bug: when using GCP secret manager with ssl_verify set to true, unable to get local issuer certificate even when ssl_trusted_certificate is configured

[kayx23]

Current Behavior

When integrating with GCP secret manager for secret fetching with ssl_trusted_certificate configured, if ssl_verify is set to true (default), you will see a 401 Unauthorized error with the following in error log:

[error] 213#213: *73737 [lua] google-cloud-oauth.lua:62: refresh_access_token(): failed to refresh google oauth access token, 20: unable to get local issuer certificate

If ssl_verify is set to false, the feature works correctly.

Expected Behavior

Not error, 200 OK

Error Logs

2024-10-16 13:51:42 2024/10/16 05:51:42 [error] 213#213: *73737 [lua] google-cloud-oauth.lua:62: refresh_access_token(): failed to refresh google oauth access token, 20: unable to get local issuer certificate, client: 192.168.65.1, server: _, request: "GET /anything HTTP/1.1", host: "127.0.0.1:9080"
2024-10-16 13:51:42 2024/10/16 05:51:42 [error] 213#213: *73737 [lua] secret.lua:180: fetch(): failed to fetch secret value: failed to retrtive data from gcp secret manager: failed to get google oauth token, client: 192.168.65.1, server: _, request: "GET /anything HTTP/1.1", host: "127.0.0.1:9080"
2024-10-16 13:51:42 2024/10/16 05:51:42 [warn] 213#213: *73737 [lua] plugin.lua:1171: run_plugin(): key-auth exits with http status code 401, client: 192.168.65.1, server: _, request: "GET /anything HTTP/1.1", host: "127.0.0.1:9080"

Steps to Reproduce

Set up APISIX 3.11.0 (currently not released and the feature is available on master).

Update config.yaml with the following config:

apisix:
  ssl:
    ssl_trusted_certificate: /etc/ssl/certs/ca-certificates.crt

Create a secret on GCP. Say the name is apisix-jack-key-auth and secret is jack-key.

Create a service account on GCP, assign the account with proper role, and get its credentials in JSON format.

Configure a GCP secret provider in APISIX for a sample user jack:

curl "http://127.0.0.1:9180/apisix/admin/secrets/gcp/jack" -X PUT -d '
{
  "auth_config": 
  {
    "client_email": "xxxx@xxxx.iam.gserviceaccount.com",
    "private_key": "-----BEGIN PRIVATE KEY----\n.....n-----END PRIVATE KEY-----\n",
    "project_id": "xxxx",
    "token_uri": "https://oauth2.googleapis.com/token"
  },
  "ssl_verify": true
}'

Here ssl_verify is explicitly set to true but this is also the default configuration, true if not set.

Create a consumer and enable key-auth. Update the key with reference to the secret on GCP:

curl "http://127.0.0.1:9180/apisix/admin/consumers" -X PUT -d '
{
  "username": "jack",
  "plugins": {
    "key-auth": {
      "key": "$secret://gcp/jack/apisix-jack-key-auth"
    }
  }
}'

Enable the key-auth plugin on a route, say /anything.

Send a request to the route with the valid credential:

curl -i "http://127.0.0.1:9080/anything" -H 'apikey: jack-key'

Expecting 200 OK but receive 401 Unauthorized with the above error in the error log.

Environment

• APISIX version (run apisix version): 3.11.0 (unreleased, master currently)

[kayx23]

#11166 this seems to be a similar issue albeit different scenario

[kayx23]

just fyi @HuanXin-Chen @nic-6443

[HuanXin-Chen]

This error seems a bit difficult to resolve, and I don't have a GCP Account on hand to test it, but it looks like it might also be related to certificates.

I developed based on the original google-cloud-logging plugin, perhaps community users have a solution?

The two errors look the same.

[kayx23]

Tested again with the 3.11.0 docker image and cannot reproduce this anymore.

Adding this config block to config.yaml is sufficient:

apisix:
  ssl:
    ssl_trusted_certificate: /etc/ssl/certs/ca-certificates.crt

Considered resolved.