Creating a custom plugin fails with an error

[shahamit]

I am following this guide to create a custom plugin for my kong ingress controller running on k3d + k3s.

Following the steps as is in the guide still causes the ingress-kong-xxx pod to crash with the error logs as below. I tried many tweaks but nothing helped. From the error it seems the schema.lua file is incorrect but its exactly what is shown in the guide.
I have also paste the output of kubectl describe deployment -n kong to share the pod config.

Could someone guide me to understand whats wrong?

Thanks.

P.S - Is there a way to start the ingress-kong pod or the proxy container in debug like it is suggested here as a troubleshooting step for kong a standalone gateway ?

Output of kubectl logs pod/ingress-kong-6599967995-swkc5 proxy -n kong command

2021/09/27 12:39:28 [warn] 1#0: the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
nginx: [warn] the "user" directive makes sense only if the master process runs with super-user privileges, ignored in /usr/local/kong/nginx.conf:6
2021/09/27 12:39:28 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:515: error loading plugin schemas: on plugin 'myheader': myheader plugin is enabled but not installed;
no plugin found
stack traceback:
[C]: in function 'assert'
/usr/local/share/lua/5.1/kong/init.lua:515: in function 'init'
init_by_lua:3: in main chunk
nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:515: error loading plugin schemas: on plugin 'myheader': myheader plugin is enabled but not installed;
no plugin found
stack traceback:
[C]: in function 'assert'
/usr/local/share/lua/5.1/kong/init.lua:515: in function 'init'
init_by_lua:3: in main chunk

Output of kubectl describe deployment -n kong command

amit@laptop:~/Documents/Sundry/Projects/K3D/kong$ kubectl describe deployment -n kong
Name:                   ingress-kong
Namespace:              kong
CreationTimestamp:      Mon, 27 Sep 2021 17:50:33 +0530
Labels:                 app=ingress-kong
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               app=ingress-kong
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:           app=ingress-kong
  Annotations:      kuma.io/gateway: enabled
                    traffic.sidecar.istio.io/includeInboundPorts: 
  Service Account:  kong-serviceaccount
  Containers:
   proxy:
    Image:       kong:2.5
    Ports:       8000/TCP, 8443/TCP, 8100/TCP
    Host Ports:  0/TCP, 0/TCP, 0/TCP
    Liveness:    http-get http://:8100/status delay=5s timeout=1s period=10s #success=1 #failure=3
    Readiness:   http-get http://:8100/status delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      KONG_PROXY_LISTEN:            0.0.0.0:8000, 0.0.0.0:8443 ssl http2
      KONG_PORT_MAPS:               80:8000, 443:8443
      KONG_ADMIN_LISTEN:            127.0.0.1:8444 ssl
      KONG_STATUS_LISTEN:           0.0.0.0:8100
      KONG_DATABASE:                off
      KONG_NGINX_WORKER_PROCESSES:  2
      KONG_ADMIN_ACCESS_LOG:        /dev/stdout
      KONG_ADMIN_ERROR_LOG:         /dev/stderr
      KONG_PROXY_ERROR_LOG:         /dev/stderr
      KONG_PLUGINS:                 bundled,myheader
      KONG_LUA_PACKAGE_PATH:        /home/amit/K3D/kong/?.lua;;
    Mounts:
      /home/amit/K3D/kong/myheader from kong-plugin-myheader (rw)
   ingress-controller:
    Image:       kong/kubernetes-ingress-controller:2.0.0-beta.2
    Ports:       8080/TCP, 10255/TCP
    Host Ports:  0/TCP, 0/TCP
    Liveness:    http-get http://:10254/healthz delay=5s timeout=1s period=10s #success=1 #failure=3
    Readiness:   http-get http://:10254/readyz delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment:
      CONTROLLER_KONG_ADMIN_URL:              https://127.0.0.1:8444
      CONTROLLER_KONG_ADMIN_TLS_SKIP_VERIFY:  true
      CONTROLLER_PUBLISH_SERVICE:             kong/kong-proxy
      POD_NAME:                                (v1:metadata.name)
      POD_NAMESPACE:                           (v1:metadata.namespace)
    Mounts:                                   <none>
  Volumes:
   kong-plugin-myheader:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      kong-plugin-myheader
    Optional:  false
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      False   MinimumReplicasUnavailable
  Progressing    True    ReplicaSetUpdated
OldReplicaSets:  <none>
NewReplicaSet:   ingress-kong-7764cbdf69 (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  78s   deployment-controller  Scaled up replica set ingress-kong-7764cbdf69 to 1

[shaneutt]

The problem you're having would seem to be because the plugin isn't in the search path:

 2021/09/27 12:39:28 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:515: error loading plugin schemas: on plugin 'myheader': myheader plugin is enabled but not installed;

This means that the plugin file is not in the lua path or otherwise findable by Kong.

The standard location for plugins is /usr/local/share/lua/5.1/kong/plugins but this guide has you use /opt/kong/plugins and change the search path to include it. Double check the following:

• configmap creation
• volume mount
• setting KONG_LUA_PACKAGE_PATH

I can't be sure from just the describe output you provided but it would seem that something got misconfigured between the above three configurations. Also note that if /home/amit/K3D/kong/?.lua; is the path you set, that wont work.

[shahamit]

Thanks. Just replacing the path to /usr/local/share/lua/5.1/kong/plugins fixed the issue. I could see the plugin working