Having more than 100 httproutes may causes excessive resource conflicts

[gjkim42]

Is there an existing issue for this?

• I have searched the existing issues

Current Behavior

When I create more than 100 httproutes, the kong ingress controller causes excessive resource conflicts on kube-apiserver side, as it cannot correctly determine the attachedRoutes of a Gateway resource but the value keeps fluctuating.

Expected Behavior

When I create more than 101 httproutes, the resource usage of kube-apiserver does not increase to an insane level, and attachedRoutes is correctly set.

Steps To Reproduce

1. Create 20 invalid httproutes and 100 valid httproutes.

$ cat test.sh
#!/usr/bin/env bash

function create_invalid_httproute {            
  local num="${1}"
  cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:    
  name: test-invalid-httproute-${num}
spec:            
  parentRefs:         
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: invalid-test-gateway
    namespace: test-gateway
  rules:           
  - backendRefs: 
    - group: "" 
      kind: Service
      name: non-existing
      port: 80
      weight: 1         
    matches:                              
    - path:
        type: PathPrefix
        value: /test-invalid-httproute-${num}/test
EOF                 
} 

function create_httproute {
  local num="${1}"
  cat <<EOF | kubectl create -f -
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: test-httproute-${num}
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: test-gateway
    namespace: test-gateway
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: test-service
      port: 80
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /test-httproute-${num}/test
EOF
}

for i in {1..20}; do
  create_invalid_httproute ${i}
done
for i in {1..100}; do
  create_httproute ${i}
done

$ ./test.sh

2. Make sure that kube-apiserver's (code_resource:apiserver_request_total:rate5m{verb="write", resource="gateways"}) metric is excessively increased.

3. Make sure that the issue is addressed after removing those httproutes.

Kong Ingress Controller version

v2.10.3

Kubernetes version

$ kubectl version
Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.5", GitCommit:"c285e781331a3785a7f436042c65c5641ce8a9e9", GitTreeState:"clean", BuildDate:"2022-03-16T15:52:18Z", GoVersion:"go1.17.8", Compiler:"gc", Platform:"linux/amd64"}

Anything else?

No response

[gjkim42]

kubernetes-ingress-controller/internal/controllers/gateway/gateway_utils.go

Lines 700 to 710 in 7e567cc

	httpRouteList := gatewayv1beta1.HTTPRouteList{}
	if err := mgrc.List(ctx, &httpRouteList, &client.ListOptions{
	Continue: continueToken,
	Limit: defaultEndpointSliceListPagingLimit,
	}); err != nil {
	return 0, err
	}
	httpRoutes = append(httpRoutes, httpRouteList.Items...)
	if httpRouteList.Continue == "" {
	break
	}

found that the issue is related to the fact that the mgrc.List returns httpRouteList without Continue token even if there are more than defaultEndpointSliceListPagingLimit(100) httpRoutes .

[gjkim42]

mgrc.List always returns exactly 100 items which are a subset of the total resources. It ends up generating different attachedRoutes values, as each given subset changes every time.

[gjkim42]

I believe that we MUST not use continue token in a cached client.
We can simply remove those paginations.

[czeslavo]

@gjkim42 Thank you for reporting and handling the fix yourself. I believe you can apply for a contributor T-shirt having a bug fixed.