kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1 won't work

[bradenwright]

I've been trying to get this going and I'm just missing something, I think there is a good chance its related to RBAC. I've tried to follow Walk Through , Deploy and Helm Chart but can't get any of them to work.

$ kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1
Error from server (NotFound): the server could not find the requested resource

I was able to get Luxas Walkthrough to work end to end. To eliminate additional complexities I tried to use madeden Helm chart with self-signed certs. And also edit above to use self-signed in case there were issues there. I've gone over RBAC and tried to add some more open privileges for testing, but have only limitedly dealt with RBAC.

I don't see any errors til the pod service starts:

I0504 17:41:44.059453       1 serve.go:85] Serving securely on 0.0.0.0:6443
I0504 17:41:44.943488       1 round_trippers.go:383] POST https://100.64.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews
I0504 17:41:44.943505       1 round_trippers.go:390] Request Headers:
I0504 17:41:44.943510       1 round_trippers.go:393]     Accept: application/json, */*
I0504 17:41:44.943515       1 round_trippers.go:393]     Content-Type: application/json
I0504 17:41:44.943519       1 round_trippers.go:393]     Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yaW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im5paGlsaXN0LW1vbmtleS1jdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXItdG9rZW4tcmQyOG4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibmloaWxpc3QtbW9ua2V5LWN1c3RvbS1tZXRyaWNzLWFwaXNlcnZlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjY3ZGQwZjA4LTRmYzItMTFlOC05YTI4LTBhNGFjOGVmYTA0NCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptb25pdG9yaW5nOm5paGlsaXN0LW1vbmtleS1jdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXIifQ.PjNg-OeUmgIs5Pm7EPY_dBj03HnjOj9P82tqHa3C6hEexrYZll96vgw2pjaCSEQz7JV7O4NE7MJ_YhAMCYrvYYO9KYosRsXzMchqG43viYxP0Hwv0T7rsvbY9DbxA0BSGDPvB8E1htN1guJnTcpZX5hfLlWUHuc5T7Gty22peTuHkJCXIi0tWMkfYDCO6yy-gU-V6-yPjFfPEKu8AtWQ7G_aGJb0WJyQyZRnY38gg_cFk7uah94UiFrROTElYjdho0a7L5EJsAm3CE94BeW3n2W_4nxR9LOdjeqedMEzzVIO5zkcu3dzUD94HafzS7Lcb1O5UVgjgKNCND4QAvEQEg
I0504 17:41:44.943528       1 round_trippers.go:393]     User-Agent: prometheus-adapter/v0.0.0 (linux/amd64) kubernetes/$Format
I0504 17:41:44.947151       1 round_trippers.go:408] Response Status: 201 Created in 3 milliseconds
I0504 17:41:44.947409       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:44.947460       1 wrap.go:42] GET /: (4.463264ms) 403 [[Go-http-client/2.0] 100.99.92.0:45210]
I0504 17:41:44.953296       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:44.953334       1 wrap.go:42] GET /: (80.866µs) 403 [[Go-http-client/2.0] 100.99.84.0:55270]
I0504 17:41:44.958829       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:44.958864       1 wrap.go:42] GET /: (78.637µs) 403 [[Go-http-client/2.0] 100.99.92.0:45210]
I0504 17:41:44.966197       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:44.966230       1 wrap.go:42] GET /: (56.14µs) 403 [[Go-http-client/2.0] 100.99.84.0:55270]
I0504 17:41:44.971338       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:44.971365       1 wrap.go:42] GET /: (52.565µs) 403 [[Go-http-client/2.0] 100.99.84.0:55270]
I0504 17:41:45.018497       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:45.018542       1 wrap.go:42] GET /: (97.306µs) 403 [[Go-http-client/2.0] 100.126.0.0:17226]
I0504 17:41:45.021977       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:45.022004       1 wrap.go:42] GET /: (57.143µs) 403 [[Go-http-client/2.0] 100.126.0.0:17226]
I0504 17:41:45.027072       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:45.027169       1 wrap.go:42] GET /: (127.241µs) 403 [[Go-http-client/2.0] 100.126.0.0:17226]
I0504 17:41:45.222450       1 authorization.go:58] Forbidden: "/", Reason: ""
I0504 17:41:45.222481       1 wrap.go:42] GET /: (59.318µs) 403 [[Go-http-client/2.0] 100.126.0.0:17226]
I0504 17:41:48.260032       1 round_trippers.go:383] POST https://100.64.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews
I0504 17:41:48.260049       1 round_trippers.go:390] Request Headers:
I0504 17:41:48.260055       1 round_trippers.go:393]     Accept: application/json, */*
I0504 17:41:48.260059       1 round_trippers.go:393]     Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yaW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im5paGlsaXN0LW1vbmtleS1jdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXItdG9rZW4tcmQyOG4iLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibmloaWxpc3QtbW9ua2V5LWN1c3RvbS1tZXRyaWNzLWFwaXNlcnZlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6IjY3ZGQwZjA4LTRmYzItMTFlOC05YTI4LTBhNGFjOGVmYTA0NCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptb25pdG9yaW5nOm5paGlsaXN0LW1vbmtleS1jdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXIifQ.PjNg-OeUmgIs5Pm7EPY_dBj03HnjOj9P82tqHa3C6hEexrYZll96vgw2pjaCSEQz7JV7O4NE7MJ_YhAMCYrvYYO9KYosRsXzMchqG43viYxP0Hwv0T7rsvbY9DbxA0BSGDPvB8E1htN1guJnTcpZX5hfLlWUHuc5T7Gty22peTuHkJCXIi0tWMkfYDCO6yy-gU-V6-yPjFfPEKu8AtWQ7G_aGJb0WJyQyZRnY38gg_cFk7uah94UiFrROTElYjdho0a7L5EJsAm3CE94BeW3n2W_4nxR9LOdjeqedMEzzVIO5zkcu3dzUD94HafzS7Lcb1O5UVgjgKNCND4QAvEQEg
I0504 17:41:48.260068       1 round_trippers.go:393]     Content-Type: application/json
I0504 17:41:48.260074       1 round_trippers.go:393]     User-Agent: prometheus-adapter/v0.0.0 (linux/amd64) kubernetes/$Format
I0504 17:41:48.261611       1 round_trippers.go:408] Response Status: 201 Created in 1 milliseconds
I0504 17:41:48.261866       1 wrap.go:42] GET /swagger.json: (2.089729ms) 404 [[] 100.126.0.0:17268]
I0504 17:41:48.571698       1 wrap.go:42] GET /swagger.json: (202.436µs) 404 [[] 100.99.92.0:45258]
I0504 17:41:48.604608       1 wrap.go:42] GET /swagger.json: (179.826µs) 404 [[] 100.99.84.0:55292]
I0504 17:41:50.440373       1 round_trippers.go:383] POST https://100.64.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews
I0504 17:41:50.440391       1 round_trippers.go:390] Request Headers:
I0504 17:41:50.440397       1 round_trippers.go:393]     Content-Type: application/json
I0504 17:41:50.440401       1 round_trippers.go:393]     User-Agent: prometheus-adapter/v0.0.0 (linux/amd64) kubernetes/$Format

When I turn on more verbose logging I'm seeing a lot of things like:

I0504 18:29:10.853028       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1: (218.596µs) 404 [[kubectl/v1.10.1 (darwin/amd64) kubernetes/d4ab475] 100.99.84.0:62308]
I0504 18:29:11.002233       1 request.go:991] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:resourcequota-controller","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":false}}
I0504 18:29:11.420526       1 round_trippers.go:386] curl -k -v -XPOST  -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJtb25pdG9yaW5nIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZWNyZXQubmFtZSI6Im5paGlsaXN0LW1vbmtleS1jdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXItdG9rZW4tMnp3NnMiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC5uYW1lIjoibmloaWxpc3QtbW9ua2V5LWN1c3RvbS1tZXRyaWNzLWFwaXNlcnZlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImY0NTgwMTJhLTRmYzgtMTFlOC05YTI4LTBhNGFjOGVmYTA0NCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDptb25pdG9yaW5nOm5paGlsaXN0LW1vbmtleS1jdXN0b20tbWV0cmljcy1hcGlzZXJ2ZXIifQ.EON5xGFxVjswqu9U3h1Dt1baRBWVU4_MuACXHFr2PzmbpqCNACTU9Os55a2PY3WJGbyUDQ2LPhmurGvW3RWc6nRcLBoL3-F1L6Ri3na7LjUt67gHa56GcyhWNeHgKHTKownKqYvR7dqqsAR7T2fHAM5oSmL1w9qokl7hMCmcXVxoF8SxVNBp-xYOCcYCeG4O0-f4XRzlfdEz5f4VbJYkHty-KDyouSUxin_VTrJZyTX-3h138MJlt-H_aBtostre9D7tp_XKKgMvMp-fRP8ff6MALXE7NkWV1DU3OVJj2OXCnv9Jnq3VEtnxMdsKkrYH8FkYFqYHCVsz5OZxT3nTEw" -H "User-Agent: prometheus-adapter/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Accept: application/json, */*" -H "Content-Type: application/json" https://100.64.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews
I0504 18:29:11.424231       1 round_trippers.go:405] POST https://100.64.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 3 milliseconds
I0504 18:29:11.424251       1 round_trippers.go:411] Response Headers:
I0504 18:29:11.424255       1 round_trippers.go:414]     Content-Type: application/json
I0504 18:29:11.424259       1 round_trippers.go:414]     Content-Length: 395
I0504 18:29:11.424263       1 round_trippers.go:414]     Date: Fri, 04 May 2018 18:29:11 GMT
I0504 18:29:11.424292       1 request.go:991] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/apis/custom.metrics.k8s.io/v1beta1","verb":"get"},"user":"system:serviceaccount:kube-system:resourcequota-controller","group":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]},"status":{"allowed":true}}
I0504 18:29:11.424432       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1: (422.406836ms) 404 [[kube-controller-manager/v1.9.7 (linux/amd64) kubernetes/dd5e1a2/system:serviceaccount:kube-system:resourcequota-controller] 100.99.84.0:62308]
I0504 18:29:11.592583       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1: (225.387µs) 404 [[kubectl/v1.10.1 (darwin/amd64) kubernetes/d4ab475] 100.99.84.0:62308]
I0504 18:29:11.648679       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1: (210.463µs) 404 [[kubectl/v1.10.1 (darwin/amd64) kubernetes/d4ab475] 100.99.84.0:62308]
I0504 18:29:12.802287       1 wrap.go:42] GET /apis/custom.metrics.k8s.io/v1beta1: (214.386µs) 404 [[kube-controller-manager/v1.9.7 (linux/amd64) kubernetes/dd5e1a2/system:serviceaccount:kube-system:resourcequota-controller] 100.99.84.0:62308]
I0504 18:29:15.228040       1 request.go:991] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}

I've been trying to look at the difference b/t the 2, but I feel like I'm spinning my wheels a little, hoping with the details and fact that I can get Luxas to work will help surface my issue. More than happy to post anything that will help.

[bradenwright]

@DirectXMan12 any ideas?

[DerrickMartinez]

You probably just need to make your own image. try to curl /apis to see what options you have.

[soorajchirag]

Same issue for me.

Got over it by getting the image 'directxman12/k8s-prometheus-adapter:advanced-config' instead of 'directxman12/k8s-prometheus-adapter:v0.1.0-centos'

[soorajchirag]

Actually I spoke too soon, the upgrade to the latest image will work for Kubernetes 1.8+.

I'm attempting to run HPA with custom-metrics on kubernetes 1.7.

While switching to the advanced-config image did help with 'kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1', but the HPA was still throwing the following errors:

Events:
FirstSeen LastSeen Count From SubObjectPath Type Reason Message

---

1h 12m 139 horizontal-pod-autoscaler Warning FailedGetPodsMetric unable to get metric active_operations: unable to fetch metrics from API: the server could not find the requested resource (get pods.custom-metrics.metrics.k8s.io *)
1h 12m 139 horizontal-pod-autoscaler Warning FailedComputeMetricsReplicas failed to get pods metric value: unable to get metric active_operations: unable to fetch metrics from API: the server could not find the requested resource (get pods.custom-metrics.metrics.k8s.io *)

Looking at these it indicates that the HPA is expecting the custom metrics to be available at the path "custom-metrics.metrics.k8s.io", and not "custom.metrics.k8s.io".

So, I changed the API Service definition to refer to the group custom-metrics.metrics.k8s.io.

Deployed the apiserver again, but now the 'kubectl raw' command failed. The custom-metrics-apiserver logs logged a 404 for the path "/apis/custom-metrics.metrics.k8s.io/v1beta1". So, it was getting the request but didnt know how to serve it.

I dug a bit into the https://github.com/kubernetes/metrics/ and found this commit:
bc84746e82d259b6ca013c3f213767138b28b96a which changed the API endpoint from custom-metrics.metrics.k8s.io to custom.metrics.k8s.io.

Taking that and from kubernetes/metrics figured out that the group version needs to be 'v1apha1'.

So, I switched back to directxman12/k8s-prometheus-adapter:v0.1.0-centos and updated the API Service yaml as follows for Kubernetes 1.7:

1. metdata.name: v1alpha1.custom-metrics.metrics.k8s.io
2. spec.group: custom-metrics.metrics.k8s.io
3. spec.version: v1alpha1

Post re-deploying the apiserver, etc I was able to successfully execute "kubectl get --raw "/apis/custom-metrics.metrics.k8s.io/v1alpha1", and even the HPA was able to get to my custom Pod metrics.

Hope this helps someone.

[DirectXMan12]

Only v0.1.0 works on Kubernetes 1.7, as noted in the release notes of v0.2.0.