Workspace metrics not working

[amomra]

Describe the bug

The resource monitor doesn't show the CPU and memory usage:

The che pod log keeps showing the following message: Unable to add metrics roles due to insufficient permissions. Workspace metrics will be disabled.

After doing some tests I found that if I add the role and the role binding manually to the metrics API group (metrics.k8s.io) in k8s to service account che-workspace in the workspace namespace the resource monitor in the Eclipse Theia starts showing the CPU and memory usage as it should. I used this file to add this role and role binding:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: workspace-metrics
  namespace: che-ws-lcvmelo
  labels:
    app: che
    component: che
rules:
- apiGroups:
  - "metrics.k8s.io"
  attributeRestrictions: null
  resources:
  - pods
  - nodes
  verbs:
  - get
  - list
  - watch
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: che-workspace-metrics
  namespace: che-ws-lcvmelo
  labels:
    app: che
    component: che
roleRef:
  kind: Role
  name: workspace-metrics
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: che-workspace
  namespace: che-ws-lcvmelo

Futher investigating the problem I found that this message is logged at line 152 of org.eclipse.che.workspace.infrastructure.kubernetes.namespace.AbstractWorkspaceServiceAccount, triggered by the exception thrown by the call at line 137 - k8sClient.supportsApiPath("/apis/metrics.k8s.io") (all lines are from tag 7.42.0). In the Eclipse Che log I forced a stack print of this exception.

All other roles (workspace-configmaps, workspace-secrets,...) are created normally.

Che version

7.42@latest

Steps to reproduce

1. Install Eclipse Che using Helm operator
2. Create a new cluster without Devworkspace
3. Create a new workspace using any example (for this test I used the python)
4. After the startup of Eclipse Theia, the resource monitor doesn't display the CPU and memory usage

Expected behavior

The resources monitor in Eclipse Theia showing the CPU and memory usage

Runtime

Kubernetes (vanilla)

Screenshots

No response

Installation method

other (please specify in additional context)

Environment

Linux

Eclipse Che Logs

io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://10.202.0.1/. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. forbidden: User "system:serviceaccount:eclipse-che:che" cannot get path "/".
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:686)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:623)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:565)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:526)
        at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:509)
        at io.fabric8.kubernetes.client.dsl.base.BaseOperation.getRootPaths(BaseOperation.java:201)
        at io.fabric8.kubernetes.client.BaseClient.rootPaths(BaseClient.java:138)
        at io.fabric8.kubernetes.client.BaseClient.supportsApiPath(BaseClient.java:143)
        at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.AbstractWorkspaceServiceAccount.ensureImplicitRolesWithBindings(AbstractWorkspaceServiceAccount.java:137)
        at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.AbstractWorkspaceServiceAccount.prepare(AbstractWorkspaceServiceAccount.java:102)
        at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.configurator.WorkspaceServiceAccountConfigurator.configure(WorkspaceServiceAccountConfigurator.java:64)
        at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespaceFactory.configureNamespace(KubernetesNamespaceFactory.java:570)
        at org.eclipse.che.workspace.infrastructure.kubernetes.namespace.KubernetesNamespaceFactory.getOrCreate(KubernetesNamespaceFactory.java:334)
        at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesRuntimeContext.getRuntime(KubernetesRuntimeContext.java:74)
        at org.eclipse.che.workspace.infrastructure.kubernetes.KubernetesRuntimeContext.getRuntime(KubernetesRuntimeContext.java:32)
        at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.startAsync(WorkspaceRuntimes.java:473)
        at org.eclipse.che.api.workspace.server.WorkspaceManager.startAsync(WorkspaceManager.java:528)
        at org.eclipse.che.api.workspace.server.WorkspaceManager.startWorkspace(WorkspaceManager.java:383)
        at org.eclipse.che.multiuser.resource.api.workspace.LimitsCheckingWorkspaceManager.startWorkspace(LimitsCheckingWorkspaceManager.java:152)
        at org.eclipse.che.api.workspace.server.WorkspaceService.startById(WorkspaceService.java:436)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:174)
        at org.everrest.core.impl.method.DefaultMethodInvoker.invokeMethod(DefaultMethodInvoker.java:61)
        at org.everrest.core.impl.RequestDispatcher.doInvokeResource(RequestDispatcher.java:329)
        at org.everrest.core.impl.RequestDispatcher.invokeSubResourceMethod(RequestDispatcher.java:319)
        at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:257)
        at org.everrest.core.impl.RequestDispatcher.dispatch(RequestDispatcher.java:131)
        at org.everrest.core.impl.RequestHandlerImpl.handleRequest(RequestHandlerImpl.java:61)
        at org.everrest.core.impl.EverrestProcessor.process(EverrestProcessor.java:130)
        at org.everrest.core.servlet.EverrestServlet.service(EverrestServlet.java:62)
        at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:777)
        at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290)
        at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280)
        at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184)
        at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85)
        at org.eclipse.che.core.metrics.ApiResponseMetricFilter.doFilter(ApiResponseMetricFilter.java:46)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at org.eclipse.che.commons.logback.filter.IdentityIdLoggerFilter.doFilter(IdentityIdLoggerFilter.java:49)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at org.eclipse.che.multiuser.api.authentication.commons.filter.MultiUserEnvironmentInitializationFilter.doFilter(MultiUserEnvironmentInitializationFilter.java:161)
        at org.eclipse.che.multiuser.keycloak.server.KeycloakEnvironmentInitializationFilter.doFilter(KeycloakEnvironmentInitializationFilter.java:99)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at org.eclipse.che.multiuser.machine.authentication.server.MachineLoginFilter.doFilter(MachineLoginFilter.java:77)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at org.eclipse.che.commons.logback.filter.RequestIdLoggerFilter.doFilter(RequestIdLoggerFilter.java:50)
        at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82)
        at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121)
        at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:185)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:158)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:119)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:769)
        at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:690)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:353)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:382)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:872)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1705)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
        at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Unknown Source)
2022-01-23 14:09:20,268[nio-8080-exec-8]  [WARN ] [bstractWorkspaceServiceAccount 151]  - Unable to add metrics roles due to insufficient permissions. Workspace metrics will be disabled.
2022-01-23 14:09:20,733[nio-8080-exec-8]  [INFO ] [o.e.c.a.w.s.WorkspaceRuntimes 486]   - Starting workspace 'lcvmelo/python-08o0' with id 'workspacem62kahf4sfx19pyd' by user 'lcvmelo'

Additional context

Tested in two clusters provisioned with kubeadm, one with k8s 1.21 and the other with 1.23 (single-node). Both running in RHEL 8.4.

Helm chart installed from here: https://eclipse-che.github.io/che-operator/charts

[amomra]

After I added the required cluster role for che service account the metrics roles in the workspace was created automatically:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: che-cluster-role
  labels:
    app: che
    component: che
rules:
- nonResourceURLs: ["/"]
  verbs:
  - get
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: che-cluster-role-binding
  labels:
    app: che
    component: che
roleRef:
  kind: ClusterRole
  name: che-cluster-role
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: che
  namespace: eclipse-che

[svor]

@amomra thank you for your investigation.
It's duplicate of #20800 and should be fixed when devworkspace-operator v0.12.0 is released https://github.com/devfile/devworkspace-operator