'rolebindings.authorization.openshift.io "exec" not found' error when start workspace with OS credentials on minishift #16534

dmytro-ndp · 2020-04-02T20:11:48Z

Describe the bug

OS OAuth E2E selenium tests had started failing on nightly Jenkins jobs from the next build
https://ci.centos.org/view/Devtools/job/devtools-che-nightly-oauth-test/47/console

[ERROR] Failures:
[ERROR] > LoginExistedUserWithOpenShiftOAuthTest.checkExistedCheUserOcpProjectCreationAndRemoval:142 » Timeout
[ERROR] > LoginNewUserWithOpenShiftOAuthTest.checkNewCheUserOcpProjectCreationAndRemoval:126 » Timeout

Eclipse Che had deployed successfully, but then it was impossible to start workspace with OS credentials:

http://artifacts.ci.centos.org/devtools/che/rc-ocp-oauth-test/30/report/site/failsafe-report.html

There were next commit during the day in Eclipse Che which might had caused regression:

db46ad4

Server logs:

2020-03-26 05:30:08,393[nio-8080-exec-5]  [INFO ] [o.e.c.a.w.s.WorkspaceManager 641]    - Workspace 'user1585200583035/workspaceowv0' with id 'workspacee8wkzi82iwhadgbq' created by user 'user1585200583035'
2020-03-26 05:30:10,982[io-8080-exec-10]  [ERROR] [c.a.c.r.RuntimeExceptionMapper 47]   - Internal Server Error occurred, error time: 2020-03-26 05:30:10
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.30.0.1/apis/authorization.openshift.io/v1/namespaces/user1585200583035-che/rolebindings. Message: rolebindings.authorization.openshift.io "exec" not found. Received status: Status(apiVersion=v1, code=404, details=StatusDetails(causes=[], group=authorization.openshift.io, kind=rolebindings, name=exec, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=rolebindings.authorization.openshift.io "exec" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=NotFound, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)

Command to deploy Eclipse Che to minishift 3.11:

chectl server:start -a operator -p openshift --k8spodreadytimeout=360000 --chenamespace=eclipse-che --che-operator-cr-yaml=/tmp/custom-resource.yaml

custom-resource.yaml

apiVersion: v1
items:
- apiVersion: org.eclipse.che/v1
  kind: CheCluster
  metadata:
    creationTimestamp: 2020-03-26T05:11:26Z
    finalizers:
    - oauthclients.finalizers.che.eclipse.org
    generation: 17
    name: eclipse-che
    namespace: eclipse-che
    resourceVersion: "3593"
    selfLink: /apis/org.eclipse.che/v1/namespaces/eclipse-che/checlusters/eclipse-che
    uid: 3e28df9e-6f20-11ea-b704-525400a648f9
  spec:
    auth:
      externalIdentityProvider: false
      identityProviderClientId: che-public
      identityProviderImage: quay.io/eclipse/che-keycloak:nightly
      identityProviderPassword: admin
      identityProviderPostgresSecret: che-identity-postgres-secret
      identityProviderRealm: che
      identityProviderSecret: che-identity-secret
      identityProviderURL: http://keycloak-eclipse-che.192.168.42.244.nip.io
      oAuthClientName: eclipse-che-openshift-identity-provider-kbltf2
      oAuthSecret: lxjP9sqONvGZ
      openShiftoAuth: true
      updateAdminPassword: false
    database:
      chePostgresDb: dbche
      chePostgresHostName: postgres
      chePostgresPort: "5432"
      chePostgresSecret: che-postgres-secret
      externalDb: false
    k8s:
      ingressDomain: 192.168.99.101.nip.io
    metrics:
      enable: false
    server:
      allowUserDefinedWorkspaceNamespaces: false
      cheDebug: "false"
      cheFlavor: che
      cheHost: che-eclipse-che.192.168.42.244.nip.io
      cheImageTag: nightly
      cheLogLevel: INFO
      customCheProperties:
        CHE_LIMITS_USER_WORKSPACES_RUN_COUNT: "-1"
      devfileRegistryImage: quay.io/eclipse/che-devfile-registry:nightly
      externalDevfileRegistry: false
      externalPluginRegistry: false
      gitSelfSignedCert: false
      pluginRegistryImage: quay.io/eclipse/che-plugin-registry:nightly
      selfSignedCert: false
      tlsSupport: false
    storage:
      preCreateSubPaths: true
      pvcClaimSize: 1Gi
      pvcStrategy: common
  status:
    cheClusterRunning: Available
    cheURL: http://che-eclipse-che.192.168.42.244.nip.io
    cheVersion: nightly
    dbProvisioned: true
    devfileRegistryURL: http://devfile-registry-eclipse-che.192.168.42.244.nip.io
    keycloakProvisioned: true
    keycloakURL: http://keycloak-eclipse-che.192.168.42.244.nip.io
    openShiftoAuthProvisioned: true
    pluginRegistryURL: http://plugin-registry-eclipse-che.192.168.42.244.nip.io/v3
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Che version

latest
nightly
other: please specify

Steps to reproduce

tests/legacy-e2e/che-selenium-test/selenium-tests.sh \
    --threads=1 \
    --host=${CHE_ROUTE} \
    --port=80 \
    --multiuser \
    --test=org.eclipse.che.selenium.site.ocpoauth.**

Runtime

kubernetes (include output of kubectl version)
Openshift (include output of oc version)
minikube (include output of minikube version and kubectl version)
minishift 3.11
docker-desktop + K8S (include output of docker version and kubectl version)
other: (please specify)

Installation method

chectl:next
che-operator
minishift-addon
I don't know

Environment

Eclipse Che Logs

Additional context

The text was updated successfully, but these errors were encountered:

tolusha · 2020-04-03T12:16:11Z

$ kubectl get rolebinding
NAME AGE
che-workspace-exec 18m

But che-server uses exec
https://github.com/eclipse/che/blob/master/infrastructures/kubernetes/src/main/java/org/eclipse/che/workspace/infrastructure/kubernetes/namespace/KubernetesWorkspaceServiceAccount.java#L78

So, it is inconsistency in the names

tolusha · 2020-04-03T12:18:34Z

/cc @sleshchenko

skabashnyuk · 2020-04-03T12:22:32Z

Might be related to incorrect method renaming during k8s client upgrade.

    if (k8sClient.rbac().kubernetesRoles().inNamespace(namespace).withName(execRoleName).get()

to

    if (k8sClient.rbac().clusterRoles().inNamespace(namespace).withName(execRoleName).get()

instead
of

    if (k8sClient.rbac().roles().inNamespace(namespace).withName(execRoleName).get()

skabashnyuk · 2020-04-03T12:23:48Z

See more fabric8io/kubernetes-client#1300

amisevsk · 2020-04-03T21:53:08Z

@skabashnyuk I assigned severity/P1 to this issue, feel free to revise.

ibuziuk · 2020-04-08T08:48:33Z

@dmytro-ndp could you confirm that this issue is minishift specific?

skabashnyuk · 2020-04-14T12:49:57Z

reproduced on crc

crc version 
crc version: 1.8.0+0a318dc
OpenShift version: 4.3.8 (embedded in binary)

2020-04-14 12:41:26,865[nio-8080-exec-7]  [ERROR] [c.a.c.r.RuntimeExceptionMapper 47]   - Internal Server Error occurred, error time: 2020-04-14 12:41:26
io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://172.30.0.1/apis/authorization.openshift.io/v1/namespaces/skabashn-che/rolebindings. Message: rolebindings.authorization.openshift.io "exec" not found. Received status: Status(apiVersion=v1, code=404, details=StatusDetails(causes=[], group=authorization.openshift.io, kind=rolebindings, name=exec, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=rolebindings.authorization.openshift.io "exec" not found, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=NotFound, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:568)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:507)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:471)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:430)
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:251)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:815)
	at io.fabric8.openshift.client.dsl.internal.OpenshiftRoleBindingOperationsImpl.handleCreate(OpenshiftRoleBindingOperationsImpl.java:72)
	at io.fabric8.openshift.client.dsl.internal.OpenshiftRoleBindingOperationsImpl.handleCreate(OpenshiftRoleBindingOperationsImpl.java:39)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.create(BaseOperation.java:333)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:394)
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.createOrReplace(BaseOperation.java:391)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftWorkspaceServiceAccount.prepare(OpenShiftWorkspaceServiceAccount.java:89)
	at org.eclipse.che.workspace.infrastructure.openshift.project.OpenShiftProjectFactory.getOrCreate(OpenShiftProjectFactory.java:94)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftRuntimeContext.getRuntime(OpenShiftRuntimeContext.java:69)
	at org.eclipse.che.workspace.infrastructure.openshift.OpenShiftRuntimeContext.getRuntime(OpenShiftRuntimeContext.java:31)
	at org.eclipse.che.api.workspace.server.WorkspaceRuntimes.startAsync(WorkspaceRuntimes.java:466)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.startAsync(WorkspaceManager.java:488)
	at org.eclipse.che.api.workspace.server.WorkspaceManager.startWorkspace(WorkspaceManager.java:373)
	at org.eclipse.che.multiuser.resource.api.workspace.LimitsCheckingWorkspaceManager.startWorkspace(LimitsCheckingWorkspaceManager.java:132)
	at org.eclipse.che.api.workspace.server.WorkspaceService.startById(WorkspaceService.java:469)

metlos · 2020-04-14T20:40:28Z

So the fix seems to be to create the roles in namespace instead of cluster-wide and to correctly reference the namespace when referencing the role when creating the role binding. All needs to be done in both the KubernetesWorkspaceServiceAccount and OpenshiftWorkspaceServiceAccount. The latter seems to be already part of #16532 but I would do it as part of a PR for this issue even if we cause a merge conflict by doing that. I will provide the PR soon.

* explicitly specify the namespace in the role reference when creating che workspace service account role bindings in openshift infra * create the roles in the namespace, not as cluster roles in k8s infra Signed-off-by: Lukas Krejci <lkrejci@redhat.com>

dmytro-ndp added kind/bug Outline of a bug - must adhere to the bug report template. area/qe labels Apr 2, 2020

dmytro-ndp added this to the Backlog - QE milestone Apr 2, 2020

dmytro-ndp mentioned this issue Apr 2, 2020

Che 7.11 Release #16478

Closed

16 tasks

dmytro-ndp assigned dmytro-ndp and SkorikSergey and unassigned dmytro-ndp Apr 3, 2020

skabashnyuk self-assigned this Apr 3, 2020

skabashnyuk added the area/che-server label Apr 3, 2020

skabashnyuk assigned metlos Apr 3, 2020

dmytro-ndp removed the area/qe label Apr 3, 2020

dmytro-ndp unassigned SkorikSergey Apr 3, 2020

dmytro-ndp removed this from the Backlog - QE milestone Apr 3, 2020

dmytro-ndp changed the title ~~OS OAuth E2E selenium tests are failing on nightly Jenkins jobs~~ 'rolebindings.authorization.openshift.io "exec" not found' error when start workspace with OS credentials on minishift Apr 3, 2020

skabashnyuk added this to the 7.12 milestone Apr 3, 2020

amisevsk added the severity/P1 Has a major impact to usage or development of the system. label Apr 3, 2020

tomgeorge mentioned this issue Apr 10, 2020

Add a role allowing che SA to stop workspaces #16532

Merged

metlos mentioned this issue Apr 15, 2020

Fix rolebinding issues in infras #16619

Merged

metlos closed this as completed in ee24066 Apr 16, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

'rolebindings.authorization.openshift.io "exec" not found' error when start workspace with OS credentials on minishift #16534

'rolebindings.authorization.openshift.io "exec" not found' error when start workspace with OS credentials on minishift #16534

dmytro-ndp commented Apr 2, 2020 •

edited

Loading

tolusha commented Apr 3, 2020 •

edited

Loading

tolusha commented Apr 3, 2020

skabashnyuk commented Apr 3, 2020

skabashnyuk commented Apr 3, 2020

amisevsk commented Apr 3, 2020

ibuziuk commented Apr 8, 2020 •

edited

Loading

skabashnyuk commented Apr 14, 2020

metlos commented Apr 14, 2020

'rolebindings.authorization.openshift.io "exec" not found' error when start workspace with OS credentials on minishift #16534

'rolebindings.authorization.openshift.io "exec" not found' error when start workspace with OS credentials on minishift #16534

Comments

dmytro-ndp commented Apr 2, 2020 • edited Loading

Describe the bug

Che version

Steps to reproduce

Runtime

Installation method

Environment

Eclipse Che Logs

Additional context

tolusha commented Apr 3, 2020 • edited Loading

tolusha commented Apr 3, 2020

skabashnyuk commented Apr 3, 2020

skabashnyuk commented Apr 3, 2020

amisevsk commented Apr 3, 2020

ibuziuk commented Apr 8, 2020 • edited Loading

skabashnyuk commented Apr 14, 2020

metlos commented Apr 14, 2020

dmytro-ndp commented Apr 2, 2020 •

edited

Loading

tolusha commented Apr 3, 2020 •

edited

Loading

ibuziuk commented Apr 8, 2020 •

edited

Loading