Che 7 java-gradle stack is not working on che.openshift.io

[ibuziuk]

Description

Che 7 java-gradle stack is not working on che.openshift.io

Successfully pulled image "gradle:5.2.1-jdk11"
Error: Error response from daemon: create 180218ddb8332c0e15487f4ada59c23a9c644b7c19ca38206b8410f38cfb4c01: error while creating volume path '/var/lib/docker/volumes/180218ddb8332c0e15487f4ada59c23a9c644b7c19ca38206b8410f38cfb4c01/_data': mkdir /var/lib/docker/volumes/180218ddb8332c0e15487f4ada59c23a9c644b7c19ca38206b8410f38cfb4c01: permission denied

Reproduction Steps

• Create and start workspace using Che 7 java-gradle stack on che.openshift.io
• ERROR: workspace is failing to start (Error: Failed to run the workspace: "Waiting for Kubernetes environment 'default' of the workspace'workspace3xu5bl7049y91xmq' reached timeout")

OS and version:
che.openshift.io Che 7 beta 4

Diagnostics:

[ibuziuk]

funny thing that gradle stack seems to work fine on dev cluster, but not on che.openshift.io:

Error: Error response from daemon: create 4eee0aa36cb85e63ca6b3128f8067a44696f6506141f1d7d9db6e935582bb05e: error while creating volume path '/var/lib/docker/volumes/4eee0aa36cb85e63ca6b3128f8067a44696f6506141f1d7d9db6e935582bb05e/_data': mkdir /var/lib/docker/volumes/4eee0aa36cb85e63ca6b3128f8067a44696f6506141f1d7d9db6e935582bb05e: permission denied

[amisevsk]

It appears that the issue is due to the dockerfile used for the community gradle image.

The issue: Docker will implicitly use host storage to mount VOLUMEs in dockerfiles. Kubernetes can (and should be, for security reasons) be configured to not allow that. To avoid the issue, any volumes in the dockerfile should have a corresponding volume mounted in the pod spec.

The solution for the gradle stack is to add

    volumes:
      - containerPath: /home/gradle/.gradle
        name: gradle

to the workspace config -- an example devfile is here (using the current spec for devfiles).

As for finding a solution to our issue, I'm not sure how to proceed.

• This is a significant issue for the goal of using community images as stacks (since the user would need to be aware of any VOLUME instructions in the dockerfiles used). This may not be a huge issue however, since presumably images being used for workspaces would be runnable on the cluster already.
• We can update our devfile/stack to use /home/gradle/.gradle as a volume, but I'm not sure if this could cause issues (especially around userid, etc). There is a lot of unexplored ocean in terms of how this interacts with Most public images used in devfile registry do not work as the base for Che 7 workspaces on OpenShift #13454.

For now I've created PR eclipse-che/che-devfile-registry#14 as an attempted fix.

[amisevsk]

Some related external discussion around this sort of issue:

https://bugzilla.redhat.com/show_bug.cgi?id=1584386
kubernetes/kube-state-metrics#471

[ibuziuk]

Java Gradle devfile is part of the getting-started devfiles and it works just fine - https://www.eclipse.org/che/getting-started/cloud/