Enable S3/NFS mounts in interactive sessions

[ableuler]

Currently, all data that users want to access when working in interactive environments has to be downloaded into the environment, either during environment launch or manually by the user later on. This approach is not very user friendly and consumes unnecessary amounts of disk space and bandwitdh in scenarios where only a small subset of all the data contained in a dataset (or data folder, etc) effectively has to be accessed.

One way to improve this would be to allow users to FUSE-mount a volume/s3-bucket into their environments. See this thread on discourse for an example use case.

Letting users simply handle this themselves is unfortunately not an option because of the escalated privileges a container (at least until now, see for example this issue) needs in order to mount volumes. Instead, we could do the mount in a different container and make the mounted volume available to the user session. This would have to be realised through some dedicated component deployed as deamonSet on each node or as init- or sidecar container running alongside the jupyterlab container in the user session pod.

Open questions that have to be addressed before we can move forward on this issue include:

• What are the security implications of running privileged and unprivileged containers in the same pod? Can a privilege escalation be prevented? Do pod security policies help here?
• What would a deamonSet based solution look like? Would a volume be mounted on every node or only on the node running the user pod? How could we reliably manage paths of the mounted volumes on the nodes?
• What would the UX be around this?
• How do we reliably handle the users credentials?

[rokroskar]

Update - this might be a feasible way forward without having to manage the mounting ourselves: https://ibm.github.io/dataset-lifecycle-framework/

[olevski]

I could not get this to work on the dev cluster. But I only tried once and was too afraid to break things for other people so I stopped.

However, I deployed the terraform-renku config that is at commit 5bf0d5fcf33db9f48e4a52ce17df24ea4d234898 on main. Then I installed the https://ibm.github.io/dataset-lifecycle-framework/ library by following the instructions (it is a one line command that applies a manifest). They still do not have a helm chart.

After creating the crds that com from the DLF library I tested:

• creating and mounting a dataset in read-only mode with gcs storage
• creating and mounting a dataset in read-write mode with openstack object storage
• creating and mounting a dataset in read-write mode with gcs storage

All of the above scenarios work. I could see the changes reflected on the openstack page for object storage in read-write mode. I could also mount the same dataset to 2 pods at the same time. And changing things from one pod was reflected in another. I am not sure what happens when you try to edit the same file from two different places at the same time. It is probably not nice.

The only problem is that the pvc and pv created from the s3 bucket is always ~9000 gb large. I posted a question on the github page for DLF if we can control this (datashim-io/datashim#87)

These tests included only simple pods that ran busybox. My next step is to deploy renku in my cluster and test it out with renku. However I do not see a reason why the user pods would not work for this if regular pods did work.

So my next step is to deploy renku and test this out in a user pod.

The way that a DLF dataset k8s resources is created and mounted is very easy and would work with the renku spawner. We would require the user to provide s3 secret and acces keys, s3 api endpoint as well as bucket names that should be mounted.

[olevski]

For reference these are my manifests for creating the dataset crds which then create a pvc with the s3 bucket data:

# source https://github.com/IBM/dataset-lifecycle-framework/tree/master/examples/templates
---
apiVersion: com.ie.ibm.hpsys/v1alpha1
kind: Dataset
metadata:
  name: example-dataset-openstack
  namespace: tasko
spec:
  local:
    type: "COS"
    secret-name: "s3-dataset-secret-openstack" #see s3-secrets.yaml for an example
    secret-namespace: "tasko" #optional if the secret is in the same ns as dataset
    endpoint: "https://os.zhdk.cloud.switch.ch"
    bucket: "dlf-dataset-test-1"
    readonly: "false" # default is false
    region: "" #it can be empty
---
kind: Namespace
apiVersion: v1
metadata:
  name: tasko
---
apiVersion: com.ie.ibm.hpsys/v1alpha1
kind: Dataset
metadata:
  name: example-dataset-gcs
  namespace: tasko
spec:
  local:
    type: "COS"
    secret-name: "s3-dataset-secret-gcs" #see s3-secrets.yaml for an example
    secret-namespace: "tasko" #optional if the secret is in the same ns as dataset
    endpoint: "https://storage.googleapis.com"
    bucket: "tasko-dlf-test-1"
    readonly: "false" # default is false
    region: "europe-west6" #it can be empty
---
apiVersion: v1
kind: Secret
metadata:
  name: s3-dataset-secret-openstack
  namespace: tasko
stringData:
  accessKeyID: "xxxxxxxxxxx"
  secretAccessKey: "xxxxxxxxxxx"
---
apiVersion: v1
kind: Secret
metadata:
  name: s3-dataset-secret-gcs
  namespace: tasko
stringData:
  accessKeyID: "xxxxxxxxxxx"
  secretAccessKey: "xxxxxxxxxxx"

Then all I have to do is mount the pvcs in any pod I want:

apiVersion: v1
kind: Pod
metadata:
  name: test-dataset-pod
spec:
  volumes:
    - name: datasetpvc
      persistentVolumeClaim:
        claimName: example-dataset-openstack
  containers:
    - name: main
      image: busybox
      command:
        - sh
        - -c
      args: 
        - sleep 30000
      volumeMounts:
        - mountPath: "/dataset"
          name: datasetpvc

[ableuler]

Thanks for the update, sounds really good so far.

The only problem is that the pvc and pv created from the s3 bucket is always ~9000 gb large. I posted a question on the github page for DLF if we can control this (IBM/dataset-lifecycle-framework#87)

For me this makes sense since s3 buckets do typically not have a fixed size. So I guess since pvc's and pv's in k8s must have an indicated size, they just went for very large value. So as long as those huge pv's just abstract an object store and have no equivalent openstack volume, those 9000GB don't worry me at all.

[rokroskar]

@ableuler we might want to have a limit imposed by kubernetes on the usage though, precisely because buckets don't have a pre-defined size. I'm thinking of case where someone would want to make a dataset available via this mechanism, but they miscalculate the size. We want some way of preventing things from blowing up and enforcing quotas.

[olevski]

So I have another update.

I deployed renku separately and tested out the following:

• mounting a S3 bucket into a renku session
• mounting a NFS share into a renku session

It took me some time to properly set things up. Especially on the NFS part but things generally worked.

Some of the potential issues and problems:

• There are two backends for mounting S3 buckets (goofys and s3fs). According to the docs goofys is more performant but it would crash for me whenever I tried to save a file in read-write mode. Using s3fs I had no such issues.
• Both in the case of S3 and NFS the permissions that a file is created with are preserved and maintained. So for example if I create a file from within a renku session it has jovyan as the user and that is maintained when I open the file from another session. Uploading files to a S3 bucket from the openstack interface results in root being the owner. But they are still editable by any user.
• The folder that is mounted into the user session has root for the owner and group. This cannot be changed - trying to change it to jovyan during init as root results in errors or all data in the S3 bucket/prefix that is mounting being deleted.
• NFS is similar with user permissions. If my share location is setup to have certain permissions by default then those permissions are preserved when the files/folders are mounted. For example if the share location on the server is owned by root then mounting that location in a user session prevents anyone from writing into it. Also any files that have strict permissions are not editable in the user session.
• For NFS I tried to deploy a standalone NFS server in a small instance on GCE. I could then mount this into my own laptop and the mac I use for work. But I could not get this to mount into a renku user session with the DLF library. I believe that this was because of me not fully properly setting up the NFS server. I could get the DLF library to work by running these k8s manifests as NFS server in k8s.

Going forward here are a few suggestions:

• Create proposal for implementing DLF only in read-only mode. The library is really new so I think this will enable us to take on less risk and have a smaller feature that we deploy. We can see how this goes and later roll out read-write mode.
• Explore the possibility of using Linux libraries for mounting S3 buckets and NFS which would run before the jupyterhub user session is started. This way we do not rely on the DLF library but we do not have the convenience of having datasets as k8s resources. These libraries would have to be included in the renku/singleuser image. Also the mounting may require root privileges which we do not have when we run the singleuser image.

[olevski]

Also I kept stuff in terraform for the gcp setup so that I can easily bring this up for testing/experimentation. I created a private repo here: https://github.com/SwissDataScienceCenter/gcp-renku.

[rokroskar]

Nice @olevski! So if I understand this correctly you also used DLF for mounting NFS? If that's the case I'm not sure that's the obvious choice since there is standard k8s support for NFS volumes.

[olevski]

@rokroskar I did use DLF for mounting NFS, yes. I suspect that if DLF worked with Renku then mounting NFS via the regular k8s feature also works.

[ableuler]

Thanks for this exploratory work @olevski, looks really promising! I think with this information we can talk start to plan this feature in more detail. Therefore turning this into an epic.

[rokroskar]

@olevski yes, for sure mounting NFS using k8s works - I don't know what DLF adds in this case. It might also be easier to set the permissions the way you want in that case. Btw, to get around the permissions issue, what we do in the init container when the interactive sessions launch is that we run the init container as root and then do a chown on the entire checked out repo before the main container starts. That way the permissions are correct for the user. I guess something similar could work here?