Use a daemonset with rshared mounts to mount FUSE

[yuvipanda]

Currently, each user mounts fuse themselves. This has negative security consequences, since they require privileged containers to do this.

Long term, the solution is to implement a Container Storage Interface driver for GCS FUSE. The CSI standard has wide adoption across multiple projects (mesos can also use it, for example), while FlexVolumes are kubernetes specific. FlexVolumes are also deprecated in Kubernetes now, and will be removed in a (far future) release. CSI is more flexible.

For the near term, it would be great to do something that lets us let go of GCS Fuse.

I'm assuming the following conditions are true for the FUSE usage:

1. Everyone has same access to the entire FUSE space (read/write)
2. We can upgrade to Kubernetes 1.10 (which should be on GKE in a few weeks)

We can use the new support for rshared mounts in kubernetes 1.10 to do the following:

1. Make a container that has all the software for doing the GCS Mount.
2. Run this container as a privileged daemonset - this makes it run on all nodes.
3. Mount GCSFuse as /data/gcsfuse on the host machine, via rshared mounts.
4. For each user node, mount /data/gcsfuse with hostPath into their user pod. They can use this for accessing GCSFuse without needing privileged access.

How does this sound?

[yuvipanda]

An alternative if we want to do this earlier is:

1. Switch node type to Ubuntu in GKE
2. Run something like https://github.com/berkeley-dsep-infra/data8xhub/tree/master/images/mounter in a daemonset. In that example, we run this script: https://github.com/berkeley-dsep-infra/data8xhub/blob/master/images/mounter/mounter.py on the host. We can instead run something that mounts GCS FUSE instead.

This can happen today if needed.

[mrocklin]

I agree that we probably don't need specialized permissions on a per-user basis. I would suggest that these be read-only. cc'ing @jhamman to verify

[jacobtomlinson]

This sounds great. We are currently using a flex volume solution for S3 access, but as you say that is deprecated.

I'm sure there will be a very wide interest for a CSI for all object stores. Not sure if this is something we want to tackle.

Otherwise I like your suggestion, the only issue from my side is that it depends on 1.10. We are using kops to build and manage our cluster right now which doesn't support 1.9 yet.

[yuvipanda]

For kops, you can enable `rshared` as an Alpha feature even in 1.8 - https://v1-8.docs.kubernetes.io/docs/concepts/storage/volumes/#mount- propagation. Alternatively, you can use the other solution I listed.

…

On Tue, Apr 3, 2018 at 1:32 AM, Jacob Tomlinson ***@***.***> wrote: This sounds great. We are currently using a flex volume solution for S3 access, but as you say that is deprecated. I'm sure there will be a very wide interest for a CSI for all object stores. Not sure if this is something we want to tackle. Otherwise I like your suggestion, the only issue from my side is that it depends on 1.10. We are using kops to build and manage our cluster right now which doesn't support 1.9 yet. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#190 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAB23pql1zgvNPfBlxkFHnu3eFMifeh-ks5tkzOBgaJpZM4TD0sd> .

-- Yuvi Panda T http://yuvi.in/blog

[jacobtomlinson]

Nice thanks!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not seen recent activity. The issue can always be reopened at a later date.

[yuvipanda]

I built https://github.com/yuvipanda/jupyterhub-roothooks/ to solve this!

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[yuvipanda]

Please, let this be, @github-actions bot.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[yuvipanda]

Can we turn off stale-bot? :(

[jhamman]

@yuvipanda - can this be closed out now? I wonder if we can transition this to a z2jh or 2i2c issue.

[yuvipanda]

yeah, I think we can probably turn this into a 'how to enable FUSE safely' issue on z2jh. Think I can convince you to do that, @jhamman? :D

[jhamman]

Thanks @yuvipanda - see jupyterhub/zero-to-jupyterhub-k8s#3300

[yuvipanda]

yay, thanks @jhamman!