Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Give notebook containers elevated privileges #379

Closed
mrocklin opened this issue Jan 5, 2018 · 10 comments
Closed

Give notebook containers elevated privileges #379

mrocklin opened this issue Jan 5, 2018 · 10 comments

Comments

@mrocklin
Copy link
Contributor

mrocklin commented Jan 5, 2018

Hi,

I would like to use a FUSE file system in my containers. To do this locally it seems that I need to pass the /dev/fuse device to the docker container and run it with elevated permissions. Locally I do the following:

docker run -it --device /dev/fuse --cap-add SYS_ADMIN --privileged daskdev/pangeo-notebook

Is there a way for me to specify these flags in my config.yaml file?
@yuvipanda
Copy link
Collaborator

Try the following (using a v0.6.0- chart)

hub:
  extraConfig: |
    from kubernetes import client
    def modify_pod_hook(spawner, pod):
        pod.spec.containers[0].security_context = client.V1SecurityContext(
            privileged=True,
            capabilities=client.V1Capabilities(
                add=['SYS_ADMIN']
            )
        )
        return pod
    c.KubeSpawner.modify_pod_hook = modify_pod_hook

singleuser:
  uid: 0
  gid: 0
  storage:
    extraVolumes:
      - name: fuse
        hostPath:
          - path: /dev/fuse
    extraVolumeMounts:
      - name: fuse
        mountPath: /dev/fuse

@mrocklin
Copy link
Contributor Author

mrocklin commented Jan 5, 2018

OK, I've given this a shot and I'm now getting 500 errors. What is the right way to diagnose this?

@yuvipanda
Copy link
Collaborator

yuvipanda commented Jan 5, 2018 via email

@mrocklin
Copy link
Contributor Author

mrocklin commented Jan 5, 2018

Directly after trying to log in.

500 : Internal Server Error
Failed to start your server. Please contact admin.

You can try restarting your server from the home page.

Logs

mrocklin@carbon:~/workspace/pangeo/gce$ kubectl logs hub-594749b96d-jj6v6 --namespace pangeo | tail -100
    HTTP response headers: HTTPHeaderDict({'Transfer-Encoding': 'chunked', 'Date': 'Fri, 05 Jan 2018 20:06:41 GMT', 'Content-Type': 'application/json', 'Audit-Id': '25aad988-fd44-4272-8f8b-c75e63a40d21'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod in version \"v1\" cannot be handled as a Pod: v1.Pod: Spec: v1.PodSpec: Volumes: []v1.Volume: v1.Volume: VolumeSource: HostPath: readObjectStart: expect { or n, parsing 417 ...stPath\": [... at {\"apiVersion\": \"v1\", \"kind\": \"Pod\", \"metadata\": {\"name\": \"jupyter-myuser\", \"labels\": {\"component\": \"singleuser-server\", \"app\": \"jupyterhub\", \"heritage\": \"jupyterhub\", \"hub.jupyter.org/username\": \"myuser\"}}, \"spec\": {\"initContainers\": [], \"securityContext\": {\"runAsUser\": 0, \"fsGroup\": 1000}, \"volumes\": [{\"persistentVolumeClaim\": {\"claimName\": \"claim-myuser\"}, \"name\": \"volume-myuser\"}, {\"name\": \"fuse\", \"hostPath\": [{\"path\": \"/dev/fuse\"}]}], \"containers\": [{\"args\": [\"jupyterhub-singleuser\", \"--ip=\\\"0.0.0.0\\\"\", \"--port=8888\", \"--NotebookApp.default_url=\\\"/lab\\\"\"], \"name\": \"notebook\", \"image\": \"daskdev/pangeo-notebook:latest\", \"env\": [{\"name\": \"JUPYTERHUB_API_URL\", \"value\": \"http://10.23.242.51:8081/hub/api\"}, {\"name\": \"GRANT_SUDO\", \"value\": \"yes\"}, {\"name\": \"MEM_GUARANTEE\", \"value\": \"1073741824\"}, {\"name\": \"JUPYTERHUB_HOST\", \"value\": \"\"}, {\"name\": \"DASK_SCHEDULER_ADDRESS\", \"value\": \"dask-scheduler:8786\"}, {\"name\": \"JPY_API_TOKEN\", \"value\": \"a52a9915474c4e708f0cde1d10c4d445\"}, {\"name\": \"GIT_COMMITTER_NAME\", \"value\": \"myuser\"}, {\"name\": \"EMAIL\", \"value\": \"myuser@local\"}, {\"name\": \"GIT_AUTHOR_NAME\", \"value\": \"myuser\"}, {\"name\": \"JUPYTERHUB_CLIENT_ID\", \"value\": \"user-myuser\"}, {\"name\": \"JUPYTERHUB_SERVICE_PREFIX\", \"value\": \"/user/myuser/\"}, {\"name\": \"JUPYTERHUB_ADMIN_ACCESS\", \"value\": \"1\"}, {\"name\": \"JUPYTERHUB_BASE_URL\", \"value\": \"/\"}, {\"name\": \"JUPYTERHUB_USER\", \"value\": \"myuser\"}, {\"name\": \"EXTRA_PIP_PACKAGES\", \"value\": \"gcsfs git+https://github.com/pydata/xarray.git git+https://github.com/alimanfoo/zarr.git\"}, {\"name\": \"JUPYTERHUB_API_TOKEN\", \"value\": \"a52a9915474c4e708f0cde1d10c4d445\"}, {\"name\": \"JUPYTERHUB_OAUTH_CALLBACK_URL\", \"value\": \"/user/myuser/oauth_callback\"}], \"imagePullPolicy\": \"IfNotPresent\", \"volumeMounts\": [{\"mountPath\": \"/home/jovyan\", \"name\": \"volume-myuser\"}, {\"mountPath\": \"/dev/fuse\", \"name\": \"fuse\"}], \"lifecycle\": {}, \"ports\": [{\"containerPort\": 8888, \"name\": \"notebook-port\"}], \"resources\": {\"limits\": {}, \"requests\": {\"memory\": 1073741824}}}]}}","reason":"BadRequest","code":400}
    
    
    
[E 2018-01-05 20:16:31.077 JupyterHub log:114] {
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36",
      "Cookie": "jupyter-hub-token=\"2|1:0|10:1515175555|17:jupyter-hub-token|44:ZmQ2NzE5YTVhMmNhNGNkMzgzYmJkOGY4OTNkMjRjMzU=|0fd7c7d1f8f043ad72331952662de4931fd49f3049bfedc8d5dd2c7a97d359ec\"; _xsrf=2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400",
      "Host": "35.188.222.190",
      "Accept-Language": "en-US,en;q=0.9",
      "X-Forwarded-Proto": "http,http",
      "X-Scheme": "http",
      "Referer": "http://35.188.222.190/user/myuser/lab?redirects=1",
      "Accept-Encoding": "gzip, deflate",
      "X-Forwarded-For": "10.128.0.38,10.20.5.12",
      "X-Xsrftoken": "2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400",
      "Connection": "close",
      "X-Forwarded-Host": "35.188.222.190",
      "Accept": "*/*",
      "X-Forwarded-Port": "80,80",
      "X-Real-Ip": "10.128.0.38",
      "X-Original-Uri": "/hub/user/myuser/api/terminals?1515183390974"
    }
[E 2018-01-05 20:16:31.077 JupyterHub log:122] 500 GET /hub/user/myuser/api/terminals?1515183390974 (myuser@10.128.0.38) 6.12ms
[I 2018-01-05 20:16:35.390 JupyterHub log:122] 302 GET /user/myuser/api/contents/?content=1&1515183395350 → /hub/user/myuser/api/contents/?content=1&1515183395350 (@10.128.0.38) 0.93ms
[E 2018-01-05 20:16:35.462 JupyterHub base:711] Preventing implicit spawn for myuser because last spawn failed: (400)
    Reason: error
    HTTP response headers: HTTPHeaderDict({'Transfer-Encoding': 'chunked', 'Date': 'Fri, 05 Jan 2018 20:06:41 GMT', 'Content-Type': 'application/json', 'Audit-Id': '25aad988-fd44-4272-8f8b-c75e63a40d21'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod in version \"v1\" cannot be handled as a Pod: v1.Pod: Spec: v1.PodSpec: Volumes: []v1.Volume: v1.Volume: VolumeSource: HostPath: readObjectStart: expect { or n, parsing 417 ...stPath\": [... at {\"apiVersion\": \"v1\", \"kind\": \"Pod\", \"metadata\": {\"name\": \"jupyter-myuser\", \"labels\": {\"component\": \"singleuser-server\", \"app\": \"jupyterhub\", \"heritage\": \"jupyterhub\", \"hub.jupyter.org/username\": \"myuser\"}}, \"spec\": {\"initContainers\": [], \"securityContext\": {\"runAsUser\": 0, \"fsGroup\": 1000}, \"volumes\": [{\"persistentVolumeClaim\": {\"claimName\": \"claim-myuser\"}, \"name\": \"volume-myuser\"}, {\"name\": \"fuse\", \"hostPath\": [{\"path\": \"/dev/fuse\"}]}], \"containers\": [{\"args\": [\"jupyterhub-singleuser\", \"--ip=\\\"0.0.0.0\\\"\", \"--port=8888\", \"--NotebookApp.default_url=\\\"/lab\\\"\"], \"name\": \"notebook\", \"image\": \"daskdev/pangeo-notebook:latest\", \"env\": [{\"name\": \"JUPYTERHUB_API_URL\", \"value\": \"http://10.23.242.51:8081/hub/api\"}, {\"name\": \"GRANT_SUDO\", \"value\": \"yes\"}, {\"name\": \"MEM_GUARANTEE\", \"value\": \"1073741824\"}, {\"name\": \"JUPYTERHUB_HOST\", \"value\": \"\"}, {\"name\": \"DASK_SCHEDULER_ADDRESS\", \"value\": \"dask-scheduler:8786\"}, {\"name\": \"JPY_API_TOKEN\", \"value\": \"a52a9915474c4e708f0cde1d10c4d445\"}, {\"name\": \"GIT_COMMITTER_NAME\", \"value\": \"myuser\"}, {\"name\": \"EMAIL\", \"value\": \"myuser@local\"}, {\"name\": \"GIT_AUTHOR_NAME\", \"value\": \"myuser\"}, {\"name\": \"JUPYTERHUB_CLIENT_ID\", \"value\": \"user-myuser\"}, {\"name\": \"JUPYTERHUB_SERVICE_PREFIX\", \"value\": \"/user/myuser/\"}, {\"name\": \"JUPYTERHUB_ADMIN_ACCESS\", \"value\": \"1\"}, {\"name\": \"JUPYTERHUB_BASE_URL\", \"value\": \"/\"}, {\"name\": \"JUPYTERHUB_USER\", \"value\": \"myuser\"}, {\"name\": \"EXTRA_PIP_PACKAGES\", \"value\": \"gcsfs git+https://github.com/pydata/xarray.git git+https://github.com/alimanfoo/zarr.git\"}, {\"name\": \"JUPYTERHUB_API_TOKEN\", \"value\": \"a52a9915474c4e708f0cde1d10c4d445\"}, {\"name\": \"JUPYTERHUB_OAUTH_CALLBACK_URL\", \"value\": \"/user/myuser/oauth_callback\"}], \"imagePullPolicy\": \"IfNotPresent\", \"volumeMounts\": [{\"mountPath\": \"/home/jovyan\", \"name\": \"volume-myuser\"}, {\"mountPath\": \"/dev/fuse\", \"name\": \"fuse\"}], \"lifecycle\": {}, \"ports\": [{\"containerPort\": 8888, \"name\": \"notebook-port\"}], \"resources\": {\"limits\": {}, \"requests\": {\"memory\": 1073741824}}}]}}","reason":"BadRequest","code":400}
    
    
[E 2018-01-05 20:16:35.463 JupyterHub web:1590] Uncaught exception GET /hub/user/myuser/api/contents/?content=1&1515183395350 (10.128.0.38)
    HTTPServerRequest(protocol='http', host='35.188.222.190', method='GET', uri='/hub/user/myuser/api/contents/?content=1&1515183395350', version='HTTP/1.1', remote_ip='10.128.0.38', headers={'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36', 'Cookie': 'jupyter-hub-token="2|1:0|10:1515175555|17:jupyter-hub-token|44:ZmQ2NzE5YTVhMmNhNGNkMzgzYmJkOGY4OTNkMjRjMzU=|0fd7c7d1f8f043ad72331952662de4931fd49f3049bfedc8d5dd2c7a97d359ec"; _xsrf=2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400', 'Host': '35.188.222.190', 'Accept-Language': 'en-US,en;q=0.9', 'X-Forwarded-Proto': 'http,http', 'X-Scheme': 'http', 'Referer': 'http://35.188.222.190/user/myuser/lab?redirects=1', 'Accept-Encoding': 'gzip, deflate', 'X-Forwarded-For': '10.128.0.38,10.20.5.12', 'X-Xsrftoken': '2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400', 'Connection': 'close', 'X-Forwarded-Host': '35.188.222.190', 'Accept': '*/*', 'X-Forwarded-Port': '80,80', 'X-Real-Ip': '10.128.0.38', 'X-Original-Uri': '/hub/user/myuser/api/contents/?content=1&1515183395350'})
    Traceback (most recent call last):
      File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 1511, in _execute
        result = yield result
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py", line 713, in get
        raise copy.copy(exc).with_traceback(exc.__traceback__)
      File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 1511, in _execute
        result = yield result
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py", line 747, in get
        yield self.spawn_single_user(current_user)
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py", line 475, in spawn_single_user
        yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future)
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py", line 445, in finish_user_spawn
        yield spawn_future
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/user.py", line 439, in spawn
        raise e
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/user.py", line 378, in spawn
        ip_port = yield gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/usr/local/lib/python3.5/dist-packages/kubespawner/spawner.py", line 917, in start
        pod
      File "/usr/lib/python3.5/concurrent/futures/_base.py", line 398, in result
        return self.__get_result()
      File "/usr/lib/python3.5/concurrent/futures/_base.py", line 357, in __get_result
        raise self._exception
      File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
        result = self.fn(*self.args, **self.kwargs)
      File "/usr/local/lib/python3.5/dist-packages/kubespawner/spawner.py", line 888, in asynchronize
        return method(*args, **kwargs)
      File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6561, in create_namespaced_pod
        (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
      File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6651, in create_namespaced_pod_with_http_info
        collection_formats=collection_formats)
      File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/api_client.py", line 335, in call_api
        _preload_content, _request_timeout)
      File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/api_client.py", line 148, in __call_api
        _request_timeout=_request_timeout)
      File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/api_client.py", line 393, in request
        body=body)
      File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/rest.py", line 287, in POST
        body=body)
      File "/usr/local/lib/python3.5/dist-packages/kubernetes/client/rest.py", line 240, in request
        raise ApiException(http_resp=r)
    kubernetes.client.rest.ApiException: (400)
    Reason: error
    HTTP response headers: HTTPHeaderDict({'Transfer-Encoding': 'chunked', 'Date': 'Fri, 05 Jan 2018 20:06:41 GMT', 'Content-Type': 'application/json', 'Audit-Id': '25aad988-fd44-4272-8f8b-c75e63a40d21'})
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod in version \"v1\" cannot be handled as a Pod: v1.Pod: Spec: v1.PodSpec: Volumes: []v1.Volume: v1.Volume: VolumeSource: HostPath: readObjectStart: expect { or n, parsing 417 ...stPath\": [... at {\"apiVersion\": \"v1\", \"kind\": \"Pod\", \"metadata\": {\"name\": \"jupyter-myuser\", \"labels\": {\"component\": \"singleuser-server\", \"app\": \"jupyterhub\", \"heritage\": \"jupyterhub\", \"hub.jupyter.org/username\": \"myuser\"}}, \"spec\": {\"initContainers\": [], \"securityContext\": {\"runAsUser\": 0, \"fsGroup\": 1000}, \"volumes\": [{\"persistentVolumeClaim\": {\"claimName\": \"claim-myuser\"}, \"name\": \"volume-myuser\"}, {\"name\": \"fuse\", \"hostPath\": [{\"path\": \"/dev/fuse\"}]}], \"containers\": [{\"args\": [\"jupyterhub-singleuser\", \"--ip=\\\"0.0.0.0\\\"\", \"--port=8888\", \"--NotebookApp.default_url=\\\"/lab\\\"\"], \"name\": \"notebook\", \"image\": \"daskdev/pangeo-notebook:latest\", \"env\": [{\"name\": \"JUPYTERHUB_API_URL\", \"value\": \"http://10.23.242.51:8081/hub/api\"}, {\"name\": \"GRANT_SUDO\", \"value\": \"yes\"}, {\"name\": \"MEM_GUARANTEE\", \"value\": \"1073741824\"}, {\"name\": \"JUPYTERHUB_HOST\", \"value\": \"\"}, {\"name\": \"DASK_SCHEDULER_ADDRESS\", \"value\": \"dask-scheduler:8786\"}, {\"name\": \"JPY_API_TOKEN\", \"value\": \"a52a9915474c4e708f0cde1d10c4d445\"}, {\"name\": \"GIT_COMMITTER_NAME\", \"value\": \"myuser\"}, {\"name\": \"EMAIL\", \"value\": \"myuser@local\"}, {\"name\": \"GIT_AUTHOR_NAME\", \"value\": \"myuser\"}, {\"name\": \"JUPYTERHUB_CLIENT_ID\", \"value\": \"user-myuser\"}, {\"name\": \"JUPYTERHUB_SERVICE_PREFIX\", \"value\": \"/user/myuser/\"}, {\"name\": \"JUPYTERHUB_ADMIN_ACCESS\", \"value\": \"1\"}, {\"name\": \"JUPYTERHUB_BASE_URL\", \"value\": \"/\"}, {\"name\": \"JUPYTERHUB_USER\", \"value\": \"myuser\"}, {\"name\": \"EXTRA_PIP_PACKAGES\", \"value\": \"gcsfs git+https://github.com/pydata/xarray.git git+https://github.com/alimanfoo/zarr.git\"}, {\"name\": \"JUPYTERHUB_API_TOKEN\", \"value\": \"a52a9915474c4e708f0cde1d10c4d445\"}, {\"name\": \"JUPYTERHUB_OAUTH_CALLBACK_URL\", \"value\": \"/user/myuser/oauth_callback\"}], \"imagePullPolicy\": \"IfNotPresent\", \"volumeMounts\": [{\"mountPath\": \"/home/jovyan\", \"name\": \"volume-myuser\"}, {\"mountPath\": \"/dev/fuse\", \"name\": \"fuse\"}], \"lifecycle\": {}, \"ports\": [{\"containerPort\": 8888, \"name\": \"notebook-port\"}], \"resources\": {\"limits\": {}, \"requests\": {\"memory\": 1073741824}}}]}}","reason":"BadRequest","code":400}
    
    
    
[E 2018-01-05 20:16:35.471 JupyterHub log:114] {
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36",
      "Cookie": "jupyter-hub-token=\"2|1:0|10:1515175555|17:jupyter-hub-token|44:ZmQ2NzE5YTVhMmNhNGNkMzgzYmJkOGY4OTNkMjRjMzU=|0fd7c7d1f8f043ad72331952662de4931fd49f3049bfedc8d5dd2c7a97d359ec\"; _xsrf=2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400",
      "Host": "35.188.222.190",
      "Accept-Language": "en-US,en;q=0.9",
      "X-Forwarded-Proto": "http,http",
      "X-Scheme": "http",
      "Referer": "http://35.188.222.190/user/myuser/lab?redirects=1",
      "Accept-Encoding": "gzip, deflate",
      "X-Forwarded-For": "10.128.0.38,10.20.5.12",
      "X-Xsrftoken": "2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400",
      "Connection": "close",
      "X-Forwarded-Host": "35.188.222.190",
      "Accept": "*/*",
      "X-Forwarded-Port": "80,80",
      "X-Real-Ip": "10.128.0.38",
      "X-Original-Uri": "/hub/user/myuser/api/contents/?content=1&1515183395350"
    }
[E 2018-01-05 20:16:35.471 JupyterHub log:122] 500 GET /hub/user/myuser/api/contents/?content=1&1515183395350 (myuser@10.128.0.38) 10.68ms
mrocklin@carbon:~/workspace/pangeo/gce$

@yuvipanda
Copy link
Collaborator

@mrocklin looks like the error is in:

    extraVolumes:
      - name: fuse
        hostPath:
          - path: /dev/fuse

Specifically, the '-' before the last line. Can you replace with:

    extraVolumes:
      - name: fuse
        hostPath:
          path: /dev/fuse

and try?

@mrocklin
Copy link
Contributor Author

mrocklin commented Jan 5, 2018

Still failing

mrocklin@carbon:~/workspace/pangeo/gce$ kubectl logs jupyter-mrocklin --namespace pangeo
+ '[' -e /opt/app/environment.yml ']'
+ echo 'no environment.yml'
+ '[' '' ']'
+ '[' 'gcsfs git+https://github.com/pydata/xarray.git git+https://github.com/alimanfoo/zarr.git' ']'
+ echo 'EXTRA_PIP_PACKAGES environment variable found.  Installing.'
+ /opt/conda/bin/pip install gcsfs git+https://github.com/pydata/xarray.git git+https://github.com/alimanfoo/zarr.git
no environment.yml
EXTRA_PIP_PACKAGES environment variable found.  Installing.
Collecting git+https://github.com/pydata/xarray.git
  Cloning https://github.com/pydata/xarray.git to /tmp/pip-528fg1jr-build
Collecting git+https://github.com/alimanfoo/zarr.git
  Cloning https://github.com/alimanfoo/zarr.git to /tmp/pip-bi40oti8-build
Requirement already satisfied: gcsfs in /opt/conda/lib/python3.6/site-packages
Requirement already satisfied: numpy>=1.11 in /opt/conda/lib/python3.6/site-packages (from xarray==0.10.0+dev24.g5a28b89)
Requirement already satisfied: pandas>=0.18.0 in /opt/conda/lib/python3.6/site-packages (from xarray==0.10.0+dev24.g5a28b89)
Collecting asciitree (from zarr==2.2.0a2.dev176)
Collecting fasteners (from zarr==2.2.0a2.dev176)
  Using cached fasteners-0.14.1-py2.py3-none-any.whl
Collecting numcodecs>=0.5.2 (from zarr==2.2.0a2.dev176)
Requirement already satisfied: oauth2client>=1.5.2 in /opt/conda/lib/python3.6/site-packages (from gcsfs)
Requirement already satisfied: requests in /opt/conda/lib/python3.6/site-packages (from gcsfs)
Requirement already satisfied: python-dateutil>=2 in /opt/conda/lib/python3.6/site-packages (from pandas>=0.18.0->xarray==0.10.0+dev24.g5a28b89)
Requirement already satisfied: pytz>=2011k in /opt/conda/lib/python3.6/site-packages (from pandas>=0.18.0->xarray==0.10.0+dev24.g5a28b89)
Collecting monotonic>=0.1 (from fasteners->zarr==2.2.0a2.dev176)
  Using cached monotonic-1.4-py2.py3-none-any.whl
Requirement already satisfied: six in /opt/conda/lib/python3.6/site-packages (from fasteners->zarr==2.2.0a2.dev176)
Requirement already satisfied: pyasn1-modules>=0.0.5 in /opt/conda/lib/python3.6/site-packages (from oauth2client>=1.5.2->gcsfs)
Requirement already satisfied: httplib2>=0.9.1 in /opt/conda/lib/python3.6/site-packages (from oauth2client>=1.5.2->gcsfs)
Requirement already satisfied: pyasn1>=0.1.7 in /opt/conda/lib/python3.6/site-packages (from oauth2client>=1.5.2->gcsfs)
Requirement already satisfied: rsa>=3.1.4 in /opt/conda/lib/python3.6/site-packages (from oauth2client>=1.5.2->gcsfs)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/lib/python3.6/site-packages (from requests->gcsfs)
Requirement already satisfied: idna<2.7,>=2.5 in /opt/conda/lib/python3.6/site-packages (from requests->gcsfs)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in /opt/conda/lib/python3.6/site-packages (from requests->gcsfs)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.6/site-packages (from requests->gcsfs)
Installing collected packages: xarray, asciitree, monotonic, fasteners, numcodecs, zarr
  Found existing installation: xarray 0.10.0
    Uninstalling xarray-0.10.0:
      Successfully uninstalled xarray-0.10.0
  Running setup.py install for xarray: started
mrocklin@carbon:~/workspace/pangeo/gce$ kubectl logs jupyter-mrocklin --namespace pangeo
failed to get container status {"" ""}: rpc error: code = OutOfRange desc = EOFmrocklin@carbon:~/workspace/pangeo/gce$ kubectl logs jupyter-mrocklin --namespace pangeo
failed to get container status {"" ""}: rpc error: code = OutOfRange desc = EOFmrocklin@carbon:~/workspace/pangeo/gce$ kubectl logs jupyter-mrocklin --namespace pangeo
failed to get container status {"" ""}: rpc error: code = OutOfRange desc = EOFmrocklin@carbon:~/workspace/pangeo/gce$ kubectl logs jupyter-mrocklin --namespace pangeo
Error from server (NotFound): pods "jupyter-mrocklin" not found

@mrocklin
Copy link
Contributor Author

mrocklin commented Jan 5, 2018

logs from the hub

[E 2018-01-05 20:40:11.087 JupyterHub log:122] 500 GET /hub/user/myuser/api/terminals?1515184810974 (myuser@10.128.0.41) 6.30ms
[I 2018-01-05 20:40:11.382 JupyterHub log:122] 302 GET /user/myuser/api/contents/?content=1&1515184811351 → /hub/user/myuser/api/contents/?content=1&1515184811351 (@10.128.0.41) 0.71ms
[E 2018-01-05 20:40:11.439 JupyterHub base:711] Preventing implicit spawn for myuser because last spawn failed: Server at http://10.20.4.28:8888/user/myuser/ didn't respond in 30 seconds
[E 2018-01-05 20:40:11.439 JupyterHub web:1590] Uncaught exception GET /hub/user/myuser/api/contents/?content=1&1515184811351 (10.128.0.41)
    HTTPServerRequest(protocol='http', host='35.188.222.190', method='GET', uri='/hub/user/myuser/api/contents/?content=1&1515184811351', version='HTTP/1.1', remote_ip='10.128.0.41', headers={'Accept-Encoding': 'gzip, deflate', 'X-Forwarded-Proto': 'http,http', 'X-Scheme': 'http', 'Host': '35.188.222.190', 'X-Real-Ip': '10.128.0.41', 'Connection': 'close', 'Accept': '*/*', 'Referer': 'http://35.188.222.190/user/myuser/lab?redirects=1', 'Cookie': 'jupyter-hub-token="2|1:0|10:1515175555|17:jupyter-hub-token|44:ZmQ2NzE5YTVhMmNhNGNkMzgzYmJkOGY4OTNkMjRjMzU=|0fd7c7d1f8f043ad72331952662de4931fd49f3049bfedc8d5dd2c7a97d359ec"; _xsrf=2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400', 'X-Forwarded-For': '10.128.0.41,10.20.5.12', 'X-Original-Uri': '/hub/user/myuser/api/contents/?content=1&1515184811351', 'Accept-Language': 'en-US,en;q=0.9', 'X-Forwarded-Host': '35.188.222.190', 'X-Forwarded-Port': '80,80', 'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36', 'X-Xsrftoken': '2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400'})
    Traceback (most recent call last):
      File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 1511, in _execute
        result = yield result
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py", line 713, in get
        raise copy.copy(exc).with_traceback(exc.__traceback__)
      File "/usr/local/lib/python3.5/dist-packages/tornado/web.py", line 1511, in _execute
        result = yield result
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py", line 720, in get
        yield gen.with_timeout(timedelta(seconds=self.slow_spawn_timeout), spawner._spawn_future)
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/handlers/base.py", line 445, in finish_user_spawn
        yield spawn_future
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/user.py", line 476, in spawn
        raise e
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/user.py", line 450, in spawn
        resp = yield server.wait_up(http=True, timeout=spawner.http_timeout)
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/utils.py", line 180, in wait_for_http_server
        timeout=timeout
      File "/usr/local/lib/python3.5/dist-packages/jupyterhub/utils.py", line 135, in exponential_backoff
        raise TimeoutError(fail_message)
    TimeoutError: Server at http://10.20.4.28:8888/user/myuser/ didn't respond in 30 seconds
    
[E 2018-01-05 20:40:11.444 JupyterHub log:114] {
      "Accept-Encoding": "gzip, deflate",
      "X-Forwarded-Proto": "http,http",
      "X-Scheme": "http",
      "Host": "35.188.222.190",
      "X-Real-Ip": "10.128.0.41",
      "Connection": "close",
      "Accept": "*/*",
      "Referer": "http://35.188.222.190/user/myuser/lab?redirects=1",
      "Cookie": "jupyter-hub-token=\"2|1:0|10:1515175555|17:jupyter-hub-token|44:ZmQ2NzE5YTVhMmNhNGNkMzgzYmJkOGY4OTNkMjRjMzU=|0fd7c7d1f8f043ad72331952662de4931fd49f3049bfedc8d5dd2c7a97d359ec\"; _xsrf=2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400",
      "X-Forwarded-For": "10.128.0.41,10.20.5.12",
      "X-Original-Uri": "/hub/user/myuser/api/contents/?content=1&1515184811351",
      "Accept-Language": "en-US,en;q=0.9",
      "X-Forwarded-Host": "35.188.222.190",
      "X-Forwarded-Port": "80,80",
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.108 Safari/537.36",
      "X-Xsrftoken": "2|ff97bf3b|f91d3a1baea6ed59e6e3d8326818f465|1515176400"
    }
[E 2018-01-05 20:40:11.445 JupyterHub log:122] 500 GET /hub/user/myuser/api/contents/?content=1&1515184811351 (myuser@10.128.0.41) 6.84ms

@yuvipanda
Copy link
Collaborator

We spent some more time and figured it all out :)

@yuvipanda
Copy link
Collaborator

It was the singleuser.cmd, which we default to jupyterhub-singleuser. It needed to be set to start.sh instead I believe.

@ChenLong2014 ChenLong2014 mentioned this issue Jun 22, 2019
Closed
@avinson
Copy link

avinson commented Jun 11, 2020

If anyone comes across this issue in the future, I've created a jupyter image and a helm config fragment for mounting GCS storage via fuse here: https://github.com/noahhomes/r-mkl-notebook

@unkcpz unkcpz mentioned this issue Nov 18, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@yuvipanda @mrocklin @avinson