List other UI servers running

[kinow]

We will have one UI server per user. And each UI server will look after one or more workflows.

We want to allow users to access each other's UI server and see their running workflows - subjected to proper authorization, of course.

We need to define what is the workflow going to look like.

• Is user A going to configure his/her workflow to authorize that user B can access it?
• Is user A going to share the link of his UI server with user B, or will user B have a way to automatically see what suites s/he has access to?

[matthewrmshin]

From the perspective of user B, a discoverable UI would be the better option. (It is also clearer what views/commands are available to user B, so the UI can be better tested.)

[kinow]

Shared on Riot, copying here

Scenario: cylc-ui assets built, then started jupyterhub with cylc-singleuser (LocalProcessSpawner, normal settings). Open the UI in a browser tab, and take note of the UI Server pid

Case 1: kill the Hub
What happens: UI server still running, configurable-http-proxy too. Once the Hub is restarted, it keeps using the old proxy and UI server with no issues, for me in the browser it's as if nothing happened.

Case 2: kill the proxy
What happens: The Hub logs that the proxy stopped with some exit code (-9) and restarts it. Everything working as normal in 1 second.

Case 3: kill the UI server
What happens: Refreshing the tab I get a message " 503 : Service Unavailable " and the text " Your server appears to be down. Try restarting it from the hub ". The Hub logs that the user server stopped with exit code 0, and removes my user URL's from the proxy. But I can go to the hub page and start it again, then everything works fine.

Case 4: kill the Hub and the Proxy
What happens: The cylc-singleuser (UI Server) keeps running with no issues. Starting jupyterhub, it also starts the configurable-http-proxy again. Reloading the browser tab, the proxy reused the existing UI Server. It even logged that it found a missing route: "[W 2019-06-05 22:42:25.148 JupyterHub proxy:344] Adding missing route for /user/kinow/ (Server(url=http://127.0.0.1:33761/user/kinow/, bind_url=http://127.0.0.1:33761/user/kinow/))". So I think the proxy is keeping track of the routes and possibly keeping the route table on disk (will investigate)

[kinow]

When testing the SSH Spawner I had to troubleshoot the proxy too (had to troubleshoot pretty much the whole setup 😫).

Learned a good trick that might be helpful here, so note to self (or to whoever works on this one):

• Start or build cylc-ui
• Run export CONFIGPROXY_AUTH_TOKEN=$(openssl rand -hex 32) and echo $CONFIGPROXY_AUTH_TOKEN, copy the token value
• Now you can start cylc-uiserver, running jupyterhub. The Hub reads this config (search for it in the jupyterhub_config.py, it is mentioned there) and uses the token to communicate with the proxy via the REST API
• Now you can also query the proxy with that token with curl -XGET -H "Authorization: token $TOKEN" http://localhost:8001/api/routes

Example response:

{
    "/":{
        "hub":true,
        "target":"http://127.0.0.1:8081",
        "jupyterhub":true,
        "last_activity":"2019-06-05T11:09:56.477Z"
    },
    "/user/kinow":{
        "user":"kinow",
        "server_name":"",
        "target":"http://127.0.0.1:44755",
        "jupyterhub":true,
        "last_activity":"2019-06-05T11:09:57.888Z"
    }
}

[kinow]

One important thing, that I have also been postponing investigating. To save resources - I assume - if you run jupyterhub and spawns a cylc_singleuser process... but you do not use it, the Hub eventually kills that UI Server.

Question: Is that desired for Cylc UI Server?

Reason for asking, is that a UI Server may start a workflow that runs... well... forever? If the UI Server dies, then the feature discussed in this issue is directly affected.

Example: Bruno starts workflow five-forever, and goes home. Then the UI Server is killed by the Hub. Hilary logs in to the Hub, and wants to list the UI servers running... he won't find my UI Server that was killed due to inactivity (he may have rights to start it, but only if he wants/needs to, he will do it). So even though I have a workflow that could affect Hilary's work, or that he could be looking for, my UI Server won't be listed, as it's not in the proxy route table/sqlite DB.

[kinow]

Hmmm, looks like it's not killing inactive notebooks/cylc_singleuser. Maybe it was during development and I may have killed a process or two.

There is a default setting for inactivity, that defaults to 300 seconds. It's possible to see that after a while and without looking at the UI Server, the Hub keeps "checking routes". I believe it will remove the ones that do not reply after each inactivity interval.

It is also possible to see the same PID running for well over 10 minutes, without any use, and not being killed.

So ignore my previous question/comment 😬

[dpmatthews]

Even if we're not yet, I think we should we killing (or shutting down) inactive UI servers

[kinow]

Spent the afternoon trying to use the REST Services from within the hub, but without much success. Only got 403, but learned a bit more about the endpoints, its access control... will try again another day and try to document how much we can rely on the hub for listing other user's UI servers.

[kinow]

@dpmatthews

Even if we're not yet, I think we should we killing (or shutting down) inactive UI servers

I haven't fully tested JupyterHub, but my understanding is that it won't kill the UI Server, even if inactive. But it provides an example of a JupyterHub Service: cull-idle

If you start that service, it will read the jupyterhub_config.py, and look for a timeout. Then it will query a service in the UI Service to check its activity. The timeout setting is global, so it wouldn't be possible to cull idle server based on different timeouts (unless we extended the example).

But at least that's yet another thing we can tell users to use something from JupyterHub project :)

[kinow]

Prepared a draft pull request that lists all the UI servers running in a Hub, using the Configurable HTTP Proxy REST API.

No authorization layer here, so in theory with that change you would be able to see all the UI servers, and also open all the UI servers (perhaps useful for development).

[dpmatthews]

I'd have thought what we need to do is present users with a list of what UI servers they are authorised to access (whether they are running or not). I don't think normal users should ever need to worry about what UI servers are currently running.

[kinow]

Oh I see. So for that we may not even need to access the hub or proxy apis. That info could be stored somewhere else like a Db (maybe the same with authorization info).

But at least I learned a bit more about JupyterHub with this experiment :-)

[kinow]

There's some other work happening on JupyterHub that may benefit us: jupyterhub/jupyterhub#1758 (comment)

Specifically this part:

### Fetch list of running servers

Use cases:
- as user: give me the list of my servers
- as collaborator: give me the list of servers I have access to
- as teacher of a class, give me the list of my student's server (filtering by the computing environment)

If they improve the REST API, and we get methods for this, maybe backed by some changes in their database, then we won't have too much work left to get that in the UI&UIServer.

[oliver-sanders]

From CylcCon2020:

We will implement a configuration file for assigning authorisation to users in a read/write/execute type fashion (with some finer grained privilege control).

• The default should be no access.
• Users will be able to delegate authorisation.
• Sites should be able to restrict delegations.

[oliver-sanders]

Not mentioned at CylcCon2020:

It would be nice to permit temporary authorisation, e.g. for support purposes, ideally this would fit in with the same configuration file scheme.

[hjoliver]

(For authorization, see #10 )

[oliver-sanders]

Authorisation closed with #10

Listing other users servers is now covered by #273