Test Cylc UI Server with SSH Spawner #46

kinow · 2019-05-27T04:58:31Z

The current set up used by Python UI Server and JS Vue.js UI uses the LocalProcessSpawner.

We also tested the PBS Batch Spawner. But we have not concluded the SSH Spawner tests yet.

@MartinRyan has a work-in-progress SSH spawner (private repository), and had forked the uiserver too.

As the requirement here seems to be simply validating spawning the UI Server via SSH, I think an alternative could be the existing sshspawner by NERSC (National Energy Research Scientific Computing Center, US), which appears to be an implementation that has not been deprecated (the zonca one is not being maintained).

Martin's implementation used paramiko, which is an excellent tool (I had to use it some time ago, can't remember why, but remember it worked well). While NERSC's uses asyncssh, which also implements SSHv2, but tailored for Python 3.4. I think both are OK.

Other posts seem to suggest using the BatchSpawner as replacement for SSH spawning, but that's actually as a replacement for users who were triggering HPC batch schedulers like PBS. So not exactly what we are looking for.

kinow · 2019-05-27T05:23:13Z

I will be using Docker for the tests, but won't create new containers, as they current images need to be updated for Cylc 8.

Instead, I will savage the distributed images, but just for the SSH part. Once docker-compose up is done, I will docker-exec ... /bin/bash then execute pip, apt-get, etc.

Bad practice, but I just want to get two containers with everything that I need, and some mapped volumes. Then copy files where they need to be according to the ssh spawner by NERSC.

kinow · 2019-05-27T07:33:18Z

Started container with CYLC_SSH_PUBKEY=$(cat id_rsa.pub) docker-compose up -d --scale cylc-ssh=1 --scale cylc-jump-box=1.

Logged in to both with docker-compose exec cylc-ssh /bin/bash and docker-compose exec cylc-jump-box /bin/bash. The re-created caches with apt update in both boxes.

Then cd /opt and cloned the NERSC tool to both boxes git clone https://github.com/NERSC/sshspawner.git, followed by pip install ..

As the images are using Ubuntu 16.xx, I had to install Python 3 through Anaconda to both boxes too.

Cloned cylc-uiserver too to both boxes, git clone https://github.com/cylc/cylc-uiserver.git, followed by pip install ..

Then for cylc-ui, installed only on the the ssh box (not to the jump box one). git clone https://github.com/kinow/cylc-ui.git, followed by manually installing nvm from git, then nvm install 12.0.0 and nvm nvm alias default 12.0.0. Producing the final archives with npm run build.

And for the jupyterhub_config.py:

c.JupyterHub.spawner_class = 'sshspawner.sshspawner.SSHSpawner'

c.SSHSpawner.remote_hosts = ['cylc-ssh']
c.SSHSpawner.remote_port = '22'
c.SSHSpawner.ssh_command = 'ssh'

c.SSHSpawner.path = '/root/anaconda3/bin:/root/anaconda3/condabin:/opt/sshspawner/scripts/:/opt/cylc/bin:/usr/bin/:/opt/cylc/bin:/usr/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'
c.SSHSpawner.remote_port_command = '/root/anaconda3/bin/python /opt/sshspawner/scripts/get_port.py'

c.Spawner.cmd = ['cylc-singleuser']
c.Spawner.args = ['-s', '/opt/cylc-ui/dist/']

Oh, also installed nvm on the jump box, to install configurable-http-proxy.

Had some issues with PAM on Docker, and had forgotten to map a port (and had to hack my way through iptables). Will continue tomorrow, but I think quite close to being able to starting the remote process.

kinow · 2019-05-27T23:32:27Z

The Docker images would need a few more kicks until they were ready to be used. But having issues with PAM is no fun. So this morning tried something different. I believe this can be used to validate the NERSC sshspawner.

Followed the simple instructions from Docker docs to run a sshd container
Logged in via localhost:$randomport, adduser kinow, then copied my public key
Installed some utilities like net-tools, wget, vim, bzip2, then added Anaconda Python 3, nvm, git
Cloned cylc-ui and did all npm commands to get the dist/ folder with the UI HTML+JS+etc files
Cloned cylc-uiserver and sshspawner projects and installed both with pip
In my local computer, configured the the jupyterhub_config.py settings to spawn via SSH a UI server in the Docker container (important to match directories & user name)

c.JupyterHub.spawner_class = 'sshspawner.sshspawner.SSHSpawner'
c.JupyterHub.hub_ip = "172.17.0.1"
c.JupyterHub.hub_connect_ip = '172.17.0.1'
c.JupyterHub.proxy_api_ip = '172.17.0.1'
#c.JupyterHub.api_url = 'http://172.17.0.1:8081'
#c.JupyterHub.hub_bind_url = 'http://172.17.0.1:8081'
c.ConfigurableHTTPProxy.api_url = 'http://172.17.0.1:8100'
c.JupyterHub.port = 8099

c.SSHSpawner.remote_hosts = ['localhost']
c.SSHSpawner.remote_port = '32768'
c.SSHSpawner.ssh_command = 'ssh'
c.SSHSpawner.ssh_keyfile = '/home/kinow/.ssh/id_rsa'

c.SSHSpawner.path = '/home/kinow/Development/python/workspace/sshspawner:/home/kinow/Development/python/workspace/cylc-uiserver/venv/bin:/home/kinow/.nvm/versions/node/v11.6.0/bin:/home/kinow/bin:/opt/ant/bin::/home/kinow/bin:/home/kinow/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin:/home/kinow/.rvm/bin'
c.SSHSpawner.remote_port_command = '/home/kinow/anaconda3/bin/python /home/kinow/Development/python/workspace/sshspawner/scripts/get_port.py --ip'

c.Spawner.args = ['-s', '/home/kinow/Development/python/workspace/cylc-ui/dist/']

c.Spawner.cmd = ['/home/kinow/anaconda3/bin/cylc-singleuser']

Started jupyterhub, logged-in

Unfortunately there are a few issues with that spawner, but that we can work around or fix later.

It has a script to find the random port in the remote host - from what I could understand. The script has a function that can return port and IP, however, the code is expecting IP and port. Tried fixing that by changing c.SSHSpawner.remote_port_command to append a | awk '{print $2 $1}' but then the command failed to work properly :/
The documentation only tells you to use the c.SSHSpawner.remote_port_command, it doesn't mention that the scripts/get_port.py returns only the port (as per name), unless you specify --ip
I have a container with SSH listening to localhost on a random port 30000+, but when running the spawner, to retrieve the remote port it insisted on using the port 22, even though in another part of the code (by hacking the script and putting a print statement) I confirmed it read my configuration to use the different port number... fixed it by simply forwarding my ports with socat tcp-listen:22,reuseaddr,fork tcp:localhost:32768
The documentation states that the certificate is optional, but it is actually required, as the code fails if the Path object created is not found. So fixed it by creating a fake local cert to match the name expected by the script ssh-keygen -s ~/.ssh/id_rsa -I host_auth_server -h -n auth.example.com -V +52w ~/.ssh/id_rsa.pub (this creates ~/.ssh/id_rsa-cert.pub)
In the end it was easier for me to hack sshspawner.py and return the tuple return ("172.17.0.2", 8000) in async def remote_random_port... horrible, but I suspect there is something wrong with either docs or spawner code

Finally, once it's up and running, I can confirm it spawned the remote UI Server, but the configuration is still missing something. The spawner logging is extremely useful. It displays what is the local Shell script created in the temporary folder, that is executed via SSH to spawn the UI server. e.g.

[D 2019-05-28 11:12:20.124 JupyterHub sshspawner:245] /tmp/kinow_run.sh was written as:
    #!/bin/bash
    export PATH=/home/kinow/Development/python/workspace/sshspawner:/home/kinow/Development/python/workspace/cylc-uiserver/venv/bin:/home/kinow/.nvm/versions/node/v11.6.0/bin:/home/kinow/bin:/opt/ant/bin::/home/kinow/bin:/home/kinow/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin:/home/kinow/.rvm/bin
    export VIRTUAL_ENV=/home/kinow/Development/python/workspace/cylc-uiserver/venv
    export LANG=en_NZ.UTF-8
    export JUPYTERHUB_API_TOKEN=84bfe7addc3448d78a90a384c5eee1ac
    export JPY_API_TOKEN=84bfe7addc3448d78a90a384c5eee1ac
    export JUPYTERHUB_CLIENT_ID=jupyterhub-user-kinow
    export JUPYTERHUB_HOST=172.17.0.1
    export JUPYTERHUB_OAUTH_CALLBACK_URL=/user/kinow/oauth_callback
    export JUPYTERHUB_USER=kinow
    export JUPYTERHUB_SERVER_NAME=
    export JUPYTERHUB_API_URL=
    export JUPYTERHUB_ACTIVITY_URL=http://172.17.0.1:8081/hub/api/users/kinow/activity
    export JUPYTERHUB_BASE_URL=/
    export JUPYTERHUB_SERVICE_PREFIX=/user/kinow/
    unset XDG_RUNTIME_DIR
    touch .jupyter.log
    chmod 600 .jupyter.log
    /home/kinow/anaconda3/bin/cylc-singleuser --port=8000 -s /home/kinow/Development/python/workspace/cylc-ui/dist/ < /dev/null >> .jupyter.log 2>&1 & pid=$!
    echo $pid

And what's even nicer, is that if you pay attention to how it's running the cylc-singleuser in the last line, it actually created a log in the remote host too. I found that the cylc-singleuser was not spawning because it couldn't be located in the $PATH. That was due to Anaconda not being included without loading the bashrc script. Fixed by defining the absolute path of the cylc-singleuser script.

Finally, once everything was fixed, it started to redirect to invalid addresses:

At the top is the browser, with a redirect loop. Below, the terminal in the remote SSH box, displaying the cylc-singleuser script that was spawned by the Hub.

kinow · 2019-05-28T00:23:03Z

Also had to change in the sshspawner and add the following to exec_notebook:

env['JUPYTERHUB_HOST'] = "172.17.0.1"
env['VIRTUAL_ENV'] = ""

As otherwise it would use 127.0.0.1 to try and connect to the hub

kinow · 2019-05-28T01:07:08Z

Reporting issues back (and thanking them for the spawner as well).

kinow · 2019-05-28T08:37:57Z

So this afternoon I tried the whole setup again from home in the morning, with the same issue. Then in the afternoon tried to set it up at work in the afternoon. Same issue.

Found an issue with a similar description (jupyterhub/jupyterhub#1657) but no clear fix for this one.

Then tried something else just now. Tried to set up a virtual machine, as I thought perhaps it was a problem with the Docker images.

Created an Alpine VM, added python3, nodejs-npm and other apk packages. Everything ready, then started the jupyterhub process, it spawned cylc-singleuser too, but same error.

One curious thing is that here at home I am - by accident - using a branch with the GraphQL work from @dwsutherland . Because of this, I had one endpoint without security to test, the /graphql/graphiql.

And this endpoint works fine.

This means that the endpoints protected by @tornado.web.authenticated annotation, and whose parent class is that JupyterHub OAuth handler, are having issues with redirect.

I am not seeing any useful logs or exceptions on the cylc-singleuser Tornado app, so I guess there is something wrong between configurable-http-proxy <-> Tornado/Cookie.

kinow · 2019-05-28T08:50:01Z

Left is the Cylc UI Server running from an Alpine VM, spawned via SSH. Browsing from my computer (host to the alpine VM). To the right, the top command in that VM.

So only thing pending: figure out why the jupyterhub.services.auth handlers are not handling authentication 😬

kinow · 2019-05-28T11:56:55Z

The routes in the configurable-http-proxy before the spawner:

{
    "/":{
        "hub":true,
        "target":"http://192.168.20.2:8081",
        "jupyterhub":true,
        "last_activity":"2019-05-28T09:27:48.270Z"
    }
}

And after spawned

{
    "/":{
        "hub":true,
        "target":"http://192.168.20.2:8081",
        "jupyterhub":true,
        "last_activity":"2019-05-28T09:29:21.457Z"
    },
    "/user/kinow":{
        "user":"kinow",
        "server_name":"",
        "target":"http://192.168.20.11:39411",
        "jupyterhub":true,
        "last_activity":"2019-05-28T09:29:21.857Z"
    }
}

Note:

To request the proxy table, you need to auth the requests. The easiest way is through:

$export CONFIGPROXY_AUTH_TOKEN=$(openssl rand -hex 32)
$curl -H "Authorization: token $CONFIGPROXY_AUTH_TOKEN" http://localhost:8001/api/routes

kinow · 2019-05-28T11:58:34Z

Added a remote Python debugger (is that possible with vim @hjoliver? I thought it wouldn't be possible at all in Python) with pip install pydevd-pycharm~=191.7141.48 and two lines of code in the cylc_singleuser.py in the remote host, and attached my IDE to the remote process. First time doing in Python, but works just like remote debugging a JVM.

Then loaded the old config, and recorded the normal application redirects, and then the bad ones, with the infinite loop.

Good:

POST http://localhost:8000/hub/login?next=
GET http://localhost:8000/user/kinow/
GET http://localhost:8000/hub/api/oauth2/authorize
GET http://localhost:8000/user/kinow/oauth_callback
GET http://localhost:8000/user/kinow/

Bad:

POST http://localhost:8000/hub/login?next=
GET http://localhost:8000/user/kinow/
GET http://localhost:8000/user/kinow/192.168.20.2/hub/api/oauth2/authorize

The last URL above was wrong. The default value is oauth_authorization_url=/hub/api/oauth2/authorize, but for some reason we were getting oauth_authorization_url=/user/kinow/192.168.20.2/hub/api/oauth2/authorize.

Took me a while, but found why. I had added env['JUPYTERHUB_HOST'] = "192.168.20.2", whereas it had to be env['JUPYTERHUB_HOST'] = "http://192.168.20.2:8000". Found after debugging a few times, and chasing where that value was coming from...

kinow · 2019-05-28T12:02:20Z

Progress after the last changes, and now got:

POST http://localhost:8000/hub/login?next=
GET http://localhost:8000/user/kinow/
GET http://localhost:8000/hub/api/oauth2/authorize
GET http://localhost:8000/user/kinow/oauth_callback

This last URL instead of returning the last 302 HTTP response, it is returning a 403, with the following traceback:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/tornado/web.py", line 1699, in _execute
    result = await result
  File "/usr/lib/python3.7/site-packages/tornado/gen.py", line 191, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/jupyterhub/services/auth.py", line 973, in get
    raise HTTPError(403, "oauth state does not match. Try logging in again.")
tornado.web.HTTPError: HTTP 403: Forbidden (oauth state does not match. Try logging in again.)

kinow · 2019-05-28T12:13:45Z

Alright. So this happens because I logged in to http://localhost:8000, which redirected to the "correct" URL http://192.168.20.2 (that's the URL of my notebook, which SSH node can see).

When I log in the second time it appears to have an incorrect cookie that fails to authenticate with the current state (state is a concept in the jupyterhub auth process).

Fixed by logging in to http://192.168.20.2 directly with an incognito window. But the battle is not over yet! Now got a response 500 (sighs), with the traceback:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3.7/site-packages/urllib3/util/connection.py", line 80, in create_connection
    raise err
  File "/usr/lib/python3.7/site-packages/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen
    chunked=chunked)
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 355, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.7/http/client.py", line 1229, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1275, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1224, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.7/http/client.py", line 1016, in _send_output
    self.send(msg)
  File "/usr/lib/python3.7/http/client.py", line 956, in send
    self.connect()
  File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 183, in connect
    conn = self._new_conn()
  File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fb104a6a7b8>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 641, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3.7/site-packages/urllib3/util/retry.py", line 399, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=8081): Max retries exceeded with url: /hub/api/oauth2/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb104a6a7b8>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/jupyterhub/services/auth.py", line 332, in _api_request
    r = requests.request(method, url, **kwargs)
  File "/usr/lib/python3.7/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=8081): Max retries exceeded with url: /hub/api/oauth2/token (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fb104a6a7b8>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/site-packages/tornado/web.py", line 1699, in _execute
    result = await result
  File "/usr/lib/python3.7/site-packages/tornado/gen.py", line 191, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python3.7/site-packages/jupyterhub/services/auth.py", line 976, in get
    token = self.hub_auth.token_for_code(code)
  File "/usr/lib/python3.7/site-packages/jupyterhub/services/auth.py", line 625, in token_for_code
    headers={'Content-Type': 'application/x-www-form-urlencoded'},
  File "/usr/lib/python3.7/site-packages/jupyterhub/services/auth.py", line 345, in _api_request
    raise HTTPError(500, msg)
tornado.web.HTTPError: HTTP 500: Internal Server Error (Failed to connect to Hub API at 'http://127.0.0.1:8081/hub/api'.  Is the Hub accessible at this URL (from host: alpine1)?  Make sure to set c.JupyterHub.hub_ip to an IP accessible to single-user servers if the servers are not on the same host as the Hub.)

kinow · 2019-05-28T12:25:09Z

Phew. This one was easy. Had to add one more entry to jupyterhub_config.py. Now my configuration file is:

c.ConfigurableHTTPProxy.api_url = 'http://192.168.20.2:8100'
c.ConfigurableHTTPProxy.debug = True

c.JupyterHub.spawner_class = 'sshspawner.sshspawner.SSHSpawner'
c.JupyterHub.hub_ip = "192.168.20.2"
c.JupyterHub.hub_connect_ip = '192.168.20.2'
c.JupyterHub.proxy_api_ip = '192.168.20.2'


c.HubOAuth.hub_host = "http://192.168.20.2:8000"
c.HubOAuth.oauth_authorization_url = "/hub/api/oauth2/authorize"

c.SSHSpawner.remote_hosts = ['192.168.20.11']
c.SSHSpawner.remote_port = '22'
c.SSHSpawner.ssh_command = 'ssh'
c.SSHSpawner.ssh_keyfile = '/home/kinow/.ssh/id_rsa'
c.SSHSpawner.hub_api_url = "http://192.168.20.2:8081/hub/api"
# local path
c.SSHSpawner.path = '/home/kinow/Development/python/workspace/sshspawner:/home/kinow/Development/python/workspace/cylc-uiserver/venv/bin:/home/kinow/.nvm/versions/node/v11.6.0/bin:/home/kinow/bin:/opt/ant/bin::/home/kinow/bin:/home/kinow/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin:/home/kinow/.rvm/bin'
c.SSHSpawner.remote_port_command = '/usr/bin/python3 /home/kinow/sshspawner/scripts/get_port.py --ip'

# useful for debugging with the IDE
c.Spawner.http_timeout = 9999999
c.Spawner.args = ['-s', '/home/kinow/cylc-ui/dist/']
c.Spawner.cmd = ['/usr/bin/cylc-singleuser']

# logging
c.Application.log_level = "DEBUG"
c.JupyterHub.log_level = 10

My sshspawner/sshspawner.py is also patched with:

    # FIXME add docstring
    async def exec_notebook(self, command):
        """TBD"""

        env = super(SSHSpawner, self).get_env()
        env['JUPYTERHUB_API_URL'] = self.hub_api_url
        env['PATH'] = self.path
        env['JUPYTERHUB_HOST'] = "http://192.168.20.2:8000"
        env['JUPYTERHUB_SERVICE_URL'] = "192.168.20.2"

(Sorry, not in a git folder, so no diff, but might be useful later)

Aaaaand. Success @hjoliver & @MartinRyan . Started this final tests after our meeting. Took me only a few hours and miss the gym today. But it became personal 😠

Here's a screen shot of our UI server running on 192.168.20.11, an Alpine VM with SSH configured. It was spawned with the NERSC/sshspawner, and is accessible from my notebook (host computer) as it is possible to confirm in the browser url with my IP 192.168.20.2.

And this screen and the main screen are available only when authenticated. And I could confirm in the browser the HTTP 302 for Oauth authentication process happening.

I also disabled the AddSlashHandler as I believe that was part of the problem with infinite loop (but we have a replacement for that in the works that hopefully Just Works (TM) 🤞

hjoliver · 2019-05-30T07:15:24Z

Started this final tests after our meeting. Took me only a few hours and miss the gym today. But it became personal 😠

hahaha, I know how you feel! (but sorry you missed the gym)

kinow · 2019-05-30T07:16:59Z

All good. I wouldn't be able to focus on the work out anyway 😁

hjoliver · 2019-05-30T07:18:21Z

This kind of issue would demand serious weights and a lot of angry straining 🏋️‍♂️

kinow · 2019-07-16T21:00:31Z

I believe this testing is done, and we could successfully spawn via SSH. Closing 🎉

hjoliver · 2019-07-16T23:32:42Z

(In due course we'll need to document how to do this though ... just need to remember this Issue exists I guess.)

kinow · 2019-07-17T00:06:50Z

(In due course we'll need to document how to do this though ... just need to remember this Issue exists I guess.)

👍 good point, and as I don't trust my memory, and could end up re-doing everything from scratch... cylc/cylc-doc#36

kinow self-assigned this May 27, 2019

kinow added the question Flag this as a question for the next Cylc project meeting. label May 27, 2019

kinow mentioned this issue May 28, 2019

Is the certificate required to use asyncssh? ronf/asyncssh#214

Closed

kinow mentioned this issue May 28, 2019

Accept extra parameters available for the singleuser app without crashing #48

Closed

kinow mentioned this issue Jun 5, 2019

List other UI servers running #27

Closed

kinow closed this as completed Jul 16, 2019

kinow mentioned this issue Jul 17, 2019

Cylc UI Server: Document Hub spawners cylc/cylc-doc#36

Closed

3 tasks

kinow added this to the 0.1 milestone Sep 10, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test Cylc UI Server with SSH Spawner #46

Test Cylc UI Server with SSH Spawner #46

kinow commented May 27, 2019

kinow commented May 27, 2019

kinow commented May 27, 2019

kinow commented May 27, 2019

kinow commented May 28, 2019 •

edited

Loading

kinow commented May 28, 2019 •

edited

Loading

kinow commented May 28, 2019

kinow commented May 28, 2019

kinow commented May 28, 2019 •

edited

Loading

kinow commented May 28, 2019

kinow commented May 28, 2019

kinow commented May 28, 2019

kinow commented May 28, 2019 •

edited

Loading

hjoliver commented May 30, 2019

kinow commented May 30, 2019

hjoliver commented May 30, 2019

kinow commented Jul 16, 2019

hjoliver commented Jul 16, 2019

kinow commented Jul 17, 2019

Test Cylc UI Server with SSH Spawner #46

Test Cylc UI Server with SSH Spawner #46

Comments

kinow commented May 27, 2019

kinow commented May 27, 2019

kinow commented May 27, 2019

kinow commented May 27, 2019

kinow commented May 28, 2019 • edited Loading

kinow commented May 28, 2019 • edited Loading

kinow commented May 28, 2019

kinow commented May 28, 2019

kinow commented May 28, 2019 • edited Loading

kinow commented May 28, 2019

kinow commented May 28, 2019

kinow commented May 28, 2019

kinow commented May 28, 2019 • edited Loading

hjoliver commented May 30, 2019

kinow commented May 30, 2019

hjoliver commented May 30, 2019

kinow commented Jul 16, 2019

hjoliver commented Jul 16, 2019

kinow commented Jul 17, 2019

kinow commented May 28, 2019 •

edited

Loading

kinow commented May 28, 2019 •

edited

Loading

kinow commented May 28, 2019 •

edited

Loading

kinow commented May 28, 2019 •

edited

Loading