zmq: implement SSH tunnelling

[oliver-sanders]

Supersedes #2975
See also #2978

Implement TCP over SSH communication method.

This could be quite straight forward as ZMQ has built-in functionality.

(To make searching for this easier, it is about re-implementing the cylc 7 "ssh task communication method")

[datamel]

I have looked into this. Although ZMQ does have ssh tunnelling built in, when I tried to implement, it became apparent that we would not be able to use this.
A grep of the sshd_config on one of our vlds returned: AllowTcpForwarding no
Since TCP forwarding is prohibited, this method will not work.
I will have a look at reimplementing the old method, unless there is a preferred method to try?!

[hjoliver]

@datamel - what's a "vld"? By "we" I assume you mean "Met Office"? I'm not sure that's a good reason not to do this.

Firstly, you don't actually need "cylc ssh comms" at Met Office, do you? (It's only a back-up method for those who can't use our preferred network comms)

Secondly, if you do need it, the first resort should be to persuade your system admins to allow TCP forwarding because a critical bit of their infrastructure needs, or otherwise give us bullet-proof
defensible reasons for not doing so (which implies extra work for us and additional complexity in Cylc).

Ping @dpmatthews

[oliver-sanders]

Quick note, TCP via SSH isn't (as far as I'm aware) something we would be interested in at the Met Office.

This feature is of most use to people running on infrastructure they have little control over. The worry we have is that the requirement to enable tcp forwarding might make this feature off-limits in such cases.

https://security.stackexchange.com/questions/22782/security-concerns-with-tcp-forwarding

The old Cylc7 ssh somewhere then open a Cylc client approach only used a few lines of code and won't be hard to resurrect.

[datamel]

Apologies @hjoliver I think my post was unclear.

…I was not aware that vld was not a widespread use term, the vlds are Met Office virtual linux desktops (I’m still learning what is standard and what is met office jargon!)

By “we” I meant the Cylc team, I was aware that this was for alternative organisations that would not be able to use the standard ZMQ. I should have been more explicit – my thinking was that if (at the Met Office) we have TCP forwarding switched off, other organisations may also have it disabled. It would be unfortunate to push out a feature for users to get around not being allowed to use ZMQ, which is still not viable.
There is clearly a significant level of risk around allowing SSH tunnelling
https://www.ssh.com/ssh/tunneling/#ssh-tunneling-in-the-corporate-risk-portfolio
Anyone using Cylc may not be able to get this switched on – depends on how much risk their IT teams are willing to tolerate.
If keen to press ahead with the tunnelling, there are a number of options in the sshd config which could prohibit this:
AllowTcpForwarding
AllowStreamLocalForwarding
GatewayPorts
PermitTunnel

Would it be worth checking with some of the users who need this feature whether they could have these configured?

As @oliver-sanders has said, the Cylc7 option is still viable and easy to implement - I'm happy to go ahead with this if preferred.

Sorry again for the confusion!

[hjoliver]

@datamel sorry I forgot to come back to this. No need to apologize!

If we have to go back to the Cylc 7 way, so be it. I think the only person we know of at the moment who has expressed a strong need for this capability (one way or another) is @trwhitcomb - what do you think Tim?

[oliver-sanders]

Think we are going to resurrect the Cylc7 approach that constructs an SSH command to call out to cylc client.

[oliver-sanders]

In order to SSH back to the scheduler the job host requires three configuration values from the scheduler host:

• ssh command
• cylc path - formally cylc executable see conf: cylc executable => cylc path #4105
• use login shell

These values are available to the Scheduler from the localhost platform. We need to pass them through to the job script. Suggest adding three new fields to the contact file:

• SCHEDULER_SSH_COMMAND
• SCHEDULER_CYLC_PATH
• SCHEDULER_USE_LOGIN_SHELL

It is ok to use the contact file for this information as it is specific to the scheduler and not the install target or platform. Whilst only required on the job platforms it would be easier to add this information to the Scheduler's contact file and allow the remote-init/file-install logic to do the rest.