Configuration design: Separate Access Management?

[malex984]

Note: this is a separation of super-issue #13

Station: 4.5 Why do we need key_ref? If you configured passwordless ssh then you only need the username to connect. The correct key is picked by ssh.

Station: 4.6 Also, once again. I really don't understand why you'd ever publicize a private key and if I can't understand it and you can't explain it to me then you'll have a harder time explaining it to a customer or interested party. We shouldn't "invent" our own security mechanisms but instead use things already in place.

[porst17]

The ssh key has to be stored on the machine the management cli is running on. The cli user needs read access to to the private key anyway. Why is it more dangerous to put the ssh password or private ssh key (reference) into the config than just keeping it in ~/.ssh? If am not mistaken, the ui and cli processes need access to the key (even though it might be indirectly by calling the ssh command with same permissions that in turn reads the key).

[malex984]

Alex said:

4.5 Why do we need key_ref? If you configured passwordless ssh then
you only need the username to connect. The correct key is picked by ssh.

I assume that you refer to using ~/.ssh/config and ~/.ssh/id_rsa, right?
We will use them if they are available BUT they may be absent!
Therefore this is an OPTIONAL source for configuration of password-less ssh
to remote hosts that cannot be accessed via default ssh configuration on
the server.

• 4.6 Also, once again. I really don't understand why you'd ever
  publicize a private key and if I can't understand it and you can't explain
  it to me then you'll have a harder time explaining it to a customer or
  interested party. We shouldn't "invent" our own security mechanisms but
  instead use things already in place.

Sure no inventions here - we directly use SSH keys!

Just consider:

• how exactly completely empty server system gets those keys?
• who will configure password-less ssh access to ALL ~150 stations for us
  in advance?

Did you read the comments regarding ssh keys in the document Installation exhibition computers ESO/HITS/Imaginary?

[malex984]

Eric said:

I assume that you refer to using ~/.ssh/config and ~/.ssh/id_rsa, right?
We will use them if they are available BUT they may be absent!
Therefore this is an OPTIONAL source for configuration of password-less ssh to remote hosts that cannot be accessed via default ssh configuration on the server.

Explain the mechanism. The server has a private key and a public key, and there are no reasons why it wouldn't have them. The stations need the public key in its authorized hosts list, and this is transferred just once in a secure manner because it needs to be done using the station's password... so it makes no sense to have it in the cfg.

Is there some other mechanism? Because anyway nothing but a public key should leave a system... the private key shouldn't be transmitted or known by external systems.

Sure no inventions here - we directly use SSH keys!

Just consider:

• how exactly completely empty server system gets those keys?
• who will configure password-less ssh access to ALL ~150 stations for us in advance?

The same way they'll install an OS, Docker, setup networking, a user, backups, OMD, etc. on 150 stations... This is well beyond the scope of our tool! There are countless tools for parallel administration / deployment. Setting up passwordless ssh access is a security-sensitive operation that is done just once, so it has no reason to be in the config.

... But, how to set up passwordless ssh to 150 stations?

With a list of 150 hostnames + their passwords + a script that transfers the server's public key to the authorized_keys file in the home of the user that will run docker using ssh with the password. Then you delete the list of hostnames + passwords.

Though, once again, we don't need to take care of this with hilbert.

Did you even read the comments regarding ssh keys in the document "Installation exhibition computers ESO/HITS/Imaginary"?

I don't have access to that document I think (just searched)... but if the idea completely vulnerates the security of stations and ssh credentials then it doesn't matter.

If I'm failing to understand something and this is indeed reasonable and perfectly secure then consider a prospective customer might have the same concerns I have and go away, so we need to be clear and avoid misunderstandings.

[elondaits]

The ssh key has to be stored on the machine the management cli is running on. The cli user needs read access to to the private key anyway. Why is it more dangerous to put the ssh password or private ssh key (reference) into the config than just keeping it in ~/.ssh? If am not mistaken, the ui and cli processes need access to the key (even though it might be indirectly by calling the ssh command with same permissions that in turn reads the key).

Normal scheme: To connect to the station from the server you have both a private key and a public key in the server and have an authorized_keys file in the station with a copy of your public key.

Now, this is the relevant piece of cfg:

  supernova:
    rexec: !!Unix {}
    address: “supernova.mfo.de”
    ssh: { user: "ur", key_ref: "supernova.id_rsa" }

Problems with this:

• It's a key associated with the station. You shouldn't have that in the server, nor need it. You only need the server's key, which is the same for all stations.
• It's apparently (based on the name), the private key, which should never leave the machine so there's less reason to specify it because it's used internally by ssh.

Alex said in previous posts this was a scheme to install the ssh keys in the stations, which I don't think it's something our system should do, because it's not specific to it and its just one of many deployment tasks.

[porst17]

@elondaits, you didn't answer my question, I think. You said above that we should never publicize a private key. But if the private key or a reference to it is in the config, it is not more public than a private key that is stored in .ssh. The private key must also not be password protected. Otherwise, we can't automatically connect to the stations. Or the key passphrase for the private key has to be put into the config file. An encrypted private key together with its password is as insecure a unencrypted private key. And the hilbert system needs access to the config file as well as to .ssh (because hilbert calls ssh and therefore is the same user).

But I would agree, that we could just leave out the key or key_ref part. The stations need the public key be set in their authorized_keys file. And the management server will just have the private key in its .ssh folder. It has to be put there by the admin or by the external syncing script.

@malex984 Why would you need to put the private key into the config in order to set up password-less ssh? If password-less ssh is not configured yet, you need a password to log in. Then you place the public key into the authorized_keys files.

Conclusion: Our system relies on properly set up password-less ssh access. Setting this up is a sysadmins job.

[elondaits]

@porst17 I didn't answer it because I think we're thinking of different things... you are talking about the server's private key (which is in the same machine as this cfg), but I assume that the private key that is associated to a station in the cfg is in fact the private key of the station (because the private key of the server is the same for all the stations and because Alex said it was for deploying the ssh keys to multiple stations easily).

A reference (path) to the server's private key doesn't expose anything, of course... my issue was with putting references to the stations' private key files, which suggests they're accesible by the server.

As for putting the server's private key in the file, yes, that could be a security problem... because the private key should be non writable and only readable by the user that connects through ssh with the station... while the cfg file could be readable/writeable by a different user or group (specially if it's synced through djangoplicity).

[malex984]

It seems that for this project WE are going to generate a single ssh key pair and during the general SW installation phase our public ssh key will be added to authorized_keys on each station by others (automatically via puppet).

Let us consider the management back-end in more details:

• management backend will be running in a docker container, but its image should not contain any private ssh keys!
• it may be necessary to access the server host system via ssh from within the management container since server is to be treated just like a station (e.g. for updating its configs or turning it off). It may be easily done with a temporary ssh key pair (no need to specify this in the general config file).
• there must be no manual intervention by us or admins to the server host system since in case of server's failure - the 2nd server host must to get all the latest data & configurations from CMS automatically all by itself assuming only initial Hilbert's Server Configuration Package (.RPM) was installed.
• another ssh key pair may be required to access CMS (to get the general configuration file via scp / sftp).

The above means that private ssh keys may be installed as a part of the initial Hilbert's Server Configuration Package (which will also include all docker images and Hilbert-Station CLI with initial config for the server as a station).

AFAIK Management container may easily access private keys that where previously installed on the server in many ways.

But what about <SERVER_HOST>/<HILBERT_USER>/.ssh/config?
Should it be installed just like the private key - via some extra Server Access Configuration Package (.RPM) during during the general SW installation phase?
Then it has to know ALL possible stations and CMS + their ssh connection details right from the very beginning! Moreover in order to change or update .ssh/config IT stuff will have to recreate that that .RPM and re-install it on both servers :-(

On the other hand i propose to specify which of those pre-installed private ssh keys (if there are several) is required to access which station (with all ssh connection details) in the general configuration file. No pre-installed .ssh/config is required since it may be generated on-the-fly from the general configuration file if necessary.

In this case the initial Hilbert's Server Configuration Package will only pre-configure access to CMS for downloading the general configuration file (which is required in any case).

ps: in the above i assumed that our servers will be running exactly the same low-level Hilbert-CLI-Station scripts as any other Linux station but will be configured with a very special profile (server) to run some special services that require quite a few local settings.

NOTE in what follows i tried to describe my understanding of the general responsibilities:

• We prepare:
  1. public/private ssh key pairs
  2. SW packages (.RPMs) with
     • Hilbert-Station-Scripts for a generic linux station
     • Initial configuration for generic stations (maybe incl. a few service images)
     • Testing configuration for generic standalone interactive stations (incl. images for the standalone testing application)
     • Initial configuration for the server (including ALL docker images with service & applications)
• IT Admins will (automatically via their puppet):
  1. create special users for Hilbert and add our public ssh key into authorized_keys on all interactive stations (not sure about the server)
  2. install necessary SW packages on interactive stations and servers

NOTE: maybe the 1st item above may also be performed by one of our SW packages (i.e. be part of 2nd step).

• There must be no manual intervention by us or admins into any of stations and the server (either prior to that initial installation or afterwards).
• Upon Power-On ANY station with Hilbert has to automatically start low-level Hilbert-CLI-Station's script and proceed according to its local configuration.

NOTE: servers are stations with special configurations - they start: Registry, OMD Server, Management Dashboard etc.

Configuration Update procedure:

If triggered Management Dashboard should be able to:

• pull ALL the necessary docker images & general config file from CMS (or update them)
• create/update the server's registry with the new docker images
• (re-)configure OMD service to monitor ALL stations (according to the updated general config file)
• for each station:
  • (re-)generate station's local Hilbert-Configuration (in simple bash)
  • Power It On (if necessary according to the configuration)
  • deploy newly generated Station's configurations (during station's autostart pause)

[porst17]

@elondaits I think, I have a better understanding of your remarks now.

@malex984 This is again a lot of text. Do you have any specific questions?

If you don't want to expose private keys to the management backend container, we could ask for ssh-agent running on the server and then forward the agent into the container. (Does not work on docker+mac right now, though. See docker/for-mac#483 and docker/for-mac#410.)

If you really need ~/.ssh/config, you would need to copy/mount it from the host into the container.

Managing and deploying keys should not be our concern. This is the IT admins job. To us, it doesn't matter if the keys are set up in advance by hand, using puppet or self-tailored shell scripts. Also, replication to a second backup server is not our task.

The management of the ssh keys and ssh config can be part of the syncing. This should solve most of the issues you stated above. The sync script will be specific for each installation of hilbert. I don't indent to provide a generic solution for the syncing.

Hilbert has some requirements. Hilbert does not care, if the IT admin is fulfilling these requirements by hand, via puppet or via a hand-written bash-script that syncs via some external CMS.

Hilbert takes the host name/ip (address) specified in the config and connects via ssh. If the host, the management container is running on, is able to connect and log in to address, then hilbert must be able to do so as well. That's all.

[malex984]

@malex984 This is again a lot of text. Do you have any specific questions?

Sorry, i will try to be as brief as possible.

I think we are safe to assume that ssh key pairs for stations and access to CMS are fixed and installed separately ahead of time and they will probably never change later on, correct?

If you really need ~/.ssh/config,

Either

1. we rely on IT Admins to maintain ~/.ssh/config on all servers for us or
2. corresponding connection details go into our single general config file

NOTE:

• 1. means another configuration file for IT Admins to maintain
• 2. means that Hilbert-Server can automatically create .ssh/config if necessary

The management of the ssh keys and ssh config can be part of the syncing.

Do you mean Hilbert-Server getting them from CMS? This is how it is done in the current prototype. I thought that was not appropriate for you guys?

Hilbert does not care, if the IT admin is fulfilling these requirements by hand, via puppet or via a hand-written bash-script that syncs via some external CMS.

I considered the case of primary server crash: they power on a backup server - upon power-on it has to automatically download something from CMS and become as good as the failed one. It seems possible to me only if backup server has pre-configured access to up-to-date backup of all data & SW packages & configurations. Therefore either:

1. Admins regularly backup primary server or
2. All that data/SW/configs originates externally and can be used to automatically restore a server.

[elondaits]

I think we are safe to assume that ssh key pairs for stations and access to CMS are fixed and installed separately ahead of time and they will probably never change later on, correct?

When you say "ssh key pairs" it sounds like they're a lot. But it's just the private/public pair in the server. The public has to be copied to the authorized_keys of every station... just that. If the server accesses the CMS then it's the same private, with the public copied to the CMS. If the CMS accesses the server then it will need to install its own public key in the server... but that's something outside the scope of hilbert.

[malex984]

If the server accesses the CMS then it's the same private, with the public copied to the CMS.

AFAIR, CMS access was never discussed in this regard.

When you say "ssh key pairs" it sounds like they're a lot.

Yes, for the current project we may do with a single pair for accessing:

1. CMS and
2. all stations (including the server host system)

from within the container with the Hilbert management backend.

Note: that ssh pair will be installed on the server host system! I can see the following way to use it by management container:

1. mounting the place with ssh keys as a data volume

• the place with ssh keys has to be a part of server configuration (as a station). Which is fine as they will likely be installed simultaneously by the same .RPM.

2. using plain text environment variables

• as above, also
• may be cumbersome if it becomes necessary to handle several keys in general case

3. setting up ssh agent socket on the host and mounting it into the container

• requires ssh agent running on the host server system
• may not work if we use a different non-root user within the management container

Currently i tend to read-only mount ${HOME}/ into the management container (e.g. as /HOME).

[porst17]

@malex984 This is again a lot of text. Do you have any specific questions?

Sorry, i will try to be as brief as possible.

I think we are safe to assume that ssh key pairs for stations and access to CMS are fixed and installed separately ahead of time and they will probably never change later on, correct?

What do you mean by never? If a key pair has been compromised, I am sure, IT admins will want to replace it. We shouldn't rule out the possibility of changing the login credentials. But what is your actual question and is it relevant to the discussion?

If you really need ~/.ssh/config,

Either

1. we rely on IT Admins to maintain ~/.ssh/config on all servers for us or
2. corresponding connection details go into our single general config file

NOTE:

• 1. means another configuration file for IT Admins to maintain
• 2. means that Hilbert-Server can automatically create .ssh/config if necessary

If you want to put the info relevant for generating the ~/.ssh/config into the hilbert-cli config, then you would also need to place the ssh public and private keys there (to access the stations). If you don't, then we end up having the login and connection data in very different places which is in my opinion bad design and counterintuitive. But then we would be at the beginning of our discussion again.

Your argument that maintaining ~/.ssh/config is additional effort for IT admins is not really valid. IT admins have to take care of setting up the login via ssh in any case, e.g. they have to put the public key of the management server into the ~/.ssh/authorized_keys file of each station. But if they need to touch the ssh configuration of any host anyway, why not let them set up ~/.ssh/config at the same time?

The management of the ssh keys and ssh config can be part of the syncing.

Do you mean Hilbert-Server getting them from CMS? This is how it is done in the current prototype. I thought that was not appropriate for you guys?

I seems like you still have a misunderstanding here. The main point was that hilbert does not care how or who set up ssh access to the stations. All that matters to hilbert is that it can access the keys and maybe the ssh config on the local machine (via mounting, ssh-agent or whatever).
I do not want to pollute this discussion with details on the syncing. Therefore, I added my explanations to a new issue #29.

Hilbert does not care, if the IT admin is fulfilling these requirements by hand, via puppet or via a hand-written bash-script that syncs via some external CMS.

I considered the case of primary server crash: they power on a backup server - upon power-on it has to automatically download something from CMS and become as good as the failed one. It seems possible to me only if backup server has pre-configured access to up-to-date backup of all data & SW packages & configurations. Therefore either:

1. Admins regularly backup primary server or
2. All that data/SW/configs originates externally and can be used to automatically restore a server.

I have no problem with hosting the data externally. All that I am saying is that hilbert relies on a working local copy of that data. And hilbert is not responsible for putting the data there nor for keeping it up to date.

[porst17]

Note: that ssh pair will be installed on the server host system! I can see the following way to use it by management container:

1. mounting the place with ssh keys as a data volume

• the place with ssh keys has to be a part of server configuration (as a station). Which is fine as they will likely be installed simultaneously by the same .RPM.

2. using plain text environment variables

• as above, also
• may be cumbersome if it becomes necessary to handle several keys in general case

3. setting up ssh agent socket on the host and mounting it into the container

• requires ssh agent running on the host server system
• may not work if we use a different non-root user within the management container

Currently i tend to read-only mount ${HOME}/ into the management container (e.g. as /HOME).

Options 1. and 3. have the same permission problem. The files in ~/.ssh have restricted permissions just like the socket used for ssh-agent. The advantage of the ssh-agent is that you don't expose the private keys to the container AFAIK (which is not really an advantage if the private key is not encrypted on the management server and you are able to log into it via ssh from within the container ...).

I am ok with option 1.

[elondaits]

Question (out of ignorance): Why do we need to ssh INTO the docker container? Stopping and starting the station and changing apps is done through the host system. What actions does hilbert have to perform "inside" the container that requires to ssh into it? Can we do those things in some other fashion and just avoid the issue altogether?

[malex984]

Why do we need to ssh INTO the docker container?

@elondaits You are wrong! We only need ssh OUT of docker container to the server host itself since it is treated by the management backend as just another station. Therefore it may need to do the following on the server:

• update its configuration as a Hilbert station (used by Hilbert-Cli-Station that initially starts all the services on the Server) or
• start/stop/restart some service/application running on the Server or
• power-off the server...

[elondaits]

I'm not sure I understand your explanation, perhaps there's a word or two missing, but what I understand is the following:

• We only need to ssh out of the docker container that runs hilbert on the hilbert server host.
• The hilbert container is tracked as a station by hilbert and updated through hilbert.
• The services required in the hilbert server host are started / stopped through hilbert.
• The hilbert server host can be powered off through hilbert.

Is this what you're saying?

If so:

• I still don't see why we should concern with the ssh keys for the stations if only the server has this problem.
• I really can't imagine how the hilbert server can be tracked through hilbert or why you would want to shut it off through hillbert if you can't start it the same way.

[malex984]

@porst17 Ok, i understand. Let me summarize:

1. Hilbert-Cli-Station will be installed on all stations (including the server) by others
2. Hilbert-Cli-Station automatically starts upon station's boot-up, under some special user that has enough permissions. Let ${HOME} denote the home directory of such a special user on the server host.
3. we rely on ${HOME}/.ssh to contain everything necessary for password-less ssh access to all stations (including the server itself) under the name of those special users from 2. above. It will be maintained by others.
4. on the server: Hilbert-Cli-Station is pre-configured to start services according certain server profile, which includes the management service.
5. docker container with the management service mounts (as read-only) ${HOME}/.ssh into it.

Consequences for the General Configuration (provided via CMS) are as follows:

1. RemoteExecSSH should refer to pre-configured Host names listed in ${HOME}/.ssh/config and needs no additional ssh-related details
2. Stations may require for some static address field since OMD server needs to contact monitored stations via network (e.g. if no ssh access was configured in ${HOME}/.ssh/config for them)
3. WOL requires MAC address (and maybe the static address field from 2.)

ps: I would make the address field from 2. mandatory for all station definitions

1. for consistency and
2. since getting correct DNS name (or static IP) from a host configuration inside ${HOME}/.ssh/config may be quite tricky IMO.

[malex984]

@elondaits I hope my comment above explains the architecture. Here are some more details:

• We only need to ssh out of the docker container that runs hilbert on the hilbert server host.

Docker container for Hilbert Management Backend runs the Dashboard Back-end that uses Hilber-CLI-Server (server part), which can

• power-on station
• access station via ssh to execute some command via pre-installed Hilber-CLI-Station
• re-configure Hilber-CLI-Station on remote station (also for its standalone operation)

NOTE: there is no need to install Hilber-CLI-Server** anywhere outside of Hilbert Management image.

• The hilbert container is tracked as a station by hilbert and updated through hilbert.

No. Server Host itself is a station with pre-installed Hilbert-CLI-Station (client part), whose local configuration (e.g. in ${HOME}/.config/hilbert-cli-station) may need to be updated upon changes to the Global Configuration.

• The services required in the hilbert server host are started / stopped through hilbert.

yes, Hilbert-CLI-Server (inside management container) commands Hilbert-CLI-Station (on the server host) via ssh.

• The hilbert host server can be powered off through hilbert.

yes, that is one of commands Hilbert-CLI-Server asks Hilbert-CLI-Station to execute.

[porst17]

1. The user on the stations might be a different one than on the management server if that's configured in ~/.ssh/config. I assume that's not a big problem since you can detect your own user name on the station once logged in via id -u -n.
2. How is the address of a station determined if address would not be given? The station ID? The station name? If there are no other means to set this, we obviously need the address field.
3. I would require that address can be used for everything network-related in the local network. It must be a name that can generally be used by RemoteExecSSH, OMD and all other network related services. Hence, you can not use an ssh alias in address. If someone really wants to set up additional network aliases, he can do via /etc/hosts on the management server and we could think of using docker run --net=host ... in order to have correct resolution of /etc/hosts aliases if there is no DNS server in the local network.

[malex984]

@porst17 I totally support your proposal above. Just a few comments:

1. Yes, no need to care about any usernames according to items 2 and 3 at the beginning of Configuration design: Separate Access Management? #25 (comment)
2. Yes, it is much better to always have address for all stations (even though there are several faulty ways to determine host's DNS name or static IP like parsing ${HOME}/.ssh/config and whatnot).
3. Yes i think it is not too much to require that ssh aliases in ${HOME}/.ssh/config are exactly equal to the corresponding address (AFAIK one can use full DNS names and IPs to configure ssh aliases).

Decision recap:

• each station must have address for static host address (DNS or Static IP).
• address is to be used as ssh alias within ${HOME}/.ssh/config if ssh access to corresponding host is configured).

For reference please check out sample .ssh/config

Let us vote on that

ps: I do not see why additional network aliases might be necessary... AFAIK we will be able to cope with them if required. I suggest we do not discuss them now.

[elondaits]

I think the things being discussed and decided on now have not much to do with the original issue, and the discussion has opened to such a wide span of things it's really impossible for me to understand what I'm voting on or what I'm quietly accepting "by default". It seems it all has to do with ssh and credentials, but we're talking about the ssh between the server and the stations, between the stations and the hosts, credentials and security, what goes or doesn't go in the configuration, etc.

Can we have a simple diagram that shows the server and three sample stations and that indicates what containers / components exist on each of those and what forms of communication are established between each? (e.g. the ssh connections as an arrow that indicates who establishes them, etc)

On the "decision recap" above:

• Yes, each station must have an address, of course.
• I wouldn't rely on .ssh/config ... all the devOps software I've seen or used configured host, port (optional) and user (optional) in their "stations" and it seems reasonable as it allows configuration in a single file, doesn't ruin security, and allows using different user/port settings from the ones in .ssh/config (the defaults for console use)... and it'd allow setting up stations sharing IP/address (proxy schemes where each host uses a different port or user) without it being our responsibility (if it works, it works, and we don't care why/how).

But on a different note:

• In don't understand what gain we get by using hilbert to track and update the hilbert server. It seems to add extra complications, it's not the purpose of hilbert (which is to deploy, monitor and operate exhibition stations), and I can't even imagine how it would work. The fact itself that you can stop the hilbert server through hilbert but not power it back on seems more like a bug than a feature.

[malex984]

Can we have a simple diagram that shows the server and three sample stations and that indicates what containers / components exist on each of those and what forms of communication are established between each? (e.g. the ssh connections as an arrow that indicates who establishes them, etc)

[malex984]

@elondaits @porst17 Here is extended and updated system architecture diagram: 995afb3

Partial description were already given in

• brief description of Hilbert's station architecture
• brief description of the operation of Management Backends
• Reloading of General Configuration from CMS by Hilbert-CLI-Server

Which parts require more explanation?

Update: the following comment contains an outline of general station start-up sequence

[malex984]

I think the things being discussed and decided on now have not much to do with the original issue,

Yes, unfortunately management backend runs isolated inside a docker container - it cannot directly access the host system (execution only via ssh, data has to be previously mounted). See the current architecture diagram with the Server and a Station (since stations differ only by Services and Application running on them).

• I don't understand what gain we get by using hilbert to track and update the hilbert server.

All systems use exactly the same simple Hilbert-CLI-Station tool that starts Services and Applications according to host's local configurations.

Hilbert-CLI-Station tool and its configuration is all we need to pre-install on all systems with Docker Engine.

Conceptually Server is just another Station. Any Station may be turned into a Server by changing its configuration accordingly.

It seems to add extra complications, it's not the purpose of hilbert (which is to deploy, monitor and operate exhibition stations), and I can't even imagine how it would work. The fact itself that you can stop the hilbert server through hilbert but not power it back on seems more like a bug than a feature.

I do not understand: at closing time Dashboard will stop all stations before shutting down the server. After that one would not be able to remotely power-on any stations (at least via Dashboard since it become unavailable) before starting the server again...

The sequence is:

• Server is powered on either automatically on RTC or manually,
• it boots and uses Hilbert-CLI-Station to start OMD, Dashboard, Registry etc. according to local config.
• now Dashboard can power-on the rest of stations according to the general configuration (which have beed stored locally from previous sync. time or initial installation).
• Dashboard can be triggered to re-load the general configuration from CMS on user's demand
• Dashboard can update local configuration of each accessible station upon corresponding changes to the general configuration.

Update: this PowerOn/StartUp sequence is has been added to the architecture diagram with c380a48

NOTE: the above is a part of the general system architecture design (hilbert/hilbert-docker-images#28)

[malex984]

Now back to access management:

According to proposals by @porst17 and @elondaits we may choose to assume that ~/.ssh/ (which is located on the server host system) always provides up-to-date external resources for the global YAML configuration (located at CMS). Those external resources include:

1. ssh credentials (the initial ssh key pair) if they may be referenced by that YAML config
2. and maybe config (which defines ssh access to all stations mentioned in YAML config), if we want to keep the whole ssh access database in one place

what goes or doesn't go in the configuration, etc.

Note:

• for 2. we require ssh aliases for hosts .ssh/config to be in the YAML configuration.
• for 1. we will have to add all ssh details like port, user, host address and reference to external ssh credentials directly into the YAML configuration

[malex984]

• Yes, each station must have an address, of course.

Good, it is decided then!

[malex984]

• I wouldn't rely on .ssh/config ... all the devOps software I've seen or used configured host, port (optional) and user (optional) in their "stations" and it seems reasonable as it allows configuration in a single file, doesn't ruin security, and allows using different user/port settings from the ones in .ssh/config (the defaults for console use)... and it'd allow setting up stations sharing IP/address (proxy schemes where each host uses a different port or user) without it being our responsibility (if it works, it works, and we don't care why/how).

Well, this is basically my initial proposal to have all ssh details like ssh_user, ssh_port, ssh_key_ref etc attached to each station (mapping ssh_options). This way i can use any low-level library for ssh access or generate .ssh/config on-the-fly if necessary...

BUT then it was suggested in #25 (comment) (if i understood @porst17 correctly) to remove all those details from general YAML configuration and rely on ~/.ssh/config that will not be maintained by us.

Please vote on:

• should we make both ways possible for example by using station's ssh_options if it is available inside a station configuration and try to use ${HOME}/.ssh/config otherwise?

[porst17]

@malex984 @elondaits I leave the decision to you.

My two cents are:

Contents of ${HOME}/.ssh/ and /etc/ssh/ssh_config (don't forget global settings!) are used by default. Additional values given in the ssh_options of the hilbert config overwrite these values, just like command line parameters of the SSH client overwrite what's given in ${HOME}/.ssh/ and /etc/ssh/ssh_config.

[malex984]

Yes, as this is out of our control, admins may also configure some things (e.g. default settings) in /etc/ssh/ssh_config in addition to ${SPECIAL_USERS_HOME}/.ssh/. That means: Access DB may const of several parts/locations and thus our management container will need to mount them all. Moreover /etc/ssh/ssh_config may reference further files on the host system...

Currently i can see only one way to re-enable the mentioned default overwriting routine within a container: to forget /etc/ssh/ that is originally provided by the docker image via mounting the system's directory on top of it when corresponding container starts.

QUESTION: is there any other way to re-enable the mentioned default overwriting routine?

Well, currently we do not really use /etc/ssh/ but it may easily change in a future (e.g. we may choose to run a separate SSHD within a container)...

Please let me list all discussed points so far concerning where to put ssh access details:

1. my original proposal: everything is embedded into a big general YAML file (including ALL ssh credentials and settings)
2. for security you wanted: ssh keys to be separately installed on the Server host system (${SPECIAL_USERS_HOME}/.ssh/). YAML then has to know how to reference them.
3. @porst17 suggested to additionally separate all remaining ssh settings for accessing stations into ${SPECIAL_USERS_HOME}/.ssh/config in order to keep them together with ssh credentials. YAML has to know ssh aliases.
4. as @porst17 noted ${SPECIAL_USERS_HOME}/.ssh/config may depend on the host's global ssh settings: /etc/ssh/ which is also to be maintained by admins. How does YAML know about that? by default?

I would really prefer if only one of /etc/ssh/ and ${SPECIAL_USERS_HOME}/.ssh/ would be used, which should also be entirely self-contained. Can Hilbert require that?

[porst17]

• Solution 1: Mount /etc/ssh/ and ${SPECIAL_USERS_HOME}/.ssh/ into the the container. Require that all data that ssh requires is contained in these folders and nowhere else (symlinks won't work, hardlinks do of course). Due to the mounting, ssh configuration data changes will immediately be reflected in the container.
• Solution 2: During startup of hilbert, scan through the hilbert YAML config for address: fields and ask hosts ssh for the corresponding configuration using ssh -G address. You can output these options to your own ssh config file into a Host address section. Then you can grep for all options containing .*File to find out which files of the main host are needed in addition (identityfile, globalknownhostsfile, userknownhostsfile et. al.) and add these files together with your auto-generated ssh config into the container. External ssh configuration changes would only be updated after restarting hilbert.

With both solutions, values given in the hilbert config can be used to overwrite the main hosts defaults.

The second option may require more work, but may be more in line with our restart-hilbert-to-update-config philosophy.

[malex984]

@porst17 I 👍 the 2nd solution! It sounds a lot like a special auto_detection action execution before starting the Management container. That way we could create a temporary self-contained directory (e.g. /tmp/.../.ssh) and mount it into the management container anywhere.

See for details

• Designing new command line interface (hilbert) hilbert-docker-images#28 (comment)
• Designing new command line interface (hilbert) hilbert-docker-images#28 (comment)
• Designing new command line interface (hilbert) hilbert-docker-images#28 (comment)

[elondaits]

The 2nd solution still requires someone to configure which port and user to use for each host (the command outputs the defaults for the user it's running under, otherwise), so I'm not sure we're gaining anything.

Also, my original comment's intention was "we should not ruin security by doing security sensitive stuff without being experts"... but right now we're adding a new vector, namely:

• /tmp is one of the lowest security directories (might be world writable, even). You shouldn't place one of the most secure directories inside one of the least secure ones.
• Also, when creating temp files / directories for sensitive information the normal practice is to make the path impossible to guess, with a secure hash (like browser caches directories)... but that's an extra problem we should not have. I can think of these things and we can think of how to solve them but I'm not a security expert... so if I can think of two security problems right away it means there are 20 at least, and we're not gonna get all.

So I would go with "the simplest thing that will work" which in this case is Solution 1.

[porst17]

@elondaits port and user can be specified in ~/.ssh/config together with all other ssh related things.

Your security related concerns are only valid in the context of @malex984 implementation. It is not a problem of approach 2 in general.

Also: Everybody knows the paths to ssh keys on a machine. The paths are ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub (if not changed by ~/.ssh/config). Obscuring the path is not part of ssh's security. So putting the information extracted automatically from ssh's config into a known place but giving it the same access restricting ~/.ssh/id_rsa has is secure enough. Also, the temp location can be in the users home e.g. in '~/.config/hilbert'.

[elondaits]

@elondaits port and user can be specified in ~/.ssh/config together with all other ssh related things.

Yes. That's why I don't understand why we need a more complex solution than the first one.

Your security related concerns are only valid in the context of @malex984 implementation. It is not a problem of approach 2 in general.

Yes. But it's a more complex solution that requires manipulating possibly sensitive information. The first idea of how to implement it had problems... that's why I think we should aim at the simplest. Specially since we're allowed to be restrictive to the users because these is a dedicated server and dedicated stations.

Also: Everybody knows the paths to ssh keys on a machine. The paths are ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub (if not changed by ~/.ssh/config). Obscuring the path is not part of ssh's security.

I Agree.

So putting the information extracted automatically from ssh's config into a known place but giving it the same access restricting ~/.ssh/id_rsa has is secure enough.

No. That does not follow from the above. You have to be sure nobody else guessed and created the path with extra permissions, for instance. You have to set the umask before creating the files so they're not readable/writeable. I don't claim these are unsolvable problems... just that it's not the same, and since it's not the same we have to think of the ways in which it's not the same. And that's the extra complexity we should not force ourselves to worry about by making things harder than the easiest possible.

Also, the temp location can be in the users home e.g. in '~/.config/hilbert'.

Yes. And so our spec or code (and our tests) would need some notice/comment that said "the temp path should be secure and only readable by the user, don't use the /tmp directory, etc." to reflect this idea we had right now because we're thinking more than 30 seconds in the problem... otherwise someone might change it and it's all for naught... If we can do it simpler, we can code with less concerns.

[malex984]

Well, the current problem is: ssh access is pre-configured on the Server host (e.g. in /etc/ssh/ & ${SPECIAL_USERS_HOME}/.ssh/ with further references to outside of these 2 directories) but we need to use it from within a docker container.

In #25 (comment) @porst17 showed how to find out all possible references (hopefully accessible by that SPECIAL_USER) to outside so that they can all be collected together in a single (temporary but well secured) place to be mounted into the container.

Update: Another way to solve this issue: to proxy ALL ssh connections from within the docker container via Server Host, e.g. access any station STATION via ssh SERVER ssh STATION command but i am not sure whether that is a good idea or not...?

[elondaits]

I understand, but it's not a huge problem. The hilbert server is a dedicated server that must run under whatever OS we support and with the dependencies we need. If we indicate we're only going to mount /etc/ssh and ~/.ssh in the container then it's not a huge problem.

And if someone, for some good reason, really needs to mount extra cfg files or directores, he can add them to the server Dockerfile. And then we can see if we should add that ourselves as well.

A more complex auto-detect feature is non-trivial to develop, needs extra tests, extra docs, and requires us to be careful with security. And it's REALLY likely no one will ever need it, nor use it.

[malex984]

And if someone, for some good reason, really needs to mount extra cfg files or directores, he can add them to the server Dockerfile.

Do you mean docker-compose.yml?

A more complex auto-detect feature is non-trivial to develop, needs extra tests, extra docs, and requires us to be careful with security. And it's REALLY likely no one will ever need it, nor use it.

We already use it a lot in the prototype, where they are part of configuration files which include scripts.

Clearly, there is no need in running any auto-detection on the host system if ALL services and applications were previously containerized by us.
Alas, in general case do need to communicate with some native SW (or use dynamically configured devices, e.g via xinput) or use host data (with configurable location) from inside our containers.

The very trivial case of using auto-detection is already relying on the user's previously set environment variables for mounting some dynamic location into containers (e.g. ${HOME}/.ssh/ or /run/user/${UID}/pulse/native).

Furthermore:

• different systems like to configure standard services in non-standard way (Docker Engine's socket may change location of be disabled by admins in favor of tcp access)
• using X11 from inside a docker container requires correct X11 socket detection (and mounting) + generation of a special XAuthority file (in the same context where X11 server itself runs - e.g. on the host system).
• some special devices may need to be detected and prepared before using them to avoid the dangerous privileged mode (e.g. QR Reader).

ps: @elondaits please remember the main topic for this issue.

[elondaits]

By auto-detection I meant @porst17 second solution.

[porst17]

@malex984 Your ssh SERVER ssh station has the problem that you need to know ssh credentials to log into SERVER, i.e. you are going round in circles. If you have a good solution for this, I am OK using this as option 3.

I agree with @elondaits that solution 2 might cause problems we are not aware of, because it tries to deal with all possible OpenSSH configurations.

Therefore, I think we should stick to method 1 if @malex984 doesn't come up with well thought-through solution for option 3. Option 1 seems easiest to implement and to maintain for now. Method 1 can be extended in the future by utilizing ideas of method 2: Mount ~/.ssh/ and /etc/ssh as well as all files detected by the ssh -G address approach. This would be a backwards compatible change.

I am curios to see if option 3 can work, but please don't post half-backed ideas.

[malex984]

I am curios to see if option 3 can work, but please don't post half-backed ideas.

@porst17 Here is a working proof-of-concept script. It runs on a host, takes an ssh alias (known to the current user) as its argument and demonstrates all the necessary steps to access account@host specified by the given ssh alias from inside any docker container. Currently it mounts ssh-agent's socket but alternatively it may just mount a single private key.

Conceptually its only benefit is that: the actual proxy may be anywhere => admins may choose a separate (but accessible) host to contain the whole Access DB...
IMO it is not worth the trouble in our situation.

Therefore: i vote for option 1.

[porst17]

OK, let's implement option 1 then.

Your proof-of-concept for option 3 looks very promising and we may implement this in the future if necessary and if we all agree that is doesn't open additional security holes.

[malex984]

Currently hilbert tool uses ~/.ssh/config. It can also check whether address is a valid ssh alias, provided the remote station is accessible via SSH...

Mounting ~/.ssh/config into management image will be implemented within #14

[malex984]

@porst17 DONE: mng service relies on some HILBERT_SERVER_CONFIG_SSH_PATH to contain config + private ssh keys

[porst17]

Ok, we can reopen this issue if we find out that the current implementation isn't good enough.