Issue with private keys when updating to AiiDA 1.3.1

[JPchico]

Bug description

After updating aiida-core from the development branch (git hash 0094f70) computers previously configured are unreachable.
verdi computer test computer_name fails

Info: Testing computer<computer_name> for user<user_email>...
* Opening connection... 09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR] Unknown exception: q must be exactly 160, 224, or 256 bits long
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR] Traceback (most recent call last):
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR]   File "/home/user_name/ENVS/aiida_py3/lib/python3.6/site-packages/paramiko/transport.py", line 2109, in run
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR]     handler(self.auth_handler, m)
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR]   File "/home/user_name/ENVS/aiida_py3/lib/python3.6/site-packages/paramiko/auth_handler.py", line 298, in _parse_service_accept
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR]     sig = self.private_key.sign_ssh_data(blob)
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR]   File "/home/user_name/ENVS/aiida_py3/lib/python3.6/site-packages/paramiko/dsskey.py", line 116, in sign_ssh_data
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR]     ).private_key(backend=default_backend())
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR]   File "/home/user_name/ENVS/aiida_py3/lib/python3.6/site-packages/cryptography/hazmat/primitives/asymmetric/dsa.py", line 250, in private_key
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR]     return backend.load_dsa_private_numbers(self)
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR]   File "/home/user_name/ENVS/aiida_py3/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 854, in load_dsa_private_numbers
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR]     dsa._check_dsa_private_numbers(numbers)
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR]   File "/home/user_name/ENVS/aiida_py3/lib/python3.6/site-packages/cryptography/hazmat/primitives/asymmetric/dsa.py", line 147, in _check_dsa_private_numbers
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR]     _check_dsa_parameters(parameters)
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR]   File "/home/user_name/ENVS/aiida_py3/lib/python3.6/site-packages/cryptography/hazmat/primitives/asymmetric/dsa.py", line 139, in _check_dsa_parameters
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR]     raise ValueError("q must be exactly 160, 224, or 256 bits long")
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR] ValueError: q must be exactly 160, 224, or 256 bits long
09/15/2020 01:30:52 PM <7153> paramiko.transport: [ERROR] 
09/15/2020 01:30:52 PM <7153> aiida.transport.SshTransport: [ERROR] Error connecting to 'hostname' through SSH: [SshTransport] q must be exactly 160, 224, or 256 bits long, connect_args were: {'username': 'user_name', 'port': 22, 'look_for_keys': True, 'key_filename': '/home/user_name/.ssh/id_rsa', 'timeout': 60, 'allow_agent': True, 'compress': True, 'gss_auth': False, 'gss_kex': False, 'gss_deleg_creds': False, 'gss_host': 'hostname'}
[FAILED]: Error while trying to connect to the computer
  Use the `--print-traceback` option to see the full traceback.
Warning: 1 out of 0 tests failed

verdi computer configure show computer_name

* username               user_name
* port                   22
* look_for_keys          True
* key_filename           /home/user_name/.ssh/id_rsa
* timeout                60
* allow_agent            True
* proxy_command
* compress               True
* gss_auth               False
* gss_kex                False
* gss_deleg_creds        False
* gss_host               hostname
* load_system_host_keys  True
* key_policy             RejectPolicy
* use_login_shell        False
* safe_interval          5.0

The lack of communication with the computer results in the inability to submit any jobs to the affected computer.

To solve the problem one can change the private key specified in the configuration for the public key.

Steps to reproduce

1. Have an installation of AiiDA 1.2
2. configure an ssh computer using the private key
3. Update AiiDA to hash 0094f70
4. Run verdi computer test computer_name

Your environment

• Ubuntu 18.04.5 LTS (GNU/Linux 5.0.0-1029-azure x86_64)
• Python 3.6.9 (default, Jul 17 2020, 12:50:27) [GCC 8.4.0]
• aiida-core hash 0094f70
• RabbitMQ 3.7.6
• PostgreSQL 10.14 (Ubuntu 10.14-0ubuntu0.18.04.1) on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
• Running in an Azure VM

[sphuber]

Thanks for the report @JPchico . Could you please add the version of paramiko that is installed in the environment? We had the suspicion that a new version of paramiko might play a role as well.

[JPchico]

Sure the version of paramiko that is currently installed in my virtualenv is paramiko==2.7.2

[ltalirz]

There are a couple of reports on the internet of q must be exactly 160, 224, or 256 bits long from other frameworks that depend on paramiko... it does seem like something changed.

The fact that the problem goes away if you specify the public key (which cannot be used to authenticate) suggests to me that somehow that leads paramiko to another codepath which then picks up the private key (or even another private key) nevertheless.

I just want to stress that this is not a bug across the board (I personally cannot reproduce it, even with the latest version of paramiko).

@JPchico Would you mind creating a new private key using ssh-keygen and ssh-copy-id -i keyfile computername and try whether you can get aiida to connect using that one?

[ltalirz]

P.S. What is the first line of your private key - is it

-----BEGIN RSA PRIVATE KEY-----

[ltalirz]

Just to exclude another possibility (that the old key is broken somehow and ssh previously ended up using a different key), please check that the old key is indeed working for the connection using something like

ssh computername -i /home/user_name/.ssh/id_rsa whoami

[JPchico]

So I tried producing a new ssh key with ssh-keygen and that works without problem. I can connect without issue to the computer using AiiDA.

I checked that the previous ssh key that I was using is in the openssh format.

-----BEGIN OPENSSH PRIVATE KEY-----

Which is a bit surprising, I wonder if it was something due to a version of ssh-keygen that was installed in my Azure VM when I configured my profile the first time?

[ltalirz]

Hi @JPchico , great to hear that you're back to a proper setup.

What I can tell you looking from the outside:

• paramiko below version 2.7 did not support parsing keys that start with -----BEGIN OPENSSH PRIVATE KEY----- - i.e.
  something else must have been going on under the hood (perhaps getting the key from a running ssh-agent does not have the same limitation; I don't know)
• I guess it now starts complaining because it is actually parsing the key. This can mean one of two things:
  A. The key is malformed.
  B. There is a bug in parsing the key (or somehow its particular format is not supported)

It would be nice to figure out whether it is A or B, and so I'd still like you to do two things:

1. Can you try the whoami command I suggested above?
   This will tell us whether the key is malformed.
2. If the key works fine, you could consider removing its public counterpart from the "authorized_keys" in the places you connected from your VM and we could open an issue on paramiko with the private key itself attached (that is if you no longer need it and can use the new one you generated).

P.S. I did notice that paramiko has a whopping 196 open PRs which is a bit worrying... there does seem to still be development activity, though: https://github.com/paramiko/paramiko/graphs/contributors

[JPchico]

Hi @ltalirz

I'm very surprised that it was working at all I wonder what was happening ..... I have used this key before with paramiko to do a small dashboard tracking some processes in that cluster, so I have no clue on what is the issue.

I tried the whoami command above and that works fine for both the previous and current key.
I have removed the offending key from the authorized_keys as you suggested and everything works fine.

The only thing I can imagine is that I'm using the cyclecloud service from Azure, and I'm using an ssh agent to ensure communication between server and remote.

There seems to be some issues with paramiko that might be related to this problem?

P.S. There seems to be some issues in the development of paramiko so the number of PRs is not surprising.

[ltalirz]

I tried the whoami command above and that works fine for both the previous and current key.
I have removed the offending key from the authorized_keys as you suggested and everything works fine.

Ok, thanks. As I mentioned, you may then want to open an issue on paramiko's tracker attaching your key.
It is supposed to support OPENSSH private keys since 2.7.0 but it seems like it doesn't like your key for some reason.

I'm using an ssh agent to ensure communication between server and remote.

I suspect that is what saved you earlier. paramiko has an interface to retrieve keys from the agent - @giovannipizzi may know more.
Perhaps this was used as a fallback option when parsing the original key failed.

There seems to be some issues with paramiko that might be related to this problem?

The issue that you linked is very old and I suspect it is outdated as of the 2.7 release.

P.S. There seems to be some issues in the development of paramiko so the number of PRs is not surprising.

Thanks for making me aware of this, I didn't know. @csadorf Perhaps paramiko should be added to the "watchlist".
On the other hand, admitting the problem is indeed the first step to a solution, so let's see how things go.

As mentioned in #3929, we may in any case consider switching to a different library such as parallel-ssh, both for performance and feature reasons.

I'll close this issue for now since this seems to be an issue of paramiko with your specific OPENSSH key.
If others run into this, creating a new key pair is likely the solution.