Creating Hetzner robot admin sub-account fails (2019)

[basvandijk]

@aszlig like in NixOS/nixops#778 it looks like Hetzner has changed its login mechanism again:

mymachine...> creating an exclusive robot admin sub-account for ‘mymachine’... error: 
  Login to robot web interface failed.

This is with the nixops from release-18.09. If I try it with nixops-HEAD (NixOS/nixops@330094b) and hetzner-HEAD (aszlig/hetzner@0df67db) I get the following failure:

mymachine...> creating an exclusive robot admin sub-account for ‘mymachine’... Traceback (most recent call last):
  File "/nix/store/5ihzkfgv8iv4klpbr5pr61pirc538bd6-nixops-1.6.1pre0_b61177/bin/..nixops-wrapped-wrapped", line 990, in <module>
    args.op()
  File "/nix/store/5ihzkfgv8iv4klpbr5pr61pirc538bd6-nixops-1.6.1pre0_b61177/bin/..nixops-wrapped-wrapped", line 411, in op_deploy
    max_concurrent_activate=args.max_concurrent_activate)
  File "/nix/store/5ihzkfgv8iv4klpbr5pr61pirc538bd6-nixops-1.6.1pre0_b61177/lib/python2.7/site-packages/nixops/deployment.py", line 1057, in deploy
    self.run_with_notify('deploy', lambda: self._deploy(**kwargs))
  File "/nix/store/5ihzkfgv8iv4klpbr5pr61pirc538bd6-nixops-1.6.1pre0_b61177/lib/python2.7/site-packages/nixops/deployment.py", line 1046, in run_with_notify
    f()
  File "/nix/store/5ihzkfgv8iv4klpbr5pr61pirc538bd6-nixops-1.6.1pre0_b61177/lib/python2.7/site-packages/nixops/deployment.py", line 1057, in <lambda>
    self.run_with_notify('deploy', lambda: self._deploy(**kwargs))
  File "/nix/store/5ihzkfgv8iv4klpbr5pr61pirc538bd6-nixops-1.6.1pre0_b61177/lib/python2.7/site-packages/nixops/deployment.py", line 990, in _deploy
    nixops.parallel.run_tasks(nr_workers=-1, tasks=self.active_resources.itervalues(), worker_fun=worker)
  File "/nix/store/5ihzkfgv8iv4klpbr5pr61pirc538bd6-nixops-1.6.1pre0_b61177/lib/python2.7/site-packages/nixops/parallel.py", line 44, in thread_fun
    result_queue.put((worker_fun(t), None, t.name))
  File "/nix/store/5ihzkfgv8iv4klpbr5pr61pirc538bd6-nixops-1.6.1pre0_b61177/lib/python2.7/site-packages/nixops/deployment.py", line 963, in worker
    r.create(self.definitions[r.name], check=check, allow_reboot=allow_reboot, allow_recreate=allow_recreate)
  File "/nix/store/5ihzkfgv8iv4klpbr5pr61pirc538bd6-nixops-1.6.1pre0_b61177/lib/python2.7/site-packages/nixops/backends/hetzner.py", line 614, in create
    self.robot_admin_pass) = server.admin.create()
  File "/nix/store/jq7irfcmj5qsgg9kds427glhvjh7iyfr-python2.7-hetzner-0.8.1/lib/python2.7/site-packages/hetzner/server.py", line 440, in admin
    self._admin_account = AdminAccount(self)
  File "/nix/store/jq7irfcmj5qsgg9kds427glhvjh7iyfr-python2.7-hetzner-0.8.1/lib/python2.7/site-packages/hetzner/server.py", line 182, in __init__
    self.update_info()
  File "/nix/store/jq7irfcmj5qsgg9kds427glhvjh7iyfr-python2.7-hetzner-0.8.1/lib/python2.7/site-packages/hetzner/server.py", line 188, in update_info
    self._scraper.login()
  File "/nix/store/jq7irfcmj5qsgg9kds427glhvjh7iyfr-python2.7-hetzner-0.8.1/lib/python2.7/site-packages/hetzner/robot.py", line 127, in login
    login_conn.request('GET', auth_url[len(ROBOT_LOGINHOST) + 8:], None)
  File "/nix/store/6qlyc5b20r861ljkgagr8hqwk4sb8x06-python-2.7.15/lib/python2.7/httplib.py", line 1042, in request
    self._send_request(method, url, body, headers)
  File "/nix/store/6qlyc5b20r861ljkgagr8hqwk4sb8x06-python-2.7.15/lib/python2.7/httplib.py", line 1082, in _send_request
    self.endheaders(body)
  File "/nix/store/6qlyc5b20r861ljkgagr8hqwk4sb8x06-python-2.7.15/lib/python2.7/httplib.py", line 1038, in endheaders
    self._send_output(message_body)
  File "/nix/store/6qlyc5b20r861ljkgagr8hqwk4sb8x06-python-2.7.15/lib/python2.7/httplib.py", line 882, in _send_output
    self.send(msg)
  File "/nix/store/6qlyc5b20r861ljkgagr8hqwk4sb8x06-python-2.7.15/lib/python2.7/httplib.py", line 844, in send
    self.connect()
  File "/nix/store/jq7irfcmj5qsgg9kds427glhvjh7iyfr-python2.7-hetzner-0.8.1/lib/python2.7/site-packages/hetzner/util/http.py", line 68, in connect
    ca_certs=cafile)
  File "/nix/store/6qlyc5b20r861ljkgagr8hqwk4sb8x06-python-2.7.15/lib/python2.7/ssl.py", line 949, in wrap_socket
    ciphers=ciphers)
  File "/nix/store/6qlyc5b20r861ljkgagr8hqwk4sb8x06-python-2.7.15/lib/python2.7/ssl.py", line 617, in __init__
    self.do_handshake()
  File "/nix/store/6qlyc5b20r861ljkgagr8hqwk4sb8x06-python-2.7.15/lib/python2.7/ssl.py", line 846, in do_handshake
    self._sslobj.do_handshake()
socket.error: [Errno 104] Connection reset by peer

[aszlig]

@basvandijk: Geesh, thanks for the report... will fix that until tomorrow.

[basvandijk]

Great! Please let me know how I can help.

[aszlig]

@basvandijk: Hm, for all of my robot accounts this works... Did you by chance enable 2FA?

[basvandijk]

No 2FA is not enabled. One thing I did do differently when ordering a new server was choosing to log into the rescue system using my public key instead of via a password but I don't think this matters for logging into the robot interface.

[aszlig]

@basvandijk: Can you check whether the hetznerctl admin command of the hetzner Python package works for you?

[FPtje]

Hi, colleague here. When making a new attempt to provision the device mentioned in the OP I hit the same issue. The hetznerctl admin command works as intended. In the end I managed to provision the device by temporarily setting deployment.hetzner.robotUser and deployment.hetzner.robotPass in its nix configuration.

The fact that this works probably means the environment variables HETZNER_ROBOT_USER and/or HETZNER_ROBOT_PASS were either not passed to nixops properly or were ignored. I've found that our deploy command (part of a Haskell shake script) runs nixops in a nix-shell:

nix-shell -E 'with (import ./.); myEnvFun { name = "nixops";  buildInputs = [ nixops nix ]; }' --run 'nixops deploy -s MyState.nixops -d stalling-net --include MyDevice'

The running in a nix shell was added by @basvandijk to run Nixops HEAD (as mentioned in the OP), but it's extremely likely that either our Haskell shake tool or the Nix shell fail to pass the environment variables.

I've found that running an echo $TESTVARIABLE within the shell does not print anything, even when setting it right before running the command. Only exporting the variable works. This is a strong hint suggesting that the hetzner variables weren't being passed properly.

The machine is provisioned now, making it hard to test, but it seems more than likely that the issue was on our side. As such I think the issue can be closed. @basvandijk Do you agree?

[flicaflow]

@FPtje workaround does not work for me. It does not make any difference wether I defiene robot user and pass in the nix file or in the environment. Is there any progress in solving this issue?
The last time I created a hetzner deployment was end of February and it worked flawless. This time not so much. I'm on the 18.09 branch.

[flicaflow]

the hetznerctl tool does not work for me as well, I tracked the problem down to https://github.com/aszlig/hetzner/blob/master/hetzner/robot.py#L173 , it seems the server does not set the expected cookie. Maybe hetzner changed there api?

[aszlig]

@flicaflow: This is a different issue than the one reported here, but you're right, Hetzner now uses a CSRF token. Will publish a new version with the fix shortly. Fix published.

[aszlig]

@basvandijk: Given @FPtje's comment, can this be closed?