Creating Hetzner robot admin sub-account fails (2017)

[basvandijk]

I get a hetzner.WebRobotError exception when provisioning a new Hetzner machine. This is with nixops-1.5.2 on NixOS-17.09.

When I upgrade nixops to the latest HEAD (9203440) I get the same error:

$ HETZNER_ROBOT_PASS=XXX nixops deploy -d my-net --include my-machine
my-machine...............................> 
creating an exclusive robot admin sub-account for ‘my-machine’... 
Traceback (most recent call last):
  File "/nix/store/p2a5cm95zwv67zawb3fm7jf5awqp5x64-nixops-1.6pre2276_9203440/bin/..nixops-wrapped-wrapped", line 955, in <module>
    args.op()
  File "/nix/store/p2a5cm95zwv67zawb3fm7jf5awqp5x64-nixops-1.6pre2276_9203440/bin/..nixops-wrapped-wrapped", line 382, in op_deploy
    repair=args.repair, dry_activate=args.dry_activate)
  File "/nix/store/p2a5cm95zwv67zawb3fm7jf5awqp5x64-nixops-1.6pre2276_9203440/lib/python2.7/site-packages/nixops/deployment.py", line 983, in deploy
    self._deploy(**kwargs)
  File "/nix/store/p2a5cm95zwv67zawb3fm7jf5awqp5x64-nixops-1.6pre2276_9203440/lib/python2.7/site-packages/nixops/deployment.py", line 944, in _deploy
    nixops.parallel.run_tasks(nr_workers=-1, tasks=self.active_resources.itervalues(), worker_fun=worker)
  File "/nix/store/p2a5cm95zwv67zawb3fm7jf5awqp5x64-nixops-1.6pre2276_9203440/lib/python2.7/site-packages/nixops/parallel.py", line 41, in thread_fun
    result_queue.put((worker_fun(t), None))
  File "/nix/store/p2a5cm95zwv67zawb3fm7jf5awqp5x64-nixops-1.6pre2276_9203440/lib/python2.7/site-packages/nixops/deployment.py", line 917, in worker
    r.create(self.definitions[r.name], check=check, allow_reboot=allow_reboot, allow_recreate=allow_recreate)
  File "/nix/store/p2a5cm95zwv67zawb3fm7jf5awqp5x64-nixops-1.6pre2276_9203440/lib/python2.7/site-packages/nixops/backends/hetzner.py", line 599, in create
    self.robot_admin_pass) = server.admin.create()
  File "/nix/store/m6wybn3vn6fwjnb5wwlr48q0wyyhxcs4-python2.7-hetzner-0.7.5/lib/python2.7/site-packages/hetzner/server.py", line 374, in admin
    self._admin_account = AdminAccount(self)
  File "/nix/store/m6wybn3vn6fwjnb5wwlr48q0wyyhxcs4-python2.7-hetzner-0.7.5/lib/python2.7/site-packages/hetzner/server.py", line 170, in __init__
    self.update_info()
  File "/nix/store/m6wybn3vn6fwjnb5wwlr48q0wyyhxcs4-python2.7-hetzner-0.7.5/lib/python2.7/site-packages/hetzner/server.py", line 176, in update_info
    self._scraper.login()
  File "/nix/store/m6wybn3vn6fwjnb5wwlr48q0wyyhxcs4-python2.7-hetzner-0.7.5/lib/python2.7/site-packages/hetzner/robot.py", line 142, in login
    " page".format(response.status))
hetzner.WebRobotError

When I upgrade the hetzner python library to the latest HEAD with the following bugfix: aszlig/hetzner#28 I get a different error:

$ HETZNER_ROBOT_PASS=XXX nixops deploy -d my-net --include my-machine
my-machine...............................> 
creating an exclusive robot admin sub-account for ‘my-machine’... 
error: Invalid status code 302 while visiting login page

@aszlig any idea what's going wrong?

[aszlig]

@basvandijk: Yep, Hetzner has changed their login mechanism for the Robot, see aszlig/hetzner#26. I tried to fix it but it's a bit more involved, so maybe by the end of the week if there aren't any more surprises.

[aszlig]

As a side note, we might want to make it easier to do that step manually if the scraper fails the next time this happens. Right now you need to temporarily set createSubAccount to false, create the account manually and pass it via HETZNER_ROBOT_USER/HETZNER_ROBOT_PASS to the new machine like eg.:

HETZNER_ROBOT_USER=foo HETZNER_ROBOT_PASS="$(cat)" nixops deploy --include newmachine

[basvandijk]

@aszlig thanks for showing the workaround!

Let me know if I can help in any way with fixing the login problem. Do you have to reverse engineer the login procedure or is there any API documentation?

[aszlig]

@basvandijk: I've fixed the login with aszlig/hetzner@2ed0b03, can you try whether it works now with the latest master version of the hetzner library?

[basvandijk]

@aszlig with the latest master of the hetzner library nixops can now successfully create a sub-account. Thanks!

Now I face a new problem:

$ HETZNER_ROBOT_PASS=XXX nixops deploy -d my-net --include my-machine
my-machine...............................> installing machine...
my-machine...............................> rebooting machine ‘my-machine’ (XXX.XXX.XXX.XXX) into rescue system
my-machine...............................> sending reboot command... 
my-machine...............................> could not connect to ‘root@XXX.XXX.XXX.XXX’, retrying in 1 seconds...
my-machine...............................> could not connect to ‘root@XXX.XXX.XXX.XXX’, retrying in 2 seconds...
my-machine...............................> could not connect to ‘root@XXX.XXX.XXX.XXX’, retrying in 4 seconds...
my-machine...............................> could not connect to ‘root@XXX.XXX.XXX.XXX’, retrying in 8 seconds...
error: unable to start SSH master connection to ‘root@XXX.XXX.XXX.XXX’

[basvandijk]

It's working now. The problem was that by default I specify:

deployment.targetPort = nonStandardSshPort

I had to temporarily switch it to 22.

[aszlig]

@basvandijk: Hm, I wonder why it failed on reboot and not way before rebooting while sshing to the rescue system?

[basvandijk]

@aszlig it did fail when trying to reboot to the rescue system.

I think it's a good idea to make a new release of your hetzner library, upgrade it in nixpkgs and cherry-pick it on release-17.09. After that we can close this issue.