Add ansible config for dev desktops #79

oli-obk · 2021-10-14T14:44:11Z

Discussion in https://rust-lang.zulipchat.com/#narrow/stream/242791-t-infra/topic/cloud.20dev.20desktops

Mark-Simulacrum · 2021-10-14T14:47:33Z

It probably makes sense to just inline all the files into simpleinfra rather than checking out a separate repository -- we'd want to move that into rust-lang at least, and it doesn't seem like there's much benefit to keeping it separate rather than integrated into this repository.

ansible/roles/dev-desktop/templates/skel/init.sh

ansible/roles/dev-desktop/templates/team_login/src/main.rs

pietroalbini · 2021-10-26T17:39:19Z

Hmm, is there a reason why the tool to create the users is a full Rust program depending on reqwest? A simple bash script with jq to parse the JSON will likely be enough.

oli-obk · 2021-10-26T18:23:20Z

Hmm, is there a reason why the tool to create the users is a full Rust program depending on reqwest? A simple bash script with jq to parse the JSON will likely be enough.

It started out as a tool that was run on each login and that took the username as an argument. The first few bash iterations of this got "hacked" by ppl I showed it to, just by looking at it. So I made it a Rust program where I was able to reason about things. Even reading articles about making bash scripts secure, I was not certain I could create one.

Later when the script became a cron job, this was much less important, the only insecure input is the json dump from github, and if that is taken over we have lost anyway.

So... in essence, yes we could go back to a bash script using jq, it's just not straight forward to understand what is going on in it (to me, even though I'm the author).

Mark-Simulacrum · 2021-12-09T20:42:08Z

Pushed up a few commits with fixes/improvements while experimenting with the sample machine. One likely blocker I noted is removing the global Rust installation, which conflicts with usage of rustup (or at least rustup complains about it).

ansible/roles/dev-desktop/tasks/main.yml

Mark-Simulacrum · 2022-03-16T15:10:16Z

TASK [dev-desktop : Build team login cron job] **********************************************************************
fatal: [dev-desktop.infra.rust-lang.org]: FAILED! => {"changed": true, "cmd": "cd /root/team_login && cargo build", "delta": "0:00:00.002152", "end": "2022-03-16 15:04:41.149915", "msg": "non-zero return code", "rc": 127, "start": "2022-03-16 15:04:41.147763", "stderr": "/bin/sh: 1: cargo: not found", "stderr_lines": ["/bin/sh: 1: cargo: not found"], "stdout": "", "stdout_lines": []}

It looks like we're not finding the rustup install -- maybe an explicit PATH is needed or something, not sure.

Mark-Simulacrum · 2022-03-16T16:12:47Z

That did not seem to help; I'm still seeing the same error after pulling the latest changes.

Mark-Simulacrum · 2022-03-16T17:07:49Z

TASK [dev-desktop : Build team login cron job] ***************************************************************************************************************************************************************
fatal: [dev-desktop.infra.rust-lang.org]: FAILED! => {"changed": true, "cmd": "cd /root/team_login && PATH=$PATH:$HOME/.cargo/bin cargo build", "delta": "0:00:26.167428", "end": "2022-03-16 17:03:56.949407", "msg": "non-zero return code", "rc": 101, "start": "2022-03-16 17:03:30.781979", "stderr": "    Updating crates.io index\n Downloading crates ...\n  Downloaded openssl-sys v0.9.67\n  Downloaded autocfg v1.0.1\n  Downloaded unicode-xid v0.2.2\n  Downloaded proc-macro2 v1.0.29\n  Downloaded pkg-config v0.3.20\n  Downloaded socket2 v0.4.2\n  Downloaded ryu v1.0.5\n  Downloaded openssl-probe v0.1.4\n  Downloaded itoa v0.4.8\n  Downloaded quote v1.0.9\n  Downloaded syn v1.0.77\n  Downloaded curl v0.4.41\n  Downloaded libc v0.2.103\n  Downloaded cc v1.0.70\n  Downloaded libz-sys v1.1.3\n  Downloaded curl-sys v0.4.51+curl-7.80.0\n  Downloaded mini-internal v0.1.16\n  Downloaded miniserde v0.1.16\n   Compiling pkg-config v0.3.20\n   Compiling cc v1.0.70\n   Compiling libc v0.2.103\n   Compiling autocfg v1.0.1\n   Compiling proc-macro2 v1.0.29\n   Compiling unicode-xid v0.2.2\n   Compiling syn v1.0.77\n   Compiling ryu v1.0.5\n   Compiling curl v0.4.41\n   Compiling itoa v0.4.8\n   Compiling openssl-probe v0.1.4\nerror: linker `cc` not found\n  |\n  = note: No such file or directory (os error 2)\n\nerror: could not compile `curl` due to previous error\nwarning: build failed, waiting for other jobs to finish...\nerror: build failed", "stderr_lines": ["    Updating crates.io index", " Downloading crates ...", "  Downloaded openssl-sys v0.9.67", "  Downloaded autocfg v1.0.1", "  Downloaded unicode-xid v0.2.2", "  Downloaded proc-macro2 v1.0.29", "  Downloaded pkg-config v0.3.20", "  Downloaded socket2 v0.4.2", "  Downloaded ryu v1.0.5", "  Downloaded openssl-probe v0.1.4", "  Downloaded itoa v0.4.8", "  Downloaded quote v1.0.9", "  Downloaded syn v1.0.77", "  Downloaded curl v0.4.41", "  Downloaded libc v0.2.103", "  Downloaded cc v1.0.70", "  Downloaded libz-sys v1.1.3", "  Downloaded curl-sys v0.4.51+curl-7.80.0", "  Downloaded mini-internal v0.1.16", "  Downloaded miniserde v0.1.16", "   Compiling pkg-config v0.3.20", "   Compiling cc v1.0.70", "   Compiling libc v0.2.103", "   Compiling autocfg v1.0.1", "   Compiling proc-macro2 v1.0.29", "   Compiling unicode-xid v0.2.2", "   Compiling syn v1.0.77", "   Compiling ryu v1.0.5", "   Compiling curl v0.4.41", "   Compiling itoa v0.4.8", "   Compiling openssl-probe v0.1.4", "error: linker `cc` not found", "  |", "  = note: No such file or directory (os error 2)", "", "error: could not compile `curl` due to previous error", "warning: build failed, waiting for other jobs to finish...", "error: build failed"], "stdout": "", "stdout_lines": []}

Looks like this needs curl installed externally.

Mark-Simulacrum · 2022-03-16T19:57:26Z

Couple notes:

It looks like the message of the day doesn't quite work, or at least logging in myself I'm not seeing it yet
Install the team login binary when run through ansible after the initial deploy can fail with "cannot create regular file" if the binary is currently running. Ideally we can seamlessly redeploy as needed without impacting existing users, so this seems like it's good to fix. (Particularly if we e.g. modify the source code of the team login script).
the "common : remove old group memberships" task in Ansible will drop allow-ssh from a bunch of users that get created by the cron job - I'm not sure what the right fix here is, but presumably this means that deploying ansible will erase a bunch of permissions, which seems not great. (They will get restored in ~5 minutes, but still.)

oli-obk · 2022-03-16T20:12:44Z

It looks like the message of the day doesn't quite work, or at least logging in myself I'm not seeing it yet

Yea I was worried about that. The default sshd config disables motd and I'm not sure if reenabling it actually works in this setup. We could change the default sshd config, but that would change it for all servers.

Install the team login binary when run through ansible after the initial deploy can fail with "cannot create regular file" if the binary is currently running.

Huh? I thought linux can overwrite binaries that are running and that there are no locks on them? I'll look into that. Maybe we just spawn a script that keeps trying to copy the new binary until that succeeds.

the "common : remove old group memberships" task in Ansible will drop allow-ssh from a bunch of users that get created by the cron job -

Launch the cron script once manually right after deploy?

Mark-Simulacrum · 2022-03-16T20:41:01Z

Yea I was worried about that. The default sshd config disables motd and I'm not sure if reenabling it actually works in this setup. We could change the default sshd config, but that would change it for all servers.

I do see some message of the day, I think, but seems like something to fix. I don't think the particular fix is too important -- if we need to enable messages of the day across all servers, that seems OK, for example.

Huh? I thought linux can overwrite binaries that are running and that there are no locks on them? I'll look into that. Maybe we just spawn a script that keeps trying to copy the new binary until that succeeds.

fatal: [dev-desktop.infra.rust-lang.org]: FAILED! => {"changed": true, "cmd": "cp /root/team_login/target/debug/team_login /etc/cron.team_login", "delta": "0:00:00.002736", "end": "2022-03-16 19:55:11.805673", "msg": "non-zero return code", "rc": 1, "start": "2022-03-16 19:55:11.802937", "stderr": "cp: cannot create regular file '/etc/cron.team_login': Text file busy", "stderr_lines": ["cp: cannot create regular file '/etc/cron.team_login': Text file busy"], "stdout": "", "stdout_lines": []}

Launch the cron script once manually right after deploy?

I think we probably want to not run the drop groups task for this server, and potentially that means it needs a different common base (or parts of the common base need factoring out); it's our first server with a non-empty set of additional users that aren't just the infra admins and aren't created by Ansible.

(An alternative approach could be to move the user creation out of the server cron job and into Ansible, and then add logic to e.g. rust-lang/sync-team to run Ansible to deploy changes across servers after team repo updates. But that is a larger change, for sure.)

oli-obk · 2022-03-17T16:10:11Z

It looks like the message of the day doesn't quite work, or at least logging in myself I'm not seeing it yet

check if it works now. According to the update-motd docs there is a cronjob updating it every 10 minutes

Mark-Simulacrum · 2022-03-17T16:38:46Z

TASK [dev-desktop : Set up the team login cron job] 
fatal: [dev-desktop.infra.rust-lang.org]: FAILED! => {"msg": "The conditional check 'task_result.rc == 0' failed. The error was: error while evaluating conditional (task_result.rc == 0): 'dict object' has no attribute 'rc'"}

Mark-Simulacrum · 2022-03-17T16:53:11Z

OK, that seemed to work in terms of deploying. I'm not yet seeing the message of the day this is trying to configure though:

Welcome to Ubuntu 20.04.4 LTS (GNU/Linux 5.13.0-1017-aws x86_64)

 * Documentation:  https://help.ubuntu.com
 * Management:     https://landscape.canonical.com
 * Support:        https://ubuntu.com/advantage

  System information as of Thu Mar 17 16:51:48 UTC 2022

  System load:  0.05               Processes:             364
  Usage of /:   2.2% of 193.82GB   Users logged in:       0
  Memory usage: 1%                 IPv4 address for ens5: 10.0.0.85
  Swap usage:   0%

 * Ubuntu Pro delivers the most comprehensive open source security and
   compliance features.

   https://ubuntu.com/aws/pro

21 updates can be applied immediately.
3 of these updates are standard security updates.
To see these additional updates run: apt list --upgradable


*** System restart required ***
Last login: Wed Mar 16 19:54:09 2022 from ...

Mark-Simulacrum · 2022-03-17T17:03:56Z

No obvious effect. FWIW, it seems like a good idea to replace the default MOTD text with the one we are trying to provide, both to make it more prominent and since I think the default isn't particularly useful.

Mark-Simulacrum · 2022-03-17T17:25:22Z

No more motd showing at all. FWIW, you should have ssh access, so you can probably try to make it work on the machine directly and then reproduce whatever steps into Ansible.

oli-obk · 2022-03-17T17:26:38Z

could you remove the IP address lockdown? That would make it easier and we'll soon have to do it anyway

Mark-Simulacrum · 2022-03-17T17:26:50Z

I don't think it's currently in place already.

Mark-Simulacrum · 2022-03-17T17:55:26Z

OK, that seems to have worked:

Continued use of this server implies that you accept the rules.
You can find the rules at
https://foundation.rust-lang.org/policies/cloud-compute-program/
Last login: Thu Mar 17 17:24:33 2022 from ...
simulacrum@dev-desktop:~$

I think this is nearly ready to go then, probably we just need to button up the policy and confirm with Joel that this will work as the only/primary prompt users get.

@pietroalbini might also want to take a look at the Terraform and Ansible configurations, but I figure in the alpha/beta period with just a few testers the current machine should likely be OK.

pietroalbini · 2022-03-17T18:05:49Z

I'll take a look in the coming days.

Mark-Simulacrum · 2022-03-18T15:02:42Z

We are getting an alert from our monitoring infra:

motd-news.service on dev-desktop failed

Can we disable that systemd service or otherwise make it not actually fail to run?

pietroalbini

Left some comments.

In general, I'd still prefer for the sync script not to be in Rust but to be in either bash or even better python: it'd remove the need to manage the rustup installation on the root user, the problem with replacing binaries, and it'd make on the fly changes easier.

If you don't have the time to do it I can push a commit changing the script over to Python.

ansible/roles/common/tasks/users.yml

ansible/roles/dev-desktop/files/team_login/src/main.rs

ansible/roles/dev-desktop/tasks/main.yml

terraform/dev-desktops/instance.tf

ansible/roles/dev-desktop/files/team_login/src/main.rs

pietroalbini · 2022-03-19T22:29:42Z

Also, how do we expect users to authenticate with GitHub to push code? I might've missed that discussion.

Co-authored-by: Joel Marcey <joelmarcey@rustfoundation.org>

I didn't know why I made them that way after the weekend

Otherwise it clashes with the admin ssh group, which is reset on redeploy. Thus users would not have access for 5 mins after redeploy, as only the next cronjob run would give the perms back.

pietroalbini · 2022-05-21T09:08:21Z

Merging this, we can iterate with other PRs.

Mark-Simulacrum reviewed Oct 14, 2021

View reviewed changes

ansible/roles/dev-desktop/templates/skel/init.sh Outdated Show resolved Hide resolved

Mark-Simulacrum reviewed Oct 26, 2021

View reviewed changes

ansible/roles/dev-desktop/templates/team_login/src/main.rs Outdated Show resolved Hide resolved

Mark-Simulacrum force-pushed the dev-desktop branch from 12b5fbc to 571521a Compare December 9, 2021 21:25

Mark-Simulacrum reviewed Dec 15, 2021

View reviewed changes

ansible/roles/dev-desktop/tasks/main.yml Outdated Show resolved Hide resolved

Mark-Simulacrum reviewed Dec 15, 2021

View reviewed changes

ansible/roles/dev-desktop/tasks/main.yml Outdated Show resolved Hide resolved

pierwill reviewed Dec 20, 2021

View reviewed changes

ansible/roles/dev-desktop/tasks/main.yml Outdated Show resolved Hide resolved

pierwill reviewed Jan 11, 2022

View reviewed changes

ansible/roles/dev-desktop/tasks/main.yml Outdated Show resolved Hide resolved

Mark-Simulacrum force-pushed the dev-desktop branch from fdb9346 to ca3710d Compare March 16, 2022 15:05

pietroalbini reviewed Mar 19, 2022

View reviewed changes

oli-obk and others added 22 commits May 9, 2022 13:16

Update ansible/roles/dev-desktop/files/skel/config.toml

2e3eee3

Co-authored-by: Joel Marcey <joelmarcey@rustfoundation.org>

ansible has pip magic, too

34f7a7b

Set the users' default shell to bash

6e5ef9b

Fix the init script to work without login capabilities

0ab6969

Make init script executable by everyone

7d07c2d

Make all scripts executable by everyone

b63a614

Remove the autogenerated ubuntu header

427776f

Add explainer for all the scripts

96c8fc5

Make help script bash-friendly

baa8ffa

Use ansible to delete files instead of shelling out to rm

a211d7d

Disable motd-news service

b70547b

Also stop the timer via ansible

0d751d1

Fix scripts

2f4974d

Automatically register the worktrees with rustup

0a5f1b7

Add some basic instructions for Github App setup

72c0e08

All your instructions are belong to server

0033849

Reorder steps

70170e2

I didn't know why I made them that way after the weekend

Create git credential helper

f78c741

Make the git credential helper available globally

ebfa0db

Add a seperate ssh group for dev-desktop users

b355200

Otherwise it clashes with the admin ssh group, which is reset on redeploy. Thus users would not have access for 5 mins after redeploy, as only the next cronjob run would give the perms back.

Only add the dev-desktop-allow-ssh group on the dev-desktop

f0df09e

Add worflows permission to instructions

03573ed

oli-obk force-pushed the dev-desktop branch from e486e62 to 03573ed Compare May 9, 2022 13:20

oli-obk added 2 commits May 9, 2022 13:24

Load the correct user when using the app

2364c06

Move docs to forge

0cc3e3c

oli-obk force-pushed the dev-desktop branch from c104484 to 0cc3e3c Compare May 9, 2022 13:28

oli-obk mentioned this pull request May 9, 2022

Move docs over from infra repo rust-lang/rust-forge#626

Merged

pietroalbini approved these changes May 21, 2022

View reviewed changes

pietroalbini merged commit 3006cc7 into rust-lang:master May 21, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ansible config for dev desktops #79

Add ansible config for dev desktops #79

oli-obk commented Oct 14, 2021

Mark-Simulacrum commented Oct 14, 2021

pietroalbini commented Oct 26, 2021

oli-obk commented Oct 26, 2021

Mark-Simulacrum commented Dec 9, 2021

Mark-Simulacrum commented Mar 16, 2022

Mark-Simulacrum commented Mar 16, 2022

Mark-Simulacrum commented Mar 16, 2022

Mark-Simulacrum commented Mar 16, 2022

oli-obk commented Mar 16, 2022

Mark-Simulacrum commented Mar 16, 2022

oli-obk commented Mar 17, 2022

Mark-Simulacrum commented Mar 17, 2022

Mark-Simulacrum commented Mar 17, 2022

Mark-Simulacrum commented Mar 17, 2022

Mark-Simulacrum commented Mar 17, 2022

oli-obk commented Mar 17, 2022

Mark-Simulacrum commented Mar 17, 2022

Mark-Simulacrum commented Mar 17, 2022

pietroalbini commented Mar 17, 2022

Mark-Simulacrum commented Mar 18, 2022

pietroalbini left a comment

pietroalbini commented Mar 19, 2022

pietroalbini commented May 21, 2022

Add ansible config for dev desktops #79

Add ansible config for dev desktops #79

Conversation

oli-obk commented Oct 14, 2021

Mark-Simulacrum commented Oct 14, 2021

pietroalbini commented Oct 26, 2021

oli-obk commented Oct 26, 2021

Mark-Simulacrum commented Dec 9, 2021

Mark-Simulacrum commented Mar 16, 2022

Mark-Simulacrum commented Mar 16, 2022

Mark-Simulacrum commented Mar 16, 2022

Mark-Simulacrum commented Mar 16, 2022

oli-obk commented Mar 16, 2022

Mark-Simulacrum commented Mar 16, 2022

oli-obk commented Mar 17, 2022

Mark-Simulacrum commented Mar 17, 2022

Mark-Simulacrum commented Mar 17, 2022

Mark-Simulacrum commented Mar 17, 2022

Mark-Simulacrum commented Mar 17, 2022

oli-obk commented Mar 17, 2022

Mark-Simulacrum commented Mar 17, 2022

Mark-Simulacrum commented Mar 17, 2022

pietroalbini commented Mar 17, 2022

Mark-Simulacrum commented Mar 18, 2022

pietroalbini left a comment

Choose a reason for hiding this comment

pietroalbini commented Mar 19, 2022

pietroalbini commented May 21, 2022