provision mac minis at nearform as osx machines in jenkins CI

[mhdawson]

We've realized we dont' have an issue covering this as email was used to do the initial arrangements with the Foundation.

We have 2 mac minis at NearForm (bought by the Foundation) to be added to the CI.

We have VMware so we should be able to have 3 machines on each (1 bare metal and 2 VMs).

Next steps include:

• Nearform to configure with ssh and vnc access
• Installation of vmware (not sure if build team members can do this through ssh and vnc or not)
• Creation of the virtual machines, key question here is what osx versions can we get/install
• Assign the IP given by nearform to a VM
• Verify we can reach the VM using this IP
• Run and tweak the ansible playbooks on the VM
• Node build/test passes
• Re-check Compatability tests - (build binaries on latest, run tests on 10.10)
• Decide which OS level and how many we want (ATM all 10.15 - 1 release, 3 test. Can revaluate later if needed)
• Set up and ansible the remaining VMs (reinstall any 10.14 as 10.15)
• Add the VMs to jenkins
• Add the 3 test machines to node-test-commit-osx, and 1 release machine to iojs+release
• Notarization: should be possible on these release builds, confirm that it works (or possibly complete the notarization process ongoing in macOS Installer Requires Notarization To Be Run Under Catalina and Beyond node#29216)
• Check the nightlys do work on all platforms
• Check that all other jobs using osx can run on 10.15/nearform (v8, etc.)
• Update all jobs to use these machines (Which jobs use OSX?)

[mhdawson]

@BethGriggs, @jasnell I'm hoping we only need Nearform to do the first step on the list above.

[jasnell]

We have it most of the way. I need to follow up this week with our infrastructure guy

[BethGriggs]

@jasnell, any updates?

[rvagg]

I'm not sure running bare is going to be a good idea, isolating via vm is nice for security and the ease of clean-up & replace as needed

[mhdawson]

Using just the 2 VMs makes sense to me. I thought somebody had mentioned using bare metal as the third, but avoiding that seems safer to me as well. Don't think that really changes any of the steps above except we won't do it for the bare metal machine.

[sam-github]

@AshCripps and I have started looking at this. ssh and vnc was already setup, its working.

The next steps are VMWare setup.

@jasnell this can (EDIT: "must") only be done locally, but we are still looking into it, it might not be possible to use VMWare at all on a mac.

[rvagg]

it might not be possible to use VMWare at all on a mac.

It is, I ran test-requireio-osx-* and release-requireio-osx-* from a single MacMini for years and it was the only machine we had when Voxer dropped us and before we got MacStadium sponsorship.

You have to use VMWare Fusion on a standard Mac though, Apple have a strong-arm against VMWare, they force them to special-case macOS. VSphere like in MacStadium might be an option but I've never tried it.

[AshCripps]

Got VMware Fusion installed on one of the mac minis downloading on the other one right now but internet is being slow.

Was able to spin up a 10.14 vm succesfully but Im not able to get ahold of any earlier versions of macOS

[sam-github]

Next steps would be:
0. any manual setup documented in nodejs/build:ansible in the manual setup instructions

1. ansible the 10.14 machine
2. confirm it can do a nodejs build
3. modify ansible and/or manual setup docs if necessary/problems arose
4. connect it up to ci.nodejs.org

[sam-github]

@rvagg We're banging our heads against getting JumpHost configuration working through the host macmini into its guests. We'll keep working at it, but in case you have some advice for us, here is where we are at, @AshCripps and I will take another crack at it tomorrow.

Manual ~/.ssh/config setup is trival (ProxyJump admin@HOSTIP), that's good.

Not so much luck yet with ansible. It looks to me that ansible is getting into the host, but then trying to auth as the "admin" user into the guest OS.

I suspect that there are some assumptions about the user accounts that must exist on the macs. The manual setup is missing any information about what users to create, so I'm poking around macstadium to see how they are setup.

We have made an iojs user on the guests, but it looks like ansible is trying to login to the guest OS as the "admin" user. I'm not sure why yet, and macstadium uses the name "administrator"... mostly, the 10.12 machines don't have administrator, they have a "root1". We hit the end of the day, next step is to figure out what we need to setup on the guest VMs in terms of users, and what we need to config in ansible to mirror the macstadium config.

% pssh -i -h hosts/macos "dscl . list /Users | grep -v _|grep -v daemon|grep -v nobody; ls -l /Users|grep -v Shared|grep -v total"
[3] 10:37:13 [SUCCESS] test-macstadium-macos10.10-x64-1
administrator
iojs
root
drwxr-xr-x+ 36 administrator  staff  1224 23 Sep 07:17 administrator
drwxr-xr-x  26 iojs           iojs    884  8 Oct 07:21 iojs
[4] 10:37:13 [SUCCESS] test-macstadium-macos10.10-x64-2
administrator
iojs
root
drwxr-xr-x+ 17 administrator  staff   578 22 Jan  2018 administrator
drwxr-xr-x  33 iojs           iojs   1122 24 Oct 06:14 iojs
[5] 10:37:13 [SUCCESS] test-macstadium-macos10.11-x64-1
administrator
iojs
root
drwxr-xr-x+ 21 administrator  staff   714 24 Sep 04:13 administrator
drwxr-xr-x  33 iojs           iojs   1122 24 Oct 05:28 iojs
[6] 10:37:13 [SUCCESS] test-macstadium-macos10.11-x64-2
administrator
iojs
root
drwxr-xr-x+ 19 administrator  staff   646 14 Jun 12:29 administrator
drwxr-xr-x  31 iojs           iojs   1054 24 Oct 08:57 iojs
[7] 10:37:13 [SUCCESS] test-macstadium-macos10.12-x64-1
iojs
root
root1
drwxr-xr-x  22 iojs   iojs   748 22 Oct 04:08 iojs
drwxr-xr-x+ 18 root1  staff  612 23 Apr  2019 root1
[8] 10:37:13 [SUCCESS] test-macstadium-macos10.12-x64-2
iojs
root
root1
drwxr-xr-x  17 iojs   iojs   578  4 Jul 12:31 iojs
drwxr-xr-x+ 20 root1  staff  680 18 Jun 05:44 root1

[sam-github]

Can I safely run the jenkins/worker/create.yml against any of the current macs?

It would be helpful to have a sucessful run against a mac to compare the nearform setup to.

[mhdawson]

I'm not sure I understand the comment about the JumpBox. For the Nearform setup if we can get an additional IP for each of the virtual machines that will be much easier.

[sam-github]

OK, I'll contact nearform. If we take that approach we can't go any further with ansible until they allocate IPs.

[rvagg]

see ansible.cfg, the [hosts:requireio] should be instructive if you really need a jump host

[sam-github]

We found and copied that, no luck. I emailed nearform, if we get static IPs, its a problem that doesn't need solving. And I guess we can PR our WIP.

[sam-github]

If you or michael have a comment on whether I can ansible macstadium hosts, I'd appreciate it. There are machines that haven't been ansibled in ages, and its risky to run on them, and some that get ansibled regularly, so its safe. I don't know what category the macstadium fits in.

[rvagg]

@sam-github as far as I'm aware yes, I think I did 10.11's recently. I'm always hesitant with older versions because who knows what's broken in the time since they were last run but you're welcome to try. As far as I know there's nothing custom on them. To be safe, maybe not all at the same time.

[sam-github]

Discussed in nodejs/build WG meeting today -- @AshCripps needs an apple dev account to download older OS images, @mhdawson will give both @AshCripps and @sam-github access to the account credentials.

[sam-github]

Update: @AshCripps took this as far as possible, at this point we are waiting for @jasnell to organize with NearForm IT static IPs for the VMs in the macminis.

Until then, the effort to get modern OS X machines in CI (test or release) via the nearform macminis is blocked.

/cc @jasnell @MylesBorins

[jasnell]

Is this resolved now?

[sam-github]

Nope, @AshCripps is going to take it up again tomorrow, as it happens. We aren't blocked, we've the static IPs from your sysadmins.

[AshCripps]

@jasnell @nodejs/platform-macos I have updated the description with a checklist of what next for me to do. Bare in mind we only have space for 4 VMs so we have decided to go with all four as 10.15 (1 release, 3 test) as otherwise we wont have enough coverage as the absolute minimum is 1 release and 2 test of the same os level which leave one machine that can't be used for any other level. This can be changed if needed if more resource become available to the build WG.

@sam-github and I propose we use the 10.15 release machine to run all releases on all lines to ensure that notorization, when ready, can be run on all releases which cannot be done on the current release machines.

I have done some tests which shows the MACOS_DEPLOYMENT_TARGET set in common.gypi is pretty accurate and allowed for a binary to be run on a much earlier os level, but further tests to confirm this will need to be done.

[AshCripps]

Weve hit a snag with setting up the new machines and assigning them the IPs so I contacted nearforms tech and got this response.

Hi Ashley,

We found the issue, the subnet was changed which they failed to notify me. I had expected to use the same as what we had.
The is an issue however with the new subnet with how our network is configured so they have to make a change to that range I'm afraid. 
I will update on Monday regarding how long this change will take.

Thanks,
Eamonn 

So now awaiting his update on monday.

[AshCripps]

Current ETA on the network fix is Tuesday next week (28th Jan 2020)

[rvagg]

@AshCripps can we have an update on this? It would be good to get a new release machine up to at least start getting 13.x on newer macOS & Xcode with notarization. Ideally we could get two new release machines, one in nearForm and one in Macstadium and run all release lines through the same setup.

[sam-github]

Update in draft meeting minutes: https://docs.google.com/document/d/1l72fuEnKuy5uBCc3vRCj18tSko5heGP4skdYofdLmvk/edit

[jasnell]

I'll check with Eamonn again tomorrow to see where we're at but still appear to be blocked by the provider :-/

[AshCripps]

Yeah I emailed Eamonn yesterday as well but it has been blocked on the provider for many weeks now :(

[AshCripps]

Update:

Hi Ashley,

Good news, the IP's have been allocated, the network service team will be in on Monday 
which they said they would setup that on the firewall and add a small switch which is 
required due to how the network is setup onsite.

Thanks,
Eamonn

[AshCripps]

Finally some good news we have the IPs and I can confirm they work on the VMs.

Will hopefully have a draft PR in later today to kick off adding these machines to CI