Migrate off of Macstadium vsphere

[AshCripps]

In the vsphere we currently have 6 active machines - 2 Release (10.10 and 10.11) and 4 test (2 10.10 and 2 10.11).

The current issue is that Orka doesn't support anything older than 10.13 as shown below:

Ash,

Unfortunately, we don’t currently support any macOS older than High Sierra 10.13 on Orka.  Our team spent considerable time attempting to make older macOS releases work on Orka but encountered significant challenges whereupon it was determined that a lower-level rewrite that was in the works made more sense. 

This is called the, “Bring your own image” model, and we are actively developing it but it is not scheduled for release in the next sprint due May 11.  I presume not being able to support these older instances is a deal breaker with regard to bringing the entire build apparatus over to Orka? 

So we have two options:

1. Wait for the new model to be released in May and hopefully add the machines back that way.
2. Stop using the machines (asides from tests only the 10.10 machine is used to build release sources but only on 10.x)

Thought @nodejs/build? Im personally for moving off these old machines.

[rvagg]

I'm fine with either option.

We're using the old machines for testing and building source + header tarballs for 10.x. I don't believe breakage would be likely if we were to move the tarball creation to Linux, it would likely be an improvement if anything. Keeping them fixed was always just a matter of being cautious so I guess it depends on how cautious we want to be.

[richardlau]

I think migrating off the release machine (I don't think that the macOS 10.11 release machine is used now?) should be low risk.

Losing the test machines would be unfortunate. Then again we've never tested as far back as we claim to support (i.e. macOS 10.7 for Node.js 10.x).

[AshCripps]

with #2347 merged im going to wait a bit incase anything breaks and the will start the process of decomissioning the machines in vsphere

[AshCripps]

To Do List:

• Move off osx10.10 for 10.x release headers (jenkins: move 10.x release sources off osx #2347)
• Remove osx10.11 labels from jobs
• Wait for a 12.x release to ensure nothing goes amiss
• Disable and remove osx10.11 from jenkins
• Remove osx10.10 labels from jobs
• Wait for a 10.x release to ensure nothing goes amiss
• Disable and remove osx10.10 from jenkins
• Contact macstadium to begin decommissing vsphere cluster

[AshCripps]

Think ive removed all the labels for osx10.11, will check the backup repo later to see if ive missed any

[AshCripps]

Theres a 12 release that should go out tomorrow after which I can move on to the next steps (provided nothing breaks)

[AshCripps]

weve been given a kick from macstadium

in reviewing your account, I see that you are using a legacy Xserve product that has been depreciated across our markets. The Xserve is a work horse and we understand that it may be difficult to let it go, but it has been discontinued by Apple very difficult for us to support any longer. To that end, we need to migrate the contents of your Xserve to other, compatible hardware by July 31, 2020.  

With that in mind I plan to start removing the 10.10 labels from jobs and removing the 10.11 machines from CI today.
I want to do this ASAP before the next 10.x release to ensure we dont break anything huge (we shouldnt really these machines are long out of support so we have no obligation to keep them around).

[AshCripps]

So think ive removed all labels of 1010 and 1011 from jenkins from at least the important jobs. Will wait for the jenkins-config repos to update before having another pass through tomorrow then I can start deleting the 10.11 machines in jenkins. Will wait on deleting the 10.10 machines, I also need to edit the version selector script for both.

[richardlau]

I have moved https://ci.nodejs.org/job/nodereport-continuous-integration-latest/ over from osx1010 and osx1012 to osx1014 and osx1015.

[AshCripps]

As part of this im going to delete this job - https://ci.nodejs.org/job/node-test-commit-osx-macstadium/

[AshCripps]

I have deleted both 10.11 machines from jenkins

[rvagg]

Did you check the labels were removed from all the jobs they're referenced in before you removed the machines? if you don't do that then the can't be deleted from the UI and have to be done by editing the XML. I normally grep all the XML files to find where those labels are and go remove them manually from each job. If the labels aren't present on active machines then the UI won't show them anywhere but they'll linger in the configs.

[AshCripps]

@rvagg yes the only mentions of 10.11 left are descriptions not label strings. I was using the jenkins config repo to grep for them but I will double check.

[AshCripps]

Ok on the double check they are in two jobs scheduled for deletion (chakaracore osx and libuv osx (libuv/libuv#2903) and a maclover7 job I couldnt find in jenkins anyway

[richardlau]

Ok on the double check they are in two jobs scheduled for deletion (chakaracore osx and libuv osx (libuv/libuv#2903) and a maclover7 job I couldnt find in jenkins anyway

The scripts that populate the jenkins-config repos currently don't remove configs for deleted jobs (#2133 (comment)). I haven't had time to look into fixing that.

[AshCripps]

went through again and removed any mention of osx1010 and deleted the old osx-macstadium job. On monday ill have another check through the backup repo and double check everythings gone, then Ill remove the machines from jenkins and contact macstadium to remove them.

so last call to @nodejs/build is there anything worth saving on the machines hosted in the macstadium vsphere (osx10.10, 10.11 and 10.12) before they are gone forever?

[AshCripps]

Any reference Ive found in the jenkins config test repo is gone except for one osx1010 reference in citgm-smoker-private which I cant seem to see? but from looking at it its in a string so should be able to be removed from the job when the label is gone anyway. Im going to now remove the osx1010 1011 and 1012 machines from both CIs if found and send the email on to macstadium.

[rvagg]

removed from citgm-smoker-private, it wasn't a label so no big deal. no osx1011 on live configs

in ci-release there's some osx1010 and osx1011 references but they're all in test clone jobs that Michael uses for testing, so I'm not going to worry about them, the jobs themselves will probably be overwritten or deleted at some point anyway.

[AshCripps]

@rvagg thanks for that!

ive deleted the agents from CI and CI-release and sent an email to macstadium - will close the issue once I have confirmation there is nothing more for us to do.