Windows CI machine updates

[joaocgreis]

@nodejs/build

I will change and rebuild some/most of the Windows machines we have connected to CI to add VS2019 to the matrix and better balance other versions for the current needs.

I will try to keep node-test-commit-windows-* working all the time, but expect jobs to queue for longer periods when the number of machines is reduced.

I will try to keep other jobs working as well, but please let me know if something is not working as expected.

[joaocgreis]

I'm planning to include the VS version in the host names, as metadata in the OS section (similar to arm_cross machines). This reinforces the restriction that only one VS version should be installed and makes the machines easier to manage, both because Ansible is made a bit simpler and we can know with a glace what version is installed.

Please let me know if there are any objections. (cc @rvagg)

[richardlau]

I will try to keep other jobs working as well, but please let me know if something is not working as expected.

@BethGriggs is seeing Windows CI failures on v10.x against releases that weren't seen at the time the releases were made (nodejs/node#30639 (comment)). Two Windows nodes have been marked offline in #2070.

[joaocgreis]

The new machines are in place and the Windows job has been updated. PR coming soon with build repo changes.

Resuming jobs before the change does not work, I expect this to be an issue in the next few days. I'll rebuild the jobs I see failing.

I've updated the libuv job, there are a few others that I still have to get to. Please let me know if there are issues in other jobs.

[joaocgreis]

Many compile jobs have failed in the last day. I noticed this happens after a V8 canary compile job runs, every compile job of the same type that runs after it fails. I'm still investigating.

For now, if you see a compile job failing with a lot of errors like

openssl.def : error LNK2001: unresolved external symbol ...

please take the machine that ran it offline.

[joyeecheung]

I think it might be related to nodejs/node#30647 (the CI was green but that was before VS 2019 was added to the CI), though from the build logs it might not be considering both node_use_node_code_cache and node_use_node_snapshot appeared to be true in the VS2019 builds. I'll try to see if reverting it makes a difference.

[joaocgreis]

The issue of failed compile jobs has been fixed by deleting all clcache data when compilation fails. If this starts happening again, we'll probably have to disable clcache completely, unless someone has the bandwidth to investigate further.

[richardlau]

@joaocgreis Please could you check if the libuv Windows failures are related?
e.g. https://ci.nodejs.org/job/libuv-test-commit-windows/1852/nodes=win2012r2-vs2015/console

09:45:35 C:\workspace\libuv-test-commit-windows\nodes\win2012r2-vs2015>FOR /F "delims=" %F IN ('grep -oP "^#define[[:space:]]*UV_VERSION_MAJOR[[:space:]]*\K[[:digit:]]*" include\uv\version.h') DO (SET "UV_VERSION_MAJOR=%F" ) 
09:45:36 grep: -P supports only unibyte and UTF-8 locales
09:45:36 
09:45:36 C:\workspace\libuv-test-commit-windows\nodes\win2012r2-vs2015>ECHO Detected UV_VERSION_MAJOR:  
09:45:36 Detected UV_VERSION_MAJOR: 
09:45:36 2 was unexpected at this time.
09:45:36 C:\workspace\libuv-test-commit-windows\nodes\win2012r2-vs2015>IF "vs2015" == "vs2013" IF  GEQ 2 (
09:45:36 Build step 'Execute Windows batch command' marked build as failure
09:45:36 Collecting metadata...
09:45:36 Metadata collection done.
09:45:36 Notifying upstream projects of job completion
09:45:36 Finished: FAILURE

The previous build passed (from 10 days ago).

cc FYI @nodejs/libuv

[joaocgreis]

This also happens on the ARM64 machines, which should not be affected by the updates. I believe this is related to a Git for Windows update that happened a few days ago on all machines, between the last good and the first failing job. I replaced grep -P by sed, should be working again.