Fix the built-in FSMonitor, and run Scalar's Functional Tests as part of the automated builds

[dscho]

Without Scalar's Functional Tests, I would not have caught two bugs causing segmentation faults in the built-in FSMonitor (which I have fixed in time for Git for Windows v2.32.0) nor the bug where the FSMonitor just hangs (which I unfortunately did not catch in time for Git for Windows v2.32.0, but at least we can fix it in v2.32.0.vfs.*).

However, running the Functional Tests locally, I could not reproduce the bugs! The hang simply did not happen here, and the segmentation faults were only triggered when Trace2 was enabled (which I had not done locally). Therefore, the (epic) bug hunt really boiled down to running Scalar's Functional Tests using GitHub Actions, over and over and over again.

It was a bit cumbersome to run the tests using GitHub Actions, though, because the original setup required a successful run of the Azure Pipeline build git installers to build full installers for a given microsoft/git branch (which would have been a PR branch such as refs/pull/348/head), and then a PR branch of microsoft/scalar to point to that successful build. Not only does that take 30-45 minutes simply to build the installers, but there are three manual steps involved, each gated on the success of the previous step.

This PR adds a new GitHub workflow that makes all of that testing a lot nicer: just push a PR branch of microsoft/git and Scalar's Functional Tests will be built and run as part of the automated builds.

I used this very workflow to investigate the remaining problems of vfs-2.32.0 that were demonstrated by Scalar's Functional Tests:

• I first ran it with vfs-2.32.0 (plus this workflow), then reverted the built-in FSMonitor v1 -> v2 changes and ran it again, to verify that the problems were due to this upgrade (as opposed to independent changes coming in via v2.31.1 -> v2.32.0).
• Then, I added the built-in FSMonitor v1 -> v2 changes back, incrementally, to see where things break.
• To use my time efficiently, I pushed these incremental changes without waiting for the Functional Tests to finish.
• That way, two days and 55 runs later (each of which took between 25-60 minutes), I could pinpoint the exact change that caused the regression.

The result is the trio of fixup! fsmonitor--daemon: use a cookie file to sync with file system patches, which fixes the hang in Scalar's Functional Tests.

While at it, I also realized that the two fixups to built-in FSMonitor v2 that I integrated into Git for Windows v2.32.0 were actually only work-arounds, and in this PR I also provide a fix for the root cause (fixup! fsmonitor: introduce core.useBuiltinFSMonitor to call the daemon via IPC, which ensures that fsmonitor_ipc__send_query() is never called with a since_token that is NULL).

[dscho]

Let's see whether I can accelerate the Windows job in Scalar's Functional Tests by using a "more minimal" Git for Windows SDK. I'm on it.

[derrickstolee]

Thanks for bringing in the Scalar functional tests! This will make our workflow so much simpler.

I restarted your recent builds because it had the connection error from the Azure Repos side, not a real failure on our side.

[derrickstolee]

Making a note for later: After we remove the product code from Scalar, we can relax this condition.

[dscho]

Good point!

[dscho]

I restarted your recent builds because it had the connection error from the Azure Repos side, not a real failure on our side.

Thank you!

I ran into this twice during my debugging session; This is a pretty decent false positive rate, so I am not super worried. But if it persists, we might have to go with the plan @jeffhostetler laid out for dealing with 503s or other network flakiness.