VMR product validation tooling

[mmitche]

In order to productize the vertical build, we will need tooling to determine whether the outputs produced are what we expect. This generally means that shipping artifacts have the same file structures, are signed in the same ways, have expected version numbers, etc.

Develop tooling to help out with this verification.

Several questions need to be answered as part of this work:

• What kinds of assets (nugets, MSIs, rpms, debs, ...) are necessary to compare to get reasonable coverage
• What comparisons should be done (file names, metadata, etc.)?
• How will be the final tooling executed? (manually, automated runs in a pipeline)

T-Shirt Size: Medium

[jtschuster]

To validate the file structure, I'm planning to start by making an msbuild task (probably in arcade?) to compare nuget packages and find the diffs in file structure, file contents, and nuget metadata (versions, anything else important). From there we can maybe set up something to download the baseline packages from the official build feeds and compare all the outputs. We'll need another way to validate we are producing all the right packages, though. @mmitche do you see any issues with that or know if that's duplicating any work?

[mmitche]

That sounds about right to me. If integrating into an msbuild process is a pain, a simple console app (arcade seems like a good place for it) is also a good option.

[tkapin]

I've updated the description with a few questions that need to get answered as part of this effort. Also, @MichaelSimons mentioned some existing source-build tooling that might be leveraged as part of this effort.

For the record, I think the overall work necessary to develop tooling that would give us necessary confidence in the supported (asset type, platform) matrix will be higher than M (2 weeks).

[MichaelSimons]

Also, @MichaelSimons mentioned some existing source-build tooling that might be leveraged as part of this effort.

This is in reference to the following source-build smoke test - https://github.com/dotnet/dotnet/blob/main/test/Microsoft.DotNet.SourceBuild.SmokeTests/SdkContentTests.cs

The problem on the source-build side is that infrastructure is needed to flag differences between the source built and microsoft products. Differences can be intentional like the source built version doesn't contain windows specific components while other times a diff may be the result of a product bug like a new component was added for the microsoft build but source-build was overlooked. Once a difference is sign-off on, meaning it is an known expected difference, the infrastructure should no longer flag that difference.

[jtschuster]

I spoke with @agocke about nuget package validation and we came to the conclusion that it's simpler and more important to compare the final output tarball/zip of the installer/VMR builds to validate the VMR builds. I'll focus on that before returning to individual package validation.

[mthalman]

Reopening this. It was mistakenly closed from dotnet/installer#19222 which only contributes to this issue but not completely.