Enable running host tests using helix

[elinor-fung]

Host tests (src/installer/tests) currently run directly on the build machine.

• CI failure investigation is different from that for libraries/runtime tests (which use Helix)
  • Results/artifacts aren't associated in the same way - accessing in AzureDevOps or querying in Kusto is inconsistent with rest of repo
• Tests are run on a limited set of platforms
  • No arm64 coverage - only x64 and x86

Test projects

• AppHost.Bundle.Tests
• HostActivation.Tests
• Microsoft.NET.HostModel
  • Microsoft.NET.HostModel.AppHost.Tests
  • Microsoft.NET.HostModel.Bundle.Tests
  • Microsoft.NET.HostModel.ComHost.Tests
• Microsoft.DotNet.CoreSetup.Packaging.Tests
  • Not actually hosting tests - inspects built NuGet packages
  • Could probably use some investigation into whether these are still valuable, should be done a different way, etc.

Work

• Audit/reduce amount of host tests/assets that would need to be added to a helix workload
  • Remove SharedFxLookup tests #77858
  • Switch LightupAppActivation tests to component tests targeting additional deps #78139
  • Delete host test targeting old xunit packages #79177
  • Collapse/remove projects under src/installer/tests/Assets/TestProjects #77807
  • Move startup hook tests targeting StartupHookProvider out of hosting tests #77805
  • Remove testing of both build and publish outputs
    • Various tests run both build and publish outputs - shouldn't really be interesting to target both for host testing.
• Remove building of test projects as part of test run
  • Tests don't actually care about the build itself - they just want something built and redirected to use the live build.
    • Currently relies on GenerateTestRestoreSourcesNuGetConfig to point at live built packages
  • Build all test projects before the test run and copy them as test assets.
  • Delete TestProjectFixture and associated helpers #99847
  • Remove requirement for building packs to run host tests and infrastructure around restoring test project assets #100324
• Remove assumption in tests that they are run from within the runtime repo
  • RepoDirectoriesProvider builds paths based on repo structure / output directory
    • Also currently reads settings from a file created by SetupTestContextVariables
• Construct layout for whatever assets are necessary to carry with the test (separate from the runtime repo)
  • RefreshProjectTestAssets currently sets up a sharedFrameworkPublish folder and test projects that will be run
  • Maybe add a host.pretest subset, similar to libs.pretest
• Ensure tests can run successfully on other platforms
  • Likely don't run properly on arm64 right now
  • Fix HostApiInvokerApp test app setting DllImportResolver for hostfxr #108954
• Add support for creating/sending Helix payload/job for host tests
  • Equivalent of src/tests/Common/helixpublishwitharcade.proj and src/libraries/sendtohelix.proj, but for host tests
  • Add project to create helix work items for host tests #108529
• Update pipeline to send tests to Helix
  • eng/pipelines/installer/jobs/build-job.yml

[ghost]

Tagging subscribers to this area: @vitek-karas, @agocke, @VSadov
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Host tests (src/installer/tests) currently run directly on the build machine.

• CI failure investigation is different from that for libraries/runtime tests (which use Helix)
  • Results/artifacts aren't associated in the same way - accessing in AzureDevOps or querying in Kusto is inconsistent with rest of repo
• Tests are run on a limited set of platforms
  • No arm64 coverage - only x64 and x86

Test projects

• AppHost.Bundle.Tests
• HostActivation.Tests
• Microsoft.NET.HostModel
  • Microsoft.NET.HostModel.AppHost.Tests
  • Microsoft.NET.HostModel.Bundle.Tests
  • Microsoft.NET.HostModel.ComHost.Tests
• Microsoft.DotNet.CoreSetup.Packaging.Tests
  • Not actually hosting tests - inspects built NuGet packages
  • Could probably use some investigation into whether these are still valuable, should be done a different way, etc.

Work

• Audit/reduce amount of host tests/assets that would need to be added to the workload
  • Many projects in src/installer/tests/Assets/TestProjects only have minor differences - multiple projects can become one with different behaviour based on command line arguments
    • Likely candidates: PortableApp*, RuntimeProperties, AppWith*, LightupApp
  • Various tests run both build and publish outputs - shouldn't really be interesting to target both for host testing.
  • Many StartupHook tests (and test projects) are actually targeting StartupHookProvider, not the hosting side of it. They can be moved to coreclr and just call into StartupHookProvider without launching a bunch of processes to actually run an app.
• Remove building of test projects as part of test run
  • Tests don't actually care about the build itself - they just want something built and redirected to use the live build.
    • Currently relies on GenerateTestRestoreSourcesNuGetConfig to point at live built packages
  • Build all test projects before the test run and copy them as test assets.
• Remove assumption in tests that they are run from within the runtime repo
  • RepoDirectoriesProvider builds paths based on repo structure / output directory
    • Also currently reads settings from a file created by SetupTestContextVariables
  • Construct layout for whatever assets are necessary to carry with the test (separate from the runtime repo)
    • RefreshProjectTestAssets currently sets up a sharedFrameworkPublish folder and test projects that will be run
    • Maybe add a host.pretest subset, similar to libs.pretest
• Ensure tests can run successfully on other platforms
  • Likely don't run properly on arm64 right now
• Add support for creating/sending Helix payload/job for host tests
  • Equivalent of src/tests/Common/helixpublishwitharcade.proj and src/libraries/sendtohelix.proj, but for host tests
• Update pipeline to send tests to Helix
  • eng/pipelines/installer/jobs/build-job.yml

Author:	elinor-fung
Assignees:	-
Labels:	area-Host
Milestone:	-

[elinor-fung]

I don't anticipate us actually doing all this in 8.0, but putting this in that milestone for now.