Execute the Scenario-Tests in CI

[MichaelSimons]

The scenarios tests should be run as part of both the source-build and unified build CI legs. The tests should run with the SDK and runtimes produced by the build

[ViktorHofer]

I think it would make sense to hook the build and test of the scenario-tests repo onto the --test action in the build script. There's also the DotNetBuildTests property which might make sense to honor. cc @mmitche

[mmitche]

DotNetBuildTests is unrelated. It generally refers to repo level unit tests. I think scenario tests are somewhat different? I would still want to be able to run them against an official build, without having to build all tests in an official build.

Otherwise using --test sounds reasonable as a first approach. I think if we end up with executable unit tests in the VMR that may need to be re-visited.

[mthalman]

How do we want this to work in terms of building the tests and running the tests? The runtime repo allows for build/test to be one action or to only test w/o a build. Do we want similar behavior here? In that case, should there be a --test-no-build option added?

[ViktorHofer]

Yes, I think it makes sense to support --test-no-build / /p:TestNoBuild. I originally added that to runtime as the vstest msbuild integration had it as well.

[mmitche]

I think things may be more complicated. If I pass --test, what does it test? The scenario tests test an installed product. The build produces more than one installable product (e.g. an msi and a zip)

[mthalman]

In the interest of not horking up machine state, I would suggest only testing from an artifact file (zip/tar.gz), not an installer (e.g. msi). At least that would be true for dev scenarios. For CI, we may need to test them all using additional options.

[mmitche]

Alright, so it will extract the output layout and execute against that? I suggest that we include a separate option where we can execute already built tests against an installed product. The infra would not be responsible for installing that product. It would just receive the location of the install.

[mthalman]

Alright, so it will extract the output layout and execute against that? I suggest that we include a separate option where we can execute already built tests against an installed product. The infra would not be responsible for installing that product. It would just receive the location of the install.

Isn't that outside the scope of the VMR? That sounds like a scenario where you would just do that directly with the scenario-tests repo.

[mmitche]

Alright, so it will extract the output layout and execute against that? I suggest that we include a separate option where we can execute already built tests against an installed product. The infra would not be responsible for installing that product. It would just receive the location of the install.

Isn't that outside the scope of the VMR? That sounds like a scenario where you would just do that directly with the scenario-tests repo.

Probably. Or at least there's no absolute need to provide that infra at the top level,

[MichaelSimons]

@mthalman - Can this be closed now? I believe you opened issues for the follow-ups but the main work is done.

[mthalman]

Fixed by dotnet/installer#19222.