Enable scenario test execution in VMR build

[mthalman]

Fixes dotnet/source-build#3819
Fixes dotnet/source-build#4278

This enables running the scenario tests via the build scripts.

There will be more follow up work to better integrate the SB smoke tests with the scenario tests as there is some duplication going on there. But for source build scenarios, both test suites are executed.

Both the SB and UB jobs are configured to enable the running of these tests.

It looks like the unified-build has some issues with F# in the OSX jobs that will need to be looked at later.

Test runs (internal links):

• unified-build
• source-build

[ViktorHofer]

Can we please wait until #19090 is merged? That's pretty close and it will cause a conflict.

[MichaelSimons]

I was playing around with the dev UX a bit and was surprised at the overhead prior to when it appears to actually build the test projects in a VMR that was previously built. Looking through the output it looks like it is creating the package archives every time I run ./build.sh --test -sb. It doesn't seem right that this logic is running in this scenario.

  Packaged all symbols in '/workspaces/dotnet/artifacts/assets/Release/dotnet-symbols-all-9.0.100-preview.4.24202.1-fedora.39-x64.tar.gz'
  Packaged sdk symbols in '/workspaces/dotnet/artifacts/assets/Release/dotnet-symbols-sdk-9.0.100-preview.4.24202.1-fedora.39-x64.tar.gz'
  Found 0 files in prebuilt packages dir.
  Shipping packages are located in '/workspaces/dotnet/artifacts/packages/Release/Shipping/'.
  Shipping assets are located in '/workspaces/dotnet/artifacts/assets/Release/'.

[mthalman]

I was playing around with the dev UX a bit and was surprised at the overhead prior to when it appears to actually build the test projects in a VMR that was previously built. Looking through the output it looks like it is creating the package archives every time I run ./build.sh --test -sb. It doesn't seem right that this logic is running in this scenario.

  Packaged all symbols in '/workspaces/dotnet/artifacts/assets/Release/dotnet-symbols-all-9.0.100-preview.4.24202.1-fedora.39-x64.tar.gz'
  Packaged sdk symbols in '/workspaces/dotnet/artifacts/assets/Release/dotnet-symbols-sdk-9.0.100-preview.4.24202.1-fedora.39-x64.tar.gz'
  Found 0 files in prebuilt packages dir.
  Shipping packages are located in '/workspaces/dotnet/artifacts/packages/Release/Shipping/'.
  Shipping assets are located in '/workspaces/dotnet/artifacts/assets/Release/'.

This seems outside the context of testing. Does this happen when you don't use the --test option? If so, that's a separate issue.

[MichaelSimons]

I was playing around with the dev UX a bit and was surprised at the overhead prior to when it appears to actually build the test projects in a VMR that was previously built. Looking through the output it looks like it is creating the package archives every time I run ./build.sh --test -sb. It doesn't seem right that this logic is running in this scenario.

  Packaged all symbols in '/workspaces/dotnet/artifacts/assets/Release/dotnet-symbols-all-9.0.100-preview.4.24202.1-fedora.39-x64.tar.gz'
  Packaged sdk symbols in '/workspaces/dotnet/artifacts/assets/Release/dotnet-symbols-sdk-9.0.100-preview.4.24202.1-fedora.39-x64.tar.gz'
  Found 0 files in prebuilt packages dir.
  Shipping packages are located in '/workspaces/dotnet/artifacts/packages/Release/Shipping/'.
  Shipping assets are located in '/workspaces/dotnet/artifacts/assets/Release/'.

This seems outside the context of testing. Does this happen when you don't use the --test option? If so, that's a separate issue.

It seems there are two issues here. One, why is build even happening - is there not a way to avoid rebuilding? This feels related to arcade not being used. w/arcade --build is separate from --test so you have the ability to do both together or independently. Two, the symbols should not be repackaged when the build outputs haven't changed. I agree the later is outside the scope here.

[mthalman]

One, why is build even happening - is there not a way to avoid rebuilding?

That's what the --testnobuild option is for.

[ViktorHofer]

Just for my own understanding, why did you need to add this property?

[ViktorHofer]

Please file a tracking issue for that and link to it here.

[mthalman]

dotnet/source-build#4364

[ViktorHofer]

Looks great now. I went ahead and deleted src/SourceBuild/content/test/scenario-tests.proj which was a leftover to avoid others reviewing the file.

IMO this is ready to be merged after you addressed my remaining minor feedback and filed an issue to track running scenario-tests in non-official builds.

Thanks for your continued effort on this.

[mthalman]

A new error is showing up from the Aspire test. This isn't specific to my changes. I can repro it using the latest built SDK from the unified build pipeline. This seems specific to the unified build though. I can't repro it using an SDK built from the official installer pipeline. This error also doesn't occur in the source build pipeline from this PR.

Failed to instatiate template '.NET Aspire Application', the following constraints are not met:
  workload: The constraint 'workload' failed to initialize: Manifest provider Microsoft.NET.Sdk.WorkloadManifestReader.SdkDirectoryWorkloadManifestProvider returned a duplicate manifest ID 'microsoft.net.sdk.android' [/vmr/artifacts/obj/extracted-dotnet-sdk/sdk-manifests/9.0.100-preview.1/microsoft.net.sdk.android/34.99.0-preview.1.151/WorkloadManifest.json] that conflicts with existing manifest [/vmr/artifacts/obj/extracted-dotnet-sdk/sdk-manifests/9.0.100-preview.1/microsoft.net.sdk.android/34.99.0-preview.1.151/WorkloadManifest.json]
    

@Forgind - It looks like you work in the workload space. Can you help point where to look in diagnosing this error?

Build link

[mthalman]

A new error is showing up from the Aspire test. This isn't specific to my changes. I can repro it using the latest built SDK from the unified build pipeline. This seems specific to the unified build though. I can't repro it using an SDK built from the official installer pipeline. This error also doesn't occur in the source build pipeline from this PR.

Failed to instatiate template '.NET Aspire Application', the following constraints are not met:
  workload: The constraint 'workload' failed to initialize: Manifest provider Microsoft.NET.Sdk.WorkloadManifestReader.SdkDirectoryWorkloadManifestProvider returned a duplicate manifest ID 'microsoft.net.sdk.android' [/vmr/artifacts/obj/extracted-dotnet-sdk/sdk-manifests/9.0.100-preview.1/microsoft.net.sdk.android/34.99.0-preview.1.151/WorkloadManifest.json] that conflicts with existing manifest [/vmr/artifacts/obj/extracted-dotnet-sdk/sdk-manifests/9.0.100-preview.1/microsoft.net.sdk.android/34.99.0-preview.1.151/WorkloadManifest.json]
    

@Forgind - It looks like you work in the workload space. Can you help point where to look in diagnosing this error?

Build link

Now the latest build shows unified build passing but source build is failing with a similar error, but referring to a different package:

Failed to instatiate template '.NET Aspire Application', the following constraints are not met:
  workload: The constraint 'workload' failed to initialize: Manifest provider Microsoft.NET.Sdk.WorkloadManifestReader.SdkDirectoryWorkloadManifestProvider returned a duplicate manifest ID 'microsoft.net.workload.emscripten.net7' [/vmr/artifacts/obj/extracted-dotnet-sdk/sdk-manifests/9.0.100-preview.4/microsoft.net.workload.emscripten.net7/9.0.0-preview.4.24215.3/WorkloadManifest.json] that conflicts with existing manifest [/vmr/artifacts/obj/extracted-dotnet-sdk/sdk-manifests/9.0.100-preview.4/microsoft.net.workload.emscripten.net7/9.0.0-preview.4.24215.3/WorkloadManifest.json]

Build link

That would seem to imply the issue is intermittent.

[Forgind]

A new error is showing up from the Aspire test. This isn't specific to my changes. I can repro it using the latest built SDK from the unified build pipeline. This seems specific to the unified build though. I can't repro it using an SDK built from the official installer pipeline. This error also doesn't occur in the source build pipeline from this PR.

Failed to instatiate template '.NET Aspire Application', the following constraints are not met:
  workload: The constraint 'workload' failed to initialize: Manifest provider Microsoft.NET.Sdk.WorkloadManifestReader.SdkDirectoryWorkloadManifestProvider returned a duplicate manifest ID 'microsoft.net.sdk.android' [/vmr/artifacts/obj/extracted-dotnet-sdk/sdk-manifests/9.0.100-preview.1/microsoft.net.sdk.android/34.99.0-preview.1.151/WorkloadManifest.json] that conflicts with existing manifest [/vmr/artifacts/obj/extracted-dotnet-sdk/sdk-manifests/9.0.100-preview.1/microsoft.net.sdk.android/34.99.0-preview.1.151/WorkloadManifest.json]
    

@Forgind - It looks like you work in the workload space. Can you help point where to look in diagnosing this error?
Build link

Now the latest build shows unified build passing but source build is failing with a similar error, but referring to a different package:

Failed to instatiate template '.NET Aspire Application', the following constraints are not met:
  workload: The constraint 'workload' failed to initialize: Manifest provider Microsoft.NET.Sdk.WorkloadManifestReader.SdkDirectoryWorkloadManifestProvider returned a duplicate manifest ID 'microsoft.net.workload.emscripten.net7' [/vmr/artifacts/obj/extracted-dotnet-sdk/sdk-manifests/9.0.100-preview.4/microsoft.net.workload.emscripten.net7/9.0.0-preview.4.24215.3/WorkloadManifest.json] that conflicts with existing manifest [/vmr/artifacts/obj/extracted-dotnet-sdk/sdk-manifests/9.0.100-preview.4/microsoft.net.workload.emscripten.net7/9.0.0-preview.4.24215.3/WorkloadManifest.json]

Build link

That would seem to imply the issue is intermittent.

That's interesting. The workload resolver supports there being multiple workload set files. If there are, it reads them all in and composes them, erroring if there are any duplicate entries because we clearly can't have two different versions of a single workload at the same time.

It appears that in this case, we're loading the same workload set twice, which means that there are definitely going to be duplicates—in fact, the whole file should be duplicates. So I see two bugs here: first, we should accept duplicates as long as they're the exact same version (basically drop the error in this case). Second, we shouldn't be loading the same manifest twice.

The first is an easy change. The second will take more investigation, as I don't know why we're trying to load the same manifest twice...glancing very quickly through the code, I don't see a way that's possible, but I didn't look exhaustively. Perhaps this has something to do with the overlay resolver? Perhaps something external is using reflection to call LoadManifestsFromProvider without updating _initializedManifests?