HttpStress and SslStress builds are broken

[antonfirsov]

HTTP and SSL stress builds are currently broken on Windows:

https://dev.azure.com/dnceng/public/_build/results?buildId=1434301&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=8ca65746-3bfa-57bf-9141-d805b5d39db5

https://dev.azure.com/dnceng/public/_build/results?buildId=1434299&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=ac0b0b0f-051f-52f7-8fb3-a7e384b0dde9

To run stress tests with the current bits on main, our dockerfiles overwrite the 6.0 runtime preinstalled to the image with the locally built 7.0.0 testhost runtime:

runtime/eng/docker/libraries-sdk.windows.Dockerfile

Lines 17 to 19 in ce4dd5f

	COPY `
	$TESTHOST_LOCATION\$TFM-$OS-$CONFIGURATION-$ARCH\shared\$COREFX_SHARED_FRAMEWORK_NAME\$SOURCE_COREFX_VERSION\ `
	$TARGET_SHARED_FRAMEWORK\$COREFX_SHARED_FRAMEWORK_NAME\$TARGET_COREFX_VERSION\

This typically works, but now it looks like it is problematic in some cases. Note that for some reason this is only broken on Windows now.

To overcome this issue, I tried to install the 7.0 nightly SDK with dotnet-install.ps1 and replace the 7.0 runtime bits with the locally built ones as recommended by @ManickaP in her #60364 (comment), but unfortunately that leads to another build-time dll resolution error:

C:\Program Files\dotnet\sdk\7.0.100-alpha.1.21518.14\NuGet.targets(130,5): error : Unable to load the service index for source https://api.nuget.org/v3/index.json. [C:\app\Htt
pStress.sln]
C:\Program Files\dotnet\sdk\7.0.100-alpha.1.21518.14\NuGet.targets(130,5): error :   Could not load file or assembly 'System.Diagnostics.DiagnosticSource, Version=7.0.0.0, Cul
ture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified. [C:\app\HttpStress.sln]

I think the best way to overcome this is problem would be to separate building and execution of the stress project. For the former we can use the pre-installed SDK. For the latter we can utilize testhost the same way we do in normal functional tests, however I'm not an expert on SDK internals, so I don't know how to achieve the separation, the way dotnet test works in the runtime repo is a black box to me.

@ViktorHofer @safern @Anipik any tips or insights that can help us? (Either solving the dll resolution issues or using testhost runtime for executing the stress tests without touching the installed SDK.)

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

HTTP and SSL stress builds are currently broken on Windows:

https://dev.azure.com/dnceng/public/_build/results?buildId=1434301&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=8ca65746-3bfa-57bf-9141-d805b5d39db5

https://dev.azure.com/dnceng/public/_build/results?buildId=1434299&view=logs&j=2d2b3007-3c5c-5840-9bb0-2b1ea49925f3&t=ac0b0b0f-051f-52f7-8fb3-a7e384b0dde9

To run stress tests with the current bits on main, our dockerfiles overwrite the 6.0 runtime preinstalled to the image with the locally built 7.0.0 testhost runtime:

runtime/eng/docker/libraries-sdk.windows.Dockerfile

Lines 17 to 19 in ce4dd5f

	COPY `
	$TESTHOST_LOCATION\$TFM-$OS-$CONFIGURATION-$ARCH\shared\$COREFX_SHARED_FRAMEWORK_NAME\$SOURCE_COREFX_VERSION\ `
	$TARGET_SHARED_FRAMEWORK\$COREFX_SHARED_FRAMEWORK_NAME\$TARGET_COREFX_VERSION\

This typically works, but now it looks like it is problematic in some cases. Note that for some reason this is only broken on Windows now.

To overcome this issue, I tried to install the 7.0 nightly SDK with dotnet-install.ps1 and replace the 7.0 runtime bits with the locally built ones as recommended by @ManickaP in her #60364 (comment), but unfortunately that leads to another build-time dll resolution error:

C:\Program Files\dotnet\sdk\7.0.100-alpha.1.21518.14\NuGet.targets(130,5): error : Unable to load the service index for source https://api.nuget.org/v3/index.json. [C:\app\Htt
pStress.sln]
C:\Program Files\dotnet\sdk\7.0.100-alpha.1.21518.14\NuGet.targets(130,5): error :   Could not load file or assembly 'System.Diagnostics.DiagnosticSource, Version=7.0.0.0, Cul
ture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified. [C:\app\HttpStress.sln]

I think the best way to overcome this is problem would be to separate building and execution of the stress project. For the former we can use the pre-installed SDK. For the latter we can utilize testhost the same way we do in normal functional tests, however I'm not an expert on SDK internals, so I don't know how to achieve the separation, the way dotnet test works in the runtime repo is a black box to me.

@ViktorHofer @safern @Anipik any tips or insights that can help us? (Either solving the dll resolution issues or using testhost runtime for executing the stress tests without touching the installed SDK.)

Author:	antonfirsov
Assignees:	-
Labels:	area-System.Net.Http
Milestone:	-

[ManickaP]

Linux is broken as well, it just doesn't fail the build since it builds within the docker image:
https://dev.azure.com/dnceng/public/_build/results?buildId=1434299&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=6c56d998-158d-5232-283c-5140104799fc&l=50

[ViktorHofer]

the way dotnet test works in the runtime repo is a black box to me.

dotnet test respects a runsettings file which is automatically generated when a test library is built. That runsettings file hardcodes the path to the dotnet host to use:

runtime/eng/testing/.runsettings

Line 20 in a7bdb16

<DotNetHostPath>$$DOTNETHOSTPATH$$</DotNetHostPath>

. $$DOTNETHOSTPATH$$ is replaced with the path to the testhost folder, so it's just an invocation of the dotnet host under the testhost folder.

To run stress tests with the current bits on main, our dockerfiles overwrite the 6.0 runtime preinstalled to the image with the locally built 7.0.0 testhost runtime:

That's recipe for failure as the assumption that a 6.0 SDK is compatible with a 7.0 testhost or that a 7.0 live built testhost is compatible with a 7.0 SDK in some cases is wrong. Why can't you just use the live built testhost folder to invoke the stress tests?

think the best way to overcome this is problem would be to separate building and execution of the stress project. For the former we can use the pre-installed SDK.

Why not just follow the same approach as the other libraries tests and use the live built targeting pack to build the tests against and the testhost to run them?

[antonfirsov]

Why can't you just use the live built testhost folder to invoke the stress tests?

Because we used the 6.0 SDK to build them which can only target net6.0 TFM. But maybe if used 7.0 (daily) SDK to build and testhost to run, that would work.

live built targeting pack

This is a term I'm unfamiliar with. Is this the thing used to build a project against net7.0 TFM? Is there some guidance on how to use it with a project that is not integrated with the dotnet/runtime infrastructure?

[geoffkizer]

Why not just follow the same approach as the other libraries tests and use the live built targeting pack to build the tests against and the testhost to run them?

I agree, this would be the ideal approach if we could make it work.

I think part of the complication here is that these apps require ASP.NET. Maybe that's no longer a concern?

[ViktorHofer]

I think part of the complication here is that these apps require ASP.NET. Maybe that's no longer a concern?

I see, I missed that requirement. Until aspnetcore is an optional workload that can be bootstrapped, you probably need to continue using a matching pre-built SDK.

[geoffkizer]

Perhaps we should consider not using ASP.NET here. We don't use it for functional tests.

[ViktorHofer]

Perhaps we should consider not using ASP.NET here. We don't use it for functional tests.

That would probably be easiest. If aspnetcore would be a strict dependency I would probably suggest to use a pre-built SDK that supports aspnetcore but then use live targeting pack so that you can compile against the live built net7.0 assemblies. For running, if you would build the apps as self-contained, you could consume the live built runtime pack and would remove the dependency on the SDK entirely.

[antonfirsov]

Perhaps we should consider not using ASP.NET here. We don't use it for functional tests.

If aspnetcore would be a strict dependency

It is a strict dependency. Getting rid of aspnet would mean implementing and maintaining a test server that covers all stress cases, or to move the test server logic to a separate project and containerize it separately. Both seem to be overly expensive options to me.

use a pre-built SDK that supports aspnetcore but then use live targeting pack so that you can compile against the live built net7.0 assemblies

I think this is the direction we need to go, but I could use some guidance or at least some starting points on using pre-built SDK to build against local, live built dll-s. HttpStress is a standalone project today, not integrated with the dotnet/runtime infra.

[ViktorHofer]

I think this is the direction we need to go, but I could use some guidance or at least some starting points on using pre-built SDK to build against local, live built dll-s. HttpStress is a standalone project today, not integrated with the dotnet/runtime infra.

https://github.com/dotnet/runtime/blob/main/eng/targetingpacks.targets is everything that you need. That file needs to be imported and the required properties that need to be set are documented in it as well.

You would set these to:

- ProductVersion: 7.0.0
  - NetCoreAppCurrent: net7.0
  - NetCoreAppCurrentVersion: 7.0
  - MicrosoftNetCoreAppFrameworkName: Microsoft.NETCore.App
  - MicrosoftNetCoreAppRefPackDir: <path-to-runtime-repo>/artifacts/bin/microsoft.netcore.app.ref/
  - optional: MicrosoftNetCoreAppRuntimePackDir: <path-to-runtime-repo>/artifacts/bin/microsoft.netcore.app.runtime.<package-rid>/<LibrariesConfiguration>/ (ie package-rid: win-x64 on Windows, <LibrariesConfiguration>: Debug/Release)