Microsoft.NET.Build.Containers.IntegrationTests are pulling from Docker Hub

[akoeplinger]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=733130
Build error leg or test failing: Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.WriteToPrivateBasicRegistry
Pull request: https://github.com/dotnet/sdk.git/pull/42019

We need to stop pulling from upstream Docker Hub since we're hitting rate limits.

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "You have reached your pull rate limit",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=733130
Error message validated: [You have reached your pull rate limit]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/8/2024 7:20:41 PM UTC

Report

Build	Definition	Test	Pull Request
900898	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45599
900242	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45463
900231	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45522
900228	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution	#45435
900187	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#44461
899373	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution	#45559
897513	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45492
897504	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44611
895350	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45458
894962	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45419
894838	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45097
894363	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution	#45427
893558	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution	#45327
892151	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution	#45392
893220	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45293
891423	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.EndToEnd_MultiProjectSolution	#45398
891375	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45354
891351	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45042
891345	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45391
891312	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45396
891307	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45395
888298	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution	#45358
888033	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution	#45293
888027	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45278
886062	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45293
884843	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#44974
884857	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45252
884539	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44540
884534	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45264
884377	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45255
884466	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45258
884409	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44570
884187	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#44958
884160	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45179
883622	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45244
883604	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45245
883513	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45243
882355	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#44974
882187	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.EndToEnd_NoAPI_ProjectType	#45043
881570	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation
881560	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateNewImageTests.Tasks_EndToEnd_With_EnvironmentVariable_Validation	#45174
880538	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45169
880118	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45157
880099	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#44570
880085	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry	#45042
879699	dotnet/sdk	Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution	#45143
879638	dotnet/sdk	Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateImageIndexTests.CreateImageIndex_Baseline	#43085

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	5	47

[akoeplinger]

@marcpopMSFT who owns Microsoft.NET.Build.Containers?

[baronfel]

@dotnet/sdk-container-builds-maintainers

[baronfel]

@akoeplinger the image in question is registry:2, which we need/use to validate our logic for communicating with container registries via the container registry APIs. Do you know if this image is mirrored anywhere that we can pull from? It's not currently on mcr.microsoft.com.

[nagilson]

Bumping this thread since it's affecting more PRs

[nagilson]

We don't have a path forward to fix this yet - Aspire moved to using their own ACR for storing the images they use, but we need an engineering-systems-managed solution for our entire stack. There's an issue tracking this request but I can't find it at the moment.

[baronfel]

Could be fixed at a root cause level by dotnet/dnceng#3389

[MichaelSimons]

@akoeplinger the image in question is registry:2, which we need/use to validate our logic for communicating with container registries via the container registry APIs. Do you know if this image is mirrored anywhere that we can pull from? It's not currently on mcr.microsoft.com.

@baronfel - It sounds like the test has this dependency and is what is pulling the image. One possible option to workaround scenarios where tests are pulling images directly from DockerHub would be to create a wrapper buildtools image. The Dockerfile would just be a FROM statement. The buildtools images are stored on MCR and therefore you would be able to get around this rate limiting issue.

[nagilson]

@dotnet/sdk-container-builds-maintainers Do you agree with @MichaelSimons suggestion? This has a pretty large impact on PR failure rates. If this is still an issue going forward a month or two down the line, I will sadly suggest that we just disable these tests

[marcpopMSFT]

@donJoseLuis @MichalPavlik I'm told by baronfel that there may be a solution to this of setting up our own ACR mirror of these so we don't get blocked by docker from pulling too often. He said to talk to @eerhardt about that. Is that something you can potentially drive getting set up?

As you can see above, this fails a couple of dozen PRs every week that have to be rerun. It's not a crazy blocker but that's high enough to be something we want eyes on to get fixed eventually.