Azure DevOps is down - all CI jobs failing

[jkotas]

Log:

error : (NETCORE_ENGINEERING_TELEMETRY=Helix) System.Net.Http.HttpRequestException: Request to https://dev.azure.com/dnceng/public/_apis/test/Runs/47600964/results?api-version=5.1-preview.6 returned failed status 503 Service Unavailable
{"$id":"1","innerException":null,"message":"TF10216: Azure DevOps services are currently unavailable. Try again later. Activity Id: 78014780-2f16-4321-822b-32c980bee1f8","typeName":"Microsoft.VisualStudio.Services.CircuitBreaker.CircuitBreakerShortCircuitException, Microsoft.VisualStudio.Services.Common","typeKey":"CircuitBreakerShortCircuitException","errorCode":0,"eventId":3000}

{
  "errorMessage" : "Microsoft.VisualStudio.Services.CircuitBreaker.CircuitBreakerShortCircuitException"
}

Report

Build	Definition	Step Name	Console log
1776808	dotnet/runtime	Send tests to Helix (Unix)	Log
1776846	dotnet/runtime	Send to Helix	Log
1776944	dotnet/runtime	Send to Helix	Log
1776886	dotnet/runtime	Send to Helix	Log
1776865	dotnet/runtime	Send to Helix	Log
1776869	dotnet/runtime	Send tests to Helix (Unix)	Log

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
0	0	6

[ghost]

Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Log:

error : (NETCORE_ENGINEERING_TELEMETRY=Helix) System.Net.Http.HttpRequestException: Request to https://dev.azure.com/dnceng/public/_apis/test/Runs/47600964/results?api-version=5.1-preview.6 returned failed status 503 Service Unavailable
{"$id":"1","innerException":null,"message":"TF10216: Azure DevOps services are currently unavailable. Try again later. Activity Id: 78014780-2f16-4321-822b-32c980bee1f8","typeName":"Microsoft.VisualStudio.Services.CircuitBreaker.CircuitBreakerShortCircuitException, Microsoft.VisualStudio.Services.Common","typeKey":"CircuitBreakerShortCircuitException","errorCode":0,"eventId":3000}

Author:	jkotas
Assignees:	-
Labels:	area-System.Net.Http, untriaged
Milestone:	-

[jkotas]

cc @dotnet/dncenghot

[ghost]

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Log:

error : (NETCORE_ENGINEERING_TELEMETRY=Helix) System.Net.Http.HttpRequestException: Request to https://dev.azure.com/dnceng/public/_apis/test/Runs/47600964/results?api-version=5.1-preview.6 returned failed status 503 Service Unavailable
{"$id":"1","innerException":null,"message":"TF10216: Azure DevOps services are currently unavailable. Try again later. Activity Id: 78014780-2f16-4321-822b-32c980bee1f8","typeName":"Microsoft.VisualStudio.Services.CircuitBreaker.CircuitBreakerShortCircuitException, Microsoft.VisualStudio.Services.Common","typeKey":"CircuitBreakerShortCircuitException","errorCode":0,"eventId":3000}

Author:	jkotas
Assignees:	-
Labels:	area-Infrastructure, untriaged
Milestone:	-

[jkotas]

Examples of affected PR: #69363, #69373

[premun]

Thanks for the links, I will have a look at the builds in more detail.

I can see there was a high amount of these errors over the weekend and they seem to have reduced back again:

The status pages of Azure/Azure DevOps claim there was no outage so I will see if we need to open an incident with them.

[premun]

I have opened dotnet/arcade#9372 to track this and we will open an IcM with Azure DevOps

[hoyosjs]

Closing this as the incident was closed.