Pool provider is sometimes slow

[Cheesebaron]

Description

I wrote in the developer community here: https://developercommunity.visualstudio.com/t/macos-13-Pool-provider-seems-slow/10522135#T-ND10523938 Tong Wu told me to log a bug here.

We are often seeing this message in Azure Pipelines that pool provider is slow and often builds are stuck waiting for macOS-13 agents for upwards of 40 minutes. This is even though in our concurrent builds of 25 there are 20+ agents idling.

You can find some of the affected builds here:
https://trackman.visualstudio.com/Golf/_build/results?buildId=395103&view=results
https://trackman.visualstudio.com/Golf/_build/results?buildId=395072&view=results

It doesn't happen every time, but it happens several times on different pipelines during the day.

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• macOS 11
• macOS 12
• macOS 13
• Windows Server 2019
• Windows Server 2022

Image version and build link

Agent name: 'Azure Pipelines 2'
Agent machine name: 'Mac-1700593125567'
Current agent version: '3.227.2'
Operating System
macOS
13.6
22G120
Runner Image
Image: macos-13
Version: 20231025.2
Included Software: https://github.com/actions/runner-images/blob/macos-13/20231025.2/images/macos/macos-13-Readme.md
Image Release: https://github.com/actions/runner-images/releases/tag/macos-13%2F20231025.2
Runner Image Provisioner
2.0.312.1
Current image version: '20231025.2'

Is it regression?

yes

Expected behavior

Not having to wait upwards of 40 minutes for agent to become available

Actual behavior

Pool provider is slow and there is a long wait for agents

Repro steps

Just queue a macOS-13 vmpool build and observe

[vpolikarpov-akvelon]

Hey @Cheesebaron. Thank you for reaching out, but there is nothing to do on the side of the runner images generation team. Please contact GitHub Support.

[Cheesebaron]

Nice to be sent from place to place to get support.

Can GitHub support help me with Azure Pipelines?

[unrateddi]

@Cheesebaron we are facing the same issue here and changed to macos-13-arm64 which seems to work fine. So, maybe you could try that, if that image works for your needs, until they resolve the issue.

[Cheesebaron]

@Cheesebaron we are facing the same issue here and changed to macos-13-arm64 which seems to work fine. So, maybe you could try that, if that image works for your needs, until they resolve the issue.

These don't seem to be available in Azure DevOps pipelines

[unrateddi]

@Cheesebaron that's weird. I have successfully run Azure Pipelines with that image. I even tried to run it now in case something changed.

https://github.com/actions/runner-images/blob/macos-13/20231025.2/images/macos/macos-13-arm64-Readme.md

[Cheesebaron]

Maybe your organization has access to a preview of some sorts?

[unrateddi]

@Cheesebaron hmm maybe, unfortunately I can't know it for sure.

[qwertylolman]

we are facing the same issue. since we were forced to use macos-13 to build xamarin apps and we also were forced to migrate to net7. furthermore, we cannot even build the flutter apps, because of xcode lack of support for macos-12. this issue is quite old, I started to face it maybe 2-3 months ago. when I started the migration to net7.

[mmrazik]

Please note that the only supported images on Azure DevOps are the following: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#software

When it comes to MacOS 13 the only vm image available for Azure DevOps is macOS-13 (in beta).

Anything else is strongly discouraged and it may stop working for you without prior notice.

[qwertylolman]

Please note that the only supported images on Azure DevOps are the following: https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#software

When it comes to MacOS 13 the only vm image available for Azure DevOps is macOS-13 (in beta).

yeah, it's quite confusing that arm64 is not listed anywhere, but presented in the repository xD
https://github.com/actions/runner-images/blob/main/images/macos/macos-13-arm64-Readme.md

but, seems like this image doesn't have the issue. i've switched our flutter project. now it's building a bit faster and without delays. xamarin net7 projects, on the other hand, are having some strange freezes during the dotnet build step, we are currently investigating this as for now.

[Nathan-Molby]

#8971

[IainS1986]

Anything else is strongly discouraged and it may stop working for you without prior notice.

macos-13 stops working without prior notice (hence the ticket).

Is there anyone looking into this or can tell us if its an issue thats being worked on, 'macos-13 is beta' is not really an excuse that can last much longer...

[haavamoa]

Our release pipeline stopped working today due to macOS-13 providers not responding. This is blocking our planned .NET MAUI app release next week.

[cho-trackman]

This is painful:

[adminM365x34654676]

Same issue.....

[Cheesebaron]

I've provided the feedback to the product team when I was at MVP Summit, they are aware of these issues.

[akrol95]

We are affected by the same problem. I hope it will be solved soon.

[olokobayusuf]

Chiming in to say that this issue also affects us.

[jvanoeveren]

I'll also chime in and say that we are required to use macos-13 to build for iOS, due to the warning Apple is giving out in their review process that we need to be building for iOS SDK 17 by April 29 (shown below). This is because MacOS 12 uses XCode 14.2 by default, whereas MacOS 13 uses Xcode 15 which is the version we need. Sometimes I can get this VM pool to pick up right away, but most times it just hangs for a long time.

ITMS-90725: SDK version issue - This app was built with the iOS 16.2 SDK. Starting April 29, 2024, all iOS and iPadOS apps must be built with the iOS 17 SDK or later, included in Xcode 15 or later, in order to be uploaded to App Store Connect or submitted for distribution.