Windows-2016 environment removal postponed until April 1st, 2022

[alepauly]

Breaking changes

Windows 2016 hosted runners were scheduled for full removal on March 15th, 2022. However, we are still seeing significant usage from customers and we want to give them more time to migrate to the new runners. In order to give customers another chance to move to either windows-2019 or windows-2022 we will delay the removal of windows-2016 until April 1st, 2022.

To raise awareness of the removal, we will have rolling 12-hour periods where the image will be unavailable for customers. During these "brownout" periods, customers will see an error message indicating that the image is slated for removal on April 1st, 2022. The schedule for those brownout periods is as follows for both Azure DevOps and GitHub Actions:

1. Tuesday March 15, 20:00 UTC to March 16, 08:00 UTC
2. Thursday March 17, 08:00 UTC to March 17, 20:00 UTC
3. Friday March 18, 20:00 UTC to March 19, 08:00 UTC
4. Monday March 21, 08:00 UTC to March 21, 20:00 UTC
5. Tuesday March 22, 20:00 UTC to March 23, 08:00 UTC
6. Wednesday March 23, 20:00 UTC to March 24, 08:00 UTC
7. Friday March 25, 08:00 UTC to March 25, 20:00 UTC
8. Sunday March 27, 20:00 UTC to March 28, 08:00 UTC
9. Tuesday March 29 Wednesday March 30, 08:00 UTC to March 30, 20:00 UTC
10. Full removal - April 1, 15:00 UTC

See the original announcement at #4312

Target date

April 1st, 2022

The motivation for the changes

Windows Server 2016 Active support ends on 11 Jan 2022 and Windows Server 2022 VM image is going out of beta later this year.
As part of our ongoing efforts to keep GitHub and Azure Devops hosted runners updated and secure, the Windows 2016 virtual environment will be removed from GitHub Actions and Azure DevOps.

Possible impact

Workflows targeting the Windows 2016 image will fail.

Virtual environments affected

• Ubuntu 18.04
• Ubuntu 20.04
• macOS 10.15
• macOS 11
• Windows Server 2016
• Windows Server 2019
• Windows Server 2022

Mitigation ways

Change your workflows to use windows-latest, windows-2022, or windows-2019

[rdhaundiyal]

Hi, I am not using windows-2022 but still getting the error related to 2016
is it some side effect

##[Error 1]
This is a scheduled windows-2016 brownout. The windows-2016 environment is deprecated and will be removed on April 1st, 2022. For more details, see #5238

##[Error 2]
The remote provider was unable to process the request.

Deployment canceled by Microsoft.VisualStudio.Services.ReleaseManagement on 3/16/2022, 6:15 PM

[GokulKannanN]

We are using vmImageName: 'ubuntu-latest' but still we are getting this error.
what is the solution for this issue.

[rdhaundiyal]

update, it was the release agent which was by default picking up 2016-17 even after creating a new release from scratch, settign it manually to 2022 worked

[GokulKannanN]

worked

I did not understand. Can you please tell me where I have to update 2022.
because Ubuntu does not have 2022, its latest version is 20.04

[rdhaundiyal]

my issue was related to release , in the release pipeline -> edit, you have agent, clicking on the agent show you the provider
https://blog.novacare.no/devops-change-agent-specification-to-windows-latest/

[ManaliMnS]

worked

I did not understand. Can you please tell me where I have to update 2022. because Ubuntu does not have 2022, its latest version is 20.04

Go to your release pipeline. Edit . Go to agent job. Change agent specification to windows latest

[CodeIn24]

Hi, Just for better understanding, we have self-hosted agent on one of our machines having windows 2016 OS , so this means we have to upgrade the OS and re-host the agent for pipelines to keep running, am I missing anything?

This option will not work for us considering our setup:

[alepauly]

Hi, Just for better understanding, we have self-hosted agent on one of our machines having windows 2016 OS , so this means we have to upgrade the OS and re-host the agent for pipelines to keep running, am I missing anything?

@CodeIn24 - this doesn't affect self-hosted agents. it's only related to the hosted Windows 2016 virtual environment. You don't need to change anything.

[CodeIn24]

@alepauly good to know this, thanks

[Razakhel]

Hello,

As is the custom with MSVC, the SDK currently available on the windows-2019 image is broken and does not compile:

C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\um\oaidl.h(502,17): error C2059: syntax error: '/'
C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\um\oaidl.h(530,17): error C2059: syntax error: '}'
C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\um\oaidl.h(531,13): error C2059: syntax error: '}'
C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\um\oaidl.h(533,9): error C2059: syntax error: '}'
C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\um\oaidl.h(534,5): error C2059: syntax error: '}'
C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\um\oaidl.h(665,16): error C2079: 'varDefaultValue' uses undefined struct 'tagVARIANT'
C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\um\oaidl.h(950,16): error C2079: 'varValue' uses undefined struct 'tagVARIANT'
C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\um\propidlbase.h(319,24): error C2059: syntax error: '/'
C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\um\propidlbase.h(378,37): error C2371: 'pvarVal': redefinition; different basic types
C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\um\oaidl.h(510): message : see declaration of 'pvarVal'
C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\um\propidlbase.h(379,9): error C2059: syntax error: '}'
C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\um\propidlbase.h(380,5): error C2059: syntax error: '}'
C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\um\propidlbase.h(383,3): error C2059: syntax error: '}'
C:\Program Files (x86)\Windows Kits\10\Include\10.0.17763.0\um\propidlbase.h(384,1): error C2059: syntax error: '}'

I did file a report on this a couple months ago in anticipation, but I guess many users will simply not be able to upgrade the version because of this. Using the windows-2022 image & VS 2022 works fine for me, but may not be suitable for others. Please either update the SDK or explicitly state that the 2019 image cannot be used for C & C++ users, so they can safely migrate to 2019 or 2022.

Thank you!

[toddbellamy]

We are getting the windows-2016 brownout error even though our Pipeline is using windows-latest and the log even shows that windows-2022 agent was used.
Why?

[jackweldon]

@oddbellamy try
vmImageName: 'windows-2019'

[sachinbdoshi]

Well I just closed a case with Microsoft where they confirmed that Agent Pool with vs2017-2016 will be supported until June-2022 and here this article is saying it is being deprecated in April 2022. what is the real truth ? today we received the Windows 2016 brownout error in our build pipeline which is building win 2016 container images.

link to where microsoft updated the information that they will deprecate the agent pools with vs2017-2016 in June 2022.
https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=azure-devops&tabs=yaml#software

[Croydon]

@sachinbdoshi #5238 (comment)

[gene-keys]

IMPORTANT NOTE FOR THOSE OF YOU STILL STRUGGLING WITH WIN2016 BROWNOUT NOTICE.

CHECK EVERYTHING!

NOT JUST YOUR PIPELINE CODE, CHECK PIPELINE TRIGGERS POOLS. CHECK RELEASES POOLS, CHECK ALL THE POOLS EVERYWHERE ON YOUR DEVOPS ACCOUNT. WHAT WILL HAPPEN IS YOU WILL EVENTUALLY FIND SOMETHING STILL SET TO USE WIN2016 POOL. ONCE YOU FIND ALL OF THEM YOUR ACCOUNT WILL NO LONGER GET THE BROWNOUT NOTICE EVEN DURING THE SCHEDULED TIMES. SO IF YOU ARE GETTING THE NOTICE ITS NOT YOUR WIN2017 POOL, OR WIN-LATEST IN THE YML, ITS SOMETHING ELSE,

AZURE NOTICES ARE DOING A TERRIBLE JOB ON THIS WITH THEIR WORDING AND WHERE THE NOTICE APPEARS. THE NOTICE MEANS YOUR AZURE DEVOPS ACCOUNT IS STILL USING WINDWOS POOL SOMEWHERE, YOU DID NOT CHECK EVEYWHERE.

[eli-schwartz]

New announcement is here #5403

I'm still getting brownout cancellations on Azure, which... mention April 1 (confusing) and link to this ticket, As a small UX thing, this should probably update the message text.

The exact job where I saw this is here: https://dev.azure.com/jussi0947/jussi/_build/results?buildId=14019&view=logs&j=e635d226-97b8-55eb-2970-00a63a10ba8b

[miketimofeev]

@eli-schwartz sorry, we will change the message during the next brownout

[eli-schwartz]

BTW the message is still the old one :) seen today in https://dev.azure.com/jussi0947/jussi/_build/results?buildId=14088&view=logs&j=678c5357-36f0-50cc-b96e-1a6f7149eed2

[todd-fraser]

@eli-schwartz @miketimofeev is there anywhere I can find when this brownout will be over? The only dates I've found listed out specifically are for previous dates. I was asked to deploy after hours and now I'm getting this deprecation message I'm not able to fix tonight. Oops.

[eli-schwartz]

@todd-fraser the new brownout dates are listed at #5403 and hopefully for the next brownout we will get links pointing there instead. :)

[todd-fraser]

@eli-schwartz Thanks much! I'll keep working to change the agent tonight. Not my expertise exactly. Only deploying to Prod, what could possibly go wrong?

[eli-schwartz]

I dunno :) for my part, the CI for a build system just fails to test whether it correctly generates VS solution files for the windows-2016 environment (technically, for vs2017, but I think windows-2016 is the only image that has vs2017 installed) so it's a bit questionable whether we should remove those jobs at all.

In the meantime we simply ignore these cancellations. At some point we'll probably switch some jobs over to vs2019 and pray that no users still have vs2017 installed (or that we don't regress for it due to lack of test coverage).

What can go wrong? We could completely fail to generate solution files for an outdated toolchain that no one should use. 😆

[Ziya137200]

Breaking changes

Windows 2016 hosted runners were scheduled for full removal on March 15th, 2022. However, we are still seeing significant usage from customers and we want to give them more time to migrate to the new runners. In order to give customers another chance to move to either windows-2019 or windows-2022 we will delay the removal of windows-2016 until April 1st, 2022.

To raise awareness of the removal, we will have rolling 12-hour periods where the image will be unavailable for customers. During these "brownout" periods, customers will see an error message indicating that the image is slated for removal on April 1st, 2022. The schedule for those brownout periods is as follows for both Azure DevOps and GitHub Actions:

1. Tuesday March 15, 20:00 UTC to March 16, 08:00 UTC
2. Thursday March 17, 08:00 UTC to March 17, 20:00 UTC
3. Friday March 18, 20:00 UTC to March 19, 08:00 UTC
4. Monday March 21, 08:00 UTC to March 21, 20:00 UTC
5. Tuesday March 22, 20:00 UTC to March 23, 08:00 UTC
6. Wednesday March 23, 20:00 UTC to March 24, 08:00 UTC
7. Friday March 25, 08:00 UTC to March 25, 20:00 UTC
8. Sunday March 27, 20:00 UTC to March 28, 08:00 UTC
9. Tuesday March 29 Wednesday March 30, 08:00 UTC to March 30, 20:00 UTC
10. Full removal - April 1, 15:00 UTC

See the original announcement at #4312

Target date

April 1st, 2022

The motivation for the changes

Windows Server 2016 Active support ends on 11 Jan 2022 and Windows Server 2022 VM image is going out of beta later this year. As part of our ongoing efforts to keep GitHub and Azure Devops hosted runners updated and secure, the Windows 2016 virtual environment will be removed from GitHub Actions and Azure DevOps.

Possible impact

Workflows targeting the Windows 2016 image will fail.

Virtual environments affected

• Ubuntu 18.04
• Ubuntu 20.04
• macOS 10.15
• macOS 11
• Windows Server 2016
• Windows Server 2019
• Windows Server 2022

Mitigation ways

Change your workflows to use windows-latest, windows-2022, or windows-2019

Duplicate of #

[Kalmuraee]

My issue is on deployment of the release in azure, I have no idea how to change that to windows-latest.