error MSB4057: The target "*****" does not exist in the project .

[FrancescElies]

Description
Starting from today on we are seeing jobs failing with Error MSB4057 where before we had no errors.

I have narrowed two builds for the same commit that present different behaviours.
Both throw Error MSB4057 but in different jobs.

Build 1

• Image DAta: Microsoft Windows Server 2019\n10.0.17763\nDatacenter
• Agent.Id: 153
• Agent.MachineName: fv-az33-885

Build 2 (same commit as Build 1)

• Image Data: Microsoft Windows Server 2019\n10.0.17763\nDatacenter
• Agent.Id: 193
• Agent.MachineName: fv-az33-885

I guess this has to do with some sort of update on msbuild or the agents, maybe with Pre-release
win19/20210525.0?

Area for Triage:

• C/C++
• .NET Core

Question, Bug, or Feature?:
Bug

Virtual environments affected

• Windows Server 2019

Image version

• Image Data: Microsoft Windows Server 2019\n10.0.17763\nDatacenter

Expected behavior
No Error MSB4057 as it used to be.

Actual behavior
Error MSB4057 being thrown

Repro steps
See build links.

I am aware this links belong to a private repository and it makes little sense for me to make a a minimal reproducible example.

[dibir-magomedsaygitov]

Hello @FrancescElies. Thank you for your report. We will take a look.

[maxim-lobanov]

Most likely related to upgrade Visual Studio from 16.9 to 16.10.

[FrancescElies]

Seems to be related to dotnet/msbuild#6373.

As you suggested the problem seems to be in 16.10.0, will be solved with 16.0.1 .

What is the plan until this version is released and rolled out in your virtual machines?

[maxim-lobanov]

We have to wait for 16.10.0.1 to fix the issue on images.
Unfortunately, Visual Studio doesn't provide way to install non-latest version and installer always pick up latest version (16.10.0 for now) so we can't freeze 16.9 on images somehow.

Some earlier discussion and context around it: #2321 (reply in thread)

[FrancescElies]

I will try to get around this issue with vswhere or using :Rebuild in our targets as this seems to be a workaround.

I'll keep you posted. Thanks for your help

[andy-c-jones]

We are seeing this in Azure DevOps. Is there any way of use forcing version 20210516 of the image?
None of the workarounds are working for us at the minute. Edit: the workaround below does work for us

[maxgolov]

The minimal trivial reproducible example can be found here in OSS repo:
microsoft/cpp_client_telemetry#883

Failing with MSBuild from vs2019 16.10.0

msbuild solution.sln /target:zlib,sqlite

The workaround is a bit ugly - scan all affected batch files, all target lists, all vcpkg port files, etc. And append :Rebuild to each target. Works okay with MSBuild from vs2019 16.10.0, but requires manual intervention.

msbuild solution.sln /target:zlib:Rebuild,sqlite:Rebuild

Looks like the hot-fix would be available in a few weeks in 16.10.1? But the issue would affect more than one customer in the next few weeks for certain. I had at least 5 devs / different teams affected by this bug in one day. It's been a hot 🔥 issue for us.

[rhuijben]

Already reinstalled a machine in an attempt to fix this. Eagerly waiting for a fix

[aronbardsley]

This is very frustrating. I'm trying to add :Rebuild and its not working. Perhaps this scenario should be added to some sort of test framework by the MSBuild team?

I know it's no-one's fault, but these kinds of avoidable issues are a complete productivity killer

[FrancescElies]

@maxgolov Thanks for the workaround!

:Rebuild has partially solved the problem for us. We also have solutions with c# projects even after this workaround they were still broken, not all dlls are included in the output folder.

For c projects, the following works:

 msbuild solution.sln /target:zlib:Rebuild

For c# projects, the following seems to help but I got other errors now (I'll write back as soon as I know more why):

 msbuild solution.sln /target:csharpproject:Rebuild,csharpproject:Publish

We make extensive usage of /target filtering. It's a pity this was broken.

I'm also pretty disappointed with dotnet/msbuild#6373 , eagerly waiting for a fix as well.

[Bouke]

So this is breaking builds for a lot of people. How about rolling back this breaking change until Visual Studio 16.10.1 Hotfix is released?

[maxim-lobanov]

As I mentioned above, there is no way to install non-latest version of Visual Studio on VM images.
If we just roll back VMs to the previous image version, we will have to freeze image version and any VM updates until the next VS release that is not acceptable considering that Hosted VMs have a lot of software except Visual Studio.

If you upgrade VS to the 16.10 on your local machine, you won't be able to downgrade it too.

[Bouke]

I'm very disappointed by your reaction. In a professional organisation one assumes there are procedures to revert changes when there are serious problems. I would argue breaking various people's builds would warrant such a roll-back. If you don't have a way to restore the previous version of Visual Studio (16.9), then there must be something seriously wrong with the procedures you have in place.

[maxim-lobanov]

We always try to rollback images in case of any issues and fix it on our side if it is possible.
In case of Visual Studio, we install and use it as usual customers and we don't control update process from our side like selecting specific version to install and etc. If Visual Studio team doesn't provide way to install non-latest version or downgrade version, unfortunately, we won't be able to do it from our side like no one user won't be able to do it on their local machine.

[Bouke]

What about Installing an earlier release?

There's an additional paragraph that applies here as well, emphasis mine:

If your organization has standardized on a particular servicing baseline of Visual Studio 2019, we recommended staying current with the latest servicing release of that supported baseline. The minor versions that have been declared as supported servicing baselines for Visual Studio 2019 are: 16.4, 16.7 and 16.9. Refer to our support policy for additional information.

So even the Microsoft Visual Studio Team advices against always upgrading to the latest minor version.

[kim3er]

This must be related to microsoft/azure-pipelines-tasks#14922. We use the target parameter in our DevOps pipeline.

[FrancescElies]

@maxgolov it seems like Visual Studio released 16.10.1 and it's available for download.

[miketimofeev]

@FrancescElies we've already started the new image deployment. It will be finished on Monday, but some customers may expect the new version earlier since the deployment is performed gradually.

[kim3er]

I can confirm the targets cli parameter is work on VSBuild 1.187.0.

[miketimofeev]

@FrancescElies could you please confirm that the issue is resolved with the new image version?

[FrancescElies]

@miketimofeev 👌 checking at the moment, I will close the issue as soon as I removed the workaround from our codebase and everything works fine.

[FrancescElies]

Closing. We removed the workaround and everything seems to be working fine.

@maxim-lobanov @miketimofeev thanks for your support!