Deploying ASP.NET Core 2.1 application from repo fails

[hajekj]

Repro steps.

When deploying ASP.NET Core 2.1 application to Azure App Service from VSTS using CI in App Service, the deployment fails.

Project structures.

The application doesn't have any specific structure and is basically built with the default scaffolding.

The log/error given by the failure.

Command: "D:\home\site\deployments\tools\deploy.cmd"
Handling ASP.NET Core Web Application deployment.
  Restore completed in 666.09 ms for D:\home\site\repository\src\Captive365\Captive365.csproj.
Microsoft (R) Build Engine version 15.7.179.6572 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Restore completed in 168.86 ms for D:\home\site\repository\src\Captive365\Captive365.csproj.
  Captive365 -> D:\home\site\repository\src\Captive365\bin\Release\netcoreapp2.1\Captive365.dll
  Captive365 -> D:\home\site\repository\src\Captive365\bin\Release\netcoreapp2.1\Captive365.Views.dll
  Captive365 -> D:\local\Temp\8d5d06868d44b50\
Cannot removed the 'to be deleted' directory, ignoring
Error: Failed to change file that is currently being used "D:\home\site\wwwroot\Captive365.pdb"
KuduSync.NET from: 'D:\local\Temp\8d5d06868d44b50' to: 'D:\home\site\wwwroot'
Copying file: 'Captive365.deps.json'
Copying file: 'Captive365.dll'
Copying file: 'Captive365.pdb'
Failed exitCode=1, command="kudusync" -v 50 -f "D:\local\Temp\8d5d06868d44b50" -t "D:\home\site\wwwroot" -n "D:\home\site\deployments\34b8e7fda1c314d142b93e74a733ab35853c1f25\manifest" -p "D:\home\site\deployments\2f3fda43bf1753bbc34e113a02bea382f6bc0bf5\manifest" -i ".git;.hg;.deployment;deploy.cmd"
An error has occurred during web site deployment.
Error: Failed to change file that is currently being used "D:\home\site\wwwroot\Captive365.pdb"\r\nD:\Program Files (x86)\SiteExtensions\Kudu\74.10607.3431\bin\Scripts\starter.cmd "D:\home\site\deployments\tools\deploy.cmd"

Debug your Azure website remotely.

The affected site's name is captive365.azurewebsites.net

Mention any other details that might be useful.

This doesn't seem to happen on other sites I have deployed which are running ASP.NET Core 2.0.

[watashiSHUN]

looking at the kudu site, I didn't find the failed deployment log for commit 34b8e7, but I do see that your most recent deployment was successful:

Were you able to repro the issue? were you using the pdb file during the deployment?

[hajekj]

It has happened multiple times, I managed to redeploy with site restart.

[1digicoder]

I logged in remotely via Kudu and found out that it's not running .NET Core 2.1. How did you get it to recognize 2.1?

root@83992fb0c794:/home/site/wwwroot# dotnet ./FizzBuzzApi.dll
It was not possible to find any compatible framework version
The specified framework 'Microsoft.AspNetCore.App', version '2.1.0' was not found.
  - Check application dependencies and target a framework version installed at:
      /
  - Alternatively, install the framework version '2.1.0'.
root@83992fb0c794:/home/site/wwwroot# dotnet --info

Microsoft .NET Core Shared Framework Host

  Version  : 2.0.7
  Build    : 2d61d0b043915bc948ebf98836fefe9ba942be11

root@83992fb0c794:/home/site/wwwroot#

[hajekj]

@1digicoder: Your log appears to be from Azure App Service on Linux. Unrelated to this issue. There doesn't seem to be a 2.1 runtime container for App Service on Linux yet. You can either build your own container, wait until the official is published or use App Service on Windows.

[davidebbo]

@hajekj correct, Linux works differently. Let's limit this issue to the Windows side so we don't end up confusing ourselves :)

[1digicoder]

@hajekj - Noticed the difference, and thanks for the extra info, was able to get past my blocker.

[hajekj]

I can make another application deploy which is likely to fail and keep it that way for few days if you would like me to.

[watashiSHUN]

I tried with my dummy2.1core app and consecutive deployments seems to work
hmm I would say your case is special...
if you were able to deploy with a site restart, then it probably means that Captive365.pdb is somehow being used, but this does not make sense to me, and I checked your running site, you have active handle on D:\home\site\wwwroot\Captive365.dll not the pdb file

Yes, it will be helpful to investigate a case, another question I have is if you change your project to 2.0 does it solve the issue?

[hajekj]

Well I just tried pushing again and it seems to work now... Could it be because I previously had remote debugging enabled which I then disabled?

[watashiSHUN]

very likely...since the pdb is used by remote debugging. Although I am not sure what's the expected user experience for that case, I may need your complete workflow

[hajekj]

So my workflow:

1. Deploy regular version through Git
2. Some issue occurs in my code (not with deployment), need to do Remote Debugging session
3. Deploy Debug version from Visual Studio to App Service through WebDeploy
4. Attach remote debugger from Cloud Explorer
5. Perform some debugging
6. Stop the debugger in Visual Studio
7. Push changes to Git and commit the fixed version

I just tried but I couldn't reproduce the issue again... I swear this happened to me like 3-4 times. Maybe was just bad luck.

[ellern]

I've hit this issue multiple times with different web sites running on different app service plans running on Windows. Currently I have to restart the web app and then do a redeploy to make the deployment succeed. I can't remember having any issues before I updated my web apps to .NET Core 2.0.0

[watashiSHUN]

Also with debugging? the same error message? I wonder what .NET Core has changed from 2.0 to 2.1...
you can always verify in kudu, before you push your changes, check to see if the pdb file handler is taken, for instance, this is my dummy core site, and at run time only the dll file is in use

[ellern]

I'm not debugging and remote debugging is switched off. I do deployments from git - almost every single time I push, it errors out telling me the file (xxxx.Views.pdb) is in use. Restart app, redeploy works every time, but it's annoying. I found out if I push two changes within a short timespan then there is no lock on the pdb file and the deployment works as expected.

Just pushed a got the error message:

KuduSync.NET from: 'D:\local\Temp\8d5d80c09bb9d1c' to: 'D:\home\site\wwwroot'
Error: Failed to change file that is currently being used "D:\home\site\wwwroot\Site.Web.Views.pdb"
Copying file: 'Site.Web.deps.json'
Copying file: 'Site.Web.runtimeconfig.json'
Copying file: 'Site.Web.Views.dll'
Copying file: 'Site.Web.Views.pdb'
Failed exitCode=1, command="kudusync" -v 50 -f "D:\local\Temp\8d5d80c09bb9d1c" -t "D:\home\site\wwwroot" -n "D:\home\site\deployments\c764c8a77ef871df3f13953049f5bc51088cac1f\manifest" -p "D:\home\site\deployments\bd8a3c2f17acba1e45b1c14276d9530f24e5a649\manifest" -i ".git;.hg;.deployment;deploy.cmd"
An error has occurred during web site deployment.
Error: Failed to change file that is currently being used "D:\home\site\wwwroot\Site.Web.Views.pdb"\r\nD:\Program Files (x86)\SiteExtensions\Kudu\74.10611.3437\bin\Scripts\starter.cmd "D:\home\site\deployments\tools\deploy.cmd"

[davidebbo]

This feels more like a dotnet runtime issue than a Kudu issue. Kudu just does deployment, and if there is a file that is hopelessly locked by the Core runtime, there is little that Kudu can do to deploy over it.

So it's best to switch the focus on that angle, and try to figure out why the runtime is locking your PDBs if you're not debugging. It may be best to report that in a .NET Core forum or repo.

[hajekj]

I just hit the same issue today again. I restarted the site and it started to work. Then, I made some more changes (not critical) and it failed again. I can see that the pdb is once again being used. I left the app in the failed state so if you want to try to do any sort of debugging, feel free.

[watashiSHUN]

Yes, your .pdb file is locked, and after some research, it does not seem to be related to remote debugging since the .pdb file is not acquired by the debugger (usually a separate process) but by runtime itself. Talked to the dotnet core folks, this is possible if your code throws exception, and the runtime will load the pdb file and try to get the line number, does this apply to you?

[hajekj]

I don't think the code throws any exception. I will double check soon. Still, shouldn't there be some automatic mechanism to "unlock" all files when redeploying?

[davidebbo]

@hajekj one thing that could be interesting to experiment with is to use a custom deployment script that creates an app_offline.htm before KuduSync, and deletes it after. This would basically take the app offline during the copying, which should unlock everything.

[hajekj]

I will give that a try. So is it enough to create app_offline.htm and Kudu will unlock it automatically or I have to call something after that as well?

[davidebbo]

@hajekj Kudu knows nothing about this file. It's the Core Runtime that detects it and takes itself offline. You may need to add a one second wait after creating the file. Tip: you can create an empty file by just calling touch app_offline.htm (in the right folder, it should go in wwwroot).

[hajekj]

I will try it shortly and report back.

[CampsiteHelper]

I am having the same issue, I am getting "Failed to change file that is currently being used" and it is pointing to a PDB file that is the same as my webApp.dll file.

Its a web app, core 2.1, deployed from BitBucket.

[davidebbo]

@CampsiteHelper what is causing the PDB to be loaded? Were you remote debugging the app? When I test with a default 2.1 MVC app, I can't repro. My specific steps:

• New 2.1 MVC app locally
• Git push it to new Azure Web App
• Run the app in Azure
• Make a change locally. e.g. change the string in the About() action
• Commit and push again

And everything works. Are you able to repro the issue with those steps? If not, it would be interesting to try isolating what is different with your app that is causing the PDB to get locked.

[CampsiteHelper]

@davidebbo Nothing that I am aware of. I believe the cause is actually that the app is not given enough time to shut down (this is the out of the box deployment script..) as when I shut the app down, the deploy works fine.

[EddyP23]

We upgraded recently to 2.2 and we are still getting multiple errors of pdbs being locked. We are using ordinary git deploy and been getting them for quite a bit with 2.1 as well. This is extremely annoying as you even need to create a new git commit and rerun the pipeline, as git push doesn't work with the same commit, as it claims Everything up to date and doesn't try to redeploy the app.

[jvano]

Hi

If the problem persists and is related to running it on Azure App Service, please open a support incident in Azure:
https://learn.microsoft.com/en-us/azure/azure-portal/supportability/how-to-create-azure-support-request

This way we can better track and assist you on this case

Thanks,

Joaquin Vano
Azure App Service