Azure App Service: Failed to Deploy. The underlying connection was closed: An unexpected error occurred on a send.

[sanar-microsoft]

Required Information

Entering this information will route you directly to the right team and expedite traction.

Question, Bug, or Feature?
Type: Bug

Enter Task Name: Azure App Service deploy(4.163.3)

list here (V# not needed):
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks

Environment

• Server - Azure Pipelines or TFS on-premises?

  • If using Azure Pipelines, provide the account name, team project name, build definition name/build number: Account name: DevDiv, DevDiv, Release: Mindaro-RP/Rel-20200224.3-4

• Agent - Hosted or Private:

  • If using private agent, provide the OS of the machine running the agent and the agent version: Windows_NT, 2.165.0

Issue Description

Task name: Deploy Azure App Service
Environment: Canary - East US 2
The Deploy Azure App Service task fails due to connection problems, but when the same code is published using Visual Studio things work.

Only happening for web apps in Canary (East US 2 EUAP) region

Task logs

##[debug]Exit code 4294967295 received from tool 'C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe'
##[debug]Exit code 4294967295 received from tool 'C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe'
##[debug]STDIO streams have closed for tool 'C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe'
##[debug]STDIO streams have closed for tool 'C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe'
##[warning]Make sure the machine is using TLS 1.2 protocol or higher. Check https://aka.ms/enableTlsv2 for more information on how to enable TLS in your machine.
##[error]Error: Error: Could not complete the request to remote agent URL 'https://.scm.azurewebsites.net/msdeploy.axd?site='.
Error: The underlying connection was closed: An unexpected error occurred on a send.
Error: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
Error: An existing connection was forcibly closed by the remote host
Error count: 1.

Troubleshooting

Checkout how to troubleshoot failures and collect debug logs: https://docs.microsoft.com/en-us/vsts/build-release/actions/troubleshooting

Error logs

##[error]Error: Error: Could not complete the request to remote agent URL 'https://.scm.azurewebsites.net/msdeploy.axd?site='.
Error: The underlying connection was closed: An unexpected error occurred on a send.
Error: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
Error: An existing connection was forcibly closed by the remote host
Error count: 1.

[floaus]

Hi! We're getting this error on our release pipeline when trying to deploy a function app to Europe West

[NickolaiL]

Hello, we are getting the same errors when trying to deploy a Web App on West Europe too.

[NikitaKudin]

We are experiencing the same issue now. Help!

[mschouveller]

same issue since yesterday

[error]Failed to deploy App Service.
[error]Error: Could not complete the request to remote agent URL 'https://xxxx.scm.azurewebsites.net/msdeploy.axd?site=xxxx'.
Error: The underlying connection was closed: An unexpected error occurred on a send.
Error: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
Error: An existing connection was forcibly closed by the remote host
Error: C:\Program Files\IIS\Microsoft Web Deploy V3\msdeploy.exe failed with return code: 4294967295

[willynagel]

Same issue here, but only with the deployment of function apps.
Changing TLS level to 1.0 works as a workaround.

[rdeveen]

Changing TLS level to 1.0 works as a workaround.

How did you do that?

Answer: In the Azure Portal, Blade TLS/SSL settings, changing the Minimal TLS version to 1.0.

Update: don't do this, insecure! See #12444 (comment)

[willynagel]

Indeed, in the portal or in our case in the ARM templates before deploying the webapp.

[mschouveller]

Thanks willynagel and rdeveen
Works for me

[michaelspinks]

Thanks worked for us today. Fine on 1.2 prior to this.

[jayeshpatel6]

The work around worked for us, anyone know if the issue has been fixed?

[rdeveen]

This script is using [Net.ServicePointManager]::SecurityProtocol to check if Tls12 is enabled. To check if Tls12 is enabled on your build machine you can run this command in Powershell console.

PS C:\> [Net.ServicePointManager]::SecurityProtocol
Tls, Tls11, Tls12
PS C:\>

To enforce Tls12 you can add this keys in the registry:

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319]
"SchUseStrongCrypto"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\.NETFramework\v4.0.30319]
"SchUseStrongCrypto"=dword:00000001

No need to change TLS level to 1.0 in the Azure Portal.

[jvanderbiest]

@rdeveen Strange on our build server we get the same protocols when running your PowerShell command, but we have the same issue.

[jaeger-csj]

+1 for @rdeveen 's solution. Per this article, I believe that the Self-hosted Windows build agent software is targeting an earlier version of .Net. Therefore, it executes pipeline deploys using the TLS settings that are default to that version of .Net.

In that article, it suggests modifying the OS registry with the values referenced by @rdeveen and two more (all of which copied below). After I do that and re-run the [Net.ServicePointManager]::SecurityProtocol PS command, I get back "SystemDefault". As my OS is on .Net 4.7 (which uses TLS 1.2 by default), the build agent uses TLS 1.2 and we no longer have any pipeline deploy issues ran through this agent pool.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v2.0.50727]
"SystemDefaultTlsVersions"=dword:00000001
"SchUseStrongCrypto"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\.NETFramework\v4.0.30319]
"SystemDefaultTlsVersions"=dword:00000001
"SchUseStrongCrypto"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v2.0.50727]
"SystemDefaultTlsVersions"=dword:00000001
"SchUseStrongCrypto"=dword:00000001

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v4.0.30319]
"SystemDefaultTlsVersions"=dword:00000001
"SchUseStrongCrypto"=dword:00000001

[pollax]

In my case the script returns SystemDefault.

PS C:\> [Net.ServicePointManager]::SecurityProtocol
SystemDefault

My agent is on .NET 4.8, so I just rebooted the build agent and... 🎉 now it works again. Wtf?

[miguelEsteban]

Same issue on build agent with 2012 R2

[JennyLawrance]

We are aware of the issue. This was a result of App Service enforcing the minTlsVersion on the web app for the SCM endpoint. As folks pointed out already, this is a breaking change from a behavior perspective, even though it was well intended.

As some of you already figured it out, there are multiple ways to get unblocked while App Service team address this

1. Update the build environment to use TLS 1.2.
2. Consider updating the devops pipeline to use zip-deploy. The tools for zip deploy is a simple CURL, which has enough configuration on the client side tooling to pick a secure TLS version.
3. Changing the site's minTlsVersion to 1.0 is not recommended, since that comes with its own issues. TLS 1.0 is not considered secure any more by any one. Use that option as a last resort knowing what it really implies.

[frankwinfaber]

Had the same issue when trying to run a deployment on a App Service in West Europe, but only for our production environment.
The output of [Net.ServicePointManager]::SecurityProtocol on our build-server was "Ssl3, Tls", but after updating the registry is changed to "SystemDefault".

Temporarely switched to an Hosted Build Agent, but after reboot of the build server the deployment is running as smooth as before. :-)

[SumiranAgg]

Thanks @JennyLawrance for the update. Closing the issue.