Functions Host Updates for Drain Mode

[nzthiago]

Drain Mode is an internal feature of the Azure Functions platform (Functions Consumption and Premium plans) to enable graceful scale-in of workers with minimal impact on executing function invocations. Drain mode is already implemented and allowing instances to gracefully shutdown up to Functions timeout periods. The platform has been updated and this feature is available for Windows and Linux on Consumption and Premium plans. Some notes:

• Only new Windows Consumption apps created after May 2021 have the feature enabled. Windows Consumption function apps that were created before then are on a long term migration path.
• Existing and new Linux Consumption apps have the feature enabled
• Existing and new Windows and Linux Premium function apps have the feature enabled

Some optimization that doesn't affect the exiting Drain Mode feature remains during scale-out operations, and this issue will keep track of this work. It will be used for communicating progress towards completion to finalize the work started via #5672.

[liliankasem]

Tagging related issues:

Host

• Consume the drain mode api exposed by the webjobs sdk #6268
• Handle SigTerm with grace period to shutdown when running in Kubernetes #7332
• Add Inflight invocations completed status on the host #7188
• Design work for supporting resume API for drain mode #7673
• [Resume API] Investigate restart API feasbility #7781
• [Resume API] Implement resume API endpoint in the functions host #7774
• Service resolution exceptions for IDrainModeManager #7792
• Improve cancellation token experience for drain and shut down #7785
• Drain mode cleanup: remove GET, return 202 Accepted #7876

WebJobs SDK

• Implement an api to put host in drain mode. azure-webjobs-sdk#2527
• DrainModeManager is unnecessary azure-webjobs-sdk#2776
• Scale-in operation (e.g. DrainMode) on function host instances should not cancel in-flight function invocations azure-webjobs-sdk#2782

Extensions

• Connection is closed immediately when StopAsync is called on IListener  azure-functions-servicebus-extension#95
• Calling StopAsync on the Ilistener when a function execution is inflight causes the thread to hang azure-functions-eventhubs-extension#69

[swettstein]

How long is the long term migration for older Windows Consumption apps?

[nzthiago]

How long is the long term migration for older Windows Consumption apps?

It is a long term platform update with no specific end date. If you have an existing Windows Consumption app created before May 2021 and notice it is not allowing for a graceful shut down you would have to move your function app code to a new function app.

[andrewb-iq]

How long is the long term migration for older Windows Consumption apps?

It is a long term platform update with no specific end date. If you have an existing Windows Consumption app created before May 2021 and notice it is not allowing for a graceful shut down you would have to move your function app code to a new function app.

Could you explain why this is necessary? As an app developer, my first reaction is that it seems like a fix to the Azure Functions runtime could address this "behind the scenes" without the need to delete and recreate the app. Especially so when other types of apps did not require the same action in order to be fixed.

[nzthiago]

Could you explain why this is necessary? As an app developer, my first reaction is that it seems like a fix to the Azure Functions runtime could address this "behind the scenes" without the need to delete and recreate the app. Especially so when other types of apps did not require the same action in order to be fixed.

@andrewb-iq Agreed, and it is absolutely a fix being addressed behind the scenes, so over time all Windows Consumption apps will have drain mode enabled even if they were created pre-May 2021. But the upgrade process for those (pre May 2021) is long term, so that was just a workaround in case you want to be 100% sure. If you notice your app is being scaled in and your function executions are not being allowed to run for their full length, please raise a support ticket so we can help investigate.

[swettstein]

is there a way to verify if a particular function app has the new drain mode enabled? in case of any issues, i would like to be able to verify the drain mode status before i resort to deleting the app.

[liliankasem]

@swettstein we do have a drain status API you can hit:

https://myfuncapp.azurewebsites.net/admin/host/drain/status?code=<master-key>

[kelps]

https://myfuncapp.azurewebsites.net/admin/host/drain/status?code=<master-key>

@liliankasem I have an Azure Function create last month (Feb-2022) and drain mode is disabled (I just confirmed using the status API. What can I do?

this is the response I got by querying the drain status API

And how long until the app is migrated? I already noticed lost messages from Service Bus because the host was being scaled. The messages went to DLQ without ever hitting my ServiceBusTrigger function. I had to manually remove it from DQL and re-queue the messages. I found an issue #5240 with a problem similar to mine and I also filed a new issue (#8223) since the previous was closed.

[liliankasem]

https://myfuncapp.azurewebsites.net/admin/host/drain/status?code=<master-key>

@liliankasem I have an Azure Function create last month (Feb-2022) and drain mode is disabled (I just confirmed using the status API. What can I do?

this is the response I got by querying the drain status API

And how long until the app is migrated? I already noticed lost messages from Service Bus because the host was being scaled. The messages went to DLQ without ever hitting my ServiceBusTrigger function. I had to manually remove it from DQL and re-queue the messages. I found an issue #5240 with a problem similar to mine and I also filed a new issue (#8223) since the previous was closed.

Drain mode "disabled" in the context of the drain/status API isn't about if it is enabled or disabled for your application, but rather if has been "invoked" or not. So here if it's disabled, it means drain mode is not currently "active". The terminology is confusing. We might want to start using different terms for drain mode being enabled (enabled from platform) vs drain mode activated (function being put in drain mode by calling /drain mode API).

I don't think there is a public way of checking if you application has drain mode enabled for the platform. But if your application was created in Feb 2022 then it should have drain mode.

@nzthiago @soninaren for their input

[nzthiago]

is there a way to verify if a particular function app has the new drain mode enabled? in case of any issues, i would like to be able to verify the drain mode status before i resort to deleting the app.

Apologies for not replying earlier. If your Windows Consumption apps were created before May 2021, you can raise a support call to check or I'd be happy to follow up with you as well to help double check your apps.

[kelps]

is there a way to verify if a particular function app has the new drain mode enabled? in case of any issues, i would like to be able to verify the drain mode status before i resort to deleting the app.

Apologies for not replying earlier. If your Windows Consumption apps were created before May 2021, you can raise a support call to check or I'd be happy to follow up with you as well to help double check your apps.

I'm interest in this as well. Please let me know how we could check our apps. I'm starting to put them in production now and I already noticed this happening 3 times this month. It isn't much, but since we are using .NET6 in isolated mode, I don't have full control over the Service Bus message, so it goes to DLQ and could cause impact on our billing services.

[nzthiago]

is there a way to verify if a particular function app has the new drain mode enabled? in case of any issues, i would like to be able to verify the drain mode status before i resort to deleting the app.

Apologies for not replying earlier. If your Windows Consumption apps were created before May 2021, you can raise a support call to check or I'd be happy to follow up with you as well to help double check your apps.

I'm interest in this as well. Please let me know how we could check our apps. I'm starting to put them in production now and I already noticed this happening 3 times this month. It isn't much, but since we are using .NET6 in isolated mode, I don't have full control over the Service Bus message, so it goes to DLQ and could cause impact on our billing services.

Drain mode gets involved in scale in operations to ensure your currently executing Functions are allowed to finish executing before the workers are removed. Although your apps would be benefitting from drain mode, given they have been recently created, from what you are describing where messages don't even get processed I'd recommend raising a support ticket as it might not be related to Drain Mode per se.

[nzthiago]

As per above, Drain Mode is implemented through the platform. Limited documentation on Scale In behavior has been added to this documentation page.
Closing this overall tracking issue for Functions Host Updates for Drain Mode. Remaining items will be tracked separately, please raise a support ticket or a new GitHub issue if you have any scale in behavior issues.