Durable Functions Support for Python

[asavaritayal]

New Feature - Looking for votes and/or user input to gauge traction.

[priyaananthasankar]

+1 Absolutely needed. Working on a small sample with asyncio to simulate something similar.

[gled4er]

+1

[yokawasa]

+1 Yes needed! Python has many choice of data science libraries and it definitely has long running & stateful scenario!

[tjhgit]

I am experimenting with triggering an azure batch job from an azure function. So this is completely asynchronous and you can also scale up the compute ressources easily in azure batch.

[mataralhawiti]

+1 Absolutely

[stereobutter]

+1 for long running analytics/ML jobs

[crgarcia12]

+1

[Sarah-Aly]

This is absolutely a needed feature. I do have a customer who uses Python in Azure Functions for manipulating CSVs & creating time series with Pandas and they are an ideal case for function chaining in DF.

[t-eckert]

+1 Happy to contribute

[asavaritayal]

/cc @cgillum @kashimiz as FYI

[nroypf]

+1 Need it asap :)

[dajor]

+1

[FlippieCoetser]

currently running large data transformations with Python. Execution time is around 30 min. Having durable functions will help a lot!

[ericdrobinson]

I'm saddened to see that Python support for Durable Functions isn't available yet. I've chronicled my efforts to make my usage of the HTTPTrigger more effective/efficient for my workflow in the comments of issue #236. The documentation, however, keeps suggesting that I take the Durable Functions route for a more reasonable experience.

Unfortunately, it looks as though I'll need to re-implement Durable Functions manually if I want to gain the suggested benefits of that approach :(

+1, indeed.

[explora21]

+1, of course

[SpicySyntax]

+1, This would be very useful for long running ML Workloads

[reynoldsa]

Absolutely +1.

[bigdatamoore]

+1 agreed. We could use this now.

[jaryder]

+1 - Is there a timeline for when this will be available?

[svartkanin]

+1 - Definitely needed

[whataride]

+1 - Absolutely needed!

[mattc-eostar]

@tjhgit

I am experimenting with triggering an azure batch job from an azure function. So this is completely asynchronous and you can also scale up the compute ressources easily in azure batch.

@priyaananthasankar

+1 Absolutely needed. Working on a small sample with asyncio to simulate something similar.

Any luck in these adventures?

[Zieg]

@mattc-eostar not really. Depends on how you trigger your function. If you use http trigger - then you are limited by 4 minutes timeout. Function should execute and return output within this limit, otherwise it fails. And if you want to trigger your function from Data Factory (for example) - you are pretty much limited to http triggers.

[mattc-eostar]

@mattc-eostar not really. Depends on how you trigger your function. If you use http trigger - then you are limited by 4 minutes timeout. Function should execute and return output within this limit, otherwise it fails. And if you want to trigger your function from Data Factory (for example) - you are pretty much limited to http triggers.

Is this true even for functions hosted on premium plan? They advertise no limit on execution time.

[anthonychu]

HTTP triggers have a 230 second timeout for how long the request can stay open. If you need to use the max function duration (up to 10 mins consumption, 60 mins Premium, "unlimited" on other plans), the HTTP function can drop a message on a queue that a longer-running queue triggered function can process. Durable Functions does this behind the scenes, and adds the ability to checkpoint progress and aggregate results.

We're targeting early 2020 for a preview of Durable Functions for Python.

[mattc-eostar]

I see what you are saying. Yes. I was operating under the assumption my use case would be using queues regardless. And great to hear. Look forward to working with that.

[gbonventre]

I need Async API and Long-running functions. Also long-running functions that can do fan in and fan out from a queue would be perfect for many of the ETL applications my employer is currently working on.

[markl11]

"We're targeting early 2020 for a preview of Durable Functions for Python." - given that quite a few folks can't put preview features into production, when are you actually thinking that Durable Functions for Python will be GA? (frustratingly, preview features in Azure can be preview for a looooong time in my experience so I hope this will be different)

[cgillum]

@markl11 we're targeting the first half of 2020 for the GA release - i.e. no later than June, assuming things go relatively smoothly with the preview.

[WrittenLabyrinth]

+1 working on a prototype app in python and a durable function in Azure would be awesome!

[CalebHui]

+1

[anthonychu]

Just a quick update as we've been hearing from a lot of folks who can't wait to try this. Python Durable Functions is currently on track for a preview in a month or two. It'll support the core 5 workflow patterns, with the exception of durable entities.

[anthonychu]

Reopening this as we're using this issue to track progress of Python Durable Functions support.

[LouisMach]

@anthonychu any updates on when the preview will be released? Maybe a tracking page?

[anthonychu]

Waiting for some bits to be released in Core Tools and deployed to the service in Azure. Could have something you can try running locally in the next week or two.

[LouisMach]

That's really cool! I'm planning to use durable functions to host a tensorflow model for computer vision, or do you think I should host it in AzureML and call it from a function instead?

[MattConflitti]

Try it and see! Python durable functions is going to open up a lot of doors given the orchestration capabilities. From smaller scale data pipelining to serving simpler models, it will be exciting to see how it stacks up in usability and pricing. Both of which are suspected to be great.

[anthonychu]

Hey everyone. Thanks for your patience. You can now try out Durable Functions in Python here:
https://github.com/azure/azure-functions-durable-python 🎉

There are a few pieces to release in tooling and deploy to our service before we can announce a public preview. For now, check out the readme for the current limitations and instructions on how you can try it out today.

We look forward to getting some early feedback. Please create issues here. Thanks!

[wasuaje]

+1 Here for needing this !

Thanks for the preview

[anthonychu]

Python Durable Functions is now in public preview. 🎉 Thanks for your patience everybody. Please use https://github.com/Azure/azure-functions-durable-python for feedback.

Announcement: https://azure.microsoft.com/updates/durable-functions-now-supports-python/

[StevenBorg]

Congratulations!!!! This is great news. I've been forced to rewrite functions in C# just to get Durable Functions. The pain of switching from Python to C# was less than dealing with coordinating my own fan-out/fan-in. Thank you! Back to Python!

[priyaananthasankar]

That's really cool! I'm planning to use durable functions to host a tensorflow model for computer vision, or do you think I should host it in AzureML and call it from a function instead?

You can use AML to host or run ML batch processes and then use durable functions to orchestrate the pipeline through monitoring pattern. Check the samples in the repo for this use case :)

[Hazhzeng]

The Python Durable Functions is currently in Preview https://azure.microsoft.com/en-us/updates/durable-functions-now-supports-python/

Closing this thread as we're redirecting the related discussion to the new durable python repo https://github.com/Azure/azure-functions-durable-python

Thanks