Dotnet-Isolated Durable Functions On M1 Mac Breaking Function App

[Ben52]

I'm in the process of upgrading a sizealbe function app from the out of process model to the dotnet-isolated model. After doing all the upgrades, running any function would fail with this error:

A host error has occurred during startup operation 'xxx-xxxx-xxxx-xxxx'.
[2023-02-17T16:33:49.933Z] Grpc.Core: Error loading native library.
 Not found in any of the possible locations: [...]/bin/output/.azurefunctions/libgrpc_csharp_ext.arm64.dylib,[...]/bin/output/.azurefunctions/runtimes/osx-arm64/native/libgrpc_csharp_ext.arm64.dylib,[...]/bin/output/.azurefunctions/../../runtimes/osx-arm64/native/libgrpc_csharp_ext.arm64.dylib.
Value cannot be null. (Parameter 'provider')

After using the process of elimination to track down what was causing the error, I reached the conclusion that having a Durable function in the app would cause running any function in the app to fail. When I removed all the durable functions, the app ran fine. When i added just a minimal durable function as follows:

    [Function(nameof(MyActivity))]
    public async Task<string> MyActivity([ActivityTrigger] string input)
    {
        Console.WriteLine("doing some hard work...");
        await Task.Delay(1000);
        return "xxxx";
    }

    [Function(nameof(MyOrchestration))]
    public async Task MyOrchestration([OrchestrationTrigger] TaskOrchestrationContext context, string input)
    {
		await context.CallActivityAsync<string>(nameof(MyActivity), input);
    }

the app failed again with the same error.
I'm running this in an M1 MacBook Pro running Ventura 13.2, using .net 7 and Azure Functions V4 dotnet-isolated model.
Here are some relevant parts of my .csproj:

<Project Sdk="Microsoft.NET.Sdk">
	<PropertyGroup>
	        <TargetFramework>net7.0</TargetFramework>
	        <AzureFunctionsVersion>v4</AzureFunctionsVersion>
	        <OutputType>Exe</OutputType>
	        <ImplicitUsings>enable</ImplicitUsings>
	        <Nullable>enable</Nullable>
	    </PropertyGroup>
	<ItemGroup>
        <PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.10.0" />
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.ServiceBus" Version="5.7.0" />
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Timer" Version="4.1.0" />
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.0.13" />
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues" Version="5.0.0" />
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Storage.Blobs" Version="5.0.1" />
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.EventGrid" Version="3.2.1" />
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.0.0" />
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.DurableTask" Version="1.0.0" />
        <PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.7.0" OutputItemType="Analyzer" />
        <PackageReference Include="Microsoft.DurableTask.Generators" Version="1.0.0-preview.1" />
	<ItemGroup>

[cgillum]

I suspect this may be caused by a ARM64-incompatible gRPC library that is used by the Microsoft.Azure.WebJobs.Extensions.DurableTask package, which gets loaded by the Functions host. We've made fixes for this in other contexts, but I don't think we've yet made the fix in the WebJobs extension.

/cc @jviau

[Ben52]

@cgillum any timeline on when and if it will be fixed, or if there's any other work around? Its currently holding up a pretty critical upgrade.

[cgillum]

It might take some time since the primary developer with the most context just left for vacation. That said, it's an urgent enough issue that I think we'd be able to get to it by no later than next month. Just a guess though.

As for workarounds, are you able to do local development in x86_64 emulation mode (e.g., using Rosetta)?

[Ben52]

I downloaded and ran the app using the x64 version, and I'm getting the same error

[Ben52]

I'm also getting the same error when running in Windows ARM through Parallels.

[Ben52]

Is there any update on this?

[davidmrdavid]

@jviau: could you take a look?

[jviau]

See this issue for a workaround: microsoft/durabletask-dotnet#93 - you need to reference Contrib.Grpc.Core.M1 package and also have a copy command as part of your build to bring the libgrpc_csharp_ext.arm64.dylib from that package into your build output.

A proper fix for this is going to take a bit, as we need to move from Grpc.Core to Grpc.AspNetCore - which the functions host already uses, but we will need to investigate the best route for adding extension gRPC services. We are looking into the host having an official prescribed way for extensions to do this.

[davidmrdavid]

Please see this issue for explicit workaround instructions.