Assembly Resolution in Azure Functions

One of the most common sources of feedback we receive relate to .NET assembly resolution and loading issues in Azure Functions. The goal of this post is to help .NET developers avoid those problems, understand the current limitations, provide guidance on how to avoid or mitigate some of those issues, when possible, and go over some of the details on how this will be improved in the next version of Azure Functions.

In the following sections, we dive into the details of how things work today

• Assembly resolution and loading
  • What are the challenges when running on Azure Functions?
• Assembly resolution in Azure Functions 1.0
  • Pre-compiled functions
  • Dynamically compiled code (C# Scripting)
  • What can be done if I have an assembly resolution problem?
• How will this be improved?
  • Better isolation
  • Reduction of runtime dependency conflicts
  • Out-of-proc worker
• Conclusion
• Frequently Asked Questions
  • Why can't you just allow me to override the runtime assemblies and specify my own redirects?

Assembly resolution and loading

When running a .NET application, the .NET runtime is responsible for assembly resolution and loading, and when needed, there are mechanisms in place (e.g. assembly binding redirect configuration, resolution callbacks, etc.) to enable developers to influence this process. Tooling is often helping us with those tasks and performing some of this configuration work automatically for us.

Things change a bit when running in a hosted environment, such as in Azure Functions, where your assemblies are being loaded into another host's process. Some of the previously mentioned mechanisms are still available to developers, but there are limitations that will vary depending on how your assemblies are loaded by that host.

In addition to processing trigger events, bindings, compilation, dispatching and other key tasks, the Azure Functions runtime acts as the hosting environment for .NET functions, and because of limitations with the underlying frameworks and infrastructure, today, the function assemblies (and their dependencies) are loaded in the same process and application domain as the runtime, which may lead to issues in scenarios where your function dependencies conflict with runtime dependencies, particularly when your function directly interacts with the runtime, usually through a binding type exchange.

What are the challenges when running on Azure Functions?

Normally, when developing .NET solutions, one way to attempt to resolve assembly conflicts is by using assembly binding redirects, which are defined in the application's config file (Web.Config in the case of an ASP.NET web application) and perform assembly version unification. However, the Azure Functions runtime is a multi-tenant component that is deployed and managed by the App Service team, so we are unable to modify the runtime's (the Application) config file and define those binding redirect entries, and since the same application domain is used, the option to provide a config file for it is not available.

It’s important to emphasize that the Azure Functions runtime has special logic in place to resolve dependencies and conflicts, and many scenarios that would traditionally require binding redirect configurations (or would just plainly not work) in a .NET application are possible in Azure Functions today – even when using dependencies with versions that may conflict with the host. For example, if I want to write a function that has a direct or transitive dependency on Json.NET v11, and the host uses Json.NET v10, you can still pull in v11 and use it in your code. When your function is executed, we will honor the dependency versions that you have referenced. However, there can be conflicts if you try to transfer rich types between the host and the function execution.

To provide concrete examples, here are some of assembly resolution and loading issues you may run into when using Azure Functions today:

• You are using an Azure Functions binding and have an argument using JObject, which is a JSON.NET type, and you use have a dependency that uses a newer version of the library than the one supported by the runtime

  • For pre-compiled functions (e.g. Visual Studio), due to strict package version restrictions put in place in the Functions SDK package, you'll run into errors trying to install your dependency
  • You can override the restriction by adding an explicit reference to the JSON.NET version you need to use, which gets around the strict version requirement, but you'll find that, at runtime, your binding may fail (some bindings are handle this scenario) to recognize that type, since without unification, their view of JObject differ from your function assemblies, resulting in an error where the extension is unable to bind to that parameter (you will see an error stating that the type you've used is not supported).

• You are referencing a package that requires a binding redirect to address a version mismatch with one of its assemblies. As previously stated, binding redirects are not supported, so this scenario would fail.

To help show examples of this and scenarios that work and ones that may be blocked we have created this GitHub repo of samples, which include the ones mentioned above. We welcome contributions, and this a great starting point to see how you can work around assembly version conflicts in Azure Functions today.

Assembly resolution in Azure Functions 1.0

For those who want to better understand the resolution process currently in place, this section goes into the nitty gritty of assembly resolution in Azure Functions 1.0, which differs between the pre-compiled and CSX models.

Pre-compiled functions

Pre-compiled functions are deployed with their dependency closure and loaded in the load-from context. Most of the dependencies will be automatically loaded by the loader (the CLR/.NET loader), and the Azure Functions runtime will handle binding failures and attempt to load the appropriate dependencies when the loader is unable to do so.

Dynamically compiled code (C# Scripting)

Dynamically compiled code is handled a little differently. C# scripting functions are compiled into an in-memory assembly, which is not persisted to disk. That assembly is then loaded without a traditional .NET load context, and the runtime is responsible for all dependency resolution.

In order to resolve dependencies, the runtime maintains a function assembly load context, that has a dependency manifest, built during the compilation process. This manifest is used during the resolution process to locate and bind to the appropriate assemblies when they are requested.

When building the dependency manifest, the runtime does maintain some additional information about the type of dependency, which could be one of the following:

• Private reference: A private assembly reference is scoped to a function and deployed in the function's bin folder. You'd reference those assemblies using a #r directive with a simple file name (e.g. #r MyAssembly.dll). These dependencies are loaded with no context.

• Shared reference: A shared assembly reference is scoped to the Function App and shared across all functions referencing it (a single copy is loaded). You typically reference those assemblies with the #r directive, pointing to a relative path. These assemblies are loaded in the load-from context

• NuGet package assemblies: These are the assemblies coming from package references you have added to your function (using the project.json file). These assemblies are loaded with no context.

The runtime uses the function assembly load context when resolving dependencies to appropriately load the assemblies references using one of the above methods.

What can be done if I have an assembly resolution problem?

It’s important to recognize that, because of the process described above, many scenarios are possible in Azure Functions today – even using dependencies with versions that may conflict with the host. For example, if I want to write a function that has a direct or transitive dependency on Json.NET 11.x, and the host uses Json.NET 9.0.1, you can still pull in version 11.x and use it in your code. When your function is executed, we will honor the dependency versions that you have pulled in. However there can be conflicts if you try to transfer rich types between the host and the function execution.

Using the Json.NET example, if as the trigger input I specify to receive a JObject and also pull in v11, an error will likely occur at runtime because the host process is using JObject from 9.0.1, and when it tries to send to a process expecting JObject v11 there may be a data model or other conflict. To get around this though, I could simply change the trigger input from JObject to a more native type like string. Now a string will be sent into function, and inside the function I could do JObject.Parse() to parse and create a v11 JObject that works.

How will this be improved?

In the next version of the runtime, Azure Functions 2.0, a significant amount of work is being put into improving assembly resolution and loading issues, some of the current plans include:

• Better isolation
• Reduction of runtime dependency conflicts
• Out-of-proc worker

Better isolation

The next version of the Azure Functions runtime will load extensions, functions and their dependencies in custom assembly load contexts, isolated from the core runtime components. This effectively means that, aside from a core set of assemblies that are shared between the runtime and the function context (e.g. framework assemblies, core WebJobs SDK assemblies, core abstraction assemblies), the assemblies referenced and deployed with your functions will be loaded, and this includes the ability to unify extension dependencies with your function references. One example of that is having the ability to use a different version of JSON.NET or the Azure Storage SDK and still be able to bind to types coming from those libraries, as the extensions will be bound to the versions you've deployed. This will all happen without the need to explicit configure binding redirects. The goal of much of this work is to address the root cause of the problem, and not just to enable a workaround.

This also enables better probing and resolution logic in the runtime, as we no longer rely on the .NET infrastructure for that, giving us the ability to more effectively implement enhancements around assembly matching, runtime specific (Windows, Linux, etc.) dependencies, runtime binding redirection, and use of deps files. When combined with the isolation changes described above, we expect that a significant number of issues, particularly when assembly load failures you'd see manifested as FileNotFoundException errors, even when the assembly was present in the function's directory, will be addressed.

We plan to release these capabilities with a preview update sometime in May

Reduction of runtime dependency conflicts

Many of the conflicts we see today will be avoided by minimizing the number of dependencies we have out of the box in the runtime. This is being done in two ways:

• Core runtime enhancements: Some external dependencies are being removed from the runtime.
• Better extension management: The current version is deployed with every binding extension we support, while in 2.0, users will enable the extensions they need to use, and only those extensions will be deployed by the runtime. Not only this reduces the number of dependencies (and potential conflicts), but also gives us the ability to change dependencies versions in new extension versions as that would not impact existing applications.

These changes are already present in the current Azure Functions, but being expanded with every update.

Out-of-proc worker

For the cases where the above is insufficient (for example, if you need to use a version of a restricted assembly, go outside of restricted assembly version upper limits, or even go as far as using a different version of .NET), we also plan to enable an out-of-proc model, where customers will “own” the hosting environment their functions is running under.

With this model, binding support (at least binding to richer types, like CloudBlockBlob, DocumentClient and others) would be limited at first.

This would take advantage of the language extensibility features that have been added to the runtime in 2.0, and are being used by the Node.js, Java and the upcoming Python language extensions, and wouldn't be something new added specifically for .NET to unblock these scenarios.

We are not committing to this feature for the Azure Functions 2.0 GA, but expect to have it in a preview state then, or shortly after.

Conclusion

We are working hard to ensure the .NET experience is significantly improved in the upcoming version of the Azure Functions runtime, 2.0, not only by enhancing existing functionality, but by introducing new options to unblock advanced scenarios, giving .NET developers full control over their dependencies.

We appreciate all the assembly resolution and loading feedback we have received. A lot of what is described here is driven by what we've heard from you.

Please don't hesitate to reach out with questions or feedback about the information shared here. To do so, we recommend using this GitHub issue, as it has a lot of information and additional context about the problems covered here.

Frequently Asked Questions

Why can't you just allow me to override the runtime assemblies and specify my own redirects?

As a managed service, we need to ensure that customers are not impacted by service updates, which would be a real possibility if we are unable to validate those updates against a known set of dependencies. While patch releases and minor versions are, in theory safe, the ability to redirect to an arbitrary version of a given runtime dependency creates a risky situation where a breaking change could impact the way the runtime operates and lead to difficult to debug issues. To make matters worse, a given breaking change may not be hit until an update that uses that code path is deployed, breaking an application that was successfully running in production