Helper class for dealing with native shared libraries and function pointers

[mellinoe]

Approved Proposal

From https://github.com/dotnet/corefx/issues/17135#issuecomment-365364216

public sealed class NativeLibrary
{
    public static bool TryLoad(string name, Assembly caller, DllImportSearchPaths paths, out NativeLibrary result);

    public IntPtr Handle { get; }

    public bool TryGetDelegate<TDelegate>(string name, out TDelegate result) where TDelegate : class;
    public bool TryGetDelegate<TDelegate>(string name, bool exactSpelling, out TDelegate result) where TDelegate : class;
    public bool TryGetSymbolAddress(string symbolName, out IntPtr result);
}

Original proposal

Note: Proposal updated based on discussion: https://github.com/dotnet/corefx/issues/17135#issuecomment-353571556

Background

Many popular C-callable shared libraries exist which do not have a consistent name across platforms. Examples include:

• SDL2
• OpenAL
• Vulkan
• Many more (the above are just ones I've used personally)

CoreCLR does not support any notion of "remappable" PInvokes, a la DllImport in Mono. The name of the shared library must be encoded directly in the DllImport attribute, which makes it impossible to create a single wrapper assembly which works on many platforms. This imposes an unnecessary development and deployment burden on library developers who could otherwise ship a single, simple DLL. Many third-party libraries which rely on PInvoke's only work on Mono platforms because of the DllMap feature. These libraries do not work on .NET Core at all, even if they are otherwise completely compatible with its profile.

The alternative to using [DllImport] is to write logic which manually opens the shared library, discovers function pointers, and converts them to managed delegates. In a sense, this very similar to what we are doing with our "Portable Linux" version of the runtime and native shims. However, this code can be complicated, tedious, and error-prone. This proposal is for a small helper library, with optional cooperation by the runtime, to make this scenarion easier.

Proposed API

namespace System.Runtime.InteropServices
{
    // Exposes functionality for loading native shared libraries and function pointers.
    public class NativeLibrary : IDisposable
    {
        // The default search paths
        public static IEnumerable<string> NativeLibrarySearchDirectories { get; }

        // The operating system handle of the loaded library.
        public IntPtr Handle { get; }

        // Constructs a new NativeLibrary using the platform's default library loader.
        public NativeLibrary(string name);
        public NativeLibrary(string name, DllImportSearchPath paths);
        public static NativeLibrary Open(string name, DllImportSearchPath paths, bool throwOnError);

        // Loads a native function pointer by name.
        public IntPtr LoadFunction(string name);

        // Loads a function whose signature matches the given delegate type's signature.
        // This is roughly equivalent to calling LoadFunction + Marshal.GetDelegateForFunctionPointer
        public T LoadDelegate<T>(string name);

        // Lookup a symbol (not a function) from the specified dynamic library
        public IntPtr SymbolAddress(string name);

        // Frees the native library. Function pointers retrieved from this library will be void.
        public void Dispose();
    }
}

Usage

public static class Sdl2
{
    private static NativeLibrary s_sdl2NativeLib = new NativeLibrary(GetSdl2LibraryName());
    // Determine library name at runtime. This could be influenced by a DllMap-like policy if desired.
    private static string GetSdl2LibraryName();

    private delegate IntPtr SDL_CreateWindow_d(string title, int x, int y, int w, int h, int flags);
    private static SDL_CreateWindow_d SDL_CreateWindow_ptr = s_sdl2NativeLib.LoadFunctionPointer<SDL_CreateWindow_d>("SDL_CreateWindow");
    public static IntPtr CreateWindow(string title, int x, int y, int w, int h, uint flags)
        => SDL_CreateWindow_ptr(title, x, y, w, h, flags);
}

public static void Main()
{
    IntPtr window = Sdl2.CreateWindow("WindowTitle", 100, 100, 960, 540, 0);
}

Open Design Questions

Probing Path Logic

CoreCLR applies particular probing logic when discovering shared libraries listed in a DllImport. Ideally, calling new NativeLibrary("lib") would follow the same probing logic as [DllImport("lib")], so that PInvoke's can be easily converted. This could potentially be solved with some internal or public runtime helper function exposed from System.Private.CoreLib. Alternatively, the constructor for NativeLibrary` could include another parameter which controlled probing logic, allowing someone to plug their own logic in.

Generic delegate types

The simplest way to convert from a native function pointer to a clean managed delegate is through the use of Marshal.GetDelegateForFunctionPointer<T>(IntPtr). Unfortunately, this cannot be used with generic delegates, e.g. Func<string, int, int, int, int, int, IntPtr> in the above example. Given that many native function signatures include pointer types, which cannot be used as generic type arguments, other than as IntPtr, this is not a major problem. However, many cases would benefit from being able to use Action and Func types in the LoadFunction<T> method. We could probably allow this using Reflection.Emit, but it would most likely be ugly, complicated, and slower than if custom delegate types were used. If Marshal.GetDelegateForFunctionPointer<T>(IntPtr) accepted generic delegate types, this feature would greatly benefit.

Disposability and lifetime tracking

The constructor of NativeLibrary involves opening an operating system handle to a native shared library, via LoadLibrary, dlopen, etc. I have proposed that NativeLibrary be disposable, which would involve calling FreeLibrary, dlclose, etc. Is this desirable or useful?

A related issue is how the runtime currently tracks and manages these handles. Should handles opened by NativeLibrary be coordinated and tracked in the same way that other handles (via PInvoke, etc.) are?

Prototype

I have a prototype version of this library implemented here: https://github.com/mellinoe/nativelibraryloader, and I have successfully used the pattern described here in a few projects.

@yizhang82 @janvorli @danmosemsft @conniey

[danmoseley]

You've eliminated the option of mimicing Mono's solution (DllMaps, as you mention)?
Would this API ultimately work for Mono, making DllMap unnecessary?

[akoeplinger]

Related discussion (which you're probably aware of): https://github.com/dotnet/coreclr/issues/930

[mellinoe]

@danmosemsft See the link that @akoeplinger shared.

[akoeplinger]

@mellinoe since my thread is pretty old by now, do you know what the current status of MCG (which was proposed as the solution by .NET runtime folks) is?

[mellinoe]

I don't know. I'd be interested to hear an update from @yizhang82 about it.

[IS4Code]

Nobody uses unsafe pointers in extern methods, only IntPtr or managed equivalents. Usage of generic delegates should be allowed in GetDelegateForFunctionPointer, and frankly I don't see what difficulties would implementing it have. On the other hand, I think that the "LoadFunction" doesn't allow for as much marshalling options as the DllImport system can. What if you could specify "instance extern" methods on types inheriting from NativeLibrary, which would be automatically imported from the library with the specified marshalling options?

[mellinoe]

Nobody uses unsafe pointers in extern methods, only IntPtr or managed equivalents.

It is very common to use unsafe pointers in extern methods, in my experience. I certainly do it in a lot of my libraries, and I've seen it in many libraries from others, as well. Wrapping things into IntPtr's is unnecessarily verbose when you're already doing fundamentally unsafe things. Now, it's also very common to "wrap" the PInvokes into methods will fully-safe signatures, but those call the unsafe version in turn.

What if you could specify "instance extern" methods on types inheriting from NativeLibrary, which would be automatically imported from the library with the specified marshalling options?

That sounds like it would involve a lot of external machinery (IL rewriting), or actual C# language support. Certainly an interesting idea, but probably outside of scope here.

@yizhang82 Do you have any feedback for this proposal?

[akoeplinger]

Another problem with the proposed API is that calling dlopen() with arbitrary native libraries is prohibited on iOS and recent Androids so this wouldn't work there (i.e. unusable by Xamarin).

[mellinoe]

Another problem with the proposed API is that calling dlopen() with arbitrary native libraries is prohibited on iOS and recent Androids so this wouldn't work there (i.e. unusable by Xamarin).

That is a good point, and it may also be the case for UWP applications.

[qmfrederik]

Just my 2 cents on this:

• On operating systems where you cannot load a native library, this could throw a PlatformNotSupportedException, right? (Although I believe that at least on iOS, you can load native libraries which are part of your application bundle.)
• I like the idea to keep a dll-map like functionality separate from this API. It makes sense to me that this is a lowel-level API which is close to dlopen/LoadLibrary where you can pass an explicit, full path to the library you want to load. Nothing prevents you from implementing dllmap-like functionality in a separate API, which you can use to resolve the path to the library you want to load
• I'd suggest adding a static property NativeLibrarySearchDirectories which exposes AppContext.GetData("NATIVE_DLL_SEARCH_DIRECTORIES"), hence:

namespace System.Runtime.InteropServices
{
    // Exposes functionality for loading native shared libraries and function pointers.
    public class NativeLibrary : IDisposable
    {
+       // The default search paths
+       public string[] NativeLibrarySearchDirectories { get; }

        // The operating system handle of the loaded library.
        public IntPtr Handle { get; }

        // Constructs a new NativeLibrary using the platform's default library loader.
        public NativeLibrary(string name);

        // Loads a native function pointer by name.
        public IntPtr LoadFunction(string name);

        // Loads a function whose signature matches the given delegate type's signature.
        // This is roughly equivalent to calling LoadFunction + Marshal.GetDelegateForFunctionPointer
        public T LoadDelegate<T>(string name);

        // Frees the native library. Function pointers retrieved from this library will be void.
        public void Dispose();
    }
}

[migueldeicaza]

This wrapper looks good, additionally, I would add the following API, that would lookup a symbol (not a function) from the specified dynamic library:

IntPtr SymbolAddress (string name);

This would be useful to access global variables and other global symbols in the loaded library.

[tmds]

@mellinoe can this be used to perform lookups using dlvsym instead of dlsym on Linux? dlvsym is a Linux specific mechanism that allows symbols to be versioned. It allows a library to provide different versions of the same symbol. The dlsym lookup (that is used in coreclr) always gives you the oldest version of the symbol. See https://github.com/dotnet/coreclr/issues/8721 for more info.

[qmfrederik]

@tdms Looking at https://linux.die.net/man/3/dlvsym and https://www.gnu.org/software/gnulib/manual/html_node/dlvsym.html, dlvsym is a glibc-extension. This means it's not available on non-glibc Linux, BSD, macOS and Windows (GetProcAddress doesn't seem to take a version parameter, either).

I'm wondering whether it makes sense to add support for a glibc-extension in the core API?

[tmds]

I'm wondering whether it makes sense to add support for a glibc-extension in the core API?

I'm not requesting dlvsym to be baked in .NET Core. I tried this with https://github.com/dotnet/coreclr/issues/8721 (+PR) and it was not accepted.
I got feedback this should be done via some extension mechanism. I believe this proposal could fit that use-case.

[qmfrederik]

@tmds Well with this API you should be able to do something like this, right:

using(var dl = new NativeLibrary("dl"))
{
    var dlvsym = dl.LoadDelegate<dlvsym_delegate>("dlvsym");
}

I do hope that the NativeLibrary API will make it into .NET core, though ;-)

[tmds]

I think in this proposal, I'm looking to override LoadFunction. So e.g. LoadFunction("foo@VERS_1.1") calls dlvsym(Handle, "foo", "VERS1_1");.

[tmds]

The constructor of NativeLibrary involves opening an operating system handle to a native shared library, via LoadLibrary, dlopen, etc. I have proposed that NativeLibrary be disposable, which would involve calling FreeLibrary, dlclose, etc. Is this desirable or useful?

What happens when someone uses a delegate after disposing the library?

A related issue is how the runtime currently tracks and manages these handles. Should handles opened by NativeLibrary be coordinated and tracked in the same way that other handles (via PInvoke, etc.) are?

Are libraries used via PInvoke ever closed? When?

[ghost]

What happens when someone uses a delegate after disposing the library?

simple ObjectDisposedException?

[qmfrederik]

Adding the area owners @russellhadley @luqun @shrah to this thread.

One of the issues we're trying to solve is how managed code can load native libraries in a cross-platform way.

For example, System.Drawing.Common depends on libgdiplus which ships in various forms in various distros. So the code probes for a couple of files and loads the correctly library using dlopen.

One of the limitations we've hit is that dlopen also lives in separate libraries - libdl.so on most Unixes but sometimes also libdl.so.2 (CentOS) or libc (FreeBSD).

I've tried to address that in dotnet/corefx#25134 by adding dlopen to the PAL but the consensus on that issue was that System.Drawing.Common cannot call the PAL directly and any wrapper around dlopen needs to be exposed in a separate API which is part of corefx.

Hence this issue 😄 .

Can you give your feedback on this API proposal?

[jkotas]

We may need a constructor overload that takes DllImportSearchPath to control the probing paths.

[qmfrederik]

@jkotas Something like this?

namespace System.Runtime.InteropServices
{
    // Exposes functionality for loading native shared libraries and function pointers.
    public class NativeLibrary : IDisposable
    {
+       // The default search paths
+       public static IEnumerable<string> NativeLibrarySearchDirectories { get; }

        // Constructs a new NativeLibrary using the platform's default library loader.
        public NativeLibrary(string name);

+       public NativeLibrary(string name, DllImportSearchPath paths);

+       // Constructs a new NativeLibrary using user-provided search libraries.
+       public static NativeLibrary Open(string name, DllImportSearchPath paths, bool throwOnError)

        // The operating system handle of the loaded library.
        public IntPtr Handle { get; }

        // Loads a native function pointer by name.
        public IntPtr LoadFunction(string name);

+       // Lookup a symbol (not a function) from the specified dynamic library
+       public IntPtr SymbolAddress (string name);

        // Loads a function whose signature matches the given delegate type's signature.
        // This is roughly equivalent to calling LoadFunction + Marshal.GetDelegateForFunctionPointer
        public T LoadDelegate<T>(string name);

        // Frees the native library. Function pointers retrieved from this library will be void.
        public void Dispose();
    }
}

[jkotas]

I meant something like this:

public class NativeLibrary : IDisposable
{
    public NativeLibrary(string name);

+   public NativeLibrary(string name, DllImportSearchPath paths);

...
}

DllImportSearchPath is existing enum, used together with regular DllImport today.

[jkotas]

Also, to allow implementation custom probing logic like the one in https://github.com/dotnet/corefx/blob/35d0838c20965c526e05e119028dd7226084987c/src/System.Drawing.Common/src/System/Drawing/GdiplusNative.Unix.cs#L39 we may want to have non-throwing factory like:

public class NativeLibrary : IDisposable
{
+   public static NativeLibrary Open(string name, DllImportSearchPath paths, bool throwOnError);

...
}

[qmfrederik]

@jkotas Ah, that makes sense. I updated the API spec.

So the libgdiplus probing logic could be rewritten to something like this:

NativeLibrary libgdiplus = null;
string libraryName = null;

if (RuntimeInformation.IsOSPlatform(OSPlatform.OSX))
{
    libraryName = "libgdiplus.dylib";
    libgdiplus = NativeLibrary.Open("libgdiplus.dylib", DllImportSearchPath.SafeDirectories, throwOnError: false);
}
else
{
    // Various Unix package managers have chosen different names for the "libgdiplus" shared library.
    // The mono project, where libgdiplus originated, allowed both of the names below to be used, via
    // a global configuration setting. We prefer the "unversioned" shared object name, and fallback to
    // the name suffixed with ".0".
    libraryName = "libgdiplus.so";
    libgdiplus = NativeLibrary.Open("libgdiplus.so", DllImportSearchPath.SafeDirectories, throwOnError: false);
    if (libgdiplus == null)
    {
        libgdiplus = NativeLibrary.Open("libgdiplus.so.0", DllImportSearchPath.SafeDirectories, throwOnError: false);
    }
}

// If we couldn't find libgdiplus in the system search path, try to look for libgdiplus in the
// NuGet package folders. This matches the DllImport behavior.
if (libgdiplus ==null)
{
    libgdiplus = NativeLibrary.Open(libraryName, DllImportSearchPath.ApplicationDirectory,, throwOnError: false);
}

Assuming SafeDirectories and ApplicationDirectory are the DllImportSearchPath values to to express "search in system paths only" vs "search in NuGet packages"

[karelz]

@qmfrederik did you leave out @jkotas's constructor overload on purpose?

Also, should NativeLibrarySearchDirectories be IEnumerable<string> instead of string[] to prevent mutation of the array?

[joelday]

@karelz the interface alone doesn't really enforce immutability. There a standard pattern for this in corefx? I should read up on that. :)

[karelz]

I don't think there is a standard pattern documented. But in general I think we try to expose interfaces instead of underlying collections (at least I vaguely remember that being discussed in API reviews).
Evil users can cast to the underlying collection - that is not something we typically guard against. Breaking such users might not be considered a breaking change though.
Technically speaking we could create internal type that encapsulates the collection and exposes its items, but that is usually too heavy-weight for us to use.

[qmfrederik]

@karelz That was an oversight on my side, although having both a constructor and a static Open method may be a bit of overkill.

Good point on IEnumerable vs string[]. I changed that; it does help communicate the values are read-only.

[karelz]

Top post updated with latest proposal.
Naming-wise we should probably rename NativeLibrarySearchDirectories to DefaultSearchDirectories.

@jkotas can you mark it api-ready-for-review when you are ok with the API shape?

[jkotas]

My comments about API shape:

• How are LoadFunction and SymbolAddress different, except for their name? Is their implementation going to differ in any way?

• LoadFunction is not loading anything. It is just doing lookup in what was loaded already. Should it be rather called GetFunction?

• Add non-throwing variant of LoadFunction to allow probing for entrypoints that may not exist?

This is roughly equivalent to calling LoadFunction + Marshal.GetDelegateForFunctionPointer

Is it roughly equivalent, or exactly equivalent? It would be nice it LoadDelegate kept the delegates created by it alive so that people do not have to do it manually.

[qmfrederik]

For the common case where you develop and deploy your own native library (and you control the way the naming is performed), we should be able to write xplat code without having to make a if (platform_blabla) { use_stupid_extension_naming} else {...}

I agree, and the good news is you can already do so today - actually .NET Core has a pretty good story there.

If you control the naming of your native library, you can something like [DllImport("mylibrary")]| and that will automagically resolve to libmylibrary.so, libmylibrary.dylib and mylibrary.dll.

Since you control the naming, you also control the distribution, in which case you can ship your native libraries in a NuGet package and .NET Core will "do the right thing".

That's how I ship imobiledevice-net, for example; and ASP.NET ships libuv IIRC.

[xoofx]

If you control the naming of your native library, you can something like [DllImport("mylibrary")]| and that will automagically resolve to libmylibrary.so, libmylibrary.dylib and mylibrary.dll.

Yep, and that was my proposal just above (to make LoadLibrary acting similar to DllImport behavior) So you agree a bit then? 😅

[qmfrederik]

So you agree a bit then? 😅

Yes!

[chmorgan]

It sounds like there is almost consensus here, what are the next steps? I'm itching to use whatever the solution is to fix cross platform support with dotnet that previously worked with .net and mono.

[GrabYourPitchforks]

@chmorgan the next steps are for us to finish the existing PR for this feature when the checkin window opens back up.

[dotMorten]

What about the fact that on iOS callbacks must be static? That's where all my interop code diverges the most. For us to be able to create truly reusable interop code, these sort of differences needs to be addressed too.
Second when running on .NET Framework as AnyCPU we need to be able to point to two different DLLs (or 3 if we get ARM64 support) based on which architecture the app is running under.

[ghost]

https://docs.microsoft.com/en-us/nuget/create-packages/supporting-multiple-target-frameworks#architecture-specific-folders

we need to restore packages on each system

[dotMorten]

@kasper3 Good point. I really like the \runtime\[TFM]-[arch]\native\ approach that UWP uses. The other platforms need to support this as well - currently we have to create .targets files and either embed or xcopy deploy the dynamic libraries.

[Ruslan-B]

@kasper3, @dotMorten no offence, but I guess warping the native to OS packaging systems is a bit of topic, as in case you are the owner of the native library - you can maintain a library naming in a dotnet friendly way. So you need this functionality only in case you are not the owner of the original library(s) which is exposing C-API. To summarize this is needed to cover discrepancies between different OS and habits for naming these libraries.

[dotMorten]

@Ruslan-B I fail to understand the relevance to whether you own the original library or not.

The way you deploy your native libs via NuGet is quite different on each platform.
On Android you have to embed you .so files as <EmbeddedNativeLibrary/> or <AndroidNativeLibrary/> depending on whether its embedded in a library or an app (why they didn't make it one build action is beyond me). On UWP and WPF you have to xcopy deploy the DLL, same on iOS but there you also have to define several MtouchExtraArgs.

Having the \runtimes\ folder part be part of the nuget story across all platforms would simplify and unify how you get the native libs deployed in your app. While I agree that part of the discussion is probably more appropriate in the NuGet repo, it's completely valid in the context of unifying how we use native code on the various platforms.

[Ruslan-B]

@dotMorten in this case I would suggest to reread the original proposal.

[dotMorten]

@Ruslan-B Read the entire thing, and you'll find the discussion has expanded quite beyond that.

[Ruslan-B]

@dotMorten true, however, I think the only reason we need this, is to handle discrepancies between different OS with stable habits of naming things.

[qmfrederik]

This issue is an API proposal issue. It concerns an API for loading native libraries and using the functions they expose. It was created because we need to load (lib)gdiplus for System.Drawing across platforms.

How you package these native libraries with your app (and whether you should package them at all or use a different acquisition mechanism) is related but not strictly in scope for the API proposal.

I don't know about iOS and Android, but the .NET Core SDK supports the runtimes folder structure and that works very well for the projects and I'm involved in - YMMV.

[qmfrederik]

@karelz @joshfree This API didn't make it for 2.1, any way we can track it for 2.2.0 (it's currently marked as Future)?

[karelz]

Currently there is not much difference between Future and 2.2. I moved it to 2.2, but it depends if we get to agreement and the result passes code reviews (there has been quite a few people with strong opinions).

[xoofx]

Currently there is not much difference between Future and 2.2. I moved it to 2.2, but it depends if we get to agreement and the result passes code reviews (there has been quite a few people with strong opinions).

@karelz How should we proceed from there? Based on my previous pseudo-proposal should I open a new issue?

[karelz]

I will defer to @GrabYourPitchforks who worked on it and has PR ready (https://github.com/dotnet/corefx/issues/17135#issuecomment-373878407), he should be able to suggest best next steps (once he is back from vacation - one more week).

[xoofx]

What about the fact that on iOS callbacks must be static? That's where all my interop code diverges the most. For us to be able to create truly reusable interop code, these sort of differences needs to be addressed too.

@dotMorten btw, what do you mean by static? Static linking required by iOS with mono? Isn't dlopen being supported starting from iOS8+? (or you need to support older iOS?)

[joshfree]

I will defer to @GrabYourPitchforks who worked on it

Actually we should defer this to @jeffschwMSFT and the Interop team, who are interested in driving this feature area forward in .NET Core. @jeffschwMSFT could you share your team's current thoughts on direction / shape?

[jeffschwMSFT]

@qmfrederik we are taking a step back and considering the broader scenario (exploring a dllmap like mechanism). Right now we don't have a concrete design, but once we do we will share and update this thread.

[qmfrederik]

So, based on all the feedback, perhaps we should split the issue in two.

The core gist of this is that we want a way to call dlopen/LoadLibrary and dlsym/GetProcAddress; most of us can take it from there. The API @xoofx proposed as well as the original proposal for this issue seem fine.

Making cross-platform library loading (i.e. dllmap & friends) would be orthogonal to that, no?

[ghost]

Currently for 2.2, there is one inconsistency left in DllImport tracked by https://github.com/dotnet/coreclr/issues/17604.
This approved API is definitely not related to DllImport enhancement with DllMap like mechanism (which is actually tracked by @akoeplinger's https://github.com/dotnet/coreclr/issues/930). So we would love to see @GrabYourPitchforks's PR dotnet/coreclr#16409 getting resurrected.

[xoofx]

Making cross-platform library loading (i.e. dllmap & friends) would be orthogonal to that, no?

Yeah, I agree, maybe not completely orthogonal, but it is more an implementation detail or an enhancement to the resolution. It should not change the API shape, nor the default behavior if a dllmap file is not present (which is good). Dllmap is a super nice feature/addition and it should be considered, but maybe we should split first an implementation without, which was almost done it seems, and then proceed on the full dllmap story (which will take likely several months of study+shape+implem+tests...etc.)

[swaroop-sridhar]

Closing this issue, in favor of dotnet/corefx#32015