Writing native plugin in pure C

[Cedric-Thomas]

requested by @bartlomieu following our chat on the deno/lobby i'm opening an issue concerning, how can we do to write a plugin in pure C code ?
for exemple:

-- hello.c

#include <stdio.h>

void deno_plugin_init(void){
    puts("Welcome in the deno land  !");
}

void dino(void){
    puts("[OK]");
}

gcc -c -Wall -fpic hello.c
gcc -shared -o libhello.so hello.o
and the following deno code:

hello.ts

const libhello = Deno.openPlugin("./libhello.so")
const funct_hello = libhello.ops
// this thing work correctly and call deno_plugin_init from my ".so" library
funct_hello.dino()
// Type 'PluginOp' has no call signatures.

so how can we do ?

[rsp]

Will plugins like this (if allowed) require a special permission, like --allow-plugin?

Because this, while useful, could circumvent the entire permission system.

I suppose this is to have plugins that use system APIs that are not accessible directly from Deno, not as a performance optimization - for the latter I would expect this C code to be compiled to WASM and used in Deno without security concerns.

[eliassjogreen]

There already is plugin support that requires the permission --allow-plugin. I see no reason why you should be able to write plugins in Rust but not in other compiled languages like C.

[afinch7]

If you want to bind to existing C code it might be simpler to just write rust bindings for said code. That said, I still think that providing a simple way to create pure C based plugins might prove useful. I've done some research on some solutions, and I found a couple solutions here.

First and most clear solution is to just expose a really simple watered down c version of the existing api. This could be simple enough to only require a header file with some basic structs, types, and helper functions. Code might look something like this;

#include <stdio.h>
#include "deno_plugin.h"

deno_buf *op_some_op(deno_buf control /*, some kinda pinned buf here */) {
  puts("Hello from c op");
  return NULL;
}

void deno_c_plugin_init(InitContext &cx) {
  init_context_register_op(cx, "someOp", op_some_op);
}

I prefer my second solution: creating standalone bindings for the existing api. That way we don't need to maintain two sets of tests and apis. I've already started working on something like this. This one might take a bit longer though.

Either way this will probably end up being a longer term goal. In the shorter term it might make sense to create and document an example that uses rust's C compatibility to create plugin based bindings for existing C code.

[dyedgreen]

Depending on the nature of the code, you might also be able to compile the code to WASM and run in in Deno that way. (This would also avoid having to build for a large set of archs and oses)

(example: https://github.com/dyedgreen/deno-sqlite)

[Qard]

I think the safest approach would be to only allow native code via WASM and control permissions through a custom WASI.

[ry]

I don't think writing a plugin in C should be a first order goal of Deno. It likely will complicate the implementation - users can always bind to C libraries from Rust. That said, I think we should strive for simplicity in the plugin interface - and considering the C user it a good way of doing that...

[Starcounter-Jack]

https://dev.to/nickytonline/10-things-i-regret-about-nodejs-14m3

From Ryan Dahl:
regret: no use of promises for asynch/await
regret: web security, write permissions, & file access
regret: the Build System (GYP/GN/Python)
---> better would have been a Foreign Function Interface (FFI)
regret: JSON & NPM central repository dependencies
regret: node_modules too heavy & often required without the extension
regret: Index.js complicates the module loading system & overly "cute"

I also think not having an easy to use FFI is a big mistake. Having some quirky way to do native in some specific language and specific build system is indeed a mistake. I do agree and hope that that mistake is not repeated. Imagine having to learn a new language to call a native library containing foobar() { return 5; }

[Starcounter-Jack]

I think the safest approach would be to only allow native code via WASM and control permissions through a custom WASI.

WASM/WASI is nice if all we want is

a) embed some simple legacy code
b) need some pointer/memory performance advantages for some library functions; or
c) want some plain vanilla OS services such as opening files or networking.

But one of the main reasons for Foreign Function Interface is to interface with the foreign functions (pardon the pun). Foreign as in made available by the hosts. WASI will only allow simple stuff that we can do in Javascript already such as accessing the network or reading files. Interfacing with foreign native code is often needed for anything interesting as this often involves IPC to running processes, shared memory, GPUs, hardware, huge complex server software digging deep into OS services, etc etc etc.

[Starcounter-Jack]

FFI is often needed for anything interesting as this often involves IPC to running processes, SSE, shared memory, core affinity, GPUs, hardware, huge complex server software digging deep into OS services, etc etc etc.

...and unless the only goal is trendware bingo or wishful thinking, last time I checked, there are at least some operating systems, servers, drivers, services that are not entirely written in Rust.

[Starcounter-Jack]

FFI should not be considered interfacing with C, it is interfacing with native calling conventions and structure conventions of native Linux/Windows/MacOS regardless of language. Lua FFI, C# FFI has this thinking while Java and Node have not. It is not surprising that Lua and C# is chosen over Java and Javascript for things like game engines or high performance computing. Currently, Node is the best option for Javascript applications to interface with native, but anything with a decent FFI would be superior while, unfortunately, WASM/WASI would be a toy or weak bragging rights for most real use cases.

[buckle2000]

We need official support for FFI. It's much better than the clumsy plugin system.

[dyedgreen]

Interfacing with foreign native code is often needed for anything interesting as this often involves IPC to running processes, shared memory, GPUs, hardware, huge complex server software digging deep into OS services, etc etc etc.

I feel like anything that needs that deep level of access if probably better served by a lower-level language like C, C++, or Rust. And if a JavaScript runtime is desired, is probably in a better spot embedding this runtime directly. Such applications will likely be highly specialised at last to a given OS (whereas Deno supports macOS/ Linux/ Windows), and will not benefit from the main features of Deno (run everywhere easily, fetch files from the network, secure runtime with no unchecked external access).

In my opinions, all that is needed to support 99% of the use-cases is WASM(/WASI) support, with is e.g. already sufficient to run things like SQLite, and further allows to run them in a secure, trust-less manner.

I feel that this aligns closer with the "A Web Browser for Command-Line Scripts" philosophy. Needing to install external plugins, that then need to be trusted, to run some script seems to go against this.

[Starcounter-Jack]

I feel like anything that needs that deep level of access if probably better served by a lower-level language like C, C++, or Rust.

I strongly believe it is the other way around. Writing the glue code manually is practical only when doing something small and shallow.

Of course the lower level stuff is written in C, C++ or Rust, but I think it is easy to downplay how time consuming it is to interface with a broad set of functions and structures without a FFI such as libffi or pinvoke. Whenever someone offers a managed language such as Javascript, C#, Lua, Java and Ruby you soon see that the runtime of choice for anything integrating with native is the runtime with FFI. C# and Lua is used by game engines while Javascript and Java is not.

Deno native is currently similar to Java JNI. And Java is horrendous compared to Lua and C# for interoperating with your Rust, C, C++ code or host libraries (unless you are doing a hello world).

[antoniormrzz]

I believe a way to build native plugins would enable faster porting of stuff that already exist in node, there are a lot of libraries written in C and C++ and build to a .node file. the documentation (as far as I saw) didn't address how to build these plugins and right now I'm not sure that is even possible. rewriting or interfacing these huge libraries through rust doesn't seem like a good choice tbh.

[buckle2000]

@antoniormrzz You want to help with this? #5566

[antoniormrzz]

@buckle2000 I would love to, unfortunately I have no knowledge of rust or how deno internally works, I just wanted to port a library from node to deno over the weekend and hit a bump, if something like node-gyp or a solution to what it does was available, that would be awesome. I took a look at your examples in other languages, and other pull requests, and indeed it seems like people need a way to talk to FFI. @ry without this, a lot of functionality would need to be rewritten or ported to rust ( I don't know how deno works with rust either, the docs didn't help much with native plugins) which slows down deno's growth. what I needed was RobotJS btw, any advice on how to port that?

[tracker1]

@dyedgreen

In my opinions, all that is needed to support 99% of the use-cases is WASM(/WASI) support, with is e.g. already sufficient to run things like SQLite, and further allows to run them in a secure, trust-less manner.

I have to disagree that WASM/WASI is enough... even the example you give is not quite right... the WASM version of SQLite cannot interact with other processes accessing the same database file. It's actually, very specifically a feature I was hoping to find, and may implement with a native module with the rust bindings for sqlite.

[dyedgreen]

@tracker1

the WASM version of SQLite cannot interact with other processes accessing the same database file

Using the file system APIs provided by Deno, I think you can actually implement everything a regular SQLite3 could do (minus file-locking at the moment, since that's not provided by Deno, see dyedgreen/deno-sqlite#49 for details on this).

[tracker1]

@dyedgreen

I think sqlite does some things to overcome OSes that don't have a file locking API iirc... so not sure if that would/should be enough as I'm unsure how the wasm environment looks in that regard...

Even then, there are still many use cases where an FFI standard for the platform is a good idea in addition to binary modules... with Rust + FFI options, that would probably be enough for the most part.

[dyedgreen]

I completely agree that there should be an FFI, but I think that we need to be careful not to make it the go-to option for things that can be achieved with WASM and through standardised Deno APIs, which can enforce security.

[tracker1]

@dyedgreen I mostly agree.. that said there's only so much time in the day to learn new things... and wasm idiosyncrasies combined with various library platforms that I'm not familiar with (I'm not a C dev) to begin with are a much higher barrier to entry than just using FFI in many cases. ;-)

[buckle2000]

WASM should be used for pure functions for safety. Any language with LLVM backend can be compiled to WASM.

[Syn9673]

any updates? since im planning to use C++ with Deno, i tried compiling the C++ code to WASM, but WebAssmebly object gives it errors

[richard-uk1]

I think wasi is the way to go there - they already implement a lot (file descriptors, sockets, signals, args/env, preemptive yield hint - see docs) and if there are things missing, they probably should be in in some form so open a ticket.

[d3x0r]

using deno run --allow-plugin test.js ...

and

const plugin = Deno.openPlugin("./path/to/some/lib.so");

I get

Uncaught TypeError: Deno.openPlugin is not a function

I already have a system interface; it's, you know, 99% adding stuff to V8...
I don't quite see the similar entry definition like node addons

	NODE_MODULE_INIT( /*Local<Object> exports,
		Local<Value>Module,
		Local<Context> context*/ ) {
		VolumeObject::Init(context,exports);		
	}

WASM interface isn't really practical; since this is already a wrapper directly to the system (whatever target system it might be)... working through abstractions of WASM syscall dispatch just to get real syscalls seems redundant to me.

[crowlKats]

@d3x0r you need to run with the unstable flag.

[redradist]

Writing C plugins is also requires packaging this plugin ...

For such packages it would be nice to support external package managers like npm, conan (C/C++), vcpkg ...

I know that Rayne does not want the centralized package manager, but for packaging native plugins we need to support them ...

What if deno will support possibility to install npm, conan (C/C++), vcpkg and other native packages, but from the any place from the internet:

import pluginFunction from 'https://github.com/packages/{some npm package}';
...

Deno could detect package type for plugin and compile it and install either in node_modules folder if it is npm package or in .conan if it is the conan package ...

Actually to support it we do not need direct support them from Deno, we just need import hooks for Deno runtime that will do all this actions

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[d3x0r]

It almost seems like having an addon in deno involves adding it to the build process along with like v8 and building a custom deno?

Maybe if there were hints how to just use rust linker to package C library would be a good start?

[ivancuric]

WASM is sadly not a solution for many applications, especially those that require GPU and other hardware access.

[lucacasonato]

WASM can very much be used for GPU when used in combination with the experimental WebGPU API :-)

[ivancuric]

Would WebGPU allow users to use capabilities such as CUDA, OpenCL, NVENC, intel QuickSync and other technologies?
One example I'm thinking of, which is often used, is FFmpeg / Gstreamer.

[trxcllnt]

No, WebGPU is a standalone spec and implementation and does not provide access to the CUDA driver or libraries.

Consider me a +1 to adopting either a native plugin API (C++ or Rust), or (more annoyingly) a native FFI layer.

[lucacasonato]

We already have a Rust native plugin API (it is unstable).

[trxcllnt]

I wasn't sure if that'd stick around given discussions like #8490. But if it is then excellent, I've been looking for an excuse to do some Rust.

[lucacasonato]

The signature will change significantly in the coming weeks, but I think overall plugins are here to stay.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[redradist]

Is there any updates for this issue ?

[lucacasonato]

No

[bartlomieju]

Native plugins are getting removed (#8490) in 1.13 and replaced with FFI (#11152).

It will be possible to call C code from Deno code.