feat: native plugins

[afinch7]

Now that we are using futures 0.3 I should be able to finish this.

Api will look a little different this time to reflect some of the refactors that have occurred since #2385.

I'm targeting something like this on the plugin side:

use deno::CoreOp;
use deno::Op;
use deno::{Buf, PinnedBuf};
use deno::PluginInitContext;
use futures::future::FutureExt;

#[macro_use]
extern crate deno;

fn init(context: &mut dyn PluginInitContext) {
  context.register_op("some_op", Box::new(some_op));
}
init_fn!(init);

pub fn some_op(data: &[u8], zero_copy: Option<PinnedBuf>) -> CoreOp {
  if let Some(buf) = zero_copy {
    let data_str = std::str::from_utf8(&data[..]).unwrap();
    let buf_str = std::str::from_utf8(&buf[..]).unwrap();
    println!(
      "Hello from native bindings. data: {} | zero_copy: {}",
      data_str, buf_str
    );
  }
  let result = b"test";
  Op::Sync(result.into())
}

and something like this on the typescript side:

const plugin = Deno.openPlugin("./path/to/some/lib.so");
const some_op = plugin.ops.some_op;

function doSomeOp(): null | ArrayBufferView {
  const response = some_op.dispatch(new UInt8Array([1,2,3,4]));
  return response;
}

refs #3366 #2473

[bartlomieju]

There are a few things I have questions about, but I believe this might be very solid MVP for native plugins;
a) it's built on ops
b) this is the simplest iteration, doesn't expose state to op dispatcher in Rust, we can iterate on API safely

[ry]

I like that this is only +400 lines

[afinch7]

For some reason workers_startup_bench.ts doesn't terminate now. I don't think that I changed anything that would break that.

[bartlomieju]

It is very solid PR, especially because all changes introduced in it are minimal - it allows to iterate on the API before 1.0 release. It's well aligned with our ops APIs and has minimal impact on current architecture 👍 I'd love to see this landed to give users chance to use it and break it 🚀

[bartlomieju]

Can this step be somehow integrated into cli/tests/integration_tests.rs flow?

[afinch7]

I tried to add a single test to test_native_plugin, but it still doesn't build automatically when running cargo test --all-targets. Not sure if there may be some other simple workaround here.

[ry]

We should figure out a way to test this without making changes to ci.yml... I'm looking into it.

[ry]

I've merged this branch with master and moved a few things around. I think the plugin infrastructure shouldn't depend on cli, so the tests shouldn't be in cli. I'll try to do that now...

[ry]

So.. I'm still having the same trouble you were having, @afinch7, with getting the plugin to build for the test. I guess moving the test_plugin crate to the root didn't help. I've done some superficial clean ups. I'm pretty much ready to land this if we can get it to go green somehow.

[ry]

It's annoying that it's not built already when the integration tests run. I wish I knew some cargo expert who could advise us here.

[ry]

LGTM - Thank you for the long and hard work of getting this in @afinch7 !