Using the Janet C embedding API

[gamagan]

Hello Community,

OS: Windows 10
Version: Latest -- 1.23.1-dev
Language: C++20
Compiler: MSVC 2022

I'm in the process of embedding Janet into a 3D graphics engine, for the purpose of providing scripting capabilities.
I've read through the 'C API' section of the documentation. I have looked at the available C code in 'test' and 'examples' directories, and I've been looking for other locations with Janet C code. I've also done the usual Google searches.
I've been able to print "hello, world!" to std out from the 3D App, via an attached cmd console -- essentially, the code in this file:
https://github.com/janet-lang/janet/blob/master/test/amalg/main.c

I've also gained a decent idea how to go about creating libraries of C functions.

I'm ready to move onto the next phase, which is to define the Api and Environment exposed to Janet scripts, which run in the context
of the 3D App. Unfortunately for me, I've been unabled to figure out some extended use of the Janet embedding API.

If possible, can you please point me in the right direction where I can get further info/examples on how to use the embedding API?

Specifically, I'm trying to accomplish the following:

1. Compile a script into bytecode, verify correctness (for some level of 'correctness', such as syntax), and keep the bytecode around for later execution. The idea being that the App may compile some scripts during startup, and later run them on demand, possibly multiple times. It might also load other scripts, at arbitrary times, as the App executes.

2. Safely unload previously compiled bytecode.

3. Get a pointer to functions defined in the script files, to (possibly) execute them every frame.

4. Bind C functions to the Env from the C side, as opposed to by calling the (use lib) function from Janet script.

5. Bind C variables to the Env, such as ints.

6. Dealing with errors on the C side, raised by called Janet functions.

7. A way to opt-into, rather than opt-out, of functions created by the default janet_core_env() function. Ideally, I'd like to specify what exactly goes into the initial environment, so that upgrading to later versions of Janet doesn't automatically pull in new features. Since the VM will run in the context of the Rendering App, there are functions that are not appropriate for the scripts.

8. Managing the Garbage collector: Pausing, Unpausing, Collecting

9. Passing void* userData into callback functions: C -> Janet -> C (utilize userData here)

10. Way to limit the execution time of scripts, so they don't monopolize the system.

From what've been experience so far, Janet is a really pleasant and well designed language. I can't wait to do cool things with it.

Regards

[andrewchambers]

1. janet itself does this by marshalling functions/environments into a buffer ('marshal) and writing that to disk. You can simply 'unmarshal the buffer when needed.
2. Bytecode/functions are garbage collected like any other data, they won't be freed until they are also cleared from the module cache.
3. not sure.
4. I presume you can just call janet_def.
5. You cannot bind a C int in place, but you can convert one to a janet number and create a def e.g. https://github.com/andrewchambers/janet-pq/blob/54563c6b675627a2c466615c9600052ea705e94c/pq.c#L639. Another way is to define access functions.
6. I think you need to use janet_pcall();
7. I think it is just a matter of which env you pass to your script execution context.
8. Garbage collection is automatically disabled when calling a C function, you must re-enable it manually with the C api and manually manage the gc roots.
9. The 'Janet' type has an opaque pointer in its union I think you can use for this.
10. Not sure.

[gamagan]

Thanks for the reply. Some followups:

1. I'm not so much interested in writing/reading the bytecode to file. It's more like, what's an example code to the compile a file without executing it. Not sure if that's possible. There is the function janet_compile_lint, but it's not clear how use that function, or how to get at the bytecode it creates, for later execution. Or, perhaps, getting at the bytecode for later execution is the wrong way to think about it.

2. Thanks

3. Info pending. This is an important one.

4. Ok

5. Thanks for the example code. That's basically what I meant.

6. janet_pcall seems to convert errors/panics into return codes. How would getting extended info work? Or, for example, how to get at stacktraces (if available) for displaying to something other than std out (such writing it to a texture sent to the GPU for display).

7. The default core env has a lot of stuff that I'd like to include, but also stuff that I don't want to include. How to select which parts of the core env that I want to include, in an opt-in (vs opt-out) way.

8. The goal is to, sometimes, disable the GC for a while without calling into a C callback -- but just turning it off. Then, at appropriate times, when there is idle time available, turn it on and run the collection method. This is to help prevent hitches in certain situations.

9. I don't think the pointer within the Janet union is a general userData pointer. I think that's specifically for when you wrap individual pointers to pass around as opaque values.

10. Info pending. This is an important one.

[paulsnar]

I hope I can shed some more light on this and answer some of the still-open questions further, @gamagan:

(1) janet_compile and janet_compile_lint are what you're looking for (the latter additionally does some extra checks, not exactly sure.) The catch is that they compile the code in the Lispian sense, that is, the value that janet_compile accepts is a single form, such as a tuple for a function call, or a (do ...) block for a multi-statement expression, so that means that you need to already have the code parsed (see the janet_parser_ functions), and then compile will just generate the bytecode, which you can get from the JanetCompileResult. (The Janet function compile does pretty much the same thing but from inside Janet.)

(6) The JanetFiber** argument to janet_pcall will return the fiber on which the pcall executed (or you could pass one in yourself); if an error occurs or whatever, you can further inspect the fiber by yourself. At the moment the public Janet API unfortunately doesn't provide much more than just reaching directly into the JanetFiber structure and pulling out the information you need -- janet_stacktrace just prints out the stack trace just like (debug/stacktrace) would, and doing it in a custom fashion could be kind of gnarly. (You could kind of make do by hooking the *err* dynamic binding and having the stack trace printed into that via janet_stacktrace, but I'm not sure how would you do it from C without looking into it further.)

(7) When creating a fiber, you get to set its environment (essentially, the globals it has access to) -- you could take the default environment (janet_core_env), copy over the values that you want into a fresh table, and then call your callbacks with this environment as the global one, and they won't get access to anything outside it (unless you grant them access to load stuff themselves, that is.)

(8) As mentioned, there are the janet_gc{root,unroot,lock,unlock} functions which, respectively, add or remove a value from the root set (useful when you need something to keep around from C which isn't accessible from janet) and stop and start the garbage collector. Externally you can call janet_sweep and janet_collect to trigger collections manually -- the former will run even if GC is disabled by janet_gclock.

(9) Janet itself doesn't have a mechanism for something like Lua's "environment". The closest analogue is the fiber environment which you can modify in the outer C before calling into Janet, and then get the current fiber in the inner callback with janet_current_fiber and index into its env with janet_table_get. (This could break if the Janet code invokes your C callback from a fiber with a completely clean environment...) In the worst case, Janet usually has one VM instance shared as a thread-local, so you could potentially do the same with your userdata state...?

(10) Right now I don't believe the Janet VM has any mechanism for delayed preemption like Lua does with lua_sethook. (You could probably bodge it with janet_step to single-step the fiber, but that's likely to be awfully inefficient.) That said, you can interrupt the interpreter externally with janet_interpreter_interrupt, say, from a different thread or a signal handler, and it will stop at the nearest jump opcode, but this requires Janet to be compiled with (edit: without) JANET_NO_INTERPRETER_INTERRUPT and will probably make the VM main loop a tad less performant. Not sure whether it's enabled by default or not.

If I may offer a bit of advice from experience, a lot of things are possible with Janet but might require diving into the VM code to find out for yourself. It might seem intimidating at first, and things like the VM main loop are optimized to be efficient and so are less readable, but nonetheless it's the only source of truth at large, and a lot of it is fairly accessible once you feel around a bit. Hope this helps!

[undernorthernsky]

Came across 'Embedding Janet' in 'Janet for Mortals'; that explains a lot of the details, albeit in a somewhat more complicated context (Emscripten).