Is there a way to give Lua access to resource files embeded in a Rust binary?

[alerque]

I have an app that consists of a Rust CLI binary that includes a mlua based Lua interpreter. It also distributes quite a number of Lua and other resource files.

I would like to embed all the Lua resource files into the Rust binary so the app can function as a stand alone binary, but at the same time I would like the existing Lua code to function as-is with require("myapp.module.foo") to continue to function unchanged. In particular I want users to be able to supply their own Lua files to override the embedded ones if they place them at a higher-priority location in the package.path. As it is I'm adjusting the package.path such that Lua searches local to their project first, then falls back to searching relative to the app installation directory, then falls back to the system LuaRocks installation location. Basically what I would like to accomplish is have a special addition to the path inserted in that step that found resources embedded in the binary itself.

I've used rust-embed in other projects to do something similar, but in those cases the internal API just expected to be handed a text buffer and it was trivial to configure to return the file contents from the embedded or filesystem arrangement.

My question is if anybody has accomplished or can think of a way to accomplish something similar with mlua such that the Lua require() side of things can load resources from the embedded files somehow.

[khvzak]

It should be simple I think, you need to register your own package searcher with required priority.

[alerque]

Thanks for the pointers, both of you. That's quite helpful.

I'm going to play around with abstracting the PUC vs. JIT differences and coupling it with rust-embed in my project. Perhaps if I get it working that will be something that can be extracted to its own crate. I'd start with it as a crate right away but I'm not sure I have a feel for what the interface can/should look like yet.

[alerque]

Trying to put it all together in my app, it turns out this code to load a Lua module returns the module source code as a string instead of evaluating it. I'm floundering again for the right way to get it processed. Your lua.create_table_from() works to get a table back, and my lua.create_string() works to get the source as a string:

return LuaFunction::wrap(move |lua, ()| {
    let data = str::from_utf8(module.data.as_ref()).expect("Embeded content is not valid UTF-8");
    lua.create_string(data)
}).into_lua(lua)

It seems to me the Lua code needs to be evaluated at this point, so I started working from something like this:

    lua.load(data).eval()

This doesn't return an expected type, and I'm again at a loss of how to get it to an expected type. Or do I need to get even trickier and evaluate it then manually stuff the result in package.loaded[module] myself?

[alerque]

So close now. This works:

return LuaFunction::wrap(|lua, ()| {
    let data = r#"MYG = 2 * 2; return { a = 4 + 6 }"#;
    let foo: LuaValue = lua.load(data).eval()?;
    Ok(foo)
})
.into_lua(lua);

But it seems like I am still missing an optimization. The following also works if and only if the module explicitly returns a value that can be cast to a table as many modules can, but it fails to load scripts that just set global and don't return anything:

return LuaFunction::wrap(|lua, ()| {
    let data = r#"MYG = 2 * 2; return { a = 4 + 6 }"#;
    lua.load(data).eval().map(LuaValue::Table)
})
.into_lua(lua);

If data is instead just MYG = 2 * 2 this will fail because of course it can't be cast to a Table. I tried inspecting the first variant above to see what it was returning, but only came up with mlua::value::Value, which is private and can't be used in the same way.

Is there something I'm missing to directly cast to the correct type here without unwrapping and wrapping the Result again?

[khvzak]

I think just using

lua.load(data).call::<_, Value>(())

should be enough for all cases.

[alerque]

And it does!

For completeness (since all my searching for this issue now is just coming back to this discussion) here are all three types discussed in a working configuration:

use mlua::ffi;
use mlua::prelude::*;

unsafe extern "C-unwind" fn luaopen_mymodule(state: *mut ffi::lua_State) -> i32 {
    ffi::lua_newtable(state);
    ffi::lua_pushstring(state, b"world\0".as_ptr() as *const _);
    ffi::lua_setfield(state, -2, b"hello\0".as_ptr() as *const _);
    1
}

fn main() -> LuaResult<()> {
    let lua = Lua::new();

    let package: LuaTable = lua.globals().get("package")?;
    let loaders: LuaTable = package.get("loaders")?;

    // Push a loader that passes on a C module generated from Rust bindings
    loaders.push(LuaFunction::wrap(|lua, path: String| unsafe {
        if path == "mymodule" {
            return lua
                .create_c_function(luaopen_mymodule)
                .map(LuaValue::Function);
        }
        "not found".into_lua(lua)
    }))?;

    lua.load(r#"print(require("mymodule").hello)"#).exec()?;

    // Push a loader that returns a Lua module of a set type based on Rust data
    loaders.push(LuaFunction::wrap(|lua, path: String| {
        if path == "abc" {
            return LuaFunction::wrap(|lua, ()| lua.create_table_from([("hello", "world")]))
                .into_lua(lua);
        }
        "not found".into_lua(lua)
    }))?;

    lua.load(r#"print(require("abc").hello)"#).exec()?;

    // Push a loader that executes random Lua input code and loads the result as a module, whatever the return type
    loaders.push(LuaFunction::wrap(|lua, path: String| {
        if path == "foo" {
            return LuaFunction::wrap(|lua, ()| {
                let data = r#"MYG = 2 * 2; return { a = 4 + 6 }"#;
                lua.load(data).call::<_, LuaValue>(())
            })
            .into_lua(lua);
        }
        "not found".into_lua(lua)
    }))?;

    lua.load(r#"print(require("foo").a, MYG)"#).exec()?;

    Ok(())
}

[alerque]

I think I'm most of the way there with a loader for Lua modules that finds them in the embedded asset cache. However this works for Lua modules and I'm having conniptions figuring out how to do it for C modules. I can embed them just fine and get them back as bytes, but I can't figure out how to load the result into Lua. I see the package.loadlib function, but it takes a file path and reads it from disk. I don't see any equivalent that I can pass it a binary blob from memory.

Any ideas?

[khvzak]

You don't need to embed binary modules as bytes, instead you need to link your rust app with a C module, to get a function luaopen_some_module.

Then you can add a loader which will call:

let entrypoint = lua.create_c_function(luaopen_some_module)?;

and return entrypoint function to lua.

[alerque]

Thanks for the pointers. I think I'm most of the way there. I had to chase a lot of linking and path issues because I'm working with 5 libraries that were originally themselves dynamically linked to system stuff, but as a sample I have:

extern "C-unwind" {
  fn luaopen_fontmetrics(lua: *mut mlua::lua_State) -> i32;
}

Later on I cat get the function much like you suggested:

let fontmetrics = unsafe { lua.create_c_function(luaopen_fontmetrics).unwrap() };

Armed with that I'm at the point where I'm adding a loader (and successfully have a similar loader passing out Lua modules from embeded files, so the loaders.push() is at the right place:

loaders.push(LuaFunction::wrap(|lua, module: String| {
  if module == "fontmetrics" {
      return fontmetrics; // <-- What goes in this return?
  }
  format!("C Module '{module}' is not linked in Rust binary").into_lua(lua)
})).unwrap();

This does not work. I've tried every iteration of function wrappers, etc. I can think of for this return value, but I can't make out what it wants. I've tried searching for similar code for reference, but it doesn't seem to be out there. The closest thing I could find in an open source code grep was from a test in this repo, but it was just stuffing a function into a function pointer in the VM, not making a loader.

Suggestions?

[alerque]

Mostly answering my own question because after quite a bit of trial and error I landed on something that actually works. It doesn't feel right, but it's a starting place to iterate on I guess:

extern "C-unwind" {
  fn luaopen_fontmetrics(lua: *mut mlua::lua_State) -> i32;
  // similar wrappers for other linked loaders
}

loaders.push(LuaFunction::wrap(|lua, module: String| {
  match module.as_str() {
    "fontmetrics" => {
      LuaFunction::wrap(move |lua, ()| {
        let loader = unsafe { lua.create_c_function(luaopen_fontmetrics).unwrap() };
        let module: LuaTable = loader.call(()).unwrap();
        Ok(module)
      }).into_lua(lua)
    },
    // More branches, one for each module with a linked loader
    _ => format!("C Module '{module}' is not linked in Rust binary").into_lua(lua)
  }
})).unwrap();

Particularly the semantics of having to explicitly unwrap, cast to a table, call, re-wrap, etc. seems like I might be missing something, but this does give me a functional module that works with no code modification because it returns the same table interface the Lua loader was before.

[khvzak]

The 1st variant should work actually (or very close to). Take a look to the code below, it works fine. No need to do wrap/unwrap/etc.

unsafe extern "C-unwind" fn luaopen_mymodule(state: *mut ffi::lua_State) -> i32 {
    ffi::lua_newtable(state);
    ffi::lua_pushstring(state, b"world\0".as_ptr() as *const _);
    ffi::lua_setfield(state, -2, b"hello\0".as_ptr() as *const _);
    1
}

fn main() -> Result<()> {
    let lua = Lua::new();

    let package: Table = lua.globals().get("package")?;
    let loaders: Table = package.get("loaders")?;
    loaders.push(Function::wrap(|lua, path: String| unsafe {
        if path == "mymodule" {
            return lua.create_c_function(luaopen_mymodule).map(Value::Function);
        }
        "not found".into_lua(lua)
    }))?;

    lua.load(
        r#"
    print(require("mymodule").hello)
"#,
    )
    .exec()?;

    Ok(())
}

[alerque]

That arrangement rates at least one order of magnitude lower on the InsaneSpagettiCode scale. I'll take it (and credit you in the commit of course!).