We should figure out how to approach custom versions of Lua

[hugomg]

Lua is a flexible language. There are multiple ways to build a custom version of it by tweaking the luaconf.h or patching its source code, and these changes all affect the API between the Lua VM and C extension modules.

There are two different directions Pallene can take in this situation. We could require that Pallene code only works in our own custom version of Lua, or we should make it so that Pallene can also be used by the various custom versions of Lua. Currently, Pallene is halfway between these two options, and I think we should pick one side or the other:

1. Pallene modules can only run against versions of Lua that edit the luaconf.h to make all the private functions public.
2. Pallene modules can only run against versions of Lua that make all the internal ".h" header files public (lobject.h, lvm.h, etc)
3. Pallene assumes that LUAI_MAXSHORTLEN is 40
4. Pallene assumes that LUA_NUMBER is double. (This affects how to print floating-point literals in C.lua)

Option 1 wold be to require Pallene to only work with our custom version of Lua 5.4. We would need to modify the installation scripts and installation instructions to have a way to install our custom version of Lua.

Option 2 would be to allow Pallene to work with any suitably-configured version of Lua 5.4. We would need to describe what are the required settings in the luaconf.h and how to make the header files like lobject.h and lvm.h visible to Pallene. Pallene should also produce a clean error when something that it assumes is not what it expects (e.g., LUA_NUMBER is not double). These can be compile-time errors, or run-time errors at the start of the luaopen function.

Option 3 would be to find a way to allow Pallene modules to be used with a totally unmodified version of Lua 5.4, without requiring any changes to luaconf or that any internal header files be made public. In order for this to work, the Pallene library code would need to bundle a copy of all the necessary Lua interpreter functions. We would also need to find a way to ensure that this bundled version of Lua is compatible with the Lua interpreter that is calling it.

Whatever solution we end up with, I think this is going to be a key step to solve #16.

[gligneul]

I wish we could link to any Lua release during the compilation of a Pallene module. This could be an optional command line argument that receives the path to the Lua source code. Then, we copy the internal functions that we need so we don't need to modify the interpreter source code to expose them.

[hugomg]

I think what you describe would fit with option 2. AFAIK, the only modification we have right now to the Lua source code is that we change the luaconf.h to make the private functions be public.

What I am more worried about would be attempting to implement Option 3 (which I now edited in the other comment). While it would be neat to be able to use Pallene with the system version of Lua installed via the Linux package manager, or brew install, the problem is that in this context I can't think of any way to ensure that the luaconf.h settings of the bundled version of Lua would match with the luaconf.h settings of the system version of Lua.

[gligneul]

I can't think of any way to ensure that the luaconf.h settings of the bundled version of Lua would match with the luaconf.h settings of the system version of Lua.

Usually the header file is distributed with the interpreter. I think that this is up to the distribution maintainer to make right, not the actual compiler.

[hugomg]

Here in my Fedora machine only the following header files appear on /usr/include:

• luaconf.h
• lua.h
• lualib.h
• lauxlib.h

It does not include lobject.h and the other internal header files that we need.

That said, one approach we could consider for Option 3 would be to use the system's luaconf.h to build the bundled version of Lua. As long as there aren't any breaking changes in the rest of the code there is a chance that compiling our copy of the Lua interpreter using their luaconf.h could work.

[gligneul]

There are the internal headers indeed. I still think that the compiler should not worry too much and just receive the path (as an environment variable) to the Lua source code. It would be up to the distribution maintainer to handle these issues (in another repo).

[gligneul]

One thing that I would like to support is a patched version of Lua that might have modifications in these internal header files. Those modifications could affect the code generated by Pallene. For instance, I would like to use Pallene with Lunatik (Lua in kernel). To do that, I would need to pass to Pallene the source code of the modified Lua.

[hugomg]

The Lunatik use case would be a good argument against Option 1. The whole argument in favor Option 1 would is that it trades off flexibility in the customization of Lua in favor of ensuring that Pallene is easy to compile and "always works". But if we want to allow something like Lunatik we might as well go for Option 2 and/or 3.

It is also a good argument in favor of Option 2. I could see a world where we allow both Option 2 and Option 3 but the Lunatik use-case is only possible with Option 2.

In the interest of keeping it simple, I think we should focus first on getting Option 2 to work. Later, if there is demand, we can investigate the possibility of supporting Option 3 too.

I propose a roadmap for Option 2 as follows:

1. Create an installation script and installation instructions for our custom version of Lua.
2. Document how to create another version of Lua that is compatible with Pallene. Basically, explain that we need to redefine LUAI_FUNC in luaconf.h, and that the internal header files need to be put somewhere that Pallene can find them.
3. Come up with a way to tell the Pallene compiler where to find the Pallene-compatible version of Lua. Currently we assume that it is in the current working directory.

What do you think?

[hugomg]

Closed by #534