Allow the compiler to be called from any directory

[hugomg]

Currently, pallenec only works if the current directory is the root project directory. If we try to call it elsewhere it fails for the following reasons:

• The C compiler can't find the headers from our bundled version of Lua
• The require can't find the modules in the titan-compiler subdirectory. (LUA_PATH issue)
• The C linker (called in c_compiler.lua) cannot find our runtime library.

[gligneul]

Remainder: update readme when done.

[gligneul]

I don't this this is an issue for the dev version.
This issue could be solved if install Pallene properly with Luarocks.

[hugomg]

Even for the dev version, being effectively restricted to only running code inside the root directory is a big limitation. It is certainly affecting how we write our benchmark and test scripts. (See #17)

I agree that properly installing pallene via luarocks is the way to go. In fact, figuring out how to do so is exactly why I opened this issue in the first place :)

[hugomg]

I've been thinking about this problem and I think the tricky bit is going to be the Pallene runtime library. Currently we have a custom Lua interpreter and a runtime library (pallenelib). In theory we could merge one of them into the other but in any case we will need a runtime library component and that brings two problems

1. The runtime library is not a Lua module that you load with require, and Luarocks is not designed to install something like that.
2. We currently statically link the pallene library but if we would like the option of dynamically linking it then it would need to be installed to /usr/local/lib instead of the Luarocks installation prefix. Linux doesn't like when your shared library is hidden away inside the Luarocks directory hierarchy.

So what do you think about having a separate installer for the Pallene compiler and for the Pallene runtime library?

The Pallene compiler can continue as a Luarocks module. Luarocks knows how to install it and also takes care of installing our Lua dependencies.

For the Pallene runtime library, we would take our existing makefile and add a make install rule that would install the library and the pallene_core.h header file to the standard system directories. We should also try to configure things so that the local directory takes precedence during development, to make the version we are currently working on take precedence over the installed version.

[hugomg]

To solve this issue for good I think we will need to clean up the situation with the pallene runtime library and the custom Lua interpreter.

I investigated what private Lua functions we are currently using. For the record, this is the list that I got after running nm on a large Pallene module packed with as many language features as I could:

• luaC_step
• luaD_growstack
• luaH_getn
• luaH_new
• luaH_resizearray
• luaL_error
• luaL_where
• luaS_createlngstrobj
• luaS_newlstr
• luaS_newudata
• luaT_typenames

[gligneul]

The runtime library is not a Lua module that you load with require, and Luarocks is not designed to install something like that.

The runtime library could be a .so file and we link against the modules generated by Pallene compiler. We could also add a dummy luaopen function to the runtime library so we can use require to load it, but that might be ugly.

So what do you think about having a separate installer for the Pallene compiler and for the Pallene runtime library?

This seems strange. Can't we handle everything inside Luarocks? Also, I'd rather not mix compilation and installation in the same makefile. Compilation is usually generic and unix-portable; installation isn't.

To solve this issue for good I think we will need to clean up the situation with the pallene runtime library and the custom Lua interpreter.

We could merge both. If I recall correctly, this is what Titan is doing. I don't think that we will have any issues. If we choose this path, I think it is important to automate the process of extracting the functions that we need from the Lua interpreter and bundling them to our runtime library.

[hugomg]

We could also add a dummy luaopen function to the runtime library so we can use require to load it, but that might be ugly.

We would need to use package.loadlib("*") instead of require. By default, require calls dlopen with a flag that hides the non-luaopen symbols in the shared library.

To do that we would also need to pass the full path to the lua library. I think you find that like this but it needs the debug library and would also bake the assumption that Pallene is installed via Luarocks (which would be a problem if someone wanted to package it for a Linux distro for example).

This seems strange. Can't we handle everything inside Luarocks?

Luarocks is geared towards installing Lua modules and the pallenelib is not a Lua module. I think it can be done if you really want to but it is a bit unwieldy.

Annyway, I'm not sure that installing pallene lib separately would be that bad. There is the precedent of those luarocks modules that are a wrapper to some other system library (ex.: expat, sqlite, etc). Usually you need to install those libraries through your system's package manager.

Compilation is usually generic and unix-portable; installation isn't.

I think that can be a good thing. I would rather have the system-specific installation directory stuff in the makefile than inside c_compiler.lua or our generated luaopen functions.

For what it is worth, the plan would be to tell the makefile to install to /usr/local hand have it be configurable with DESTDIR and the other usual make install environment variables. It would be fairly unix portable. (Windows is another story, of course).

We could merge both. If I recall correctly, this is what Titan is doing

IIRC Titan always statically links a copy of liblua into the Titan-generated binaries.

There is as also some sr.lua stuff to be able to generate a self-sufficient executable file instead of an ".so" library. But I don't understand that part very well.

[gligneul]

To do that we would also need to pass the full path to the lua library. I think you find that like this but it needs the debug library and would also bake the assumption that Pallene is installed via Luarocks (which would be a problem if someone wanted to package it for a Linux distro for example).

We could use package.searchpath to find the library, no?

> package.searchpath('lfs', package.cpath)
/Users/gligneul/.luarocks/lib/lua/5.3/lfs.so

Annyway, I'm not sure that installing pallene lib separately would be that bad. There is the precedent of those luarocks modules that are a wrapper to some other system library (ex.: expat, sqlite, etc). Usually you need to install those libraries through your system's package manager.

I'd rather hide this from the final user. Moreover, our runtime lib isn't quite a system library, it is very specific to our compiler.

IIRC Titan always statically links a copy of liblua into the Titan-generated binaries.

Yes, but I meant that Titan also statically links the Lua functions it needs against the generated module. We could dynamically link everything we need (runtime library and Lua functions).

[hugomg]

I hadn't thought about the package.searchpath. That could certainly help.

We'd still need to solve the issue of the internal Lua header files but that is easier than the dynamic library problem.