Add types to the source code

[Ruin0x11]

I don't particularly regret choosing Lua as the language for the engine (being as it has only a single user as of present writing), but I do wish there were at least type hints for the many data structures like map objects or the bundles of parameters passed to callbacks or events. It's difficult to figure out what the contents of a params table is without searching the source code or manually adding documentation and constantly making sure it doesn't go out of sync. The problem is that I've been spoiled by the program introspection features that are central to my workflow and can't seem to give up the things that Lua offers, even though it's a dynamically typed language. Basically, what I want is a language that:

• is general purpose
• is gradually typed
• has a runtime-reloadable module system
• is production-ready

As far as I'm aware, this language does not exist yet. The crucial point here is that, unlike most dynamic languages like JavaScript or TypeScript, the import mechanism in Lua is a function instead of a statement. This means you can call require as a function to get back a dictionary containing a complete module, and furthermore that you could replace require with your own implementation that does some custom module bookkeeping, as OpenNefia does. I don't think you can do this in ES6, TypeScript or Python, where you can't hot-reload modules at runtime in the same way, or sometimes at all, because the module system is so tightly integrated with the language and its ecosystem that you can't hook into the way modules are produced or loaded. From the Python docs for importlib.reload() (emphasis mine):

Python modules’ code is recompiled and the module-level code reexecuted, defining a new set of objects which are bound to names in the module’s dictionary. The init function of extension modules is not called a second time. As with all other objects in Python the old objects are only reclaimed after their reference counts drop to zero. The names in the module namespace are updated to point to any new or changed objects. Other references to the old objects (such as names external to the module) are not rebound to refer to the new objects and must be updated in each namespace where they occur if that is desired.

We avoid this in OpenNefia by enforcing the convention of returning a complete module table from each Lua file, and using that returned table to do late binding. (Rand.rnd(42) over local rnd = Rand.rnd; rnd(42)) This way you can reload just the single module's file/namespace and have all other references to the module in the running program updated automatically, so long as late binding is being used (no upvalues). Perhaps this is why Lua is the only easily recognizable, "mainstream" programming language that shows up in the list of notable live coding environments on Wikipedia, alongside the likes of more obscure fare like Max and Sonic Pi.

But it seems that none of the general-purpose programming languages with this kind of module system have a type checker. Which is a bummer, because that means you have to choose whether you want program introspection and hotloading or the ability to catch simple type errors at compile time instead of runtime.

The good news is that seems like Teal might finally become the solution at some point. With Teal, you can essentially write "TypeScript, but Lua," catch type errors before running any code, and then compile the source to Lua and load it just as before. Since the compiled code is just Lua, you'd still be able to take advantage of hotloading where it was previously possible, so long as the types work out. And you could just ignore the typechecker if you know what you're doing, as Teal is merely a small addition of syntax to standard Lua.

As far as I know, Teal is the first programming language to both use a type system and allow for runtime module hotloading.

In regard to OpenNefia, I don't know what porting the codebase to Teal would look like, or if porting the entire codebase instead of small pieces would be feasable. As the time of writing it isn't possible at the moment to practically adopt Teal with the amount of metatable and class magic being used, since Teal currently doesn't have any way of specifying functions that create new types in a metaprogramming fashion. But I think having types would make it way, way easier to mod things when you see a function argument or table and no longer have to worry about searching through the entire codebase or putting down a breakpoint in-game in order to figure out what fields it contains.

A couple of questions I keep coming back to when it comes to adding types:

1. How will types interact with versioning? Obviously the API of the engine or a mod could change between versions. There would need to be some way of knowing the specific set of types of the engine/mod for a particular version, and treating those types as the same in the typechecker if they are structurally equivalent. Because Lua is more lax than Teal there could be some instances where a type error could be resolved gracefully, for example if an optional parameter was added to the end of a function's argument list (although currently Teal has no support for option types).
2. What kind of transition plan would we have, assuming that it will take a very long time for Teal to become stable, longer than it takes for OpenNefia to be "playable," at least in a pre-release state? We would probably need to allow people to code mods in either Teal or Lua. But then it's possible for a Lua mod to be imported by a Teal one, and Teal will always throw an error on missing type information when compiling Teal source code, so we'd need some way of adding annotations for those mods externally, in such a manner that the original mod's author doesn't need to update their mod.
3. How would we document the fields of each type? We'd probably need ldoc for Teal at that point. But even having types alone would stand as a decent source of documentation in itself, if the alternative is guessing.
4. How would types interact with the map object system? It uses some weird metatable logic to essentially set an interface declared in the schema as part of its metatable instead of creating a concrete class. There are also "fallback" fields that are optional and only used to prevent errors, some of which are only for usage internal to the engine, so it would need to be clear which ones those are.
5. How would the type system interact with hotloading? This will probably be difficult to solve. I'm thinking we'd have some way of calling the almighty table.replace() on a table returned from Teal's record statement for all types detected in a module's table and ignoring any errors for code that's already loaded.
6. How would we extend map objects or types with data added from mods? If we go with Namespace custom game object fields by the mod that adds them #15, then this becomes easier as it would just be a table mapping from mod ID to a typed data bundle registered by each mod. But how would the type checker know about these additional types for the extra map object data? We'd need to keep track of them somehow through a registration system and pass them to Teal's typechecker when it runs. I'm thinking that it might look something like this:

local opennefia = require("api.types")

local types = {
    TempCharaData = record
        time_to_removal: number
    end
}

MapObject.extend(opennefia.IChara, TempCharaData)

return types

local mod_types = require("mod.temp_chara.types")

local chara: opennefia.IChara = Chara.create("elona.putit", Map.current())
local extra_data: mod_types.TempCharaData = chara._.temp_chara
extra_data.time_to_removal = 100