What is this?

spaghettimod is a Cube 2: Sauerbraten server mod with Lua scripting. It is completely different from Hopmod (the other Lua server mod around). It is for modders who already have a good knowledge of the Sauerbraten server codebase, and just want tools to easily extend that with Lua. It is not for clan leaders who just want a personalized (read, custom message color or little more) server.

For this reason, the principles are mostly coding principles.

• minimize impact on the C++ codebase, for easier merging and predictable behavior
• expose the C++ engine as-is to Lua, making it easier to write Lua code as if it was C++ inlined into the vanilla implementation
• no bloat, and use external libraries as much as possible (fight the NIH syndrome)
• provide modular standard libraries for most wanted stuff, but configuration is done through a Lua script, not configuration variables
• no cubescript (VAR and COMMAND are mirrored to Lua)
• ease of debugging

I am available in Gamesurge as pisto. It is called spaghettimod because I'm italian.

Default configuration

There are some files in script/load.d, which enable some sane default configuration. These modules are:

1. 10-logging.lua : improved logging with date, renames, etc
2. 20-cleanshutdown.lua : gracefully kick all clients on shutdown, remove the server from master quickly
3. 100-connetcookies.lua : harden the server against (D)DoS attacks, see section [Advanced networking](# enet-cookies))
4. 100-extinfo-noip.lua : do not expose the (partial) IP of players through extinfo (either send 0, or a random IP from the same country if GeoIPCountryWhois.csv is available)
5. 100-geoip.lua : show Geoip on client connect, and provide the #geoip [cn]
6. 2000-demorecord.lua : record demos in <servertag>.demos (std.servertag is a module that returns either the port number or a user provided string to tag the server among various instances)
7. 2000-serverexec.lua : create a unix socket for a Lua interactive shell, connect with socat READLINE,history=.spaghetti_history UNIX-CLIENT:./28785.serverexec
8. 2000-stdban.lua : advanced ban and kicks support (IP ranges, access rules, bypass rules, listing and deletion...)
9. 2000-ASkidban.lua : ban proxies with ASkidban
10. 2100-mapswitch-gc.lua : run a Lua garbage collection cycle after a map load
11. off/3000-shelldetach.lua (off by default unless you have the luaposix package and symlink it in the script/load.d folder): make the server fork to background and write logs to <servertag>.log, and execute a full restart on SIGUSR1 (updating to the latest revision is as easy as git pull && make && killall -s SIGUSR1 sauer_server).

Additional assorted modules can be found in the ancillary repo spaghettimod-assorted.

The PISTOVPS configuration

If you start the server with the enviroment variable PISTOVPS set (PISTOVPS=1 ./sauer_server), you will have a clone of the server that I run myself (/connect pisto.horse 1024). 1000-sample-config.lua sports a more real life configuration, including map rotation, abuse protection, gamemode mods (quadarmours and flag switch)... Just /connect pisto.horse 1024 to check out the latest gadgets.

The ZOMBIEVPS configuration

The ZOMBIE OUTBREAK! server (/connect pisto.horse 6666) can be started with ZOMBIEVPS=1 ./sauer_server. It is a heavily modded gamemode with up to 128 bots, and showcases a variety of event hooks.

Performance

Performance is deemed to be "very good". I run athe ZOMBIE OUTBREAK! server and even in crowded situations (~20 players, ~60 bots) it still takes only 10%-15% of cpu and 25-30 MB of memory, and since Lua is called for at least every N_POS message, performance in general should not be a concern.

In my experience, Lua generates a lot of memory fragmentation together with the glibc implementation of malloc(), which means that memory may be deallocated but never returned to the system, and the server process will result to use much more memory than what is reported by collectgarbage"count". This is particularly noticeable when the 100-extinfo-noip.lua script finds the GeoIP cvs databases and generates a fake geolocalized ip: a lot of tables are allocated, then freed, but the server still takes 60 MB. I solved the problem by using a low fragmentation implementation of malloc(), jemalloc: the memory usage at boot is now ~13 MB.

Compilation

Compilation has been tested with luajit, lua 5.2, lua 5.1, on Mac OS X, Windows and Linux. The default scripts are written with a Unix environment in mind, so most probably they won't work under Windows. The only other dependency is libz.

The Makefile is different from vanilla but acts the same. make generally should suffice to create the executable in the top folder.

The Lua version is determined automatically with pkg-config, preferring in order luajit, lua5.2, lua, lua5.1. If you wish to select a version manually, pass LUAVERSION to make, e.g. make LUAVERSION=5.3. You can override the pkg-config detection with LUACFLAGS and LUALDFLAGS.

If you want to cross compile, pass the host triplet on the command line with PLATFORM, e.g. make PLATFORM=x86_64-w64-mingw32.

To change the optimization settings, you sould use the OPTFLAGS variable instead of CXXFLAGS directly.

RAM usage during compilation

Compilation of {engine,fpsgame}/server.cpp take an enormous amount of RAM (~ 700MB) because of heavy template instantiation in the Lua binding code. This can be a problem on a resource limited VPS. If you hit this problem, try this:

fallocate -l 2G /swapfile
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile

clang is known to use around one fourth less memory. Additionally, you can disable debug symbols generation with DEBUG="": this will save one fourth of memory in g++, roughly half in clang (it will also make compilation around one third faster). If you have a crash and find yourself without debugging symbols you can try to rebuild spaghettimod without the DEBUG modifier and hope that the output executable is the same as the old one at binary level, which is generally the case but not guaranteed. gdb will also spit out the warning "exec file is newer than core file.", but you can ignore that. With these hints (clang and DEBUG="") you can get down to ~ 300MB memory usage.

Debugging

You can debug the server and client code, C++ and Lua, without network timeouts, thanks to the patch enetnotimeouts.diff. Follow this procedure:

1. download and install the SDoS test client, or apply the patch to your client
2. start the server, optionally under a C++ debugger

• the standard bootstrap file tries to run either the ZeroBrane Studio or Eclipse LDT remote debuggers

3. start the client, optionally under a debugger, and connect to the server
4. issue /enetnotimeouts 1 on the client, and engine.serverhost.noTimeouts = 1 on the server

With these steps, you can stop the execution with any of the three debuggers involved, and resume at will without timeouts.

Stack traces

The C++ code takes care to call all Lua code with xpcall and a stack dumper, so in general you always get meaningful stacktraces. Unfortunately, the LDT debugger does not support break on error (and I suppose it is not possible to fully implement that without C source modding): however, you can place a breakpoint on this line to break at least on error thrown by hooks.

If you are using a C debugger and spaghettimod crashes or halts while executing Lua code, you get a rather useless C stack trace of the Lua VM. If you are using gdb and Lua 5.2, you may use these gdb scripts: just type luatrace spaghetti::L, and you hopefully will get a Lua stack trace (for Lua 5.1 or luajit, you may need to edit the script and use lua_pcall instead of lua_pcallk).

Advanced networking

The in-tree ENet source comes with two additional features, besides the aforementioned debugging switch: multihoming and a connection flood protection, akin to TCP syn cookies. Multihoming is hardcoded and cannot be turned off (without dirty hacks), while the connection flood protection needs to be explicitly activated. This is done in the default configuration script 100-connetcookies.lua.

Multihoming

This patch allows the server to run correctly on servers attached to multiple networks on unix hosts, which boils down to sending reply packets from the same interface that received the request. There are two extra functions that ENet exports:

• ENET_SOCKOPT_PKTINFO: option value to enable local interface address information on a socket, needs to be set on sockets that use the two following functions.
• int enet_socket_receive_local (ENetSocket, ENetAddress *, ENetBuffer *, size_t, ENetAddress * localAddress): same as enet_socket_send_local, but if localAddress is non null the address of the receiving interface is stored in localAddress.host (the port field is undefined).
• int enet_socket_send_local (ENetSocket, const ENetAddress *, const ENetBuffer *, size_t, ENetAddress * localAddress): similarly, same as enet_socket_send but reads the interface from which the packet need to be sent in localAddress, if non null.

At the moment, Windows is not supported because the required functions need to be loaded at runtime, and would need to be attached to each socket (but this is not so clear from the MSDN documentation).

This patch enables also creating a single connected socket for each peer if the system supports SO_REUSEPORT. This helps because traffic to a connected socket is implicitly prioritized from traffic from not-yet-connected peers (see the next paragraph for an exmplanation of why this is necessary) as one buffer is allocated specifically for that peer, and shuffling packet requires less ancillary data to be communicated with the kernel. The TOS field of the IP header for packets from these sockets is set to 4, to help in setups with iptables, where you want to do certain actions if the server acknowledges a connection.

ENet cookies

Upstream ENet is particularly vulnerable to connection slots exhaustion if an attacker can spoof the source IP, because for each new connection request a new peer is setup in a state almost equal to that of an established connection. An established connection takes quite a long time to timeout. Furthemore, processing the connection packet is not optimized to the bone.

This patch "hacks" the protocol to send a cookie with 2^16*peerCount different possible values on each connection request, and only stores a lightweight object to restore the complete peer state once an ack is received with such cookie. The cookie is sent only once, contrary to vanilla ENet which treats the connection verification packet as a normal one, and so is affected by normal timeouts and retry attempts. Once a valid ack is received, the peer is effectively setup for communication. It's important to note that all the other connection attempts to the same peerID are discarded, so you should create your ENetHost with the maximum available number of peers (ENET_PROTOCOL_MAXIMUM_PEER_ID = 0xFFF), to maximize entropy in the cookie and minimize erroneously discarded connection attempts (and spaghettimod does that).

The implementation is optional and needs to be activated explicitly. It is highly optimized to process connection requests and connection verification acknowledgements. The tradeoff between CPU (and so performance and packet drop) and memory can be tuned linearly with a parameter, and so the timeout for each connection request. A good random number generator is needed, and it is responsibility of the user to provide it through a callback, because there is no good cross platform PRNG.

Cookies can be activated with a call to int enet_host_connect_cookies(ENetHost * host, const ENetRandom * randomFunction, enet_uint32 connectingPeerTimeout, enet_uint8 windowRatio):

• randomFunction: a callback which provides entropy to ENet (if null, deactivates cookies). Fields are
  • void * context: a context to pass to the function
  • enet_uint32 (ENET_CALLBACK * generate) (void * context): the actual function pointer, which needs to provide 32 bits of entropy at each call
  • void (ENET_CALLBACK * destroy) (void * context): a function called once enet_host_connect_cookies is called on the same host again, for the purpose of finalizing the context. Can be null.
• connectingPeerTimeout: timeout for each connection request, if zero it is set to ENET_HOST_DEFAULT_CONNECTING_PEER_TIMEOUT = 2000.
• windowRatio: a percentage parameter that controls tradeoff between CPU and memory, if zero it is set to ENET_HOST_DEFAULT_CONNECTS_WINDOW_RATIO = 10.

Memory usage roughly follows this formula (size_of_cookie = 60): attack_pps * (connectingPeerTimeout / 1000) * (100 / ENET_HOST_DEFAULT_CONNECTS_WINDOW_RATIO) * size_of_cookie. With the default parameters, a 10k pps attack can be stopped without problems with 12 MB of memory, and in informal tests it has been found that this scales well at least up to the range of 150k pps (you may need to enlarge the socket receive and send buffers with the normal ENet API).

Information for Lua modders

The Lua API tries to be as much as similar to the C++ code. Generally you can write basically the same stuff in Lua and C++ (replacing -> with . maybe). This also means that there is no handholding: very little is being checked for sanity (like in C++), your lua script can crash the server, and don't even think to run Lua script sent by the client, exploits are possible.

Things that are accessible in the C++ global namespace :: are bound to Lua in the table engine, and things that are in server:: are bound in server. This means that the crypto functions are in engine. There is an extra predefined table, spaghetti, which holds user defined events (see later) and a field, spaghetti.quit: once this is set to true, it cannot be unset, and the server will shutdown as soon as possible. This way to shutdown the server is preferred to just terminate the program from lua, or even from calling the sauerbraten shutdown functions from Lua.

Cubescript has been totally stripped. variables and commands are exported to Lua in the table cs as variables that you can read/write and functions that you can call.

spaghettimod tries to minimize the modifications to the vanilla code. This is reflected also in the way that C++ and Lua interact. Whereas Lua can access almost all the internals of the sauerbraten server, the interaction C++ -> Lua happens through

• binding
• calling script/bootstrap.lua at boot
• issuing events

The default bootstrap file just export two event related helpers (see next section), and calls the files in script/load.d/, which have to follow the naming scheme # # -somename.lua, where *# # * determines the relative order in calling the files (from lower to higher).

Events

Events are calls that the C++ code makes to Lua. When a specific even occurs, the engine runs this code:

local argument_table = {
    -- event specific named arguments
}
local listener = spaghetti[event_type]
if listener then listener(argument_table) end

The arguments are usually linked directly to C++ function variables, and the modifications you do in Lua are reflected in C++. Some arguments might be read only.

If the event is cancellable (with semantics specific to the event), the argument table contains a field skip, which if set to true, once the listener returns, causes the event to be cancelled. Cancellable events are issued before "side effects" take place, and non cancellable events after.

The number and kind of events is in flux, the arguments passed correspond, most of the time, to the C++ function variables, and the exact meaning of cancellation depends on the kind of event. Hence it's rather pointless to write down a list here, since it would need to constantly refer to code lines. You can work out a list of event with some grep commands.

• cancellable events: grep -REho "spaghetti::simple(const)?hook.*\)" engine/ fpsgame/ shared/ spaghetti/spaghetti.cpp | sort | uniq
• non cancellable events: grep -REho "spaghetti::simple(const)?event.*\)" engine/ fpsgame/ shared/ spaghetti/spaghetti.cpp | sort | uniq The results are in the form spaghettimod::issueevent(event_type, args...), where issueevent is one of simplehook, simpleevent, simpleconstevent. The event_type is either a N_* enum which correspond to server.N_* in Lua, or spaghetti::hotstring::event_type, which means "event_type" in Lua.

So far this is the only hardcoded behavior, but the script/bootstrap.lua that comes with upstream adds two functions: spaghetti.addhook(event_type, your_callback, do_prepend) and spaghetti.removehook(token). They implement a simple event listeners multiplexer: you add a listener with local hook_token = spaghetti.addhook(event_type, your_callback), and you remove it with spaghetti.removehook(hook_token). Hooks are called in the order that they are installed, and you can force a hook to be put first in the list with do_prepend = true.

Caveats on bindings (important!)

ENetPacket structures are transparently wrapped to use the native reference counting. This renders enet_packet_destroy impossible to use directly without introducing a double-free bug. For this reason, enet_packet_destroy is not bound to Lua. Furthermore, packetbuf uses reference counting always, regardless of the growth value.

In C++ the cryptographic functions return generally pointers to void* and have to be freed. Lua returns and takes strings with literal or binary hashes (grep -F addFunction shared/crypto.cpp).

The original sauer implementation of hash swaps the nibbles (e.g. byte 0x4F is written as 0xF4). This is kept for compatibility, but if you want to get a correct tigersum use engine.hashstring(yourdata, true).

ucharbuf, vector<uchar>, packetbuf now have method versions for sendstring putint putuint putfloat (they return the object itself so you can make a dot chain), getstring getint getuint getfloat.

Some C++ structures that represent binary buffers map to Lua strings by accessing the char* (or void*) pointer: ENetPacket (read only), ENetBuffer (read-write), ucharbuf (read only) (grep -F lua_buff_type engine/server.cpp fpsgame/server.cpp shared/crypto.cpp).

Some functions that require a binary buffer are proxied by functions that take strings, or functions that require an output buffer just return a new string (along with the original return, if applicable): enet_packet_create, decodeutf8, encodeutf8, filtertext, hashstring, genprivkey, processmasterinput... (grep -E '\.add.*\+\[\]' engine/server.cpp fpsgame/server.cpp shared/crypto.cpp).

luabridge, the library I use to bind C++ stuff to Lua, allows only one constructor to be bound (find out which with grep -FB 1 addConstructor engine/server.cpp fpsgame/server.cpp shared/crypto.cpp).

The static const parameters in fpsgame/{ctf,capture,collect}.h are now modifiable, as well as some const arrays in fpsgame/game.h (itemstat, guninfo).

Not all fields of ENetHost and ENetPeer are exported. As a rule of thumb, those that are clearly meant for internal usage by enet (for example the lists of packet fragments) will be unavailable.

The scripting environment

The default boostrap code adds script/ to the LUA_PATH, to ease require.

utils contain some functional programming utilities, ip and ipset object that are already documented in [kidban](https://github.com/pisto/kidban/tree/master/maintaining-docs# modules), and other generic helpers.

std contains the standard modules. I am too lazy to write a documentation for these before someone actually shows interest in using my code. Feel free to contact me if you want to write your own modules.