Object Files

MLWorks Object File Format

The MLWorks compiler creates object files foo.mo from Standard ML source files foo.sml. These object files include dependency information, type information, intermediate forms, and objects, including a 'setup function': an ML function which when applied to its own closure executes any runtime behaviour (producing output, etc) and returns an ML value ('the 'module value') which represents the results of any top-level declarations in the source file.

The format of object files is platform-independent, apart from the machine code itself. There is an endian-ness check on the first word in the file, which ought to allow files generated with one endian-ness to be read with another.

Module Table

The runtime maintains a "module table", which maps a module's name to a timestamp and a module value. The module names are not filenames (although the module names produced by the compiler are in fact based on filenames).

The module table is stored in the ML heap and saved and loaded in image files, in a global called "module list", and also is accessible from ML (MLWorks.Internal.Runtime.modules, an environment item called "system module root"). When running the interpreter, this is cleared down to just the two pervasive modules; when running the batch compiler, it's cleared completely. See the documentation on image files and on environment items, when I have written it.

Loading Object Files

Object files are designed so they can be loaded by two different mechanisms:

• by the compiler, when it encounters a "require" declaration in an ML source file:

  require "../main/code_module";
  require "../rts/gen/objectfile";
  require "../debugger/debugger_types";
  require "enc_sub";
  require "encapsulate";
  

• by the runtime system, when specified on the command line:

  rts/runtime-g -MLWpass xyzzy -save pervasive.img __builtin_library.mo __pervasive_library.mo xyzzy
  

When loaded by the compiler, all parts of the object file are used: the dependency information is checked, the type information is combined into the type environment in which the current source file is elaborated, the intermediate forms can be combined with the current sources when compiling (potentially for optimisation such as inlining or even defunctorisation). Finally, the code section can be loaded. If the module value or behaviour is required (by the "interpreter"), the setup function can be created from the code section and then executed; this does not happen when running the batch compiler.

When loaded by the runtime system, much of the object file is disregarded. However, the dependency information is checked, and then the objects are loaded and the setup function is called.

Overall Structure

The object file consists of the following sections, in order:

• Header;
• Dependency (consistency) information;
• Type basis;
• Parser environment;
• Lambda environment;
• Code and other module elements.

Serialization

The object file serialization system works as follows:

• (30-bit) ints are written in 4 bytes, big-endian.
• booleans are written as single bytes (0 is false, 1 is true).
• "optimized [positive] ints" are written:
  • if under 254, in a single byte;
  • if under 65535, as a byte 254 followed by two bytes;
  • otherwise as a byte 255 followed by four bytes (big-endian).
• "extended strings" are padded to a multiple of 4 bytes with at least one zero byte.
• "sized strings" are written preceded by their size as an optimized int.
• "optimized strings" are interned: each new string is given an index, counting from 1, and each string is written either as its index (if it has been seen before) or as a zero byte followed by the sized string.
• "symbols" are written as optimized strings.
• lists are written as an "optimized int" for the length, followed by the written elements.
• maps are written as an "optimized int" for the map size, followed by the map elements (key followed by value).
• pairs and other fixed-length tuples are simply written one element at a time.

Object File Header

The object file starts with a header, with the following layout. Each field is a 4 byte big-endian integer.

Offset	Name	Description
0	magic	GOOD_MAGIC: 0x1ADE6818 (450783256) or OLD_GOOD_MAGIC: 0x3ade68b1 (987654321) (but not NOT_SO_GOOD_MAGIC 0x1868DE1A (409525786) which indicates an endianness error).
4	version	OBJECT_FILE_VERSION
8	code_start	Offset of serialized objects from start of file.
c	cons_size	Size in bytes of consistency information.
10	parser_size	Size in bytes of parser information.
14	type_size	Size in bytes of type information
18	lambda_size	Size in bytes of lambda environment.
1c	total_strings	Total number of "optimized strings".

When the compiler outputs an object file, the number it puts in the version field is the OBJECT_FILE_VERSION constant from the objectfile.h runtime source file which was used to generate the __objectfile.sml generated source compiled into the compiler. When the compiler reads an object file, it compares the version field with this same number (and raises an error if they differ).

However, when the runtime system loads an object file, it raises an error if the version field is less than the OBJECT_CODE_VERSION constant. This constant may not be the same as OBJECT_FILE_VERSION, and equality is not required. Those two factors may be useful when cross-compiling or otherwise tinkering with compatibility between compilers and runtimes. On the tip sources, both constants are 20. In MLWorks 2.0, both constants were 19.

Consistency Section

The consistency section format is written as a serialized list of consistency items. Each item is a pair of a string (a module name) and timestamp. The timestamp is an integer stored as a pair of 30-bit ints (the quotient and remainder of division by a million). The compiler gets these times from FileTime.modTime (see unix/__filetime.sml and win_nt/__filetime.sml): it's basically a number of seconds since some epoch.

Each consistency item refers to a single module, and the first item refers to the module in the current object file. The timestamp is the modification time of the source file (although this fact is not used or checked by the module loader).

The runtime module loader records the first item on the consistency list, and unless the runtime was invoked with the -relaxed switch, it goes down the rest of the consistency list, doing a consistency check. If the list includes a module name which isn't in the module table, the loader stops with an error. If the module name is found in the table but the timestamps differ, the loader prints a warning and continues.

Code Section

The code section has a header, followed by a sequence of objects. The loader reads each object, creates it on the ML heap, and puts a boxed pointer to it into the module's closure.

Offset	Name	Description
0	object	The number of objects in the code section.
4	clos_size	The number of slots in the module's closure.
8	objects	The objects themselves.

Each object is preceded by an opcode, which determines its format:

Opcode	Value	Description
OPCODE_CODESET	0	Representing a code object. Details below.
OPCODE_REAL	1	An int index, an int size (2), and a double-precision floating-point number, to be boxed and put into the module's closure at that index.
OPCODE_STRING	2	An int index, an int size (the string length + 1), and an 'extended' string (padded with zero bytes to word alignment) to be put into the module's closure at that index.
OPCODE_EXTERNAL	3	As for OPCODE_STRING, except that the string is a module name. The name is looked up in the loader's module table, and the corresponding module value is put into the module's closure at that index.

Code Objects

The serialization of code is rather complex; this section assumes a knowledge of the in-memory structure of code objects. Each code object is serialized as follows:

Offset	Name	Description
0	procs	The number of code items in the object.
4	size	The total size of the machine code, in bytes, rounded up to a multiple of 4.
8	names	The procedure names in order, as strings, each preceded by its length as an int.
-	intercept	An int: if this is non-zero then the loader will create an ancillary interception function array.
-	items	The code items themselves (see below).

Each machine code item within the object is structured like this:

Offset	Name	Description
0	position	The offset into the module's closure for the resulting code pointer.
4	spill	The number of non-GC stack slots.
8	saves	The number of callee-save slots.
c	leaf	A flag: non-zero means this is a leaf function.
10	intercept	The intercept point code offset.
14	args	The number of stack argument slots.
18	size	The number of bytes of machine code.
1c	code	The machine code itself.

Other sections

TODO: Write documentation on the parser environment, the type basis, and the lambda environment sections.

Loading

There are two bodies of code for loading object files: the runtime's loader in rts/src/loader.c and the compiler's loader in main/_encapsulate.sml.

When the runtime's loader has loaded all the objects and created the module's closure, it calls the closure (with the closure itself as argument), and adds the result to the module table, with the module name and timestamp read from the consistency section.

The compiler's loader doesn't create code objects from the code section; it creates ML objects (of type Code_Module.wordset). If the interpreter needs to run the code, it calls into the runtime to convert a wordset into a code object (and then creates and calls the setup function's closure). See interpreter/_interload.sml.