Collection of ideas for VM improvements

[alaviss]

Any structure that is not int-based or float-based is stored in the VM using PNode.

This is needlessly wasteful, given that the nodes are meant to construct the syntax tree of any given program and contains metadata that is completely unused by the VM.

With the move to DoD AST brewing in #139 (experimental implementation in #144) which will specialize the AST further to fit its purpose for codegen, it makes little sense to keep using PNode for the VM and add unnecessary constraints to the new AST design.

Since the VM is fairly self-contained, I'd propose that we move VM data presentation away from PNode before proceeding further on the DoD AST effort.

Any help in this endeavor would be appreciated!

[zerbina]

Hi, I'd be interested in working on this. I have some rough ideas on how to tackle this task and have already started to hack around in the vm related code a bit (mainly vm.nim).

If it's okay for me to work on this task, should I create a draft PR with the beginnings of my implementation attempt or should we discuss direction/implementation details here first?

[alaviss]

It'd be great if you could do both, so we can have a good idea on how you are gonna hack it out :)

If you can, please join us on our Matrix space or #nimworks-dev/#nimworks on libera.chat. That way, we would be able to help you faster. Also, I'm doing some research on VM stuff too, so it'd be a great way to compare notes :D.

Thanks for contributing.

[zerbina]

PR is up: #242

[zerbina]

As discussed on Matrix, here's an overview of the current status together with some context, ideas and an attempt at a roadmap.

To make discussion via replies easier, I tried splitting all of it up at thematic breaks

First, some overall goals and non-goals

Goals:

• Safe memory reading/writing. Prevent the guest from corrupting/crashing the host (i.e. the compiler)
• Make the VM a more first-class backend, supporting most, if not everything, that is available with the C/C++ backends
• Get the compiler to run inside the VM. Basically: nim e compiler/nim.nim -- c compiler/nim.nim. Or even
  nim e compiler/nim.nim -- e compiler/nim.nim -- c compiler/nim (Build the compiler with a compiler running inside a VM instance running inside another VM instance itself :D).
  • Why: To make sure that the VM works as it's supposed to, I'd say it's essential to run some complex real-world Nim code inside it. The Nimskull compiler itself makes for an easy choice, due to it being large and complex while also being no external dependency
• Make the VM code easier to understand by documenting opcodes, documenting procedures, reducing or eliminating the use of ref types and by using func whenever possible
• Support interactive execution of VM instructions (both as a tool for users and for compiler debugging)
• Create a framework for testing and debugging the VM

Non-goals:

• Repurpose the VM for running anything else than Nim code
• Turn the VM into a fully seperate entity
• Guard against side-channel attacks

[alaviss]

We can probably come up with an arrangement so that testament could test the VM. That'd let the VM be subjected to the full test suite of the language and is more attainable short-term than running the compiler :)

[zerbina]

Some concepts and terminology.
I'll make use of these when describing ideas. The concepts together with their names are not necessarily meant to be final; they're just what I currently arrived at.

Locations and memory

The meaning and function of a location is mostly based on the description at the top of the language manual. Basically, they act as the logical storage for values. In context of the VM, they are split into two categories: atomic locations and complex locations. Atomic locations contain values that are atomic (they can't be further decomposed into smaller values). Complex locations contain values that are made up of smaller values (which can be either atomic or complex values). VM instructions operate on locations via handles. A handle can be seen as a way to address a location.

Atomic value types are:

• int (all variants)
• float (all variants)
• seq
• string
• set
• ptr/pointer
• ref
• function object (closure, function pointer, etc.)
• NimNode

Background: I came up with this set of types by looking at what kind of values the original VM instructions operated on

Complex value types are:

• object/tuple
• array

A value is also either managed or unmanaged. Managed values are:

• complex values that are either fully or partially made up of other managed values
• atomic values of seq, string, function object, ref or NimNode type are managed values. They require special handling regarding copying/moving and destruction.

At the physical layer, values are stored flat in contiguous memory. A location containing a compound value overlays the memory region encompassing all the locations of it's sub-values.

A VmType describes the type and memory layout of a location

Pseudo code of VmType:

type VmType = object
    size: int # The size-in-bytes of the value (including all sub-values, if any)
    alignment: int # The alignment requirement of the value

    case kind: ValueKind # The kind of value type
    of vkInt, vkFloat, vkSet, vkString, vkNimNode: # These don't need extra information
    of vkSeq:
        seqElemStride: int # The amount of bytes between two consecutive elements
        seqElemType: VmType # The vm type of `T` in `seq[T]`
    of vkPtr, vkRef:
        targetType: VmType # The vm type `T` in `ptr T`
    of vkObject:
        fields: seq[tuple[offset: int, typ: VmType]] # The fields of an `object`/`tuple` type
    of vkArray:
        elemCount: int # The number of elements in the array
        elemStride: int # The amount of bytes between two consecutive elements
        elemType: VmType # The vm type of `T` in `array[..., T]`

Variant objects are currently not accounted for. I still need to come up with a solution on how to handle them.

The following nim type:

type Obj = object
    x: int32
    y: int16
    z: int32
    a: array[2, bool]

would be represented as a VmType like:

let i32T = VmType(kind: vkInt, size: 4, alignment: 4)
let i16T = VmType(kind: vkInt: size: 2, alignemnt: 2)
let aT = VmType(kind: vkArray, elemCount: 2, elemStride: 1,
    elemType: VmType(kind: vkInt, size: 1, alignment: 1)
)
result = VmType(kind: vkObject, size: 4 + 4 + 4 + 2,  # padding must be inserted after `y`
    alignemnt: 4,
    fields: @[(0, i32T), (4, i16T), (8, i32T), (12, aT)] 
)

The in-memory layout would be:

# 00   01   02   03   04   05   06   07   08   09   0A   0B   0C   0D   0E
# [ x: int32         ][y: int16] - - - - -[ z: int32         ][(0)][(1)]
# ^                                                           ^        ^                                                             
# |                                                           |____a___|
# |_________________________________Obj________________________________|                                                               

A location is required to be accessed via a handle of the same, super or sub-type (as described in the manual). Unlike the C/C++ backend, this in enforced in the VM. var a: int; cast[ptr float](addr(a))[] = 0.0 will fail at run-time (in the future, trivial examples like this should probably also fail at compile time. But that is outside the scope of the VM).

byte sequences (array, seq) are overloaded to also represent dynamic (i.e. untyped) locations. Dynamic locations don't have their size and type fixed at creation time. They can be aliased with: any unmanaged type, a ref type, function objects. The only additional requirement being that the type fits the byte sequence. In the VM, this is enforced:

• var a: array[4, byte]; cast[ptr int32](addr a)[] = 0 is valid
• var L = 3; var n = newSeq[byte](L); cast[ptr int32](addr n[0])[] = 0 is not (due to int32 not fitting into 3 byte), and will result in a runtime error
  Memory allocated with alloc(X) is treated as a byte sequence of length X

Future direction: Managed types that are only considered managed due to them containing either ref or function object types should also be allowed to alias untyped locations

The byte sequence aliasing is required to allow runtime interpretation of memory in some from. Some rule similar to the one I used here should probably be also added to the language definitions in the manual

Future direction: Overload the untyped type to declare locations as being alias-able with unmanaged types. bytes should stay as bytes

An untyped memory region refers to the memory region occupied by elements of a byte sequence.

Values can have their in-memory representation copied via copyMem with some restrictions. Both source and destination must either be typed or untyped locations.
If the location pointed to by an operand (dest/src) is typed, the size argument is required to either be 0 or match the in-memory size of the location.
If it's untyped, the size argument must be less than or equal to the size of the untyped memory region starting at the given location.

The following code is illegal and will fail during run-time:

var a, b: int32
template sub(x): cast[pointer](cast[int](addr x) + 2)
copyMem(sub(a), sub(b), 2)

When the destination is a typed location, the source must be a typed location with the same type as the destination. Due the overall architecture here, it is easy to enforce this at runtime too. When the destination is untyped, the source can be either typed or untyped.

The requirements here are more strict than those of the previous iteration in the PR. This also makes them somewhat less complex, which, I think, is a good thing.

Registers

A register holds one of the following:

• nothing (empty)
• int
• float
• address (a raw pointer basically)
• handle
• NimNode

While ints, floats and NimNodes could also be directly accessed in memory via handles (seqs, sets and strings are handled this way), they are loaded into registers first. This has a two reasons:

• Compatibility with vmgen
• Efficiency. They are small, self-contained and often used.

To de-reference pointers, they have to be first loaded into a register. The rationale here is to speed up repeated de-referencing of an unchanging pointer, while also making memory safety easier. When memory is deallocated (either via dealloc or when a ref/seq/string gets gc'ed), all loaded addresses are marked as potentially invalid. They're are validated again at the moment of their next de-referencing. If validation succeeds, they are marked as valid again and if not, the VM exits.

handles that reference locations of newed objects keep them alive for the duration of the reference.

Future direction: Track the id of the owning memory region for addresses. Only invalidate address and handle registers that contain references into the freed region.

[zerbina]

Idea: Restructuring the execution loop

Right now, all of the execution logic is tangled up into one big function (rawExecute). On a logical level it can be broadly split up into the following:

• Instruction fetching. In the current implementation, instructions must come from a linear buffer (a seq)
• Stack frame management. Pushing/popping stack frames, handling guest function returns/yields. This part overlaps with control-flow handling a bit
• Control-flow handling. Jumping/Branching, returning/yielding and also exception handling (only the control-flow part, like jumping between finally blocks, etc.)
• Lazy code generation for newly called functions
• Instruction execution. The heart of the engine

The basic flow is like this: Fetch instruction -> execute instruction -> handle control flow / frame management -> repeat.

Instruction execution at the lowest level should not need to know about things like: program counter, stack, (guest) exception handling. It should simply execute the instruction fed to it and return an expectation on what should happen next, along with some data. Expectations are:

• nothing special needs to happen. Just continue with the next instruction
• jump: go to a non-adjacent instruction
• call: enter a given function
• return/yield: leave the current frame
• raise: the guest code raised an exception
• fail: the instruction was given invalid by the guest (e.g. nil pointer, index out-of-bounds)
• abort: an instruction's internal preconditions were violated (e.g. an assertion failed). The VM might be in an unstable state and should probably terminate

The instruction execution stage doesn't know what do with these, it just reports them back to the one who invoked it. During normal operation of the VM, this would be some form of director/orchestrator/supervisor (can't think of better name right now). It would then examine the execution result and act accordingly. It could, for example: return in-VM control flow back to the calling guest function (in case of return), run code generation for a new function (in case of call), find and go to the closest encompassing finally block (in case of raise), quit VM execution.

The instruction execution stage may only read and modify VM heap memory and the registers that are operands to the executed instruction. Things like file-system I/O and echoing should be done in VM callbacks.

Splitting up the execution into multiple physical components could be done in several ways. Probably the most simple one, would be to just put them into their own procs (execInstruction, handleCall, handleRaise, etc.). A "director" would then invoke each, as required, and handle communication between them. While simple and easy, this would most likely be a lot slower than the loop we have today.
Another approach would be to use continuations (CPS; @alaviss also mentioned this on Matrix. Could also function as a good benchmark for Nim-CPS :D). This would most likely require macros (true in case of Nim-CPS) and I'm not sure if they are available during bootstrapping.

Some cool things all of this would enable (or make a whole lot easier):

• Fuzzing of instructions. Testing of the VM in general
• Easier (and maybe also more efficient) integration of debugging facilities
• Easier separation of the VM from the rest of the compiler (only meant for better testing/debugging, not for separating it from Nimskull as a whole)
• A small tool for manually feeding instructions to the execution engine running in a standalone environment (separate from the compiler). This could be useful when designing new instructions)
• Resumable VM invocations
• Fetching instructions from disk

All of this is just an idea, nothing has to be done the way described here.

[saem]

Much of what you're describing, fetch, decode, execute, etc, reminds me of a CPU pipeline -- that's a very good thing.

The lazy code generation being lifted out makes a lot of sense right off the bat. Much more so than say treating it like an "instruction cache miss".

The part I'm not sure I understand is the separation of execution, stack frame, and control flow aspects. Frame allocation for example is a form of memory management and PC/control flow along with various instruction execution seems connected. I'm missing something.

[zerbina]

The idea there was to be able to have the core instruction execution as a function over memory (registers and heap) and some execution state, yielding a result. This would, in my opinion, allow for the instruction execution code to be simpler while also making both the run-time (e.g. for debugging) and build-time (e.g. for unit testing) swapping of control-flow/stack-frame behaviour possible in a simple manner.

It's not strictly necessary for the instruction execution to know where the registers it operates on are located nor how they are managed. If stack frame handling is not managed by the core execution engine, it is easier to change how it works (could be useful for experimenting but also for interactive debugging)

The control flow related things are a bit trickier. While I originally thought about removing all knowledge of the program counter from the instruction execution, it's probably a good idea to still let instruction execution have access to it, both for simple in-function jumping and for the case where instruction behaviour changes depending the PC. All other control-flow related things would be offloaded into a dedicated component

[zerbina]

Debugger/Debugging

My overall idea regarding debugging would be, to only provide a low-level framework in the VM. The debuggers' implementation would live outside the VM's code.

Things the framework/API should provide/allow:

• modifying and querying the contents of the heap and registers. Querying of constant data (constants and types)
• querying of generated code for functions
• insertion of new instructions. Either at arbitrary places or only before the next instruction, not sure yet
• manual control-flow (force return, function call or jump). Modifying the program counter
• event hooking (call, return, yield, raise, error, etc.). Ability to override the default behaviour from inside the hook (e.g. to ignore a raise)
• (maybe) a facility to receive notifications of modifications to for specified registers and heap locations. This could be emulated by the debugger with instruction stepping and then comparing pre and post state, but that might be too slow to be practical

Stepping, breakpoints, watchpoints and similar things are all implemented in the debugger via the aforementioned APIs

[zerbina]

Current state

This describes my local changes. I haven't pushed them yet, so as to not make review of my PR any harder.

Overview of what I got so far:

• Flat memory representation/storage of values
• Logic to translate PType to internal VmType
• PNode is no longer used for storing data. It's only used for NimNodes and typdesc
• The base to support enforcing of memory/location safety
• A separate VM heap. Previously, PNodes were abused for newed values

Details

In order to not throw too much working parts away at once, I tried to not change instruction semantics in ways that would require large adjustments or structural changes in vmgen (to keep working for now). Most changes to vmgen are modifications to some leaf procedures/logic (e.g. logic for constant data creation had to be changed).

Only one new instruction was added. It is needed as a temporary measure to keep vmgen working while correctly supporting new

Values are stored flat in memory (described in one of the posts above) and read/modified via handles and VmType. Globals and constants own and store the pointer to their value together with it's VmType. When loading globals or constants, the pointer is turned into a handle and then stored, together with the type, in a register.

A handle is simply a ptr Atom internally. To ease development, Atom is currently a helper {.union.} type. The memory of a value's location is allocated and managed by VmMemoryManager.

To match the expectations of vmgen, registers can 'own' a location. Once an owning register is transitioned to a different state (i.e. reused), the location is freed (deallocated)

At the end of execution, the result of a VM invocation is unmarshalled from it's atom based representation back to it's PNode based one. This allows the rest of the compiler to stay oblivious to the new data representation for now.

What is missing (to restore previous functionality):

• var {int|float|ptr|pointer|NimNode} function parameters don't work yet, as I removed direct register referencing from the code.
  The correct location to address this at would be vmgen, but since I'd try to avoid that for now, a simple temporary workaround would be to treat the register's content as a location and just produce a handle (pointer) to it. This won't be a problem, since it's ensured that registers always have a fixed address.
• getAst support. Haven't really look into this but should be simple to get back working
• Variant object support. Probably the hardest task in this list
• FFI support. Shouldn't be too hard
• Some instructions are still commented out but most of them are easy to reinstate
• opcOf might be a bit more complex
• Lots of callbacks currently don't work anymore but easy to restore

At this point, bootstrapping should work again and all test should be green

What is left to get it to a stable state:

• locations are not properly reset in a consistent manner (thus leaking memory)
• Ref-counting and cleanup of ref objects created via new is missing
• The code needs to be cleaned up (e.g. dead code removal, name changes, code style adjustments, etc.)

Other things:

• More efficient data structures need to be used for a bunch of things. Right now, I simply use seq for more or less everything
• set still uses the early seq-based representation and operation
• The implementation of string and seq atoms (add, setLen, etc.) is rather inefficient

Proposed roadmap (ordered)

Near term

• Split up and merge the stack frame handling changes
• Rebase the big PR onto devel and push local changes
• Restore missing functionality and get the PR to a stable state (described above)
• Gather reviews for the PR / act on them
• Fully implement memory/location safety feature
• Next review round
• Fix the other things
• Next review round
• Merge the PR

After that (These are all meant as single PRs)

• Unrelated to the VM, but I think the DoD AST effort can proceed at this point
• Document all opcodes
• Rewrite vmgen and get rid of vmgen related hacks in the execution engine
• Use the knowledge gained from the vmgen rewrite to refine the instruction set
• Rethink/rework error reporting in the execution engine
• Design VM debug interface / interactive execution
• Implement VM debug facilities
• Gather what features are missing to support running the compiler in the VM
  • Probably something like: alloc, copyMem, full openArray support, GC_ref and GC_unref
• Implement the missing features and get the compiler running in the VM
• Make the compiler running in the VM produce a working compiler binary
• Setup test cases for doing the above as part of the test suite
• Tune the performance of both vmgen and the execution engine

Splitting up the PNode change into smaller tasks won't be easy and would probably also induce quite some overhead (in terms of work), in order to only make stable increments (as required for devel).

As an alternative, I thought that it might be a good idea to create a temporary branch, where, after an initial review round, the big PR is merged. The remaining items in the near-term bracket could then be done as smaller standalone PRs (easier to review) against the temporary branch, without the hard requirement of each PR being perfectly stable/finished.

When finished, the temporary branch would be rebased/fixed/squashed and then merged into devel. All items in the "After that" bracket are then done as standalone PRs against devel again.

[saem]

Near term

Looks good, anything that we're worried about is likely going to be best addressed by adding some specification (tests) and if those are green at least we'll have memorialized the intention(s) and should any issues arise we can use those to navigate.

After that

Yes, DOD AST should be unlocked, and all the remaining items are big value adds.

Splitting up the PNode change into smaller tasks won't be easy and would probably also induce quite some overhead (in terms of work), in order to only make stable increments (as required for devel).

Splitting up PNode change only works if it's done by use case within the VM and not for all PNode related items in the VM. Then again I didn't look at it as deeply, so I'll go with your assessment.

As an alternative, I thought that it might be a good idea to create a temporary branch, where, after an initial review round, the big PR is merged. The remaining items in the near-term bracket could then be done as smaller standalone PRs (easier to review) against the temporary branch, without the hard requirement of each PR being perfectly stable/finished.

When finished, the temporary branch would be rebased/fixed/squashed and then merged into devel. All items in the "After that" bracket are then done as standalone PRs against devel again.

The incremental additions to a bigger branch would make it easier to digest in logical chunks.

[zerbina]

Splitting up PNode change only works if it's done by use case within the VM and not for all PNode related items in the VM. Then again I didn't look at it as deeply, so I'll go with your assessment.

PNode is, broadly speaking, used for two things: data representation and AST. In my opinion, doing the data representation change in multiple steps (e.g. heap, registers, constants) doesn't make much sense as it's parts are heavily intertwined.

But maybe I'm only looking at it from the wrong angle due to having the change completed already.

[saem]

Hard to argue with working code.

[saem]

Stack frame changes are in. Yay!

If you want to put up a draft that's be cool.

[saem]

Just to make it explicit, the roadmap is good.

[zerbina]

Error handling

Where and what kind of errors can happen in the VM?

• Internal errors in the execution engine (e.g. due to instruction preconditions being violated)
• Internal errors during VM codegen (vmgen)
• Codegen issues due to code unsupported by the VM
• Errors due to invalid guest input/action (e.g. index out of bounds, trying to modify sem-checked NimNode)
• Errors inside VM callbacks. These can be further split up into:
  • Input validation failure
  • Exceptions raised by callback code due to either bugs or un-validated guest input
  • External failures (e.g. I/O errors during file system interaction)
• Uncaught exceptions raised by the guest
• Errors raised via the error proc
• There's also quit, of which I'm unsure of what it is (in regards to error handling/propagation)

The current situation

• Internal errors are raised via assert or internalAssert/internalError
  • This is for both the the execution engine and vmgen
• VM codegen issues (except "cannot eval") are raised via globalReport
  • "cannot eval" errors are raised as exceptions which are then caught in sem.tryConstExpr
• Inside the execution engine (this excludes callbacks), errors due to invalid guest input are inconsistently raised via either stackTrace, globalReport or internalAssert/internalError. Not using stackTrace means no guest stack-trace.
  • What stackTrace currently does: Generate a stack-trace; report it; if in REPL-mode, this is done with the doRaise strategy; do a localReport of the failure report; exit the VM
  • Exiting the VM due to errors is done via an injected return statement inside the stackTrace template (also documented by @haxscramper here). If it's not used at the scope of rawExecute, the VM is not terimnated. This currently leads to uncaught guest exception not terminating the VM (exectuion just continues after the raise statement; finally blocks are still executed though)
• Error handling inside callbacks:
  • Except at two places, guest input is never validated. Even when it is, execution of neither the callback nor the VM is aborted.
  • Stack-traces generated inside callbacks also don't show the correct invocation location due to the callback having no access to this information
  • If the callback raises due to either invalid guest input (e.g. IndexDefect) or external issues (e.g. IOError), in most cases, the compiler will simply exit with an uncaught exception, with no indication of where the issue originated:

    import std/hashes
    static:
      var arr: array[256, byte]
      let v = hash(arr, 0, 512)

    yields only:

    fatal.nim(53)            sysFatal
    Error: unhandled exception: index out of bounds [IndexDefect]
    

• error uses stackTrace and thus aborts execution
• quit in VM code reports that quit was called and then terminates the compiler (via msgQuit -> system.quit)

Callers of rawExecute can't, as of now, reliably detect if execution exited abnormally (which would also only include via stackTrace right now). In case of failure, an empty register is returned (which is also valid if the guest code has no result)

Guest exception handling is heavily centered around opcRaise, which currently means that no other instruction can kick-off exception propagation directly. Instead, instructions may signal that an exception should be raised via an error flag which then has to be tested by the guest (parseExpr/parseStmt does it this way).

---

Improvement proposal

Note that this proposal doesn't aim to "fix" all issues with VM error handling at once. Some fixes only make sense after further progress on the general VM overhaul happened (e.g. new vmgen, VM callback redesign/overhaul)

General problems:

• How to raise/propagate errors inside sub-systems (i.e. VM, callbacks, vmgen)?
• How to report the errors at the edge of the sub-systems?

The edges to consider are:

• compiler <- vmgen (prior to executing a statement/expression/proc/macro, codegen has to take place (only if it didn't already))
• compiler <- VM
• This includes the cases where:
  • A NimNode (PNode) tree result is expected (when executing a macro)
  • A value (PNode based) is expected (static expression)
  • No result is expected (static statements)
• VM <- vmgen (lazy codegen)
• VM <- callback

My proposal:

For errors that can't be handled inside vmgen (which are basically all of them) raise an exception that stores a SemReport. Return a result-like value at the edge of vmgen (genProc, genStmt, genExpr) and do the exception -> result translation there. Replace all usages of globalReport with raising an exception (internalError would stay for now).

Since the execution engine code has a call depth of 1 for the most part, it makes sense to do error reporting via rawExectutes return value. Instead of returning a register value (as is done now), a wrapped register index is returned. There are multiple approaches to do the wrapping, and I'm unsure which is the most fitting.

In addition, replace all internalAsserts and globalReports that are used in input validation with the new reporting mechanism.

Guest input validation issues in callbacks are reported either via an Option[SemReport] return value or a SemReport+hasError flag inside VmArgs. Ideally, callback functions would be annotated with {.raises: [].} in order to make sure that no catchable exception (e.g. IOError) escapes and is translated, but the csources compiler seems to be unable to do raise tracking reliably. Defects would still be problem.

Errors during both callback execution and codegen abort VM execution with a failure, propagating the report upwards.

If VM execution or codegen happened in a context where a result (both value and NimNode) is expected, in the event of failure, turn the report into a nkError. The guest stack-trace would also be generated outside of rawExcute now.

If VM execution or codegen happened in a context where no result is expected, turn the failure into a globalReport.

I'd propose re-using SemReport for reports for now (it's used for the existing reports), but there probably is a better way

---

While the error handling situation in regards to the VM would still not be perfect, it should be a bit better after the changes proposed here.

Thoughts?

[haxscramper]

I read through the proposal only once so maybe I kissed some nuances, but it sounds pretty reasonable.

As for SemReport reuse - it might make sense to implement a separate report category (VmReport) - I merged those tow together at the first run because they were interleaved in a lot of places. And sematics report errors can store VM errors if you find it necessary (rsemVmError, with vmError: VmError field)

[zerbina]

With the newly introduced handle/address checks, VM errors happen further down the call chain now, so it might make sense to use exceptions in the execution engine too.

I also thought about errors in vmgen a bit more, and as the VM is moving towards also being a first-class back-end, maybe they should be BackendReports instead of SemReport or VmReport.

For vmgen usage during constant expression or macro evaluation however, this would be a bit incorrect, since vmgen is not really used as a back-end there.

[saem]

Idea: Instruction Set Extension

There is a current push to move things into callbacks via vmops. I think this makes a ton of sense for 'syscall' like things, but I don't believe this is the correct thing for "instruction set extension".

Loose descriptions:

• syscall: call host functionality from the guest program
• extended ISA: adding new instructions to the base instruction set

The dividing line in the above is pretty fuzzy, admittedly. Looking at all the metaprogramming related opcodes and the code within vm.rawExecute that backs those instructions, I think instead of making them "syscalls" (via registered callbacks) a better bet would to keep them as instructions but an extended variety as they have to do plenty of register fiddling and the like.

---

Improvement Proposal

I think a slightly different facility that's more inspired by "instruction set extension" might be a better way to go. Perhaps TInstrType can dedicate a bit to indicate it's not the typical TOpcode and should instead be interpreted as a secondary enum (extended opcodes). The major change would be reworking the vm module to support the extended instruction set and a potentially parameterized TCtx.

# compiler/vm/vm.nim, assuming no parameterized TCtx

template makeExecute*(extRawExecuteSetup: untyped, extRawExecute: untyped): untyped =
  proc rawExecute(c: var TCtx, pc: var int, tos: var StackFrameIndex): YieldReason =
    rawExecuteSetup(c, pc, tos)       # template that does the pre-while loop setup
    extRawExecuteSetup(c, pc, tos)
    while true:
      let instr = c.code[pc]
      let ra = instr.regA

      if instr.isExtended:
        # extRawExecute(c, pc, tos, instr, ra)
      else:
        # rawExecute's core while loop

      c.profiler.leave(c)
      inc pc

  proc execute(c: var TCtx, thread: var VmThread): YieldReason =
    var
      pc = thread.pc
      sframe = thread.frame

    try:
      result = rawExecute(c, pc, sframe, extRawExecuteSetup, extRawExecute)
    # ... rest of code is mostly the same

The above is a very rough sketch, still need to sort out all the bind/import issues. But effectively one could declare an extended VM this way, the template shenanigans would be simliar to nimdebugutils. So start with the sketch above and then follow the necessary changes.

There would be a few major VM instances I imagine:

1. the vm without the compiler extension
2. the vm with compiler extensions
3. if someone is using the vm as a lib, then their extensions