Lua: Cooperative Multitasking Using Event Driven Messaging

[wpferguson]

Is your feature request related to a problem? Please describe.

Lua is single threaded, therefore only one script can be executing at a time. Lua, as embedded in darktable, uses an event driven architecture so that multiple scripts can be started and waiting on an event to perform an action. If one script is executing and an event occurs that would start a different script then the event is "held" until the currently executing script finishes it's task. So, we have FIFO multitasking.

When a long executing script, such as cache generation for ~3000 images at 2 to 3 images/second, starts running all other Lua scripts that are called to execute and held until the long running script finishes executing.

Lua has a multitasking concept called cooperative multitasking. It consists of a main script with co-routines which yield to let other co-routines run. The event driven architecture used by Lua in darktable doesn't lend itself to a yield/resume in the standard Lua way.

Describe the solution you'd like

Using event driven messaging, as described in #17295, it would possible to yield and resume. It would require a simple FIFO scheduler script to work. It would work as follows:

• The FIFO scheduler would start
• The long running script would start.
• Long running script would query the scheduler to see if it's running
• If the scheduler is running then cooperative multitasking is enabled
• The long running script starts executing and performs some work.
• Long running script sends a yield message to the scheduler
• The scheduler queues up a resume message for the long running script
• The scheduler sleeps for a small amount of time
• Any other Lua script that is waiting starts and completes until no more scripts are waiting
• The scheduler pops a resume message off the queue and sends it to the appropriate script
• The long running script performs another block of work then yields....