Support of coroutines

[dwursteisen]

👋 Hello,

Recently, I've been enjoying playing around with luak.

I created a small game engine with Kotlin Multiplatform. The great thing about luak is that it allows you to create games using LUA scripting.

Behind the scene, it's using luak and as you can see, It's working splendidly 👏.

During my exploration, I delved into the LUA coroutine support and discovered that the implementation relies on the notify/wait system provided by Java (lua4j legacy). However, when it comes to the JavaScript version, the implementation is currently missing.

This led me to wonder if there's a way to address this gap. Despite my best efforts, I couldn't find a suitable solution to enable coroutine support in luak on the JavaScript platform.

I was hoping you might have some insights or knowledge about whether it's possible to support coroutines in luak on the JavaScript platform.

Thank you!

[soywiz]

Awesome! That looks great.

Regarding luak, I ported it sometime ago and I don't remember the details, but we can try to figure out if we can make it someway compatible with the kotlin coroutines. How are you currently implementing LUA coroutines? Is there a loop where you resume coroutines?

[soywiz]

Checking at it, I would say the best option would be making LuaValue.invoke and inheritors suspend. Which looks pretty impractical, since everything would end being suspend.

Since this is an interpreter, and it seems that a linear set of opcodes is executed:

korge-luak/luak/src/commonMain/kotlin/org/luaj/vm2/LuaClosure.kt

Line 213 in bd22c9f

when (i and 0x3f) {

Maybe if we could suspend the execution of the LuaClosure and resume it later, it could work. But looks like calls are done recursively, so we should be able to resume that execution too.

But again, that LuaClosure in the OP_CALL, ends invoking LuaValue.invoke, so... there should be a mechanism to restore that call stack anyway. Though at least the function evaluation could be suspend at any time because the interpreter is a while + switch. Not sure.

If we could for example make the execution of closures calling other closures linear (keeping a stack):

val functionLocals = Stack()
val functionCodes = Stack()
var currentLocals = Locals()...
while (...) {
   when (op) {
      OP_CALL -> {
            val result = functionCall() // or try catch and throw an exception when a LuaClosure would be going to be executed
            if (result is ExecuteFunction) {
              functionCodes.push(currentCode)
              currentCode = result.code
              functionLocals.push(currentLocals)
              currentLocals = Locals()
            } else {
               valueStack.push(result)
            }
      }
      OP_RET -> {
             currentLocals = functionLocals.pop()
      }
      ...
   }
}

Then we could suspend and resume it easier. There is a mix of normal functions and closure functions. So when executing a closure function we should return a value instead of executing the closure, so our linear executor can execute it.
All this don't seem trivial and I might be missing things. So this might be even more complicated. Though it should be achieveable with enough patience.

Another option is to wait for WASM+threads, and totally ignore the JS target.

[dwursteisen]

👋 thanks for your quick feedback.

I tried to see with kotlin coroutines if it’s possible to keep a continuation and use it later to run a suspendable code. You might guess it’s not possible. Another way would force, as you said, to mark every methods ´suspend´.

I agree with you that it’s not a good idea to go this way.

I didn’t thought about updating the LuaClosure. I’ll think about it doesn’t seems easy to do. 😅

as there is no viable solution yet, I’ll close the issue.

[soywiz]

BTW @dwursteisen could we chat somehow?

[soywiz]

Also we might try to replace in the whole codebase “fun “ with “suspend fun “ and then remove places where it is not required and see how it behaves in terms of performance. Might even be feasible.

[soywiz]

Good news. I managed to get it working in a somehow reasonable way.

[dwursteisen]

👏 Congrats !

I checked yesterday and a bit today: the lua website provide documentation and example of implementation of the LUA VM, including coroutines implementation. (see https://www.lua.org/source/5.4/lcorolib.c.html for example)

It seems that they implemented it without thread but by playing with the stack. I didn't took the time to fully understand how it's working in the C implementation.

Regarding your implementation, my understanding is that, if there is a coroutine in the LUA script, you have to call the callSuspend method instead of the call method? Am I right?

I saw that you tag it. Is it available on a public repository so I can try it right away in my engine?

Thanks :)

[soywiz]

LuaThread in java is implemented as https://en.wikipedia.org/wiki/Cooperative_multitasking.
While one thread is running, the other is paused.

In the case of C, if you can switch the stack pointer + the program counter, you can do preemptive threading super fast, at the cost of changing two cpu registers. Maybe that's the case.
IIRC: setjmp/longjump could do the trick: https://en.cppreference.com/w/cpp/utility/program/setjmp
In the case of the Kotlin implementation, this is different. We cannot switch the stack, since Kotlin doesn't offer that functionality; we could do that in K/N somehow, but wouldn't be able to do that on JS or the JVM.

So in the JVM luaj used Threads. Each one having their own stack, but emulated the non-preemptive multitasking by pausing the thread calling the coroutine, and then the thread of the coroutine, so only one could be executed at a time.

In the case of Kotlin I implementing using suspend functions. Instead of switching stacks/pointers, Kotlin does what C# started doing when the async/await pattern appeared. Suspendable functions, instead of having the locals in the stack, they are allocated in the heap, and functions are implemented as state machines. Like a giant while

• switch changing the state every time the function can be suspended, so you can resume the function by just calling it again with a label to where resume it.

---

In any case, this should work. But I might have not covered all the cases. So feel free to report if you find something strange.

BTW: If you have time to chat for a totally different matter at some point, let me know :)

[dwursteisen]

Sure, we can organise something.
You should be able to contact me through the Kotlin Slack or through twitter to arrange a chat.

I tried build locally the project. I didn't check how the internal plugin of korge is working but I didn't find any task to publish the project in any maven repository (ie: maven central or even maven local). Do you plan to publish it somewhere or maybe you can guide me so I can publish locally the JVM + JS version to test it? (I saw that I can comment other platform to build only what I need in the build.gradle.kts)

[soywiz]

I'm going to take a few days off. Let's talk after my break.

Regarding to using the project. Right now there are no tasks for publishing. I was not aware of anyone using this outside korge.

In any case, you should be able to use it anyway, since it doesn't depend on korge, and it uses something called kproject that should be compatible with other kind of projects.

If you need to publish to maven, you can get the root build.gradle.kts and add something like subprojects { ...publish to local... } but it shouldn't be necessary.

Instead in your final project where you generate the final executable. You can do:

settings.gradle.kts

pluginManagement { repositories {  mavenLocal(); mavenCentral(); google(); gradlePluginPortal()  }  }

plugins {
    id("com.soywiz.kproject.settings") version "0.2.9"
}

kproject("./deps")

deps.kproject.yml

https://store.korge.org/module/luak/

dependencies:
- https://github.com/korlibs/korge-luak/tree/0.1.0/luak##e05688f6048db8ca049237ba66a645893858bf5a

build.gradle.kts

dependencies {
    add("commonMainApi", project(":deps"))
}

kproject is intended to share source-code projects/snippets without having to publish to maven.

You can find an explanation here: https://docs.google.com/presentation/d/1LCd31z9Ke2_gqtba504R2bnjuGX52OlQwBX58Fg3Mho/edit?usp=sharing

[soywiz]

I'm back from a small break. I have written you via Kotlin Slack 👍