Tictactoe benchmark

[petrihakkinen]

Hi! I'm comparing the performance of Luau and LuaJIT interpreters (with JIT off for fairness). One thing which stands out is the performance of the tictactoe benchmark that comes bundled with Luau:

luajit 153ms
luau 421ms

This is on MSVC 2017 on Windows. (For some reason the performance is even worse for this benchmark when compiled with clang, when usually it's the other way around.)

I would expect that the perf difference would be in the same ballpark (within 40% or so) as it is with other benchmarks. Is there something particularly nasty in this benchmark for Luau?

[zeux]

FWIW this is a gap that's much wider than we've seen before, but we've mostly switched to VS 2019. VS 2017 doesn't use some important optimizations that affect the interpreter, so the code ends up slower. The timings from my machine:

luajit 74ms
luau (vs2019): 116ms
luau (vs2022): 105ms
luau (clang): 132ms

My recollection was that vs2017 => vs2019 was a ~10% delta on average so I'm surprised that you're seeing an effect that significant. I don't think the benchmark is doing anything particularly unusual - we haven't analyzed it closely though.

And yeah unusure what is going on with clang here, we'd need to look closely at the benchmark.

[zeux]

Looked into this a little bit, this benchmark broadly has 4 hotspots:

• Reading from the array via an index. This is a path that is well optimized for us, and is a little codegen sensitive on x64 because the code needs to cast between double and integer to recover the actual index. Would need to look at what clang is doing here carefully.
• Writing to the array which causes the array to grow (in move_candidates?). This is mostly bottlenecked in table resize, although the table is small - potentially something we can improve in general.
• Calling pairs in the inner loop - we generally haven't bothered to specialize pairs but here it looks like a significant cost. Could be easy wins for making pairs a builtin (generalized iteration also solves this since you can just drop the call to pairs...)
• Comparing to numbers. This is a case where luajit has a dedicated opcode for reg-number comparison, whereas we only have a somewhat more generic opcode for reg-constant comparison that internally branches based on constant type. We've looked into this a little bit in the past and results wrt performance were inconclusive, but it looks like we might need to take another look.

I suspect luajit is better on points 3 and definitely 4, based on past analysis it's similar in point 1 but only if the compiler codegen is reasonable, and maybe it has better code for 2 but that should be reasonably simple for us to remedy depending on what exact characteristics of table growth are seen here.

[zeux]

As an illustration for pairs, merely removing three calls to pairs without making any other changes goes from 105ms (vs2022) to 95ms :) -- this is possible because of generalized iteration, so that makes the code not runnable in luajit of course.

pairs in an inner loop is something that we rarely see, but here it's definitely an issue.

[zeux]

(finally, just in case, double check that you have compiler optimizations fully enabled and, critically, assertions disabled - we have very extensive assertions in the interpreter loop, something we should probably revisit at some point. luajit is much less sensitive to build configuration since most of the interpreter is in assembly so you get the same code no matter what the compiler or compilation options are)

[petrihakkinen]

Thanks for the detailed answers! I'm considering moving my indie game projects from LuaJIT to Luau and I'm indeed using pairs quite heavily as that has not been a bottleneck with LuaJIT, so I'd definitely be interested in faster pairs :)

Also good to know that the performance difference should not be as big. I need to dig deeper what the issue here is. Maybe I have messed up compiler settings or something...

[zeux]

Note that we have fully optimized pairs iteration but this one is a curious case where what's slow(ish) isn't the iteration but the actual call to pairs() (which returns next, t) that happens every time the inner loop starts :) We do have a mechanism for fixing issues like this (fastcall/builtins), we just haven't used it on pairs/ipairs since outside of this benchmark we haven't seen this called that frequently.

[petrihakkinen]

Ah got it!

[zeux]

Also thanks for highlighting this, we'll study this benchmark more closely and see what we can improve in the VM, the disparity between clang and MSVC in the wrong direction if anything is something that we should fix or file an LLVM bug for.

[petrihakkinen]

No problem! :)

There is indeed something wrong with my build environment. I compiled Luau from commandline using the instructions from the github landing page and I got these results instead:

luajit 153ms
luau 250ms

(the gap is larger than yours but I have an old PC... Intel i5-3570k @ 3.4 GHz, not sure if that matters)

[petrihakkinen]

btw. I get a 5% perf improvement in this benchmark if I disable /GS compiler option (security check for buffer overruns). That option is enabled by default at least in ReleaseWithDebInfo configuration.

Edit: Other benchmarks are affected too. Might be worth considering disabling that option?

[petrihakkinen]

Solved my original problem -- I had forgotten to call luaL_sandbox. Oops!

[JDaance]

What was the original problem? The big performance gap? After reading this whole thread some numbers would be interesting 🙂