[Proposal] Add some timings

[delneg]

I've decided to try & run those on my machine (mbp, 16" 2019, 2,4 GHz 8-Core Intel Core i9, 32gb), and I've got the following times (couple runs each):

# test-js
Raytracer running...
Ray tracing:
 - rendered image size: (1024x1024)
 - elapsed: 1805.3647939562798 ms
 
 
# test-wasm
 Raytracer running...
Ray tracing done:
 - rendered image size: (1024x1024)
 - elapsed: 970.5676089525223 ms

# test-web
Ray tracing done:
 - image rendered at 2x for super-sampled anti-aliasing
 - rendered image size: (1024x1024)
 - elapsed: 1067 ms
 
 # test-rust (lto = false)
 Raytracer running...
Ray tracing:
 - rendered image size: (1024x1024)
 - elapsed: 2143.8712720000003 ms
 
 # test-dotnet
 Raytracer running...
Ray tracing:
 - rendered image size: (1024x1024)
 - elapsed: 915.9425 ms

# test-native (had to change vars to osx-x64)
Raytracer running...
Ray tracing:
 - rendered image size: (1024x1024)
 - elapsed: 949.2975 ms

Seems kinda interesting that dotnet is on par with nodejs, and rust is kinda the slowest. I guess JIT does it work quite well

P.S. the license file is dated 2017 lol

[ncave]

@delneg The native Rust binary seems to be slower on Windows (but not on Linux), for some reason. See more results here.
I'll consider adding a benchmark results page, thanks for testing.

[delneg]

Thanks, I was testing on macos, maybe that's why it's slower
A benchmark page would be great

[yowl]

What exactly is the test-dotnet running, NativeAOT ? I.e. are you uncommenting

fable-raytracer/fable-raytracer.fsproj

Line 19 in 014db00

<!-- <PackageReference Include="Microsoft.DotNet.ILCompiler" Version="7.0.0-*" /> -->

for that?

[ncave]

@yowl
No, test-dotnet runs a standard .NET release build (whatever version you have installed, but .NET 6.0 is recommended).
If you want to try .NET NativeAOT, you can try build-native / test-native. Make sure test-native's runtime is correct for your platform (see package.json), and yes, you need to uncomment the AOT compiler package in the .fsproj and the commented MS repos in nuget.config.

That should be enough, but if not, follow the MS instructions for NativeAOT. There is also a NativeAOT-LLVM branch but I haven't tried it. Overall, I got more or less the same performance from NativeAOT as normal (jitted) .NET 6.0 (for this specific demo, which is numeric-heavy). The native binaries it produces are quite large, so I didn't bother trying to make a cross-platform script to build/run it. I haven't tried it with .NET 7 either, so YMMV.

[delneg]

Some more results with an M1 Mac (aarch64):

test-js - 1200-1270ms
test-wasm - 891-893ms
test-rust - 1008-1012ms
test-dotnet - 690 - 700ms
test-native - not possible: error : Native compilation does not support targeting osx-arm64 yet. https://github.com/dotnet/corert/issues/4589

[ncave]

@alfonsogarciacaro
Switching just one line (the ray intersection from value type back to reference type) made .NET times much closer to Rust, and now .NET Native is 20% faster than the Rust native binary (on Linux, build-native/test-native). Of course, this is testing generated F# to Rust code for this particular demo, not hand-crafted Rust code, so it is expected to some extent. There is plenty of room for generating more performance-oriented Rust code in the future. Still, it's good that the currently generated Rust code can at least hold some parity and it's not magnitudes slower.

[alfonsogarciacaro]

Thanks for letting me know @ncave. This is a good thing to remember to avoid the mantra that GC destroys performance and we forget that passing by value also has a cost :) Even if Rust is not much faster that .NET I think it's still very valuable to be able to compile F# to native/wasm through Rust and generate small binaries without the full .NET runtime. And of course, having the potential to get better performance by improving the generated Rust from the same F# code is a big plus 👍

[delneg]

Btw, chatting with .net JIT developer Egor some time ago, he mentioned that optimal struct size is 64b (4 fields), anymore than that and you typically lose more than you gain.
The more you know :)

[ncave]

In the generated Rust code we're passing in everything by reference, so size normally should not matter for function parameters. But this particular type is a function return type (returned by value), so size does matter. But, it was already a reference type in the Rust code, so Rust times didn't change, only .NET times were improved. Usually it's a bit more nuanced than this, with speculative execution, data locality, caches and GC pressure all interacting to complicate things.

[alfonsogarciacaro]

I added the original Raytracer sample from the Typescript playground (it seems to be deleted now). These are the results in my machine, I get the best timing out of 4 runs. The code always renders the same 1024x1024 scene in all cases.

Browser (Chrome)

• Typescript > JS (code): 2754ms
• F# > JS (code in repl): 1575ms
• F# > Rust > WASM (npm run test-web): 1052ms

CLI (Ubuntu on Windows WSL2)

• npm run test-dotnet: 583ms
• npm run test-rust: 464ms

Not sure why the Fable JS version is faster than the TS version, the main difference in the generated code seems to be that Fable compiles class methods as module functions instead of attaching them to the prototype.

As stated above, the Rust version is much slower on Windows. Dotnet is almost on par when not using structs, but the other advantage of the Rust version is the binary size is much smaller.

# Also adding <PublishSingleFile>true</PublishSingleFile> to .fsproj
dotnet publish -c Release -r linux-x64 --self-contained > 65MB

cargo build --release > 1.8M