Benchmark between make and ninja build for kvrocks

[PragmaTwice]

Motivation

In #621, @tisonkun came up with a great idea: benchmark ninja build and make build of kvrocks.
I did this test after making some fixes to make cmake configuration of kvrocks support ninja.
A PR for that will follow.

Preparation

mkdir bm-make-build && cd bm-make-build && cmake .. -DCMAKE_BUILD_TYPE=RelWithDbgInfo
mkdir bm-ninja-build && cd bm-ninja-build && cmake .. -DCMAKE_BUILD_TYPE=RelWithDbgInfo -G Ninja

Clean Build

$ hyperfine -r 1 "cd bm-make-build; make -j8 kvrocks kvrocks2redis unittest" "cd bm-ninja-build; ninja -j8 kvrocks kvrocks2redis unittest"
Benchmark 1: cd bm-make-build; make -j8 kvrocks kvrocks2redis unittest
  Time (abs ≡):        186.883 s               [User: 1139.316 s, System: 62.312 s]
 
Benchmark 2: cd bm-ninja-build; ninja -j8 kvrocks kvrocks2redis unittest
  Time (abs ≡):        212.749 s               [User: 1129.823 s, System: 60.877 s]
 
Summary
  'cd bm-make-build; make -j8 kvrocks kvrocks2redis unittest' ran
    1.14 times faster than 'cd bm-ninja-build; ninja -j8 kvrocks kvrocks2redis unittest'

No-op Incremental Build

$ hyperfine -r 1 "cd bm-make-build; make -j8 kvrocks kvrocks2redis unittest" "cd bm-ninja-build; ninja -j8 kvrocks kvrocks2redis unittest"
Benchmark 1: cd bm-make-build; make -j8 kvrocks kvrocks2redis unittest
  Time (abs ≡):        657.1 ms               [User: 707.8 ms, System: 224.1 ms]
 
Benchmark 2: cd bm-ninja-build; ninja -j8 kvrocks kvrocks2redis unittest
  Time (abs ≡):         39.0 ms               [User: 25.7 ms, System: 17.7 ms]
 
Summary
  'cd bm-ninja-build; ninja -j8 kvrocks kvrocks2redis unittest' ran
   16.86 times faster than 'cd bm-make-build; make -j8 kvrocks kvrocks2redis unittest

Incremental Build after modifying a source file

$ hyperfine -r 1 "cd bm-make-build; make -j8 kvrocks kvrocks2redis unittest" "cd bm-ninja-build; ninja -j8 kvrocks kvrocks2redis unittest"
Benchmark 1: cd bm-make-build; make -j8 kvrocks kvrocks2redis unittest
  Time (abs ≡):        16.158 s               [User: 14.762 s, System: 1.640 s]
 
Benchmark 2: cd bm-ninja-build; ninja -j8 kvrocks kvrocks2redis unittest
  Time (abs ≡):         6.797 s               [User: 16.019 s, System: 1.530 s]
 
Summary
  'cd bm-ninja-build; ninja -j8 kvrocks kvrocks2redis unittest' ran
    2.38 times faster than 'cd bm-make-build; make -j8 kvrocks kvrocks2redis unittest'

Incremental Build after modifying a header file

$ hyperfine -r 1 "cd bm-make-build; make -j8 kvrocks kvrocks2redis unittest" "cd bm-ninja-build; ninja -j8 kvrocks kvrocks2redis unittest"
Benchmark 1: cd bm-make-build; make -j8 kvrocks kvrocks2redis unittest
  Time (abs ≡):        39.906 s               [User: 143.227 s, System: 7.891 s]
 
Benchmark 2: cd bm-ninja-build; ninja -j8 kvrocks kvrocks2redis unittest
  Time (abs ≡):        27.103 s               [User: 163.895 s, System: 8.882 s]
 
Summary
  'cd bm-ninja-build; ninja -j8 kvrocks kvrocks2redis unittest' ran
    1.47 times faster than 'cd bm-make-build; make -j8 kvrocks kvrocks2redis unittest'

Conclusion

As you can see, ninja is slightly slower than make build in clean build, but has huge advantages in incremental build, which is very friendly to kvrocks developers who need to do frequent incremental builds.

I would like to make kvrocks use make by default, but have a --ninja option to enable ninja.

[tisonkun]

@PragmaTwice I guess this can be a reply on #621 or since this issue has been created, we can move it to Discussions?

It doesn't seem like an issue to resolve but a general topic about benchmark.

[PragmaTwice]

@PragmaTwice I guess this can be a reply on #621 or since this issue has been created, we can move it to Discussions?

It doesn't seem like an issue to resolve but a general topic about benchmark.

Oh yeah, I also think moving to Discussions is better.

[tisonkun]

Done.

[PragmaTwice]

Thanks!

[tisonkun]

As you can see, ninja is slightly slower than make build in clean build, but has huge advantages in incremental build, which is very friendly to kvrocks developers who need to do frequent incremental builds.

That sounds good. We can introduce ninja build option as you describe and if developers find it a frequent better choice, switch it to default later.

[PragmaTwice]

Done in #626.