Add an implementation of mirage-crypto-rng-miou to initialize the RNG with Miou

[dinosaure]

This is my first attempt at implementing mirage-crypto-rng-miou. The idea of this initialization is to create a process that will periodically refill the random number generator. I've just tested a program that lasts several hours and I haven't observed any memory leakage: the tasks created to refill the generator are cleaned up properly (every second by default).

The initialization creates a witness to the promise that the user must stop before the end of the application (otherwise Miou raises the Still_has_children exception). Mirage_crypto_rng_miou.kill just cancels the task and finishes all its sub-tasks.

Finally, I've re-implemented Fortuna to be domain-friendly. This is because access may involve changes to the state of the random number generator. We need to ensure that several domains can generate random numbers without conflicts (data-race). A mutex is therefore used to protect exclusive access to the state of the generator. I will verify my implementation with tsan.

[dinosaure]

Note for reviewers, currently a big diff between lwt/async and this initializer is about sources. lwt/async add a new source which corresponds to a function which is executed at every iterations of the scheduler (as a hook). I tried to provides with Miou such a way to register a hook. However, actually such hook can not emit an effect (Miou does not attach an effect handler on them). Due to this fact, it's impossible to run add a new source (such as G.accumulate) for the rng as a hook because these functions perform effects (mainly to lock/unlock the mutex).

[dinosaure]

I can confirm via TSan that the current implementation Pfortuna does not generates data-races and Fortuna does.

[dinosaure]

I force-pushed the final version which is rechecked with TSan (I found an issue about Miou itself). The documentation was improved and the rng task can be run in parallel or concurrently depending on the number of domains available.

NOTE: On OCaml 5.1, I was unable to compile tests without the (modes native), so I added them to be able to link correctly C stubs. The error is probably related to the bytecode target and, in some ways, dune is not available to link correctly C stubs and bytecode executable.

[hannesm]

great to see this happening, but I'm rather uncomfortable with the fork of fortuna.ml -- this has a high maintenance cost. any chance to have a single code base (is there noticable performance if we use mutex everywhere?)

[dinosaure]

this has a high maintenance cost. any chance to have a single code base (is there noticable performance if we use mutex everywhere?)

I don't have strong opinion and we probably can just use Mutex (instead of Miou.Mutex) to be sure that the Fortuna engine is domain-safe for any schedulers. But yes, it requires some benchmarks.

[hannesm]

But yes, it requires some benchmarks.

We have dune build --rel bench/speed.exe && _build/default/bench/speed.exe fortuna -- which generates random bytes (but there's no entropy collection happening during that phase, so it may be not the right thing).

[dinosaure]

I got this result:

➜  mirage-crypto git:(miou-rng) ✗ dune build --rel bench/speed.exe && _build/default/bench/speed.exe fortuna pfortuna
accel: XOR AES GHASH             

* [fortuna]
       16:  220.113893 MB/s  (28879114 iters in 2.002 s)
       64:  705.788863 MB/s  (23177441 iters in 2.004 s)
      256:  2716.850044 MB/s  (22237640 iters in 1.998 s)
     1024:  5060.986346 MB/s  (10351918 iters in 1.997 s)
     8192:  5159.824384 MB/s  (1317303 iters in 1.995 s)

* [pfortuna]
       16:  65.395947 MB/s  (8762398 iters in 2.045 s)
       64:  261.715039 MB/s  (8594904 iters in 2.004 s)
      256:  867.953303 MB/s  (7247164 iters in 2.039 s)
     1024:  1246.217245 MB/s  (2548325 iters in 1.997 s)
     8192:  1758.976589 MB/s  (475333 iters in 2.111 s)

With this patch:

+  bm "pfortuna" (fun name ->
+    let open Mirage_crypto_rng_miou_unix.Pfortuna in
+    Miou_unix.run ~domains:2 @@ fun () ->
+    let rng = Mirage_crypto_rng_miou_unix.(initialize (module Pfortuna)) in
+    let g = create () in
+    reseed ~g "abcd" ;
+    throughput name (fun buf ->
+        let buf = Bytes.unsafe_of_string buf in
+        generate_into ~g buf ~off:0 (Bytes.length buf));
+    Mirage_crypto_rng_miou_unix.kill rng) ;

I deleted the initialize function - so we have no entropy collection during the benchmark and it does not affect results. I also just add a Mutex.t into the Fortuna module (as I did for Pfortuna but it uses Miou.Mutex.t) and I got the same result than pfortuna above. In other words, if we add Mutex.t or Miou.Mutex.t into the Fortuna implementation we got the pfortuna results above (and the initialize does not have an impact).

So we are ~3 times slower than the initial implementation.

[dinosaure]

The PR is ready to be reviewed and merged 👍.

[hannesm]

looks mostly fine, I had some minor comments. thanks for your contribution.

[dinosaure]

It seems that test_entropy.exe fails on MacOS (and it's unrelated to this PR). Otherwise, I splitted out the benchmark about pfortuna and added a comment about why we re-implemented the module.

[hannesm]

It seems that test_entropy.exe fails on MacOS (and it's unrelated to this PR).

Indeed, tracked as #216

[hannesm]

Thanks, I rebased on main and force-pushed, after merging #232 there was the only conflict in .cirrus.yml. Also, I added a comment to keep fortuna and pfortuna in sync :)

[hannesm]

Once CI passes (see notes in #232 (comment)), I'll go ahead and squash-merge this PR.

[hannesm]

Hm, I've no clue what to do with bench/miou -- marking it optional doesn't lead to the OCaml-CI not building it -- and marking it with (package mirage-crypto-rng-miou-unix) results in a dune error since there's no public_name -- but we don't want a public_name since we don't want it to be installed when installing mirage-crypto-rng-miou-unix.

Any pointers how to get out of this situation? Maybe commenting the dune rule out for the time being?

[hannesm]

Maybe commenting the dune rule out for the time being?

This is what I pushed to this PR. CI looks good. I'll squash & merge if @dinosaure is happy with this PR (esp. with my changes).

[hannesm]

NOTE: On OCaml 5.1, I was unable to compile tests without the (modes native), so I added them to be able to link correctly C stubs. The error is probably related to the bytecode target and, in some ways, dune is not available to link correctly C stubs and bytecode executable.

Please note, I had this issue as well, the problem is that if you have mirage-crypto installed, it uses in bytecode mode the installed C stubs instead of the locally compiled ones... And I've a deja-vu ocaml/dune#9979

So, let's remove the (modes native). It is a dune issue, but our tests shouldn't be a scapegoat for that.

[dinosaure]

I'm ok with your changes 👍