Memoization function

[rudihorn]

This is referencing the computation model specified in #1280. The goal has been to implement a memoization function which takes a computation of the form 'a -> 'b Fiber.t and returns a computation 'a -> 'b Fiber.t which additionally performs:

• Dependency tracking
• Output caching with early cut-off
• Cycle detection

Currently the memoization function has been implemented and is able to perform these three functions. An example of how to use the memoization functions is shown in test/unit-tests/memoize.mlt. Below is an example of how one could use the system to implement a Fibonacci sequence which is automatically cached between computations.

let mfib = 
  let mfib = CRef.deferred () in
  let compfib x =
    let mfib = CRef.get mfib in
    counter := !counter + 1;
    if x <= 1 then
      Fiber.return x
    else
      mfib (x - 1)
      >>= (fun r1 ->
        mfib (x - 2)
        >>| fun r2 -> r1 + r2) in
  memoization iimemcached "fib" int_input_spec int_output_spec compfib |> CRef.set mfib;
  CRef.get mfib;;

There has been a little bit of progress to change build_system.ml so that the actual build process is a little more abstracted to make it easier to drop in the memoization function and remove the current caching etc. This bit is still a little bit of a mess however, and in essence it just breaks existing caching (for now).

Feedback welcome.

[ghost]

Using print : 'a -> Sexp.t seems better here, so that we leave the formatting to the caller

[rudihorn]

currently I'm only using serialize, but this should probably be changed to also include the "human readable" form. I'm wondering if it is better to store this as a lazy function (which would prevent the input from being garbage collected) or as the printed input (which would require computation of print for every function).

[ghost]

This is dead code, let's delete it

[rudihorn]

This code may be nice to register as computation at a later stage, even if it isn't used currently in the code. It essentially builds a files dependencies without building the file itself.

[ghost]

Ok. Could you add a comment? Otherwise, future readers will wonder why it's here.

[ghost]

What about putting Static_deps.t inside Internal_rule.t?

[rudihorn]

I've now fixed this, but it does mean that rule.static_deps (which is defined as lazy) is evaluated twice, once in prepare_rule and once in build_rule. Otherwise one would need to memoize this.

[ghost]

Hmm, evaluating it twice doesn't seem good. We should keep it separate then

[rudihorn]

I've decided to introduce a Once module, which similarly to lazy only evaluates something when Once.get is called, but that also caches the result so Once.get only causes the expression to be evaluated a single time.

[rgrinberg]

@rudihorn Once this is ready to be merged, could you remove the WIP tag? I'd like to give this a read when it's ready. To make sure I understand the basics.

[ghost]

I did a bit more work on this PR and I now get very similar timings between the base and tip of this PR. For null builds of the platform, there is now only a 0.1s slowdown, which is likely due to the new cycle detection algorithm. This is fine given that the one in master is broken. We can now rebase and merge this PR.

[ghost]

@rudihorn there is an issue with the signature, could you update the email in author field of the commit and force-push?

@rgrinberg @emillon could you have another look as I changed a couple of things (and @rudihorn as well if you want).

[emillon]

Thanks! I have a couple minor comments but I look forward to having this merged 👍

[rgrinberg]

Bootstrap seems to be broken now.

[rgrinberg]

@diml feel free to merge after fixing bootstrap. I'd still like to study some parts of this PR, but I don't think we should delay merging. Rebasing these pr's over and over is a waste of time.

[ghost]

Ok, following the slack conversation, I'll wait for 1.6 and then merge.