Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Effects on incremental compilation of compile time user code #1578

Closed
jakemac53 opened this issue Apr 12, 2021 · 1 comment
Closed

Effects on incremental compilation of compile time user code #1578

jakemac53 opened this issue Apr 12, 2021 · 1 comment
Labels
static-metaprogramming Issues related to static metaprogramming

Comments

@jakemac53
Copy link
Contributor

(Forked from the discussion in #1483)

In the discussion on modular compilation, some issues were also brought up around incremental compilation (generally in the frontend_server).

Advanced Invalidation

In the new advanced invalidation strategy, edits to function bodies only require the containing library to be re-compiled. If that function could be used at compile time, then we can't do that optimization, because it might affect the generated code (or constants) in any library depending on the containing library.

Proposed Solution

Add a single boolean flag to each function ast node, indicating if it was evaluated at compile time. In the constant evaluator (or macro expander) whenever you handle a function invocation you flip the bit on that ast node to true.

Whenever a function body is modified, you check the bit and invalidate all transitive deps if it was set, and then you flip the bit back to false.

Incremental Serialization

The above proposed solution causes problems for both incremental serialization and modular serialization, because you want to serialize kernel files in a Depth-first fashion (in terms of the import graph). That means you don't know which functions are used or not at kernel serialization time, and so you can't accurately set the bit.

There is a similar issue here with the accept/reject flow between the incremental compiler and the VM, where if the VM rejects a library but compiling that library edited some of its deps (by flipping these bits), then we wouldn't be able to revert to the old state because it would no longer be valid. I believe this is covered by the same solution below, but I don't fully understand the workflow here still so I could be missing something.

Proposed Solution

Don't serialize this bit to disk. This implies some extra work when initializing from dill - either you need to re-evaluate all constants in order to set all the bits properly or you need to revert to the old invalidation strategy for the first build of any given library. Note that the latter approach could have some pathological cases, if you edited the libraries in your app in a breadth-first order (import graph wise).

cc @jensjoha @munificent @leafpetersen

@davidmorgan
Copy link
Contributor

I suspect this is obsolete, if not please reopen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
static-metaprogramming Issues related to static metaprogramming
Projects
Development

No branches or pull requests

2 participants