Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multithreaded evaluator #10938

Draft
wants to merge 71 commits into
base: master
Choose a base branch
from
Draft

Multithreaded evaluator #10938

wants to merge 71 commits into from

Commits on May 20, 2024

  1. EvalState: Make the parse/eval caches thread-safe

    @edolstra
    edolstra committed May 20, 2024
    Configuration menu
    Copy the full SHA
    6760b39 View commit details
    Browse the repository at this point in the history

  2. LRUCache: Mark size() as const

    @edolstra
    edolstra committed May 20, 2024
    Configuration menu
    Copy the full SHA
    d3854d1 View commit details
    Browse the repository at this point in the history

  3. Sync: Add support for shared locks

    @edolstra
    edolstra committed May 20, 2024
    Configuration menu
    Copy the full SHA
    945cd69 View commit details
    Browse the repository at this point in the history

  4. WIP

    @edolstra
    edolstra committed May 20, 2024
    Configuration menu
    Copy the full SHA
    5f3b1a3 View commit details
    Browse the repository at this point in the history

Commits on May 22, 2024

  1. WIP3

    @edolstra
    edolstra committed May 22, 2024
    Configuration menu
    Copy the full SHA
    9ddca98 View commit details
    Browse the repository at this point in the history

Commits on May 23, 2024

  1. WIP4

    @edolstra
    edolstra committed May 23, 2024
    Configuration menu
    Copy the full SHA
    d133aca View commit details
    Browse the repository at this point in the history

Commits on May 24, 2024

  1. Disable some blackhole tests for now

    @edolstra
    edolstra committed May 24, 2024
    Configuration menu
    Copy the full SHA
    1a55754 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2024

  1. WIP working

    @edolstra
    edolstra committed May 28, 2024
    Configuration menu
    Copy the full SHA
    d623dfb View commit details
    Browse the repository at this point in the history

Commits on May 29, 2024

  1. Better hash

    @edolstra
    edolstra committed May 29, 2024
    Configuration menu
    Copy the full SHA
    a9e3594 View commit details
    Browse the repository at this point in the history

  2. Symbol table concurrency hack

    @edolstra
    edolstra committed May 29, 2024
    Configuration menu
    Copy the full SHA
    b63a132 View commit details
    Browse the repository at this point in the history

  3. Hacks

    @edolstra
    edolstra committed May 29, 2024
    Configuration menu
    Copy the full SHA
    76f822f View commit details
    Browse the repository at this point in the history

Commits on May 31, 2024

  1. Fix failures due to value reuse

    @edolstra
    edolstra committed May 31, 2024
    Configuration menu
    Copy the full SHA
    6a85af7 View commit details
    Browse the repository at this point in the history

  2. Revive the Boehm GC alloc cache

    @edolstra
    edolstra committed May 31, 2024
    Configuration menu
    Copy the full SHA
    6eafc52 View commit details
    Browse the repository at this point in the history

  3. Make RegexCache thread-safe

    @edolstra
    edolstra committed May 31, 2024
    Configuration menu
    Copy the full SHA
    f018a55 View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2024

  1. Add some stats

    @edolstra
    edolstra committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    ec8593d View commit details
    Browse the repository at this point in the history

  2. Cleanup

    @edolstra
    edolstra committed Jun 3, 2024
    Configuration menu
    Copy the full SHA
    105dea5 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2024

  1. Make EvalState::srcToStore thread-safe

    @edolstra
    edolstra committed Jun 4, 2024
    Configuration menu
    Copy the full SHA
    27fb652 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2024

  1. PosixSourceAccessor: Use SharedSync

    @edolstra
    edolstra committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    d990974 View commit details
    Browse the repository at this point in the history

  2. FileParseCache, FileEvalCache: Use read lock

    @edolstra
    edolstra committed Jun 5, 2024
    Configuration menu
    Copy the full SHA
    eba54f5 View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2024

  1. Add getOptional()

    @edolstra
    edolstra committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    a25a5b7 View commit details
    Browse the repository at this point in the history

  2. EvalState: Add importResolutionCache 

    This is a mapping from paths to "resolved" paths (i.e. with
`default.nix` added, if appropriate). `fileParseCache` and
`fileEvalCache` are now keyed on the resolved path *only*.
    @edolstra
    edolstra committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    ca11328 View commit details
    Browse the repository at this point in the history

  3. Make fileEvalCache insertion more efficient

    @edolstra
    edolstra committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    c2c01d8 View commit details
    Browse the repository at this point in the history

  4. Ensure that files are parsed/evaluated only once 

    Previously, the optimistic concurrency approach in `evalFile()` meant
that a `nix search nixpkgs ^` would do hundreds of duplicated
parsings/evaluations. Now, we reuse the thunk locking mechanism to
ensure it's done only once.
    @edolstra
    edolstra committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    9b88021 View commit details
    Browse the repository at this point in the history

  5. Small optimization

    @edolstra
    edolstra committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    708e0e8 View commit details
    Browse the repository at this point in the history

  6. SymbolStr: Remove std::string conversion 

    This refactoring allows the symbol table to be stored as something
other than std::strings.
    @edolstra
    edolstra committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    cc38822 View commit details
    Browse the repository at this point in the history

  7. Use a contiguous arena for storing symbols 

    This allows symbol IDs to be offsets into an arena whose base offset
never moves, and can therefore be dereferenced without any locks.
    @edolstra
    edolstra committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    424e01e View commit details
    Browse the repository at this point in the history

  8. Executor: Randomize the work queue 

    This makes it less likely that we concurrently execute tasks that
would block on a common subtask, e.g. evaluating `libfoo` and
`libfoo_variant` are likely to have common dependencies.
    @edolstra
    edolstra committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    c663076 View commit details
    Browse the repository at this point in the history

  9. Provide std::hash<SourcePath>

    @edolstra
    edolstra committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    adcc351 View commit details
    Browse the repository at this point in the history

  10. Provide std::hash<Symbol>

    @edolstra
    edolstra committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    3988faf View commit details
    Browse the repository at this point in the history

  11. Remove unused #include

    @edolstra
    edolstra committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    a70ec9e View commit details
    Browse the repository at this point in the history

  12. Split the symbol table into domains

    @edolstra
    edolstra committed Jun 6, 2024
    Configuration menu
    Copy the full SHA
    0cd29fe View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2024

  1. Fix --disable-gc build

    @edolstra
    edolstra committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    0c87ead View commit details
    Browse the repository at this point in the history

  2. Don't use finishValue() for thunks

    @edolstra
    edolstra committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    33f50ae View commit details
    Browse the repository at this point in the history

  3. Remove debug statement

    @edolstra
    edolstra committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    5e87cf4 View commit details
    Browse the repository at this point in the history

  4. Specify memory order 

    Probably doesn't matter much though.
    @edolstra
    edolstra committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    400a670 View commit details
    Browse the repository at this point in the history

  5. Split the PosixSourceAccessor lstat cache

    @edolstra
    edolstra committed Jun 7, 2024
    Configuration menu
    Copy the full SHA
    5c6eb1a View commit details
    Browse the repository at this point in the history

Commits on Jun 13, 2024

  1. nix search: Restore output

    @edolstra
    edolstra committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    3353f9a View commit details
    Browse the repository at this point in the history

  2. Make the max-call-depth check thread-local

    @edolstra
    edolstra committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    9b814c4 View commit details
    Browse the repository at this point in the history

  3. Move code

    @edolstra
    edolstra committed Jun 13, 2024
    Configuration menu
    Copy the full SHA
    fd5c32b View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2024

  1. nix flake show: Make multi-threaded

    @edolstra
    edolstra committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    1bdf907 View commit details
    Browse the repository at this point in the history

  2. Disable some failing tests for now

    @edolstra
    edolstra committed Jun 14, 2024
    Configuration menu
    Copy the full SHA
    3cc1319 View commit details
    Browse the repository at this point in the history

Commits on Jun 18, 2024

  1. Merge remote-tracking branch 'origin/master' into multithreaded-eval

    @edolstra
    edolstra committed Jun 18, 2024
    Configuration menu
    Copy the full SHA
    f6cbd6a View commit details
    Browse the repository at this point in the history

Commits on Jun 19, 2024

  1. Cleanups

    @edolstra
    edolstra committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    6103246 View commit details
    Browse the repository at this point in the history

  2. Re-enable assertNoSymlinks()

    @edolstra
    edolstra committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    576a03e View commit details
    Browse the repository at this point in the history

  3. Formatting

    @edolstra
    edolstra committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    52bd994 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/master' into multithreaded-eval

    @edolstra
    edolstra committed Jun 19, 2024
    Configuration menu
    Copy the full SHA
    b713591 View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2024

  1. Merge remote-tracking branch 'origin/master' into multithreaded-eval

    @edolstra
    edolstra committed Jul 3, 2024
    Configuration menu
    Copy the full SHA
    fcbdc9f View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2024

  1. Make the default GC_INITIAL_HEAP_SIZE a lot bigger 

    On Linux, we now use 80% of free memory. If it's free, we may as well
use it, and hopefully avoid some expensive stop-the-world GC cycles.
    @edolstra
    edolstra committed Jul 5, 2024
    Configuration menu
    Copy the full SHA
    997af66 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2024

  1. Merge remote-tracking branch 'origin/master' into multithreaded-eval

    @edolstra
    edolstra committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    d3397d7 View commit details
    Browse the repository at this point in the history

  2. Remove obsolete comment

    @edolstra
    edolstra committed Jul 11, 2024
    Configuration menu
    Copy the full SHA
    2b4c36f View commit details
    Browse the repository at this point in the history

Commits on Jul 12, 2024

  1. Merge remote-tracking branch 'origin/master' into multithreaded-eval

    @edolstra
    edolstra committed Jul 12, 2024
    Configuration menu
    Copy the full SHA
    d131a02 View commit details
    Browse the repository at this point in the history

Commits on Jul 26, 2024

  1. Merge remote-tracking branch 'origin/master' into multithreaded-eval

    @edolstra
    edolstra committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    4482fe4 View commit details
    Browse the repository at this point in the history

  2. nix repl: Remove unnecessary call to evalString 

    This crashes with the multithreaded evaluator, which checks against
attempts to finish an already finished value.
    @edolstra
    edolstra committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    8cf80c9 View commit details
    Browse the repository at this point in the history

  3. Remove FIXME

    @edolstra
    edolstra committed Jul 26, 2024
    Configuration menu
    Copy the full SHA
    67ff326 View commit details
    Browse the repository at this point in the history

Commits on Aug 5, 2024

  1. Merge remote-tracking branch 'origin/master' into multithreaded-eval

    @edolstra
    edolstra committed Aug 5, 2024
    Configuration menu
    Copy the full SHA
    c8c9500 View commit details
    Browse the repository at this point in the history

Commits on Aug 12, 2024

  1. Merge remote-tracking branch 'origin/master' into multithreaded-eval

    @edolstra
    edolstra committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    dd44b26 View commit details
    Browse the repository at this point in the history

  2. Make AllowListSourceAccessor thread-safe

    @edolstra
    edolstra committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    9102baf View commit details
    Browse the repository at this point in the history

  3. Make positionToDocComment thread-safe

    @edolstra
    edolstra committed Aug 12, 2024
    Configuration menu
    Copy the full SHA
    998a289 View commit details
    Browse the repository at this point in the history

Commits on Aug 13, 2024

  1. callFunction(): Use correct environment in error messages

    @edolstra
    edolstra committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    5310b0f View commit details
    Browse the repository at this point in the history

  2. callFunction(): Create the primop app chain safely 

    We should never call reset() on a value (such as vRes) than can be
seen by another thread.

This was causing random failures about 'partially applied built-in
function' etc.
    @edolstra
    edolstra committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    839aec2 View commit details
    Browse the repository at this point in the history

  3. Debug

    @edolstra
    edolstra committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    4f90786 View commit details
    Browse the repository at this point in the history

  4. Move perf counters into EvalState

    @edolstra
    edolstra committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    6357885 View commit details
    Browse the repository at this point in the history

  5. Debug

    @edolstra
    edolstra committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    4086c1c View commit details
    Browse the repository at this point in the history

  6. Remove "SPURIOUS" message

    @edolstra
    edolstra committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    a6d8217 View commit details
    Browse the repository at this point in the history

  7. Fix formatting

    @edolstra
    edolstra committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    ea4e981 View commit details
    Browse the repository at this point in the history

  8. Fix meson build

    @edolstra
    edolstra committed Aug 13, 2024
    Configuration menu
    Copy the full SHA
    d36ea2e View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2024

  1. finishAll(): Propagate an arbitrary exception

    @edolstra
    edolstra committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    114d1a0 View commit details
    Browse the repository at this point in the history

  2. nix flake show: Make sure the visit() closure is still alive in case … 

    …of an exception
    @edolstra
    edolstra committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    f947b63 View commit details
    Browse the repository at this point in the history

  3. Introduce ValueType::nFailed 

    This fixes a crash in Printer if a value is in a failed state.
    @edolstra
    edolstra committed Aug 14, 2024
    Configuration menu
    Copy the full SHA
    ceeb648 View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2024

  1. Make 'nix search --json' thread-safe

    @edolstra
    edolstra committed Aug 15, 2024
    Configuration menu
    Copy the full SHA
    8b7d5b4 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2024

  1. Merge remote-tracking branch 'origin/master' into multithreaded-eval

    @edolstra
    edolstra committed Aug 22, 2024
    Configuration menu
    Copy the full SHA
    8020c0c View commit details
    Browse the repository at this point in the history