Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nomad: include snapshot index when submitting plans #5791

Merged
merged 4 commits into from
Jul 17, 2019

Commits on Jun 24, 2019

  1. nomad: include snapshot index when submitting plans

    Plan application should use a state snapshot at or after the Raft index
    at which the plan was created otherwise it risks being rejected based on
    stale data.
    
    This commit adds a Plan.SnapshotIndex which is set by workers when
    submitting plan. SnapshotIndex is set to the Raft index of the snapshot
    the worker used to generate the plan.
    
    Plan.SnapshotIndex plays a similar role to PlanResult.RefreshIndex.
    While RefreshIndex informs workers their StateStore is behind the
    leader's, SnapshotIndex is a way to prevent the leader from using a
    StateStore behind the worker's.
    
    Plan.SnapshotIndex should be considered the *lower bound* index for
    consistently handling plan application.
    
    Plans must also be committed serially, so Plan N+1 should use a state
    snapshot containing Plan N. This is guaranteed for plans *after* the
    first plan after a leader election.
    
    The Raft barrier on leader election ensures the leader's statestore has
    caught up to the log index at which it was elected. This guarantees its
    StateStore is at an index > lastPlanIndex.
    schmichael committed Jun 24, 2019
    Configuration menu
    Copy the full SHA
    1d670f2 View commit details
    Browse the repository at this point in the history
  2. nomad: evaluate plans after previous plan index

    The previous commit prevented evaluating plans against a state snapshot
    which is older than the snapshot at which the plan was created.  This is
    correct and prevents failures trying to retrieve referenced objects that
    may not exist until the plan's snapshot. However, this is insufficient
    to guarantee consistency if the following events occur:
    
    1. P1, P2, and P3 are enqueued with snapshot @ 100
    2. Leader evaluates and applies Plan P1 with snapshot @ 100
    3. Leader evaluates Plan P2 with snapshot+P1 @ 100
    4. P1 commits @ 101
    4. Leader evaluates applies Plan P3 with snapshot+P2 @ 100
    
    Since only the previous plan is optimistically applied to the state
    store, the snapshot used to evaluate a plan may not contain the N-2
    plan!
    
    To ensure plans are evaluated and applied serially we must consider all
    previous plan's committed indexes when evaluating further plans.
    
    Therefore combined with the last PR, the minimum index at which to
    evaluate a plan is:
    
        min(previousPlanResultIndex, plan.SnapshotIndex)
    schmichael committed Jun 24, 2019
    Configuration menu
    Copy the full SHA
    f82f2a6 View commit details
    Browse the repository at this point in the history
  3. nomad: SnapshotAfter -> SnapshotMinIndex

    Rename SnapshotAfter to SnapshotMinIndex. The old name was not
    technically accurate. SnapshotAtOrAfter is more accurate, but wordy and
    still lacks context about what precisely it is at or after (the index).
    
    SnapshotMinIndex was chosen as it describes the action (snapshot), a
    constraint (minimum), and the object of the constraint (index).
    schmichael committed Jun 24, 2019
    Configuration menu
    Copy the full SHA
    bcad687 View commit details
    Browse the repository at this point in the history

Commits on Jun 26, 2019

  1. Configuration menu
    Copy the full SHA
    887b49e View commit details
    Browse the repository at this point in the history