Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce deployments to make rolling updates based on allocation health #2799

Merged
merged 105 commits into from
Jul 7, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
105 commits
Select commit Hold shift + click to select a range
bb65241
cancel deployments
dadgar May 18, 2017
df40bd8
initial reconciler
dadgar May 22, 2017
c16195a
Only upsert a job if the spec changes and push deployment creation in…
dadgar May 23, 2017
0d3c836
Small cleanup
dadgar May 23, 2017
4bbf24a
Split reconcile file
dadgar May 23, 2017
3ae7aba
Some comments and cleanup
dadgar May 23, 2017
a7df708
Todos
dadgar May 23, 2017
662bc39
handle annotations
dadgar May 23, 2017
66789e8
Show canaries on plan
dadgar May 23, 2017
201ad8b
Populate desired state per tg
dadgar May 23, 2017
2dc0268
Plan reuses job where possible
dadgar May 23, 2017
4c9e5c3
Remove old
dadgar May 23, 2017
1dabd20
handle batch filtering
dadgar May 24, 2017
85e0d6f
assign names
dadgar May 31, 2017
d72b270
Pull out in-place updating into a passed in function; reduce inputs t…
dadgar May 31, 2017
f32a9a5
Non-Canary/Deployment Tests
dadgar Jun 1, 2017
369a04b
Deployment tests
dadgar Jun 2, 2017
af7f93b
Fix canary handling
dadgar Jun 6, 2017
ff8c057
SpecChanged doesn't mutate passed job
dadgar Jun 21, 2017
0d29972
cleanup limit detection
dadgar Jun 21, 2017
0ec6d74
update description of the alloc update factory function
dadgar Jun 21, 2017
53f4952
initial impl
dadgar Jun 26, 2017
b6277af
state store tests
dadgar Jun 27, 2017
9213bb4
FSM Tests
dadgar Jun 27, 2017
7d9d85b
Deployments list
dadgar Jun 27, 2017
05c710d
Watcher
dadgar Jun 27, 2017
7e43ed7
Deployment watcher tests
dadgar Jun 28, 2017
bd02a84
more tests
dadgar Jun 28, 2017
5d5e287
Fix unnecessary evals
dadgar Jun 28, 2017
3615ffd
batch test
dadgar Jun 28, 2017
aeeb0a6
comments
dadgar Jun 28, 2017
c86af4d
Add watcher to server
dadgar Jun 28, 2017
6aae18e
fix integration slightly
dadgar Jun 28, 2017
6019915
FailDeployment
dadgar Jun 28, 2017
d3e02c6
Tests
dadgar Jun 29, 2017
bb0d97a
Fix tests
dadgar Jun 29, 2017
28e4fe7
fix index
dadgar Jul 3, 2017
1a4bd26
comments on watcher
dadgar Jul 3, 2017
e6a1266
Remove setters
dadgar Jul 3, 2017
174e4f7
simplify the batcher's timers
dadgar Jul 3, 2017
0ba6722
more comment fixes
dadgar Jul 3, 2017
31daf93
HTTP Endpoints
dadgar Jun 29, 2017
aa5bf9b
deployment api
dadgar Jun 29, 2017
89d86bc
Deployments HTTP docs
dadgar Jun 29, 2017
d0a8332
job deployment endpoint + api
dadgar Jul 1, 2017
86e49b9
Add config options
dadgar Jun 29, 2017
24635f8
Deployment GC
dadgar Jun 29, 2017
9220836
JobVersions returns struct with optional diff
dadgar Jun 30, 2017
1c425de
job history
dadgar Jun 30, 2017
3935656
Show submit time
dadgar Jun 30, 2017
fbd2b73
job revert
dadgar Jun 30, 2017
3d77a58
small fixes
dadgar Jun 30, 2017
dc3d500
deployment list
dadgar Jun 30, 2017
bab25f6
deployment status
dadgar Jun 30, 2017
d18a3df
fail,pause,resume commands
dadgar Jun 30, 2017
004a766
deployment promote
dadgar Jun 30, 2017
4ce7b62
job deployments
dadgar Jul 1, 2017
468c886
Formatting abilities
dadgar Jul 1, 2017
7456351
Small fixes
dadgar Jul 4, 2017
7e50771
Add deployment id to alloc
dadgar Jul 2, 2017
da82a6e
initial watcher
dadgar Jul 3, 2017
8e58ddc
Update index
dadgar Jul 3, 2017
d165f65
watcher per alloc
dadgar Jul 3, 2017
f72bbaa
Client watches for allocation health using task state and Consul checks
dadgar Jul 4, 2017
b8ba29b
Warn log
dadgar Jul 6, 2017
591ef9c
Rename CreateDeployments and remove cancelling behavior in state_store
dadgar Jul 4, 2017
c3626cd
vendor file
dadgar Jul 4, 2017
e5b1e31
Remove promoted bit from allocation
dadgar Jul 4, 2017
71c7c45
Change canary handling
dadgar Jul 5, 2017
0c6a1c0
Mark complete
dadgar Jul 5, 2017
29e31af
Attach eval id
dadgar Jul 6, 2017
598748d
Fix handling of failed job
dadgar Jul 6, 2017
690fc78
Plan apply handles canaries and success is set via update
dadgar Jul 6, 2017
463d20e
Test marking as complete
dadgar Jul 6, 2017
127d43d
Test scheduler's handling of canaries/inplace updates
dadgar Jul 6, 2017
b7088b3
plan apply tests
dadgar Jul 6, 2017
989aa56
Remove canary
dadgar Jul 6, 2017
b3ec146
Respond to comments
dadgar Jul 6, 2017
62550c1
Complete deployments mark jobs as stable
dadgar Jul 6, 2017
aaf5ab0
Job stability
dadgar Jul 6, 2017
65abc9a
HTTP API docs for job
dadgar Jul 6, 2017
4e4c4a0
feedback
dadgar Jul 6, 2017
34679cf
add reverted job version to deployment update response
dadgar Jul 6, 2017
fec6f2e
Return the reverted job version
dadgar Jul 6, 2017
6925255
add to docs
dadgar Jul 6, 2017
78d1580
Deployment from inplace updates tracks placed properly.
dadgar Jul 6, 2017
f0a090c
Disallow update stanza on batch jobs
dadgar Jul 7, 2017
6639f31
Fix some commands test
dadgar Jul 7, 2017
53c1294
Vendor columnize
dadgar Jul 7, 2017
fe990ca
Fix some tests, eval monitor shows deployment id and deployment cance…
dadgar Jul 7, 2017
09af3f9
Fix broken things on deployment promote
dadgar Jul 7, 2017
5bcf2dd
deployment status indicates whether canaries were promoted
dadgar Jul 7, 2017
f3ffc3d
Fix JobModifyIndex changing when job is marked stable
dadgar Jul 7, 2017
c5b1e2d
alloc-list shows version
dadgar Jul 7, 2017
e5d8c08
alloc-status
dadgar Jul 7, 2017
466702a
deployment status in job status view
dadgar Jul 7, 2017
e1c6310
@jippi Changed my mind! Good suggestion
dadgar Jul 7, 2017
27048bb
events
dadgar Jul 7, 2017
9eb2c8f
Status description shows requiring promotion
dadgar Jul 7, 2017
0244721
an user -> a user
schmichael Jul 7, 2017
0129455
Rolling node drains using max_parallel and stagger
dadgar Jul 7, 2017
38e50ee
check id method name changed
dadgar Jul 7, 2017
50e3178
vendor
dadgar Jul 7, 2017
beb01f1
test fixes
dadgar Jul 7, 2017
3beaafc
Vet and small improvement on watcher failure detection
dadgar Jul 7, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions api/allocations.go
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,8 @@ type Allocation struct {
ClientStatus string
ClientDescription string
TaskStates map[string]*TaskState
DeploymentID string
DeploymentStatus *AllocDeploymentStatus
PreviousAllocation string
CreateIndex uint64
ModifyIndex uint64
Expand Down Expand Up @@ -136,17 +138,27 @@ type AllocationListStub struct {
Name string
NodeID string
JobID string
JobVersion uint64
TaskGroup string
DesiredStatus string
DesiredDescription string
ClientStatus string
ClientDescription string
TaskStates map[string]*TaskState
DeploymentStatus *AllocDeploymentStatus
CreateIndex uint64
ModifyIndex uint64
CreateTime int64
}

// AllocDeploymentStatus captures the status of the allocation as part of the
// deployment. This can include things like if the allocation has been marked as
// heatlhy.
type AllocDeploymentStatus struct {
Healthy *bool
ModifyIndex uint64
}

// AllocIndexSort reverse sorts allocs by CreateIndex.
type AllocIndexSort []*AllocationListStub

Expand Down
233 changes: 233 additions & 0 deletions api/deployments.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,233 @@
package api

import (
"sort"
)

// Deployments is used to query the deployments endpoints.
type Deployments struct {
client *Client
}

// Deployments returns a new handle on the deployments.
func (c *Client) Deployments() *Deployments {
return &Deployments{client: c}
}

// List is used to dump all of the evaluations.
func (d *Deployments) List(q *QueryOptions) ([]*Deployment, *QueryMeta, error) {
var resp []*Deployment
qm, err := d.client.query("/v1/deployments", &resp, q)
if err != nil {
return nil, nil, err
}
sort.Sort(DeploymentIndexSort(resp))
return resp, qm, nil
}

func (d *Deployments) PrefixList(prefix string) ([]*Deployment, *QueryMeta, error) {
return d.List(&QueryOptions{Prefix: prefix})
}

// Info is used to query a single evaluation by its ID.
func (d *Deployments) Info(deploymentID string, q *QueryOptions) (*Deployment, *QueryMeta, error) {
var resp Deployment
qm, err := d.client.query("/v1/deployment/"+deploymentID, &resp, q)
if err != nil {
return nil, nil, err
}
return &resp, qm, nil
}

// Allocations is used to retrieve a set of allocations that are part of the
// deployment
func (d *Deployments) Allocations(deploymentID string, q *QueryOptions) ([]*AllocationListStub, *QueryMeta, error) {
var resp []*AllocationListStub
qm, err := d.client.query("/v1/deployment/allocations/"+deploymentID, &resp, q)
if err != nil {
return nil, nil, err
}
sort.Sort(AllocIndexSort(resp))
return resp, qm, nil
}

// Fail is used to fail the given deployment.
func (d *Deployments) Fail(deploymentID string, q *WriteOptions) (*DeploymentUpdateResponse, *WriteMeta, error) {
var resp DeploymentUpdateResponse
req := &DeploymentFailRequest{
DeploymentID: deploymentID,
}
wm, err := d.client.write("/v1/deployment/fail/"+deploymentID, req, &resp, q)
if err != nil {
return nil, nil, err
}
return &resp, wm, nil
}

// Pause is used to pause or unpause the given deployment.
func (d *Deployments) Pause(deploymentID string, pause bool, q *WriteOptions) (*DeploymentUpdateResponse, *WriteMeta, error) {
var resp DeploymentUpdateResponse
req := &DeploymentPauseRequest{
DeploymentID: deploymentID,
Pause: pause,
}
wm, err := d.client.write("/v1/deployment/pause/"+deploymentID, req, &resp, q)
if err != nil {
return nil, nil, err
}
return &resp, wm, nil
}

// PromoteAll is used to promote all canaries in the given deployment
func (d *Deployments) PromoteAll(deploymentID string, q *WriteOptions) (*DeploymentUpdateResponse, *WriteMeta, error) {
var resp DeploymentUpdateResponse
req := &DeploymentPromoteRequest{
DeploymentID: deploymentID,
All: true,
}
wm, err := d.client.write("/v1/deployment/promote/"+deploymentID, req, &resp, q)
if err != nil {
return nil, nil, err
}
return &resp, wm, nil
}

// PromoteGroups is used to promote canaries in the passed groups in the given deployment
func (d *Deployments) PromoteGroups(deploymentID string, groups []string, q *WriteOptions) (*DeploymentUpdateResponse, *WriteMeta, error) {
var resp DeploymentUpdateResponse
req := &DeploymentPromoteRequest{
DeploymentID: deploymentID,
Groups: groups,
}
wm, err := d.client.write("/v1/deployment/promote/"+deploymentID, req, &resp, q)
if err != nil {
return nil, nil, err
}
return &resp, wm, nil
}

// SetAllocHealth is used to set allocation health for allocs that are part of
// the given deployment
func (d *Deployments) SetAllocHealth(deploymentID string, healthy, unhealthy []string, q *WriteOptions) (*DeploymentUpdateResponse, *WriteMeta, error) {
var resp DeploymentUpdateResponse
req := &DeploymentAllocHealthRequest{
DeploymentID: deploymentID,
HealthyAllocationIDs: healthy,
UnhealthyAllocationIDs: unhealthy,
}
wm, err := d.client.write("/v1/deployment/allocation-health/"+deploymentID, req, &resp, q)
if err != nil {
return nil, nil, err
}
return &resp, wm, nil
}

// Deployment is used to serialize an deployment.
type Deployment struct {
ID string
JobID string
JobVersion uint64
JobModifyIndex uint64
JobCreateIndex uint64
TaskGroups map[string]*DeploymentState
Status string
StatusDescription string
CreateIndex uint64
ModifyIndex uint64
}

// DeploymentState tracks the state of a deployment for a given task group.
type DeploymentState struct {
PlacedCanaries []string
AutoRevert bool
Promoted bool
DesiredCanaries int
DesiredTotal int
PlacedAllocs int
HealthyAllocs int
UnhealthyAllocs int
}

// DeploymentIndexSort is a wrapper to sort deployments by CreateIndex. We
// reverse the test so that we get the highest index first.
type DeploymentIndexSort []*Deployment

func (d DeploymentIndexSort) Len() int {
return len(d)
}

func (d DeploymentIndexSort) Less(i, j int) bool {
return d[i].CreateIndex > d[j].CreateIndex
}

func (d DeploymentIndexSort) Swap(i, j int) {
d[i], d[j] = d[j], d[i]
}

// DeploymentUpdateResponse is used to respond to a deployment change. The
// response will include the modify index of the deployment as well as details
// of any triggered evaluation.
type DeploymentUpdateResponse struct {
EvalID string
EvalCreateIndex uint64
DeploymentModifyIndex uint64
RevertedJobVersion *uint64
WriteMeta
}

// DeploymentAllocHealthRequest is used to set the health of a set of
// allocations as part of a deployment.
type DeploymentAllocHealthRequest struct {
DeploymentID string

// Marks these allocations as healthy, allow further allocations
// to be rolled.
HealthyAllocationIDs []string

// Any unhealthy allocations fail the deployment
UnhealthyAllocationIDs []string

WriteRequest
}

// DeploymentPromoteRequest is used to promote task groups in a deployment
type DeploymentPromoteRequest struct {
DeploymentID string

// All is to promote all task groups
All bool

// Groups is used to set the promotion status per task group
Groups []string

WriteRequest
}

// DeploymentPauseRequest is used to pause a deployment
type DeploymentPauseRequest struct {
DeploymentID string

// Pause sets the pause status
Pause bool

WriteRequest
}

// DeploymentSpecificRequest is used to make a request specific to a particular
// deployment
type DeploymentSpecificRequest struct {
DeploymentID string
QueryOptions
}

// DeploymentFailRequest is used to fail a particular deployment
type DeploymentFailRequest struct {
DeploymentID string
WriteRequest
}

// SingleDeploymentResponse is used to respond with a single deployment
type SingleDeploymentResponse struct {
Deployment *Deployment
QueryMeta
}
1 change: 1 addition & 0 deletions api/evaluations.go
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ type Evaluation struct {
JobModifyIndex uint64
NodeID string
NodeModifyIndex uint64
DeploymentID string
Status string
StatusDescription string
Wait time.Duration
Expand Down
Loading