swarm: Add update/rollback order #30261

aaronlehmann · 2017-01-18T22:29:09Z

Update order is a new parameter to control the order of operations when rolling out an updated task. Either the old task is stopped before starting the new one, or the new task is started first, and the running tasks will briefly overlap.

Previously, stop_first was the only supported behavior. The behavior of start_first could be emulated by temporarily scaling up the service, so for awhile we hesitated to add it. However, there are some cases where it is awkward to change the number of replicas in a service just to support temporary overlap during updates. It doesn't work for some new features we have in mind like automatic preemption.

This PR adds Order to the API, and --update-order / --rollback-order flags to the CLI. It adds an integration test that performs a rolling update in the new start_first mode.

cc @aluzzardi @dongluochen

LK4D4 · 2017-01-27T19:33:24Z

Still depends on swarmkit PR

aaronlehmann · 2017-02-27T19:56:56Z

Rebased. This is still waiting for the swarmkit PR, but I've updated it to match a change in the swarmkit PR. --update-rollout was renamed to --update-order.

An additional option called --rollback-order will be needed. Where this is done will depend on whether this PR or #31108 is merged first.

aaronlehmann · 2017-03-10T19:54:52Z

Rebased and updated to support the rollback case. Still waiting for the swarmkit PR.

aaronlehmann · 2017-03-21T01:41:10Z

Swarmkit PR was merged

Updated this one to bring it in line, such as Order replacing RolloutMode.

This should finally be ready to go now.

thaJeztah · 2017-03-21T16:50:15Z

Looks like CI is failing @aaronlehmann

aaronlehmann · 2017-03-21T17:09:38Z

CI will be fixed when #31714 is merged. Until then, vendoring swarmkit is blocked. Let's try to make that happen.

aluzzardi · 2017-03-21T21:45:47Z

nit: Underscores vs dashes. I believe in the CLI we favor dashes (e.g. on-failure, unless-stopped, ...) while this PR is using underscores (start_first).

aaronlehmann · 2017-03-21T22:16:44Z

@aluzzardi: I think you're right about dashes vs. underscores. @thaJeztah can you please confirm?

thaJeztah · 2017-03-21T23:36:35Z

Oh, correct, yes. And, actually; the same is used for the API; https://github.com/docker/docker/blob/b36ce6f2f6cfdee4d376b58137dde2bc20fc4312/api/swagger.yaml#L304-L313

aaronlehmann · 2017-03-22T00:24:43Z

Updated to use start-first and stop-first in the API and CLI

dongluochen · 2017-03-24T18:21:40Z

cli/command/service/opts.go

@@ -494,6 +496,8 @@ func addServiceFlags(flags *pflag.FlagSet, opts *serviceOptions) {
 	flags.StringVar(&opts.update.onFailure, flagUpdateFailureAction, "pause", `Action on update failure ("pause"|"continue"|"rollback")`)
 	flags.Var(&opts.update.maxFailureRatio, flagUpdateMaxFailureRatio, "Failure rate to tolerate during an update")
 	flags.SetAnnotation(flagUpdateMaxFailureRatio, "version", []string{"1.25"})
+	flags.StringVar(&opts.update.order, flagUpdateOrder, "start-first", `Update order ("start-first"|"stop-first")`)


Default should be stop-first. stop-first is usually safe as tasks would not compete for resources, like CPU/mem/TCP ports, while start-first is not.

https://github.com/docker/swarmkit/blob/master/cmd/swarmctl/service/flagparser/flags.go#L39

Thanks, that was a mistake.

dongluochen

sgtm

dongluochen · 2017-03-24T18:52:22Z

docs/reference/commandline/service_update.md

@@ -74,6 +74,7 @@ Options:
      --rollback-max-failure-ratio float   Failure rate to tolerate during a rollback
      --rollback-monitor duration          Duration after each task rollback to monitor for failure
                                           (ns|us|ms|s|m|h) (default 0s)
+      --rollback-order string              Rollback order ("start-first"|"stop-first") (default "stop-first")


Stating default in service update command is a little misleading. If you don't specify the flag, it keeps existing value.

Agreed, however the other --update-* flags have the same problem.

--update-delay duration Delay between updates (ns|us|ms|s|m|h) (default 0s) --update-failure-action string Action on update failure ("pause"|"continue"|"rollback") (default "pause") --update-monitor duration Duration after each task update to monitor for failure (ns|us|ms|s|m|h) --update-parallelism uint Maximum number of tasks updated simultaneously (0 to update all at once) (default 1)

Let's deal with this in a separate PR. Would you mind filing an issue?

dongluochen · 2017-03-24T18:54:25Z

integration-cli/docker_api_swarm_service_test.go

@@ -175,6 +175,116 @@ func (s *DockerSwarmSuite) TestAPISwarmServicesUpdate(c *check.C) {
 		map[string]int{image1: instances})
 }

+func (s *DockerSwarmSuite) TestAPISwarmServicesUpdateStartFirst(c *check.C) {
+	const nodeCount = 3


nodeCount is not used here.

aaronlehmann · 2017-04-04T20:07:15Z

I've tried to rebuild rebuild/windowsRS1 several times and it always fails. Is this a known issue?

aaronlehmann · 2017-04-06T23:54:59Z

ping @docker/core-swarmkit-maintainers

Any chance of getting this in before the freeze?

dperny · 2017-04-06T23:59:57Z

I am not a maintainer, but it LGTM.

vieux · 2017-04-07T00:05:38Z

LGTM ping @aluzzardi

nishanttotla · 2017-04-07T00:21:04Z

daemon/cluster/convert/service.go

+	case types.UpdateOrderStartFirst:
+		converted.Order = swarmapi.UpdateConfig_START_FIRST
+	default:
+		return nil, fmt.Errorf("unrecongized update order %s", updateConfig.Order)


nit: typo unrecongized

Thanks. Fixed.

nishanttotla

LGTM (IANAM)

This parameter controls the order of operations when rolling out an update task. Either the old task is stopped before starting the new one, or the new task is started first, and the running tasks will briefly overlap. This commit adds Rollout to the API, and --update-order / --rollback-order flags to the CLI. Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>

cpuguy83

LGTM

aaronlehmann added area/swarm status/1-design-review status/needs-vendoring labels Jan 18, 2017

GordonTheTurtle added the status/0-triage label Jan 18, 2017

aaronlehmann removed the status/0-triage label Jan 18, 2017

thaJeztah mentioned this pull request Jan 23, 2017

Zero-downtime deployments with rolling upgrades #30321

Closed

thaJeztah mentioned this pull request Feb 2, 2017

Docker swarm mode rebalancing | Resurrection #30666

Closed

GordonTheTurtle assigned crosbymichael Feb 10, 2017

aaronlehmann force-pushed the rollout-mode branch from bc1f8c2 to 71b5f61 Compare February 27, 2017 19:55

thaJeztah mentioned this pull request Mar 6, 2017

Service update failure thresholds and rollback #26421

Merged

aaronlehmann force-pushed the rollout-mode branch from 71b5f61 to 1e4820a Compare March 10, 2017 19:54

thaJeztah mentioned this pull request Mar 20, 2017

A better docker service update behaviour toward services with replicas = 1 #31955

Open

aaronlehmann force-pushed the rollout-mode branch from 1e4820a to b748c60 Compare March 21, 2017 01:33

aaronlehmann changed the title ~~[WIP] swarm: Add rollout mode~~ swarm: Add update/rollback order Mar 21, 2017

aaronlehmann added the status/failing-ci Indicates that the PR in its current state fails the test suite label Mar 21, 2017

aaronlehmann force-pushed the rollout-mode branch from b748c60 to 72d1e5f Compare March 22, 2017 00:24

aaronlehmann removed the status/needs-vendoring label Mar 22, 2017

dongluochen suggested changes Mar 24, 2017

View reviewed changes

aaronlehmann force-pushed the rollout-mode branch from 72d1e5f to b836795 Compare March 24, 2017 18:33

dongluochen reviewed Mar 24, 2017

View reviewed changes

GordonTheTurtle removed rebuild/windowsRS1 labels Apr 3, 2017

aaronlehmann added the rebuild/windowsRS1 label Apr 4, 2017

GordonTheTurtle removed the rebuild/windowsRS1 label Apr 4, 2017

nishanttotla reviewed Apr 7, 2017

View reviewed changes

nishanttotla approved these changes Apr 7, 2017

View reviewed changes

aaronlehmann added 2 commits April 6, 2017 17:23

Add integration test for START_FIRST update order

6763641

Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>

aaronlehmann force-pushed the rollout-mode branch from 428fe0b to 6763641 Compare April 7, 2017 00:23

cpuguy83 approved these changes Apr 7, 2017

View reviewed changes

cpuguy83 added status/4-merge and removed status/2-code-review labels Apr 7, 2017

cpuguy83 merged commit 3de58eb into moby:master Apr 7, 2017

GordonTheTurtle added this to the 17.05.0 milestone Apr 7, 2017

thaJeztah added impact/api impact/changelog impact/cli impact/documentation labels Apr 7, 2017

aaronlehmann mentioned this pull request Apr 12, 2017

Support update-order/rollback-order in stack deploy #32586

Closed

aaronlehmann deleted the rollout-mode branch May 3, 2017 16:29

aaronlehmann mentioned this pull request May 3, 2017

[1.12.0-rc] docker service update --image should pull image before stopping services #24447

Open

albers mentioned this pull request Jun 27, 2017

Add bash completion for service create|update (update|rollback)-order docker/cli#256

Merged

sirlatrom mentioned this pull request Jul 14, 2017

Service update rollback with "start-first" replaces existing healthy tasks #34111

Closed

deviantony mentioned this pull request Aug 9, 2017

Add the ability to specify the Update configuration order of a service portainer/portainer#1100

Closed

This was referenced Jan 10, 2018

Need more documentation for "startup-order" / "rollback-order" docker/cli#795

Open

Docker Swarm update killing old containers too early before container is fully initialized #35881

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

swarm: Add update/rollback order #30261

swarm: Add update/rollback order #30261

aaronlehmann commented Jan 18, 2017 •

edited

Loading

LK4D4 commented Jan 27, 2017

aaronlehmann commented Feb 27, 2017

aaronlehmann commented Mar 10, 2017

aaronlehmann commented Mar 21, 2017

thaJeztah commented Mar 21, 2017

aaronlehmann commented Mar 21, 2017

aluzzardi commented Mar 21, 2017

aaronlehmann commented Mar 21, 2017

thaJeztah commented Mar 21, 2017

aaronlehmann commented Mar 22, 2017

dongluochen Mar 24, 2017

aaronlehmann Mar 24, 2017

dongluochen left a comment

dongluochen Mar 24, 2017

aaronlehmann Mar 24, 2017

dongluochen Mar 24, 2017

aaronlehmann commented Apr 4, 2017

aaronlehmann commented Apr 6, 2017

dperny commented Apr 6, 2017

vieux commented Apr 7, 2017

nishanttotla Apr 7, 2017

aaronlehmann Apr 7, 2017

nishanttotla left a comment

cpuguy83 left a comment

swarm: Add update/rollback order #30261

swarm: Add update/rollback order #30261

Conversation

aaronlehmann commented Jan 18, 2017 • edited Loading

LK4D4 commented Jan 27, 2017

aaronlehmann commented Feb 27, 2017

aaronlehmann commented Mar 10, 2017

aaronlehmann commented Mar 21, 2017

thaJeztah commented Mar 21, 2017

aaronlehmann commented Mar 21, 2017

aluzzardi commented Mar 21, 2017

aaronlehmann commented Mar 21, 2017

thaJeztah commented Mar 21, 2017

aaronlehmann commented Mar 22, 2017

dongluochen Mar 24, 2017

Choose a reason for hiding this comment

aaronlehmann Mar 24, 2017

Choose a reason for hiding this comment

dongluochen left a comment

Choose a reason for hiding this comment

dongluochen Mar 24, 2017

Choose a reason for hiding this comment

aaronlehmann Mar 24, 2017

Choose a reason for hiding this comment

dongluochen Mar 24, 2017

Choose a reason for hiding this comment

aaronlehmann commented Apr 4, 2017

aaronlehmann commented Apr 6, 2017

dperny commented Apr 6, 2017

vieux commented Apr 7, 2017

nishanttotla Apr 7, 2017

Choose a reason for hiding this comment

aaronlehmann Apr 7, 2017

Choose a reason for hiding this comment

nishanttotla left a comment

Choose a reason for hiding this comment

cpuguy83 left a comment

Choose a reason for hiding this comment

aaronlehmann commented Jan 18, 2017 •

edited

Loading