Merge pull request #132 from phylake/hotswap

Bug fix

CHANGELOG.md

@@ -1,3 +1,8 @@
+### v3.0.1
+
+- allow 10 mins for service payload download+install during hot swap
+- check for egress rules before writing `SecurityGroupEgress`
+
 ### v3.0.0
 
 - updated to Amazon Linux 2016.09

commands/build/provision.go

@@ -477,13 +477,15 @@ func hotswapStackPoll(log log15.Logger, environment *conf.Environment,
 	// 2. variable: time to download and start the service
 	// 3. HC_HealthyThreshold * HC_Interval seconds: health check on each container
 	// 4. ~ 1min: to complete haproxy reload which is the Keep-Alive time from ELB
+	// 5. 1min: timeout on hot swap signal
 	//
-	// That's 2m 15s excluding step 2
+	// That's 3m 15s excluding step 2
+	// Allow 10 mins for step 2
 	receiveSuccess := 0
 	loopCount := 0
 	for asgSize != receiveSuccess {
 
-		if loopCount == 30 {
+		if loopCount == 40 {
 			log.Error("Never received messages from all EC2 instances")
 			return
 		}

docs/detailed_design/hotswap.md

@@ -18,9 +18,35 @@ For these reasons porter enforces that hot swap can only occur for a maximum of
 between a fast feedback loop, a good default security posture, and avoiding
 infrastructure drift.
 
+Assumptions
+-----------
+
+The assumptions exist in normal provisioning as well but they are slightly
+different in the hot swap flow and worth calling out.
+
+### Short-lived connections
+
+ELB keep-alive to HAProxy is 60 seconds. Per-host, porter waits an additional
+120 seconds before declaring hot swap a failure. The primary assumption of hot
+swap is short-lived connections. If you have long-lived connections you'll want
+to avoid hot swap. That said you can't have longer-lived connections than what
+you set your ELB connection draining to anyway (max is 3600 seconds).
+
+### Version compatibility
+
+Partial failures can occur and porter doesn't try to repair them. In the worst
+case a multi-region hot swap could see a whole region fail to hot swap and other
+regions partially fail to hot swap.
+
+If version A of your software is deployed and can not coexist with version B
+then you should not use hot swap.
+
 When not to use it
 ------------------
 
+An extension of assumptions relates to best practices around delivering SaaS -
+namely testing.
+
 Often integration tests are run in the `pre_promote` hook as a quality gate.
 Since promotion is skipped there's no chance to run tests against your running
 code before it starts serving traffic.
@@ -30,13 +56,6 @@ environment. Both can be configured to hot swap and in addition to a
 `pre_promote` hook you would configure stage to run a `post_hotswap` hook that
 performs the same testing as a gate to allow code through to production.
 
-Caveats
--------
-
-The AutoScalingGroup's min/max size isn't matched with what currently deployed
-meaning any changes that were made to the AutoScaling group after porter
-deployed it are lost.
-
 How it works
 ------------
 

provision/map_resource.go

@@ -216,6 +216,10 @@ func overwriteASGSecurityGroupEgress(recv *stackCreator, template *cfn.Template,
 		return true
 	}
 
+	if len(recv.region.AutoScalingGroup.SecurityGroupEgress) == 0 {
+		return true
+	}
+
 	if props, ok = resource["Properties"].(map[string]interface{}); !ok {
 		recv.log.Error("Missing Properties on resource")
 		return false