Deployment hooks proposal #1224

ironcladlou · 2015-03-04T21:57:15Z

This PR is a WIP to discuss the design of a deployment hook mechanism for https://trello.com/c/Zea4kjvo.

danmcp · 2015-03-04T22:01:26Z

docs/proposals/post-deployment-hooks.md

+
+## Use cases
+
+1. As a MySQL user, I want my application deployment to seed the database with data the first time the database is used.


This sounds like the first user has to wait for the db to be populated when they hit the app in the browser. But I think you mean it's part of the deployment of a component(s) that consume the database.

I took another stab at this based on your feedback.

ironcladlou · 2015-03-05T21:53:55Z

@pmorie @bparees @ncdc @smarterclayton @pweil- PTAL- redid most of it in terms of a run-once pod design.

ironcladlou · 2015-03-05T21:56:39Z

How could I forget @danmcp

pmorie · 2015-03-05T23:09:50Z

docs/proposals/post-deployment-hooks.md

+3. Describe a deployment hook API
+
+
+## Comparison of potential approaches


I would make a note here saying that fundamentally, these are the choices you have to evaluate the use-cases in terms of, since this ties into the next section.

I revised this section, ptal.

smarterclayton · 2015-03-07T21:41:38Z

The "notification" use case mentioned here should be separated from the "hook" use case as I describe in #1228. Hooks run during deployments (pre, mid, post) and are dealing with the execution environment of the deployment itself. Notifications happen after deployments transition into a terminal phase.

ironcladlou · 2015-03-09T14:41:35Z

The "notification" use case mentioned here should be separated from the "hook" use case as I describe in #1228. Hooks run during deployments (pre, mid, post) and are dealing with the execution environment of the deployment itself. Notifications happen after deployments transition into a terminal phase.

Do you agree that my call-a-webhook use case here is a "hook" rather than a "notification" if the outcome of the execution is allowed to impact the outcome of the deployment? Keep in mind that in the design section I point out that we don't currently have any proposal for how to handle post-deployment hooks failures other than manual intervention- so without a solution to that failure mode, I don't know what impact ANY post hook could have on the deployment other than reporting warnings.

smarterclayton · 2015-03-09T15:33:24Z

Yes. Hooks are imperative "do this as part of the innate lifecycle of this object". Notifications are post-hoc, triggers are pre-hoc.

----- Original Message -----

The "notification" use case mentioned here should be separated from the
"hook" use case as I describe in #1228. Hooks run during deployments (pre,
mid, post) and are dealing with the execution environment of the deployment
itself. Notifications happen after deployments transition into a terminal
phase.

Do you agree that my call-a-webhook use case here is a "hook" rather than a
"notification" if the outcome of the execution is allowed to impact the
outcome of the deployment? Keep in mind that in the design section I point
out that we don't currently have any proposal for how to handle
post-deployment hooks failures other than manual intervention- so without a
solution to that failure mode, I don't know what impact ANY post hook could
have on the deployment other than reporting warnings.

Reply to this email directly or view it on GitHub:
#1224 (comment)

bparees · 2015-03-09T15:37:26Z

@smarterclayton that would imply that a post-deployment-hook failure must rollback (or otherwise impact) the deployment, otherwise it's not an innate part of the lifecycle any more so than a notification would be.

smarterclayton · 2015-03-09T19:49:57Z

If a post deployment hook fails most users will want to know. It may not rollback, but it should certainly tell a user. The thing about most hooks is that they are not implicitly recoverable - migrations are one way, updates in a remote service are one way. I think hooks rolling back is somewhat wishful thinking on our part.

----- Original Message -----

@smarterclayton that would imply that a post-deployment-hook failure must
rollback (or otherwise impact) the deployment, otherwise it's not an innate
part of the lifecycle any more so than a notification would be.

Reply to this email directly or view it on GitHub:
#1224 (comment)

ironcladlou · 2015-03-10T19:09:14Z

Added a simple API proposal. Could it be that simple?

ironcladlou · 2015-03-10T19:11:21Z

cc @abhgupta

ironcladlou · 2015-03-12T13:29:02Z

@smarterclayton This needs more review.

ironcladlou · 2015-03-12T15:23:15Z

Something we might still need are new deployment statuses distinct from "Running" to represent hook executions. Otherwise implementation is going to be really difficult. Hopefully those new statuses would have value outside the context of an implementation detail so I can justify them.

smarterclayton · 2015-03-12T16:22:24Z

If hooks are run once you should create a status / equiv field for "I ran this". Derek can explain what he's doing for namespaces and help guide that. Status should have limited states.

On Mar 12, 2015, at 11:23 AM, Dan Mace notifications@github.com wrote:

Something we might still need are new deployment statuses distinct from "Running" to represent hook executions. Otherwise implementation is going to be really difficult. Hopefully those new statuses would have value outside the context of an implementation detail so I can justify them.

—
Reply to this email directly or view it on GitHub.

smarterclayton · 2015-03-12T19:08:41Z

docs/proposals/post-deployment-hooks.md

+Deployment hooks implemented as run-once pods external to the Kubelet provide a different set of guarantees:
+
+1. Hooks can be bound to the logical deployment lifecycle, enabling hook executions decoupled from replication mechanics.
+  1. Races can be avoided by defining a hook that executes exactly once per deployment regardless of the replica count.


At most once

i'd want at least once. or at least an option for at least once. if you bring up my app w/o having successfully run my db migration i'm not going to be happy. if you run my db migration twice, that might be ok (if i wrote my db migration intelligently)

It's also cloudier.

On Mar 12, 2015, at 3:21 PM, Ben Parees notifications@github.com wrote:

In docs/proposals/post-deployment-hooks.md:

+2. The status of the pod is linked to the status and outcome of the hook execution.

The pod will not enter a ready status until the hook has completed successfully.

Service endpoints will not be created for the pod until the pod has a ready status.

If the hook fails, the pod's creation is considered a failure, and the retry behavior is restart-policy driven in the usual way.

+Because deployments are represented as replication controllers, lifecycle hooks defined for containers are executed for every container in the replica set for the deployment. This behavior has complexity implications when applied to deployment use cases:
+
+1. The hooks for all pods in the deployment will race, placing a burden on hook authors (e.g., the hooks would generally need to be tolerant of concurrent hook execution and implement manual coordination.)
+
+
+##### Deployment hooks
+
+Deployment hooks implemented as run-once pods external to the Kubelet provide a different set of guarantees:
+
+1. Hooks can be bound to the logical deployment lifecycle, enabling hook executions decoupled from replication mechanics.

Races can be avoided by defining a hook that executes exactly once per deployment regardless of the replica count.
i'd want at least once. or at least an option for at least once. if you bring up my app w/o having successfully run my db migration i'm not going to be happy. if you run my db migration twice, that might be ok (if i wrote my db migration intelligently)

—
Reply to this email directly or view it on GitHub.

Agree w/ "at least," changed.

ironcladlou · 2015-03-19T20:30:28Z

@smarterclayton Digested your comments and studies the upstream container lifecycle API more closey. I added some commits which rework the hook API to be much closer to the upstream lifecycle/handler setup. PTAL.

pmorie · 2015-03-19T21:56:58Z

@smarterclayton are saying that this hook would win in a fight against
ExecAction?
On Thu, Mar 19, 2015 at 9:30 PM Dan Mace notifications@github.com wrote:

@smarterclayton https://github.com/smarterclayton Digested your
comments and studies the upstream container lifecycle API more closey. I
added some commits which rework the hook API to be much closer to the
upstream lifecycle/handler setup. PTAL.

—
Reply to this email directly or view it on GitHub
#1224 (comment).

pmorie · 2015-03-23T14:25:05Z

docs/proposals/post-deployment-hooks.md

+
+##### Deployment hooks
+
+Deployment hooks implemented as run-once pods external to the Kubelet provide a different set of guarantees:


You refer to deployment hooks here without definining them. Add a couple sentences saying what this is:

An alternative to the upstream-provided lifecycle hooks is to have a notion of a hook which is a property of an OpenShift deployment.

^^ something like that

Good idea; revised.

smarterclayton · 2015-03-23T17:19:59Z

Proposal looks good minus that one issue - fix that (add something to the example), and then squash and I'll merge. Do you have a prototype yet?

ironcladlou · 2015-03-23T17:27:10Z

Added env to API and example, and squashed. No prototype yet, but now that the API and behavior is pretty well specified, I expect the implementation will come along quickly.

smarterclayton · 2015-03-23T17:31:44Z

docs/proposals/post-deployment-hooks.md

+            "command": ["rake", "db:migrate"],
+            "env": [
+              {
+                "key": "CUSTOM_VAR",


Key is not required, better to remove from the example

Oops, fixed.

smarterclayton · 2015-03-23T17:47:09Z

give your commit a title

ironcladlou · 2015-03-23T17:49:51Z

Hrm, rebase fail. Fixed.

smarterclayton · 2015-03-23T17:59:56Z

LGTM [merge]

openshift-bot · 2015-03-23T18:00:42Z

continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/merge_pull_requests_openshift3/1267/) (Image: devenv-fedora_1112)

openshift-bot · 2015-03-23T18:00:43Z

Evaluated for origin up to 8f765fb

Merged by openshift-bot

…service-catalog/' changes from ef63307bdb..ae6b643caf ae6b643caf Use oc adm instead of oadm which might not exist in various installations. 66a4eb2a2c Update instructions... will remove once documented elsewhere 1b704d1530 replace build context setup with init containers ee4df18c7f hack/lib: dedup os::util::host_platform and os::build::host_platform 1cd6dfa998 origin: Switch out owners to Red Hatters 664f4d318f Add instructions for syncing repos 2f2cdd546b origin-build: delete files with colon in them cdf8b12848 origin-build: don't build user-broker ebfede9056 origin build: add _output to .gitignore 55412c7e3d origin build: make build-go and build-cross work 68c74ff4ae origin build: modify hard coded path 3d41a217f6 origin build: add origin tooling a8fc27d Fix typos in walkthrough (openshift#1224) e77edbf openshift#1157: Limit the amount of time for reconciliations (openshift#1196) 1b1a749 temporarily disabled verify-links.sh from the verify target (openshift#1226) acf8fab Send originating identity headers in OSB requests (openshift#1162) 821ba16 new admission controller to block updates to service instance updates that (openshift#1210) d69c5e5 Minor improvement to godoc in binaries (openshift#1211) 5b81814 fix typos (openshift#1221) 836dc4a Adds how to download Helm chart (charts/catalog) (openshift#1219) 2fd0115 Fix "visit the project on github" link. (openshift#1217) 325e4b6 Add how to set $GOPATH. (openshift#1218) 68b775f Update the installation (openshift#1199) 6e3a3c1 v0.0.19 (openshift#1207) 8b69791 Removing errexit from TLS setup script (openshift#1206) 273260f Instance deletion lifecycle enhancements, issue openshift#820 (openshift#1159) c050713 fix cleaning of build output for non-root users (openshift#1205) 5995df1 Merge branch 'pr/1204' 72f4802 Remove osb prefix from example ServiceClass (openshift#1201) f9dbd4e pin all dependencies in glide to current version except for glog where we want to pick up the prior version to fix issue 1187. f148bc5 v0.0.18 (openshift#1202) b86ab8d Removing the helm install command (openshift#1185) 3cff482 Remove Alpha* prefix on all API fields for issue openshift#1180 (openshift#1184) 154b74d Fix gofmt issue (openshift#1192) 2ee894a do the clean before building an arch (openshift#1179) b4976ef Fix bad URL (openshift#1189) cd3dede Fix hrefs again (openshift#1190) f066226 Design: Instance/Binding parameters (openshift#1075) eb37682 This generated file is missing from master (openshift#1191) 28c0ae7 Use generation instead of checksum for Instances and InstanceCredentials (openshift#1151) 5cdd323 Fix bad href (openshift#1188) 8a892f0 handle lingering polling cases (openshift#1174) f5fabd6 remove TPRs from Jenkins e2e pipeline (openshift#1175) 717df78 Add godoc explaining that Instance and InstanceCredential specs are immutable (openshift#1182) REVERT: ef63307bdb origin build: add origin tooling git-subtree-dir: cmd/service-catalog/go/src/github.com/kubernetes-incubator/service-catalog git-subtree-split: ae6b643cafd3a17412f173e70ed7c1a2e39ee549

* Updates walkthrough-1.6.md Updates incorrect contents in walkthrough-1.6.md. * Updates install-1.6.md Updates the clean up instruction in install-1.6.md. * Updates walkthrough-1.7.md Applies the change in walkthrough-1.6.md to walkthrough-1.7.md. * Troubleshooting is out of clean up section Move troubleshooting section out of cleanup-section. * Moves troubleshooting out of cleanup section Troubleshooting section is not part of cleanup section.

danmcp reviewed Mar 4, 2015
View reviewed changes

ironcladlou changed the title ~~WIP: Post-deployment hooks proposal~~ WIP: Deployment hooks proposal Mar 5, 2015

pmorie reviewed Mar 5, 2015
View reviewed changes

soltysh mentioned this pull request Mar 6, 2015

add a build completed notification so that we can chain builds together to make a Deployment Pipline #1228

Closed

ironcladlou mentioned this pull request Mar 9, 2015

[deployment config] stuck endlessly creating pods that will error #1214

Closed

smarterclayton reviewed Mar 12, 2015
View reviewed changes

ironcladlou force-pushed the post-deployment-hook-proposal branch from ad4ff5e to 95efb4c Compare March 23, 2015 14:04

pmorie reviewed Mar 23, 2015
View reviewed changes

ironcladlou force-pushed the post-deployment-hook-proposal branch from e7bfd5c to 5320be5 Compare March 23, 2015 17:25

smarterclayton reviewed Mar 23, 2015
View reviewed changes

ironcladlou force-pushed the post-deployment-hook-proposal branch from 5320be5 to 77b52ef Compare March 23, 2015 17:34

smarterclayton changed the title ~~WIP: Deployment hooks proposal~~ Deployment hooks proposal Mar 23, 2015

Introduce deployment hook proposal

8f765fb

ironcladlou force-pushed the post-deployment-hook-proposal branch from 77b52ef to 8f765fb Compare March 23, 2015 17:49

openshift-bot pushed a commit that referenced this pull request Mar 23, 2015

Merge pull request #1224 from ironcladlou/post-deployment-hook-proposal

b5f78c7

Merged by openshift-bot

openshift-bot merged commit b5f78c7 into openshift:master Mar 23, 2015

ironcladlou mentioned this pull request Mar 25, 2015

WIP: Implement deployment lifecycle hooks #1456

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Deployment hooks proposal #1224

Deployment hooks proposal #1224

ironcladlou commented Mar 4, 2015

danmcp Mar 4, 2015

ironcladlou Mar 4, 2015

ironcladlou commented Mar 5, 2015

ironcladlou commented Mar 5, 2015

pmorie Mar 5, 2015

ironcladlou Mar 6, 2015

smarterclayton commented Mar 7, 2015

ironcladlou commented Mar 9, 2015

smarterclayton commented Mar 9, 2015

bparees commented Mar 9, 2015

smarterclayton commented Mar 9, 2015

ironcladlou commented Mar 10, 2015

ironcladlou commented Mar 10, 2015

ironcladlou commented Mar 12, 2015

ironcladlou commented Mar 12, 2015

smarterclayton commented Mar 12, 2015

smarterclayton Mar 12, 2015

bparees Mar 12, 2015

smarterclayton Mar 12, 2015

ironcladlou Mar 12, 2015

ironcladlou commented Mar 19, 2015

pmorie commented Mar 19, 2015

pmorie Mar 23, 2015

ironcladlou Mar 23, 2015

smarterclayton commented Mar 23, 2015

ironcladlou commented Mar 23, 2015

smarterclayton Mar 23, 2015

ironcladlou Mar 23, 2015

smarterclayton commented Mar 23, 2015

ironcladlou commented Mar 23, 2015

smarterclayton commented Mar 23, 2015

openshift-bot commented Mar 23, 2015

openshift-bot commented Mar 23, 2015


		## Use cases

		1. As a MySQL user, I want my application deployment to seed the database with data the first time the database is used.

		3. Describe a deployment hook API


		## Comparison of potential approaches


		##### Deployment hooks

		Deployment hooks implemented as run-once pods external to the Kubelet provide a different set of guarantees:

Deployment hooks proposal #1224

Deployment hooks proposal #1224

Conversation

ironcladlou commented Mar 4, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ironcladlou commented Mar 5, 2015

ironcladlou commented Mar 5, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smarterclayton commented Mar 7, 2015

ironcladlou commented Mar 9, 2015

smarterclayton commented Mar 9, 2015

bparees commented Mar 9, 2015

smarterclayton commented Mar 9, 2015

ironcladlou commented Mar 10, 2015

ironcladlou commented Mar 10, 2015

ironcladlou commented Mar 12, 2015

ironcladlou commented Mar 12, 2015

smarterclayton commented Mar 12, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ironcladlou commented Mar 19, 2015

pmorie commented Mar 19, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smarterclayton commented Mar 23, 2015

ironcladlou commented Mar 23, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smarterclayton commented Mar 23, 2015

ironcladlou commented Mar 23, 2015

smarterclayton commented Mar 23, 2015

openshift-bot commented Mar 23, 2015

openshift-bot commented Mar 23, 2015