-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Deployment hooks proposal #1224
Deployment hooks proposal #1224
Conversation
|
||
## Use cases | ||
|
||
1. As a MySQL user, I want my application deployment to seed the database with data the first time the database is used. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This sounds like the first user has to wait for the db to be populated when they hit the app in the browser. But I think you mean it's part of the deployment of a component(s) that consume the database.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I took another stab at this based on your feedback.
@pmorie @bparees @ncdc @smarterclayton @pweil- PTAL- redid most of it in terms of a run-once pod design. |
How could I forget @danmcp |
3. Describe a deployment hook API | ||
|
||
|
||
## Comparison of potential approaches |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would make a note here saying that fundamentally, these are the choices you have to evaluate the use-cases in terms of, since this ties into the next section.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I revised this section, ptal.
The "notification" use case mentioned here should be separated from the "hook" use case as I describe in #1228. Hooks run during deployments (pre, mid, post) and are dealing with the execution environment of the deployment itself. Notifications happen after deployments transition into a terminal phase. |
Do you agree that my call-a-webhook use case here is a "hook" rather than a "notification" if the outcome of the execution is allowed to impact the outcome of the deployment? Keep in mind that in the design section I point out that we don't currently have any proposal for how to handle post-deployment hooks failures other than manual intervention- so without a solution to that failure mode, I don't know what impact ANY post hook could have on the deployment other than reporting warnings. |
Yes. Hooks are imperative "do this as part of the innate lifecycle of this object". Notifications are post-hoc, triggers are pre-hoc. ----- Original Message -----
|
@smarterclayton that would imply that a post-deployment-hook failure must rollback (or otherwise impact) the deployment, otherwise it's not an innate part of the lifecycle any more so than a notification would be. |
If a post deployment hook fails most users will want to know. It may not rollback, but it should certainly tell a user. The thing about most hooks is that they are not implicitly recoverable - migrations are one way, updates in a remote service are one way. I think hooks rolling back is somewhat wishful thinking on our part. ----- Original Message -----
|
Added a simple API proposal. Could it be that simple? |
cc @abhgupta |
@smarterclayton This needs more review. |
Something we might still need are new deployment statuses distinct from "Running" to represent hook executions. Otherwise implementation is going to be really difficult. Hopefully those new statuses would have value outside the context of an implementation detail so I can justify them. |
If hooks are run once you should create a status / equiv field for "I ran this". Derek can explain what he's doing for namespaces and help guide that. Status should have limited states.
|
Deployment hooks implemented as run-once pods external to the Kubelet provide a different set of guarantees: | ||
|
||
1. Hooks can be bound to the logical deployment lifecycle, enabling hook executions decoupled from replication mechanics. | ||
1. Races can be avoided by defining a hook that executes exactly once per deployment regardless of the replica count. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At most once
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
i'd want at least once. or at least an option for at least once. if you bring up my app w/o having successfully run my db migration i'm not going to be happy. if you run my db migration twice, that might be ok (if i wrote my db migration intelligently)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's also cloudier.
On Mar 12, 2015, at 3:21 PM, Ben Parees notifications@github.com wrote:
In docs/proposals/post-deployment-hooks.md:
+2. The status of the pod is linked to the status and outcome of the hook execution.
- The pod will not enter a ready status until the hook has completed successfully.
- Service endpoints will not be created for the pod until the pod has a ready status.
- If the hook fails, the pod's creation is considered a failure, and the retry behavior is restart-policy driven in the usual way.
+Because deployments are represented as replication controllers, lifecycle hooks defined for containers are executed for every container in the replica set for the deployment. This behavior has complexity implications when applied to deployment use cases:
+
+1. The hooks for all pods in the deployment will race, placing a burden on hook authors (e.g., the hooks would generally need to be tolerant of concurrent hook execution and implement manual coordination.)
+
+
+##### Deployment hooks
+
+Deployment hooks implemented as run-once pods external to the Kubelet provide a different set of guarantees:
+
+1. Hooks can be bound to the logical deployment lifecycle, enabling hook executions decoupled from replication mechanics.
- Races can be avoided by defining a hook that executes exactly once per deployment regardless of the replica count.
i'd want at least once. or at least an option for at least once. if you bring up my app w/o having successfully run my db migration i'm not going to be happy. if you run my db migration twice, that might be ok (if i wrote my db migration intelligently)—
Reply to this email directly or view it on GitHub.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agree w/ "at least," changed.
@smarterclayton Digested your comments and studies the upstream container lifecycle API more closey. I added some commits which rework the hook API to be much closer to the upstream lifecycle/handler setup. PTAL. |
@smarterclayton are saying that this hook would win in a fight against
|
ad4ff5e
to
95efb4c
Compare
|
||
##### Deployment hooks | ||
|
||
Deployment hooks implemented as run-once pods external to the Kubelet provide a different set of guarantees: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You refer to deployment hooks here without definining them. Add a couple sentences saying what this is:
An alternative to the upstream-provided lifecycle hooks is to have a notion of a hook which is a property of an OpenShift deployment.
^^ something like that
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good idea; revised.
Proposal looks good minus that one issue - fix that (add something to the example), and then squash and I'll merge. Do you have a prototype yet? |
e7bfd5c
to
5320be5
Compare
Added |
"command": ["rake", "db:migrate"], | ||
"env": [ | ||
{ | ||
"key": "CUSTOM_VAR", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Key is not required, better to remove from the example
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops, fixed.
5320be5
to
77b52ef
Compare
give your commit a title |
77b52ef
to
8f765fb
Compare
Hrm, rebase fail. Fixed. |
LGTM [merge] |
continuous-integration/openshift-jenkins/merge SUCCESS (https://ci.openshift.redhat.com/jenkins/job/merge_pull_requests_openshift3/1267/) (Image: devenv-fedora_1112) |
Evaluated for origin up to 8f765fb |
Merged by openshift-bot
…service-catalog/' changes from ef63307bdb..ae6b643caf ae6b643caf Use oc adm instead of oadm which might not exist in various installations. 66a4eb2a2c Update instructions... will remove once documented elsewhere 1b704d1530 replace build context setup with init containers ee4df18c7f hack/lib: dedup os::util::host_platform and os::build::host_platform 1cd6dfa998 origin: Switch out owners to Red Hatters 664f4d318f Add instructions for syncing repos 2f2cdd546b origin-build: delete files with colon in them cdf8b12848 origin-build: don't build user-broker ebfede9056 origin build: add _output to .gitignore 55412c7e3d origin build: make build-go and build-cross work 68c74ff4ae origin build: modify hard coded path 3d41a217f6 origin build: add origin tooling a8fc27d Fix typos in walkthrough (openshift#1224) e77edbf openshift#1157: Limit the amount of time for reconciliations (openshift#1196) 1b1a749 temporarily disabled verify-links.sh from the verify target (openshift#1226) acf8fab Send originating identity headers in OSB requests (openshift#1162) 821ba16 new admission controller to block updates to service instance updates that (openshift#1210) d69c5e5 Minor improvement to godoc in binaries (openshift#1211) 5b81814 fix typos (openshift#1221) 836dc4a Adds how to download Helm chart (charts/catalog) (openshift#1219) 2fd0115 Fix "visit the project on github" link. (openshift#1217) 325e4b6 Add how to set $GOPATH. (openshift#1218) 68b775f Update the installation (openshift#1199) 6e3a3c1 v0.0.19 (openshift#1207) 8b69791 Removing errexit from TLS setup script (openshift#1206) 273260f Instance deletion lifecycle enhancements, issue openshift#820 (openshift#1159) c050713 fix cleaning of build output for non-root users (openshift#1205) 5995df1 Merge branch 'pr/1204' 72f4802 Remove osb prefix from example ServiceClass (openshift#1201) f9dbd4e pin all dependencies in glide to current version except for glog where we want to pick up the prior version to fix issue 1187. f148bc5 v0.0.18 (openshift#1202) b86ab8d Removing the helm install command (openshift#1185) 3cff482 Remove Alpha* prefix on all API fields for issue openshift#1180 (openshift#1184) 154b74d Fix gofmt issue (openshift#1192) 2ee894a do the clean before building an arch (openshift#1179) b4976ef Fix bad URL (openshift#1189) cd3dede Fix hrefs again (openshift#1190) f066226 Design: Instance/Binding parameters (openshift#1075) eb37682 This generated file is missing from master (openshift#1191) 28c0ae7 Use generation instead of checksum for Instances and InstanceCredentials (openshift#1151) 5cdd323 Fix bad href (openshift#1188) 8a892f0 handle lingering polling cases (openshift#1174) f5fabd6 remove TPRs from Jenkins e2e pipeline (openshift#1175) 717df78 Add godoc explaining that Instance and InstanceCredential specs are immutable (openshift#1182) REVERT: ef63307bdb origin build: add origin tooling git-subtree-dir: cmd/service-catalog/go/src/github.com/kubernetes-incubator/service-catalog git-subtree-split: ae6b643cafd3a17412f173e70ed7c1a2e39ee549
* Updates walkthrough-1.6.md Updates incorrect contents in walkthrough-1.6.md. * Updates install-1.6.md Updates the clean up instruction in install-1.6.md. * Updates walkthrough-1.7.md Applies the change in walkthrough-1.6.md to walkthrough-1.7.md. * Troubleshooting is out of clean up section Move troubleshooting section out of cleanup-section. * Moves troubleshooting out of cleanup section Troubleshooting section is not part of cleanup section.
This PR is a WIP to discuss the design of a deployment hook mechanism for https://trello.com/c/Zea4kjvo.