Support addon Deployments, make heapster a deployment with a nanny. #22893

Q-Lee · 2016-03-12T01:45:58Z

Heapster uses a deployment here.
Changed kube-addon-update.sh to include deployments (this seems like a transitional change while we're moving all addons to deployments).
Adding a nanny to resize heapster with the number of nodes.

k8s-github-robot · 2016-03-12T01:55:52Z

Labelling this PR as size/L

k8s-bot · 2016-03-12T02:20:10Z

GCE e2e build/test failed for commit 5a3c51892cf210760ba4cbbc40d821e5e7c97b82.

Please reference the list of currently known flakes when examining this failure. If you request a re-test, you must reference the issue describing the flake.

davidopp · 2016-03-12T06:25:03Z

Is this for 1.2?

cc/ @bgrant0607 @mikedanese

bgrant0607 · 2016-03-12T06:31:22Z

No, this can't go into 1.2. That ship has sailed.

bgrant0607 · 2016-03-12T06:33:11Z

Ref #7459, #9467

Q-Lee · 2016-03-14T20:10:51Z

We're aiming for 1.2.1. Adding zml@ for context on kube-addon-update.sh.
cc @zmerlynn

mikedanese · 2016-03-15T06:34:46Z

cluster/saltbase/salt/kube-addons/kube-addon-update.sh

@@ -475,6 +475,7 @@ function update-addons() {
    local -r addon_path=$1
    # be careful, reconcile-objects uses global variables
    reconcile-objects ${addon_path} ReplicationController "-" &
+    reconcile-objects ${addon_path} Deployment "-" &


Comment bellow this:

We don't expect names to be versioned for the following kinds

I would expect this to be true for deployments as well.

Q-Lee · 2016-03-16T23:37:48Z

@bgrant0607 Is there anything in particular I need to consider for cluster upgrades if heapster is becoming a deployment?

k8s-bot · 2016-03-17T00:31:53Z

GCE e2e build/test failed for commit 82682f5d993c99f33e4d104675fbbb9cb864e1a2.

Please reference the list of currently known flakes when examining this failure. If you request a re-test, you must reference the issue describing the flake.

bgrant0607 · 2016-03-17T02:34:18Z

What's going to kill off the old RCs?

cc @fgrzadkowski @vishh

bgrant0607 · 2016-03-17T02:34:55Z

cluster/addons/cluster-monitoring/googleinfluxdb/heapster-controller-combined.yaml

+        - image: gcr.io/google_containers/addon-resizer:1.0
+          name: heapster-nanny
+          resources:
+            limits:


If you just specify limits, request should be set from limits.

#18216 added requests to all addons.

One of the config best practices is:

Don’t specify default values unnecessarily, in order to simplify and minimize configs, and to reduce error.

I don't really understand the argument behind #18216 but it seems to violate that best practice.

Then why don't we do this for other singletons (e.g., heapster itself)?

Because of #18216. Can you explain why this is important @gmarek? Here's the defaulting code https://github.com/kubernetes/kubernetes/blob/master/pkg/api/v1/defaults.go#L99-L116

When we were designing QoS we decided to have this defaulting as a backward compatibility feature. I didn't had impression that it's there to stay forever, so I think we should have requests explicit. Generally all those QoS stuff is really confusing as it is today. @piosz

Interesting: I didn't know about that default behavior. It actually conflicts with the addon-resizer, which explicitly sets requests=limits to keep its dependents in the guaranteed class. The addon-resizer also performs an update on any qualitative difference (e.g., requests being expected but not found). This would manifest as a single deployment update at startup.

It feels weird to add a flag to disregard requests, but we'll need to if we want to utilize this default behavior. But since literally all of our addons specify both requests and limits, why don't I cut an issue for 1.3 to rectify both the pod_nanny and all of our addon yamls with best practices?

This is another proof that the current state is confusing. We should work on that at some point.

Opened #23229

bgrant0607 · 2016-03-17T02:36:13Z

cc @janetkuo @pwittrock

bgrant0607 · 2016-03-17T02:36:38Z

cluster/addons/cluster-monitoring/googleinfluxdb/heapster-controller-combined.yaml

@@ -17,7 +17,8 @@ metadata:
 spec:
  replicas: 1


Since this only has 1 replica, you might consider the Recreate update policy. The default is RollingUpdate.

Interesting idea. The code for the container in question lives in contrib/addon-resizer, where that would be an easy (1-line) change. I'll definitely consider it.

fgrzadkowski · 2016-03-17T17:59:41Z

@Q-Lee Please cc people who owns a particular feature/component if you change its configuration etc. @mwielgus @piosz

fgrzadkowski · 2016-03-17T18:01:42Z

cluster/addons/cluster-monitoring/google/heapster-controller.yaml

+                  fieldPath: metadata.namespace
+          command:
+            - /pod_nanny
+            - --cpu=100m


@mwielgus Please verify these values based on your perf tests. AFAIR we need more than 100m cpu in large clusters and 4MB of memory for each node.

The 1.2 cut gives 4MB per node, but leaves CPU at 100m. I've changed this to use the same per-node value as static sizing.

On 1000 node cluster we are using 50 millis on average with occasional spikes to 500 (every minute for a couple seconds).

Where is this define?

Q-Lee · 2016-03-17T20:07:29Z

@fgrzadkowski how do I know which person/people own a particular configuration file?

Q-Lee · 2016-03-17T20:36:23Z

"What's going to kill off the old RCs?"

Ideally I would think the addon-update system should delete the old replication controller when its configuration is no longer found. Thoughts?

@bgrant0607 @fgrzadkowski @vishh

k8s-bot · 2016-03-17T21:08:55Z

GCE e2e build/test failed for commit 1c794555072a5ddd5ef3a26fc5970e6f04ec1c26.

Please reference the list of currently known flakes when examining this failure. If you request a re-test, you must reference the issue describing the flake.

vishh · 2016-03-17T21:50:01Z

@Q-Lee:
+1 for #22893 (comment)

k8s-bot · 2016-03-22T20:53:20Z

GCE e2e build/test passed for commit 726bbc4bfce110601fc17bf568386e413b4bd3de.

k8s-github-robot · 2016-03-22T20:54:34Z

PR changed after LGTM, removing LGTM.

k8s-bot · 2016-03-22T21:22:42Z

GCE e2e build/test passed for commit 81ba98a.

Support addon Deployments, make heapster a deployment with a nanny.

fgrzadkowski · 2016-03-23T14:45:41Z

cluster/saltbase/salt/kube-addons/kube-addon-update.sh

@@ -481,6 +481,7 @@ function update-addons() {
    # That's why we pass an empty string as the version separator.
    # If the description differs on disk, the object should be recreated.
    # This is not implemented in this version.
+    reconcile-objects ${addon_path} Deployment "" &


How is this going to update Deployments? AFAIU name for an object will not change, so how addon-updater will realise it has to do an update? Has this been tested?

ping @Q-Lee
Are you sure this won't broke addons update procedure?

This reverts commit 1f8773e, reversing changes made to 9dfbcae.

k8s-cherrypick-bot · 2016-04-06T20:45:52Z

Removing label cherrypick-candidate because no release milestone was set. This is an invalid state and thus this PR is not being considered for cherry-pick to any release branch. Please add an appropriate release milestone and then re-add the label.

#23512-upstream-release-1.2 Automated cherry pick of #22893 #23512

k8s-cherrypick-bot · 2016-04-06T23:42:51Z

Commit found in the "release-1.2" branch appears to be this PR. Removing the "cherrypick-candidate" label. If this is an error find help to get your PR picked.

andyzheng0831 · 2016-04-07T17:31:13Z

@Q-Lee as I said in #23434 (comment), "Please notify me when you decide to cherry pick." Your change broke us in master branch and I fixed that. Now it was cherry picked in release-1.2 without telling us, so it breaks us again. Thanks to our Jenkins for catching the breakage. Hopefully I will still have time to cherry pick my fix into release-1.2 before 1.2.2 is cut.

…ck-of-#22893-kubernetes#23512-upstream-release-1.2 Automated cherry pick of kubernetes#22893 kubernetes#23512

googlebot added the cla: yes label Mar 12, 2016

k8s-github-robot assigned davidopp Mar 12, 2016

k8s-github-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Mar 12, 2016

mikedanese assigned mikedanese and unassigned davidopp Mar 12, 2016

mikedanese reviewed Mar 15, 2016
View reviewed changes

Q-Lee force-pushed the nanny-heapster branch from 5a3c518 to 82682f5 Compare March 16, 2016 23:35

bgrant0607 reviewed Mar 17, 2016
View reviewed changes

fgrzadkowski reviewed Mar 17, 2016
View reviewed changes

Adding e2e test support for monitoring deployments.

81ba98a

Q-Lee force-pushed the nanny-heapster branch from 726bbc4 to 81ba98a Compare March 22, 2016 20:47

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 22, 2016

Q-Lee added a commit that referenced this pull request Mar 22, 2016

Merge pull request #22893 from Q-Lee/nanny-heapster

1f8773e

Support addon Deployments, make heapster a deployment with a nanny.

Q-Lee merged commit 1f8773e into kubernetes:master Mar 22, 2016

fgrzadkowski reviewed Mar 23, 2016
View reviewed changes

andyzheng0831 mentioned this pull request Mar 24, 2016

Trusty: Update heapster manifest handling code #23434

Merged

piosz mentioned this pull request Mar 25, 2016

Bumped Heapster to v1.1.0-beta1 #23470

Merged

fgrzadkowski mentioned this pull request Mar 25, 2016

kube-addon-updater.sh not working properly for Deployments #23471

Closed

mikedanese added a commit to mikedanese/kubernetes that referenced this pull request Mar 25, 2016

Revert "Merge pull request kubernetes#22893 from Q-Lee/nanny-heapster"

b4739b5

This reverts commit 1f8773e, reversing changes made to 9dfbcae.

mikedanese mentioned this pull request Mar 25, 2016

Revert "Merge pull request #22893 from Q-Lee/nanny-heapster" #23499

Closed

janetkuo mentioned this pull request Apr 6, 2016

Create a new Deployment in kube-system for every version. #23512

Merged

Q-Lee added the cherrypick-candidate label Apr 6, 2016

k8s-cherrypick-bot removed the cherrypick-candidate label Apr 6, 2016

zmerlynn added this to the v1.2 milestone Apr 6, 2016

zmerlynn added the cherrypick-candidate label Apr 6, 2016

This was referenced Apr 6, 2016

Automated cherry pick of #22893 #23512 #23941

Merged

Temp downgrade heapster from v1.1.0.beta1 to v1.0.2 for cherrypick #23946

Closed

zmerlynn added cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Apr 6, 2016

zmerlynn added a commit that referenced this pull request Apr 6, 2016

Merge pull request #23941 from janetkuo/automated-cherry-pick-of-#22893-

9b4f3db

#23512-upstream-release-1.2 Automated cherry pick of #22893 #23512

k8s-cherrypick-bot removed the cherrypick-candidate label Apr 6, 2016

shyamjvs pushed a commit to shyamjvs/kubernetes that referenced this pull request Dec 1, 2016

Merge pull request kubernetes#23941 from janetkuo/automated-cherry-pi…

86525b9

…ck-of-#22893-kubernetes#23512-upstream-release-1.2 Automated cherry pick of kubernetes#22893 kubernetes#23512

shouhong pushed a commit to shouhong/kubernetes that referenced this pull request Feb 14, 2017

Merge pull request kubernetes#23941 from janetkuo/automated-cherry-pi…

ba7d271

…ck-of-#22893-kubernetes#23512-upstream-release-1.2 Automated cherry pick of kubernetes#22893 kubernetes#23512

Support addon Deployments, make heapster a deployment with a nanny. #22893

Support addon Deployments, make heapster a deployment with a nanny. #22893

Conversation

Q-Lee commented Mar 12, 2016

k8s-github-robot commented Mar 12, 2016

k8s-bot commented Mar 12, 2016

davidopp commented Mar 12, 2016

bgrant0607 commented Mar 12, 2016

bgrant0607 commented Mar 12, 2016

Q-Lee commented Mar 14, 2016

Choose a reason for hiding this comment

Q-Lee commented Mar 16, 2016

k8s-bot commented Mar 17, 2016

bgrant0607 commented Mar 17, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bgrant0607 commented Mar 17, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fgrzadkowski commented Mar 17, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Q-Lee commented Mar 17, 2016

Q-Lee commented Mar 17, 2016

k8s-bot commented Mar 17, 2016

vishh commented Mar 17, 2016

k8s-bot commented Mar 22, 2016

k8s-github-robot commented Mar 22, 2016

k8s-bot commented Mar 22, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-cherrypick-bot commented Apr 6, 2016

k8s-cherrypick-bot commented Apr 6, 2016

andyzheng0831 commented Apr 7, 2016