Add Daemon Set resource #133

karanthukral · 2017-07-18T17:20:17Z

What?

Adds daemon set as a kubernetes resource
Replicates logic built into the Replica Set resource here
Adds a test for successful and timed out deploy for a daemon set
The timeout currently is the same as a replica set

cc/ @Shopify/cloudplatform

karanthukral · 2017-07-19T13:33:29Z

I have another branch ready for stateful sets but I'll wait for review on this before opening that one since they might have similar comments

KnVerey · 2017-07-19T18:44:30Z

lib/kubernetes-deploy/kubernetes_resource/daemon_set.rb

+
+      if @found
+        daemonset_data = JSON.parse(raw_json)
+        @desired_number = daemonset_data["status"]["desiredNumberScheduled"]


Nit: since this is coming from the status data for this type, no need to store it separately--use the copy in the @rollout_data slice.

KnVerey · 2017-07-19T18:45:08Z

lib/kubernetes-deploy/kubernetes_resource/daemon_set.rb

+    end
+
+    def deploy_succeeded?
+      @desired_number == @rollout_data["desiredNumberScheduled"].to_i &&


This is comparing @rollout_data["desiredNumberScheduled"] to itself, no?

🤦‍♂️

KnVerey · 2017-07-19T18:48:33Z

lib/kubernetes-deploy/kubernetes_resource/daemon_set.rb

+
+    def deploy_succeeded?
+      @desired_number == @rollout_data["desiredNumberScheduled"].to_i &&
+      @desired_number == @rollout_data["numberReady"].to_i


Looking at what's available to us, I think what we want to verify is desiredNumberScheduled == updatedNumberScheduled == numberAvailable. In other words, all required have been updated and are available (which means ready for at least minReadySeconds).

KnVerey · 2017-07-19T19:07:30Z

lib/kubernetes-deploy/kubernetes_resource/daemon_set.rb

+      container_names.each_with_object({}) do |container_name, container_logs|
+        out, _err, _st = kubectl.run(
+          "logs",
+          id,


This command isn't implemented for daemonsets unfortunately.

༶ kubectl logs ds/dd-agent error: cannot get the logs from extensions/__internal, Kind=DaemonSet

Ref kubernetes/kubernetes#40927, which implemented it for deployments, jobs and statefulsets. We'll need to pick a pod ourselves and use its logs. The most_useful_pod logic from fetch_events is probably good enough for now, though it isn't as smart as what kubectl does. Note that there are tradeoffs to prioritizing failing pods, which I discuss in #138.

KnVerey · 2017-07-19T19:11:16Z

lib/kubernetes-deploy/kubernetes_resource/daemon_set.rb

+
+    private
+
+    def unmanaged?


This concept doesn't apply to DaemonSets--they aren't generated by other resources.

KnVerey · 2017-07-19T19:15:56Z

test/fixtures/hello-cloud/daemon_set.yml

+  template:
+    metadata:
+      labels:
+        app: busybox


Nit: the "app" label for this fixture set is usually set to "hello-cloud"

I fixed that but still need to push it :P

KnVerey · 2017-07-19T19:17:19Z

test/helpers/fixture_set.rb

+      daemon_sets.each do |ds|
+        found = true if ds.metadata.name == name
+      end
+      assert found


We should also have an availability assertion like we do for deployments and replicasets.

I'd also add the assertion message

KnVerey · 2017-07-19T19:19:03Z

test/integration/kubernetes_deploy_test.rb

+
+    assert_logs_match_all([
+      'Successfully deployed 1 resource',
+      '1 currentNumberSchedule, 1 desiredNumberSchedule, 1 numberRead'


Since these tests are expensive, I'd recommend adding this assertion to the group at L17 (including a resource type prefix like the others there have) instead of having a separate test.

KnVerey · 2017-07-19T19:22:41Z

test/integration/kubernetes_deploy_test.rb

+    ])
+  end
+
+  def test_timed_out_daemon_set_deploy


Since the timeout has not special logic for this resource type, I'd rather swap this out for a test of a failure scenario. There are a few deployment tests you can use as a starting point (no need to duplicate all of them, since they're ultimately using the same pod logic). One thing it is important to show is that we're successfully fetching events from both the DS and a pod, as well as logs.

KnVerey · 2017-07-19T19:27:45Z

lib/kubernetes-deploy/kubernetes_resource/daemon_set.rb

+
+      all_pods = JSON.parse(raw_json)["items"]
+      all_pods.each_with_object([]) do |pod_data, relevant_pods|
+        next unless pod_data["metadata"]["ownerReferences"].any? { |ref| ref["uid"] == ds_data["metadata"]["uid"] }


Seemingly this will select all pods belonging to the DaemonSet, both old and updated. Is there a way to select only the ones in the updated generation? It's the intermediate ReplicaSet layer that provides this guarantee for Deployments. If it isn't possible to filter down to the relevant pods, then we'll need to change our logic for success/failure.

Thought I added the revision check too. I'll add that

karanthukral · 2017-07-24T15:03:29Z

Made all the requested changes. Should be good for another set of 👀 . Will rebase the commits before merging

KnVerey · 2017-07-24T21:43:28Z

lib/kubernetes-deploy/kubernetes_resource/daemon_set.rb

+
+    def deploy_succeeded?
+      @rollout_data["desiredNumberScheduled"] == @rollout_data["currentNumberScheduled"].to_i &&
+      @rollout_data["desiredNumberScheduled"] == @rollout_data["numberAvailable"].to_i


I believe this could generate a false positive if we look right before the rollout of the new pods begins. I.e. at that point desired == current == available, and all of the pods in question are from the old generation. The status includes a updatedNumberScheduled number, doesn't it? Do you think we should also look at @pods at all?

I'll look into the updatedNumberScheduled. I skipped checking the pods since numberAvailable should handle the case when a pod deploy fails. If you see a reason for it to be there, I can look into adding it.

I skipped checking the pods since numberAvailable should handle the case when a pod deploy fails. If you see a reason for it to be there, I can look into adding it.

Theoretically you should be right. However, in debugging a test failure that happened about 1% of the time locally and more often on CI, I discovered that the following is possible for deployments:

Deployment template is up to date

New replica set exists and is scaled to zero (so its desired/current/available are equal)

Deployment's status still describes the old replica set, which is fully scaled still (so the overall desired/current/available and even updated are also equal)

That's why Deployment's success condition looks at the latest replicaSet too, not just the status fields. I don't know that DS have the same issue, but it seems plausible. An in-between option would be to include @pods.length == desiredNumberScheduled. WDYT?

One more note on this: don't forget to convert all of them to integers (you've got a mix right now)

Looking at the code you linked in your other comment, we should be safe without looking at pods as long as we also check status.observedGeneration == metadata.Generation. Looks like that strategy would be available for Deployment too actually.

KnVerey · 2017-07-24T21:45:28Z

lib/kubernetes-deploy/kubernetes_resource/daemon_set.rb

+    private
+
+    def container_names
+      @definition["spec"]["template"]["spec"]["containers"].map { |c| c["name"] }


Can DaemonSets have init containers too?

It can. Sorry forgot to add it back while jumping between too many branches last week

KnVerey · 2017-07-24T21:50:48Z

lib/kubernetes-deploy/kubernetes_resource/daemon_set.rb

+
+      latest_pods = all_pods.find_all do |pods|
+        pods["metadata"]["ownerReferences"].any? { |ref| ref["uid"] == ds_data["metadata"]["uid"] } &&
+        pods["metadata"]["labels"]["pod-template-generation"] == current_generation


This sounds like the right thing to look at, but for my education (since this is different from Deployment) do you have a doc or PR link that describes Generation?

I mostly figured this out by testing it and seeing what we had access too. 👀 at the k8s repo I did find isPodUpdated and rollout status ref. Hopefully that helps. If not I can investigate more tomorrow

I started from those links and poked around a bit. tl;dr I think what you have here is correct for 1.6, but it'll change to controller-revision-hash in 1.7 (deprecation)/1.8(final). Some of the stuff I just read:

Daemonset Update Proposal with some relevant details about the field changes (looks like the old one will be kept and the new one added when a DS rolls in 1.7)

pod-template-generation is deprecated

Issue mentioning the DS/deploy/SS labels for 1.7 (it was apparently going to be daemonset-controller-hash for DS but was changed before release for consistency)

Since 1.6 we don't have access to the revision, I'll make an issue to update this when we look at migrating to 1.7

Works for me--that'll make our lives easier when the time comes. The test coverage should also prevent us from doing an upgrade without fixing it if it comes down to it.

KnVerey · 2017-07-24T21:51:48Z

test/integration/kubernetes_deploy_test.rb

+    assert_logs_match_all([
+      "DaemonSet/nginx: FAILED",
+      "The following containers are in a state that is unlikely to be recoverable:",
+      "Logs from container 'nginx' (last 250 lines shown):",


Add an assertion on an event too please

KnVerey · 2017-07-24T21:53:23Z

test/integration/kubernetes_deploy_test.rb

    ], in_order: true)

    assert_logs_match_all([
      %r{ReplicaSet/bare-replica-set\s+1 replica, 1 availableReplica, 1 readyReplica},
      %r{Deployment/web\s+1 replica, 1 updatedReplica, 1 availableReplica},
-      %r{Service/web\s+Selects at least 1 pod}
+      %r{Service/web\s+Selects at least 1 pod},
+      %r{DaemonSet/nginx\s+1 currentNumberSchedule, 1 desiredNumberSchedule, 1 numberRead, 1 numberAvailabl}


Why are "numberRead" and "numberAvailabl" truncated?

Fixing now :)

KnVerey · 2017-07-25T03:17:11Z

test/integration/kubernetes_deploy_test.rb

@@ -51,7 +52,7 @@ def test_pruning_works
      'deployment "web"',
      'ingress "web"'
    ] # not necessarily listed in this order
-    expected_msgs = [/Pruned 5 resources and successfully deployed 3 resources/]
+    expected_msgs = [/Pruned 6 resources and successfully deployed 3 resources/]


let's also add a message to the expected_prune list above

KnVerey · 2017-07-25T15:41:53Z

lib/kubernetes-deploy/kubernetes_resource/daemon_set.rb

+
+      if @found
+        daemonset_data = JSON.parse(raw_json)
+        @metadata = daemonset_data["metadata"]


Nit: Since all we want out of this is the generation, I'd suggest storing @current_generation instead of the whole big blob. Similarly, observedGeneration is a kinda different from the rest of the "rollout data", so I'd prefer to store that on its own as well. Either way, we need to make the else reset these new values.

KnVerey · 2017-07-25T15:42:53Z

lib/kubernetes-deploy/kubernetes_resource/daemon_set.rb

+        @metadata = daemonset_data["metadata"]
+        @rollout_data = daemonset_data["status"]
+          .slice("currentNumberScheduled", "desiredNumberScheduled", "numberReady", "numberAvailable", "observedGeneration")
+        @status = @rollout_data.map { |state_replicas, num| "#{num} #{state_replicas}" }.join(", ")


Even if you keep observedGeneration as part of the rollout data, I don't think it is useful to include it in the status, which is end-user-facing.

KnVerey · 2017-07-25T15:45:56Z

lib/kubernetes-deploy/kubernetes_resource/daemon_set.rb

+
+    def fetch_logs
+      most_useful_pod = @pods.find(&:deploy_failed?) || @pods.find(&:deploy_timed_out?) || @pods.first
+      container_names.each_with_object({}) do |container_name, container_logs|


Is there a reason we can't do most_useful_pod.fetch_logs for this?

KnVerey · 2017-07-25T15:48:32Z

lib/kubernetes-deploy/kubernetes_resource/daemon_set.rb

+
+    def container_names
+      regular_containers = @definition["spec"]["template"]["spec"]["containers"].map { |c| c["name"] }
+      init_containers = @definition["spec"]["template"]["spec"].fetch("initContainers", {}).map { |c| c["name"] }


Nit: .fetch("initContainers", {}) should technically be .fetch("initContainers", []) since it'd be an array if it were present (the behaviour is the same though). I just fixed this for Pod in another PR. (then again, we don't need this method if we can use Pod's version directly)

KnVerey · 2017-07-25T15:52:20Z

lib/kubernetes-deploy/kubernetes_resource/daemon_set.rb

+      return [] unless st.success?
+
+      all_pods = JSON.parse(raw_json)["items"]
+      current_generation = ds_data["metadata"]["generation"].to_s


Why to_s--isn't this already a string, which really contains an integer?

KnVerey · 2017-07-25T15:54:56Z

lib/kubernetes-deploy/kubernetes_resource/daemon_set.rb

+
+      latest_pods = all_pods.find_all do |pods|
+        pods["metadata"]["ownerReferences"].any? { |ref| ref["uid"] == ds_data["metadata"]["uid"] } &&
+        pods["metadata"]["labels"]["pod-template-generation"] == current_generation


Works for me--that'll make our lives easier when the time comes. The test coverage should also prevent us from doing an upgrade without fixing it if it comes down to it.

KnVerey · 2017-07-25T16:06:37Z

test/integration/kubernetes_deploy_test.rb

+    assert_logs_match_all([
+      "DaemonSet/nginx: FAILED",
+      "The following containers are in a state that is unlikely to be recoverable:",
+      "Events: None found. Please check your usual logging service (e.g. Splunk).",


Hm, really? I'd expect there to be some events here. Indeed, I see two when I run this test locally:

[Pod/nginx-cnkzq] BackOff: Back-off restarting failed container (2 events) [Pod/nginx-cnkzq] FailedSync: Error syncing pod, skipping: failed to "StartContainer" for "nginx" with CrashLoopBackOff: "Back-off 10s restarting failed container=nginx pod=nginx-cnkzq_k8sdeploy-test-bad-container-on-daemon-sets-fa-be1043b5b97dfe2b(969dd56c-7152-11e7-be12-76b0ec54df39)" (2 events)

My local tests for some reason is not printing any logs or events so I wanted to see if buildkite would do it

KnVerey · 2017-07-25T16:07:27Z

test/unit/kubernetes-deploy/google_friendly_config_test.rb

@@ -14,7 +14,7 @@ def teardown
  def test_auth_use_default_gcp_success
    config = KubernetesDeploy::KubeclientBuilder::GoogleFriendlyConfig.new(kubeconfig, "")

-    stub_request(:post, 'https://www.googleapis.com/oauth2/v3/token')
+    stub_request(:post, 'https://www.googleapis.com/oauth2/v4/token')


This is on master if you rebase FYI

kirs · 2017-07-25T20:56:22Z

lib/kubernetes-deploy/kubernetes_resource/daemon_set.rb

+    end
+
+    def deploy_failed?
+      @pods.present? && @pods.all?(&:deploy_failed?)


Why .all? and not any?? What if daemon set has one crashing and one starting pod?

Good point. I'll fix that

I think all? is correct actually. Since these are clones, we don't want to fail the deploy if a large deployment has a single bad pod, which will automatically be rescheduled. We're trying to catch the case where something is systemically wrong. ReplicaSets also use all? for this reason.

I feel like all? makes sense for replica_sets since they are redundant whereas a daemon set is meant to make sure there is a pod on each node. I don't have strong feelings either way though

Very true. And when a single pod is bad, it is often because of a problem with the node, so in the case of a DS pod it is pretty doomed. I'm especially nervous that we'll wrongly fail deploys due to transient registry errors that unfortunately can surface as 404s, but I definitely see the logic. We can always try any? if we feel it's the strictly correct solution as long as we keep an eye on our DaemonSet success metrics and switch to all? if it is an actual problem.

Sounds good 👍

karanthukral · 2017-07-25T21:33:33Z

I'll merge this first thing tomorrow since I don't want to ship it EOD

karanthukral force-pushed the daemonsets branch from 2fa0d5b to 8d944d1 Compare July 18, 2017 17:50

karanthukral changed the title ~~[WIP] Add Daemon Set resource~~ Add Daemon Set resource Jul 18, 2017

karanthukral requested a review from KnVerey July 18, 2017 18:18

karanthukral force-pushed the daemonsets branch 2 times, most recently from 01a4e2d to 8bd237d Compare July 19, 2017 14:25

KnVerey requested a review from kirs July 19, 2017 18:40

KnVerey suggested changes Jul 19, 2017

View reviewed changes

karanthukral force-pushed the daemonsets branch from 6e0ef23 to baaebc2 Compare July 24, 2017 14:12

karanthukral force-pushed the daemonsets branch from 249367c to 4195c6c Compare July 24, 2017 15:04

KnVerey reviewed Jul 24, 2017

View reviewed changes

KnVerey reviewed Jul 25, 2017

View reviewed changes

karanthukral force-pushed the daemonsets branch 3 times, most recently from 8e44ad3 to 8eda84c Compare July 25, 2017 17:51

KnVerey approved these changes Jul 25, 2017

View reviewed changes

karanthukral force-pushed the daemonsets branch from 8eda84c to dbc7981 Compare July 25, 2017 18:21

kirs approved these changes Jul 25, 2017

View reviewed changes

Add daemon set resource

8301ed3

karanthukral force-pushed the daemonsets branch from dbc7981 to 8301ed3 Compare July 25, 2017 21:32

karanthukral merged commit 5bea54a into master Jul 26, 2017

karanthukral deleted the daemonsets branch July 26, 2017 12:55

karanthukral deployed to rubygems July 26, 2017 13:09 Active

karanthukral restored the daemonsets branch July 26, 2017 18:21

karanthukral mentioned this pull request Jul 26, 2017

Revert "Add Daemon Set resource" #147

Merged

Add Daemon Set resource #133

Add Daemon Set resource #133

Conversation

karanthukral commented Jul 18, 2017 • edited Loading

What?

karanthukral commented Jul 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karanthukral commented Jul 24, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

KnVerey Jul 25, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karanthukral commented Jul 25, 2017

karanthukral commented Jul 18, 2017 •

edited

Loading

KnVerey Jul 25, 2017 •

edited

Loading