Scheduler can recieve its policy configuration from a ConfigMap #43892

bsalamat · 2017-03-31T07:01:22Z

What this PR does / why we need it: This PR adds the ability to scheduler to receive its policy configuration from a ConfigMap. Before this, scheduler could receive its policy config only from a file. The logic to watch the ConfigMap object will be added in a subsequent PR.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

Release note:

k8s-reviewable · 2017-03-31T07:01:29Z

This change is

bsalamat · 2017-03-31T07:02:23Z

@kubernetes/sig-scheduling-pr-reviews

bsalamat · 2017-03-31T07:04:49Z

@davidopp

bsalamat · 2017-03-31T07:26:22Z

Looks like I need to manually add some conversion functions. I will add them soon.

gyliu513 · 2017-04-02T15:06:30Z

plugin/cmd/kube-scheduler/app/options/options.go

@@ -63,7 +63,9 @@ func (s *SchedulerServer) AddFlags(fs *pflag.FlagSet) {
 	fs.Int32Var(&s.Port, "port", s.Port, "The port that the scheduler's http service runs on")
 	fs.StringVar(&s.Address, "address", s.Address, "The IP address to serve on (set to 0.0.0.0 for all interfaces)")
 	fs.StringVar(&s.AlgorithmProvider, "algorithm-provider", s.AlgorithmProvider, "The scheduling algorithm provider to use, one of: "+factory.ListAlgorithmProviders())
-	fs.StringVar(&s.PolicyConfigFile, "policy-config-file", s.PolicyConfigFile, "File with scheduler policy configuration")
+	fs.StringVar(&s.PolicyConfigFile, "policy-config-file", s.PolicyConfigFile, "File with scheduler policy configuration. This file is used if policy ConfigMap object does not exists or scheduler is using legacy policy config.")
+	fs.StringVar(&s.PolicyConfigMapName, "policy-configmap", s.PolicyConfigMapName, "Name of the ConfigMap object that contains scheduler's policy configuration. It must exist in the system namespace before scheduler initialization.")


s/It must exist in the system namespace before scheduler initialization./It must exist in the system namespace before scheduler initialization if scheduler is not using legacy policy config.

gyliu513 · 2017-04-02T15:10:36Z

plugin/cmd/kube-scheduler/app/options/options.go

-	fs.StringVar(&s.PolicyConfigFile, "policy-config-file", s.PolicyConfigFile, "File with scheduler policy configuration")
+	fs.StringVar(&s.PolicyConfigFile, "policy-config-file", s.PolicyConfigFile, "File with scheduler policy configuration. This file is used if policy ConfigMap object does not exists or scheduler is using legacy policy config.")
+	fs.StringVar(&s.PolicyConfigMapName, "policy-configmap", s.PolicyConfigMapName, "Name of the ConfigMap object that contains scheduler's policy configuration. It must exist in the system namespace before scheduler initialization.")
+	fs.BoolVar(&s.UseLegacyPolicyConfig, "use-legacy-policy-config", false, "When set to true, scheduler will ignore PolicyConfigMapName and uses PolicyConfigFile")


As the PolicyConfigMapName and PolicyConfigFile are some internal used words, how about s/PolicyConfigMapName and uses PolicyConfigFile/policy config map and uses policy config file.

Good point. Done.

timothysc · 2017-04-03T15:14:01Z

pkg/apis/componentconfig/types.go

+	// uses PolicyConfigFile. If UseLegacyPolicyConfig is false and
+	// PolicyConfigMapName is not empty, the ConfigMap object with this name must
+	// exist before scheduler initialization.
+	PolicyConfigMapName string


If our component config reduces to a config map, should all other params shuffle to config map and be removed from the component config?

Also we need a means to generate a config map with all the defaults.

Also do we need to know it's a configmap? Isn't it just loading a file and the config map is an implementation detail.

@timothysc This PR is not going to make any changes to scheduler's component config. This is only about scheduler's policy config. Policy config used to be a file whose path is provided in the component config. With this PR we support the file and also a ConfigMap to configure scheduler's policy.
This PR implements part of "Approach 2" as described in:
https://docs.google.com/document/d/19AKH6V6ejOeIvyGtIPNvRMR4Yi_X8U3Q1zz2fgTNhvM

timothysc · 2017-04-03T15:15:54Z

This is super important we get this right, and @k8s-mirror-cluster-lifecycle-misc will need to be involved.

timothysc

I have some chicken-egg questions that need to be resolved 1st.

k82cn · 2017-04-04T05:50:32Z

plugin/cmd/kube-scheduler/app/configurator.go

+				return nil, fmt.Errorf("ConfigMap %v has %v entries in its 'Data'. It must have only one.", len(policyConfigMap.Data), sc.policyConfigMap)
+			}
+			policyConfigMapFound = true
+			// This loop should iterate only once, as we have already checked the length of Data.


Why not just get data directly?

The key can be any arbitrary string and we don't know it.

oh, yes. never mind :).

k82cn · 2017-04-04T07:14:45Z

test/integration/scheduler/scheduler_test.go

+	stop := make(chan struct{})
+	informerFactory.Start(stop)
+
+	defer close(stop)


k82cn · 2017-04-04T07:14:54Z

test/integration/scheduler/scheduler_test.go

+	informerFactory.Start(stop)
+
+	sched.Run()
+	defer close(stop)


k82cn · 2017-04-04T07:20:34Z

test/integration/scheduler/scheduler_test.go

+	informerFactory.Start(stop)
+
+	sched.Run()
+	defer close(stop)


prefer just after "stop".

Is there any style guide or recommendation on this? Other tests in the file use this pattern and I think consistency is more important, unless there is a strong reason to use a different pattern.

IMO, the style is from golang, prefer to call "defer" close to where the object was build, e.g. for "sync.Mutex", we'll call

l.Lock() defer l.Unlock() // other codes.

For the consistency, I'd like to correct other lines in this file :).

BTW, which line do you mean "Other tests in the file use this pattern"? Here's an example about defer.

56 func TestUnschedulableNodes(t *testing.T) { 57 _, s := framework.RunAMaster(nil) 58 defer s.Close() 59 60 ns := framework.CreateTestingNamespace("unschedulable-nodes", s, t) 61 defer framework.DeleteTestingNamespace(ns, s, t)

Oh, sorry, I didn't understand what you meant by your first comment. Done.

k82cn · 2017-04-04T07:22:09Z

test/integration/scheduler/scheduler_test.go

+	sched.Run()
+	defer close(stop)
+
+	DoTestUnschedulableNodes(t, clientSet, ns, informerFactory.Core().V1().Nodes().Lister())


This verification can not make sure the config from ConfigMap was took place; default config can also pass this test, right?

Similar to the following test, whether return err can not make sure the config was updated as expected; I'd like suggest to find a way to check the data/config with the scheduler.

Scheduler does not expose its internal configuration for testing. I could change things in scheduler to make it testable, but but I am not changing any logic around applying the config to the scheduler in this PR. That's why I thought those tests may not belong to this PR.

I think it's probably fine to do in a different PR, but I think as part of this overall feature it's important to test that the values read from the ConfigMap were actually applied by the scheduler. (I realize we had no useful testing for the file case previously.) I think you were mentioning the other day that the factory test could be adapted to do this? Or something simple like what @k82cn suggested in the next comment (just tweak one policy and see that it is respected in how a pod is scheduled).

k82cn · 2017-04-04T07:37:03Z

plugin/cmd/kube-scheduler/app/configurator.go

+			}
+			policyConfigMapFound = true
+			// This loop should iterate only once, as we have already checked the length of Data.
+			for _, val := range policyConfigMap.Data {


prefer to get data directly and remove the comments :).

Please see my previous answer.

k82cn · 2017-04-04T07:38:06Z

test/integration/scheduler/scheduler_test.go

+	informerFactory.Start(stop)
+
+	sched.Run()
+	defer close(stop)


prefer just after "stop:=..."

k82cn · 2017-04-04T07:42:14Z

test/integration/scheduler/scheduler_test.go

+	sched.Run()
+	defer close(stop)
+
+	DoTestUnschedulableNodes(t, clientSet, ns, informerFactory.Core().V1().Nodes().Lister())


The verification can not check whether the config from ConfigMap was set, as the default config can also pass it.

Similar comments to the following cases, the err from app.CreateScheduler can not make sure the config was set in sched; prefer to check sched's data directly or only enable part of predicates/priorities to check the result.

I guess this is the same point as above. I will try to add a test.

Yes, similar point; used to close and re-open browser, the previous comments was "lost" but it's here when I submit new comments :(. Sorry for the confuse.

The sched's did not expose data to check; but I think we can only enable part of predicates/priorities to check, e.g. disable "PortFit" predicates and dispatch two pods with same port to the same host , the second one should NOT be Unscheduable. But I think you will get a better test than that :).

k82cn · 2017-04-04T07:42:27Z

test/integration/scheduler/scheduler_test.go

+	stop := make(chan struct{})
+	informerFactory.Start(stop)
+
+	defer close(stop)


k82cn · 2017-04-04T07:48:48Z

pkg/apis/componentconfig/v1alpha1/types.go

+	// PolicyConfigMapName is the name of the ConfigMap object that specifies
+	// the scheduler's policy config. If UseLegacyPolicyConfig is true, scheduler
+	// uses PolicyConfigFile. If UseLegacyPolicyConfig is false and
+	// PolicyConfigMapName is not empty, the ConfigMap object with this name must


davidopp

I only skimmed it, looks good overall, just a few comments.

davidopp · 2017-04-05T06:42:17Z

plugin/cmd/kube-scheduler/app/configurator.go

-	configData, err := ioutil.ReadFile(sc.policyFile)
-	if err != nil {
-		return nil, fmt.Errorf("unable to read policy config: %v", err)
+	// If there we are in legacy mode or ConfigMap name is empty, try to use


s/there we/we/

davidopp · 2017-04-05T06:42:39Z

plugin/cmd/kube-scheduler/app/configurator.go

+}
+
+// Create implements the interface for the Configurator, hence it is exported
+// even through the struct is not.


s/through/though/

davidopp · 2017-04-05T06:43:37Z

plugin/cmd/kube-scheduler/app/options/options.go

@@ -63,7 +63,9 @@ func (s *SchedulerServer) AddFlags(fs *pflag.FlagSet) {
 	fs.Int32Var(&s.Port, "port", s.Port, "The port that the scheduler's http service runs on")
 	fs.StringVar(&s.Address, "address", s.Address, "The IP address to serve on (set to 0.0.0.0 for all interfaces)")
 	fs.StringVar(&s.AlgorithmProvider, "algorithm-provider", s.AlgorithmProvider, "The scheduling algorithm provider to use, one of: "+factory.ListAlgorithmProviders())
-	fs.StringVar(&s.PolicyConfigFile, "policy-config-file", s.PolicyConfigFile, "File with scheduler policy configuration")
+	fs.StringVar(&s.PolicyConfigFile, "policy-config-file", s.PolicyConfigFile, "File with scheduler policy configuration. This file is used if policy ConfigMap is not provided or scheduler is using legacy policy config.")


I would say "or --use-legacy-policy-config==true" instead of "or scheduler is using legacy policy config"

davidopp · 2017-04-05T06:57:43Z

plugin/cmd/kube-scheduler/app/options/options.go

@@ -63,7 +63,9 @@ func (s *SchedulerServer) AddFlags(fs *pflag.FlagSet) {
 	fs.Int32Var(&s.Port, "port", s.Port, "The port that the scheduler's http service runs on")
 	fs.StringVar(&s.Address, "address", s.Address, "The IP address to serve on (set to 0.0.0.0 for all interfaces)")
 	fs.StringVar(&s.AlgorithmProvider, "algorithm-provider", s.AlgorithmProvider, "The scheduling algorithm provider to use, one of: "+factory.ListAlgorithmProviders())
-	fs.StringVar(&s.PolicyConfigFile, "policy-config-file", s.PolicyConfigFile, "File with scheduler policy configuration")
+	fs.StringVar(&s.PolicyConfigFile, "policy-config-file", s.PolicyConfigFile, "File with scheduler policy configuration. This file is used if policy ConfigMap is not provided or scheduler is using legacy policy config.")
+	fs.StringVar(&s.PolicyConfigMapName, "policy-configmap", s.PolicyConfigMapName, "Name of the ConfigMap object that contains scheduler's policy configuration. It must exist in the system namespace before scheduler initialization if scheduler is not using legacy policy config.")


I would say "if --use-legacy-policy-config==false" instead of "if scheduler is not using legacy policy config"

davidopp · 2017-04-05T07:34:02Z

test/integration/scheduler/scheduler_test.go

+	sched.Run()
+	defer close(stop)
+
+	DoTestUnschedulableNodes(t, clientSet, ns, informerFactory.Core().V1().Nodes().Lister())


I think it's probably fine to do in a different PR, but I think as part of this overall feature it's important to test that the values read from the ConfigMap were actually applied by the scheduler. (I realize we had no useful testing for the file case previously.) I think you were mentioning the other day that the factory test could be adapted to do this? Or something simple like what @k82cn suggested in the next comment (just tweak one policy and see that it is respected in how a pod is scheduled).

timothysc · 2017-04-05T18:33:57Z

pkg/apis/componentconfig/types.go

+	// the scheduler's policy config. If UseLegacyPolicyConfig is true, scheduler
+	// uses PolicyConfigFile. If UseLegacyPolicyConfig is false and
+	// PolicyConfigMapName is not empty, the ConfigMap object with this name must
+	// exist in the default system namespace ("kube-system") before scheduler


Given multi-schedulers and #42961, I think we should probably allow for the flexibility of specifying another namespace.

wanghaoran1988 · 2017-04-06T03:05:52Z

@bsalamat I think maybe you should update the rbac bootstrap policy for the scheduler as you involve new resource type to access.

wanghaoran1988 · 2017-04-06T03:21:13Z

plugin/cmd/kube-scheduler/app/configurator.go

+	var policy schedulerapi.Policy
+
+	namespace := sc.policyConfigMapNamespace
+	if namespace == "" {


Is it better to add to the defaults.go ?

Good point. Done.

wanghaoran1988 · 2017-04-06T03:38:49Z

plugin/cmd/kube-scheduler/app/configurator.go

+	}
+	// If not in legacy mode, try to find policy ConfigMap.
+	if !sc.useLegacyPolicyConfig && len(sc.policyConfigMap) != 0 {
+		policyConfigMap, err := sc.GetClient().CoreV1().ConfigMaps(namespace).Get(sc.policyConfigMap, metav1.GetOptions{})


Do we have default policyConfigMapName ?

No, we don't. If it is empty, we ignore it.

bsalamat · 2017-04-06T07:27:19Z

@wanghaoran1988 is the rbac permissions needed to let scheduler access its ConfigMap?

wanghaoran1988 · 2017-04-06T08:41:18Z

@bsalamat I am not completely sure about that, I met that problem when I update the scheduler lock location in this #42961 , you can have a try to enable rbac in your cluster.

timothysc · 2017-04-08T15:09:43Z

lgtm to me now, if someone else wants to once over.

k82cn · 2017-04-09T03:18:11Z

plugin/cmd/kube-scheduler/app/options/options.go

-	fs.StringVar(&s.PolicyConfigFile, "policy-config-file", s.PolicyConfigFile, "File with scheduler policy configuration")
+	fs.StringVar(&s.PolicyConfigFile, "policy-config-file", s.PolicyConfigFile, "File with scheduler policy configuration. This file is used if policy ConfigMap is not provided or --use-legacy-policy-config==true")
+	fs.StringVar(&s.PolicyConfigMapName, "policy-configmap", s.PolicyConfigMapName, "Name of the ConfigMap object that contains scheduler's policy configuration. It must exist in the system namespace before scheduler initialization if --use-legacy-policy-config==false")
+	fs.StringVar(&s.PolicyConfigMapNamespace, "policy-configmap-namespace", s.PolicyConfigMapNamespace, "The namespace where policy ConfigMap is located. The system namespace will be used if this is not provided or is empty.")


The default value should be "kube-system"

Wait, we do not need that if we have that set in the defaults.go @k82cn

When the default.go will execute? AFAIK, the default will be set by apiserver; not sure when the componentconfig is update.

En, it's set in NewSchedulerServer manually. Also prefer to the set the default value to kube-system so the help message [default=kube-system] will be included.

Anyway, not a block comments.

k82cn · 2017-04-09T03:19:16Z

plugin/pkg/scheduler/algorithm/scheduler_interface.go

@@ -41,4 +41,10 @@ type SchedulerExtender interface {
 // onto machines.
 type ScheduleAlgorithm interface {
 	Schedule(*v1.Pod, NodeLister) (selectedMachine string, err error)
+	// Predicates() returns a pointer to a map of predicate functions. This is


d/a pointer to/
or
s/pointer/reference/

k82cn · 2017-04-09T03:28:26Z

LGTM, minor comments.

wanghaoran1988 · 2017-04-09T04:28:47Z

@liggitt @jayunit100 I am not sure whether openshift will enable user to use configMap change the policy config, but I know that openshift are try to reuse the controller rolebindings, if so, we need update the bootstrap policy for scheduler controller, and I prefer we have an default policy-configmap if we enabled them by default.

liggitt · 2017-04-09T04:35:21Z

This is a very similar discussion to the one kube-dns had, which wanted to make use of dynamic config, and started by making the pod watch a config map. That was modified to allow reading config from a file instead, which allowed mounting the config map holding the config into the pod. This meant no API entanglement or permissions needed for purposes of reading the config, and no modifications to the dns pod whether it was reading from a static file or one mounted from a config map.

k82cn · 2017-04-09T05:36:37Z

/lgtm

k8s-github-robot · 2017-04-09T05:36:56Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bsalamat, k82cn
We suggest the following additional approver: @smarterclayton

Needs approval from an approver in each of these OWNERS Files:

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

k8s-github-robot · 2017-04-09T05:37:03Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2017-04-09T06:19:43Z

Automatic merge from submit-queue

smarterclayton · 2017-04-12T16:40:27Z

I would expect these comments to be fixed - is there an issue for that?

…

On Tue, Apr 11, 2017 at 10:41 PM, Jordan Liggitt ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In plugin/cmd/kube-scheduler/app/configurator.go <#43892 (comment)> : > +func (sc schedulerConfigurator) getSchedulerPolicyConfig() (*schedulerapi.Policy, error) { + var configData []byte + var policyConfigMapFound bool + var policy schedulerapi.Policy + + // If not in legacy mode, try to find policy ConfigMap. + if !sc.useLegacyPolicyConfig && len(sc.policyConfigMap) != 0 { + namespace := sc.policyConfigMapNamespace + policyConfigMap, err := sc.GetClient().CoreV1().ConfigMaps(namespace).Get(sc.policyConfigMap, metav1.GetOptions{}) + if err != nil { + return nil, fmt.Errorf("Error getting scheduler policy ConfigMap: %v.", err) + } + if policyConfigMap != nil { + // We expect the first element in the Data member of the ConfigMap to + // contain the policy config. + if len(policyConfigMap.Data) != 1 { the scheduler should either look for a fixed key or be able to be told which key to use — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#43892 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p9YR9BBzSps-l6qLiJ6L1n339Qquks5rvDm-gaJpZM4MvS1C> .

liggitt · 2017-04-14T03:04:56Z

test/integration/scheduler/scheduler_test.go

+
+// TestSchedulerCreationInLegacyMode ensures that creation of the scheduler
+// works fine when legacy mode is enabled.
+func TestSchedulerCreationInLegacyMode(t *testing.T) {


Seeing test failures on this

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/pr-logs/pull/39716/pull-kubernetes-unit/26428/

bsalamat · 2017-04-26T17:23:38Z

/ref #41600

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Mar 31, 2017

k8s-github-robot assigned erictune Mar 31, 2017

bsalamat force-pushed the sched_conf1 branch from 29fcfd5 to 5bf0765 Compare March 31, 2017 07:13

bsalamat force-pushed the sched_conf1 branch 2 times, most recently from 054a7a1 to 9c16267 Compare April 1, 2017 01:43

gyliu513 reviewed Apr 2, 2017

View reviewed changes

timothysc reviewed Apr 3, 2017

View reviewed changes

timothysc self-assigned this Apr 3, 2017

timothysc added the sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. label Apr 3, 2017

timothysc added this to the v1.7 milestone Apr 3, 2017

timothysc suggested changes Apr 3, 2017

View reviewed changes

k82cn reviewed Apr 4, 2017

View reviewed changes

bsalamat force-pushed the sched_conf1 branch 2 times, most recently from 7d7c7c6 to 66ba4c1 Compare April 5, 2017 00:02

davidopp reviewed Apr 5, 2017

View reviewed changes

timothysc reviewed Apr 5, 2017

View reviewed changes

wanghaoran1988 reviewed Apr 6, 2017

View reviewed changes

bsalamat force-pushed the sched_conf1 branch from c9e9a41 to a453a9f Compare April 6, 2017 07:24

davidopp unassigned erictune Apr 7, 2017

bsalamat force-pushed the sched_conf1 branch from 7e9ec44 to a7fd1d7 Compare April 7, 2017 23:56

bsalamat added 3 commits April 7, 2017 17:03

Scheduler can recieve its policy configuration from a ConfigMap

d1bc320

Generated files

7edc249

Add flags to known-flags

c55e5b6

bsalamat force-pushed the sched_conf1 branch from a7fd1d7 to a23a818 Compare April 8, 2017 00:08

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 8, 2017

Addressed reviewers comments

f9d1333

bsalamat force-pushed the sched_conf1 branch from a23a818 to f9d1333 Compare April 8, 2017 00:32

timothysc added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 8, 2017

wanghaoran1988 mentioned this pull request Apr 8, 2017

Flaky SchedulerPriorities e2e #44127

Closed

k82cn reviewed Apr 9, 2017

View reviewed changes

k8s-ci-robot assigned k82cn Apr 9, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 9, 2017

k8s-github-robot merged commit 6e3bd08 into kubernetes:master Apr 9, 2017

bsalamat deleted the sched_conf1 branch April 10, 2017 18:04

bsalamat mentioned this pull request Apr 12, 2017

Scheduler configurator looks for a specific key in ConfigMap.Data #44404

Merged

liggitt reviewed Apr 14, 2017

View reviewed changes

whs mentioned this pull request Oct 6, 2017

Added PolicyConfigMap and PolicyConfigMapNamespace to KubeSchedulerConfig kubernetes/kops#3546

Merged

Scheduler can recieve its policy configuration from a ConfigMap #43892

Scheduler can recieve its policy configuration from a ConfigMap #43892

Conversation

bsalamat commented Mar 31, 2017

k8s-reviewable commented Mar 31, 2017

bsalamat commented Mar 31, 2017

bsalamat commented Mar 31, 2017

bsalamat commented Mar 31, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timothysc commented Apr 3, 2017

timothysc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidopp left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wanghaoran1988 commented Apr 6, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bsalamat commented Apr 6, 2017

wanghaoran1988 commented Apr 6, 2017

timothysc commented Apr 8, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k82cn commented Apr 9, 2017

wanghaoran1988 commented Apr 9, 2017

liggitt commented Apr 9, 2017

k82cn commented Apr 9, 2017

k8s-github-robot commented Apr 9, 2017

k8s-github-robot commented Apr 9, 2017

k8s-github-robot commented Apr 9, 2017

smarterclayton commented Apr 12, 2017 via email

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bsalamat commented Apr 26, 2017