-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
run scheduler by wiring up to a command #16015
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,53 @@ | ||
package start | ||
|
||
import ( | ||
"github.com/golang/glog" | ||
|
||
kerrors "k8s.io/apimachinery/pkg/util/errors" | ||
schedulerapp "k8s.io/kubernetes/plugin/cmd/kube-scheduler/app" | ||
scheduleroptions "k8s.io/kubernetes/plugin/cmd/kube-scheduler/app/options" | ||
_ "k8s.io/kubernetes/plugin/pkg/scheduler/algorithmprovider" | ||
|
||
cmdflags "github.com/openshift/origin/pkg/cmd/util/flags" | ||
) | ||
|
||
func newScheduler(kubeconfigFile, schedulerConfigFile string, cmdLineArgs map[string][]string) (*scheduleroptions.SchedulerServer, error) { | ||
if cmdLineArgs == nil { | ||
cmdLineArgs = map[string][]string{} | ||
} | ||
if len(cmdLineArgs["kubeconfig"]) == 0 { | ||
cmdLineArgs["kubeconfig"] = []string{kubeconfigFile} | ||
} | ||
if len(cmdLineArgs["policy-config-file"]) == 0 { | ||
cmdLineArgs["policy-config-file"] = []string{schedulerConfigFile} | ||
} | ||
// disable serving http since we didn't used to expose it | ||
if len(cmdLineArgs["port"]) == 0 { | ||
cmdLineArgs["port"] = []string{"-1"} | ||
} | ||
|
||
// resolve arguments | ||
schedulerServer := scheduleroptions.NewSchedulerServer() | ||
if err := cmdflags.Resolve(cmdLineArgs, schedulerServer.AddFlags); len(err) > 0 { | ||
return nil, kerrors.NewAggregate(err) | ||
} | ||
|
||
return schedulerServer, nil | ||
} | ||
|
||
func runEmbeddedScheduler(kubeconfigFile, schedulerConfigFile string, cmdLineArgs map[string][]string) { | ||
for { | ||
// TODO we need a real identity for this. Right now it's just using the loopback connection like it used to. | ||
scheduler, err := newScheduler(kubeconfigFile, schedulerConfigFile, cmdLineArgs) | ||
if err != nil { | ||
glog.Error(err) | ||
continue | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. It seems to me that if there is any parsing error or anything, it will keep filling the logs. How continuing here is helpful? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
It keeps the entire process (including the apiserver) from dying. |
||
} | ||
// this does a second leader election, but doing the second leader election will allow us to move out process in | ||
// 3.8 if we so choose. | ||
if err := schedulerapp.Run(scheduler); err != nil { | ||
glog.Error(err) | ||
continue | ||
} | ||
} | ||
} |
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I dont see we are making sure now that connection overrides still apply, since now the client will be created in scheduler itself, are we ok with not having this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. The path forward will be to configure the differences we need in the scheduler CLI. If the scheduler doesn't support it feature we need, we'll be pursuing them upstream and perhaps picking them back.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thats ok I agree with this. I am concerned if someone supplied QPS/Burst/ContentType in openshift, and doesn't realize that these values wont be passed to scheduler any more? wouldn't it create regressions?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are the defaults bad? If the defaults upstream are good, I'm probably willing to crack some eggs. If they're bad, we do some shenanigans to override. The overrides in openshift are pretty coarse grained and targeted at loopback (which we'll probably be disabling) and controllers (which we have to keep).
Just remember that everything we plumb through special is another thing you'll be describing to ansible to special-case during the (hopefully) 3.8 migration.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I am not really sure upstream defaults are good or bad, lets say they are good. But I agree with your ansible thing.
But It is getting a bit confusing now as there are 4 ways to set QPS/Burst etc
Lets say overrides are not in picture any more, I think I still see issue in createClient function: https://github.com/openshift/origin/blob/master/vendor/k8s.io/kubernetes/plugin/cmd/kube-scheduler/app/configurator.go#L57
Even if a user does not set any scheduler flags for QPS/burst etc, QPS/Burst kubeconfig values are being overwritten by defaults, as there is no way to differentiate if these were set by user or are defaults.
I think we will have to decide if we are ok with kubeconfig supplied values being overwritten by defaults. I think this issue is something I will have to look into kube upstream too.
Though, one thing I am wondering, I am not sure, in practice what is preferred way of configuration for these QPS/Burst, via kubeconfig, or via scheduler's flags etc.