Support out-of-process and out-of-tree cloud providers #88
Comments
|
Benefits:
Just a note, |
|
cc @kubernetes/sig-cluster-lifecycle @kubernetes/sig-network @kubernetes/sig-storage @kubernetes/sig-aws @kubernetes/sig-openstack |
|
I endorse this idea in general. I think the built-in cloud provider logic has served its purpose and its time to modularize. I think there are a number of facets to this that we have to work out including but not limited to:
I think it would be worthwhile to start building a doc that details these and explores options for ejecting each one. I don't think there's anything here that hasn't been considered at SOME point. Once we get that written down, we can craft a roadmap... |
|
Sounds useful. @errordeveloper, do we have any mailing lists or GitHub discussions with this question what we can refer to? |
|
@idvoretskyi not yet, this is probably very much on the radar of @kubernetes/sig-cluster-lifecycle. |
|
I don't know that anyone is working on speccing this. It touches on a few On Tue, Sep 20, 2016 at 5:35 AM, Ilya Dmitrichenko <notifications@github.com
|
|
@thockin you are right. May be we should form |
|
so. many. sigs. I don't think we need a SIG for this. I doubt if it is On Wed, Sep 21, 2016 at 12:07 AM, Ilya Dmitrichenko <
|
|
@errordeveloper no need in yet another SIG (SIG-Cloud sounds like an abstract and umbrella one). I agree with @thockin - the primary SIG has to be @kubernetes/sig-cluster-lifecycle; while on behalf of @kubernetes/sig-openstack I'm going to track this item. Hope other cloud-SIG's will be involved in the process as well. |
|
@justinsb and I have discussed this on Slack, and looks like we may be able to get closer to getting similar user-facing value by exposing flags via component config. It also turns out |
|
I think there is additional value to moving code out to add-ons: it will enable further cloud providers to be added without enlarging the core of Kubernetes. Example: kubernetes/kubernetes#32419 |
|
Ah, but it also looks like someone is working on this: On Wed, 28 Sep 2016, 18:19 Bryan Boreham, notifications@github.com wrote:
|
|
That does not read as someone working on it, to me. This is a big problem On Wed, Sep 28, 2016 at 10:27 AM, Ilya Dmitrichenko <
|
|
@thockin In reference to kubernetes/kubernetes#32419, Rancher would be up for being a guinea pig for this. @wlan0 will be working on this and if the scope is massive we will see if we can pull in more resources. I want to see if understand the approach you were proposing in kubernetes/kubernetes#32419 and see if we are on the same page. What we would do is implement the existing cloudprovider.Interface with a new cloud provider called "external". Ideally we wouldn't change the existing Interface, but if we hit some oddities it might make sense to modify it. This new external implementation will not delegate via a plugin model but instead through k8s resources and expect one to write controllers. Upfront it seems like we would need some new resources like CloudProviderLoadBalancer, Instance, Zone, Cluster, Route. A new cloud provider would need to be a controller that interacted with these resources. That all seems pretty straight forward to me. Now the weird part is volume plugins. While it's not part of the CloudProvider interface, there seems to be a back channel relationship between volume plugins and cloud providers. To decouple those I'd have to spend a bit more time researching. @thockin Is this the basic approach you were thinking? |
|
I replied to @wlan0, but for the record... simpler. My "external" suggestion was more about designating that we are not using a Once the built-in controllers are nullified, we run a cloud-specific One possible result is a library pkg that accepts a I would suggest leaving volumes for last :) On Thu, Sep 29, 2016 at 2:37 PM, Darren Shepherd notifications@github.com
|
|
Why is one linked in at all? What is the difference between Or maybe we're on the same page? And then you're proposing a generic implementation of these standalone controllers (for now?) that can take the existing |
|
On Fri, Sep 30, 2016 at 12:16 AM, Cole Mickens notifications@github.com wrote:
Without inspecting, I don't know if "" disables the controllers today,
I think same page. As a starting point, we would decompose the single |
|
So the controller manager that embeds certain control loops. Some of these loops are cloud provider specific:
but most are provider agnostic:
I wonder if it would make sense to split the controller into 2 parts: base-controller (k8s code base). And provider-specific-controller (external repo, deployed by the user by choice). This way it would be more similar to the ingress controller path with the only slight difference: controller loops should be maintained as a library as all the providers will share them. Only implementation - attach/detachDisk/etc - will be provider specific. To make it backwards compatible, we can disble initializing cloud provider specific controllers in the current controller-manager code if the provider is passed as empty on the kubernetes start. Or may be I'm just stating what you've already meant by "keep maintaining the cloud controller |
|
I think we are saying the same thing. kube-controller-manager will still I would leave volumes for VERY LAST :) On Fri, Sep 30, 2016 at 4:10 PM, Alena Prokharchyk <notifications@github.com
|
Addresses: kubernetes/enhancements#88 This commit starts breaking the controller manager into two pieces, namely, 1. cloudprovider dependent piece 2. coudprovider agnostic piece the controller manager has the following control loops - - nodeController - volumeController - routeController - serviceController - replicationController - endpointController - resourcequotacontroller - namespacecontroller - deploymentController etc.. among the above controller loops, - nodeController - volumeController - routeController - serviceController are cloud provider dependent. As kubernetes has evolved tremendously, it has become difficult for different cloudproviders (currently 8), to make changes and iterate quickly. Moreover, the cloudproviders are constrained by the kubernetes build/release lifecycle. This commit is the first step in moving towards a kubernetes code base where cloud providers specific code will move out of the core repository, and will be maintained by the cloud providers themselves. I have added a new cloud provider called "external", which signals the controller-manager that cloud provider specific loops are being run by another controller. I have added these changes in such a way that the existing cloud providers are not affected. This change is completely backwards compatible, and does not require any changes to the way kubernetes is run today. Finally, along with the controller-manager, the kubelet also has cloud-provider specific code, and that will be addressed in a different commit/issue.
|
Hi @thockin Is this the one you mentioned to me in the hall way conversation in Barcelona? Just making sure! |
|
Is this planned for v1.6 or what's the plan? |
|
@dims yeah, this is the one. On Mon, Oct 31, 2016 at 8:17 PM, Davanum Srinivas notifications@github.com
|
|
This feature will remain beta in 1.13. In Q4 we're planning to break this feature down a bit because the "out-of-tree" cloud provider model has a lot of different components and not all the pieces need to be feature gated. We're hoping to have final consensus on this at KubeCON Seattle. |
|
We're going to have more discussions around this during KubeCON, but my sentiment for this enhancement at the moment is that it would be more effective if we created enhancements per in-tree provider (e.g. #631 & kubernetes/kubernetes#50752) and close this one out. The reasoning is that we've already built the functionality to support out-of-tree providers and need to have more providers executing. This catch-all enhancement makes it difficult to track the progress of each provider going forward (some providers may already support "beta" level out-of-tree cloud providers while others have not even started). |
|
@andrewsykim what if it was renamed "deprecate in-tree cloud providers"? Or something along the lines of removing all of them. I agree that every vendor will be working on their own implementation, but not sure if they are going to take the time to deprecate nor remove the code from the core. thoughts? |
|
Yes, removal is definitely on our roadmap! I would still be in favor of a new enhancement for that as this issue is pretty cluttered as it is. What do you think? |
|
On second thought, removal of in-tree providers may also depend on the progress of each provider so might be good to keep that close to each cloud provider's out-of-tree enhancement issue. |
|
I believe a tracking issue should be made for every out of tree provider that also has a native in-cloud provider. Beyond that, it's a larger discussion to see if a tracking issue needs to be created for "every" provider since it's an out-of-tree driver and won't effect anything in k/k. |
Totally agree with this statement, and @andrewsykim's point as well. As this is going kinda well now from an architectural perspective (it's possible to move out in practice now), it makes sense to ONLY track the individual in-tree -> out-of-tree migrations we have, and once they provider is totally out of the core k8s tree, it will not be tracked here anymore but have its own release cycle and all that. With that agree with the previous saying and am saying we should not add e.g. a "Kubernetes enhancement issue" here for an all-out-of-core (from the start) cloud provider like Digital Ocean. |
|
+1 to having separate enhancement tracking issues (and associated KEPs) for each provider. |
|
@andrewsykim nice work! I believe Photon may be in a limbo state. @dougm or @frapposelli can you comment on that? |
|
CloudStack, oVirt and Photon are in the process of being removed for 1.14: kubernetes/kubernetes#72178 |
|
What is tracking the fact that once the in tree cloud providers are removed , the external cloud controller manager has to be by default installed as an add on when installing on gcp/ aws otherwise things like getting a service with load balancer won’t work |
|
@krmayankk installation and configuration of individual cloud providers will have to be done on a per-use case basis. Each cloud provider currently has their implementation in their respective SIGs. Documentation will be required for each cloud provider. I don't foresee any tracking beyond that since the vendor is responsible for making it consumer friendly. |
|
@wlan0 Hello - I’m the enhancement’s lead for 1.14 and I’m checking in on this issue to see what work (if any) is being planned for the 1.14 release. Enhancements freeze is Jan 29th and I want to remind that all enhancements must have a KEP |
|
@claurence thanks for the ping! We're going to be closing this issue in the next few days in favor of having an enhancement issue per cloud provider (see #88 (comment)). I will follow up with leads for each provider to update their enhancement issues (and respective KEPs if there aren't any). |
|
/close |
|
@andrewsykim: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Add audit logs to cluster-logging-log-forwarding enhancement proposal
* Add a unique identifier to each fature * Change IDs to strings * Update feature IDs
and others
Feature Description:
Support out-of-tree and out-of-process cloud providers, a.k.a pluggable cloud providers.
Feature Progress:
In order to complete this feature, cloud provider dependencies need to be moved out the the following Kubernetes binaries, then docs and tests need to be added. The Links to the right hand side of the binary denote the PRs that lead to the completion of the sub-feature
The cloud-specific functionality of the above features needs to be moved into a new binary called cloud-controller-manager that support a plugin architecture.
Primary Contact: @wlan0
Responsible SIG: @k8s-mirror-cluster-lifecycle-feature-re
Design Proposal Link: kubernetes/community#128
Reviewers:
@luxas
@roberthbailey
@thockin
Approver:
@thockin
Feature Target:
Alpha: 1.7
Beta: 1.8
Stable: 1.10
Here's an updated status report for this feature, please let me know if anything needs clarification:
Beta (starting v1.11)
Reasoning for Graduation
There were a few things on our TODO list that we wanted to get done before graduating to beta such as collecting E2E tests from all providers & improving out-of-tree storage. However, many of these initiatives require collaboration from external parties that was delaying progress on this effort. In addition, there was uncertainty since we do not develop some of the components we rely on, a good example is whether CSI would be able to meet demands for out-of-tree storage that was on par with in-tree storage support. Though in hindsight we have more confidence in CSI, prior to its beta release it was unclear if it would meet our requirements. With this context in mind, we had decided to graduate to beta because:
Goals for GA (targetted for v1.13/v1.14)
The text was updated successfully, but these errors were encountered: