🌱 [WIP] Proposal for Phase 2 Plugins #1801

rashmigottipati · 2020-11-09T05:55:47Z

Description of the change
This PR adds a design proposal for Kubebuilder Phase 2 Plugins

Motivation for the change
As the proposed scaffold plugin system is implemented, looking forward to achieving agreement on a way forward for Kubebuilder Phase 2 Plugins so that kubebuilder and operator sdk projects can align on requirements and progress on phase 2, which adds support for chaining plugins together

k8s-ci-robot · 2020-11-09T05:55:54Z

Hi @rashmigottipati. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot · 2020-11-09T05:55:55Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rashmigottipati
To complete the pull request process, please assign directxman12 after the PR has been reviewed.
You can assign the PR to them by writing /assign @directxman12 in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

camilamacedo86

It is nice. I added a few questions.

camilamacedo86 · 2020-11-09T10:48:12Z

designs/plugins-phase-2.md

+
+## Overview
+
+Phase 2 of the scaffold plugin system adds support for chaining plugins together. The idea for plugins phase 2 is to have support for a plugin architecture that enables plugins to be separate binaries discoverable by the Kubebuilder CLI binary via user specified plugin file paths and that they are language indepedent. Some of the phase 2 high-level requirements summarized from the feature request and discussions are: 


what do you mean with enables plugins to be separate binaries? We will build each plugin? Will we not so longer extending the code implementation?

camilamacedo86 · 2020-11-09T10:50:58Z

designs/plugins-phase-2.md

+
+##### Go built-in plugin package vs Hashicorp plugin library vs custom plugin library
+
+Based on the evaluation of plugin libraries such as the built-in go-plugin library and Hashicorp’s plugin library, we have come to the conclusion that it is more suitable to write our own custom typed plugin library.


Could we add the link as a reference/link of the libraries that are cited here?

camilamacedo86 · 2020-11-09T10:53:38Z

designs/plugins-phase-2.md

+    Config *config.Config `json:"config,omitempty"`
+
+    // Files contains the model of the files that are being scaffolded
+    Files map[string]File `json:"files,omitempty"`	


are we speaking here about an array with all boilerplates?

I think it is talking about already templated files. Is there anyway that allows us to pass it before templating and templating after all plugins have run? I think we may be able to abstract part of the templating and scaffolding code so that plugins are easier to develop.

camilamacedo86 · 2020-11-09T10:54:24Z

designs/plugins-phase-2.md

+The proposed plugin universe is: 
+
+```go
+type File struct {


ìs the File our boilerplate? (template used to scaffold the files)

camilamacedo86 · 2020-11-09T10:55:28Z

designs/plugins-phase-2.md

+    Contents string `json:"contents,omitempty"`
+
+    // Info is the os.FileInfo 
+    Info os.FileInfo  `json:"info,omitempty"`	


why we need to have an attribute such as Info os.FileInfo? Is to allow write a dir for example? Could you please add the use cases?

camilamacedo86 · 2020-11-09T10:59:22Z

designs/plugins-phase-2.md

+To support phase 2 plugins, we propose that we modify the "layout" PROJECT file key to either of these options as order matters for certain plugins: 
+    * Ordered list of plugin {name}/{version} or
+    * Ordered map of {name}/{version} to plugin-specific config


The PROJECT file represents the input/config of the project. so, are you saying here that we need to store the plugins used in the order that they were executed? Am I right?

camilamacedo86 · 2020-11-09T11:01:37Z

designs/plugins-phase-2.md

+
+```yaml
+version: "3-alpha"
+layout: ["go/v1.0.0", "go.kubebuilder.io/v3-alpha"]


the layout means the father of the plugins used to scaffold the project. So, what would mean "go/v1.0.0", "go.kubebuilder.io/v3-alpha"? Could we give an example of a scenario that we would like to address and then have the project file as it would be in this scenario for a better understand?

I think this example is confusing, as both plugins are the base go plugin (one with the short form and the other one the full name) which doesn't make sense. Additionally, we have v2 and v3 versions of that plugin, no v1.0.0 nor v3-alpha, to add up to the confussion.

camilamacedo86 · 2020-11-09T11:04:07Z

designs/plugins-phase-2.md

+  version: v1
+plugins:
+  - go.sdk.operatorframework.io/v2-alpha: {
+      type: JSON


Since the tool has access to the plugin implementation, why we need to add the type here? Could you please add the example scenario and then describe why/when the Type is used in the process to illustrate why it is required at all?

I think the type is meant to determine how the universe is passed. JSON doesn't seem to be a good name election as it is just a encoding language. I would say in-tree or built-in or something similar for the ones that we already have and stdin/stdout or something similar for the ones that you are proposing in this document.

camilamacedo86 · 2020-11-09T11:05:22Z

designs/plugins-phase-2.md

+      type: JSON
+  }
+``` 
+


Could we add a real/valid scenario for we have a better understanding of the purposed idea? E.g Something such as: add the use case then address the scenario with the example.

I am as X, would like to N, so that I can Z.

Run the command : otherbin init --plugins=[a,b]

Then, update the path path of the plugin

Do the scaffold

The Project file would be like as:

camilamacedo86 · 2020-11-09T11:11:08Z

designs/plugins-phase-2.md

+    * Either an ordered list of plugin {name}/{version}, or ordered map of {name}/{version} to plugin-specific config.
+    * Order matters for certain plugins, ex. a bazel plugin would need to run after all Go files are generated.
+
+## Evaluation of Plugin libraries 


Could we add here the use cases that we are trying to address with this proposal (I am as X, would like to N, so that I can Z.)? Also, could we add how we will test the plugins and ensure its quality?

camilamacedo86 · 2020-11-09T11:35:14Z

designs/plugins-phase-2.md

+
+## Comments & Questions
+
+* How to handle help text? Are plugins allowed to define their own flags?


Could you please add an example over who the consumer will consume and customize the plugins in phase 2? Or are we not covering it in this phase? For example, I am as a consumer(sdk) of kubebuilder plugins, would like to extend the init subcommand/plugin and define that the main.go will be a scaffold in a new path, so that for my tool can do the same scaffolds but in another path.

Also, would be nice we have here what are the goals and no goals of this EP. So, if it is not a goal of this proposal we would easily know.

Adirio

Several questions, mostly to clarify myself:

Adirio · 2020-11-09T16:16:12Z

designs/plugins-phase-2.md

+
+## Overview
+
+Phase 2 of the scaffold plugin system adds support for chaining plugins together. The idea for plugins phase 2 is to have support for a plugin architecture that enables plugins to be separate binaries discoverable by the Kubebuilder CLI binary via user specified plugin file paths and that they are language indepedent. Some of the phase 2 high-level requirements summarized from the feature request and discussions are: 


Interpreted languages, i.e. Python, do not generate binaries. Are we leaving them apart intentionally? I think that interpreted languages are well suited for this kind of scripting purposes and considering them an option would be interesting.

Adirio · 2020-11-09T16:17:55Z

designs/plugins-phase-2.md

+
+* mdbook-like structure
+    * A Plugin receives a JSON blob plugin "universe" from upstream plugins containing file "nodes", each of which contain file metadata.
+    * The plugin can add or delete nodes (nodes are not mutable), then returns the modified universe so downstream plugins can do the same.


I would like to discuss why nodes need to be inmutable. There are probably good reasons for you to consider them like that, I just want to know them.

Adirio · 2020-11-09T16:20:41Z

designs/plugins-phase-2.md

+
+The built-in plugin library seems to be a non-starter as it is more suitable for in-tree plugins rather than out-of-tree and it doesn’t offer cross language support. Hashicorp’s go plugin system seems more suitable than built-in go-plugin library as it enables cross language/platform support. However, it is more suited for long running plugins as opposed to short lived plugins and it could be overkill with the usage of protobuf.
+
+For the stated reasons, the proposal is to write our own plugin system that passes JSON blobs back and forth across stdin/stdout.


Again, probably good reasons to choose stdin/stdout. Could we addd some justification why we chose this approach isntead of, for example, sockets?

Adirio · 2020-11-09T16:25:48Z

designs/plugins-phase-2.md

+    Config *config.Config `json:"config,omitempty"`
+
+    // Files contains the model of the files that are being scaffolded
+    Files map[string]File `json:"files,omitempty"`	


I think it is talking about already templated files. Is there anyway that allows us to pass it before templating and templating after all plugins have run? I think we may be able to abstract part of the templating and scaffolding code so that plugins are easier to develop.

Adirio · 2020-11-09T16:32:16Z

designs/plugins-phase-2.md

+
+```yaml
+version: "3-alpha"
+layout: ["go/v1.0.0", "go.kubebuilder.io/v3-alpha"]


I think this example is confusing, as both plugins are the base go plugin (one with the short form and the other one the full name) which doesn't make sense. Additionally, we have v2 and v3 versions of that plugin, no v1.0.0 nor v3-alpha, to add up to the confussion.

Adirio · 2020-11-09T16:34:49Z

designs/plugins-phase-2.md

+  version: v1
+plugins:
+  - go.sdk.operatorframework.io/v2-alpha: {
+      type: JSON


I think the type is meant to determine how the universe is passed. JSON doesn't seem to be a good name election as it is just a encoding language. I would say in-tree or built-in or something similar for the ones that we already have and stdin/stdout or something similar for the ones that you are proposing in this document.

Adirio · 2020-11-09T16:35:22Z

designs/plugins-phase-2.md

+  kind: Captain
+  version: v1
+plugins:
+  - go.sdk.operatorframework.io/v2-alpha: {


This doesn't match any of the plugins in layout. How should we interpret this?

Adirio · 2020-11-09T16:40:44Z

designs/plugins-phase-2.md

+
+## Comments & Questions
+
+* How to handle help text? Are plugins allowed to define their own flags?


I would say that a *.yaml file that defines the characteristics of the plugin could be used. For example it could define that we should call it with python file-of-the-plugin.py and that uses stdin and stdout to communicate with the main executable. When the main executable discovers the plugins it would read this file and register that plugin. In this file we could also add thing like allowed flags, commands, descriptions, etc.

Adirio · 2020-11-09T16:42:49Z

designs/plugins-phase-2.md

+## Comments & Questions
+
+* How to handle help text? Are plugins allowed to define their own flags?
+* Do we allow plugins to depend on environment variables? 


Cons: the same plugin call would do different things depending on the environment
Difficulties: hard to enforce a plugin not to use environmental variables when it is an external executable for you.

Adirio · 2020-11-15T17:11:16Z

Prow only detects WIP if they are the first word in the title, so I'm manually holding it as it won't get the hold:work-in-progress label automatically. Feel free to remove the hold once it is not a WIP.

/hold
/ok-to-test

Adirio

Some additional considerations

Adirio · 2021-01-30T11:09:57Z

designs/plugins-phase-2.md

+
+## Overview
+
+Phase 2 of the scaffold plugin system adds support for chaining plugins together. The idea for plugins phase 2 is to have support for a plugin architecture that enables plugins to be separate binaries discoverable by the Kubebuilder CLI binary via user specified plugin file paths and that they are language indepedent. Some of the phase 2 high-level requirements summarized from the feature request and discussions are: 


I feel like there are 2 steps here:

Chain plugins

Enable external/out-of-tree plugins
I will refer to Phase 1.5 to achieve the first step and Phase 2 to achieve both.

Adirio · 2021-01-30T11:14:24Z

designs/plugins-phase-2.md

+* Pre-generate a project file for handling complex plugin use-cases.
+* Enhanced "layout" PROJECT file key.
+    * Either an ordered list of plugin {name}/{version}, or ordered map of {name}/{version} to plugin-specific config.
+    * Order matters for certain plugins, ex. a bazel plugin would need to run after all Go files are generated.


The current project file supports a comma-separated lists of plugins in the layout field and offers a unstructured field were plugins can store their config. While I do like the suggested approach of having an ordered map with plugin-keys to configs, that is a breaking change and would require project version 4. Project version 3 (current) is still able to support our needs so this can be implemented without having to wait for the fourth project version.

Adirio · 2021-01-30T11:29:21Z

designs/plugins-phase-2.md

+The built-in plugin library seems to be a non-starter as it is more suitable for in-tree plugins rather than out-of-tree and it doesn’t offer cross language support. Hashicorp’s go plugin system seems more suitable than built-in go-plugin library as it enables cross language/platform support. However, it is more suited for long running plugins as opposed to short lived plugins and it could be overkill with the usage of protobuf.
+
+For the stated reasons, the proposal is to write our own plugin system that passes JSON blobs back and forth across stdin/stdout.
+The plugin specification will include type metadata to allow the potential for using other plugin libraries in the future if the need arises. From an implementation standpoint, we could create a new implementation of our plugin interface based on the type specified in the project file. We can do a type switch on the type of plugin the data is being passed to. For ex: if it is native go, then we pass the universe directly & if it is a binary wrapper, then we pass the serialized stream of JSON bytes to it. 


From the implementation viewpoint, instead of making the cli understand all different kind of plugins, we could code a plugin with the current interface that is in charge of doing this translation. For example, we have a JSON plugin that has the current plugin interface and creates the JSON blobs and pass them to the external plugin.

Now, imagine I'm a C++ plugin developer, I would copy this JSON plugin and develop my C++ plugin that interacts with the JSON-based exposed API.

By doing this, we would only need to support out-of-tree Go plugins, as the integration with other languages would be done by this JSON (and potentially others in a future) plugins. What do you hink about this approach? And would this approach bring the built-in plugin library back to the table?

By the way, I think that these plugins should work under demand more than sending all the data. The C++ plugin would request the admiral_types.go file representation and the plugin would sent it in a JSON blob. Obviously the plugin will send some general metadata first (for example, but not limitd to, the plugin config).

camilamacedo86

Hi @rashmigottipati,

It is open for a while and without follow up to address the comments. Also, we have the 1.5 EP which was accepted and is implemented already and very close to be merged. See: #2060

This shows that the plugin phase 2.0 after that get merged and the issues in #2016 are addressed might only be more about check the possibility of the plugins be as separate binaries that can be discovered by the Kubebuilder CLI binary via user-specified plugin file paths.

So, could you please follow that and check if this one could be closed or updated to address what will be missing accordingly?

rashmigottipati · 2021-03-17T14:51:58Z

Closing this as it makes more sense to create a new proposal that incorporates plugin phase 1.5 implementation and discusses design of plugins as separate binaries that can be discovered by the Kubebuilder CLI binary via user-specified plugin file paths.

Enhancement proposal for phase 2 plugins

57a3136

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Nov 9, 2020

k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Nov 9, 2020

k8s-ci-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Nov 9, 2020

k8s-ci-robot requested review from DirectXMan12 and pwittrock November 9, 2020 05:55

rashmigottipati changed the title ~~🌱 : [WIP] Proposal for Phase 2 Plugins~~ 🌱 [WIP] Proposal for Phase 2 Plugins Nov 9, 2020

camilamacedo86 reviewed Nov 9, 2020

View reviewed changes

Adirio reviewed Nov 9, 2020

View reviewed changes

joelanford mentioned this pull request Nov 17, 2020

[:warning:] bump k8s dependencies to 1.19 in 0.6.x release kubernetes-sigs/controller-runtime#1266

Closed

Adirio mentioned this pull request Nov 17, 2020

⚠ pkg/cli overhaul #1828

Merged

2 tasks

estroz mentioned this pull request Dec 3, 2020

operator-sdk 1.0 needs a way to use podman instead of docker operator-framework/operator-sdk#3623

Closed

Adirio reviewed Jan 30, 2021

View reviewed changes

camilamacedo86 reviewed Mar 6, 2021

View reviewed changes

rashmigottipati closed this Mar 17, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🌱 [WIP] Proposal for Phase 2 Plugins #1801

🌱 [WIP] Proposal for Phase 2 Plugins #1801

rashmigottipati commented Nov 9, 2020

k8s-ci-robot commented Nov 9, 2020

k8s-ci-robot commented Nov 9, 2020

camilamacedo86 left a comment

camilamacedo86 Nov 9, 2020

camilamacedo86 Nov 9, 2020

camilamacedo86 Nov 9, 2020

Adirio Nov 9, 2020

camilamacedo86 Nov 9, 2020

camilamacedo86 Nov 9, 2020

camilamacedo86 Nov 9, 2020

camilamacedo86 Nov 9, 2020

Adirio Nov 9, 2020

camilamacedo86 Nov 9, 2020

Adirio Nov 9, 2020

camilamacedo86 Nov 9, 2020

camilamacedo86 Nov 9, 2020

camilamacedo86 Nov 9, 2020

Adirio left a comment

Adirio Nov 9, 2020

Adirio Nov 9, 2020

Adirio Nov 9, 2020

Adirio Nov 9, 2020

Adirio Nov 9, 2020

Adirio Nov 9, 2020

Adirio Nov 9, 2020

Adirio Nov 9, 2020

Adirio Nov 9, 2020

Adirio commented Nov 15, 2020

Adirio left a comment

Adirio Jan 30, 2021

Adirio Jan 30, 2021

Adirio Jan 30, 2021

camilamacedo86 left a comment

rashmigottipati commented Mar 17, 2021


		## Overview

		Phase 2 of the scaffold plugin system adds support for chaining plugins together. The idea for plugins phase 2 is to have support for a plugin architecture that enables plugins to be separate binaries discoverable by the Kubebuilder CLI binary via user specified plugin file paths and that they are language indepedent. Some of the phase 2 high-level requirements summarized from the feature request and discussions are:


		##### Go built-in plugin package vs Hashicorp plugin library vs custom plugin library

		Based on the evaluation of plugin libraries such as the built-in go-plugin library and Hashicorp’s plugin library, we have come to the conclusion that it is more suitable to write our own custom typed plugin library.


		## Comments & Questions

		* How to handle help text? Are plugins allowed to define their own flags?


		The built-in plugin library seems to be a non-starter as it is more suitable for in-tree plugins rather than out-of-tree and it doesn’t offer cross language support. Hashicorp’s go plugin system seems more suitable than built-in go-plugin library as it enables cross language/platform support. However, it is more suited for long running plugins as opposed to short lived plugins and it could be overkill with the usage of protobuf.

		For the stated reasons, the proposal is to write our own plugin system that passes JSON blobs back and forth across stdin/stdout.

🌱 [WIP] Proposal for Phase 2 Plugins #1801

🌱 [WIP] Proposal for Phase 2 Plugins #1801

Conversation

rashmigottipati commented Nov 9, 2020

k8s-ci-robot commented Nov 9, 2020

k8s-ci-robot commented Nov 9, 2020

camilamacedo86 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Adirio left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Adirio commented Nov 15, 2020

Adirio left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

camilamacedo86 left a comment

Choose a reason for hiding this comment

rashmigottipati commented Mar 17, 2021