Context Sharing POC #1855

dsimansk · 2023-08-22T14:16:42Z

Description

This's a bit simplified approach from original Feature Track proposal. I wonder if we should simply start with a few well known keys, that can be consumed in kn core + plugins

Changes

Reference

Fixes #

Release Note

TBD

/cc @rhuss

dsimansk · 2023-08-22T14:31:01Z

pkg/kn/plugin/context.go

+		mockData := `
+{  
+  "service":"hello",
+  "namespace":"default"
+}


This represents very minimal shared context between every Consumer and Producer to read/write. E.g. in a seems floating value changed on latest usage of kn or plugins.

That's a looser contract then with predefined manifest etc.

This could be further split into plugin specific sub-context and shared sub-context. Where plugin specific is only accessed by it's owner. In addition with read/write permission field maybe?

dsimansk · 2023-08-24T10:34:44Z

pkg/kn/plugin/context.go

+	return ctxManager, nil
+}
+
+func (c *ContextDataManager) Find(key string) string {


Then defining the API as a set of access methods to retrieve and store the well-known values.

rhuss · 2023-08-25T09:57:54Z

@dsimansk thanks for this initial work. I wouldn't mind to restrict ourselves to a set of fixed keys, but I'm not sure what this will buy us. Of course, the core commands themselves should check the Context for any mandatory argument that the user has not provided. These are are well known keys like "service" or "namespace". However, I would not prohibit that a plugin could add any key to the context manager, as this is not harmful as the core commands never will lookup this. However other plugins might make use of it, so that e.g. the function plugin could communicate with the event plugin.

The tricky part really is, how external, non-inlined plugins can add to the context. For inline plugins, this is trivial as they just can add to a global map. For external plugins. that would need to an external call. For optimization, the core should know which plugin supports which context key, therefor the idea of the manifest that is obtained once during startup by calling every plugin that is reachable. This might be dangerous if plugins are looked up from the path and there are some dangling plugins or bogus plugins that would be called. So not really sure if this is the proper way. Alternatively we could keep a central registry remotely that contains this information in a plain file. We could fetch this once from a central place (e.g. GitHub) or fallback to a default one that we have compile into kn. This information ("which plugin supports which keys") should be quite static and don't changes very often.

rhuss · 2023-08-25T09:59:10Z

on the other hand, we also could reconsider to implement the "sharing via a local file" first, too. Not sure how far we should go.

dsimansk · 2023-08-31T12:26:25Z

@rhuss updated with a building blocks to Manifest structures and retrieving those from inlined and external plugins. It's starting to be messy, be careful. :)

dsimansk

I'm not quiet sure how to allow external plugins to store any context data though. Other than by directly writting to context.json file.

docs/cmd/kn.md

dsimansk · 2023-08-31T12:29:04Z

pkg/kn/plugin/context_sharing.go

+
+// TODO: We should cautiously execute external binaries
+// fetchExternalManifest returns Manifest from external plugin by exec `$plugin manifest get`
+func fetchExternalManifest(p Plugin) *Manifest {


That's a bit tricky part. Executing binary half blind if it's desired one even?

Preview of manifest flag: knative-extensions/kn-plugin-service-log@c214f72

That can be a standalone cmd. I.e. kn service log manifest, but it kind of makes single purpose plugins grow the list of cmds. :)

Indeed, the tricky part is not blindly calling each plugin in the path somehow, if possible. Of course if a plugin is in the path, it can be called anyway (maybe by accident), but chances are higher that a malicious plugin is called when it is called by itself. A simple solution could be to keep track of which plugins have been called explicitly and only can them when picking up all manifests (which might be hard to maintain, though ...)

dsimansk · 2023-09-01T08:58:55Z

pkg/kn/commands/service/describe.go

+				c, _ := plugin.NewContextManager()
+				serviceName = c.GetDefault()["service"]
+				if serviceName == "" {
+					return errors.New("'service describe' requires the service name given as single argument")
+				}


@rhuss what would you say about using Golang's buily-in context.Context since map[string]string is our context now anyway? We can pass it through ExecuteContext() function that's already part of Cobra lib.

Or shoud we be a bit more bold here and inject the params directly into args & flags?

rhuss · 2023-09-01T09:59:22Z

Thanks! I hope i can review it today, if not, Iwill do it early next week.

codecov · 2023-09-14T15:29:10Z

Codecov Report

Attention: 90 lines in your changes are missing coverage. Please review.

Comparison is base (1719498) 79.62% compared to head (7e5bef5) 79.20%.
Report is 6 commits behind head on main.

❗ Current head 7e5bef5 differs from pull request most recent head e2d3a49. Consider uploading reports for the commit e2d3a49 to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1855      +/-   ##
==========================================
- Coverage   79.62%   79.20%   -0.43%     
==========================================
  Files         179      180       +1     
  Lines       13927    14075     +148     
==========================================
+ Hits        11090    11148      +58     
- Misses       2070     2155      +85     
- Partials      767      772       +5

Files	Coverage Δ
pkg/kn/config/testing.go	`100.00% <100.00%> (ø)`
pkg/kn/plugin/manager.go	`84.30% <100.00%> (+0.11%)`	⬆️
pkg/kn/config/config.go	`57.71% <0.00%> (-0.79%)`	⬇️
pkg/kn/commands/service/describe.go	`70.07% <50.00%> (-1.35%)`	⬇️
cmd/kn/main.go	`76.69% <15.00%> (-11.14%)`	⬇️
pkg/kn/plugin/context_sharing.go	`43.96% <43.96%> (ø)`

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

dsimansk · 2023-09-21T13:45:31Z

@rhuss I'll be adding unit test, but it's ready for a review and feedback I believe.

rhuss

Thank, I did a first round of review. Beside the usual bike shedding some more substantial comments inline:

Consider cleaning the cache, too, if some external plugin is not in the list anymore.
Only do the lookup dance and external call to a plugin if really needed. E.g. it should be lazily done, e.g. when the service describe does not find a service name provided, instead of doing it always. That would keep the current behavior and only fallback to a context lookup when needed.

rhuss · 2023-09-25T07:43:38Z

cmd/kn/main.go

+
+		defer func(ctxManager *pluginpkg.ContextDataManager) {
+			if err := ctxManager.WriteCache(); err != nil {
+				println("error during write")


This error message should contain more context (pun intended :-)

also, you propbably want to write this to standard error.

cmd/kn/main.go

rhuss · 2023-09-25T07:54:56Z

pkg/kn/commands/service/describe.go

@@ -182,7 +190,7 @@ func describe(w io.Writer, service *servingv1.Service, revisions []*revisionDesc
 	return nil
 }

-// Write out main service information. Use colors for major items.
+// WriteCache out main service information. Use colors for major items.


seems to be a wrong search/replace thingy. Same as in the comment below.

rhuss · 2023-09-25T07:55:29Z

pkg/kn/config/config_test.go

@@ -117,7 +117,7 @@ func setupConfig(t *testing.T, configContent string) (string, func()) {
 	// Save old args
 	backupArgs := os.Args

-	// Write out a temporary configContent file
+	// WriteCache out a temporary configContent file


wrong comment

rhuss · 2023-09-25T08:10:23Z

pkg/kn/plugin/context_sharing.go

+	Path string `json:"path,omitempty"`
+
+	// Plugin declares its own manifest to be included in Context Sharing feature
+	HasManifest bool `json:"hasManifest"`


Is this flag really necessary or wouldn't it be just good enough to have the slices below, which then could be also be empty to indicate that the plugin does not support context data sharing.

rhuss · 2023-09-25T08:46:53Z

pkg/kn/plugin/context_sharing.go

+func (c *ContextDataManager) FetchContextData(pluginManager *Manager) context.Context {
+	ctx := context.Background()
+	for _, p := range pluginManager.GetInternalPlugins() {
+		if p.Name() == "kn func" {


I guess this is just a test case for the prototype and will be generalized later on ?

rhuss · 2023-09-25T08:53:08Z

pkg/kn/plugin/context_sharing.go

+	for _, plugin := range pluginManager.GetInternalPlugins() {
+		manifest := &Manifest{}
+		// For the integrity build the same name format as external plugins
+		pluginName := "kn-" + strings.Join(plugin.CommandParts(), "-")


I don't remember whether we haven't added some more sophisticated name handling (e.g. like converting - in command parts to _ before joining them), but I think we should do the same handling here.

rhuss · 2023-09-25T08:54:42Z

pkg/kn/plugin/context_sharing.go

+		// Add manifest to map
+		c.Manifests[pluginName] = *manifest
+
+		if manifest.HasManifest {


as said above, maybe just check if there are keys produced by a plugin, just to get rid of this boolean flag.

You could even just call populateDataKeys as this is a noop if the slices for the keys are empty anyways.

rhuss · 2023-09-25T09:01:37Z

pkg/kn/plugin/context_sharing.go

+	if err != nil {
+		return err
+	}
+	for _, plugin := range plugins {


I would be also cool, to remove plugins from the cache that are not listed anymore. So checking which keys of the c.Manifests[..] have been touched and remove at the end every entry with a key that is neither an internal or external plugin.

pkg/kn/plugin/manager.go

dsimansk · 2023-10-20T10:24:58Z

A corresponding client-pkg with types at least: knative/client-pkg#127.

rhuss

This goes definitely in the write direction. See some comments inline (and the other comments I did before).

rhuss · 2023-10-20T13:59:28Z

cmd/kn/main.go

-
+		if config.GlobalConfig.ContextSharing() {
+			if pwm, ok := plugin.(pluginpkg.PluginWithManifest); ok {
+				data, _ := ctxManager.FetchContextData()


You probably should not ignore the error here. Either remove the error part of the FectContextData() signature or return the error (I hope, the second arg is an error ;-), nevertheless ignore this here has some smell)

rhuss · 2023-10-20T14:01:35Z

pkg/kn/commands/service/describe.go

+					return errors.New("'service describe' requires the service name given as single argument")
+				}
+			} else {
+				serviceName = args[0]


I'm not sure, but I have the feeling that his access pattern will be repeated in several commands (all that require a service name), so maybe we could extract this handling in a utility method. Usually, I wouldn't opt for this, but this looks complex enough to avoid a C&P approach.

rhuss · 2023-10-20T14:12:47Z

Let's get it merged and iterate from there on. Not sure how to preserve the unresolved comments for the next update. But I think they are still kind of valid.

/approve
/lgtm
/hold

knative-prow · 2023-10-20T14:12:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dsimansk, rhuss

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [dsimansk,rhuss]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dsimansk · 2023-10-20T14:35:46Z

/unhold

knative-prow bot requested a review from rhuss August 22, 2023 14:16

dsimansk commented Aug 22, 2023

View reviewed changes

dsimansk commented Aug 24, 2023

View reviewed changes

dsimansk commented Aug 31, 2023

View reviewed changes

dsimansk commented Sep 1, 2023

View reviewed changes

dsimansk force-pushed the pr/context-sharing branch from 03f2293 to 4e2e85f Compare September 21, 2023 13:15

knative-prow-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 21, 2023

knative-prow bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 21, 2023

dsimansk force-pushed the pr/context-sharing branch 2 times, most recently from 63dded1 to c8d0a42 Compare September 21, 2023 13:21

knative-prow-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 21, 2023

dsimansk force-pushed the pr/context-sharing branch from 425df21 to ec469a7 Compare September 24, 2023 19:04

WIP: Context Sharing POC

c0c511c

dsimansk force-pushed the pr/context-sharing branch from ec469a7 to c0c511c Compare September 24, 2023 19:08

knative-prow bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Sep 24, 2023

rhuss reviewed Sep 25, 2023

View reviewed changes

More lazy init

7e5bef5

dsimansk changed the title ~~WIP: Context Sharing POC~~ Context Sharing POC Oct 20, 2023

knative-prow bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 20, 2023

Fix imports

e2d3a49

rhuss reviewed Oct 20, 2023

View reviewed changes

knative-prow bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 20, 2023

knative-prow bot assigned rhuss Oct 20, 2023

knative-prow bot added the lgtm Indicates that a PR is ready to be merged. label Oct 20, 2023

knative-prow bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 20, 2023

knative-prow bot merged commit b658574 into knative:main Oct 20, 2023
22 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Context Sharing POC #1855

Context Sharing POC #1855

dsimansk commented Aug 22, 2023

dsimansk Aug 22, 2023 •

edited

Loading

dsimansk Aug 22, 2023

dsimansk Aug 24, 2023

rhuss commented Aug 25, 2023

rhuss commented Aug 25, 2023

dsimansk commented Aug 31, 2023

dsimansk left a comment

dsimansk Aug 31, 2023

dsimansk Sep 1, 2023

rhuss Sep 4, 2023

dsimansk Sep 1, 2023 •

edited

Loading

rhuss commented Sep 1, 2023

codecov bot commented Sep 14, 2023 •

edited

Loading

dsimansk commented Sep 21, 2023

rhuss left a comment

rhuss Sep 25, 2023

rhuss Sep 25, 2023

rhuss Sep 25, 2023

rhuss Sep 25, 2023

rhuss Sep 25, 2023

rhuss Sep 25, 2023

rhuss Sep 25, 2023

rhuss Sep 25, 2023

rhuss Sep 25, 2023

rhuss Sep 25, 2023

dsimansk commented Oct 20, 2023

rhuss left a comment

rhuss Oct 20, 2023

rhuss Oct 20, 2023

rhuss commented Oct 20, 2023

knative-prow bot commented Oct 20, 2023

dsimansk commented Oct 20, 2023

Context Sharing POC #1855

Context Sharing POC #1855

Conversation

dsimansk commented Aug 22, 2023

Description

Changes

Reference

dsimansk Aug 22, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhuss commented Aug 25, 2023

rhuss commented Aug 25, 2023

dsimansk commented Aug 31, 2023

dsimansk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsimansk Sep 1, 2023 • edited Loading

Choose a reason for hiding this comment

rhuss commented Sep 1, 2023

codecov bot commented Sep 14, 2023 • edited Loading

Codecov Report

dsimansk commented Sep 21, 2023

rhuss left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dsimansk commented Oct 20, 2023

rhuss left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rhuss commented Oct 20, 2023

knative-prow bot commented Oct 20, 2023

dsimansk commented Oct 20, 2023

dsimansk Aug 22, 2023 •

edited

Loading

dsimansk Sep 1, 2023 •

edited

Loading

codecov bot commented Sep 14, 2023 •

edited

Loading