[PROPOSAL] Introduce an OpenSearch-Dashboards SDK

[dblock]

Is your feature request related to a problem? Please describe.

Around 2015, Kibana started offering a plugin system, but no public plugin APIs. By 2020, before the OpenSearch fork, it has launched the Kibana Development Platform that significantly improved plugin development, including foundational APIs, and runtime isolation. Plugins can register HTTP endpoints and UI applications, query and create back-end data, and provide generic services to other plugins.

Post-fork, the major problems in extensibility that remain unsolved are as follows:

1. Developers of plugins are forced to release new versions of plugins for each patch version. This is terrible developer experience and makes it very difficult for users of OpenSearch Dashboards to upgrade as they must wait for all the plugins they use to become available, which may be never.
2. Developing plugins requires checking out and building a specific version of OpenSearch Dashboards, which is fairly sizable and rather ... specific. For all the runtime isolation and independence of plugins, developers are forced to bring in a very specific version of the kitchen sink.

Describe the solution you'd like

tl;dr An OpenSearch Dashboards SDK, similar to OpenSearch: https://github.com/opensearch-project/opensearch-sdk-java.

Since OpenSearch adopted semver at fork it may seem like we can now relax the exact version check (problem 1), and assume that a plugin developed during 2.3.0 will work with the next release of OpenSearch Dashboards, 2.4.0. This is because per semver, as plugins reach deep into @osd/core/server we can safely assume that those interfaces don't change between 2.3.0 and 2.4.0. Or do they? Well, maintaining semver in all OpenSearch Dashboards interfaces seems quite difficult, and relies on maintainers to be extra careful about changing anything. TODO: @kavilla a specific example pls?

An SDK solves this by providing a logical and physical separation.

An SDK provides a logical separation. The SDK will only contain the set of APIs that need to follow semver, significantly reducing the surface of APIs that OpenSearch Dashboards needs to worry about since plugins only take a dependency on the SDK. Backwards incompatible changes can be immediately caught by testing every API in an already shipped SDK against all released versions of OpenSearch Dashboards (including the next version). Thus, we guarantee backwards and forwards compatibility. With a semver-stable SDK plugins should be able to confidently declare that they work with OpenSearch Dashboards >= 2.3.0 or ~> 2.5. The SDK could provide support for integration testing against those versions. Finally, an SDK can begin selecting common functionality that all plugins may need, such as storing credentials or saving objects, and be strongly opinionated about what constitutes a semver-compatible extension point for OpenSearch Dashboards.

An SDK provides a physical separation. The SDK would be much smaller than Dashboards. Thus, to develop a plugin on top of an SDK, no need to check out and build the kitchen sink, much less a specific version of the kitchen sink. The SDK can be published to npm, and follow its own semver, and document only the public interfaces it contains. It can also implement wrappers that translate for multiple major versions of OpenSearch Dashboards, extending compatibility much further, enabling developers to write plugins once for several versions of OpenSearch. Testing your custom plugin should be done against a released, downloaded, and stable version of OpenSearch Dashboards.

There are downsides to an SDK: it's a standalone additional project. It will have to forward/copy code/interfaces. What else?

Describe alternatives you've considered

1. Relax version checks.
2. Mark extension points clearly and continue relying on humans to follow semver.

Additional context

I propose we call plugins that use the SDK extensions to make it easy to identify those by a set of new capabilities (broad version compatibility), and match how OpenSearch is thinking about extensibility.

The remaining questions are:

1. Plugins can also be extended. There are plugins that are part of OpenSearch dashboards that would need to also publish interfaces into the SDK. And there are plugins that will be using the SDK that will want to publish their own public interfaces and SDK. As a plugin developer I want to be able to say: my plugin needs OpenSearch Dashboards ~> 2.5 and alerting ~> 7.3 (alerting 7.3+ works with any version of OpenSearch Dashboards 2.x).
2. The security model can/should/will be moved behind the SDK and offered by the SDK. Where/how/what?

Some related issues:

• [Research] Inventory existing visualization abstractions and APIs #2584
• yarn opensearch arg to setup dependencies  #2544
• Allow specifying plugin version requirement as semver range elastic/kibana#135781
• [RFC] What type of visualization abstractions do developers need? #2507
• [Plugin as Extension][SPIKE] Runtime as extension #2145
• [DISCUSS] What constitutes a new plugin being a core plugin #2065
• [Docs] Add document on "how to create OSD plugin" in the Developer doc site #2036
• [FEATURE] Dashboards List Extensions #1866
• get current user in plugins #1724
• [Meta] Plugin Decoupling #287
• [Bug]: OpenSearch Dashboards needs to rebuild core every time (make --component work) opensearch-build#606

[kavilla]

Developers of plugins are forced to release new versions of plugins for each patch version. This is terrible developer experience and makes it very difficult for users of OpenSearch Dashboards to upgrade as they must wait for all the plugins they use to become available, which may be never.

They actually don't have to if they set a reserve word in the opensearch_dashboards.json. For the key opensearchDashboardsVersion they can set to opensearchDashboards. And it will work, but devs need to trust that there plugin works.

[ashwin-pc]

@dblock I have a few question's about this approach.

1. How do we want to handle UI extensibility? Currently since there is an expectation that plugins and OSD share a common react dependency and ecosystem, components built directly using React can be injected into the different parts of the exiting app. (Micro frontends maybe?)
2. Do we want to support exchange of API's between plugins? Currently it is possible for plugins that arent a part of the minimal distribution to expose api's that other plugins can optionally consume if they exist. And since almost everything in OpenSearch Dashboards is a plugin, this opens up the door in future for many existing core plugins in OpenSearch Dashboards to be pulled out into its own separate repo in future to minimize the core applications footprint. Not sure how this plays into the extensions project though. But i can also see the two architectures (plugin and SDK) work side by side where extension authors use the SDK and plugin authors continue to build as is.

[dblock]

1. How do we want to handle UI extensibility? Currently since there is an expectation that plugins and OSD share a common react dependency and ecosystem, components built directly using React can be injected into the different parts of the existing app. (Micro frontends maybe?)

This will need a design from people who understand the underlying tech better. UI extension points will need to be documented and follow semver is a given. Should I be able to author react components in a different version of React and Dashboards support that (microfrontends)? Maybe. From the developer point of view in a scenario of alerts/monitors I'd like to be able to create a bar chart visualization in extension A, allow a user to right click on a column, choose "Create Monitor" (coming from extension B), and then see an alert rendered on that visualization (implemented in extension C that displays alerts). I don't know how best to accomplish that.

2. Do we want to support exchange of API's between plugins? Currently it is possible for plugins that arent a part of the minimal distribution to expose api's that other plugins can optionally consume if they exist. And since almost everything in OpenSearch Dashboards is a plugin, this opens up the door in future for many existing core plugins in OpenSearch Dashboards to be pulled out into its own separate repo in future to minimize the core applications footprint. Not sure how this plays into the extensions project though. But i can also see the two architectures (plugin and SDK) work side by side where extension authors use the SDK and plugin authors continue to build as is.

Yes. An extension should be able to expose its own extension points and follow its own semver. An extension consuming that would declare that it needs Dashboards ~> X.Y and Extension ~> Y.Z installed.

[seraphjiang]

for extensibility, I'm looking forward to see SDK extension experience like https://vscode.dev/

[seraphjiang]

1. Relax version checks.

@dblock this is a little bit vague, need clarification

Relax version check from OSD to OS, or release plugin's version check to OSD. or both

[peternied]

There are a couple of areas that I often see from the orbiting the Security space to include in this new design, if we built up these areas we can create better user experiences and unlock plugins for Dashboards we've never seen before.

Ways to know users permissions

There are some ugly user experiences where Dashboards plugins are unaware of what the user's capabilities are. This often manifests in buttons / actions they are shown, but fail after the action is rejected by an OpenSearch API call. While building up this SDK for Dashboards extension there is a opportunity to make sure permissions are accessible and that features added by plugins can express those permissions.

Create rich plugins without a OpenSearch Core plugin

Most of the todays Dashboards plugins that have in a 'read-only' way - there is a backend component that has read/write capabilities but the visualization component don't have a separate data layer. This necessitates that a Dashboards plugins require a OpenSearch Plugin running on the backend - a big adoption bottleneck. Some plugins will absolutely depend on an OpenSearch Plugin, but for visualization focused experiences its not necessary with data storage is easy and trustworthy.

Secure all metadata

I am guessing on the utility of this one - today this isolation is only possible with a side-by-side OpenSearch plugin

Today dashboards data is all stored in an index that is fully accessible. Consider sensitive annotations like 'security breach detected' that should have limited read or update access. More than just moving to a hidden index, Dashboard Extension shouldn't be able to read everything by default, its should have limits to what it can do.

A simpler example, user specific settings it could be nice to store them on Dashboards so if you changed browser windows or computers the settings followed you.

[zengyan-amazon]

@dblock if we want to address the build overhead due to versioning, do we really need to build a sophisticated extension framework? As @kavilla mentioned in this thread, developers can specify their plugin version to match building OpenSearch Dashboards version in the plugin manifest file, so that they can just build and release new version without making code change.

Regarding the release process, shall we consider doing something like merging all plugins into one single OpenSearch Dashboards repository, so that all build and release can be unified into the same place? That mono-repo approach is what Elasticsearch and Kibana have been doing for years. They even have their commercial feature and OSS feature in the same repo and control what to be released in their build process.
Prior to fork, we only have several plugins, and have no control of the core repos, so each plugin needs a separate repository. Now we forked from their repo, we may consider to merge our plugins into the core software repo, so that all development, testing and release can be unified in the mono-repo.

[kavilla]

shall we consider doing something like merging all plugins into one single OpenSearch Dashboards repository

I think one of the major goals of this SDK is the empower extension developers more. The ideal state being we have a bunch of community extensions not under the OpenSearch project. So if abstract stuff into the SDK and verify compatibility with the SDK, then we and extension developers have more insight about versioning and we can feel more confident about breaking changes.

[zengyan-amazon]

If we want to solve the version compatibility problem, I think we need to ensure complete functional test being executed for all plugins with each new version of OpenSearch Dashboards, to ensure all functions are working as expected. A mono-repo can solve this in a pretty straight forward way. Or with separate repositories, we can built some sort of notification mechanism so that all changes to OpenSearch Dashboards repo will generate a notification event. Then all plugins can subscribe to that event and execute their tests to ensure the plugin works with new OpenSearch Dashboards changes. Community 3rd party plugin developers can also subscribe to that event and automate their tests with new Dashboards if they want.

With regard to the SDK and extensions, does the extensions run as a separate process out side of OpenSearch Dashboards process or it still runs in the Dashboards process like plugins today?

[davidlago]

Secure all metadata

I am guessing on the utility of this one - today this isolation is only possible with a side-by-side OpenSearch plugin

Today dashboards data is all stored in an index that is fully accessible. Consider sensitive annotations like 'security breach detected' that should have limited read or update access. More than just moving to a hidden index, Dashboard Extension shouldn't be able to read everything by default, its should have limits to what it can do.

👍 to this one. Right now the saved objects construct in Dashboards is way too coupled with OpenSearch. As we think of the future of sharing with Dashboards (i.e. what today is somewhat fulfilled by Tenants, cc @shanilpa), we will have to provide data isolation purely inside of the Dashboards application (decoupling the metadata store), and we should account for extensions in doing so, being able to give them access to some metadata (i.e. "sharing" with the extension).

[saratvemulapalli]

Thanks @dblock for the proposal.
I love the idea and always onboard with lowering the entry bar for developers to hop on to Dashboards platform.

1. Developers of plugins are forced to release new versions of plugins for each patch version. This is terrible developer experience and makes it very difficult for users of OpenSearch Dashboards to upgrade as they must wait for all the plugins they use to become available, which may be never.

2. Developing plugins requires checking out and building a specific version of OpenSearch Dashboards, which is fairly sizable and rather ... specific. For all the runtime isolation and independence of plugins, developers are forced to bring in a very specific version of the kitchen sink.

💯 We should strive to expose simple interfaces for developers to consume while not having to worry about versions of Dashboards.
As we are working through similar problems on OpenSearch, the time I've spent(probably very little) on Dashboards I see few more problems I believe in solving as well:

• Library dependency tree is global. I would really love to run 1000's of plugins/extensions and this architecture prevents me from running them and also lowers the velocity of Dashboards releases as we have to take care of all the plugins in the distribution.
• As an operator I would love to install/modify/remove plugins without taking down Dashboards node which provides a great experience for upgrading individual plugins/extensions and not really worry about Dashboards versioning.
• And the last one, a plugin/extension should not really take down Dashboards, they should able be run in isolated environment and not really impact the cluster.

I love where we are starting and excited to see this rolling.
Folks working on OpenSearch extensions your thoughts/learnings so far will be really helpful. @dbwiddis @owaiskazi19 @joshpalis @ryanbogan

[kavilla]

Related #2880 to interfaces that plugins utilized.