Separate public and private API for plugins

[timroes]

In the current development state of the plugin system, you have to include several modules to create some basic functionality, e.g. the ui/filter_manager module if you want to add a filter by clicking in your custom visualization. Basically you could require every module.

It is (at the moment) pretty unclear which of the modules are meant for external usage in plugins (e.g. I would consider the filter_manager to be such a module) and which are not meant for usage in plugins, because their API might change during releases often/a lot.

Are there any plans on how this might be solved to make a clear separation on what modules plugins can rely and on which they shouldn't?

My suggestion would be: Don't just try to solve that by documentation, but perhaps make a new folder src/api in which all modules that are considered public will have a wrapper. I.e. a user plugin require('api/filter_manager'); that module basically just delegates everything to ui/filter_manager. It would be pretty clear (and can be documented), that user plugins should only rely on modules in the api path. Also, if the ui/filter_manager needs updating at some point the api/filter_manager could still offer the old interface for backward compatibility reasons (or there could be versioned modules like api/filter_manager/v1 which will always have the same interface).

[rashidkpc]

The answer here is that there are no public plugin APIs right now. We will get to that point, but we're still determining the components that should be public and should be private.

[rashidkpc]

See here: https://github.com/elastic/kibana/blob/master/src/plugins/README.txt

Despite that file saying tickets asking about plugins will be closed, I'm going to leave this open to foster discussion of the eventual plugin API.

[timroes]

Since 4.2.0 will introduce (afaik) the installing of plugins from remote sources, is there also any estimated target version for making a public plugin API?

[rashidkpc]

Currently no, we're still sorting out what we need to make available. Feel free to add ideas to this thread

[spalger]

@timroes I expect that we will "expose" the api in pieces using, like you said, api-specific modules.

We don't have any official plans for how this will look in code though, perhaps it's a good time to pull together some dreamcode.

[timroes]

As heard from ElasticON Tour in Munich there are now plans for supporting development of custom plugins for beginning of 2016. So I guess there have been some thoughts about how to separate private and public APIs, bundling libraries, etc.

For the sake of documentation would you care to elaborate on your plans about how you want to handle separation and how that public API might look like? Is this ticket the correct place for leaving suggestions what parts of Kibana I wish to access from plugins and I would like to see in the public API.

Have there been any considerations for versioned APIs, so Kibana plugins are not bound strictly to the underlying Kibana version?

[timroes]

/cc @rashidkpc @spalger

I really would like your feedback about that topic.

[spalger]

@timroes I like your suggestion of api/* module patterns for separating "public" and "internal" APIs, but we don't have any right now. We are now focusing on internal plugins (like marvel and sense).

As for versioning, we will only make stable and backwards compatible changes to public APIs in minor/patch versions. In major versions these APIs may get breaking changes.

I appreciate the explicitness of something like api/*/v1, but we just aren't ready to commit to supporting multiple version of the same api at once. It may be the best way to go though, we will see.

If you want to being writing plugins now I recommend that you use your own dependencies where possible. This should prevent us from breaking things unintentionally.

[timroes]

@spalger thanks for the informations.

The problem with me writing custom plugins at the moment (and we get several client requests for that) is that most of them would require internal APIs (which cannot be satisfied by external libraries), e.g. access to the currently set filters, the current ES query, etc.

All these are available via some kind of code, e.g. via the es_response variable in the visualizations controller (currently not having my code here, so guessing the variable name). I am just very unsure if it is a great idea, writing plugins against these, since there is no guarantee (though very likely in case of some of the APIs) that they will still be the same in the next minor kibana version, since it's not official public API.

So if we offer a client kibana plugins it only can go with the warning: they might be fixed to your current version exactly. Every patch version might break the plugin, since we might have used something that wasn't meant to be used and has been refactored in a patch version. This let's us end up in a kind of unsecure situation (kind of similar to ES plugin development). Since I would really like to be able to offer a client a great plugin, that we can kind of guarantee it will run for some versions (hopefully years), I am very interested in the progress in this and bother often here, to get some progress.

Is the separation of APIs isn't something to come in the near future, but you anyway would like to see third party plugins? Or is a public API planned for near future releases, or isn't it planned and you actually don't want to see third party plugins at the current state of art?

If you could give me some heads up about that it would be nice, because it might be, that I have a nearly finished tutorial on how to write Kibana plugins for the current (not explicit public) API on hold and just not publishing, because I thought there might be an explicit public API in the near future. :-)

Thanks so much for an answer!

[spalger]

If you start developing plugins for clients then you will likely have to update those plugins with each release of Kibana (as you guessed).

We are also unlikely to release a public API that would give you access to all of the things you seek. Certain API's will be exposed in the near future, but it doesn't sound like the APIs you are looking for will be available soon.

Sorry, hope this is helpful though.

[jerrac]

Well, I had the wonderful idea of writing a Kibana plugin this morning, then I stumbled onto this issue while trying to figure out how. Is there any progress on making a plugin system work for Kibana?

Would a description of what I want to do help? Or would it just clutter the discussion?

[mkozjak]

+1. Are there any news on this?

[havidarou]

Any news on this?

[timroes]

@mkozjak @havidarou There is currently an ongoing effort to build a new underlying platform for Kibana, that will also offer proper interfaces and exports from plugins. You can find a detailed explanation in #9675

[timroes]

Kibana has come an incredible way since I wrote that issue. And with the near finalization of the Kibana platform the principles outlined in this issues has come into reality now. Plugins have a clear contract they expose and importing from other Plugins is limited to specific API and pathes.

Thanks everyone for the effort and journey that brought us here. Thanks @joshdover and everyone involved.

/closing my oldest issue at this time