Remove legacy platform support

[joshdover]

We have decided to remove legacy plugin support in a future 7.x minor version.

Plan (WIP - more to be fleshed out)

Blockers for removing client-side plugin support

• Migrate remaining 1st party plugins to NP (owned by other teams)
  • Timelion
  • Management
  • Maps
  • Status Page (Migrate Status Page UI #67979)
• ui/styles (Migrate ui/styles to New Platform #58369)

Blockers for removing server-side plugin support

• Migrate remaining 1st party plugins to NP
  • Stack Monitoring (has small piece left blocked by Migrate status service and status page to New Platform #41983)
  • Security remove legacy security plugin #75648
  • Deprecated import/export API in legacy kibana plugin
  • Beats Management Migrate Beats Management server-side #67980
  • Dashboard-only mode
• remove legacy plugin discovery Remove legacy plugin discovery #71927

Blockers for all legacy being gone

Migrate core functionality out of src/legacy

• UiSettings injected via legacy metadata (Migrate injected UiSettings from legacyMetadata #22779)
• i18n (Migrate i18n server part #50647)
• UiSettings definitions (Migrate uiSettings defintions to NP #59755)
• bootstrap.js endpoint (Migrate the bootstrap.js endpoint to Kibana Platform #54375)
• utils/streams (Move utils/streams out of legacy #54128)
• CSP config (Migrate CSP configuration  #50644)
• Keystore src/legacy/server/keystore
• PID file (Migrate PID file management to core #77019)
• Status HTTP API (Migrate status service and status page to New Platform #41983)
• Optimize mixin (Migrate optimize mixin to KP #77020)
• Remaining config schema src/legacy/server/config/schema.js
• Legacy types in KP contracts Remove the legacy types from the platform public contract #77997
• Remove injectedVars

Original Discussion

With the Kibana Platform nearing completion, I'd like to open the discussion about the timing of when we should remove support for legacy plugins.

At first glance, it may seem appropriate to wait until 8.0 to remove support for legacy plugins since this is definitely a major breaking change for 3rd party plugins. However, during the migration of 1st party code, we have not maintained great support for some of the legacy extension points in Kibana. This has caused many 3rd party plugins to break on newer versions of Kibana in the 7.x series.

If most plugins are already broken and need to be re-written to work in the Kibana Platform, it's a bit hard to justify maintaining support for legacy plugins after 7.8 (when all 1st party plugins are targeted to be completely migrated over).

In addition, there are a number of issues that can be resolved once the legacy system is removed:

• The old optimizer can be deleted (superseded by @kbn/optimizer)
  • This improves the experience of administering Kibana with custom plugins because UI bundles won't need to be built at install time.
• Upgrading Hapi.js will become much easier
  • Once all plugins are using Core's HttpService abstraction, upgrading Hapi is much simplified. We're long overdue on an upgrade here.
• Removal of a lot of legacy cruft / code
  • There may not be much direct benefit here other than a simpler codebase for new developers to ramp up on?

Open Questions

• Which legacy extension points are easy enough to maintain support for post-migration?
  • @ppisljar @flash1293 Do either of you have a good grasp on which extension points still work (and will or could continue working after migration)? By extension points, I'm thinking things like vis types, filter manager, timepicker, kbn_top_nav, etc.
• How many legacy plugins in the community have actually kept up with recent Kibana releases?
  • @timroes I think you have the best grasp of the plugin dev community. Do you have a good idea of how many plugins have been keeping up lately and why or why not?
• Which legacy extension points are most commonly used by 3rd parties and customers?
  • This may be hard to audit, but with the answer to the question above, a good starting place could be looking at actively maintained community plugins. I will also track down someone in Product or Support who may have information about plugins developed by customers (internal to their orgs) and the common reasons these plugins are built for.
• Are there other concerns with removing legacy plugin support?

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[flash1293]

@joshdover For the Kibana App extension points so far we moved them to shimmed/NP plugins and documented in dev docs. It would be possible to add a layer providing ~ 90% backward compatibility, but so far we didn't take care of that. 100% backward compatibility will be difficult because the angular injection mechanism exposed a lot of stuff people could rely on - I don't have a good idea what happened on that level. If we exclude the angular injections (which are probably broken really bad anyway), I would say maintaining the legacy platform isn't worth it because in most cases the switch to the new apis is rather easy (they kept their general shape, just got moved).

[ruflin]

There is Beats Central Management (beta) which will be replaced by our Fleet effort. BCM is still on the old platform. We can't remove it before 8.0 as this is a breaking change and at the same time we would not prefer to touch it.

[mshustov]

For the Kibana App extension points so far we moved them to shimmed/NP plugins and documented in dev docs.

Would it alleviate the plugin authors' frustration if we collect them in one place and create something similar to the migration guide?

BCM is still on the old platform.

We need to review what extension points BCM uses. There is a chance that we have to update code to keep it in the working state. @joshdover is this list complete #56349 ?

[joshdover]

We need to review what extension points BCM uses. There is a chance that we have to update code to keep it in the working state. @joshdover is this list complete #56349 ?

It does not include BCM yet. I will do an audit here soon to see what we need to do there.

[jinmu03]

@spalger @alexfrancoeur If you happen to have some information about 3rd party plugins, please comment here

[joshdover]

One idea that has come up is to add a config option or CLI flag to run Kibana with the legacy system disabled, meaning no legacy server code is started and no legacy plugin bundles are loaded. While this wouldn't enable us to start removing code, we could begin testing an approximation of the legacy platform being removed.

[joshdover]

I audited the Beats Central Management plugin for migration. While I don't think it's very much work, I am afraid of breaking it (particularly on the backend) due to the very low test coverage.

Frontend

No uiExport dependencies.

Legacy modules:

• ui/autoload - obsolete, can be removed
• ui/chrome - 2 modules
  • chrome.breadcrumbs
  • chrome.getXsrfToken
  • chrome.getBasePath
  • chrome.getKibanaVersion
• ui/routes - 1 module
  • used to hack into angular router for management route
• ui/management - 1 module
  • used to register management section menu
• ui/modules - 1 module
  • used to get a string from itself? not sure why that is necessary

Backend

• Uses the framework adapter boilerplate that is used by Infra, SIEM, etc. Has a few layers that will need to be adapted to make it work correctly, but we may be able to steal from prior migrations to the other plugins.
• None of the routes have tests (and all of the business logic is in the routes)

Legacy APIs in use:

• server.plugins.xpack_main
• server.plugins.elasticsearch (including status API)
• server.log
• server.config
• server.plugins.security.getUser
• server.inject (only used by 4 tests)

[flash1293]

@restrry

Would it alleviate the plugin authors' frustration if we collect them in one place and create something similar to the migration guide?

I'm not sure whether adding another source of truth would be the right thing. My suggestion: Maintaining dev docs, adding changes to the existing migration guide (already made changes should already be listed there: https://github.com/elastic/kibana/blob/master/src/core/MIGRATION.md#plugins-for-shared-application-services ) and (there's still work to be done here) adding example applications showing how to consume the new APIs (e.g. for visualizations, doc views and so on). At least for me it feels like giving the developer an actually working starting point helps a lot, plus we can test those in CI (which isn't the case for guides which can become stale rather quickly). Wdyt?

[mshustov]

(there's still work to be done here) adding example applications showing how to consume the new APIs (e.g. for visualizations, doc views and so on). At least for me it feels like giving the developer an actually working starting point helps a lot, plus we can test those in CI (which isn't the case for guides which can become stale rather quickly). Wdyt?

Sounds good. Just to note: there is a separate doc to give migration examples https://github.com/elastic/kibana/blob/master/src/core/MIGRATION_EXAMPLES.md
You can extend it with your guidance and/or reference your tests.

One idea that has come up is to add a config option or CLI flag to run Kibana with the legacy system disabled, meaning no legacy server code is started and no legacy plugin bundles are loaded. While this wouldn't enable us to start removing code, we could begin testing an approximation of the legacy platform being removed.

On the one hand, it would simplify the transition: for example, we can run FTR for already migrated modules in this mode. OTOH it will introduce an additional mode that we have to test somehow because Kibana still has to work with and without enabled legacy plugins.

[flash1293]

@joshdover I initially didn't think about it because I focussed on 3rd party integrations / extension points, but there is also the Timelion app and the "dashboard mode" functionality maintained by Kibana app. Those were scheduled to be removed with 8.0, so we planned on not migrating them at all. We would like to keep them running in the remaining 7.x releases though (to prevent breaking changes), preferably without migrating.

There's also the import/export dashboard api (also planned to be left behind with 7.x) we have to consider - not sure who would own the migration though.

Dashboard import/export api

Server side: Thin wrapper around saved objects client, doesn't look like much work

No client side

Dashboard mode

Server side: Intercepting incoming requests and preventing the actual handler from being invoked - not sure how that works in the NP, but there probably is a way to do it.

Client side: Easy to migrate, basically rendering the dashboard app with a flag

Timelion app

No server side

Client side: Angular app still rendering using the global angular. The local angular approach is possible but we probably will have to inline a lot of legacy stuff into the plugin to make it work. Non-trivial but also not a ton of work.

[timroes]

@joshdover sorry for the late reply, I just now saw your question.

How many legacy plugins in the community have actually kept up with recent Kibana releases?
@timroes I think you have the best grasp of the plugin dev community. Do you have a good idea of how many plugins have been keeping up lately and why or why not?

There are currently not many plugins keeping up to date quickly with Kibana releases. I think the most notable one might be the enhanced table plugin by @fbaligand, who usually does a great job making it work usually only days or weeks after a release. I would say from the plugins we list in the known plugins section, most likely 80% don't even work with 7.x at all, and of those 20% who do, most are on a rather early version of 7.x.

There are a couple of plugins maintained by Elastic employees (e.g. the swimlane vis), which are also more or less up to date usually.

Given the why? I think it's what you'd suspect. The amount of releases and the amount of changes in each release is just too high, to keep up for most people who develop those open plugins in their spare time. Sometimes our APIs even break in a way, that you cannot be done with a "simple change", but need to significantly refactor some features or get rid of them (ask Fabien about total count in datatable ;) ).

In my experience, the higher amount of maintained plugins are the ones developed internally at companies. We have a couple of users who extend and customize Kibana to their need. They have the advantage of not needing or striving towards supporting all versions, but only a couple of them they upgrade to, and can usually time the upgrade and development work together.

Which legacy extension points are most commonly used by 3rd parties and customers?
This may be hard to audit, but with the answer to the question above, a good starting place could be looking at actively maintained community plugins. I will also track down someone in Product or Support who may have information about plugins developed by customers (internal to their orgs) and the common reasons these plugins are built for.

I think the most common extension point I've seen used (for public and private plugins) is the extension point of creating custom visualization types. That's why we tried to keep this more stable then other extension points for a long time (though with the change to expressions and general refactoring required also besides Kibana Platform work), we needed to introduce several breaking changes also there.

I think the 2nd most common "extension point" I've seen used are actually "hacks", for a wide variety of use-cases, often adjustments around look and feel, sometimes minor interaction changes, sometimes building features that should plugin into other Kibana features that are not extensible.

[joshdover]

Thanks all for the feedback and information here, super helpful! 🎉

Update on the Platform Team's thinking on this issue:

• We're not committing to removing legacy plugin support in 7.x yet, but we do plan for full removal in 8.0.
• We will explore adding a config flag for disabling legacy plugin support for testing purposes, once all 1st party plugins have been migrated (ignoring the ones that are being deprecated/removed like Beats CM and Timelion). This should de-risk the removal of the legacy plugin system earlier.
• We are reserving the right to remove this in 7.x once we get further into the 7.x series. This decision will be based on:
  1. The effort of migrating any 1st party plugins that were not planned to be migrated (again, like Beats CM and Timelion);
  2. Feedback received from 3rd party developers and consultants; and
  3. The maintenance burden of support legacy plugins in 7.last during the long-term support period. We do fear this maintenance burden may be high and will make backporting to from 8.x to 7.last much more complex and error-prone.

[joshdover]

One of the "features" of completing the new platform migration is the elimination of the full-page refresh between applications in the UI. For the plugins that are not planning to migrate (due to removal in 8.0), I think we should still consider migrating at least their UI to eliminate the full page refresh.

• Beats Central Management: should be a fairly trivial move, only uses 2-3 legacy APIs in the frontend and it's all in React. I think this one is most important because without migrating this, we cannot migrate the rest of the Management app, meaning we'll always have full-page refreshes when going to Management until 8.0
• Dashboard Only Mode: should be easy to move, but probably doesn't make any sense because users of this feature cannot navigate to other apps anyways
• Timelion: Sounds like more work, may or may not be worth the effort depending on usage.

[flash1293]

@joshdover About timelion usage - I just checked telemetry and ~99% of reporting clusters have 0 saved timelion sheets (this number is not shared with timelion visualizations). I don't have a strong opinion on either way here.

[flash1293]

Another thought about keeping legacy platform around - it would give people more time to migrate their plugins if they rely on a global angular that is bootstrapped for them.

[fbaligand]

Really great to learn that in v8.0, there will be no more full page refresh when we navigate between applications!

[fbaligand]

To answer one of your questions, on enhanced table plugin, I would say that I use :
Vis types, filter manager, build pipeline expression

[fbaligand]

As a community plugin developer, I’m very sad to hear that...

[timroes]

@fbaligand Are you seeing larger problems converting your plugin to Kibana platform (expect of course the time it will need to do so)? From 7.9 onwards pretty much everything will be in New Platform, and from 7.8 onwards I think there should be no service left you're using, that is not consumable from the Kibana Platform. Let me know if there's anything I can support you with.

[fbaligand]

Hi @timroes ,

I think the most critical part I use is a custom request/response handler coupled with a custom expression function. Could you confirm that it will still be available in new platform?

Then, currently, I try to make enhanced table plugin compatible with kibana 7.7. And the most annoying part is that visEditor does not support anymore a angularjs directive. If you know a way to encapsulate angularjs directive into visEditor, I’m strongly interested!

[timroes]

Hi @fbaligand,

while moving the code to the new platform, you should have the same capabilities then in the legacy world. I am not aware that we removed custom request/response handlers in new platform (if it would have been removed as part of API cleanup, it should have been removed in both legacy and Kibana Platform usually). But maybe @ppisljar has more insights here.

But beside moving to new platform we're also cleaning up the APIs a lot and especially are moving away from Angular, since we're removing Angular altogether in Kibana. So this should be unrelated to you moving to new platform or not, you won't be able to use Angular via any of our APIs in the future directly, but if you want to use Angular need to bootstrap it manually inside your plugin. We're doing this in a couple of places inside Kibana, where we haven't succeeded removing it all yet, e.g. you can look at the get_inner_angular file of the data table visualization, which still uses Angular and needs to instantiate it itself. But since we converted all of the default editor code to React, we don't have any example of how to use angular in an editor component. You'd basically also need to bootstrap Angular manually, and render your Angular component manually into a specific part of the DOM.

Cheers,
Tim

[fbaligand]

Hi @timroes,

I know the get_inner_angular file. I use it since Kibana 7.6 to render visualization (without it, it would do not work). You can see it there:
https://github.com/fbaligand/kibana-enhanced-table/blob/master/public/get_inner_angular.ts

But in Kibana 7.7, my problem is on visEditor. I mean editorConfig.optionsTemplate in visTypeDefinition.
And despite my multiple tries, I don't arrive to apply same thing or similar.
visualization entry in visTypeDefinition requires a class with render method:
https://github.com/fbaligand/kibana-enhanced-table/blob/master/public/vis_controller.ts

But editorConfig.optionsTemplate requires a TSX function. Like this:
https://github.com/elastic/kibana/blob/7.7/src/legacy/core_plugins/vis_type_table/public/components/table_vis_options.tsx

So I can't use the same structure (I tried but without success).

So my question is: how to inject a AngularJS directive wrapper into editorConfig.optionsTemplate?

[joshdover]

Hey y'all, could I ask that we move this Q/A discussion to discuss.elastic.co? This issue is intended to discuss the plan for removing legacy plugin support and is creating unnecessary pings to everyone on this thread.

[joshdover]

This was completed in 7.9