Use the Canvas rendering pipeline/expressions for all visualizations

[timroes]

To align rendering between Canvas and Kibana and to create a future proof architecture for all visualizations, we are planning to switch to the Canvas rendering pipeline for all visualizations.

That would mean, that all regular visualizations you do in Kibana are backed by a pipeline script.

To switch over we'll need to provide the following functions to the pipeline:

• kibana (or named differently) to provide filters and queries from the Kibana context
• esaggs to load aggregated data and provide it in a tabular format
• visualization functions for each of our charts.

6.3

• Remove angular from Schemas (removing angular from Schema and AggParams #17565)
• Remove angular from fieldFormatts (removes angular from fieldFormatts #17581)
• Remove angular from AggTypes (remove angular from AggTypes #17681)
• Remove angular from AggConfigs (remove angular from AggConfigs #17682) (remove angular from AggConfig #17668)
• Remove vis dependency from Tabify (removes angular and vis dependency from tabify #17566)

6.4

• PR Pass context down to visualize (explicitly passing filters and query to visualize #19172)
• PR Remove vis dependency from AggConfigs (adding indexPattern to agg so its not accessed from vis.indexPattern #20491)
• PR Reactify (reactifying visualization #16425)
• PR Implement new Inspector (Implement new inspector #16185)
• Reactify

6.5

• Refactor tabify to produce future tabular format (simplified tabify #19061)
• Rewrite point series response handler to be based on top of tabify (simplified tabify #19061)
• remove dashboardContext
• move inspector from Vis
• cleanup vis.API.events

6.6

• Remove IndexPatterns and queryFilter from Vis
• move canvas interpreter to OSS

7.0 alpha

• Write additional pipeline functions:
  • Kibana and KibanaContext (kibana & kibana_context)
  • Aggregate (esaggs)
  • timelion, tsvb, vega proxy functions
  • functions for every visualization
• change how we handle fieldFormatters ([WIP] visualizations field formatting refactoring #26951)
• change how we handle schemas ([WIP] visualizations field formatting refactoring #26951)

7.0

• Using canvas pipeline to render visualizations (Use the Canvas rendering pipeline/expressions for all visualizations #19813)
• Toggle for switching over the pipeline data loader (Toggle between Expression Pipeline & Visualize Data loader #29874) @lukeelmers
• Remove websockets (Remove WebSockets from Canvas expressions interpreter #29792) @chrisdavies
• Explore reducing Canvas bundle size
• Develop test plan for initial cutover (Expression Language Test Plan (Short Term) #30279)

7.2

• Improve performance by using batching (Optimize Canvas REST calls via batching #29847)
• Resolve issue with importing disabled plugins (Interpreter plugin imports from other plugins #29324) @ppisljar
• Strategy for performance benchmarking
• Create pipeline testing framework (Testing framework for new pipeline #22139)
• Broader test coverage in the interpreter & surrounding infra (core interpreter functions - unit tests #32377, Add unit tests for embedded_visualize_handler #32657)
• Move more parts of the infrastructure over to TypeScript
• Documentation (inline), and README with whole system overview
• Toggle rendering pipeline/expressions back on
• Replace canvas specific naming with generic interpreter naming
• add interpreter uiExports (adding interpreter uiExports #33391)

7.x

Make Kibana visualizations useable inside Canvas:

• change how we handle filters
• move to canvas datatable table format (remove kibana_table)
• move to canvas context (remove kibana_context)
• write visualize render function for canvas (related: Embed Kibana visualizations in Canvas #23035)
• improve semantics of all visualize functions
• Convert to New Platform plugin system
  • Explore server side migration first, since support is already there for it

Refactoring & cleanup

• Split Registry code into it's own plugin
• Cleanup after removing the visualize toggle
• Move stateful registries out of kbn-interpreter - either keep it purely a utils package or move entirely to a plugin.
• Cleanup after new Table Visualization is complete (Rewrite table visualization in React/EUI #16639)
• Cleanup after new EUI chart components are complete
• Refactor Input Controls
• ~Split uiState into two new visStates (Split visualization's uiState into two states #21644)
• Broaden test coverage on pipeline utilities & refactor for greater stability (Add unit tests for getSchema pipeline utility #33196)

Functionality improvements & enhancements

• Refactor Inspector to work with canvas pipeline (debugging, server side support, ...)

8.0

• Implement new saved object format (New saved object format for visualize #21645)
• Migrate old saved object format to new format
• New loader API (New Visual loader API #25374)

[ppisljar]

reopening since #25996 didn't address all of the topics mentioned above.

[epixa]

@timroes Can you describe at this point what you're currently targeting for 7.0? Specifically, will visualizations now be pipeline based, and will there be a required or automatic migration for existing visualizations to convert them over to the new pipeline-based format?

[ppisljar]

we build expression out of our current savedobject/Vis on-the-fly
we still use old saved object/old vis for the editors, once you want to render it we convert it to expression and execute that expression with the interpreter.

[ppisljar]

closing this as we have separate issues tracing remaining work