Plug-in System

[benoitf]

This is a follow up on the previous issue #1347

Provide plug-in API

There is a current extension model in Theia powered by inversify allowing to use/extend any extension point available but the current model is conflicting with some Eclipse Che goals.
While this model fit nicely the ISV goals, it may not fit for 'cloud IDE developer'

Eclipse Che Goals

Fast loading Che workspace

In Che, a new IDE needs to be loaded each time we create a new workspace. (If user select java, java extensions needs to be loaded, etc)
So, we need to avoid any build/compile operations that can take time. Loading 10 or 100 extensions should be a fast as possible.

Secure loading

As we need to create/assemble/load a new IDE for each workspace, we always need to have something runnable at the end. If one extension is failing, only this extension should be disabled but you should have all the other extensions available and the main IDE there. No blank screen.
We need to avoid as much as possible network operation as well.

Code isolation

• At runtime, a plugin shouldn't impact performance of the main UI thread.
• A plugin could use any compliant library even if another plugin is using the same library in a different version.

Code isolation example

on web worker and performance we shouldn't use 1 web worker per plugin even if for example firefox limits the number of workers to 512 (can be configured) but too many workers = bad performance
So probably a pool of webworkers (< navigator.hardwareConcurrency ) should be used

POC

example: https://youtu.be/v_a1J_cM5lQ

[AlexTugarev]

Just asking, how would the rendering part of plugins fit into this picture? At some point those plugins want to render something, right?

[svenefftinge]

It would be nice to evaluate whether we could just support the VS Code extension API.

[benoitf]

@AlexTugarev sure it's a question that come quite often. it's flexibility over control. It's where you put the cursor between customization and maintenance.
It's not because there is an API that you can not render anything. If the core API is providing a rendering of image in editor, then the extension can reuse that rendering as well.
But if you're thinking about adding some html popups anywhere on the screen, it can break maintenance to authorize that as for example the underlying layout can be changed and then the plugin is out of date on colors/rendering/etc

[benoitf]

@svenefftinge funny that you add that comment as we were also thinking at a time to have an abstract layer on top vscode extension API allowing vscode plugins to be easily migrated as well. (it may work out of the box for many plugins as well without changes in vscode plugins)

Based on the previous comment, vscode is clearly preventing any UI rendering from extension while theia API could authorize in some way that.
But as it's high level API, they can be both provided.

vscode model also use the lazy loading pattern. (activation events)
For Che, as we build the IDE on demand, it's less a requirement as for example we won't load python plugins if you open a java project, etc.

[akosyakov]

I don't think we can/should run VS Code extensions in web workers:

• they allow any Node.js based dependencies which cannot be used in web workers
• accessing the filesystem, git and so on from the browser is expensive because of the network overhead. I assume that extensions can perform more efficient on the backend and send minimal increments to the frontend to reduce network overhead.

[benoitf]

@akosyakov about the first point, extension can be run on the server side (with threads instead of worker or with libs like https://www.npmjs.com/package/tiny-worker) as we're still sending messages we can use any node dependencies in that case.

But yes obviously it means that we send more messages between client and the server.

[benoitf]

but maybe vscode extension support can be discussed in another issue :-)

[svenefftinge]

but maybe vscode extension support can be discussed in another issue :-)

I would say we either support vscode extension through this or not. It doesn't make sense to have two different sandboxed / high-level extension mechanism, does it?

[svenefftinge]

As you already started prototyping on this (https://github.com/theia-ide/theia/tree/isolated-extension-api), I'd like to get the discussion going on why we just can't support the VS Code extension system instead. It seems to fulfill all the requirements you listed above?

The advantages are:

• ~6000 existing extensions (+ growing fast)
• many developers familiar with that
• lots of articles and documentation
• tools to create and publish them
• solid, thought out & mature API

[benoitf]

I think that some arguments were given by Anton. VScode assume that extension is running close to the client (Desktop IDE). So while you're talking on a big advantage, you may also have a lot of network usage when client and server are separated.
So it may be nice think (BTW you mentioned only advantages and no drawbacks) but philosophy is quite different for cloud IDE where you might want to optimize the network usage.

Also it's difficult to follow a model that is evolving on its side, so you might not be 100% compatible as well. You could have a "emulated vscode layer" with 'try do the best` but you could never assume it's 100% compatible.

[svenefftinge]

Could you be more concrete about your concerns? Which information don't you want to send over the network and why could we never support the API fully?

I think the benefits are compelling so we should have very good reasons to invent our own thing instead. Also please list any drawbacks you see.

[akosyakov]

Is not it our case that we have a powerful scalable backend and potentially limited on resources frontend?

I imagine a worst case as someone who has a budget laptop and wants to use remote workspaces with Theia. Would she be happy to find out that extensions are doing some heavy networking and computations in multiple web workers? How could it be prevented with web workers? How could we control communications? How can we scale it, ask her to buy a new laptop?

Giving that would not it be reasonable to exploit the backend as much as possible and don't overuse the frontend?

VS Code assumes that both frontend and backend are running at the same distance to all heavy resources, as documents, filesystem, git and so on. The most of these resources are on the backend already. We can bring the rest by moving relevant frontend state to the backend via controllable syncing, as @svenefftinge suggested for documents: #1470. By controllable I mean that our code will be responsible for all communications between frontend and backend to keep them low and small.

With having it we are back to VS code case and can run extensions efficient on the backend without misusing frontend resources. Extension developers will be able to use familiar backend apis, as Node.js, and VS Code API (or similar) to access the frontend state from the main process. The latter, being implemented by us, let us as well control communications between extensions and frontend again to keep them low and small.

[benoitf]

I will not on this comment on which API model to stick but more on the global picture. All model can be plugged as it's quite a high level view that I'm showing.

That said, for some case, we require to have plugin running on the client side, for some other cases we can use server side. so instead of choosing for users, they could have the choice. (we may say one is the default like using any nodejs module so being server side plugin)

Note : the same API is exposed for plugins whatever they will run on client’s side (like in worker) or on server side (inside nodejs instance)

For Che, plugins to load will be defined when Theia docker container will start.
Theia container in Che is immutable It means that all users/workspaces will mainly run the same docker image but we need to provide some settings to each container to define which plugins will be used.

Plugins could be provided through an environment variable to the container. (it could use a different mechanism, like providing REST API endpoint to grab it, etc)

Probably, registries will have to be provided to theia container as well.
What is a registry ? it is where we can find the plugins to install

So, providing list of plugins should also provide on which registry they’re located.
Registries could use different pluggable types. It’s part of the registry type to know how to grab plugin.

Example of what could be done for the “bootstrapping of the container”

Provide registries (with json or yaml format)

• Npmjs
  • url: http://npmjs.com
  • type: npm
    -Maven
  • url: http://central.maven.org
  • type: “maven”
  • Github
    • url: http://github.com
    • type: “github”

2. Provide set of plugins

Then plugins could be described by using their registry as prefix.

Env variable PLUGINS: npmjs/my-custom-plugin@2.0.0,github/johndoe/customplugin@1.0.0, maven/org.eclipse.che:che-custom-plugin:1.2.3:zip

So the extension manager is able to collect all plugins that need to be installed, from which registry, and each type of registry provides how to grab resources.

At this point, container knows what are the plugins to install, and how to grab them.

Backend Handling

Here is an example of what could do the theia backend.

Metadata are analyzed by some scanners that can be plugged and contribute to Theia plugin’s metadata available both from frontend and backend extension manager parts.
Resources can be registered as well from a plugin like icons/images

Note: to speed up the install of plugins, all plugins should be “precompiled” if they have external dependencies or static requirements. (This is mainly because the startup of a workspace should be as fast as possible)

Frontend Handling

When the frontend is starting, it should exchange data with backend in order to get what are the plugins to start on browser side and also to get backend plugins commands to register lazily (we register the command but plugin is loaded only when we first call this command)

API

All resources added through high level API (disposable resources) when plugin is loaded should be automatically cleanup if plugin is disabled.
For the other parts, it’s plugin’s responsability (like closing files, closing network connections opened when starting plugin, etc) to release resources when plugin is disabled (callback called on the plugin)

→ Provide a way to refresh the whole frontend/backend if a plugin is disabled if user want to be sure it’s cleanly removed

User experience

For development mode, add a new “tab browser” with a new instance of theia running with the plugin being currently developed. (hosted mode)
Hosted mode will have a dedicated UI/server.
You can code in the first “theia” and you can see the result of plugins in the other “theia”

[benoitf]

@akosyakov @svenefftinge do you have comments on my previous comment ? does it fit well for you on global picture ?

[svenefftinge]

Hey @benoitf,
Could you outline the requirements that lead to the need to have frontend extensions?
I still think it would be best to directly go for the vs code model unless we have concrete reasons not to do it. Maybe we could set up a meeting to talk through the requirements and options?

[benoitf]

yes we could discuss that in a meeting. BTW I don't see why we could not let the user to write code that could run on frontend. (developer may have the choice even if default is server side where you can use all nodejs API)
Even on cloud, it's not because it can scale a lot that you want to run it remotely and that you're granted infinite resources.

Let say I want to implement a monitor or check connectivity from my Theia IDE. I expect that for this design to be able to run 'ping/check' from the browser side, not from the server.

There are many other cases where you want to have plugin executed on frontend but it doesn't mean all of them need to be executed on frontend side as well. (When you design the plugin you could just pickup which side)

Also it is still compliant if someone want to apply vscode mode. In that case these vscode kindof extensions will be executed on the server.

[svenefftinge]

We had a meeting and agreed on the following approach:

• use term 'plug-in' (since 'extension' is already taken)
• go forward in smaller steps inspired by the VS code API
• Eventually try to fully support the VS Code API
• @evidolob will create a new branch, and create a PR with his current work against it, so feedback/discussions on the code level can happen more easily and transparently.

Regarding VS Code API support, @gorkem raised concerns that it will be hard to maintain and achieve 100% compatibility, due to its lack of a real spec. The surface is indeed huge and directly going for that as the ultimate goal would consume many developer hours (and nerves ;-)). So it seems to be better to move there in smaller steps with useful intermediate results.