Official Plugin API

[hiddeninthesand]

Your use case

What would you like to do?

An official plugin/modification API similar to BetterDiscord which would allow expanding the functionality of Element in a much more dynamic way than committing directly to the project.

Why would you like to do it?

Electron was built specifically to be easily modifiable by the user, ever since it was Atom Shell, and eventually renamed to Electron. Many other projects, rather officially or not, make good use of this, such as previously mentioned BetterDiscord. Many features that some people may want are rather too niche or not worth the Element developers' time to justify adding to the main Element client, such as automatic message translation, advanced GUIs for search and room management, or a reimplementation of Discord's emoji system (all notable BD plugins). An upside that may be more important to the Element developers is that if this catches on, it'd be an easy way to gauge what features people do want added to the main project via plugin popularity.

How would you like to achieve it?

Possibly the same way BetterDiscord does it, an injector with plugin script files. If BD is anything to go by, if you give people a way to modify a client, they will modify a client.

Have you considered any alternatives?

There are really only three alternatives to this, and all of them are less than ideal.

Unofficial Injector

The best of these would be to make an unofficial plugin injector, and while that is perfectly functional, especially since as an open source project I imagine the developers would be more likely to play ball with the maintainers of an unofficial injector to ensure stability, this would guarantee some loss of stability and security. This specific example is not a bad alternative, but it's certainly not the best one.

Contributing directly to the Element project

This comes with many more issues, including but not limited to being more difficult to create (as it would obviously come with more stringent security and stability criteria than a user modification), being less likely to be approved at all (too niche for the main project/out of scope), or simply because it'd make the project bigger than it has to be. Not ideal for users, developers, or the Element project maintainers.

Forking Element

The worst option of them all, as it'd mean minor changes would be forced to be maintained on an entirely separate fork of Element, with no upstream improvements likely to come, and being even more difficult for a would-be plugin developer or plugin user than the other two. It's very rare that someone who wants to modify a client would prefer to have to maintain their own version of said client.

Additional context

This issue is related to #15650, which was previously closed as the specific examples the OP submitted could be done using an alternative (a local service). I'd like to make clear that the specific examples I've listed are not the only reason I'd like such a thing to exist, and if you require more examples there's an official list of BetterDiscord plugins that contain popularity statistics to further my case here.

[sweidac]

I would love to see this feature. Our use-case would be a plugin for the company which could e.g. generate links to our Jitsi server, or insert some funny GIF sourced from some company insider jokes.
I just want to emphasize the point that such things are out-of-scope of the main repo.

[t3chguy]

@sweidac both of those can be done using Widget integrations and Bots already today.

[sweidac]

Sounds nice @t3chguy! Unfortunately I couldn't find documentation about Widget's after excessive querying the web (my fault most probably). Can you provide a link to some resources?

[0xdevalias]

both of those can be done using Widget integrations and Bots already today.

This was my thought as well.

Unfortunately I couldn't find documentation about Widget's after excessive querying the web

I've also tried a few times to find good info/resources on this, and I keep struggling to do so. Would love to see the official/comprehensive/etc docs for it!

The best I've found (that I can remember off the top of my head) has just been searching the issues here for 'widget' and combing through the disparate threads to try and get snippets of info:

• https://github.com/vector-im/element-web/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+widget

Edit: Based on that, here are a few relevant snippets i've (re-)discovered just now:

• Matrix Widget API v2 matrix-org/matrix-spec-proposals#3803
• Add support for m.widget (spec) widgets #13111
• Optimized widget integration and user experience element-meta#1381
• Optimize widget discoverability #15737
• Add some very thin UI that allows adding and renaming custom widgets #21219
• Add information about widget author and url to approve widget permissions dialog #20953
• Clicking "Cancel" on the integration manager's ToS spawns a never-loading modal #19011
• UIFeature.widgets flag breaks Jitsi calls #18055
  • https://github.com/vector-im/element-web/blob/develop/docs/config.md#ui-features
  • https://github.com/vector-im/element-web/blob/develop/docs/config.md#voip--jitsi-calls
  • https://github.com/vector-im/element-web/blob/develop/docs/config.md#integration-managers

    • Integration managers are embedded applications within Element to help the user configure bots, bridges, and widgets. An integration manager is a separate piece of software not typically available with your homeserver.

• Widgets use undocumented $theme variable at runtime #20917
  • MSC2873: Identifying clients and user settings in widgets matrix-org/matrix-spec-proposals#2873
• Standalone widget renderer element-meta#1653
  • [WIP] Theoretical concept of a "thin widget wrapper" for popout-style widgets #16122
• custom widgets need to support custom names to differentiate them from each other element-meta#1270

  • For the icon you can specify avatar_url with an mxc resource in the same widget event content

• A room with two custom widgets doesn't let you distinguish them in RightPanel #15992
• MSC1957: Integration manager discovery matrix-org/matrix-spec-proposals#1957
  • [WIP] MSC1956: Integrations API (base) matrix-org/matrix-spec-proposals#1956
    • Formally spec an API for interacting with integration managers matrix-org/matrix-spec#296

      • Integration managers are applications that help clients bring integrations (bridges, bots, etc) to users.

      • There are 2 integration managers at the time of writing this, each with different goals and APIs. Scalar (aka Modular) is the proprietary manager that ships with Riot and is developed specifically for Riot. Dimension (a project of mine) is open source and caters towards self-hosted integrations.

        • https://github.com/turt2live/matrix-dimension

          • An open source integration manager for matrix clients, like Element.

            • https://dimension.t2bot.io/
          • https://github.com/turt2live/matrix-dimension/blob/master/docs/reference/element_widget_api.md

            • Element's Widget API

          • https://github.com/turt2live/matrix-dimension/blob/master/docs/reference/scalar_client_api.md

            • Scalar API (Element)

• Inline widgets element-meta#1319
  • MSC2192: Inline widgets matrix-org/matrix-spec-proposals#2192
  • [WIP] Inline widgets matrix-org/matrix-react-sdk#3252
• When calling the Widget API using .readRoomEvents just after a page reload, only a subset of events are returned #20897

  • Use the example code below. You can create a new React app, add the matrix-widget-api package and paste the code into App.js.

  • https://github.com/matrix-org/matrix-widget-api

    • JavaScript/TypeScript API for widgets & web clients to communicate

    • For help and support, visit #matrix-widgets:matrix.org on Matrix.

    • Disclaimer: Widgets are not yet in the Matrix spec, so this library may not work with other implementations.

• Inline widgets which unfurl URLs element-meta#1269

And from Google/etc:

• https://matrix.org/blog/2017/08/23/introducing-matrix-widgets

  • Introducing Matrix Widgets - including Jitsi video conferencing!

• https://docs.google.com/document/u/0/d/1uPF7XWY_dXTKVKV7jZQ2KmsI19wn9-kFRgQ1tFQP7wQ/mobilebasic

  • Matrix.org Widgets (RFC), Version: 2.0, State: PROPOSED

  • Widgets are HTML and Javascript content / applications that can be embedded within, and communicate with Matrix clients.

  • This doc actually seems super detailed/comprehensive. I haven't tried to follow all of the content in it yet to see if it actually works as described.. but it seems like a pretty solid starting point.
• https://docs.google.com/document/d/1TiWNDcEOULeRYQpkJHQDjgIW32ohIJSi5MKv9oRdzCo/edit

  • Riot Widget API

  • What’s a widget? It’s an embedded iframe in Riot. So it’s just an arbitrary URL.

  • This seems like it was an earlier/less detailed document
• https://codesandbox.io/examples/package/matrix-widget-api

  • Matrix Widget Api Examples

  • Learn how to use matrix-widget-api by viewing and forking example apps that make use of matrix-widget-api on CodeSandbox.

• https://matrix-org.github.io/matrix-hookshot/latest/advanced/widgets.html
  • Not sure if these are specifically Matrix widgets, or widgets for matrix-hookshot, but I figured I would include it just in case anyway
• https://github.com/nordeck/matrix-widget-toolkit

  • Matrix Widget Toolkit

  • Reusable framework for creating Matrix widgets.

  • This repository contains multiple packages that provide reusable components that can be used to write Matrix/Element widgets.

  • @matrix-widget-toolkit/api: A package that wraps matrix-widget-api to provide a more convenient API.

  • @matrix-widget-toolkit/react: A package that provides a Widget API integration for React apps.

  • @matrix-widget-toolkit/semantic-ui: A package that provides a matching Semantic UI theme for Element.

  • @matrix-widget-toolkit/mui: A package that provides a matching Mui theme for Element.

  • @matrix-widget-toolkit/testing: Testing support for @matrix-widget-toolkit/api.

  • @matrix-widget-toolkit/widget-server: A container to host a Matrix widget in a production environment.

  • More packages to follow…

  • https://github.com/nordeck/matrix-widget-toolkit/tree/main/example-widget-mui

    • Matrix Example Widget (Material UI)

    • This is an example that shows how you can write a Matrix/Element widget. You can use this package as a starting point for writing new widgets.

• https://github.com/matrix-org/matrix-js-sdk

  • Matrix Client-Server SDK for JavaScript

• https://github.com/matrix-org/matrix-react-sdk

  • Matrix SDK for React Javascript

• https://github.com/matrix-org/matrix-events-sdk

  • JS/TS SDK for handling (extensible) events in Matrix

• https://github.com/turt2live/matrix-bot-sdk

  • TypeScript/JavaScript SDK for Matrix bots

  • https://github.com/turt2live/matrix-bot-sdk-bot-template

    • A template for creating bots with matrix-bot-sdk

  • https://github.com/turt2live/matrix-bot-sdk-bot-template

    • Repository of the bot we created in the matrix-bot-sdk-tutorial

[PseudoResonance]

A potential use-case outside the scope of Element may be file compression. Because of Discord's abysmal 8MB upload limit, I threw together a BetterDiscord plugin to compress images, video and audio automatically (using FFmpeg for the video/audio) which is extremely helpful. Even with the increased upload sizes today, I like to use it to quickly normalize audio files or clip video I send when I share things, and it would be great to have similar functionality on Element.

Maybe it would be appropriate to have a simplified version of it built into Element with a hidden advanced menu for all the other stuff? But I think it's probably better suited to being an optional plugin.