Richer bot interactions

[pontaoski]

Suggestion
As a developer of assorted multiplatform amusement and utility bots, I find Matrix's plaintext interactions to be somewhat lacking, especially in compraison to how richly formatted the messages themselves can be.

Some grokking around for "button" "drop down" etc. didn't really indicate that anyone else had requested richer interactions, so I guess I will.

There are two main components to what I need for richer bot interactions that I'm not getting out of Matrix:

• the ability for a user to send something to another user (bot) in the context of a room that does not necessarily form a visible message in the room (through e.g. buttons, dropdowns, inline keyboards, whatever features you want to take from other platforms)
• the ability for the bot to send feedback in response to this interaction that again may not necessarily form a visible message in the room (e.g. through an ephemeral message or a popup/snackbar/whatever you want to call it)

The underlying thing (I think) needed for both of these is the ability to dispatch events that are only visible to a user and another user within a room. Everything else would just be building on top of that base thing.

[pontaoski]

matrix-org/matrix-spec-proposals#3006 i was made aware of this older MSC proposal

[tezlm]

This would be really nice, is #1 feature I miss from discord, and is the #2 pain point I have with matrix (#1 being push rules, of course). Letting bots be able to embed extra ui like this in their messages would make bots much more flexible. (the conduit admin bot could benefit immensely).

ideas for bot ui stuff

• Buttons are a core part of ui. Even if the entire rest of this issue + comment is ignored, being able to specify preset reactions on a message would increase usability by 10x
• Dropdowns are particularily useful - arbitrary items, user ids/room members, and room ids. (Discord implements this as "lots of buttons in one list" - selecting an item automatically sends a response. Is this the best option?)
• Text inputs and full forms (fill out stuff, then click submit) could be nice to have.
• As said in the parent issue, temporary/ephemeral responses could be nice.
  • Though, from what I've seen in practice, half of ephemeral responses are "interaction was successful" and the other half are "a lot of data that shouldn't be persisted". The first half could have a special "interaction complete" annotation relation while the second one isn't much of a problem since matrix html supports <details>/<summary>
• "Typedchecked" commands would be nice, rather than parsing message bodies
  • This is *extremely* painful, considering reply fallbacks + client having different plain text mention formats, and should you choose to parsing rich text there's so much complexity compared to "split by whitespace"
• Quick replies (Selectable replies to messages sent by bots [canned responses] #232) generally could be replaced by one of the above options, but are an excellent lowest common denominator

Of course not all of these need to be implemented (The two that I think would have the highest effort/payoff ratio by far are buttons and dropdowns-as-lists-of-buttons).

why i think widgets are a bad idea

The main way bot interactions seem to be shaping up are with widgets, where webpages can be embedded in clients. Imo, this is a bad idea because

• Clients wanting to support widgets need to embed an entire browser.
• Bot developers would need to write out a custom html ui + use a separate widget api instead of interacting with events
• Since widgets seem to be able to do interactions on behalf of the user(?), clients need to take extra care in implementing sandboxing and access control. Bot-based ui would let bots use standard power levels and thus reduce the attack surface
• Bot-based ui would it in with the rest of the application's ui, and may be more accessible

The sole benefit I can think of for widgets is flexibility (you get all the features of html/css/js), but the downsides seem to outweight the benefits and in practice discord seems to do fine.

Bonus point: since its a particularily user facing feature, it gives more leverage to nerds like me for getting friends to move to matrix - imagine reiterating "yeah but its federated and e2ee" vs "but look at all the things you can do with this shiny new bot" ;)