[NEW] Connection of external search providers in spotlight

[mrsimpson]

Motivation

Finding rooms based on message content is a requirement which has been around for quite a long time (see e. g. RocketChat/Rocket.Chat#1615). There is even an open PR RocketChat/Rocket.Chat#7628 . However, there are (and will always be) concerns with respect to performance and DB load.
When it comes down to searching huge installations, I believe there's no other way than using an external search engine. Replicating data to them is no issue as existing hooks already exist, but getting it integrated into the results is currently not possible.
In order to

• integrate those results, a hook shall be provided in the spotlight-method which retrieves results asynchronously.
• ensure proper access permissions, all potentially accessible rooms for the current user shall be determined (which should be a fast, index-based query) and passed to the hook
• allow understanding of search results, the spotlight result should be enhanced by additional information why a particular result has been found (e. g. [subscription, public channel, message content])

Proposal for implementation

For flexibility reasons, a modular integration of configurable search providers shall be realized.
Since a database-connection cannot be assumed to be available, all replication shall be performed on the application layer (e. g. using http-requests).

• Provide a package in Rocket.Chat core which exposes an API for registering external search providers and an interface definition to be implemented by them. Follow model of RoomTypes (see registry and the interface definition. The active search provider's interface methods shall be triggered explicitly from the rocket.chat runtime as alternative to (untyped) webhooks. Relevant events: Room created, message sent, user added/removed, permission changed
• Provide a settings group which generates sections with the settings provided by the search providers
• Provide a generic UI template which is the default visualization for the search provider implementation
• Provide a second package with a reference implementation

To be clarified: How to perform an initial load? Most probably, a dedicated Method triggering the proper interface methods for all existing objects is appropriate.

• How should this potentially long-running background-task be executed?
• Does this need a particular authorization?

[ruKurz]

Just a few thoughts:

• Define a search engine independent interface
• Create a Rocket.Chat Plugin that implements a external search
• make the user's read permission for subscriptions, channels, messages part of the external search index

[mrsimpson]

@geekgonecrazy I extended the description in the way we discussed it - at least I hope so.
In case I forgot something, feel free to extend it.
As I said we're aiming to provide the API, an implementation and a docker image so that an on-premise full-text-search can be provided at low cost.
@lobo87, @tkurz anything to add from your side?

[mrsimpson]

How should this potentially long-running background-task be executed?

@geekgonecrazy any idea or reference implementation?

[geekgonecrazy]

Awesome!

There are products out there that do one thing and one thing very well: Search. So it makes sense to be able to utilize them in large deployments for very fast search.

Being able to register search providers similar to how roomtypes can be registered would be great. I like this model a lot.

In an ideal world I think we would do:

• Api to register search providers
• Default search provider for that same minimal out of box configuration we strive for. [NEW] Global search  Rocket.Chat#7628 might be great for that.
• A search provider included for larger deployments that is very easy for users to deploy. As discussed sounds like you have one in mind.

As for the api sounds like we need active search provider to be able to receive events regarding the following:

• Channel Creation
• User Join Channel
• Permission related to Channel / User
• Message Send.

Also needs to be able to provide settings that can be surfaced in the administration.

It also needs a way to feed the search provider information on activation. Lets say I enable it on a server that is already active. Of course its great that it starts handling things from then on forward.

But it also needs to be able to index existing content.

So on enable I would imagine it needs to be provided rooms created, user membership and permissions. That's the bare minimum for it to function. From there new messages would be sent to it and it would index them.

But then the big one... Messages sent prior to activation would need to be synced to it. Doing this one well could be difficult.

Then of course the obvious one. We need to be able to call the search provider potentially with advanced search params (like limits) to actually get the data and surface in the UI.

[geekgonecrazy]

Also talked a bit with @karlprieb the search is currently planned to be implemented in the flex tab similar to how slack does. From my understanding we will be putting it in a more prominent spot at the top of the UI for faster and easier access

[mrsimpson]

@karlprieb The flex tab has got very limited space for visualizing search results.
What do you think of adding it to the full-modal?
Of course, a dedicated interaction was necessary in order to trigger it.
On the downside, a modal makes it inefficient to skip though the results...