[PROPOSAL] Make the User Behavior Insights (UBI) plugin a part of the OpenSearch plugin ecosystem

[jzonthemtn]

What/Why

What are you proposing?

The goal of the User Behavior Insights (UBI) project is to unify client-side events with OpenSearch queries for purposes of improving the user search experience and search relevance. The project currently consists of an OpenSearch plugin that facilitates the storage of client-side events and queries in OpenSearch indexes and a JavaScript reference implementation for capturing client-side events and sending the events to the plugin. It is likely that UBI will also contain a set of dashboards, perhaps implemented as an OpenSearch Dashboards plugin.

OpenSearch User Behavior Insights (UBI) Plugin repository

What users have asked for this feature?

This capability has commonly be requested by e-commerce websites, but other search systems can benefit, too. Improving the user search journey has wide use-cases.

What problems are you trying to solve?

Improving search relevance is a common goal for search systems. E-commerce sites, in particular, would benefit from the ability to analyze client-side along with OpenSearch queries to improve their user search experience. Additionally, having the plugin a part of the OpenSearch ecosystem may encourage more community engagement and promote adoption.

What is the developer experience going to be?

The UBI plugin has a REST API that provides methods for:

• Managing UBI stores which hold client-side events and queries.
• Receiving and persisting client-side events.
• Passively capturing queries, and associating queries with client-side events.

Are there any security considerations?

I am not currently aware of any security considerations.

Are there any breaking changes to the API

No, there are no breaking changes proposed.

What is the user experience going to be?

The user experience is described by the plugin's documentation. A user installs the plugin in OpenSearch, then using the plugin's API, the user initializes a new UBI store. The user can now send client-side events to the plugin's API and the plugin can begin passively capturing OpenSearch queries while associating the two together.

Are there breaking changes to the User Experience?

No, there are no breaking changes to the user experience.

Why should it be built? Any reason not to?

The plugin and its functionality as described above has been built. At the time of writing it is in a beta status and being tested and validated prior to a 1.0 release.

What will it take to execute?

Providing a repository under the opensearch-project organization for the opensearch-ubi plugin. This repository can be transferred to the opensearch-project organization as-is.

Any remaining open questions?

No, other open questions.

[epugh]

@bbarani @anirudha @smacrakis I wanted to draw your attention to this ticket.

[dblock]

We've transferred repos into the opensearch-project in the past, e.g. https://github.com/opensearch-project/opensearch-learning-to-rank-base, so in principle this is totally doable. There's some internal Amazon steps around security and IP to follow, @anirudha or @smacrakis, is this something you're willing to take on?

[dblock]

@epugh Do you want to move the repo for discoverability, or are you after trying to include the plugin into the standard distribution?

[smacrakis]

The UBI plugin has two functions:

• Capturing server-side information useful for User Behavior Insights, notably the actual query sent to OS.
• Capturing client-side information as an HTTP endpoint and inserting it in an index. This is mostly for prototyping and very small uses -- production level capture will use more scalable mechanisms. This function may well be removed by the time we release -- we may replace it by direct use of OS APIs.

These functions should become a standard plugin in the distribution.

[bbarani]

@dblock I had a discussion with @epugh . The ultimate goal here is to include this plugin by default as part of release distribution. For 2.13, they are planning to release the plugin on their own (as standalone plugin) but they would like to bundle this plugin as part of default distribution.

Btw, we had a similar request to support external plugin outside of OpenSearch GitHub project before for Prometheus plugin but difference being that the UBI team is ready to move the repo to OpenSearch GitHub project.

[msfroh]

In case we need at least one Amazon employee as a maintainer on the repo to satisfy the bureaucracy, I'm happy to join in. (Also, I am genuinely interested in helping to review the plugin and participate in development.)

[jzonthemtn]

Thanks everyone. It has been decided to try to transition UBI from a plugin to more of a core functionality of OpenSearch, so I am closing this issue. We do have a 2.13.0 release as a plugin that's available from our GitHub repository.