[MDS] Support Vega Visualizations #5975

huyaboo · 2024-02-28T00:43:51Z

Description

This PR adds Vega support for MDS. If data_source.enabled: true and a data_source_id field is provided in the Vega spec, then the Vega visualization will render with data from the data source (assuming the data source has the specified index).

Issues Resolved

closes #5927

Screenshot

Screen.Recording.2024-03-06.at.2.01.38.PM.mov

Testing the changes

Pre-requisites

Enable MDS (setdata_source.enabled: true in opensearch_dashboards.yml)
Add data source (in this example, it is vegaDataSource)
Add the newly created data source name to the data_source_name field under the url body

Testing

Go to Visualize -> Create visualization -> Vega
Copy-paste existing Vega visualization from the data source (or create a new one)
Add a field data_source_id to the vega spec under url and set the value to the data source id.
Vega should render, update when timefield is updated, and save.

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

BionIT · 2024-02-28T17:52:24Z

src/plugins/vis_type_vega/public/data_model/search_api.ts

@@ -70,7 +71,9 @@ export class SearchAPI {
          requestResponders[requestId].json(params.body);
        }

-        return search({ params }, { abortSignal: this.abortSignal }).pipe(
+        const searchApiParams = dataSourceId ? { params, dataSourceId } : { params };


I think we should check if data source plugin is enabled or not

Is this API public? I mean without UI, can I make API call?

What is dataSourceId is empty and there is no OpenSearch running locally (or opensearch.hosts:[] in OSD YML?

Here are the four cases and expected behaviors:

MDS is disabled and no datasource name is provided: visualization will fetch data from local cluster as usual

MDS is disabled but a datasource name is provided: visualization will throw error that MDS is disabled

MDS is enabled and no datasource name is provided: visualization will fetch data from local cluster as usual

MDS is enabled and a datasource name is provided: visualization will (attempt) to fetch data from remote

What if - MDS is enabled and no datasource name is provided and opensearch.hosts:[]

BionIT · 2024-02-28T18:04:34Z

src/plugins/vis_type_vega/public/data_model/search_api.ts

-        return search({ params }, { abortSignal: this.abortSignal }).pipe(
+        const searchApiParams = dataSourceId ? { params, dataSourceId } : { params };
+
+        return search(searchApiParams, { abortSignal: this.abortSignal }).pipe(


How do we support imported visualization for vega?

+1. Also I am curious about how do we support Sample Vega when we add sample data from remote data source?

+1 for import/export

Import visualization will be limited in support.

If Vega visualization were to be imported from a non-MDS enabled cluster -> MDS enabled cluster, then if a datasource is specified, the associated datasource name can be added to each opensearch url body (which would require editing the Vega spec directly). This is what can happen when importing sample data.

However, if the visualization were to be imported from an MDS enabled cluster -> MDS enabled cluster, then there is no current method to map new datasource names to url bodies. This is because the import saved objects can only determine 1 datasource. Since the Vega spec supports multiple url queries to multiple datasources, the import would inevitably break.

@huyaboo import and export can be addressed in separate issue, we can check the vega script inside the saved object to see if datasource name is specified, if not, then we can add the data source name, if yes, then we don't add but checks if the data source exists. Does that simplify it?

Flyingliuhub · 2024-02-28T18:27:38Z

src/plugins/vis_type_vega/public/data_model/search_api.ts

@@ -61,6 +61,7 @@ export class SearchAPI {
    return combineLatest(
      searchRequests.map((request) => {
        const requestId = request.name;
+        const dataSourceId = request.data_source_id;


Can we add test coverage? thanks

Added some test cases to parse the id from the data_source_name

bandinib-amzn · 2024-03-02T00:04:51Z

This need not to be scope in this PR

Instead of adding a field data_source_id to the vega spec under url, can we show a screen to choose data source? cc: @kgcreative

Once we have selected datasource, we will have the data source id and we can inject that data source id into vega spec(under url) under the hood. I think this will improve user experience overall.

On top of my mind, flow will remain same as existing vizualization. Instead of showing index pattern, searches, we will show data sources.

bandinib-amzn · 2024-03-02T00:06:32Z

CHANGELOG.md

@@ -70,6 +70,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - [Multiple Datasource] Concatenate data source name with index pattern name and change delimiter to double colon ([#5907](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5907))
 - [Multiple Datasource] Refactor client and legacy client to use authentication registry ([#5881](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5881))
 - [Multiple Datasource] Improved error handling for the search API when a null value is passed for the dataSourceId ([#5882](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5882))
+- [Multiple Datasource] Add Vega support to MDS by specifying a data source id in the Vega spec ([#5975](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5975))


can we move this change log to unreleased?

Moved to unreleased

kgcreative · 2024-03-02T00:25:34Z

I think the challenge here is Vega allows for customers to select both index patterns, but also raw indexes

seraphjiang · 2024-03-03T04:57:23Z

vega is for advanced user who could write scripting.

I'm leaning for a simple user experience in the initial release if possible.

In the future we might considering some auto completion or code intelligence feature, or integrate with AI assistant based on requirements

BTW
@bandinib-amzn @Flyingliuhub @ZilongX @BionIT appreciate if you could help to review the current implementation

seraphjiang · 2024-03-03T06:06:42Z

@BionIT @yujin-emma

Do we need to change saved object export code in order to support vega in import/export scenarios for MDS. If so, could we create a follow up issue to track?

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

xinruiba · 2024-03-08T05:47:22Z

src/plugins/vis_type_vega/public/data_model/search_api.ts

+    const dataSources = await this.dataSourceFindQuery(dataSourceName);
+
+    // In the case that data_source_name is a prefix of another name, match exact data_source_name
+    const possibleDataSourceIds = dataSources.savedObjects.filter(
+      (obj) => obj.attributes.title === dataSourceName
    );
+
+    if (possibleDataSourceIds.length !== 1) {
+      throw new Error(
+        `Expected exactly 1 result for data_source_name "${dataSourceName}" but got ${possibleDataSourceIds.length} results`
+      );
+    }


This logic is to check no duplicate datasource names between all datasources.

But we set the paging size 10 here: https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5975/files#diff-f1c265abe9e3b3c11a64d4110cf3e6a010dfe369cf6eed9a20ea532763489fbdR125

In that case, will the duplication check only applies to first 10 datasources?

So there are two cases where possibleDataSourceIds before filtering is > 1:

The data_source_name is a duplicate of another (which means querying cannot be done)

The name is a prefix of other data_source_names. For example, if the name were Vega Data Source and the other were Vega Data Source Copy, then the find query for Vega Data Source will return both these data source names. The filtering is done to ensure that Vega Data Source is chosen and not Vega Data Source Copy.

The paging limit of 10 helps keep the number of possible datasources low since this is client-side.

Is it possible for the initial paging to be free of duplicates, but duplication happens when applied across all data sources?

This comment is not a blocker of this PR.
I don't think we are able to create datasource with duplication name, so this logic should be fine.

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

* Add MDS support for Vega Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Refactor field to data_source_id Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Add to CHANGELOG.md Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Added test cases and renamed field to use data_source_name Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Add prefix datasource name test case and add example in default hjson Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Move CHANGELOG to appropriate section Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Increased test coverage of search() method Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> --------- Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> (cherry picked from commit 1c5ad6c) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md

* Add MDS support for Vega Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Refactor field to data_source_id Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Add to CHANGELOG.md Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Added test cases and renamed field to use data_source_name Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Add prefix datasource name test case and add example in default hjson Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Move CHANGELOG to appropriate section Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Increased test coverage of search() method Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> --------- Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> (cherry picked from commit 1c5ad6c) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add MDS support for Vega Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Refactor field to data_source_id Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Add to CHANGELOG.md Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Added test cases and renamed field to use data_source_name Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Add prefix datasource name test case and add example in default hjson Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Move CHANGELOG to appropriate section Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Increased test coverage of search() method Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> --------- Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> (cherry picked from commit 1c5ad6c) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> # Conflicts: # CHANGELOG.md Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> (cherry picked from commit 05ede47) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add MDS support for Vega * Refactor field to data_source_id * Add to CHANGELOG.md * Added test cases and renamed field to use data_source_name * Add prefix datasource name test case and add example in default hjson * Move CHANGELOG to appropriate section * Increased test coverage of search() method --------- (cherry picked from commit 1c5ad6c) # Conflicts: # CHANGELOG.md (cherry picked from commit 05ede47) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

huyaboo added 2 commits February 28, 2024 00:10

Add MDS support for Vega

efe5116

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

Refactor field to data_source_id

eb9c9ef

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

github-actions bot added the first-time-contributor label Feb 28, 2024

Add to CHANGELOG.md

ddba426

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

huyaboo force-pushed the vega-mds-support branch from 330f37e to ddba426 Compare February 28, 2024 00:46

huyaboo marked this pull request as ready for review February 28, 2024 02:12

huyaboo requested review from ananzh, kavilla, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, kristenTian, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, BSFishy, curq, bandinib-amzn and SuZhou-Joe as code owners February 28, 2024 02:12

BionIT reviewed Feb 28, 2024

View reviewed changes

Flyingliuhub reviewed Feb 28, 2024

View reviewed changes

bandinib-amzn reviewed Mar 2, 2024

View reviewed changes

Added test cases and renamed field to use data_source_name

76aa282

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

huyaboo added 2 commits March 8, 2024 00:51

Increased test coverage of search() method

be65384

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

Merge branch 'main' into vega-mds-support

a5aa803

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

BionIT previously approved these changes Mar 8, 2024

View reviewed changes

BionIT added the multiple datasource multiple datasource project label Mar 8, 2024

bandinib-amzn previously approved these changes Mar 8, 2024

View reviewed changes

Merge branch 'main' into vega-mds-support

fb8fe84

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

huyaboo dismissed stale reviews from bandinib-amzn and BionIT via fb8fe84 March 8, 2024 04:04

BionIT previously approved these changes Mar 8, 2024

View reviewed changes

bandinib-amzn previously approved these changes Mar 8, 2024

View reviewed changes

xinruiba reviewed Mar 8, 2024

View reviewed changes

xinruiba approved these changes Mar 8, 2024

View reviewed changes

Merge branch 'main' into vega-mds-support

f9747f3

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

huyaboo dismissed stale reviews from bandinib-amzn and BionIT via f9747f3 March 8, 2024 20:09

BionIT approved these changes Mar 8, 2024

View reviewed changes

bandinib-amzn approved these changes Mar 8, 2024

View reviewed changes

bandinib-amzn merged commit 1c5ad6c into opensearch-project:main Mar 8, 2024
68 checks passed

huyaboo mentioned this pull request Mar 18, 2024

[DOC] Add documentation about how to use Vega Visualization plugin in OpenSearch Dashboards opensearch-project/documentation-website#6709

Closed

4 tasks

Flyingliuhub added backport 2.x v2.13.0 labels Mar 19, 2024

opensearch-trigger-bot bot mentioned this pull request Mar 19, 2024

[Backport 2.x] [MDS] Support Vega Visualizations #6212

Merged

BionIT mentioned this pull request Mar 19, 2024

[Meta][2.13] Security, bug fix and enhancemanet for dashboards anywhere #5752

Open

28 tasks

huyaboo mentioned this pull request Mar 19, 2024

[MDS] Install Vega sample data #6218

Merged

7 tasks

huyaboo mentioned this pull request Apr 11, 2024

[MDS] Full Timeline Support #6410

Closed

huyaboo deleted the vega-mds-support branch April 18, 2024 00:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MDS] Support Vega Visualizations #5975

[MDS] Support Vega Visualizations #5975

huyaboo commented Feb 28, 2024 •

edited

Loading

BionIT Feb 28, 2024

bandinib-amzn Mar 2, 2024

huyaboo Mar 6, 2024

bandinib-amzn Mar 8, 2024

BionIT Feb 28, 2024

bandinib-amzn Mar 1, 2024

seraphjiang Mar 3, 2024

huyaboo Mar 6, 2024

BionIT Mar 7, 2024

Flyingliuhub Feb 28, 2024

huyaboo Mar 6, 2024

bandinib-amzn commented Mar 2, 2024

bandinib-amzn Mar 2, 2024

BionIT Mar 5, 2024

huyaboo Mar 7, 2024

kgcreative commented Mar 2, 2024

seraphjiang commented Mar 3, 2024

seraphjiang commented Mar 3, 2024

xinruiba Mar 8, 2024 •

edited

Loading

huyaboo Mar 8, 2024

xinruiba Mar 8, 2024

xinruiba Mar 8, 2024 •

edited

Loading

[MDS] Support Vega Visualizations #5975

[MDS] Support Vega Visualizations #5975

Conversation

huyaboo commented Feb 28, 2024 • edited Loading

Description

Issues Resolved

Screenshot

Testing the changes

Pre-requisites

Testing

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bandinib-amzn commented Mar 2, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kgcreative commented Mar 2, 2024

seraphjiang commented Mar 3, 2024

seraphjiang commented Mar 3, 2024

xinruiba Mar 8, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xinruiba Mar 8, 2024 • edited Loading

Choose a reason for hiding this comment

huyaboo commented Feb 28, 2024 •

edited

Loading

xinruiba Mar 8, 2024 •

edited

Loading

xinruiba Mar 8, 2024 •

edited

Loading