[Multiple Datasource][Version Decoupling] Support Version Decoupling in Index Patterns Dashboards Plugin #7100

ZilongX · 2024-06-24T20:57:28Z

Description

This PR is to add support of Version Decoupling in Index Patterns Dashboards Plugin
This is achieved mainly by add a new entry supportedOSDataSourceVersions in the plugin manifest file
For Index Patterns Plugin it would support all versions of OS clusters so the supported range would be >=1.0.0
The current behavior is to filter out the incompatible data sources out of the data source list.

Issues Resolved

#7099

Screenshot

Screen.Recording.2024-06-24.at.1.30.34.PM.mov

Changelog

fix: [MDS][Version Decoupling] Add support of Version Decoupling in Index Patterns Dashboards Plugin

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

…in Index Patterns Dashboards Plugin Signed-off-by: Zilong Xia <zilongx@amazon.com>

codecov · 2024-06-24T21:14:15Z

Codecov Report

Attention: Patch coverage is 25.00000% with 3 lines in your changes missing coverage. Please review.

Project coverage is 67.45%. Comparing base (e6aa9d6) to head (2b6db3b).

Files	Patch %	Lines
...ents/step_data_source/components/header/header.tsx	0.00%	2 Missing ⚠️
...ndex_pattern_management/public/components/utils.ts	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7100      +/-   ##
==========================================
+ Coverage   67.43%   67.45%   +0.01%     
==========================================
  Files        3448     3448              
  Lines       67960    67964       +4     
  Branches    11057    11057              
==========================================
+ Hits        45832    45845      +13     
+ Misses      19454    19446       -8     
+ Partials     2674     2673       -1

Flag	Coverage Δ
Linux_1	`33.10% <0.00%> (-0.01%)`	⬇️
Linux_2	`55.06% <100.00%> (+<0.01%)`	⬆️
Linux_3	`45.25% <0.00%> (+<0.01%)`	⬆️
Linux_4	`34.83% <0.00%> (-0.01%)`	⬇️
Windows_1	`33.12% <0.00%> (?)`
Windows_2	`55.01% <100.00%> (+<0.01%)`	⬆️
Windows_3	`45.25% <0.00%> (-0.01%)`	⬇️
Windows_4	`34.83% <0.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

yujin-emma

LGTM

bandinib-amzn · 2024-06-25T00:09:47Z

Can you please add more UT for uncovered lines so that codecov CI would pass?

bandinib-amzn · 2024-06-25T00:16:23Z

For Index Patterns Plugin it would support all versions of OS clusters so the supported range would be >=1.0.0

Do we have any test result to support above sentence? Can we attach that here as source of truth?

bandinib-amzn · 2024-06-25T00:19:37Z

...ponents/create_index_pattern_wizard/components/step_data_source/components/header/header.tsx

 import { useOpenSearchDashboards } from '../../../../../../../../../plugins/opensearch_dashboards_react/public';
 import { getDataSources } from '../../../../../../components/utils';
 import { DataSourceTableItem, StepInfo } from '../../../../types';
 import { LoadingState } from '../../../loading_state';
+import * as pluginManifest from '../../../../../../../opensearch_dashboards.json';


nit: can we use absolute path here instead of relative path with directory traversal?

yes, this one just to keep the save relative style as existing imports lol

bandinib-amzn · 2024-06-25T00:21:28Z

src/plugins/index_pattern_management/public/components/create_index_pattern_wizard/types.ts

@@ -92,4 +92,5 @@ export interface DataSourceTableItem {
  sort: string;
  checked?: 'on' | 'off';
  label: string;
+  datasourceversion: string;


Is it expected that each plugin needs to have this change in order to onboard version decoupling?

Yes, datasource would be the minimal fields to check in order to decide the version compatibility

bandinib-amzn · 2024-06-25T00:21:55Z

src/plugins/index_pattern_management/public/components/utils.ts

@@ -99,13 +99,15 @@ export async function getDataSources(savedObjectsClient: SavedObjectsClientContr
            const id = dataSource.id;
            const type = dataSource.type;
            const title = dataSource.get('title');
+            const datasourceversion = dataSource.get('dataSourceVersion');


same as above.

zhongnansu · 2024-06-25T01:05:34Z

...ponents/create_index_pattern_wizard/components/step_data_source/components/header/header.tsx

@@ -68,6 +70,12 @@ export const Header: React.FC<HeaderProps> = (props: HeaderProps) => {
      .then((fetchedDataSources: DataSourceTableItem[]) => {
        setIsLoading(false);
        if (fetchedDataSources?.length) {
+          fetchedDataSources = fetchedDataSources.filter((dataSource) =>
+            semver.satisfies(
+              dataSource.datasourceversion,


what about for existing/old data sources that doesn't have this field? Does it break backwards compatibility?

for older data sources it won't work yest as they won't support version decoupling, MDS initially just list out all data sources created regardless so there would be a bunch of unexpected behaviors, version decoupling is aiming to improve that user experiences by only allowing consumption against compatible data sources.

besides there is a refresh flow of data sources planning in progress which would also resolve the concern here, which will refresh the current saved object to bring it up-to-date

ZilongX · 2024-06-27T20:15:22Z

Can you please add more UT for uncovered lines so that codecov CI would pass?

Yes I was planned yet later found the whole section was not covered in the first place, I'll try to add coverage for the whole section or maybe in separate PR

ZilongX · 2024-06-27T20:16:40Z

For Index Patterns Plugin it would support all versions of OS clusters so the supported range would be >=1.0.0

Do we have any test result to support above sentence? Can we attach that here as source of truth?

Yes, we have test results indicating latest Index Pattern would only support ES 7.9 + which for OS it would be 1.0 +

…in Index Patterns Dashboards Plugin (#7100) * [Multiple Datasource][Version Decoupling] Support Version Decoupling in Index Patterns Dashboards Plugin Signed-off-by: Zilong Xia <zilongx@amazon.com> * Changeset file for PR #7100 created/updated --------- Signed-off-by: Zilong Xia <zilongx@amazon.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com> (cherry picked from commit 5ee6f58) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…in Index Patterns Dashboards Plugin (#7100) (#7123) * [Multiple Datasource][Version Decoupling] Support Version Decoupling in Index Patterns Dashboards Plugin * Changeset file for PR #7100 created/updated --------- (cherry picked from commit 5ee6f58) Signed-off-by: Zilong Xia <zilongx@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: opensearch-changeset-bot[bot] <154024398+opensearch-changeset-bot[bot]@users.noreply.github.com>

[Multiple Datasource][Version Decoupling] Support Version Decoupling …

b17902a

…in Index Patterns Dashboards Plugin Signed-off-by: Zilong Xia <zilongx@amazon.com>

ZilongX requested review from ananzh, kavilla, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, zhongnansu, manasvinibs, Flyingliuhub, curq, bandinib-amzn, SuZhou-Joe, ruanyl, BionIT, xinruiba, zhyuanqi and mengweieric as code owners June 24, 2024 20:57

github-actions bot added the seasoned-contributor label Jun 24, 2024

ZilongX added backport 2.x v2.16.0 labels Jun 24, 2024

Changeset file for PR opensearch-project#7100 created/updated

fdf8a86

ZilongX added multiple datasource multiple datasource project version decoupling labels Jun 24, 2024

xinruiba approved these changes Jun 24, 2024

View reviewed changes

yujin-emma approved these changes Jun 24, 2024

View reviewed changes

bandinib-amzn reviewed Jun 25, 2024

View reviewed changes

zhongnansu reviewed Jun 25, 2024

View reviewed changes

Merge branch 'main' into index_pattern_version_decoupling

1305621

ZilongX requested a review from LDrago27 as a code owner June 26, 2024 19:55

ZilongX added 2 commits June 27, 2024 12:50

Merge branch 'main' into index_pattern_version_decoupling

e649ca0

Merge branch 'main' into index_pattern_version_decoupling

2b6db3b

bandinib-amzn approved these changes Jun 27, 2024

View reviewed changes

ZilongX merged commit 5ee6f58 into opensearch-project:main Jun 27, 2024
66 of 67 checks passed

opensearch-trigger-bot bot mentioned this pull request Jun 27, 2024

[Backport 2.x] [Multiple Datasource][Version Decoupling] Support Version Decoupling in Index Patterns Dashboards Plugin #7123

Merged

ZilongX deleted the index_pattern_version_decoupling branch June 28, 2024 21:01

wanglam mentioned this pull request Jul 9, 2024

Add version decoupling meta for MDS opensearch-project/ml-commons-dashboards#338

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Multiple Datasource][Version Decoupling] Support Version Decoupling in Index Patterns Dashboards Plugin #7100

[Multiple Datasource][Version Decoupling] Support Version Decoupling in Index Patterns Dashboards Plugin #7100

ZilongX commented Jun 24, 2024 •

edited

Loading

codecov bot commented Jun 24, 2024 •

edited

Loading

yujin-emma left a comment

bandinib-amzn commented Jun 25, 2024

bandinib-amzn commented Jun 25, 2024

bandinib-amzn Jun 25, 2024

ZilongX Jun 27, 2024

bandinib-amzn Jun 25, 2024

ZilongX Jun 27, 2024

bandinib-amzn Jun 25, 2024

zhongnansu Jun 25, 2024

ZilongX Jun 27, 2024

ZilongX commented Jun 27, 2024

ZilongX commented Jun 27, 2024

[Multiple Datasource][Version Decoupling] Support Version Decoupling in Index Patterns Dashboards Plugin #7100

[Multiple Datasource][Version Decoupling] Support Version Decoupling in Index Patterns Dashboards Plugin #7100

Conversation

ZilongX commented Jun 24, 2024 • edited Loading

Description

Issues Resolved

Screenshot

Changelog

Check List

codecov bot commented Jun 24, 2024 • edited Loading

Codecov Report

yujin-emma left a comment

Choose a reason for hiding this comment

bandinib-amzn commented Jun 25, 2024

bandinib-amzn commented Jun 25, 2024

bandinib-amzn Jun 25, 2024

Choose a reason for hiding this comment

ZilongX Jun 27, 2024

Choose a reason for hiding this comment

bandinib-amzn Jun 25, 2024

Choose a reason for hiding this comment

ZilongX Jun 27, 2024

Choose a reason for hiding this comment

bandinib-amzn Jun 25, 2024

Choose a reason for hiding this comment

zhongnansu Jun 25, 2024

Choose a reason for hiding this comment

ZilongX Jun 27, 2024

Choose a reason for hiding this comment

ZilongX commented Jun 27, 2024

ZilongX commented Jun 27, 2024

ZilongX commented Jun 24, 2024 •

edited

Loading

codecov bot commented Jun 24, 2024 •

edited

Loading