[Backport 2.x] MultiDataSource feature merge (#2334) #2399

kavilla · 2022-09-21T17:06:58Z

Backport 890a4b6 from #2334

* Instantiate credential management plugin code structure (opensearch-project#1996) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Data source inside stack management setup (opensearch-project#2017) (opensearch-project#2030) Signed-off-by: Kristen Tian <tyarong@amazon.com> * enable CI for feature branch (opensearch-project#2010) Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add empty data source plugin (opensearch-project#2052) Adds empty data source plugin. Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Add initial credential management CRUD pages (opensearch-project#2040) * Add credential management CRUD pages 1. List all credentials 2. Create your saved credential 3. Edit your credential 4. Delete credentials Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Register Data source to savedObject & Update license header (opensearch-project#2037) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Move credential saved object to data source plugin (opensearch-project#2062) Signed-off-by: Louis Chu <clingzhi@amazon.com> Move credential saved object to data source plugin Resolve follow up comments on UI Signed-off-by: Kristen Tian <tyarong@amazon.com> * breadcrumbfix for datasource management (opensearch-project#2066) * breadcrumbfix for datasource management Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * breadcrumbfix for datasource management - refactoring code Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * using services to update breadcrumb on data sources management page Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Changing the license header on breadcrumbs.ts datasource management Signed-off-by: Kristen Tian <tyarong@amazon.com> * Fix breadcrumb on listing page and update saved object mapping (opensearch-project#2069) Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add data source step into IndexPattern with Mock switch (opensearch-project#2064) (opensearch-project#2086) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add delete button for credential detailed page (opensearch-project#2067) Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Link datasource to indexpattern (opensearch-project#2118) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add encrypt/decrypt module on data source plugin (opensearch-project#2120) Signed-off-by: Louis Chu <clingzhi@amazon.com> 1. Add encrypt/decrypt module with UT 2. Add client factory wrapper for encrypt credential 3. Add encryption config support 4. Bugfix on Jest interpret Buffer Signed-off-by: Kristen Tian <tyarong@amazon.com> * Integrate index pattern with new data client (opensearch-project#2146) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add noAuth to dataSource attributes (opensearch-project#2154) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Datasource Management - creation & listing - UI only (opensearch-project#2128) * data source management - creation & Listing UI only * data source management - creation & Listing UI only * Create/edit data source feature * toggling default value * refactoring code as per review comments * toggling server flag to false Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Apply dataSource plugin as dependent for cm and dsm plugins (opensearch-project#2150) Signed-off-by: Louis Chu <clingzhi@amazon.com> Apply dataSource plugin as dependent for cm and dsm plugins (opensearch-project#2150) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Leverage datasource enablement in index pattern management Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Add client management module and register `getClient()` to route handler context (opensearch-project#2121) * Add client management module and register `getClient()` interface to route handler context Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Make step info in index pattern creation dynamic (opensearch-project#2164) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Refactor for credential listing page & add loading effect (opensearch-project#2142) Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Integration sequal - replace data client placeholders (opensearch-project#2167) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Integrate with crypto module to decrpt password (opensearch-project#2170) Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] noAuth integration, credential & endpoint validation (opensearch-project#2165) * noAuth integration, credential & endpoint validation Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Refactoring validation message Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Adding back accidentally deleted file home/tutorials/haproxy_metrics/index.ts Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Refactor search strategy to conditionally user datasource client (opensearch-project#2171) Signed-off-by: Kristen Tian <tyarong@amazon.com> * adding relation between credential selection and no auth checkbox (opensearch-project#2175) Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update getIndicesViaSearch with datasource (opensearch-project#2176) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add null header to child client spawn (opensearch-project#2188) Signed-off-by: Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD][IP]Update button position & Add UT & Add skip option (opensearch-project#2195) * Add UT - index pattern related Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update button position Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add skip option to allow use default os data source Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD]Refactor layout and validate input fields for listing and create pages (opensearch-project#2202) Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Enable datasource link in saveObjectManagement (opensearch-project#2209) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update configure data source per UX input (opensearch-project#2235) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Enable data source audit log to file (opensearch-project#2215) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Refactor of credential editing page layout & refactor backend field validation method (opensearch-project#2222) * Refactor of credential editing page layout & refactor backend field validation method * Resolved the comments & fix the multiple call for one operation Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Tweak fetch data back to original (opensearch-project#2238) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Revamped UX for data source management (opensearch-project#2239) * revamped UX for data source management Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * refactored datasource screens as per PR comments Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Credentials security redesign (opensearch-project#2253) 1. Data model changes for data source saved object 2. Server side changes for data source saved object a. Implement data_source_saved_objects_client_wrapper to integrate with CryptographyClient for password encryption / decryption. b. Change data_source_service to fetch credentials directly from data source (still decrypt via CryptographyClient) c. Fix unit tests accordingly Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add step data source UI test (opensearch-project#2264) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD]Improve datasource server side error handling (opensearch-project#2236) Signed-off-by: Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Datasource management new API change integrations (opensearch-project#2282) * changing datasource management design to integrate with new API changes Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Update edit_data_source_form.tsx moving masked password to constants Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update stream test to bypass CI env generate domain attribute Signed-off-by: Kristen Tian <tyarong@amazon.com> * Delete credential management Signed-off-by: Kristen Tian <tyarong@amazon.com> * Address comments Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Datasource Management - Create data source - Unit tests (opensearch-project#2341) * Unit test cases for data source management - create Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * adding tests to utils.ts & changing it to test Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Fix update data source & block update endpint (opensearch-project#2364) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * MD datasource management-datasource table-UTs (opensearch-project#2350) Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * reafctor based on PR 2334 comments to merge to main (opensearch-project#2375) Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * removing invalid urls as CI fails on link checker for inavlid urls in git (opensearch-project#2376) Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Address comments Signed-off-by: Kristen Tian <tyarong@amazon.com> * Fix DS snapshot test Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add https://test.com/ to lychee exclude Signed-off-by: Kristen Tian <tyarong@amazon.com> * Address comments Signed-off-by: Kristen Tian <tyarong@amazon.com> * Remove unnessacry check Signed-off-by: Kristen Tian <tyarong@amazon.com> * Remove not needed check Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Su <szhongna@amazon.com> Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com> Co-authored-by: Zhongnan Su <szhongna@amazon.com> Co-authored-by: Yan Zeng <46499415+zengyan-amazon@users.noreply.github.com> Co-authored-by: Manideep Pabba <109986843+mpabba3003@users.noreply.github.com> Co-authored-by: Yibo Wang <109543558+yibow98@users.noreply.github.com>

kavilla · 2022-09-21T21:39:45Z

...a_source_management/public/components/create_data_source_wizard/components/header/header.tsx

+            <p>
+              <FormattedMessage
+                id="dataSourcesManagement.createDataSource.description"
+                defaultMessage="A data source is an OpenSearch cluster endpoint (for now) to query against."


I would drop the (for now) it's really ambiguous.

#2400 (comment)

will address in #2434, need input from UI/UX

kavilla · 2022-09-21T21:51:46Z

src/plugins/data_source_management/public/components/text_content/text_content.ts

+  'dataSourcesManagement.dataSourcesTable.description',
+  {
+    defaultMessage:
+      'Create and manage the data sources that help you retrieve your data from multiple Elasticsearch clusters',


!!!!!!!!!!

#2400 (comment)

will address in #2434, need input from UI/UX

kavilla · 2022-09-21T22:54:34Z

Made a couple mentions about validation when connecting to a data source: #2400 (comment)

kavilla · 2022-09-21T23:09:01Z

src/plugins/index_pattern_management/opensearch_dashboards.json

  "requiredPlugins": ["management", "data", "urlForwarding"],
-  "requiredBundles": ["opensearchDashboardsReact", "opensearchDashboardsUtils"]
+  "requiredBundles": ["opensearchDashboardsReact", "opensearchDashboardsUtils", "savedObjects"]


Does this imply a breaking change?

@kavilla Why do you think this is a breaking change? All that this changes is the fact that we now use types and objects from the savedObject plugin. no feature is removed in the process.

+1 to Ashwin, no breaking change

ashwin-pc

I havent been able to get through the whole PR yet, but i was able to review the changes to the existing plugins to make sure there weren't any regressions. All my comments so far are minor and not really blockers. The change looks really good so far. Great job on isolating most of the code to the new data_source and data_source_management plugins

ashwin-pc · 2022-09-23T02:14:47Z

src/core/server/http/router/router.ts

+          opensearchDashboardsResponseFactory.badRequest({ body: e.message })
+        );
+      }
+      // TODO: add legacy data source client config error handling


Nit: I didnt quite understand this TODO. what other error handling are we missing?

handled in #2204

ashwin-pc · 2022-09-23T02:24:47Z

src/fixtures/stubbed_saved_object_index_pattern.ts

@@ -33,8 +33,8 @@ import stubbedLogstashFields from './logstash_fields';

 const mockLogstashFields = stubbedLogstashFields();

-export function stubbedSavedObjectIndexPattern(id: string | null = null) {
-  return {
+export function stubbedSavedObjectIndexPattern(id: string | null = null, withDataSource?: false) {


nit: This should be withDataSource = false.

ashwin-pc · 2022-09-23T11:02:00Z

src/plugins/data/common/index_patterns/types.ts

@@ -184,6 +185,11 @@ export interface FieldSpec {

 export type IndexPatternFieldMap = Record<string, FieldSpec>;

+export interface SavedObjectReference {


This type is already available in src/core/types/saved_objects.ts and the name property is not optional there.

@kristenTian Can you take a look at all the comments about index_pattern, and create issues then add to this meta issue for tracking? #2433

Sure. -- As for this one, the repo currently forbids import fromcoreintocommonpath. Also as you point out, the name could be optional here, hence I can rename the interface name here to avoid confusion.

ashwin-pc · 2022-09-23T11:10:31Z

src/plugins/data/common/index_patterns/index_patterns/index_patterns.ts

@@ -562,8 +567,11 @@ export class IndexPatternsService {
    }

    const body = indexPattern.getAsSavedObjectBody();
+    const references = indexPattern.getSaveObjectReference();


nit: The createSavedObject function does not have a test, but it would be nice to ensure that the flow here is being tested too.

ashwin-pc · 2022-09-23T11:22:49Z

src/plugins/data/server/search/opensearch_search/decide_client.ts

+import { OpenSearchClient, RequestHandlerContext } from 'src/core/server';
+import { IOpenSearchSearchRequest } from '..';
+
+export const decideClient = async (


Looks like this is almost identical to the decideClient function in src/plugins/data/server/index_patterns/routes.ts

ashwin-pc · 2022-09-23T11:25:42Z

src/plugins/data_source_management/README.md

@@ -0,0 +1,11 @@
+# dataSourceManagement
+
+An OpenSearch Dashboards plugin


Can we add more content here? something like

# Plugin_name Category: application, service, helper etc Brief description of what it does ## Usage How to use the plugin, ideally with code snippets or links to example plugins in `./examples` with instructions on how to use them. #``` Code snippets #``` ## Setup Contract We need this in the readme because sometimes it is quite hard to tell from the `plugin.ts` file what the start and setup contracts look like. e.g. `src/plugins/visualize/public/plugin.ts` ## Start Contract

tracked in #2199

ashwin-pc · 2022-09-23T11:26:22Z

src/plugins/data_source/README.md

@@ -0,0 +1,13 @@
+# data_source


Can we add more content here? something like

# Plugin_name Category: application, service, helper etc Brief description of what it does ## Usage How to use the plugin, ideally with code snippets or links to example plugins in `./examples` with instructions on how to use them. #``` Code snippets #``` ## Setup Contract We need this in the readme because sometimes it is quite hard to tell from the `plugin.ts` file what the start and setup contracts look like. e.g. `src/plugins/visualize/public/plugin.ts` ## Start Contract

tracked in #2199

ashwin-pc · 2022-09-23T11:40:27Z

src/plugins/index_pattern_management/opensearch_dashboards.json

  "requiredPlugins": ["management", "data", "urlForwarding"],
-  "requiredBundles": ["opensearchDashboardsReact", "opensearchDashboardsUtils"]
+  "requiredBundles": ["opensearchDashboardsReact", "opensearchDashboardsUtils", "savedObjects"]


@kavilla Why do you think this is a breaking change? All that this changes is the fact that we now use types and objects from the savedObject plugin. no feature is removed in the process.

ashwin-pc · 2022-09-23T11:56:47Z

...ugins/index_pattern_management/public/components/index_pattern_table/index_pattern_table.tsx

+    if (!dataSourceEnabled) {
+      if (!hasDataIndices && !remoteClustersExist) {


nit: These can be combined into a single if condition

ashwin-pc · 2022-09-23T11:59:24Z

.../index_pattern_management/public/components/create_index_pattern_wizard/lib/creation_flow.ts

+export const TIME_FIELD_STEP = 'TIME_FIELD_STEP';
+export const DATA_SOURCE_STEP = 'DATA_SOURCE_STEP';
+
+const CREATION_FLOW_WITH_DATA_SOURCE_MAP = new Map();


seraphjiang · 2022-09-23T21:28:56Z

let's close this as feature team has actual official backport request at #2409

kavilla requested a review from a team as a code owner September 21, 2022 17:06

kavilla commented Sep 21, 2022

View reviewed changes

zhongnansu added the multiple datasource multiple datasource project label Sep 21, 2022

kavilla commented Sep 21, 2022

View reviewed changes

ashwin-pc reviewed Sep 23, 2022

View reviewed changes

ananzh mentioned this pull request Sep 23, 2022

[Backport 2.x] MultiDataSource feature merge (#2334) #2409

Merged

seraphjiang closed this Sep 23, 2022

This was referenced Sep 28, 2022

[MD] Documentation for "data_source" plugin #2436

Closed

[MD] Documentation for "data_source_management" plugin #2437

Closed

zhongnansu mentioned this pull request Sep 29, 2022

[MD] Meta issue from feedback session & comments in closed PR #2433

Closed

7 tasks

mpabba3003 mentioned this pull request Oct 6, 2022

[MD] UX updates on data source page & validation changes #2521

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Backport 2.x] MultiDataSource feature merge (#2334) #2399

[Backport 2.x] MultiDataSource feature merge (#2334) #2399

kavilla commented Sep 21, 2022

kavilla Sep 21, 2022

zhongnansu Sep 28, 2022

kavilla Sep 21, 2022 •

edited

Loading

zhongnansu Sep 28, 2022

kavilla commented Sep 21, 2022

kavilla Sep 21, 2022

ashwin-pc Sep 23, 2022

zhongnansu Sep 28, 2022

ashwin-pc left a comment

ashwin-pc Sep 23, 2022

zhongnansu Sep 28, 2022

ashwin-pc Sep 23, 2022

ashwin-pc Sep 23, 2022

zhongnansu Sep 28, 2022 •

edited

Loading

kristenTian Sep 29, 2022

ashwin-pc Sep 23, 2022

ashwin-pc Sep 23, 2022

ashwin-pc Sep 23, 2022

zhongnansu Sep 28, 2022

ashwin-pc Sep 23, 2022

zhongnansu Sep 28, 2022

ashwin-pc Sep 23, 2022

ashwin-pc Sep 23, 2022

ashwin-pc Sep 23, 2022

seraphjiang commented Sep 23, 2022

		@@ -184,6 +185,11 @@ export interface FieldSpec {

		export type IndexPatternFieldMap = Record<string, FieldSpec>;

		export interface SavedObjectReference {

		@@ -0,0 +1,11 @@
		# dataSourceManagement

		An OpenSearch Dashboards plugin

		if (!dataSourceEnabled) {
		if (!hasDataIndices && !remoteClustersExist) {

[Backport 2.x] MultiDataSource feature merge (#2334) #2399

[Backport 2.x] MultiDataSource feature merge (#2334) #2399

Conversation

kavilla commented Sep 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kavilla Sep 21, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kavilla commented Sep 21, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ashwin-pc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhongnansu Sep 28, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seraphjiang commented Sep 23, 2022

kavilla Sep 21, 2022 •

edited

Loading

zhongnansu Sep 28, 2022 •

edited

Loading