[MD] Apply dataSource plugin as dependent for cm and dsm plugins #2150

noCharger · 2022-08-16T22:21:46Z

Signed-off-by: Louis Chu clingzhi@amazon.com

Description

Apply dataSource plugin as dependent for cm and dsm plugins

Default data_source.enabled is false

After update data_source.enabled to true in osd.yml

Issues Resolved

#1978

Check List

Signed-off-by: Louis Chu <clingzhi@amazon.com>

seraphjiang

where is code for data_source.enabled

seraphjiang · 2022-08-17T07:00:51Z

Help me understand why we need this PR? and where is implementation for data_source.enabled

noCharger · 2022-08-17T07:11:18Z

where is code for data_source.enabled

Previous PR defines the config. During OSD launch, the plugin dependency topology is built. When the dataSource plugin is set as a required plugin (See src/plugins/credential_management/opensearch_dashboards.json)and data source.enabled is false, the cm and dsm plugins will not be built, and vise versa.

zengyan-amazon · 2022-08-17T07:11:38Z

src/plugins/credential_management/public/components/edit_credential/edit_credential.tsx

@@ -25,7 +25,7 @@ import {
  EuiConfirmModal,
 } from '@elastic/eui';
 import { DocLinksStart } from 'src/core/public';
-import { Credential } from '../../../../data_source/common';


do we move Credential type from common to public ?

The Credential type is still defined in common path, but because the CM plugin now depends on the DS plugin, when importing a non-public component, it will report the following error.

ERROR in ./public/components/create_credential_wizard/create_credential_wizard.tsx │ Module not found: Error: import [../../../../data_source/common] references a non-public export of the [dataSource] bundle and must point to one of the public directories: [public] │ @ ./public/components/create_credential_wizard/create_credential_wizard.tsx 11:0-60 89:13-23 │ @ ./public/components/create_credential_wizard/index.tsx │ @ ./public/components/index.tsx │ @ ./public/management_app/mount_management_section.tsx │ @ ./public/management_app/index.tsx │ @ ./public/plugin.ts │ @ ./public/index.ts │ @ /Volumes/workplace/DEV/packages/osd-optimizer/target/worker/entry_point_creator.js

zengyan-amazon

looks good to me overall

seraphjiang · 2022-08-17T07:21:02Z

Previous PR defines the config. During OSD launch, the plugin dependency topology is built. When the dataSource plugin is set as a required plugin (See src/plugins/credential_management/opensearch_dashboards.json)and data source.enabled is false, the cm and dsm plugins will not be built, and vise versa.

it is not good practice to mix two critical functionalities in one PR. the encryption PR should not include any code for feature flag

seraphjiang · 2022-08-17T07:31:38Z

please create PR to add doc for feature flag. https://github.com/opensearch-project/OpenSearch-Dashboards/tree/feature/multi-datasource/src/plugins/data_source
communicate to @kristenTian @zengyan-amazon about what is strategy for functional test, should it be enabled to run ftr test. and do we need to change https://github.com/opensearch-project/OpenSearch-Dashboards/blob/2d6eef9679dcc225757873d50d40dccc8abeceea/test/common/config.js
any impact to docker generator https://github.com/opensearch-project/OpenSearch-Dashboards/blob/2d6eef9679dcc225757873d50d40dccc8abeceea/src/dev/build/tasks/os_packages/docker_generator/resources/bin/opensearch-dashboards-docker

noCharger · 2022-08-17T07:32:38Z

Previous PR defines the config. During OSD launch, the plugin dependency topology is built. When the dataSource plugin is set as a required plugin (See src/plugins/credential_management/opensearch_dashboards.json)and data source.enabled is false, the cm and dsm plugins will not be built, and vise versa.

it is not good practice to mix two critical functionalities in one PR. the encryption PR should not include any code for feature flag

The feature flag was not added in encryption PR. Check the commit history - 2abcc21

…ch-project#2150) Signed-off-by: Louis Chu <clingzhi@amazon.com> Apply dataSource plugin as dependent for cm and dsm plugins (opensearch-project#2150)

…ch-project#2150) Signed-off-by: Louis Chu <clingzhi@amazon.com> Apply dataSource plugin as dependent for cm and dsm plugins (opensearch-project#2150) Signed-off-by: Kristen Tian <tyarong@amazon.com>

* Instantiate credential management plugin code structure (#1996) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Data source inside stack management setup (#2017) (#2030) Signed-off-by: Kristen Tian <tyarong@amazon.com> * enable CI for feature branch (#2010) Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add empty data source plugin (#2052) Adds empty data source plugin. Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Add initial credential management CRUD pages (#2040) * Add credential management CRUD pages 1. List all credentials 2. Create your saved credential 3. Edit your credential 4. Delete credentials Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Register Data source to savedObject & Update license header (#2037) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Move credential saved object to data source plugin (#2062) Signed-off-by: Louis Chu <clingzhi@amazon.com> Move credential saved object to data source plugin Resolve follow up comments on UI Signed-off-by: Kristen Tian <tyarong@amazon.com> * breadcrumbfix for datasource management (#2066) * breadcrumbfix for datasource management Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * breadcrumbfix for datasource management - refactoring code Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * using services to update breadcrumb on data sources management page Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Changing the license header on breadcrumbs.ts datasource management Signed-off-by: Kristen Tian <tyarong@amazon.com> * Fix breadcrumb on listing page and update saved object mapping (#2069) Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add data source step into IndexPattern with Mock switch (#2064) (#2086) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add delete button for credential detailed page (#2067) Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Link datasource to indexpattern (#2118) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add encrypt/decrypt module on data source plugin (#2120) Signed-off-by: Louis Chu <clingzhi@amazon.com> 1. Add encrypt/decrypt module with UT 2. Add client factory wrapper for encrypt credential 3. Add encryption config support 4. Bugfix on Jest interpret Buffer Signed-off-by: Kristen Tian <tyarong@amazon.com> * Integrate index pattern with new data client (#2146) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add noAuth to dataSource attributes (#2154) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Datasource Management - creation & listing - UI only (#2128) * data source management - creation & Listing UI only * data source management - creation & Listing UI only * Create/edit data source feature * toggling default value * refactoring code as per review comments * toggling server flag to false Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Apply dataSource plugin as dependent for cm and dsm plugins (#2150) Signed-off-by: Louis Chu <clingzhi@amazon.com> Apply dataSource plugin as dependent for cm and dsm plugins (#2150) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Leverage datasource enablement in index pattern management Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Add client management module and register `getClient()` to route handler context (#2121) * Add client management module and register `getClient()` interface to route handler context Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Make step info in index pattern creation dynamic (#2164) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Refactor for credential listing page & add loading effect (#2142) Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Integration sequal - replace data client placeholders (#2167) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Integrate with crypto module to decrpt password (#2170) Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] noAuth integration, credential & endpoint validation (#2165) * noAuth integration, credential & endpoint validation Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Refactoring validation message Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Adding back accidentally deleted file home/tutorials/haproxy_metrics/index.ts Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Refactor search strategy to conditionally user datasource client (#2171) Signed-off-by: Kristen Tian <tyarong@amazon.com> * adding relation between credential selection and no auth checkbox (#2175) Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update getIndicesViaSearch with datasource (#2176) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add null header to child client spawn (#2188) Signed-off-by: Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD][IP]Update button position & Add UT & Add skip option (#2195) * Add UT - index pattern related Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update button position Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add skip option to allow use default os data source Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD]Refactor layout and validate input fields for listing and create pages (#2202) Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Enable datasource link in saveObjectManagement (#2209) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update configure data source per UX input (#2235) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Enable data source audit log to file (#2215) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Refactor of credential editing page layout & refactor backend field validation method (#2222) * Refactor of credential editing page layout & refactor backend field validation method * Resolved the comments & fix the multiple call for one operation Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Tweak fetch data back to original (#2238) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Revamped UX for data source management (#2239) * revamped UX for data source management Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * refactored datasource screens as per PR comments Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Credentials security redesign (#2253) 1. Data model changes for data source saved object 2. Server side changes for data source saved object a. Implement data_source_saved_objects_client_wrapper to integrate with CryptographyClient for password encryption / decryption. b. Change data_source_service to fetch credentials directly from data source (still decrypt via CryptographyClient) c. Fix unit tests accordingly Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add step data source UI test (#2264) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD]Improve datasource server side error handling (#2236) Signed-off-by: Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Datasource management new API change integrations (#2282) * changing datasource management design to integrate with new API changes Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Update edit_data_source_form.tsx moving masked password to constants Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update stream test to bypass CI env generate domain attribute Signed-off-by: Kristen Tian <tyarong@amazon.com> * Delete credential management Signed-off-by: Kristen Tian <tyarong@amazon.com> * Address comments Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Datasource Management - Create data source - Unit tests (#2341) * Unit test cases for data source management - create Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * adding tests to utils.ts & changing it to test Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Fix update data source & block update endpint (#2364) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * MD datasource management-datasource table-UTs (#2350) Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * reafctor based on PR 2334 comments to merge to main (#2375) Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * removing invalid urls as CI fails on link checker for inavlid urls in git (#2376) Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Address comments Signed-off-by: Kristen Tian <tyarong@amazon.com> * Fix DS snapshot test Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add https://test.com/ to lychee exclude Signed-off-by: Kristen Tian <tyarong@amazon.com> * Address comments Signed-off-by: Kristen Tian <tyarong@amazon.com> * Remove unnessacry check Signed-off-by: Kristen Tian <tyarong@amazon.com> * Remove not needed check Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Su <szhongna@amazon.com> Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com> Co-authored-by: Zhongnan Su <szhongna@amazon.com> Co-authored-by: Yan Zeng <46499415+zengyan-amazon@users.noreply.github.com> Co-authored-by: Manideep Pabba <109986843+mpabba3003@users.noreply.github.com> Co-authored-by: Yibo Wang <109543558+yibow98@users.noreply.github.com>

* Instantiate credential management plugin code structure (opensearch-project#1996) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Data source inside stack management setup (opensearch-project#2017) (opensearch-project#2030) Signed-off-by: Kristen Tian <tyarong@amazon.com> * enable CI for feature branch (opensearch-project#2010) Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add empty data source plugin (opensearch-project#2052) Adds empty data source plugin. Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Add initial credential management CRUD pages (opensearch-project#2040) * Add credential management CRUD pages 1. List all credentials 2. Create your saved credential 3. Edit your credential 4. Delete credentials Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Register Data source to savedObject & Update license header (opensearch-project#2037) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Move credential saved object to data source plugin (opensearch-project#2062) Signed-off-by: Louis Chu <clingzhi@amazon.com> Move credential saved object to data source plugin Resolve follow up comments on UI Signed-off-by: Kristen Tian <tyarong@amazon.com> * breadcrumbfix for datasource management (opensearch-project#2066) * breadcrumbfix for datasource management Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * breadcrumbfix for datasource management - refactoring code Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * using services to update breadcrumb on data sources management page Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Changing the license header on breadcrumbs.ts datasource management Signed-off-by: Kristen Tian <tyarong@amazon.com> * Fix breadcrumb on listing page and update saved object mapping (opensearch-project#2069) Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add data source step into IndexPattern with Mock switch (opensearch-project#2064) (opensearch-project#2086) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add delete button for credential detailed page (opensearch-project#2067) Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Link datasource to indexpattern (opensearch-project#2118) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add encrypt/decrypt module on data source plugin (opensearch-project#2120) Signed-off-by: Louis Chu <clingzhi@amazon.com> 1. Add encrypt/decrypt module with UT 2. Add client factory wrapper for encrypt credential 3. Add encryption config support 4. Bugfix on Jest interpret Buffer Signed-off-by: Kristen Tian <tyarong@amazon.com> * Integrate index pattern with new data client (opensearch-project#2146) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add noAuth to dataSource attributes (opensearch-project#2154) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Datasource Management - creation & listing - UI only (opensearch-project#2128) * data source management - creation & Listing UI only * data source management - creation & Listing UI only * Create/edit data source feature * toggling default value * refactoring code as per review comments * toggling server flag to false Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Apply dataSource plugin as dependent for cm and dsm plugins (opensearch-project#2150) Signed-off-by: Louis Chu <clingzhi@amazon.com> Apply dataSource plugin as dependent for cm and dsm plugins (opensearch-project#2150) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Leverage datasource enablement in index pattern management Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Add client management module and register `getClient()` to route handler context (opensearch-project#2121) * Add client management module and register `getClient()` interface to route handler context Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Make step info in index pattern creation dynamic (opensearch-project#2164) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Refactor for credential listing page & add loading effect (opensearch-project#2142) Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Integration sequal - replace data client placeholders (opensearch-project#2167) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Integrate with crypto module to decrpt password (opensearch-project#2170) Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] noAuth integration, credential & endpoint validation (opensearch-project#2165) * noAuth integration, credential & endpoint validation Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Refactoring validation message Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Adding back accidentally deleted file home/tutorials/haproxy_metrics/index.ts Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Refactor search strategy to conditionally user datasource client (opensearch-project#2171) Signed-off-by: Kristen Tian <tyarong@amazon.com> * adding relation between credential selection and no auth checkbox (opensearch-project#2175) Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update getIndicesViaSearch with datasource (opensearch-project#2176) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add null header to child client spawn (opensearch-project#2188) Signed-off-by: Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD][IP]Update button position & Add UT & Add skip option (opensearch-project#2195) * Add UT - index pattern related Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update button position Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add skip option to allow use default os data source Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD]Refactor layout and validate input fields for listing and create pages (opensearch-project#2202) Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Enable datasource link in saveObjectManagement (opensearch-project#2209) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update configure data source per UX input (opensearch-project#2235) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Enable data source audit log to file (opensearch-project#2215) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Refactor of credential editing page layout & refactor backend field validation method (opensearch-project#2222) * Refactor of credential editing page layout & refactor backend field validation method * Resolved the comments & fix the multiple call for one operation Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Tweak fetch data back to original (opensearch-project#2238) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Revamped UX for data source management (opensearch-project#2239) * revamped UX for data source management Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * refactored datasource screens as per PR comments Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Credentials security redesign (opensearch-project#2253) 1. Data model changes for data source saved object 2. Server side changes for data source saved object a. Implement data_source_saved_objects_client_wrapper to integrate with CryptographyClient for password encryption / decryption. b. Change data_source_service to fetch credentials directly from data source (still decrypt via CryptographyClient) c. Fix unit tests accordingly Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add step data source UI test (opensearch-project#2264) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD]Improve datasource server side error handling (opensearch-project#2236) Signed-off-by: Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Datasource management new API change integrations (opensearch-project#2282) * changing datasource management design to integrate with new API changes Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Update edit_data_source_form.tsx moving masked password to constants Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update stream test to bypass CI env generate domain attribute Signed-off-by: Kristen Tian <tyarong@amazon.com> * Delete credential management Signed-off-by: Kristen Tian <tyarong@amazon.com> * Address comments Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Datasource Management - Create data source - Unit tests (opensearch-project#2341) * Unit test cases for data source management - create Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * adding tests to utils.ts & changing it to test Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Fix update data source & block update endpint (opensearch-project#2364) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * MD datasource management-datasource table-UTs (opensearch-project#2350) Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * reafctor based on PR 2334 comments to merge to main (opensearch-project#2375) Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * removing invalid urls as CI fails on link checker for inavlid urls in git (opensearch-project#2376) Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Address comments Signed-off-by: Kristen Tian <tyarong@amazon.com> * Fix DS snapshot test Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add https://test.com/ to lychee exclude Signed-off-by: Kristen Tian <tyarong@amazon.com> * Address comments Signed-off-by: Kristen Tian <tyarong@amazon.com> * Remove unnessacry check Signed-off-by: Kristen Tian <tyarong@amazon.com> * Remove not needed check Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Su <szhongna@amazon.com> Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com> Co-authored-by: Zhongnan Su <szhongna@amazon.com> Co-authored-by: Yan Zeng <46499415+zengyan-amazon@users.noreply.github.com> Co-authored-by: Manideep Pabba <109986843+mpabba3003@users.noreply.github.com> Co-authored-by: Yibo Wang <109543558+yibow98@users.noreply.github.com>

* Instantiate credential management plugin code structure (opensearch-project#1996) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Data source inside stack management setup (opensearch-project#2017) (opensearch-project#2030) Signed-off-by: Kristen Tian <tyarong@amazon.com> * enable CI for feature branch (opensearch-project#2010) Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add empty data source plugin (opensearch-project#2052) Adds empty data source plugin. Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Add initial credential management CRUD pages (opensearch-project#2040) * Add credential management CRUD pages 1. List all credentials 2. Create your saved credential 3. Edit your credential 4. Delete credentials Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Register Data source to savedObject & Update license header (opensearch-project#2037) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Move credential saved object to data source plugin (opensearch-project#2062) Signed-off-by: Louis Chu <clingzhi@amazon.com> Move credential saved object to data source plugin Resolve follow up comments on UI Signed-off-by: Kristen Tian <tyarong@amazon.com> * breadcrumbfix for datasource management (opensearch-project#2066) * breadcrumbfix for datasource management Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * breadcrumbfix for datasource management - refactoring code Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * using services to update breadcrumb on data sources management page Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Changing the license header on breadcrumbs.ts datasource management Signed-off-by: Kristen Tian <tyarong@amazon.com> * Fix breadcrumb on listing page and update saved object mapping (opensearch-project#2069) Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add data source step into IndexPattern with Mock switch (opensearch-project#2064) (opensearch-project#2086) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add delete button for credential detailed page (opensearch-project#2067) Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Link datasource to indexpattern (opensearch-project#2118) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add encrypt/decrypt module on data source plugin (opensearch-project#2120) Signed-off-by: Louis Chu <clingzhi@amazon.com> 1. Add encrypt/decrypt module with UT 2. Add client factory wrapper for encrypt credential 3. Add encryption config support 4. Bugfix on Jest interpret Buffer Signed-off-by: Kristen Tian <tyarong@amazon.com> * Integrate index pattern with new data client (opensearch-project#2146) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add noAuth to dataSource attributes (opensearch-project#2154) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Datasource Management - creation & listing - UI only (opensearch-project#2128) * data source management - creation & Listing UI only * data source management - creation & Listing UI only * Create/edit data source feature * toggling default value * refactoring code as per review comments * toggling server flag to false Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Apply dataSource plugin as dependent for cm and dsm plugins (opensearch-project#2150) Signed-off-by: Louis Chu <clingzhi@amazon.com> Apply dataSource plugin as dependent for cm and dsm plugins (opensearch-project#2150) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Leverage datasource enablement in index pattern management Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Add client management module and register `getClient()` to route handler context (opensearch-project#2121) * Add client management module and register `getClient()` interface to route handler context Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Make step info in index pattern creation dynamic (opensearch-project#2164) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Refactor for credential listing page & add loading effect (opensearch-project#2142) Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Integration sequal - replace data client placeholders (opensearch-project#2167) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Integrate with crypto module to decrpt password (opensearch-project#2170) Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] noAuth integration, credential & endpoint validation (opensearch-project#2165) * noAuth integration, credential & endpoint validation Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Refactoring validation message Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Adding back accidentally deleted file home/tutorials/haproxy_metrics/index.ts Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Refactor search strategy to conditionally user datasource client (opensearch-project#2171) Signed-off-by: Kristen Tian <tyarong@amazon.com> * adding relation between credential selection and no auth checkbox (opensearch-project#2175) Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update getIndicesViaSearch with datasource (opensearch-project#2176) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add null header to child client spawn (opensearch-project#2188) Signed-off-by: Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD][IP]Update button position & Add UT & Add skip option (opensearch-project#2195) * Add UT - index pattern related Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update button position Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add skip option to allow use default os data source Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD]Refactor layout and validate input fields for listing and create pages (opensearch-project#2202) Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Enable datasource link in saveObjectManagement (opensearch-project#2209) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update configure data source per UX input (opensearch-project#2235) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Enable data source audit log to file (opensearch-project#2215) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Refactor of credential editing page layout & refactor backend field validation method (opensearch-project#2222) * Refactor of credential editing page layout & refactor backend field validation method * Resolved the comments & fix the multiple call for one operation Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Tweak fetch data back to original (opensearch-project#2238) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Revamped UX for data source management (opensearch-project#2239) * revamped UX for data source management Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * refactored datasource screens as per PR comments Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Credentials security redesign (opensearch-project#2253) 1. Data model changes for data source saved object 2. Server side changes for data source saved object a. Implement data_source_saved_objects_client_wrapper to integrate with CryptographyClient for password encryption / decryption. b. Change data_source_service to fetch credentials directly from data source (still decrypt via CryptographyClient) c. Fix unit tests accordingly Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add step data source UI test (opensearch-project#2264) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD]Improve datasource server side error handling (opensearch-project#2236) Signed-off-by: Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Datasource management new API change integrations (opensearch-project#2282) * changing datasource management design to integrate with new API changes Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Update edit_data_source_form.tsx moving masked password to constants Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update stream test to bypass CI env generate domain attribute Signed-off-by: Kristen Tian <tyarong@amazon.com> * Delete credential management Signed-off-by: Kristen Tian <tyarong@amazon.com> * Address comments Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Datasource Management - Create data source - Unit tests (opensearch-project#2341) * Unit test cases for data source management - create Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * adding tests to utils.ts & changing it to test Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Fix update data source & block update endpint (opensearch-project#2364) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * MD datasource management-datasource table-UTs (opensearch-project#2350) Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * reafctor based on PR 2334 comments to merge to main (opensearch-project#2375) Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * removing invalid urls as CI fails on link checker for inavlid urls in git (opensearch-project#2376) Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Address comments Signed-off-by: Kristen Tian <tyarong@amazon.com> * Fix DS snapshot test Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add https://test.com/ to lychee exclude Signed-off-by: Kristen Tian <tyarong@amazon.com> * Address comments Signed-off-by: Kristen Tian <tyarong@amazon.com> * Remove unnessacry check Signed-off-by: Kristen Tian <tyarong@amazon.com> * Remove not needed check Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Su <szhongna@amazon.com> Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com> Co-authored-by: Zhongnan Su <szhongna@amazon.com> Co-authored-by: Yan Zeng <46499415+zengyan-amazon@users.noreply.github.com> Co-authored-by: Manideep Pabba <109986843+mpabba3003@users.noreply.github.com> Co-authored-by: Yibo Wang <109543558+yibow98@users.noreply.github.com> Signed-off-by: Sergey V. Osipov <sipopo@yandex.ru>

noCharger self-assigned this Aug 16, 2022

noCharger added the multiple datasource multiple datasource project label Aug 16, 2022

noCharger force-pushed the md-config-louis-dev branch from d2d337c to 189b267 Compare August 17, 2022 06:35

noCharger marked this pull request as ready for review August 17, 2022 06:47

noCharger requested a review from a team as a code owner August 17, 2022 06:47

noCharger requested review from seraphjiang, zengyan-amazon, zhongnansu and kristenTian August 17, 2022 06:47

Apply dataSource plugin as dependent for cm and dsm plugins

97e1054

Signed-off-by: Louis Chu <clingzhi@amazon.com>

noCharger force-pushed the md-config-louis-dev branch from 189b267 to 97e1054 Compare August 17, 2022 06:49

seraphjiang suggested changes Aug 17, 2022

View reviewed changes

zengyan-amazon reviewed Aug 17, 2022

View reviewed changes

noCharger requested review from zengyan-amazon and seraphjiang August 17, 2022 07:16

seraphjiang approved these changes Aug 17, 2022

View reviewed changes

seraphjiang changed the title ~~Apply dataSource plugin as dependent for cm and dsm plugins~~ [MD]Apply dataSource plugin as dependent for cm and dsm plugins Aug 17, 2022

zengyan-amazon approved these changes Aug 17, 2022

View reviewed changes

Merge branch 'feature/multi-datasource' into md-config-louis-dev

86f5de8

noCharger merged commit 44e35f8 into opensearch-project:feature/multi-datasource Aug 17, 2022

noCharger mentioned this pull request Aug 17, 2022

[MD]Credential management in stack management #1733

Closed

54 tasks

noCharger changed the title ~~[MD]Apply dataSource plugin as dependent for cm and dsm plugins~~ [MD] Apply dataSource plugin as dependent for cm and dsm plugins Aug 17, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MD] Apply dataSource plugin as dependent for cm and dsm plugins #2150

[MD] Apply dataSource plugin as dependent for cm and dsm plugins #2150

noCharger commented Aug 16, 2022 •

edited

Loading

seraphjiang left a comment

seraphjiang commented Aug 17, 2022

noCharger commented Aug 17, 2022 •

edited

Loading

zengyan-amazon Aug 17, 2022

noCharger Aug 17, 2022

zengyan-amazon left a comment

seraphjiang commented Aug 17, 2022

seraphjiang commented Aug 17, 2022

noCharger commented Aug 17, 2022

[MD] Apply dataSource plugin as dependent for cm and dsm plugins #2150

[MD] Apply dataSource plugin as dependent for cm and dsm plugins #2150

Conversation

noCharger commented Aug 16, 2022 • edited Loading

Description

Issues Resolved

Check List

seraphjiang left a comment

Choose a reason for hiding this comment

seraphjiang commented Aug 17, 2022

noCharger commented Aug 17, 2022 • edited Loading

zengyan-amazon Aug 17, 2022

Choose a reason for hiding this comment

noCharger Aug 17, 2022

Choose a reason for hiding this comment

zengyan-amazon left a comment

Choose a reason for hiding this comment

seraphjiang commented Aug 17, 2022

seraphjiang commented Aug 17, 2022

noCharger commented Aug 17, 2022

noCharger commented Aug 16, 2022 •

edited

Loading

noCharger commented Aug 17, 2022 •

edited

Loading