MultiDataSource feature merge (opensearch-project#2334)

* Instantiate credential management plugin code structure (opensearch-project#1996) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Data source inside stack management setup (opensearch-project#2017) (opensearch-project#2030) Signed-off-by: Kristen Tian <tyarong@amazon.com> * enable CI for feature branch (opensearch-project#2010) Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add empty data source plugin (opensearch-project#2052) Adds empty data source plugin. Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Add initial credential management CRUD pages (opensearch-project#2040) * Add credential management CRUD pages 1. List all credentials 2. Create your saved credential 3. Edit your credential 4. Delete credentials Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Register Data source to savedObject & Update license header (opensearch-project#2037) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Move credential saved object to data source plugin (opensearch-project#2062) Signed-off-by: Louis Chu <clingzhi@amazon.com> Move credential saved object to data source plugin Resolve follow up comments on UI Signed-off-by: Kristen Tian <tyarong@amazon.com> * breadcrumbfix for datasource management (opensearch-project#2066) * breadcrumbfix for datasource management Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * breadcrumbfix for datasource management - refactoring code Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * using services to update breadcrumb on data sources management page Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Changing the license header on breadcrumbs.ts datasource management Signed-off-by: Kristen Tian <tyarong@amazon.com> * Fix breadcrumb on listing page and update saved object mapping (opensearch-project#2069) Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add data source step into IndexPattern with Mock switch (opensearch-project#2064) (opensearch-project#2086) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add delete button for credential detailed page (opensearch-project#2067) Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Link datasource to indexpattern (opensearch-project#2118) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add encrypt/decrypt module on data source plugin (opensearch-project#2120) Signed-off-by: Louis Chu <clingzhi@amazon.com> 1. Add encrypt/decrypt module with UT 2. Add client factory wrapper for encrypt credential 3. Add encryption config support 4. Bugfix on Jest interpret Buffer Signed-off-by: Kristen Tian <tyarong@amazon.com> * Integrate index pattern with new data client (opensearch-project#2146) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add noAuth to dataSource attributes (opensearch-project#2154) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Datasource Management - creation & listing - UI only (opensearch-project#2128) * data source management - creation & Listing UI only * data source management - creation & Listing UI only * Create/edit data source feature * toggling default value * refactoring code as per review comments * toggling server flag to false Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Apply dataSource plugin as dependent for cm and dsm plugins (opensearch-project#2150) Signed-off-by: Louis Chu <clingzhi@amazon.com> Apply dataSource plugin as dependent for cm and dsm plugins (opensearch-project#2150) Signed-off-by: Kristen Tian <tyarong@amazon.com> * Leverage datasource enablement in index pattern management Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Add client management module and register `getClient()` to route handler context (opensearch-project#2121) * Add client management module and register `getClient()` interface to route handler context Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Make step info in index pattern creation dynamic (opensearch-project#2164) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Refactor for credential listing page & add loading effect (opensearch-project#2142) Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Integration sequal - replace data client placeholders (opensearch-project#2167) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Integrate with crypto module to decrpt password (opensearch-project#2170) Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] noAuth integration, credential & endpoint validation (opensearch-project#2165) * noAuth integration, credential & endpoint validation Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Refactoring validation message Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Adding back accidentally deleted file home/tutorials/haproxy_metrics/index.ts Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Refactor search strategy to conditionally user datasource client (opensearch-project#2171) Signed-off-by: Kristen Tian <tyarong@amazon.com> * adding relation between credential selection and no auth checkbox (opensearch-project#2175) Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update getIndicesViaSearch with datasource (opensearch-project#2176) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add null header to child client spawn (opensearch-project#2188) Signed-off-by: Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD][IP]Update button position & Add UT & Add skip option (opensearch-project#2195) * Add UT - index pattern related Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update button position Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add skip option to allow use default os data source Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD]Refactor layout and validate input fields for listing and create pages (opensearch-project#2202) Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Enable datasource link in saveObjectManagement (opensearch-project#2209) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update configure data source per UX input (opensearch-project#2235) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Enable data source audit log to file (opensearch-project#2215) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Refactor of credential editing page layout & refactor backend field validation method (opensearch-project#2222) * Refactor of credential editing page layout & refactor backend field validation method * Resolved the comments & fix the multiple call for one operation Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Tweak fetch data back to original (opensearch-project#2238) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Revamped UX for data source management (opensearch-project#2239) * revamped UX for data source management Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * refactored datasource screens as per PR comments Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Credentials security redesign (opensearch-project#2253) 1. Data model changes for data source saved object 2. Server side changes for data source saved object a. Implement data_source_saved_objects_client_wrapper to integrate with CryptographyClient for password encryption / decryption. b. Change data_source_service to fetch credentials directly from data source (still decrypt via CryptographyClient) c. Fix unit tests accordingly Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add step data source UI test (opensearch-project#2264) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD]Improve datasource server side error handling (opensearch-project#2236) Signed-off-by: Su <szhongna@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Datasource management new API change integrations (opensearch-project#2282) * changing datasource management design to integrate with new API changes Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * Update edit_data_source_form.tsx moving masked password to constants Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Update stream test to bypass CI env generate domain attribute Signed-off-by: Kristen Tian <tyarong@amazon.com> * Delete credential management Signed-off-by: Kristen Tian <tyarong@amazon.com> * Address comments Signed-off-by: Kristen Tian <tyarong@amazon.com> * [MD] Datasource Management - Create data source - Unit tests (opensearch-project#2341) * Unit test cases for data source management - create Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> * adding tests to utils.ts & changing it to test Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Fix update data source & block update endpint (opensearch-project#2364) Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * MD datasource management-datasource table-UTs (opensearch-project#2350) Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * reafctor based on PR 2334 comments to merge to main (opensearch-project#2375) Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * removing invalid urls as CI fails on link checker for inavlid urls in git (opensearch-project#2376) Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> * Address comments Signed-off-by: Kristen Tian <tyarong@amazon.com> * Fix DS snapshot test Signed-off-by: Kristen Tian <tyarong@amazon.com> * Add https://test.com/ to lychee exclude Signed-off-by: Kristen Tian <tyarong@amazon.com> * Address comments Signed-off-by: Kristen Tian <tyarong@amazon.com> * Remove unnessacry check Signed-off-by: Kristen Tian <tyarong@amazon.com> * Remove not needed check Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Kristen Tian <tyarong@amazon.com> Signed-off-by: Zhongnan Su <szhongna@amazon.com> Signed-off-by: Louis Chu <clingzhi@amazon.com> Signed-off-by: Yibo Wang <yibow@amazon.com> Signed-off-by: mpabba3003 <amazonmanideep@gmail.com> Signed-off-by: Su <szhongna@amazon.com> Co-authored-by: Louis Chu <lingzhichu.clz@gmail.com> Co-authored-by: Zhongnan Su <szhongna@amazon.com> Co-authored-by: Yan Zeng <46499415+zengyan-amazon@users.noreply.github.com> Co-authored-by: Manideep Pabba <109986843+mpabba3003@users.noreply.github.com> Co-authored-by: Yibo Wang <109543558+yibow98@users.noreply.github.com> Signed-off-by: Sergey V. Osipov <sipopo@yandex.ru>
sipopo · Dec 16, 2022 · 86a19dc · 86a19dc
1 parent 607ad19
commit 86a19dc
Show file tree

Hide file tree

Showing 142 changed files with 16,133 additions and 154 deletions.
diff --git a/.github/workflows/build_and_test_workflow.yml b/.github/workflows/build_and_test_workflow.yml
@@ -38,7 +38,7 @@ jobs:
       - name: Setup Node
         uses: actions/setup-node@v2
         with:
-          node-version-file: ".nvmrc"
+          node-version-file: '.nvmrc'
           registry-url: 'https://registry.npmjs.org'
 
       - name: Setup Yarn
@@ -77,7 +77,7 @@ jobs:
       image: docker://opensearchstaging/ci-runner:ci-runner-rockylinux8-opensearch-dashboards-integtest-v2
       options: --user 1001
     name: Run functional tests
-    strategy: 
+    strategy:
       matrix:
         group: [ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13 ]
     steps:
@@ -89,7 +89,7 @@ jobs:
       - name: Setup Node
         uses: actions/setup-node@v2
         with:
-          node-version-file: ".nvmrc"
+          node-version-file: '.nvmrc'
           registry-url: 'https://registry.npmjs.org'
 
       - name: Setup Yarn
@@ -125,7 +125,7 @@ jobs:
     defaults:
       run:
         working-directory: ./artifacts
-    strategy: 
+    strategy:
       matrix:
         include:
           - name: Linux x64
@@ -145,7 +145,7 @@ jobs:
       - name: Setup Node
         uses: actions/setup-node@v2
         with:
-          node-version-file: "./artifacts/.nvmrc"
+          node-version-file: './artifacts/.nvmrc'
           registry-url: 'https://registry.npmjs.org'
 
       - name: Setup Yarn
@@ -173,9 +173,9 @@ jobs:
           name: ${{ matrix.suffix }}-${{ env.VERSION }}
           path: ./artifacts/target/${{ env.ARTIFACT_BUILD_NAME }}
           retention-days: 1
-    
+
   bwc-tests:
-    needs: [ build-min-artifact-tests ]
+    needs: [build-min-artifact-tests]
     runs-on: ubuntu-latest
     container:
       image: docker://opensearchstaging/ci-runner:ci-runner-rockylinux8-opensearch-dashboards-integtest-v2
@@ -184,7 +184,7 @@ jobs:
     defaults:
       run:
         working-directory: ./artifacts
-    strategy: 
+    strategy:
       matrix:
         version: [ osd-2.0.0, osd-2.1.0, osd-2.2.0 ]
     steps:
@@ -199,7 +199,7 @@ jobs:
       - name: Setup Node
         uses: actions/setup-node@v2
         with:
-          node-version-file: "./artifacts/.nvmrc"
+          node-version-file: './artifacts/.nvmrc'
           registry-url: 'https://registry.npmjs.org'
 
       - name: Setup Yarn
@@ -221,15 +221,15 @@ jobs:
           if curl -I -L ${{ env.OPENSEARCH_URL }}; then
             echo "::set-output name=version-exists::true"
           fi
-      
+
       - name: Skipping tests
         if: steps.verify-opensearch-exists.outputs.version-exists != 'true'
         run: echo Tests were skipped because an OpenSearch release build does not exist for this version yet!
 
       - name: Setting environment variable to run tests for ${{ matrix.version }}
         if: steps.verify-opensearch-exists.outputs.version-exists == 'true'
         run: echo "BWC_VERSIONS=${{ matrix.version }}" >> $GITHUB_ENV
-    
+
       - name: Download OpenSearch Dashboards
         uses: actions/download-artifact@v3
         id: download
@@ -242,7 +242,7 @@ jobs:
         if: steps.verify-opensearch-exists.outputs.version-exists == 'true'
         run: |
           ./bwctest.sh -s false -o ${{ env.OPENSEARCH_URL }} -d ${{ steps.download.outputs.download-path }}/opensearch-dashboards-${{ env.VERSION }}-linux-x64.tar.gz
-      
+
       - uses: actions/upload-artifact@v3
         if: ${{ failure() && steps.verify-opensearch-exists.outputs.version-exists == 'true' }}
         with:

diff --git a/.lycheeexclude b/.lycheeexclude
@@ -35,6 +35,7 @@ http://noone.nowhere.none/
 http://bar
 http://foo
 http://test.com/
+https://test.com/
 https://manifest.foobar
 https://files.foobar/
 https://tiles.foobar/

diff --git a/package.json b/package.json
@@ -110,6 +110,7 @@
     ]
   },
   "dependencies": {
+    "@aws-crypto/client-node": "^3.1.1",
     "@elastic/datemath": "5.0.3",
     "@elastic/eui": "npm:@opensearch-project/oui@1.0.0",
     "@elastic/good": "^9.0.1-kibana3",

diff --git a/packages/osd-config-schema/src/types/stream_type.test.ts b/packages/osd-config-schema/src/types/stream_type.test.ts
@@ -68,13 +68,18 @@ test('includes namespace in failure', () => {
 describe('#defaultValue', () => {
   test('returns default when undefined', () => {
     const value = new Stream();
-    expect(schema.stream({ defaultValue: value }).validate(undefined)).toMatchInlineSnapshot(`
-      Stream {
-        "_events": Object {},
-        "_eventsCount": 0,
-        "_maxListeners": undefined,
-      }
-    `);
+    expect(schema.stream({ defaultValue: value }).validate(undefined)).toHaveProperty(
+      '_events',
+      {}
+    );
+    expect(schema.stream({ defaultValue: value }).validate(undefined)).toHaveProperty(
+      '_eventsCount',
+      0
+    );
+    expect(schema.stream({ defaultValue: value }).validate(undefined)).toHaveProperty(
+      '_maxListeners',
+      undefined
+    );
   });
 
   test('returns value when specified', () => {

diff --git a/src/core/server/http/router/router.ts b/src/core/server/http/router/router.ts
@@ -300,11 +300,23 @@ export class Router implements IRouter {
       if (LegacyOpenSearchErrorHelpers.isNotAuthorizedError(e)) {
         return e;
       }
+
+      if (isDataSourceConfigError(e)) {
+        return hapiResponseAdapter.handle(
+          opensearchDashboardsResponseFactory.badRequest({ body: e.message })
+        );
+      }
+      // TODO: add legacy data source client config error handling
+
       return hapiResponseAdapter.toInternalError();
     }
   }
 }
 
+const isDataSourceConfigError = (error: any) => {
+  return error.constructor.name === 'DataSourceConfigError' && error.statusCode === 400;
+};
+
 const convertOpenSearchUnauthorized = (
   e: OpenSearchNotAuthorizedError
 ): ErrorHttpResponseOptions => {

diff --git a/src/dev/jest/config.js b/src/dev/jest/config.js
@@ -101,4 +101,7 @@ export default {
     '<rootDir>/node_modules/enzyme-to-json/serializer',
   ],
   reporters: ['default', '<rootDir>/src/dev/jest/junit_reporter.js'],
+  globals: {
+    Uint8Array: Uint8Array,
+  },
 };
diff --git a/src/fixtures/stubbed_saved_object_index_pattern.ts b/src/fixtures/stubbed_saved_object_index_pattern.ts
@@ -33,8 +33,8 @@ import stubbedLogstashFields from './logstash_fields';
 
 const mockLogstashFields = stubbedLogstashFields();
 
-export function stubbedSavedObjectIndexPattern(id: string | null = null) {
-  return {
+export function stubbedSavedObjectIndexPattern(id: string | null = null, withDataSource?: false) {
+  const indexPattern: any = {
     id,
     type: 'index-pattern',
     attributes: {
@@ -45,4 +45,16 @@ export function stubbedSavedObjectIndexPattern(id: string | null = null) {
     },
     version: '2',
   };
+
+  if (withDataSource) {
+    indexPattern.reference = [
+      {
+        id: 'id',
+        name: 'name',
+        type: 'data-source',
+      },
+    ];
+  }
+
+  return indexPattern;
 }