Skip to content

Commit

Permalink
[Backport opensearch-project#6395] Do not support import data source …
Browse files Browse the repository at this point in the history
…object to Local cluster when not enable data source

Signed-off-by: yujin-emma <yujin.emma.work@gmail.com>
  • Loading branch information
yujin-emma authored and zhongnansu committed Jun 6, 2024
1 parent d260f9e commit 369c205
Show file tree
Hide file tree
Showing 10 changed files with 277 additions and 6 deletions.
31 changes: 31 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,37 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
### 📈 Features/Enhancements

- [Multiple Datasource] Add TLS configuration for multiple data sources ([#6171](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6171))
<<<<<<< HEAD
=======
- [Multiple Datasource] Add multi selectable data source component ([#6211](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6211))
- [Multiple Datasource] Do not support import data source object to Local cluster when not enable data source ([#6395](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6395))
- [Multiple Datasource] Refactor data source menu and interface to allow cleaner selection of component and related configurations ([#6256](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6256))
- [Multiple Datasource] Allow top nav menu to mount data source menu for use case when both menus are mounted ([#6268](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6268))
- [Workspace] Add create workspace page ([#6179](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6179))
- [Workspace] Add update workspace page ([#6270](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6270))
- [Multiple Datasource] Make sure customer always have a default datasource ([#6237](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6237))
- [Workspace] Add workspace list page ([#6182](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6182))
- [Workspace] Add API to duplicate saved objects among workspaces ([#6288](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6288))
- [Workspace] Add workspaces column to saved objects page ([#6225](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6225))
- [Multiple Datasource] Enhanced data source selector with default datasource shows as first choice ([#6293](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6293))
- [Multiple Datasource] Add multi data source support to sample vega visualizations ([#6218](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6218))
- [Multiple Datasource] Fetch data source title for DataSourceView when only id is provided ([#6315](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6315)
- [Multiple Datasource] Get data source label when only id is provided in DataSourceSelectable ([#6358](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6358)
- [Workspace] Add permission control logic ([#6052](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6052))
- [Multiple Datasource] Add default icon for selectable component and make sure the default datasource shows automatically ([#6327](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6327))
- [Multiple Datasource] Pass selected data sources to plugin consumers when the multi-select component initially loads ([#6333](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6333))
- [Multiple Datasource] Add installedPlugins list to data source saved object ([#6348](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6348))
- [Multiple Datasource] Add default icon in multi-selectable picker ([#6357](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6357))
- [Workspace] Add APIs to support plugin state in request ([#6303](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6303))
- [Workspace] Filter left nav menu items according to the current workspace ([#6234](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6234))
- [Multiple Datasource] Refactor data source selector component to include placeholder and add tests ([#6372](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6372))
- Replace control characters before logging ([#6402](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6402))
- [Dynamic Configurations] Improve dynamic configurations by adding cache and simplifying client fetch ([#6364](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6364))
- [CSP Handler] Update CSP handler to only query and modify frame ancestors instead of all CSP directives ([#6398](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6398))
- [MD] Add OpenSearch cluster group label to top of single selectable dropdown ([#6400](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6400))
- [Workspace] Support workspace in saved objects client in server side. ([#6365](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6365))
- [MD] Add dropdown header to data source single selector ([#6431](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6431))
>>>>>>> 97355995c8... [Multiple DataSource] Do not support import data source object to Local cluster when not enable data source (#6395)
### 🐛 Bug Fixes

Expand Down
105 changes: 103 additions & 2 deletions src/core/server/saved_objects/import/import_saved_objects.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -96,11 +96,12 @@ describe('#importSavedObjectsFromStream', () => {
let savedObjectsClient: jest.Mocked<SavedObjectsClientContract>;
let typeRegistry: jest.Mocked<ISavedObjectTypeRegistry>;
const namespace = 'some-namespace';
const testDataSourceId = 'some-datasource';
const testDataSourceId = uuidv4();

const setupOptions = (
createNewCopies: boolean = false,
dataSourceId: string | undefined = undefined
dataSourceId: string | undefined = undefined,
dataSourceEnabled: boolean | undefined = false
): SavedObjectsImportOptions => {
readStream = new Readable();
savedObjectsClient = savedObjectsClientMock.create();
Expand Down Expand Up @@ -135,6 +136,17 @@ describe('#importSavedObjectsFromStream', () => {
attributes: { title: 'some-title' },
};
};

const createDataSourceObject = (): SavedObject<{
title: string;
}> => {
return {
type: 'data-source',
id: uuidv4(),
references: [],
attributes: { title: 'some-title' },
};
};
const createError = (): SavedObjectsImportError => {
const title = 'some-title';
return {
Expand Down Expand Up @@ -589,5 +601,94 @@ describe('#importSavedObjectsFromStream', () => {
const expectedErrors = errors.map(({ type, id }) => expect.objectContaining({ type, id }));
expect(result).toEqual({ success: false, successCount: 0, errors: expectedErrors });
});

test('early return if import data source objects to non-MDS cluster', async () => {
const options = setupOptions(false, testDataSourceId, false);
const dsObj = createDataSourceObject();
const dsExportedObj = createObject(testDataSourceId);
const collectedObjects = [dsObj, dsExportedObj];

const errors = [
{
type: dsObj.type,
id: dsObj.id,
title: dsObj.attributes.title,
meta: { title: dsObj.attributes.title },
error: { type: 'unsupported_type' },
},
{
type: dsExportedObj.type,
id: dsExportedObj.id,
title: dsExportedObj.attributes.title,
meta: { title: dsExportedObj.attributes.title },
error: { type: 'unsupported_type' },
},
];
getMockFn(collectSavedObjects).mockResolvedValue({
errors: [],
collectedObjects,
importIdMap: new Map(),
});
const result = await importSavedObjectsFromStream(options);
const expectedErrors = errors.map(({ type, id }) => expect.objectContaining({ type, id }));
expect(result).toEqual({ success: false, successCount: 0, errors: expectedErrors });
});

test('early return if import mixed non/data source objects to non-MDS cluster', async () => {
const options = setupOptions(false, testDataSourceId, false);
const dsObj = createDataSourceObject();
const dsExportedObj = createObject(testDataSourceId);
const nonDsExportedObj = createObject();
const collectedObjects = [dsObj, dsExportedObj, nonDsExportedObj];

const errors = [
{
type: dsObj.type,
id: dsObj.id,
title: dsObj.attributes.title,
meta: { title: dsObj.attributes.title },
error: { type: 'unsupported_type' },
},
{
type: dsExportedObj.type,
id: dsExportedObj.id,
title: dsExportedObj.attributes.title,
meta: { title: dsExportedObj.attributes.title },
error: { type: 'unsupported_type' },
},
];
getMockFn(collectSavedObjects).mockResolvedValue({
errors: [],
collectedObjects,
importIdMap: new Map(),
});
const result = await importSavedObjectsFromStream(options);
const expectedErrors = errors.map(({ type, id }) => expect.objectContaining({ type, id }));
expect(result).toEqual({ success: false, successCount: 0, errors: expectedErrors });
});

test('early return if import single data source objects to non-MDS cluster', async () => {
const options = setupOptions(false, testDataSourceId, false);
const dsObj = createDataSourceObject();
const collectedObjects = [dsObj];

const errors = [
{
type: dsObj.type,
id: dsObj.id,
title: dsObj.attributes.title,
meta: { title: dsObj.attributes.title },
error: { type: 'unsupported_type' },
},
];
getMockFn(collectSavedObjects).mockResolvedValue({
errors: [],
collectedObjects,
importIdMap: new Map(),
});
const result = await importSavedObjectsFromStream(options);
const expectedErrors = errors.map(({ type, id }) => expect.objectContaining({ type, id }));
expect(result).toEqual({ success: false, successCount: 0, errors: expectedErrors });
});
});
});
25 changes: 25 additions & 0 deletions src/core/server/saved_objects/import/import_saved_objects.ts
Original file line number Diff line number Diff line change
Expand Up @@ -33,13 +33,15 @@ import {
SavedObjectsImportError,
SavedObjectsImportResponse,
SavedObjectsImportOptions,
SavedObjectsImportUnsupportedTypeError,
} from './types';
import { validateReferences } from './validate_references';
import { checkOriginConflicts } from './check_origin_conflicts';
import { createSavedObjects } from './create_saved_objects';
import { checkConflicts } from './check_conflicts';
import { regenerateIds } from './regenerate_ids';
import { checkConflictsForDataSource } from './check_conflict_for_data_source';
import { isSavedObjectWithDataSource } from './validate_object_id';

/**
* Import saved objects from given stream. See the {@link SavedObjectsImportOptions | options} for more
Expand All @@ -58,6 +60,7 @@ export async function importSavedObjectsFromStream({
dataSourceId,
dataSourceTitle,
workspaces,
dataSourceEnabled,
}: SavedObjectsImportOptions): Promise<SavedObjectsImportResponse> {
let errorAccumulator: SavedObjectsImportError[] = [];
const supportedTypes = typeRegistry.getImportableAndExportableTypes().map((type) => type.name);
Expand All @@ -69,6 +72,28 @@ export async function importSavedObjectsFromStream({
supportedTypes,
dataSourceId,
});
// if not enable data_source, throw error early
if (!dataSourceEnabled) {
const notSupportedErrors: SavedObjectsImportError[] = collectSavedObjectsResult.collectedObjects.reduce(
(errors: SavedObjectsImportError[], obj) => {
if (obj.type === 'data-source' || isSavedObjectWithDataSource(obj.id)) {
const error: SavedObjectsImportUnsupportedTypeError = { type: 'unsupported_type' };
const { title } = obj.attributes;
errors.push({ error, type: obj.type, id: obj.id, title, meta: { title } });
}
return errors; // Return the accumulator in each iteration
},
[]
);
if (notSupportedErrors?.length > 0) {
return {
successCount: 0,
success: false,
errors: notSupportedErrors,
};
}
}

errorAccumulator = [...errorAccumulator, ...collectSavedObjectsResult.errors];
/** Map of all IDs for objects that we are attempting to import; each value is empty by default */
let importIdMap = collectSavedObjectsResult.importIdMap;
Expand Down
2 changes: 1 addition & 1 deletion src/core/server/saved_objects/import/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ export interface SavedObjectsImportOptions {
createNewCopies: boolean;
dataSourceId?: string;
dataSourceTitle?: string;
/** if specified, will import in given workspaces */
dataSourceEnabled?: boolean;
workspaces?: SavedObjectsBaseOptions['workspaces'];
}

Expand Down
59 changes: 59 additions & 0 deletions src/core/server/saved_objects/import/validate_object_id.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/

import { isSavedObjectWithDataSource } from './validate_object_id';

describe('isObjectWithDataSource', () => {
test('should return false for valid object with data source ID but in wrong format', () => {
// Valid ID with two parts separated by underscore, and both parts being UUIDs
const inValidId = 'invalid_uuid_1234-invalid_uuid_5678';
expect(isSavedObjectWithDataSource(inValidId)).toBe(false);
});

test('should return false for invalid IDs', () => {
// Missing underscore
const invalidId1 = 'missingunderscore';
expect(isSavedObjectWithDataSource(invalidId1)).toBe(false);

// Invalid UUID in the second part
const invalidId2 = 'valid_uuid_1234-invalid_uuid';
expect(isSavedObjectWithDataSource(invalidId2)).toBe(false);

// Missing second part
const invalidId3 = 'valid_uuid_1234';
expect(isSavedObjectWithDataSource(invalidId3)).toBe(false);

// More than two parts
const invalidId4 = 'valid_uuid_1234-valid_uuid_5678-extra_part';
expect(isSavedObjectWithDataSource(invalidId4)).toBe(false);
});

test('should return false for non-UUID parts', () => {
// First part is not a UUID
const invalidId1 = 'not_a_uuid_valid_uuid_1234';
expect(isSavedObjectWithDataSource(invalidId1)).toBe(false);

// Second part is not a UUID
const invalidId2 = 'valid_uuid_1234_not_a_uuid';
expect(isSavedObjectWithDataSource(invalidId2)).toBe(false);

// Both parts are not UUIDs
const invalidId3 = 'not_a_uuid_not_a_uuid';
expect(isSavedObjectWithDataSource(invalidId3)).toBe(false);
});

test('should return false for string with underscore but not with UUID', () => {
// First part is not a UUID
const invalidId = 'saved_object_with_index_pattern_conflict';
expect(isSavedObjectWithDataSource(invalidId)).toBe(false);
});

test('should return false for string with underscore but with three UUIDs', () => {
// First part is not a UUID
const invalidId =
'7cbd2350-2223-11e8-b802-5bcf64c2cfb4_7cbd2350-2223-11e8-b802-5bcf64c2cfb4_7cbd2350-2223-11e8-b802-5bcf64c2cfb4';
expect(isSavedObjectWithDataSource(invalidId)).toBe(false);
});
});
40 changes: 40 additions & 0 deletions src/core/server/saved_objects/import/validate_object_id.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/

/**
* When enable multiple data source, exported objects from a data source will maintain object id like
* "69a34b00-9ee8-11e7-8711-e7a007dcef99_7cbd2350-2223-11e8-b802-5bcf64c2cfb4"
* two UUIDs are connected with a underscore,
* before the underscore, the UUID represents the data source
* after the underscore, the UUID is the original object id
* when disable multiple data source, the exported object from local cluster will look like 7cbd2350-2223-11e8-b802-5bcf64c2cfb4
* we can use this format to tell out whether a single object is exported from MDS enabled/disabled cluster
*
* This file to going to group some validate function to tell source of object based on the object id
*/

/**
*
* @param candidate: string without underscore
* @returns
*/
const isUUID = (candidate: string): boolean => {
// Regular expression pattern for UUID
const uuidPattern: RegExp = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
return uuidPattern.test(candidate);
};

/**
*
* @param id single object id
* @returns
*/
export const isSavedObjectWithDataSource = (id: string): boolean => {
const idParts = id.split('_');
/**
* check with the
*/
return idParts && idParts.length === 2 && idParts.every(isUUID);
};
4 changes: 4 additions & 0 deletions src/core/server/saved_objects/routes/import.ts
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@ export const registerImportRoute = (router: IRouter, config: SavedObjectConfig)
workspaces: schema.maybe(
schema.oneOf([schema.string(), schema.arrayOf(schema.string())])
),
dataSourceEnabled: schema.maybe(schema.boolean({ defaultValue: false })),
},
{
validate: (object) => {
Expand Down Expand Up @@ -116,6 +117,8 @@ export const registerImportRoute = (router: IRouter, config: SavedObjectConfig)
workspaces = [workspaces];
}

const dataSourceEnabled = req.query.dataSourceEnabled;

const result = await importSavedObjectsFromStream({
savedObjectsClient: context.core.savedObjects.client,
typeRegistry: context.core.savedObjects.typeRegistry,
Expand All @@ -126,6 +129,7 @@ export const registerImportRoute = (router: IRouter, config: SavedObjectConfig)
dataSourceId,
dataSourceTitle,
workspaces,
dataSourceEnabled,
});

return res.ok({ body: result });
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -41,14 +41,18 @@ export async function importFile(
http: HttpStart,
file: File,
{ createNewCopies, overwrite }: ImportMode,
selectedDataSourceId?: string
selectedDataSourceId?: string,
dataSourceEnabled?: boolean
) {
const formData = new FormData();
formData.append('file', file);
const query = createNewCopies ? { createNewCopies } : { overwrite };
if (selectedDataSourceId) {
query.dataSourceId = selectedDataSourceId;
}
if (dataSourceEnabled) {
query.dataSourceEnabled = dataSourceEnabled;
}
return await http.post<ImportResponse>('/api/saved_objects/_import', {
body: formData,
headers: {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,7 @@ describe('Flyout', () => {
createNewCopies: true,
overwrite: true,
},
undefined,
undefined
);
expect(component.state()).toMatchObject({
Expand Down
Loading

0 comments on commit 369c205

Please sign in to comment.