[Cases] Migrate connector ID to references #104221

jonathan-buttner · 2021-07-01T21:36:36Z

Background: #105677

This PR Migrates the actions connector ID fields we have stored in the Configuration and Case saved objects. We still need to finish user actions, which will be done in a subsequent PR.

The general approach I took was to move all transformations into the service layer. There is now an external model which is returned the UI and used in most places within the backend. The internal (ES) model is used only within the service layer.

Going from the external model to the ES model

Prior to this change we were already transform the connector.fields from an object to an array of objects that had a key and value (i.e. for jira {key: 'issueType', value: 'bug'})

I moved this transformation into the service layer as well.

In addition to the fields transformation I remove the connector.id (for the configuration and cases saved objects) and move it to the references array.

Going from the ES model to the external model

Prior to this change we already transformed the connector.fields from the ES representation of an array of objects I mentioned above into an object with the fields like: {issueType: bug} to follow the example above.

This was also moved to the service layer. So after the saved object client does a get, post, update, find, etc we transform the ES model into the external model by converting the connector.fields, finding the id from the references array and placing it back in connector.id and external_service.connector_id.

Saved Object Migration

In addition to moving the transformation logic to the service layer we also need to the existing saved objects using a migration. I split out the configuration and cases migrations into their own files since the migrations.ts file was getting kind of large.

The migration removes the connector.id for the cases and configuration saved objects and moves it to the references array. For external_service.connector_id it does the same thing.

Errors

Going from the ES model to the external model

If the transformation can't find a reference called connectorId inside the references array it will set the connector field to the default none connector (this is what the cases and configuration saved objects should have if no connector is defined.)

If the transformation can't find a reference called pushConnectorId inside the references array it will set the external_service.connector_id field to null. It leaves the rest of the fields in external_service. I did this intentionally so that we wouldn't accidentally get data loss since the external_service saves a good bit of information about the push details.

Going from the external model to the ES model

If the connector.id or external_service.connector_id are value none the transformation will not create a reference for them. it will still remove the connector id field though. If they are defined but not none then it will move the value into the references field.

If the connector.id is undefined for some reason, we won't create a reference.
If the external_service is null or the external_service.connector_id is null it won't create a reference.

Testing

I tried to add good coverage using unit tests for the migration and transformations. I also added a few integration tests that look in the saved object to ensure that the ids are removed appropriately.

…ctor-id

jonathan-buttner · 2021-07-30T17:58:52Z

x-pack/plugins/cases/common/api/cases/case.ts

+ * The external service fields. Exporting here for use in the service transformation code so I can define
+ * a type without the connector_id field.
+ */
+export const CaseExternalServiceBasicRt = rt.type({


Exporting here for use in the service transformation code so I can define a type without the connector_id field.

jonathan-buttner · 2021-07-30T17:59:51Z

x-pack/plugins/cases/common/api/cases/case.ts

-  rt.intersection([
-    CaseExternalServiceBasicRt,
-    rt.type({
-      pushed_at: rt.string,


Moved these up into the basic definition above.

jonathan-buttner · 2021-07-30T18:01:08Z

x-pack/plugins/cases/common/api/cases/case.ts

-export type ESCasePatchRequest = Omit<CasePatchRequest, 'connector'> & {
-  connector?: ESCaseConnector;
-};
-


These have been moved into the cases service because they are isolated to the service layer. Other parts of cases no longer need to reference the types. Instead they will always reference the CaseAttributes or equivalent type.

jonathan-buttner · 2021-07-30T18:01:54Z

x-pack/plugins/cases/common/api/cases/configure.ts

-export type ESCasesConfigureAttributes = Omit<CasesConfigureAttributes, 'connector'> & {
-  connector: ESCaseConnector;
-};
-


This has been moved into the configure service because it is isolated to the service layer. Other parts of the cases plugin no longer need to reference the type. Instead they will reference the CasesConfigureAttributes type directly.

jonathan-buttner · 2021-07-30T18:03:21Z

x-pack/plugins/cases/common/api/connectors/index.ts

-  name: string;
-  type: ESCaseConnectorTypes;
-  fields: ESConnectorFields | null;
-}


These were moved into the service layer as well since they are only referenced by service layer functionality.

jonathan-buttner · 2021-07-30T18:36:37Z

x-pack/plugins/cases/server/services/configure/index.ts

    } catch (error) {
      this.log.debug(`Error on UPDATE case configuration ${configurationId}: ${error}`);
      throw error;
    }
  }
 }
+
+function transformUpdateResponseToExternalModel(


🔴 : Configuration saved object transformation logic

jonathan-buttner · 2021-07-30T18:37:09Z

x-pack/plugins/cases/server/services/test_utils.ts

@@ -0,0 +1,199 @@
+/*


This file holds test utility functions.

jonathan-buttner · 2021-07-30T18:37:51Z

x-pack/plugins/cases/server/services/transform.ts

@@ -0,0 +1,100 @@
+/*


🔴 : Shared transformation logic between the case and configuration saved objects

jonathan-buttner · 2021-07-30T18:38:56Z

x-pack/plugins/cases/server/services/user_actions/helpers.ts

-    field: UserActionFieldType
-  ) => {
-    return field === 'connector' && so.attributes.connector
-      ? transformESConnectorToCaseConnector(so.attributes.connector)


Since we no longer need to do the transformation, we no longer need the abstraction for getField both can just use get(so, ['attributes', field]) directly as done above.

jonathan-buttner · 2021-07-30T18:39:56Z

x-pack/test/case_api_integration/common/lib/utils.ts

@@ -605,6 +609,74 @@ export const getConnectorMappingsFromES = async ({ es }: { es: KibanaClient }) =
  return mappings;
 };

+type ESConnectorFields = Array<{


These make the integration test types easier. Since these types are no longer defined in the cases common area we need to define them here so we can access them.

I think is ok to import them from x-pack/plugins/cases/server/services/cases

Oh, I'll give that a shot, I think it was giving me an error, I suppose for this we could just ignore the eslint. Good point.

cnasikas · 2021-08-02T15:52:00Z

x-pack/plugins/cases/server/services/cases/transform.ts

+  const { connector, external_service, ...restAttributes } = caseAttributes;
+
+  let transformedAttributes: Partial<ESCaseAttributes> = { ...restAttributes };
+  let pushConnectorId: string | undefined | null;
+
+  if (external_service) {
+    let restExternalService: ExternalServicesWithoutConnectorId | null | undefined;
+    ({ connector_id: pushConnectorId, ...restExternalService } = external_service);
+    transformedAttributes = {
+      ...transformedAttributes,
+      external_service: restExternalService,
+    };
+  } else if (external_service === null) {
+    transformedAttributes = { ...transformedAttributes, external_service: null };
+  }
+
+  if (connector) {
+    transformedAttributes = {
+      ...transformedAttributes,
+      connector: {
+        name: connector.name,
+        type: connector.type,
+        fields: transformFieldsToESModel(connector),
+      },
+    };
+  }


I cannot think of a way to remove the casting (restExternalService as ExternalServicesWithoutConnectorId ). TS won this fight 😄 .

Suggested change

const { connector, external_service, ...restAttributes } = caseAttributes;

let transformedAttributes: Partial<ESCaseAttributes> = { ...restAttributes };

let pushConnectorId: string | undefined | null;

if (external_service) {

let restExternalService: ExternalServicesWithoutConnectorId | null | undefined;

({ connector_id: pushConnectorId, ...restExternalService } = external_service);

transformedAttributes = {

...transformedAttributes,

external_service: restExternalService,

};

} else if (external_service === null) {

transformedAttributes = { ...transformedAttributes, external_service: null };

}

if (connector) {

transformedAttributes = {

...transformedAttributes,

connector: {

name: connector.name,

type: connector.type,

fields: transformFieldsToESModel(connector),

},

};

}

const { connector, external_service, ...restAttributes } = caseAttributes;

const { connector_id: pushConnectorId, ...restExternalService } = external_service ?? {};

const transformedAttributes = {

...restAttributes,

...(external_service != null

? {

external_service: restExternalService as ExternalServicesWithoutConnectorId,

}

: { external_service: null }),

...(connector != null

? {

connector: {

name: connector.name,

type: connector.type,

fields: transformFieldsToESModel(connector),

},

}

: {}),

};

Haha, thanks, I took another stab at this let me know what you think.

cnasikas · 2021-08-02T16:42:54Z

x-pack/plugins/cases/server/services/cases/transform.ts

+  if (connectorId === noneConnectorId) {
+    connectorRef.push({ name: connectorIdReferenceName });
+  } else if (connectorId) {
+    connectorRef.push({
+      name: connectorIdReferenceName,
+      ref: { id: connectorId, name: connectorIdReferenceName, type: ACTION_SAVED_OBJECT_TYPE },
+    });
+  }


nit:

Suggested change

if (connectorId === noneConnectorId) {

connectorRef.push({ name: connectorIdReferenceName });

} else if (connectorId) {

connectorRef.push({

name: connectorIdReferenceName,

ref: { id: connectorId, name: connectorIdReferenceName, type: ACTION_SAVED_OBJECT_TYPE },

});

}

connectorRef.push({

name: connectorIdReferenceName,

...(connectorId !== noneConnectorId ? { ref: { id: connectorId, name: connectorIdReferenceName, type: ACTION_SAVED_OBJECT_TYPE }} : {}),

});

I refactored this with the introduction of the ConnectorReferenceHandler. Let me know what you think.

cnasikas · 2021-08-02T16:54:05Z

x-pack/plugins/cases/server/services/index.ts

+ * The name of the saved object reference indicating the action connector ID. This is stored in the Saved Object reference
+ * field's name property.
+ */
+export const connectorIdReferenceName = 'connectorId';


nit: connectorIdReferenceName -> CONNECTOR_ID_REFERENCE_NAME

Maybe we could move these variables to a constants.ts file in x-pack/plugins/cases/server

…ctor-id

jonathan-buttner · 2021-08-02T20:47:50Z

x-pack/plugins/cases/server/client/cases/update.ts

-          })
-        : { id: thisCase.id, version: thisCase.version };
-    });
+    const updateCases: UpdateRequestWithOriginalCase[] = query.cases.reduce(


This block now zips togethers the original case if it existed and the updated request. This is particularly useful for when we call the patch request so that we can populate the originalCase field. If the original case cannot be found within the map we skip the request. This shouldn't even happen though because the code above should handle finding requests that don't exist.

cnasikas · 2021-08-03T11:30:55Z

x-pack/plugins/cases/server/services/cases/index.test.ts

+        `);
+      });
+
+      it('builds references for connector_id, connector.id, and includes the existing references', async () => {


nit: for connector_id, connector.id -> for external service connector id, case connector id or we can put a comment to explain what are those in the whole file test.

cnasikas · 2021-08-03T12:48:12Z

x-pack/plugins/cases/server/services/transform.test.ts

+} from './transform';
+
+describe('service transform helpers', () => {
+  describe('findConnectorIdReference', () => {


Is it possible to have two references with the same id?

…ctor-id

kibanamachine · 2021-08-03T15:11:09Z

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`cases`	529.0KB	529.1KB	+32.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`cases`	131.7KB	131.9KB	+231.0B

Saved Objects .kibana field count

Every field in each saved object type adds overhead to Elasticsearch. Kibana needs to keep the total field count below Elasticsearch's default limit of 1000 fields. Only specify field mappings for the fields you wish to search on or query. See https://www.elastic.co/guide/en/kibana/master/development-plugin-saved-objects.html#_mappings

id	before	after	diff
`cases`	42	40	-2
`cases-configure`	20	19	-1
total			-3

Unknown metric groups

API count

id	before	after	diff
`cases`	451	443	-8

API count missing comments

id	before	after	diff
`cases`	409	401	-8

History

💚 Build #142488 succeeded 290ec82
💛 Build #142445 was flaky 17cb102
💚 Build #142446 succeeded 7689a83
💔 Build #142098 failed 13d8ffd
💚 Build #142070 succeeded d418f91

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @jonathan-buttner

cnasikas

Great job as always!

I created a few connectors on master. Then I switched to your PR to test the migration. I also played around with the UI. I could not break anything 🚀. LGTM!

kibanamachine · 2021-08-06T14:48:42Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 104221 or prevent reminders by adding the backport:skip label.

* Starting configure migration * Initial refactor of configuration connector id * Additional clean up and tests * Adding some tests * Finishing configure tests * Starting case attributes transformation refactor * adding more tests for the cases service * Adding more functionality and tests for cases migration * Finished unit tests for cases transition * Finished tests and moved types * Cleaning up type names * Fixing types and renaming * Adding more tests directly for the transformations * Fixing tests and renaming some functions * Adding transformation helper tests * Adding migration utility tests and some clean up * Begining logic to remove references when it is the none connector * Fixing merge reference bug * Addressing feedback * Changing test name and creating constants file

kibanamachine · 2021-08-09T14:57:06Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 104221 or prevent reminders by adding the backport:skip label.

kibanamachine · 2021-08-10T15:07:07Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 104221 or prevent reminders by adding the backport:skip label.

kibanamachine · 2021-08-11T15:17:07Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 104221 or prevent reminders by adding the backport:skip label.

* Starting configure migration * Initial refactor of configuration connector id * Additional clean up and tests * Adding some tests * Finishing configure tests * Starting case attributes transformation refactor * adding more tests for the cases service * Adding more functionality and tests for cases migration * Finished unit tests for cases transition * Finished tests and moved types * Cleaning up type names * Fixing types and renaming * Adding more tests directly for the transformations * Fixing tests and renaming some functions * Adding transformation helper tests * Adding migration utility tests and some clean up * Begining logic to remove references when it is the none connector * Fixing merge reference bug * Addressing feedback * Changing test name and creating constants file

jonathan-buttner added 3 commits July 1, 2021 09:20

Starting configure migration

e6dc819

Merge branch 'master' of github.com:elastic/kibana into migrate-conne…

a62c282

…ctor-id

Initial refactor of configuration connector id

ab586b0

jonathan-buttner added v8.0.0 v7.15.0 labels Jul 1, 2021

jonathan-buttner added 2 commits July 1, 2021 17:37

Merge branch 'master' of github.com:elastic/kibana into migrate-conne…

6a0d9e2

…ctor-id

Merge branch 'master' of github.com:elastic/kibana into migrate-conne…

d3be649

…ctor-id

jonathan-buttner added Feature:Cases Cases feature release_note:skip Skip the PR/issue when compiling release notes Team:Threat Hunting Security Solution Threat Hunting Team labels Jul 12, 2021

jonathan-buttner added 6 commits July 13, 2021 16:19

Additional clean up and tests

dbf84dd

Merge branch 'master' of github.com:elastic/kibana into migrate-conne…

adbb016

…ctor-id

Adding some tests

6c00c6b

Finishing configure tests

54cd2d2

Starting case attributes transformation refactor

aed1b30

adding more tests for the cases service

636c586

jonathan-buttner mentioned this pull request Jul 27, 2021

[Cases][Timeline] Saved Object Type Conversion #105677

Closed

8 tasks

jonathan-buttner added 10 commits July 27, 2021 17:21

Adding more functionality and tests for cases migration

37ef451

Finished unit tests for cases transition

48a449d

Merge branch 'master' of github.com:elastic/kibana into migrate-conne…

68137ea

…ctor-id

Finished tests and moved types

2029b4e

Cleaning up type names

3b530ba

Fixing types and renaming

f653634

Adding more tests directly for the transformations

a1a01d2

Fixing tests and renaming some functions

3affb65

Merge branch 'master' of github.com:elastic/kibana into migrate-conne…

b27d7b8

…ctor-id

Adding transformation helper tests

3be7768

jonathan-buttner commented Jul 30, 2021

View reviewed changes

Adding migration utility tests and some clean up

d418f91

jonathan-buttner requested a review from cnasikas July 30, 2021 19:11

cnasikas reviewed Aug 2, 2021

View reviewed changes

jonathan-buttner added 3 commits August 2, 2021 14:57

Fixing merge reference bug

17cb102

Merge branch 'master' of github.com:elastic/kibana into migrate-conne…

7689a83

…ctor-id

Addressing feedback

290ec82

jonathan-buttner commented Aug 2, 2021

View reviewed changes

cnasikas reviewed Aug 3, 2021

View reviewed changes

jonathan-buttner added 2 commits August 3, 2021 08:50

Changing test name and creating constants file

7f7a35b

Merge branch 'master' of github.com:elastic/kibana into migrate-conne…

eb94ffd

…ctor-id

jonathan-buttner mentioned this pull request Aug 3, 2021

[Cases] Migrate Connector ID Fields to References #107561

Closed

2 tasks

cnasikas approved these changes Aug 4, 2021

View reviewed changes

jonathan-buttner merged commit 96f27b9 into elastic:master Aug 4, 2021

jonathan-buttner deleted the migrate-connector-id branch August 4, 2021 14:39

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Aug 6, 2021

jonathan-buttner mentioned this pull request Aug 11, 2021

[7.x] [Cases] Migrate connector ID to references (#104221) #108266

Merged

jonathan-buttner mentioned this pull request Aug 11, 2021

[Cases] Migrate user actions connector ID #108272

Merged

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Aug 11, 2021

jonathan-buttner mentioned this pull request Aug 18, 2021

[Cases] Migrate Connector ID Fields to References #109177

Closed

jonathan-buttner mentioned this pull request Sep 21, 2021

[Cases] Migrate user actions connector ID V2 #112710

Merged

cnasikas mentioned this pull request Sep 30, 2021

[Cases][Connectors] Migrate to using Saved Object References array to reference action saved objects from Cases #107067

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Cases] Migrate connector ID to references #104221

[Cases] Migrate connector ID to references #104221

jonathan-buttner commented Jul 1, 2021 •

edited

Loading

jonathan-buttner Jul 30, 2021

jonathan-buttner Jul 30, 2021

jonathan-buttner Jul 30, 2021

jonathan-buttner Jul 30, 2021

jonathan-buttner Jul 30, 2021

jonathan-buttner Jul 30, 2021

jonathan-buttner Jul 30, 2021

jonathan-buttner Jul 30, 2021

jonathan-buttner Jul 30, 2021

jonathan-buttner Jul 30, 2021

cnasikas Aug 2, 2021

jonathan-buttner Aug 2, 2021

cnasikas Aug 2, 2021

jonathan-buttner Aug 2, 2021

cnasikas Aug 2, 2021

jonathan-buttner Aug 2, 2021

cnasikas Aug 2, 2021

jonathan-buttner Aug 2, 2021

cnasikas Aug 3, 2021 •

edited

Loading

cnasikas Aug 3, 2021

kibanamachine commented Aug 3, 2021

API count

API count missing comments

cnasikas left a comment

kibanamachine commented Aug 6, 2021

kibanamachine commented Aug 9, 2021

kibanamachine commented Aug 10, 2021

kibanamachine commented Aug 11, 2021

[Cases] Migrate connector ID to references #104221

[Cases] Migrate connector ID to references #104221

Conversation

jonathan-buttner commented Jul 1, 2021 • edited Loading

Going from the external model to the ES model

Going from the ES model to the external model

Saved Object Migration

Errors

Going from the ES model to the external model

Going from the external model to the ES model

Testing

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cnasikas Aug 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibanamachine commented Aug 3, 2021

💚 Build Succeeded

Metrics [docs]

Async chunks

Page load bundle

Saved Objects .kibana field count

API count

API count missing comments

History

cnasikas left a comment

Choose a reason for hiding this comment

kibanamachine commented Aug 6, 2021

kibanamachine commented Aug 9, 2021

kibanamachine commented Aug 10, 2021

kibanamachine commented Aug 11, 2021

jonathan-buttner commented Jul 1, 2021 •

edited

Loading

cnasikas Aug 3, 2021 •

edited

Loading