Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Security] Adds pre-packaged rule updates through the "Prebuilt Security Detection Rules" Fleet integration #96698

Merged
merged 11 commits into from
Apr 14, 2021
Merged

[Security] Adds pre-packaged rule updates through the "Prebuilt Security Detection Rules" Fleet integration #96698

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
850856d
Make the prepackaged rules functions async
rw-access Apr 9, 2021
38ba797
Fix type for getPrepackagedRules mock
rw-access Apr 9, 2021
93e3159
Install updates from saved objects & FS
rw-access Apr 9, 2021
4a855ee
Merge remote-tracking branch 'origin/master' into install-security-ru…
rw-access Apr 9, 2021
a1febb9
Mock getLatestPrepackagedRules instead of getPrepackagedRules
rw-access Apr 10, 2021
dc48e1a
Cleanup ruleAssetSavedObjectsClientFactory.all
rw-access Apr 12, 2021
1e56114
Fix comment for "most recent version"
rw-access Apr 12, 2021
3df98ad
Switch to ruleMap.get() for less typescript errors
rw-access Apr 12, 2021
84248b3
Remove unneeded constants
rw-access Apr 12, 2021
41536f0
Fix SO.attributes sig and use custom validation
rw-access Apr 12, 2021
df517f8
Merge branch 'master' into install-security-rules-via-pkg
kibanamachine Apr 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...ity_solution/server/lib/detection_engine/routes/rules/add_prepackaged_rules_route.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ import { elasticsearchClientMock } from 'src/core/server/elasticsearch/client/mo

jest.mock('../../rules/get_prepackaged_rules', () => {
return {
getPrepackagedRules: (): AddPrepackagedRulesSchemaDecoded[] => {
getLatestPrepackagedRules: async (): Promise<AddPrepackagedRulesSchemaDecoded[]> => {
return [
{
author: ['Elastic'],
Expand Down
11 changes: 6 additions & 5 deletions ...security_solution/server/lib/detection_engine/routes/rules/add_prepackaged_rules_route.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,13 @@ import { SetupPlugins } from '../../../../plugin';
import { buildFrameworkRequest } from '../../../timeline/utils/common';

import { getIndexExists } from '../../index/get_index_exists';
import { getPrepackagedRules } from '../../rules/get_prepackaged_rules';
import { getLatestPrepackagedRules } from '../../rules/get_prepackaged_rules';
import { installPrepackagedRules } from '../../rules/install_prepacked_rules';
import { updatePrepackagedRules } from '../../rules/update_prepacked_rules';
import { getRulesToInstall } from '../../rules/get_rules_to_install';
import { getRulesToUpdate } from '../../rules/get_rules_to_update';
import { getExistingPrepackagedRules } from '../../rules/get_existing_prepackaged_rules';
import { ruleAssetSavedObjectsClientFactory } from '../../rules/rule_asset_saved_objects_client';

import { transformError, buildSiemResponse } from '../utils';
import { AlertsClient } from '../../../../../../alerting/server';
Expand Down Expand Up @@ -110,7 +111,7 @@ export const createPrepackagedRules = async (
const savedObjectsClient = context.core.savedObjects.client;
const exceptionsListClient =
context.lists != null ? context.lists.getExceptionListClient() : exceptionsClient;

const ruleAssetsClient = ruleAssetSavedObjectsClientFactory(savedObjectsClient);
if (!siemClient || !alertsClient) {
throw new PrepackagedRulesError('', 404);
}
Expand All @@ -120,10 +121,10 @@ export const createPrepackagedRules = async (
await exceptionsListClient.createEndpointList();
}

const rulesFromFileSystem = getPrepackagedRules();
const latestPrepackagedRules = await getLatestPrepackagedRules(ruleAssetsClient);
const prepackagedRules = await getExistingPrepackagedRules({ alertsClient });
const rulesToInstall = getRulesToInstall(rulesFromFileSystem, prepackagedRules);
const rulesToUpdate = getRulesToUpdate(rulesFromFileSystem, prepackagedRules);
const rulesToInstall = getRulesToInstall(latestPrepackagedRules, prepackagedRules);
const rulesToUpdate = getRulesToUpdate(latestPrepackagedRules, prepackagedRules);
const signalsIndex = siemClient.getSignalsIndex();
if (rulesToInstall.length !== 0 || rulesToUpdate.length !== 0) {
const signalsIndexExists = await getIndexExists(esClient.asCurrentUser, signalsIndex);
Expand Down
2 changes: 1 addition & 1 deletion ...ution/server/lib/detection_engine/routes/rules/get_prepackaged_rules_status_route.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ import {

jest.mock('../../rules/get_prepackaged_rules', () => {
return {
getPrepackagedRules: () => {
getLatestPrepackagedRules: async () => {
return [
{
rule_id: 'rule-1',
Expand Down
11 changes: 7 additions & 4 deletions ...y_solution/server/lib/detection_engine/routes/rules/get_prepackaged_rules_status_route.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,12 @@ import {
import type { SecuritySolutionPluginRouter } from '../../../../types';
import { DETECTION_ENGINE_PREPACKAGED_URL } from '../../../../../common/constants';
import { transformError, buildSiemResponse } from '../utils';
import { getPrepackagedRules } from '../../rules/get_prepackaged_rules';
import { getRulesToInstall } from '../../rules/get_rules_to_install';
import { getRulesToUpdate } from '../../rules/get_rules_to_update';
import { findRules } from '../../rules/find_rules';
import { getLatestPrepackagedRules } from '../../rules/get_prepackaged_rules';
import { getExistingPrepackagedRules } from '../../rules/get_existing_prepackaged_rules';
import { ruleAssetSavedObjectsClientFactory } from '../../rules/rule_asset_saved_objects_client';
import { buildFrameworkRequest } from '../../../timeline/utils/common';
import { ConfigType } from '../../../../config';
import { SetupPlugins } from '../../../../plugin';
Expand All @@ -40,15 +41,17 @@ export const getPrepackagedRulesStatusRoute = (
},
},
async (context, request, response) => {
const savedObjectsClient = context.core.savedObjects.client;
const siemResponse = buildSiemResponse(response);
const alertsClient = context.alerting?.getAlertsClient();
const ruleAssetsClient = ruleAssetSavedObjectsClientFactory(savedObjectsClient);

if (!alertsClient) {
return siemResponse.error({ statusCode: 404 });
}

try {
const rulesFromFileSystem = getPrepackagedRules();
const latestPrepackagedRules = await getLatestPrepackagedRules(ruleAssetsClient);
const customRules = await findRules({
alertsClient,
perPage: 1,
Expand All @@ -61,8 +64,8 @@ export const getPrepackagedRulesStatusRoute = (
const frameworkRequest = await buildFrameworkRequest(context, security, request);
const prepackagedRules = await getExistingPrepackagedRules({ alertsClient });

const rulesToInstall = getRulesToInstall(rulesFromFileSystem, prepackagedRules);
const rulesToUpdate = getRulesToUpdate(rulesFromFileSystem, prepackagedRules);
const rulesToInstall = getRulesToInstall(latestPrepackagedRules, prepackagedRules);
const rulesToUpdate = getRulesToUpdate(latestPrepackagedRules, prepackagedRules);
const prepackagedTimelineStatus = await checkTimelinesStatus(frameworkRequest);
const [validatedprepackagedTimelineStatus] = validate(
prepackagedTimelineStatus,
Expand Down
8 changes: 8 additions & 0 deletions x-pack/plugins/security_solution/server/lib/detection_engine/rules/constants.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

export const EPR_PACKAGE = 'security_detection_engine';
rw-access marked this conversation as resolved.
Show resolved Hide resolved
6 changes: 4 additions & 2 deletions ...plugins/security_solution/server/lib/detection_engine/rules/get_prepackaged_rules.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,8 +41,10 @@ describe('get_existing_prepackaged_rules', () => {
});

test('should throw an exception with a message having rule_id and name in it', () => {
// @ts-expect-error intentionally invalid argument
expect(() => getPrepackagedRules([{ name: 'rule name', rule_id: 'id-123' }])).toThrow(
expect(() =>
// @ts-expect-error intentionally invalid argument
getPrepackagedRules([{ name: 'rule name', rule_id: 'id-123' }])
).toThrow(
'name: "rule name", rule_id: "id-123" within the folder rules/prepackaged_rules is not a valid detection engine rule. Expect the system to not work with pre-packaged rules until this rule is fixed or the file is removed. Error is: Invalid value "undefined" supplied to "description",Invalid value "undefined" supplied to "risk_score",Invalid value "undefined" supplied to "severity",Invalid value "undefined" supplied to "type",Invalid value "undefined" supplied to "version", Full rule contents are:\n{\n "name": "rule name",\n "rule_id": "id-123"\n}'
);
});
Expand Down
39 changes: 38 additions & 1 deletion x-pack/plugins/security_solution/server/lib/detection_engine/rules/get_prepackaged_rules.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@ import { BadRequestError } from '../errors/bad_request_error';

// TODO: convert rules files to TS and add explicit type definitions
import { rawRules } from './prepackaged_rules';
import { RuleAssetSavedObjectsClient } from './rule_asset_saved_objects_client';

/**
* Validate the rules from the file system and throw any errors indicating to the developer
Expand Down Expand Up @@ -52,7 +53,43 @@ export const validateAllPrepackagedRules = (
});
};

/**
* Retrieve and validate rules that were installed from Fleet as saved objects.
*/
export const getFleetInstalledRules = async (
client: RuleAssetSavedObjectsClient
): Promise<AddPrepackagedRulesSchemaDecoded[]> => {
const fleetResponse = await client.all();
const fleetRules = fleetResponse.map(
// @ts-expect-error data is too loosely typed
(so) => so.attributes as AddPrepackagedRulesSchema
Copy link
Contributor

@FrankHassanabad FrankHassanabad Apr 12, 2021 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you're shoe horning a bit here and this isn't the validateAllPrepackagedRules as it is written is really what you want maybe.

The ts-expect-error kind of gives it away as a bit as maybe something doesn't seem right.

If validateAllPrepackagedRules throws if the rule is not validated and was written when rules on a file system are a thing. Is that what you want? You want a throw here if it is not valid? Also it will say in the error message:

within the folder rules/prepackaged_rules

I don't know if that's what you want either. The best bet would be to do something like this where you add a new function that is specific to checking and validating and throwing a specific error:

export const validateAllPrepackagedRulesFromSavedObjects = (
  rules: Array<Array<SavedObject<IRuleAssetSOAttributes & SavedObjectAttributes>>>
): AddPrepackagedRulesSchemaDecoded[] => {
  return rules.map((rule) => {
    const decoded = addPrepackagedRulesSchema.decode(rule);
    const checked = exactCheck(rule, decoded);

    const onLeft = (errors: t.Errors): AddPrepackagedRulesSchemaDecoded => {
      const ruleName = rule[0].attributes.name ? rule[0].attributes.name : '(rule name unknown)';
      const ruleId = rule[0].attributes.rule_id
        ? rule[0].attributes.rule_id
        : '(rule rule_id unknown)';
      throw new BadRequestError(
        `name: "${ruleName}", rule_id: "${ruleId}" within the saved object rules/prepackaged_rules ` +
          `is not a valid detection engine rule. Expect the system ` +
          `to not work with pre-packaged rules until this rule is fixed ` +
          `or the file is removed. Error is: ${formatErrors(
            errors
          ).join()}, Full rule contents are:\n${JSON.stringify(rule, null, 2)}`
      );
    };

    const onRight = (schema: AddPrepackagedRulesSchema): AddPrepackagedRulesSchemaDecoded => {
      return schema as AddPrepackagedRulesSchemaDecoded;
    };
    return pipe(checked, fold(onLeft, onRight));
  });
};

/**
 * Retrieve and validate rules that were installed from Fleet as saved objects.
 */
export const getFleetInstalledRules = async (
  client: RuleAssetSavedObjectsClient
): Promise<AddPrepackagedRulesSchemaDecoded[]> => {
  const fleetResponse = await client.all();
  const fleetRules = fleetResponse.map((so) => so.attributes);
  return validateAllPrepackagedRulesFromSavedObjects(fleetRules);
};

Then you can remove the ts-expect-error but also look at that bit of weirdness where I have two arrays?

      const ruleName = rule[0].attributes.name ? rule[0].attributes.name : '(rule name unknown)';
      const ruleId = rule[0].attributes.rule_id
        ? rule[0].attributes.rule_id
        : '(rule rule_id unknown)';

I don't think even that is 100% but it's closer to your intent of validating and then casting the data from a SO key/value generic data structure into a more non-generic data structure which has guaranteed fields such as the io-ts ones. I think maybe there just needs to be a loop over the inner array or something else is possibly off, but it does show at least progress here to where restructuring with a new function would be better.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, fair enough to make a separate function to validate here, so we don't assume any more than absolutely necessary about the incoming dynamically typed data.

I definitely need to do this check, I just mistakenly assumed that the validateAllPrepackagedRules would also be appropriate here.

    const decoded = addPrepackagedRulesSchema.decode(rule);
    const checked = exactCheck(rule, decoded);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

41536f0

);
return validateAllPrepackagedRules(fleetRules);
};

export const getPrepackagedRules = (
// @ts-expect-error mock data is too loosely typed
rules: AddPrepackagedRulesSchema[] = rawRules
): AddPrepackagedRulesSchemaDecoded[] => validateAllPrepackagedRules(rules);
): AddPrepackagedRulesSchemaDecoded[] => {
return validateAllPrepackagedRules(rules);
};

export const getLatestPrepackagedRules = async (
client: RuleAssetSavedObjectsClient
): Promise<AddPrepackagedRulesSchemaDecoded[]> => {
// build a map of the most version of each rule
rw-access marked this conversation as resolved.
Show resolved Hide resolved
const prepackaged = getPrepackagedRules();
const ruleMap = new Map(prepackaged.map((r) => [r.rule_id, r]));

// check the rules installed via fleet and create/update if the version is newer
const fleetRules = await getFleetInstalledRules(client);
const fleetUpdates = fleetRules.filter(
// @ts-expect-error data ruleMap.get() is safe because of ruleMap.has() check
Copy link
Contributor

@FrankHassanabad FrankHassanabad Apr 12, 2021 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fwiw, @ts-expect-error count outside of test files is increasing per release when it should be be decreasing or staying steady or rising only slightly.

Maintainers, operators, and general outsiders from security_solutions who are cutting across our solution pepper them in and then later I think that creates the situation where it feels like the norm to whip them out and use them as if it is a normal typescript thing when in reality they're considered a code smell and something that should be fixed soon if possible.

Granted in a few situations where typescript is very wrong or we have to do an import of JS it is something we cannot avoid. However, we should avoid them altogether even if typescript is slightly wrong but we can work around typescript.

Can we can this code for example to be this:

  const fleetUpdates = fleetRules.filter((r) => {
    const ruleId = ruleMap.get(r.rule_id);
    return ruleId == null || ruleId.version < r.version;
  });

Then that removes the ts-expect-error.

For example, a maintainer can now change that code to have accidental character addition like this below and if it passes the unit tests it will end up shipping broken.

  const fleetUpdates = fleetRules.filter(
    // @ts-expect-error data ruleMap.get() is safe because of ruleMap.has() check
    (r) => !ruleMap_extra_chars.has(r.rule_id) || ruleMap.get(r.rule_id).version < r.version
  );

Even in tests I try to avoid them if I can. My scale of good to bad goes like this:

  • When possible use regular typescript/javascript, including generics, templates, etc...
  • Ok, I have to work a bit around typescript with a change in code, but I still get type safety.
  • Ok, if I have to use an "as cast" like "as " I will, but I tried to avoid it.
  • Ok, I have to add a bang, "!" to turn something off. I really tried not to and don't want to but it's needed. I did try to use ? or another technique but failed.
  • Ok, if I have to use an "as unknown as case", things are getting worse, but still not too bad, and the alternatives are worse than just doing it in the code.
  • Finally, I really have no choice here, I must use a ts-expect-error and hope that it goes away very soon. I understand it's easy to break things :-( in this area of code by accident.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the explanation, that's good to understand more of your thought process and approach!
i am definitely not a fan of ts-expect-error style suppression for a whole line either. especially since a future error could be introduced and it's already suppressed. that's not good.

in this case, it seems the typesystem was unable to infer how has and get relate to each other, so even though what i did looks safe it guessed wrong.

storing .get() in a variable seems like just the right solution. you can explicitly check that against null, and then the other check it knows is non-null. i'm trying that change out locally now and running through the type checking

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is == null or === undefined more idiomatic in this case?

Copy link
Contributor Author

@rw-access rw-access Apr 12, 2021 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated in 3df98ad (for == null)

(r) => !ruleMap.has(r.rule_id) || ruleMap.get(r.rule_id).version < r.version
);

// add the new or updated rules to the map
fleetUpdates.forEach((r) => ruleMap.set(r.rule_id, r));

return Array.from(ruleMap.values());
};
50 changes: 50 additions & 0 deletions ...ns/security_solution/server/lib/detection_engine/rules/rule_asset_saved_objects_client.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import {
SavedObjectsClientContract,
SavedObjectsFindOptions,
SavedObjectsFindResponse,
} from '../../../../../../../src/core/server';
import { ruleAssetSavedObjectType } from '../rules/saved_object_mappings';
import { IRuleAssetSavedObject } from '../rules/types';

const DEFAULT_PAGE_SIZE = 100;

export interface RuleAssetSavedObjectsClient {
find: (
options?: Omit<SavedObjectsFindOptions, 'type'>
) => Promise<SavedObjectsFindResponse<IRuleAssetSavedObject>>;
all: () => Promise<IRuleAssetSavedObject[]>;
}

export const ruleAssetSavedObjectsClientFactory = (
savedObjectsClient: SavedObjectsClientContract
): RuleAssetSavedObjectsClient => {
return {
find: (options) =>
savedObjectsClient.find<IRuleAssetSavedObject>({
...options,
type: ruleAssetSavedObjectType,
}),
all: async () => {
const finder = savedObjectsClient.createPointInTimeFinder({
perPage: DEFAULT_PAGE_SIZE,
type: ruleAssetSavedObjectType,
});
const responses: IRuleAssetSavedObject[] = [];
for await (const response of finder.find()) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bummer the .find doesn't take a template input and forces you to do that so as IRuleAssetSavedObject below.

response.saved_objects.forEach((so) => responses.push(so as IRuleAssetSavedObject));
if (response.total === 0) {
break;
}
Copy link
Contributor

@FrankHassanabad FrankHassanabad Apr 12, 2021 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this total is zero, won't this exit the loop automatically? Just curious, I have used this newer API before. I couldn't tell from the docs.

In other parts of the code base for example I see:

  const ruleUuids: string[] = [];

  for await (const response of pitFinder.find()) {
    ruleUuids.push(...response.saved_objects.map((object) => object.id));
  }

  await pitFinder.close();

  return ruleUuids;

Something like that where they're not looking for totals and just looping until done and then doing a close.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good question. i couldn't tell either. any guess who we should ping to double check?

}
await finder.close();
return responses;
},
};
};
13 changes: 13 additions & 0 deletions x-pack/plugins/security_solution/server/lib/detection_engine/rules/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,19 @@ export interface IRuleStatusFindType {
saved_objects: IRuleStatusSavedObject[];
}

// eslint-disable-next-line @typescript-eslint/no-explicit-any
export interface IRuleAssetSOAttributes extends Record<string, any> {
rule_id: string | null | undefined;
version: string | null | undefined;
name: string | null | undefined;
}

export interface IRuleAssetSavedObject {
type: string;
id: string;
attributes: Array<SavedObject<IRuleAssetSOAttributes & SavedObjectAttributes>>;
}

export interface HapiReadableStream extends Readable {
hapi: {
filename: string;
Expand Down