Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution][Endpoint][Exceptions] Only write manifest to policy when there are changes #72000

Merged
merged 18 commits into from
Jul 17, 2020
Merged

[Security Solution][Endpoint][Exceptions] Only write manifest to policy when there are changes #72000

Show file tree
Hide file tree
Changes from 16 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
936b498
Refactor security_solution policy creation callback - part 1
madirey Jul 15, 2020
8392372
Fix manifest dispatch
madirey Jul 15, 2020
17316b4
Change how dispatches are performed
madirey Jul 16, 2020
ac6a453
simplify manifest types
madirey Jul 16, 2020
a8cc143
Merge branch 'master' of github.com:elastic/kibana into policy-creati…
madirey Jul 16, 2020
c8516fb
Remove unused mock
madirey Jul 16, 2020
045c470
Fix tests
madirey Jul 16, 2020
eaf1cfa
Merge branch 'master' of github.com:elastic/kibana into policy-creati…
madirey Jul 16, 2020
4ec51cb
one place to construct artifact ids
madirey Jul 16, 2020
a2fc28a
fixing linter exceptions
madirey Jul 16, 2020
8e2cd14
Add tests for stable hashes
madirey Jul 17, 2020
85f873d
Additional testing and type cleanup
madirey Jul 17, 2020
e5296ce
Merge branch 'master' of github.com:elastic/kibana into policy-creati…
madirey Jul 17, 2020
e5731e5
Remove unnecessary log
madirey Jul 17, 2020
08afa76
Minor fixup
madirey Jul 17, 2020
5bf5655
jsdoc
madirey Jul 17, 2020
87f8ee4
type fixup
madirey Jul 17, 2020
807f80f
Additional type adjustments
madirey Jul 17, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
87 changes: 87 additions & 0 deletions x-pack/plugins/ingest_manager/common/mocks.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,90 @@ export const createPackageConfigMock = (): PackageConfig => {
],
};
};

export const createPackageConfigWithInitialManifestMock = (): PackageConfig => {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe me more specific in the naming, any reason to no keep this in the security_solution plugin? it will make future change easier no?

Suggested change
export const createPackageConfigWithInitialManifestMock = (): PackageConfig => {
export const createEndpointPackageConfigWithInitialManifestMock = (): PackageConfig => {

Copy link
Contributor Author

@madirey madirey Jul 17, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@nchaulet These are mocks related to the policy, so IMO they belong here. The manifest is part of the policy. I'm happy to rename or move though if that's the consensus. Can we address in a quick follow-up to this one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We discussed and in our next PR, I'll see if I can make the names more descriptive. Leaving them here for now, as they may be useful for future tests. We can move them to security_solution if they don't end up being useful here.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@madirey sorry for the late response.
I agree with @nchaulet on this here - these are endpoint specific mocks that Ingest knows nothing about. We only use their data structure to store our Policy data in (the area where we store it accepts any json blob).

We should really move them to our Plugin (probably under the generator we have in /common/endpoint.

const packageConfig = createPackageConfigMock();
packageConfig.inputs[0].config!.artifact_manifest = {
value: {
artifacts: {
'endpoint-exceptionlist-linux-v1': {
compression_algorithm: 'zlib',
encryption_algorithm: 'none',
decoded_sha256: 'd801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658',
encoded_sha256: 'f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda',
decoded_size: 14,
encoded_size: 22,
relative_url:
'/api/endpoint/artifacts/download/endpoint-exceptionlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658',
},
'endpoint-exceptionlist-macos-v1': {
compression_algorithm: 'zlib',
encryption_algorithm: 'none',
decoded_sha256: 'd801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658',
encoded_sha256: 'f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda',
decoded_size: 14,
encoded_size: 22,
relative_url:
'/api/endpoint/artifacts/download/endpoint-exceptionlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658',
},
'endpoint-exceptionlist-windows-v1': {
compression_algorithm: 'zlib',
encryption_algorithm: 'none',
decoded_sha256: 'd801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658',
encoded_sha256: 'f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda',
decoded_size: 14,
encoded_size: 22,
relative_url:
'/api/endpoint/artifacts/download/endpoint-exceptionlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658',
},
},
manifest_version: 'a9b7ef358a363f327f479e31efc4f228b2277a7fb4d1914ca9b4e7ca9ffcf537',
schema_version: 'v1',
},
};
return packageConfig;
};

export const createPackageConfigWithManifestMock = (): PackageConfig => {
const packageConfig = createPackageConfigMock();
packageConfig.inputs[0].config!.artifact_manifest = {
value: {
artifacts: {
'endpoint-exceptionlist-linux-v1': {
compression_algorithm: 'zlib',
encryption_algorithm: 'none',
decoded_sha256: '0a5a2013a79f9e60682472284a1be45ab1ff68b9b43426d00d665016612c15c8',
encoded_sha256: '57941169bb2c5416f9bd7224776c8462cb9a2be0fe8b87e6213e77a1d29be824',
decoded_size: 292,
encoded_size: 131,
relative_url:
'/api/endpoint/artifacts/download/endpoint-exceptionlist-linux-v1/0a5a2013a79f9e60682472284a1be45ab1ff68b9b43426d00d665016612c15c8',
},
'endpoint-exceptionlist-macos-v1': {
compression_algorithm: 'zlib',
encryption_algorithm: 'none',
decoded_sha256: '96b76a1a911662053a1562ac14c4ff1e87c2ff550d6fe52e1e0b3790526597d3',
encoded_sha256: '975382ab55d019cbab0bbac207a54e2a7d489fad6e8f6de34fc6402e5ef37b1e',
decoded_size: 432,
encoded_size: 147,
relative_url:
'/api/endpoint/artifacts/download/endpoint-exceptionlist-macos-v1/96b76a1a911662053a1562ac14c4ff1e87c2ff550d6fe52e1e0b3790526597d3',
},
'endpoint-exceptionlist-windows-v1': {
compression_algorithm: 'zlib',
encryption_algorithm: 'none',
decoded_sha256: '96b76a1a911662053a1562ac14c4ff1e87c2ff550d6fe52e1e0b3790526597d3',
encoded_sha256: '975382ab55d019cbab0bbac207a54e2a7d489fad6e8f6de34fc6402e5ef37b1e',
decoded_size: 432,
encoded_size: 147,
relative_url:
'/api/endpoint/artifacts/download/endpoint-exceptionlist-windows-v1/96b76a1a911662053a1562ac14c4ff1e87c2ff550d6fe52e1e0b3790526597d3',
},
},
manifest_version: '520f6cf88b3f36a065c6ca81058d5f8690aadadf6fe857f8dec4cc37589e7283',
schema_version: 'v1',
},
};

return packageConfig;
};
5 changes: 5 additions & 0 deletions x-pack/plugins/security_solution/common/endpoint/schema/common.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ export const compressionAlgorithm = t.keyof({
});
export type CompressionAlgorithm = t.TypeOf<typeof compressionAlgorithm>;

export const compressionAlgorithmDispatch = t.keyof({
zlib: null,
});
export type CompressionAlgorithmDispatch = t.TypeOf<typeof compressionAlgorithmDispatch>;

export const encryptionAlgorithm = t.keyof({
none: null,
});
Expand Down
46 changes: 41 additions & 5 deletions x-pack/plugins/security_solution/common/endpoint/schema/manifest.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
import * as t from 'io-ts';
import {
compressionAlgorithm,
compressionAlgorithmDispatch,
encryptionAlgorithm,
identifier,
manifestSchemaVersion,
Expand All @@ -16,25 +17,60 @@ import {
size,
} from './common';

export const manifestEntrySchema = t.exact(
export const manifestEntryBaseSchema = t.exact(
t.type({
relative_url: relativeUrl,
decoded_sha256: sha256,
decoded_size: size,
encoded_sha256: sha256,
encoded_size: size,
compression_algorithm: compressionAlgorithm,
encryption_algorithm: encryptionAlgorithm,
})
);

export const manifestSchema = t.exact(
export const manifestEntrySchema = t.intersection([
manifestEntryBaseSchema,
t.exact(
t.type({
compression_algorithm: compressionAlgorithm,
})
),
]);
export type ManifestEntrySchema = t.TypeOf<typeof manifestEntrySchema>;

export const manifestEntryDispatchSchema = t.intersection([
manifestEntryBaseSchema,
t.exact(
t.type({
compression_algorithm: compressionAlgorithmDispatch,
})
),
]);
export type ManifestEntryDispatchSchema = t.TypeOf<typeof manifestEntryDispatchSchema>;

export const manifestBaseSchema = t.exact(
t.type({
manifest_version: manifestVersion,
schema_version: manifestSchemaVersion,
artifacts: t.record(identifier, manifestEntrySchema),
})
);

export type ManifestEntrySchema = t.TypeOf<typeof manifestEntrySchema>;
export const manifestSchema = t.intersection([
manifestBaseSchema,
t.exact(
t.type({
artifacts: t.record(identifier, manifestEntrySchema),
})
),
]);
export type ManifestSchema = t.TypeOf<typeof manifestSchema>;

export const manifestDispatchSchema = t.intersection([
manifestBaseSchema,
t.exact(
t.type({
artifacts: t.record(identifier, manifestEntryDispatchSchema),
})
),
]);
export type ManifestDispatchSchema = t.TypeOf<typeof manifestDispatchSchema>;
105 changes: 70 additions & 35 deletions x-pack/plugins/security_solution/server/endpoint/ingest_integration.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,87 +4,122 @@
* you may not use this file except in compliance with the Elastic License.
*/

// eslint-disable-next-line @kbn/eslint/no-restricted-paths
import { loggerMock } from 'src/core/server/logging/logger.mock';
import { loggingSystemMock } from 'src/core/server/mocks';
import { createNewPackageConfigMock } from '../../../ingest_manager/common/mocks';
import { factory as policyConfigFactory } from '../../common/endpoint/models/policy_config';
import { getManifestManagerMock } from './services/artifacts/manifest_manager/manifest_manager.mock';
import {
getManifestManagerMock,
ManifestManagerMockType,
} from './services/artifacts/manifest_manager/manifest_manager.mock';
import { getPackageConfigCreateCallback } from './ingest_integration';
import { ManifestConstants } from './lib/artifacts';

describe('ingest_integration tests ', () => {
describe('ingest_integration sanity checks', () => {
test('policy is updated with manifest', async () => {
const logger = loggerMock.create();
const manifestManager = getManifestManagerMock();
test('policy is updated with initial manifest', async () => {
const logger = loggingSystemMock.create().get('ingest_integration.test');
const manifestManager = getManifestManagerMock({
mockType: ManifestManagerMockType.InitialSystemState,
});

const callback = getPackageConfigCreateCallback(logger, manifestManager);
const policyConfig = createNewPackageConfigMock();
const newPolicyConfig = await callback(policyConfig);
const policyConfig = createNewPackageConfigMock(); // policy config without manifest
const newPolicyConfig = await callback(policyConfig); // policy config WITH manifest

expect(newPolicyConfig.inputs[0]!.type).toEqual('endpoint');
expect(newPolicyConfig.inputs[0]!.config!.policy.value).toEqual(policyConfigFactory());
expect(newPolicyConfig.inputs[0]!.config!.artifact_manifest.value).toEqual({
artifacts: {
'endpoint-exceptionlist-linux-v1': {
compression_algorithm: 'zlib',
decoded_sha256: '1a8295e6ccb93022c6f5ceb8997b29f2912389b3b38f52a8f5a2ff7b0154b1bc',
decoded_size: 287,
encoded_sha256: 'c3dec543df1177561ab2aa74a37997ea3c1d748d532a597884f5a5c16670d56c',
encoded_size: 133,
decoded_sha256: 'd801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658',
decoded_size: 14,
encoded_sha256: 'f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda',
encoded_size: 22,
encryption_algorithm: 'none',
relative_url:
'/api/endpoint/artifacts/download/endpoint-exceptionlist-linux-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658',
},
'endpoint-exceptionlist-macos-v1': {
compression_algorithm: 'zlib',
decoded_sha256: 'd801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658',
decoded_size: 14,
encoded_sha256: 'f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda',
encoded_size: 22,
encryption_algorithm: 'none',
relative_url:
'/api/endpoint/artifacts/download/endpoint-exceptionlist-macos-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658',
},
'endpoint-exceptionlist-windows-v1': {
compression_algorithm: 'zlib',
decoded_sha256: 'd801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658',
decoded_size: 14,
encoded_sha256: 'f8e6afa1d5662f5b37f83337af774b5785b5b7f1daee08b7b00c2d6813874cda',
encoded_size: 22,
encryption_algorithm: 'none',
relative_url:
'/api/endpoint/artifacts/download/endpoint-exceptionlist-linux-v1/1a8295e6ccb93022c6f5ceb8997b29f2912389b3b38f52a8f5a2ff7b0154b1bc',
'/api/endpoint/artifacts/download/endpoint-exceptionlist-windows-v1/d801aa1fb7ddcc330a5e3173372ea6af4a3d08ec58074478e85aa5603e926658',
},
},
manifest_version: 'WzAsMF0=',
manifest_version: 'a9b7ef358a363f327f479e31efc4f228b2277a7fb4d1914ca9b4e7ca9ffcf537',
schema_version: 'v1',
});
});

test('policy is returned even if error is encountered during artifact sync', async () => {
const logger = loggerMock.create();
test('policy is returned even if error is encountered during artifact creation', async () => {
const logger = loggingSystemMock.create().get('ingest_integration.test');
const manifestManager = getManifestManagerMock();
manifestManager.syncArtifacts = jest.fn().mockRejectedValue([new Error('error updating')]);
const lastDispatched = await manifestManager.getLastDispatchedManifest();
manifestManager.pushArtifacts = jest.fn().mockResolvedValue([new Error('error updating')]);
const lastComputed = await manifestManager.getLastComputedManifest(
ManifestConstants.SCHEMA_VERSION
);

const callback = getPackageConfigCreateCallback(logger, manifestManager);
const policyConfig = createNewPackageConfigMock();
const newPolicyConfig = await callback(policyConfig);

expect(newPolicyConfig.inputs[0]!.type).toEqual('endpoint');
expect(newPolicyConfig.inputs[0]!.config!.policy.value).toEqual(policyConfigFactory());
expect(newPolicyConfig.inputs[0]!.config!.artifact_manifest.value).toEqual(
lastDispatched.toEndpointFormat()
lastComputed!.toEndpointFormat()
);
});

test('initial policy creation succeeds if snapshot retrieval fails', async () => {
const logger = loggerMock.create();
const manifestManager = getManifestManagerMock();
const lastDispatched = await manifestManager.getLastDispatchedManifest();
manifestManager.getSnapshot = jest.fn().mockResolvedValue(null);
test('initial policy creation succeeds if manifest retrieval fails', async () => {
const logger = loggingSystemMock.create().get('ingest_integration.test');
const manifestManager = getManifestManagerMock({
mockType: ManifestManagerMockType.InitialSystemState,
});
const lastComputed = await manifestManager.getLastComputedManifest(
ManifestConstants.SCHEMA_VERSION
);
expect(lastComputed).toEqual(null);

manifestManager.buildNewManifest = jest.fn().mockRejectedValue(new Error('abcd'));
const callback = getPackageConfigCreateCallback(logger, manifestManager);
const policyConfig = createNewPackageConfigMock();
const newPolicyConfig = await callback(policyConfig);

expect(newPolicyConfig.inputs[0]!.type).toEqual('endpoint');
expect(newPolicyConfig.inputs[0]!.config!.policy.value).toEqual(policyConfigFactory());
expect(newPolicyConfig.inputs[0]!.config!.artifact_manifest.value).toEqual(
lastDispatched.toEndpointFormat()
);
});

test('subsequent policy creations succeed', async () => {
const logger = loggerMock.create();
const logger = loggingSystemMock.create().get('ingest_integration.test');
const manifestManager = getManifestManagerMock();
const snapshot = await manifestManager.getSnapshot();
manifestManager.getLastDispatchedManifest = jest.fn().mockResolvedValue(snapshot!.manifest);
manifestManager.getSnapshot = jest.fn().mockResolvedValue({
manifest: snapshot!.manifest,
diffs: [],
});
const lastComputed = await manifestManager.getLastComputedManifest(
ManifestConstants.SCHEMA_VERSION
);

manifestManager.buildNewManifest = jest.fn().mockResolvedValue(lastComputed); // no diffs
const callback = getPackageConfigCreateCallback(logger, manifestManager);
const policyConfig = createNewPackageConfigMock();
const newPolicyConfig = await callback(policyConfig);

expect(newPolicyConfig.inputs[0]!.type).toEqual('endpoint');
expect(newPolicyConfig.inputs[0]!.config!.policy.value).toEqual(policyConfigFactory());
expect(newPolicyConfig.inputs[0]!.config!.artifact_manifest.value).toEqual(
snapshot!.manifest.toEndpointFormat()
lastComputed!.toEndpointFormat()
);
});
});
Expand Down
Loading