[Security][Fleet] Install the security_detection_engine package automatically #97191

rw-access · 2021-04-14T21:55:03Z

Summary

Related to https://github.com/elastic/security-team/issues/17
Continuation of #96698

Automatically install the security_detection_engine package alongside the Endpoint package. This will make security rule updates automatically available to users.

Screenshot from a fresh snapshot build:

Checklist

Delete any items that are not applicable to this PR.

Documentation was added for features that require explanation or tutorials
Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

elasticmachine · 2021-04-14T21:55:05Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2021-04-14T21:55:08Z

Pinging @elastic/fleet (Team:Fleet)

spong · 2021-04-14T23:09:38Z

x-pack/plugins/fleet/common/constants/epm.ts

@@ -15,6 +15,7 @@ export const requiredPackages = {
  System: 'system',
  Endpoint: 'endpoint',
  ElasticAgent: 'elastic_agent',
+  SecurityDetectionEngine: 'security_detection_engine',


What happens if this dependency is removed in the future -- is the package just gracefully uninstalled and SO's removed on upgrade?

yeah you got it.
fleet manages all the SOs, so they will be removed during an upgrade/uninstall gracefully. the detection engine will then just use the rules compiled in

maybe I misunderstood the question but if security_detection_engine is removed from this list of required packages in the future, Fleet will not automatically uninstall the package if it's already installed

Ahhh, so we can have lingering SO's from this package then? This would result in the rules installing from these SO's still, or would the fact the package is no-longer present short circuit that logic?

How can they be manually removed -- will it still show up in fleet until it's un-installed, or is there another way to perform this cleanup?

…cess/kibana into autoinstall-security-detection-pkg

jen-huang

Code LGTM

ruflin · 2021-04-15T18:26:08Z

Does this indirectly mean, the detection rules are also installed the first time the user visits the Fleet page or /setup is called? I'm ok to add this for now but I'm also concerned that /setup takes longer and longer: #96026

jen-huang · 2021-04-15T18:39:08Z

I had the same thought @ruflin, this will add overhead to /setup, we need to make /setup ops more performant outside of this PR.

rw-access · 2021-04-15T19:14:47Z

so for this package in particular the only "install" is just the creation of the detection rules SOs. hopefully the overhead of that is smaller and not too obtrusive?

yeah I wish there was a way to make these dependencies conditional. requiring the user to navigate to a particular solution could be useful for both endpoint and this package

…y-detection-pkg

…cess/kibana into autoinstall-security-detection-pkg

rw-access · 2021-04-20T10:28:04Z

For some reason, pulling in the 0.18.0 endpoint package via the dockerImage update caused some failures for the endpoint list functional tests, which previously worked with 0.17.1.

At first, I thought it was just the destination_index changes, but changing that data caused some security_solution_endpoint_api_int failures.

Instead of fighting an unrelated change in this PR, I think it's most appropriate to change the KQL query acaa76a to something that allows the PR to pass tests. I switched to an agent.id filter, which is testing locally.

This will unblock the PR, and we can investigate a potential 7.12 bug where there appears to be an inconsistency with this particular combination of:

0.18.0 endpoint package
7.12+ endpoint metadata api logic
x-pack/test/functional/es_archives/endpoint/metadata/destination_index/data.json test data
testing logic

We can look into this in a follow on PR, since it's completely unrelated to the scope of the PR, and only surfaced when dockerImage was updated.

kibanamachine · 2021-04-20T17:07:10Z

💚 Build Succeeded

Metrics [docs]

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	346.1KB	346.2KB	+52.0B

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…atically (elastic#97191) * Automatically install the security_detection_engine package via fleet * Update dockerImage to include the security_detection_engine package * Update api/fleet/setup install test * Update test data for Endpoint package * Fix JSON token * Update firis json entry in destination_index * Update destination_index structure * Update destination_index structure * Change KQL query to unblock testing * Restore KQL and fix JSON instead * update timestamps to pass tests

kibanamachine · 2021-04-20T17:10:20Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

…atically (#97191) (#97680) * Automatically install the security_detection_engine package via fleet * Update dockerImage to include the security_detection_engine package * Update api/fleet/setup install test * Update test data for Endpoint package * Fix JSON token * Update firis json entry in destination_index * Update destination_index structure * Update destination_index structure * Change KQL query to unblock testing * Restore KQL and fix JSON instead * update timestamps to pass tests Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>

…atically (elastic#97191) * Automatically install the security_detection_engine package via fleet * Update dockerImage to include the security_detection_engine package * Update api/fleet/setup install test * Update test data for Endpoint package * Fix JSON token * Update firis json entry in destination_index * Update destination_index structure * Update destination_index structure * Change KQL query to unblock testing * Restore KQL and fix JSON instead * update timestamps to pass tests

Automatically install the security_detection_engine package via fleet

124bdc5

rw-access requested a review from kevinlog April 14, 2021 21:55

rw-access requested a review from a team as a code owner April 14, 2021 21:55

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Apr 14, 2021

spong reviewed Apr 14, 2021

View reviewed changes

kibanamachine and others added 4 commits April 15, 2021 00:08

Merge branch 'master' into autoinstall-security-detection-pkg

35ff135

Update dockerImage to include the security_detection_engine package

43b52a3

Merge branch 'autoinstall-security-detection-pkg' of github.com:rw-ac…

5610c81

…cess/kibana into autoinstall-security-detection-pkg

Merge branch 'master' into autoinstall-security-detection-pkg

c2501a9

rw-access enabled auto-merge (squash) April 15, 2021 07:25

rw-access disabled auto-merge April 15, 2021 07:25

rw-access enabled auto-merge (squash) April 15, 2021 07:25

rw-access added 2 commits April 15, 2021 09:27

Update api/fleet/setup install test

51147f8

Merge branch 'autoinstall-security-detection-pkg' of github.com:rw-ac…

9c48e96

…cess/kibana into autoinstall-security-detection-pkg

jen-huang approved these changes Apr 15, 2021

View reviewed changes

Merge branch 'master' into autoinstall-security-detection-pkg

7e21c86

rw-access and others added 3 commits April 15, 2021 14:50

Merge remote-tracking branch 'origin/master' into autoinstall-securit…

340ab0a

…y-detection-pkg

Merge branch 'master' into autoinstall-security-detection-pkg

820ef5c

Merge branch 'master' into autoinstall-security-detection-pkg

287828e

rw-access added 4 commits April 20, 2021 02:52

Update destination_index structure

5515757

Update destination_index structure

d95a0ef

Merge branch 'autoinstall-security-detection-pkg' of github.com:rw-ac…

61057aa

…cess/kibana into autoinstall-security-detection-pkg

Change KQL query to unblock testing

acaa76a

rw-access disabled auto-merge April 20, 2021 11:44

rw-access and others added 2 commits April 20, 2021 06:43

Restore KQL and fix JSON instead

4a06e1f

update timestamps to pass tests

d425a8a

elastic deleted a comment from kibanamachine Apr 20, 2021

kevinlog approved these changes Apr 20, 2021

View reviewed changes

elastic deleted a comment from kibanamachine Apr 20, 2021

rw-access enabled auto-merge (squash) April 20, 2021 15:17

pzl approved these changes Apr 20, 2021

View reviewed changes

rw-access merged commit 1925cea into elastic:master Apr 20, 2021

rw-access deleted the autoinstall-security-detection-pkg branch April 20, 2021 17:07

kibanamachine mentioned this pull request Apr 20, 2021

[7.x] [Security][Fleet] Install the security_detection_engine package automatically (#97191) #97680

Merged

rw-access mentioned this pull request Apr 30, 2021

[Fleet] Make security rules package optional (revert #97191) #98854

Merged

rw-access added a commit that referenced this pull request Apr 30, 2021

Make security rules optional (revert #97191) (#98854)

0ddea04

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Apr 30, 2021

Make security rules optional (revert elastic#97191) (elastic#98854)

55a4ed0

kibanamachine mentioned this pull request Apr 30, 2021

[7.13] Make security rules optional (revert #97191) (#98854) #98860

Merged

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Apr 30, 2021

Make security rules optional (revert elastic#97191) (elastic#98854)

da35549

kibanamachine mentioned this pull request Apr 30, 2021

[7.x] Make security rules optional (revert #97191) (#98854) #98861

Merged

kibanamachine added a commit that referenced this pull request Apr 30, 2021

Make security rules optional (revert #97191) (#98854) (#98860)

3d7966b

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>

kibanamachine added a commit that referenced this pull request Apr 30, 2021

Make security rules optional (revert #97191) (#98854) (#98861)

5809263

Co-authored-by: Ross Wolf <31489089+rw-access@users.noreply.github.com>

madirey pushed a commit to madirey/kibana that referenced this pull request May 11, 2021

Make security rules optional (revert elastic#97191) (elastic#98854)

1f34fec

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security][Fleet] Install the security_detection_engine package automatically #97191

[Security][Fleet] Install the security_detection_engine package automatically #97191

rw-access commented Apr 14, 2021 •

edited

Loading

elasticmachine commented Apr 14, 2021

elasticmachine commented Apr 14, 2021

spong Apr 14, 2021

rw-access Apr 15, 2021

jen-huang Apr 15, 2021

spong Apr 20, 2021

jen-huang left a comment

ruflin commented Apr 15, 2021

jen-huang commented Apr 15, 2021 •

edited

Loading

rw-access commented Apr 15, 2021

rw-access commented Apr 20, 2021 •

edited

Loading

kibanamachine commented Apr 20, 2021

kibanamachine commented Apr 20, 2021

[Security][Fleet] Install the security_detection_engine package automatically #97191

[Security][Fleet] Install the security_detection_engine package automatically #97191

Conversation

rw-access commented Apr 14, 2021 • edited Loading

Summary

Checklist

For maintainers

elasticmachine commented Apr 14, 2021

elasticmachine commented Apr 14, 2021

spong Apr 14, 2021

Choose a reason for hiding this comment

rw-access Apr 15, 2021

Choose a reason for hiding this comment

jen-huang Apr 15, 2021

Choose a reason for hiding this comment

spong Apr 20, 2021

Choose a reason for hiding this comment

jen-huang left a comment

Choose a reason for hiding this comment

ruflin commented Apr 15, 2021

jen-huang commented Apr 15, 2021 • edited Loading

rw-access commented Apr 15, 2021

rw-access commented Apr 20, 2021 • edited Loading

kibanamachine commented Apr 20, 2021

💚 Build Succeeded

Metrics [docs]

Page load bundle

kibanamachine commented Apr 20, 2021

💚 Backport successful

rw-access commented Apr 14, 2021 •

edited

Loading

jen-huang commented Apr 15, 2021 •

edited

Loading

rw-access commented Apr 20, 2021 •

edited

Loading