[FR] [DAC] further decouple reliance on default rule dir locations

[eric-forte-elastic]

Issues

#3619

Summary

The primary purpose of this PR is to remove the hard coding of rules directories and replace them with the directories loaded from a config file. Either the default one, or a user generated one.

Additionally, there are a few other considerations that are related to the implementation:

• This PR should make BBR rules directories optional
• In the config, we should have a separate path list for BBR rules dirs as they are handled differently
• This PR updates the current loading of rules to support a list of paths to be loaded, instead of one path.
• This PR updates the example config generation to match the formatting specified from the above considerations.

Open Design Quetsions:

• In the config, should also have an RTA section (expected to be optional as well)?
  • Currently this is not done, but maybe it should be?
• Need to decide whether or not deprecated rules are optional and whether or not we should make this be a configurable directory location (as opposed to _deprecated relative to the rules directory.)
  • Current decision is to store deprecated rules in _deprecated folder relative to rules directory.

Testing

To test one will need to run the following commands with both the default config and a custom config.

• make build
• make test-cli
• make test-remote-cli

Testing Results

Custom config

make

make_build.txt

make test-cli

make_test_cli.txt

make test-remote-cli

detection-rules on  3619-frdac-further-decouple-reliance-on-default-rule-dir-locations [$✘!?] is  v0.1.0 via  v3.12.3 (detection-rules-build) on  eric.forte
❯ make test-remote-cli
Executing test_remote_cli script...
Running detection-rules remote CLI tests...
Performing a quick rule alerts search...
Requires .detection-rules-cfg.json credentials file set.
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

kibana_user: eric.forte
kibana_password:
No alerts detected
Performing a rule export...
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

kibana_user: eric.forte
kibana_password:
- skipping First Time Seen AWS Secret Value Accessed in Secrets Manager - ValidationError
3 rules exported
2 rules converted
2 saved to tmp-export
1 errors saved to tmp-export/_errors.txt
adfind_command_activity.toml  clearing_windows_console_history.toml  _errors.txt
Removing generated files...
Detection-rules CLI tests completed!

[eric-forte-elastic]

LGTM!

I think we should run the make test-cli. there may be a few gaps still missing.

Agreed, I think it passed for me due to a specific config setup in for my environment. Will take a look 👀

[eric-forte-elastic]

Couple thoughts:

1. We should probably update the custom_rules setup-config to match the bbr rule changes made in here.
2. When we load optional directories, we should probably add resiliency to handle when its none

load_directories(DEFAULT_PREBUILT_RULES_DIRS)
load_directories(DEFAULT_PREBUILT_BBR_DIRS)

will error if either are set to NoneType

Good catch! Added in a default to not be None in the creation and adding resiliency
bbr_rules_dirs: Optional[List[Path]] = field(default_factory=list)

[Mikaayenson]

This will give people the ability to place rules where they want which is nice!

[Mikaayenson]

⭐ Great work. It lgtm. Especially after making the final changes based on the make test files.

[traut]

looks good!