[FR] Added Schema Check for Data View ID and Index

[eric-forte-elastic]

Issues

#3817

Summary

This PR adds a Schema check for QueryRuleData that checks to make sure only Index or Data View ID are set for a given rule and not both. Currently we only support rules using one or the other and this will catch cases where rules may inadvertently reference both.

Testing

Use the rule from #3510, and an index field so that it has both the index and data_view_id set.

Example Rule

[metadata]
creation_date = "2024/03/13"
maturity = "development"
updated_date = "2024/03/13"

[rule]
author = ["me"]
index = ["logs-endpoint.events.*"]
data_view_id = "logs-*"
description = "ere add aims aewerdas"
language = "eql"
name = "test"
risk_score = 21
rule_id = "8f6eb3b6-e9f2-4c10-a72b-cf48b4e90c2d"
severity = "low"
type = "eql"

query = '''
process where true
'''

Now run view-rule and check to make sure there is now a validation error.

Validation Error

detection-rules on  3817-bug-schema-should-not-allow-index-and-dataview [!?] is  v0.1.0 via  v3.12.4 (detection-rules-build) on  eric.forte took 2s 
❯ python -m detection_rules view-rule rules/linux/command_and_control_development.toml
Loaded config file: /home/forteea1/Code/clean_mains/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Error loading rule in /home/forteea1/Code/clean_mains/detection-rules/rules/linux/command_and_control_development.toml
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/__main__.py", line 35, in <module>
    main()
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/__main__.py", line 32, in main
    root(prog_name="detection_rules")
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/main.py", line 224, in view_rule
    rule = RuleCollection().load_file(rule_file)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/rule_loader.py", line 435, in load_file
    return self.load_dict(obj, path=path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/rule_loader.py", line 412, in load_dict
    contents = TOMLRuleContents.from_dict(obj)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/detection_rules/mixins.py", line 142, in from_dict
    return schema.load(obj)
           ^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow_dataclass/__init__.py", line 910, in load
    all_loaded = super().load(data, many=many, **kwargs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 723, in load
    return self._do_load(
           ^^^^^^^^^^^^^^
  File "/home/forteea1/Code/clean_mains/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/marshmallow/schema.py", line 910, in _do_load
    raise exc
marshmallow.exceptions.ValidationError: {'rule': [ValidationError({'_schema': ['Only one of index or data_view_id should be set.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.'], 'query': ['Unknown field.'], 'data_view_id': ['Unknown field.'], 'language': ['Unknown field.'], 'index': ['Unknown field.']}), ValidationError({'type': ['Must be equal to query.']}), ValidationError({'type': ['Must be equal to new_terms.'], 'new_terms': ['Missing data for required field.']})]}

Example Run

[terrancedejesus]

Looks good, testing looks fine as well!
Double checked we had no existing rules leveraging this so no further changes required.

[shashank-elastic]

Manual Testing from the branch Looks good 🟢