New Output: Elastic ES|QL

[WildDogOne]

I am building the converter and output pipeline für Elastic.
There will be two plattforms that can be used, esql (ES|QL) or eql.
However it should always be esql prefered since it supports more sigma features.

When it comes to raw rule import, both are valid options, since both languages have their own advantages.
I will not implement KQL (Kibana Query Language) since there are no obvious benefits.

Todos:

• Renaming Plattform to elastic-xxx
• Add Documentation of new Variables
• Use variables for the policy executions
• Add Search Export
• Add Run Search
• Add Raw Rules for ES|QL & EQL -> Here it makes sense to do both languages
• Testcases
• Have a discussion about how to push the data: One at a time is the solution, I've converted to JSON import instead of
  NDJSON which already spedup the process by a lot
• Add disabling of rules
• Add deleting rules
• Add Integrity check
• Implement push only on rule change
• Have a think about this problem: https://www.elastic.co/guide/en/security/8.15/rules-ui-create.html#esql-non-agg-query-dedupe
• Fix the index selection for raw eql rules, right now they are done via custom field index, but that's just not OK
• Add option for BuildingBlock prefix, right now it's automatically added if it's a BB, but maybe not everyone wants that

[WildDogOne]

by switching to allow both ES|QL and EQL I introduced a new problem.
It is now easily possible to accidentally push policies in one language and the next time pushing in another language.
This will obviously crash.
There are two solutions.
Deleting and Recreating the detection rule when a collision is noticed
Or just giving an error to the user so they can decide themselves.

Right now I just added an error

[WildDogOne]

Random nuissance, at the end of an export run I always get this error, but the run was absolutely OK...?

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\linus\Documents\git\droid-pipeline\.venv\Scripts\droid.exe\__main__.py", line 7, in <module>
  File "c:\Users\linus\Documents\git\droid-pipeline\.venv\Lib\site-packages\droid\__main__.py", line 306, in main
    conversion_error, search_warning = convert_rules(parameters, droid_platform_config(args, config_path), base_config)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: cannot unpack non-iterable bool object

[WildDogOne]

I have added a custom field for raw rule imports.

custom:
  raw_language: esql/eql

If no raw_language is set it will take the platform that is used.
Also big issue with the Kibana API, it will gladly accept queries in the wrong language...
This is quite dangerous behaviour, because the import will succeed but the alert will fail

I am thinking if it should be mandatory to set the custom raw_language field, to avoid accidental failures

[WildDogOne]

I am not sure which SIEMs support this, but Elastic just like QRadar have a concept called building blocks. Basically a building block (BB) will not trigger an alert, but it well trigger an event. These events can then later be used to build alerts or for investigation purposes.

My old implementation is to have a tag in the Sigma file called "elastic.bb" which I then use to make the rule into a BB.
However it could make sense to have a custom field for that as well, depending on if other SIEMs have this feature too

custom:
  building_block: true/false

[0xFustang]

Deleting and Recreating the detection rule when a collision is noticed

To be tested but it's fine to delete and recreate it.

I am thinking if it should be mandatory to set the custom raw_language field, to avoid accidental failures

I would go for that path until we find a better way to validate the language in the process

As for the building_block, this implementation is fine.

[0xFustang]

As for the index selection, the index can be provided from the general configuration of droid and the rule level config:

For instance:

from a droid_config.toml :

...
[platforms.esql]

url = "foo"
...

    [platforms.esql.pipelines.windows_process_access]

    pipelines = ["pipelines/esql_process_access.yml"]
    product = "windows"
    category = "process_access"
    index = "windows-sysmon*"

from a custom field in a rule :

custom:
   index: "windows-sysmon-special*"

[WildDogOne]

About the index thing, pysigma uses the pipeline for that.

  - id: windows_active_directory
    type: set_state
    key: "index"
    val: "logs-system.*"
    rule_conditions:
      - type: logsource
        product: windows
        category: active_directory

but right now I am too stupid to understand how I would use the result of the pipeline inside the elastic integration
things for the future maybe, since it is only an issue with EQL and EQL is secondary anyhow