Refresh Kibana module with API updates #3466

brokensound77 · 2024-02-22T04:22:03Z

Issues

resolves #3404
resolves #2806
resolves https://github.com/elastic/ia-trade-team/issues/224
related to #3298
resolves #3512

Summary

This refreshes the built in kibana module to use the updated rule APIs, since it was currently built on the soon to be deprecated _bulk_create API.

These new APIs are much more stable, allowing for new features built on top of them.

Details

This maintains the existing calls using the old API for backwards compatibility and adds the new bulk_action API, along with methods for each type of action call. It also adds support for import-rules and export-rules to and from Kibana.

Lastly, it adds commands to wrap the import and export functionality under python -m detection_rules kibana ...

New API methods:

bulk_action
bulk_enable
bulk_disable
bulk_delete
bulk_duplicate
bulk_edit
bulk_export

New commands:

kibana import-rules
kibana export-rules

import-rules as it is works only with multi-collection which is restricted to built in rules, but this will be expanded in future PR (DAC POC)

Testing

python -m detection_rules kibana export-rules -d test-export-rules --skip-errors

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

DEBUG MODE ENABLED
- skipping Stolen Credentials Used to Login to Okta Account After MFA Reset - ValidationError
- skipping First Occurrence of Okta User Session Started via Proxy - ValidationError
- skipping ESQL test: cmd child of Explorer - ValidationError
- skipping Potential Persistence Through Run Control Detected - ValidationError
- skipping First Time Seen AWS Secret Value Accessed in Secrets Manager - ValidationError
- skipping Potential Shadow File Read via Command Line Utilities - ValidationError
- skipping Abnormal Process ID or Lock File Created - ValidationError
- skipping New service installed in last 24 hours - ValidationError
- skipping Scheduled Task or Driver added - KqlParseError
- skipping Scheduled Task or Driver removed - KqlParseError
- skipping name - ValidationError
33 rules exported
22 rules converted
22 saved to test-export-rules
11 errors saved to test-export-rules/_errors.txt

ls test-export-rules

_errors.txt
collection_exchange_mailbox_export_via_powershell.toml.toml
credential_access_multiple_okta_user_auth_events_with_same_device_token_hash_behind_a_proxy.toml.toml
credential_access_potential_okta_mfa_bombing_via_push_notifications.toml.toml
defense_evasion_agent_spoofing_multiple_hosts_using_same_agent.toml.toml
defense_evasion_attempt_to_disable_syslog_service.toml.toml
defense_evasion_kernel_module_removal.toml.toml
discovery_enumeration_of_kernel_modules.toml.toml
execution_interactive_terminal_spawned_via_python.toml.toml
initial_access_multiple_okta_client_addresses_for_a_single_user_session.toml.toml
initial_access_new_okta_authentication_behavior_detected.toml.toml
initial_access_okta_fastpass_phishing_detection.toml.toml
initial_access_okta_sign_in_events_via_third_party_idp.toml.toml
initial_access_okta_user_sessions_started_from_different_geolocations.toml.toml
lateral_movement_multiple_okta_sessions_detected_for_a_single_user.toml.toml
my_first_alert.toml.toml
persistence_new_okta_identity_provider_idp_added_by_admin.toml.toml
test_data_view.toml.toml
test_noisy.toml.toml
test_suppress.toml.toml
web_application_suspicious_activity_post_request_declined.toml.toml
web_application_suspicious_activity_sqlmap_user_agent.toml.toml
web_application_suspicious_activity_unauthorized_method.toml.toml

cat test-export-rules/_errors.txt 
- Stolen Credentials Used to Login to Okta Account After MFA Reset - {'_schema': ['Setup header found in both note and setup fields.']}
- First Occurrence of Okta User Session Started via Proxy - {'rule': [ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to query.']}), ValidationError({'new_terms': ['Missing data for required field.']})]}
- ESQL test: cmd child of Explorer - {'rule': [ValidationError({'type': ['Must be equal to eql.'], 'threat': {0: {'tactic': {'reference': ['String does not match expected pattern.']}, 'technique': {0: {'reference': ['String does not match expected pattern.']}}}}, 'language': ['Must be equal to eql.']}), ValidationError({'threat': {0: {'tactic': {'reference': ['String does not match expected pattern.']}, 'technique': {0: {'reference': ['String does not match expected pattern.']}}}}}), ValidationError({'type': ['Must be equal to threshold.'], 'threat': {0: {'tactic': {'reference': ['String does not match expected pattern.']}, 'technique': {0: {'reference': ['String does not match expected pattern.']}}}}, 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat': {0: {'tactic': {'reference': ['String does not match expected pattern.']}, 'technique': {0: {'reference': ['String does not match expected pattern.']}}}}, 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'threat': {0: {'tactic': {'reference': ['String does not match expected pattern.']}, 'technique': {0: {'reference': ['String does not match expected pattern.']}}}}, 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to query.'], 'threat': {0: {'tactic': {'reference': ['String does not match expected pattern.']}, 'technique': {0: {'reference': ['String does not match expected pattern.']}}}}}), ValidationError({'type': ['Must be equal to new_terms.'], 'threat': {0: {'tactic': {'reference': ['String does not match expected pattern.']}, 'technique': {0: {'reference': ['String does not match expected pattern.']}}}}, 'new_terms': ['Missing data for required field.']})]}
- Potential Persistence Through Run Control Detected - {'rule': [ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to query.']}), ValidationError({'new_terms': ['Missing data for required field.']})]}
- First Time Seen AWS Secret Value Accessed in Secrets Manager - {'rule': [ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to query.']}), ValidationError({'new_terms': ['Missing data for required field.']})]}
- Potential Shadow File Read via Command Line Utilities - {'rule': [ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to query.']}), ValidationError({'new_terms': ['Missing data for required field.']})]}
- Abnormal Process ID or Lock File Created - {'rule': [ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to query.']}), ValidationError({'new_terms': ['Missing data for required field.']})]}
- New service installed in last 24 hours - {'rule': [ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to query.']}), ValidationError({'new_terms': ['Missing data for required field.']})]}
- Scheduled Task or Driver added - Error at line:1,column:75
Unknown field
data_stream.dataset:osquery_manager.result and osquery_meta.counter>0 and osquery_meta.type:diff and osquery.last_run_code:0 and osquery_meta.action:added
                                                                          ^^^^^^^^^^^^^^^^^
stack: 8.9.0, beats: 8.9.0, ecs: 8.9.0
- Scheduled Task or Driver removed - Error at line:1,column:75
Unknown field
data_stream.dataset:osquery_manager.result and osquery_meta.counter>0 and osquery_meta.type:diff and osquery.last_run_code:0 and osquery_meta.action:removed
                                                                          ^^^^^^^^^^^^^^^^^
stack: 8.9.0, beats: 8.9.0, ecs: 8.9.0
- name - {'rule': [ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equal to esql.']}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], 'threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to query.']}), ValidationError({'new_terms': ['Missing data for required field.']})]}(venv312) ➜  detection-rules-fork git:(refresh-kibana-module-with-new-APIs) ✗

Detailed commands output

Help output:

(venv312) ➜  detection-rules-fork git:(refresh-kibana-module-with-new-APIs) ✗ python -m detection_rules kibana import-rules -h

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

DEBUG MODE ENABLED
Kibana client:
Options:
  --ignore-ssl-errors TEXT
  --space TEXT                 Kibana space
  --provider-name TEXT         Elastic Cloud providers: cloud-basic and cloud-
                               saml (for SSO)
  --provider-type TEXT         Elastic Cloud providers: basic and saml (for
                               SSO)
  -ku, --kibana-user TEXT
  --kibana-url TEXT
  -kp, --kibana-password TEXT
  -kc, --kibana-cookie TEXT    Cookie from an authed session
  --cloud-id TEXT              ID of the cloud instance.

Usage: detection_rules kibana import-rules [OPTIONS]

  Import custom rules into Kibana.

Options:
  -f, --rule-file FILE
  -d, --directory DIRECTORY       Recursively load rules from a directory
  -id, --rule-id TEXT
  -o, --overwrite                 Overwrite existing rules
  -e, --overwrite-exceptions      Overwrite exceptions in existing rules
  -a, --overwrite-action-connectors
                                  Overwrite action connectors in existing
                                  rules
  -h, --help                      Show this message and exit.

Existing rule fails as expected:

(venv312) ➜  detection-rules-fork git:(refresh-kibana-module-with-new-APIs) ✗ python -m detection_rules kibana import-rules -f test-export-rules/credential_access_EXISTING_RULE.toml

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

DEBUG MODE ENABLED
1 rule(s) failed to import!
 - 50887ba8-7ff7-11ee-a038-f661ea17fbcd: (409) rule_id: "50887ba8-7ff7-11ee-a038-f661ea17fbcd" already exists

-o overwrite forces the import successfully

(venv312) ➜  detection-rules-fork git:(refresh-kibana-module-with-new-APIs) ✗ python -m detection_rules kibana import-rules -f test-export-rules/credential_access_EXISTING_RULE.toml -o

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

DEBUG MODE ENABLED
1 rule(s) successfully imported
 - 50887ba8-7ff7-11ee-a038-f661ea17fbcd

New rule successfully imports:

(venv312) ➜  detection-rules-fork git:(refresh-kibana-module-with-new-APIs) ✗ python -m detection_rules kibana import-rules -f test-export-rules/credential_access_NEW_RULE.toml        

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

DEBUG MODE ENABLED
1 rule(s) successfully imported
 - 50887ba8-aaaa-bbbb-a038-f661ea17fbcd

The rule loader detects a collision in name and fails as intended:

(venv312) ➜  detection-rules-fork git:(refresh-kibana-module-with-new-APIs) ✗ python -m detection_rules kibana import-rules -d test-export-rules                               

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

DEBUG MODE ENABLED
Error loading rule in /Users/jibarra/PycharmProjects/detection-rules-fork/test-export-rules/credential_access_NEW_RULE.toml
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/__main__.py", line 34, in <module>
    main()
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/__main__.py", line 31, in main
    root(prog_name="detection_rules")
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/venv312/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/venv312/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/venv312/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/venv312/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/venv312/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/venv312/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/cli_utils.py", line 83, in get_collection
    rules.load_directories(Path(d) for d in directories)
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/rule_loader.py", line 499, in load_directories
    self.load_directory(path, recursive=recursive, obj_filter=obj_filter)
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/rule_loader.py", line 494, in load_directory
    self.load_files(paths)
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/rule_loader.py", line 487, in load_files
    self.load_file(path)
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/rule_loader.py", line 435, in load_file
    return self.load_dict(obj, path=path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/rule_loader.py", line 415, in load_dict
    self.add_rule(rule)
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/rule_loader.py", line 392, in add_rule
    self._assert_new(rule)
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/rule_loader.py", line 383, in _assert_new
    assert rule.name not in name_map, \
AssertionError: Rule Name Multiple Okta User Auth Events with Same Device Token Hash Behind a Proxy for 50887ba8-aaaa-bbbb-a038-f661ea17fbcd collides with rule ID 50887ba8-7ff7-11ee-a038-f661ea17fbcd

Expected failure on rule_id collision:

(venv312) ➜  detection-rules-fork git:(refresh-kibana-module-with-new-APIs) ✗ python -m detection_rules kibana import-rules -d test-export-rules
Loaded config file: /Users/jibarra/PycharmProjects/detection-rules-fork/.detection-rules-cfg.yaml

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

DEBUG MODE ENABLED
Error loading rule in /Users/jibarra/PycharmProjects/detection-rules-fork/test-export-rules/credential_access_multiple_okta_user_auth_events_with_same_device_token_hash_behind_a_proxy.toml
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/__main__.py", line 34, in <module>
    main()
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/__main__.py", line 31, in main
    root(prog_name="detection_rules")
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/venv312/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/venv312/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/venv312/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/venv312/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/venv312/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/venv312/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/cli_utils.py", line 83, in get_collection
    rules.load_directories(Path(d) for d in directories)
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/rule_loader.py", line 499, in load_directories
    self.load_directory(path, recursive=recursive, obj_filter=obj_filter)
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/rule_loader.py", line 494, in load_directory
    self.load_files(paths)
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/rule_loader.py", line 487, in load_files
    self.load_file(path)
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/rule_loader.py", line 435, in load_file
    return self.load_dict(obj, path=path)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/rule_loader.py", line 415, in load_dict
    self.add_rule(rule)
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/rule_loader.py", line 392, in add_rule
    self._assert_new(rule)
  File "/Users/jibarra/PycharmProjects/detection-rules-fork/detection_rules/rule_loader.py", line 381, in _assert_new
    assert rule.id not in id_map, \
AssertionError: Rule ID 50887ba8-7ff7-11ee-a038-f661ea17fbcd for Multiple Okta User Auth Events with Same Device Token Hash Behind a Proxy collides with rule Multiple Okta User Auth Events with Same Device Token Hash Behind a Proxy

Import a full directory - all fail as expected:

(venv312) ➜  detection-rules-fork git:(refresh-kibana-module-with-new-APIs) ✗ python -m detection_rules kibana import-rules -d test-export-rules

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

DEBUG MODE ENABLED
23 rule(s) failed to import!
 - ee663abc-fb77-49d2-a7c5-204b9cf888ca: (409) rule_id: "ee663abc-fb77-49d2-a7c5-204b9cf888ca" already exists
 - 50887ba8-aaaa-bbbb-a038-f661ea17fbcd: (409) rule_id: "50887ba8-aaaa-bbbb-a038-f661ea17fbcd" already exists
 - 50887ba8-7ff7-11ee-a038-f661ea17fbcd: (409) rule_id: "50887ba8-7ff7-11ee-a038-f661ea17fbcd" already exists
 - 8a0fbd26-867f-11ee-947c-f661ea17fbcd: (409) rule_id: "8a0fbd26-867f-11ee-947c-f661ea17fbcd" already exists
 - aaaaaaaa-f861-414c-8602-150d5505b777: (409) rule_id: "aaaaaaaa-f861-414c-8602-150d5505b777" already exists
 - 2f8a1226-5720-437d-9c20-e0029deb6194: (409) rule_id: "2f8a1226-5720-437d-9c20-e0029deb6194" already exists
 - cd66a5af-e34b-4bb0-8931-57d0a043f2ef: (409) rule_id: "cd66a5af-e34b-4bb0-8931-57d0a043f2ef" already exists
 - 2d8043ed-5bda-4caf-801c-c1feb7410504: (409) rule_id: "2d8043ed-5bda-4caf-801c-c1feb7410504" already exists
 - d76b02ef-fc95-4001-9297-01cb7412232f: (409) rule_id: "d76b02ef-fc95-4001-9297-01cb7412232f" already exists
 - cc382a2e-7e52-11ee-9aac-f661ea17fbcd: (409) rule_id: "cc382a2e-7e52-11ee-9aac-f661ea17fbcd" already exists
 - 260486ee-7d98-11ee-9599-f661ea17fbcd: (409) rule_id: "260486ee-7d98-11ee-9599-f661ea17fbcd" already exists
 - ee39a9f7-5a79-4b0a-9815-d36b3cf28d3e: (409) rule_id: "ee39a9f7-5a79-4b0a-9815-d36b3cf28d3e" already exists
 - 1ceb05c4-7d25-11ee-9562-f661ea17fbcd: (409) rule_id: "1ceb05c4-7d25-11ee-9562-f661ea17fbcd" already exists
 - 2e56e1bc-867a-11ee-b13e-f661ea17fbcd: (409) rule_id: "2e56e1bc-867a-11ee-b13e-f661ea17fbcd" already exists
 - 621e92b6-7e54-11ee-bdc0-f661ea17fbcd: (409) rule_id: "621e92b6-7e54-11ee-bdc0-f661ea17fbcd" already exists
 - a198fbbd-9413-45ec-a269-47ae4ccf59ce: (409) rule_id: "a198fbbd-9413-45ec-a269-47ae4ccf59ce" already exists
 - 29b53942-7cd4-11ee-b70e-f661ea17fbcd: (409) rule_id: "29b53942-7cd4-11ee-b70e-f661ea17fbcd" already exists
 - aaec44bc-d691-4874-99b2-48ab7392dfd5: (409) rule_id: "aaec44bc-d691-4874-99b2-48ab7392dfd5" already exists
 - 40e1f208-0f70-47d4-98ea-378ccf504ad3: (409) rule_id: "40e1f208-0f70-47d4-98ea-378ccf504ad3" already exists
 - 5e9bc07c-7e7a-415b-a6c0-1cae4a0d256e: (409) rule_id: "5e9bc07c-7e7a-415b-a6c0-1cae4a0d256e" already exists
 - 17d99572-793d-41ae-8b55-cee30db13fa2: (409) rule_id: "17d99572-793d-41ae-8b55-cee30db13fa2" already exists
 - 38accba8-894a-4f32-98d5-7cb01c82f5d6: (409) rule_id: "38accba8-894a-4f32-98d5-7cb01c82f5d6" already exists
 - e1b7d2a6-d23a-4747-b621-d249d83162ea: (409) rule_id: "e1b7d2a6-d23a-4747-b621-d249d83162ea" already exists

Import a fulle directory, with -o forcing the updates successfully

(venv312) ➜  detection-rules-fork git:(refresh-kibana-module-with-new-APIs) ✗ python -m detection_rules kibana import-rules -d test-export-rules -o
Loaded config file: /Users/jibarra/PycharmProjects/detection-rules-fork/.detection-rules-cfg.yaml

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

DEBUG MODE ENABLED
23 rule(s) successfully imported
 - ee663abc-fb77-49d2-a7c5-204b9cf888ca
 - 50887ba8-aaaa-bbbb-a038-f661ea17fbcd
 - 50887ba8-7ff7-11ee-a038-f661ea17fbcd
 - 8a0fbd26-867f-11ee-947c-f661ea17fbcd
 - aaaaaaaa-f861-414c-8602-150d5505b777
 - 2f8a1226-5720-437d-9c20-e0029deb6194
 - cd66a5af-e34b-4bb0-8931-57d0a043f2ef
 - 2d8043ed-5bda-4caf-801c-c1feb7410504
 - d76b02ef-fc95-4001-9297-01cb7412232f
 - cc382a2e-7e52-11ee-9aac-f661ea17fbcd
 - 260486ee-7d98-11ee-9599-f661ea17fbcd
 - ee39a9f7-5a79-4b0a-9815-d36b3cf28d3e
 - 1ceb05c4-7d25-11ee-9562-f661ea17fbcd
 - 2e56e1bc-867a-11ee-b13e-f661ea17fbcd
 - 621e92b6-7e54-11ee-bdc0-f661ea17fbcd
 - a198fbbd-9413-45ec-a269-47ae4ccf59ce
 - 29b53942-7cd4-11ee-b70e-f661ea17fbcd
 - aaec44bc-d691-4874-99b2-48ab7392dfd5
 - 40e1f208-0f70-47d4-98ea-378ccf504ad3
 - 5e9bc07c-7e7a-415b-a6c0-1cae4a0d256e
 - 17d99572-793d-41ae-8b55-cee30db13fa2
 - 38accba8-894a-4f32-98d5-7cb01c82f5d6
 - e1b7d2a6-d23a-4747-b621-d249d83162ea

detailed API calls

    from kibana import definitions

    rids = ['40e1f208-0f70-47d4-98ea-378ccf504ad3', '5e9bc07c-7e7a-415b-a6c0-1cae4a0d256e']

    # with TypedDict, either is valid, both with static type checking
    st = definitions.RuleBulkSetTags(type='set_tags', value=['tag1', 'tag2'])
    dt: definitions.RuleBulkDeleteTags = {'type': 'delete_tags', 'value': ['tag1', 'tag2']}
    with kibana:
        r1 = RuleResource.bulk_enable(rids, dry_run=True)
        r2 = RuleResource.bulk_disable(rids, dry_run=True)
        r3 = RuleResource.bulk_duplicate(rids, dry_run=True)
        r4 = RuleResource.bulk_export(rids)
        r5 = RuleResource.bulk_edit(edit_object=[st, dt], rule_ids=rids, dry_run=True)
        r6 = RuleResource.bulk_delete(rids, dry_run=True)

Returned data:

kibana/connector.py

detection_rules/kbwrap.py

Mikaayenson · 2024-02-25T02:31:13Z

BTW, is this related? #3296

kibana/resources.py

eric-forte-elastic · 2024-04-25T21:38:28Z

Tested additional case with having specified a Kibana config and functions correctly 🟢

Output

detection-rules on  refresh-kibana-module-with-new-APIs is  v0.1.0 via  v3.12.2 (detection-rules-build) on  eric.forte took 12s
❯ python -m detection_rules kibana --cloud-id *** export-rules -d test-export-rules --skip-errors

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

kibana_user: eric.forte
kibana_password:
- skipping First Time Seen AWS Secret Value Accessed in Secrets Manager - ValidationError
3 rules exported
2 rules converted
2 saved to test-export-rules
1 errors saved to test-export-rules/_errors.txt

detection-rules on  refresh-kibana-module-with-new-APIs [?] is  v0.1.0 via  v3.12.2 (detection-rules-build) on  eric.forte took 13s
❯ ls test-export-rules/
adfind_command_activity.toml  clearing_windows_console_history.toml  _errors.txt

detection-rules on  refresh-kibana-module-with-new-APIs [?] is  v0.1.0 via  v3.12.2 (detection-rules-build) on  eric.forte
❯ cat test-export-rules/_errors.txt
───────┬───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
       │ File: test-export-rules/_errors.txt
───────┼───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
   1   │ - First Time Seen AWS Secret Value Accessed in Secrets Manager - {'rule': [ValidationError({'type': ['Must be equal to eql.'], 'language': ['Must be equal to eql.']}), ValidationError({'type': ['Must be equal to esql.'], 'language': ['Must be equ
       │ al to esql.']}), ValidationError({'type': ['Must be equal to threshold.'], 'threshold': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to threat_match.'], 'threat_mapping': ['Missing data for required field.'], '
       │ threat_index': ['Missing data for required field.']}), ValidationError({'type': ['Must be equal to machine_learning.'], 'anomaly_threshold': ['Missing data for required field.'], 'machine_learning_job_id': ['Missing data for required field.']}),
       │ ValidationError({'type': ['Must be equal to query.']}), ValidationError({'new_terms': ['Missing data for required field.']})]}
───────┴───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

detection_rules/mixins.py

detection_rules/rule.py

detection_rules/mixins.py

lib/kibana/kibana/resources.py

eric-forte-elastic · 2024-04-26T16:43:58Z

Also since we are updating lib/kibana/ I think we may need to update the library version and the dependency right?

Mikaayenson

Great work! LGTM! 🚢

brokensound77 · 2024-04-26T16:54:10Z

Also since we are updating lib/kibana/ I think we may need to update the library version and the dependency right?

Good catch, thanks! We should explore adding another workflow to catch when bumps are missed

* Refresh Kibana module with API updates * add import/export commands * rename repo commands * add RawRuleCollection and DictRule objects * save exported rules to files; rule.from_rule_resource * strip unknown fields in schema * add remote cli test * update docs * bump kibana lib version --------- Co-authored-by: brokensound77 <brokensound77@users.noreply.github.com> (cherry picked from commit c567d37)

brokensound77 commented Feb 22, 2024

View reviewed changes

kibana/connector.py Outdated Show resolved Hide resolved

brokensound77 commented Feb 22, 2024

View reviewed changes

kibana/connector.py Outdated Show resolved Hide resolved

Refresh Kibana module with API updates

8f826e1

brokensound77 force-pushed the refresh-kibana-module-with-new-APIs branch from f183117 to 8f826e1 Compare February 22, 2024 04:25

add import/export commands

4f18a52

brokensound77 commented Feb 22, 2024

View reviewed changes

detection_rules/kbwrap.py Outdated Show resolved Hide resolved

Mikaayenson reviewed Feb 22, 2024

View reviewed changes

detection_rules/kbwrap.py Show resolved Hide resolved

Mikaayenson reviewed Feb 22, 2024

View reviewed changes

detection_rules/kbwrap.py Show resolved Hide resolved

This comment was marked as resolved.

Sign in to view

brokensound77 requested a review from eric-forte-elastic March 14, 2024 02:48

brokensound77 and others added 3 commits March 14, 2024 09:28

Merge branch 'main' into refresh-kibana-module-with-new-APIs

aa1a824

rename repo commands

4d031b0

merge upstream updates

9a3d809

brokensound77 marked this pull request as ready for review March 14, 2024 16:34

brokensound77 requested a review from terrancedejesus as a code owner March 14, 2024 16:34

github-actions bot added the backport: auto label Mar 14, 2024

brokensound77 added enhancement New feature or request kibana-module related to the kibana module blocked and removed backport: auto labels Mar 14, 2024

github-actions bot added the backport: auto label Mar 14, 2024

brokensound77 commented Mar 14, 2024

View reviewed changes

kibana/resources.py Outdated Show resolved Hide resolved

botelastic bot added the python Internal python for the repository label Mar 14, 2024

Mikaayenson assigned brokensound77 Mar 15, 2024

This comment was marked as resolved.

Sign in to view

eric-forte-elastic reviewed Apr 25, 2024

View reviewed changes

detection_rules/mixins.py Outdated Show resolved Hide resolved

eric-forte-elastic reviewed Apr 25, 2024

View reviewed changes

detection_rules/rule.py Outdated Show resolved Hide resolved

brokensound77 removed the blocked label Apr 26, 2024

brokensound77 commented Apr 26, 2024

View reviewed changes

detection_rules/mixins.py Outdated Show resolved Hide resolved

brokensound77 commented Apr 26, 2024

View reviewed changes

lib/kibana/kibana/resources.py Outdated Show resolved Hide resolved

brokensound77 commented Apr 26, 2024

View reviewed changes

lib/kibana/kibana/resources.py Outdated Show resolved Hide resolved

lint

eaf7b65

eric-forte-elastic reviewed Apr 26, 2024

View reviewed changes

lib/kibana/kibana/resources.py Outdated Show resolved Hide resolved

eric-forte-elastic approved these changes Apr 26, 2024

View reviewed changes

Mikaayenson approved these changes Apr 26, 2024

View reviewed changes

bump kibana lib version

96aa750

brokensound77 force-pushed the refresh-kibana-module-with-new-APIs branch from 2b3c2fe to 96aa750 Compare April 26, 2024 16:52

brokensound77 merged commit c567d37 into elastic:main Apr 26, 2024
14 checks passed

brokensound77 deleted the refresh-kibana-module-with-new-APIs branch April 26, 2024 21:25

brokensound77 mentioned this pull request Apr 27, 2024

[Discussions] DaC Community Feedback #3298

Open

Mikaayenson mentioned this pull request Apr 27, 2024

[Docs] Update docs based on alpha release elastic/DaC-Reference#5

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refresh Kibana module with API updates #3466

Refresh Kibana module with API updates #3466

brokensound77 commented Feb 22, 2024 •

edited

Loading

Mikaayenson commented Feb 25, 2024

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

eric-forte-elastic commented Apr 25, 2024

eric-forte-elastic commented Apr 26, 2024

Mikaayenson left a comment

brokensound77 commented Apr 26, 2024

Refresh Kibana module with API updates #3466

Refresh Kibana module with API updates #3466

Conversation

brokensound77 commented Feb 22, 2024 • edited Loading

Issues

Summary

Details

Testing

Mikaayenson commented Feb 25, 2024

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

eric-forte-elastic commented Apr 25, 2024

eric-forte-elastic commented Apr 26, 2024

Mikaayenson left a comment

Choose a reason for hiding this comment

brokensound77 commented Apr 26, 2024

brokensound77 commented Feb 22, 2024 •

edited

Loading