Add export-rules command

[brokensound77]

Issues

None

Summary

Add export-rules command to CLI which takes all specified rules (and rule files) and exports them into a consolidated ndjson file which is importable by the Kibana security app.

Details

python -m detection_rules export-rules rules-dir

creates:

{"author": ["Elastic"], "description": "Adversaries may collect the keychain storage data from a system to acquire credentials. Keychains are the built-in way for macOS to keep track of users' passwords and credentials for many services and features such as WiFi passwords, websites, secure notes, certificates, and Kerberos.", "from": "now-9m", "index": ["auditbeat-*", "logs-endpoint.events.*"], "language": "kuery", "license": "Elastic License", "meta": {"original": {"creation_date": "2020/08/14", "id": "96e90768-c3b7-4df6-b5d9-6237f8bc36a8", "maturity": "production", "updated_date": "2020/11/03"}}, "name": "Compression of Keychain Credentials Directories", "query": "event.category:process and event.type:(start or process_started) and process.name:(zip or tar or gzip or 7za or hdiutil) and process.args:(\"/Library/Keychains/\" or \"/Network/Library/Keychains/\" or \"~/Library/Keychains/\")", "references": ["https://objective-see.com/blog/blog_0x25.html"], "risk_score": 73, "rule_id": "a1fa5c12-90da-4a43-9bab-4d6d8a9ff053", "severity": "high", "tags": ["Elastic", "Host", "macOS", "Threat Detection", "Credential Access"], "threat": [{"framework": "MITRE ATT&CK", "tactic": {"id": "TA0006", "name": "Credential Access", "reference": "https://attack.mitre.org/tactics/TA0006/"}, "technique": [{"id": "T1142", "name": "Keychain", "reference": "https://attack.mitre.org/techniques/T1142/"}]}], "type": "query", "version": 1}
{"author": ["Elastic"], "description": "Identifies the use of the Kerberos credential cache (kcc) utility to dump locally cached Kerberos tickets.", "from": "now-9m", "index": ["auditbeat-*", "logs-endpoint.events.*"], "language": "kuery", "license": "Elastic License", "meta": {"original": {"creation_date": "2020/08/14", "id": "ad88231f-e2ab-491c-8fc6-64746da26cfe", "maturity": "production", "updated_date": "2020/11/03"}}, "name": "Kerberos Cached Credentials Dumping", "query": "event.category:process and event.type:(start or process_started) and process.name:kcc and process.args:copy_cred_cache", "references": ["https://github.com/EmpireProject/EmPyre/blob/master/lib/modules/collection/osx/kerberosdump.py", "https://opensource.apple.com/source/Heimdal/Heimdal-323.12/kuser/kcc-commands.in.auto.html"], "risk_score": 73, "rule_id": "d8d6b06b-5f4f-44ef-846d-e66745e7e5e4", "severity": "high", "tags": ["Elastic", "Host", "macOS", "Threat Detection", "Credential Access"], "threat": [{"framework": "MITRE ATT&CK", "tactic": {"id": "TA0006", "name": "Credential Access", "reference": "https://attack.mitre.org/tactics/TA0006/"}, "technique": [{"id": "T1003", "name": "OS Credential Dumping", "reference": "https://attack.mitre.org/techniques/T1003/"}]}], "type": "query", "version": 1}
{"author": ["Elastic"], "description": "Identifies a high number (20) of macOS SSH KeyGen process executions from the same host. An adversary may attempt a brute force attack to obtain unauthorized access to user accounts.", "from": "now-9m", "index": ["auditbeat-*", "logs-endpoint.events.*"], "language": "kuery", "license": "Elastic License", "meta": {"original": {"creation_date": "2020/11/16", "id": "ace1e989-a541-44df-93a8-a8b0591b63c0", "maturity": "production", "updated_date": "2020/11/16"}}, "name": "Potential SSH Brute Force Detected", "query": "event.category:process and event.type:start and process.name:\"sshd-keygen-wrapper\" and process.parent.name:launchd", "references": ["https://themittenmac.com/detecting-ssh-activity-via-process-monitoring/"], "risk_score": 47, "rule_id": "f60da14b-f7be-436b-8471-934df7214268", "severity": "medium", "tags": ["Elastic", "Host", "macOS", "Threat Detection", "Credential Access"], "threat": [{"framework": "MITRE ATT&CK", "tactic": {"id": "TA0006", "name": "Credential Access", "reference": "https://attack.mitre.org/tactics/TA0006/"}, "technique": [{"id": "T1110", "name": "Brute Force", "reference": "https://attack.mitre.org/techniques/T1110/"}]}], "threshold": {"field": "host.id", "value": 20}, "type": "threshold", "version": 1}
{"author": ["Elastic"], "description": "Detects use of the systemsetup command to enable remote SSH Login.", "from": "now-9m", "index": ["auditbeat-*", "logs-endpoint.events.*"], "language": "kuery", "license": "Elastic License", "meta": {"original": {"creation_date": "2020/08/18", "id": "5ae4e6f8-d1bf-40fa-96ba-e29645e1e4dc", "maturity": "production", "updated_date": "2020/11/03"}}, "name": "Remote SSH Login Enabled via systemsetup Command", "query": "event.category:process and event.type:(start or process_started) and process.name:systemsetup and process.args:(\"-f\" and \"-setremotelogin\" and on)", "references": ["https://documents.trendmicro.com/assets/pdf/XCSSET_Technical_Brief.pdf", "https://ss64.com/osx/systemsetup.html"], "risk_score": 47, "rule_id": "cbb41b02-8c4c-4068-8695-b1764a2a67f2", "severity": "medium", "tags": ["Elastic", "Host", "macOS", "Threat Detection", "Lateral Movement"], "threat": [{"framework": "MITRE ATT&CK", "tactic": {"id": "TA0008", "name": "Lateral Movement", "reference": "https://attack.mitre.org/tactics/TA0008/"}, "technique": [{"id": "T1021", "name": "Remote Services", "reference": "https://attack.mitre.org/techniques/T1021/"}]}], "type": "query", "version": 1}

Exporting rules for older versions of Kibana

python -m detection_rules export-rules --downgrade-version 7.9 -d rules-dir --skip-unsupported 

Exported 4 rules into exports/20201218T233226L.ndjson
Skipped 9 unsupported rules: 
- Unsupported rule type eql: 47f76567-d58a-4fed-b32b-21f571e28910 - Apple Script Execution followed by Network Connection
- Unsupported rule type eql: f0b48bbc-549e-4bcf-8ee0-a7a72586c6a7 - Attempt to Remove File Quarantine Attribute
- Unsupported rule type eql: f0eb70e9-71e9-40cd-813f-bf8e8c812cb1 - Execution with Explicit Credentials via Apple Scripting
- Unsupported rule type eql: 082e3f8c-6f80-485c-91eb-5b112cb79b28 - Launch Agent Creation or Modification and Immediate Loading
- Unsupported rule type eql: 9d19ece6-c20e-481a-90c5-ccca596537de - LaunchDaemon Creation or Modification and Immediate Loading
- Unsupported rule type eql: c292fa52-4115-408a-b897-e14f684b3cb7 - Persistence via Folder Action Script
- Unsupported rule type eql: 5d0265bf-dea9-41a9-92ad-48a8dcd05080 - Persistence via Login or Logout Hook
- Unsupported rule type eql: 38948d29-3d5d-42e3-8aec-be832aaaf8eb - Prompt for Credentials with OSASCRIPT
- Unsupported rule type eql: d461fac0-43e8-49e2-85ea-3a58fe120b4f - Shell Execution via Apple Scripting

[threat-punter]

Looks great! I tested the --rule-file, --directory, --outfile, and --replace-id options successfully. I imported a rule in Kibana and that worked.

Shall we add the exports directory to .gitignore?

[brokensound77]

Looks great! I tested the --rule-file, --directory, --outfile, and --replace-id options successfully. I imported a rule in Kibana and that worked.

Shall we add the exports directory to .gitignore?

Great idea!

[brokensound77]

I added the ability to downgrade rules on export as we discussed @rw-access. Should be good for a final review from you

[rw-access]

one nit, otherwise looks good