[Bug] Index information on the TOML files does not make it to Kibana SIEM rules

[ArdRay]

Describe the bug
During testing of the new Kibana upload feature, we have identified that the index information on the TOML configuration does not propagate to the rules in the Kibana SIEM app.

To Reproduce
Steps to reproduce the behavior:

1. Create a TOML detection rule which follows the guidelines provided. Example:

[metadata]
creation_date = "2020/08/11"
ecs_version = ["1.5.0"]
maturity = "development"
updated_date = "2020/08/11"

[rule]
author = ["Rule export"]
description = "The contents of crontab have been modified by a non-root user."
tags = ["prod"]
index = ["filebeat-*"]
language = "kuery"
license = "Elastic License"
name = "Cron - Crontab entry changed"
risk_score = 15
rule_id = "9bad9907-1969-4a70-bfbd-367f0dcc0203"
severity = "low"
type = "query"

query = '''
event.module: cron and event.action: changed and not user.name: root
'''

[[rule.threat]]
framework = "MITRE ATT&CK"
[[rule.threat.technique]]
id = "T1053"
name = "Scheduled Task"
reference = "https://attack.mitre.org/techniques/T1053/"


[rule.threat.tactic]
id = "TA0002"
name = "Execution"
reference = "https://attack.mitre.org/tactics/TA0002/"

[[rule.threat]]
framework = "MITRE ATT&CK"
[[rule.threat.technique]]
id = "T1053"
name = "Scheduled Task"
reference = "https://attack.mitre.org/techniques/T1053/"


[rule.threat.tactic]
id = "TA0003"
name = "Persistence"
reference = "https://attack.mitre.org/tactics/TA0003/"

2. Ensure all tests pass successfully.

python3 -m detection_rules toml-lint
python3 -m detection_rules validate-all
python3 -m detection_rules test 

3. Verify that the index information makes it to the api-call.

python3 -m detection_rules view-rule -f ./rules/cron_crontab_entry_changed.toml --as-api

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█
{
  "author": [
    "Rule export"
  ],
  "description": "The contents of crontab have been modified by a non-root user.",
  "index": [
    "filebeat-*"
  ],
  "language": "kuery",
  "license": "Elastic License",
  "name": "Cron - Crontab entry changed",
  "query": "event.module: cron and event.action: changed and not user.name: root",
  "risk_score": 15,
  "rule_id": "9bad9907-1969-4a70-bfbd-367f0dcc0203",
  "severity": "low",
  "tags": [
    "prod"
  ],
  "threat": [
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0002",
        "name": "Execution",
        "reference": "https://attack.mitre.org/tactics/TA0002/"
      },
      "technique": [
        {
          "id": "T1053",
          "name": "Scheduled Task",
          "reference": "https://attack.mitre.org/techniques/T1053/"
        }
      ]
    },
    {
      "framework": "MITRE ATT&CK",
      "tactic": {
        "id": "TA0003",
        "name": "Persistence",
        "reference": "https://attack.mitre.org/tactics/TA0003/"
      },
      "technique": [
        {
          "id": "T1053",
          "name": "Scheduled Task",
          "reference": "https://attack.mitre.org/techniques/T1053/"
        }
      ]
    }
  ],
  "type": "query"
}

4. Push to kibana.

python3 -m detection_rules kibana-upload --url $KIBANA_URL -u $KIBANA_USERNAME -p $KIBANA_PASSWORD rules/*.toml
█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█
Loading rules from /opt/rule_engine/detection-rules/rules
Loaded 1 rules
Successfully uploaded 1 rules

5. Verify the rule within the SIEM Kibana app.
   
   

Expected behavior
The index should have propagated to the rule within the SIEM app, although as it can be seen from the screenshots above, that information is missing. The index field is empty.

Desktop (please complete the following information):

• OS: MacOS
• Version: 10.15.5
• Python version: 3.6.8
• ELK version: Tested on 7.8.0 and 7.8.1, the issue persist on both.

[rw-access]

@spong do you have any ideas about this behavior at the API level?

[rw-access]

ah could also be at the schema downgrade level. I'll check the python side