[Security Solutions] Detection Engine and Security Pages do not operate with runtime fields defined in Kibana Indexes

[FrankHassanabad]

Describe the bug:
When you define a runtime mapping field in a Kibana index, the runtime field is there to be used for runtime-search-requests and is actually not directly in an Elastic mapping.

When you use Discover it knows to pull the runtime fields from the Kibana index and knows to attach the runtime mappings directly to the query.

However, the SecuritySolution pages and the Detection Engine both do not utilize this runtime mapping information from a Kibana Index and you will get inconsistent results between Discover, SecuritySolution pages, and the Detection Engine when you query the same Kibana Indexes.

Really, we should attach the runtime mappings to the queries we find from multiple indexes best we can/best effort and I do say best we can/best effort since SecuritySolution pages and detection engine can both utilize multiple Kibana indexes for their source index produces single response results. So you could end up with conflicts between runtime mappings between multiple Kibana index patterns just like you can with multiple Elasticsearch index patterns. However, in most cases I would imagine that you could merge the runtime fields in the Kibana index patterns push them down to the services and use them without conflicts.

As of right now, since we allow Discover to make it incredibly simple to add these runtime mapping fields directly in Discover to Kibana indexes (A Kibana Index could also be a signals index), we should fix this to operate as expected even if we ignore conflicts or choose one conflict of a runtime field over another for those mis-use cases.

Steps to reproduce:

Open dev tools and add a simple mapping

# Add a mapping which contains a "@timestamp" property
DELETE frank-test-delme-16
PUT frank-test-delme-16
{
   "mappings": {
    "dynamic": "strict",
    "properties": {
      "@timestamp": {
        "type": "date"
      }
    }
  }
}

Add a single simple document to it:

# Add a single document with the latest time
PUT frank-test-delme-16/_doc/1
{
  "@timestamp": "2021-06-28T23:19:26.744Z"
}

Create a Kibana Index from the mapping:

Create a runtime mapping within Discover for it which emits a host.name as an example:

Notice you get host.name in Discover:

Open dev tools and confirm that Discover attaches the runtime field as a query mapping from the Kibana Index pattern:

Go to SecuritySolution and select the index as the data source but notice that it does not show up in the tables and also notice that it does not attach the runtime mapping as part of the query in dev tools:

If you go to the detections page, you can create a detections using host.name or *:* and you will notice the same effect since we do not introspect and push down data such as runtime mappings from Kibana indexes as runtime query mappings.

Expected behavior:
You get the same behavior and data from discover also in your detections results and in your SecuritySolution queries by using runtime mapping query fields from a Kibana Index.

Kibana version:
7.13.0

Elasticsearch version:
7.13.0

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[stephmilovic]

not a bug, feature not implemented yet. Issue tracking this feature: https://github.com/elastic/security-team/issues/772

[yctercero]

Per @stephmilovic - this feature is being addressed by her open PRs (#114806, #114907)

[jethr0null]

Related issue:
[Booking.com] Provide list of index patterns when creating a Detection rule
#12835

[jethr0null]

Data view support was added in 8.4. Detection rules now have access to runtime fields configured in the referenced data view.