[BUG] nested field type in index template/pattern breaks detector creation

[mmguero]

What is the bug?

When attempting to creating a security analytics detector with an index matching an index pattern where the index template has field of type nested this error is given: [security_analytics_exception] null cannot be cast to non-null type kotlin.collections.MutableMap<kotlin.String, kotlin.Any>. The nested field need not be populated in any documents actually stored in the index. Removing the nested fields from the JSON of my composable index template prior to its creation is a workaround for the error.

Error message in console:

Security Analytics - DetectorsService - createDetector: StatusCodeError: [security_analytics_exception] null cannot be cast to non-null type kotlin.collections.MutableMap<kotlin.String, kotlin.Any>
    at respond (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:349:15)
    at checkRespForFailure (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/transport.js:306:7)
    at HttpConnector.<anonymous> (/usr/share/opensearch-dashboards/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)
    at IncomingMessage.wrapper (/usr/share/opensearch-dashboards/node_modules/lodash/lodash.js:4991:19)
    at IncomingMessage.emit (node:events:529:35)
    at IncomingMessage.emit (node:domain:489:12)
    at endReadableNT (node:internal/streams/readable:1400:12)
    at processTicksAndRejections (node:internal/process/task_queues:82:21) {
  status: 500,
  displayName: 'InternalServerError',
  path: '/_plugins/_security_analytics/detectors',
  query: {},
  body: {
    error: {
      root_cause: [Array],
      type: 'security_analytics_exception',
      reason: 'null cannot be cast to non-null type kotlin.collections.MutableMap<kotlin.String, kotlin.Any>',
      caused_by: [Object]
    },
    status: 500
  },
  statusCode: 500,
  response: '{"error":{"root_cause":[{"type":"security_analytics_exception","reason":"null cannot be cast to non-null type kotlin.collections.MutableMap<kotlin.String, kotlin.Any>"}],"type":"security_analytics_exception","reason":"null cannot be cast to non-null type kotlin.collections.MutableMap<kotlin.String, kotlin.Any>","caused_by":{"type":"exception","reason":"org.opensearch.commons.alerting.util.AlertingException: null cannot be cast to non-null type kotlin.collections.MutableMap<kotlin.String, kotlin.Any>"}},"status":500}',
  toString: [Function (anonymous)],
  toJSON: [Function (anonymous)]
}

How can one reproduce the bug?

I've broken it all down into a self-contained example, which is stored here in a repository where I keep various docker-based projects.

As a little bit of context before I go through these steps, this started with me trying to get security analytics to work with an index template I am using that is based on the Elastic Common Schema. My process involves downloading the ECS sample templates and then doing some massaging of that template prior to its import into OpenSearch (changing some field types, mainly) so that it will import. However, the nested field type should be supported, I think, or at the very least it should not cause security analytics to break, as it's defined as a field type within OpenSearch as well.

Steps to reproduce the behavior:

1. Have opensearch 2.18.0 and opensearch-dashboards running 2.18.0 (I am doing this within docker compose, with some slight modifications using the official opensearch / dashboards images as the base of mine)
2. Create a composable index template where one of the components has a nested field. I did this by downloading the ECS sample templates and then doing some massaging of that template. The one that finally helped me narrow in on this was the dll template, which has a nested field and also a flattened field (which I change to nested prior to import). I'm scripting the creation of the templates after opensearch and dashboards have started up.
3. Index a document (see the video example below), I'm using this curl command:

curl -k --config .creds.curlrc -sSL -XPOST -H 'Content-Type: application/json'   "https://localhost:9200/ecs-$(date -u +'%Y%m%d')/_doc" -d"
{
 \"dns.answers.type\": \"CNAME\",
 \"dns.question.name\": \"www.example.com\",
 \"dns.question.registered_domain\": \"www.example.com\",
 \"ecs.version\": \"1.0.0\",
 \"@timestamp\": \"$(date -u +'%Y-%m-%dT%H:%M:%S.%3NZ')\"
}"

4. In Dashboards Management, create an index pattern (in my example, the index pattern is ecs-*
5. In Security Analytics, try to create a detector, specifying the index created by the document above, selecting the DNS ruleset, and mapping the @timestamp field to timestamp (see the video below)
6. The error [security_analytics_exception] null cannot be cast to non-null type kotlin.collections.MutableMap<kotlin.String, kotlin.Any> is given

Compare this to a working scenario, where everything is exactly the same, except for one difference: I use jq to remove the nested fields from the template .json files prior to their creation. Everything else is exactly the same, and the detector is created correctly.

What is the expected behavior?

The security analytics detector should be created correctly regardless of there being nested field mappings defined in the index template

What is your host/environment?

• OS: Linux x86_64
• Version: 2.18.0
• Plugins: lguillaud/osd_transform_vis is the only non-default plugin installed in my environment, but I don't believe that has bearing on this issue.

Do you have any screenshots?

Screenshot of error:

Video of entire workflow (broken state):

test.mp4

Do you have any additional context?

• I originally brought this up on the forums last fall but ended up on other things and am just now coming back to it.
• [BUG] Create detector on datastream #768 also mentions this error, but it was closed by its creator with the comment "The indices mapping was not consistant" and no further follow up or details

[mmguero]

Hmmmm... interesting. I just found another wrinkle, which I'm going to pull on on my end. I mentioned above that in some of my massaging of the ECS templates one of the things I do is change flattened to nested. If I change my code to completely remove the flattened fields, and leave the nested ones in place, I don't get the error! I'm not sure exactly what that means. I think the opensearch flat_object field type which is kind of new-ish (when I originally wrote this code, it was pre 2.7.0 so flat_object wasn't an option). That's probably the right mapping for this, so I'll try that.

[mmguero]

So continuing down that thread I did that: changed my code that was changing the Elasticsearch ECS flattened field to nested to rather changed flattened to flat_object. Then, I left all of the nested fields alone. And I am no longer getting this error.

So I guess I'll leave it up to you, the devs of this project, whether this is something you want to look at anyway or just close and mark it as "invalid."

TL;DR: To make Elasticsearch ECS templates compatible with OpenSearch I was changing flattened to nested, for example, for fields like this. Doing so was causing the error listed in my original report above. I'm now changing flattened to flat_object and no longer have the error.

[mmguero]

Moving beyond the ECS schema though, for some of my other schemas that still use the nested field type, I do still have this bug, so I think it is still an issue. I'll find another example where nested breaks as well if it's useful.

[mmguero]

The more I play with it, the more I do think it's a bug. I'm using nested field types in what I believe are totally valid ways and getting the error. I'm still trying to figure out what's the distinction between nested fields that do cause the error and nested fields that don't.

[mmguero]

It seems like it might be an issue when the type is nested but no other fields are initially listed as properties, maybe?

I.e.,

"details": {
  "type": "nested",
  "properties": {
    "target": { "type": "keyword" },
    "size": { "type": "long" },
    "used": { "type": "long" },
    "avail": { "type": "long" }
  }
}

seems to be okay, while:

"details": { "type": "nested" }

is a problem? Maybe that's not a valid use of nested, the documentation isn't totally clear on that.

[mmguero]

I do think my last comment narrows down on the issue. It seems to me that if the type is nested it only causes the problem with the creation of the template if there are no subfields initially listed in the properties for that nested field.

I think this is a valid use case, though, as the documentation says that for nested fields, if the dynamic property is true (the default), "specifies whether new fields can be dynamically added to the object," which to me implies that there don't have to be fields explicitly added to it beforehand.