[Auto Import] Improve the ECS mapping extraction logic

[ilyannn]

Release Notes

Automatic Import is more forgiving if an LLM returns an ECS mapping in a slightly unexpected format.

Summary

When implementing #194386 an issue has been encountered where Claude returns the field name date_format instead of expected date_formats and the ECS chain breaks down.

We add this case as a test to x-pack/plugins/integration_assistant/server/graphs/ecs/validate.test. Without the changes in this PR the list returned by findInvalidEcsFields is

      [
        'Reserved ECS field mapping identified for event.created : ai_postgres_202410050058.logs.column1.target',
        'Invalid ECS field mapping identified for 0.9 : ai_postgres_202410050058.logs.column1.confidence, ai_postgres_202410050058.logs.column5.confidence',
        'Invalid ECS field mapping identified for date : ai_postgres_202410050058.logs.column1.type, ai_postgres_202410050058.logs.column9.type',
        'Invalid ECS field mapping identified for 0.95 : ai_postgres_202410050058.logs.column12.confidence',
        'Invalid ECS field mapping identified for string : ai_postgres_202410050058.logs.column12.type, ai_postgres_202410050058.logs.column14.type, ai_postgres_202410050058.logs.column24.type, ai_postgres_202410050058.logs.column5.type, ai_postgres_202410050058.logs.column3.type, ai_postgres_202410050058.logs.column2.type',
        'Invalid ECS field mapping identified for 0.8 : ai_postgres_202410050058.logs.column9.confidence, ai_postgres_202410050058.logs.column3.confidence',
        'Invalid ECS field mapping identified for 0.7 : ai_postgres_202410050058.logs.column14.confidence, ai_postgres_202410050058.logs.column2.confidence',
        'Invalid ECS field mapping identified for 0.85 : ai_postgres_202410050058.logs.column24.confidence'
      ]

while with these changes the result does not contain any Invalid ECS field messages.

The key changes are in the processMapping function:

1. We made function more forgiving in regards to the input, accepting date_format in lieu of date_formats.
2. We have removed the collection of "other paths", that is, the reverse index for simple values like 0.8.

The latter change generally limits the impact of any other format issues in the ECS mapping in the future.

Additionally, the function has been renamed to extractECSMapping, its output type validated, and documentation has been added.

Checklist

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/security-scalability (Team:Security-Scalability)

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: b35c290

Metrics [docs]

✅ unchanged

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ilyannn