[Security Solution] Allow users to edit required_fields field for custom rules

[nikitaindik]

Resolves: #173594
Flaky test runner: https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5915

Summary

This PR adds an ability to add and edit custom rule's required fields. "Required fields" is an optional field that shows the user which Elasticsearch fields are needed for the rule to run properly. The values in "required fields" don't affect rule execution in any way. It's purely documentational, similar to "setup guide" and "investigation guide". This functionality is added to both rule creation and rule editing screens. It's available for all rule types except ML.

Details

The basic flow goes like this: first you specify your index patterns (or a data view), then you can set required fields for these index patterns. Once a user adds a required field and chooses its name, he can then choose its type from the dropdown. The first available type for the field name selected automatically. User can also add their own custom names and types.

Warnings

If a field that is not present in the selected index pattern, you will see a warning message.
This can happen in the following cases:

• You have specified an index pattern, selected a required field from this index pattern, and then removed this index pattern.
• The index doesn't yet exist. For example, you have installed a prebuilt rule but the data for it hasn't been ingested yet, so there's no index yet.
• The index was removed.
• The mappings for the index were changed and the field is no longer present.

In any of these cases, you'll see a general warning message above the form section. And then also a more specific warning message next to the field that is causing the issue.

ESQL and ML rules

Here's how available dropdown options are determined for different rule types:

For all rule types except ESQL and ML, we take the index patterns specified by the user and fetch their mappings. Then we use these fields and types to populate the dropdowns.

For ESQL rules we parse index patterns out of the query since there's no explicit index pattern form field. We then fetch the mappings for these index patterns and use them to populate the dropdowns.

For ML rules, we don't show "required fields" at all. ML rules are a special case.

1. The concept of "required fields" is sort of handled during creation of the ML job itself, where the user specifies the fields that are required for the job to run.
2. In the ML rule creation/editing interface, we don't display the index patterns a rule operates on. So, even if we allowed specifying required fields, the user would need to check the ML job details to see the index patterns the job uses.
3. The ML job dropdown includes both existing and not-yet-created jobs. We can't get index patterns for jobs that don't exist yet, so we can't fill the dropdowns with fields and types.

Screenshots

[banderror]

@nikitaindik Can you give an update on the progress on this PR? Is anything blocking you from completing the test coverage and opening it for review?

[nikitaindik]

@banderror Sorry for the late reply. My plan is to wrap up the tests and open the PR for review today.

[nikitaindik]

/ci

[nikitaindik]

/ci

[joepeeples]

Feature looks great! I suggested some tweaks to UI text here and there, let me know if you have any questions. Thanks!

[xcrzx]

Nit: Do we still need to use useMemo in this component? There are a lot of moving parts, and most of the variables in this component are written without using useMemo. I'd say we either need to wrap all of them with memoization or none, but not a mix of the two approaches. If the performance is okay, given the watch optimization, I'd say we can omit using useMemo.

[nikitaindik]

I thought about removing this, but then decided to leave it in, because fieldsWithTypes array can be really long – almost 2000 items for default index patterns.

[xcrzx]

Tested the PR locally, and everything works as expected. The code changes also look good to me. Thank you for the implementation, @nikitaindik 👍

[dplumlee]

Thanks for addressing my comments @nikitaindik, I like the look of the default state a lot more now!

[jpdjere]

Exploratory testing

Tested and working as expected:

• UI to add required fields works as expected, no noticeable bugs ✅
• Fields appearing as options -with their expected types- when indexing documents into a single index that matches the selected index pattern ✅
• Fields appearing as options when indexing documents into a multiple indices that match the selected index pattern(s) ✅
• Fields appearing as options when selecting a Data View that contains the same indices from the previous point as before ✅
• Warning displayed when:
  • You have specified an index pattern, selected a required field from this index pattern, and then removed this index pattern. ✅ (warning doesn't appear until another index pattern or data view is selected, but it's acceptable)
  • The index doesn't yet exist. For example, you have installed a prebuilt rule but the data for it hasn't been ingested yet, so there's no index yet. ✅
  • The index was removed. ✅
  • The mappings for the index were changed and the field is no longer present.
• Different types of fields appear with correct icons in Rules Details ✅
• Different types of fields appear with correct icons in Discover ✅
• Creating a rule with required fields via API and editing required field via API is correctly reflected in new UI ✅

[joepeeples]

@nikitaindik Here's some revised popover text, after our chat about custom fields. I also wanted to make it clearer that this is just informational — the user isn't actually configuring the rule's logic to require these fields, they're basically just documenting how the rule works.

Suggested change

	defaultMessage="Choose the fields and data types needed for this rule to function. You can select the fields available in the rule's {source} index patterns or data view."
	defaultMessage="Create an informational list of fields and data types this rule needs to function. Select fields in the rule's {source} index patterns or data view, or type in custom fields."

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: d5988c5
• Cloud Deployment
• Kibana Serverless Image: docker.elastic.co/kibana-ci/kibana-serverless:pr-180682-d5988c56c6ef
• Security Deployment

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
securitySolution	5482	5492	+10

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
lists	142.0KB	142.1KB	+48.0B
securitySolution	15.1MB	15.1MB	+13.4KB
total			+13.4KB

Canvas Sharable Runtime

The Canvas "shareable runtime" is an bundle produced to enable running Canvas workpads outside of Kibana. This bundle is included in third-party webpages that embed canvas and therefor should be as slim as possible.

id	before	after	diff
module count	-	5405	+5405
total size	-	8.8MB	+8.8MB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
core	409.6KB	409.6KB	+48.0B

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
securitySolution	522	521	-1

Total ESLint disabled count

id	before	after	diff
securitySolution	604	603	-1

History

• 💚 Build #210380 succeeded f1bfb0d
• 💔 Build #210322 failed 9489507
• 💔 Build #210290 failed 1b1221b
• 💔 Build #210243 failed af57ac5

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @nikitaindik