Ingest remove processor to support wildcards and excludes

[warrickhill]

Is your feature request related to a problem? Please describe.
When receiving a doc to index it could contain fields I do not wish it index / store. Ingest pipelines have the handy "remove" processor however you must know the name of the field you want removed, wildcards do not work.

Describe the solution you'd like
I would like the ingest "remove" processor to support wildcards and excludes, in a similar way to how .gitignore files work. I don't believe that is would be a breaking change.

Describe alternatives you've considered
I could use a script processor to remove fields i don't need / want but this is way more complex to maintain.

Additional context
None

[gaobinlong]

Maybe it's not easy to support wildcards for remove processor, because field name supports *, users can set field name to a* or *b, so it's dangerous to support wildcard, users may want to delete the single field a* but all fields starts with a are deleted. However, it makes sense that remove processor supports excludes, something like a deny list.

[gaobinlong]

In PR #10967, we add a new parameter exclude_field to the remove ingest processor, the parameter can be used to remove the fields which are not in the specified list. As we discussed in that PR, we'll implement a new ingest processor called remove_by_pattern processor, because if we support field patterns(like a*) in the remove ingest processor, we should add two extra parameters like field_pattern and exclude_filed_pattern, combined with the existing field and exclude_field, there're too many options to remove fields, which makes the usage complicated and confuses users, so implementing a new ingest processor which supports removing fields by wildcards can make the usage more clear.

[kiranprakash154]

Hi, are we on track for this to be released in 2.12 ?

[gaobinlong]

Hi, are we on track for this to be released in 2.12 ?

Yeah, there are two related PRs for this issue, both of them target for 2.12.0.

[andrross]

Resolving as it appears this was released in 2.12