Add regex support functionality

[gonzaloe]

• Enhancement

What does this PR do?

Added regex support for drop_fields processor as requested in #12923

Fields that are declared between slashes (as in Painless scripts) are going to be considered regex and all matches within field names will be removed

Why is it important?

So deleting field-by-field with name can be avoided while using modules that declare a big amount of fields.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Configure a field using a regex pattern under drop_fields processor, on fields array, between slashes /. For e.g.:

drop_fields:
  fields: ["/field_to_be_deleted.*/"]

Related issues

• Closes [filebeat] Allow wildcards for drop_fields #12923

[cla-checker-service]

💚 CLA has been signed

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2022-05-01T18:12:43.474+0000

• Duration: 155 min 24 sec

Test stats 🧪

Test	Results
Failed	0
Passed	22259
Skipped	1934
Total	24193

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages and run the E2E tests.

• /beats-tester : Run the installation tests with beats-tester.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[rdner]

@kvch I suspect this PR will be affected by your refactoring here, right?

should we wait until the migration is merged?

[kvch]

Please wait. My PR is gigantic already.

[gonzaloe]

@kvch could you ping me when your PR gets merged? I'll adapt my changes and rebase then

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b drop-fields-support-regex upstream/drop-fields-support-regex
git merge upstream/main
git push upstream drop-fields-support-regex

[rdner]

@gonzaloe thank you for waiting, now the big change is merged, please update your PR.

[gonzaloe]

thanks for the notification @rdner, seems i'll have to move the mapstr implementation to elastic-agent-libs project, then import a new version here once that code is merged, is there a label to pause the review for now? or should i ping back once the other PR gets merged and everything working here again?

[gonzaloe]

@rdner in case you can take a quick look there -> elastic/elastic-agent-libs#45

[rdner]

@gonzaloe it's approved, feel free to proceed.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b drop-fields-support-regex upstream/drop-fields-support-regex
git merge upstream/main
git push upstream drop-fields-support-regex

[gonzaloe]

@rdner thanks! I've updated the dependency already :)
one more doubt so far:

• about the bug I pointed out on the drop_field package, should I go ahead and fix it, or is that code not being used? (deletion is being done while traversing the array from left to right)

[rdner]

about the bug I pointed out on the drop_field package, should I go ahead and fix it, or is that code not being used? (deletion is being done while traversing the array from left to right)

@gonzaloe sorry I missed where you mentioned the bug. Regardless, I think we should make one change at a time. Let's finish with this PR first and address any other bug later.

After updating a dependency one must run make update from the root of the repository and it seems like there is one build error reported by the linter too.

[gonzaloe]

@rdner perfect about addressing the other bug, i'll create a separate PR after. I understand about the make update, now should be addressed that (and the change on dependency for the lint warning). Thanks

[rdner]

@gonzaloe I have to ask you to run goimport on libbeat/processors/actions/drop_fields.go and libbeat/processors/actions/drop_fields_test.go, otherwise the CI (libbeat-checks) would not pass. It runs goimport and checks that there is no diff afterwards.

You can also run this step locally with make check.

[gonzaloe]

@rdner sorry for the inconvenience, I had first the problem with elastic-agent-libs and ran the update, but then with the new config package, but forgot to run make update now 😅 . I believe things should be in order

[rdner]

/test

[rdner]

I see e2e tests passed here https://beats-ci.elastic.co/blue/organizations/jenkins/e2e-tests%2Fe2e-testing-mbp/detail/main/906/pipeline but it's not reported to Github for some reason. I think it's okay to merge it now.

[gonzaloe]

@rdner good morning, E2E Tests are configured to run in every branch once the previous steps are successful? or only run in main/version branches?

[rdner]

@gonzaloe the E2E tests passed (https://beats-ci.elastic.co/blue/organizations/jenkins/e2e-tests%2Fe2e-testing-mbp/detail/main/908/pipeline) I think something has been changed and they are not reported to Github correctly anymore. I'm going to approve the PR, we can merge it now.

[rdner]

LGTM,

@gonzaloe please open another PR with the documentation change.