Add community_id ingest processor

[gaobinlong]

Description

This PR is to add a new ingest processor called community_id processor, which is used to generate hash value for network flow tuple, so that the network flow data can be tracked by the unique hash value. The processor supports TCP, UDP, SCTP, ICMP and IPv6-ICMP protocols. The algorithm of generating the hash value is defined in https://github.com/corelight/community-id-spec.

Here are the short description about the hash input for the SHA-1 algorithm:

1. TCP/UDP/SCTP
   source ip | destination ip | protocol | source port | dest port
   4 bytes(ipv4) or 16 bytes(ipv6) | 4 bytes(ipv4) or 16 bytes(ipv6) | 1 bytes | 2 bytes | 2 bytes

2. ICMP/IPv6-ICMP
   source ip | destination ip | protocol | type | code
   4 bytes(ipv4) or 16 bytes(ipv6) | 4 bytes(ipv4) or 16 bytes(ipv6) | 1 bytes | 2 bytes | 2 bytes

In order to make the generated hash value stable for the given tuple, some unit test and yml test cases check that whether the hash value generated by this processor equals to the known value or not.

Related Issues

#2787

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change a5cf254

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/performance-analyzer.git]

[github-actions]

❌ Gradle check result for 8d4148f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[gaobinlong]

Hi @reta @msfroh @nknize @andrross, could you help to review this PR? Any comments about it will be really appreciated.

[codecov]

Codecov Report

Attention: 26 lines in your changes are missing coverage. Please review.

Comparison is base (0d50525) 71.38% compared to head (a5cf254) 71.32%.
Report is 2 commits behind head on main.

Files	Patch %	Lines
...opensearch/ingest/common/CommunityIdProcessor.java	89.36%	10 Missing and 15 partials ⚠️
...search/ingest/common/IngestCommonModulePlugin.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #12121      +/-   ##
============================================
- Coverage     71.38%   71.32%   -0.06%     
- Complexity    59707    59719      +12     
============================================
  Files          4952     4953       +1     
  Lines        280639   280875     +236     
  Branches      40773    40816      +43     
============================================
  Hits         200341   200341              
- Misses        63629    63853     +224     
- Partials      16669    16681      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❕ Gradle check result for c035c72: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing
      1 org.opensearch.indices.replication.SegmentReplicationAllocationIT.testSingleIndexShardAllocation

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[github-actions]

❕ Gradle check result for a5cf254: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-12121-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 a0b519826b2dab19f022e16e613f630a0bce253e
# Push it to GitHub
git push --set-upstream origin backport/backport-12121-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-12121-to-2.x.

[reta]

@gaobinlong could you please backport to 2.x manually? thank you