Fix simulate remove ingest processor throwing illegal_argument_exception #11607

gaobinlong · 2023-12-13T09:28:17Z

Description

This PR fixes the bug of simulating remove ingest processor always throwing illegal_argument_exception, the bug is caused by the new merged PR which disallowing removing some metadata fields, but when simulating processors, the ingesting document does not have the metadata fields like _version and _version_type, so document.getFieldValue(IngestDocument.Metadata.VERSION_TYPE.getFieldName(), String.class) throws illegal_argument_exception rather than return null value, in order to return null value, use another method instead with setting ignore_missing to true.

Both the unit test code and yml test do not cover this case, I've added some yml tests and unit test code for it.

Related Issues

#11604

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

gaobinlong · 2023-12-13T09:35:57Z

@reta , could you help to review this PR? Thanks!

github-actions · 2023-12-13T09:50:01Z

Compatibility status:

Checks if related components are compatible with change 3d29a28

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/performance-analyzer.git]

github-actions · 2023-12-13T10:22:08Z

✅ Gradle check result for 22d295c: SUCCESS

modules/ingest-common/src/test/java/org/opensearch/ingest/common/RemoveProcessorTests.java

github-actions · 2023-12-15T05:05:15Z

❕ Gradle check result for 598c74a: UNSTABLE

TEST FAILURES:

      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testSnapshotWithLargeSegmentFiles
      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-12-15T05:20:51Z

❕ Gradle check result for 7d90ad6: UNSTABLE

TEST FAILURES:

      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

github-actions · 2023-12-15T06:41:43Z

❕ Gradle check result for 5968814: UNSTABLE

TEST FAILURES:

      1 org.opensearch.repositories.gcs.GoogleCloudStorageBlobContainerRetriesTests.testWriteLargeBlob

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

github-actions · 2024-01-02T06:43:15Z

❕ Gradle check result for bb39c9b: UNSTABLE

TEST FAILURES:

      1 org.opensearch.smoketest.SmokeTestMultiNodeClientYamlTestSuiteIT.test {yaml=pit/10_basic/Delete all}
      1 org.opensearch.remotestore.RemoteIndexPrimaryRelocationIT.testPrimaryRelocationWhileIndexing

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2024-01-03T04:07:20Z

❌ Gradle check result for e707b70: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-01-04T08:44:15Z

✅ Gradle check result for 3d29a28: SUCCESS

gaobinlong · 2024-01-04T08:48:58Z

Hi @reta , can this PR be merged now? I've resolved the conflict and now all checks have passed yet.

…ion (#11607) * Fix simulate remove ingest processor throwing illegal_argument_exception Signed-off-by: Gao Binlong <gbinlong@amazon.com> * modify change log Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Create a new test mothod Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Use old method to get field value Signed-off-by: Gao Binlong <gbinlong@amazon.com> --------- Signed-off-by: Gao Binlong <gbinlong@amazon.com> (cherry picked from commit f647515) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ion (#11607) (#11745) * Fix simulate remove ingest processor throwing illegal_argument_exception * modify change log * Create a new test mothod * Use old method to get field value --------- (cherry picked from commit f647515) Signed-off-by: Gao Binlong <gbinlong@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ion (opensearch-project#11607) * Fix simulate remove ingest processor throwing illegal_argument_exception Signed-off-by: Gao Binlong <gbinlong@amazon.com> * modify change log Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Create a new test mothod Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Use old method to get field value Signed-off-by: Gao Binlong <gbinlong@amazon.com> --------- Signed-off-by: Gao Binlong <gbinlong@amazon.com>

…ion (opensearch-project#11607) * Fix simulate remove ingest processor throwing illegal_argument_exception Signed-off-by: Gao Binlong <gbinlong@amazon.com> * modify change log Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Create a new test mothod Signed-off-by: Gao Binlong <gbinlong@amazon.com> * Use old method to get field value Signed-off-by: Gao Binlong <gbinlong@amazon.com> --------- Signed-off-by: Gao Binlong <gbinlong@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

reta reviewed Dec 14, 2023

View reviewed changes

modules/ingest-common/src/test/java/org/opensearch/ingest/common/RemoveProcessorTests.java Show resolved Hide resolved

gaobinlong added 4 commits December 15, 2023 13:49

Fix simulate remove ingest processor throwing illegal_argument_exception

e56c8dd

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

modify change log

25f690f

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

Create a new test mothod

2ed08dd

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

Use old method to get field value

5968814

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

gaobinlong force-pushed the fix_remove branch from 7d90ad6 to 5968814 Compare December 15, 2023 05:50

reta approved these changes Dec 15, 2023

View reviewed changes

merge main

bb39c9b

Signed-off-by: Gao Binlong <gbinlong@amazon.com>

Merge remote-tracking branch 'upstream/main' into fix_remove

e707b70

Merge remote-tracking branch 'upstream/main' into fix_remove

3d29a28

reta self-requested a review January 4, 2024 13:46

reta approved these changes Jan 4, 2024

View reviewed changes

reta added bug Something isn't working v3.0.0 Issues and PRs related to version 3.0.0 v2.12.0 Issues and PRs related to version 2.12.0 labels Jan 4, 2024

reta merged commit f647515 into opensearch-project:main Jan 4, 2024
42 checks passed

reta added the backport 2.x Backport to 2.x branch label Jan 4, 2024

opensearch-trigger-bot bot mentioned this pull request Jan 4, 2024

[Backport 2.x] Fix simulate remove ingest processor throwing illegal_argument_exception #11745

Merged

BrewTestBot mentioned this pull request Feb 21, 2024

opensearch 2.12.0 Homebrew/homebrew-core#163463

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix simulate remove ingest processor throwing illegal_argument_exception #11607

Fix simulate remove ingest processor throwing illegal_argument_exception #11607

gaobinlong commented Dec 13, 2023 •

edited

Loading

gaobinlong commented Dec 13, 2023

github-actions bot commented Dec 13, 2023 •

edited

Loading

github-actions bot commented Dec 13, 2023

github-actions bot commented Dec 15, 2023

github-actions bot commented Dec 15, 2023

github-actions bot commented Dec 15, 2023

github-actions bot commented Jan 2, 2024

github-actions bot commented Jan 3, 2024

github-actions bot commented Jan 4, 2024

gaobinlong commented Jan 4, 2024

Fix simulate remove ingest processor throwing illegal_argument_exception #11607

Fix simulate remove ingest processor throwing illegal_argument_exception #11607

Conversation

gaobinlong commented Dec 13, 2023 • edited Loading

Description

Related Issues

Check List

gaobinlong commented Dec 13, 2023

github-actions bot commented Dec 13, 2023 • edited Loading

Compatibility status:

Incompatible components

Skipped components

Compatible components

github-actions bot commented Dec 13, 2023

github-actions bot commented Dec 15, 2023

github-actions bot commented Dec 15, 2023

github-actions bot commented Dec 15, 2023

github-actions bot commented Jan 2, 2024

github-actions bot commented Jan 3, 2024

github-actions bot commented Jan 4, 2024

gaobinlong commented Jan 4, 2024

gaobinlong commented Dec 13, 2023 •

edited

Loading

github-actions bot commented Dec 13, 2023 •

edited

Loading