Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Data Prepper is writting empty DLQ objects #3644

Closed
dlvenable opened this issue Nov 13, 2023 · 1 comment · Fixed by #3660
Closed

[BUG] Data Prepper is writting empty DLQ objects #3644

dlvenable opened this issue Nov 13, 2023 · 1 comment · Fixed by #3660
Assignees
Labels
bug Something isn't working
Milestone

Comments

@dlvenable
Copy link
Member

Describe the bug

The Data Prepper opensearch sink can write empty DLQ objects to S3 in certain case.

From what I can tell, this mostly occurs from high-level errors from the _bulk request which do not result in underlying items provided with failures.

To Reproduce
Steps to reproduce the behavior:

Configure an index that you do not have permission to write to.

Run Data Prepper and provide input.

I see the following errors:

2023-11-13T18:09:23,902 [dynamodb-pipeline-sink-worker-2-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.BulkRetryStrategy - Bulk Operation Failed. Number of retries 5. Retrying...
2023-11-13T18:09:23,906 [dynamodb-pipeline-sink-worker-2-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.BulkRetryStrategy - operation = Index, error = OpenSearch exception [type=authorization_exception, reason=User does not have permissions for the requested resource]
2023-11-13T18:09:23,909 [dynamodb-pipeline-sink-worker-2-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.BulkRetryStrategy - operation = Index, error = OpenSearch exception [type=authorization_exception, reason=User does not have permissions for the requested resource]
2023-11-13T18:09:50,577 [dynamodb-pipeline-sink-worker-2-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.BulkRetryStrategy - Bulk Operation Failed. Number of retries 10. Retrying...
2023-11-13T18:09:50,579 [dynamodb-pipeline-sink-worker-2-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.BulkRetryStrategy - operation = Index, error = OpenSearch exception [type=authorization_exception, reason=User does not have permissions for the requested resource]
2023-11-13T18:09:50,579 [dynamodb-pipeline-sink-worker-2-thread-1] WARN  org.opensearch.dataprepper.plugins.sink.opensearch.BulkRetryStrategy - operation = Index, error = OpenSearch exception [type=authorization_exception, reason=User does not have permissions for the requested resource]

I also got 8 S3 objects in the DLQ. Each looked like:

{"dlqObjects":[]}

Expected behavior

The DLQ should include one item per failed document. It should include the high-level error instead of the individual item error.

Environment (please complete the following information):

Data Prepper main working toward 2.6.0.

Additional context

N/A

@dlvenable dlvenable added bug Something isn't working untriaged labels Nov 13, 2023
@dlvenable dlvenable self-assigned this Nov 13, 2023
@dlvenable dlvenable added this to the v2.6 milestone Nov 14, 2023
@dlvenable
Copy link
Member Author

I am getting this error with an Amazon OpenSearch Serverless collection. In particular, I can see it when the role I use has access to the collection, but not the necessary permissions to create an index.

dlvenable added a commit to dlvenable/data-prepper that referenced this issue Nov 14, 2023
…yable failure. Resolves opensearch-project#3644

Signed-off-by: David Venable <dlv@amazon.com>
dlvenable added a commit that referenced this issue Nov 15, 2023
…yable failure. Resolves #3644 (#3660)

Signed-off-by: David Venable <dlv@amazon.com>
@github-project-automation github-project-automation bot moved this from Unplanned to Done in Data Prepper Tracking Board Nov 15, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Archived in project
1 participant