-
Notifications
You must be signed in to change notification settings - Fork 206
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] Data prepper cannot start if otel-v1-apm-span index exists #3342
Comments
I had a quick look into the OpenSearch sink. My understanding is, that when the alias is lost, the bulk operation will create an index with the next batch to be indexed in OpenSearch. The OpenSearch Bulk API [1] has the option to if (isEstimateBulkSizeUsingCompression && isRequestCompressionEnabled) {
final int maxLocalCompressionsForEstimation = openSearchSinkConfig.getIndexConfiguration().getMaxLocalCompressionsForEstimation();
bulkRequestSupplier = () -> new JavaClientAccumulatingCompressedBulkRequest(new BulkRequest.Builder(), bulkSize, maxLocalCompressionsForEstimation);
} else if (isEstimateBulkSizeUsingCompression) {
LOG.warn("Estimate bulk request size using compression was enabled but request compression is disabled. " +
"Estimating bulk request size without compression.");
bulkRequestSupplier = () -> new JavaClientAccumulatingUncompressedBulkRequest(new BulkRequest.Builder());
} else {
bulkRequestSupplier = () -> new JavaClientAccumulatingUncompressedBulkRequest(new BulkRequest.Builder());
}
} I suggest to make [1] https://opensearch.org/docs/1.2/opensearch/rest-api/document-apis/bulk/ |
User experience is quite ugly: Users would assume complete loss of data as Opensearch Dashboards observability plugin shows "Trace Analytics not set up" because otel-v1-apm-span does not match otel-v1-apm-span-* pattern (which otel-v1-apm-span-000001 would match). This is a huge blocker for us and the proposal by @KarstenSchnitter is looks quite nice: Would appreciate to get some feedback. |
@KarstenSchnitter , @juergen-walter , Thank you both for looking into this issue. If I understand the situation, it is:
I think the solution you proposed to use From an implementation perspective I think we'd want to connect this flag to the |
Describe the bug
Data Prepper pods cannot (re)start (specifically the opensearch plugin) when there is a index named like an alias that is managed by data prepper. This bug occurs when no index alias exists (for example, if all OTel span indices were deleted), ingestion to data prepper is still ongoing and then data prepper restarts.
To Reproduce
Steps to reproduce the behavior:
An index exists with the same name as the reserved index alias name [otel-v1-apm-span], please delete or migrate the existing index
Expected behavior
Data prepper should be able to start in situations where customers did not do anything obviously wrong.
One mitigation idea would be to automatically rename otel-v1-apm-span index to otel-v1-apm-span-000001 and add an otel-v1-apm-span alias pointing to the renamed index.
Stack trace
Environment:
The text was updated successfully, but these errors were encountered: