[BUG] Unhelpful error message when failing to authenticate with Amazon OpenSearch Service

[chenqi0805]

What is the bug?

This is related to our opensearch sink use case with Amazon OpenSearch backend sts role authentication: opensearch-project/data-prepper#2655

How can one reproduce the bug?

When sts role is not configured with the right permissions, one gets

[security_exception] authentication/authorization failure

What is the expected behavior?

Ideally the cause message could be

1. more specific about the operation that fails authentication
2. print the sts role used in authentication

What is your host/environment?

all

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

Add any other context about the problem.

[dblock]

What's the actual HTTP response for this?

Looking at the code it does try to parse the response and pull out the complete message.

[chenqi0805]

@dblock Sorry for the late response. I also had a hard time to reproduce it. But it is with getClusterSettings:

org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [security_exception] authentication/authorization failure
	at org.opensearch.client.transport.aws.AwsSdk2Transport.parseResponse(AwsSdk2Transport.java:517) ~[opensearch-java-2.2.0.jar:?]
	at org.opensearch.client.transport.aws.AwsSdk2Transport.executeSync(AwsSdk2Transport.java:438) ~[opensearch-java-2.2.0.jar:?]
	at org.opensearch.client.transport.aws.AwsSdk2Transport.performRequest(AwsSdk2Transport.java:241) ~[opensearch-java-2.2.0.jar:?]
	at org.opensearch.client.opensearch.cluster.OpenSearchClusterClient.getSettings(OpenSearchClusterClient.java:282) ~[opensearch-java-2.2.0.jar:?]
	at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.checkISMEnabled(AbstractIndexManager.java:187) ~[opensearch-2.2.1.jar:?]
	at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.checkAndCreateIndexTemplate(AbstractIndexManager.java:208) ~[opensearch-2.2.1.jar:?]
	at org.opensearch.dataprepper.plugins.sink.opensearch.index.AbstractIndexManager.setupIndex(AbstractIndexManager.java:203) ~[opensearch-2.2.1.jar:?]
	at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.doInitializeInternal(OpenSearchSink.java:174) ~[opensearch-2.2.1.jar:?]
	at org.opensearch.dataprepper.plugins.sink.opensearch.OpenSearchSink.doInitialize(OpenSearchSink.java:139) ~[opensearch-2.2.1.jar:?]
	at org.opensearch.dataprepper.model.sink.SinkThread.run(SinkThread.java:23) ~[data-prepper-api-2.2.1.jar:?]
	at java.lang.Thread.run(Thread.java:829) [?:?]

but for some reason when I try to call the API with the wrong role, it does yield clear message:

Exception in thread "main" org.opensearch.client.opensearch._types.OpenSearchException: Request failed: [security_exception] User: arn:aws:sts::************:assumed-role/******/****** is not authorized to perform: es:ESHttpGet because no resource-based policy allows the es:ESHttpGet action
	at org.opensearch.client.transport.aws.AwsSdk2Transport.parseResponse(AwsSdk2Transport.java:517)
	at org.opensearch.client.transport.aws.AwsSdk2Transport.executeSync(AwsSdk2Transport.java:438)
	at org.opensearch.client.transport.aws.AwsSdk2Transport.performRequest(AwsSdk2Transport.java:241)
	at org.opensearch.client.opensearch.cluster.OpenSearchClusterClient.getSettings(OpenSearchClusterClient.java:282)

This seems to be a corner case failed to be covered. Is it possible that the fine-grained access control of Amazon OpenSearch might cause missing response message be overriden?

[dblock]

Yes, it's possible. First, try to do this with awscurl or curl, see the responses and how different they are?

[chenqi0805]

@dblock I was able to reproduce the issue with FGAC domain that the client does not have proper permission/role. It seems we should be parsing reason in this scenario instead of message or Message. Let me know if you have suggestions on the fix.