AWSCredentialsProviderChain should return Exceptions from providers, not just the combined message

[ivankhoosty]

Upcoming End-of-Support

• I acknowledge the upcoming end-of-support for AWS SDK for Java v1 was announced, and migration to AWS SDK for Java v2 is recommended.

Describe the bug

If an error occurs when fetching secrets from the secretsmanager during Spring startup, no logs are available as described in awspring/spring-cloud-aws#165 . The only information that is available to diagnose what went wrong with DefaultCredentialsProvider is the exception stacktrace/message generated on https://github.com/aws/aws-sdk-java/blob/df2ce8afe8f1c1ecebc110b9c451de4c904250fc/aws-java-sdk-core/src/main/java/com/amazonaws/auth/AWSCredentialsProviderChain.java#L142C38-L142C100

In my situation, ContainerCredentialsProvider provided message "Failed to load credentials from metadata service.", but the original (proxy/network/etc related) cause Exception was not visible in the logs, as

aws-sdk-java/aws-java-sdk-core/src/main/java/com/amazonaws/auth/AWSCredentialsProviderChain.java

Line 132 in df2ce8a

log.debug("Unable to load credentials from " + message);

doesn't actually produce any logs at this early stage.

The workaround was to replace DefaultCredentialsProvider with ContainerCredentialsProvider to see the cause Exception and fix the issue.

@maciejwalkowiak

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

I would want to see the exceptions that caused each provider to fail in the stacktrace of the

aws-sdk-java/aws-java-sdk-core/src/main/java/com/amazonaws/auth/AWSCredentialsProviderChain.java

Line 142 in df2ce8a

throw new SdkClientException("Unable to load AWS credentials from any provider in the chain: "

Current Behavior

only messages from the exceptions that caused each provider to fail are part of the exception generated on

aws-sdk-java/aws-java-sdk-core/src/main/java/com/amazonaws/auth/AWSCredentialsProviderChain.java

Line 142 in df2ce8a

throw new SdkClientException("Unable to load AWS credentials from any provider in the chain: "

Reproduction Steps

this is only reproducible if your app is fetching secrets via spring.config.import: aws-secretsmanager . Failures during this boostrap stage do not have any logs printed.

Possible Solution

Chain causes of failures of all providers and set as the cause for the exception thrown on line 142 ?

Additional Information/Context

No response

AWS Java SDK version used

2.28.28

JDK version used

21.0.1

Operating System and version

unix

[debora-ito]

Hi @ivankhoosty Java SDK 1.x is in Maintenance Mode, it'll only get security updates and critical bug fixes. We don't have plans to change the logging.

The workaround was to replace DefaultCredentialsProvider with ContainerCredentialsProvider to see the cause Exception and fix the issue.

This is exactly the instruction we give when troubleshooting credential issues: replace the default chain for the credential provider you expect to be used, and check the logs.

[ivankhoosty]

@debora-ito I have just noticed I pasted the links to sdk 1.x by mistake, I had both open to compare the code, and 2.x classes code is the same.

[debora-ito]

In 2.x if you set the log level to DEBUG you'll see the stacktraces of the individual credential provider calls in the chain:

<Logger name="software.amazon.awssdk" level="DEBUG" />

[ivankhoosty]

That’s not what happens because of the stage in spring boot loader. I will prepare a sample spring boot app to show you the issue

[maciejwalkowiak]

@ivankhoosty perhaps this issue should be moved to https://github.com/awspring/spring-cloud-aws project?

[ivankhoosty]

@maciejwalkowiak I dont know what you can do when Spring is not logging DeferredLog during boostrap failure.

I think AwsCredentialsProviderChain appending error messages from various credentialsProviders to use it as the SdkClientException message is not consistent, they should be chaining the actual exceptions into the cause of SdkClientException as well. @debora-ito makes a comment above "This is exactly the instruction we give when troubleshooting credential issues: replace the default chain for the credential provider you expect to be used, and check the logs." - this would not be necessary at all if exceptions from credentialsProviders were preserved and stored in SdkClientException

[debora-ito]

"... replace the default chain for the credential provider you expect to be used, and check the logs" is the instruction when using Java SDK 1.x.

In Java SDK 2.x you can see the full stacktraces when you use DEBUG level logging. Maybe this is a configuration that spring-cloud-aws needs to expose? We wouldn't like to change to print those at regular level as they are pretty verbose and will be printed with every client that uses the default chain.

[ivankhoosty]

@debora-ito the code in AWSCredentialsProviderChain and AwsCredentialsProviderChain resolveCredentials() methods is pretty much the same as to how it logs credentialProvider exceptions. I am not talking about changing logging level, that would make no difference to this issue.

I am interested in exceptions not being lost, so if ALL providers fail, we can still see them in the stacktrace:

public AwsCredentials resolveCredentials() {
if (reuseLastProviderEnabled && lastUsedProvider != null) {
return CredentialUtils.toCredentials(CompletableFutureUtils.joinLikeSync(lastUsedProvider.resolveIdentity()));
}

List<String> exceptionMessages = new ArrayList<>();
**List<Throwable> exceptions = new ArrayList<>();**

for (IdentityProvider<? extends AwsCredentialsIdentity> provider : credentialsProviders) {
    try {
        AwsCredentialsIdentity credentials = CompletableFutureUtils.joinLikeSync(provider.resolveIdentity());

        log.debug(() -> "Loading credentials from " + provider);

        lastUsedProvider = provider;
        return CredentialUtils.toCredentials(credentials);
    } catch (RuntimeException e) {
        // Ignore any exceptions and move onto the next provider
        String message = provider + ": " + e.getMessage();
        log.debug(() -> "Unable to load credentials from " + message, e);

        exceptionMessages.add(message);
        **exceptions.add(e);**
    }
}

SdkClientException sdkClientException = SdkClientException.builder()
    .message("Unable to load credentials from any of the providers in the chain " + this + " : " + exceptionMessages)
    .build();

//SdkClientException.builder doesn't support addSuppressed, so have to add directly to the exception
for (Throwable exception : exceptions) {
sdkClientException.addSuppressed(exception);
}

throw sdkClientException;

}

can be optimised to generate exceptionMessages from the exceptions list.