Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] RepositoryS3EcsClientYamlTestSuiteIT test {yaml=repository_s3/50_repository_ecs_credentials/Restore a non existing snapshot} failing #117711

Closed
elasticsearchmachine opened this issue Nov 28, 2024 · 2 comments · Fixed by #117675
Closed

[CI] RepositoryS3EcsClientYamlTestSuiteIT test {yaml=repository_s3/50_repository_ecs_credentials/Restore a non existing snapshot} failing #117711

elasticsearchmachine opened this issue Nov 28, 2024 · 2 comments · Fixed by #117675
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs low-risk An open issue or test failure that is a low risk to future releases Team:Distributed Coordination Meta label for Distributed Coordination team >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

Build Scans:

Reproduction Line:

./gradlew ":modules:repository-s3:yamlRestTest" --tests "org.elasticsearch.repositories.s3.RepositoryS3EcsClientYamlTestSuiteIT.test {yaml=repository_s3/50_repository_ecs_credentials/Delete a non existing snapshot}" -Dtests.seed=100082FF2DF14682 -Dtests.locale=de-AT -Dtests.timezone=America/Kentucky/Louisville -Druntime.java=17 -Dtests.fips.enabled=true

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure at [repository_s3/50_repository_ecs_credentials:7]: expected [2xx] status code but api [snapshot.create_repository] returned [500 Internal Server Error] [{"error":{"root_cause":[{"type":"repository_verification_exception","reason":"[repository_ecs] path [ecs_base_path] is not accessible on master node","stack_trace":"org.elasticsearch.repositories.RepositoryVerificationException: [repository_ecs] path [ecs_base_path] is not accessible on master node\nCaused by: java.io.IOException: Unable to upload object [ecs_base_path/tests-cj1XVnNaSW2HOV2u3TBfdA/master.dat] using a single upload\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:466)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2205)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: com.amazonaws.SdkClientException: Unable to calculate a request signature: Unable to calculate a request signature: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:109)\n\tat com.amazonaws.auth.AWS4Signer.newSigningKey(AWS4Signer.java:639)\n\tat com.amazonaws.auth.AWS4Signer.deriveSigningKey(AWS4Signer.java:404)\n\tat com.amazonaws.auth.AWS4Signer.sign(AWS4Signer.java:253)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1320)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)\n\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)\n\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)\n\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:464)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:464)\n\t... 14 more\nCaused by: com.amazonaws.SdkClientException: Unable to calculate a request signature: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:132)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:105)\n\t... 37 more\nCaused by: org.bouncycastle.crypto.IllegalKeyException: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.jcajce.provider.BaseHMac.engineInit(Unknown Source)\n\tat java.base/javax.crypto.Mac.init(Mac.java:432)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:127)\n\t... 38 more\n"}],"type":"repository_verification_exception","reason":"[repository_ecs] path [ecs_base_path] is not accessible on master node","caused_by":{"type":"i_o_exception","reason":"Unable to upload object [ecs_base_path/tests-cj1XVnNaSW2HOV2u3TBfdA/master.dat] using a single upload","caused_by":{"type":"sdk_client_exception","reason":"Unable to calculate a request signature: Unable to calculate a request signature: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC","caused_by":{"type":"sdk_client_exception","reason":"Unable to calculate a request signature: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC","caused_by":{"type":"illegal_key_exception","reason":"Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC","stack_trace":"org.bouncycastle.crypto.IllegalKeyException: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.jcajce.provider.BaseHMac.engineInit(Unknown Source)\n\tat java.base/javax.crypto.Mac.init(Mac.java:432)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:127)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:105)\n\tat com.amazonaws.auth.AWS4Signer.newSigningKey(AWS4Signer.java:639)\n\tat com.amazonaws.auth.AWS4Signer.deriveSigningKey(AWS4Signer.java:404)\n\tat com.amazonaws.auth.AWS4Signer.sign(AWS4Signer.java:253)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1320)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)\n\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)\n\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)\n\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:464)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:464)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2205)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\n"},"stack_trace":"com.amazonaws.SdkClientException: Unable to calculate a request signature: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:132)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:105)\n\tat com.amazonaws.auth.AWS4Signer.newSigningKey(AWS4Signer.java:639)\n\tat com.amazonaws.auth.AWS4Signer.deriveSigningKey(AWS4Signer.java:404)\n\tat com.amazonaws.auth.AWS4Signer.sign(AWS4Signer.java:253)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1320)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)\n\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)\n\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)\n\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:464)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:464)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2205)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.bouncycastle.crypto.IllegalKeyException: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.jcajce.provider.BaseHMac.engineInit(Unknown Source)\n\tat java.base/javax.crypto.Mac.init(Mac.java:432)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:127)\n\t... 38 more\n"},"stack_trace":"com.amazonaws.SdkClientException: Unable to calculate a request signature: Unable to calculate a request signature: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:109)\n\tat com.amazonaws.auth.AWS4Signer.newSigningKey(AWS4Signer.java:639)\n\tat com.amazonaws.auth.AWS4Signer.deriveSigningKey(AWS4Signer.java:404)\n\tat com.amazonaws.auth.AWS4Signer.sign(AWS4Signer.java:253)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1320)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)\n\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)\n\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)\n\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:464)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:464)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2205)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: com.amazonaws.SdkClientException: Unable to calculate a request signature: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:132)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:105)\n\t... 37 more\nCaused by: org.bouncycastle.crypto.IllegalKeyException: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.jcajce.provider.BaseHMac.engineInit(Unknown Source)\n\tat java.base/javax.crypto.Mac.init(Mac.java:432)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:127)\n\t... 38 more\n"},"stack_trace":"java.io.IOException: Unable to upload object [ecs_base_path/tests-cj1XVnNaSW2HOV2u3TBfdA/master.dat] using a single upload\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:466)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2205)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: com.amazonaws.SdkClientException: Unable to calculate a request signature: Unable to calculate a request signature: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:109)\n\tat com.amazonaws.auth.AWS4Signer.newSigningKey(AWS4Signer.java:639)\n\tat com.amazonaws.auth.AWS4Signer.deriveSigningKey(AWS4Signer.java:404)\n\tat com.amazonaws.auth.AWS4Signer.sign(AWS4Signer.java:253)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1320)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)\n\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)\n\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)\n\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:464)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:464)\n\t... 14 more\nCaused by: com.amazonaws.SdkClientException: Unable to calculate a request signature: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:132)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:105)\n\t... 37 more\nCaused by: org.bouncycastle.crypto.IllegalKeyException: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.jcajce.provider.BaseHMac.engineInit(Unknown Source)\n\tat java.base/javax.crypto.Mac.init(Mac.java:432)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:127)\n\t... 38 more\n"},"stack_trace":"org.elasticsearch.repositories.RepositoryVerificationException: [repository_ecs] path [ecs_base_path] is not accessible on master node\nCaused by: java.io.IOException: Unable to upload object [ecs_base_path/tests-cj1XVnNaSW2HOV2u3TBfdA/master.dat] using a single upload\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:466)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2205)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: com.amazonaws.SdkClientException: Unable to calculate a request signature: Unable to calculate a request signature: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:109)\n\tat com.amazonaws.auth.AWS4Signer.newSigningKey(AWS4Signer.java:639)\n\tat com.amazonaws.auth.AWS4Signer.deriveSigningKey(AWS4Signer.java:404)\n\tat com.amazonaws.auth.AWS4Signer.sign(AWS4Signer.java:253)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1320)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)\n\tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)\n\tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)\n\tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:464)\n\tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:464)\n\t... 14 more\nCaused by: com.amazonaws.SdkClientException: Unable to calculate a request signature: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:132)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:105)\n\t... 37 more\nCaused by: org.bouncycastle.crypto.IllegalKeyException: Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.jcajce.provider.BaseHMac.engineInit(Unknown Source)\n\tat java.base/javax.crypto.Mac.init(Mac.java:432)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:127)\n\t... 38 more\n"},"status":500}]

Issue Reasons:

  • [8.x] 2 consecutive failures in step openjdk17_checkpart1_java-fips-matrix
  • [8.x] 5 failures in test test {yaml=repository_s3/50_repository_ecs_credentials/Restore a non existing snapshot} (1.3% fail rate in 373 executions)
  • [8.x] 5 failures in step openjdk17_checkpart1_java-fips-matrix (71.4% fail rate in 7 executions)
  • [8.x] 5 failures in pipeline elasticsearch-periodic (50.0% fail rate in 10 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.
@elasticsearchmachine elasticsearchmachine added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI needs:risk Requires assignment of a risk label (low, medium, blocker) Team:Distributed Coordination Meta label for Distributed Coordination team labels Nov 28, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

@DaveCTurner
Copy link
Contributor

Just a test bug, see #117675 for the fix.

@DaveCTurner DaveCTurner added low-risk An open issue or test failure that is a low risk to future releases and removed needs:risk Requires assignment of a risk label (low, medium, blocker) labels Nov 28, 2024
@ywangd ywangd mentioned this issue Nov 28, 2024
Merged
ywangd added a commit to ywangd/elasticsearch that referenced this issue Nov 29, 2024
@ywangd
[Test] Increase test secret key length (elastic#117675)
1db39a7 
Running with FIPS approved mode requires secret keys to be at least 114
bits long.

Relates: elastic#117324 Resolves: elastic#117596 Resolves: elastic#117709 Resolves: elastic#117710
Resolves: elastic#117711 Resolves: elastic#117712
(cherry picked from commit 24bc505)

# Conflicts:
#	modules/repository-s3/src/javaRestTest/java/org/elasticsearch/repositories/s3/RepositoryS3RestReloadCredentialsIT.java
#	muted-tests.yml
#	test/fixtures/s3-fixture/src/main/java/fixture/s3/S3HttpFixtureWithSTS.java
elasticsearchmachine pushed a commit that referenced this issue Nov 29, 2024
@ywangd
[Test] Increase test secret key length (#117675) (#117738)
f52b8c6 
Running with FIPS approved mode requires secret keys to be at least 114
bits long.

Relates: #117324 Resolves: #117596 Resolves: #117709 Resolves: #117710
Resolves: #117711 Resolves: #117712
(cherry picked from commit 24bc505)

# Conflicts:
#	modules/repository-s3/src/javaRestTest/java/org/elasticsearch/repositories/s3/RepositoryS3RestReloadCredentialsIT.java
#	muted-tests.yml
#	test/fixtures/s3-fixture/src/main/java/fixture/s3/S3HttpFixtureWithSTS.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs low-risk An open issue or test failure that is a low risk to future releases Team:Distributed Coordination Meta label for Distributed Coordination team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Test] Increase test secret key length ywangd/elasticsearch
2 participants
@DaveCTurner @elasticsearchmachine