Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] RepositoryS3ClientYamlTestSuiteIT class failing #117596

Closed
elasticsearchmachine opened this issue Nov 26, 2024 · 2 comments · Fixed by #117675
Closed

[CI] RepositoryS3ClientYamlTestSuiteIT class failing #117596

elasticsearchmachine opened this issue Nov 26, 2024 · 2 comments · Fixed by #117675
Assignees
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs low-risk An open issue or test failure that is a low risk to future releases Team:Distributed Coordination Meta label for Distributed Coordination team >test-failure Triaged test failures from CI

Comments

@elasticsearchmachine
Copy link
Collaborator

elasticsearchmachine commented Nov 26, 2024

Build Scans:

Reproduction Line:

./gradlew ":modules:repository-s3:yamlRestTest" --tests "org.elasticsearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT.test {yaml=repository_s3/40_repository_ec2_credentials/Get a non existing snapshot}" -Dtests.seed=FAA9DF498EC1EED4 -Dtests.locale=ro -Dtests.timezone=Europe/Helsinki -Druntime.java=17 -Dtests.fips.enabled=true

Applicable branches:
8.x

Reproduces locally?:
N/A

Failure History:
See dashboard

Failure Message:

java.lang.AssertionError: Failure at [repository_s3/40_repository_ec2_credentials:7]: expected [2xx] status code but api [snapshot.create_repository] returned [500 Internal Server Error] [---
error:
  root_cause:
  - type: "repository_verification_exception"
    reason: "[repository_ec2] path [ec2_base_path] is not accessible on master node"
    stack_trace: "org.elasticsearch.repositories.RepositoryVerificationException:\
      \ [repository_ec2] path [ec2_base_path] is not accessible on master node\nCaused\
      \ by: java.io.IOException: Unable to upload object [ec2_base_path/tests-X8eE8K6AQYesIdw10Xk6vw/master.dat]\
      \ using a single upload\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:466)\n\
      \tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144)\n\
      \tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\
      \tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38)\n\
      \tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142)\n\
      \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124)\n\
      \tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304)\n\
      \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2205)\n\
      \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373)\n\
      \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37)\n\
      \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\
      \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\
      \tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\
      \tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\
      \tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: com.amazonaws.SdkClientException:\
      \ Unable to calculate a request signature: Unable to calculate a request signature:\
      \ Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\
      \tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:109)\n\
      \tat com.amazonaws.auth.AWS4Signer.newSigningKey(AWS4Signer.java:639)\n\tat\
      \ com.amazonaws.auth.AWS4Signer.deriveSigningKey(AWS4Signer.java:404)\n\tat\
      \ com.amazonaws.auth.AWS4Signer.sign(AWS4Signer.java:253)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1320)\n\
      \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\
      \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\
      \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\
      \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\
      \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\
      \tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\
      \tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\
      \tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\
      \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\
      \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\
      \tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)\n\
      \tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)\n\
      \tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)\n\
      \tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)\n\
      \tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:464)\n\
      \tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\
      \tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\
      \tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\
      \tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:464)\n\
      \t... 14 more\nCaused by: com.amazonaws.SdkClientException: Unable to calculate\
      \ a request signature: Key size for HMAC must be at least 112 bits in approved\
      \ mode: SHA-256/HMAC\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:132)\n\
      \tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:105)\n\
      \t... 37 more\nCaused by: org.bouncycastle.crypto.IllegalKeyException: Key size\
      \ for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown\
      \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown\
      \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown\
      \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown\
      \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.jcajce.provider.BaseHMac.engineInit(Unknown\
      \ Source)\n\tat java.base/javax.crypto.Mac.init(Mac.java:432)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:127)\n\
      \t... 38 more\n"
  type: "repository_verification_exception"
  reason: "[repository_ec2] path [ec2_base_path] is not accessible on master node"
  caused_by:
    type: "i_o_exception"
    reason: "Unable to upload object [ec2_base_path/tests-X8eE8K6AQYesIdw10Xk6vw/master.dat]\
      \ using a single upload"
    caused_by:
      type: "sdk_client_exception"
      reason: "Unable to calculate a request signature: Unable to calculate a request\
        \ signature: Key size for HMAC must be at least 112 bits in approved mode:\
        \ SHA-256/HMAC"
      caused_by:
        type: "sdk_client_exception"
        reason: "Unable to calculate a request signature: Key size for HMAC must be\
          \ at least 112 bits in approved mode: SHA-256/HMAC"
        caused_by:
          type: "illegal_key_exception"
          reason: "Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC"
          stack_trace: "org.bouncycastle.crypto.IllegalKeyException: Key size for\
            \ HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat\
            \ org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown\
            \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown\
            \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown\
            \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown\
            \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.jcajce.provider.BaseHMac.engineInit(Unknown\
            \ Source)\n\tat java.base/javax.crypto.Mac.init(Mac.java:432)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:127)\n\
            \tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:105)\n\
            \tat com.amazonaws.auth.AWS4Signer.newSigningKey(AWS4Signer.java:639)\n\
            \tat com.amazonaws.auth.AWS4Signer.deriveSigningKey(AWS4Signer.java:404)\n\
            \tat com.amazonaws.auth.AWS4Signer.sign(AWS4Signer.java:253)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1320)\n\
            \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\
            \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\
            \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\
            \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\
            \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\
            \tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\
            \tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\
            \tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\
            \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\
            \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\
            \tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)\n\
            \tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)\n\
            \tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)\n\
            \tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)\n\
            \tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:464)\n\
            \tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\
            \tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\
            \tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\
            \tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:464)\n\
            \tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144)\n\
            \tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\
            \tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38)\n\
            \tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142)\n\
            \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124)\n\
            \tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304)\n\
            \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2205)\n\
            \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373)\n\
            \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37)\n\
            \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\
            \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\
            \tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\
            \tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\
            \tat java.base/java.lang.Thread.run(Thread.java:833)\n"
        stack_trace: "com.amazonaws.SdkClientException: Unable to calculate a request\
          \ signature: Key size for HMAC must be at least 112 bits in approved mode:\
          \ SHA-256/HMAC\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:132)\n\
          \tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:105)\n\
          \tat com.amazonaws.auth.AWS4Signer.newSigningKey(AWS4Signer.java:639)\n\t\
          at com.amazonaws.auth.AWS4Signer.deriveSigningKey(AWS4Signer.java:404)\n\
          \tat com.amazonaws.auth.AWS4Signer.sign(AWS4Signer.java:253)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1320)\n\
          \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\
          \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\
          \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\
          \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\
          \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\
          \tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\
          \tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\
          \tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\
          \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\
          \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\
          \tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)\n\
          \tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)\n\
          \tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)\n\
          \tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)\n\
          \tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:464)\n\
          \tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\
          \tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\
          \tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\
          \tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:464)\n\
          \tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144)\n\
          \tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\
          \tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38)\n\
          \tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142)\n\
          \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124)\n\
          \tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304)\n\
          \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2205)\n\
          \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373)\n\
          \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37)\n\
          \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\
          \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\
          \tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\
          \tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\
          \tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: org.bouncycastle.crypto.IllegalKeyException:\
          \ Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\
          \tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown\
          \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown\
          \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown\
          \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown\
          \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.jcajce.provider.BaseHMac.engineInit(Unknown\
          \ Source)\n\tat java.base/javax.crypto.Mac.init(Mac.java:432)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:127)\n\
          \t... 38 more\n"
      stack_trace: "com.amazonaws.SdkClientException: Unable to calculate a request\
        \ signature: Unable to calculate a request signature: Key size for HMAC must\
        \ be at least 112 bits in approved mode: SHA-256/HMAC\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:109)\n\
        \tat com.amazonaws.auth.AWS4Signer.newSigningKey(AWS4Signer.java:639)\n\t\
        at com.amazonaws.auth.AWS4Signer.deriveSigningKey(AWS4Signer.java:404)\n\t\
        at com.amazonaws.auth.AWS4Signer.sign(AWS4Signer.java:253)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1320)\n\
        \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\
        \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\
        \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\
        \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\
        \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\
        \tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\
        \tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\
        \tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\
        \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\
        \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\
        \tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)\n\
        \tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)\n\
        \tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)\n\
        \tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)\n\
        \tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:464)\n\
        \tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\
        \tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\
        \tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\
        \tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:464)\n\
        \tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144)\n\
        \tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\
        \tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38)\n\
        \tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142)\n\
        \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124)\n\
        \tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304)\n\
        \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2205)\n\
        \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373)\n\
        \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37)\n\
        \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\
        \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\
        \tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\
        \tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\
        \tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: com.amazonaws.SdkClientException:\
        \ Unable to calculate a request signature: Key size for HMAC must be at least\
        \ 112 bits in approved mode: SHA-256/HMAC\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:132)\n\
        \tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:105)\n\
        \t... 37 more\nCaused by: org.bouncycastle.crypto.IllegalKeyException: Key\
        \ size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\
        \tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown\
        \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown\
        \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown\
        \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown\
        \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.jcajce.provider.BaseHMac.engineInit(Unknown\
        \ Source)\n\tat java.base/javax.crypto.Mac.init(Mac.java:432)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:127)\n\
        \t... 38 more\n"
    stack_trace: "java.io.IOException: Unable to upload object [ec2_base_path/tests-X8eE8K6AQYesIdw10Xk6vw/master.dat]\
      \ using a single upload\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:466)\n\
      \tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144)\n\
      \tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\
      \tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38)\n\
      \tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142)\n\
      \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124)\n\
      \tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304)\n\
      \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2205)\n\
      \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373)\n\
      \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37)\n\
      \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\
      \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\
      \tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\
      \tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\
      \tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: com.amazonaws.SdkClientException:\
      \ Unable to calculate a request signature: Unable to calculate a request signature:\
      \ Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\
      \tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:109)\n\
      \tat com.amazonaws.auth.AWS4Signer.newSigningKey(AWS4Signer.java:639)\n\tat\
      \ com.amazonaws.auth.AWS4Signer.deriveSigningKey(AWS4Signer.java:404)\n\tat\
      \ com.amazonaws.auth.AWS4Signer.sign(AWS4Signer.java:253)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1320)\n\
      \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\
      \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\
      \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\
      \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\
      \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\
      \tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\
      \tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\
      \tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\
      \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\
      \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\
      \tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)\n\
      \tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)\n\
      \tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)\n\
      \tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)\n\
      \tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:464)\n\
      \tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\
      \tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\
      \tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\
      \tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:464)\n\
      \t... 14 more\nCaused by: com.amazonaws.SdkClientException: Unable to calculate\
      \ a request signature: Key size for HMAC must be at least 112 bits in approved\
      \ mode: SHA-256/HMAC\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:132)\n\
      \tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:105)\n\
      \t... 37 more\nCaused by: org.bouncycastle.crypto.IllegalKeyException: Key size\
      \ for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown\
      \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown\
      \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown\
      \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown\
      \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.jcajce.provider.BaseHMac.engineInit(Unknown\
      \ Source)\n\tat java.base/javax.crypto.Mac.init(Mac.java:432)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:127)\n\
      \t... 38 more\n"
  stack_trace: "org.elasticsearch.repositories.RepositoryVerificationException: [repository_ec2]\
    \ path [ec2_base_path] is not accessible on master node\nCaused by: java.io.IOException:\
    \ Unable to upload object [ec2_base_path/tests-X8eE8K6AQYesIdw10Xk6vw/master.dat]\
    \ using a single upload\n\tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:466)\n\
    \tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$writeBlob$1(S3BlobContainer.java:144)\n\
    \tat java.base/java.security.AccessController.doPrivileged(AccessController.java:569)\n\
    \tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedIOException(SocketAccess.java:38)\n\
    \tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlob(S3BlobContainer.java:142)\n\
    \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.blobstore.BlobContainer.writeBlob(BlobContainer.java:124)\n\
    \tat org.elasticsearch.repositories.s3.S3BlobContainer.writeBlobAtomic(S3BlobContainer.java:304)\n\
    \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.blobstore.BlobStoreRepository.startVerification(BlobStoreRepository.java:2205)\n\
    \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.repositories.RepositoriesService.lambda$validatePutRepositoryRequest$11(RepositoriesService.java:373)\n\
    \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:37)\n\
    \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1023)\n\
    \tat org.elasticsearch.server@8.18.0-SNAPSHOT/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)\n\
    \tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\
    \tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\
    \tat java.base/java.lang.Thread.run(Thread.java:833)\nCaused by: com.amazonaws.SdkClientException:\
    \ Unable to calculate a request signature: Unable to calculate a request signature:\
    \ Key size for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\
    \tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:109)\n\t\
    at com.amazonaws.auth.AWS4Signer.newSigningKey(AWS4Signer.java:639)\n\tat com.amazonaws.auth.AWS4Signer.deriveSigningKey(AWS4Signer.java:404)\n\
    \tat com.amazonaws.auth.AWS4Signer.sign(AWS4Signer.java:253)\n\tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1320)\n\
    \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)\n\
    \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)\n\
    \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)\n\
    \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)\n\
    \tat com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)\n\
    \tat com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)\n\
    \tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)\n\t\
    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)\n\t\
    at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)\n\
    \tat com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)\n\
    \tat com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)\n\
    \tat com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)\n\
    \tat com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)\n\
    \tat com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)\n\
    \tat org.elasticsearch.repositories.s3.S3BlobContainer.lambda$executeSingleUpload$16(S3BlobContainer.java:464)\n\
    \tat org.elasticsearch.repositories.s3.SocketAccess.lambda$doPrivilegedVoid$0(SocketAccess.java:47)\n\
    \tat java.base/java.security.AccessController.doPrivileged(AccessController.java:318)\n\
    \tat org.elasticsearch.repositories.s3.SocketAccess.doPrivilegedVoid(SocketAccess.java:46)\n\
    \tat org.elasticsearch.repositories.s3.S3BlobContainer.executeSingleUpload(S3BlobContainer.java:464)\n\
    \t... 14 more\nCaused by: com.amazonaws.SdkClientException: Unable to calculate\
    \ a request signature: Key size for HMAC must be at least 112 bits in approved\
    \ mode: SHA-256/HMAC\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:132)\n\
    \tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:105)\n\t\
    ... 37 more\nCaused by: org.bouncycastle.crypto.IllegalKeyException: Key size\
    \ for HMAC must be at least 112 bits in approved mode: SHA-256/HMAC\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown\
    \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsSHS$MACOperatorFactory.createMAC(Unknown\
    \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown\
    \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.crypto.fips.FipsMACOperatorFactory.createOutputMACCalculator(Unknown\
    \ Source)\n\tat org.bouncycastle.fips.core/org.bouncycastle.jcajce.provider.BaseHMac.engineInit(Unknown\
    \ Source)\n\tat java.base/javax.crypto.Mac.init(Mac.java:432)\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:127)\n\
    \t... 38 more\n"
status: 500
]

Issue Reasons:

  • [8.x] 2 consecutive failures in step openjdk17_checkpart1_java-fips-matrix
  • [8.x] 6 failures in class org.elasticsearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT (1.5% fail rate in 413 executions)
  • [8.x] 6 failures in step openjdk17_checkpart1_java-fips-matrix (75.0% fail rate in 8 executions)
  • [8.x] 6 failures in pipeline elasticsearch-periodic (54.5% fail rate in 11 executions)

Note:
This issue was created using new test triage automation. Please report issues or feedback to es-delivery.

@elasticsearchmachine elasticsearchmachine added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs >test-failure Triaged test failures from CI needs:risk Requires assignment of a risk label (low, medium, blocker) Team:Distributed Coordination Meta label for Distributed Coordination team labels Nov 26, 2024
@elasticsearchmachine
Copy link
Collaborator Author

Pinging @elastic/es-distributed-coordination (Team:Distributed Coordination)

@elasticsearchmachine
Copy link
Collaborator Author

This has been muted on branch main

Mute Reasons:

  • [main] 4 failures in class org.elasticsearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT (1.6% fail rate in 243 executions)
  • [main] 2 failures in step openjdk21_checkpart1_java-fips-matrix (40.0% fail rate in 5 executions)
  • [main] 2 failures in step part-1-fips (50.0% fail rate in 4 executions)
  • [main] 2 failures in pipeline elasticsearch-periodic (40.0% fail rate in 5 executions)
  • [main] 2 failures in pipeline elasticsearch-pull-request (1.9% fail rate in 103 executions)

Build Scans:

elasticsearchmachine added a commit that referenced this issue Nov 27, 2024
…eIT org.elasticsearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT #117596
cbuescher pushed a commit to cbuescher/elasticsearch that referenced this issue Nov 27, 2024
…eIT org.elasticsearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT elastic#117596
@ywangd ywangd added low-risk An open issue or test failure that is a low risk to future releases and removed needs:risk Requires assignment of a risk label (low, medium, blocker) labels Nov 28, 2024
ywangd added a commit to ywangd/elasticsearch that referenced this issue Nov 28, 2024
Running with FIPS approved mode requires secret keys to be at least
114 bits long.

Resolves: elastic#117596
@ywangd ywangd self-assigned this Nov 28, 2024
alexey-ivanov-es pushed a commit to alexey-ivanov-es/elasticsearch that referenced this issue Nov 28, 2024
…eIT org.elasticsearch.repositories.s3.RepositoryS3ClientYamlTestSuiteIT elastic#117596
ywangd added a commit to ywangd/elasticsearch that referenced this issue Nov 29, 2024
Running with FIPS approved mode requires secret keys to be at least 114
bits long.

Relates: elastic#117324 Resolves: elastic#117596 Resolves: elastic#117709 Resolves: elastic#117710
Resolves: elastic#117711 Resolves: elastic#117712
(cherry picked from commit 24bc505)

# Conflicts:
#	modules/repository-s3/src/javaRestTest/java/org/elasticsearch/repositories/s3/RepositoryS3RestReloadCredentialsIT.java
#	muted-tests.yml
#	test/fixtures/s3-fixture/src/main/java/fixture/s3/S3HttpFixtureWithSTS.java
elasticsearchmachine pushed a commit that referenced this issue Nov 29, 2024
Running with FIPS approved mode requires secret keys to be at least 114
bits long.

Relates: #117324 Resolves: #117596 Resolves: #117709 Resolves: #117710
Resolves: #117711 Resolves: #117712
(cherry picked from commit 24bc505)

# Conflicts:
#	modules/repository-s3/src/javaRestTest/java/org/elasticsearch/repositories/s3/RepositoryS3RestReloadCredentialsIT.java
#	muted-tests.yml
#	test/fixtures/s3-fixture/src/main/java/fixture/s3/S3HttpFixtureWithSTS.java
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs low-risk An open issue or test failure that is a low risk to future releases Team:Distributed Coordination Meta label for Distributed Coordination team >test-failure Triaged test failures from CI
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants