Parallel execution with "Use Jenkins source" results S3 upload error and strange error message

[ikedam]

Running awsCodeBuild with sourceControlType: 'jenkins' often results following error:

[AWS CodeBuild Plugin] Unexpected exception upon uploading source zip to S3: The request to the service failed with a retryable reason, but resetting the request input stream has failed. See exception.getExtraInfo or debug-level logging for the original failure that caused this retry.;  If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int)
[AWS CodeBuild Plugin] A versioned S3 bucket is required.

The message is strange as S3 versioning feature is enabled.
I found that this happens when multiple builds run at the same time and multiple awsCodeBuild for the same codebuild project run at the same time.

[ikedam]

Example pipeline to reproduce this issue:

node {
  // Code Build that does nothing at all.
  writeFile(
    file: 'buildspec.yaml',
    text: 'version: 0.2\nphases:\n  build:\n    commands:\n      - true\n',
  )

  // Create a large file.
  // A larger file makes it easier to reproduce the issue.
  sh 'dd if=/dev/urandom of=file.txt bs=1M count=5'
  sh 'rm -f file.zip && zip file.zip file.txt buildspec.yaml && ls -l'
  def file = sh(
    script: 'echo $(pwd)/file.zip',
    returnStdout: true,
  ).trim()

  // Run builds parallel to reproduce the confliction of S3 upload.
  // More parallel builds make it easier to reproduce the issue.
  def buildNum = 5
  def tasks = [:]
  buildNum.times { i ->
    def label = "task${i}"
    tasks[label] = {
      node {
        awsCodeBuild(
          projectName: 'codebuildtest',
          credentialsType: 'jenkins',
          credentialsId: 'codebuild-credentials',
          localSourcePath: file,
          region: 'us-west-2',
          sourceControlType: 'jenkins',
        )
      }
    }
  }
  parallel(tasks)
}

This build always results the issue in the description in my environment.
It might depends on the performance of your environment, and you might need to tune the output count of dd and buildNum.

[ikedam]

I believe this relates to https://stackoverflow.com/questions/63238344/amazon-s3-how-parallel-puts-to-the-same-key-are-resolved-in-versioned-buckets .
The root cause is the limitation (not rate limit!) of S3, and clients must retry the upload if the request fail with 5xx.

[ikedam]

It thows ResetExcetion:

com.amazonaws.ResetException: The request to the service failed with a retryable reason, but resetting the request input stream has failed. See exception.getExtraInfo or debug-level logging for the original failure that caused this retry.;  If the request involves an input stream, the maximum stream buffer size can be configured via request.getRequestClientOptions().setReadLimit(int)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.resetRequestInputStream(AmazonHttpClient.java:1511)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1284)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
	at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:421)
	at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6531)
	at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1861)
	at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1821)
	at S3DataManager.uploadSourceToS3(S3DataManager.java:91)
	at CodeBuilder.perform(CodeBuilder.java:429)
	at CodeBuilder.perform(CodeBuilder.java:264)
	at CodeBuildStep$CodeBuildExecution.run(CodeBuildStep.java:654)
	at CodeBuildStep$CodeBuildExecution.run(CodeBuildStep.java:612)
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
	at hudson.security.ACL.impersonate2(ACL.java:451)
	at hudson.security.ACL.impersonate(ACL.java:463)
	at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.io.IOException: Resetting to invalid mark
	at java.base/java.io.BufferedInputStream.reset(Unknown Source)
	at com.amazonaws.internal.SdkBufferedInputStream.reset(SdkBufferedInputStream.java:106)
	at com.amazonaws.internal.SdkFilterInputStream.reset(SdkFilterInputStream.java:120)
	at com.amazonaws.event.ProgressInputStream.reset(ProgressInputStream.java:168)
	at com.amazonaws.internal.SdkFilterInputStream.reset(SdkFilterInputStream.java:120)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.resetRequestInputStream(AmazonHttpClient.java:1509)
	... 29 more

It states "retryable", but isRetryable() is set to false: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/AmazonClientException.html#isRetryable--

[ikedam]

Called zipFileInputStream.mark(1024);, but that doesn't fix the issue.

[kb-fusus]

same issue here.