Skip to content

Commit

Permalink
Reduce memory usage for chunked artifact uploads to MinIO (#31325) (#…
Browse files Browse the repository at this point in the history
…31338)

Backport #31325 by @bohde

When using the MinIO storage driver for Actions Artifacts, we found that
the chunked artifact required significantly more memory usage to both
upload and merge than the local storage driver. This seems to be related
to hardcoding a value of `-1` for the size to the MinIO client [which
has a warning about memory usage in the respective
docs](https://pkg.go.dev/github.com/minio/minio-go/v7#Client.PutObject).
Specifying the size in both the upload and merge case reduces memory
usage of the MinIO client.

Co-authored-by: Rowan Bohde <rowan.bohde@gmail.com>
Co-authored-by: Kyle D <kdumontnu@gmail.com>
  • Loading branch information
3 people authored Jun 12, 2024
1 parent 758f84f commit b1ad8cc
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions routers/api/actions/artifacts_chunks.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ func saveUploadChunkBase(st storage.ObjectStorage, ctx *ArtifactContext,
r = io.TeeReader(r, hasher)
}
// save chunk to storage
writtenSize, err := st.Save(storagePath, r, -1)
writtenSize, err := st.Save(storagePath, r, contentSize)
if err != nil {
return -1, fmt.Errorf("save chunk to storage error: %v", err)
}
Expand Down Expand Up @@ -208,7 +208,7 @@ func mergeChunksForArtifact(ctx *ArtifactContext, chunks []*chunkFileItem, st st

// save merged file
storagePath := fmt.Sprintf("%d/%d/%d.%s", artifact.RunID%255, artifact.ID%255, time.Now().UnixNano(), extension)
written, err := st.Save(storagePath, mergedReader, -1)
written, err := st.Save(storagePath, mergedReader, artifact.FileCompressedSize)
if err != nil {
return fmt.Errorf("save merged file error: %v", err)
}
Expand Down

0 comments on commit b1ad8cc

Please sign in to comment.