Uploads From MC Client Are Killed #3376
Comments
|
Check the exit code. When copying a lot of files, I am running mine in kubernetes so I needed to look at the node logs to get the error I get this when copying lots ( > many thousand ) files and my guess is that |
|
Mirroring 153 local files (6.5GB) to S3 does use over 2GB of RAM, getting it OOM-killed on a 2GB VM. I'm not familiar with the internals of Edit : This behavior is not exhibited during local-to-local copy (no S3 usage). It happens on both AWS S3 and Scaleway Object Storage remotes. On a less memory-limited machine mirroring the same payload reaches 6GB memory usage on 60-63 threads. |
|
Here is a mem profile top : https://github.com/minio/minio-go/blob/master/api-put-object-streaming.go#L152 seems incriminated. |
|
I am suffering from a similar issue. In Kubernetes, if I remove resource limits |
|
Will investigate this is a change that was necessary for |
This code is needed because there is no way to seek and read from multiple offsets in parallel on POSIX - *os.File is not locked. Introducing a mutex would simply serialize the entire thing. Using independent memory is the only way to upload content in parallel |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions. |
|
I've hit the same issue, tried to mirror large video files using Not sure how should I mirror(rsync?) my files to minio. Thinking of looping files in my own shell script, and then using streaming upload by piping the file to |
this workaround solved the memory issue for now. |
It also speed up my |
|
Thanks @furkanmustafa, it worked like a charm! |
|
@harshavardhana do you mind if we can rename title of this issue "Increased memory utilization for multi part upload" 🤗 or is there better issue to discuss this?
Thanks for the explanation! I have a couple of ideas. But before I am trying to understand the issue minio/minio-go#1355, what was race about? Because if it's about book-keeping of offset we can just open file multiple times or calculate offsets manually, WDYT? 🤗 |
The race is concurrent Seek's on *os.File are not possible and is not thread safe - leading to potentially incorrect content on the destination |
|
I understand that, you have essentially one "offset" pointer. But we don't need to use Seek, we could just implement our own offset checking or just... open same file 4 times. I think it's better than allocating 0,5GB for single upload 🤗 |
Correct but the API is not designed to take 4 different *os.File @bwplotka this is only possible if the entire path name is provided to minio-go client so that it can open as many For now setting a proper PartSize() to ensure that for most common use-cases you don't have to use 128MiB * 4 buffers locally @bwplotka |
|
Ah yea, we have only a reader.
I would love to avoid this, I don't think that makes sense. PartSize should control how I interact with HTTP protocol and minio target. It should not suddenly control how much is allocated. In the same way when I upload 6GB of file without multi-part I don't want 6GB of mem to be allocated, right? I think this is an major in-efficiency that we need to remove. You can already see how many issues and project were impacted. Let's see how we can improve this. I think we can rely on
If I am not wrong, we should be able to avoid using Seek just fine, no? We can just readAt offset we want into smaller fixed size buffer. Alternatively I would check how AWS SDK is doing it. Maybe something we can be inspired by? |
ReadAt we had other issues as well in the past @bwplotka - that's why we had re-implemented to be a much simpler implementation (predictable) although using more memory at the same time.
AWS SDK is also buffering as they all have to due to S3 API requirements.
Allocation is necessary no matter what @bwplotka because content-md5 calculation or sha256 needs to be calculated prior to uploading. If you re-seek your fd if you use *os.File and it is very tricky to manage these offsets cleanly in a safe way. Usage of ReadAt() makes it tricky because then you need to reflect upon an io.Seeker() to seek back anyways and that is not safe. We also need to handle transparent retries - this is why all of this becomes even more tricky. A transparent retry requires re-seek to the beginning of the required offset. |
|
You can re-seek virtually though within a buffer. Yes, looks like we have many things in place here (((: It escalated quickly to quite complex implementation, potentially for a good reason. Thanks for pointing out. This brings the question: What is the negative impact of reducing multi part size to e.g 16MB? 🤗 |
There is really no impact, in-fact I have a PR to change that as the default value for Users who have more leeway can use a higher buffer if needed. I suspect most of your files to be less than this value in terms of the full size of the object - or are you uploading with |
Thanks! Let's update default then if we can! 🤗
Yea, it depends if we can or cannot guess the size. Sometimes we cannot, so we use -1. GCS API is amazing because it deduces multipart or not from the reader directly - no need to pass full size. No idea how they do it 🙃 Magic. Probably prefetching to check. But our files can have a lot, sometimes around 50GB. Then question is, at what size we should start doing multipart... always? 32MB if we have 16MB Part size? 🤗 We can definitely experiment and benchmark, that would be the best |
NOTE: you need to remember that at max you can upload upto 10,000 parts in S3 multipart API so you should set the value such that it caters to the maximum object size. Internally in So essentially your best partSize is always So anyone uploading an object that is like 128GiB has to spend 128MiB worth of memory per multipart upload, as there is no way around it. This type of cacophony API is why sometimes we love GCS API where it takes an undefined upload to be uploaded without much memory requirement on the client-side. Then there are other ways such as you can write your own uploader using https://github.com/minio/minio-go/blob/master/core.go where you have full control over all the underlying details but that puts some burden on your end where you can fully optimize the memory requirement. |
|
Made some benchmarks and indeed it can be significant for our use case: thanos-io/thanos#3917 (comment) Is there a way for upload logic to accept some kind of reader that locks things? Or ability to tell minio that we don't use multiple
Thanks! This is useful, but I wonder if this is always good to set 10 000 parts. e.g for 1MB file, it sounds like inefficiency to send 10kb parts only, no? Since this is blocking Thanos release, I am considering reverting race fix as sounds like we are not affected and figure long term solution later ): bwplotka/minio-go@f9927e5 I still have high hopes for some Reader implementation that will lock e.g |
The minimum per part size is 5MiB @bwplotka only the last part can be < 5MiB
you can pass a ReadAt() which blocks uploads one by one or you change the concurrency default from '4' to '1' https://pkg.go.dev/github.com/minio/minio-go/v7#PutObjectOptions |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions. |
|
I think this issue still exists. Can we reopen? We have 4 workers (we don't change So either:
OR
|
From what I looked you are passing stream without the length - if we can't know the length we have to use a higher memory to cater to AWS S3 multipart requirement. If the size is provided we know what the optimal part size is, if you want to control that further, you can set whatever part size you want. |
|
We don't always know, but can double check!
Still, I would love to revisit if copying is necessary.
Kind Regards
Bartek Plotka
…On Thu, 16 Jun 2022, 18:47 Harshavardhana, ***@***.***> wrote:
2. The issue is unexisting and the current copying is not worth huge
performance regressions. Our tests in the worst case shows
<thanos-io/thanos#3968 (comment)> 1.5 MB of
mem allocated per upload vs 400 MB in the latest minio-go version and
almost twice slower operation without our revert.
From what I looked you are passing Content-Length without length if we
can't know the length we have to use a higher memory to cater to AWS S3
multipart requirement.
If the size is provided we know what the optimal part size is, if you want
to control that further, you can set whatever part size you want.
—
Reply to this email directly, view it on GitHub
<#3376 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABVA3OYDBSDSZWCZTS7U7XLVPNLC7ANCNFSM4QJUU6MQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
If you cannot know you can still set a partSize that you are okay with - you can't have an unending stream - maximum allowed is 5TiB which requires a 500MiB buffer per part to upload to satisfy 10,000 part upper limit. We need to calculate sha256sum, md5sum - we can't re-seek back on the stream either what else can the client do? other than buffering it? - If a Seeker or ReaderAt can be implemented then that is the best-case scenario for minio-go. We can't use Treating this as some kind of memory leak is naive, if you want to control the memory properly you have PartSize configurable - you should use that to control what you want to do with memory and how much leverage you have on the system where this is running. Also, this is simply an allocated memory, not an active memory - it is given back to runtime when minio-go is done with it - unfortunately Go's GC might not kick in that soon. The forked code you have @bwplotka is wrong - you are prone to races. |
|
However, to not bike shed this further, I will explore ways to reduce memory usage @bwplotka would you mind writing a sample code to reproduce this in https://github.com/minio/minio-go/issues |
Expected behavior
The client should upload the file
Actual behavior
The client stops the upload with a "Killed" output.
Here is the issue I created earlier: here
And here is the debug created when uploading the file:
`mc: GET /movies/?location= HTTP/1.1
Host: minio.cleverapps.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5 mc/2020-08-20T00:23:01Z
Authorization: AWS4-HMAC-SHA256 Credential=capriciousduck/20200824/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=REDACTED
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20200824T154923Z
Accept-Encoding: gzip
mc: HTTP/1.1 200 OK
Content-Length: 128
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Mon, 24 Aug 2020 15:49:23 GMT
Server: MinIO/RELEASE.2020-08-18T19-41-00Z
Sozu-Id: 1820f1e6-026a-427e-960e-110c66c9d1e7
Vary: Origin
X-Amz-Request-Id: 162E3EFA8A9FCE51
X-Xss-Protection: 1; mode=block
mc: Response Time: 141.98146ms
mc: HEAD /movies/ HTTP/1.1
Host: minio.cleverapps.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5 mc/2020-08-20T00:23:01Z
Authorization: AWS4-HMAC-SHA256 Credential=capriciousduck/20200824/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=REDACTED
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200824T154923Z
mc: HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Mon, 24 Aug 2020 15:49:23 GMT
Server: MinIO/RELEASE.2020-08-18T19-41-00Z
Sozu-Id: 1820f1e6-026a-427e-960e-110c66c9d1e7
Vary: Origin
X-Amz-Request-Id: 162E3EFA8AF3A52B
X-Xss-Protection: 1; mode=block
Content-Length: 0
mc: Response Time: 4.144945ms
mc: HEAD /movies/ HTTP/1.1
Host: minio.cleverapps.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5 mc/2020-08-20T00:23:01Z
Authorization: AWS4-HMAC-SHA256 Credential=capriciousduck/20200824/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=REDACTED
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200824T154923Z
mc: HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Mon, 24 Aug 2020 15:49:23 GMT
Server: MinIO/RELEASE.2020-08-18T19-41-00Z
Sozu-Id: b2b9cd51-0156-4c65-acc5-bfe5bc25715d
Vary: Origin
X-Amz-Request-Id: 162E3EFA8B41E64A
X-Xss-Protection: 1; mode=block
Content-Length: 0
mc: Response Time: 3.915626ms
0 B / ? ┃░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░▓┃mc: HEAD /movies/ HTTP/1.1
Host: minio.cleverapps.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5 mc/2020-08-20T00:23:01Z
Authorization: AWS4-HMAC-SHA256 Credential=capriciousduck/20200824/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=REDACTED
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200824T154923Z
mc: HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Date: Mon, 24 Aug 2020 15:49:23 GMT
Server: MinIO/RELEASE.2020-08-18T19-41-00Z
Sozu-Id: f3dfa6ec-df7a-45ef-bccf-681caca2f90a
Vary: Origin
X-Amz-Request-Id: 162E3EFA8BE7440F
X-Xss-Protection: 1; mode=block
Content-Length: 0
mc: Response Time: 4.594106ms
mc: GET /movies/?object-lock= HTTP/1.1
Host: minio.cleverapps.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5 mc/2020-08-20T00:23:01Z
Authorization: AWS4-HMAC-SHA256 Credential=capriciousduck/20200824/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=REDACTED
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20200824T154923Z
Accept-Encoding: gzip
mc: HTTP/1.1 404 Not Found
Content-Length: 330
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Mon, 24 Aug 2020 15:49:23 GMT
Server: MinIO/RELEASE.2020-08-18T19-41-00Z
Sozu-Id: 853f3437-e49f-4775-9e36-7e143ddf2e6c
Vary: Origin
X-Amz-Request-Id: 162E3EFA8C38CE7F
X-Xss-Protection: 1; mode=block
ObjectLockConfigurationNotFoundErrorObject Lock configuration does not exist for this bucketmovies/movies/162E3EFA8C38CE7F1f5c4a5e-43c3-416f-92c5-0fddc0b2c28amc: Response Time: 4.243974msmc: POST /movies/The.One.and.Only.Ivan.2020.1080p.DSNP.WEBRip.DDP5.1.x264-CM.mkv?uploads= HTTP/1.1
Host: minio.cleverapps.io
User-Agent: MinIO (linux; amd64) minio-go/v7.0.5 mc/2020-08-20T00:23:01Z
Content-Length: 0
Authorization: AWS4-HMAC-SHA256 Credential=capriciousduck/20200824/us-east-1/s3/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=REDACTED
Content-Type: video/x-matroska
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20200824T154923Z
Accept-Encoding: gzip
mc: HTTP/1.1 200 OK
Content-Length: 304
Accept-Ranges: bytes
Content-Security-Policy: block-all-mixed-content
Content-Type: application/xml
Date: Mon, 24 Aug 2020 15:49:23 GMT
Server: MinIO/RELEASE.2020-08-18T19-41-00Z
Sozu-Id: 0b9f5d17-7178-4470-907a-f060d45e50fc
Vary: Origin
X-Amz-Request-Id: 162E3EFA8C88E2B1
X-Xss-Protection: 1; mode=block
mc: Response Time: 4.883374ms`
The text was updated successfully, but these errors were encountered: