-
-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use MultipartUpload for uploading chunks to s3 #27034
Conversation
108d4e6
to
b1690b2
Compare
b1690b2
to
6cf9a74
Compare
6cf9a74
to
ea62004
Compare
ea62004
to
e69b0b9
Compare
The change was needed to avoid doubling of upload time as the original solution does a complete re-read of the S3 object on final MOVE of NextCloud chunked upload. See nextcloud#27034 for details. Signed-off-by: Bernd.Rederlechner@t-systems.com <bernd.rederlechner@t-systems.com>
89e2e83
to
31a3a8e
Compare
@nextcloud/server-backend I didn't get that far in finishing this as expected, but some intermediate feedback on the approach with the storage abstraction and the chunking plugin would already be much appreciated. |
31a3a8e
to
68e2a44
Compare
The change was needed to avoid doubling of upload time as the original solution does a complete re-read of the S3 object on final MOVE of NextCloud chunked upload. See nextcloud#27034 for details. Signed-off-by: Bernd.Rederlechner@t-systems.com <bernd.rederlechner@t-systems.com>
The change was needed to avoid doubling of upload time as the original solution does a complete re-read of the S3 object on final MOVE of NextCloud chunked upload. See nextcloud#27034 for details. Signed-off-by: Bernd.Rederlechner@t-systems.com <bernd.rederlechner@t-systems.com>
The change was needed to avoid doubling of upload time as the original solution does a complete re-read of the S3 object on final MOVE of NextCloud chunked upload. See nextcloud#27034 for details. Signed-off-by: Bernd.Rederlechner@t-systems.com <bernd.rederlechner@t-systems.com>
0610055
to
6a1abcd
Compare
/backport to stable26 |
🚀 This is amazing :) Thanks! (hope it is ok to "pollute" github PR with praising :) it is easy to complain about bugs, but we should also learn to celebrate :) ) |
The backport to stable-23 failed. Please do this backport manually. |
The backport to stable26 failed. Please do this backport manually. |
@juliushaertl great work, thank you very much! |
The backport to stable-22 failed. Please do this backport manually. |
The backport to stable22 failed. Please do this backport manually. |
The backport to stable23 failed. Please do this backport manually. |
The backport to stable24 failed. Please do this backport manually. |
Thanks, good catch indeed. I'll take care of that in the follow up. |
Awesome feature :) ! Thanks a lot Are there any reasons for it not being backported to v25 ? |
We usually only backport security fixes or critical bugfixes. Any kind of performance improvement as well as features gets pinned to the upcoming major release. |
Implements #19414
This allows chunked upload streaming the chunks directly to S3 so that the final MOVE request no longer needs to assemble the final file on the Nextcloud server.
Ref: https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html
Current API changes to trigger the new upload mechanism (required changes on the client side)
The DAV capabilities expose a feature flag
s3-multipart
if available.Additional requirements on top of the current chunking API in order to trigger it:
Destination: http://nextcloud.dev.local/remote.php/dav/files/admin/myfiletarget
Summary of how S3 handles MultipartUpload
How it currently works
Test script
Bash script to test the upload