Large Object support for OpenStack Swift #18955
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Until now, you wouldn't be able to create objects larger that 5GB.
It's worth noting that it doesn't improve efficiency. Here is how it work for default configuration:
Client divide file to chunks and upload each of those chunks as separate file
All chunks are pushed to object storage as they are received by Nextcloud
Client send MOVE request which assembly file and store it. In case of Swift (and probably other object storages too), it download all chunks, assembly it locally into one file, and then uploads again to Swift
You can improve efficiency by setting
cache_path
to for e.g./tmp/nextcloud_cache
, then file chunks wont be uploaded to Swift, but you will still have to wait when the assembled file is being uploaded to Swift. For large file it can lead to timeout on the client side.Setting
cache_path
may not be appropriate for multi-node installation, in that case you probably need to have sticky-session or place cache on shared storage.Edit: I see that issue has been already reported for e.g.: #6105, #7070