Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large Object support for OpenStack Swift #18955

Merged
merged 1 commit into from
May 6, 2020
Merged

Large Object support for OpenStack Swift #18955

merged 1 commit into from
May 6, 2020

Conversation

adrb
Copy link

@adrb adrb commented Jan 17, 2020

Until now, you wouldn't be able to create objects larger that 5GB.

It's worth noting that it doesn't improve efficiency. Here is how it work for default configuration:

  1. Client divide file to chunks and upload each of those chunks as separate file

  2. All chunks are pushed to object storage as they are received by Nextcloud

  3. Client send MOVE request which assembly file and store it. In case of Swift (and probably other object storages too), it download all chunks, assembly it locally into one file, and then uploads again to Swift

You can improve efficiency by setting cache_path to for e.g. /tmp/nextcloud_cache, then file chunks wont be uploaded to Swift, but you will still have to wait when the assembled file is being uploaded to Swift. For large file it can lead to timeout on the client side.

Setting cache_path may not be appropriate for multi-node installation, in that case you probably need to have sticky-session or place cache on shared storage.

Edit: I see that issue has been already reported for e.g.: #6105, #7070

Until now, you wouldn't be able to create
objects larger that 5GB.

It's somewhat related with pull #18883

Signed-off-by: Adrian Brzezinski <adrian.brzezinski@eo.pl>
@jospoortvliet
Copy link
Member

ping @rullzer can this still get in 19?

@rullzer rullzer added this to the Nextcloud 19 milestone May 6, 2020
@rullzer rullzer added the 4. to release Ready to be released and/or waiting for tests to finish label May 6, 2020
@rullzer rullzer merged commit d5850eb into nextcloud:master May 6, 2020
@rullzer rullzer mentioned this pull request May 6, 2020
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
4. to release Ready to be released and/or waiting for tests to finish
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants