-
Notifications
You must be signed in to change notification settings - Fork 195
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
S3 / Swift object storage support #639
Comments
Nearly there ! Just cant find any explanations. |
Hello, |
Mhmm very interesting, however, why do not support multi buckets. It's well known that buckets with millions of objects start to get slow at a certain moment and that's why multi buckets kicks in. I think it would assure stability, nextcloud for example supports it in its enterprise edition. And after getting through testing it I really think using multiple buckets for a lot of users is better. |
Hello,
Can you give me more details on the issue you know having million of objects in a bucket ?
According to the AWS and Nextcloud documentation I only read about multibuckets as a way to avoid API limitations. On AWS S3 provider this limit is if 3500 write per seconds and that’s way enough for us up to millions users. The number of objects is also not marked as being a potential issue.
But as you said this is not a big work to do, so we can add this to our backlogs and also check with Linshare which is going to become our “drive” app soon.
https://aws.amazon.com/about-aws/whats-new/2018/07/amazon-s3-announces-increased-request-rate-performance/
https://autoize.ch/multi-bucket-primary-storage-for-nextcloud/
… On 8 Dec 2020, at 08:01, Poli ***@***.***> wrote:
Hello,
We do support Swift using what we call the OpenStack Adapters.
You should not need multi-bucket to have an horizontally scalable deployment, https://min.io/ for instance can give you S3 buckets API compatible with Twake.
You just need to put the S3 endpoint and credentials in Twake parameters.
The documentation on how to change default configuration is here: https://doc.twake.app/configuration
Mhmm very interesting, however, why do not support multi buckets. It's well known that buckets with millions of objects start to get slow at a certain moment and that's why multi buckets kicks in. I think it would assure stability, nextcloud for example supports it in its enterprise edition. And after getting through testing it I really think using multiple buckets for a lot of users is better.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
Hey, So basically I'll give you an example we have one shity bucket that has about 10 million objects on Swift. Each of the files of this bucket contains a JSON about who is where. Every time we need just to access one file it can take up to 1-2 seconds just to find in the index which file it is, and it gets on the client-side. So because of the fact that the index isn't stored on an SSD disk it takes a lot of time to retrieve data in the index. To be honest I don't know exactly how I could explain this but multibucketing has really a lot of advantages. |
@RomaricMourgues Any possibilities to implement that 😄 |
This is a good point, but I cannot add it to our roadmaps right now because the Twake Drive is going to be removed and replaced by Linshare :/ This not very easy for us because we need to do some load balancing on buckets and store the right bucket on database, we need to take little time to think about it! |
No worries, if needed take a look at how the multibucketing on nextcloud works (tip the version 9 of the manual shows how to do it) EDIT : Between isn't linshare like ultra old? EDIT 2 : Wikipedia says shit |
Up,
Could you please elaborate as documentation isn't clear at all. Thank you for your assistance. |
Twake looks like an really awesome alternative to nextcloud.
But I have a few issues, I would love to use it on a scalable structure (using some database nodes that are always up).
The ideal to do this usage would be to add object storage support using multiple buckets.
It isn't quite hard to setup, and to use. The objects location are all set in the database. And then you can simply retrieve them using bucket:objectid for example.
I don't think it would be hard to implement but it would be a true alternative to nextcloud.
The text was updated successfully, but these errors were encountered: