Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

S3 / Swift object storage support #639

Open
IIPoliII opened this issue Dec 7, 2020 · 9 comments
Open

S3 / Swift object storage support #639

IIPoliII opened this issue Dec 7, 2020 · 9 comments

Comments

@IIPoliII
Copy link

IIPoliII commented Dec 7, 2020

Twake looks like an really awesome alternative to nextcloud.

But I have a few issues, I would love to use it on a scalable structure (using some database nodes that are always up).
The ideal to do this usage would be to add object storage support using multiple buckets.

It isn't quite hard to setup, and to use. The objects location are all set in the database. And then you can simply retrieve them using bucket:objectid for example.

I don't think it would be hard to implement but it would be a true alternative to nextcloud.

@IIPoliII
Copy link
Author

IIPoliII commented Dec 7, 2020

https://github.com/TwakeApp/Twake/blob/main/twake/backend/core/src/Twake/Upload/Services/AWS_Uploader.php

Nearly there ! Just cant find any explanations.
And multi buckets

@RomaricMourgues
Copy link
Contributor

Hello,
We do support Swift using what we call the OpenStack Adapters.
You should not need multi-bucket to have an horizontally scalable deployment, https://min.io/ for instance can give you S3 buckets API compatible with Twake.
You just need to put the S3 endpoint and credentials in Twake parameters.
The documentation on how to change default configuration is here: https://doc.twake.app/configuration

@IIPoliII
Copy link
Author

IIPoliII commented Dec 8, 2020

Hello,
We do support Swift using what we call the OpenStack Adapters.
You should not need multi-bucket to have an horizontally scalable deployment, https://min.io/ for instance can give you S3 buckets API compatible with Twake.
You just need to put the S3 endpoint and credentials in Twake parameters.
The documentation on how to change default configuration is here: https://doc.twake.app/configuration

Mhmm very interesting, however, why do not support multi buckets. It's well known that buckets with millions of objects start to get slow at a certain moment and that's why multi buckets kicks in. I think it would assure stability, nextcloud for example supports it in its enterprise edition. And after getting through testing it I really think using multiple buckets for a lot of users is better.

@RomaricMourgues
Copy link
Contributor

RomaricMourgues commented Dec 8, 2020 via email

@IIPoliII
Copy link
Author

IIPoliII commented Dec 8, 2020

Hey,
Thanks for your reply. The main issue with multibucketing, is when you don't have SSDs in for the indexes (as dumb as it sounds).

So basically I'll give you an example we have one shity bucket that has about 10 million objects on Swift. Each of the files of this bucket contains a JSON about who is where. Every time we need just to access one file it can take up to 1-2 seconds just to find in the index which file it is, and it gets on the client-side. So because of the fact that the index isn't stored on an SSD disk it takes a lot of time to retrieve data in the index.
This could be fixed if we used a relational database with multi buckets. It's not the fact that it needs to find the file in the database that takes time it's the fact that it needs to retrieve which file corresponds to which metadata.

To be honest I don't know exactly how I could explain this but multibucketing has really a lot of advantages.
nextcloud/server#1084

@IIPoliII
Copy link
Author

IIPoliII commented Dec 9, 2020

@RomaricMourgues Any possibilities to implement that 😄

@RomaricMourgues
Copy link
Contributor

This is a good point, but I cannot add it to our roadmaps right now because the Twake Drive is going to be removed and replaced by Linshare :/
What I can do is sharing your proposal with them.

This not very easy for us because we need to do some load balancing on buckets and store the right bucket on database, we need to take little time to think about it!
But anyway, I let this issue open and when we have more time I will go back on this :)

@IIPoliII
Copy link
Author

IIPoliII commented Dec 9, 2020

No worries, if needed take a look at how the multibucketing on nextcloud works (tip the version 9 of the manual shows how to do it)

EDIT : Between isn't linshare like ultra old?

EDIT 2 : Wikipedia says shit

@RamsesDeux
Copy link

Up,

Hello,
We do support Swift using what we call the OpenStack Adapters.
You should not need multi-bucket to have an horizontally scalable deployment, https://min.io/ for instance can give you S3 buckets API compatible with Twake.
You just need to put the S3 endpoint and credentials in Twake parameters.
The documentation on how to change default configuration is here: https://doc.twake.app/configuration

Could you please elaborate as documentation isn't clear at all.
I'm running MinIO in a docker.
Setting up S3 in Parameters.php doesn't seem to work. When uploading a file, these aren't stored nowhere.
It is necessary to create an Application for the S3 server as well? If yes, any example you could share?

Thank you for your assistance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants