Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Integration with Azure blob storage #2027

Closed
shyamal890 opened this issue Nov 6, 2016 · 31 comments
Closed

Integration with Azure blob storage #2027

shyamal890 opened this issue Nov 6, 2016 · 31 comments
Labels
enhancement feature: external storage integration Nice to have

Comments

@shyamal890
Copy link

Will NextCloud support Azure blob storage as external storage in near future?
@MariusBluem
Copy link
Member

Hm ... Let's put this as an enhancement. However - I do not see this in the near future 😁 ... Maybe you want to step in - or know somebody who wants so 😅

We have some external storage backends in Nextcloud, (Amazon) S3 seems to be finished for (primary) storage for Nextcloud 11: #768

This may be an alternative ... Sharepoint is also on the roadmap for 11, if you rely on Microsoft 😁

cc @icewind1991 @blizzz

@pascalsimi
Copy link

I find it rather puzzling that blob is not supported.
If you look at the gartner quadrant then there is Amazon and Microsoft as the big players.
Supporting only one of them cuts down potential user base by approximately 50%. Take into account how many personal computers actually run on Microsoft it's even more puzzling to me why Microsoft blob is not on the roadmap. But hey its just my thoughts. Thanks for all the hard work you put into next cloud. All you guys are awesome!

@nickvergessen nickvergessen changed the title Feature Request : Integration with Azure blob storage Integration with Azure blob storage Nov 7, 2016
@gjonespf
Copy link

gjonespf commented Nov 7, 2016

Would love to see this also. May be worth mentioning that (and I haven't tested this) but Azure File Store allows SMB access, so you could potentially run Nextcloud on an Azure VM with external storage mapped to the Azure File Store, as long as the VM and storage were in the same region (otherwise you need SMB3 encryption support, which isn't there yet in Linux). Something I will have to investigate.

@pascalsimi
Copy link

I have a few projects with Microsoft azure for processing and nextcloud for storage and sync right now and with their new push for openness on azure I think there is a good chance their evangelists might be interested in helping to implement that feature.
I'll mention it in the next meeting.

@pilcherd
Copy link

pilcherd commented Feb 9, 2017

Support for Azure Blob would be good. Interesting about Azure File Store and SMB. Will look into that.

@MariusBluem
Copy link
Member

https://github.com/icewind1991/files_external_azure

Digging for old code 😜😜 What was the state there @icewind1991 🤔🙈

@icewind1991
Copy link
Member

Not working atm, will see if I can fix it when I get some time

@strategywise
Copy link

We're also interested in this ability. If the SMB mapping on the host box works would it in theory at least be able to read/write to the blob? If I have to get a blob mounted on the VM and it works well I don't think that would be a very hard work around for the time being.

@SpiraMirabilis
Copy link

SpiraMirabilis commented Sep 16, 2017 •
edited
Loading

I've been tinkering with this, using the S3 backend as a template. It's a bit of a work in progress. However, as a temporary measure you can use a Windows server (2008 or greater), with WebDAV authoring role for IIS and essentially act a proxy in between your users and azure blob storage, see this how-to: http://fabriccontroller.net/mounting-your-azure-file-shares-on-premises-through-webdav/ Edit: Actually, it looks like that only supports the Azure Files which is much more expensive than blob.

Should it support serving from the verizon/akamai CDN if it is linked to that storage account?

Also, either nextcloud will need dedicated access to the blob container, or maybe we can use a Service Bus subscription and just mandate that anyone adding/changing/removing a file triggers a message sent, that way we can confidently cache the metadata/contents and there wouldnt be polling which increases cost a lot.

@MorrisJobke
Copy link
Member

This should be definitely be implemented as an app - see the files_external_dropbox external storage for example: https://github.com/icewind1991/files_external_dropbox

@CyrilHupor
Copy link

CyrilHupor commented Mar 6, 2018 •
edited
Loading

+1 on the feature which I believe is very important.

I can't however understand why this support should be an app when out of the box support is already included for AWS and OpenStack. Keep in mind Azure is leader together with AWS in the cloud space. I think this matter should be re-visited.

Btw, the reason as to why Blow Storage might be considered as opposed to Azure Files or additional storage on the VM is simply cost. Blob storage is much cheaper.

On another note, congrats to dev's on nextcloud, would just very much like support for Azure Blob.

@jbraes
Copy link

jbraes commented Mar 15, 2018 •
edited
Loading

I am always amazed how people use software is such different ways.
running your "personal" cloud on-top of "public" cloud infrastructure.
Isn't the idea of having your own cloud infrastructure , to be independent of azure, Amazon , google etc , to have full controle over your hardware

@MourIdri
Copy link

+1

@wvh-shanee
Copy link

wvh-shanee commented Jul 2, 2018 •
edited
Loading

+1

@jbraes with e2e encryption the public cloud just ends up as a big storage location to stash encrypted files. Some prefer Azure to Amazon so it would be nice to support multiple storage options. I personally like the public cloud because of automatic backups, geographic redundancy, unlimited storage, etc. Running my own box at the house (or a VPS in the cloud) just adds overhead, because I then I have to deal with redundancy and backup myself. My father-in-law is tired of me swapping hard drives out of his gun safe. ;)

@SpiraMirabilis
Copy link

SpiraMirabilis commented Jul 21, 2018 via email

@ohthehugemanatee
Copy link

@MorrisJobke Can you give some insight into your decision process here? Nextcloud supports AWS and Openstack out of the box. Why not the number 2 biggest cloud provider?

@MorrisJobke
Copy link
Member

@MorrisJobke Can you give some insight into your decision process here? Nextcloud supports AWS and Openstack out of the box. Why not the number 2 biggest cloud provider?

As this sounds like a nice feature, there are currently no plans to implement such a feature. This does not mean we don't want this feature, but it is simply not on our roadmap for the near future. If somebody wants to implement this feature nevertheless we are happy to assist and help out. I hope that explains a bit why it isn't there yet.

If you wish to have this feature implemented by the Nextcloud GmbH there is the option for consulting work on top of your Nextcloud Enterprise subscription to get your features implemented.

@powareverb
Copy link

Thanks for the clarification @MorrisJobke is what I figured. You guys only have so much time in the day. I'll be looking into Minio on blob storage, as the s3 backend for Nextcloud seems more stable now. Might be worth looking at for others.
https://azuremarketplace.microsoft.com/en-us/marketplace/apps/minio.minioblob

@ohthehugemanatee
Copy link

Thanks for the clarification.

For those who are curious, there are ways to mount Azure Blob storage directly on your VM or in k8s. That's what I ended up doing. Honestly if you're going to use it as primary storage, this is a better option than implementing in the application layer IMO.

@see

Hope this unblocks someone else in the future!

@z2z
Copy link

z2z commented Jan 28, 2019 •
edited
Loading

Maybe in future, use of https://flysystem.thephpleague.com/docs/ it already has a lot of adapters.

@mstenz
Copy link

mstenz commented Sep 16, 2019

also instead of BlobFUSE you could potentially map Azure Files with SMB directoy to the host vm.
But as you can mount it only with one access rights you need to fix some error.

Please change the permissions to 0770 so that the directory cannot be listed by other users.

f(\OC::$server->getConfig()->getSystemValue('check_data_directory_permissions', true) === false) {

I have not checked further but i think you can change the falue to false in the configuration which will fix this issue, but as i have not yet done it (still in the process) there may be issues afterwards.

@MourIdri
Copy link

MourIdri commented Jan 8, 2020

AZUREBLOBNFS3

It seems that it is possible to mount Blob natively via NFS now. Cheaper and easier than object storage (Blob, S3 or other)

@SpiraMirabilis
Copy link

SpiraMirabilis commented Jan 8, 2020 via email

@MourIdri
Copy link

MourIdri commented Jan 8, 2020

Interesting. I wonder how efficient it is with regards to unnecessary metadata requests which Azure charges for. Ideally if you’re using blob you would cache the metadata aggressively for that reason.
…
On Wed, Jan 8, 2020 at 3:44 AM chapeau @.***> wrote: [image: AZUREBLOBNFS3] https://user-images.githubusercontent.com/19348103/71972009-1ca82780-320c-11ea-99aa-ab571c21f699.JPG It seems that it is possible to mount Blob natively via NFS now. Cheaper and easier than object storage (Blob, S3 or other) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2027?email_source=notifications&email_token=AAL7NADARSB62Z2HAHAUHQ3Q4WVATA5CNFSM4CVKCH32YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIL6ZZQ#issuecomment-571993318>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAL7NAEVI6GUV7I4BDLQXBLQ4WVATANCNFSM4CVKCH3Q .

It is all about $$$ I guess, I use to have a 28 Millions requests per week on my S3 bucket for around 1000 files. In that case you just get the data and the metadata will be limited to files's GID/UID and embedded in the offering. Let's see .

@ThumbGen
Copy link

Did anyone tried the Azure Files NFS v3 as primary storage for NextCloud? Any hints/tips?

@ajdinzutic
Copy link

would also be intersted. +1 Azure Blob with features like Soft Delete and a WORM Solution with Azure Storage Explorer, it would be very good product for the clients.

@sudipmandal
Copy link

sudipmandal commented Mar 24, 2022 •
edited
Loading

For Anyone who might be interested in getting this working, here are the steps with the Nextcloud docker container, adjust accordingly if you are running on bare metal

My setup:
Host : Debian 11, Docker 20.11
Nextcloud : Docker image 23.0.3

On Host:

  1. Install Blob fuse
    As of this writing there is no blobfuse package available in the MS repo for debian 11, do the following to get this installed if using debian 11 (for other distros follow Blob fuse documentation)
    curl https://packages.microsoft.com/debian/10/prod/pool/main/b/blobfuse/blobfuse-1.4.3-Debian10.0-x86_64.deb --output blobfuse-1.4.3-Debian10.0-x86_64.deb
    sudo apt-get update
    sudo apt-get install fuse
    sudo dpkg -i blobfuse-1.4.3-Debian10.0-x86_64.deb
  2. Create a mount folder for your nextcloud "/var/www/html" folder on host say /mnt/nextcloud
  3. Create an empty folder say media inside this to be used as volume mounting point in container later, /mnt/nextcloud/media
  4. Create a mount folder for your azure blob under this like/mnt/blob
  5. Create a cache folder for blob fuse say /mnt/cache
  6. Create a connection.cfg file say in /connection.cfg and add the following line to it (without the curly brackets)
    accountName {Azure Storage Acc Name}
    accountKey {Azure Storage Access Key}
    authType Key
    containerName {Azure Blob Container Name}
  7. Run the following command to mount blob to dir
    sudo blobfuse /mnt/blob --tmp-path=/mnt/cache --config-file=/connection.cfg -o allow_other
  8. Add the above command to fstab or a new systemd service to ensure this directory is mounted automatically after server reboots

On Docker:
Create the nextcloud container as documented in docker hub, just add a new volume to the above folder as follows
docker run ... -v /mnt/nextcloud:/var/www/html -v /mnt/blob:/var/www/html/media ....

For Existing running containers, on portainer duplicate/recreate and add the volumes

Run Container, add the External Storage app to Nextcloud
Then mount the local folder /var/www/html/media to Nextcloud as external storage.

Enjoy!

P.S : Also tried to use Azure S3 gateway, Minio Azure Gateway etc, long story short, they dont work with Nextcloud External storage very well, this is the only method which worked for me 100% with good performance.

@brosahay
Copy link

brosahay commented Nov 2, 2022

I was able to set up and use blobfuse pretty decently, only annoying thing I am unable to figure out yet is the fact that nextcloud goes on telling I am out of space since the blob storage is not reporting the storage space properly. It is showing and calculating the space that is left on the OS drive.

@ThumbGen
Copy link

@brosahay Did you find any solution for the wrongly reported size? Does it influence the way Nextcloud works? Does it prevent uploading files to Nextcloud?

@z2z
Copy link

z2z commented Jan 2, 2023

@brosahay @ThumbGen report at https://github.com/Azure/azure-storage-fuse/issues

Also jfyi, both sftp and nfsv3 have cost. https://azure.microsoft.com/en-us/pricing/details/storage/blobs/

@brosahay
Copy link

@brosahay Did you find any solution for the wrongly reported size? Does it influence the way Nextcloud works? Does it prevent uploading files to Nextcloud?

It does not affect the operations in any way as such. I am working on getting the blobfuse2 working right now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement feature: external storage integration Nice to have
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests