Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

32-bit limits file uploads to 2GB? #4125

Closed
rsteans opened this issue Sep 6, 2017 · 13 comments
Closed

32-bit limits file uploads to 2GB? #4125

rsteans opened this issue Sep 6, 2017 · 13 comments

Comments

@rsteans
Copy link

rsteans commented Sep 6, 2017

Hi y'all,

It is definitely possible that we're the only ones having this issue, but due to the limits of 32-bit, we've been unable to use the interface of dataverse to upload files larger than 2 GB. Tar'd files, etc... that need extraction are even smaller.

https://en.wikipedia.org/wiki/2_GB_limit

Mentioned this on Google Groups and Phil suggested I log it as a ticket.

@rsteans
Copy link
Author

rsteans commented Sep 6, 2017

Let me be clear - I think we decided that this was the reason we couldn't do better than 2GB. This may be incorrect.

@pdurbin
Copy link
Member

pdurbin commented Sep 6, 2017

Thanks for opening this issue, @rsteans and yes, this is a follow on from https://groups.google.com/d/msg/dataverse-community/yMi4KHy-T00/2gtmUYrxAAAJ

My understanding is that you're running Dataverse 4.7.1 on AWS with files stored on EBS (Elastic Block Store). I'm not sure why you're hitting this 2 GB limit. I guess we're hoping that when you move to S3 (#3921), the limit will be gone?

@rsteans
Copy link
Author

rsteans commented Sep 6, 2017 via email

@pdurbin
Copy link
Member

pdurbin commented Sep 7, 2017

@rsteans could it be that you're thinking about the :MaxFileUploadSizeInBytes setting? As it says at http://guides.dataverse.org/en/4.7.1/installation/config.html#maxfileuploadsizeinbytes , "If the :MaxFileUploadSizeInBytes is NOT set, uploads, including SWORD may be of unlimited size." So you might have set this to prevent uploads of unlimited size. #2169 is related but I don't want to throw too much at you at once.

@4tikhonov
Copy link
Contributor

Well, DANS has the same problem with file limitation, we can upload only 5GB even with :MaxFileUploadSizeInBytes adjusted to 1TB. We think that bottleneck is proxy timeout (Apache) as we tested upload directly to glassfish by creating tunnel to our server and it did work even for 20-30Gb.

@rsteans
Copy link
Author

rsteans commented Sep 20, 2017

Hi Phil,

We'll take a look at that setting next week and see if we can sort through this on our end. I'd like to make sure we're not setting anyone on a wild goose chase.

@pdurbin
Copy link
Member

pdurbin commented Sep 20, 2017

Sounds good. Thanks @rsteans . @4tikhonov I'm glad to hear you aren't affected. 30 GB is pretty big! Maybe you should try out the Data Capture Module ( http://guides.dataverse.org/en/4.7.1/installation/data-capture-module.html ) once it's farther along!

@pdurbin
Copy link
Member

pdurbin commented Oct 24, 2017

@rsteans any news? Also, I saw your tweet! https://twitter.com/TDLRyan/status/920835247826178048

@djbrooke
Copy link
Contributor

I'm going to go ahead and close this one out.

TDL and DANS folks - @CCMumma and @4tikhonov - if y'all have some more info here and feel it needs deeper investigation, feel free to open it back up. Thanks!

@pdurbin
Copy link
Member

pdurbin commented Nov 15, 2017

I don't believe @CCMumma and @4tikhonov have permission to re-open issues and @rsteans has moved on (congrats! Check out the Dataverse mention at https://groups.google.com/d/msg/samvera-community/HMR1xK9JfmM/oxUVASKqAwAJ ) but they (or others) should definitely feel free to open a new issue if needed!

@rsteans
Copy link
Author

rsteans commented Nov 16, 2017

I have moved on, but these emails are now following me to Northwestern, so let me know how I can help.

@CCMumma
Copy link

CCMumma commented Nov 21, 2017

I think we'll know more once we upgrade to S3, which we plan to do with our next production upgrade once 5.0 is released. If we still see the issue, we'll re-open. Thanks, all,
Courtney

@pdurbin
Copy link
Member

pdurbin commented Mar 1, 2018

#4439 is pretty much the new version of this issue where we're continuing to work with TDL folks like @CCMumma .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants