-
Notifications
You must be signed in to change notification settings - Fork 492
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AWS/S3 configuration and upload problem #5062
Comments
I should have added I used the install directions for setting the storage to S3.
|
@jmjamison thanks for opening this issue and for all the chatter in various places:
It's weird to me that deleting that One thing I wanted to point out is that the Thanks again. At minimum, I believe we need to fix up the docs. It's also quite possible you've found a bug. |
FWIW - dataverse.files.directory is used to determine the temp dir where files are written before transfer to S3. (If not set the default is /tmp/files/temp). So having it set probably isn't a problem itself, but if that dir doesn't exist or isn't writable... |
I've confirmed through testing it behaves the way @qqmyers described. Additionally, if I configure a path the glassfish user does not have write access to, you can select files to upload, they appear to upload but then do not appear in the uploaded list on the upload files page -they disappear. |
Hey guys, IMHO there should be an error message that the files cannot be uploaded right now before the potentially large upload starts. At least there should be a clear error message. Maybe add check at startup time that ensures a writeable path? And maybe an extra check before the upload starts? Ideally there should also be a check if the file can be uploaded in terms of storage capacity within the temp folder, but this is another story. |
@poikilotherm there is definitely room for improvement. Any interest in poking around in the code and maybe making a pull request? |
FWIW: The permission issue that this issue started with should be a one-time thing, i.e. once you're setup correctly, users shouldn't be running into it. A check could be written, but it might still be overkill to run it for every upload. Managing disk space would trickier in many ways. Since multiple users could be uploading in parallel and one user can drag files into the upload interface sequentially, attempts to avoid running out of space would have to reserve space and effectively release it if/when not needed and the GUI would need logic to stop additions to an upload but allow the existing files to complete. Having just been through the code to find places where temp files are being left (pre 4.9.3, when users hit cancel on an upload, or deleted some files before save, temp files were left) I know there are multiple places any reservation would have to be released. (There's still the cases where a user leaves the page without cancelling or saving or a network drop where I haven't yet tracked down how to delete temp files where reservations would also have to be cancelled). At a minimum, if you have limited disk space for temp files, 4.9.3 should be better at making sure it doesn't fill up. Some good news perhaps is that I think the upload problem is visible if it happens and probably could be made more visible: files dragged into the upload pane are only transferred to the bottom if the ingest completes and writes the temp file. I did submit code a while back to catch and show non-20x responses from the server during upload - I don't know if that's already catching when there's a permission of out-of-space issue (it should if the response is not 201). If so, there's already a GUI display when the error occurs and one could add more logic to the javascript triggering that to decide whether the failure is one that should shut down other ongoing uploads (or warn the user to do a 'cancel', etc.). |
@jmjamison hi! Are you still having this problem? Can we close this issue? Thanks. |
Apologies. I got side tracked to another (non-Dataverse) problem. I'll go ahead and close this. |
Apologies. I got side tracked to another (non-Dataverse) problem. I'll go ahead and close this. |
I have a test dataverse on AWS using S3 storage. (http://54.67.118.35:8080/). We were unable to upload files.
The jvm dataverse.files.directory is default set to -Ddataverse.files.directory=/usr/local/dvn/data. Once that option was deleted we could upload and display files.
Although it appears to work I'm really worried about just deleting a jvm option without know what it should be set to.
Jamie Jamison
UCLA data science center
The text was updated successfully, but these errors were encountered: