-
Notifications
You must be signed in to change notification settings - Fork 493
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
6505 optimize zip downloads #6986
Conversation
…ate the external "custom download" service. Everything else is done by an outside standalone program (a java program with its own pom file). (#6505)
still working on the documentation, so will need to check it in later.
added some info to the documentation explaining how the zipper does its thing. (#6505)
(fixed merge conflicts w/develop - mostly the POST handling added for the /api/access/datafiles/ API)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
...nload/src/main/java/edu/harvard/iq/dataverse/custom/service/download/ZipDownloadService.java
Outdated
Show resolved
Hide resolved
...nload/src/main/java/edu/harvard/iq/dataverse/custom/service/download/ZipDownloadService.java
Outdated
Show resolved
Hide resolved
@@ -0,0 +1,2 @@ | |||
-- maybe temporary? - work in progress | |||
CREATE TABLE IF NOT EXISTS CUSTOMZIPSERVICEREQUEST (KEY VARCHAR(63), STORAGELOCATION VARCHAR(255), FILENAME VARCHAR(255), ISSUETIME TIMESTAMP); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we use our usual @Entity
convention and have JPA create and manage this table? And remove this SQL script?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That was on purpose - I wanted this table to live outside what's managed by JPA; and I didn't want to make its entries entities.
In part because the table is going to be modified externally (the zipper job cleaning after itself and deleting the entries for the jobs it has served). But also to emphasize the custom-ness and hacky-ness of this setup?
src/main/java/edu/harvard/iq/dataverse/FileDownloadServiceBean.java
Outdated
Show resolved
Hide resolved
@pdurbin Thanks again - as I said earlier, that was super useful feedback; including the things we discussed off github. |
Co-authored-by: Philip Durbin <philip_durbin@harvard.edu>
…tom/service/download/ZipDownloadService.java Co-authored-by: Philip Durbin <philip_durbin@harvard.edu>
…/dataverse into 6505-optimize-zip-downloads
Co-authored-by: Philip Durbin <philip_durbin@harvard.edu>
…/dataverse into 6505-optimize-zip-downloads
Got tripped up by selinux, will review docs |
Yes, the selinux thing does deserve to be mentioned specifically - it can be confusing.
|
Added some extra text to the instruction, as discussed. |
We are seeing a browser-specific issue: Download works with FF but fails with Chrome and Microsoft Edge with download failed due to network issue. There is a warning that appears in the browser console of failing browsers (not FF): There is no server log error |
…rom working in some browsers. (#6505)
Turned out, the headers were fine as they were. It was a small error in how I was formatting the "chunks", in the "chunked encoding". It's just that Firefox, among others, happens to be more forgiving, and was willing to accept and decode the stream anyway. And Chrome is, apparently, a stickler for the rules. |
Do I need a new download zipper jar file to see the change and not just another build of dataverse? |
No Dataverse changes, only the zipper .jar file. That can be obtained on dvn-build.hmdc.harvard.edu again. |
What this PR does / why we need it:
The much debated "zipper service" - an experimental way to take zipped file downloads (extra long running jobs by design) outside of the main Application Service.
Which issue(s) this PR closes:
Closes #6505
Special notes for your reviewer:
Suggestions on how to test this:
Does this PR introduce a user interface change? If mockups are available, please link/include them here:
Is there a release notes update needed for this change?:
Additional documentation: