feat: Use nginx mod_zip to generate multi-file zip downloads #1102
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Test Results81 tests 81 ✅ 21s ⏱️ Results for commit 0680063. ♻️ This comment has been updated with latest results. |
This change installs and configures the `mod_zip` nginx module [1], which allows nginx to stream ZIP files directly. It includes a workaround needed to correctly calculate CRC-32 values for included files, by including a new `server` section listening at port 8081, only used for the file requests to be upstream subrequests that correctly trigger the CRC-32 calculation logic. Also, to be able to provide a `m3u` file generated on the fly, we add a `/decode` endpoint fully implemented in nginx using NJS, which receives a `value` URL param, and decodes it using base64. The decoded value is returned as the response. That way, the contents of the `m3u` file is base64-encoded, and set as part of the response, for `mod_zip` to include it in the ZIP file. [1] https://github.com/evanmiller/mod_zip
adamantike
force-pushed
the
feat/use-nginx-mod-zip
branch
from
1327f0a
to
0fad8ac
Compare
gantoine
reviewed
Aug 21, 2024
gantoine
approved these changes
Aug 21, 2024
spiceratops
referenced
this pull request
in spiceratops/k8s-gitops
Sep 1, 2024
This PR contains the following updates: | Package | Update | Change | |---|---|---| | [rommapp/romm](https://github.com/rommapp/romm) | minor | `3.4.0` -> `3.5.0` | --- > [!WARNING] > Some dependencies could not be looked up. Check the Dependency Dashboard for more information. --- ### Release Notes <details> <summary>rommapp/romm (rommapp/romm)</summary> ### [`v3.5.0`](https://github.com/rommapp/romm/releases/tag/3.5.0) [Compare Source](https://github.com/rommapp/romm/compare/3.4.0...3.5.0) #### What's Changed - misc: Improve typing for feed schemas by [@​adamantike](https://github.com/adamantike) in [https://github.com/rommapp/romm/pull/1064](https://github.com/rommapp/romm/pull/1064) - \[ROMM-1063] Add Amiga CD32 to emulatorjs game list by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1065](https://github.com/rommapp/romm/pull/1065) - fix: fab overlay fixed by [@​zurdi15](https://github.com/zurdi15) in [https://github.com/rommapp/romm/pull/1073](https://github.com/rommapp/romm/pull/1073) - fix: Readable setup text on white theme by [@​zurdi15](https://github.com/zurdi15) in [https://github.com/rommapp/romm/pull/1072](https://github.com/rommapp/romm/pull/1072) - fix: Allow access to Tinfoil feed when download auth is disabled by [@​adamantike](https://github.com/adamantike) in [https://github.com/rommapp/romm/pull/1078](https://github.com/rommapp/romm/pull/1078) - fix: Set Gunicorn option forwarded-allow-ips by [@​adamantike](https://github.com/adamantike) in [https://github.com/rommapp/romm/pull/1077](https://github.com/rommapp/romm/pull/1077) - Improve speed of fetching siblings for roms by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1076](https://github.com/rommapp/romm/pull/1076) - Ruffle flash emulator by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1049](https://github.com/rommapp/romm/pull/1049) - Calculate and store hashes for rom files by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1005](https://github.com/rommapp/romm/pull/1005) - Hotfix scans when running HASH_SCAN by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1081](https://github.com/rommapp/romm/pull/1081) - Fix playing emulatorjs + better platform icon loading by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1086](https://github.com/rommapp/romm/pull/1086) - Fix home icon size on safari/ios by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1088](https://github.com/rommapp/romm/pull/1088) - misc: Upgrade Python to v3.12 and Alpine to v3.20 by [@​adamantike](https://github.com/adamantike) in [https://github.com/rommapp/romm/pull/1092](https://github.com/rommapp/romm/pull/1092) - feat: Use X-Accel-Redirect to improve file download speed by [@​adamantike](https://github.com/adamantike) in [https://github.com/rommapp/romm/pull/1084](https://github.com/rommapp/romm/pull/1084) - misc: Pin Node version to v20 by [@​adamantike](https://github.com/adamantike) in [https://github.com/rommapp/romm/pull/1095](https://github.com/rommapp/romm/pull/1095) - Add icon to gallery header for current platform by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1093](https://github.com/rommapp/romm/pull/1093) - Upload progress bars by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1096](https://github.com/rommapp/romm/pull/1096) - Skip compressed files if theyre invalid by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1097](https://github.com/rommapp/romm/pull/1097) - Bump emulatorjs to 4.1.1 by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1100](https://github.com/rommapp/romm/pull/1100) - feat: Use nginx mod_zip to generate multi-file zip downloads by [@​adamantike](https://github.com/adamantike) in [https://github.com/rommapp/romm/pull/1102](https://github.com/rommapp/romm/pull/1102) - misc: Use single SQLAlchemy engine and session maker by [@​adamantike](https://github.com/adamantike) in [https://github.com/rommapp/romm/pull/1105](https://github.com/rommapp/romm/pull/1105) - Switch funding to open collective by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1108](https://github.com/rommapp/romm/pull/1108) - Replace illegal fs chars in filenames by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1109](https://github.com/rommapp/romm/pull/1109) - Upload dialog hotfixed + clear button by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1118](https://github.com/rommapp/romm/pull/1118) - Fix detecting if platform is flash games by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1120](https://github.com/rommapp/romm/pull/1120) - feat(UX) - Allow links to open in a new tab by [@​SaraVieira](https://github.com/SaraVieira) in [https://github.com/rommapp/romm/pull/1116](https://github.com/rommapp/romm/pull/1116) - fix(icons) - match icon names to platform names by [@​SaraVieira](https://github.com/SaraVieira) in [https://github.com/rommapp/romm/pull/1122](https://github.com/rommapp/romm/pull/1122) - Disable auth on rom content get endpoint when env variable is present by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1125](https://github.com/rommapp/romm/pull/1125) - Add support for formatting vuejs with trunk by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1124](https://github.com/rommapp/romm/pull/1124) - Add known bios files from retropie project by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1126](https://github.com/rommapp/romm/pull/1126) - Add titledb field to tinfoil response by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1127](https://github.com/rommapp/romm/pull/1127) - Better performance for large collections by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1128](https://github.com/rommapp/romm/pull/1128) - \[ROMM-1113] Add file path in rom edit window by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1129](https://github.com/rommapp/romm/pull/1129) - misc: Tag Docker image with major version only by [@​adamantike](https://github.com/adamantike) in [https://github.com/rommapp/romm/pull/1131](https://github.com/rommapp/romm/pull/1131) - build(deps): bump axios from 1.6.8 to 1.7.4 in /frontend by [@​dependabot](https://github.com/dependabot) in [https://github.com/rommapp/romm/pull/1132](https://github.com/rommapp/romm/pull/1132) - Bump axios to 1.7.4 by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1134](https://github.com/rommapp/romm/pull/1134) - \[ROMM-1107] Add env variable to disable in-browser emulation by [@​gantoine](https://github.com/gantoine) in [https://github.com/rommapp/romm/pull/1133](https://github.com/rommapp/romm/pull/1133) - Default both fast forward and rewind to enabled by [@​SaraVieira](https://github.com/SaraVieira) in [https://github.com/rommapp/romm/pull/1136](https://github.com/rommapp/romm/pull/1136) - Allow user to unmatch rom by [@​SaraVieira](https://github.com/SaraVieira) in [https://github.com/rommapp/romm/pull/1138](https://github.com/rommapp/romm/pull/1138) #### New Contributors - [@​SaraVieira](https://github.com/SaraVieira) made their first contribution in [https://github.com/rommapp/romm/pull/1116](https://github.com/rommapp/romm/pull/1116) **Full Changelog**: rommapp/romm@3.4.0...3.5.0 </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOC41OS4yIiwidXBkYXRlZEluVmVyIjoiMzguNTkuMiIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsicmVub3ZhdGUvY29udGFpbmVyIiwidHlwZS9taW5vciJdfQ==-->
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change installs and configures the
mod_zipnginx module [1], which allows nginx to stream ZIP files directly.It includes a workaround needed to correctly calculate CRC-32 values for included files, by including a new
serversection listening at port 8081, only used for the file requests to be upstream subrequests that correctly trigger the CRC-32 calculation logic.Also, to be able to provide a
m3ufile generated on the fly, we add a/decodeendpoint fully implemented in nginx using NJS, which receives avalueURL param, and decodes it using base64. The decoded value is returned as the response.That way, the contents of the
m3ufile is base64-encoded, and set as part of the response, formod_zipto include it in the ZIP file.[1] https://github.com/evanmiller/mod_zip