Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Use nginx mod_zip to generate multi-file zip downloads #1102

Merged
merged 2 commits into from
Aug 21, 2024

Conversation

adamantike
Copy link
Collaborator

This change installs and configures the mod_zip nginx module [1], which allows nginx to stream ZIP files directly.

It includes a workaround needed to correctly calculate CRC-32 values for included files, by including a new server section listening at port 8081, only used for the file requests to be upstream subrequests that correctly trigger the CRC-32 calculation logic.

Also, to be able to provide a m3u file generated on the fly, we add a /decode endpoint fully implemented in nginx using NJS, which receives a value URL param, and decodes it using base64. The decoded value is returned as the response.

That way, the contents of the m3u file is base64-encoded, and set as part of the response, for mod_zip to include it in the ZIP file.

[1] https://github.com/evanmiller/mod_zip

Copy link

github-actions bot commented Aug 21, 2024

Test Results

81 tests   81 ✅  21s ⏱️
 1 suites   0 💤
 1 files     0 ❌

Results for commit 0680063.

♻️ This comment has been updated with latest results.

This change installs and configures the `mod_zip` nginx module [1],
which allows nginx to stream ZIP files directly.

It includes a workaround needed to correctly calculate CRC-32 values for
included files, by including a new `server` section listening at port
8081, only used for the file requests to be upstream subrequests that
correctly trigger the CRC-32 calculation logic.

Also, to be able to provide a `m3u` file generated on the fly, we add a
`/decode` endpoint fully implemented in nginx using NJS, which receives
a `value` URL param, and decodes it using base64. The decoded value is
returned as the response.

That way, the contents of the `m3u` file is base64-encoded, and set as
part of the response, for `mod_zip` to include it in the ZIP file.

[1] https://github.com/evanmiller/mod_zip
backend/endpoints/rom.py Show resolved Hide resolved
backend/endpoints/rom.py Show resolved Hide resolved
backend/endpoints/rom.py Show resolved Hide resolved
backend/utils/hashing.py Show resolved Hide resolved
frontend/src/services/api/rom.ts Show resolved Hide resolved
backend/utils/nginx.py Show resolved Hide resolved
docker/Dockerfile Show resolved Hide resolved
docker/nginx/default.conf Show resolved Hide resolved
@adamantike adamantike merged commit 419f3ba into master Aug 21, 2024
9 checks passed
@adamantike adamantike deleted the feat/use-nginx-mod-zip branch August 21, 2024 03:20
spiceratops referenced this pull request in spiceratops/k8s-gitops Sep 1, 2024
This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [rommapp/romm](https://github.com/rommapp/romm) | minor |
`3.4.0` -> `3.5.0` |

---

> [!WARNING]
> Some dependencies could not be looked up. Check the Dependency
Dashboard for more information.

---

### Release Notes

<details>
<summary>rommapp/romm (rommapp/romm)</summary>

###
[`v3.5.0`](https://github.com/rommapp/romm/releases/tag/3.5.0)

[Compare
Source](https://github.com/rommapp/romm/compare/3.4.0...3.5.0)

#### What's Changed

- misc: Improve typing for feed schemas by
[@&#8203;adamantike](https://github.com/adamantike) in
[https://github.com/rommapp/romm/pull/1064](https://github.com/rommapp/romm/pull/1064)
- \[ROMM-1063] Add Amiga CD32 to emulatorjs game list by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1065](https://github.com/rommapp/romm/pull/1065)
- fix: fab overlay fixed by
[@&#8203;zurdi15](https://github.com/zurdi15) in
[https://github.com/rommapp/romm/pull/1073](https://github.com/rommapp/romm/pull/1073)
- fix: Readable setup text on white theme by
[@&#8203;zurdi15](https://github.com/zurdi15) in
[https://github.com/rommapp/romm/pull/1072](https://github.com/rommapp/romm/pull/1072)
- fix: Allow access to Tinfoil feed when download auth is disabled by
[@&#8203;adamantike](https://github.com/adamantike) in
[https://github.com/rommapp/romm/pull/1078](https://github.com/rommapp/romm/pull/1078)
- fix: Set Gunicorn option forwarded-allow-ips by
[@&#8203;adamantike](https://github.com/adamantike) in
[https://github.com/rommapp/romm/pull/1077](https://github.com/rommapp/romm/pull/1077)
- Improve speed of fetching siblings for roms by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1076](https://github.com/rommapp/romm/pull/1076)
- Ruffle flash emulator by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1049](https://github.com/rommapp/romm/pull/1049)
- Calculate and store hashes for rom files by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1005](https://github.com/rommapp/romm/pull/1005)
- Hotfix scans when running HASH_SCAN by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1081](https://github.com/rommapp/romm/pull/1081)
- Fix playing emulatorjs + better platform icon loading by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1086](https://github.com/rommapp/romm/pull/1086)
- Fix home icon size on safari/ios by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1088](https://github.com/rommapp/romm/pull/1088)
- misc: Upgrade Python to v3.12 and Alpine to v3.20 by
[@&#8203;adamantike](https://github.com/adamantike) in
[https://github.com/rommapp/romm/pull/1092](https://github.com/rommapp/romm/pull/1092)
- feat: Use X-Accel-Redirect to improve file download speed by
[@&#8203;adamantike](https://github.com/adamantike) in
[https://github.com/rommapp/romm/pull/1084](https://github.com/rommapp/romm/pull/1084)
- misc: Pin Node version to v20 by
[@&#8203;adamantike](https://github.com/adamantike) in
[https://github.com/rommapp/romm/pull/1095](https://github.com/rommapp/romm/pull/1095)
- Add icon to gallery header for current platform by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1093](https://github.com/rommapp/romm/pull/1093)
- Upload progress bars by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1096](https://github.com/rommapp/romm/pull/1096)
- Skip compressed files if theyre invalid by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1097](https://github.com/rommapp/romm/pull/1097)
- Bump emulatorjs to 4.1.1 by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1100](https://github.com/rommapp/romm/pull/1100)
- feat: Use nginx mod_zip to generate multi-file zip downloads by
[@&#8203;adamantike](https://github.com/adamantike) in
[https://github.com/rommapp/romm/pull/1102](https://github.com/rommapp/romm/pull/1102)
- misc: Use single SQLAlchemy engine and session maker by
[@&#8203;adamantike](https://github.com/adamantike) in
[https://github.com/rommapp/romm/pull/1105](https://github.com/rommapp/romm/pull/1105)
- Switch funding to open collective by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1108](https://github.com/rommapp/romm/pull/1108)
- Replace illegal fs chars in filenames by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1109](https://github.com/rommapp/romm/pull/1109)
- Upload dialog hotfixed + clear button by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1118](https://github.com/rommapp/romm/pull/1118)
- Fix detecting if platform is flash games by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1120](https://github.com/rommapp/romm/pull/1120)
- feat(UX) - Allow links to open in a new tab by
[@&#8203;SaraVieira](https://github.com/SaraVieira) in
[https://github.com/rommapp/romm/pull/1116](https://github.com/rommapp/romm/pull/1116)
- fix(icons) - match icon names to platform names by
[@&#8203;SaraVieira](https://github.com/SaraVieira) in
[https://github.com/rommapp/romm/pull/1122](https://github.com/rommapp/romm/pull/1122)
- Disable auth on rom content get endpoint when env variable is present
by [@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1125](https://github.com/rommapp/romm/pull/1125)
- Add support for formatting vuejs with trunk by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1124](https://github.com/rommapp/romm/pull/1124)
- Add known bios files from retropie project by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1126](https://github.com/rommapp/romm/pull/1126)
- Add titledb field to tinfoil response by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1127](https://github.com/rommapp/romm/pull/1127)
- Better performance for large collections by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1128](https://github.com/rommapp/romm/pull/1128)
- \[ROMM-1113] Add file path in rom edit window by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1129](https://github.com/rommapp/romm/pull/1129)
- misc: Tag Docker image with major version only by
[@&#8203;adamantike](https://github.com/adamantike) in
[https://github.com/rommapp/romm/pull/1131](https://github.com/rommapp/romm/pull/1131)
- build(deps): bump axios from 1.6.8 to 1.7.4 in /frontend by
[@&#8203;dependabot](https://github.com/dependabot) in
[https://github.com/rommapp/romm/pull/1132](https://github.com/rommapp/romm/pull/1132)
- Bump axios to 1.7.4 by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1134](https://github.com/rommapp/romm/pull/1134)
- \[ROMM-1107] Add env variable to disable in-browser emulation by
[@&#8203;gantoine](https://github.com/gantoine) in
[https://github.com/rommapp/romm/pull/1133](https://github.com/rommapp/romm/pull/1133)
- Default both fast forward and rewind to enabled by
[@&#8203;SaraVieira](https://github.com/SaraVieira) in
[https://github.com/rommapp/romm/pull/1136](https://github.com/rommapp/romm/pull/1136)
- Allow user to unmatch rom by
[@&#8203;SaraVieira](https://github.com/SaraVieira) in
[https://github.com/rommapp/romm/pull/1138](https://github.com/rommapp/romm/pull/1138)

#### New Contributors

- [@&#8203;SaraVieira](https://github.com/SaraVieira) made
their first contribution in
[https://github.com/rommapp/romm/pull/1116](https://github.com/rommapp/romm/pull/1116)

**Full Changelog**:
rommapp/romm@3.4.0...3.5.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Renovate
Bot](https://github.com/renovatebot/renovate).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOC41OS4yIiwidXBkYXRlZEluVmVyIjoiMzguNTkuMiIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsicmVub3ZhdGUvY29udGFpbmVyIiwidHlwZS9taW5vciJdfQ==-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants