Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: ZIP streaming decompression #1037

Open
wants to merge 59 commits into
base: dev/exp
Choose a base branch
from

Conversation

iamscottxu
Copy link
Contributor

@iamscottxu iamscottxu commented Aug 16, 2024

新增功能: #861 利用HTTP分片下载机制实现ZIP流式下载,且支持分卷压缩文件,支持限速。经过测试,目前支持原神、崩坏:星穹铁道(2.5.0完整包用了7z格式且固实压缩,任支持增量包更新)、绝区零中国官方、全球官方、哔哩哔哩的安装和更新。崩坏3因为使用了7z压缩格式,暂不支持。
大致流程:

  1. 下载前先获取压缩文件的中央目录数据,并保存为zipcdr文件;
  2. 解析并遍历中央目录数据中的目录实体,创建目录;
  3. 解析并遍历中央目录数据中的文件实体,获取各文件的文件名、修改日期、压缩算法、压缩文件大小和在压缩文件中的偏移信息;
  4. 如果有已经存在的同名文件,检查其大小和创建时间(更新时不检查创建时间,进行CRC校验);
  5. 利用HTTP分片下载机制从压缩文件的特定位置开始获取压缩文件数据头,中央目录数据进行核对并校验有效性;
  6. 根据中央目录数据创建文件,并预分配磁盘空间;
  7. 从压缩文件头后开始下载压缩文件,并缓存到内存(不存储到硬盘),边下载边解压,下载完成即解压完成,下载中如果出现网络错误延迟1秒重试;
  8. 下载解压完成后设置文件创建时间和修改时间,并进行CRC和文件大小校验,保证文件完整性。

效果: 缩减下载时硬盘占用近一半,且大幅缩减游戏安装时间。
新增依赖: SharpZipLib: 1.4.2

image
image
image
image

2024年10月17日更新:
2.5.0完整包用了7z的固实压缩格式,字块数为21。相当于将未压缩的文件分成21份,每份中的文件看成一个整体去执行压缩算法。理论上能实现流式解压缩,但是实现难度大大增加。(可能涉及到文件系统底层操作,要预分配一段连续的存储空间进行下载和解压,然后再按照顺序划分给各文件)

Signed-off-by: scottxu <scottxu@scottxublog.com>
Signed-off-by: scottxu <scottxu@scottxublog.com>

# Conflicts:
#	src/Starward.Language/Lang.resx
#	src/Starward.Language/Lang.zh-CN.resx
Signed-off-by: scottxu <scottxu@scottxublog.com>
…ry files.

Signed-off-by: scottxu <scottxu@scottxublog.com>
…ion Path Description`

Signed-off-by: scottxu <scottxu@scottxublog.com>
@Scighost
Copy link
Owner

太强了,这么多代码光审查都要很久

Signed-off-by: scottxu <scottxu@scottxublog.com>
Signed-off-by: scottxu <scottxu@scottxublog.com>
Signed-off-by: scottxu <scottxu@scottxublog.com>
…cause unexpected behavior after a traditional download mode task was started

Signed-off-by: scottxu <scottxu@scottxublog.com>
…a streaming download was retried

Signed-off-by: scottxu <scottxu@scottxublog.com>
…oads

Signed-off-by: scottxu <scottxu@scottxublog.com>
@iamscottxu
Copy link
Contributor Author

iamscottxu commented Aug 17, 2024

修了一些Bug,现在发现几个待处理的问题。

  • 快速点击暂停和启动出现行为异常,怀疑是暂停按钮发出取消令牌后,未等到任务完全取消就将继续按钮置为可用状态,导致在下次任务启动时,上次的任务没有完全取消,导致行为异常。(因为流式下载在取消命令发出后不能立即取消,导致该问题,新增取消状态解决)
  • 下载到一半报服务器上的文件在下载途中被修改了,经过反复解析域名测试发现米哈游的域名解析服务器会随机返回IP地址,估计是不同服务器上的文件创建时间不同导致校验不通过引发此异常。(单个文件下载时已固定IP)
  • 服务器不定时关闭与客户端的连接,目前只能通过重试来解决问题,测试通过设置SocketHttpHandler的HTTP最大并发数参数可减少这种情况,怀疑是服务器流控。(限制单服务器并发连接数到5后,大幅缓解,未严重影响下载速度,暂不处理。)
  • 暂停后再启动,因为多线程问题,有概率造成队列顺序与先前不一致,先下载音频资源,导致进度条与暂停前不一致。(将流式下载解压的任务放在其他任务前单线程执行(流式下载自身为多线程任务,方法内部会启动线程),使用线程不安全队列,任务完成后再出队)

…rrectly caught as `TaskCanceledException`

Signed-off-by: scottxu <scottxu@scottxublog.com>
…tialDownloadException was thrown

Signed-off-by: scottxu <scottxu@scottxublog.com>
…ZIP downloads

Signed-off-by: scottxu <scottxu@scottxublog.com>
@iamscottxu
Copy link
Contributor Author

iamscottxu commented Aug 18, 2024

image
@Scighost 下载包好像报错了,怕不是服务器网络抽风?

@Scighost
Copy link
Owner

我重新运行一下

Signed-off-by: scottxu <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
…wnloads

Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
…ancel

Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
…ming download

Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
… fixed the bug in which the progress was displayed incorrectly after the download was paused and restarted

Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
…completion or cancellation when streaming downloads

Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>

# Conflicts:
#	src/Starward.Core.ZipStreamDownload/FastZipStreamDownloadTask.partial.cs
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
…e few files in the compressed package during streaming download

Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
… times by multiple threads (Scighost#1083) (Scighost#1082)

Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>

# Conflicts:
#	src/Starward/Services/Download/InstallGameService.cs
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>

# Conflicts:
#	src/Starward/Services/Download/InstallGameService.cs
…re were multiple steps of the task when installing the game

Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
…leted

Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>

# Conflicts:
#	src/Starward.Language/Lang.de-DE.resx
#	src/Starward.Language/Lang.es-ES.resx
#	src/Starward.Language/Lang.it-IT.resx
#	src/Starward.Language/Lang.ja-JP.resx
#	src/Starward.Language/Lang.ko-KR.resx
#	src/Starward.Language/Lang.ru-RU.resx
#	src/Starward.Language/Lang.th-TH.resx
#	src/Starward.Language/Lang.vi-VN.resx
#	src/Starward/Services/Download/InstallGameManager.cs
#	src/Starward/Services/Download/InstallGameService.cs
…using speed limiter.

Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>

# Conflicts:
#	src/Starward/Services/Download/InstallGameService.cs
…the list of files to be deleted, delete the files in the list, then download the files to a temporary folder, and finally move the files.

Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Signed-off-by: OsakaRuma <scottxu@scottxublog.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants