Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GZip compression #195

Closed
1 task
mdedetrich opened this issue Apr 14, 2022 · 0 comments · Fixed by #196
Closed
1 task

Add GZip compression #195

mdedetrich opened this issue Apr 14, 2022 · 0 comments · Fixed by #196

Comments

@mdedetrich
Copy link
Contributor

mdedetrich commented Apr 14, 2022

What is currently missing?

Guardian is currently missing compression. For saving storage space (especially with large backups) compression is usually a necessary feature

How could this be improved?

Adding a gzip compression setting to Backup configuration which will create backup files/objects that are compressed. There are certain questions/topics that arise from this

  • How we should store the fact that a key/object is compressed? The best solution to me is that compressed keys/files have a different filename, so standard backup keys/files have a .json filename where as compressed keys/files have a .json.gz
  • One corner case that needs to be thought of is if someone originally does a backup without compression but then restarts Guardian with compression setting enabled, you have currently existing state with keys/files that are uncompressed. Because of how cloud storage's work you cannot overwrite currently existing uncompleted chunked uploads (even if you could you wouldn't want to because its quite taxing, you would have to redownload, compress and reupload the compressed version for an unbounded amount of data). The easiest way to solve this issue would be that if you change the compression setting and reload Guardian, this won't actually apply until a completely new object/key is made (this should also be documented).
  • Similarly to the previous point, one would need to confirm if there are any issues in switching compression levels between restarts of Guardian.

Is this a feature you would work on yourself?

  • I plan to open a pull request for this feature
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant