Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How to migrate from invalid chunker parameters? #7590

Closed
ThomasWaldmann opened this issue May 19, 2023 Discussed in #7586 · 2 comments
Closed

How to migrate from invalid chunker parameters? #7590

ThomasWaldmann opened this issue May 19, 2023 Discussed in #7586 · 2 comments
Milestone
1.2.5

Comments

@ThomasWaldmann
Copy link
Member

Discussed in #7586

Originally posted by ethus3h May 19, 2023
Hi, when I use the latest version of Borg packaged in Guix (updating from 1.1.11 to 1.2.3), I get invalid ChunkerParams value because the script I was using used parameters 10,14,16,4095 that do not pass the validation (parseformat.py#L164).

For now I've rolled back to the old version that allowed those parameters, but I'm wondering what parameters should I put to allow deduplication to continue working against the chunks made with those invalid values?

(I'd like to avoid having to recompress the whole repository since I've already stored several terabytes/many millions of files so re-compressing would be really inconvenient, or I'd just do that.)

Thank you!
@ThomasWaldmann
Copy link
Member Author

PR: #7589

@ThomasWaldmann ThomasWaldmann mentioned this issue May 19, 2023
Closed
@ThomasWaldmann ThomasWaldmann added this to the 1.2.5 milestone May 19, 2023
ThomasWaldmann added a commit to ThomasWaldmann/borg that referenced this issue May 19, 2023
@ThomasWaldmann
tests: only warn about "invalid" chunker params, fixes borgbackup#7590
5cc4e3a 
we previously allowed some weird chunker params, so we just warn
about them instead of rejecting them.

e.g. one could use super small chunk sizes. borg can do that,
but you'll end up with a huge amount of chunks and very large
hash tables (RAM usage) to manage them.

also, one can violate the min <= mask <= max chunker param condition
as in borgbackup#7590 and it will somehow work, but will likely not dedup as good
as when not violating that.

borg2 **will** reject such strange chunker params, see borgbackup#7586 for
some ideas how to "fix" your repos.
@ThomasWaldmann
Copy link
Member Author

ThomasWaldmann commented May 23, 2023 •
edited
Loading

fixed in 1.2-maint (will be in 1.2.5) be relaxing the chunker params checks.

master (borg 2) will do the more strict checks though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
1.2.5
Development

No branches or pull requests
1 participant
@ThomasWaldmann