Multi-threaded Borg / Scaling performance with CPU cores

[Hammit]

I'm currently on holiday and only have my tablet, but I'm looking to migrate away from Deja Dup / Duplicity due to poor backup performance. It runs on one CPU core only and takes many many hours to do a backup. Based on what I've seen in the docs and the issues on GitHub, Borg will have the same problem.

It seems like this is something that's on the roadmap though so I'm interested in learning what the current state of affairs is. Is this planned for any particular version in the future? Any and all info appreciated

P.s. Borg looks great from what I've read. Really hope I can send up using it

[ThomasWaldmann]

Having the same problem: yes and no (usually no).

Correct is that borg is currently single-threaded and your first backup using borg will take a relatively long time.

But (and this is different from older backup tools), due to the chunk-based deduplication, all following backups of the same data sets will be much much faster, because it will not process/store data chunks again which it already has in the repository (== backup storage).

Additionally, borg uses some clever tricks with the "files cache" and in-memory hash tables, always making full backups super fast (if a lot of files did not change at all, which is often the case).

Thus, while multithreading is on the roadmap, it is not as important and not as pressing as one might think it is. It's often just a "one-time patience issue" (initial backup) and avoiding CPU work is always better than parallelising cpu work.

Of course, multithreading becomes increasingly more important with growing amounts of changed chunks between backups. Folks who produce lots of new data all the time need it.

Thus, next releases (2.0, 2.1) will address mostly improving crypto and getting rid of old code and some other long-term open issues that we could not yet fix without breaking compatibility. The better crypto will also unblock/solve some multi-repo / multi-client issues of the past.

Once we're finished with that, guess it will be the time for adding multithreading.

[Hammit]

Ok. Interesting. Thanks for your comment on the subject. I shouldn't be saying this, but I look forward to getting back from holidays and trying my first backup 🙂

[Hammit]

Just a follow up on this. I got back from holidays and did a backup using lz4 compression and it was actually very fast. I didn't even see it hit 100% of the CPU. Single threaded performance was good. It took me roughly 1hr to backup 650GB. I think that's roughly 185MB/s. Quite good.

[ThomasWaldmann]

How many files per second is that? Maybe use --stats --info --list to check.

[Hammit]

1074727 files total I ran cancelled and resumed the backup a few time so I'm not sure exactly how long it took. Does the final printout of stats give total runtime or the time of the most recent backup run? from my mobile

…

On Wed, 24 Aug 2022, 7:59 pm TW, ***@***.***> wrote: How many files per second is that? Maybe use --stats --info --list to check. — Reply to this email directly, view it on GitHub <#6958 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABOIXVLHC22HR7EU3XRDNYLV2YFCRANCNFSM56AQPPSQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ThomasWaldmann]

It refers to the current run and archiving start/end refer to the time needed for the actual backup (not including startup/shutdown overhead of borg).

[ShaiMagal]

@ThomasWaldmann @Hammit
Guys, is there any update after more than 2 years? Multithread borg can be great, one core is not enough and it's bottleneck.

[Hammit]

I'm using Ubuntu and the latest version (24.10) is only at 2.0.0b
Given that it was it wasn't scheduled for 2.0.1, I think it's fair to say this is generally not available yet. Like you though, I'd be interested in an update to know if the intention is still to implement it soon after that

[ThomasWaldmann]

There was some progress in master branch by moving away from a transactional, log-like (always append) repository with a single "manifest" chunk having the directory of archives inside it. Also moving away from using reference counting.

With such a structure it would be hard to do any parallel operations and it required locking to avoid that.

Now we have (in master branch / in borg2 betas):

• separate repository objects (currently, each object is stored in a separate file) (*)
• the "directory of archives" is not in a single object anymore, but is a separate directory with separate objects (each pointing to an archive). adding or removing a directory entry is not a r-m-w operation anymore and does not require a lock anymore.
• borg compact now cleans up by determining used vs. unused objects (that is a boolean) instead of the previous reference counter.
• prune and delete are super fast now
• a lot of operations have a much simpler implementation now
• a new and quite flexible hashtable: borghash (in Cython, the old one was quite a bit of C)
• a new very simple store with misc. backends: borgstore

That now enables:

• multiple borg processes can use the repo in parallel, as their operations can not conflict anymore for most operations. the only operations needing an exclusive lock are compact and check. so we have multi-processing now already. each process has one thread, but as there can be multiple processes working on same repo: there you have it: multi-threading! :-)
• repos via file, ssh, sftp, rclone protocols

So while we still don't have single-process multithreading, there was quite a bit work done that was required before it would make sense to even try that.

(*) we likely need another big change there using client-side built pack files - there is another ticket about that, don't discuss this in here.