New to borg backup and quite baffled...

[lddubeau]

Since at least 2008 I've been running a very idiosyncratic script to perform backups from my laptop to a NAS in my home. The backups are encrypted but I do not have deduplication or compression. It is essentially the rsync with hard links method that some were promoting a few years back. Unfortunately, this script is super idiosyncratic. I translated it from Python 2 to 3 essentially by running a tool on it a while back and answering y/n to its questions. No intelligence went into the conversion. I'm still using it, but it would need a rewrite. At any rate backing up an entire home used to take about 6 hours.

Enter borg who does just about everything I want, and has deduplication and compression. So far so good, but I ran tests that got me super concerned about performance. I was never able to get it to do a backup in 6 hours. After 6 hours I have a small portion of my home backed up. At first I was using borgmatic but I removed it and now just use my own sh scripts to run borg. I then turned off compression entirely. My own backup script that I mention above did not use compression. Still too slow. Is my next step to turn off the deduplication algorithm??? I've seen --chunker-param but I don't know if that's what I should use to turn off deduplication. I did not see a flag that would just be an off switch for deduplication. Note that I could live with borg without compression and deduplication, because my script did not have these features. But I'm not sure how to turn off deduplication and whether it is going to make much of a difference.

At first I thought borg was just freezing on a file but after I turned off compression I find that it stops at another file when I kill it. So I do not think it freezes.

I'd very much to put my backup script to rest, but the current performance of borg gives me pause. And I truly cannot say that my comparison is unfair. In both cases (my script and borg) the backup is done over WiFi, and no I don't plan to run cable from the basement to the 2nd floor in a house that wasn't built for a network.

ETA: Oops. I made a mistake above. My script did have deduplication between backups but not inside a backup. If you did two backups in a row and nothing changed in your files, the new backup would take a minimal amount of new disk space but most of the backup would be a bunch of hard links to the previous backup. However, if you copied file /home/foo/a a bunch of times and backed all of that up, it would not detect that duplication and do anything about it. I believe borg backup detects this kind of duplication.

[ThomasWaldmann]

I am familiar with the rsync+hardlinks method, I also used such scripts for at least a decade. But at some time I felt "there should be something better than this meanwhile" and started to search. Then found attic and a bit later borg was forked. :-)

Rsync with hardlinks doesn't do much: it just copies files to backup IF they are modified or creates a hardlink if not. And (IIRC) the modifcation check by default does not even involve a checksum, but only compares mtime (or ctime?) timestamp.

Borg has a lot more to do (and that's the reason why it is slower (for first backup) and uses more CPU and RAM):

• it chunks the file (using the relatively expensive buzhash algorithm),
• then it computes the "id" of the chunk (hmac-sha256 (often slow, except if your CPU has sha256 acceleration) or blake2b (fast, in software)),
• then it checks whether this chunk is already in the repo (local hashtable lookup, fast). if so, the processing of the chunk is completed here. otherwise it needs to process the chunk:
• compresses the chunks (the default lz4 is super fast),
• then encrypts (AES, usually fast if your CPU has AES acceleration as usual since about 10y),
• then authenticates ("signs") using hmac-sha256 or blake2b (see above),
• then transmits to repo (remote repo), usually involving an ssh connection (also does a lot of crypto),
• then stores the chunk into a key/value store (the key is the chunk id, the value is the data). while doing that, it computes a crc32 of the data (repo low-level checksum, used by borg check --repository) and also updates the repo index (another hashtable).

The key/value store being used as storage is the reason why you can't switch of deduplication in borg - it is not able to store data without deduplicating it. But nobody would want that anyway.

What you can try:

• make sure borg has enough RAM (depends on how big your repo is / how many files you have)
• use one of the blake2 modes for --encryption except if you positively know your CPU (and openssl) accelerates sha256 (then stay with hmac-sha256).
• don't use any expensive compression. the default is lz4 and super fast. uncompressed is often slower than lz4.
• just wait. you can also interrupt it and start it again as often as you like, it will converge against a valid "completed" state (see --checkpoint-interval, maybe use the default, but in any case don't make it too short). It is starting from the beginning each time, but it is still faster then as it does not store data into the repo which it already has there from last checkpoint.

Much more important than the first backup run are the many runs after that and these are usually much faster as a lot of data did not change and is already in the repository. Thus, borg can usually process about 1000 files per second in a backup run and finish only after a few minutes (depends on a lot of factors, of course).

"Freezing on a file":

• borg < 1.2 announces file names after finishing with the file. For very big files (like ISOs or VM images) this usually leads to the progress display show some previous short file for a long time while it processes the big one. People then think something is wrong because they don't see the big file's filename yet.
• with borg 1.2 this was changed to announcing the filename before starting to process it.

WiFi: a lot of people use borg over WiFi (including me). Slow for first backup, fast for all other backups. Of course a LAN connection would be faster (esp. for first backup, so that is another option you have).

Kinds of borg deduplication:

• inner dedup inside data set (of a single host, of a single backup run)
• inter-machine dedup (BUT: it is recommended to use 1 repo per host for other reasons, see FAQ)
• historical dedup (a lot of data doesn't change between backups)

[ThomasWaldmann]

From a recent backup (using borg 1.2.0rc1+):

Duration: 5 minutes 50.17 seconds
Number of files: 648467

------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              652.71 GB            623.34 GB            116.86 MB
All archives:               77.63 TB             72.27 TB            384.14 GB

Notable:

• 1800 files per second. 😁
• almost 2GBytes per second (at least it looks like that, of course it optimised away most of that by deduplication, so only 117MB of new data needed processing and storing into remote repo)
• massive amount of historical dedup

Setup: workstation class machine --> ~30Mbit/s internet connection --> small and rather old mini server

[lddubeau]

First, thank you for the intelligent response. That's much appreciated. I'm getting tired of CSRs that tell me that if I have a timezone problem in such and such device, then I need to rebuild my house. 😬

I don't think it is worth going into the finer points right now because I'm thinking my CPU may be outdated for borg. I see this in /proc/cpuinfo:

model name      : Intel(R) Celeron(R) CPU  J1900  @ 1.99GHz

AFAIK it does not have AES or SHA built in. I built this computer in April 2015 as far as I can tell. And to tell you the truth I did not remember what I put in. Maybe it is time to swap the board and CPU for something more powerful?? I am definitely going to replace the disks soon because they are quite old (older than the server itself, actually) and maybe make other changes to the server at the same time.

I have a server rack coming today. I have maybe a couple things that are currently rack mount and was planing to put stuff on shelves until I rotate my hardware out and replace it with rack mount... but maybe I should swap the file server for a rack mount server.

Suggestions appreciated, if there are any suggestions to make. That server is only used for storing backups. I'm not planing to have a media server in my house any time soon, and I run Home Assistant and Zoneminder on their own servers.

[lddubeau]

Oops. The spec for the CPU I have above is on the server. My client has this processor:

model name      : Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz

And I think that's the one that would be used for computing hashes and encrypting. It does have AES support, but I'm not sure about SHA. I'd expect it to be present.

[ThomasWaldmann]

https://ark.intel.com/content/www/us/en/ark/products/78867/intel-celeron-processor-j1900-2m-cache-up-to-2-42-ghz.html

That's an older atom family cpu - rather cheap and weak (even as new part and by back-then standards, 2013). No AES acceleration and (of course) also no sha256 acceleration (which is only present in some rather new CPUs).

But, on the repo server (== where borg serve is executed), neither AES nor sha256 are used by borg. The CPU heaviest operation there are crc32 computations (borg check, repository part) and also the sshd process uses quite some CPU. So it might be fine as long as repo index and borg fit into RAM.

On the client (== where borg create is executed) you have a decent CPU with AES hw acceleration, but no sha256 hw acceleration (I guess, check cpu flags). Most of borg's processing happens client-side. So, use repokey-blake2 for a bit more speed there. Also make sure files cache, chunks index and borg fit into RAM.

RAM usage / swap activity can be checked with atop or similar tool while borg is running.

[lddubeau]

I think I don't have sha support on the client (or the server, not that it matters for this purpose). I looked at the flags and I assume I would see the letters sha in the flags if I had it. I switched to blake2 and will see how it goes.

I have 32G of RAM. I think everything should fit in there. I don't launch backups while having things like Blender renders running.

I've not done further testing since I last posted since I had a server rack arrive and dealt with that, which meant taking the file server offline for a bit.

[jdchristensen]

@lddubeau You shouldn't have any issues with your cpus. Definitely keep compression and deduplication enabled. The first backup might be slow, but after that things should be much faster. You could try initially backing up subsets of your source. Once they are done, a full backup should be fast, and you can delete the partial backups. I don't think you mentioned how big your source is...

[lddubeau]

Is there a way to get a du that reads only the same files as borg. I'm using a pattern file that eliminates gigabytes of data so getting a figure is not as simple as doing df /home or du /home. For instance I eliminate a Windows VM image that I can recreate if a disaster were to happen. I also exclude cached files. (Well, those that I know about.) A cache can always be recreated. In the past I've been able to know how much data I was backing up by looking at the backed up files, but I've made a bunch of recent changes that make that method unreliable. (e.g. I've moved a 100GB of historical data into cold storage. I can access it if I need it but it is no longer on my laptop.)

[ThomasWaldmann]

If you use borg create --stats ... or borg info repo::archive you can see the size of the input files in the "This Archive:" leftmost column (IIRC "Original size").

[ThomasWaldmann]

The actual disk space usage can of course be higher than that due to the allocation block sizes of the filesystem (e.g. if a 100 bytes file takes a 4096 bytes block).

[lddubeau]

I'll use borg create --stats... and see. But I'm not doing it right this minute and I'll explain in another, more general, post.

[lddubeau]

Ok, the plot deepens... I might be dealing with wonky equipment???

This morning I forgot that I should be checking on my backup. I got up, made coffee, got my breakfast, watched last night's Colbert show, YouTube, what-have-you and I had zero problems with the streaming. Then I remembered I had started a backup and went to my laptop to see if it was running and it was. It is still running as we speak. I'm baffled. All my previous backups would be killed by me in the morning because I couldn't watch TV. As soon as I killed the backup, I could stream. At first, I was not using --remote-ratelimit, but I added it quickly to my create command because of the TV problems. But even when I was rate limiting, I still couldn't watch TV in the morning. (All our TV is through streaming.) I thought the limit was too high, but I did not lower it further and now it works just fine.

The only things that changed between my previous test, and the backup I have running now:

1. I installed a rack in the network closet. But I doubt the rack made any difference by itself.

2. Everything got reset because I had to unplug everything while installing the rack. This may have had an impact????

3. I dropped my ASUS router and broke two antennas. (Yay for me.) After that happened my network was super wonky. I replaced the antennas with those from another router. I reset the main router. It still acted wonkily so I reset a node that I use in the mesh. (We have a chimney that blocks WiFi signals to the living room.) It was still acting wonkily so I reset the main router again. After that, everything looked fine. Did my router resets inadvertently cleared a bug???

In other news:

1. I probably won't get a rack mount file server. I've searched and people complain about the noise. I've read the reasoning and it makes sense that pushing lots of air into tight spaces would be noisy. I might upgrade my file server but it won't be a rack mount.

2. On the other hand, I'm tired of fixing bugs in the mesh I have with my ASUS RT-AC5300. I've pretty much reached the end of my rope. I should not have had to reboot and reboot again yesterday. And I have needs that are not typical of most home users. For instance, I segregate IoT devices from dubious sources onto their own network, separate from the main network. I'm doing it now with an additional wireless router but I'd like to ditch this setup. The current system was put together in a kind of haphazard way... run into a problem and fix that problem, run into another problem and fix that other problem, rinse and repeat. And I'm sure I've learned bad ways of managing a network because the equipment I'm using is subpar. I'm thinking of moving to a Ubiquiti setup. And this would be rack mount.

I'm also concerned about what my wife would do if I were to become incapacitated or die. It is more than theoretical. I'm doing pretty well for the moment, but I was diagnosed with a primary CNS lymphoma and almost died from it. Again, I'm fine now but relapse is possible. My current setup is held together by snot and seaweed. I suspect a Ubiquiti setup would be more likely to stay up and running for longer if I were to kick the bucket, and would be more easily manageable by someone else. (Like my stepson, who is a software engineer like I am.)

[ThomasWaldmann]

Maybe resetting the router fixed something, could be.

I have unifi stuff at home and at some customer locations, their APs and the unifi controller sw is quite nice (only that mongodb dependency is a bit of a pain on linux). The setup is made for bigger installations, but one can also use it for a single AP. It's a bit more complex setup than the usual home-user wifi routers though. If one does not want to setup the unifi controller sw on some linux machine / VM, one can also get their "cloud key" (name is a bit irritating, considering you don't need to use the cloud for it) which is basically a small computer that locally runs the controller software.

Wish you the best!

[lddubeau]

Success! But I still need to fine tune things. I'll start with questions:

1. Is there a way to change the rate limit at run time? I expect the answer to this to be no. My use case would be to let my backup run without rate limit at night, but if it takes too long and runs my work day then I'd want to limit the rate when I get up.

2. If the answer above is "no", then what should I do if I don't set a rate limit and the backup is still running when I get up. I realize I can kill borg and run the backup again the next night, but that's not optimal. With the rate limit I had set I could let the backup run while I was working during the day, and this would be optimal.

3. I have Ubiquiti hardware on order. Is there a way to deprioritize the borg backup stream of data?? So that other stuff would have priority over it? (I've never messed with my default router settings regarding how to schedule my various streams of data. I suspect it is doable but I've not looked into it.) This would obviate the need to use the solutions above.

And now the results of my first backups...

I was using --remote-ratelimit but I put this in place when I had a bug and I may have slowed the backup too much. At any rate my initial backup ran for a few days and then I had my files backed up. If I run borg info on that backup this is what I get:

+ borg info ::x-2022-02-18T04:00:02
Archive name: x-2022-02-18T04:00:02
Archive fingerprint: [something]
Comment: 
Hostname: x
Username: root
Time (start): Thu, 2022-02-17 23:00:04
Time (end): Sun, 2022-02-20 15:42:33
Duration: 2 days 16 hours 42 minutes 29.47 seconds
Number of files: 1688697
Command line: /usr/bin/borg create --patterns-from /home/q/.config/backup/patterns --remote-ratelimit 500 --compression lz4 --list --debug '::x-{utcnow}' /home/q /etc /var/log/syslog /var/log/syslog.1 /var/log/syslog.2.gz /var/log/syslog.3.gz /var/log/syslog.4.gz /var/log/syslog.6.gz /var/log/syslog.7.gz
Utilization of maximum supported archive size: 1%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              163.82 GB            135.69 GB            317.38 MB
All archives:              400.26 GB            339.65 GB            126.66 GB

                       Unique chunks         Total chunks
Chunk index:                  687665              3501668

[Sensitive information changed to protect the innocent. 🤣]

Last night the first backup after the initial backup ran. I think this is the interesting portion of --stats of that backup:

------------------------------------------------------------------------------
Archive name: x-2022-02-21T04:00:02
Archive fingerprint: [something else]
Time (start): Sun, 2022-02-20 23:00:03
Time (end):   Mon, 2022-02-21 03:44:12
Duration: 4 hours 44 minutes 8.24 seconds
Number of files: 1689906
Utilization of max. archive size: 1%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              172.97 GB            144.02 GB              8.50 GB
All archives:              400.26 GB            339.65 GB            126.66 GB

                       Unique chunks         Total chunks
Chunk index:                  687665              3501668
------------------------------------------------------------------------------
RemoteRepository: 8.50 GB bytes sent, 412.78 kB bytes received, 8287 messages sent

Almost 5 hours is slow but I still was using the rate limit flag. I think I'll drop it tonight and see how fast it goes. (Also, there's going to be less modifications because it will just have one day of modifications instead of 2-3 days.)

[ThomasWaldmann]

No dynamic changing of the rate-limit possible, IIRC.

What you could maybe do on your router is use QoS somehow and prioritze your important daytime stuff (esp. realtime stuff like VOIP, Video/Audio-Conferencing, interactive ssh) higher than the borg backup related packets.

[jdchristensen]

@lddubeau With just one day's data and no rate limit, you should be fine having it complete during the night, so there will be no need to dynamically change the rate limit or get fancy with your network. Note that 8500000/500/60/60 = 4.72 hours = 4 hours 43.2 minutes, so the rate limit you set is the only reason that it took almost five hours to upload 8.5GB.

[lddubeau]

Here the latest backup that ran last night after I went to bed. This was run without rate limit. About 4 minutes and a half.

------------------------------------------------------------------------------
Archive name: x-2022-02-22T04:00:01
Archive fingerprint: [something]
Time (start): Mon, 2022-02-21 23:00:03
Time (end):   Mon, 2022-02-21 23:04:31
Duration: 4 minutes 27.99 seconds
Number of files: 1689976
Utilization of max. archive size: 1%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:              172.62 GB            143.65 GB              1.06 GB
All archives:              572.88 GB            483.30 GB            127.72 GB

                       Unique chunks         Total chunks
Chunk index:                  691348              5242538
------------------------------------------------------------------------------
RemoteRepository: 1.06 GB bytes sent, 58.14 kB bytes received, 3692 messages sent

It should be even faster tonight because I did not do anything today. I got a 4th shot of vaccine yesterday and I don't feel great today. (Immunocompromised folks get more boosters than the general population.) I guess that's great because it means my immune system responded robustly to the vaccine. 🥳

[m3nu]

Added some material from this discussion to our FAQ. Let me know if there is anything important that's still missing with regards to performance.

https://github.com/borgbackup/borg/pull/6425/files

[ThomasWaldmann]

Merged: https://borgbackup.readthedocs.io/en/master/faq.html#what-s-the-expected-backup-performance