Empty dir as source is not backuped #1157

Zesko · 2024-05-22T13:20:19Z

Hi, I tried to test rustic version 0.7.0 on Arch Linux with Btrfs in VM.

The output of $ rustic snapshots

snapshots for (host [archlinux], label [label], paths [/btrfs/@home/test/Desktop,/btrfs/@home/test/Documents,/btrfs/@home/test/Downloads])
| ID       | Time                | Host      | Label | Tags | Paths                       | Files | Dirs |     Size |
|----------|---------------------|-----------|-------|------|-----------------------------|-------|------|----------|
| 6dd641f9 | 2024-05-22 14:35:20 | archlinux | label |      | /btrfs/@home/test/Desktop   |   244 |  268 |  8.0 GiB |
|          |                     |           |       |      | /btrfs/@home/test/Documents |       |      |          |
|          |                     |           |       |      | /btrfs/@home/test/Downloads |       |      |          |
| 0cff64d2 | 2024-05-22 14:35:37 | archlinux | label |      | /btrfs/@home/test/Desktop   |   327 |  268 | 10.7 GiB |
|          |                     |           |       |      | /btrfs/@home/test/Documents |       |      |          |
|          |                     |           |       |      | /btrfs/@home/test/Downloads |       |      |          |
| 8bdb6d5c | 2024-05-22 14:40:11 | archlinux | label |      | /btrfs/@home/test/Desktop   |   415 |  268 | 13.7 GiB |
|          |                     |           |       |      | /btrfs/@home/test/Documents |       |      |          |
|          |                     |           |       |      | /btrfs/@home/test/Downloads |       |      |          |
| 766e74ca | 2024-05-22 14:44:18 | archlinux | label |      | /btrfs/@home/test/Desktop   |   495 |  268 | 16.5 GiB |
|          |                     |           |       |      | /btrfs/@home/test/Documents |       |      |          |
|          |                     |           |       |      | /btrfs/@home/test/Downloads |       |      |          |
| 1c42e574 | 2024-05-22 14:44:50 | archlinux | label |      | /btrfs/@home/test/Desktop   |   576 |  267 | 19.3 GiB |
|          |                     |           |       |      | /btrfs/@home/test/Documents |       |      |          |
|          |                     |           |       |      | /btrfs/@home/test/Downloads |       |      |          |
| b2381bfb | 2024-05-22 14:46:25 | archlinux | label |      | /btrfs/@home/test/Desktop   |   658 |  267 | 22.2 GiB |
|          |                     |           |       |      | /btrfs/@home/test/Documents |       |      |          |
|          |                     |           |       |      | /btrfs/@home/test/Downloads |       |      |          |
6 snapshot(s)

Each snapshot has 3 different sources: Desktop, Documents and Downloads , that is right.
Some of them should have blank Documents.

Issue:

I checked to run ls the latest snapshot b2381bfb:

$ rustic ls b2381bfb:btrfs/@home/test/ shows:

[INFO] using config /etc/rustic/rustic.toml
[INFO] repository local:/home/test/Desktop/rustic: password is correct.
[INFO] using cache at /home/test/.cache/rustic/e1787b1f622aa5a803b70a94e11175bb8f004ed3d01f4be97a4e976de018476a
[00:00:00] reading index...               ████████████████████████████████████████          8/8                                                                                                                                                                                            [00:00:00] getting snapshot...            ████████████████████████████████████████          0                                                                                                                                                                                              [INFO] getting snapshot...
"Desktop" 
"Downloads"

But where is the empty directory "Documents"?

I checked restic to mount this snapshot, but there is no "Documents" in this snapshot.

I checked some older snapshots they have no "Documents".

Config:

Here is a config for test on VM.

[global]
log-level = "info"

[repository]
repository = "/home/test/Desktop/rustic"
password = "test"

[forget]
keep-last = 10

[backup]
git-ignore = true # Respect all gitignore files anywhere
label = "label" 
glob = [
  "!.snapshots/",
  "!.cache/",
]

[[backup.sources]]
source = "/btrfs/@home/test/Desktop /btrfs/@home/test/Documents /btrfs/@home/test/Downloads"

The text was updated successfully, but these errors were encountered:

Zesko · 2024-05-22T14:41:14Z

I forgot to say that this is a circle backup problem, because:

[repository]
repository = "/home/test/Desktop/rustic"
...

[[backup.sources]]
source = "/btrfs/@home/test/Desktop ...

The Rustic repository is backed up in this same repository.

I should prevent the circle backup problem, but I have another problem:

When I moved the repository /home/test/Desktop/rustic to /rustic and tried to run backup, the backup fails and shows the error message:

[INFO] using config /etc/rustic/rustic.toml
[INFO] repository local:/rustic: password is correct.
[INFO] using cache at /home/test/.cache/rustic/e1787b1f622aa5a803b70a94e11175bb8f004ed3d01f4be97a4e976de018476a
[00:00:00] reading index...               ████████████████████████████████████████          2/2 
[WARN] error sanitizing source="/btrfs/@home/test/Desktop /btrfs/@home/test/Documents /btrfs/@home/test/Downloads" in config file
[INFO] using all backup sources from config file.

aawsome · 2024-05-22T18:41:28Z

@Zesko Thanks for opening this issue! Seems that Documents is not even backuped. Can you run rustic backup --log-level debug to see what is actually backed up?

Zesko · 2024-05-22T19:09:18Z

That does not work after moving the repository /home/test/Desktop/rustic to /rustic
$ rustic backup --log-level debug

[INFO] using config /etc/rustic/rustic.toml
[INFO] repository local:/rustic: password is correct.
[INFO] using cache at /home/test/.cache/rustic/e1787b1f622aa5a803b70a94e11175bb8f004ed3d01f4be97a4e976de018476a
[00:00:00] reading index...               ████████████████████████████████████████          2/2                                                                                                                                                                                          
[WARN] error sanitizing source="/btrfs/@home/test/Desktop /btrfs/@home/test/Documents /btrfs/@home/test/Downloads" in config file
[INFO] using all backup sources from config file.

Edit: Wait, I notice the VM storage is 97% full

Zesko · 2024-05-22T19:24:23Z

I tried to delete all snapshots, but the repository size is still 24 GB. The VM storage (35 GB) is still 97% full after deleting all snapshots.

rustic snapshots --log-level debug

[INFO] using config /etc/rustic/rustic.toml
[INFO] repository local:/rustic: password is correct.
[INFO] using cache at /home/test/.cache/rustic/e1787b1f622aa5a803b70a94e11175bb8f004ed3d01f4be97a4e976de018476a

total: 0 snapshot(s)

Check a size of the repository du -sh /rustic , the output:

24G     /rustic

PS: Btrfs does not have any snapshot in VM.

aawsome · 2024-05-22T19:27:43Z

If you want to remove all snapshots, it is easier to remove the repo and re-create it. In general, removing snapshots only frees space if you run a prune afterwards.

Zesko · 2024-05-22T19:31:13Z

In general, removing snapshots only frees space if you run a prune afterwards.

I did.

$ rustic prune

[INFO] using config /etc/rustic/rustic.toml
[INFO] repository local:/rustic: password is correct.
[INFO] using cache at /home/test/.cache/rustic/e1787b1f622aa5a803b70a94e11175bb8f004ed3d01f4be97a4e976de018476a
[00:00:00] reading index...               ████████████████████████████████████████          1/1                                                                                                                                                                                            [00:00:00] reading snapshots...           ████████████████████████████████████████          0/0                                                                                                                                                                                            [00:00:00] finding used blobs...          ████████████████████████████████████████          0/0                                                                                                                                                                                            [00:00:00] getting packs from repository...                                                                                                                                                                                                                                                [DEBUG] (1) rustic_rs::commands::prune: used:            0 blobs,        0 B
[DEBUG] (1) rustic_rs::commands::prune: unused:          0 blobs,        0 B
[DEBUG] (1) rustic_rs::commands::prune: total:           0 blobs,        0 B
to repack:          0 packs,          0 blobs,        0 B
this removes:                         0 blobs,        0 B
to delete:          0 packs,          0 blobs,        0 B
unindexed:          0 packs,         ?? blobs,        0 B
total prune:                          0 blobs,        0 B
remaining:                            0 blobs,        0 B
unused size after prune:        0 B (NaN% of remaining size)

packs marked for deletion:        689,   23.9 GiB
 - complete deletion:               0,        0 B
 - keep marked:                   689,   23.9 GiB
 - recover:                         0,        0 B
[DEBUG] (1) rustic_rs::commands::prune: index files to rebuild: 0 / 1
[INFO] nothing to do!

$ du -sh /rustic

24G     /rustic

aawsome · 2024-05-22T19:34:28Z

Yes, prune does a two-phase pruning by default which allows parallel operations on a repository. Therefore all data is just marked for removal. Either you wait 23h and re-run, or you re-run using --keep-delete to a small value like some seconds, or directly --instant-delete.

Zesko · 2024-05-22T19:35:54Z

Ah, I see, thanks.

Zesko · 2024-05-22T19:44:14Z

The same error

$ rustic backup --log-level debug

[INFO] using config /etc/rustic/rustic.toml
[INFO] repository local:/rustic: password is correct.
[INFO] using cache at /home/test/.cache/rustic/e1787b1f622aa5a803b70a94e11175bb8f004ed3d01f4be97a4e976de018476a
[00:00:00] reading index...               ████████████████████████████████████████          0/0   
[WARN] error sanitizing source="/btrfs/@home/test/Desktop /btrfs/@home/test/Documents /btrfs/@home/test/Downloads" in config file
[INFO] using all backup sources from config file.

How can I fix this error without re-creating a new repo?

aawsome · 2024-05-22T19:47:32Z

This error comes when the paths can't be sanitized because they (some?) don't exist.
Can you please run rustic backup --log-level debug by manually giving the paths? I think the most important thing is first to check why you snapshot misses some dir, or am I interpreting something wrong?

Zesko · 2024-05-22T19:53:46Z

Ah, your are right. I was blind that /btrfs is not mounted after reboot in VM.
$ ls -al /btrfs

total 0
drwxr-xr-x 1 root root   0 22. Mai 14:01 ./
drwxr-xr-x 1 root root 164 22. Mai 21:42 ../

I will fix fstab

Zesko · 2024-05-22T19:58:44Z

Can you please run rustic backup --log-level debug by manually giving the paths?

Of course:

$ rustic backup --log-level debug

[INFO] using config /etc/rustic/rustic.toml
[INFO] repository local:/home/test/rustic: password is correct.
[INFO] using cache at /home/test/.cache/rustic/e1787b1f622aa5a803b70a94e11175bb8f004ed3d01f4be97a4e976de018476a
[00:00:00] reading index...               ████████████████████████████████████████          0/0                                                                                                                                                                                            [INFO] using all backup sources from config file.
[INFO] merging source="/btrfs/@home/test/Desktop","/btrfs/@home/test/Documents","/btrfs/@home/test/Downloads" section from config file
[00:00:00] getting latest snapshot...     ████████████████████████████████████████          0/0                                                                                                                                                                                            [INFO] using no parent
[INFO] starting to backup "/btrfs/@home/test/Desktop","/btrfs/@home/test/Documents","/btrfs/@home/test/Downloads"...
[00:00:00] backing up...                  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░       50 B/1.78 KiB   12.56 KiB/s  (ETA 0s)                                                                                                                                                             [DEBUG] (1) rustic_core::archiver::tree_archiver: new       file: "/btrfs/@home/test/Desktop/.directory"
[00:00:00] backing up...                  ████████████████████████████████████████   1.78 KiB/1.78 KiB   25.88 KiB/s  (ETA 0s)                                                                                                                                                             [DEBUG] (1) rustic_core::archiver::tree_archiver: new       file: "/btrfs/@home/test/Desktop/text.log"
[DEBUG] (1) rustic_core::archiver::tree_archiver: new       tree: "/btrfs/@home/test/Desktop" 740 B
[DEBUG] (1) rustic_core::archiver::tree_archiver: new       tree: "/btrfs/@home/test/Downloads" 13 B
[DEBUG] (1) rustic_core::archiver::tree_archiver: new       tree: "/btrfs/@home/test" 507 B
[DEBUG] (1) rustic_core::archiver::tree_archiver: new       tree: "/btrfs/@home" 144 B
[DEBUG] (1) rustic_core::archiver::tree_archiver: new       tree: "/btrfs" 145 B
[DEBUG] (1) rustic_core::archiver::tree_archiver: new       tree: "" 145 B
[00:00:00] backing up...                  ████████████████████████████████████████   1.78 KiB/1.78 KiB   48.75 KiB/s  (ETA 0s)                                                                                                                                                             Files:       2 new, 0 changed, 0 unchanged
Dirs:        6 new, 0 changed, 0 unchanged
[DEBUG] (1) rustic_rs::commands::backup: Data Blobs:  2 new
[DEBUG] (1) rustic_rs::commands::backup: Tree Blobs:  6 new
Added to the repo: 1.6 kiB (raw: 3.4 kiB)
processed 2 files, 1.8 kiB
snapshot b56cf6c6 successfully saved.
[INFO] backup of "/btrfs/@home/test/Desktop","/btrfs/@home/test/Documents","/btrfs/@home/test/Downloads" done.

$ rustic ls b56cf6c6:btrfs/@home/test/

[INFO] using config /etc/rustic/rustic.toml
[INFO] repository local:/home/test/rustic: password is correct.
[INFO] using cache at /home/test/.cache/rustic/e1787b1f622aa5a803b70a94e11175bb8f004ed3d01f4be97a4e976de018476a
[00:00:00] reading index...               ████████████████████████████████████████          1/1                                                                                                                                                                                            [00:00:00] getting snapshot...            ████████████████████████████████████████          0                                                                                                                                                                                              [INFO] getting snapshot...
"Desktop"
"Downloads"

The Documents is missing.

$ rustic snapshots --log-level debug

[INFO] using config /etc/rustic/rustic.toml
[INFO] repository local:/home/test/rustic: password is correct.
[INFO] using cache at /home/test/.cache/rustic/e1787b1f622aa5a803b70a94e11175bb8f004ed3d01f4be97a4e976de018476a

snapshots for (host [archlinux], label [label], paths [/btrfs/@home/test/Desktop,/btrfs/@home/test/Documents,/btrfs/@home/test/Downloads])
| ID       | Time                | Host      | Label | Tags | Paths                       | Files | Dirs |    Size |
|----------|---------------------|-----------|-------|------|-----------------------------|-------|------|---------|
| b56cf6c6 | 2024-05-22 21:56:26 | archlinux | label |      | /btrfs/@home/test/Desktop   |     2 |    6 | 1.8 kiB |
|          |                     |           |       |      | /btrfs/@home/test/Documents |       |      |         |
|          |                     |           |       |      | /btrfs/@home/test/Downloads |       |      |         |
1 snapshot(s)

total: 1 snapshot(s)

the list of Desktop, Documents and Downloads: tree ~/Desktop/ ~/Documents/ ~/Downloads/

/home/test/Desktop/
└── text.log
/home/test/Documents/
/home/test/Downloads/

aawsome · 2024-05-23T06:58:07Z

Ok, I tried to reproduce the issue and can confirm that an empty dir (as part of the source) is simply not backuped. There might be some exceptions when using multiple dirs, but just running a backup of Desktop should give you the same result.

github-actions bot added the S-triage Status: Waiting for a maintainer to triage this issue/PR label May 22, 2024

aawsome changed the title ~~The option ls shows missing directory if that directory is empty~~ Empty dir as source is not backuped May 23, 2024

aawsome added C-bug Category: Something isn't working as expected A-core Area: Generally related to `rustic_core` and removed S-triage Status: Waiting for a maintainer to triage this issue/PR labels May 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Empty dir as source is not backuped #1157

Empty dir as source is not backuped #1157

Zesko commented May 22, 2024

Zesko commented May 22, 2024 •

edited

Loading

aawsome commented May 22, 2024

Zesko commented May 22, 2024 •

edited

Loading

Zesko commented May 22, 2024 •

edited

Loading

aawsome commented May 22, 2024

Zesko commented May 22, 2024

aawsome commented May 22, 2024

Zesko commented May 22, 2024

Zesko commented May 22, 2024

aawsome commented May 22, 2024

Zesko commented May 22, 2024 •

edited

Loading

Zesko commented May 22, 2024 •

edited

Loading

aawsome commented May 23, 2024

Empty dir as source is not backuped #1157

Empty dir as source is not backuped #1157

Comments

Zesko commented May 22, 2024

Issue:

Config:

Zesko commented May 22, 2024 • edited Loading

aawsome commented May 22, 2024

Zesko commented May 22, 2024 • edited Loading

Zesko commented May 22, 2024 • edited Loading

aawsome commented May 22, 2024

Zesko commented May 22, 2024

aawsome commented May 22, 2024

Zesko commented May 22, 2024

Zesko commented May 22, 2024

aawsome commented May 22, 2024

Zesko commented May 22, 2024 • edited Loading

Zesko commented May 22, 2024 • edited Loading

aawsome commented May 23, 2024

Zesko commented May 22, 2024 •

edited

Loading

Zesko commented May 22, 2024 •

edited

Loading

Zesko commented May 22, 2024 •

edited

Loading

Zesko commented May 22, 2024 •

edited

Loading

Zesko commented May 22, 2024 •

edited

Loading