Automatic grub-update only works on snapshots in parent folder found grub-btrfs.path

[FGD-Garuda]

Hi there,

I migrated from Kubuntu to Arch lately (Garuda Linux) so my username has not been wisely chosen. loll

Garuda installs by default Timeshift and uses a modified path in grub-btrfs.path.
I disabled Timeshift and edited the path for the standard /.snapshots so my grub-btrfs.path is a direct copy of the one found in github.

I did reboot twice since that time.

I noticed grub-btrfs.cfg only updates automatically when I create a snapshot in the parent directory, which is /.snapshots.
When snapshots are created in sub-dirs, like /.snapshots/ad-hoc, grub-btrfs.cfg does not update.

However if I run grubup (grub-mkconfig alias) no matter where the snapshots are located it does regenerate grub-btrfs.cfg.

journalctl -ru grub-btrfs.service is kinda strange, I have a umount error, was there before I rebooted but it wasn't there 2-3 days ago or I just never noticed. More on that below.

Feb 01 21:34:59 Garuda systemd[1]: Failed to start Regenerate grub-btrfs.cfg.
Feb 01 21:34:59 Garuda systemd[1]: grub-btrfs.service: Failed with result 'exit-code'.
Feb 01 21:34:59 Garuda systemd[1]: grub-btrfs.service: Main process exited, code=exited, status=32/n/a
Feb 01 21:34:59 Garuda bash[184617]: umount: /tmp/tmp.EMDs3U1ogX: target is busy.
Feb 01 21:34:59 Garuda bash[184320]: Found snapshot: 2021-01-27 20:29:55 | @snapshots/sync/SSD1/@.old
Feb 01 21:34:59 Garuda bash[184320]: Found snapshot: 2021-01-29 19:11:49 | @snapshots/daily/root2021-01-29_19H11
Feb 01 21:34:59 Garuda bash[184320]: Found snapshot: 2021-01-29 19:12:33 | @snapshots/daily/root2021-01-29_19H12
Feb 01 21:34:59 Garuda bash[184320]: Found snapshot: 2021-01-29 19:13:20 | @snapshots/daily/root2021-01-29_19H13
Feb 01 21:34:59 Garuda bash[184320]: Found snapshot: 2021-01-29 19:14:49 | @snapshots/daily/root2021-01-29_19H14
Feb 01 21:34:59 Garuda bash[184320]: Found snapshot: 2021-01-30 19:05:01 | @snapshots/daily/root2021-01-30_19H05
Feb 01 21:34:59 Garuda bash[184320]: Found snapshot: 2021-01-31 18:11:13 | @snapshots/ad-hoc/root_upd_2021-01-31_18H11
Feb 01 21:34:59 Garuda bash[184320]: Found snapshot: 2021-01-31 18:12:50 | @snapshots/ad-hoc/@.b4_custom_kernel_2021-01-31_18H12
Feb 01 21:34:59 Garuda bash[184320]: Found snapshot: 2021-01-31 19:05:01 | @snapshots/daily/root2021-01-31_19H05
Feb 01 21:34:59 Garuda bash[184320]: Found snapshot: 2021-02-01 15:36:18 | @snapshots/autosnap/root2021-02-01_15H36
Feb 01 21:34:59 Garuda bash[184320]: Found snapshot: 2021-02-01 15:48:13 | @snapshots/autosnap/root2021-02-01_15H48
Feb 01 21:34:59 Garuda bash[184320]: Found snapshot: 2021-02-01 15:49:21 | @snapshots/autosnap/root2021-02-01_15H49
Feb 01 21:34:59 Garuda bash[184320]: Found snapshot: 2021-02-01 19:05:01 | @snapshots/daily/root2021-02-01_19H05
Feb 01 21:34:59 Garuda bash[184320]: Found snapshot: 2021-02-01 20:41:43 | @snapshots/autosnap/root2021-02-01_20H41
Feb 01 21:34:59 Garuda bash[184320]: Found snapshot: 2021-02-01 20:51:52 | @snapshots/autosnap/root2021-02-01_20H51
Feb 01 21:34:59 Garuda bash[184320]: Found snapshot: 2021-02-01 21:34:56 | @snapshots/@test_2021-02-01_21H34
Feb 01 21:34:59 Garuda bash[184320]: Info: Separate boot partition not detected
Feb 01 21:34:58 Garuda bash[184320]: Detecting snapshots ...
Feb 01 21:34:58 Garuda systemd[1]: Starting Regenerate grub-btrfs.cfg...
Feb 01 21:34:57 Garuda systemd[1]: Failed to start Regenerate grub-btrfs.cfg.
Feb 01 21:34:57 Garuda systemd[1]: grub-btrfs.service: Failed with result 'exit-code'.
Feb 01 21:34:57 Garuda systemd[1]: grub-btrfs.service: Main process exited, code=exited, status=32/n/a
Feb 01 21:34:57 Garuda bash[184142]: umount: /tmp/tmp.Dl4QKHZQSb: target is busy.
Feb 01 21:34:56 Garuda bash[183845]: Found snapshot: 2021-01-27 20:29:55 | @snapshots/sync/SSD1/@.old
Feb 01 21:34:56 Garuda bash[183845]: Found snapshot: 2021-01-29 19:11:49 | @snapshots/daily/root2021-01-29_19H11
Feb 01 21:34:56 Garuda bash[183845]: Found snapshot: 2021-01-29 19:12:33 | @snapshots/daily/root2021-01-29_19H12
Feb 01 21:34:56 Garuda bash[183845]: Found snapshot: 2021-01-29 19:13:20 | @snapshots/daily/root2021-01-29_19H13
Feb 01 21:34:56 Garuda bash[183845]: Found snapshot: 2021-01-29 19:14:49 | @snapshots/daily/root2021-01-29_19H14
Feb 01 21:34:56 Garuda bash[183845]: Found snapshot: 2021-01-30 19:05:01 | @snapshots/daily/root2021-01-30_19H05
Feb 01 21:34:56 Garuda bash[183845]: Found snapshot: 2021-01-31 18:11:13 | @snapshots/ad-hoc/root_upd_2021-01-31_18H11
Feb 01 21:34:56 Garuda bash[183845]: Found snapshot: 2021-01-31 18:12:50 | @snapshots/ad-hoc/@.b4_custom_kernel_2021-01-31_18H12
Feb 01 21:34:56 Garuda bash[183845]: Found snapshot: 2021-01-31 19:05:01 | @snapshots/daily/root2021-01-31_19H05
Feb 01 21:34:56 Garuda bash[183845]: Found snapshot: 2021-02-01 15:36:18 | @snapshots/autosnap/root2021-02-01_15H36
Feb 01 21:34:56 Garuda bash[183845]: Found snapshot: 2021-02-01 15:48:13 | @snapshots/autosnap/root2021-02-01_15H48
Feb 01 21:34:56 Garuda bash[183845]: Found snapshot: 2021-02-01 15:49:21 | @snapshots/autosnap/root2021-02-01_15H49
Feb 01 21:34:56 Garuda bash[183845]: Found snapshot: 2021-02-01 19:05:01 | @snapshots/daily/root2021-02-01_19H05
Feb 01 21:34:56 Garuda bash[183845]: Found snapshot: 2021-02-01 20:41:43 | @snapshots/autosnap/root2021-02-01_20H41
Feb 01 21:34:56 Garuda bash[183845]: Found snapshot: 2021-02-01 20:51:52 | @snapshots/autosnap/root2021-02-01_20H51
Feb 01 21:34:56 Garuda bash[183845]: Found snapshot: 2021-02-01 21:34:56 | @snapshots/@test_2021-02-01_21H34
Feb 01 21:34:56 Garuda bash[183845]: Info: Separate boot partition not detected
Feb 01 21:34:56 Garuda bash[183845]: Detecting snapshots ...
Feb 01 21:34:56 Garuda systemd[1]: Starting Regenerate grub-btrfs.cfg...

It says it's failing but it works, grub-btrfs.cfg is updated with all the snapshots' data. The one I tested here that was new is @snapshots/@test_2021-02-01_21H34

About that strange /tmp umount target busy, I think it's related to the ZRAM swap Garuda is using by default.

TARGET                   SOURCE                      FSTYPE     OPTIONS
/proc                    proc                        proc       rw,nosuid,nodev,noexec,relatime
/sys                     sys                         sysfs      rw,nosuid,nodev,noexec,relatime
/dev                     dev                         devtmpfs   rw,nosuid,relatime,size=8143212k,nr_inodes=2035803,mode=755,inode64
/run                     run                         tmpfs      rw,nosuid,nodev,relatime,mode=755,inode64
/dev/pts                 devpts                      devpts     rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000
/                        /dev/nvme0n1p1[/@]          btrfs      rw,noatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=256,subvol=/@
/sys/kernel/security     securityfs                  securityfs rw,nosuid,nodev,noexec,relatime
/dev/shm                 tmpfs                       tmpfs      rw,nosuid,nodev,inode64
/sys/fs/cgroup           cgroup2                     cgroup2    rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot
/sys/fs/pstore           pstore                      pstore     rw,nosuid,nodev,noexec,relatime
/sys/fs/bpf              none                        bpf        rw,nosuid,nodev,noexec,relatime,mode=700
/proc/sys/fs/binfmt_misc systemd-1                   autofs     rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=20807
/dev/mqueue              mqueue                      mqueue     rw,nosuid,nodev,noexec,relatime
/dev/hugepages           hugetlbfs                   hugetlbfs  rw,relatime,pagesize=2M
/sys/kernel/debug        debugfs                     debugfs    rw,nosuid,nodev,noexec,relatime
/sys/kernel/tracing      tracefs                     tracefs    rw,nosuid,nodev,noexec,relatime
/tmp                     tmpfs                       tmpfs      rw,nosuid,nodev,size=8199908k,nr_inodes=409600,inode64
/sys/kernel/config       configfs                    configfs   rw,nosuid,nodev,noexec,relatime
/sys/fs/fuse/connections fusectl                     fusectl    rw,nosuid,nodev,noexec,relatime
/.snapshots              /dev/nvme0n1p1[/@snapshots] btrfs      rw,noatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=280,subvol=/@snapshots
/home                    /dev/nvme0n1p1[/@home]      btrfs      rw,noatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=257,subvol=/@home
/root                    /dev/nvme0n1p1[/@root]      btrfs      rw,noatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=258,subvol=/@root
/srv                     /dev/nvme0n1p1[/@srv]       btrfs      rw,noatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=259,subvol=/@srv
/var/cache               /dev/nvme0n1p1[/@cache]     btrfs      rw,noatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=260,subvol=/@cache
/var/log                 /dev/nvme0n1p1[/@log]       btrfs      rw,noatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=261,subvol=/@log
/var/tmp                 /dev/nvme0n1p1[/@tmp]       btrfs      rw,noatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=262,subvol=/@tmp
/home/guest              tmpfs                       tmpfs      rw,nosuid,nodev,mode=750,uid=1001,gid=985,inode64
/xfs                     /dev/nvme0n1p2              xfs        rw,noatime,nodiratime,attr2,inode64,allocsize=131072k,logbufs=8,logbsize=256k,noquota
/run/user/1000           tmpfs                       tmpfs      rw,nosuid,nodev,relatime,size=1639980k,nr_inodes=409995,mode=700,uid=1000,gid=1000,inode64
/tmp/tmp.HAJHMZspNC      /dev/nvme0n1p1              btrfs      rw,relatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=5,subvol=/
/tmp/tmp.hm7TH1mb6S      /dev/nvme0n1p1              btrfs      rw,relatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=5,subvol=/
/tmp/tmp.XMNeE0I7oG      /dev/nvme0n1p1              btrfs      rw,relatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=5,subvol=/
/tmp/tmp.Dl4QKHZQSb      /dev/nvme0n1p1              btrfs      rw,relatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=5,subvol=/
/tmp/tmp.EMDs3U1ogX      /dev/nvme0n1p1              btrfs      rw,relatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=5,subvol=/
/tmp/tmp.m1U6vMTHQ0      /dev/nvme0n1p1              btrfs      rw,relatime,compress=zstd:3,ssd,space_cache,autodefrag,subvolid=5,subvol=/

systemctl tells me grub-btrfs is active:

● grub-btrfs.path - Monitors for new snapshots
     Loaded: loaded (/usr/lib/systemd/system/grub-btrfs.path; enabled; vendor preset: disabled)
    Drop-In: /etc/systemd/system/grub-btrfs.path.d
             └─override.conf
     Active: active (waiting) since Mon 2021-02-01 21:06:36 EST; 35min ago
   Triggers: ● grub-btrfs.service

Feb 01 21:06:36 Garuda systemd[1]: Started Monitors for new snapshots.

Not sure where to go from there...

[Antynea]

Hello,

I noticed grub-btrfs.cfg only updates automatically when I create a snapshot in the parent directory, which is /.snapshots.
When snapshots are created in sub-dirs, like /.snapshots/ad-hoc, grub-btrfs.cfg does not update.

This is the expected behavior of systemd.path.
Systemd.path isn't recursive, it's a weakness.
I know inotifywait, offers recursion, maybe abandoned systemd.path to use inotifywait instead.
Work is underway to improve the grub-btrfs.path unit #139

However if I run grubup (grub-mkconfig alias) no matter where the snapshots are located it does regenerate grub-btrfs.cfg.

When you invoke grub-mkconfig, all the files in /etc/grub.d/ are executed.
Unlike systemd.path, grub-btrfs uses the btrfs sub list -s / command to detect existing snapshots.
Nothing weird, this is the expected behavior.

journalctl -ru grub-btrfs.service is kinda strange, I have a umount error, was there before I rebooted but it wasn't there 2-3 days ago or I just never noticed

This is very strange, and needs to be fixed.
Do you have a process open for writing at this time on the mount point?

I'm thinking about mounting the root of btrfs as read-only, this makes sense because grub-btrfs doesn't write to your root filesystem, it just looks for information.

[FGD-Garuda]

Oh ok! Interesting, I don't recall having that recursive folder issue when I was with Kubuntu.

I guess it's not possible to include more than 1 folder in grub-btrfs.path? :( That would be way too easy, right. loll

As for my /tmp issue it's probably related to somekind of crap-up from DKMS from an update on Sunday. I have to investigate or reinstall worst case. At least it's not related to grub-btrfs meaning it does not affect it from detecting my snapshots.

Now I have to change my userid... let's see if I can.

[FGD-Garuda]

@Antynea , actually it seems many people are having that issue of / being mounted to /tmp/tmp.*
This happened since a recent update of grub-btrfs. We are trying to downgrade to a previous version of grub-btrfs and see if that changes anything. Still a very fresh issue opened on Garuda forums but one person fixes the issue by downgrading grub-btrfs.

Is there a conflict with some specific packages somewhere I don't know, but many people on the Garuda forum are having the same issue.

I'm not sure how to check if I have a process opened for writing at the time of mount point. If there is, then it's most probably the same for us using Garuda.

[FGD-Garuda]

Confirmed.

I downgraded from 4.8.1.1 to 4.7.2.1 and sudo update-grub now works.

╭─frank at Garuda in ⌁
╰─λ sudo update-grub
Generating grub configuration file ...
Found theme: /usr/share/grub/themes/garuda/theme.txt
Found linux image: /boot/vmlinuz-linux-tkg-bmq
Found initrd image: /boot/amd-ucode.img /boot/initramfs-linux-tkg-bmq.img
Found fallback initrd image(s) in /boot: initramfs-linux-tkg-bmq-fallback.img
Detecting snapshots ...
Info: Separate boot partition not detected
Found snapshot: 2021-02-02 09:52:34 | @snapshots/ad-hoc/@.nvidia
Found snapshot: 2021-01-27 20:28:05 | @snapshots/ad-hoc/@.b4_bkp_2021-01-27-20H28
Found snapshot: 2021-01-24 20:42:25 | @snapshots/ad-hoc/@.locale_2021-01-24_20H41
Found snapshot: 2021-01-16 15:52:05 | @.bkp
Found 4 snapshot(s)
done

Something is conflicting (or bugged?) with up to date Arch/Garuda packages when using grub-btrfs 4.8.1.1.

I guess we should split this issue into another one of its own and close the one regarding non-recursive path. What do you think?

[Antynea]

I guess it's not possible to include more than 1 folder in grub-btrfs.path? :( That would be way too easy, right. loll

Sure, just added a another PathModified=/.snapshots/whatever/ line in grub-btrfs.path
systemctl edit grub-btrfs.path

[Path]
PathModified=/.snapshots/wathever

You can check if systemd has supported the additional parameter by :
systemctl show grub-btrfs.path

...
Paths=/.snapshots/wathever (PathModified)
Paths=/.snapshots (PathModified)
...

I'm not sure how to check if I have a process opened for writing at the time of mount point.

Use the lsof command:
lsof /tmp
Files opened will be indicated in the FD column, so
lsof /tmp | awk '$4 ~ /[0-9].*[r|w|u]/'
This will list the processes that use mount point (read/write/both)

guess we should split this issue into another one of its own and close the one regarding non-recursive path. What do you think?

Yes, please.

[FGD-Garuda]

Ok thank you I'll do that for grub-btgrfs.path, I didn't think it could be that easy. loll

I am splitting to new issue for the /tmp thing. Closing this one.