Consider changing the BTRFS subvolume layout

[KhalilSantana]

Hello,

I'm very happy to see that v2.3.X includes BTRFS subvolume support as a beta feature! This was a major blocker for me and now it's (mostly) gone.

However, I would like to comment on the current subvolume layout used by Archinstall's guided mode, and why (and how) I think it could be improved.

The Current Layout

For reference this is the current layout used by guided mode as of 2021-12-1 using archinstall v2.3.0:

# btrfs subvol list -t /
ID	gen	top level	path	
--	---	---------	----	
256	21	5		home
258	21	5		var/log
259	17	5		var/cache/pacman/pkg
260	10	5		.snapshots
264	14	5		var/lib/portables
265	15	5		var/lib/machines

How BTRFS handles subvolumes, and snapshots

(this section exists mostly to add context, skip it if you are familiar with BTRFS)

In BTRFS, each subvolume has an ID and a path, relative to the toplevel (a.k.a subvolid=5), these subvolumes can be mounted anywhere, and can be used for many things just like folders, however, the main differentiator between a regular folder and a subvol is that the later has special powers: atomic snapshots, etc.

Snapshots are just a clone of that filesystem root (subvol) at that time, and it shares extents with the snapped subvol, this is one of reasons snaps are so useful: they are fast and small.

However, there are some important quirks/behaviors that BTRFS currently does:

• Rule 0: You cannot replace the toplevel (subvolid=5, subvol=/) with another subvol.
• Rule 1: btrfs subvol snap won't recurse into nested subvolumes. So you cannot create atomic snaps of nested subvolumes and their parents at once.
• Rule 2: As btrfs subvol snap doesn't recurse, the resulting subvolume (the snapshot) won't contain ANY of it's nested subvols.

With this in mind, I hope the next section becomes easy to understand.

What's wrong with the current layout?

The current layout basically prohibits using snapshots as a rollback feature, why?

• Per Rule 0, you cannot rollback (without lots of jiggling mv's and rm's) the root filesystem as it's been mounted at the toplevel subvol.
• Per Rule 1 and 2, even if you manually rollback using methods, like using the (often confusing) btrfs sub set-default, your homedir will be empty! You'd need to manually snap RW the old home subvol into the right place.

In short the main issues with the current subvol layout are:

• The root filesystem using the toplevel subvolume directly.
• The home dir0 being a nested subvolume.

What would be a better layout?

• Reserve the toplevel subvolume for managing other subvolumes
• Utilize a subvolume as the root filesystem
• Utilize a non-nested subvolume for the home dir.

This is often called the "flat-layout", often used with the ubuntu-style naming scheme (@ as /, @home as /home), and it's documented on the wiki

What other distros are using?

• Ubuntu's installer uses the flat layout, with @ and @home, mounted at / and /home, respectively
• Manjaro's installer uses the flat layout, identical to the Ubuntu one.
• Fedora's Installer uses the flat layout, with a different naming scheme (root and home AFAIK)

My layout suggestion

Follow the flat layout, with the Ubuntu-style naming scheme, but keep the extra subvols (/var/log & /var/cache/pacman/pkg), those extras not nested, of course.

Basically, this:

# btrfs subvolume list -t /
[sudo] senha para khalil: 
ID      gen     top level       path
--      ---     ---------       ----
256     1408526 5               @
257     1408526 5               @home
258     1408526 5               @pkg
259     1408526 5               @log
260     1391176 5               @snapshots

In tree view for clarity's sake:

# exa -TL1 /mnt/toplevel # this is the mount point of subvolid=5 on my system, for maintance purposes)
/mnt/toplevel
├── @
├── @home
├── @pkg
├── @log
└── @snapshots

Why the Ubuntu naming scheme? Why not Fedora's? Or something else entirely?

Mostly because of Timeshift, as it only supports that standard. And while it might seem like a silly reason, I feel like it's reasonable: Timeshift is the one of the few snapshot utilities that has an incredibly easy to use restore button, which makes undoing changes incredibly easy, and also un-breaking a system after a bad upgrade.

---

Thank you for your time!

[Torxed]

Thank you for this very detailed feedback! Really awesome to see everyone suggesting good things to improve the btrfs subvolume support. We'll track it in this issue and in #718 where the topic came up during an issue of re-using btrfs subvolumes that was anything other than what we supply.

[wllacer]

@KhalilSantana Great explanation for what i also do empirically,

With a slightly different layout (i don't care -still- for snapshooting but more for usage flexibility. At the end of issue #718 you will find an unoffical patch that allows archinstall to work with flat layouts IF they are defined and mounted previously. I would be glad i could have some feedback about it. I could make a PR of it, but i'm not that sure of its quality (works for me, but this is what i can guarantee).
I want to expand it to full support of flat schemas. Any suggestion and ideas would be welcome

[Torxed]

I would be glad i could have some feedback about it.

It's coming in a bit : ) Gathering my thoughts to not just blurp out unfinished thoughts :)

[rexhent]

This is definitely important. Btrfs is a bit of a pain to remember how to use when doing the arch install "the old way", and one of my main reasons to use EndeavourOS right now.

[DrymarchonShaun]

Just to put it out there as well, why not go with opensuse's layout, so that snapper can be fully utilized? Timeshift works, but the last time I tested it, it left like it lacked alot that Snapper has; rolling back with snapper is as easy as booting a snapshot of your working system, and running snapper rollback when you are using the right subvolume layout.

From https://en.opensuse.org/SDB:BTRFS

/home
If /home does not reside on a separate partition, it is excluded to avoid data loss on rollbacks.

/opt
Third-party products usually get installed to /opt. It is excluded to avoid uninstalling these applications on rollbacks.

/root
The root users home directory should also be preserved during a rollback

/srv
Contains data for Web and FTP servers. It is excluded to avoid data loss on rollbacks.

/tmp
All directories containing temporary files and caches are excluded from snapshots.

/usr/local
This directory is used when manually installing software. It is excluded to avoid uninstalling these installations on rollbacks.

/var
This directory contains many variable files, including logs, temporary caches, third party products in /var/opt, and is the default location for many virtual machine images and databases. Therefore this subvolume is created to exclude all of this variable data from snapshots and is created with Copy-On-Write disabled.

What this ends up looking like (with the top-level subvolume mounted at /btrfs)

/Btrfs
└── /@
      ├── /home                       /home
      ├── /opt                        /op
      ├── /root                       /root
      ├── /srv                        /srv
      ├── /tmp                        /tmp
      ├── /usr_local                  /usr/local
      ├── /var                        /var      (cow disabled)
      └── /.snapshots                 /.snapshots
              └── /1/snapshot       /

Putting the root subvolume in .snapshots makes using snapshots much cleaner, as the original root subvolume is essentially a snapshot, so what happens after you boot into a read-only snapshot and run snapper rollback is 1. A read only snapshot of the current (broken) system is created, 2. a read write copy of the currently booted snapshot is created and set as the default btrfs partition, so that when you boot back up, it gets mounted as the new root subvolume.

It is more complicated to wrap your head around to begin with, but once you understand it, it makes way more sense.

If it helps make it understandable at all, I just finished fixing a (janky) script that installs arch with this layout (albeit with a lot of separation of /var) https://github.com/ShaunTheQuietGamer/Arch-Setup-Script/blob/main/install.sh
It might help explain the process to get this functioning, but I understand that there's probably a better way to do all the stuff its doing, and some of is important to get the system functional (like modifying fstab, and adding some stuff to the grub config,) a lot of it is not.

EDIT: I didn't fully understand how this whole thing worked, and still probably don't, but isn't it theoretically possible to add both layouts as an option for users? Instead of it asking Yes or No if they want a default subvolume layout, it could ask if they

1. Don't want subvolumes
2. Want the Ubuntu style flat layout
3. Want the opensuse layout with snapper support
4. Manual subvolume setup (like the disk partitioning section)

(Something else to look at is possibly adding an extra Y/N if the Opensuse layout is selected to also install and set up snapper, grub-btrfs, and snap-pac with snapper rollback functioning.)

[Torxed]

Before we close this, and you'll have to correct me if I'm wrong because there's almost too much context here so I might have gotten a bit lost in all the reading. But the desired outcome is:

/Btrfs
└── /@
            ├── /home                       /home
            ├── /opt                        /opt
            ├── /root                       /root
            ├── /srv                        /srv
            ├── /tmp                        /tmp
            ├── /usr_local                  /usr/local
            ├── /var                        /var      (cow disabled)
            └── /.snapshots                 /.snapshots
                      └── /1/snapshot       /

Which we currently can't support because we don't support a mountpoint (/ in this case) inside another subvolume, is this a correct assumption @wllacer (pinging you because you've done more work on the subvolumes than me at this point).

If we can live without mountpoints inside subvlumes, this issue is fulfilled. Otherwise I'll keep this open but I'll move it for v2.4.0 as I'd like to get v2.3.1 out ASAP for patch reasons.

[wllacer]

With the code just merged you can create almost any setup you can imagine (most of them dead wrong). Subvolume hierachy and File System hierachies can (and should be) totally distinct. The only things to take care are:

• A mount point must be either the partition or a subvolume, but not necesarily a first level one. (imagine a subvolume structure /snapshots, /snapshots/root /snapshots/yesterday .... you can mount /snapshots/root as /, f.i.
• It is dumb to replicate file system hierachy into subvolume one (unless there are specific btrfs reasons (nodatacow, compression, snapshots policy, ...). To create / and /home is dumb to make subv. @ and @/home (parent / child), but a good idea to make @ and @home (siblings, see there is no slash here).
  As a matter of fact, it might not even work. Using the still defaults at archinstall (fiercely copying FS hierachy at the subvolume level), results in only / being mounted

[KhalilSantana]

Commenting a little bit on @ShaunTheQuietGamer's layout:

/srv
Contains data for Web and FTP servers. It is excluded to avoid data loss on rollbacks.

I also do this on my system, but omitted from my original proposal for KISS principles, but it's a pretty sound suggestion in my opinion.

/opt
Third-party products usually get installed to /opt. It is excluded to avoid uninstalling these applications on rollbacks.
/usr/local
This directory is used when manually installing software. It is excluded to avoid uninstalling these installations on rollbacks.

While I can understand the point you are making, I also think this is somewhat of a foot gun: applications seldom exist within a vacuum, they have dependencies that may be scattered elsewhere on the filesystem, thus rolling back the rootfs without also rolling back these may not lead to a working (read: consistent) set of packages.

And yes, one could configure snapper to also snapshot this in sync with the rootfs subvol, then figure out a way to rollback from that (either manually, or via snapper itself somehow??), but that feels like overcomplicating things.

/tmp
All directories containing temporary files and caches are excluded from snapshots.

I'm not sure what openSUSE does with /tmp, but on ArchLinux, /tmp is a ramdisk (tempfs) by default, so I don't see the point on this being a subvol.

/var
This directory contains many variable files, including logs, temporary caches, third party products in /var/opt, and is the default location for many virtual machine images and databases. Therefore this subvolume is created to exclude all of this variable data from snapshots and is created with Copy-On-Write disabled.

I don't fully understand how Archlinux package manager (libalpm) works, but I believe it uses /var/lib/pacman/local to keep track on what's installed or not, thus having /var out of sync (ie: after a restore) from the rootfs would lead from pacman -Q disagreeing on what's on the rest of the filesystem (say, /usr/bin/firefoxbeing installed or not).

If my understanding is correct, splitting the entire /var into a subvol should be avoided, as it would break pacman each and every time you rollback.

--

Regarding CoW vs no-CoW I also disagree, nodatacow is basically an escape hatch, that disables or otherwise breaks many core features of BTRFS, including, but not limited to:

• Disables checksums
• Disables compression
• Scrub can't find corruption on nodatacow stuff as there are no checksums to verify
• Even trivial cases like a -mraid1 -draid1 cannot detect which disk has the newest or correct version of a nodatacow file, so there's no (good) way to "sync" an array with nodatacow files.
• and probably more

Those are very big downsides which should be decided on a per-application basis, either by the user or by the application itself, not a blanket disable on the entire /var directory.

[DrymarchonShaun]

(To Khalil) Your points make total sense, I see no issue with modifying the layout to work better with arch, every distro is different and is going to require different things. I guess I should specify/have specified that it was more that I think it's worth looking at opensuse's layout to see if anything they are doing would makes sense to add for the (or one of the) defaults for arch. I think the only thing that is absolutely required (subvolume wise) for snapper is for the original root be mounted in /@/.snapshots/1/snapshot, having .snapshots mounted at /@/.snapshots may also be required. Speaking as the one that's not having to do any of the work, (you or others may have a different opinion as the ones actually implemented this,) I would think is important to give choices. Rather than have the only default option be timeshift, give the choice between the two (or a totally manual layout, obviously.) Part of the problem with choosing one over the other is that, for snapper at least, you have to modify some configs during the installation to make it function correctly. (Setting an option to true in grub, modifying fstab to not force /@/.snapshots/1/snapshot to be mounted, and instead tell it to mount the default subvolume, etc.)

[DrymarchonShaun]

I really suck at explaining things, and snapper doesn't have nearly enough documentation on the required layout...
I'm going to try to make it a bit more understandable, and try to change it to match the points Kalil made.

Subvolumes	Filesystem mount point
/mnt/@/	N/A
/mnt/@/home	/home
/mnt/@/root	/root
/mnt/@/srv	/srv
/mnt/@/var_log	/var/log
/mnt/@/var_pkg	/var/cache/pacman/pkg
/mnt/@/.snapshots	/.snapshots
/mnt/@/.snapshots/1/snapshot	/

just for the sake of clarity from here down, / is the filesystem root, \ is the partition root that can be mounted somewhere within the filesystem root but doesn't have to be mounted. I've been using / for both and I think that's the main reason why there is some confusion
I'm not entirely sure why opensuse does \@/* instead of \@*, I think the idea is to distinguish the file system root / and the files in it from the partition root \@/ and the subvolumes in it. If I say \@/.snapshots I'm talking about the .snapshots subvolume, if I say /.snapshots I'm talking about the filesystem location, its to stop the confusion that is currently happening. Regardless, its what is required for snapper.

Actually, I think I might have an analogy. Think of each of the subvolumes as physical disks on a normal system within /dev/*. They are all in /dev so you know they are physical drives, just like the subvolumes are in \@/* so you know they are subvolumes, because technically you could create a text file in the partition root right next to '@', (although I imagine its bad practice,) and if it was all just in the partition root it could get confusing.

Anyways, the filesystem root is mounted as @/.snapshots/1/snapshots (1 is just a folder in the @/.snapshsots subvolume,) because it is essentially the first snapshot of the system. When you cd into @/.snapshots/1/snapshots its your current filesystem. The next snapshot that is created is @/.snapshots/2/snapshot/ and if you cd into it, its a copy, a snapshot, of a root filesystem.

[Torxed]

Subvolumes Filesystem mount point
/mnt/@/ N/A
/mnt/@/home /home
/mnt/@/root /root
/mnt/@/srv /srv
/mnt/@/var_log /var/log
/mnt/@/var_pkg /var/cache/pacman/pkg
/mnt/@/.snapshots /.snapshots
/mnt/@/.snapshots/1/snapshot /

This looks pretty similar to what we got with the exception of the root subvolume not being mounted and a separate @/root being created. And I think the current code can handle this one with a breeze. @wllacer will have to correct me if my assumptions here are wrong.

[wllacer]

Subvolumes Filesystem mount point
/mnt/@/ N/A
/mnt/@/home /home
/mnt/@/root /root
/mnt/@/srv /srv
/mnt/@/var_log /var/log
/mnt/@/var_pkg /var/cache/pacman/pkg
/mnt/@/.snapshots /.snapshots
/mnt/@/.snapshots/1/snapshot /

This looks pretty similar to what we got with the exception of the root subvolume not being mounted and a separate @/root being created. And I think the current code can handle this one with a breeze. @wllacer will have to correct me if my assumptions here are wrong.

Happy New Year to all !
@Torxed is right, this is a flat version of the current layout, and I don't dislike it for a basic layout (although I separate var as subvolume. Just a comment , both @var_log and @var_pkg don't need to be mounted IF there is no need from the OS side, just for backups and are created as embedded suvolumes f.i. if @var is created as a subvolume (which is interesting) (edited)
Default layouts is a tricky matter, because it depends on the end users need. A couple of samples

• I used to need a lot of databases, so for me it was CRITICAL to have the corresponding subdirectories as separated subvolumes (or at least with a nodatacow flag). Both mysql and PgSql use directories in var (can't remember for Oracle, DB2 and SQLServer, and sqlite can go anywhere) and the targets at btrfs have to be correctly defined BEFORE use, either marked subdirectories only, subvolumes or mounted subvolumes. Which depends on backup policies -the first two- or flexibility at the OS level (the third.)
• The same with VM images, this time aggravated by the fact that they are usually inside one user's home directory. (in this case I simply play with the nodatacow atrribute). Or for the KDE users, the directory where the baloo index file goes ...
• Another case can be made for a limited "distro hopping" inside the machine. With an adecuate use of subvolumes you can have, f.i. two or more totally different ArchLinux instances running almost isolated on the same machine, without disk partitioning (size limitations). BTW, researching this question is what sent me into the Archinstall rabbit hole.

The sad true is that most of us lack, for the time being, serious experience with Btrfs and the impact of alternatives. At least, since yesterday, when PR #787 was merged we can play the different alternatives on "real metal" (well, VMs) with easy - if you don't mind editing json files. If you read the comments there, we have tried to document what and how it can do it.

Let's play for a while before taking some decision (and try to involve in the discussion the more wide Archlinux communiy)

[Torxed]

and try to involve in the discussion the more wide Archlinux communiy

I think this might be a good idea.
This topic seems too complex for me to merge any alternative on my own.

[DrymarchonShaun]

Sounds good to me.
(everything after this is reliant on figuring out the non snapper/timeshift specific default subvolumes, (as in extra ones that may be necessary for pacman, etc)

I've kind of been creating a mental image of how I was thinking i'd set up the configuration menus for subvolumes and snapshots. I'm almost certainly getting ahead of myself (and everything else,) but I though I would but it out there regardless. This is once again coming from someone that doesn't know how to do the work, or the standards/rules being followed, so I'm sure changes/tweaks would be necessary, if you all think it fits into the scope of this installer at all.

First it would ask what layout the user wants

Select a subvolume layout
1. Ubuntu style layout (Compatible with Timeshift)
2. openSUSE style layout (Compatible with Snapper)
3. Manual configuration
4. None //could be under the manual configuration menu?

The the manual configuration would again be similar to the manual configuration for partitions. If 1 or 2 are selected, there could be a yes/no question for people to manually adjust the layout for certain use-cases as @wllacer outlined.

Would you like to manually adjust the subvolume layout?
1. Yes
2. No

If yes, it would go into the same manual configuration menu as selecting 3 in the prior step, but with the previously selected layout already configured. ( After This is where i'm thinking it might start to get away from the scope of the installer. ) After either skipping manual adjustment, or finishing it, then it would have another Yes/No pulling the user input from the first question

Automatically install [Timeshift/Snapper]?
1. Yes
2. No

and then if no, it just moves on to the next step of the install, if yes it sets up timeshift or snapper, (this is were I might actually be able to help with the snapper configuration, as its not just installing the package, you have to change some configs for snapper to fully function.)

[Torxed]

Select a subvolume layout

1. Ubuntu style layout (Compatible with Timeshift)
2. openSUSE style layout (Compatible with Snapper)
3. Manual configuration
4. None //could be under the manual configuration menu?

We could have this menu instead of the "Do you want to use subvolumes?" as this one packs more information but would add no additional steps.

[ghost]

Hi, just a quick question, why /var/cache/pacman/pkg instead of just /var/cache ? It makes little or no difference as most files are in /pacman/pkg/, but what is there to gain in going 2 steps deeper instead of excluding all cache from snapshots ?

[Torxed]

Hi, just a quick question, why /var/cache/pacman/pkg instead of just /var/cache ? It makes little or no difference as most files are in /pacman/pkg/, but what is there to gain in going 2 steps deeper instead of excluding all cache from snapshots ?

It's probably to strictly snapshot package states, for easier rollbacks or to use snapper (etc) to snapshot a point in time of a package store. For instance a stable snapshot when you knew packages were working, and you could (wild pseudo commands to follow) do something like pacman -U /var/cache/pacman/pkg/* and know it'll be stuff that works :)

[ghost]

Anything in the subvolume is not included in snapshots... that will be true, whether the subvol is /var/cache or /var/cache/pacman/pkg.
The point being to not fill the file system up with ever changing versions of cached packages

[Torxed]

Then I've got it backwards. I for sure would rather like to have it included in snapshots. Or at least I looked at it as a great way to take snapshots of known states :) But I guess then again, you could just snapshot your installation folders instead and store the known states there.. But then you'd might lose data if they're stored alongside the binaries etc, then those snapshotted installation packages would come in handy for a rollback.

Anyway, that's why I thought peopled wanted that subvolume :)

[ghost]

I think the installed state is in /var/lib/pacman/local
/var/cache/pacman/pkgis just the cache of installer files downloaded by pacman. The ones you clear with paccache.

$ ls /var/cache/pacman/pkg/
a52dec-0.7.4-11-x86_64.pkg.tar.zst
a52dec-0.7.4-11-x86_64.pkg.tar.zst.sig
aalib-1.4rc5-14-x86_64.pkg.tar.zst
accountsservice-0.6.55-3-x86_64.pkg.tar.zst
acl-2.3.1-1-x86_64.pkg.tar.zst
adobe-source-code-pro-fonts-2.038ro+1.058it+1.018var-1-any.pkg.tar.zst
adwaita-icon-theme-41.0-1-any.pkg.tar.zst
.....

They don't need to be backed up.

[KhalilSantana]

@Torxed,

Btrfs snapshot stops at the subvolume boundary, so putting /var/cache/pacman/pkg into a subvol means btrfs subv snap / /.snapshot/foo won't include anything inside the pkg folder, which is, in my opinion, the best way to do something like this.

By excluding pkg from the main rootfs subvol you can make snapshots of it much smaller, in case you want to do stuff like btrfs send... or just for diff purposes. Plus, having a pkg folder that's unchanged across rollbacks (assuming proper use of the non-nested subvol like I explained before), you could save a bit of network bandwidth by not having to re-download everything again.

Regarding /var/cache being a subvol or not, I'm not entirely sure, I don't think would cause any issues, but I also don't think it adds too much versus just putting pacman's pkg folder into a subvol, as that's by far the largest folder in my /var/cache/with netdata taking the second place at some 200MB's.

[ghost]

Regarding /var/cache being a subvol or not, I'm not entirely sure, I don't think would cause any issues, but I also don't think it adds too much versus just putting pacman's pkg folder into a subvol, as that's by far the largest folder in my /var/cache/with netdata taking the second place at some 200MB's.

I agree it doesn't add too much, but it still adds a little and makes it simpler at the same time, that's why I don't really see any reason not to use it instead of /var/cache/pacman/pkg.

Plus, having a pkg folder that's unchanged across rollbacks (assuming proper use of the non-nested subvol like I explained before), you could save a bit of network bandwidth by not having to re-download everything again.

It's a little off-topic so feel free to ignore, but in which scenario would you re-download the same installers for packages which are most likely still installed ? I struggle to understand why the usage is to keep so much cached packages, if I install something I most likely want to download the latest version, and if I uninstall something I'm unlikely to change my mind and install it again.

Back to the topic, is any thought given to suggesting a subvolume layout for @home ?

There are also good candidates for excluding from snapshots, like .cache, .local/Trash, .local/libvirt/images ... but it can be a little more complicated to deal with..

[KhalilSantana]

@clavelc

It's a little off-topic but in which scenario would you re-download the same installers for packages which are most likely still installed ? I struggle to understand why the usage is to keep so much cached packages, if I install something I most likely want to download the latest version, and if I uninstall something I'm unlikely to change my mind and install it again.

It depends on the user really. If someone's building a "golden image" of their Arch install, perhaps they'll want to test multiple drivers/settings before committing to it, so the user builds their install step-by-step using snapshots as checkpoints. Creating this "golden image" might take several attempts before figuring out which exact packages are required, which ones are bloat, and which settings or quirks to apply due to their hardware.

So having an unchanging pkg would be helpful, the user could snapshot his working system, then mess around with drivers, DEs, etc, then revert and try again, without needing to redownload most stuff.

[KhalilSantana]

My original proposal was rather simple, but as everyone is suggesting their own (more advanced/sophisticated) layouts, I'll dabble a bit on my own more featured layout, mostly for my own uses, but I'll try to make an effort to make this cover as many use cases as I can. (I'll also edit this comment as ideas come along).

I'll also use a table to make this more readable. The priority column is how much I care about that particular choice, or how much I think it's going to matter for users of specific software.

Subvol path	Mount point	Priority	Comment
/@	/	High	The root filesystem must be separated from user data (ie: home) for ease of reinstall, rollback, etc.
/@home	/home	High	User data must be separated from system data to enable rolling back either without affecting the other, ease of reinstall, etc.
/@pkg	/var/cache/pacman/pkg	Medium	Separating this dataset from the others means users of btrfs-send (or wrappers like btrbk) will have smaller send streams.
/@VMs	/var/lib/libvirt/	High	Rolling back the rootfs shouldn't affect VMs. And these are usually huge so the btrfs-send argument also applies here.
/@containers	???	Medium	Stuff for podman/docker so it doesn't implode when you rollback the rootfs.I'm not sure if docker uses the BTRFS driver by default on Arch, but podman doesn't. Mount point varies if user is using podman or docker, thus I left it blank
/@srv	/srv	Low	A nice place to dump random daemon homedirs, as pacman mostly won't touch this dir anyway. Basically a miscellaneous folder
/@log	/var/log/	Medium	Logs should be consistent across rootfs rollbacks

---

EDIT-0: Added the /@log subvol here, even though it was on my original comment I forgot to include it here =p. Thanks @Forza-tng via IRC

EDIT-1: Changed the /@VMssubvol mount point to /var/lib/libvirt instead of just it's images subvol, as the VM's NVRAM are stored on a sibling directory and those must be consistent. Thanks Jannik2099 via IRC for the suggestion.

[KhalilSantana]

@ShaunTheQuietGamer

just for the sake of clarity from here down, / is the filesystem root, \ is the partition root that can be mounted somewhere within the filesystem root but doesn't have to be mounted. I've been using / for both and I think that's the main reason why there is some confusion

Consider using "toplevel" or "subvolid=5" to describe filesystem root of all roots, using \ is best left for Windows folks =p.

Actually, I think I might have an analogy. Think of each of the subvolumes as physical disks on a normal system within /dev/. They are all in /dev so you know they are physical drives, just like the subvolumes are in @/ so you know they are subvolumes, because technically you could create a text file in the partition root right next to '@', (although I imagine its bad practice,) and if it was all just in the partition root it could get confusing.

I didn't really understand your analogy, for one it "compares" subvols to block devs, which is a rather bad way of understanding it in my opinion because subvols aren't block devs, they are independently mountable filesystem roots.

The best way to explain a subvol to a newbie would be something like "subvols are like normal directories with special semantics such as atomic snapshots and separate inode numbering, thus they share the same free space".

---

@wllacer

The same with VM images, this time aggravated by the fact that they are usually inside one user's home directory. (in this case I simply play with the nodatacow atrribute). Or for the KDE users, the directory where the baloo index file goes ...

Not really needed. KDE already does this on its own. Try this on your system and see for yourself:

% find ~/ -print0 | xargs -0 lsattr 2>/dev/null | grep -v -- '----------------------' | grep -- '-C-'
---------------C------ /home/khalil/.local/share/baloo
---------------C------ /home/khalil/.local/share/baloo/index
---------------C------ /home/khalil/.local/share/baloo/index-lock
---------------C------ /home/khalil/.local/share/baloo/index
---------------C------ /home/khalil/.local/share/baloo/index-lock
---------------C------ /home/khalil/.local/share/akonadi/db_data
---------------C------ /home/khalil/.local/share/akonadi/db_data/aria_log_control
... # and more

---

@clavelc

Back to the topic, is any thought given to suggesting a subvolume layout for @home ?

There are also good candidates for excluding from snapshots, like .cache, .local/Trash, .local/libvirt/images ... but it can be a little more complicated to deal with..

This is a more tricky subject in my opinion, for one, subvols have a few quirks associated with them that might not be immediately obvious to newbies, sometimes resulting in data loss. See this borgbackup issue as an example.

This doesn't mean I don't think subvols can't be used on homedirs, they can, it just requires the user to understand its quirks. For example, fragmentation is a common issue when torrenting on BTRFS, so you can (ab)use the same feature/bug (subvols being treated as different filesystems) to have your torrent copied + unliked when it finishes, thus defragging the file without the torrent client being aware of BTRFS itself.

[DrymarchonShaun]

I didn't really understand your analogy, for one it "compares" subvols to block devs, which is a rather bad way of understanding it in my opinion because subvols aren't block devs, they are independently mountable filesystem roots.

The reason it didn't make sense to you was because I wasn't trying to explain subvolumes, I was trying to explain what's i'm guessing the thinking was behind opensuse's use of toplevel/@/* instead of toplevel/@* to delineate subvolumes from regular folders. It doesn't actually do anything system-wise, it's just a different way of making things easier for people to understand

[KhalilSantana]

@KhalilSantana I assume if we just get the mounting done correctly, this should be all good?

@Torxed,

I didn't read the code (Python doesn't click for me sadly), but just going by observing the final output, Archinstall needs to:

• Before running the equivalent of pacstrap, create the subvols on the toplevel
• Umount the toplevel and mount the rootfs subvol (/@) into /mnt/archinstall
• Mount other subvolumes such as /@homeinto their place inside the /mnt/archinstall
• Run pacstrap in /mnt/archinstall
• Run genfstab -U >> /mnt/archinstall/etc/fstab (the currently generated /etc/fstab is incorrect, but it should fix itself after solving the items above).
• Make sure the kernel cmdline has the rootflags=subvol=/@ parameter, so it mounts the correct subvolume as root without having to use btrfs subvolume set-default. To accomplish this, configure the bootloader to pass that flag to the Linux kernel, example for systemd-boot:

title   Arch Linux
linux   /vmlinuz-linux
initrd  /intel-ucode.img
initrd  /initramfs-linux.img
options  rootflags=subvol=/@

The checked boxes are seemingly done/correct from my point of view, while the unchecked ones need to be addressed for a functional Arch system that works out-of-the-box with Timeshift (or anything else that expects the Flat layout with the Ubuntu-style naming scheme).

---

Regarding /etc/fstab vs systemd-mount units, as far as I'm aware, systemd-mount parses and generates its own .mountunits from fstab anyways[1], I personally haven't fiddled with it, so I'd rather keep everything on /etc/fstab and let systemd-mount figure the units by itself.

[1] - Try using sudo systemctl list-units | grep mount on a system which you didn't explicitly configure systemd-mount

[Torxed]

@KhalilSantana Thank you! We'll stick to /etc/fstab for now :)
And I think we're on to a solution regarding why the mountpoint's aren't mounted properly. Stay tuned! : )

[Torxed]

I've created a draft PR that hopefully solves this. I suspect wllacer would solve this in 5 minutes.
But I'll give it a go :) Good learning experience!

[wllacer]

@wllacer Sorry for pinging, but I'm trying to figure out the last issue before release. And I think it's because

archinstall/archinstall/lib/disk/btrfs.py

Line 12 in becb423

def mount_subvolume(installation, subvolume_location :Union[pathlib.Path, str], force=False) -> bool:

is never called after creating the "mount structure" here:

archinstall/archinstall/lib/disk/btrfs.py

Lines 177 to 178 in becb423

	if partition['mountpoint']:
	mountpoints[partition['mountpoint']] = partition

Is this a correct assumption? Note that I'm working with v2.3.1-dev here and not master where mount_subvolume is

Yes , I never needed to call it as I create standard partition dictionary entries in *manage_btrfs_... for each mountable subvolume since day one. There is (i believe) where I mounted / unmonted the root partition @KhalilSantana misses.
In my machine I never ever got (with the standard layout) more than the root subvolume mounted, not even with the original code.
I just assumed mount would silently "forget" them as they were already reachable. If you give me a few minutes, i'll do some tests and report back

[Torxed]

Thank you @wllacer.
In my PR I've started to tinker with it. And I'm starting to understand the logic a bit more.
I assumed manage_btrfs_subvolumes returned a flat structure of all the mountpoints that we needed to deal with, but it returns essentially {"/": {..., "subvolume" : {...}}.

Since the variable was called mountpoints I assumed it would contain all the mountpoints needed to be setup in a flat structure. I think that's what's missing tbh. If we can get manage_btrfs_subvolumes to instruct the following calls which mountpoints to setup, that would be it : )

I could have this done within the hour probably if you're busy. And the PR is almost done.

[wllacer]

Got the problem
In btrfs.manage_btrfs_subvolumes

archinstall/archinstall/lib/disk/btrfs.py

Lines 134 to 136 in becb423

	# we do not mount if THE basic partition will be mounted or if we exclude explicitly this subvolume
	if not partition['mountpoint'] and location is not None:
	# we begin to create a fake partition entry. First we copy the original -the one that corresponds to

look at the generated json

           },
            {
                "btrfs": {
                    "subvolumes": {
                        "@": "/",
                        "@.snapshots": "/.snapshots",
                        "@home": "/home",
                        "@log": "/var/log",
                        "@pkg": "/var/cache/pacman/pkg"
                    }
                },
                "encrypted": false,
                "filesystem": {
                    "format": "btrfs"
                },
                "format": true,
                "mountpoint": "/",        #here is the culprit
                "size": "100%",
                "start": "513MiB",
                "type": "primary"
            }

I never touched the part where the entry is generated (ny bad at user_guides and, as the partition entry has a mountpoint in the partition entry , all the procedures to create subvolumes don't work. I think i documented it somewhere, but never got to the code.
The fastest patch should be

archinstall/archinstall/lib/disk/user_guides.py

Lines 36 to 44 in becb423

	layout[block_device.path]['partitions'].append({
	# Root
	"type" : "primary",
	"start" : "518MB",
	"encrypted" : False,
	"format" : True,
	"mountpoint" : "/",
	"filesystem" : {
	"format" : default_filesystem

at line 42 something like

     mountpoint = '/' if not using_subvolumes else None

Can you do it ? Else I have to make a path for master and then you've to port it back

[Torxed]

I can try it : )
I think however we'll still have a bunch of issues after this fix.
For instance, we should ensure that the subvolumes get mounted in the correct places.

I can't see how that would work with the current code that is in my PR.

Here's the current state:

Formatting /dev/loop0p2 -> btrfs
Mounting Partition(path=/dev/loop0p2, size=19.5, PARTUUID=b84cb72e-70f1-4f0b-8192-593b797992a7, fs=btrfs) to /mnt/archinstall/
Creating a subvolume on /mnt/archinstall/@
Creating a subvolume on /mnt/archinstall/@.snapshots
Creating a subvolume on /mnt/archinstall/@home
Creating a subvolume on /mnt/archinstall/@log
Creating a subvolume on /mnt/archinstall/@pkg
Mounting Partition(path=/dev/loop0p2, size=19.5, PARTUUID=b84cb72e-70f1-4f0b-8192-593b797992a7, fs=btrfs) as / to /mnt/archinstall/ using options None
Mounting Partition(path=/dev/loop0p2, size=19.5, PARTUUID=b84cb72e-70f1-4f0b-8192-593b797992a7, fs=btrfs) to /mnt/archinstall/
Getting mount information for device path /mnt/archinstall/
Mounting Partition(path=/dev/loop0p2, size=19.5, PARTUUID=b84cb72e-70f1-4f0b-8192-593b797992a7, fs=btrfs, mounted=/mnt/archinstall/) as /.snapshots to /mnt/archinstall/.snapshots using options subvolid=2

[wllacer]

I got it fine

└─/mnt/archinstall                /dev/loop0p2[/@]     btrfs      rw,relatime,ssd,space_cache=v2,subvolid=256,subvol=/@
  ├─/mnt/archinstall/boot         /dev/loop0p1         vfat       rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro
  ├─/mnt/archinstall/.snapshots   /dev/loop0p2[/@.snapshots]
  │                                                    btrfs      rw,relatime,ssd,space_cache=v2,subvolid=260,subvol=/@.snapshots
  ├─/mnt/archinstall/home         /dev/loop0p2[/@home] btrfs      rw,relatime,ssd,space_cache=v2,subvolid=257,subvol=/@home
  ├─/mnt/archinstall/var/cache/pacman/pkg
  │                               /dev/loop0p2[/@pkg]  btrfs      rw,relatime,ssd,space_cache=v2,subvolid=259,subvol=/@pkg
  └─/mnt/archinstall/var/log      /dev/loop0p2[/@log]  btrfs      rw,relatime,ssd,space_cache=v2,subvolid=258,subvol=/@log

PR #906 for master

[wllacer]

.snapshots should not be mounted according to the SuSe people, but is just fine ;-)

[Torxed]

Weird, it didn't work for me (Edit: just realized because I'm testing against v2.3.1-dev)
But then again, I was messing with the code so much.

Where do we call mount -t btrfs -o subvolume=@/ /mnt/archinstall and all the other ones?

[Torxed]

I'll try to fix this in v2.3.1-dev as well, but since the logic is backwards compatible and not as heavily modified to support the complex nature of btrfs, it can get tricky to do so.

Currently I got:

Formatting /dev/loop0p2 -> btrfs
Mounting Partition(path=/dev/loop0p2, size=19.5, PARTUUID=8180918a-a9b4-4b35-8cb2-0f84c0e47d0c, fs=btrfs) to /mnt/archinstall/
Creating a subvolume on /mnt/archinstall/@
Creating a subvolume on /mnt/archinstall/@.snapshots
Creating a subvolume on /mnt/archinstall/@home
Creating a subvolume on /mnt/archinstall/@log
Creating a subvolume on /mnt/archinstall/@pkg
Mounting Partition(path=/dev/loop0p2, size=19.5, PARTUUID=8180918a-a9b4-4b35-8cb2-0f84c0e47d0c, fs=btrfs) as / to /mnt/archinstall/ using options None
Mounting Partition(path=/dev/loop0p2, size=19.5, PARTUUID=8180918a-a9b4-4b35-8cb2-0f84c0e47d0c, fs=btrfs) to /mnt/archinstall/
Getting mount information for device path /mnt/archinstall/
Mounting Partition(path=/dev/loop0p2, size=19.5, PARTUUID=8180918a-a9b4-4b35-8cb2-0f84c0e47d0c, fs=btrfs, mounted=/mnt/archinstall/) as /.snapshots to /mnt/archinstall/.snapshots using options subvol=@/.snapshots
Getting mount information for device path /mnt/archinstall/.snapshots
Target /mnt/archinstall/.snapshots never got mounted properly (unable to get mount information using findmnt).

[wllacer]

Weird, it didn't work for me (Edit: just realized because I'm testing against v2.3.1-dev) But then again, I was messing with the code so much.

Where do we call mount -t btrfs -o subvolume=@/ /mnt/archinstall and all the other ones?

At mount_ordered_partition (just like normal partitions). The result of manage_btrfs_subvolumes is a bunch of new entries there, one for each subvolume to be mounted.
I'll try to download and test v231-dev (NOPE. keep getting the master branch when running)

[wllacer]

Just for documentation

@KhalilSantana I assume if we just get the mounting done correctly, this should be all good?

@Torxed,

I didn't read the code (Python doesn't click for me sadly), but just going by observing the final output, Archinstall needs to:

* [x]  Before running the equivalent of pacstrap, create the subvols on the toplevel

done at btrfs.manage_btrfs_subvolumes, where it also decides which subvolumes are to be mounted (there is no need for a 1:1 correspondency)

* [ ]  Umount the toplevel and mount the rootfs subvol (`/@`) into `/mnt/archinstall`

* [ ]  Mount other subvolumes such as `/@home`into their place inside the `/mnt/archinstall`

(The umount was done at the previous routine in 2.3.1, IIRC). This both are done at installer.mount_ordered_partition: We have a list (dict in in 2.3.1) of partition entries to be mounted and each of them are mounted in lexicographical order (so / is always the first)

* [x]  Run `pacstrap` in `/mnt/archinstall`

* [x]  Run `genfstab -U >> /mnt/archinstall/etc/fstab` (the currently generated /etc/fstab is incorrect, but it should fix itself after solving the items above).

* [ ]  Make sure the kernel cmdline has the `rootflags=subvol=/@` parameter, so it mounts the correct subvolume as root without having to use `btrfs subvolume set-default`. To accomplish this, configure the bootloader to pass that flag to the Linux kernel, example for systemd-boot:

It is done at installer.add_bootloader. Currently only for gummi-bootsystemd-bootclt and grub

title   Arch Linux
linux   /vmlinuz-linux
initrd  /intel-ucode.img
initrd  /initramfs-linux.img
options  rootflags=subvol=/@

The checked boxes are seemingly done/correct from my point of view, while the unchecked ones need to be addressed for a functional Arch system that works out-of-the-box with Timeshift (or anything else that expects the Flat layout with the Ubuntu-style naming scheme).

We can generate whatever subvolume/partition scheme I could think of -sadly, still no UI, but working directly with the Json files-. The only restriction at 2.3.1 is that we didn't allowed to encrypt a btrfs partition with subvolumes. We have this restriction lifted already in master

Regarding /etc/fstab vs systemd-mount units, as far as I'm aware, systemd-mount parses and generates its own .mountunits from fstab anyways[1], I personally haven't fiddled with it, so I'd rather keep everything on /etc/fstab and let systemd-mount figure the units by itself.

[1] - Try using sudo systemctl list-units | grep mount on a system which you didn't explicitly configure systemd-mount

[Torxed]

This is the current state of the code in v2.3.1-dev -fix branch:

[root@bigrigv2 archinstall]# findmnt --real
TARGET                                    SOURCE                     FSTYPE      OPTIONS
└─/mnt/archinstall                        /dev/loop0p2[/@]           btrfs       rw,relatime,ssd,space_cache=v2,subvolid=256,subvol=/@
  ├─/mnt/archinstall/boot                 /dev/loop0p1               vfat        rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro
  ├─/mnt/archinstall/.snapshots           /dev/loop0p2[/@.snapshots] btrfs       rw,relatime,ssd,space_cache=v2,subvolid=257,subvol=/@.snapshots
  ├─/mnt/archinstall/home                 /dev/loop0p2[/@home]       btrfs       rw,relatime,ssd,space_cache=v2,subvolid=258,subvol=/@home
  ├─/mnt/archinstall/var/cache/pacman/pkg /dev/loop0p2[/@pkg]        btrfs       rw,relatime,ssd,space_cache=v2,subvolid=260,subvol=/@pkg
  └─/mnt/archinstall/var/log              /dev/loop0p2[/@log]        btrfs       rw,relatime,ssd,space_cache=v2,subvolid=259,subvol=/@log

Not sure this is enough, I'm missing the @home when doing ls -l /mnt/archinstall

Fstab looks ok I think:

[root@bigrigv2 /]# cat /etc/fstab
# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
# /dev/loop0p2
UUID=ff13bad7-374a-47d4-9683-c9baba085296	/         	btrfs     	rw,relatime,ssd,space_cache=v2,subvolid=256,subvol=/@	0 0

# /dev/loop0p1
UUID=7DB7-F6FA      	/boot     	vfat      	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro	0 2

# /dev/loop0p2
UUID=ff13bad7-374a-47d4-9683-c9baba085296	/.snapshots	btrfs     	rw,relatime,ssd,space_cache=v2,subvolid=257,subvol=/@.snapshots	0 0

# /dev/loop0p2
UUID=ff13bad7-374a-47d4-9683-c9baba085296	/home     	btrfs     	rw,relatime,ssd,space_cache=v2,subvolid=258,subvol=/@home	0 0

# /dev/loop0p2
UUID=ff13bad7-374a-47d4-9683-c9baba085296	/var/cache/pacman/pkg	btrfs     	rw,relatime,ssd,space_cache=v2,subvolid=260,subvol=/@pkg	0 0

# /dev/loop0p2
UUID=ff13bad7-374a-47d4-9683-c9baba085296	/var/log  	btrfs     	rw,relatime,ssd,space_cache=v2,subvolid=259,subvol=/@log	0 0

[Torxed]

I suspect subvol=/@ should have been subvol=@ in the OPTIONS.
But it was mounted with:

Mount command: /usr/bin/mount -t btrfs -o subvol=@ /dev/loop0p2 /mnt/archinstall/

[Torxed]

The state of the PR is that it appears to do everything we expect.
However, GRUB does not boot with subvolumes (encrypted or otherwise).

Most likely because a lack of rootflags=subvol=/@, which I haven't figured out how to "put in" yet.
I'll adopt the systemd-boot logic to the GRUB section of the code, but I'm very unfamiliar with GRUB as I haven't used it in somewhat 10 years.

I'm still performing some last tweaks, mainly compatibility between GRUB and systemd-boot.
But PR and release of v2.3.1 should be ready by tomorrow. Got my hopes up now! : )

[Torxed]

• systemd-boot with subvolumes
• systemd-boot + luks2 with subvolumes
• grub in BIOS with subvolumes
• grub in BIOS + luks2 with subvolumes
• grub in UEFI with subvolumes
• grub in UEFI + luks2 with subvolumes

Grub appears broken with UEFI, and I think the reason for the screenshot was simply because fstab wasn't properly generated (due to a setting of timezone issue at the very last step of the installer)

[wllacer]

Funny. When i tried grub i had a lot of environmental troubles, (could only execute It inside virtualbox) but the execution of grub-mkconfig (in the add_bootloader) created everything fine without need of me touching anything. El mar., 25 ene. 2022 23:15, Anton Hvornum ***@***.***> escribió:

…

The state of the PR is that it appears to do everything we expect. However, GRUB does not boot with subvolumes (encrypted or otherwise). Most likely because a lack of rootflags=subvol=/@, which I haven't figured out how to "put in" yet. I'll adopt the systemd-boot logic to the GRUB section of the code, but I'm very unfamiliar with GRUB as I haven't used it in somewhat 10 years. I'm still performing some last tweaks, mainly compatibility between GRUB and systemd-boot. But PR and release of v2.3.1 should be ready by tomorrow. Got my hopes up now! : ) — Reply to this email directly, view it on GitHub <#781 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACSDTYHL6REO3NR7YOZ6FW3UX4OJ7ANCNFSM5JF3YX3A> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[Torxed]

Yepp, it was just genfstab that didn't execute upon exit of the installer (due to another error I'd like to fix).
Will test encrypted GRUb again.

[Torxed]

Giving up for today.
GRUB in UEFI doesn't work. I'm not sure why. Probably something to do with efivars or something not properly installed in the right place. I'm perfectly fine accepting this limitation and simply not allow subvolumes on GRUB in EFI mode, and instead instruct users to use systemd-boot for that if they really need the subvolumes in v2.3.1.

I'll try again tomorrow, but chances are slim that I understand GRUB well enough to sort this out.

[KhalilSantana]

@Torxed

I suspect subvol=/@ should have been subvol=@ in the OPTIONS.

It works either way, one is an absolute path, the other is a relative path to toplevel, both point to the same subvolume in the end.

Most likely because a lack of rootflags=subvol=/@, which I haven't figured out how to "put in" yet.

I also don't use GRUB, but the ArchWiki details how to use GRUB_CMDLINE_LINUX, which should accomplish this goal.

---

The outputs show on this comment look correct for me, I'll probably test this again on my other SSD soon just to be sure.

[Torxed]

@KhalilSantana Glad to hear! Then I feel more confident merging the PR.

I'll test a little bit more with GRUB + UEFI tomorrow before merging.
And after that I need to sort out #908.

After which, release time : )

[Torxed]

It now works on GRUB + systemd-boot.
Both with, and without subvolumes.
And both with, and without luks2 encryption.

The default layout will be merged and look like this for v2.3.1:

[Torxed]

I'd like to thank everyone that contributed to this pretty large thread.
And I apologize for the spam the last two days, been keeping busy pushing this out of the way and got carried away :)

This change has now made it's way into master and v2.3.1-dev, and I'm currently in the process of releasing v2.3.1 tonight. Thank you, and great work ironing out the details and carefully putting together explanations, hugely appreciated!