Update destroy.sh - refuse to destroy jail with mounted filesystem

[tschettervictor]

This PR fixes an issue where files could be deleted when bastille attempts to destroy a jail that still has mounted filesystems when using ZFS. It will throw an error and exit if it detects a filesystem is still mounted inside the jail.

To test

# bastille create temp 14.0-RELEASE 192.168.1.10 lo1
...
# mkdir test
# cp /usr/bin/less test/
# mkdir /usr/local/bastille/jails/temp/root/test
# bastille mount temp $(realpath test) test
[temp]:
Added: /root/admin/bastille/test /usr/local/bastille/jails/temp/root/test nullfs ro 0 0
# /usr/local/bastille/jails/temp/root/test/less -f /dev/stdin &
# bastille destroy force temp
rdr-anchor not found in pf.conf
[temp]:
temp: removed
umount: unmount of /usr/local/bastille/jails/temp/root/test failed: Device busy
jail: temp: /sbin/umount -t nullfs /usr/local/bastille/jails/temp/root/test: failed

Deleting Jail: temp.
Jail has mounted filesystems:
/usr/local/bastille/jails/temp/root/test

With the PR in place, bastille will error upon finding existing mounts.

[tschettervictor]

This won't work.
Since jails are always mounted, there will always be a mount point present, and jails can never be destroyed with this.

[tschettervictor]

Tested and working as expected now.

[bmac2]

need testing on this one to merge
@yaazkal @cedwards

[tschettervictor]

The concept is this.

Jails have their mount points defined in the fstab file, but are also mounted and can be found using the mount command. Any mount points that follow "/root" need to be unmounted before destroying the jail.

But there is also the root dataset of the jail itself that does need to be destroyed, that's why we grep for "${bastille_jail_base}/root/" with the trailing slash in our mount points. If we would not include the trailing slash, we would end up also find the root dataset of the jail, and could never delete a jail

[bmac2]

give me an example of how you are mounting filesystems inside your jail. I want to make sure I am doing it the way you are for testing.

[tschettervictor]

#662

Basically, sometimes a mount point will fail to unmount, and the destroy command will then delete content inside the mounted directory.

This PR will grep for any mount points below root/ of the jail, which should normally already be unmounted when the jail is stopped.

Easiest way to test is to mount a directory into the jail, copy a binary into it, and run the binary from the host.

Then stop the jail and try to destroy it.

With this PR it should fail because it detects an existing mount point.

Without the PR it destroys all the data inside the mount point.

[tschettervictor]

I could also add something I've added to my own fork, which will attempt to force unmount any mounts before it does the destroy.

It's basically an additional safety feature.

[bmac2]

tested and as long as tghere is a running process in a mount, it will not destroy the jail, but it will stop it and throw a message that there is a mounted filesystem.

@yaazkal test this one also yourself. This is the 3rd one ready for you.

[yaazkal]

what should be the behavior when the user uses destroy -f to force the stop of the jail?

[tschettervictor]

It simply will stop the jail before it attempts the destroy. If -f is not set, and the jail is running, it will just error and say so.

The only thing this PR adds is a check for any mount point that were not unmounted properly when the jail was stopped. Without this PR, bastille would just delete everything leading to potential data loss if a mount point was not unmounted for any reason. With this PR destroy does not do anything in the above case, but simply warns about the existing mounts.

[yaazkal]

@tschettervictor said:

If -f is not set, and the jail is running, it will just error and say so.

But running bastille destroy TARGET already says it can't destroy it because the jail is running, so actually it doesn't even get to that message in this PR.

So I guess the issue is as @tschettervictor said:

Basically, sometimes a mount point will fail to unmount, and the destroy command will then delete content inside the mounted directory.

I can see @tschettervictor in your original comment that one way to replicate that, is to have something running in the host from the jail tree. Are there one other scenarios to replicate an unmount failure? I just want to think if we are really getting to the root cause, because that initial scenario is not Bastille's fault.

[tschettervictor]

EDIT the -f flag has always been part of the command btw

There can be multiple causes. I have seen it where I was running uptimekuma with the data stored in the same mount point as the executable.

This was on iocage though, and they do have the exact same style of error checking.

Bottom line is IMO that bastille should not attempt to destroy if mounts are present.

BTW bastille already has the -f flag to allow stopping the jail first before destroying. It will not attempt destroy if the jail is running anyway, so you have to stop it first. The problem (as mentioned above) is that sometimes in certain cases a mount will fail to unmount, and data can get deleted if bastille doesn't recognize this.

[yaazkal]

yes, I wonder if we have to do something in the stop command in order to check/force the umount there? I guess there could be more side effects of a failing umount and we are just working on one.

[tschettervictor]

Hmmmm...
I actually tried what you are suggesting, and it didn't work.

There is a -f option in the umount (official, not bastille umount) the attempts a force unmount.

If you want I can go ahead and implement that into the destroy command. It's only a few lines of code

Basically it would check, attempt force unmount, check again, and if it failed the unmount, give an error.

[yaazkal]

I'll left the current message in this PR.

And to help the user managing other scenarios:

1. Also use the -f option you are talking in the umount system command if it's safe to do it, but I feel it has more sense in the stop command. Will this mean a new -f option to the stop command? maybe are the users that needs to decide if the umount -f is what they need?

2. And what about to also giving an error_exit in the stop command if the unmount was not possible for any given reason? Also, will this mean a new -f option to the stop command?

3. destroy -f will use stop -f if gets implemented.

Just thinking loud, what do you think?

[yaazkal]

@tschettervictor I'll merge this as it is. Maybe we need to think in a stop -force|-f later in another issue.

cc @bmac2

[tschettervictor]

The -f option in all scripts at this point has the following function.

"If the jail needs to be stopped to perform the script action, the -f will stop it first, otherwise an error is shown telling users that the jail needs to be stopped first. This works the same way for jails that need to be started."

If we do a "force unmount" flag id make it -u or something like that.

Keeping -f for stopping/starting a jail if the script requires it. Or we could have the start stop be -s. I'd just like consistency

[tschettervictor]

But stop -f could have a force unmount option, as stop itself is to stop a jail.

That could work.