k0s reset deleted all data on persistent volumes
#4318
Comments
|
The |
|
Would having a prompt make the UX better, and perhaps reduce the chance of such occurrences happening. I'd add a Goes without saying, this would be a breaking change |
|
Well, there is of course the safeguard that k0s must be not running. The main problem in this case is that first reset tries to unmount such volumes, but even if it fails to do so, it does not abort and proceeds to recursively delete data under datadir/rundir, which includes data on any volumes mounted under them. |
|
Ah that's useful to know, thanks! I didn't realise mounted volumes came under the potential data parts up for deletion during a reset. The docs
So I perhaps should have procedded with a little more caution! I do wonder whether a bullet point could be added to the docs there that includes something similar to this line
Happy to make a PR for it, if it helps! Otherwise good to close this issue. |
|
The issue is marked as stale since no activity has been recorded in 30 days |
|
@devsjc PRs are always welcome :D |
|
The issue is marked as stale since no activity has been recorded in 30 days |
|
@jnummelin Please Reopen this issue. This is a serious problem and a critical design flaw (yes even 'just' clearing /var/lib/k0s) IMO. This behaviour recently caused a serious data loss incident at my organisation where a k0s reset command was executed prior to un-mounting critical data as a part of a large and complex release. If we did not have backup policies in place, this would have been an existential problem for our organisation, as it was, it caused a week of headaches and a large Cloud Egress bill for restoring deleted data. There is just no way that silent deletion of data is an acceptable behaviour for any software that is intended for enterprise use. We have DevOps capability in my team, happy to contribute with some guidance. //from pkg/cleanup/directories.go
var dataDirMounted bool
// search and unmount kubelet volume mounts
for _, v := range procMounts {
if v.Path == filepath.Join(d.Config.dataDir, "kubelet") {
logrus.Debugf("%v is mounted! attempting to unmount...", v.Path)
if err = mounter.Unmount(v.Path); err != nil {
logrus.Warningf("failed to unmount %v", v.Path)
}
} else if v.Path == d.Config.dataDir {
dataDirMounted = true
}
}
...etc
if err := os.RemoveAll(d.Config.dataDir); err != nil {.... as discussed above, failing to unmount does not prevent the RemoveAll from executing |
|
I think the safest option is to implement recursive directory removal using openat2 along with |
|
We definitively need to refactor this. The problem is bigger than crossing mount points. It can delete completely unrelated stuff. mkdir -p /tmp/camerun/netnsomething/foo
echo data > /tmp/camerun/netnsomething/foo/data
sudo mount --bind /tmp/camerun/netnsomething/foo /tmp/camerun/netnsomething/start and boom! the EDIT: this is a separate issue. |
|
A bind mount is a mount point, isn't it? Btw, I've been investigating this for quite some days already. |
Make sure that we don't have anything mounted under those directories so we don't delete persistent data. We do this py parsing /proc/mounts in reverse order as it is listed in mount order, and then we unmount anything that is under our directories before we delete them. Don't umount datadir itself if it is on a separate partition/mount Fixes k0sproject#4318 Signed-off-by: Natanael Copa <ncopa@mirantis.com>
Make sure that we don't have anything mounted under those directories so we don't delete persistent data. We do this py parsing /proc/mounts in reverse order as it is listed in mount order, and then we unmount anything that is under our directories before we delete them. Don't umount datadir itself if it is on a separate partition/mount Fixes k0sproject#4318 Signed-off-by: Natanael Copa <ncopa@mirantis.com>
Make sure that we don't have anything mounted under those directories so we don't delete persistent data. We do this py parsing /proc/mounts in reverse order as it is listed in mount order, and then we unmount anything that is under our directories before we delete them. Don't umount datadir itself if it is on a separate partition/mount Fixes k0sproject#4318 Signed-off-by: Natanael Copa <ncopa@mirantis.com>
Make sure that we don't have anything mounted under those directories so we don't delete persistent data. We do this py parsing /proc/mounts in reverse order as it is listed in mount order, and then we unmount anything that is under our directories before we delete them. Don't umount datadir itself if it is on a separate partition/mount Fixes #4318 Signed-off-by: Natanael Copa <ncopa@mirantis.com> (cherry picked from commit 7ab81aa)
Make sure that we don't have anything mounted under those directories so we don't delete persistent data. We do this py parsing /proc/mounts in reverse order as it is listed in mount order, and then we unmount anything that is under our directories before we delete them. Don't umount datadir itself if it is on a separate partition/mount Fixes #4318 Signed-off-by: Natanael Copa <ncopa@mirantis.com> (cherry picked from commit 7ab81aa)
Make sure that we don't have anything mounted under those directories so we don't delete persistent data. We do this py parsing /proc/mounts in reverse order as it is listed in mount order, and then we unmount anything that is under our directories before we delete them. Don't umount datadir itself if it is on a separate partition/mount Fixes k0sproject#4318 Signed-off-by: Natanael Copa <ncopa@mirantis.com> (cherry picked from commit 7ab81aa) (cherry picked from commit 54fb78c)
Before creating an issue, make sure you've checked the following:
Platform
Linux 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/LinuxVersion
v1.29.3+k0s.0
Sysinfo
`k0s sysinfo`
Machine ID: "bf8222b2f95c64426d710b2c00cc6e5eb49618a69a861782356308a69f99328c" (from machine) (pass) Total memory: 15.3 GiB (pass) Disk space available for /var/lib/k0s: 13.5 GiB (pass) Name resolution: localhost: [::1 127.0.0.1] (pass) Operating system: Linux (pass) Linux kernel release: 6.1.0-18-amd64 (pass) Max. file descriptors per process: current: 1048576 / max: 1048576 (pass) AppArmor: active (pass) Executable in PATH: modprobe: exec: "modprobe": executable file not found in $PATH (warning) Executable in PATH: mount: /usr/bin/mount (pass) Executable in PATH: umount: /usr/bin/umount (pass) /proc file system: mounted (0x9fa0) (pass) Control Groups: version 2 (pass) cgroup controller "cpu": available (is a listed root controller) (pass) cgroup controller "cpuacct": available (via cpu in version 2) (pass) cgroup controller "cpuset": available (is a listed root controller) (pass) cgroup controller "memory": available (is a listed root controller) (pass) cgroup controller "devices": unknown (warning: insufficient permissions, try with elevated permissions) cgroup controller "freezer": available (cgroup.freeze exists) (pass) cgroup controller "pids": available (is a listed root controller) (pass) cgroup controller "hugetlb": available (is a listed root controller) (pass) cgroup controller "blkio": available (via io in version 2) (pass) CONFIG_CGROUPS: Control Group support: built-in (pass) CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass) CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass) CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass) CONFIG_CPUSETS: Cpuset support: built-in (pass) CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass) CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass) CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass) CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass) CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass) CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass) CONFIG_BLK_CGROUP: Block IO controller: built-in (pass) CONFIG_NAMESPACES: Namespaces support: built-in (pass) CONFIG_UTS_NS: UTS namespace: built-in (pass) CONFIG_IPC_NS: IPC namespace: built-in (pass) CONFIG_PID_NS: PID namespace: built-in (pass) CONFIG_NET_NS: Network namespace: built-in (pass) CONFIG_NET: Networking support: built-in (pass) CONFIG_INET: TCP/IP networking: built-in (pass) CONFIG_IPV6: The IPv6 protocol: built-in (pass) CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass) CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass) CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass) CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass) CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass) CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass) CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass) CONFIG_NETFILTER_XT_SET: set target and match support: module (pass) CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass) CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass) CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass) CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass) CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass) CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass) CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass) CONFIG_NETFILTER_NETLINK: module (pass) CONFIG_NF_NAT: module (pass) CONFIG_IP_SET: IP set support: module (pass) CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass) CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass) CONFIG_IP_VS: IP virtual server support: module (pass) CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass) CONFIG_IP_VS_SH: Source hashing scheduling: module (pass) CONFIG_IP_VS_RR: Round-robin scheduling: module (pass) CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass) CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning) CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass) CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning) CONFIG_IP_NF_IPTABLES: IP tables support: module (pass) CONFIG_IP_NF_FILTER: Packet filtering: module (pass) CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass) CONFIG_IP_NF_NAT: iptables NAT support: module (pass) CONFIG_IP_NF_MANGLE: Packet mangling: module (pass) CONFIG_NF_DEFRAG_IPV4: module (pass) CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning) CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning) CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass) CONFIG_IP6_NF_FILTER: Packet filtering: module (pass) CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass) CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass) CONFIG_NF_DEFRAG_IPV6: module (pass) CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass) CONFIG_LLC: module (pass) CONFIG_STP: module (pass) CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: module (pass) CONFIG_PROC_FS: /proc file system support: built-in (pass)What happened?
k0s reset, followed by a node reboot deleted all files from all persistent volumes, irrespective of theirRetainpolicies. Folders remained, but were completely empty.Steps to reproduce
I have not managed to reproduce the error (thankfully!)
Expected behavior
Persistent volumes mounted with the
Retainpolicy are untouched on a reset, only k0s'/var/lib/k0sdirectory gets cleaned.Actual behavior
See "What Happened?"
Screenshots and logs
I have the full set of logs from
sudo journalctl -u k0scontroller -r -U 2024-04-13 -S 2024-04-11but I imagine a more focussed subset is more useful!Additional context
Firstly, thanks for the great tool!
I have previously run
k0s reseta fair few times without issue with no changes to the way volumes were mounted or to the services running in the cluster. All I can think that seperates this one from the others is the context of the reset regarding why it was needed:This specific reset was prompted by an issue with the helm extensions: removal of a chart from the k0s config yaml, instead of uninstalling the chart, put the cluster in an unstartable state. The config was installed into the controller by
$ sudo k0s install controller --single -c ~/hs-infra/k0s.yamlAnd
k0s stopwas run before making changes to the config. The only changes tok0s.yamlwere fromto
k0s startwould then error with logs such asor
Before running a k0s reset, to try to resolve the error and start the cluster successfully, I modified
/var/lib/k0s/helmhome/repositories.yamlto remove reference to the charts, but this didn't work either. Therefore I rank0s resetas I had a few times before, and performed the node reboot as requested. However on restarting the server, all files in any mounted volumes were deleted - definitely deleted and not just hidden or moved as an inspection of available space revealed. Strangely enough the folder structure within the volumes remained, but just all empty directories.If it helps, here's some manifest snippets!
Persistent Volume manifest example
Persistent Volume Claims manifest example
Volume mount in deployment example
The text was updated successfully, but these errors were encountered: