-
Notifications
You must be signed in to change notification settings - Fork 376
K8s emptyDir on virtio-fs provisioned on memory filesystem #1818
Comments
Just spent some time with @amshinde on this issue. The problem is the same for both 9p and virtio-fs. Because you create the The simple solution is to not ignore the The only question raised was about performance because now we will have to go over the guest/host boundary when using this empty dir. Well the answer is that if you expect performance out of this directory, you should use it as |
Bind mounting the shared directory created by k8s on the host sounds like a solid plan. Where would a good place to start to solve this? Ideally if someone wanted performance with an emptyDir, they should be using |
@awprice I thought with #1485, we are creating the emptydirs in the sandbox rootfs.
I think the |
Nice find @amshinde! This is where the path is constructed - runtime/virtcontainers/kata_agent.go Line 1297 in 3d4729d
As you said, maybe we need to move |
That's an option, but you have to keep in mind that by doing so, you're using the space made available by the container rootfs overlay snapshot. And this space might not be as large as you could expect the |
@sboeuf The original intention behind #1485 was to provision emptyDir inside the sandbox container, and so if the user was using devicemapper, emptyDir would benefit from the speed of devicemapper.
I've tested this behaviour with virtio-fs, and reverting the changes made in #1485 would achieve this. I haven't tested 9p, but I think it would work the same as virtio-fs. Where you lose out is if you use devicemapper, the emptyDir will be shared with virtio-fs/9p and you won't benefit from the performance of devicemapper. |
True, but as we discussed, one should use And secondly, I expect virtio-fs to be as performant as block if used with correct parameters, which means you should have equivalent performances. |
@sboeuf @awprice I would like to keep #1485 around, till we move virtio-fs to a stable feature. @awprice I think this patch should fix the issue you are seeing:
Let me know if that works for you. |
With kata-containers#1485, we moved the default medium empty-dir creation to the sandbox rootfs. This worked for devicemapper, but in case of overlay the "local" directory was being created outside the sandbox rootfs. As a result we were seeing the behaviour seen in kata-containers#1818. Fixes kata-containers#1818 Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>
Thanks @amshinde, I will give that a test as soon as possible. |
@amshinde I've tested your PR and can confirm it fixes this issue. When running the |
With kata-containers#1485, we moved the default medium empty-dir creation to the sandbox rootfs. This worked for devicemapper, but in case of overlay the "local" directory was being created outside the sandbox rootfs. As a result we were seeing the behaviour seen in kata-containers#1818. Fixes kata-containers#1818 Signed-off-by: Archana Shinde <archana.m.shinde@intel.com>
Description of problem
When using virtio-fs for the shared_fs, Kubernetes emptyDirs are being provisioned on a memory based filesystem. This is incorrect, as emptyDir should be on a disk.
I believe this is due to the changes made in #1485, where the emptyDir is provisioned on the file system of the sandbox container. This worked fine when using 9p or devicemapper for rootfs, but when using virtio-fs, the rootfs of the sandbox container is being provisioned on a memory filesystem.
Note: we are NOT referring to emptyDir
medium: Memory
, but the default medium for emptyDir.Expected result
emptyDir to be provisioned with virtio-fs and to have the same performance as the rootfs of of the non-sandbox containers.
Actual result
emptyDir is being provisioned in the sandbox container's filesystem, which is memory backed. Thus, the IO performance of emptyDir, is the same as a emptyDir with
medium: Memory
.Just to prove that the emptyDir is being provisioned on memory, I ran the following command in an emptyDir -
sync; dd if=/dev/zero of=./tempfile bs=1M count=102400; sync
, which will create a 100GB file.We can then see the tmpfs for /run now has a 100GB file in it:
Other stuff
The pod I am using to test this:
Show kata-collect-data.sh details
Meta details
Running
kata-collect-data.sh
version1.8.0-alpha1 (commit 7aaf61d44d1e79e792af707387bbbc7e809e98c2)
at2019-06-21.06:50:14.771979354+0000
.Runtime is
/opt/kata/bin/kata-runtime
.kata-env
Output of "
/opt/kata/bin/kata-runtime kata-env
":Runtime config files
Runtime default config files
Runtime config file contents
Output of "
cat "/etc/kata-containers/configuration.toml"
":Output of "
cat "/opt/kata/share/defaults/kata-containers/configuration.toml"
":Config file
/usr/share/defaults/kata-containers/configuration.toml
not foundKSM throttler
version
Output of "
--version
":systemd service
Image details
Initrd details
No initrd
Logfiles
Runtime logs
No recent runtime problems found in system journal.
Proxy logs
No recent proxy problems found in system journal.
Shim logs
No recent shim problems found in system journal.
Throttler logs
No recent throttler problems found in system journal.
Container manager details
Have
docker
Docker
Output of "
docker version
":Output of "
docker info
":Output of "
systemctl show docker
":No
kubectl
No
crio
Have
containerd
containerd
Output of "
containerd --version
":Output of "
systemctl show containerd
":Output of "
cat /etc/containerd/config.toml
":Packages
No
dpkg
No
rpm
The text was updated successfully, but these errors were encountered: