Ability to Store VM images on NAS

[lanycrost]

Currently, tart utilizes the fcntl system call for locking VMs, which limits compatibility with SMB or AFP. Enabling the ability to run VMs on shared storage would be highly valuable. This enhancement would allow for storing VM images for multiple machines in a centralized location, alleviating concerns about individual device storage limitations.

Pulling machine images from a registry isn't an ideal option due to image sizes ranging from 30 to 50GBs and even 100GBs, resulting in prolonged retrieval times and significant network consumption. I suggest considering the use of files for locking up VMs, with the option to make it configurable.

Thanks!

https://chris.improbable.org/2010/12/16/everything-you-never-wanted-to-know-about-file-locking/

[edigaryev]

Hello Khachatur 👋

Unfortunately, the Tart right now is pretty much married to file locking: this allows it to be 100% daemon-less, which is what makes it so simple on the outside and easy to begin building with.

Without both file locking or running a daemon (dockerd-fashion, etc), it's pretty hard to guarantee operation atomicity and almost impossible implement things like tart stop, as the latter command uses fcntl(2)-based locking to find out the tart run's process PID.

While we could limit the fcntl(2) usage only to tart run and tart stop and give an option to avoid placing a fcntl(2) lock in tart run, doing this this will effectively make the tart stop a no-op, which might be confusing for some people.

Which NAS protocol are you planning to use? NFSv3? NFSv4?

Also, as a work around/experiment you can try to symlink the local VM's disk file to the VM's disk file on NAS (as the disk is not locked by Tart) to see if the over-the-network performance is tolerable for you at all.

[lanycrost]

I'm trying to use SMB and AFP. You've given me an idea: I'll try using an image file because you're only locking the config file. Maybe that will work. After trying it out, I'll give you an update.

Thanks!

[lanycrost]

I've tried to use only disk.img file from SMB and APF share (using symbolic links), unfortunately it's not working. The nvram.bin and config.json is stored on local disk (because you are locking config.json).

Thanks!

[fkorotkov]

@lanycrost how do you envision your workflow? You plan to clone VMs within the NAS and run it from multiple Macs? Do you have a single NAS instance? Are you concerned about network limits in case of running multiple VMs in parallel?

[lanycrost]

Yes, as you described. Mac Minis support 10Gbps links so most probably it will work I guess I need more deep research to understand the bottlenecks of this approach, but theoretically running 50Gb image with maximum speed will take about 1 minute to bring up the machine.

[fkorotkov]

It seems that tart stop functionality relies on fcntl since we "store" process id for a given VM.

If you have a folder with your images on a remote NAS then you can symlink it to ~/.tart/cache/OCIs/ghcr.io/foo/bar. Then Tart will think that you VM name on a NAS is a tag for ghcr.io/foo/bar so you'll be able to do:

tart clone ghcr.io/foo/bar:nas-vm local-vm`

Which will theoretically copt over the VM from the NAS without fcntl.

[lanycrost]

It's really great idea, guess it will work :D will try to use this approach!

[fkorotkov]

Will close the issue for now. But please share how it will work for you in terms of IO.

We have some thoughts about how to support launching ephemeral VMs of a remote storage like S3 and so on but haven't experimented with it yet. Waiting on update from Apple during WWDC before investing time into it.