vmrc-instructions.txt

################################################################ LICENSE
#
# Copyright (c) 2012-2016,2018 Michael Dexter <editor@callfortesting.org>
#
# Permission to use, copy, modify, and distribute this software for any
# purpose with or without fee is hereby granted, provided that the above
# copyright notice and this permission notice appear in all copies.
#
# THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
# WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
# ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
# WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
# ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
# OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
#
############################################################ INFORMATION
#
# Title: virtual machine rc script instructions
# Version: v.0.9.8
#
# vmrc usage instructions and developer notes
#
########################################################################

Thank you for trying vmrc, the Virtual Machine rc script.

Latest development snapshot: http://vmrc.bsd.lv/vmrc-snapshot.tar
GitHub repository: https://github.com/michaeldexter/vmrc


### PHILOSOPHY ###

"Go away or I will replace you with a very small shell script."

Given the countless large "cloud" options available, this project intends to be
the bare minimum /bin/sh code to manage virtual machines using multiple
hypervisors, emulators and containers.


### OVERVIEW ###

This tool has come a long way from a 2005 jail(8) provisioning and management
script to a 2012 FreeBSD 9.0 bhyve beta build script, to a series of sequential
FreeBSD 10.* VM provisioning and management scripts.

Because bhyve is only the latest multiplicity solution on FreeBSD, this
rc script includes multiple container support including bhyve and Xen,
and aims to include jail, qemu, VirtualBox, simh and GXemul support. While this
notion may be disconcerting to some, you will find that virtual machines care
little about their containers and are increasingly container-agnostic.

vmrc helped bhyve arrive in FreeBSD 10.0 in January of 2014, rather than 11.0.


### REQUIREMENTS ###

The Intel or AMD "POPCNT" (EPT) processor feature is required for bhyve and Xen.

A working IOMMU is required for bhyve PCI pass-through (not implemented in vmrc)
and Xen. Xen is also only available on FreeBSD 11/CURRENT.


### INSTALLATION ###

vmrc does not require ZFS but will create and install to datasets rather than
directories if a pool name is provided. If the directories exist,
install-vmrc.sh will copy in replacement vmrc.conf (the syntax may change),
vm rc script and templates. It will not delete obsolete templates.

Edit the vmrc.conf file PRIOR TO INSTALLATION to include existing directories
or datasets on your system for use with virtual machines managed by vmrc.

These are the default locations:

host_templates="/vmrc/templates/"	# VM Templates
host_distdir="/vmrc/distributions/"	# OS Distributions
host_vmdir="/vmrc/vm/"			# Virtual Machines
host_zpool=""				# Optional Host zpool

Also edit the remaining parameters as needed, notably "bridge0nic" which will
be the network interface used for bridged networking of virtual machines.

Run "sh install-vmrc.sh" as root or read it and perform the steps manually,
adjusting the paths to suit your system. Running it on an installed system
will update/overwrite the installed files, including vmrc.conf.

vmrc currently depends on the sysutils/tmux package for normal operation and
uses the sysutils/grub2-bhyve package to boot VMs with GRUB.

The vm rc script is assumed to reside in /usr/local/etc/rc.d and vmrc.conf is
assumed to reside in /usr/local/etc/ in line with standard practices.


### VIRTUAL MACHINE PROVISIONING ###

The mkvm.sh script provisions Virtual Machines based on the templates in
the templates directory.

Note that the master_template in the templates directory is a runtime
requirement as it is used to zero out virtual machine variables.

mkvm.sh can be used in a captive manner with:

sh mkvm.sh

or by passing in a template name and a VM name. i.e.:

sh mkvm.sh fbsd103iso freebsd 

...which will create a VM named "freebsd#" where "#" is a unique, auto-generated
VM ID number such as "0", resulting in a VM named "freebsd0" in:

/vmrc/vm/fbsdbsd0

Some templates are named based on their configuration variables.

For example, fbsd103amd64_img_gpt_ufs_dist represents:

fbsd103:	FreeBSD 10.3 Operating System
amd64:		AMD64 (64-bit Intel) Architecture (i386 is also supported)
img:		Install to a disk image, rather than zvol
gpt:		Format using the gpart utility
ufi:		Install to a UFS file system
dist:		Install via distribution sets i.e.: base.txz kernel.txz


All vmrc VMs are distinguished by unique trailing ID numbers:

NON CONFLICTING: vm0, freebsd1, openbsd2

CONFLICTING: vm0, freebsd0 and openbsd0 (trailing ID number must be unique)


Once a VM is provisioned, it is managed with the "vm" rc script, hence "vmrc".

All templates are currently set to a tmux console.


To boot a new virtual machine and attach to its serial console:

service vm instead freebsd0 ; service vm oneattach freebsd0


If booting to an ISO installer:

service vm oneinstall freebsd0 ; service vm oneattach freebsd0


If the UEFI-GOP video console is used with a UEFI installation:

service vm onevnc freebsd0


The "one" prefix is not needed if you add:

vm_enable="YES"

to /etc/rc.conf


If the virtual machine cannot be accessed but appears to be hung, it can be
stopped with:

service vm onestop freebsd0


### INSTALL MODE ###

Templates ending with "iso" or "img" are installed by booting to the ISO or
"memstick" image and formatting the storage devices with the OS's native
installer.


### DRY RUN AND DRY BOOT ###

vmrc generates stand-alone scripts to boot a given VM in bhyve, xen or qemu,
named bhyve.sh, xen.sh and qemu.sh respectively. These are also accompanied by
support configuration files such as grub.cfg and device.map in the case of
bhyve GRUB VMs and xen.cfg in the case of Xen VMs. The scripts are run in tmux
or screen upon "start" and attached to with those respective tools with "attach".

vmrc includes also "dryrun" and "dryruninstall" directives, which will only
generate, but not execute the boot script in the VM directory, i.e.: bhyve.sh

This script will prove essential to debugging.


### GRUB MODE ###

The 'grub' directive exists to launch the VM in the GRUB boot loader. This is
particularly useful for determining the on-disk layout of the operating system.

Useful GRUB prompt commands are:

grub> ls		# List devices
grub> ls (cd0)		# Get information for cd0
grub> ls (cd0)/		# List contents of cd0
grub> ls (hd0)		# Get information for hd0
grub> ls (hd0)/		# List contents of hd0
grub> set		# Show set arguments

linux (cd0)/vmlinuz
initrc (cd0)/initrd.gz
boot

kopenbsd -h com0 (cd0)/5.6/amd64/bsd.rd
boot

knetbsd -h -r cd0a (cd0)/netbsd
boot

In a perfect world, GNU/Linux distros would use a standard location for
vmlinuz and initrd.gz, but this is not that world. BSD gets this right.

Debian/Ubuntu GNU/Linux are some of the simplest distributions to configure:

iso_grub_r="-r cd0"		# Default to dc0 for ISO boot
img_grub_r="-r hd0,msdos1"	# Default to hd0 for img boot

grub-bhyve then correctly finds /boot/grub/grub.cfg on these devices as they
are the default GRUB locations.

If your VM OS (such as OpenBSD, NetBSD and many GNU/Linux distributions)
requires an external grub.cfg, a \n separated one can be added to the template.
For example:

iso_grub_cfg="linux (cd0)/vmlinuz\ninitrc (cd0)/initrd.gz\nboot" 

This will auto-generate a ~/grub.cfg file with:

linux (cd0)/vmlinuz
initrc (cd0)/initrd.gz
boot

vmrc will also auto-generate a device.map file:

/usr/local/vmrc/vm/centos7amd64iso0/device.map

Whic reads:

(hd0) /usr/local/vmrc/vm//centos7amd64iso0/centos7amd64iso0.img
(cd0) /usr/local/vmrc/vm//centos7amd64iso0/centos7amd64iso0.iso

Centos7 GRUB with the defice.map file:

grub> ls
(hd0) (hd0,msdos2) (hd0,msdos1) (cd0) (cd0,msdos2) (host) (lvm/centos-root) (lv
m/centos-swap)

Without the device.map:

grub> ls
(host)

If a grub.cfg is not supported by a given GNU/Linux distribution for ISO boot,
the arguments can be entered in at the grub prompt or be read from grub.cfg
on the host file system.

For example:

/usr/local/vmrc/vm/centos7amd64iso0/grub.cfg

linux (cd0)/vmlinuz
initrc (cd0)/initrd.gz
boot

Once installed, the CentOS 7 grub.cfg exists in:

(hd0,msdos1)/grub2/grub.cfg

The grub.cfg arguments to use this would be:

/usr/local/sbin/grub-bhyve -r hd0,msdos1 -d /grub2/ \
-m /usr/local/vmrc/vm/centos7amd64iso0/device.map -M 1024 centos7amd64iso0

Other examples:

(cd0)/isolinux/grub.conf

(cd0)/EFI/BOOT/grub.cfg

If an external grub.cfg is used, the -r command can be used to specify the
boot drive, i.e. -r hd0,msdos1 and the commands can be issued without the
(hd0,msdos1) prefix.

Note that GNU/Linux paths may vary depending of the type of file system you use.

grub-bhyve kfreebsd support is experimental.

Tip: 'set pager=1' will give you a "more/less" style pager in GRUB.


### DIRECTIVES ###

Top level commands without a VM name (i.e. /usr/local/etc/rc.d/vm start)

start		Load and boot all VMs in /usr/local/vm/
stop		Ungracefully terminate all VMs in /usr/local/vm/

"reboot" and "shutdown" within VM should work as expected.
An ACPI shutdown is available after the 10.0-RELEASE and is included in vmrc.

VM-specific commands (i.e. /etc/rc.d/vm attach vm0)

start		Load and boot VMs
stop		kill and 'destroy' VMs
attach		Attach to a VM's nmdm serial port, tmux or screen session
vnc		Attach to a VM's VNC session
dryrun		Only generate VM configuration files and boot script
dryboot		Boot the generated boot script
dryruninstall   Same as "dryrun" but in "install" mode
dryrunlivecd	Same as "dryrun" but in live CD mode
install		Boot the VM in "install" mode
grub		Launch grub with the VM's disk and ISO images
bhyve		Override "vm_container" with bhyve
bhyveinstall	Same as "bhyve" but in install mode
qemu		Override "vm_container" with qemu
xen		Override "vm_container" with Xen
xeninstall	Same as "xen" but in install mode
list		List all VMs in the VM directory
status		List running bhyve VMs


### vmrc/VM LAYOUT ###

/vmrc/distributions/		OS distribution sets and ISOs
/vmrc/templates/		VM templates
/vmrc/vm/<vm#>/vm.conf		VM configuration file
/vmrc/vm/<vm#>/diskN.img	VM raw disk image (format or copy in)
/vmrc/vm/<vm#>/install.iso|img	Soft link to bootable ISO image


### CONSOLES ###

Because bhyve console output is inherently captive, vmrc supports several
"console" options:

tmux	Output the VM console to a detached tmux session
screen	Output the VM console to a detached screen session


To attach to a tmux or screen console, use the "attach" directive:

/usr/local/etc/rc.d/vm oneattach freebsd3

To detach from each console, type:

tmux: "CTRL-b d"
screen: "C-a DD"
nmdm: "~ CTRL-d"


### NETWORKING ###

By default, vmrc uses on network bridge on one interface on the host as defined
in /usr/local/etc/vmrc.conf :

bridge0_nic="em0"		# Host's active NIC for bridged networking

Should you want to use multiple bridges that are associated with multiple host
interfaces, you should be able to simply add these arguments to the VM template
as they are sourced sequentially.


### DEPENDENCIES ###

tmux functionality requires ports/sysutils/tmux
screen functionality requires ports/sysutils/screen
vnc functionality requires ports/net/tightvnc
Non-FreeBSD VM support requires ports/sysutils/grub2-bhyve 
QEMU functionality requires ports/emulators/qemu(-devel|-sbruno)
Xen functionality requires ports/emulators/xen and ports/net/tightvnc
UEFI firmware can be found in ports/sysutils/bhyve-firmware


### TEMPLATES ###

The template naming attempts to convey as much information about a VM in as
little space possible. VM names over 31 characters will cause bhyve load to
give the error:

vm_create: Invalid argument

The simple solution is to not use the full template name as a VM name and note
that the automatic numbering done by mkvm.sh will add characters to the end of
a given VM name.

The master_template includes all VM arguments but the examples
will make more sense. 'diff' similar FreeBSD ones to see how they differ.

Many templates have OS-specific notes at the bottom in the Comments section.

Note the "required" fields in the templates. They are required.


### MISC ###

Please send questions, bug fixes, love mail and hate mail to:
Michael Dexter <editor@callfortesting.org>

FreeBSD guests can be provisioned with "iso" mode but it will not set the needed
console. (A "smart" console arrived mid-June 2014 and is available in FreeBSD
CURRENT and some STABLE branches.

cat >> /etc/ttys << EOF
ttyu0 "/usr/libexec/getty 3wire.9600" vt100 on secure
EOF

GNU/Linux users will find the "grub" command very useful for locating disks and
grub configuration files.

A few GNU/Linux-related features did not make it into the FreeBSD 10.0 release:

>3000M RAM support
The -w flag for better performance on many OSs including CentOS and RHEL.


### bhyve OVERVIEW ###

The bhyve hypervisor consists of the following components:

/boot/kernel/vmm.ko  Kernel module
/dev/vmm/*           Index of loaded virtual machines
/usr/sbin/bhyveload  Userspace FreeBSD loader
/usr/sbin/bhyve      Virtual Machine boot utility
/usr/sbin/bhyvectl   Virtual Machine shutdown and diagnostics utility
/usr/local/sbin/grub-bhyve  Optional userspace GRUB bootloader

Both bhyveload(8) and bhyve(8) are provided the VM memory allocation and a
bootable block device in the form of a /dev/ device or a disk image plus a
virtual machine name. These must be consistent between both utilities.

bhyve(8) is provided additional arguments describing PCI devices (notably for
VirtIO storage and network) plus miscellaneous operational parameters. 

A bhyve(8) virtual machine is shutdown with a reboot(8) command and the
/dev/vmm/<vmname> entry is cleaned up with the bhyvect(8) command.

VM network is primarily achieved by providing an if_tap(4) network device that is
associated with an if_bridge(4) bridge. Alternatively a hardware network device
could be passed to the virtual machine but this is not supported by vmrc.

The first bhyve-ready FreeBSD snapshot "raw" VM-IMAGEs arrived on June 15th,
2014 and do not require any modification for use with bhyve. vmrc has templates
for these raw images and they can also be used with the
/usr/share/examples/bhyve/vmrun.sh script that is included with FreeBSD. These
images can be found at ftp://ftp.freebsd.org/snapshots/

Alternatively, virtual machines can be built in a manner similar to jail(8)
systems or hardware-based systems. The FreeBSD ISO install images can be used
for this purpose and this the approach taken by vmrun.sh. Note that all but the
most recent FreeBSD 11.0-CURRENT and 10.0-STABLE releases require a /etc/ttys
modification to work properly. vmrc will perform this modification as needed for
most FreeBSD versions and FreeBSD-derived operating systems.

The vmrc template syntax should be self-explanatory and attempts to accommodate
most desired system configuration options short of complex partitioning and the
creation of users or installation of ports and packages. Once built and booted,
a bhyve virtual machine behaves like a normal system and can be operated with
your tools of choice.


### DEVELOPER NOTES ###

vmrc Design Goals

* "Unix" design philosophy, simple architecture and transparent operation
* Rapid, flexible development using in-base tools
* Zero dependencies for basic operation with significant-value add-ons
* Fundamentally-portable design
* Open architecture to accept additional multiplicity strategies

While patches are welcome but please first submit a feature request or take the
time to understand how this script works before proposing solutions to problems
that might be more simple or complex than they appear. Ask questions!

vmrc aims to have zero dependencies to operate. It optionally uses external
utilities like grub2-bhyve, tmux, qemu, screen and the like to add
functionality but will hopefully never depend on them.

The vm rc script and mkvm.sh perform two key functions:

1. Programmatically construct virtual machine boot devices (mkvm.sh/"iso" mode)
2. Manage the lifecycle operation of the virtual machines it creates

A "Virtual Machine" is comprised of a "vm.conf" configuration file, a soft link
to an install ISO or image "install.iso|img", a boot disk "disk0.img", optional
additional disks i.e. disk1.img and the generated boot script i.e. bhyve.sh,
xen.sh or qemu.sh.

mkvm.sh is largely a simplified FreeBSD installer that is comprised of
preflight, primary and debug functions.

The vm rc script primarily creates bhyveload(8) and bhyve(8) command syntax for
loading and booting virtual machines based on the criteria in the configuration
files such as the VM RAM allocation, block device to boot, network interfaces
etc. The vm rc script will output the bhyveload and bhyve command strings that
it generates prior to executing them. If you copy these out, they should work
on their own along with the preflight script if included.

Each VM directory in /usr/local/vmrc/vm/ is fully self-contained and is
only referenced by the vm script looking for VMs in the vm/ directory and then
using the unique numeric VM ID number at the end of the virtual machine.

For example: vm0 has the arbitrary name of "vm" and the VM ID of "0". VMs can
share the same name (vm0, vm1, vm2) but must have a unique ID (vm0, freenas1,
openbsd3). VMs vm0 and linux0 would have conflicting VM IDs.

This naming is extended to the contents of the vm/ directory (vm/vm0/vm0.img)

The VM ID is used to generate unique, non-conflicting tap# network interfaces.

With the introduction of multiple VM virtual network devices, a new convention
was needed to keep VM tap devices unique while preserving their obvious
association with a given VM. The solution is to add a suffix to the tap device,
indicating the NIC ID (vm0's tap0 -> tap01 or vm3's tap3 ->tap31).

This revealed a problem: ifconfig will now allow for tap ID's to be
prefixed with "0" and thus vm0's tap03 would be invalid. To solve this, vmrc 
employs a double digit VMID that is prefixed with the arbitrary "8", resulting
in vm0's fourth network interface being:

tap8033 [ 8 | VM ID of 3 | Network Device ID of 4 (counting from 0) ]

Network device "3" is the fourth device because the first one is "0".


### SPECIAL THANKS ###

Thank you Neel, Peter, John B. and Tycho for your hard work on bhyve.

Thank you Devin for greatly improving this script, Bryan V for valuable input,
and Allan at ScaleEngine for testing it and providing resources for it and
bhyve's improvement.

We miss you Ben.