forked from michaeldexter/vmrc
-
Notifications
You must be signed in to change notification settings - Fork 0
/
vmrc-instructions.txt
executable file
·373 lines (274 loc) · 14.7 KB
/
vmrc-instructions.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
Copyright (c) 2012-2014 Michael Dexter <editor@callfortesting.org>
Version: v.0.6.1
Thank you for trying vmrc, the Virtual Machine rc script.
If you did NOT just download this, redownload it for the lastest version
as it is highly-experimental and changing rapidly. See the "TO DO" section
for known issues and feature ideas.
CHANGES from .0.5 to .0.6
* Introduces multiple VM NIC support. Simply add 1-7 in "vm_addtl_nics"
in the VM configuration file. This introduced a limit of 100 VM's but
that can be increased with a documented, one-character change.
* Adds updated pfSense
* Minor cosmetic cleanup
SIGNIFICANT CHANGES from .0.4 to .0.5
* The host_vmroot layout has changed.
* /usr/local/vm/ is now /usr/local/vmrc/vm/
* The "provision" option has been replaced by mkvm.sh which is based on a
library of functions and templates.
* Experimental preflight script and multiple block device support added.
* Experimental ZFS on root support is included but requires the included
zfs-boot components or a recent snapshot.
This has come a long way from a 2005 jail(8) provisioning and management script
to a 2012 FreeBSD 9.0 bhyve build script to a series of sequential FreeBSD 10.*
VM provisioning and management scripts.
Because bhyve is only the latest multiplicity solution on FreeBSD, this
rc script includes basic jail and qemu support and could include chroot, simh
and GXemul support. Of these, QEMU is intended to help prepare OSs like OpenBSD
that require VirtIO and console adjustments prior to booting under bhyve.
### CHEAT SHEET ###
Unpack vmrc.tar or retrieve the latest version from GitHub:
https://github.com/michaeldexter/vmrc
## FreeBSD VMs ##
Run 'sh install-vmrc.sh' or read it and perform the steps manually.
Run templates/install-templates.sh to install all templates into:
/usr/local/vmrc/templates
Verify your host_nic and optional host_zpool in /usr/local/etc/vm.conf
The mkvm.sh script can provision a Virtual Machine based on one of the
included templates in /usr/local/vmrc/templates
mkvm.sh can be used in a captive manner or by passing in a template name and a
VM name. i.e.: sh mkvm.sh t_freebsd10 freebsd which will create a VM named
freebsd followed by an auto-generated number such as 3: freebsd3
Note that all VMs are distinguished by unique trailing ID numbers:
OK: vm0, freebsd1, openbsd2
NOT OK: vm0, freebsd0 and openbsd0
Consider using a numeric word to describe an OS version such as "freebsd-ten3"
Once a VM is provisioned, run:
/usr/local/etc/rc.d/vm (Show usage)
/usr/local/etc/rc.d/vm start (Loads and boots all VMs)
/usr/local/etc/rc.d/vm start vm0 (Start a specific VM)
The vm rc script can also be called with the service(8) command:
service vm start vm0
Alternatively there is "ISO" mode:
/usr/local/etc/rc.d/vm iso vm0 (Fetches and boots and ISO image w/ grub)
"ISO" mode will prepare a disk image to install the OS to and includes fetch
support for FreeBSD, Ubuntu and CentOS.
## Linux VMs ##
Copy a template to a VM directory such as /usr/local/vmrc/vm/linux0/linux0.cfg
and use "ISO" mode:
/usr/local/etc/rc.d/vm iso linux0
### COMMAND USAGE ###
/usr/local/etc/rc.d/vm commands can take in zero or more VM arguments.
'start', 'stop' and 'status' will act on all VMs in /usr/local/vm/ or
multiple VMs i.e.:
/usr/local/etc/rc.d/vm start vm0 vm2 vm3
/usr/local/etc/rc.d/vm status
/usr/local/etc/rc.d/vm status vm3
### COMMANDS ###
Top level commands without a VM name (i.e. /usr/local/etc/rc.d/vm start)
start Load and boot all VMs in /usr/local/vm/
stop Ungracefully terminate all VMs in /usr/local/vm/
status List running VMs
All VMs in /usr/local/vm/ will be launched on boot! (see "CONSOLES" below)
Issue a "reboot" within the VM to shut it down. Cleaner ACPI shutdown is
available after the 10.0-RELEASE and is included in vmrc.
VM-specific commands (i.e. /etc/rc.d/vm attach vm0)
start Load and boot VMs
stop kill and 'destroy' VMs
load Load a VM with bhyveload(8) or grub-bhyve()
boot Boot a VM with bhyve(8)
destroy 'destroy' a VM
restart Run the stop and start functions on a VM
attach Attach to a VM's nmdm serial port or tmux session
debug List bhyvectl(8) --get-all output
iso Format storage device, fetch an ISO image, boot to it
grub Launch grub with the VM's disk and ISO images
mount fsck(8) and mount(8) a FreeBSD VM disk image (UFS only)
umount umount(8) a FreeBSD VM disk image
jail Launch a FreeBSD VM's disk image with jail(8)
format Create and format a FreeBSD VM disk image
delete Delete a VM and its images and directory tree
### VM LAYOUT ###
/usr/local/vmrc/vm/distributions/ OS distribution sets and ISOs
/usr/local/vmrc/vm/templates/ VM templates
/usr/local/vmrc/vm/vm#/vm#.conf VM config file
/usr/local/vmrc/vm/vm#/vm#.img VM raw disk image (format or copy in)
/usr/local/vmrc/vm/vm#/vm#.iso Link to bootable ISO image (fetch or copy in)
/usr/local/vmrc/vm/vm#/mnt/ VM mount point for raw disk image
### CONSOLES ###
The included templates direct each VM's output to an "nmdm" serial device.
Alternatively it can be directed to "stdio" or a tmux session,
"Detached" nmdm and tmux-detached "vm_console" types are available.
All other examples are set to "nmdm" and a bonmdm and tmux-detached consoles
can be attached to with using the "attach" command, i.e.:
/usr/local/etc/rc.d/vm attach vm0
You can detach from nmdm consoles with "~ CTRL-d" and from tmux sessions with
"CTRL-b d".
NOTE: nmdm consoles have the advantage of using the in-base cu(1) command but
had a bug in 10.0R that would cause the VM to boot only upon first attachment.
tmux may be a wiser choice in production environments.
### DEPENDENCIES ###
tmux functionality requires ports/sysutils/tmux
ISO and non-FreeBSD VM support requires ports/sysutils/grub2-bhyve
QEMU functionality requires ports/emulators/qemu(-devel)
Note: Early versions of the grub2-bhyve port include a gcc runtime requirement
which is in fact only a build requirement. This has been fixed.
grub2-bhyve version .22 and later support OpenBSD bsd and bsd.mp kernels but
not the bsd.rd install kernel. Consider flashrd (documentation to come)
### MISC ###
Please send questions, bug fixes, love mail and hate mail to:
Michael Dexter <editor@callfortesting.org>
FreeBSD VMs have their disk images fsck'd before boot given that fsck at boot
time is not always comprehensive and may result in VM boot failure. Jailed VMs
with dirty file systems may cause kernel panics. (bug?)
FreeBSD guests can be provisioned with "iso" mode but it will not set the needed
console. (A "smart" console is undergoing testing that would eliminate the need
for any console changes). For now, just as with vmrun.sh, start a "manual
configuration" console and run:
cat >> /etc/ttys << EOF
console "/usr/libexec/getty std.9600" vt100 on secure
EOF
GNU/Linux users will find the "grub" command very useful for locating disks and
grub configuration files.
A few GNU/Linux-related features did not make it into the FreeBSD 10.0 release:
>3000M RAM support
The -w flag for better CentOS and RHEL performance
### DEVELOPER NOTES ###
GitHub: https://github.com/michaeldexter/vmrc
While patches are welcome, please take the time to understand how this script
works before proposing solutions to problems that might be more complex than
they appear. Ask questions!
vmrc aims to have zero dependencies to operate. It optionally uses external
utilities like grub2-bhyve, tmux, qemu and the like to add functionality but
will hopefully never depend on them.
The vm rc script and mkvm.sh perform two key functions:
1. Programmatically construct virtual machine boot devices (mkvm.sh/"iso" mode)
2. Manage the lifecycle operation of the virtual machines it creates
A "Virtual Machine" is comprised of a directory containing a bootable disk
image, a configuration file derived from a template, a mnt/ directory for the
attachment and mounting of the disk image (FreeBSD only) and an optional
preflight script.
mkvm.sh is largely a simplified FreeBSD installer that is comprised of
functions.
The vm rc script largely creates bhyveload(8) and bhyve(8) command syntax for
loading and booting virtual machines based on the criteria in the configuration
files such as the VM RAM allocation, block device to boot, network interfaces
etc. The vm rc script will output the bhyveload and bhyve command strings that
it generates prior to executing them. If you copy these out, they should work
on their own along with the preflight script if included.
Each VM directory in /usr/local/vmrc/vm/ is fully self-contained and is
only referenced by the vm script looking for VMs in the vm/ directory and then
using the unique numeric VM ID number at the end of the virtual machine.
For example: vm0 has the arbitrary name of "vm" and the VM ID of "0". VMs can
share the same name (vm0, vm1, vm2) but must have a unique ID (vm0, freenas1,
openbsd3). VMs vm0 and linux0 would have conflicting VM IDs.
This naming is extended to the contents of the vm/ directory (vm/vm0/vm0.img)
The VM ID is used to generate unique, non-conflicting tap# network interfaces.
With the introduction of multiple VM virtual network devices, a new convention
was needed to keep VM tap devices unique while preserving their obvious
association with a given VM. The solution is to add a suffix to the tap device,
indicating the NIC ID (vm0's tap0 -> tap01 or vm3's tap3 ->tap31).
This revealed a problem: ifconfig will now allow for tap ID's to be
prefixed with "0" and thus vm0's tap03 would be invalid. To solve this, we use
a double digit VMID that is prefixed with the arbitrary "8", resulting in vm0's
fourth network interface being tap8033 (8 | VM ID | Network Device ID).
Network device "3" is the fourth device because the first one is "0".
### SPECIAL THANKS ###
Thank you Neel, Peter, John and Tycho for your hard work on bhyve.
Thank you Devin for greatly improving this script and Allan at ScaleEngine for
testing it and providing resources for it and bhyve's improvement.
### mkvm.sh TO DO ###
fix: Homogenize templates to master
fix: Make sure devices and zvols are not left attached after installation
fix: ${var_name:=variable} syntax if not set...
fix: Correct rc behavior... vm_enable="YES"
fix: Shorten FreeBSD boot timeout: autoboot_delay="3"
test: Check extraction destinations. zvols will not mount to directories
with .snap in them etc.
ext bug: The PC-BSD automounter will delete a mount point you use. Fix coming.
f_mount does create a directory but may not help all cases.
feature: Implement zfs "mount" and "umount"
feature: Implement make installworld installation
note: Expanding .tar is faster than .txz. Maybe institutionalize.
feature: Bring back 8.* dist set support. Maybe repackage to new .txz layout.
feature: ZFS clone-based storage.
feature: ZFS dataset for each /dev/vm/* for versioning.
feature: getopts? Use bsdconfig-style environment variables?
feature: Have an IP prefix/subnet in vm.conf? Override in mkvm.sh?
feature: Better preparation of VMs for jail use
sendmail_enable="NO"
inetd_flags="-wW -a 192.0.2.23"
rpcbind_enable="NO
feature: beadm?
feature: panicmail, firstboot?
### vm TO DO ###
*NOT* PACKAGE THIS YET - IT IS NOT READY AND IS CHANGING RAPIDLY
note: Tested on 10.0-RELEASE, PC-BSD 10.0 and FreeBSD 11 snapshots
fix: QEMU support - not seeing disk images even if present
fix: vm_destroy after tmux session
news: bhyve automatic console detection is being tested, eliminating tty issues
feature: a "disable" option for VMs to prevent boot upon host boot
test: Try every incorrect sequence of commands and see if checks work
See "# fix: " throughout the script
concern: loaders can be captive and hang: detach loader steps, check flow
feature: disable vm option - move to ~/disabled
missing: mount zpool to ~/vm#/mnt
test: Why are ubuntu VMs read-only upon (first) boot?
test: TrueOS, Does android have VirtIO?
fix: Test: acpi kill in FreeBSD 11-current
linux needs acpi-utils, openbsd doesn't support power button?
hw.allowpowerdown=1 ?
fix: Ubuntu often has a captive loader, halting host system boot
fix: Test: host_vmroot without trailing / (make idiot-proof)
fix: any way to avoid //'s in paths but safely not add trailing one in conf?
fix: CHECK FOR CONFLICTING VM ID NUMBERS BEFORE BOOT
(Consider a PID or otherwise style of keeping VMs unique)
fix: Cosmetic: stalled return after cu session clean up (cleared with ENTER)
fix: Line wrapping!
fix: Formatting: Remaining spaces in place of tabs, for multiline info()?
fix: Formatting: Any way to elegantly wrap long URL's?
fix: Merge f_iso and vm_load/vm_boot
fix: Concern: could a VM booted at startup precede full networking?
fix: Homogenize info/error/debug messages, lose some
fix: Check for a valid tmux session before attempting to attach?
fix: Add zvol and raw device support to grub and load
fix: Destroy VM zvols in delete function
fix: Add root-on-ZFS configuration for when the zfs_boot change is made
fix: Add pty console attachment option (/dev/pts/<n>), try a physical port
fix: Why does second run of 'bhvye' on a loaded vm simply give a new line?
fix: Does grub boot need the CD ISO?
fix: Can grub NOT wipe out all preceding on screen (debug) text?
fix: f_vmmcheck: Is vmm going to be built in by default? (not yet!)
fix: f_vmmcheck: Note left-justified reverse indenting
fix: Remove info's and DEBUG's when satisfied?
fix: NEVER too many checks
fix: Push more checks to separate f_functions?
fyi: bhyve: not loaded first: vm_open: Invalid argument or error 33
fix: Elegant way to use official VM images? (qemu-img conversion works)
qemu-img convert -O raw freebsd.qcow2 freebsd.raw (needs console!)
fix: idea: Dump the load and boot commands plus diagnostic info to a log file.
fix: idea: Add a 'top' command function 'top -uaP' ? How to show only bhyve?
fix: idea: Consider what other commands may want to act on multiple VMs
fix: idea: Cloud! (What was the question?) (ssh is your friend)
fix: idea: Popular jail functionality
fix: idea: Have a separate jail shell-only (no rc) command for maintenance?
fix: idea: Config file hooks for commands at various steps?
fix: idea: "Master" VM zvol cloning?
fix: idea: Add chroot, simh and GXemul support if they have serial output
fix: idea: freebsd-update on boot? (external problem?)
fix: idea: Probe disk layout and fs types? (vmrun.sh probe fails on zvols)
fix: idea: Maybe check for /mnt/.snap on new fs's as a test
fix: idea: Consider an "installworld" method as per the jail man page?
fix: idea: config file-> flags like vm_flags="em0 root ..."
fix: idea: Some flag to indicate "don't boot this VM" (move from dir)?
fix: idea: Add full ext2/3/4 support to the "mount" option (fuse ext4)
Note e2fsprogs and e2fsck /dev/md0s1 and mount -t ext2fs /dev/md0s1 /mnt
Linux ZFS may be very helpful as a LCD FS
fix: Note Neel's vmrun.sh loop:
while [ 1 ]; do
<destroy>
<load>
<boot>
if [ $? -ne 0 ]; then
break
fi # exit if you get a non 0 return
done