docs/bare-metal: Discuss iso ignition embed more (coreos#264)

Came up on IRC - this is such a powerful and cool feature, we
should discuss it in docs.

modules/ROOT/pages/bare-metal.adoc

@@ -14,7 +14,7 @@ NOTE: If you have servers with different types and/or number of hard drives, you
 
 == Installing from live ISO
 
-To install FCOS onto bare metal using the live ISO, follow these steps:
+To install FCOS onto bare metal using the live ISO interactively, follow these steps:
 
 - Download the latest ISO image from the https://getfedora.org/coreos/download?tab=metal_virtualized&stream=stable[download page] or with podman (see https://coreos.github.io/coreos-installer/cmd/download/[documentation] for options):
 [source, bash]
@@ -23,6 +23,8 @@ podman run --privileged --pull=always --rm -v .:/data -w /data \
     quay.io/coreos/coreos-installer:release download -s stable -p metal -f iso
 ----
 
+Note this is just using `coreos-installer` as a tool to download the ISO.
+
 NOTE: You can boot the live ISO in either legacy BIOS or UEFI mode, regardless of what mode the OS will use once installed.
 
 - Burn the ISO to disk. On Linux and macOS, you can use `dd`. On Windows, you can use https://rufus.ie/[Rufus] in "DD Image" mode.
@@ -36,6 +38,8 @@ sudo coreos-installer install /dev/sda \
 
 Once the installation is complete, you can simply `sudo reboot`. After rebooting, the first boot process begins. It is at this time that Ignition ingests the configuration file and provisions the system as specified.
 
+For more advanced ISO installs, including automation, see below.
+
 TIP: Check out `coreos-installer install --help` for more options on how to install Fedora CoreOS.
 
 == Installing from PXE
@@ -113,3 +117,78 @@ IPAPPEND 2
 == PXE rootfs image
 
 include::pxe-artifacts.adoc[]
+
+== Automated ISO/PXE installs with Ignition embedding
+
+The Fedora CoreOS live environment is also CoreOS in the sense that it can boot via Ignition, execute containers, etc.  It includes all of the same content.
+
+As noted above for live PXE, the ISO live environment does not have to actually perform a persistent installation.  You can boot it from a read-only medium such as a physical CD-ROM/DVD, and do everything you do on any other Fedora CoreOS environment.  It also works to boot from a USB stick.  Each boot will re-run the Ignition config, and changes will not persist by default.
+
+For the ISO, the mechanism to do this is `coreos-installer ignition iso embed`, which will create a new `.iso` file that combines your configuration with the ISO.  Similarly, there is `coreos-installer pxe ignition wrap` for the PXE case.
+
+However, many system administrators will want to perform fully unattended persistent installations instead of running stateless.
+
+Some documentation on this is on the upstream installer site: https://coreos.github.io/coreos-installer/customizing-install/[customizing install].
+
+To emphasize, there are *two* Ignition configurations here; the first config ("ISO Ignition") will commonly embed a second rendered configuration that runs on the "target" installation.
+
+First, generate `target.ign` (the file can be named anything) - the configuration that will be passed to `coreos-installer`.  Then, using e.g. `butane`, embed it as a file in your ISO ignition, and use a custom systemd unit to pass it to `coreos-installer`:
+
+[source,yaml]
+----
+variant: fcos
+version: 1.1.0
+storage:
+  files:
+    - path: /etc/target.ign
+      contents: 
+        inline: |
+          Replace this bit with a real butane directive that
+          fetches the target Ignition however you like e.g.:
+          local: target.ign
+      mode: 0644
+systemd:
+  units:
+    - name: my-coreos-installer.service
+      enabled: true
+      contents: |
+        [Unit]
+        Description=Run CoreOS Installer
+        Requires=coreos-installer-pre.target
+        After=coreos-installer-pre.target
+        OnFailure=emergency.target
+        OnFailureJobMode=replace-irreversibly
+
+        # Can be removed if install doesn't reference remote resources with
+        # --stream, --image-url, or --ignition-url
+        After=network-online.target
+        Wants=network-online.target
+
+        [Service]
+        Type=oneshot
+        ExecStart=/usr/bin/coreos-installer install -i /etc/target.ign /dev/sda
+        ExecStart=/usr/bin/systemctl --no-block reboot
+        StandardOutput=kmsg+console
+        StandardError=kmsg+console
+
+        [Install]
+        RequiredBy=default.target
+
+----
+
+Note that this configuration is completely independent of the config passed for the ISO/PXE boot, in the sense that e.g. no systemd units and files written from the live Ignition will be preserved when booted into the final installed system, unless you take explicit action to preserve it.
+
+There is however explicit support for https://docs.fedoraproject.org/en-US/fedora-coreos/sysconfig-network-configuration/#_via_coreos_installer_copy_network[copying network configuration] with `coreos-installer`.
+
+A generally useful technique is to add more systemd units that run before or after the systemd unit that invokes `coreos-installer`.  For example, you can run a systemd unit which pulls a container and does hardware validation.
+
+An example post-install action: Some provisioning systems may require a callback to the PXE server to be switched to "boot from local disk" via a HTTP request; this can similarly be done via a systemd unit that is scheduled `After=my-coreos-installer.service` that uses
+`ExecStart=/usr/bin/curl` or pulling a container which makes the HTTP request.
+
+=== ISO installation on diverse hardware
+
+Commonly bare metal systems will have a diversity of hardware - some systems may have NVMe drives `/dev/nvme*`, whereas others have `/dev/sd*` for example.  You will almost certainly have to template the value of `/dev/sda` above.
+
+A useful approach is to script generating a per-machine `.iso`.  If you have a hardware database (whether a text file in git or relational database) then it will work to generate a per-machine `target-dell.ign` and `target-hp.ign` for example, embed that with the generic `iso.ign` to generate `fedora-coreos-install-dell.iso` and `fedora-coreos-install-hp.iso`.
+
+Alternatively, instead of generating per-machine ISOs, you can have the ISO Ignition pull a privileged container which inspects the target system, and dynamically invokes `coreos-installer`.