Update documentation to reflect PR #1

jbtrystram · Oct 17, 2023 · 69ab115 · 69ab115
1 parent cdcc195
commit 69ab115
Show file tree

Hide file tree

Showing 5 changed files with 114 additions and 37 deletions.
diff --git a/README.md b/README.md
@@ -1,57 +1,134 @@
-# Create SCSI LUN disks in container
+# CoreOS iscsi test setup
 
-Use targetcli in containers.
-This **MUST** be run as root. 
+This repo contains a few bits needed to boot a coreOS VM from an iSCSI root disk. 
+This is still a work in progress
 
-## Build the image
+## Overview
+
+```mermaid
+flowchart BT
+    A(iSCSI Target)
+    B[FCOS VM] --> |container volume| A
+    C(COSA) --> D
+    C --> |virtioFS| B 
+    D[diskless VM] <--> |netboot| A
+    R{HOST RAW image} --> |container volume| C
+```
+
+## target container
+
+Targetcli in a container. It requires root privileges. 
+
+### Build the image
 ```bash
 # podman build -t targetcli -f Dockerfile .
 ```
 
-## Volumes
-The config in `saveconfig.json` expose `/var/shared/fcos.raw` as a fileio backstore.
-See [`targetcli commands`](targetcli_script) to see the details.
-My use case here is to mount a CoreOS raw image disk in a QEMU VM through virtiofs. This VM act as the iSCSI server.
-
-## Run the container and exec to create the disks
+### Run
+If you want to run the container on its own (without through the ignition config provided), here is the required arguments: 
 ```bash
 # podman run -d \
 	--name targetclid \
 	-v /lib/modules:/lib/modules \
 	-v /var/run/dbus:/var/run/dbus \
     -v /sys/kernel/config:/sys/kernel/config \
-	-v /var/shared/fcos.raw:/fcos.raw \
 	--privileged --cap-add=CAP_SYS_MODULE \
 	--net host \
+	-v /var/shared/fcos.raw:/fcos.raw \
 	quay.io/jbtrystram/targetcli
 ```
+then `podman exec -it targetclid targetcli`, you can then create portals, luns and all iSCSI things.
 
 Note: `--net host` is used here because `-p 3260:3260` made the LUN to bind to the container network address.
 
-## Mounting the iSCSI target
 
-The target is `iqn.2023-09.centos.vm:coreos`. No auth is required. 
+## CoreOS Virtual machine
+
+Here we focus on running the iSCSI target container in a virtual machine created through [cosa](https://coreos.github.io/coreos-assembler/cosa/run/)
+
+### Components
+The `ignition` folders contains the necessary butane configuration to launch and configure everything:
+ - `target.container` sets up the container explained above as a systemd service.
+ - `setup-targetcli.service` sets up a systemd service to execute `targetcli_script`, only on the first boot and only once.  
+ - `targetcli_script` is simply a bash script that execute `targetcli` commands inside the container to configure it.
+ - `target.bu` ties it all together, essentially copying the above bits in the correct places.
+
+
+### Pre-requisites
+
+You will need a metal image of fedora coreOS and a regular image :
+```
+cosa fetch && cosa build && cosa buildextend-metal
+```
+Then, copy the metal raw file in the tmp workdir, expand it and make it writeable: 
+```
+cp builds/latest/x86_64/fedora-coreos-*.raw tmp/fcos.raw
+chmod 664 temp/fcos.raw
+```
+Neatly, `cosa run` will mount `tmp/` at `/var/mnt/workdir-tmp` in the VM through virtioFS.
+
+### Start the VM
+
+Build the butane config:
+```
+ butane --pretty --strict target.bu -d . > target.ign
+```
+
+Then `cosa run -i /path/to/target.ign`
+
+At this point, the iSCSI target setup is complete.
+
+## Testing
+
+Now, we create a second VM, which should boot from the iSCSI target.
+We'll need an iPXE boot script: 
+``` 
+$ cat /mnt/workdir-tmp/boot.ipxe
+#!ipxe
+sanboot iscsi:10.0.2.15::::iqn.2023-10.coreos.target.vm:core
+```
+
+Then boot !
+```
+podman run -ti --privileged --net=host --rm -v /mnt/workdir:/mnt/workdir \
+quay.io/coreos-assembler/coreos-assembler shell -- \
+kola qemuexec --netboot /mnt/workdir/tmp/boot.ipxe --usernet-addr 10.0.3.0/24 -boot order=n
+```
+Note : you will need https://github.com/coreos/coreos-assembler/pull/3645
+
+Without the above PRs : 
+```
+podman run -ti --privileged --net=host --rm -v /mnt/workdir:/mnt/workdir \
+quay.io/coreos-assembler/coreos-assembler shell -- \
+kola qemuexec -- -netdev user,id=iscsi,tftp=/mnt/workdir-tmp/,bootfile=boot.ipxe,net=10.0.3.0/24 -boot n
+```
+
+
+At this point, you should get a GRUB screen, pulled from that initial `metal raw` file, booted off iscsi ! Happy hacking.
+
+## Manual iscsi mount
+
+The setup target in the ignition config is `iqn.2023-10.coreos.target.vm:coreos`. No auth is required. 
 So on the initator, to mount the volume
 Discover : 
 ```
 # iscsiadm -m discovery -t st -p 192.168.122.2
-10.88.0.2:3260,1 iqn.2023-09.centos.vm:coreos
+10.88.0.2:3260,1 iqn.2023-10.coreos.target.vm:coreos
 ```
-
 Mount :
 ```
-sudo  iscsiadm -m node -T iqn.2023-09.centos.vm:coreos -l
+sudo  iscsiadm -m node -T iqn.2023-10.coreos.target.vm:coreos -l
 Logging in to [iface: default, target: iqn.2023-09.centos.vm:coreos, portal: 192.168.122.2,3260]
 Login to [iface: default, target: iqn.2023-09.centos.vm:coreos, portal: 192.168.122.2,3260] successful.
 ```
 
 Then find where it was mounted :  `journalctl --no-pager | grep "Attached SCSI"`
-The core OS disk iamge : 
+The core OS disk image : 
 ```
-sudo blkid /dev/sdc*                                                                                                                                                         main  ✭ ✱
+sudo blkid /dev/sdc*
 /dev/sdc: PTUUID="a1c653e9-8857-4eb1-b69b-6f53952a569d" PTTYPE="gpt"
 /dev/sdc1: PARTLABEL="BIOS-BOOT" PARTUUID="6498509e-b163-4660-b067-04ea5db88b1d"
-/dev/sdc2: SEC_TYPE="msdos" LABEL_FATBOOT="EFI-SYSTEM" LABEL="EFI-SYSTEM" UUID="3E45-2746" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI-SYSTEM" PARTUUID="60f508bc-0dee-4355-b052-aae90650b550"
+/dev/sdc2: SEC_TYPE="msdos" LABEL_FATBOOT="EFI-SYSTEM" LABEL="EFI-SYSTEM" UUID="E45-2746" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI-SYSTEM" PARTUUID="60f508bc-0dee-4355-b052-aae90650b550"
 /dev/sdc3: LABEL="boot" UUID="78ec57df-7912-4cf1-b9c9-2a782bf6ea83" BLOCK_SIZE="1024" TYPE="ext4" PARTLABEL="boot" PARTUUID="3f1a5f26-36e1-4411-807c-b579d20d855d"
 /dev/sdc4: LABEL="root" UUID="0f296b65-e139-4729-9890-c9312eb91bbd" BLOCK_SIZE="512" TYPE="xfs" PARTLABEL="root" PARTUUID="0a29aecb-1b5b-450b-9964-1760b3b72acf"
 ```

diff --git a/ignition/README.md b/ignition/README.md
@@ -25,17 +25,13 @@ IGNITION_FILE=path/to/target.ign
 # virt-install --connect="qemu:///system" --name="fcos-target" --vcpus=2 --memory=2048                      \
     --os-variant="fedora-coreos-stable" --import --graphics=none                                            \
     --disk="size=10,backing_store=${FCOS_IMAGE}"                                                            \
-    --network bridge=virbr0 --qemu-commandline="-fw_cfg name=opt/com.coreos/config,file=${IGNITION_FILE}"     \
+    --network bridge=virbr0 --qemu-commandline="-fw_cfg name=opt/com.coreos/config,file=${IGNITION_FILE}"   \
     --filesystem=${RAW_DISK},var-shared,driver.type=virtiofs                                                \
     --memorybacking=source.type=memfd,access.mode=shared
 ```
 
 Here we use the virtioFS filesystem to mount the coreOS raw disk image into the VM. 
 
-# TODO / ideas
-
-- [] iPXE boot script
-- [] add a DHCP service as well (to help iPXE boot)
 
 # Resources
 

diff --git a/ignition/setup-targetcli.service b/ignition/setup-targetcli.service
@@ -0,0 +1,14 @@
+[Unit]
+Description=Setup targetcli
+ConditionPathExists=/var/mnt/workdir-tmp/fcos.raw
+Requires=target.service
+After=target.service
+ConditionFirstBoot=true
+
+[Service]
+Type=oneshot
+RemainAfterExit=yes
+ExecStart=/usr/local/bin/targetcli_script
+
+[Install]
+WantedBy=multi-user.target
diff --git a/ignition/target.bu b/ignition/target.bu
@@ -13,16 +13,4 @@ systemd:
   units:
     - name: setup-targetcli.service
       enabled: true
-      contents: |
-          [Unit]
-          Description=Setup targetcli
-          ConditionPathExists=/var/mnt/workdir-tmp/fcos.raw
-          Requires=target.service
-          After=target.service
-          ConditionFirstBoot=true
-          [Service]
-          Type=oneshot
-          RemainAfterExit=yes
-          ExecStart=/usr/local/bin/targetcli_script
-          [Install]
-          WantedBy=multi-user.target
+      contents_local: setup-targetcli.service
diff --git a/ignition/targetcli_script b/ignition/targetcli_script
@@ -7,3 +7,5 @@ podman exec target bash -exc "
   targetcli iscsi/iqn.2023-10.coreos.target.vm:coreos/tpg1/ set attribute authentication=0 demo_mode_write_protect=0 generate_node_acls=1 cache_dynamic_acls=1
 "
 
+# Will return 0 if the discovery yield a valid portal
+iscsiadm -m discovery -p 127.0.0.1 -t st | grep iqn.2023-10.coreos.target.vm:coreos