OKD/OpenShift install on Baremetal: Proxmox/QEMU

[GannonTdW]

Hello,

This is more a question than a bug report, but I didn't really find where to put this question

I would like to install OKD (or OpenShift) on my home Lab proxmox.
I have configured all DNS records on my OPNsense and I have the oc client and the openshift-install.

But I don't understand how to create VMs.

With the openshift-instal create manifest|cluster command, I have the choice between azure, gcp, openstack, vsphere ... but not proxmox / qemu.

In some old blog post or the lastest okd doc (installation-initializing-manual_installing-bare-metal), I see mention of a install-config.yaml file but I don't have it in openshift-install-linux-4.12.0-0.okd-2023-01-21-055900.tar.gz

I simply create it based on #27 ?

Thank you for your help.

PS: To be closer to what I have at work, I would like to use Fedora CoreOS

[tothlali]

My install-config.yaml (3 masters, 2 workers) (service.okd.local is the machine where oc runs) :

apiVersion: v1
baseDomain: local
compute:
- hyperthreading: Enabled   
  name: worker
  replicas: 2 
controlPlane:
  hyperthreading: Enabled   
  name: master 
  replicas: 3 
metadata:
  name: okd 
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14 
    hostPrefix: 23 
  networkType: OpenShiftSDN
  serviceNetwork: 
  - 172.30.0.0/16
platform:
  none: {} 
pullSecret: '{"auths": {"fake": { "auth" : "bar"}}}' 
sshKey: 'ssh-ed25519 ... root@service.okd.local' 

[titou10titou10]

Mine.
Currently Proxmox 7.3-4 + OKD v4.12.0-0.okd-2023-01-21-055900 upgraded from v4.9
VM nodes: 1 bootstrap, 3 masters, 3 workers, created with no OS in Proxmox, all FCOS installed by the OKD installer with iPXE
VM other: 1 load balancer (API+Ingress/Routes+iPXE): AlmaLinux v9.1 + nginx, 1 "utility" VM with AlmaLinux v9.1 where the OKD installer run

apiVersion: v1
baseDomain: denis.prive
metadata:
  name: oc4
compute:
- hyperthreading: Enabled   
  name: worker
  replicas: 0 
controlPlane:
  hyperthreading: Enabled   
  name: master 
  replicas: 3 
networking:
  clusterNetwork:
  - cidr: 10.128.0.0/14 
    hostPrefix: 23 
  networkType: OVNKubernetes
  serviceNetwork: 
  - 172.30.0.0/16
platform:
  none: {} 
fips: false 
sshKey: 'ssh-rsa AAAA{...}' 
pullSecret: '{"auths":{"cloud.openshift.com":"auth":{...},"quay.io":"auth":"{...},"registry.connect.redhat.com":{"auth":"{...}},"registry.redhat.io":{"auth":"{...}"}}}' 

[GannonTdW]

Thank you for your answer.
But where do we fill in the credentials for Proxmox?
Is it possible to use the terraform/proxmox provider?

Is there a documentation on this file somewhere?
Apart from the 2 following links I can't find much:

• https://docs.okd.io/4.11/installing/installing_openstack/installing-openstack-installer-restricted.html#installation-osp-restricted-config-yaml_installing-openstack-installer-restricted
• https://docs.okd.io/4.11/installing/installing_openstack/installing-openstack-user.html

[titou10titou10]

What Proxmox credential?
The link you provided points to OpenStack..Are you installing OpenStack, and then OKD on OpenStack?
Sorry, I can't help as my install is not OKD on OpenStack on Proxmox.... Just OKD UPI (User Provisioned Infrastructure) on "bare" Proxmox VM
The ref for that is https://docs.okd.io/latest/installing/installing_bare_metal/installing-bare-metal.html

[tothlali]

Dunno. I made the 'service' machine (everything: haproxy,dns,pxe-server,tftp,...) and nodes (bootstrap, master, worker) were booted by PXE with the appropriate ignition-file and coreos-image. I didn't use coreos-installer.

[titou10titou10]

if I understand correctly, I have to create each FCOS VM on proxmox with the iso obtained by

openshift-install coreos print-stream-json | grep '\.iso[^.]'

Boot them and run the command

sudo coreos-installer install --ignition-url=http://<HTTP_server>/<node_type>.ign <device> --ignition-hash=sha512-<digest>  

Not in my case as I use iPXE. As said before, I create VM with no OS (and option to boot on net first) and with a specific mac address for each. my DHCP server is configure to assign an ip and name for each mac address. For the nodes and bootstrap, create iPXE install scripts (1 bs, 1 master, 1 worker) upload the iPXE config files with the FCOS "live-kernel", "initramfs" and "rootfs" iso to the web server . I use the same web server (with nginx) that will be used for load balancing later, then boot from Proxmox all the nodes. open the console of each node (in order bs,then masters, then workers) and boot from the iPXE file. basically

CTRL+B
dhcp
chain http://<web server>/dhcp_bs.ipxe (or dhcp_master.ipxe or dhcp_worker.ipxe)

and the node will install themselves, as the masters and workers and fetch and update the FCOS version is needed
my ipxe file for the masters uploaded close to the FCOS images :

#!ipxe
set p_version 34.20210904.3.0 <- version from "./openshift-install coreos print-stream-json | grep coreos"
set p_role master
set p_base http://<web server>:9080
kernel ${p_base}/fedora-coreos-${p_version}-live-kernel-x86_64 \
       rd.neednet=1 console=tty0 \
       coreos.inst=yes \
       coreos.inst.install_dev=sda \
       coreos.inst.ignition_url=${p_base}/${p_role}.ign
initrd ${p_base}/fedora-coreos-${p_version}-live-initramfs.x86_64.img 
initrd ${p_base}/fedora-coreos-${p_version}-live-rootfs.x86_64.img
boot

Instead of assign the mac address to the ip+host name for your nodes in your DHCP server, you can create one ipxe file per node, and add to the kernel arguments the detail for network. eg

kernel ${p_base}/fedora-coreos-${p_version}-live-kernel-x86_64 \
    ip=192.168.1.<ip>::192.168.1.1:255.255.255.0:<hostname>:ens18:none nameserver=<nameserver ip> \
    {...}

[tothlali]

I made very similarly like titou10, but my network has original DHCP server which is out of my land, so cannot modify its config. So I used the ipxe.iso to boot nodes.

[GannonTdW]

Hello @titou10titou10,
At first I didn't want to use the solution with pxe because I don't know how to do that.
But why not, I will know how to do it.

I think I understood almost all your message, but there is a point where I wonder.
You enter manually

CTRL+B
dhcp
chain http://10.10.10.10/dhcp_bs.ipxe

It's ok when you have 2 or 3 VM but it will quickly become boring.
Is there any way to do this automatically?

proxmox template provider it can create an empty vm and activate the options for the pxe boot; but it does not allow to fill the string "dhcp; chain http://10.10.10.10/dhcp_bs.ipxe".

Thank you for your help

PS:

• all DNS, DHCP, LB (haproxy) is config on OPNsense
• I have create a lxc with nginx for all ipxe script and *.img

[GannonTdW]

@titou10titou10 @tothlali ,

ok I have found how send *.ipxe file with the OPNsense DHCP service.

OPNsense Web UI -> Services -> DHCPv4 -> Network Interface -> Network booting :

• Enable network booting
• Set iPXE boot filename : http://10.10.10.10/dhcp_bootstrap.ipxe

The only problem at the moment is that you can only put one file.