Microstack
Microstack is Snap based OpenStack distribution.
- Homepage: https://microstack.run/docs/single-node
- Source: https://opendev.org/x/microstack
- Snap page: https://snapcraft.io/microstack
WARNING!
There are currently two important things to know:
- current version - 245 (tested on Mar 2023) has several issues (see Troubleshooting)
- The project is already 1 year inactive (last Snap published on 26 January 2022)
If you want to use living (and working) single-node OpenStack distribution please follow rather DevStack Quick Start
Currently Ubuntu 20.04 LTS or 22.04 LTS required and:
- at least 8 GB RAM (tested 32 GB) - Azure "Size" (instance type):
Standard D8s v3 (8 vcpus, 32 GiB memory) - the more CPU cores the better
- should support virtualisation - accessible device
/dev/kvm - officially 100GB of disk space (however I play with tiny instances on 32GB disk)
Theoretically the installation is very simple - by following https://microstack.run/docs/single-node - in 3 steps:
-
Install snap:
$ sudo snap install microstack --beta microstack (beta) ussuri from Canonical✓ installed $ snap list microstack Name Version Rev Tracking Publisher Notes microstack ussuri 245 latest/beta canonical✓ -
-
Initialize Microstack (you must start with Control node and can later add more Compute nodes):
- WARNING! Before setup below,
You should have properly working HOSTNAME and FQDN, so these commands have to work properly:
Be sure that Request FQDN matches Response FQDN:
ping `hostname` ping `hostname -f`
$ ping `hostname` # vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv - forward FQDN PING hp-mstack20.internal.cloudapp.net (10.101.0.5) 56(84) bytes of data. # vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv - reverse FQDN - must MATCH! 64 bytes from hp-mstack20.internal.cloudapp.net (10.101.0.5): icmp_seq=1 ttl=64 time=0.017 ms
$ sudo microstack init --auto --control 15:00:53,807 - microstack_init - INFO - Configuring clustering ... 15:00:54,065 - microstack_init - INFO - Setting up as a control node. ... 15:08:47,570 - microstack_init - INFO - Hardware virtualization is supported - KVM will be used for Nova instances ... 15:15:25,395 - microstack_init - INFO - Complete. Marked microstack as initialized!
- WARNING! Before setup below,
You should have properly working HOSTNAME and FQDN, so these commands have to work properly:
-
Launch instance (using unprivileged account):
$ microstack launch cirros -n testYou will receive optimistic instructions how to connect to above instance
test, but it did not work in my case.
When you finished setup and trying to connect to VM you will get error like:
sign_and_send_pubkey: no mutual signature supported
RSA key was ditched from Ubuntu 22.04.2 LTS. You can verify it with:
$ file ~/snap/microstack/common/.ssh/id_microstack
~/snap/microstack/common/.ssh/id_microstack: PEM RSA private keyWorkaround: accept RSA keys again:
ssh -o 'PubkeyAcceptedKeyTypes=+ssh-rsa' \
-i ~/snap/microstack/common/.ssh/id_microstack \
cirros@VM_IP
On bare-metal machine (2 cores, 8GB RAM, 100GB disk) I got this error on setup:
2023-03-30 06:01:57,517 - microstack_init - INFO - Configuring Neutron
Waiting for 192.168.0.51:9696
Traceback (most recent call last):
File "/snap/microstack/245/bin/microstack", line 11, in <module>
load_entry_point('microstack==0.0.1', 'console_scripts', 'microstack')()
File "/snap/microstack/245/lib/python3.8/site-packages/microstack/main.py", line 44, in main
cmd()
File "/snap/microstack/245/lib/python3.8/site-packages/init/main.py", line 60, in wrapper
return func(*args, **kwargs)
File "/snap/microstack/245/lib/python3.8/site-packages/init/main.py", line 228, in init
question.ask()
File "/snap/microstack/245/lib/python3.8/site-packages/init/questions/question.py", line 210, in ask
self.yes(awr)
File "/snap/microstack/245/lib/python3.8/site-packages/init/questions/__init__.py", line 887, in yes
check('openstack', 'network', 'create', 'test')
File "/snap/microstack/245/lib/python3.8/site-packages/init/shell.py", line 69, in check
raise subprocess.CalledProcessError(proc.returncode, " ".join(args))
subprocess.CalledProcessError: Command 'openstack network create test' returned non-zero exit status 1.To fix this error we have to increase random sleep(5) to sleep(50) in Python code:
sudo snap remove --purge microstack
mkdir ~/snap-repo
cd ~/snap-repo
snap download microstack --beta
mkdir ~/snap-tree
cd ~/snap-tree
sudo unsquashfs ../snap-repo/microstack_245.snap
# edit files
sudo vim ./squashfs-root/lib/python3.8/site-packages/init/questions/__init__.pyReplace both sleeps (otherwise Compute node configuration will fail):
diff -u ~/questions-init.py ./squashfs-root/lib/python3.8/site-packages/init/questions/__init__.py
--- /home/ansible/questions-init.py 2023-03-30 06:51:12.410802382 +0000
+++ ./squashfs-root/lib/python3.8/site-packages/init/questions/__init__.py 2023-03-30 08:26:42.342265206 +0000
@@ -739,7 +739,7 @@
nc_wait(_env['compute_ip'], '8774')
- sleep(5) # TODO: log_wait
+ sleep(55) # TODO: log_wait
if not call('openstack', 'service', 'show', 'compute'):
check('openstack', 'service', 'create', '--name', 'nova',
@@ -881,7 +881,7 @@
nc_wait(_env['control_ip'], '9696')
- sleep(5) # TODO: log_wait
+ sleep(50) # TODO: log_wait
if not call('openstack', 'network', 'show', 'test'):
check('openstack', 'network', 'create', 'test')And try again
# try this bind-mounted version of snap
# must use --devmode otherwise libvirt will fail and thus compute
sudo snap try --devmode ~/snap-tree/squashfs-root/
# note that version is not "245" anymore but "x1"
snap list microstack
Name Version Rev Tracking Publisher Notes
microstack ussuri x1 - - try
# again setup Microstack
sudo microstack init --auto --controlNow Neutron configuration should proceed without error(!)
Unfortunately Nova is unable to find compute node - there is some error with Libvirt:
$ systemctl --failed
UNIT LOAD ACTIVE SUB DESCRIPTION
● snap.microstack.libvirtd.service loaded failed failed Service for snap application microstack.libvirtd
$ sudo journalctl -u snap.microstack.libvirtd.service
libvirtd[103536]: Unable to initialize audit layer: Operation not permitted
libvirtd[103536]: cannot connect to netlink socket with protocol 15: Operation not permitted
$ sudo dmesg | fgrep libvirt
16466.315157] audit: type=1326 audit(1680169649.940:4419): auid=4294967295 uid=0 gid=0 ses=4294967295 subj=snap.microstack.libvirtd pid=103417 comm="libvirtd" exe="/snap/microstack/x1/usr/sbin/libvirtd" sig=0 arch=c000003e syscall=41 compat=0 ip=0x7fa1f3cd77ab code=0x50000By default Azure has inconsistent FQDN:
- request FQDN contains additional
ID.zoneprefix (XXXXXXXX.fracin my case) - but reverse DNS record is without it:
$ ping `hostname`
# vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
PING hp-mstack20.XXXXXXXX.frax.internal.cloudapp.net (10.101.0.5) 56(84) bytes of data.
# reverse PTR vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv - MISMATCH!!!
64 bytes from hp-mstack20.internal.cloudapp.net (10.101.0.5): icmp_seq=1 ttl=64 time=0.024 msTo make consistent FQDN I have to put reveres lookup FQDN hp-mstack20.internal.cloudapp.net into:
-
/etc/systemd/resolved.confDomains=internal.cloudapp.net -
/etc/netplan/50-cloud-init.yaml- addeduse-domains: false:network: ethernets: eth0: dhcp4: true dhcp4-overrides: route-metric: 100 use-domains: false dhcp6: false match: driver: hv_netvsc macaddress: xx:xx:xx:xx:xx:xx set-name: eth0 version: 2
- test it:
sudo netplan generate sudo netplan try
- to disable override on reboot (from cloud-init) we have to follow instructions:
echo 'network: {config: disabled}' | sudo tee -a /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
- and try reboot:
sudo init 6
- after reboot verify that
ping hostnamereturns same FQDN for both request and response:$ ping `hostname` PING hp-mstack20.internal.cloudapp.net (10.101.0.5) 56(84) bytes of data. 64 bytes from hp-mstack20.internal.cloudapp.net (10.101.0.5): icmp_seq=1 ttl=64 time=0.017 ms
Now you should have completely remove Microstack:
sudo snap remove --purge microstackAnd repeat all installation steps. This shoudl fix the problem and you should be able to launch VM without error.
Here sample output of working Microstack in Azure VM:
- Host list:
$ microstack.openstack host list +-----------------------------------+-----------+----------+ | Host Name | Service | Zone | +-----------------------------------+-----------+----------+ | hp-mstack20.internal.cloudapp.net | conductor | internal | | hp-mstack20.internal.cloudapp.net | scheduler | internal | | hp-mstack20.internal.cloudapp.net | compute | nova | +-----------------------------------+-----------+----------+
- WARNING! if Service=compute is missing it indicates likely libvirt daemon error (!)
- Network Agent list - notice that they are OVN agents:
microstack.openstack network agent list -f yaml
- output:
- Agent Type: OVN Controller Gateway agent Alive: true Availability Zone: '' Binary: ovn-controller Host: hp-mstack20.internal.cloudapp.net ID: hp-mstack20.internal.cloudapp.net State: true - Agent Type: OVN Metadata agent Alive: true Availability Zone: '' Binary: networking-ovn-metadata-agent Host: hp-mstack20.internal.cloudapp.net ID: 1b1790d8-d752-44cc-83e8-b6b1c4c15728 State: true
- output:
- Network list, output of:
microstack.openstack network list -f yaml- ID: 071fcb30-2e58-4df9-8402-1c12018e3f97 Name: test Subnets: - 0089ce30-1ad6-487d-9199-c33703453d81 - ID: 2aaee59f-8d0a-46d9-a165-3c76e8a40e8c Name: external Subnets: - 829f4ec4-3ed9-457c-a8ea-429730b2cd14
- the
testnetwork si ofgeneve(Overlay for VMs) type:$ microstack.openstack network show 071fcb30-2e58-4df9-8402-1c12018e3f97 -f value -c 'provider:network_type' geneve - the
externalnetwork is of `` type:$ microstack.openstack network show 2aaee59f-8d0a-46d9-a165-3c76e8a40e8c -f value -c 'provider:network_type' flat - listing Ports of Virtual Switch - port are used to Connect VMs or other components to Virtual Switch.
using command
microstack.openstack port list -f yaml, output:- Fixed IP Addresses: - ip_address: 192.168.222.198 subnet_id: 0089ce30-1ad6-487d-9199-c33703453d81 ID: 4f827709-dcf8-4190-a031-e0cdef9f2f4a MAC Address: fa:16:3e:50:a9:6c Name: '' Status: ACTIVE - Fixed IP Addresses: - ip_address: 10.20.20.10 subnet_id: 829f4ec4-3ed9-457c-a8ea-429730b2cd14 ID: 72ae1825-07a7-4032-a1d2-309f231592ee MAC Address: fa:16:3e:76:bb:7e Name: '' Status: N/A - Fixed IP Addresses: - ip_address: 10.20.20.155 subnet_id: 829f4ec4-3ed9-457c-a8ea-429730b2cd14 ID: 7960787e-048b-4e87-8fd1-0826d539eade MAC Address: fa:16:3e:89:30:e5 Name: '' Status: ACTIVE - Fixed IP Addresses: [] ID: a4bc904c-27dd-4bcb-a7a9-84833bfed1c2 MAC Address: fa:16:3e:ed:55:b7 Name: '' Status: DOWN - Fixed IP Addresses: - ip_address: 192.168.222.2 subnet_id: 0089ce30-1ad6-487d-9199-c33703453d81 ID: cfaa3b79-b844-400d-827b-8f2339dd8d1b MAC Address: fa:16:3e:92:9f:e5 Name: '' Status: DOWN - Fixed IP Addresses: - ip_address: 192.168.222.1 subnet_id: 0089ce30-1ad6-487d-9199-c33703453d81 ID: fe1f5bcb-893e-469e-910a-1845800de914 MAC Address: fa:16:3e:e3:cf:b4 Name: '' Status: ACTIVE
- list Virtual Machines:
$ microstack.openstack server list +--------------------------------------+------+--------+-----------------------------------+--------+---------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+------+--------+-----------------------------------+--------+---------+ | c0ca3f6b-9da9-45f6-9445-80fe5b15a678 | test | ACTIVE | test=192.168.222.198, 10.20.20.10 | cirros | m1.tiny | +--------------------------------------+------+--------+-----------------------------------+--------+---------+
Now tricky stuff - listing OVN configurations:
- few commands from: https://docs.ovn.org/en/stable/tutorials/ovn-openstack.html
- show
OVN southbound db- only Geneve tunnel using LAN card:$ sudo microstack.ovn-sbctl show Chassis hp-mstack20.internal.cloudapp.net hostname: hp-mstack20.internal.cloudapp.net Encap geneve ip: "10.101.0.5" options: {csum="true"} Port_Binding "4f827709-dcf8-4190-a031-e0cdef9f2f4a" Port_Binding cr-lrp-7960787e-048b-4e87-8fd1-0826d539eade - show
OVN northbound database$ sudo microstack.ovn-nbctl show switch 8d94192b-b57a-4382-a152-28684cfda579 (neutron-071fcb30-2e58-4df9-8402-1c12018e3f97) (aka test) port 4f827709-dcf8-4190-a031-e0cdef9f2f4a addresses: ["fa:16:3e:50:a9:6c 192.168.222.198"] port cfaa3b79-b844-400d-827b-8f2339dd8d1b type: localport addresses: ["fa:16:3e:92:9f:e5 192.168.222.2"] port fe1f5bcb-893e-469e-910a-1845800de914 type: router router-port: lrp-fe1f5bcb-893e-469e-910a-1845800de914 switch b2b8e843-3649-4a68-b4f3-edeb69c2799a (neutron-2aaee59f-8d0a-46d9-a165-3c76e8a40e8c) (aka external) port provnet-e2363fcd-3cea-46d8-89a2-3fc36f2635a8 type: localnet addresses: ["unknown"] port a4bc904c-27dd-4bcb-a7a9-84833bfed1c2 type: localport addresses: ["fa:16:3e:ed:55:b7"] port 7960787e-048b-4e87-8fd1-0826d539eade type: router router-port: lrp-7960787e-048b-4e87-8fd1-0826d539eade router 1a25abf8-df53-450f-ac51-74974142e8f1 (neutron-ba9912e1-fcbd-4c5b-bb53-fbf77d432beb) (aka test-router) port lrp-fe1f5bcb-893e-469e-910a-1845800de914 mac: "fa:16:3e:e3:cf:b4" networks: ["192.168.222.1/24"] port lrp-7960787e-048b-4e87-8fd1-0826d539eade mac: "fa:16:3e:89:30:e5" networks: ["10.20.20.155/24"] gateway chassis: [hp-mstack20.internal.cloudapp.net] nat 3bf7a3ad-ea36-40a9-9207-973e2ef7307a external ip: "10.20.20.10" logical ip: "192.168.222.198" type: "dnat_and_snat" nat be072635-646b-4511-83bd-f6a782c1c250 external ip: "10.20.20.155" logical ip: "192.168.222.0/24" type: "snat" - now to show how Northbound (high-level DB) is connected to Southbound (low-level switch tables)
we have to first list bindings:
$ sudo microstack.ovn-sbctl list datapath_binding _uuid : 75239af1-5c73-446f-9056-cc60582c656f external_ids : {logical-router="1a25abf8-df53-450f-ac51-74974142e8f1", name=neutron-ba9912e1-fcbd-4c5b-bb53-fbf77d432beb, name2=test-router} tunnel_key : 3 _uuid : 329ee08e-82d9-450e-a4ac-f67dacb78e2a external_ids : {logical-switch="8d94192b-b57a-4382-a152-28684cfda579", name=neutron-071fcb30-2e58-4df9-8402-1c12018e3f97, name2=test} tunnel_key : 1 _uuid : 7c9ae157-f20f-4ca2-bfee-5689c7019e86 external_ids : {logical-switch="b2b8e843-3649-4a68-b4f3-edeb69c2799a", name=neutron-2aaee59f-8d0a-46d9-a165-3c76e8a40e8c, name2=external} tunnel_key : 2
- and now Southbound tables:
$ sudo microstack.ovn-sbctl lflow-list test ... ingress rules for test network ... Datapath: "neutron-071fcb30-2e58-4df9-8402-1c12018e3f97" aka "test" (329ee08e-82d9-450e-a4ac-f67dacb78e2a) Pipeline: ingress table=0 (ls_in_port_sec_l2 ), priority=100 , match=(eth.src[40]), action=(drop;) table=0 (ls_in_port_sec_l2 ), priority=100 , match=(vlan.present), action=(drop;) table=0 (ls_in_port_sec_l2 ), priority=50 , match=(inport == "4f827709-dcf8-4190-a031-e0cdef9f2f4a" && eth.src == {fa:16:3e:50:a9:6c}), action=(next;) ... many more rules... ... egress rules for test network Datapath: "neutron-071fcb30-2e58-4df9-8402-1c12018e3f97" aka "test" (329ee08e-82d9-450e-a4ac-f67dacb78e2a) Pipeline: egress table=0 (ls_out_pre_lb ), priority=110 , match=(eth.src == e2:89:25:72:8f:85), action=(next;) table=0 (ls_out_pre_lb ), priority=110 , match=(nd || nd_rs || nd_ra || icmp4.type == 3 ||icmp6.type == 1 || (tcp && tcp.flags == 20)), action=(next;) table=0 (ls_out_pre_lb ), priority=0 , match=(1), action=(next;) table=1 (ls_out_pre_acl ), priority=110 , match=(ip && outport == "fe1f5bcb-893e-469e-910a-1845800de914"), action=(next;) table=1 (ls_out_pre_acl ), priority=110 , match=(nd || nd_rs || nd_ra || icmp4.type == 3 || icmp6.type == 1 || (tcp && tcp.flags == 20)), action=(next;) ... many more rules ...
And here lot of details from https://docs.openstack.org/networking-ovn/latest/admin/refarch/refarch.html#accessing-ovn-database-content for Northbound:
Here we can see why Neutron fails to find Chassis when FQDN is incosistent:
$ sudo microstack.ovn-nbctl list Gateway_Chassis
_uuid : 815825d7-123b-4b15-bedb-f0285bc60e00
chassis_name : hp-mstack20.internal.cloudapp.net
external_ids : {}
name : lrp-7960787e-048b-4e87-8fd1-0826d539eade_hp-mstack20.internal.cloudapp.net
options : {}
priority : 1- above chassis name is FQDN.
Similarly overlay network for VMs (called geneve):
$ sudo microstack.ovn-sbctl list Encap
_uuid : 1f922921-9268-4e6d-ae87-55afacab9185
chassis_name : hp-mstack20.internal.cloudapp.net
ip : "10.101.0.5"
options : {csum="true"}
type : geneveUff, how to graps all of this....
Microstack ver. 245 and current DevStack sets-pu OVN as default - software defined L3 layer switch/router and OVS - software defined L2 level switch. Please see OVN
Copyright © Henryk Paluch. All rights reserved.
