clab hung at "Running postdeploy actions" Centos9 Stream

[chpradeep23]

I am trying to get clab running on Centos9-Stream and when I deploy a lab its hung at Running postdeploy actions. and also I am seeing a warning "WARN[0000] Unable to load kernel module "ip_tables" automatically "load ip_tables failed: key was rejected by service"

is there any solution for this warning?.

[hellt]

Can you share your topology file and docker version output?

[chpradeep23]

test-lab.txt

Docker version 24.0.2, build cb74dfc

[chpradeep23]

also, same topology and docker version works on Centos8 without issues.

[hellt]

you can do docker logs -f <node-name> to watch what ceos emits in the logs. It doesn't boot properly for some reason

[chpradeep23]

this is what I am seeing in logs
[root@pchallagulla-centos9 DEV-LABS]# docker logs -f eos-01
Waiting for all 2 interfaces to be connected
Connected 1 interfaces out of 2
Sleeping 0 seconds before boot

Welcome to CentOS Linux 7 (AltArch)!

[ OK ] Listening on Syslog Socket.
[ OK ] Listening on Journal Socket.
Mounting Huge Pages File System...
[ OK ] Reached target Swap.
[ OK ] Listening on initctl Compatibility Named Pipe.
Mounting Kernel Debug File System...
Starting Process EosKernel cmdline...
[ OK ] Set up automount Arbitrary Executab…rmats File System Automount Point.
[ OK ] Listening on Journal Socket (/dev/log).
Starting Journal Service...
[ OK ] Reached target Remote File Systems.
Starting Load legacy module configuration...
[ OK ] Created slice system-selinux\x2dpol…migrate\x2dlocal\x2dchanges.slice.
[ OK ] Listening on Device-mapper event daemon FIFOs.
[ OK ] Created slice User and Session Slice.
[ OK ] Listening on LVM2 metadata daemon socket.
Starting Monitoring of LVM2 mirrors…ng dmeventd or progress polling...
[ OK ] Reached target Slices.
[ OK ] Listening on LVM2 poll daemon socket.
Mounting Kernel Configuration File System...
Mounting FUSE Control File System...
Starting Apply net.conf.ipv*.all workaround...
[ OK ] Reached target Paths.
[ OK ] Listening on Process Core Dump Socket.
Starting Read and set NIS domainname from /etc/sysconfig/network...
[ OK ] Reached target Local Encrypted Volumes.
Starting Remount Root and Kernel File Systems...
[ OK ] Started Journal Service.
[ OK ] Mounted Huge Pages File System.
[ OK ] Mounted Kernel Debug File System.
[ OK ] Started Process EosKernel cmdline.
[ OK ] Started Load legacy module configuration.
[ OK ] Mounted Kernel Configuration File System.
[ OK ] Mounted FUSE Control File System.
[ OK ] Started Read and set NIS domainname from /etc/sysconfig/network.
[ OK ] Started Remount Root and Kernel File Systems.
[ OK ] Started LVM2 metadata daemon.
Starting Configure read-only root support...
Starting Rebuild Hardware Database...
Starting Create System Users...
Starting Arista stuff from rc.sysinit...
[ OK ] Started Create System Users.
Starting Create Static Device Nodes in /dev...
[ OK ] Started Configure read-only root support.
[ OK ] Started Create Static Device Nodes in /dev.
[ OK ] Started Monitoring of LVM2 mirrors,…sing dmeventd or progress polling.
[ OK ] Reached target Local File Systems (Pre).
Mounting /run/netns...
Mounting /var/lib/docker for cEOS-lab...
Mounting /.deltas...
Mounting /var/log for cEOS-lab...
Mounting /monitor...
Mounting /var/shmem for cEOS-lab...
Mounting /var/core for cEOS-lab...
Mounting /tmp for cEOS-lab...
Mounting /var/tmp for cEOS-lab...
[ OK ] Mounted /run/netns.
[ OK ] Mounted /var/lib/docker for cEOS-lab.
[ OK ] Mounted /.deltas.
[ OK ] Mounted /var/log for cEOS-lab.
[ OK ] Mounted /monitor.
[ OK ] Mounted /var/shmem for cEOS-lab.
[ OK ] Mounted /var/core for cEOS-lab.
[ OK ] Mounted /tmp for cEOS-lab.
Mounting /var/shmem/stashes for cEOS-lab...
Starting Flush Journal to Persistent Storage...
[ OK ] Mounted /var/tmp for cEOS-lab.
[ OK ] Mounted /var/shmem/stashes for cEOS-lab.
[ OK ] Reached target Local File Systems.
Starting Rebuild Journal Catalog...
Starting Mark the need to relabel after reboot...
Starting CoreUtil systemd service t…duce annotation for core files....
[ OK ] Started Mark the need to relabel after reboot.
[ OK ] Started Rebuild Journal Catalog.
[ OK ] Started Flush Journal to Persistent Storage.
Starting Create Volatile Files and Directories...
[ OK ] Started Create Volatile Files and Directories.
Starting Update UTMP about System Boot/Shutdown...
[ OK ] Started Update UTMP about System Boot/Shutdown.
[ OK ] Started Apply net.conf.ipv*.all workaround.
Starting Apply Kernel Variables...
[ OK ] Started Arista stuff from rc.sysinit.
[ OK ] Started Apply Kernel Variables.
[ OK ] Started Rebuild Hardware Database.
Starting Update is Completed...
[ OK ] Started Update is Completed.
[ OK ] Reached target System Initialization.
[ OK ] Started Discard unused blocks once a week.
[ OK ] Reached target Timers.
[ OK ] Listening on D-Bus System Message Bus Socket.
[ OK ] Reached target Sockets.
[ OK ] Reached target Basic System.
Starting Dynamic EosKernel configuration...
[ OK ] Started Monitors for unusually high numbers of running processes..
Starting System Logging Service...
Starting Process kernelcrash on /mnt/flash...
Starting Xinetd A Powerful Replacement For Inetd...
Starting Displays the welcome banner...
Starting Install PSP...
[20630.719449] EosBanner[265]: Welcome to Arista Networks EOS 4.29.3M
Starting Mount and fsck the SSD device for use by Arista....
Starting Login Service...
[ OK ] Started D-Bus System Message Bus.
Starting Configure loopback interface...
Starting Permit User Sessions...
[ OK ] Started RAS daemon to log the RAS events.
Starting Setup /persist from /mnt/flash/persist...
Starting EOS Diagnostic Mode...
[ OK ] Started Watchdog service..
[ OK ] Started System Logging Service.
[ OK ] Started Process kernelcrash on /mnt/flash.
[ OK ] Started Displays the welcome banner.
[20630.740145] EosBanner[265]: Architecture: i386
[ OK ] Started Install PSP.
Starting Perform cEOS specific initialization if we're running cEOS...
Starting SsuRestore systemd service....
[ OK ] Started Permit User Sessions.
[ OK ] Started Xinetd A Powerful Replacement For Inetd.
[ OK ] Started CoreUtil systemd service to…roduce annotation for core files..
[ OK ] Started Command Scheduler.
[ OK ] Started Login Service.
[ OK ] Started Configure loopback interface.
Starting Publish /etc/port-numbering if needed...
Starting Insert dmamem module if needed...
[ OK ] Started Publish /etc/port-numbering if needed.
[ OK ] Stopped RAS daemon to log the RAS events.
[ OK ] Started RAS daemon to log the RAS events.
[20630.891123] EosKernelConfig[259]: Failed to set /proc/sys/vm/rss_warn_thresh to 3500: [Errno 2] No such file or directory: '/proc/sys/vm/rss_warn_thresh'
[ OK ] Started Dynamic EosKernel configuration.
[ OK ] Started Perform cEOS specific initialization if we're running cEOS.
[ OK ] Started Insert dmamem module if needed.
[ OK ] Started Setup /persist from /mnt/flash/persist.
[ OK ] Started SsuRestore systemd service..
[ OK ] Started AsuFastPktTransmit systemd service..
[ OK ] Started Mount and fsck the SSD device for use by Arista..
Starting service to copy fossil files from temp storage to flash...
Starting BootExtension systemd service....
Starting Setup log archiving for EOS....
[ OK ] Started service to copy fossil files from temp storage to flash.
[ OK ] Started Setup log archiving for EOS..
[ OK ] Started BootExtension systemd service..
Starting EosReadPrefdl systemd serv…fdl and saves it as /etc/prefdl...
[ OK ] Started EosReadPrefdl systemd servi…refdl and saves it as /etc/prefdl.
Starting Run rc.eos...
Starting Removes any pkgs not neede…uld be run before netnsd launch...
[ OK ] Started Run rc.eos.
Starting DPM Update...
[ OK ] Started Removes any pkgs not needed…hould be run before netnsd launch.
Starting SYSV: ProcMgr is an Arastr…that keeps our daemons running....
Starting RPM Correction service...
[20632.258928] ProcMgr[568]: Starting ProcMgr: Removing all files in all subdirs of /etc/ProcMgr.d/run
[ OK ] Started DPM Update.
[ OK ] Started RPM Correction service.
[20632.456855] ProcMgr[568]: [ OK ]
[ OK ] Started SYSV: ProcMgr is an Arastra…l that keeps our daemons running..
Starting watchdog service. This ser… stops watchdog before ProcMgr....
Starting SYSV: Eos system stage 1 i…t (runs after POST, as early as...
[ OK ] Started watchdog service. This service stops watchdog before ProcMgr..
[20632.608502] EosStage1[589]: Starting EOS initialization stage 1: [ OK ]
[ OK ] Started SYSV: Eos system stage 1 in…ipt (runs after POST, as early as.
Starting NorCal systemd service...
Starting SYSV: EosCloudInit is used…unning in Cloud (AWS, Azure,..)...
[ OK ] Started SYSV: EosCloudInit is used … running in Cloud (AWS, Azure,..).
[20633.967654] NorCal[593]: Starting NorCal initialization: [ OK ]
[ OK ] Started NorCal systemd service.
Starting Logs a warning message whe…n unsupported OS Arch is booted...
Starting SYSV: Exclusive ssh service for internal mgmt i/f...
[ OK ] Started SYSV: Exclusive ssh service for internal mgmt i/f.
Starting SYSV: Adjusts the oom_scor…n-procmanager processes so that...
[ OK ] Started SYSV: Adjusts the oom_score…non-procmanager processes so that.
Starting SYSV: Eos system init scri…uns after POST, before ProcMgr)...
[ OK ] Started Logs a warning message when an unsupported OS Arch is booted.
[ OK ] Started SYSV: Eos system init scrip…(runs after POST, before ProcMgr).
Starting Power-On Self Test...
Starting EOS Warmup Service...
[20634.426115] EosStage2[626]: Starting EOS initialization stage 2: [ OK ]
[ OK ] Started Power-On Self Test.
[ OK ] Reached target EOS regular mode.
[ OK ] Started EOS Diagnostic Mode.
Stopping System Logging Service...
[ OK ] Stopped System Logging Service.
Starting System Logging Service...
[ OK ] Started System Logging Service.

[chpradeep23]

let me also check with an another cEOS version