Tracing/debugging in talosctl and friends

[boldandbusted]

Howdy. I'm troubleshooting a problem on my local Gentoo workstation. I've almost got talosctl working with QEMU, but I'm getting an obscured bug.

❯ sudo --preserve-env=HOME talosctl cluster create --provisioner qemu --with-debug
validating CIDR and reserving IPs
generating PKI and tokens
creating state directory in "/home/jesse/.talos/clusters/talos-default"
creating network talos-default
creating load balancer
creating dhcpd
creating controlplane nodes
creating worker nodes
WARNING: failed to preallocate disk space for "/home/jesse/.talos/clusters/talos-default/talos-default-worker-1-0.disk" (size 6442450944): interrupted system cal
lrenamed talosconfig context "talos-default" -> "talos-default-2"
waiting for API
^Cbootstrap error: 3 error(s) occurred:
        rpc error: code = DeadlineExceeded desc = context deadline exceeded
        rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.5.0.2:50000: connect: no route to host"
        context canceled
Signal received, aborting, press Ctrl+C once again to abort immediately...

A good five minutes of no perceivable activity presages the Ctrl-C.

It is the WARNING: failed to preallocate disk space for "/home/jesse/.talos/clusters/talos-default/talos-default-worker-1-0.disk" (size 6442450944): interrupted system call that I would like to drill down on.

I'm about to break out strace to start looking, but is there some way to enable debug logging that I can see what is triggering the 'interrupted system call'? Thanks in advance.

P.S. I suspect that there is a config in the kernel that is required to be set or unset for Talos and QEMU, but I'm not at that stage to guess that yet.

P.P.S. Ah, I added the --with-debug to try to get more verbose output - it had no effect, and the result and output is the same.

[smira]

Please see the documentation: https://www.talos.dev/v1.6/talos-guides/install/local-platforms/qemu/#logs

The interrupted system call can be ignored.

[boldandbusted]

Aaand... FWIW, I do not have any Docker component on this system - so odd that there's a DOCKER-USER iptables chain.

[smira]

DOCKER-USER is something we create as a workaround for Docker issues, it's not referenced without docker.

The error comes from tc-redirect-tap, not sure - something is missing in your kernel/loaded modules, probably qdisc?

[boldandbusted]

Thanks for your continuing help here, @smira. I had some missing Linux Kernel configuration options, and now I've got virtual networking working. However, now, qemu-system-x86_64 is running, but apparently never starts its networking services. Is there a way to pass additional options to qemu-system-x86_64 to assist in debugging, like -curses or -display curses so I can see the guest kernel console output?

[smira]

The console output goes to the log, ~/.talos/cluster/<name>/<machine>.log

[boldandbusted]

Thanks. Yeah, then there's no output at all, sadly. I was unable to get QEMU to boot Talos via talosctl. But, I was able to get minikube to start a working cluster and QEMU to launch Linux Mint (to just validate that QEMU can work), so I'm not sure what is happening with talosctl.

Thanks anyway for your help, @smira.