Fedora VM Drivers on minikube 1.33 cannot pull images (resolved.conf) #18705
Comments
llegolas
changed the title
|
I confirm the same on my MacBook M2 2023. Same behavior. The minikube addons fails. ingress and ingress-dns clearly also stating not supported(compiled image missing). I have submitted a fix for ingress-dns but they decided ignore. |
|
not sure how |
|
@llegolas can you paste the full logs , does that only affect when you enable a specific addon or only on quay.io images? and also can you try with a different container runtime to see if that solves it ? minikube delete --all |
|
btw I just tired this myself and it seems good for me for Qemu Driver on M1 (arm64) |
|
btw I do not see DENSSEC=yes in my resolved.conf cat /version.json llegolas @cirix |
|
|
|
here is the whole session: logs are attached |
|
the full systemd-resolved.service logs: |
|
@llegolas and @cirix **Question1 :**When you downgrade to 1.32 does minikube start still give you the same warning ? Question2: (this runs the same check that minikube does to check access to registry.k8s.io) @llegolas and @cirix are you both on a corp network with a custom DNS? or do you need to set a Proxy to access registries ? (if yes then you would need to set the HTTP_PROXY envs) in the same terminal in our website we have a known issue for Builtin Network (not socket_vm network) does that help you for socket vmnet too ? btw for reference, |
medyagh
added
the
triage/needs-information
|
can you the try ISO before this PR ? https://github.com/kubernetes/minikube/pull/18277/files Other PRS to try: |
as expected by
Settiing the DNS on eth1 to 8.8.8.8 instead of 192.168.122.1 solves the problem. Disabling the DNS on the 'default' kvm network used for eth1 solves the problem too. It looks like dnsmasq used by libvirt for dns and dhcp is not having dnssec enabled by default. I'm using driver kvm2 not qemu so network=mvnet_socket is not applicable With all that in mind modifying the 'default' libvirt network (disable dns altogether) just to accommodate minikube seem unacceptable provided the previous version 1.32 which had DNSSEC=no and not DNSSEC=allow-downgrade was working OK. I see no point of trying with the ISOs from the prs above as just one of them is x86 related but just introduces dm-multipath. By the way cilium CNI fails to start after all but it is another issue "FailedPostStartHook" |
|
I think is not relevant to the architecture. I enable on my macbook again the network for my qemu2 and I have the same problem. I did the same I reconfigured the DNS manually as @llegolas and it worked and boils down to what is reported. The behavior of the image has changed. br |
|
with some git archeology I traced the behavior change to aeed263 . The previous version of buildroot had |
|
Thank you for providing more info, i agree this should not happen to you and we need to find a solution that can help most users without lowering security standards. I would like to know what is special about your setup or workstation and why does it time out, and why it doesnt happen to me? Does it affect you only it you deploy an image from quay.io ? curl -sS -m 2 https://registry.k8s.iocurl: (28) Resolving timed out after 2000 milliseconds Since that commands run on your computer and not on minikube. @cirix the command I provided above —iso-url lets you try minikube with different ISO (would need to delete previous one first) And you can get the ISO build number from the PR, that way you dont have to wait to |
|
@medyagh the curl command is run inside the minikube VM! "It looks like dnsmasq used by libvirt for dns and dhcp is not having dnssec enabled by default." "With all that in mind modifying the 'default' libvirt network (disable dns altogether) just to accommodate minikube seem unacceptable provided the previous version 1.32 which had DNSSEC=no and not DNSSEC=allow-downgrade was working OK." And to repeat - why testing different PR builds when I've narrowed down the problem to aeed263 ?? |
|
Hi @llegolas, thank you for your investigation and discovering the cause. I think the best path going forward is I can add a patch to the ISO to revert buildroot/buildroot@b16ae93 and we can release a patch release to resolve this issue. Then we can work on a long term solution to the issue for the next minor release. |
|
Happy to test the PR #18737 ISO once available |
|
Would you mind testing now @llegolas?
|
|
Unfortunately it does not solve the problem. I still have DNNSEC=allow-downgrade with all the consequences of it. DNSSEC=allow-downgrade is enabled if libgcrypt is installed which it is. thus preserving all other default whatever they might be now or in the future. |
|
Thank you llegolas for staying with us by trying and providing the logs, you were right the minikube check for regsitry connection happens inside minikube (that we should do outside as well if iit fails #18754) I still have not been able to replicate your issue myself, I know you said you dont have any Proxy or Corp network, but is there any thing else that you could think of that is different with your environment ? it would greatly help us if we find out whats is different about your environment than mine or the one we created in the cloud, that way we could guard against it in the code and help more users... |
|
@medyagh lets start by comparing how we run it at first place. In a broader sense to replicate it you should run in on linux host with KVM driver. There are too many subtle differences in the way the different hypervisors or container runtimes set the network for the VM/containers to rely on the assumption that this can be replicated across all of them. |
|
@llegolas sure thing ! btw you did delete the old minikube before trying out the ISO in the PR ? @spowelljr tried it with KVM on linux (ubuntu) unfortunately we dont have fedoras, I really wish we had Fedora machines for integration tests, is there anything special about your DNS config ? is everything fully vanila fedora 40? |
|
I don't know if the problem is with the host per se, but more related to the hypervisor and the configuration of dns on the linux image of minikube. In my case on a Macosx M2 with qemu2 the logs on the minikube services is exactly the same as the ones with Fedora + KVM. Now, if I manipulate the bridge interface that gets created or the resolv conf to set DNSSEC off the delete and recreate of the pods make the problem go away. My intuition is that is related to the os+hyperv setup.
Now on the other hand, I have deployed version 33 marked the binary minikube 33 and you will see a total different log setup:
Comparing side by side the above , I think becomes evident that on the second scenario DNSSEC is killing the traffic. Hope the above help. p.s. If I manipulate the dnssec myself and reconfigure dns on minikube33 the problem goes away, but I don't consider it a viable solution. Also for osx based oses is clearly documented on Apple that DNSSEC is not supported and it can only happen by changing the dns server(i.e. use unbound). As you will see has not received any votes to be implemented: Overall: The linux iso used to start minikube has major differences between 32 and 33 image. Add to that the behavior of the underlying virtual interfaces varying between os versions and the hypervisor used is not that evident regards |
Minikube 1.33.0 adds useful features, fixes, and performance improvements, but we could not use it because of a regression in systemd-resolved[1]. A critical change in 1.33.0 is upgrading the kernel to 5.10.207. This version fixes bad bug with minikube 1.32.0, when qemu assertion fails while starting a kubevirt VM[2] on newer Intel CPUs (i7-12700k). Now that we setup systemd-resolved configuration we can upgrade to minikube 1.33.0, and Alex can run drenv kubevirt environment without manual hacks. [1] kubernetes/minikube#18705 [2] https://gitlab.com/qemu-project/qemu/-/issues/237 Thanks: Alex Kalenyuk <akalenyu@redhat.com> Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Minikube 1.33.0 adds useful features, fixes, and performance improvements, but we could not use it because of a regression in systemd-resolved[1]. A critical change in 1.33.0 is upgrading the kernel to 5.10.207. This version fixes bad bug with minikube 1.32.0, when qemu assertion fails while starting a kubevirt VM[2] on newer Intel CPUs (i7-12700k). Now that we setup systemd-resolved configuration we can upgrade to minikube 1.33.0, and Alex can run drenv kubevirt environment without manual hacks. [1] kubernetes/minikube#18705 [2] https://gitlab.com/qemu-project/qemu/-/issues/237 Thanks: Alex Kalenyuk <akalenyu@redhat.com> Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Minikube 1.33.0 adds useful features, fixes, and performance improvements, but we could not use it because of a regression in systemd-resolved[1]. A critical change in 1.33.0 is upgrading the kernel to 5.10.207. This version fixes bad bug with minikube 1.32.0, when qemu assertion fails while starting a kubevirt VM[2] on newer Intel CPUs (i7-12700k). Now that we setup systemd-resolved configuration we can upgrade to minikube 1.33.0, and Alex can run drenv kubevirt environment without manual hacks. [1] kubernetes/minikube#18705 [2] https://gitlab.com/qemu-project/qemu/-/issues/237 Thanks: Alex Kalenyuk <akalenyu@redhat.com> Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Minikube 1.33.0 adds useful features, fixes, and performance improvements, but we could not use it because of a regression in systemd-resolved[1]. A critical change in 1.33.0 is upgrading the kernel to 5.10.207. This version fixes bad bug with minikube 1.32.0, when qemu assertion fails while starting a kubevirt VM[2] on newer Intel CPUs (i7-12700k). Now that we setup systemd-resolved configuration we can upgrade to minikube 1.33.0, and Alex can run drenv kubevirt environment without manual hacks. [1] kubernetes/minikube#18705 [2] https://gitlab.com/qemu-project/qemu/-/issues/237 Thanks: Alex Kalenyuk <akalenyu@redhat.com> Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Minikube 1.33.0 adds useful features, fixes, and performance improvements, but we could not use it because of a regression in systemd-resolved[1]. A critical change in 1.33.0 is upgrading the kernel to 5.10.207. This version fixes bad bug with minikube 1.32.0, when qemu assertion fails while starting a kubevirt VM[2] on newer Intel CPUs (i7-12700k). Now that we setup systemd-resolved configuration we can upgrade to minikube 1.33.0, and Alex can run drenv kubevirt environment without manual hacks. [1] kubernetes/minikube#18705 [2] https://gitlab.com/qemu-project/qemu/-/issues/237 Thanks: Alex Kalenyuk <akalenyu@redhat.com> Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Minikube 1.33.0 adds useful features, fixes, and performance improvements, but we could not use it because of a regression in systemd-resolved[1]. A critical change in 1.33.0 is upgrading the kernel to 5.10.207. This version fixes bad bug with minikube 1.32.0, when qemu assertion fails while starting a kubevirt VM[2] on newer Intel CPUs (i7-12700k). Now that we setup systemd-resolved configuration we can upgrade to minikube 1.33.0, and Alex can run drenv kubevirt environment without manual hacks. This change: - Updates the docs that latest minikube test is 1.33.0 - Setup minikube by default when creating the virtual environment, so minikube 1.33.0 support is added transparently for developers - Setup drenv in the e2e job to support minikube 1.33.0 - Cleanup drenv in the e2e job so setup from previous build will not leak into the next build [1] kubernetes/minikube#18705 [2] https://gitlab.com/qemu-project/qemu/-/issues/237 Thanks: Alex Kalenyuk <akalenyu@redhat.com> Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Minikube 1.33.0 adds useful features, fixes, and performance improvements, but we could not use it because of a regression in systemd-resolved[1]. A critical change in 1.33.0 is upgrading the kernel to 5.10.207. This version fixes bad bug with minikube 1.32.0, when qemu assertion fails while starting a kubevirt VM[2] on newer Intel CPUs (i7-12700k). Now that we setup systemd-resolved configuration we can upgrade to minikube 1.33.0, and Alex can run drenv kubevirt environment without manual hacks. This change: - Updates the docs that latest minikube test is 1.33.0 - Setup minikube by default when creating the virtual environment, so minikube 1.33.0 support is added transparently for developers - Setup drenv in the e2e job to support minikube 1.33.0 - Cleanup drenv in the e2e job so setup from previous build will not leak into the next build [1] kubernetes/minikube#18705 [2] https://gitlab.com/qemu-project/qemu/-/issues/237 Thanks: Alex Kalenyuk <akalenyu@redhat.com> Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Minikube 1.33.0 adds useful features, fixes, and performance improvements, but we could not use it because of a regression in systemd-resolved[1]. A critical change in 1.33.0 is upgrading the kernel to 5.10.207. This version fixes bad bug with minikube 1.32.0, when qemu assertion fails while starting a kubevirt VM[2] on newer Intel CPUs (i7-12700k). Now that we setup systemd-resolved configuration we can upgrade to minikube 1.33.0, and Alex can run drenv kubevirt environment without manual hacks. This change: - Updates the docs that latest minikube test is 1.33.0 - Setup minikube by default when creating the virtual environment, so minikube 1.33.0 support is added transparently for developers - Setup drenv in the e2e job to support minikube 1.33.0 - Cleanup drenv in the e2e job so setup from previous build will not leak into the next build [1] kubernetes/minikube#18705 [2] https://gitlab.com/qemu-project/qemu/-/issues/237 Thanks: Alex Kalenyuk <akalenyu@redhat.com> Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Minikube 1.33.0 adds useful features, fixes, and performance improvements, but we could not use it because of a regression in systemd-resolved[1]. A critical change in 1.33.0 is upgrading the kernel to 5.10.207. This version fixes bad bug with minikube 1.32.0, when qemu assertion fails while starting a kubevirt VM[2] on newer Intel CPUs (i7-12700k). Now that we setup systemd-resolved configuration we can upgrade to minikube 1.33.0, and Alex can run drenv kubevirt environment without manual hacks. This change: - Updates the docs that latest minikube test is 1.33.0 - Setup minikube by default when creating the virtual environment, so minikube 1.33.0 support is added transparently for developers - Setup drenv in the e2e job to support minikube 1.33.0 - Cleanup drenv in the e2e job so setup from previous build will not leak into the next build [1] kubernetes/minikube#18705 [2] https://gitlab.com/qemu-project/qemu/-/issues/237 Thanks: Alex Kalenyuk <akalenyu@redhat.com> Signed-off-by: Nir Soffer <nsoffer@redhat.com>
Minikube 1.33.0 adds useful features, fixes, and performance improvements, but we could not use it because of a regression in systemd-resolved[1]. A critical change in 1.33.0 is upgrading the kernel to 5.10.207. This version fixes bad bug with minikube 1.32.0, when qemu assertion fails while starting a kubevirt VM[2] on newer Intel CPUs (i7-12700k). Now that we setup systemd-resolved configuration we can upgrade to minikube 1.33.0, and Alex can run drenv kubevirt environment without manual hacks. This change: - Updates the docs that latest minikube test is 1.33.0 - Setup minikube by default when creating the virtual environment, so minikube 1.33.0 support is added transparently for developers - Setup drenv in the e2e job to support minikube 1.33.0 - Cleanup drenv in the e2e job so setup from previous build will not leak into the next build [1] kubernetes/minikube#18705 [2] https://gitlab.com/qemu-project/qemu/-/issues/237 Thanks: Alex Kalenyuk <akalenyu@redhat.com> Signed-off-by: Nir Soffer <nsoffer@redhat.com>
@llegolas isnt that commented ? it is a linux convention that lines that start with # are comments. They are not interpreted as configuration options by systemd-resolved does still give you the pull image problem as well ? |
Minikube 1.33.0 adds useful features, fixes, and performance improvements, but we could not use it because of a regression in systemd-resolved[1]. A critical change in 1.33.0 is upgrading the kernel to 5.10.207. This version fixes bad bug with minikube 1.32.0, when qemu assertion fails while starting a kubevirt VM[2] on newer Intel CPUs (i7-12700k). Now that we setup systemd-resolved configuration we can upgrade to minikube 1.33.0, and Alex can run drenv kubevirt environment without manual hacks. This change: - Updates the docs that latest minikube test is 1.33.0 - Setup minikube by default when creating the virtual environment, so minikube 1.33.0 support is added transparently for developers - Setup drenv in the e2e job to support minikube 1.33.0 - Cleanup drenv in the e2e job so setup from previous build will not leak into the next build [1] kubernetes/minikube#18705 [2] https://gitlab.com/qemu-project/qemu/-/issues/237 Thanks: Alex Kalenyuk <akalenyu@redhat.com> Signed-off-by: Nir Soffer <nsoffer@redhat.com>
thank you for pointing this out, this work arround worth a try |
medyagh
changed the title
Made a PR to implment this: #18830 |
It is commented out indeed but is is also linux convention that the commented out lines document the defaults the application in this case systemd-resolved uses. I've explained that two weeks ago here #18705 (comment) @spowelljr will test the new ISO once the build is complete. Please drop me a link |
|
I tested the ISO, it looks good: |
|
#18830 ISO image fixes it |
Minikube 1.33.0 adds useful features, fixes, and performance improvements, but we could not use it because of a regression in systemd-resolved[1]. A critical change in 1.33.0 is upgrading the kernel to 5.10.207. This version fixes bad bug with minikube 1.32.0, when qemu assertion fails while starting a kubevirt VM[2] on newer Intel CPUs (i7-12700k). Now that we setup systemd-resolved configuration we can upgrade to minikube 1.33.0, and Alex can run drenv kubevirt environment without manual hacks. This change: - Updates the docs that latest minikube test is 1.33.0 - Setup minikube by default when creating the virtual environment, so minikube 1.33.0 support is added transparently for developers - Setup drenv in the e2e job to support minikube 1.33.0 - Cleanup drenv in the e2e job so setup from previous build will not leak into the next build [1] kubernetes/minikube#18705 [2] https://gitlab.com/qemu-project/qemu/-/issues/237 Thanks: Alex Kalenyuk <akalenyu@redhat.com> Signed-off-by: Nir Soffer <nsoffer@redhat.com>
medyagh
changed the title
@llegolas do you mind confirming one more time that this issue did happen to you for ubuntu ? I have had 3 different people try on ubuntu and none of them could reproduce this issue, it would be nice if we can confirm this is only related to Fedora so we could have some sort of Pre-Detection Mechasim for this |
|
i think the originally reported issue is to test, i created a fedora 39 vm and successfully replicated with minikube v1.33.0 to confirm this, can you please try the following:
note: although i tested various things, i think that the above are minimal (but not proper nor permanent) changes needed to be made to fedora host only to make dns resolution work in minikube vm (without any additional changes needed in minikube) beforeat host: at vm: afterat host: at vm: |
The setup I had for the ubuntu test was: host machine fedora 40(39 works all the same) ubuntu 23.04 VM inside the fedora and minikube inside the ubuntu. Sort of |
|
Fix is included in latest release of minikube: https://github.com/kubernetes/minikube/releases/tag/v1.33.1 |
What Happened?
minikube fails to start network cni (resolved.conf)
Attach the log file
It seem as a problem with systemd-resolved.service
running
times out with the following in the logs:
if I run
all is good
Operating System
Redhat/Fedora
Driver
KVM2
The text was updated successfully, but these errors were encountered: