Firewall randomly messing with Multipass networking on macOS #2387
Comments
|
Hey @milopersic, It's not clear to me at all why qemu crashes, but it looks like it does start again to the point where the instance's virtual network adapter comes up. If you don't mind, could you post the contents of |
|
Sure thing, thank you. This shows some of the other instances I created previously on the first installation of multipass. For what it's worth, I get "status unknown" on my instances after attempting a restart.
/var/db/dhcpd_leases below: (I'm assuming you wanted these from my host machine, the M1 mac) |
|
Is that the full file contents? I ask because the top of what you posted looks wrong, ie, the is a dangling entry and shows a corrupted leases file. Multipass only opens this file read-only, so if this is the case, then something in macOS is corrupting this file and causing us issues. If those two lines are truly like that in your file, please remove those two lines only and try again. |
|
My mistake, apologies! I didn't post the full file. Repost: |
|
Ok, that looks better as far as a corrupted leases file 😉 But gah, stupid Apple's bootp has duplicate entries in the file and past experience shows that if there are entries with the same name, none of the entries work. Please remove any duplicate entries based in the |
|
:) thanks. I'll try it! To be clear, by "duplicates" you mean duplicates by name? No other info is relevant? Wait you just said that. Sorry, long day. |
|
Sorry, I mean the whole entry. For example, there are 2 entries for the and You will need to remove both of those full entries, from the |
|
Okay, that helps. The instance you pasted in on top here (ip_address=192.168.64.11) is my latest instance and one that I was using. Can you confirm that I should remove that? What should be kept? |
|
Please stop any "running" instances, ie, |
|
Sadly still not able to start the instance. I stopped and exited multipass, edited out the duplicates and both entries for dbarn, restarted multipass, and attempted to restart dbarn: At present, my dhcpd_leases file only has "primary" in it, which also does not start. Here are the latest logs (I think I'm going far back enough but let me know): |
|
Ok, I'm betting the bootp has some entries cached in memory. The easiest way to clear that up is to restart the machine. Please let me know if that is not possible. Sorry for all of the headaches with this |
|
Hey Chris, can't tell you how much I appreciate the help. I can get a reboot in. I'll do that now and retry starting the instance(s) and let you know what happens. |
|
Well, no luck after a restart: |
|
Ugh, and the Also, let's see if starting it by hand in a terminal will work. Try I tried to escape spaces in the paths, but you may have to play around with those. In theory, this should open a QEMU window and you should be able to observe the boot process and see if there are any errors or that the instance is getting stuck booting. |
|
I think you escaped it correctly because no error was thrown in the terminal. However it launched a QEMU terminal window and locked my cursor. Tried alt+cmd+G (as indicated) and no luck. Also experienced some UI glitches. Had to perform two successive hard resets to get my input ability back. Weird stuff! After coming back up, I tried starting dbarn again via the menu bar app. No luck, still times out. Anything else I should try? |
|
Oh, good grief! Well, when the QEMU window was open, did you happen to see if it booted to a login prompt? Also, do you see a I'm really running out of ideas here. I'll give it some thought and post anything I can think of tomorrow. |
|
I don't recall what was in the terminal. Sounds like a plan, thanks! In the meantime I can spool up a remote dev environment on my server and keep on truckin' with that. |
|
Hi guys, |
|
Well what do you know. Completely disabling the firewall on macOS actually worked and I was able to start my instances normally. I didn't think this was the problem, as you can see I'm not blocking all incoming connections and I have checked "automatically allow built-in software to receive incoming connections" (I guess Multipass is not treated as built-in software?).
Canonical's troubleshooting page mentions that if the firewall is enabled, it should not "block all incoming connections" so I thought I was good. It does not say the firewall should be fully disabled. https://multipass.run/docs/troubleshooting-networking-on-macos Turning off the firewall completely doesn't seem like the ideal solution. I turned my firewall back on and added a rule for Multipass, to Allow all incoming connections. This did not work! Even with that rule the instances do not start. I'm nervous about using Multipass if I have to fully disable the firewall, but maybe I'm wrong. |
|
Hi @milopersic, Glad you've made some progress on diagnosing what the problem is! Our guide is based more on macOS 10.x and it appears there have been changes surrounding the firewall in newer versions of macOS. I don't have a M1-based Mac at my disposal, but I do have an Intel Mac and will try to see if I can reproduce this and if so, determine what can be done aside from completely disabling the firewall. I agree, that is not an ideal solution. |
|
Thank you @townsend2010 ! M1 is still very new and we're all figuring out the quirks. Good luck on researching the issue and fix. If I can assist you by trying anything let me know. Multipass is an amazing application and this is just a bump in the road. |
|
Thanks for the kind words @milopersic 🙂 I'll definitely let you know if I find anything. |
|
Awesome! For what it's worth, I'll recap or add a few facts re: my experience in case any of this is relevant:
What's confusing to me is why it would work at all for any length of time if the firewall is the issue, but who knows what macOS is doing under the hood. |
|
Great summary @milopersic! The macOS networking is certainly mysterious. I suppose no updates to Monterey happened around the time the firewall began to cause Multipass issues, right? |
|
You know, MacOS 12.1 was released December 13, 2021. I don't recall the exact day I had the first crash but it may well have been right after the update. |
|
Ok, good to know. I'm updating my Intel Mac to 12.1. We'll see what happens 😉 |
|
I would like to add I am also using a similar set up (M1 Pro, Monterey 12.0.1) and have encountered today what I believe is the same issue as @milopersic. Unfortunately this is a company issued machine that I do not have privileges to disable the firewall on. Brief description below and logs similar to what has already been reported follow: Last night I had Multipass working just fine. Today after rebooting my machine my VMs failed to start. I tried clearing out entries for my VMs from I also confirmed a simple multipassd.log |
|
I'm in the same situation as @rjoelnorgren (work laptop) but I have been able to work around this by adding the following from |
|
@morphologue, thank you for the suggestion. Unfortunately that did not resolve the issue on my end. |
|
@ricab - annoyingly, that reporting process seems to require you to be on the a beta program to submit feedback. I can run the Feedback Assistant app but not submit anything. If there is a way to enroll in the MacOS beta program, I probably wouldn't be able to on my primary work device. I want to check I'm experiencing the same issue because unfortunately none of the steps mentioned here seem to resolve the problem at the moment. I'm still experiencing this on MacOS Sonoma 14.1.1 (23B81), Multipass 1.12.2+mac with:
And I get this timeout starting instances: And in |
|
FYI, I have much success by running the below commands for the bootpd to start dhcp leases to work. I have to run once per boot to fix my dhcp leasing. The issue is more prominent when the internet sharing is enabled for me. This is definitely not a permanent solution, but a temporary work around, I am running Sonoma, M1 Max. Hope it helps. |
|
Unfortunately my MacBook has the firewall enabled via an MDM profile so those commands give this error: The firewall isn't currently set to block incoming connections and I also excluded <dict>
<key>ProfileDescription</key>
<string>The configuration profile enables your company’s technical support to enforce security policies on your mobile device</string>
<key>ProfileDisplayName</key>
<string>Firewall Profile</string>
<key>ProfileIdentifier</key>
<string>www.windowsintune.com.security.firewall</string>
<key>ProfileInstallDate</key>
<string>2023-11-02 20:49:03 +0000</string>
<key>ProfileItems</key>
<array>
<dict>
<key>PayloadContent</key>
<dict>
<key>Applications</key>
<array>
<dict>
<key>Allowed</key>
<true/>
<key>BundleID</key>
<string>com.apple.bootpd</string>
</dict>
</array>
<key>EnableFirewall</key>
<true/>
</dict>
<key>PayloadDisplayName</key>
<string>Firewall Profile</string>
<key>PayloadIdentifier</key>
<string>www.windowsintune.com.security.firewall.payload</string>
<key>PayloadOrganization</key>
<string>ORG</string>
<key>PayloadType</key>
<string>com.apple.security.firewall</string>
<key>PayloadUUID</key>
<string>ef5ef3d1-a3bf-465a-b887-885f78367e39</string>
<key>PayloadVersion</key>
<integer>1</integer>
</dict>
</array>
<key>ProfileOrganization</key>
<string>ORG</string>
<key>ProfileRemovalDisallowed</key>
<string>true</string>
<key>ProfileType</key>
<string>Configuration</string>
<key>ProfileUUID</key>
<string>7863fde5-8080-423e-a20d-3e365970ee2b</string>
<key>ProfileVerificationState</key>
<string>verified</string>
<key>ProfileVersion</key>
<integer>1</integer>
</dict>Which at least shows up in the Firewall UI OK: |
I added multipass and actually the qemu as well to the firewall "allow incoming" and it worked |
|
@yossicohn-hs Make sure this didn't happen because of restart or something like that. You'd know 100% that it works if it starts working the moment you've added the firewall rules and not after rebooting. When restarting, multipass might work without making any changes. It's happened to me over and over. |
|
I found this whole thread helpful actually - I think that the reason this appears random is due to the GUI for the firewall in MacOS not accurately representing the state of the firewall - specifically, if you change settings on the dialog that pops up when you click "Options..." they will apparently not be applied until the firewall re-loads settings (which happens on a proper (important!) reboot). In my instance, here is the sequence of events. Last week I was feeling a bit paranoid and so took a pass through the Firewall settings on my work Mac and noticed that while the Firewall was "turned on" which "enables built-in software" and "signed software" to receive connections. This is different from the default configuration on my Linux machines which is to block all incoming connections so I changed it to "Block all incoming connections". I figured this might break Multipass so after doing this I immediately started a couple of my Multipass instances and they were fine so I figured something like "huh, it's clever enough to know not to apply firewall rules to the virtual interfaces - nice!". On Monday this week, I applied the 14.3 update which of course rebooted my Mac, but it did so in the weird "not-quite-entire-reboot" way that software updates do (I've previously noticed that software update reboots are different from "real" reboots and various things persist but not had time or cared enough to investigate further). Today, before an important meeting (of course) I did my usual "brew update" and this updated the Wacom tablet drivers. Because they are a cask, they don't... like that very much and usually rebooting fixes it. It fixed the tablet, but immediately broke multipass. I tried all sorts of things before I found this thread (rebooting, removing and reinstalling multipass, removing the tablet drivers...)- even changing the firewall settings in the GUI to no avail, until someone further up suggested rebooting after changing the settings. And indeed, with further testing it looks like changes to settings you make in the GUI under "Options..." do not immediately take effect. This is quite bad for all sorts of reasons, for example, I thought for about a week that my firewall had a "higher level" of protection than it actually had, but also makes the behaviour here seem random when in actual fact, it's just "delayed". Hope this helps someone. |
|
@milopersic Dude you're a lifesaver. Totally bewildered why multipass was not working after I had just used it yesterday - and now I'm back on track. |
|
Just adding this here in case it might be useful to someone. Ran into the same issue and MacOS version: 14.4.1
Allow the following:
After multiple times turning off & on the system it seemed to stay working yesterday (and still does today). Fingers crossed it stays this way. EDIT: Testing if it worked was done AFTER the first "turning it off and on again". Also note that the Macbook was turned off & on, not restarted. EDIT2: It stopped working again... very weird... |
|
Hi @svandenhoek for me even after disabling firewall as you mentioned above, i have to run these commands still after a bootup/reboot for inorder it to work, may be worth a try: |
|
@AravindGopala even after disabling the firewall?! Seems there's some very weird behaviour going on. For me running Next time it occurs I'll just try running that again and if it indeed fixes it consistently will either just create an alias or some script that runs during startup. |
|
I have this issue as well Will running these commands one-off fix the issue, or does this need to happen every time at startup? I am on macOS 14.4.1 and my laptop is managed by IT. I can have them run the commands once but I can't have this done on each reboot. |
|
What the above commands do, is register bootpd with the firewall and unblock it. You only need to do it once. That did not help me. I need to find a way to get my instances to listen to me. But its complains about network timeouts, which appear to be SSH is not up (which it is). This means incompatible network apparently. New instances created in this release work well. I had updated to the latest release. Which jumped a couple of releases. Seems network configuration might have changed enough but can't recover without manual edits. Any guidance someone can provide is appreciated. I mostly rebuilt my Instances and rewote the code, but I'd like to recover the old Instances. if nothing else, it provides less of a feeling that I am living on a knife's edge using Multipass. I love the product. Hate the risk. |
|
The interesting thing is that the person next to me is using 14.4.1 with the same firewall settings and it works for them. |
|
Some clarification is required. Any Instances created with this release seem solid. It's those created in a prior release I could not open as soon as the update took place. They were always solid, until the update. And then no network. Can't be opened. I did create a test instance in the new release which worked, after which I added to (192.168.64.X.) a Bridge network (10.0.1.X), since I had a similar Bridge network before. 10 is the gateway IP. When Test instance lot its 10 IP, I can get into Test with 192. But not the VMs form a prior Multipass release - they have lots of config and code. And while I have a backup image, it doesn't work so no value either - no network, so I'm stuck with dead dev VMs . And I should also add, I am aware of MacOS Firewall opaque behaviour. I run into them in dev as well. So I get that part. But I am surprised there is no way to open up these perfectly good VM instances. That's the knife edge. It can happen again, and that is unsettling. So recovering these VMs, is a bit of insurance on future failures. |
|
Hi @BainMcKay, if you can start some VMs but not others your problems are unrelated to the firewall. The symptoms you report can have different causes, some of them recoverable, some not so much. This thread is quite long as it is so let's continue discussing in the issue you've already opened. Please allow us a little time to reply. Thank you. |
|
Great. Thank you
|
|
I was able to disable the firewall with IT on Friday to A/B test this and I can confirm this solves the problem. I was not able to re-enable the firewall and add specific rules. If anyone can document exactly what multipass/qemu needs we can probably add this to the docs while a fix is found. I am available to test this if anyone has ideas, what I am trying right now is this. ❯ /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
Application at path ( /usr/libexec/bootpd ) added to firewall
❯ sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd
The application is not part of the firewallP.S. https://skowronski.tech/posts/2023-01-30-macos-application-firewall-and-multipass-debugging/ |
|
Ok. Got Something. #!/bin/bash
set -e
COUNTER=1
while true; do
echo ""
echo "set firewall ran $COUNTER times"
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --add /usr/libexec/bootpd
sudo /usr/libexec/ApplicationFirewall/socketfilterfw --unblock /usr/libexec/bootpd
COUNTER=$((COUNTER+1))
sleep 5
done
Firstly, it doesn't work without ... but the mystery continues ... I had a hunch that keeping the settings window open was racing to write the settings, so I closed it and re-ran the script (the updated version). It ran for few minutes, this time without resetting the rule again, which would validate my hypothesis. After verifying that launching a new VM worked (YAY), I opened my settings again and found the
I hope this helps someone, meanwhile I will keep updating this thread with my findings. |
and others
Update: If you are using an unmanaged Mac and run across this issue, the following commands may fix this issue:
Original post
Multipass crashes unexpectedly, and all instances (including primary) will not restart. I am using multipass 1.8.1+mac,
multipassd 1.8.1+mac. (multipass --version), on macOS Monterey 12.1, M1 Max, 32 GB RAM. I installed multipass for macOS directly from Canonical's website.
This has happened twice. The first time I uninstalled multipass, reinstalled, and re-built my instances thinking it was a fluke. Multipass performed fine for roughly a week, then crashed again and I lost some work. I am not able to determine what the cause is, or how to reproduce it. System restarts do not help. I'm hoping someone can spot the issue in the logs.
Logs here:
The text was updated successfully, but these errors were encountered: