Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

port redirects broken when corectld is running corectld #105

Open
vitaly opened this issue Nov 30, 2016 · 12 comments
Open

port redirects broken when corectld is running corectld #105

vitaly opened this issue Nov 30, 2016 · 12 comments

Comments

@vitaly
Copy link

vitaly commented Nov 30, 2016

I have a custom anchor for my port redirects (port 80 to mehserve), but once corectld is running those rules disappear.

I have a script to print PF rules: https://gist.github.com/vitaly/cd0024d232a3a134b00e5496e4e779b8

here is the output w/o corectld: https://gist.github.com/vitaly/10b9f4cd467370c226daf78424f02816
here is the output with corectld: https://gist.github.com/vitaly/b40d230cc72b88fbcb7c1ade87a12163

The difference is this:

--- 1	2016-11-30 15:10:05.000000000 +0100
+++ 2	2016-11-30 15:10:27.000000000 +0100
@@ -1,9 +1,6 @@
-                                        r     scrub-anchor "com.apple/*" all fragment reassemble
-                                        r     anchor "com.apple/*" all
-                                        n     nat-anchor "com.apple/*" all
-                                        n     nat-anchor "custom" all
-                                        n     rdr-anchor "com.apple/*" all
-                                        n     rdr-anchor "custom" all
+                                        n     nat-anchor "com.apple.internet-sharing" all
+                                        n     rdr-anchor "corectl-dns-forwarding" all
+                                        n     rdr-anchor "com.apple.internet-sharing" all
                                         A       com.apple
                                         A       com.apple.internet-sharing
                                         A       corectl-dns-forwarding

Note that not only my 'custom' anchors are skipped but the default apple ones as well.

@vitaly
Copy link
Author

vitaly commented Nov 30, 2016

I don't know go, so I'd rather not try to change the code (for now ;)). but I think I do know what needs to be done in terms of PF.

right now it does pfctl -F pfC_file which in turn loads pfR_file. pfC declares 2 anchors: corectl-dns-forwarding, and com.apple.internet-sharing, and then loads corectl-dns-forwarding from pfR. The problem is that it clears the default "root" config (e.g. all com.apple/* anchors).

but default config already uses rdr rules from com.apple/*, so instead we can just load pfR directly into this namespace:

pfctl -a com.apple/corectl-dns-forwarding -f pfR_file

and to remove it:

pfctl -a com.apple/corectl-dns-forwarding -Fa

@AntonioMeireles
Copy link
Member

It'a a bug, my fault - you're spot on. I' m on the road ATM, will fix it tomorrow. And many, many thanks!

@vitaly
Copy link
Author

vitaly commented Nov 30, 2016

@AntonioMeireles it looks like kube-solo also messes up with it. Not sure if directly or through corectl though.

AntonioMeireles added a commit that referenced this issue Dec 1, 2016
- handles #105. corectld now respects all sorts of custom host pf
  settings

Signed-off-by: António Meireles <antonio.meireles@reformi.st>
@AntonioMeireles
Copy link
Member

@vitaly

Hi again, could you please fetch latest tip, built it, and confirm that it fixes this for you ?
Thanks in advance!

@vitaly
Copy link
Author

vitaly commented Dec 2, 2016

@AntonioMeireles please see my comment on #106

couldn't build it, not a golang user.

@vitaly
Copy link
Author

vitaly commented Dec 2, 2016

OK, so I finally got it to compile using hints from /Applications/corectl.app/Contents/Resources/get_go_binaries/corectl_compile_from_source_corectl.sh

DNS port configuration seems to be working, but PF rules are still messed up. Actually it seems to be worse, as I can't even connect outside from a VM.

Note: I'm again using my pf rules dump script, as it produces a much shorter output and much easier to diff. You can see the script here: https://gist.github.com/vitaly/cd0024d232a3a134b00e5496e4e779b8

rules after firewall reset:

                                        r     scrub-anchor "com.apple/*" all fragment reassemble
                                        r     anchor "com.apple/*" all
                                        n     nat-anchor "com.apple/*" all
                                        n     rdr-anchor "com.apple/*" all
                                        A       com.apple
                                        A       com.apple.internet-sharing
com.apple                               r     anchor "200.AirDrop/*" all
com.apple                               r     anchor "250.ApplicationFirewall/*" all
com.apple                               A       com.apple/200.AirDrop
com.apple                               A       com.apple/250.ApplicationFirewall
com.apple                               A       com.apple/mehserve
com.apple/200.AirDrop                   A       com.apple/200.AirDrop/Bonjour
com.apple/200.AirDrop/Bonjour           r     pass in on p2p0 inet6 proto udp from any to any port = 5353 keep state
com.apple/200.AirDrop/Bonjour           r     pass out on p2p0 proto tcp all flags any keep state
com.apple/250.ApplicationFirewall       r     scrub in all fragment reassemble
com.apple/250.ApplicationFirewall       r     block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall       r     block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall       r     block drop in inet6 proto ipv6-icmp all icmp6-type echoreq
com.apple/mehserve                      n     rdr pass inet proto tcp from any to 127.0.0.1 port = 80 -> 127.0.0.1 port 12439

rules after corectld started:

                                        r     scrub-anchor "com.apple/*" all fragment reassemble
                                        r     anchor "com.apple/*" all
                                        n     nat-anchor "com.apple/*" all
                                        n     rdr-anchor "com.apple/*" all
                                        A       com.apple
                                        A       com.apple.internet-sharing
com.apple                               r     anchor "200.AirDrop/*" all
com.apple                               r     anchor "250.ApplicationFirewall/*" all
com.apple                               A       com.apple/200.AirDrop
com.apple                               A       com.apple/250.ApplicationFirewall
com.apple                               A       com.apple/corectl-dns-forwarding
com.apple                               A       com.apple/mehserve
com.apple/200.AirDrop                   A       com.apple/200.AirDrop/Bonjour
com.apple/200.AirDrop/Bonjour           r     pass in on p2p0 inet6 proto udp from any to any port = 5353 keep state
com.apple/200.AirDrop/Bonjour           r     pass out on p2p0 proto tcp all flags any keep state
com.apple/250.ApplicationFirewall       r     scrub in all fragment reassemble
com.apple/250.ApplicationFirewall       r     block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall       r     block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall       r     block drop in inet6 proto ipv6-icmp all icmp6-type echoreq
com.apple/corectl-dns-forwarding        n     nat-anchor "com.apple.internet-sharing" all
com.apple/corectl-dns-forwarding        n     rdr-anchor "corectl-dns-forwarding" all
com.apple/corectl-dns-forwarding        n     rdr-anchor "com.apple.internet-sharing" all
com.apple/corectl-dns-forwarding        A       com.apple/corectl-dns-forwarding/com.apple.internet-sharing
com.apple/corectl-dns-forwarding        A       com.apple/corectl-dns-forwarding/corectl-dns-forwarding
com.apple/mehserve                      n     rdr pass inet proto tcp from any to 127.0.0.1 port = 80 -> 127.0.0.1 port 12439

here is the diff:

--- 1-after-reset	2016-12-02 03:49:00.000000000 +0100
+++ 2-after-corectld-started	2016-12-02 03:50:50.000000000 +0100
@@ -9,6 +9,7 @@
 com.apple                               r     anchor "250.ApplicationFirewall/*" all
 com.apple                               A       com.apple/200.AirDrop
 com.apple                               A       com.apple/250.ApplicationFirewall
+com.apple                               A       com.apple/corectl-dns-forwarding
 com.apple                               A       com.apple/mehserve
 com.apple/200.AirDrop                   A       com.apple/200.AirDrop/Bonjour
 com.apple/200.AirDrop/Bonjour           r     pass in on p2p0 inet6 proto udp from any to any port = 5353 keep state
@@ -17,4 +18,9 @@
 com.apple/250.ApplicationFirewall       r     block drop in inet proto icmp all icmp-type echoreq
 com.apple/250.ApplicationFirewall       r     block drop in inet proto icmp all icmp-type echoreq
 com.apple/250.ApplicationFirewall       r     block drop in inet6 proto ipv6-icmp all icmp6-type echoreq
+com.apple/corectl-dns-forwarding        n     nat-anchor "com.apple.internet-sharing" all
+com.apple/corectl-dns-forwarding        n     rdr-anchor "corectl-dns-forwarding" all
+com.apple/corectl-dns-forwarding        n     rdr-anchor "com.apple.internet-sharing" all
+com.apple/corectl-dns-forwarding        A       com.apple/corectl-dns-forwarding/com.apple.internet-sharing
+com.apple/corectl-dns-forwarding        A       com.apple/corectl-dns-forwarding/corectl-dns-forwarding
 com.apple/mehserve                      n     rdr pass inet proto tcp from any to 127.0.0.1 port = 80 -> 127.0.0.1 port 12439

Note the weird anchors: com.apple/corectl-dns-forwarding/com.apple.internet-sharing, and com.apple/corectl-dns-forwarding/corectl-dns-forwarding.

Once you properly nest your rules inside com.apple/*, there is no need to declare the anchors, just the rules.

Since at this stage you don't yet add any commands, I think instead of adding nat-anchor and rdr-anchor you only need to reset them (Also note the renamed internet-sharing anchor to properly nest it inside com.apple/):

sudo pfctl -a com.apple/corectl-dns-forwarding -Fa
sudo pfctl -a com.apple/internet-sharing -Fa

But at least at this point my mehserve is still working.

Now, here is how it looks once I run a VM with it:

                                        r     scrub-anchor "com.apple/*" all fragment reassemble
                                        r     scrub-anchor "com.apple.internet-sharing" all fragment reassemble
                                        r     anchor "com.apple/*" all
                                        r     anchor "com.apple.internet-sharing" all
                                        n     nat-anchor "com.apple/*" all
                                        n     nat-anchor "com.apple.internet-sharing" all
                                        n     rdr-anchor "com.apple/*" all
                                        n     rdr-anchor "com.apple.internet-sharing" all
                                        A       com.apple
                                        A       com.apple.internet-sharing
com.apple                               r     anchor "200.AirDrop/*" all
com.apple                               r     anchor "250.ApplicationFirewall/*" all
com.apple                               A       com.apple/200.AirDrop
com.apple                               A       com.apple/250.ApplicationFirewall
com.apple                               A       com.apple/corectl-dns-forwarding
com.apple                               A       com.apple/mehserve
com.apple/200.AirDrop                   A       com.apple/200.AirDrop/Bonjour
com.apple/200.AirDrop/Bonjour           r     pass in on p2p0 inet6 proto udp from any to any port = 5353 keep state
com.apple/200.AirDrop/Bonjour           r     pass out on p2p0 proto tcp all flags any keep state
com.apple/250.ApplicationFirewall       r     scrub in all fragment reassemble
com.apple/250.ApplicationFirewall       r     block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall       r     block drop in inet proto icmp all icmp-type echoreq
com.apple/250.ApplicationFirewall       r     block drop in inet6 proto ipv6-icmp all icmp6-type echoreq
com.apple/corectl-dns-forwarding        n     nat-anchor "com.apple.internet-sharing" all
com.apple/corectl-dns-forwarding        n     rdr-anchor "corectl-dns-forwarding" all
com.apple/corectl-dns-forwarding        n     rdr-anchor "com.apple.internet-sharing" all
com.apple/corectl-dns-forwarding        A       com.apple/corectl-dns-forwarding/com.apple.internet-sharing
com.apple/corectl-dns-forwarding        A       com.apple/corectl-dns-forwarding/corectl-dns-forwarding
com.apple/mehserve                      n     rdr pass inet proto tcp from any to 127.0.0.1 port = 80 -> 127.0.0.1 port 12439
com.apple.internet-sharing              r     scrub-anchor "shared_v4" all fragment reassemble
com.apple.internet-sharing              r     anchor "shared_v4" all
com.apple.internet-sharing              n     nat-anchor "shared_v4" all
com.apple.internet-sharing              n     rdr-anchor "shared_v4" all
com.apple.internet-sharing              A       com.apple.internet-sharing/shared_v4
com.apple.internet-sharing/shared_v4    r     scrub on en0 all no-df fragment reassemble
com.apple.internet-sharing/shared_v4    r     scrub on bridge100 all no-df max-mss 1460 fragment reassemble
com.apple.internet-sharing/shared_v4    r     scrub on bridge100 proto esp all no-df fragment reassemble
com.apple.internet-sharing/shared_v4    r     pass on en0 all flags any keep state
com.apple.internet-sharing/shared_v4    r     pass on en0 proto esp all no state
com.apple.internet-sharing/shared_v4    n     nat on en0 inet from 192.168.64.0/24 to any -> (en0:0) extfilter ei
com.apple.internet-sharing/shared_v4    n     no nat on bridge100 inet from 192.168.64.1 to 192.168.64.0/24

here is the diff:

--- 2-after-corectld-started	2016-12-02 03:50:50.000000000 +0100
+++ 3-after-corectl-run	2016-12-02 03:53:11.000000000 +0100
@@ -1,8 +1,12 @@
 ./pfdump
                                         r     scrub-anchor "com.apple/*" all fragment reassemble
+                                        r     scrub-anchor "com.apple.internet-sharing" all fragment reassemble
                                         r     anchor "com.apple/*" all
+                                        r     anchor "com.apple.internet-sharing" all
                                         n     nat-anchor "com.apple/*" all
+                                        n     nat-anchor "com.apple.internet-sharing" all
                                         n     rdr-anchor "com.apple/*" all
+                                        n     rdr-anchor "com.apple.internet-sharing" all
                                         A       com.apple
                                         A       com.apple.internet-sharing
 com.apple                               r     anchor "200.AirDrop/*" all
@@ -24,3 +28,15 @@
 com.apple/corectl-dns-forwarding        A       com.apple/corectl-dns-forwarding/com.apple.internet-sharing
 com.apple/corectl-dns-forwarding        A       com.apple/corectl-dns-forwarding/corectl-dns-forwarding
 com.apple/mehserve                      n     rdr pass inet proto tcp from any to 127.0.0.1 port = 80 -> 127.0.0.1 port 12439
+com.apple.internet-sharing              r     scrub-anchor "shared_v4" all fragment reassemble
+com.apple.internet-sharing              r     anchor "shared_v4" all
+com.apple.internet-sharing              n     nat-anchor "shared_v4" all
+com.apple.internet-sharing              n     rdr-anchor "shared_v4" all
+com.apple.internet-sharing              A       com.apple.internet-sharing/shared_v4
+com.apple.internet-sharing/shared_v4    r     scrub on en0 all no-df fragment reassemble
+com.apple.internet-sharing/shared_v4    r     scrub on bridge100 all no-df max-mss 1460 fragment reassemble
+com.apple.internet-sharing/shared_v4    r     scrub on bridge100 proto esp all no-df fragment reassemble
+com.apple.internet-sharing/shared_v4    r     pass on en0 all flags any keep state
+com.apple.internet-sharing/shared_v4    r     pass on en0 proto esp all no state
+com.apple.internet-sharing/shared_v4    n     nat on en0 inet from 192.168.64.0/24 to any -> (en0:0) extfilter ei
+com.apple.internet-sharing/shared_v4    n     no nat on bridge100 inet from 192.168.64.1 to 192.168.64.0/24

Seems like it is rewriting the global scope again, which is not needed if you just nest the rules inside com.apple/.

At this stage when I corectl ssh into the VM I can't even ping 8.8.8.8, where previously I could ping it, just couldn't resolve any hostnames (due to DNS port conflict, the forwarding was working)

Note that the DNS rules are missing now. I tried adding them, but it didn't help (din't expect it to, it has nothing to do with ping)

I tried to mess with the rules for a while, but couldn't get it to work, so I reverted to the old version for now.

AntonioMeireles added a commit that referenced this issue Dec 2, 2016
Signed-off-by: António Meireles <antonio.meireles@reformi.st>
@AntonioMeireles
Copy link
Member

@vitaly

sorry lag, can you try again against latest tip ? (and many thanks for your patience!)

AntonioMeireles added a commit that referenced this issue Dec 3, 2016
Signed-off-by: António Meireles <antonio.meireles@reformi.st>
@vitaly
Copy link
Author

vitaly commented Dec 5, 2016

just tried the latest tip.

result:

  • DNS and ping work from inside VM
  • my port redirection doesn't ;(

PF dumps: https://gist.github.com/vitaly/fc0cb6986a2a3d92fb86b2239c8cf1cf

Looking at the current corectl pf config, I see that it was done outside of 'com.apple' on purpose ;). and, to my untrained eye it actually looks OK vs breaking other rules. i.e. I can't see how it can break my redirect, but it does ;(. On the other hand I also don't understand how your corectld-forwarding-dns anchor works. it is not referenced from root or any other anchors (btw, can you please explain this? ;)). it clearly does work though, because I do have DNS inside the VM, but I really can't figure out how it manages to do that. which means I have no idea how this PF thing actually works ;)

@vitaly
Copy link
Author

vitaly commented Dec 5, 2016

I think I already mentioned this, but it might be worth pointing out again.

My redirect still works once corectld is started and stops working as soon as I run any VM

BUT

it still doesn't work even after I stop VM and corectd, even though PF dump looks exactly like before corectld was started

only once I run sudo pfctl -f /etc/pf.conf it starts working again, but I can not detect any difference in PF config before and after I do that. The only idea that I have is that its some kind of transient state that is getting cleared when I re–run the root scope config.

Hopefully this helps ... ;)

@AntonioMeireles
Copy link
Member

interesting... can you paste your exact redirect rules and how you are loading them please... (and once again MANY thanks for your patience!)

@genevera
Copy link
Contributor

Pinging as I have a similar problem. corectl and sshuttle also don't get along so well.

@genevera
Copy link
Contributor

@vitaly and @AntonioMeireles maybe pfctl needs to be called with -E, the token stored somewhere, then torn down with -X + that token? That looks like that might solve this particular problem.

AntonioMeireles added a commit that referenced this issue Sep 12, 2017
Signed-off-by: António Meireles <antonio.meireles@reformi.st>
genevera pushed a commit to genevera/corectl that referenced this issue Jan 8, 2019
Signed-off-by: António Meireles <antonio.meireles@reformi.st>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants