Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Port isolation on sandboxing on OS X doesn't work #2669

Closed
ittaiz opened this issue Mar 12, 2017 · 28 comments
Closed

Port isolation on sandboxing on OS X doesn't work #2669

ittaiz opened this issue Mar 12, 2017 · 28 comments
Assignees
@philwo
Labels
category: sandboxing type: feature request under investigation

Comments

@ittaiz
Copy link
Member

ittaiz commented Mar 12, 2017

Please provide the following information. The more we know about your system and use case, the more easily and likely we can help.

Description of the problem / feature request / question:

I'm trying to run tests in parallel which both use the same port. I was under the assumption that sandboxing applies to networking as well as to filesystem.

If possible, provide a minimal example to reproduce the problem:

https://github.com/ittaiz/bazel-port-isolation

Environment info

  • Operating System:
    OS X el capitan (10.11.6)

  • Bazel version (output of bazel info release):
    development version

  • If bazel info release returns "development version" or "(@non-git)", please tell us what source tree you compiled Bazel from; git commit hash is appreciated (git rev-parse HEAD):
    54b9d2c (Feb 20, 2017)

Have you found anything relevant by searching the web? (e.g. GitHub issues, email threads in the bazel-discuss@googlegroups.com archive)

Anything else, information or logs or outputs that would be helpful?

(If they are large, please upload as attachment or provide link).
@ittaiz
Copy link
Member Author

ittaiz commented Mar 12, 2017

Two more notes:

  1. tags = ["block-network"] doesn't help
  2. This sounds to me like this is a real limitation on the degree of parallelism one can gain from moving to bazel, isn't it?

@ulfjack
Copy link
Contributor

ulfjack commented Mar 14, 2017

Looking at the code, "block-network" ought to work. At least it's hooked up in the sandbox strategy.

@ulfjack
Copy link
Contributor

ulfjack commented Mar 14, 2017

On Linux only. There's no network sandboxing on Mac as far as I know. Oh, I see.

@ulfjack
Copy link
Contributor

ulfjack commented Mar 14, 2017

I can't find any supported way to virtualize the network on MacOS. I can't see how Docker is doing it, if it is doing it at all.

@ulfjack
Copy link
Contributor

ulfjack commented Mar 14, 2017

Maybe we can provide a way to reliably pick a per-test port in a way that doesn't conflict? Would that work for you @ittaiz ?

@philwo
Copy link
Member

philwo commented Mar 14, 2017

I'm not aware of any mechanism on macOS to virtualize the network and I thus don't think that there's any other software out there that can do it.

I would recommend to use a dynamic port (listen on TCP port 0, then the operating system will automatically and atomically assign you a random free port), then write the port number to a file that is read by whatever wants to connect to it. :)

Another idea would be to add alias IPs to your localhost interface: sudo ifconfig lo0 alias 127.0.0.2 (as many as you like), then you can use the same port (but would have to shard among those localhost alias IPs).

I'd be happy to add a network isolation feature to our macOS sandbox - if anyone has an idea how it could work, hints would be very welcome.

@ittaiz
Copy link
Member Author

ittaiz commented Mar 15, 2017 via email

@ittaiz
Copy link
Member Author

ittaiz commented Mar 15, 2017

This blog post about unikernels and docker for mac looks like a good place to start.
https://blog.docker.com/2016/05/docker-unikernels-open-source/

@damienmg damienmg assigned philwo and unassigned damienmg Mar 15, 2017
@ittaiz
Copy link
Member Author

ittaiz commented Mar 15, 2017

Specifically VPNKit sounds really relevant but I'm guessing @philwo will be a much better judge

@philwo
Copy link
Member

philwo commented Mar 15, 2017

Thanks for the pointers! I had a look at that stuff. Docker's HyperKit and VPNKit is pretty exciting technology, but - unless I'm completely missing something here - what they in the end do, is launch a Linux VM for each container that you're running. VPNKit is only usable when you also use HyperKit to run stuff in a VM, not on its own.

That's why Docker does not support running native macOS software inside a container. Their stuff is basically a more lightweight version of VirtualBox - and if you go to that length, of course you can do whatever network isolation you want, because you're emulating an entire machine with its own operating system for the purpose of running a single command.

The fact that you can't run macOS software with that approach means that this is not a viable sandboxing mechanism for Bazel on macOS in general. :(

That said, for Linux this would not be an issue and it would be very interesting if we could use Docker for sandboxing in general and I always wanted to give that a try. The performance characteristics make me rather sceptical though:

On my MacBook Pro, I'm seeing a 0.5s wall time on average for "while true; do time docker run docker/whalesay cowsay boo; done". If I run this in four terminals in parallel, this goes up to 2 seconds. This would mean that if Bazel were to execute all build actions using Docker, then for a build that runs 200 actions, 400 seconds will be spent just in Docker overhead.

I guess on Linux the overhead is lower, because they don't have to use full featured VMs there, but I can't benchmark that at the moment. It would also not help you with your macOS based tests. :|

@ittaiz
Copy link
Member Author

ittaiz commented Mar 15, 2017

I see...
Do these help as inspirations?
https://github.com/NetBSDfr/sailor (mainly with concepts to look at chroot and pkgin)
https://github.com/vito/houdini (mac garden backend)

@ulfjack
Copy link
Contributor

ulfjack commented Mar 15, 2017

I also looked at VPNKit, and I had the same impression as Philipp.

@ittaiz ittaiz changed the title Port isolation on sandboxing Port isolation on sandboxing on OS X doesn't work Apr 4, 2017
@gorset
Copy link

gorset commented May 6, 2017 •
edited
Loading

I'm struggling with this issue too. Testing with 0.5rc3, I find that block-network works flawlessly on linux, but on macos I get:

Caused by: java.net.SocketException: Operation not permitted
        at sun.nio.ch.Net.bind0(Native Method)
        at sun.nio.ch.Net.bind(Net.java:433)
        at sun.nio.ch.Net.bind(Net.java:425)
        at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
        at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
        at org.apache.cassandra.net.MessagingService.getServerSockets(MessagingService.java:461)

I tried falling back to exclusive using something like:

tags = select({":darwin": ["exclusive"], "//conditions:default": ["block-network"]}),

but this is unfortunately not supported:

attribute "tags" is not configurable.

Are there any other workarounds here (other than rewriting the tests)? An alternative approach that would have worked well for us is to have a version of exclusive that only require running exclusively in relation to other tests with the same tag. Then such tests will only run one at a time, but still let other tests run concurrently, making the build faster. Ideally, block-network could do this as a fallback on macos instead of making all tests using the tag fail when using the network.

This was referenced May 8, 2017
Closed
Closed
@philwo
Copy link
Member

philwo commented May 8, 2017

@gorset Thanks for the feedback. I filed two feature requests for your ideas.

Unfortunately, as we still don't know how to virtualize "localhost" on macOS, I'm not sure what we can do about the root cause here. :(

@ittaiz
Copy link
Member Author

ittaiz commented May 8, 2017 via email

@philwo
Copy link
Member

philwo commented May 8, 2017

one could try and use a custom sandbox script which uses docker

"bazel build" on Mac produces macOS binaries, but Docker(-on-Mac) runs everything inside a Linux VM, so it can't run macOS binaries. How would this work? I think this could only work if your project, when built, only consists of 100% cross-platform binaries (Java JARs without JNI, shell scripts, Python)? And this would also only work due to luck - it is not guaranteed that java_binary or java_test, which currently result in a bash shell script (as a wrapper) + the Java jars, won't give you a native executable on a different platform.

I think solving this correctly would mean to teach Bazel that it is running on macOS, but has to compile for Linux, which it can then run using Docker. This would be a great project. If I remember correctly, @lberki and @ulfjack worked on "platforms" for Bazel in the past and might know more about our current plans.

Personally, if I had the problem of wanting to run thousands of end-to-end tests today, I would just solve it by running the tests on a Linux server (or Linux VM on the local system - which gives you the same performance as Docker would). This gives you a solution right now. It could be as simple as this:

run-all-my-tests.sh:

#!/bin/bash
rsync -av --delete --checksum ~/workspace/ linux:~/workspace/ &&
ssh -t linux "cd ~/workspace && bazel test //..."

@alexeagle
Copy link
Contributor

I'm also trying to run bazel in docker-on-mac and believe I'm hitting this issue, though the error does not have a stack dump pointing to a networking issue, I simply get
WARNING: Sandboxed execution is not supported on your system and thus hermeticity of actions cannot be guaranteed. See http://bazel.build/docs/bazel-user-manual.html#sandboxing for more information. You can turn off this warning via --ignore_unsupported_sandboxing.

@ittaiz
Copy link
Member Author

ittaiz commented May 26, 2017 via email

@alexeagle
Copy link
Contributor

I'm trying to get sandboxing working at all (for the filesystem first). Maybe I should file a separate one.

@ittaiz
Copy link
Member Author

ittaiz commented May 26, 2017 via email

@or-shachar
Copy link
Contributor

or-shachar commented Jun 20, 2017 •
edited
Loading

Hey everyone,
Just conducted an experiment with 0.5.1 on Linux - seems like it's not working there as well.
https://stackoverflow.com/questions/44655417/port-isolation-with-bazel-and-linux

I did not get any warnings regarding sandbox execution like @alexeagle ... that's why I suspect something is wrong with how I tested.

Before I'm opening a new issue - Will be happy to understand if I am failing to run it in the correct way.

UPDATE: solved by unprivileged_userns_clone

@philwo
Copy link
Member

philwo commented Jun 20, 2017 •
edited
Loading

@or-shachar I'll answer on StackOverflow in more detail, but can you please try to add the "block-network" tag to your tests that require it? This should make it work on Linux. We can't make this the default, because of a very big performance hit due to a bottleneck in the Linux kernel regarding creation / deletion of network namespaces.

The documentation for this tag is here: https://bazel.build/versions/master/docs/be/common-definitions.html#common-attributes (search for "block-network").

@or-shachar
Copy link
Contributor

or-shachar commented Jun 20, 2017 •
edited
Loading

@philwo - thanks! I found out the issue was with unprivileged_userns_clone. Once I enabled that - it worked well. Now my issue is running this from within docker privileged container. Opened a new SO question for that: https://stackoverflow.com/questions/44656481/port-isolation-with-bazel-inside-docker

UPDATE: works on docker with current HEAD (0433a20) Not sure what solved it but it was solved.

@gorset gorset mentioned this issue Jul 26, 2017
Closed
@talya talya mentioned this issue May 16, 2018
Closed
@philwo
Copy link
Member

philwo commented Jul 19, 2018

Closing, because supporting this is just not feasible on macOS.

@philwo philwo closed this as completed Jul 19, 2018
@wildmolasses wildmolasses mentioned this issue Feb 1, 2019
Closed
@ittaiz
Copy link
Member Author

ittaiz commented Aug 15, 2020

I remember Microsoft releasing something over the past year or two that does something like this for Mac. Has anyone had a chance to take a look at this and see if it's relevant for bazel? can't seem to find links for it right now

@philwo
Copy link
Member

philwo commented Aug 21, 2020

@laszlocsomor Do you remember what came out of the GSoC project in this area?

@ittaiz I think this might be the link you were looking for: https://github.com/microsoft/BuildXL/blob/master/Documentation/Wiki/Installation.md#macos-sandbox-kernel-extension

@laszlocsomor
Copy link
Contributor

@philwo : Yes! See #5136 (comment)

@ittaiz
Copy link
Member Author

ittaiz commented Aug 26, 2020

Thanks @philwo!
Exactly what I was referring to. Why did you think of the experimental windows sandbox? Because one might be able to combine the two together?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@philwo philwo

Labels
category: sandboxing type: feature request under investigation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@alexeagle @gorset @steren @philwo @ittaiz @or-shachar @damienmg @ulfjack @laszlocsomor