Continuous testing for FreeBSD

[asomers]

I'm working on a buildbot cluster that could support continuous testing on FreeBSD for several projects. I've got a prototype running, and you can see a PR in action at asomers/mio-aio#2 . The hardest question is how to secure it. Since anybody can open a PR, that means that anybody can run arbitrary code on the buildslaves. The potential damage is limited; each project gets its own worker, each worker runs in its own jail, and there's a timeout on each build. I could use the firewall to prevent workers from sending email and stuff, but I can't completely isolate workers from the internet without breaking a lot of builds. There are a few options to improve the security situation.

1. Don't automatically build a PR until a maintainer posts a specific comment. This is what open-zfs does. It completely eliminates unreviewed code from running on the workers. However, it's inconvenient for people who are accustomed to Travis building stuff without needing to be asked.
2. Destroy and reclone the worker for each build. This is what Travis does. The worker still runs untrusted code, but not for long. The untrusted code can't modify the filesystem in any persistent way.
3. A hybrid of the previous two. Do a build when a maintainer give the magic comment, but also do builds automatically whenever a PR is posted by a known-good contributor. The list of contributors would probably have to be maintained by hand, but it could be much larger than the list of maintainers.

Does anybody have any better ideas?

[notriddle]

Since nix uses bors, your cluster doesn't need to have its own support for magic comments. All it needs is to be able to build branches on nix's repository and report results using the GitHub Commit Status API. The builds will run if a reviewer calls bors try or bors r+.

On the other hand, if you want to run all the pull requests, and you want suggestions to be better able to isolate the builds, you could use a project-owner-managed whitelist of internet hosts. That would be about as close as you could possibly get to eliminating all risk of bad actors using your build cluster to send comment spam.

[asomers]

Interesting. How would bors request a build from buildbot?

[notriddle]

It drops a merge commit into the right branch, either "trying" for a try build or "staging" for a merge build. BuildBot needs to have a webhook set up so that it automatically builds when those branches change.

[asomers]

Great news! It's easy to setup that kind of hook, as long as the branch name matches a regex. I'll try it out tonight.

[Susurrus]

I wanted to point out here that GitLab CI already supports FreeBSD with their runners. We'd need to have this repo on GitLab to utilize this, but I wanted to point this out in case it's a reasonable path forward. If they've already solved the hard part and it just requires installing a runner, then that seems pretty low-effort. I do not, however, know their plans for future platform support moving forward, but it is a FOSS project, so there's that benefit.

[notriddle]

BuildBot supports FreeBSD fine, as well. The question is how it should be configured.

[asomers]

@Susurrus that's nice. I didn't know GitLab had its own CI system. And it's hosted version looks free, which is nice. Price will be an issue, though, since you need someplace to host the runners. Gitlab's free shared runners are all Linux, and it looks like they don't even have container-level isolation. I'll have to do some more research to see how hard it would be to share a FreeBSD runner (with container-level isolation) amongst multiple projects.

One big downside to switching to GitLab would be losing test coverage on OSX. While there are OSX VPS services, their pricing is unattractive, especially without containers.
https://www.hostmyapple.com/macvps.html

Thanks for the tip. It looks like I've got some more homework to do.

[asomers]

I got a FreeBSD runner setup to work with a GitLab project, and successfully completed a build. Unfortunately, GitLab doesn't yet have any support for sharing FreeBSD runners between projects in a containerized way. My next task will be to see how hard it would be to add that feature.
https://gitlab.com/asomers/mio-aio/commit/0323a4ab6a6e299d5930fb7ae628a6a23c90d8cb/pipelines

[Susurrus]

Your GitLab link is broken. Is that repo public there?

[asomers]

Whoops! Try again.

[Susurrus]

I can see the project, but nothing in it. No files, commits, or pipelines are public.

[asomers]

I'm clearly a beginner at Gitlab. Try again. I think you should be able to see everything and submit pull requests, but not trigger CI builds.

[Susurrus]

Interesting, I see everything but the pipelines now on the GitLab page. I see that it "Passed" with their badge, but clicking on it takes me to a 404. Interesting.

[asomers]

@Susurrus Try again. And if you get another 404, please tell me the URL that failed.

[Susurrus]

Ha! Finally it worked. Did you have to enable permissions on the pipelines or something?

[asomers]

Yes. But the settings dialog implies that everyone now has the ability to submit pipeline jobs. It doesn't seem that there's such a thing as read-only access to the pipeline. I'd be curious if you can submit a pull request and have it attempt to start a build (if so, the build will hang because my runner VM is shut down ATM).

[Susurrus]

Yeah, it's definitely a little weird, but I guess you either want builds for everyone or builds just for you. Would you like me to submit an MR?

[asomers]

Sure. Let's see what happens. Just try to merge a syntax error or something.

[Susurrus]

https://gitlab.com/asomers/mio-aio/merge_requests/3

Actually a useful MR. If you'd prefer one that errors, I can do another one.

[asomers]

It's pretty much all working now. I have asomers/mio-aio setup to use buildbot, and it uses a fresh jail for each build. That takes a little longer, but it mitigates the security concerns of automatically building PRs. It also uses more upload bandwidth on the build server, but my host doesn't charge for bandwidth in that direction. Checkout the latest PR results here:
asomers/mio-aio#6
https://162.213.36.222/buildbot/#/builders/2/builds/28

GitLab has a nicer UI. It also makes it possible for multiple projects to share a worker, without sharing any build UI. I think I could even make GitLab use a separate jail for each build, though I haven't yet. But GitLab has one major drawback: a lack of free OSX runners. Travis CI doesn't work with GitLab, and ever other OSX hosting service I've found is quite expensive. Even if Nix decides that it doesn't need automatic OSX builds, many other projects are addicted to them, thanks to Travis. So I think BuildBot is a more general solution.

[Susurrus]

Well it sounds like we have a working solution here. Man I wish there was a CI service that just ran buildbots on the correct systems.

As to GitLab, I'd love to give them this feedback. Would you be interested in emailing them and CCing me? Otherwise I could write this up if you'd like. Because I have the same issues with my cross-platform library crates hosted there. Maybe they should have bought Travis CI instead of Gitter.

[asomers]

There might be a buildbot hosting service soon. This company says they're going to do it, but they aren't open for business yet: https://exana.io/buildbot

Do you have any contact info for GitLab? I could write it up, but I'm not sure where to send it.

[Susurrus]

sales [at] gitlab.com would be a good place to do it I think.

I might suggest they should work closely with the buildbot team. That'd be great if that was the direction they went with their CI service. Really need that mac testing man.

[asomers]

I just hooked up buildbot, and it failed every pull request. It looks like it's trying to pull the PR from the wrong URL. I'll fix it, but in the meantime don't take a buildbot failure as blocker for any PR. Everybody can go ahead and merge stuff.

[Susurrus]

Yeah, they passed for #591 tho, but failed for a bunch of other PRs. Looks like we're close. I'm pretty surprised however that all the FreeBSD tests passed since we've never run them.

[asomers]

@Susurrus That's just because since I fixed it, very few PRs have been updated. #591 is one of them. And what do you mean "we've" never run the FreeBSD tests? Speak for yourself ;) .

I think everything should be working now, but we won't know for sure until Bors trys to merge something.

[Susurrus]

Well I'm glad someone's been running those tests, didn't realize you had been. Thanks!

Do you wanna give #591 a review and r+ for Bors if you're okay with it? Let's do a live test!

[notriddle]

You need to modify the bors.toml for it to actually gate on your BuildBot instance.