Continuous build system

[ariya]

As with many other large software projects, it is imperative to have a stable and functional continuous build system to facilitate a better collaboration between the contributors.

[ariya]

Let's clarify the problems that we're facing without a CI system so far. This might be obvious but let's ensure that we are all on the same page.

The first and foremost, there is no automatic way to verify that every pull request will not cause a regression. With a CI system that integrates with GitHub pull request workflow, a build can be produced for each feature branch and the test suite against that build can be executed automatically.

Binary package is done manually by some of us, PhantomJS contributors. This means that the packager builds the binaries for Windows/Linux/OS X on their own machine. The packager becomes the bottleneck every time there has to be a new release. On top of that, this does not permit a binary build for any development versions, whether it is for every commit or for milestone releases (alpha, beta, preview, etc).

[ariya]

Now, how about some challenges?

Challenge 1: PhantomJS code base is huge and it takes a lot of time to build. Since there are tons of object files, 4 GB RAM is also required for the linker to produce the final executable. Hence, the build process must run on a beefy system because we can't wait forever until the build finishes.

Challenge 2: Since we want to treat every OS equally (no second-class citizen), the build system needs to facilitate producing the build and running the test suite for all supported platforms (Windows, OS X, Linux).

(Of course, there are also the following bits not necessarily specific to PhantomJS)

Challenge 3: The cost for running it should be zero (ideally) or very low (realistically).
Technically speaking, this limitation can be alleviated if there is an entity that is willing to cover the cost (without asking for anything in return). Even more awesome if there is a capable hosted CI system that will waive the cost.

Challenge 4: The build system should require zero (ideally) or minimal (realistically) maintenance effort.
This is quite hard to overcome. Every PhantomJS contributor rather focuses on developing PhantomJS than babysitting the build system. It is thus not likely to be negotiate-able (we can't magically "create" time for additional maintenance work).

[ariya]

Can't you install and use Jenkins on AWS?

In theory it is possible. In practice, someone needs to volunteer their time to do that (see also Challenge 4 described above).

Beside, Jenkins still needs a build slave to operate and that requires time, effort, and money (see Challenge 3).

[ariya]

Can't you use Travis CI?

This has been attempted before. The problem is that Travis CI build machine has limited resources and therefore the compilation will fail at some point because it exhausts all the available memory.

Beside, Travis CI does not offer Windows build.

[ariya]

Can't you use CircleCI?

According to https://circleci.com/pricing, the free tier of CircleCI only offers 1,500 build minutes per month. See also #13721.

Beside, CircleCI does not provide Windows or OS X build.

[ariya]

Can't you use HostedCI?

Already signed up for the "free" plan, https://hosted-ci.com/plan/free, and still not heard from them. There is also a limitation of 20 hours build time per month for this plan.

Beside, HostedCI only offers OS X build.

[ariya]

Can't you use AppVeyor?

According to http://www.appveyor.com/pricing, there is a build time limitation of 30 minutes per run. This is likely insufficient for PhantomJS.

Beside, AppVeyor is only for building on Windows.

[zackw]

Half-baked idea: The builds would be much cheaper, and therefore we would have more options, if we didn't have to build qtwebkit on every single iteration. I wonder how difficult it would be to arrange to build qtbase and qtwebkit separately from the core pjs code, then pull in previously compiled libraries for changes that don't touch them.

[ariya]

@zackw I think eventually we have to migrate to the modular approach in the near future. Yet, hopefully there is still a suitable solution for the current situation.

[ariya]

Can't you use Snap CI?

From a quick attempt, Snap CI build setup is limited to 2 GB RAM. Eventually, the build fails.

Beside, Snap CI provides only Linux build.

[ariya]

Currently I'm exploring Semaphore (https://semaphoreci.com/) and it seems to be promising.

Its build machine is, as promised, quite mighty: lots of RAM and CPU power. I'm sure it's still somehow virtualized, yet lscpu reports 8 available cores. Running the full PhantomJS build takes about 15 minutes, pretty impressive for a hosted CI system.

Since the host system is Linux 64-bit only, this is hardly useful to run the full build of all PhantomJS supported platforms. However, it should be sufficient as a mechanism to verify a pull request (build + run tests), consider its fast build time.

Although the wordings are a bit ambiguous, https://semaphoreci.com/blog/2014/08/14/semaphore-gets-free.html mentions that FOSS project gets only 100 builds per month. I think this should be sufficient given the current rate of our pull request.

[strika]

@ariya To address the ambiguity, Semaphore offers up to 100 free builds per month for private projects and unlimited builds for open source projects.

I'm a Semaphore engineer so feel free to get in touch if you need any help. PhantomJS is very popular on Semaphore so we would be more than happy to help. Cheers.

[ariya]

Thanks for the clarifications @strika! I'll keep everyone posted with my experiment and perhaps I will reach out if I need some more help.

[Vanuan]

@ariya how did you achieve 15 minutes? Mine is killed after 1 hour: https://semaphoreci.com/vanuan/phantomjs/branches/master/builds/3

[Vanuan]

openssl + icu build takes 5 minutes without Docker.

[Vanuan]

Ok, build succeeds, but it picks up shared libraries for some reason. Probably, pkg-config returns shared library options.

[ariya]

@Vanuan What is your build machine setup? Unless it is equipped with an 8-core CPU, the build will take forever. Add a build step to run lscpu and compare with what I have:

CPU Information:
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 60
Stepping:              3
CPU MHz:               3115.250
BogoMIPS:              6799.63
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
NUMA node0 CPU(s):     0-7

[Vanuan]

What is your build machine setup?

I used Ubuntu 14.04 LTS v1512 (beta with Docker support) as a platform in settings
Probably, with Docker it's slower

[MikeMcQuaid]

@ariya You have some vaguely similar needs to Homebrew here (we also have automatic phantomjs builds). The only real option we've found for long-running builds is having dedicated bare metal machines (or beefy VMs) donated by someone or, in our case, bought through a Kickstarter. Feel free to mail me directly if you'd like to talk any more about this.

[Vanuan]

(we also have automatic phantomjs builds)

Did you mean brew builds?

[MikeMcQuaid]

Did you mean brew builds?

Yes. It involves building PhantomJS from source across three OS X versions so is comparable to doing the same on Windows/Mac/Linux, I think.

[Vanuan]

Ah, got it. So you prepare phantomjs brew packages by building from source.
Sadly, nobody's preparing deb and rpm packages for different Linux distributions.

[MikeMcQuaid]

Ah, got it. So you prepare phantomjs brew packages by building from source.
Sadly, nobody's preparing deb and rpm packages for different Linux distributions.

@Vanuan The OpenSUSE Build Service is a bit unwieldy but, in my past life doing cross-platform Qt development, was the tool I found best for this.

I'm sure this is well known but the main source of your troubles here is the Qt build; if you can use an unpatched Qt version and link against it dynamically then the distros will do the legwork for you and Travis CI and friends suddenly become a lot more useful. That may end up being less work than managing your own CI/build infrastructure.

[ariya]

The only real option we've found for long-running builds...

I remember reading an article on how this was setup for Homebrew. I think the difference here is that Homebrew builds thousands of packages, hence the effort is amortized over time. In our case, it would be ideal if we don't spend too much time for the setup and the maintenance.

[ariya]

I'm sure this is well known but the main source of your troubles here is the Qt build...

Yes, this is well known. I explained how we end up going that path already: https://groups.google.com/forum/#!topic/phantomjs/r0hPOmnCUpc.

Using upstream Qt warrants a separate discussion, but frankly looking at the current state of affair with Qt 5.x and its QtWebKit module, I doubt that it's going to be promising.

[MikeMcQuaid]

I remember reading an article on how this was setup for Homebrew. I think the difference here is that Homebrew builds thousands of packages, hence the effort is amortized over time. In our case, it would be ideal if we don't spend too much time for the setup and the maintenance.

Agreed; even for Homebrew it's time that takes away from development and I'd rather we didn't spend. Unfortunately though from my personal research/experience given the requirements you have it's pretty much the only option. Your best bet might be trying to convince sponsors to run those machines for you as part of their infrastructure.

Yes, this is well known. I explained how we end up going that path already: https://groups.google.com/forum/#!topic/phantomjs/r0hPOmnCUpc.

Using upstream Qt warrants a separate discussion, but frankly looking at the current state of affair with Qt 5.x and its QtWebKit module, I doubt that it's going to be promising.

Sure, I understand the logic behind it. I think it's related to this discussion because a dynamically linked Qt suddenly solves a lot of your CI problems. That's just my opinion, though, so I'll drop it for now.

[jakoch]

You could also setup Buildbot on a custom server, add some docker slaves for building and reuse some of the scripts from Chromium/Webkit/Mozilla to save time.

[ervinb]

Hi! I'm working on the Semaphore platform. The current plan is to double the number of cores available in the Docker enabled platform, starting from the next platform update, scheduled for Tuesday (19.01.2016). I hope this will have a positive effect on your project. Let me know if there's anything else I can help you with.

[Vanuan]

@ervinb Sounds awesome! I hope it will be enough to have a build in less than an hour.

[vassilevsky]

Semaphore is a very good service.

Mozilla has published an interesting description of their build farm:

http://taras.glek.net/blog/2014/05/09/how-amazon-ec2-got-15x-cheaper-in-6-months/

They say that it costs them $0.035 / hour.

Now of course this is not free. It will be necessary to raise funding for the farm.

It seems that this can be done pretty well on https://salt.bountysource.com

[ariya]

Hi @ervinb, thanks for chiming in!

Just to clarify: currently we can't use Semaphore for this purpose unless it allows building for OS X and Windows as well. However, we plan to use Semaphore for a subset of the task, i.e. building and testing a feature branch in a pull request, see #13850 for the details. For this particular task, we don't need Docker support as we can run in the bare system just fine.

[ariya]

@jakoch @vassilevsky Thanks for the suggestions. But keep in mind that the whole point of this investigation is to find out a ready-to-use system that we can use right away. Yes, we can setup Bamboo/Jenkins/TeamCity/what-have-you ourselves, but that is the main goal here. See again four challenges I've mentioned earlier.

[ariya]

Your best bet might be trying to convince sponsors to run those machines for you as part of their infrastructure.

In fact, I won't object to that idea! Even if it costs money, it is better to spend that money to fund an entity that is specialized in the field rather than us spending our time/effort doing tasks not directly related to the development.

[Vanuan]

For this particular task, we don't need Docker support as we can run in the bare system just fine.

But we need it to provide pre-release static binaries
#13848

[ariya]

But we need it to provide pre-release static binaries

Unless every binary is available for all platforms, I won't support that idea. Otherwise, we'll get another mob attack just like the current situation.

[ariya]

We'll settle with CircleCi for the minimum build for now. Once we revive the development (see #15344), we'll revisit more CI systems for macOS and Windows. Thank you!