Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CI infrastructure design #302

Closed
layus opened this issue Aug 17, 2018 · 10 comments
Closed

CI infrastructure design #302

layus opened this issue Aug 17, 2018 · 10 comments

Comments

@layus
Copy link
Member

layus commented Aug 17, 2018

Dear mozart2 enthusiasts,

@azarzadavila and I are trying very hard to get CI rolling for mozart2. The point is to avoid the current situation where mozart2 failed to get new releases because we do not have the right machines and lost the knowledge of how to do so.
At this point, we are again able to build mozart2 on Windows (via appveyor), and on Linux (with Travis or locally). We dismissed Darwin for now.

Using llvm-4.0.1, packaged at apt.llvm.org, we are even able to compile and run the generator, but only on Travis.
These generated sources are still a pain to obtain. Building LLVM/Clang is not an option on appveyor, where the build already approaches the 1h limit.
For the appveyor build, we added the generated sources to git in our own repos.

To make things simple, I propose the following idea, partially discussed in #266:

Add generated sources to git

We should add the generated sources in the git repo. If we do that, we could easily setup a Travis task to check that they are always up-to-date with the source code, and avoid de-syncs.

Pros

  • We can build and release mozart2 on freely available CI platforms
  • Mozart2 gets much easier to build for downstream packagers as "repo checkout == source archive"

Cons

  • We add generated stuff to the source repo
  • The build gets more confusing for developers, as manual actions will be needed to regenerate the sources.
  • Artefacts built on Travis may depend on outdated package versions (to be confirmed).

As I spent too many hours trying to get the generator working on too many different platforms, I must say that I am much in favour of this solution. However, here is an alternative for the sake of comparison:

Use a ad-hoc CI pipeline

With Jenkins (and possibly other tools) we are theoretically able to configure a custom pipeline. The first stage (on Linux) would be to release a pre-generated source archive containing the generated sources. Afterwards, one build is started for each of the platforms we want to support (Linux (deb/rpm), Darwin, Windows).

We are currently discussing with CloudBees to get access to their Jenkins infra. We could also use the Jenkins available here at UCL, but we need to add workers for windows and mac, which is not going to happen anytime soon.

Your reactions and comments are much awaited :-).

/cc @sjrd @sjmackenzie @eregon

@sjrd
Copy link
Member

sjrd commented Aug 17, 2018

IIRC, generated sources were somewhat machine-dependent. I'm not sure whether we're still trying to build 32-bit versions, but for such builds, the generated sources are different. If that's still the case, putting the generated sources in the repo is just plain wrong, as they will only be valid for some build environments.

Otherwise, and given the harsh constraints, I think your idea is fine, as long as this:

we could easily setup a Travis task to check that they are always up-to-date with the source code, and avoid de-syncs.

is actually implemented.

@eregon
Copy link
Member

eregon commented Aug 17, 2018

As before, I'm all for adding the generated sources in the repo.
Checking they are the same in a CI job is indeed important to make sure they don't go out of sync.

The build gets more confusing for developers, as manual actions will be needed to regenerate the sources.

But OTOH devs don't need to spend hours building LLVM.

@layus
Copy link
Member Author

layus commented Aug 17, 2018

But OTOH devs don't need to spend hours building LLVM.

Indeed, as long as they do not change anything that requires an update to the generated files.

@eregon
Copy link
Member

eregon commented Aug 17, 2018

Indeed, as long as they do not change anything that requires an update to the generated files.

The CI job checking that could handily show the expected file contents :)

@sjmackenzie
Copy link
Member

i'm more than happy to include mozart2 on my nix CI system hydra.fractalide.com , you can tie this up with travis so travis just pulls the latest artefacts from the nix store. Travis build times are typically reduced to a few minutes, this makes it very quick and easy to iterate on the mozart2 code. We've already got the code demonstrating how to tie it all together. https://github.com/fractalide/fractalide/blob/master/support/utils/nix-build-travis-fold.sh

@layus
Copy link
Member Author

layus commented Aug 29, 2018

@sjmackenzie Let's discuss that at NixCon (if you are attending of course).

@layus
Copy link
Member Author

layus commented Aug 29, 2018

Closing, as there is a working CI for windows (#310) and linux (https://github.com/mozart/mozart2/blob/master/.travis.yml) with gensources checking (#309) as discussed here.

@layus layus closed this as completed Aug 29, 2018
@sjmackenzie
Copy link
Member

@layus i'll probably be at FOSDEM 2019 iirc nixcon is about the same time. Might be there :-)

@layus
Copy link
Member Author

layus commented Aug 29, 2018

We might meet at fosdem (Brussels, january 2019), but nixcon is in october 2018, in London https://nixcon2018.org/

@sjmackenzie
Copy link
Member

okay great hope to see you at FOSDEM then!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants