CI Infrastructure #100
Comments
ghost
mentioned this issue
daviddias
changed the title
|
In today's sprint hangout, CI was brought up as one of the main points that are blocking our development teams from moving faster. I was going to open a issue in ipfs/community, but since this one already existed, let's continue here. Current issues:
Requirements:
Possible solutions:
@chriscool you have an incredible amount of experience testing different projects with very eccentric needs, if you feel like Captaining this endeavour and optimising our CI infrastructure to enable our dev team to move forward faster, either by leading or building, that would be amazing! @yoshuawuyts, I know you are also really excited with CI and have been doing a lot of work with BuildKite, your thoughts on this are greatly appreciated. Let's gather all ideas before Wednesday, August 17, so that we can have a call and commit to a plan to move forward. Does 4:30pm UTC work for everyone? |
|
|
Can I be on this call? |
|
Hey @daviddias, thanks for pinging me. I won't be able to make the call (yay, You're right in saying I'm quite a fan of buildkite. I've been using it for the
But there's one major drawback to all this: they don't have public build So I've reached out to the @buildkite people today, asking them what's up with Another option @toolmantim suggested was creating a custom frontend for builds. ┌───────────────────────────┐ ┌────────────┐ ┌─────────────────────────┐
│ github.com/ipfs/project │ │ Buildkite │ │ https://builds.ipfs.io │
└───────────────────────────┘ └────────────┘ └─────────────────────────┘
│ │ │
│ │ │
├───────────PR opened / ─────────▶ │
│ fork push │ │
│ webhook │ │
│ └───────────Events via───────────▶
│ webhooks │
│ │
│ │
│ POST │
◀─────────────────────────────commit ─────────────────────────────┤
│ status │
│ │
│ │
│ │
│ Contributor can │
└──────────────────────click through or go ───────────────────────▶
directly to
Buildkite handles jobs dispatching, running agents, streaming logs,
capturing artifacts etc.
builds.ipfs.io is a Node app that:
- Receives Buildkite webhooks, posts commit status to GitHub
- Display builds using Buildkite REST or GraphQL API (can use artifacts and
meta-data to instrument/customize the build results)
- Serves up dynamic build badges based on Buildkite build info Some of the things that were also discussed were full-public orgs; where every Ueh, and I think that's about it. I think if you holler at @toolmantim for Hope you find a good solution. Cheers! ✌️ See Also |
|
👋 I just had a chat with @yoshuawuyts about it all. Just to clarify, nobody does the auto-invite thing just yet. But we're free for open-source, we just have to hit a button for you, and we're happy to support IPFS if you wanted to go w/ Buildkite. I'd recommend two completely separate Buildkite orgs in your case… one for third-party PRs, and another for core/internal use. |
|
Hi @toolmantim, I am evaluating a CI system for (This is a little separate from the whole IPFS team goals, so take my questions with a grain of salt 😃 ) |
|
@hermanjunge yeah it does. Have been using it for the past few days on google super tiny (micro?) nodes; and a few odd restarts aside (yay, low ram) they seem to be holding up quite well. See Also✌️ |
|
@hermanjunge yeah, the build agent runs on Docker and Kubernetes just fine, we just don't have the equivalent to buildkite/elastic-ci-stack-for-aws for Kubernetes just yet. But we have buildkite/helm-charts. And then for the actual build tasks themselves, they can run in Docker (for stable agent see the guide or for beta agents we've moved it into a plugin: buildkite-plugins/docker-compose-buildkite-plugin). Might want to move this to another issue to talk more tho? |
|
@toolmantim Amazing! I have some tests scenarios that involve building docker images, and deploying into kubernetes before running the tests. Will sign up, learn and provide feedback of Buildkite. Thank you very much for answering. |
|
@toolmantim thanks for stopping by, buildkite looks really nice :) It looks like with buildkite we still have to manually set up each repo we want CI for (same with teamcity). Are there any nice automations for setting up a 'stock CI setup for environment X'? Similar to how easy it is to get travis CI running? teamcity provides really nice metrics such as build time per step, total tests run, number of tests passed and ignored. The latter few are because teamcity has plugins to parse and understand test output, does buildkite have anything similar? Following along the previous point, We currently fail our tests if the number of tests executed drops significantly from the previous successful run. This is because teamcity allows us to define custom failure conditions based on both preset and user defined parameters. I think this is super cool and would hope that we have similar capabilities in any system we switch to. It looks like buildkite supports windows, does it run scripts in cmd or git bash? (or can we specify?) I notice in some of the documentation (that I admittedly havent read too far into) that there are buildkite hook files that get committed into vcs. It seems odd to have configuration both in the repo and on the buildkite admin page. Whats the purpose of that? Anyways, I look forward to trying it out. I made an account and created an ipfs org, but haven't yet had time to make it any farther. Thanks! |
|
@diasdavid yeah I have some experience especially with Jenkins but it is not part of the possible solutions. |
|
It seems to me that we have some solid action items before jumping onto a call, these are:
Let's postpone the call until we have these items done, otherwise we might not be able to decide anything more than what we've discussed on this thread. It would be also good to have everyone when it happens. |
@diasdavid I can commit to try buildkite, and do a quick video with a flow. EDIT: I was pestering everybody for the when and where of the call... |
|
@hermanjunge If you want, i can invite you to the ipfs buildkite org and you can try setting something up there. It might make it easier for the rest of us to see |
|
@whyrusleeping Excellent. Do you have my email? |
|
@hermanjunge yeap! sent out an invite |
|
About GitLab CI it looks like the best is to setup Repository Mirroring on GitLab: http://docs.gitlab.com/ee/workflow/repository_mirroring.html, as GitLab now integrates GitLab CI. |
|
Can gitlab CI work as a github CI indicator? Could ease the transition
|
|
(Of their potential users) On Mon, Aug 22, 2016 at 23:35 Juan Benet juan2@benet.ai wrote:
|
|
@jbenet I think so. By the way you may want to read dak1 comments on this HN thread about yesterday's GitLab release: https://news.ycombinator.com/item?id=12338096 |
|
Hi @toolmantim Our buildkite trial just expired and haven't had the chance to build a proper test yet. Can we get an extension trial, so I can describe my experience here ??? 😬 Here is the email
(cc @whyrusleeping @yoshuawuyts ) |
|
@hermanjunge all fixed, I just extended it for a month. If this Buildkite org is for all third-party and public builds, I'll switch it to a free/open-source account if you like. But I'd suggest a separate org for private/sensitive/secure build pipelines, for doing all the releases etc, with a completely separate set of build agents (in their own VPC etc). |
|
Stumbled upon this article about popular self-hosted CI exploits; figured it might be relevant to the topics discussed here |
|
Hey guys. The buildkite team linked me to this issue. If you would like to take a look I just implemented buildkite for an open source project I work on in my free time. See this PR: rubocop/rubocop-rspec#205 |
|
@backus thats awesome :) Any idea how you would go about doing tests on windows and/or OSX? |
|
@whyrusleeping that I can't comment on if you need the OS inside the docker container to be windows or OS X. |
@toolmantim can activate the IPFS organization as an free/open-source account, then we get public builds. Then, if we start having private builds, we have a different BuildKite organization, we just need to find a way of sharing the pipelines between them, possible via setting up a repo for composing pipelines together. The BuildKite IPFS Organization lives under https://buildkite.com/ipfs quote from above
|
|
@victorbjelkholm thanks-- missed that. yeah i'd say lets make it public |
|
I sent an message to the BuildKite folks about activating public builds but seems like I misunderstood the relationship between public builds and a free/open source account. So they have a free plan for open source projects, but there is no public builds. Asked them about a timeline for public builds but they don't have any ideas about the timeline for it... The quest goes on... |
|
I am working on using GitLab CI to test Sharness on both Linux and Windows. https://gitlab.com/chriscool/sharness/ is a mirror of https://github.com/chriscool/sharness/ that is automatically refreshed. I installed a "GitLab CI Runner" on my Windows machine and I am using some shared Runners provided by http://gitlab.com to run the tests on Linux. Currently there is something wrong with the way my Windows Runner is configured so the tests on Windows fail immediately. See: https://gitlab.com/chriscool/sharness/pipelines/4356014 But hopefully it will work soon. |
|
@chriscool are you setting up the runner with privilege so you can do DIND (docker-in-docker) or mounting the .sock for the docker api? Seems like there is something missing with the api connection |
|
Yeah, I think I shouldn't use docker on Windows. I will try another configuration for the runner. |
|
This might be intersting to look at as well https://medium.com/@hichaelmart/lambci-4c3e29d6599b#.3ydwhefgk |
|
Some GitLab related links:
From the main GCE page it looks like Google Compute Engine (GCE) has support for "Debian, CentOS, CoreOS, SUSE, Ubuntu, Red Hat, FreeBSD, or Windows 2008 R2 and 2012 R2" and "shared image from the Cloud Platform community", or our own images. |
|
So I've sat down, looked over multiple different solutions and tried them out. Problems needing solution
All these are aggregated with searching through discussions on Github in IPFS Review of systemsFirst part of my review was to go through a list of the CI systems I could find Second part of my review was to actually try the CI's out, and here is what I found so far ConcourseSelf-hosted. Does most things we want to do, is blazing fast and very easy to setup and maintain. BuildKiteRun our own agents, perfect. UI that makes sense. Configuration of jobs that is reusable TeamCitySelf-Hosted. Giant in CI, configurable until you can't recongnize it anymore. Does pretty much JenkinsSelf-Hosted. Second giant in CI. Also very configurable, but the more you configure, the GoCDSelf-Hosted. Focus on pipelines from the ground up. Simple UI. Not very extensible, small CircleCIHosted service. Allows us to easily setup simple jobs. Not very good for bigger TravisCIHosted service. We're currently using Travis, and having troubles with waiting Gitlab-CIGitlabs solution for doing CI. Works well, when inside the Gitlab ecosystem. ConclusionIn the end, we end up with two different viable options. TeamCity and Jenkins. Pitting them against each other, they fit all the needed features, either natively TeamCity
Jenkins
In the end, they are mostly the same, but after going back and fourth, trying both I'm pretty confident Jenkins in the end would fit us better than TeamCity. Main I would like to give a larger try to Jenkins to review the new changes in Jenkins |
|
re CircleCI: re TeamCity: |
|
re GitLab CI:
Also what do you mean by "Status Matrix" and by "Metrics"? I am wondering how you evaluated it. |
|
@chriscool I made the evaluation about Gitlab-CI with the assumption that we still want to host our projects on Github and not migrate them to Gitlab. "Configuration in SCM" refers to the setup of the server, which with Gitlab.com we don't control that. If we setup our own Gitlab instance, that seems possible, but not sure we want to host Gitlab just to get the CI.
Sure, that is true that we could. However, we can't host just the CI, was my thinking regarding that.
That seems to be a list of the current builds and pipelines, which is good but is missing an overview of the status of the pipelines/jobs, aggregated data basically, to be able to get a quick glance if the project "healthy" or not.
Status Matrix would mean if we can build a table with current build status for different builds on different platforms. Metrics would be some sort of analytics. Gitlab includes "Cycle Analytics" which seems limited in the way that you need to use Gitlab to have use of that. In general, those features are not deal breakers, but rather the lack of most of the other things. However, if I missed something about Gitlab-CI or misunderstand, it would be great if you clarified it. |
|
@victorbjelkholm about GitLab CI you also say above "Can't find any ways of doing on-demand builds About "Self Hosted", even if we would have to host GitLab, if we want GitLab CI, it seems more fair to say "Yes" than "No" (or at least something like "Yes-"). In general as with GitLab you can easily and automatically mirror any repo on GitHub or elsewhere, I don't think it is a big problem to have to use GitLab, if we want GitLab CI. |
Yeah, I should probably have clarified the columns before sharing the spreadsheet. I'll add some definitions. Gitlab CI is marked as can do on-demand builds but no parameterized ones.
Sure, Gitlab we is self-hosted, my point was to evaluate the CI only. But yeah, you're right, and changed that.
Not sure it's a great idea to replicate all repositories to Gitlab (unless we want to commit to the full Gitlab ecosystem) just to be able to use the CI, which in general doesn't fit us anyways. At least that's the idea I got from trying to get into it and trying it out. |
|
@ipfs/go-ipfs-team @ipfs/javascript-team Some things that would be good to hear from you
|
|
I don't think there is anything we could not do with Jenkins. My only thought right now is that I want us to stabilize on something. |
|
@victorbjelkholm one thing we were having issues with is having Chrome/Firefox using xvfb running on TeamCity. I would like to make sure that is addressed in whatever system we are moving to. |
|
There is no way around it no matter which runner we use I think.I f it is still a problem I can promise to take a look at it. |
|
For future reference: https://wiki.jenkins-ci.org/display/JENKINS/TAP+Plugin |
|
@victorbjelkholm I think triggers can be used to do parametrized builds in GitLab: https://docs.gitlab.com/ce/ci/triggers/ especially: https://docs.gitlab.com/ce/ci/triggers/#pass-build-variables-to-a-trigger See also: |
|
We've had a nicely automated Jenkins instance for a while now at https://ci.ipfs.team -- the code lives at https://github.com/ipfs/jenkins |
As we investigate golang/build (see issue #99) we might want to figure out what would make us super happy both in the short and long term.
The basic goals are:
And two constraints are:
Related issues:
The text was updated successfully, but these errors were encountered: