Buildbot end of life date is March 1, 2019. As of March 1, all Chrome/Chromium builds hosted on chromium.org will be running on LUCI.
LUCI which stands for Layered Universal Continuous Integration is a replacement for Buildbot (our existing single-threaded monolithic continuous integration system). Over time, we have been swapping out Buildbot responsibilities with LUCI services. As such, most builds today on Buildbot are already running using LUCI services. The next step is to migrate Buildbot builders to LUCI builders, with the end result to turndown Buildbot.
As a Chromium developer, read this to understand the CQ and CI user expectations and how to contact us.
[TOC]
As of March 1, 2019, all Buildbot masters run by Chrome Operations will be permanently shut down. All existing builders will have been migrated to LUCI before this date.
A migration guide is available to help you with migrations. Please reach out to the Chrome Operations team at infra-dev@chromium.org for references and scheduling or timeframe info.
The existing Buildbot builders are not running on the LUCI stack, as such, they are still prone to Buildbot limitations such as single process scheduling and deployment issues (restarts needed) to name a few. We have been in preparation to migrate builders to LUCI with as minimal impact as possible to Chromium devs.
We have started migration of existing Buildbot builders to LUCI builders in Q1 2018 and will first focus on the Chromium CQ builder sets. A Chromium CQ builder set consists of try-builders that are part of the commit queue and their corresponding waterfall (continuous integration) ci-builders. A builder set will be migrated together whenever possible. Target migration time per builder set is approx. 1 week per set with a few days of overlap with existing builders to ensure correctness and performance. Note that times can vary per builder set.
We have created a LUCI migration task force (infra-dev@chromium.org) comprised of Chrome Operations developers. Owners of Chromium builders are responsible for approvals for final go ahead to switch a LUCI builder to production.
A tour of LUCI is available in presentation format. Also, you can learn more about the LUCI UI under Chromium source.
No, there should be minimal to no disruption. Our migration plan is to make it as seamless as possible for Chromium devs to continue their work and use our CQ and CI systems effectively.
While we're working to make the transition as seamless as possible, there can be a few hiccups that may occur when a builder is migrated. Below are the known limitations:
- The blamelist will not be calculated correctly between the first LUCI build and the last Buildbot build. The second and subsequent LUCI builds will have correct blamelists, however.
All Chromium developers should already be using the new LUCI UI (ci.chromium.org). The views will continue to look the same once LUCI builders are switched to production. Sheriff-O-Matic, Gatekeeper, and Findit build integrations will all work as intended. The main differences to expect are:
- CQ try builders will show a LUCI tag in the Code Review UI. See below for details.
- LUCI builder pages will have a different URL path. See UI tour for details.
- LUCI build pages will have a different URL path. See UI tour for details.
- When the switch is made, build numbers will be incremented by approximately +10 to separate Buildbot and LUCI builds.
- Blamelist limitation between first LUCI and last Buildbot builds. See above for details.
Builds on LUCI use a different URL path that
starts with /p/chromium/builders/…
If the build is still running on Buildbot,
"buildbot" will still be part of the URL path. Also, in
the Code Review UI, try-builders on Buildbot is shown
with a "Buildbot" tag. The default with no tag represents
a LUCI build.
In some cases, builds on LUCI can also have the
following URL
ci.chromium.org/p/chromium/builds/b<buildbucket_build_id>.
This occurs when the build does not have a build number
which should not occur for any Chromium build
Buildbot builders will be turned down progressively and won't be turned down all at once. Once a CQ builder set has completed migration, we will schedule a date within the following month to turn down the corresponding Buildbot builders. This date will be coordinated between the development teams and Chrome Operations.
Contact the Chrome Operations Foundation team (infra-dev@chromium.org) and include dpranke@chromium.org, estaab@chromium.org, efoo@chromium.org and nodir@chromium.org directly for an immediate response. Also, refer to the Contact us section below for other options to reach us.
Builder migration involves 3 major steps.
- Preparation (machine allocation/configs).
- Making builds work on LUCI (recipes).
- Flipping to LUCI builder on prod (UI).
There are many ways you can help. Either through contributing to LUCI, helping with builder migrations or providing feedback. Reach out to the Chrome Operations Foundation team (infra-dev@chromium.org) and CC efoo@chromium.org and estaab@chromium.org to let us know.
We are continually making improvements to the migration process and to LUCI services. Our initial goal is to migrate all Chromium CQ builders sets first as seamless as possible for Chromium devs to continue to work unimpacted and use our CQ and CI systems effectively.
If you have issues or feedback you would like to share with us, we would love to hear it and incorporate it into our ongoing LUCI improvements.
- Use the feedback button
on a LUCI page. - File a migration bug using the following template.
- File a feature request using the following template.
- To share your feedback, please fill out this short survey.
- Ask a question using IRC under the #chromium channel.
- Contact us directly by emailing us at infra-dev@chromium.org.
