Build Stages Part 2: Stages Order and Conditions

[svenfuchs]

We have shipped iteration 2 for the Build Stages feature.

Along with a lot of bug fixes and small improvements this includes:

• Build Stages section: Define the order of stages, optionally with a condition
• Conditional builds, stages, jobs: Define conditions for accepting/rejecting a given build, stage, or job

Find out more about this on our blog, and documentation here and here.

We would love to hear your feedback. Please leave all thoughts, comments, ideas related to this feature here.

Happy Testing!

FAQ

What feature requests have you received for improving Build Stages so far?

We are adding this list of feature requests (with a rough indication of how likely we are going to prioritize it) so you don't have to ask about these again :)

• Pausing a build after a given stage, and proceeding only after an interactive confirmation. (Yes)
• Add a native build artifacts feature that can be used for sharing artifacts across stages, or another way to more easily share storage or artifacts. (Yes)
• Making the native cache feature more configurable by specifying cache keys per job. This would allow you to reuse caches in more flexible ways in subsequent stages. (Probably yes)
• Allow script etc. on the new stages section, so they can be shared across all jobs in one stage. (Yes, but not before [1])
• Allow specifying allow_failure: true per job on jobs.include. (Probably yes, but not before [1].)
• Allow a stage key on the deploy section. Turn this into a stage and a job. (Not quite sure, not before [1].)
• Add a name/description to jobs, in order to reveal the intend, show them in the UI. (Not quite sure.)
• Consider skip: all, so one does not have to overwrite before_install, install, and script. (Sounds like a good idea? Not before [1])
• Silence skipped commands: no log output. (Not quite sure.)
• Consider "embedded matrix expansion", e.g. jobs.include.env: [FOO=foo, BAR=bar]. (Not quite sure)
• Build config .travis.yml editor/web tool. (Yes, based on the specification produced in [1])
• Allow grouping jobs into arbitrary groups, not depending on each other, like stages, just for visual presentation on the UI. (Probably not)
• Update the GitHub commit status per stage, add with more detailed commit status information. (Unsure, definitely not before [2])
• Consider automatically restarting jobs in later stages that have been canceled because a job on a previous stage failed when this job is restarted. (So far uncertain. There's a lot of complexity involved.)
• In the running tab, consider grouping jobs per build (and possibly stage). (Interesting thought. We are working on improving this UI, and might consider this in a later iteration.)

Other improvements:

• [1] A strict travis-yml parser has been shipped.
• [2] The GitHub commit status API has the known limitation that new updates are being rejected after the 1000th update. They are working on improving this, and providing a way for us to post more updates. Until this is unblocked we are unlikely to make any changes to our commit status updates.

[ljharb]

Heads up, the comment on #11 points to itself instead of to here.

[23Skidoo]

Is there any chance you could implement a more fine-grained job dependency graph feature than just "wait until the previous stage is done"? E.g. in my case I have a bunch of different compiler/OS configurations, and "populate cache" and "build" stages that each have a bunch of jobs corresponding to every configuration. In principle, a "build" job could start as soon as the corresponding "populate cache" job for that configuration was done, but right now the "build" stage doesn't begin until all of the jobs in the "populate cache" stage have finished.

See haskell/cabal#4556 for more details.

[szpak]

Is it possible to skip cache update for a particular stage?

I have a acceptance test job which should leverage cache from the previous stage. However, I don't care about the new artifacts it creates/installed (which would be cached in default configuration) and I don't want to have cache updated at the end of that particular stage (while cache still should be fetched at the beginning of that job to have build faster).

[ujovlado]

Looks like there's a small bug. If I have multiple stages (e.g. 4) and I add conditions to run only 1 stage then the main job will stuck in "created" state.

[ujovlado]

Looks like there's a small bug. If I have multiple stages (e.g. 4) and I add conditions to run only 1 stage then the main job will stuck in "created" state.

Hm, that's also true for multiple jobs (e.g. if conditions remove 2 jobs and 2 remain).

[BanzaiMan]

@ujovlado I think that is travis-ci/travis-ci#8415. We will look into it shortly.

[ujovlado]

@BanzaiMan thanks!

[BanzaiMan]

@ujovlado No worries. That is one long standing source of confusion, which is tracked in travis-ci/travis-ci#1066.

[webknjaz]

• [1] A strict travis-yml parser has been shipped.

Does this mean we'd have CLI tool for config validation? Maybe embed it into existing travis CLI utility?

[BanzaiMan]

Somewhat related to:

Allow specifying allow_failure: true per job on jobs.include. (Probably yes, but not before [1].)

Feature: Set the build status when all remaining jobs and stages are allowed to fail. travis-ci/travis-ci#8425 also asks this in the case where the last stage consists of jobs which are allowed to fail.

[webknjaz]

Guys, plz take a look at this build: https://travis-ci.org/aio-libs/multidict/jobs/275530708#L848

There must be deploy step in this job, but it's missing according to logs. I'm going to guess that some of your changes might've affected build steps resolution.

Any ideas? Should I file a separate issue?

[abbeycode]

One thing that's frustrating when working on a build with stages, compared to travis-after-all, is that it seems like it takes much longer for a stage to start after the previous stage completes. I'm not sure if this is what's happening, but it's like the next stage's jobs get tossed onto the end of the queue. It would be nice if subsequent stages could run more quickly after the earlier matrixed jobs have already started. I can clarify if that isn't clear.

[BanzaiMan]

@abbeycode Each job is independently queued (in other words, only when the execution gets to that job, is it inserted into the queue). This is due to how the queue is processed; if it is in the queue, it is assumed to be runnable. If we put the last job on the queue, it may be ready to run before it is supposed to.

[BanzaiMan]

@webknjaz We are not just "guys", so please refrain from using that. Thanks. :-)

As for your deployment not triggering, it is reported as travis-ci/travis-ci#8337. The current workaround is to not skip, but use an actual command (true, or echo skip, or whatever).

[cristopher-rodrigues]

Hey guys, it is being really nice to test the stage features. I would like to suggest running the stages concurrently, it would be fantastic!

[BanzaiMan]

@cristopher-rodrigues We are not just "guys", so please refrain from using that. Thanks. :-)

I would like to suggest running the stages concurrently, it would be fantastic!

Could you elaborate on this? If the jobs in these stages can be run concurrently, they can be in the same stage and not lose any meaning, can't they? Or did I miss some broader context?

[hmottestad]

Would be nice with a label option when running multiple stages with the same name. As a workaround we have put a comment in an environment variable (which is displayed)

[BanzaiMan]

@hmottestad This request is listed in @svenfuchs's list above.

[cristopher-rodrigues]

@BanzaiMan Sorry for that, my bad.

Look for some use case:

• I execute unitary tests and e2e in a different stages for example, they can be executed in parallel to speed things up a little bit.

• I run different journeys of e2e tests and they do not depend on each other. My E2e tests are complex and time consuming, if we would be able to execute asynchronously, it would br way more faster.

Something like that:

...

jobs:
  include:
    - stage: Tests Unit
      async: true
      script: ...
    - stage: Tests E2E Journey Foo
      async: true
      script: ...
    - stage: Tests E2E Journey Bar
      script: ...
      async: true

[hawkrives]

@cristopher-rodrigues If you give those three the same stage, they'll execute in parallel, won't they? Do they need to be in their own stages?

[jayvdb]

Following on from my little experiment with sentinel jobs (travis-ci/travis-ci#2062 (comment)), I am wanting to have fast_finish: true in the stage definition, so that I can have it be conditional, such as

stages:
  - name: test
    fast_finish: true
    if: branch != master OR type = pull_request
  - name: test
    fast_finish: false
    if: branch = master AND type = push

I've tried with jobs: fast_finish: true and matrix: fast_finish: true. None appear to work, and I wouldnt expect them to work without this being an intentional feature.

[mattrayner]

I'm not sure if this has been covered, but I keep getting emails saying by build has errored, regardless of wether there is an error, the build passes, or the build fails. Sometimes I get these emails up to half an hour after a build has finished.

[atodorov]

I am having troubles getting allow_failures to work, see:
https://travis-ci.org/kiwitcms/Kiwi/builds/394767964

The pylint job is allowed to fail. Any hints ?

[Seluj78]

Hey there ! Amazing feature !

Why is the lint stage running only on python2.7?

Here's my travis file:
https://github.com/OpenApprenticeFoundation/openapprentice-blog/blob/779ac3b416ba6d0e43529ed1d2af3169e05e98a8/.travis.yml#L1-L48

Oh and one of my jobs failed weirdly:
Please override the script: key in your .travis.yml to run tests.

The build:
https://travis-ci.org/OpenApprenticeFoundation/openapprentice-blog/builds/395339275

[zaypen]

Awesome work!
But really complicated yaml schema you guys providing. Spent plenty of time getting my stages work.
Please make it simple and easy-use.
Take something good from Buildkite.

[lfre]

Hey there! My suggestion is along the lines of Allow script etc. in the new stages section, so they can be shared across all jobs in one stage. I know you can do this by declaring it at the top level and every stage will inherit it, but if you override it at the stage level i.e before_script, it overrides the whole thing. Right now I'm solving it by grouping the stage steps in a different step script, but it'd be great if there was an option for each step to indicate addition, so global steps can run before/after stage steps. For example:

before_script:
  - echo "global"

jobs:
  include:
     - stage: before
       before_script:
          - merge: "before"
          - script:
            - echo "before";
     - stage: after
       before_script:
          - merge: "after"
          - script:
            - echo "after";
     - stage: override
       before_script:
          - merge: "none" # default
          - script:
            - echo "override";

This would execute as:

stage: before

• before
• global

stage: after

• global
• after

stage: override

• override

This could also be handled by separated optional stages before_global_[step], after_global_[step]. Right now, my yml file is so long simply because I have to include shared steps on each stage because on different stages I want to add or pre-add steps.

[svenfuchs]

For all of you who have asked for named jobs (“job labels”, “job descriptions”) ... this has been added and released yesterday: https://blog.travis-ci.com/2018-07-18-build-stages-officially-released

Many of the questions asked here were actually support cases (some of which have been perfectly answered by some of you, such an amazing community! ❤️). Since Build Stages are officially out of beta I am going to close this GitHub issue, and recommend using our new forum for further conversations about this feature: https://travis-ci.community

Thanks everyone!

[webknjaz]

@svenfuchs thank you for all your hard work!

named jobs (“job labels”, “job descriptions”) ... this has been added and released yesterday

Since I had to click-through several links to confirm how it works, here's just an extract for others: add name: on the job definition level (in items of jobs.include/matrix.include).

[webknjaz]

@svenfuchs I've added a proposal on improvements for job labels. I hope it has enough reasoning :) https://travis-ci.community/t/job-labels-name-improvement-proposal/79?u=webknjaz

[phoenix1480]

I love build stages!

[kgcarr]

Being able to specify jobs from previous stages as dependencies for a job in a later stage would be nice, so if one job fails in an early stage only the jobs in later stages that are dependent on it's success are cancelled.

[yahliu]

Previous stage have two jobs, if one job failed, so the next stage will not be run. Actually, two jobs in stage1 are absolute, stage2-1 just only depend on stage1-1, but no dependencies with stage1-2.
so it would be good to set specify job in previous stage as dependency a job in the next stage.

[KeyWeeUsr]

Hey there ! Amazing feature !

Why is the lint stage running only on python2.7?

Here's my travis file:
https://github.com/OpenApprenticeFoundation/openapprentice-blog/blob/779ac3b416ba6d0e43529ed1d2af3169e05e98a8/.travis.yml#L1-L48

Oh and one of my jobs failed weirdly:
Please override the script: key in your .travis.yml to run tests.

The build:
https://travis-ci.org/OpenApprenticeFoundation/openapprentice-blog/builds/395339275

@Seluj78 Same thing for me with the random wild Test stage throwing basically no error and failing the whole build. It appeared after I included osx to the os: but you don't have such a thing in your YML. I resolved it with actually adding it to the stages as a conditional stage:

stages:
    - name: test
      if: branch = gibberish

so that it is never run. Ugly workaround af, but works.

[henrik242]

I create jobs programatically through the Travis API (using Github Enterprise). Usually there are two stages with some jobs in each, but some times there's a single stage with a single job. In the latter cases, the Travis UI does not display the stage and job name in a list, it just proceeds as if it was a regular "simple" job. It would have been really nice to see that stage in the UI.

This is similiar to matrix builds. If the matrix only has a single job, the Travis UI does not show the matrix list. I would rather that it displayed that single entry!

[illeatmyhat]

I tried to use the build stages to essentially better document the CI/CD process and have multiple things building and installing at the same time.
You really should better document that each build stage and job are separate environments.
How are people supposed to "Build-Test-Deploy" when they are all separate environments?

[rweickelt]

How are people supposed to "Build-Test-Deploy" when they are all separate environments?

See travis-ci/travis-ci#7841

If you have are not just building Docker images, but need to pass file artifacts, then the official answer is something like "Use amazon s3". If you don't store a lot of stuff and remove your files often, then S3 costs you nothing. It's not a very convenient solution though.

You can also abuse github releases to store your intermediate artifacts. Create a dummy tag and use the deploy step to upload your artifacts always to that tag and include the TRAVIS_BUILD_NUMBER into the file name. In a subsequent stage, download it with curl since the URL of the artifact is predictable.

[cyberhck]

the way I do is this:
test phase: build (just for linting), unit test, integration tests, postman tests in parallel, before postman is done, it builds a docker image, tests against postman tests the other integration tests, and pushes to docker hub if it passes with tag same as commit hash

deploy phase (I use semantic release): pull the docker image, run semantic-release and see if we need to release, if we do, then tag the image with appropriate version (see semantic-release for version info)

[mriedem]

As a relatively new Travis user I like build stages. I'm working on a project with a test and deploy stage and the test stage only had a single job to run unit tests. I wanted to add a linter job to the test stage and it was easy. My only complaint is that to organize the two jobs within the test build stage I gave them separate names but the names don't show up in the build page in the UI at all. I can see them in the "View config" tab though. If they aren't shown in the UI rather than an auto-incrementing job number, e.g. #3008.2, I'm not sure what use naming the jobs serves except organizing within the config file, which could be achieved with comments just as easily. Maybe I'm missing something about where the job names show up?

[Saviq]

Hi @mriedem language and environment does show up in the list:

[eine]

@Saviq, I think that @mriedem means https://docs.travis-ci.com/user/build-stages/#naming-your-jobs-within-build-stages. Honestly, I am not sure about the expected usage of that field.

[mriedem]

@Saviq, I think that @mriedem means https://docs.travis-ci.com/user/build-stages/#naming-your-jobs-within-build-stages. Honestly, I am not sure about the expected usage of that field.

Correct, I'm wondering why if a job has a name it's not used in the UI. For example, in the screenshot in the comment above if all of those jobs had names why wouldn't the job ID, e.g. 2696.1, be replaced with the job name in the UI. The ID is useful for computers, the name is useful for humans.

[edurocher]

Hi, not sure if it is known or not but it looks like the build times displayed in the UI are not always correct?
For example, our build would show "Ran for 14 min 7 sec", but the first stage lasted 14 min and the longest job in the second stage lasted 13 min, so it seems unlikely that the total could be 14 min...
Does not happen all the time though. Maybe I missed something on what these times really mean?

[shockdm]

It seems that the build time on the PR level is the sum of all of the jobs in every stage, rather than the total real run time of the build. That statistic is available when you drill down into the PR build:

It would be more useful to see the Run for stat on the PR level, rather than Total time. Is this configurable?

[shockdm]

There seems to be a problem with the way that jobs clone the +refs/pull/<PR#>/merge during the PR build. Travis relies on Github for +refs/pull/<PR#>/merge state. But that state may change if master changes, or if the PR branch itself changes. So you may end up with different commits/code in different stages of the same build, or if you rerun a particular job of a build (it may have mismatched artifacts with the other stages/jobs). This can be worked around by creating a temporary branch that concertizes +refs/pull/<PR#>/merge state of the pr branch in the first job of the build, and having the rest of the jobs refer to this temporary branch, rather than a moving target of +refs/pull/<PR#>/merge. While there is a workaround - it seems that something like this should be handled by Travis itself, as it introduces a lot of instability, especially for very active codebases.

[xavimb]

Build stages take about 1 minute to get executed, even when the if condition is not met. So even for stages that do not do anything at all, it still tries to find a place to execute them and then it takes about 22-27s to figure out that it doesn't meet the if condition and it should not do anything for that stage. Is there a way to speed up the evaluation of the if condition?

[Sarquamon]

Enjoying so far the build stages, keeps everything nice and tidy. But maybe there could be a way of adding a "pre-stage" tag to allow some sort of previous configuration.

Or find some sort of way of storing and sharing docker images locally. That would be nice

[nielspardon]

FYI: some Travis CI Enterprise installations still point to this issue for feedback for this "beta feature" although it's been long out of beta.