Compose GitHub actions as a single GitHub action

[ghost]

What I like from GitHub action is we can ensure that our action is up-to-date with available fixes in the upstream. E.g.: we can use actions/checkout@v2 and the CI will still run with the latest version of checkout@v2.

Unfortunately, this facility can't be used while we're creating a GitHub action. I think it's better if we can compose multiple GitHub actions into a single GitHub action, or maybe just use one or two GitHub actions for the trivial tasks and continue the other in Javascript or Docker container.

This is an imaginary action.yml to serve as an example, maybe it's cool if we can do this:

name: 'imaginary-extension'
description: 'Imaginary extension about composing actions'
runs:
  using: composite
  steps:
    - name: Python 3
      uses: actions/setup-python@v1
      with:
        python-version: 3.8
    - name: Requirements
      run: pip install -r requirements.txt

[TingluoHuang]

It feels like a template reuse scenario, so you can share a list of reusable steps with others.

@chrispat @ericsciple for an idea.

[ericsciple]

Yep it's something we want to do. Currently behind other priorities.

[ghost]

Please tell me about the progress later. I'm interested in this feature.

[hross]

We are picking up work here so keep an eye out for activity. We will update things as we go.

[ghost]

Super cool. Thank you!

[pelotom]

This is our team's number 1 pain point with GH actions, would love to see something like this implemented, or alternatively support for YAML anchors (how we used to accomplish DRY with Travis).

[vanppo]

This is our team's number 1 pain point with GH actions, would love to see something like this implemented, or alternatively support for YAML anchors (how we used to accomplish DRY with Travis).

YAML anchors may not be good enough. For the action below, we can not share common steps between jobs by YAML anchors.

test-job:
  steps:
    - checkout
    - cache
    - dependencies
    - test

build-job:
  steps:
    - checkout
    - cache
    - dependencies
    - build

Syntax below may be the resolution.

defaults:
  steps:
    - checkout
    - cache
    - dependencies
jobs:
  test-job:
    - test
  build-job:
    - build

[Enrico2]

A single step must be able to reference outputs of previous steps, and so in order to define a step in a centralized way, there must be a way to provide inputs to that step.
I like the idea of being able to define a composed set of steps as an action with inputs:
E.g.

composites:
  init: 
    input:
      - branch
    steps:
      - uses: actions/checkout@v2
        with: {{ input.branch }}
      - actions/cache...
      - actions/test
        
job1:
  steps:
    - if: {{ event == release }}
      run: ... set some output, on a e.g. my_var
      id: is_release
    - uses: composites.init
       with:
         branch: steps.is_release.outputs.my_var
    - run: ...

[hross]

We are working through this scenario. The first feature we need to get working is multiple run steps, after which we would allow more complex action composition. Right now we plan to do it with a separate action template/definition.

You can see this work in progress for how we are going to support multiple run steps (then imagine this gets extended to more complicated steps (there is more technical work to support things like uses/etc): #591

Also see the full ADR for more detail on syntax/usage: https://github.com/actions/runner/blob/main/docs/adrs/0549-composite-run-steps.md

[Vampire]

As the mentioned PR is now merged, when can we expect this to be publicly available? :-)

[ThibaultVlacich]

@timharris777 The comment right above yours...

[timharris777]

@timharris777 The comment right above yours...

Sorry @ThibaultVlacich, I'm not following you. This is already released and I have been using it today. It was in the latest runner features from like 9 days ago and it was announced today. https://github.blog/changelog/2020-08-07-github-actions-composite-run-steps/

[sidvishnoi]

@timharris777 See #646

[timharris777]

@timharris777 See #646

Thanks for pointing that out @sidvishnoi , just read it. Looks like their long term plan is to support actions within actions and this is a step in that direction-- but for now they are just supporting run commands from different shells.

[mikedrexler]

Hi @timharris777 ! It would appear that your answer arrived before I saw your question. Tracking in the Chick-Fil-A customer repository at https://github.com/githubcustomers/Chick-Fil-A/issues/11.

[nsheff]

The 'composite' action type allows us to run sequential commands natively, but is there any way to put together docker-based actions? For instance, I'd like one of the steps in the composite action to be running a docker action.

[razor-x]

@nsheff Check out #646

[hross]

We still have work to do on this feature. I'm linking this issue: #646 so everyone can see we are totally aware of the future work to get "actions in actions" working fully. See that issues for what's left to do and the state of the world for this feature.

[reitzig]

FWIW and for googlers landing here, I don't see how the closing MR offers more than YAML replacement for simple intermediate/wrapper scripts. Qualitively speaking; there seem to be some consequences how workflow runs show up in the UI, which are of secondary concern IMHO.
In particular, the core issue of (transparently) calling another action has not been addressed, at all.

Both the MR and #646 express that being able to express multiple steps in YAML is a major stepping stone towards that. I don't necessarily follow -- another route, such as before-steps: <array of uses> would have seemed feasible as well -- but I'll take your word for it. #612 was dealing with the core issue, though, so #646 remains the issue to watch.

[emilong]

YAML anchors do everything I want, which includes sharing configuration not related to steps, i.e. stuff I want to share but don't want to run like container configuration.

To prove this to myself, I decided to create a .github/workflows-src/ directory with workflow yaml files which included anchors, but only defined under the top-level anchors key.

Here's an example (simplified) workflow yaml source file:

---
name: Integration

"on": [pull_request]

anchors:
  docker_hub_credentials: &docker_hub_credentials
    credentials:
      username: my-dockerhub-username
      password: ${{ secrets.DOCKER_HUB_PASSWORD }}

  postgres_service: &postgres_service
    postgres:
      image: mdillon/postgis:9.6
      <<: *docker_hub_credentials

  cache_gems: &cache_gems
    name: Cache Ruby Gems
    uses: actions/cache@v2
    env:
      cache-name: cache-gems
    with:
      path: vendor/bundle
      key: ${{ runner.os }}-build-${{ env.cache-name }}-gems-${{ hashFiles('Gemfile.lock') }}
      restore-keys: |
        ${{ runner.os }}-build-${{ env.cache-name }}-gems-
        ${{ runner.os }}-build-${{ env.cache-name }}-
        ${{ runner.os }}-build-
        ${{ runner.os }}-

jobs:
  lint_and_dependencies:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - *cache_gems
      - run: bundle install
      - name: Run Pronto
        run: >
          PRONTO_PULL_REQUEST_ID="${{ github.event.number }}"
          PRONTO_GITHUB_ACCESS_TOKEN="${{ github.token }}"
          bundle exec pronto run -f github_status github_pr -c origin/${{ github.base_ref }}
  run_rspec_tests:
    needs: lint_and_dependencies
    runs-on: ubuntu-latest
    services:
      <<: *postgres_service
    steps:
      - uses: actions/checkout@v2
      - *cache_gems
      - run: bundle install
      - name: Prepare database
        run: psql -h postgres -U test test_db < db/structure.sql
      - name: Run RSpec
        run: bundle exec rspec

Then I preprocess in a pre-commit hook with this script:

require "yaml"
require "json"

parsed = JSON.parse(YAML.load_file(ARGV.first).to_json)
parsed.delete("anchors")

puts YAML.dump(parsed)

Maybe this is of some use to others here, but also possibly an example of what many of us are looking for with YAML anchors for the GitHub team.

(P.S. I'm a little confused by the anchor parsing concerns as I'm not sure if I could turn off anchors in the transformation script above, but you all know your codebase better than me, of course)

[rhyek]

Per @emilong's example I tried it in an example repo of mine and it's working wonderfully. I included functionality to automatically merge/flatten steps lists, as well.

You can check out the example here. Hope someone finds it useful.