How to generate, mask, and output a token from one job in a workflow to the next

[jnahmias]

I'm building a GH Actions workflow for running tests that needs an instance of the MS SQL Server database. So, in my unit-test job, I've defined a service to spawn the mssql db in a container. This service requires an initial password to be passed via env. Rather than hardcoding some value for the password, I'd like to generate a random one as part of the workflow and pass it in. However, I seem to have run into some limitation/bug where if the generated password is masked, then it doesn't flow to subsequent jobs in the workflow.

Example code:

name: Tests
on:
  push

jobs:

  gen-passwords:
    runs-on: ubuntu-latest
    outputs:
      SAPW:  ${{ steps.genpw.outputs.sapw  }}
    steps:
      - name: install password generator
        run: sudo apt install --quiet --assume-yes apg
      - name: generate database passwords
        id: genpw
        run: |
          RAND_PW=$(apg -a 1 -n 1 -m 32 -M SNCL -E \''@:;[]{}()\"')
          printf "::add-mask::%s\n" "$RAND_PW"
          printf "::set-output name=sapw::%s\n" "$RAND_PW"
          echo "Generated password has character class distribution:"
          printf '%s' "$RAND_PW" \
            | sed -e 's/[[:lower:]]/a/g' \
                  -e 's/[[:upper:]]/A/g' \
                  -e 's/[[:digit:]]/0/g' \
                  -e 's/[^Aa0]/*/g' \
                  -e 's/./&\n/g' \
            | sort | uniq -c

  pw-test:
    runs-on: ubuntu-latest
    needs: gen-passwords
    env:
      SA_PW: ${{ needs.gen-passwords.outputs.SAPW }}
    steps:
      - name: check passing of generated passwords from a prior job
        run: |
          printf 'Got SA_PW length %s, distribution:\n' \
            $(printf '%s' "$SA_PW" | wc -c)
          printf '%s' "$SA_PW" \
            | sed -e 's/[[:lower:]]/a/g' \
                  -e 's/[[:upper:]]/A/g' \
                  -e 's/[[:digit:]]/0/g' \
                  -e 's/[^Aa0]/*/g' \
                  -e 's/./&\n/g' \
            | sort | uniq -c

This fails with the second job, pw-test, getting a blank value (0 characters) for SA_PW despite it being properly generated in the prior job.
If I comment the call to ::add-mask:: (line 18), then everything works fine.

Am I doing something wrong?
How do I enable masking on an output so that it's properly passed from one job to the next?
If there's some other way of accomplishing what I want, I'm open to that as well.
Note that the password generation cannot happen in a step of the same job in which it's used, since the password is needed for the creation of a service, which runs before any of the job steps.

[philip-gai]

This isn't something currently supported 😞

https://github.com/orgs/community/discussions/13082 is a similar discussion and has an example workaround of encrypting and later decrypting the secrets instead of masking.

You could also temporarily save the password as a repository secret, use it in the next job and then clean it up. Here's an example of doing that. Unfortunately it requires using a personal access token (PAT) to save the secret, so at that point you may prefer just encrypting and decrypting the secrets and not masking them at all.¹

Footnotes

1. This suggestion was made on my own, not as an employee of GitHub 🙂 ↩

[jnahmias]

Thanks for the reply, even though it's not really what I wanted to hear... :(

I found another thread that brings up this issue here: https://github.com/orgs/community/discussions/25225#discussioncomment-3246944
Seems like I'm not the only one wanting this functionality. How can we get it added to the roadmap?

[rdhar]

BACKGROUND CONTEXT

Although there are now docs on masking and passing secrets between jobs or workflows, it didn't meet requirements like:

• Storing and retrieving from a secret store seems a tad over-the-top (as @philip-gai mentioned).
  • The same goes for encrypted artifact upload/download.
• For ephemeral, temporary OIDC tokens, it's not feasible to store/retrieve dynamic secrets (as @jnahmias raised).

PROBLEM STATEMENTS

With these factors in mind, the main blockers are:

• Per docs, to prevent accidental disclosures, GitHub Actions redacts any configured secrets as well as common encodings of those values, like Base64.
• For dynamically-generated secrets, those have to remain masked from the caller workflow to the reusable workflow to avoid exposing their their values in the workflow logs.

PROPOSED SOLUTION

1. Before going to output secrets from the caller workflow, Base64 encode the values twice.
   1. This enables GitHub Actions to output the secrets without exposing them.
2. Per docs, pass the encoded values to the reusable workflow as secret inputs.
   1. This enables the reusable workflow accept and treat the encoded values as secrets.
3. In the reusable workflow, decode the values from Base64 twice before masking them.
   1. This ensures that the secrets remain masked in the workflow logs throughout.
   2. They can now be populated as environment variables for use in the subsequent steps of the reusable workflow.

WORKING EXAMPLE

These methods are sourced from DevSecTop/TF-via-PR (permalink) repository, which hosts a reusable workflow to run Terraform commands via PR comments, like a CLI.

As a bonus, any number of secrets can be securely passed into the reusable workflow to be used as environment variables, for example. The repository also contains recent GitHub Actions workflow runs to verify that the secrets remain masked throughout.

# caller-workflow.yml

jobs:
  credentials:
    runs-on: ubuntu-latest
    outputs:
      CREDENTIAL1: ${{ steps.credentials.outputs.CREDENTIAL1 }}
      CREDENTIAL2: ${{ steps.credentials.outputs.CREDENTIAL2 }}
    steps:
      - name: Output encoded credentials
        id: credentials
        env:
          CREDENTIAL1: ${{ secrets.CREDENTIAL1 }}
          CREDENTIAL2: ${{ secrets.CREDENTIAL2 }}
        run: |
          echo "CREDENTIAL1=$(echo $CREDENTIAL1 | base64 -w0 | base64 -w0)" >> $GITHUB_OUTPUT
          echo "CREDENTIAL2=$(echo $CREDENTIAL2 | base64 -w0 | base64 -w0)" >> $GITHUB_OUTPUT

  reusable-workflow:
    needs: credentials
    uses: reusable-workflow.yml
    secrets:
      env_vars: |
        CREDENTIAL1=${{ needs.credentials.outputs.CREDENTIAL1 }}
        CREDENTIAL2=${{ needs.credentials.outputs.CREDENTIAL2 }}

# reusable-workflow.yml

on:
  workflow_call:
    secrets:
      env_vars:
        required: true
jobs:
  parse-credentials:
    runs-on: ubuntu-latest
    env:
      env_vars: ${{ secrets.env_vars }}
    steps:
      - name: Decode credentials as environment variables
        run: |
          for i in $env_vars; do
            i=$(echo $i | sed 's/=.*//g')=$(echo ${i#*=} | base64 -di | base64 -di)
            echo ::add-mask::${i#*=}
            printf '%s\n' $i >> $GITHUB_ENV
          done
      - name: Validate credentials
        run: |
          # Secrets are now available as masked environment variable.
          echo $CREDENTIAL1 # or ${{ env.CREDENTIAL1 }}
          echo $CREDENTIAL2 # or ${{ env.CREDENTIAL2 }}

Where:

• base64 -w0: Ensures that the encoded values are outputted in a single line, regardless of the length of the input.
• base64 -di: Decodes the values from Base64, ignoring any newlines from echo'd strings.
• echo ${i#*=}: Removes the key of the key-value pair using parameter expansion, leaving only the value.
• sed 's/=.*//g': Removes the value of the key-value pair, leaving only the key.
• echo ::add-mask::${i#*=}: Masks the value of the key-value pair without outputting.