Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistent cypress runs between github action and cypress dashboard #124

Closed
MaximeHeckel opened this issue Apr 3, 2020 · 14 comments
Closed

Comments

@MaximeHeckel
Copy link

MaximeHeckel commented Apr 3, 2020

Hi 👋

I've recently updated my cypress github workflow to run only on pull_requests against the master branch instead of push events and I'm seeing a weird side effect when opening a new Pull Requests and re-running the cypress job:

  • Cypress will run the test suite of the master branch (not the head branch that is being submitted as a PR to merge on top of master). On the github status check for the cypress dashboard, I can see it's called default-group (merge) and the title of the run contains the commit hash of my head branch and the HEAD commit hash of the master branch we're merging into.

Screen Shot 2020-04-03 at 12 03 11 PM

  • If I rerun the same job, Cypress will run the test suite of the head branch (as expected). On the github status check for the cypress dashboard, I can see it's simply called default-group and the title of the run

Screen Shot 2020-04-03 at 12 03 00 PM

It looks like Cypress Dashboard is somehow capturing 2 different events on the same github ci run (from the same workflow file). I'm pretty convinced that my update to run cypress only on pull_request events triggered.

Am I doing something wrong?

Here's a screenshot of the pull requests checks:
Screen Shot 2020-04-03 at 12 01 44 PM

Here's my Github Workflow file:

name: Run End-to-End Test Suite

on:
  pull_request:
    branches:
      - master

jobs:
  build-and-e2e-test:
    runs-on: ubuntu-16.04
    strategy:
      matrix:
        node-version: [11.x]
    steps:
      - name: Checkout Commit
        uses: actions/checkout@v1
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v1
        with:
          node-version: ${{ matrix.node-version }}
      - name: Cache NPM Dependencies
        id: cache-dependencies
        uses: actions/cache@v1
        with:
          path: node_modules
          key: node-modules-${{ runner.OS }}-npm-cache-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            node-modules-${{ runner.OS }}-npm-cache
      - name: Cache Cypress Binaries
        id: cache-cypress-binaries
        uses: actions/cache@v1
        with:
          path: ~/.cache/Cypress
          key: cypress-binaries-${{ runner.OS }}-npm-cache-${{ hashFiles('**/yarn.lock') }}
          restore-keys: |
            cypress-binaries-${{ runner.OS }}-npm-cache
      - name: Install Dependencies
        if: steps.cache-dependencies.outputs.cache-hit != 'true' || steps.cache-cypress-binaries.outputs.cache-hit != 'true'
        run: |
          yarn install --force --non-interactive
      - name: Build Application
        run: yarn build
        env:
          CI: true
      - name: Run E2E Tests without recording (if not on master branch)
        uses: cypress-io/github-action@v1
        with:
          browser: chrome
          record: true
          headless: true
          start: yarn serve-built-files
          wait-on: "http://localhost:8081"
          config: defaultCommandTimeout=100000,pageLoadTimeout=100000,watchForFileChanges=false,video=false
        env:
          NODE_ENV: "test"
          CI: true
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
@jennifer-shehane
Copy link
Member

GitHub status checks will trigger differently for CI providers that utilize "Pull Requests builds". Pull request builds are generated by the CI provider (if configured) by merging the head branch with the base branch specified in the pull request (in your case - master). These are helpful in answering the question “What if this PR merged into target (base) branch”. You'll often see the commit message for the associated build look something like "Merge 2f5a7afe97740f99f5c4f5e40a3d7347bc058b30 into 1f40d78961e5d268ac072dca86176f9fb2aa7b13".

These types of builds trigger Cypress's GitHub status checks notated with the note (merge) on the status check.

If your build also runs the action against its current branch (without the generated merge 'pull request build') you would see 2 events triggered from Cypress since there are 2 actions being run.

I'm a bit unsure about whether you are saying this happens on every single PR or whether this only triggers 2 events when you 'rerun' your action. Could you clarify?

@MaximeHeckel
Copy link
Author

Thank you for your answer!

I'm a bit unsure about whether you are saying this happens on every single PR or whether this only triggers 2 events when you 'rerun' your action. Could you clarify?

I get the 1st merge event when I open the pull request and the 2nd event only when I rerun the PR. I'd have expected it to rerun as the 1st event.

@gotbahn
Copy link

gotbahn commented Jun 25, 2020

@jennifer-shehane is it possible to specify custom message, or point Cypress to proper commit sha? As I understand by default it takes github.sha, however github.event.after would serve better

Now our dashboard doesn't look good
image

@jennifer-shehane
Copy link
Member

@godban We use these environment variables to display the information in the Dashboard https://on.cypress.io/continuous-integration#Git-information So, if you have some way to overwrite this when GitHub auto creates a merge commit - then you could pass that (I'm not sure if this is possible).

@gotbahn
Copy link

gotbahn commented Jul 2, 2020

@jennifer-shehane no, this will lead to unexpected behaviour even if possible.

@bahmutov
Copy link
Contributor

bahmutov commented Jul 2, 2020

I have looked at what is going on by opening bahmutov/cypress-gh-action-example#14

The workflow is running both on push (commit) and on pull_request events

name: merge
# run this action on Push and on Pull Request
# to demonstrate the difference in commit SHA
on: [push, pull_request]

You can see the Dashboard at https://dashboard.cypress.io/projects/yz8qku and filter by tags since I pass the event name as a tag

- name: Cypress run on ${{ github.event_name }}
  uses: cypress-io/github-action@v2
  with:
    record: true
    group: merge
    tag: ${{ github.event_name }}
  env:
    CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

Every commit with open pull request thus triggers 2 events = 2 action runs

For example here in GH Actions we see merge #3 which is a push event with the current commit title "print", and merge #4 which is the pull request that uses the original PR title "add pr workflow".

Screen Shot 2020-07-02 at 2 35 37 PM

Here is what our Dashboard shows in this case

Screen Shot 2020-07-02 at 2 31 57 PM

Build 164 is the recording of the "push" event, has the expected title from the commit "print" and is ok.

Build 162 is the recording of the "pull_request" event, and it uses the default commit title created by GitHub (merge commit A into B), and this is what we would like to change.

When we look closely at the build 162, its commit also is not as nice as it could be

Screen Shot 2020-07-02 at 2 39 51 PM

which leads to the merge commit and not to the pull request

Screen Shot 2020-07-02 at 2 39 39 PM

Solution

To me the largest problem is the commit message for pull request. Thus a good solution is to set it for pull requests as an environment variable to overwrite the Git information. You can find the file at https://github.com/bahmutov/cypress-gh-action-example/blob/master/.github/workflows/merge.yml and here is the relevant portion

- name: Cypress run on ${{ github.event_name }}
  uses: cypress-io/github-action@v2
  with:
    record: true
    group: merge
    # tag will be either "push" or "pull_request"
    tag: ${{ github.event_name }}
  env:
    CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
    # - if the event is push, the title will be undefined
    # and Cypress will get the commit message from Git information
    # - if the event is pull_request, then we set the commit
    # message to the pull request title
    COMMIT_INFO_MESSAGE: ${{ github.event.pull_request.title }}

The Dashboard shows then:

Screen Shot 2020-07-02 at 2 58 35 PM

"push" event shows the commit's subject line.
"pull_request" event shows the PR title.

@bahmutov bahmutov closed this as completed Jul 2, 2020
bahmutov added a commit that referenced this issue Jul 2, 2020
@gotbahn
Copy link

gotbahn commented Jul 3, 2020

@bahmutov alright, it works. Thank you!

image

@eplatek-safyr
Copy link

@bahmutov Thanks for this solution that fixes the dashboard labels, but it breaks the PR comments on github. After applying this solution to my project I don't have cypress bot comments on my PR, after removing the COMMIT_INFO_MESSAGE variable the comments are back.

Any idea on how to keep both of those features working ?

@HartiganHM
Copy link

Checking to see if there have been any updates on this issue as I experienced the same thing as @eliasplatekdkt. The solution @bahmutov provided worked for the title but broke all Cypress status updates to the PR in question.

It would be great to configure this option with more nuance, somehow. The current title displayed from the screenshots already posted is not only overly verbose but somewhat unhelpful as to the context of the run.

@conversayShawn
Copy link
Contributor

conversayShawn commented Mar 23, 2022

@eplatek-safyr Try adding COMMIT_INFO_SHA: ${{ github.event.pull_request.head.sha }} as an env var.

Depending on your setup, it could look something like this:

env:
 CYPRESS_PROJECT_ID: ${{ secrets.CYPRESS_PROJECT_ID }}
 CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
 GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 COMMIT_INFO_MESSAGE: ${{ github.event.pull_request.title }}
 COMMIT_INFO_SHA: ${{ github.event.pull_request.head.sha }}

@mellis481
Copy link

mellis481 commented Jul 20, 2022

@bahmutov's solution above (which does successfully update the commit message) not only breaks Cypress bot comments as @eplatek-safyr points out, but it also prevented status checks that I had configured in my parallelized workflows from ever running.

The solution above of adding COMMIT_INFO_SHA: ${{ github.event.pull_request.head.sha }} appears to correct these issues for PR workflows. @conversaShawn: Is there only a need to set COMMIT_INFO_SHA for PR workflows? I have a workflow that uses the workflow_dispatch trigger to do manual deployments which I don't think I would need to add it to after setting a custom COMMIT_INFO_MESSAGE, but want to confirm.

@conversayShawn
Copy link
Contributor

@mellis481 I am currently investigating

@conversayShawn
Copy link
Contributor

@mellis481 After some testing, COMMIT_INFO_MESSAGE is only necessary to obtain the message in the Dashboard when using pull_request as part of the workflow. The COMMIT_INFO_SHA seems to be necessary when using pull_request or workflow_dispatch workflows to definitively determine the run has ended.

@mellis481
Copy link

@conversaShawn I'm confused why you say COMMIT_INFO_SHA is needed for workflows that use the workflow_dispatch trigger since my manually-initiated workflows are currently working without it. Plus what value would you use for it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

10 participants