Improvements to e2e tests

[laurentsimon]

It would be useful to have the following to improve troubleshooting and maintaining our e2e tests:

1. In each test repo, name the workflows as: trigger.branch.(<optional params>)yml, e.g. schedule.main.yml, workflow_dispatch.main.<user-options>
2. Give each workflow a name to make it easier to locate in the Actions tab: Trigger - Branch should work?
3. Add more information to the body of the issues created when the e2e test fails:

Repo: <repo name@branch>
(optional) Workflow name: <name of workflow>
Workflow file: <path/to/workflow.yaml>
Trigger: <trigger-name>
Branch: <branch-name>
Private repo : <true/false>

@naveensrinivasan wdut?

/cc @azeemshaikh38

[azeemshaikh38]

Let's prioritize (2) and (3) since we need this for debugging any failures in the upcoming Golang-based action. Since workflow_dispatch and pull_request are supposed to be experimental and not expected to have full-support in our upcoming release, let's attempt those when we plan to add support for it.

[laurentsimon]

So let's wait until these minor changes (2 and 3) are landed to cut the release?

@naveensrinivasan I don't know if you've created PRs on the other repos for these, but I think I'd like to be a reviewer. Not that I'll block anything, but mostly to follow and learn how it's setup a little more in details :-) Thanks!

[azeemshaikh38]

So let's wait until these minor changes (2 and 3) are landed to cut the release?

Yes, if you mean the Golang release. Not needed for the minor release maybe.

[laurentsimon]

OK, then we have only one item left for the minor, which @rohankh532 is working on #196
Almost there!

[laurentsimon]

An additional feature is to crate an entry only if the issue is not already crated. If the issue already exists, we can add a comment with the information for the new run. I've implemented this in https://github.com/slsa-framework/example-package/blob/main/.github/workflows/e2e-report-failure.sh, I think it can be re-used

[naveensrinivasan]

So let's wait until these minor changes (2 and 3) are landed to cut the release?

@naveensrinivasan I don't know if you've created PRs on the other repos for these, but I think I'd like to be a reviewer. Not that I'll block anything, but mostly to follow and learn how it's setup a little more in details :-) Thanks!

Yes, I will include you as the reviewer!

[laurentsimon]

Cool, thanks. Fyi you can see the setup for SLSA in https://github.com/slsa-framework/example-package/blob/main/.github/workflows/

[naveensrinivasan]

@laurentsimon FYI

1. Included details message in issue ossf-tests/scorecard-action#1
2. https://github.com/test-organization-ls/scorecard-action-private-repo-tests/pull/1

[naveensrinivasan]

Non-main branch
https://github.com/ossf-tests/scorecard-action-non-main-branch/blob/80fd39d8425aaa571fc06b8b4b536c6c83777eb6/.github/workflows/scorecard-golang.yml#L64

https://github.com/ossf-tests/scorecard-action-non-main-branch/blob/80fd39d8425aaa571fc06b8b4b536c6c83777eb6/.github/workflows/scorecard-main.yml#L64

[laurentsimon]

You also need to set THIS_FILE. I could not find a GITHUB_ for it. Let me know if you find one

[naveensrinivasan]

You also need to set THIS_FILE. I could not find a GITHUB_ for it. Let me know if you find one

Thanks, I wasn't able to find one either. I set them manually so that it will help to debug. Let me know. If you are good merge those PR's. Thanks

[laurentsimon]

Go ahead merge. No further comment on my end.

[naveensrinivasan]

They are merged. Can we close this ticket and open another if we want any other enhancements?

[laurentsimon]

SGTM. Feel free to close and add the newly-created issue to the release issue. I think in the new issue we want:

1. Re-use existing issue instead of creating new one
2. Verify push/dispatch_workflow work properly
3. Auto-close issues when e2e tests pass

Are these what you had in mind or there were others?