[SPARK-35101][INFRA] Add GitHub status check in PR instead of a comment

### What changes were proposed in this pull request?

TL;DR: now it shows green yellow read status of tests instead of relying on a comment in a PR, **see HyukjinKwon#41 for an example**.

This PR proposes the GitHub status checks instead of a comment that link to the build (from forked repository) in PRs.

This is how it works:

1. **forked repo**: "Build and test" workflow is triggered when you create a branch to create a PR which uses your resources in GitHub Actions.
1. **main repo**: "Notify test workflow" (previously created a comment) now creates a in-progress status (yellow status) as a GitHub Actions check to your current PR.
1.  **main repo**: "Update build status workflow" regularly (every 15 mins) checks open PRs, and updates the status of GitHub Actions checks at PRs according to the status of workflows in the forked repositories (status sync).

**NOTE** that creating/updating statuses in the PRs is only allowed from the main repo. That's why the flow is as above.

### Why are the changes needed?

The GitHub status shows a green although the tests are running, which is confusing.

### Does this PR introduce _any_ user-facing change?

No, dev-only.

### How was this patch tested?

Manually tested at:
- HyukjinKwon#41
- HyukjinKwon#42
- HyukjinKwon#43
- HyukjinKwon#37

**queued**:
<img width="861" alt="Screen Shot 2021-04-16 at 10 56 03 AM" src="https://user-images.githubusercontent.com/6477701/114960831-c9a73080-9ea2-11eb-8442-ddf3f6008a45.png">

**in progress**:
<img width="871" alt="Screen Shot 2021-04-16 at 12 14 39 PM" src="https://user-images.githubusercontent.com/6477701/114966359-59ea7300-9ead-11eb-98cb-1e63323980ad.png">

**passed**:
![Screen Shot 2021-04-16 at 2 04 07 PM](https://user-images.githubusercontent.com/6477701/114974045-a12c3000-9ebc-11eb-9be5-653393a863e6.png)

**failure**:
![Screen Shot 2021-04-16 at 10 46 10 PM](https://user-images.githubusercontent.com/6477701/115033584-90ec7300-9f05-11eb-8f2e-0fc2ef986a70.png)

Closes #32193 from HyukjinKwon/update-checks-pr-poc.

Lead-authored-by: HyukjinKwon <gurwls223@apache.org>
Co-authored-by: Hyukjin Kwon <gurwls223@apache.org>
Co-authored-by: Yikun Jiang <yikunkero@gmail.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>

.github/workflows/labeler.yml

@@ -17,11 +17,18 @@
 # under the License.
 #
 
-name: "Pull Request Labeler"
+# Intentionally has a general name.
+# because the test status check created in GitHub Actions
+# currently randomly picks any associated workflow.
+# So, the name was changed to make sense in that context too.
+# See also https://git.luolix.topmunity/t/specify-check-suite-when-creating-a-checkrun/118380/10
+
+name: "On pull requests"
 on: pull_request_target
 
 jobs:
   label:
+    name: Label pull requests
     runs-on: ubuntu-latest
     steps:
     # In order to get back the negated matches like in the old config,

.github/workflows/notify_test_workflow.yml

@@ -1,10 +1,35 @@
-name: Notify test workflow
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+# Intentionally has a general name.
+# because the test status check created in GitHub Actions
+# currently randomly picks any associated workflow.
+# So, the name was changed to make sense in that context too.
+# See also https://git.luolix.topmunity/t/specify-check-suite-when-creating-a-checkrun/118380/10
+name: On pull request update
 on:
   pull_request_target:
     types: [opened, reopened, synchronize]
 
 jobs:
   notify:
+    name: Notify test workflow
     runs-on: ubuntu-20.04
     steps:
       - name: "Notify test workflow"
@@ -13,28 +38,53 @@ jobs:
         with:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           script: |
-            const endpoint = "GET /repos/:owner/:repo/actions/workflows/:id/runs?&branch=:branch"
+            const endpoint = 'GET /repos/:owner/:repo/actions/workflows/:id/runs?&branch=:branch'
+
+            // TODO: Should use pull_request.user and pull_request.user.repos_url?
+            // If a different person creates a commit to another forked repo,
+            // it wouldn't be able to detect.
             const params = {
               owner: context.payload.pull_request.head.repo.owner.login,
               repo: context.payload.pull_request.head.repo.name,
-              id: "build_and_test.yml",
+              id: 'build_and_test.yml',
               branch: context.payload.pull_request.head.ref,
             }
 
+            console.log('Ref: ' + context.payload.pull_request.head.ref)
+            console.log('SHA: ' + context.payload.pull_request.head.sha)
+
+            // Wait 3 seconds to make sure the fork repository triggered a workflow.
+            await new Promise(r => setTimeout(r, 3000))
+
             const runs = await github.request(endpoint, params)
-            var runID = runs.data.workflow_runs[0].id
-
-            var msg = "**[Test build #" + runID + "]"
-              + "(https://github.com/" +  context.payload.pull_request.head.repo.full_name
-              + "/actions/runs/" + runID + ")** "
-              + "for PR " + context.issue.number
-              + " at commit [`" + context.payload.pull_request.head.sha.substring(0, 7) + "`]"
-              + "(https://github.com/" + context.payload.pull_request.head.repo.full_name
-              + "/commit/" + context.payload.pull_request.head.sha + ")."
-
-            github.issues.createComment({
-              issue_number: context.issue.number,
-              owner: context.payload.repository.owner.login,
-              repo: context.payload.repository.name,
-              body: msg
+            const runID = runs.data.workflow_runs[0].id
+            // TODO: If no workflows were found, it's likely GitHub Actions was not enabled
+
+            if (runs.data.workflow_runs[0].head_sha != context.payload.pull_request.head.sha) {
+              throw new Error('There was a new unsynced commit pushed. Please retrigger the workflow.');
+            }
+
+            const runUrl = 'https://github.com/'
+              + context.payload.pull_request.head.repo.full_name
+              + '/actions/runs/'
+              + runID
+
+            const name = 'Build and test'
+            const head_sha = context.payload.pull_request.head.sha
+            const status = 'queued'
+
+            github.checks.create({
+              ...context.repo,
+              name,
+              head_sha,
+              status,
+              output: {
+                title: 'Test results',
+                summary: runUrl,
+                text: JSON.stringify({
+                  owner: context.payload.pull_request.head.repo.owner.login,
+                  repo: context.payload.pull_request.head.repo.name,
+                  run_id: runID
+                })
+              }
             })

.github/workflows/update_build_status.yml

@@ -0,0 +1,95 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+name: Update build status workflow
+
+on:
+  schedule:
+  - cron: "*/15 * * * *"
+
+jobs:
+  update:
+    name: Update build status
+    runs-on: ubuntu-20.04
+    steps:
+      - name: "Update build status"
+        uses: actions/github-script@v3
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          script: |
+            const endpoint = 'GET /repos/:owner/:repo/pulls?state=:state'
+            const params = {
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              state: 'open'
+            }
+
+            // See https://docs.github.com/en/graphql/reference/enums#mergestatestatus
+            const maybeReady = ['behind', 'clean', 'draft', 'has_hooks', 'unknown', 'unstable'];
+
+            // Iterate open PRs
+            for await (const prs of github.paginate.iterator(endpoint,params)) {
+              // Each page
+              for await (const pr of prs.data) {
+                console.log('SHA: ' + pr.head.sha)
+                console.log('  Mergeable status: ' + pr.mergeable_state)
+                if (pr.mergeable_state == null || maybeReady.includes(pr.mergeable_state)) {
+                  const checkRuns = await github.request('GET /repos/{owner}/{repo}/commits/{ref}/check-runs', {
+                    owner: context.repo.owner,
+                    repo: context.repo.repo,
+                    ref: pr.head.sha
+                  })
+
+                  // Iterator GitHub Checks in the PR
+                  for await (const cr of checkRuns.data.check_runs) {
+                    if (cr.name == 'Build and test') {
+                      // text contains parameters to make request in JSON.
+                      const params = JSON.parse(cr.output.text)
+
+                      // Get the workflow run in the forked repository
+                      const run = await github.request('GET /repos/{owner}/{repo}/actions/runs/{run_id}', params)
+
+                      // Keep syncing the status of the checks
+                      if (run.data.status == 'completed') {
+                        console.log('    Run ' + cr.id + ': set status (' + run.data.status + ') and conclusion (' + run.data.conclusion + ')')
+                        const response = await github.request('PATCH /repos/{owner}/{repo}/check-runs/{check_run_id}', {
+                          owner: context.repo.owner,
+                          repo: context.repo.repo,
+                          check_run_id: cr.id,
+                          output: cr.output,
+                          status: run.data.status,
+                          conclusion: run.data.conclusion
+                        })
+                      } else {
+                        console.log('    Run ' + cr.id + ': set status (' + run.data.status + ')')
+                        const response = await github.request('PATCH /repos/{owner}/{repo}/check-runs/{check_run_id}', {
+                          owner: context.repo.owner,
+                          repo: context.repo.repo,
+                          check_run_id: cr.id,
+                          output: cr.output,
+                          status: run.data.status,
+                        })
+                      }
+
+                      break
+                    }
+                  }
+                }
+              }
+            }