✨ Enable GitHub Enterprise Server (GHES) support #2788

rajbos · 2023-03-24T11:38:58Z

What kind of change does this PR introduce?

Support GHES by using the Enterprise client for the GitHub calls.

PR title follows the guidelines defined in our pull request documentation

What is the current behavior?

$ scorecard --repo github.corp.net/nv35/myrepo
unsupported host: github.corp.net

Which issue(s) this PR fixes

Fixes Support private git servers (like Github Enterprise instances) #660

Special notes for your reviewer

There is one error that needs to be fixed, since this returns a non-zero exit code at the moment, hence the Draft status.

Error: check runtime error: Branch-Protection: internal error: error during branchesHandler.setup: internal error: githubv4.Query: Field 'requireLastPushApproval' doesn't exist on type 'BranchProtectionRule'
2023/03/24 12:04:20 error during command execution: check runtime error: Branch-Protection: internal error: error during branchesHandler.setup: internal error: githubv4.Query: Field 'requireLastPushApproval' doesn't exist on type 'BranchProtectionRule'
exit status 1

I expect this is coming from our GHES instance (3.7) that does not support this field yet. If I remove the field requireLastPushApproval all checks run normally, so we might need to check the GHES version and include/exclude this field based on that.

Does this PR introduce a user-facing change?

Not for existing users, as it did not support GHES before. No changes needed by existing users.

Discussion

I am using os.Getenv("GITHUB_API_URL") to pick up the API url for the server, as this is where I'd expect this to run from an automation context. We could pick up the url from the scorecard --repo github.corp.net/nv35/myrepo command, but that means wiring up all the places from front to back to pick this up and pass it through and then update CreateGithubRepoClient calls everywhere.

 Support for GitHub Enterprise Server has been added.

Signed-off-by: GitHub <noreply@github.com>

github-actions · 2023-04-04T01:54:06Z

Stale pull request message

rajbos · 2023-04-04T08:34:19Z

I'm going to work on this again later this week. Not stale.

azeemshaikh38 · 2023-04-04T14:37:20Z

@raghavkaul could you help review this PR whenever its ready?

raghavkaul · 2023-04-04T15:11:16Z

Yes, happy to review whenever this is ready.

Signed-off-by: GitHub <noreply@github.com>

rajbos · 2023-04-13T17:23:03Z

Tested against GHES 3.8.1 today and it works: the field in the GraphQL search exists.

Two things left to discuss:

I need some help with the failing unit test: I have no clue how this works in Go and why this is failing (I get that I changed the setup partially)
Should this new change be reflected in the REAMDE (GHES is supported by looking at the GITHUB_API_URL variable. Is so: where? We can then note that this works from 3.8 and up.

rajbos · 2023-04-13T17:29:31Z

Tagging @raghavkaul for visiblity.

naveensrinivasan

Thanks!

naveensrinivasan · 2023-04-20T16:21:12Z

clients/githubrepo/client.go

@@ -126,7 +128,7 @@ func (client *Client) InitRepo(inputRepo clients.Repo, commitSHA string, commitD

 // URI implements RepoClient.URI.
 func (client *Client) URI() string {
-	return fmt.Sprintf("github.com/%s/%s", client.repourl.owner, client.repourl.repo)
+	return fmt.Sprintf("%s/%s/%s", client.repourl.host, client.repourl.owner, client.repourl.repo)


Can you please write a test for this?

naveensrinivasan · 2023-04-20T16:21:35Z

clients/githubrepo/client.go

@@ -259,8 +261,33 @@ func CreateGithubRepoClientWithTransport(ctx context.Context, rt http.RoundTripp
 	httpClient := &http.Client{
 		Transport: rt,
 	}
-	client := github.NewClient(httpClient)
-	graphClient := githubv4.NewClient(httpClient)
+


Can you please include tests for these?

@naveensrinivasan , can you give me some pointers / help on where to get started? (not a Go programmer, so testing in Go is new to me).

I see a generic setup for creating the clients during the tests here: clients/githubrepo/githubrepo_suite_test.go, and another generic mock here clients/mockclients/repo_client.go.

github-actions · 2023-05-01T01:54:13Z

Stale pull request message

github-actions · 2023-05-13T01:51:16Z

Stale pull request message

rajbos · 2023-05-13T09:45:34Z

Still want to work on this Stale bot! Looking for the time to do so 😄

github-actions · 2023-06-03T02:01:58Z

Stale pull request message

esnible · 2023-06-05T12:22:57Z

The error messages were not obvious to me:

go run main.go --repo https://github.ibm.com/<myorg>/<myproject>
2023/06/05 08:05:42 error during command execution: RunScorecard: repo unreachable: Get "/api/v3/repos/<myorg>/<myproject>": internal error: innerTransport.RoundTrip: internal error: innerTransport.RoundTrip: unsupported protocol scheme ""

Could this message remind people that you expect $GITHUB_API_URL, and the value it should have?

GITHUB_API_URL=https://api.github.ibm.com go run main.go --repo https://github.ibm.com/<myorg>/<myproject>
Error: RunScorecard: repo unreachable: GET https://api.github.ibm.com/repos/<myorg>/<myproject>: 401 Must authenticate to access this API. []

Could this message include the usual one GitHub token env var is not set. Please read https://github.com/ossf/scorecard#authentication error="an error occurred while getting GitHub credentials"

GITHUB_AUTH_TOKEN=<mytoken> GITHUB_API_URL=https://api.github.ibm.com go run main.go --repo https://github.ibm.com/<myorg>/<myproject>
Error: RunScorecard: internal error: ListCommits:error during graphqlHandler.setup: internal error: githubv4.Query: non-200 OK status code: 404 Not Found body: "{\"message\":\"Not Found\",\"documentation_url\":\"https://docs.github.com/enterprise-server@3.3/rest\"}"

I gave up before solving this one. Does it want the enterprise URL instead of docs.github.com? How do I provide that?

github-actions · 2023-06-17T01:56:45Z

Stale pull request message

rajbos · 2023-06-19T18:51:40Z

Missed the feedback from @esnible, will try to address that.

esnible · 2023-06-21T17:55:16Z

clients/githubrepo/client.go

+		}
+		// trim trailing slash to prevent issues with the graphql client
+		githubSERVERURL = strings.TrimSuffix(githubSERVERURL, "/")
+		githubGRAPHQLURL := fmt.Sprintf("%s/api/graphql", githubSERVERURL)


This line didn't work for me. To run the scorecards, I used GITHUB_API_URL=https://api.github.ibm.com and changed this line to

githubGRAPHQLURL := fmt.Sprintf("%s/graphql", githubSERVERURL)

Using "%s/api/graphql" caused 404s, using /graphql allowed the scorecard to generate. I only have access to my own company's GHE, so I don't know if you should always use /graphql or if the path needs to be parameterized for different configurations.

spencerschrock · 2023-06-21T20:40:17Z

FYI, #2999 was merged while this was marked stale.

Can we see if @esnible's feedback applies to what has already been merged? If not, can we re-purpose this PR (or close and open a new one) to fixing what's already been merged?

esnible · 2023-06-21T21:43:54Z

Using the current HEAD I was able to GITHUB_AUTH_TOKEN=$GITHUB_ENTERPRISE_TOKEN GH_HOST=github.ibm.com go run main.go --repo https://github.ibm.com/<owner>/<project> --show-details --checks Vulnerabilities and it worked just fine.

I had incorrectly tried GH_HOST=api.github.ibm.com with HEAD, as that is the host I use when I make GraphQL queries, and that is the host that this PR expected, and I found the error message confusing. However, I was not confused enough to be stuck. The error message was:

error during command execution: GetClients: error making github repo: error in IsValid: unsupported host: github.ibm.com

That confused me because

I wasn't making a github repo
I was passing in a GH_HOST that I knew was supported
I thought the error message referred to $GH_HOST

In terms of functionality the code in HEAD is great.

github-actions · 2023-07-02T02:10:50Z

Stale pull request message

rajbos · 2023-07-10T15:16:40Z

I've validated with v4.11.0 and it works against our internal GHES environment, so I am ready to close this issue. Thanks for adding the support!

spencerschrock · 2023-07-12T20:03:29Z

I've validated with v4.11.0 and it works against our internal GHES environment, so I am ready to close this issue. Thanks for adding the support!

All the thanks goes to you and Niket!

rajbos changed the title ~~✨ Enable GitHub Enterprise Server (GHES) support, closes #660~~ ✨ Enable GitHub Enterprise Server (GHES) support Mar 24, 2023

Support GitHub Enterprise Server hosts

b443372

Signed-off-by: GitHub <noreply@github.com>

rajbos force-pushed the ghes-support branch from 6f0c285 to b443372 Compare March 24, 2023 11:50

rajbos had a problem deploying to integration-test March 24, 2023 21:26 — with GitHub Actions Failure

github-actions bot added the no-pr-activity label Apr 4, 2023

github-actions bot removed the no-pr-activity label Apr 5, 2023

Handle error

8ab5e6d

Signed-off-by: GitHub <noreply@github.com>

rajbos force-pushed the ghes-support branch from 7147e54 to 8ab5e6d Compare April 13, 2023 17:13

Fix typo

630d00d

Signed-off-by: GitHub <noreply@github.com>

Merge branch 'main' into ghes-support

b663652

rajbos marked this pull request as ready for review April 13, 2023 17:28

rajbos requested review from azeemshaikh38, justaugustus, laurentsimon, naveensrinivasan, spencerschrock and raghavkaul as code owners April 13, 2023 17:28

naveensrinivasan removed request for naveensrinivasan, justaugustus, azeemshaikh38, spencerschrock and laurentsimon April 20, 2023 16:18

rajbos had a problem deploying to integration-test April 20, 2023 16:19 — with GitHub Actions Failure

naveensrinivasan reviewed Apr 20, 2023

View reviewed changes

github-actions bot added no-pr-activity and removed no-pr-activity labels May 1, 2023

patelniketm mentioned this pull request May 12, 2023

✨ Add support for github GHES #2999

Merged

2 tasks

github-actions bot added the no-pr-activity label May 13, 2023

github-actions bot removed the no-pr-activity label May 14, 2023

github-actions bot added the no-pr-activity label Jun 3, 2023

github-actions bot removed the no-pr-activity label Jun 6, 2023

github-actions bot added the no-pr-activity label Jun 17, 2023

github-actions bot removed the no-pr-activity label Jun 20, 2023

esnible reviewed Jun 21, 2023

View reviewed changes

github-actions bot added the no-pr-activity label Jul 2, 2023

github-actions bot removed the no-pr-activity label Jul 11, 2023

spencerschrock closed this Jul 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Enable GitHub Enterprise Server (GHES) support #2788

✨ Enable GitHub Enterprise Server (GHES) support #2788

rajbos commented Mar 24, 2023 •

edited

Loading

github-actions bot commented Apr 4, 2023

rajbos commented Apr 4, 2023

azeemshaikh38 commented Apr 4, 2023

raghavkaul commented Apr 4, 2023

rajbos commented Apr 13, 2023

rajbos commented Apr 13, 2023

naveensrinivasan left a comment

naveensrinivasan Apr 20, 2023

naveensrinivasan Apr 20, 2023

rajbos May 23, 2023

github-actions bot commented May 1, 2023

github-actions bot commented May 13, 2023

rajbos commented May 13, 2023

github-actions bot commented Jun 3, 2023

esnible commented Jun 5, 2023

github-actions bot commented Jun 17, 2023

rajbos commented Jun 19, 2023

esnible Jun 21, 2023 •

edited

Loading

spencerschrock commented Jun 21, 2023 •

edited

Loading

esnible commented Jun 21, 2023

github-actions bot commented Jul 2, 2023

rajbos commented Jul 10, 2023

spencerschrock commented Jul 12, 2023

✨ Enable GitHub Enterprise Server (GHES) support #2788

✨ Enable GitHub Enterprise Server (GHES) support #2788

Conversation

rajbos commented Mar 24, 2023 • edited Loading

What kind of change does this PR introduce?

What is the current behavior?

Which issue(s) this PR fixes

Special notes for your reviewer

Does this PR introduce a user-facing change?

Discussion

github-actions bot commented Apr 4, 2023

rajbos commented Apr 4, 2023

azeemshaikh38 commented Apr 4, 2023

raghavkaul commented Apr 4, 2023

rajbos commented Apr 13, 2023

rajbos commented Apr 13, 2023

naveensrinivasan left a comment

Choose a reason for hiding this comment

naveensrinivasan Apr 20, 2023

Choose a reason for hiding this comment

naveensrinivasan Apr 20, 2023

Choose a reason for hiding this comment

rajbos May 23, 2023

Choose a reason for hiding this comment

github-actions bot commented May 1, 2023

github-actions bot commented May 13, 2023

rajbos commented May 13, 2023

github-actions bot commented Jun 3, 2023

esnible commented Jun 5, 2023

github-actions bot commented Jun 17, 2023

rajbos commented Jun 19, 2023

esnible Jun 21, 2023 • edited Loading

Choose a reason for hiding this comment

spencerschrock commented Jun 21, 2023 • edited Loading

esnible commented Jun 21, 2023

github-actions bot commented Jul 2, 2023

rajbos commented Jul 10, 2023

spencerschrock commented Jul 12, 2023

rajbos commented Mar 24, 2023 •

edited

Loading

esnible Jun 21, 2023 •

edited

Loading

spencerschrock commented Jun 21, 2023 •

edited

Loading