-
Notifications
You must be signed in to change notification settings - Fork 474
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot scan self-hosted (private) GitLab repositories #3696
Comments
Our code tries to create a GitLab client first, and then falls back to GitHub. Lines 58 to 75 in 04ea8be
Since it's falling back to GitHub, there was some sort of error making the GitLab one, which we unfortunately aren't surfacing. My guess is this has to do with auth tokens since we don't pass one in here. scorecard/clients/gitlabrepo/repo.go Lines 98 to 114 in 04ea8be
|
If you're able to try building locally, can you try this patch? diff --git a/checker/client.go b/checker/client.go
index 1b5d28a3..d9570773 100644
--- a/checker/client.go
+++ b/checker/client.go
@@ -61,6 +61,7 @@ func GetClients(ctx context.Context, repoURI, localURI string, logger *log.Logge
}
if makeRepoError != nil || repo == nil {
+ fmt.Println(makeRepoError)
repo, makeRepoError = ghrepo.MakeGithubRepo(repoURI)
if makeRepoError != nil {
return repo,
diff --git a/clients/gitlabrepo/repo.go b/clients/gitlabrepo/repo.go
index 65a44783..1a371ed6 100644
--- a/clients/gitlabrepo/repo.go
+++ b/clients/gitlabrepo/repo.go
@@ -97,7 +97,7 @@ func (r *repoURL) String() string {
// IsValid implements Repo.IsValid.
func (r *repoURL) IsValid() error {
- if strings.Contains(r.host, "gitlab.") {
+ if strings.Contains(r.host, "gitlab") {
return nil
}
|
Hey, thanks for the fast answer! I tried to install but I am not very familiar with go, getting lot of errors... I guess you will be faster getting the Your patch makes sense to me, the url looks like this:
|
@spencerschrock Building or running on the machine (centos) needed to access the private gitlab instance I am just getting this strange error, and I cannot find any useful info how to fix that: I tried on my personal laptop adding your print statement, but of course there it is not reachable: If you know how to fix this asm error I could try again |
Hmm, I can't say I've seen the error before. A quick search says it's something preventing you from running the asm binary, either antivirus, or maybe all of the Go binaries aren't in your path, or there's conflicting versions of Go installed? Maybe check you're seeing the expected things when running: which go Depending on the results, you can try setting your GOROOT and GOPATH. export GOROOT=`which go`
export PATH=$GOROOT/bin:$PATH Another alternative is compiling on your laptop and transfering the binary to the centos machine (if possible) |
I fixed it via installing a fresh go bin and running scorecard directly. Seems like there is an issue reaching the gitlab instance, but strange error regarding "invalid character <" is thrown. However trying wget the connection works: Update: I exported a valid gitlab token before running scorecard, like mentioned in the docs |
Thanks, I think this error helps confirm the GitLab server is responding with HTML (possibly a 404 since Scorecard isn't using the user-provided token for this call). The library we use is trying to parse the HTML as a JSON response, leading to the invalid character message.
Just to confirm, exporting the token didn't fix it? If you want to try something while we discuss fixing it, i would try replacing scorecard/clients/gitlabrepo/repo.go Line 108 in 5dc03b7
Resulting in: client, err := gitlab.NewClient(os.Getenv("GITLAB_AUTH_TOKEN"), gitlab.WithBaseURL(fmt.Sprintf("%s://%s", r.scheme, r.host))) @raghavkaul Thoughts on solving this? My suggested patch in the other comment (#3696 (comment)) wouldn't be enough since the "gitlab" part of the URL is in the path not the host. We could try a Or there's the fix to the "liveness" check I mention in this comment, but are there situations where we should be worried about doing this, in terms of sending a PAT to an instance it may not correspond to? Although I'm not sure any solution would prevent that, as the actual analysis part of the code will send the token. |
Hmm, i wonder if the API link is different from what we're expecting: So with gitlab.com, the API link is: Does https://foo.com/gitlab/api/v4/projects resolve for you (after replacing foo.com with your domain?) If so, the problematic code changes to here, where we just add Lines 59 to 62 in 2ef20f1
|
Hi, yes https://foo.com/gitlab/api/v4/projects resolves for me! I added this code:
Still same error. Token exported. Maybe you are also missing See this curl command works for me: So the URL has to be id: The ID or URL-encoded path of the project Let me know if I can help testing or anything. |
I think we've found the issue. Your code patch was close, but I've pushed a quick workaround to a branch in my fork, where the only patch compared to Can you give that a shot, but make sure you set the GL_HOST=foo.com/gitlab/ go run main.go --repo https://foo.com/gitlab/ssdlc/scorecard-scanner --format json If that works, I'll clean it up before sending a PR. |
And if that complains about "couldn't reach gitlab instance", it probably needs to be combined with token patch above. host := r.host
if h := os.Getenv("GL_HOST"); h != "" {
// avoid duplication of the scheme when constructing baseURL below
host = strings.TrimPrefix(h, r.scheme+"://")
}
baseURL := fmt.Sprintf("%s://%s", r.scheme, host)
client, err := gitlab.NewClient(os.Getenv("GITLAB_AUTH_TOKEN"), gitlab.WithBaseURL(baseURL)) |
Nope, my patch was just an incomplete solution, but we've confirmed the issue in question. Will need to think about how to handle this. Probably would involve changes to the how we parse the repo: scorecard/clients/gitlabrepo/repo.go Line 53 in 45425b6
I might have some time later this week to poke around again. |
Can you give it another shot with the changes I pushed to the same branch today? (sorry for all the "try this", don't have a setup to test on) I made sure to add the token changes too. So hopefully the branch HEAD (d3d5ba6) works. If it doesn't, give HEAD~1 (22d788c) a try too. |
In this particular case, I think the fix is in |
Nice, let me know if I can help review & test the PR... |
Thank you so much! 🫶🏻 |
Describe the bug
Cannot scan self-hosted (private) GitLab repositories
Reproduction steps
Steps to reproduce the behavior:
Shouldnt this be possible acc. to your blogpost?
You can run Scorecard today on a GitLab.com (or self-hosted GitLab) repository by running Scorecard as you normally would
https://openssf.org/blog/2023/08/28/openssf-scorecard-launches-v4-12-with-support-for-gitlab/
Thank you!
The text was updated successfully, but these errors were encountered: