Cannot scan self-hosted (private) GitLab repositories #3696

mwager · 2023-11-28T07:28:55Z

Describe the bug
Cannot scan self-hosted (private) GitLab repositories

Reproduction steps
Steps to reproduce the behavior:

See screenshot

Shouldnt this be possible acc. to your blogpost?

You can run Scorecard today on a GitLab.com (or self-hosted GitLab) repository by running Scorecard as you normally would

https://openssf.org/blog/2023/08/28/openssf-scorecard-launches-v4-12-with-support-for-gitlab/

Thank you!

spencerschrock · 2023-11-28T17:35:03Z

Our code tries to create a GitLab client first, and then falls back to GitHub.

scorecard/checker/client.go

Lines 58 to 75 in 04ea8be

    
           repo, makeRepoError = glrepo.MakeGitlabRepo(repoURI) 
        
           if repo != nil && makeRepoError == nil { 
        
           	repoClient, makeRepoError = glrepo.CreateGitlabClient(ctx, repo.Host()) 
        
           } 
        
           if makeRepoError != nil || repo == nil { 
        
           	repo, makeRepoError = ghrepo.MakeGithubRepo(repoURI) 
        
           	if makeRepoError != nil { 
        
           		return repo, 
        
           			nil, 
        
           			nil, 
        
           			nil, 
        
           			nil, 
        
           			fmt.Errorf("error making github repo: %w", makeRepoError) 
        
           	} 
        
           	repoClient = ghrepo.CreateGithubRepoClient(ctx, logger) 
        
           }

Since it's falling back to GitHub, there was some sort of error making the GitLab one, which we unfortunately aren't surfacing.

My guess is this has to do with auth tokens since we don't pass one in here.

scorecard/clients/gitlabrepo/repo.go

Lines 98 to 114 in 04ea8be

    
           // IsValid implements Repo.IsValid. 
        
           func (r *repoURL) IsValid() error { 
        
           	if strings.Contains(r.host, "gitlab.") { 
        
           		return nil 
        
           	} 
        
           	if strings.EqualFold(r.host, "github.com") { 
        
           		return fmt.Errorf("%w: %s", errInvalidGitlabRepoURL, r.host) 
        
           	} 
        
           	client, err := gitlab.NewClient("", gitlab.WithBaseURL(fmt.Sprintf("%s://%s", r.scheme, r.host))) 
        
           	if err != nil { 
        
           		return sce.WithMessage(err, 
        
           			fmt.Sprintf("couldn't create gitlab client for %s", r.host), 
        
           		) 
        
           	}

spencerschrock · 2023-11-28T17:38:31Z

If you're able to try building locally, can you try this patch?

diff --git a/checker/client.go b/checker/client.go
index 1b5d28a3..d9570773 100644
--- a/checker/client.go
+++ b/checker/client.go
@@ -61,6 +61,7 @@ func GetClients(ctx context.Context, repoURI, localURI string, logger *log.Logge
 	}
 
 	if makeRepoError != nil || repo == nil {
+		fmt.Println(makeRepoError)
 		repo, makeRepoError = ghrepo.MakeGithubRepo(repoURI)
 		if makeRepoError != nil {
 			return repo,
diff --git a/clients/gitlabrepo/repo.go b/clients/gitlabrepo/repo.go
index 65a44783..1a371ed6 100644
--- a/clients/gitlabrepo/repo.go
+++ b/clients/gitlabrepo/repo.go
@@ -97,7 +97,7 @@ func (r *repoURL) String() string {
 
 // IsValid implements Repo.IsValid.
 func (r *repoURL) IsValid() error {
-	if strings.Contains(r.host, "gitlab.") {
+	if strings.Contains(r.host, "gitlab") {
 		return nil
 	}

mwager · 2023-11-30T08:42:44Z

Hey, thanks for the fast answer! I tried to install but I am not very familiar with go, getting lot of errors... I guess you will be faster getting the makeRepoError printed

Your patch makes sense to me, the url looks like this:

https://foo.bar.com/gitlab/org/repo

mwager · 2023-11-30T10:11:40Z

@spencerschrock Building or running on the machine (centos) needed to access the private gitlab instance I am just getting this strange error, and I cannot find any useful info how to fix that:

I tried on my personal laptop adding your print statement, but of course there it is not reachable:

If you know how to fix this asm error I could try again

spencerschrock · 2023-12-06T21:43:34Z

If you know how to fix this asm error I could try again

Hmm, I can't say I've seen the error before. A quick search says it's something preventing you from running the asm binary, either antivirus, or maybe all of the Go binaries aren't in your path, or there's conflicting versions of Go installed?

Maybe check you're seeing the expected things when running:

which go

Depending on the results, you can try setting your GOROOT and GOPATH.

export GOROOT=`which go`
export PATH=$GOROOT/bin:$PATH

Another alternative is compiling on your laptop and transfering the binary to the centos machine (if possible)

mwager · 2023-12-11T10:21:00Z

I fixed it via installing a fresh go bin and running scorecard directly.

Seems like there is an issue reaching the gitlab instance, but strange error regarding "invalid character <" is thrown.

However trying wget the connection works:

Update: I exported a valid gitlab token before running scorecard, like mentioned in the docs export GITLAB_AUTH_TOKEN=glpat-xxxx...

spencerschrock · 2023-12-11T17:51:43Z

Seems like there is an issue reaching the gitlab instance, but strange error regarding "invalid character <" is thrown.

Thanks, I think this error helps confirm the GitLab server is responding with HTML (possibly a 404 since Scorecard isn't using the user-provided token for this call). The library we use is trying to parse the HTML as a JSON response, leading to the invalid character message.

Update: I exported a valid gitlab token

Just to confirm, exporting the token didn't fix it? If you want to try something while we discuss fixing it, i would try replacing "" in this line with os.Getenv("GITLAB_AUTH_TOKEN"), in order to send your PAT to your instance:

scorecard/clients/gitlabrepo/repo.go

Line 108 in 5dc03b7

    
           client, err := gitlab.NewClient("", gitlab.WithBaseURL(fmt.Sprintf("%s://%s", r.scheme, r.host)))

Resulting in:

client, err := gitlab.NewClient(os.Getenv("GITLAB_AUTH_TOKEN"), gitlab.WithBaseURL(fmt.Sprintf("%s://%s", r.scheme, r.host)))

@raghavkaul Thoughts on solving this? My suggested patch in the other comment (#3696 (comment)) wouldn't be enough since the "gitlab" part of the URL is in the path not the host. We could try a GL_HOST env var similar to what we do in the github version of IsValid() (GH_HOST)?

Or there's the fix to the "liveness" check I mention in this comment, but are there situations where we should be worried about doing this, in terms of sending a PAT to an instance it may not correspond to? Although I'm not sure any solution would prevent that, as the actual analysis part of the code will send the token.

mwager · 2023-12-12T07:07:22Z

Oh yes I thought this error reminds me on something :)

I tried your patch passing the token as first paramneter to NewClient(, it does not solve the issue:

spencerschrock · 2023-12-19T15:36:35Z

Hmm, i wonder if the API link is different from what we're expecting:

So with gitlab.com, the API link is: https://gitlab.com/api/v4/ and then the rest of the path, for example: https://gitlab.com/api/v4/projects
But based on your screenshots, your instance is at https://foo.com/gitlab/, so I'm guessing the api link is different.
Maybe something like: https://foo.com/gitlab/api/v4/?

Does https://foo.com/gitlab/api/v4/projects resolve for you (after replacing foo.com with your domain?)

If so, the problematic code changes to here, where we just add api/v4/ to the end of whatever the host is:

scorecard/checker/client.go

Lines 59 to 62 in 2ef20f1

    
           if repo != nil && makeRepoError == nil { 
        
           	repoClient, makeRepoError = glrepo.CreateGitlabClient(ctx, repo.Host()) 
        
           }

mwager · 2023-12-20T07:08:29Z

Hi,

yes https://foo.com/gitlab/api/v4/projects resolves for me!

I added this code:

repoURI is smt like https://foo.com/gitlab/someorg/someproject

Still same error. Token exported.

Maybe you are also missing projects like documented here: https://docs.gitlab.com/ee/api/api_resources.html#project-resources

See this curl command works for me:

So the URL has to be https://foo.com/gitlab/api/v4/projects/namespace%2Fname

See this: https://stackoverflow.com/questions/54717065/get-the-id-of-gitlab-project-via-gitlab-api-or-gitlab-cli

id: The ID or URL-encoded path of the project

Let me know if I can help testing or anything.

spencerschrock · 2023-12-28T21:29:07Z

yes https://foo.com/gitlab/api/v4/projects resolves for me!
See this curl command works for me:

I think we've found the issue. Your code patch was close, but repo.Host() only takes the (sub)domain portion of the URL (foo.com in this case), which doesn't include the gitlab part of your URL. So you still ended up with https://foo.com/api/v4/ instead of https://foo.com/gitlab/api/v4/

I've pushed a quick workaround to a branch in my fork, where the only patch compared to main is 22bc273.

Can you give that a shot, but make sure you set the GL_HOST environment variable to foo.com/gitlab/

GL_HOST=foo.com/gitlab/ go run main.go --repo https://foo.com/gitlab/ssdlc/scorecard-scanner --format json

If that works, I'll clean it up before sending a PR.

spencerschrock · 2023-12-28T21:38:56Z

And if that complains about "couldn't reach gitlab instance", it probably needs to be combined with token patch above.

host := r.host
if h := os.Getenv("GL_HOST"); h != "" {
	// avoid duplication of the scheme when constructing baseURL below
	host = strings.TrimPrefix(h, r.scheme+"://")
}
baseURL := fmt.Sprintf("%s://%s", r.scheme, host)
client, err := gitlab.NewClient(os.Getenv("GITLAB_AUTH_TOKEN"), gitlab.WithBaseURL(baseURL))

mwager · 2023-12-29T09:46:37Z

Tried your patch, incl the token patch above. Looks better now, now it is using the host including /gitlab but then also tries to fetch gitlab/ssdlc/scorecard-scanner instead of just ssdlc/scorecard-scanner

Did I do smt wrong?

spencerschrock · 2024-01-09T19:07:35Z

Tried your patch, incl the token patch above. Looks better now, now it is using the host including /gitlab but then also tries to fetch gitlab/ssdlc/scorecard-scanner instead of just ssdlc/scorecard-scanner

Did I do smt wrong?

Nope, my patch was just an incomplete solution, but we've confirmed the issue in question. Will need to think about how to handle this. Probably would involve changes to the how we parse the repo:

scorecard/clients/gitlabrepo/repo.go

Line 53 in 45425b6

func (r *repoURL) parse(input string) error {

I might have some time later this week to poke around again.

spencerschrock · 2024-01-19T23:22:54Z

Can you give it another shot with the changes I pushed to the same branch today? (sorry for all the "try this", don't have a setup to test on)

I made sure to add the token changes too. So hopefully the branch HEAD (d3d5ba6) works.

If it doesn't, give HEAD~1 (22d788c) a try too.

mwager · 2024-01-23T06:53:47Z

No problem, I would be very glad to help :)

I pulled your repo, and did git checkout fix/gitlab-host-path

Looks good!

Throws an error after the results:

spencerschrock · 2024-01-23T18:51:45Z

Throws an error after the results

In this particular case, I think the fix is in main already (I haven't pulled in updates to this branch since I forked it ~1 month ago). I'll work later this week on getting this up as a PR

mwager · 2024-01-24T06:04:41Z

Nice, let me know if I can help review & test the PR...

mwager · 2024-02-01T06:30:22Z

Thank you so much! 🫶🏻

mwager added the kind/bug Something isn't working label Nov 28, 2023

spencerschrock added the gitlab Issue related to Scorecard's GitLab client label Nov 28, 2023

spencerschrock mentioned this issue Jan 25, 2024

🐛 Support self-hosted GitLab instances where base URL has a path component #3819

Merged

2 tasks

spencerschrock closed this as completed in #3819 Jan 31, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot scan self-hosted (private) GitLab repositories #3696

Cannot scan self-hosted (private) GitLab repositories #3696

mwager commented Nov 28, 2023 •

edited

Loading

spencerschrock commented Nov 28, 2023

spencerschrock commented Nov 28, 2023

mwager commented Nov 30, 2023 •

edited

Loading

mwager commented Nov 30, 2023

spencerschrock commented Dec 6, 2023

mwager commented Dec 11, 2023 •

edited

Loading

spencerschrock commented Dec 11, 2023

mwager commented Dec 12, 2023 •

edited

Loading

spencerschrock commented Dec 19, 2023

mwager commented Dec 20, 2023

spencerschrock commented Dec 28, 2023

spencerschrock commented Dec 28, 2023 •

edited

Loading

mwager commented Dec 29, 2023

spencerschrock commented Jan 9, 2024

spencerschrock commented Jan 19, 2024 •

edited

Loading

mwager commented Jan 23, 2024

spencerschrock commented Jan 23, 2024

mwager commented Jan 24, 2024

mwager commented Feb 1, 2024

Cannot scan self-hosted (private) GitLab repositories #3696

Cannot scan self-hosted (private) GitLab repositories #3696

Comments

mwager commented Nov 28, 2023 • edited Loading

spencerschrock commented Nov 28, 2023

spencerschrock commented Nov 28, 2023

mwager commented Nov 30, 2023 • edited Loading

mwager commented Nov 30, 2023

spencerschrock commented Dec 6, 2023

mwager commented Dec 11, 2023 • edited Loading

spencerschrock commented Dec 11, 2023

mwager commented Dec 12, 2023 • edited Loading

spencerschrock commented Dec 19, 2023

mwager commented Dec 20, 2023

spencerschrock commented Dec 28, 2023

spencerschrock commented Dec 28, 2023 • edited Loading

mwager commented Dec 29, 2023

spencerschrock commented Jan 9, 2024

spencerschrock commented Jan 19, 2024 • edited Loading

mwager commented Jan 23, 2024

spencerschrock commented Jan 23, 2024

mwager commented Jan 24, 2024

mwager commented Feb 1, 2024

mwager commented Nov 28, 2023 •

edited

Loading

mwager commented Nov 30, 2023 •

edited

Loading

mwager commented Dec 11, 2023 •

edited

Loading

mwager commented Dec 12, 2023 •

edited

Loading

spencerschrock commented Dec 28, 2023 •

edited

Loading

spencerschrock commented Jan 19, 2024 •

edited

Loading