feat: use repo search to select repositories #397

jamestelfer · 2023-10-24T11:27:29Z

What does this change

Using the --repo-search option, it is possible to supply a GitHub repository search query that is used to generate the set of repositories to process.

This feature is GItHub-only at the moment.

This allows for searches that reference:

partial names with boolean queries: owner:buildkite-plugins in:name docker AND plugin
one or multiple owners owner:cultureamp owner:jamestelfer fork:true in:name plugin
README contents
license

I initially started this as a "warm up" to implementing something similar with code search, but the partial name search and license search are potentially quite useful in themselves.

Obviously this means that search syntax and results depend on the underlying source being queried, so implementations for GitLab et al would be required to see this feature properly take root.

References:

GitHub repository search API
GitHub repo search qualifiers
GitHub search query syntax (this is the legacy search, as the API does not yet use the new search engine).

$ multi-gitter run examples/general/test.sh --repo-search "user:jamestelfer fork:only" --dry-run --commit-message "chore: foo"

<log output elided />
INFO[0102] Cloning and running script                    repo=jamestelfer/multi-gitter
INFO[0103] Script output: jamestelfer/multi-gitter       repo=jamestelfer/multi-gitter
DEBU[0103] diff --git a/.features.yaml b/.features.yaml
new file mode 100644
index 0000000000000000000000000000000000000000..73311e25f5ad0766cd0576c7302b96f9bc76a4b7
--- /dev/null
+++ b/.features.yaml
@@ -0,0 +1,3 @@
+
+features: ~
+
INFO[0103] Skipping pushing changes because of dry run   repo=jamestelfer/multi-gitter
Repositories with a successful run:
  jamestelfer/jenkins-inheritance-plugin #0
  jamestelfer/slow-cheetah.xdt #0
  jamestelfer/global-build-stats-plugin #0
  jamestelfer/wheel-of-misfortune #0
  jamestelfer/blast-1 #0
  jamestelfer/cloudformation-cli #0
  jamestelfer/step-templates-buildkite-plugin #0
  jamestelfer/fusionauth-site #0
  jamestelfer/aws-lambda-go #0
  jamestelfer/testza #0
  jamestelfer/avro #0
  jamestelfer/ca-go #0
  jamestelfer/rocketcrab #0
  jamestelfer/terraform-cdk #0
  jamestelfer/aws-cdk #0
  jamestelfer/multi-gitter #0

What issue does it fix

Closes #398

Refer also to original discussion.

Notes for the reviewer

Happy for feedback/changes of direction.

Checklist

Made sure the PR follows the CONTRIBUTING.md guidelines
Tests if something new is added

Using the `--repo-search` option, it is possible to supply a GitHub repository search query that is used to generate the set of repositories to process.

jamestelfer · 2023-10-25T04:07:31Z

@lindell I've managed to get some tests on the go, so I've taken this out of draft. Very open to feedback and changes of direction, let me know what you think!

lindell · 2023-10-25T06:52:34Z

Nice! Will try to take a look this evening :D

lindell

Thanks for the contribution!

It looks very good. The biggest concern I have is accidentally writing a query that returns too many results. Which will hit the incomplete results error, and give the user an error that they don't know what to do with. Do you have any ideas about it?

internal/scm/github/github.go

cmd/platform.go

lindell · 2023-10-25T18:35:17Z

PS. I fixed the linting timeout. If you merge master, it should fix the error.

This is the default behaviour of the repo search API.

jamestelfer · 2023-10-25T22:32:42Z

The biggest concern I have is accidentally writing a query that returns too many results. Which will hit the incomplete results error, and give the user an error that they don't know what to do with. Do you have any ideas about it?

Fair point, and to be honest one that I hadn't considered enough.

The doco states that:

To keep the REST API fast for everyone, we limit how long any individual query can run. For queries that exceed the time limit, the API returns the matches that were already found prior to the timeout, and the response has the incomplete_results property set to true.

We can handle this by either:

failing with an error that directs the user to refine their search (so that it's less likely to return incomplete results), or
continue with the incomplete results and emit a warning.

In my opinion, the warning will likely get lost, and the user will get unexpected results instead. I'm going to go ahead and refine the error message (option 1), but I'm happy to change this if you would like (2) or you have a different approach.

Now suggests possible remediation for the user, and includes a test for this case.

jamestelfer · 2023-10-25T22:47:19Z

I've updated it now so the error message reads:

search timed out on GitHub and was marked incomplete: try refining the search to return fewer results or be less complex

Let me know what you think.

The test was flakey because the times were all the same and map ordering is non-deterministic. Changing the time slightly for each ensures that they're going to come out in order.

lindell

I think the new error for incomplete results is good 👍

When playing around with it a bit further, for the query "org:google", where the total repos returned is 2569. It did not error on incomplete results, but instead

GET https://api.github.com/search/repositories?page=11&per_page=100&q=fork%3Atrue+org%3Agoogle: 422 Only the first 1000 search results are available

This message came after making 10 requests.

So we should also add a check if the total results are >1000 and error right away in that case.

internal/scm/github/github.go

Wraps up nil check, easier to read.

The search will eventually fail, so short circuit the waiting for the user.

jamestelfer · 2023-10-26T05:29:45Z

Fantastic feedback, much appreciated. Thanks for testing out the upper bound like that, it's much better to get a fast failure.

lindell

LGTM 👍 Thanks for the contribution!

github-actions · 2023-10-27T04:44:05Z

Included in release v0.48.0 🎉

jamestelfer added 2 commits October 24, 2023 18:42

feat: use repo search to select repositories

5f92bbb

Using the `--repo-search` option, it is possible to supply a GitHub repository search query that is used to generate the set of repositories to process.

test: add tests to exercise search function

a704080

jamestelfer marked this pull request as ready for review October 25, 2023 04:05

fix: always include forks in repo search

7da2928

jamestelfer mentioned this pull request Oct 25, 2023

feat: use code search to select repositories #399

Merged

2 tasks

josealdaco mentioned this pull request Oct 25, 2023

feat: Including Regular Expression filter when targeting Github Repositories josealdaco/multi-gitter#1

Closed

2 tasks

lindell reviewed Oct 25, 2023

View reviewed changes

internal/scm/github/github.go Outdated Show resolved Hide resolved

internal/scm/github/github.go Outdated Show resolved Hide resolved

cmd/platform.go Outdated Show resolved Hide resolved

jamestelfer added 4 commits October 26, 2023 08:39

fix: exclude forks by default from repo search results

b7b4f21

This is the default behaviour of the repo search API.

docs: clarified repo-search flag doco

59bd196

fix: check empty string with equality not length

9c305b2

Merge branch 'lindell:master' into use-repo-search

29be33f

jamestelfer requested a review from lindell October 25, 2023 22:25

fix: improve error for incomplete repo search

5d8a137

Now suggests possible remediation for the user, and includes a test for this case.

test: use date to fix ordering of results

9474644

The test was flakey because the times were all the same and map ordering is non-deterministic. Changing the time slightly for each ensures that they're going to come out in order.

lindell reviewed Oct 26, 2023

View reviewed changes

internal/scm/github/github.go Outdated Show resolved Hide resolved

fix: use accessor on GitHub struct

c8d13c4

Wraps up nil check, easier to read.

jamestelfer force-pushed the use-repo-search branch from 8a023ac to c8d13c4 Compare October 26, 2023 05:18

fix: fail early when > 1000 results are found

9000581

The search will eventually fail, so short circuit the waiting for the user.

jamestelfer requested a review from lindell October 26, 2023 05:29

lindell approved these changes Oct 27, 2023

View reviewed changes

lindell merged commit 0f8c2dc into lindell:master Oct 27, 2023
8 checks passed

github-actions bot mentioned this pull request Oct 27, 2023

chore(master): release 0.48.0 #389

Merged

jamestelfer mentioned this pull request Nov 24, 2023

feat: support CVSS3 scores in rendered annotation cultureamp/ecr-scan-results-buildkite-plugin#26

Merged

This was referenced Jan 6, 2024

chore(master): release 1.0.0 #448

Closed

chore(master): release 1.0.0 #449

Closed

github-actions bot mentioned this pull request Jul 2, 2024

chore(master): release 0.52.1 #491

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: use repo search to select repositories #397

feat: use repo search to select repositories #397

jamestelfer commented Oct 24, 2023 •

edited

Loading

jamestelfer commented Oct 25, 2023

lindell commented Oct 25, 2023

lindell left a comment •

edited

Loading

lindell commented Oct 25, 2023

jamestelfer commented Oct 25, 2023

jamestelfer commented Oct 25, 2023

lindell left a comment •

edited

Loading

jamestelfer commented Oct 26, 2023

lindell left a comment

github-actions bot commented Oct 27, 2023

feat: use repo search to select repositories #397

feat: use repo search to select repositories #397

Conversation

jamestelfer commented Oct 24, 2023 • edited Loading

What does this change

What issue does it fix

Notes for the reviewer

Checklist

jamestelfer commented Oct 25, 2023

lindell commented Oct 25, 2023

lindell left a comment • edited Loading

Choose a reason for hiding this comment

lindell commented Oct 25, 2023

jamestelfer commented Oct 25, 2023

jamestelfer commented Oct 25, 2023

lindell left a comment • edited Loading

Choose a reason for hiding this comment

jamestelfer commented Oct 26, 2023

lindell left a comment

Choose a reason for hiding this comment

github-actions bot commented Oct 27, 2023

jamestelfer commented Oct 24, 2023 •

edited

Loading

lindell left a comment •

edited

Loading

lindell left a comment •

edited

Loading