Cache and store primer packages via the default branch

[DanielNoord]

• Add yourself to CONTRIBUTORS if you are a new contributor.
• Write a good description on what the PR does.

Type of Changes

	Type
✓	🔨 Refactoring

Description

Ref. #5364.

This is in preparation of a full fledged primer which can warn us about false positives and negatives.

This adds:

1. A workflow triggered on every push to main that caches and downloads all packages in tests/primer/packages_to_prime.json.
2. A workflow triggered on all relevant PRs that searches for the last successful run on main and gets its cache from there.

The cache is managed by what I call the commitstring, which is just a concatenation of all commits that the packages to prime are on.

Because this needs to be merged into main to work you can see a working example here:
Job on main:
https://github.com/DanielNoord/pylint/runs/6606171195?check_suite_focus=true
PR that gets triggered after that push:
DanielNoord#145
Successful job on the PR:
https://github.com/DanielNoord/pylint/runs/6606228882?check_suite_focus=true

You can look at the restore cache jobs to see that this works.

In itself this PR doesn't really add much, but the extra run time is about 60 seconds for both jobs together and only about 20 seconds for a PR. Since the workflow for primer depends on stuff being already merged on main I'd like to do some preparation PRs to build on like this.

[coveralls]

Pull Request Test Coverage Report for Build 2396032378

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 95.42%

---

Totals
Change from base Build 2386903518:	0.0%
Covered Lines:	16229
Relevant Lines:	17008

---

💛 - Coveralls

[Pierre-Sassoulas]

Should we set the commit we're analyzing for each repositories ? I.e. we set the sha for each package to prime. This way we can't have a new message that appears simply because something was added on the primed repo. We could upgrade the primer repo to the latest version before release so we're still somewhat up to date.

[DanielNoord]

Should we set the commit we're analyzing for each repositories ? I.e. we set the sha for each package to prime. This way we can't have a new message that appears simply because something was added on the primed repo. We could upgrade the primer repo to the latest version before release so we're still somewhat up to date.

I don't think so.

The idea is to run main over a set of packages on specific commit. That commit is the latest commit of each package on their default branch at the time of the run. Then main records those commits as the commitstring.
A PR then gets the commitstring and retrieves the cache created by main and validates it. That way main and PR are both on the same commits of the packages (so even if the package adds new commits between the main and the PR run, those don't affect us).

Lastly, we run PR over the packages and compare its output with main. Notifying of any added messages or any removed messages. In that way it is similar to the current primer: we don't really care about which messages are emitted when with --enable=all, --enable-all-extensions. We only care about the difference between main and PR.

[jacobtylerwalls]

Should we add a .gitignore to this directory to ensure it always exists? I'm thinking for local testing etc.

[Pierre-Sassoulas]

Isn't this a .gitkeep ?

[DanielNoord]

Added!

[jacobtylerwalls]

Probably should add to the project's .gitignore.

[DanielNoord]

I have changed this to go into the primer directory instead of writing to root.

[DanielNoord]

Test run on my fork main:
https://github.com/DanielNoord/pylint/runs/6624159814
And subsequent run of the PR workflow on a PR:
https://github.com/DanielNoord/pylint/runs/6624174015

[Pierre-Sassoulas]

This is pretty great ! We'll have even more confidence when releasing.