RF: forum post on RF5 metrics

[ccerv1]

What is it?

Explain the metrics, where they come from, how they're calculated

[ccerv1]

Draft...

Open Source Observer is happy to support the latest Retro Funding round with another suite of impact metrics.

As RF5 features OP Stack contributions, the metrics we’ve developed for the round are focused on GitHub repo-level contributions. They are intended to give human voters some helpful “data in the loop” as a complement to projects’ self-reported impact statements.

The metrics comes in three flavors:

• Basic vanilla GitHub stats. We took a snapshot of projects’ GitHub repo stats (stars, forks, etc) at the end of the review process (2024-09-23). Nothing fancy here. You can double-check any of these by clicking on the project’s repo link.
• Contributor counts. We indexed the event history for each repo to identify unique, non-bot contributors over the lifetime and the past 6 months of the project. We also present a count of “trusted” contributors.
• Trust weighted metrics. With our friends at OpenRank, we implemented a “developer rank” and a “repo rank” algorithm. The algorithm considers contributions to over 40K repos, seeded by a set of “high impact” repos identified by core devs. These trust-weighted metrics offer a better approximation of the impact a project has had specifically on the Optimism ecosystem versus the broader Ethereum / open source universe.

As always, you can see the source code and underlying data behind metrics here.

A few caveats:

• The metrics come exclusively from public GitHub activity.
• The majority - but not all - projects in the round submitted GitHub repos as part of their impact statements.
• Metrics are presented only at the repo level (we don’t roll them up to the project level). Some projects submitted more than one repo and therefore have multiple sets of metrics.
• The underlying data behind the metrics comes from gharchive. We only capture contributions to public repos; we don’t capture any private contributions that happened before the repo turned public. This may cause minor differences between the metrics on the project’s voting page and the ones you see on the project’s actual GitHub.

Metrics are continuously evolving. Feedback or questions on our GitHub forum are always welcome!

The Metrics

The following fields are included in the GitHub-based metrics analysis for Retro Funding 5:

1. artifact_url: Repository URL associated with the project based on the application. A project may have multiple repos in its application.
2. project_name: Name of the project.
3. project_category_id: Category ID of the project (1, 2, 3).
4. num_contributors: Total number of contributors before August 1, 2024. A contributor is defined as any non-bot user that has contributed to the repository (since 2015) by committing code directly to a repository, opening an issue, or opening/reviewing a pull request.
5. num_trusted_contributors: Number of trusted contributors before August 1, 2024. A subset of the contributors defined above, this is the number of contributors that are also in the top 420 of the OpenRank developer trust score.
6. num_contributors_last_6_months: Number of contributors over the period Feburary 1, 2024 - August 1, 2024.
7. num_stars: Total number of stars as of September 23, 2024.
8. num_trusted_stars: Number of stars from trusted users before August 1, 2024. A subset of the stars defined above, this is the number of stars from users that are also in the top 420 of the OpenRank developer trust score.
9. trust_weighted_stars: This metric is a percentage score between 0% and 100%, representing the sum of the reputation share of the developers who starred the repo. If all developers in OpenRank developer ranking have starred a particular repo, the metric's value is going to be 100% for this particular repo. The more and the higher ranked developer who starred the repo, the higher the percentage value, the higher impact and quality of this repo. We calculate this metric by first calculating every developer's reputation share (%) based on their OpenRank score, then sum them up if they starred the target repo.
10. num_forks: Total number of forks as of September 23, 2024.
11. num_trusted_forks: Number of forks from trusted users before August 1, 2024. A subset of the forks defined above, this is the number of forks from users that are also in the top 420 of the OpenRank developer trust score.
12. trust_weighted_forks: This metric is a percentage score between 0% and 100%, representing the sum of the reputation share of the developers who forked the repo. If all developers in OpenRank developer ranking have forked a particular repo, the metric's value is going to be 100% for this particular repo. The more and the higher ranked developer who forked the target repo, the higher the percentage value, the higher impact and quality of this repo. We calculate this metric by first calculating every developer's reputation share (%) based on their OpenRank score, then sum them up if they forked the target repo.
13. trust_rank_for_repo_in_category: Ranking of the repository's OpenRank trust score within its category. A score of 1 indicates the highest ranking repo in its category.
14. age_of_project_years: Age of the project in years, measured from the project's first public commit to August 1, 2024.
15. license(s): License(s) used by the project.
16. application_id: Application ID in the sign-up and voting UIs.